upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-09-30 18:08:18 -05:00
Code Issues Packages Projects Releases Wiki Activity
1,077 Commits 437 Branches 119 Tags
consensus-bootstrap-fail
Commit Graph
34 Commits
Author SHA1 Message Date
Andrey Vasnetsov dad331fc26 fix colletion params update (#1208) 2022-11-09 13:10:54 +01:00
Arnaud GourlayEgor IvkovAndrey Vasnetsov
 		286ca9f5e4 0.11.0 cherry picks (#1174) 
* implement concern-factor, forbig disabling the last node, improve tes… (#1168)

* implement concern-factor, forbig disabling the last node, improve test stability

* upd docs

* improve test_recover_dead_node

* rename to write_consistency_factor

* Fix bug: applied index higher than commit on restart (#1172)

* create collection with default status = dead + await for activation on create_collecton_op_submit

* fmt

* fix unit tests

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>

* upd integration tests

* fix transaction handling

* upd protobuf

* upd to v0.11.0

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>
 2022-10-27 10:16:21 +02:00
Egor IvkovandArnaud Gourlay c6c0afa781 [replication] Suggest replica set changes on repl_factor change (#1059) 
* Suggest replica set changes on repl_factor change

* Review fix: account for replication_factor being 1

* Update lib/collection/src/collection.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update lib/storage/src/content_manager/toc.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Review fix: accept higher repl factor than n peers

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-09-28 17:05:11 +03:00
Egor Ivkov 9a3fad44b2 Revert "Revert replication changes (#1055)" (#1056) 
This reverts commit a2ed0cf5db.
 2022-09-23 16:46:06 +03:00
Egor IvkovandAndrey Vasnetsov a2ed0cf5db Revert replication changes (#1055) 
* Revert "Replication set `Change` enum"

This reverts commit 28e8487542.

* Revert changes (#1053)

* Revert "Generate API - replication follow up (#1043)"

This reverts commit 183ea9e5e3.

* Revert "Replication related API changes (#1037)"

This reverts commit d690b700ff.

* Revert "Persist replica state (#1035)"

This reverts commit 31c3ae482d.

* Revert "[replication] Account for replicas in shard distribution (#1009)"

This reverts commit e332d482b0.

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-09-23 16:33:07 +03:00
Egor Ivkov d690b700ff Replication related API changes (#1037) 
* Replication related API changes

* minor proto file fix
 2022-09-21 18:49:47 +03:00
Andrey VasnetsovandArnaud Gourlay dc07b01e1f remove deprecated fields from API (#1030) 
* remove depricated fields from API

* fmt

* upd openapi and integration tests

* fix grpc test

* regenerate storage reference data

* improve docs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-09-19 13:51:03 +02:00
Ivan PleshkovandAndrey Vasnetsov f6b2186193 [WIP] Many named vectors per point (#958) 
* many named vectors per point (segment-level)

* operation result for dim function

* beautifulized vector name

* fix naming bug

* segment version migration

* fmt

* add segment tests

* are you happy clippy

* fix build

* [WIP] many named vectors per point (collection-level) (#975)

* config and search

* fix placeholders for proxy segment move

* remove VectorType from collection

* are you happy fmt

* vectors in grps messages

* create collections with vectors

* segment holder fixes

* are you happy fmt

* remove default vector name placeholders

* are you happy fmt

* are you happy clippy

* fix build

* fix web api

* are you happy clippy

* are you happy fmt

* record vector&vectors

* openapi update

* fix openapi integration tests

* segment builder fix todo

* vector names for update from segment

* remove unwrap

* backward compatibility

* upd openapi

* backward compatible PointStruct

* upd openapi

* fix record back-comp

* fmt

* vector configuration backward compatibility

* fix vetor storage size estimation

* fmt

* multi-vec segment test + index test

* fmt

* api integration tests

* [WIP] Named vectors struct (#1002)

* move to separate file

* named vectors as struct

* use cow

* fix build

* keys iterator

* avoid copy in PointStruct -> get_vectors

* avoid another copy

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-09-11 20:59:23 +02:00
dependabot[bot] a261d4f25e Bump tokio from 1.20.1 to 1.21.0 (#983) 2022-09-05 09:31:21 +00:00
Russ Cam 8e1f2ca353 Use tempfile (#922) 
This commit replaces tempdir with tempfile.
tempdir is archived.

Closes #544
 2022-08-10 09:39:21 +02:00
Andrey Vasnetsov c53f19c0dd Disable mmap threshold (#877) 
* allow to disable mmap by not specifying mmap threshold

* fmt

* smart default max_segment_size

* upd openapi for max_segment_size
 2022-08-01 15:18:40 +02:00
Andrey VasnetsovandEgor Ivkov 38cf822518 Shard transmission (#830) 
* forward proxy shard

* fmt

* transmission routine

* fmt

* add channel service to collection

* WIP: shard transfer handlelling

* WIP: transfer task pool

* WIP: fmt

* fmt

* handle commands from consensus: start, finish & abort transmission

* fmt

* initiate temp shard on remote

* Update lib/collection/src/collection.rs

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>

* Update lib/collection/src/collection.rs

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>

* review fixes

* fmt

* review fixes 2

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>
 2022-07-27 10:12:00 +02:00
Andrey Vasnetsov bd02f835aa Refactoring: Split consensus state into multiple files (#838) 
* fmt

* move dispatcher into separate file

* fmt
 2022-07-21 16:18:40 +02:00
Egor Ivkov 026bd040b0 Add import formatting rules (#820) 
* Add import formatting rules

* Review fix: update rusty hook
 2022-07-15 15:42:25 +03:00
Andrey VasnetsovandArnaud Gourlay fac87018c4 Snapshoting API (#764) 
* wip: rest api for snapshots

* fmt

* fix tests

* fmt

* wip: collection snapshoting and recovery

* fmt

* remote shard snapshots

* remote shard test

* fmt

* extend proxy snapshot test + fix double read lock

* fmt + clippy

* openapi schema

* Update openapi/openapi-snapshots.ytt.yaml

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update openapi/openapi-snapshots.ytt.yaml

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update src/main.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update src/main.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* reduce writes on snapshots location

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-07-01 13:21:57 +02:00
Egor Ivkov 533fd208b6 Move out consensus logic from ToC (#675) 2022-06-15 16:18:26 +03:00
Andrey Vasnetsov fc2f579ec6 smarter default number of shards (#668) 
* smarter default number of shards

* upd openapi
 2022-06-07 16:59:10 +02:00
Gabriel VeloIvan PleshkovAndrey Vasnetsov
 		c15981092a [WIP] [real-time index] Implement payloadstorage for structpayloadindex (#642) 
* [real-time index] Extend FieldIndex enum and StructPayloadIndex with method from PayloadStorage

* [real-time index] add missing remove_point methods

* [real-time index] add new index to FieldIndex enum

* fix compile

* are you happy fmt

* merge load and remove

* fix test generics

* decrement points count

* remove from histogram

* simplify histogram usage

* [real-time index] remove old tests and fix clippy warnings

* histogram: method to derive range by size (#657)

* [real-time index] add histogram based payload_blocks implementation.

* payload blocks

* fmt

* clippy

* [real-time index] refactor Segment to use PayloadIndex instead of PayloadStorage.

* fix tests

* fmt

* clippy

* rename indexes

* remove redundent params

* add struct payload deletion test + fix delete payload in map index

* remove payload threshold

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-06-06 17:14:20 +02:00
Andrey Vasnetsov 1b458780eb On disk payload storage (#634) 
* implement on-disk payload storage

* fmt + clippy

* config param for on-disk payload storage

* upd openapi definitions

* add integration test with on-disk payload

* fix clippy

* review fixes

* fmt
 2022-06-01 17:23:34 +02:00
Arnaud Gourlay c7396525f9 Switch consensus to a runtime configuration (#565) 
* switch consensus to a runtime configuration

* propose_sender is always present

* better toggle switch with ConsensusEnabled
 2022-05-06 17:56:00 +02:00
Egor Ivkov fd1b8a457c Adds this_peer_id generation as part of raft state (#503) 
Also some raft state refactoring
 2022-04-25 12:14:01 +03:00
Egor Ivkov 99dd3d4704 Adds PeerAddressById to StorageConfig (#501) 2022-04-21 19:02:14 +03:00
Egor Ivkov d0d1f615f2 Fixes test_alias_operation (#497) 2022-04-20 18:04:51 +03:00
Egor Ivkov aeea31dfa4 Wait for operation commit (#436) 
* Wait for operation commit

* Review: Option with timeout, return execution err to API

* Review: Add comment about error forwarding in ???
 2022-04-06 18:00:44 +03:00
Egor Ivkov d3e0d7dcec Integrate consensus into execution pipeline (#410) 
* Integrate consensus into execution pipeline

* Review: Remove unnecessary config
 2022-03-31 17:54:55 +03:00
Egor Ivkov 2343078ea1 Collection meta as global state (#394) 
* Collection meta as global state

* Improve collection wal entry error handling
 2022-03-22 18:27:03 +03:00
Egor Ivkov 960937b58b Sharding related collection API changes (#381) 
* NonZeroU32 shard number

* Inconsistent Failure option

* Generate docs and apis
 2022-03-11 19:26:09 +03:00
Andrey VasnetsovandArnaud Gourlay c6441f06c1 [gPRC] docs + refactoring (#363) 
* wip: doc generator script

* fix enum default value + CollectionInfo upd + collections comments

* refactor grpc + generate docs

* disable clippy for auto-generated files

* Update src/tonic/proto/points.proto

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-03-03 15:30:09 +01:00
Egor Ivkov 3b62670a1a Introduce Shard abstraction (#318) 
* Introduce Shard abstraction

* Optimizer thread name improved and avg_vectors moved
 2022-02-18 17:51:44 +03:00
Andrey Vasnetsov 898b692f39 [WIP] Limit segment size #135 (#195) 
* add parameters to optimizer config

* benchmark search speed in different segment sizes

* use constructor for FilteredScorer

* * Implement benchmarks for HNSW index search with different number of
  stored points
* Fix minor issue in HNSW graph edge assignment
* Update profiler with call-graph report generation
* Add profiling guide
* Add HNSW graph statistics test function (debug inly)

* limit resulting segment size in merge optimizer

* fix clippy

* stop the music

* fix clippy once again

* fmt once again
 2022-01-18 15:49:35 +01:00
Andrey Vasnetsov d7148126ce Split collection update API into several endpoints (#126) 
* split storage operation structures #32

* cargo fmt #32

* split collection update api into several endpoints #32

* cargo fmt #32

* fix tonic-related code with new structures

* upd alias structures

* use ytt teplate engine for OpenAPI Endpoint schema generation
 2021-12-13 09:38:33 +01:00
Anton VandAndrey Vasnetsov eb786ab64b Multithread optimizer (#134) 
* run optimizers on tokio thread pool for cpu-bound tasks

* [WIP] move check condition to another thread

* [WIP] optimizer iter live not long enough

* [WIP] change Box to Arc in optimizers vector

* add blocking handles management

* cargo fmt apply

* Update lib/collection/src/update_handler.rs

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

* [WIP] optimizer iter live not long enough

* [WIP] change Box to Arc in optimizers vector

* add blocking handles management

* fix code review issues

* use CollectionConfig available cpu value

* apply updated fmt

* [WIP] move count of optimization threads to OptimizersConfig

* optimization options

* fix formatting

* fix proto for optimizer config

* fmt

* Update config/config.yaml

related task: https://github.com/qdrant/qdrant/issues/30

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2021-12-09 18:18:20 +01:00
Marcin Puc c1a790a7af Add more refactorings (#122) 2021-11-12 19:16:00 +01:00
Andrey Vasnetsov cca7839e18 fix alias operation deadlock (#103) (#105) 
* fix alias operation deadlock (#103)

* cargo fmt (#103)
 2021-10-06 18:50:51 +03:00