upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-09-21 13:37:46 -05:00
Code Issues Packages Projects Releases Wiki Activity
6,949 Commits 438 Branches 119 Tags
edge-docs-diff
Commit Graph
84 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Egor Ivkov a96f92cbfe Record message send failures and stop consensus on service error (#1152) 
* Record message send failures and stop consensus on service error

* Generate openapi

* Review fixes: log first error and remove on success
 2022-10-22 23:23:37 +02:00
Egor Ivkov d3849d712b Allow peer force removal (#1139) 
* Allow peer force removal

* Remove shards
 2022-10-19 17:01:13 +02:00
Egor Ivkov da039fd533 Telemetry data for replica set shard (#1115) 2022-10-11 16:31:02 +03:00
Egor Ivkov e0b64e61bf Fix: Do not rely on empty message for is_leader_established flag (#1103) 
* Fix: Do not rely on empty message for `is_leader_established` flag

* Record last consensus update

* Update OpenAPI
 2022-10-11 09:53:38 +02:00
Egor Ivkov 03ec804bef Adds panic hook (#1106) 2022-10-07 19:17:08 +03:00
Egor Ivkov ca01b8ac8e Fix: Do not rely on empty message for is_leader_established flag (#1103) 
* Fix: Do not rely on empty message for `is_leader_established` flag

* Record last consensus update

* Update OpenAPI
 2022-10-05 18:10:33 +03:00
Egor Ivkov 4a3dd91530 Fix minor todos in for ReplicaSet (#1091) 2022-10-04 18:34:28 +03:00
Egor Ivkov db39c07d05 Add sync and sync_source for transfers (#1073) 
* Add `sync` and `sync_source` for transfers
 2022-09-29 18:06:47 +03:00
Egor IvkovandArnaud Gourlay c6c0afa781 [replication] Suggest replica set changes on repl_factor change (#1059) 
* Suggest replica set changes on repl_factor change

* Review fix: account for replication_factor being 1

* Update lib/collection/src/collection.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update lib/storage/src/content_manager/toc.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Review fix: accept higher repl factor than n peers

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-09-28 17:05:11 +03:00
Egor Ivkov 9a3fad44b2 Revert "Revert replication changes (#1055)" (#1056) 
This reverts commit a2ed0cf5db.
 2022-09-23 16:46:06 +03:00
Egor IvkovandAndrey Vasnetsov a2ed0cf5db Revert replication changes (#1055) 
* Revert "Replication set `Change` enum"

This reverts commit 28e8487542.

* Revert changes (#1053)

* Revert "Generate API - replication follow up (#1043)"

This reverts commit 183ea9e5e3.

* Revert "Replication related API changes (#1037)"

This reverts commit d690b700ff.

* Revert "Persist replica state (#1035)"

This reverts commit 31c3ae482d.

* Revert "[replication] Account for replicas in shard distribution (#1009)"

This reverts commit e332d482b0.

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-09-23 16:33:07 +03:00
Egor Ivkov cf2a2f3022 Replication set Change enum (#1052) 2022-09-23 13:24:12 +03:00
Egor Ivkov 183ea9e5e3 Generate API - replication follow up (#1043) 
Also removes `JsonSchema` from structs not exposed in API
 2022-09-22 14:53:00 +03:00
Egor Ivkov d690b700ff Replication related API changes (#1037) 
* Replication related API changes

* minor proto file fix
 2022-09-21 18:49:47 +03:00
Egor Ivkov e332d482b0 [replication] Account for replicas in shard distribution (#1009) 
* Account for replicas in shard distribution

* fix fmt

* Review comments fixes: hashset, comment for replicaset

* Review fix: replica count instead of fan out percentage
 2022-09-19 18:06:02 +03:00
Egor Ivkov d8a36c5bbe Drop shard before deleting folder in ReplicaSet (#999) 2022-09-12 11:26:01 +03:00
Egor Ivkov d1dda7e7f6 Adds replica info to consensus snapshots (#996) 
* Adds replica info to consensus snapshots

* fix clippy

* Review fix: add debug_assert
 2022-09-08 18:24:40 +03:00
Egor Ivkov 167774aade SetShardReplicaState operation (#986) 
* `SetShardReplicaState` operation

* Review fix: assume RemotePeer manages several remote peers
 2022-09-06 15:37:34 +03:00
Egor Ivkov 517e6c7a0e Replica set (#982) 
* Introduces ReplicaSet
 2022-09-02 20:06:52 +03:00
Egor Ivkov dc0314201e Add replication factor (#966) 
* Add replication_factor config and ops to change it
 2022-08-31 11:26:52 +03:00
Egor Ivkov a4e760af34 AddPeer in one consensus operation (#923) 
* AddPeer in one consensus operation
 2022-08-09 17:00:38 +03:00
Egor Ivkov 540044f280 Raft snapshot logic for handling transfers (#918) 2022-08-05 17:49:18 +02:00
Egor Ivkov 0a634453e3 Peer removal API (#894) 
* Remove peer api

* Keep the same cluster status api for compatibility
 2022-08-02 15:56:41 +03:00
Egor Ivkov 95019c74ca Stop consensus on error (#872) 2022-07-29 18:17:41 +03:00
Egor Ivkov a6922234cc Include shards transfer state in raft snapshot (#868) 
* Include shards transfer state in snapshot

* Review fix
 2022-07-27 19:17:05 +03:00
Egor Ivkov e8ad2731ff Save shard transfers state on disk (#831) 
* Save shard transfers state on disk

* Use write closure instead of save on drop

* Fix clippy

* Review fixes
 2022-07-27 14:58:57 +03:00
Egor IvkovandAndrey Vasnetsov fd70b662d1 Recreate transport channels on network error (#845) 
* Recreate transport channels on network error

* fmt

* Recreate channel upd (#847)

* recoverable channels

* recoverable channels

* retry on specific response error codes

* rename: ChannelsToUri -> ChannelPool

* rename: get_last_created -> get_created_at

* fmt

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-07-25 12:22:16 +02:00
Egor Ivkov d81e3c9f86 Retry entries in case of service errors (#839) 2022-07-22 17:09:04 +03:00
Egor Ivkov 06afc5234e Default value for apply_progress_queue (#821) 2022-07-15 18:17:14 +03:00
Egor Ivkov 026bd040b0 Add import formatting rules (#820) 
* Add import formatting rules

* Review fix: update rusty hook
 2022-07-15 15:42:25 +03:00
Egor Ivkov ef3c6c5e15 Fair hash ring (#816) 
* Fair hash ring

* Python test processes removal fix
 2022-07-14 19:27:53 +02:00
Egor IvkovandArnaud Gourlay bd55b143e2 Consensus operations for shard transfer (#793) 
* Consensus operations for shard transfer

* service error corrected

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Review fix: add transfer abortion reason

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-07-12 17:37:20 +03:00
Egor IvkovandAndrey Vasnetsov 8330ab583e Fix: bring back shard number suggestion (#799) 
* Fix: bring back shard number suggestion

* Review fix: use shard number to assert

* `debug_assert_eq`

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-07-11 18:12:54 +03:00
Egor Ivkov 8c9bfd7757 Remove deprecated API for collections and points update (#787) 2022-07-07 16:49:08 +03:00
Egor Ivkov 18d4e2795b Warnings for big consensus tick period (#785) 
* Print info about approximate leader election time

* Warn about exceeding timeout
 2022-07-06 15:34:36 +03:00
Egor Ivkov 75008d8d57 Kill qdrant processes after each python test (#782) 2022-07-05 15:01:17 +03:00
Egor Ivkov 4a6b185dad Retry bootstrap if term is 0 (#776) 2022-07-05 10:21:02 +02:00
Egor Ivkov 1753811e6a Property tests for Storage trait impl (#760) 
* Property tests for Storage trait impl

* Review fix: add comments to `CollectionContainer`
 2022-07-04 15:38:09 +03:00
Egor Ivkov 8f8f55f993 Learners switch to voters test (#753) 2022-06-28 11:28:11 +02:00
Egor Ivkov e12b91e10c Promote learner (#735) 
* Promote learner

Also fixes ERROR voters not found

* Review fixes

* Review: Find learner first
 2022-06-24 11:27:58 +03:00
Egor Ivkov 17a6b25e29 Add new peers as learners (#710) 
* Add new peers as learners

* Fix Storage impl

* test added

* Test fixes
 2022-06-22 19:50:27 +03:00
Egor Ivkov d52dff2aa1 Fix entry reapplication on restart (#715) 
* Fix entry reapplication on restart

* Review fixes

* Rename `UnappliedEntries` to `EntryApplyProgressQueue`

* Remove `snapshot_was_applied`
 2022-06-21 13:28:44 +03:00
Egor Ivkov e9cb75bf90 Fix unapplied entries len (#703) 2022-06-15 19:29:40 +03:00
Egor Ivkov 533fd208b6 Move out consensus logic from ToC (#675) 2022-06-15 16:18:26 +03:00
Egor Ivkov 9ddaa1c749 Add leader_id and role to cluster API (#660) 2022-06-03 22:37:50 +02:00
Egor IvkovandAndrey Vasnetsov 88d14732ea Wait for leader to be established (#658) 
* Wait for leader to be established

* use condwar instead of channel (#659)

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-06-03 15:58:07 +03:00
Egor Ivkov 1d36910b93 Test shards with restart and add volumes (#654) 2022-06-02 12:37:21 +03:00
Egor Ivkov 4b83464061 Skip consensus init on restart (#638) 
* Skip consensus init on restart

* Added test

* CI
 2022-06-01 11:38:23 +03:00
Egor Ivkov 5bebd07520 Move out consensus test setup into util (#629) 2022-05-25 19:19:16 +03:00
Egor Ivkov 08b8de8692 Fix deadlock on collection creation from snapshot (#615) 2022-05-20 18:05:43 +03:00
Egor Ivkov 8f3f7ca336 Remove serialization wrappers (#612) 2022-05-20 14:03:24 +03:00
Egor Ivkov 0cf8c070f8 Log only h2 errors (#608) 2022-05-19 12:35:21 +03:00
Egor IvkovandAndrey Vasnetsov 55ccf1fff5 Run consensus in docker-compose (#609) 
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-05-18 19:28:17 +03:00
Egor Ivkov 27440937bc Add multipeer deployment test (#590) 
* Add multipeer deployment test

And corresponding fixes for snapshot and wal logic to make test green

* Add CI for multipeer test

* Increase number of peers to 5

* Fix pytest installation

* Fix python version

* Install requests lib
 2022-05-18 15:22:38 +03:00
Egor Ivkov 75f424a5ef Raft configuration updates - add peer (#578) 
* Raft configuration updates - add peer

* Review: clarify send error

* Review: move `first_peer` into `ConsensusEnabled`

* Fix consensus test
 2022-05-11 17:10:18 +03:00
Egor Ivkov f3fc921e51 Derive this peer uri if not supplied (#546) 2022-05-03 16:17:06 +03:00
Egor Ivkov 40c4c1ec2c Allows bootstrapping peer on addition (#529) 
* Allows bootstrapping peer on addition

* Review: warn on address replacement

* Review: more detailed comments
 2022-05-02 18:02:35 +03:00
Egor Ivkov d4bb6e4f49 Send Raft messages with gRPC client (#509) 
* Send Raft messages with gRPC client

* Review: more specific thread names
 2022-04-27 13:01:41 +03:00
Egor Ivkov fd1b8a457c Adds this_peer_id generation as part of raft state (#503) 
Also some raft state refactoring
 2022-04-25 12:14:01 +03:00
Egor Ivkov 99dd3d4704 Adds PeerAddressById to StorageConfig (#501) 2022-04-21 19:02:14 +03:00
Egor Ivkov d0d1f615f2 Fixes test_alias_operation (#497) 2022-04-20 18:04:51 +03:00
Egor Ivkov 77b7e666be Add gRPC RaftService (#490) 
* Add gRPC `RaftService`

* Review: Back-pressure with bounded channel
 2022-04-20 14:23:26 +03:00
Egor Ivkov 9885628f0a Remove java related proto3 options (#485) 
As the project does not use java code generation
 2022-04-14 18:39:52 +03:00
Egor Ivkov e7784cc001 Add comment to UnappliedEntries (#482) 2022-04-14 15:30:09 +03:00
Egor Ivkov ca62c734e4 Save unapplied committed entries indexes (#477) 2022-04-14 11:05:20 +03:00
Egor Ivkov 754dd482c6 Persistent Raft State (#465) 2022-04-12 16:05:04 +03:00
Egor Ivkov ee4cd768ea Allows specifying collection meta operation commit timeout in API (#444) 
* Allows specifying collection meta operation commit timeout in API

Also restructures a part of gRPC collection API

* Review: link to dev docs

* Review: Timeout error description

Also committed generated protobuf code that somehow was left out

* Review: seconds in API and minor proto fixes
 2022-04-08 14:42:38 +03:00
Egor Ivkov aeea31dfa4 Wait for operation commit (#436) 
* Wait for operation commit

* Review: Option with timeout, return execution err to API

* Review: Add comment about error forwarding in ???
 2022-04-06 18:00:44 +03:00
Egor Ivkov 3826b22c0b Handle aliases in Raft state snapshot (#428) 
* Handle aliases in Raft state snapshot

* Persist alias update
 2022-04-01 14:47:45 +03:00
Egor Ivkov d3e0d7dcec Integrate consensus into execution pipeline (#410) 
* Integrate consensus into execution pipeline

* Review: Remove unnecessary config
 2022-03-31 17:54:55 +03:00
Egor Ivkov 2343078ea1 Collection meta as global state (#394) 
* Collection meta as global state

* Improve collection wal entry error handling
 2022-03-22 18:27:03 +03:00
Egor Ivkov dcfb44de6f Consensus with Raft feature (#385) 
Consensus will be used to synchronize global state which is:
- Collection meta
- Shard to Peer mapping
 2022-03-17 15:15:06 +03:00
Egor Ivkov a5bd04f446 Remove unused cli.rs (#383) 2022-03-14 18:47:47 +03:00
Egor Ivkov 960937b58b Sharding related collection API changes (#381) 
* NonZeroU32 shard number

* Inconsistent Failure option

* Generate docs and apis
 2022-03-11 19:26:09 +03:00
Egor Ivkov cb36ca8e90 Fixes shard drop on collection creation (#380) 
* Fixes shard drop on collection creation

* Concurrent waiting for the shard drop

* Handle create_dir_all error

* Decrease n segments per shard
 2022-03-10 19:16:56 +03:00
Egor Ivkov 8a2c17f0d4 Migrate legacy collection format (#367) 2022-03-07 20:20:43 +03:00
Egor Ivkov 2bb52abc71 Multiple shards in collection (#360) 
* Multiple shards in collection

* Collection and Shard async drop

* Generate APIs for shard_number param
 2022-03-07 16:22:47 +03:00
Egor Ivkov 59160eff9f Add consistent hashing library (#348) 2022-02-28 20:37:06 +03:00
Egor Ivkov b008e3dede Select shard for operation (#340) 2022-02-28 11:38:43 +03:00
Egor Ivkov b07aa392a6 Periodic flushing (#332) 
* Periodic flushing

* Save flush error
 2022-02-23 16:52:33 +03:00
Egor Ivkov 3b62670a1a Introduce Shard abstraction (#318) 
* Introduce Shard abstraction

* Optimizer thread name improved and avg_vectors moved
 2022-02-18 17:51:44 +03:00
Egor Ivkov 8ac513dae5 Avoid sending stop signal more than once (#276) 2022-02-02 17:22:39 +03:00
Egor Ivkov 2912042fb6 Use tokio::sync channels instead of async-channel (#273) 2022-02-01 20:37:05 +03:00
Egor Ivkov 009d473733 Logging updates (#150) 
- Log deadlocks as `error` instead of printing them
- Minor stylistic changes to messages for consistency
- Removes global export of log macros with `macro_use`
 2021-12-23 13:00:07 +01:00