* Resume resharding driver on restart if resharding is active
* Update resharding state comment started text
Using since versus started. 'Since' better clarifies that the current
state has been active since that time. While 'started at' could lead to
confusion on whether that time is for starting the whole resharding
operation or just that state.
* Add resharding resumption test during migrating points
* Recover resharding hash ring after loading shards
* Update resharding resume test, interrupt at multiple stages, less points
* Remove unused parameter
* Update collection shard count at resharding shard creation/destruction
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Use debug_assert_eq
* Remove trailing comma
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Apply resharding filter only to existing shards, include shard selection
* Add test for stable exact point count during resharding
* Improve filtering, only two separate requests, filtered and non-filtered
* Make scrolling stable while resharding
* Add test for stable scroll during resharding
* Make search stable while resharding
* Add test for stable search during resharding
* Remove resharding post filter in retrieve
* Also assert cardinality point count in resharding test
* Fix typos and some tweaks
* Only clone filter if resharding is active
* Also minimize cloning with resharding filter on count request
* Add test for stable exact point count during resharding with indexing
* Update lib/collection/src/shards/shard_holder/mod.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Update lib/collection/src/collection/point_ops.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Also apply resharding filter to retrieve
* Restructure resharding filter usage in `Collection::retrieve`
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add basic resharding test, asserting shard and point counts
* Add resharding replica balancing test
* Always propagate deletes with wait, fix first resharding test flakyness
* Don't count points in balance test for now
* Fix resharding balance test, initial shard may not be on first node
* Count points in resharding balance test
* Test resharding with concurrent updates
* Add point data consistency check to resharding tests
* Fix not using given collection name, list point counts in debug output
* Extract logic for creating thresholds config
* Put collection params and threshold config in update handler
* Add function to add a new appendable segment if all are over capacity
* Make new method static, call it before each optimization loop
* Update error message formatting
* Use exact point count in replication consensus test
* Add a test to assert segment creation when all are over capacity
* Suffix optimizer thresholds with _kb to clarify unit
* Move segment capacity check logic, run if optimizers are disabled
* fix: add -> mul
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add API key to HTTP client
* Add API key field to snapshot recovery requests
* Add API key to channel service
* Provide API key when doing snapshot transfer
* Configure API key header name constant in a central place
* Reformat
* Update OpenAPI spec
* Remove suffixed spaces from configuration file
* Allow to specify HTTP headers in some consensus test utility functions
* Add snapshot transfer test with configured API key
* Use random API key in test
* Fix compilation errors and clippy warnings
* separate access tests from validation tests
* bring changes from rbac-integration-tests review fixes
* bring changes from rbac-payload-access-tests
* update from base branch
* use const instead of magic number
* move COLL_NAME
* force usage of kill_all_processes()
* create exhaustive access test (missing body stubs)
* use json body in rest requests
* use better stubs
* green test for create_shard_key
* make it work for delete_shard_key
* make it work for list_collections
* make it work for get_collection
* make it work for create_collection
* make it work for update_collection_params
* make it work for delete_collection
* make it work for update_aliases
* make it work for create_index
* make it work for collection_exists
* make it work for delete_index
* make it work for get_collection_cluster_info
* don't build grpcio and use `uv` for faster deps installation
* partially prepare for update cluster ops
* make it work for list_collection_aliases
* make it work for list_aliases
* make it work for list_collection_snapshots
* make it work for create_collection_snapshot
* huge refactor + make it work for delete_collection_snapshot
* make it work for download_collection_snapshot
* test_upload_collection_snapshot
* test_recover_collection_snapshot
* test_recover_collection_snapshot
* test_upload_shard_snapshot
* test_recover_shard_snapshot
* test_list_shard_snapshots
* test_create_shard_snapshot
* test_delete_shard_snapshot
* test_download_shard_snapshot
* test_list_snapshots and test_create_snapshot
* test_delete_snapshot
* test_download_full_snapshot
* test_get_cluster
* test_recover_raft_state
* test_delete_peer
* prepare for splitting into another PR
* skip api exhaustiveness checks
* lil cleanup
* fix integration tests workflow
* test search api
* test recommend api
* test discover api
* test count and scroll
* test get and upsert points
* test update points batch
* test update and delete vectors
* test set, overwrite, delete, and clear payload
* test service level actions
* finish testing (with errors), introduce not needing success
* make tests much faster
* simplify delete tests
* complete collection cluster operations
* cleanup
* update after rebase
* change permissions for create and delete shard keys
* review fixes
* add newline
* Add shard transfer consensus method to restart shard transfer
* Arrange shard transfer fallback through consensus
* Properly handle stopping existing transfer, don't finish
* Use user specified shard transfer method as default when falling back
* Report correct fallback transfer method being used
* Always make WAL delta transfer fall back to stream records
* Remove unnecessary clones
* When doing shard transfer fallback, confirm consensus accepted it
* Confirm shard transfer restarts through consensus, rather than state
* Update WAL delta fallback test, assert we switch to stream_records
* Decrease queue proxy batch size to 10
This means each batch will hold 10 user operations. Each user operation
could already be a batch.
* Add test barrier to wait for transfer progress
* Don't assert WAL delta transfer by logs, assert from status if possible
* Fix typo
* Add WAL delta transfer integration test for empty delta
* Report size of WAL delta, assert this in integration tests
* Add WAL delta transfer integration test for shard recovery
* Add WAL delta transfer integration test for falling back
* Minor improvements
* Fix typo
* Fix data race in WAL delta integration test, kill upsertions on count
* In WAL delta tests, compare all points, vectors and payload exactly
* add test for fallback in into straming if diff is big
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Do not select ourselves when finding auto shard transfer source
* Limit automatic shard transfers with basic transfer count check
* Extract shard recovery transfer request logic to separate function
* Add global shard transfer tracker structure to allow rate limiting
* Count and limit incoming and outgoing shard transfers separately
* Make automatic shard transfer limit configurable
* Move shard transfer tracker from global to collection level
* Comment out new config parameters
* Fix incorrect comment
* Fix missing space in log message
* Fix negated condition
* Remove logic for requesting shard transfer on replica state change
* Check shard transfer limits in consensus sync, use consensus state
Instead of bothering with tracking proposed shard transfers, this now
purely relies on state that is already in consensus. Each time we sync
consensus, we request the maximum number of shard transfers up to the
limit at that time.
* Remove now obsolete shard transfer tracker
* Revert now obsolete changes
* Improve transfer IO counting comment, revert now obsolete code
* Fix typos
* Fix flipped variables, don't take self reference on Copyable type
* Update lib/collection/src/collection/shard_transfer.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Handle incoming/outgoing transfer counts separately, don't tuple it
* Improve loop for counting incoming/outgoing transfers
* Remove unused test function
* Add consensus tests for automatic shard transfer limits
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Remove debug lines from new test
* Reorder last test a bit to resolve flakyness
* We can have one more transfer for recovery on the other alive node
* Attempt to reduce test flakyness, more points and more frequent polling
* Update config/config.yaml
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Explicitly note default shard transfer limit in configuration
* Use default for shard transfer IO everywhere
* Rename transfer limit check function to be more explicit
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* add update_concurrency to control the parallelism of updating shards
* reformat the files
* reformat
* add update_concurrency control to shard updates
* use unwrap_or
* test using buffer_ordered
* use buffered (testing)
* Pre-allocate space for update futures
* use NonZeroUsize
* linter
* use NonZeroUsize in the test
* consensus test for update_concurrency
* update config commit
---------
Co-authored-by: Di Zhao <diz@twitter.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
* Forward write request according to write consistency
* use gRPC internal update API by making shard_id optional
* improve and test
* mark leader peer as failed in case of service error during forward
* forward write ordering param
* WIP: introduce local state
* WIP: sync local state with consensus in idle
* fmt
* rm extensive debug
* update triple replication test
* update triple replication test
* rm debug logs
* fix check for established leader + only sync local if no proposals
* remove unused file
* test compatible with python 3.8
* test compatible with python 3.8
* longer wait for consensus
* longer wait for consensus
* extra sleep in test
* remove extra sleep
* Fixing missed leader inconsistency - transfers (#1298)
* explicit request timeout
* explicit request timeout
* explicit request timeout
* explicit request timeout
* explicit request timeout
* prevent double handelling of the transfer termination
* kill the process
* log on inconsistency
* log on inconsistency
* log on inconsistency
* debug
* revert debug in test
* forward updates to partial shards, abort transactions on dead node report
* fmt
* disable retry of transfer, if the transfer was cancelled + allow predictable ports in test
* fix import
