* Add integration test to assert all queued updates are also transferred
* Plunge update queue in stream records transfers
* Migrate existing plunger usages to new plunge helper
* Skip test if not compiled with staging flag
* Only send delay operation when staging feature is enabled
* Reformat
* Fix review remarks
* introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert
* add test
* upd dockstring
* require resharding once all peers have updated version
* use service error
* fix clippy again
* wait for same version before resharding in tests
* Add test to reproduce broken WAL delta after aborting stream records
* Add staging env var to slow down stream records transfers for test
* Tweak test formatting and utilities a bit
* Add comment to test, link to PR describing bug
* Update test so it still succeeds with patched behavior
* Fix broken WAL delta after stream records abort (#7791)
* Make set_replica_state async
* Add function called when active state of local replica changes
* Add snapshot for newest clocks
* Bump newest clocks snapshot on replica deactivation
* Use newest clocks snapshot during recovery
* Add enum for specifying whether to take or clear clocks snapshot
* Store clock snapshot inside clock map, removing extra file
This greatly simplifies state handling. It also prevent any kind of
desynchronization because all newest clocks are always persisted
atomically.
* Immediately persist clocks after taking snapshot
* Always update snapshot, only take if missing
* Take clock snapshots through each shard flavor, including proxies
* Propagate dedicated functions for taking and clearing clocks snapshot
* Only persist clocks immediately if changed on snapshot/clear
* Simplify recovery point logic, always take clocks snapshot if exists
* Remove unwrap
* Fix typo
* Fix doc comment
* Transfer driver is async, use Tokio sleep
* Reduce visibility
Do not release the port before the process is really gone
otherwise it can be wrongly reused.
E.g.
2025-11-14T10:39:20.156573Z ERROR qdrant::startup: Panic occurred in file src/tonic/mod.rs at line 358:
called `Result::unwrap()` on an `Err` value: tonic::transport::Error(Transport, hyper::Error(Listen, Os { code: 98, kind: AddrInUse, message: "Address already in use" }))
* Respect payload and hashring filters while replicating point
* Merge points instead of replacing for filtered stream records transfer
* cleanup
* Update utils
* Allow filters in transfer
* drop hashring filter since its not required when destination has single shard
* test comment
* Pass actual filter
* use exact=True
* Forward updates that satisfy the filter
* make test long enough for extra points to be upserted
* minor improvements
* fix test
* try with only inserting new points
* Temporarily forward points that match before or after
* Fix failing test
* Only do inserts
* Dont merge points and update old points in test
* Fix the bug in integration test
* fix read_batch_with_hashring fn name and docs
* changes after rebase
* add validation check
* apply suggestions
* Always trim clock tag when forwarding to a different shard ID
* review fixes
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* remove default explicit config for strict mode
* Don't prefix commented out properties with space to remain consistent
* Don't crash test if strict mode config is not set at all
---------
Co-authored-by: timvisee <tim@visee.me>
* Fix force delete peer flaky test
* Temporarily only run force delete peer test
* Run till failure
* Add mutable transfer field
* Introduce wait_with_return and use it to solve TOCTOU
* Revert "Run till failure"
This reverts commit 53319a19c8.
* Revert "Temporarily only run force delete peer test"
This reverts commit 5ea47e30a6.
* Simplify the test instead of introducing wait_with_return
* revert changes to utils
* Temporarily run fewer tests in CI to reproduce flakiness
* disable other integration test
* log collection topology if request fails
* Run in CI until failure
* update tests
* Catch failures and log
* improve logging for search result comparison flakiness
* fix test
* Simplify test
* Use scroll instead of search and simplify existing code
* Trigger CI
* new util function
* debug again
* print collection topology on mismatch
* wait for deletion of flag after recovery is done
* use wait_for
* Revert integration-tests.yml
* Introduce coverage reports for integration tests
* Install cargo-llvm-cov
* Use multiline script
* Explicityl setup COVERAGE env var
* fix integration tests and log generated data
* fix ls path
* upload lcov file to GH artifacts
* integration profraw dynamic filename
* Fix llvm profile filename template
* Use interrupt instead of kill and merge consensus test results into same file
* Drop upload artifact stage
* install llvm-cov
* upload as artifact and export coverage files
* try simplifying workflow
* Migrate coverage generation to existing dedicated gh workflow
* trigger on coverage related branches
* Build only if qdrant binary with cov doesnt exist
* Use valid yaml
* include mode in profraw filename
* split coverage workflow into parallel jobs
* add poetry version to env
* log poetry version to install
* clean up integration test workflow
* Simplify comments
* Test behaviour of Qdrant with shard initializing flag
* Corrupt shard directory and let Qdrant panic like prod
* Wait for shard transfer
* Restore dirty shards using other replicas
* remove unused code
* Request transfer only if replica is dead or dirty
* fmt
* remove comment
* fix clippy
* Delete shard initializing flag after initializing empty local shard
* Expect test to recover shard in existing test
* Review suggestions
* Run tests for longer
* Simplify tests
* Use 2k points
* condition for point_count
* Add comment
* fix flaky tests
* fix flaky tests
* handle edge case
* Include Active in expected states list
* Introduce is_recovery
* simplify tests
* get rid of is_dirty bool in DummyShard
* add missing negation in condition
* fix condition
* final fix for transfer condition
* Don't auto recover if in recovery mode, simplify state checking
* minor comment improvements
* tests scenario where node is killed after deleting shard initializing flag
* Fix failing CI
* Only automatically recover dead replicas
* Mark replica as dead to recover dummy shard
* fix failing test
* Sleep one second after killing peer, give time to release WAL lock
* Prevent waiting for peer to come online indefinitely
* update comment
* minor typo
---------
Co-authored-by: timvisee <tim@visee.me>
* Measure update operations hardware IO
* Add support for distributed setups
* also measure update_local
* Add consensus tests for HW metrics of update operations
* add test for upserting without waiting
* Disable HW usage reporting when not waiting for update API
* Review remarks
* Fix resharding collecting hw measurements
* Fix metric type
* New struct HardwareData for better accumulation
* Ensure we always apply CPU multiplier
* Apply suggestions from code review
* Update src/actix/api/update_api.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Fix assert_with_upper_bound_error threshold calculation.
* Clarifying why we don't measure shard cleanup
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add retry-after header for gRPC
* propagate retry-after within inter node communication
* snipe
* tweak and test
* log all parsing errors
* Update lib/collection/src/operations/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Restructure process spawning in consensus tests, build list of args
* Extend consensus test, also test if providing URIs through env vars work
* Set URI and bootstrap variables unconditionally
* Strict Mode: distributed checking of max collection size
* add size projections in distributed mode
* Add consensus tests
* New Test: All nodes in cluster
* fix tests
* Update lib/collection/src/collection/mod.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* increase upsert delay
* add TODO for resharding
* wait for strict mode config to be applied on second node
* remove delays
* Also wait for strict mode in other test
* clearify strict mode config option
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* option to clear consensus state while preserving peer id
* compact or clear consensus WAL on re-init
* Add test on reinit
* minor changes
* more points check suggestion
---------
Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
* Resume resharding driver on restart if resharding is active
* Update resharding state comment started text
Using since versus started. 'Since' better clarifies that the current
state has been active since that time. While 'started at' could lead to
confusion on whether that time is for starting the whole resharding
operation or just that state.
* Add resharding resumption test during migrating points
* Recover resharding hash ring after loading shards
* Update resharding resume test, interrupt at multiple stages, less points
* Remove unused parameter
* Update collection shard count at resharding shard creation/destruction
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Use debug_assert_eq
* Remove trailing comma
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Apply resharding filter only to existing shards, include shard selection
* Add test for stable exact point count during resharding
* Improve filtering, only two separate requests, filtered and non-filtered
* Make scrolling stable while resharding
* Add test for stable scroll during resharding
* Make search stable while resharding
* Add test for stable search during resharding
* Remove resharding post filter in retrieve
* Also assert cardinality point count in resharding test
* Fix typos and some tweaks
* Only clone filter if resharding is active
* Also minimize cloning with resharding filter on count request
* Add test for stable exact point count during resharding with indexing
* Update lib/collection/src/shards/shard_holder/mod.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Update lib/collection/src/collection/point_ops.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Also apply resharding filter to retrieve
* Restructure resharding filter usage in `Collection::retrieve`
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add basic resharding test, asserting shard and point counts
* Add resharding replica balancing test
* Always propagate deletes with wait, fix first resharding test flakyness
* Don't count points in balance test for now
* Fix resharding balance test, initial shard may not be on first node
* Count points in resharding balance test
* Test resharding with concurrent updates
* Add point data consistency check to resharding tests
* Fix not using given collection name, list point counts in debug output
* Extract logic for creating thresholds config
* Put collection params and threshold config in update handler
* Add function to add a new appendable segment if all are over capacity
* Make new method static, call it before each optimization loop
* Update error message formatting
* Use exact point count in replication consensus test
* Add a test to assert segment creation when all are over capacity
* Suffix optimizer thresholds with _kb to clarify unit
* Move segment capacity check logic, run if optimizers are disabled
* fix: add -> mul
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add API key to HTTP client
* Add API key field to snapshot recovery requests
* Add API key to channel service
* Provide API key when doing snapshot transfer
* Configure API key header name constant in a central place
* Reformat
* Update OpenAPI spec
* Remove suffixed spaces from configuration file
* Allow to specify HTTP headers in some consensus test utility functions
* Add snapshot transfer test with configured API key
* Use random API key in test
* Fix compilation errors and clippy warnings
* separate access tests from validation tests
* bring changes from rbac-integration-tests review fixes
* bring changes from rbac-payload-access-tests
* update from base branch
* use const instead of magic number
* move COLL_NAME
* force usage of kill_all_processes()
* create exhaustive access test (missing body stubs)
* use json body in rest requests
* use better stubs
* green test for create_shard_key
* make it work for delete_shard_key
* make it work for list_collections
* make it work for get_collection
* make it work for create_collection
* make it work for update_collection_params
* make it work for delete_collection
* make it work for update_aliases
* make it work for create_index
* make it work for collection_exists
* make it work for delete_index
* make it work for get_collection_cluster_info
* don't build grpcio and use `uv` for faster deps installation
* partially prepare for update cluster ops
* make it work for list_collection_aliases
* make it work for list_aliases
* make it work for list_collection_snapshots
* make it work for create_collection_snapshot
* huge refactor + make it work for delete_collection_snapshot
* make it work for download_collection_snapshot
* test_upload_collection_snapshot
* test_recover_collection_snapshot
* test_recover_collection_snapshot
* test_upload_shard_snapshot
* test_recover_shard_snapshot
* test_list_shard_snapshots
* test_create_shard_snapshot
* test_delete_shard_snapshot
* test_download_shard_snapshot
* test_list_snapshots and test_create_snapshot
* test_delete_snapshot
* test_download_full_snapshot
* test_get_cluster
* test_recover_raft_state
* test_delete_peer
* prepare for splitting into another PR
* skip api exhaustiveness checks
* lil cleanup
* fix integration tests workflow
* test search api
* test recommend api
* test discover api
* test count and scroll
* test get and upsert points
* test update points batch
* test update and delete vectors
* test set, overwrite, delete, and clear payload
* test service level actions
* finish testing (with errors), introduce not needing success
* make tests much faster
* simplify delete tests
* complete collection cluster operations
* cleanup
* update after rebase
* change permissions for create and delete shard keys
* review fixes
* add newline
* Add shard transfer consensus method to restart shard transfer
* Arrange shard transfer fallback through consensus
* Properly handle stopping existing transfer, don't finish
* Use user specified shard transfer method as default when falling back
* Report correct fallback transfer method being used
* Always make WAL delta transfer fall back to stream records
* Remove unnecessary clones
* When doing shard transfer fallback, confirm consensus accepted it
* Confirm shard transfer restarts through consensus, rather than state
* Update WAL delta fallback test, assert we switch to stream_records
* Decrease queue proxy batch size to 10
This means each batch will hold 10 user operations. Each user operation
could already be a batch.
* Add test barrier to wait for transfer progress
* Don't assert WAL delta transfer by logs, assert from status if possible
* Fix typo
* Add WAL delta transfer integration test for empty delta
* Report size of WAL delta, assert this in integration tests
* Add WAL delta transfer integration test for shard recovery
* Add WAL delta transfer integration test for falling back
* Minor improvements
* Fix typo
* Fix data race in WAL delta integration test, kill upsertions on count
* In WAL delta tests, compare all points, vectors and payload exactly
* add test for fallback in into straming if diff is big
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Do not select ourselves when finding auto shard transfer source
* Limit automatic shard transfers with basic transfer count check
* Extract shard recovery transfer request logic to separate function
* Add global shard transfer tracker structure to allow rate limiting
* Count and limit incoming and outgoing shard transfers separately
* Make automatic shard transfer limit configurable
* Move shard transfer tracker from global to collection level
* Comment out new config parameters
* Fix incorrect comment
* Fix missing space in log message
* Fix negated condition
* Remove logic for requesting shard transfer on replica state change
* Check shard transfer limits in consensus sync, use consensus state
Instead of bothering with tracking proposed shard transfers, this now
purely relies on state that is already in consensus. Each time we sync
consensus, we request the maximum number of shard transfers up to the
limit at that time.
* Remove now obsolete shard transfer tracker
* Revert now obsolete changes
* Improve transfer IO counting comment, revert now obsolete code
* Fix typos
* Fix flipped variables, don't take self reference on Copyable type
* Update lib/collection/src/collection/shard_transfer.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Handle incoming/outgoing transfer counts separately, don't tuple it
* Improve loop for counting incoming/outgoing transfers
* Remove unused test function
* Add consensus tests for automatic shard transfer limits
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Remove debug lines from new test
* Reorder last test a bit to resolve flakyness
* We can have one more transfer for recovery on the other alive node
* Attempt to reduce test flakyness, more points and more frequent polling
* Update config/config.yaml
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Explicitly note default shard transfer limit in configuration
* Use default for shard transfer IO everywhere
* Rename transfer limit check function to be more explicit
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
