Under stress, three concurrent user-requested snapshot transfers (10 000
points each) running while the killed peer recovers can starve the
leader's heartbeats long enough to trigger a raft election. If the
recovery transfer's `RecoveryToPartial` proposal is submitted while no
leader exists, raft drops it silently — `recovered_switch_to_partial`
returns Ok regardless because it only sends to a channel — and the
retry path then waits a full CONSENSUS_CONFIRM_TIMEOUT (10s) before
trying again. Combined with sequential per-shard recovery (auto
transfer limit = 1 × 3 shards), the 30s `/readyz` budget runs out.
Add an optional `wait_for_timeout` to `wait_for_peer_online` and bump
this test's wait to 60s. Default behaviour for other callers is
unchanged.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The test killed the last peer immediately after start_cluster, but
start_cluster only waits for cluster size and a known leader — not for
all peers to be promoted from learner to voter. If the last peer caught
up first, it became the only other voter alongside the leader; killing
it left a 2-of-2 quorum with one voter dead, and the subsequent
CreateCollection commit timed out after 10s.
Wait for all peers to be voters before killing one, so the survivors
form a 2-of-3 voter quorum.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
During a snapshot shard transfer recovery, the receiving peer
temporarily takes its local shard before installing the snapshot, so
the cluster info endpoint returns local_shards: [] for a brief window.
The test helper indexed [0] on that list and crashed with IndexError,
making test_triple_replication flaky. Treat the empty case as a
non-Active state so callers poll again instead.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [ai] On shard snapshot transfer recovery, drop existing shard before recovery
* [ai] Add integration test to assert clearing behavior
* [ai] Debug assert our replica is not active when we clear it
* [ai] Tweak assertion
* Fix flaky test, replica may temporarily not be visible
* Replace debug assertion with runtime error
* Fix consensus test port collision across xdist workers
When a peer was started with `get_port()` for each of p2p/grpc/http,
only the OS-allocated p2p_port was guaranteed free. On restart with
`port=p.p2p_port`, the framework derives grpc=port+1, http=port+2 — but
those neighbor ports were never reserved at original startup. Under
`pytest -n auto --dist=loadfile`, another xdist worker could legitimately
bind one of them, causing the restarted peer's REST bind to fail with
EADDRINUSE while the test still talks to that URL and gets a misleading
404 from the unrelated peer.
Add `get_port_triple()` which allocates a base port and probes bind() on
base+1 and base+2 to verify the contiguous slot is free across processes,
then route start_peer / start_first_peer (and their two callers in
test_peer_snapshot_bootstrap / test_cluster_rejoin) through it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Partition port allocation by xdist worker
Each pytest-xdist worker now draws peer ports from a disjoint slice
(BASE + worker_index * SIZE), so concurrent workers can never compete
for the same triple. Within the slice we still probe-bind() each
candidate to skip ports occupied by unrelated processes, and fall back
to OS allocation if the slice is exhausted.
This eliminates the cross-worker race that prompted the original fix,
and removes the need for the ±2 buffer dance against busy_ports.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: parallelize consensus tests with pytest-xdist
Enable pytest-xdist for consensus_tests to run tests across multiple
workers in parallel, significantly reducing CI wall time (~20min → ~5-7min).
Changes:
- Add `-n auto --dist=loadfile` to the consensus test pytest invocation
- Remove hardcoded port_seed from tests that don't need fixed ports for
restart/rejoin (test_order_by, test_consensus_compaction,
test_named_vector_crud, test_listener_node)
- Give test_cluster_rejoin its own PORT_SEED=15000 to avoid port
conflicts with auth tests (PORT_SEED=10000)
- Derive restart ports from killed PeerProcess objects instead of
hardcoded arithmetic where possible
- Add xdist_group("auth") marker to auth test files to ensure they
run on the same worker (they share PORT_SEED=10000)
Made-with: Cursor
* fix: remove remaining hardcoded port_seed=20000 causing parallel test conflicts
8 test files were using port_seed=20000 as a positional argument to
start_cluster(), which was missed in the initial change. When running
in parallel with pytest-xdist, multiple workers would try to bind to
the same port range (20000-20x02), causing port conflicts and cascading
test failures.
Also remove port_seed=23000 from test_snapshot_recovery_kill.py since
it doesn't need fixed ports for restart.
Made-with: Cursor
* fix: use saved port for restart in test_two_follower_nodes_down
The test was restarting killed peers on hardcoded ports (20200/20100)
that previously matched port_seed=20000. After switching to random
ports, the restart ports no longer match the original peer ports,
causing raft state URI mismatches and peer startup failures.
Save the p2p_port from the killed PeerProcess and reuse it for restart.
Made-with: Cursor
* Reuse p2p ports when restarting killed peers in consensus tests
When a peer is killed and restarted with random ports, it gets a new
consensus URI. The cluster needs a Raft operation to update this URI,
which under CPU contention from parallel test workers can exceed the
30-second timeout. Fix by capturing each peer's p2p_port before killing
and reusing it on restart, so the URI stays the same and no consensus
update is needed.
Made-with: Cursor
* A few improvements for parallel runs (#8731)
* fix: make auth tests' PORT_SEED per-worker to avoid port collisions
* ci: improve failure visibility for parallel consensus tests
* Three small changes to make hangs, interleaved output, and coverage runs behave predictably under pytest-xdist
* fix: two test bugs surfaced by parallel runs and revert drop PR_SET_PDEATHSIG helper
* fix: wait for count convergence in test_triple_replication
* fix: clean leaked peer processes at test start
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add integration test to assert all queued updates are also transferred
* Plunge update queue in stream records transfers
* Migrate existing plunger usages to new plunge helper
* Skip test if not compiled with staging flag
* Only send delay operation when staging feature is enabled
* Reformat
* Fix review remarks
* introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert
* add test
* upd dockstring
* require resharding once all peers have updated version
* use service error
* fix clippy again
* wait for same version before resharding in tests
* Add test to reproduce broken WAL delta after aborting stream records
* Add staging env var to slow down stream records transfers for test
* Tweak test formatting and utilities a bit
* Add comment to test, link to PR describing bug
* Update test so it still succeeds with patched behavior
* Fix broken WAL delta after stream records abort (#7791)
* Make set_replica_state async
* Add function called when active state of local replica changes
* Add snapshot for newest clocks
* Bump newest clocks snapshot on replica deactivation
* Use newest clocks snapshot during recovery
* Add enum for specifying whether to take or clear clocks snapshot
* Store clock snapshot inside clock map, removing extra file
This greatly simplifies state handling. It also prevent any kind of
desynchronization because all newest clocks are always persisted
atomically.
* Immediately persist clocks after taking snapshot
* Always update snapshot, only take if missing
* Take clock snapshots through each shard flavor, including proxies
* Propagate dedicated functions for taking and clearing clocks snapshot
* Only persist clocks immediately if changed on snapshot/clear
* Simplify recovery point logic, always take clocks snapshot if exists
* Remove unwrap
* Fix typo
* Fix doc comment
* Transfer driver is async, use Tokio sleep
* Reduce visibility
Do not release the port before the process is really gone
otherwise it can be wrongly reused.
E.g.
2025-11-14T10:39:20.156573Z ERROR qdrant::startup: Panic occurred in file src/tonic/mod.rs at line 358:
called `Result::unwrap()` on an `Err` value: tonic::transport::Error(Transport, hyper::Error(Listen, Os { code: 98, kind: AddrInUse, message: "Address already in use" }))
* Respect payload and hashring filters while replicating point
* Merge points instead of replacing for filtered stream records transfer
* cleanup
* Update utils
* Allow filters in transfer
* drop hashring filter since its not required when destination has single shard
* test comment
* Pass actual filter
* use exact=True
* Forward updates that satisfy the filter
* make test long enough for extra points to be upserted
* minor improvements
* fix test
* try with only inserting new points
* Temporarily forward points that match before or after
* Fix failing test
* Only do inserts
* Dont merge points and update old points in test
* Fix the bug in integration test
* fix read_batch_with_hashring fn name and docs
* changes after rebase
* add validation check
* apply suggestions
* Always trim clock tag when forwarding to a different shard ID
* review fixes
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* remove default explicit config for strict mode
* Don't prefix commented out properties with space to remain consistent
* Don't crash test if strict mode config is not set at all
---------
Co-authored-by: timvisee <tim@visee.me>
* Fix force delete peer flaky test
* Temporarily only run force delete peer test
* Run till failure
* Add mutable transfer field
* Introduce wait_with_return and use it to solve TOCTOU
* Revert "Run till failure"
This reverts commit 53319a19c8.
* Revert "Temporarily only run force delete peer test"
This reverts commit 5ea47e30a6.
* Simplify the test instead of introducing wait_with_return
* revert changes to utils
* Temporarily run fewer tests in CI to reproduce flakiness
* disable other integration test
* log collection topology if request fails
* Run in CI until failure
* update tests
* Catch failures and log
* improve logging for search result comparison flakiness
* fix test
* Simplify test
* Use scroll instead of search and simplify existing code
* Trigger CI
* new util function
* debug again
* print collection topology on mismatch
* wait for deletion of flag after recovery is done
* use wait_for
* Revert integration-tests.yml
* Introduce coverage reports for integration tests
* Install cargo-llvm-cov
* Use multiline script
* Explicityl setup COVERAGE env var
* fix integration tests and log generated data
* fix ls path
* upload lcov file to GH artifacts
* integration profraw dynamic filename
* Fix llvm profile filename template
* Use interrupt instead of kill and merge consensus test results into same file
* Drop upload artifact stage
* install llvm-cov
* upload as artifact and export coverage files
* try simplifying workflow
* Migrate coverage generation to existing dedicated gh workflow
* trigger on coverage related branches
* Build only if qdrant binary with cov doesnt exist
* Use valid yaml
* include mode in profraw filename
* split coverage workflow into parallel jobs
* add poetry version to env
* log poetry version to install
* clean up integration test workflow
* Simplify comments
* Test behaviour of Qdrant with shard initializing flag
* Corrupt shard directory and let Qdrant panic like prod
* Wait for shard transfer
* Restore dirty shards using other replicas
* remove unused code
* Request transfer only if replica is dead or dirty
* fmt
* remove comment
* fix clippy
* Delete shard initializing flag after initializing empty local shard
* Expect test to recover shard in existing test
* Review suggestions
* Run tests for longer
* Simplify tests
* Use 2k points
* condition for point_count
* Add comment
* fix flaky tests
* fix flaky tests
* handle edge case
* Include Active in expected states list
* Introduce is_recovery
* simplify tests
* get rid of is_dirty bool in DummyShard
* add missing negation in condition
* fix condition
* final fix for transfer condition
* Don't auto recover if in recovery mode, simplify state checking
* minor comment improvements
* tests scenario where node is killed after deleting shard initializing flag
* Fix failing CI
* Only automatically recover dead replicas
* Mark replica as dead to recover dummy shard
* fix failing test
* Sleep one second after killing peer, give time to release WAL lock
* Prevent waiting for peer to come online indefinitely
* update comment
* minor typo
---------
Co-authored-by: timvisee <tim@visee.me>
* Measure update operations hardware IO
* Add support for distributed setups
* also measure update_local
* Add consensus tests for HW metrics of update operations
* add test for upserting without waiting
* Disable HW usage reporting when not waiting for update API
* Review remarks
* Fix resharding collecting hw measurements
* Fix metric type
* New struct HardwareData for better accumulation
* Ensure we always apply CPU multiplier
* Apply suggestions from code review
* Update src/actix/api/update_api.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Fix assert_with_upper_bound_error threshold calculation.
* Clarifying why we don't measure shard cleanup
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add retry-after header for gRPC
* propagate retry-after within inter node communication
* snipe
* tweak and test
* log all parsing errors
* Update lib/collection/src/operations/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Restructure process spawning in consensus tests, build list of args
* Extend consensus test, also test if providing URIs through env vars work
* Set URI and bootstrap variables unconditionally
* Strict Mode: distributed checking of max collection size
* add size projections in distributed mode
* Add consensus tests
* New Test: All nodes in cluster
* fix tests
* Update lib/collection/src/collection/mod.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* increase upsert delay
* add TODO for resharding
* wait for strict mode config to be applied on second node
* remove delays
* Also wait for strict mode in other test
* clearify strict mode config option
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* option to clear consensus state while preserving peer id
* compact or clear consensus WAL on re-init
* Add test on reinit
* minor changes
* more points check suggestion
---------
Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
* Resume resharding driver on restart if resharding is active
* Update resharding state comment started text
Using since versus started. 'Since' better clarifies that the current
state has been active since that time. While 'started at' could lead to
confusion on whether that time is for starting the whole resharding
operation or just that state.
* Add resharding resumption test during migrating points
* Recover resharding hash ring after loading shards
* Update resharding resume test, interrupt at multiple stages, less points
* Remove unused parameter
* Update collection shard count at resharding shard creation/destruction
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Use debug_assert_eq
* Remove trailing comma
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Apply resharding filter only to existing shards, include shard selection
* Add test for stable exact point count during resharding
* Improve filtering, only two separate requests, filtered and non-filtered
* Make scrolling stable while resharding
* Add test for stable scroll during resharding
* Make search stable while resharding
* Add test for stable search during resharding
* Remove resharding post filter in retrieve
* Also assert cardinality point count in resharding test
* Fix typos and some tweaks
* Only clone filter if resharding is active
* Also minimize cloning with resharding filter on count request
* Add test for stable exact point count during resharding with indexing
* Update lib/collection/src/shards/shard_holder/mod.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Update lib/collection/src/collection/point_ops.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Also apply resharding filter to retrieve
* Restructure resharding filter usage in `Collection::retrieve`
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add basic resharding test, asserting shard and point counts
* Add resharding replica balancing test
* Always propagate deletes with wait, fix first resharding test flakyness
* Don't count points in balance test for now
* Fix resharding balance test, initial shard may not be on first node
* Count points in resharding balance test
* Test resharding with concurrent updates
* Add point data consistency check to resharding tests
* Fix not using given collection name, list point counts in debug output
* Extract logic for creating thresholds config
* Put collection params and threshold config in update handler
* Add function to add a new appendable segment if all are over capacity
* Make new method static, call it before each optimization loop
* Update error message formatting
* Use exact point count in replication consensus test
* Add a test to assert segment creation when all are over capacity
* Suffix optimizer thresholds with _kb to clarify unit
* Move segment capacity check logic, run if optimizers are disabled
* fix: add -> mul
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add API key to HTTP client
* Add API key field to snapshot recovery requests
* Add API key to channel service
* Provide API key when doing snapshot transfer
* Configure API key header name constant in a central place
* Reformat
* Update OpenAPI spec
* Remove suffixed spaces from configuration file
* Allow to specify HTTP headers in some consensus test utility functions
* Add snapshot transfer test with configured API key
* Use random API key in test
* Fix compilation errors and clippy warnings
* separate access tests from validation tests
* bring changes from rbac-integration-tests review fixes
* bring changes from rbac-payload-access-tests
* update from base branch
* use const instead of magic number
* move COLL_NAME
* force usage of kill_all_processes()
* create exhaustive access test (missing body stubs)
* use json body in rest requests
* use better stubs
* green test for create_shard_key
* make it work for delete_shard_key
* make it work for list_collections
* make it work for get_collection
* make it work for create_collection
* make it work for update_collection_params
* make it work for delete_collection
* make it work for update_aliases
* make it work for create_index
* make it work for collection_exists
* make it work for delete_index
* make it work for get_collection_cluster_info
* don't build grpcio and use `uv` for faster deps installation
* partially prepare for update cluster ops
* make it work for list_collection_aliases
* make it work for list_aliases
* make it work for list_collection_snapshots
* make it work for create_collection_snapshot
* huge refactor + make it work for delete_collection_snapshot
* make it work for download_collection_snapshot
* test_upload_collection_snapshot
* test_recover_collection_snapshot
* test_recover_collection_snapshot
* test_upload_shard_snapshot
* test_recover_shard_snapshot
* test_list_shard_snapshots
* test_create_shard_snapshot
* test_delete_shard_snapshot
* test_download_shard_snapshot
* test_list_snapshots and test_create_snapshot
* test_delete_snapshot
* test_download_full_snapshot
* test_get_cluster
* test_recover_raft_state
* test_delete_peer
* prepare for splitting into another PR
* skip api exhaustiveness checks
* lil cleanup
* fix integration tests workflow
* test search api
* test recommend api
* test discover api
* test count and scroll
* test get and upsert points
* test update points batch
* test update and delete vectors
* test set, overwrite, delete, and clear payload
* test service level actions
* finish testing (with errors), introduce not needing success
* make tests much faster
* simplify delete tests
* complete collection cluster operations
* cleanup
* update after rebase
* change permissions for create and delete shard keys
* review fixes
* add newline
* Add shard transfer consensus method to restart shard transfer
* Arrange shard transfer fallback through consensus
* Properly handle stopping existing transfer, don't finish
* Use user specified shard transfer method as default when falling back
* Report correct fallback transfer method being used
* Always make WAL delta transfer fall back to stream records
* Remove unnecessary clones
* When doing shard transfer fallback, confirm consensus accepted it
* Confirm shard transfer restarts through consensus, rather than state
* Update WAL delta fallback test, assert we switch to stream_records
