* test: add a gated proxy for peer RPCs
Pause one selected internal request while other peer traffic continues.
Preserve payloads, metadata, deadlines, and cancellation so consensus
tests can control transfer timing without blocking unrelated requests.
Cover forwarding, independent gates, and cleanup with socket tests.
* test: connect peer proxies to consensus clusters
Let consensus tests route internal RPCs through request gates. Keep each
proxy alive across peer restarts so advertised addresses remain stable,
and close all proxies during test cleanup.
Wait for the upstream gRPC connection before returning from proxied
startup. Verify consensus progress during a held WAL-delta request,
recovery data, and restart behavior with both URI configuration modes.
* test: fix potentially misleading peer proxy method names
Explicitly state the guarantees, or lack of.
Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com>
* test: add support for hold_snapshot_download
Removes flakiness from snapshot-related consensus tests too
Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com>
* test: improve asserts when force deleting peer
Actually verify survivors recover and retain the expected data.
Making sure no data loss happens.
Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com>
* test: add OsError socket handling + explicit wal_delta tests
Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com>
* test: reject zero as a defined consensus leader
* test: recheck leader agreement on each poll
After a restart, the leader can change during election. Let the cluster
wait resample the leader on each poll and require agreement on a nonzero
leader before the snapshot test starts its transfer.
Keep explicit leader checks for existing callers, membership-size checks,
and the existing timeout. Cover election changes and offline peers.
* test: verify independent snapshot download gates
* test: use a positive peer connection deadline
* test: cover recovery after the removed source exits
* chore: add clarifying comment on timeout=0 usage
It's not obvious at first why it's like so.
Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com>
* test: share consensus response gates
Move response gates, their tests, and Raft decoding from the leader
removal proof into the base test infrastructure. Both removal scenarios
can then use the same successful-response check.
* test: support selective RPC blocking
Keep a removed source unaware of membership changes while its transfer
continues. Block its Raft traffic in both directions so election
attempts cannot disrupt survivor recovery.
* test: make source removal scenarios deterministic
Separate recovery after source exit from late data sent by a removed
source. Require a successful receiver response in the late scenario,
and retain complete data and replica-state checks in both cases.
* test: refactor timeouts and deadlines
* Cancellation happens after observing the intended phase, without an RPC deadline.
* Separate deadline tests cover held requests, upstream work, and held responses.
Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com>
* test: bound peer probes and removal requests
Give cluster probes and peer removal finite client timeouts so a stalled
HTTP request cannot leave the test waiting indefinitely.
* test: separate RPC release from termination
Keep the upstream handler blocked until the test releases it or the RPC
terminates. Use a separate termination event for cancellation assertions,
and release the handler during teardown instead of racing a fixture timer.
* test: use monotonic polling deadlines
Measure elapsed polling time with a monotonic clock so system clock
adjustments cannot shorten or extend the wait.
* test: allow more time to observe proxy events
Allow ten seconds for proxy observations and ordinary test requests.
Event and future waits still return as soon as they complete. Keep the
one-second expiry tests and document the HTTP deadline setup race.
* test: bound leader and replication requests
Limit how long leader lookup and transfer submission wait for an HTTP
response. A stalled submission must fail so the test can release its
transfer gates and clean up the peers.
* test: preserve readiness failures in diagnostics
Catch request failures while collecting cluster diagnostics, including
read timeouts. Report the original readiness failure instead of replacing
it with a diagnostic error.
* test: assert points calls for the correct collection
Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com>
* test: make sure check_cluster_size and check_leader cannot stall
Have an explicit timeout.
Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com>
* test: retry timeouts during initial leader lookup
Treat request timeouts as retryable while discovering the expected leader,
matching the subsequent leader and membership checks. Keep polling after
a transient timeout instead of aborting the cluster-status wait.
* test: ensure batch data is different
Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com>
---------
Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com>
The test compared raft commit indices across peers before issuing shard
cleanup. A follower advances its commit index before it applies the entry,
so cleanup could still race the follower applying CommitRead, which
invalidates running clean tasks and makes the endpoint return 500.
Wait for every peer to report the applied `read_hash_ring_committed`
resharding stage via telemetry instead. Reading it takes the same shard
holder lock as the consensus handler, so an observed stage is fully applied.
Repurpose the unused stage check helper to read the stage from telemetry,
since the `comment` field it read no longer exists.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
test_recover_from_snapshot_2 and test_upload_snapshot_2 start snapshot
recovery on a freshly joined peer as soon as it lists the collection. The
collection appears once the creation entry is applied, while the peer is
still replaying the rest of the raft log, including the removal of the
killed peer. Recovery then decides which other replicas to remove or mark
dead from that stale local view and drops a healthy replica, leaving a
shard with a single replica.
Add a helper that waits until all peers share the same commit index and
have no pending operations, and use it in both tests before recovering.
Also fix a misleading comment in the recovery replica cleanup branch.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Under stress, three concurrent user-requested snapshot transfers (10 000
points each) running while the killed peer recovers can starve the
leader's heartbeats long enough to trigger a raft election. If the
recovery transfer's `RecoveryToPartial` proposal is submitted while no
leader exists, raft drops it silently — `recovered_switch_to_partial`
returns Ok regardless because it only sends to a channel — and the
retry path then waits a full CONSENSUS_CONFIRM_TIMEOUT (10s) before
trying again. Combined with sequential per-shard recovery (auto
transfer limit = 1 × 3 shards), the 30s `/readyz` budget runs out.
Add an optional `wait_for_timeout` to `wait_for_peer_online` and bump
this test's wait to 60s. Default behaviour for other callers is
unchanged.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The test killed the last peer immediately after start_cluster, but
start_cluster only waits for cluster size and a known leader — not for
all peers to be promoted from learner to voter. If the last peer caught
up first, it became the only other voter alongside the leader; killing
it left a 2-of-2 quorum with one voter dead, and the subsequent
CreateCollection commit timed out after 10s.
Wait for all peers to be voters before killing one, so the survivors
form a 2-of-3 voter quorum.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
During a snapshot shard transfer recovery, the receiving peer
temporarily takes its local shard before installing the snapshot, so
the cluster info endpoint returns local_shards: [] for a brief window.
The test helper indexed [0] on that list and crashed with IndexError,
making test_triple_replication flaky. Treat the empty case as a
non-Active state so callers poll again instead.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [ai] On shard snapshot transfer recovery, drop existing shard before recovery
* [ai] Add integration test to assert clearing behavior
* [ai] Debug assert our replica is not active when we clear it
* [ai] Tweak assertion
* Fix flaky test, replica may temporarily not be visible
* Replace debug assertion with runtime error
* Fix consensus test port collision across xdist workers
When a peer was started with `get_port()` for each of p2p/grpc/http,
only the OS-allocated p2p_port was guaranteed free. On restart with
`port=p.p2p_port`, the framework derives grpc=port+1, http=port+2 — but
those neighbor ports were never reserved at original startup. Under
`pytest -n auto --dist=loadfile`, another xdist worker could legitimately
bind one of them, causing the restarted peer's REST bind to fail with
EADDRINUSE while the test still talks to that URL and gets a misleading
404 from the unrelated peer.
Add `get_port_triple()` which allocates a base port and probes bind() on
base+1 and base+2 to verify the contiguous slot is free across processes,
then route start_peer / start_first_peer (and their two callers in
test_peer_snapshot_bootstrap / test_cluster_rejoin) through it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Partition port allocation by xdist worker
Each pytest-xdist worker now draws peer ports from a disjoint slice
(BASE + worker_index * SIZE), so concurrent workers can never compete
for the same triple. Within the slice we still probe-bind() each
candidate to skip ports occupied by unrelated processes, and fall back
to OS allocation if the slice is exhausted.
This eliminates the cross-worker race that prompted the original fix,
and removes the need for the ±2 buffer dance against busy_ports.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: parallelize consensus tests with pytest-xdist
Enable pytest-xdist for consensus_tests to run tests across multiple
workers in parallel, significantly reducing CI wall time (~20min → ~5-7min).
Changes:
- Add `-n auto --dist=loadfile` to the consensus test pytest invocation
- Remove hardcoded port_seed from tests that don't need fixed ports for
restart/rejoin (test_order_by, test_consensus_compaction,
test_named_vector_crud, test_listener_node)
- Give test_cluster_rejoin its own PORT_SEED=15000 to avoid port
conflicts with auth tests (PORT_SEED=10000)
- Derive restart ports from killed PeerProcess objects instead of
hardcoded arithmetic where possible
- Add xdist_group("auth") marker to auth test files to ensure they
run on the same worker (they share PORT_SEED=10000)
Made-with: Cursor
* fix: remove remaining hardcoded port_seed=20000 causing parallel test conflicts
8 test files were using port_seed=20000 as a positional argument to
start_cluster(), which was missed in the initial change. When running
in parallel with pytest-xdist, multiple workers would try to bind to
the same port range (20000-20x02), causing port conflicts and cascading
test failures.
Also remove port_seed=23000 from test_snapshot_recovery_kill.py since
it doesn't need fixed ports for restart.
Made-with: Cursor
* fix: use saved port for restart in test_two_follower_nodes_down
The test was restarting killed peers on hardcoded ports (20200/20100)
that previously matched port_seed=20000. After switching to random
ports, the restart ports no longer match the original peer ports,
causing raft state URI mismatches and peer startup failures.
Save the p2p_port from the killed PeerProcess and reuse it for restart.
Made-with: Cursor
* Reuse p2p ports when restarting killed peers in consensus tests
When a peer is killed and restarted with random ports, it gets a new
consensus URI. The cluster needs a Raft operation to update this URI,
which under CPU contention from parallel test workers can exceed the
30-second timeout. Fix by capturing each peer's p2p_port before killing
and reusing it on restart, so the URI stays the same and no consensus
update is needed.
Made-with: Cursor
* A few improvements for parallel runs (#8731)
* fix: make auth tests' PORT_SEED per-worker to avoid port collisions
* ci: improve failure visibility for parallel consensus tests
* Three small changes to make hangs, interleaved output, and coverage runs behave predictably under pytest-xdist
* fix: two test bugs surfaced by parallel runs and revert drop PR_SET_PDEATHSIG helper
* fix: wait for count convergence in test_triple_replication
* fix: clean leaked peer processes at test start
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add integration test to assert all queued updates are also transferred
* Plunge update queue in stream records transfers
* Migrate existing plunger usages to new plunge helper
* Skip test if not compiled with staging flag
* Only send delay operation when staging feature is enabled
* Reformat
* Fix review remarks
* introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert
* add test
* upd dockstring
* require resharding once all peers have updated version
* use service error
* fix clippy again
* wait for same version before resharding in tests
* Add test to reproduce broken WAL delta after aborting stream records
* Add staging env var to slow down stream records transfers for test
* Tweak test formatting and utilities a bit
* Add comment to test, link to PR describing bug
* Update test so it still succeeds with patched behavior
* Fix broken WAL delta after stream records abort (#7791)
* Make set_replica_state async
* Add function called when active state of local replica changes
* Add snapshot for newest clocks
* Bump newest clocks snapshot on replica deactivation
* Use newest clocks snapshot during recovery
* Add enum for specifying whether to take or clear clocks snapshot
* Store clock snapshot inside clock map, removing extra file
This greatly simplifies state handling. It also prevent any kind of
desynchronization because all newest clocks are always persisted
atomically.
* Immediately persist clocks after taking snapshot
* Always update snapshot, only take if missing
* Take clock snapshots through each shard flavor, including proxies
* Propagate dedicated functions for taking and clearing clocks snapshot
* Only persist clocks immediately if changed on snapshot/clear
* Simplify recovery point logic, always take clocks snapshot if exists
* Remove unwrap
* Fix typo
* Fix doc comment
* Transfer driver is async, use Tokio sleep
* Reduce visibility
Do not release the port before the process is really gone
otherwise it can be wrongly reused.
E.g.
2025-11-14T10:39:20.156573Z ERROR qdrant::startup: Panic occurred in file src/tonic/mod.rs at line 358:
called `Result::unwrap()` on an `Err` value: tonic::transport::Error(Transport, hyper::Error(Listen, Os { code: 98, kind: AddrInUse, message: "Address already in use" }))
* Respect payload and hashring filters while replicating point
* Merge points instead of replacing for filtered stream records transfer
* cleanup
* Update utils
* Allow filters in transfer
* drop hashring filter since its not required when destination has single shard
* test comment
* Pass actual filter
* use exact=True
* Forward updates that satisfy the filter
* make test long enough for extra points to be upserted
* minor improvements
* fix test
* try with only inserting new points
* Temporarily forward points that match before or after
* Fix failing test
* Only do inserts
* Dont merge points and update old points in test
* Fix the bug in integration test
* fix read_batch_with_hashring fn name and docs
* changes after rebase
* add validation check
* apply suggestions
* Always trim clock tag when forwarding to a different shard ID
* review fixes
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* remove default explicit config for strict mode
* Don't prefix commented out properties with space to remain consistent
* Don't crash test if strict mode config is not set at all
---------
Co-authored-by: timvisee <tim@visee.me>
* Fix force delete peer flaky test
* Temporarily only run force delete peer test
* Run till failure
* Add mutable transfer field
* Introduce wait_with_return and use it to solve TOCTOU
* Revert "Run till failure"
This reverts commit 53319a19c8.
* Revert "Temporarily only run force delete peer test"
This reverts commit 5ea47e30a6.
* Simplify the test instead of introducing wait_with_return
* revert changes to utils
* Temporarily run fewer tests in CI to reproduce flakiness
* disable other integration test
* log collection topology if request fails
* Run in CI until failure
* update tests
* Catch failures and log
* improve logging for search result comparison flakiness
* fix test
* Simplify test
* Use scroll instead of search and simplify existing code
* Trigger CI
* new util function
* debug again
* print collection topology on mismatch
* wait for deletion of flag after recovery is done
* use wait_for
* Revert integration-tests.yml
* Introduce coverage reports for integration tests
* Install cargo-llvm-cov
* Use multiline script
* Explicityl setup COVERAGE env var
* fix integration tests and log generated data
* fix ls path
* upload lcov file to GH artifacts
* integration profraw dynamic filename
* Fix llvm profile filename template
* Use interrupt instead of kill and merge consensus test results into same file
* Drop upload artifact stage
* install llvm-cov
* upload as artifact and export coverage files
* try simplifying workflow
* Migrate coverage generation to existing dedicated gh workflow
* trigger on coverage related branches
* Build only if qdrant binary with cov doesnt exist
* Use valid yaml
* include mode in profraw filename
* split coverage workflow into parallel jobs
* add poetry version to env
* log poetry version to install
* clean up integration test workflow
* Simplify comments
* Test behaviour of Qdrant with shard initializing flag
* Corrupt shard directory and let Qdrant panic like prod
* Wait for shard transfer
* Restore dirty shards using other replicas
* remove unused code
* Request transfer only if replica is dead or dirty
* fmt
* remove comment
* fix clippy
* Delete shard initializing flag after initializing empty local shard
* Expect test to recover shard in existing test
* Review suggestions
* Run tests for longer
* Simplify tests
* Use 2k points
* condition for point_count
* Add comment
* fix flaky tests
* fix flaky tests
* handle edge case
* Include Active in expected states list
* Introduce is_recovery
* simplify tests
* get rid of is_dirty bool in DummyShard
* add missing negation in condition
* fix condition
* final fix for transfer condition
* Don't auto recover if in recovery mode, simplify state checking
* minor comment improvements
* tests scenario where node is killed after deleting shard initializing flag
* Fix failing CI
* Only automatically recover dead replicas
* Mark replica as dead to recover dummy shard
* fix failing test
* Sleep one second after killing peer, give time to release WAL lock
* Prevent waiting for peer to come online indefinitely
* update comment
* minor typo
---------
Co-authored-by: timvisee <tim@visee.me>
* Measure update operations hardware IO
* Add support for distributed setups
* also measure update_local
* Add consensus tests for HW metrics of update operations
* add test for upserting without waiting
* Disable HW usage reporting when not waiting for update API
* Review remarks
* Fix resharding collecting hw measurements
* Fix metric type
* New struct HardwareData for better accumulation
* Ensure we always apply CPU multiplier
* Apply suggestions from code review
* Update src/actix/api/update_api.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Fix assert_with_upper_bound_error threshold calculation.
* Clarifying why we don't measure shard cleanup
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add retry-after header for gRPC
* propagate retry-after within inter node communication
* snipe
* tweak and test
* log all parsing errors
* Update lib/collection/src/operations/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Restructure process spawning in consensus tests, build list of args
* Extend consensus test, also test if providing URIs through env vars work
* Set URI and bootstrap variables unconditionally
* Strict Mode: distributed checking of max collection size
* add size projections in distributed mode
* Add consensus tests
* New Test: All nodes in cluster
* fix tests
* Update lib/collection/src/collection/mod.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* increase upsert delay
* add TODO for resharding
* wait for strict mode config to be applied on second node
* remove delays
* Also wait for strict mode in other test
* clearify strict mode config option
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* option to clear consensus state while preserving peer id
* compact or clear consensus WAL on re-init
* Add test on reinit
* minor changes
* more points check suggestion
---------
Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
* Resume resharding driver on restart if resharding is active
* Update resharding state comment started text
Using since versus started. 'Since' better clarifies that the current
state has been active since that time. While 'started at' could lead to
confusion on whether that time is for starting the whole resharding
operation or just that state.
* Add resharding resumption test during migrating points
* Recover resharding hash ring after loading shards
* Update resharding resume test, interrupt at multiple stages, less points
* Remove unused parameter
* Update collection shard count at resharding shard creation/destruction
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Use debug_assert_eq
* Remove trailing comma
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Apply resharding filter only to existing shards, include shard selection
* Add test for stable exact point count during resharding
* Improve filtering, only two separate requests, filtered and non-filtered
* Make scrolling stable while resharding
* Add test for stable scroll during resharding
* Make search stable while resharding
* Add test for stable search during resharding
* Remove resharding post filter in retrieve
* Also assert cardinality point count in resharding test
* Fix typos and some tweaks
* Only clone filter if resharding is active
* Also minimize cloning with resharding filter on count request
* Add test for stable exact point count during resharding with indexing
* Update lib/collection/src/shards/shard_holder/mod.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Update lib/collection/src/collection/point_ops.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Also apply resharding filter to retrieve
* Restructure resharding filter usage in `Collection::retrieve`
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add basic resharding test, asserting shard and point counts
* Add resharding replica balancing test
* Always propagate deletes with wait, fix first resharding test flakyness
* Don't count points in balance test for now
* Fix resharding balance test, initial shard may not be on first node
* Count points in resharding balance test
* Test resharding with concurrent updates
* Add point data consistency check to resharding tests
* Fix not using given collection name, list point counts in debug output
* Extract logic for creating thresholds config
* Put collection params and threshold config in update handler
* Add function to add a new appendable segment if all are over capacity
* Make new method static, call it before each optimization loop
* Update error message formatting
* Use exact point count in replication consensus test
* Add a test to assert segment creation when all are over capacity
* Suffix optimizer thresholds with _kb to clarify unit
* Move segment capacity check logic, run if optimizers are disabled
* fix: add -> mul
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Add API key to HTTP client
* Add API key field to snapshot recovery requests
* Add API key to channel service
* Provide API key when doing snapshot transfer
* Configure API key header name constant in a central place
* Reformat
* Update OpenAPI spec
* Remove suffixed spaces from configuration file
* Allow to specify HTTP headers in some consensus test utility functions
* Add snapshot transfer test with configured API key
* Use random API key in test
* Fix compilation errors and clippy warnings
* separate access tests from validation tests
* bring changes from rbac-integration-tests review fixes
* bring changes from rbac-payload-access-tests
* update from base branch
* use const instead of magic number
* move COLL_NAME
* force usage of kill_all_processes()
* create exhaustive access test (missing body stubs)
* use json body in rest requests
* use better stubs
* green test for create_shard_key
* make it work for delete_shard_key
* make it work for list_collections
* make it work for get_collection
* make it work for create_collection
* make it work for update_collection_params
* make it work for delete_collection
* make it work for update_aliases
* make it work for create_index
* make it work for collection_exists
* make it work for delete_index
* make it work for get_collection_cluster_info
* don't build grpcio and use `uv` for faster deps installation
* partially prepare for update cluster ops
* make it work for list_collection_aliases
* make it work for list_aliases
* make it work for list_collection_snapshots
* make it work for create_collection_snapshot
* huge refactor + make it work for delete_collection_snapshot
* make it work for download_collection_snapshot
* test_upload_collection_snapshot
* test_recover_collection_snapshot
* test_recover_collection_snapshot
* test_upload_shard_snapshot
* test_recover_shard_snapshot
* test_list_shard_snapshots
* test_create_shard_snapshot
* test_delete_shard_snapshot
* test_download_shard_snapshot
* test_list_snapshots and test_create_snapshot
* test_delete_snapshot
* test_download_full_snapshot
* test_get_cluster
* test_recover_raft_state
* test_delete_peer
* prepare for splitting into another PR
* skip api exhaustiveness checks
* lil cleanup
* fix integration tests workflow
* test search api
* test recommend api
* test discover api
* test count and scroll
* test get and upsert points
* test update points batch
* test update and delete vectors
* test set, overwrite, delete, and clear payload
* test service level actions
* finish testing (with errors), introduce not needing success
* make tests much faster
* simplify delete tests
* complete collection cluster operations
* cleanup
* update after rebase
* change permissions for create and delete shard keys
* review fixes
* add newline
Anton Antonov