When the restarted peer's dummy shard is auto-recovered by the
cluster's recovery loop before the test issues its manual
`replicate_shard` call, the manual call returns 400 "already involved
in transfer". Skip the manual call when a transfer is already in
flight — the existing wait_for / transfer-count / replica assertions
still verify the shard recovers.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Raw requests already use QDRANT_HOST, but bypass request_with_validation
and missed QDRANT_HOST_HEADERS. Without these headers, tests cannot run
behind host-based reverse proxies or mocks.
Use unique point IDs across all phases instead of overwriting the same
id repeatedly. The gridstore payload storage size estimate is
bitmask-based: overwrites keep old blocks allocated until a periodic
flush reclaims them, so the test was racing the 5s flush worker. With
unique ids every block stays live and the post-flush size still
reflects all inserted points.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The test waited a fixed 0.5s after each PUT/DELETE before reading from
every peer, which raced with raft apply on followers under CI load.
Replace the fixed sleeps with wait_for-based polling so each per-peer
read retries until the expected value is observed.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Under stress, three concurrent user-requested snapshot transfers (10 000
points each) running while the killed peer recovers can starve the
leader's heartbeats long enough to trigger a raft election. If the
recovery transfer's `RecoveryToPartial` proposal is submitted while no
leader exists, raft drops it silently — `recovered_switch_to_partial`
returns Ok regardless because it only sends to a channel — and the
retry path then waits a full CONSENSUS_CONFIRM_TIMEOUT (10s) before
trying again. Combined with sequential per-shard recovery (auto
transfer limit = 1 × 3 shards), the 30s `/readyz` budget runs out.
Add an optional `wait_for_timeout` to `wait_for_peer_online` and bump
this test's wait to 60s. Default behaviour for other callers is
unchanged.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [ai] Add integration test for triggering inconsistent resharding state
* [ai] Also add test for resharding down
* Update test
* Resolve resharding idempotency through setting replica states
* Remove resharding down test
* Reformat
* [ai] Remove resharding abort order, abort before setting replica state
* [ai] Resolve test flakiness
* Collapse matches into helper function
* Check preconditions before aborting resharding
* Fix test flakiness by not waiting for a dead node
* Abort resharding before aborting transfer for idempotency
* Update comment
* Release shard holder lock on transfer/reshard abort to prevent deadlock
* Remove now unused shard holder parameter
* Split handle_replica_changes to eliminate need for juggling locking
* In resharding tests, import all utils to enable every_test cleanup
Resolves flakiness I've been seeing in
test_set_replica_dead_clears_resharding_state test
The test killed the last peer immediately after start_cluster, but
start_cluster only waits for cluster size and a known leader — not for
all peers to be promoted from learner to voter. If the last peer caught
up first, it became the only other voter alongside the leader; killing
it left a 2-of-2 quorum with one voter dead, and the subsequent
CreateCollection commit timed out after 10s.
Wait for all peers to be voters before killing one, so the survivors
form a 2-of-3 voter quorum.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The rapid drop/create loops in test_rejoin_cluster intentionally use
short 3s timeouts to accumulate Raft log entries quickly. Under CI
load the consensus apply for CreateCollection can exceed 3s (segment
setup competes with background flushes/optimizations), and the API
returns 500 even though the operation reaches consensus right after.
The matching upserts already pass `fail_on_error=False`; do the same
for `create_collection` to make the test resilient to that race.
These tests rely on the collection size stats cache being refreshed to
detect that a size limit has been exceeded. Without wait=true, upsert
operations are written to WAL and acknowledged immediately without being
applied to segments. When the cache refreshes, it reads segment data
which may not yet reflect the pending WAL operations, causing the size
check to see stale values and not reject the request.
Adding wait=true ensures operations are applied to segments before the
response returns, so the cache refresh sees the correct sizes.
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The deadline loop only waited for the download phase to become
"streaming", but the assertion also required bytes > 0. On CI the
assertion could fire before iter_content yielded the first chunk,
causing a flaky failure. Wait for bytes > 0 in the deadline loop
and improve the error message to show bytes/error state.
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: keep optimizers disabled during snapshot transfer in deferred test
The snapshot variant of test_shard_transfer_includes_deferred_points was
flaky because optimizers were enabled before the transfer, letting the
optimizer race ahead and fully index the segment before the snapshot was
captured (~1s of HNSW build for 500 small vectors fits comfortably before
the snapshot is taken). The deferred-state assertion then fails since all
points are already visible.
Only enable optimizers before the transfer for stream_records (which needs
them for its internal wait=true). For snapshot, leave optimizers disabled
through the transfer so deferred state is preserved on the wire, then
enable them afterwards for trigger_upsert_wait_true. The hung server-side
wait=true from the timeout-and-retry block does not block the snapshot —
wait_for_deferred_points_ready runs in a detached tokio::spawn.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: skip wait=true probe for snapshot variant
CI showed that with optimizers kept disabled through the snapshot transfer
(needed to preserve deferred state on the wire), the wait=true probe at
the start of the test leaves a hung server-side request: update_local
holds local.read() until the deferred wait resolves, and there is no
optimizer to resolve it. The subsequent shard transfer's apply path
deadlocks against that held read lock when queue_proxify_local tries to
take local.write().
For stream_records the config update later cancels the hung worker, so
the probe is fine there. Move the probe (and config update) under the
stream_records branch so the snapshot variant doesn't leave a hung
update around. The probe was auxiliary behaviour verification, not
central to the snapshot-of-deferred-points assertion.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Revert "test: skip wait=true probe for snapshot variant"
This reverts commit d07aac78a263d7a91691e43444b9dae44e3d179f.
* test: add reproducer for deferred-wait shard-transfer deadlock
Adds test_shard_transfer_with_hung_deferred_wait_does_not_deadlock as a
focused reproducer for the engine bug surfaced by the snapshot variant
of test_shard_transfer_includes_deferred_points.
Lock-ordering chain:
1. With prevent_unoptimized=true and max_optimization_threads=0, a
wait=true upsert on deferred points enters
wait_for_deferred_points_ready (update_worker.rs:241), which loops
on tokio::select over cancel and optimization_finished. The
optimization_worker hits limit==0 and `continue`s without firing
optimization_finished_sender (optimization_worker.rs:172-174), so
neither branch of the select ever fires.
2. update_local (replica_set/update.rs:49) holds self.local.read()
across the entire update await. actix-web does not cancel the
response future on client disconnect, so the read guard stays alive
even after the client's 5s timeout.
3. A subsequent snapshot transfer eventually calls queue_proxify_local
(replica_set/shard_transfer.rs:122), which needs self.local.write().
tokio::sync::RwLock is write-preferring: the queued writer blocks
new readers, including is_local() calls on the consensus apply
path itself (shard_transfer.rs:129-130). The apply never returns,
the consensus broadcast never fires, POST /cluster times out with
"Waiting for consensus operation commit failed".
The new test asserts the symptom (POST /cluster must return promptly)
without papering over the bug, so it stays red until the engine is
fixed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(replica_set): release local read guard around deferred-points wait (#8862)
* fix(replica_set): drop remotes read guard early in update_impl
`update_impl` was holding `self.remotes.read()` and `self.local.read()`
across the entire update await, including the deferred-points wait that
can park indefinitely under prevent_unoptimized + max_optimization_threads=0.
When a shard transfer is started concurrently with a parked wait=true
update, the consensus apply runs `add_remote`, which calls
`self.remotes.write().await`. tokio::sync::RwLock is write-preferring:
the queued writer is blocked behind the held read, the apply never
returns, and `POST /cluster` times out with "Waiting for consensus
operation commit failed".
Fix: snapshot updatable remote shards into owned `Vec<RemoteShard>` and
drop the read guard before the await. The remote_update futures now own
the cloned RemoteShards, so they no longer borrow from the guard.
The `local` guard is still held across the await (futures borrow
`&Shard` from it). Releasing it would unblock `queue_proxify_local`'s
`local.write()` too, but that requires wrapping `Shard` in `Arc` —
deferred to a follow-up. For the consensus-commit-timeout deadlock
exposed by `test_shard_transfer_with_hung_deferred_wait_does_not_deadlock`,
dropping `remotes` is sufficient.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(updater): wake deferred wait on caller-receiver drop
`wait_for_deferred_points_ready` parked on a `tokio::select` over
`cancel.cancelled()` and `optimization_finished_receiver.changed()`.
Under prevent_unoptimized + max_optimization_threads=0, neither fires:
optimization_worker.rs:171-174 hits `limit == 0` and `continue`s
without notifying, and the cancel token is the worker's lifecycle
token (only fired by stop_update_worker on config update / shutdown).
The top-of-loop `is_closed()` poll didn't help — the loop never
re-runs once the select parks.
Take `feedback_sender` by `&mut` and add `feedback_sender.closed()`
as a third select branch. When the matching `Receiver` is dropped
(by upstream cancellation, client-supplied timeout, or any future
cancellation), the detached task wakes immediately and exits with
WaitTimeout instead of staying parked until the next worker restart.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [AI] split update operarion into submit and independent wait function
* [AI] refactor `update_local` to drop local shard lock after submitting update operation
* [AI] refactor `update_impl` for early release of the lock in case of local shard update
* fmt
* Apply suggestion from @generall
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
During a snapshot shard transfer recovery, the receiving peer
temporarily takes its local shard before installing the snapshot, so
the cluster info endpoint returns local_shards: [] for a brief window.
The test helper indexed [0] on that list and crashed with IndexError,
making test_triple_replication flaky. Treat the empty case as a
non-Active state so callers poll again instead.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Make resharding state transitions idempotent on replay
Why: consensus entries may be re-applied after a crash (partial state
on disk) or during raft recovery. The unchecked state-transition
helpers used `debug_assert!` to require a specific starting state, so
a replay would panic in debug or silently overwrite in release.
How to apply: use write_optional so the state file is only touched
when the in-memory state needs to change. This also avoids unnecessary
fsyncs when a replay is a no-op.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Skip replica set creation on resharding start replay
Why: on replay, `create_replica_set` would call `create_shard_dir`,
which removes and recreates the existing shard directory -- wiping
any shard contents that had already been migrated or written since
the first apply.
How to apply: check `contains_shard(shard_id)` before creating the
replica set and pass `None` to `start_resharding_unchecked` when a
replica set with the target shard id is already present. Also
relaxes `check_start_resharding` to return `Ok` (instead of a
swallowed `bad_request`) when a matching resharding state is
persisted, so the caller can fall through each idempotent step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Derive resharding shard count from shard id for idempotency
Why: the start/finish/abort paths used ++/-- on the persisted shard
number with a `debug_assert` pinning the expected starting value. On
replay (e.g. after a crash between the shard holder mutation and the
config save) this either panics or produces a wrong count.
Since resharding always targets the last shard (auto sharding assigns
contiguous ids from zero), the target count is a pure function of the
shard id: `shard_id + 1` for start up, `shard_id` for finish down and
abort up. Set it directly and skip the save when it already matches,
so replay converges to the same value without touching the file.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fall through resharding finish/abort steps on replay
Why: the early return on `resharding_state.is_none()` assumed that a
missing state means the operation was fully applied. But a crash can
leave the state cleared while the shard drop, key mapping removal, or
shard count update are still pending. Short-circuiting then skips the
reconciling work those replays need to do.
How to apply: drop the early return so every step runs; each step is
already individually idempotent (check_*, drop_and_remove_shard,
remove_shard_from_key_mapping, the set-based shard count update).
The abort_resharding down-invalidation path now reads nodes from the
router regardless of its variant, so a replay over an already-rolled-
back ring doesn't trip the removed debug_assert!s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Skip shard key mapping write when nothing to remove
Why: `write_optional` returns `Some` whenever the shard key is
present, even if the shard id we're removing is already gone. That
still triggers a JSON rewrite and a cache notification on every
replay of a finished finish/abort.
How to apply: check that the shard id is actually in the set before
returning `Some`. If the set is missing or already doesn't contain
the id, return `None` so the on-disk file and in-memory data are
left untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add tests for resharding replay idempotency
Covers the paths that must not error or panic on replay:
- `check_start_resharding` when matching state is already present
- `start_resharding_unchecked` preserves matching state verbatim
- `finish_resharding_unchecked` when state is already cleared
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Simplify finish_resharding_unchecked closure
Replace a manual `match` on `Option` with `as_ref().map(...)` to
satisfy clippy::manual_map.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: add a test for resharding replay (#8839)
* [ai] add a debug_assert on shard_number (#8846)
* Update lib/collection/src/shards/shard_holder/resharding.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Reformat
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* [ai] On shard snapshot transfer recovery, drop existing shard before recovery
* [ai] Add integration test to assert clearing behavior
* [ai] Debug assert our replica is not active when we clear it
* [ai] Tweak assertion
* Fix flaky test, replica may temporarily not be visible
* Replace debug assertion with runtime error
The test asserted segment count growth and payload index backfill
immediately after upsert, but both are produced asynchronously by the
optimization worker (new appendable segment via
`ensure_appendable_segment_with_capacity`, payload index backfill).
Replace the bare asserts with `wait_for` polling.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The PUT with wait=true only awaits consensus apply on the leader, so
followers can still serve a stale value from their local persistent
state shortly after. The fixed 0.5s sleep was occasionally too short
on slow CI runners (observed ~1.15s follower lag), causing
test_consensus_compaction to fail with a None metadata value on
peers[1].
Replace the sleep + one-shot assertion with per-peer wait_for polling.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix consensus test port collision across xdist workers
When a peer was started with `get_port()` for each of p2p/grpc/http,
only the OS-allocated p2p_port was guaranteed free. On restart with
`port=p.p2p_port`, the framework derives grpc=port+1, http=port+2 — but
those neighbor ports were never reserved at original startup. Under
`pytest -n auto --dist=loadfile`, another xdist worker could legitimately
bind one of them, causing the restarted peer's REST bind to fail with
EADDRINUSE while the test still talks to that URL and gets a misleading
404 from the unrelated peer.
Add `get_port_triple()` which allocates a base port and probes bind() on
base+1 and base+2 to verify the contiguous slot is free across processes,
then route start_peer / start_first_peer (and their two callers in
test_peer_snapshot_bootstrap / test_cluster_rejoin) through it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Partition port allocation by xdist worker
Each pytest-xdist worker now draws peer ports from a disjoint slice
(BASE + worker_index * SIZE), so concurrent workers can never compete
for the same triple. Within the slice we still probe-bind() each
candidate to skip ports occupied by unrelated processes, and fall back
to OS allocation if the slice is exhausted.
This eliminates the cross-worker race that prompted the original fix,
and removes the need for the ±2 buffer dance against busy_ports.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [ai] Add config boolean to disable URL based snapshot restore
* Merge if-statement
* Comment-out config option by default
* [ai] Also block partial snapshots from remote URLs
* [ai] Only run clock consistency test when staging feature is present
* [ai] Add integration test
* ci: parallelize consensus tests with pytest-xdist
Enable pytest-xdist for consensus_tests to run tests across multiple
workers in parallel, significantly reducing CI wall time (~20min → ~5-7min).
Changes:
- Add `-n auto --dist=loadfile` to the consensus test pytest invocation
- Remove hardcoded port_seed from tests that don't need fixed ports for
restart/rejoin (test_order_by, test_consensus_compaction,
test_named_vector_crud, test_listener_node)
- Give test_cluster_rejoin its own PORT_SEED=15000 to avoid port
conflicts with auth tests (PORT_SEED=10000)
- Derive restart ports from killed PeerProcess objects instead of
hardcoded arithmetic where possible
- Add xdist_group("auth") marker to auth test files to ensure they
run on the same worker (they share PORT_SEED=10000)
Made-with: Cursor
* fix: remove remaining hardcoded port_seed=20000 causing parallel test conflicts
8 test files were using port_seed=20000 as a positional argument to
start_cluster(), which was missed in the initial change. When running
in parallel with pytest-xdist, multiple workers would try to bind to
the same port range (20000-20x02), causing port conflicts and cascading
test failures.
Also remove port_seed=23000 from test_snapshot_recovery_kill.py since
it doesn't need fixed ports for restart.
Made-with: Cursor
* fix: use saved port for restart in test_two_follower_nodes_down
The test was restarting killed peers on hardcoded ports (20200/20100)
that previously matched port_seed=20000. After switching to random
ports, the restart ports no longer match the original peer ports,
causing raft state URI mismatches and peer startup failures.
Save the p2p_port from the killed PeerProcess and reuse it for restart.
Made-with: Cursor
* Reuse p2p ports when restarting killed peers in consensus tests
When a peer is killed and restarted with random ports, it gets a new
consensus URI. The cluster needs a Raft operation to update this URI,
which under CPU contention from parallel test workers can exceed the
30-second timeout. Fix by capturing each peer's p2p_port before killing
and reusing it on restart, so the URI stays the same and no consensus
update is needed.
Made-with: Cursor
* A few improvements for parallel runs (#8731)
* fix: make auth tests' PORT_SEED per-worker to avoid port collisions
* ci: improve failure visibility for parallel consensus tests
* Three small changes to make hangs, interleaved output, and coverage runs behave predictably under pytest-xdist
* fix: two test bugs surfaced by parallel runs and revert drop PR_SET_PDEATHSIG helper
* fix: wait for count convergence in test_triple_replication
* fix: clean leaked peer processes at test start
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
submit_collection_meta_op was returning after the local apply on the
leader for these two ops, so a follow-up GET on a different peer could
still see the old vectors config until that peer applied the consensus
entry. Add them to the do_sync_nodes branch so the API only returns once
all reachable peers have caught up, matching CreateCollection /
CreateShardKey.
Drop the sleep(1) workarounds in test_vector_crud_with_consensus_snapshot
and tighten the client timeout on calls made while a peer is killed so the
server-side sync bounds quickly on the unreachable node.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The compatibility tests previously only checked that collections loaded
with "ok" status. This adds actual queries against every collection to
catch errors/panics in on-disk code paths during version upgrades.
For each collection, we now run:
- Dense vector search (image, dim 256)
- Sparse vector search (text)
- Multivector search (multi-image, dim 128)
- Scroll with filters for all payload index types: keyword, float,
integer, boolean, geo bounding box, full-text, uuid, datetime
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
The WAL delta manual recovery tests SIGKILL the uploader python process
and then SIGKILL the qdrant peer. If the uploader's last HTTP upsert was
still in-flight when the peer dies, the peer may have partially
replicated the batch — some replicas get it, some don't — and the peer
dies before it can mark the lagging replicas Dead. That divergence
persists across the subsequent recovery chain, because the untouched
replicas never participate in any transfer.
Observed as `test_shard_wal_delta_transfer_manual_recovery_chain`
failing with "Data on all nodes should be consistent": peer_0_1 had
batch 500006-500008 (forwarded by peer_5 pre-kill), peer_0_0 did not,
and neither peer was a source or destination of any transfer, so the
gap was never reconciled.
Fix: give the uploader loop a `stop_event` that it checks between
upserts. Add `stop_update_process` helper that sets the event, joins
the process, and only SIGKILLs as a timeout fallback. Use it before
every `processes.pop().kill()` so no upsert is in-flight when the
peer dies.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [AI] strict mode parameter for limiting update requests if ram usage is over threshold
* opanAPI update
* [AI] end-to-end test
* fmt
* Fix e2e test: memory rejection check broken by string truncation
UnexpectedResponse.__str__() truncates the raw response body, cutting
off the `max_resident_memory_percent` hint at the end of the error
message. Use `resident memory usage` instead, which appears early
enough to survive the truncation.
Made-with: Cursor
* add grpc validation
* test check_resident_memory
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
The documented format `2023-02-08T10:49` was not accepted because the
parser only had `%Y-%m-%d %H:%M` (space separator) but was missing
`%Y-%m-%dT%H:%M` (T separator). Add the missing format variant and
tests.
Closes#8718
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
The test restarts a killed peer, waits for automatic recovery, and
asserts that no more than 1 shard transfer is ongoing at a time.
However, it also issues 3 user-requested replicate_shard operations
(peer[0] -> peer_4) before restarting the killed peer. The check
reads /collections/<name>/cluster, which returns the whole
collection's shard_transfers set — so those 3 user transfers are
counted against the auto-recovery limit of 1, and
transfers_below_limit_or_done raises "3/1" as soon as the loop runs.
User-requested transfers intentionally bypass the automatic
shard-transfer limit (see check_auto_shard_transfer_limit, only
consulted when proposing automatic recoveries). The limit applies to
transfers involving the peer being checked. Filter shard_transfers
accordingly: only count ones where the recovering peer is either
source or destination. Concurrent user transfers between unrelated
peers no longer trigger a false failure.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: add failing tests for non-idempotent resharding consensus ops
Add unit tests proving that resharding operations (abort, commit_read,
commit_write, finish) return `BadRequest` errors instead of `Ok` when
re-applied or when local state diverges between peers.
Since `apply_entries` silently swallows all non-`ServiceError` results
(including `BadRequest`), these errors cause permanent resharding state
divergence between peers while consensus term/commit remain identical.
All 6 tests fail on dev, demonstrating the bug.
Made-with: Cursor
* fix: make resharding consensus operations idempotent (#8523)
* fix: make resharding consensus operations idempotent
When apply_entries processes a resharding operation that returns
BadRequest (not ServiceError), the error is silently swallowed and the
entry is marked as applied. If local state on a peer diverges (crash
during partial apply, prior swallowed error), the same committed Raft
entry produces different outcomes on different peers — permanent
resharding state divergence despite identical consensus term/commit.
Fix by making all resharding operations return Ok when the desired
post-condition is already met:
ShardHolder level:
- check_abort_resharding: return Ok when no resharding active or
different resharding active (already aborted)
- commit_read_hashring: return Ok when no resharding or stage already
past ReadHashRingCommitted
- commit_write_hashring: return Ok when no resharding or stage already
past WriteHashRingCommitted
- check_finish_resharding: return Ok when no resharding (already
finished)
- check_start_resharding: warn instead of error when shard already
exists for Up direction (leftover from crashed previous attempt)
Collection level (defense in depth):
- start_resharding: return Ok if same resharding already in progress
- commit_read/write_hashring: return Ok if no resharding or past stage
- finish_resharding: return Ok if no resharding active
- abort_resharding: return Ok if no resharding active
Made-with: Cursor
* Fix formatting issue
* Remove resharding idempotency note from logs
* Don't hold shard holder lock
* Committing read/write hash rings are idempotent now and are allowed
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: timvisee <tim@visee.me>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: timvisee <tim@visee.me>
* Test if io_uring handles EINTR properly
* Fix unit test compilation after read_iter API change
Update test_io_uring_eintr_handling to match the new read_iter signature
that takes (Meta, ReadRange) tuples and returns Result<impl Iterator>.
Made-with: Cursor
* Install no-op SIGUSR1 handler in debug mode on Unix
Prevents SIGUSR1 from terminating the process with the default
disposition, so that io_uring EINTR tests can safely bombard
the process with signals.
Made-with: Cursor
* Enter tokio runtime context for SIGUSR1 handler, fix clippy
tokio::signal::unix::signal requires a reactor context, so enter
the runtime handle before installing the handler.
Also fix manual_let_else clippy warning in the EINTR unit test.
Made-with: Cursor
* Cleanup 🙄
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add consistency to test_resharding_deferred
Add consistency to the test_resharding_transfer_deferred_points test,
which puts shards in the Active state from the ReshardingScaleDown state
* Review fixes
* Sleep once after activating shards when resharding down
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add mincore-based memory stats to MmapFile
Add `resident_bytes()`, `disk_bytes()`, and `probe_memory_stats()` methods
to `MmapFile` for measuring page cache residency via `mincore(2)`. This is
the foundation for per-collection memory usage reporting.
Also extract `page_size()` as a public function in `mmap::advice`, replacing
the internal `PAGE_SIZE_MASK` with a direct page size cache.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] introduce trait for reporting memory usage per component
* [AI] memory reporter implementation for vector storage
* [AI] implement MemoryReporter for QuantizedVectors
* [AI] implement MemoryReporter for VectorIndexEnum
* Implement MemoryReporter for IdTrackerEnum with RAM estimation
Add ram_usage_bytes() to all ID tracker types and their data structures:
- PointMappings, CompressedPointMappings, CompressedVersions,
CompressedInternalToExternal, CompressedExternalToInternal
- MutableIdTracker, ImmutableIdTracker, InMemoryIdTracker
All ID trackers load their data into RAM (none use mmap for working data).
Files are reported as OnDisk (persistence only), actual RAM footprint
is reported via extra_ram_bytes. Uses struct destructuring to ensure
new fields trigger compile errors if not accounted for.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] implement MemoryReporter for PayloadStorageEnum and adjust FileStorageIntent
* [AI] implement MemoryReporter for PayloadStorageEnum and adjust FileStorageIntent
* [AI] implement MemoryReporter for payload indexes: in-ram structures memory consumtion computation + caching
* [AI] implement MemoryReporter for payload indexes: in-ram structures memory consumtion computation + caching
* [AI] segment-level memory usage report
* [AI] Block 3: Aggregation Layer and Data Model + internal api for remote shard
* [AI] REST API handler
* fmt
* [AI] clippy fixes
* [AI] macos fix + proxy segment fix
* [AI] make text index estimation a bit more correct
* fix is_on_disk reporting for dense_vector_storage
* fix after rebase
* [AI] deep account for quantized vectors RAM usage + unify chunk size + shring volatile storage after load
* remove debug log
* cache in test
* make manual test easier to run
* rollback chunk size diff, but keep it for test only
* review fixes
* Use exhaustive match
* Use div_ceil on bits everywhere
It does not seem to be strictly necessary because the number of bits
should already be a multiple of the used container size bytes. Still
it's good practice to be careful with this calculation.
* Improve heap size bytes for encoded product quantization vectors
* Include vector stats for binary quantized vectors
* In volatile chunked vectors, include heap allocated vector
* Include rest of heap allocated structures for mutable map index
* In mutable geo index, the hash map is also heap allocated
* Update tests/manual/test_memory_reporting.py
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Add empty placeholder vector storage types for named vector CRUD
Introduce EmptyDenseVectorStorage and EmptySparseVectorStorage as
placeholder storages for newly created named vectors on immutable
segments. These report all vectors as deleted, consume no disk space,
and are reconstructed from segment config on load via the new
VectorStorageType::Empty and SparseVectorStorageType::Empty variants.
Key design decisions:
- is_on_disk is derived from original user config, not hardcoded
- MultiVectorConfig is preserved for multi-vector support
- Config mismatch optimizer skips Empty storage to avoid false rebuilds
- Quantization delegates normally (handles 0 vectors gracefully)
- get_vector includes debug_assert to catch unexpected access
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] segment-level operations for creating and deleting anmed vectors
* [AI] implement named vector creation and deleting in proxy segment
* [AI] Step 3: Proxy Segment Handling for Named Vector Operations
* [AI] implement for Edge
* [AI] implement consensus operations for named vector operations
* [AI] refactor VectorNameConfig, remove VectorNameConfigInternal
* [AI] handle vector schema inconsistency in raft snapshot recovery
* [AI] rest + grpc API
* [AI] clippy
* [AI] generate openAPI schema
* fmt
* ci fixes
* [AI] fix jwt access test
* [AI] nop operation for awaiting of consensus-commited update ops
* [AI] move vector name operations into points service
* [AI] implement internal api for vector name operations
* [AI] change collection-level config along with segment level operation
* [AI] vector schema reconceliation instead of error
* fmt
* missing compile-time option
* [AI] integration test
* [AI] fix missing JWT tests
* [AI] remove NOP
* [AI] openapi test
* [AI] fix initialization of mutable segment
* [AI] more simple integration tests
* fmt
* [AI] make cluster test a bit harder
* [AI] make test less flacky
* [AI] rabbit comments
* [AI] check params compatibility before writing vector config
* [AI] make sure to register vector storages in structure payload index
* [AI] vector name validation
* lower vector length validation to 200 chars to account for prefix in filename
* [AI] proxy segment: prevent stale data leak through optimization
* fmt
* [AI] filter out removed vectors from proxy response
* [AI] handle vector name in proxy
* fmt
* adjust proxy info based on dropped vectors
* [AI] proxy segment: update filters to correct has_vector condition
* fmt
* clippy
* Fix consensus snapshot applicaiton for vector schema
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix empty sparse vector name validation
* Reorder validation to check empty sparse name before duplicate name
---------
Co-authored-by: leohenon <77656081+lhenon999@users.noreply.github.com>
