* test: add openapi regression test for values_count on missing fields
Reproduces https://github.com/qdrant/qdrant/issues/9586 where points with
a missing payload field are not matched by values_count filters that should
treat the count as 0.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: use module fixture for values_count missing field test
Follow the standard openapi test structure with setup/teardown fixture
instead of explicit drop_collection calls in the test body.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: treat missing field as count 0 for values_count filter (#9607)
`FieldCondition::check_empty()` ignored the `values_count` condition, so a
point with a missing payload field was never matched by a `values_count`
filter. The desired semantics (and the existing `ValuesCount::check_empty`)
treat a missing field as having a value count of 0.
Forward the empty check to `values_count.check_empty()` so bounds like
`lt: 1`, `gte: 0` and `lte: 0` correctly match points whose field is absent.
Fixes#9586
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add routing token structure
* Implement routing token in read operation executor as per design doc
* Add TODO to glue routing token to user requests
* Implement routing header for REST API
* Source routing token from request, not from JWT token
* Implement routing token in gRPC API
* Add test
* Review remarks
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Use lower case header name to prevent panic
* Rename header to X-Qdrant-Route-Affinity
* Assert routing consistency in test on all peers
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
The test asserts that creating a partial snapshot between two in-sync peers
returns 304 (empty diff). It only waited for the write peer to become green,
but read the read peer's manifest for the comparison. An async optimization
reshaping the read peer's segments after collection-snapshot recovery makes
its manifest diverge from the write peer's files, producing 200 instead of
304 (assert 200 == 304).
Wait for the read peer to become green as well before comparing manifests,
mirroring the earlier flaky-test fixes (#7358, #7360).
Co-authored-by: Cursor <cursoragent@cursor.com>
The test relied on concurrent background upserts adding *new* matching
points to the destination shard during the streamed transfer, asserting
the destination count was strictly greater than the original snapshot.
Whether new matching points land in the destination before the transfer
completes is timing-dependent (especially under pytest-xdist load), so
the strict `>` assertion is racy and occasionally fails with equal
counts (e.g. `assert 5031 > 5031`).
Relax to `>=` and keep the strict `dest == src` consistency check, which
is the actual invariant being verified.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add block level timeout to snapshot stream writer
* Add tooling to exercise streaming snapshot stalls
- tests/manual/slow_snapshot_download.py: slowly / partially download a
streaming shard snapshot from a URL to exercise sender-side backpressure.
Supports hold / rst / fin / blackhole termination to simulate a stalled,
killed, or offline (network-partitioned) consumer against a remote node.
Stdlib only; read-only against the target.
- tests/consensus_tests/test_streaming_snapshot_receiver_kill.py: throttled
receiver killed mid-flight + a second receiver, to observe whether the
sender releases the SegmentHolder lock and recovers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: reproduce MatchAny(any=[]) strict-mode rejection on integer index
Adds an openapi test that demonstrates the strict-mode bug where
`match: {"any": []}` on an integer-indexed payload field is rejected with:
Bad request: Index required but not found for "<field>" of one of
the following types: [keyword, uuid]
even though the field is indexed as integer. Root cause is that an empty
`any` list deserializes as `AnyVariants::Strings(empty)` (untagged enum;
Strings variant listed first), and strict mode infers a keyword/uuid
index requirement from the variant tag — ignoring that the list is
empty (i.e. a no-op condition).
The test covers:
- Baseline no-op semantics with and without indexes (passes today).
- Reproduction under strict mode for `must`, `must_not`, and a
FormulaQuery FieldCondition (currently failing; will pass once
empty `any`/`except` no longer requires an index).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: treat empty MatchAny/MatchExcept as no-op in strict-mode index check
An empty `match: {"any": []}` (or `{"except": []}`) is a no-op: `any: []`
matches nothing and `except: []` excludes nothing, regardless of the
field's data type. Because an empty list cannot carry type information it
deserializes as the keyword `AnyVariants::Strings(empty)` variant, which
previously caused strict mode to demand a keyword/uuid index and reject
the request with:
Index required but not found for "<field>" of one of the following
types: [keyword, uuid]
even on a field indexed as integer.
Fix:
- `infer_index_from_any_variants` returns no required index for an empty
variant set.
- `Extractor::update_from_condition` skips a condition whose required
index set is empty, so a no-op condition is never reported as needing
an index (this also covers the FormulaQuery condition path).
This makes the openapi reproduction in test_match_any_empty.py pass.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
The initial 20k-point insert was sent as a single HTTP request (no
batch_size), saturating all cores long enough to starve the consensus
thread (cascading leader elections) and to exceed the 2000ms per-shard
update healthcheck deadline, returning a flaky 408 before the test's
actual snapshot-transfer logic even started.
Batch the insert into 1k-point requests, matching the convention used
by every other large initial upsert in the suite.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Strict mode: add `max_disk_usage_percent`
Mirrors `max_resident_memory_percent`: rejects disk-consuming update ops
(upsert, set/overwrite payload, update vectors) when the filesystem hosting
Qdrant storage is filled above the configured percentage. Delete-style ops
remain allowed so callers can free disk.
Disk usage is sampled via `statvfs` and TTL-cached for 5s (same cadence as
the resident-memory reader) so high-RPS request paths don't hammer the
syscall. Reader is keyed by path in `common::disk_usage` and returns `None`
on stat failure — callers (the strict-mode check) treat `None` as "skip",
matching the memory-check behaviour.
Plumbing follows the existing pattern: field on `StrictModeConfig` (+
output/diff/Hash), gRPC proto field `22`, validation 1..=100, REST/proto
conversions, and the hook into `check_strict_mode_toc_batch` alongside the
memory check (both guarded by `any_consumes_memory`).
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix CI: Windows disk_usage test + e2e WAL config
- `missing_path_returns_none` panicked on Windows because
`GetDiskFreeSpaceEx` succeeds for non-existent paths (it resolves up to
the containing drive). Relax the assertion to "must not panic; if a
value is returned it must be well-formed". The contract we care about
(None on failure) is platform-defined, not something we can portably
force.
- e2e test failed at batch 0 with "WAL buffer size exceeds available disk
space": Qdrant's existing per-shard `DiskUsageWatcher` enforces
`free >= 2 * wal_capacity_mb` and the default WAL didn't fit in the
50 MB tmpfs. Bump tmpfs to 200 MB and shrink `wal_capacity_mb` to 1 MB
(same pattern as `test_low_disk.py`) so our strict-mode gate is the
one that fires, not the WAL pre-check. Raise the gate threshold to
50% to match the larger headroom.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
The test fired a single 20k-point `wait=true` upsert into a fresh
3-shard x 2-replica collection. Under CI load that one op keeps a replica
busy past the hardcoded 2s inter-node health-check (transport_channel_pool
HEALTH_CHECK_TIMEOUT), so the coordinator fails the forward with a transient
"Healthcheck timeout 2000ms exceeded" 408 before resharding even starts.
It's the only resharding test doing a 20k single-shot upsert (others do
~1k), which is why it flakes and they don't. Batch the load at 1000 points
so each op stays well under the health-check window, matching the
proven-stable pattern. Data and transfer behavior are unchanged.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test_resharding_down_abort_converges_when_killed_mid_abort fired the
fire-and-forget abort_transfer request at alive_uri after waiting only for
the *victim* to apply the transfer-start. When alive_uri lagged in replicating
that consensus entry, the abort handler's local check_transfer_exists returned
404 (swallowed by the fire-and-forget thread); the transfer then completed
naturally, resharding was never aborted, and _victim_in_window never opened,
timing out after 30s.
Wait until both alive_uri (the abort target that gates the 404) and the victim
have applied the transfer-start before firing the abort.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [ai] TQDT in the API
* [ai] unify new sparse error for TQDT
* Rename to `turbo4`
* Add `Turbo4` to comments and doc strings.
* Also validate named sparse vector creation
* Abort resharding before we abort transfer
* Test resharding-down abort converges when a peer is killed mid-abort
---------
Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
* test: wait for WAL flock release on peer restart in dirty-shard test
The flake addressed by #9124 (bumping `wait_for_peer_online` to 60s) was
misdiagnosed as CPU contention. Logs show the restarted peer panics within
~1s of startup at `consensus_wal.rs:36` with:
Wal error: Can't init WAL: Kind(WouldBlock)
Panic: Can't open consensus WAL: Kind(WouldBlock)
`wal::Wal::open` calls `fs4::FileExt::try_lock` (non-blocking `flock`) on
the WAL directory fd. After `p.kill()` (SIGKILL + waitpid) the kernel
normally releases the killed peer's flock immediately, but under
pytest-xdist load there is a small window where it lags. The fresh peer's
startup then races and panics. After the panic `/readyz` never returns
200, so neither 30s nor 60s rescues the test.
Fix: add a `wait_for_wal_unlocked` helper that polls both the consensus
WAL directory and the local-shard WAL directory with the same exclusive
non-blocking flock that qdrant uses, and call it after every `p.kill()`
that is followed by a `start_peer` on the same `peer_dir`. The two
60s timeouts are restored to the default 30s now that the underlying
race is gone.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: fail fast if sync restart crashes in dirty-shard test
The sync restart in `test_dirty_shard_survives_update_collection` used
plain `wait_for_peer_online(sync_uri)`, which only polls `/readyz`. If
the freshly started peer panics on startup (e.g. WAL `WouldBlock`), the
test waits the full 30s timeout and then reports a `/readyz` timeout
instead of the actual panic message and exit code.
Switch the sync restart to `wait_for_peer_online_or_crash(...)` (same
helper already used for the dirty restart). On crash it dumps the peer
log tail so the next CI failure shows the real reason instead of a
generic timeout.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: drop speculative WAL flock wait, keep crash-detection switch
Reverts the `wait_for_wal_unlocked` helper that polled the WAL
directories with non-blocking flock. The kernel-side flock race it was
guarding against could not be reproduced in isolation (0/300 iters of
SIGKILL+wait+re-flock on bare Linux), so it was speculative.
Kept:
- Sync restart now uses `wait_for_peer_online_or_crash(...)` instead of
plain `wait_for_peer_online`, so any startup panic surfaces fast with
the actual log tail instead of hiding behind a 30s `/readyz` timeout.
- The two 60s timeouts bumped in #9124 are restored to the default 30s.
If the flake recurs in CI, the new failure output will tell us the
actual cause (panic message + exit code), which is more useful than
papering over it.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* tests: use uniform `from .utils import *` in consensus tests
Eight consensus tests imported specific symbols from `utils` instead of
using `from .utils import *`. The most consequential side effect was
missing the `every_test` autouse fixture, which is responsible for
cleaning up leaked `processes` and resetting the port-slice allocator
between tests. Without it, leaks from one test can carry into the next
on the same xdist worker, causing `processes.pop(target_idx)` to return
the wrong peer in tests like `test_dirty_shard_crash_loop` and triggering
WAL-lock conflicts when the intended target is left running.
Make the imports uniform across the package so the autouse fixture is
always in scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tests: keep `test_issues_api` on explicit imports
This module uses a `@pytest.fixture(scope="module")` cluster setup
shared across all tests in the file. With `from .utils import *` the
function-scoped `every_test` autouse fixture comes into scope and runs
after the module-scoped `setup`, so on the first test it sees the
already-populated `processes` and calls `kill_all_processes()` — killing
the shared cluster before the test body runs. The test then fails with
connection refused on the cluster's port.
Keep explicit imports here so `every_test` is not autoloaded; the
module teardown already cleans up via `kill_all_processes()`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`commit_read_hashring` returns once the leader has applied the entry,
but followers may not have applied it yet. The CommitRead handler calls
`invalidate_clean_local_shards` for `old.nodes()`, which cancels any
ongoing shard clean task. If the test's subsequent `cleanup?wait=true`
request to a follower arrives just before that follower applies
CommitRead, the cleanup task is started and then cancelled mid-flight,
and the endpoint returns HTTP 500 "Failed to clean shard points due to
cancellation, please try again".
Add `wait_for_same_commit` after `commit_read_hashring` so every peer
has applied the entry (and thus already invalidated any clean tasks
that don't exist yet) before we issue cleanup.
Observed flake:
https://github.com/qdrant/qdrant/actions/runs/26259341982/job/77289143792
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(consensus): snapshot transfer with set-payload for missing points
Add a consensus test reproducing a shard transfer abort caused by
set-payload (and other partial-update) operations targeting non-existing
points.
Such operations are written to the WAL before the point-existence check
rejects them, so the queue proxy replays them to the receiver during a
snapshot transfer. The receiver applies them with force=true, bypassing
the missing-point tolerance in handle_failed_replicas, and the operation
hard-fails with `NotFound: No point with id ... found`. Under sustained
load the bounded queue/driver retries are exhausted, the receiver replica
is marked Dead and the transfer is aborted.
The test keeps the missing-point load running while checking the result,
because consensus auto-recovers Dead replicas: stopping the load first
would let the next recovery transfer succeed and mask the bug. It must
FAIL on current code and PASS once the receiver tolerates missing-point
operations during recovery.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(transfer): skip non-transient errors during queue proxy WAL replay (#9126)
During a shard transfer the queue proxy replays operations from the
sender's WAL to the receiver. Some of these are partial updates
(set_payload, update_vectors, ...) that the receiver rejects with a
non-transient error - most commonly `NotFound: No point with id ...`
for a point that does not exist on the receiver, but also any other
client-caused bad request.
These operations were replayed from the WAL, meaning they were already
applied (and rejected the same way) on the sender, so the sender's state
reflects them as no-ops. Propagating the error aborted the whole transfer;
under sustained load the bounded queue/driver retries were exhausted and
the receiver replica was marked Dead.
Handle the error where the semantic context lives - the transmitter
(`transfer_operations_batch`): skip operations the remote rejects with a
non-transient error and keep going, while still propagating transient
errors so the caller retries delivery. Because the batch update API aborts
at the first failing operation, a non-transient batch error falls back to
one-by-one sending to isolate and skip the offending operation(s).
This complements PR #5991, which handles missing points on the live
forwarded-update path (handle_failed_replicas) but not the WAL replay path.
Fixes the abort reproduced by
test_shard_snapshot_transfer_with_missing_point_updates.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Under parallel pytest-xdist load the rejoining peer needs more than the
default 30s budget to replicate the UpdateCollection entry and report
ready, causing intermittent CI failures:
Exception: Timeout waiting for condition peer_is_online to be satisfied
in 30 seconds
Bump both peer-online waits in the dirty shard crash-loop test to 60s,
matching the precedent set by #8963 for similar CPU-contention flakes.
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Recreate optimizers in non-blocking fashion from consensus calls
* Update comments
* On optimizer config update failure, report error status to local shard
* Add a test to confirm we don't block consensus
* Rerun recreation if called multiple times
* Use atomics instead
* Move to the bottom
* Reformat
* Use cluster default shard transfer method for fallback
When a WAL delta automatic transfer fails, the driver falls back to the
method passed via `fallback_method`. This was hard-coded to
`StreamRecords` (unless `prevent_unoptimized` was enabled), which is
inconsistent with the 1.18.0+ default of `Snapshot` and ignores any
configured `default_shard_transfer_method`.
Use `Collection::default_shard_transfer_method()` instead, so the
fallback matches the cluster default. With `prevent_unoptimized` we
still pin to `Snapshot` to preserve deferred point state exactly (raw
segment copy); stream_records would send deferred points but they
would not be deferred on the target.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Avoid wal_delta fallback, update fallback test
If the cluster default transfer method is wal_delta, the same-method
fallback would be refused by the driver. Use snapshot as a safe fallback
in that case; snapshot is also the 1.18.0+ default.
Update test_shard_wal_delta_transfer_fallback to assert the new
snapshot fallback (was stream_records).
Co-authored-by: Cursor <cursoragent@cursor.com>
* Address clippy wildcard_enum_match_arm
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add test to show cleanup may conflict with update queue
* When invoking clean task, first wait for current update queue
* Don't hold shard holder lock for a long time
* Also assert the clean task finished completely
The test_shard_snapshot_transfer_throttled_updates test was flaky because
it checked data consistency immediately after killing background upload
processes, without waiting for in-flight writes to propagate across peers.
All sibling tests (test_shard_snapshot_transfer_fast_burst,
test_shard_stream_transfer_throttled_updates, etc.) already include a
sleep(1) after killing uploaders. This was the only variant missing it.
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Pre-pull the fullstorydev/grpcurl Docker image before running the test
and retry the gRPC health check up to 3 times, so transient Docker
networking hiccups on CI runners no longer cause a hard failure.
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: validate vector dimensions before WAL write for async upserts
When upserting points with wait=false (the default), dimension
mismatches were silently discarded during background processing.
The API returned 200 "acknowledged" but the points were never stored,
causing silent data loss with no error feedback to the user.
This adds an early dimension validation check in do_upsert_points()
that runs before the operation is written to WAL. This ensures that
dimension errors are returned to the client regardless of the wait
parameter, matching the behavior of wait=true.
The validation handles all vector types:
- Dense single vectors
- Multi-dense vectors
- Named vectors (dense, multi-dense, sparse)
- Sparse vectors are skipped (no fixed dimension)
Closes#9039
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor: move vector dimension validation into dedicated module
Extract validate_vector_dimensions and helper functions from update.rs
into src/common/validate_vectors.rs for better code organization.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: update shard update test for early dimension validation
The test expected a shard-level error message, but now dimension
mismatches are caught before reaching the shards. Update the assertion
to accept either the early validation error or the shard-level error.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: assert actual dimension error message in shard update test
Check for the descriptive error ("Vector dimension error: expected dim: 4, got 3")
rather than the generic shard failure wrapper.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add integration test for `match: {except: []}` with integer index
Regression test for https://github.com/qdrant/qdrant/issues/9050
An empty `except` list (NOT IN []) should always match all points that
have the field, both with and without a payload index. Currently the
integer (and keyword) indexed path incorrectly returns zero results
because:
1. serde deserializes `except: []` as `AnyVariants::Strings([])` (first
variant of the untagged enum)
2. The map index filter_impl returns `iter::empty()` for empty
cross-type variant, instead of matching everything
The test covers:
- except: [] without any index (baseline, currently works)
- except: [] with an integer index (currently broken)
- except: [] with a keyword index (currently broken)
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix `match: {except: []}` returning zero results with payload index
Fixes https://github.com/qdrant/qdrant/issues/9050
Root cause: `except: []` deserializes as `AnyVariants::Strings([])`
due to the untagged serde enum trying `Strings` first. On an integer
index, the filter_impl matched `Except + Strings(empty)` and returned
`iter::empty()` (zero results). The same issue existed symmetrically
on keyword indexes with `Integers(empty)` and on UUID indexes with
`Integers(empty)`.
The fix: when the cross-type variant reaches the Except branch (e.g.
Strings on an integer index, Integers on a keyword/UUID index), return
`None` unconditionally — regardless of whether the set is empty. This
delegates to the fallback condition checker, which already handles
`except: []` correctly by matching all values.
The `estimate_cardinality_impl` functions had the same bug (returning
`CardinalityEstimation::exact(0)` for the empty cross-type case) and
are fixed the same way.
Affected index types: integer, keyword (str), UUID.
Bool index is NOT affected — it already returns `None` for all
Any/Except conditions.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Expand match-except test to cover all index types and cross-type filters
Extends the regression test for #9050 to comprehensively cover:
- Integer, keyword, and UUID field indexes
- Empty except list (the original bug) for each index type
- Non-empty except list with matching types (normal filtering)
- Cross-type except values (e.g. strings on integer index, integers on
keyword/UUID index) — type mismatch should exclude nothing → all match
- Exclude-all: listing every value should return zero results
- All scenarios tested both with and without a payload index
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
When the restarted peer's dummy shard is auto-recovered by the
cluster's recovery loop before the test issues its manual
`replicate_shard` call, the manual call returns 400 "already involved
in transfer". Skip the manual call when a transfer is already in
flight — the existing wait_for / transfer-count / replica assertions
still verify the shard recovers.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Raw requests already use QDRANT_HOST, but bypass request_with_validation
and missed QDRANT_HOST_HEADERS. Without these headers, tests cannot run
behind host-based reverse proxies or mocks.
Use unique point IDs across all phases instead of overwriting the same
id repeatedly. The gridstore payload storage size estimate is
bitmask-based: overwrites keep old blocks allocated until a periodic
flush reclaims them, so the test was racing the 5s flush worker. With
unique ids every block stays live and the post-flush size still
reflects all inserted points.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The test waited a fixed 0.5s after each PUT/DELETE before reading from
every peer, which raced with raft apply on followers under CI load.
Replace the fixed sleeps with wait_for-based polling so each per-peer
read retries until the expected value is observed.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Under stress, three concurrent user-requested snapshot transfers (10 000
points each) running while the killed peer recovers can starve the
leader's heartbeats long enough to trigger a raft election. If the
recovery transfer's `RecoveryToPartial` proposal is submitted while no
leader exists, raft drops it silently — `recovered_switch_to_partial`
returns Ok regardless because it only sends to a channel — and the
retry path then waits a full CONSENSUS_CONFIRM_TIMEOUT (10s) before
trying again. Combined with sequential per-shard recovery (auto
transfer limit = 1 × 3 shards), the 30s `/readyz` budget runs out.
Add an optional `wait_for_timeout` to `wait_for_peer_online` and bump
this test's wait to 60s. Default behaviour for other callers is
unchanged.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
