test_recover_from_snapshot_2 and test_upload_snapshot_2 start snapshot
recovery on a freshly joined peer as soon as it lists the collection. The
collection appears once the creation entry is applied, while the peer is
still replaying the rest of the raft log, including the removal of the
killed peer. Recovery then decides which other replicas to remove or mark
dead from that stale local view and drops a healthy replica, leaving a
shard with a single replica.
Add a helper that waits until all peers share the same commit index and
have no pending operations, and use it in both tests before recovering.
Also fix a misleading comment in the recovery replica cleanup branch.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Reject snapshot upload without collection config before loading it
The raw IO error from `CollectionConfigInternal::load` embedded the
server-side temporary path in the API response. Check for the file first
and return a fixed bad-input error instead.
Part of #10553
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A3o9eWSZNMa6WAs5F2HMZC
* Fix missing-config check to run before restore_snapshot loads config
The path-leak guard lived after Collection::restore_snapshot, but that
function already calls CollectionConfigInternal::load and surfaced the
temp path as a 500. Require a regular config.json file before loading,
and cover a directory-shaped config entry in the openapi test.
* Use a valid empty TAR in the missing-config snapshot upload test
Avoid depending on malformed-archive handling; exercise the missing
collection-config path with a real TAR that has no entries.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
Thread a `Populate` through the disk-resident id tracker's open paths
(`DiskMappingReader`, `DiskIdTracker::open`, `ReadOnlyDiskIdTracker`,
`ReadOnlyIdTrackerEnum`) so a `cached` placement primes the page cache with the
mapping files on load instead of leaving them to page in on demand. The
populate is derived from the segment config's placement at load time, clamped
by low-memory mode, in both the writable segment open and the read-only one.
The update-only lookup path keeps its transfer-nothing policy, and the
build-time open stays cold: the built segment is reloaded anyway.
`cached` is no longer rejected by validation.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Add `id_tracker: { memory: cold | pinned }` to CollectionParams,
CollectionParamsDiff and CreateCollection (REST + gRPC `IdTrackerParams`),
mirroring `payload: { memory }`. `cold` builds the disk-resident id tracker,
`pinned` the in-RAM immutable one. Unset keeps the current behavior: the
`serverless_compatible` feature flag decides.
The requested placement is persisted as an optional `id_tracker_memory` on
SegmentConfig (skipped when unset, so existing configs are unchanged); the
segment builder resolves it through `SegmentConfig::id_tracker_memory_placement`
instead of reading the feature flag directly.
The config mismatch optimizer rebuilds non-appendable segments whose effective
placement differs from the requested one. Appendable segments are skipped: they
always use the mutable tracker and get the current config when indexed.
`cached` is rejected by validation: the disk mapping reader has no
populate-on-open path.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Fix flaky shard snapshot API CI readiness race
Run the prebuilt binary and poll /readyz instead of cargo run + fixed sleep, which can miss startup when cargo recompiles.
* Move shard snapshot API CI runner into a dedicated script
Keep workflow YAML thin by starting Qdrant, waiting for /readyz, and invoking shard-snapshot-api.sh from tests/shard-snapshot-api-tests.sh.
* docs(schema): declare enforced 1..=65536 bound on VectorParams.size
The REST layer enforces an upper bound of 65536 on VectorParams.size
via a custom validator (validate_nonzerou64_range_min_1_max_65536),
but custom validators contribute no bounds to the generated JSON
schema - so the published OpenAPI document only declared minimum: 1,
while DenseVectorConfig.size already documented both bounds.
Add an explicit #[schemars(range(min = 1, max = 65536))] attribute so
clients validating requests against the schema see the same contract
the server enforces, update docs/redoc/master/openapi.json
accordingly, and pin the bound with a unit test asserting the
generated schema.
Fixes#9942
* test(openapi): accept documented size bound as a rejection path
test_vector_dimension_limit asserted that an oversized VectorParams.size
reaches the server and returns the exact runtime 422 message. Now that the
enforced 1..=65536 bound is documented in the served OpenAPI schema (#9942),
request_with_validation rejects such payloads client-side before sending.
Accept either layer: a client-side jsonschema.ValidationError or the
server-side validation error.
* test(openapi): handle both rejection layers in dimension limit
pytest.raises only covered the client-side jsonschema rejection; if the
request reached the server instead, the test would fail on an unhandled
response. Use try/except around request_with_validation and assert the
server-side status and exact error message in the else branch.
* test(openapi): assert exact HTTP 422 on server-side rejection
A broad not-ok check would pass on any error status carrying the same
error text; pin the documented contract to 422.
* refactor(tests): address review feedback
Remove the unit test asserting the generated VectorParams schema shape -
it only restates the schemars attribute and adds maintenance cost.
Reduce test_vector_dimension_limit to its actual contract: an oversized
dimension is rejected by the documented OpenAPI schema before the
request is sent.
Mirror the fix from #9938 for test_upload_snapshot: assert every shard
has n_replicas active replicas across local+remote shards instead of
assuming peer 0 always sees exactly 2*n_replicas remote shards.
Follow-up to #10287, which added `max`. Expressing a minimum still
required spelling out `(a + b - |a - b|) / 2`, the sign flip of the max
identity — drop the `neg` and you silently get a maximum instead. It also
only works for two operands and mentions each one twice, so the scorer
walks every sub-tree twice per candidate point.
The pair is what makes clamping expressible:
{"max": [0.0, {"min": [1.0, "$score"]}]}
`min` mirrors `max` throughout, and both guard helpers introduced in
#10287 already took an `operator: &str`, so they are reused unchanged: an
empty operand list is rejected at parse time rather than folding to
+infinity, and the Edge FFI rejects it at construction time. The result
needs no `is_finite` check, since `min` cannot produce a non-finite value
from finite inputs.
The unindexed-field walker shares one arm for `Max | Min` as the bodies
are identical, with a test pinning `min` separately so a later split
cannot silently drop it.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: add a dedicated max operator to score formulas
Expressing a maximum in a score formula required spelling out the
arithmetic identity `(a + b + |a - b|) / 2`. That is easy to get wrong
(the `/ 2` is load-bearing), only works for two operands, and mentions
each operand twice, so the scorer evaluates every sub-tree twice per
candidate point.
`max` is variadic, mirroring `sum` and `mult`:
{"max": ["$score", {"mult": [0.5, "popularity"]}]}
Unlike `sum` and `mult`, `max` has no identity element for the empty
case, so an empty operand list is rejected at parse time rather than
folding to -infinity and scoring every point with a non-finite value.
The check lives in `ExpressionInternal::parse_and_convert`, which every
entry point passes through, and the Edge FFI additionally rejects it at
construction time to match how that crate validates elsewhere.
The result needs no `is_finite` check: unlike `log10`, `exp`, `div`,
`sqrt` and `pow`, `max` cannot produce a non-finite value from finite
inputs, so it follows the existing `sum` convention.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: cover max error propagation and datetime operands
An operand that fails must fail the whole expression rather than being
passed over in favour of a finite sibling. Covered with the failure both
before and after the finite operand: `mult` short-circuits on zero and
so can skip evaluating later operands, and this pins down that `max`
must not grow a similar shortcut that would swallow an error.
Also covers `max` over datetime operands, which reach the scorer through
a separate conversion to seconds, so that "score by whichever timestamp
is newer" is verified rather than assumed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
`test_reinit_removed_peer_readyz_ignores_old_cluster` restarted the
reinitialized peer on a fresh port. A changed `--uri` makes the peer
announce its new address to every address-book entry, including the
injected old first peer, which re-adds it to the *old* cluster as a
learner and starts replicating its log to it. Normally the restarted
peer is a term ahead and ignores those messages, but when the reinit
run's hard state save did not finish before the kill, it restarted at
the old term, accepted the old leader, took its log (commit 13 > 12)
and failed the guard.
The scenario is a plain restart, so keep the URI; then nothing is
announced and only the `/readyz` membership filter is exercised.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Unary inverse hyperbolic cosine, parallel to sqrt/ln/exp/log10, in REST,
gRPC, and edge (FFI + Python) interfaces. Inputs below 1 produce the same
NonFiniteNumber error as an invalid sqrt or ln.
Closes#10186
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* test: cover TurboQuant, turbo4 datatype and keyword prefix in compat data
Extend the storage compatibility fixture with vector and payload index
features that landed since the generator was last updated:
* TurboQuant quantization, one collection per persisted blob layout
(bits1_5 and the default bits4)
* the turbo4 storage datatype, on both dense and multivector storage
* the keyword index `prefix` option, plus a matching prefix scroll in the
query battery
Sparse vector configs reject the turbo4 datatype, so create_collection omits
the sparse datatype for that collection rather than forwarding it. This is a
no-op for every other collection.
Archives are generated once per release and keep the collection set of their
own generation, so expected collections are now resolved per version and the
new ones are only required from v1.19.0 onward. The prefix scroll stays
ungated: archives without the prefix index answer it by scanning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: fail instead of skip when a compatibility archive is missing
A 404 from the compatibility bucket means the archive was never published,
which no amount of retrying fixes. Skipping it reported the version as
covered while nothing ran, so a pull request adding a version could stay
green with its new coverage never executing.
Fail on 404 and keep skipping connection resets and timeouts, so a bucket
outage still does not turn unrelated pull requests red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tolerate not-yet-ready collection upserts in test_rejoin_cluster and give
JWT snapshot uploads more headroom while still bounding auth-rejection hangs.
The crashing peer was picked positionally, so it could be the raft leader.
The staging crash exits inside the apply of the `Dead` entry while raft
messages leave through an async send queue, so a crashing leader takes the
append carrying the new commit index down with it. The other live peer is
then left holding that entry appended but uncommitted and, as one voter out
of three, can never commit it: it never aborts resharding, and the test
times out waiting for its resharding state to clear — the restart that
would restore quorum only comes after that wait.
Pick the victim after the receiver is killed instead: wait for the two live
peers to agree on a live leader, keep that leader as the survivor and crash
the follower. The survivor then commits and applies the abort locally, with
no commit index left to escape a dying process.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Add /profiler/consensus_lag to measure apply lag between peers
Raft commit index advances on a peer whose apply loop is stalled, so the
existing signals - `raft_info.commit` and the `all_nodes_have_same_commit`
test helper - report a stuck peer as healthy. Nothing exposes how long a
peer has been behind at *applying* entries, which is what shard transfer's
`await_consensus_sync` barrier actually waits on.
Each peer now keeps a ring of the last 32 entries it applied, stamped with
its own wall clock and the time that entry took to apply. The ring is in
memory on ConsensusManager, not in Persistent, so the on-disk format is
untouched.
`/profiler/consensus_lag` collects those rings from every peer over a new
internal RPC and lines them up on the entry indices they share. Each entry
is measured from whichever peer applied it first, so a lag is never
negative; the peer that is first can differ per entry, so the baseline is
per entry rather than a single chosen peer. Entries only one peer still
remembers are excluded, otherwise a peer would be measured against itself.
A peer stalled part-way through an entry keeps healthy lag statistics -
everything it did apply, it applied on time - so the report carries
`behind_entries` and `newest_applied_age_ms` alongside, which is what
actually exposes the stall.
Peers that fail or time out are listed rather than failing the request: a
partial answer is more useful than none when the point is to find a peer
that stopped answering.
The endpoint follows `/profiler/slow_requests`: manage access, and outside
OpenAPI, so no endpoint-count or ACTION_ACCESS guard applies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Move applied-entry log into its own module
Keeps the new code out of files that are already large. The ring, its entry
type and the snapshot served over RPC move to
`content_manager/consensus/applied_log.rs`, alongside the other consensus
internals; `ConsensusManager` is left with a field, an accessor and the one
`record` call in the apply loop.
The grpc encoding moves next to the decoding it mirrors, in
`common/consensus_lag.rs`, leaving the internal service handler three lines
instead of thirty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Test that a consensus stall is still in the report after the peer catches up
* Take each peer's applied index from consensus state, not its ring
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
* feat: global quota API
Memory and disk are node-wide resources, so configuring their thresholds
per collection through strict mode makes little sense. Move them behind a
single cluster-wide `QuotaManager`.
The quota config is seeded from `storage.quotas` in the settings (and so
from env vars), overridden by `quota.json` in the storage directory, and
updated cluster-wide through a new `SetQuotaConfig` consensus operation
which rewrites that file on every peer. Raft snapshots carry it too, so a
peer that joins by snapshot picks it up.
Quotas are enforced wherever the strict mode memory and disk checks used
to run, but no longer gated behind `strict_mode.enabled`: a value set in
an enabled strict mode config still wins per resource, the quota is the
default. Rejections name both the condition that tripped and the config
that governs it.
`GET /quotas` reports the config plus current utilization to global read
users; `PUT /quotas` replaces it for global manage users.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: cover the quota endpoints in the API consistency checks
`test_all_rest_endpoints_are_covered` and the OpenAPI endpoint count both
break on any new REST endpoint. Add `GET`/`PUT /quotas` to `ACTION_ACCESS`
with their JWT access tests, and bump the expected API count. The quota
endpoints stay out of `REST_ENDPOINT_WHITELIST`: that list is for
data-plane endpoints reported per-endpoint in metrics.
Also add a Raft snapshot CBOR compatibility test — snapshots are exchanged
between peers of different versions during a rolling upgrade, so
`quota_config` must be absent-tolerant in both directions.
Review feedback: persist through `SaveOnDisk`, which already implements the
write-before-swap protocol this was doing by hand; validate the config at
both persistence boundaries, since a hand-edited quota file or a config
arriving through consensus does not pass the REST handler's validation, and
a `0%` limit would reject every update forever.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: assert seeding a quota from invalid settings persists nothing
Follow-up to review feedback claiming `SaveOnDisk::load_or_init` writes the
init value before it is validated. It does not — only `SaveOnDisk::new`
persists — but the property matters: were seeding to persist first, invalid
settings would leave a `quota.json` that fails validation on every
subsequent start, and the node could only be recovered by deleting it by
hand. Pin it down with a test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: make QuotaManager the single reader of memory and disk
The quota checks measured memory and disk themselves, while the optimizer
and the WAL disk watcher each called `fs4::available_space` behind their
own ad-hoc caches. Fold all of it into QuotaManager: it owns the readings,
the freshness policy, and the limits they are compared against.
Moves the module to `lib/shard`, since the optimizer sits below `storage`
and has to reach it; `storage::quota` re-exports it, so consensus, the
`/quotas` API and StorageConfig are unchanged. The manager is installed as
a process singleton by TableOfContent, ahead of loading any collection.
- Callers hand in QuotaLimits overrides instead of a StrictModeConfig, and
an override can now only tighten. A collection-level admin could raise
`max_disk_usage_percent` past a cluster-wide limit that needed global
manage rights to set; ties resolve to the quota so the rejection names
the knob that actually has to change.
- Measurements are cached for 5s, but a reading at or above its limit is
never reused: a rejected client retries, and freeing the resource has to
take effect on the next request rather than a TTL later.
- `fits_on_disk` sizes an optimization against physical free space only,
never the configured limits. Optimizations are what free a full disk, so
the quota must not be what stops one.
- `percent_of` widens to u128 instead of saturating the multiply, which
under-reported utilization (failing open) above ~184 PB.
- StorageConfig::quotas is optional; absent means no quota is enforced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: don't recover dead replicas onto a node at a resource limit
Recovering a dead replica pulls a whole copy of its shard onto this node.
If it is already at its memory or disk quota that transfer cannot finish,
and starting it only pushes the node further past the limit. Skip it and
reconsider on a later sync, once the resource frees up.
Adds QuotaManager::check_capacity for work that lands bytes here without
being an update. Unlike fits_on_disk the configured limits do apply:
taking on a replica is not what frees a full node, so there is no deadlock
to avoid by letting it through.
The check is hoisted out of the per-shard loop because a node over its
limit re-measures on every call, so checking per dead shard would cost a
statvfs each. It is free when no quota is configured.
Also trims the comments across the quota module, which had grown well past
what the code needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: drop trivial and duplicated quota tests
Six tests removed, ~140 lines, with no loss of coverage:
- a_rejection_names_the_knob_that_has_to_change asserted that a format!
contains its own literals; the message is covered end-to-end by the
override test and by test_global_quota.py.
- a_node_over_its_quota_has_no_capacity_to_take_on_a_replica was 30 lines
for check_capacity, a one-line delegation to check_update the test above
it already calls.
- a_rejecting_measurement_is_never_served_from_the_cache duplicated the
meter test, which proves the same rule with an injected reader instead
of inferring it from the real filesystem.
- free_space_is_reported_without_enforcing_anything covered a one-line
accessor, and its point is what the fits_on_disk test is for.
- The two resolve tests and the three meter tests each collapse into one.
DiskFit::Unknown keeps its coverage as two lines inside the fits_on_disk
test rather than its own fixture. The snapshot compat pair becomes one
test: the second only asserted cluster_metadata.is_empty(), which says
nothing about quotas — the real check was the deserialize, now an expect
that states it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: re-measure free space as the disk fills, and drop a Windows-only assert
Two CI failures, both from this branch.
e2e test_low_disk: the DiskUsageWatcher I replaced escalated to checking
on every call once free space fell below 512 MB. Folding it into the quota
manager lost that — available_bytes passed no limit, so a reading was
reused for the full 5s however little space was left. On a disk filling as
fast as that test fills it, 5s blind is enough to actually run out and the
WAL write dies instead of returning "No space left on device".
available_bytes now takes a watch_below level and never reuses a reading
under it, which is what the old ladder was expressing. The watcher passes
max(min_free, 512 MB), so the escalation point is back; above it the 5s
cache still costs fewer syscalls than the old 128-call ladder. fits_on_disk
gets the same rule by passing required_bytes, so a merge that does not fit
re-checks rather than sitting on a stale sample.
Windows: fits_on_disk on a missing path was asserted to be Unknown, but
GetDiskFreeSpaceEx resolves up to the containing drive and succeeds — as
common::disk_usage's own test documents. Dropped; the branch is a two-line
else and is not portably reachable.
Also renames an_optimization_is_sized_against_the_disk_not_the_quota, which
needed explaining to be understood.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: report the quota config in telemetry
Reads it from the quota manager rather than the settings, so it is the
config the node is actually enforcing: a peer that missed a consensus
update reports what it is applying, not what the cluster agreed on.
Gated on global access, the same access `GET /quotas` requires, and left
out of `PeerTelemetry` — a quota is per-node state, so each peer reports
its own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: regenerate OpenAPI, and cover the quota in the telemetry key sets
Two CI failures from the previous commit.
Referencing QuotaConfig from TelemetryData moves its definition earlier
in `components/schemas`, because TelemetryData is generated ahead of
QuotaStatus. Regenerated rather than hand-patched, so the schema is a
pure move.
test_telemetry_detail asserts the exact set of top-level telemetry keys.
The quota is reported at every level, including 0 — it is three scalars,
it is the default the endpoint serves, and it is what explains an update
being rejected — so both key sets gain it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: remove `max_disk_usage_percent` from strict mode
Disk is a node-wide resource, so a per-collection percentage of it never
meant anything a caller could act on: the limit describes how full the
*node* is, and which collection the write happens to target has nothing
to do with it. The global quota is where it belongs.
It shipped in 1.18.2 without documentation, so this drops it outright
rather than deprecating. Removal is soft in every direction: StrictModeConfig
has no `deny_unknown_fields`, so a client still sending it gets it ignored
rather than a 400, and the same struct deserializes the persisted collection
config, so collections created on 1.18.2+ keep loading. Proto field 22 is
reserved so the number is never reused.
The e2e test becomes a quota test — the fixture and the timing are the
interesting parts and they carry over unchanged; only how the threshold is
configured differs.
`max_resident_memory_percent` was documented and stays for now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: enforce the strict mode memory limit outside the quota
`max_resident_memory_percent` was folded into the quota as an override,
which meant the quota check had to know about strict mode, and retiring
the setting would mean unpicking `EffectiveLimit` and `LimitSource` from
the resolution logic.
It is now a check of its own in `verification/mod.rs`, next to the strict
mode checks it belongs with, borrowing only the measurement from the quota
manager — which stays the node's single reader of process memory, so both
checks still share one reading. Deleting the setting later is deleting one
function and its one caller.
`QuotaManager::check_update` takes no arguments and consults the quota
alone. A collection can still only tighten the limit for itself, because
its own check runs in addition rather than in place of the quota's, and
each rejection now names the config that has to change without having to
carry a `LimitSource` to say so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: enforce the quota on the update path, not in strict mode
The quota check sat inside `check_strict_mode_toc_batch` only because that
was the one place holding the collection's strict mode config. It doesn't
need one any more, and the placement had a real cost: coverage depended on
each handler remembering to ask for a strict mode check, and four of the
internal update RPCs do — `sync_internal`, which moves the most bytes onto
a node, does not.
It now runs in `Collection::update_from_client` and `update_from_peer`,
which every update passes through. `update_from_client` checks ahead of the
shard split, so an operation is accepted or refused whole rather than
landing on some shards and being refused by others.
Classification moves with it, from ~10 `consumes_memory` impls on request
DTOs to one exhaustive `CollectionUpdateOperations::consumes_quota`. The
internal enum has variants — raw upserts, conditional upserts, the syncs —
that have no client-facing request type, so per-DTO impls structurally
could not classify them.
Shard-transfer syncs stay excluded, as they are today: a transfer is sized
up once before it starts, and refusing its batches partway abandons work
that is nearly done only for it to restart from the beginning.
Index and named-vector creation reach shards through consensus, past this
check — a peer must not refuse what the cluster agreed to — so they keep
their pre-consensus check, now against the quota manager directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: deprecate `max_resident_memory_percent` in strict mode
Same reason the disk threshold went: memory is node-wide, so a
per-collection percentage of it caps how full the *node* is, which has
nothing to do with which collection is being written to. The node-wide
quota caps it once for everything.
Unlike the disk threshold this one shipped documented, in 1.18.0, so it
keeps working — as a limit a collection can tighten for itself, never lift
— and gets the usual markers: `#[deprecated]` on both Rust structs,
`[deprecated = true]` on proto field 21, and `deprecated: true` in the
OpenAPI schema, which schemars derives from the attribute.
The note names 1.21 as the removal. Recording a version matters here: the
audit in docs/plans/overdue-deprecations.md found that this repo has never
written a removal deadline down, and members of the 1.15.0 deprecation
batch are still in tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: reconcile the quota readers with #9891#9891 landed effective (cgroup) figures in telemetry while this branch was
making QuotaManager the single reader of memory and disk. Two collisions,
neither of which git sees.
`segment::utils::mem::total_memory_bytes` is now a shared accessor with a
5s TTL, so a cgroup resize is picked up. The quota module had its own
`OnceLock` copy that froze the value at startup — exactly what #9891 set
out to fix — so it delegates to the shared one instead.
Telemetry's new `disk_size` called `common::disk_usage::disk_usage`
directly. That reader lost its TTL cache on this branch when the caching
moved into the quota manager's meter, so it would have taken an uncached
`statvfs` on every telemetry request, and it put a second disk reader back
in the tree. It goes through `QuotaManager::disk_capacity_bytes` now,
sharing the reading the quota check already takes. Verified it still
reports the storage filesystem, matching `df`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: split the quota manager by what each half does
`manager.rs` had grown to 450 lines holding three separate jobs: owning
the config and its file, taking the readings, and comparing one against
the other.
- `manager/store.rs` — the `Store` enum, `QUOTA_CONFIG_FILE`, and config
validation, which is now the store's own business rather than something
every caller has to remember to do first.
- `manager/measure.rs` — every reading, and `DiskFit`. The "nothing else
calls `statvfs` or reads process RSS" claim is now checkable by looking
at one file.
- `manager/enforce.rs` — `check_update` / `check_capacity` and the
threshold comparison.
- `manager/mod.rs` — the struct, its construction, and the config
accessors: what a reader needs to see first.
Tests move with their subject. No behaviour change: `set_config` used to
validate before delegating to the store, and now the store validates on
write, which is the same order of operations from the outside.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: count copy-on-write deletes toward the quota
Dropping a vector or a payload key does not free anything on its own:
copy-on-write rewrites the point to produce the version without that
field, so storage grows first and is only reclaimed once the optimizer
gets to it. Gating those as if they were reclaiming space let a full node
keep taking writes that make it fuller.
Deleting whole points stays exempt. That is the one operation that has to
work on a node at its limit, or there is no way back under it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: say that the quota's reported usage is per node
`GET /quotas` returns one cluster-wide config and one set of utilization
figures, which reads as though both describe the cluster. They do not:
memory and disk are node-local, so `usage` is whatever the peer that
served the request is seeing, and a peer under its limit says nothing
about the others.
Also corrects `resident_memory_percent`, which claimed to be a share of
total system memory. It is a share of the memory available to the
process, which under a cgroup is the limit rather than the host's RAM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: treat a node over its quota as a failed replica, not a bad request
A quota rejection described the request as invalid (400) and was classified
non-transient, which is how the replica set recognises errors that every
replica would produce alike. A quota is the opposite: the input is fine and
the answer depends on which machine you ask. On the default `wait=false`
path that combination silently dropped the write — `update.rs` only
deactivates transient failures when nothing completed — leaving the replica
Active and permanently missing data its co-replicas had.
It is now `InsufficientStorage`, transient, HTTP 507 / gRPC
`ResourceExhausted`. So a node that is out of room is handled like one that
is offline:
- last active replica, or every replica over quota: nothing could take the
write, and the client is told the cluster is out of room.
- more than one replica: the full node is deactivated through the same path
a dead peer takes, and the update stands if enough replicas accepted it.
`check_capacity` already keeps recovery off that node until it has room.
The check also moves off `update_from_client`, which applied the
coordinator's own limit to the whole operation even when it held no replica
of the shards being written. Each replica set now gates its own local write
and records the refusal as a failure of this peer, so a node only ever
answers for itself.
`ResourceExhausted` is shared with rate limiting, and the reverse conversion
mapped it straight to `RateLimitExceeded` — a forwarded rejection came back
as 429. Statuses now carry a marker so the two stay distinguishable across
the wire.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: report quota pressure per node, and across the cluster
A quota is node-local, so finding out which node has hit one meant asking
each of them in turn — and nothing at all showed up in monitoring.
`/metrics` gains a `quota_exceeded` gauge for the local node. It is emitted
only while the quota is enabled: with it off the value would be a constant
0 that says nothing about the node, and an alert built on it would go quiet
rather than fire if someone disabled the quota. Telemetry's `quota` field
carries the same verdict alongside the config, since that is where the
metric is derived from.
`GET /quotas` now answers for the whole cluster. A new `GetQuotaUsage` RPC
on the internal `QdrantInternal` service returns what one peer is using,
and the handler fans it out to every known peer in parallel. Peers that do
not answer are left out rather than failing the request — the nodes that
are out of room are exactly the ones most likely to time out, and a partial
answer still names them. Outside distributed mode the field is absent
rather than a map of one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: report the quota metric per resource
`quota_exceeded` was one flag for the whole node, which does not say what
to go and fix — disk is freed by deleting or optimizing, memory by
unloading. It now carries a `resource` label:
quota_exceeded{resource="memory"} 0
quota_exceeded{resource="disk"} 1
A resource with no limit gets no series at all, for the same reason the
metric is absent while the quota is disabled: a series that can never reach
1 reads as healthy and would quietly carry an alert that cannot fire.
`QuotaManager::exceeded` returns the per-resource verdict, with `None` for
a resource this node does not cap. Telemetry reports the same breakdown,
since the metric is derived from it. The peer usage RPC keeps a single
flag — it sits next to both percentages, so it only has to answer "is this
peer refusing writes".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: drop a no-op error conversion the linter caught
`check_global_access` already returns a `StorageError`, so mapping it
through `StorageError::from` converted the type to itself and tripped
`clippy::useless_conversion`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: hold a tripped quota until usage clears a release margin
A resource resting on its limit crosses it in both directions on the noise
between two readings, and each crossing is expensive: the node refuses a
write, its replica is deactivated, usage dips, recovery starts sending a
whole shard copy back, and the arriving data pushes it over again. The loop
sustains itself, and every lap costs a shard transfer.
A limit now trips at its configured value but only clears once usage has
fallen 5 percentage points below it, so the crossing has to be real. The
margin is floored at 1%, since a limit smaller than the margin would
otherwise be impossible to fall back under and would strand the node.
The verdict is carried on the manager rather than recomputed, which makes
it the thing reporting shows: expect `exceeded` to be set while the
utilization next to it is already back under the limit. Rejections say so
too, rather than claiming a limit that is no longer exceeded:
Disk usage is at 87% of total capacity. It reached the configured limit
of 90% and has to fall below 85% before this node takes writes again.
Changing the config clears the verdicts. New limits are a deliberate act,
and should not be held back by the margin of a limit that no longer exists.
Both resources are now evaluated on every check instead of stopping at the
first failure, so a verdict is never left behind reporting a reading that
has since been superseded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: make the quota release margin configurable
5 points is a guess about how noisy a deployment's usage is, which is not
something one number can be right about: a node whose disk moves in
gigabyte steps needs a wider margin than one that creeps, and an operator
who wants the old flip-on-every-reading behaviour should be able to ask
for it.
`release_margin_percent` joins the rest of the quota config, so it seeds
from `QDRANT__STORAGE__QUOTAS__RELEASE_MARGIN_PERCENT`, replicates through
consensus, and changes with `PUT /quotas`. Defaults to 5 and is filled in
when a request omits it, so it always answers with the margin actually in
force rather than leaving the caller to assume one. `0` releases as soon as
usage is back under the limit.
`QuotaConfig` grows a hand-written `Default` for it, since deriving one
would have quietly defaulted the margin to 0 and disabled the hysteresis
for anyone constructing a config in code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: leave the release margin unset by default, and hold verdicts in atomics
`release_margin_percent` is `null` unless someone sets it, rather than
materialising 5 into every config. A quota written today then does not pin a
number a later release may want to revise, and `{"enabled": false}` still
round-trips as itself. `QuotaConfig::limits` resolves it, next to `enabled`,
so enforcement never sees the unset case.
The verdicts move from a `Mutex<QuotaExceeded>` to one `AtomicBool` per
resource. They are judged independently and nothing reads them as a pair, so
the lock only added contention to the path every update takes; a verdict
that races a concurrent check is re-decided by the next one from a fresh
reading.
That also drops the tri-state. Only "was this over its limit" has to survive
between checks — whether a resource is enforced at all follows from the
config and the reading, so it is derived when reporting rather than stored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: drop a comment arguing with a design that was never here
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Remove deprecated search/recommend/discover endpoints from OpenAPI
Remove deprecated REST API endpoint definitions from the OpenAPI
generator. These endpoints were deprecated in v1.13.3 (`f4ced2567`,
#5907, 2025-01-30) in favor of the universal `/points/query` endpoint:
- POST /points/search
- POST /points/search/batch
- POST /points/search/groups
- POST /points/recommend
- POST /points/recommend/batch
- POST /points/recommend/groups
- POST /points/discover
- POST /points/discover/batch
Also removes the corresponding request types from the schema generator
and updates the expected API count in the consistency check.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Migrate OpenAPI integration tests to /points/query
The deprecated /points/search, /points/recommend and /points/discover
endpoints (along with their /batch and /groups variants) were removed
from the OpenAPI spec, which caused validation failures in the Python
integration test harness.
This commit migrates the affected tests to the universal /points/query
endpoint:
- Delete tests dedicated to the deprecated endpoints:
test_recommend.py, test_discover.py, test_multicollection_reco.py,
test_recommendation_multivector.py
- Refactor remaining tests to call /points/query (and /query/batch,
/query/groups), translating request bodies (vector -> query / using,
positive/negative -> query.recommend, target/context -> query.discover)
and unwrapping the new result.points response shape.
- Drop equivalence assertions against the now-removed legacy endpoints.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Relax non-empty assertions in migrated recommend/discover tests
The previous migration added `len(...) > 0` assertions to tests that
previously only checked equivalence between the deprecated and new
API. These assertions are too strict because the parametrized
`query_filter` cases legitimately produce empty result sets.
Drop the `> 0` assertion and rely on `request_with_validation` to
verify the response is well-formed and HTTP OK.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Migrate remaining OpenAPI tests off deprecated search endpoints
Tests added to dev after the original migration was written still call
/points/search and /points/recommend/groups through
`request_with_validation`, which resolves the endpoint against the
OpenAPI spec and therefore breaks once the endpoint is not in the spec:
- test_turbo4_storage.py, test_sparse_idf_corpus.py, test_validation.py:
translate /points/search to /points/query (vector{name,vector} ->
query + using, result -> result.points).
- test_group.py: drop the /points/recommend/groups half of the
lookup_from validation test in favour of the query equivalent.
test_sparse_idf_corpus.py's test_query_api_supports_idf_corpus goes
away: with the helper on /points/query every test in the file now
exercises what it asserted.
Also record why test_recommend_group cannot assert on its groups: it
uses every point in the collection as a recommend example, so all of
them are excluded and the result is legitimately empty.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Regenerate openapi.json without the deprecated search endpoints
Drops the 8 deprecated paths and the request schemas that only they
referenced: Search/Recommend/Discover request (+Batch, +Groups) types
and their exclusive dependencies (NamedVector, NamedSparseVector,
NamedVectorStruct, UsingVector, RecommendExample, ContextExamplePair).
Regenerated output is a strict subset of the previous spec, and every
remaining $ref still resolves.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Deprecate the search/recommend/discover RPCs in gRPC
The REST counterparts have carried `deprecated: true` since v1.13.3 and
are now gone from the OpenAPI spec, while the gRPC RPCs never got any
deprecation annotation at all. Mark all 8 with `option deprecated = true`
so generated clients warn, and point each doc comment at its `Query`
replacement.
tonic puts `#[deprecated]` on the generated client methods only; the
server trait gets the doc comment alone, so our own `impl` is unaffected.
The RPCs keep serving traffic — this is annotation only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Restore the deleted recommend/discover suites on /points/query
The earlier migration deleted these four files outright, but the
query-side tests it left behind are all shallow smoke tests
(`len(result) > 0`, `"points" in result[0]`). The deleted ones carried
invariants with no query-API equivalent anywhere, so deleting them was a
real loss of coverage rather than de-duplication:
- test_recommend.py: default strategy equals average_vector; batch
results identical to sequential singles across six request shapes;
best_score with only negatives yields all-negative scores; best_score
with a single positive orders identically to a nearest query; raw
vectors as examples equal ids as examples.
- test_discover.py: context-only scores are all <= 0; target-only orders
identically to a nearest query but scores differently; with a fixed
context the integer part of the score is stable while the decimal part
moves, and vice versa with a fixed target; batch equals singles;
lookup_from by id equals by vector.
- test_multicollection_reco.py: cross-collection lookup_from, plus
wrong-vector-size, unknown-collection and unknown-vector rejections.
- test_recommendation_multivector.py: the same recommend invariants over
a max_sim multivector collection, which the query suite never covered.
Only test_recommend_missing_lookup_from_collection_with_raw_vector is
dropped as genuinely redundant — test_query.py's
test_query_missing_lookup_from_collection covers query, query/batch and
prefetch.
Two request-shape differences the translation had to absorb:
- Giving no examples at all is 422 (a RecommendInput validation rule),
where the legacy API reported 400 from the query itself. A malformed
example, such as an empty vector, is still 400.
- DiscoverInput requires the `context` key and accepts only an explicit
null to mean "no context", so target-only discover must spell it out.
The legacy API let it be omitted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Security release addressing CVE-2026-59884, CVE-2026-59885, and CVE-2026-59886.
Equivalent to #9973 for the dev branch.
Co-authored-by: Cursor <cursoragent@cursor.com>
POST /cluster/recover can return 200 while raft silently drops the
snapshot request when no leader is known yet, leaving the test stuck
on a missing collection until timeout.
Co-authored-by: Cursor <cursoragent@cursor.com>
Wait for green on write (and read after recover_read) so collection and
partial snapshots are not taken mid-indexing. Otherwise a leftover
appendable segment survives partial merge and breaks manifest equality.
Co-authored-by: Cursor <cursoragent@cursor.com>
#9013 skipped the manual replicate_shard when a transfer was already
visible, but the recovery loop can still start one between that check
and the POST. Accept 400 "already involved in transfer" as success so
the remaining wait assertions still cover recovery.
Co-authored-by: Cursor <cursoragent@cursor.com>
Right after the no_sync snapshot recovery, the recovered replica serves local
reads immediately, but a remote read to it can transiently fail for a short
window. The read path then falls back to the other replica in hash order on
just the requesting peer, so a single routing token momentarily resolves to
different replicas across peers (observed as {A, B, B}), failing the
determinism assertion.
Wait until token-routed reads are stable across all peers for every token the
test asserts on before measuring, so the transient post-recovery fallback
window is passed. Pure test-side change; routing behaviour is unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: make test_upload_snapshot robust to shard placement balance
The final assertion in recover_from_uploaded_snapshot assumed a perfectly
balanced shard placement (peer 0 having exactly 2*n_replicas remote shards).
Shard placement across peers is not guaranteed to be balanced, so this made
the test flaky (e.g. peer 0 ended up hosting all shards locally, leaving only
3 remote replicas instead of 4).
Instead, verify the full replica layout is healthy: peer 0 observes every
replica through its local + remote shards, so assert that all replicas are
Active and every shard has exactly n_replicas copies across the cluster.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: fetch cluster info once for shard validation
Read local and remote shards from a single /cluster response so both lists
come from the same cluster revision, per review feedback.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
A transfer source that misses the transfer abort (e.g. while partitioned
or paused) keeps its local shard wrapped in a proxy. When such a peer can
only catch up via consensus snapshot, snapshot application re-creates
payload indexes with an update operation that the stale forward proxy
forwards to a transfer target which may no longer have the shard. The
resulting precondition error fails snapshot application and stops the
consensus thread ("No target shard N found for update"), leaving the
peer unable to ever catch up.
Snapshot application now explicitly cleans up transfers that are no
longer registered in consensus: the transfer task is stopped and the
proxy is reverted via the new `ShardReplicaSet::discard_proxy_local`,
which is infallible, never contacts the remote, and forgets queued
updates (replica states in the same snapshot already reflect the
transfer outcome).
The consensus test reproduces the incident: pause the transfer source
mid-transfer, restart the other peers so the aborted transfer can only
be learned via snapshot, and verify the source recovers.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat: slice filtering condition for sliced scroll and deterministic sampling
Add a `slice` filter condition selecting points where
`stable_hash(point_id) % total == index`. The hash is SipHash-2-4 with a
zero key over canonical id bytes (8 LE bytes for numeric ids, 16 RFC 4122
bytes for UUIDs) — a frozen public contract, independent of the internal
resharding ring hash, reproducible by clients to predict membership.
For a fixed `total`, slices are disjoint and cover all points, enabling
parallel scroll streams (ES sliced-scroll style) and reproducible sampling
that composes with any other filter condition.
- REST: `{"slice": {"total": N, "index": R}}`; gRPC: `SliceCondition` in
the condition oneof (tag 8)
- Evaluated per point via id_tracker external-id lookup; no payload index
needed; cardinality estimated as `points / total` with no primary clause
- `total >= 1` enforced by NonZeroU32 at parse time, `index < total` by
validation in both REST and gRPC paths
- Hash contract locked by test vectors independently reproduced with a
reference SipHash-2-4 implementation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* tests: minimal OpenAPI test for slice filter condition
Scrolls all slices of a fixed total over numeric + UUID ids asserting
disjointness and full coverage, checks must_not inversion, and pins the
two rejection paths (422 for index >= total, 400 for total = 0). Requests
and responses are validated against the regenerated OpenAPI spec by the
test harness.
Note: the spec cannot itself reject total = 0 client-side — the Condition
anyOf falls through to the permissive Filter schema, as with any invalid
condition — so rejection is asserted via the server response.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Fix resharding, on queries filter shards on all shard selectors
* Add failing consensus test: search during resharding with shard keys (#9880)
Reproduces a known bug: after resharding is initialized on a custom
sharded collection with a shard key, searches (with and without the
shard key selector) fail with "does not have enough active replicas",
because the new resharding shard is included in reads before it has
an active replica.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Exempt explicit shard id selection from resharding read filter
Explicit shard id selection is only used by internal per-shard
operations (local shard API, internal gRPC reads), including the
resharding driver reading back migrated points from the new shard.
These must reach the resharding shard before it becomes visible to
user-facing selectors, and filtering them also made per-shard reads
return silently empty results on peers lagging on hashring commits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Explicitly set resharding filtering per match branch
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add per-query IDF corpus for sparse vector search
Let the caller choose, per query, which population sparse IDF statistics
are computed over. `params.idf` is either `"global"` (default, unchanged
behavior) or `{"corpus": <filter>}`, where the corpus filter is
independent of - and usually broader than - the retrieval filter.
Decoupling the two keeps the score scale stable when the retrieval
filter tightens: term importance is measured against a population the
user names, not against whatever subset the filter happens to select.
Design decisions:
- Corpus grammar is restricted to a conjunction (`must`) of `match`
conditions on payload fields; loosening later is backward compatible.
- Strict mode validates the corpus filter like a read filter
(unindexed fields rejected).
- `idf` on a vector without the IDF modifier is a validation error,
never silently ignored.
- An empty corpus yields degenerate but corpus-scoped scores (smoothed
IDF over N=0), never a fallback to global statistics - in multi-tenant
collections a fallback would leak term statistics across tenants.
Implementation:
- QueryContext IDF stats are keyed by corpus, so one batch can mix
requests with different corpora.
- Statistics come from the sparse index: df(term) is counted over the
query terms' posting lists only, never by scanning stored vectors.
Small corpora (under ~1/32 of the segment, by cardinality estimate)
are kept as a sorted id list galloping through posting lists via
skip_to; large ones as a dense membership mask filled streaming from
the filtered-points iterator. A misestimated small corpus degrades
into the mask.
- Exposed uniformly: REST (`params.idf`), gRPC (`IdfParams` message),
edge python bindings; OpenAPI schema regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply rustfmt
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix clippy manual_is_multiple_of in sparse IDF corpus test.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Allow any filter as IDF corpus
Drop the must+match grammar restriction on the corpus filter. A
restriction enforced only as a validation step over the full Filter
type buys nothing; if a narrower corpus syntax is ever wanted, it
should be a dedicated API-level type instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix build: add memory field to SparseIndexConfig in idf corpus test
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Reset first_voter and prune address book on first-peer --reinit
A peer removed from consensus and killed after `RemoveNode(self)` was
committed but before it was applied keeps the old cluster's
`first_voter` and peer addresses in `raft_state.json`. First-peer
`--reinit` reset `conf_state` to a single voter (itself) but left both
untouched (`first_voter` is in fact never reset, even when the removal
is applied).
Both values are served to peers bootstrapping onto the reinitialized
cluster. A joining peer seeds its initial `conf_state` with the
advertised `first_voter`, and Raft conf-changes are deltas on top of
that base - so a stale `first_voter` permanently corrupts the joining
peer's voter set: it ends up with {old first peer, itself}, missing the
actual leader. Its `/readyz` then treats the still-alive old peer as a
cluster member and waits for the old cluster's commit index, which its
own consensus never reaches.
This only manifests when the reinitialized leader replicates its log as
plain entries (nothing applied before the kill, so the log anchor is
index 0). If the leader sends a snapshot instead, the snapshot's full
`conf_state` heals the corrupted seed - which is why
test_reinit_removed_peer only failed sporadically on CI.
Fix first-peer `--reinit` to behave like founding a fresh cluster:
reset `first_voter` to this peer (not `None`, or `recover_first_voter`
would re-derive the old first voter from the retained Raft log) and
prune `peer_address_by_id` to this peer only.
The kill-before-apply state is now injected deterministically into
test_reinit_removed_peer, reproducing the exact CI failure against the
unfixed binary. Since `--reinit` now prunes the address book, the
stale-address injection in
test_reinit_removed_peer_readyz_ignores_old_cluster moved to a restart
without `--reinit`, so it keeps exercising the /readyz `conf_state`
membership filter from #9688.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix test_reinit_consensus expecting stale address book after --reinit
The test waited for cluster size 2 right after starting the reinitialized
first peer, before the second peer was even started. That only passed
because first-peer --reinit used to keep the old cluster's addresses in
the address book - the stale state the previous commit removes. A
reinitialized first peer is a fresh single-member cluster; the other
peers re-join and re-register their new addresses right after.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The end-of-test teardown killed the uploader processes and slept for one
second before scrolling all peers for the consistency check. Killing the
client does not cancel an already-sent upsert server-side: on slow CI the
last PUT can take longer than the sleep, so its replication is still
propagating while the peers are scrolled at slightly different times,
making the scrolls diverge by the last batch of points.
Observed in test_shard_wal_delta_transfer_abort_and_retry: peer 0 was
scrolled at 19.089s, received the forwarded batch at 19.179s, while
peer 1 (which had already applied it locally) was scrolled at 19.267s.
Use stop_update_process() (introduced in #8713 for the pre-peer-kill
case) so the uploader exits between requests. Since uploads use
wait=true, once the last PUT returns all active replicas have applied
it, and the scrolls can no longer race.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
TestSnapshotsInterferenceWithConsensus flakes when the initial
create_collection exceeds the 10s client read timeout. Cluster logs
from a failing run show the primary stalling its consensus loop for
~5s while creating local shards on a loaded runner, which triggered
a leader election that delayed full shard activation to ~10.3s.
Setup operations now get a 30s client timeout. The regression check
for #7489 is unaffected: it relies on the server-side operation
timeout passed to delete_collection, not the client read timeout.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Wait for the collection metadata to propagate to the newly added extra
peer before querying its collection cluster info. Being online and
present in consensus does not guarantee the peer has already applied the
collection-creation Raft entry locally, so get_collection_cluster_info
could race and return 404.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(consensus): de-flake replicate_points_stream_transfer_updates override case
With override_points=True the background writer re-upserts points
9990-9999, re-rolling their city payload. Points flipping away from
"London" legitimately drop out of the filtered count on both shards, so
asserting dest_filtered_count >= original snapshot count is not a valid
invariant. On a slow CI runner the sleep(1)+kill() stopped the writer
right after the overrides, before new inserts could compensate, making
a net-negative flip likely (observed: 4954 >= 4959 failure).
Replace the blind kill with a bounded workload (60 points) joined
cleanly before the ~10s transfer of 10k points can finish, assert the
writer exit code, and allow the filtered count to drop by up to the
number of overridden points.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(consensus): guard that writer finishes while transfer is running
The exact count consistency check requires every concurrent write to go
through the transfer proxy. Make that precondition explicit: if the
transfer ever finishes before the writer, fail with a clear message
instead of a confusing count mismatch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(consensus): make replicate_points update consistency checks exact
Assign the city payload deterministically by point ID parity so filter
membership can never change under concurrent overwrites. All assertions
become exact ID-set comparisons with no slack: random city re-rolls made
count-based checks unsound, since the forward proxy filters forwarded
updates by post-update state and a point flipping out of the filter
legitimately goes stale or missing on the destination.
Replace the background writer process (sleep/kill/join choreography)
with synchronous wait=true upserts issued while the transfer streams the
initial points. Leave low point IDs unoccupied and insert into them
during the transfer: the stream cursor passes them immediately, so these
points can only reach the destination through live update forwarding,
which the previous layout (writes at the stream tail) never verified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add unified `memory: cold|cached|pinned` placement parameter for collection components
Introduce a single `memory` parameter that controls how each collection
component's data is held in RAM, replacing the inconsistent zoo of
`on_disk` / `always_ram` / `on_disk_payload` flags:
- `cold`: not pre-loaded from disk, cached with usage
- `cached`: pre-populated into page cache on load, evictable under pressure
- `pinned`: materialized on heap, never evicted by cache pressure
The parameter is available on dense vectors, HNSW config, all quantization
configs, the sparse index, all payload field index types, and payload
storage (as a new `payload: { memory }` sub-object on collection params).
When set, it overrides the deprecated legacy flag; when unset, behavior is
unchanged. Legacy flags are marked deprecated (Rust + proto) but keep
working; conflicts are resolved in favor of `memory` with a warning.
New capabilities enabled by the tri-state model:
- HNSW graph links can be pinned (first production caller of the existing
`GraphLinksResidency::Pinned`)
- sparse mmap index, quantized vectors and on-disk payload field indexes
gain a `cached` tier (mmap + populate on open)
`pinned` is rejected by API validation for components without a heap
variant (dense vector storage, payload storage). Low-memory mode degrades
placements at load time via `Memory::clamp_to_low_memory`, matching the
existing `prefer_disk`/`skip_populate` behavior. Effective-placement
comparison in the config-mismatch optimizer avoids spurious rebuilds when
the same placement is expressed through the new parameter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix gpu-gated tests for the new `memory` field
CI clippy runs with --all-features, which compiles the gpu-gated tests
that were missed locally: add the `memory` field to config literals and
allow deprecated placement params, same as in the rest of the tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add OpenAPI tests for memory placement, keep sparse config downgrade-clean
- OpenAPI tests: create/update collections with `memory` on every component,
assert the parameters are echoed in collection info, assert legacy-only
collections expose no new fields, and assert `pinned` is rejected (422)
for dense vector storage and payload storage on both create and update.
- Persist only the explicitly requested `memory` parameter in
`sparse_index_config.json` instead of the legacy-resolved placement, so
configurations using only the deprecated `on_disk` flag keep byte-identical
files that older Qdrant versions load without unknown fields.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Validate collection meta ops at construction, not only in the API layer
The `memory: pinned` rejection for dense vectors and payload storage
lived in `Validate` impls on the internal request types, which only ran
through the REST actix extractor. gRPC validates just the proto message,
so a gRPC client could persist `pinned` where it is not supported and
have it silently treated as `cached`.
Run the derived validation in `CreateCollectionOperation::new` and
`UpdateCollectionOperation::new` instead: the constructors are the
common chokepoint for all API paths, before the operation is proposed
to consensus. This covers every validator on these types, not just the
`memory` checks, and keeps consensus-apply unaffected so mixed-version
clusters never reject already-committed operations.
`UpdateCollectionOperation::new` becomes fallible; `remove_replica` now
uses `new_empty` since it carries no user config. Regression tests drive
the gRPC conversion path and assert `InvalidArgument` for `pinned` on
create and update, with `cold`/`cached` accepted as a control.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Resolve filter-based update operations to point ids before WAL write
Filter/condition-resolving operations (delete-by-filter, conditional
upsert, the *-by-filter payload/vector operations) stored their filter
in the WAL and re-resolved it against live segment state on every
apply. Replay-time state can differ from the original apply-time state
(the optimizer drops deleted points and their version records during
compaction), so WAL replay was not a deterministic function of the log
and could resurrect filter-deleted points.
Resolve such operations into concrete point ids at submit time, under a
fence that guarantees the resolution sees exactly the operations that
precede it in WAL order. The WAL now only ever contains id-based
operations (pre-existing variants only — no format change), so replay
applies the exact same point set as the original run.
Fixes#9575
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DfVDMFQcy9Ww791x8sHobW
* Fix rustfmt
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DfVDMFQcy9Ww791x8sHobW
* Drop coordinator-side resolution: every replica resolves locally
Replicas holding the same data resolve the same filter to the same point
set, and replicas that already diverged would not become consistent by
agreeing on a filter's resolution. Forward the original filter operation
as usual and let each replica's submit fallback resolve it under its own
fence — one uniform path regardless of where the update lands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DfVDMFQcy9Ww791x8sHobW
* Guard against is_filter_resolving / resolve_operation drift
A resolved operation must never still classify as filter-resolving,
otherwise a filter-carrying record could reach the WAL again (#9575).
Catch one direction of drift between the gate and the rewriter with a
debug assertion right after resolution.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Dedup points-vs-filter precedence into resolve_points_or_filter
The "explicit id list wins over the filter" rule was written twice on
the resolver side (DeletePayload arm and resolve_set_payload); a future
tweak landing in one copy only would make SetPayload and DeletePayload
silently diverge in what gets persisted to the WAL.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Assert rewritten WAL record reuses the incoming clock tag
The single-record-reuses-the-tag property is what WAL-delta recovery
and replica dedup rely on, but no test asserted it: submit the
delete-by-filter with a real clock tag and check the resolved
DeletePoints record carries it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Test replay of old-style filter records left in the WAL
Upgraded nodes can still hold WALs with unresolved filter operations;
the by-filter apply paths are kept so they replay one final time with
the old semantics. No test covered that path (the new submit flow can
no longer produce such WALs), so append a raw DeletePointsByFilter
record at the WAL layer, reload, and assert the matched points are
gone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add consensus test for per-replica filter-op resolution
Exercises the replicated path for filter/condition-resolving updates:
the coordinator forwards the original filter op and each replica
resolves it locally (delete-by-filter, insert-only and update-filter
conditional upserts, set-payload-by-filter, including per-shard empty
resolutions on a 2-shard collection). Asserts both replicas hold
identical state (reads prefer the local replica), then restarts the
whole cluster and asserts each replica replays its id-based WAL to the
same state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Applying `RemoveNode(self)` prunes all other peers from the removed
peer's persisted address book, but the process may be stopped after the
removal is committed and before the entry is applied. The old first
peer's address then survives `--reinit`, and the readiness checker -
which treated every `peer_address_by_id` entry as a cluster member -
would wait for the reinitialized peer to reach the *old* cluster's
commit index: a foreign consensus it can never catch up with, so
`/readyz` never passed.
Filter the address book by current `conf_state` membership instead,
falling back to all known addresses while `conf_state` is still empty
(a bootstrapping node that has not applied any configuration change
yet). After `--reinit` the `conf_state` is reset to a single voter, so
the readiness check correctly ignores peers of the old cluster.
Fixes flaky `test_reinit_removed_peer`, which hit this race when the
removed peer was killed before applying its own `RemoveNode`. The new
regression test simulates that state and fails with the exact CI error
without the fix.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Roman Titov