* test(model_testing): add slice matcher to generated scroll filter
Extend ScrollFilter with a Slice variant so paginated scroll exercises
Condition::Slice. The generator draws small totals (1/2/3/4/5/8) and a
valid index; the model verifier mirrors membership via Slice::check —
the same hash contract the engine uses — so the existing paged-scroll
id-set assertion covers sliced scroll under soak (optimizer, WAL reload,
multi-shard, mixed UUID/numeric ids).
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(model_testing): add CountBySlice verification op
Exercise Condition::Slice through the exact count API under soak.
Shares the slice generator with ScrollPaged; the model oracle uses
Slice::check so engine and in-memory counts must agree.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(model_testing): compose slice with num on scroll and delete-by-filter
- ScrollFilter::NumAndSlice: indexed num drives candidates; slice is a
per-candidate check via Filter::merge.
- DeleteByFilter { num, slice: Option<Slice> }: half the deletes also
restrict by slice so submit-time filter resolution and WAL-replayed
id lists exercise Condition::Slice.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: slice filtering condition for sliced scroll and deterministic sampling
Add a `slice` filter condition selecting points where
`stable_hash(point_id) % total == index`. The hash is SipHash-2-4 with a
zero key over canonical id bytes (8 LE bytes for numeric ids, 16 RFC 4122
bytes for UUIDs) — a frozen public contract, independent of the internal
resharding ring hash, reproducible by clients to predict membership.
For a fixed `total`, slices are disjoint and cover all points, enabling
parallel scroll streams (ES sliced-scroll style) and reproducible sampling
that composes with any other filter condition.
- REST: `{"slice": {"total": N, "index": R}}`; gRPC: `SliceCondition` in
the condition oneof (tag 8)
- Evaluated per point via id_tracker external-id lookup; no payload index
needed; cardinality estimated as `points / total` with no primary clause
- `total >= 1` enforced by NonZeroU32 at parse time, `index < total` by
validation in both REST and gRPC paths
- Hash contract locked by test vectors independently reproduced with a
reference SipHash-2-4 implementation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* tests: minimal OpenAPI test for slice filter condition
Scrolls all slices of a fixed total over numeric + UUID ids asserting
disjointness and full coverage, checks must_not inversion, and pins the
two rejection paths (422 for index >= total, 400 for total = 0). Requests
and responses are validated against the regenerated OpenAPI spec by the
test harness.
Note: the spec cannot itself reject total = 0 client-side — the Condition
anyOf falls through to the permissive Filter schema, as with any invalid
condition — so rejection is asserted via the server response.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Benches: use SmallRng instead of ChaCha12-based generators
All benchmarks used StdRng or rand::rng() (ThreadRng), both backed by the
ChaCha12 block cipher in rand 0.10. Benchmarks do not need crypto-strength
randomness, and several draw random values inside the timed closure, so
cipher work was included in the measurement itself.
Switch every bench target to SmallRng (Xoshiro256++), and key the HNSW
graph cache and sparse index cache by RNG algorithm so stale caches built
from the old generator are not reused against newly generated vectors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Benches: replace free-function rand::random with local SmallRng
Addresses review: rand::random draws from the thread RNG (ChaCha12),
including inside the timed loop of the pq score benchmark.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Model tester: cover all quantization types
Add a `quantization` field to `VectorCandidate` so candidates can carry
any `QuantizationConfig` variant, and materialize the configs in the
fixture (`quantization_config`). The inline-storage vector "i" keeps its
scalar Int8 config, now declared on the candidate instead of hard-coded
in the fixture.
New candidates:
- "p" Dense(8) + Product x4
- "v" Dense(6) + Binary (non-byte-aligned dim, trailing-bit padding)
- "r" Dense(8) + Turbo (search-side TQ over Float32 storage)
Quantization x datatype combos:
- "l" Dense(6) Float16 + Binary
- "d" Dense(8) Turbo4 + Turbo default bits (keep-source-rotated branch
of `should_keep_source_rotated`)
- "g" Dense(8) Turbo4 + Turbo Bits1_5 (Padded rotation, rotate-back
branch)
This is model-safe: schema quantization keeps the original vectors, so
read-back predictions are untouched; the approximate quantized scoring
only feeds the membership-only Search/Query/Recommend checks.
`assert_candidates_predictable` enforces the wiring constraints:
quantization is dense-only (the fixture only wires the dense arm) and
requires `initially_active` (CreateVectorName's `DenseVectorConfig`
carries no quantization).
Verified: seeds 1/2/3/7/42 soaks (5k ops, restarts, optimizer on) green;
quantized codes confirmed on disk for every quantized candidate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Model tester: make inline_storage a VectorCandidate knob, enable on "l" and "d"
Replaces the fixture's name-based INLINE_STORAGE_VECTOR special case with an
inline_storage field on VectorCandidate (requires quantization, enforced by the
startup assert). Enables it on "l" (Float16 base + padded Binary links) and "d"
(Turbo4 base + TQ links) to cover more (base layout, link encoding) pairs of the
CompressedWithVectors format; "v", "r", "g" and "p" keep the non-inline paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Fix resharding, on queries filter shards on all shard selectors
* Add failing consensus test: search during resharding with shard keys (#9880)
Reproduces a known bug: after resharding is initialized on a custom
sharded collection with a shard key, searches (with and without the
shard key selector) fail with "does not have enough active replicas",
because the new resharding shard is included in reads before it has
an active replica.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Exempt explicit shard id selection from resharding read filter
Explicit shard id selection is only used by internal per-shard
operations (local shard API, internal gRPC reads), including the
resharding driver reading back migrated points from the new shard.
These must reach the resharding shard before it becomes visible to
user-facing selectors, and filtering them also made per-shard reads
return silently empty results on peers lagging on hashring commits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Explicitly set resharding filtering per match branch
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The WAL is only truncated past operations whose segment flush was
confirmed, so first_index is a durable lower bound on the applied
sequence. The persisted applied_seq can legitimately lag behind it by
more than one save interval: it is saved every 64 update-worker calls
from a counter that restarts at zero on process start, and synchronous
WAL replay never feeds it. A replay target computed from such a stale
applied_seq can then sit before first_index, tripping the debug_assert
from #8454 (flaky model_testing gate, #9844) and, in release builds,
enqueueing already-truncated indices that fail with spurious
"Operation not found in WAL" errors.
Clamp the replay target at first_index: nothing before it ever needs
replay.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Every point id the workload draws now comes from an IdSpace pool
precomputed at startup. A --uuid-id-fraction (default 0.5) of the
--id-pool slots are well-formed v4 UUIDs built from the seeded rng via
uuid::Builder::from_random_bytes, the rest stay numeric. Precomputing
the pool keeps the id-reuse semantics (upserts overwrite live points,
deletes and retrieves hit them) that fresh per-op random UUIDs would
lose, and keeps runs seed-reproducible.
Sampling consumes a single range draw per id, exactly like the previous
NumId draw, so fraction 0 consumes no extra rng draws and reproduces
the numeric-only op stream byte-for-byte. The harness smoke tests run
with fraction 0.5, and the fraction is recorded in the trace header.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Preparation for capping appendable segment growth in the update path
(#9158): a dedicated error for "all appendable segments reached
max_segment_size", so the update pipeline can recognize it and provision a
fresh appendable segment before re-applying the operation.
Maps to a transient service error at the collection level: if it ever
escapes recovery, failed-operation recovery re-applies the operation.
Part 1/5 of the appendable segment overflow fix.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Every vector candidate now carries a distance metric instead of the
hardcoded Dot, threaded into the fixture schema, the CreateVectorName
generator and the read-back prediction.
The model predicts Cosine read-backs exactly by mirroring the engine's
ingestion preprocessing: metric_preprocess follows
NamedVectors::preprocess_dense_vector's per-datatype dispatch and calls
Distance::preprocess_vector itself, so predictions track the engine by
construction (including the identity preprocess of the byte metric,
which stores Uint8 vectors un-normalized). Stored vectors are
preprocessed exactly once (optimizer and CoW moves transfer raw bytes),
so predictions stay exact across moves, including Cosine + Float16.
New candidates: "e" (dense Cosine), "n" (multi-dense Cosine, per-row
normalization), "x" (dense Cosine + Float16), "o" (dense Cosine +
Turbo4, padding-free dim), "j" (dense Euclid), "k" (dense Manhattan).
Euclid/Manhattan preprocess is an identity, so their value is engine
side: Order::SmallBetter comparator coverage.
The startup predictability check now also rejects sparse + non-Dot
(sparse schemas carry no distance) and Turbo4 + Euclid/Manhattan (TQ's
L1/L2 modes store lengths differently from Dot/Cosine and their
copy-on-write re-quantization fixed point is not soak-validated yet).
Soak-validated on seeds 1/2/4/5/6/7/8 (30k ops), including two
restart runs (restart probability 0.002) with the optimizer enabled.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add Float16 and Uint8 storage datatypes to the model tester
Extend VectorCandidate with a datatype override and fold the DenseTurbo
kind into Dense + Some(Turbo4) so storage datatype has a single source
of truth. Two new initially-active candidates exercise half-precision
("h", dense 6) and unsigned-byte ("y", dense 4) storage; "c" carries an
explicit Some(Float32) to cover schema configs that spell the default
datatype out.
The model predicts lossy read-backs through the engine's own
PrimitiveVectorElement impls (as Turbo4 reuses turbo_storage_roundtrip)
and compares them exactly: both round-trips are deterministic and
idempotent, so they stay bit-stable across optimizer moves, WAL replay,
and reloads. Uint8 components are drawn from 0.0..256.0 since the
storage truncates with `x as u8` and unit-range draws would collapse to
zeros.
A compile-time assertion rejects datatype overrides on non-Dense
candidates: the fixture's sparse/multi-dense arms ignore the field and
multi-dense read-backs are compared without a round-trip prediction, so
a lossy multi-dense candidate would soak-panic with a false divergence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Plumb Float16 and Uint8 multi-dense support in the model tester
Multi-dense storage converts the flattened matrix component-wise
(from_float_multivector), so per-row round-trips through the same
PrimitiveVectorElement impls predict read-backs exactly. The fixture's
multi-dense arm now applies the candidate datatype (matching the
CreateVectorName path), model_vector predicts per-row, and two new
initially-active candidates exercise the combination: "w"
(MultiDense(5), Float16) and "z" (MultiDense(3), Uint8).
The compile-time candidate check narrows to the combinations that
remain unpredicted: Turbo4 multi-dense (the multivector quantization
path differs from the per-vector turbo_storage_roundtrip) and sparse
with any datatype override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Make datatype match exhaustive in random_dense_vec
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review findings on datatype plumbing
- Start candidate "c" active so the explicit Float32 schema path runs in
default soaks (CreateVectorName is FORCE_OFF by default)
- Single Float16/Uint8 roundtrip dispatch shared by the dense and
multi-dense arms of model_vector
- Hoist shared fixture builder plumbing into dense_params_builder
- Build one DenseVectorConfig literal in the CreateVectorName generator
- Inline single-caller datatype_of wrapper
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Replace const-eval candidate check with a startup assert
Const eval forbids iterators, forcing an index-based while loop. A plain
function called at the top of run() reads better, still fails before any
op is applied, and names the offending candidate in the panic message.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fold INITIAL_ACTIVE into an initially_active candidate field
The hand-maintained list duplicated ALL_CANDIDATES (11 of 12 names) and
had to be kept in sync when adding candidates; forgetting it was silent
since CreateVectorName is FORCE_OFF by default, so a forgotten name got
zero default-soak coverage. Each candidate now declares its activation
inline and the fixture and run() filter on it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
An empty update channel only means the last operation was received,
not that it finished applying in spawn_blocking. Use plunge_async as
a barrier before asserting point counts.
Fixes#9831
Co-authored-by: Cursor <cursoragent@cursor.com>
Drain the update worker queue after setup upserts with WaitUntil::Wal.
Wal only waits for the WAL write, so on slow CI (notably Windows) B could
time out while queued behind the setup backlog rather than because the
worker was blocked on A's deferred wait.
Fixes#9814
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add per-query IDF corpus for sparse vector search
Let the caller choose, per query, which population sparse IDF statistics
are computed over. `params.idf` is either `"global"` (default, unchanged
behavior) or `{"corpus": <filter>}`, where the corpus filter is
independent of - and usually broader than - the retrieval filter.
Decoupling the two keeps the score scale stable when the retrieval
filter tightens: term importance is measured against a population the
user names, not against whatever subset the filter happens to select.
Design decisions:
- Corpus grammar is restricted to a conjunction (`must`) of `match`
conditions on payload fields; loosening later is backward compatible.
- Strict mode validates the corpus filter like a read filter
(unindexed fields rejected).
- `idf` on a vector without the IDF modifier is a validation error,
never silently ignored.
- An empty corpus yields degenerate but corpus-scoped scores (smoothed
IDF over N=0), never a fallback to global statistics - in multi-tenant
collections a fallback would leak term statistics across tenants.
Implementation:
- QueryContext IDF stats are keyed by corpus, so one batch can mix
requests with different corpora.
- Statistics come from the sparse index: df(term) is counted over the
query terms' posting lists only, never by scanning stored vectors.
Small corpora (under ~1/32 of the segment, by cardinality estimate)
are kept as a sorted id list galloping through posting lists via
skip_to; large ones as a dense membership mask filled streaming from
the filtered-points iterator. A misestimated small corpus degrades
into the mask.
- Exposed uniformly: REST (`params.idf`), gRPC (`IdfParams` message),
edge python bindings; OpenAPI schema regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply rustfmt
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix clippy manual_is_multiple_of in sparse IDF corpus test.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Allow any filter as IDF corpus
Drop the must+match grammar restriction on the corpus filter. A
restriction enforced only as a validation step over the full Filter
type buys nothing; if a narrower corpus syntax is ever wanted, it
should be a dedicated API-level type instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix build: add memory field to SparseIndexConfig in idf corpus test
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
drop_shard_key force-aborted any resharding on the key being dropped, but
swallowed a failed abort_resharding with a log-only error and dropped the
shards anyway. That could leave resharding_state.json referencing the
just-dropped key — a latent inconsistent load-time state.
Propagate the error with `?`. A ServiceError halts consensus and retries after
restart, which is safe because the abort and everything else in drop_shard_key
is replay-tolerant; a user error dismisses the entry before any shard is
dropped, which is equally consistent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A ServiceError returned from apply halts consensus on every peer (the entry can
never be applied), deterministically stalling the whole cluster. The transfer
Start validations that report a missing source or destination shard are
reachable — e.g. a committed Start racing a resharding-abort or shard-key-drop
that removed the shard — and all run before any durable write, so dismissing
the entry with a user error is safe and correct.
Convert five such sites to CollectionError::bad_request:
- validate_transfer: source shard missing, and destination shard missing in
both the resharding and filtered branches (helpers.rs);
- start_shard_transfer: the source and target get_shard lookups
(shard_transfer.rs).
The genuine "single node deployment" service_errors in collection_meta_ops.rs
are left untouched — those are real misconfiguration guards.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Start's first durable write registers the transfer record; its last write sets
the destination replica state. On re-apply after a crash between the two,
check_transfer_conflicts found the operation's own half-applied transfer and
returned bad_request, which is dismissed and never retried — so the peer
permanently lacked the destination replica entry. Worse, a later Finish on
that peer then silently skips both the destination promotion and the source
removal, pinning a replica-set divergence.
Exclude the transfer's own key from the conflict scan so a replay falls through
and re-runs the (idempotent) start: register_start_shard_transfer is a
set-insert and the destination replica-state write is absolute, so re-running
reconciles the partial state instead of dismissing it. A genuinely conflicting
transfer (different key touching the same shard/peers) is still rejected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
start_resharding, finish_resharding and abort_resharding each saved the
updated collection config with a log-only `if let Err(err) = config.save(..)`
that swallowed the failure. A swallowed save let the operation report success
with a stale shard_number persisted on disk, arming a shard-dir/loader panic
(or a silently unloaded shard) at the next restart.
Propagate the error with `config.save(&self.path)?;` instead. The resulting IO
error is a ServiceError, so consensus halts and retries the entry after
restart. That is safe because in all three functions the config save is
value-idempotent (guarded by `shard_number != new_shard_number`) and every
step is replay-tolerant, so the retry converges.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In the up-direction of abort_resharding, ShardHolder::abort_resharding drops
the new shard's directory, but the config.params.shard_number update was
persisted only at the very end of the function — after the transfers abort.
A crash anywhere in that wide window left shard_number pointing at an
already-deleted shard directory, which makes the auto-sharding loader panic
on the missing dir at startup (crash loop), before the consensus replay that
would reconcile the state can run.
Move the shard-count update block to before the shard_holder.abort_resharding
call. The block keeps its value-idempotence guard, so replay converges. The
config write lock is taken while holding the shard_holder write guard, matching
the shard_holder -> config ordering already used in finish_resharding, and is
released before abort_resharding. The reverse crash window (dir still present
but count already decremented) is benign and reconciled by replay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In the down-direction of finish_resharding, the config.params.shard_number
update was persisted *after* drop_and_remove_shard. A crash between the two
left shard_number pointing at an already-deleted shard directory, which makes
the auto-sharding loader panic on the missing dir at startup (crash loop) —
before the consensus replay that would reconcile the state can run.
Move the shard-count update block to before drop_and_remove_shard (still after
remove_shard_from_key_mapping). The block keeps its value-idempotence guard, so
replay converges. The reverse crash window (dir still present but count already
decremented) is benign and reconciled by replay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When Medium/Strong write-ordering forwards an update to the leader and it
fails with a transient error, the failure path proposed deactivating the
leader replica while passing its raw peer state as from_state. Unlike the
sibling deactivation site, this did not filter out transient/resharding
states, so a Resharding/ReshardingScaleDown leader could be proposed
Dead with from_state=Some(Resharding*).
On re-apply after a crash inside abort-resharding (which reverts the gated
replica to Active or removes it), the from_state gate no longer matches the
current state -> bad_input -> the entry is dismissed and never retried, so
the peer keeps Active while the rest of the cluster has Dead.
Filter the leader state with `.filter(|state| !state.is_partial_or_recovery())`,
matching the sibling site (PR #7849), so the proposal omits from_state for
transient/resharding states and converges on replay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(recovery): advance newest clocks for WAL-replay tail on load
`load_from_wal` replays the WAL synchronously up to `to` and queues the
remaining `[to, last_wal_index)` tail to the update worker. Whether that
tail exists depends on how far the persisted `applied_seq` lags the WAL
end, independently of the `prevent_unoptimized` flag's value: the flag only
gates the worker's deferred-points wait, not whether the tail is queued.
The synchronous replay advances `newest_clocks` from each entry's clock tag,
but the queued tail did not: the update worker deserializes only the
operation and discards the clock tag.
As a result the newest-clocks recovery point regressed across a graceful
restart by up to the update-queue size, even though those operations are
durably in the WAL (which is exactly what the recovery point tracks). Worse,
the first post-restart updates would then be assigned clock ticks that
earlier WAL entries already carry for different operations, which corrupts
WAL-delta resolution in a cluster.
Advance `newest_clocks` over the queued tail during load, mirroring the
synchronous replay, before handing it to the worker. The range is
end-exclusive because `last_wal_index` is one past the last entry.
Regression dates to WAL replay honoring `applied_seq` (#8008), which
narrowed synchronous replay from the whole WAL to `[from, to)`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(model): assert clock recovery point survives close+reopen
Capture each shard's newest-clocks recovery point (flattened to
shard_id -> (peer_id, clock_id) -> tick, tokens dropped) on both sides of
every close+reopen in the model testing harness (mid-run restart and final
reload) and assert exact equality.
Both mismatch directions are bugs: a lost tick means clock durability broke,
a gained tick means the reload path over-advanced a clock. The check runs
after the existing model check so a lost WAL tail keeps its established
extra/missing-id postmortem signature, and a clocks-only divergence surfaces
distinctly.
This is what caught the WAL-replay-tail clock regression fixed in the
previous commit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(recovery): tolerate unreadable WAL tail entries in clock advance
Review follow-ups for the deferred-tail clock advance:
- Log and skip a tail entry that fails to read instead of propagating
the error: the update worker tolerates the same failure when it
re-reads the entry, and failing here turns one bad tail record (or
applied_seq/truncation index skew) into a shard, and by default a
node, that cannot start. Add a red-green-verified regression test
that injects an undeserializable record into the deferred tail.
- Fix the pre-existing off-by-one in the send loop: last_wal_index is
one past the last entry, so `to..=last_wal_index` enqueued a phantom
op_num on every restart with a deferred tail.
- Reword new comments (em dashes, and the range-bound note is obsolete
now that both loops use the same exclusive bound; document the
two-pass shape instead: the WAL iterator is not Send across await).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: return error for missing point IDs in Query API recommend
The Query API recommend handler (RecommendAverageVector, RecommendBestScore, RecommendSumScores) used filter_map in resolve_reco_reference, which silently dropped point IDs that could not be resolved. The legacy /points/recommend API correctly returns a PointNotFound error in this case.
Fix: change resolve_reco_reference to return CollectionResult and propagate errors instead of filtering them out, matching the pattern used by Nearest, Discover, Context, and Feedback query types.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
* fix: apply cargo fmt to collection_query.rs resolve_reco_reference
Format resolve_reference() calls as method chains to satisfy rustfmt.
Addresses reviewer feedback from timvisee: CI lint failure.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
* test: add unit tests for resolve_reco_reference error on missing point IDs (Fixes#9390)
- test_missing_point_id: verifies error when ID not in ReferencedVectors
- test_valid_point_id: verifies success when ID is present
- test_mixed_point_ids: verifies error when mix of valid/invalid IDs
Requested by @timvisee in PR review.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
* fix: apply cargo fmt to resolve_reco_reference tests
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
* fix: apply cargo fmt to resolve_reco_reference test imports
Move `use super::*` after external crate imports to satisfy rustfmt.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: read-only edge grouping/matrix search + object-storage read path
Add query_groups (group by a payload field) and search_matrix (single-shard distance matrix over a random sample) to the read-only edge shard's EdgeShardRead API, in new grouping and matrix modules plus edge test helpers.
Make the object-storage read path available outside tests: drop the #[cfg(test)] gate on the BlobFile UniversalReadExt impl and move io_bridge_object_store/object_store to segment's normal dependencies, so a ReadOnlyEdgeShard can serve segments read from S3.
* Share group-by building blocks between server and edge
Move GroupsAggregator, group candidate query shaping (is-empty filter,
group_by payload selector, prefetch limit scaling) and result-order
derivation into shard::grouping / shard::query, so the collection and
edge grouping implementations cannot silently diverge.
Edge grouping now handles multi-valued group keys, u64 keys, wildcard
group_by paths, prefetch limits and score-ordered groups the same way
as the server.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drive group-by through a shared sans-IO state machine
Extract the multi-request collect/fill loop into
shard::grouping::GroupByDriver: next_request() yields shaped backend
queries, add_points() advances the state, distill() returns the groups.
Query execution stays with the caller, so the async server path and the
sync edge path drive the same machine, and the request shaping helpers
become private to shard::grouping.
Edge now uses the same request budget (5 collect + 5 fill requests) and
per-request candidates limit (groups * group_size, computed inside the
driver) as the server, replacing its single 4x-oversampled request.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add coverage for TurboQuant datatype in model tester
Co-authored-by: Cursor <cursoragent@cursor.com>
* Address review feedback: tolerance, doc comment, exhaustive match
- Tighten dense_matches Turbo4 tolerance to 16 ulps relative and drop
the absolute floor, so near-zero sign flips and small systematic
quantization drift fail instead of passing
- Fix ALL_CANDIDATES doc comment to match INITIAL_ACTIVE (six names
start active, "c" and "u" via CreateVectorName)
- Make model_vector match exhaustive so new VectorKind variants force
a compile error
- Use explicit DistanceType::from(distance) in turbo_storage_roundtrip
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The write_limit_type: bool parameter of CollectionError::rate_limit_error
relied on a trailing comment to document its meaning (false = read,
true = write). A swapped literal at a call site would compile and report
the wrong limiter type in the error message.
Introduce RateLimiterKind { Read, Write } so the intent is explicit at
call sites and checked at compile time.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Remove 8 `#[allow(clippy::...)]` attributes that no longer suppress any
lint. Each was verified redundant by rewriting it to `#[expect(...)]` and
confirming the workspace stays clippy-clean under the CI config
(`cargo clippy --workspace --all-targets --all-features -- -D warnings`).
Attribute-only deletions, no behavior change.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Read-only edge-shard followers need to distinguish "essential segment file
missing because the leader removed the segment mid-reload" (re-check the
manifest, absorb) from real corruption (escalate). The classification
already exists at the universal-io layer (UniversalIoError::NotFound,
mmap MissingFile, both carrying the path), but died at the OperationError
boundary where everything collapsed into ServiceError strings.
- Add OperationError::FileNotFound { path } and route the structured
sources into it: UniversalIoError::NotFound, MmapError::MissingFile,
and GridstoreError::UniversalIo(NotFound) (the route payload-storage
live-reload errors take).
- Implement IsNotFound for OperationError so follower code can classify
(and OperationResult::ok_not_found() works where lazily-created files
are legitimately absent).
- Raw io::Error NotFound intentionally stays ServiceError: it has no
structured path; universal-io wraps not-found at the call site via
extract_not_found, so classification belongs at the source.
- CollectionError maps FileNotFound like a service error, keeping
external API behavior unchanged.
Groundwork for not-found handling in ReadOnlyEdgeShard live-reload.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [AI] make ReadOnlyRoaringFlags bitmap and bool index counts lazy
Opening a read-only segment scanned every flags file end to end:
`ReadOnlyRoaringFlags::open` materialized the whole RoaringBitmap via
`iter_ones()`. Every payload field carries a null index, so this was paid
per field per segment, for bitmaps most queries never touch.
Make the bitmap a `OnceLock`, filled by a scan on first access. Open now
reads only the tiny status file. `ReadOnlyBoolIndex`'s three eager count
fields collapse into one lazily-derived, cached `BoolCounts`; its
`live_reload` refreshes them in place when present and leaves them unset
otherwise, so reloading an index nothing queries stays scan-free.
Propagate the resulting `OperationResult` through `RoaringFlagsRead`,
`PayloadFieldIndexRead::count_indexed_points`, `FieldIndexRead`,
`PayloadIndexRead::{indexed_points, get_telemetry_data}`, `build_info` /
`build_telemetry` and `SegmentEntry::{info, get_telemetry_data}`, out
into shard, edge and collection.
`ram_usage_bytes` stays infallible: an unmaterialized bitmap holds no
RAM, so it reports 0 via the new `bitmap_if_materialized`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [AI] correct `preopen` comment: `open` no longer scans the flags file
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [AI] fix edge examples for fallible `info()`
`EdgeShardRead::info` now returns `OperationResult<ShardInfo>`. The
examples live in their own workspace (lib/edge/publish), so the main
`cargo check --workspace` never saw them.
Every call site sits in `fn main() -> Result<(), Box<dyn Error>>`, so
propagate with `?`. `bm25-search` compiled either way but would have
printed the `Result` rather than the `ShardInfo`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: replace flush_all sync bool with FlushMode enum
SegmentHolder::flush_all took two adjacent bools (sync, force), and call
sites read as bare literal pairs like flush_all(true, false). Swapping
the arguments compiles and silently changes flush semantics: a swapped
pair at the snapshot site would make snapshots skip flushing entirely
when a background flush is running.
Introduce FlushMode { Sync, Background } for the first parameter so the
pair is no longer transposable and the behavior is named at each call
site. The force flag stays a bool since it feeds the
SegmentEntry::flusher(force) trait in lib/segment. No behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: exhaustive match on FlushMode instead of equality check
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Bound request snapshots in the slow requests log
Entries in the slow requests log retain a full serde_json::Value snapshot
of the (internal) shard request indefinitely. The distance matrix API
internally generates a batch of `sample` queries, each carrying a has_id
filter with all `sample` sampled ids, so a single log entry ballooned to
sample^2 ids expanded into a JSON tree (~90 bytes/id): ~90MB per entry for
sample=1000, ~2GB for sample=5000. Random sample ids give every request a
fresh content hash, so each call added a new entry until the 32-slot
queue filled — OOM long before that for larger samples.
Truncate all arrays in logged request bodies to 64 elements plus an
omission marker. Query batches are serialized per element up to the cap,
so the full untruncated JSON tree is never materialized even transiently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix rustfmt
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Remove from_iter_instead_of_collect from workspace lints
The lint was removed from clippy (beta) and now triggers
renamed_and_removed_lints warnings in every crate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::chunks_exact_to_as_chunks
Replace chunks_exact with a constant chunk size by as_chunks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::needless_late_init
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::useless_borrows_in_formatting
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::uninlined_format_args
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::for_kv_map
Iterate map values directly instead of discarding keys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_large_err on QueueProxyShard::new_from_version
The Err variant intentionally hands the LocalShard back to the caller.
Same pattern as the existing allow on ForwardProxyShard::new.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_unit_err on wait_for_consensus_commit
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add unified `memory: cold|cached|pinned` placement parameter for collection components
Introduce a single `memory` parameter that controls how each collection
component's data is held in RAM, replacing the inconsistent zoo of
`on_disk` / `always_ram` / `on_disk_payload` flags:
- `cold`: not pre-loaded from disk, cached with usage
- `cached`: pre-populated into page cache on load, evictable under pressure
- `pinned`: materialized on heap, never evicted by cache pressure
The parameter is available on dense vectors, HNSW config, all quantization
configs, the sparse index, all payload field index types, and payload
storage (as a new `payload: { memory }` sub-object on collection params).
When set, it overrides the deprecated legacy flag; when unset, behavior is
unchanged. Legacy flags are marked deprecated (Rust + proto) but keep
working; conflicts are resolved in favor of `memory` with a warning.
New capabilities enabled by the tri-state model:
- HNSW graph links can be pinned (first production caller of the existing
`GraphLinksResidency::Pinned`)
- sparse mmap index, quantized vectors and on-disk payload field indexes
gain a `cached` tier (mmap + populate on open)
`pinned` is rejected by API validation for components without a heap
variant (dense vector storage, payload storage). Low-memory mode degrades
placements at load time via `Memory::clamp_to_low_memory`, matching the
existing `prefer_disk`/`skip_populate` behavior. Effective-placement
comparison in the config-mismatch optimizer avoids spurious rebuilds when
the same placement is expressed through the new parameter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix gpu-gated tests for the new `memory` field
CI clippy runs with --all-features, which compiles the gpu-gated tests
that were missed locally: add the `memory` field to config literals and
allow deprecated placement params, same as in the rest of the tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add OpenAPI tests for memory placement, keep sparse config downgrade-clean
- OpenAPI tests: create/update collections with `memory` on every component,
assert the parameters are echoed in collection info, assert legacy-only
collections expose no new fields, and assert `pinned` is rejected (422)
for dense vector storage and payload storage on both create and update.
- Persist only the explicitly requested `memory` parameter in
`sparse_index_config.json` instead of the legacy-resolved placement, so
configurations using only the deprecated `on_disk` flag keep byte-identical
files that older Qdrant versions load without unknown fields.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Validate collection meta ops at construction, not only in the API layer
The `memory: pinned` rejection for dense vectors and payload storage
lived in `Validate` impls on the internal request types, which only ran
through the REST actix extractor. gRPC validates just the proto message,
so a gRPC client could persist `pinned` where it is not supported and
have it silently treated as `cached`.
Run the derived validation in `CreateCollectionOperation::new` and
`UpdateCollectionOperation::new` instead: the constructors are the
common chokepoint for all API paths, before the operation is proposed
to consensus. This covers every validator on these types, not just the
`memory` checks, and keeps consensus-apply unaffected so mixed-version
clusters never reject already-committed operations.
`UpdateCollectionOperation::new` becomes fallible; `remove_replica` now
uses `new_empty` since it carries no user config. Regression tests drive
the gRPC conversion path and assert `InvalidArgument` for `pinned` on
create and update, with `cold`/`cached` accepted as a control.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Resolve filter-based update operations to point ids before WAL write
Filter/condition-resolving operations (delete-by-filter, conditional
upsert, the *-by-filter payload/vector operations) stored their filter
in the WAL and re-resolved it against live segment state on every
apply. Replay-time state can differ from the original apply-time state
(the optimizer drops deleted points and their version records during
compaction), so WAL replay was not a deterministic function of the log
and could resurrect filter-deleted points.
Resolve such operations into concrete point ids at submit time, under a
fence that guarantees the resolution sees exactly the operations that
precede it in WAL order. The WAL now only ever contains id-based
operations (pre-existing variants only — no format change), so replay
applies the exact same point set as the original run.
Fixes#9575
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DfVDMFQcy9Ww791x8sHobW
* Fix rustfmt
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DfVDMFQcy9Ww791x8sHobW
* Drop coordinator-side resolution: every replica resolves locally
Replicas holding the same data resolve the same filter to the same point
set, and replicas that already diverged would not become consistent by
agreeing on a filter's resolution. Forward the original filter operation
as usual and let each replica's submit fallback resolve it under its own
fence — one uniform path regardless of where the update lands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DfVDMFQcy9Ww791x8sHobW
* Guard against is_filter_resolving / resolve_operation drift
A resolved operation must never still classify as filter-resolving,
otherwise a filter-carrying record could reach the WAL again (#9575).
Catch one direction of drift between the gate and the rewriter with a
debug assertion right after resolution.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Dedup points-vs-filter precedence into resolve_points_or_filter
The "explicit id list wins over the filter" rule was written twice on
the resolver side (DeletePayload arm and resolve_set_payload); a future
tweak landing in one copy only would make SetPayload and DeletePayload
silently diverge in what gets persisted to the WAL.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Assert rewritten WAL record reuses the incoming clock tag
The single-record-reuses-the-tag property is what WAL-delta recovery
and replica dedup rely on, but no test asserted it: submit the
delete-by-filter with a real clock tag and check the resolved
DeletePoints record carries it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Test replay of old-style filter records left in the WAL
Upgraded nodes can still hold WALs with unresolved filter operations;
the by-filter apply paths are kept so they replay one final time with
the old semantics. No test covered that path (the new submit flow can
no longer produce such WALs), so append a raw DeletePointsByFilter
record at the WAL layer, reload, and assert the matched points are
gone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add consensus test for per-replica filter-op resolution
Exercises the replicated path for filter/condition-resolving updates:
the coordinator forwards the original filter op and each replica
resolves it locally (delete-by-filter, insert-only and update-filter
conditional upserts, set-payload-by-filter, including per-shard empty
resolutions on a 2-shard collection). Asserts both replicas hold
identical state (reads prefer the local replica), then restarts the
whole cluster and asserts each replica replays its id-based WAL to the
same state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Add prefix keyword index coverage to model_testing soak harness.
Exercise prefix filters on a new `url` payload field through count/scroll,
paginated scroll, facet, search, and query ops, with CreateIndex/DropIndex
toggling a prefix-enabled keyword index to stress index lifecycle and fallback.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Apply rustfmt to model_testing prefix index changes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Address review: preserve keyword-equality coverage, reuse Filter::merge_opts.
- Arms 8/9 now randomly toggle either the plain keyword index on `tag`
(exact-match path) or the prefix-enabled keyword index on `url`, keeping
coverage of both index paths.
- optional_read_filter now delegates to Filter::merge_opts over the existing
match_num_filter / match_url_prefix_filter constructors instead of
hand-building conditions.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add prefix matching option to keyword index
Introduce an opt-in `prefix` option for the keyword payload index and a
new `match: { "prefix": ... }` filter condition, enabling efficient
byte-wise prefix filtering over keyword values (e.g. URL prefixes,
web-ui value autocompletion via facet + prefix filter).
Index side: a new `prefix_index.bin` file stores a sorted, front-coded
key dictionary with a resident block index (cumulative counts per
block); it is an ordered view over the keys of `values_to_points.bin`
and stores no postings. Presence of the file signals prefix support at
load time, so legacy segments load unchanged and enabling the option
goes through the standard incompatible-schema rebuild. The mutable
variant keeps an in-RAM ordered key set (not persisted), the immutable
variant builds a sorted key vector at load, and the on-disk variant
reads the dictionary lazily (block index resident, 1-2 block reads per
prefix lookup; reader is generic over UniversalRead).
Query side: prefix conditions are served from the dictionary when
available (filter + cardinality estimation from per-block aggregates),
from the forward index as per-point checks, and degrade to the payload
full-scan fallback otherwise - same execution model as other match
conditions. Strict mode (`unindexed_filtering_*`) rejects prefix
queries on fields without a prefix-enabled keyword index via a new
KeywordPrefix capability.
HNSW payload blocks: prefix-enabled indexes additionally emit prefix
blocks for heavy branching trie nodes (single-child chains collapsed to
their longest common prefix, one block per distinct point set, emitted
largest-first) so filtered search with prefix conditions gets navigable
subgraphs without rebuilding the same subset repeatedly.
API: `prefix` flag on KeywordIndexParams (REST bool, gRPC empty
message for extensibility), `prefix` variant in the Match oneof, edge
python bindings, regenerated OpenAPI spec.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Split prefix index into a dedicated module, fix clippy in tests
Reorganize the flat prefix_index.rs / prefix_read.rs into a
map_index/prefix_index/ module: format.rs (on-disk layout primitives),
writer.rs, reader.rs (PrefixIndex), map_read.rs (StrMapIndexPrefixRead
with per-variant impls) and tests.rs, with a file-format diagram and a
read-path walkthrough in the module docs. No logic changes.
Also fix clippy --all-targets complaints in test code: replace a
wildcard Match arm with an exhaustive list and a field-reassign-with-
default with a struct literal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add OpenAPI test for prefix match and snapshot file-tracking test
- tests/openapi/test_prefix_match.py: index-less fallback, prefix index
creation with schema echo, scroll/count parity against ground truth,
facet + prefix filter (the autocompletion flow), strict-mode rejection
without the prefix capability.
- test_prefix_index_file_tracking: `prefix_index.bin` is listed in
`files()` / `immutable_files()` exactly when built with the option.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Replace hand-rolled varint parsing with bytemuck Pod records
Per review: the prefix index format now uses fixed-size little-endian
Pod records (BlockEntry 24 B, KeyEntry 12 B, Header 40 B) written with
bytemuck::bytes_of and read back by copy via pod_read_unaligned — no
manual varint encode/decode, no alignment requirement, one shared
read_record helper. Costs ~9 bytes per key on disk versus LEB128; the
raw key bytes dominate dictionary size, so the simplification wins.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fetch the whole candidate block range with a single storage read
Candidate key blocks of a prefix lookup are contiguous in the file, so
enumerate them from one ranged read instead of one read per block; the
over-read versus the exact key range is bounded by the two boundary
blocks. Block decoding is split into a storage-free helper reused by
the per-block path of stats estimation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Align prefix payload blocks with the geo index granularity principle
Geo's large_hashes emits only the smallest geohash regions above the
threshold — a disjoint antichain, never a parent nested with its
children. Prefix payload blocks now follow the same rule: a heavy
collapsed trie node is emitted only if nothing heavy is nested inside
it, counting both deeper qualifying prefixes and single heavy values
(which already get their own exact-match blocks). Emitted blocks are
therefore mutually disjoint and disjoint from exact-value blocks; no
near-collection-sized ancestor subgraphs, no reliance on the HNSW
connectivity check to skip nested duplicates.
Implemented as a `covered` flag propagated through the existing
LCP-interval scan, still one O(total key bytes) pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Document block wire format and unaligned-read rationale in decode_block
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix: unblock optimizer after deleting a named vector
Deleting a named vector could permanently block the config-mismatch
optimizer. The source-superset check in SegmentBuilder::update cancelled
every rebuild that found the deleted vector still in old segment files,
and each retry cancelled again, so optimizations got stuck forever.
Removing the check (as in #9609) would fix delete but reintroduce data
loss for the CreateVectorName race. Instead, tell the two cases apart
with the live collection schema: prune a source vector that is gone from
the schema (a real deletion), but cancel when it is still present (a
freshly created vector this optimizer has not yet seen). This is safe
because the schema is persisted before the op reaches segments, and the
live schema is read after the source segments are frozen.
The live set covers dense and sparse vectors, since a segment stores both
together. When no live source is wired in, the conservative always-cancel
behavior is kept.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: wire live vector names into edge optimizers
Deleting a named vector left the edge path with the pre-fix behavior:
segment_optimizer_config hardcoded live_vector_names to None, so a merge
touching a segment that still carried the deleted vector cancelled, and
EdgeShard::optimize() propagated the cancellation as a hard error forever.
Share the shard config behind an Arc and hand the blocking optimizers a
provider that reads the current vector names on every call. Same safety
argument as the server wiring: update() holds the segments read guard
across both the segment application and the config update, so any name a
frozen source segment carries is visible to the live read.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: share vector-name enumeration via CollectionParams::vector_names
The optimizer's live-schema set and the WAL-recovery valid-name set are
the same dense+sparse enumeration and must stay in lockstep; a drift
between them would reintroduce a wrong prune/cancel decision. Replace
the private helper in optimizers_builder and the inline block in WAL
recovery with a single CollectionParams method.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: drop SegmentOptimizer::live_vector_names forwarding hop
The default trait method only forwarded to the config getter and had a
single caller; ShardOptimizationStrategy now reads the config directly,
removing one layer of indirection.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix optimizer recreation dropping sibling shard restarts on error
Replace try_join_all with join_all in recreate_optimizers so a failing
shard does not cancel non-cancel-safe on_optimizer_config_update futures
that have already stopped their update workers.
Fixes#9670
Co-authored-by: Cursor <cursoragent@cursor.com>
* Apply cargo fmt
Co-authored-by: Cursor <cursoragent@cursor.com>
* Apply suggestion from @timvisee
Co-authored-by: Tim Visée <tim+github@visee.me>
* Move optimizer config update tests into dedicated file
Extract worker restart test hooks and unit tests from updaters.rs into
optimizer_config_update_tests.rs, following the snapshot_tests pattern.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
