* Measure and persist document length in the mutable text index
BM25 length normalization needs the total token count per point, which nothing
stored: `point_to_tokens_count` is the distinct count, and its meaning is fixed
by the user-visible `values_count` filter.
- `doc_len` is measured in `add_many`, the only place that still sees every
token, and persisted in the stored record alongside the tokens.
- It is a parameter on `index_str_tokens` and `MutableInvertedIndexBuilder::add`,
never derived. Without phrase matching the stored tokens are sorted and
deduplicated, and the index is rebuilt from those records on every segment
open, so a derived length would degrade to the distinct-term count on restart.
- Array boundary sentinels are discounted by inserted count, not by value:
`tokenize_doc` does not strip that character from user text the way
`tokenize_query` does, so a payload containing it has those tokens indexed
and they must be counted.
- `MutableInvertedIndex` gains `point_to_doc_len` and a running `total_tokens`,
maintained across add, overwrite and remove, so `avgdl` is a division rather
than a scan. `set_doc_len` is the only writer, so an absent length means the
same thing on every path: the slot is zeroed, never left stale.
- Recording is gated behind `TextIndexParams::scoring()`, a private const for
now. Nothing can ask for a ranked query yet, so recording on every text index
would rebuild every collection to produce data no query can reach. State
lives in the data rather than in a flag: `point_to_doc_len` is an `Option`
the way `point_to_doc` already encodes positions, and `add_many` asks the
index whether it records lengths rather than asking the config.
- `StoredDocument::doc_len` is an `Option` skipped on write when absent, so a
non-scoring index writes byte-identical records to today's and a legacy
record reads back as `None`. A document whose tokens were all filtered is a
real `Some(0)`, and stays distinguishable from one that was never measured.
The immutable and on-disk backends drop the value for now and grow their own
sidecar next.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Address text index document length review feedback
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Share the text index post-tokenization indexing path
Extract `MutableInvertedIndex::index_str_tokens` so the write path and
read-only live reload cannot drift apart, and collapse the two identical
on-disk file listings into one.
The extracted helper gates the ordered document on `point_to_doc.is_some()`
rather than re-reading `config.phrase_matching`. Equivalent, since
`point_to_doc` is built from that flag, and it skips a needless clone if the
two ever disagree.
No behavior change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Cover the text index live reload path
`ReadOnlyAppendableFullTextIndex::live_reload` had no direct test: it replays
stored documents through the same post-tokenization indexing as the write path,
but nothing pinned that down.
Asserts the incremental reload lands on the same state as a fresh
`open_appendable` after a writer deletes one point and appends two, over both
`phrase_matching` values. The phrase leg matters because adjacency depends on
the ordered document being indexed, not just the token set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Apply suggestion from @timvisee
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
* fix: do not claim an unfinished operation when flushing
A flush pass can capture a segment between the separately locked steps of
one update operation. Persisting it under that operation's version marks
the segment clean while the rest is still in memory, so every later pass
skips it and the WAL acknowledge moves past the operation.
Clamp what a flush claims to the last fully applied operation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Fix rustfmt in alias_mapping test after merging dev
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
* Remove MessagePack (rmp-serde)
The WAL switched from msgpack to CBOR in v0.3.5 (2021-07-11), so v0.3.4
is the last version that wrote msgpack entries. Drop the read fallback
kept for those entries, plus the remaining test and bench usages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Drop unused fs4 dependency from collection
Not referenced anywhere in the crate. Still used by wal and common, so
the workspace entry stays.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Add optional dial9 Tokio telemetry behind a `dial9` feature
Integrate dial9 so storage runtimes can emit production-friendly Tokio
traces. Recording is off unless the crate is built with `--features dial9`
and DIAL9_ENABLED=true is set at runtime; with the feature off, runtime
construction is byte-for-byte unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y7X6MkjY3P7wpP2MfHdTDY
* Enable dial9 CPU and schedule profiling
Turn on cpu-profiling and sched events behind the same `dial9` feature,
add the DIAL9_CPU_* / DIAL9_SCHEDULE_* env knobs, and document the frame
pointer rustflags the stack unwinder needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y7X6MkjY3P7wpP2MfHdTDY
* Harden dial9 env parsing and the writer-failure path
- Reset Cargo.lock to the branch point and re-resolve, so the diff is
additive instead of re-resolving unrelated packages. This drops the
heck 0.5.0 -> 0.4.1 downgrade, which sat in the default build graph and
would have changed proto codegen identifier casing. The remaining
non-additive entry, toml_parser 1.0.9 -> 1.1.3, is forced by
proc-macro-crate via dial9-trace-format-derive.
- Parse DIAL9_* booleans the way dial9 does, accepting 1/y/yes/on and
0/n/no/off and warning on anything else. `str::parse::<bool>` took only
exact lowercase true/false, so DIAL9_CPU_PROFILE_ENABLED=0 silently left
99 Hz sampling on and DIAL9_ENABLED=1 silently left recording off.
- Require the numeric knobs to be positive. A zero disk budget made dial9
evict everything and stop recording within seconds while the log still
reported telemetry enabled.
- Treat a set-but-empty DIAL9_TRACE_DIR as unset. It skipped the /tmp
fallback and wrote up to the full budget into the working directory,
which is /qdrant next to storage/ in the official image.
- Return a disabled guard as soon as the trace writer fails, before
with_cpu_profiling and with_sched_events run. Those start their profilers
eagerly, opening a perf event per thread and installing a process-global
signal handler that build() would then discard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y7X6MkjY3P7wpP2MfHdTDY
* Correct the dial9 docs and give them their own section
- `--cfg tokio_unstable` is required for any task data at all, not merely
for fuller coverage: dial9's poll, spawn and terminate hooks are all
`#[cfg(tokio_unstable)]`, and nothing in the repo sets the flag. Without
it there is no task timeline and DIAL9_TASK_TRACKING_ENABLED does nothing.
- Document `-C debuginfo=2`. `[profile.perf]` inherits `release` and sets no
`debug` key, so the documented build symbolized off the ELF symtab with
inlined callees collapsed and no file or line, unlike `[profile.bench]`
which sets `debug = true` for this reason.
- Move the dial9 material out from between the feature list and the prose
that belongs to it. Those paragraphs describe `tracing` instrumentation
and read as dial9's when the example is wedged in front of them, which
points readers at `#[tracing::instrument]` for a tool that records Tokio
runtime events and no tracing spans.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y7X6MkjY3P7wpP2MfHdTDY
* Use cfg_select!
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: timvisee <tim@visee.me>
* Add SetFlushInterval op to the model tester
Changes the collection's flush_interval_sec mid-run through the same path
update_collection takes (persist the optimizer-config diff, then recreate
the optimizers in the background). The model is untouched: what it perturbs
is the flush cadence, so how much of the workload is still WAL-only when a
restart hits, plus the worker stop/start race in on_optimizer_config_update.
Kept in FORCE_OFF for now: with the optimizer on it makes stale point state
visible within a few ops of the config change. Narrowed to
recreate_optimizers_background, see the comment on Swarm::BASE.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011AjmS5GFeGztfP3JqutnXj
* Keep SetFlushInterval enabled in the swarm
Drops it from FORCE_OFF so the divergence it surfaces is reachable without
--enable-force-off (which would also enable the broken vector-name ops).
The evidence moves from the FORCE_OFF comment onto the op's own doc.
The two optimizer-on harness gates now fail whenever the swarm draws the op.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011AjmS5GFeGztfP3JqutnXj
* Propagate proxied changes when unwrapping proxies on optimization failure
unwrap_proxy puts the wrapped segments back into the segment holder, so the
changes recorded on the proxy while the optimization ran (deleted points,
index and vector-name changes) have to reach the wrapped segment first. They
did not, so every point deleted or overwritten during the optimization kept
its pre-optimization copy live next to the new copy in the write segment, and
reads saw both: counts too high, scroll and search returning the stale copy.
The snapshot unproxy path already does this; the optimizer failure path was
the only place putting a wrapped segment back without it. It is reachable
whenever the shard outlives the cancellation, in particular an update_collection
that recreates the optimizers while an optimization is in flight.
Lock order is holder-then-updates, matching try_unproxy_segment: updates-then-
holder-write deadlocks against the snapshot path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011AjmS5GFeGztfP3JqutnXj
* Drop the stale failure note from the SetFlushInterval doc
The divergence it described is fixed in this branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011AjmS5GFeGztfP3JqutnXj
* test as well with 0s as flushing interval
* Fail optimization unwrapping when proxy propagation fails
Losing proxied deletes and index changes is data corruption, so return the
error instead of logging it: no proxy is unwrapped and the changes stay
served by the proxies. The cancelled-segment cleanup moves ahead of
unwrap_proxy so the orphan is still removed when that error fires.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpoaAtGbAQAEuxEi8ScHc5
* Drop the model tester --flush-interval-sec flag
SetFlushInterval covers the interval now, so the run starts at the shipped
5s default (fixture::INITIAL_FLUSH_INTERVAL_SEC, still traced in the header)
and the ops move it from there. Also documents what 0 does now that it is a
generated value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpoaAtGbAQAEuxEi8ScHc5
* Fail snapshot unproxying when proxy propagation fails
Both paths logged the error and unwrapped anyway, dropping the deletes and
index changes that never reached the wrapped segment. Same reasoning as
unwrap_proxy in the optimizer.
try_unproxy_segment hands the lock back and leaves the proxy installed, the
failure mode its doc already describes: the caller keeps it in `proxies` and
unproxy_all_segments retries the propagation right after. unproxy_all_segments
returns before touching the holder, so the temp segment the surviving proxies
write into stays in place (remove_segment_if_not_needed only checks whether it
is empty and appendable, not whether a proxy still references it).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CpoaAtGbAQAEuxEi8ScHc5
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
ast-grep 0.45 no longer parses a leading `::` fragment as a pattern, so
`pattern: ::$MOD` stopped matching and every `::common::` / `::wal::` path
survived into the generated qdrant-edge crate, failing `just rs-check`.
Matching the node text instead keeps the rule working on 0.44 and 0.45: the
amalgamation output is byte identical to what 0.44 produced before.
Claude-Session: https://claude.ai/code/session_01CpoaAtGbAQAEuxEi8ScHc5
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Keep consensus operation awaiters alive for concurrent waiters
Callers proposing an identical consensus operation deduplicate onto one
broadcast channel, and the map holds its only sender. Removing the entry on
timeout therefore closed the channel for every other waiter, failing their
still in-flight operation with "Channel sender dropped".
Only remove the entry once no receiver is left, dropping our own receiver
first so the last caller out cleans up. Apply the same to
await_for_multiple_operations, which registered awaiters but never
deregistered them when it timed out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Add debug assert to ensure we clean up consensus operation waiters
* Deregister consensus operation awaiters when the waiter is dropped
Dispatcher::submit_collection_meta_op registers the expected operations before
proposing, then drops that future unpolled whenever the proposal itself fails.
The awaiters stayed in the map with no receiver left, so the next identical
request deduplicated onto a dead entry and never heard back. This is what
tripped the new debug assert in CI: a rejected create-collection left a
SetShardReplicaState awaiter behind, and the next run of the same test hit it.
Move registration into an OperationAwaiters guard that deregisters on drop, so
timeout, drop-before-poll and request cancellation are all covered by one path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Close the race the awaiter debug assert trips on
The assert is sound only if no one can observe an entry whose receivers are all
gone. Both cleanup sites dropped their receiver before taking the map lock, so a
concurrent register could see exactly that and panic. Drop the receiver while
holding the lock instead, and take that lock once per batch rather than once per
operation: creating a collection registers an awaiter per replica, on the mutex
the consensus thread needs for every entry it applies.
Collect the awaiters into the guard as we go, so giving up part way still
deregisters the ones already registered, and only build the broadcast channel
when the operation is not already in-flight.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: timvisee <tim@visee.me>
* Drop obsolete clippy large-error-threshold override
The 256 threshold was pinned for clippy 1.87 while tonic's `Status` was a
large error type. Upstream boxed its contents in `5de7bad` (hyperium/tonic#2253),
which is in the pinned 0.14.6 fork, so `Status` is now a single `Box` and the
default threshold of 128 passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Remove stale clippy allows
These 11 allows no longer suppress anything under any of the three CI clippy
configurations (default, --all-targets, --all-targets --all-features).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drops 6 crates from the release build and 7 from the workspace test
build, with no source changes.
- geo: no triangulation, only Contains/Intersects/Haversine (spade, earcut)
- jsonwebtoken: HS256 from_secret only, no PEM keys (pem, simple_asn1)
- tar: nothing sets unpack_xattrs, which defaults to false (xattr)
- duplicate: every duplicate_item names its module (proc-macro2-diagnostics)
- pprof: no C++ frames to demangle (cpp_demangle)
Also promotes duplicate to a workspace dependency.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Steer writes away from appendable segments at max_segment_size
* pick a write target that stays under the configured size cap, instead of
growing an appendable segment past it
* clamp the deferred points threshold to max_segment_size, treating a zero cap
as uncapped
* apply the same cap when replaying the WAL, so recovery matches live updates
* plumb the cap through the update worker and cover the silent-failure gaps
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Extract the per-segment capacity check into a helper
Lets has_appendable_segment_with_capacity short-circuit on the first segment
below the cap instead of collecting every eligible ID.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Replace cgroups-rs with direct cgroup memory file reads
We used cgroups-rs in exactly one place, to read the memory limit and
usage of our own cgroup, so read those files directly instead. Drops 34
crates from the lockfile, including the zbus stack that carries
RUSTSEC-2026-0221.
Also fixes two latent cgroup v1 bugs (the LONG_MAX unlimited sentinel
reported ~9 EB of total memory, an unreadable limit file reported 0
bytes) and the hierarchy mix-up on hybrid hosts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Decline cgroup memory reporting when the usage read fails
Reporting a usage of 0 made available_memory_bytes claim the whole cgroup
limit as free. Fall back to sysinfo when the usage file cannot be read at
init, and keep the last known value on a failed refresh.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Keep the last known memory limit when its read fails
A transient read failure cleared the cached limit and silently fell back
to host memory while the process was still capped, the same direction of
over-reporting as the usage read. Both now keep their last known value,
and a limit lifted at runtime still clears.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Treat a malformed memory limit as an error, not as unlimited
Parse failures returned Ok(None), so garbage in the limit file cleared a
valid cached limit on refresh and read as unlimited at init. Reserve
Ok(None) for "max" and the v1 sentinel, and report anything else as
InvalidData so the last known limit survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The model_testing harness gates preallocate a 32 MB gridstore page per
storage per segment, so ~440 live points occupy ~600 MB and the five
gates peak around 4.5 GB. On a disk-backed temp dir that preallocation
plus the flush and snapshot traffic dominates their runtime: measured on
ext4/NVMe the snapshot gate takes 337s versus 54s on tmpfs, at 24% CPU,
and on btrfs a set finishing in ~30s on tmpfs was reported not to finish
in 7 minutes.
Point TMPDIR at /dev/shm on the Linux test jobs. Whole-suite peak there
is 4.6 GB against the ~8 GB a GitHub runner provides, and all 3066 tests
pass. Linux only: macOS and Windows have no /dev/shm.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Fix clippy warnings from Rust 1.98 beta
* drop a redundant trait import in an io_bridge test module, it already
arrives through `use super::*`
* rewrite two `chunks_exact(CONST)` sites as `as_chunks::<{ CONST }>()`
for the new `chunks_exact_to_as_chunks` lint
* return `bool` from `wait_for_consensus_commit` instead of
`Result<(), ()>`, which `result_unit_err` now flags on `async fn`. Its
only caller did `.is_ok()` on it
* allow `result_large_err` on `QueueProxyShard::new_from_version`, which
hands the `LocalShard` back to the caller on failure. Mirrors the allow
already on `ForwardProxyShard::new`
* migrate three `Atomic::fetch_update` calls to `try_update`, the name it
is renamed to in 1.99. The new name already exists at our 1.97 MSRV
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* keep guarantee on caller
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(shard): flush CoW destinations before the payload-index pre-build flush
create_field_index force-flushes each segment before building an index on it
(flush-before-build, #9767), one segment at a time, outside flush_all's
all-segment lock capture and copy-on-write dependency ordering. That flush
durably advances a CoW source past the delete halves of its pending moves.
The appendable-first iteration order usually flushes the destination before
the source, but not always: a destination proxy-wrapped by a running
optimization is classified non-appendable and can skip its flush entirely
through the already_indexed short-circuit (the proxy reports the field as
present), and a move landing mid-pass is ordered behind nothing. Once the
source flushes, the move's WAL entry stops being replayable: the pre-image
is durably deleted while the only current copy sits in the unflushed
destination, and a graceful close then loses the point.
This is the root cause of the nightly model-testing reload divergence
(#10095), traced end-to-end in CI runs 31583878492 and 31583871346: cow move
op 5197 into a freshly proxied destination, index op ~5252 flushing every
source past it while skipping the proxy, destination reloading at 5181,
replay declining with 'No point with id'.
The fix mirrors flush_all's invariant at the only per-segment flush site:
before flushing a segment, flush the destinations of its pending
flush_dependency edges (one hop suffices, destinations are appendable and
never CoW sources). Destination guards are taken before the flush lock to
keep the documented [segment locks -> flush lock] ordering.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(shard): regression test for the CoW-destination flush in create_field_index
Reproduces the #10095 loss shape deterministically: a pending copy-on-write
move out of a non-appendable source, a destination whose own pre-build flush
is skipped by the already_indexed short-circuit, then a holder-wide
create_field_index. Verified failing with the dependency-aware flush
neutralized (destination stays behind the move while the source flushes past
it) and passing with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(shard): move the CoW-aware single-segment flush into SegmentHolder
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
limits_only_apply_while_the_quota_is_enabled assumed the system temp dir
is on a filesystem at least 1% full, the smallest configurable disk
limit. On a tmpfs /tmp the usage floors to 0% and the limit never trips.
Measure the tempdir's filesystem first and fall back to a tempdir in the
crate directory when it is under 1% used.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`cargo audit` flags rand 0.7.3 as unsound (RUSTSEC-2026-0097), and
permutation_iterator 0.1.2 is its sole importer. The crate is
unmaintained, so the finding is permanent for as long as we depend on
it.
Everything we used it for is "pick k distinct random indices out of n",
which is exactly `rand::seq::index::sample` from the workspace rand.
Switch the three src call sites and the two benches over, and drop the
dependency. 11 crates leave Cargo.lock.
Also fix a comment in quantile.rs claiming the permutation was
deterministic per count: the old crate keyed itself from thread_rng on
every call, so it never was.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The app telemetry was the only user of the sys-info crate, while segment
already depends on sysinfo for cgroup-aware memory accounting. Read the
distribution id/version via sysinfo statics, and the disk size fallback
via common::disk_usage, so the whole sys-info crate (and its bundled C
sources) drops out of the build. sysinfo is hoisted to a workspace
dependency, shared by the root crate and segment.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`TableOfContent::new` installs the process-global quota manager, and
`set_global` treated a second install as a startup-order bug worth a
`debug_assert!`. A test binary runs all of its tests in a single process,
and every test that builds a table of contents installs the manager again,
so all but the first one panic.
Building several tables of contents in one process is legitimate for test
harnesses, so the second install is no longer fatal. A node that installs
twice still reports it loudly through the existing error log.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: cover TurboQuant, turbo4 datatype and keyword prefix in compat data
Extend the storage compatibility fixture with vector and payload index
features that landed since the generator was last updated:
* TurboQuant quantization, one collection per persisted blob layout
(bits1_5 and the default bits4)
* the turbo4 storage datatype, on both dense and multivector storage
* the keyword index `prefix` option, plus a matching prefix scroll in the
query battery
Sparse vector configs reject the turbo4 datatype, so create_collection omits
the sparse datatype for that collection rather than forwarding it. This is a
no-op for every other collection.
Archives are generated once per release and keep the collection set of their
own generation, so expected collections are now resolved per version and the
new ones are only required from v1.19.0 onward. The prefix scroll stays
ungated: archives without the prefix index answer it by scanning.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: fail instead of skip when a compatibility archive is missing
A 404 from the compatibility bucket means the archive was never published,
which no amount of retrying fixes. Skipping it reported the version as
covered while nothing ran, so a pull request adding a version could stay
green with its new coverage never executing.
Fail on 404 and keep skipping connection resets and timeouts, so a bucket
outage still does not turn unrelated pull requests red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
POSIX_FADV_DONTNEED cannot evict pages of a tmpfs file: the page cache
is the backing store, so there is nothing to drop them to. On systems
where /tmp is tmpfs (Ubuntu 24.10+ default) the test fails with all
pages still resident. Detect tmpfs via statfs and skip.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The turbo model tests, HNSW graph properties tests, and id-tracker
mapping tests generate their datasets with StdRng (ChaCha12 in rand
0.10). Switching to SmallRng (Xoshiro256++) makes the turbo model tests
~10-14% faster and the 400k-mapping id-tracker test ~30% faster.
All affected tests pass with the changed seeded sequences, including
the tolerance-carrying turbo model comparisons.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`load_from_wal` splits WAL recovery in two: it replays
`[first_index, applied_seq + APPLIED_SEQ_SAVE_INTERVAL + 1)` synchronously and
hands the remaining tail to the update worker, which applies it in the
background *after* `LocalShard::load` has returned and the shard has started
serving reads.
That split was introduced by #8008 and applies to every collection, so up to
`update_queue_size - 1` operations already acknowledged to a client with
`wait=true` can be missing from reads right after a restart, reappearing one by
one as the worker catches up.
Only `prevent_unoptimized` needs that routing: the update worker signals the
optimizer per operation, and optimization is the only thing that makes deferred
points visible. Everywhere else the synchronous replay is sufficient, so gate
the use of `applied_seq` on the flag -- the same condition that already gates
the worker's deferred-points wait -- and replay the whole WAL before load
returns, as it did before #8008.
Found by the crasher: after a crash-restart cycle it reported 72 missing points
out of a confirmed 3202, with the shard counting 3202 points while 3930 had been
acknowledged and 332 WAL entries were still queued across two shards.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The persisted_hashmap and disk_cache tests generate their datasets with
StdRng (ChaCha12 in rand 0.10). String keys draw one random_range call
per character, so the crypto generator dominated the test runtime:
test_k_str_* drop from ~3.3s to ~1.4s each with SmallRng (Xoshiro256++).
Assertions are self-consistency checks (write then read back), so the
changed sequences carry no retuning risk.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Benches: use SmallRng instead of ChaCha12-based generators
All benchmarks used StdRng or rand::rng() (ThreadRng), both backed by the
ChaCha12 block cipher in rand 0.10. Benchmarks do not need crypto-strength
randomness, and several draw random values inside the timed closure, so
cipher work was included in the measurement itself.
Switch every bench target to SmallRng (Xoshiro256++), and key the HNSW
graph cache and sparse index cache by RNG algorithm so stale caches built
from the old generator are not reused against newly generated vectors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Benches: replace free-function rand::random with local SmallRng
Addresses review: rand::random draws from the thread RNG (ChaCha12),
including inside the timed loop of the pq score benchmark.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Model tester: cover all quantization types
Add a `quantization` field to `VectorCandidate` so candidates can carry
any `QuantizationConfig` variant, and materialize the configs in the
fixture (`quantization_config`). The inline-storage vector "i" keeps its
scalar Int8 config, now declared on the candidate instead of hard-coded
in the fixture.
New candidates:
- "p" Dense(8) + Product x4
- "v" Dense(6) + Binary (non-byte-aligned dim, trailing-bit padding)
- "r" Dense(8) + Turbo (search-side TQ over Float32 storage)
Quantization x datatype combos:
- "l" Dense(6) Float16 + Binary
- "d" Dense(8) Turbo4 + Turbo default bits (keep-source-rotated branch
of `should_keep_source_rotated`)
- "g" Dense(8) Turbo4 + Turbo Bits1_5 (Padded rotation, rotate-back
branch)
This is model-safe: schema quantization keeps the original vectors, so
read-back predictions are untouched; the approximate quantized scoring
only feeds the membership-only Search/Query/Recommend checks.
`assert_candidates_predictable` enforces the wiring constraints:
quantization is dense-only (the fixture only wires the dense arm) and
requires `initially_active` (CreateVectorName's `DenseVectorConfig`
carries no quantization).
Verified: seeds 1/2/3/7/42 soaks (5k ops, restarts, optimizer on) green;
quantized codes confirmed on disk for every quantized candidate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Model tester: make inline_storage a VectorCandidate knob, enable on "l" and "d"
Replaces the fixture's name-based INLINE_STORAGE_VECTOR special case with an
inline_storage field on VectorCandidate (requires quantization, enforced by the
startup assert). Enables it on "l" (Float16 base + padded Binary links) and "d"
(Turbo4 base + TQ links) to cover more (base layout, link encoding) pairs of the
CompressedWithVectors format; "v", "r", "g" and "p" keep the non-inline paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The HashSet bought nothing: names are unique by construction (map keys)
and all callers only iterate. A sorted Vec skips the hashing and set
allocation, and makes the iteration order deterministic.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The WAL is only truncated past operations whose segment flush was
confirmed, so first_index is a durable lower bound on the applied
sequence. The persisted applied_seq can legitimately lag behind it by
more than one save interval: it is saved every 64 update-worker calls
from a counter that restarts at zero on process start, and synchronous
WAL replay never feeds it. A replay target computed from such a stale
applied_seq can then sit before first_index, tripping the debug_assert
from #8454 (flaky model_testing gate, #9844) and, in release builds,
enqueueing already-truncated indices that fail with spurious
"Operation not found in WAL" errors.
Clamp the replay target at first_index: nothing before it ever needs
replay.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Every point id the workload draws now comes from an IdSpace pool
precomputed at startup. A --uuid-id-fraction (default 0.5) of the
--id-pool slots are well-formed v4 UUIDs built from the seeded rng via
uuid::Builder::from_random_bytes, the rest stay numeric. Precomputing
the pool keeps the id-reuse semantics (upserts overwrite live points,
deletes and retrieves hit them) that fresh per-op random UUIDs would
lose, and keeps runs seed-reproducible.
Sampling consumes a single range draw per id, exactly like the previous
NumId draw, so fraction 0 consumes no extra rng draws and reproduces
the numeric-only op stream byte-for-byte. The harness smoke tests run
with fraction 0.5, and the fraction is recorded in the trace header.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Every vector candidate now carries a distance metric instead of the
hardcoded Dot, threaded into the fixture schema, the CreateVectorName
generator and the read-back prediction.
The model predicts Cosine read-backs exactly by mirroring the engine's
ingestion preprocessing: metric_preprocess follows
NamedVectors::preprocess_dense_vector's per-datatype dispatch and calls
Distance::preprocess_vector itself, so predictions track the engine by
construction (including the identity preprocess of the byte metric,
which stores Uint8 vectors un-normalized). Stored vectors are
preprocessed exactly once (optimizer and CoW moves transfer raw bytes),
so predictions stay exact across moves, including Cosine + Float16.
New candidates: "e" (dense Cosine), "n" (multi-dense Cosine, per-row
normalization), "x" (dense Cosine + Float16), "o" (dense Cosine +
Turbo4, padding-free dim), "j" (dense Euclid), "k" (dense Manhattan).
Euclid/Manhattan preprocess is an identity, so their value is engine
side: Order::SmallBetter comparator coverage.
The startup predictability check now also rejects sparse + non-Dot
(sparse schemas carry no distance) and Turbo4 + Euclid/Manhattan (TQ's
L1/L2 modes store lengths differently from Dot/Cosine and their
copy-on-write re-quantization fixed point is not soak-validated yet).
Soak-validated on seeds 1/2/4/5/6/7/8 (30k ops), including two
restart runs (restart probability 0.002) with the optimizer enabled.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Profiling showed SipHash over JsonPath keys (RandomState::hash_one)
dominating field index lookups in
StructPayloadIndexReadView::estimate_field_condition: the map holds
only a handful of indexed fields, but it is queried per condition per
filter evaluation per query, and hashing the path is the entire
lookup cost.
Switch the field index lookup maps (IndexesMap, ReadOnlyIndexesMap,
the read view borrow, and the query_checker / condition_converter /
value_retriever signatures that receive them) from std HashMap to
AHashMap, already used throughout the crate. Serialized and API-level
schema maps keep std HashMap: they are cold and their type leaks into
conversions.
conditional_search bench (criterion, 100 samples, vs saved baseline):
- struct-conditional-search-query-points: 583.7 us -> 443.5 us (-23.1% median, p ~ 0)
- struct-conditional-search-context-check: 73.2 us -> 64.8 us (-12.6% median, p ~ 0)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add Float16 and Uint8 storage datatypes to the model tester
Extend VectorCandidate with a datatype override and fold the DenseTurbo
kind into Dense + Some(Turbo4) so storage datatype has a single source
of truth. Two new initially-active candidates exercise half-precision
("h", dense 6) and unsigned-byte ("y", dense 4) storage; "c" carries an
explicit Some(Float32) to cover schema configs that spell the default
datatype out.
The model predicts lossy read-backs through the engine's own
PrimitiveVectorElement impls (as Turbo4 reuses turbo_storage_roundtrip)
and compares them exactly: both round-trips are deterministic and
idempotent, so they stay bit-stable across optimizer moves, WAL replay,
and reloads. Uint8 components are drawn from 0.0..256.0 since the
storage truncates with `x as u8` and unit-range draws would collapse to
zeros.
A compile-time assertion rejects datatype overrides on non-Dense
candidates: the fixture's sparse/multi-dense arms ignore the field and
multi-dense read-backs are compared without a round-trip prediction, so
a lossy multi-dense candidate would soak-panic with a false divergence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Plumb Float16 and Uint8 multi-dense support in the model tester
Multi-dense storage converts the flattened matrix component-wise
(from_float_multivector), so per-row round-trips through the same
PrimitiveVectorElement impls predict read-backs exactly. The fixture's
multi-dense arm now applies the candidate datatype (matching the
CreateVectorName path), model_vector predicts per-row, and two new
initially-active candidates exercise the combination: "w"
(MultiDense(5), Float16) and "z" (MultiDense(3), Uint8).
The compile-time candidate check narrows to the combinations that
remain unpredicted: Turbo4 multi-dense (the multivector quantization
path differs from the per-vector turbo_storage_roundtrip) and sparse
with any datatype override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Make datatype match exhaustive in random_dense_vec
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review findings on datatype plumbing
- Start candidate "c" active so the explicit Float32 schema path runs in
default soaks (CreateVectorName is FORCE_OFF by default)
- Single Float16/Uint8 roundtrip dispatch shared by the dense and
multi-dense arms of model_vector
- Hoist shared fixture builder plumbing into dense_params_builder
- Build one DenseVectorConfig literal in the CreateVectorName generator
- Inline single-caller datatype_of wrapper
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Replace const-eval candidate check with a startup assert
Const eval forbids iterators, forcing an index-based while loop. A plain
function called at the top of run() reads better, still fails before any
op is applied, and names the offending candidate in the panic message.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fold INITIAL_ACTIVE into an initially_active candidate field
The hand-maintained list duplicated ALL_CANDIDATES (11 of 12 names) and
had to be kept in sync when adding candidates; forgetting it was silent
since CreateVectorName is FORCE_OFF by default, so a forgotten name got
zero default-soak coverage. Each candidate now declares its activation
inline and the fixture and run() filter on it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The inverse Hadamard rotation is linear, so it distributes over
subtraction: |R'd1 - R'd2| = |R'(d1 - d2)|. Subtract the dequantized
vectors in rotated space and inverse-rotate the difference once,
instead of inverse-rotating both vectors per score call.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Speed up TurboQuant Hadamard rotation by ~3x
Profiling apply_inverse showed 44% of cycles in the Fisher-Yates swap
replay (serial LCG plus a 64-bit modulo per element), not in the WHT.
- Materialize the shared permutations as index maps once in
HadamardRotation::new; each apply-time permutation becomes a flat
gather pass ping-ponging between the vector and a thread-local
scratch buffer. The replay path stays as the cfg(test) parity oracle.
- Fuse consecutive WHT outer butterfly stage pairs (h, 2h) into one
pass over the array (AVX2), halving memory traffic for h >= 16.
- Fuse the normalization multiply into the transform's final-stage
stores (wht_dispatch_scaled), removing the separate normalize pass.
Output is bit-identical on all paths: pinned by the existing
struct-vs-replay and SIMD-vs-scalar bit-equal tests plus a new
wht_dispatch_scaled parity test. NEON is unchanged.
Criterion hadamard bench (Zen 5): apply 2.8-3.7x faster across dims
128-4096 (1024: 6.98us -> 2.03us), apply_inverse 2.8-3.5x
(1024: 6.10us -> 2.05us).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: harden gather contract, small-dim parity, grow-only scratch
- Mark gather_permuted as unsafe fn: the get_unchecked justification relied
on a caller invariant (map indices in range) that the signature did not
surface; document it as a # Safety contract instead.
- Add dims 5 and 50 to static_rotation_matches_struct_and_roundtrips to pin
the map path against the replay oracle on degenerate chunk splits.
- Make the thread-local gather scratch grow-only, so threads alternating
between dims no longer shrink and re-zero the buffer on every call.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Hard-assert input length in apply/apply_inverse
Fail fast at the public boundary: with debug_assert only, a release
build would WHT-normalize a wrong-length slice before the gather's
hard length asserts panic. Flagged by CodeRabbit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
StructPayloadIndex::open took two adjacent bools (is_appendable, create)
and call sites passed every literal combination: (true, true),
(true, false) and (false, true) all exist. A transposed pair compiles
and silently yields e.g. non-appendable + create instead of
appendable + load-only.
The target enum already existed: open immediately converted the bool
into the private StorageType { Appendable, NonAppendable }, so the bool
survived only at the API boundary, exactly where the swap hazard lives.
Make StorageType public, take it directly, and introduce
IndexLoadMode { CreateIfMissing, LoadExisting } for the create flag.
create_segment had the same trailing create: bool with bare literals at
both callers, so its parameter is lifted to IndexLoadMode as well:
load_segment passes LoadExisting, build_segment passes CreateIfMissing.
No behavior change.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(recovery): advance newest clocks for WAL-replay tail on load
`load_from_wal` replays the WAL synchronously up to `to` and queues the
remaining `[to, last_wal_index)` tail to the update worker. Whether that
tail exists depends on how far the persisted `applied_seq` lags the WAL
end, independently of the `prevent_unoptimized` flag's value: the flag only
gates the worker's deferred-points wait, not whether the tail is queued.
The synchronous replay advances `newest_clocks` from each entry's clock tag,
but the queued tail did not: the update worker deserializes only the
operation and discards the clock tag.
As a result the newest-clocks recovery point regressed across a graceful
restart by up to the update-queue size, even though those operations are
durably in the WAL (which is exactly what the recovery point tracks). Worse,
the first post-restart updates would then be assigned clock ticks that
earlier WAL entries already carry for different operations, which corrupts
WAL-delta resolution in a cluster.
Advance `newest_clocks` over the queued tail during load, mirroring the
synchronous replay, before handing it to the worker. The range is
end-exclusive because `last_wal_index` is one past the last entry.
Regression dates to WAL replay honoring `applied_seq` (#8008), which
narrowed synchronous replay from the whole WAL to `[from, to)`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(model): assert clock recovery point survives close+reopen
Capture each shard's newest-clocks recovery point (flattened to
shard_id -> (peer_id, clock_id) -> tick, tokens dropped) on both sides of
every close+reopen in the model testing harness (mid-run restart and final
reload) and assert exact equality.
Both mismatch directions are bugs: a lost tick means clock durability broke,
a gained tick means the reload path over-advanced a clock. The check runs
after the existing model check so a lost WAL tail keeps its established
extra/missing-id postmortem signature, and a clocks-only divergence surfaces
distinctly.
This is what caught the WAL-replay-tail clock regression fixed in the
previous commit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(recovery): tolerate unreadable WAL tail entries in clock advance
Review follow-ups for the deferred-tail clock advance:
- Log and skip a tail entry that fails to read instead of propagating
the error: the update worker tolerates the same failure when it
re-reads the entry, and failing here turns one bad tail record (or
applied_seq/truncation index skew) into a shard, and by default a
node, that cannot start. Add a red-green-verified regression test
that injects an undeserializable record into the deferred tail.
- Fix the pre-existing off-by-one in the send loop: last_wal_index is
one past the last entry, so `to..=last_wal_index` enqueued a phantom
op_num on every restart with a deferred tail.
- Reword new comments (em dashes, and the range-bound note is obsolete
now that both loops use the same exclusive bound; document the
two-pass shape instead: the WAL iterator is not Send across await).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add coverage for TurboQuant datatype in model tester
Co-authored-by: Cursor <cursoragent@cursor.com>
* Address review feedback: tolerance, doc comment, exhaustive match
- Tighten dense_matches Turbo4 tolerance to 16 ulps relative and drop
the absolute floor, so near-zero sign flips and small systematic
quantization drift fail instead of passing
- Fix ALL_CANDIDATES doc comment to match INITIAL_ACTIVE (six names
start active, "c" and "u" via CreateVectorName)
- Make model_vector match exhaustive so new VectorKind variants force
a compile error
- Use explicit DistanceType::from(distance) in turbo_storage_roundtrip
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The write_limit_type: bool parameter of CollectionError::rate_limit_error
relied on a trailing comment to document its meaning (false = read,
true = write). A swapped literal at a call site would compile and report
the wrong limiter type in the error message.
Introduce RateLimiterKind { Read, Write } so the intent is explicit at
call sites and checked at compile time.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Remove 8 `#[allow(clippy::...)]` attributes that no longer suppress any
lint. Each was verified redundant by rewriting it to `#[expect(...)]` and
confirming the workspace stays clippy-clean under the CI config
(`cargo clippy --workspace --all-targets --all-features -- -D warnings`).
Attribute-only deletions, no behavior change.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: replace flush_all sync bool with FlushMode enum
SegmentHolder::flush_all took two adjacent bools (sync, force), and call
sites read as bare literal pairs like flush_all(true, false). Swapping
the arguments compiles and silently changes flush semantics: a swapped
pair at the snapshot site would make snapshots skip flushing entirely
when a background flush is running.
Introduce FlushMode { Sync, Background } for the first parameter so the
pair is no longer transposable and the behavior is named at each call
site. The force flag stays a bool since it feeds the
SegmentEntry::flusher(force) trait in lib/segment. No behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: exhaustive match on FlushMode instead of equality check
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
One call site in segment tests was not updated when
SparseIndexConfig::new gained the Option<Memory> parameter in #9684.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Remove from_iter_instead_of_collect from workspace lints
The lint was removed from clippy (beta) and now triggers
renamed_and_removed_lints warnings in every crate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::chunks_exact_to_as_chunks
Replace chunks_exact with a constant chunk size by as_chunks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::needless_late_init
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::useless_borrows_in_formatting
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::uninlined_format_args
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::for_kv_map
Iterate map values directly instead of discarding keys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_large_err on QueueProxyShard::new_from_version
The Err variant intentionally hands the LocalShard back to the caller.
Same pattern as the existing allow on ForwardProxyShard::new.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_unit_err on wait_for_consensus_commit
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix: unblock optimizer after deleting a named vector
Deleting a named vector could permanently block the config-mismatch
optimizer. The source-superset check in SegmentBuilder::update cancelled
every rebuild that found the deleted vector still in old segment files,
and each retry cancelled again, so optimizations got stuck forever.
Removing the check (as in #9609) would fix delete but reintroduce data
loss for the CreateVectorName race. Instead, tell the two cases apart
with the live collection schema: prune a source vector that is gone from
the schema (a real deletion), but cancel when it is still present (a
freshly created vector this optimizer has not yet seen). This is safe
because the schema is persisted before the op reaches segments, and the
live schema is read after the source segments are frozen.
The live set covers dense and sparse vectors, since a segment stores both
together. When no live source is wired in, the conservative always-cancel
behavior is kept.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: wire live vector names into edge optimizers
Deleting a named vector left the edge path with the pre-fix behavior:
segment_optimizer_config hardcoded live_vector_names to None, so a merge
touching a segment that still carried the deleted vector cancelled, and
EdgeShard::optimize() propagated the cancellation as a hard error forever.
Share the shard config behind an Arc and hand the blocking optimizers a
provider that reads the current vector names on every call. Same safety
argument as the server wiring: update() holds the segments read guard
across both the segment application and the config update, so any name a
frozen source segment carries is visible to the live read.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: share vector-name enumeration via CollectionParams::vector_names
The optimizer's live-schema set and the WAL-recovery valid-name set are
the same dense+sparse enumeration and must stay in lockstep; a drift
between them would reintroduce a wrong prune/cancel decision. Replace
the private helper in optimizers_builder and the inline block in WAL
recovery with a single CollectionParams method.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: drop SegmentOptimizer::live_vector_names forwarding hop
The default trait method only forwarded to the config getter and had a
single caller; ShardOptimizationStrategy now reads the config directly,
removing one layer of indirection.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expand the regression suite to the full {optimizer on/off} x {restarts
on/off} matrix via a shared `smoke` helper, and retain storage on panic
(with seed) through a `StorageGuard` for postmortem repro. Op counts
unified to 8000; the optimizer+restarts cell runs at OP_NUM/4 to avoid
being the suite long pole. Module skipped on Windows (too slow).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* model_testing: add QueryFusion op (prefetch + fusion coverage)
The model tester's Query op only issued a top-level Nearest query, leaving
the Query API's prefetch + fusion path untested. QueryFusion issues 1-3
independent Nearest prefetch sources fused with RRF or DBSF, plus an
optional outer num filter.
Fusion ranking is score-based and approximate, so the oracle is upper-bound
only (same convention as Search/Query): every result must be some prefetch's
candidate and pass the outer filter, and the result size can't exceed the
outer-filtered candidate union capped at limit. Scores and order are not
checked. Single prefetch level (no nesting).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* model_testing: address review on QueryFusion
- assert no duplicate ids before collapsing fusion results into a set, so
an engine that fails to dedup overlapping prefetch hits is caught
- log full prefetch descriptors (vector_name, limit, filter_num) in the
trace instead of just vector names
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: add nightly model_testing workflow
Run the model_testing binary nightly on dev with two sequential passes
(async-scorer and without io_uring), reusing a single built binary.
Note: a temporary pull_request trigger is included to test-drive the
workflow on the PR; it is to be removed before merge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: fix checkout ref for pull_request event
github.ref_name resolves to <pr>/merge on pull_request, which checkout
treats as a branch and fails to fetch. Use the default ref outside the
nightly schedule.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: drop mold/clang linker from model_testing workflow
free-disk-space (large-packages) removes the llvm/clang packages, so the
clang linker used for mold no longer exists on the runner. Build time is
dominated by the long model-testing runs, so the default linker is fine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: random seed, constants, both-run, backtrace, dispatch inputs, failure issue
- Random seed per run (logged for repro), shared by both passes
- OP_NUM/SHARD_COUNT/RESTART_PROBABILITY/ID_POOL as env constants
- no-io_uring run uses if: !cancelled() so both passes always run
- RUST_BACKTRACE=1 for debuggable panics
- workflow_dispatch inputs to override op_num/seed
- rust-cache saves on the nightly (schedule) run, not the never-matching dev ref
- open/update a GitHub issue on scheduled failures
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: run model_testing with 2 shards by default
Exercises the cross-shard reload/WAL-replay path (the 2-shard-only revert
class) instead of single-shard only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: bump model_testing OP_NUM to 200000
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: remove temporary pull_request trigger from model_testing workflow
Test-drive validated on the PR; the nightly now runs only on schedule and
workflow_dispatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(optimizer): defer source segment destruction until durable
Optimizations copy-on-write move points out of their source segments in
memory, and WAL replay can only re-derive those moves from the sources'
on-disk pre-images. Dropping the sources immediately in finish_optimization
is unsafe: the moved copies may still sit unflushed in appendable segments,
so a restart before they are persisted loses the points ("No point with id"
during replay).
Replace the immediate proxy.drop_data() with a generic post-flush action
mechanism on SegmentHolder:
- register_post_flush_action(ready_at, ack_pin, action) queues a retryable
closure (FnMut returning PostFlushOutcome) to run once a flush proves its
data durable.
- flush_all runs every action whose ready_at is covered by the durable
waterline, and caps the returned version (and thus the WAL acknowledge) at
the minimum ack_pin of the actions still pending, so every operation the
not-yet-cleaned data contradicts (deletions in particular) stays replayable
until the files are gone.
- finish_optimization registers each source's drop via register_segment_drop,
pinned at the source's persisted version.
A crash before an action runs loads the old files next to their replacement;
load-time deduplication resolves the overlap.
Robustness:
- LockedSegment::try_drop_data hands the segment back on a StillInUse failure
(data untouched) with a short timeout, so a failed drop is retried on a
later flush instead of leaking its ack pin or blocking the flush for up to
an hour; drop_data keeps its long timeout for callers without a retry path.
- run_ready_post_flush_actions records the ack-pin floor of the actions it is
running (briefly out of the queue) so a concurrent flush on the background
early-return path cannot advance the WAL acknowledge past them.
Optimizer tests that assert source files are gone now flush_all first to run
the deferred action before counting dirs / asserting. New SegmentHolder unit
tests cover the retry, hard-failure, and in-flight-pin-visibility paths.
The perf caveat (a fresh appendable segment reporting persistent_version()
== 0 drags the waterline down) is documented inline as a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* capture knowledge
* Update lib/shard/src/segment_holder/mod.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
Build the multi_thread runtime by hand instead of via the `#[tokio::main]`
macro so the soak test can seed Tokio's internal RNG from `--seed`. Under
`--cfg tokio_unstable` this pins the in-poll random draws (notably `select!`
branch selection), shrinking the space of possible executions for a given
seed. It does not make the scheduler fully deterministic (thread interleaving,
work-stealing timing and I/O readiness still vary), but it removes one source
of variance at near-zero cost; without the flag the binary still builds and
runs unseeded.
Also add `--worker-threads` to optionally pin the worker count (defaults to
the CPU core count), so a `--seed` repro can share the same runtime shape
across machines.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Buffer multi-dense offsets store to prevent reload corruption
The appendable multi-dense storage flushes its `vectors` and `offsets`
chunked stores independently. Each flusher snapshots its `status.len` at
creation time but msyncs chunk bytes at execution time. A re-upsert that
grows a point past its reserved capacity rewrites the point's `offsets`
entry in place to a freshly-appended row region; if that lands in a
flush's creation-to-execution window, the relocated entry becomes durable
while the `vectors` store's recorded length still predates the rows it
references. On reload that point is unreadable and a WAL append reuses the
rows, clobbering another point.
Wrap the offsets store in a write-back buffer (`BufferedOffsets`) so the
durable offsets can never reference rows beyond the durable `vectors`
length. Offset writes stage in a pending overlay and only land in the
durable store while a flush executes; the flusher snapshots the pending
set at creation time, so any write after that stays buffered for the next
flush. Both flushers snapshot at the same instant, yielding a consistent
durable cut. Rows written after the cut are unreferenced garbage the next
append overwrites. This prevents the skew at the source instead of
patching it on reload, and also closes the residual offsets-length smear.
The buffer follows the Gridstore flusher convention: pending writes live
inside the single lock-guarded store, reads consult the overlay then the
durable bytes under one lock, and the durable msync runs after releasing
the write lock so it never stalls concurrent reads.
Enable the "m" multivector in the collection model test now that its
reload divergence is resolved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Narrow offsets flush write-lock scope
Build and sort the pending snapshot before taking the write lock; only the
apply + reconcile need it. Shortens the lock hold so concurrent reads block
less. Addresses review feedback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf: use Entry API to avoid redundant map double-lookups
Replace get_mut/contains_key followed by insert with the entry API
across several maps, collapsing two hash lookups into one. Limited to
sites where the key is Copy or already owned and moved, so no extra
key clone is added to any hot path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* More Entry API usage in mutable_geo_index
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
A DropIndex (or vector name deletion) landing between flusher capture
and execution drops the component's storage, making its captured
flusher return Cancelled. Aborting the rest of the flush sequence at
that point leaves the components flushed so far durably ahead of
payload storage and point versions. WAL replay then re-derives
filter-based operations through the too-new field index and silently
skips points whose payload still needs the operation re-applied,
losing the update.
Skip the dropped storage and keep flushing instead. The drop itself is
a versioned operation that WAL replay re-applies, so skipping it is
safe.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`delete_vector_name` removes a vector's storage/index directories, but that
removal is best-effort: it logs a warning and continues on failure (and a crash
mid-delete leaves them too), so the directories can still be on disk after a
delete.
When the same name is later recreated, `create_vector_name` reopened whatever
was on disk: `open_vector_storage` reuses existing data and
`prefill_deleted_entries` only pads missing entries, so a point that was never
re-upserted silently regained its old vector after reload.
Remove any leftover storage/index directories in `create_vector_name_impl`
before opening, so a recreated name always starts empty regardless of whether
the earlier delete fully removed them.
Adds deterministic regression tests (dense + sparse) that reproduce the
resurrection by restoring the storage directory after the delete, standing in
for leftover files the delete did not remove.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(gridstore): honor pending pointer-unset in batched reads
`Tracker::get_batch` only handled the `Some` arm of a point's pending
pointer update. When a point had a pending pointer-unset (`current ==
None`) over a value already flushed to disk, the batched lookup fell
through to the stale persisted pointer, so `read_values` resurrected the
deleted value while the single-read `get`/`get_value` correctly reported
it gone.
This split the payload read paths: `retrieve`-by-id (batched
`read_payloads`) returned a stale payload while `scroll`/filter (single
`get_value`) returned the correct empty one — surfacing as a payload
mismatch after a filter-driven clear.
Honor the pending update either way in `get_batch`, mirroring `get`.
Add a regression test covering put+flush then delete-without-flush; the
pre-existing congruence test missed it by only deleting never-flushed
points (no stale persisted pointer).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sigh
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Fix proxy deleted_mask race dropping live points from scored search
ProxySegment::new snapshots the wrapped segment's deleted_mask while the
optimizer holds only the upgradable-read lock, before the write lock freezes
the segment. An upsert racing onto the still-appendable wrapped segment in that
window lands at an internal offset past the snapshot. The scored search path
(PlainVectorIndex::search) consults the proxy mask in place of the segment's
live deleted state, and check_deleted_condition defaults any out-of-range
offset to deleted (unwrap_or(true)) — so the live point is silently dropped
from filtered KNN while scroll/count/retrieve still return it.
Re-snapshot deleted_mask in the optimizer once the holder write lock is held
(segment frozen) and before the proxy goes live, so the mask covers the
segment's full final point range. The fresh read also captures any deletes
that raced in, closing the ghost direction too.
Adds a proxy-level regression test that reproduces the race without the
model-testing harness.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Make proxy deleted_mask sync a type-state, read once
ProxySegment::new now returns UnsyncedProxySegment instead of a
ready-to-use ProxySegment. The deleted_mask snapshot is deferred to
UnsyncedProxySegment::finalize(), which reads the wrapped segment's
deleted bitvec exactly once. The only way to obtain a ProxySegment
(and thus put it in a SegmentHolder) is via finalize(), so the sync
under the holder write lock can no longer be forgotten, and the mask
is no longer read twice (once in new, once in resync).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Allow clippy::new_ret_no_self on ProxySegment::new
new deliberately returns the unsynced UnsyncedProxySegment stage rather
than Self, since a usable ProxySegment only exists once deleted_mask is
synced via finalize().
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Name the real constructor UnsyncedProxySegment::new, keep ProxySegment::new for tests
Instead of allowing clippy::new_ret_no_self on a ProxySegment::new that
returned UnsyncedProxySegment, give the two-phase constructor its natural
home: UnsyncedProxySegment::new returns Self and is what production code
(optimize, snapshot) uses, finalizing under the holder write lock.
ProxySegment::new becomes a #[cfg(feature = "testing")] convenience that
builds and finalizes in one step (returns Self), so existing test call
sites stay terse and don't need an explicit .finalize(). The shard
testing feature is enabled for both shard's own tests and collection's
dev-dependency, and excluded from production builds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* fix: serialize optimization proxy install against shard updates
execute_optimization captures `target_config` from the optimizer's frozen
config and then wraps source segments in proxies. Between those two points,
`CollectionUpdater::update` can apply a `CreateVectorName(V)` to the source
segments via `apply_segments`, leaving the optimizer with sources that have
V but a target_config that does not. The optimization then produces a
merged segment without V, and a follow-up optimization (running with the
refreshed config that includes V) fails to use that segment as a source:
"Cannot update from other segment because it is missing vector name X".
Close the race by extending the scope of the existing
`LockedSegmentHolder::acquire_updates_lock` to cover the proxy install
window. `CollectionUpdater::update` already takes this lock before
processing any shard update, so concurrent writers wait until proxies are
in place — at which point further mutations hit the proxies (recorded as
intent and propagated to the merged segment in `finish_optimization`)
instead of the originals. The guard is dropped right after proxy install so
the slow build phase does not extend it.
Tests:
- Three `SegmentBuilder::update` tests document the precondition the lock
now guarantees: with a target schema that adds a named vector the source
lacks, update errors with "missing vector name X". Quantized and
mixed-source variants exercise the same error path.
- `test_optimize_blocks_proxy_install_on_updates_lock` asserts the
invariant directly: while the updates lock is held, proxies are not yet
installed. Verified to fail when the new guard is removed (otherwise it
passes because `finish_optimization` also takes the same lock, so a
naive "did optimize finish?" check would not catch a missing proxy-install
guard).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(optimize): finish_optimization lock order; drop redundant tests
Address review of #9110:
1. `finish_optimization` was acquiring `upgradable_read` before
`acquire_updates_lock`, while the new guard at the start of
`execute_optimization` acquires them in the reverse order. With two
optimizer threads in flight, thread A in `finish_optimization` could
hold `upgradable_read` and wait on `updates_lock` while thread B at the
top of `execute_optimization` held `updates_lock` and waited on
`upgradable_read` (parking_lot allows only one upgradable reader),
deadlocking. Swap `finish_optimization` to take `updates_lock` first so
both halves agree.
2. Drop the quantized and mixed-source variants of the inverted
`SegmentBuilder` unit test — all three asserted the same error path
(the mismatch check fires before quantization training or per-source
branching), so only one is useful as documentation of the precondition.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: reject source-superset schema mismatch at SegmentBuilder
Drop the lock approach (deadlocked test_continuous_snapshot) and fix the bug
at the merge layer instead.
Snapshot's proxy_all_segments_and_apply acquires the segment_holder
upgradable_read first and then takes acquire_updates_lock tactically inside
the snapshot operation. The previous commits' lock-extension acquired
updates_lock before upgradable_read, so a snapshot in flight and an
optimization just entering execute_optimization could deadlock holding each
other's required next lock. Snapshot cannot easily reverse its order — that
would hold updates_lock for the entire snapshot duration, blocking all
writes.
Move the fix to where the actual harm happens: SegmentBuilder::update
iterates the target's vector_data and silently drops source vectors that
aren't in target. That silent drop is what produces the broken merged
segment in the CreateVectorName-vs-optimizer race. Add a check that every
source vector name is in the target schema; the optimization aborts cleanly
on mismatch and the next round (with refreshed config) merges correctly.
This is strictly stronger than the lock: the lock only closed the window
where V arrived *during* the proxy-install region. The schema check catches
both that window and the window where V's apply_segments completed before
the optimizer's lock acquisition.
Diff is contained to lib/segment; no locking changes, no cross-crate
plumbing. test_continuous_snapshot passes again.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(segment_builder): use Cancelled instead of ServiceError for schema mismatch
ServiceError flips the shard to RED status (via `report_optimizer_error` →
`segments.optimizer_errors`) and stays sticky until the next
`recreate_optimizers_blocking` clears it. That's the right shape for
hardware/IO failures but wrong for the schema-mismatch case here, which is
an expected, recoverable race outcome — the next optimizer round with a
refreshed target_config merges the same originals cleanly.
`Cancelled` is the variant the optimization worker treats as a recoverable
cancellation: logged at debug, tracker marked Cancelled, no
`report_optimizer_error` call, no RED status.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(segment_builder): also use Cancelled for the existing target-superset error
The existing "missing vector name" check at the start of the merge loop
also fires during a race — specifically the optimizer-vs-DeleteVectorName
shape, where V is removed from originals before J wraps proxies but J's
frozen target_config still has V. Like the new source-superset check, this
is an expected, recoverable race outcome, so use Cancelled instead of
ServiceError to avoid flipping the shard to RED.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(segment): immutable map index skips values with no live points on load
When MmapMapIndex::open ORs the id-tracker's runtime deletion bitvec
into the on-disk one at open time, ImmutableMapIndex::open_mmap could
insert a zero-count entry into value_to_points for any value whose live
points were all deleted, then immediately trip its own post-build sort
assert. Skip such values, mirroring the runtime invariant maintained by
remove_idx_from_value_list which already removes entries when their
count drops to zero.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(segment): file-system immutability test for payload indices
Builds an immutable segment with all 8 PayloadSchemaType variants
indexed, snapshots every byte under payload_index/, then asserts
byte-for-byte equality plus per-field query correctness after
delete_point, flush, drop+reload, and a second deletes+flush on the
reloaded segment. Each query exercises a different read path of an
immutable index variant: map exact-match (keyword/uuid/integer),
numeric range (float), datetime range, geo bounding box, full-text
token match, bool match. Reproduces the regression fixed in the
previous commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
