* strip fs from id tracker
* strip fs from Gridstore and Logstore
* strip fs from UpdateOnlyBlobstore
* strip fs out of null and bool indexes
* strip fs out of chunked vectors
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Remove MessagePack (rmp-serde)
The WAL switched from msgpack to CBOR in v0.3.5 (2021-07-11), so v0.3.4
is the last version that wrote msgpack entries. Drop the read fallback
kept for those entries, plus the remaining test and bench usages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Drop unused fs4 dependency from collection
Not referenced anywhere in the crate. Still used by wal and common, so
the workspace entry stays.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
- 'mising' -> 'missing' in .github/review-rules.md (flagged by the
repo's own codespell config)
- 'Do not create segments larger this size' -> 'larger than this size'
in config.yaml, the optimizer builder/diff sources and the grpc
proto + generated rust comment
- 'bigger then' -> 'bigger than' in the query scorer rustdoc
- three duplicated-word rustdoc fixes ('override in in', 'any of of',
'and and')
* Direct call for shard transfer method and keys
* Reuse cardinality estimate in sparse plain search
* Avoid recounting available points in segment size info
* Avoid cloning segment config when updating quantization
* Avoid cloning search request for load profile
* Direct call for counting read-only segments
* Avoid re-reading point range for values count
* Direct call to check replica states when initializing collection
* Direct call to look up transfer on restart
* Direct call for shard replicas after snapshot recovery
* Direct call for local replica states in health check
* Direct call for payload index schema keys when applying state
* Direct calls for sharding method and key mapping when creating shard key
* Direct call to check if peer has shards
* Direct call for sharding method and keys when dropping shard key
* Avoid cloning collection params for group by ordering
* Avoid cloning collection params in local shard search
* Direct call for peer address when sending Raft messages
* Direct call for peer address in who_is
* Avoid cloning remote query batch request
* Avoid cloning operation in queue proxy update
* Avoid cloning gRPC search groups request
* Fetch cluster status once in cluster telemetry
* Direct call to validate transfer exists on finish
* Direct call for sharding method when dropping shard key
* Avoid cloning peer address map when listing peers
* Avoid cloning peer address map when adding peer to known
* Avoid cloning shard key mapping when routing writes with fallback
* Avoid cloning shard key mapping when checking resharding start
* Avoid cloning gRPC recommend groups request
* Avoid cloning operation when retaining forwarded point IDs
* Direct call for counting collections in telemetry
* Direct call to validate transfer exists on recovery
* Direct call for shard IDs by shard key
* Direct call for shard keys
* Direct call to check if peer has shards in consensus
* Direct call for replica state on transfer recovery
* Direct call to check for active replicas when routing writes with fallback
* Direct call to validate transfer exists on abort
- `TurboVectorStorageImpl::{insert_vector, insert_tq_bytes}`: return
error right away. `QuantizedStorage::upsert_vector` will return error
anyway, so why pretend it will not. Bonus: `quantization_buffer`
no longer needed. Related: #9953.
- `DenseVectorStorageImpl::insert_vector`: `Err`, not `panic!`. For
consistency with `TurboVectorStorageImpl`. It's unreachable anyway.
- `QuantizedStorage::reload`: reopen via `self.storage.live_reload()`.
The same, but now we don't need to keep `fs`.
These were added for named-vector CRUD (acfb650), never used. The actual
placeholder storages `EmptyDenseVectorStorage` and
`EmptySparseVectorStorage` stay.
* Fall back to per-vector TQ scoring when runs are short
Run-batched scoring pays off on plain and dense filtered scans but
regresses HNSW, where neighbor ids rarely form consecutive runs and batch
setup dominates. Gate both EncodedVectorsTQ::score_points and Turbo
score_query_batch on offsets_worth_batch_scoring, which takes the run path
only when the ids split into runs averaging BATCH_SCORE_MIN_MEAN_RUN
vectors or more.
The average decides, not the longest run: a sorted id list -- what a
filtered scan hands the scorer -- already contains adjacent pairs at ~1%
density while its runs still average one vector, so a "contains a run of
>= 2" test sends those down the run path to pay setup per vector, measured
at up to +75% against the better path. An average also stays independent
of the batch size the driver slices ids into, which a longest-run test does
not. The threshold comes from the measured crossover -- mean run 2.0-3.6,
stable across dims 128/512/1024, both RAM storages and the 1/2/4-bit
widths -- and a fully contiguous block is recognized in O(1), so a plain
scan pays nothing for the gate.
* io_uring: never gate run-batched scoring
The gate exists because run batching costs setup that short runs do not
repay on RAM and mmap storages. io_uring is the opposite: one run-granular
read beats the batched per-vector path at every density measured -- 27% on
HNSW-shaped id lists, 43% at 25% filter density, 93% on a full scan --
because per-request submission and completion bookkeeping dominates once
the data sits in the page cache. Gating it costs 36% on HNSW-shaped lists.
Add EncodedStorage::prefers_run_reads, defaulting to false so every storage
keeps its current routing, and override it for the single-file quantized
storage when its backend is io_uring. Remote backends (object stores, a
gRPC peer) deliberately keep the per-vector path: their reads pipeline
across a batch, while run-granular reads would serialize the round trips.
QuantizedStorage::is_in_ram_or_mmap() still reports true for every backend,
which is what routes io_uring into the gate in the first place. Correcting
that would also change how the multivector storage picks between its
in-memory and uring scoring paths, so it is left to a separate change.
* Rename prefers_run_reads to prefers_contiguous_reads
"Run reads" is easy to misread as "execute reads"; contiguous makes the
storage I/O preference explicit.
* QuantizedStorage::for_each_run: pipeline run reads on async backends
With `prefers_contiguous_reads()` true for io_uring, every batch goes
through `for_each_run`, which read each run synchronously: a scattered
id list (HNSW neighbours) waited on one disk read per vector, where
`for_each_in_batch` kept the whole batch in flight through `read_batch`.
Submit all runs of a batch together, still one read per run, so
scattered reads stay pipelined while a scan still reads each run in one
request. Backends without async reads keep the sequential loop.
`turbo_vector_search` (dim 1024, 200k vectors, 4096 shuffled ids per
iteration) against dev: cold scattered io_uring 187 ms -> 26.5 ms
(dev 29.5 ms); the warm scan keeps 198 ms -> 21 ms. Warm scattered
lands at 5.06 ms (dev 4.67 ms), giving up the 3.68 ms of synchronous
reads, which only holds with the data already in the page cache.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRiRaiPsm5CEBgpQ7VAQab
* Iterate consecutive runs and share the run-scoring gate
`for_each_consecutive_run` becomes `consecutive_runs()`, a lazy iterator
over `Run { first, start, len }`, so a storage can feed runs straight into
a read pipeline. `QuantizedStorage::for_each_run` loses the intermediate
`Vec` and the duplicated `ReadRange` construction: the pipelined branch
maps the iterator into `read_batch`, the synchronous branch keeps the
per-run `Sequential`/`Random` hint that picks between mmap's two mappings.
The routing condition duplicated at both scoring call sites moves into
`EncodedStorage::prefers_run_scoring`: same expression, one place.
`for_each_run`'s contract no longer promises run order: pipelined
backends report reads as they complete, so callers address results by
`first`. Add a contract test over the mmap and disk-cache backends; the
latter is the async-capable backend that runs on every platform and
covers the `read_batch` branch io_uring takes on Linux, which no test
exercised before.
Measured on Apple M3 against 1f2d1264e, interleaved A/B/B/A: the run
path is unchanged on all four storages (-0.3%, +0.0%, +0.9%, -1.1%,
within replicate noise).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Rename Run to ConsecutiveRun
`Run` on its own reads as "execute", the same ambiguity that got
`prefers_run_reads` renamed earlier in this branch. `ConsecutiveRun`
names what the value is and pairs with `consecutive_runs()`, the iterator
that yields it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* io_uring pipeline: submit eagerly only while reads are outstanding
`IoUringPipeline::wait()` called `submit_and_wait(0)` whenever a completion
was ready and anything was enqueued. `read_batch` enqueues one entry per
consumed completion, so on a warm page cache — where reads complete inline
at submission — that was one `io_uring_enter` per read. Scattered quantized
scoring over io_uring ran at ~630 ns/point warm against ~460 for the same
reads issued synchronously (which, without direct_io, are plain `pread`s).
Now the eager submission happens only while the kernel still has reads
outstanding (`in_progress` minus the completions already waiting in the
queue). When everything submitted so far has completed — the warm case —
the enqueued entries wait and go down together once the ready completions
run out. On a cold device nothing changes: a completion is answered with a
submission as before, so the in-flight depth never sags. Two fixed rules
tried first (submit only when nothing is ready; submit once half the queue
piled up) both cost the cold path, +6 % and +3 %, in proportion to how long
the device sat idle while ready completions were drained.
turbo_vector_search / turbo_uring_ab, Zen 4, `taskset -c 7`, prebuilt
binaries run alternately, cold rows with the page cache dropped:
warm scattered, uring hnsw: 634 -> 472 ns/point (-25 %)
warm scattered, uring p0.25: 477 -> 369 ns/point (-23 %)
warm sequential, uring p1.00: 45 -> 45 (flat)
cold scattered, uring: 29.9 -> 29.9 ms/iter (flat, 4 reps each within 0.7 %)
mmap rows (control): within ±2 %
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix: reject mismatched dense dims in recommend average
Stop silently truncating oversized negative examples during
average_vector merge. Validate dense dimensions within each example
group and between positive/negative averages before zip-merge.
Fixes#10369
* Simplify: keep only the merge-time dimension check
The zip truncation in merge_positive_and_negative_avg is the only place
an oversized negative can silently pass the downstream dimension check;
within-group mismatches already grow the average to the max length and
fail the segment-entry check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style(query): make recommendation conversion explicit
* test: assert recommendation dimension errors
Issue: #10369
Make the regression test verify the exact WrongVectorDimension payload for mismatched recommendation vectors.
---------
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* TurboQuantizer::score_precomputed_batch: score a contiguous run of vectors
Batch counterpart of `score_precomputed` for vectors stored back to
back at `quantized_size()`: the width's kernel scores the whole run of
codes in one `dotprod_batch` call, then a second pass applies each
vector's extras. L1 dequantizes per vector and stays a plain loop.
Tested against per-vector `score_precomputed` for every width,
distance, and mode over run lengths that leave every group remainder.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* EncodedStorage::for_each_run: serve consecutive offsets as contiguous runs
`for_each_run(offsets, callback(first, count, bytes))` splits the
offsets into maximal runs of consecutive ids the storage can serve
from one contiguous slice, so a sequential scan resolves chunk lookups
and reads once per run instead of once per vector. The default serves
every vector as its own run; `for_each_consecutive_run` is the shared
run detection for storages that override it, with a per-run cap for
chunk boundaries. The test storage overrides it (its data is one flat
buffer).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* EncodedVectors::score_points: batched scoring entry point, run-batched for TQ
`score_points(query, offsets, scores)` scores a batch of points. The
default keeps the per-vector loop the scorers run today, so SQ/PQ/BQ
are unchanged. TurboQuant overrides it: on RAM/mmap storages it walks
`for_each_run` and scores each contiguous run with one
`score_precomputed_batch` call, hoisting the score inversion out of
the loop; backends with async reads keep the pipelined per-vector
path. Non-consecutive offsets degrade to single-vector runs, so
scattered access keeps its previous cost.
Integration test: `score_points` vs `score_point` for every bit width
and mode, Dot and inverted L2, over sequential, scattered and
descending id orders.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Quantized storages: for_each_run over their contiguous regions
The RAM storage and both chunked mmap storages cap runs at their chunk
boundary and serve each run with one `get_many`; the single-file mmap
storage serves any run as one sequential read. Unit test on the RAM
storage: runs cover every offset once, in order, with bytes identical
to per-point reads, across the internal chunk boundary.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* QuantizedQueryScorer: score batches through EncodedVectors::score_points
Routes `score_stored_batch` through the batched entry point, so
TurboQuant-as-quantization scans score contiguous runs with one kernel
call per run; SQ/PQ/BQ keep the per-vector loop via the default.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* TurboScoring::score_query_batch: run-batched scoring for Turbo4 storages
Adds the batch counterpart of `score_query_bytes` to the trait, with
one shared implementation over the storage's `EncodedStorage`:
consecutive ids are coalesced into contiguous runs, each run scored by
a single `score_precomputed_batch` call, and the metric sign applied
once over the batch. Backends with async reads keep the pipelined
per-vector path. `TurboQueryScorer::score_stored_batch` now calls it.
The batch-vs-single storage test grows to 8192 vectors so a full
ascending scan crosses a chunk boundary of the chunked backend, and
runs that scan on the chunked, mmap and io_uring backends.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* score_precomputed_batch: keep the extras pass in L1
The kernel pass and the extras pass now alternate over sub-runs of 64
vectors instead of each covering the whole run: for a run of several
hundred vectors the second pass otherwise refetched every vector's
extras from L2. Measured with 512-vector runs from the full-scan
driver at dim 512: the regression against 64-vector runs went from
+11 % to +2 %.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Bench: exhaustive search over Turbo4 storages through the plain-index driver
`turbo4_full_scan` runs `BatchFilteredSearcher::peek_top_visible` —
the exact path of a non-indexed search — over 200k normalized random
vectors for Turbo4 as datatype (appendable chunked, in RAM) and Turbo4
as quantization (over a RAM dense storage), at dims 64 to 1024, so the
fixed per-point cost of the scan driver is measured next to the kernel.
`TURBO_SCAN_DIMS=64,128` narrows the dims while iterating.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Split async IO into extension traits; only async-capable backends implement them
Move `read_bytes_async` / `open_async` off the universal `UniversalRead` /
`UniversalReadFs` traits into dedicated extension traits, `UniversalReadAsync`
and `UniversalReadFsAsync` (traits/async_io.rs). Only backends with a genuine
async story implement them — the blob family, the disk caches layered over it,
and a trivial ready-impl for mmap (tests and the mmap lookup path) — each in a
dedicated async_io.rs next to its sync impl.
`CachedFs` now requires its inner filesystem to be `UniversalReadFsAsync`; the
requirement reaches segment code through one supertrait bound on
`UniversalReadExt`. io_uring implements no async surface anymore: the
tokio_uring bridge thread, its tests, the musl-gated tokio-uring dependency,
and the `IoUringFile` read-only-segment wiring (`UniversalReadExt` impl and
the *RoIoUring condition-checker variants) are deleted — io_uring is not a
read-only-segment backend.
The payoff for live reload: `CachedFs::resolve_prefetched` awaits every parked
prefetch, and the edge refresh flow now runs preload -> resolve -> reload, so
the per-segment write locks never wait on IO.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Decouple UniversalReadExt from the async filesystem requirement
UniversalReadExt is condition-checker dispatch; it never consumed the async
surface itself. Drop its `Fs: UniversalReadFsAsync` supertrait bound and relax
CachedFs's struct-level bound back to `UniversalReadFs` — the async requirement
now lives on the one impl that consumes it, `CachedReadFs for CachedFs`
(schedule_open parks the inner filesystem's `open_async` futures).
The bound then surfaces only on the lifecycle/preload impl blocks that go
through CachedReadFs (segment open, live-preload/reload, config reload, edge
load/refresh); the search path carries no async bounds at all.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* existing segments: wait for IO outside of search pool
* new segments: wait for IO outside of search pool
* extract reload into separate function
* Update lib/edge/Cargo.toml
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* rename `reopen`->`live_reload` and `schedule_reopen`->`live_preload`
* `UniversalRead::live_preload` returns a shared future
* assert snapshot-miss eagerly on `live_preload`
`live_reload` cannot see the failed preload: its blocking fallback
re-resolves the length from the remote and succeeds. The error
surfaces at preload time, as callers (`ok_not_found`) expect.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [CachedFs] new `schedule` and `wait_all` primitives
* [AppendableIdTracker] don't reopen if just opened
* eager NotFound in `schedule_open`
* add traces for async reads
* finish `preopen`/`preload` with `wait_all`
* lock all segments in parallel for `live_reload`
* LIST before everything
to do: we don't have whole-fetch in async mode. to prevent sequential
`len`, we won't overlap static files with LIST.
* `wait_all` returns nothing
* `UniversalReadFs::open_async`
* `schedule_open` polls once
Scheduled opens must start eagerly: sync backends complete their
`open_async` on the first poll, preserving the prefetch contract
(handles outlive later file deletions/replacements). Moved down from
the integration branch so this PR stays green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Read vector runs that straddle a chunk boundary
Resolve a run into per-chunk parts instead of a single range, borrowing
when it lands in one chunk and copying when it spans two. The read
pipeline schedules one range per read, so a straddling run is read
outside it.
No writer produces such a run yet, so this changes nothing on its own.
It is what a reader needs before one does — including edge and
live-reload readers, which read files a different version wrote.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Place multivector runs without regard to chunk boundaries
Writers appended a multivector's inner vectors at the end of the row
space unless the run would cross a chunk boundary, in which case they
skipped the chunk tail — the batch writers padding the skipped rows with
explicit zero rows. That made chunk geometry part of the interface every
multivector storage had to reuse.
Runs now go at the end unconditionally and the chunked storage splits
the write across chunks, as it already did for a batch of single
vectors.
What is left of the geometry is a size cap: a multivector may not exceed
one chunk. It is fill-independent, so it constrains nothing about
placement, and it is what the volatile storage needs anyway — that one
returns a plain slice and so cannot serve a straddling run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Split a run at chunk boundaries in one place
Reading, writing in place and appending each derived the split from
`remaining_chunk_capacity`, so every one of them had to know that a run
does not necessarily fit where it starts.
`split_run` hands out the parts instead: one per chunk the run covers,
each carrying where it goes and how much of the run it takes. Nothing
asks how much room is left any more, and `get_chunk_offset` goes with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Keep straddling runs on the read pipeline
Reading a straddling run outside the pipeline blocked the scheduling
loop on one read, which costs a round trip on a backend that fetches
remotely and drops the batch back to sequential.
A run is now scheduled as one read per chunk it covers. Parts complete
in any order, so each run holds what has landed until the last part
does, then hands the callback the stitched vectors. Runs taking a single
read carry the caller's data in the tag and never touch that table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Stop capping a multivector at one chunk
The cap outlived its reason on disk, but the volatile storage still
needed it: its `get_many` handed out a slice of one chunk, so a run that
crossed a boundary had nowhere to come from. And since a volatile
storage is a target of the batched copy that builds a segment, dropping
the cap only on disk would have turned a rejected write into a failed
merge.
So the volatile storage splits and stitches too. Both are a few lines
each, and placing a run no longer skips a chunk tail, so `extend` is now
`insert_many` at the end of the storage.
Nothing user-facing moves: `MAX_MULTIVECTOR_FLATTENED_LEN` caps a
multivector at 1M elements, far inside a 32 MiB chunk, so the storages
only ever rejected what reached them unvalidated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Schedule a single-read run without the queue
The scheduling loop resolved every run into the queue and then took it
straight back out, so the overwhelmingly common run — one that fits a
chunk — paid a push and a pop for nothing. It now goes to the pipeline
directly, and the queue holds only what a straddling run leaves behind.
Worth ~10% on the multivector read benchmark, and it collapses the
"top up, then take" pair into one decision. Extracting that bookkeeping
into helpers instead was measured and is much worse: the mmap pipeline
alternates one schedule with one wait, so the loop body is a few dozen
nanoseconds, and a helper carrying the cold map and stitching paths is
too big for the compiler to inline back into it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Repoint the multivector WAL-replay test at a live rejection
The test upserted a multivector too large for a storage chunk, which no
longer fails: the storages stopped capping one at a chunk. Nothing else
covered a multivector operation that only the apply path rejects.
A raw blob that is not a whole number of quantized records still does,
so the test now uses that, alongside its dense and sparse siblings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Test reading multivectors with legacy chunk-tail padding
Locks the compatibility contract that pre-straddle files — runs that
skip a chunk's leftover slots — still reopen as single-chunk borrows.
* chore: retrigger CI after flaky test-consensus-compose
* Move ReadTag into for_each_vector, its only user
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AN4Hgbd65gDhesthJk5bUY
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
Unindexed Match::Text and Match::Phrase previously shared a String::contains
arm, so phrase order was ignored and queries matched across token boundaries.
Use the default Word tokenizer for best-effort parity with indexed fields.
Fixes#10182
Add stop checks during the pre-HNSW build setup phase so cancellation
is observed promptly, and widen timing tolerance for noisy Windows debug
builds where post-stop delays can exceed 1s.
* Drop obsolete clippy large-error-threshold override
The 256 threshold was pinned for clippy 1.87 while tonic's `Status` was a
large error type. Upstream boxed its contents in `5de7bad` (hyperium/tonic#2253),
which is in the pinned 0.14.6 fork, so `Status` is now a single `Box` and the
default threshold of 128 passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Remove stale clippy allows
These 11 allows no longer suppress anything under any of the three CI clippy
configurations (default, --all-targets, --all-targets --all-features).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drops 6 crates from the release build and 7 from the workspace test
build, with no source changes.
- geo: no triangulation, only Contains/Intersects/Haversine (spade, earcut)
- jsonwebtoken: HS256 from_secret only, no PEM keys (pem, simple_asn1)
- tar: nothing sets unpack_xattrs, which defaults to false (xattr)
- duplicate: every duplicate_item names its module (proc-macro2-diagnostics)
- pprof: no C++ frames to demangle (cpp_demangle)
Also promotes duplicate to a workspace dependency.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf: skip external-id resolution in search post-processing
process_search_result hands its scored points to retrieve(), which resolves
every external id back into an internal offset — although the offsets are
already known there: they come straight from the vector index as
ScoredPointOffset. Every request therefore performs top_k x segments
redundant external->internal lookups.
Whether that costs anything depends on the id tracker being mutable: a
freshly optimized segment looks ids up through a BTree, while a restarted
node maps an immutable tracker and resolves in constant time. So the effect
shows up right after ingestion or optimization and disappears after a
restart, which is what makes it easy to miss — a benchmark that starts from
a freshly booted node never sees it.
Split retrieve() into resolution + retrieve_resolved() and pass the offsets
from process_search_result directly, applying the deferred cutoff by offset
instead of resolving ids just to filter them. retrieve() behaviour is
unchanged for all other callers; retrieve_resolved() is private and states
that deferred filtering is the caller's responsibility.
Measured on glove-100-angular (1.18M points, 21 segments, top_k=10) at a
fixed request rate, on a node that had just finished ingesting: median
latency ~15-24% lower, ~8% less CPU per request. Both figures come from the
same node before and after the change.
* review fix
* review fix
---------
Co-authored-by: Ivan Dashchinskiy <iadashchinskiy@sbertech.ru>
* Optimization: Skipping items before pushing into PriorityQueue.
* Apply suggestion from top-k-update branch
* Stabilize order of equal scored items in tests
* Replace cgroups-rs with direct cgroup memory file reads
We used cgroups-rs in exactly one place, to read the memory limit and
usage of our own cgroup, so read those files directly instead. Drops 34
crates from the lockfile, including the zbus stack that carries
RUSTSEC-2026-0221.
Also fixes two latent cgroup v1 bugs (the LONG_MAX unlimited sentinel
reported ~9 EB of total memory, an unreadable limit file reported 0
bytes) and the hierarchy mix-up on hybrid hosts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Decline cgroup memory reporting when the usage read fails
Reporting a usage of 0 made available_memory_bytes claim the whole cgroup
limit as free. Fall back to sysinfo when the usage file cannot be read at
init, and keep the last known value on a failed refresh.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Keep the last known memory limit when its read fails
A transient read failure cleared the cached limit and silently fell back
to host memory while the process was still capped, the same direction of
over-reporting as the usage read. Both now keep their last known value,
and a limit lifted at runtime still clears.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Treat a malformed memory limit as an error, not as unlimited
Parse failures returned Ok(None), so garbage in the limit file cleared a
valid cached limit on refresh and read as unlimited at init. Reserve
Ok(None) for "max" and the v1 sentinel, and report anything else as
InvalidData so the last known limit survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up to #10287, which added `max`. Expressing a minimum still
required spelling out `(a + b - |a - b|) / 2`, the sign flip of the max
identity — drop the `neg` and you silently get a maximum instead. It also
only works for two operands and mentions each one twice, so the scorer
walks every sub-tree twice per candidate point.
The pair is what makes clamping expressible:
{"max": [0.0, {"min": [1.0, "$score"]}]}
`min` mirrors `max` throughout, and both guard helpers introduced in
#10287 already took an `operator: &str`, so they are reused unchanged: an
empty operand list is rejected at parse time rather than folding to
+infinity, and the Edge FFI rejects it at construction time. The result
needs no `is_finite` check, since `min` cannot produce a non-finite value
from finite inputs.
The unindexed-field walker shares one arm for `Max | Min` as the bodies
are identical, with a test pinning `min` separately so a later split
cannot silently drop it.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
11 names across 7 files. Renames that did not reach the comment above them
(further_searches for further_results, query_context for segment_query_context,
block_ranges for local_block_ranges, op for operation twice, request for
requests, max_threads for max_kmeans_threads), and 3 arguments that were
removed from a signature and left documented (is_on_disk, collection_params,
search_runtime_handle with timeout).
Documentation only, no behaviour change.
* feat: add a dedicated max operator to score formulas
Expressing a maximum in a score formula required spelling out the
arithmetic identity `(a + b + |a - b|) / 2`. That is easy to get wrong
(the `/ 2` is load-bearing), only works for two operands, and mentions
each operand twice, so the scorer evaluates every sub-tree twice per
candidate point.
`max` is variadic, mirroring `sum` and `mult`:
{"max": ["$score", {"mult": [0.5, "popularity"]}]}
Unlike `sum` and `mult`, `max` has no identity element for the empty
case, so an empty operand list is rejected at parse time rather than
folding to -infinity and scoring every point with a non-finite value.
The check lives in `ExpressionInternal::parse_and_convert`, which every
entry point passes through, and the Edge FFI additionally rejects it at
construction time to match how that crate validates elsewhere.
The result needs no `is_finite` check: unlike `log10`, `exp`, `div`,
`sqrt` and `pow`, `max` cannot produce a non-finite value from finite
inputs, so it follows the existing `sum` convention.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: cover max error propagation and datetime operands
An operand that fails must fail the whole expression rather than being
passed over in favour of a finite sibling. Covered with the failure both
before and after the finite operand: `mult` short-circuits on zero and
so can skip evaluating later operands, and this pins down that `max`
must not grow a similar shortcut that would swallow an error.
Also covers `max` over datetime operands, which reach the scorer through
a separate conversion to seconds, so that "score by whichever timestamp
is newer" is verified rather than assumed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* impl live_preload for payload indexes
enable live_preload for bool and null indexes
* (not) impl live_preload for `VectorIndexReadEnum`
* impl live_preload for `ReadOnlyPayloadStorage`
Extend the Windows-only ignore pattern to the remaining long-pole
unit/integration tests that dominate the Windows nextest phase:
- custom_query_scorer_equivalency::compare_scoring_equivalency
(~130s from product_x4 cases alone; rstest via test_attr)
- test_appendable_multi_turbo_vector_storage::{congruent_upsert_read_all_distances,
congruent_random_ops_{dot,cosine}} (~80s)
- test_appendable_turbo_vector_storage::{upsert_flush_reload_in_ram_matches_independent_oracle,
turbo_model_test_random_ops_{dot,cosine}} (~55s)
- scroll_filtering_test::test_filtering_context_consistency (~28s)
None of these exercise Windows-specific behavior; Linux/macOS keep
full coverage. Roughly ~300s of Windows test time removed on top of
the multivector rstest fix.
`multivector_filtrable_hnsw_test::test_multi_filterable_hnsw` and
`multivector_quantization_test::test_multivector_quantization_hnsw`
were meant to be ignored on Windows, but the
#[cfg_attr(target_os = "windows", ignore = "...")]
#[rstest]
pattern places the attribute BEFORE `#[rstest]`, so it never reaches
the per-case functions rstest generates — the cases were still running
on Windows CI (visible in the streaming test log).
Switch to the pattern that already works for
`byte_storage_quantization_test.rs`:
#[rstest]
#[cfg_attr(target_os = "windows",
test_attr(ignore = "..."))]
which uses rstest's `test_attr(...)` forwarding, so the `#[ignore]` is
applied to each generated per-case test.
Removes ~160 s of Windows test time (multivector_filtrable_hnsw ~100 s,
multivector_quantization ~64 s), which with ~3.6× test parallelism
should trim the Windows job wall-clock by ~45 s. Coverage on
Linux/macOS is unchanged.
Jojii