* HNSWIndex::build(): add `inline_vectors` arg
Let the caller decide whether to use `inline_vectors` format.
* SegmentBuilder::build: finalize GraphInline vector storage
Instead of old "graph-with-vectors plus regular vector storage",
keep only the graph-with-vectors storage.
* Gate the GraphInline segment build behind a feature flag
universal_map_index::UniversalMapIndex does not exist; the on-disk
variant is on_disk_map_index::OnDiskMapIndex (the third MapIndex
variant). Repointed the three rustdoc links accordingly.
* shard: persist proxy pending changes on flush, stop holding back WAL ack
Hook the pending changes component into the proxy segment's flush: the
proxy flusher first persists the buffered operations into the pending
changes log, then passes the flush along to the wrapped segment. The
proxy's `persistent_version` now covers what the log durably holds on
top of what the wrapped segment persisted itself.
That is what lifts the WAL cap proxies imposed so far. `flush_all`
compares each segment's version against its persistent version; a
proxy used to report only the wrapped segment's persisted version while
its own version climbed with every buffered operation, so the WAL could
never be acknowledged past the point the proxy was created at, and a
restart replayed all of it — potentially very expensive operations,
such as an update by filter, all over again. With the buffered state
durable on disk the generic rule acknowledges the full version, and a
restart recovers it from the log instead.
Dropping a proxy's data drops the component first, which waits for any
in-flight pending changes flusher so it cannot append to the segment
directory while that is being deleted.
Update the proxy flush test to the new semantics, add a segment holder
test asserting the acknowledged version advances past a proxied delete,
and update the ack pin rationale in `finish_optimization`: the pin is
still needed after the proxies leave the holder, it just snapshots a
persistent version that now includes the log.
* Persist wrapped segment before pending changes
Prevents raising version of proxy segment too early
* collection: test crash recovery through persisted proxy changes
End-to-end test of the persisted pending changes: wrap every segment of
a local shard in a proxy, delete points so the deletes are only
buffered, flush, and assert the acknowledgeable version covers them.
Then acknowledge the WAL up to that version, drop the shard without
ever propagating the proxies, and load it again: the deletes are gone
from the WAL and must come back through the pending changes logs.
The delete under test is deliberately not the last WAL entry, as the
acknowledge never passes the last entry and that one is always
replayed.
* test: cover untested pending proxy changes paths
* test: pin persisted proxy changes findings
* Fix bad merge
* Remove corrupt length test, we cannot detect if last entry was corrupt
Remove a test that asserts an entry that isn't the last cannot have a
corrupt length. We cannot reliably detect whether the invalid length was
the last entry or not, because nothing else tells us how many entries we
expect in the file. At the same time we don't expect random bit flips.
So I removed the test.
* Simulate segments flush to clear pending changes log file
* Update test, also assert proxy segment version
* Fix bad merge
* Fix blocked test
* Enable necessary feature flags in tests (2/2)
---------
Co-authored-by: timvisee <tim@visee.me>
* segment: move proxy pending change types into segment crate
Move the types describing the changes a proxy segment buffers — point
deletes (`ProxyDeletedPoint`), payload index changes (`ProxyIndexChange`,
`ProxyIndexChanges`) and vector name changes (`IntendedVector`,
`ProxyVectorNameChanges`) — from `shard::proxy_segment` into a new
`segment::pending_changes` module.
Pure move, no behavior change: the proxy segment re-exports them from
their old location. Having them in the segment crate lets both the proxy
segment and the segment load path share them, in preparation for
persisting pending proxy changes to disk and replaying them on restart.
* segment: add PendingChange describing a persisted proxy operation
Add the `PendingChange` enum with one variant per operation type a proxy
segment buffers — point delete, payload index change, vector name
change — each carrying the operation version it was issued with. This
is the shape in which pending proxy changes are persisted to disk.
Derive serde on it and on the buffered change types it embeds, so
entries can be serialized into a log file and read back. `PartialEq` on
those types lets a persisted batch be matched against the in-memory
pending buffer after a flush.
* segment: add PendingChanges component persisting proxy changes to a log
Add `PendingChanges`, the component that manages the operations a proxy
segment buffers for one proxy layer, and persists them to disk so they
no longer only live in memory.
It keeps the same per-type buffers the proxy segment served its reads
from (point deletes, payload index changes, vector name changes), plus a
single registration-ordered buffer of everything not yet persisted.
`flusher()` writes that buffer into an append-only log file inside the
wrapped segment's directory: `pending_changes.log` for the inner most
proxy layer, with the layer number as a suffix for each layer above it.
Appends follow the mutable ID tracker: all new entries are serialized
into one buffer and written with a single call on an append-mode file,
then fsynced, so a crash can only leave a torn entry at the very end.
Loading truncates such an entry — its operations were never durable and
thus never acknowledged in the WAL — but fails hard on a malformed entry
in the middle, which cannot be explained by a torn append.
The component tracks the highest operation version the log covers.
Every registered operation at or below it is either durable in the log
or was a no-op that does not need recovery; a flusher advances it to the
proxy's version even when there is nothing to write. The pending buffer
is deliberately not cleared when the proxy propagates its changes to
the wrapped segment, as that only makes them durable once the wrapped
segment flushes. Replaying an entry twice is a version-gated no-op.
A log file left behind by a previous proxy on the same segment is
adopted by `open()`: new entries are appended after it and its highest
version is taken over, while its entries are not loaded into the
buffers as they are already applied to the segment. `load()` also
reconstructs the buffers, for callers that do want the buffered state.
* segment: replay persisted pending proxy changes onto a segment on load
Add `recover_pending_changes`, to be called when a segment is loaded on
restart, before regular WAL replay. If the segment directory holds
pending changes log files, the proxies that wrote them did not
propagate their buffered state into the segment before the process
stopped. Instead of reconstructing the proxies, replay all logged
operations directly onto the segment: inner most proxy layer first,
each file in append order, through the regular version-gated segment
operations (`apply_change`). Entries the segment already applied are
silently skipped, so a stale file is harmless.
The segment is force-flushed before the files are removed; a crash in
between merely replays the files once more.
* segment: test PendingChanges component
Cover the pending changes component: registering and flushing each
operation type and reconstructing the buffers from the log, log file
naming per proxy layer and gap-tolerant listing, covering the proxy
version without entries, operations registered while a flusher is
captured, flushers of a dropped component, torn-tail truncation versus
mid-file corruption, adoption of an existing log, and replaying logs
onto a real segment: fresh, stale (already applied), multi-layer, and
vector name changes.
* segment: include pending changes logs in segment snapshots
Register the pending changes log files of a segment in its snapshot:
add them to `snapshot_files` next to the segment state and version
files, existence-guarded, and to the segment manifest as unversioned
files. Full, partial and streamed snapshots therefore all carry them.
The recovery side needs no changes: a restored segment is loaded like
any other, which replays and removes the logs.
* shard: back proxy segment pending changes by PendingChanges component
Replace the proxy segment's separate `deleted_points`, `changed_indexes`
and `changed_vector_names` fields with a single `PendingChanges`
component. Reads keep going through the same per-type buffers, now
behind accessors; writes go through the component's `register_*`
methods, which additionally queue every operation for persistence.
Opening the component is fallible, as it adopts a pending changes log
a previous proxy may have left in the wrapped segment's directory, so
`UnsyncedProxySegment::new` now returns a result. Wrapping another proxy
opens the next proxy layer up, writing to its own dedicated log file.
No behavior change yet: the proxy still flushes and reports persistence
exactly as before, nothing is written to the log.
* shard: persist proxy pending changes on flush, stop holding back WAL ack
Hook the pending changes component into the proxy segment's flush: the
proxy flusher first persists the buffered operations into the pending
changes log, then passes the flush along to the wrapped segment. The
proxy's `persistent_version` now covers what the log durably holds on
top of what the wrapped segment persisted itself.
That is what lifts the WAL cap proxies imposed so far. `flush_all`
compares each segment's version against its persistent version; a
proxy used to report only the wrapped segment's persisted version while
its own version climbed with every buffered operation, so the WAL could
never be acknowledged past the point the proxy was created at, and a
restart replayed all of it — potentially very expensive operations,
such as an update by filter, all over again. With the buffered state
durable on disk the generic rule acknowledges the full version, and a
restart recovers it from the log instead.
Dropping a proxy's data drops the component first, which waits for any
in-flight pending changes flusher so it cannot append to the segment
directory while that is being deleted.
Update the proxy flush test to the new semantics, add a segment holder
test asserting the acknowledged version advances past a proxied delete,
and update the ack pin rationale in `finish_optimization`: the pin is
still needed after the proxies leave the holder, it just snapshots a
persistent version that now includes the log.
* shard: propagate proxy changes when unwrapping on optimizer cancel
When an optimization is cancelled or fails, `unwrap_proxy` puts the
wrapped segments back into the segment holder. Propagate the changes
buffered in each proxy into its wrapped segment first, as the snapshot
unproxy path already does, instead of dropping them with the proxy.
The pending changes log is deliberately left in place when unwrapping:
deleting it before the wrapped segment has flushed the propagated
changes would not be crash safe. It is cleaned up on restart and when
the segment directory is dropped, and a new proxy on the same segment
adopts and appends to it; replaying a stale file is safe because all
operations are version gated.
* shard: test persisted proxy pending changes
Test the proxy segment against its persisted pending changes: buffered
changes survive dropping the proxy without propagation and are replayed
onto the segment when it is loaded again; unwrapping leaves the log in
place and a new proxy on the same segment adopts and appends to it;
layered proxies each persist into their own log file and a restart
replays both; and a persisted log is part of the segment manifest and
snapshot.
* collection, edge: recover persisted proxy changes on segment load
Replay the pending changes logs left behind by proxy segments onto each
segment when a shard loads its segments, right after consistency
repair and before the payload index rebuild, vector name reconciliation
and WAL replay. Proxy state that made it to disk no longer holds back
the WAL acknowledge, so this is where it must be recovered from.
Proxies are not reconstructed: the segment holder starts with plain
segments carrying the replayed operations, and the logs are removed
once the segment flushed them.
* collection: test crash recovery through persisted proxy changes
End-to-end test of the persisted pending changes: wrap every segment of
a local shard in a proxy, delete points so the deletes are only
buffered, flush, and assert the acknowledgeable version covers them.
Then acknowledge the WAL up to that version, drop the shard without
ever propagating the proxies, and load it again: the deletes are gone
from the WAL and must come back through the pending changes logs.
The delete under test is deliberately not the last WAL entry, as the
acknowledge never passes the last entry and that one is always
replayed.
* segment: make replaying persisted proxy changes on load an explicit mode
Add `PersistedProxyChanges` to state whether persisted pending proxy
changes are replayed onto a segment when it is loaded. `Replay`, the
default, recovers them and removes the logs as before. `Ignore` leaves
both the segment and the log files untouched and logs at debug level
that replaying was skipped; it is for segment files that mirror those
of another writer, where replaying would make the local copy diverge
from what the writer's manifest describes.
All callers pass `Replay` for now, no behavior change.
* collection: do not replay persisted proxy changes on partial snapshot recovery
Partial snapshots are recovered by read replicas in a read/write
segregation setup. A read replica must not mutate its segments, so it
cannot replay the persisted proxy segment changes on load and must
ignore them instead: its segment files are a local copy of the writer's
that must stay a faithful mirror of them, as later partial snapshots are
diffed against what the writer's manifest describes. Replaying would
mutate the segment files and remove the logs, making the copy diverge.
Thread the replay mode through `LocalShard::load` as a dedicated
`PersistedProxyChanges` argument, derived from the recovery type:
`RecoveryType::Full` replays as before, `RecoveryType::Partial` ignores
the persisted changes and leaves the logs in place. Regular shard loads
replay.
Extend the crash recovery test with an ignoring load first: the delete
under test must not come back and the logs must survive, before a
replaying load recovers it.
* Persist wrapped segment before pending changes
Prevents raising version of proxy segment too early
* Fix comment
* Fix crash window, only ready optimized segment after propagating changes
The optimizer renamed a newly built segment into segments_path and wrote
its version file before finish_optimization propagated the proxies'
buffered changes into it. A crash in that window left the segment
restart-loadable but stale, permanently losing or resurrecting points.
Defer the version file save until finish_optimization has fully
reconciled proxy changes into the segment, including the post-swap dedup
pass, so it stays invisible to restart and snapshot recovery until then.
SegmentBuilder::build() gains a `ready` flag; load_segment gains
`ignore_missing_version` for the one caller reloading before that point.
Incidentally also closes the crash-unsafe cancellation-orphan cleanup
gap noted in #9217, since a cancelled build is discarded on restart the
same way.
* Force flush optimized segment, otherwise we may lose proxy changes
* Don't force flush after replay, defer deleting log files until flush
* Include persisted proxy changes log file in segment manifest
* Add random ID to proxy log files, prevent instance conflicts
* Rename proxy log file, always include level
* Delete proxy log file on unproxy, defer until next flush cycle
* Fix truncation
* Reformat
* Lock persisted segments behind runtime feature flag
* Enable necessary feature flags in tests
* Fix linters
* Measure and persist document length in the mutable text index
BM25 length normalization needs the total token count per point, which nothing
stored: `point_to_tokens_count` is the distinct count, and its meaning is fixed
by the user-visible `values_count` filter.
- `doc_len` is measured in `add_many`, the only place that still sees every
token, and persisted in the stored record alongside the tokens.
- It is a parameter on `index_str_tokens` and `MutableInvertedIndexBuilder::add`,
never derived. Without phrase matching the stored tokens are sorted and
deduplicated, and the index is rebuilt from those records on every segment
open, so a derived length would degrade to the distinct-term count on restart.
- Array boundary sentinels are discounted by inserted count, not by value:
`tokenize_doc` does not strip that character from user text the way
`tokenize_query` does, so a payload containing it has those tokens indexed
and they must be counted.
- `MutableInvertedIndex` gains `point_to_doc_len` and a running `total_tokens`,
maintained across add, overwrite and remove, so `avgdl` is a division rather
than a scan. `set_doc_len` is the only writer, so an absent length means the
same thing on every path: the slot is zeroed, never left stale.
- Recording is gated behind `TextIndexParams::scoring()`, a private const for
now. Nothing can ask for a ranked query yet, so recording on every text index
would rebuild every collection to produce data no query can reach. State
lives in the data rather than in a flag: `point_to_doc_len` is an `Option`
the way `point_to_doc` already encodes positions, and `add_many` asks the
index whether it records lengths rather than asking the config.
- `StoredDocument::doc_len` is an `Option` skipped on write when absent, so a
non-scoring index writes byte-identical records to today's and a legacy
record reads back as `None`. A document whose tokens were all filtered is a
real `Some(0)`, and stays distinguishable from one that was never measured.
The immutable and on-disk backends drop the value for now and grow their own
sidecar next.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Address text index document length review feedback
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Share the text index post-tokenization indexing path
Extract `MutableInvertedIndex::index_str_tokens` so the write path and
read-only live reload cannot drift apart, and collapse the two identical
on-disk file listings into one.
The extracted helper gates the ordered document on `point_to_doc.is_some()`
rather than re-reading `config.phrase_matching`. Equivalent, since
`point_to_doc` is built from that flag, and it skips a needless clone if the
two ever disagree.
No behavior change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Cover the text index live reload path
`ReadOnlyAppendableFullTextIndex::live_reload` had no direct test: it replays
stored documents through the same post-tokenization indexing as the write path,
but nothing pinned that down.
Asserts the incremental reload lands on the same state as a fresh
`open_appendable` after a writer deletes one point and appends two, over both
`phrase_matching` values. The phrase leg matters because adjacency depends on
the ordered document being indexed, not just the token set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Apply suggestion from @timvisee
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
`map_get_err` is handed the key it was reading, but only the `NotFound`
arm kept it; every other error boxed the underlying failure and dropped
the key on the floor.
Callers that report such a failure are then unable to say what it was
reading. The read-only segment open is the clearest case: it logs one
warning per skipped segment, so a failure anywhere among a segment's
objects — state file, id tracker, payload storage, per-vector storage
and index, payload indexes — produces the same line, naming only the
segment uuid. A recent load test hit exactly this: 667 warnings, all
byte-identical, none of them saying which file failed.
Give the `S3` variant an explicit `path` beside its source error, rather
than folding the key into the message, so the object stays a field
callers can read. It is optional because most construction sites are not
about one particular object — a short or overlapping read from the
scatter buffer, the append context's protocol errors — and those keep
using `s3()` unchanged. `s3_at()` sets it, and the object-store read
surface (every `map_get_err` caller, plus `list_files` and `exists`) now
does.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
External->internal id resolution is the expensive half of estimating a
`has_id` filter, and the query API's rescore stage turns every prefetch
into exactly such a filter. Two paths paid for it more than once:
- `search_vectors_plain` re-estimated the filter its caller had already
estimated to pick the plain strategy, so every segment resolved the
whole id list twice. It now takes the estimation as an argument, and
the dispatcher makes it once for every strategy.
- Formula rescoring resolved the prefetched ids one at a time, which the
disk-resident id tracker turns into an unpipelined block read per
point, instead of the batched single pass it offers.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Thread a `Populate` through the disk-resident id tracker's open paths
(`DiskMappingReader`, `DiskIdTracker::open`, `ReadOnlyDiskIdTracker`,
`ReadOnlyIdTrackerEnum`) so a `cached` placement primes the page cache with the
mapping files on load instead of leaving them to page in on demand. The
populate is derived from the segment config's placement at load time, clamped
by low-memory mode, in both the writable segment open and the read-only one.
The update-only lookup path keeps its transfer-nothing policy, and the
build-time open stays cold: the built segment is reloaded anyway.
`cached` is no longer rejected by validation.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Add `id_tracker: { memory: cold | pinned }` to CollectionParams,
CollectionParamsDiff and CreateCollection (REST + gRPC `IdTrackerParams`),
mirroring `payload: { memory }`. `cold` builds the disk-resident id tracker,
`pinned` the in-RAM immutable one. Unset keeps the current behavior: the
`serverless_compatible` feature flag decides.
The requested placement is persisted as an optional `id_tracker_memory` on
SegmentConfig (skipped when unset, so existing configs are unchanged); the
segment builder resolves it through `SegmentConfig::id_tracker_memory_placement`
instead of reading the feature flag directly.
The config mismatch optimizer rebuilds non-appendable segments whose effective
placement differs from the requested one. Appendable segments are skipped: they
always use the mutable tracker and get the current config when indexed.
`cached` is rejected by validation: the disk mapping reader has no
populate-on-open path.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix: do not claim an unfinished operation when flushing
A flush pass can capture a segment between the separately locked steps of
one update operation. Persisting it under that operation's version marks
the segment clean while the rest is still in memory, so every later pass
skips it and the WAL acknowledge moves past the operation.
Clamp what a flush claims to the last fully applied operation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Fix rustfmt in alias_mapping test after merging dev
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
* introduce UniversalAppendFs helper
* AI: migrate to UniversalAppendFs bound
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* don't use it in Gridstore
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* HnswGraph: put into Arc
Later it will be shared by vector index and vector storage.
* Read the inline base vectors through HnswGraph
* HnswGraph knows its residency
* strip fs from id tracker
* strip fs from Gridstore and Logstore
* strip fs from UpdateOnlyBlobstore
* strip fs out of null and bool indexes
* strip fs out of chunked vectors
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Remove MessagePack (rmp-serde)
The WAL switched from msgpack to CBOR in v0.3.5 (2021-07-11), so v0.3.4
is the last version that wrote msgpack entries. Drop the read fallback
kept for those entries, plus the remaining test and bench usages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Drop unused fs4 dependency from collection
Not referenced anywhere in the crate. Still used by wal and common, so
the workspace entry stays.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
- 'mising' -> 'missing' in .github/review-rules.md (flagged by the
repo's own codespell config)
- 'Do not create segments larger this size' -> 'larger than this size'
in config.yaml, the optimizer builder/diff sources and the grpc
proto + generated rust comment
- 'bigger then' -> 'bigger than' in the query scorer rustdoc
- three duplicated-word rustdoc fixes ('override in in', 'any of of',
'and and')
* Direct call for shard transfer method and keys
* Reuse cardinality estimate in sparse plain search
* Avoid recounting available points in segment size info
* Avoid cloning segment config when updating quantization
* Avoid cloning search request for load profile
* Direct call for counting read-only segments
* Avoid re-reading point range for values count
* Direct call to check replica states when initializing collection
* Direct call to look up transfer on restart
* Direct call for shard replicas after snapshot recovery
* Direct call for local replica states in health check
* Direct call for payload index schema keys when applying state
* Direct calls for sharding method and key mapping when creating shard key
* Direct call to check if peer has shards
* Direct call for sharding method and keys when dropping shard key
* Avoid cloning collection params for group by ordering
* Avoid cloning collection params in local shard search
* Direct call for peer address when sending Raft messages
* Direct call for peer address in who_is
* Avoid cloning remote query batch request
* Avoid cloning operation in queue proxy update
* Avoid cloning gRPC search groups request
* Fetch cluster status once in cluster telemetry
* Direct call to validate transfer exists on finish
* Direct call for sharding method when dropping shard key
* Avoid cloning peer address map when listing peers
* Avoid cloning peer address map when adding peer to known
* Avoid cloning shard key mapping when routing writes with fallback
* Avoid cloning shard key mapping when checking resharding start
* Avoid cloning gRPC recommend groups request
* Avoid cloning operation when retaining forwarded point IDs
* Direct call for counting collections in telemetry
* Direct call to validate transfer exists on recovery
* Direct call for shard IDs by shard key
* Direct call for shard keys
* Direct call to check if peer has shards in consensus
* Direct call for replica state on transfer recovery
* Direct call to check for active replicas when routing writes with fallback
* Direct call to validate transfer exists on abort
- `TurboVectorStorageImpl::{insert_vector, insert_tq_bytes}`: return
error right away. `QuantizedStorage::upsert_vector` will return error
anyway, so why pretend it will not. Bonus: `quantization_buffer`
no longer needed. Related: #9953.
- `DenseVectorStorageImpl::insert_vector`: `Err`, not `panic!`. For
consistency with `TurboVectorStorageImpl`. It's unreachable anyway.
- `QuantizedStorage::reload`: reopen via `self.storage.live_reload()`.
The same, but now we don't need to keep `fs`.
These were added for named-vector CRUD (acfb650), never used. The actual
placeholder storages `EmptyDenseVectorStorage` and
`EmptySparseVectorStorage` stay.
* Fall back to per-vector TQ scoring when runs are short
Run-batched scoring pays off on plain and dense filtered scans but
regresses HNSW, where neighbor ids rarely form consecutive runs and batch
setup dominates. Gate both EncodedVectorsTQ::score_points and Turbo
score_query_batch on offsets_worth_batch_scoring, which takes the run path
only when the ids split into runs averaging BATCH_SCORE_MIN_MEAN_RUN
vectors or more.
The average decides, not the longest run: a sorted id list -- what a
filtered scan hands the scorer -- already contains adjacent pairs at ~1%
density while its runs still average one vector, so a "contains a run of
>= 2" test sends those down the run path to pay setup per vector, measured
at up to +75% against the better path. An average also stays independent
of the batch size the driver slices ids into, which a longest-run test does
not. The threshold comes from the measured crossover -- mean run 2.0-3.6,
stable across dims 128/512/1024, both RAM storages and the 1/2/4-bit
widths -- and a fully contiguous block is recognized in O(1), so a plain
scan pays nothing for the gate.
* io_uring: never gate run-batched scoring
The gate exists because run batching costs setup that short runs do not
repay on RAM and mmap storages. io_uring is the opposite: one run-granular
read beats the batched per-vector path at every density measured -- 27% on
HNSW-shaped id lists, 43% at 25% filter density, 93% on a full scan --
because per-request submission and completion bookkeeping dominates once
the data sits in the page cache. Gating it costs 36% on HNSW-shaped lists.
Add EncodedStorage::prefers_run_reads, defaulting to false so every storage
keeps its current routing, and override it for the single-file quantized
storage when its backend is io_uring. Remote backends (object stores, a
gRPC peer) deliberately keep the per-vector path: their reads pipeline
across a batch, while run-granular reads would serialize the round trips.
QuantizedStorage::is_in_ram_or_mmap() still reports true for every backend,
which is what routes io_uring into the gate in the first place. Correcting
that would also change how the multivector storage picks between its
in-memory and uring scoring paths, so it is left to a separate change.
* Rename prefers_run_reads to prefers_contiguous_reads
"Run reads" is easy to misread as "execute reads"; contiguous makes the
storage I/O preference explicit.
* QuantizedStorage::for_each_run: pipeline run reads on async backends
With `prefers_contiguous_reads()` true for io_uring, every batch goes
through `for_each_run`, which read each run synchronously: a scattered
id list (HNSW neighbours) waited on one disk read per vector, where
`for_each_in_batch` kept the whole batch in flight through `read_batch`.
Submit all runs of a batch together, still one read per run, so
scattered reads stay pipelined while a scan still reads each run in one
request. Backends without async reads keep the sequential loop.
`turbo_vector_search` (dim 1024, 200k vectors, 4096 shuffled ids per
iteration) against dev: cold scattered io_uring 187 ms -> 26.5 ms
(dev 29.5 ms); the warm scan keeps 198 ms -> 21 ms. Warm scattered
lands at 5.06 ms (dev 4.67 ms), giving up the 3.68 ms of synchronous
reads, which only holds with the data already in the page cache.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VRiRaiPsm5CEBgpQ7VAQab
* Iterate consecutive runs and share the run-scoring gate
`for_each_consecutive_run` becomes `consecutive_runs()`, a lazy iterator
over `Run { first, start, len }`, so a storage can feed runs straight into
a read pipeline. `QuantizedStorage::for_each_run` loses the intermediate
`Vec` and the duplicated `ReadRange` construction: the pipelined branch
maps the iterator into `read_batch`, the synchronous branch keeps the
per-run `Sequential`/`Random` hint that picks between mmap's two mappings.
The routing condition duplicated at both scoring call sites moves into
`EncodedStorage::prefers_run_scoring`: same expression, one place.
`for_each_run`'s contract no longer promises run order: pipelined
backends report reads as they complete, so callers address results by
`first`. Add a contract test over the mmap and disk-cache backends; the
latter is the async-capable backend that runs on every platform and
covers the `read_batch` branch io_uring takes on Linux, which no test
exercised before.
Measured on Apple M3 against 1f2d1264e, interleaved A/B/B/A: the run
path is unchanged on all four storages (-0.3%, +0.0%, +0.9%, -1.1%,
within replicate noise).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Rename Run to ConsecutiveRun
`Run` on its own reads as "execute", the same ambiguity that got
`prefers_run_reads` renamed earlier in this branch. `ConsecutiveRun`
names what the value is and pairs with `consecutive_runs()`, the iterator
that yields it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* io_uring pipeline: submit eagerly only while reads are outstanding
`IoUringPipeline::wait()` called `submit_and_wait(0)` whenever a completion
was ready and anything was enqueued. `read_batch` enqueues one entry per
consumed completion, so on a warm page cache — where reads complete inline
at submission — that was one `io_uring_enter` per read. Scattered quantized
scoring over io_uring ran at ~630 ns/point warm against ~460 for the same
reads issued synchronously (which, without direct_io, are plain `pread`s).
Now the eager submission happens only while the kernel still has reads
outstanding (`in_progress` minus the completions already waiting in the
queue). When everything submitted so far has completed — the warm case —
the enqueued entries wait and go down together once the ready completions
run out. On a cold device nothing changes: a completion is answered with a
submission as before, so the in-flight depth never sags. Two fixed rules
tried first (submit only when nothing is ready; submit once half the queue
piled up) both cost the cold path, +6 % and +3 %, in proportion to how long
the device sat idle while ready completions were drained.
turbo_vector_search / turbo_uring_ab, Zen 4, `taskset -c 7`, prebuilt
binaries run alternately, cold rows with the page cache dropped:
warm scattered, uring hnsw: 634 -> 472 ns/point (-25 %)
warm scattered, uring p0.25: 477 -> 369 ns/point (-23 %)
warm sequential, uring p1.00: 45 -> 45 (flat)
cold scattered, uring: 29.9 -> 29.9 ms/iter (flat, 4 reps each within 0.7 %)
mmap rows (control): within ±2 %
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix: reject mismatched dense dims in recommend average
Stop silently truncating oversized negative examples during
average_vector merge. Validate dense dimensions within each example
group and between positive/negative averages before zip-merge.
Fixes#10369
* Simplify: keep only the merge-time dimension check
The zip truncation in merge_positive_and_negative_avg is the only place
an oversized negative can silently pass the downstream dimension check;
within-group mismatches already grow the average to the max length and
fail the segment-entry check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style(query): make recommendation conversion explicit
* test: assert recommendation dimension errors
Issue: #10369
Make the regression test verify the exact WrongVectorDimension payload for mismatched recommendation vectors.
---------
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* TurboQuantizer::score_precomputed_batch: score a contiguous run of vectors
Batch counterpart of `score_precomputed` for vectors stored back to
back at `quantized_size()`: the width's kernel scores the whole run of
codes in one `dotprod_batch` call, then a second pass applies each
vector's extras. L1 dequantizes per vector and stays a plain loop.
Tested against per-vector `score_precomputed` for every width,
distance, and mode over run lengths that leave every group remainder.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* EncodedStorage::for_each_run: serve consecutive offsets as contiguous runs
`for_each_run(offsets, callback(first, count, bytes))` splits the
offsets into maximal runs of consecutive ids the storage can serve
from one contiguous slice, so a sequential scan resolves chunk lookups
and reads once per run instead of once per vector. The default serves
every vector as its own run; `for_each_consecutive_run` is the shared
run detection for storages that override it, with a per-run cap for
chunk boundaries. The test storage overrides it (its data is one flat
buffer).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* EncodedVectors::score_points: batched scoring entry point, run-batched for TQ
`score_points(query, offsets, scores)` scores a batch of points. The
default keeps the per-vector loop the scorers run today, so SQ/PQ/BQ
are unchanged. TurboQuant overrides it: on RAM/mmap storages it walks
`for_each_run` and scores each contiguous run with one
`score_precomputed_batch` call, hoisting the score inversion out of
the loop; backends with async reads keep the pipelined per-vector
path. Non-consecutive offsets degrade to single-vector runs, so
scattered access keeps its previous cost.
Integration test: `score_points` vs `score_point` for every bit width
and mode, Dot and inverted L2, over sequential, scattered and
descending id orders.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Quantized storages: for_each_run over their contiguous regions
The RAM storage and both chunked mmap storages cap runs at their chunk
boundary and serve each run with one `get_many`; the single-file mmap
storage serves any run as one sequential read. Unit test on the RAM
storage: runs cover every offset once, in order, with bytes identical
to per-point reads, across the internal chunk boundary.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* QuantizedQueryScorer: score batches through EncodedVectors::score_points
Routes `score_stored_batch` through the batched entry point, so
TurboQuant-as-quantization scans score contiguous runs with one kernel
call per run; SQ/PQ/BQ keep the per-vector loop via the default.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* TurboScoring::score_query_batch: run-batched scoring for Turbo4 storages
Adds the batch counterpart of `score_query_bytes` to the trait, with
one shared implementation over the storage's `EncodedStorage`:
consecutive ids are coalesced into contiguous runs, each run scored by
a single `score_precomputed_batch` call, and the metric sign applied
once over the batch. Backends with async reads keep the pipelined
per-vector path. `TurboQueryScorer::score_stored_batch` now calls it.
The batch-vs-single storage test grows to 8192 vectors so a full
ascending scan crosses a chunk boundary of the chunked backend, and
runs that scan on the chunked, mmap and io_uring backends.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* score_precomputed_batch: keep the extras pass in L1
The kernel pass and the extras pass now alternate over sub-runs of 64
vectors instead of each covering the whole run: for a run of several
hundred vectors the second pass otherwise refetched every vector's
extras from L2. Measured with 512-vector runs from the full-scan
driver at dim 512: the regression against 64-vector runs went from
+11 % to +2 %.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Bench: exhaustive search over Turbo4 storages through the plain-index driver
`turbo4_full_scan` runs `BatchFilteredSearcher::peek_top_visible` —
the exact path of a non-indexed search — over 200k normalized random
vectors for Turbo4 as datatype (appendable chunked, in RAM) and Turbo4
as quantization (over a RAM dense storage), at dims 64 to 1024, so the
fixed per-point cost of the scan driver is measured next to the kernel.
`TURBO_SCAN_DIMS=64,128` narrows the dims while iterating.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Split async IO into extension traits; only async-capable backends implement them
Move `read_bytes_async` / `open_async` off the universal `UniversalRead` /
`UniversalReadFs` traits into dedicated extension traits, `UniversalReadAsync`
and `UniversalReadFsAsync` (traits/async_io.rs). Only backends with a genuine
async story implement them — the blob family, the disk caches layered over it,
and a trivial ready-impl for mmap (tests and the mmap lookup path) — each in a
dedicated async_io.rs next to its sync impl.
`CachedFs` now requires its inner filesystem to be `UniversalReadFsAsync`; the
requirement reaches segment code through one supertrait bound on
`UniversalReadExt`. io_uring implements no async surface anymore: the
tokio_uring bridge thread, its tests, the musl-gated tokio-uring dependency,
and the `IoUringFile` read-only-segment wiring (`UniversalReadExt` impl and
the *RoIoUring condition-checker variants) are deleted — io_uring is not a
read-only-segment backend.
The payoff for live reload: `CachedFs::resolve_prefetched` awaits every parked
prefetch, and the edge refresh flow now runs preload -> resolve -> reload, so
the per-segment write locks never wait on IO.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Decouple UniversalReadExt from the async filesystem requirement
UniversalReadExt is condition-checker dispatch; it never consumed the async
surface itself. Drop its `Fs: UniversalReadFsAsync` supertrait bound and relax
CachedFs's struct-level bound back to `UniversalReadFs` — the async requirement
now lives on the one impl that consumes it, `CachedReadFs for CachedFs`
(schedule_open parks the inner filesystem's `open_async` futures).
The bound then surfaces only on the lifecycle/preload impl blocks that go
through CachedReadFs (segment open, live-preload/reload, config reload, edge
load/refresh); the search path carries no async bounds at all.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* existing segments: wait for IO outside of search pool
* new segments: wait for IO outside of search pool
* extract reload into separate function
* Update lib/edge/Cargo.toml
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* rename `reopen`->`live_reload` and `schedule_reopen`->`live_preload`
* `UniversalRead::live_preload` returns a shared future
* assert snapshot-miss eagerly on `live_preload`
`live_reload` cannot see the failed preload: its blocking fallback
re-resolves the length from the remote and succeeds. The error
surfaces at preload time, as callers (`ok_not_found`) expect.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [CachedFs] new `schedule` and `wait_all` primitives
* [AppendableIdTracker] don't reopen if just opened
* eager NotFound in `schedule_open`
* add traces for async reads
* finish `preopen`/`preload` with `wait_all`
* lock all segments in parallel for `live_reload`
* LIST before everything
to do: we don't have whole-fetch in async mode. to prevent sequential
`len`, we won't overlap static files with LIST.
* `wait_all` returns nothing
* `UniversalReadFs::open_async`
* `schedule_open` polls once
Scheduled opens must start eagerly: sync backends complete their
`open_async` on the first poll, preserving the prefetch contract
(handles outlive later file deletions/replacements). Moved down from
the integration branch so this PR stays green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
xzfc