* Add `match: { substring }` filter condition
Unindexed `text` and `text_any` matching became token-aware in #10341 and
#10593. Users who relied on the old raw substring behaviour get it back as
an explicit condition: `match: { "substring": "..." }` selects points with a
string value containing the given string, byte-wise and case-sensitive,
consistent with exact keyword and prefix matching.
Execution: a keyword index (with or without the `prefix` option) serves the
condition by scanning its value dictionary and uniting the postings of the
matching keys; cardinality reuses the prefix estimator, generalised into
`keys_union_cardinality`. The per-point checker goes through the forward
index. Without a keyword index the condition falls back to reading the
payload. Text, bool, integer and uuid indexes decline it.
Strict mode: the condition requires the `KeywordMatch` capability, so with
`unindexed_filtering_retrieve: false` it is rejected on unindexed and on
text-indexed fields and allowed on any keyword index.
API: `MatchSubstring` in the REST `Match` union with regenerated OpenAPI,
gRPC `Match.substring = 12`, edge python `MatchSubstring`, edge ffi
`Match::Substring`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Test substring fallback on a text-indexed field, document estimator params
A text index cannot serve `substring`, so on a field that has only a text
index the condition runs through the payload fallback; only strict mode may
reject it. Pin that in the OpenAPI suite and reword the strict-mode unit
test comment, which read as if the text index itself blocked the query.
Also spell out what `keys` and `postings` mean in `keys_union_cardinality`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Serve `match: { substring }` from the keyword key dictionary
The condition used to enumerate keys through `MapIndexRead::for_each_value`,
which on the on-disk variant drags the whole `value_to_points` file through
`for_each_entry`, plus one random read per matching key for its postings
count. Query planning paid that scan in full, before deciding whether to use
the clause at all.
Route it through the `prefix_index.bin` key dictionary instead: front-coded
keys with their postings counts inline, no postings. Estimation now reads
keys only and never touches `value_to_points`; filtering takes the matched
key list and resolves postings in one batched read, as prefix matching
already does.
This makes the `prefix` option a requirement: a keyword index without it has
no key dictionary, so it declines the condition and falls back to the payload
scan, the same as a text index. Strict mode follows — substring now infers
`KeywordPrefix`, so `unindexed_filtering_retrieve: false` names
`keyword (with prefix: true)` as the index to create.
`PrefixIndex::for_each_key` reads blocks in ~1 MiB chunks rather than the
whole key section at once: a substring cannot be pruned by the block index,
so the one-shot read would grow with the dictionary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Sync MatchSubstring OpenAPI description with Rust docs
After routing substring matching through the prefix key dictionary,
the schema docstring required regenerating so OpenAPI stays consistent.
* Scan the map index keys for substring match without a dictionary
Without the keyword dictionary a substring condition was declined by the
field index and left to the per-point condition checker, which reads the
forward index for every candidate point. Enumerate the distinct keys of
`values_to_points` instead: the same one-pass-over-distinct-values shape as
the dictionary scan, only over a structure that interleaves keys with their
postings. Filtering and cardinality estimation are then always served, so
the condition can act as a primary clause on a plain keyword index.
Prefix matching keeps its per-point fallback: an ordered dictionary is what
makes a prefix a bounded range, and enumerating every key to answer one is
not a trade worth making implicitly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Reject substring matching in strict mode
A substring condition is answered by looking at every distinct value of the
field: no index gives it a bounded access path, so there is no index a user
could create to make it affordable. Reject it under strict mode instead,
wherever a filter reaches verification — read and write filters, nested
sub-filters, and prefetch filters.
Filter limits are now checked before the unindexed-field check, so the
rejection is not reported as "create an index for this key", advice that
would lead to the same rejection afterwards.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Update the strict mode substring test to the new rejection
The test asserted that substring filtering under strict mode asks for a
keyword index with the `prefix` option. It is now rejected whatever index
the field carries, so every case in the test gets the same answer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Estimate a substring condition without scanning
Counting the keys a substring matches costs the same scan as answering the
condition, and `filter` then repeats it to collect those keys. Report the
uninformed estimate instead — the one an unindexed condition has always
reported — and keep the primary clause, so the scan happens once, in
`filter`, and only when the planner picks the condition to drive iteration.
With no counts to collect, `substring_scan` collapses into `substring_keys`:
the in-RAM variants no longer look up a posting count per matched key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Don't to parse everything as UTF-8
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: timvisee <tim@visee.me>
* Remove MessagePack (rmp-serde)
The WAL switched from msgpack to CBOR in v0.3.5 (2021-07-11), so v0.3.4
is the last version that wrote msgpack entries. Drop the read fallback
kept for those entries, plus the remaining test and bench usages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Drop unused fs4 dependency from collection
Not referenced anywhere in the crate. Still used by wal and common, so
the workspace entry stays.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* TurboQuantizer::score_precomputed_batch: score a contiguous run of vectors
Batch counterpart of `score_precomputed` for vectors stored back to
back at `quantized_size()`: the width's kernel scores the whole run of
codes in one `dotprod_batch` call, then a second pass applies each
vector's extras. L1 dequantizes per vector and stays a plain loop.
Tested against per-vector `score_precomputed` for every width,
distance, and mode over run lengths that leave every group remainder.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* EncodedStorage::for_each_run: serve consecutive offsets as contiguous runs
`for_each_run(offsets, callback(first, count, bytes))` splits the
offsets into maximal runs of consecutive ids the storage can serve
from one contiguous slice, so a sequential scan resolves chunk lookups
and reads once per run instead of once per vector. The default serves
every vector as its own run; `for_each_consecutive_run` is the shared
run detection for storages that override it, with a per-run cap for
chunk boundaries. The test storage overrides it (its data is one flat
buffer).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* EncodedVectors::score_points: batched scoring entry point, run-batched for TQ
`score_points(query, offsets, scores)` scores a batch of points. The
default keeps the per-vector loop the scorers run today, so SQ/PQ/BQ
are unchanged. TurboQuant overrides it: on RAM/mmap storages it walks
`for_each_run` and scores each contiguous run with one
`score_precomputed_batch` call, hoisting the score inversion out of
the loop; backends with async reads keep the pipelined per-vector
path. Non-consecutive offsets degrade to single-vector runs, so
scattered access keeps its previous cost.
Integration test: `score_points` vs `score_point` for every bit width
and mode, Dot and inverted L2, over sequential, scattered and
descending id orders.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Quantized storages: for_each_run over their contiguous regions
The RAM storage and both chunked mmap storages cap runs at their chunk
boundary and serve each run with one `get_many`; the single-file mmap
storage serves any run as one sequential read. Unit test on the RAM
storage: runs cover every offset once, in order, with bytes identical
to per-point reads, across the internal chunk boundary.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* QuantizedQueryScorer: score batches through EncodedVectors::score_points
Routes `score_stored_batch` through the batched entry point, so
TurboQuant-as-quantization scans score contiguous runs with one kernel
call per run; SQ/PQ/BQ keep the per-vector loop via the default.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* TurboScoring::score_query_batch: run-batched scoring for Turbo4 storages
Adds the batch counterpart of `score_query_bytes` to the trait, with
one shared implementation over the storage's `EncodedStorage`:
consecutive ids are coalesced into contiguous runs, each run scored by
a single `score_precomputed_batch` call, and the metric sign applied
once over the batch. Backends with async reads keep the pipelined
per-vector path. `TurboQueryScorer::score_stored_batch` now calls it.
The batch-vs-single storage test grows to 8192 vectors so a full
ascending scan crosses a chunk boundary of the chunked backend, and
runs that scan on the chunked, mmap and io_uring backends.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* score_precomputed_batch: keep the extras pass in L1
The kernel pass and the extras pass now alternate over sub-runs of 64
vectors instead of each covering the whole run: for a run of several
hundred vectors the second pass otherwise refetched every vector's
extras from L2. Measured with 512-vector runs from the full-scan
driver at dim 512: the regression against 64-vector runs went from
+11 % to +2 %.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Bench: exhaustive search over Turbo4 storages through the plain-index driver
`turbo4_full_scan` runs `BatchFilteredSearcher::peek_top_visible` —
the exact path of a non-indexed search — over 200k normalized random
vectors for Turbo4 as datatype (appendable chunked, in RAM) and Turbo4
as quantization (over a RAM dense storage), at dims 64 to 1024, so the
fixed per-point cost of the scan driver is measured next to the kernel.
`TURBO_SCAN_DIMS=64,128` narrows the dims while iterating.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* `UniversalReadFs::open_async`
* `schedule_open` polls once
Scheduled opens must start eagerly: sync backends complete their
`open_async` on the first poll, preserving the prefetch contract
(handles outlive later file deletions/replacements). Moved down from
the integration branch so this PR stays green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Drops 6 crates from the release build and 7 from the workspace test
build, with no source changes.
- geo: no triangulation, only Contains/Intersects/Haversine (spade, earcut)
- jsonwebtoken: HS256 from_secret only, no PEM keys (pem, simple_asn1)
- tar: nothing sets unpack_xattrs, which defaults to false (xattr)
- duplicate: every duplicate_item names its module (proc-macro2-diagnostics)
- pprof: no C++ frames to demangle (cpp_demangle)
Also promotes duplicate to a workspace dependency.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Replace cgroups-rs with direct cgroup memory file reads
We used cgroups-rs in exactly one place, to read the memory limit and
usage of our own cgroup, so read those files directly instead. Drops 34
crates from the lockfile, including the zbus stack that carries
RUSTSEC-2026-0221.
Also fixes two latent cgroup v1 bugs (the LONG_MAX unlimited sentinel
reported ~9 EB of total memory, an unreadable limit file reported 0
bytes) and the hierarchy mix-up on hybrid hosts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Decline cgroup memory reporting when the usage read fails
Reporting a usage of 0 made available_memory_bytes claim the whole cgroup
limit as free. Fall back to sysinfo when the usage file cannot be read at
init, and keep the last known value on a failed refresh.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Keep the last known memory limit when its read fails
A transient read failure cleared the cached limit and silently fell back
to host memory while the process was still capped, the same direction of
over-reporting as the usage read. Both now keep their last known value,
and a limit lifted at runtime still clears.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Treat a malformed memory limit as an error, not as unlimited
Parse failures returned Ok(None), so garbage in the limit file cleared a
valid cached limit on refresh and read as unlimited at init. Reserve
Ok(None) for "max" and the v1 sentinel, and report anything else as
InvalidData so the last known limit survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The app telemetry was the only user of the sys-info crate, while segment
already depends on sysinfo for cgroup-aware memory accounting. Read the
distribution id/version via sysinfo statics, and the disk size fallback
via common::disk_usage, so the whole sys-info crate (and its bundled C
sources) drops out of the build. sysinfo is hoisted to a workspace
dependency, shared by the root crate and segment.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Gridstore: introduce storage operating mode in config
Add a mode field to the gridstore config, selecting between the dynamic
mode (current behavior, the default) and the upcoming serverless mode.
The mode is specified through StorageOptions on creation, persisted in
config.json, and read back first when opening so the correct variant can
be selected automatically. Configs written before this field existed
deserialize as dynamic.
For now, selecting the serverless mode returns an error; the variant
itself is added in follow-up commits.
* Gridstore: move dynamic implementation into dedicated module
Mechanical move of the current Gridstore implementation into
gridstore/dynamic.rs as DynamicGridstore. The public Gridstore struct
becomes a thin wrapper holding a mode variant enum, propagating every
call into the selected variant. For now the enum only has the dynamic
variant; the serverless variant is added in follow-up commits.
No logic changes to the dynamic implementation itself: only visibility,
the config parameter now passed into open (the wrapper reads it first to
select the mode), and open_or_create staying on the wrapper.
* Gridstore: add serverless tracker
Add the append-only mapping tracker for the serverless storage mode.
The tracker file is a plain array of 16-byte mapping entries without any
header: the number of mappings is defined by the exact file length, and
the entry index is the point offset. The file starts empty and only ever
grows by appending, existing bytes are never rewritten. Mappings must be
set in monotonically increasing point offset order; skipped offsets are
backfilled as zeroed entries which decode as None.
New mappings are buffered in memory and appended with a single write per
flush. A flush with a stale target is a no-op so bytes are never written
twice. A torn trailing entry (file length not a multiple of the entry
size) is ignored when reading and truncated away when opening writable.
Unlike the dynamic tracker, the file is read and written directly with
positional file IO instead of memory mapping, as serverless environments
do not handle memory mapped files well.
* Gridstore: add serverless storage variant
Add the append-only gridstore variant for serverless deployments, which
restrict IO to appending to files: existing bytes can never be
rewritten, and IO is expensive so as few files as possible are used.
The variant stores all value data in a single page file next to the
serverless tracker and the storage config, three files in total. Both
data files start empty and only ever grow by appending; there is no
preallocation, no used-block bitmask and no gap/region bookkeeping.
Values are appended at put time at the next block aligned offset, with
the zero padding included in the write so it lands exactly at the end
of the file. Mappings are buffered and appended to the tracker with a
single write per flush, after the page file is synced, so a mapping on
disk never points at data that is not durable.
Values cannot be updated or deleted, and must be put at monotonically
increasing point offsets; violations are rejected before any data is
written. Files are read and written directly, never memory mapped.
The mode is selected through StorageOptions on creation and picked up
automatically from the persisted config when opening.
* Gridstore: serverless support in reader and view
Extend the read-only GridstoreReader and the GridstoreView with the
serverless mode, keeping both public types unchanged: like the writable
Gridstore they now hold a mode variant internally, selected
automatically from the persisted config when opening.
The serverless reader holds the tracker and page directly and reads the
files positionally, without memory mapping. A live reload re-reads the
mapping count from the exact tracker file length (there is no size
header), ignoring a torn trailing entry, and never truncates as it is
read-only. Value reads always go directly to the file, so newly
appended data is readable without remapping anything.
* Gridstore: document storage operating modes
* Gridstore: review fixes for the serverless mode
Hardening and cleanup from a review pass over the new serverless
storage variant:
- Batch the reader side iteration like the writer already did, instead
of materializing tracker mappings for the full range in one go, which
could transiently allocate gigabytes on large storages.
- Recover the append cursors when a positional write fails partway:
truncate the file back to the tracked length so a retried append or
flush never rewrites bytes that already landed in the file.
- Validate page addressability before appending value data, a rejected
put must not grow the page file.
- Cross-check tracker and page consistency when opening: mappings that
reference value data past the end of the page file (e.g. after a
partial copy or restore) now fail fast instead of surfacing as
opaque read errors per point.
- Reject value pointers into any page other than page 0 on the
serverless read path with PageNotFound, matching the dynamic mode
contract, instead of silently reading from a wrong location.
- Refresh the reported storage size on reader live reload even when no
new mappings were flushed, unflushed value data may have grown the
page file already.
- Validate configs read from disk: a corrupt config with zero sized
blocks, pages or regions is now rejected when opening instead of
panicking on a division by zero later.
- Classify rejected serverless puts as UnsupportedOperation, consistent
with rejected deletes, so they don't surface as user-facing
validation errors at the segment level.
- Deduplicate the compression dispatch into Compression::compress and
Compression::decompress, and the serverless file create/open patterns
into shared direct IO helpers, so the two modes and files can't
silently drift apart.
* Gridstore: cover both operating modes in mode-agnostic tests
Parameterize the gridstore tests that exercise mode-agnostic behavior
over both the dynamic and serverless mode with rstest, using a
single and bulk put/get roundtrips, storage files, basic persistence,
corrupt config rejection, batched read congruence, reader live reload,
and the different block sizes.
Mode specific expectations branch inside the tests: expected file
names, storage size semantics (whole blocks vs exactly packed bytes),
value pointer layout (page spill over vs a single packed page), and
gaps (created by deletes in dynamic mode, by skipped puts in serverless
mode). Dynamic-only internals assertions are kept behind a mode check.
Tests around updates, deletes, page spanning, block reuse and other
dynamic-only behavior intentionally stay dynamic; the serverless
specific format invariants remain covered by the dedicated serverless
tests.
* Gridstore: port serverless specific tests from sibling branch
Source the serverless specific test cases that the
serverless-gridstore-updates branch added, adapted to the dedicated
variant implemented here (distinct file names, headerless tracker
with 16 byte entries, a single packed page without trailing padding,
and rejected re-puts):
- writes only ever append: tracker and page files only grow and
previously written bytes stay byte-for-byte untouched
- new mappings land exactly at the end of the tracker file, which
always covers the exact number of mappings
- mapping gaps are zero-padded on disk and survive reopening
- values are packed back to back at block aligned offsets, the page
file ends exactly at the last value
- serverless mode never creates nor reports block flag files
- a flusher persists exactly the mappings that existed at its
creation, later puts stay pending
- a config claiming the wrong mode fails loudly in both directions
instead of loading the incompatible file format of the other mode
Tests around their mode switching, page spanning and tolerated deletes
don't apply to this design and are intentionally not ported.
* Gridstore: test serverless production risk scenarios
Add tests for the operational aspects that matter before serverless
mode goes to production, each covering a scenario that wasn't
evaluated yet:
- Replayed puts of already persisted offsets (a WAL redo after a
crash where the flush completed but was never acknowledged) are
rejected without appending anything, and max_point_offset is the
exact offset a replay must resume at.
- The accepted crash case of a tracker file extended with zeroed
bytes: the entries count as permanent None mappings, can never be
put again, and the storage stays consistent and writable past them.
- The read-only reader never modifies the files: opening over a torn
tracker tail, reading, iterating and live reloading leave both
files byte-for-byte untouched.
- A multi-round put/flush/reopen cycle always exposes exactly the
flushed prefix, with the mapping count matching the exact tracker
file length and unflushed offsets reusable.
- An append beyond the maximum addressable block offset is rejected
before writing anything, keeping retried puts from growing the page
file unboundedly.
* Gridstore: rename serverless mode to append-only, split into module
Rename the mode after its defining characteristic instead of its
deployment target: files only ever grow, existing bytes are never
rewritten. Renames Mode::Serverless to Mode::AppendOnly (persisted as
"mode": "append_only") and the on-disk file names to
append_only_tracker.dat and append_only_page_0.dat. The serverless
deployment motivation stays in the documentation.
Also split the single 2300 line serverless.rs into an append_only
module with dedicated files for the storage, page, view, reader and
tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use universal IO in Gridstore
* Include upstream preopen logic in new Gridstore variant
* Gridstore: buffer append-only value writes until flush
In append-only mode, put previously wrote the value data to the page
file right away, one write operation per put, while mappings were
already buffered and batch persisted on flush. Buffer value writes the
same way: both the value and its mapping now only land on disk once a
flush cycle executes.
This batches all new value data into a single write operation per
flush, which is significantly more efficient on S3 based storage where
every write is a costly operation. A flush now performs exactly two
writes: one appending all buffered value data to the page file, one
appending all pending mappings to the tracker file, in that order, so
a mapping on disk never points at value data that is not durable.
The page mirrors the tracker's pending mechanism: an in-memory buffer
that is byte for byte the next append (zero padding between block
aligned values included), a watermark captured at flusher creation so
puts made during a flush stay buffered, a stale-flush no-op guard so
appended bytes are never written twice, and truncate-back recovery on
failed writes. Reads transparently serve buffered values from memory.
As a side effect, a crash between flushes now leaves nothing on disk
at all, where the write-through approach left orphaned value bytes in
the page file. The buffered data is held in memory until the next
flush, bounded by the flush cadence.
Universal IO filesystem handles are now required to be Send + Sync, so
the flusher closure can carry one to grow the page file at flush time;
all existing backends already satisfied this.
* Gridstore: rename inner DynamicGridstore to Gridstore
The dynamic variant keeps the Gridstore name; the outer dispatching type
will be renamed to Blobstore in a follow-up. Until then the inner type is
referred to as dynamic::Gridstore to distinguish it from the outer type.
* Gridstore: rename append-only variant to Arenastore
The append-only variant stores all value data in a single ever-growing
page, allocating space by appending, hence: arena store.
* Gridstore: rename outer storage type to Blobstore
The outer type dispatching between the two storage variants is now called
Blobstore, being more generic than Gridstore. This frees up the Gridstore
name, which now exclusively refers to the dynamic mode variant, next to
Arenastore for the append-only variant. Storage components keep using the
outer type, so they now use Blobstore.
The gridstore crate name, GridstoreError, and the persisted names
(config.json mode, payload config storage_type) are unchanged.
* Gridstore: split Gridstore and Arenastore into dedicated modules
The outer module is now blobstore, matching the Blobstore type it
defines. The two storage variants each get their own submodule: the
dynamic Gridstore moves from dynamic.rs into gridstore/ with its reader
and view extracted from the shared files, mirroring the arenastore/
module (previously append_only/) which already had this layout.
* Rename gridstore crate to blobstore
The crate is named after the outer Blobstore storage type it provides.
The gridstore name lives on in the dynamic mode variant. GridstoreError
and the persisted names (config.json mode, payload config storage_type)
are unchanged.
* Arenastore: pack values back to back across multiple pages
Drop the block alignment from the append-only mode: values are packed
byte to byte, without blocks, and the tracker offset is now a plain byte
offset within the page. Blocks and regions are dynamic mode concepts;
their page size constraints no longer apply to append-only configs.
Bring back support for multiple pages. Once appending a value would
grow the current page beyond the configured page size, a new page is
started, bounding the size of and the number of appends to each file:
object stores like S3 Express limit the number of appends per object.
A value larger than the page size gets a page of its own; values never
span pages.
A rollover creates the new, empty page file at put time; the value data
itself stays buffered until the next flush, which appends to each
touched page with a single write, using per-page watermarks captured at
flusher creation. The reader scans for consecutively numbered page
files when opening, validates the most recent mappings against them,
and adopts pages created since on a live reload.
* Blobstore: rename dynamic mode to mutable
Rename Mode::Dynamic to Mode::Mutable, and the persisted config value
with it: config.json now writes "mode": "mutable". There is no
compatibility alias for "dynamic", released versions never wrote the
mode field (a missing field still defaults to mutable), only unreleased
storages did.
The Gridstore type and module names for the mutable variant are
unchanged.
* Fix Edge compilation due to package rename
* Review remarks
* Extract Gridstore preopen into module
* Rename Arenastore files
* Use universal IO for append operations
* Rename GridstoreError to BlobstoreError
The error type belongs to the Blobstore crate and is shared by both the
Gridstore and Arenastore variants, so it follows the crate naming. Also
update the user-facing error messages that referred to the old name.
* Split config into per-variant types
* Rename Arenastore to Logstore
Rename the Arenastore type to Logstore, including the reader, view,
config, module and variant names. The storage file names follow:
log_page_{n}.dat and log_tracker.dat. The persisted mode tag stays
"append_only".
* Move bitmask module into the Gridstore variant
The bitmask tracks free blocks, which only exists in the mutable mode.
Move the module from the crate root into the Gridstore variant that
owns it. It stays re-exported at the crate root because the bitmask
benchmark needs a public path.
* Move pages module into the Gridstore variant
Like the bitmask, the block based pages module is only used by the
mutable mode. Move it from the crate root into the Gridstore variant
that owns it. The Logstore variant has its own page implementation.
* Use universal IO for every Logstore operation
Replace the direct_io module with universal IO in the append-only
tracker, making the whole Logstore go through a universal IO backend
bounded by UniversalRead and UniversalAppend:
- The tracker is generic over the backend now. Reads go through
UniversalRead with the caller's access pattern, flushes land as one
atomic append with the same offset compare-and-swap recovery as the
pages: a retried append after a lost acknowledgement is adopted
instead of appended twice. A torn trailing entry is still truncated
away on writable open, through a fresh handle since shrinking is not
supported through an open one.
- The reader now schedules a prefetch for the tracker file too, it no
longer bypasses the backend.
- The config write, clear and wipe use the backend file operations
instead of local filesystem calls, matching the Gridstore variant.
* Batch reads in Logstore read_values
Apply the same batching logic as the Gridstore variant: resolve all
mappings first, then fetch the value data, both through the backend's
read pipeline so async backends can serve the reads in parallel.
The tracker gains a batched lookup mirroring the mutable tracker's
iter, serving pending mappings and out of range point offsets directly
from memory. The pages gain a batched value read; unflushed values are
served from the in-memory buffers, and since values never span pages
each value is a single read without reassembly.
Like in the Gridstore variant, the callback may now be invoked in a
different order than the requested point offsets.
* Better describe logstore live reload ordering
* use enum for options, swap `*Options`<->`*Config` naming
* don't wrap enum in struct
* ditch unused `StorageConfig`, make deserialization more ergonomic
* rename `*Options`->`*Config`
* make `preopen` non-blocking
* fixup! ditch unused `StorageConfig`, make deserialization more ergonomic
* fixup! use enum for options, swap `*Options`<->`*Config` naming
* fixup! don't wrap enum in struct
* fix rebase
* use `populate` param in Logstore
* test: failing repro of stale page after live reload across rollover
A reader that live-reloads between a page rollover and the following
flush adopts the new, still empty page. The previous page is then no
longer the last one and is never reloaded again, so the tail that the
next flush appends to it stays invisible to the reader forever:
value pointer at byte 100 with length 100 is out of range
AppendOnlyPages::live_reload only reloads the last held page, assuming
earlier pages never change once a newer page exists. But the rollover
creates the new page file eagerly at put time, while the previous
page's buffered tail only lands at the next flush (see
test_rollover_writes_no_value_data_before_flush), so a page can keep
growing on disk after its successor exists.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: reload all pages that grew
* use Fs in `open_or_create`
* fix: publish tracker mappings only after the pages reload
`AppendOnlyTracker::live_reload` observed the mapping count and made it
visible in one step, before `LogstoreReader::live_reload` reloaded the
pages. Every failure path in the page reload -- `list_files`, reopening a
grown page, opening an adopted one, the truncation check -- therefore left
the reader with mappings referencing value data it never loaded, so reads
in the new offset range fail until a later reload happens to succeed. The
edge refresh loop keeps a segment whose reload failed, expecting it to keep
serving its pre-refresh state, which it then does not.
Split observing from publishing: `reload_count` refreshes the handle and
returns the count as a `PendingReload` token, `commit_reload` publishes it.
The reader still observes the tracker first, as the writer persists pages
before the mappings referencing them, but only commits once the pages are
loaded. Reopening without committing is harmless: reads stay bounded by the
unchanged count, and the bytes below it never change.
A partial failure inside the page reload needs no unwinding, pages running
ahead of the tracker is the safe direction.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf: batch the value reads in Logstore iteration
`LogstoreView::iter_range`, the path behind `Logstore::iter` and
`LogstoreReader::iter`, fetched the mappings for the whole range with a
single read but then read the values themselves one at a time, serially.
Gridstore routes its `iter` through `read_values` and pipelines both stages,
so a full scan of an append-only storage was the one read path without
batching -- one blocking round trip per value on the object store backends
this variant exists for. It is reached by payload storage iteration and by
the payload index build, which scans every payload.
Feed the pointers into `read_batch_values` instead, keeping the single
contiguous tracker read, which is better than the per-offset pipeline
scheduling Gridstore does on that side.
Values are now delivered through the read pipeline, so the callback may be
invoked out of order, as it already could be for Gridstore's `iter` and for
`read_values` in both variants. Both segment callers are order independent.
Tests that happened to rely on the mmap backend completing reads in
scheduling order now sort before comparing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: don't run the failed-page-reload test on Windows
The test shrinks a page file out of band to make the page reload fail, but
Windows refuses to resize a file while the reader holds it mapped, which it
does by construction here: "the requested operation cannot be performed on a
file with a user-mapped section open". The panic is on the injection itself,
the code under test never runs.
There is no portable injection. Truncating a page the reader holds is what
the check under test detects, so the mapping cannot be avoided; failing the
adopted page open instead needs a listed but unopenable file, and
`local_list_files` descends into matching directories rather than listing
them; failing the directory listing needs the storage directory removed,
which Windows also refuses while pages are mapped.
The storage itself is fine on Windows, its append path grows mapped pages
there and every other Logstore test passes. The logic under test is platform
independent and stays covered elsewhere, with the tracker half of the
guarantee pinned by `test_live_reload`, which runs on every target.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* feat: slice filtering condition for sliced scroll and deterministic sampling
Add a `slice` filter condition selecting points where
`stable_hash(point_id) % total == index`. The hash is SipHash-2-4 with a
zero key over canonical id bytes (8 LE bytes for numeric ids, 16 RFC 4122
bytes for UUIDs) — a frozen public contract, independent of the internal
resharding ring hash, reproducible by clients to predict membership.
For a fixed `total`, slices are disjoint and cover all points, enabling
parallel scroll streams (ES sliced-scroll style) and reproducible sampling
that composes with any other filter condition.
- REST: `{"slice": {"total": N, "index": R}}`; gRPC: `SliceCondition` in
the condition oneof (tag 8)
- Evaluated per point via id_tracker external-id lookup; no payload index
needed; cardinality estimated as `points / total` with no primary clause
- `total >= 1` enforced by NonZeroU32 at parse time, `index < total` by
validation in both REST and gRPC paths
- Hash contract locked by test vectors independently reproduced with a
reference SipHash-2-4 implementation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* tests: minimal OpenAPI test for slice filter condition
Scrolls all slices of a fixed total over numeric + UUID ids asserting
disjointness and full coverage, checks must_not inversion, and pins the
two rejection paths (422 for index >= total, 400 for total = 0). Requests
and responses are validated against the regenerated OpenAPI spec by the
test harness.
Note: the spec cannot itself reject total = 0 client-side — the Condition
anyOf falls through to the permissive Filter schema, as with any invalid
condition — so rejection is asserted via the server response.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Introduce request-specific LoadProfile with per-component placement
A read-only shard opened for one known request (the serverless cold-start
path) doesn't have to warm components the request will never touch.
LoadProfile captures that from the request: warm components keep the
persisted-config placement, everything else is parked cold. All placement
decisions live in one place, so the memory placement of a whole segment
under a profile is reviewable in one file.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais
* Thread LoadProfile through the read-only segment open
ReadOnlySegment::open takes an optional profile; first_preopen and
open_via resolve it into per-component populate overrides so the opens
make the same placement decisions the prefetches did. Pinned components
that materialize on open regardless (quantized RAM storage kinds, the
immutable-RAM sparse index) and appendable components ignore the
override; the HNSW graph and immutable payload indexes demote fully.
Config reloads follow the new config alone and pass no override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais
* Open ReadOnlyEdgeShard under a request-derived load profile
ReadOnlyEdgeShard::open takes an optional LoadProfile, applies it to
every segment open and keeps it so segments discovered by a later
refresh load with the same placement. ScrollRequestInternal and
CoreSearchRequest gain load_profile() constructors, and edge-shard-query
builds the request before the open and passes its profile (opt out with
--no-load-profile).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais
* Demote pinned quantized vectors and sparse index under a cold profile
Within the immutable layout the quantized RAM and mmap loaders share the
on-disk format — only how the data is brought into memory differs — and
the immutable-RAM sparse index has the same lazy mmap open low-memory
mode already downgrades to. So a cold populate override now demotes the
effective placement itself (Memory::with_populate_override, shared with
the HNSW residency mapping) instead of only skipping cache priming: a
pinned quantized storage opens the mmap kind cold, and a pinned sparse
index opens as Mmap, so neither reads its data on a cold start.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais
* Add LoadProfile::merge for composite queries
A composite query runs multiple core requests — e.g. a hybrid search
runs one core search per vector, each with its own filter. Its profile
is the union of its parts': merge extends the warm sets and ORs the
payload-storage flag, so a component either part needs warm stays warm.
The union is sound because every placement method is monotone in the
warm sets (growing them only turns "park cold" into "keep configured
placement"), so the merged profile dominates each input; and minimal,
warming nothing no part asked for.
Combine profiles with reduce, not fold: merge's identity element is
the coldest profile (empty warm sets), the opposite of passing no
profile at all — deliberately no empty()/Default constructor exists.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Adapt vanished-segment test to the profile-aware open signature
The test landed on dev (#9777) after the load-profile signature change
was written, so the rebase left its ReadOnlySegment::open calls without
the new load_profile argument.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Defer vector index open entirely under a cold load profile
A cold placement is not enough for the vector index on remote backends:
GraphLinksView requires the whole links file as one contiguous slice,
and the disk cache can only lend a borrowed slice once every block is
locally present — so even a Cold HNSW open mirrors the entire
links_compressed.bin (8.2 MB / 1.1 s per segment in the serverless
cold-start trace), plus the unconditional graph.bin metadata read.
The only way not to fetch the index is not to open it.
LoadProfile::vector_index_placement is replaced by
vector_index_deferred: a vector the request never scores now gets a
DeferredVectorIndex — a new VectorIndexReadEnum variant holding the
open arguments (an owned clone of the segment's raw backend, path,
config, shared component handles) and a OnceLock. Nothing is opened or
prefetched for it at segment open.
Per-method policy of the deferred variant:
- search, fill_idf_statistics and populate open the index on first use
(with the cold placement the profile chose), so the profile contract
holds: a request the profile did not predict still works, just pays
the open then;
- is_index reports true without opening (deferral only ever wraps a
real HNSW or sparse index; plain opens no files and is never
deferred);
- telemetry, indexed_vector_count and sizes answer conservative
defaults rather than trigger a remote fetch for a statistic.
Tests: deleting the vector_index directory before an open under a
scroll profile leaves open, filtered reads and payload reads working —
proof that nothing of the index is read — while a search surfaces the
missing files; and a segment opened under a scroll profile answers
searches identically to an eagerly opened one via the transparent
first-use open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Make --vector optional in edge-shard-query: random query after open
Omitting --vector on the search sub-command now searches with a random
vector. The request is still built before the shard opens — the load
profile only needs the vector name, not its values — with an empty
placeholder; once the shard is open, fill_random_vector reads the
dimension of the queried vector from the derived shard config and
fills in uniform-random f32s (with a clear error if the named vector
is not in the config).
The vector is generated once, so live-reload iterations re-run the
identical random query and the printed diffs stay meaningful.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Scope index deferral to the HNSW graph via a lazy OnceLock load
Replace the DeferredVectorIndex wrapper (and the VectorIndexReadEnum::
Deferred variant) with deferral inside ReadOnlyHNSWIndex itself: the
graph lives in a OnceLock (same first-wins arbitration as
ReadOnlyRoaringFlags::bitmap) alongside the retained raw backend and
residency, and loads on first use with a cold placement. The config
read stays eager — one tiny, absence-tolerated file — so telemetry,
is_on_disk and indexed_vector_count report real values where the
Deferred arms answered with hard defaults.
The sparse index needs no deferral: its mmap open reads lazily, with
only small JSON metadata eager. A profile that never scores the vector
now passes a cold placement override (LoadProfile::
vector_index_placement) into the eager open_sparse, which demotes
ImmutableRam to the lazy Mmap open like low-memory mode.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Express HNSW graph deferral as a populate override, not a bool param
Replace the `deferred: bool` on the read-only HNSW open/preopen (and
the VectorIndexReadEnum pass-through) with the same
`populate_override: Option<Populate>` every other component takes. A
cold *override* defers the graph load — graph_deferred() mirrors the
cold-override match of open_sparse — while a config-derived cold
placement (or the low-memory clamp) keeps the eager load, since only a
request-specific override carries the "never scored" prediction.
With dense and sparse now consuming the same signal,
LoadProfile::vector_index_deferred is gone: a single
vector_index_placement() serves both index kinds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Serialize the deferred graph load via once_cell's get_or_try_init
Loading outside the lock (std OnceLock's fallible init is still
unstable) let a search burst on a deferred vector fetch the whole
graph once per thread. Swap the cell for once_cell::sync::OnceCell:
the fallible load runs inside the cell's lock, concurrent first users
block on the one load, and a failed load leaves the cell empty so the
next caller retries.
Addresses https://github.com/qdrant/qdrant/pull/9797#discussion_r3573727836
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat: read-only edge grouping/matrix search + object-storage read path
Add query_groups (group by a payload field) and search_matrix (single-shard distance matrix over a random sample) to the read-only edge shard's EdgeShardRead API, in new grouping and matrix modules plus edge test helpers.
Make the object-storage read path available outside tests: drop the #[cfg(test)] gate on the BlobFile UniversalReadExt impl and move io_bridge_object_store/object_store to segment's normal dependencies, so a ReadOnlyEdgeShard can serve segments read from S3.
* Share group-by building blocks between server and edge
Move GroupsAggregator, group candidate query shaping (is-empty filter,
group_by payload selector, prefetch limit scaling) and result-order
derivation into shard::grouping / shard::query, so the collection and
edge grouping implementations cannot silently diverge.
Edge grouping now handles multi-valued group keys, u64 keys, wildcard
group_by paths, prefetch limits and score-ordered groups the same way
as the server.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drive group-by through a shared sans-IO state machine
Extract the multi-request collect/fill loop into
shard::grouping::GroupByDriver: next_request() yields shaped backend
queries, add_points() advances the state, distill() returns the groups.
Query execution stays with the caller, so the async server path and the
sync edge path drive the same machine, and the request shaping helpers
become private to shard::grouping.
Edge now uses the same request budget (5 collect + 5 fill requests) and
per-request candidates limit (groups * group_size, computed inside the
driver) as the server, replacing its single 4x-oversampled request.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [AI + manual] initial impl
[AI] read_batch which actually batches
manual nits
[AI] better handling of local and remote paths
manual refactor, respect open options
don't delete local file
dumbify read_batch
we want to refactor it anyway
simplify
rename to `DiskCache` in `simple_disk_cache` module
* refactor to use always use ReadPipeline
pass meta to remote pipeline
* nits
* run tests for more Remotes
* fix no more <T> in UniversalRead
* fmt
* chore(deps): unify roaring as workspace dep, move duplicate to dev-deps
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* propagate to BufferedDynamicFlags
* use duplicate for tests
* propagate to Bitvec/Roaring flags
* propagate to RoaringFlags
* fixup! propagate to BufferedDynamicFlags
* propagate to BitvecFlags
* Skip broken tests
* Add rocksdb dropper
This is a temporary tool. It will be removed later in this PR.
* [automated] Drop rocksdb
This commit is made by running tools/rocksdb/drop.sh
* Touch-up after ast-grep
The previous automated commit removed items, but not their comments.
Also, some blocks are left with only one item.
Also, ast-grep can't handle macros like `vec![]`.
This commit completes the job.
* Remove RocksDB dropper
* Remove mentions of rocksdb feature in CI
* Fix clippy warnings
These `FIXME` comments added previously in this PR by ast-grep by
"peeling" cfg_attr like this:
-#[cfg_attr(not(feature = "rocksdb"), expect(...))]
+#[expect(...)] // FIXME(rocksdb): ...
This commit removes these allow/expect attributes and fixes clippy
lints.
* Remove leftover rocksdb-related code
Removed:
- Cargo.toml: rocksdb cargo feature flag and dependencies.
- code: rocksdb-related items that was not under feature flag.
- flags.rs: rocksdb-related qdrant feature flags.
Disabled:
- Some benchmarks because these do not compile now.
* Print warning if rocksdb leftovers found in snapshots
* Remove Clone from tokenizer
* Regenerate openapi.json
Instead of `IdTracker` to have `iter_*` methods, it now has a `point_mappings()` method, returning a guard to `PointMapping` value. It prepares for moving the point_mappings under a lock to allow parallel searches and deletions.
Also, an incorrect unsafe in `Segment::iter_points` is replaced by a safe version with `self_cell`.
* chore: gitignore .idea and .vscode globally
Reason: `lib/edge/publish` is a separate Cargo workspace and I'd like to
have a separate rust-analyzier configuration for it, which is stored in
`.vscode/settings.json`. Not sure about `.idea`, but I guess the same
logic applies.
* chore: Use Iterator::is_sorted (stabilized in Rust 1.82)
* chore: remove unrelated files
These were added in #5501 (42fd2e27), perhaps by accident?
* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
