* Make deprecated `on_disk_payload` optional in the API schema
`CollectionParams::on_disk_payload` was the last place the deprecated flag was
still a bare bool, in both the REST schema and the gRPC `CollectionParams`
message. Clients generated from those schemas model it as a required bool, so
removing the field in a future version would break them.
Make it optional in both schemas while keeping it populated on every path that
builds `CollectionParams`, so responses and the persisted collection config
still carry a value and existing clients keep working until it is removed.
The gRPC change is wire-compatible: proto3 `optional` only adds a synthetic
oneof for presence tracking, the field number and wire type are unchanged. It
does change what an absent field means, though, so decode absent as `false` when
reading a remote peer's collection info -- peers that predate the change encode
a plain bool, which is omitted from the wire when false.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Keep gRPC `on_disk_payload` a plain bool
Adding proto3 `optional` would let a new client tell "unset" from `false`, but
it also changes what an absent field means. A pre-upgrade server encodes a plain
bool, which is omitted from the wire when false, so a client generated from the
new proto would decode `false` as "unset" -- and upgrading the client first is
the order we recommend.
gRPC does not need the change anyway: protobuf tolerates a missing field by
design, so a gRPC client does not break at runtime when we stop sending this
one. The breakage this addresses is on the REST side, where a generated model
with a required non-nullable bool fails to deserialize a response that omits the
field. JSON encodes `false` explicitly, so the REST schema change carries no
equivalent ambiguity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Gridstore: introduce storage operating mode in config
Add a mode field to the gridstore config, selecting between the dynamic
mode (current behavior, the default) and the upcoming serverless mode.
The mode is specified through StorageOptions on creation, persisted in
config.json, and read back first when opening so the correct variant can
be selected automatically. Configs written before this field existed
deserialize as dynamic.
For now, selecting the serverless mode returns an error; the variant
itself is added in follow-up commits.
* Gridstore: move dynamic implementation into dedicated module
Mechanical move of the current Gridstore implementation into
gridstore/dynamic.rs as DynamicGridstore. The public Gridstore struct
becomes a thin wrapper holding a mode variant enum, propagating every
call into the selected variant. For now the enum only has the dynamic
variant; the serverless variant is added in follow-up commits.
No logic changes to the dynamic implementation itself: only visibility,
the config parameter now passed into open (the wrapper reads it first to
select the mode), and open_or_create staying on the wrapper.
* Gridstore: add serverless tracker
Add the append-only mapping tracker for the serverless storage mode.
The tracker file is a plain array of 16-byte mapping entries without any
header: the number of mappings is defined by the exact file length, and
the entry index is the point offset. The file starts empty and only ever
grows by appending, existing bytes are never rewritten. Mappings must be
set in monotonically increasing point offset order; skipped offsets are
backfilled as zeroed entries which decode as None.
New mappings are buffered in memory and appended with a single write per
flush. A flush with a stale target is a no-op so bytes are never written
twice. A torn trailing entry (file length not a multiple of the entry
size) is ignored when reading and truncated away when opening writable.
Unlike the dynamic tracker, the file is read and written directly with
positional file IO instead of memory mapping, as serverless environments
do not handle memory mapped files well.
* Gridstore: add serverless storage variant
Add the append-only gridstore variant for serverless deployments, which
restrict IO to appending to files: existing bytes can never be
rewritten, and IO is expensive so as few files as possible are used.
The variant stores all value data in a single page file next to the
serverless tracker and the storage config, three files in total. Both
data files start empty and only ever grow by appending; there is no
preallocation, no used-block bitmask and no gap/region bookkeeping.
Values are appended at put time at the next block aligned offset, with
the zero padding included in the write so it lands exactly at the end
of the file. Mappings are buffered and appended to the tracker with a
single write per flush, after the page file is synced, so a mapping on
disk never points at data that is not durable.
Values cannot be updated or deleted, and must be put at monotonically
increasing point offsets; violations are rejected before any data is
written. Files are read and written directly, never memory mapped.
The mode is selected through StorageOptions on creation and picked up
automatically from the persisted config when opening.
* Gridstore: serverless support in reader and view
Extend the read-only GridstoreReader and the GridstoreView with the
serverless mode, keeping both public types unchanged: like the writable
Gridstore they now hold a mode variant internally, selected
automatically from the persisted config when opening.
The serverless reader holds the tracker and page directly and reads the
files positionally, without memory mapping. A live reload re-reads the
mapping count from the exact tracker file length (there is no size
header), ignoring a torn trailing entry, and never truncates as it is
read-only. Value reads always go directly to the file, so newly
appended data is readable without remapping anything.
* Gridstore: document storage operating modes
* Gridstore: review fixes for the serverless mode
Hardening and cleanup from a review pass over the new serverless
storage variant:
- Batch the reader side iteration like the writer already did, instead
of materializing tracker mappings for the full range in one go, which
could transiently allocate gigabytes on large storages.
- Recover the append cursors when a positional write fails partway:
truncate the file back to the tracked length so a retried append or
flush never rewrites bytes that already landed in the file.
- Validate page addressability before appending value data, a rejected
put must not grow the page file.
- Cross-check tracker and page consistency when opening: mappings that
reference value data past the end of the page file (e.g. after a
partial copy or restore) now fail fast instead of surfacing as
opaque read errors per point.
- Reject value pointers into any page other than page 0 on the
serverless read path with PageNotFound, matching the dynamic mode
contract, instead of silently reading from a wrong location.
- Refresh the reported storage size on reader live reload even when no
new mappings were flushed, unflushed value data may have grown the
page file already.
- Validate configs read from disk: a corrupt config with zero sized
blocks, pages or regions is now rejected when opening instead of
panicking on a division by zero later.
- Classify rejected serverless puts as UnsupportedOperation, consistent
with rejected deletes, so they don't surface as user-facing
validation errors at the segment level.
- Deduplicate the compression dispatch into Compression::compress and
Compression::decompress, and the serverless file create/open patterns
into shared direct IO helpers, so the two modes and files can't
silently drift apart.
* Gridstore: cover both operating modes in mode-agnostic tests
Parameterize the gridstore tests that exercise mode-agnostic behavior
over both the dynamic and serverless mode with rstest, using a
single and bulk put/get roundtrips, storage files, basic persistence,
corrupt config rejection, batched read congruence, reader live reload,
and the different block sizes.
Mode specific expectations branch inside the tests: expected file
names, storage size semantics (whole blocks vs exactly packed bytes),
value pointer layout (page spill over vs a single packed page), and
gaps (created by deletes in dynamic mode, by skipped puts in serverless
mode). Dynamic-only internals assertions are kept behind a mode check.
Tests around updates, deletes, page spanning, block reuse and other
dynamic-only behavior intentionally stay dynamic; the serverless
specific format invariants remain covered by the dedicated serverless
tests.
* Gridstore: port serverless specific tests from sibling branch
Source the serverless specific test cases that the
serverless-gridstore-updates branch added, adapted to the dedicated
variant implemented here (distinct file names, headerless tracker
with 16 byte entries, a single packed page without trailing padding,
and rejected re-puts):
- writes only ever append: tracker and page files only grow and
previously written bytes stay byte-for-byte untouched
- new mappings land exactly at the end of the tracker file, which
always covers the exact number of mappings
- mapping gaps are zero-padded on disk and survive reopening
- values are packed back to back at block aligned offsets, the page
file ends exactly at the last value
- serverless mode never creates nor reports block flag files
- a flusher persists exactly the mappings that existed at its
creation, later puts stay pending
- a config claiming the wrong mode fails loudly in both directions
instead of loading the incompatible file format of the other mode
Tests around their mode switching, page spanning and tolerated deletes
don't apply to this design and are intentionally not ported.
* Gridstore: test serverless production risk scenarios
Add tests for the operational aspects that matter before serverless
mode goes to production, each covering a scenario that wasn't
evaluated yet:
- Replayed puts of already persisted offsets (a WAL redo after a
crash where the flush completed but was never acknowledged) are
rejected without appending anything, and max_point_offset is the
exact offset a replay must resume at.
- The accepted crash case of a tracker file extended with zeroed
bytes: the entries count as permanent None mappings, can never be
put again, and the storage stays consistent and writable past them.
- The read-only reader never modifies the files: opening over a torn
tracker tail, reading, iterating and live reloading leave both
files byte-for-byte untouched.
- A multi-round put/flush/reopen cycle always exposes exactly the
flushed prefix, with the mapping count matching the exact tracker
file length and unflushed offsets reusable.
- An append beyond the maximum addressable block offset is rejected
before writing anything, keeping retried puts from growing the page
file unboundedly.
* Gridstore: rename serverless mode to append-only, split into module
Rename the mode after its defining characteristic instead of its
deployment target: files only ever grow, existing bytes are never
rewritten. Renames Mode::Serverless to Mode::AppendOnly (persisted as
"mode": "append_only") and the on-disk file names to
append_only_tracker.dat and append_only_page_0.dat. The serverless
deployment motivation stays in the documentation.
Also split the single 2300 line serverless.rs into an append_only
module with dedicated files for the storage, page, view, reader and
tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use universal IO in Gridstore
* Include upstream preopen logic in new Gridstore variant
* Gridstore: buffer append-only value writes until flush
In append-only mode, put previously wrote the value data to the page
file right away, one write operation per put, while mappings were
already buffered and batch persisted on flush. Buffer value writes the
same way: both the value and its mapping now only land on disk once a
flush cycle executes.
This batches all new value data into a single write operation per
flush, which is significantly more efficient on S3 based storage where
every write is a costly operation. A flush now performs exactly two
writes: one appending all buffered value data to the page file, one
appending all pending mappings to the tracker file, in that order, so
a mapping on disk never points at value data that is not durable.
The page mirrors the tracker's pending mechanism: an in-memory buffer
that is byte for byte the next append (zero padding between block
aligned values included), a watermark captured at flusher creation so
puts made during a flush stay buffered, a stale-flush no-op guard so
appended bytes are never written twice, and truncate-back recovery on
failed writes. Reads transparently serve buffered values from memory.
As a side effect, a crash between flushes now leaves nothing on disk
at all, where the write-through approach left orphaned value bytes in
the page file. The buffered data is held in memory until the next
flush, bounded by the flush cadence.
Universal IO filesystem handles are now required to be Send + Sync, so
the flusher closure can carry one to grow the page file at flush time;
all existing backends already satisfied this.
* Gridstore: rename inner DynamicGridstore to Gridstore
The dynamic variant keeps the Gridstore name; the outer dispatching type
will be renamed to Blobstore in a follow-up. Until then the inner type is
referred to as dynamic::Gridstore to distinguish it from the outer type.
* Gridstore: rename append-only variant to Arenastore
The append-only variant stores all value data in a single ever-growing
page, allocating space by appending, hence: arena store.
* Gridstore: rename outer storage type to Blobstore
The outer type dispatching between the two storage variants is now called
Blobstore, being more generic than Gridstore. This frees up the Gridstore
name, which now exclusively refers to the dynamic mode variant, next to
Arenastore for the append-only variant. Storage components keep using the
outer type, so they now use Blobstore.
The gridstore crate name, GridstoreError, and the persisted names
(config.json mode, payload config storage_type) are unchanged.
* Gridstore: split Gridstore and Arenastore into dedicated modules
The outer module is now blobstore, matching the Blobstore type it
defines. The two storage variants each get their own submodule: the
dynamic Gridstore moves from dynamic.rs into gridstore/ with its reader
and view extracted from the shared files, mirroring the arenastore/
module (previously append_only/) which already had this layout.
* Rename gridstore crate to blobstore
The crate is named after the outer Blobstore storage type it provides.
The gridstore name lives on in the dynamic mode variant. GridstoreError
and the persisted names (config.json mode, payload config storage_type)
are unchanged.
* Arenastore: pack values back to back across multiple pages
Drop the block alignment from the append-only mode: values are packed
byte to byte, without blocks, and the tracker offset is now a plain byte
offset within the page. Blocks and regions are dynamic mode concepts;
their page size constraints no longer apply to append-only configs.
Bring back support for multiple pages. Once appending a value would
grow the current page beyond the configured page size, a new page is
started, bounding the size of and the number of appends to each file:
object stores like S3 Express limit the number of appends per object.
A value larger than the page size gets a page of its own; values never
span pages.
A rollover creates the new, empty page file at put time; the value data
itself stays buffered until the next flush, which appends to each
touched page with a single write, using per-page watermarks captured at
flusher creation. The reader scans for consecutively numbered page
files when opening, validates the most recent mappings against them,
and adopts pages created since on a live reload.
* Blobstore: rename dynamic mode to mutable
Rename Mode::Dynamic to Mode::Mutable, and the persisted config value
with it: config.json now writes "mode": "mutable". There is no
compatibility alias for "dynamic", released versions never wrote the
mode field (a missing field still defaults to mutable), only unreleased
storages did.
The Gridstore type and module names for the mutable variant are
unchanged.
* Fix Edge compilation due to package rename
* Review remarks
* Extract Gridstore preopen into module
* Rename Arenastore files
* Use universal IO for append operations
* Rename GridstoreError to BlobstoreError
The error type belongs to the Blobstore crate and is shared by both the
Gridstore and Arenastore variants, so it follows the crate naming. Also
update the user-facing error messages that referred to the old name.
* Split config into per-variant types
* Rename Arenastore to Logstore
Rename the Arenastore type to Logstore, including the reader, view,
config, module and variant names. The storage file names follow:
log_page_{n}.dat and log_tracker.dat. The persisted mode tag stays
"append_only".
* Move bitmask module into the Gridstore variant
The bitmask tracks free blocks, which only exists in the mutable mode.
Move the module from the crate root into the Gridstore variant that
owns it. It stays re-exported at the crate root because the bitmask
benchmark needs a public path.
* Move pages module into the Gridstore variant
Like the bitmask, the block based pages module is only used by the
mutable mode. Move it from the crate root into the Gridstore variant
that owns it. The Logstore variant has its own page implementation.
* Use universal IO for every Logstore operation
Replace the direct_io module with universal IO in the append-only
tracker, making the whole Logstore go through a universal IO backend
bounded by UniversalRead and UniversalAppend:
- The tracker is generic over the backend now. Reads go through
UniversalRead with the caller's access pattern, flushes land as one
atomic append with the same offset compare-and-swap recovery as the
pages: a retried append after a lost acknowledgement is adopted
instead of appended twice. A torn trailing entry is still truncated
away on writable open, through a fresh handle since shrinking is not
supported through an open one.
- The reader now schedules a prefetch for the tracker file too, it no
longer bypasses the backend.
- The config write, clear and wipe use the backend file operations
instead of local filesystem calls, matching the Gridstore variant.
* Batch reads in Logstore read_values
Apply the same batching logic as the Gridstore variant: resolve all
mappings first, then fetch the value data, both through the backend's
read pipeline so async backends can serve the reads in parallel.
The tracker gains a batched lookup mirroring the mutable tracker's
iter, serving pending mappings and out of range point offsets directly
from memory. The pages gain a batched value read; unflushed values are
served from the in-memory buffers, and since values never span pages
each value is a single read without reassembly.
Like in the Gridstore variant, the callback may now be invoked in a
different order than the requested point offsets.
* Better describe logstore live reload ordering
* use enum for options, swap `*Options`<->`*Config` naming
* don't wrap enum in struct
* ditch unused `StorageConfig`, make deserialization more ergonomic
* rename `*Options`->`*Config`
* make `preopen` non-blocking
* fixup! ditch unused `StorageConfig`, make deserialization more ergonomic
* fixup! use enum for options, swap `*Options`<->`*Config` naming
* fixup! don't wrap enum in struct
* fix rebase
* use `populate` param in Logstore
* test: failing repro of stale page after live reload across rollover
A reader that live-reloads between a page rollover and the following
flush adopts the new, still empty page. The previous page is then no
longer the last one and is never reloaded again, so the tail that the
next flush appends to it stays invisible to the reader forever:
value pointer at byte 100 with length 100 is out of range
AppendOnlyPages::live_reload only reloads the last held page, assuming
earlier pages never change once a newer page exists. But the rollover
creates the new page file eagerly at put time, while the previous
page's buffered tail only lands at the next flush (see
test_rollover_writes_no_value_data_before_flush), so a page can keep
growing on disk after its successor exists.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: reload all pages that grew
* use Fs in `open_or_create`
* fix: publish tracker mappings only after the pages reload
`AppendOnlyTracker::live_reload` observed the mapping count and made it
visible in one step, before `LogstoreReader::live_reload` reloaded the
pages. Every failure path in the page reload -- `list_files`, reopening a
grown page, opening an adopted one, the truncation check -- therefore left
the reader with mappings referencing value data it never loaded, so reads
in the new offset range fail until a later reload happens to succeed. The
edge refresh loop keeps a segment whose reload failed, expecting it to keep
serving its pre-refresh state, which it then does not.
Split observing from publishing: `reload_count` refreshes the handle and
returns the count as a `PendingReload` token, `commit_reload` publishes it.
The reader still observes the tracker first, as the writer persists pages
before the mappings referencing them, but only commits once the pages are
loaded. Reopening without committing is harmless: reads stay bounded by the
unchanged count, and the bytes below it never change.
A partial failure inside the page reload needs no unwinding, pages running
ahead of the tracker is the safe direction.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf: batch the value reads in Logstore iteration
`LogstoreView::iter_range`, the path behind `Logstore::iter` and
`LogstoreReader::iter`, fetched the mappings for the whole range with a
single read but then read the values themselves one at a time, serially.
Gridstore routes its `iter` through `read_values` and pipelines both stages,
so a full scan of an append-only storage was the one read path without
batching -- one blocking round trip per value on the object store backends
this variant exists for. It is reached by payload storage iteration and by
the payload index build, which scans every payload.
Feed the pointers into `read_batch_values` instead, keeping the single
contiguous tracker read, which is better than the per-offset pipeline
scheduling Gridstore does on that side.
Values are now delivered through the read pipeline, so the callback may be
invoked out of order, as it already could be for Gridstore's `iter` and for
`read_values` in both variants. Both segment callers are order independent.
Tests that happened to rely on the mmap backend completing reads in
scheduling order now sort before comparing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: don't run the failed-page-reload test on Windows
The test shrinks a page file out of band to make the page reload fail, but
Windows refuses to resize a file while the reader holds it mapped, which it
does by construction here: "the requested operation cannot be performed on a
file with a user-mapped section open". The panic is on the injection itself,
the code under test never runs.
There is no portable injection. Truncating a page the reader holds is what
the check under test detects, so the mapping cannot be avoided; failing the
adopted page open instead needs a listed but unopenable file, and
`local_list_files` descends into matching directories rather than listing
them; failing the directory listing needs the storage directory removed,
which Windows also refuses while pages are mapped.
The storage itself is fine on Windows, its append path grows mapped pages
there and every other Logstore test passes. The logic under test is platform
independent and stays covered elsewhere, with the tracker half of the
guarantee pinned by `test_live_reload`, which runs on every target.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Add doc comments to `StartResharding` fields so the generated OpenAPI
spec explains what a user has to pass, and regenerate the spec.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
`load_from_wal` splits WAL recovery in two: it replays
`[first_index, applied_seq + APPLIED_SEQ_SAVE_INTERVAL + 1)` synchronously and
hands the remaining tail to the update worker, which applies it in the
background *after* `LocalShard::load` has returned and the shard has started
serving reads.
That split was introduced by #8008 and applies to every collection, so up to
`update_queue_size - 1` operations already acknowledged to a client with
`wait=true` can be missing from reads right after a restart, reappearing one by
one as the worker catches up.
Only `prevent_unoptimized` needs that routing: the update worker signals the
optimizer per operation, and optimization is the only thing that makes deferred
points visible. Everywhere else the synchronous replay is sufficient, so gate
the use of `applied_seq` on the flag -- the same condition that already gates
the worker's deferred-points wait -- and replay the whole WAL before load
returns, as it did before #8008.
Found by the crasher: after a crash-restart cycle it reported 72 missing points
out of a confirmed 3202, with the shard counting 3202 points while 3930 had been
acknowledged and 332 WAL entries were still queued across two shards.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A transfer source that misses the transfer abort (e.g. while partitioned
or paused) keeps its local shard wrapped in a proxy. When such a peer can
only catch up via consensus snapshot, snapshot application re-creates
payload indexes with an update operation that the stale forward proxy
forwards to a transfer target which may no longer have the shard. The
resulting precondition error fails snapshot application and stops the
consensus thread ("No target shard N found for update"), leaving the
peer unable to ever catch up.
Snapshot application now explicitly cleans up transfers that are no
longer registered in consensus: the transfer task is stopped and the
proxy is reverted via the new `ShardReplicaSet::discard_proxy_local`,
which is infallible, never contacts the remote, and forgets queued
updates (replica states in the same snapshot already reflect the
transfer outcome).
The consensus test reproduces the incident: pause the transfer source
mid-transfer, restart the other peers so the aborted transfer can only
be learned via snapshot, and verify the source recovers.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* test(model_testing): add slice matcher to generated scroll filter
Extend ScrollFilter with a Slice variant so paginated scroll exercises
Condition::Slice. The generator draws small totals (1/2/3/4/5/8) and a
valid index; the model verifier mirrors membership via Slice::check —
the same hash contract the engine uses — so the existing paged-scroll
id-set assertion covers sliced scroll under soak (optimizer, WAL reload,
multi-shard, mixed UUID/numeric ids).
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(model_testing): add CountBySlice verification op
Exercise Condition::Slice through the exact count API under soak.
Shares the slice generator with ScrollPaged; the model oracle uses
Slice::check so engine and in-memory counts must agree.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(model_testing): compose slice with num on scroll and delete-by-filter
- ScrollFilter::NumAndSlice: indexed num drives candidates; slice is a
per-candidate check via Filter::merge.
- DeleteByFilter { num, slice: Option<Slice> }: half the deletes also
restrict by slice so submit-time filter resolution and WAL-replayed
id lists exercise Condition::Slice.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: slice filtering condition for sliced scroll and deterministic sampling
Add a `slice` filter condition selecting points where
`stable_hash(point_id) % total == index`. The hash is SipHash-2-4 with a
zero key over canonical id bytes (8 LE bytes for numeric ids, 16 RFC 4122
bytes for UUIDs) — a frozen public contract, independent of the internal
resharding ring hash, reproducible by clients to predict membership.
For a fixed `total`, slices are disjoint and cover all points, enabling
parallel scroll streams (ES sliced-scroll style) and reproducible sampling
that composes with any other filter condition.
- REST: `{"slice": {"total": N, "index": R}}`; gRPC: `SliceCondition` in
the condition oneof (tag 8)
- Evaluated per point via id_tracker external-id lookup; no payload index
needed; cardinality estimated as `points / total` with no primary clause
- `total >= 1` enforced by NonZeroU32 at parse time, `index < total` by
validation in both REST and gRPC paths
- Hash contract locked by test vectors independently reproduced with a
reference SipHash-2-4 implementation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* tests: minimal OpenAPI test for slice filter condition
Scrolls all slices of a fixed total over numeric + UUID ids asserting
disjointness and full coverage, checks must_not inversion, and pins the
two rejection paths (422 for index >= total, 400 for total = 0). Requests
and responses are validated against the regenerated OpenAPI spec by the
test harness.
Note: the spec cannot itself reject total = 0 client-side — the Condition
anyOf falls through to the permissive Filter schema, as with any invalid
condition — so rejection is asserted via the server response.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Benches: use SmallRng instead of ChaCha12-based generators
All benchmarks used StdRng or rand::rng() (ThreadRng), both backed by the
ChaCha12 block cipher in rand 0.10. Benchmarks do not need crypto-strength
randomness, and several draw random values inside the timed closure, so
cipher work was included in the measurement itself.
Switch every bench target to SmallRng (Xoshiro256++), and key the HNSW
graph cache and sparse index cache by RNG algorithm so stale caches built
from the old generator are not reused against newly generated vectors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Benches: replace free-function rand::random with local SmallRng
Addresses review: rand::random draws from the thread RNG (ChaCha12),
including inside the timed loop of the pq score benchmark.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Model tester: cover all quantization types
Add a `quantization` field to `VectorCandidate` so candidates can carry
any `QuantizationConfig` variant, and materialize the configs in the
fixture (`quantization_config`). The inline-storage vector "i" keeps its
scalar Int8 config, now declared on the candidate instead of hard-coded
in the fixture.
New candidates:
- "p" Dense(8) + Product x4
- "v" Dense(6) + Binary (non-byte-aligned dim, trailing-bit padding)
- "r" Dense(8) + Turbo (search-side TQ over Float32 storage)
Quantization x datatype combos:
- "l" Dense(6) Float16 + Binary
- "d" Dense(8) Turbo4 + Turbo default bits (keep-source-rotated branch
of `should_keep_source_rotated`)
- "g" Dense(8) Turbo4 + Turbo Bits1_5 (Padded rotation, rotate-back
branch)
This is model-safe: schema quantization keeps the original vectors, so
read-back predictions are untouched; the approximate quantized scoring
only feeds the membership-only Search/Query/Recommend checks.
`assert_candidates_predictable` enforces the wiring constraints:
quantization is dense-only (the fixture only wires the dense arm) and
requires `initially_active` (CreateVectorName's `DenseVectorConfig`
carries no quantization).
Verified: seeds 1/2/3/7/42 soaks (5k ops, restarts, optimizer on) green;
quantized codes confirmed on disk for every quantized candidate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Model tester: make inline_storage a VectorCandidate knob, enable on "l" and "d"
Replaces the fixture's name-based INLINE_STORAGE_VECTOR special case with an
inline_storage field on VectorCandidate (requires quantization, enforced by the
startup assert). Enables it on "l" (Float16 base + padded Binary links) and "d"
(Turbo4 base + TQ links) to cover more (base layout, link encoding) pairs of the
CompressedWithVectors format; "v", "r", "g" and "p" keep the non-inline paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Fix resharding, on queries filter shards on all shard selectors
* Add failing consensus test: search during resharding with shard keys (#9880)
Reproduces a known bug: after resharding is initialized on a custom
sharded collection with a shard key, searches (with and without the
shard key selector) fail with "does not have enough active replicas",
because the new resharding shard is included in reads before it has
an active replica.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Exempt explicit shard id selection from resharding read filter
Explicit shard id selection is only used by internal per-shard
operations (local shard API, internal gRPC reads), including the
resharding driver reading back migrated points from the new shard.
These must reach the resharding shard before it becomes visible to
user-facing selectors, and filtering them also made per-shard reads
return silently empty results on peers lagging on hashring commits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Explicitly set resharding filtering per match branch
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The WAL is only truncated past operations whose segment flush was
confirmed, so first_index is a durable lower bound on the applied
sequence. The persisted applied_seq can legitimately lag behind it by
more than one save interval: it is saved every 64 update-worker calls
from a counter that restarts at zero on process start, and synchronous
WAL replay never feeds it. A replay target computed from such a stale
applied_seq can then sit before first_index, tripping the debug_assert
from #8454 (flaky model_testing gate, #9844) and, in release builds,
enqueueing already-truncated indices that fail with spurious
"Operation not found in WAL" errors.
Clamp the replay target at first_index: nothing before it ever needs
replay.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Every point id the workload draws now comes from an IdSpace pool
precomputed at startup. A --uuid-id-fraction (default 0.5) of the
--id-pool slots are well-formed v4 UUIDs built from the seeded rng via
uuid::Builder::from_random_bytes, the rest stay numeric. Precomputing
the pool keeps the id-reuse semantics (upserts overwrite live points,
deletes and retrieves hit them) that fresh per-op random UUIDs would
lose, and keeps runs seed-reproducible.
Sampling consumes a single range draw per id, exactly like the previous
NumId draw, so fraction 0 consumes no extra rng draws and reproduces
the numeric-only op stream byte-for-byte. The harness smoke tests run
with fraction 0.5, and the fraction is recorded in the trace header.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Preparation for capping appendable segment growth in the update path
(#9158): a dedicated error for "all appendable segments reached
max_segment_size", so the update pipeline can recognize it and provision a
fresh appendable segment before re-applying the operation.
Maps to a transient service error at the collection level: if it ever
escapes recovery, failed-operation recovery re-applies the operation.
Part 1/5 of the appendable segment overflow fix.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Every vector candidate now carries a distance metric instead of the
hardcoded Dot, threaded into the fixture schema, the CreateVectorName
generator and the read-back prediction.
The model predicts Cosine read-backs exactly by mirroring the engine's
ingestion preprocessing: metric_preprocess follows
NamedVectors::preprocess_dense_vector's per-datatype dispatch and calls
Distance::preprocess_vector itself, so predictions track the engine by
construction (including the identity preprocess of the byte metric,
which stores Uint8 vectors un-normalized). Stored vectors are
preprocessed exactly once (optimizer and CoW moves transfer raw bytes),
so predictions stay exact across moves, including Cosine + Float16.
New candidates: "e" (dense Cosine), "n" (multi-dense Cosine, per-row
normalization), "x" (dense Cosine + Float16), "o" (dense Cosine +
Turbo4, padding-free dim), "j" (dense Euclid), "k" (dense Manhattan).
Euclid/Manhattan preprocess is an identity, so their value is engine
side: Order::SmallBetter comparator coverage.
The startup predictability check now also rejects sparse + non-Dot
(sparse schemas carry no distance) and Turbo4 + Euclid/Manhattan (TQ's
L1/L2 modes store lengths differently from Dot/Cosine and their
copy-on-write re-quantization fixed point is not soak-validated yet).
Soak-validated on seeds 1/2/4/5/6/7/8 (30k ops), including two
restart runs (restart probability 0.002) with the optimizer enabled.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add Float16 and Uint8 storage datatypes to the model tester
Extend VectorCandidate with a datatype override and fold the DenseTurbo
kind into Dense + Some(Turbo4) so storage datatype has a single source
of truth. Two new initially-active candidates exercise half-precision
("h", dense 6) and unsigned-byte ("y", dense 4) storage; "c" carries an
explicit Some(Float32) to cover schema configs that spell the default
datatype out.
The model predicts lossy read-backs through the engine's own
PrimitiveVectorElement impls (as Turbo4 reuses turbo_storage_roundtrip)
and compares them exactly: both round-trips are deterministic and
idempotent, so they stay bit-stable across optimizer moves, WAL replay,
and reloads. Uint8 components are drawn from 0.0..256.0 since the
storage truncates with `x as u8` and unit-range draws would collapse to
zeros.
A compile-time assertion rejects datatype overrides on non-Dense
candidates: the fixture's sparse/multi-dense arms ignore the field and
multi-dense read-backs are compared without a round-trip prediction, so
a lossy multi-dense candidate would soak-panic with a false divergence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Plumb Float16 and Uint8 multi-dense support in the model tester
Multi-dense storage converts the flattened matrix component-wise
(from_float_multivector), so per-row round-trips through the same
PrimitiveVectorElement impls predict read-backs exactly. The fixture's
multi-dense arm now applies the candidate datatype (matching the
CreateVectorName path), model_vector predicts per-row, and two new
initially-active candidates exercise the combination: "w"
(MultiDense(5), Float16) and "z" (MultiDense(3), Uint8).
The compile-time candidate check narrows to the combinations that
remain unpredicted: Turbo4 multi-dense (the multivector quantization
path differs from the per-vector turbo_storage_roundtrip) and sparse
with any datatype override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Make datatype match exhaustive in random_dense_vec
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review findings on datatype plumbing
- Start candidate "c" active so the explicit Float32 schema path runs in
default soaks (CreateVectorName is FORCE_OFF by default)
- Single Float16/Uint8 roundtrip dispatch shared by the dense and
multi-dense arms of model_vector
- Hoist shared fixture builder plumbing into dense_params_builder
- Build one DenseVectorConfig literal in the CreateVectorName generator
- Inline single-caller datatype_of wrapper
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Replace const-eval candidate check with a startup assert
Const eval forbids iterators, forcing an index-based while loop. A plain
function called at the top of run() reads better, still fails before any
op is applied, and names the offending candidate in the panic message.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fold INITIAL_ACTIVE into an initially_active candidate field
The hand-maintained list duplicated ALL_CANDIDATES (11 of 12 names) and
had to be kept in sync when adding candidates; forgetting it was silent
since CreateVectorName is FORCE_OFF by default, so a forgotten name got
zero default-soak coverage. Each candidate now declares its activation
inline and the fixture and run() filter on it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
An empty update channel only means the last operation was received,
not that it finished applying in spawn_blocking. Use plunge_async as
a barrier before asserting point counts.
Fixes#9831
Co-authored-by: Cursor <cursoragent@cursor.com>
Drain the update worker queue after setup upserts with WaitUntil::Wal.
Wal only waits for the WAL write, so on slow CI (notably Windows) B could
time out while queued behind the setup backlog rather than because the
worker was blocked on A's deferred wait.
Fixes#9814
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add per-query IDF corpus for sparse vector search
Let the caller choose, per query, which population sparse IDF statistics
are computed over. `params.idf` is either `"global"` (default, unchanged
behavior) or `{"corpus": <filter>}`, where the corpus filter is
independent of - and usually broader than - the retrieval filter.
Decoupling the two keeps the score scale stable when the retrieval
filter tightens: term importance is measured against a population the
user names, not against whatever subset the filter happens to select.
Design decisions:
- Corpus grammar is restricted to a conjunction (`must`) of `match`
conditions on payload fields; loosening later is backward compatible.
- Strict mode validates the corpus filter like a read filter
(unindexed fields rejected).
- `idf` on a vector without the IDF modifier is a validation error,
never silently ignored.
- An empty corpus yields degenerate but corpus-scoped scores (smoothed
IDF over N=0), never a fallback to global statistics - in multi-tenant
collections a fallback would leak term statistics across tenants.
Implementation:
- QueryContext IDF stats are keyed by corpus, so one batch can mix
requests with different corpora.
- Statistics come from the sparse index: df(term) is counted over the
query terms' posting lists only, never by scanning stored vectors.
Small corpora (under ~1/32 of the segment, by cardinality estimate)
are kept as a sorted id list galloping through posting lists via
skip_to; large ones as a dense membership mask filled streaming from
the filtered-points iterator. A misestimated small corpus degrades
into the mask.
- Exposed uniformly: REST (`params.idf`), gRPC (`IdfParams` message),
edge python bindings; OpenAPI schema regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply rustfmt
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix clippy manual_is_multiple_of in sparse IDF corpus test.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Allow any filter as IDF corpus
Drop the must+match grammar restriction on the corpus filter. A
restriction enforced only as a validation step over the full Filter
type buys nothing; if a narrower corpus syntax is ever wanted, it
should be a dedicated API-level type instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix build: add memory field to SparseIndexConfig in idf corpus test
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
drop_shard_key force-aborted any resharding on the key being dropped, but
swallowed a failed abort_resharding with a log-only error and dropped the
shards anyway. That could leave resharding_state.json referencing the
just-dropped key — a latent inconsistent load-time state.
Propagate the error with `?`. A ServiceError halts consensus and retries after
restart, which is safe because the abort and everything else in drop_shard_key
is replay-tolerant; a user error dismisses the entry before any shard is
dropped, which is equally consistent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A ServiceError returned from apply halts consensus on every peer (the entry can
never be applied), deterministically stalling the whole cluster. The transfer
Start validations that report a missing source or destination shard are
reachable — e.g. a committed Start racing a resharding-abort or shard-key-drop
that removed the shard — and all run before any durable write, so dismissing
the entry with a user error is safe and correct.
Convert five such sites to CollectionError::bad_request:
- validate_transfer: source shard missing, and destination shard missing in
both the resharding and filtered branches (helpers.rs);
- start_shard_transfer: the source and target get_shard lookups
(shard_transfer.rs).
The genuine "single node deployment" service_errors in collection_meta_ops.rs
are left untouched — those are real misconfiguration guards.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Start's first durable write registers the transfer record; its last write sets
the destination replica state. On re-apply after a crash between the two,
check_transfer_conflicts found the operation's own half-applied transfer and
returned bad_request, which is dismissed and never retried — so the peer
permanently lacked the destination replica entry. Worse, a later Finish on
that peer then silently skips both the destination promotion and the source
removal, pinning a replica-set divergence.
Exclude the transfer's own key from the conflict scan so a replay falls through
and re-runs the (idempotent) start: register_start_shard_transfer is a
set-insert and the destination replica-state write is absolute, so re-running
reconciles the partial state instead of dismissing it. A genuinely conflicting
transfer (different key touching the same shard/peers) is still rejected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
start_resharding, finish_resharding and abort_resharding each saved the
updated collection config with a log-only `if let Err(err) = config.save(..)`
that swallowed the failure. A swallowed save let the operation report success
with a stale shard_number persisted on disk, arming a shard-dir/loader panic
(or a silently unloaded shard) at the next restart.
Propagate the error with `config.save(&self.path)?;` instead. The resulting IO
error is a ServiceError, so consensus halts and retries the entry after
restart. That is safe because in all three functions the config save is
value-idempotent (guarded by `shard_number != new_shard_number`) and every
step is replay-tolerant, so the retry converges.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In the up-direction of abort_resharding, ShardHolder::abort_resharding drops
the new shard's directory, but the config.params.shard_number update was
persisted only at the very end of the function — after the transfers abort.
A crash anywhere in that wide window left shard_number pointing at an
already-deleted shard directory, which makes the auto-sharding loader panic
on the missing dir at startup (crash loop), before the consensus replay that
would reconcile the state can run.
Move the shard-count update block to before the shard_holder.abort_resharding
call. The block keeps its value-idempotence guard, so replay converges. The
config write lock is taken while holding the shard_holder write guard, matching
the shard_holder -> config ordering already used in finish_resharding, and is
released before abort_resharding. The reverse crash window (dir still present
but count already decremented) is benign and reconciled by replay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In the down-direction of finish_resharding, the config.params.shard_number
update was persisted *after* drop_and_remove_shard. A crash between the two
left shard_number pointing at an already-deleted shard directory, which makes
the auto-sharding loader panic on the missing dir at startup (crash loop) —
before the consensus replay that would reconcile the state can run.
Move the shard-count update block to before drop_and_remove_shard (still after
remove_shard_from_key_mapping). The block keeps its value-idempotence guard, so
replay converges. The reverse crash window (dir still present but count already
decremented) is benign and reconciled by replay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When Medium/Strong write-ordering forwards an update to the leader and it
fails with a transient error, the failure path proposed deactivating the
leader replica while passing its raw peer state as from_state. Unlike the
sibling deactivation site, this did not filter out transient/resharding
states, so a Resharding/ReshardingScaleDown leader could be proposed
Dead with from_state=Some(Resharding*).
On re-apply after a crash inside abort-resharding (which reverts the gated
replica to Active or removes it), the from_state gate no longer matches the
current state -> bad_input -> the entry is dismissed and never retried, so
the peer keeps Active while the rest of the cluster has Dead.
Filter the leader state with `.filter(|state| !state.is_partial_or_recovery())`,
matching the sibling site (PR #7849), so the proposal omits from_state for
transient/resharding states and converges on replay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(recovery): advance newest clocks for WAL-replay tail on load
`load_from_wal` replays the WAL synchronously up to `to` and queues the
remaining `[to, last_wal_index)` tail to the update worker. Whether that
tail exists depends on how far the persisted `applied_seq` lags the WAL
end, independently of the `prevent_unoptimized` flag's value: the flag only
gates the worker's deferred-points wait, not whether the tail is queued.
The synchronous replay advances `newest_clocks` from each entry's clock tag,
but the queued tail did not: the update worker deserializes only the
operation and discards the clock tag.
As a result the newest-clocks recovery point regressed across a graceful
restart by up to the update-queue size, even though those operations are
durably in the WAL (which is exactly what the recovery point tracks). Worse,
the first post-restart updates would then be assigned clock ticks that
earlier WAL entries already carry for different operations, which corrupts
WAL-delta resolution in a cluster.
Advance `newest_clocks` over the queued tail during load, mirroring the
synchronous replay, before handing it to the worker. The range is
end-exclusive because `last_wal_index` is one past the last entry.
Regression dates to WAL replay honoring `applied_seq` (#8008), which
narrowed synchronous replay from the whole WAL to `[from, to)`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(model): assert clock recovery point survives close+reopen
Capture each shard's newest-clocks recovery point (flattened to
shard_id -> (peer_id, clock_id) -> tick, tokens dropped) on both sides of
every close+reopen in the model testing harness (mid-run restart and final
reload) and assert exact equality.
Both mismatch directions are bugs: a lost tick means clock durability broke,
a gained tick means the reload path over-advanced a clock. The check runs
after the existing model check so a lost WAL tail keeps its established
extra/missing-id postmortem signature, and a clocks-only divergence surfaces
distinctly.
This is what caught the WAL-replay-tail clock regression fixed in the
previous commit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(recovery): tolerate unreadable WAL tail entries in clock advance
Review follow-ups for the deferred-tail clock advance:
- Log and skip a tail entry that fails to read instead of propagating
the error: the update worker tolerates the same failure when it
re-reads the entry, and failing here turns one bad tail record (or
applied_seq/truncation index skew) into a shard, and by default a
node, that cannot start. Add a red-green-verified regression test
that injects an undeserializable record into the deferred tail.
- Fix the pre-existing off-by-one in the send loop: last_wal_index is
one past the last entry, so `to..=last_wal_index` enqueued a phantom
op_num on every restart with a deferred tail.
- Reword new comments (em dashes, and the range-bound note is obsolete
now that both loops use the same exclusive bound; document the
two-pass shape instead: the WAL iterator is not Send across await).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: return error for missing point IDs in Query API recommend
The Query API recommend handler (RecommendAverageVector, RecommendBestScore, RecommendSumScores) used filter_map in resolve_reco_reference, which silently dropped point IDs that could not be resolved. The legacy /points/recommend API correctly returns a PointNotFound error in this case.
Fix: change resolve_reco_reference to return CollectionResult and propagate errors instead of filtering them out, matching the pattern used by Nearest, Discover, Context, and Feedback query types.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
* fix: apply cargo fmt to collection_query.rs resolve_reco_reference
Format resolve_reference() calls as method chains to satisfy rustfmt.
Addresses reviewer feedback from timvisee: CI lint failure.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
* test: add unit tests for resolve_reco_reference error on missing point IDs (Fixes#9390)
- test_missing_point_id: verifies error when ID not in ReferencedVectors
- test_valid_point_id: verifies success when ID is present
- test_mixed_point_ids: verifies error when mix of valid/invalid IDs
Requested by @timvisee in PR review.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
* fix: apply cargo fmt to resolve_reco_reference tests
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
* fix: apply cargo fmt to resolve_reco_reference test imports
Move `use super::*` after external crate imports to satisfy rustfmt.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: read-only edge grouping/matrix search + object-storage read path
Add query_groups (group by a payload field) and search_matrix (single-shard distance matrix over a random sample) to the read-only edge shard's EdgeShardRead API, in new grouping and matrix modules plus edge test helpers.
Make the object-storage read path available outside tests: drop the #[cfg(test)] gate on the BlobFile UniversalReadExt impl and move io_bridge_object_store/object_store to segment's normal dependencies, so a ReadOnlyEdgeShard can serve segments read from S3.
* Share group-by building blocks between server and edge
Move GroupsAggregator, group candidate query shaping (is-empty filter,
group_by payload selector, prefetch limit scaling) and result-order
derivation into shard::grouping / shard::query, so the collection and
edge grouping implementations cannot silently diverge.
Edge grouping now handles multi-valued group keys, u64 keys, wildcard
group_by paths, prefetch limits and score-ordered groups the same way
as the server.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drive group-by through a shared sans-IO state machine
Extract the multi-request collect/fill loop into
shard::grouping::GroupByDriver: next_request() yields shaped backend
queries, add_points() advances the state, distill() returns the groups.
Query execution stays with the caller, so the async server path and the
sync edge path drive the same machine, and the request shaping helpers
become private to shard::grouping.
Edge now uses the same request budget (5 collect + 5 fill requests) and
per-request candidates limit (groups * group_size, computed inside the
driver) as the server, replacing its single 4x-oversampled request.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add coverage for TurboQuant datatype in model tester
Co-authored-by: Cursor <cursoragent@cursor.com>
* Address review feedback: tolerance, doc comment, exhaustive match
- Tighten dense_matches Turbo4 tolerance to 16 ulps relative and drop
the absolute floor, so near-zero sign flips and small systematic
quantization drift fail instead of passing
- Fix ALL_CANDIDATES doc comment to match INITIAL_ACTIVE (six names
start active, "c" and "u" via CreateVectorName)
- Make model_vector match exhaustive so new VectorKind variants force
a compile error
- Use explicit DistanceType::from(distance) in turbo_storage_roundtrip
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The write_limit_type: bool parameter of CollectionError::rate_limit_error
relied on a trailing comment to document its meaning (false = read,
true = write). A swapped literal at a call site would compile and report
the wrong limiter type in the error message.
Introduce RateLimiterKind { Read, Write } so the intent is explicit at
call sites and checked at compile time.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Remove 8 `#[allow(clippy::...)]` attributes that no longer suppress any
lint. Each was verified redundant by rewriting it to `#[expect(...)]` and
confirming the workspace stays clippy-clean under the CI config
(`cargo clippy --workspace --all-targets --all-features -- -D warnings`).
Attribute-only deletions, no behavior change.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Read-only edge-shard followers need to distinguish "essential segment file
missing because the leader removed the segment mid-reload" (re-check the
manifest, absorb) from real corruption (escalate). The classification
already exists at the universal-io layer (UniversalIoError::NotFound,
mmap MissingFile, both carrying the path), but died at the OperationError
boundary where everything collapsed into ServiceError strings.
- Add OperationError::FileNotFound { path } and route the structured
sources into it: UniversalIoError::NotFound, MmapError::MissingFile,
and GridstoreError::UniversalIo(NotFound) (the route payload-storage
live-reload errors take).
- Implement IsNotFound for OperationError so follower code can classify
(and OperationResult::ok_not_found() works where lazily-created files
are legitimately absent).
- Raw io::Error NotFound intentionally stays ServiceError: it has no
structured path; universal-io wraps not-found at the call site via
extract_not_found, so classification belongs at the source.
- CollectionError maps FileNotFound like a service error, keeping
external API behavior unchanged.
Groundwork for not-found handling in ReadOnlyEdgeShard live-reload.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [AI] make ReadOnlyRoaringFlags bitmap and bool index counts lazy
Opening a read-only segment scanned every flags file end to end:
`ReadOnlyRoaringFlags::open` materialized the whole RoaringBitmap via
`iter_ones()`. Every payload field carries a null index, so this was paid
per field per segment, for bitmaps most queries never touch.
Make the bitmap a `OnceLock`, filled by a scan on first access. Open now
reads only the tiny status file. `ReadOnlyBoolIndex`'s three eager count
fields collapse into one lazily-derived, cached `BoolCounts`; its
`live_reload` refreshes them in place when present and leaves them unset
otherwise, so reloading an index nothing queries stays scan-free.
Propagate the resulting `OperationResult` through `RoaringFlagsRead`,
`PayloadFieldIndexRead::count_indexed_points`, `FieldIndexRead`,
`PayloadIndexRead::{indexed_points, get_telemetry_data}`, `build_info` /
`build_telemetry` and `SegmentEntry::{info, get_telemetry_data}`, out
into shard, edge and collection.
`ram_usage_bytes` stays infallible: an unmaterialized bitmap holds no
RAM, so it reports 0 via the new `bitmap_if_materialized`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [AI] correct `preopen` comment: `open` no longer scans the flags file
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [AI] fix edge examples for fallible `info()`
`EdgeShardRead::info` now returns `OperationResult<ShardInfo>`. The
examples live in their own workspace (lib/edge/publish), so the main
`cargo check --workspace` never saw them.
Every call site sits in `fn main() -> Result<(), Box<dyn Error>>`, so
propagate with `?`. `bm25-search` compiled either way but would have
printed the `Result` rather than the `ShardInfo`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: replace flush_all sync bool with FlushMode enum
SegmentHolder::flush_all took two adjacent bools (sync, force), and call
sites read as bare literal pairs like flush_all(true, false). Swapping
the arguments compiles and silently changes flush semantics: a swapped
pair at the snapshot site would make snapshots skip flushing entirely
when a background flush is running.
Introduce FlushMode { Sync, Background } for the first parameter so the
pair is no longer transposable and the behavior is named at each call
site. The force flag stays a bool since it feeds the
SegmentEntry::flusher(force) trait in lib/segment. No behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: exhaustive match on FlushMode instead of equality check
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Bound request snapshots in the slow requests log
Entries in the slow requests log retain a full serde_json::Value snapshot
of the (internal) shard request indefinitely. The distance matrix API
internally generates a batch of `sample` queries, each carrying a has_id
filter with all `sample` sampled ids, so a single log entry ballooned to
sample^2 ids expanded into a JSON tree (~90 bytes/id): ~90MB per entry for
sample=1000, ~2GB for sample=5000. Random sample ids give every request a
fresh content hash, so each call added a new entry until the 32-slot
queue filled — OOM long before that for larger samples.
Truncate all arrays in logged request bodies to 64 elements plus an
omission marker. Query batches are serialized per element up to the cap,
so the full untruncated JSON tree is never materialized even transiently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix rustfmt
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Remove from_iter_instead_of_collect from workspace lints
The lint was removed from clippy (beta) and now triggers
renamed_and_removed_lints warnings in every crate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::chunks_exact_to_as_chunks
Replace chunks_exact with a constant chunk size by as_chunks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::needless_late_init
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::useless_borrows_in_formatting
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::uninlined_format_args
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::for_kv_map
Iterate map values directly instead of discarding keys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_large_err on QueueProxyShard::new_from_version
The Err variant intentionally hands the LocalShard back to the caller.
Same pattern as the existing allow on ForwardProxyShard::new.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_unit_err on wait_for_consensus_commit
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
