Lets callbacks fail with the caller's own error type (e.g. OperationError)
instead of smuggling it out of an infallible callback: lookup_batch now
takes a fallible callback and resolve_internal_batch drops its first_err
pattern. E is not inferable from Ok(())-only callbacks, so existing
infallible call sites annotate it in the turbofish.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add batch id-tracker lookups, pipelined in the disk-resident trackers
Introduce batch counterparts for the per-point IdTracker read operations
(external->internal resolution, internal->external lookup, version lookup,
deleted checks) and implement them with pipelined reads in the disk-resident
trackers, where the per-point path costs one storage round-trip per lookup:
- StoredBitSlice::get_bits_batch: dedupe the containing u64 elements and
fetch them through one read_batch pass.
- DiskMappingReader::lookup_batch: group e2i keys by run block, read each
unique block once, binary-search all its keys.
- DiskMappingReader::external_ids_batch: schedule all i2e data slots plus
the deduplicated is_uuid bitmap bytes in one pass.
- DiskMappingsSource::{points_deleted_batch,resolve_internal_batch,
resolve_external_batch}: shared batched resolution for both disk trackers.
- IdTrackerRead::{external_ids_batch,internal_versions_batch}: trait-level
batch methods with loop defaults for in-RAM trackers; overridden (along
with resolve_external_ids) in DiskIdTracker and ReadOnlyDiskIdTracker,
and forwarded through both tracker enums so the overrides dispatch.
Wire the batch operations into the read paths: search result processing
(external ids + versions), scroll filtered_read_by_index (chunked), and
HasId condition conversion/cardinality estimation. retrieve() already goes
through resolve_external_ids and picks up the batched dispatch.
Batch read-by-id keeps the read-only tracker's laziness: no full
deleted-set materialization (covered by test).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Rework batch id-tracker interfaces to iterator inputs and callback output
Review follow-up: the slice-in/Vecs-out shapes forced intermediate collects
at every call site, costing the in-RAM trackers (whose defaults are plain
loops) allocations the pre-batch code never had.
- resolve_external_ids now takes `impl IntoIterator<Item = PointIdType>`
and delivers `(id, offset)` pairs through a callback, in input order:
has_id conditions stream the id set straight into the offsets set/vec,
retrieve fills its parallel vectors directly.
- external_ids_batch / internal_versions_batch take iterator inputs but
keep `Vec<Option<_>>` outputs (search result processing needs one aligned
slot per input); search drops its two offset collects, scroll passes the
itertools chunk directly.
- In-RAM trait defaults stream with no intermediate allocation; the disk
overrides buffer the input once internally, where the block-grouped
reads need the whole batch anyway.
Also document on the writable DiskIdTracker why points_deleted_batch and
internal_versions_batch are deliberately not overridden there: deleted and
versions are RAM-resident by design, so the default loop has no IO to
pipeline; only the read-only tracker keeps them on disk.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove accidentally committed local example
inspect_mutable_id_tracker.rs is a local debugging harness that was swept
into the previous commit by a directory-level git add; it is not part of
the batch-interface change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add PointIdBatch and thread it through resolve_external_ids
Replace the `impl IntoIterator` / `&[PointIdType]` inputs on the
id-tracker external->internal resolve path with a small `PointIdBatch`
trait (Copy + num_ids + iter_ids), implemented for `&[PointIdType]` and
`&AHashSet<PointIdType>`. Being re-iterable and length-known lets the
disk trackers batch the reads without first collecting the input.
On the disk path this drops several intermediate allocations:
- no input `Vec<PointIdType>` collect in resolve_external_ids_batch;
- lookup_batch returns compact `(id, offset)` pairs (id rebuilt from
is_uuid + key) instead of a size-N `Option` reorder buffer;
- lookup_batch no longer groups keys by block up front: a block is one
~16 KiB DiskCache block, so same-block reads are deduplicated by the
cache (piggybacked in flight, a plain hit once resident);
- resolve_internal_batch filters deleted pairs in place and feeds
points_deleted_batch an offset iterator, dropping the offsets Vec.
Delivery is no longer input-ordered; the id travels with the offset into
the callback, so callers that need positions still have them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Stream disk id resolve through a callback; prefetch deleted flags whole
Follow-up to the PointIdBatch rework, removing the last buffers from the
disk external->internal resolve path and dropping the redundant deleted
batching.
- `lookup_batch` and `resolve_internal_batch` now hand each resolved
`(id, offset)` to a callback as its block completes instead of
returning a `Vec`; `resolve_external_ids_batch` just forwards the
caller's callback and logs on error. No buffer is built on the path.
- Drop `points_deleted_batch`: the deleted set is resident (writable
tracker) or prefetched whole (read-only), so a per-point `point_deleted`
check is as cheap as a pipelined read would be. `resolve_internal_batch`
captures the first check error out of band and surfaces it after the
pass.
- Prefetch the deleted flags whole (`Populate::PreferBackground`) in both
`try_preopen` and `try_open`, via a shared `deleted_open_options`, so
the per-point checks stay off remote storage.
- `PointIdBatch` loses the now-unused `num_ids`; only `iter_ids` remains.
Delivery is fully streamed, so on a mid-pass storage error the callbacks
for already-resolved live points have fired before the error surfaces —
acceptable at the best-effort `IdTrackerRead` boundary.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Surface id resolve errors to the trait; drop redundant batch wrapper
`IdTrackerRead::resolve_external_ids` now returns `OperationResult<()>`
instead of swallowing storage errors. The disk path previously logged
and dropped a `resolve_internal_batch` error at this boundary; now it
propagates so callers see failed id resolution.
- The in-RAM default returns `Ok(())`; both enum forwarders and both disk
overrides propagate. The three call sites (retrieve, HasId cardinality,
HasId checker) are already in `OperationResult` functions, so they just
gain a `?`.
- Drop the `resolve_external_ids_batch` free function: with its
log-and-swallow gone it only forwarded to `resolve_internal_batch`, so
the two overrides call that directly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Stream internal_versions_batch through a callback
Give internal_versions_batch the same treatment as resolve_external_ids:
it takes a callback and returns OperationResult<()> instead of a
Vec<Option<..>>, so the disk override no longer allocates. It walks the
input lazily (no collect), tags each pipelined read with its internal_id,
and streams (internal_id, version) to the callback as reads complete;
out-of-range offsets are skipped and a storage error propagates instead
of being logged and swallowed.
The in-RAM default and both enum forwarders match. process_search_result
now builds a small internal_id -> version map from the callback and looks
up by scored offset (handling duplicate offsets naturally), keeping the
same missing-version-is-an-error behaviour.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Stream external_ids_batch through a callback; drop chunked scroll batching
Align external_ids_batch with internal_versions_batch: iterator input,
(offset, id) callback output, no internal buffering, storage errors
propagated. The deleted filter runs inside the lazy range iterator, so
each tracker implements the override directly against its own deleted
source (resident bitvec vs prefetched on-disk file) and the redundant
DiskMappingsSource::resolve_external_batch pass-through is removed,
along with the now-unused log_lookup_err_batch.
filtered_read_by_index feeds the whole candidate iterator into one
batch pass and lets the IO layer pipeline the reads; the limit case
keeps the top-smallest ids in a bounded priority queue. This retires
ID_TRACKER_BATCH_SIZE. Search result processing pairs external ids by
internal id, mirroring the versions map.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove accidentally committed local example, again
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop unused StoredBitSlice::get_bits_batch
Its only caller, the read-only tracker's points_deleted_batch override,
was removed when the deleted flags became whole-file prefetched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Write deleted_open_options fields explicitly, comment the why
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Simplify read-only external_ids_batch error handling via try_filter
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add infallible point_deleted helper to the writable disk tracker
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop PointIdBatch in favor of impl IntoIterator<Item = PointIdType>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* test(model_testing): add slice matcher to generated scroll filter
Extend ScrollFilter with a Slice variant so paginated scroll exercises
Condition::Slice. The generator draws small totals (1/2/3/4/5/8) and a
valid index; the model verifier mirrors membership via Slice::check —
the same hash contract the engine uses — so the existing paged-scroll
id-set assertion covers sliced scroll under soak (optimizer, WAL reload,
multi-shard, mixed UUID/numeric ids).
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(model_testing): add CountBySlice verification op
Exercise Condition::Slice through the exact count API under soak.
Shares the slice generator with ScrollPaged; the model oracle uses
Slice::check so engine and in-memory counts must agree.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(model_testing): compose slice with num on scroll and delete-by-filter
- ScrollFilter::NumAndSlice: indexed num drives candidates; slice is a
per-candidate check via Filter::merge.
- DeleteByFilter { num, slice: Option<Slice> }: half the deletes also
restrict by slice so submit-time filter resolution and WAL-replayed
id lists exercise Condition::Slice.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Setting metadata to an empty object does not clear it: the update merges
key by key, so an empty object is a no-op (and over gRPC an empty map is
indistinguishable from an absent one). Per-key removal via null values is
the mechanism that was actually implemented and tested in #7123.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat: slice filtering condition for sliced scroll and deterministic sampling
Add a `slice` filter condition selecting points where
`stable_hash(point_id) % total == index`. The hash is SipHash-2-4 with a
zero key over canonical id bytes (8 LE bytes for numeric ids, 16 RFC 4122
bytes for UUIDs) — a frozen public contract, independent of the internal
resharding ring hash, reproducible by clients to predict membership.
For a fixed `total`, slices are disjoint and cover all points, enabling
parallel scroll streams (ES sliced-scroll style) and reproducible sampling
that composes with any other filter condition.
- REST: `{"slice": {"total": N, "index": R}}`; gRPC: `SliceCondition` in
the condition oneof (tag 8)
- Evaluated per point via id_tracker external-id lookup; no payload index
needed; cardinality estimated as `points / total` with no primary clause
- `total >= 1` enforced by NonZeroU32 at parse time, `index < total` by
validation in both REST and gRPC paths
- Hash contract locked by test vectors independently reproduced with a
reference SipHash-2-4 implementation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* tests: minimal OpenAPI test for slice filter condition
Scrolls all slices of a fixed total over numeric + UUID ids asserting
disjointness and full coverage, checks must_not inversion, and pins the
two rejection paths (422 for index >= total, 400 for total = 0). Requests
and responses are validated against the regenerated OpenAPI spec by the
test harness.
Note: the spec cannot itself reject total = 0 client-side — the Condition
anyOf falls through to the permissive Filter schema, as with any invalid
condition — so rejection is asserted via the server response.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Replace the mixed read interface (internal types, explicit parameter
enumeration, ad-hoc custom types) with edge-owned request structs, one
file per request in src/requests/. Each has a new() constructor taking
only the required parameters, a no-macro fluent builder in src/builders/,
and a From conversion into the internal request type in
requests/conversions/, grouped by request type. Conversions construct
and destructure with full field lists, so a parameter added on either
side fails compilation instead of being silently dropped.
The old reexport aliases (ScrollRequest = ScrollRequestInternal, etc.)
are replaced by the edge types under the same names; retrieve() takes a
RetrieveRequest instead of a parameter triple. Python bindings wrap the
edge types, and the published Rust examples use the builders.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Non-generic TransformInto::transform
I think dyn is fine here because it is executed once per query, not in a
hot loop.
* Query::score_by impls: manual loops
Iterator chains (`.iter().map(..).sum()`) produce a lot work for the
compiler to do.
* QuantizedCustomQueryScorer: move generics from struct to `new` method
* refactor: group edge crate internals into edge_shard and read_view modules
Restructure lib/edge/src so the top level only contains the modules that
form the public crate surface. Implementation files move under the type
they implement:
- edge_shard/: the EdgeShard struct with its load/config-resolution
helpers (previously inlined in lib.rs), plus optimize, shard_read,
snapshots, and update
- read_view/: the EdgeShardRead/ReadSegmentHandle traits and
EdgeReadView (previously read_view.rs), plus the per-operation impl
files count, facet, grouping, info, matrix, query, retrieve, scroll,
and search; build_search_pool (previously pool.rs) is folded into
read_view/mod.rs next to par_map_segments, the seam it powers
lib.rs is now a thin facade of module declarations and re-exports. The
public API is unchanged: all previously exported names resolve exactly
as before, verified against the edge tests, the python bindings,
edge-shard-query, and the regenerated publish amalgamation with its
examples.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: trim EdgeShardRead surface and split read_view/read_only modules
Trait surface:
- Drop query_scroll and rescore_with_formula from EdgeShardRead and the
EdgeShard inherent wrappers; only the internal query pipeline used
them, via the pub(crate) EdgeReadView methods that remain.
- Hide the snapshot plumbing (read_segments, search_pool, plus
config_snapshot/path providers) in a crate-private ReadViewProvider
trait. EdgeShardRead now declares only user-facing methods and is
implemented for every provider through a blanket impl, so the
plumbing is not callable from user code (verified with a negative
compile test; a private supertrait alone leaves supertrait methods
callable through generic bounds). An empty sealed marker supertrait
keeps the trait unimplementable downstream.
- config_snapshot and path stay public: used by edge-shard-query and
the python bindings.
Module layout:
- read_view/: mod.rs keeps EdgeReadView and build_search_pool;
ReadSegmentHandle moves to handle.rs, the trait machinery to
shard_read.rs, and the nine per-operation impl files into ops/.
- read_only/: the follower's ReadViewProvider impl moves out of mod.rs
into shard_read.rs, mirroring edge_shard/shard_read.rs.
Verified: edge tests, python bindings, edge-shard-query, regenerated
publish amalgamation with all examples compiling and running.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat: segment manifest optimizing state with lease
* fix: clippy
* fix: exhaustive manifest state matching
* fix: merge manifest rebuilds under the write lock
* refactor: named state predicates, drop redundant enumerator test
Review follow-up: move the enumerator's filter into
SegmentManifestState::is_usable, name the preserving predicate
is_optimizer_mark, delete the enumerator test that re-tested serde.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The persisted_hashmap and disk_cache tests generate their datasets with
StdRng (ChaCha12 in rand 0.10). String keys draw one random_range call
per character, so the crypto generator dominated the test runtime:
test_k_str_* drop from ~3.3s to ~1.4s each with SmallRng (Xoshiro256++).
Assertions are self-consistency checks (write then read back), so the
changed sequences carry no retuning risk.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
copy_to_path returned Ok(()) without flushing, so the copy was not guaranteed on disk. Flush after the copy, matching create() and flush() in the same file. Also add a round-trip test for copy_to_path, which had none.
* Benches: use SmallRng instead of ChaCha12-based generators
All benchmarks used StdRng or rand::rng() (ThreadRng), both backed by the
ChaCha12 block cipher in rand 0.10. Benchmarks do not need crypto-strength
randomness, and several draw random values inside the timed closure, so
cipher work was included in the measurement itself.
Switch every bench target to SmallRng (Xoshiro256++), and key the HNSW
graph cache and sparse index cache by RNG algorithm so stale caches built
from the old generator are not reused against newly generated vectors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Benches: replace free-function rand::random with local SmallRng
Addresses review: rand::random draws from the thread RNG (ChaCha12),
including inside the timed loop of the pq score benchmark.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Model tester: cover all quantization types
Add a `quantization` field to `VectorCandidate` so candidates can carry
any `QuantizationConfig` variant, and materialize the configs in the
fixture (`quantization_config`). The inline-storage vector "i" keeps its
scalar Int8 config, now declared on the candidate instead of hard-coded
in the fixture.
New candidates:
- "p" Dense(8) + Product x4
- "v" Dense(6) + Binary (non-byte-aligned dim, trailing-bit padding)
- "r" Dense(8) + Turbo (search-side TQ over Float32 storage)
Quantization x datatype combos:
- "l" Dense(6) Float16 + Binary
- "d" Dense(8) Turbo4 + Turbo default bits (keep-source-rotated branch
of `should_keep_source_rotated`)
- "g" Dense(8) Turbo4 + Turbo Bits1_5 (Padded rotation, rotate-back
branch)
This is model-safe: schema quantization keeps the original vectors, so
read-back predictions are untouched; the approximate quantized scoring
only feeds the membership-only Search/Query/Recommend checks.
`assert_candidates_predictable` enforces the wiring constraints:
quantization is dense-only (the fixture only wires the dense arm) and
requires `initially_active` (CreateVectorName's `DenseVectorConfig`
carries no quantization).
Verified: seeds 1/2/3/7/42 soaks (5k ops, restarts, optimizer on) green;
quantized codes confirmed on disk for every quantized candidate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Model tester: make inline_storage a VectorCandidate knob, enable on "l" and "d"
Replaces the fixture's name-based INLINE_STORAGE_VECTOR special case with an
inline_storage field on VectorCandidate (requires quantization, enforced by the
startup assert). Enables it on "l" (Float16 base + padded Binary links) and "d"
(Turbo4 base + TQ links) to cover more (base layout, link encoding) pairs of the
CompressedWithVectors format; "v", "r", "g" and "p" keep the non-inline paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Fix resharding, on queries filter shards on all shard selectors
* Add failing consensus test: search during resharding with shard keys (#9880)
Reproduces a known bug: after resharding is initialized on a custom
sharded collection with a shard key, searches (with and without the
shard key selector) fail with "does not have enough active replicas",
because the new resharding shard is included in reads before it has
an active replica.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Exempt explicit shard id selection from resharding read filter
Explicit shard id selection is only used by internal per-shard
operations (local shard API, internal gRPC reads), including the
resharding driver reading back migrated points from the new shard.
These must reach the resharding shard before it becomes visible to
user-facing selectors, and filtering them also made per-shard reads
return silently empty results on peers lagging on hashring commits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Explicitly set resharding filtering per match branch
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The HashSet bought nothing: names are unique by construction (map keys)
and all callers only iterate. A sorted Vec skips the hashing and set
allocation, and makes the iteration order deterministic.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* DiskIdTracker: RAM-resident is_uuid stored-bitmask sidecar + module split
Move the is_uuid flags out of the i2e file into a separate
id_tracker.is_uuid file in the compact StoredBitmask format (#9871),
loaded whole into RAM as a RoaringBitmap on open and prefetched in
preopen, so slot decoding never reads the flag from disk. Bump the
on-disk format version to 2 (DiskIdTracker is unreleased; no migration).
Add StoredBitmask::read_ones() in common to normalize any stored
encoding into a bitmap of set positions, with logical_len validated
against the u32 position space at open.
Restructure disk_id_tracker: read_only.rs becomes read_only/{mod,
lifecycle,live_reload,id_tracker_read} mirroring the immutable tracker,
and reader.rs becomes reader/{mod,lifecycle,lookup,iter}.
Also unbox the DiskMappingsRef iterators (impl Iterator instead of
Box<dyn>), and make IdTrackerRead::iter_internal_versions fallible so
the read-only disk tracker propagates storage errors instead of
silently truncating; its implementation now reads the versions file in
one pass (cleanup-on-open drains it anyway).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Split disk_id_tracker mod.rs into lifecycle / read / write files
Mirror the read_only/ layout: mod.rs keeps the struct and resident-RAM
helpers, lifecycle.rs the build/open paths, id_tracker_read.rs the
DiskMappingsSource + IdTrackerRead impls, id_tracker.rs the mutable
IdTracker impl. Code moved verbatim; only imports and a field doc touched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Tighten disk_id_tracker docstrings around guarantees
State contracts (residency, laziness, error semantics, atomicity)
instead of narrating which structures hold or use what.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Deduplicate plain and raw upsert drivers via PointToUpsert trait
upsert_points and upsert_points_raw were ~60-line clones differing only
in how the point struct writes itself into a segment. Extract a private
PointToUpsert trait with the two variation points — upsert_into (in-place
write) and write_moved (CoW-move record transform) — implemented for
PointStructPersisted and PointStructRawPersisted, and fold the chunked
driver into a single generic upsert_points_impl.
The public functions keep their names and signatures as thin wrappers.
The duplicated upsert_with_payload/upsert_raw_with_payload tails collapse
into one shared set_full_or_clear_payload helper, which also carries the
single has_point debug assertion.
No behavior changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Deduplicate plain and raw sync drivers via PointToSync trait
sync_points and sync_points_raw were ~80-line clones of the same 5-step
algorithm, differing only in the retrieval call (retrieve vs retrieve_raw)
and the stored-record type compared against. Extend the upsert approach
with a PointToSync subtrait carrying an associated StoredRecord type,
retrieve_stored, and is_equal_to (delegating to the existing inherent
methods), and fold the drivers into a single generic sync_points_impl.
Step 5 calls upsert_points_impl directly; PointToUpsert and
upsert_points_impl become pub(super) to be visible within points/.
Public signatures unchanged.
No behavior changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
lib/shard/src/update.rs grew to 2200+ lines. Split it into an update/
directory by operation kind, moving code verbatim:
- mod.rs: process_* dispatch entry points + re-exports (public API and
crate-internal paths are unchanged)
- points/: upsert.rs (plain, conditional and raw), delete.rs (by id and
by filter), sync.rs (plain and raw)
- vectors.rs, payload.rs, field_index.rs: per-kind apply functions
- helpers.rs: shared filter-based point selection (incl. the deferred
points corner case) and check_unprocessed_points
- tests.rs: the test module, unchanged
Only additions are per-file imports, re-export lists and two
pub(super) visibility bumps for now-cross-module helpers.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The WAL is only truncated past operations whose segment flush was
confirmed, so first_index is a durable lower bound on the applied
sequence. The persisted applied_seq can legitimately lag behind it by
more than one save interval: it is saved every 64 update-worker calls
from a counter that restarts at zero on process start, and synchronous
WAL replay never feeds it. A replay target computed from such a stale
applied_seq can then sit before first_index, tripping the debug_assert
from #8454 (flaky model_testing gate, #9844) and, in release builds,
enqueueing already-truncated indices that fail with spurious
"Operation not found in WAL" errors.
Clamp the replay target at first_index: nothing before it ever needs
replay.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Compact stored bitmask for on-disk field index deleted masks
Add StoredBitmask: a compact persisted bitmask written and read as a
whole. The payload is a roaring bitmap of whichever bit value is the
minority (mostly-0 and mostly-1 masks both stay tiny), falling back to
raw dense bits when roaring would not be smaller, so the file is never
larger than the dense representation. Files are replaced atomically via
UniversalWriteFileOps::atomic_save; there is no in-place mutation.
Use it for the write-once "no values" masks of the on-disk numeric,
geo, map and full-text indexes, replacing the raw dense bitslice files
sized at point_count/8 bytes regardless of content.
Writing the new format is gated by the compact_bitmask feature flag
(default off; enabled by `all` and `serverless_compatible`). Reading is
format-agnostic regardless of the flag: the compact deleted_mask.bin is
tried first, then the index-specific legacy file, so old segments stay
readable and flag-off builds produce byte-identical legacy files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Split save_bitmask into named helpers
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Regenerate OpenAPI spec for compact_bitmask feature flag
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review findings on stored bitmask
- Avoid u64 overflow in the payload bound check when opening a mask
with a corrupted payload_len.
- Reject roaring payload positions beyond logical_len at read time,
enforcing the BitmaskContent range contract for corrupted files.
- Remove the opposite-format mask file after a successful save, so a
rebuild with a flipped compact_bitmask flag can't leave a stale
compact file shadowing the fresh legacy one (or an orphaned legacy
file next to a compact one).
- Make the compact-open numeric test tolerate builds that already
wrote the compact format.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Every point id the workload draws now comes from an IdSpace pool
precomputed at startup. A --uuid-id-fraction (default 0.5) of the
--id-pool slots are well-formed v4 UUIDs built from the seeded rng via
uuid::Builder::from_random_bytes, the rest stay numeric. Precomputing
the pool keeps the id-reuse semantics (upserts overwrite live points,
deletes and retrieves hit them) that fresh per-op random UUIDs would
lose, and keeps runs seed-reproducible.
Sampling consumes a single range draw per id, exactly like the previous
NumId draw, so fraction 0 consumes no extra rng draws and reproduces
the numeric-only op stream byte-for-byte. The harness smoke tests run
with fraction 0.5, and the fraction is recorded in the trace header.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Align the workspace dependency pin with the already-resolved
Cargo.lock version (includes GHSA-7gcf-g7xr-8hxj fix).
Co-authored-by: Cursor <cursoragent@cursor.com>
Preparation for capping appendable segment growth in the update path
(#9158): a dedicated error for "all appendable segments reached
max_segment_size", so the update pipeline can recognize it and provision a
fresh appendable segment before re-applying the operation.
Maps to a transient service error at the collection level: if it ever
escapes recovery, failed-operation recovery re-applies the operation.
Part 1/5 of the appendable segment overflow fix.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Every vector candidate now carries a distance metric instead of the
hardcoded Dot, threaded into the fixture schema, the CreateVectorName
generator and the read-back prediction.
The model predicts Cosine read-backs exactly by mirroring the engine's
ingestion preprocessing: metric_preprocess follows
NamedVectors::preprocess_dense_vector's per-datatype dispatch and calls
Distance::preprocess_vector itself, so predictions track the engine by
construction (including the identity preprocess of the byte metric,
which stores Uint8 vectors un-normalized). Stored vectors are
preprocessed exactly once (optimizer and CoW moves transfer raw bytes),
so predictions stay exact across moves, including Cosine + Float16.
New candidates: "e" (dense Cosine), "n" (multi-dense Cosine, per-row
normalization), "x" (dense Cosine + Float16), "o" (dense Cosine +
Turbo4, padding-free dim), "j" (dense Euclid), "k" (dense Manhattan).
Euclid/Manhattan preprocess is an identity, so their value is engine
side: Order::SmallBetter comparator coverage.
The startup predictability check now also rejects sparse + non-Dot
(sparse schemas carry no distance) and Turbo4 + Euclid/Manhattan (TQ's
L1/L2 modes store lengths differently from Dot/Cosine and their
copy-on-write re-quantization fixed point is not soak-validated yet).
Soak-validated on seeds 1/2/4/5/6/7/8 (30k ops), including two
restart runs (restart probability 0.002) with the optimizer enabled.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Profiling showed SipHash over JsonPath keys (RandomState::hash_one)
dominating field index lookups in
StructPayloadIndexReadView::estimate_field_condition: the map holds
only a handful of indexed fields, but it is queried per condition per
filter evaluation per query, and hashing the path is the entire
lookup cost.
Switch the field index lookup maps (IndexesMap, ReadOnlyIndexesMap,
the read view borrow, and the query_checker / condition_converter /
value_retriever signatures that receive them) from std HashMap to
AHashMap, already used throughout the crate. Serialized and API-level
schema maps keep std HashMap: they are cold and their type leaks into
conversions.
conditional_search bench (criterion, 100 samples, vs saved baseline):
- struct-conditional-search-query-points: 583.7 us -> 443.5 us (-23.1% median, p ~ 0)
- struct-conditional-search-context-check: 73.2 us -> 64.8 us (-12.6% median, p ~ 0)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add Ram variant, loads from vector storage
* impl live reload
* preopen sparse storage when using ram index
* Batch storage reads when building sparse RAM index
Rebuild `build_ram_index` around a single `read_vectors::<Sequential>`
pass instead of a per-point `get_vector_opt` loop, so Gridstore
coalesces the IO into block reads instead of a round-trip per point.
Benefits both the read-only mutable-RAM rebuild and the writable
`SparseVectorIndex::plan` build path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Skip the three slowest Windows rust-tests (>60s each): turbo multi
random-ops model tests and the turbo Manhattan HNSW quantization case.
Use rstest test_attr for the parametrized case since a top-level ignore
does not apply to all generated tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
* universal_io: add UniversalAppend for atomic single-operation appends
Growing a file previously took a separate set_len + reopen + write dance
(bypassing universal_io and leaving a zero-filled window on crash), and
there was no way to express appends for backends without random-offset
writes.
UniversalAppend::append grows the file by writing at the current end of
file in one atomic grow+write operation and returns the offset at which
the data landed; append_batch lands multiple buffers contiguously in as
few operations as the backend allows. flusher() moves from
UniversalWrite into a new UniversalFlush supertrait so append-only
handles can require it without duplicating the method.
Local backends, both single-syscall:
- MmapFile appends through a dedicated O_APPEND fd (every write(2) /
writev(2) is an atomic grow+write at EOF), then remaps via reopen().
Its flusher also fdatasyncs after appends, since msync alone does not
persist file-size metadata.
- IoUringFile appends via pwritev2(RWF_APPEND). O_APPEND is not an
option there: on Linux, pwrite on an O_APPEND fd appends regardless
of the given offset, which would break positioned writes on clones
sharing the fd.
Concurrent appenders are out of contract (single logical writer);
object-store backends surface the new AppendOffsetConflict error and
recover via reopen() + retry.
* io_bridge: add AsyncWrite/AsyncAppend and appendable BlobFile
Add the write-side backend traits reserved next to AsyncRead: AsyncWrite
(create/remove/save) powers a UniversalWriteFileOps impl on BlobFs
(create-or-truncate put, delete, atomic whole-object save; directory ops
are no-ops), and AsyncAppend — a single-request append where the offset
must equal the current object size, acting as a compare-and-swap token —
powers UniversalAppend on BlobFile.
BlobFile caches the object size across appends (one HEAD for N appends;
a missing object counts as empty so the first append creates it),
concatenates batches into a single request, and drops the cache on
reopen() — the documented recovery path after AppendOffsetConflict. Its
flusher is a no-op: appends are durable once the backend acknowledges
them.
* io_bridge_object_store: native single-request S3 append
object_store has no append support, so issue the PutObject +
x-amz-write-offset-bytes request ourselves, reusing the store's
credential chain (AmazonS3::credentials) and object_store's SigV4
AwsAuthorizer, which signs every header present on the request — no
hand-rolled signing and no direct reqwest dependency. The offset doubles
as a compare-and-swap token: a mismatch (400 InvalidWriteOffset, or 412
on some S3-compatibles) maps to AppendOffsetConflict.
The write-offset append API exists on AWS S3 Express One Zone directory
buckets and compatible stores (e.g. MinIO AiStor) — plain S3 Standard
buckets reject it, and real Express zonal endpoints / session auth are
not verified yet; MinIO-AiStor-compatible stores are the primary target
for now. GCS and Azure sources simply do not implement AsyncAppend.
ObjectStoreSource carries an AppendContext (HTTP client + object URL
base + signing region) built per backend from its config, and gains a
generic AsyncWrite impl (single-put create/save, delete). A test-only
multi-request CAS emulation over InMemory exercises the BlobFile append
stack hermetically; an end-to-end flow against a real append-capable
store is gated behind S3_APPEND_INTEGRATION_TEST=1.
* simple_disk_cache: write-through UniversalAppend for DiskCache
Append to the remote (the single grow+write operation), then write the
same bytes through into the local mirror so tail reads do not re-fetch
what was just uploaded. LocalState::append_local keeps the fetched
bitmap accurate: blocks fully covered by the appended range are marked
fetched, and the pre-append partial tail block — which resize() drops
because set_len zero-fills its gap — is re-marked only when its prefix
was already fetched. If the mirror turns out stale (the remote grew
behind our back), append falls back to resize-only and lazy fetches
heal the gap on the next read.
Writeable opens are now allowed on DiskCacheFs solely to enable append;
DiskCache still never implements UniversalWrite. The writeable flag
propagates to the remote handle, which is opened buffered instead of
O_DIRECT: appends write through the page cache, which O_DIRECT reads on
the same fd would fight (and IoUringFile rejects appends on
prevent_caching handles). The remote-immutability docs are relaxed to
append-only with an immutable prefix, matching what reopen() already
assumed.
Includes a full-stack composition test: DiskCache write-through over
BlobFile offset tracking over an in-memory object store.
* io_bridge_object_store: build the append HTTP client lazily
Opening a source from an AwsConfig eagerly built the reqwest client
(TLS setup, connection pool) even when append was never used. Keep the
AppendContext construction to pure config (allow_http flag, object URL
base, signing region) and build the client on first append instead,
cached in an Arc<OnceLock> shared across clones of the source — and
thus across the file handles opened from it. Sources that never append
now pay nothing; client-construction errors surface on the first append
instead of at open.
* universal_io: test that append grows the regular file on disk
The conformance suite reads appended bytes back through universal-io
handles; also assert the underlying regular file itself — created
outside universal_io, verified with plain fs reads — for both local
backends.
* Mention why we use custom HTTP client, object_store crate has no support
* io_bridge_object_store: reject appends unconfirmed by the size header
A store without write-offset support may accept the signed PutObject as
a plain put — replacing the object with just the appended bytes — and
return 2xx (community MinIO did exactly this before 2025-05, commit
minio/minio@6d18dba9). The old success path fabricated the new length
when x-amz-object-size was missing, so the destruction stayed invisible
while every subsequent append repeated it.
Require the x-amz-object-size response header (returned by AWS and
MinIO AiStor appends) for any append at offset > 0 and fail loudly
without it. Offset-0 appends are equivalent to a whole-object write, so
they remain valid either way — a misconfigured store now fails on the
second append instead of never.
* io_bridge_object_store: honor endpoint/region env vars for appends
With AwsCredentials::Default the store is built via
AmazonS3Builder::from_env, which honors AWS_ENDPOINT_URL_S3,
AWS_ENDPOINT_URL, AWS_ENDPOINT, AWS_REGION and AWS_DEFAULT_REGION — but
append_context derived the append URL and SigV4 region only from the
typed config fields. An env-configured deployment would read from one
host while signing and sending appends to
https://{bucket}.s3.us-east-1.amazonaws.com.
Resolve the append endpoint and region the same way build_store does:
explicit config first, then (default credential chain only) the same
environment variables, with AWS_ENDPOINT_URL_S3 taking precedence as in
from_env. The resolution is a pure function over an injected env lookup
so the test does not touch process-global environment state.
* simple_disk_cache: delegate the append flusher to the remote
DiskCache's UniversalFlush impl was an unconditional no-op, justified by
object-store appends being durable on acknowledgement — but the impl is
generic over any appendable remote, and for local remotes (MmapFile,
IoUringFile, exactly the compositions the tests instantiate) that
silently dropped the fdatasync the UniversalAppend contract requires:
append, flush Ok, power loss, appended bytes gone.
Delegate to the remote's flusher once the cache is materialized: local
remotes get their sync, object-store flushers remain no-ops, and a
never-materialized cache has made no appends so a no-op stays correct.
* io_bridge_object_store: retry transient append failures
The append RPC was a single unretried HTTP attempt, while every other
request in this stack goes through object_store's retry layer — a
routine transient 503 SlowDown or connection reset failed the append
hard where a concurrent read would have silently recovered.
Retry connection errors, 5xx and 429 up to three attempts with a short
linear backoff, re-signing per attempt (the SigV4 signature embeds the
request date). Retrying is safe because the write offset is a
compare-and-swap; the one ambiguity — an attempt that landed but whose
acknowledgement was lost — surfaces as a write-offset conflict on the
retry, which is reconciled with a HEAD: under the single-writer
contract, an object size of exactly offset + data_len proves the tail
is ours, so the append reports success instead of a spurious conflict
(whose reopen-and-retry recovery would duplicate the record).
* universal_io: forward TypedStorage::flusher for any UniversalFlush
The flusher forwarding lived in TypedStorage's S: UniversalWrite impl
block, so append-only storages (DiskCache, BlobFile — UniversalAppend +
UniversalFlush but not UniversalWrite) offered append through the
wrapper while the durability flusher the append contract mandates was
unreachable without going through .inner.
Move it to an S: UniversalFlush block: UniversalWrite implies
UniversalFlush, so existing callers resolve unchanged, and duplicating
the method instead would have hit E0592 on backends implementing both —
the very ambiguity UniversalFlush was extracted to avoid.
* io_bridge_object_store: surface unbuildable append requests as errors
Request building could panic on two reachable paths: url accepts URIs
the http crate rejects (IPv6 zone identifiers, URIs beyond u16::MAX
bytes), and AppendContext::new is public so the object URL base is not
guaranteed to be a base URL. Both expects become S3Config errors, so a
configuration edge case fails the append instead of panicking the
thread driving the bridge runtime.
* simple_disk_cache: don't fail appends the remote already committed
append_impl committed to the remote first and returned Err when the
subsequent local-mirror update failed (e.g. ENOSPC on the cache volume)
— indistinguishable from "nothing was appended", so a retrying caller
would duplicate the record on the remote.
The mirror is cache maintenance, not part of the append: on a failed
write-through, log and degrade to bare growth so lazy fetches heal the
unmarked blocks (safe — blocks are only marked fetched after their
bytes landed). Only an unresizable mirror still surfaces an error, and
the UniversalAppend contract now documents that an append Err does not
guarantee nothing was appended: reopen() and re-check the length before
retrying.
* universal_io: bounds-check positioned io_uring writes against EOF
The UniversalAppend contract states that UniversalWrite::write beyond
the end-of-file fails and append is the only growth path — mmap
enforces it, but IoUringFile's write/write_batch/write_multi were
unchecked pwrites that silently extended the file with a zero-filled
hole, inflating subsequent append offsets.
Check every positioned write against the file length (fstat once per
call), matching mmap's OutOfBounds semantics, and generalize the
regression test to run on both local backends including the batched
path.
* io_bridge: reject appends on handles opened without writeable
BlobFs::open dropped OpenOptions entirely, so a BlobFile opened with
writeable: false still accepted appends — mmap and DiskCache enforce
the writeable requirement, the blob backend silently didn't, and a
stray append through a nominally read-only handle would mutate a shared
object.
Thread OpenOptions::writeable into BlobFile and reject appends with
PermissionDenied when it is unset, mirroring the other backends.
Directly-constructed handles (BlobFile::new/open, which take no
OpenOptions) remain writeable. With every backend now enforcing the
flag, the UniversalAppend contract drops its 'where the backend
enforces open modes' hedge.
* simple_disk_cache: answer empty appends from the mirror
An empty append still went through remote.append_batch, so it returned
the remote's live end-of-file — which can diverge from what this
handle's len() and reads observe when the remote grew behind our back —
while leaving the stale mirror unhealed (unlike a non-empty append in
the same state, which resizes).
Accept empty appends early: return the mirror length without touching
the remote at all, keeping the answer consistent with the handle's own
view. The trait contract now spells out that empty appends return the
handle's view of the end of file without growth I/O.
* universal_io: grow the mmap in place after appends
Every mmap append ended in a full reopen(): an open+fstat+close by path
just to learn the new length, and — on the non-Linux fallback, which
rebuilds the mapping with the open-time populate flag — a re-population
of the ENTIRE file per append, making appends O(file size) for handles
opened with Populate::Blocking. Populating after an append is pointless
anyway: we just touched the data we wrote.
Extract the remap machinery into remap_to() (reopen() keeps its exact
semantics, populate included) and add grow_mapping(): a stat-free grow
that never re-populates. Appends learn the new length from a single
fstat on the already-open O_APPEND fd — kept rather than trusting the
mapping length, which is stale exactly in the externally-grown-remote
scenario the disk cache heals through lazy fetches (the foreign-growth
tests catch the difference). The mirror's resize() passes the length it
just set_len'd, dropping its stat round-trip entirely.
Per small append this is write+fstat+mremap, down from
write+open+fstat+close+mremap, with no populate anywhere.
* universal_io: share the mmap append fd across clones
The flusher captured the per-clone append_file at flusher-creation
time, so a flusher obtained from a sibling clone — or created before
the handle's first append — msynced the shared mapping's data pages but
skipped the fdatasync that persists the appended file size: a
half-persist where a crash loses the acknowledged tail even though a
flusher ran after the appends (writer thread + long-lived flush-worker
clone is exactly the natural WAL shape).
Store the fd in an Arc<OnceLock> shared by all clones and read it at
flush time instead of capture time: any clone's append makes every
handle's flusher sync the size metadata, whatever the clone/flusher
creation order. Initialization races between clones keep exactly one
fd. No hot-path cost: reads and positioned writes never touch the cell,
and the append path pays one atomic load next to its syscalls. The
interior mutability also lets append_fd take &self.
* universal_io: document the clone remap hazard truthfully
The remap SAFETY comment claimed moving is safe "since we are holding
&mut self" — which says nothing about clones: they share the mapping
but keep their own raw ptr/len copies, so after a moving (or, on
non-Linux, replacing) remap a sibling clone's next read dereferences an
unmapped address. The trait contract understated the same hazard as a
concurrent-read constraint, while the UB persists after append returns.
State the contract once on MmapFile (clones must reopen before reading
after any growth; a stale clone read is undefined behavior, not a stale
view), correct the SAFETY argument to rely on it explicitly, annotate
the as_bytes unsafe blocks that depend on it, and sharpen the
UniversalAppend contract bullet accordingly. Making clones structurally
safe (resolving ptr/len through the shared Arc) is deliberately left as
a separate change.
* universal_io: share the vectored append machinery between backends
The IOV_MAX-chunking / EINTR-retry / WriteZero / advance_slices loop
existed twice — as local_file_ops::write_all_vectored (mmap) and
inlined around pwritev2 in IoUringFile::append_slices — along with a
verbatim collect/cast/filter-empties preamble in both append_batch
impls. Two copies of subtle short-write handling introduced by one
branch will diverge the first time only one of them gets a fix.
Add an io::Write adapter whose write_vectored issues
pwritev2(RWF_APPEND), letting the io_uring append delegate to the
shared write_all_vectored, and hoist the slice collection into
local_file_ops::collect_append_slices. IOV_MAX becomes private to the
one function enforcing it. No behavior change; the existing conformance
tests (including the beyond-IOV_MAX batch) cover both backends through
the shared path.
* io_bridge_object_store: add s3_express to the test config helper
The AwsConfig struct gained the s3_express field; update the
resolve-endpoint test helper accordingly.
* universal_io: run the append conformance suite over the S3 stack
Promote the backend-generic UniversalAppend battery (offsets, batches
across IOV_MAX, empty appends, read-after-append, reopen visibility,
flusher) from a private test into universal_io::conformance, exposed
under the testing feature so backend crates can run the identical
suite. mmap and io_uring keep running it as before; the object-store
bridge now runs it too, over BlobFs/BlobFile with the in-memory
offset-CAS append emulation — so local file system and S3 append
behavior are asserted by the same test. The real write-offset RPC
remains covered by the gated test_native_append_flow integration test.
* Swap order
* universal_io: disambiguate the io_uring crate import
The import reorder dropped the leading `::`, making `io_uring`
ambiguous with this very module (pulled into scope by the
`use super::*` glob) and breaking the build.
* io_bridge: don't materialize the mock object on rejected appends
MutableMockSource::append called get_or_insert_with before validating
the offset, so a rejected stale append against a missing object left an
empty entry behind (exists() flipping true) — a fidelity gap versus the
real backends, where a rejected append has no side effects. Check the
offset against the current length first and only materialize the buffer
on a match.
* universal_io: disambiguate the io_uring crate import
Restore the leading `::` on the io_uring crate import — without it the
name is ambiguous with this very module, which the `use super::*` glob
pulls into scope, and the crate fails to compile. Matches the sibling
files (pool.rs, runtime.rs), which already import via `::io_uring`.
* io_bridge_object_store: treat 404 under a nonzero append offset as a conflict
A missing object while the handle expected a nonzero end-of-file is a
stale view (the object was deleted behind our back) — the same
situation as an offset mismatch, with the same reopen-and-retry
recovery, and it is exactly what the in-memory emulation and the
io_bridge mock already report. The RPC path mapped every 404 to
NotFound instead, so the three implementations disagreed on the same
logical case. Keep NotFound for offset-0 appends, where a 404 is a
genuine missing-target error (e.g. a missing bucket) that retrying
cannot heal.
* Preallocate vector
* universal_io: disallow appends through the disk cache
Appends must go directly to the backing storage (mmap, io_uring, S3) —
the disk cache is strictly read-only again. Remove DiskCache's
UniversalAppend/UniversalFlush impls and the mirror write-through
machinery (LocalState::append_local), reject writeable opens at
DiskCacheFs::open, and drop the writeable/prevent_caching plumbing that
existed solely for cached appends, restoring read-only remote handles.
Attempting to append through the cache is now a compile-time error (the
trait impl no longer exists), and opening a cached handle writeable is
rejected at runtime, covered by a test in each backend variant.
* universal_io: make append idempotent via caller-supplied offset
Append now takes the byte offset where the data must land:
append(offset, data) -> Result<()>. Every backend validates that the
offset equals the current end of file before writing (mmap and io_uring
fstat the fd, object stores validate server-side via
x-amz-write-offset-bytes); on mismatch nothing is written and the append
fails with AppendOffsetConflict. Retrying an already-landed append
therefore conflicts instead of appending twice, and recovery is
re-deriving the offset from len().
BlobFile no longer tracks the object length locally; the store's own
offset check is the compare-and-swap.
* Review remarks
* Validate file length in S3 append response
* Fix linting, we don't mind a large enum variant on index builder
* universal_io: conformance-test stale-handle append conflict recovery
Promote the two-handle conflict scenario from the in-memory BlobFile
test into the backend-generic conformance battery: a second writeable
handle grows the file, the stale handle's append conflicts cleanly (the
offset check runs against the file, not the handle's view), and the
contract's documented recovery — reopen, re-check the length, append at
the real end — lands the data exactly once. Now exercised over mmap,
io_uring, and the object-store stack instead of only the in-memory
emulation.
* io_bridge_object_store: stub-server tests for append response handling
The native append's HTTP state machine was only exercised by the gated
live-store integration test (S3_APPEND_INTEGRATION_TEST=1), so none of
its branches ran in CI. Cover them hermetically against a minimal local
HTTP stub — one connection per canned response, no new dependencies:
- the signed write-offset PUT, and the new-size validation on success
(matching, mismatching, unparseable, and absent size headers — the
absent case at offset zero and past it);
- conflict mapping for 400 InvalidWriteOffset, 412, and 404 under a
nonzero offset, with 404 at offset zero staying NotFound, and a 400
without the conflict code staying a plain error;
- 429/5xx retries re-sending the same offset, giving up after
MAX_ATTEMPTS, and the lost-acknowledgement reconciliation via HEAD
(accepted when the object ends at offset + len, rejected otherwise);
- the status + body excerpt on unexpected failures.
* simple_disk_cache: statically assert the cache stays read-only
Disallowing appends through the disk cache made them a compile-time
error by removing the impls; pin that with assert_not_impl_any so the
UniversalAppend/UniversalFlush/UniversalWrite impls cannot quietly
return. Runtime rejection of writeable opens stays covered per backend
variant.
* Add Float16 and Uint8 storage datatypes to the model tester
Extend VectorCandidate with a datatype override and fold the DenseTurbo
kind into Dense + Some(Turbo4) so storage datatype has a single source
of truth. Two new initially-active candidates exercise half-precision
("h", dense 6) and unsigned-byte ("y", dense 4) storage; "c" carries an
explicit Some(Float32) to cover schema configs that spell the default
datatype out.
The model predicts lossy read-backs through the engine's own
PrimitiveVectorElement impls (as Turbo4 reuses turbo_storage_roundtrip)
and compares them exactly: both round-trips are deterministic and
idempotent, so they stay bit-stable across optimizer moves, WAL replay,
and reloads. Uint8 components are drawn from 0.0..256.0 since the
storage truncates with `x as u8` and unit-range draws would collapse to
zeros.
A compile-time assertion rejects datatype overrides on non-Dense
candidates: the fixture's sparse/multi-dense arms ignore the field and
multi-dense read-backs are compared without a round-trip prediction, so
a lossy multi-dense candidate would soak-panic with a false divergence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Plumb Float16 and Uint8 multi-dense support in the model tester
Multi-dense storage converts the flattened matrix component-wise
(from_float_multivector), so per-row round-trips through the same
PrimitiveVectorElement impls predict read-backs exactly. The fixture's
multi-dense arm now applies the candidate datatype (matching the
CreateVectorName path), model_vector predicts per-row, and two new
initially-active candidates exercise the combination: "w"
(MultiDense(5), Float16) and "z" (MultiDense(3), Uint8).
The compile-time candidate check narrows to the combinations that
remain unpredicted: Turbo4 multi-dense (the multivector quantization
path differs from the per-vector turbo_storage_roundtrip) and sparse
with any datatype override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Make datatype match exhaustive in random_dense_vec
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review findings on datatype plumbing
- Start candidate "c" active so the explicit Float32 schema path runs in
default soaks (CreateVectorName is FORCE_OFF by default)
- Single Float16/Uint8 roundtrip dispatch shared by the dense and
multi-dense arms of model_vector
- Hoist shared fixture builder plumbing into dense_params_builder
- Build one DenseVectorConfig literal in the CreateVectorName generator
- Inline single-caller datatype_of wrapper
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Replace const-eval candidate check with a startup assert
Const eval forbids iterators, forcing an index-based while loop. A plain
function called at the top of run() reads better, still fails before any
op is applied, and names the offending candidate in the panic message.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fold INITIAL_ACTIVE into an initially_active candidate field
The hand-maintained list duplicated ALL_CANDIDATES (11 of 12 names) and
had to be kept in sync when adding candidates; forgetting it was silent
since CreateVectorName is FORCE_OFF by default, so a forgotten name got
zero default-soak coverage. Each candidate now declares its activation
inline and the fixture and run() filter on it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [AI] piggyback on in-flight requests
* cleanup tests
* Insert in-flight fetch through Slab's VacantEntry (#9840)
Replace the peek-key-then-insert pattern with slab's vacant_entry() API:
inserting through the reserved entry guarantees the fetch lands on the
key the remote read was tagged with, instead of relying on nothing
touching the slab between the peek and the insert. A failed remote
schedule still leaves no trace, as VacantEntry allocates nothing until
insert.
get_or_init_remote_pipeline now takes the field instead of &mut self so
the VacantEntry can hold a disjoint borrow of in_flight across the
schedule call.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Standalone usage reference for `edge-shard-query`: backends (S3 / GCS /
uio-grpc), connection and tuning flags, the scroll / search / search-sparse
sub-commands, filtering, and the live-reload diff mode.
Written to stand on its own, so it can be shared as a link without also
sharing the source.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The inverse Hadamard rotation is linear, so it distributes over
subtraction: |R'd1 - R'd2| = |R'(d1 - d2)|. Subtract the dequantized
vectors in rotated space and inverse-rotate the difference once,
instead of inverse-rotating both vectors per score call.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
