* Benches: use SmallRng instead of ChaCha12-based generators
All benchmarks used StdRng or rand::rng() (ThreadRng), both backed by the
ChaCha12 block cipher in rand 0.10. Benchmarks do not need crypto-strength
randomness, and several draw random values inside the timed closure, so
cipher work was included in the measurement itself.
Switch every bench target to SmallRng (Xoshiro256++), and key the HNSW
graph cache and sparse index cache by RNG algorithm so stale caches built
from the old generator are not reused against newly generated vectors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Benches: replace free-function rand::random with local SmallRng
Addresses review: rand::random draws from the thread RNG (ChaCha12),
including inside the timed loop of the pq score benchmark.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The inverse Hadamard rotation is linear, so it distributes over
subtraction: |R'd1 - R'd2| = |R'(d1 - d2)|. Subtract the dequantized
vectors in rotated space and inverse-rotate the difference once,
instead of inverse-rotating both vectors per score call.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Speed up TurboQuant Hadamard rotation by ~3x
Profiling apply_inverse showed 44% of cycles in the Fisher-Yates swap
replay (serial LCG plus a 64-bit modulo per element), not in the WHT.
- Materialize the shared permutations as index maps once in
HadamardRotation::new; each apply-time permutation becomes a flat
gather pass ping-ponging between the vector and a thread-local
scratch buffer. The replay path stays as the cfg(test) parity oracle.
- Fuse consecutive WHT outer butterfly stage pairs (h, 2h) into one
pass over the array (AVX2), halving memory traffic for h >= 16.
- Fuse the normalization multiply into the transform's final-stage
stores (wht_dispatch_scaled), removing the separate normalize pass.
Output is bit-identical on all paths: pinned by the existing
struct-vs-replay and SIMD-vs-scalar bit-equal tests plus a new
wht_dispatch_scaled parity test. NEON is unchanged.
Criterion hadamard bench (Zen 5): apply 2.8-3.7x faster across dims
128-4096 (1024: 6.98us -> 2.03us), apply_inverse 2.8-3.5x
(1024: 6.10us -> 2.05us).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: harden gather contract, small-dim parity, grow-only scratch
- Mark gather_permuted as unsafe fn: the get_unchecked justification relied
on a caller invariant (map indices in range) that the signature did not
surface; document it as a # Safety contract instead.
- Add dims 5 and 50 to static_rotation_matches_struct_and_roundtrips to pin
the map path against the replay oracle on degenerate chunk splits.
- Make the thread-local gather scratch grow-only, so threads alternating
between dims no longer shrink and re-zero the buffer on every call.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Hard-assert input length in apply/apply_inverse
Fail fast at the public boundary: with debug_assert only, a release
build would WHT-normalize a wrong-length slice before the gather's
hard length asserts panic. Flagged by CodeRabbit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Fix use-after-free in u8-quantized vector reads on owning storages
EncodedVectorsU8::get_vec_ptr extracted a raw pointer from the buffer
returned by EncodedStorage::get_vector_data and dropped the buffer
before the pointer was dereferenced. For storages returning
Cow::Borrowed (mmap) this happened to be sound, but for storages that
return Cow::Owned (disk-cache misses, uring backends) every user of
get_vec_ptr read freed memory: the internal SIMD scorers,
encode_internal_vector, and get_quantized_vector_offset_and_code, which
exported the dangling buffer through a safe &[u8].
Make parse_vec_data return the code as a slice borrowing the input, so
the borrow checker forces callers to keep the storage buffer alive
while reading it, and drop get_vec_ptr entirely. This matches how
encoded_vectors_binary already binds the buffers at its scoring sites.
Change get_quantized_vector_offset_and_code to return the code as a
sub-view of the buffer Cow itself, and adapt its GPU caller.
The new regression test drives these paths through a storage that
always returns owned buffers; under Miri it reproduces the
use-after-free on the previous code and passes with this fix.
Fixes#9799
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* additional assertion
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* Remove from_iter_instead_of_collect from workspace lints
The lint was removed from clippy (beta) and now triggers
renamed_and_removed_lints warnings in every crate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::chunks_exact_to_as_chunks
Replace chunks_exact with a constant chunk size by as_chunks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::needless_late_init
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::useless_borrows_in_formatting
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::uninlined_format_args
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::for_kv_map
Iterate map values directly instead of discarding keys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_large_err on QueueProxyShard::new_from_version
The Err variant intentionally hands the LocalShard back to the caller.
Same pattern as the existing allow on ForwardProxyShard::new.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_unit_err on wait_for_consensus_commit
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* quantization over tq strorage
are you happy fmt
are you happy clippy
rotation refactor
rotation refactor
fix tests
better test name
review remarks
dont rotate query in raw scorer
tq sources revert
quantized_scoring_datatype revert
simplify
are you happy fmt
* remove old comments
* review remarks
* remove unused iter_offsets
* Replace MultivectorOffsetsStorage::iter_offsets with callback-based for_each_offset
First step of removing the iterator-returning read API (whose deep,
composable generic types blow up mangled symbol size). Convert the
offsets read from an iterator to a callback the caller pushes into:
- trait method iter_offsets -> for_each_offset(ids, FnMut(usize, MultivectorOffset))
returning common::universal_io::Result<()>
- Mmap impl now uses the callback read_batch (drops one read_iter use)
- Ram / Chunked impls push into the callback; Chunked still goes through
iter_vectors for now (converted in a later step)
- the single caller (for_each_in_multi_batch) passes a closure
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Implement read_batch directly on ReadPipeline, not via read_iter
read_batch now drives the pipeline itself (refill-then-wait loop, like
read_multi_iter) and invokes the callback per result, instead of
consuming the iterator returned by read_iter. A step toward removing the
iterator-returning read API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove the ReadMulti RPC from the StorageRead gRPC service
ReadMulti was the only real consumer of UniversalRead::read_multi (which
itself relies on read_multi_iter). Removing the RPC end-to-end clears the
path to dropping that read API. StorageReadService keeps all its other
RPCs (ListFiles, FileExists, FileLength, ReadBytes, ReadBytesStream,
ReadWhole, ReadBatch).
- proto: drop `rpc ReadMulti` + ReadMulti{Entry,Request,Response}
- regenerated lib/api + uio-client generated code; drop ReadMulti
validation rules in lib/api/build.rs
- tonic: delete the read_multi handler + its 2 tests
- uio-client: delete Client::read_multi, the mock-server impl, and 2 tests
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove UniversalRead::read_multi
Its only real consumer was the StorageRead ReadMulti gRPC handler (removed
in the previous commit); the two wrapper forwarders had no callers. Drop
the trait method and both forwarders (typed/read_only), and remove the
io_uring test that only existed to compare read_multi vs read_multi_iter
(read_multi_iter stays covered by the other tests). Another step toward
removing the iterator-returning read API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drive ReadPipeline directly in gridstore read_from_pages
Replace the read_multi_iter call in Pages::read_from_pages with a direct
pipeline loop (refill-then-drain), scheduling each multi-page read on its
own page file. Behavior unchanged; another step toward removing the
iterator-returning read_multi_iter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drive ReadPipeline directly in gridstore read_batch_from_pages
Replace the second read_multi_iter call (in Pages::read_batch_from_pages)
with a direct pipeline loop, scheduling each (ReadMeta, page, range) on its
own page file and propagating errors via GridstoreError. Single/multi-page
buffering and out-of-order reassembly are unchanged. No more read_multi_iter
in gridstore.
Measured overhead on warm mmap (both paths are zero-copy borrows): ~0.3 ns
per read of fixed control cost, flat across read sizes — well under 0.1% of
a real payload read.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drive ReadPipeline directly in on-disk postings with_posting_views
Replace read_iter in OnDiskPostings::with_posting_views with a direct
pipeline loop. wait_bytemuck yields a file-borrowed Cow, so postings are
still stored zero-copy in raw_postings (a read_batch swap would have forced
an owned copy of every posting list per query on the mmap backend).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Read on-disk posting headers via read_batch, drop the HeadersBatch iterator
headers_iter now reads headers with the callback read_batch API: each header
is parsed (copied) out of the read bytes, so nothing borrows the file past the
read — no pipeline needed. Since the read is now eager, HeadersBatch holds the
collected Vec<HeaderResult> directly instead of a Box<dyn Iterator>, dropping
the boxing, the dynamic dispatch, and the struct's lifetime parameter.
with_posting_views takes the Vec and still pipelines the posting reads.
Removes the last read_iter use in this file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Use read_batch in simple_disk_cache populate_from
populate_from reads one byte per block only to fault blocks into the local
cache, discarding the bytes — a no-op-callback read_batch fits exactly. Drives
the same DiskCachePipeline as before; one less read_iter caller.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Implement read_iter directly on ReadPipeline, not via read_multi_iter
read_iter now drives the pipeline itself (refill-then-wait loop, mirroring
read_bytes_iter) instead of mapping its ranges onto self and calling
read_multi_iter. Same signature and iterator contract, so all callers are
unchanged. Leaves iter_vectors as read_multi_iter's only remaining caller.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "Drive ReadPipeline directly in on-disk postings with_posting_views"
This reverts commit c02c41f17b.
* Add callback-based for_each_vector next to iter_vectors
for_each_vector drives the ReadPipeline directly across chunk files and
invokes a fallible callback per flattened multi-vector, returning
OperationResult, instead of returning an iterator built on read_multi_iter.
Callers will migrate onto it so iter_vectors (read_multi_iter's last caller)
can be removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Make dense for_each_in_batch / for_each_in_dense_batch fallible
Thread OperationResult up the dense batch-read path so io_uring read
errors propagate instead of being .expect()ed deep inside the storage.
The for_each_in_dense_batch scorer path (custom/metric query scorers)
now carries the Result to the infallible score() boundary where it is
.expect()ed; read_vectors keeps its () signature and .expect()s locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Convert dense read_vectors to for_each_vector
The read-only and appendable dense storage read_vectors impls drove
ChunkedVectors::iter_vectors directly; switch them to the callback-based
for_each_vector and .expect() the result at the (infallible) read_vectors
boundary. Removes the last dense-path iter_vectors callers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Convert quantized for_each_offset to for_each_vector + OperationResult
The two chunked MultivectorOffsetsStorage impls drove iter_vectors to read
the offset table; switch them to the callback-based for_each_vector.
for_each_vector returns OperationResult, so upgrade the for_each_offset
trait (and all four impls) from universal_io::Result to OperationResult
(the universal_io -> Operation direction, via ?). No error downgrade.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Convert EncodedStorage/EncodedVectors iter_batch to callback for_each_batch
The two chunked-mmap EncodedStorage impls drove ChunkedVectors::iter_vectors
to back iter_batch. Replace the iterator-returning iter_batch on both the
EncodedStorage and EncodedVectors traits (quantization crate) with a callback
for_each_batch(FnMut(usize, &[u8])), and switch the chunked impls to
for_each_vector. The callback is infallible: the chunked impls .expect() the
read internally, matching iter_vectors' prior panic-on-read-error behavior,
so no OperationError is downgraded. Scorers and the multivector readers adopt
the callback; the accumulating multivector path owns (to_vec) only when it
must buffer across components.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Convert multivector read paths to for_each_vector
The read_only multivector free fn chained two iter_vectors (offsets feeding
vectors) - the recursive iterator nesting behind the worst symbol bloat.
Replace it with a callback for_each_vector that resolves the per-point
offsets into a Vec first, then drives ChunkedVectors::for_each_vector over
the flattened vectors. Migrate both multivector storages' read_vectors and
for_each_in_batch_multi accordingly, .expect()ing at their infallible
boundaries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove read_multi_iter and ChunkedVectors::iter_vectors
With every caller migrated to callback-based for_each_vector/for_each_batch,
delete the last iterator-returning multi-read APIs: ChunkedVectors::iter_vectors
(segment) and the read_multi_iter trait method plus its mmap override, the
TypedStorage/ReadOnly wrapper forwarders, and the two io_uring unit tests.
These deeply-nested monomorphized iterator types (read_multi_iter feeding
read_multi_iter) produced >1 MiB mangled drop_in_place symbols that overflowed
the macOS ld symbol-name limit; the callback rewrite eliminates them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Pass owned Cow through for_each_vector/for_each_batch to avoid a copy
The callback-based readers handed the callback a borrowed `&[u8]`/`&[T]`,
forcing the quantized multivector batch read to `to_vec()` each sub-vector
into its per-point buffer. But io_uring-like backends already return a freshly
owned buffer per read (`ACow::Owned`), so that was a redundant second copy.
Change `ChunkedVectorsRead::for_each_vector` and the `EncodedStorage` /
`EncodedVectors` `for_each_batch` callbacks to receive `Cow<[..]>` by value.
The buffering path now `into_owned()`s it — a move when the backend returned
owned (the case this path targets), a copy only for a borrowed Cow (mmap),
which never reaches this path. Immediate-use callers (scorers,
score_point_max_similarity, the dense/multivector readers) just deref the Cow;
the dense readers drop their now-redundant `Cow::Borrowed` wraps.
Also clarifies the multivector reader: `SubVectorOwner`/`owners`/
`sub_vector_offsets` naming, docs, and a corrected comment noting the per-point
buffer is what makes regrouping order-independent under out-of-order completion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drop the removed ReadMulti JWT access test; fix clippy unwrap_or_default
The ReadMulti StorageRead RPC was removed earlier in this branch, so the
consensus JWT-access test (and its registry entry) for it must go too. Also
switch the multivector buffer's `or_insert_with(SmallVec::new)` to
`or_default()` per clippy::unwrap_or_default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add MmapFile::read_batch
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* feat: live_reload for read-only quantized vectors
* Propagate live_reload params through quantized chunked storage
Thread fs, deleted_points, new_points and hw_counter through the full
quantized live_reload chain instead of synthesizing empty deltas and a
disposable hardware counter at the leaf storages.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: use LiveReload trait in chunked mmap
* fix: use LiveReload trait
* fix: compiler errors
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Partial function impls for TurboVectorStorage
* create/load functionality + tests
# Conflicts:
# lib/segment/src/vector_storage/turbo/mod.rs
# lib/segment/src/vector_storage/turbo/turbo_encoded_vectors.rs
* Fix vectors are not rotated back
* update_from impl
* Rebase fixes
* Proper update_from for ChunkedMmap + better tests
* Improve test coverage
* Move update_from to new DenseTQVectorStorage trait
* Review remark quantization::DistanceType
* Add populate and clear_cache to TurboVectorStorage and TurboEncodedVectorStorage
* feat: add read-only QuantizedVectors generic over UniversalRead
Introduce `QuantizedVectorsRead<S>` / `QuantizedVectorStorageRead<S>`, a
read-only counterpart of `QuantizedVectors` organized like
`VectorStorageReadEnum`: generic over the `UniversalRead` backend `S`,
opened from existing on-disk data, with no create/upsert/builder path and
no disk writes.
Highlights:
- Keep both in-RAM (`*Ram`) and read-only mmap (`*Mmap`) variants; drop the
appendable `*ChunkedMmap` variants (the only mutable ones).
- All bulk reads go through `S`: add `QuantizedRamStorage::from_universal_read`
and `MultivectorOffsetsStorageRam::open`, and make
`MultivectorOffsetsStorageMmap` generic over `S` (default `MmapFile`).
- Share scorer construction between the read-write and read-only enums via a
`QuantizedScorerDispatch` trait, so only the per-variant match is duplicated
while the datatype/distance and per-query dispatch live once in the builder.
Tested: read-only vs read-write scorer parity (scalar/binary/product, single
and multivector, RAM and mmap), covering both `raw_scorer` and
`raw_internal_scorer`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: route quantized RAM loading through UniversalRead
Follow-up to the read-only quantized work, removing direct-filesystem reads
and load-path duplication:
- `EncodedVectors{U8,PQ,Bin,TQ}::load` now take a `UniversalRead` filesystem
and read metadata via `read_json_via` instead of `fs::read_to_string` —
every read flows through universal IO. Single-vector `load` returns
`common::universal_io::Result`; `validate_storage_vector_size` stays
`std::io::Result`.
- Add `common::universal_io::OneshotFile<S>`: a thin RAII wrapper over any
`UniversalRead` handle that evicts the data from cache via `clear_ram_cache`
on drop (the universal-IO counterpart of `fs::OneshotFile`).
- Collapse `QuantizedRamStorage::{from_file, from_universal_read}` into one
`from_file<S: UniversalRead>`. It reads the whole file in a single access
(no separate `len()` round-trip — cheaper on S3-like backends) and loads via
the new `VolatileChunkedVectors::extend`, which inserts one chunk per
`copy_from_slice` instead of one vector at a time.
- RW callers pass the local `READ_FS` (mmap) backend; the read-only loader
passes its `S`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: load appendable (chunked) quantization format read-only
`storage_type` only selects the on-disk layout, so the read-only quantized
storage must be able to load both — the immutable flat format and the
appendable chunked format (produced only by Binary/TurboQuant). Previously the
read-only loader rejected the Mutable layout outright.
- Add `QuantizedChunkedStorageRead<S>` and `MultivectorOffsetsStorageChunkedRead<S>`,
read-only wrappers over the existing `ChunkedVectorsRead<_, S>` primitive
(mirrors the dense read-view's chunked read storage). Generic over the
`UniversalRead` backend, on-disk, no write path.
- Add `BinaryChunked`/`TQChunked` (+ multi) variants to `QuantizedVectorStorageRead`
and wire them through every accessor and the scorer dispatch.
- Route `storage_type == Mutable` to the chunked read variants in the loader and
drop the blanket rejection.
- Tests: parametrize the read-only/read-write parity tests over `storage_type`
and add chunked (Mutable) cases for binary/turbo, single and multivector.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: split quantized chunked & multivector storage into modules
`quantized_chunked_mmap_storage.rs` and `quantized_multivector_storage.rs` had
grown to hold several unrelated structures, making them hard to navigate.
Convert each into a module (pure code movement, no behavior change):
- quantized_chunked_mmap_storage/{read_write,read_only}.rs — the appendable
mmap storage + builder vs. the read-only chunked storage.
- quantized_multivector_storage/{mod,offsets}.rs — the core
`QuantizedMultivectorStorage` + offset traits stay in mod.rs; the four
`MultivectorOffsetsStorage*` backends move to offsets.rs.
Public paths are unchanged via re-exports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: single source of truth for quantized vector size
The on-disk stride (bytes per quantized vector), including the binary
`u8`(multi)/`u128`(single) word-type choice, was independently re-derived in
every load/open site with no compile-time link — the kind of value-level
duplication that silently diverges (it already caused the binary multi
`u8`/`u128` bug).
Extract it into one place:
- `QuantizedVectors::quantized_vector_size(quantization_config, vector_parameters, is_multi)`
and the `QuantizedVectorsConfig::quantized_vector_size(is_multi)` convenience.
Route every reader through it:
- read-only `open_single`/`open_multi` and the read-write `{scalar,pq,binary,turbo}`
loaders now hoist `config.quantized_vector_size(is_multi)` once instead of
recomputing the per-method formula per branch.
The create (write) path keeps its own computation for now; it stays guarded by
`validate_storage_vector_size` and the read-only/read-write parity tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: flat-match quantized loaders on a shared QuantizedStorageKind
The loaders chose the concrete storage variant via a nested
`method × (storage_type / is_ram)` match, re-derived independently in the
read-only and read-write paths — hard to follow and easy to drift.
- Add `QuantizedStorageKind` + `QuantizedVectorsConfig::storage_kind(on_disk)`
(and `is_ram(on_disk)`): the single place the method × backend decision lives.
- Both loaders now compute the kind once and use a flat 10-arm match:
- read-only `open_single`/`open_multi`;
- read-write `load_single`/`load_multi` (consolidating the four per-method
`{scalar,pq,binary,turbo}/load.rs` files, which are removed).
Both matches are exhaustive over the same enum, so the read and read-write
variants can no longer fall out of sync without a compile error.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: genericize write-capable chunked quantized storages over Fs
Drop the hardcoded `MmapFile` specialization from `QuantizedChunkedStorage`,
`QuantizedChunkedStorageBuilder`, and `MultivectorOffsetsStorageChunked`. They
now expose the `Fs` backend (defaulting to `MmapFile`) and accept the fs handle
as a parameter, matching the read-only variants.
Introduce a single shared `ReadFile` type alias and `READ_FS` value handle in
the `quantized_vectors` module root, used by the create, load, and storage enum
paths so the local-file backend is named in one place.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: move is_in_ram_or_mmap match into UniversalKind
Deduplicate the identical kind-to-residency match in the read-only and
write-capable chunked quantized storages by adding
UniversalKind::is_in_ram_or_mmap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: adapt TurboVectorStorage to quantized rename + update_from move
Integration fix after rebasing onto dev's TurboQuant work:
- QuantizedChunkedMmapStorage -> QuantizedChunkedStorage<MmapFile>
- update_from moved off the VectorStorage trait onto an inherent method,
matching the per-kind sub-trait refactor; TurboVectorStorage implements
neither DenseVectorStorage<T> nor the other kind traits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EncodedVectorsTQ was the only quantizer that did not override the
EncodedVectors::heap_size_bytes() trait method, so it fell back to the
default of 0. For the RAM-backed variants (TQRam/TQRamMulti) this meant
the entire resident quantized dataset was reported as 0 bytes and
misclassified as fully on-disk by the MemoryReporter; the always-resident
quantizer tables (rotation + TQ+ error-correction vectors) and encoding
buffer were also uncounted for every variant.
Make heap_size_bytes() a required trait method (remove the default impl)
so every quantizer must account for its own heap explicitly, then add the
missing TurboQuant implementation: storage backend + quantizer tables +
encoding buffer.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* use generic in QuantizedMmapStorage
* rename to QuantizedStorage
* be explicit about S
* rename builder to `QuantizedStorageBuilder`
* rename file to `quantized_storage.rs`
* Add `EncodedStorage::for_each_in_batch` method
* Implement `for_each_in_batch` method for `QuantizedChunkedMmapStorage`
* Add `EncodedVectors::for_each_in_batch` method
* Add `EncodedVectors::score` method
* Implement `score_stored_batch` for `QuantizedQueryScorer` and `Quanti…
* Remove `TElement` and `TMetric` type parameters from `QuantizedMultiQ…
* Use `QuantizedMultiQueryScorer` when building `raw_internal_scorer`...
* Add TQ+ ErrorCorrection on top of renorm
Per-coordinate shift+scale fits each rotated, length-rescaled coord onto
the codebook's N(0, 1) grid before quantization. EncodedVectorsTQ::encode
runs a first pass to fit the stats when TQMode::Plus.
Scoring stays correct under renorm's `scaling_factor` framework:
- Asymmetric: precompute_query scales `Q .* D'` and stashes `qm = ⟨Q, M⟩`
on EncodedQueryTQ; score_precomputed adds qm to raw_dot before applying
scaling_factor.
- Symmetric: scalar slow path computes `Σ X+_a X+_b D'_i² + xm_a + xm_b
− ⟨M, M⟩` (xm stored per vector in extras, mm_const cached on
ErrorCorrection). Result feeds the existing `* v1_scale * v2_scale` arms.
SIMD reuse for this path is a follow-up.
Storage layout: TQMode::Plus extras are 4 bytes longer (xm appended after
scaling_factor). Zero-vector inputs skip EC application so renorm's
existing zero-norm guard keeps producing score ≈ 0 within tolerance.
VectorStats refactor: streaming `VectorStatsBuilder` so the Plus first
pass can feed Welford with a reused buffer; `build` now takes `dim`
directly and is generic over `T: Into<f64>`.
Integration tests run on both Normal and Plus via rstest cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix TQ+ recall regression on non-uniform per-coord variance data
Two compounding bugs in the renorm + TQ+ composition:
1. centroid_norm was measured on `X+` centroids, which have chi-squared
norm distribution across vectors (~10% spread for d=256). renorm
assumed `cn` should be deterministic `sqrt(d)` (just quantization
drift), so the per-vector correction ended up amplifying intrinsic
chi-squared noise into ranking error. Fix: revert EC per coord before
measuring (`c · D' + M`), matching llama-turbo-quant. The reverted
centroids approximate `rescaled` which has length `sqrt(d)` exactly
by construction. dequantize follows the same convention so the
stored `scaling_factor = l2/cn` round-trips back to the original l2.
2. Asymmetric query path pre-scaled `Q* = R_q · D'` before SIMD encoding.
The SIMD encoder normalizes by `max(|input|)`, so a query whose coords
span 5× magnitude (which `R_q · D'` does on real data) loses precision
on the small-D' coords. Fix: keep `rotated` unscaled, store it as a
side field on `EncodedQueryTQ`, and use a scalar decode-and-dot path
for TQ+ (`Σ R_q_i · c_i · D'_i + qm`). SIMD support for this is a
follow-up.
Catches both via `recall_skewed_data` test on data with 8 spike-variance
input coords. Without the fixes, Bits4 Plus dropped to 0.93 vs Normal
0.98; after the fixes, Plus tracks Normal within 2%.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ asymmetric: skip the SIMD encoding entirely
The TQ+ asymmetric path doesn't use the SIMD-encoded query — it goes
through `score_precomputed_ec` with `rotated_query`. So building the
SIMD form was wasted work + memory. Make `data` an `Option` and only
populate it for the cases that actually use it (Normal mode any distance,
TQ+ L1 via dequantize fallback).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Revert TQ+ to SIMD scoring path
The whole point of TQ+ is that scoring code-paths stay identical between
Normal and Plus modes — only the query precomputation changes. We
pre-scale `Q* = R_q · D'` so the existing SIMD raw_dot computes
`⟨Q · D', X+⟩` directly, then add `qm` and apply renorm's scaling_factor.
Drops `score_precomputed_ec` and `EncodedQueryTQ::rotated_query`. The
recall regression that motivated the scalar fallback was entirely from
the `compute_centroid_norm` bug (measuring `‖X+‖` instead of `‖rescaled‖`)
fixed in the prior commit; SIMD precision was a red herring.
`recall_skewed_data` confirms: Bits4 Normal=0.984 / Plus=0.978, Bits2
Normal=0.902 / Plus=0.908.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ Bits1: widen query encoding to 12 bits
For 1-bit storage with TQ+, the per-coord `D' = 1/scale` pre-scaling on
the query can push some coords toward the small end of the SIMD encoder's
integer range (which normalizes by `max(|input|)`). At 8 bits those small
coords lose precision; at 12 bits the rounding error drops ~10× per the
existing `test_query_dotprod_matches_reference` parity test.
`Query1bitSimd` is already generic over BITS so this is just a new
`EncodedQueryTQData::Bits1Wide(Query1bitSimd<12>)` variant + a TQ+/Bits1
dispatch in `precompute_query`. Bits2/Bits4 don't need this — their
storage is fine-grained enough that query precision isn't the bottleneck,
and their SIMD encoders aren't generic over BITS today.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* are you happy fmt
* TQ+ Bits1Wide: bump to 16-bit query quantization (kernel max)
12 bits helped on real datasets but not enough — push to the kernel's
ceiling of 16. `Query1bitSimd<BITS>` asserts `BITS ∈ [2, 16]`, so this
is the most precision the existing SIMD path can give us before needing
a wider integer kernel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+: shift by per-coord median, not mean
For 1-bit storage the codebook boundary sits at 0, so post-shift values
are quantized purely by sign. Median is the sign-balance point of the
distribution; mean isn't (skewed coords pull mean off the median).
On anisotropic embeddings — dbpedia-openai being the reference case —
mean-based shift produced a ~60/40 biased sign distribution per coord,
losing 1-bit's representational capacity. Median-based shift restores
50/50 and matches llama-turbo-quant's behavior. Higher bit-widths are
less sensitive but still benefit; the codebook boundaries still lie at
distribution-percentile-aware positions when the data is centered on
the median.
Median requires per-coord samples in memory, so cap the stats pass at
10K vectors. Estimates converge fast (~√N) — 10K is plenty even for
million-vector indexes. The encoding pass still processes every vector.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ Bits1Wide: revert to 12-bit query quantization
The recall regression on anisotropic data was the mean-vs-median shift,
not query precision. 12 bits is enough headroom for the per-coord D'
pre-scaling and avoids the extra storage of the 16-bit form.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ Bits1Wide: bump back to 16-bit query quantization
12 bits helped a bit but not enough on the real dataset. Bump to the
kernel's ceiling. If 16 still isn't enough, the next step is checking
whether the gap is real (re-measure llama branch) before widening the
SIMD integer kernel itself.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* are you happy clippy
* use mean
* 1bit error correction
* review remarks
* are you happy clippy
* review remarks
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add mincore-based memory stats to MmapFile
Add `resident_bytes()`, `disk_bytes()`, and `probe_memory_stats()` methods
to `MmapFile` for measuring page cache residency via `mincore(2)`. This is
the foundation for per-collection memory usage reporting.
Also extract `page_size()` as a public function in `mmap::advice`, replacing
the internal `PAGE_SIZE_MASK` with a direct page size cache.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] introduce trait for reporting memory usage per component
* [AI] memory reporter implementation for vector storage
* [AI] implement MemoryReporter for QuantizedVectors
* [AI] implement MemoryReporter for VectorIndexEnum
* Implement MemoryReporter for IdTrackerEnum with RAM estimation
Add ram_usage_bytes() to all ID tracker types and their data structures:
- PointMappings, CompressedPointMappings, CompressedVersions,
CompressedInternalToExternal, CompressedExternalToInternal
- MutableIdTracker, ImmutableIdTracker, InMemoryIdTracker
All ID trackers load their data into RAM (none use mmap for working data).
Files are reported as OnDisk (persistence only), actual RAM footprint
is reported via extra_ram_bytes. Uses struct destructuring to ensure
new fields trigger compile errors if not accounted for.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] implement MemoryReporter for PayloadStorageEnum and adjust FileStorageIntent
* [AI] implement MemoryReporter for PayloadStorageEnum and adjust FileStorageIntent
* [AI] implement MemoryReporter for payload indexes: in-ram structures memory consumtion computation + caching
* [AI] implement MemoryReporter for payload indexes: in-ram structures memory consumtion computation + caching
* [AI] segment-level memory usage report
* [AI] Block 3: Aggregation Layer and Data Model + internal api for remote shard
* [AI] REST API handler
* fmt
* [AI] clippy fixes
* [AI] macos fix + proxy segment fix
* [AI] make text index estimation a bit more correct
* fix is_on_disk reporting for dense_vector_storage
* fix after rebase
* [AI] deep account for quantized vectors RAM usage + unify chunk size + shring volatile storage after load
* remove debug log
* cache in test
* make manual test easier to run
* rollback chunk size diff, but keep it for test only
* review fixes
* Use exhaustive match
* Use div_ceil on bits everywhere
It does not seem to be strictly necessary because the number of bits
should already be a multiple of the used container size bytes. Still
it's good practice to be careful with this calculation.
* Improve heap size bytes for encoded product quantization vectors
* Include vector stats for binary quantized vectors
* In volatile chunked vectors, include heap allocated vector
* Include rest of heap allocated structures for mutable map index
* In mutable geo index, the hash map is also heap allocated
* Update tests/manual/test_memory_reporting.py
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* [ai] Replace manual into mappings with Into::into
* Reformat
* [ai] Use implicit .iter
* Don't iterate over keys too
* [ai] Replace unwrap_or
* Reformat
* [ai] Use as_deref and then_some
* [ai] Use more to_string
* [ai] Use explicitly typed into conversions
* Reformat
* [ai] More explicit into conversions
* Reformat
