* make `EncodedStorage::for_each_batch` mandatory
* make `DenseVectorStorageRead::for_each_in_dense_batch` mandatory
* make `DenseTQVectorStorage::for_each_in_dense_batch` mandatory
* make `DenseTQVectorStorage::read_dense_tq_bytes` mandatory
* make `QueryScorer::score_stored_batch` mandatory
...and implement for tq multivectors
* [AI] make `IdTrackerRead::internal_versions_batch` mandatory
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* [AI] make `IdTrackerRead::external_ids_batch` mandatory
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* [AI] make `DiskMappingsSource::resolve_internal_batch` mandatory
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Fix use-after-free in u8-quantized vector reads on owning storages
EncodedVectorsU8::get_vec_ptr extracted a raw pointer from the buffer
returned by EncodedStorage::get_vector_data and dropped the buffer
before the pointer was dereferenced. For storages returning
Cow::Borrowed (mmap) this happened to be sound, but for storages that
return Cow::Owned (disk-cache misses, uring backends) every user of
get_vec_ptr read freed memory: the internal SIMD scorers,
encode_internal_vector, and get_quantized_vector_offset_and_code, which
exported the dangling buffer through a safe &[u8].
Make parse_vec_data return the code as a slice borrowing the input, so
the borrow checker forces callers to keep the storage buffer alive
while reading it, and drop get_vec_ptr entirely. This matches how
encoded_vectors_binary already binds the buffers at its scoring sites.
Change get_quantized_vector_offset_and_code to return the code as a
sub-view of the buffer Cow itself, and adapt its GPU caller.
The new regression test drives these paths through a storage that
always returns owned buffers; under Miri it reproduces the
use-after-free on the previous code and passes with this fix.
Fixes#9799
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* additional assertion
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* quantization over tq strorage
are you happy fmt
are you happy clippy
rotation refactor
rotation refactor
fix tests
better test name
review remarks
dont rotate query in raw scorer
tq sources revert
quantized_scoring_datatype revert
simplify
are you happy fmt
* remove old comments
* review remarks
* feat: add read-only QuantizedVectors generic over UniversalRead
Introduce `QuantizedVectorsRead<S>` / `QuantizedVectorStorageRead<S>`, a
read-only counterpart of `QuantizedVectors` organized like
`VectorStorageReadEnum`: generic over the `UniversalRead` backend `S`,
opened from existing on-disk data, with no create/upsert/builder path and
no disk writes.
Highlights:
- Keep both in-RAM (`*Ram`) and read-only mmap (`*Mmap`) variants; drop the
appendable `*ChunkedMmap` variants (the only mutable ones).
- All bulk reads go through `S`: add `QuantizedRamStorage::from_universal_read`
and `MultivectorOffsetsStorageRam::open`, and make
`MultivectorOffsetsStorageMmap` generic over `S` (default `MmapFile`).
- Share scorer construction between the read-write and read-only enums via a
`QuantizedScorerDispatch` trait, so only the per-variant match is duplicated
while the datatype/distance and per-query dispatch live once in the builder.
Tested: read-only vs read-write scorer parity (scalar/binary/product, single
and multivector, RAM and mmap), covering both `raw_scorer` and
`raw_internal_scorer`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: route quantized RAM loading through UniversalRead
Follow-up to the read-only quantized work, removing direct-filesystem reads
and load-path duplication:
- `EncodedVectors{U8,PQ,Bin,TQ}::load` now take a `UniversalRead` filesystem
and read metadata via `read_json_via` instead of `fs::read_to_string` —
every read flows through universal IO. Single-vector `load` returns
`common::universal_io::Result`; `validate_storage_vector_size` stays
`std::io::Result`.
- Add `common::universal_io::OneshotFile<S>`: a thin RAII wrapper over any
`UniversalRead` handle that evicts the data from cache via `clear_ram_cache`
on drop (the universal-IO counterpart of `fs::OneshotFile`).
- Collapse `QuantizedRamStorage::{from_file, from_universal_read}` into one
`from_file<S: UniversalRead>`. It reads the whole file in a single access
(no separate `len()` round-trip — cheaper on S3-like backends) and loads via
the new `VolatileChunkedVectors::extend`, which inserts one chunk per
`copy_from_slice` instead of one vector at a time.
- RW callers pass the local `READ_FS` (mmap) backend; the read-only loader
passes its `S`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: load appendable (chunked) quantization format read-only
`storage_type` only selects the on-disk layout, so the read-only quantized
storage must be able to load both — the immutable flat format and the
appendable chunked format (produced only by Binary/TurboQuant). Previously the
read-only loader rejected the Mutable layout outright.
- Add `QuantizedChunkedStorageRead<S>` and `MultivectorOffsetsStorageChunkedRead<S>`,
read-only wrappers over the existing `ChunkedVectorsRead<_, S>` primitive
(mirrors the dense read-view's chunked read storage). Generic over the
`UniversalRead` backend, on-disk, no write path.
- Add `BinaryChunked`/`TQChunked` (+ multi) variants to `QuantizedVectorStorageRead`
and wire them through every accessor and the scorer dispatch.
- Route `storage_type == Mutable` to the chunked read variants in the loader and
drop the blanket rejection.
- Tests: parametrize the read-only/read-write parity tests over `storage_type`
and add chunked (Mutable) cases for binary/turbo, single and multivector.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: split quantized chunked & multivector storage into modules
`quantized_chunked_mmap_storage.rs` and `quantized_multivector_storage.rs` had
grown to hold several unrelated structures, making them hard to navigate.
Convert each into a module (pure code movement, no behavior change):
- quantized_chunked_mmap_storage/{read_write,read_only}.rs — the appendable
mmap storage + builder vs. the read-only chunked storage.
- quantized_multivector_storage/{mod,offsets}.rs — the core
`QuantizedMultivectorStorage` + offset traits stay in mod.rs; the four
`MultivectorOffsetsStorage*` backends move to offsets.rs.
Public paths are unchanged via re-exports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: single source of truth for quantized vector size
The on-disk stride (bytes per quantized vector), including the binary
`u8`(multi)/`u128`(single) word-type choice, was independently re-derived in
every load/open site with no compile-time link — the kind of value-level
duplication that silently diverges (it already caused the binary multi
`u8`/`u128` bug).
Extract it into one place:
- `QuantizedVectors::quantized_vector_size(quantization_config, vector_parameters, is_multi)`
and the `QuantizedVectorsConfig::quantized_vector_size(is_multi)` convenience.
Route every reader through it:
- read-only `open_single`/`open_multi` and the read-write `{scalar,pq,binary,turbo}`
loaders now hoist `config.quantized_vector_size(is_multi)` once instead of
recomputing the per-method formula per branch.
The create (write) path keeps its own computation for now; it stays guarded by
`validate_storage_vector_size` and the read-only/read-write parity tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: flat-match quantized loaders on a shared QuantizedStorageKind
The loaders chose the concrete storage variant via a nested
`method × (storage_type / is_ram)` match, re-derived independently in the
read-only and read-write paths — hard to follow and easy to drift.
- Add `QuantizedStorageKind` + `QuantizedVectorsConfig::storage_kind(on_disk)`
(and `is_ram(on_disk)`): the single place the method × backend decision lives.
- Both loaders now compute the kind once and use a flat 10-arm match:
- read-only `open_single`/`open_multi`;
- read-write `load_single`/`load_multi` (consolidating the four per-method
`{scalar,pq,binary,turbo}/load.rs` files, which are removed).
Both matches are exhaustive over the same enum, so the read and read-write
variants can no longer fall out of sync without a compile error.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: genericize write-capable chunked quantized storages over Fs
Drop the hardcoded `MmapFile` specialization from `QuantizedChunkedStorage`,
`QuantizedChunkedStorageBuilder`, and `MultivectorOffsetsStorageChunked`. They
now expose the `Fs` backend (defaulting to `MmapFile`) and accept the fs handle
as a parameter, matching the read-only variants.
Introduce a single shared `ReadFile` type alias and `READ_FS` value handle in
the `quantized_vectors` module root, used by the create, load, and storage enum
paths so the local-file backend is named in one place.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: move is_in_ram_or_mmap match into UniversalKind
Deduplicate the identical kind-to-residency match in the read-only and
write-capable chunked quantized storages by adding
UniversalKind::is_in_ram_or_mmap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: adapt TurboVectorStorage to quantized rename + update_from move
Integration fix after rebasing onto dev's TurboQuant work:
- QuantizedChunkedMmapStorage -> QuantizedChunkedStorage<MmapFile>
- update_from moved off the VectorStorage trait onto an inherent method,
matching the per-kind sub-trait refactor; TurboVectorStorage implements
neither DenseVectorStorage<T> nor the other kind traits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add TQ+ ErrorCorrection on top of renorm
Per-coordinate shift+scale fits each rotated, length-rescaled coord onto
the codebook's N(0, 1) grid before quantization. EncodedVectorsTQ::encode
runs a first pass to fit the stats when TQMode::Plus.
Scoring stays correct under renorm's `scaling_factor` framework:
- Asymmetric: precompute_query scales `Q .* D'` and stashes `qm = ⟨Q, M⟩`
on EncodedQueryTQ; score_precomputed adds qm to raw_dot before applying
scaling_factor.
- Symmetric: scalar slow path computes `Σ X+_a X+_b D'_i² + xm_a + xm_b
− ⟨M, M⟩` (xm stored per vector in extras, mm_const cached on
ErrorCorrection). Result feeds the existing `* v1_scale * v2_scale` arms.
SIMD reuse for this path is a follow-up.
Storage layout: TQMode::Plus extras are 4 bytes longer (xm appended after
scaling_factor). Zero-vector inputs skip EC application so renorm's
existing zero-norm guard keeps producing score ≈ 0 within tolerance.
VectorStats refactor: streaming `VectorStatsBuilder` so the Plus first
pass can feed Welford with a reused buffer; `build` now takes `dim`
directly and is generic over `T: Into<f64>`.
Integration tests run on both Normal and Plus via rstest cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix TQ+ recall regression on non-uniform per-coord variance data
Two compounding bugs in the renorm + TQ+ composition:
1. centroid_norm was measured on `X+` centroids, which have chi-squared
norm distribution across vectors (~10% spread for d=256). renorm
assumed `cn` should be deterministic `sqrt(d)` (just quantization
drift), so the per-vector correction ended up amplifying intrinsic
chi-squared noise into ranking error. Fix: revert EC per coord before
measuring (`c · D' + M`), matching llama-turbo-quant. The reverted
centroids approximate `rescaled` which has length `sqrt(d)` exactly
by construction. dequantize follows the same convention so the
stored `scaling_factor = l2/cn` round-trips back to the original l2.
2. Asymmetric query path pre-scaled `Q* = R_q · D'` before SIMD encoding.
The SIMD encoder normalizes by `max(|input|)`, so a query whose coords
span 5× magnitude (which `R_q · D'` does on real data) loses precision
on the small-D' coords. Fix: keep `rotated` unscaled, store it as a
side field on `EncodedQueryTQ`, and use a scalar decode-and-dot path
for TQ+ (`Σ R_q_i · c_i · D'_i + qm`). SIMD support for this is a
follow-up.
Catches both via `recall_skewed_data` test on data with 8 spike-variance
input coords. Without the fixes, Bits4 Plus dropped to 0.93 vs Normal
0.98; after the fixes, Plus tracks Normal within 2%.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ asymmetric: skip the SIMD encoding entirely
The TQ+ asymmetric path doesn't use the SIMD-encoded query — it goes
through `score_precomputed_ec` with `rotated_query`. So building the
SIMD form was wasted work + memory. Make `data` an `Option` and only
populate it for the cases that actually use it (Normal mode any distance,
TQ+ L1 via dequantize fallback).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Revert TQ+ to SIMD scoring path
The whole point of TQ+ is that scoring code-paths stay identical between
Normal and Plus modes — only the query precomputation changes. We
pre-scale `Q* = R_q · D'` so the existing SIMD raw_dot computes
`⟨Q · D', X+⟩` directly, then add `qm` and apply renorm's scaling_factor.
Drops `score_precomputed_ec` and `EncodedQueryTQ::rotated_query`. The
recall regression that motivated the scalar fallback was entirely from
the `compute_centroid_norm` bug (measuring `‖X+‖` instead of `‖rescaled‖`)
fixed in the prior commit; SIMD precision was a red herring.
`recall_skewed_data` confirms: Bits4 Normal=0.984 / Plus=0.978, Bits2
Normal=0.902 / Plus=0.908.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ Bits1: widen query encoding to 12 bits
For 1-bit storage with TQ+, the per-coord `D' = 1/scale` pre-scaling on
the query can push some coords toward the small end of the SIMD encoder's
integer range (which normalizes by `max(|input|)`). At 8 bits those small
coords lose precision; at 12 bits the rounding error drops ~10× per the
existing `test_query_dotprod_matches_reference` parity test.
`Query1bitSimd` is already generic over BITS so this is just a new
`EncodedQueryTQData::Bits1Wide(Query1bitSimd<12>)` variant + a TQ+/Bits1
dispatch in `precompute_query`. Bits2/Bits4 don't need this — their
storage is fine-grained enough that query precision isn't the bottleneck,
and their SIMD encoders aren't generic over BITS today.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* are you happy fmt
* TQ+ Bits1Wide: bump to 16-bit query quantization (kernel max)
12 bits helped on real datasets but not enough — push to the kernel's
ceiling of 16. `Query1bitSimd<BITS>` asserts `BITS ∈ [2, 16]`, so this
is the most precision the existing SIMD path can give us before needing
a wider integer kernel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+: shift by per-coord median, not mean
For 1-bit storage the codebook boundary sits at 0, so post-shift values
are quantized purely by sign. Median is the sign-balance point of the
distribution; mean isn't (skewed coords pull mean off the median).
On anisotropic embeddings — dbpedia-openai being the reference case —
mean-based shift produced a ~60/40 biased sign distribution per coord,
losing 1-bit's representational capacity. Median-based shift restores
50/50 and matches llama-turbo-quant's behavior. Higher bit-widths are
less sensitive but still benefit; the codebook boundaries still lie at
distribution-percentile-aware positions when the data is centered on
the median.
Median requires per-coord samples in memory, so cap the stats pass at
10K vectors. Estimates converge fast (~√N) — 10K is plenty even for
million-vector indexes. The encoding pass still processes every vector.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ Bits1Wide: revert to 12-bit query quantization
The recall regression on anisotropic data was the mean-vs-median shift,
not query precision. 12 bits is enough headroom for the per-coord D'
pre-scaling and avoids the extra storage of the 16-bit form.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ Bits1Wide: bump back to 16-bit query quantization
12 bits helped a bit but not enough on the real dataset. Bump to the
kernel's ceiling. If 16 still isn't enough, the next step is checking
whether the gap is real (re-measure llama branch) before widening the
SIMD integer kernel itself.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* are you happy clippy
* use mean
* 1bit error correction
* review remarks
* are you happy clippy
* review remarks
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* use vector statistics for scalar bq query
* fix minor error
* remove test_binary_scalar_internal test
* do NOT use special file for storing vector stats
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* # This is a combination of 7 commits.
* SameAsStorage default value
* fix coderabbit warnings
* neon for u8 bq
* sse for u8 bq
* fix windows build
* rename function
* add comments
* fmt
* fix arm build
* review remarks
* bq encodings
* are you happy clippy
* are you happy clippy
* are you happy clippy
* are you happy clippy
* gpu tests
* update models
* are you happy fmt
* move additional bits to the end
* fix tests
* Welford's Algorithm
* review remarks
* are you happy clippy
* remove debug println in test
* coderabit nitpicks
* remove unnecessary clone and partialeq
* Use f64 for Welford's Algorithm
* try fix ci
* revert cargo-nextest
* add debug assertions
* Bump Rust edition to 2024
* gen is a reserved keyword now
* Remove ref mut on references
* Mark extern C as unsafe
* Wrap unsafe function bodies in unsafe block
* Geo hash implements Copy, don't reference but pass by value instead
* Replace secluded self import with parent
* Update execute_cluster_read_operation with new match semantics
* Fix lifetime issue
* Replace map_or with is_none_or
* set_var is unsafe now
* Reformat
* bump and migrate to rand 0.9.0
also bump rand_distr to 0.5.0 to match it
* Migrate AVX2 and SSE implementations
* Remove unused thread_rng placeholders
* More random migrations
* Migrate GPU tests
* bump seed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
