* feat: add read-only QuantizedVectors generic over UniversalRead
Introduce `QuantizedVectorsRead<S>` / `QuantizedVectorStorageRead<S>`, a
read-only counterpart of `QuantizedVectors` organized like
`VectorStorageReadEnum`: generic over the `UniversalRead` backend `S`,
opened from existing on-disk data, with no create/upsert/builder path and
no disk writes.
Highlights:
- Keep both in-RAM (`*Ram`) and read-only mmap (`*Mmap`) variants; drop the
appendable `*ChunkedMmap` variants (the only mutable ones).
- All bulk reads go through `S`: add `QuantizedRamStorage::from_universal_read`
and `MultivectorOffsetsStorageRam::open`, and make
`MultivectorOffsetsStorageMmap` generic over `S` (default `MmapFile`).
- Share scorer construction between the read-write and read-only enums via a
`QuantizedScorerDispatch` trait, so only the per-variant match is duplicated
while the datatype/distance and per-query dispatch live once in the builder.
Tested: read-only vs read-write scorer parity (scalar/binary/product, single
and multivector, RAM and mmap), covering both `raw_scorer` and
`raw_internal_scorer`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: route quantized RAM loading through UniversalRead
Follow-up to the read-only quantized work, removing direct-filesystem reads
and load-path duplication:
- `EncodedVectors{U8,PQ,Bin,TQ}::load` now take a `UniversalRead` filesystem
and read metadata via `read_json_via` instead of `fs::read_to_string` —
every read flows through universal IO. Single-vector `load` returns
`common::universal_io::Result`; `validate_storage_vector_size` stays
`std::io::Result`.
- Add `common::universal_io::OneshotFile<S>`: a thin RAII wrapper over any
`UniversalRead` handle that evicts the data from cache via `clear_ram_cache`
on drop (the universal-IO counterpart of `fs::OneshotFile`).
- Collapse `QuantizedRamStorage::{from_file, from_universal_read}` into one
`from_file<S: UniversalRead>`. It reads the whole file in a single access
(no separate `len()` round-trip — cheaper on S3-like backends) and loads via
the new `VolatileChunkedVectors::extend`, which inserts one chunk per
`copy_from_slice` instead of one vector at a time.
- RW callers pass the local `READ_FS` (mmap) backend; the read-only loader
passes its `S`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: load appendable (chunked) quantization format read-only
`storage_type` only selects the on-disk layout, so the read-only quantized
storage must be able to load both — the immutable flat format and the
appendable chunked format (produced only by Binary/TurboQuant). Previously the
read-only loader rejected the Mutable layout outright.
- Add `QuantizedChunkedStorageRead<S>` and `MultivectorOffsetsStorageChunkedRead<S>`,
read-only wrappers over the existing `ChunkedVectorsRead<_, S>` primitive
(mirrors the dense read-view's chunked read storage). Generic over the
`UniversalRead` backend, on-disk, no write path.
- Add `BinaryChunked`/`TQChunked` (+ multi) variants to `QuantizedVectorStorageRead`
and wire them through every accessor and the scorer dispatch.
- Route `storage_type == Mutable` to the chunked read variants in the loader and
drop the blanket rejection.
- Tests: parametrize the read-only/read-write parity tests over `storage_type`
and add chunked (Mutable) cases for binary/turbo, single and multivector.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: split quantized chunked & multivector storage into modules
`quantized_chunked_mmap_storage.rs` and `quantized_multivector_storage.rs` had
grown to hold several unrelated structures, making them hard to navigate.
Convert each into a module (pure code movement, no behavior change):
- quantized_chunked_mmap_storage/{read_write,read_only}.rs — the appendable
mmap storage + builder vs. the read-only chunked storage.
- quantized_multivector_storage/{mod,offsets}.rs — the core
`QuantizedMultivectorStorage` + offset traits stay in mod.rs; the four
`MultivectorOffsetsStorage*` backends move to offsets.rs.
Public paths are unchanged via re-exports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: single source of truth for quantized vector size
The on-disk stride (bytes per quantized vector), including the binary
`u8`(multi)/`u128`(single) word-type choice, was independently re-derived in
every load/open site with no compile-time link — the kind of value-level
duplication that silently diverges (it already caused the binary multi
`u8`/`u128` bug).
Extract it into one place:
- `QuantizedVectors::quantized_vector_size(quantization_config, vector_parameters, is_multi)`
and the `QuantizedVectorsConfig::quantized_vector_size(is_multi)` convenience.
Route every reader through it:
- read-only `open_single`/`open_multi` and the read-write `{scalar,pq,binary,turbo}`
loaders now hoist `config.quantized_vector_size(is_multi)` once instead of
recomputing the per-method formula per branch.
The create (write) path keeps its own computation for now; it stays guarded by
`validate_storage_vector_size` and the read-only/read-write parity tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: flat-match quantized loaders on a shared QuantizedStorageKind
The loaders chose the concrete storage variant via a nested
`method × (storage_type / is_ram)` match, re-derived independently in the
read-only and read-write paths — hard to follow and easy to drift.
- Add `QuantizedStorageKind` + `QuantizedVectorsConfig::storage_kind(on_disk)`
(and `is_ram(on_disk)`): the single place the method × backend decision lives.
- Both loaders now compute the kind once and use a flat 10-arm match:
- read-only `open_single`/`open_multi`;
- read-write `load_single`/`load_multi` (consolidating the four per-method
`{scalar,pq,binary,turbo}/load.rs` files, which are removed).
Both matches are exhaustive over the same enum, so the read and read-write
variants can no longer fall out of sync without a compile error.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: genericize write-capable chunked quantized storages over Fs
Drop the hardcoded `MmapFile` specialization from `QuantizedChunkedStorage`,
`QuantizedChunkedStorageBuilder`, and `MultivectorOffsetsStorageChunked`. They
now expose the `Fs` backend (defaulting to `MmapFile`) and accept the fs handle
as a parameter, matching the read-only variants.
Introduce a single shared `ReadFile` type alias and `READ_FS` value handle in
the `quantized_vectors` module root, used by the create, load, and storage enum
paths so the local-file backend is named in one place.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: move is_in_ram_or_mmap match into UniversalKind
Deduplicate the identical kind-to-residency match in the read-only and
write-capable chunked quantized storages by adding
UniversalKind::is_in_ram_or_mmap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: adapt TurboVectorStorage to quantized rename + update_from move
Integration fix after rebasing onto dev's TurboQuant work:
- QuantizedChunkedMmapStorage -> QuantizedChunkedStorage<MmapFile>
- update_from moved off the VectorStorage trait onto an inherent method,
matching the per-kind sub-trait refactor; TurboVectorStorage implements
neither DenseVectorStorage<T> nor the other kind traits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add TQ+ ErrorCorrection on top of renorm
Per-coordinate shift+scale fits each rotated, length-rescaled coord onto
the codebook's N(0, 1) grid before quantization. EncodedVectorsTQ::encode
runs a first pass to fit the stats when TQMode::Plus.
Scoring stays correct under renorm's `scaling_factor` framework:
- Asymmetric: precompute_query scales `Q .* D'` and stashes `qm = ⟨Q, M⟩`
on EncodedQueryTQ; score_precomputed adds qm to raw_dot before applying
scaling_factor.
- Symmetric: scalar slow path computes `Σ X+_a X+_b D'_i² + xm_a + xm_b
− ⟨M, M⟩` (xm stored per vector in extras, mm_const cached on
ErrorCorrection). Result feeds the existing `* v1_scale * v2_scale` arms.
SIMD reuse for this path is a follow-up.
Storage layout: TQMode::Plus extras are 4 bytes longer (xm appended after
scaling_factor). Zero-vector inputs skip EC application so renorm's
existing zero-norm guard keeps producing score ≈ 0 within tolerance.
VectorStats refactor: streaming `VectorStatsBuilder` so the Plus first
pass can feed Welford with a reused buffer; `build` now takes `dim`
directly and is generic over `T: Into<f64>`.
Integration tests run on both Normal and Plus via rstest cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix TQ+ recall regression on non-uniform per-coord variance data
Two compounding bugs in the renorm + TQ+ composition:
1. centroid_norm was measured on `X+` centroids, which have chi-squared
norm distribution across vectors (~10% spread for d=256). renorm
assumed `cn` should be deterministic `sqrt(d)` (just quantization
drift), so the per-vector correction ended up amplifying intrinsic
chi-squared noise into ranking error. Fix: revert EC per coord before
measuring (`c · D' + M`), matching llama-turbo-quant. The reverted
centroids approximate `rescaled` which has length `sqrt(d)` exactly
by construction. dequantize follows the same convention so the
stored `scaling_factor = l2/cn` round-trips back to the original l2.
2. Asymmetric query path pre-scaled `Q* = R_q · D'` before SIMD encoding.
The SIMD encoder normalizes by `max(|input|)`, so a query whose coords
span 5× magnitude (which `R_q · D'` does on real data) loses precision
on the small-D' coords. Fix: keep `rotated` unscaled, store it as a
side field on `EncodedQueryTQ`, and use a scalar decode-and-dot path
for TQ+ (`Σ R_q_i · c_i · D'_i + qm`). SIMD support for this is a
follow-up.
Catches both via `recall_skewed_data` test on data with 8 spike-variance
input coords. Without the fixes, Bits4 Plus dropped to 0.93 vs Normal
0.98; after the fixes, Plus tracks Normal within 2%.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ asymmetric: skip the SIMD encoding entirely
The TQ+ asymmetric path doesn't use the SIMD-encoded query — it goes
through `score_precomputed_ec` with `rotated_query`. So building the
SIMD form was wasted work + memory. Make `data` an `Option` and only
populate it for the cases that actually use it (Normal mode any distance,
TQ+ L1 via dequantize fallback).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Revert TQ+ to SIMD scoring path
The whole point of TQ+ is that scoring code-paths stay identical between
Normal and Plus modes — only the query precomputation changes. We
pre-scale `Q* = R_q · D'` so the existing SIMD raw_dot computes
`⟨Q · D', X+⟩` directly, then add `qm` and apply renorm's scaling_factor.
Drops `score_precomputed_ec` and `EncodedQueryTQ::rotated_query`. The
recall regression that motivated the scalar fallback was entirely from
the `compute_centroid_norm` bug (measuring `‖X+‖` instead of `‖rescaled‖`)
fixed in the prior commit; SIMD precision was a red herring.
`recall_skewed_data` confirms: Bits4 Normal=0.984 / Plus=0.978, Bits2
Normal=0.902 / Plus=0.908.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ Bits1: widen query encoding to 12 bits
For 1-bit storage with TQ+, the per-coord `D' = 1/scale` pre-scaling on
the query can push some coords toward the small end of the SIMD encoder's
integer range (which normalizes by `max(|input|)`). At 8 bits those small
coords lose precision; at 12 bits the rounding error drops ~10× per the
existing `test_query_dotprod_matches_reference` parity test.
`Query1bitSimd` is already generic over BITS so this is just a new
`EncodedQueryTQData::Bits1Wide(Query1bitSimd<12>)` variant + a TQ+/Bits1
dispatch in `precompute_query`. Bits2/Bits4 don't need this — their
storage is fine-grained enough that query precision isn't the bottleneck,
and their SIMD encoders aren't generic over BITS today.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* are you happy fmt
* TQ+ Bits1Wide: bump to 16-bit query quantization (kernel max)
12 bits helped on real datasets but not enough — push to the kernel's
ceiling of 16. `Query1bitSimd<BITS>` asserts `BITS ∈ [2, 16]`, so this
is the most precision the existing SIMD path can give us before needing
a wider integer kernel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+: shift by per-coord median, not mean
For 1-bit storage the codebook boundary sits at 0, so post-shift values
are quantized purely by sign. Median is the sign-balance point of the
distribution; mean isn't (skewed coords pull mean off the median).
On anisotropic embeddings — dbpedia-openai being the reference case —
mean-based shift produced a ~60/40 biased sign distribution per coord,
losing 1-bit's representational capacity. Median-based shift restores
50/50 and matches llama-turbo-quant's behavior. Higher bit-widths are
less sensitive but still benefit; the codebook boundaries still lie at
distribution-percentile-aware positions when the data is centered on
the median.
Median requires per-coord samples in memory, so cap the stats pass at
10K vectors. Estimates converge fast (~√N) — 10K is plenty even for
million-vector indexes. The encoding pass still processes every vector.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ Bits1Wide: revert to 12-bit query quantization
The recall regression on anisotropic data was the mean-vs-median shift,
not query precision. 12 bits is enough headroom for the per-coord D'
pre-scaling and avoids the extra storage of the 16-bit form.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ Bits1Wide: bump back to 16-bit query quantization
12 bits helped a bit but not enough on the real dataset. Bump to the
kernel's ceiling. If 16 still isn't enough, the next step is checking
whether the gap is real (re-measure llama branch) before widening the
SIMD integer kernel itself.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* are you happy clippy
* use mean
* 1bit error correction
* review remarks
* are you happy clippy
* review remarks
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* use vector statistics for scalar bq query
* fix minor error
* remove test_binary_scalar_internal test
* do NOT use special file for storing vector stats
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* # This is a combination of 7 commits.
* SameAsStorage default value
* fix coderabbit warnings
* neon for u8 bq
* sse for u8 bq
* fix windows build
* rename function
* add comments
* fmt
* fix arm build
* review remarks
* bq encodings
* are you happy clippy
* are you happy clippy
* are you happy clippy
* are you happy clippy
* gpu tests
* update models
* are you happy fmt
* move additional bits to the end
* fix tests
* Welford's Algorithm
* review remarks
* are you happy clippy
* remove debug println in test
* coderabit nitpicks
* remove unnecessary clone and partialeq
* Use f64 for Welford's Algorithm
* try fix ci
* revert cargo-nextest
* add debug assertions
* Bump Rust edition to 2024
* gen is a reserved keyword now
* Remove ref mut on references
* Mark extern C as unsafe
* Wrap unsafe function bodies in unsafe block
* Geo hash implements Copy, don't reference but pass by value instead
* Replace secluded self import with parent
* Update execute_cluster_read_operation with new match semantics
* Fix lifetime issue
* Replace map_or with is_none_or
* set_var is unsafe now
* Reformat
* bump and migrate to rand 0.9.0
also bump rand_distr to 0.5.0 to match it
* Migrate AVX2 and SSE implementations
* Remove unused thread_rng placeholders
* More random migrations
* Migrate GPU tests
* bump seed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
