* TurboQuantizer::score_precomputed_batch: score a contiguous run of vectors
Batch counterpart of `score_precomputed` for vectors stored back to
back at `quantized_size()`: the width's kernel scores the whole run of
codes in one `dotprod_batch` call, then a second pass applies each
vector's extras. L1 dequantizes per vector and stays a plain loop.
Tested against per-vector `score_precomputed` for every width,
distance, and mode over run lengths that leave every group remainder.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* EncodedStorage::for_each_run: serve consecutive offsets as contiguous runs
`for_each_run(offsets, callback(first, count, bytes))` splits the
offsets into maximal runs of consecutive ids the storage can serve
from one contiguous slice, so a sequential scan resolves chunk lookups
and reads once per run instead of once per vector. The default serves
every vector as its own run; `for_each_consecutive_run` is the shared
run detection for storages that override it, with a per-run cap for
chunk boundaries. The test storage overrides it (its data is one flat
buffer).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* EncodedVectors::score_points: batched scoring entry point, run-batched for TQ
`score_points(query, offsets, scores)` scores a batch of points. The
default keeps the per-vector loop the scorers run today, so SQ/PQ/BQ
are unchanged. TurboQuant overrides it: on RAM/mmap storages it walks
`for_each_run` and scores each contiguous run with one
`score_precomputed_batch` call, hoisting the score inversion out of
the loop; backends with async reads keep the pipelined per-vector
path. Non-consecutive offsets degrade to single-vector runs, so
scattered access keeps its previous cost.
Integration test: `score_points` vs `score_point` for every bit width
and mode, Dot and inverted L2, over sequential, scattered and
descending id orders.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Quantized storages: for_each_run over their contiguous regions
The RAM storage and both chunked mmap storages cap runs at their chunk
boundary and serve each run with one `get_many`; the single-file mmap
storage serves any run as one sequential read. Unit test on the RAM
storage: runs cover every offset once, in order, with bytes identical
to per-point reads, across the internal chunk boundary.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* QuantizedQueryScorer: score batches through EncodedVectors::score_points
Routes `score_stored_batch` through the batched entry point, so
TurboQuant-as-quantization scans score contiguous runs with one kernel
call per run; SQ/PQ/BQ keep the per-vector loop via the default.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* TurboScoring::score_query_batch: run-batched scoring for Turbo4 storages
Adds the batch counterpart of `score_query_bytes` to the trait, with
one shared implementation over the storage's `EncodedStorage`:
consecutive ids are coalesced into contiguous runs, each run scored by
a single `score_precomputed_batch` call, and the metric sign applied
once over the batch. Backends with async reads keep the pipelined
per-vector path. `TurboQueryScorer::score_stored_batch` now calls it.
The batch-vs-single storage test grows to 8192 vectors so a full
ascending scan crosses a chunk boundary of the chunked backend, and
runs that scan on the chunked, mmap and io_uring backends.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* score_precomputed_batch: keep the extras pass in L1
The kernel pass and the extras pass now alternate over sub-runs of 64
vectors instead of each covering the whole run: for a run of several
hundred vectors the second pass otherwise refetched every vector's
extras from L2. Measured with 512-vector runs from the full-scan
driver at dim 512: the regression against 64-vector runs went from
+11 % to +2 %.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Bench: exhaustive search over Turbo4 storages through the plain-index driver
`turbo4_full_scan` runs `BatchFilteredSearcher::peek_top_visible` —
the exact path of a non-indexed search — over 200k normalized random
vectors for Turbo4 as datatype (appendable chunked, in RAM) and Turbo4
as quantization (over a RAM dense storage), at dims 64 to 1024, so the
fixed per-point cost of the scan driver is measured next to the kernel.
`TURBO_SCAN_DIMS=64,128` narrows the dims while iterating.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [UpdateOnly] implement the appendable quantized-vector overlay (dense, Binary/Turbo)
Appendable/plain segments can carry live quantized vectors today: PlainVectorIndex::
update_vector calls quantized_vectors.upsert_vector alongside the raw vector on every
insert (lib/segment/src/index/plain_vector_index/lifecycle.rs), auto-created for a
fresh segment when appendable_quantization is on and the method supports it
(QuantizationConfig::supports_appendable — Binary and Turbo only; Scalar/Product are
policy-gated off regardless of storage backend). The update-only vector-storage stack
(this PR's base) had no equivalent: UpdateOnlyVectorStorage::open never read
quantization_config, and nothing under vector_storage/*/update_only/ mentioned
quantization at all — a segment configured with quantization would silently lose it
end-to-end once written through this path.
This adds UpdateOnlyQuantizedVectors, mirroring QuantizedVectors' auto-create/reopen
behavior but scoped to dense (single-vector) Binary/Turbo — the two methods that
support incremental appends, matching current capability exactly (multivector support
is a follow-up: it needs its own append-only offsets storage, mirroring
MultivectorOffsetsStorageChunked the same way this mirrors QuantizedChunkedStorage).
The only new machinery is UpdateOnlyQuantizedChunkedStorage, an EncodedStorage backed
by UpdateOnlyChunkedVectors (append-only, S: UniversalAppend) instead of
ChunkedVectors' positional writes (S: UniversalWrite) — everything else reuses the
quantization crate's EncodedVectorsBin::encode/load and EncodedVectorsTQ::encode/load
completely unchanged, since both are already generic over the storage backend. It
writes files in the exact layout QuantizedChunkedStorage reads, so a promoted segment's
quantized data reads through the existing, unmodified reader with no new reading code.
UpdateOnlyChunkedVectors gains one addition: a `get` method to read back a single
vector, needed because EncodedVectors::load validates the storage's vector size by
reading vector 0 (skipped when the store is still empty).
Verified: the update-only writer's persisted bytes, read back through the standard
(non-update-only) QuantizedChunkedStorage + EncodedVectorsBin/TQ::load, match a
RAM-backed reference fed the same vectors one at a time through upsert_vector,
byte-for-byte, for both Binary and Turbo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* [UpdateOnly] fix quantized reopen: resume writing shouldn't validate stored reads
The previous commit made reopening a non-empty quantized overlay panic
(EncodedVectorsBin/TQ::load validates a non-empty store by reading its
vector 0, which UpdateOnlyQuantizedChunkedStorage's write-only design
cannot serve) and worked around it with a redundant pre-check plus a
todo!(), narrowing the tests to single-session-only writes.
Both of those were the wrong fix. A writer resuming appends doesn't need
`load`'s read-and-validate — it only needs the fitted metadata (encoding,
stats) to keep encoding consistently, and that invariant already holds by
construction: every vector this writer ever encodes is sized from the same
`quantized_vector_size` `load` and the new path both read. Added
`EncodedVectorsBin`/`EncodedVectorsTQ::reopen_for_write` to the
quantization crate — identical to `load` minus the validating read — and
switched `open_existing` to it. `UpdateOnlyQuantizedChunkedStorage` stays
write-only as originally designed; no new read capability, no pre-check,
no todo.
Tests restored to the original two-writer split (write half, drop, reopen,
write the rest), now genuinely exercising resume-with-data instead of
avoiding it, and still passing byte-for-byte against the reference.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* [UpdateOnly] split EncodedStorage into EncodedStorageWrite + EncodedStorage
A write-only storage (the update-only quantized overlay) had to fake a
full EncodedStorage impl with unreachable!() read stubs just to satisfy
EncodedVectorsBin/TQ's generic bound. Split the trait so a write-only
backend only needs to implement EncodedStorageWrite; EncodedStorage adds
the read methods on top. The overlay now implements EncodedStorageWrite
alone — no panicking stand-ins for methods that don't exist.
* [UpdateOnly] remove UpdateOnlyQuantizedVectors::create
Nothing in this stack builds the first appendable segment of a
collection yet (that's still a todo!() in edge/src/update_only), so
create() had no real caller and open() had to guess from file absence
whether to invoke it. open() now only reopens an overlay create()
already persisted; the bootstrap logic moved into tests.rs as a
private fixture helper, since tests still need it to build fixtures.
* [UpdateOnly] fix CI: codespell typo and lint dead-code on unwired write path
codespell flagged "implementors" (wants "implementers") in two doc
comments. Separately, CI's lint job runs clippy without --all-targets,
so the update-only quantized write path — genuinely unreachable from
any non-test code until #10152 wires it into a segment — trips
-D warnings dead-code. Scope #![allow(dead_code)] to the two files
that are only exercised by their own tests today, and allow the now
test-only UpdateOnlyQuantizedChunkedStorageBuilder re-export.
* [UpdateOnly] fix ast-grep: use expect(dead_code) instead of allow
* fix CI: remove unused EncodedStorageWrite import in gpu vector storage
Left over from splitting EncodedStorage into EncodedStorageWrite +
EncodedStorage; only caught under --all-features since gpu is gated
behind a feature flag.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
`EncodedVectorsTQ::layout()` declared `align_of::<f32>()`, but the size it
reports is the packed dimensions plus a 4-byte-multiple extras trailer, which
is not a multiple of 4 for three quarters of all dimensions (e.g. dim=756,
Bits4: 378 + 4 = 382 bytes).
The claim was never true — the encoded storage packs vectors at
`id * quantized_vector_size` with no per-vector padding — and nothing relies on
it: packed dimensions are read through unaligned SIMD loads (`loadu` / `vld1`)
and the extras trailer through `f32::from_le_bytes` on a byte slice.
It is also actively harmful. Inline HNSW storage packs link vectors
back-to-back using this layout and rejects one whose size is not a multiple of
its alignment, so building an index with `inline_storage` enabled fails for
those dimensions — and retries forever as an optimization crashloop.
Use `align_of::<u8>()`, matching scalar and product quantization. Old links
files stay readable: both layouts are persisted in the file header and the
reader takes size and alignment from there, never from the live quantizer.
Add a test covering the `size % align == 0` invariant across awkward
dimensions, bit widths, distances and modes — `layout()` had no coverage, which
is why the mismatch went unnoticed on the multiple-of-32 dimensions everyone
uses in practice.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* make `EncodedStorage::for_each_batch` mandatory
* make `DenseVectorStorageRead::for_each_in_dense_batch` mandatory
* make `DenseTQVectorStorage::for_each_in_dense_batch` mandatory
* make `DenseTQVectorStorage::read_dense_tq_bytes` mandatory
* make `QueryScorer::score_stored_batch` mandatory
...and implement for tq multivectors
* [AI] make `IdTrackerRead::internal_versions_batch` mandatory
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* [AI] make `IdTrackerRead::external_ids_batch` mandatory
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* [AI] make `DiskMappingsSource::resolve_internal_batch` mandatory
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Fix use-after-free in u8-quantized vector reads on owning storages
EncodedVectorsU8::get_vec_ptr extracted a raw pointer from the buffer
returned by EncodedStorage::get_vector_data and dropped the buffer
before the pointer was dereferenced. For storages returning
Cow::Borrowed (mmap) this happened to be sound, but for storages that
return Cow::Owned (disk-cache misses, uring backends) every user of
get_vec_ptr read freed memory: the internal SIMD scorers,
encode_internal_vector, and get_quantized_vector_offset_and_code, which
exported the dangling buffer through a safe &[u8].
Make parse_vec_data return the code as a slice borrowing the input, so
the borrow checker forces callers to keep the storage buffer alive
while reading it, and drop get_vec_ptr entirely. This matches how
encoded_vectors_binary already binds the buffers at its scoring sites.
Change get_quantized_vector_offset_and_code to return the code as a
sub-view of the buffer Cow itself, and adapt its GPU caller.
The new regression test drives these paths through a storage that
always returns owned buffers; under Miri it reproduces the
use-after-free on the previous code and passes with this fix.
Fixes#9799
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* additional assertion
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* quantization over tq strorage
are you happy fmt
are you happy clippy
rotation refactor
rotation refactor
fix tests
better test name
review remarks
dont rotate query in raw scorer
tq sources revert
quantized_scoring_datatype revert
simplify
are you happy fmt
* remove old comments
* review remarks
* feat: add read-only QuantizedVectors generic over UniversalRead
Introduce `QuantizedVectorsRead<S>` / `QuantizedVectorStorageRead<S>`, a
read-only counterpart of `QuantizedVectors` organized like
`VectorStorageReadEnum`: generic over the `UniversalRead` backend `S`,
opened from existing on-disk data, with no create/upsert/builder path and
no disk writes.
Highlights:
- Keep both in-RAM (`*Ram`) and read-only mmap (`*Mmap`) variants; drop the
appendable `*ChunkedMmap` variants (the only mutable ones).
- All bulk reads go through `S`: add `QuantizedRamStorage::from_universal_read`
and `MultivectorOffsetsStorageRam::open`, and make
`MultivectorOffsetsStorageMmap` generic over `S` (default `MmapFile`).
- Share scorer construction between the read-write and read-only enums via a
`QuantizedScorerDispatch` trait, so only the per-variant match is duplicated
while the datatype/distance and per-query dispatch live once in the builder.
Tested: read-only vs read-write scorer parity (scalar/binary/product, single
and multivector, RAM and mmap), covering both `raw_scorer` and
`raw_internal_scorer`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: route quantized RAM loading through UniversalRead
Follow-up to the read-only quantized work, removing direct-filesystem reads
and load-path duplication:
- `EncodedVectors{U8,PQ,Bin,TQ}::load` now take a `UniversalRead` filesystem
and read metadata via `read_json_via` instead of `fs::read_to_string` —
every read flows through universal IO. Single-vector `load` returns
`common::universal_io::Result`; `validate_storage_vector_size` stays
`std::io::Result`.
- Add `common::universal_io::OneshotFile<S>`: a thin RAII wrapper over any
`UniversalRead` handle that evicts the data from cache via `clear_ram_cache`
on drop (the universal-IO counterpart of `fs::OneshotFile`).
- Collapse `QuantizedRamStorage::{from_file, from_universal_read}` into one
`from_file<S: UniversalRead>`. It reads the whole file in a single access
(no separate `len()` round-trip — cheaper on S3-like backends) and loads via
the new `VolatileChunkedVectors::extend`, which inserts one chunk per
`copy_from_slice` instead of one vector at a time.
- RW callers pass the local `READ_FS` (mmap) backend; the read-only loader
passes its `S`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: load appendable (chunked) quantization format read-only
`storage_type` only selects the on-disk layout, so the read-only quantized
storage must be able to load both — the immutable flat format and the
appendable chunked format (produced only by Binary/TurboQuant). Previously the
read-only loader rejected the Mutable layout outright.
- Add `QuantizedChunkedStorageRead<S>` and `MultivectorOffsetsStorageChunkedRead<S>`,
read-only wrappers over the existing `ChunkedVectorsRead<_, S>` primitive
(mirrors the dense read-view's chunked read storage). Generic over the
`UniversalRead` backend, on-disk, no write path.
- Add `BinaryChunked`/`TQChunked` (+ multi) variants to `QuantizedVectorStorageRead`
and wire them through every accessor and the scorer dispatch.
- Route `storage_type == Mutable` to the chunked read variants in the loader and
drop the blanket rejection.
- Tests: parametrize the read-only/read-write parity tests over `storage_type`
and add chunked (Mutable) cases for binary/turbo, single and multivector.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: split quantized chunked & multivector storage into modules
`quantized_chunked_mmap_storage.rs` and `quantized_multivector_storage.rs` had
grown to hold several unrelated structures, making them hard to navigate.
Convert each into a module (pure code movement, no behavior change):
- quantized_chunked_mmap_storage/{read_write,read_only}.rs — the appendable
mmap storage + builder vs. the read-only chunked storage.
- quantized_multivector_storage/{mod,offsets}.rs — the core
`QuantizedMultivectorStorage` + offset traits stay in mod.rs; the four
`MultivectorOffsetsStorage*` backends move to offsets.rs.
Public paths are unchanged via re-exports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: single source of truth for quantized vector size
The on-disk stride (bytes per quantized vector), including the binary
`u8`(multi)/`u128`(single) word-type choice, was independently re-derived in
every load/open site with no compile-time link — the kind of value-level
duplication that silently diverges (it already caused the binary multi
`u8`/`u128` bug).
Extract it into one place:
- `QuantizedVectors::quantized_vector_size(quantization_config, vector_parameters, is_multi)`
and the `QuantizedVectorsConfig::quantized_vector_size(is_multi)` convenience.
Route every reader through it:
- read-only `open_single`/`open_multi` and the read-write `{scalar,pq,binary,turbo}`
loaders now hoist `config.quantized_vector_size(is_multi)` once instead of
recomputing the per-method formula per branch.
The create (write) path keeps its own computation for now; it stays guarded by
`validate_storage_vector_size` and the read-only/read-write parity tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: flat-match quantized loaders on a shared QuantizedStorageKind
The loaders chose the concrete storage variant via a nested
`method × (storage_type / is_ram)` match, re-derived independently in the
read-only and read-write paths — hard to follow and easy to drift.
- Add `QuantizedStorageKind` + `QuantizedVectorsConfig::storage_kind(on_disk)`
(and `is_ram(on_disk)`): the single place the method × backend decision lives.
- Both loaders now compute the kind once and use a flat 10-arm match:
- read-only `open_single`/`open_multi`;
- read-write `load_single`/`load_multi` (consolidating the four per-method
`{scalar,pq,binary,turbo}/load.rs` files, which are removed).
Both matches are exhaustive over the same enum, so the read and read-write
variants can no longer fall out of sync without a compile error.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: genericize write-capable chunked quantized storages over Fs
Drop the hardcoded `MmapFile` specialization from `QuantizedChunkedStorage`,
`QuantizedChunkedStorageBuilder`, and `MultivectorOffsetsStorageChunked`. They
now expose the `Fs` backend (defaulting to `MmapFile`) and accept the fs handle
as a parameter, matching the read-only variants.
Introduce a single shared `ReadFile` type alias and `READ_FS` value handle in
the `quantized_vectors` module root, used by the create, load, and storage enum
paths so the local-file backend is named in one place.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: move is_in_ram_or_mmap match into UniversalKind
Deduplicate the identical kind-to-residency match in the read-only and
write-capable chunked quantized storages by adding
UniversalKind::is_in_ram_or_mmap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: adapt TurboVectorStorage to quantized rename + update_from move
Integration fix after rebasing onto dev's TurboQuant work:
- QuantizedChunkedMmapStorage -> QuantizedChunkedStorage<MmapFile>
- update_from moved off the VectorStorage trait onto an inherent method,
matching the per-kind sub-trait refactor; TurboVectorStorage implements
neither DenseVectorStorage<T> nor the other kind traits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add TQ+ ErrorCorrection on top of renorm
Per-coordinate shift+scale fits each rotated, length-rescaled coord onto
the codebook's N(0, 1) grid before quantization. EncodedVectorsTQ::encode
runs a first pass to fit the stats when TQMode::Plus.
Scoring stays correct under renorm's `scaling_factor` framework:
- Asymmetric: precompute_query scales `Q .* D'` and stashes `qm = ⟨Q, M⟩`
on EncodedQueryTQ; score_precomputed adds qm to raw_dot before applying
scaling_factor.
- Symmetric: scalar slow path computes `Σ X+_a X+_b D'_i² + xm_a + xm_b
− ⟨M, M⟩` (xm stored per vector in extras, mm_const cached on
ErrorCorrection). Result feeds the existing `* v1_scale * v2_scale` arms.
SIMD reuse for this path is a follow-up.
Storage layout: TQMode::Plus extras are 4 bytes longer (xm appended after
scaling_factor). Zero-vector inputs skip EC application so renorm's
existing zero-norm guard keeps producing score ≈ 0 within tolerance.
VectorStats refactor: streaming `VectorStatsBuilder` so the Plus first
pass can feed Welford with a reused buffer; `build` now takes `dim`
directly and is generic over `T: Into<f64>`.
Integration tests run on both Normal and Plus via rstest cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix TQ+ recall regression on non-uniform per-coord variance data
Two compounding bugs in the renorm + TQ+ composition:
1. centroid_norm was measured on `X+` centroids, which have chi-squared
norm distribution across vectors (~10% spread for d=256). renorm
assumed `cn` should be deterministic `sqrt(d)` (just quantization
drift), so the per-vector correction ended up amplifying intrinsic
chi-squared noise into ranking error. Fix: revert EC per coord before
measuring (`c · D' + M`), matching llama-turbo-quant. The reverted
centroids approximate `rescaled` which has length `sqrt(d)` exactly
by construction. dequantize follows the same convention so the
stored `scaling_factor = l2/cn` round-trips back to the original l2.
2. Asymmetric query path pre-scaled `Q* = R_q · D'` before SIMD encoding.
The SIMD encoder normalizes by `max(|input|)`, so a query whose coords
span 5× magnitude (which `R_q · D'` does on real data) loses precision
on the small-D' coords. Fix: keep `rotated` unscaled, store it as a
side field on `EncodedQueryTQ`, and use a scalar decode-and-dot path
for TQ+ (`Σ R_q_i · c_i · D'_i + qm`). SIMD support for this is a
follow-up.
Catches both via `recall_skewed_data` test on data with 8 spike-variance
input coords. Without the fixes, Bits4 Plus dropped to 0.93 vs Normal
0.98; after the fixes, Plus tracks Normal within 2%.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ asymmetric: skip the SIMD encoding entirely
The TQ+ asymmetric path doesn't use the SIMD-encoded query — it goes
through `score_precomputed_ec` with `rotated_query`. So building the
SIMD form was wasted work + memory. Make `data` an `Option` and only
populate it for the cases that actually use it (Normal mode any distance,
TQ+ L1 via dequantize fallback).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Revert TQ+ to SIMD scoring path
The whole point of TQ+ is that scoring code-paths stay identical between
Normal and Plus modes — only the query precomputation changes. We
pre-scale `Q* = R_q · D'` so the existing SIMD raw_dot computes
`⟨Q · D', X+⟩` directly, then add `qm` and apply renorm's scaling_factor.
Drops `score_precomputed_ec` and `EncodedQueryTQ::rotated_query`. The
recall regression that motivated the scalar fallback was entirely from
the `compute_centroid_norm` bug (measuring `‖X+‖` instead of `‖rescaled‖`)
fixed in the prior commit; SIMD precision was a red herring.
`recall_skewed_data` confirms: Bits4 Normal=0.984 / Plus=0.978, Bits2
Normal=0.902 / Plus=0.908.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ Bits1: widen query encoding to 12 bits
For 1-bit storage with TQ+, the per-coord `D' = 1/scale` pre-scaling on
the query can push some coords toward the small end of the SIMD encoder's
integer range (which normalizes by `max(|input|)`). At 8 bits those small
coords lose precision; at 12 bits the rounding error drops ~10× per the
existing `test_query_dotprod_matches_reference` parity test.
`Query1bitSimd` is already generic over BITS so this is just a new
`EncodedQueryTQData::Bits1Wide(Query1bitSimd<12>)` variant + a TQ+/Bits1
dispatch in `precompute_query`. Bits2/Bits4 don't need this — their
storage is fine-grained enough that query precision isn't the bottleneck,
and their SIMD encoders aren't generic over BITS today.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* are you happy fmt
* TQ+ Bits1Wide: bump to 16-bit query quantization (kernel max)
12 bits helped on real datasets but not enough — push to the kernel's
ceiling of 16. `Query1bitSimd<BITS>` asserts `BITS ∈ [2, 16]`, so this
is the most precision the existing SIMD path can give us before needing
a wider integer kernel.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+: shift by per-coord median, not mean
For 1-bit storage the codebook boundary sits at 0, so post-shift values
are quantized purely by sign. Median is the sign-balance point of the
distribution; mean isn't (skewed coords pull mean off the median).
On anisotropic embeddings — dbpedia-openai being the reference case —
mean-based shift produced a ~60/40 biased sign distribution per coord,
losing 1-bit's representational capacity. Median-based shift restores
50/50 and matches llama-turbo-quant's behavior. Higher bit-widths are
less sensitive but still benefit; the codebook boundaries still lie at
distribution-percentile-aware positions when the data is centered on
the median.
Median requires per-coord samples in memory, so cap the stats pass at
10K vectors. Estimates converge fast (~√N) — 10K is plenty even for
million-vector indexes. The encoding pass still processes every vector.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ Bits1Wide: revert to 12-bit query quantization
The recall regression on anisotropic data was the mean-vs-median shift,
not query precision. 12 bits is enough headroom for the per-coord D'
pre-scaling and avoids the extra storage of the 16-bit form.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TQ+ Bits1Wide: bump back to 16-bit query quantization
12 bits helped a bit but not enough on the real dataset. Bump to the
kernel's ceiling. If 16 still isn't enough, the next step is checking
whether the gap is real (re-measure llama branch) before widening the
SIMD integer kernel itself.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* are you happy clippy
* use mean
* 1bit error correction
* review remarks
* are you happy clippy
* review remarks
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* use vector statistics for scalar bq query
* fix minor error
* remove test_binary_scalar_internal test
* do NOT use special file for storing vector stats
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* # This is a combination of 7 commits.
* SameAsStorage default value
* fix coderabbit warnings
* neon for u8 bq
* sse for u8 bq
* fix windows build
* rename function
* add comments
* fmt
* fix arm build
* review remarks
* bq encodings
* are you happy clippy
* are you happy clippy
* are you happy clippy
* are you happy clippy
* gpu tests
* update models
* are you happy fmt
* move additional bits to the end
* fix tests
* Welford's Algorithm
* review remarks
* are you happy clippy
* remove debug println in test
* coderabit nitpicks
* remove unnecessary clone and partialeq
* Use f64 for Welford's Algorithm
* try fix ci
* revert cargo-nextest
* add debug assertions
* Bump Rust edition to 2024
* gen is a reserved keyword now
* Remove ref mut on references
* Mark extern C as unsafe
* Wrap unsafe function bodies in unsafe block
* Geo hash implements Copy, don't reference but pass by value instead
* Replace secluded self import with parent
* Update execute_cluster_read_operation with new match semantics
* Fix lifetime issue
* Replace map_or with is_none_or
* set_var is unsafe now
* Reformat
* bump and migrate to rand 0.9.0
also bump rand_distr to 0.5.0 to match it
* Migrate AVX2 and SSE implementations
* Remove unused thread_rng placeholders
* More random migrations
* Migrate GPU tests
* bump seed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
qdrant-cloud-bot