mirror of
https://github.com/qdrant/qdrant.git
synced 2026-07-29 14:11:32 -05:00
80acb6c55d
* feat: add read-only QuantizedVectors generic over UniversalRead Introduce `QuantizedVectorsRead<S>` / `QuantizedVectorStorageRead<S>`, a read-only counterpart of `QuantizedVectors` organized like `VectorStorageReadEnum`: generic over the `UniversalRead` backend `S`, opened from existing on-disk data, with no create/upsert/builder path and no disk writes. Highlights: - Keep both in-RAM (`*Ram`) and read-only mmap (`*Mmap`) variants; drop the appendable `*ChunkedMmap` variants (the only mutable ones). - All bulk reads go through `S`: add `QuantizedRamStorage::from_universal_read` and `MultivectorOffsetsStorageRam::open`, and make `MultivectorOffsetsStorageMmap` generic over `S` (default `MmapFile`). - Share scorer construction between the read-write and read-only enums via a `QuantizedScorerDispatch` trait, so only the per-variant match is duplicated while the datatype/distance and per-query dispatch live once in the builder. Tested: read-only vs read-write scorer parity (scalar/binary/product, single and multivector, RAM and mmap), covering both `raw_scorer` and `raw_internal_scorer`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: route quantized RAM loading through UniversalRead Follow-up to the read-only quantized work, removing direct-filesystem reads and load-path duplication: - `EncodedVectors{U8,PQ,Bin,TQ}::load` now take a `UniversalRead` filesystem and read metadata via `read_json_via` instead of `fs::read_to_string` — every read flows through universal IO. Single-vector `load` returns `common::universal_io::Result`; `validate_storage_vector_size` stays `std::io::Result`. - Add `common::universal_io::OneshotFile<S>`: a thin RAII wrapper over any `UniversalRead` handle that evicts the data from cache via `clear_ram_cache` on drop (the universal-IO counterpart of `fs::OneshotFile`). - Collapse `QuantizedRamStorage::{from_file, from_universal_read}` into one `from_file<S: UniversalRead>`. It reads the whole file in a single access (no separate `len()` round-trip — cheaper on S3-like backends) and loads via the new `VolatileChunkedVectors::extend`, which inserts one chunk per `copy_from_slice` instead of one vector at a time. - RW callers pass the local `READ_FS` (mmap) backend; the read-only loader passes its `S`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: load appendable (chunked) quantization format read-only `storage_type` only selects the on-disk layout, so the read-only quantized storage must be able to load both — the immutable flat format and the appendable chunked format (produced only by Binary/TurboQuant). Previously the read-only loader rejected the Mutable layout outright. - Add `QuantizedChunkedStorageRead<S>` and `MultivectorOffsetsStorageChunkedRead<S>`, read-only wrappers over the existing `ChunkedVectorsRead<_, S>` primitive (mirrors the dense read-view's chunked read storage). Generic over the `UniversalRead` backend, on-disk, no write path. - Add `BinaryChunked`/`TQChunked` (+ multi) variants to `QuantizedVectorStorageRead` and wire them through every accessor and the scorer dispatch. - Route `storage_type == Mutable` to the chunked read variants in the loader and drop the blanket rejection. - Tests: parametrize the read-only/read-write parity tests over `storage_type` and add chunked (Mutable) cases for binary/turbo, single and multivector. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: split quantized chunked & multivector storage into modules `quantized_chunked_mmap_storage.rs` and `quantized_multivector_storage.rs` had grown to hold several unrelated structures, making them hard to navigate. Convert each into a module (pure code movement, no behavior change): - quantized_chunked_mmap_storage/{read_write,read_only}.rs — the appendable mmap storage + builder vs. the read-only chunked storage. - quantized_multivector_storage/{mod,offsets}.rs — the core `QuantizedMultivectorStorage` + offset traits stay in mod.rs; the four `MultivectorOffsetsStorage*` backends move to offsets.rs. Public paths are unchanged via re-exports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: single source of truth for quantized vector size The on-disk stride (bytes per quantized vector), including the binary `u8`(multi)/`u128`(single) word-type choice, was independently re-derived in every load/open site with no compile-time link — the kind of value-level duplication that silently diverges (it already caused the binary multi `u8`/`u128` bug). Extract it into one place: - `QuantizedVectors::quantized_vector_size(quantization_config, vector_parameters, is_multi)` and the `QuantizedVectorsConfig::quantized_vector_size(is_multi)` convenience. Route every reader through it: - read-only `open_single`/`open_multi` and the read-write `{scalar,pq,binary,turbo}` loaders now hoist `config.quantized_vector_size(is_multi)` once instead of recomputing the per-method formula per branch. The create (write) path keeps its own computation for now; it stays guarded by `validate_storage_vector_size` and the read-only/read-write parity tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: flat-match quantized loaders on a shared QuantizedStorageKind The loaders chose the concrete storage variant via a nested `method × (storage_type / is_ram)` match, re-derived independently in the read-only and read-write paths — hard to follow and easy to drift. - Add `QuantizedStorageKind` + `QuantizedVectorsConfig::storage_kind(on_disk)` (and `is_ram(on_disk)`): the single place the method × backend decision lives. - Both loaders now compute the kind once and use a flat 10-arm match: - read-only `open_single`/`open_multi`; - read-write `load_single`/`load_multi` (consolidating the four per-method `{scalar,pq,binary,turbo}/load.rs` files, which are removed). Both matches are exhaustive over the same enum, so the read and read-write variants can no longer fall out of sync without a compile error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: genericize write-capable chunked quantized storages over Fs Drop the hardcoded `MmapFile` specialization from `QuantizedChunkedStorage`, `QuantizedChunkedStorageBuilder`, and `MultivectorOffsetsStorageChunked`. They now expose the `Fs` backend (defaulting to `MmapFile`) and accept the fs handle as a parameter, matching the read-only variants. Introduce a single shared `ReadFile` type alias and `READ_FS` value handle in the `quantized_vectors` module root, used by the create, load, and storage enum paths so the local-file backend is named in one place. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: move is_in_ram_or_mmap match into UniversalKind Deduplicate the identical kind-to-residency match in the read-only and write-capable chunked quantized storages by adding UniversalKind::is_in_ram_or_mmap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: adapt TurboVectorStorage to quantized rename + update_from move Integration fix after rebasing onto dev's TurboQuant work: - QuantizedChunkedMmapStorage -> QuantizedChunkedStorage<MmapFile> - update_from moved off the VectorStorage trait onto an inherent method, matching the per-kind sub-trait refactor; TurboVectorStorage implements neither DenseVectorStorage<T> nor the other kind traits. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>