Commit Graph

47 Commits

Author SHA1 Message Date
Daniel Boros
c79ca1bdd1 feat: live_reload for read-only quantized vectors (#9375)
* feat: live_reload for read-only quantized vectors

* Propagate live_reload params through quantized chunked storage

Thread fs, deleted_points, new_points and hw_counter through the full
quantized live_reload chain instead of synthesizing empty deltas and a
disposable hardware counter at the leaf storages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: use LiveReload trait in chunked mmap

* fix: use LiveReload trait

* fix: compiler errors

---------

Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 16:58:02 +02:00
Andrey Vasnetsov
80acb6c55d feat: read-only QuantizedVectors generic over UniversalRead (#9346)
* feat: add read-only QuantizedVectors generic over UniversalRead

Introduce `QuantizedVectorsRead<S>` / `QuantizedVectorStorageRead<S>`, a
read-only counterpart of `QuantizedVectors` organized like
`VectorStorageReadEnum`: generic over the `UniversalRead` backend `S`,
opened from existing on-disk data, with no create/upsert/builder path and
no disk writes.

Highlights:
- Keep both in-RAM (`*Ram`) and read-only mmap (`*Mmap`) variants; drop the
  appendable `*ChunkedMmap` variants (the only mutable ones).
- All bulk reads go through `S`: add `QuantizedRamStorage::from_universal_read`
  and `MultivectorOffsetsStorageRam::open`, and make
  `MultivectorOffsetsStorageMmap` generic over `S` (default `MmapFile`).
- Share scorer construction between the read-write and read-only enums via a
  `QuantizedScorerDispatch` trait, so only the per-variant match is duplicated
  while the datatype/distance and per-query dispatch live once in the builder.

Tested: read-only vs read-write scorer parity (scalar/binary/product, single
and multivector, RAM and mmap), covering both `raw_scorer` and
`raw_internal_scorer`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor: route quantized RAM loading through UniversalRead

Follow-up to the read-only quantized work, removing direct-filesystem reads
and load-path duplication:

- `EncodedVectors{U8,PQ,Bin,TQ}::load` now take a `UniversalRead` filesystem
  and read metadata via `read_json_via` instead of `fs::read_to_string` —
  every read flows through universal IO. Single-vector `load` returns
  `common::universal_io::Result`; `validate_storage_vector_size` stays
  `std::io::Result`.
- Add `common::universal_io::OneshotFile<S>`: a thin RAII wrapper over any
  `UniversalRead` handle that evicts the data from cache via `clear_ram_cache`
  on drop (the universal-IO counterpart of `fs::OneshotFile`).
- Collapse `QuantizedRamStorage::{from_file, from_universal_read}` into one
  `from_file<S: UniversalRead>`. It reads the whole file in a single access
  (no separate `len()` round-trip — cheaper on S3-like backends) and loads via
  the new `VolatileChunkedVectors::extend`, which inserts one chunk per
  `copy_from_slice` instead of one vector at a time.
- RW callers pass the local `READ_FS` (mmap) backend; the read-only loader
  passes its `S`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: load appendable (chunked) quantization format read-only

`storage_type` only selects the on-disk layout, so the read-only quantized
storage must be able to load both — the immutable flat format and the
appendable chunked format (produced only by Binary/TurboQuant). Previously the
read-only loader rejected the Mutable layout outright.

- Add `QuantizedChunkedStorageRead<S>` and `MultivectorOffsetsStorageChunkedRead<S>`,
  read-only wrappers over the existing `ChunkedVectorsRead<_, S>` primitive
  (mirrors the dense read-view's chunked read storage). Generic over the
  `UniversalRead` backend, on-disk, no write path.
- Add `BinaryChunked`/`TQChunked` (+ multi) variants to `QuantizedVectorStorageRead`
  and wire them through every accessor and the scorer dispatch.
- Route `storage_type == Mutable` to the chunked read variants in the loader and
  drop the blanket rejection.
- Tests: parametrize the read-only/read-write parity tests over `storage_type`
  and add chunked (Mutable) cases for binary/turbo, single and multivector.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor: split quantized chunked & multivector storage into modules

`quantized_chunked_mmap_storage.rs` and `quantized_multivector_storage.rs` had
grown to hold several unrelated structures, making them hard to navigate.
Convert each into a module (pure code movement, no behavior change):

- quantized_chunked_mmap_storage/{read_write,read_only}.rs — the appendable
  mmap storage + builder vs. the read-only chunked storage.
- quantized_multivector_storage/{mod,offsets}.rs — the core
  `QuantizedMultivectorStorage` + offset traits stay in mod.rs; the four
  `MultivectorOffsetsStorage*` backends move to offsets.rs.

Public paths are unchanged via re-exports.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor: single source of truth for quantized vector size

The on-disk stride (bytes per quantized vector), including the binary
`u8`(multi)/`u128`(single) word-type choice, was independently re-derived in
every load/open site with no compile-time link — the kind of value-level
duplication that silently diverges (it already caused the binary multi
`u8`/`u128` bug).

Extract it into one place:
- `QuantizedVectors::quantized_vector_size(quantization_config, vector_parameters, is_multi)`
  and the `QuantizedVectorsConfig::quantized_vector_size(is_multi)` convenience.

Route every reader through it:
- read-only `open_single`/`open_multi` and the read-write `{scalar,pq,binary,turbo}`
  loaders now hoist `config.quantized_vector_size(is_multi)` once instead of
  recomputing the per-method formula per branch.

The create (write) path keeps its own computation for now; it stays guarded by
`validate_storage_vector_size` and the read-only/read-write parity tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor: flat-match quantized loaders on a shared QuantizedStorageKind

The loaders chose the concrete storage variant via a nested
`method × (storage_type / is_ram)` match, re-derived independently in the
read-only and read-write paths — hard to follow and easy to drift.

- Add `QuantizedStorageKind` + `QuantizedVectorsConfig::storage_kind(on_disk)`
  (and `is_ram(on_disk)`): the single place the method × backend decision lives.
- Both loaders now compute the kind once and use a flat 10-arm match:
  - read-only `open_single`/`open_multi`;
  - read-write `load_single`/`load_multi` (consolidating the four per-method
    `{scalar,pq,binary,turbo}/load.rs` files, which are removed).

Both matches are exhaustive over the same enum, so the read and read-write
variants can no longer fall out of sync without a compile error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor: genericize write-capable chunked quantized storages over Fs

Drop the hardcoded `MmapFile` specialization from `QuantizedChunkedStorage`,
`QuantizedChunkedStorageBuilder`, and `MultivectorOffsetsStorageChunked`. They
now expose the `Fs` backend (defaulting to `MmapFile`) and accept the fs handle
as a parameter, matching the read-only variants.

Introduce a single shared `ReadFile` type alias and `READ_FS` value handle in
the `quantized_vectors` module root, used by the create, load, and storage enum
paths so the local-file backend is named in one place.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor: move is_in_ram_or_mmap match into UniversalKind

Deduplicate the identical kind-to-residency match in the read-only and
write-capable chunked quantized storages by adding
UniversalKind::is_in_ram_or_mmap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: adapt TurboVectorStorage to quantized rename + update_from move

Integration fix after rebasing onto dev's TurboQuant work:
- QuantizedChunkedMmapStorage -> QuantizedChunkedStorage<MmapFile>
- update_from moved off the VectorStorage trait onto an inherent method,
  matching the per-kind sub-trait refactor; TurboVectorStorage implements
  neither DenseVectorStorage<T> nor the other kind traits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-08 12:56:57 +02:00
Tim Visée
4e54bbc94a Fix OOB heap read crash with malicious snapshot (#9268)
* Validate quantized u8 data size on load

* Validate other quantization types

* Add test

* Use fs_err
2026-06-03 15:17:04 +02:00
Arnaud Gourlay
abc53d717f Remove unecesssary Clippy allows (#9267) 2026-06-02 16:54:27 +02:00
Roman Titov
65dda5ff45 Merge pull request #9034
* Add `iter_batch` method to `EncodedStorage` and `EncodedVectors`

* Remove `EncodedVectors::for_each_in_batch` method

* Add `for_each_in_multi_batch` and `score_vector_max_similarity` metho…

* Refactor `score_stored_batch` for quantized multi-vector scorers...

* Remove `QuantizedMultivectorStorage::score_multi`

* Implement `score_points_batch_mmap` and `score_points_batch_uring`...

* Implement runtime routing between mmap and io_uring batch scoring met…

* review: rename + comments
2026-05-25 17:00:47 +02:00
Luis Cossío
757a168167 make quantized_vector_size method static (#9060) 2026-05-15 18:31:23 -04:00
Roman Titov
2a47997654 Merge pull request #8791
* Add `EncodedStorage::for_each_in_batch` method

* Implement `for_each_in_batch` method for `QuantizedChunkedMmapStorage`

* Add `EncodedVectors::for_each_in_batch` method

* Add `EncodedVectors::score` method

* Implement `score_stored_batch` for `QuantizedQueryScorer` and `Quanti…

* Remove `TElement` and `TMetric` type parameters from `QuantizedMultiQ…

* Use `QuantizedMultiQueryScorer` when building `raw_internal_scorer`...
2026-05-07 20:20:54 +02:00
Ivan Pleshkov
eb2e3b8416 Tq plus (#8836)
* Add TQ+ ErrorCorrection on top of renorm

Per-coordinate shift+scale fits each rotated, length-rescaled coord onto
the codebook's N(0, 1) grid before quantization. EncodedVectorsTQ::encode
runs a first pass to fit the stats when TQMode::Plus.

Scoring stays correct under renorm's `scaling_factor` framework:
- Asymmetric: precompute_query scales `Q .* D'` and stashes `qm = ⟨Q, M⟩`
  on EncodedQueryTQ; score_precomputed adds qm to raw_dot before applying
  scaling_factor.
- Symmetric: scalar slow path computes `Σ X+_a X+_b D'_i² + xm_a + xm_b
  − ⟨M, M⟩` (xm stored per vector in extras, mm_const cached on
  ErrorCorrection). Result feeds the existing `* v1_scale * v2_scale` arms.
  SIMD reuse for this path is a follow-up.

Storage layout: TQMode::Plus extras are 4 bytes longer (xm appended after
scaling_factor). Zero-vector inputs skip EC application so renorm's
existing zero-norm guard keeps producing score ≈ 0 within tolerance.

VectorStats refactor: streaming `VectorStatsBuilder` so the Plus first
pass can feed Welford with a reused buffer; `build` now takes `dim`
directly and is generic over `T: Into<f64>`.

Integration tests run on both Normal and Plus via rstest cases.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Fix TQ+ recall regression on non-uniform per-coord variance data

Two compounding bugs in the renorm + TQ+ composition:

1. centroid_norm was measured on `X+` centroids, which have chi-squared
   norm distribution across vectors (~10% spread for d=256). renorm
   assumed `cn` should be deterministic `sqrt(d)` (just quantization
   drift), so the per-vector correction ended up amplifying intrinsic
   chi-squared noise into ranking error. Fix: revert EC per coord before
   measuring (`c · D' + M`), matching llama-turbo-quant. The reverted
   centroids approximate `rescaled` which has length `sqrt(d)` exactly
   by construction. dequantize follows the same convention so the
   stored `scaling_factor = l2/cn` round-trips back to the original l2.

2. Asymmetric query path pre-scaled `Q* = R_q · D'` before SIMD encoding.
   The SIMD encoder normalizes by `max(|input|)`, so a query whose coords
   span 5× magnitude (which `R_q · D'` does on real data) loses precision
   on the small-D' coords. Fix: keep `rotated` unscaled, store it as a
   side field on `EncodedQueryTQ`, and use a scalar decode-and-dot path
   for TQ+ (`Σ R_q_i · c_i · D'_i + qm`). SIMD support for this is a
   follow-up.

Catches both via `recall_skewed_data` test on data with 8 spike-variance
input coords. Without the fixes, Bits4 Plus dropped to 0.93 vs Normal
0.98; after the fixes, Plus tracks Normal within 2%.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* TQ+ asymmetric: skip the SIMD encoding entirely

The TQ+ asymmetric path doesn't use the SIMD-encoded query — it goes
through `score_precomputed_ec` with `rotated_query`. So building the
SIMD form was wasted work + memory. Make `data` an `Option` and only
populate it for the cases that actually use it (Normal mode any distance,
TQ+ L1 via dequantize fallback).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Revert TQ+ to SIMD scoring path

The whole point of TQ+ is that scoring code-paths stay identical between
Normal and Plus modes — only the query precomputation changes. We
pre-scale `Q* = R_q · D'` so the existing SIMD raw_dot computes
`⟨Q · D', X+⟩` directly, then add `qm` and apply renorm's scaling_factor.

Drops `score_precomputed_ec` and `EncodedQueryTQ::rotated_query`. The
recall regression that motivated the scalar fallback was entirely from
the `compute_centroid_norm` bug (measuring `‖X+‖` instead of `‖rescaled‖`)
fixed in the prior commit; SIMD precision was a red herring.

`recall_skewed_data` confirms: Bits4 Normal=0.984 / Plus=0.978, Bits2
Normal=0.902 / Plus=0.908.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* TQ+ Bits1: widen query encoding to 12 bits

For 1-bit storage with TQ+, the per-coord `D' = 1/scale` pre-scaling on
the query can push some coords toward the small end of the SIMD encoder's
integer range (which normalizes by `max(|input|)`). At 8 bits those small
coords lose precision; at 12 bits the rounding error drops ~10× per the
existing `test_query_dotprod_matches_reference` parity test.

`Query1bitSimd` is already generic over BITS so this is just a new
`EncodedQueryTQData::Bits1Wide(Query1bitSimd<12>)` variant + a TQ+/Bits1
dispatch in `precompute_query`. Bits2/Bits4 don't need this — their
storage is fine-grained enough that query precision isn't the bottleneck,
and their SIMD encoders aren't generic over BITS today.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* are you happy fmt

* TQ+ Bits1Wide: bump to 16-bit query quantization (kernel max)

12 bits helped on real datasets but not enough — push to the kernel's
ceiling of 16. `Query1bitSimd<BITS>` asserts `BITS ∈ [2, 16]`, so this
is the most precision the existing SIMD path can give us before needing
a wider integer kernel.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* TQ+: shift by per-coord median, not mean

For 1-bit storage the codebook boundary sits at 0, so post-shift values
are quantized purely by sign. Median is the sign-balance point of the
distribution; mean isn't (skewed coords pull mean off the median).

On anisotropic embeddings — dbpedia-openai being the reference case —
mean-based shift produced a ~60/40 biased sign distribution per coord,
losing 1-bit's representational capacity. Median-based shift restores
50/50 and matches llama-turbo-quant's behavior. Higher bit-widths are
less sensitive but still benefit; the codebook boundaries still lie at
distribution-percentile-aware positions when the data is centered on
the median.

Median requires per-coord samples in memory, so cap the stats pass at
10K vectors. Estimates converge fast (~√N) — 10K is plenty even for
million-vector indexes. The encoding pass still processes every vector.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* TQ+ Bits1Wide: revert to 12-bit query quantization

The recall regression on anisotropic data was the mean-vs-median shift,
not query precision. 12 bits is enough headroom for the per-coord D'
pre-scaling and avoids the extra storage of the 16-bit form.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* TQ+ Bits1Wide: bump back to 16-bit query quantization

12 bits helped a bit but not enough on the real dataset. Bump to the
kernel's ceiling. If 16 still isn't enough, the next step is checking
whether the gap is real (re-measure llama branch) before widening the
SIMD integer kernel itself.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* are you happy clippy

* use mean

* 1bit error correction

* review remarks

* are you happy clippy

* review remarks

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 13:39:05 +02:00
Arnaud Gourlay
354bbb35e6 Delete unused code (#8771)
* Delete unused code

* restore initialize_global

* drop BadShardSelection

* Remove now obsolete allow(dead_code) attributes

* Remove more dead code

---------

Co-authored-by: timvisee <tim@visee.me>
2026-04-24 11:44:21 +02:00
Jojii
4f35b72f0a TQ Scoring (#8753)
* [AI + Manual] Cosine scoring

* [ai + manual] Add support for dot product

* [ai] re-implement tests including covering both distances (dot+cosine)

* Clarify Cosine/Dot equivalence behavior for pre-TQ quantization storages

* review remarks

---------

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
2026-04-23 12:01:53 +02:00
Andrey Vasnetsov
5a899b74de deep memory reporting (#8606)
* Add mincore-based memory stats to MmapFile

Add `resident_bytes()`, `disk_bytes()`, and `probe_memory_stats()` methods
to `MmapFile` for measuring page cache residency via `mincore(2)`. This is
the foundation for per-collection memory usage reporting.

Also extract `page_size()` as a public function in `mmap::advice`, replacing
the internal `PAGE_SIZE_MASK` with a direct page size cache.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [AI] introduce trait for reporting memory usage per component

* [AI] memory reporter implementation for vector storage

* [AI] implement MemoryReporter for QuantizedVectors

* [AI] implement MemoryReporter for VectorIndexEnum

* Implement MemoryReporter for IdTrackerEnum with RAM estimation

Add ram_usage_bytes() to all ID tracker types and their data structures:
- PointMappings, CompressedPointMappings, CompressedVersions,
  CompressedInternalToExternal, CompressedExternalToInternal
- MutableIdTracker, ImmutableIdTracker, InMemoryIdTracker

All ID trackers load their data into RAM (none use mmap for working data).
Files are reported as OnDisk (persistence only), actual RAM footprint
is reported via extra_ram_bytes. Uses struct destructuring to ensure
new fields trigger compile errors if not accounted for.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [AI] implement MemoryReporter for PayloadStorageEnum and adjust FileStorageIntent

* [AI] implement MemoryReporter for PayloadStorageEnum and adjust FileStorageIntent

* [AI] implement MemoryReporter for payload indexes: in-ram structures memory consumtion computation + caching

* [AI] implement MemoryReporter for payload indexes: in-ram structures memory consumtion computation + caching

* [AI] segment-level memory usage report

* [AI] Block 3: Aggregation Layer and Data Model + internal api for remote shard

* [AI] REST API handler

* fmt

* [AI] clippy fixes

* [AI] macos fix + proxy segment fix

* [AI] make text index estimation a bit more correct

* fix is_on_disk reporting for dense_vector_storage

* fix after rebase

* [AI] deep account for quantized vectors RAM usage + unify chunk size + shring volatile storage after load

* remove debug log

* cache in test

* make manual test easier to run

* rollback chunk size diff, but keep it for test only

* review fixes

* Use exhaustive match

* Use div_ceil on bits everywhere

It does not seem to be strictly necessary because the number of bits
should already be a multiple of the used container size bytes. Still
it's good practice to be careful with this calculation.

* Improve heap size bytes for encoded product quantization vectors

* Include vector stats for binary quantized vectors

* In volatile chunked vectors, include heap allocated vector

* Include rest of heap allocated structures for mutable map index

* In mutable geo index, the hash map is also heap allocated

* Update tests/manual/test_memory_reporting.py

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2026-04-14 12:37:31 +02:00
Luis Cossío
5403ea68ab Chunked vectors with UniversalWrite storage (#8233)
* use CowMultiVector as return type from storages

* add advice to OpenOptions

* Implement ChunkedVectors with generic storage

* rename ChunkedVectors->VolatileChunkedVectors and ChunkedMmapVectors-> ChunkedVectors

* propagate everywhere

fix tests

* [auto] rename BytesRange -> ElementsRange

* [auto] rename BytesOffset -> ElementOffset

* coderabbit nits

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2026-02-27 12:47:15 -03:00
xzfc
4cabb7fd8e Merge io and memory into common (#8155)
* Unify parking_lot/arc_lock feature

* Move lib/common/{io,memory}/* -> lib/common/common/*

- Mmap-related items are grouped into `common::mmap` sub-module:
  - `memory/src/chunked_utils.rs`      -> `common/src/mmap/chunked.rs`
  - `memory/src/madvise.rs`            -> `common/src/mmap/advice.rs`
  - `memory/src/mmap_ops.rs`           -> `common/src/mmap/ops.rs`
  - `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
  - `memory/src/mmap_type.rs`          -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
  - `common/src/fs.rs`          -> `common/src/fs/sync.rs`
  - `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
  - `io/src/move_files.rs`      -> `common/src/fs/move.rs`
  - `io/src/safe_delete.rs`     -> `common/src/fs/safe_delete.rs`
  - `memory/src/checkfs.rs`     -> `common/src/fs/check.rs`
  - `memory/src/fadvise.rs`     -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
  - `io/src/storage_version.rs` -> `common/src/storage_version.rs`

The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.

* Replace uses of `io` and `memory` with new paths in `common`

Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.

* Remove `io` and `memory` crates
2026-02-17 10:58:59 +01:00
Ivan Boldyrev
984a9e8bd1 Fix gridstore Option unsoundness (#8011)
* Implement `Optional<T>` type

Define a type with the same presumed layout as `Option<T>`, but with defined behavior.

* Make `transmute_*` functions unsafe

The functions `memory::mmap_ops::transmute_*` are inherently unsafe, but
are not marked as are.  Their usage is documented, but it is not always clear
if the code is correct.

* Add `CsrHeader` to resolve another unsoundness

Tuples have no defined layout.
2026-02-03 17:57:09 +07:00
Ivan Pleshkov
793b261016 avx512 for binary quantization (#7052)
* avx512 for binary quantization

fix build

* change checks order
2025-10-29 13:26:55 +01:00
xzfc
a0d62330c7 Use fs-err (#7319) 2025-09-29 12:47:10 +00:00
Arnaud Gourlay
360d7bc26d Clippy 1.90 (#7253)
* Fix Clippy 1.90

* fmt
2025-09-15 20:12:11 +02:00
Ivan Pleshkov
2d1b92c16b Create and load for an appendable quantization (#7193)
* Create and load for an appendable quantization

remove feature flag

more todo

review remarks

review remarks

* fix after rebase

* Rename is_appendable to supports_appendable

---------

Co-authored-by: timvisee <tim@visee.me>
2025-09-01 15:49:09 +02:00
Ivan Pleshkov
b03853c33e Quantization storage builder for chunked mmap (#7174)
* Quantization storage builder for chunked mmap

* fix after rebase

* Update lib/segment/src/vector_storage/quantized/quantized_chunked_mmap_storage.rs

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
2025-08-29 16:10:30 +02:00
Ivan Pleshkov
879da486af Remove quantization load from trait (#7113)
* remove quantization load

* refactor load

* fix fmt error

* fix clippy
2025-08-22 14:37:32 +02:00
Ivan Pleshkov
f3b61bbf4a Retrieve files list from quantization storage (#7095)
* Get files list from quantization

* fix ci
2025-08-20 14:15:09 +02:00
Ivan Pleshkov
3f75fae516 EncodedStorage upsert vector (#7048)
* EncodedStorage upsert vector
2025-08-18 17:39:23 +02:00
xzfc
acfdf02da8 Optional scorer methods via TBool (#6989)
* Add common::typelevel module

* Use common::typelevel for score_bytes

* Revert "Split FilteredScorer into FilteredScorer and FilteredQuantizedScorer"

This reverts commit ff3a93ee95.

* FilteredQuantizedScorer

* Extract `struct Filters` from `FilteredScorer` and `FilteredQuantizedScorer`

* Revert "QuantizedVectors::query_scorer_bytes"

This reverts commit 9dee824afd.

* Partial revert "Add QueryScorerBytes::score_bytes"

This reverts commit d80af4f698, but keeps
the QueryScorerBytes trait while removing implementations.

* Partial revert "Add EncodedVectorsBytes::score_point_vs_bytes"

This partially reverts commit c741e42f9a.
- trait `EncodedVectorsBytes` and its implementations: reverted/removed
- calls to `EncodedVectorsBytes::score_point_vs_bytes`: replaced with
  calls to `EncodedVectors::score_bytes`

* Remove unused method FilteredScorer::new_from_raw
2025-08-18 14:40:12 +00:00
Ivan Pleshkov
8961eeaf0a Remove quantization save functions (#7043)
* remove quantization save

* revert convert_binary_encoding fn

* review remarks

* review remark

* review remarks
2025-08-18 13:55:07 +02:00
Ivan Pleshkov
0d1b0ead53 Use PointOffsetType in quantization (#7058)
* Use PointOffsetType in quantization

* review remarks

* review remarks
2025-08-14 16:16:08 +02:00
Ivan Pleshkov
7eb562e891 Quantization flusher (#7046)
* quantization flusher

* review remarks
2025-08-13 12:57:16 +02:00
Ivan Pleshkov
1616806b8f BQ features without vector stats (#7009)
* bq all features without vector stats

* fix tests

* review suggestions

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-08-12 14:58:43 +02:00
xzfc
079ef6fb13 Implement GraphLinksFormat::CompressedWithVectors (#6982)
* Simplify graph_links tests

* Rstest-ify hnsw_index::tests::test_save_load

* Rename EncodedVectorsU8::{get_quantized_vector => get_quantized_vector_offset_and_code}

Reason: it was inconsistent with other implementations of
get_quantized_vector().

* Add QuantizedVectors::{get_quantized_vectors, get_quantized_vector_layout}

* Rename `links_map` to `for_each_link`

* pack_links: expose the updated `raw_links` order

* TestRawScorerProducer: support quantization

* Make GraphLinksSerializer::new return Result

* Rename links/compressed_links into `neighbors`

* Implement GraphLinksFormat::CompressedWithVectors

* GraphLayersWithVectors

* bitpacking: add packed_links_size

* Reorder neighbors: N__VVVVVVVcccccccccc -> Ncccccccccc__VVVVVVV
2025-08-11 17:28:43 +00:00
Ivan Pleshkov
766b527ac8 Appendable quantization storage (#6935)
* appendable qunatization storage

fmt

deprecate count in quantization config

fix arm tests build

qunatized storage vectors count

fix ci

are you happy clippy

fix compat tests

undo rename to `deprecated_count`

remove flusher

remove obsolete functions, don't use hardware counter in ram storage

are you happy clippy

fix gpu build

check ci when revert option

revert last commit

* debug ci

* log offsets

* log offsets

* fix compatibility tests

* deprecate count

* fix arm tests

* fix benches

* Update lib/quantization/src/encoded_vectors_binary.rs

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>

---------

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
2025-08-06 13:51:01 +02:00
xzfc
83005cd2db QueryScorerBytes (#6945)
* Add EncodedVectorsBytes::score_point_vs_bytes

* Add QueryScorerBytes::score_bytes

* QuantizedVectors::query_scorer_bytes

* Split FilteredScorer into FilteredScorer and FilteredQuantizedScorer
2025-07-31 03:05:10 +00:00
Ivan Pleshkov
6521b0df7d bq scalar query benches (#6831)
* bq scalar query benches

* review remarks
2025-07-14 19:28:07 +02:00
Ivan Pleshkov
bac28164ed use vector statistics for scalar bq query (#6835)
* use vector statistics for scalar bq query

* fix minor error

* remove test_binary_scalar_internal test

* do NOT use special file for storing vector stats

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-07-11 10:15:28 +02:00
Ivan Pleshkov
a38fd0a1e5 simplify 2bit encoding method (#6824) 2025-07-08 12:19:04 +02:00
Ivan Pleshkov
b414a402bb Merge pull request #6728
* # This is a combination of 7 commits.

* SameAsStorage default value

* fix coderabbit warnings

* neon for u8 bq

* sse for u8 bq

* fix windows build

* rename function

* add comments

* fmt

* fix arm build

* review remarks
2025-07-03 14:09:12 +02:00
Ivan Pleshkov
de54049e55 Don't encode quantization query while hnsw build (#6729)
* dont encode quantization query while hnsw build

* add comments

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-06-23 10:34:00 +02:00
Ivan Pleshkov
fe5becdadd Merge pull request #6663
* bq encodings

* are you happy clippy

* are you happy clippy

* are you happy clippy

* are you happy clippy

* gpu tests

* update models

* are you happy fmt

* move additional bits to the end

* fix tests

* Welford's Algorithm

* review remarks

* are you happy clippy

* remove debug println in test

* coderabit nitpicks

* remove unnecessary clone and partialeq

* Use f64 for Welford's Algorithm

* try fix ci

* revert cargo-nextest

* add debug assertions
2025-06-19 18:21:00 +02:00
xzfc
6717d38cc4 Replace some generic parameters with associated types (#6711)
* Make EncodedVectors::EncodedQuery associated type

* Make EncodedStorageBuilder::Storage associated type

* Make QueryScorer::TVector associated type

* Remove TVector from RawScorerImpl generic parameters
2025-06-17 07:32:22 +00:00
Arnaud Gourlay
1fb02a693c Fix vector_io_read for multivector recommendation (#6622)
* Fix vector_io_read for multivector recommendation

* fix

* cleanup

* single disposable

* do not use disposable hardware counter for quantized multivectors
2025-06-05 19:32:57 +02:00
Andrey Vasnetsov
0266508736 disk cache hygiene (#6323)
* wip: implement explicit populate and clear_cache functions for all components

* fmt

* implement clear and populate for vector storages

* fmt

* implement clear and populate for payload storage

* wip: implement explicit populate and clear_cache functions payload indexes

* implement explicit populate and clear_cache functions payload indexes

* fix clippy on CI

* only compile posix_fadvise on linux

* only compile posix_fadvise on linux

* implement explicit populate and clear_cache functions for quantized vectors

* fmt

* remove post-load prefault

* fix typo

* implement is-on-disk for payload indexes, implement clear on drop for segment, implement clear after segment build

* fmt

* also evict quantized vectors after optimization

* re-use and replace advise_dontneed
2025-04-09 10:54:30 +02:00
Andrey Vasnetsov
82af7aa481 vector-io-read measurement on query (#6197)
* remove mut getters from HardwareCounterCell, as mutability is not useful

* introduce vector-io multiplier

* remove RealCpuMeasurement structure

* set vector-io reads multipliers

* account vector reads in dense scorers

* fmt

* fix tests

* propagate hw_counter into posting list iterator

* fmt

* fix test

* wip: measure of sparse iterator

* fmt

* optimize skip_to

* minor refactoring

* keep current PointOffset in iterator to prevent unnecessary reads from memory

* adjust sparse search cpu cost - account for datatype

* fix test

* refactor search_context tests

* move tests into a module

* introduce more tests

* grammar

* review fixes

* fix clippy

* fix clippy again

* change disposable -> new
2025-03-24 13:15:04 +01:00
Arnaud Gourlay
423a39f301 Remove duplicated mmap utils (#6129) 2025-03-10 12:20:22 +01:00
Tim Visée
3e536347e1 Bump Rust edition to 2024 (#6042)
* Bump Rust edition to 2024

* gen is a reserved keyword now

* Remove ref mut on references

* Mark extern C as unsafe

* Wrap unsafe function bodies in unsafe block

* Geo hash implements Copy, don't reference but pass by value instead

* Replace secluded self import with parent

* Update execute_cluster_read_operation with new match semantics

* Fix lifetime issue

* Replace map_or with is_none_or

* set_var is unsafe now

* Reformat
2025-02-25 11:21:25 +01:00
Ivan Pleshkov
1447d1efa8 atomic save quantization metadata (#5627) 2024-12-09 21:24:24 +01:00
xzfc
4548ebe4bf Reduce debug size (#5556)
* debug size: RawScorerImpl::peek_top_iter

* debug size: TypedMultiDenseVector::multi_vectors() uses

* debug size: stop_condition -> stopped
2024-12-02 14:33:35 +00:00
Ivan Pleshkov
a770a7f3a0 Gpu vectors (#5339)
* parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012991 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012977 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012901 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012766 +0100

parent 33b2c176cb
author Ivan Pleshkov <pleshkov.ivan@gmail.com> 1729676438 +0200
committer Ivan Pleshkov <pleshkov.ivan@gmail.com> 1732012730 +0100

vulkan api wrapper crate

fmt

gpu is not a workspace member

exclude gpu from workspace

disable tests for gpu crate

ash version up

change collection for pipeline builder

gpu test

gpu is not optional

check device features

remove feature

make all fields private

more comments

are you happy fmt

qdrant team as authors

gpu vectors

move changes from magic branch

move storage updates

are you happy codespell

are you happy clippy

don't make storage_impl public

review remarks

are you happy clippy

are you happy fmt

* Update lib/segment/src/index/hnsw_index/gpu/gpu_vector_storage/tests.rs

Co-authored-by: Tim Visée <tim+github@visee.me>

* review remarks

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
2024-11-27 09:50:10 +01:00
Jojii
7b677fb7fa Hardware counting for quantization (#5369)
* add cpu measurement for quantization

* clippy

* fix tests

* discard hardware counters in benchmarks

* Forwarding hardware counter in distributed setup (#5371)

* make hardware counter available in distributed setup

* clippy

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-06 16:23:02 +01:00
Ivan Pleshkov
f27b6909bd Move quantization repo (#5096)
* move quantization repo

* are you happy fmt

* are you happy clippy

* remove dumping pq to image

* workspace deps

* are you happy clippy
2024-09-17 16:30:26 +02:00