* make `EncodedStorage::for_each_batch` mandatory
* make `DenseVectorStorageRead::for_each_in_dense_batch` mandatory
* make `DenseTQVectorStorage::for_each_in_dense_batch` mandatory
* make `DenseTQVectorStorage::read_dense_tq_bytes` mandatory
* make `QueryScorer::score_stored_batch` mandatory
...and implement for tq multivectors
* [AI] make `IdTrackerRead::internal_versions_batch` mandatory
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* [AI] make `IdTrackerRead::external_ids_batch` mandatory
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* [AI] make `DiskMappingsSource::resolve_internal_batch` mandatory
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add Ram variant, loads from vector storage
* impl live reload
* preopen sparse storage when using ram index
* Batch storage reads when building sparse RAM index
Rebuild `build_ram_index` around a single `read_vectors::<Sequential>`
pass instead of a per-point `get_vector_opt` loop, so Gridstore
coalesces the IO into block reads instead of a round-trip per point.
Benefits both the read-only mutable-RAM rebuild and the writable
`SparseVectorIndex::plan` build path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* [AI] piggyback on in-flight requests
* cleanup tests
* Insert in-flight fetch through Slab's VacantEntry (#9840)
Replace the peek-key-then-insert pattern with slab's vacant_entry() API:
inserting through the reserved entry guarantees the fetch lands on the
key the remote read was tagged with, instead of relying on nothing
touching the slab between the peek and the insert. A failed remote
schedule still leaves no trace, as VacantEntry allocates nothing until
insert.
get_or_init_remote_pipeline now takes the field instead of &mut self so
the VacantEntry can hold a disjoint borrow of in_flight across the
schedule call.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* impl `OnDiskMapIndex::preopen`
* Prepare for preopen all payload indexes
* impl `ReadOnlyAppendableMapIndex::preopen`
* also use it for int and uuid index
* respect low memory mode
* [AI + manual] Include file size in `UniversalReadFileOps::list_files`
* Use dedicated `ListedFile` struct instead of `(PathBuf, u64)` pair
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* hold init_guard during reopen
* add `from` parameter to `OwnedReadPipeline::schedule_whole`
* add `OwnedReadPipeline::into_inner` to extract inner file
* implement proper background populate on `reopen`
* clippy
* coderabbit
* explicit owned uring pipeline destructuring
* refactor to join local and remote into one state
* @ffuugoo's nits
* override `for_values_map` for `OnDiskMapIndex`
* refactor and apply to filter as id iterator
* add comment
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* feat(facet): add sampling strategy for high-cardinality fields
For approximate facet queries the current per-segment implementation
walks every distinct value in the field index — O(unique_values_count)
even when the user asks for a tiny top-K. On UUID-style fields with
millions of unique values this dominates the request latency even
after #9208 capped the cross-shard payload.
This commit adds a parallel sampling strategy that runs in O(limit)
instead of O(unique_values_count):
1. Phase 1 — iterative novelty sampling. Stream point IDs in random
order (filtered if requested), look up each point's value via the
facet index, and collect distinct values into a candidate set until
`limit * 10` (min 1000) candidates have been gathered. Uses a
batch size of 32 to amortise the inner `for_points_values` call,
and bails out early after 128 consecutive empty batches when the
long tail is too thin to keep finding novel values.
2. Phase 2 — exact-count post-pass. For each candidate value, compose
`field == value` with the user filter and count via the payload
index. This guarantees the returned counts are exact (matching the
semantics of the full-scan path); only the *set* of returned values
is approximate.
The two strategies live side by side; `SegmentReadView::approximate_facet`
picks between them per-request based on
`unique_values_count > limit * FACET_FULL_SCAN_FACTOR` (FACTOR = 4).
Below that, the existing scan path runs unchanged — it'd visit most of
the index either way, and the post-pass adds no value.
The Monte-Carlo simulation behind this design (see thread context for
Zipf-distributed fields with cardinality up to 10^5 in ~1000 samples,
and trivially-correct results on UUID-style fields where every value
has count 1.
Adds a new `unique_values_count` method on the `FacetIndex` trait
(implemented for `MapIndex`, `ReadOnlyMapIndex`, `BoolIndex`,
`ReadOnlyBoolIndex`, and the `FacetIndexEnum` dispatcher) so the
strategy switch can run without touching the index.
Co-authored-by: Cursor <cursoragent@cursor.com>
* [AI] simplify, use single file
[AI] better selection of filtering approach
fmt
[AI] simplify, use single file
* manual simplification
* [AI] implement candidate-based lookups
[AI] 🧹
* precollect filter into bitmap
* avoid sampling with restrictive filter
* fix rebase + clippy
* refactor tests
* no duplicate values in map index
* polish comments
---------
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
`GridstoreReader`)
- Make the entrypoint of payload storage choose the `Populate` variant
- Mutable payload indexes now populate gridstore blockingly before using
it to load
- Propagate `Populate` into `flags` module
- Add `UniversalRead::populate_auto` to know whether a backend chooses
to populate or not when `Populate::Auto`
* rename to OnDiskFullTextIndex
* rename module to `on_disk_text_index`
* rename to OnDiskPostings
* rename module to `on_disk_inverted_index`
* rename to OnDiskInvertedIndex
* remove `Box`ing
* get rid of `is_on_disk` field, propagate Populate
* add separate OnDisk and Immutable readonly variants
* rename in tests too
* nits
* rename to OnDiskMapIndex
* rename module
* remove `is_on_disk` field from `OnDiskMapIndex`
* add separate Immutable and OnDisk ReadOnlyMapIndex
* rename StoredPointToValues to OnDiskPointToValues
* rename module to `on_disk_point_to_values`
* rename to `build_from_iter` and add `populate` arg
* fix test
* rename IndexSelector variants
* rename MapIndex::Mmap to MapIndex::OnDisk
* rename NumericIndex::Mmap to NumericIndex::OnDisk
* rename BoolIndex::Mmap to BoolIndex::Mutable
* rename FullTextIndex::Mmap to FullTextIndex::OnDisk
* rename GeoMapIndex::Storage to GeoMapIndex::OnDisk
Also rename a few functions and test variants
* Standardize numeric index (#9294)
* remove `UniversalNumericIndex.is_on_disk` field
* have separate Immutable and OnDisk variants for ReadOnlyNumericIndexInner
* rename UniversalNumericIndex -> OnDiskNumericIndex
* rename module to `on_disk_numeric_index`
* actually load as immutable if on_disk=false
* fixup! Standardize numeric index (#9294)
Make builder also return proper Immutable variant
* review nits
* load with any universal io
* Don't populate with sequential advice
* use appropriate fs for `exists`
* use `read_whole_via` more
* TODO
* clear ram cache after read_whole
* Oversample with approx facet
* return early for limit=0
* Document distributed limit flow on Collection::facet
Add an ASCII diagram showing how the facet limit is oversampled once on
the entry node and how peer nodes enter via facet_internal without
re-oversampling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [AI + manual] initial impl
[AI] read_batch which actually batches
manual nits
[AI] better handling of local and remote paths
manual refactor, respect open options
don't delete local file
dumbify read_batch
we want to refactor it anyway
simplify
rename to `DiskCache` in `simple_disk_cache` module
* refactor to use always use ReadPipeline
pass meta to remote pipeline
* nits
* run tests for more Remotes
* fix no more <T> in UniversalRead
* fmt
* chore(deps): unify roaring as workspace dep, move duplicate to dev-deps
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* use generic in QuantizedMmapStorage
* rename to QuantizedStorage
* be explicit about S
* rename builder to `QuantizedStorageBuilder`
* rename file to `quantized_storage.rs`
* bytemuck::Pod already implies 'static
* no supertraits
* regions gaps with TypedStorage
* StoredBitSlice with TypedStorage
* bytemuck for PostingsHeader
* bytemuck for TrackerHeader
* bytemuck for MmapRange
* bytemuck for stored_point_to_values::Header
* derive bytemuck::Pod on `OptionalPointer`, get rid of legacy transmute
* remove one layer of Option in `get_raw`
* keep `UniversalRead<u8>` bound, do bytemuck conversion at call site (#8953)
Drop the `+ UniversalRead<OptionalPointer>` / `+ UniversalWrite<OptionalPointer>`
trait bounds added in the previous two commits. Reads and writes of
`OptionalPointer` and `TrackerHeader` now go through `bytemuck::from_bytes` /
`bytemuck::bytes_of` on the existing `u8` byte slices instead.
This keeps the unsafe-transmute removal but avoids the turbofish noise
(`UniversalWrite::<u8>::flusher(&self.storage)`, `<S as UniversalRead<u8>>::open`,
…) that two `T`s on `S` forced everywhere.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* extra refactoring after #8955
* explicit zeroed instantiation
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* genericize ChunkedVectors.status, remove `Sized` bound
* check exists with `UniversalReadFileOps`
* Inline UioChunkedVectors bound, drop the alias (#8952)
The empty trait + blanket impl was a stable-Rust trait-alias workaround
that hid a fairly short bound (UniversalWrite<T> + UniversalWrite<Status>
+ Send + 'static) at the cost of an indirection readers had to mentally
unwind. Spelling it out at the three sites that need it is shorter overall
and immediately tells the reader what is required.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* propagate to BufferedDynamicFlags
* use duplicate for tests
* propagate to Bitvec/Roaring flags
* propagate to RoaringFlags
* fixup! propagate to BufferedDynamicFlags
* propagate to BitvecFlags
* derive Pod for Point<T>
* remove clones on Copy type
* fix annoying clippy lint
* serde skip padding
* extract point into a new file
* assert aligment
* feat(universal_io): add storage-agnostic BitSliceStorage with read/write support`
Add BitSliceStorage<S> generic over
UniversalRead<u64>/UniversalWrite<u64>,
providing bit-level read and write operations over u64-element storage.
Read operations (S: UniversalRead<u64>):
- read_all: returns Cow<BitSlice> (zero-copy for mmap, owned for others)
- get_bit: single bit read via u64 element fetch
- read_bit_range: arbitrary bit range read
Write operations (S: UniversalWrite<u64>):
- set_bit / replace_bit: single bit write (skips write if unchanged)
- write_bit_range: arbitrary bit range write from BitSlice source
- fill_bit_range: fill range with a value
- set_bits_batch: batch individual bit updates coalesced by element
- flusher: flush underlying storage
* refactor: replace MmapBitSlice with BitSliceStorage in MmapBitSliceBufferedUpdateWrapper
Migrate all deleted-flag bitslice storage from the legacy MmapBitSlice
(Deref-based mmap wrapper) to BitSliceStorage<MmapUniversal<u64>>
(storage-agnostic universal IO backend).
Updated consumers:
- MmapMapIndex
- MmapNumericIndex
- MmapGeoMapIndex
- MmapInvertedIndex (fulltext)
- ImmutableIdTracker
- Benchmark
Additionally:
- Add BitSliceStorage::create(path, num_bits) to encapsulate
file creation + sizing + open (eliminates duplicated size math
across 5 call sites)
- Add MmapBitSliceStorage type alias for
BitSliceStorage<MmapUniversal<u64>>
- Add count_ones() convenience method
- Use set_bits_batch() in wrapper flusher and all build paths
instead of per-bit set_bit() loops (coalesces u64 read-modify-writes)
- Fix silent error swallowing in wrapper get(): log + debug_assert
on I/O errors instead of .ok().flatten()
- Remove dead bitmap_mmap_size function from immutable_id_tracker
* use bitvec's approach for offset
* less api
* misc improvements
* refactor write method
* move to segment crate
* handle creation of file outside of StoredBitSlice logic
* fix codespell
* document bitwise calculations
* coalesce more updates into a single write, batch all writes
* use native `extend_from_bitslice` instead of iterating.
* clarity refactor
* clippyyy
* use `u64` as BitStore everywhere
* assume iterator is sorted
* only use chunk_by for generating runs
* fix update wrapper flusher
* review fixes
* larger set bits batch test
* remove reintroduced file
* oops, fix new test
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* refactor PointToValues to use UniversalRead
It includes a significant change to `MmapValue` trait to be able to
handle Cow reads, instead of just references.
* fix str parsing
* fix incorrect path
* use fallible casting
* clippy
* No eager allocation
this also makes it so that borrowed strs can keep happening :heart-eyes:
* Lifetime cleanup
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* feat: add "any" to "match" statements
Adds support for using "any" to match on a list of different possible values
* enable proper counting in cardinality estimation
* fmt
* upd OpenAPI
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Luis Cossío