* existing segments: wait for IO outside of search pool
* new segments: wait for IO outside of search pool
* extract reload into separate function
* Update lib/edge/Cargo.toml
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* rename `reopen`->`live_reload` and `schedule_reopen`->`live_preload`
* `UniversalRead::live_preload` returns a shared future
* assert snapshot-miss eagerly on `live_preload`
`live_reload` cannot see the failed preload: its blocking fallback
re-resolves the length from the remote and succeeds. The error
surfaces at preload time, as callers (`ok_not_found`) expect.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [CachedFs] new `schedule` and `wait_all` primitives
* [AppendableIdTracker] don't reopen if just opened
* eager NotFound in `schedule_open`
* add traces for async reads
* finish `preopen`/`preload` with `wait_all`
* lock all segments in parallel for `live_reload`
* LIST before everything
to do: we don't have whole-fetch in async mode. to prevent sequential
`len`, we won't overlap static files with LIST.
* `wait_all` returns nothing
* `UniversalReadFs::open_async`
* `schedule_open` polls once
Scheduled opens must start eagerly: sync backends complete their
`open_async` on the first poll, preserving the prefetch contract
(handles outlive later file deletions/replacements). Moved down from
the integration branch so this PR stays green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* impl live_preload for payload indexes
enable live_preload for bool and null indexes
* (not) impl live_preload for `VectorIndexReadEnum`
* impl live_preload for `ReadOnlyPayloadStorage`
* Extract stored bitmask encoding into `bitmask_file_bytes`
Also single-source the u32 position-space bound as `MAX_LOGICAL_LEN`.
* Add `MutableStoredBitmask`, collecting bitmask changes in RAM
Materializes via the existing reader without keeping the file handle open, tracks diverged positions, and atomically rewrites the whole file on save - skipping the write when nothing changed.
* Rename payload to bits
* Use changed boolean
* Remove now obsolete test
* Borrow the bitmap in bitmask encoding via Cow, avoiding a clone on save
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* add `UniversalIoError::UnchangedOpen`
* add & impl `CachedReadFs::reschedule_prefetch`
* clippy
* add `OkUnchanged` helper
* propagate scheduling errors immediately
* drop lock before reacquiring it
* clear prefetched_files on new snapshot
* only avoid prefetch on full FileInfo equality
* `UnchangedOpen` maps to `Cancelled`
* match-all match
* AI + manual: impl `UpdateOnlyChunkedVectors`
* AI: simplify
* graceful handling of unexpected file lengths
fix test
* incorporate updates from #10119
* drop the unused status read on open
The vector count loaded at open was never consulted: every batch carries the
offset it starts at, and the chunks are reconciled against that offset. Drop
the field and the read, and fold both watermark writes into `save_len`.
A corrupt status file no longer blocks opening the writer — the first batch
overwrites it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix clippy: ensure_chunk_lengths no longer needs &mut self
Dropping the status field left it with nothing to mutate. `append_many` keeps
`&mut self` — nothing in this module is exported, so the lint reaches it too,
but the exclusive borrow is what enforces the single-writer contract the
appends rest on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* AI: implement 2 stage reopen
* manual: simplification refactor
* AI: simplify further
* upd trait interface
* use closure instead of `&CachedReadFs`
* open a new remote for the tail fetch
* rename to `cached_file_info`
* only resize after fallible op
* use consistent remote openoptions
* make `EncodedStorage::for_each_batch` mandatory
* make `DenseVectorStorageRead::for_each_in_dense_batch` mandatory
* make `DenseTQVectorStorage::for_each_in_dense_batch` mandatory
* make `DenseTQVectorStorage::read_dense_tq_bytes` mandatory
* make `QueryScorer::score_stored_batch` mandatory
...and implement for tq multivectors
* [AI] make `IdTrackerRead::internal_versions_batch` mandatory
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* [AI] make `IdTrackerRead::external_ids_batch` mandatory
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* [AI] make `DiskMappingsSource::resolve_internal_batch` mandatory
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add Ram variant, loads from vector storage
* impl live reload
* preopen sparse storage when using ram index
* Batch storage reads when building sparse RAM index
Rebuild `build_ram_index` around a single `read_vectors::<Sequential>`
pass instead of a per-point `get_vector_opt` loop, so Gridstore
coalesces the IO into block reads instead of a round-trip per point.
Benefits both the read-only mutable-RAM rebuild and the writable
`SparseVectorIndex::plan` build path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* [AI] piggyback on in-flight requests
* cleanup tests
* Insert in-flight fetch through Slab's VacantEntry (#9840)
Replace the peek-key-then-insert pattern with slab's vacant_entry() API:
inserting through the reserved entry guarantees the fetch lands on the
key the remote read was tagged with, instead of relying on nothing
touching the slab between the peek and the insert. A failed remote
schedule still leaves no trace, as VacantEntry allocates nothing until
insert.
get_or_init_remote_pipeline now takes the field instead of &mut self so
the VacantEntry can hold a disjoint borrow of in_flight across the
schedule call.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* impl `OnDiskMapIndex::preopen`
* Prepare for preopen all payload indexes
* impl `ReadOnlyAppendableMapIndex::preopen`
* also use it for int and uuid index
* respect low memory mode
* [AI + manual] Include file size in `UniversalReadFileOps::list_files`
* Use dedicated `ListedFile` struct instead of `(PathBuf, u64)` pair
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* hold init_guard during reopen
* add `from` parameter to `OwnedReadPipeline::schedule_whole`
* add `OwnedReadPipeline::into_inner` to extract inner file
* implement proper background populate on `reopen`
* clippy
* coderabbit
* explicit owned uring pipeline destructuring
* refactor to join local and remote into one state
* @ffuugoo's nits
* override `for_values_map` for `OnDiskMapIndex`
* refactor and apply to filter as id iterator
* add comment
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* feat(facet): add sampling strategy for high-cardinality fields
For approximate facet queries the current per-segment implementation
walks every distinct value in the field index — O(unique_values_count)
even when the user asks for a tiny top-K. On UUID-style fields with
millions of unique values this dominates the request latency even
after #9208 capped the cross-shard payload.
This commit adds a parallel sampling strategy that runs in O(limit)
instead of O(unique_values_count):
1. Phase 1 — iterative novelty sampling. Stream point IDs in random
order (filtered if requested), look up each point's value via the
facet index, and collect distinct values into a candidate set until
`limit * 10` (min 1000) candidates have been gathered. Uses a
batch size of 32 to amortise the inner `for_points_values` call,
and bails out early after 128 consecutive empty batches when the
long tail is too thin to keep finding novel values.
2. Phase 2 — exact-count post-pass. For each candidate value, compose
`field == value` with the user filter and count via the payload
index. This guarantees the returned counts are exact (matching the
semantics of the full-scan path); only the *set* of returned values
is approximate.
The two strategies live side by side; `SegmentReadView::approximate_facet`
picks between them per-request based on
`unique_values_count > limit * FACET_FULL_SCAN_FACTOR` (FACTOR = 4).
Below that, the existing scan path runs unchanged — it'd visit most of
the index either way, and the post-pass adds no value.
The Monte-Carlo simulation behind this design (see thread context for
Zipf-distributed fields with cardinality up to 10^5 in ~1000 samples,
and trivially-correct results on UUID-style fields where every value
has count 1.
Adds a new `unique_values_count` method on the `FacetIndex` trait
(implemented for `MapIndex`, `ReadOnlyMapIndex`, `BoolIndex`,
`ReadOnlyBoolIndex`, and the `FacetIndexEnum` dispatcher) so the
strategy switch can run without touching the index.
Co-authored-by: Cursor <cursoragent@cursor.com>
* [AI] simplify, use single file
[AI] better selection of filtering approach
fmt
[AI] simplify, use single file
* manual simplification
* [AI] implement candidate-based lookups
[AI] 🧹
* precollect filter into bitmap
* avoid sampling with restrictive filter
* fix rebase + clippy
* refactor tests
* no duplicate values in map index
* polish comments
---------
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
`GridstoreReader`)
- Make the entrypoint of payload storage choose the `Populate` variant
- Mutable payload indexes now populate gridstore blockingly before using
it to load
- Propagate `Populate` into `flags` module
- Add `UniversalRead::populate_auto` to know whether a backend chooses
to populate or not when `Populate::Auto`
* rename to OnDiskFullTextIndex
* rename module to `on_disk_text_index`
* rename to OnDiskPostings
* rename module to `on_disk_inverted_index`
* rename to OnDiskInvertedIndex
* remove `Box`ing
* get rid of `is_on_disk` field, propagate Populate
* add separate OnDisk and Immutable readonly variants
* rename in tests too
* nits
* rename to OnDiskMapIndex
* rename module
* remove `is_on_disk` field from `OnDiskMapIndex`
* add separate Immutable and OnDisk ReadOnlyMapIndex
* rename StoredPointToValues to OnDiskPointToValues
* rename module to `on_disk_point_to_values`
* rename to `build_from_iter` and add `populate` arg
* fix test
* rename IndexSelector variants
* rename MapIndex::Mmap to MapIndex::OnDisk
* rename NumericIndex::Mmap to NumericIndex::OnDisk
* rename BoolIndex::Mmap to BoolIndex::Mutable
* rename FullTextIndex::Mmap to FullTextIndex::OnDisk
* rename GeoMapIndex::Storage to GeoMapIndex::OnDisk
Also rename a few functions and test variants
* Standardize numeric index (#9294)
* remove `UniversalNumericIndex.is_on_disk` field
* have separate Immutable and OnDisk variants for ReadOnlyNumericIndexInner
* rename UniversalNumericIndex -> OnDiskNumericIndex
* rename module to `on_disk_numeric_index`
* actually load as immutable if on_disk=false
* fixup! Standardize numeric index (#9294)
Make builder also return proper Immutable variant
* review nits
* load with any universal io
* Don't populate with sequential advice
* use appropriate fs for `exists`
* use `read_whole_via` more
* TODO
* clear ram cache after read_whole
* Oversample with approx facet
* return early for limit=0
* Document distributed limit flow on Collection::facet
Add an ASCII diagram showing how the facet limit is oversampled once on
the entry node and how peer nodes enter via facet_internal without
re-oversampling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [AI + manual] initial impl
[AI] read_batch which actually batches
manual nits
[AI] better handling of local and remote paths
manual refactor, respect open options
don't delete local file
dumbify read_batch
we want to refactor it anyway
simplify
rename to `DiskCache` in `simple_disk_cache` module
* refactor to use always use ReadPipeline
pass meta to remote pipeline
* nits
* run tests for more Remotes
* fix no more <T> in UniversalRead
* fmt
* chore(deps): unify roaring as workspace dep, move duplicate to dev-deps
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* use generic in QuantizedMmapStorage
* rename to QuantizedStorage
* be explicit about S
* rename builder to `QuantizedStorageBuilder`
* rename file to `quantized_storage.rs`
* bytemuck::Pod already implies 'static
* no supertraits
* regions gaps with TypedStorage
* StoredBitSlice with TypedStorage
* bytemuck for PostingsHeader
* bytemuck for TrackerHeader
* bytemuck for MmapRange
* bytemuck for stored_point_to_values::Header
* derive bytemuck::Pod on `OptionalPointer`, get rid of legacy transmute
* remove one layer of Option in `get_raw`
* keep `UniversalRead<u8>` bound, do bytemuck conversion at call site (#8953)
Drop the `+ UniversalRead<OptionalPointer>` / `+ UniversalWrite<OptionalPointer>`
trait bounds added in the previous two commits. Reads and writes of
`OptionalPointer` and `TrackerHeader` now go through `bytemuck::from_bytes` /
`bytemuck::bytes_of` on the existing `u8` byte slices instead.
This keeps the unsafe-transmute removal but avoids the turbofish noise
(`UniversalWrite::<u8>::flusher(&self.storage)`, `<S as UniversalRead<u8>>::open`,
…) that two `T`s on `S` forced everywhere.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* extra refactoring after #8955
* explicit zeroed instantiation
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* genericize ChunkedVectors.status, remove `Sized` bound
* check exists with `UniversalReadFileOps`
* Inline UioChunkedVectors bound, drop the alias (#8952)
The empty trait + blanket impl was a stable-Rust trait-alias workaround
that hid a fairly short bound (UniversalWrite<T> + UniversalWrite<Status>
+ Send + 'static) at the cost of an indirection readers had to mentally
unwind. Spelling it out at the three sites that need it is shorter overall
and immediately tells the reader what is required.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* propagate to BufferedDynamicFlags
* use duplicate for tests
* propagate to Bitvec/Roaring flags
* propagate to RoaringFlags
* fixup! propagate to BufferedDynamicFlags
* propagate to BitvecFlags
Luis Cossío