mirror of
https://github.com/qdrant/qdrant.git
synced 2026-07-25 20:21:08 -05:00
dev
2272 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Ivan Pleshkov
|
7469834d9b |
[TQDT] TQ dense/multi vector storage consistency (#9953)
* [TQDT] Align TQ vector storage layout with the reference dense storage
Make the TurboQuant vector storage structurally mirror the reference dense
(and multi_dense) storages, down to file names and their contents:
- Rename files + structs to the dense convention:
- immutable.rs -> turbo_vector_storage.rs (ImmutableTurboVectorStorage ->
TurboVectorStorageImpl)
- appendable.rs -> appendable_turbo_vector_storage.rs
(AppendableTurboVectorStorage -> AppendableMmapTurboVectorStorage)
- multi.rs -> multi_turbo/appendable_mmap_multi_turbo_vector_storage.rs
(TurboMultiVectorStorage -> AppendableMmapMultiTurboVectorStorage)
- ReadOnlyTurboMultiVectorStorage -> ReadOnlyChunkedMultiTurboVectorStorage
- Thin out turbo/mod.rs to module declarations + re-exports: open_* fns move
into their storage files, consts + turbo_storage_roundtrip into shared.rs,
and TurboScoring / TurboMultiScoring join the other TQ traits in
vector_storage_base.rs.
- Split read_only/ into the chunked storage (read_only/) and the single-file
storage (read_only/immutable/), each with the mod/lifecycle/live_reload/
read_ops 4-file layout, mirroring dense/read_only/.
- Introduce multi_turbo/ mirroring multi_dense/, with its own read_only/
submodule holding ReadOnlyChunkedMultiTurboVectorStorage.
- Relocate the storage test suites to
tests/test_appendable_turbo_vector_storage.rs and
tests/test_appendable_multi_turbo_vector_storage.rs, paralleling the
dense/multi_dense integration test files (tests moved verbatim, no new
tests added).
- Fix a gpu-gated VectorStorageEnum match that referenced stale DenseTurbo /
DenseTurboAppendable variant names.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* are you happy fmt
* fix after rebase
* [TQDT] Address review feedback on TQ vector storage split
- gpu tests: use the real `VectorStorageEnum::DenseTurboAppendableMemmap`
variant (the old `DenseTurboAppendable` name never existed post-rename, so
the gpu-feature test failed to compile — missed because `cargo build
--features gpu` does not compile the `#[cfg(test)]` code).
- memory_reporter: report `DenseTurboUring` files as `FileStorageIntent::OnDisk`
like the other io_uring variants; io_uring never mmap-caches, so delegating
to `is_on_disk()` could wrongly report `Cached` for a populated backend.
- turbo_vector_storage: fix the misleading `insert_tq_bytes` doc comment — the
single-file backend rejects the upsert via `?`, so `set_deleted` is never
reached.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [TQDT] Fix clippy::wildcard_enum_match_arm in read-only routing test
Spell out the non-routing `VectorStorageType` variants instead of `_`, so a
future added variant fails the match rather than silently mapping to `false`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [TQDT] Fix stale Turbo4 storage-variant assertion in quantization test
The segment is built with the default (appendable/chunked) storage type, so a
Turbo4 datatype now lands in `DenseTurboAppendableMemmap`, not the single-file
`DenseTurboMemmap`. The assertion was left on the pre-split variant; align it
with the non-turbo branch, which already expects the appendable variants.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* are you happy fmt
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Andrey Vasnetsov
|
94fdd0e746 |
Support memory placement in service-level storage config defaults (#9950)
* Support memory placement in service-level storage config defaults Follow-up to #9684: `storage.payload.memory` and `storage.collection.vectors.memory` set service-wide placement defaults for newly created collections, deprecating `storage.on_disk_payload` and `storage.collection.vectors.on_disk`. Defaults resolve as: request `memory` > request legacy flag > service `memory` > service legacy flag; exactly one level is filled to avoid spurious memory-vs-legacy mismatch warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Mark hnsw_index.on_disk deprecated in config.yaml, document memory option Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Daniel Boros
|
3b77388f7e |
fix: fold appended points' deletion into read-only live_reload (#9948)
* fix: fold appended points' deletion into read-only live_reload * fix: batch appended-deletion reads in read-only live_reload * fix: clamp appended-deletion reload to persisted flag length |
||
|
Luis Cossío
|
d32f738c1f |
[CI] Enforce no default impl for batch methods (#9939)
* [AI] Add ast-grep rule to avoid default batch trait methods * reword * fix `FullTextIndexRead::check_match_batch` * move to `tools/ast-grep/` * Add tests * pin ast-grep version * fix spelling |
||
|
Jojii
|
89cf9295a9 | Remove wrong debug assertion (#9945) | ||
|
Ivan Pleshkov
|
542f722105 |
[TQDT] Support Turbo4 in the read-only vector storage path (#9925)
* feat(segment): support Turbo4 in the read-only vector storage path Read-only segments (UniversalRead-backed: mmap / cache / remote) could not open a `Turbo4`-typed vector storage — every dispatch arm returned "not yet supported". This adds the missing read-only TurboQuant storages so read-only segments can retrieve and score TQ-typed vectors. - Reuse the existing, correct TQ scoring: extract `TurboScoring` / `TurboMultiScoring` traits and generalize `TurboQueryScorer` / `TurboCustomQueryScorer` (+ multi) and `raw_turbo_*_scorer_impl` from the concrete `TurboVectorStorage` to `&impl TurboScoring`. Scoring logic is untouched. - Split `DenseTQVectorStorage` / `MultiTQVectorStorage` into read-only (`*Read`) + write supertraits, mirroring the dense storages, so the read-only storage need not implement the ingest path. - Add `ReadOnlyTurboVectorStorage<S>` (single-file + chunked backends) and `ReadOnlyTurboMultiVectorStorage<S>` over `QuantizedStorage<S>` / `QuantizedChunkedStorageRead<S>` + `InMemoryBitvecFlags`; the quantizer is rebuilt from `(dim, distance)`, so nothing beyond the encoded bytes and flags is read from disk. - Wire `DenseTurbo` / `MultiDenseTurbo` variants into `VectorStorageReadEnum` (dispatch, scorer, live-reload) and replace the `Turbo4` stubs in the read-only `open` / `preopen`. - Round-trip tests: writable TQ -> reopen read-only, asserting byte-exact raw TQ bytes, identical dequantization, deletion flags and per-point scores matching the writable scorer (dense single-file + chunked, and multi). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * review remarks * Split turbo read_only.rs into a module Mirror the layout of the other read-only vector storages: split the oversized read_only.rs into mod.rs (struct definitions + shared encoded backend enum), lifecycle.rs (preopen/open), read_ops.rs (retrieval + scoring trait impls) and live_reload.rs. Co-authored-by: Cursor <cursoragent@cursor.com> * review remarks * fix after rebase --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Arnaud Gourlay
|
f8b5a9ce60 |
Tests: use SmallRng for RNG-bound test data generation in segment (#9889)
The turbo model tests, HNSW graph properties tests, and id-tracker mapping tests generate their datasets with StdRng (ChaCha12 in rand 0.10). Switching to SmallRng (Xoshiro256++) makes the turbo model tests ~10-14% faster and the 400k-mapping id-tracker test ~30% faster. All affected tests pass with the changed seeded sequences, including the tolerance-carrying turbo model comparisons. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
xzfc
|
48e710ce0b |
Cleanup UniversalRead methods interface (#9934)
* use common::generic_consts::{Random, Sequential};
* UniversalRead::read_batch: generic over E
* UniversalRead::read_batch: pass `AccessPattern` as ZST arg
* UniversalRead::read_bytes_iter: pass `AccessPattern` as ZST arg
* UniversalRead::read_iter: pass `AccessPattern` as ZST arg
* UniversalRead::read: pass `AccessPattern` as ZST arg
* UniversalRead::read_bytes: pass `AccessPattern` as ZST arg
|
||
|
xzfc
|
7e99cdd86a | UioResult (#9933) | ||
|
Luis Cossío
|
fca67539f4 |
Mandatory batch impls (#9935)
* make `EncodedStorage::for_each_batch` mandatory * make `DenseVectorStorageRead::for_each_in_dense_batch` mandatory * make `DenseTQVectorStorage::for_each_in_dense_batch` mandatory * make `DenseTQVectorStorage::read_dense_tq_bytes` mandatory * make `QueryScorer::score_stored_batch` mandatory ...and implement for tq multivectors * [AI] make `IdTrackerRead::internal_versions_batch` mandatory Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * [AI] make `IdTrackerRead::external_ids_batch` mandatory Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * [AI] make `DiskMappingsSource::resolve_internal_batch` mandatory Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
xzfc
|
ba3043b343 | ConditionChecker::check_batched: use in hnsw (#9845) | ||
|
xzfc
|
9bb75ea2d1 |
Batched ConditionChecker (#9740)
* ConditionChecker::check_batched: trait + ConditionCheckerEnum * check_batched for OptimizedFilter * OnDiskPointToValues::values_iter_batch: improve performance * OnDiskPointToValues::values_iter_batch: update interface Accept bitvec, call on every point, pass UserData. * check_batched for geo index * geo_index tests: add same_geo_index_between_points_with_dups_test Catches broken load_from_on_disk/for_all_points_values. * check_batched for map index * check_batched for numeric index * check_batched for full-text index * tests for check_batched * Review fixups - default_check_batched: avoid running pred twice at boundary - condition_checker: use explicit match over matches!/if-else - Partitioner: use assert! over debug_assert! Co-authored-by: Luis Cossío <luis.cossio@outlook.com> * ConditionChecker::check_batched: &mut self -> &self * ConditionChecker::check_batched: make mandatory --------- Co-authored-by: Luis Cossío <luis.cossio@outlook.com> |
||
|
Jojii
|
0745c36c8f | Rename WrongVectorBytesSize -> MalformedVectorBlob (#9929) | ||
|
Jojii
|
c561e433c6 |
[TQDT] upsert raw malformed blob multi + sparse vectors (#9904)
* Fix malformed vector upsertion for multivecs and sparse-vecs too * Shorten comment * Fix after rebase |
||
|
Ivan Pleshkov
|
1353d54eb5 | [TQDT] Fix/upsert raw malformed blob badinput (#9886) | ||
|
Andrey Vasnetsov
|
0595333a65 |
Batch IdTracker read operations, pipelined in disk-resident trackers (#9809)
* Add batch id-tracker lookups, pipelined in the disk-resident trackers
Introduce batch counterparts for the per-point IdTracker read operations
(external->internal resolution, internal->external lookup, version lookup,
deleted checks) and implement them with pipelined reads in the disk-resident
trackers, where the per-point path costs one storage round-trip per lookup:
- StoredBitSlice::get_bits_batch: dedupe the containing u64 elements and
fetch them through one read_batch pass.
- DiskMappingReader::lookup_batch: group e2i keys by run block, read each
unique block once, binary-search all its keys.
- DiskMappingReader::external_ids_batch: schedule all i2e data slots plus
the deduplicated is_uuid bitmap bytes in one pass.
- DiskMappingsSource::{points_deleted_batch,resolve_internal_batch,
resolve_external_batch}: shared batched resolution for both disk trackers.
- IdTrackerRead::{external_ids_batch,internal_versions_batch}: trait-level
batch methods with loop defaults for in-RAM trackers; overridden (along
with resolve_external_ids) in DiskIdTracker and ReadOnlyDiskIdTracker,
and forwarded through both tracker enums so the overrides dispatch.
Wire the batch operations into the read paths: search result processing
(external ids + versions), scroll filtered_read_by_index (chunked), and
HasId condition conversion/cardinality estimation. retrieve() already goes
through resolve_external_ids and picks up the batched dispatch.
Batch read-by-id keeps the read-only tracker's laziness: no full
deleted-set materialization (covered by test).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Rework batch id-tracker interfaces to iterator inputs and callback output
Review follow-up: the slice-in/Vecs-out shapes forced intermediate collects
at every call site, costing the in-RAM trackers (whose defaults are plain
loops) allocations the pre-batch code never had.
- resolve_external_ids now takes `impl IntoIterator<Item = PointIdType>`
and delivers `(id, offset)` pairs through a callback, in input order:
has_id conditions stream the id set straight into the offsets set/vec,
retrieve fills its parallel vectors directly.
- external_ids_batch / internal_versions_batch take iterator inputs but
keep `Vec<Option<_>>` outputs (search result processing needs one aligned
slot per input); search drops its two offset collects, scroll passes the
itertools chunk directly.
- In-RAM trait defaults stream with no intermediate allocation; the disk
overrides buffer the input once internally, where the block-grouped
reads need the whole batch anyway.
Also document on the writable DiskIdTracker why points_deleted_batch and
internal_versions_batch are deliberately not overridden there: deleted and
versions are RAM-resident by design, so the default loop has no IO to
pipeline; only the read-only tracker keeps them on disk.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove accidentally committed local example
inspect_mutable_id_tracker.rs is a local debugging harness that was swept
into the previous commit by a directory-level git add; it is not part of
the batch-interface change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add PointIdBatch and thread it through resolve_external_ids
Replace the `impl IntoIterator` / `&[PointIdType]` inputs on the
id-tracker external->internal resolve path with a small `PointIdBatch`
trait (Copy + num_ids + iter_ids), implemented for `&[PointIdType]` and
`&AHashSet<PointIdType>`. Being re-iterable and length-known lets the
disk trackers batch the reads without first collecting the input.
On the disk path this drops several intermediate allocations:
- no input `Vec<PointIdType>` collect in resolve_external_ids_batch;
- lookup_batch returns compact `(id, offset)` pairs (id rebuilt from
is_uuid + key) instead of a size-N `Option` reorder buffer;
- lookup_batch no longer groups keys by block up front: a block is one
~16 KiB DiskCache block, so same-block reads are deduplicated by the
cache (piggybacked in flight, a plain hit once resident);
- resolve_internal_batch filters deleted pairs in place and feeds
points_deleted_batch an offset iterator, dropping the offsets Vec.
Delivery is no longer input-ordered; the id travels with the offset into
the callback, so callers that need positions still have them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Stream disk id resolve through a callback; prefetch deleted flags whole
Follow-up to the PointIdBatch rework, removing the last buffers from the
disk external->internal resolve path and dropping the redundant deleted
batching.
- `lookup_batch` and `resolve_internal_batch` now hand each resolved
`(id, offset)` to a callback as its block completes instead of
returning a `Vec`; `resolve_external_ids_batch` just forwards the
caller's callback and logs on error. No buffer is built on the path.
- Drop `points_deleted_batch`: the deleted set is resident (writable
tracker) or prefetched whole (read-only), so a per-point `point_deleted`
check is as cheap as a pipelined read would be. `resolve_internal_batch`
captures the first check error out of band and surfaces it after the
pass.
- Prefetch the deleted flags whole (`Populate::PreferBackground`) in both
`try_preopen` and `try_open`, via a shared `deleted_open_options`, so
the per-point checks stay off remote storage.
- `PointIdBatch` loses the now-unused `num_ids`; only `iter_ids` remains.
Delivery is fully streamed, so on a mid-pass storage error the callbacks
for already-resolved live points have fired before the error surfaces —
acceptable at the best-effort `IdTrackerRead` boundary.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Surface id resolve errors to the trait; drop redundant batch wrapper
`IdTrackerRead::resolve_external_ids` now returns `OperationResult<()>`
instead of swallowing storage errors. The disk path previously logged
and dropped a `resolve_internal_batch` error at this boundary; now it
propagates so callers see failed id resolution.
- The in-RAM default returns `Ok(())`; both enum forwarders and both disk
overrides propagate. The three call sites (retrieve, HasId cardinality,
HasId checker) are already in `OperationResult` functions, so they just
gain a `?`.
- Drop the `resolve_external_ids_batch` free function: with its
log-and-swallow gone it only forwarded to `resolve_internal_batch`, so
the two overrides call that directly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Stream internal_versions_batch through a callback
Give internal_versions_batch the same treatment as resolve_external_ids:
it takes a callback and returns OperationResult<()> instead of a
Vec<Option<..>>, so the disk override no longer allocates. It walks the
input lazily (no collect), tags each pipelined read with its internal_id,
and streams (internal_id, version) to the callback as reads complete;
out-of-range offsets are skipped and a storage error propagates instead
of being logged and swallowed.
The in-RAM default and both enum forwarders match. process_search_result
now builds a small internal_id -> version map from the callback and looks
up by scored offset (handling duplicate offsets naturally), keeping the
same missing-version-is-an-error behaviour.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Stream external_ids_batch through a callback; drop chunked scroll batching
Align external_ids_batch with internal_versions_batch: iterator input,
(offset, id) callback output, no internal buffering, storage errors
propagated. The deleted filter runs inside the lazy range iterator, so
each tracker implements the override directly against its own deleted
source (resident bitvec vs prefetched on-disk file) and the redundant
DiskMappingsSource::resolve_external_batch pass-through is removed,
along with the now-unused log_lookup_err_batch.
filtered_read_by_index feeds the whole candidate iterator into one
batch pass and lets the IO layer pipeline the reads; the limit case
keeps the top-smallest ids in a bounded priority queue. This retires
ID_TRACKER_BATCH_SIZE. Search result processing pairs external ids by
internal id, mirroring the versions map.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove accidentally committed local example, again
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop unused StoredBitSlice::get_bits_batch
Its only caller, the read-only tracker's points_deleted_batch override,
was removed when the deleted flags became whole-file prefetched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Write deleted_open_options fields explicitly, comment the why
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Simplify read-only external_ids_batch error handling via try_filter
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add infallible point_deleted helper to the writable disk tracker
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop PointIdBatch in favor of impl IntoIterator<Item = PointIdType>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Andrey Vasnetsov
|
446d140c2d |
Slice filtering condition: sliced scroll / deterministic sampling (#9899)
* feat: slice filtering condition for sliced scroll and deterministic sampling
Add a `slice` filter condition selecting points where
`stable_hash(point_id) % total == index`. The hash is SipHash-2-4 with a
zero key over canonical id bytes (8 LE bytes for numeric ids, 16 RFC 4122
bytes for UUIDs) — a frozen public contract, independent of the internal
resharding ring hash, reproducible by clients to predict membership.
For a fixed `total`, slices are disjoint and cover all points, enabling
parallel scroll streams (ES sliced-scroll style) and reproducible sampling
that composes with any other filter condition.
- REST: `{"slice": {"total": N, "index": R}}`; gRPC: `SliceCondition` in
the condition oneof (tag 8)
- Evaluated per point via id_tracker external-id lookup; no payload index
needed; cardinality estimated as `points / total` with no primary clause
- `total >= 1` enforced by NonZeroU32 at parse time, `index < total` by
validation in both REST and gRPC paths
- Hash contract locked by test vectors independently reproduced with a
reference SipHash-2-4 implementation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* tests: minimal OpenAPI test for slice filter condition
Scrolls all slices of a fixed total over numeric + UUID ids asserting
disjointness and full coverage, checks must_not inversion, and pins the
two rejection paths (422 for index >= total, 400 for total = 0). Requests
and responses are validated against the regenerated OpenAPI spec by the
test harness.
Note: the spec cannot itself reject total = 0 client-side — the Condition
anyOf falls through to the permissive Filter schema, as with any invalid
condition — so rejection is asserted via the server response.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Luis Cossío
|
956e5dde29 |
make VectorStorageRead::read_vector_bytes mandatory (#9893)
Additionally: - homogenize enum dispatch - impl batch multivector read bytes |
||
|
xzfc
|
9d4b4b91a0 |
Optimize debug build time (less generics) (#9895)
* Non-generic TransformInto::transform I think dyn is fine here because it is executed once per query, not in a hot loop. * Query::score_by impls: manual loops Iterator chains (`.iter().map(..).sum()`) produce a lot work for the compiler to do. * QuantizedCustomQueryScorer: move generics from struct to `new` method |
||
|
Luis Cossío
|
5c269b9525 |
Misc nits (#9894)
* use `WithVector::is_enabled` * suppress unused var lint * fix non linux "useless mut" lint |
||
|
Arnaud Gourlay
|
c196d2eb1a |
Benches: use SmallRng instead of ChaCha12-based generators (#9887)
* Benches: use SmallRng instead of ChaCha12-based generators All benchmarks used StdRng or rand::rng() (ThreadRng), both backed by the ChaCha12 block cipher in rand 0.10. Benchmarks do not need crypto-strength randomness, and several draw random values inside the timed closure, so cipher work was included in the measurement itself. Switch every bench target to SmallRng (Xoshiro256++), and key the HNSW graph cache and sparse index cache by RNG algorithm so stale caches built from the old generator are not reused against newly generated vectors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Benches: replace free-function rand::random with local SmallRng Addresses review: rand::random draws from the thread RNG (ChaCha12), including inside the timed loop of the pq score benchmark. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Jojii
|
c03ee39456 | Less generics in retrieve raw (#9874) | ||
|
Arnaud Gourlay
|
247dbbbb07 |
Return sorted Vec from SegmentEntry::vector_names (#9870)
The HashSet bought nothing: names are unique by construction (map keys) and all callers only iterate. A sorted Vec skips the hashing and set allocation, and makes the iteration order deterministic. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
0eedad2e38 |
DiskIdTracker: RAM-resident is_uuid stored-bitmask sidecar + module split (#9878)
* DiskIdTracker: RAM-resident is_uuid stored-bitmask sidecar + module split Move the is_uuid flags out of the i2e file into a separate id_tracker.is_uuid file in the compact StoredBitmask format (#9871), loaded whole into RAM as a RoaringBitmap on open and prefetched in preopen, so slot decoding never reads the flag from disk. Bump the on-disk format version to 2 (DiskIdTracker is unreleased; no migration). Add StoredBitmask::read_ones() in common to normalize any stored encoding into a bitmap of set positions, with logical_len validated against the u32 position space at open. Restructure disk_id_tracker: read_only.rs becomes read_only/{mod, lifecycle,live_reload,id_tracker_read} mirroring the immutable tracker, and reader.rs becomes reader/{mod,lifecycle,lookup,iter}. Also unbox the DiskMappingsRef iterators (impl Iterator instead of Box<dyn>), and make IdTrackerRead::iter_internal_versions fallible so the read-only disk tracker propagates storage errors instead of silently truncating; its implementation now reads the versions file in one pass (cleanup-on-open drains it anyway). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Split disk_id_tracker mod.rs into lifecycle / read / write files Mirror the read_only/ layout: mod.rs keeps the struct and resident-RAM helpers, lifecycle.rs the build/open paths, id_tracker_read.rs the DiskMappingsSource + IdTrackerRead impls, id_tracker.rs the mutable IdTracker impl. Code moved verbatim; only imports and a field doc touched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Tighten disk_id_tracker docstrings around guarantees State contracts (residency, laziness, error semantics, atomicity) instead of narrating which structures hold or use what. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Luis Cossío
|
a9a8e7669b |
[edge] Actually use batched vector reads (#9855)
* override default impls * Fix clippy needless_borrow in batched dense read ops Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Andrey Vasnetsov
|
a6ea303e4b |
Compact stored bitmask for on-disk field index deleted masks (#9871)
* Compact stored bitmask for on-disk field index deleted masks Add StoredBitmask: a compact persisted bitmask written and read as a whole. The payload is a roaring bitmap of whichever bit value is the minority (mostly-0 and mostly-1 masks both stay tiny), falling back to raw dense bits when roaring would not be smaller, so the file is never larger than the dense representation. Files are replaced atomically via UniversalWriteFileOps::atomic_save; there is no in-place mutation. Use it for the write-once "no values" masks of the on-disk numeric, geo, map and full-text indexes, replacing the raw dense bitslice files sized at point_count/8 bytes regardless of content. Writing the new format is gated by the compact_bitmask feature flag (default off; enabled by `all` and `serverless_compatible`). Reading is format-agnostic regardless of the flag: the compact deleted_mask.bin is tried first, then the index-specific legacy file, so old segments stay readable and flag-off builds produce byte-identical legacy files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Split save_bitmask into named helpers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Regenerate OpenAPI spec for compact_bitmask feature flag Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review findings on stored bitmask - Avoid u64 overflow in the payload bound check when opening a mask with a corrupted payload_len. - Reject roaring payload positions beyond logical_len at read time, enforcing the BitmaskContent range contract for corrupted files. - Remove the opposite-format mask file after a successful save, so a rebuild with a flipped compact_bitmask flag can't leave a stale compact file shadowing the fresh legacy one (or an orphaned legacy file next to a compact one). - Make the compact-open numeric test tolerate builds that already wrote the compact format. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
561aa419f7 |
Add recoverable OutOfAppendableCapacity operation error (#9860)
Preparation for capping appendable segment growth in the update path (#9158): a dedicated error for "all appendable segments reached max_segment_size", so the update pipeline can recognize it and provision a fresh appendable segment before re-applying the operation. Maps to a transient service error at the collection level: if it ever escapes recovery, failed-operation recovery re-applies the operation. Part 1/5 of the appendable segment overflow fix. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Jojii
|
6d6e88aff3 | IO uring for TQDT (#9852) | ||
|
Arnaud Gourlay
|
6b3f8e876e |
Use AHashMap for JsonPath-keyed field index lookups (#9854)
Profiling showed SipHash over JsonPath keys (RandomState::hash_one) dominating field index lookups in StructPayloadIndexReadView::estimate_field_condition: the map holds only a handful of indexed fields, but it is queried per condition per filter evaluation per query, and hashing the path is the entire lookup cost. Switch the field index lookup maps (IndexesMap, ReadOnlyIndexesMap, the read view borrow, and the query_checker / condition_converter / value_retriever signatures that receive them) from std HashMap to AHashMap, already used throughout the crate. Serialized and API-level schema maps keep std HashMap: they are cold and their type leaks into conversions. conditional_search bench (criterion, 100 samples, vs saved baseline): - struct-conditional-search-query-points: 583.7 us -> 443.5 us (-23.1% median, p ~ 0) - struct-conditional-search-context-check: 73.2 us -> 64.8 us (-12.6% median, p ~ 0) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Luis Cossío
|
e147d2f8e4 |
[edge] Add readonly ram sparse index (#9841)
* Add Ram variant, loads from vector storage * impl live reload * preopen sparse storage when using ram index * Batch storage reads when building sparse RAM index Rebuild `build_ram_index` around a single `read_vectors::<Sequential>` pass instead of a per-point `get_vector_opt` loop, so Gridstore coalesces the IO into block reads instead of a round-trip per point. Benefits both the read-only mutable-RAM rebuild and the writable `SparseVectorIndex::plan` build path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
qdrant-cloud-bot
|
94cc5eeec9 |
Ignore slow Windows CI tests for turbo multi model and quantization (#9850)
Skip the three slowest Windows rust-tests (>60s each): turbo multi random-ops model tests and the turbo Manhattan HNSW quantization case. Use rstest test_attr for the parametrized case since a top-level ignore does not apply to all generated tests. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Tim Visée
|
71e6c70458 |
Universal IO: append to file (#9720)
* universal_io: add UniversalAppend for atomic single-operation appends Growing a file previously took a separate set_len + reopen + write dance (bypassing universal_io and leaving a zero-filled window on crash), and there was no way to express appends for backends without random-offset writes. UniversalAppend::append grows the file by writing at the current end of file in one atomic grow+write operation and returns the offset at which the data landed; append_batch lands multiple buffers contiguously in as few operations as the backend allows. flusher() moves from UniversalWrite into a new UniversalFlush supertrait so append-only handles can require it without duplicating the method. Local backends, both single-syscall: - MmapFile appends through a dedicated O_APPEND fd (every write(2) / writev(2) is an atomic grow+write at EOF), then remaps via reopen(). Its flusher also fdatasyncs after appends, since msync alone does not persist file-size metadata. - IoUringFile appends via pwritev2(RWF_APPEND). O_APPEND is not an option there: on Linux, pwrite on an O_APPEND fd appends regardless of the given offset, which would break positioned writes on clones sharing the fd. Concurrent appenders are out of contract (single logical writer); object-store backends surface the new AppendOffsetConflict error and recover via reopen() + retry. * io_bridge: add AsyncWrite/AsyncAppend and appendable BlobFile Add the write-side backend traits reserved next to AsyncRead: AsyncWrite (create/remove/save) powers a UniversalWriteFileOps impl on BlobFs (create-or-truncate put, delete, atomic whole-object save; directory ops are no-ops), and AsyncAppend — a single-request append where the offset must equal the current object size, acting as a compare-and-swap token — powers UniversalAppend on BlobFile. BlobFile caches the object size across appends (one HEAD for N appends; a missing object counts as empty so the first append creates it), concatenates batches into a single request, and drops the cache on reopen() — the documented recovery path after AppendOffsetConflict. Its flusher is a no-op: appends are durable once the backend acknowledges them. * io_bridge_object_store: native single-request S3 append object_store has no append support, so issue the PutObject + x-amz-write-offset-bytes request ourselves, reusing the store's credential chain (AmazonS3::credentials) and object_store's SigV4 AwsAuthorizer, which signs every header present on the request — no hand-rolled signing and no direct reqwest dependency. The offset doubles as a compare-and-swap token: a mismatch (400 InvalidWriteOffset, or 412 on some S3-compatibles) maps to AppendOffsetConflict. The write-offset append API exists on AWS S3 Express One Zone directory buckets and compatible stores (e.g. MinIO AiStor) — plain S3 Standard buckets reject it, and real Express zonal endpoints / session auth are not verified yet; MinIO-AiStor-compatible stores are the primary target for now. GCS and Azure sources simply do not implement AsyncAppend. ObjectStoreSource carries an AppendContext (HTTP client + object URL base + signing region) built per backend from its config, and gains a generic AsyncWrite impl (single-put create/save, delete). A test-only multi-request CAS emulation over InMemory exercises the BlobFile append stack hermetically; an end-to-end flow against a real append-capable store is gated behind S3_APPEND_INTEGRATION_TEST=1. * simple_disk_cache: write-through UniversalAppend for DiskCache Append to the remote (the single grow+write operation), then write the same bytes through into the local mirror so tail reads do not re-fetch what was just uploaded. LocalState::append_local keeps the fetched bitmap accurate: blocks fully covered by the appended range are marked fetched, and the pre-append partial tail block — which resize() drops because set_len zero-fills its gap — is re-marked only when its prefix was already fetched. If the mirror turns out stale (the remote grew behind our back), append falls back to resize-only and lazy fetches heal the gap on the next read. Writeable opens are now allowed on DiskCacheFs solely to enable append; DiskCache still never implements UniversalWrite. The writeable flag propagates to the remote handle, which is opened buffered instead of O_DIRECT: appends write through the page cache, which O_DIRECT reads on the same fd would fight (and IoUringFile rejects appends on prevent_caching handles). The remote-immutability docs are relaxed to append-only with an immutable prefix, matching what reopen() already assumed. Includes a full-stack composition test: DiskCache write-through over BlobFile offset tracking over an in-memory object store. * io_bridge_object_store: build the append HTTP client lazily Opening a source from an AwsConfig eagerly built the reqwest client (TLS setup, connection pool) even when append was never used. Keep the AppendContext construction to pure config (allow_http flag, object URL base, signing region) and build the client on first append instead, cached in an Arc<OnceLock> shared across clones of the source — and thus across the file handles opened from it. Sources that never append now pay nothing; client-construction errors surface on the first append instead of at open. * universal_io: test that append grows the regular file on disk The conformance suite reads appended bytes back through universal-io handles; also assert the underlying regular file itself — created outside universal_io, verified with plain fs reads — for both local backends. * Mention why we use custom HTTP client, object_store crate has no support * io_bridge_object_store: reject appends unconfirmed by the size header A store without write-offset support may accept the signed PutObject as a plain put — replacing the object with just the appended bytes — and return 2xx (community MinIO did exactly this before 2025-05, commit minio/minio@6d18dba9). The old success path fabricated the new length when x-amz-object-size was missing, so the destruction stayed invisible while every subsequent append repeated it. Require the x-amz-object-size response header (returned by AWS and MinIO AiStor appends) for any append at offset > 0 and fail loudly without it. Offset-0 appends are equivalent to a whole-object write, so they remain valid either way — a misconfigured store now fails on the second append instead of never. * io_bridge_object_store: honor endpoint/region env vars for appends With AwsCredentials::Default the store is built via AmazonS3Builder::from_env, which honors AWS_ENDPOINT_URL_S3, AWS_ENDPOINT_URL, AWS_ENDPOINT, AWS_REGION and AWS_DEFAULT_REGION — but append_context derived the append URL and SigV4 region only from the typed config fields. An env-configured deployment would read from one host while signing and sending appends to https://{bucket}.s3.us-east-1.amazonaws.com. Resolve the append endpoint and region the same way build_store does: explicit config first, then (default credential chain only) the same environment variables, with AWS_ENDPOINT_URL_S3 taking precedence as in from_env. The resolution is a pure function over an injected env lookup so the test does not touch process-global environment state. * simple_disk_cache: delegate the append flusher to the remote DiskCache's UniversalFlush impl was an unconditional no-op, justified by object-store appends being durable on acknowledgement — but the impl is generic over any appendable remote, and for local remotes (MmapFile, IoUringFile, exactly the compositions the tests instantiate) that silently dropped the fdatasync the UniversalAppend contract requires: append, flush Ok, power loss, appended bytes gone. Delegate to the remote's flusher once the cache is materialized: local remotes get their sync, object-store flushers remain no-ops, and a never-materialized cache has made no appends so a no-op stays correct. * io_bridge_object_store: retry transient append failures The append RPC was a single unretried HTTP attempt, while every other request in this stack goes through object_store's retry layer — a routine transient 503 SlowDown or connection reset failed the append hard where a concurrent read would have silently recovered. Retry connection errors, 5xx and 429 up to three attempts with a short linear backoff, re-signing per attempt (the SigV4 signature embeds the request date). Retrying is safe because the write offset is a compare-and-swap; the one ambiguity — an attempt that landed but whose acknowledgement was lost — surfaces as a write-offset conflict on the retry, which is reconciled with a HEAD: under the single-writer contract, an object size of exactly offset + data_len proves the tail is ours, so the append reports success instead of a spurious conflict (whose reopen-and-retry recovery would duplicate the record). * universal_io: forward TypedStorage::flusher for any UniversalFlush The flusher forwarding lived in TypedStorage's S: UniversalWrite impl block, so append-only storages (DiskCache, BlobFile — UniversalAppend + UniversalFlush but not UniversalWrite) offered append through the wrapper while the durability flusher the append contract mandates was unreachable without going through .inner. Move it to an S: UniversalFlush block: UniversalWrite implies UniversalFlush, so existing callers resolve unchanged, and duplicating the method instead would have hit E0592 on backends implementing both — the very ambiguity UniversalFlush was extracted to avoid. * io_bridge_object_store: surface unbuildable append requests as errors Request building could panic on two reachable paths: url accepts URIs the http crate rejects (IPv6 zone identifiers, URIs beyond u16::MAX bytes), and AppendContext::new is public so the object URL base is not guaranteed to be a base URL. Both expects become S3Config errors, so a configuration edge case fails the append instead of panicking the thread driving the bridge runtime. * simple_disk_cache: don't fail appends the remote already committed append_impl committed to the remote first and returned Err when the subsequent local-mirror update failed (e.g. ENOSPC on the cache volume) — indistinguishable from "nothing was appended", so a retrying caller would duplicate the record on the remote. The mirror is cache maintenance, not part of the append: on a failed write-through, log and degrade to bare growth so lazy fetches heal the unmarked blocks (safe — blocks are only marked fetched after their bytes landed). Only an unresizable mirror still surfaces an error, and the UniversalAppend contract now documents that an append Err does not guarantee nothing was appended: reopen() and re-check the length before retrying. * universal_io: bounds-check positioned io_uring writes against EOF The UniversalAppend contract states that UniversalWrite::write beyond the end-of-file fails and append is the only growth path — mmap enforces it, but IoUringFile's write/write_batch/write_multi were unchecked pwrites that silently extended the file with a zero-filled hole, inflating subsequent append offsets. Check every positioned write against the file length (fstat once per call), matching mmap's OutOfBounds semantics, and generalize the regression test to run on both local backends including the batched path. * io_bridge: reject appends on handles opened without writeable BlobFs::open dropped OpenOptions entirely, so a BlobFile opened with writeable: false still accepted appends — mmap and DiskCache enforce the writeable requirement, the blob backend silently didn't, and a stray append through a nominally read-only handle would mutate a shared object. Thread OpenOptions::writeable into BlobFile and reject appends with PermissionDenied when it is unset, mirroring the other backends. Directly-constructed handles (BlobFile::new/open, which take no OpenOptions) remain writeable. With every backend now enforcing the flag, the UniversalAppend contract drops its 'where the backend enforces open modes' hedge. * simple_disk_cache: answer empty appends from the mirror An empty append still went through remote.append_batch, so it returned the remote's live end-of-file — which can diverge from what this handle's len() and reads observe when the remote grew behind our back — while leaving the stale mirror unhealed (unlike a non-empty append in the same state, which resizes). Accept empty appends early: return the mirror length without touching the remote at all, keeping the answer consistent with the handle's own view. The trait contract now spells out that empty appends return the handle's view of the end of file without growth I/O. * universal_io: grow the mmap in place after appends Every mmap append ended in a full reopen(): an open+fstat+close by path just to learn the new length, and — on the non-Linux fallback, which rebuilds the mapping with the open-time populate flag — a re-population of the ENTIRE file per append, making appends O(file size) for handles opened with Populate::Blocking. Populating after an append is pointless anyway: we just touched the data we wrote. Extract the remap machinery into remap_to() (reopen() keeps its exact semantics, populate included) and add grow_mapping(): a stat-free grow that never re-populates. Appends learn the new length from a single fstat on the already-open O_APPEND fd — kept rather than trusting the mapping length, which is stale exactly in the externally-grown-remote scenario the disk cache heals through lazy fetches (the foreign-growth tests catch the difference). The mirror's resize() passes the length it just set_len'd, dropping its stat round-trip entirely. Per small append this is write+fstat+mremap, down from write+open+fstat+close+mremap, with no populate anywhere. * universal_io: share the mmap append fd across clones The flusher captured the per-clone append_file at flusher-creation time, so a flusher obtained from a sibling clone — or created before the handle's first append — msynced the shared mapping's data pages but skipped the fdatasync that persists the appended file size: a half-persist where a crash loses the acknowledged tail even though a flusher ran after the appends (writer thread + long-lived flush-worker clone is exactly the natural WAL shape). Store the fd in an Arc<OnceLock> shared by all clones and read it at flush time instead of capture time: any clone's append makes every handle's flusher sync the size metadata, whatever the clone/flusher creation order. Initialization races between clones keep exactly one fd. No hot-path cost: reads and positioned writes never touch the cell, and the append path pays one atomic load next to its syscalls. The interior mutability also lets append_fd take &self. * universal_io: document the clone remap hazard truthfully The remap SAFETY comment claimed moving is safe "since we are holding &mut self" — which says nothing about clones: they share the mapping but keep their own raw ptr/len copies, so after a moving (or, on non-Linux, replacing) remap a sibling clone's next read dereferences an unmapped address. The trait contract understated the same hazard as a concurrent-read constraint, while the UB persists after append returns. State the contract once on MmapFile (clones must reopen before reading after any growth; a stale clone read is undefined behavior, not a stale view), correct the SAFETY argument to rely on it explicitly, annotate the as_bytes unsafe blocks that depend on it, and sharpen the UniversalAppend contract bullet accordingly. Making clones structurally safe (resolving ptr/len through the shared Arc) is deliberately left as a separate change. * universal_io: share the vectored append machinery between backends The IOV_MAX-chunking / EINTR-retry / WriteZero / advance_slices loop existed twice — as local_file_ops::write_all_vectored (mmap) and inlined around pwritev2 in IoUringFile::append_slices — along with a verbatim collect/cast/filter-empties preamble in both append_batch impls. Two copies of subtle short-write handling introduced by one branch will diverge the first time only one of them gets a fix. Add an io::Write adapter whose write_vectored issues pwritev2(RWF_APPEND), letting the io_uring append delegate to the shared write_all_vectored, and hoist the slice collection into local_file_ops::collect_append_slices. IOV_MAX becomes private to the one function enforcing it. No behavior change; the existing conformance tests (including the beyond-IOV_MAX batch) cover both backends through the shared path. * io_bridge_object_store: add s3_express to the test config helper The AwsConfig struct gained the s3_express field; update the resolve-endpoint test helper accordingly. * universal_io: run the append conformance suite over the S3 stack Promote the backend-generic UniversalAppend battery (offsets, batches across IOV_MAX, empty appends, read-after-append, reopen visibility, flusher) from a private test into universal_io::conformance, exposed under the testing feature so backend crates can run the identical suite. mmap and io_uring keep running it as before; the object-store bridge now runs it too, over BlobFs/BlobFile with the in-memory offset-CAS append emulation — so local file system and S3 append behavior are asserted by the same test. The real write-offset RPC remains covered by the gated test_native_append_flow integration test. * Swap order * universal_io: disambiguate the io_uring crate import The import reorder dropped the leading `::`, making `io_uring` ambiguous with this very module (pulled into scope by the `use super::*` glob) and breaking the build. * io_bridge: don't materialize the mock object on rejected appends MutableMockSource::append called get_or_insert_with before validating the offset, so a rejected stale append against a missing object left an empty entry behind (exists() flipping true) — a fidelity gap versus the real backends, where a rejected append has no side effects. Check the offset against the current length first and only materialize the buffer on a match. * universal_io: disambiguate the io_uring crate import Restore the leading `::` on the io_uring crate import — without it the name is ambiguous with this very module, which the `use super::*` glob pulls into scope, and the crate fails to compile. Matches the sibling files (pool.rs, runtime.rs), which already import via `::io_uring`. * io_bridge_object_store: treat 404 under a nonzero append offset as a conflict A missing object while the handle expected a nonzero end-of-file is a stale view (the object was deleted behind our back) — the same situation as an offset mismatch, with the same reopen-and-retry recovery, and it is exactly what the in-memory emulation and the io_bridge mock already report. The RPC path mapped every 404 to NotFound instead, so the three implementations disagreed on the same logical case. Keep NotFound for offset-0 appends, where a 404 is a genuine missing-target error (e.g. a missing bucket) that retrying cannot heal. * Preallocate vector * universal_io: disallow appends through the disk cache Appends must go directly to the backing storage (mmap, io_uring, S3) — the disk cache is strictly read-only again. Remove DiskCache's UniversalAppend/UniversalFlush impls and the mirror write-through machinery (LocalState::append_local), reject writeable opens at DiskCacheFs::open, and drop the writeable/prevent_caching plumbing that existed solely for cached appends, restoring read-only remote handles. Attempting to append through the cache is now a compile-time error (the trait impl no longer exists), and opening a cached handle writeable is rejected at runtime, covered by a test in each backend variant. * universal_io: make append idempotent via caller-supplied offset Append now takes the byte offset where the data must land: append(offset, data) -> Result<()>. Every backend validates that the offset equals the current end of file before writing (mmap and io_uring fstat the fd, object stores validate server-side via x-amz-write-offset-bytes); on mismatch nothing is written and the append fails with AppendOffsetConflict. Retrying an already-landed append therefore conflicts instead of appending twice, and recovery is re-deriving the offset from len(). BlobFile no longer tracks the object length locally; the store's own offset check is the compare-and-swap. * Review remarks * Validate file length in S3 append response * Fix linting, we don't mind a large enum variant on index builder * universal_io: conformance-test stale-handle append conflict recovery Promote the two-handle conflict scenario from the in-memory BlobFile test into the backend-generic conformance battery: a second writeable handle grows the file, the stale handle's append conflicts cleanly (the offset check runs against the file, not the handle's view), and the contract's documented recovery — reopen, re-check the length, append at the real end — lands the data exactly once. Now exercised over mmap, io_uring, and the object-store stack instead of only the in-memory emulation. * io_bridge_object_store: stub-server tests for append response handling The native append's HTTP state machine was only exercised by the gated live-store integration test (S3_APPEND_INTEGRATION_TEST=1), so none of its branches ran in CI. Cover them hermetically against a minimal local HTTP stub — one connection per canned response, no new dependencies: - the signed write-offset PUT, and the new-size validation on success (matching, mismatching, unparseable, and absent size headers — the absent case at offset zero and past it); - conflict mapping for 400 InvalidWriteOffset, 412, and 404 under a nonzero offset, with 404 at offset zero staying NotFound, and a 400 without the conflict code staying a plain error; - 429/5xx retries re-sending the same offset, giving up after MAX_ATTEMPTS, and the lost-acknowledgement reconciliation via HEAD (accepted when the object ends at offset + len, rejected otherwise); - the status + body excerpt on unexpected failures. * simple_disk_cache: statically assert the cache stays read-only Disallowing appends through the disk cache made them a compile-time error by removing the impls; pin that with assert_not_impl_any so the UniversalAppend/UniversalFlush/UniversalWrite impls cannot quietly return. Runtime rejection of writeable opens stays covered per backend variant. |
||
|
Arnaud Gourlay
|
0d2f5c7e85 | Miscellaneous cleanups (#9849) | ||
|
Aditya Painuli
|
72e5d9b42b |
Improve error message for non-string key in filter conditions (#9847)
* Improve error message for non-string 'key' in filter conditions * Reformat --------- Co-authored-by: timvisee <tim@visee.me> |
||
|
Jojii
|
40332d271d |
[TQDT] Roundtrip fix for clone_and_mutate_point (#9761)
* Fix TQDT roundtrip issue for `clone_and_mutate_point` * Use TinyMap * Fix doc string * Rebase docstring fixes * [TQDT] Reduce allocations (#9833) * Reduce allocations * Fix edge |
||
|
Luis Cossío
|
da63a3cdab | remove expect (#9839) | ||
|
Luis Cossío
|
6ac3c828d4 | exhaustive match on async-like backends (#9837) | ||
|
ARYAN SINGH
|
631d9a2045 |
align scalar f16 metrics precision with SIMD thresholds (#9788) (#9811)
* - [AI] fix: align scalar f16 metrics precision with SIMD thresholds (#9788) * remove generated tests |
||
|
Andrey Vasnetsov
|
c3ff8ee44b |
Add optional block-index sidecar to accelerate on-disk binary searches (#9810)
* Add optional block-index sidecar to accelerate on-disk binary searches Binary search directly over an unpopulated mmap costs O(log n) random reads scattered across the whole file — one page fault (or remote range read) per probe on high-latency storage. Introduce SortedBlockIndex: an optional sidecar file storing the first element of every 16KiB block of a sorted on-disk array, read whole into RAM at open. A lookup becomes an in-RAM partition_point over the block firsts plus a single contiguous read of one block, bisected in RAM. Wire it into the three remaining scatter-probe sites: - on-disk numeric index: binary_search_pairs over data.bin - on-disk geo index: counts_of_hash over counts_per_hash.bin - on-disk geo index: all_points start boundary over points_map.bin The sidecar is backward and forward compatible: absent (old segments) or failing validation, readers fall back to the existing plain binary search; old code simply ignores the extra file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Validate block-index sidecar snapshot recovery and preopen coverage - snapshot round-trip test (Regular + Streamable): the sidecars written by the on-disk numeric and geo indexes are collected via files() into the snapshot tar and restored byte-for-byte, with correct query results on the reloaded segment - preopen tests for both indexes: after schedule_prefetch, open loads the sidecar from the CachedFs prefetch pool even when the file is unlinked from disk; an absent sidecar neither fails preopen nor open (fallback) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Yash Singh
|
7c2a045574 | fix(segment): count trailing deleted offset on mmap sparse reload (#9798) | ||
|
Andrey Vasnetsov
|
1d4d6f02da |
Per-query IDF corpus for sparse vector search (#9661)
* Add per-query IDF corpus for sparse vector search
Let the caller choose, per query, which population sparse IDF statistics
are computed over. `params.idf` is either `"global"` (default, unchanged
behavior) or `{"corpus": <filter>}`, where the corpus filter is
independent of - and usually broader than - the retrieval filter.
Decoupling the two keeps the score scale stable when the retrieval
filter tightens: term importance is measured against a population the
user names, not against whatever subset the filter happens to select.
Design decisions:
- Corpus grammar is restricted to a conjunction (`must`) of `match`
conditions on payload fields; loosening later is backward compatible.
- Strict mode validates the corpus filter like a read filter
(unindexed fields rejected).
- `idf` on a vector without the IDF modifier is a validation error,
never silently ignored.
- An empty corpus yields degenerate but corpus-scoped scores (smoothed
IDF over N=0), never a fallback to global statistics - in multi-tenant
collections a fallback would leak term statistics across tenants.
Implementation:
- QueryContext IDF stats are keyed by corpus, so one batch can mix
requests with different corpora.
- Statistics come from the sparse index: df(term) is counted over the
query terms' posting lists only, never by scanning stored vectors.
Small corpora (under ~1/32 of the segment, by cardinality estimate)
are kept as a sorted id list galloping through posting lists via
skip_to; large ones as a dense membership mask filled streaming from
the filtered-points iterator. A misestimated small corpus degrades
into the mask.
- Exposed uniformly: REST (`params.idf`), gRPC (`IdfParams` message),
edge python bindings; OpenAPI schema regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply rustfmt
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix clippy manual_is_multiple_of in sparse IDF corpus test.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Allow any filter as IDF corpus
Drop the must+match grammar restriction on the corpus filter. A
restriction enforced only as a validation step over the full Filter
type buys nothing; if a narrower corpus syntax is ever wanted, it
should be a dedicated API-level type instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix build: add memory field to SparseIndexConfig in idf corpus test
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
Arnaud Gourlay
|
dd84044a3e |
refactor: replace StructPayloadIndex::open bool flags with StorageType and IndexLoadMode (#9754)
StructPayloadIndex::open took two adjacent bools (is_appendable, create)
and call sites passed every literal combination: (true, true),
(true, false) and (false, true) all exist. A transposed pair compiles
and silently yields e.g. non-appendable + create instead of
appendable + load-only.
The target enum already existed: open immediately converted the bool
into the private StorageType { Appendable, NonAppendable }, so the bool
survived only at the API boundary, exactly where the swap hazard lives.
Make StorageType public, take it directly, and introduce
IndexLoadMode { CreateIfMissing, LoadExisting } for the create flag.
create_segment had the same trailing create: bool with bare literals at
both callers, so its parameter is lifted to IndexLoadMode as well:
load_segment passes LoadExisting, build_segment passes CreateIfMissing.
No behavior change.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Andrey Vasnetsov
|
43e3d6ea8d |
[UIO] Request-specific load profile for read-only opens (#9797)
* Introduce request-specific LoadProfile with per-component placement A read-only shard opened for one known request (the serverless cold-start path) doesn't have to warm components the request will never touch. LoadProfile captures that from the request: warm components keep the persisted-config placement, everything else is parked cold. All placement decisions live in one place, so the memory placement of a whole segment under a profile is reviewable in one file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Thread LoadProfile through the read-only segment open ReadOnlySegment::open takes an optional profile; first_preopen and open_via resolve it into per-component populate overrides so the opens make the same placement decisions the prefetches did. Pinned components that materialize on open regardless (quantized RAM storage kinds, the immutable-RAM sparse index) and appendable components ignore the override; the HNSW graph and immutable payload indexes demote fully. Config reloads follow the new config alone and pass no override. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Open ReadOnlyEdgeShard under a request-derived load profile ReadOnlyEdgeShard::open takes an optional LoadProfile, applies it to every segment open and keeps it so segments discovered by a later refresh load with the same placement. ScrollRequestInternal and CoreSearchRequest gain load_profile() constructors, and edge-shard-query builds the request before the open and passes its profile (opt out with --no-load-profile). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Demote pinned quantized vectors and sparse index under a cold profile Within the immutable layout the quantized RAM and mmap loaders share the on-disk format — only how the data is brought into memory differs — and the immutable-RAM sparse index has the same lazy mmap open low-memory mode already downgrades to. So a cold populate override now demotes the effective placement itself (Memory::with_populate_override, shared with the HNSW residency mapping) instead of only skipping cache priming: a pinned quantized storage opens the mmap kind cold, and a pinned sparse index opens as Mmap, so neither reads its data on a cold start. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Add LoadProfile::merge for composite queries A composite query runs multiple core requests — e.g. a hybrid search runs one core search per vector, each with its own filter. Its profile is the union of its parts': merge extends the warm sets and ORs the payload-storage flag, so a component either part needs warm stays warm. The union is sound because every placement method is monotone in the warm sets (growing them only turns "park cold" into "keep configured placement"), so the merged profile dominates each input; and minimal, warming nothing no part asked for. Combine profiles with reduce, not fold: merge's identity element is the coldest profile (empty warm sets), the opposite of passing no profile at all — deliberately no empty()/Default constructor exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Adapt vanished-segment test to the profile-aware open signature The test landed on dev (#9777) after the load-profile signature change was written, so the rebase left its ReadOnlySegment::open calls without the new load_profile argument. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Defer vector index open entirely under a cold load profile A cold placement is not enough for the vector index on remote backends: GraphLinksView requires the whole links file as one contiguous slice, and the disk cache can only lend a borrowed slice once every block is locally present — so even a Cold HNSW open mirrors the entire links_compressed.bin (8.2 MB / 1.1 s per segment in the serverless cold-start trace), plus the unconditional graph.bin metadata read. The only way not to fetch the index is not to open it. LoadProfile::vector_index_placement is replaced by vector_index_deferred: a vector the request never scores now gets a DeferredVectorIndex — a new VectorIndexReadEnum variant holding the open arguments (an owned clone of the segment's raw backend, path, config, shared component handles) and a OnceLock. Nothing is opened or prefetched for it at segment open. Per-method policy of the deferred variant: - search, fill_idf_statistics and populate open the index on first use (with the cold placement the profile chose), so the profile contract holds: a request the profile did not predict still works, just pays the open then; - is_index reports true without opening (deferral only ever wraps a real HNSW or sparse index; plain opens no files and is never deferred); - telemetry, indexed_vector_count and sizes answer conservative defaults rather than trigger a remote fetch for a statistic. Tests: deleting the vector_index directory before an open under a scroll profile leaves open, filtered reads and payload reads working — proof that nothing of the index is read — while a search surfaces the missing files; and a segment opened under a scroll profile answers searches identically to an eagerly opened one via the transparent first-use open. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Make --vector optional in edge-shard-query: random query after open Omitting --vector on the search sub-command now searches with a random vector. The request is still built before the shard opens — the load profile only needs the vector name, not its values — with an empty placeholder; once the shard is open, fill_random_vector reads the dimension of the queried vector from the derived shard config and fills in uniform-random f32s (with a clear error if the named vector is not in the config). The vector is generated once, so live-reload iterations re-run the identical random query and the printed diffs stay meaningful. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Scope index deferral to the HNSW graph via a lazy OnceLock load Replace the DeferredVectorIndex wrapper (and the VectorIndexReadEnum:: Deferred variant) with deferral inside ReadOnlyHNSWIndex itself: the graph lives in a OnceLock (same first-wins arbitration as ReadOnlyRoaringFlags::bitmap) alongside the retained raw backend and residency, and loads on first use with a cold placement. The config read stays eager — one tiny, absence-tolerated file — so telemetry, is_on_disk and indexed_vector_count report real values where the Deferred arms answered with hard defaults. The sparse index needs no deferral: its mmap open reads lazily, with only small JSON metadata eager. A profile that never scores the vector now passes a cold placement override (LoadProfile:: vector_index_placement) into the eager open_sparse, which demotes ImmutableRam to the lazy Mmap open like low-memory mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Express HNSW graph deferral as a populate override, not a bool param Replace the `deferred: bool` on the read-only HNSW open/preopen (and the VectorIndexReadEnum pass-through) with the same `populate_override: Option<Populate>` every other component takes. A cold *override* defers the graph load — graph_deferred() mirrors the cold-override match of open_sparse — while a config-derived cold placement (or the low-memory clamp) keeps the eager load, since only a request-specific override carries the "never scored" prediction. With dense and sparse now consuming the same signal, LoadProfile::vector_index_deferred is gone: a single vector_index_placement() serves both index kinds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Serialize the deferred graph load via once_cell's get_or_try_init Loading outside the lock (std OnceLock's fallible init is still unstable) let a search burst on a deferred vector fetch the whole graph once per thread. Swap the cell for once_cell::sync::OnceCell: the fallible load runs inside the cell's lock, concurrent first users block on the one load, and a failed load leaves the cell empty so the next caller retries. Addresses https://github.com/qdrant/qdrant/pull/9797#discussion_r3573727836 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
bb7b995d14 |
[UIO] implement VectorIndexReadEnum::preopen (#9782)
* [UIO] implement VectorIndexReadEnum::preopen Schedule background prefetch of the HNSW graph (config, graph data, links) and sparse index (config, inverted index, version, indices tracker) files, wired into the segment's first_preopen for every dense and sparse vector. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Adapt index preopen to caller-controlled populate Now that CachedFs respects the caller's populate: prefetch Cached HNSW links with a background populate instead of the open's blocking one; populate the immutable-RAM sparse index data (read in full on open) while the mmap variant stays cold; apply the low-memory ImmutableRam downgrade in preopen_sparse since it now changes populate behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Decide sparse index preopen populate from the index type All readable sparse index variants share the compressed-mmap on-disk format, so the datatype dispatch in preopen_sparse selected nothing. Replace the per-TInvertedIndex preopen_ro trait machinery with one populate-parameterized preopen in the sparse crate: the effective index type alone decides whether the index data is warmed (immutable-RAM reads it in full) or parked cold (mmap reads lazily). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Derive sparse index preopen populate from the segment config The segment config's sparse_vector_data already carries the SparseIndexConfig, so preopen_sparse doesn't need to read the persisted copy at all: it derives populate from the segment-side index type and merely schedules the config file for open_sparse to consume — still a single fetch, without threading the parsed config through open_via. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Resolve index preopen placement via memory_placement The HNSW read-only open and preopen used the deprecated on_disk flag directly; resolve the graph residency like the writable open instead — memory parameter with on_disk fallback, clamped by low-memory mode, including the pinned placement. The sparse index preopen likewise derives its populate from the effective memory placement, so the cached mmap index is prefetched warm. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
1a61d0b09c |
Fix live-reload staleness of in-place-mutated files on caching backends (#9812)
* Mutate same-operation slots in place in append-only mode With append_only_mutations enabled, every mutation of an existing point clones it to a fresh internal id. Shard-level updates decompose one point write into several SegmentEntry steps (upsert_with_payload issues upsert_point plus set_full_payload/clear_payload), so a single upsert burned one slot per step, leaving a chain of immediately-dead clones. A slot whose version already equals op_num was written by an earlier step of the current operation. It cannot be durable yet — the segment write lock is held across the whole operation, so no flush (and no read-only follower) can have observed it, and versions flush last so a crash discards it and WAL replay re-applies the whole operation. Mutate such slots in place: one operation now allocates exactly one slot regardless of how many steps it decomposes into. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stamp payload storage version in write_point_parts overwrite_payload mutates payload storage, but the fused write never bumped version_tracker's payload version — the old CoW path did, via set_full_payload. The segment manifest would stamp payload files with a stale version, letting a partial snapshot skip payload storage that contains the moved point's row, so the restored id tracker would point at an offset with no payload. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Resync ReadOnlyRoaringFlags from disk on live-reload The flags file is preallocated to power-of-two capacity and mutated in place within its length, which the held handle's reopen() — an append-only-growth contract on caching backends — never picks up: bits the writer changes inside already-cached DiskCache blocks stayed stale forever on shard followers. Drop the `impl LiveReload for ReadOnlyRoaringFlags` altogether: this storage holds arbitrary flags with no notion of points, so a point-delta interface (deleted/new points) did not belong here — open never applied deleted points either. Replace it with an inherent live_reload(fs) that opens a fresh StoredBitSlice (a fresh open always mirrors the current remote bytes), resyncs the materialized bitmap from it, and swaps the handle. The on-disk flags are the sole source of truth. To make refresh-by-fresh-open safe while the old handle is still alive, every DiskCache open now mirrors into a uniquely-named local file (.{pid}-{counter} suffix) and removes it on drop. Mirrors were already truncated on every open (block validity is in-memory only), so the stable name carried no state. Also add trace logging for live-reload consumed mapping changes and pending deltas. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Resync immutable id tracker deleted bitmap via fresh handle on live-reload The immutable tracker's id_tracker.deleted file is a fixed-size bitmap whose bits the writer flips in place — its length never changes — so the held handle's reopen(), an append-only-growth contract on caching backends, never picked it up: the pre-deletion state cached at open was served forever and live-reload never reported deletions on shard followers. live_reload now takes the fs, opens a fresh StoredBitSlice (a fresh open always mirrors the current remote bytes), diffs it against the tracker's current state — mappings already reflect every previously reported deletion, so they are the baseline — and swaps the handle in. ReadOnlyIdTrackerEnum and the segment-level reload pass the fs through; the appendable and disk-resident variants keep their no-arg reloads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Resync disk id tracker deleted bitmap via fresh handle on live-reload Same staleness as the immutable tracker: the deleted file is a fixed-size bitmap mutated in place, so reopen() — append-only-growth on caching backends — served the pre-deletion state forever. live_reload now takes the fs, opens a fresh StoredBitSlice, and swaps it into deleted_file, so the per-point get_bit lookups read fresh state from then on too. The deleted_full take/diff/set baseline logic is unchanged. The enum's DiskResident arm passes the fs through. The regression test now covers both trackers over DiskCacheFs; the disk leg was verified to fail under the old reopen-based behavior. The disk tracker's versions file staleness is a separate, still-open case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Introduce header-stateless ReadOnlyTracker for gridstore live-reload The gridstore tracker file is preallocated and mutated in place (header rewrites, slot updates), which the reader's held handle never picked up on caching backends: Tracker::live_reload read the header through the stale handle — before reopen(), and reopen() would not have refreshed cached blocks anyway — concluded "no new pointers", and never reloaded pages. New points' payloads read as empty on shard followers. Replace the reader's tracker with a dedicated ReadOnlyTracker that holds nothing but the storage handle: reads don't need header state (slot addressing is positional, unwritten slots read as None in the zero-initialized file) and readers have no pending-updates buffer. Its live_reload opens a fresh handle and swaps it in. Tracker::live_reload is deleted. A new TrackerRead trait (max_point_offset/get/iter) is implemented by both the writable Tracker and ReadOnlyTracker, and GridstoreView is generic over it, so writer and reader share the read logic. max_point_offset reads the stored header count as plain data through the current handle (fresh as of the last reload) rather than deriving it from slot capacity: sparse vector storage builds total_vector_count from it, and the capacity of the preallocated file (1MB -> 65536 slots) would inflate every follower segment's sparse count. The method is now fallible; consumers propagate the error. GridstoreReader::live_reload refreshes tracker and pages unconditionally — the tracker carries no reliable cheap change signal. Making this incremental again is a deferred follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Re-open last held chunk on chunked-vectors live-reload Chunk files are preallocated to full size, so appended vectors are in-place writes within the existing file length. On caching backends a held handle keeps serving blocks fetched earlier — and a block fetched near the old tail extends into then-unwritten space — so vectors appended into that block read back as stale bytes after a reload. Mirror Pages::live_reload: on a len change (the status file is read through a fresh open every reload, so it stays a reliable gate), drop and re-open the last held chunk — the only one that can have gained vectors — alongside adopting newly created chunk files. Earlier, fully-filled chunks keep their handles; their contents cannot change. Covers dense, multi-dense, and quantized-chunked storages, which all delegate to ChunkedVectorsRead::live_reload. Avoiding the whole-chunk refetch per reload is a deferred follow-up, together with the analogous gridstore pages/tracker case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix lint: fs_err::create_dir_all in tests, drop unnecessary cast CI clippy runs with --all-targets and disallows std::fs methods in favor of fs_err; the new live-reload regression tests used std::fs::create_dir_all directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Tombstone-only deletes on non-appendable segments in append-only mode The tombstone-only delete path existed but was gated on is_append_only(), which requires the segment to be appendable. Deletes landing on non-appendable segments — notably the CoW update deleting the point's old copy — still went through delete_point_internal, which clears the payload row in place before the id-tracker drop. Clearing the payload destroys committed state of an offset that stays visible to live-reload followers until the drop is flushed: a follower refreshing in that window resolves the point through its (unchanged) id-tracker view and reads an empty payload — observed as a one-refresh {} payload flicker on a point update. Writer-side flush ordering cannot close the window, since the follower samples the id tracker and the payload storage at different instants; the versions commit protocol protects inserts exactly because the marker is read first, and deletes have no marker. This is the same failure class for which vector-deletion propagation was disabled (see the comment in delete_point_internal). Gate deletes on the new is_append_only_delete() — append_only_mutations alone, no appendability requirement: tombstoning needs nothing from the segment but the id tracker. Normal deployments keep eager payload clearing and prompt space reuse; clone-based mutations keep requiring appendability via is_append_only(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: chunked vectors reload --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Daniel Boros <dancixx@gmail.com> |
||
|
Andrey Vasnetsov
|
3a9bff3e5b |
[UIO] implement ReadOnlyQuantizedVectors::preopen (#9781)
* [UIO] implement ReadOnlyQuantizedVectors::preopen Schedule background prefetch of the quantization config, per-method metadata, quantized data and multivector offsets, wired into the segment's first_preopen for every dense vector with quantization configured. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * additional stage for quantized config preopen * better quantized preopening * Partially populate the first quantized vector in preopen The load reads the first vector off quantized.data (and off the first chunk in the chunked layout) to validate the stored vector size — on a cold open that was a round-trip. Use Populate::Partial (#9769) to prefetch exactly that prefix; the exact size comes from the quantized config the preopen already read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Restore the layout-probing quantized preopen Recovers the pre-rewrite design: preopen never reads the quantization config — it probes both layout candidates against the listing snapshot (a segment only contains the layout its real config selects), derives warmth from the segment-side quantization config (placement resolution incl. cached and low-memory), iterates storage types exhaustively for layout coverage, and schedules the config for open to parse once off the parked handle — no threading, no second preopen stage. Keeps the first-vector partial populate, with the size now derived from the segment config via construct_vector_parameters, and keeps VectorStorageType::is_pinned. Restores the full preopen test matrix and OneshotFile::preopen. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Luis Cossío <luis.cossio@outlook.com> |
||
|
Daniel Boros
|
f982ae63fc |
feat: read-only edge grouping/matrix search + object-storage read path (#9691)
* feat: read-only edge grouping/matrix search + object-storage read path Add query_groups (group by a payload field) and search_matrix (single-shard distance matrix over a random sample) to the read-only edge shard's EdgeShardRead API, in new grouping and matrix modules plus edge test helpers. Make the object-storage read path available outside tests: drop the #[cfg(test)] gate on the BlobFile UniversalReadExt impl and move io_bridge_object_store/object_store to segment's normal dependencies, so a ReadOnlyEdgeShard can serve segments read from S3. * Share group-by building blocks between server and edge Move GroupsAggregator, group candidate query shaping (is-empty filter, group_by payload selector, prefetch limit scaling) and result-order derivation into shard::grouping / shard::query, so the collection and edge grouping implementations cannot silently diverge. Edge grouping now handles multi-valued group keys, u64 keys, wildcard group_by paths, prefetch limits and score-ordered groups the same way as the server. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Drive group-by through a shared sans-IO state machine Extract the multi-request collect/fill loop into shard::grouping::GroupByDriver: next_request() yields shaped backend queries, add_points() advances the state, distill() returns the groups. Query execution stays with the caller, so the async server path and the sync edge path drive the same machine, and the request shaping helpers become private to shard::grouping. Edge now uses the same request budget (5 collect + 5 fill requests) and per-request candidates limit (groups * group_size, computed inside the driver) as the server, replacing its single 4x-oversampled request. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
273bbe1729 |
Fix use-after-free in u8-quantized vector reads on owning storages (#9801)
* Fix use-after-free in u8-quantized vector reads on owning storages EncodedVectorsU8::get_vec_ptr extracted a raw pointer from the buffer returned by EncodedStorage::get_vector_data and dropped the buffer before the pointer was dereferenced. For storages returning Cow::Borrowed (mmap) this happened to be sound, but for storages that return Cow::Owned (disk-cache misses, uring backends) every user of get_vec_ptr read freed memory: the internal SIMD scorers, encode_internal_vector, and get_quantized_vector_offset_and_code, which exported the dangling buffer through a safe &[u8]. Make parse_vec_data return the code as a slice borrowing the input, so the borrow checker forces callers to keep the storage buffer alive while reading it, and drop get_vec_ptr entirely. This matches how encoded_vectors_binary already binds the buffers at its scoring sites. Change get_quantized_vector_offset_and_code to return the code as a sub-view of the buffer Cow itself, and adapt its GPU caller. The new regression test drives these paths through a storage that always returns owned buffers; under Miri it reproduces the use-after-free on the previous code and passes with this fix. Fixes #9799 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * additional assertion --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com> |
||
|
Andrey Vasnetsov
|
da7eedeaf3 |
Allocate a single slot per operation in append-only mutations (#9804)
* Fuse CoW point move into a single destination write The CoW arm of apply_points_with_conditional_move wrote a moved point in three SegmentEntry steps: upsert_point_raw, update_vectors, set_full_payload. With append_only_mutations enabled every step clones the whole point to a fresh internal id, so one moved point burned three slots (the first holding an empty point for plain upserts, which clear raw_vectors) and left two immediately-dead clones behind, tripling the id-tracker changelog and vector writes. Add SegmentEntry::upsert_moved_point, which writes raw vectors, the decoded overlay, and the payload in one operation — allocating exactly one slot — and use it on the CoW move path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Mutate same-operation slots in place in append-only mode With append_only_mutations enabled, every mutation of an existing point clones it to a fresh internal id. Shard-level updates decompose one point write into several SegmentEntry steps (upsert_with_payload issues upsert_point plus set_full_payload/clear_payload), so a single upsert burned one slot per step, leaving a chain of immediately-dead clones. A slot whose version already equals op_num was written by an earlier step of the current operation. It cannot be durable yet — the segment write lock is held across the whole operation, so no flush (and no read-only follower) can have observed it, and versions flush last so a crash discards it and WAL replay re-applies the whole operation. Mutate such slots in place: one operation now allocates exactly one slot regardless of how many steps it decomposes into. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stamp payload storage version in write_point_parts overwrite_payload mutates payload storage, but the fused write never bumped version_tracker's payload version — the old CoW path did, via set_full_payload. The segment manifest would stamp payload files with a stale version, letting a partial snapshot skip payload storage that contains the moved point's row, so the restored id tracker would point at an offset with no payload. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
e7340ed160 |
Stamp payload storage version on append-only clone paths (#9806)
clone_and_mutate_point and clone_and_replace_point_raw rewrite the point's payload row at a fresh internal id, mutating payload storage — but never bumped version_tracker's payload version. Payload operations bump it at the caller, so their clones were stamped; vectors-only operations (upsert_point, upsert_point_raw, update_vectors, delete_vector) never stamp payload, leaving the manifest with a stale payload version. A partial snapshot could then skip payload storage files that changed, and the restored id tracker would point at offsets with no payload row. Stamp inside the clone helpers, where the mutation happens. The payload entry points move their bump into the in-place closure so the clone arm is not double-bumped: a same-version double bump collapses the tracked version to None, degrading the stamp to the segment version. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |