The Debug requirement on UserData (#9588) left two impl blocks without
the trait bound, breaking compilation on dev.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add a `Debug` supertrait bound to the `UserData` marker trait and propagate
it through the universal-io read pipelines and the read APIs built on them.
This lets pipeline user-data (request ids, point ids, internal read metadata)
be formatted for diagnostics. Wrapper types that carry user data through the
pipelines (`RemoteMeta`, gridstore `ReadMeta`, hashmap `Entry`) gain `Debug`,
and the `U: UserData` bound is threaded through `read_vectors`/`read_payloads`/
`read_values`/`iter_vectors` and their implementors in gridstore and segment.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf: read whole objects in a single GET instead of HEAD+GET
* feat: read object tail from offset to EOF in a single GET
Generalize the whole-object single-GET read so a tail read from a known
offset also avoids the separate len()/HEAD round-trip. The backend
primitive becomes `read_from(path, from)`, issuing an open-ended
`GetRange::Offset(from)` GET (or a plain GET for from == 0) and reporting
the object's total size from the response; `read_whole` is now a thin
wrapper over it.
The owned blob pipeline's `schedule_whole` no longer HEADs the remote to
size a tail read. An offset at/past EOF is an unsatisfiable range (HTTP
416) rather than an empty body, so the buffer builder disambiguates with
a single len() only on the error path, yielding an empty read when the
tail is genuinely empty. The disk cache's reopen prefiller is made
tolerant of that empty tail so it no longer truncates the local mirror.
Also split the now-hard-to-follow simple_disk_cache `file.rs` into a
`file/` module (type/state, init state machine, reopen, read surface)
and document the FromScratch vs Prefiller init sources.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* hold init_guard during reopen
* add `from` parameter to `OwnedReadPipeline::schedule_whole`
* add `OwnedReadPipeline::into_inner` to extract inner file
* implement proper background populate on `reopen`
* clippy
* coderabbit
* explicit owned uring pipeline destructuring
* refactor to join local and remote into one state
* @ffuugoo's nits
Follow-up to #9530 and #9558.
The shard-level segment manifest was written inside the `segments/`
directory (`segments/manifest.json`). Older versions of Qdrant choke on
an unknown file inside `segments/`, so move it next to the directory as
`segments_manifest.json` instead.
- `SEGMENT_MANIFEST_FILE` is now `segments_manifest.json` and
`segment_manifest_path()` points at the shard root.
- The manifest is added to `ShardDataFiles` so clear/move handle it.
- Snapshots write the manifest to the snapshot root (next to
`segments/`), and restore/partial-snapshot loaders no longer need to
skip it inside `segments/`.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: feature-flagged segment manifest for LocalShard
Add an on-disk segment manifest (`segments/manifest.json`) that lists a
shard's segments and their state, so out-of-process readers (e.g. a
read-only follower, possibly over object storage) can discover segments
without scanning the filesystem. Gated by the new `write_segment_manifest`
feature flag (off by default).
shard: define the structure + helpers (`SegmentsManifest`,
`SegmentManifestState`, `from_segment_holder`) in a new `segment_manifest`
module, plus the `SEGMENT_MANIFEST_FILE` constant and path helper. The
manifest is a flat `{ "<uuid>": "<state>" }` map; only `active` is written
today, with `under_construction`/`retiring` defined so the format can be
extended without breaking compatibility.
collection: LocalShard owns the writing logic. The manifest is persisted
via `SaveOnDisk<SegmentsManifest>`, initialized from the live segment set
on load/build and refreshed by the optimization worker whenever the
segment set changes (the helper re-derives from the holder and no-ops when
unchanged). No changes to `lib/shard/src/optimize.rs` internals.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor
* feat: make append_only_mutations a proper feature flag
Replace the debug-only `QDRANT_APPEND_ONLY_MUTATIONS=1` env-var escape
hatch in segment construction with a `FeatureFlags::append_only_mutations`
flag, so it works in release builds and is configurable like the other
flags (config / `QDRANT__FEATURE_FLAGS__APPEND_ONLY_MUTATIONS`).
Deliberately left out of `FeatureFlags::all()`: it changes mutation
semantics and `all` is enabled in dev and e2e configs, so it stays
explicit opt-in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor
* upd openapi schema
* feat: register segments in manifest via must-use token, holder builder
Wire segment-manifest maintenance through the segment lifecycle so a
newly created segment is registered as soon as it exists on disk, and
construction can't silently skip it:
- build_segment now returns a #[must_use] NewSegmentToken carrying the
new segment UUID; the lint forces callers to register or drop it.
- SegmentHolder owns the manifest and reconciles it on sync; new
segments are registered ASAP (even before being added to the holder)
via the token, before they can receive writes.
- SegmentHolderBuilder is the only way to obtain a shard's holder; its
build() wires up the manifest, so it can't be forgotten. init/set
manifest helpers are now private / test-only.
- Optimization registers the optimized segment before dropping the
superseded segments' data; deletion is intentionally lenient.
- Document the consistency assumptions on SegmentsManifest: it is a
superset-biased view that may list not-yet-finalized or already-deleted
segments, which readers must tolerate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`GridstoreReader`)
- Make the entrypoint of payload storage choose the `Populate` variant
- Mutable payload indexes now populate gridstore blockingly before using
it to load
- Propagate `Populate` into `flags` module
- Add `UniversalRead::populate_auto` to know whether a backend chooses
to populate or not when `Populate::Auto`
* Add DynConditionChecker type alias
* Add DynConditionChecker type alias (point_scorer.rs)
* Move trait ConditionChecker to `common`
Reason: will be used both in `segment` and `sparse` crates.
* feat: add open for read-only sparse vector index + enum sparse dispatcher
* fix: universal-IO loads for read-only sparse index open
* refactor: rename load_via/open_via to load_universal/open_universal
* refactor: drop StorageVersion::load in favor of load_universal
The regular-IO `load` duplicated `load_universal` over plain `File` IO.
Remove it and route all callers through `load_universal(&MmapFs, ..)`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(sparse): decouple inverted index from concrete storage; split segment_constructor_base (#9461)
Make the read-only index enum generic over storage `S` (no concrete MmapFile),
and remove construction callbacks from the sparse index open paths.
- InvertedIndex is now a pure read/search trait: `open`/`from_ram_index`/
`type Fs` moved off the trait to inherent methods on each concrete index
type, so construction no longer requires `S::Fs: Default`.
- SparseVectorIndex open split into a generic `plan` (load-vs-build decision +
RAM-index build) and generic `finish` (assembly); callers do the concrete
per-type construction, so no construction callbacks are needed.
- ReadOnlySparseVectorIndex::open takes the already-constructed inverted index
and caller-loaded config instead of a `load_inverted_index` callback.
- VectorIndexReadEnum is generic over `S: UniversalRead`; sparse mmap variants
hold `InvertedIndexCompressedMmap<_, S>` rather than a concrete `MmapFile`.
- Split the 1182-line segment_constructor_base.rs into a module (paths,
vector_storage, payload_storage, id_tracker, vector_index,
sparse_vector_index, create_segment, segment, legacy_state); the sparse
dispatcher's match arms collapse into three per-family helpers.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat: LiveReload (no-op) dispatch for read-only vector index enum (#9436)
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(common): don't fsync read-only file handles on Windows (#9132)
* Make sync_dir_with_fsync unix-only so non-unix is a clean no-op; use fs_err in test
* feat(debug): QDRANT_APPEND_ONLY_MUTATIONS env override
Debug-only escape hatch so newly built segments default to append-only
mutation routing when QDRANT_APPEND_ONLY_MUTATIONS=1 (or true/yes) is
set in the environment. Lets us run the existing test suites against
the append-only path without wiring a collection-level config knob
first.
Release builds compile this out — the function is a const false.
Logs a single warn-level message the first time the override fires.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(append-only): skip deleted vectors on snapshot, always write payload
Two correctness fixes to clone_and_mutate_point uncovered by running
the openapi suite with QDRANT_APPEND_ONLY_MUTATIONS=1:
1. The snapshot loop read every named vector via get_vector_opt, which
returns slot bytes even when the per-vector deletion bit is set.
For sparse-only points (or any point that had update_vector(_, None)
applied) this materialised the default-zero vector as if it were
real data, wrote it at new_id, and never re-tombstoned the slot —
so dense search started scoring phantom vectors. Now we check
is_deleted_vector(old_id) and skip the read, letting the writer
loop emit update_vector(new_id, None) and re-mark the slot deleted.
2. The payload write was skipped when the snapshot ended up empty.
That dropped two side effects the field indexes rely on:
payload_storage.overwrite(new_id, empty), and the remove_point
fan-out across configured field indexes that bumps each index's
total_point_count to cover new_id. Without the bump the null index
doesn't see new_id, so is_empty / is_null filters lose the point
even though its mapping is live in the id tracker. The skip was an
optimisation, not a contract; remove it so the field indexes get
the same registration they'd get from the standard clear_payload
path.
Also collapses the debug env override to an inline cfg!()-gated
check at the struct literal — the helper with one-time logging and
multi-value matching was disproportionate for a debug-only escape
hatch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(id_tracker): split active vs deferred maps in PointMappings (PR A)
First step of moving the mutable id tracker to a "two-track" model so
append-only mutations into a deferred segment can keep both the
visible (active) version and the latest mutated (deferred) version of
a point at the same time.
This PR is purely structural. On-disk format is unchanged: the loader
still produces a single combined map, and `PointMappings::new`
partitions it at construction time based on `deferred_internal_id`.
Every observable behaviour at the existing API surface is preserved.
Concretely:
- New fields: `external_to_internal_num_deferred`,
`external_to_internal_uuid_deferred`, and a `shadowed: BitVec` for
future use by PR B (lazy-grown, default-false).
- `internal_id(ext)` checks active first, falls through to deferred —
matches the pre-split "any matching id" contract for ext ids whose
internal id sat above the cutoff.
- `set_link(ext, new_id)` now routes by cutoff: writes below the
cutoff land in active, writes at or above land in deferred. Any
prior head in the other track is tombstoned, so each ext still owns
exactly one slot — same observable result as the pre-split
single-map insert. PR B replaces the cross-track tombstone with a
shadow-bit flip; PR A keeps current semantics on purpose.
- `drop(ext)` clears entries from both tracks and tombstones each
one, again matching the prior single-map behaviour.
- `iter_external` and `iter_from` merge the active and deferred
BTreeMap views into one sorted-by-key stream (dedup'd in case an
ext exists in both tracks).
- `available_point_count` counts distinct external ids across both
tracks — preserves the prior observable count for segments where
some entries used to sit above the cutoff in the single map.
No write-path or read-path behaviour change. Reads still filter
`internal_id >= cutoff` exactly as before; mutations still tombstone
prior heads. The shadow bit and the deferred-aware lookup wiring
land in PR B and PR C.
All existing id_tracker tests pass against the split layout.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(id_tracker): shadow active head on deferred writes (PR B)
Step 2 of the deferred-aware mutable id tracker. Replaces PR A's
"tombstone the other track" cross-track cleanup with shadow-bit logic
so a deferred mutation no longer hides the visible active version.
Behaviour by case:
- Active write (`internal_id < cutoff`, or no cutoff): unchanged
semantically. Any prior deferred head for the same ext is dropped
and tombstoned (the new active is the single visible head). Same
observable result as PR A.
- Deferred write (`internal_id >= cutoff`): the prior deferred head
(if any) is dropped and tombstoned. The active head, if it exists,
is **shadowed** — its bit is set in `shadowed: BitVec` but its slot
stays alive in the active map. Read paths in `Exclude` mode
continue to return that active version; PR C will teach
`IncludeAll` paths (the optimiser) to skip shadowed actives and
prefer the deferred head.
Also adds `is_shadowed(internal_id)` and `shadowed_bitslice()`
accessors (the latter for PR C's filter pipeline). New unit tests
cover the routing matrix:
- no cutoff — active replacement, no shadow,
- below-cutoff replacement — active path, no shadow,
- deferred-on-top-of-active — shadow set, active retained,
- two deferred writes — prior deferred tombstoned, shadow persists,
- fresh insert above cutoff — no shadow,
- `drop(ext)` clears both tracks plus the shadow bit.
WAL replay reuses the same `set_link`, so deferred routing on
replay falls out of this change with no extra wiring.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(id_tracker): IncludeAll skips shadowed actives, deferred-aware lookups (PR C)
Step 3 of the deferred-aware id tracker stack. PR B introduced the
shadow bit on `set_link`; this PR teaches the read paths to honour it
so the optimiser (and any other `DeferredBehavior::IncludeAll`
consumer) sees each external id exactly once, with the deferred head
winning over the now-shadowed active.
Concretely:
- `PointMappings::internal_id_with_behavior(ext, behavior)`:
* `Exclude` returns the active head only — `None` for a
deferred-only ext, so query paths never see a deferred mutation;
* `IncludeAll` prefers the deferred head and falls back to active
for points that never crossed the cutoff. Yields at most one
internal id per ext.
- `IdTrackerRead::internal_id_with_behavior` mirrors the new method
with a default impl that delegates to `internal_id` for trackers
that don't carry deferred mutations.
- `PointMappingsRefEnum::iter_internal_with_behavior(IncludeAll)`
now filters shadowed actives via the new
`PointMappings::shadowed_bitslice()` accessor.
- `PointMappingsRefEnum::filter_deferred_and_deleted(IncludeAll)`
also filters shadowed actives — same single-yield-per-external
guarantee for external iterator sources like field-index outputs.
- `IdTrackerRead::resolve_external_ids` switches to the deferred-aware
lookup. No more post-lookup `id >= cutoff` filter — the behaviour
enum lookup gets it right at the source.
New unit tests cover the two new entry points:
- IncludeAll prefers the deferred head when an active is shadowed;
- Exclude returns None for deferred-only ext ids;
- `filter_deferred_and_deleted` over a mixed candidate list yields
the expected per-mode result (Exclude: actives below cutoff;
IncludeAll: every visible head, no shadowed actives).
No queries observable behaviour change today — production Exclude
paths still resolve via `internal_id` and the active head. The
optimiser will start using `IncludeAll` (and reap the dedup) in
follow-up work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(id_tracker): expose shadowed_point_count in SegmentInfo (PR D)
Final step of the deferred-aware id tracker stack. Adds a single new
counter — number of active heads currently shadowed by a deferred
mutation — and plumbs it through the id-tracker trait, the segment
read-view helper, and `SegmentInfo` for telemetry.
Preserves existing semantics:
- `available_point_count` is unchanged. Each distinct external id
still counts once, regardless of which track holds its head.
- `deferred_point_count`, `deferred_internal_id`,
`num_deleted_deferred_points` keep their current values.
- Non-appendable trackers default `shadowed_point_count()` to `0`,
so the new `SegmentInfo.num_shadowed_points` is `None` for them.
Concrete changes:
- `PointMappings::shadowed_count()` — popcount of the shadowed
bitslice.
- `IdTrackerRead::shadowed_point_count()` trait method with a `0`
default; wired through `MutableIdTracker`, `InMemoryIdTracker`,
the mutable read-only tracker, and both enum dispatchers.
- `SegmentReadView::shadowed_point_count()` helper.
- New `SegmentInfo.num_shadowed_points: Option<usize>`, populated
with `Some(_)` for appendable segments and proxied through
`ProxySegment` from the wrapped segment's value.
New unit test covers the counter lifecycle:
- active-only writes don't grow it,
- a deferred write over an active adds one shadow,
- a second deferred write supersedes the prior deferred head but
the shadow stays put (still one active being shadowed),
- `drop(ext)` clears the shadow bit,
- a fresh deferred insert with no active prior doesn't add a shadow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(id_tracker): rename DeferredBehavior, push behavior down to PointMappings
Three connected pieces of polish on top of the deferred-aware id
tracker stack:
1. Rename `DeferredBehavior::Exclude` → `VisibleOnly` and
`IncludeAll` → `WithDeferred`. The pre-PR-C semantics of
"include/exclude the deferred cutoff" became misleading once
`IncludeAll` started skipping shadowed actives — it doesn't
"include all" anymore, it yields one slot per external (deferred
head preferred, active fallback). New names describe what each
variant returns instead of how it relates to the cutoff. The
helper method `include_all_points` becomes
`with_deferred_points`. Updates ~90 call sites across the
workspace; behaviour is unchanged.
2. Push `iter_internal_with_behavior` down from
`PointMappingsRefEnum` into `PointMappings`. The per-mode
logic (cutoff `take_while`, shadowed `filter`) now lives next
to the data it consults; the enum layer becomes a two-arm
`Either` dispatcher. `CompressedPointMappings` short-circuits
to `iter_internal()` since compressed mappings can't carry
deferred mutations.
3. Add a short docstring on `internal_to_external` describing the
two-track model: no active-vs-deferred bias, shadowed pairs
occupy two slots with the same value, and reads must gate on
`deleted` because `set_link`'s same-track replacement leaves a
real-looking stale ext id in place.
Returning `impl Iterator` instead of `Box<dyn Iterator>` for
`iter_internal`, `iter_internal_excluding`, `iter_internal_visible`,
and `iter_internal_with_behavior` removes the double-Box at the
enum boundary. The original branching structure is preserved with
`itertools::Either` instead of restructured into one big filter
chain.
All existing `id_tracker` tests still pass (47 total).
`cargo check --all-targets` + `cargo clippy --all-targets` clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(id_tracker): drop shadowed API surface, unbox iter_from/iter_random
Cleanup pass on top of the deferred-aware id tracker stack.
API surface:
- Drop SegmentInfo.num_shadowed_points + its trait method, read-view
helper, tracker impls and ProxySegment passthrough. Cross-segment
shadow already exists in the appendable update flow (the source
segment keeps its copy when the new write lands above the cutoff,
see SegmentHolder::apply_points_with_conditional_move) and is
handled at the aggregation layer, not reported per-tracker.
- Keep PointMappings::shadowed_count() as a private popcount helper
guarded by expect(dead_code) so a future caller can opt back into
the dedup'd count without re-plumbing the trait.
available_point_count:
- Replace the cross-track dedup (active + deferred-only via
contains_key filter) with a straight 4-term sum. A shadowed ext
contributes two slots, matching the two non-tombstoned internal
ids it actually occupies. Old dedup broke the invariant that
deleted_point_count == deleted_bitslice.count_ones(): for each
shadow it overcounted deletions by one without any tombstone
actually being set.
DeferredBehavior pushdown:
- iter_random_with_behavior: caps the sampling range at the
deferred threshold in VisibleOnly mode (no wasted samples above
cutoff), filters shadowed actives via the bit in WithDeferred.
- iter_from_with_behavior: VisibleOnly walks the active maps only
(no merge with deferred); WithDeferred delegates to iter_from's
existing merge.
- scroll.rs read_by_id_stream / filtered_read_by_id_stream collapse
their manual if-deferred-behavior branching into a single
iter_from_with_behavior call.
- Old iter_random (no behavior) at PointMappings + ref enum was
unused after the migration, deleted.
Unboxing iter_from / iter_random / iter_from_with_behavior:
- PointMappings::iter_from returns impl Iterator + '_ via Either
inside the merged_num/merged_uuid closures (BTreeMap::iter vs
range), Either at the outer match (num+uuid chain vs uuid-only).
- PointMappings::iter_from_with_behavior unboxed with a triple
Either (behavior, external-id arm, closure start).
- CompressedPointMappings::iter_from unboxed (Either over None/Some).
- PointMappingsRefEnum::{iter_from, iter_from_with_behavior,
iter_from_visible} all return impl Iterator + 'a via Either on
the Plain/Compressed dispatch.
Other:
- Update outdated PR-A/PR-B comment on PointMappings::drop.
- Make the max_internal match in iter_random_with_behavior
exhaustive (VisibleOnly+None | WithDeferred+_ instead of `_`).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(id_tracker): unbox iter_external
Return impl Iterator from PointMappings::iter_external,
CompressedPointMappings::iter_external and the PointMappingsRefEnum
wrapper, matching the style of the other iter_* helpers. The wrapper
dispatches via Either. The remaining Box::new at Segment::iter_points
stays because self_cell's BoxedPointIdIterator alias needs a sized
type.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(id_tracker): count set_link tombstones in deferred_deleted_count
PointMappings::set_link tombstones the prior slot at two sites — when
an active write supersedes a deferred head (cross-track) and when any
write replaces a same-track head — using a bare deleted.set(old, true).
Neither path updated deferred_deleted_count, so tombstones above the
cutoff weren't reflected in the counter.
The append-only flow exposes this constantly: every set_full_payload
after upsert_point routes through clone_and_mutate_point, which
re-issues set_link with a fresh internal id and tombstones the prior
one. Most tombstones land above the cutoff, so deferred_point_count
(total - cutoff - deferred_deleted_count) over-reports by the missing
count. The openapi test_deferred_points integration test caught this
as `num_points - num_deferred_points = -1800` across two segments.
Extracted the tombstone bookkeeping into PointMappings::tombstone_slot
and routed both set_link sites + drop's loop through it. Behaviour on
drop is unchanged; set_link now bumps the counter on the
live → tombstoned transition for slots at or above the cutoff.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style(id_tracker): collapse set_link tombstone if-let chains
Clippy's collapsible_if on the two if-let blocks added by the
deferred_deleted_count fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update lib/segment/src/id_tracker/point_mappings.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* refactor(id_tracker): size shadowed BitVec once instead of growing lazily
Collect the shadowed active ids up front and allocate the BitVec to the
highest offset in a single resize, avoiding repeated reallocations while
marking shadows. Addresses review feedback.
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert: restore lazily-grown shadowed BitVec
Roll back the up-front sizing of the shadowed BitVec; last_entry doesn't
fit here and swapping one allocation for another isn't worthwhile.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(id_tracker): prefer deferred head in iter_from merge
When an external id has both an active and a deferred head, iter_from's
merge collapsed the pair to the active (stale) offset. That contradicts
the WithDeferred contract used everywhere else: internal_id_with_behavior
and iter_random_with_behavior both surface the deferred head (the latest
mutation) over the shadowed active. Consumers that use the returned
internal id (payload-filter checks, the optimizer's version merge, HNSW
old->new mapping) therefore saw the stale copy.
Flip the Both arm to take the deferred operand and document the rule.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(id_tracker): preserve active+deferred heads when loading mappings
read_mappings replayed the persisted change log into a single flat
external -> internal map per id-type, then split it by the deferred
cutoff in PointMappings::new. A flat map holds one head per external id,
so it collapsed the active+deferred coexistence case (an external id
linked first to an active slot, then to a deferred one via sequential
set_link) down to the last write — silently dropping the other head,
orphaning its slot, and leaving the shadowed bit unset. The split in
new() could not recover what was already lost before it.
Replay the log through the canonical set_link/drop mutators on a
PointMappings seeded with the cutoff instead. The log is the sequence of
set_link/drop calls that produced the live in-memory state, so this
reconstructs that state exactly — both heads, the shadowed bit, and
deferred_deleted_count — with no logic duplication. Drops the
debug_assert-guarded corruption-recovery branch (subsumed by set_link's
re-link handling) and the now-unused Uuid/PointIdType imports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(id_tracker): make deferred behavior explicit at point resolution
Replace the ambiguous "any matching head" point resolution with an
explicit DeferredBehavior at every resolution boundary, so callers no
longer rely on a hidden active-vs-deferred policy.
- Remove PointMappings::internal_id and the bare IdTrackerRead::internal_id
(the active-first-else-deferred hybrid). internal_id_with_behavior is now
the single required resolution method; immutable/compressed trackers
implement it by ignoring the behavior (they never carry deferred heads).
- Migrate every caller to an explicit behavior, audit-driven:
- writes (upsert/delete/payload/vectors), point_version, point_is_deferred,
get_internal_id, drop, consistency + builder dedup -> WithDeferred (the
latest/live head);
- single-point payload/vector retrieval and formula rescore -> VisibleOnly;
- HasId/CustomIdChecker/cardinality resolution -> the request's behavior,
threaded through the filter chain from iter_filtered_points (other entry
points default to VisibleOnly).
- lookup_internal_id takes an explicit DeferredBehavior instead of assuming
VisibleOnly internally.
- has_point takes an explicit DeferredBehavior (drop the has_point_with_behavior
wrapper). Thread it through read_points/_read_points/read_points_locked so
retrieve_blocking passes its request behavior to the existence filter; all
other existence/dedup callers pass WithDeferred (unchanged behavior).
- set_link now detaches a stale live occupant of a reused internal id, keeping
the forward and reverse maps consistent when recovering a corrupted log.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(id_tracker): failing tests for two correctness findings in #9249 (#9311)
* test(id_tracker): failing tests for two review findings
Two intentionally-failing tests pinning correctness gaps in the
deferred-aware id tracker (#9249). Both fail on the final assertion;
the earlier assertions establish the expected/consistent behavior.
1. iter_from_with_behavior(WithDeferred) resolves an active/deferred
shadow collision to the stale ACTIVE internal id, while its siblings
internal_id_with_behavior and iter_internal_with_behavior correctly
surface the DEFERRED (latest) head. Consumers that use the yielded
internal id (optimizer merge via for_each_unique_point,
filtered_read_by_id_stream) therefore observe the pre-mutation
version.
left: [(NumId(7), 2)] right: [(NumId(7), 9)]
2. The PR-B shadow/visible invariant is not durable: the on-disk single
combined map cannot represent a shadowed ext, so a plain mappings
flush + reload collapses the shadow to deferred-only and the visible
(active) head is lost (VisibleOnly resolves None where it resolved
Some(2) live). Restoration then depends entirely on WAL replay, i.e.
on flush-vs-WAL-truncate ordering.
left: None right: Some(2)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(id_tracker): strengthen shadow tests + merge-primitive proof
Follow-up to the two failing tests, addressing self-review:
- Add for_each_unique_point_keeps_deferred_head_for_shadowed_point: the
optimizer merge primitive (used by segment_builder::update_from) yields
the stale active copy (internal 2, version 5) for a shadowed point and
drops the deferred latest (internal 9, version 8). This directly
exercises the data-loss consequence of finding #1 at the merge layer.
left: [(NumId(7), 2, 5)] right: [(NumId(7), 9, 8)]
- Tighten shadow_visible_head_survives_mapping_flush_reload: pin the exact
reload failure mode. After flush+reload the mapping collapses to
deferred-only (internal_id == Some(9)) and the active slot survives as a
live orphan in the inverse map (external_id(2) == Some(7), not deleted) —
a torn state where a VisibleOnly scroll still surfaces the stale copy
while by-id VisibleOnly resolution breaks. Reframe as the live-vs-reload
divergence the PR introduces (Some(2) live -> None reload; dev is
consistently None).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix test
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* feat: add read-only QuantizedVectors generic over UniversalRead
Introduce `QuantizedVectorsRead<S>` / `QuantizedVectorStorageRead<S>`, a
read-only counterpart of `QuantizedVectors` organized like
`VectorStorageReadEnum`: generic over the `UniversalRead` backend `S`,
opened from existing on-disk data, with no create/upsert/builder path and
no disk writes.
Highlights:
- Keep both in-RAM (`*Ram`) and read-only mmap (`*Mmap`) variants; drop the
appendable `*ChunkedMmap` variants (the only mutable ones).
- All bulk reads go through `S`: add `QuantizedRamStorage::from_universal_read`
and `MultivectorOffsetsStorageRam::open`, and make
`MultivectorOffsetsStorageMmap` generic over `S` (default `MmapFile`).
- Share scorer construction between the read-write and read-only enums via a
`QuantizedScorerDispatch` trait, so only the per-variant match is duplicated
while the datatype/distance and per-query dispatch live once in the builder.
Tested: read-only vs read-write scorer parity (scalar/binary/product, single
and multivector, RAM and mmap), covering both `raw_scorer` and
`raw_internal_scorer`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: route quantized RAM loading through UniversalRead
Follow-up to the read-only quantized work, removing direct-filesystem reads
and load-path duplication:
- `EncodedVectors{U8,PQ,Bin,TQ}::load` now take a `UniversalRead` filesystem
and read metadata via `read_json_via` instead of `fs::read_to_string` —
every read flows through universal IO. Single-vector `load` returns
`common::universal_io::Result`; `validate_storage_vector_size` stays
`std::io::Result`.
- Add `common::universal_io::OneshotFile<S>`: a thin RAII wrapper over any
`UniversalRead` handle that evicts the data from cache via `clear_ram_cache`
on drop (the universal-IO counterpart of `fs::OneshotFile`).
- Collapse `QuantizedRamStorage::{from_file, from_universal_read}` into one
`from_file<S: UniversalRead>`. It reads the whole file in a single access
(no separate `len()` round-trip — cheaper on S3-like backends) and loads via
the new `VolatileChunkedVectors::extend`, which inserts one chunk per
`copy_from_slice` instead of one vector at a time.
- RW callers pass the local `READ_FS` (mmap) backend; the read-only loader
passes its `S`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: load appendable (chunked) quantization format read-only
`storage_type` only selects the on-disk layout, so the read-only quantized
storage must be able to load both — the immutable flat format and the
appendable chunked format (produced only by Binary/TurboQuant). Previously the
read-only loader rejected the Mutable layout outright.
- Add `QuantizedChunkedStorageRead<S>` and `MultivectorOffsetsStorageChunkedRead<S>`,
read-only wrappers over the existing `ChunkedVectorsRead<_, S>` primitive
(mirrors the dense read-view's chunked read storage). Generic over the
`UniversalRead` backend, on-disk, no write path.
- Add `BinaryChunked`/`TQChunked` (+ multi) variants to `QuantizedVectorStorageRead`
and wire them through every accessor and the scorer dispatch.
- Route `storage_type == Mutable` to the chunked read variants in the loader and
drop the blanket rejection.
- Tests: parametrize the read-only/read-write parity tests over `storage_type`
and add chunked (Mutable) cases for binary/turbo, single and multivector.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: split quantized chunked & multivector storage into modules
`quantized_chunked_mmap_storage.rs` and `quantized_multivector_storage.rs` had
grown to hold several unrelated structures, making them hard to navigate.
Convert each into a module (pure code movement, no behavior change):
- quantized_chunked_mmap_storage/{read_write,read_only}.rs — the appendable
mmap storage + builder vs. the read-only chunked storage.
- quantized_multivector_storage/{mod,offsets}.rs — the core
`QuantizedMultivectorStorage` + offset traits stay in mod.rs; the four
`MultivectorOffsetsStorage*` backends move to offsets.rs.
Public paths are unchanged via re-exports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: single source of truth for quantized vector size
The on-disk stride (bytes per quantized vector), including the binary
`u8`(multi)/`u128`(single) word-type choice, was independently re-derived in
every load/open site with no compile-time link — the kind of value-level
duplication that silently diverges (it already caused the binary multi
`u8`/`u128` bug).
Extract it into one place:
- `QuantizedVectors::quantized_vector_size(quantization_config, vector_parameters, is_multi)`
and the `QuantizedVectorsConfig::quantized_vector_size(is_multi)` convenience.
Route every reader through it:
- read-only `open_single`/`open_multi` and the read-write `{scalar,pq,binary,turbo}`
loaders now hoist `config.quantized_vector_size(is_multi)` once instead of
recomputing the per-method formula per branch.
The create (write) path keeps its own computation for now; it stays guarded by
`validate_storage_vector_size` and the read-only/read-write parity tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: flat-match quantized loaders on a shared QuantizedStorageKind
The loaders chose the concrete storage variant via a nested
`method × (storage_type / is_ram)` match, re-derived independently in the
read-only and read-write paths — hard to follow and easy to drift.
- Add `QuantizedStorageKind` + `QuantizedVectorsConfig::storage_kind(on_disk)`
(and `is_ram(on_disk)`): the single place the method × backend decision lives.
- Both loaders now compute the kind once and use a flat 10-arm match:
- read-only `open_single`/`open_multi`;
- read-write `load_single`/`load_multi` (consolidating the four per-method
`{scalar,pq,binary,turbo}/load.rs` files, which are removed).
Both matches are exhaustive over the same enum, so the read and read-write
variants can no longer fall out of sync without a compile error.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: genericize write-capable chunked quantized storages over Fs
Drop the hardcoded `MmapFile` specialization from `QuantizedChunkedStorage`,
`QuantizedChunkedStorageBuilder`, and `MultivectorOffsetsStorageChunked`. They
now expose the `Fs` backend (defaulting to `MmapFile`) and accept the fs handle
as a parameter, matching the read-only variants.
Introduce a single shared `ReadFile` type alias and `READ_FS` value handle in
the `quantized_vectors` module root, used by the create, load, and storage enum
paths so the local-file backend is named in one place.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: move is_in_ram_or_mmap match into UniversalKind
Deduplicate the identical kind-to-residency match in the read-only and
write-capable chunked quantized storages by adding
UniversalKind::is_in_ram_or_mmap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: adapt TurboVectorStorage to quantized rename + update_from move
Integration fix after rebasing onto dev's TurboQuant work:
- QuantizedChunkedMmapStorage -> QuantizedChunkedStorage<MmapFile>
- update_from moved off the VectorStorage trait onto an inherent method,
matching the per-kind sub-trait refactor; TurboVectorStorage implements
neither DenseVectorStorage<T> nor the other kind traits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Use assert_matches!
* Add trailing commas
* Use more assert_matches!
Also, drop now redundant `expected blah but got blah` messages because
`assert_matches!` will print these.
* Use debug_assert_matches!
---------
Co-authored-by: xzfc <xzfcpw@gmail.com>
`validate_multi_vector_len(N, &[])` with N > 0 previously returned Ok:
it passes the `vectors_count != 0` check, an empty `flatten_dense_vector`
clears the size check, and `0.is_multiple_of(N)` is true, so it falls into
the Ok branch. The unvalidated value then reaches
`convert_to_plain_multi_vector`, where `dim = data.len() / vectors_count = 0`,
the divisibility check `dim * vectors_count != data.len()` (0 == 0) passes,
and `data.into_iter().chunks(0)` panics (itertools asserts the chunk size is
non-zero). This is reachable from a single malformed gRPC Upsert via the
deprecated `vectors_count` field.
Add an `is_empty()` guard to `validate_multi_vector_len`, mirroring the
sibling `validate_multi_vector_by_length`, so empty flattened data is
rejected with a clear validation error before any conversion. Also add a
defensive `vectors_count == 0 || data.is_empty()` guard at the top of
`convert_to_plain_multi_vector`, which aligns it with the already-guarded
`MultiDenseVectorInternal::try_from_flatten` and closes the same panic on the
internal node-to-node `sync` path (where validation is log-only and
`SyncPoints.points` is not validated).
This completes the empty-vector hardening started in #9070, which covered the
REST side only and did not touch the gRPC flattened `vectors_count` form.
Includes a regression test covering both the rejected (empty) and accepted
(consistent multivector) cases.
* feat: add open methods for map
* fix: review comments
* feat: detect not-found via error instead of path.exists in map index open
Replace the path.exists() pre-check in the read-only appendable map index
open path with error-driven detection through ok_not_found(). Generalize
OkNotFound over an IsNotFound trait, implemented for io::Error, mmap::Error,
UniversalIoError, and GridstoreError so a NotFound surfacing through any
layer (including mmap's inner io::Error) maps to Ok(None).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* review: revert internal error conversion
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Strict mode: add `max_disk_usage_percent`
Mirrors `max_resident_memory_percent`: rejects disk-consuming update ops
(upsert, set/overwrite payload, update vectors) when the filesystem hosting
Qdrant storage is filled above the configured percentage. Delete-style ops
remain allowed so callers can free disk.
Disk usage is sampled via `statvfs` and TTL-cached for 5s (same cadence as
the resident-memory reader) so high-RPS request paths don't hammer the
syscall. Reader is keyed by path in `common::disk_usage` and returns `None`
on stat failure — callers (the strict-mode check) treat `None` as "skip",
matching the memory-check behaviour.
Plumbing follows the existing pattern: field on `StrictModeConfig` (+
output/diff/Hash), gRPC proto field `22`, validation 1..=100, REST/proto
conversions, and the hook into `check_strict_mode_toc_batch` alongside the
memory check (both guarded by `any_consumes_memory`).
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix CI: Windows disk_usage test + e2e WAL config
- `missing_path_returns_none` panicked on Windows because
`GetDiskFreeSpaceEx` succeeds for non-existent paths (it resolves up to
the containing drive). Relax the assertion to "must not panic; if a
value is returned it must be well-formed". The contract we care about
(None on failure) is platform-defined, not something we can portably
force.
- e2e test failed at batch 0 with "WAL buffer size exceeds available disk
space": Qdrant's existing per-shard `DiskUsageWatcher` enforces
`free >= 2 * wal_capacity_mb` and the default WAL didn't fit in the
50 MB tmpfs. Bump tmpfs to 200 MB and shrink `wal_capacity_mb` to 1 MB
(same pattern as `test_low_disk.py`) so our strict-mode gate is the
one that fires, not the WAL pre-check. Raise the gate threshold to
50% to match the larger headroom.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Migrate InvertedIndexCompressedMmap to UniversalRead
* Rehaul search_scratch.rs (was scores_memory_pool.rs)
- Use `typed_arena::Arena` instead of `bumpalo::Bump`.
Reason: bump don't drop.
Drawback: `Arena::new()` is not free, and `Arena::clear()` doesn't
exist; but this commit has workarounds.
- Use names that make more sense.
* Misc fixes
* InvertedIndexCompressedMmap: explicit S type parameter
* load with any universal io
* Don't populate with sequential advice
* use appropriate fs for `exists`
* use `read_whole_via` more
* TODO
* clear ram cache after read_whole
* [ai] TQDT in the API
* [ai] unify new sparse error for TQDT
* Rename to `turbo4`
* Add `Turbo4` to comments and doc strings.
* Also validate named sparse vector creation
* feat: add io_bridge
* fix: cached dispatcher
* feat: add open_with_handle
* fix: naming
* fix: wording
* fix: rebase io_bridge onto split BorrowedReadPipeline/OwnedReadPipeline
* fix: linter
* feat: add S3 backend
* chore: remove io_bridge
* fix: linter
* fix: linter
* fix: read handle
* chore: simplify S3Source
* fix: s3 test
* feat: support multi runtime
* fix: clippy errors
* fix: review comments
* feat: add io design
* feat: add S3 backend
* chore: fix docs
* fix: dev changes
* chore: add some docs
* chore: remove explicit type
* feat: add new methods
* [WIP] review refactor
* fmt
* fix: bytes alignment
* fix: linter
* feat: remove Bytes
* fix: ci/cd
* fix: tests
* fix: tests
* refactor: simplify io_bridge pipeline to Handle-based dispatch
Replace the BridgeRuntime worker thread + request channel + boxed
BridgeRequest with direct tokio Handle usage:
- BridgeRuntime is now just an Arc<Runtime>; schedule() spawns the read
future via Handle::spawn instead of routing it through a dispatcher
thread. Removes BridgeRequest and the now-unreachable S3RuntimeShutDown
error variant.
- Guard against a panicking read task hanging wait() forever: the spawned
task catches unwinds and converts them into a TaskPanicked error reply,
so every scheduled slot is always answered.
- Encapsulate slot bookkeeping in PendingSlots, exposing only the needed
operations instead of a public map + counter.
- Split the grown pipeline.rs into a pipeline/ module (slots / inner /
borrowed / owned), de-duplicating the shared read-future construction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: move pipeline buffer ownership into the read future
Instead of the pipeline owning the destination Vec<T> in a slot map and
the future writing through a SendBytePtr raw pointer, let the future
allocate the buffer itself and return it through BridgeResponse. The
buffer crosses the worker-thread boundary as a normal move via the reply
channel, wrapped in a SendableVec<T> newtype that asserts Send for
T: bytemuck::Pod only.
This removes the entire unsafe SendBytePtr apparatus from the pipeline:
no raw pointer, no unsafe fn, no per-call-site unsafe blocks, no
heap-stability invariants. The only remaining unsafe in the crate is one
bounded `unsafe impl<T: Pod> Send for SendableVec<T>` with a trivially
true invariant (Pod types are plain bytes).
PendingSlots collapses back to PendingSlots<U>: slots no longer carry
buffers. AlignedBufWriter::from_raw_bytes (used only by the SendBytePtr
path) and its test are removed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: drop SendableVec wrapper now that Item: Send
UniversalRead's element type is now bound to `Item` (`Pod + Send`),
so the io_bridge pipeline no longer needs a hand-rolled `Send` wrapper
around `Vec<T>` to ship buffers through the reply channel. Replace
`SendableVec<T>` with `Vec<T>` end-to-end and tighten the local impls
to `T: Item`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: implement UniversalRead::reopen for BlobFile
BlobFile has no cached file metadata or mapping — `len()` queries the
object store fresh on each call — so reopen is a no-op, matching the
io_uring impl.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: add new fs impl for Blob
* fix: is_in_ram_or_mmap for S3
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [UIO] Split UniversalReadFileOps into filesystem + file traits
`UniversalReadFileOps` is now an instance-based trait describing a
filesystem handle (list/exists with `&self`, plus `from_context`). A new
`UniversalReadFs: UniversalReadFileOps` subtrait adds the
`open(&self, path, options) -> Self::File` capability with `type File:
UniversalRead`. `UniversalRead` no longer extends `UniversalReadFileOps`
and is purely a file-handle trait.
This separates "filesystem instance" from "file handle". Backends that
need per-instance configuration (S3 bucket name + credentials, mmap
default advice, io_uring runtime, block-cache controller `Arc`) gain a
typed home in `Self::ContextConfig`, and `list_files`/`exists`/`open`
become `&self` methods on the filesystem handle.
Concrete filesystem handles introduced for the three existing backends:
- `MmapFs` — unit struct; `ContextConfig = ()`; produces `MmapFile`.
- `IoUringFs` — carries `prevent_caching`; `ContextConfig =
IoUringConfigContext`; produces `IoUringFile`. `IoUringConfigContext`
becomes the construction input rather than a per-open argument.
- `BlockCacheFs` — carries `Arc<CacheController>`; `ContextConfig =
BlockCacheConfigContext`; produces `CachedSlice`.
`TConfigContext` (universal builder methods) is kept so generic-over-`Fs`
code can still set cross-backend knobs via
`Fs::ContextConfig::default().with_prevent_caching(true)`.
Wrappers (`ReadOnly<S>`, `TypedStorage<S, T>`, `StoredStruct<S, T>`),
higher-level storages (`StoredBitSlice<S>`, `UniversalHashMap<K, V, S>`)
keep `<S: UniversalRead>` parameterization but their `open(...)`
constructors now grow a `fs: &Fs` argument bound by
`Fs: UniversalReadFs<File = S>`. `read_json_via` becomes
`read_json_via(fs: &Fs, path)`.
All test code and benches in `common` updated to construct
`MmapFs`/`IoUringFs` inline as needed. `common` compiles cleanly with
tests and benches. `gridstore`, `segment`, and `tonic` caller updates
are in flight in subsequent commits.
* WIP: gridstore + segment caller sweep (partial)
Threads `fs: &Fs` through gridstore's `BitmaskGaps`, `Bitmask`, `Pages`,
and `Gridstore::new`/`open`/`create_new_page`. Most segment callers
have `OpenOptions { extra: ... }` removed and `S::open(path, opts, ctx)`
sites updated mechanically but the trait change is not yet propagated.
Does NOT compile yet. Tracker still has static `S::open` calls, segment
generic constructors (`MmapInvertedIndex<S>`, `UniversalMapIndex`,
`StoredGeoMapIndex`, etc.) still call `S::open`/`S::list_files`/`S::exists`
statically — they need an `fs: &Fs` parameter added. Tonic API
`StorageReadService<S>` also unconverted.
Committed as branch checkpoint; cascade continues in subsequent work.
* gridstore: thread `fs: &Fs` through Bitmask, BitmaskGaps, Pages, Tracker
Per the new `UniversalReadFs` shape, every constructor/method that opens
files takes an `fs: &Fs` parameter. Gridstore is currently mmap-only,
so the top-level `Gridstore` / `GridstoreReader::open` callers in the
crate pass `&MmapFs` inline. Tests do the same.
gridstore lib + tests now compile cleanly. Segment + tonic cascade
still pending.
* WIP: segment caller sweep — dynamic_stored_flags first
* WIP: segment cascade - id_tracker partial
* common benches: update to new UniversalReadFs::open shape (clippy clean)
* segment flags: thread Fs through BufferedDynamicFlags / Bitvec / Roaring
DynamicStoredFlags::set_len now takes `fs: &Fs`. BufferedDynamicFlags
stores an `Arc<Fs>` so the flusher closure can call `set_len` on resize.
BitvecFlags and RoaringFlags expose a new `Fs` type parameter and the
flag tests now pass `Fs::default()` (MmapFs/IoUringFs via duplicate_item).
Concrete consumers (bool/null index, mmap dense/multi/sparse storages)
pin `Fs = MmapFs` and pass `&MmapFs` to inner opens.
* segment: thread Fs through field-index lifecycle methods
Apply the new UniversalReadFs::open shape across:
- full_text_index (MmapInvertedIndex, MmapFullTextIndex, UniversalPostings)
- geo_index (StoredGeoMapIndex build/open + tests + builders)
- numeric_index lifecycle (UniversalNumericIndex build/open)
- map_index lifecycle (UniversalMapIndex build/open)
- stored_point_to_values (open / from_iter)
Concrete consumers pin Fs = MmapFs and pass &MmapFs inline; generic
open paths thread `fs: &Fs` where Fs: UniversalReadFs<File = S>.
* segment: thread Fs through chunked vectors and id-tracker callers
- ChunkedVectors gains `Fs` generic so add_chunk can call create_chunk
after open. ChunkedVectorsRead/load_config and chunks::{read_chunks,
create_chunk} take `fs: &Fs`. Concrete callers (dense / multi-dense /
sparse / quantized) pass MmapFs inline.
- DenseVectorStorageImpl stores `fs: Fs` so `update_from` can reopen
ImmutableDenseVectors. ImmutableDenseVectors::open takes `fs: &Fs`.
- VectorStorageEnum DenseUring* variants thread IoUringFs alongside
IoUringFile.
- segment_builder + segment_constructor_base pass MmapFs to
ImmutableIdTracker::{new, open}.
- QuantizedStorage::from_file takes `fs: &Fs`; quantized_vectors callers
pass &MmapFs.
* segment: apply nightly rustfmt after Fs refactor
cargo +nightly fmt --all over the segment crate after the
UniversalReadFs cascade. No semantic changes.
* segment: thread Fs through benches and id-tracker tests
Update the dynamic-mmap-flags and buffered-update-bitslice benches to
the new UniversalReadFs::open shape (pass `&MmapFs`). Update the
immutable-id-tracker test suite to forward `&MmapFs` to
`from_in_memory_tracker` / `open`.
* uio: pin Fs via UniversalRead::Fs assoc type; per-call OpenExtra
Two design changes that fall out of the per-instance Fs refactor:
1. Bidirectional Fs ↔ File pinning. `UniversalRead::Fs:
UniversalReadFs<File = Self>` lets generic-over-`S` code refer to
`S::Fs` directly instead of carrying an extra `<Fs: UniversalReadFs<File = S>>`
generic param. `ReadOnly<S>` wraps a file but has no natural
filesystem; a phantom `ReadOnlyFs<S::Fs>` satisfies the constraint
while inherent `ReadOnly::open` keeps taking `&S::Fs` directly.
2. `prevent_caching` moves from filesystem-instance state to per-call
`UniversalReadFs::OpenExtra: Default`. Was previously a knob on
`IoUringConfigContext` / `IoUringFs`, conflating "how this fs is
built" with "how this file is opened." Now `IoUringFs::OpenExtra =
IoUringOpenExtra { prevent_caching }`; mmap and block-cache use `()`.
`IoUringConfigContext` is gone, `TConfigContext` slims to a `Default`
marker.
Tonic StorageReadService holds `Arc<S::Fs>` (was `PhantomData<S>`); its
`new()` builds via `S::Fs::from_context(default)` and the spawn_blocking
closures clone the Arc to call instance methods.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: cfg-gate IoUringOpenExtra import for non-linux builds
The IoUringOpenExtra reexport from `universal_io` is gated on
`target_os = "linux"`. The previous commit left an unconditional import
in `persisted_hashmap/tests.rs`, breaking macOS/Windows CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: migrate simple_disk_cache to per-instance Fs API
PR #9097 (merged into dev concurrently with this branch) introduced a
`DiskCache<R: UniversalRead>` using the pre-refactor trait shape:
file-handle-as-filesystem (`R::open`, `R::list_files`), an
`OpenOptionsExtra` field on `OpenOptions`, and trait methods without
`&self`. The Fs-instance refactor on this branch removed all three.
Reshape `simple_disk_cache` to match the new design without breaking the
lazy mirror semantics:
- New `DiskCacheFs<R>` is the filesystem handle. Holds a clone of the
remote `R::Fs`; `list_files`/`exists` delegate; `from_context`
forwards to the inner Fs context. `open` constructs a `DiskCache<R>`
via the global `DiskCacheConfig`.
- `DiskCache<R>` now stores `remote_fs: R::Fs` + `remote_extra:
<R::Fs as UniversalReadFs>::OpenExtra`, so lazy remote opens go
through `self.remote_fs.open(path, options, extra)` instead of the
removed `R::open`. `open_with_config` takes the remote Fs + extra
explicitly (no more hard-coded `prevent_caching: true`; callers pass
the appropriate `OpenExtra`).
- `UniversalRead for DiskCache<R>` now declares `type Fs =
DiskCacheFs<R>` (no more `open` method on the file trait).
- Propagate the necessary bounds (`R::Fs: Clone`,
`<R::Fs as UniversalReadFs>::OpenExtra: Clone`,
`R::OwnedReadPipeline<u8, Range<u32>>: Send`) through `pipeline.rs`
free functions and impl blocks that reach into `DiskCache::remote` /
`local_state`.
- Drop the now-removed `extra: _` destructure in `LocalState::new`.
- Tests construct the remote Fs via `R::Fs::from_context(Default::default())`
and exercise `DiskCache::open_with_config`. 17 simple_disk_cache
tests pass; the 3 `empty_read_does_not_materialize_local_file`
failures pre-exist on dev (verified) and are unrelated.
`mold -run cargo clippy --all-targets` clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt on simple_disk_cache migration
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt for local_state imports after rebase
Co-authored-by: Cursor <cursoragent@cursor.com>
* uio: split DiskCacheFs into its own module
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: replace TConfigContext with OpenExtra trait; move DiskCacheConfig onto DiskCacheFs
- Drop the empty TConfigContext marker; ContextConfig is now unconstrained
so backends can require explicit construction.
- Add OpenExtra trait with with_prevent_caching for backend-agnostic
per-call knobs; impl for () (no-op) and IoUringOpenExtra.
- DiskCacheFs now carries Arc<DiskCacheConfig> via the new
DiskCacheFsContext<C>; the prefill flow moves from the deleted
open_with_config into DiskCacheFs::open so Populate::Blocking /
PreferBackground work through the trait API.
- Remove the DiskCacheConfig global; callers must construct the context
explicitly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt after OpenExtra refactor
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: fix spelling — Implementors → Implementers
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: possible panic insetad of error propagation
* fix: missing cfg annotation
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
Introduce a `pub trait Item: bytemuck::Pod + Send` marker (mirroring the
existing `UserData` pattern) and use it in place of `T: bytemuck::Pod`
on the read side of `UniversalRead`, its pipeline traits, and all
impls/wrappers. Some implementations buffer `Vec<MaybeUninit<T>>` that
may be transferred across threads in future backends; tightening the
bound makes that explicit at the trait level instead of leaving callers
to add `+ Send` ad-hoc. Write paths keep the looser `bytemuck::Pod`
bound since they only borrow `&[T]`.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* dataset: add features to fs-err
Reason: fix `cargo check --all-targets -p dataset`. Feature unification
pulls `fs-err/debug` + `fs-err/tokio` without `fs-err/debug_tokio`.
* dataset: run `cargo metadata` to get target dir
Reason: avoid re-downloading in separate worktrees. The old
`cargo locate-project` ignores custom target dirs.
* sparse/benches/search: cache vectors/indices
The segment builder evicts the page cache of vector storage, quantized
vectors, payload, payload index and the vector index after a build to
avoid cache pollution, but the id tracker was never cleared. Its on-disk
files (mappings, versions, deleted bitslice) are written during the build
and stay resident in the page cache, so after each optimization the id
tracker files linger as cache even though they are meant to be on-disk
only (expected_cache_bytes == 0).
Add `IdTracker::clear_cache` (default no-op) plus a `clear_cache_if_on_disk`
policy wrapper, and implement the eviction for the mutable and immutable
trackers:
- ImmutableIdTracker pages out its two mmap-backed storages via madvise
and drops the RAM-loaded mappings file via fadvise(DONTNEED).
- MutableIdTracker drops its append-only log files via fadvise(DONTNEED).
The builder calls `clear_cache_if_on_disk`, mirroring the payload index.
The id tracker has no on-disk mode yet, so this always clears for now;
a TODO marks where to gate it once an on-disk mode exists.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
