* segment: move proxy pending change types into segment crate
Move the types describing the changes a proxy segment buffers — point
deletes (`ProxyDeletedPoint`), payload index changes (`ProxyIndexChange`,
`ProxyIndexChanges`) and vector name changes (`IntendedVector`,
`ProxyVectorNameChanges`) — from `shard::proxy_segment` into a new
`segment::pending_changes` module.
Pure move, no behavior change: the proxy segment re-exports them from
their old location. Having them in the segment crate lets both the proxy
segment and the segment load path share them, in preparation for
persisting pending proxy changes to disk and replaying them on restart.
* segment: add PendingChange describing a persisted proxy operation
Add the `PendingChange` enum with one variant per operation type a proxy
segment buffers — point delete, payload index change, vector name
change — each carrying the operation version it was issued with. This
is the shape in which pending proxy changes are persisted to disk.
Derive serde on it and on the buffered change types it embeds, so
entries can be serialized into a log file and read back. `PartialEq` on
those types lets a persisted batch be matched against the in-memory
pending buffer after a flush.
* segment: add PendingChanges component persisting proxy changes to a log
Add `PendingChanges`, the component that manages the operations a proxy
segment buffers for one proxy layer, and persists them to disk so they
no longer only live in memory.
It keeps the same per-type buffers the proxy segment served its reads
from (point deletes, payload index changes, vector name changes), plus a
single registration-ordered buffer of everything not yet persisted.
`flusher()` writes that buffer into an append-only log file inside the
wrapped segment's directory: `pending_changes.log` for the inner most
proxy layer, with the layer number as a suffix for each layer above it.
Appends follow the mutable ID tracker: all new entries are serialized
into one buffer and written with a single call on an append-mode file,
then fsynced, so a crash can only leave a torn entry at the very end.
Loading truncates such an entry — its operations were never durable and
thus never acknowledged in the WAL — but fails hard on a malformed entry
in the middle, which cannot be explained by a torn append.
The component tracks the highest operation version the log covers.
Every registered operation at or below it is either durable in the log
or was a no-op that does not need recovery; a flusher advances it to the
proxy's version even when there is nothing to write. The pending buffer
is deliberately not cleared when the proxy propagates its changes to
the wrapped segment, as that only makes them durable once the wrapped
segment flushes. Replaying an entry twice is a version-gated no-op.
A log file left behind by a previous proxy on the same segment is
adopted by `open()`: new entries are appended after it and its highest
version is taken over, while its entries are not loaded into the
buffers as they are already applied to the segment. `load()` also
reconstructs the buffers, for callers that do want the buffered state.
* segment: replay persisted pending proxy changes onto a segment on load
Add `recover_pending_changes`, to be called when a segment is loaded on
restart, before regular WAL replay. If the segment directory holds
pending changes log files, the proxies that wrote them did not
propagate their buffered state into the segment before the process
stopped. Instead of reconstructing the proxies, replay all logged
operations directly onto the segment: inner most proxy layer first,
each file in append order, through the regular version-gated segment
operations (`apply_change`). Entries the segment already applied are
silently skipped, so a stale file is harmless.
The segment is force-flushed before the files are removed; a crash in
between merely replays the files once more.
* segment: test PendingChanges component
Cover the pending changes component: registering and flushing each
operation type and reconstructing the buffers from the log, log file
naming per proxy layer and gap-tolerant listing, covering the proxy
version without entries, operations registered while a flusher is
captured, flushers of a dropped component, torn-tail truncation versus
mid-file corruption, adoption of an existing log, and replaying logs
onto a real segment: fresh, stale (already applied), multi-layer, and
vector name changes.
* segment: include pending changes logs in segment snapshots
Register the pending changes log files of a segment in its snapshot:
add them to `snapshot_files` next to the segment state and version
files, existence-guarded, and to the segment manifest as unversioned
files. Full, partial and streamed snapshots therefore all carry them.
The recovery side needs no changes: a restored segment is loaded like
any other, which replays and removes the logs.
* shard: back proxy segment pending changes by PendingChanges component
Replace the proxy segment's separate `deleted_points`, `changed_indexes`
and `changed_vector_names` fields with a single `PendingChanges`
component. Reads keep going through the same per-type buffers, now
behind accessors; writes go through the component's `register_*`
methods, which additionally queue every operation for persistence.
Opening the component is fallible, as it adopts a pending changes log
a previous proxy may have left in the wrapped segment's directory, so
`UnsyncedProxySegment::new` now returns a result. Wrapping another proxy
opens the next proxy layer up, writing to its own dedicated log file.
No behavior change yet: the proxy still flushes and reports persistence
exactly as before, nothing is written to the log.
* shard: persist proxy pending changes on flush, stop holding back WAL ack
Hook the pending changes component into the proxy segment's flush: the
proxy flusher first persists the buffered operations into the pending
changes log, then passes the flush along to the wrapped segment. The
proxy's `persistent_version` now covers what the log durably holds on
top of what the wrapped segment persisted itself.
That is what lifts the WAL cap proxies imposed so far. `flush_all`
compares each segment's version against its persistent version; a
proxy used to report only the wrapped segment's persisted version while
its own version climbed with every buffered operation, so the WAL could
never be acknowledged past the point the proxy was created at, and a
restart replayed all of it — potentially very expensive operations,
such as an update by filter, all over again. With the buffered state
durable on disk the generic rule acknowledges the full version, and a
restart recovers it from the log instead.
Dropping a proxy's data drops the component first, which waits for any
in-flight pending changes flusher so it cannot append to the segment
directory while that is being deleted.
Update the proxy flush test to the new semantics, add a segment holder
test asserting the acknowledged version advances past a proxied delete,
and update the ack pin rationale in `finish_optimization`: the pin is
still needed after the proxies leave the holder, it just snapshots a
persistent version that now includes the log.
* shard: propagate proxy changes when unwrapping on optimizer cancel
When an optimization is cancelled or fails, `unwrap_proxy` puts the
wrapped segments back into the segment holder. Propagate the changes
buffered in each proxy into its wrapped segment first, as the snapshot
unproxy path already does, instead of dropping them with the proxy.
The pending changes log is deliberately left in place when unwrapping:
deleting it before the wrapped segment has flushed the propagated
changes would not be crash safe. It is cleaned up on restart and when
the segment directory is dropped, and a new proxy on the same segment
adopts and appends to it; replaying a stale file is safe because all
operations are version gated.
* shard: test persisted proxy pending changes
Test the proxy segment against its persisted pending changes: buffered
changes survive dropping the proxy without propagation and are replayed
onto the segment when it is loaded again; unwrapping leaves the log in
place and a new proxy on the same segment adopts and appends to it;
layered proxies each persist into their own log file and a restart
replays both; and a persisted log is part of the segment manifest and
snapshot.
* collection, edge: recover persisted proxy changes on segment load
Replay the pending changes logs left behind by proxy segments onto each
segment when a shard loads its segments, right after consistency
repair and before the payload index rebuild, vector name reconciliation
and WAL replay. Proxy state that made it to disk no longer holds back
the WAL acknowledge, so this is where it must be recovered from.
Proxies are not reconstructed: the segment holder starts with plain
segments carrying the replayed operations, and the logs are removed
once the segment flushed them.
* collection: test crash recovery through persisted proxy changes
End-to-end test of the persisted pending changes: wrap every segment of
a local shard in a proxy, delete points so the deletes are only
buffered, flush, and assert the acknowledgeable version covers them.
Then acknowledge the WAL up to that version, drop the shard without
ever propagating the proxies, and load it again: the deletes are gone
from the WAL and must come back through the pending changes logs.
The delete under test is deliberately not the last WAL entry, as the
acknowledge never passes the last entry and that one is always
replayed.
* segment: make replaying persisted proxy changes on load an explicit mode
Add `PersistedProxyChanges` to state whether persisted pending proxy
changes are replayed onto a segment when it is loaded. `Replay`, the
default, recovers them and removes the logs as before. `Ignore` leaves
both the segment and the log files untouched and logs at debug level
that replaying was skipped; it is for segment files that mirror those
of another writer, where replaying would make the local copy diverge
from what the writer's manifest describes.
All callers pass `Replay` for now, no behavior change.
* collection: do not replay persisted proxy changes on partial snapshot recovery
Partial snapshots are recovered by read replicas in a read/write
segregation setup. A read replica must not mutate its segments, so it
cannot replay the persisted proxy segment changes on load and must
ignore them instead: its segment files are a local copy of the writer's
that must stay a faithful mirror of them, as later partial snapshots are
diffed against what the writer's manifest describes. Replaying would
mutate the segment files and remove the logs, making the copy diverge.
Thread the replay mode through `LocalShard::load` as a dedicated
`PersistedProxyChanges` argument, derived from the recovery type:
`RecoveryType::Full` replays as before, `RecoveryType::Partial` ignores
the persisted changes and leaves the logs in place. Regular shard loads
replay.
Extend the crash recovery test with an ignoring load first: the delete
under test must not come back and the logs must survive, before a
replaying load recovers it.
* Persist wrapped segment before pending changes
Prevents raising version of proxy segment too early
* Fix comment
* Fix crash window, only ready optimized segment after propagating changes
The optimizer renamed a newly built segment into segments_path and wrote
its version file before finish_optimization propagated the proxies'
buffered changes into it. A crash in that window left the segment
restart-loadable but stale, permanently losing or resurrecting points.
Defer the version file save until finish_optimization has fully
reconciled proxy changes into the segment, including the post-swap dedup
pass, so it stays invisible to restart and snapshot recovery until then.
SegmentBuilder::build() gains a `ready` flag; load_segment gains
`ignore_missing_version` for the one caller reloading before that point.
Incidentally also closes the crash-unsafe cancellation-orphan cleanup
gap noted in #9217, since a cancelled build is discarded on restart the
same way.
* Force flush optimized segment, otherwise we may lose proxy changes
* Don't force flush after replay, defer deleting log files until flush
* Include persisted proxy changes log file in segment manifest
* Add random ID to proxy log files, prevent instance conflicts
* Rename proxy log file, always include level
* Delete proxy log file on unproxy, defer until next flush cycle
* Fix truncation
* Reformat
* Lock persisted segments behind runtime feature flag
* Enable necessary feature flags in tests
* Fix linters
* feat(debug): QDRANT_APPEND_ONLY_MUTATIONS env override
Debug-only escape hatch so newly built segments default to append-only
mutation routing when QDRANT_APPEND_ONLY_MUTATIONS=1 (or true/yes) is
set in the environment. Lets us run the existing test suites against
the append-only path without wiring a collection-level config knob
first.
Release builds compile this out — the function is a const false.
Logs a single warn-level message the first time the override fires.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(append-only): skip deleted vectors on snapshot, always write payload
Two correctness fixes to clone_and_mutate_point uncovered by running
the openapi suite with QDRANT_APPEND_ONLY_MUTATIONS=1:
1. The snapshot loop read every named vector via get_vector_opt, which
returns slot bytes even when the per-vector deletion bit is set.
For sparse-only points (or any point that had update_vector(_, None)
applied) this materialised the default-zero vector as if it were
real data, wrote it at new_id, and never re-tombstoned the slot —
so dense search started scoring phantom vectors. Now we check
is_deleted_vector(old_id) and skip the read, letting the writer
loop emit update_vector(new_id, None) and re-mark the slot deleted.
2. The payload write was skipped when the snapshot ended up empty.
That dropped two side effects the field indexes rely on:
payload_storage.overwrite(new_id, empty), and the remove_point
fan-out across configured field indexes that bumps each index's
total_point_count to cover new_id. Without the bump the null index
doesn't see new_id, so is_empty / is_null filters lose the point
even though its mapping is live in the id tracker. The skip was an
optimisation, not a contract; remove it so the field indexes get
the same registration they'd get from the standard clear_payload
path.
Also collapses the debug env override to an inline cfg!()-gated
check at the struct literal — the helper with one-time logging and
multi-value matching was disproportionate for a debug-only escape
hatch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(id_tracker): split active vs deferred maps in PointMappings (PR A)
First step of moving the mutable id tracker to a "two-track" model so
append-only mutations into a deferred segment can keep both the
visible (active) version and the latest mutated (deferred) version of
a point at the same time.
This PR is purely structural. On-disk format is unchanged: the loader
still produces a single combined map, and `PointMappings::new`
partitions it at construction time based on `deferred_internal_id`.
Every observable behaviour at the existing API surface is preserved.
Concretely:
- New fields: `external_to_internal_num_deferred`,
`external_to_internal_uuid_deferred`, and a `shadowed: BitVec` for
future use by PR B (lazy-grown, default-false).
- `internal_id(ext)` checks active first, falls through to deferred —
matches the pre-split "any matching id" contract for ext ids whose
internal id sat above the cutoff.
- `set_link(ext, new_id)` now routes by cutoff: writes below the
cutoff land in active, writes at or above land in deferred. Any
prior head in the other track is tombstoned, so each ext still owns
exactly one slot — same observable result as the pre-split
single-map insert. PR B replaces the cross-track tombstone with a
shadow-bit flip; PR A keeps current semantics on purpose.
- `drop(ext)` clears entries from both tracks and tombstones each
one, again matching the prior single-map behaviour.
- `iter_external` and `iter_from` merge the active and deferred
BTreeMap views into one sorted-by-key stream (dedup'd in case an
ext exists in both tracks).
- `available_point_count` counts distinct external ids across both
tracks — preserves the prior observable count for segments where
some entries used to sit above the cutoff in the single map.
No write-path or read-path behaviour change. Reads still filter
`internal_id >= cutoff` exactly as before; mutations still tombstone
prior heads. The shadow bit and the deferred-aware lookup wiring
land in PR B and PR C.
All existing id_tracker tests pass against the split layout.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(id_tracker): shadow active head on deferred writes (PR B)
Step 2 of the deferred-aware mutable id tracker. Replaces PR A's
"tombstone the other track" cross-track cleanup with shadow-bit logic
so a deferred mutation no longer hides the visible active version.
Behaviour by case:
- Active write (`internal_id < cutoff`, or no cutoff): unchanged
semantically. Any prior deferred head for the same ext is dropped
and tombstoned (the new active is the single visible head). Same
observable result as PR A.
- Deferred write (`internal_id >= cutoff`): the prior deferred head
(if any) is dropped and tombstoned. The active head, if it exists,
is **shadowed** — its bit is set in `shadowed: BitVec` but its slot
stays alive in the active map. Read paths in `Exclude` mode
continue to return that active version; PR C will teach
`IncludeAll` paths (the optimiser) to skip shadowed actives and
prefer the deferred head.
Also adds `is_shadowed(internal_id)` and `shadowed_bitslice()`
accessors (the latter for PR C's filter pipeline). New unit tests
cover the routing matrix:
- no cutoff — active replacement, no shadow,
- below-cutoff replacement — active path, no shadow,
- deferred-on-top-of-active — shadow set, active retained,
- two deferred writes — prior deferred tombstoned, shadow persists,
- fresh insert above cutoff — no shadow,
- `drop(ext)` clears both tracks plus the shadow bit.
WAL replay reuses the same `set_link`, so deferred routing on
replay falls out of this change with no extra wiring.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(id_tracker): IncludeAll skips shadowed actives, deferred-aware lookups (PR C)
Step 3 of the deferred-aware id tracker stack. PR B introduced the
shadow bit on `set_link`; this PR teaches the read paths to honour it
so the optimiser (and any other `DeferredBehavior::IncludeAll`
consumer) sees each external id exactly once, with the deferred head
winning over the now-shadowed active.
Concretely:
- `PointMappings::internal_id_with_behavior(ext, behavior)`:
* `Exclude` returns the active head only — `None` for a
deferred-only ext, so query paths never see a deferred mutation;
* `IncludeAll` prefers the deferred head and falls back to active
for points that never crossed the cutoff. Yields at most one
internal id per ext.
- `IdTrackerRead::internal_id_with_behavior` mirrors the new method
with a default impl that delegates to `internal_id` for trackers
that don't carry deferred mutations.
- `PointMappingsRefEnum::iter_internal_with_behavior(IncludeAll)`
now filters shadowed actives via the new
`PointMappings::shadowed_bitslice()` accessor.
- `PointMappingsRefEnum::filter_deferred_and_deleted(IncludeAll)`
also filters shadowed actives — same single-yield-per-external
guarantee for external iterator sources like field-index outputs.
- `IdTrackerRead::resolve_external_ids` switches to the deferred-aware
lookup. No more post-lookup `id >= cutoff` filter — the behaviour
enum lookup gets it right at the source.
New unit tests cover the two new entry points:
- IncludeAll prefers the deferred head when an active is shadowed;
- Exclude returns None for deferred-only ext ids;
- `filter_deferred_and_deleted` over a mixed candidate list yields
the expected per-mode result (Exclude: actives below cutoff;
IncludeAll: every visible head, no shadowed actives).
No queries observable behaviour change today — production Exclude
paths still resolve via `internal_id` and the active head. The
optimiser will start using `IncludeAll` (and reap the dedup) in
follow-up work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(id_tracker): expose shadowed_point_count in SegmentInfo (PR D)
Final step of the deferred-aware id tracker stack. Adds a single new
counter — number of active heads currently shadowed by a deferred
mutation — and plumbs it through the id-tracker trait, the segment
read-view helper, and `SegmentInfo` for telemetry.
Preserves existing semantics:
- `available_point_count` is unchanged. Each distinct external id
still counts once, regardless of which track holds its head.
- `deferred_point_count`, `deferred_internal_id`,
`num_deleted_deferred_points` keep their current values.
- Non-appendable trackers default `shadowed_point_count()` to `0`,
so the new `SegmentInfo.num_shadowed_points` is `None` for them.
Concrete changes:
- `PointMappings::shadowed_count()` — popcount of the shadowed
bitslice.
- `IdTrackerRead::shadowed_point_count()` trait method with a `0`
default; wired through `MutableIdTracker`, `InMemoryIdTracker`,
the mutable read-only tracker, and both enum dispatchers.
- `SegmentReadView::shadowed_point_count()` helper.
- New `SegmentInfo.num_shadowed_points: Option<usize>`, populated
with `Some(_)` for appendable segments and proxied through
`ProxySegment` from the wrapped segment's value.
New unit test covers the counter lifecycle:
- active-only writes don't grow it,
- a deferred write over an active adds one shadow,
- a second deferred write supersedes the prior deferred head but
the shadow stays put (still one active being shadowed),
- `drop(ext)` clears the shadow bit,
- a fresh deferred insert with no active prior doesn't add a shadow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(id_tracker): rename DeferredBehavior, push behavior down to PointMappings
Three connected pieces of polish on top of the deferred-aware id
tracker stack:
1. Rename `DeferredBehavior::Exclude` → `VisibleOnly` and
`IncludeAll` → `WithDeferred`. The pre-PR-C semantics of
"include/exclude the deferred cutoff" became misleading once
`IncludeAll` started skipping shadowed actives — it doesn't
"include all" anymore, it yields one slot per external (deferred
head preferred, active fallback). New names describe what each
variant returns instead of how it relates to the cutoff. The
helper method `include_all_points` becomes
`with_deferred_points`. Updates ~90 call sites across the
workspace; behaviour is unchanged.
2. Push `iter_internal_with_behavior` down from
`PointMappingsRefEnum` into `PointMappings`. The per-mode
logic (cutoff `take_while`, shadowed `filter`) now lives next
to the data it consults; the enum layer becomes a two-arm
`Either` dispatcher. `CompressedPointMappings` short-circuits
to `iter_internal()` since compressed mappings can't carry
deferred mutations.
3. Add a short docstring on `internal_to_external` describing the
two-track model: no active-vs-deferred bias, shadowed pairs
occupy two slots with the same value, and reads must gate on
`deleted` because `set_link`'s same-track replacement leaves a
real-looking stale ext id in place.
Returning `impl Iterator` instead of `Box<dyn Iterator>` for
`iter_internal`, `iter_internal_excluding`, `iter_internal_visible`,
and `iter_internal_with_behavior` removes the double-Box at the
enum boundary. The original branching structure is preserved with
`itertools::Either` instead of restructured into one big filter
chain.
All existing `id_tracker` tests still pass (47 total).
`cargo check --all-targets` + `cargo clippy --all-targets` clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(id_tracker): drop shadowed API surface, unbox iter_from/iter_random
Cleanup pass on top of the deferred-aware id tracker stack.
API surface:
- Drop SegmentInfo.num_shadowed_points + its trait method, read-view
helper, tracker impls and ProxySegment passthrough. Cross-segment
shadow already exists in the appendable update flow (the source
segment keeps its copy when the new write lands above the cutoff,
see SegmentHolder::apply_points_with_conditional_move) and is
handled at the aggregation layer, not reported per-tracker.
- Keep PointMappings::shadowed_count() as a private popcount helper
guarded by expect(dead_code) so a future caller can opt back into
the dedup'd count without re-plumbing the trait.
available_point_count:
- Replace the cross-track dedup (active + deferred-only via
contains_key filter) with a straight 4-term sum. A shadowed ext
contributes two slots, matching the two non-tombstoned internal
ids it actually occupies. Old dedup broke the invariant that
deleted_point_count == deleted_bitslice.count_ones(): for each
shadow it overcounted deletions by one without any tombstone
actually being set.
DeferredBehavior pushdown:
- iter_random_with_behavior: caps the sampling range at the
deferred threshold in VisibleOnly mode (no wasted samples above
cutoff), filters shadowed actives via the bit in WithDeferred.
- iter_from_with_behavior: VisibleOnly walks the active maps only
(no merge with deferred); WithDeferred delegates to iter_from's
existing merge.
- scroll.rs read_by_id_stream / filtered_read_by_id_stream collapse
their manual if-deferred-behavior branching into a single
iter_from_with_behavior call.
- Old iter_random (no behavior) at PointMappings + ref enum was
unused after the migration, deleted.
Unboxing iter_from / iter_random / iter_from_with_behavior:
- PointMappings::iter_from returns impl Iterator + '_ via Either
inside the merged_num/merged_uuid closures (BTreeMap::iter vs
range), Either at the outer match (num+uuid chain vs uuid-only).
- PointMappings::iter_from_with_behavior unboxed with a triple
Either (behavior, external-id arm, closure start).
- CompressedPointMappings::iter_from unboxed (Either over None/Some).
- PointMappingsRefEnum::{iter_from, iter_from_with_behavior,
iter_from_visible} all return impl Iterator + 'a via Either on
the Plain/Compressed dispatch.
Other:
- Update outdated PR-A/PR-B comment on PointMappings::drop.
- Make the max_internal match in iter_random_with_behavior
exhaustive (VisibleOnly+None | WithDeferred+_ instead of `_`).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(id_tracker): unbox iter_external
Return impl Iterator from PointMappings::iter_external,
CompressedPointMappings::iter_external and the PointMappingsRefEnum
wrapper, matching the style of the other iter_* helpers. The wrapper
dispatches via Either. The remaining Box::new at Segment::iter_points
stays because self_cell's BoxedPointIdIterator alias needs a sized
type.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(id_tracker): count set_link tombstones in deferred_deleted_count
PointMappings::set_link tombstones the prior slot at two sites — when
an active write supersedes a deferred head (cross-track) and when any
write replaces a same-track head — using a bare deleted.set(old, true).
Neither path updated deferred_deleted_count, so tombstones above the
cutoff weren't reflected in the counter.
The append-only flow exposes this constantly: every set_full_payload
after upsert_point routes through clone_and_mutate_point, which
re-issues set_link with a fresh internal id and tombstones the prior
one. Most tombstones land above the cutoff, so deferred_point_count
(total - cutoff - deferred_deleted_count) over-reports by the missing
count. The openapi test_deferred_points integration test caught this
as `num_points - num_deferred_points = -1800` across two segments.
Extracted the tombstone bookkeeping into PointMappings::tombstone_slot
and routed both set_link sites + drop's loop through it. Behaviour on
drop is unchanged; set_link now bumps the counter on the
live → tombstoned transition for slots at or above the cutoff.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style(id_tracker): collapse set_link tombstone if-let chains
Clippy's collapsible_if on the two if-let blocks added by the
deferred_deleted_count fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update lib/segment/src/id_tracker/point_mappings.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* refactor(id_tracker): size shadowed BitVec once instead of growing lazily
Collect the shadowed active ids up front and allocate the BitVec to the
highest offset in a single resize, avoiding repeated reallocations while
marking shadows. Addresses review feedback.
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert: restore lazily-grown shadowed BitVec
Roll back the up-front sizing of the shadowed BitVec; last_entry doesn't
fit here and swapping one allocation for another isn't worthwhile.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(id_tracker): prefer deferred head in iter_from merge
When an external id has both an active and a deferred head, iter_from's
merge collapsed the pair to the active (stale) offset. That contradicts
the WithDeferred contract used everywhere else: internal_id_with_behavior
and iter_random_with_behavior both surface the deferred head (the latest
mutation) over the shadowed active. Consumers that use the returned
internal id (payload-filter checks, the optimizer's version merge, HNSW
old->new mapping) therefore saw the stale copy.
Flip the Both arm to take the deferred operand and document the rule.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(id_tracker): preserve active+deferred heads when loading mappings
read_mappings replayed the persisted change log into a single flat
external -> internal map per id-type, then split it by the deferred
cutoff in PointMappings::new. A flat map holds one head per external id,
so it collapsed the active+deferred coexistence case (an external id
linked first to an active slot, then to a deferred one via sequential
set_link) down to the last write — silently dropping the other head,
orphaning its slot, and leaving the shadowed bit unset. The split in
new() could not recover what was already lost before it.
Replay the log through the canonical set_link/drop mutators on a
PointMappings seeded with the cutoff instead. The log is the sequence of
set_link/drop calls that produced the live in-memory state, so this
reconstructs that state exactly — both heads, the shadowed bit, and
deferred_deleted_count — with no logic duplication. Drops the
debug_assert-guarded corruption-recovery branch (subsumed by set_link's
re-link handling) and the now-unused Uuid/PointIdType imports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(id_tracker): make deferred behavior explicit at point resolution
Replace the ambiguous "any matching head" point resolution with an
explicit DeferredBehavior at every resolution boundary, so callers no
longer rely on a hidden active-vs-deferred policy.
- Remove PointMappings::internal_id and the bare IdTrackerRead::internal_id
(the active-first-else-deferred hybrid). internal_id_with_behavior is now
the single required resolution method; immutable/compressed trackers
implement it by ignoring the behavior (they never carry deferred heads).
- Migrate every caller to an explicit behavior, audit-driven:
- writes (upsert/delete/payload/vectors), point_version, point_is_deferred,
get_internal_id, drop, consistency + builder dedup -> WithDeferred (the
latest/live head);
- single-point payload/vector retrieval and formula rescore -> VisibleOnly;
- HasId/CustomIdChecker/cardinality resolution -> the request's behavior,
threaded through the filter chain from iter_filtered_points (other entry
points default to VisibleOnly).
- lookup_internal_id takes an explicit DeferredBehavior instead of assuming
VisibleOnly internally.
- has_point takes an explicit DeferredBehavior (drop the has_point_with_behavior
wrapper). Thread it through read_points/_read_points/read_points_locked so
retrieve_blocking passes its request behavior to the existence filter; all
other existence/dedup callers pass WithDeferred (unchanged behavior).
- set_link now detaches a stale live occupant of a reused internal id, keeping
the forward and reverse maps consistent when recovering a corrupted log.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(id_tracker): failing tests for two correctness findings in #9249 (#9311)
* test(id_tracker): failing tests for two review findings
Two intentionally-failing tests pinning correctness gaps in the
deferred-aware id tracker (#9249). Both fail on the final assertion;
the earlier assertions establish the expected/consistent behavior.
1. iter_from_with_behavior(WithDeferred) resolves an active/deferred
shadow collision to the stale ACTIVE internal id, while its siblings
internal_id_with_behavior and iter_internal_with_behavior correctly
surface the DEFERRED (latest) head. Consumers that use the yielded
internal id (optimizer merge via for_each_unique_point,
filtered_read_by_id_stream) therefore observe the pre-mutation
version.
left: [(NumId(7), 2)] right: [(NumId(7), 9)]
2. The PR-B shadow/visible invariant is not durable: the on-disk single
combined map cannot represent a shadowed ext, so a plain mappings
flush + reload collapses the shadow to deferred-only and the visible
(active) head is lost (VisibleOnly resolves None where it resolved
Some(2) live). Restoration then depends entirely on WAL replay, i.e.
on flush-vs-WAL-truncate ordering.
left: None right: Some(2)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(id_tracker): strengthen shadow tests + merge-primitive proof
Follow-up to the two failing tests, addressing self-review:
- Add for_each_unique_point_keeps_deferred_head_for_shadowed_point: the
optimizer merge primitive (used by segment_builder::update_from) yields
the stale active copy (internal 2, version 5) for a shadowed point and
drops the deferred latest (internal 9, version 8). This directly
exercises the data-loss consequence of finding #1 at the merge layer.
left: [(NumId(7), 2, 5)] right: [(NumId(7), 9, 8)]
- Tighten shadow_visible_head_survives_mapping_flush_reload: pin the exact
reload failure mode. After flush+reload the mapping collapses to
deferred-only (internal_id == Some(9)) and the active slot survives as a
live orphan in the inverse map (external_id(2) == Some(7), not deleted) —
a torn state where a VisibleOnly scroll still surfaces the stale copy
while by-id VisibleOnly resolution breaks. Reframe as the live-vs-reload
divergence the PR introduces (Some(2) live -> None reload; dev is
consistently None).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix test
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Refactor: `SearchSegmentEntry` for search ops
* Move `has_deferrend_points_method`
to `SearchSegmentEntry`
* Reorg imports
* Renamings per review
* fixup! Renamings per review
* `StorageSegmentEntry` trait
It contains methods dealing with storage and syncronization
* Move index methods from `ReadSegmentEntry`
to `SegmentEntry`
* Add `_concurrent` suffix to some methods
* move field index methods into NonAppendableSegmentEntry
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
* Split `SegmentEntry`
Add `ImmutableSegmentEntry` for operations that can be applied to
immutable segments, and make the `SegmentEntry` as it subtrait.
* Rename to `NonAppendableSegmentEntry`
It differs semantically from `ImmutableSegmentEntry` by allowing point
deletion. Move point deletion to the trait too.
* Fix docstring
* use NonAppendableSegmentEntry where possible
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Replace `Option<Segment>` with `enum LoadSegmentOutcome`
* Replace some Path/PathBuf with str/String
* Rename field Segment::{current_path -> segment_path}
* safe_delete
* flush all segments in one thread sequentially
* bonus: move flush-related functions into dedicated file
* Minor comment tweaks
* await for flush on segment holder level
* fmt
* Minor improvement, preallocate vector for payload index flushers
* Remove invalid comment
---------
Co-authored-by: timvisee <tim@visee.me>
* Bump Rust edition to 2024
* gen is a reserved keyword now
* Remove ref mut on references
* Mark extern C as unsafe
* Wrap unsafe function bodies in unsafe block
* Geo hash implements Copy, don't reference but pass by value instead
* Replace secluded self import with parent
* Update execute_cluster_read_operation with new match semantics
* Fix lifetime issue
* Replace map_or with is_none_or
* set_var is unsafe now
* Reformat
* bump and migrate to rand 0.9.0
also bump rand_distr to 0.5.0 to match it
* Migrate AVX2 and SSE implementations
* Remove unused thread_rng placeholders
* More random migrations
* Migrate GPU tests
* bump seed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Measure read io for payload storage
* Add Hardware Counter to update functions
* Fix tests and benches
* Rename (some) *_measured functions back to original
* minor conversion improvement
* use NamedVectors in update_vectors
* remove merge from VectorStruct
* rename Multi -> Named in vector struct
* add multi-dense vectors option into VectorStruct
* generate openapi
* rename VectorStruct -> VectorStructInternal
* add conversion for anonymous multivec in grpc
* renames for BatchVectorStruct
* implement multi-dense for batch
* allow multi-dense in batch upserts
* test and fixes
* Add min_should field in Filter struct
* min_should clause checks whether at least given number (min_count) of conditions are met
* modify test cases due to change in Filter struct (set min_should: None)
* add simple condition check unit test
* docs, cardinality estimation, grpc not implemented yet
* Add min_should field in Filter struct
* min_should clause checks whether at least given number (min_count) of conditions are met
* modify test cases due to change in Filter struct (set min_should: None)
* add simple condition check unit test
* Impl min_should clause in REST API
* perform cardinality estimation by estimating cardinalities of intersection and combining as union
* add openapi spec with docs update
* add integration test
* Impl min_should clause in gRPC
* Cargo fmt & clippy
* Fix minor comments
* add equivalence test between min_should and must
* shortcut at min_count matches
* use `Filter::new_*` whenever possible
* Add missing min_should field
* Fix gRPC field ordering & remove deny_unknown_fields
* Empty commit
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* add enum for vector query on segment search
* rename newly introduced types
* fix: handle QueryVector on async scorer
* handle QueryVector in QuantizedVectors impl
* fix async scorer test after refactor
* rebase + refactor on queue_proxy_shard.rs
* constrain refactor propagation to segment_searcher
* fmt
* fix after rebase
* add ignore_plain_index to speed up search
* remove unnecessary & for vectors_batch
* format
* add special handle for proxy segments where the wrapped segment is plain
indexed
* review refactoring
* rollback changes in google.protobuf.rs
---------
Co-authored-by: Di Zhao <diz@twitter.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* pass atomic bool from local shard to raw scorer
* pass atomic bool from local shard to raw scorer
* is_stopped in async scorer
* fmt
* is_stopped in quantized scorer
* terminating scorer if stopped
* enable timeout in local_shard
* allow timeout configuration
* use tokio spawn to ensure timeout handling if request is dropped
* Revert "use tokio spawn to ensure timeout handling if request is dropped"
This reverts commit 1068cf48d4.
* use stopping guard instead of task
* report error if search request is stopped
* fmt
* refactor transient error handelling
Tim Visée