* feat: read-only edge shard follower (ReadOnlyEdgeShard)
Add a read-only follower of EdgeShard, mirroring the segment-level
ReadOnlySegment + live_reload design one level up at the shard. A leader
process owns a read-write EdgeShard; one or more followers open the same
on-disk directory as a ReadOnlyEdgeShard, serve reads, and periodically
refresh() to pick up the leader's flushed writes and optimizations.
Shared read logic lives once in a crate-internal EdgeReadView<H>, generic
over a ReadSegmentHandle: the follower is monomorphized over the concrete
ReadOnlySegment<S> (no dynamic dispatch), while the read-write shard's
heterogeneous Segment/ProxySegment holder uses the LockedSegment enum
(the only dyn ReadSegmentEntry left, mirroring LockedSegment::get_read).
EdgeShardRead is the public read API: it exposes the read methods
(search/query/retrieve/count/facet/scroll/info/...) as default methods,
requiring implementors only to provide read_segments() + config_snapshot().
EdgeShard keeps its inherent read methods for backwards compatibility.
Segment discovery is injected via a SegmentEnumerator (temporary seam):
LocalSegmentEnumerator scans the segments/ directory for the local/mmap
case; S3 followers supply their own. This will be replaced by an on-disk
segment manifest in follow-up work.
shard: add LockedSegment::get_read_arc(); split retrieve_blocking into a
handle-collecting wrapper + generic retrieve_over; generalize the private
_read_points over the segment type (public signatures unchanged); remove
the now-unused read_points_locked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor
* feat: manifest-based segment loading for ReadOnlyEdgeShard
Replace the read-only follower's directory-scan discovery with the segment
manifest from the leader, the proper mechanism the SegmentEnumerator seam
was a placeholder for.
shard: add SegmentsManifest::load so readers can read manifest.json.
edge (leader): EdgeShard now writes segments/manifest.json (gated by the
write_segment_manifest feature flag), initialized from the live segment
set on new/load and refreshed after optimize() swaps segments.
edge (follower): ManifestSegmentEnumerator reads the manifest's `active`
segments instead of scanning, and open_mmap uses it. It requires a manifest
and errors when none is present (no silent scan fallback); discover by
scanning explicitly via open() with a LocalSegmentEnumerator instead.
Since the manifest only reports ready segments (and future versions will
mark segments retiring/under-construction rather than removing them from
active immediately), the read path no longer races with the leader, so the
defensive `is_transient_open_error` skip handling is removed — a failure to
open a reported segment is now a genuine error and propagates.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor
* feat: edge-s3-scroll experiment binary for ReadOnlyEdgeShard over S3
Adds a standalone `edge-s3-scroll` binary that opens a ReadOnlyEdgeShard
directly over an S3 (or S3-compatible) bucket and runs a single scroll
request with a hard-coded filter, for experimentation.
Takes the bucket endpoint, credentials and key prefix as CLI flags/env
vars, builds an S3-backed BlobFs, and discovers segments via a custom
enumerator that reads the segment manifest over object storage. The
edge_config.json is fetched to a local temp dir because
ReadOnlyEdgeShard::open reads config from the local filesystem.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* filter config in s3 scroll tool
* fix: read segment manifest from new location after dev rebase
dev #9564 moved the segment manifest next to (rather than inside) the
segments/ directory and named it segments_manifest.json. Adapt the
read-only follower enumerators accordingly so the follower reads the
manifest where the leader now writes it:
- ManifestSegmentEnumerator reads via segment_manifest_path() (shard root)
while still resolving segment dirs under segments/<uuid>.
- The S3 scroll tool's enumerator mirrors the same layout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: read-only edge shard works over object storage without edge_config
Make ReadOnlyEdgeShard self-sufficient over arbitrary backends (S3/GCS),
and harden the read-only segment loading it depends on.
- ReadOnlyEdgeShard derives its config from the segments via
EdgeConfig::from_segment_config instead of requiring edge_config.json
(open and refresh both derive); a follower never has that file.
- ReadOnlyEdgeShard::open(fs, path) no longer takes an enumerator: a
read-only follower always discovers segments through the manifest.
open_with_enumerator remains as a pub(crate) seam for tests.
- ManifestSegmentEnumerator is generic over UniversalReadFs (reads the
manifest via read_json_via), so it works over local mmap or a blob/S3
backend; the tool's bespoke enumerator is removed.
- Read-only mutable ID tracker tolerates absent mappings/versions files
(they are not written while empty), matching MutableIdTracker::open:
files are opened lazily and NotFound is treated as empty, avoiding an
extra exists() round-trip on object storage.
edge-s3-scroll experiment tool:
- Supports AWS S3 / S3-compatible and GCS backends (--backend), with a
hard-coded-free --filter-key/--filter-value scroll filter.
- Reads segment data through a DiskCache so remote blocks are fetched
once and served from a local mirror afterwards.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: remove unused SegmentsManifest::load
Its only caller (edge's ManifestSegmentEnumerator) now reads the manifest
via read_json_via over UniversalReadFs, so the local-only load helper is
dead. Drop it and its now-unused read_json/Path imports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(bm25): add explicit Disabled stemmer; deprecate language hack
Adds a `Disabled` variant to `StemmingAlgorithm` (`stemmer: {"type": "none"}`)
so stemming can be turned off explicitly in both the main engine and Edge,
instead of relying on the undocumented `language: "none"` footgun that
silently disabled both stemming and stopwords.
For language-neutral text processing the supported setup is now:
1. set the stemmer to disabled, and
2. configure an empty stopword set.
The main engine still tolerates unsupported languages (so existing
`language: "none"` configs keep working on upgrade) but now logs a
deprecation warning pointing users to the explicit setup. Edge continues
to reject unsupported languages, and now has a real way to disable stemming.
Refs: https://github.com/qdrant/qdrant/issues/9289
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(edge-py): handle Disabled stemmer in python bindings; fix openapi schema
- Handle the new StemmingAlgorithm::Disabled variant in the qdrant-edge-py
bindings (FromPyObject/IntoPyObject/Repr) and add a DisabledStemmer pyclass
plus its .pyi stub entry.
- Match generator output for the StemmingAlgorithm OpenAPI schema (plain $ref
in anyOf) so docs/redoc/master/openapi.json stays consistent.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openapi): regenerate StemmingAlgorithm schema with generator output
Ran tools/generate_openapi_models.sh so docs/redoc/master/openapi.json
exactly matches generator output: DisabledStemmerParams/NoStemmer are placed
after SnowballLanguage, and the StemmingAlgorithm anyOf entry is a plain $ref
(the schema2openapi step flattens the allOf+description wrapper).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): avoid wildcard enum match arm in bm25 sparse_len helper
clippy --all-targets flags `other => panic!()` as wildcard_enum_match_arm;
match the Dense/MultiDense variants explicitly instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: issues
* fix: log::warn as call once
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* feat(debug): QDRANT_APPEND_ONLY_MUTATIONS env override
Debug-only escape hatch so newly built segments default to append-only
mutation routing when QDRANT_APPEND_ONLY_MUTATIONS=1 (or true/yes) is
set in the environment. Lets us run the existing test suites against
the append-only path without wiring a collection-level config knob
first.
Release builds compile this out — the function is a const false.
Logs a single warn-level message the first time the override fires.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(append-only): skip deleted vectors on snapshot, always write payload
Two correctness fixes to clone_and_mutate_point uncovered by running
the openapi suite with QDRANT_APPEND_ONLY_MUTATIONS=1:
1. The snapshot loop read every named vector via get_vector_opt, which
returns slot bytes even when the per-vector deletion bit is set.
For sparse-only points (or any point that had update_vector(_, None)
applied) this materialised the default-zero vector as if it were
real data, wrote it at new_id, and never re-tombstoned the slot —
so dense search started scoring phantom vectors. Now we check
is_deleted_vector(old_id) and skip the read, letting the writer
loop emit update_vector(new_id, None) and re-mark the slot deleted.
2. The payload write was skipped when the snapshot ended up empty.
That dropped two side effects the field indexes rely on:
payload_storage.overwrite(new_id, empty), and the remove_point
fan-out across configured field indexes that bumps each index's
total_point_count to cover new_id. Without the bump the null index
doesn't see new_id, so is_empty / is_null filters lose the point
even though its mapping is live in the id tracker. The skip was an
optimisation, not a contract; remove it so the field indexes get
the same registration they'd get from the standard clear_payload
path.
Also collapses the debug env override to an inline cfg!()-gated
check at the struct literal — the helper with one-time logging and
multi-value matching was disproportionate for a debug-only escape
hatch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(id_tracker): split active vs deferred maps in PointMappings (PR A)
First step of moving the mutable id tracker to a "two-track" model so
append-only mutations into a deferred segment can keep both the
visible (active) version and the latest mutated (deferred) version of
a point at the same time.
This PR is purely structural. On-disk format is unchanged: the loader
still produces a single combined map, and `PointMappings::new`
partitions it at construction time based on `deferred_internal_id`.
Every observable behaviour at the existing API surface is preserved.
Concretely:
- New fields: `external_to_internal_num_deferred`,
`external_to_internal_uuid_deferred`, and a `shadowed: BitVec` for
future use by PR B (lazy-grown, default-false).
- `internal_id(ext)` checks active first, falls through to deferred —
matches the pre-split "any matching id" contract for ext ids whose
internal id sat above the cutoff.
- `set_link(ext, new_id)` now routes by cutoff: writes below the
cutoff land in active, writes at or above land in deferred. Any
prior head in the other track is tombstoned, so each ext still owns
exactly one slot — same observable result as the pre-split
single-map insert. PR B replaces the cross-track tombstone with a
shadow-bit flip; PR A keeps current semantics on purpose.
- `drop(ext)` clears entries from both tracks and tombstones each
one, again matching the prior single-map behaviour.
- `iter_external` and `iter_from` merge the active and deferred
BTreeMap views into one sorted-by-key stream (dedup'd in case an
ext exists in both tracks).
- `available_point_count` counts distinct external ids across both
tracks — preserves the prior observable count for segments where
some entries used to sit above the cutoff in the single map.
No write-path or read-path behaviour change. Reads still filter
`internal_id >= cutoff` exactly as before; mutations still tombstone
prior heads. The shadow bit and the deferred-aware lookup wiring
land in PR B and PR C.
All existing id_tracker tests pass against the split layout.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(id_tracker): shadow active head on deferred writes (PR B)
Step 2 of the deferred-aware mutable id tracker. Replaces PR A's
"tombstone the other track" cross-track cleanup with shadow-bit logic
so a deferred mutation no longer hides the visible active version.
Behaviour by case:
- Active write (`internal_id < cutoff`, or no cutoff): unchanged
semantically. Any prior deferred head for the same ext is dropped
and tombstoned (the new active is the single visible head). Same
observable result as PR A.
- Deferred write (`internal_id >= cutoff`): the prior deferred head
(if any) is dropped and tombstoned. The active head, if it exists,
is **shadowed** — its bit is set in `shadowed: BitVec` but its slot
stays alive in the active map. Read paths in `Exclude` mode
continue to return that active version; PR C will teach
`IncludeAll` paths (the optimiser) to skip shadowed actives and
prefer the deferred head.
Also adds `is_shadowed(internal_id)` and `shadowed_bitslice()`
accessors (the latter for PR C's filter pipeline). New unit tests
cover the routing matrix:
- no cutoff — active replacement, no shadow,
- below-cutoff replacement — active path, no shadow,
- deferred-on-top-of-active — shadow set, active retained,
- two deferred writes — prior deferred tombstoned, shadow persists,
- fresh insert above cutoff — no shadow,
- `drop(ext)` clears both tracks plus the shadow bit.
WAL replay reuses the same `set_link`, so deferred routing on
replay falls out of this change with no extra wiring.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(id_tracker): IncludeAll skips shadowed actives, deferred-aware lookups (PR C)
Step 3 of the deferred-aware id tracker stack. PR B introduced the
shadow bit on `set_link`; this PR teaches the read paths to honour it
so the optimiser (and any other `DeferredBehavior::IncludeAll`
consumer) sees each external id exactly once, with the deferred head
winning over the now-shadowed active.
Concretely:
- `PointMappings::internal_id_with_behavior(ext, behavior)`:
* `Exclude` returns the active head only — `None` for a
deferred-only ext, so query paths never see a deferred mutation;
* `IncludeAll` prefers the deferred head and falls back to active
for points that never crossed the cutoff. Yields at most one
internal id per ext.
- `IdTrackerRead::internal_id_with_behavior` mirrors the new method
with a default impl that delegates to `internal_id` for trackers
that don't carry deferred mutations.
- `PointMappingsRefEnum::iter_internal_with_behavior(IncludeAll)`
now filters shadowed actives via the new
`PointMappings::shadowed_bitslice()` accessor.
- `PointMappingsRefEnum::filter_deferred_and_deleted(IncludeAll)`
also filters shadowed actives — same single-yield-per-external
guarantee for external iterator sources like field-index outputs.
- `IdTrackerRead::resolve_external_ids` switches to the deferred-aware
lookup. No more post-lookup `id >= cutoff` filter — the behaviour
enum lookup gets it right at the source.
New unit tests cover the two new entry points:
- IncludeAll prefers the deferred head when an active is shadowed;
- Exclude returns None for deferred-only ext ids;
- `filter_deferred_and_deleted` over a mixed candidate list yields
the expected per-mode result (Exclude: actives below cutoff;
IncludeAll: every visible head, no shadowed actives).
No queries observable behaviour change today — production Exclude
paths still resolve via `internal_id` and the active head. The
optimiser will start using `IncludeAll` (and reap the dedup) in
follow-up work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(id_tracker): expose shadowed_point_count in SegmentInfo (PR D)
Final step of the deferred-aware id tracker stack. Adds a single new
counter — number of active heads currently shadowed by a deferred
mutation — and plumbs it through the id-tracker trait, the segment
read-view helper, and `SegmentInfo` for telemetry.
Preserves existing semantics:
- `available_point_count` is unchanged. Each distinct external id
still counts once, regardless of which track holds its head.
- `deferred_point_count`, `deferred_internal_id`,
`num_deleted_deferred_points` keep their current values.
- Non-appendable trackers default `shadowed_point_count()` to `0`,
so the new `SegmentInfo.num_shadowed_points` is `None` for them.
Concrete changes:
- `PointMappings::shadowed_count()` — popcount of the shadowed
bitslice.
- `IdTrackerRead::shadowed_point_count()` trait method with a `0`
default; wired through `MutableIdTracker`, `InMemoryIdTracker`,
the mutable read-only tracker, and both enum dispatchers.
- `SegmentReadView::shadowed_point_count()` helper.
- New `SegmentInfo.num_shadowed_points: Option<usize>`, populated
with `Some(_)` for appendable segments and proxied through
`ProxySegment` from the wrapped segment's value.
New unit test covers the counter lifecycle:
- active-only writes don't grow it,
- a deferred write over an active adds one shadow,
- a second deferred write supersedes the prior deferred head but
the shadow stays put (still one active being shadowed),
- `drop(ext)` clears the shadow bit,
- a fresh deferred insert with no active prior doesn't add a shadow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(id_tracker): rename DeferredBehavior, push behavior down to PointMappings
Three connected pieces of polish on top of the deferred-aware id
tracker stack:
1. Rename `DeferredBehavior::Exclude` → `VisibleOnly` and
`IncludeAll` → `WithDeferred`. The pre-PR-C semantics of
"include/exclude the deferred cutoff" became misleading once
`IncludeAll` started skipping shadowed actives — it doesn't
"include all" anymore, it yields one slot per external (deferred
head preferred, active fallback). New names describe what each
variant returns instead of how it relates to the cutoff. The
helper method `include_all_points` becomes
`with_deferred_points`. Updates ~90 call sites across the
workspace; behaviour is unchanged.
2. Push `iter_internal_with_behavior` down from
`PointMappingsRefEnum` into `PointMappings`. The per-mode
logic (cutoff `take_while`, shadowed `filter`) now lives next
to the data it consults; the enum layer becomes a two-arm
`Either` dispatcher. `CompressedPointMappings` short-circuits
to `iter_internal()` since compressed mappings can't carry
deferred mutations.
3. Add a short docstring on `internal_to_external` describing the
two-track model: no active-vs-deferred bias, shadowed pairs
occupy two slots with the same value, and reads must gate on
`deleted` because `set_link`'s same-track replacement leaves a
real-looking stale ext id in place.
Returning `impl Iterator` instead of `Box<dyn Iterator>` for
`iter_internal`, `iter_internal_excluding`, `iter_internal_visible`,
and `iter_internal_with_behavior` removes the double-Box at the
enum boundary. The original branching structure is preserved with
`itertools::Either` instead of restructured into one big filter
chain.
All existing `id_tracker` tests still pass (47 total).
`cargo check --all-targets` + `cargo clippy --all-targets` clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(id_tracker): drop shadowed API surface, unbox iter_from/iter_random
Cleanup pass on top of the deferred-aware id tracker stack.
API surface:
- Drop SegmentInfo.num_shadowed_points + its trait method, read-view
helper, tracker impls and ProxySegment passthrough. Cross-segment
shadow already exists in the appendable update flow (the source
segment keeps its copy when the new write lands above the cutoff,
see SegmentHolder::apply_points_with_conditional_move) and is
handled at the aggregation layer, not reported per-tracker.
- Keep PointMappings::shadowed_count() as a private popcount helper
guarded by expect(dead_code) so a future caller can opt back into
the dedup'd count without re-plumbing the trait.
available_point_count:
- Replace the cross-track dedup (active + deferred-only via
contains_key filter) with a straight 4-term sum. A shadowed ext
contributes two slots, matching the two non-tombstoned internal
ids it actually occupies. Old dedup broke the invariant that
deleted_point_count == deleted_bitslice.count_ones(): for each
shadow it overcounted deletions by one without any tombstone
actually being set.
DeferredBehavior pushdown:
- iter_random_with_behavior: caps the sampling range at the
deferred threshold in VisibleOnly mode (no wasted samples above
cutoff), filters shadowed actives via the bit in WithDeferred.
- iter_from_with_behavior: VisibleOnly walks the active maps only
(no merge with deferred); WithDeferred delegates to iter_from's
existing merge.
- scroll.rs read_by_id_stream / filtered_read_by_id_stream collapse
their manual if-deferred-behavior branching into a single
iter_from_with_behavior call.
- Old iter_random (no behavior) at PointMappings + ref enum was
unused after the migration, deleted.
Unboxing iter_from / iter_random / iter_from_with_behavior:
- PointMappings::iter_from returns impl Iterator + '_ via Either
inside the merged_num/merged_uuid closures (BTreeMap::iter vs
range), Either at the outer match (num+uuid chain vs uuid-only).
- PointMappings::iter_from_with_behavior unboxed with a triple
Either (behavior, external-id arm, closure start).
- CompressedPointMappings::iter_from unboxed (Either over None/Some).
- PointMappingsRefEnum::{iter_from, iter_from_with_behavior,
iter_from_visible} all return impl Iterator + 'a via Either on
the Plain/Compressed dispatch.
Other:
- Update outdated PR-A/PR-B comment on PointMappings::drop.
- Make the max_internal match in iter_random_with_behavior
exhaustive (VisibleOnly+None | WithDeferred+_ instead of `_`).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactor(id_tracker): unbox iter_external
Return impl Iterator from PointMappings::iter_external,
CompressedPointMappings::iter_external and the PointMappingsRefEnum
wrapper, matching the style of the other iter_* helpers. The wrapper
dispatches via Either. The remaining Box::new at Segment::iter_points
stays because self_cell's BoxedPointIdIterator alias needs a sized
type.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(id_tracker): count set_link tombstones in deferred_deleted_count
PointMappings::set_link tombstones the prior slot at two sites — when
an active write supersedes a deferred head (cross-track) and when any
write replaces a same-track head — using a bare deleted.set(old, true).
Neither path updated deferred_deleted_count, so tombstones above the
cutoff weren't reflected in the counter.
The append-only flow exposes this constantly: every set_full_payload
after upsert_point routes through clone_and_mutate_point, which
re-issues set_link with a fresh internal id and tombstones the prior
one. Most tombstones land above the cutoff, so deferred_point_count
(total - cutoff - deferred_deleted_count) over-reports by the missing
count. The openapi test_deferred_points integration test caught this
as `num_points - num_deferred_points = -1800` across two segments.
Extracted the tombstone bookkeeping into PointMappings::tombstone_slot
and routed both set_link sites + drop's loop through it. Behaviour on
drop is unchanged; set_link now bumps the counter on the
live → tombstoned transition for slots at or above the cutoff.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style(id_tracker): collapse set_link tombstone if-let chains
Clippy's collapsible_if on the two if-let blocks added by the
deferred_deleted_count fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update lib/segment/src/id_tracker/point_mappings.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* refactor(id_tracker): size shadowed BitVec once instead of growing lazily
Collect the shadowed active ids up front and allocate the BitVec to the
highest offset in a single resize, avoiding repeated reallocations while
marking shadows. Addresses review feedback.
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert: restore lazily-grown shadowed BitVec
Roll back the up-front sizing of the shadowed BitVec; last_entry doesn't
fit here and swapping one allocation for another isn't worthwhile.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(id_tracker): prefer deferred head in iter_from merge
When an external id has both an active and a deferred head, iter_from's
merge collapsed the pair to the active (stale) offset. That contradicts
the WithDeferred contract used everywhere else: internal_id_with_behavior
and iter_random_with_behavior both surface the deferred head (the latest
mutation) over the shadowed active. Consumers that use the returned
internal id (payload-filter checks, the optimizer's version merge, HNSW
old->new mapping) therefore saw the stale copy.
Flip the Both arm to take the deferred operand and document the rule.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(id_tracker): preserve active+deferred heads when loading mappings
read_mappings replayed the persisted change log into a single flat
external -> internal map per id-type, then split it by the deferred
cutoff in PointMappings::new. A flat map holds one head per external id,
so it collapsed the active+deferred coexistence case (an external id
linked first to an active slot, then to a deferred one via sequential
set_link) down to the last write — silently dropping the other head,
orphaning its slot, and leaving the shadowed bit unset. The split in
new() could not recover what was already lost before it.
Replay the log through the canonical set_link/drop mutators on a
PointMappings seeded with the cutoff instead. The log is the sequence of
set_link/drop calls that produced the live in-memory state, so this
reconstructs that state exactly — both heads, the shadowed bit, and
deferred_deleted_count — with no logic duplication. Drops the
debug_assert-guarded corruption-recovery branch (subsumed by set_link's
re-link handling) and the now-unused Uuid/PointIdType imports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(id_tracker): make deferred behavior explicit at point resolution
Replace the ambiguous "any matching head" point resolution with an
explicit DeferredBehavior at every resolution boundary, so callers no
longer rely on a hidden active-vs-deferred policy.
- Remove PointMappings::internal_id and the bare IdTrackerRead::internal_id
(the active-first-else-deferred hybrid). internal_id_with_behavior is now
the single required resolution method; immutable/compressed trackers
implement it by ignoring the behavior (they never carry deferred heads).
- Migrate every caller to an explicit behavior, audit-driven:
- writes (upsert/delete/payload/vectors), point_version, point_is_deferred,
get_internal_id, drop, consistency + builder dedup -> WithDeferred (the
latest/live head);
- single-point payload/vector retrieval and formula rescore -> VisibleOnly;
- HasId/CustomIdChecker/cardinality resolution -> the request's behavior,
threaded through the filter chain from iter_filtered_points (other entry
points default to VisibleOnly).
- lookup_internal_id takes an explicit DeferredBehavior instead of assuming
VisibleOnly internally.
- has_point takes an explicit DeferredBehavior (drop the has_point_with_behavior
wrapper). Thread it through read_points/_read_points/read_points_locked so
retrieve_blocking passes its request behavior to the existence filter; all
other existence/dedup callers pass WithDeferred (unchanged behavior).
- set_link now detaches a stale live occupant of a reused internal id, keeping
the forward and reverse maps consistent when recovering a corrupted log.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(id_tracker): failing tests for two correctness findings in #9249 (#9311)
* test(id_tracker): failing tests for two review findings
Two intentionally-failing tests pinning correctness gaps in the
deferred-aware id tracker (#9249). Both fail on the final assertion;
the earlier assertions establish the expected/consistent behavior.
1. iter_from_with_behavior(WithDeferred) resolves an active/deferred
shadow collision to the stale ACTIVE internal id, while its siblings
internal_id_with_behavior and iter_internal_with_behavior correctly
surface the DEFERRED (latest) head. Consumers that use the yielded
internal id (optimizer merge via for_each_unique_point,
filtered_read_by_id_stream) therefore observe the pre-mutation
version.
left: [(NumId(7), 2)] right: [(NumId(7), 9)]
2. The PR-B shadow/visible invariant is not durable: the on-disk single
combined map cannot represent a shadowed ext, so a plain mappings
flush + reload collapses the shadow to deferred-only and the visible
(active) head is lost (VisibleOnly resolves None where it resolved
Some(2) live). Restoration then depends entirely on WAL replay, i.e.
on flush-vs-WAL-truncate ordering.
left: None right: Some(2)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(id_tracker): strengthen shadow tests + merge-primitive proof
Follow-up to the two failing tests, addressing self-review:
- Add for_each_unique_point_keeps_deferred_head_for_shadowed_point: the
optimizer merge primitive (used by segment_builder::update_from) yields
the stale active copy (internal 2, version 5) for a shadowed point and
drops the deferred latest (internal 9, version 8). This directly
exercises the data-loss consequence of finding #1 at the merge layer.
left: [(NumId(7), 2, 5)] right: [(NumId(7), 9, 8)]
- Tighten shadow_visible_head_survives_mapping_flush_reload: pin the exact
reload failure mode. After flush+reload the mapping collapses to
deferred-only (internal_id == Some(9)) and the active slot survives as a
live orphan in the inverse map (external_id(2) == Some(7), not deleted) —
a torn state where a VisibleOnly scroll still surfaces the stale copy
while by-id VisibleOnly resolution breaks. Reframe as the live-vs-reload
divergence the PR introduces (Some(2) live -> None reload; dev is
consistently None).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix test
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Use assert_matches!
* Add trailing commas
* Use more assert_matches!
Also, drop now redundant `expected blah but got blah` messages because
`assert_matches!` will print these.
* Use debug_assert_matches!
---------
Co-authored-by: xzfc <xzfcpw@gmail.com>
* Add Rust edge example: add-named-vector
Port `lib/edge/python/examples/add-named-vector.py` to Rust under
`lib/edge/publish/examples/src/bin/add-named-vector.rs`.
Re-export `VectorNameOperations`, `CreateVectorName`, `DeleteVectorName`,
`VectorNameConfig`, `DenseVectorConfig`, and `SparseVectorConfig` from
`qdrant_edge` so the public Rust API can create/delete named vectors
without reaching into the internal `shard` crate.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fmt
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci(windows): skip IO-heavy tests that aren't OS-specific
On the Windows CI runner, several tests are 3-25x slower than on Ubuntu
purely due to slow filesystem IO. These tests exercise platform-agnostic
logic (optimizer, snapshot, WAL recovery, dedup, deferred points) and
are fully covered by the Linux and macOS jobs.
Mark them with `#[cfg_attr(target_os = "windows", ignore = "...")]` so:
- Windows CI skips them and finishes faster.
- They're still listed and runnable via `cargo test -- --ignored`
on Windows for local debugging.
Based on JUnit timings from CI run 26462785436 (PR #8827), this should
save ~5 minutes wall-clock on the Windows job, taking it closer to the
~13min Ubuntu and ~9min macOS jobs (currently 20m24s).
Tests affected:
- lib/wal: check_wal, check_last_index, check_clear, check_reopen,
check_truncate, check_prefix_truncate, test_prefix_truncate_parametric
- lib/edge/optimize: full tests module
- lib/segment deferred-point tests: read_operations,
dense_segment_combinations, sparse, facets
- lib/collection: snapshot_test, points_dedup, wal_recovery,
collection_test::test_ordered_read_api, snapshot_recovery_test
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert(ci/windows): keep WAL and WAL-recovery tests on Windows
Reviewer correctly pointed out that WAL is mmap-backed and has
substantial Windows-specific code paths:
- Different segment allocation (fs4 vs rustix::ftruncate)
- Windows-specific delete_windows() with mmap-drop + retry loop
- Windows-specific sync_all() because directory fsync is unavailable
- Windows-specific lock proxy file (directories aren't lockable)
So those tests genuinely need Windows coverage. Reverted skips for:
- lib/wal/src/lib.rs: all check_* tests and test_prefix_truncate_parametric
- lib/collection/src/tests/wal_recovery_test.rs: all three tests
Still skipped on Windows (no OS-specific code in their production paths):
- lib/edge/optimize.rs (no cfg(windows) in source)
- lib/segment deferred-point tests (segment/ has no cfg(windows))
- lib/collection snapshot/dedup tests (collection/ has no cfg(windows))
- lib/collection integration snapshot_recovery + ordered_read_api
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert(ci/windows): keep collection integration and snapshot_test
Per reviewer request, keep running these on Windows:
- lib/collection/tests/integration/* (snapshot_recovery_test,
collection_test::test_ordered_read_api)
- lib/collection/src/tests/snapshot_test.rs
These exercise higher-level collection/snapshot behavior that benefits
from cross-platform validation.
Remaining Windows skips (production code has no cfg(windows) branches):
- lib/edge/src/optimize.rs: 14 optimizer tests
- lib/segment/src/segment/tests/mod.rs: 4 deferred-point tests
- lib/collection/src/tests/points_dedup.rs: 2 dedup tests
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci(windows): also skip HNSW/quantization integration tests
Per reviewer, also skip these segment integration test modules on Windows:
- hnsw_quantized_search_test::* (25 tests)
- multivector_filtrable_hnsw_test::* (rstest cases)
- multivector_quantization_test::* (rstest cases)
- byte_storage_quantization_test::* (rstest cases)
- payload_index_test::test_struct_payload_index_nested_fields
These exercise pure HNSW/quantization correctness on top of standard
segment IO that is already covered by tests we keep running on Windows.
Adds ~930s of sequential time to the Windows skip list, bringing the
expected wall-clock saving from ~3 min to ~8-10 min.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(edge): expose WAL options via EdgeShard::load_with_wal_options
Adds a sibling method to EdgeShard::load that accepts custom
WalOptions, alongside the existing load() which keeps using
default_wal_options().
Motivation: embedded/mobile deployments (e.g. Flutter plugins on
iOS/Android) need much smaller WAL segments than the 32 MiB default —
typically 4 MiB. On filesystems with sparse files (APFS/ext4/F2FS)
the physical footprint is small, but the visible/reported size is
the full segment capacity, which is surfaced by iCloud backup, OS
size pickers, etc.
Changes:
* New EdgeShard::load_with_wal_options(path, config, wal_options).
* EdgeShard::load delegates to it with default_wal_options() — no
behavior change for existing callers.
* ensure_dirs_and_open_wal now takes WalOptions explicitly; called
from both EdgeShard::new (with default) and load_with_wal_options.
* WalOptions re-exported from edge crate root.
* Two regression tests in lib/edge/tests/wal_options.rs.
WAL options are intentionally a runtime parameter, not part of
EdgeConfig (which is persisted to edge_config.json) — different
processes may legitimately open the same shard with different WAL
options.
* test(edge): verify mismatching WAL options on reload preserve data
Addresses upstream review feedback (qdrant/qdrant#9067): demonstrate
that reloading an existing shard with WAL options different from the
ones it was created with is safe and preserves all previously written
points.
Two new tests in lib/edge/tests/wal_options.rs:
* reload_with_smaller_wal_capacity_after_upsert:
create shard with default 32 MiB WAL -> upsert point 42 ->
drop -> reload with 4 MiB WAL options -> point 42 still readable ->
upsert point 43 under smaller WAL -> count == 2.
* reload_with_larger_wal_capacity_after_upsert (symmetric):
create -> reload with 4 MiB -> upsert point 100 -> drop ->
reload with default 32 MiB -> point 100 readable -> upsert
point 101 -> count == 2.
Both pass. The WAL crate does not validate segment_capacity on
reload — existing segments on disk keep their original size,
subsequent appends honor the runtime options. This confirms the
PR description's claim that WalOptions is a runtime hint, not
persisted shard state.
* style(edge): cargo fmt for wal_options.rs mismatch tests
* refactor(edge): replace load_with_wal_options with builder-based options
Addresses upstream feedback (timvisee, generall): the sibling
constructor pattern doesn't scale as runtime configurability grows.
Replaces `EdgeShard::load_with_wal_options(path, config, wal_options)`
with `EdgeShard::load_with_options(path, config, EdgeShardOptions)`,
where `EdgeShardOptions` is a builder-style runtime-options struct.
* `EdgeShardOptions::new().with_wal_options(opts)` is the equivalent
of the old call.
* Adding new runtime options later (e.g. wal flush interval, initial
indexing threshold) is now an additive change — new builder method
on `EdgeShardOptions`, no new constructor variant on `EdgeShard`.
* `EdgeShardOptions` is intentionally not persisted (no
Serialize/Deserialize). The reload-mismatch tests above confirm
that WAL options are a runtime hint, not shard identity, so they
don't belong in `EdgeConfig` (which is persisted to
`edge_config.json`).
* `EdgeShard::load(path, config)` unchanged for existing callers —
delegates to `load_with_options(..., EdgeShardOptions::default())`.
Tests in `lib/edge/tests/wal_options.rs` migrated to the new shape;
all four still pass:
test load_with_options_accepts_custom_wal_capacity ... ok
test load_still_works_with_default_wal_options ... ok
test reload_with_smaller_wal_capacity_after_upsert ... ok
test reload_with_larger_wal_capacity_after_upsert ... ok
* refactor(edge): add fluent builders; move wal_options into EdgeConfig
Replaces the EdgeShardOptions side-channel with a single EdgeConfig that
carries wal_options inline, and introduces fluent builders for the three
user-facing config types.
* WalOptions now derives Clone/PartialEq/Eq/Serialize/Deserialize so it
can live inside EdgeConfig and round-trip through edge_config.json.
* EdgeConfig.wal_options (Option<WalOptions>) replaces EdgeShardOptions.
EdgeShard::load_with_options is folded into EdgeShard::load; both new
and load drive the WAL from config.wal_options.unwrap_or_default().
* New builders/ module hosts EdgeConfigBuilder, EdgeVectorParamsBuilder,
and EdgeSparseVectorParamsBuilder. Each builder has explicit per-field
storage and constructs its target via an exhaustive struct literal in
build(), so adding a field to the target forces a compile error in the
builder.
* publish example switched to the new builder API.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style(edge): cargo fmt after builder refactor
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous description was outdated: it claimed that enabling this
option "blocks updates at the request level" until segments are
re-optimized. In practice the implementation uses "deferred points":
new points written to large unoptimized segments are persisted but
excluded from read/search results until the segments are optimized.
Updates are not blocked; only `wait=true` clients are made to wait for
the deferred points to become visible. Update this in the REST schema
(via `OptimizersConfig` / `OptimizersConfigDiff`), in the gRPC proto,
in the edge config docstrings, and regenerate the OpenAPI bundle via
`tools/generate_openapi_models.sh`.
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add empty placeholder vector storage types for named vector CRUD
Introduce EmptyDenseVectorStorage and EmptySparseVectorStorage as
placeholder storages for newly created named vectors on immutable
segments. These report all vectors as deleted, consume no disk space,
and are reconstructed from segment config on load via the new
VectorStorageType::Empty and SparseVectorStorageType::Empty variants.
Key design decisions:
- is_on_disk is derived from original user config, not hardcoded
- MultiVectorConfig is preserved for multi-vector support
- Config mismatch optimizer skips Empty storage to avoid false rebuilds
- Quantization delegates normally (handles 0 vectors gracefully)
- get_vector includes debug_assert to catch unexpected access
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] segment-level operations for creating and deleting anmed vectors
* [AI] implement named vector creation and deleting in proxy segment
* [AI] Step 3: Proxy Segment Handling for Named Vector Operations
* [AI] implement for Edge
* [AI] implement consensus operations for named vector operations
* [AI] refactor VectorNameConfig, remove VectorNameConfigInternal
* [AI] handle vector schema inconsistency in raft snapshot recovery
* [AI] rest + grpc API
* [AI] clippy
* [AI] generate openAPI schema
* fmt
* ci fixes
* [AI] fix jwt access test
* [AI] nop operation for awaiting of consensus-commited update ops
* [AI] move vector name operations into points service
* [AI] implement internal api for vector name operations
* [AI] change collection-level config along with segment level operation
* [AI] vector schema reconceliation instead of error
* fmt
* missing compile-time option
* [AI] integration test
* [AI] fix missing JWT tests
* [AI] remove NOP
* [AI] openapi test
* [AI] fix initialization of mutable segment
* [AI] more simple integration tests
* fmt
* [AI] make cluster test a bit harder
* [AI] make test less flacky
* [AI] rabbit comments
* [AI] check params compatibility before writing vector config
* [AI] make sure to register vector storages in structure payload index
* [AI] vector name validation
* lower vector length validation to 200 chars to account for prefix in filename
* [AI] proxy segment: prevent stale data leak through optimization
* fmt
* [AI] filter out removed vectors from proxy response
* [AI] handle vector name in proxy
* fmt
* adjust proxy info based on dropped vectors
* [AI] proxy segment: update filters to correct has_vector condition
* fmt
* clippy
* Fix consensus snapshot applicaiton for vector schema
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix edge sparse vector search panic on score postprocessing
The distance lookup for score postprocessing only checked dense vector
configs, causing a panic when searching sparse vectors. Fall back to
Distance::Dot for sparse vectors, matching the full server behavior.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix formatting
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Address PR review: return error instead of panic, fix example
- Replace panic! with OperationError::service_error for unknown vector
names in edge search, avoiding process crash on bad client input
- Update stale "panics" comments and add assertions in sparse-search example
Made-with: Cursor
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <agent@cursor.com>
* Refactor: `SearchSegmentEntry` for search ops
* Move `has_deferrend_points_method`
to `SearchSegmentEntry`
* Reorg imports
* Renamings per review
* fixup! Renamings per review
* `StorageSegmentEntry` trait
It contains methods dealing with storage and syncronization
* Move index methods from `ReadSegmentEntry`
to `SegmentEntry`
* Add `_concurrent` suffix to some methods
* move field index methods into NonAppendableSegmentEntry
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Don't account for deferred points in some places
# Conflicts:
# lib/collection/src/shards/local_shard/scroll.rs
* Add in QueryContext
* Cover more places
* Coderabbit review remarks
* Properly count amount of deleted deferred points (#8386)
* Properly count amount of deleted deferred points
* Prevent double-counting of the same point
* Remove hints to estimations
* Properly handle counts in ProxySegment
* Add tests and fix deleted point count issue
* Adjusts tests + fix issues
* Separte fields for deferred points in telemetry (SegmentInfo)
* Remove deferred_points_count()
* openapi
* Fix test by manually calculating visible points
* Adjust ProxySegment test to revertion of SegmentInfo
* Throw error if collection was not found in telemetry (e2e Test)
* Update lib/segment/src/segment/segment_ops.rs
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Make new fields in API optional
* Don't take range if no deferred point exist
* Review remarks
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* refactor(edge): split EdgeShard load into new() and load()
- new(path, config): create edge shard only when path has no existing segments
- load(path, config?): load from existing files; config optional (load from
edge_config.json or infer from segments)
- Helpers: has_existing_segments, ensure_dirs_and_open_wal, resolve_initial_config,
load_segments, ensure_appendable_segment
- Python: EdgeShard.create_new(path, config); __init__ still calls load()
- Examples use new() for creation and load(path, None) for reopen
- Error instead of panic on config mismatch; explicit load/create in Python
- fix: use OptimizerThresholds.deferred_internal_id for dev compatibility
Made-with: Cursor
* rollback threshold changes
* fix(edge): address dancixx review comments
- Persist inferred config in load() when edge_config.json does not exist
(SaveOnDisk::new only saves when file exists)
- Python: add #[new] delegating to load() for backward compatibility
- qdrant_edge.pyi: Optional return types for deleted_threshold,
vacuum_min_vector_number, default_segment_number; add __init__ doc
Made-with: Cursor
* Revert "fix(edge): address dancixx review comments"
This reverts commit 370e21a6c74c41210e1658f89814395cb86ed81c.
* fix: optional returns
* fix: save config it is not exist
* fix: save data on disk always
* fix: python examples
* chore: remove edge.close()
* fix: wal lock
* fix: rename of EdgeShardConfig -> EdgeConfig
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* refactor(edge): let edge re-export segment/shard/sparse
* chore(ast-grep-rules): handle path like `::shard::blah::Blah`
This commit was required during the refactoring. It turns out we don't
need it right now, but it might be useful in the future. Or not.
* refactor(edge-rs): Rehaul public API
* Add ignore_deferred flag for internal scroll API
* Ignore deferred points in ProxyShard::update
* Use enum instead of bool as type in function parameters
* Use consistent parameter name
* fmt
* Apply suggestions from code review
Co-authored-by: Tim Visée <tim+github@visee.me>
* More explicit naming
* Add DeferredBehavior to `count()` and disable filtering in stream_records
* Don't filter `retrieve()` everywhere
* Use consistent filter behavior in read operations
* Add TODO for ignored parameter in remote_shard
* Rebase fixes
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Adjust points selection for deferred points update
* adjust proxy segment implementation
* simplify
* use simpler proxy impl
* stick to Entry API
* renaming to stay closer to the original
* two passes and simpler impl.
* fmt
* fmt
* use predefined deferred internal id
* calculate deferred point id
* move deferred check to the entry
* fix after rebase
* fmt
* fix tests
* review remarks
* fix tests
* codespell fix
* are you happy clippy
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Introduce EdgeShardConfig for edge shard
- Add EdgeShardConfig and EdgeOptimizersConfig in lib/edge/src/config.rs
- Segment config (vector_data, sparse_vector_data, payload_storage_type)
- Global hnsw_config and per-vector HNSW in segment config
- Optimizer params: deleted_threshold, vacuum_min_vector_number,
default_segment_number, max_segment_size, indexing_threshold,
prevent_unoptimized (excludes memmap_threshold, flush_interval_sec,
max_optimization_threads)
- Persist/load as edge_config.json in shard path
- EdgeShard uses RwLock<EdgeShardConfig>; load() accepts Option<EdgeShardConfig>,
falls back to file or infer from segments; compatibility checked on load
- load_with_segment_config() for backward compatibility (SegmentConfig -> EdgeShardConfig)
- optimize() uses EdgeShardConfig for hnsw and optimizer thresholds
- Public methods: set_hnsw_config(), set_vector_hnsw_config(), set_optimizers_config()
(update and persist)
- Python and examples use load_with_segment_config with existing config API
Made-with: Cursor
* Refactor EdgeShardConfig: user-facing params only, config module
- Replace SegmentConfig inside EdgeShardConfig with user-facing fields:
- on_disk_payload (bool) instead of payload_storage_type
- vectors: HashMap<VectorNameBuf, EdgeVectorParams> with on_disk per vector,
no per-vector quantization; global quantization_config only
- sparse_vectors: HashMap<VectorNameBuf, EdgeSparseVectorParams> with on_disk
- EdgeVectorParams / EdgeSparseVectorParams use on_disk (bool) instead of
storage_type; conversion to VectorDataConfig/SparseVectorDataConfig in
to_segment_config()
- Add config module: mod.rs, optimizers.rs, vectors.rs, shard.rs
- from_segment_config(&SegmentConfig) fills all inferrable params
- to_segment_config() builds SegmentConfig for segments and optimize()
- load_with_segment_config takes Option<SegmentConfig>, uses from_segment_config
Made-with: Cursor
* Move optimizer threshold helpers to shard crate
- Add get_number_segments, get_indexing_threshold_kb, get_max_segment_size_kb,
get_deferred_points_threshold_bytes in shard::optimizers::config
- Collection OptimizersConfig and edge EdgeOptimizersConfig delegate to these
- Single place for threshold logic; collection and edge use shard helpers
Made-with: Cursor
* Use destructuring in config conversions to avoid missing new fields
- EdgeVectorParams: destructure VectorDataConfig in from_*, destructure self in to_vector_data_config
- EdgeSparseVectorParams: destructure SparseVectorDataConfig and SparseIndexConfig in from_*, destructure self in to_sparse_vector_data_config
- EdgeShardConfig: destructure SegmentConfig in from_segment_config, destructure self in to_segment_config
Adding new fields to source structs will now cause compile errors until conversions are updated.
Made-with: Cursor
* refactor: centralize on_disk_payload→payload_storage_type, on_disk→storage_type, and appendable quantization logic
- PayloadStorageType::from_on_disk_payload(bool) in segment (Mmap/InRamMmap)
- VectorStorageType::from_on_disk(bool) in segment (ChunkedMmap/InRamChunkedMmap)
- QuantizationConfig::for_appendable_segment(Option<&Self>) in segment (feature flag + supports_appendable)
- collection: use from_on_disk_payload in non-rocksdb branch
- edge shard/vectors: use new helpers; remove duplicated conditionals
- shard optimizers: use from_on_disk and for_appendable_segment
Made-with: Cursor
* refactor(edge): use EdgeShardConfig directly, drop segment_config
- Add plain_segment_config() for create_appendable_segment (no HNSW)
- Add segment_optimizer_config() built from EdgeShardConfig for blocking optimizers
- Add vector_data_config(name) for query/MMR
- build_blocking_optimizers: use segment_optimizer_config() instead of SegmentConfig
- create_appendable_segment: use plain_segment_config()
- search/query: use config().vectors and vector_data_config() instead of segment_config()
- Remove segment_config() from EdgeShardConfig and EdgeShard
- Add to_plain_vector_data_config on EdgeVectorParams
Made-with: Cursor
* [manual] review changes
* refactor(edge-py): wrap EdgeShardConfig, add EdgeVectorParams/EdgeSparseVectorParams
- PyEdgeConfig now wraps EdgeShardConfig (vectors, sparse_vectors, on_disk_payload, etc.)
- PyEdgeVectorParams / PyEdgeSparseVectorParams wrap edge config types
- PyEdgeOptimizersConfig for optional optimizer settings
- EdgeShard.load() uses EdgeShardConfig; edge::config made pub for Python crate
- cargo fmt + clippy (remove map_identity)
Made-with: Cursor
* refactor(edge-py): simplify config API, remove unused Py* types, add EdgeConfig
- Remove unused PyPayloadStorageType, PyVectorDataConfig, PyVectorStorageType,
PySparseVectorDataConfig, PySparseVectorStorageType from Python bindings
- Move PyEdgeOptimizersConfig to lib/edge/python/src/config/optimizers.rs
- Update qdrant_edge.pyi: EdgeConfig with vectors/sparse_vectors,
EdgeVectorParams, EdgeSparseVectorParams, EdgeOptimizersConfig
- Update examples (common.py, repr.py) to use new config API
- Run cargo fmt
Made-with: Cursor
* [manual] review changes
* [manual] review changes
* [manual] fix test
* Address CodeRabbit review comments for PR 8322 (#8324)
* Address CodeRabbit review comments for PR 8322
- Python examples: explicit imports (repr.py, common.py) and new EdgeConfig API
- HnswIndexConfig: add max_indexing_threads param and property in .pyi and Rust bindings
- EdgeConfig: make vectors optional for sparse-only configs; validate at least one of vectors/sparse_vectors
- EdgeShardConfig::load: use try_exists(), propagate I/O errors
- from_segment_config: infer hnsw_config from per-vector HNSW when all agree
- EdgeShard setters: atomic clone-mutate-save-then-replace; persist config save errors
- Segment compat: prefix vector name in error messages; resolve None datatype to Float32
- max_indexing_threads: preserve 0 (auto) sentinel in trait default; remove per-optimizer overrides
- SegmentOptimizerConfig:🆕 build plain and optimizer maps in single pass
- config_mismatch_optimizer tests: use VectorNameBuf::from() instead of .into()
- vectors.rs: doc updates for per-vector quantization
Made-with: Cursor
* Address @generall review: SaveOnDisk for config, resolve num_rayon_threads in optimizer
- Use SaveOnDisk<EdgeShardConfig> for EdgeShard config (generall: 'We have SaveOnDisk struct for this')
- Create via SaveOnDisk::new() after resolving config; setters use .write() for atomic persist
- set_vector_hnsw_config: clone then mutate then write (fallible setter)
- max_indexing_threads: resolve 0 (auto) via num_rayon_threads inside impl (generall: 'proper solution would be to resolve num_rayon_threads inside the optimizer impl')
- max_indexing_threads_sentinel_aware() now returns Some(num_rayon_threads(raw)) so callers get actual thread count
Made-with: Cursor
* [manual] reorganize num_rayon_threads -> get_num_indexing_threads to better account per-vector configuration
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* update docstring and pyi
* fmt
* fmt
* clipy
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
Change pub mod to mod for count, facet, info, optimize, query,
retrieve, scroll, search, snapshots, and update in lib/edge.
The public API (EdgeShard, ShardInfo) is unchanged.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* Fixups of amalgamator
Fix issues that break `qdrant-edge` build process:
- `use … as segment;` - this causes `ast-grep` rules to replace wrong
paths. So, rename to avoid collisions.
- `#[macro_use]` and `extern crate` required be in the top-level
`lib.rs`.
- `format!("…", crate::something::…)` - `ast-grep` can't fix paths
inside macros. Fixed by moving `crate::something::…` out of the macro.
* Add lib/edge/publish workspace and amalgamation script
* Move `lib/edge/examples` into `lib/edge/publish/` workspace
And fix them to use the generated `qdrant-edge` crate.
* Add github workflow
* Cleanup `qdrant-edge` public API
Removes empty modules. Checked by `cargo doc`.
* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
* handle `score_threshold` in Formula queries
* AI: add openapi test
prompt: Upload 4 points with a numeric payload, use the payload as the
score, and set a score threshold. We can assert which ids should be in
the result and which shouldn't
* weighted rrf implementation
* test
* fmt
* fix edge
* validate number of sources and number of weights
* do not partial match
* upd schema
* review fixes
* update formula
* remove calcualtions from tests
* update comment, because AI have OCD
* fmt
* Split `SegmentEntry`
Add `ImmutableSegmentEntry` for operations that can be applied to
immutable segments, and make the `SegmentEntry` as it subtrait.
* Rename to `NonAppendableSegmentEntry`
It differs semantically from `ImmutableSegmentEntry` by allowing point
deletion. Move point deletion to the trait too.
* Fix docstring
* use NonAppendableSegmentEntry where possible
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Replace `Option<Segment>` with `enum LoadSegmentOutcome`
* Replace some Path/PathBuf with str/String
* Rename field Segment::{current_path -> segment_path}
* safe_delete
* Timeout on point retrieve when acquiring segment holder
* review: less radical and cleaner
* highly embarassing I hope no one sees this!
* coderabbit
