mirror of
https://github.com/qdrant/qdrant.git
synced 2026-07-31 15:11:35 -05:00
read-batch-generic-error
82 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Andrey Vasnetsov
|
446d140c2d |
Slice filtering condition: sliced scroll / deterministic sampling (#9899)
* feat: slice filtering condition for sliced scroll and deterministic sampling
Add a `slice` filter condition selecting points where
`stable_hash(point_id) % total == index`. The hash is SipHash-2-4 with a
zero key over canonical id bytes (8 LE bytes for numeric ids, 16 RFC 4122
bytes for UUIDs) — a frozen public contract, independent of the internal
resharding ring hash, reproducible by clients to predict membership.
For a fixed `total`, slices are disjoint and cover all points, enabling
parallel scroll streams (ES sliced-scroll style) and reproducible sampling
that composes with any other filter condition.
- REST: `{"slice": {"total": N, "index": R}}`; gRPC: `SliceCondition` in
the condition oneof (tag 8)
- Evaluated per point via id_tracker external-id lookup; no payload index
needed; cardinality estimated as `points / total` with no primary clause
- `total >= 1` enforced by NonZeroU32 at parse time, `index < total` by
validation in both REST and gRPC paths
- Hash contract locked by test vectors independently reproduced with a
reference SipHash-2-4 implementation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* tests: minimal OpenAPI test for slice filter condition
Scrolls all slices of a fixed total over numeric + UUID ids asserting
disjointness and full coverage, checks must_not inversion, and pins the
two rejection paths (422 for index >= total, 400 for total = 0). Requests
and responses are validated against the regenerated OpenAPI spec by the
test harness.
Note: the spec cannot itself reject total = 0 client-side — the Condition
anyOf falls through to the permissive Filter schema, as with any invalid
condition — so rejection is asserted via the server response.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Andrey Vasnetsov
|
4ff6acaf8c |
Edge: request-specific structures for EdgeShardRead (#9901)
Replace the mixed read interface (internal types, explicit parameter enumeration, ad-hoc custom types) with edge-owned request structs, one file per request in src/requests/. Each has a new() constructor taking only the required parameters, a no-macro fluent builder in src/builders/, and a From conversion into the internal request type in requests/conversions/, grouped by request type. Conversions construct and destructure with full field lists, so a parameter added on either side fails compilation instead of being silently dropped. The old reexport aliases (ScrollRequest = ScrollRequestInternal, etc.) are replaced by the edge types under the same names; retrieve() takes a RetrieveRequest instead of a parameter triple. Python bindings wrap the edge types, and the published Rust examples use the builders. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
842f701aae |
refactor: group edge crate internals into edge_shard and read_view modules (#9898)
* refactor: group edge crate internals into edge_shard and read_view modules Restructure lib/edge/src so the top level only contains the modules that form the public crate surface. Implementation files move under the type they implement: - edge_shard/: the EdgeShard struct with its load/config-resolution helpers (previously inlined in lib.rs), plus optimize, shard_read, snapshots, and update - read_view/: the EdgeShardRead/ReadSegmentHandle traits and EdgeReadView (previously read_view.rs), plus the per-operation impl files count, facet, grouping, info, matrix, query, retrieve, scroll, and search; build_search_pool (previously pool.rs) is folded into read_view/mod.rs next to par_map_segments, the seam it powers lib.rs is now a thin facade of module declarations and re-exports. The public API is unchanged: all previously exported names resolve exactly as before, verified against the edge tests, the python bindings, edge-shard-query, and the regenerated publish amalgamation with its examples. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: trim EdgeShardRead surface and split read_view/read_only modules Trait surface: - Drop query_scroll and rescore_with_formula from EdgeShardRead and the EdgeShard inherent wrappers; only the internal query pipeline used them, via the pub(crate) EdgeReadView methods that remain. - Hide the snapshot plumbing (read_segments, search_pool, plus config_snapshot/path providers) in a crate-private ReadViewProvider trait. EdgeShardRead now declares only user-facing methods and is implemented for every provider through a blanket impl, so the plumbing is not callable from user code (verified with a negative compile test; a private supertrait alone leaves supertrait methods callable through generic bounds). An empty sealed marker supertrait keeps the trait unimplementable downstream. - config_snapshot and path stay public: used by edge-shard-query and the python bindings. Module layout: - read_view/: mod.rs keeps EdgeReadView and build_search_pool; ReadSegmentHandle moves to handle.rs, the trait machinery to shard_read.rs, and the nine per-operation impl files into ops/. - read_only/: the follower's ReadViewProvider impl moves out of mod.rs into shard_read.rs, mirroring edge_shard/shard_read.rs. Verified: edge tests, python bindings, edge-shard-query, regenerated publish amalgamation with all examples compiling and running. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Daniel Boros
|
0982e8699c |
feat: segment manifest optimizing state with lease (#9873)
* feat: segment manifest optimizing state with lease * fix: clippy * fix: exhaustive manifest state matching * fix: merge manifest rebuilds under the write lock * refactor: named state predicates, drop redundant enumerator test Review follow-up: move the enumerator's filter into SegmentManifestState::is_usable, name the preserving predicate is_optimizer_mark, delete the enumerator test that re-tested serde. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
1d4d6f02da |
Per-query IDF corpus for sparse vector search (#9661)
* Add per-query IDF corpus for sparse vector search
Let the caller choose, per query, which population sparse IDF statistics
are computed over. `params.idf` is either `"global"` (default, unchanged
behavior) or `{"corpus": <filter>}`, where the corpus filter is
independent of - and usually broader than - the retrieval filter.
Decoupling the two keeps the score scale stable when the retrieval
filter tightens: term importance is measured against a population the
user names, not against whatever subset the filter happens to select.
Design decisions:
- Corpus grammar is restricted to a conjunction (`must`) of `match`
conditions on payload fields; loosening later is backward compatible.
- Strict mode validates the corpus filter like a read filter
(unindexed fields rejected).
- `idf` on a vector without the IDF modifier is a validation error,
never silently ignored.
- An empty corpus yields degenerate but corpus-scoped scores (smoothed
IDF over N=0), never a fallback to global statistics - in multi-tenant
collections a fallback would leak term statistics across tenants.
Implementation:
- QueryContext IDF stats are keyed by corpus, so one batch can mix
requests with different corpora.
- Statistics come from the sparse index: df(term) is counted over the
query terms' posting lists only, never by scanning stored vectors.
Small corpora (under ~1/32 of the segment, by cardinality estimate)
are kept as a sorted id list galloping through posting lists via
skip_to; large ones as a dense membership mask filled streaming from
the filtered-points iterator. A misestimated small corpus degrades
into the mask.
- Exposed uniformly: REST (`params.idf`), gRPC (`IdfParams` message),
edge python bindings; OpenAPI schema regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply rustfmt
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix clippy manual_is_multiple_of in sparse IDF corpus test.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Allow any filter as IDF corpus
Drop the must+match grammar restriction on the corpus filter. A
restriction enforced only as a validation step over the full Filter
type buys nothing; if a narrower corpus syntax is ever wanted, it
should be a dedicated API-level type instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix build: add memory field to SparseIndexConfig in idf corpus test
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
Andrey Vasnetsov
|
43e3d6ea8d |
[UIO] Request-specific load profile for read-only opens (#9797)
* Introduce request-specific LoadProfile with per-component placement A read-only shard opened for one known request (the serverless cold-start path) doesn't have to warm components the request will never touch. LoadProfile captures that from the request: warm components keep the persisted-config placement, everything else is parked cold. All placement decisions live in one place, so the memory placement of a whole segment under a profile is reviewable in one file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Thread LoadProfile through the read-only segment open ReadOnlySegment::open takes an optional profile; first_preopen and open_via resolve it into per-component populate overrides so the opens make the same placement decisions the prefetches did. Pinned components that materialize on open regardless (quantized RAM storage kinds, the immutable-RAM sparse index) and appendable components ignore the override; the HNSW graph and immutable payload indexes demote fully. Config reloads follow the new config alone and pass no override. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Open ReadOnlyEdgeShard under a request-derived load profile ReadOnlyEdgeShard::open takes an optional LoadProfile, applies it to every segment open and keeps it so segments discovered by a later refresh load with the same placement. ScrollRequestInternal and CoreSearchRequest gain load_profile() constructors, and edge-shard-query builds the request before the open and passes its profile (opt out with --no-load-profile). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Demote pinned quantized vectors and sparse index under a cold profile Within the immutable layout the quantized RAM and mmap loaders share the on-disk format — only how the data is brought into memory differs — and the immutable-RAM sparse index has the same lazy mmap open low-memory mode already downgrades to. So a cold populate override now demotes the effective placement itself (Memory::with_populate_override, shared with the HNSW residency mapping) instead of only skipping cache priming: a pinned quantized storage opens the mmap kind cold, and a pinned sparse index opens as Mmap, so neither reads its data on a cold start. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Add LoadProfile::merge for composite queries A composite query runs multiple core requests — e.g. a hybrid search runs one core search per vector, each with its own filter. Its profile is the union of its parts': merge extends the warm sets and ORs the payload-storage flag, so a component either part needs warm stays warm. The union is sound because every placement method is monotone in the warm sets (growing them only turns "park cold" into "keep configured placement"), so the merged profile dominates each input; and minimal, warming nothing no part asked for. Combine profiles with reduce, not fold: merge's identity element is the coldest profile (empty warm sets), the opposite of passing no profile at all — deliberately no empty()/Default constructor exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Adapt vanished-segment test to the profile-aware open signature The test landed on dev (#9777) after the load-profile signature change was written, so the rebase left its ReadOnlySegment::open calls without the new load_profile argument. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Defer vector index open entirely under a cold load profile A cold placement is not enough for the vector index on remote backends: GraphLinksView requires the whole links file as one contiguous slice, and the disk cache can only lend a borrowed slice once every block is locally present — so even a Cold HNSW open mirrors the entire links_compressed.bin (8.2 MB / 1.1 s per segment in the serverless cold-start trace), plus the unconditional graph.bin metadata read. The only way not to fetch the index is not to open it. LoadProfile::vector_index_placement is replaced by vector_index_deferred: a vector the request never scores now gets a DeferredVectorIndex — a new VectorIndexReadEnum variant holding the open arguments (an owned clone of the segment's raw backend, path, config, shared component handles) and a OnceLock. Nothing is opened or prefetched for it at segment open. Per-method policy of the deferred variant: - search, fill_idf_statistics and populate open the index on first use (with the cold placement the profile chose), so the profile contract holds: a request the profile did not predict still works, just pays the open then; - is_index reports true without opening (deferral only ever wraps a real HNSW or sparse index; plain opens no files and is never deferred); - telemetry, indexed_vector_count and sizes answer conservative defaults rather than trigger a remote fetch for a statistic. Tests: deleting the vector_index directory before an open under a scroll profile leaves open, filtered reads and payload reads working — proof that nothing of the index is read — while a search surfaces the missing files; and a segment opened under a scroll profile answers searches identically to an eagerly opened one via the transparent first-use open. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Make --vector optional in edge-shard-query: random query after open Omitting --vector on the search sub-command now searches with a random vector. The request is still built before the shard opens — the load profile only needs the vector name, not its values — with an empty placeholder; once the shard is open, fill_random_vector reads the dimension of the queried vector from the derived shard config and fills in uniform-random f32s (with a clear error if the named vector is not in the config). The vector is generated once, so live-reload iterations re-run the identical random query and the printed diffs stay meaningful. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Scope index deferral to the HNSW graph via a lazy OnceLock load Replace the DeferredVectorIndex wrapper (and the VectorIndexReadEnum:: Deferred variant) with deferral inside ReadOnlyHNSWIndex itself: the graph lives in a OnceLock (same first-wins arbitration as ReadOnlyRoaringFlags::bitmap) alongside the retained raw backend and residency, and loads on first use with a cold placement. The config read stays eager — one tiny, absence-tolerated file — so telemetry, is_on_disk and indexed_vector_count report real values where the Deferred arms answered with hard defaults. The sparse index needs no deferral: its mmap open reads lazily, with only small JSON metadata eager. A profile that never scores the vector now passes a cold placement override (LoadProfile:: vector_index_placement) into the eager open_sparse, which demotes ImmutableRam to the lazy Mmap open like low-memory mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Express HNSW graph deferral as a populate override, not a bool param Replace the `deferred: bool` on the read-only HNSW open/preopen (and the VectorIndexReadEnum pass-through) with the same `populate_override: Option<Populate>` every other component takes. A cold *override* defers the graph load — graph_deferred() mirrors the cold-override match of open_sparse — while a config-derived cold placement (or the low-memory clamp) keeps the eager load, since only a request-specific override carries the "never scored" prediction. With dense and sparse now consuming the same signal, LoadProfile::vector_index_deferred is gone: a single vector_index_placement() serves both index kinds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Serialize the deferred graph load via once_cell's get_or_try_init Loading outside the lock (std OnceLock's fallible init is still unstable) let a search burst on a deferred vector fetch the whole graph once per thread. Swap the cell for once_cell::sync::OnceCell: the fallible load runs inside the cell's lock, concurrent first users block on the one load, and a failed load leaves the cell empty so the next caller retries. Addresses https://github.com/qdrant/qdrant/pull/9797#discussion_r3573727836 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Daniel Boros
|
f982ae63fc |
feat: read-only edge grouping/matrix search + object-storage read path (#9691)
* feat: read-only edge grouping/matrix search + object-storage read path Add query_groups (group by a payload field) and search_matrix (single-shard distance matrix over a random sample) to the read-only edge shard's EdgeShardRead API, in new grouping and matrix modules plus edge test helpers. Make the object-storage read path available outside tests: drop the #[cfg(test)] gate on the BlobFile UniversalReadExt impl and move io_bridge_object_store/object_store to segment's normal dependencies, so a ReadOnlyEdgeShard can serve segments read from S3. * Share group-by building blocks between server and edge Move GroupsAggregator, group candidate query shaping (is-empty filter, group_by payload selector, prefetch limit scaling) and result-order derivation into shard::grouping / shard::query, so the collection and edge grouping implementations cannot silently diverge. Edge grouping now handles multi-valued group keys, u64 keys, wildcard group_by paths, prefetch limits and score-ordered groups the same way as the server. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Drive group-by through a shared sans-IO state machine Extract the multi-request collect/fill loop into shard::grouping::GroupByDriver: next_request() yields shaped backend queries, add_points() advances the state, distill() returns the groups. Query execution stays with the caller, so the async server path and the sync edge path drive the same machine, and the request shaping helpers become private to shard::grouping. Edge now uses the same request budget (5 collect + 5 fill requests) and per-request candidates limit (groups * group_size, computed inside the driver) as the server, replacing its single 4x-oversampled request. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
f82d7584a2 |
Resolve vanished segments against the manifest in follower refresh (#9792)
Third step of the ReadOnlyEdgeShard not-found handling (after #9763 and #9777): shard-level resolution of live-reload failures, per the live-reload design requirements. refresh() is now a bounded retry loop over single-manifest-snapshot attempts. Within an attempt every survivor live-reloads even if one fails (they are independent); failures split by classification: - Not-found: resolved against a re-read manifest. Gone from the manifest means the leader removed the segment mid-reload — drop it and re-run the attempt to pick up its replacements immediately. Still listed means essential files are genuinely missing — escalate. - Anything else: escalate after reloading the other survivors, replacing the previous warn-and-swallow that could hide a corrupted segment forever. Safe: a failed segment keeps serving its pre-refresh state and pending_reload replays the unapplied delta on the next refresh. If attempts run out (leader churning segments continuously) refresh logs a warning and returns Ok: every swap was atomic, so the shard is consistent, just possibly not the newest; the next refresh continues. open_with_enumerator now reuses the refresh machinery: empty holder + refresh() + the open-only merge_follower_config overlay, removing the duplicated load/derive logic. At open there are no survivors, so open behavior is unchanged (#9762 already made it tolerant of unloadable segments via the superset-biased manifest contract). Load-side failures stay as settled by #9762: unloadable listed segments are skipped and retried on every refresh. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
qdrant-cloud-bot
|
d773266c55 |
Fix edge shard exact count double-counting across segments (#9790)
Collect per-segment point ids into a BTreeSet before counting so points visible in multiple segments (e.g. during proxy/merge optimization) are counted once, matching the canonical collection count path. Fixes #9789 Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Arnaud Gourlay
|
35bbf0487a |
Remove unnecessary clippy allow attributes (#9775)
Remove 8 `#[allow(clippy::...)]` attributes that no longer suppress any lint. Each was verified redundant by rewriting it to `#[expect(...)]` and confirming the workspace stays clippy-clean under the CI config (`cargo clippy --workspace --all-targets --all-features -- -D warnings`). Attribute-only deletions, no behavior change. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Boros
|
66ce17e709 | feat: skip unloadable segments in read-only follower open (#9762) | ||
|
Andrey Vasnetsov
|
fb681b7d9d |
Lazy roaring flags bitmap and bool index counts (#9749)
* [AI] make ReadOnlyRoaringFlags bitmap and bool index counts lazy
Opening a read-only segment scanned every flags file end to end:
`ReadOnlyRoaringFlags::open` materialized the whole RoaringBitmap via
`iter_ones()`. Every payload field carries a null index, so this was paid
per field per segment, for bitmaps most queries never touch.
Make the bitmap a `OnceLock`, filled by a scan on first access. Open now
reads only the tiny status file. `ReadOnlyBoolIndex`'s three eager count
fields collapse into one lazily-derived, cached `BoolCounts`; its
`live_reload` refreshes them in place when present and leaves them unset
otherwise, so reloading an index nothing queries stays scan-free.
Propagate the resulting `OperationResult` through `RoaringFlagsRead`,
`PayloadFieldIndexRead::count_indexed_points`, `FieldIndexRead`,
`PayloadIndexRead::{indexed_points, get_telemetry_data}`, `build_info` /
`build_telemetry` and `SegmentEntry::{info, get_telemetry_data}`, out
into shard, edge and collection.
`ram_usage_bytes` stays infallible: an unmaterialized bitmap holds no
RAM, so it reports 0 via the new `bitmap_if_materialized`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [AI] correct `preopen` comment: `open` no longer scans the flags file
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [AI] fix edge examples for fallible `info()`
`EdgeShardRead::info` now returns `OperationResult<ShardInfo>`. The
examples live in their own workspace (lib/edge/publish), so the main
`cargo check --workspace` never saw them.
Every call site sits in `fn main() -> Result<(), Box<dyn Error>>`, so
propagate with `?`. `bm25-search` compiled either way but would have
printed the `Result` rather than the `ShardInfo`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Arnaud Gourlay
|
bd35dc1189 |
Replace flush_all sync bool with FlushMode enum (#9750)
* refactor: replace flush_all sync bool with FlushMode enum
SegmentHolder::flush_all took two adjacent bools (sync, force), and call
sites read as bare literal pairs like flush_all(true, false). Swapping
the arguments compiles and silently changes flush semantics: a swapped
pair at the snapshot site would make snapshots skip flushing entirely
when a background flush is running.
Introduce FlushMode { Sync, Background } for the first parameter so the
pair is no longer transposable and the behavior is named at each call
site. The force flag stays a bool since it feeds the
SegmentEntry::flusher(force) trait in lib/segment. No behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: exhaustive match on FlushMode instead of equality check
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Andrey Vasnetsov
|
ab0d3ecc62 |
Add unified memory: cold|cached|pinned placement parameter for collection components (#9684)
* Add unified `memory: cold|cached|pinned` placement parameter for collection components
Introduce a single `memory` parameter that controls how each collection
component's data is held in RAM, replacing the inconsistent zoo of
`on_disk` / `always_ram` / `on_disk_payload` flags:
- `cold`: not pre-loaded from disk, cached with usage
- `cached`: pre-populated into page cache on load, evictable under pressure
- `pinned`: materialized on heap, never evicted by cache pressure
The parameter is available on dense vectors, HNSW config, all quantization
configs, the sparse index, all payload field index types, and payload
storage (as a new `payload: { memory }` sub-object on collection params).
When set, it overrides the deprecated legacy flag; when unset, behavior is
unchanged. Legacy flags are marked deprecated (Rust + proto) but keep
working; conflicts are resolved in favor of `memory` with a warning.
New capabilities enabled by the tri-state model:
- HNSW graph links can be pinned (first production caller of the existing
`GraphLinksResidency::Pinned`)
- sparse mmap index, quantized vectors and on-disk payload field indexes
gain a `cached` tier (mmap + populate on open)
`pinned` is rejected by API validation for components without a heap
variant (dense vector storage, payload storage). Low-memory mode degrades
placements at load time via `Memory::clamp_to_low_memory`, matching the
existing `prefer_disk`/`skip_populate` behavior. Effective-placement
comparison in the config-mismatch optimizer avoids spurious rebuilds when
the same placement is expressed through the new parameter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix gpu-gated tests for the new `memory` field
CI clippy runs with --all-features, which compiles the gpu-gated tests
that were missed locally: add the `memory` field to config literals and
allow deprecated placement params, same as in the rest of the tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add OpenAPI tests for memory placement, keep sparse config downgrade-clean
- OpenAPI tests: create/update collections with `memory` on every component,
assert the parameters are echoed in collection info, assert legacy-only
collections expose no new fields, and assert `pinned` is rejected (422)
for dense vector storage and payload storage on both create and update.
- Persist only the explicitly requested `memory` parameter in
`sparse_index_config.json` instead of the legacy-resolved placement, so
configurations using only the deprecated `on_disk` flag keep byte-identical
files that older Qdrant versions load without unknown fields.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Validate collection meta ops at construction, not only in the API layer
The `memory: pinned` rejection for dense vectors and payload storage
lived in `Validate` impls on the internal request types, which only ran
through the REST actix extractor. gRPC validates just the proto message,
so a gRPC client could persist `pinned` where it is not supported and
have it silently treated as `cached`.
Run the derived validation in `CreateCollectionOperation::new` and
`UpdateCollectionOperation::new` instead: the constructors are the
common chokepoint for all API paths, before the operation is proposed
to consensus. This covers every validator on these types, not just the
`memory` checks, and keeps consensus-apply unaffected so mixed-version
clusters never reject already-committed operations.
`UpdateCollectionOperation::new` becomes fallible; `remove_replica` now
uses `new_empty` since it carries no user config. Regression tests drive
the gRPC conversion path and assert `InvalidArgument` for `pinned` on
create and update, with `cold`/`cached` accepted as a control.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
Andrey Vasnetsov
|
729dc6af43 |
Make EdgeConfig tunables optional with a layered fallback chain (#9714)
Tunable EdgeConfig parameters (on_disk_payload, hnsw_config, optimizers) are now Option, and every tunable resolves through the fallback chain provided -> persisted -> derived from segments -> default when loading an existing shard. Leaving a parameter unspecified keeps the shard as it is; an explicit value overwrites it and existing segments converge to it through the optimizers. vectors/sparse_vectors are excluded from overwrite semantics: an empty map inherits the persisted/segment-derived definitions, a non-empty map is validated for compatibility against the loaded segments (size, distance, multivector, datatype, sparse modifier) and fails the load on mismatch. The derived layer folds over all segments in UUID order instead of taking an arbitrary first segment, so a plain appendable segment (which carries no HNSW parameters) can never mask an indexed segment's actual build parameters. Previously a lost edge_config.json could resolve unspecified HNSW params to compiled-in defaults and silently trigger a full re-index via ConfigMismatchOptimizer. The read-only follower accepts an optional config on open: provided tunables are applied once over the segment-derived config (vectors always come from the segments), and refresh re-derives from segments alone. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Daniel Boros
|
61a1eb80ba |
chore: reexport types (#9696)
* feat: re-export ShardKey and GeoLineString from edge * fix: linter * chore: remove shardkey |
||
|
Andrey Vasnetsov
|
efab63d024 |
Add prefix matching option to keyword index (#9683)
* Add prefix matching option to keyword index
Introduce an opt-in `prefix` option for the keyword payload index and a
new `match: { "prefix": ... }` filter condition, enabling efficient
byte-wise prefix filtering over keyword values (e.g. URL prefixes,
web-ui value autocompletion via facet + prefix filter).
Index side: a new `prefix_index.bin` file stores a sorted, front-coded
key dictionary with a resident block index (cumulative counts per
block); it is an ordered view over the keys of `values_to_points.bin`
and stores no postings. Presence of the file signals prefix support at
load time, so legacy segments load unchanged and enabling the option
goes through the standard incompatible-schema rebuild. The mutable
variant keeps an in-RAM ordered key set (not persisted), the immutable
variant builds a sorted key vector at load, and the on-disk variant
reads the dictionary lazily (block index resident, 1-2 block reads per
prefix lookup; reader is generic over UniversalRead).
Query side: prefix conditions are served from the dictionary when
available (filter + cardinality estimation from per-block aggregates),
from the forward index as per-point checks, and degrade to the payload
full-scan fallback otherwise - same execution model as other match
conditions. Strict mode (`unindexed_filtering_*`) rejects prefix
queries on fields without a prefix-enabled keyword index via a new
KeywordPrefix capability.
HNSW payload blocks: prefix-enabled indexes additionally emit prefix
blocks for heavy branching trie nodes (single-child chains collapsed to
their longest common prefix, one block per distinct point set, emitted
largest-first) so filtered search with prefix conditions gets navigable
subgraphs without rebuilding the same subset repeatedly.
API: `prefix` flag on KeywordIndexParams (REST bool, gRPC empty
message for extensibility), `prefix` variant in the Match oneof, edge
python bindings, regenerated OpenAPI spec.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Split prefix index into a dedicated module, fix clippy in tests
Reorganize the flat prefix_index.rs / prefix_read.rs into a
map_index/prefix_index/ module: format.rs (on-disk layout primitives),
writer.rs, reader.rs (PrefixIndex), map_read.rs (StrMapIndexPrefixRead
with per-variant impls) and tests.rs, with a file-format diagram and a
read-path walkthrough in the module docs. No logic changes.
Also fix clippy --all-targets complaints in test code: replace a
wildcard Match arm with an exhaustive list and a field-reassign-with-
default with a struct literal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add OpenAPI test for prefix match and snapshot file-tracking test
- tests/openapi/test_prefix_match.py: index-less fallback, prefix index
creation with schema echo, scroll/count parity against ground truth,
facet + prefix filter (the autocompletion flow), strict-mode rejection
without the prefix capability.
- test_prefix_index_file_tracking: `prefix_index.bin` is listed in
`files()` / `immutable_files()` exactly when built with the option.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Replace hand-rolled varint parsing with bytemuck Pod records
Per review: the prefix index format now uses fixed-size little-endian
Pod records (BlockEntry 24 B, KeyEntry 12 B, Header 40 B) written with
bytemuck::bytes_of and read back by copy via pod_read_unaligned — no
manual varint encode/decode, no alignment requirement, one shared
read_record helper. Costs ~9 bytes per key on disk versus LEB128; the
raw key bytes dominate dictionary size, so the simplification wins.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fetch the whole candidate block range with a single storage read
Candidate key blocks of a prefix lookup are contiguous in the file, so
enumerate them from one ranged read instead of one read per block; the
over-read versus the exact key range is bounded by the two boundary
blocks. Block decoding is split into a storage-free helper reused by
the per-block path of stats estimation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Align prefix payload blocks with the geo index granularity principle
Geo's large_hashes emits only the smallest geohash regions above the
threshold — a disjoint antichain, never a parent nested with its
children. Prefix payload blocks now follow the same rule: a heavy
collapsed trie node is emitted only if nothing heavy is nested inside
it, counting both deeper qualifying prefixes and single heavy values
(which already get their own exact-match blocks). Emitted blocks are
therefore mutually disjoint and disjoint from exact-value blocks; no
near-collection-sized ancestor subgraphs, no reliance on the HNSW
connectivity check to skip nested duplicates.
Implemented as a `covered` flag propagated through the existing
LCP-interval scan, still one O(total key bytes) pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Document block wire format and unaligned-read rationale in decode_block
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Arnaud Gourlay
|
0346ea66cd |
fix: unblock optimizer after deleting a named vector (#9641)
* fix: unblock optimizer after deleting a named vector Deleting a named vector could permanently block the config-mismatch optimizer. The source-superset check in SegmentBuilder::update cancelled every rebuild that found the deleted vector still in old segment files, and each retry cancelled again, so optimizations got stuck forever. Removing the check (as in #9609) would fix delete but reintroduce data loss for the CreateVectorName race. Instead, tell the two cases apart with the live collection schema: prune a source vector that is gone from the schema (a real deletion), but cancel when it is still present (a freshly created vector this optimizer has not yet seen). This is safe because the schema is persisted before the op reaches segments, and the live schema is read after the source segments are frozen. The live set covers dense and sparse vectors, since a segment stores both together. When no live source is wired in, the conservative always-cancel behavior is kept. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: wire live vector names into edge optimizers Deleting a named vector left the edge path with the pre-fix behavior: segment_optimizer_config hardcoded live_vector_names to None, so a merge touching a segment that still carried the deleted vector cancelled, and EdgeShard::optimize() propagated the cancellation as a hard error forever. Share the shard config behind an Arc and hand the blocking optimizers a provider that reads the current vector names on every call. Same safety argument as the server wiring: update() holds the segments read guard across both the segment application and the config update, so any name a frozen source segment carries is visible to the live read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share vector-name enumeration via CollectionParams::vector_names The optimizer's live-schema set and the WAL-recovery valid-name set are the same dense+sparse enumeration and must stay in lockstep; a drift between them would reintroduce a wrong prune/cancel decision. Replace the private helper in optimizers_builder and the inline block in WAL recovery with a single CollectionParams method. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: drop SegmentOptimizer::live_vector_names forwarding hop The default trait method only forwarded to the config getter and had a single caller; ShardOptimizationStrategy now reads the config directly, removing one layer of indirection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
qdrant-cloud-bot
|
9977421331 |
perf(edge): load ReadOnlyEdgeShard segments in parallel (#9594)
* perf(edge): load ReadOnlyEdgeShard segments in parallel Open each segment on a dedicated thread during initial open and refresh, reducing follower startup time for shards with many segments. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(edge): resolve clippy type_complexity in parallel segment load Co-authored-by: Cursor <cursoragent@cursor.com> * perf(edge): run per-segment reads on a configurable thread pool Replace the per-segment sequential read loops and the spawn-a-thread-per-segment loader with a single fixed-size rayon thread pool owned by each shard. - Add EdgeConfig::max_search_threads (Option<usize>, None = CPU-derived default matching the core search runtime via common::defaults::search_thread_count). - Build a long-lived pool in EdgeShard and ReadOnlyEdgeShard; reuse it for parallel segment loading on open/refresh instead of std::thread::spawn. - Add EdgeReadView::par_map_segments as the single seam that runs per-segment work on the pool; use it in search, scroll, count, facet and rescore-formula. Each task mints its own HardwareCounterCell from the shared accumulator. - Expose max_search_threads through the builder and the Python binding (+ stub). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(edge): return error instead of panicking on search pool creation ThreadPoolBuilder::build() can fail (e.g. thread spawn / resource exhaustion). This runs during EdgeShard open/load and ReadOnlyEdgeShard follower open, so a transient failure must not abort the process. Propagate it as an OperationError through the existing OperationResult-returning constructors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
xzfc
|
db2c68e004 |
Replace DynConditionChecker with ConditionCheckerEnum (#9560)
* Add UniversalReadExt Need this trait for the upcoming `ConditionCheckerEnum`. * Replace DynConditionChecker with ConditionCheckerEnum |
||
|
Andrey Vasnetsov
|
f70a1462fb |
feat: read-only edge shard follower (ReadOnlyEdgeShard) (#9529)
* feat: read-only edge shard follower (ReadOnlyEdgeShard) Add a read-only follower of EdgeShard, mirroring the segment-level ReadOnlySegment + live_reload design one level up at the shard. A leader process owns a read-write EdgeShard; one or more followers open the same on-disk directory as a ReadOnlyEdgeShard, serve reads, and periodically refresh() to pick up the leader's flushed writes and optimizations. Shared read logic lives once in a crate-internal EdgeReadView<H>, generic over a ReadSegmentHandle: the follower is monomorphized over the concrete ReadOnlySegment<S> (no dynamic dispatch), while the read-write shard's heterogeneous Segment/ProxySegment holder uses the LockedSegment enum (the only dyn ReadSegmentEntry left, mirroring LockedSegment::get_read). EdgeShardRead is the public read API: it exposes the read methods (search/query/retrieve/count/facet/scroll/info/...) as default methods, requiring implementors only to provide read_segments() + config_snapshot(). EdgeShard keeps its inherent read methods for backwards compatibility. Segment discovery is injected via a SegmentEnumerator (temporary seam): LocalSegmentEnumerator scans the segments/ directory for the local/mmap case; S3 followers supply their own. This will be replaced by an on-disk segment manifest in follow-up work. shard: add LockedSegment::get_read_arc(); split retrieve_blocking into a handle-collecting wrapper + generic retrieve_over; generalize the private _read_points over the segment type (public signatures unchanged); remove the now-unused read_points_locked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor * feat: manifest-based segment loading for ReadOnlyEdgeShard Replace the read-only follower's directory-scan discovery with the segment manifest from the leader, the proper mechanism the SegmentEnumerator seam was a placeholder for. shard: add SegmentsManifest::load so readers can read manifest.json. edge (leader): EdgeShard now writes segments/manifest.json (gated by the write_segment_manifest feature flag), initialized from the live segment set on new/load and refreshed after optimize() swaps segments. edge (follower): ManifestSegmentEnumerator reads the manifest's `active` segments instead of scanning, and open_mmap uses it. It requires a manifest and errors when none is present (no silent scan fallback); discover by scanning explicitly via open() with a LocalSegmentEnumerator instead. Since the manifest only reports ready segments (and future versions will mark segments retiring/under-construction rather than removing them from active immediately), the read path no longer races with the leader, so the defensive `is_transient_open_error` skip handling is removed — a failure to open a reported segment is now a genuine error and propagates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor * feat: edge-s3-scroll experiment binary for ReadOnlyEdgeShard over S3 Adds a standalone `edge-s3-scroll` binary that opens a ReadOnlyEdgeShard directly over an S3 (or S3-compatible) bucket and runs a single scroll request with a hard-coded filter, for experimentation. Takes the bucket endpoint, credentials and key prefix as CLI flags/env vars, builds an S3-backed BlobFs, and discovers segments via a custom enumerator that reads the segment manifest over object storage. The edge_config.json is fetched to a local temp dir because ReadOnlyEdgeShard::open reads config from the local filesystem. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * filter config in s3 scroll tool * fix: read segment manifest from new location after dev rebase dev #9564 moved the segment manifest next to (rather than inside) the segments/ directory and named it segments_manifest.json. Adapt the read-only follower enumerators accordingly so the follower reads the manifest where the leader now writes it: - ManifestSegmentEnumerator reads via segment_manifest_path() (shard root) while still resolving segment dirs under segments/<uuid>. - The S3 scroll tool's enumerator mirrors the same layout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: read-only edge shard works over object storage without edge_config Make ReadOnlyEdgeShard self-sufficient over arbitrary backends (S3/GCS), and harden the read-only segment loading it depends on. - ReadOnlyEdgeShard derives its config from the segments via EdgeConfig::from_segment_config instead of requiring edge_config.json (open and refresh both derive); a follower never has that file. - ReadOnlyEdgeShard::open(fs, path) no longer takes an enumerator: a read-only follower always discovers segments through the manifest. open_with_enumerator remains as a pub(crate) seam for tests. - ManifestSegmentEnumerator is generic over UniversalReadFs (reads the manifest via read_json_via), so it works over local mmap or a blob/S3 backend; the tool's bespoke enumerator is removed. - Read-only mutable ID tracker tolerates absent mappings/versions files (they are not written while empty), matching MutableIdTracker::open: files are opened lazily and NotFound is treated as empty, avoiding an extra exists() round-trip on object storage. edge-s3-scroll experiment tool: - Supports AWS S3 / S3-compatible and GCS backends (--backend), with a hard-coded-free --filter-key/--filter-value scroll filter. - Reads segment data through a DiskCache so remote blocks are fetched once and served from a local mirror afterwards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: remove unused SegmentsManifest::load Its only caller (edge's ManifestSegmentEnumerator) now reads the manifest via read_json_via over UniversalReadFs, so the local-only load helper is dead. Drop it and its now-unused read_json/Path imports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
qdrant-cloud-bot
|
8c487a17e8 |
feat(bm25): explicit Disabled stemmer; deprecate language: "none" hack (#9376)
* feat(bm25): add explicit Disabled stemmer; deprecate language hack
Adds a `Disabled` variant to `StemmingAlgorithm` (`stemmer: {"type": "none"}`)
so stemming can be turned off explicitly in both the main engine and Edge,
instead of relying on the undocumented `language: "none"` footgun that
silently disabled both stemming and stopwords.
For language-neutral text processing the supported setup is now:
1. set the stemmer to disabled, and
2. configure an empty stopword set.
The main engine still tolerates unsupported languages (so existing
`language: "none"` configs keep working on upgrade) but now logs a
deprecation warning pointing users to the explicit setup. Edge continues
to reject unsupported languages, and now has a real way to disable stemming.
Refs: https://github.com/qdrant/qdrant/issues/9289
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(edge-py): handle Disabled stemmer in python bindings; fix openapi schema
- Handle the new StemmingAlgorithm::Disabled variant in the qdrant-edge-py
bindings (FromPyObject/IntoPyObject/Repr) and add a DisabledStemmer pyclass
plus its .pyi stub entry.
- Match generator output for the StemmingAlgorithm OpenAPI schema (plain $ref
in anyOf) so docs/redoc/master/openapi.json stays consistent.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openapi): regenerate StemmingAlgorithm schema with generator output
Ran tools/generate_openapi_models.sh so docs/redoc/master/openapi.json
exactly matches generator output: DisabledStemmerParams/NoStemmer are placed
after SnowballLanguage, and the StemmingAlgorithm anyOf entry is a plain $ref
(the schema2openapi step flattens the allOf+description wrapper).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): avoid wildcard enum match arm in bm25 sparse_len helper
clippy --all-targets flags `other => panic!()` as wildcard_enum_match_arm;
match the Dense/MultiDense variants explicitly instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: issues
* fix: log::warn as call once
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
|
||
|
Andrey Vasnetsov
|
ba3472ea49 |
feat(id_tracker): deferred-aware mutable id tracker (#9249)
* feat(debug): QDRANT_APPEND_ONLY_MUTATIONS env override Debug-only escape hatch so newly built segments default to append-only mutation routing when QDRANT_APPEND_ONLY_MUTATIONS=1 (or true/yes) is set in the environment. Lets us run the existing test suites against the append-only path without wiring a collection-level config knob first. Release builds compile this out — the function is a const false. Logs a single warn-level message the first time the override fires. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(append-only): skip deleted vectors on snapshot, always write payload Two correctness fixes to clone_and_mutate_point uncovered by running the openapi suite with QDRANT_APPEND_ONLY_MUTATIONS=1: 1. The snapshot loop read every named vector via get_vector_opt, which returns slot bytes even when the per-vector deletion bit is set. For sparse-only points (or any point that had update_vector(_, None) applied) this materialised the default-zero vector as if it were real data, wrote it at new_id, and never re-tombstoned the slot — so dense search started scoring phantom vectors. Now we check is_deleted_vector(old_id) and skip the read, letting the writer loop emit update_vector(new_id, None) and re-mark the slot deleted. 2. The payload write was skipped when the snapshot ended up empty. That dropped two side effects the field indexes rely on: payload_storage.overwrite(new_id, empty), and the remove_point fan-out across configured field indexes that bumps each index's total_point_count to cover new_id. Without the bump the null index doesn't see new_id, so is_empty / is_null filters lose the point even though its mapping is live in the id tracker. The skip was an optimisation, not a contract; remove it so the field indexes get the same registration they'd get from the standard clear_payload path. Also collapses the debug env override to an inline cfg!()-gated check at the struct literal — the helper with one-time logging and multi-value matching was disproportionate for a debug-only escape hatch. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(id_tracker): split active vs deferred maps in PointMappings (PR A) First step of moving the mutable id tracker to a "two-track" model so append-only mutations into a deferred segment can keep both the visible (active) version and the latest mutated (deferred) version of a point at the same time. This PR is purely structural. On-disk format is unchanged: the loader still produces a single combined map, and `PointMappings::new` partitions it at construction time based on `deferred_internal_id`. Every observable behaviour at the existing API surface is preserved. Concretely: - New fields: `external_to_internal_num_deferred`, `external_to_internal_uuid_deferred`, and a `shadowed: BitVec` for future use by PR B (lazy-grown, default-false). - `internal_id(ext)` checks active first, falls through to deferred — matches the pre-split "any matching id" contract for ext ids whose internal id sat above the cutoff. - `set_link(ext, new_id)` now routes by cutoff: writes below the cutoff land in active, writes at or above land in deferred. Any prior head in the other track is tombstoned, so each ext still owns exactly one slot — same observable result as the pre-split single-map insert. PR B replaces the cross-track tombstone with a shadow-bit flip; PR A keeps current semantics on purpose. - `drop(ext)` clears entries from both tracks and tombstones each one, again matching the prior single-map behaviour. - `iter_external` and `iter_from` merge the active and deferred BTreeMap views into one sorted-by-key stream (dedup'd in case an ext exists in both tracks). - `available_point_count` counts distinct external ids across both tracks — preserves the prior observable count for segments where some entries used to sit above the cutoff in the single map. No write-path or read-path behaviour change. Reads still filter `internal_id >= cutoff` exactly as before; mutations still tombstone prior heads. The shadow bit and the deferred-aware lookup wiring land in PR B and PR C. All existing id_tracker tests pass against the split layout. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(id_tracker): shadow active head on deferred writes (PR B) Step 2 of the deferred-aware mutable id tracker. Replaces PR A's "tombstone the other track" cross-track cleanup with shadow-bit logic so a deferred mutation no longer hides the visible active version. Behaviour by case: - Active write (`internal_id < cutoff`, or no cutoff): unchanged semantically. Any prior deferred head for the same ext is dropped and tombstoned (the new active is the single visible head). Same observable result as PR A. - Deferred write (`internal_id >= cutoff`): the prior deferred head (if any) is dropped and tombstoned. The active head, if it exists, is **shadowed** — its bit is set in `shadowed: BitVec` but its slot stays alive in the active map. Read paths in `Exclude` mode continue to return that active version; PR C will teach `IncludeAll` paths (the optimiser) to skip shadowed actives and prefer the deferred head. Also adds `is_shadowed(internal_id)` and `shadowed_bitslice()` accessors (the latter for PR C's filter pipeline). New unit tests cover the routing matrix: - no cutoff — active replacement, no shadow, - below-cutoff replacement — active path, no shadow, - deferred-on-top-of-active — shadow set, active retained, - two deferred writes — prior deferred tombstoned, shadow persists, - fresh insert above cutoff — no shadow, - `drop(ext)` clears both tracks plus the shadow bit. WAL replay reuses the same `set_link`, so deferred routing on replay falls out of this change with no extra wiring. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(id_tracker): IncludeAll skips shadowed actives, deferred-aware lookups (PR C) Step 3 of the deferred-aware id tracker stack. PR B introduced the shadow bit on `set_link`; this PR teaches the read paths to honour it so the optimiser (and any other `DeferredBehavior::IncludeAll` consumer) sees each external id exactly once, with the deferred head winning over the now-shadowed active. Concretely: - `PointMappings::internal_id_with_behavior(ext, behavior)`: * `Exclude` returns the active head only — `None` for a deferred-only ext, so query paths never see a deferred mutation; * `IncludeAll` prefers the deferred head and falls back to active for points that never crossed the cutoff. Yields at most one internal id per ext. - `IdTrackerRead::internal_id_with_behavior` mirrors the new method with a default impl that delegates to `internal_id` for trackers that don't carry deferred mutations. - `PointMappingsRefEnum::iter_internal_with_behavior(IncludeAll)` now filters shadowed actives via the new `PointMappings::shadowed_bitslice()` accessor. - `PointMappingsRefEnum::filter_deferred_and_deleted(IncludeAll)` also filters shadowed actives — same single-yield-per-external guarantee for external iterator sources like field-index outputs. - `IdTrackerRead::resolve_external_ids` switches to the deferred-aware lookup. No more post-lookup `id >= cutoff` filter — the behaviour enum lookup gets it right at the source. New unit tests cover the two new entry points: - IncludeAll prefers the deferred head when an active is shadowed; - Exclude returns None for deferred-only ext ids; - `filter_deferred_and_deleted` over a mixed candidate list yields the expected per-mode result (Exclude: actives below cutoff; IncludeAll: every visible head, no shadowed actives). No queries observable behaviour change today — production Exclude paths still resolve via `internal_id` and the active head. The optimiser will start using `IncludeAll` (and reap the dedup) in follow-up work. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(id_tracker): expose shadowed_point_count in SegmentInfo (PR D) Final step of the deferred-aware id tracker stack. Adds a single new counter — number of active heads currently shadowed by a deferred mutation — and plumbs it through the id-tracker trait, the segment read-view helper, and `SegmentInfo` for telemetry. Preserves existing semantics: - `available_point_count` is unchanged. Each distinct external id still counts once, regardless of which track holds its head. - `deferred_point_count`, `deferred_internal_id`, `num_deleted_deferred_points` keep their current values. - Non-appendable trackers default `shadowed_point_count()` to `0`, so the new `SegmentInfo.num_shadowed_points` is `None` for them. Concrete changes: - `PointMappings::shadowed_count()` — popcount of the shadowed bitslice. - `IdTrackerRead::shadowed_point_count()` trait method with a `0` default; wired through `MutableIdTracker`, `InMemoryIdTracker`, the mutable read-only tracker, and both enum dispatchers. - `SegmentReadView::shadowed_point_count()` helper. - New `SegmentInfo.num_shadowed_points: Option<usize>`, populated with `Some(_)` for appendable segments and proxied through `ProxySegment` from the wrapped segment's value. New unit test covers the counter lifecycle: - active-only writes don't grow it, - a deferred write over an active adds one shadow, - a second deferred write supersedes the prior deferred head but the shadow stays put (still one active being shadowed), - `drop(ext)` clears the shadow bit, - a fresh deferred insert with no active prior doesn't add a shadow. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(id_tracker): rename DeferredBehavior, push behavior down to PointMappings Three connected pieces of polish on top of the deferred-aware id tracker stack: 1. Rename `DeferredBehavior::Exclude` → `VisibleOnly` and `IncludeAll` → `WithDeferred`. The pre-PR-C semantics of "include/exclude the deferred cutoff" became misleading once `IncludeAll` started skipping shadowed actives — it doesn't "include all" anymore, it yields one slot per external (deferred head preferred, active fallback). New names describe what each variant returns instead of how it relates to the cutoff. The helper method `include_all_points` becomes `with_deferred_points`. Updates ~90 call sites across the workspace; behaviour is unchanged. 2. Push `iter_internal_with_behavior` down from `PointMappingsRefEnum` into `PointMappings`. The per-mode logic (cutoff `take_while`, shadowed `filter`) now lives next to the data it consults; the enum layer becomes a two-arm `Either` dispatcher. `CompressedPointMappings` short-circuits to `iter_internal()` since compressed mappings can't carry deferred mutations. 3. Add a short docstring on `internal_to_external` describing the two-track model: no active-vs-deferred bias, shadowed pairs occupy two slots with the same value, and reads must gate on `deleted` because `set_link`'s same-track replacement leaves a real-looking stale ext id in place. Returning `impl Iterator` instead of `Box<dyn Iterator>` for `iter_internal`, `iter_internal_excluding`, `iter_internal_visible`, and `iter_internal_with_behavior` removes the double-Box at the enum boundary. The original branching structure is preserved with `itertools::Either` instead of restructured into one big filter chain. All existing `id_tracker` tests still pass (47 total). `cargo check --all-targets` + `cargo clippy --all-targets` clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(id_tracker): drop shadowed API surface, unbox iter_from/iter_random Cleanup pass on top of the deferred-aware id tracker stack. API surface: - Drop SegmentInfo.num_shadowed_points + its trait method, read-view helper, tracker impls and ProxySegment passthrough. Cross-segment shadow already exists in the appendable update flow (the source segment keeps its copy when the new write lands above the cutoff, see SegmentHolder::apply_points_with_conditional_move) and is handled at the aggregation layer, not reported per-tracker. - Keep PointMappings::shadowed_count() as a private popcount helper guarded by expect(dead_code) so a future caller can opt back into the dedup'd count without re-plumbing the trait. available_point_count: - Replace the cross-track dedup (active + deferred-only via contains_key filter) with a straight 4-term sum. A shadowed ext contributes two slots, matching the two non-tombstoned internal ids it actually occupies. Old dedup broke the invariant that deleted_point_count == deleted_bitslice.count_ones(): for each shadow it overcounted deletions by one without any tombstone actually being set. DeferredBehavior pushdown: - iter_random_with_behavior: caps the sampling range at the deferred threshold in VisibleOnly mode (no wasted samples above cutoff), filters shadowed actives via the bit in WithDeferred. - iter_from_with_behavior: VisibleOnly walks the active maps only (no merge with deferred); WithDeferred delegates to iter_from's existing merge. - scroll.rs read_by_id_stream / filtered_read_by_id_stream collapse their manual if-deferred-behavior branching into a single iter_from_with_behavior call. - Old iter_random (no behavior) at PointMappings + ref enum was unused after the migration, deleted. Unboxing iter_from / iter_random / iter_from_with_behavior: - PointMappings::iter_from returns impl Iterator + '_ via Either inside the merged_num/merged_uuid closures (BTreeMap::iter vs range), Either at the outer match (num+uuid chain vs uuid-only). - PointMappings::iter_from_with_behavior unboxed with a triple Either (behavior, external-id arm, closure start). - CompressedPointMappings::iter_from unboxed (Either over None/Some). - PointMappingsRefEnum::{iter_from, iter_from_with_behavior, iter_from_visible} all return impl Iterator + 'a via Either on the Plain/Compressed dispatch. Other: - Update outdated PR-A/PR-B comment on PointMappings::drop. - Make the max_internal match in iter_random_with_behavior exhaustive (VisibleOnly+None | WithDeferred+_ instead of `_`). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(id_tracker): unbox iter_external Return impl Iterator from PointMappings::iter_external, CompressedPointMappings::iter_external and the PointMappingsRefEnum wrapper, matching the style of the other iter_* helpers. The wrapper dispatches via Either. The remaining Box::new at Segment::iter_points stays because self_cell's BoxedPointIdIterator alias needs a sized type. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(id_tracker): count set_link tombstones in deferred_deleted_count PointMappings::set_link tombstones the prior slot at two sites — when an active write supersedes a deferred head (cross-track) and when any write replaces a same-track head — using a bare deleted.set(old, true). Neither path updated deferred_deleted_count, so tombstones above the cutoff weren't reflected in the counter. The append-only flow exposes this constantly: every set_full_payload after upsert_point routes through clone_and_mutate_point, which re-issues set_link with a fresh internal id and tombstones the prior one. Most tombstones land above the cutoff, so deferred_point_count (total - cutoff - deferred_deleted_count) over-reports by the missing count. The openapi test_deferred_points integration test caught this as `num_points - num_deferred_points = -1800` across two segments. Extracted the tombstone bookkeeping into PointMappings::tombstone_slot and routed both set_link sites + drop's loop through it. Behaviour on drop is unchanged; set_link now bumps the counter on the live → tombstoned transition for slots at or above the cutoff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style(id_tracker): collapse set_link tombstone if-let chains Clippy's collapsible_if on the two if-let blocks added by the deferred_deleted_count fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update lib/segment/src/id_tracker/point_mappings.rs Co-authored-by: Tim Visée <tim+github@visee.me> * refactor(id_tracker): size shadowed BitVec once instead of growing lazily Collect the shadowed active ids up front and allocate the BitVec to the highest offset in a single resize, avoiding repeated reallocations while marking shadows. Addresses review feedback. Co-authored-by: Cursor <cursoragent@cursor.com> * revert: restore lazily-grown shadowed BitVec Roll back the up-front sizing of the shadowed BitVec; last_entry doesn't fit here and swapping one allocation for another isn't worthwhile. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(id_tracker): prefer deferred head in iter_from merge When an external id has both an active and a deferred head, iter_from's merge collapsed the pair to the active (stale) offset. That contradicts the WithDeferred contract used everywhere else: internal_id_with_behavior and iter_random_with_behavior both surface the deferred head (the latest mutation) over the shadowed active. Consumers that use the returned internal id (payload-filter checks, the optimizer's version merge, HNSW old->new mapping) therefore saw the stale copy. Flip the Both arm to take the deferred operand and document the rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(id_tracker): preserve active+deferred heads when loading mappings read_mappings replayed the persisted change log into a single flat external -> internal map per id-type, then split it by the deferred cutoff in PointMappings::new. A flat map holds one head per external id, so it collapsed the active+deferred coexistence case (an external id linked first to an active slot, then to a deferred one via sequential set_link) down to the last write — silently dropping the other head, orphaning its slot, and leaving the shadowed bit unset. The split in new() could not recover what was already lost before it. Replay the log through the canonical set_link/drop mutators on a PointMappings seeded with the cutoff instead. The log is the sequence of set_link/drop calls that produced the live in-memory state, so this reconstructs that state exactly — both heads, the shadowed bit, and deferred_deleted_count — with no logic duplication. Drops the debug_assert-guarded corruption-recovery branch (subsumed by set_link's re-link handling) and the now-unused Uuid/PointIdType imports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(id_tracker): make deferred behavior explicit at point resolution Replace the ambiguous "any matching head" point resolution with an explicit DeferredBehavior at every resolution boundary, so callers no longer rely on a hidden active-vs-deferred policy. - Remove PointMappings::internal_id and the bare IdTrackerRead::internal_id (the active-first-else-deferred hybrid). internal_id_with_behavior is now the single required resolution method; immutable/compressed trackers implement it by ignoring the behavior (they never carry deferred heads). - Migrate every caller to an explicit behavior, audit-driven: - writes (upsert/delete/payload/vectors), point_version, point_is_deferred, get_internal_id, drop, consistency + builder dedup -> WithDeferred (the latest/live head); - single-point payload/vector retrieval and formula rescore -> VisibleOnly; - HasId/CustomIdChecker/cardinality resolution -> the request's behavior, threaded through the filter chain from iter_filtered_points (other entry points default to VisibleOnly). - lookup_internal_id takes an explicit DeferredBehavior instead of assuming VisibleOnly internally. - has_point takes an explicit DeferredBehavior (drop the has_point_with_behavior wrapper). Thread it through read_points/_read_points/read_points_locked so retrieve_blocking passes its request behavior to the existence filter; all other existence/dedup callers pass WithDeferred (unchanged behavior). - set_link now detaches a stale live occupant of a reused internal id, keeping the forward and reverse maps consistent when recovering a corrupted log. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(id_tracker): failing tests for two correctness findings in #9249 (#9311) * test(id_tracker): failing tests for two review findings Two intentionally-failing tests pinning correctness gaps in the deferred-aware id tracker (#9249). Both fail on the final assertion; the earlier assertions establish the expected/consistent behavior. 1. iter_from_with_behavior(WithDeferred) resolves an active/deferred shadow collision to the stale ACTIVE internal id, while its siblings internal_id_with_behavior and iter_internal_with_behavior correctly surface the DEFERRED (latest) head. Consumers that use the yielded internal id (optimizer merge via for_each_unique_point, filtered_read_by_id_stream) therefore observe the pre-mutation version. left: [(NumId(7), 2)] right: [(NumId(7), 9)] 2. The PR-B shadow/visible invariant is not durable: the on-disk single combined map cannot represent a shadowed ext, so a plain mappings flush + reload collapses the shadow to deferred-only and the visible (active) head is lost (VisibleOnly resolves None where it resolved Some(2) live). Restoration then depends entirely on WAL replay, i.e. on flush-vs-WAL-truncate ordering. left: None right: Some(2) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(id_tracker): strengthen shadow tests + merge-primitive proof Follow-up to the two failing tests, addressing self-review: - Add for_each_unique_point_keeps_deferred_head_for_shadowed_point: the optimizer merge primitive (used by segment_builder::update_from) yields the stale active copy (internal 2, version 5) for a shadowed point and drops the deferred latest (internal 9, version 8). This directly exercises the data-loss consequence of finding #1 at the merge layer. left: [(NumId(7), 2, 5)] right: [(NumId(7), 9, 8)] - Tighten shadow_visible_head_survives_mapping_flush_reload: pin the exact reload failure mode. After flush+reload the mapping collapses to deferred-only (internal_id == Some(9)) and the active slot survives as a live orphan in the inverse map (external_id(2) == Some(7), not deleted) — a torn state where a VisibleOnly scroll still surfaces the stale copy while by-id VisibleOnly resolution breaks. Reframe as the live-vs-reload divergence the PR introduces (Some(2) live -> None reload; dev is consistently None). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix test --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: generall <andrey@vasnetsov.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Tim Visée <tim+github@visee.me> Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com> |
||
|
Tim Visée
|
7041b81f76 |
Use assert_matches! (#9231)
* Use assert_matches! * Add trailing commas * Use more assert_matches! Also, drop now redundant `expected blah but got blah` messages because `assert_matches!` will print these. * Use debug_assert_matches! --------- Co-authored-by: xzfc <xzfcpw@gmail.com> |
||
|
qdrant-cloud-bot
|
65b7f52d18 |
Add Rust edge example: add-named-vector (#9230)
* Add Rust edge example: add-named-vector Port `lib/edge/python/examples/add-named-vector.py` to Rust under `lib/edge/publish/examples/src/bin/add-named-vector.rs`. Re-export `VectorNameOperations`, `CreateVectorName`, `DeleteVectorName`, `VectorNameConfig`, `DenseVectorConfig`, and `SparseVectorConfig` from `qdrant_edge` so the public Rust API can create/delete named vectors without reaching into the internal `shard` crate. Co-authored-by: Cursor <cursoragent@cursor.com> * fmt Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
qdrant-cloud-bot
|
6d8f3c6ce9 |
ci(windows): skip IO-heavy tests that aren't OS-specific (#9188)
* ci(windows): skip IO-heavy tests that aren't OS-specific On the Windows CI runner, several tests are 3-25x slower than on Ubuntu purely due to slow filesystem IO. These tests exercise platform-agnostic logic (optimizer, snapshot, WAL recovery, dedup, deferred points) and are fully covered by the Linux and macOS jobs. Mark them with `#[cfg_attr(target_os = "windows", ignore = "...")]` so: - Windows CI skips them and finishes faster. - They're still listed and runnable via `cargo test -- --ignored` on Windows for local debugging. Based on JUnit timings from CI run 26462785436 (PR #8827), this should save ~5 minutes wall-clock on the Windows job, taking it closer to the ~13min Ubuntu and ~9min macOS jobs (currently 20m24s). Tests affected: - lib/wal: check_wal, check_last_index, check_clear, check_reopen, check_truncate, check_prefix_truncate, test_prefix_truncate_parametric - lib/edge/optimize: full tests module - lib/segment deferred-point tests: read_operations, dense_segment_combinations, sparse, facets - lib/collection: snapshot_test, points_dedup, wal_recovery, collection_test::test_ordered_read_api, snapshot_recovery_test Co-authored-by: Cursor <cursoragent@cursor.com> * revert(ci/windows): keep WAL and WAL-recovery tests on Windows Reviewer correctly pointed out that WAL is mmap-backed and has substantial Windows-specific code paths: - Different segment allocation (fs4 vs rustix::ftruncate) - Windows-specific delete_windows() with mmap-drop + retry loop - Windows-specific sync_all() because directory fsync is unavailable - Windows-specific lock proxy file (directories aren't lockable) So those tests genuinely need Windows coverage. Reverted skips for: - lib/wal/src/lib.rs: all check_* tests and test_prefix_truncate_parametric - lib/collection/src/tests/wal_recovery_test.rs: all three tests Still skipped on Windows (no OS-specific code in their production paths): - lib/edge/optimize.rs (no cfg(windows) in source) - lib/segment deferred-point tests (segment/ has no cfg(windows)) - lib/collection snapshot/dedup tests (collection/ has no cfg(windows)) - lib/collection integration snapshot_recovery + ordered_read_api Co-authored-by: Cursor <cursoragent@cursor.com> * revert(ci/windows): keep collection integration and snapshot_test Per reviewer request, keep running these on Windows: - lib/collection/tests/integration/* (snapshot_recovery_test, collection_test::test_ordered_read_api) - lib/collection/src/tests/snapshot_test.rs These exercise higher-level collection/snapshot behavior that benefits from cross-platform validation. Remaining Windows skips (production code has no cfg(windows) branches): - lib/edge/src/optimize.rs: 14 optimizer tests - lib/segment/src/segment/tests/mod.rs: 4 deferred-point tests - lib/collection/src/tests/points_dedup.rs: 2 dedup tests Co-authored-by: Cursor <cursoragent@cursor.com> * ci(windows): also skip HNSW/quantization integration tests Per reviewer, also skip these segment integration test modules on Windows: - hnsw_quantized_search_test::* (25 tests) - multivector_filtrable_hnsw_test::* (rstest cases) - multivector_quantization_test::* (rstest cases) - byte_storage_quantization_test::* (rstest cases) - payload_index_test::test_struct_payload_index_nested_fields These exercise pure HNSW/quantization correctness on top of standard segment IO that is already covered by tests we keep running on Windows. Adds ~930s of sequential time to the Windows skip list, bringing the expected wall-clock saving from ~3 min to ~8-10 min. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Daniel Boros
|
c4f70cc445 |
feat/edge-bm25 (#8827)
* feat: integrate bm25 into edge * fix: linter * fix: imports * fix: qdrant-edge build errors * fix: spell check * feat: integrate inference * fix: api missmatch * chore: remove inference * fix: coderabbit comments * fix: compiler error * chore: remove wrapper type * feat: add some temp tests for legacy check * add example --------- Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> |
||
|
Sasha Denisov
|
797779a159 |
feat(edge): expose WAL options via EdgeShard::load_with_wal_options (#9067)
* feat(edge): expose WAL options via EdgeShard::load_with_wal_options Adds a sibling method to EdgeShard::load that accepts custom WalOptions, alongside the existing load() which keeps using default_wal_options(). Motivation: embedded/mobile deployments (e.g. Flutter plugins on iOS/Android) need much smaller WAL segments than the 32 MiB default — typically 4 MiB. On filesystems with sparse files (APFS/ext4/F2FS) the physical footprint is small, but the visible/reported size is the full segment capacity, which is surfaced by iCloud backup, OS size pickers, etc. Changes: * New EdgeShard::load_with_wal_options(path, config, wal_options). * EdgeShard::load delegates to it with default_wal_options() — no behavior change for existing callers. * ensure_dirs_and_open_wal now takes WalOptions explicitly; called from both EdgeShard::new (with default) and load_with_wal_options. * WalOptions re-exported from edge crate root. * Two regression tests in lib/edge/tests/wal_options.rs. WAL options are intentionally a runtime parameter, not part of EdgeConfig (which is persisted to edge_config.json) — different processes may legitimately open the same shard with different WAL options. * test(edge): verify mismatching WAL options on reload preserve data Addresses upstream review feedback (qdrant/qdrant#9067): demonstrate that reloading an existing shard with WAL options different from the ones it was created with is safe and preserves all previously written points. Two new tests in lib/edge/tests/wal_options.rs: * reload_with_smaller_wal_capacity_after_upsert: create shard with default 32 MiB WAL -> upsert point 42 -> drop -> reload with 4 MiB WAL options -> point 42 still readable -> upsert point 43 under smaller WAL -> count == 2. * reload_with_larger_wal_capacity_after_upsert (symmetric): create -> reload with 4 MiB -> upsert point 100 -> drop -> reload with default 32 MiB -> point 100 readable -> upsert point 101 -> count == 2. Both pass. The WAL crate does not validate segment_capacity on reload — existing segments on disk keep their original size, subsequent appends honor the runtime options. This confirms the PR description's claim that WalOptions is a runtime hint, not persisted shard state. * style(edge): cargo fmt for wal_options.rs mismatch tests * refactor(edge): replace load_with_wal_options with builder-based options Addresses upstream feedback (timvisee, generall): the sibling constructor pattern doesn't scale as runtime configurability grows. Replaces `EdgeShard::load_with_wal_options(path, config, wal_options)` with `EdgeShard::load_with_options(path, config, EdgeShardOptions)`, where `EdgeShardOptions` is a builder-style runtime-options struct. * `EdgeShardOptions::new().with_wal_options(opts)` is the equivalent of the old call. * Adding new runtime options later (e.g. wal flush interval, initial indexing threshold) is now an additive change — new builder method on `EdgeShardOptions`, no new constructor variant on `EdgeShard`. * `EdgeShardOptions` is intentionally not persisted (no Serialize/Deserialize). The reload-mismatch tests above confirm that WAL options are a runtime hint, not shard identity, so they don't belong in `EdgeConfig` (which is persisted to `edge_config.json`). * `EdgeShard::load(path, config)` unchanged for existing callers — delegates to `load_with_options(..., EdgeShardOptions::default())`. Tests in `lib/edge/tests/wal_options.rs` migrated to the new shape; all four still pass: test load_with_options_accepts_custom_wal_capacity ... ok test load_still_works_with_default_wal_options ... ok test reload_with_smaller_wal_capacity_after_upsert ... ok test reload_with_larger_wal_capacity_after_upsert ... ok * refactor(edge): add fluent builders; move wal_options into EdgeConfig Replaces the EdgeShardOptions side-channel with a single EdgeConfig that carries wal_options inline, and introduces fluent builders for the three user-facing config types. * WalOptions now derives Clone/PartialEq/Eq/Serialize/Deserialize so it can live inside EdgeConfig and round-trip through edge_config.json. * EdgeConfig.wal_options (Option<WalOptions>) replaces EdgeShardOptions. EdgeShard::load_with_options is folded into EdgeShard::load; both new and load drive the WAL from config.wal_options.unwrap_or_default(). * New builders/ module hosts EdgeConfigBuilder, EdgeVectorParamsBuilder, and EdgeSparseVectorParamsBuilder. Each builder has explicit per-field storage and constructs its target via an exhaustive struct literal in build(), so adding a field to the target forces a compile error in the builder. * publish example switched to the new builder API. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style(edge): cargo fmt after builder refactor Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
qdrant-cloud-bot
|
26aeb9c0b2 |
docs: refresh prevent_unoptimized description (#9133)
The previous description was outdated: it claimed that enabling this option "blocks updates at the request level" until segments are re-optimized. In practice the implementation uses "deferred points": new points written to large unoptimized segments are persisted but excluded from read/search results until the segments are optimized. Updates are not blocked; only `wait=true` clients are made to wait for the deferred points to become visible. Update this in the REST schema (via `OptimizersConfig` / `OptimizersConfigDiff`), in the gRPC proto, in the edge config docstrings, and regenerate the OpenAPI bundle via `tools/generate_openapi_models.sh`. Co-authored-by: Cursor Agent <agent@cursor.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
xzfc
|
faf2714f4b | Warn on clippy::wildcard_enum_match_arm (#9096) | ||
|
Arnaud Gourlay
|
8ab30ee089 | Reinstate edge add_vector API (#8925) | ||
|
Arnaud Gourlay
|
354bbb35e6 |
Delete unused code (#8771)
* Delete unused code * restore initialize_global * drop BadShardSelection * Remove now obsolete allow(dead_code) attributes * Remove more dead code --------- Co-authored-by: timvisee <tim@visee.me> |
||
|
Andrey Vasnetsov
|
acfb6503b1 |
crud named vectors (#8605)
* Add empty placeholder vector storage types for named vector CRUD Introduce EmptyDenseVectorStorage and EmptySparseVectorStorage as placeholder storages for newly created named vectors on immutable segments. These report all vectors as deleted, consume no disk space, and are reconstructed from segment config on load via the new VectorStorageType::Empty and SparseVectorStorageType::Empty variants. Key design decisions: - is_on_disk is derived from original user config, not hardcoded - MultiVectorConfig is preserved for multi-vector support - Config mismatch optimizer skips Empty storage to avoid false rebuilds - Quantization delegates normally (handles 0 vectors gracefully) - get_vector includes debug_assert to catch unexpected access Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * [AI] segment-level operations for creating and deleting anmed vectors * [AI] implement named vector creation and deleting in proxy segment * [AI] Step 3: Proxy Segment Handling for Named Vector Operations * [AI] implement for Edge * [AI] implement consensus operations for named vector operations * [AI] refactor VectorNameConfig, remove VectorNameConfigInternal * [AI] handle vector schema inconsistency in raft snapshot recovery * [AI] rest + grpc API * [AI] clippy * [AI] generate openAPI schema * fmt * ci fixes * [AI] fix jwt access test * [AI] nop operation for awaiting of consensus-commited update ops * [AI] move vector name operations into points service * [AI] implement internal api for vector name operations * [AI] change collection-level config along with segment level operation * [AI] vector schema reconceliation instead of error * fmt * missing compile-time option * [AI] integration test * [AI] fix missing JWT tests * [AI] remove NOP * [AI] openapi test * [AI] fix initialization of mutable segment * [AI] more simple integration tests * fmt * [AI] make cluster test a bit harder * [AI] make test less flacky * [AI] rabbit comments * [AI] check params compatibility before writing vector config * [AI] make sure to register vector storages in structure payload index * [AI] vector name validation * lower vector length validation to 200 chars to account for prefix in filename * [AI] proxy segment: prevent stale data leak through optimization * fmt * [AI] filter out removed vectors from proxy response * [AI] handle vector name in proxy * fmt * adjust proxy info based on dropped vectors * [AI] proxy segment: update filters to correct has_vector condition * fmt * clippy * Fix consensus snapshot applicaiton for vector schema --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
ecae89ef4d |
Fix edge sparse vector search panic on score postprocessing (#8543)
* Fix edge sparse vector search panic on score postprocessing The distance lookup for score postprocessing only checked dense vector configs, causing a panic when searching sparse vectors. Fall back to Distance::Dot for sparse vectors, matching the full server behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix formatting Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Address PR review: return error instead of panic, fix example - Replace panic! with OperationError::service_error for unknown vector names in edge search, avoiding process crash on bad client input - Update stale "panics" comments and add assertions in sparse-search example Made-with: Cursor --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor Agent <agent@cursor.com> |
||
|
Ivan Boldyrev
|
2a0b95b08d |
Further split segment traits (#8434)
* Refactor: `SearchSegmentEntry` for search ops * Move `has_deferrend_points_method` to `SearchSegmentEntry` * Reorg imports * Renamings per review * fixup! Renamings per review * `StorageSegmentEntry` trait It contains methods dealing with storage and syncronization * Move index methods from `ReadSegmentEntry` to `SegmentEntry` * Add `_concurrent` suffix to some methods * move field index methods into NonAppendableSegmentEntry --------- Co-authored-by: generall <andrey@vasnetsov.com> |
||
|
Jojii
|
356040312b |
Correct calculation of deferred point counts (#8366)
* Don't account for deferred points in some places # Conflicts: # lib/collection/src/shards/local_shard/scroll.rs * Add in QueryContext * Cover more places * Coderabbit review remarks * Properly count amount of deleted deferred points (#8386) * Properly count amount of deleted deferred points * Prevent double-counting of the same point * Remove hints to estimations * Properly handle counts in ProxySegment * Add tests and fix deleted point count issue * Adjusts tests + fix issues * Separte fields for deferred points in telemetry (SegmentInfo) * Remove deferred_points_count() * openapi * Fix test by manually calculating visible points * Adjust ProxySegment test to revertion of SegmentInfo * Throw error if collection was not found in telemetry (e2e Test) * Update lib/segment/src/segment/segment_ops.rs Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> * Make new fields in API optional * Don't take range if no deferred point exist * Review remarks --------- Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> |
||
|
xzfc
|
9aaa7c649b | Propagate OperationResult [2/4]: SegmentEntry (#8445) | ||
|
qdrant-cloud-bot
|
09b3ff00cb |
refactor(edge): split EdgeShard load into new() and load() (#8326)
* refactor(edge): split EdgeShard load into new() and load() - new(path, config): create edge shard only when path has no existing segments - load(path, config?): load from existing files; config optional (load from edge_config.json or infer from segments) - Helpers: has_existing_segments, ensure_dirs_and_open_wal, resolve_initial_config, load_segments, ensure_appendable_segment - Python: EdgeShard.create_new(path, config); __init__ still calls load() - Examples use new() for creation and load(path, None) for reopen - Error instead of panic on config mismatch; explicit load/create in Python - fix: use OptimizerThresholds.deferred_internal_id for dev compatibility Made-with: Cursor * rollback threshold changes * fix(edge): address dancixx review comments - Persist inferred config in load() when edge_config.json does not exist (SaveOnDisk::new only saves when file exists) - Python: add #[new] delegating to load() for backward compatibility - qdrant_edge.pyi: Optional return types for deleted_threshold, vacuum_min_vector_number, default_segment_number; add __init__ doc Made-with: Cursor * Revert "fix(edge): address dancixx review comments" This reverts commit 370e21a6c74c41210e1658f89814395cb86ed81c. * fix: optional returns * fix: save config it is not exist * fix: save data on disk always * fix: python examples * chore: remove edge.close() * fix: wal lock * fix: rename of EdgeShardConfig -> EdgeConfig --------- Co-authored-by: Cursor Agent <agent@cursor.com> Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> Co-authored-by: Daniel Boros <dancixx@gmail.com> |
||
|
xzfc
|
40e12be1a8 |
Edge flat api (#8381)
* refactor(edge): let edge re-export segment/shard/sparse * chore(ast-grep-rules): handle path like `::shard::blah::Blah` This commit was required during the refactoring. It turns out we don't need it right now, but it might be useful in the future. Or not. * refactor(edge-rs): Rehaul public API |
||
|
xzfc
|
8ab0057566 | DiscoveryQuery -> DiscoverQuery (#8378) | ||
|
Jojii
|
e7e5f4cc4f |
Filter deferred points: facets (#8313)
* Filter deferred: facets * Add test (facet) * Review remark * Review remarks * Fix unique_values returning values even if occurring only in deferred points * Clippy * Rename Filter=>Exclude |
||
|
Jojii
|
f91e2179d5 |
Disable deferred point filtering in resharding (#8310)
* Add ignore_deferred flag for internal scroll API * Ignore deferred points in ProxyShard::update * Use enum instead of bool as type in function parameters * Use consistent parameter name * fmt * Apply suggestions from code review Co-authored-by: Tim Visée <tim+github@visee.me> * More explicit naming * Add DeferredBehavior to `count()` and disable filtering in stream_records * Don't filter `retrieve()` everywhere * Use consistent filter behavior in read operations * Add TODO for ignored parameter in remote_shard * Rebase fixes --------- Co-authored-by: timvisee <tim@visee.me> Co-authored-by: Tim Visée <tim+github@visee.me> |
||
|
Ivan Pleshkov
|
4b9b39c596 |
Use predefined deferred ID (#8329)
* Adjust points selection for deferred points update * adjust proxy segment implementation * simplify * use simpler proxy impl * stick to Entry API * renaming to stay closer to the original * two passes and simpler impl. * fmt * fmt * use predefined deferred internal id * calculate deferred point id * move deferred check to the entry * fix after rebase * fmt * fix tests * review remarks * fix tests * codespell fix * are you happy clippy --------- Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com> |
||
|
qdrant-cloud-bot
|
7e15d343c2 |
Introduce EdgeShardConfig for edge shard (#8322)
* Introduce EdgeShardConfig for edge shard
- Add EdgeShardConfig and EdgeOptimizersConfig in lib/edge/src/config.rs
- Segment config (vector_data, sparse_vector_data, payload_storage_type)
- Global hnsw_config and per-vector HNSW in segment config
- Optimizer params: deleted_threshold, vacuum_min_vector_number,
default_segment_number, max_segment_size, indexing_threshold,
prevent_unoptimized (excludes memmap_threshold, flush_interval_sec,
max_optimization_threads)
- Persist/load as edge_config.json in shard path
- EdgeShard uses RwLock<EdgeShardConfig>; load() accepts Option<EdgeShardConfig>,
falls back to file or infer from segments; compatibility checked on load
- load_with_segment_config() for backward compatibility (SegmentConfig -> EdgeShardConfig)
- optimize() uses EdgeShardConfig for hnsw and optimizer thresholds
- Public methods: set_hnsw_config(), set_vector_hnsw_config(), set_optimizers_config()
(update and persist)
- Python and examples use load_with_segment_config with existing config API
Made-with: Cursor
* Refactor EdgeShardConfig: user-facing params only, config module
- Replace SegmentConfig inside EdgeShardConfig with user-facing fields:
- on_disk_payload (bool) instead of payload_storage_type
- vectors: HashMap<VectorNameBuf, EdgeVectorParams> with on_disk per vector,
no per-vector quantization; global quantization_config only
- sparse_vectors: HashMap<VectorNameBuf, EdgeSparseVectorParams> with on_disk
- EdgeVectorParams / EdgeSparseVectorParams use on_disk (bool) instead of
storage_type; conversion to VectorDataConfig/SparseVectorDataConfig in
to_segment_config()
- Add config module: mod.rs, optimizers.rs, vectors.rs, shard.rs
- from_segment_config(&SegmentConfig) fills all inferrable params
- to_segment_config() builds SegmentConfig for segments and optimize()
- load_with_segment_config takes Option<SegmentConfig>, uses from_segment_config
Made-with: Cursor
* Move optimizer threshold helpers to shard crate
- Add get_number_segments, get_indexing_threshold_kb, get_max_segment_size_kb,
get_deferred_points_threshold_bytes in shard::optimizers::config
- Collection OptimizersConfig and edge EdgeOptimizersConfig delegate to these
- Single place for threshold logic; collection and edge use shard helpers
Made-with: Cursor
* Use destructuring in config conversions to avoid missing new fields
- EdgeVectorParams: destructure VectorDataConfig in from_*, destructure self in to_vector_data_config
- EdgeSparseVectorParams: destructure SparseVectorDataConfig and SparseIndexConfig in from_*, destructure self in to_sparse_vector_data_config
- EdgeShardConfig: destructure SegmentConfig in from_segment_config, destructure self in to_segment_config
Adding new fields to source structs will now cause compile errors until conversions are updated.
Made-with: Cursor
* refactor: centralize on_disk_payload→payload_storage_type, on_disk→storage_type, and appendable quantization logic
- PayloadStorageType::from_on_disk_payload(bool) in segment (Mmap/InRamMmap)
- VectorStorageType::from_on_disk(bool) in segment (ChunkedMmap/InRamChunkedMmap)
- QuantizationConfig::for_appendable_segment(Option<&Self>) in segment (feature flag + supports_appendable)
- collection: use from_on_disk_payload in non-rocksdb branch
- edge shard/vectors: use new helpers; remove duplicated conditionals
- shard optimizers: use from_on_disk and for_appendable_segment
Made-with: Cursor
* refactor(edge): use EdgeShardConfig directly, drop segment_config
- Add plain_segment_config() for create_appendable_segment (no HNSW)
- Add segment_optimizer_config() built from EdgeShardConfig for blocking optimizers
- Add vector_data_config(name) for query/MMR
- build_blocking_optimizers: use segment_optimizer_config() instead of SegmentConfig
- create_appendable_segment: use plain_segment_config()
- search/query: use config().vectors and vector_data_config() instead of segment_config()
- Remove segment_config() from EdgeShardConfig and EdgeShard
- Add to_plain_vector_data_config on EdgeVectorParams
Made-with: Cursor
* [manual] review changes
* refactor(edge-py): wrap EdgeShardConfig, add EdgeVectorParams/EdgeSparseVectorParams
- PyEdgeConfig now wraps EdgeShardConfig (vectors, sparse_vectors, on_disk_payload, etc.)
- PyEdgeVectorParams / PyEdgeSparseVectorParams wrap edge config types
- PyEdgeOptimizersConfig for optional optimizer settings
- EdgeShard.load() uses EdgeShardConfig; edge::config made pub for Python crate
- cargo fmt + clippy (remove map_identity)
Made-with: Cursor
* refactor(edge-py): simplify config API, remove unused Py* types, add EdgeConfig
- Remove unused PyPayloadStorageType, PyVectorDataConfig, PyVectorStorageType,
PySparseVectorDataConfig, PySparseVectorStorageType from Python bindings
- Move PyEdgeOptimizersConfig to lib/edge/python/src/config/optimizers.rs
- Update qdrant_edge.pyi: EdgeConfig with vectors/sparse_vectors,
EdgeVectorParams, EdgeSparseVectorParams, EdgeOptimizersConfig
- Update examples (common.py, repr.py) to use new config API
- Run cargo fmt
Made-with: Cursor
* [manual] review changes
* [manual] review changes
* [manual] fix test
* Address CodeRabbit review comments for PR 8322 (#8324)
* Address CodeRabbit review comments for PR 8322
- Python examples: explicit imports (repr.py, common.py) and new EdgeConfig API
- HnswIndexConfig: add max_indexing_threads param and property in .pyi and Rust bindings
- EdgeConfig: make vectors optional for sparse-only configs; validate at least one of vectors/sparse_vectors
- EdgeShardConfig::load: use try_exists(), propagate I/O errors
- from_segment_config: infer hnsw_config from per-vector HNSW when all agree
- EdgeShard setters: atomic clone-mutate-save-then-replace; persist config save errors
- Segment compat: prefix vector name in error messages; resolve None datatype to Float32
- max_indexing_threads: preserve 0 (auto) sentinel in trait default; remove per-optimizer overrides
- SegmentOptimizerConfig:🆕 build plain and optimizer maps in single pass
- config_mismatch_optimizer tests: use VectorNameBuf::from() instead of .into()
- vectors.rs: doc updates for per-vector quantization
Made-with: Cursor
* Address @generall review: SaveOnDisk for config, resolve num_rayon_threads in optimizer
- Use SaveOnDisk<EdgeShardConfig> for EdgeShard config (generall: 'We have SaveOnDisk struct for this')
- Create via SaveOnDisk::new() after resolving config; setters use .write() for atomic persist
- set_vector_hnsw_config: clone then mutate then write (fallible setter)
- max_indexing_threads: resolve 0 (auto) via num_rayon_threads inside impl (generall: 'proper solution would be to resolve num_rayon_threads inside the optimizer impl')
- max_indexing_threads_sentinel_aware() now returns Some(num_rayon_threads(raw)) so callers get actual thread count
Made-with: Cursor
* [manual] reorganize num_rayon_threads -> get_num_indexing_threads to better account per-vector configuration
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* update docstring and pyi
* fmt
* fmt
* clipy
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
|
||
|
qdrant-cloud-bot
|
b3a6cba0ff |
Make edge crate submodules private (#8319)
Change pub mod to mod for count, facet, info, optimize, query, retrieve, scroll, search, snapshots, and update in lib/edge. The public API (EdgeShard, ShardInfo) is unchanged. Made-with: Cursor Co-authored-by: Cursor Agent <agent@cursor.com> |
||
|
Daniel Boros
|
d4cb6a58d9 |
feat/edge segment opt (#8224)
* feat: add edge shard optimize * feat: refactor edge optimize logic * chore: remove unused &self * feat: add more tests * fix: linter * fix: missing threshold prop * fix: local nightly version * fix: linter * fix: linter issues * fix: use of explicit from * feat: add some notes * fix: feature_flags call once * [manual] review refactor * fix: - default_segment_number -> move shard - rename: default_hnsw_config -> hnsw_config - infer existing hnsw_config * feat: add optimize to python * feat: add python optimize example * feat: add hnsw config load tests * fix: linter * feat: make unified build config * fix: linter * fix: openapi definition' * fix: review comments * fix: remove mut self & reset_temp_segments_dir * fix: linter * review: rename for simpler public name + use explicit strucuture deconstruction * clipy --------- Co-authored-by: generall <andrey@vasnetsov.com> |
||
|
Ivan Pleshkov
|
30cf43382b |
Deferred threshold integration (#8246)
* Deferred threshold integration * update deferred id * apply update_deferred_internal_id * fix segment inspector * use avaliable bytes count * renamings * review remarks * review remarks * move has_deferred_points * remove todo * update comments |
||
|
dependabot[bot]
|
18a7587d4b |
build(deps): bump rand_distr from 0.5.1 to 0.6.0 (#8148)
* build(deps): bump rand_distr from 0.5.1 to 0.6.0 Bumps [rand_distr](https://github.com/rust-random/rand_distr) from 0.5.1 to 0.6.0. - [Release notes](https://github.com/rust-random/rand_distr/releases) - [Changelog](https://github.com/rust-random/rand_distr/blob/master/CHANGELOG.md) - [Commits](https://github.com/rust-random/rand_distr/compare/0.5.1...0.6.0) --- updated-dependencies: - dependency-name: rand_distr dependency-version: 0.6.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Migrate main code base to rand 0.10 * Migrate tests * Migrate benches --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: timvisee <tim@visee.me> |
||
|
xzfc
|
28d9c5be12 |
Single edge crate (#8173)
* Fixups of amalgamator
Fix issues that break `qdrant-edge` build process:
- `use … as segment;` - this causes `ast-grep` rules to replace wrong
paths. So, rename to avoid collisions.
- `#[macro_use]` and `extern crate` required be in the top-level
`lib.rs`.
- `format!("…", crate::something::…)` - `ast-grep` can't fix paths
inside macros. Fixed by moving `crate::something::…` out of the macro.
* Add lib/edge/publish workspace and amalgamation script
* Move `lib/edge/examples` into `lib/edge/publish/` workspace
And fix them to use the generated `qdrant-edge` crate.
* Add github workflow
* Cleanup `qdrant-edge` public API
Removes empty modules. Checked by `cargo doc`.
|
||
|
xzfc
|
4cabb7fd8e |
Merge io and memory into common (#8155)
* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
|