mirror of
https://github.com/qdrant/qdrant.git
synced 2026-08-05 09:31:05 -05:00
read-batch-generic-error
164 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Andrey Vasnetsov
|
446d140c2d |
Slice filtering condition: sliced scroll / deterministic sampling (#9899)
* feat: slice filtering condition for sliced scroll and deterministic sampling
Add a `slice` filter condition selecting points where
`stable_hash(point_id) % total == index`. The hash is SipHash-2-4 with a
zero key over canonical id bytes (8 LE bytes for numeric ids, 16 RFC 4122
bytes for UUIDs) — a frozen public contract, independent of the internal
resharding ring hash, reproducible by clients to predict membership.
For a fixed `total`, slices are disjoint and cover all points, enabling
parallel scroll streams (ES sliced-scroll style) and reproducible sampling
that composes with any other filter condition.
- REST: `{"slice": {"total": N, "index": R}}`; gRPC: `SliceCondition` in
the condition oneof (tag 8)
- Evaluated per point via id_tracker external-id lookup; no payload index
needed; cardinality estimated as `points / total` with no primary clause
- `total >= 1` enforced by NonZeroU32 at parse time, `index < total` by
validation in both REST and gRPC paths
- Hash contract locked by test vectors independently reproduced with a
reference SipHash-2-4 implementation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* tests: minimal OpenAPI test for slice filter condition
Scrolls all slices of a fixed total over numeric + UUID ids asserting
disjointness and full coverage, checks must_not inversion, and pins the
two rejection paths (422 for index >= total, 400 for total = 0). Requests
and responses are validated against the regenerated OpenAPI spec by the
test harness.
Note: the spec cannot itself reject total = 0 client-side — the Condition
anyOf falls through to the permissive Filter schema, as with any invalid
condition — so rejection is asserted via the server response.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Andrey Vasnetsov
|
4ff6acaf8c |
Edge: request-specific structures for EdgeShardRead (#9901)
Replace the mixed read interface (internal types, explicit parameter enumeration, ad-hoc custom types) with edge-owned request structs, one file per request in src/requests/. Each has a new() constructor taking only the required parameters, a no-macro fluent builder in src/builders/, and a From conversion into the internal request type in requests/conversions/, grouped by request type. Conversions construct and destructure with full field lists, so a parameter added on either side fails compilation instead of being silently dropped. The old reexport aliases (ScrollRequest = ScrollRequestInternal, etc.) are replaced by the edge types under the same names; retrieve() takes a RetrieveRequest instead of a parameter triple. Python bindings wrap the edge types, and the published Rust examples use the builders. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
842f701aae |
refactor: group edge crate internals into edge_shard and read_view modules (#9898)
* refactor: group edge crate internals into edge_shard and read_view modules Restructure lib/edge/src so the top level only contains the modules that form the public crate surface. Implementation files move under the type they implement: - edge_shard/: the EdgeShard struct with its load/config-resolution helpers (previously inlined in lib.rs), plus optimize, shard_read, snapshots, and update - read_view/: the EdgeShardRead/ReadSegmentHandle traits and EdgeReadView (previously read_view.rs), plus the per-operation impl files count, facet, grouping, info, matrix, query, retrieve, scroll, and search; build_search_pool (previously pool.rs) is folded into read_view/mod.rs next to par_map_segments, the seam it powers lib.rs is now a thin facade of module declarations and re-exports. The public API is unchanged: all previously exported names resolve exactly as before, verified against the edge tests, the python bindings, edge-shard-query, and the regenerated publish amalgamation with its examples. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: trim EdgeShardRead surface and split read_view/read_only modules Trait surface: - Drop query_scroll and rescore_with_formula from EdgeShardRead and the EdgeShard inherent wrappers; only the internal query pipeline used them, via the pub(crate) EdgeReadView methods that remain. - Hide the snapshot plumbing (read_segments, search_pool, plus config_snapshot/path providers) in a crate-private ReadViewProvider trait. EdgeShardRead now declares only user-facing methods and is implemented for every provider through a blanket impl, so the plumbing is not callable from user code (verified with a negative compile test; a private supertrait alone leaves supertrait methods callable through generic bounds). An empty sealed marker supertrait keeps the trait unimplementable downstream. - config_snapshot and path stay public: used by edge-shard-query and the python bindings. Module layout: - read_view/: mod.rs keeps EdgeReadView and build_search_pool; ReadSegmentHandle moves to handle.rs, the trait machinery to shard_read.rs, and the nine per-operation impl files into ops/. - read_only/: the follower's ReadViewProvider impl moves out of mod.rs into shard_read.rs, mirroring edge_shard/shard_read.rs. Verified: edge tests, python bindings, edge-shard-query, regenerated publish amalgamation with all examples compiling and running. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Daniel Boros
|
0982e8699c |
feat: segment manifest optimizing state with lease (#9873)
* feat: segment manifest optimizing state with lease * fix: clippy * fix: exhaustive manifest state matching * fix: merge manifest rebuilds under the write lock * refactor: named state predicates, drop redundant enumerator test Review follow-up: move the enumerator's filter into SegmentManifestState::is_usable, name the preserving predicate is_optimizer_mark, delete the enumerator test that re-tested serde. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
8888046cd3 |
Add usage skill file for edge-shard-query tool (#9836)
Standalone usage reference for `edge-shard-query`: backends (S3 / GCS / uio-grpc), connection and tuning flags, the scroll / search / search-sparse sub-commands, filtering, and the live-reload diff mode. Written to stand on its own, so it can be shared as a link without also sharing the source. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
a5a0ecc5e9 |
Add search-sparse sub-command to edge-shard-query tool (#9834)
Sparse nearest-neighbour search over a ReadOnlyEdgeShard, reusing the same SearchRequest path as dense search with VectorInternal::Sparse. The query vector is accepted as a JSON object or an index:value pair list, sorted and validated before use. --vector is required (no random fallback: the sparse vocabulary is not recoverable from the shard config), and --hnsw-ef is omitted since it does not apply to the sparse index. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
1d4d6f02da |
Per-query IDF corpus for sparse vector search (#9661)
* Add per-query IDF corpus for sparse vector search
Let the caller choose, per query, which population sparse IDF statistics
are computed over. `params.idf` is either `"global"` (default, unchanged
behavior) or `{"corpus": <filter>}`, where the corpus filter is
independent of - and usually broader than - the retrieval filter.
Decoupling the two keeps the score scale stable when the retrieval
filter tightens: term importance is measured against a population the
user names, not against whatever subset the filter happens to select.
Design decisions:
- Corpus grammar is restricted to a conjunction (`must`) of `match`
conditions on payload fields; loosening later is backward compatible.
- Strict mode validates the corpus filter like a read filter
(unindexed fields rejected).
- `idf` on a vector without the IDF modifier is a validation error,
never silently ignored.
- An empty corpus yields degenerate but corpus-scoped scores (smoothed
IDF over N=0), never a fallback to global statistics - in multi-tenant
collections a fallback would leak term statistics across tenants.
Implementation:
- QueryContext IDF stats are keyed by corpus, so one batch can mix
requests with different corpora.
- Statistics come from the sparse index: df(term) is counted over the
query terms' posting lists only, never by scanning stored vectors.
Small corpora (under ~1/32 of the segment, by cardinality estimate)
are kept as a sorted id list galloping through posting lists via
skip_to; large ones as a dense membership mask filled streaming from
the filtered-points iterator. A misestimated small corpus degrades
into the mask.
- Exposed uniformly: REST (`params.idf`), gRPC (`IdfParams` message),
edge python bindings; OpenAPI schema regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply rustfmt
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix clippy manual_is_multiple_of in sparse IDF corpus test.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Allow any filter as IDF corpus
Drop the must+match grammar restriction on the corpus filter. A
restriction enforced only as a validation step over the full Filter
type buys nothing; if a narrower corpus syntax is ever wanted, it
should be a dedicated API-level type instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix build: add memory field to SparseIndexConfig in idf corpus test
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
Andrey Vasnetsov
|
43e3d6ea8d |
[UIO] Request-specific load profile for read-only opens (#9797)
* Introduce request-specific LoadProfile with per-component placement A read-only shard opened for one known request (the serverless cold-start path) doesn't have to warm components the request will never touch. LoadProfile captures that from the request: warm components keep the persisted-config placement, everything else is parked cold. All placement decisions live in one place, so the memory placement of a whole segment under a profile is reviewable in one file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Thread LoadProfile through the read-only segment open ReadOnlySegment::open takes an optional profile; first_preopen and open_via resolve it into per-component populate overrides so the opens make the same placement decisions the prefetches did. Pinned components that materialize on open regardless (quantized RAM storage kinds, the immutable-RAM sparse index) and appendable components ignore the override; the HNSW graph and immutable payload indexes demote fully. Config reloads follow the new config alone and pass no override. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Open ReadOnlyEdgeShard under a request-derived load profile ReadOnlyEdgeShard::open takes an optional LoadProfile, applies it to every segment open and keeps it so segments discovered by a later refresh load with the same placement. ScrollRequestInternal and CoreSearchRequest gain load_profile() constructors, and edge-shard-query builds the request before the open and passes its profile (opt out with --no-load-profile). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Demote pinned quantized vectors and sparse index under a cold profile Within the immutable layout the quantized RAM and mmap loaders share the on-disk format — only how the data is brought into memory differs — and the immutable-RAM sparse index has the same lazy mmap open low-memory mode already downgrades to. So a cold populate override now demotes the effective placement itself (Memory::with_populate_override, shared with the HNSW residency mapping) instead of only skipping cache priming: a pinned quantized storage opens the mmap kind cold, and a pinned sparse index opens as Mmap, so neither reads its data on a cold start. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Add LoadProfile::merge for composite queries A composite query runs multiple core requests — e.g. a hybrid search runs one core search per vector, each with its own filter. Its profile is the union of its parts': merge extends the warm sets and ORs the payload-storage flag, so a component either part needs warm stays warm. The union is sound because every placement method is monotone in the warm sets (growing them only turns "park cold" into "keep configured placement"), so the merged profile dominates each input; and minimal, warming nothing no part asked for. Combine profiles with reduce, not fold: merge's identity element is the coldest profile (empty warm sets), the opposite of passing no profile at all — deliberately no empty()/Default constructor exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Adapt vanished-segment test to the profile-aware open signature The test landed on dev (#9777) after the load-profile signature change was written, so the rebase left its ReadOnlySegment::open calls without the new load_profile argument. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Defer vector index open entirely under a cold load profile A cold placement is not enough for the vector index on remote backends: GraphLinksView requires the whole links file as one contiguous slice, and the disk cache can only lend a borrowed slice once every block is locally present — so even a Cold HNSW open mirrors the entire links_compressed.bin (8.2 MB / 1.1 s per segment in the serverless cold-start trace), plus the unconditional graph.bin metadata read. The only way not to fetch the index is not to open it. LoadProfile::vector_index_placement is replaced by vector_index_deferred: a vector the request never scores now gets a DeferredVectorIndex — a new VectorIndexReadEnum variant holding the open arguments (an owned clone of the segment's raw backend, path, config, shared component handles) and a OnceLock. Nothing is opened or prefetched for it at segment open. Per-method policy of the deferred variant: - search, fill_idf_statistics and populate open the index on first use (with the cold placement the profile chose), so the profile contract holds: a request the profile did not predict still works, just pays the open then; - is_index reports true without opening (deferral only ever wraps a real HNSW or sparse index; plain opens no files and is never deferred); - telemetry, indexed_vector_count and sizes answer conservative defaults rather than trigger a remote fetch for a statistic. Tests: deleting the vector_index directory before an open under a scroll profile leaves open, filtered reads and payload reads working — proof that nothing of the index is read — while a search surfaces the missing files; and a segment opened under a scroll profile answers searches identically to an eagerly opened one via the transparent first-use open. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Make --vector optional in edge-shard-query: random query after open Omitting --vector on the search sub-command now searches with a random vector. The request is still built before the shard opens — the load profile only needs the vector name, not its values — with an empty placeholder; once the shard is open, fill_random_vector reads the dimension of the queried vector from the derived shard config and fills in uniform-random f32s (with a clear error if the named vector is not in the config). The vector is generated once, so live-reload iterations re-run the identical random query and the printed diffs stay meaningful. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Scope index deferral to the HNSW graph via a lazy OnceLock load Replace the DeferredVectorIndex wrapper (and the VectorIndexReadEnum:: Deferred variant) with deferral inside ReadOnlyHNSWIndex itself: the graph lives in a OnceLock (same first-wins arbitration as ReadOnlyRoaringFlags::bitmap) alongside the retained raw backend and residency, and loads on first use with a cold placement. The config read stays eager — one tiny, absence-tolerated file — so telemetry, is_on_disk and indexed_vector_count report real values where the Deferred arms answered with hard defaults. The sparse index needs no deferral: its mmap open reads lazily, with only small JSON metadata eager. A profile that never scores the vector now passes a cold placement override (LoadProfile:: vector_index_placement) into the eager open_sparse, which demotes ImmutableRam to the lazy Mmap open like low-memory mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Express HNSW graph deferral as a populate override, not a bool param Replace the `deferred: bool` on the read-only HNSW open/preopen (and the VectorIndexReadEnum pass-through) with the same `populate_override: Option<Populate>` every other component takes. A cold *override* defers the graph load — graph_deferred() mirrors the cold-override match of open_sparse — while a config-derived cold placement (or the low-memory clamp) keeps the eager load, since only a request-specific override carries the "never scored" prediction. With dense and sparse now consuming the same signal, LoadProfile::vector_index_deferred is gone: a single vector_index_placement() serves both index kinds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Serialize the deferred graph load via once_cell's get_or_try_init Loading outside the lock (std OnceLock's fallible init is still unstable) let a search burst on a deferred vector fetch the whole graph once per thread. Swap the cell for once_cell::sync::OnceCell: the fallible load runs inside the cell's lock, concurrent first users block on the one load, and a failed load leaves the cell empty so the next caller retries. Addresses https://github.com/qdrant/qdrant/pull/9797#discussion_r3573727836 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Daniel Boros
|
f982ae63fc |
feat: read-only edge grouping/matrix search + object-storage read path (#9691)
* feat: read-only edge grouping/matrix search + object-storage read path Add query_groups (group by a payload field) and search_matrix (single-shard distance matrix over a random sample) to the read-only edge shard's EdgeShardRead API, in new grouping and matrix modules plus edge test helpers. Make the object-storage read path available outside tests: drop the #[cfg(test)] gate on the BlobFile UniversalReadExt impl and move io_bridge_object_store/object_store to segment's normal dependencies, so a ReadOnlyEdgeShard can serve segments read from S3. * Share group-by building blocks between server and edge Move GroupsAggregator, group candidate query shaping (is-empty filter, group_by payload selector, prefetch limit scaling) and result-order derivation into shard::grouping / shard::query, so the collection and edge grouping implementations cannot silently diverge. Edge grouping now handles multi-valued group keys, u64 keys, wildcard group_by paths, prefetch limits and score-ordered groups the same way as the server. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Drive group-by through a shared sans-IO state machine Extract the multi-request collect/fill loop into shard::grouping::GroupByDriver: next_request() yields shaped backend queries, add_points() advances the state, distill() returns the groups. Query execution stays with the caller, so the async server path and the sync edge path drive the same machine, and the request shaping helpers become private to shard::grouping. Edge now uses the same request budget (5 collect + 5 fill requests) and per-request candidates limit (groups * group_size, computed inside the driver) as the server, replacing its single 4x-oversampled request. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
f82d7584a2 |
Resolve vanished segments against the manifest in follower refresh (#9792)
Third step of the ReadOnlyEdgeShard not-found handling (after #9763 and #9777): shard-level resolution of live-reload failures, per the live-reload design requirements. refresh() is now a bounded retry loop over single-manifest-snapshot attempts. Within an attempt every survivor live-reloads even if one fails (they are independent); failures split by classification: - Not-found: resolved against a re-read manifest. Gone from the manifest means the leader removed the segment mid-reload — drop it and re-run the attempt to pick up its replacements immediately. Still listed means essential files are genuinely missing — escalate. - Anything else: escalate after reloading the other survivors, replacing the previous warn-and-swallow that could hide a corrupted segment forever. Safe: a failed segment keeps serving its pre-refresh state and pending_reload replays the unapplied delta on the next refresh. If attempts run out (leader churning segments continuously) refresh logs a warning and returns Ok: every swap was atomic, so the shard is consistent, just possibly not the newest; the next refresh continues. open_with_enumerator now reuses the refresh machinery: empty holder + refresh() + the open-only merge_follower_config overlay, removing the duplicated load/derive logic. At open there are no survivors, so open behavior is unchanged (#9762 already made it tolerant of unloadable segments via the superset-biased manifest contract). Load-side failures stay as settled by #9762: unloadable listed segments are skipped and retried on every refresh. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
bb0e5d7544 |
Fix flaky disk exhaustion in edge-test CI job (#9807)
The publish/ directory is a separate Cargo workspace, so it does not inherit the root workspace's [profile.dev] debug = "line-tables-only" override. Example binaries were built with full debug info, each statically linking qdrant-edge at ~700 MB per binary. Linking several of them concurrently ran the runner out of disk, surfacing as "mold: failed to write to an output file. Disk full?" + SIGBUS. Mirror the debug info override in the publish workspace and free ~20 GB of unused preinstalled software on Linux runners as headroom. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
8f54076cbe |
Add uio-grpc backend and key-triggered live-reload to edge-shard-query (#9795)
* Add uio-grpc backend and key-triggered live-reload to edge-shard-query --backend uio-grpc opens the shard directly over a running Qdrant peer's StorageRead gRPC service (public gRPC endpoint, api-key aware), addressed by --collection/--shard-id — no object storage involved. The UioGrpcSource backend existed since #9634 but was never wired into the tool's CLI. --bucket is now per-backend optional (required for aws/gcs), and the default cache dir is scoped by collection/shard for uio-grpc, whose mirror has no distinguishing key prefix. --live-reload-key runs the same watch loop as --live-reload with each reload triggered by pressing Enter instead of a timer — easier when stepping through a debug scenario. Timer mode is unchanged; the two flags are mutually exclusive. Closed stdin ends the loop gracefully, so the flag cannot busy-loop on piped input (and is documented as incompatible with @- stdin request arguments). Verified end-to-end against a live instance started with QDRANT__FEATURE_FLAGS__WRITE_SEGMENT_MANIFEST=true: scroll over uio-grpc returns all points with payloads, timer mode picks up an upsert + delete as +/- diff lines, and key mode fires one reload per Enter and exits cleanly on EOF. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Log StorageRead not-found gRPC failures at debug level A read-only follower routinely probes files the writer creates lazily (e.g. the mutable id tracker's mappings and versions before the first flush), so every uio-grpc follower poll spammed INFO logs like: gRPC /qdrant.StorageRead/FileLength failed with NotFound "File not found: .../mutable_id_tracker.versions" NotFound on /qdrant.StorageRead/* now logs at debug; NotFound on all other services (missing collection etc.) stays at info. The service is matched by parsing the path's service component and comparing it to the tonic-generated storage_read_server::SERVICE_NAME constant rather than a hard-coded path string. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
a43abbd3b8 |
Add --live-reload watch mode to edge-shard-query (#9791)
With --live-reload <SECONDS> the tool keeps running after the first answer: every interval it refreshes the ReadOnlyEdgeShard from object storage, re-runs the same request, and prints the difference against the previous results — `+` id appeared, `-` id disappeared, `~ old -> new` for changed content (payload/vector/score/version). Each cycle prints a summary line; pure reordering prints nothing. The request is parsed once into a PreparedRequest (ScrollRequest / SearchRequest are Clone), so every iteration re-runs the identical request and the filter/vector parsing and logging no longer repeat. The first run prints the full result set in the existing format. A failed refresh is logged and retried next tick — the shard keeps serving its previous state, so a transiently unreachable bucket does not kill the watch loop. Status lines go to stderr via log, diff rows to stdout, keeping stdout machine-consumable. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
qdrant-cloud-bot
|
d773266c55 |
Fix edge shard exact count double-counting across segments (#9790)
Collect per-segment point ids into a BTreeSet before counting so points visible in multiple segments (e.g. during proxy/merge optimization) are counted once, matching the canonical collection count path. Fixes #9789 Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Arnaud Gourlay
|
35bbf0487a |
Remove unnecessary clippy allow attributes (#9775)
Remove 8 `#[allow(clippy::...)]` attributes that no longer suppress any lint. Each was verified redundant by rewriting it to `#[expect(...)]` and confirming the workspace stays clippy-clean under the CI config (`cargo clippy --workspace --all-targets --all-features -- -D warnings`). Attribute-only deletions, no behavior change. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Daniel Boros
|
66ce17e709 | feat: skip unloadable segments in read-only follower open (#9762) | ||
|
Andrey Vasnetsov
|
fb681b7d9d |
Lazy roaring flags bitmap and bool index counts (#9749)
* [AI] make ReadOnlyRoaringFlags bitmap and bool index counts lazy
Opening a read-only segment scanned every flags file end to end:
`ReadOnlyRoaringFlags::open` materialized the whole RoaringBitmap via
`iter_ones()`. Every payload field carries a null index, so this was paid
per field per segment, for bitmaps most queries never touch.
Make the bitmap a `OnceLock`, filled by a scan on first access. Open now
reads only the tiny status file. `ReadOnlyBoolIndex`'s three eager count
fields collapse into one lazily-derived, cached `BoolCounts`; its
`live_reload` refreshes them in place when present and leaves them unset
otherwise, so reloading an index nothing queries stays scan-free.
Propagate the resulting `OperationResult` through `RoaringFlagsRead`,
`PayloadFieldIndexRead::count_indexed_points`, `FieldIndexRead`,
`PayloadIndexRead::{indexed_points, get_telemetry_data}`, `build_info` /
`build_telemetry` and `SegmentEntry::{info, get_telemetry_data}`, out
into shard, edge and collection.
`ram_usage_bytes` stays infallible: an unmaterialized bitmap holds no
RAM, so it reports 0 via the new `bitmap_if_materialized`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [AI] correct `preopen` comment: `open` no longer scans the flags file
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [AI] fix edge examples for fallible `info()`
`EdgeShardRead::info` now returns `OperationResult<ShardInfo>`. The
examples live in their own workspace (lib/edge/publish), so the main
`cargo check --workspace` never saw them.
Every call site sits in `fn main() -> Result<(), Box<dyn Error>>`, so
propagate with `?`. `bm25-search` compiled either way but would have
printed the `Result` rather than the `ShardInfo`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Arnaud Gourlay
|
bd35dc1189 |
Replace flush_all sync bool with FlushMode enum (#9750)
* refactor: replace flush_all sync bool with FlushMode enum
SegmentHolder::flush_all took two adjacent bools (sync, force), and call
sites read as bare literal pairs like flush_all(true, false). Swapping
the arguments compiles and silently changes flush semantics: a swapped
pair at the snapshot site would make snapshots skip flushing entirely
when a background flush is running.
Introduce FlushMode { Sync, Background } for the first parameter so the
pair is no longer transposable and the behavior is named at each call
site. The force flag stays a bool since it feeds the
SegmentEntry::flusher(force) trait in lib/segment. No behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: exhaustive match on FlushMode instead of equality check
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Daniel Boros
|
78688733cb | feat: support S3 Express One Zone buckets in the S3 bridge (#9757) | ||
|
Andrey Vasnetsov
|
02cacfe192 |
debug logging of the s3 connection (#9597)
* debug logging of the s3 connection * refactor(io_bridge): make S3 latency logs opt-in and low-overhead Move the blob-backend latency traces onto a dedicated `io_bridge::latency` log target at `trace` level, so they are silent by default and can be toggled as one group at runtime without a rebuild (e.g. `RUST_LOG=io_bridge::latency=trace`). Guard the timing `Instant::now()` behind `log_enabled!` so there is no overhead when the target is disabled. Also add `list_files` timing, and switch the shard_query CLI logger to millisecond timestamp resolution. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
8311ab62a9 |
feat(edge): add --search-threads to shard_query tool (#9736)
Add a `--search-threads` option to the edge-shard-query tool so the number of threads in the shard's search thread pool can be specified. When set, it builds an EdgeConfig with `max_search_threads` and passes it to `ReadOnlyEdgeShard::open`, overriding the CPU-derived default used for both parallel segment reads at open and running searches. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
ab0d3ecc62 |
Add unified memory: cold|cached|pinned placement parameter for collection components (#9684)
* Add unified `memory: cold|cached|pinned` placement parameter for collection components
Introduce a single `memory` parameter that controls how each collection
component's data is held in RAM, replacing the inconsistent zoo of
`on_disk` / `always_ram` / `on_disk_payload` flags:
- `cold`: not pre-loaded from disk, cached with usage
- `cached`: pre-populated into page cache on load, evictable under pressure
- `pinned`: materialized on heap, never evicted by cache pressure
The parameter is available on dense vectors, HNSW config, all quantization
configs, the sparse index, all payload field index types, and payload
storage (as a new `payload: { memory }` sub-object on collection params).
When set, it overrides the deprecated legacy flag; when unset, behavior is
unchanged. Legacy flags are marked deprecated (Rust + proto) but keep
working; conflicts are resolved in favor of `memory` with a warning.
New capabilities enabled by the tri-state model:
- HNSW graph links can be pinned (first production caller of the existing
`GraphLinksResidency::Pinned`)
- sparse mmap index, quantized vectors and on-disk payload field indexes
gain a `cached` tier (mmap + populate on open)
`pinned` is rejected by API validation for components without a heap
variant (dense vector storage, payload storage). Low-memory mode degrades
placements at load time via `Memory::clamp_to_low_memory`, matching the
existing `prefer_disk`/`skip_populate` behavior. Effective-placement
comparison in the config-mismatch optimizer avoids spurious rebuilds when
the same placement is expressed through the new parameter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix gpu-gated tests for the new `memory` field
CI clippy runs with --all-features, which compiles the gpu-gated tests
that were missed locally: add the `memory` field to config literals and
allow deprecated placement params, same as in the rest of the tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add OpenAPI tests for memory placement, keep sparse config downgrade-clean
- OpenAPI tests: create/update collections with `memory` on every component,
assert the parameters are echoed in collection info, assert legacy-only
collections expose no new fields, and assert `pinned` is rejected (422)
for dense vector storage and payload storage on both create and update.
- Persist only the explicitly requested `memory` parameter in
`sparse_index_config.json` instead of the legacy-resolved placement, so
configurations using only the deprecated `on_disk` flag keep byte-identical
files that older Qdrant versions load without unknown fields.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Validate collection meta ops at construction, not only in the API layer
The `memory: pinned` rejection for dense vectors and payload storage
lived in `Validate` impls on the internal request types, which only ran
through the REST actix extractor. gRPC validates just the proto message,
so a gRPC client could persist `pinned` where it is not supported and
have it silently treated as `cached`.
Run the derived validation in `CreateCollectionOperation::new` and
`UpdateCollectionOperation::new` instead: the constructors are the
common chokepoint for all API paths, before the operation is proposed
to consensus. This covers every validator on these types, not just the
`memory` checks, and keeps consensus-apply unaffected so mixed-version
clusters never reject already-committed operations.
`UpdateCollectionOperation::new` becomes fallible; `remove_replica` now
uses `new_empty` since it carries no user config. Regression tests drive
the gRPC conversion path and assert `InvalidArgument` for `pinned` on
create and update, with `cold`/`cached` accepted as a control.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
Andrey Vasnetsov
|
729dc6af43 |
Make EdgeConfig tunables optional with a layered fallback chain (#9714)
Tunable EdgeConfig parameters (on_disk_payload, hnsw_config, optimizers) are now Option, and every tunable resolves through the fallback chain provided -> persisted -> derived from segments -> default when loading an existing shard. Leaving a parameter unspecified keeps the shard as it is; an explicit value overwrites it and existing segments converge to it through the optimizers. vectors/sparse_vectors are excluded from overwrite semantics: an empty map inherits the persisted/segment-derived definitions, a non-empty map is validated for compatibility against the loaded segments (size, distance, multivector, datatype, sparse modifier) and fails the load on mismatch. The derived layer folds over all segments in UUID order instead of taking an arbitrary first segment, so a plain appendable segment (which carries no HNSW parameters) can never mask an indexed segment's actual build parameters. Previously a lost edge_config.json could resolve unspecified HNSW params to compiled-in defaults and silently trigger a full re-index via ConfigMismatchOptimizer. The read-only follower accepts an optional config on open: provided tunables are applied once over the segment-derived config (vectors always come from the segments), and refresh re-derives from segments alone. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Daniel Boros
|
61a1eb80ba |
chore: reexport types (#9696)
* feat: re-export ShardKey and GeoLineString from edge * fix: linter * chore: remove shardkey |
||
|
Andrey Vasnetsov
|
efab63d024 |
Add prefix matching option to keyword index (#9683)
* Add prefix matching option to keyword index
Introduce an opt-in `prefix` option for the keyword payload index and a
new `match: { "prefix": ... }` filter condition, enabling efficient
byte-wise prefix filtering over keyword values (e.g. URL prefixes,
web-ui value autocompletion via facet + prefix filter).
Index side: a new `prefix_index.bin` file stores a sorted, front-coded
key dictionary with a resident block index (cumulative counts per
block); it is an ordered view over the keys of `values_to_points.bin`
and stores no postings. Presence of the file signals prefix support at
load time, so legacy segments load unchanged and enabling the option
goes through the standard incompatible-schema rebuild. The mutable
variant keeps an in-RAM ordered key set (not persisted), the immutable
variant builds a sorted key vector at load, and the on-disk variant
reads the dictionary lazily (block index resident, 1-2 block reads per
prefix lookup; reader is generic over UniversalRead).
Query side: prefix conditions are served from the dictionary when
available (filter + cardinality estimation from per-block aggregates),
from the forward index as per-point checks, and degrade to the payload
full-scan fallback otherwise - same execution model as other match
conditions. Strict mode (`unindexed_filtering_*`) rejects prefix
queries on fields without a prefix-enabled keyword index via a new
KeywordPrefix capability.
HNSW payload blocks: prefix-enabled indexes additionally emit prefix
blocks for heavy branching trie nodes (single-child chains collapsed to
their longest common prefix, one block per distinct point set, emitted
largest-first) so filtered search with prefix conditions gets navigable
subgraphs without rebuilding the same subset repeatedly.
API: `prefix` flag on KeywordIndexParams (REST bool, gRPC empty
message for extensibility), `prefix` variant in the Match oneof, edge
python bindings, regenerated OpenAPI spec.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Split prefix index into a dedicated module, fix clippy in tests
Reorganize the flat prefix_index.rs / prefix_read.rs into a
map_index/prefix_index/ module: format.rs (on-disk layout primitives),
writer.rs, reader.rs (PrefixIndex), map_read.rs (StrMapIndexPrefixRead
with per-variant impls) and tests.rs, with a file-format diagram and a
read-path walkthrough in the module docs. No logic changes.
Also fix clippy --all-targets complaints in test code: replace a
wildcard Match arm with an exhaustive list and a field-reassign-with-
default with a struct literal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add OpenAPI test for prefix match and snapshot file-tracking test
- tests/openapi/test_prefix_match.py: index-less fallback, prefix index
creation with schema echo, scroll/count parity against ground truth,
facet + prefix filter (the autocompletion flow), strict-mode rejection
without the prefix capability.
- test_prefix_index_file_tracking: `prefix_index.bin` is listed in
`files()` / `immutable_files()` exactly when built with the option.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Replace hand-rolled varint parsing with bytemuck Pod records
Per review: the prefix index format now uses fixed-size little-endian
Pod records (BlockEntry 24 B, KeyEntry 12 B, Header 40 B) written with
bytemuck::bytes_of and read back by copy via pod_read_unaligned — no
manual varint encode/decode, no alignment requirement, one shared
read_record helper. Costs ~9 bytes per key on disk versus LEB128; the
raw key bytes dominate dictionary size, so the simplification wins.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fetch the whole candidate block range with a single storage read
Candidate key blocks of a prefix lookup are contiguous in the file, so
enumerate them from one ranged read instead of one read per block; the
over-read versus the exact key range is bounded by the two boundary
blocks. Block decoding is split into a storage-free helper reused by
the per-block path of stats estimation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Align prefix payload blocks with the geo index granularity principle
Geo's large_hashes emits only the smallest geohash regions above the
threshold — a disjoint antichain, never a parent nested with its
children. Prefix payload blocks now follow the same rule: a heavy
collapsed trie node is emitted only if nothing heavy is nested inside
it, counting both deeper qualifying prefixes and single heavy values
(which already get their own exact-match blocks). Emitted blocks are
therefore mutually disjoint and disjoint from exact-value blocks; no
near-collection-sized ancestor subgraphs, no reliance on the HNSW
connectivity check to skip nested duplicates.
Implemented as a `covered` flag propagated through the existing
LCP-interval scan, still one O(total key bytes) pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Document block wire format and unaligned-read rationale in decode_block
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Arnaud Gourlay
|
0346ea66cd |
fix: unblock optimizer after deleting a named vector (#9641)
* fix: unblock optimizer after deleting a named vector Deleting a named vector could permanently block the config-mismatch optimizer. The source-superset check in SegmentBuilder::update cancelled every rebuild that found the deleted vector still in old segment files, and each retry cancelled again, so optimizations got stuck forever. Removing the check (as in #9609) would fix delete but reintroduce data loss for the CreateVectorName race. Instead, tell the two cases apart with the live collection schema: prune a source vector that is gone from the schema (a real deletion), but cancel when it is still present (a freshly created vector this optimizer has not yet seen). This is safe because the schema is persisted before the op reaches segments, and the live schema is read after the source segments are frozen. The live set covers dense and sparse vectors, since a segment stores both together. When no live source is wired in, the conservative always-cancel behavior is kept. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: wire live vector names into edge optimizers Deleting a named vector left the edge path with the pre-fix behavior: segment_optimizer_config hardcoded live_vector_names to None, so a merge touching a segment that still carried the deleted vector cancelled, and EdgeShard::optimize() propagated the cancellation as a hard error forever. Share the shard config behind an Arc and hand the blocking optimizers a provider that reads the current vector names on every call. Same safety argument as the server wiring: update() holds the segments read guard across both the segment application and the config update, so any name a frozen source segment carries is visible to the live read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: share vector-name enumeration via CollectionParams::vector_names The optimizer's live-schema set and the WAL-recovery valid-name set are the same dense+sparse enumeration and must stay in lockstep; a drift between them would reintroduce a wrong prune/cancel decision. Replace the private helper in optimizers_builder and the inline block in WAL recovery with a single CollectionParams method. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: drop SegmentOptimizer::live_vector_names forwarding hop The default trait method only forwarded to the config getter and had a single caller; ShardOptimizationStrategy now reads the config directly, removing one layer of indirection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
5dec49802e |
Split io_bridge core out of io_bridge_object_store; add uio backend (#9634)
* Split io_bridge core out of io_bridge_object_store; add uio backend Separate the backend-agnostic sync<->async bridge from the object-store backends so a second backend can plug in, and add a uio-client backend. - io_bridge (new): the bridge core (AsyncRead, BridgeRuntime, BlobFile, BlobFs, pipeline), with no object_store/tonic deps. BridgeRuntime::block_on is now pub so sibling crates can drive it. - io_bridge_object_store (slimmed): keeps BlobBackend + S3/GCS/Azure. The blanket `AsyncRead for Arc<S>` would now break the orphan rule, so it moves onto a local newtype `ObjectStoreSource<S>`. Re-exports the bridge core. - io_bridge_uio (new): UioSource implements AsyncRead over uio_client::Client, pinned to one (collection, shard_id) replica. Streams gRPC chunks into the aligned buffer; read_from = FileLength + tail stream; writes are Unsupported (read-only service). The client is built lazily on first use inside the bridge runtime, because tonic's connect_lazy captures the reactor at construction and would panic in the runtime-free AsyncRead::open. - uio-client: add sync connect_lazy, read_bytes_stream_raw (BoxStream<Bytes>), NOT_FOUND -> UniversalIoError::NotFound mapping, and an api-key auth interceptor (UioConfig::api_key) injecting the `api-key` gRPC header. - common: add UniversalKind::Uio. - Consumers (segment, edge-shard-query): Arc<AmazonS3> -> ObjectStoreSource<_>. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Rename uio crates/types to uio-grpc Clarify that this "uio" backend is the gRPC transport variant (the "uio" shorthand for Universal IO is used elsewhere in the tree and is left alone). - crate uio-client -> uio-grpc-client (dir lib/uio-client -> lib/uio-grpc-client) - crate io_bridge_uio -> io_bridge_uio_grpc - UniversalKind::Uio -> UniversalKind::UioGrpc - UioConfig/UioSource/UioFile/UioFs -> UioGrpc{Config,Source,File,Fs} - doc/import references updated accordingly Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
qdrant-cloud-bot
|
9977421331 |
perf(edge): load ReadOnlyEdgeShard segments in parallel (#9594)
* perf(edge): load ReadOnlyEdgeShard segments in parallel Open each segment on a dedicated thread during initial open and refresh, reducing follower startup time for shards with many segments. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(edge): resolve clippy type_complexity in parallel segment load Co-authored-by: Cursor <cursoragent@cursor.com> * perf(edge): run per-segment reads on a configurable thread pool Replace the per-segment sequential read loops and the spawn-a-thread-per-segment loader with a single fixed-size rayon thread pool owned by each shard. - Add EdgeConfig::max_search_threads (Option<usize>, None = CPU-derived default matching the core search runtime via common::defaults::search_thread_count). - Build a long-lived pool in EdgeShard and ReadOnlyEdgeShard; reuse it for parallel segment loading on open/refresh instead of std::thread::spawn. - Add EdgeReadView::par_map_segments as the single seam that runs per-segment work on the pool; use it in search, scroll, count, facet and rescore-formula. Each task mints its own HardwareCounterCell from the shared accumulator. - Expose max_search_threads through the builder and the Python binding (+ stub). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(edge): return error instead of panicking on search pool creation ThreadPoolBuilder::build() can fail (e.g. thread spawn / resource exhaustion). This runs during EdgeShard open/load and ReadOnlyEdgeShard follower open, so a transient failure must not abort the process. Propagate it as an OperationError through the existing OperationResult-returning constructors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Andrey Vasnetsov
|
33771fb6c1 |
Generalize edge S3 scroll tool into edge-shard-query (#9598)
Turn the single-purpose `edge-s3-scroll` tool into a sub-command-based `edge-shard-query` that can run different read requests against a ReadOnlyEdgeShard opened over object storage. - Add `scroll` and `search` sub-commands; shared connection/cache args live at the top level. - Accept arbitrary payload filters as JSON via `--filter` (curl `--data` style: literal JSON, `@file`, or `@-` for stdin), parsed straight into the filter DSL. Keep `--filter-key`/`--filter-value` as a shortcut. - `search` accepts a query vector (JSON array, comma list, or `@file`/`@-`), named vector, offset, score threshold, and HNSW params. - Rename the package/binary/dir (`s3_scroll` -> `shard_query`) since the tool is no longer scroll-only nor S3-only. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Luis Cossío
|
a3c313b9a6 |
Add s3_proxy.sh tool to simulate an S3 bucket locally (#9589)
|
||
|
xzfc
|
db2c68e004 |
Replace DynConditionChecker with ConditionCheckerEnum (#9560)
* Add UniversalReadExt Need this trait for the upcoming `ConditionCheckerEnum`. * Replace DynConditionChecker with ConditionCheckerEnum |
||
|
Andrey Vasnetsov
|
f70a1462fb |
feat: read-only edge shard follower (ReadOnlyEdgeShard) (#9529)
* feat: read-only edge shard follower (ReadOnlyEdgeShard) Add a read-only follower of EdgeShard, mirroring the segment-level ReadOnlySegment + live_reload design one level up at the shard. A leader process owns a read-write EdgeShard; one or more followers open the same on-disk directory as a ReadOnlyEdgeShard, serve reads, and periodically refresh() to pick up the leader's flushed writes and optimizations. Shared read logic lives once in a crate-internal EdgeReadView<H>, generic over a ReadSegmentHandle: the follower is monomorphized over the concrete ReadOnlySegment<S> (no dynamic dispatch), while the read-write shard's heterogeneous Segment/ProxySegment holder uses the LockedSegment enum (the only dyn ReadSegmentEntry left, mirroring LockedSegment::get_read). EdgeShardRead is the public read API: it exposes the read methods (search/query/retrieve/count/facet/scroll/info/...) as default methods, requiring implementors only to provide read_segments() + config_snapshot(). EdgeShard keeps its inherent read methods for backwards compatibility. Segment discovery is injected via a SegmentEnumerator (temporary seam): LocalSegmentEnumerator scans the segments/ directory for the local/mmap case; S3 followers supply their own. This will be replaced by an on-disk segment manifest in follow-up work. shard: add LockedSegment::get_read_arc(); split retrieve_blocking into a handle-collecting wrapper + generic retrieve_over; generalize the private _read_points over the segment type (public signatures unchanged); remove the now-unused read_points_locked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor * feat: manifest-based segment loading for ReadOnlyEdgeShard Replace the read-only follower's directory-scan discovery with the segment manifest from the leader, the proper mechanism the SegmentEnumerator seam was a placeholder for. shard: add SegmentsManifest::load so readers can read manifest.json. edge (leader): EdgeShard now writes segments/manifest.json (gated by the write_segment_manifest feature flag), initialized from the live segment set on new/load and refreshed after optimize() swaps segments. edge (follower): ManifestSegmentEnumerator reads the manifest's `active` segments instead of scanning, and open_mmap uses it. It requires a manifest and errors when none is present (no silent scan fallback); discover by scanning explicitly via open() with a LocalSegmentEnumerator instead. Since the manifest only reports ready segments (and future versions will mark segments retiring/under-construction rather than removing them from active immediately), the read path no longer races with the leader, so the defensive `is_transient_open_error` skip handling is removed — a failure to open a reported segment is now a genuine error and propagates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor * feat: edge-s3-scroll experiment binary for ReadOnlyEdgeShard over S3 Adds a standalone `edge-s3-scroll` binary that opens a ReadOnlyEdgeShard directly over an S3 (or S3-compatible) bucket and runs a single scroll request with a hard-coded filter, for experimentation. Takes the bucket endpoint, credentials and key prefix as CLI flags/env vars, builds an S3-backed BlobFs, and discovers segments via a custom enumerator that reads the segment manifest over object storage. The edge_config.json is fetched to a local temp dir because ReadOnlyEdgeShard::open reads config from the local filesystem. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * filter config in s3 scroll tool * fix: read segment manifest from new location after dev rebase dev #9564 moved the segment manifest next to (rather than inside) the segments/ directory and named it segments_manifest.json. Adapt the read-only follower enumerators accordingly so the follower reads the manifest where the leader now writes it: - ManifestSegmentEnumerator reads via segment_manifest_path() (shard root) while still resolving segment dirs under segments/<uuid>. - The S3 scroll tool's enumerator mirrors the same layout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: read-only edge shard works over object storage without edge_config Make ReadOnlyEdgeShard self-sufficient over arbitrary backends (S3/GCS), and harden the read-only segment loading it depends on. - ReadOnlyEdgeShard derives its config from the segments via EdgeConfig::from_segment_config instead of requiring edge_config.json (open and refresh both derive); a follower never has that file. - ReadOnlyEdgeShard::open(fs, path) no longer takes an enumerator: a read-only follower always discovers segments through the manifest. open_with_enumerator remains as a pub(crate) seam for tests. - ManifestSegmentEnumerator is generic over UniversalReadFs (reads the manifest via read_json_via), so it works over local mmap or a blob/S3 backend; the tool's bespoke enumerator is removed. - Read-only mutable ID tracker tolerates absent mappings/versions files (they are not written while empty), matching MutableIdTracker::open: files are opened lazily and NotFound is treated as empty, avoiding an extra exists() round-trip on object storage. edge-s3-scroll experiment tool: - Supports AWS S3 / S3-compatible and GCS backends (--backend), with a hard-coded-free --filter-key/--filter-value scroll filter. - Reads segment data through a DiskCache so remote blocks are fetched once and served from a local mirror afterwards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: remove unused SegmentsManifest::load Its only caller (edge's ManifestSegmentEnumerator) now reads the manifest via read_json_via over UniversalReadFs, so the local-only load helper is dead. Drop it and its now-unused read_json/Path imports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
qdrant-cloud-bot
|
8c487a17e8 |
feat(bm25): explicit Disabled stemmer; deprecate language: "none" hack (#9376)
* feat(bm25): add explicit Disabled stemmer; deprecate language hack
Adds a `Disabled` variant to `StemmingAlgorithm` (`stemmer: {"type": "none"}`)
so stemming can be turned off explicitly in both the main engine and Edge,
instead of relying on the undocumented `language: "none"` footgun that
silently disabled both stemming and stopwords.
For language-neutral text processing the supported setup is now:
1. set the stemmer to disabled, and
2. configure an empty stopword set.
The main engine still tolerates unsupported languages (so existing
`language: "none"` configs keep working on upgrade) but now logs a
deprecation warning pointing users to the explicit setup. Edge continues
to reject unsupported languages, and now has a real way to disable stemming.
Refs: https://github.com/qdrant/qdrant/issues/9289
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(edge-py): handle Disabled stemmer in python bindings; fix openapi schema
- Handle the new StemmingAlgorithm::Disabled variant in the qdrant-edge-py
bindings (FromPyObject/IntoPyObject/Repr) and add a DisabledStemmer pyclass
plus its .pyi stub entry.
- Match generator output for the StemmingAlgorithm OpenAPI schema (plain $ref
in anyOf) so docs/redoc/master/openapi.json stays consistent.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openapi): regenerate StemmingAlgorithm schema with generator output
Ran tools/generate_openapi_models.sh so docs/redoc/master/openapi.json
exactly matches generator output: DisabledStemmerParams/NoStemmer are placed
after SnowballLanguage, and the StemmingAlgorithm anyOf entry is a plain $ref
(the schema2openapi step flattens the allOf+description wrapper).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): avoid wildcard enum match arm in bm25 sparse_len helper
clippy --all-targets flags `other => panic!()` as wildcard_enum_match_arm;
match the Dense/MultiDense variants explicitly instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: issues
* fix: log::warn as call once
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
|
||
|
qdrant-cloud-bot
|
4c169d571c |
build(deps): bump pyo3 from 0.28.3 to 0.29.0 (#9462)
Bumps [pyo3](https://github.com/pyo3/pyo3) from 0.28.3 to 0.29.0. - [Release notes](https://github.com/pyo3/pyo3/releases) - [Changelog](https://github.com/PyO3/pyo3/blob/main/CHANGELOG.md) - [Commits](https://github.com/pyo3/pyo3/compare/v0.28.3...v0.29.0) --- updated-dependencies: - dependency-name: pyo3 dependency-version: 0.29.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Arnaud Gourlay
|
f2b1221669 | Remove unecessary bm25 dep on Edge (#9430) | ||
|
Andrey Vasnetsov
|
ba3472ea49 |
feat(id_tracker): deferred-aware mutable id tracker (#9249)
* feat(debug): QDRANT_APPEND_ONLY_MUTATIONS env override Debug-only escape hatch so newly built segments default to append-only mutation routing when QDRANT_APPEND_ONLY_MUTATIONS=1 (or true/yes) is set in the environment. Lets us run the existing test suites against the append-only path without wiring a collection-level config knob first. Release builds compile this out — the function is a const false. Logs a single warn-level message the first time the override fires. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(append-only): skip deleted vectors on snapshot, always write payload Two correctness fixes to clone_and_mutate_point uncovered by running the openapi suite with QDRANT_APPEND_ONLY_MUTATIONS=1: 1. The snapshot loop read every named vector via get_vector_opt, which returns slot bytes even when the per-vector deletion bit is set. For sparse-only points (or any point that had update_vector(_, None) applied) this materialised the default-zero vector as if it were real data, wrote it at new_id, and never re-tombstoned the slot — so dense search started scoring phantom vectors. Now we check is_deleted_vector(old_id) and skip the read, letting the writer loop emit update_vector(new_id, None) and re-mark the slot deleted. 2. The payload write was skipped when the snapshot ended up empty. That dropped two side effects the field indexes rely on: payload_storage.overwrite(new_id, empty), and the remove_point fan-out across configured field indexes that bumps each index's total_point_count to cover new_id. Without the bump the null index doesn't see new_id, so is_empty / is_null filters lose the point even though its mapping is live in the id tracker. The skip was an optimisation, not a contract; remove it so the field indexes get the same registration they'd get from the standard clear_payload path. Also collapses the debug env override to an inline cfg!()-gated check at the struct literal — the helper with one-time logging and multi-value matching was disproportionate for a debug-only escape hatch. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(id_tracker): split active vs deferred maps in PointMappings (PR A) First step of moving the mutable id tracker to a "two-track" model so append-only mutations into a deferred segment can keep both the visible (active) version and the latest mutated (deferred) version of a point at the same time. This PR is purely structural. On-disk format is unchanged: the loader still produces a single combined map, and `PointMappings::new` partitions it at construction time based on `deferred_internal_id`. Every observable behaviour at the existing API surface is preserved. Concretely: - New fields: `external_to_internal_num_deferred`, `external_to_internal_uuid_deferred`, and a `shadowed: BitVec` for future use by PR B (lazy-grown, default-false). - `internal_id(ext)` checks active first, falls through to deferred — matches the pre-split "any matching id" contract for ext ids whose internal id sat above the cutoff. - `set_link(ext, new_id)` now routes by cutoff: writes below the cutoff land in active, writes at or above land in deferred. Any prior head in the other track is tombstoned, so each ext still owns exactly one slot — same observable result as the pre-split single-map insert. PR B replaces the cross-track tombstone with a shadow-bit flip; PR A keeps current semantics on purpose. - `drop(ext)` clears entries from both tracks and tombstones each one, again matching the prior single-map behaviour. - `iter_external` and `iter_from` merge the active and deferred BTreeMap views into one sorted-by-key stream (dedup'd in case an ext exists in both tracks). - `available_point_count` counts distinct external ids across both tracks — preserves the prior observable count for segments where some entries used to sit above the cutoff in the single map. No write-path or read-path behaviour change. Reads still filter `internal_id >= cutoff` exactly as before; mutations still tombstone prior heads. The shadow bit and the deferred-aware lookup wiring land in PR B and PR C. All existing id_tracker tests pass against the split layout. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(id_tracker): shadow active head on deferred writes (PR B) Step 2 of the deferred-aware mutable id tracker. Replaces PR A's "tombstone the other track" cross-track cleanup with shadow-bit logic so a deferred mutation no longer hides the visible active version. Behaviour by case: - Active write (`internal_id < cutoff`, or no cutoff): unchanged semantically. Any prior deferred head for the same ext is dropped and tombstoned (the new active is the single visible head). Same observable result as PR A. - Deferred write (`internal_id >= cutoff`): the prior deferred head (if any) is dropped and tombstoned. The active head, if it exists, is **shadowed** — its bit is set in `shadowed: BitVec` but its slot stays alive in the active map. Read paths in `Exclude` mode continue to return that active version; PR C will teach `IncludeAll` paths (the optimiser) to skip shadowed actives and prefer the deferred head. Also adds `is_shadowed(internal_id)` and `shadowed_bitslice()` accessors (the latter for PR C's filter pipeline). New unit tests cover the routing matrix: - no cutoff — active replacement, no shadow, - below-cutoff replacement — active path, no shadow, - deferred-on-top-of-active — shadow set, active retained, - two deferred writes — prior deferred tombstoned, shadow persists, - fresh insert above cutoff — no shadow, - `drop(ext)` clears both tracks plus the shadow bit. WAL replay reuses the same `set_link`, so deferred routing on replay falls out of this change with no extra wiring. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(id_tracker): IncludeAll skips shadowed actives, deferred-aware lookups (PR C) Step 3 of the deferred-aware id tracker stack. PR B introduced the shadow bit on `set_link`; this PR teaches the read paths to honour it so the optimiser (and any other `DeferredBehavior::IncludeAll` consumer) sees each external id exactly once, with the deferred head winning over the now-shadowed active. Concretely: - `PointMappings::internal_id_with_behavior(ext, behavior)`: * `Exclude` returns the active head only — `None` for a deferred-only ext, so query paths never see a deferred mutation; * `IncludeAll` prefers the deferred head and falls back to active for points that never crossed the cutoff. Yields at most one internal id per ext. - `IdTrackerRead::internal_id_with_behavior` mirrors the new method with a default impl that delegates to `internal_id` for trackers that don't carry deferred mutations. - `PointMappingsRefEnum::iter_internal_with_behavior(IncludeAll)` now filters shadowed actives via the new `PointMappings::shadowed_bitslice()` accessor. - `PointMappingsRefEnum::filter_deferred_and_deleted(IncludeAll)` also filters shadowed actives — same single-yield-per-external guarantee for external iterator sources like field-index outputs. - `IdTrackerRead::resolve_external_ids` switches to the deferred-aware lookup. No more post-lookup `id >= cutoff` filter — the behaviour enum lookup gets it right at the source. New unit tests cover the two new entry points: - IncludeAll prefers the deferred head when an active is shadowed; - Exclude returns None for deferred-only ext ids; - `filter_deferred_and_deleted` over a mixed candidate list yields the expected per-mode result (Exclude: actives below cutoff; IncludeAll: every visible head, no shadowed actives). No queries observable behaviour change today — production Exclude paths still resolve via `internal_id` and the active head. The optimiser will start using `IncludeAll` (and reap the dedup) in follow-up work. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(id_tracker): expose shadowed_point_count in SegmentInfo (PR D) Final step of the deferred-aware id tracker stack. Adds a single new counter — number of active heads currently shadowed by a deferred mutation — and plumbs it through the id-tracker trait, the segment read-view helper, and `SegmentInfo` for telemetry. Preserves existing semantics: - `available_point_count` is unchanged. Each distinct external id still counts once, regardless of which track holds its head. - `deferred_point_count`, `deferred_internal_id`, `num_deleted_deferred_points` keep their current values. - Non-appendable trackers default `shadowed_point_count()` to `0`, so the new `SegmentInfo.num_shadowed_points` is `None` for them. Concrete changes: - `PointMappings::shadowed_count()` — popcount of the shadowed bitslice. - `IdTrackerRead::shadowed_point_count()` trait method with a `0` default; wired through `MutableIdTracker`, `InMemoryIdTracker`, the mutable read-only tracker, and both enum dispatchers. - `SegmentReadView::shadowed_point_count()` helper. - New `SegmentInfo.num_shadowed_points: Option<usize>`, populated with `Some(_)` for appendable segments and proxied through `ProxySegment` from the wrapped segment's value. New unit test covers the counter lifecycle: - active-only writes don't grow it, - a deferred write over an active adds one shadow, - a second deferred write supersedes the prior deferred head but the shadow stays put (still one active being shadowed), - `drop(ext)` clears the shadow bit, - a fresh deferred insert with no active prior doesn't add a shadow. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(id_tracker): rename DeferredBehavior, push behavior down to PointMappings Three connected pieces of polish on top of the deferred-aware id tracker stack: 1. Rename `DeferredBehavior::Exclude` → `VisibleOnly` and `IncludeAll` → `WithDeferred`. The pre-PR-C semantics of "include/exclude the deferred cutoff" became misleading once `IncludeAll` started skipping shadowed actives — it doesn't "include all" anymore, it yields one slot per external (deferred head preferred, active fallback). New names describe what each variant returns instead of how it relates to the cutoff. The helper method `include_all_points` becomes `with_deferred_points`. Updates ~90 call sites across the workspace; behaviour is unchanged. 2. Push `iter_internal_with_behavior` down from `PointMappingsRefEnum` into `PointMappings`. The per-mode logic (cutoff `take_while`, shadowed `filter`) now lives next to the data it consults; the enum layer becomes a two-arm `Either` dispatcher. `CompressedPointMappings` short-circuits to `iter_internal()` since compressed mappings can't carry deferred mutations. 3. Add a short docstring on `internal_to_external` describing the two-track model: no active-vs-deferred bias, shadowed pairs occupy two slots with the same value, and reads must gate on `deleted` because `set_link`'s same-track replacement leaves a real-looking stale ext id in place. Returning `impl Iterator` instead of `Box<dyn Iterator>` for `iter_internal`, `iter_internal_excluding`, `iter_internal_visible`, and `iter_internal_with_behavior` removes the double-Box at the enum boundary. The original branching structure is preserved with `itertools::Either` instead of restructured into one big filter chain. All existing `id_tracker` tests still pass (47 total). `cargo check --all-targets` + `cargo clippy --all-targets` clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(id_tracker): drop shadowed API surface, unbox iter_from/iter_random Cleanup pass on top of the deferred-aware id tracker stack. API surface: - Drop SegmentInfo.num_shadowed_points + its trait method, read-view helper, tracker impls and ProxySegment passthrough. Cross-segment shadow already exists in the appendable update flow (the source segment keeps its copy when the new write lands above the cutoff, see SegmentHolder::apply_points_with_conditional_move) and is handled at the aggregation layer, not reported per-tracker. - Keep PointMappings::shadowed_count() as a private popcount helper guarded by expect(dead_code) so a future caller can opt back into the dedup'd count without re-plumbing the trait. available_point_count: - Replace the cross-track dedup (active + deferred-only via contains_key filter) with a straight 4-term sum. A shadowed ext contributes two slots, matching the two non-tombstoned internal ids it actually occupies. Old dedup broke the invariant that deleted_point_count == deleted_bitslice.count_ones(): for each shadow it overcounted deletions by one without any tombstone actually being set. DeferredBehavior pushdown: - iter_random_with_behavior: caps the sampling range at the deferred threshold in VisibleOnly mode (no wasted samples above cutoff), filters shadowed actives via the bit in WithDeferred. - iter_from_with_behavior: VisibleOnly walks the active maps only (no merge with deferred); WithDeferred delegates to iter_from's existing merge. - scroll.rs read_by_id_stream / filtered_read_by_id_stream collapse their manual if-deferred-behavior branching into a single iter_from_with_behavior call. - Old iter_random (no behavior) at PointMappings + ref enum was unused after the migration, deleted. Unboxing iter_from / iter_random / iter_from_with_behavior: - PointMappings::iter_from returns impl Iterator + '_ via Either inside the merged_num/merged_uuid closures (BTreeMap::iter vs range), Either at the outer match (num+uuid chain vs uuid-only). - PointMappings::iter_from_with_behavior unboxed with a triple Either (behavior, external-id arm, closure start). - CompressedPointMappings::iter_from unboxed (Either over None/Some). - PointMappingsRefEnum::{iter_from, iter_from_with_behavior, iter_from_visible} all return impl Iterator + 'a via Either on the Plain/Compressed dispatch. Other: - Update outdated PR-A/PR-B comment on PointMappings::drop. - Make the max_internal match in iter_random_with_behavior exhaustive (VisibleOnly+None | WithDeferred+_ instead of `_`). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(id_tracker): unbox iter_external Return impl Iterator from PointMappings::iter_external, CompressedPointMappings::iter_external and the PointMappingsRefEnum wrapper, matching the style of the other iter_* helpers. The wrapper dispatches via Either. The remaining Box::new at Segment::iter_points stays because self_cell's BoxedPointIdIterator alias needs a sized type. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(id_tracker): count set_link tombstones in deferred_deleted_count PointMappings::set_link tombstones the prior slot at two sites — when an active write supersedes a deferred head (cross-track) and when any write replaces a same-track head — using a bare deleted.set(old, true). Neither path updated deferred_deleted_count, so tombstones above the cutoff weren't reflected in the counter. The append-only flow exposes this constantly: every set_full_payload after upsert_point routes through clone_and_mutate_point, which re-issues set_link with a fresh internal id and tombstones the prior one. Most tombstones land above the cutoff, so deferred_point_count (total - cutoff - deferred_deleted_count) over-reports by the missing count. The openapi test_deferred_points integration test caught this as `num_points - num_deferred_points = -1800` across two segments. Extracted the tombstone bookkeeping into PointMappings::tombstone_slot and routed both set_link sites + drop's loop through it. Behaviour on drop is unchanged; set_link now bumps the counter on the live → tombstoned transition for slots at or above the cutoff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style(id_tracker): collapse set_link tombstone if-let chains Clippy's collapsible_if on the two if-let blocks added by the deferred_deleted_count fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update lib/segment/src/id_tracker/point_mappings.rs Co-authored-by: Tim Visée <tim+github@visee.me> * refactor(id_tracker): size shadowed BitVec once instead of growing lazily Collect the shadowed active ids up front and allocate the BitVec to the highest offset in a single resize, avoiding repeated reallocations while marking shadows. Addresses review feedback. Co-authored-by: Cursor <cursoragent@cursor.com> * revert: restore lazily-grown shadowed BitVec Roll back the up-front sizing of the shadowed BitVec; last_entry doesn't fit here and swapping one allocation for another isn't worthwhile. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(id_tracker): prefer deferred head in iter_from merge When an external id has both an active and a deferred head, iter_from's merge collapsed the pair to the active (stale) offset. That contradicts the WithDeferred contract used everywhere else: internal_id_with_behavior and iter_random_with_behavior both surface the deferred head (the latest mutation) over the shadowed active. Consumers that use the returned internal id (payload-filter checks, the optimizer's version merge, HNSW old->new mapping) therefore saw the stale copy. Flip the Both arm to take the deferred operand and document the rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(id_tracker): preserve active+deferred heads when loading mappings read_mappings replayed the persisted change log into a single flat external -> internal map per id-type, then split it by the deferred cutoff in PointMappings::new. A flat map holds one head per external id, so it collapsed the active+deferred coexistence case (an external id linked first to an active slot, then to a deferred one via sequential set_link) down to the last write — silently dropping the other head, orphaning its slot, and leaving the shadowed bit unset. The split in new() could not recover what was already lost before it. Replay the log through the canonical set_link/drop mutators on a PointMappings seeded with the cutoff instead. The log is the sequence of set_link/drop calls that produced the live in-memory state, so this reconstructs that state exactly — both heads, the shadowed bit, and deferred_deleted_count — with no logic duplication. Drops the debug_assert-guarded corruption-recovery branch (subsumed by set_link's re-link handling) and the now-unused Uuid/PointIdType imports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(id_tracker): make deferred behavior explicit at point resolution Replace the ambiguous "any matching head" point resolution with an explicit DeferredBehavior at every resolution boundary, so callers no longer rely on a hidden active-vs-deferred policy. - Remove PointMappings::internal_id and the bare IdTrackerRead::internal_id (the active-first-else-deferred hybrid). internal_id_with_behavior is now the single required resolution method; immutable/compressed trackers implement it by ignoring the behavior (they never carry deferred heads). - Migrate every caller to an explicit behavior, audit-driven: - writes (upsert/delete/payload/vectors), point_version, point_is_deferred, get_internal_id, drop, consistency + builder dedup -> WithDeferred (the latest/live head); - single-point payload/vector retrieval and formula rescore -> VisibleOnly; - HasId/CustomIdChecker/cardinality resolution -> the request's behavior, threaded through the filter chain from iter_filtered_points (other entry points default to VisibleOnly). - lookup_internal_id takes an explicit DeferredBehavior instead of assuming VisibleOnly internally. - has_point takes an explicit DeferredBehavior (drop the has_point_with_behavior wrapper). Thread it through read_points/_read_points/read_points_locked so retrieve_blocking passes its request behavior to the existence filter; all other existence/dedup callers pass WithDeferred (unchanged behavior). - set_link now detaches a stale live occupant of a reused internal id, keeping the forward and reverse maps consistent when recovering a corrupted log. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(id_tracker): failing tests for two correctness findings in #9249 (#9311) * test(id_tracker): failing tests for two review findings Two intentionally-failing tests pinning correctness gaps in the deferred-aware id tracker (#9249). Both fail on the final assertion; the earlier assertions establish the expected/consistent behavior. 1. iter_from_with_behavior(WithDeferred) resolves an active/deferred shadow collision to the stale ACTIVE internal id, while its siblings internal_id_with_behavior and iter_internal_with_behavior correctly surface the DEFERRED (latest) head. Consumers that use the yielded internal id (optimizer merge via for_each_unique_point, filtered_read_by_id_stream) therefore observe the pre-mutation version. left: [(NumId(7), 2)] right: [(NumId(7), 9)] 2. The PR-B shadow/visible invariant is not durable: the on-disk single combined map cannot represent a shadowed ext, so a plain mappings flush + reload collapses the shadow to deferred-only and the visible (active) head is lost (VisibleOnly resolves None where it resolved Some(2) live). Restoration then depends entirely on WAL replay, i.e. on flush-vs-WAL-truncate ordering. left: None right: Some(2) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(id_tracker): strengthen shadow tests + merge-primitive proof Follow-up to the two failing tests, addressing self-review: - Add for_each_unique_point_keeps_deferred_head_for_shadowed_point: the optimizer merge primitive (used by segment_builder::update_from) yields the stale active copy (internal 2, version 5) for a shadowed point and drops the deferred latest (internal 9, version 8). This directly exercises the data-loss consequence of finding #1 at the merge layer. left: [(NumId(7), 2, 5)] right: [(NumId(7), 9, 8)] - Tighten shadow_visible_head_survives_mapping_flush_reload: pin the exact reload failure mode. After flush+reload the mapping collapses to deferred-only (internal_id == Some(9)) and the active slot survives as a live orphan in the inverse map (external_id(2) == Some(7), not deleted) — a torn state where a VisibleOnly scroll still surfaces the stale copy while by-id VisibleOnly resolution breaks. Reframe as the live-vs-reload divergence the PR introduces (Some(2) live -> None reload; dev is consistently None). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix test --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: generall <andrey@vasnetsov.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Tim Visée <tim+github@visee.me> Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com> |
||
|
Tarun Jain
|
7caf415958 |
[qdrant-edge] Hybrid Search with Dbsf fusion (#9356)
* [edge]dbsf fusion example along with hybrid search * Reduce query limits in hybrid search example |
||
|
xzfc
|
99abba9556 |
edge: use cargo metadata instead of parsing Cargo.toml (#9287)
* edge: refine ast-grep rules * edge: use `cargo metadata` instead of parsing Cargo.toml |
||
|
Tim Visée
|
7041b81f76 |
Use assert_matches! (#9231)
* Use assert_matches! * Add trailing commas * Use more assert_matches! Also, drop now redundant `expected blah but got blah` messages because `assert_matches!` will print these. * Use debug_assert_matches! --------- Co-authored-by: xzfc <xzfcpw@gmail.com> |
||
|
Arnaud Gourlay
|
4f1f9707ee | Remove unused dependencies (#9262) | ||
|
qdrant-cloud-bot
|
07f94e1fae |
Bump edge packages (Python + Rust) to 0.7.2 (#9252)
Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
qdrant-cloud-bot
|
65b7f52d18 |
Add Rust edge example: add-named-vector (#9230)
* Add Rust edge example: add-named-vector Port `lib/edge/python/examples/add-named-vector.py` to Rust under `lib/edge/publish/examples/src/bin/add-named-vector.rs`. Re-export `VectorNameOperations`, `CreateVectorName`, `DeleteVectorName`, `VectorNameConfig`, `DenseVectorConfig`, and `SparseVectorConfig` from `qdrant_edge` so the public Rust API can create/delete named vectors without reaching into the internal `shard` crate. Co-authored-by: Cursor <cursoragent@cursor.com> * fmt Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Jojii
|
7a8703f166 |
[TQDT] API (#9172)
* [ai] TQDT in the API * [ai] unify new sparse error for TQDT * Rename to `turbo4` * Add `Turbo4` to comments and doc strings. * Also validate named sparse vector creation |
||
|
qdrant-cloud-bot
|
b84f8c84c6 |
Bump edge packages (Python + Rust) to 0.7.1 (#9228)
Co-authored-by: Cursor Agent <agent@cursor.com> |
||
|
qdrant-cloud-bot
|
6d8f3c6ce9 |
ci(windows): skip IO-heavy tests that aren't OS-specific (#9188)
* ci(windows): skip IO-heavy tests that aren't OS-specific On the Windows CI runner, several tests are 3-25x slower than on Ubuntu purely due to slow filesystem IO. These tests exercise platform-agnostic logic (optimizer, snapshot, WAL recovery, dedup, deferred points) and are fully covered by the Linux and macOS jobs. Mark them with `#[cfg_attr(target_os = "windows", ignore = "...")]` so: - Windows CI skips them and finishes faster. - They're still listed and runnable via `cargo test -- --ignored` on Windows for local debugging. Based on JUnit timings from CI run 26462785436 (PR #8827), this should save ~5 minutes wall-clock on the Windows job, taking it closer to the ~13min Ubuntu and ~9min macOS jobs (currently 20m24s). Tests affected: - lib/wal: check_wal, check_last_index, check_clear, check_reopen, check_truncate, check_prefix_truncate, test_prefix_truncate_parametric - lib/edge/optimize: full tests module - lib/segment deferred-point tests: read_operations, dense_segment_combinations, sparse, facets - lib/collection: snapshot_test, points_dedup, wal_recovery, collection_test::test_ordered_read_api, snapshot_recovery_test Co-authored-by: Cursor <cursoragent@cursor.com> * revert(ci/windows): keep WAL and WAL-recovery tests on Windows Reviewer correctly pointed out that WAL is mmap-backed and has substantial Windows-specific code paths: - Different segment allocation (fs4 vs rustix::ftruncate) - Windows-specific delete_windows() with mmap-drop + retry loop - Windows-specific sync_all() because directory fsync is unavailable - Windows-specific lock proxy file (directories aren't lockable) So those tests genuinely need Windows coverage. Reverted skips for: - lib/wal/src/lib.rs: all check_* tests and test_prefix_truncate_parametric - lib/collection/src/tests/wal_recovery_test.rs: all three tests Still skipped on Windows (no OS-specific code in their production paths): - lib/edge/optimize.rs (no cfg(windows) in source) - lib/segment deferred-point tests (segment/ has no cfg(windows)) - lib/collection snapshot/dedup tests (collection/ has no cfg(windows)) - lib/collection integration snapshot_recovery + ordered_read_api Co-authored-by: Cursor <cursoragent@cursor.com> * revert(ci/windows): keep collection integration and snapshot_test Per reviewer request, keep running these on Windows: - lib/collection/tests/integration/* (snapshot_recovery_test, collection_test::test_ordered_read_api) - lib/collection/src/tests/snapshot_test.rs These exercise higher-level collection/snapshot behavior that benefits from cross-platform validation. Remaining Windows skips (production code has no cfg(windows) branches): - lib/edge/src/optimize.rs: 14 optimizer tests - lib/segment/src/segment/tests/mod.rs: 4 deferred-point tests - lib/collection/src/tests/points_dedup.rs: 2 dedup tests Co-authored-by: Cursor <cursoragent@cursor.com> * ci(windows): also skip HNSW/quantization integration tests Per reviewer, also skip these segment integration test modules on Windows: - hnsw_quantized_search_test::* (25 tests) - multivector_filtrable_hnsw_test::* (rstest cases) - multivector_quantization_test::* (rstest cases) - byte_storage_quantization_test::* (rstest cases) - payload_index_test::test_struct_payload_index_nested_fields These exercise pure HNSW/quantization correctness on top of standard segment IO that is already covered by tests we keep running on Windows. Adds ~930s of sequential time to the Windows skip list, bringing the expected wall-clock saving from ~3 min to ~8-10 min. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
qdrant-cloud-bot
|
4829bbd255 |
Bump edge packages (Python + Rust) to 0.7.0 (#9187)
Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Daniel Boros
|
c4f70cc445 |
feat/edge-bm25 (#8827)
* feat: integrate bm25 into edge * fix: linter * fix: imports * fix: qdrant-edge build errors * fix: spell check * feat: integrate inference * fix: api missmatch * chore: remove inference * fix: coderabbit comments * fix: compiler error * chore: remove wrapper type * feat: add some temp tests for legacy check * add example --------- Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> |
||
|
Sasha Denisov
|
797779a159 |
feat(edge): expose WAL options via EdgeShard::load_with_wal_options (#9067)
* feat(edge): expose WAL options via EdgeShard::load_with_wal_options Adds a sibling method to EdgeShard::load that accepts custom WalOptions, alongside the existing load() which keeps using default_wal_options(). Motivation: embedded/mobile deployments (e.g. Flutter plugins on iOS/Android) need much smaller WAL segments than the 32 MiB default — typically 4 MiB. On filesystems with sparse files (APFS/ext4/F2FS) the physical footprint is small, but the visible/reported size is the full segment capacity, which is surfaced by iCloud backup, OS size pickers, etc. Changes: * New EdgeShard::load_with_wal_options(path, config, wal_options). * EdgeShard::load delegates to it with default_wal_options() — no behavior change for existing callers. * ensure_dirs_and_open_wal now takes WalOptions explicitly; called from both EdgeShard::new (with default) and load_with_wal_options. * WalOptions re-exported from edge crate root. * Two regression tests in lib/edge/tests/wal_options.rs. WAL options are intentionally a runtime parameter, not part of EdgeConfig (which is persisted to edge_config.json) — different processes may legitimately open the same shard with different WAL options. * test(edge): verify mismatching WAL options on reload preserve data Addresses upstream review feedback (qdrant/qdrant#9067): demonstrate that reloading an existing shard with WAL options different from the ones it was created with is safe and preserves all previously written points. Two new tests in lib/edge/tests/wal_options.rs: * reload_with_smaller_wal_capacity_after_upsert: create shard with default 32 MiB WAL -> upsert point 42 -> drop -> reload with 4 MiB WAL options -> point 42 still readable -> upsert point 43 under smaller WAL -> count == 2. * reload_with_larger_wal_capacity_after_upsert (symmetric): create -> reload with 4 MiB -> upsert point 100 -> drop -> reload with default 32 MiB -> point 100 readable -> upsert point 101 -> count == 2. Both pass. The WAL crate does not validate segment_capacity on reload — existing segments on disk keep their original size, subsequent appends honor the runtime options. This confirms the PR description's claim that WalOptions is a runtime hint, not persisted shard state. * style(edge): cargo fmt for wal_options.rs mismatch tests * refactor(edge): replace load_with_wal_options with builder-based options Addresses upstream feedback (timvisee, generall): the sibling constructor pattern doesn't scale as runtime configurability grows. Replaces `EdgeShard::load_with_wal_options(path, config, wal_options)` with `EdgeShard::load_with_options(path, config, EdgeShardOptions)`, where `EdgeShardOptions` is a builder-style runtime-options struct. * `EdgeShardOptions::new().with_wal_options(opts)` is the equivalent of the old call. * Adding new runtime options later (e.g. wal flush interval, initial indexing threshold) is now an additive change — new builder method on `EdgeShardOptions`, no new constructor variant on `EdgeShard`. * `EdgeShardOptions` is intentionally not persisted (no Serialize/Deserialize). The reload-mismatch tests above confirm that WAL options are a runtime hint, not shard identity, so they don't belong in `EdgeConfig` (which is persisted to `edge_config.json`). * `EdgeShard::load(path, config)` unchanged for existing callers — delegates to `load_with_options(..., EdgeShardOptions::default())`. Tests in `lib/edge/tests/wal_options.rs` migrated to the new shape; all four still pass: test load_with_options_accepts_custom_wal_capacity ... ok test load_still_works_with_default_wal_options ... ok test reload_with_smaller_wal_capacity_after_upsert ... ok test reload_with_larger_wal_capacity_after_upsert ... ok * refactor(edge): add fluent builders; move wal_options into EdgeConfig Replaces the EdgeShardOptions side-channel with a single EdgeConfig that carries wal_options inline, and introduces fluent builders for the three user-facing config types. * WalOptions now derives Clone/PartialEq/Eq/Serialize/Deserialize so it can live inside EdgeConfig and round-trip through edge_config.json. * EdgeConfig.wal_options (Option<WalOptions>) replaces EdgeShardOptions. EdgeShard::load_with_options is folded into EdgeShard::load; both new and load drive the WAL from config.wal_options.unwrap_or_default(). * New builders/ module hosts EdgeConfigBuilder, EdgeVectorParamsBuilder, and EdgeSparseVectorParamsBuilder. Each builder has explicit per-field storage and constructs its target via an exhaustive struct literal in build(), so adding a field to the target forces a compile error in the builder. * publish example switched to the new builder API. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style(edge): cargo fmt after builder refactor Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
qdrant-cloud-bot
|
26aeb9c0b2 |
docs: refresh prevent_unoptimized description (#9133)
The previous description was outdated: it claimed that enabling this option "blocks updates at the request level" until segments are re-optimized. In practice the implementation uses "deferred points": new points written to large unoptimized segments are persisted but excluded from read/search results until the segments are optimized. Updates are not blocked; only `wait=true` clients are made to wait for the deferred points to become visible. Update this in the REST schema (via `OptimizersConfig` / `OptimizersConfigDiff`), in the gRPC proto, in the edge config docstrings, and regenerate the OpenAPI bundle via `tools/generate_openapi_models.sh`. Co-authored-by: Cursor Agent <agent@cursor.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
xzfc
|
faf2714f4b | Warn on clippy::wildcard_enum_match_arm (#9096) |