* Fix MMR pagination with offsets
(cherry picked from commit 6a778bbde8)
* Clamp MMR selection capacity at the candidate count
Move the guard into `maximal_marginal_relevance`, which every MMR caller
routes through, so the collection, edge and local-shard paths are covered
instead of just the collection one. The collection-level limit is then
plainly `limit + offset`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit a63d744131)
---------
Co-authored-by: mikemikimike <13286568797@163.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Remove MessagePack (rmp-serde)
The WAL switched from msgpack to CBOR in v0.3.5 (2021-07-11), so v0.3.4
is the last version that wrote msgpack entries. Drop the read fallback
kept for those entries, plus the remaining test and bench usages.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Drop unused fs4 dependency from collection
Not referenced anywhere in the crate. Still used by wal and common, so
the workspace entry stays.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Direct call for shard transfer method and keys
* Reuse cardinality estimate in sparse plain search
* Avoid recounting available points in segment size info
* Avoid cloning segment config when updating quantization
* Avoid cloning search request for load profile
* Direct call for counting read-only segments
* Avoid re-reading point range for values count
* Direct call to check replica states when initializing collection
* Direct call to look up transfer on restart
* Direct call for shard replicas after snapshot recovery
* Direct call for local replica states in health check
* Direct call for payload index schema keys when applying state
* Direct calls for sharding method and key mapping when creating shard key
* Direct call to check if peer has shards
* Direct call for sharding method and keys when dropping shard key
* Avoid cloning collection params for group by ordering
* Avoid cloning collection params in local shard search
* Direct call for peer address when sending Raft messages
* Direct call for peer address in who_is
* Avoid cloning remote query batch request
* Avoid cloning operation in queue proxy update
* Avoid cloning gRPC search groups request
* Fetch cluster status once in cluster telemetry
* Direct call to validate transfer exists on finish
* Direct call for sharding method when dropping shard key
* Avoid cloning peer address map when listing peers
* Avoid cloning peer address map when adding peer to known
* Avoid cloning shard key mapping when routing writes with fallback
* Avoid cloning shard key mapping when checking resharding start
* Avoid cloning gRPC recommend groups request
* Avoid cloning operation when retaining forwarded point IDs
* Direct call for counting collections in telemetry
* Direct call to validate transfer exists on recovery
* Direct call for shard IDs by shard key
* Direct call for shard keys
* Direct call to check if peer has shards in consensus
* Direct call for replica state on transfer recovery
* Direct call to check for active replicas when routing writes with fallback
* Direct call to validate transfer exists on abort
* perf: skip retrieval in scroll when no payload or vectors are requested
Every scroll variant went through SegmentsSearcher::retrieve to build its
records, even when neither payload nor vectors were asked for. That is a
has_point lookup per id per segment plus a version and id resolution per
hit, only to yield records holding nothing but the id. The universal query
API always scrolls this way and fetches payload separately afterwards.
Build the bare records from the ids directly in that case. The retrieve
could only have dropped ids deleted in between, which the update lock held
across the scroll rules out.
* perf: fetch payload and vectors in the leaf of plain query requests
A query without prefetches and without rescoring is served by a single
leaf search or scroll whose result is returned as is. The planner still
built that leaf without payload or vectors and filled them in afterwards
through SegmentsSearcher::retrieve, which resolves every result id in
every segment again: the same cost #10312 removed from the search API,
paid once more at the end of each query.
Let the leaf carry the requested payload and vectors instead, so the
segment attaches them to the results it already holds by offset, and
clear the root plan so the fill step is skipped. Prefetch leaves and
rescored roots (MMR) are unchanged. As with the search API, this fetches
payload for each segment's candidates rather than for the merged top
`limit` alone.
* Use new_empty function
* fix: fetch payload and vectors in scroll leaves only (#10384)
A search leaf hydrates every segment's local top-k before merging, so
`with_payload` there multiplies payload I/O by the segment count — the
regression #6279 fixed and `test_payload_io_read_is_within_limit[query]`
guards. Scroll leaves retrieve once for the merged page, so they keep
fetching directly; search leaves stay bare and the root plan retrieves
for the final result.
Claude-Session: https://claude.ai/code/session_01SUWh5PqUeSefrUqUwXxU3E
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* test: parameterize collection fixture and share resharding consensus stub
Let integration tests build a collection with a custom optimizers
config, and move NoopReshardingConsensus from the consensus idempotency
test into the shared test module.
* test: reproduce scale-down resharding abort hanging on deferred points
abort_resharding holds the shard holder write lock while
scale_down_cleanup_points deletes migrated points with
WaitUntil::Visible and no timeout. With prevent_unoptimized enabled that
wait only resolves once the optimizer has cleared every deferred point
of the shard, so a stalled optimizer wedges the write lock, and with it
every shard holder reader and - in a cluster - the consensus apply
thread driving the abort (SetShardReplicaState(Dead) on a
ReshardingScaleDown replica, e.g. after that peer is killed).
The test drives such an abort with the optimizer disabled and asserts it
completes. It currently fails by hanging into its 30s timeout, and must
pass once the cleanup delete no longer waits for visibility.
* When cleaning up old points, don't wait until visible
[audit-K] Make create_shard_key crash-safe with a single commit point
create_shard_key persisted shard_key_mapping.json incrementally, one add_shard
per placement entry, so the operation was not atomic across a crash. The
re-apply gate state.shards_key_mapping.contains_key(&shard_key) becomes true
after the first loop iteration, so a crash mid-loop on a multi-shard placement
left the peer permanently holding a subset of the key's shards while every other
peer had all of them.
Write the mapping exactly once instead, after every shard of the key exists on
disk, so there is no partial state to observe. SaveOnDisk writes atomically,
load_shards derives the shard id list from the mapping alone (unreferenced
directories are invisible after restart), create_shard_dir wipes leftovers, and
max_shard_id reads only the mapping - so a replay allocates exactly the ids the
crashed attempt did, which are the ids every other peer allocated too. The
contains_key gate then holds as intended: it fires only for an operation that
already completed in full, or for a genuine duplicate.
ShardHolder::add_shards registers a batch and persists the mapping once, with
add_shard as a one-element wrapper; the mapping write moved ahead of the
in-memory updates, being the only fallible step. No caller changes behavior:
Collection::new and load_shards already hit the write_optional early return, and
start_resharding_unchecked still adds a single id. An empty placement is now
rejected rather than silently returning Ok without creating the key.
Covered by create_shard_key_test.rs: id allocation, replay before and after the
commit, duplicate rejection, and the empty placement guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Add per-query IDF corpus for sparse vector search
Let the caller choose, per query, which population sparse IDF statistics
are computed over. `params.idf` is either `"global"` (default, unchanged
behavior) or `{"corpus": <filter>}`, where the corpus filter is
independent of - and usually broader than - the retrieval filter.
Decoupling the two keeps the score scale stable when the retrieval
filter tightens: term importance is measured against a population the
user names, not against whatever subset the filter happens to select.
Design decisions:
- Corpus grammar is restricted to a conjunction (`must`) of `match`
conditions on payload fields; loosening later is backward compatible.
- Strict mode validates the corpus filter like a read filter
(unindexed fields rejected).
- `idf` on a vector without the IDF modifier is a validation error,
never silently ignored.
- An empty corpus yields degenerate but corpus-scoped scores (smoothed
IDF over N=0), never a fallback to global statistics - in multi-tenant
collections a fallback would leak term statistics across tenants.
Implementation:
- QueryContext IDF stats are keyed by corpus, so one batch can mix
requests with different corpora.
- Statistics come from the sparse index: df(term) is counted over the
query terms' posting lists only, never by scanning stored vectors.
Small corpora (under ~1/32 of the segment, by cardinality estimate)
are kept as a sorted id list galloping through posting lists via
skip_to; large ones as a dense membership mask filled streaming from
the filtered-points iterator. A misestimated small corpus degrades
into the mask.
- Exposed uniformly: REST (`params.idf`), gRPC (`IdfParams` message),
edge python bindings; OpenAPI schema regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply rustfmt
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix clippy manual_is_multiple_of in sparse IDF corpus test.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Allow any filter as IDF corpus
Drop the must+match grammar restriction on the corpus filter. A
restriction enforced only as a validation step over the full Filter
type buys nothing; if a narrower corpus syntax is ever wanted, it
should be a dedicated API-level type instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix build: add memory field to SparseIndexConfig in idf corpus test
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* [AI] fix: bound random-sample pre-allocation by available point count
`LocalShard::scroll_randomly` pre-allocated its result set with
`HashSet::with_capacity(limit)`, where `limit` is the client-supplied request
limit and is unbounded by default (`StrictModeConfig::max_query_limit` is `None`
unless an operator sets it). A very large limit made the up-front allocation
fail and abort the process (`handle_alloc_error`), which cannot be caught and
returned as an error.
The sample loop inserts at most `min(limit, total points available across
segments)` points, so bound the pre-allocation to what is actually available.
Capacity is unchanged for normal requests; only the pathological case is bounded.
Same class as the search top-k fix (#4321 / #4328) and the in-flight edge fix
(#9374). Adds a regression test.
* [AI] fix: cap random-sample preallocation by filtered candidate count
Review feedback. `available_point_count_without_deferred()` is filter-unaware,
so `availability.iter().sum()` could still drive `HashSet::with_capacity()` to
reserve a buffer proportional to the full collection on a highly selective
filtered request, re-opening the allocator-abort path on large collections.
The sampling loop only ever inserts points read into `segments_reads`, which is
filter-aware and already capped at `limit` per segment, so bound the
preallocation by that candidate count instead.
* Update lib/collection/src/shards/local_shard/scroll.rs
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
---------
Co-authored-by: Andrey Vasnetsov <vasnetsov93@gmail.com>
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
* Add routing token structure
* Implement routing token in read operation executor as per design doc
* Add TODO to glue routing token to user requests
* Implement routing header for REST API
* Source routing token from request, not from JWT token
* Implement routing token in gRPC API
* Add test
* Review remarks
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Use lower case header name to prevent panic
* Rename header to X-Qdrant-Route-Affinity
* Assert routing consistency in test on all peers
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Use assert_matches!
* Add trailing commas
* Use more assert_matches!
Also, drop now redundant `expected blah but got blah` messages because
`assert_matches!` will print these.
* Use debug_assert_matches!
---------
Co-authored-by: xzfc <xzfcpw@gmail.com>
* [ai] Replace manual into mappings with Into::into
* Reformat
* [ai] Use implicit .iter
* Don't iterate over keys too
* [ai] Replace unwrap_or
* Reformat
* [ai] Use as_deref and then_some
* [ai] Use more to_string
* [ai] Use explicitly typed into conversions
* Reformat
* [ai] More explicit into conversions
* Reformat
Reduce test parameters across 5 areas to cut test runtime without
sacrificing coverage:
1. HNSW PQ tests: lower vector dimensionality from 131 to 64 for product
quantization variants, cutting PQ codebook training time (~31s -> ~8s)
2. HNSW index build: use 2 threads instead of 1 for index construction;
the tests only check accuracy above a threshold so determinism is not
required
3. Search attempts: reduce from 10 to 5 query vectors per test
4. Rescoring: skip the expensive re-upsert-all-as-zeros rescoring check
for PQ tests (already covered by scalar quantization variants)
5. Near-miss speedups:
- WAL: reduce QuickCheck iterations from 100 to 50
- Gridstore: halve operation counts, drop 64-byte block size case,
reduce proptest cases for gap search
- Continuous snapshot: reduce timeout from 20s to 10s
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* Move `DenseVector`/`MultiDenseVector` from `api` to `segment`
* Move `OrderByInterface` from `api` to `segment`
Reason: it's used in `edge` which shouldn't depend on `api`.
* Make `shard` -> `api` dependency optional
* Remove `api` from the amalgamation
* Don't install protoc in edge Actions
* [manual] generate snapshot download link with respect of enabled tls
* Simplify http(s) selection
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* download tar
* compute sha256 for stream download
* wip: propagate unpacking into down to the logic, todo: validation
* unpacked snapshot validation
* Minor tweaks
* Fix typo
* validation during unpack
* cancellation token
* update docstring
* remove redundant dep
* Rearrange unpack functions
- Rename `safe_unpack.rs` into `tar_unpack.rs` so it would be listed
near `tar_ext.rs` in IDEs.
- Replace calls like `ar = open_snapshot_archive(…); safe_unpack(ar, …);`
with a single call to `tar_unpack_file(…)`.
- Put calls to `Archive::new(); Archive::set_overwrite(false);` inside
`tar_unpack_reader` (was `safe_unpack`). So, now it is the only place
that does `set_overwrite`.
* Let clippy complain if tar::Archive::unpack used
* Mock snapshot download URL
Instead of downloading from storage.googleapis.com every time the test
runs, put small snapshot file to the repo.
The snapshot file is created using this command:
curl -s \
https://storage.googleapis.com/qdrant-benchmark-snapshots/test-shard.snapshot \
| tar \
--delete segments/4ea958d8-0b64-4312-9a53-0cd857e93535.tar \
--delete segments/65ac6276-8cca-4f5c-b767-9722190cee8b.tar \
> lib/storage/src/content_manager/snapshots/test-shard.snapshot
File contents:
$ tar tf lib/storage/src/content_manager/snapshots/test-shard.snapshot
wal/
wal/closed-255
newest_clocks.json
replica_state.json
shard_config.json
$ du -sh lib/storage/src/content_manager/snapshots/test-shard.snapshot
12K lib/storage/src/content_manager/snapshots/test-shard.snapshot
$ sha256sum < lib/storage/src/content_manager/snapshots/test-shard.snapshot
5d94eac5c1ede3994a28bc406120046c37370d5d45b489a0d2252531b4e3e1f2 -
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* sync function to ask to stop workers
fmt
update blocking_ask_workers_to_stop
* graceful shard stop on ShardReplicaSet changes
* make stop consuming
* implement graceful stop for replica set
* graceful shard stop in start_resharding_unchecked
* stop_gracefully fn on shard_holder
* collection graceful stop on delete
* explain when it is ok to shutdown
* fix tests and ensure blocking callsa are not called from async runtime
* fmt
* Annotate cancel safety for do_recover_from_snapshot function
* Minor comment nit
* make drop not wait on stop thread
* graceful stop in more tests
* graceful stop in more tests
* avoid background thread is shard was stopped gracefully
* annotate cancel safety
* fmt
* More cancel safety annotations
* Add two spawns to ensure cancel safety
---------
Co-authored-by: timvisee <tim@visee.me>
* Fix non visible points during snapshots
* Update lib/shard/src/segment_holder/mod.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* code review from mr.rabbit
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Retain more closed segments
* Add back rocksdb
* Update WalOptions/WalConfig across the code
* Use NonZeroUsize
* Expose via APIs
* Update gRPC docs
* Fix stoarge compat test
* recompile openapi.json with rocksdb
* default wal retain closed fn
* update openapi.json
* Use qdrant/wal latest commit and remove from config.yaml
* Deprecate init_from
* Mark memmap_threshold as deprecated
* Mark locks API as deprecated
* Mark RBAC collection access payload filter as deprecated
* Allow deprecations in tests and benchmarks
* deprecate inserting order_value in payload, use the one in the struct instead
* fix for test
* dedup by (order_value, id) tuple
* Scroll dedup test must also consider order value
* Update assert message, include order value
---------
Co-authored-by: timvisee <tim@visee.me>
* Propagate shard key mapping wrapper deeper
* Also bump reverse shard mapping when applying shard keys from snapshot
* Also apply new shard key mappings to existing replicas
* Apply shard key to replica set through apply_state directly
* Set both new shard key mappings through utility function
* Set shard key mappings directly during directory creation
* Rename get_key to key
* Update comment
* Measure update operations hardware IO
* Add support for distributed setups
* also measure update_local
* Add consensus tests for HW metrics of update operations
* add test for upserting without waiting
* Disable HW usage reporting when not waiting for update API
* Review remarks
* Fix resharding collecting hw measurements
* Fix metric type
* New struct HardwareData for better accumulation
* Ensure we always apply CPU multiplier
* Apply suggestions from code review
* Update src/actix/api/update_api.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Fix assert_with_upper_bound_error threshold calculation.
* Clarifying why we don't measure shard cleanup
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Bump Rust edition to 2024
* gen is a reserved keyword now
* Remove ref mut on references
* Mark extern C as unsafe
* Wrap unsafe function bodies in unsafe block
* Geo hash implements Copy, don't reference but pass by value instead
* Replace secluded self import with parent
* Update execute_cluster_read_operation with new match semantics
* Fix lifetime issue
* Replace map_or with is_none_or
* set_var is unsafe now
* Reformat
* rename cpu_budget -> resource_budget
* clippy
* add io budget to resources
* fmt
* move budget structures into a separate file
* add extend permit function
* dont extend existing permit
* switch from IO to CPU permit
* do not release resource before aquiring an extension
* fmt
* Review remarks
* Improve resource permit number assertion
* Make resource permit replace_with only acquire extra needed permits
* Remove obsolete drop implementation
* allocate IO budget same as CPU
* review fixes
---------
Co-authored-by: timvisee <tim@visee.me>
* bump and migrate to rand 0.9.0
also bump rand_distr to 0.5.0 to match it
* Migrate AVX2 and SSE implementations
* Remove unused thread_rng placeholders
* More random migrations
* Migrate GPU tests
* bump seed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Measure read io for payload storage
* Add Hardware Counter to update functions
* Fix tests and benches
* Rename (some) *_measured functions back to original
