* Add per-query IDF corpus for sparse vector search
Let the caller choose, per query, which population sparse IDF statistics
are computed over. `params.idf` is either `"global"` (default, unchanged
behavior) or `{"corpus": <filter>}`, where the corpus filter is
independent of - and usually broader than - the retrieval filter.
Decoupling the two keeps the score scale stable when the retrieval
filter tightens: term importance is measured against a population the
user names, not against whatever subset the filter happens to select.
Design decisions:
- Corpus grammar is restricted to a conjunction (`must`) of `match`
conditions on payload fields; loosening later is backward compatible.
- Strict mode validates the corpus filter like a read filter
(unindexed fields rejected).
- `idf` on a vector without the IDF modifier is a validation error,
never silently ignored.
- An empty corpus yields degenerate but corpus-scoped scores (smoothed
IDF over N=0), never a fallback to global statistics - in multi-tenant
collections a fallback would leak term statistics across tenants.
Implementation:
- QueryContext IDF stats are keyed by corpus, so one batch can mix
requests with different corpora.
- Statistics come from the sparse index: df(term) is counted over the
query terms' posting lists only, never by scanning stored vectors.
Small corpora (under ~1/32 of the segment, by cardinality estimate)
are kept as a sorted id list galloping through posting lists via
skip_to; large ones as a dense membership mask filled streaming from
the filtered-points iterator. A misestimated small corpus degrades
into the mask.
- Exposed uniformly: REST (`params.idf`), gRPC (`IdfParams` message),
edge python bindings; OpenAPI schema regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply rustfmt
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix clippy manual_is_multiple_of in sparse IDF corpus test.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Allow any filter as IDF corpus
Drop the must+match grammar restriction on the corpus filter. A
restriction enforced only as a validation step over the full Filter
type buys nothing; if a narrower corpus syntax is ever wanted, it
should be a dedicated API-level type instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix build: add memory field to SparseIndexConfig in idf corpus test
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* [AI] fix: bound random-sample pre-allocation by available point count
`LocalShard::scroll_randomly` pre-allocated its result set with
`HashSet::with_capacity(limit)`, where `limit` is the client-supplied request
limit and is unbounded by default (`StrictModeConfig::max_query_limit` is `None`
unless an operator sets it). A very large limit made the up-front allocation
fail and abort the process (`handle_alloc_error`), which cannot be caught and
returned as an error.
The sample loop inserts at most `min(limit, total points available across
segments)` points, so bound the pre-allocation to what is actually available.
Capacity is unchanged for normal requests; only the pathological case is bounded.
Same class as the search top-k fix (#4321 / #4328) and the in-flight edge fix
(#9374). Adds a regression test.
* [AI] fix: cap random-sample preallocation by filtered candidate count
Review feedback. `available_point_count_without_deferred()` is filter-unaware,
so `availability.iter().sum()` could still drive `HashSet::with_capacity()` to
reserve a buffer proportional to the full collection on a highly selective
filtered request, re-opening the allocator-abort path on large collections.
The sampling loop only ever inserts points read into `segments_reads`, which is
filter-aware and already capped at `limit` per segment, so bound the
preallocation by that candidate count instead.
* Update lib/collection/src/shards/local_shard/scroll.rs
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
---------
Co-authored-by: Andrey Vasnetsov <vasnetsov93@gmail.com>
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
* Add routing token structure
* Implement routing token in read operation executor as per design doc
* Add TODO to glue routing token to user requests
* Implement routing header for REST API
* Source routing token from request, not from JWT token
* Implement routing token in gRPC API
* Add test
* Review remarks
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Use lower case header name to prevent panic
* Rename header to X-Qdrant-Route-Affinity
* Assert routing consistency in test on all peers
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Use assert_matches!
* Add trailing commas
* Use more assert_matches!
Also, drop now redundant `expected blah but got blah` messages because
`assert_matches!` will print these.
* Use debug_assert_matches!
---------
Co-authored-by: xzfc <xzfcpw@gmail.com>
* [ai] Replace manual into mappings with Into::into
* Reformat
* [ai] Use implicit .iter
* Don't iterate over keys too
* [ai] Replace unwrap_or
* Reformat
* [ai] Use as_deref and then_some
* [ai] Use more to_string
* [ai] Use explicitly typed into conversions
* Reformat
* [ai] More explicit into conversions
* Reformat
Reduce test parameters across 5 areas to cut test runtime without
sacrificing coverage:
1. HNSW PQ tests: lower vector dimensionality from 131 to 64 for product
quantization variants, cutting PQ codebook training time (~31s -> ~8s)
2. HNSW index build: use 2 threads instead of 1 for index construction;
the tests only check accuracy above a threshold so determinism is not
required
3. Search attempts: reduce from 10 to 5 query vectors per test
4. Rescoring: skip the expensive re-upsert-all-as-zeros rescoring check
for PQ tests (already covered by scalar quantization variants)
5. Near-miss speedups:
- WAL: reduce QuickCheck iterations from 100 to 50
- Gridstore: halve operation counts, drop 64-byte block size case,
reduce proptest cases for gap search
- Continuous snapshot: reduce timeout from 20s to 10s
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* Move `DenseVector`/`MultiDenseVector` from `api` to `segment`
* Move `OrderByInterface` from `api` to `segment`
Reason: it's used in `edge` which shouldn't depend on `api`.
* Make `shard` -> `api` dependency optional
* Remove `api` from the amalgamation
* Don't install protoc in edge Actions
* [manual] generate snapshot download link with respect of enabled tls
* Simplify http(s) selection
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* download tar
* compute sha256 for stream download
* wip: propagate unpacking into down to the logic, todo: validation
* unpacked snapshot validation
* Minor tweaks
* Fix typo
* validation during unpack
* cancellation token
* update docstring
* remove redundant dep
* Rearrange unpack functions
- Rename `safe_unpack.rs` into `tar_unpack.rs` so it would be listed
near `tar_ext.rs` in IDEs.
- Replace calls like `ar = open_snapshot_archive(…); safe_unpack(ar, …);`
with a single call to `tar_unpack_file(…)`.
- Put calls to `Archive::new(); Archive::set_overwrite(false);` inside
`tar_unpack_reader` (was `safe_unpack`). So, now it is the only place
that does `set_overwrite`.
* Let clippy complain if tar::Archive::unpack used
* Mock snapshot download URL
Instead of downloading from storage.googleapis.com every time the test
runs, put small snapshot file to the repo.
The snapshot file is created using this command:
curl -s \
https://storage.googleapis.com/qdrant-benchmark-snapshots/test-shard.snapshot \
| tar \
--delete segments/4ea958d8-0b64-4312-9a53-0cd857e93535.tar \
--delete segments/65ac6276-8cca-4f5c-b767-9722190cee8b.tar \
> lib/storage/src/content_manager/snapshots/test-shard.snapshot
File contents:
$ tar tf lib/storage/src/content_manager/snapshots/test-shard.snapshot
wal/
wal/closed-255
newest_clocks.json
replica_state.json
shard_config.json
$ du -sh lib/storage/src/content_manager/snapshots/test-shard.snapshot
12K lib/storage/src/content_manager/snapshots/test-shard.snapshot
$ sha256sum < lib/storage/src/content_manager/snapshots/test-shard.snapshot
5d94eac5c1ede3994a28bc406120046c37370d5d45b489a0d2252531b4e3e1f2 -
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* sync function to ask to stop workers
fmt
update blocking_ask_workers_to_stop
* graceful shard stop on ShardReplicaSet changes
* make stop consuming
* implement graceful stop for replica set
* graceful shard stop in start_resharding_unchecked
* stop_gracefully fn on shard_holder
* collection graceful stop on delete
* explain when it is ok to shutdown
* fix tests and ensure blocking callsa are not called from async runtime
* fmt
* Annotate cancel safety for do_recover_from_snapshot function
* Minor comment nit
* make drop not wait on stop thread
* graceful stop in more tests
* graceful stop in more tests
* avoid background thread is shard was stopped gracefully
* annotate cancel safety
* fmt
* More cancel safety annotations
* Add two spawns to ensure cancel safety
---------
Co-authored-by: timvisee <tim@visee.me>
* Fix non visible points during snapshots
* Update lib/shard/src/segment_holder/mod.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* code review from mr.rabbit
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Retain more closed segments
* Add back rocksdb
* Update WalOptions/WalConfig across the code
* Use NonZeroUsize
* Expose via APIs
* Update gRPC docs
* Fix stoarge compat test
* recompile openapi.json with rocksdb
* default wal retain closed fn
* update openapi.json
* Use qdrant/wal latest commit and remove from config.yaml
* Deprecate init_from
* Mark memmap_threshold as deprecated
* Mark locks API as deprecated
* Mark RBAC collection access payload filter as deprecated
* Allow deprecations in tests and benchmarks
* deprecate inserting order_value in payload, use the one in the struct instead
* fix for test
* dedup by (order_value, id) tuple
* Scroll dedup test must also consider order value
* Update assert message, include order value
---------
Co-authored-by: timvisee <tim@visee.me>
* Propagate shard key mapping wrapper deeper
* Also bump reverse shard mapping when applying shard keys from snapshot
* Also apply new shard key mappings to existing replicas
* Apply shard key to replica set through apply_state directly
* Set both new shard key mappings through utility function
* Set shard key mappings directly during directory creation
* Rename get_key to key
* Update comment
* Measure update operations hardware IO
* Add support for distributed setups
* also measure update_local
* Add consensus tests for HW metrics of update operations
* add test for upserting without waiting
* Disable HW usage reporting when not waiting for update API
* Review remarks
* Fix resharding collecting hw measurements
* Fix metric type
* New struct HardwareData for better accumulation
* Ensure we always apply CPU multiplier
* Apply suggestions from code review
* Update src/actix/api/update_api.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Fix assert_with_upper_bound_error threshold calculation.
* Clarifying why we don't measure shard cleanup
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Bump Rust edition to 2024
* gen is a reserved keyword now
* Remove ref mut on references
* Mark extern C as unsafe
* Wrap unsafe function bodies in unsafe block
* Geo hash implements Copy, don't reference but pass by value instead
* Replace secluded self import with parent
* Update execute_cluster_read_operation with new match semantics
* Fix lifetime issue
* Replace map_or with is_none_or
* set_var is unsafe now
* Reformat
* rename cpu_budget -> resource_budget
* clippy
* add io budget to resources
* fmt
* move budget structures into a separate file
* add extend permit function
* dont extend existing permit
* switch from IO to CPU permit
* do not release resource before aquiring an extension
* fmt
* Review remarks
* Improve resource permit number assertion
* Make resource permit replace_with only acquire extra needed permits
* Remove obsolete drop implementation
* allocate IO budget same as CPU
* review fixes
---------
Co-authored-by: timvisee <tim@visee.me>
* bump and migrate to rand 0.9.0
also bump rand_distr to 0.5.0 to match it
* Migrate AVX2 and SSE implementations
* Remove unused thread_rng placeholders
* More random migrations
* Migrate GPU tests
* bump seed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Measure read io for payload storage
* Add Hardware Counter to update functions
* Fix tests and benches
* Rename (some) *_measured functions back to original
* Add UUID to collection...
...and recreate collection, when applying Raft snapshot, if UUID of collection is different
* fixup! Add UUID to collection...
Remove UUID field from gRPC and exclude it from OpenAPI spec 🤡
* fixup! fixup! Add UUID to collection...
Always generate collection UUID 🤦♀️
* Raft snapshot recreate collection no expose UUID (#5452)
* separate colleciton config structure from API
* fmt
* Update lib/collection/src/operations/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Send current replica state with disable replica proposals
* Don't be strict about from state if in shard transfer related state
* Link to PR, update formatting
* Fix typo
* Fix test compilation failures
* Provide peer state when deactivating leader, remove unnecessary TODO
* ReplicaState is Copy
* include document & image objects into grpc API
* introduce image and object to rest api
* minor refactoring
* rename Vector -> VectorInternal
* decompose vector data structures
* add schema
* fmt
* grpc docs
* fix conversion
* fix clippy
* fix another conversion
* rename VectorInput -> VectorInputInternal
* replace grpc TryFrom with async functions
* fmt
* replace rest TryFrom with async functions
* add image and object into query rest
* separate inference related conversions
* move json-related conversions into a separate file
* move vector-related transformations into a separate file
* move more vector related-conversions into dedicated module
