* Skip audit logging for telemetry endpoints
Telemetry access checks (telemetry_memory, telemetry_requests,
telemetry_cluster, cluster_telemetry) fire on every metrics scrape and
produce very noisy audit logs with no security-relevant signal.
Switch all telemetry callers to use `auth.unlogged_access()` to bypass
audit logging, consistent with the existing metrics and prepare_data
handlers that already did this.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* trigger CI
* feat: add some tests
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* Skip broken tests
* Add rocksdb dropper
This is a temporary tool. It will be removed later in this PR.
* [automated] Drop rocksdb
This commit is made by running tools/rocksdb/drop.sh
* Touch-up after ast-grep
The previous automated commit removed items, but not their comments.
Also, some blocks are left with only one item.
Also, ast-grep can't handle macros like `vec![]`.
This commit completes the job.
* Remove RocksDB dropper
* Remove mentions of rocksdb feature in CI
* Fix clippy warnings
These `FIXME` comments added previously in this PR by ast-grep by
"peeling" cfg_attr like this:
-#[cfg_attr(not(feature = "rocksdb"), expect(...))]
+#[expect(...)] // FIXME(rocksdb): ...
This commit removes these allow/expect attributes and fixes clippy
lints.
* Remove leftover rocksdb-related code
Removed:
- Cargo.toml: rocksdb cargo feature flag and dependencies.
- code: rocksdb-related items that was not under feature flag.
- flags.rs: rocksdb-related qdrant feature flags.
Disabled:
- Some benchmarks because these do not compile now.
* Print warning if rocksdb leftovers found in snapshots
* Remove Clone from tokenizer
* Regenerate openapi.json
* Test resharding down with deferred points
* clean
* test resharding up as well
* delete and fixes
* Moved the cleanup step between commit_read_hashring and commit_write_hashring.
* feat: add search_max_batchsize to strict mode config
* added test case for search_max_batchsize
* Changes for fixing CI issue dure openapi
* Modify check_strict_mode_batch
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Enable async_scorer in integration tests
Enable `async_scorer=true` for all openapi and shell-based integration
tests by setting the environment variable in `integration-tests.sh`.
This exercises the io_uring vector storage path (on Linux) across all
vector configurations tested there, with graceful fallback if io_uring
is unavailable on the CI runner.
Made-with: Cursor
* Scope async_scorer to test_points_search consensus test only
Revert the blanket async_scorer=true from integration-tests.sh to keep
the normal mmap path covered. Instead, enable it via extra_env in just
the test_points_search consensus test, so we exercise the io_uring path
without losing coverage of the non-io_uring path on Linux.
Made-with: Cursor
* Parametrize test_points_search for sync/async scorer modes
Use pytest.mark.parametrize to run test_points_search twice: once with
the default scorer (sync_scorer) and once with async_scorer=true. This
exercises both the normal mmap and io_uring paths in a single test.
Made-with: Cursor
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
* Test all shard transfer methods for deferred points
* WAL delta carries deferred points
* add point count assertions
- snapshot: asserts target_visible == source_visible — deferred state preserved exactly
- stream_records: asserts target_visible >= source_visible — points may not be deferred on target
* [ai] Allow peer to bootstrap with existing URI if peer has no shards
Prompt:
I have a Qdrant cluster in distributed mode with some nodes registered
in consensus. If I bootstrap a new node with an URL that is already used
it is rejected and an error is returned. I would like to change this
behavior and allow this to happen. This would effectively replace a peer
because internally we'd drop the existing peer first, and then we'd add
the new peer so we can reuse the same peer URL. We must still reject
bootstrapping with the same URL if the peer that used the URL before us
still has any shards on it.
* [ai] Add test to assert new behavior, can rejoin if empty
Prompt:
Add two tests to assert the new behavior.
The first test should:
- create a cluster
- create a collection
- bootstrap a new peer
- kill and delete the local data for this peer without removing it from consensus
- bootstrap a new peer but reuse the URI of the peer we just killed to rejoin
- bootstrapping is expected to succeed
The second test should:
- create a cluster
- create a collection
- bootstrap a new peer but reuse the URI of the last node
- bootstrapping is expected to fail because the node being replaced has data on it
* [ai] Attempt to fix new tests
* Reformat
* [ai] Fix deadlock when rejoining with same peer URI
* [ai] Add test to ensure existing peer stops consensus on replace
* Don't account for deferred points in some places
# Conflicts:
# lib/collection/src/shards/local_shard/scroll.rs
* Add in QueryContext
* Cover more places
* Coderabbit review remarks
* Properly count amount of deleted deferred points (#8386)
* Properly count amount of deleted deferred points
* Prevent double-counting of the same point
* Remove hints to estimations
* Properly handle counts in ProxySegment
* Add tests and fix deleted point count issue
* Adjusts tests + fix issues
* Separte fields for deferred points in telemetry (SegmentInfo)
* Remove deferred_points_count()
* openapi
* Fix test by manually calculating visible points
* Adjust ProxySegment test to revertion of SegmentInfo
* Throw error if collection was not found in telemetry (e2e Test)
* Update lib/segment/src/segment/segment_ops.rs
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Make new fields in API optional
* Don't take range if no deferred point exist
* Review remarks
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Add consensus test for deferred points in snapshot shard transfer
Verify that deferred points (those beyond the indexing threshold in
appendable segments) are included when performing a shard transfer
with the snapshot method, and become visible after optimization on
both source and target nodes.
Made-with: Cursor
* fix
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Add basic test that shows shard holder is locked for too long
* Rework test, move into stream records test
* Fix typos
* Don't lock shard holder for a long time in stream records transfer (#8374)
* [ai] Detach shard holder lock from sending update batch
* Fix clippy warning
* Move stream records delay up to sleep in the middle of the batch
* remove unused code
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Default shard transfer with snapshot when prevent_unoptimized
* extract heuristic
* adjust all possible call sites
* keep previous style
* rework default shard transfer method
* Ensure we don't combine non-streaming transfers with filters
* Clippy
* add http ok assertion before json()
* Use exhaustive match
* keep stream records as fallback
* use effective_optimizers_config
* fix fallback success signal
---------
Co-authored-by: jojii <jojii@gmx.net>
Co-authored-by: timvisee <tim@visee.me>
* chore: gitignore .idea and .vscode globally
Reason: `lib/edge/publish` is a separate Cargo workspace and I'd like to
have a separate rust-analyzier configuration for it, which is stored in
`.vscode/settings.json`. Not sure about `.idea`, but I guess the same
logic applies.
* chore: Use Iterator::is_sorted (stabilized in Rust 1.82)
* chore: remove unrelated files
These were added in #5501 (42fd2e27), perhaps by accident?
* restrict recovery from snapshot folder
* fmt
* clippy
* Extend test, path must be inside snapshots directory
* In release builds, hide detailed tar error message to not leak contents
* Change from bad request to forbidden
* fix(auth_tests): place snapshot in peer snapshots dir for recover_collection_snapshot
The recover from snapshot API now requires file:// paths to be inside the
configured snapshots directory. Write the test snapshot into the peer's
snapshots directory instead of a temp file so the request is allowed.
Made-with: Cursor
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Cursor Agent <agent@cursor.com>
* [manual] make untagged enum for consistent API for FeedbackStrategy
* add #[validate(nested)]
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Add integration test to assert all queued updates are also transferred
* Plunge update queue in stream records transfers
* Migrate existing plunger usages to new plunge helper
* Skip test if not compiled with staging flag
* Only send delay operation when staging feature is enabled
* Reformat
* Fix review remarks
* add rest and grpc interfaces
Also handle inference for this query
* follow refactor from base branch
* renaming from Ms Cooper
* get started on validation testing
* more validations
* test equivalence with query when less than 2 feedback elements
* rename feedback query to relevance_feedback query
* fix extraction of context pairs
* rename feedback vector to example
* make coderabbit happier
* upd test
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* handle `score_threshold` in Formula queries
* AI: add openapi test
prompt: Upload 4 points with a numeric payload, use the payload as the
score, and set a score threshold. We can assert which ids should be in
the result and which shouldn't
* weighted rrf implementation
* test
* fmt
* fix edge
* validate number of sources and number of weights
* do not partial match
* upd schema
* review fixes
* update formula
* remove calcualtions from tests
* update comment, because AI have OCD
* fmt
* Update queue info into CollectionInfo
* basic test
* fix
* cleaner
* Update lib/collection/src/shards/local_shard/mod.rs
Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
* fix the fix
* rename to op_num
* skip_serializing_if just in case
* first step
* test queue length with staging feature
* gate integration test based on binary feature
---------
Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
* introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert
* add test
* upd dockstring
* require resharding once all peers have updated version
* use service error
* fix clippy again
* wait for same version before resharding in tests
