* Skip broken tests
* Add rocksdb dropper
This is a temporary tool. It will be removed later in this PR.
* [automated] Drop rocksdb
This commit is made by running tools/rocksdb/drop.sh
* Touch-up after ast-grep
The previous automated commit removed items, but not their comments.
Also, some blocks are left with only one item.
Also, ast-grep can't handle macros like `vec![]`.
This commit completes the job.
* Remove RocksDB dropper
* Remove mentions of rocksdb feature in CI
* Fix clippy warnings
These `FIXME` comments added previously in this PR by ast-grep by
"peeling" cfg_attr like this:
-#[cfg_attr(not(feature = "rocksdb"), expect(...))]
+#[expect(...)] // FIXME(rocksdb): ...
This commit removes these allow/expect attributes and fixes clippy
lints.
* Remove leftover rocksdb-related code
Removed:
- Cargo.toml: rocksdb cargo feature flag and dependencies.
- code: rocksdb-related items that was not under feature flag.
- flags.rs: rocksdb-related qdrant feature flags.
Disabled:
- Some benchmarks because these do not compile now.
* Print warning if rocksdb leftovers found in snapshots
* Remove Clone from tokenizer
* Regenerate openapi.json
* feat: add search_max_batchsize to strict mode config
* added test case for search_max_batchsize
* Changes for fixing CI issue dure openapi
* Modify check_strict_mode_batch
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Add `#[must_use]` to RAII guard and task handle types
These types rely on being held for a scope — dropping them immediately is
almost always a bug:
- `StoppingGuard`: sets `is_stopped` flag on drop
- `UpdateGuard`: decrements update counter on drop
- `UpdatesGuard`: releases mutex preventing concurrent updates on drop
- `ClockGuard`: marks clock as inactive on drop
- `IsAliveGuard`: releases liveness lock on drop
- `ScopeTrackerGuard`: decrements scope counter on drop
- `CancellableAsyncTaskHandle`: detaches task on drop
- `StoppableTaskHandle`: may abort task on drop (AbortOnDropHandle)
Made-with: Cursor
* Remove redundant function-level #[must_use] now covered by type
With ScopeTrackerGuard marked #[must_use] at the type level, the
function-level attributes on measure_scope(), measure(),
track_create_snapshot_request(), and count_snapshot_creation() are
redundant and trigger clippy::double_must_use.
Made-with: Cursor
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
* Add REST API for reading audit logs across the cluster
Introduces a new `GET /audit/logs` endpoint that retrieves audit log
entries from all peers in the cluster. The API supports filtering by
time range (time_from/time_to), dynamic key=value field filters, and
a built-in limit parameter to prevent reading too many logs at once.
- Add `audit_reader` module in storage crate for efficient file-based
log retrieval, selecting only files whose date range overlaps the
query window
- Add `GetAuditLog` internal gRPC RPC for cross-peer log retrieval
- Add `GET /audit/logs` REST endpoint restricted to management access
- Aggregate and sort results from all peers by timestamp (newest first)
Made-with: Cursor
* [AI] Update filter_files_by_time_range make it aware of the log rotation granulatity (either hourly or daily)
* [AI] For AuditLogParams, try to use DateTime types natively into serde, instead of manual parsing
* [AI] Refactor API into separate file, propagate timeout, use spawn-blocking
* [AI] introduce cancellation token
* [AI] move timestamp to constant
* small manual fixes
* review fixes part 1
* review: switch to POST instead of GET
* [AI] review: sorting update
* [AI] use strict typing
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Refactor: `SearchSegmentEntry` for search ops
* Move `has_deferrend_points_method`
to `SearchSegmentEntry`
* Reorg imports
* Renamings per review
* fixup! Renamings per review
* `StorageSegmentEntry` trait
It contains methods dealing with storage and syncronization
* Move index methods from `ReadSegmentEntry`
to `SegmentEntry`
* Add `_concurrent` suffix to some methods
* move field index methods into NonAppendableSegmentEntry
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Adjust debug logline's format of
ShardTransferRestart to match with ShardTransfer
* Convert into shardtransfer
* Deconstruct ShardTransferRestart struct
Instead of `IdTracker` to have `iter_*` methods, it now has a `point_mappings()` method, returning a guard to `PointMapping` value. It prepares for moving the point_mappings under a lock to allow parallel searches and deletions.
Also, an incorrect unsafe in `Segment::iter_points` is replaced by a safe version with `self_cell`.
* Add deferred point count to UpdateQueueInfo
* Hide if prevent_unoptimized is false
* Clippy
* Openapi
* Iterate over appendable segments only
* Only calculate deferred point count if prevent_unoptimized is true
* Rebase fixes
* Codespell
* Don't account for deferred points in some places
# Conflicts:
# lib/collection/src/shards/local_shard/scroll.rs
* Add in QueryContext
* Cover more places
* Coderabbit review remarks
* Properly count amount of deleted deferred points (#8386)
* Properly count amount of deleted deferred points
* Prevent double-counting of the same point
* Remove hints to estimations
* Properly handle counts in ProxySegment
* Add tests and fix deleted point count issue
* Adjusts tests + fix issues
* Separte fields for deferred points in telemetry (SegmentInfo)
* Remove deferred_points_count()
* openapi
* Fix test by manually calculating visible points
* Adjust ProxySegment test to revertion of SegmentInfo
* Throw error if collection was not found in telemetry (e2e Test)
* Update lib/segment/src/segment/segment_ops.rs
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Make new fields in API optional
* Don't take range if no deferred point exist
* Review remarks
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
In two places we collected an iterator of Results into a
Result<Vec<_>> only to immediately iterate the Vec again.
Use `itertools::process_results` instead to pipe the unwrapped
items straight into the downstream iterator chain, removing
the unnecessary heap allocation.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* Log individual payload index load time per field
Add per-field timing to `load_all_fields()` in `StructPayloadIndex`,
using the existing `LOAD_TIMING_LOG_TARGET` infrastructure. Each
payload field index that takes >= 5ms to load now gets its own log
line, making it easy to identify slow-loading fields.
Made-with: Cursor
* Move `log_load_timing` to common module and use everywhere
Move the `log_load_timing` helper from `segment_constructor_base` into
`common::defaults` so it can be reused across crates. Convert all raw
`log::debug!(target: LOAD_TIMING_LOG_TARGET, ...)` call sites to use
the shared function, giving consistent formatting and min-duration
suppression everywhere.
Made-with: Cursor
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Reduce test parameters across 5 areas to cut test runtime without
sacrificing coverage:
1. HNSW PQ tests: lower vector dimensionality from 131 to 64 for product
quantization variants, cutting PQ codebook training time (~31s -> ~8s)
2. HNSW index build: use 2 threads instead of 1 for index construction;
the tests only check accuracy above a threshold so determinism is not
required
3. Search attempts: reduce from 10 to 5 query vectors per test
4. Rescoring: skip the expensive re-upsert-all-as-zeros rescoring check
for PQ tests (already covered by scalar quantization variants)
5. Near-miss speedups:
- WAL: reduce QuickCheck iterations from 100 to 50
- Gridstore: halve operation counts, drop 64-byte block size case,
reduce proptest cases for gap search
- Continuous snapshot: reduce timeout from 20s to 10s
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* Replace `has_deferred_points` with `deferred_points_count` in SegmentEntry
Change the `SegmentEntry` trait method from returning a boolean to
returning the actual count of deferred points (`usize`). This gives
callers more information without needing a separate method, while
existing boolean checks simply compare against zero.
Made-with: Cursor
* Fix clippy collapsible_if warning in deferred_points_count
Made-with: Cursor
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
* Add basic test that shows shard holder is locked for too long
* Rework test, move into stream records test
* Fix typos
* Don't lock shard holder for a long time in stream records transfer (#8374)
* [ai] Detach shard holder lock from sending update batch
* Fix clippy warning
* Move stream records delay up to sleep in the middle of the batch
* remove unused code
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Default shard transfer with snapshot when prevent_unoptimized
* extract heuristic
* adjust all possible call sites
* keep previous style
* rework default shard transfer method
* Ensure we don't combine non-streaming transfers with filters
* Clippy
* add http ok assertion before json()
* Use exhaustive match
* keep stream records as fallback
* use effective_optimizers_config
* fix fallback success signal
---------
Co-authored-by: jojii <jojii@gmx.net>
Co-authored-by: timvisee <tim@visee.me>
* Add debug-level load timing logs for storage components
Add log::debug! calls with target "qdrant::load_timing" that report
how long each storage component takes to load during startup:
- Total shard load time (including WAL replay)
- Total segment load time + load_state
- Per-segment: payload_storage, id_tracker, payload_index
- Per-vector: vector_storage (dense/sparse), quantized_vectors,
vector_index (dense/sparse)
These logs are at DEBUG level so they are silent by default. Enable
selectively via log_level config or env, for example:
log_level: "INFO,qdrant::load_timing=debug"
No config fields, no global flags, no function signature changes.
Made-with: Cursor
* Use seconds with 10ms resolution for load timing logs, extract log target constant
Switch from `{:.2}ms` with `as_secs_f64() * 1000.0` to `{:.2}s` with
`as_secs_f64()` for cleaner output. Move the "qdrant::load_timing"
log target string to a LOAD_TIMING_LOG_TARGET constant in each file.
Made-with: Cursor
* Move LOAD_TIMING_LOG_TARGET to common::defaults
Single definition shared by segment and collection crates, avoiding
duplication of the log target string.
Made-with: Cursor
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
* Add ignore_deferred flag for internal scroll API
* Ignore deferred points in ProxyShard::update
* Use enum instead of bool as type in function parameters
* Use consistent parameter name
* fmt
* Apply suggestions from code review
Co-authored-by: Tim Visée <tim+github@visee.me>
* More explicit naming
* Add DeferredBehavior to `count()` and disable filtering in stream_records
* Don't filter `retrieve()` everywhere
* Use consistent filter behavior in read operations
* Add TODO for ignored parameter in remote_shard
* Rebase fixes
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Change backpressure policy for deferred points
* back pressure waits for deferred points
* Remove prevent_unoptimized_threshold_kb
* tests
* are you happy fmt
* better optimization wait
* fix after rebase
* send signal if optimizer is over the budget
* Make wait_for_deferred_points_ready cancel safe
* fix test
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: timvisee <tim@visee.me>
* [manunal] points deduplication after optimization
* take a list of deferred point ids
* test
* review remarks
* review remarks
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* Use array_windows when statically sized windows were used
* Bump MSRV to 1.94
* fix in gpu code
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
