Match the REST API by adding an optional `WriteOrdering` field to
`CreateVectorNameRequest` and `DeleteVectorNameRequest`, and propagate
it through the tonic handlers and remote-shard forwarding paths.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [ai] Always set default transfer method on entry peer
This will propagate a specific transfer type through consensus to all
peers. It ensures all peers will use the same transfer method when
applying the operation.
* [ai] Enforce the shard method to be specified
* Still allow unspecified transfer method for older peer versions
* Use snapshot shard transfers by default in Qdrant 1.18.0 and up
* New method does not have to be async
* Use stream records based transfer for replicate points with filter
* [ai] On shard snapshot transfer recovery, drop existing shard before recovery
* [ai] Add integration test to assert clearing behavior
* [ai] Debug assert our replica is not active when we clear it
* [ai] Tweak assertion
* Fix flaky test, replica may temporarily not be visible
* Replace debug assertion with runtime error
* UniversalReadPipeline
* Performance: cache pointers in MmapFile
The previous commit removed `MmapFile::read_batch` method override. So,
`MmapFile` now re-uses the default `UniversalRead::read_batch` impl,
which is implemented using `UniversalReadPipeline` interface.
Unsurprisingly, it caused a slowdown in the benchmarks, particularly
this one:
cargo bench -p common --bench universal_io -- mmap/8bytes/read_batch_full
This commit reclaims the performance back.
* remove unfulfilled lint
* Review suggestions
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* [ai] Add config boolean to disable URL based snapshot restore
* Merge if-statement
* Comment-out config option by default
* [ai] Also block partial snapshots from remote URLs
* [ai] Only run clock consistency test when staging feature is present
* [ai] Add integration test
`StrictModeConfigOutput` is an output-only type that is never validated,
so remove its dead `#[validate(...)]` field attributes.
`ShardSnapshotRecover` does need validation: add the missing `Validate`
derive and switch the actix handler from `web::Json` to `valid::Json`
so the SHA256 checksum constraint is actually enforced.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
Extract self-contained pieces of `main()` into helper functions and group
the remaining body into labeled sections. No functional changes — the
order of operations, arguments, and control flow are preserved.
Extracted helpers:
- install_default_crypto_provider
- run_stacktrace_collector
- check_filesystem_compatibility
- init_gpu_devices (gpu feature)
- resolve_bootstrap_uri
- recover_collections_from_snapshot_args
- init_channel_service
- spawn_deadlock_checker (service_debug feature)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Gate `FsType` enum and impl with `#[cfg(fs_type_check_supported)]` to
suppress dead_code warnings on platforms where the fs type check is not
supported (e.g. Windows).
Move `CollectionError` import into the `#[cfg(unix)]` block where it is
actually used, fixing unused_imports on non-unix.
Add `#[cfg(unix)]` to `OTHER_COLLECTION_NAME` test constant that is
only referenced in unix-gated tests.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* [AI] strict mode parameter for limiting update requests if ram usage is over threshold
* opanAPI update
* [AI] end-to-end test
* fmt
* Fix e2e test: memory rejection check broken by string truncation
UnexpectedResponse.__str__() truncates the raw response body, cutting
off the `max_resident_memory_percent` hint at the end of the error
message. Use `resident memory usage` instead, which appears early
enough to survive the truncation.
Made-with: Cursor
* add grpc validation
* test check_resident_memory
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Test if io_uring handles EINTR properly
* Fix unit test compilation after read_iter API change
Update test_io_uring_eintr_handling to match the new read_iter signature
that takes (Meta, ReadRange) tuples and returns Result<impl Iterator>.
Made-with: Cursor
* Install no-op SIGUSR1 handler in debug mode on Unix
Prevents SIGUSR1 from terminating the process with the default
disposition, so that io_uring EINTR tests can safely bombard
the process with signals.
Made-with: Cursor
* Enter tokio runtime context for SIGUSR1 handler, fix clippy
tokio::signal::unix::signal requires a reactor context, so enter
the runtime handle before installing the handler.
Also fix manual_let_else clippy warning in the EINTR unit test.
Made-with: Cursor
* Cleanup 🙄
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
Previously `clear_all_tmp_directories()` was called in `main.rs` after
`TableOfContent::new()` had already loaded all collections and applied
WAL. Stale temp files from a previous crash (e.g. interrupted snapshot
transfers) could interfere with the recovery process.
Move the cleanup into `TableOfContent::new()` so it runs before the
collection loading loop. Extract a standalone `clear_tmp_directories()`
function that only needs `StorageConfig`, and delegate the existing
method to it.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* Add mincore-based memory stats to MmapFile
Add `resident_bytes()`, `disk_bytes()`, and `probe_memory_stats()` methods
to `MmapFile` for measuring page cache residency via `mincore(2)`. This is
the foundation for per-collection memory usage reporting.
Also extract `page_size()` as a public function in `mmap::advice`, replacing
the internal `PAGE_SIZE_MASK` with a direct page size cache.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] introduce trait for reporting memory usage per component
* [AI] memory reporter implementation for vector storage
* [AI] implement MemoryReporter for QuantizedVectors
* [AI] implement MemoryReporter for VectorIndexEnum
* Implement MemoryReporter for IdTrackerEnum with RAM estimation
Add ram_usage_bytes() to all ID tracker types and their data structures:
- PointMappings, CompressedPointMappings, CompressedVersions,
CompressedInternalToExternal, CompressedExternalToInternal
- MutableIdTracker, ImmutableIdTracker, InMemoryIdTracker
All ID trackers load their data into RAM (none use mmap for working data).
Files are reported as OnDisk (persistence only), actual RAM footprint
is reported via extra_ram_bytes. Uses struct destructuring to ensure
new fields trigger compile errors if not accounted for.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] implement MemoryReporter for PayloadStorageEnum and adjust FileStorageIntent
* [AI] implement MemoryReporter for PayloadStorageEnum and adjust FileStorageIntent
* [AI] implement MemoryReporter for payload indexes: in-ram structures memory consumtion computation + caching
* [AI] implement MemoryReporter for payload indexes: in-ram structures memory consumtion computation + caching
* [AI] segment-level memory usage report
* [AI] Block 3: Aggregation Layer and Data Model + internal api for remote shard
* [AI] REST API handler
* fmt
* [AI] clippy fixes
* [AI] macos fix + proxy segment fix
* [AI] make text index estimation a bit more correct
* fix is_on_disk reporting for dense_vector_storage
* fix after rebase
* [AI] deep account for quantized vectors RAM usage + unify chunk size + shring volatile storage after load
* remove debug log
* cache in test
* make manual test easier to run
* rollback chunk size diff, but keep it for test only
* review fixes
* Use exhaustive match
* Use div_ceil on bits everywhere
It does not seem to be strictly necessary because the number of bits
should already be a multiple of the used container size bytes. Still
it's good practice to be careful with this calculation.
* Improve heap size bytes for encoded product quantization vectors
* Include vector stats for binary quantized vectors
* In volatile chunked vectors, include heap allocated vector
* Include rest of heap allocated structures for mutable map index
* In mutable geo index, the hash map is also heap allocated
* Update tests/manual/test_memory_reporting.py
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Add empty placeholder vector storage types for named vector CRUD
Introduce EmptyDenseVectorStorage and EmptySparseVectorStorage as
placeholder storages for newly created named vectors on immutable
segments. These report all vectors as deleted, consume no disk space,
and are reconstructed from segment config on load via the new
VectorStorageType::Empty and SparseVectorStorageType::Empty variants.
Key design decisions:
- is_on_disk is derived from original user config, not hardcoded
- MultiVectorConfig is preserved for multi-vector support
- Config mismatch optimizer skips Empty storage to avoid false rebuilds
- Quantization delegates normally (handles 0 vectors gracefully)
- get_vector includes debug_assert to catch unexpected access
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] segment-level operations for creating and deleting anmed vectors
* [AI] implement named vector creation and deleting in proxy segment
* [AI] Step 3: Proxy Segment Handling for Named Vector Operations
* [AI] implement for Edge
* [AI] implement consensus operations for named vector operations
* [AI] refactor VectorNameConfig, remove VectorNameConfigInternal
* [AI] handle vector schema inconsistency in raft snapshot recovery
* [AI] rest + grpc API
* [AI] clippy
* [AI] generate openAPI schema
* fmt
* ci fixes
* [AI] fix jwt access test
* [AI] nop operation for awaiting of consensus-commited update ops
* [AI] move vector name operations into points service
* [AI] implement internal api for vector name operations
* [AI] change collection-level config along with segment level operation
* [AI] vector schema reconceliation instead of error
* fmt
* missing compile-time option
* [AI] integration test
* [AI] fix missing JWT tests
* [AI] remove NOP
* [AI] openapi test
* [AI] fix initialization of mutable segment
* [AI] more simple integration tests
* fmt
* [AI] make cluster test a bit harder
* [AI] make test less flacky
* [AI] rabbit comments
* [AI] check params compatibility before writing vector config
* [AI] make sure to register vector storages in structure payload index
* [AI] vector name validation
* lower vector length validation to 200 chars to account for prefix in filename
* [AI] proxy segment: prevent stale data leak through optimization
* fmt
* [AI] filter out removed vectors from proxy response
* [AI] handle vector name in proxy
* fmt
* adjust proxy info based on dropped vectors
* [AI] proxy segment: update filters to correct has_vector condition
* fmt
* clippy
* Fix consensus snapshot applicaiton for vector schema
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Return error instead of panicking for corrupted aliases file on startup
* common::fs::ops: provide file name in error messages
And drop FileStorageError in favor of std::io::Error, since we always
convert all kinds of errors into ServiceError anyway.
* TableOfContent:🆕 return errors instead of panics
Also, drop context strings. We use fs_err anyway, that should be enough.
---------
Co-authored-by: leohenon <77656081+lhenon999@users.noreply.github.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* [ai] Replace manual into mappings with Into::into
* Reformat
* [ai] Use implicit .iter
* Don't iterate over keys too
* [ai] Replace unwrap_or
* Reformat
* [ai] Use as_deref and then_some
* [ai] Use more to_string
* [ai] Use explicitly typed into conversions
* Reformat
* [ai] More explicit into conversions
* Reformat
* Add optional `api` field to audit log events
Add a new `api` field to audit log entries that records the API method
path (REST path or gRPC method name). Controlled by the `log_api` audit
config option. For denied auth requests, `api` is always logged and
`method` is omitted since there is no internal operation name available.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix formatting
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* deconstruct
* fix: test edge case
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* IoUringState: generic `RequestId`
* UniversalRead: generic `RequestId`
* Simplify `gridstore::Pages::get_page_value_ranges`
Now we don't need two separate `SmallVec`s as we can put `buffer_offset`
into `RequestId`.
* Better doc comment
* Rename `RequestId` -> `Meta`
Replace .to_str().unwrap() with .display() to handle paths
containing non-UTF-8 characters gracefully.
This matches the pattern used elsewhere in the codebase (e.g.,
actix/api/snapshot_api.rs:101, tonic/auth.rs:59).
Fixes#8563
Co-authored-by: easonysliu <easonysliu@tencent.com>
* Skip audit logging for telemetry endpoints
Telemetry access checks (telemetry_memory, telemetry_requests,
telemetry_cluster, cluster_telemetry) fire on every metrics scrape and
produce very noisy audit logs with no security-relevant signal.
Switch all telemetry callers to use `auth.unlogged_access()` to bypass
audit logging, consistent with the existing metrics and prepare_data
handlers that already did this.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* trigger CI
* feat: add some tests
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* Skip broken tests
* Add rocksdb dropper
This is a temporary tool. It will be removed later in this PR.
* [automated] Drop rocksdb
This commit is made by running tools/rocksdb/drop.sh
* Touch-up after ast-grep
The previous automated commit removed items, but not their comments.
Also, some blocks are left with only one item.
Also, ast-grep can't handle macros like `vec![]`.
This commit completes the job.
* Remove RocksDB dropper
* Remove mentions of rocksdb feature in CI
* Fix clippy warnings
These `FIXME` comments added previously in this PR by ast-grep by
"peeling" cfg_attr like this:
-#[cfg_attr(not(feature = "rocksdb"), expect(...))]
+#[expect(...)] // FIXME(rocksdb): ...
This commit removes these allow/expect attributes and fixes clippy
lints.
* Remove leftover rocksdb-related code
Removed:
- Cargo.toml: rocksdb cargo feature flag and dependencies.
- code: rocksdb-related items that was not under feature flag.
- flags.rs: rocksdb-related qdrant feature flags.
Disabled:
- Some benchmarks because these do not compile now.
* Print warning if rocksdb leftovers found in snapshots
* Remove Clone from tokenizer
* Regenerate openapi.json
* feat: add search_max_batchsize to strict mode config
* added test case for search_max_batchsize
* Changes for fixing CI issue dure openapi
* Modify check_strict_mode_batch
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Snapshot endpoints used raw `web::Path<String>` / `web::Path<(String, ...)>`
instead of validated path types, allowing invalid collection names (including
NUL bytes) to bypass `validate_collection_name` checks and reach the
filesystem, causing 500 errors.
Replace all raw path extractors with `valid::Path<CollectionPath>` and new
validated structs (`CollectionSnapshotPath`, `CollectionShardPath`,
`CollectionShardSnapshotPath`) that enforce legacy collection-name validation,
consistent with the rest of the API.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* Add REST API for reading audit logs across the cluster
Introduces a new `GET /audit/logs` endpoint that retrieves audit log
entries from all peers in the cluster. The API supports filtering by
time range (time_from/time_to), dynamic key=value field filters, and
a built-in limit parameter to prevent reading too many logs at once.
- Add `audit_reader` module in storage crate for efficient file-based
log retrieval, selecting only files whose date range overlaps the
query window
- Add `GetAuditLog` internal gRPC RPC for cross-peer log retrieval
- Add `GET /audit/logs` REST endpoint restricted to management access
- Aggregate and sort results from all peers by timestamp (newest first)
Made-with: Cursor
* [AI] Update filter_files_by_time_range make it aware of the log rotation granulatity (either hourly or daily)
* [AI] For AuditLogParams, try to use DateTime types natively into serde, instead of manual parsing
* [AI] Refactor API into separate file, propagate timeout, use spawn-blocking
* [AI] introduce cancellation token
* [AI] move timestamp to constant
* small manual fixes
* review fixes part 1
* review: switch to POST instead of GET
* [AI] review: sorting update
* [AI] use strict typing
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Refactor: `SearchSegmentEntry` for search ops
* Move `has_deferrend_points_method`
to `SearchSegmentEntry`
* Reorg imports
* Renamings per review
* fixup! Renamings per review
* `StorageSegmentEntry` trait
It contains methods dealing with storage and syncronization
* Move index methods from `ReadSegmentEntry`
to `SegmentEntry`
* Add `_concurrent` suffix to some methods
* move field index methods into NonAppendableSegmentEntry
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* [ai] Allow peer to bootstrap with existing URI if peer has no shards
Prompt:
I have a Qdrant cluster in distributed mode with some nodes registered
in consensus. If I bootstrap a new node with an URL that is already used
it is rejected and an error is returned. I would like to change this
behavior and allow this to happen. This would effectively replace a peer
because internally we'd drop the existing peer first, and then we'd add
the new peer so we can reuse the same peer URL. We must still reject
bootstrapping with the same URL if the peer that used the URL before us
still has any shards on it.
* [ai] Add test to assert new behavior, can rejoin if empty
Prompt:
Add two tests to assert the new behavior.
The first test should:
- create a cluster
- create a collection
- bootstrap a new peer
- kill and delete the local data for this peer without removing it from consensus
- bootstrap a new peer but reuse the URI of the peer we just killed to rejoin
- bootstrapping is expected to succeed
The second test should:
- create a cluster
- create a collection
- bootstrap a new peer but reuse the URI of the last node
- bootstrapping is expected to fail because the node being replaced has data on it
* [ai] Attempt to fix new tests
* Reformat
* [ai] Fix deadlock when rejoining with same peer URI
* [ai] Add test to ensure existing peer stops consensus on replace
* Default shard transfer with snapshot when prevent_unoptimized
* extract heuristic
* adjust all possible call sites
* keep previous style
* rework default shard transfer method
* Ensure we don't combine non-streaming transfers with filters
* Clippy
* add http ok assertion before json()
* Use exhaustive match
* keep stream records as fallback
* use effective_optimizers_config
* fix fallback success signal
---------
Co-authored-by: jojii <jojii@gmx.net>
Co-authored-by: timvisee <tim@visee.me>
* restrict recovery from snapshot folder
* fmt
* clippy
* Extend test, path must be inside snapshots directory
* In release builds, hide detailed tar error message to not leak contents
* Change from bad request to forbidden
* fix(auth_tests): place snapshot in peer snapshots dir for recover_collection_snapshot
The recover from snapshot API now requires file:// paths to be inside the
configured snapshots directory. Write the test snapshot into the peer's
snapshots directory instead of a temp file so the request is allowed.
Made-with: Cursor
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Cursor Agent <agent@cursor.com>
* Use array_windows when statically sized windows were used
* Bump MSRV to 1.94
* fix in gpu code
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
