Snapshot endpoints used raw `web::Path<String>` / `web::Path<(String, ...)>`
instead of validated path types, allowing invalid collection names (including
NUL bytes) to bypass `validate_collection_name` checks and reach the
filesystem, causing 500 errors.
Replace all raw path extractors with `valid::Path<CollectionPath>` and new
validated structs (`CollectionSnapshotPath`, `CollectionShardPath`,
`CollectionShardSnapshotPath`) that enforce legacy collection-name validation,
consistent with the rest of the API.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* Don't insert deferred points into sparse index
# Conflicts:
# lib/segment/src/segment/tests.rs
# lib/segment/src/segment_constructor/segment_constructor_base.rs
# Conflicts:
# lib/segment/src/segment_constructor/segment_constructor_base.rs
* Clippy
* Assert consistency of deferred_internal_id var
* Return before SparseVector conversion in case of deferred point
* Use debug_assert instead
* Add `#[must_use]` to RAII guard and task handle types
These types rely on being held for a scope — dropping them immediately is
almost always a bug:
- `StoppingGuard`: sets `is_stopped` flag on drop
- `UpdateGuard`: decrements update counter on drop
- `UpdatesGuard`: releases mutex preventing concurrent updates on drop
- `ClockGuard`: marks clock as inactive on drop
- `IsAliveGuard`: releases liveness lock on drop
- `ScopeTrackerGuard`: decrements scope counter on drop
- `CancellableAsyncTaskHandle`: detaches task on drop
- `StoppableTaskHandle`: may abort task on drop (AbortOnDropHandle)
Made-with: Cursor
* Remove redundant function-level #[must_use] now covered by type
With ScopeTrackerGuard marked #[must_use] at the type level, the
function-level attributes on measure_scope(), measure(),
track_create_snapshot_request(), and count_snapshot_creation() are
redundant and trigger clippy::double_must_use.
Made-with: Cursor
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
* Add REST API for reading audit logs across the cluster
Introduces a new `GET /audit/logs` endpoint that retrieves audit log
entries from all peers in the cluster. The API supports filtering by
time range (time_from/time_to), dynamic key=value field filters, and
a built-in limit parameter to prevent reading too many logs at once.
- Add `audit_reader` module in storage crate for efficient file-based
log retrieval, selecting only files whose date range overlaps the
query window
- Add `GetAuditLog` internal gRPC RPC for cross-peer log retrieval
- Add `GET /audit/logs` REST endpoint restricted to management access
- Aggregate and sort results from all peers by timestamp (newest first)
Made-with: Cursor
* [AI] Update filter_files_by_time_range make it aware of the log rotation granulatity (either hourly or daily)
* [AI] For AuditLogParams, try to use DateTime types natively into serde, instead of manual parsing
* [AI] Refactor API into separate file, propagate timeout, use spawn-blocking
* [AI] introduce cancellation token
* [AI] move timestamp to constant
* small manual fixes
* review fixes part 1
* review: switch to POST instead of GET
* [AI] review: sorting update
* [AI] use strict typing
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Refactor: `SearchSegmentEntry` for search ops
* Move `has_deferrend_points_method`
to `SearchSegmentEntry`
* Reorg imports
* Renamings per review
* fixup! Renamings per review
* `StorageSegmentEntry` trait
It contains methods dealing with storage and syncronization
* Move index methods from `ReadSegmentEntry`
to `SegmentEntry`
* Add `_concurrent` suffix to some methods
* move field index methods into NonAppendableSegmentEntry
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Enable async_scorer in integration tests
Enable `async_scorer=true` for all openapi and shell-based integration
tests by setting the environment variable in `integration-tests.sh`.
This exercises the io_uring vector storage path (on Linux) across all
vector configurations tested there, with graceful fallback if io_uring
is unavailable on the CI runner.
Made-with: Cursor
* Scope async_scorer to test_points_search consensus test only
Revert the blanket async_scorer=true from integration-tests.sh to keep
the normal mmap path covered. Instead, enable it via extra_env in just
the test_points_search consensus test, so we exercise the io_uring path
without losing coverage of the non-io_uring path on Linux.
Made-with: Cursor
* Parametrize test_points_search for sync/async scorer modes
Use pytest.mark.parametrize to run test_points_search twice: once with
the default scorer (sync_scorer) and once with async_scorer=true. This
exercises both the normal mmap and io_uring paths in a single test.
Made-with: Cursor
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
* Test all shard transfer methods for deferred points
* WAL delta carries deferred points
* add point count assertions
- snapshot: asserts target_visible == source_visible — deferred state preserved exactly
- stream_records: asserts target_visible >= source_visible — points may not be deferred on target
* Adjust debug logline's format of
ShardTransferRestart to match with ShardTransfer
* Convert into shardtransfer
* Deconstruct ShardTransferRestart struct
* [ai] Allow peer to bootstrap with existing URI if peer has no shards
Prompt:
I have a Qdrant cluster in distributed mode with some nodes registered
in consensus. If I bootstrap a new node with an URL that is already used
it is rejected and an error is returned. I would like to change this
behavior and allow this to happen. This would effectively replace a peer
because internally we'd drop the existing peer first, and then we'd add
the new peer so we can reuse the same peer URL. We must still reject
bootstrapping with the same URL if the peer that used the URL before us
still has any shards on it.
* [ai] Add test to assert new behavior, can rejoin if empty
Prompt:
Add two tests to assert the new behavior.
The first test should:
- create a cluster
- create a collection
- bootstrap a new peer
- kill and delete the local data for this peer without removing it from consensus
- bootstrap a new peer but reuse the URI of the peer we just killed to rejoin
- bootstrapping is expected to succeed
The second test should:
- create a cluster
- create a collection
- bootstrap a new peer but reuse the URI of the last node
- bootstrapping is expected to fail because the node being replaced has data on it
* [ai] Attempt to fix new tests
* Reformat
* [ai] Fix deadlock when rejoining with same peer URI
* [ai] Add test to ensure existing peer stops consensus on replace
* delete_points_by_filter with deferred
* versions check
* tests
* Check for deferred points in first loop
* fix after rebase
* refactor
* fix tests
* fix build after rebase
* refactor
* fix test
* refactor
* fix after rebase
* refactor
---------
Co-authored-by: timvisee <tim@visee.me>
* UIO traits: require Sized
I don't think we ever going to add unsized implementation. So, lets,
require it on the whole trait rather than on individual methods.
* Move `AccessPattern` to the `common` package
Was: segment::vector_storage::vector_storage_base::AccessPattern
Now: common::generic_consts::AccessPattern
* UIO traits: replace `bool` with `AccessPattern`
Instead of `IdTracker` to have `iter_*` methods, it now has a `point_mappings()` method, returning a guard to `PointMapping` value. It prepares for moving the point_mappings under a lock to allow parallel searches and deletions.
Also, an incorrect unsafe in `Segment::iter_points` is replaced by a safe version with `self_cell`.
* feat(universal_io): add storage-agnostic BitSliceStorage with read/write support`
Add BitSliceStorage<S> generic over
UniversalRead<u64>/UniversalWrite<u64>,
providing bit-level read and write operations over u64-element storage.
Read operations (S: UniversalRead<u64>):
- read_all: returns Cow<BitSlice> (zero-copy for mmap, owned for others)
- get_bit: single bit read via u64 element fetch
- read_bit_range: arbitrary bit range read
Write operations (S: UniversalWrite<u64>):
- set_bit / replace_bit: single bit write (skips write if unchanged)
- write_bit_range: arbitrary bit range write from BitSlice source
- fill_bit_range: fill range with a value
- set_bits_batch: batch individual bit updates coalesced by element
- flusher: flush underlying storage
* refactor: replace MmapBitSlice with BitSliceStorage in MmapBitSliceBufferedUpdateWrapper
Migrate all deleted-flag bitslice storage from the legacy MmapBitSlice
(Deref-based mmap wrapper) to BitSliceStorage<MmapUniversal<u64>>
(storage-agnostic universal IO backend).
Updated consumers:
- MmapMapIndex
- MmapNumericIndex
- MmapGeoMapIndex
- MmapInvertedIndex (fulltext)
- ImmutableIdTracker
- Benchmark
Additionally:
- Add BitSliceStorage::create(path, num_bits) to encapsulate
file creation + sizing + open (eliminates duplicated size math
across 5 call sites)
- Add MmapBitSliceStorage type alias for
BitSliceStorage<MmapUniversal<u64>>
- Add count_ones() convenience method
- Use set_bits_batch() in wrapper flusher and all build paths
instead of per-bit set_bit() loops (coalesces u64 read-modify-writes)
- Fix silent error swallowing in wrapper get(): log + debug_assert
on I/O errors instead of .ok().flatten()
- Remove dead bitmap_mmap_size function from immutable_id_tracker
* use bitvec's approach for offset
* less api
* misc improvements
* refactor write method
* move to segment crate
* handle creation of file outside of StoredBitSlice logic
* fix codespell
* document bitwise calculations
* coalesce more updates into a single write, batch all writes
* use native `extend_from_bitslice` instead of iterating.
* clarity refactor
* clippyyy
* use `u64` as BitStore everywhere
* assume iterator is sorted
* only use chunk_by for generating runs
* fix update wrapper flusher
* review fixes
* larger set bits batch test
* remove reintroduced file
* oops, fix new test
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Add deferred point count to UpdateQueueInfo
* Hide if prevent_unoptimized is false
* Clippy
* Openapi
* Iterate over appendable segments only
* Only calculate deferred point count if prevent_unoptimized is true
* Rebase fixes
* Codespell
* Don't account for deferred points in some places
# Conflicts:
# lib/collection/src/shards/local_shard/scroll.rs
* Add in QueryContext
* Cover more places
* Coderabbit review remarks
* Properly count amount of deleted deferred points (#8386)
* Properly count amount of deleted deferred points
* Prevent double-counting of the same point
* Remove hints to estimations
* Properly handle counts in ProxySegment
* Add tests and fix deleted point count issue
* Adjusts tests + fix issues
* Separte fields for deferred points in telemetry (SegmentInfo)
* Remove deferred_points_count()
* openapi
* Fix test by manually calculating visible points
* Adjust ProxySegment test to revertion of SegmentInfo
* Throw error if collection was not found in telemetry (e2e Test)
* Update lib/segment/src/segment/segment_ops.rs
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Make new fields in API optional
* Don't take range if no deferred point exist
* Review remarks
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
