* Don't insert deferred points into sparse index
# Conflicts:
# lib/segment/src/segment/tests.rs
# lib/segment/src/segment_constructor/segment_constructor_base.rs
# Conflicts:
# lib/segment/src/segment_constructor/segment_constructor_base.rs
* Clippy
* Assert consistency of deferred_internal_id var
* Return before SparseVector conversion in case of deferred point
* Use debug_assert instead
* Add `#[must_use]` to RAII guard and task handle types
These types rely on being held for a scope — dropping them immediately is
almost always a bug:
- `StoppingGuard`: sets `is_stopped` flag on drop
- `UpdateGuard`: decrements update counter on drop
- `UpdatesGuard`: releases mutex preventing concurrent updates on drop
- `ClockGuard`: marks clock as inactive on drop
- `IsAliveGuard`: releases liveness lock on drop
- `ScopeTrackerGuard`: decrements scope counter on drop
- `CancellableAsyncTaskHandle`: detaches task on drop
- `StoppableTaskHandle`: may abort task on drop (AbortOnDropHandle)
Made-with: Cursor
* Remove redundant function-level #[must_use] now covered by type
With ScopeTrackerGuard marked #[must_use] at the type level, the
function-level attributes on measure_scope(), measure(),
track_create_snapshot_request(), and count_snapshot_creation() are
redundant and trigger clippy::double_must_use.
Made-with: Cursor
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
* Add REST API for reading audit logs across the cluster
Introduces a new `GET /audit/logs` endpoint that retrieves audit log
entries from all peers in the cluster. The API supports filtering by
time range (time_from/time_to), dynamic key=value field filters, and
a built-in limit parameter to prevent reading too many logs at once.
- Add `audit_reader` module in storage crate for efficient file-based
log retrieval, selecting only files whose date range overlaps the
query window
- Add `GetAuditLog` internal gRPC RPC for cross-peer log retrieval
- Add `GET /audit/logs` REST endpoint restricted to management access
- Aggregate and sort results from all peers by timestamp (newest first)
Made-with: Cursor
* [AI] Update filter_files_by_time_range make it aware of the log rotation granulatity (either hourly or daily)
* [AI] For AuditLogParams, try to use DateTime types natively into serde, instead of manual parsing
* [AI] Refactor API into separate file, propagate timeout, use spawn-blocking
* [AI] introduce cancellation token
* [AI] move timestamp to constant
* small manual fixes
* review fixes part 1
* review: switch to POST instead of GET
* [AI] review: sorting update
* [AI] use strict typing
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Refactor: `SearchSegmentEntry` for search ops
* Move `has_deferrend_points_method`
to `SearchSegmentEntry`
* Reorg imports
* Renamings per review
* fixup! Renamings per review
* `StorageSegmentEntry` trait
It contains methods dealing with storage and syncronization
* Move index methods from `ReadSegmentEntry`
to `SegmentEntry`
* Add `_concurrent` suffix to some methods
* move field index methods into NonAppendableSegmentEntry
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Adjust debug logline's format of
ShardTransferRestart to match with ShardTransfer
* Convert into shardtransfer
* Deconstruct ShardTransferRestart struct
* [ai] Allow peer to bootstrap with existing URI if peer has no shards
Prompt:
I have a Qdrant cluster in distributed mode with some nodes registered
in consensus. If I bootstrap a new node with an URL that is already used
it is rejected and an error is returned. I would like to change this
behavior and allow this to happen. This would effectively replace a peer
because internally we'd drop the existing peer first, and then we'd add
the new peer so we can reuse the same peer URL. We must still reject
bootstrapping with the same URL if the peer that used the URL before us
still has any shards on it.
* [ai] Add test to assert new behavior, can rejoin if empty
Prompt:
Add two tests to assert the new behavior.
The first test should:
- create a cluster
- create a collection
- bootstrap a new peer
- kill and delete the local data for this peer without removing it from consensus
- bootstrap a new peer but reuse the URI of the peer we just killed to rejoin
- bootstrapping is expected to succeed
The second test should:
- create a cluster
- create a collection
- bootstrap a new peer but reuse the URI of the last node
- bootstrapping is expected to fail because the node being replaced has data on it
* [ai] Attempt to fix new tests
* Reformat
* [ai] Fix deadlock when rejoining with same peer URI
* [ai] Add test to ensure existing peer stops consensus on replace
* delete_points_by_filter with deferred
* versions check
* tests
* Check for deferred points in first loop
* fix after rebase
* refactor
* fix tests
* fix build after rebase
* refactor
* fix test
* refactor
* fix after rebase
* refactor
---------
Co-authored-by: timvisee <tim@visee.me>
* UIO traits: require Sized
I don't think we ever going to add unsized implementation. So, lets,
require it on the whole trait rather than on individual methods.
* Move `AccessPattern` to the `common` package
Was: segment::vector_storage::vector_storage_base::AccessPattern
Now: common::generic_consts::AccessPattern
* UIO traits: replace `bool` with `AccessPattern`
Instead of `IdTracker` to have `iter_*` methods, it now has a `point_mappings()` method, returning a guard to `PointMapping` value. It prepares for moving the point_mappings under a lock to allow parallel searches and deletions.
Also, an incorrect unsafe in `Segment::iter_points` is replaced by a safe version with `self_cell`.
* feat(universal_io): add storage-agnostic BitSliceStorage with read/write support`
Add BitSliceStorage<S> generic over
UniversalRead<u64>/UniversalWrite<u64>,
providing bit-level read and write operations over u64-element storage.
Read operations (S: UniversalRead<u64>):
- read_all: returns Cow<BitSlice> (zero-copy for mmap, owned for others)
- get_bit: single bit read via u64 element fetch
- read_bit_range: arbitrary bit range read
Write operations (S: UniversalWrite<u64>):
- set_bit / replace_bit: single bit write (skips write if unchanged)
- write_bit_range: arbitrary bit range write from BitSlice source
- fill_bit_range: fill range with a value
- set_bits_batch: batch individual bit updates coalesced by element
- flusher: flush underlying storage
* refactor: replace MmapBitSlice with BitSliceStorage in MmapBitSliceBufferedUpdateWrapper
Migrate all deleted-flag bitslice storage from the legacy MmapBitSlice
(Deref-based mmap wrapper) to BitSliceStorage<MmapUniversal<u64>>
(storage-agnostic universal IO backend).
Updated consumers:
- MmapMapIndex
- MmapNumericIndex
- MmapGeoMapIndex
- MmapInvertedIndex (fulltext)
- ImmutableIdTracker
- Benchmark
Additionally:
- Add BitSliceStorage::create(path, num_bits) to encapsulate
file creation + sizing + open (eliminates duplicated size math
across 5 call sites)
- Add MmapBitSliceStorage type alias for
BitSliceStorage<MmapUniversal<u64>>
- Add count_ones() convenience method
- Use set_bits_batch() in wrapper flusher and all build paths
instead of per-bit set_bit() loops (coalesces u64 read-modify-writes)
- Fix silent error swallowing in wrapper get(): log + debug_assert
on I/O errors instead of .ok().flatten()
- Remove dead bitmap_mmap_size function from immutable_id_tracker
* use bitvec's approach for offset
* less api
* misc improvements
* refactor write method
* move to segment crate
* handle creation of file outside of StoredBitSlice logic
* fix codespell
* document bitwise calculations
* coalesce more updates into a single write, batch all writes
* use native `extend_from_bitslice` instead of iterating.
* clarity refactor
* clippyyy
* use `u64` as BitStore everywhere
* assume iterator is sorted
* only use chunk_by for generating runs
* fix update wrapper flusher
* review fixes
* larger set bits batch test
* remove reintroduced file
* oops, fix new test
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Add deferred point count to UpdateQueueInfo
* Hide if prevent_unoptimized is false
* Clippy
* Openapi
* Iterate over appendable segments only
* Only calculate deferred point count if prevent_unoptimized is true
* Rebase fixes
* Codespell
* Don't account for deferred points in some places
# Conflicts:
# lib/collection/src/shards/local_shard/scroll.rs
* Add in QueryContext
* Cover more places
* Coderabbit review remarks
* Properly count amount of deleted deferred points (#8386)
* Properly count amount of deleted deferred points
* Prevent double-counting of the same point
* Remove hints to estimations
* Properly handle counts in ProxySegment
* Add tests and fix deleted point count issue
* Adjusts tests + fix issues
* Separte fields for deferred points in telemetry (SegmentInfo)
* Remove deferred_points_count()
* openapi
* Fix test by manually calculating visible points
* Adjust ProxySegment test to revertion of SegmentInfo
* Throw error if collection was not found in telemetry (e2e Test)
* Update lib/segment/src/segment/segment_ops.rs
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Make new fields in API optional
* Don't take range if no deferred point exist
* Review remarks
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
In two places we collected an iterator of Results into a
Result<Vec<_>> only to immediately iterate the Vec again.
Use `itertools::process_results` instead to pipe the unwrapped
items straight into the downstream iterator chain, removing
the unnecessary heap allocation.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* refactor(edge): split EdgeShard load into new() and load()
- new(path, config): create edge shard only when path has no existing segments
- load(path, config?): load from existing files; config optional (load from
edge_config.json or infer from segments)
- Helpers: has_existing_segments, ensure_dirs_and_open_wal, resolve_initial_config,
load_segments, ensure_appendable_segment
- Python: EdgeShard.create_new(path, config); __init__ still calls load()
- Examples use new() for creation and load(path, None) for reopen
- Error instead of panic on config mismatch; explicit load/create in Python
- fix: use OptimizerThresholds.deferred_internal_id for dev compatibility
Made-with: Cursor
* rollback threshold changes
* fix(edge): address dancixx review comments
- Persist inferred config in load() when edge_config.json does not exist
(SaveOnDisk::new only saves when file exists)
- Python: add #[new] delegating to load() for backward compatibility
- qdrant_edge.pyi: Optional return types for deleted_threshold,
vacuum_min_vector_number, default_segment_number; add __init__ doc
Made-with: Cursor
* Revert "fix(edge): address dancixx review comments"
This reverts commit 370e21a6c74c41210e1658f89814395cb86ed81c.
* fix: optional returns
* fix: save config it is not exist
* fix: save data on disk always
* fix: python examples
* chore: remove edge.close()
* fix: wal lock
* fix: rename of EdgeShardConfig -> EdgeConfig
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* refactor(edge): let edge re-export segment/shard/sparse
* chore(ast-grep-rules): handle path like `::shard::blah::Blah`
This commit was required during the refactoring. It turns out we don't
need it right now, but it might be useful in the future. Or not.
* refactor(edge-rs): Rehaul public API
