mirror of
https://github.com/qdrant/qdrant.git
synced 2026-08-05 01:20:53 -05:00
30f00477c6
* Add disk-resident id tracker Introduce a disk-resident id tracker family that keeps the point-id mapping on disk instead of loading it into RAM, so resident memory no longer scales with total point count. This removes the last component of a segment that forces a full RAM load, enabling object-storage / edge followers and cutting RAM for regular deployments that opt in. New on-disk mapping format (`disk_id_tracker/on_disk_format.rs`): random-access `id_tracker.i2e` (internal->external) + `id_tracker.e2i` (external->internal as sorted num/uuid runs with a resident sparse block index). `id_tracker.versions` and `id_tracker.deleted` are reused unchanged. Trackers (sharing a lazy `DiskMappingReader` core and a `DiskMappingsSource` trait): - `DiskIdTracker` — writable, deletion-only; a new `IdTrackerEnum` variant used by regular segments, keeping deleted+versions resident and the mapping on disk. - `ReadOnlyDiskIdTracker` — read-only live-reload mirror for followers; per-point `get_bit` deletion checks on read-by-id, full deleted set materialized lazily. Selection: created when the `serverless_compatible` feature flag is set (in `segment_builder`); loaded by attempting each format in `ReadOnlyIdTrackerEnum:: detect_and_load` (no per-file `exists` round-trips) and by file-presence detection unified in `IdTrackerFormat`. `PointMappingsRefEnum` is generic over the read backend so the disk variant is a concrete `DiskMappingsRef` (no trait object). The `DiskMappingsSource` read surface returns `OperationResult`; errors are only swallowed at the infallible `IdTrackerRead` boundary. `ImmutableIdTracker` is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style: apply rustfmt Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix clippy: collapse nested if in DiskIdTracker::drop Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Align on-disk id tracker headers and sections to 16 bytes Pad the i2e header to 32 bytes and the e2i header to 48 bytes, and zero-pad between e2i sections so every section starts on a 16-byte boundary. This keeps the files mmap+transmute-friendly: the u128 arrays (i2e slots, e2i uuid sparse index) need 16-byte alignment in Rust. Add on_disk_sections_are_aligned test pinning the invariant and the store/parse padding agreement via exact file-length checks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>