mirror of
https://github.com/qdrant/qdrant.git
synced 2026-08-01 07:30:54 -05:00
5a899b74de
* Add mincore-based memory stats to MmapFile Add `resident_bytes()`, `disk_bytes()`, and `probe_memory_stats()` methods to `MmapFile` for measuring page cache residency via `mincore(2)`. This is the foundation for per-collection memory usage reporting. Also extract `page_size()` as a public function in `mmap::advice`, replacing the internal `PAGE_SIZE_MASK` with a direct page size cache. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * [AI] introduce trait for reporting memory usage per component * [AI] memory reporter implementation for vector storage * [AI] implement MemoryReporter for QuantizedVectors * [AI] implement MemoryReporter for VectorIndexEnum * Implement MemoryReporter for IdTrackerEnum with RAM estimation Add ram_usage_bytes() to all ID tracker types and their data structures: - PointMappings, CompressedPointMappings, CompressedVersions, CompressedInternalToExternal, CompressedExternalToInternal - MutableIdTracker, ImmutableIdTracker, InMemoryIdTracker All ID trackers load their data into RAM (none use mmap for working data). Files are reported as OnDisk (persistence only), actual RAM footprint is reported via extra_ram_bytes. Uses struct destructuring to ensure new fields trigger compile errors if not accounted for. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * [AI] implement MemoryReporter for PayloadStorageEnum and adjust FileStorageIntent * [AI] implement MemoryReporter for PayloadStorageEnum and adjust FileStorageIntent * [AI] implement MemoryReporter for payload indexes: in-ram structures memory consumtion computation + caching * [AI] implement MemoryReporter for payload indexes: in-ram structures memory consumtion computation + caching * [AI] segment-level memory usage report * [AI] Block 3: Aggregation Layer and Data Model + internal api for remote shard * [AI] REST API handler * fmt * [AI] clippy fixes * [AI] macos fix + proxy segment fix * [AI] make text index estimation a bit more correct * fix is_on_disk reporting for dense_vector_storage * fix after rebase * [AI] deep account for quantized vectors RAM usage + unify chunk size + shring volatile storage after load * remove debug log * cache in test * make manual test easier to run * rollback chunk size diff, but keep it for test only * review fixes * Use exhaustive match * Use div_ceil on bits everywhere It does not seem to be strictly necessary because the number of bits should already be a multiple of the used container size bytes. Still it's good practice to be careful with this calculation. * Improve heap size bytes for encoded product quantization vectors * Include vector stats for binary quantized vectors * In volatile chunked vectors, include heap allocated vector * Include rest of heap allocated structures for mutable map index * In mutable geo index, the hash map is also heap allocated * Update tests/manual/test_memory_reporting.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: timvisee <tim@visee.me> Co-authored-by: Tim Visée <tim+github@visee.me> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>