mirror of
https://github.com/qdrant/qdrant.git
synced 2026-08-06 01:50:57 -05:00
6edf837e22
* fix: evict page cache of files that are mapped twice `clear_cache()` on a `memory: cold` vector storage was a silent no-op: after optimization the whole storage stayed resident in the page cache. `MmapFile` opened with `need_sequential` holds two mappings of the same file (`MADV_RANDOM` + `MADV_SEQUENTIAL`), and `MADV_PAGEOUT` skips any page carrying more than one page-table reference. Any page faulted through both mappings was therefore never reclaimed, no matter which mapping was advised. Quantization is one way to get there: it reads the raw vectors through the sequential mapping while `populate_vector_storages()` populated the random one, so with quantization enabled `matrix.dat` stayed 100% cached after the build, and without it the same build evicted down to ~1%. `POSIX_FADV_DONTNEED` alone does not help either, as it skips pages with any page-table reference. So zap the page tables of both mappings first (`MADV_DONTNEED` on a shared file mapping only drops the PTEs; the data stays in the page cache and refaults on the next access) and then evict through the file. Dirty pages are still kept, exactly as before — `MADV_PAGEOUT` did not write back filesystem pages either — so callers that flush first, like `SegmentBuilder::build`, get a complete eviction. Also affects the payload storage (gridstore pages), the disk id tracker reader and quantized multivector offsets, which open with `need_sequential` too. Measured on a 200k x 256 build with quantization: `matrix.dat` 100% -> 0.0% resident, whole segment 67% -> 2.4%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: satisfy clippy::cast_lossless in the eviction test Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * MADV_DONTNEED should not be safe * Document why Madviseable::clear_cache might not be enough --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: xzfc <xzfcpw@gmail.com>