mirror of
https://github.com/qdrant/qdrant.git
synced 2026-08-05 01:20:53 -05:00
c3ff8ee44b
* Add optional block-index sidecar to accelerate on-disk binary searches Binary search directly over an unpopulated mmap costs O(log n) random reads scattered across the whole file — one page fault (or remote range read) per probe on high-latency storage. Introduce SortedBlockIndex: an optional sidecar file storing the first element of every 16KiB block of a sorted on-disk array, read whole into RAM at open. A lookup becomes an in-RAM partition_point over the block firsts plus a single contiguous read of one block, bisected in RAM. Wire it into the three remaining scatter-probe sites: - on-disk numeric index: binary_search_pairs over data.bin - on-disk geo index: counts_of_hash over counts_per_hash.bin - on-disk geo index: all_points start boundary over points_map.bin The sidecar is backward and forward compatible: absent (old segments) or failing validation, readers fall back to the existing plain binary search; old code simply ignores the extra file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Validate block-index sidecar snapshot recovery and preopen coverage - snapshot round-trip test (Regular + Streamable): the sidecars written by the on-disk numeric and geo indexes are collected via files() into the snapshot tar and restored byte-for-byte, with correct query results on the reloaded segment - preopen tests for both indexes: after schedule_prefetch, open loads the sidecar from the CachedFs prefetch pool even when the file is unlinked from disk; an absent sidecar neither fails preopen nor open (fallback) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>