mirror of
https://github.com/qdrant/qdrant.git
synced 2026-07-23 11:11:00 -05:00
9bcef50544
* feat(universal_io): add storage-agnostic BitSliceStorage with read/write support` Add BitSliceStorage<S> generic over UniversalRead<u64>/UniversalWrite<u64>, providing bit-level read and write operations over u64-element storage. Read operations (S: UniversalRead<u64>): - read_all: returns Cow<BitSlice> (zero-copy for mmap, owned for others) - get_bit: single bit read via u64 element fetch - read_bit_range: arbitrary bit range read Write operations (S: UniversalWrite<u64>): - set_bit / replace_bit: single bit write (skips write if unchanged) - write_bit_range: arbitrary bit range write from BitSlice source - fill_bit_range: fill range with a value - set_bits_batch: batch individual bit updates coalesced by element - flusher: flush underlying storage * refactor: replace MmapBitSlice with BitSliceStorage in MmapBitSliceBufferedUpdateWrapper Migrate all deleted-flag bitslice storage from the legacy MmapBitSlice (Deref-based mmap wrapper) to BitSliceStorage<MmapUniversal<u64>> (storage-agnostic universal IO backend). Updated consumers: - MmapMapIndex - MmapNumericIndex - MmapGeoMapIndex - MmapInvertedIndex (fulltext) - ImmutableIdTracker - Benchmark Additionally: - Add BitSliceStorage::create(path, num_bits) to encapsulate file creation + sizing + open (eliminates duplicated size math across 5 call sites) - Add MmapBitSliceStorage type alias for BitSliceStorage<MmapUniversal<u64>> - Add count_ones() convenience method - Use set_bits_batch() in wrapper flusher and all build paths instead of per-bit set_bit() loops (coalesces u64 read-modify-writes) - Fix silent error swallowing in wrapper get(): log + debug_assert on I/O errors instead of .ok().flatten() - Remove dead bitmap_mmap_size function from immutable_id_tracker * use bitvec's approach for offset * less api * misc improvements * refactor write method * move to segment crate * handle creation of file outside of StoredBitSlice logic * fix codespell * document bitwise calculations * coalesce more updates into a single write, batch all writes * use native `extend_from_bitslice` instead of iterating. * clarity refactor * clippyyy * use `u64` as BitStore everywhere * assume iterator is sorted * only use chunk_by for generating runs * fix update wrapper flusher * review fixes * larger set bits batch test * remove reintroduced file * oops, fix new test --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: generall <andrey@vasnetsov.com>