mirror of
https://github.com/qdrant/qdrant.git
synced 2026-07-26 12:41:04 -05:00
ad8334928d
* feat(universal_io): add storage-agnostic BitSliceStorage with read/write support` Add BitSliceStorage<S> generic over UniversalRead<u64>/UniversalWrite<u64>, providing bit-level read and write operations over u64-element storage. Read operations (S: UniversalRead<u64>): - read_all: returns Cow<BitSlice> (zero-copy for mmap, owned for others) - get_bit: single bit read via u64 element fetch - read_bit_range: arbitrary bit range read Write operations (S: UniversalWrite<u64>): - set_bit / replace_bit: single bit write (skips write if unchanged) - write_bit_range: arbitrary bit range write from BitSlice source - fill_bit_range: fill range with a value - set_bits_batch: batch individual bit updates coalesced by element - flusher: flush underlying storage * refactor: replace MmapBitSlice with BitSliceStorage in MmapBitSliceBufferedUpdateWrapper Migrate all deleted-flag bitslice storage from the legacy MmapBitSlice (Deref-based mmap wrapper) to BitSliceStorage<MmapUniversal<u64>> (storage-agnostic universal IO backend). Updated consumers: - MmapMapIndex - MmapNumericIndex - MmapGeoMapIndex - MmapInvertedIndex (fulltext) - ImmutableIdTracker - Benchmark Additionally: - Add BitSliceStorage::create(path, num_bits) to encapsulate file creation + sizing + open (eliminates duplicated size math across 5 call sites) - Add MmapBitSliceStorage type alias for BitSliceStorage<MmapUniversal<u64>> - Add count_ones() convenience method - Use set_bits_batch() in wrapper flusher and all build paths instead of per-bit set_bit() loops (coalesces u64 read-modify-writes) - Fix silent error swallowing in wrapper get(): log + debug_assert on I/O errors instead of .ok().flatten() - Remove dead bitmap_mmap_size function from immutable_id_tracker * use bitvec's approach for offset * less api * misc improvements * refactor write method * move to segment crate * handle creation of file outside of StoredBitSlice logic * fix codespell * document bitwise calculations * coalesce more updates into a single write, batch all writes * use native `extend_from_bitslice` instead of iterating. * clarity refactor * clippyyy * use `u64` as BitStore everywhere * assume iterator is sorted * only use chunk_by for generating runs * fix update wrapper flusher * review fixes * larger set bits batch test * remove reintroduced file * oops, fix new test --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: generall <andrey@vasnetsov.com>
23 lines
739 B
TOML
23 lines
739 B
TOML
# You can run the following command in the repository root:
|
|
# uv tool run codespell --toml tools/codespell.toml
|
|
|
|
[tool.codespell]
|
|
|
|
skip = [
|
|
"**/stop_words/*.rs",
|
|
"./lib/segment/src/index/field_index/full_text_index/tokenizers/ascii_folding.rs", # contains character mappings
|
|
"./static",
|
|
"./tests/e2e_tests/test_data/cert/key.pem",
|
|
"openapi.json", # openapi.json is typically a released version and could carry old typos
|
|
"target",
|
|
]
|
|
|
|
ignore-words-list = [
|
|
"ABD", # Geohash example
|
|
"INDX", # bitvec crate const
|
|
"generall", # The boss
|
|
"hel", # Query token
|
|
"mmapped", # Memory mapped
|
|
"ser", # Serde package
|
|
]
|