mirror of
https://github.com/qdrant/qdrant.git
synced 2026-07-29 14:11:32 -05:00
c196d2eb1a
* Benches: use SmallRng instead of ChaCha12-based generators All benchmarks used StdRng or rand::rng() (ThreadRng), both backed by the ChaCha12 block cipher in rand 0.10. Benchmarks do not need crypto-strength randomness, and several draw random values inside the timed closure, so cipher work was included in the measurement itself. Switch every bench target to SmallRng (Xoshiro256++), and key the HNSW graph cache and sparse index cache by RNG algorithm so stale caches built from the old generator are not reused against newly generated vectors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Benches: replace free-function rand::random with local SmallRng Addresses review: rand::random draws from the thread RNG (ChaCha12), including inside the timed loop of the pq score benchmark. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
gridstore
New storage for variable-sized values using mmap.
Design
- IDs are sequential integers, starting at 0.
- The storage is divided into file pages of fixed size (32MB).
- Data can be written and read across multiple pages.
- Those pages are mapped into memory using mmap.
- Data units are blocks of fixed size (128 bytes).
- Values span an integer number of contiguous blocks.
- Values are compressed with lz4
- Each block is mapped to a bit in the bitmask.
- A region is a fixed number of contiguous blocks.
- Gaps of free blocks in each region are tracked in a file.
- Deletes mark the block as deleted (in-memory) & updates their region
- Updates:
- not done in place, always a new value is inserted
- calculation of the new regions gaps is done on the fly
- the tracker is updated in-memory, only persisted on flush
- Supports multiple threads reading and single thread writing
- One file per page, one file for tracker, one file for bitmask, and one file for gaps.
TODOs
- dictionary compression to optimize payload key repetition
- validate the usage with a block storage via HTTP range requests