mirror of
https://github.com/qdrant/qdrant.git
synced 2026-09-29 09:27:53 -05:00
xzfc
66d0c36009
* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
gridstore
New storage for variable-sized values using mmap.
Design
- IDs are sequential integers, starting at 0.
- The storage is divided into file pages of fixed size (32MB).
- Data can be written and read across multiple pages.
- Those pages are mapped into memory using mmap.
- Data units are blocks of fixed size (128 bytes).
- Values span an integer number of contiguous blocks.
- Values are compressed with lz4
- Each block is mapped to a bit in the bitmask.
- A region is a fixed number of contiguous blocks.
- Gaps of free blocks in each region are tracked in a file.
- Deletes mark the block as deleted (in-memory) & updates their region
- Updates:
- not done in place, always a new value is inserted
- calculation of the new regions gaps is done on the fly
- the tracker is updated in-memory, only persisted on flush
- Supports multiple threads reading and single thread writing
- One file per page, one file for tracker, one file for bitmask, and one file for gaps.
TODOs
- dictionary compression to optimize payload key repetition
- validate the usage with a block storage via HTTP range requests