mirror of
https://github.com/qdrant/qdrant.git
synced 2026-08-05 09:31:05 -05:00
68146a7103
* build(deps): bump lz4_flex from 0.13.1 to 0.14.0 Bumps [lz4_flex](https://github.com/pseitz/lz4_flex) from 0.13.1 to 0.14.0. - [Release notes](https://github.com/pseitz/lz4_flex/releases) - [Changelog](https://github.com/PSeitz/lz4_flex/blob/main/CHANGELOG.md) - [Commits](https://github.com/pseitz/lz4_flex/compare/0.13.1...0.14.0) --- updated-dependencies: - dependency-name: lz4_flex dependency-version: 0.14.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * fix: enable lz4_flex alloc feature for Vec-returning APIs lz4_flex 0.14.0 gates compress_prepend_size/decompress_size_prepended behind the new alloc feature when default-features are disabled. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
gridstore
New storage for variable-sized values using mmap.
Design
- IDs are sequential integers, starting at 0.
- The storage is divided into file pages of fixed size (32MB).
- Data can be written and read across multiple pages.
- Those pages are mapped into memory using mmap.
- Data units are blocks of fixed size (128 bytes).
- Values span an integer number of contiguous blocks.
- Values are compressed with lz4
- Each block is mapped to a bit in the bitmask.
- A region is a fixed number of contiguous blocks.
- Gaps of free blocks in each region are tracked in a file.
- Deletes mark the block as deleted (in-memory) & updates their region
- Updates:
- not done in place, always a new value is inserted
- calculation of the new regions gaps is done on the fly
- the tracker is updated in-memory, only persisted on flush
- Supports multiple threads reading and single thread writing
- One file per page, one file for tracker, one file for bitmask, and one file for gaps.
TODOs
- dictionary compression to optimize payload key repetition
- validate the usage with a block storage via HTTP range requests