mirror of
https://github.com/qdrant/qdrant.git
synced 2026-08-06 10:00:58 -05:00
45ae3e048b
* wip: chunked mmap * Fix typo * insert and get methods * dynamic bitvec * clippy * wip: vector storage * wip: fmt * wip: mmap chunks * wip: mmap problems * Share transmuted mutable reference over mmap * option to enable appendable mmap vectors * fmt * rename storage status file * update tests * fix get deleted value range * add recovery to vector storage tests * add flush to tests * fix transmute from immutable to mutable * make transmuted pointer private * remove unused unsafe functions * force WAL flush if wait=true * move wal flush into updater thread * remove flush from update api * Minimize pub visibility for specialized/dangerous functions * Allocate vector with predefined capacity * Inline format parameters * Assert we have multiple chunks while testing, test is useless otherwise * Remove unnecessary scope * Remove unnecessary dereference * Random bool has 0.5 as standard distribution, use iter::repeat_with * Replace RemovableMmap::new with Default derive * Rename len to num_flags * Use Option replace as it is convention alongside take * Add FileId enum to replace error prone manual ID rotating * Use debug_assert_eq where applicable * Refactor drop and set to replace * Change default chunk size for chunked mmap vectors to 32MB This change is made as per GitHub review, because allocating a few storages with 128MB would take a significant amount of time and storage. See: https://github.com/qdrant/qdrant/pull/1838#discussion_r1187215475 * Replace for-loops with iterators * Draft: add typed mmap to improve code safety (#1860) * Add typed mmap * Replace some crude mmap usages with typed mmap * Use typed mmap for deleted flags * Simplify dynamic mmap flags a lot with new typed mmap, remove flags option * Reformat * Remove old mmap functions that are now unused * Reimplement mmap locking for mmap_vectors * Add MmapBitSlice tests * Replace MmapChunk with new typed mmap * Update docs * Clean-up * Disable alignment assertions on Windows for now * Rename mmap lock to mlock to prevent confusion with lockable types * one more small test * Some review fixes * Add aliasing note * Add basic error handling in typed mmap constructors * Use typed mmap error handling throughout project * Move mmap type module to common * Fix transmute functions being unsound See https://github.com/qdrant/qdrant/pull/1860#discussion_r1188593854 --------- Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> --------- Co-authored-by: timvisee <tim@visee.me> Co-authored-by: Tim Visée <tim+github@visee.me>