* use pageout to clear mmap cache
* Also clear cache of deleted flags in mmap dense vector storage
* Add reference to madvise man pages for probe logic
* use deconstruct
---------
Co-authored-by: timvisee <tim@visee.me>
* [ai] Replace manual into mappings with Into::into
* Reformat
* [ai] Use implicit .iter
* Don't iterate over keys too
* [ai] Replace unwrap_or
* Reformat
* [ai] Use as_deref and then_some
* [ai] Use more to_string
* [ai] Use explicitly typed into conversions
* Reformat
* [ai] More explicit into conversions
* Reformat
* IoUringState: generic `RequestId`
* UniversalRead: generic `RequestId`
* Simplify `gridstore::Pages::get_page_value_ranges`
Now we don't need two separate `SmallVec`s as we can put `buffer_offset`
into `RequestId`.
* Better doc comment
* Rename `RequestId` -> `Meta`
* UIO traits: require Sized
I don't think we ever going to add unsized implementation. So, lets,
require it on the whole trait rather than on individual methods.
* Move `AccessPattern` to the `common` package
Was: segment::vector_storage::vector_storage_base::AccessPattern
Now: common::generic_consts::AccessPattern
* UIO traits: replace `bool` with `AccessPattern`
* feat(universal_io): add storage-agnostic BitSliceStorage with read/write support`
Add BitSliceStorage<S> generic over
UniversalRead<u64>/UniversalWrite<u64>,
providing bit-level read and write operations over u64-element storage.
Read operations (S: UniversalRead<u64>):
- read_all: returns Cow<BitSlice> (zero-copy for mmap, owned for others)
- get_bit: single bit read via u64 element fetch
- read_bit_range: arbitrary bit range read
Write operations (S: UniversalWrite<u64>):
- set_bit / replace_bit: single bit write (skips write if unchanged)
- write_bit_range: arbitrary bit range write from BitSlice source
- fill_bit_range: fill range with a value
- set_bits_batch: batch individual bit updates coalesced by element
- flusher: flush underlying storage
* refactor: replace MmapBitSlice with BitSliceStorage in MmapBitSliceBufferedUpdateWrapper
Migrate all deleted-flag bitslice storage from the legacy MmapBitSlice
(Deref-based mmap wrapper) to BitSliceStorage<MmapUniversal<u64>>
(storage-agnostic universal IO backend).
Updated consumers:
- MmapMapIndex
- MmapNumericIndex
- MmapGeoMapIndex
- MmapInvertedIndex (fulltext)
- ImmutableIdTracker
- Benchmark
Additionally:
- Add BitSliceStorage::create(path, num_bits) to encapsulate
file creation + sizing + open (eliminates duplicated size math
across 5 call sites)
- Add MmapBitSliceStorage type alias for
BitSliceStorage<MmapUniversal<u64>>
- Add count_ones() convenience method
- Use set_bits_batch() in wrapper flusher and all build paths
instead of per-bit set_bit() loops (coalesces u64 read-modify-writes)
- Fix silent error swallowing in wrapper get(): log + debug_assert
on I/O errors instead of .ok().flatten()
- Remove dead bitmap_mmap_size function from immutable_id_tracker
* use bitvec's approach for offset
* less api
* misc improvements
* refactor write method
* move to segment crate
* handle creation of file outside of StoredBitSlice logic
* fix codespell
* document bitwise calculations
* coalesce more updates into a single write, batch all writes
* use native `extend_from_bitslice` instead of iterating.
* clarity refactor
* clippyyy
* use `u64` as BitStore everywhere
* assume iterator is sorted
* only use chunk_by for generating runs
* fix update wrapper flusher
* review fixes
* larger set bits batch test
* remove reintroduced file
* oops, fix new test
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
Reduce test parameters across 5 areas to cut test runtime without
sacrificing coverage:
1. HNSW PQ tests: lower vector dimensionality from 131 to 64 for product
quantization variants, cutting PQ codebook training time (~31s -> ~8s)
2. HNSW index build: use 2 threads instead of 1 for index construction;
the tests only check accuracy above a threshold so determinism is not
required
3. Search attempts: reduce from 10 to 5 query vectors per test
4. Rescoring: skip the expensive re-upsert-all-as-zeros rescoring check
for PQ tests (already covered by scalar quantization variants)
5. Near-miss speedups:
- WAL: reduce QuickCheck iterations from 100 to 50
- Gridstore: halve operation counts, drop 64-byte block size case,
reduce proptest cases for gap search
- Continuous snapshot: reduce timeout from 20s to 10s
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* lib/edge/publish/cargo: improve target directory discovery
Don't hardcode workspace root, ask cargo instead.
* Fix warning in qdrant-edge amalgamation
`cargo check` in `lib/edge/publish` produces the following warning:
warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending`
* lib/edge/publish/examples: fix compiler warnings
* refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs
To match python examples that have the same constant in `common.py`.
* Use `lib/edge/data` dir for both Python and Rust examples
Before this commit, `lib/edge/publish/data` and `lib/edge/python/data`
were used. I'd like Python and Rust to use the same prepared resources
in the future.
* ❗Add lib/edge/Justfile with recipes to build/run examples
I constantly keep forgetting how to compile and run edge stuff.
Also, this would be used in CI in subsequent commits.
* ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes
Also now we can rename `creates-io-readme.md` to `README.md` like
reasonable people.
* ❗doc: Add qdrant-edge-py README.md
Since the previous commit removed qdrant-edge-py README.md, we need to
add something in its place instead.
The new README is adapted from Rust edge README.
And the previous README wasn't good anyway, see [1], it mentions maturin
which is irrelevant for users.
[1]: https://pypi.org/project/qdrant-edge-py/0.5.0/
* GHA: Add "Setup Qdrant" composite action
We need running Qdrant instance to test edge examples on CI. We can't
use docker for this because we want to run tests on Windows and macOS.
OTOH, this action downloads binaries for all platforms provided in
https://github.com/qdrant/qdrant/releases/tag/v1.17.0
* ❗GHA: Use single worflow for python and rust edge examples
Merge workflows to reuse build cache. Also, modernize it by using `uv`,
and recently introduced `setup-qdrant` and Justfile. Also, make it run
on three platforms (only on merges).
* ❗GHA: edge-{py,rust}-release: also run examples before publishing
Use similar approach as in the previous commit. But note that this time
we build/test also on ARM Linux and Windows.
* feat(gridstore): migrate Tracker to universal IO
- Tracker<S> generic over S: UniversalRead<u8> + UniversalWrite<u8>
- Replace MmapSlice<u8> with S; use S::open, read, write, flusher, populate
- On file growth: flush, create_and_ensure_length, re-open S
- Map UniversalIoError::NotFound to tracker file missing error
- get_raw returns Result<Option<Option<ValuePointer>>>; get returns Result<Option<ValuePointer>>
- has_pointer, unset, write_pending return Result where needed
- flusher returns crate::gridstore::Flusher (universal_io Flusher mapped to GridstoreError)
- Type alias Tracker = Tracker<MmapUniversal<u8>> in lib.rs
- Tests use TestTracker = Tracker<MmapUniversal<u8>>; mapping_len/mmap_file_size .unwrap()
Made-with: Cursor
* refactor(gridstore): review follow-ups – generic view, From conversion, iter errors
- GridstoreView: use Tracker<S> with same S as Page (S: UniversalRead + UniversalWrite)
- Tracker: use ? and .map_err(Into::into) instead of .map_err(GridstoreError::from)
- View iter: propagate tracker read errors via Err(e) instead of silently skipping
Made-with: Cursor
* Split Tracker into read/write impl blocks; GridstoreView only requires UniversalRead
- Tracker: impl<S> for files/pointer_count, impl<S: UniversalRead<u8>> for
get/get_raw/iter_pointers/has_pointer/populate and test helpers,
impl<S: UniversalRead+UniversalWrite> for new/open/write_pending/set/unset
and other write methods (like Page).
- GridstoreView: require S: UniversalRead<u8> only so read-only views don't
need UniversalWrite.
Made-with: Cursor
* Move Tracker::open to read-only impl; add FILE_NAME/tracker_file_name to unbound impl
open() only uses S::open and storage.read(), so it belongs in impl<S: UniversalRead<u8>>.
read_config_and_tracker can thus use a read-only tracker open. FILE_NAME and
tracker_file_name moved to unbound impl so both read and write sections use them.
Made-with: Cursor
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
* [manunal] Gridstore page use universal IO
* fmt
* Apply review feedback for universal IO gridstore pages (#8230)
* Apply review feedback from PR #8223
- Use `super::Result` import in mmap.rs instead of fully-qualified `crate::universal_io::Result`
- Restructure ValuePointer destructuring in get_value and delete_value to
first match Some(pointer), then destructure separately
* Replace Either<E, GridstoreError> with E: From<GridstoreError> in Gridstore::iter
Use a trait bound instead of Either to combine callback and gridstore
errors, allowing `?` to work directly on GridstoreError. This simplifies
callers by removing Either matching and io::Error conversion workarounds.
---------
Co-authored-by: qdrant-claw <qdrant-claw@users.noreply.github.com>
---------
Co-authored-by: qdrant-claw <qdrant-claw@users.noreply.github.com>
* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
* Implement `Optional<T>` type
Define a type with the same presumed layout as `Option<T>`, but with defined behavior.
* Make `transmute_*` functions unsafe
The functions `memory::mmap_ops::transmute_*` are inherently unsafe, but
are not marked as are. Their usage is documented, but it is not always clear
if the code is correct.
* Add `CsrHeader` to resolve another unsoundness
Tuples have no defined layout.
* Enrich Gridstore model testing
* better
* no clear on Windows
* less windows
* sigh
* change strategy
* never give up
* x100 scale down for Windows
* last touch
* decrease for others too
* remove Iter test - the quadratic cost is not worth it
* Revert "remove Iter test - the quadratic cost is not worth it"
This reverts commit f0e98cb33b.
* try with limited Iter
* less logging as it is hurting crasher
* tracker.mapping_len() is slow
* hard limit good enough for now
* introduce IsAliveLock
* add tests
* rename to mark_dead
* update gridstore comment
* Update some comments
* Add must_use attribute to lock_if_alive
* Rename mark_dead to blocking_mark_dead
* make the handle take a `Weak` reference
* make dropping explicit
* use `Mutex::lock_arc` instead
* less nesting
* clippy
---------
Co-authored-by: timvisee <tim@visee.me>
* Separate set and unset lists for each Gridstore pointer update
This makes the list of pointer updates much easier to grasp. The
simplification is desired because this exact structure has been a cause
for bugs multiple times now.
* Add more aggressive debug assertions
* Use consistent terminology
* Correct removal of set
* Simplify drain function
* Add safe guard to ignore empty pointer updates
* Rework tracker getter, explicitly branch variants
* Patch transmute, require Sized type
* rename and adjust comments
- set/unset is now current/to_free
- adjusted descriptions and comments for this nomenclature too
* clippy
* clippyyy
not 4, not 2, but 3 spaces
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Gridstore decompress failure
* more logs
* more logs
* hardening
* clean
* no compression
* cleanup
* typo
* misc
* wait for flush & less logs
* add compression type to the mix
* optimize
* Fix Gridstore tracker drain and persist not working properly
* Patch existing test
* Add new test to assert buggy scenario we found
* Mention PR in test
* Add lease structure to invalidate pending flushers after wipe/clear
* We don't use bitmask as barrier anymore
* Add missing early return
* Add test
* Replace flush lease with is alive boolean we also use elsewhere
* Reimplement test, now it does fail on the old implementation
* In test, cover both new fixed and old broken flushing
* Remove old test case because it is flaky
* Fix clippy
* Fix outdated comment
* Fix delete_page not dropping sequential mmap
* In Gridstore Page, support using a single mmap
* In Gridstore Page, only open multiple mmaps if supported at runtime
* Support opening single mmap in UniversalMmapChunk
* Rename environment var, log warning when QDRANT_NO_MULTI_MMAP is set
* Support opening single mmap in MmapDenseVectors
* Remove unused result response
* Import LazyLock
* Wrap the tracker and bitmask in Arc
* Implement deferred flusher for gridstore, defer tracker writes
* Remove clone
* Dynamically adjust list of pending updates to drain what is flushed
* Return proper flusher, defer premature Gridstore flushing errors
* Use deferred Gridstore flusher across storages
* Remove all Arc<RwLock<_>> wrappers around Gridstore in storages
* Flush pages inside closure, also wrap them in a lock
* Remove direct flush function from Gridstore
* Add test to assert behavior of deferred flushing in Gridstore
* Add much more extensive test, including concurrent flushes and deletes
* Add test for draining value pointer, make sure to drop when flushed
* Feature gate RocksDB components
* Explicitly use C memory representation for Gridstore value pointers
* Inline pure function
* Grow tracker file to be next multiple of two
* In debug builds, panic if getting out of bound point from Gridstore
* Also represent region gaps as C struct
* Remove debug assertion
* Adjust file size test
