* feat(gridstore): migrate Tracker to universal IO
- Tracker<S> generic over S: UniversalRead<u8> + UniversalWrite<u8>
- Replace MmapSlice<u8> with S; use S::open, read, write, flusher, populate
- On file growth: flush, create_and_ensure_length, re-open S
- Map UniversalIoError::NotFound to tracker file missing error
- get_raw returns Result<Option<Option<ValuePointer>>>; get returns Result<Option<ValuePointer>>
- has_pointer, unset, write_pending return Result where needed
- flusher returns crate::gridstore::Flusher (universal_io Flusher mapped to GridstoreError)
- Type alias Tracker = Tracker<MmapUniversal<u8>> in lib.rs
- Tests use TestTracker = Tracker<MmapUniversal<u8>>; mapping_len/mmap_file_size .unwrap()
Made-with: Cursor
* refactor(gridstore): review follow-ups – generic view, From conversion, iter errors
- GridstoreView: use Tracker<S> with same S as Page (S: UniversalRead + UniversalWrite)
- Tracker: use ? and .map_err(Into::into) instead of .map_err(GridstoreError::from)
- View iter: propagate tracker read errors via Err(e) instead of silently skipping
Made-with: Cursor
* Split Tracker into read/write impl blocks; GridstoreView only requires UniversalRead
- Tracker: impl<S> for files/pointer_count, impl<S: UniversalRead<u8>> for
get/get_raw/iter_pointers/has_pointer/populate and test helpers,
impl<S: UniversalRead+UniversalWrite> for new/open/write_pending/set/unset
and other write methods (like Page).
- GridstoreView: require S: UniversalRead<u8> only so read-only views don't
need UniversalWrite.
Made-with: Cursor
* Move Tracker::open to read-only impl; add FILE_NAME/tracker_file_name to unbound impl
open() only uses S::open and storage.read(), so it belongs in impl<S: UniversalRead<u8>>.
read_config_and_tracker can thus use a read-only tracker open. FILE_NAME and
tracker_file_name moved to unbound impl so both read and write sections use them.
Made-with: Cursor
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
* [manunal] Gridstore page use universal IO
* fmt
* Apply review feedback for universal IO gridstore pages (#8230)
* Apply review feedback from PR #8223
- Use `super::Result` import in mmap.rs instead of fully-qualified `crate::universal_io::Result`
- Restructure ValuePointer destructuring in get_value and delete_value to
first match Some(pointer), then destructure separately
* Replace Either<E, GridstoreError> with E: From<GridstoreError> in Gridstore::iter
Use a trait bound instead of Either to combine callback and gridstore
errors, allowing `?` to work directly on GridstoreError. This simplifies
callers by removing Either matching and io::Error conversion workarounds.
---------
Co-authored-by: qdrant-claw <qdrant-claw@users.noreply.github.com>
---------
Co-authored-by: qdrant-claw <qdrant-claw@users.noreply.github.com>
* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
* Implement `Optional<T>` type
Define a type with the same presumed layout as `Option<T>`, but with defined behavior.
* Make `transmute_*` functions unsafe
The functions `memory::mmap_ops::transmute_*` are inherently unsafe, but
are not marked as are. Their usage is documented, but it is not always clear
if the code is correct.
* Add `CsrHeader` to resolve another unsoundness
Tuples have no defined layout.
* Enrich Gridstore model testing
* better
* no clear on Windows
* less windows
* sigh
* change strategy
* never give up
* x100 scale down for Windows
* last touch
* decrease for others too
* remove Iter test - the quadratic cost is not worth it
* Revert "remove Iter test - the quadratic cost is not worth it"
This reverts commit f0e98cb33b.
* try with limited Iter
* less logging as it is hurting crasher
* tracker.mapping_len() is slow
* hard limit good enough for now
* introduce IsAliveLock
* add tests
* rename to mark_dead
* update gridstore comment
* Update some comments
* Add must_use attribute to lock_if_alive
* Rename mark_dead to blocking_mark_dead
* make the handle take a `Weak` reference
* make dropping explicit
* use `Mutex::lock_arc` instead
* less nesting
* clippy
---------
Co-authored-by: timvisee <tim@visee.me>
* Separate set and unset lists for each Gridstore pointer update
This makes the list of pointer updates much easier to grasp. The
simplification is desired because this exact structure has been a cause
for bugs multiple times now.
* Add more aggressive debug assertions
* Use consistent terminology
* Correct removal of set
* Simplify drain function
* Add safe guard to ignore empty pointer updates
* Rework tracker getter, explicitly branch variants
* Patch transmute, require Sized type
* rename and adjust comments
- set/unset is now current/to_free
- adjusted descriptions and comments for this nomenclature too
* clippy
* clippyyy
not 4, not 2, but 3 spaces
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Gridstore decompress failure
* more logs
* more logs
* hardening
* clean
* no compression
* cleanup
* typo
* misc
* wait for flush & less logs
* add compression type to the mix
* optimize
* Fix Gridstore tracker drain and persist not working properly
* Patch existing test
* Add new test to assert buggy scenario we found
* Mention PR in test
* Add lease structure to invalidate pending flushers after wipe/clear
* We don't use bitmask as barrier anymore
* Add missing early return
* Add test
* Replace flush lease with is alive boolean we also use elsewhere
* Reimplement test, now it does fail on the old implementation
* In test, cover both new fixed and old broken flushing
* Remove old test case because it is flaky
* Fix clippy
* Fix outdated comment
* Fix delete_page not dropping sequential mmap
* In Gridstore Page, support using a single mmap
* In Gridstore Page, only open multiple mmaps if supported at runtime
* Support opening single mmap in UniversalMmapChunk
* Rename environment var, log warning when QDRANT_NO_MULTI_MMAP is set
* Support opening single mmap in MmapDenseVectors
* Remove unused result response
* Import LazyLock
* Wrap the tracker and bitmask in Arc
* Implement deferred flusher for gridstore, defer tracker writes
* Remove clone
* Dynamically adjust list of pending updates to drain what is flushed
* Return proper flusher, defer premature Gridstore flushing errors
* Use deferred Gridstore flusher across storages
* Remove all Arc<RwLock<_>> wrappers around Gridstore in storages
* Flush pages inside closure, also wrap them in a lock
* Remove direct flush function from Gridstore
* Add test to assert behavior of deferred flushing in Gridstore
* Add much more extensive test, including concurrent flushes and deletes
* Add test for draining value pointer, make sure to drop when flushed
* Feature gate RocksDB components
* Explicitly use C memory representation for Gridstore value pointers
* Inline pure function
* Grow tracker file to be next multiple of two
* In debug builds, panic if getting out of bound point from Gridstore
* Also represent region gaps as C struct
* Remove debug assertion
* Adjust file size test
