* feat: add open methods for map
* fix: review comments
* feat: detect not-found via error instead of path.exists in map index open
Replace the path.exists() pre-check in the read-only appendable map index
open path with error-driven detection through ok_not_found(). Generalize
OkNotFound over an IsNotFound trait, implemented for io::Error, mmap::Error,
UniversalIoError, and GridstoreError so a NotFound surfacing through any
layer (including mmap's inner io::Error) maps to Ok(None).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* review: revert internal error conversion
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [UIO] Split UniversalReadFileOps into filesystem + file traits
`UniversalReadFileOps` is now an instance-based trait describing a
filesystem handle (list/exists with `&self`, plus `from_context`). A new
`UniversalReadFs: UniversalReadFileOps` subtrait adds the
`open(&self, path, options) -> Self::File` capability with `type File:
UniversalRead`. `UniversalRead` no longer extends `UniversalReadFileOps`
and is purely a file-handle trait.
This separates "filesystem instance" from "file handle". Backends that
need per-instance configuration (S3 bucket name + credentials, mmap
default advice, io_uring runtime, block-cache controller `Arc`) gain a
typed home in `Self::ContextConfig`, and `list_files`/`exists`/`open`
become `&self` methods on the filesystem handle.
Concrete filesystem handles introduced for the three existing backends:
- `MmapFs` — unit struct; `ContextConfig = ()`; produces `MmapFile`.
- `IoUringFs` — carries `prevent_caching`; `ContextConfig =
IoUringConfigContext`; produces `IoUringFile`. `IoUringConfigContext`
becomes the construction input rather than a per-open argument.
- `BlockCacheFs` — carries `Arc<CacheController>`; `ContextConfig =
BlockCacheConfigContext`; produces `CachedSlice`.
`TConfigContext` (universal builder methods) is kept so generic-over-`Fs`
code can still set cross-backend knobs via
`Fs::ContextConfig::default().with_prevent_caching(true)`.
Wrappers (`ReadOnly<S>`, `TypedStorage<S, T>`, `StoredStruct<S, T>`),
higher-level storages (`StoredBitSlice<S>`, `UniversalHashMap<K, V, S>`)
keep `<S: UniversalRead>` parameterization but their `open(...)`
constructors now grow a `fs: &Fs` argument bound by
`Fs: UniversalReadFs<File = S>`. `read_json_via` becomes
`read_json_via(fs: &Fs, path)`.
All test code and benches in `common` updated to construct
`MmapFs`/`IoUringFs` inline as needed. `common` compiles cleanly with
tests and benches. `gridstore`, `segment`, and `tonic` caller updates
are in flight in subsequent commits.
* WIP: gridstore + segment caller sweep (partial)
Threads `fs: &Fs` through gridstore's `BitmaskGaps`, `Bitmask`, `Pages`,
and `Gridstore::new`/`open`/`create_new_page`. Most segment callers
have `OpenOptions { extra: ... }` removed and `S::open(path, opts, ctx)`
sites updated mechanically but the trait change is not yet propagated.
Does NOT compile yet. Tracker still has static `S::open` calls, segment
generic constructors (`MmapInvertedIndex<S>`, `UniversalMapIndex`,
`StoredGeoMapIndex`, etc.) still call `S::open`/`S::list_files`/`S::exists`
statically — they need an `fs: &Fs` parameter added. Tonic API
`StorageReadService<S>` also unconverted.
Committed as branch checkpoint; cascade continues in subsequent work.
* gridstore: thread `fs: &Fs` through Bitmask, BitmaskGaps, Pages, Tracker
Per the new `UniversalReadFs` shape, every constructor/method that opens
files takes an `fs: &Fs` parameter. Gridstore is currently mmap-only,
so the top-level `Gridstore` / `GridstoreReader::open` callers in the
crate pass `&MmapFs` inline. Tests do the same.
gridstore lib + tests now compile cleanly. Segment + tonic cascade
still pending.
* WIP: segment caller sweep — dynamic_stored_flags first
* WIP: segment cascade - id_tracker partial
* common benches: update to new UniversalReadFs::open shape (clippy clean)
* segment flags: thread Fs through BufferedDynamicFlags / Bitvec / Roaring
DynamicStoredFlags::set_len now takes `fs: &Fs`. BufferedDynamicFlags
stores an `Arc<Fs>` so the flusher closure can call `set_len` on resize.
BitvecFlags and RoaringFlags expose a new `Fs` type parameter and the
flag tests now pass `Fs::default()` (MmapFs/IoUringFs via duplicate_item).
Concrete consumers (bool/null index, mmap dense/multi/sparse storages)
pin `Fs = MmapFs` and pass `&MmapFs` to inner opens.
* segment: thread Fs through field-index lifecycle methods
Apply the new UniversalReadFs::open shape across:
- full_text_index (MmapInvertedIndex, MmapFullTextIndex, UniversalPostings)
- geo_index (StoredGeoMapIndex build/open + tests + builders)
- numeric_index lifecycle (UniversalNumericIndex build/open)
- map_index lifecycle (UniversalMapIndex build/open)
- stored_point_to_values (open / from_iter)
Concrete consumers pin Fs = MmapFs and pass &MmapFs inline; generic
open paths thread `fs: &Fs` where Fs: UniversalReadFs<File = S>.
* segment: thread Fs through chunked vectors and id-tracker callers
- ChunkedVectors gains `Fs` generic so add_chunk can call create_chunk
after open. ChunkedVectorsRead/load_config and chunks::{read_chunks,
create_chunk} take `fs: &Fs`. Concrete callers (dense / multi-dense /
sparse / quantized) pass MmapFs inline.
- DenseVectorStorageImpl stores `fs: Fs` so `update_from` can reopen
ImmutableDenseVectors. ImmutableDenseVectors::open takes `fs: &Fs`.
- VectorStorageEnum DenseUring* variants thread IoUringFs alongside
IoUringFile.
- segment_builder + segment_constructor_base pass MmapFs to
ImmutableIdTracker::{new, open}.
- QuantizedStorage::from_file takes `fs: &Fs`; quantized_vectors callers
pass &MmapFs.
* segment: apply nightly rustfmt after Fs refactor
cargo +nightly fmt --all over the segment crate after the
UniversalReadFs cascade. No semantic changes.
* segment: thread Fs through benches and id-tracker tests
Update the dynamic-mmap-flags and buffered-update-bitslice benches to
the new UniversalReadFs::open shape (pass `&MmapFs`). Update the
immutable-id-tracker test suite to forward `&MmapFs` to
`from_in_memory_tracker` / `open`.
* uio: pin Fs via UniversalRead::Fs assoc type; per-call OpenExtra
Two design changes that fall out of the per-instance Fs refactor:
1. Bidirectional Fs ↔ File pinning. `UniversalRead::Fs:
UniversalReadFs<File = Self>` lets generic-over-`S` code refer to
`S::Fs` directly instead of carrying an extra `<Fs: UniversalReadFs<File = S>>`
generic param. `ReadOnly<S>` wraps a file but has no natural
filesystem; a phantom `ReadOnlyFs<S::Fs>` satisfies the constraint
while inherent `ReadOnly::open` keeps taking `&S::Fs` directly.
2. `prevent_caching` moves from filesystem-instance state to per-call
`UniversalReadFs::OpenExtra: Default`. Was previously a knob on
`IoUringConfigContext` / `IoUringFs`, conflating "how this fs is
built" with "how this file is opened." Now `IoUringFs::OpenExtra =
IoUringOpenExtra { prevent_caching }`; mmap and block-cache use `()`.
`IoUringConfigContext` is gone, `TConfigContext` slims to a `Default`
marker.
Tonic StorageReadService holds `Arc<S::Fs>` (was `PhantomData<S>`); its
`new()` builds via `S::Fs::from_context(default)` and the spawn_blocking
closures clone the Arc to call instance methods.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: cfg-gate IoUringOpenExtra import for non-linux builds
The IoUringOpenExtra reexport from `universal_io` is gated on
`target_os = "linux"`. The previous commit left an unconditional import
in `persisted_hashmap/tests.rs`, breaking macOS/Windows CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: migrate simple_disk_cache to per-instance Fs API
PR #9097 (merged into dev concurrently with this branch) introduced a
`DiskCache<R: UniversalRead>` using the pre-refactor trait shape:
file-handle-as-filesystem (`R::open`, `R::list_files`), an
`OpenOptionsExtra` field on `OpenOptions`, and trait methods without
`&self`. The Fs-instance refactor on this branch removed all three.
Reshape `simple_disk_cache` to match the new design without breaking the
lazy mirror semantics:
- New `DiskCacheFs<R>` is the filesystem handle. Holds a clone of the
remote `R::Fs`; `list_files`/`exists` delegate; `from_context`
forwards to the inner Fs context. `open` constructs a `DiskCache<R>`
via the global `DiskCacheConfig`.
- `DiskCache<R>` now stores `remote_fs: R::Fs` + `remote_extra:
<R::Fs as UniversalReadFs>::OpenExtra`, so lazy remote opens go
through `self.remote_fs.open(path, options, extra)` instead of the
removed `R::open`. `open_with_config` takes the remote Fs + extra
explicitly (no more hard-coded `prevent_caching: true`; callers pass
the appropriate `OpenExtra`).
- `UniversalRead for DiskCache<R>` now declares `type Fs =
DiskCacheFs<R>` (no more `open` method on the file trait).
- Propagate the necessary bounds (`R::Fs: Clone`,
`<R::Fs as UniversalReadFs>::OpenExtra: Clone`,
`R::OwnedReadPipeline<u8, Range<u32>>: Send`) through `pipeline.rs`
free functions and impl blocks that reach into `DiskCache::remote` /
`local_state`.
- Drop the now-removed `extra: _` destructure in `LocalState::new`.
- Tests construct the remote Fs via `R::Fs::from_context(Default::default())`
and exercise `DiskCache::open_with_config`. 17 simple_disk_cache
tests pass; the 3 `empty_read_does_not_materialize_local_file`
failures pre-exist on dev (verified) and are unrelated.
`mold -run cargo clippy --all-targets` clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt on simple_disk_cache migration
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt for local_state imports after rebase
Co-authored-by: Cursor <cursoragent@cursor.com>
* uio: split DiskCacheFs into its own module
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: replace TConfigContext with OpenExtra trait; move DiskCacheConfig onto DiskCacheFs
- Drop the empty TConfigContext marker; ContextConfig is now unconstrained
so backends can require explicit construction.
- Add OpenExtra trait with with_prevent_caching for backend-agnostic
per-call knobs; impl for () (no-op) and IoUringOpenExtra.
- DiskCacheFs now carries Arc<DiskCacheConfig> via the new
DiskCacheFsContext<C>; the prefill flow moves from the deleted
open_with_config into DiskCacheFs::open so Populate::Blocking /
PreferBackground work through the trait API.
- Remove the DiskCacheConfig global; callers must construct the context
explicitly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt after OpenExtra refactor
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: fix spelling — Implementors → Implementers
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: possible panic insetad of error propagation
* fix: missing cfg annotation
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* bytemuck::Pod already implies 'static
* no supertraits
* regions gaps with TypedStorage
* StoredBitSlice with TypedStorage
* bytemuck for PostingsHeader
* bytemuck for TrackerHeader
* bytemuck for MmapRange
* bytemuck for stored_point_to_values::Header
* derive bytemuck::Pod on `OptionalPointer`, get rid of legacy transmute
* remove one layer of Option in `get_raw`
* keep `UniversalRead<u8>` bound, do bytemuck conversion at call site (#8953)
Drop the `+ UniversalRead<OptionalPointer>` / `+ UniversalWrite<OptionalPointer>`
trait bounds added in the previous two commits. Reads and writes of
`OptionalPointer` and `TrackerHeader` now go through `bytemuck::from_bytes` /
`bytemuck::bytes_of` on the existing `u8` byte slices instead.
This keeps the unsafe-transmute removal but avoids the turbofish noise
(`UniversalWrite::<u8>::flusher(&self.storage)`, `<S as UniversalRead<u8>>::open`,
…) that two `T`s on `S` forced everywhere.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* extra refactoring after #8955
* explicit zeroed instantiation
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Move element type from UniversalRead trait to method generics
Lifts the `T` parameter off `trait UniversalRead<T>` (and the matching
`UniversalWrite<T>`) and onto the read/write methods themselves. The
`ReadPipeline` associated type becomes a GAT over `T`. With per-method
generics, callers that need to read several element types from one storage
just write `S: UniversalRead` instead of stacking
`UniversalRead<u8> + UniversalRead<Counts> + ...`.
Removes the workarounds the old shape required:
- `TypedStorage<S, T>` newtype (sole purpose was disambiguating multi-bounds)
- `UniversalReadFamily` HKT shim
- `StoredGeoMapIndexStorage` four-bound trait alias
- `CachedSlice<T>` is now non-generic; `T` moves to `get_range`/`len`
No runtime behavior change: alignment in `IoUringRuntime` and `CachedSlice`
is preserved because `T` is still known at each call site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Restore TypedStorage as a typed-access fail-safe
Reintroduces `TypedStorage<S, T>` as a transparent wrapper around
`UniversalRead`/`UniversalWrite` storage that fixes the element type to
`T`. With per-method generics on the underlying traits, callers can
otherwise read or write any `T` from the same handle; this wrapper
binds it at the type level so accidental cross-type access fails to
compile.
The wrapper exposes inherent typed methods (`read::<P>`, `read_iter`,
`write`, `len`, …) that delegate to the inner storage with `T` fixed.
It does not implement `UniversalRead`/`UniversalWrite` itself — those
are intentionally avoided to prevent the typed binding from being
bypassed via the generic trait methods.
Restores the wrapping at the previous call sites: `StoredStruct`'s
inner storage, `ImmutableIdTracker`'s version mmap, the geo and
numeric index storages, the chunked-vectors chunks, and the
immutable dense vector storage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Silence clippy::len_without_is_empty on TypedStorage
`TypedStorage::len` returns `Result<u64>` (a fallible byte length from
the underlying storage), so an `is_empty` companion would also be
fallible and offer nothing over `len()? == 0`. Suppress the lint at
the impl block, matching how the underlying `UniversalRead::len` is
already exempted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fmt
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Windows CI takes ~28min vs ~18min on Linux, with some individual tests
running 10-50x slower due to Windows filesystem IO overhead. Since we
only need functional compatibility on Windows (not performance testing),
reduce iteration counts for the worst offenders:
- WAL quickcheck: 10 → 3 iterations (saves ~350s)
- Consensus manager proptests: 256 → 10 cases (saves ~100s)
- Deferred point tests: reduce loop combinations (saves ~250s)
- Gridstore test_behave_like_hashmap: 50k → 10k operations (saves ~200s)
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* use pageout to clear mmap cache
* Also clear cache of deleted flags in mmap dense vector storage
* Add reference to madvise man pages for probe logic
* use deconstruct
---------
Co-authored-by: timvisee <tim@visee.me>
* [ai] Replace manual into mappings with Into::into
* Reformat
* [ai] Use implicit .iter
* Don't iterate over keys too
* [ai] Replace unwrap_or
* Reformat
* [ai] Use as_deref and then_some
* [ai] Use more to_string
* [ai] Use explicitly typed into conversions
* Reformat
* [ai] More explicit into conversions
* Reformat
* IoUringState: generic `RequestId`
* UniversalRead: generic `RequestId`
* Simplify `gridstore::Pages::get_page_value_ranges`
Now we don't need two separate `SmallVec`s as we can put `buffer_offset`
into `RequestId`.
* Better doc comment
* Rename `RequestId` -> `Meta`
* Skip broken tests
* Add rocksdb dropper
This is a temporary tool. It will be removed later in this PR.
* [automated] Drop rocksdb
This commit is made by running tools/rocksdb/drop.sh
* Touch-up after ast-grep
The previous automated commit removed items, but not their comments.
Also, some blocks are left with only one item.
Also, ast-grep can't handle macros like `vec![]`.
This commit completes the job.
* Remove RocksDB dropper
* Remove mentions of rocksdb feature in CI
* Fix clippy warnings
These `FIXME` comments added previously in this PR by ast-grep by
"peeling" cfg_attr like this:
-#[cfg_attr(not(feature = "rocksdb"), expect(...))]
+#[expect(...)] // FIXME(rocksdb): ...
This commit removes these allow/expect attributes and fixes clippy
lints.
* Remove leftover rocksdb-related code
Removed:
- Cargo.toml: rocksdb cargo feature flag and dependencies.
- code: rocksdb-related items that was not under feature flag.
- flags.rs: rocksdb-related qdrant feature flags.
Disabled:
- Some benchmarks because these do not compile now.
* Print warning if rocksdb leftovers found in snapshots
* Remove Clone from tokenizer
* Regenerate openapi.json
* UIO traits: require Sized
I don't think we ever going to add unsized implementation. So, lets,
require it on the whole trait rather than on individual methods.
* Move `AccessPattern` to the `common` package
Was: segment::vector_storage::vector_storage_base::AccessPattern
Now: common::generic_consts::AccessPattern
* UIO traits: replace `bool` with `AccessPattern`
* feat(universal_io): add storage-agnostic BitSliceStorage with read/write support`
Add BitSliceStorage<S> generic over
UniversalRead<u64>/UniversalWrite<u64>,
providing bit-level read and write operations over u64-element storage.
Read operations (S: UniversalRead<u64>):
- read_all: returns Cow<BitSlice> (zero-copy for mmap, owned for others)
- get_bit: single bit read via u64 element fetch
- read_bit_range: arbitrary bit range read
Write operations (S: UniversalWrite<u64>):
- set_bit / replace_bit: single bit write (skips write if unchanged)
- write_bit_range: arbitrary bit range write from BitSlice source
- fill_bit_range: fill range with a value
- set_bits_batch: batch individual bit updates coalesced by element
- flusher: flush underlying storage
* refactor: replace MmapBitSlice with BitSliceStorage in MmapBitSliceBufferedUpdateWrapper
Migrate all deleted-flag bitslice storage from the legacy MmapBitSlice
(Deref-based mmap wrapper) to BitSliceStorage<MmapUniversal<u64>>
(storage-agnostic universal IO backend).
Updated consumers:
- MmapMapIndex
- MmapNumericIndex
- MmapGeoMapIndex
- MmapInvertedIndex (fulltext)
- ImmutableIdTracker
- Benchmark
Additionally:
- Add BitSliceStorage::create(path, num_bits) to encapsulate
file creation + sizing + open (eliminates duplicated size math
across 5 call sites)
- Add MmapBitSliceStorage type alias for
BitSliceStorage<MmapUniversal<u64>>
- Add count_ones() convenience method
- Use set_bits_batch() in wrapper flusher and all build paths
instead of per-bit set_bit() loops (coalesces u64 read-modify-writes)
- Fix silent error swallowing in wrapper get(): log + debug_assert
on I/O errors instead of .ok().flatten()
- Remove dead bitmap_mmap_size function from immutable_id_tracker
* use bitvec's approach for offset
* less api
* misc improvements
* refactor write method
* move to segment crate
* handle creation of file outside of StoredBitSlice logic
* fix codespell
* document bitwise calculations
* coalesce more updates into a single write, batch all writes
* use native `extend_from_bitslice` instead of iterating.
* clarity refactor
* clippyyy
* use `u64` as BitStore everywhere
* assume iterator is sorted
* only use chunk_by for generating runs
* fix update wrapper flusher
* review fixes
* larger set bits batch test
* remove reintroduced file
* oops, fix new test
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
Reduce test parameters across 5 areas to cut test runtime without
sacrificing coverage:
1. HNSW PQ tests: lower vector dimensionality from 131 to 64 for product
quantization variants, cutting PQ codebook training time (~31s -> ~8s)
2. HNSW index build: use 2 threads instead of 1 for index construction;
the tests only check accuracy above a threshold so determinism is not
required
3. Search attempts: reduce from 10 to 5 query vectors per test
4. Rescoring: skip the expensive re-upsert-all-as-zeros rescoring check
for PQ tests (already covered by scalar quantization variants)
5. Near-miss speedups:
- WAL: reduce QuickCheck iterations from 100 to 50
- Gridstore: halve operation counts, drop 64-byte block size case,
reduce proptest cases for gap search
- Continuous snapshot: reduce timeout from 20s to 10s
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* lib/edge/publish/cargo: improve target directory discovery
Don't hardcode workspace root, ask cargo instead.
* Fix warning in qdrant-edge amalgamation
`cargo check` in `lib/edge/publish` produces the following warning:
warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending`
* lib/edge/publish/examples: fix compiler warnings
* refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs
To match python examples that have the same constant in `common.py`.
* Use `lib/edge/data` dir for both Python and Rust examples
Before this commit, `lib/edge/publish/data` and `lib/edge/python/data`
were used. I'd like Python and Rust to use the same prepared resources
in the future.
* ❗Add lib/edge/Justfile with recipes to build/run examples
I constantly keep forgetting how to compile and run edge stuff.
Also, this would be used in CI in subsequent commits.
* ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes
Also now we can rename `creates-io-readme.md` to `README.md` like
reasonable people.
* ❗doc: Add qdrant-edge-py README.md
Since the previous commit removed qdrant-edge-py README.md, we need to
add something in its place instead.
The new README is adapted from Rust edge README.
And the previous README wasn't good anyway, see [1], it mentions maturin
which is irrelevant for users.
[1]: https://pypi.org/project/qdrant-edge-py/0.5.0/
* GHA: Add "Setup Qdrant" composite action
We need running Qdrant instance to test edge examples on CI. We can't
use docker for this because we want to run tests on Windows and macOS.
OTOH, this action downloads binaries for all platforms provided in
https://github.com/qdrant/qdrant/releases/tag/v1.17.0
* ❗GHA: Use single worflow for python and rust edge examples
Merge workflows to reuse build cache. Also, modernize it by using `uv`,
and recently introduced `setup-qdrant` and Justfile. Also, make it run
on three platforms (only on merges).
* ❗GHA: edge-{py,rust}-release: also run examples before publishing
Use similar approach as in the previous commit. But note that this time
we build/test also on ARM Linux and Windows.
* feat(gridstore): migrate Tracker to universal IO
- Tracker<S> generic over S: UniversalRead<u8> + UniversalWrite<u8>
- Replace MmapSlice<u8> with S; use S::open, read, write, flusher, populate
- On file growth: flush, create_and_ensure_length, re-open S
- Map UniversalIoError::NotFound to tracker file missing error
- get_raw returns Result<Option<Option<ValuePointer>>>; get returns Result<Option<ValuePointer>>
- has_pointer, unset, write_pending return Result where needed
- flusher returns crate::gridstore::Flusher (universal_io Flusher mapped to GridstoreError)
- Type alias Tracker = Tracker<MmapUniversal<u8>> in lib.rs
- Tests use TestTracker = Tracker<MmapUniversal<u8>>; mapping_len/mmap_file_size .unwrap()
Made-with: Cursor
* refactor(gridstore): review follow-ups – generic view, From conversion, iter errors
- GridstoreView: use Tracker<S> with same S as Page (S: UniversalRead + UniversalWrite)
- Tracker: use ? and .map_err(Into::into) instead of .map_err(GridstoreError::from)
- View iter: propagate tracker read errors via Err(e) instead of silently skipping
Made-with: Cursor
* Split Tracker into read/write impl blocks; GridstoreView only requires UniversalRead
- Tracker: impl<S> for files/pointer_count, impl<S: UniversalRead<u8>> for
get/get_raw/iter_pointers/has_pointer/populate and test helpers,
impl<S: UniversalRead+UniversalWrite> for new/open/write_pending/set/unset
and other write methods (like Page).
- GridstoreView: require S: UniversalRead<u8> only so read-only views don't
need UniversalWrite.
Made-with: Cursor
* Move Tracker::open to read-only impl; add FILE_NAME/tracker_file_name to unbound impl
open() only uses S::open and storage.read(), so it belongs in impl<S: UniversalRead<u8>>.
read_config_and_tracker can thus use a read-only tracker open. FILE_NAME and
tracker_file_name moved to unbound impl so both read and write sections use them.
Made-with: Cursor
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
* [manunal] Gridstore page use universal IO
* fmt
* Apply review feedback for universal IO gridstore pages (#8230)
* Apply review feedback from PR #8223
- Use `super::Result` import in mmap.rs instead of fully-qualified `crate::universal_io::Result`
- Restructure ValuePointer destructuring in get_value and delete_value to
first match Some(pointer), then destructure separately
* Replace Either<E, GridstoreError> with E: From<GridstoreError> in Gridstore::iter
Use a trait bound instead of Either to combine callback and gridstore
errors, allowing `?` to work directly on GridstoreError. This simplifies
callers by removing Either matching and io::Error conversion workarounds.
---------
Co-authored-by: qdrant-claw <qdrant-claw@users.noreply.github.com>
---------
Co-authored-by: qdrant-claw <qdrant-claw@users.noreply.github.com>
* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
* Implement `Optional<T>` type
Define a type with the same presumed layout as `Option<T>`, but with defined behavior.
* Make `transmute_*` functions unsafe
The functions `memory::mmap_ops::transmute_*` are inherently unsafe, but
are not marked as are. Their usage is documented, but it is not always clear
if the code is correct.
* Add `CsrHeader` to resolve another unsoundness
Tuples have no defined layout.
