* ci(windows): skip IO-heavy tests that aren't OS-specific
On the Windows CI runner, several tests are 3-25x slower than on Ubuntu
purely due to slow filesystem IO. These tests exercise platform-agnostic
logic (optimizer, snapshot, WAL recovery, dedup, deferred points) and
are fully covered by the Linux and macOS jobs.
Mark them with `#[cfg_attr(target_os = "windows", ignore = "...")]` so:
- Windows CI skips them and finishes faster.
- They're still listed and runnable via `cargo test -- --ignored`
on Windows for local debugging.
Based on JUnit timings from CI run 26462785436 (PR #8827), this should
save ~5 minutes wall-clock on the Windows job, taking it closer to the
~13min Ubuntu and ~9min macOS jobs (currently 20m24s).
Tests affected:
- lib/wal: check_wal, check_last_index, check_clear, check_reopen,
check_truncate, check_prefix_truncate, test_prefix_truncate_parametric
- lib/edge/optimize: full tests module
- lib/segment deferred-point tests: read_operations,
dense_segment_combinations, sparse, facets
- lib/collection: snapshot_test, points_dedup, wal_recovery,
collection_test::test_ordered_read_api, snapshot_recovery_test
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert(ci/windows): keep WAL and WAL-recovery tests on Windows
Reviewer correctly pointed out that WAL is mmap-backed and has
substantial Windows-specific code paths:
- Different segment allocation (fs4 vs rustix::ftruncate)
- Windows-specific delete_windows() with mmap-drop + retry loop
- Windows-specific sync_all() because directory fsync is unavailable
- Windows-specific lock proxy file (directories aren't lockable)
So those tests genuinely need Windows coverage. Reverted skips for:
- lib/wal/src/lib.rs: all check_* tests and test_prefix_truncate_parametric
- lib/collection/src/tests/wal_recovery_test.rs: all three tests
Still skipped on Windows (no OS-specific code in their production paths):
- lib/edge/optimize.rs (no cfg(windows) in source)
- lib/segment deferred-point tests (segment/ has no cfg(windows))
- lib/collection snapshot/dedup tests (collection/ has no cfg(windows))
- lib/collection integration snapshot_recovery + ordered_read_api
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert(ci/windows): keep collection integration and snapshot_test
Per reviewer request, keep running these on Windows:
- lib/collection/tests/integration/* (snapshot_recovery_test,
collection_test::test_ordered_read_api)
- lib/collection/src/tests/snapshot_test.rs
These exercise higher-level collection/snapshot behavior that benefits
from cross-platform validation.
Remaining Windows skips (production code has no cfg(windows) branches):
- lib/edge/src/optimize.rs: 14 optimizer tests
- lib/segment/src/segment/tests/mod.rs: 4 deferred-point tests
- lib/collection/src/tests/points_dedup.rs: 2 dedup tests
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci(windows): also skip HNSW/quantization integration tests
Per reviewer, also skip these segment integration test modules on Windows:
- hnsw_quantized_search_test::* (25 tests)
- multivector_filtrable_hnsw_test::* (rstest cases)
- multivector_quantization_test::* (rstest cases)
- byte_storage_quantization_test::* (rstest cases)
- payload_index_test::test_struct_payload_index_nested_fields
These exercise pure HNSW/quantization correctness on top of standard
segment IO that is already covered by tests we keep running on Windows.
Adds ~930s of sequential time to the Windows skip list, bringing the
expected wall-clock saving from ~3 min to ~8-10 min.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: add io_bridge
* fix: cached dispatcher
* feat: add open_with_handle
* fix: naming
* fix: wording
* fix: rebase io_bridge onto split BorrowedReadPipeline/OwnedReadPipeline
* fix: linter
* feat: add S3 backend
* chore: remove io_bridge
* fix: linter
* fix: linter
* fix: read handle
* chore: simplify S3Source
* fix: s3 test
* feat: support multi runtime
* fix: clippy errors
* fix: review comments
* feat: add io design
* feat: add S3 backend
* chore: fix docs
* fix: dev changes
* chore: add some docs
* chore: remove explicit type
* feat: add new methods
* [WIP] review refactor
* fmt
* fix: bytes alignment
* fix: linter
* feat: remove Bytes
* fix: ci/cd
* fix: tests
* fix: tests
* refactor: simplify io_bridge pipeline to Handle-based dispatch
Replace the BridgeRuntime worker thread + request channel + boxed
BridgeRequest with direct tokio Handle usage:
- BridgeRuntime is now just an Arc<Runtime>; schedule() spawns the read
future via Handle::spawn instead of routing it through a dispatcher
thread. Removes BridgeRequest and the now-unreachable S3RuntimeShutDown
error variant.
- Guard against a panicking read task hanging wait() forever: the spawned
task catches unwinds and converts them into a TaskPanicked error reply,
so every scheduled slot is always answered.
- Encapsulate slot bookkeeping in PendingSlots, exposing only the needed
operations instead of a public map + counter.
- Split the grown pipeline.rs into a pipeline/ module (slots / inner /
borrowed / owned), de-duplicating the shared read-future construction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: move pipeline buffer ownership into the read future
Instead of the pipeline owning the destination Vec<T> in a slot map and
the future writing through a SendBytePtr raw pointer, let the future
allocate the buffer itself and return it through BridgeResponse. The
buffer crosses the worker-thread boundary as a normal move via the reply
channel, wrapped in a SendableVec<T> newtype that asserts Send for
T: bytemuck::Pod only.
This removes the entire unsafe SendBytePtr apparatus from the pipeline:
no raw pointer, no unsafe fn, no per-call-site unsafe blocks, no
heap-stability invariants. The only remaining unsafe in the crate is one
bounded `unsafe impl<T: Pod> Send for SendableVec<T>` with a trivially
true invariant (Pod types are plain bytes).
PendingSlots collapses back to PendingSlots<U>: slots no longer carry
buffers. AlignedBufWriter::from_raw_bytes (used only by the SendBytePtr
path) and its test are removed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: drop SendableVec wrapper now that Item: Send
UniversalRead's element type is now bound to `Item` (`Pod + Send`),
so the io_bridge pipeline no longer needs a hand-rolled `Send` wrapper
around `Vec<T>` to ship buffers through the reply channel. Replace
`SendableVec<T>` with `Vec<T>` end-to-end and tighten the local impls
to `T: Item`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: implement UniversalRead::reopen for BlobFile
BlobFile has no cached file metadata or mapping — `len()` queries the
object store fresh on each call — so reopen is a no-op, matching the
io_uring impl.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: add new fs impl for Blob
* fix: is_in_ram_or_mmap for S3
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(edge): expose WAL options via EdgeShard::load_with_wal_options
Adds a sibling method to EdgeShard::load that accepts custom
WalOptions, alongside the existing load() which keeps using
default_wal_options().
Motivation: embedded/mobile deployments (e.g. Flutter plugins on
iOS/Android) need much smaller WAL segments than the 32 MiB default —
typically 4 MiB. On filesystems with sparse files (APFS/ext4/F2FS)
the physical footprint is small, but the visible/reported size is
the full segment capacity, which is surfaced by iCloud backup, OS
size pickers, etc.
Changes:
* New EdgeShard::load_with_wal_options(path, config, wal_options).
* EdgeShard::load delegates to it with default_wal_options() — no
behavior change for existing callers.
* ensure_dirs_and_open_wal now takes WalOptions explicitly; called
from both EdgeShard::new (with default) and load_with_wal_options.
* WalOptions re-exported from edge crate root.
* Two regression tests in lib/edge/tests/wal_options.rs.
WAL options are intentionally a runtime parameter, not part of
EdgeConfig (which is persisted to edge_config.json) — different
processes may legitimately open the same shard with different WAL
options.
* test(edge): verify mismatching WAL options on reload preserve data
Addresses upstream review feedback (qdrant/qdrant#9067): demonstrate
that reloading an existing shard with WAL options different from the
ones it was created with is safe and preserves all previously written
points.
Two new tests in lib/edge/tests/wal_options.rs:
* reload_with_smaller_wal_capacity_after_upsert:
create shard with default 32 MiB WAL -> upsert point 42 ->
drop -> reload with 4 MiB WAL options -> point 42 still readable ->
upsert point 43 under smaller WAL -> count == 2.
* reload_with_larger_wal_capacity_after_upsert (symmetric):
create -> reload with 4 MiB -> upsert point 100 -> drop ->
reload with default 32 MiB -> point 100 readable -> upsert
point 101 -> count == 2.
Both pass. The WAL crate does not validate segment_capacity on
reload — existing segments on disk keep their original size,
subsequent appends honor the runtime options. This confirms the
PR description's claim that WalOptions is a runtime hint, not
persisted shard state.
* style(edge): cargo fmt for wal_options.rs mismatch tests
* refactor(edge): replace load_with_wal_options with builder-based options
Addresses upstream feedback (timvisee, generall): the sibling
constructor pattern doesn't scale as runtime configurability grows.
Replaces `EdgeShard::load_with_wal_options(path, config, wal_options)`
with `EdgeShard::load_with_options(path, config, EdgeShardOptions)`,
where `EdgeShardOptions` is a builder-style runtime-options struct.
* `EdgeShardOptions::new().with_wal_options(opts)` is the equivalent
of the old call.
* Adding new runtime options later (e.g. wal flush interval, initial
indexing threshold) is now an additive change — new builder method
on `EdgeShardOptions`, no new constructor variant on `EdgeShard`.
* `EdgeShardOptions` is intentionally not persisted (no
Serialize/Deserialize). The reload-mismatch tests above confirm
that WAL options are a runtime hint, not shard identity, so they
don't belong in `EdgeConfig` (which is persisted to
`edge_config.json`).
* `EdgeShard::load(path, config)` unchanged for existing callers —
delegates to `load_with_options(..., EdgeShardOptions::default())`.
Tests in `lib/edge/tests/wal_options.rs` migrated to the new shape;
all four still pass:
test load_with_options_accepts_custom_wal_capacity ... ok
test load_still_works_with_default_wal_options ... ok
test reload_with_smaller_wal_capacity_after_upsert ... ok
test reload_with_larger_wal_capacity_after_upsert ... ok
* refactor(edge): add fluent builders; move wal_options into EdgeConfig
Replaces the EdgeShardOptions side-channel with a single EdgeConfig that
carries wal_options inline, and introduces fluent builders for the three
user-facing config types.
* WalOptions now derives Clone/PartialEq/Eq/Serialize/Deserialize so it
can live inside EdgeConfig and round-trip through edge_config.json.
* EdgeConfig.wal_options (Option<WalOptions>) replaces EdgeShardOptions.
EdgeShard::load_with_options is folded into EdgeShard::load; both new
and load drive the WAL from config.wal_options.unwrap_or_default().
* New builders/ module hosts EdgeConfigBuilder, EdgeVectorParamsBuilder,
and EdgeSparseVectorParamsBuilder. Each builder has explicit per-field
storage and constructs its target via an exhaustive struct literal in
build(), so adding a field to the target forces a compile error in the
builder.
* publish example switched to the new builder API.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style(edge): cargo fmt after builder refactor
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [UIO] Split UniversalReadFileOps into filesystem + file traits
`UniversalReadFileOps` is now an instance-based trait describing a
filesystem handle (list/exists with `&self`, plus `from_context`). A new
`UniversalReadFs: UniversalReadFileOps` subtrait adds the
`open(&self, path, options) -> Self::File` capability with `type File:
UniversalRead`. `UniversalRead` no longer extends `UniversalReadFileOps`
and is purely a file-handle trait.
This separates "filesystem instance" from "file handle". Backends that
need per-instance configuration (S3 bucket name + credentials, mmap
default advice, io_uring runtime, block-cache controller `Arc`) gain a
typed home in `Self::ContextConfig`, and `list_files`/`exists`/`open`
become `&self` methods on the filesystem handle.
Concrete filesystem handles introduced for the three existing backends:
- `MmapFs` — unit struct; `ContextConfig = ()`; produces `MmapFile`.
- `IoUringFs` — carries `prevent_caching`; `ContextConfig =
IoUringConfigContext`; produces `IoUringFile`. `IoUringConfigContext`
becomes the construction input rather than a per-open argument.
- `BlockCacheFs` — carries `Arc<CacheController>`; `ContextConfig =
BlockCacheConfigContext`; produces `CachedSlice`.
`TConfigContext` (universal builder methods) is kept so generic-over-`Fs`
code can still set cross-backend knobs via
`Fs::ContextConfig::default().with_prevent_caching(true)`.
Wrappers (`ReadOnly<S>`, `TypedStorage<S, T>`, `StoredStruct<S, T>`),
higher-level storages (`StoredBitSlice<S>`, `UniversalHashMap<K, V, S>`)
keep `<S: UniversalRead>` parameterization but their `open(...)`
constructors now grow a `fs: &Fs` argument bound by
`Fs: UniversalReadFs<File = S>`. `read_json_via` becomes
`read_json_via(fs: &Fs, path)`.
All test code and benches in `common` updated to construct
`MmapFs`/`IoUringFs` inline as needed. `common` compiles cleanly with
tests and benches. `gridstore`, `segment`, and `tonic` caller updates
are in flight in subsequent commits.
* WIP: gridstore + segment caller sweep (partial)
Threads `fs: &Fs` through gridstore's `BitmaskGaps`, `Bitmask`, `Pages`,
and `Gridstore::new`/`open`/`create_new_page`. Most segment callers
have `OpenOptions { extra: ... }` removed and `S::open(path, opts, ctx)`
sites updated mechanically but the trait change is not yet propagated.
Does NOT compile yet. Tracker still has static `S::open` calls, segment
generic constructors (`MmapInvertedIndex<S>`, `UniversalMapIndex`,
`StoredGeoMapIndex`, etc.) still call `S::open`/`S::list_files`/`S::exists`
statically — they need an `fs: &Fs` parameter added. Tonic API
`StorageReadService<S>` also unconverted.
Committed as branch checkpoint; cascade continues in subsequent work.
* gridstore: thread `fs: &Fs` through Bitmask, BitmaskGaps, Pages, Tracker
Per the new `UniversalReadFs` shape, every constructor/method that opens
files takes an `fs: &Fs` parameter. Gridstore is currently mmap-only,
so the top-level `Gridstore` / `GridstoreReader::open` callers in the
crate pass `&MmapFs` inline. Tests do the same.
gridstore lib + tests now compile cleanly. Segment + tonic cascade
still pending.
* WIP: segment caller sweep — dynamic_stored_flags first
* WIP: segment cascade - id_tracker partial
* common benches: update to new UniversalReadFs::open shape (clippy clean)
* segment flags: thread Fs through BufferedDynamicFlags / Bitvec / Roaring
DynamicStoredFlags::set_len now takes `fs: &Fs`. BufferedDynamicFlags
stores an `Arc<Fs>` so the flusher closure can call `set_len` on resize.
BitvecFlags and RoaringFlags expose a new `Fs` type parameter and the
flag tests now pass `Fs::default()` (MmapFs/IoUringFs via duplicate_item).
Concrete consumers (bool/null index, mmap dense/multi/sparse storages)
pin `Fs = MmapFs` and pass `&MmapFs` to inner opens.
* segment: thread Fs through field-index lifecycle methods
Apply the new UniversalReadFs::open shape across:
- full_text_index (MmapInvertedIndex, MmapFullTextIndex, UniversalPostings)
- geo_index (StoredGeoMapIndex build/open + tests + builders)
- numeric_index lifecycle (UniversalNumericIndex build/open)
- map_index lifecycle (UniversalMapIndex build/open)
- stored_point_to_values (open / from_iter)
Concrete consumers pin Fs = MmapFs and pass &MmapFs inline; generic
open paths thread `fs: &Fs` where Fs: UniversalReadFs<File = S>.
* segment: thread Fs through chunked vectors and id-tracker callers
- ChunkedVectors gains `Fs` generic so add_chunk can call create_chunk
after open. ChunkedVectorsRead/load_config and chunks::{read_chunks,
create_chunk} take `fs: &Fs`. Concrete callers (dense / multi-dense /
sparse / quantized) pass MmapFs inline.
- DenseVectorStorageImpl stores `fs: Fs` so `update_from` can reopen
ImmutableDenseVectors. ImmutableDenseVectors::open takes `fs: &Fs`.
- VectorStorageEnum DenseUring* variants thread IoUringFs alongside
IoUringFile.
- segment_builder + segment_constructor_base pass MmapFs to
ImmutableIdTracker::{new, open}.
- QuantizedStorage::from_file takes `fs: &Fs`; quantized_vectors callers
pass &MmapFs.
* segment: apply nightly rustfmt after Fs refactor
cargo +nightly fmt --all over the segment crate after the
UniversalReadFs cascade. No semantic changes.
* segment: thread Fs through benches and id-tracker tests
Update the dynamic-mmap-flags and buffered-update-bitslice benches to
the new UniversalReadFs::open shape (pass `&MmapFs`). Update the
immutable-id-tracker test suite to forward `&MmapFs` to
`from_in_memory_tracker` / `open`.
* uio: pin Fs via UniversalRead::Fs assoc type; per-call OpenExtra
Two design changes that fall out of the per-instance Fs refactor:
1. Bidirectional Fs ↔ File pinning. `UniversalRead::Fs:
UniversalReadFs<File = Self>` lets generic-over-`S` code refer to
`S::Fs` directly instead of carrying an extra `<Fs: UniversalReadFs<File = S>>`
generic param. `ReadOnly<S>` wraps a file but has no natural
filesystem; a phantom `ReadOnlyFs<S::Fs>` satisfies the constraint
while inherent `ReadOnly::open` keeps taking `&S::Fs` directly.
2. `prevent_caching` moves from filesystem-instance state to per-call
`UniversalReadFs::OpenExtra: Default`. Was previously a knob on
`IoUringConfigContext` / `IoUringFs`, conflating "how this fs is
built" with "how this file is opened." Now `IoUringFs::OpenExtra =
IoUringOpenExtra { prevent_caching }`; mmap and block-cache use `()`.
`IoUringConfigContext` is gone, `TConfigContext` slims to a `Default`
marker.
Tonic StorageReadService holds `Arc<S::Fs>` (was `PhantomData<S>`); its
`new()` builds via `S::Fs::from_context(default)` and the spawn_blocking
closures clone the Arc to call instance methods.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: cfg-gate IoUringOpenExtra import for non-linux builds
The IoUringOpenExtra reexport from `universal_io` is gated on
`target_os = "linux"`. The previous commit left an unconditional import
in `persisted_hashmap/tests.rs`, breaking macOS/Windows CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: migrate simple_disk_cache to per-instance Fs API
PR #9097 (merged into dev concurrently with this branch) introduced a
`DiskCache<R: UniversalRead>` using the pre-refactor trait shape:
file-handle-as-filesystem (`R::open`, `R::list_files`), an
`OpenOptionsExtra` field on `OpenOptions`, and trait methods without
`&self`. The Fs-instance refactor on this branch removed all three.
Reshape `simple_disk_cache` to match the new design without breaking the
lazy mirror semantics:
- New `DiskCacheFs<R>` is the filesystem handle. Holds a clone of the
remote `R::Fs`; `list_files`/`exists` delegate; `from_context`
forwards to the inner Fs context. `open` constructs a `DiskCache<R>`
via the global `DiskCacheConfig`.
- `DiskCache<R>` now stores `remote_fs: R::Fs` + `remote_extra:
<R::Fs as UniversalReadFs>::OpenExtra`, so lazy remote opens go
through `self.remote_fs.open(path, options, extra)` instead of the
removed `R::open`. `open_with_config` takes the remote Fs + extra
explicitly (no more hard-coded `prevent_caching: true`; callers pass
the appropriate `OpenExtra`).
- `UniversalRead for DiskCache<R>` now declares `type Fs =
DiskCacheFs<R>` (no more `open` method on the file trait).
- Propagate the necessary bounds (`R::Fs: Clone`,
`<R::Fs as UniversalReadFs>::OpenExtra: Clone`,
`R::OwnedReadPipeline<u8, Range<u32>>: Send`) through `pipeline.rs`
free functions and impl blocks that reach into `DiskCache::remote` /
`local_state`.
- Drop the now-removed `extra: _` destructure in `LocalState::new`.
- Tests construct the remote Fs via `R::Fs::from_context(Default::default())`
and exercise `DiskCache::open_with_config`. 17 simple_disk_cache
tests pass; the 3 `empty_read_does_not_materialize_local_file`
failures pre-exist on dev (verified) and are unrelated.
`mold -run cargo clippy --all-targets` clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt on simple_disk_cache migration
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt for local_state imports after rebase
Co-authored-by: Cursor <cursoragent@cursor.com>
* uio: split DiskCacheFs into its own module
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: replace TConfigContext with OpenExtra trait; move DiskCacheConfig onto DiskCacheFs
- Drop the empty TConfigContext marker; ContextConfig is now unconstrained
so backends can require explicit construction.
- Add OpenExtra trait with with_prevent_caching for backend-agnostic
per-call knobs; impl for () (no-op) and IoUringOpenExtra.
- DiskCacheFs now carries Arc<DiskCacheConfig> via the new
DiskCacheFsContext<C>; the prefill flow moves from the deleted
open_with_config into DiskCacheFs::open so Populate::Blocking /
PreferBackground work through the trait API.
- Remove the DiskCacheConfig global; callers must construct the context
explicitly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt after OpenExtra refactor
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: fix spelling — Implementors → Implementers
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: possible panic insetad of error propagation
* fix: missing cfg annotation
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* refactor: pass point_id into handle_point_version_and_failure
Drop the external_id reverse lookup used for error_status correlation
and take the external point_id as an explicit argument instead. All
callers already have it in scope, and decoupling it from op_point_offset
keeps error correlation correct in flows where the old internal_id is
tombstoned before the recovery check runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: filter_deferred_and_deleted also consults the deleted bitslice
Field-index primary-clause iterators (and the analogous plain/sparse
vector-index paths) are routed through PointMappingsRefEnum to apply the
deferred-threshold cutoff. The old `filter_deferred` only applied that
threshold, so any soft-deleted internal id sitting below the cutoff
slipped through whenever its field-index posting was still live. This
was fine while the only source of mid-range tombstones was the
deferred-tail design, but it breaks the moment a tombstone can land
anywhere in the id range.
Add an unconditional deleted-bitslice check (single bit test per
element) and rename to `filter_deferred_and_deleted` so the contract is
visible at the call site. The Either split is preserved so the
no-threshold path still avoids the cutoff comparison.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: cargo +nightly fmt
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* style: collapse error-status recovery match arm
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduce a `pub trait Item: bytemuck::Pod + Send` marker (mirroring the
existing `UserData` pattern) and use it in place of `T: bytemuck::Pod`
on the read side of `UniversalRead`, its pipeline traits, and all
impls/wrappers. Some implementations buffer `Vec<MaybeUninit<T>>` that
may be transferred across threads in future backends; tightening the
bound makes that explicit at the trait level instead of leaving callers
to add `+ Send` ad-hoc. Write paths keep the looser `bytemuck::Pod`
bound since they only borrow `&[T]`.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* dataset: add features to fs-err
Reason: fix `cargo check --all-targets -p dataset`. Feature unification
pulls `fs-err/debug` + `fs-err/tokio` without `fs-err/debug_tokio`.
* dataset: run `cargo metadata` to get target dir
Reason: avoid re-downloading in separate worktrees. The old
`cargo locate-project` ignores custom target dirs.
* sparse/benches/search: cache vectors/indices
The segment builder evicts the page cache of vector storage, quantized
vectors, payload, payload index and the vector index after a build to
avoid cache pollution, but the id tracker was never cleared. Its on-disk
files (mappings, versions, deleted bitslice) are written during the build
and stay resident in the page cache, so after each optimization the id
tracker files linger as cache even though they are meant to be on-disk
only (expected_cache_bytes == 0).
Add `IdTracker::clear_cache` (default no-op) plus a `clear_cache_if_on_disk`
policy wrapper, and implement the eviction for the mutable and immutable
trackers:
- ImmutableIdTracker pages out its two mmap-backed storages via madvise
and drops the RAM-loaded mappings file via fadvise(DONTNEED).
- MutableIdTracker drops its append-only log files via fadvise(DONTNEED).
The builder calls `clear_cache_if_on_disk`, mirroring the payload index.
The id tracker has no on-disk mode yet, so this always clears for now;
a TODO marks where to gate it once an on-disk mode exists.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous description was outdated: it claimed that enabling this
option "blocks updates at the request level" until segments are
re-optimized. In practice the implementation uses "deferred points":
new points written to large unoptimized segments are persisted but
excluded from read/search results until the segments are optimized.
Updates are not blocked; only `wait=true` clients are made to wait for
the deferred points to become visible. Update this in the REST schema
(via `OptimizersConfig` / `OptimizersConfigDiff`), in the gRPC proto,
in the edge config docstrings, and regenerate the OpenAPI bundle via
`tools/generate_openapi_models.sh`.
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix proxy deleted_mask race dropping live points from scored search
ProxySegment::new snapshots the wrapped segment's deleted_mask while the
optimizer holds only the upgradable-read lock, before the write lock freezes
the segment. An upsert racing onto the still-appendable wrapped segment in that
window lands at an internal offset past the snapshot. The scored search path
(PlainVectorIndex::search) consults the proxy mask in place of the segment's
live deleted state, and check_deleted_condition defaults any out-of-range
offset to deleted (unwrap_or(true)) — so the live point is silently dropped
from filtered KNN while scroll/count/retrieve still return it.
Re-snapshot deleted_mask in the optimizer once the holder write lock is held
(segment frozen) and before the proxy goes live, so the mask covers the
segment's full final point range. The fresh read also captures any deletes
that raced in, closing the ghost direction too.
Adds a proxy-level regression test that reproduces the race without the
model-testing harness.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Make proxy deleted_mask sync a type-state, read once
ProxySegment::new now returns UnsyncedProxySegment instead of a
ready-to-use ProxySegment. The deleted_mask snapshot is deferred to
UnsyncedProxySegment::finalize(), which reads the wrapped segment's
deleted bitvec exactly once. The only way to obtain a ProxySegment
(and thus put it in a SegmentHolder) is via finalize(), so the sync
under the holder write lock can no longer be forgotten, and the mask
is no longer read twice (once in new, once in resync).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Allow clippy::new_ret_no_self on ProxySegment::new
new deliberately returns the unsynced UnsyncedProxySegment stage rather
than Self, since a usable ProxySegment only exists once deleted_mask is
synced via finalize().
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Name the real constructor UnsyncedProxySegment::new, keep ProxySegment::new for tests
Instead of allowing clippy::new_ret_no_self on a ProxySegment::new that
returned UnsyncedProxySegment, give the two-phase constructor its natural
home: UnsyncedProxySegment::new returns Self and is what production code
(optimize, snapshot) uses, finalizing under the holder write lock.
ProxySegment::new becomes a #[cfg(feature = "testing")] convenience that
builds and finalizes in one step (returns Self), so existing test call
sites stay terse and don't need an explicit .finalize(). The shard
testing feature is enabled for both shard's own tests and collection's
dev-dependency, and excluded from production builds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* test(consensus): snapshot transfer with set-payload for missing points
Add a consensus test reproducing a shard transfer abort caused by
set-payload (and other partial-update) operations targeting non-existing
points.
Such operations are written to the WAL before the point-existence check
rejects them, so the queue proxy replays them to the receiver during a
snapshot transfer. The receiver applies them with force=true, bypassing
the missing-point tolerance in handle_failed_replicas, and the operation
hard-fails with `NotFound: No point with id ... found`. Under sustained
load the bounded queue/driver retries are exhausted, the receiver replica
is marked Dead and the transfer is aborted.
The test keeps the missing-point load running while checking the result,
because consensus auto-recovers Dead replicas: stopping the load first
would let the next recovery transfer succeed and mask the bug. It must
FAIL on current code and PASS once the receiver tolerates missing-point
operations during recovery.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(transfer): skip non-transient errors during queue proxy WAL replay (#9126)
During a shard transfer the queue proxy replays operations from the
sender's WAL to the receiver. Some of these are partial updates
(set_payload, update_vectors, ...) that the receiver rejects with a
non-transient error - most commonly `NotFound: No point with id ...`
for a point that does not exist on the receiver, but also any other
client-caused bad request.
These operations were replayed from the WAL, meaning they were already
applied (and rejected the same way) on the sender, so the sender's state
reflects them as no-ops. Propagating the error aborted the whole transfer;
under sustained load the bounded queue/driver retries were exhausted and
the receiver replica was marked Dead.
Handle the error where the semantic context lives - the transmitter
(`transfer_operations_batch`): skip operations the remote rejects with a
non-transient error and keep going, while still propagating transient
errors so the caller retries delivery. Because the batch update API aborts
at the first failing operation, a non-transient batch error falls back to
one-by-one sending to isolate and skip the offending operation(s).
This complements PR #5991, which handles missing points on the live
forwarded-update path (handle_failed_replicas) but not the WAL replay path.
Fixes the abort reproduced by
test_shard_snapshot_transfer_with_missing_point_updates.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: serialize optimization proxy install against shard updates
execute_optimization captures `target_config` from the optimizer's frozen
config and then wraps source segments in proxies. Between those two points,
`CollectionUpdater::update` can apply a `CreateVectorName(V)` to the source
segments via `apply_segments`, leaving the optimizer with sources that have
V but a target_config that does not. The optimization then produces a
merged segment without V, and a follow-up optimization (running with the
refreshed config that includes V) fails to use that segment as a source:
"Cannot update from other segment because it is missing vector name X".
Close the race by extending the scope of the existing
`LockedSegmentHolder::acquire_updates_lock` to cover the proxy install
window. `CollectionUpdater::update` already takes this lock before
processing any shard update, so concurrent writers wait until proxies are
in place — at which point further mutations hit the proxies (recorded as
intent and propagated to the merged segment in `finish_optimization`)
instead of the originals. The guard is dropped right after proxy install so
the slow build phase does not extend it.
Tests:
- Three `SegmentBuilder::update` tests document the precondition the lock
now guarantees: with a target schema that adds a named vector the source
lacks, update errors with "missing vector name X". Quantized and
mixed-source variants exercise the same error path.
- `test_optimize_blocks_proxy_install_on_updates_lock` asserts the
invariant directly: while the updates lock is held, proxies are not yet
installed. Verified to fail when the new guard is removed (otherwise it
passes because `finish_optimization` also takes the same lock, so a
naive "did optimize finish?" check would not catch a missing proxy-install
guard).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(optimize): finish_optimization lock order; drop redundant tests
Address review of #9110:
1. `finish_optimization` was acquiring `upgradable_read` before
`acquire_updates_lock`, while the new guard at the start of
`execute_optimization` acquires them in the reverse order. With two
optimizer threads in flight, thread A in `finish_optimization` could
hold `upgradable_read` and wait on `updates_lock` while thread B at the
top of `execute_optimization` held `updates_lock` and waited on
`upgradable_read` (parking_lot allows only one upgradable reader),
deadlocking. Swap `finish_optimization` to take `updates_lock` first so
both halves agree.
2. Drop the quantized and mixed-source variants of the inverted
`SegmentBuilder` unit test — all three asserted the same error path
(the mismatch check fires before quantization training or per-source
branching), so only one is useful as documentation of the precondition.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: reject source-superset schema mismatch at SegmentBuilder
Drop the lock approach (deadlocked test_continuous_snapshot) and fix the bug
at the merge layer instead.
Snapshot's proxy_all_segments_and_apply acquires the segment_holder
upgradable_read first and then takes acquire_updates_lock tactically inside
the snapshot operation. The previous commits' lock-extension acquired
updates_lock before upgradable_read, so a snapshot in flight and an
optimization just entering execute_optimization could deadlock holding each
other's required next lock. Snapshot cannot easily reverse its order — that
would hold updates_lock for the entire snapshot duration, blocking all
writes.
Move the fix to where the actual harm happens: SegmentBuilder::update
iterates the target's vector_data and silently drops source vectors that
aren't in target. That silent drop is what produces the broken merged
segment in the CreateVectorName-vs-optimizer race. Add a check that every
source vector name is in the target schema; the optimization aborts cleanly
on mismatch and the next round (with refreshed config) merges correctly.
This is strictly stronger than the lock: the lock only closed the window
where V arrived *during* the proxy-install region. The schema check catches
both that window and the window where V's apply_segments completed before
the optimizer's lock acquisition.
Diff is contained to lib/segment; no locking changes, no cross-crate
plumbing. test_continuous_snapshot passes again.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(segment_builder): use Cancelled instead of ServiceError for schema mismatch
ServiceError flips the shard to RED status (via `report_optimizer_error` →
`segments.optimizer_errors`) and stays sticky until the next
`recreate_optimizers_blocking` clears it. That's the right shape for
hardware/IO failures but wrong for the schema-mismatch case here, which is
an expected, recoverable race outcome — the next optimizer round with a
refreshed target_config merges the same originals cleanly.
`Cancelled` is the variant the optimization worker treats as a recoverable
cancellation: logged at debug, tracker marked Cancelled, no
`report_optimizer_error` call, no RED status.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(segment_builder): also use Cancelled for the existing target-superset error
The existing "missing vector name" check at the start of the merge loop
also fires during a race — specifically the optimizer-vs-DeleteVectorName
shape, where V is removed from originals before J wraps proxies but J's
frozen target_config still has V. Like the new source-superset check, this
is an expected, recoverable race outcome, so use Cancelled instead of
ServiceError to avoid flipping the shard to RED.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Recreate optimizers in non-blocking fashion from consensus calls
* Update comments
* On optimizer config update failure, report error status to local shard
* Add a test to confirm we don't block consensus
* Rerun recreation if called multiple times
* Use atomics instead
* Move to the bottom
* Reformat
* Use cluster default shard transfer method for fallback
When a WAL delta automatic transfer fails, the driver falls back to the
method passed via `fallback_method`. This was hard-coded to
`StreamRecords` (unless `prevent_unoptimized` was enabled), which is
inconsistent with the 1.18.0+ default of `Snapshot` and ignores any
configured `default_shard_transfer_method`.
Use `Collection::default_shard_transfer_method()` instead, so the
fallback matches the cluster default. With `prevent_unoptimized` we
still pin to `Snapshot` to preserve deferred point state exactly (raw
segment copy); stream_records would send deferred points but they
would not be deferred on the target.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Avoid wal_delta fallback, update fallback test
If the cluster default transfer method is wal_delta, the same-method
fallback would be refused by the driver. Use snapshot as a safe fallback
in that case; snapshot is also the 1.18.0+ default.
Update test_shard_wal_delta_transfer_fallback to assert the new
snapshot fallback (was stream_records).
Co-authored-by: Cursor <cursoragent@cursor.com>
* Address clippy wildcard_enum_match_arm
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add test to show cleanup may conflict with update queue
* When invoking clean task, first wait for current update queue
* Don't hold shard holder lock for a long time
* Also assert the clean task finished completely
EncodedVectorsTQ was the only quantizer that did not override the
EncodedVectors::heap_size_bytes() trait method, so it fell back to the
default of 0. For the RAM-backed variants (TQRam/TQRamMulti) this meant
the entire resident quantized dataset was reported as 0 bytes and
misclassified as fully on-disk by the MemoryReporter; the always-resident
quantizer tables (rotation + TQ+ error-correction vectors) and encoding
buffer were also uncounted for every variant.
Make heap_size_bytes() a required trait method (remove the default impl)
so every quantizer must account for its own heap explicitly, then add the
missing TurboQuant implementation: storage backend + quantizer tables +
encoding buffer.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [AI + manual] initial impl
[AI] read_batch which actually batches
manual nits
[AI] better handling of local and remote paths
manual refactor, respect open options
don't delete local file
dumbify read_batch
we want to refactor it anyway
simplify
rename to `DiskCache` in `simple_disk_cache` module
* refactor to use always use ReadPipeline
pass meta to remote pipeline
* nits
* run tests for more Remotes
* fix no more <T> in UniversalRead
* fmt
* chore(deps): unify roaring as workspace dep, move duplicate to dev-deps
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* BorrowedReadPipeline/OwnedReadPipeline
* `fn new(file: &Self::File)` -> `fn new(file: Self::File)`
Also, partially revert some related changes from the previous commit.
* refactor: move deferred-point ownership into the ID tracker
Re-implements the idea from #8512 against current `dev`.
Deferred-point state (`deferred_internal_id` + `deferred_deleted_count`)
moves out of `Segment.deferred_point_status` and the cached
`SparseVectorIndex.deferred_internal_id` field into `PointMappings`,
exposed through `IdTrackerRead`. The threshold is set once at
`MutableIdTracker::open` time; `PointMappings::drop` now maintains the
deleted counter inline (with double-delete protection), removing the
manual increment in `delete_point_internal` and the
`calculate_deleted_deferred_point_count` rescan.
Read paths consume the threshold through the id tracker:
- The segment read view drops the `deferred_point_status` field and
`with_view` no longer threads it in; `read_view/{deferred,info}.rs`
call `self.id_tracker.deferred_*()` directly.
- `SparseVectorIndex` no longer stores its own copy and its
`update_vector` / search debug-assert read from
`self.id_tracker.borrow().deferred_internal_id()`.
- `VectorQueryContext.deferred_internal_id` and the
`SegmentQueryContext::get_vector_context` parameter are gone; the
three downstream readers (`plain_vector_index`, sparse search, sparse
`update_vector`) consult their own id tracker.
`PointMappingsRefEnum` centralises the dispatch:
- `iter_internal_with_behavior(DeferredBehavior)` replaces ad-hoc
branches in `iter_filtered_points` impls.
- `external_iter_cutoff(DeferredBehavior)` covers iterators sourced
outside the mapping (field-index outputs in
`struct_payload_index::iter_filtered_points`).
- The internal `deferred_internal_id()` accessor is private; the raw
threshold no longer leaks to consumers.
- `iter_from_visible` / `iter_random_visible` read the mapping's own
threshold; callers that previously passed
`DeferredBehavior::apply(...)` now branch on
`deferred_behavior.include_all_points()` (scroll / order_by) or simply
drop the argument (sampling / facet).
`PayloadIndexRead::query_points` drops the now-redundant
`deferred_internal_id` parameter; `iter_filtered_points` takes
`DeferredBehavior` directly so HNSW build/search can request
`IncludeAll` while normal reads request `Exclude`.
RocksDB-related parts of the original PR are skipped — that tracker is
already gone from `dev`.
Tests adapted: sites that mutated `segment.deferred_point_status`
directly now construct a parallel non-deferred segment via
`create_deferred_segment(..., 0)` for comparison;
`test_deleted_deferred_point_count` reads counters through the id
tracker.
See `docs/plans/deferred-points-owned-by-id-tracker.md` for the design
write-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(benches): drop stale deferred_internal_id arg from query_points calls
The boolean / range / conditional bench files weren't built by
`cargo test -p segment`, so they slipped through. `cargo clippy
--workspace --all-targets` catches them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: drop id_tracker / point_mappings args from iter_filtered_points
Both impls already hold an id tracker on `self`:
- `StructPayloadIndexReadView` carries `id_tracker: &'a I`, so
`self.id_tracker.point_mappings()` borrows from `'a` and the lazy
iterator chain keeps working unchanged.
- `PlainPayloadIndex` carries `id_tracker: Arc<AtomicRefCell<...>>`,
where the mapping borrow is local; collect into a `Vec` and return
`into_iter()`. PlainPayloadIndex::iter_filtered_points has no direct
callers — only `query_points` was using it — so eager collection is a
non-issue.
While here, take `self` by value on `iter_internal_visible`,
`iter_from_visible`, `iter_random_visible`,
`iter_internal_with_behavior`, and `external_iter_cutoff`.
`PointMappingsRefEnum` is `Copy`; this matches the existing
`iter_internal` / `iter_from` / `iter_random` shape and lets the
iterator outlive a local `let point_mappings = ...;` binding.
The HNSW `condition_points` helper drops its now-unused `id_tracker`
parameter.
All callers (sampling, scroll, order_by, facet ×2, hnsw build/search)
just drop the two arguments.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: ignore /docs/plans/ and untrack the previously-committed plan
`docs/plans/` is a scratch directory for per-feature planning notes —
not something we want under source control. Add it to `.gitignore` and
drop the deferred-points plan that slipped into history; the design is
captured in the PR description.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: replace external_iter_cutoff with filter_deferred iterator wrapper
Instead of exposing a raw `Option<PointOffsetType>` cutoff that every
caller has to apply with their own `.filter(...)`, give
`PointMappingsRefEnum` an iterator wrapper:
fn filter_deferred<I: Iterator<Item = PointOffsetType>>(
self,
iter: I,
deferred_behavior: DeferredBehavior,
) -> impl Iterator<Item = PointOffsetType>
It returns the iterator unchanged for `IncludeAll` (or when the mapping
has no threshold) and otherwise wraps it in a cutoff `.filter`,
dispatched via `itertools::Either` so the no-cutoff path stays
allocation-free.
The struct payload index's `iter_filtered_points` swaps its open-coded
filter for a single `point_mappings.filter_deferred(...)` call. The
deferred threshold no longer leaks out of `PointMappingsRefEnum`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: move deferred wrapping out of peek_top_all, gate it as test-only
`BatchFilteredSearcher::peek_top_all` baked the deferred cutoff into
its iterator construction, which was the last place outside
`PointMappingsRefEnum` that knew about the threshold. Split the
deleted-iteration concern out into a new accessor:
fn iter_not_deleted(&self) -> impl Iterator<Item = PointOffsetType> + 'a
It borrows `&'a BitSlice` directly (not via `&self`), so callers can
chain `filter_deferred` and then move `self` into `peek_top_iter`
without lifetime conflicts.
Sparse + plain vector index call sites now do:
let iter = id_tracker
.point_mappings()
.filter_deferred(searcher.iter_not_deleted(), DeferredBehavior::Exclude);
searcher.peek_top_iter(iter, &is_stopped)
leaving `BatchFilteredSearcher` completely ignorant of deferred state.
With deferred handling lifted out, `peek_top_all` itself is now used
only by tests (3 inline `#[cfg(test)] mod tests`, 1 integration test,
1 bench) — gate it under `#[cfg(feature = "testing")]` to match
`new_for_test`. Production code goes through the
`iter_not_deleted` + `filter_deferred` + `peek_top_iter` composition.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: optimize Segment::retrieve and thread user data through read_vectors
Two interlocking changes that together collapse the per-point lookups
and intermediate allocations in `Segment::retrieve` down to one
external-to-internal pass.
## `IdTrackerRead::resolve_external_ids` (new default trait method)
Single-pass translation of a `&[PointIdType]` slice into two parallel
vectors `(Vec<PointIdType>, Vec<PointOffsetType>)`. Folds deferred
filtering (compare offset against the threshold inline — no separate
`point_is_deferred` lookup) and missing-id errors (eager
`PointIdError`) into resolution.
Lives on the trait so the deferred threshold never leaks out of the id
tracker; the parallel-vector shape lets a future batched payload /
vector fetcher consume `&offsets` straight without unzipping. The
`appendable_flag` guard previously in `point_is_deferred` is gone:
non-appendable trackers always carry `deferred_internal_id() == None`
(set only via `MutableIdTracker::open`, guarded by the segment
constructor), so the check was load-bearing nowhere.
## User-data threading through `read_vectors`
`VectorStorageRead::read_vectors` now takes
`IntoIterator<Item = (U, PointOffsetType)>` and yields
`(U, PointOffsetType, CowVector)`. The user-data tag rides alongside
each offset all the way through, so callers can map results back into a
parallel array without keeping a separate `offset → ...` lookup table.
- Default trait impl: one-line per-key loop.
- Dense impl: `unzip()` into parallel `(Vec<U>, Vec<PointOffsetType>)`
in a single pass — same allocation count as before, just U riding
alongside.
- Enum delegations (`VectorStorageEnum`, `VectorStorageReadEnum`)
forward unchanged.
- `for_each_in_batch` and below stay untouched.
`SegmentReadView::vectors_by_offsets<U: Copy>` becomes a lazy filter
chain — no parallel `Vec<(orig_idx, offset)>` allocation. The dead
`SegmentReadView::read_vectors` helper is removed.
## `Segment::retrieve` end-to-end
Per N points / V vectors / payload:
| Operation | Before | After |
|----------------------------|---------------------|-------|
| `id_tracker.internal_id` | N × (1 + V + 1) | N |
| `id_tracker.external_id` | N × V | 0 |
| `point_is_deferred` | N (when applicable) | 0 |
| `offset_to_id` HashMap | N entries | none |
| `Vec` in `vectors_by_offsets` | 1 | 0 |
The vectors stage passes the external id as `read_vectors`'s user
data — the callback gets `id` directly without any index lookup.
The payload stage uses `payload_by_offset` against the already-resolved
offsets. The shape is also batch-friendly: swapping in a future
`IdTrackerRead::batch_internal_id` or `payload_index.batch_get_payload`
needs no changes outside the two call sites.
## Behavioural notes
- Missing-id now errors eagerly inside resolution, instead of in the
vectors stage (`WithVector::Bool(true)` / `Selector`) or payload
stage (`with_payload.enable`). The previous `WithVector::Bool(false)`
+ no-payload path silently inserted an empty record; that is now also
an error. None of the existing callers (search post-processing,
external retrieve API, the deferred-points test on tests/mod.rs:1179)
pass non-existent ids.
- Added a per-payload `check_stopped`; the vectors stage already had
`stop_if` on its iterator chain.
- `vector_by_offset` (the single-element helper) passes `()` as the
no-op user data.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: apply rustfmt to optimised retrieve / read_vectors paths
Pre-push hook failure on the previous commit was rustfmt. Same content,
formatted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* do not error out on missing points in retrieve
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Empty commit to open a PR
* Refactor `MultivectorOffsetsStorageMmap` to use `MmapFile` instead of `MmapSlice`
* simpler result types
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* fix: validate vector dimensions before WAL write for async upserts
When upserting points with wait=false (the default), dimension
mismatches were silently discarded during background processing.
The API returned 200 "acknowledged" but the points were never stored,
causing silent data loss with no error feedback to the user.
This adds an early dimension validation check in do_upsert_points()
that runs before the operation is written to WAL. This ensures that
dimension errors are returned to the client regardless of the wait
parameter, matching the behavior of wait=true.
The validation handles all vector types:
- Dense single vectors
- Multi-dense vectors
- Named vectors (dense, multi-dense, sparse)
- Sparse vectors are skipped (no fixed dimension)
Closes#9039
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor: move vector dimension validation into dedicated module
Extract validate_vector_dimensions and helper functions from update.rs
into src/common/validate_vectors.rs for better code organization.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: update shard update test for early dimension validation
The test expected a shard-level error message, but now dimension
mismatches are caught before reaching the shards. Update the assertion
to accept either the early validation error or the shard-level error.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: assert actual dimension error message in shard update test
Check for the descriptive error ("Vector dimension error: expected dim: 4, got 3")
rather than the generic shard failure wrapper.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* use generic in QuantizedMmapStorage
* rename to QuantizedStorage
* be explicit about S
* rename builder to `QuantizedStorageBuilder`
* rename file to `quantized_storage.rs`
