* ci(windows): skip IO-heavy tests that aren't OS-specific
On the Windows CI runner, several tests are 3-25x slower than on Ubuntu
purely due to slow filesystem IO. These tests exercise platform-agnostic
logic (optimizer, snapshot, WAL recovery, dedup, deferred points) and
are fully covered by the Linux and macOS jobs.
Mark them with `#[cfg_attr(target_os = "windows", ignore = "...")]` so:
- Windows CI skips them and finishes faster.
- They're still listed and runnable via `cargo test -- --ignored`
on Windows for local debugging.
Based on JUnit timings from CI run 26462785436 (PR #8827), this should
save ~5 minutes wall-clock on the Windows job, taking it closer to the
~13min Ubuntu and ~9min macOS jobs (currently 20m24s).
Tests affected:
- lib/wal: check_wal, check_last_index, check_clear, check_reopen,
check_truncate, check_prefix_truncate, test_prefix_truncate_parametric
- lib/edge/optimize: full tests module
- lib/segment deferred-point tests: read_operations,
dense_segment_combinations, sparse, facets
- lib/collection: snapshot_test, points_dedup, wal_recovery,
collection_test::test_ordered_read_api, snapshot_recovery_test
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert(ci/windows): keep WAL and WAL-recovery tests on Windows
Reviewer correctly pointed out that WAL is mmap-backed and has
substantial Windows-specific code paths:
- Different segment allocation (fs4 vs rustix::ftruncate)
- Windows-specific delete_windows() with mmap-drop + retry loop
- Windows-specific sync_all() because directory fsync is unavailable
- Windows-specific lock proxy file (directories aren't lockable)
So those tests genuinely need Windows coverage. Reverted skips for:
- lib/wal/src/lib.rs: all check_* tests and test_prefix_truncate_parametric
- lib/collection/src/tests/wal_recovery_test.rs: all three tests
Still skipped on Windows (no OS-specific code in their production paths):
- lib/edge/optimize.rs (no cfg(windows) in source)
- lib/segment deferred-point tests (segment/ has no cfg(windows))
- lib/collection snapshot/dedup tests (collection/ has no cfg(windows))
- lib/collection integration snapshot_recovery + ordered_read_api
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert(ci/windows): keep collection integration and snapshot_test
Per reviewer request, keep running these on Windows:
- lib/collection/tests/integration/* (snapshot_recovery_test,
collection_test::test_ordered_read_api)
- lib/collection/src/tests/snapshot_test.rs
These exercise higher-level collection/snapshot behavior that benefits
from cross-platform validation.
Remaining Windows skips (production code has no cfg(windows) branches):
- lib/edge/src/optimize.rs: 14 optimizer tests
- lib/segment/src/segment/tests/mod.rs: 4 deferred-point tests
- lib/collection/src/tests/points_dedup.rs: 2 dedup tests
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci(windows): also skip HNSW/quantization integration tests
Per reviewer, also skip these segment integration test modules on Windows:
- hnsw_quantized_search_test::* (25 tests)
- multivector_filtrable_hnsw_test::* (rstest cases)
- multivector_quantization_test::* (rstest cases)
- byte_storage_quantization_test::* (rstest cases)
- payload_index_test::test_struct_payload_index_nested_fields
These exercise pure HNSW/quantization correctness on top of standard
segment IO that is already covered by tests we keep running on Windows.
Adds ~930s of sequential time to the Windows skip list, bringing the
expected wall-clock saving from ~3 min to ~8-10 min.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: add io_bridge
* fix: cached dispatcher
* feat: add open_with_handle
* fix: naming
* fix: wording
* fix: rebase io_bridge onto split BorrowedReadPipeline/OwnedReadPipeline
* fix: linter
* feat: add S3 backend
* chore: remove io_bridge
* fix: linter
* fix: linter
* fix: read handle
* chore: simplify S3Source
* fix: s3 test
* feat: support multi runtime
* fix: clippy errors
* fix: review comments
* feat: add io design
* feat: add S3 backend
* chore: fix docs
* fix: dev changes
* chore: add some docs
* chore: remove explicit type
* feat: add new methods
* [WIP] review refactor
* fmt
* fix: bytes alignment
* fix: linter
* feat: remove Bytes
* fix: ci/cd
* fix: tests
* fix: tests
* refactor: simplify io_bridge pipeline to Handle-based dispatch
Replace the BridgeRuntime worker thread + request channel + boxed
BridgeRequest with direct tokio Handle usage:
- BridgeRuntime is now just an Arc<Runtime>; schedule() spawns the read
future via Handle::spawn instead of routing it through a dispatcher
thread. Removes BridgeRequest and the now-unreachable S3RuntimeShutDown
error variant.
- Guard against a panicking read task hanging wait() forever: the spawned
task catches unwinds and converts them into a TaskPanicked error reply,
so every scheduled slot is always answered.
- Encapsulate slot bookkeeping in PendingSlots, exposing only the needed
operations instead of a public map + counter.
- Split the grown pipeline.rs into a pipeline/ module (slots / inner /
borrowed / owned), de-duplicating the shared read-future construction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: move pipeline buffer ownership into the read future
Instead of the pipeline owning the destination Vec<T> in a slot map and
the future writing through a SendBytePtr raw pointer, let the future
allocate the buffer itself and return it through BridgeResponse. The
buffer crosses the worker-thread boundary as a normal move via the reply
channel, wrapped in a SendableVec<T> newtype that asserts Send for
T: bytemuck::Pod only.
This removes the entire unsafe SendBytePtr apparatus from the pipeline:
no raw pointer, no unsafe fn, no per-call-site unsafe blocks, no
heap-stability invariants. The only remaining unsafe in the crate is one
bounded `unsafe impl<T: Pod> Send for SendableVec<T>` with a trivially
true invariant (Pod types are plain bytes).
PendingSlots collapses back to PendingSlots<U>: slots no longer carry
buffers. AlignedBufWriter::from_raw_bytes (used only by the SendBytePtr
path) and its test are removed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: drop SendableVec wrapper now that Item: Send
UniversalRead's element type is now bound to `Item` (`Pod + Send`),
so the io_bridge pipeline no longer needs a hand-rolled `Send` wrapper
around `Vec<T>` to ship buffers through the reply channel. Replace
`SendableVec<T>` with `Vec<T>` end-to-end and tighten the local impls
to `T: Item`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: implement UniversalRead::reopen for BlobFile
BlobFile has no cached file metadata or mapping — `len()` queries the
object store fresh on each call — so reopen is a no-op, matching the
io_uring impl.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: add new fs impl for Blob
* fix: is_in_ram_or_mmap for S3
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(edge): expose WAL options via EdgeShard::load_with_wal_options
Adds a sibling method to EdgeShard::load that accepts custom
WalOptions, alongside the existing load() which keeps using
default_wal_options().
Motivation: embedded/mobile deployments (e.g. Flutter plugins on
iOS/Android) need much smaller WAL segments than the 32 MiB default —
typically 4 MiB. On filesystems with sparse files (APFS/ext4/F2FS)
the physical footprint is small, but the visible/reported size is
the full segment capacity, which is surfaced by iCloud backup, OS
size pickers, etc.
Changes:
* New EdgeShard::load_with_wal_options(path, config, wal_options).
* EdgeShard::load delegates to it with default_wal_options() — no
behavior change for existing callers.
* ensure_dirs_and_open_wal now takes WalOptions explicitly; called
from both EdgeShard::new (with default) and load_with_wal_options.
* WalOptions re-exported from edge crate root.
* Two regression tests in lib/edge/tests/wal_options.rs.
WAL options are intentionally a runtime parameter, not part of
EdgeConfig (which is persisted to edge_config.json) — different
processes may legitimately open the same shard with different WAL
options.
* test(edge): verify mismatching WAL options on reload preserve data
Addresses upstream review feedback (qdrant/qdrant#9067): demonstrate
that reloading an existing shard with WAL options different from the
ones it was created with is safe and preserves all previously written
points.
Two new tests in lib/edge/tests/wal_options.rs:
* reload_with_smaller_wal_capacity_after_upsert:
create shard with default 32 MiB WAL -> upsert point 42 ->
drop -> reload with 4 MiB WAL options -> point 42 still readable ->
upsert point 43 under smaller WAL -> count == 2.
* reload_with_larger_wal_capacity_after_upsert (symmetric):
create -> reload with 4 MiB -> upsert point 100 -> drop ->
reload with default 32 MiB -> point 100 readable -> upsert
point 101 -> count == 2.
Both pass. The WAL crate does not validate segment_capacity on
reload — existing segments on disk keep their original size,
subsequent appends honor the runtime options. This confirms the
PR description's claim that WalOptions is a runtime hint, not
persisted shard state.
* style(edge): cargo fmt for wal_options.rs mismatch tests
* refactor(edge): replace load_with_wal_options with builder-based options
Addresses upstream feedback (timvisee, generall): the sibling
constructor pattern doesn't scale as runtime configurability grows.
Replaces `EdgeShard::load_with_wal_options(path, config, wal_options)`
with `EdgeShard::load_with_options(path, config, EdgeShardOptions)`,
where `EdgeShardOptions` is a builder-style runtime-options struct.
* `EdgeShardOptions::new().with_wal_options(opts)` is the equivalent
of the old call.
* Adding new runtime options later (e.g. wal flush interval, initial
indexing threshold) is now an additive change — new builder method
on `EdgeShardOptions`, no new constructor variant on `EdgeShard`.
* `EdgeShardOptions` is intentionally not persisted (no
Serialize/Deserialize). The reload-mismatch tests above confirm
that WAL options are a runtime hint, not shard identity, so they
don't belong in `EdgeConfig` (which is persisted to
`edge_config.json`).
* `EdgeShard::load(path, config)` unchanged for existing callers —
delegates to `load_with_options(..., EdgeShardOptions::default())`.
Tests in `lib/edge/tests/wal_options.rs` migrated to the new shape;
all four still pass:
test load_with_options_accepts_custom_wal_capacity ... ok
test load_still_works_with_default_wal_options ... ok
test reload_with_smaller_wal_capacity_after_upsert ... ok
test reload_with_larger_wal_capacity_after_upsert ... ok
* refactor(edge): add fluent builders; move wal_options into EdgeConfig
Replaces the EdgeShardOptions side-channel with a single EdgeConfig that
carries wal_options inline, and introduces fluent builders for the three
user-facing config types.
* WalOptions now derives Clone/PartialEq/Eq/Serialize/Deserialize so it
can live inside EdgeConfig and round-trip through edge_config.json.
* EdgeConfig.wal_options (Option<WalOptions>) replaces EdgeShardOptions.
EdgeShard::load_with_options is folded into EdgeShard::load; both new
and load drive the WAL from config.wal_options.unwrap_or_default().
* New builders/ module hosts EdgeConfigBuilder, EdgeVectorParamsBuilder,
and EdgeSparseVectorParamsBuilder. Each builder has explicit per-field
storage and constructs its target via an exhaustive struct literal in
build(), so adding a field to the target forces a compile error in the
builder.
* publish example switched to the new builder API.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style(edge): cargo fmt after builder refactor
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [UIO] Split UniversalReadFileOps into filesystem + file traits
`UniversalReadFileOps` is now an instance-based trait describing a
filesystem handle (list/exists with `&self`, plus `from_context`). A new
`UniversalReadFs: UniversalReadFileOps` subtrait adds the
`open(&self, path, options) -> Self::File` capability with `type File:
UniversalRead`. `UniversalRead` no longer extends `UniversalReadFileOps`
and is purely a file-handle trait.
This separates "filesystem instance" from "file handle". Backends that
need per-instance configuration (S3 bucket name + credentials, mmap
default advice, io_uring runtime, block-cache controller `Arc`) gain a
typed home in `Self::ContextConfig`, and `list_files`/`exists`/`open`
become `&self` methods on the filesystem handle.
Concrete filesystem handles introduced for the three existing backends:
- `MmapFs` — unit struct; `ContextConfig = ()`; produces `MmapFile`.
- `IoUringFs` — carries `prevent_caching`; `ContextConfig =
IoUringConfigContext`; produces `IoUringFile`. `IoUringConfigContext`
becomes the construction input rather than a per-open argument.
- `BlockCacheFs` — carries `Arc<CacheController>`; `ContextConfig =
BlockCacheConfigContext`; produces `CachedSlice`.
`TConfigContext` (universal builder methods) is kept so generic-over-`Fs`
code can still set cross-backend knobs via
`Fs::ContextConfig::default().with_prevent_caching(true)`.
Wrappers (`ReadOnly<S>`, `TypedStorage<S, T>`, `StoredStruct<S, T>`),
higher-level storages (`StoredBitSlice<S>`, `UniversalHashMap<K, V, S>`)
keep `<S: UniversalRead>` parameterization but their `open(...)`
constructors now grow a `fs: &Fs` argument bound by
`Fs: UniversalReadFs<File = S>`. `read_json_via` becomes
`read_json_via(fs: &Fs, path)`.
All test code and benches in `common` updated to construct
`MmapFs`/`IoUringFs` inline as needed. `common` compiles cleanly with
tests and benches. `gridstore`, `segment`, and `tonic` caller updates
are in flight in subsequent commits.
* WIP: gridstore + segment caller sweep (partial)
Threads `fs: &Fs` through gridstore's `BitmaskGaps`, `Bitmask`, `Pages`,
and `Gridstore::new`/`open`/`create_new_page`. Most segment callers
have `OpenOptions { extra: ... }` removed and `S::open(path, opts, ctx)`
sites updated mechanically but the trait change is not yet propagated.
Does NOT compile yet. Tracker still has static `S::open` calls, segment
generic constructors (`MmapInvertedIndex<S>`, `UniversalMapIndex`,
`StoredGeoMapIndex`, etc.) still call `S::open`/`S::list_files`/`S::exists`
statically — they need an `fs: &Fs` parameter added. Tonic API
`StorageReadService<S>` also unconverted.
Committed as branch checkpoint; cascade continues in subsequent work.
* gridstore: thread `fs: &Fs` through Bitmask, BitmaskGaps, Pages, Tracker
Per the new `UniversalReadFs` shape, every constructor/method that opens
files takes an `fs: &Fs` parameter. Gridstore is currently mmap-only,
so the top-level `Gridstore` / `GridstoreReader::open` callers in the
crate pass `&MmapFs` inline. Tests do the same.
gridstore lib + tests now compile cleanly. Segment + tonic cascade
still pending.
* WIP: segment caller sweep — dynamic_stored_flags first
* WIP: segment cascade - id_tracker partial
* common benches: update to new UniversalReadFs::open shape (clippy clean)
* segment flags: thread Fs through BufferedDynamicFlags / Bitvec / Roaring
DynamicStoredFlags::set_len now takes `fs: &Fs`. BufferedDynamicFlags
stores an `Arc<Fs>` so the flusher closure can call `set_len` on resize.
BitvecFlags and RoaringFlags expose a new `Fs` type parameter and the
flag tests now pass `Fs::default()` (MmapFs/IoUringFs via duplicate_item).
Concrete consumers (bool/null index, mmap dense/multi/sparse storages)
pin `Fs = MmapFs` and pass `&MmapFs` to inner opens.
* segment: thread Fs through field-index lifecycle methods
Apply the new UniversalReadFs::open shape across:
- full_text_index (MmapInvertedIndex, MmapFullTextIndex, UniversalPostings)
- geo_index (StoredGeoMapIndex build/open + tests + builders)
- numeric_index lifecycle (UniversalNumericIndex build/open)
- map_index lifecycle (UniversalMapIndex build/open)
- stored_point_to_values (open / from_iter)
Concrete consumers pin Fs = MmapFs and pass &MmapFs inline; generic
open paths thread `fs: &Fs` where Fs: UniversalReadFs<File = S>.
* segment: thread Fs through chunked vectors and id-tracker callers
- ChunkedVectors gains `Fs` generic so add_chunk can call create_chunk
after open. ChunkedVectorsRead/load_config and chunks::{read_chunks,
create_chunk} take `fs: &Fs`. Concrete callers (dense / multi-dense /
sparse / quantized) pass MmapFs inline.
- DenseVectorStorageImpl stores `fs: Fs` so `update_from` can reopen
ImmutableDenseVectors. ImmutableDenseVectors::open takes `fs: &Fs`.
- VectorStorageEnum DenseUring* variants thread IoUringFs alongside
IoUringFile.
- segment_builder + segment_constructor_base pass MmapFs to
ImmutableIdTracker::{new, open}.
- QuantizedStorage::from_file takes `fs: &Fs`; quantized_vectors callers
pass &MmapFs.
* segment: apply nightly rustfmt after Fs refactor
cargo +nightly fmt --all over the segment crate after the
UniversalReadFs cascade. No semantic changes.
* segment: thread Fs through benches and id-tracker tests
Update the dynamic-mmap-flags and buffered-update-bitslice benches to
the new UniversalReadFs::open shape (pass `&MmapFs`). Update the
immutable-id-tracker test suite to forward `&MmapFs` to
`from_in_memory_tracker` / `open`.
* uio: pin Fs via UniversalRead::Fs assoc type; per-call OpenExtra
Two design changes that fall out of the per-instance Fs refactor:
1. Bidirectional Fs ↔ File pinning. `UniversalRead::Fs:
UniversalReadFs<File = Self>` lets generic-over-`S` code refer to
`S::Fs` directly instead of carrying an extra `<Fs: UniversalReadFs<File = S>>`
generic param. `ReadOnly<S>` wraps a file but has no natural
filesystem; a phantom `ReadOnlyFs<S::Fs>` satisfies the constraint
while inherent `ReadOnly::open` keeps taking `&S::Fs` directly.
2. `prevent_caching` moves from filesystem-instance state to per-call
`UniversalReadFs::OpenExtra: Default`. Was previously a knob on
`IoUringConfigContext` / `IoUringFs`, conflating "how this fs is
built" with "how this file is opened." Now `IoUringFs::OpenExtra =
IoUringOpenExtra { prevent_caching }`; mmap and block-cache use `()`.
`IoUringConfigContext` is gone, `TConfigContext` slims to a `Default`
marker.
Tonic StorageReadService holds `Arc<S::Fs>` (was `PhantomData<S>`); its
`new()` builds via `S::Fs::from_context(default)` and the spawn_blocking
closures clone the Arc to call instance methods.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: cfg-gate IoUringOpenExtra import for non-linux builds
The IoUringOpenExtra reexport from `universal_io` is gated on
`target_os = "linux"`. The previous commit left an unconditional import
in `persisted_hashmap/tests.rs`, breaking macOS/Windows CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: migrate simple_disk_cache to per-instance Fs API
PR #9097 (merged into dev concurrently with this branch) introduced a
`DiskCache<R: UniversalRead>` using the pre-refactor trait shape:
file-handle-as-filesystem (`R::open`, `R::list_files`), an
`OpenOptionsExtra` field on `OpenOptions`, and trait methods without
`&self`. The Fs-instance refactor on this branch removed all three.
Reshape `simple_disk_cache` to match the new design without breaking the
lazy mirror semantics:
- New `DiskCacheFs<R>` is the filesystem handle. Holds a clone of the
remote `R::Fs`; `list_files`/`exists` delegate; `from_context`
forwards to the inner Fs context. `open` constructs a `DiskCache<R>`
via the global `DiskCacheConfig`.
- `DiskCache<R>` now stores `remote_fs: R::Fs` + `remote_extra:
<R::Fs as UniversalReadFs>::OpenExtra`, so lazy remote opens go
through `self.remote_fs.open(path, options, extra)` instead of the
removed `R::open`. `open_with_config` takes the remote Fs + extra
explicitly (no more hard-coded `prevent_caching: true`; callers pass
the appropriate `OpenExtra`).
- `UniversalRead for DiskCache<R>` now declares `type Fs =
DiskCacheFs<R>` (no more `open` method on the file trait).
- Propagate the necessary bounds (`R::Fs: Clone`,
`<R::Fs as UniversalReadFs>::OpenExtra: Clone`,
`R::OwnedReadPipeline<u8, Range<u32>>: Send`) through `pipeline.rs`
free functions and impl blocks that reach into `DiskCache::remote` /
`local_state`.
- Drop the now-removed `extra: _` destructure in `LocalState::new`.
- Tests construct the remote Fs via `R::Fs::from_context(Default::default())`
and exercise `DiskCache::open_with_config`. 17 simple_disk_cache
tests pass; the 3 `empty_read_does_not_materialize_local_file`
failures pre-exist on dev (verified) and are unrelated.
`mold -run cargo clippy --all-targets` clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt on simple_disk_cache migration
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt for local_state imports after rebase
Co-authored-by: Cursor <cursoragent@cursor.com>
* uio: split DiskCacheFs into its own module
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: replace TConfigContext with OpenExtra trait; move DiskCacheConfig onto DiskCacheFs
- Drop the empty TConfigContext marker; ContextConfig is now unconstrained
so backends can require explicit construction.
- Add OpenExtra trait with with_prevent_caching for backend-agnostic
per-call knobs; impl for () (no-op) and IoUringOpenExtra.
- DiskCacheFs now carries Arc<DiskCacheConfig> via the new
DiskCacheFsContext<C>; the prefill flow moves from the deleted
open_with_config into DiskCacheFs::open so Populate::Blocking /
PreferBackground work through the trait API.
- Remove the DiskCacheConfig global; callers must construct the context
explicitly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt after OpenExtra refactor
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: fix spelling — Implementors → Implementers
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: possible panic insetad of error propagation
* fix: missing cfg annotation
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* test: wait for WAL flock release on peer restart in dirty-shard test
The flake addressed by #9124 (bumping `wait_for_peer_online` to 60s) was
misdiagnosed as CPU contention. Logs show the restarted peer panics within
~1s of startup at `consensus_wal.rs:36` with:
Wal error: Can't init WAL: Kind(WouldBlock)
Panic: Can't open consensus WAL: Kind(WouldBlock)
`wal::Wal::open` calls `fs4::FileExt::try_lock` (non-blocking `flock`) on
the WAL directory fd. After `p.kill()` (SIGKILL + waitpid) the kernel
normally releases the killed peer's flock immediately, but under
pytest-xdist load there is a small window where it lags. The fresh peer's
startup then races and panics. After the panic `/readyz` never returns
200, so neither 30s nor 60s rescues the test.
Fix: add a `wait_for_wal_unlocked` helper that polls both the consensus
WAL directory and the local-shard WAL directory with the same exclusive
non-blocking flock that qdrant uses, and call it after every `p.kill()`
that is followed by a `start_peer` on the same `peer_dir`. The two
60s timeouts are restored to the default 30s now that the underlying
race is gone.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: fail fast if sync restart crashes in dirty-shard test
The sync restart in `test_dirty_shard_survives_update_collection` used
plain `wait_for_peer_online(sync_uri)`, which only polls `/readyz`. If
the freshly started peer panics on startup (e.g. WAL `WouldBlock`), the
test waits the full 30s timeout and then reports a `/readyz` timeout
instead of the actual panic message and exit code.
Switch the sync restart to `wait_for_peer_online_or_crash(...)` (same
helper already used for the dirty restart). On crash it dumps the peer
log tail so the next CI failure shows the real reason instead of a
generic timeout.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: drop speculative WAL flock wait, keep crash-detection switch
Reverts the `wait_for_wal_unlocked` helper that polled the WAL
directories with non-blocking flock. The kernel-side flock race it was
guarding against could not be reproduced in isolation (0/300 iters of
SIGKILL+wait+re-flock on bare Linux), so it was speculative.
Kept:
- Sync restart now uses `wait_for_peer_online_or_crash(...)` instead of
plain `wait_for_peer_online`, so any startup panic surfaces fast with
the actual log tail instead of hiding behind a 30s `/readyz` timeout.
- The two 60s timeouts bumped in #9124 are restored to the default 30s.
If the flake recurs in CI, the new failure output will tell us the
actual cause (panic message + exit code), which is more useful than
papering over it.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor: pass point_id into handle_point_version_and_failure
Drop the external_id reverse lookup used for error_status correlation
and take the external point_id as an explicit argument instead. All
callers already have it in scope, and decoupling it from op_point_offset
keeps error correlation correct in flows where the old internal_id is
tombstoned before the recovery check runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: filter_deferred_and_deleted also consults the deleted bitslice
Field-index primary-clause iterators (and the analogous plain/sparse
vector-index paths) are routed through PointMappingsRefEnum to apply the
deferred-threshold cutoff. The old `filter_deferred` only applied that
threshold, so any soft-deleted internal id sitting below the cutoff
slipped through whenever its field-index posting was still live. This
was fine while the only source of mid-range tombstones was the
deferred-tail design, but it breaks the moment a tombstone can land
anywhere in the id range.
Add an unconditional deleted-bitslice check (single bit test per
element) and rename to `filter_deferred_and_deleted` so the contract is
visible at the call site. The Either split is preserved so the
no-threshold path still avoids the cutoff comparison.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: cargo +nightly fmt
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* style: collapse error-status recovery match arm
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tests: use uniform `from .utils import *` in consensus tests
Eight consensus tests imported specific symbols from `utils` instead of
using `from .utils import *`. The most consequential side effect was
missing the `every_test` autouse fixture, which is responsible for
cleaning up leaked `processes` and resetting the port-slice allocator
between tests. Without it, leaks from one test can carry into the next
on the same xdist worker, causing `processes.pop(target_idx)` to return
the wrong peer in tests like `test_dirty_shard_crash_loop` and triggering
WAL-lock conflicts when the intended target is left running.
Make the imports uniform across the package so the autouse fixture is
always in scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* tests: keep `test_issues_api` on explicit imports
This module uses a `@pytest.fixture(scope="module")` cluster setup
shared across all tests in the file. With `from .utils import *` the
function-scoped `every_test` autouse fixture comes into scope and runs
after the module-scoped `setup`, so on the first test it sees the
already-populated `processes` and calls `kill_all_processes()` — killing
the shared cluster before the test body runs. The test then fails with
connection refused on the cluster's port.
Keep explicit imports here so `every_test` is not autoloaded; the
module teardown already cleans up via `kill_all_processes()`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduce a `pub trait Item: bytemuck::Pod + Send` marker (mirroring the
existing `UserData` pattern) and use it in place of `T: bytemuck::Pod`
on the read side of `UniversalRead`, its pipeline traits, and all
impls/wrappers. Some implementations buffer `Vec<MaybeUninit<T>>` that
may be transferred across threads in future backends; tightening the
bound makes that explicit at the trait level instead of leaving callers
to add `+ Send` ad-hoc. Write paths keep the looser `bytemuck::Pod`
bound since they only borrow `&[T]`.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* dataset: add features to fs-err
Reason: fix `cargo check --all-targets -p dataset`. Feature unification
pulls `fs-err/debug` + `fs-err/tokio` without `fs-err/debug_tokio`.
* dataset: run `cargo metadata` to get target dir
Reason: avoid re-downloading in separate worktrees. The old
`cargo locate-project` ignores custom target dirs.
* sparse/benches/search: cache vectors/indices
The segment builder evicts the page cache of vector storage, quantized
vectors, payload, payload index and the vector index after a build to
avoid cache pollution, but the id tracker was never cleared. Its on-disk
files (mappings, versions, deleted bitslice) are written during the build
and stay resident in the page cache, so after each optimization the id
tracker files linger as cache even though they are meant to be on-disk
only (expected_cache_bytes == 0).
Add `IdTracker::clear_cache` (default no-op) plus a `clear_cache_if_on_disk`
policy wrapper, and implement the eviction for the mutable and immutable
trackers:
- ImmutableIdTracker pages out its two mmap-backed storages via madvise
and drops the RAM-loaded mappings file via fadvise(DONTNEED).
- MutableIdTracker drops its append-only log files via fadvise(DONTNEED).
The builder calls `clear_cache_if_on_disk`, mirroring the payload index.
The id tracker has no on-disk mode yet, so this always clears for now;
a TODO marks where to gate it once an on-disk mode exists.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous description was outdated: it claimed that enabling this
option "blocks updates at the request level" until segments are
re-optimized. In practice the implementation uses "deferred points":
new points written to large unoptimized segments are persisted but
excluded from read/search results until the segments are optimized.
Updates are not blocked; only `wait=true` clients are made to wait for
the deferred points to become visible. Update this in the REST schema
(via `OptimizersConfig` / `OptimizersConfigDiff`), in the gRPC proto,
in the edge config docstrings, and regenerate the OpenAPI bundle via
`tools/generate_openapi_models.sh`.
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix proxy deleted_mask race dropping live points from scored search
ProxySegment::new snapshots the wrapped segment's deleted_mask while the
optimizer holds only the upgradable-read lock, before the write lock freezes
the segment. An upsert racing onto the still-appendable wrapped segment in that
window lands at an internal offset past the snapshot. The scored search path
(PlainVectorIndex::search) consults the proxy mask in place of the segment's
live deleted state, and check_deleted_condition defaults any out-of-range
offset to deleted (unwrap_or(true)) — so the live point is silently dropped
from filtered KNN while scroll/count/retrieve still return it.
Re-snapshot deleted_mask in the optimizer once the holder write lock is held
(segment frozen) and before the proxy goes live, so the mask covers the
segment's full final point range. The fresh read also captures any deletes
that raced in, closing the ghost direction too.
Adds a proxy-level regression test that reproduces the race without the
model-testing harness.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Make proxy deleted_mask sync a type-state, read once
ProxySegment::new now returns UnsyncedProxySegment instead of a
ready-to-use ProxySegment. The deleted_mask snapshot is deferred to
UnsyncedProxySegment::finalize(), which reads the wrapped segment's
deleted bitvec exactly once. The only way to obtain a ProxySegment
(and thus put it in a SegmentHolder) is via finalize(), so the sync
under the holder write lock can no longer be forgotten, and the mask
is no longer read twice (once in new, once in resync).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Allow clippy::new_ret_no_self on ProxySegment::new
new deliberately returns the unsynced UnsyncedProxySegment stage rather
than Self, since a usable ProxySegment only exists once deleted_mask is
synced via finalize().
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Name the real constructor UnsyncedProxySegment::new, keep ProxySegment::new for tests
Instead of allowing clippy::new_ret_no_self on a ProxySegment::new that
returned UnsyncedProxySegment, give the two-phase constructor its natural
home: UnsyncedProxySegment::new returns Self and is what production code
(optimize, snapshot) uses, finalizing under the holder write lock.
ProxySegment::new becomes a #[cfg(feature = "testing")] convenience that
builds and finalizes in one step (returns Self), so existing test call
sites stay terse and don't need an explicit .finalize(). The shard
testing feature is enabled for both shard's own tests and collection's
dev-dependency, and excluded from production builds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
`commit_read_hashring` returns once the leader has applied the entry,
but followers may not have applied it yet. The CommitRead handler calls
`invalidate_clean_local_shards` for `old.nodes()`, which cancels any
ongoing shard clean task. If the test's subsequent `cleanup?wait=true`
request to a follower arrives just before that follower applies
CommitRead, the cleanup task is started and then cancelled mid-flight,
and the endpoint returns HTTP 500 "Failed to clean shard points due to
cancellation, please try again".
Add `wait_for_same_commit` after `commit_read_hashring` so every peer
has applied the entry (and thus already invalidated any clean tasks
that don't exist yet) before we issue cleanup.
Observed flake:
https://github.com/qdrant/qdrant/actions/runs/26259341982/job/77289143792
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(consensus): snapshot transfer with set-payload for missing points
Add a consensus test reproducing a shard transfer abort caused by
set-payload (and other partial-update) operations targeting non-existing
points.
Such operations are written to the WAL before the point-existence check
rejects them, so the queue proxy replays them to the receiver during a
snapshot transfer. The receiver applies them with force=true, bypassing
the missing-point tolerance in handle_failed_replicas, and the operation
hard-fails with `NotFound: No point with id ... found`. Under sustained
load the bounded queue/driver retries are exhausted, the receiver replica
is marked Dead and the transfer is aborted.
The test keeps the missing-point load running while checking the result,
because consensus auto-recovers Dead replicas: stopping the load first
would let the next recovery transfer succeed and mask the bug. It must
FAIL on current code and PASS once the receiver tolerates missing-point
operations during recovery.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(transfer): skip non-transient errors during queue proxy WAL replay (#9126)
During a shard transfer the queue proxy replays operations from the
sender's WAL to the receiver. Some of these are partial updates
(set_payload, update_vectors, ...) that the receiver rejects with a
non-transient error - most commonly `NotFound: No point with id ...`
for a point that does not exist on the receiver, but also any other
client-caused bad request.
These operations were replayed from the WAL, meaning they were already
applied (and rejected the same way) on the sender, so the sender's state
reflects them as no-ops. Propagating the error aborted the whole transfer;
under sustained load the bounded queue/driver retries were exhausted and
the receiver replica was marked Dead.
Handle the error where the semantic context lives - the transmitter
(`transfer_operations_batch`): skip operations the remote rejects with a
non-transient error and keep going, while still propagating transient
errors so the caller retries delivery. Because the batch update API aborts
at the first failing operation, a non-transient batch error falls back to
one-by-one sending to isolate and skip the offending operation(s).
This complements PR #5991, which handles missing points on the live
forwarded-update path (handle_failed_replicas) but not the WAL replay path.
Fixes the abort reproduced by
test_shard_snapshot_transfer_with_missing_point_updates.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Under parallel pytest-xdist load the rejoining peer needs more than the
default 30s budget to replicate the UpdateCollection entry and report
ready, causing intermittent CI failures:
Exception: Timeout waiting for condition peer_is_online to be satisfied
in 30 seconds
Bump both peer-online waits in the dirty shard crash-loop test to 60s,
matching the precedent set by #8963 for similar CPU-contention flakes.
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: serialize optimization proxy install against shard updates
execute_optimization captures `target_config` from the optimizer's frozen
config and then wraps source segments in proxies. Between those two points,
`CollectionUpdater::update` can apply a `CreateVectorName(V)` to the source
segments via `apply_segments`, leaving the optimizer with sources that have
V but a target_config that does not. The optimization then produces a
merged segment without V, and a follow-up optimization (running with the
refreshed config that includes V) fails to use that segment as a source:
"Cannot update from other segment because it is missing vector name X".
Close the race by extending the scope of the existing
`LockedSegmentHolder::acquire_updates_lock` to cover the proxy install
window. `CollectionUpdater::update` already takes this lock before
processing any shard update, so concurrent writers wait until proxies are
in place — at which point further mutations hit the proxies (recorded as
intent and propagated to the merged segment in `finish_optimization`)
instead of the originals. The guard is dropped right after proxy install so
the slow build phase does not extend it.
Tests:
- Three `SegmentBuilder::update` tests document the precondition the lock
now guarantees: with a target schema that adds a named vector the source
lacks, update errors with "missing vector name X". Quantized and
mixed-source variants exercise the same error path.
- `test_optimize_blocks_proxy_install_on_updates_lock` asserts the
invariant directly: while the updates lock is held, proxies are not yet
installed. Verified to fail when the new guard is removed (otherwise it
passes because `finish_optimization` also takes the same lock, so a
naive "did optimize finish?" check would not catch a missing proxy-install
guard).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(optimize): finish_optimization lock order; drop redundant tests
Address review of #9110:
1. `finish_optimization` was acquiring `upgradable_read` before
`acquire_updates_lock`, while the new guard at the start of
`execute_optimization` acquires them in the reverse order. With two
optimizer threads in flight, thread A in `finish_optimization` could
hold `upgradable_read` and wait on `updates_lock` while thread B at the
top of `execute_optimization` held `updates_lock` and waited on
`upgradable_read` (parking_lot allows only one upgradable reader),
deadlocking. Swap `finish_optimization` to take `updates_lock` first so
both halves agree.
2. Drop the quantized and mixed-source variants of the inverted
`SegmentBuilder` unit test — all three asserted the same error path
(the mismatch check fires before quantization training or per-source
branching), so only one is useful as documentation of the precondition.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: reject source-superset schema mismatch at SegmentBuilder
Drop the lock approach (deadlocked test_continuous_snapshot) and fix the bug
at the merge layer instead.
Snapshot's proxy_all_segments_and_apply acquires the segment_holder
upgradable_read first and then takes acquire_updates_lock tactically inside
the snapshot operation. The previous commits' lock-extension acquired
updates_lock before upgradable_read, so a snapshot in flight and an
optimization just entering execute_optimization could deadlock holding each
other's required next lock. Snapshot cannot easily reverse its order — that
would hold updates_lock for the entire snapshot duration, blocking all
writes.
Move the fix to where the actual harm happens: SegmentBuilder::update
iterates the target's vector_data and silently drops source vectors that
aren't in target. That silent drop is what produces the broken merged
segment in the CreateVectorName-vs-optimizer race. Add a check that every
source vector name is in the target schema; the optimization aborts cleanly
on mismatch and the next round (with refreshed config) merges correctly.
This is strictly stronger than the lock: the lock only closed the window
where V arrived *during* the proxy-install region. The schema check catches
both that window and the window where V's apply_segments completed before
the optimizer's lock acquisition.
Diff is contained to lib/segment; no locking changes, no cross-crate
plumbing. test_continuous_snapshot passes again.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(segment_builder): use Cancelled instead of ServiceError for schema mismatch
ServiceError flips the shard to RED status (via `report_optimizer_error` →
`segments.optimizer_errors`) and stays sticky until the next
`recreate_optimizers_blocking` clears it. That's the right shape for
hardware/IO failures but wrong for the schema-mismatch case here, which is
an expected, recoverable race outcome — the next optimizer round with a
refreshed target_config merges the same originals cleanly.
`Cancelled` is the variant the optimization worker treats as a recoverable
cancellation: logged at debug, tracker marked Cancelled, no
`report_optimizer_error` call, no RED status.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(segment_builder): also use Cancelled for the existing target-superset error
The existing "missing vector name" check at the start of the merge loop
also fires during a race — specifically the optimizer-vs-DeleteVectorName
shape, where V is removed from originals before J wraps proxies but J's
frozen target_config still has V. Like the new source-superset check, this
is an expected, recoverable race outcome, so use Cancelled instead of
ServiceError to avoid flipping the shard to RED.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Recreate optimizers in non-blocking fashion from consensus calls
* Update comments
* On optimizer config update failure, report error status to local shard
* Add a test to confirm we don't block consensus
* Rerun recreation if called multiple times
* Use atomics instead
* Move to the bottom
* Reformat
* Use cluster default shard transfer method for fallback
When a WAL delta automatic transfer fails, the driver falls back to the
method passed via `fallback_method`. This was hard-coded to
`StreamRecords` (unless `prevent_unoptimized` was enabled), which is
inconsistent with the 1.18.0+ default of `Snapshot` and ignores any
configured `default_shard_transfer_method`.
Use `Collection::default_shard_transfer_method()` instead, so the
fallback matches the cluster default. With `prevent_unoptimized` we
still pin to `Snapshot` to preserve deferred point state exactly (raw
segment copy); stream_records would send deferred points but they
would not be deferred on the target.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Avoid wal_delta fallback, update fallback test
If the cluster default transfer method is wal_delta, the same-method
fallback would be refused by the driver. Use snapshot as a safe fallback
in that case; snapshot is also the 1.18.0+ default.
Update test_shard_wal_delta_transfer_fallback to assert the new
snapshot fallback (was stream_records).
Co-authored-by: Cursor <cursoragent@cursor.com>
* Address clippy wildcard_enum_match_arm
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add test to show cleanup may conflict with update queue
* When invoking clean task, first wait for current update queue
* Don't hold shard holder lock for a long time
* Also assert the clean task finished completely
