* Remove from_iter_instead_of_collect from workspace lints
The lint was removed from clippy (beta) and now triggers
renamed_and_removed_lints warnings in every crate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::chunks_exact_to_as_chunks
Replace chunks_exact with a constant chunk size by as_chunks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::needless_late_init
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::useless_borrows_in_formatting
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::uninlined_format_args
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix clippy::for_kv_map
Iterate map values directly instead of discarding keys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_large_err on QueueProxyShard::new_from_version
The Err variant intentionally hands the LocalShard back to the caller.
Same pattern as the existing allow on ForwardProxyShard::new.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow clippy::result_unit_err on wait_for_consensus_commit
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add unified `memory: cold|cached|pinned` placement parameter for collection components
Introduce a single `memory` parameter that controls how each collection
component's data is held in RAM, replacing the inconsistent zoo of
`on_disk` / `always_ram` / `on_disk_payload` flags:
- `cold`: not pre-loaded from disk, cached with usage
- `cached`: pre-populated into page cache on load, evictable under pressure
- `pinned`: materialized on heap, never evicted by cache pressure
The parameter is available on dense vectors, HNSW config, all quantization
configs, the sparse index, all payload field index types, and payload
storage (as a new `payload: { memory }` sub-object on collection params).
When set, it overrides the deprecated legacy flag; when unset, behavior is
unchanged. Legacy flags are marked deprecated (Rust + proto) but keep
working; conflicts are resolved in favor of `memory` with a warning.
New capabilities enabled by the tri-state model:
- HNSW graph links can be pinned (first production caller of the existing
`GraphLinksResidency::Pinned`)
- sparse mmap index, quantized vectors and on-disk payload field indexes
gain a `cached` tier (mmap + populate on open)
`pinned` is rejected by API validation for components without a heap
variant (dense vector storage, payload storage). Low-memory mode degrades
placements at load time via `Memory::clamp_to_low_memory`, matching the
existing `prefer_disk`/`skip_populate` behavior. Effective-placement
comparison in the config-mismatch optimizer avoids spurious rebuilds when
the same placement is expressed through the new parameter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix gpu-gated tests for the new `memory` field
CI clippy runs with --all-features, which compiles the gpu-gated tests
that were missed locally: add the `memory` field to config literals and
allow deprecated placement params, same as in the rest of the tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add OpenAPI tests for memory placement, keep sparse config downgrade-clean
- OpenAPI tests: create/update collections with `memory` on every component,
assert the parameters are echoed in collection info, assert legacy-only
collections expose no new fields, and assert `pinned` is rejected (422)
for dense vector storage and payload storage on both create and update.
- Persist only the explicitly requested `memory` parameter in
`sparse_index_config.json` instead of the legacy-resolved placement, so
configurations using only the deprecated `on_disk` flag keep byte-identical
files that older Qdrant versions load without unknown fields.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Validate collection meta ops at construction, not only in the API layer
The `memory: pinned` rejection for dense vectors and payload storage
lived in `Validate` impls on the internal request types, which only ran
through the REST actix extractor. gRPC validates just the proto message,
so a gRPC client could persist `pinned` where it is not supported and
have it silently treated as `cached`.
Run the derived validation in `CreateCollectionOperation::new` and
`UpdateCollectionOperation::new` instead: the constructors are the
common chokepoint for all API paths, before the operation is proposed
to consensus. This covers every validator on these types, not just the
`memory` checks, and keeps consensus-apply unaffected so mixed-version
clusters never reject already-committed operations.
`UpdateCollectionOperation::new` becomes fallible; `remove_replica` now
uses `new_empty` since it carries no user config. Regression tests drive
the gRPC conversion path and assert `InvalidArgument` for `pinned` on
create and update, with `cold`/`cached` accepted as a control.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Applying `RemoveNode(self)` prunes all other peers from the removed
peer's persisted address book, but the process may be stopped after the
removal is committed and before the entry is applied. The old first
peer's address then survives `--reinit`, and the readiness checker -
which treated every `peer_address_by_id` entry as a cluster member -
would wait for the reinitialized peer to reach the *old* cluster's
commit index: a foreign consensus it can never catch up with, so
`/readyz` never passed.
Filter the address book by current `conf_state` membership instead,
falling back to all known addresses while `conf_state` is still empty
(a bootstrapping node that has not applied any configuration change
yet). After `--reinit` the `conf_state` is reset to a single voter, so
the readiness check correctly ignores peers of the old cluster.
Fixes flaky `test_reinit_removed_peer`, which hit this race when the
removed peer was killed before applying its own `RemoveNode`. The new
regression test simulates that state and fails with the exact CI error
without the fix.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [AI + manual] Include file size in `UniversalReadFileOps::list_files`
* Use dedicated `ListedFile` struct instead of `(PathBuf, u64)` pair
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Since the tonic 0.14 upgrade (hyper 1.x), a client cancelling a unary
call between HEADERS and DATA surfaces as Internal "Missing request
message." instead of Cancelled, because hyper hides stream resets
during request body read (hyperium/hyper#3681). Cluster read fan-out
generates these routinely, flooding logs with spurious ERROR lines.
Log them at trace level, matching the existing Cancelled handling.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Fix reinit failing with "removed all voters" on a removed peer
When `--reinit` is run on the first peer, `conf_state` is reset to a
single voter (this peer), but the Raft log is left untouched. If this
peer had been removed from consensus before reinit, its log still holds
a committed-but-unapplied `RemoveNode(self)` conf-change. On startup that
entry is replayed on top of the freshly reset single-voter config, and
Raft aborts with "removed all voters", so the node can never start.
Resetting only the apply-progress queue is not enough: a fresh
single-node leader re-commits and re-applies any log entries still
physically present beyond `commit`, re-triggering the failure. The stale
tail has to be physically dropped from the WAL.
On the first-peer reinit path, discard committed-but-unapplied entries
inherited from the previous cluster: truncate the WAL to the last applied
index, pin `commit` to it and clear the apply-progress queue. The entry
at `commit` is retained as the snapshot anchor, so the first peer can
still serve snapshots to bootstrapping peers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add regression test for reinit of a removed peer
Starts a 2-node cluster, gracefully removes the second peer from consensus
(so it commits `RemoveNode(self)` into its own WAL), then reinitializes it
as a fresh first peer. Before the fix this panicked on startup with
"removed all voters"; the test asserts the peer comes back online, elects
itself leader, and can still seed a fresh bootstrapping peer.
Verified the test fails against the pre-fix binary with exactly:
Failed to apply configuration change entry
Caused by: Error in Raft consensus: removed all voters
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Restart consensus thread on failure instead of stopping permanently
If the consensus loop failed (e.g. due to a transient I/O error such as
running out of disk space), the consensus thread stopped permanently and
the only way to recover was a full service restart.
Now the consensus thread rebuilds the Raft node from persisted state and
retries with exponential backoff (1s initial, doubled up to 5 min cap,
retrying indefinitely). Rebuilding from persisted state is equivalent to
a process restart, so this introduces no new recovery semantics. The
`reinit` logic is never repeated on restart, and initial startup remains
fail-fast.
The consensus message channel is created outside of `Consensus` so that
internal gRPC handlers and the forward-proposals thread keep their
senders across restarts, and messages buffered during the outage are
drained after recovery.
While in restart backoff, the node reports the existing
`StoppedWithErr` cluster status with restart attempt info appended, and
flips back to `Working` after a successful restart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add staging TestTransientError op and consensus restart integration test
New staging-only cluster operation `test_transient_error` (mirroring
`test_slow_down`): applying it fails with the given probability on the
targeted peer, stopping its consensus thread with a service error. The
peer re-applies the entry on every consensus thread restart, rolling the
probability again, so it exercises the consensus restart loop end to
end. Probability 1.0 simulates a permanently failing operation.
The integration test runs a 3-peer cluster, poisons a follower through
its own API (so the simulated failure is reported deterministically in
the response), and asserts that the restart loop engages, the remaining
peers keep serving consensus operations with a quorum, and the failed
peer eventually recovers and catches up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix import order to satisfy rustfmt
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* remove unused iter_offsets
* Replace MultivectorOffsetsStorage::iter_offsets with callback-based for_each_offset
First step of removing the iterator-returning read API (whose deep,
composable generic types blow up mangled symbol size). Convert the
offsets read from an iterator to a callback the caller pushes into:
- trait method iter_offsets -> for_each_offset(ids, FnMut(usize, MultivectorOffset))
returning common::universal_io::Result<()>
- Mmap impl now uses the callback read_batch (drops one read_iter use)
- Ram / Chunked impls push into the callback; Chunked still goes through
iter_vectors for now (converted in a later step)
- the single caller (for_each_in_multi_batch) passes a closure
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Implement read_batch directly on ReadPipeline, not via read_iter
read_batch now drives the pipeline itself (refill-then-wait loop, like
read_multi_iter) and invokes the callback per result, instead of
consuming the iterator returned by read_iter. A step toward removing the
iterator-returning read API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove the ReadMulti RPC from the StorageRead gRPC service
ReadMulti was the only real consumer of UniversalRead::read_multi (which
itself relies on read_multi_iter). Removing the RPC end-to-end clears the
path to dropping that read API. StorageReadService keeps all its other
RPCs (ListFiles, FileExists, FileLength, ReadBytes, ReadBytesStream,
ReadWhole, ReadBatch).
- proto: drop `rpc ReadMulti` + ReadMulti{Entry,Request,Response}
- regenerated lib/api + uio-client generated code; drop ReadMulti
validation rules in lib/api/build.rs
- tonic: delete the read_multi handler + its 2 tests
- uio-client: delete Client::read_multi, the mock-server impl, and 2 tests
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove UniversalRead::read_multi
Its only real consumer was the StorageRead ReadMulti gRPC handler (removed
in the previous commit); the two wrapper forwarders had no callers. Drop
the trait method and both forwarders (typed/read_only), and remove the
io_uring test that only existed to compare read_multi vs read_multi_iter
(read_multi_iter stays covered by the other tests). Another step toward
removing the iterator-returning read API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drive ReadPipeline directly in gridstore read_from_pages
Replace the read_multi_iter call in Pages::read_from_pages with a direct
pipeline loop (refill-then-drain), scheduling each multi-page read on its
own page file. Behavior unchanged; another step toward removing the
iterator-returning read_multi_iter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drive ReadPipeline directly in gridstore read_batch_from_pages
Replace the second read_multi_iter call (in Pages::read_batch_from_pages)
with a direct pipeline loop, scheduling each (ReadMeta, page, range) on its
own page file and propagating errors via GridstoreError. Single/multi-page
buffering and out-of-order reassembly are unchanged. No more read_multi_iter
in gridstore.
Measured overhead on warm mmap (both paths are zero-copy borrows): ~0.3 ns
per read of fixed control cost, flat across read sizes — well under 0.1% of
a real payload read.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drive ReadPipeline directly in on-disk postings with_posting_views
Replace read_iter in OnDiskPostings::with_posting_views with a direct
pipeline loop. wait_bytemuck yields a file-borrowed Cow, so postings are
still stored zero-copy in raw_postings (a read_batch swap would have forced
an owned copy of every posting list per query on the mmap backend).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Read on-disk posting headers via read_batch, drop the HeadersBatch iterator
headers_iter now reads headers with the callback read_batch API: each header
is parsed (copied) out of the read bytes, so nothing borrows the file past the
read — no pipeline needed. Since the read is now eager, HeadersBatch holds the
collected Vec<HeaderResult> directly instead of a Box<dyn Iterator>, dropping
the boxing, the dynamic dispatch, and the struct's lifetime parameter.
with_posting_views takes the Vec and still pipelines the posting reads.
Removes the last read_iter use in this file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Use read_batch in simple_disk_cache populate_from
populate_from reads one byte per block only to fault blocks into the local
cache, discarding the bytes — a no-op-callback read_batch fits exactly. Drives
the same DiskCachePipeline as before; one less read_iter caller.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Implement read_iter directly on ReadPipeline, not via read_multi_iter
read_iter now drives the pipeline itself (refill-then-wait loop, mirroring
read_bytes_iter) instead of mapping its ranges onto self and calling
read_multi_iter. Same signature and iterator contract, so all callers are
unchanged. Leaves iter_vectors as read_multi_iter's only remaining caller.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "Drive ReadPipeline directly in on-disk postings with_posting_views"
This reverts commit c02c41f17b.
* Add callback-based for_each_vector next to iter_vectors
for_each_vector drives the ReadPipeline directly across chunk files and
invokes a fallible callback per flattened multi-vector, returning
OperationResult, instead of returning an iterator built on read_multi_iter.
Callers will migrate onto it so iter_vectors (read_multi_iter's last caller)
can be removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Make dense for_each_in_batch / for_each_in_dense_batch fallible
Thread OperationResult up the dense batch-read path so io_uring read
errors propagate instead of being .expect()ed deep inside the storage.
The for_each_in_dense_batch scorer path (custom/metric query scorers)
now carries the Result to the infallible score() boundary where it is
.expect()ed; read_vectors keeps its () signature and .expect()s locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Convert dense read_vectors to for_each_vector
The read-only and appendable dense storage read_vectors impls drove
ChunkedVectors::iter_vectors directly; switch them to the callback-based
for_each_vector and .expect() the result at the (infallible) read_vectors
boundary. Removes the last dense-path iter_vectors callers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Convert quantized for_each_offset to for_each_vector + OperationResult
The two chunked MultivectorOffsetsStorage impls drove iter_vectors to read
the offset table; switch them to the callback-based for_each_vector.
for_each_vector returns OperationResult, so upgrade the for_each_offset
trait (and all four impls) from universal_io::Result to OperationResult
(the universal_io -> Operation direction, via ?). No error downgrade.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Convert EncodedStorage/EncodedVectors iter_batch to callback for_each_batch
The two chunked-mmap EncodedStorage impls drove ChunkedVectors::iter_vectors
to back iter_batch. Replace the iterator-returning iter_batch on both the
EncodedStorage and EncodedVectors traits (quantization crate) with a callback
for_each_batch(FnMut(usize, &[u8])), and switch the chunked impls to
for_each_vector. The callback is infallible: the chunked impls .expect() the
read internally, matching iter_vectors' prior panic-on-read-error behavior,
so no OperationError is downgraded. Scorers and the multivector readers adopt
the callback; the accumulating multivector path owns (to_vec) only when it
must buffer across components.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Convert multivector read paths to for_each_vector
The read_only multivector free fn chained two iter_vectors (offsets feeding
vectors) - the recursive iterator nesting behind the worst symbol bloat.
Replace it with a callback for_each_vector that resolves the per-point
offsets into a Vec first, then drives ChunkedVectors::for_each_vector over
the flattened vectors. Migrate both multivector storages' read_vectors and
for_each_in_batch_multi accordingly, .expect()ing at their infallible
boundaries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove read_multi_iter and ChunkedVectors::iter_vectors
With every caller migrated to callback-based for_each_vector/for_each_batch,
delete the last iterator-returning multi-read APIs: ChunkedVectors::iter_vectors
(segment) and the read_multi_iter trait method plus its mmap override, the
TypedStorage/ReadOnly wrapper forwarders, and the two io_uring unit tests.
These deeply-nested monomorphized iterator types (read_multi_iter feeding
read_multi_iter) produced >1 MiB mangled drop_in_place symbols that overflowed
the macOS ld symbol-name limit; the callback rewrite eliminates them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Pass owned Cow through for_each_vector/for_each_batch to avoid a copy
The callback-based readers handed the callback a borrowed `&[u8]`/`&[T]`,
forcing the quantized multivector batch read to `to_vec()` each sub-vector
into its per-point buffer. But io_uring-like backends already return a freshly
owned buffer per read (`ACow::Owned`), so that was a redundant second copy.
Change `ChunkedVectorsRead::for_each_vector` and the `EncodedStorage` /
`EncodedVectors` `for_each_batch` callbacks to receive `Cow<[..]>` by value.
The buffering path now `into_owned()`s it — a move when the backend returned
owned (the case this path targets), a copy only for a borrowed Cow (mmap),
which never reaches this path. Immediate-use callers (scorers,
score_point_max_similarity, the dense/multivector readers) just deref the Cow;
the dense readers drop their now-redundant `Cow::Borrowed` wraps.
Also clarifies the multivector reader: `SubVectorOwner`/`owners`/
`sub_vector_offsets` naming, docs, and a corrected comment noting the per-point
buffer is what makes regrouping order-independent under out-of-order completion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drop the removed ReadMulti JWT access test; fix clippy unwrap_or_default
The ReadMulti StorageRead RPC was removed earlier in this branch, so the
consensus JWT-access test (and its registry entry) for it must go too. Also
switch the multivector buffer's `or_insert_with(SmallVec::new)` to
`or_default()` per clippy::unwrap_or_default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add MmapFile::read_batch
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
DeleteByFilter remains broken: WAL replay can resurrect filter-deleted
points (#9575). Re-mask the op in FORCE_OFF and document the issue.
Co-authored-by: Cursor <cursoragent@cursor.com>
The flushing/reload-durability bug that caused post-reload count
mismatches for live filtered deletes has been fixed, so DeleteByFilter
no longer needs to be masked off in the model tester swarm config.
Remove it from FORCE_OFF and update the related doc comments.
Co-authored-by: Cursor <cursoragent@cursor.com>
Build the multi_thread runtime by hand instead of via the `#[tokio::main]`
macro so the soak test can seed Tokio's internal RNG from `--seed`. Under
`--cfg tokio_unstable` this pins the in-poll random draws (notably `select!`
branch selection), shrinking the space of possible executions for a given
seed. It does not make the scheduler fully deterministic (thread interleaving,
work-stealing timing and I/O readiness still vary), but it removes one source
of variance at near-zero cost; without the flag the binary still builds and
runs unseeded.
Also add `--worker-threads` to optionally pin the worker count (defaults to
the CPU core count), so a `--seed` repro can share the same runtime shape
across machines.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(bm25): add explicit Disabled stemmer; deprecate language hack
Adds a `Disabled` variant to `StemmingAlgorithm` (`stemmer: {"type": "none"}`)
so stemming can be turned off explicitly in both the main engine and Edge,
instead of relying on the undocumented `language: "none"` footgun that
silently disabled both stemming and stopwords.
For language-neutral text processing the supported setup is now:
1. set the stemmer to disabled, and
2. configure an empty stopword set.
The main engine still tolerates unsupported languages (so existing
`language: "none"` configs keep working on upgrade) but now logs a
deprecation warning pointing users to the explicit setup. Edge continues
to reject unsupported languages, and now has a real way to disable stemming.
Refs: https://github.com/qdrant/qdrant/issues/9289
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(edge-py): handle Disabled stemmer in python bindings; fix openapi schema
- Handle the new StemmingAlgorithm::Disabled variant in the qdrant-edge-py
bindings (FromPyObject/IntoPyObject/Repr) and add a DisabledStemmer pyclass
plus its .pyi stub entry.
- Match generator output for the StemmingAlgorithm OpenAPI schema (plain $ref
in anyOf) so docs/redoc/master/openapi.json stays consistent.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openapi): regenerate StemmingAlgorithm schema with generator output
Ran tools/generate_openapi_models.sh so docs/redoc/master/openapi.json
exactly matches generator output: DisabledStemmerParams/NoStemmer are placed
after SnowballLanguage, and the StemmingAlgorithm anyOf entry is a plain $ref
(the schema2openapi step flattens the allOf+description wrapper).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): avoid wildcard enum match arm in bm25 sparse_len helper
clippy --all-targets flags `other => panic!()` as wildcard_enum_match_arm;
match the Dense/MultiDense variants explicitly instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: issues
* fix: log::warn as call once
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* feat(model_testing): add --duration-sec to run for a fixed wall-clock time
Bound the soak by wall-clock time instead of op count: when --duration-sec is set, the loop runs until the deadline (or Ctrl-C) and --op-num is ignored.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix clippy
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add routing token structure
* Implement routing token in read operation executor as per design doc
* Add TODO to glue routing token to user requests
* Implement routing header for REST API
* Source routing token from request, not from JWT token
* Implement routing token in gRPC API
* Add test
* Review remarks
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Use lower case header name to prevent panic
* Rename header to X-Qdrant-Route-Affinity
* Assert routing consistency in test on all peers
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* fix: treat empty API key strings as unset
Closes#8856
* fix: also filter empty secrets in JWT parser creation
* Add tests
* Apply same treatment to internal gRPC authentication
* Add startup warning when empty API key is set
---------
Co-authored-by: timvisee <tim@visee.me>
* Use assert_matches!
* Add trailing commas
* Use more assert_matches!
Also, drop now redundant `expected blah but got blah` messages because
`assert_matches!` will print these.
* Use debug_assert_matches!
---------
Co-authored-by: xzfc <xzfcpw@gmail.com>
* Fix stale snapshot transfer recovery comment on failure/cancellation
The transfer status "comment" prefers the destination-side recovery
progress over the sender task status. Recovery progress was tracked in
`active_recoveries` and only removed via `finish_shard_recovery`, which
was called on the success path only. If `recover_shard_snapshot` returned
early - clearing the local shard, downloading the snapshot, checksum
mismatch, or cancellation (the future is spawned with
`spawn_cancel_on_drop`) - the entry leaked.
A leaked entry keeps reporting its last stage with an ever-growing
elapsed time (computed live from a fixed `Instant`), so subsequent
transfers for the same shard show stale timings until the next recovery
overwrites the entry or the node restarts.
Replace the manual start/finish pair with an RAII `ShardRecoveryGuard`
that removes the progress entry on drop, covering every exit path
including early returns and cancellation. The guard only removes its own
entry (checked via `Arc::ptr_eq`) so it never clobbers a newer recovery.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Extract recovery tracking into dedicated recovery_guard module
Move ShardRecoveryGuard out of the large shard_holder/mod.rs into a
dedicated recovery_guard.rs, and replace the verbose
`Arc<Mutex<HashMap<ShardId, Arc<Mutex<RecoveryProgress>>>>>` type with a
dedicated `ActiveRecoveries` struct that encapsulates start/comment/
set_stage/remove operations. No behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Bind recovery stage updates to the guard instance, not shard id
`ActiveRecoveries::remove` is pointer-guarded so an unwinding recovery
cannot delete a newer recovery's entry, but stage updates still resolved
the progress entry by shard id. If recovery A is still unwinding after
recovery B started for the same shard, A's late `set_stage` would mutate
B's progress and resurrect incorrect transfer comments.
Thread the guard's own progress handle (`RecoveryProgressHandle`) through
`recover_shard_snapshot_impl` -> `Collection::restore_shard_snapshot` ->
`ShardHolder::restore_shard_snapshot`, so the Unpacking/Restoring stages
(and Downloading, via `ShardRecoveryGuard::set_stage`) update that exact
recovery's progress. Direct API recoveries, which are not tracked as
transfer-side recoveries, pass `None`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Allow too_many_arguments on recover_shard_snapshot_impl
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* load with any universal io
* Don't populate with sequential advice
* use appropriate fs for `exists`
* use `read_whole_via` more
* TODO
* clear ram cache after read_whole
* feat: add io_bridge
* fix: cached dispatcher
* feat: add open_with_handle
* fix: naming
* fix: wording
* fix: rebase io_bridge onto split BorrowedReadPipeline/OwnedReadPipeline
* fix: linter
* feat: add S3 backend
* chore: remove io_bridge
* fix: linter
* fix: linter
* fix: read handle
* chore: simplify S3Source
* fix: s3 test
* feat: support multi runtime
* fix: clippy errors
* fix: review comments
* feat: add io design
* feat: add S3 backend
* chore: fix docs
* fix: dev changes
* chore: add some docs
* chore: remove explicit type
* feat: add new methods
* [WIP] review refactor
* fmt
* fix: bytes alignment
* fix: linter
* feat: remove Bytes
* fix: ci/cd
* fix: tests
* fix: tests
* refactor: simplify io_bridge pipeline to Handle-based dispatch
Replace the BridgeRuntime worker thread + request channel + boxed
BridgeRequest with direct tokio Handle usage:
- BridgeRuntime is now just an Arc<Runtime>; schedule() spawns the read
future via Handle::spawn instead of routing it through a dispatcher
thread. Removes BridgeRequest and the now-unreachable S3RuntimeShutDown
error variant.
- Guard against a panicking read task hanging wait() forever: the spawned
task catches unwinds and converts them into a TaskPanicked error reply,
so every scheduled slot is always answered.
- Encapsulate slot bookkeeping in PendingSlots, exposing only the needed
operations instead of a public map + counter.
- Split the grown pipeline.rs into a pipeline/ module (slots / inner /
borrowed / owned), de-duplicating the shared read-future construction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: move pipeline buffer ownership into the read future
Instead of the pipeline owning the destination Vec<T> in a slot map and
the future writing through a SendBytePtr raw pointer, let the future
allocate the buffer itself and return it through BridgeResponse. The
buffer crosses the worker-thread boundary as a normal move via the reply
channel, wrapped in a SendableVec<T> newtype that asserts Send for
T: bytemuck::Pod only.
This removes the entire unsafe SendBytePtr apparatus from the pipeline:
no raw pointer, no unsafe fn, no per-call-site unsafe blocks, no
heap-stability invariants. The only remaining unsafe in the crate is one
bounded `unsafe impl<T: Pod> Send for SendableVec<T>` with a trivially
true invariant (Pod types are plain bytes).
PendingSlots collapses back to PendingSlots<U>: slots no longer carry
buffers. AlignedBufWriter::from_raw_bytes (used only by the SendBytePtr
path) and its test are removed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: drop SendableVec wrapper now that Item: Send
UniversalRead's element type is now bound to `Item` (`Pod + Send`),
so the io_bridge pipeline no longer needs a hand-rolled `Send` wrapper
around `Vec<T>` to ship buffers through the reply channel. Replace
`SendableVec<T>` with `Vec<T>` end-to-end and tighten the local impls
to `T: Item`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: implement UniversalRead::reopen for BlobFile
BlobFile has no cached file metadata or mapping — `len()` queries the
object store fresh on each call — so reopen is a no-op, matching the
io_uring impl.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: add new fs impl for Blob
* fix: is_in_ram_or_mmap for S3
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [UIO] Split UniversalReadFileOps into filesystem + file traits
`UniversalReadFileOps` is now an instance-based trait describing a
filesystem handle (list/exists with `&self`, plus `from_context`). A new
`UniversalReadFs: UniversalReadFileOps` subtrait adds the
`open(&self, path, options) -> Self::File` capability with `type File:
UniversalRead`. `UniversalRead` no longer extends `UniversalReadFileOps`
and is purely a file-handle trait.
This separates "filesystem instance" from "file handle". Backends that
need per-instance configuration (S3 bucket name + credentials, mmap
default advice, io_uring runtime, block-cache controller `Arc`) gain a
typed home in `Self::ContextConfig`, and `list_files`/`exists`/`open`
become `&self` methods on the filesystem handle.
Concrete filesystem handles introduced for the three existing backends:
- `MmapFs` — unit struct; `ContextConfig = ()`; produces `MmapFile`.
- `IoUringFs` — carries `prevent_caching`; `ContextConfig =
IoUringConfigContext`; produces `IoUringFile`. `IoUringConfigContext`
becomes the construction input rather than a per-open argument.
- `BlockCacheFs` — carries `Arc<CacheController>`; `ContextConfig =
BlockCacheConfigContext`; produces `CachedSlice`.
`TConfigContext` (universal builder methods) is kept so generic-over-`Fs`
code can still set cross-backend knobs via
`Fs::ContextConfig::default().with_prevent_caching(true)`.
Wrappers (`ReadOnly<S>`, `TypedStorage<S, T>`, `StoredStruct<S, T>`),
higher-level storages (`StoredBitSlice<S>`, `UniversalHashMap<K, V, S>`)
keep `<S: UniversalRead>` parameterization but their `open(...)`
constructors now grow a `fs: &Fs` argument bound by
`Fs: UniversalReadFs<File = S>`. `read_json_via` becomes
`read_json_via(fs: &Fs, path)`.
All test code and benches in `common` updated to construct
`MmapFs`/`IoUringFs` inline as needed. `common` compiles cleanly with
tests and benches. `gridstore`, `segment`, and `tonic` caller updates
are in flight in subsequent commits.
* WIP: gridstore + segment caller sweep (partial)
Threads `fs: &Fs` through gridstore's `BitmaskGaps`, `Bitmask`, `Pages`,
and `Gridstore::new`/`open`/`create_new_page`. Most segment callers
have `OpenOptions { extra: ... }` removed and `S::open(path, opts, ctx)`
sites updated mechanically but the trait change is not yet propagated.
Does NOT compile yet. Tracker still has static `S::open` calls, segment
generic constructors (`MmapInvertedIndex<S>`, `UniversalMapIndex`,
`StoredGeoMapIndex`, etc.) still call `S::open`/`S::list_files`/`S::exists`
statically — they need an `fs: &Fs` parameter added. Tonic API
`StorageReadService<S>` also unconverted.
Committed as branch checkpoint; cascade continues in subsequent work.
* gridstore: thread `fs: &Fs` through Bitmask, BitmaskGaps, Pages, Tracker
Per the new `UniversalReadFs` shape, every constructor/method that opens
files takes an `fs: &Fs` parameter. Gridstore is currently mmap-only,
so the top-level `Gridstore` / `GridstoreReader::open` callers in the
crate pass `&MmapFs` inline. Tests do the same.
gridstore lib + tests now compile cleanly. Segment + tonic cascade
still pending.
* WIP: segment caller sweep — dynamic_stored_flags first
* WIP: segment cascade - id_tracker partial
* common benches: update to new UniversalReadFs::open shape (clippy clean)
* segment flags: thread Fs through BufferedDynamicFlags / Bitvec / Roaring
DynamicStoredFlags::set_len now takes `fs: &Fs`. BufferedDynamicFlags
stores an `Arc<Fs>` so the flusher closure can call `set_len` on resize.
BitvecFlags and RoaringFlags expose a new `Fs` type parameter and the
flag tests now pass `Fs::default()` (MmapFs/IoUringFs via duplicate_item).
Concrete consumers (bool/null index, mmap dense/multi/sparse storages)
pin `Fs = MmapFs` and pass `&MmapFs` to inner opens.
* segment: thread Fs through field-index lifecycle methods
Apply the new UniversalReadFs::open shape across:
- full_text_index (MmapInvertedIndex, MmapFullTextIndex, UniversalPostings)
- geo_index (StoredGeoMapIndex build/open + tests + builders)
- numeric_index lifecycle (UniversalNumericIndex build/open)
- map_index lifecycle (UniversalMapIndex build/open)
- stored_point_to_values (open / from_iter)
Concrete consumers pin Fs = MmapFs and pass &MmapFs inline; generic
open paths thread `fs: &Fs` where Fs: UniversalReadFs<File = S>.
* segment: thread Fs through chunked vectors and id-tracker callers
- ChunkedVectors gains `Fs` generic so add_chunk can call create_chunk
after open. ChunkedVectorsRead/load_config and chunks::{read_chunks,
create_chunk} take `fs: &Fs`. Concrete callers (dense / multi-dense /
sparse / quantized) pass MmapFs inline.
- DenseVectorStorageImpl stores `fs: Fs` so `update_from` can reopen
ImmutableDenseVectors. ImmutableDenseVectors::open takes `fs: &Fs`.
- VectorStorageEnum DenseUring* variants thread IoUringFs alongside
IoUringFile.
- segment_builder + segment_constructor_base pass MmapFs to
ImmutableIdTracker::{new, open}.
- QuantizedStorage::from_file takes `fs: &Fs`; quantized_vectors callers
pass &MmapFs.
* segment: apply nightly rustfmt after Fs refactor
cargo +nightly fmt --all over the segment crate after the
UniversalReadFs cascade. No semantic changes.
* segment: thread Fs through benches and id-tracker tests
Update the dynamic-mmap-flags and buffered-update-bitslice benches to
the new UniversalReadFs::open shape (pass `&MmapFs`). Update the
immutable-id-tracker test suite to forward `&MmapFs` to
`from_in_memory_tracker` / `open`.
* uio: pin Fs via UniversalRead::Fs assoc type; per-call OpenExtra
Two design changes that fall out of the per-instance Fs refactor:
1. Bidirectional Fs ↔ File pinning. `UniversalRead::Fs:
UniversalReadFs<File = Self>` lets generic-over-`S` code refer to
`S::Fs` directly instead of carrying an extra `<Fs: UniversalReadFs<File = S>>`
generic param. `ReadOnly<S>` wraps a file but has no natural
filesystem; a phantom `ReadOnlyFs<S::Fs>` satisfies the constraint
while inherent `ReadOnly::open` keeps taking `&S::Fs` directly.
2. `prevent_caching` moves from filesystem-instance state to per-call
`UniversalReadFs::OpenExtra: Default`. Was previously a knob on
`IoUringConfigContext` / `IoUringFs`, conflating "how this fs is
built" with "how this file is opened." Now `IoUringFs::OpenExtra =
IoUringOpenExtra { prevent_caching }`; mmap and block-cache use `()`.
`IoUringConfigContext` is gone, `TConfigContext` slims to a `Default`
marker.
Tonic StorageReadService holds `Arc<S::Fs>` (was `PhantomData<S>`); its
`new()` builds via `S::Fs::from_context(default)` and the spawn_blocking
closures clone the Arc to call instance methods.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: cfg-gate IoUringOpenExtra import for non-linux builds
The IoUringOpenExtra reexport from `universal_io` is gated on
`target_os = "linux"`. The previous commit left an unconditional import
in `persisted_hashmap/tests.rs`, breaking macOS/Windows CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: migrate simple_disk_cache to per-instance Fs API
PR #9097 (merged into dev concurrently with this branch) introduced a
`DiskCache<R: UniversalRead>` using the pre-refactor trait shape:
file-handle-as-filesystem (`R::open`, `R::list_files`), an
`OpenOptionsExtra` field on `OpenOptions`, and trait methods without
`&self`. The Fs-instance refactor on this branch removed all three.
Reshape `simple_disk_cache` to match the new design without breaking the
lazy mirror semantics:
- New `DiskCacheFs<R>` is the filesystem handle. Holds a clone of the
remote `R::Fs`; `list_files`/`exists` delegate; `from_context`
forwards to the inner Fs context. `open` constructs a `DiskCache<R>`
via the global `DiskCacheConfig`.
- `DiskCache<R>` now stores `remote_fs: R::Fs` + `remote_extra:
<R::Fs as UniversalReadFs>::OpenExtra`, so lazy remote opens go
through `self.remote_fs.open(path, options, extra)` instead of the
removed `R::open`. `open_with_config` takes the remote Fs + extra
explicitly (no more hard-coded `prevent_caching: true`; callers pass
the appropriate `OpenExtra`).
- `UniversalRead for DiskCache<R>` now declares `type Fs =
DiskCacheFs<R>` (no more `open` method on the file trait).
- Propagate the necessary bounds (`R::Fs: Clone`,
`<R::Fs as UniversalReadFs>::OpenExtra: Clone`,
`R::OwnedReadPipeline<u8, Range<u32>>: Send`) through `pipeline.rs`
free functions and impl blocks that reach into `DiskCache::remote` /
`local_state`.
- Drop the now-removed `extra: _` destructure in `LocalState::new`.
- Tests construct the remote Fs via `R::Fs::from_context(Default::default())`
and exercise `DiskCache::open_with_config`. 17 simple_disk_cache
tests pass; the 3 `empty_read_does_not_materialize_local_file`
failures pre-exist on dev (verified) and are unrelated.
`mold -run cargo clippy --all-targets` clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt on simple_disk_cache migration
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt for local_state imports after rebase
Co-authored-by: Cursor <cursoragent@cursor.com>
* uio: split DiskCacheFs into its own module
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* uio: replace TConfigContext with OpenExtra trait; move DiskCacheConfig onto DiskCacheFs
- Drop the empty TConfigContext marker; ContextConfig is now unconstrained
so backends can require explicit construction.
- Add OpenExtra trait with with_prevent_caching for backend-agnostic
per-call knobs; impl for () (no-op) and IoUringOpenExtra.
- DiskCacheFs now carries Arc<DiskCacheConfig> via the new
DiskCacheFsContext<C>; the prefill flow moves from the deleted
open_with_config into DiskCacheFs::open so Populate::Blocking /
PreferBackground work through the trait API.
- Remove the DiskCacheConfig global; callers must construct the context
explicitly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style: nightly fmt after OpenExtra refactor
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: fix spelling — Implementors → Implementers
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: possible panic insetad of error propagation
* fix: missing cfg annotation
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* fix: validate vector dimensions before WAL write for async upserts
When upserting points with wait=false (the default), dimension
mismatches were silently discarded during background processing.
The API returned 200 "acknowledged" but the points were never stored,
causing silent data loss with no error feedback to the user.
This adds an early dimension validation check in do_upsert_points()
that runs before the operation is written to WAL. This ensures that
dimension errors are returned to the client regardless of the wait
parameter, matching the behavior of wait=true.
The validation handles all vector types:
- Dense single vectors
- Multi-dense vectors
- Named vectors (dense, multi-dense, sparse)
- Sparse vectors are skipped (no fixed dimension)
Closes#9039
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor: move vector dimension validation into dedicated module
Extract validate_vector_dimensions and helper functions from update.rs
into src/common/validate_vectors.rs for better code organization.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: update shard update test for early dimension validation
The test expected a shard-level error message, but now dimension
mismatches are caught before reaching the shards. Update the assertion
to accept either the early validation error or the shard-level error.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: assert actual dimension error message in shard update test
Check for the descriptive error ("Vector dimension error: expected dim: 4, got 3")
rather than the generic shard failure wrapper.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Move element type from UniversalRead trait to method generics
Lifts the `T` parameter off `trait UniversalRead<T>` (and the matching
`UniversalWrite<T>`) and onto the read/write methods themselves. The
`ReadPipeline` associated type becomes a GAT over `T`. With per-method
generics, callers that need to read several element types from one storage
just write `S: UniversalRead` instead of stacking
`UniversalRead<u8> + UniversalRead<Counts> + ...`.
Removes the workarounds the old shape required:
- `TypedStorage<S, T>` newtype (sole purpose was disambiguating multi-bounds)
- `UniversalReadFamily` HKT shim
- `StoredGeoMapIndexStorage` four-bound trait alias
- `CachedSlice<T>` is now non-generic; `T` moves to `get_range`/`len`
No runtime behavior change: alignment in `IoUringRuntime` and `CachedSlice`
is preserved because `T` is still known at each call site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Restore TypedStorage as a typed-access fail-safe
Reintroduces `TypedStorage<S, T>` as a transparent wrapper around
`UniversalRead`/`UniversalWrite` storage that fixes the element type to
`T`. With per-method generics on the underlying traits, callers can
otherwise read or write any `T` from the same handle; this wrapper
binds it at the type level so accidental cross-type access fails to
compile.
The wrapper exposes inherent typed methods (`read::<P>`, `read_iter`,
`write`, `len`, …) that delegate to the inner storage with `T` fixed.
It does not implement `UniversalRead`/`UniversalWrite` itself — those
are intentionally avoided to prevent the typed binding from being
bypassed via the generic trait methods.
Restores the wrapping at the previous call sites: `StoredStruct`'s
inner storage, `ImmutableIdTracker`'s version mmap, the geo and
numeric index storages, the chunked-vectors chunks, and the
immutable dense vector storage.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Silence clippy::len_without_is_empty on TypedStorage
`TypedStorage::len` returns `Result<u64>` (a fallible byte length from
the underlying storage), so an `is_empty` companion would also be
fallible and offer nothing over `len()? == 0`. Suppress the lint at
the impl block, matching how the underlying `UniversalRead::len` is
already exempted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fmt
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Match the REST API by adding an optional `WriteOrdering` field to
`CreateVectorNameRequest` and `DeleteVectorNameRequest`, and propagate
it through the tonic handlers and remote-shard forwarding paths.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [ai] Always set default transfer method on entry peer
This will propagate a specific transfer type through consensus to all
peers. It ensures all peers will use the same transfer method when
applying the operation.
* [ai] Enforce the shard method to be specified
* Still allow unspecified transfer method for older peer versions
* Use snapshot shard transfers by default in Qdrant 1.18.0 and up
* New method does not have to be async
* Use stream records based transfer for replicate points with filter
* [ai] On shard snapshot transfer recovery, drop existing shard before recovery
* [ai] Add integration test to assert clearing behavior
* [ai] Debug assert our replica is not active when we clear it
* [ai] Tweak assertion
* Fix flaky test, replica may temporarily not be visible
* Replace debug assertion with runtime error
* UniversalReadPipeline
* Performance: cache pointers in MmapFile
The previous commit removed `MmapFile::read_batch` method override. So,
`MmapFile` now re-uses the default `UniversalRead::read_batch` impl,
which is implemented using `UniversalReadPipeline` interface.
Unsurprisingly, it caused a slowdown in the benchmarks, particularly
this one:
cargo bench -p common --bench universal_io -- mmap/8bytes/read_batch_full
This commit reclaims the performance back.
* remove unfulfilled lint
* Review suggestions
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* [ai] Add config boolean to disable URL based snapshot restore
* Merge if-statement
* Comment-out config option by default
* [ai] Also block partial snapshots from remote URLs
* [ai] Only run clock consistency test when staging feature is present
* [ai] Add integration test
`StrictModeConfigOutput` is an output-only type that is never validated,
so remove its dead `#[validate(...)]` field attributes.
`ShardSnapshotRecover` does need validation: add the missing `Validate`
derive and switch the actix handler from `web::Json` to `valid::Json`
so the SHA256 checksum constraint is actually enforced.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
