* feat: wire up read only field indexes
* feat: add missing indexes
* feat: add try_from impl for TextIndexParams
* refactor: unify read-only field index open and drop RocksDB storage
Merge ReadOnlyFieldIndex::open_gridstore/open_mmap into a single `open`
that picks the appendable vs immutable path from the stored
FullPayloadIndexType::storage_type. The choice is modeled as a ReadMode
(Appendable/Immutable) rather than a concrete backend, since the
read-only stack is generic over UniversalRead and mmap is now just one
implementation of it.
Also remove the unsupported RocksDb variant from payload_config's
StorageType and its now-dead match arms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: linter
* fix: linter
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: log slow operations during local shard WAL recovery
Warn when applying a single WAL operation during recovery of a local
shard takes longer than 30s, including the operation type (e.g.
PointOperation::UpsertPoints) so slow recoveries can be diagnosed.
Adds CollectionUpdateOperations::label() returning a human-readable
label including the inner variant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: reuse audit operation names for slow WAL recovery logging
Move the canonical operation-name mapping next to the
CollectionUpdateOperations definition in the shard crate as an inherent
operation_name() method, and have the audit AuditableOperation impl
delegate to it. The slow WAL recovery warning now reuses these same
names (e.g. upsert_points) instead of a duplicated label mapping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add open for read only numeric index
* feat: add parent open_*, get_mutability_type for ReadOnlyNumericIndex
* fix: naming and generic fs
* feat: detect not-found via error in numeric index open
* fix: hw counter write -> read
* feat: detect not-found via error in bool, null and geo index open
* fix: error on inconsistent storage in bool and null index open
* fix: hw counter write -> read
* feat: add readonly null index open
* feat: add open method
* refactor: split variants into modules
* chore: remove duplicated impl
* feat: remove generic in Roaring flag move into open method
* Add block level timeout to snapshot stream writer
* Add tooling to exercise streaming snapshot stalls
- tests/manual/slow_snapshot_download.py: slowly / partially download a
streaming shard snapshot from a URL to exercise sender-side backpressure.
Supports hold / rst / fin / blackhole termination to simulate a stalled,
killed, or offline (network-partitioned) consumer against a remote node.
Stdlib only; read-only against the target.
- tests/consensus_tests/test_streaming_snapshot_receiver_kill.py: throttled
receiver killed mid-flight + a second receiver, to observe whether the
sender releases the SegmentHolder lock and recovers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add open methods for map
* fix: review comments
* feat: detect not-found via error instead of path.exists in map index open
Replace the path.exists() pre-check in the read-only appendable map index
open path with error-driven detection through ok_not_found(). Generalize
OkNotFound over an IsNotFound trait, implemented for io::Error, mmap::Error,
UniversalIoError, and GridstoreError so a NotFound surfacing through any
layer (including mmap's inner io::Error) maps to Ok(None).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* review: revert internal error conversion
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: append-only mutation mode for mutable segments
Adds an opt-in per-segment switch so every mutating op tombstones the old
internal_id and writes the result at a fresh one, instead of overwriting
the underlying vector storage / payload row / field index in place.
Intended for S3-backed storages that prefer pure appends — the on-disk
structures are never rewritten.
Runtime flag, not persisted: `Segment::append_only_mutations` lives next
to `appendable_flag` on the struct, defaults to false in the builder,
and `Segment::is_append_only()` only returns true when the segment is
also appendable (clone-and-tombstone needs growable storages).
The core helper is `Segment::clone_and_mutate_point`: snapshots the
point at `old_id` into owned `NamedVectors` + `Payload`, hands them to
a closure for op-specific in-memory modification, allocates `new_id`,
writes all configured vector storages and the payload at `new_id`, and
repoints the id tracker via `set_link` (which auto-tombstones `old_id`
in the deleted bitslice). Stale slots and field-index postings keyed
by `old_id` are filtered by readers via
`filter_deferred_and_deleted` and reclaimed at optimization.
The routing policy concentrates in one new dispatcher,
`Segment::handle_point_mutate`, which takes two closures (in-place and
snapshot-mutate) and picks the path based on `is_append_only()`. All
six existing entry points in `SegmentEntry` (`upsert_point`'s
existing-pid branch, `update_vectors`, `delete_vector`,
`set_full_payload`, `set_payload`, `delete_payload`, `clear_payload`)
collapse to this dispatcher; `upsert_point`'s insert path stays on the
non-mutating fast path.
`delete_point` gets a tombstone-only variant
(`delete_point_tombstone_only`) that only flips the id-tracker deleted
bit, leaving the payload row and field-index postings at `internal_id`
in place. The append-only path skips the in-place `clear_payload` call
entirely, matching the "id tracker is the only thing we write" intent.
`NamedVectors::into_owned()` is added so the snapshot closure can take
ownership of a borrowed input wholesale.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(debug): QDRANT_APPEND_ONLY_MUTATIONS env override
Debug-only escape hatch so newly built segments default to append-only
mutation routing when QDRANT_APPEND_ONLY_MUTATIONS=1 (or true/yes) is
set in the environment. Lets us run the existing test suites against
the append-only path without wiring a collection-level config knob
first.
Release builds compile this out — the function is a const false.
Logs a single warn-level message the first time the override fires.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(append-only): skip deleted vectors on snapshot, always write payload
Two correctness fixes to clone_and_mutate_point uncovered by running
the openapi suite with QDRANT_APPEND_ONLY_MUTATIONS=1:
1. The snapshot loop read every named vector via get_vector_opt, which
returns slot bytes even when the per-vector deletion bit is set.
For sparse-only points (or any point that had update_vector(_, None)
applied) this materialised the default-zero vector as if it were
real data, wrote it at new_id, and never re-tombstoned the slot —
so dense search started scoring phantom vectors. Now we check
is_deleted_vector(old_id) and skip the read, letting the writer
loop emit update_vector(new_id, None) and re-mark the slot deleted.
2. The payload write was skipped when the snapshot ended up empty.
That dropped two side effects the field indexes rely on:
payload_storage.overwrite(new_id, empty), and the remove_point
fan-out across configured field indexes that bumps each index's
total_point_count to cover new_id. Without the bump the null index
doesn't see new_id, so is_empty / is_null filters lose the point
even though its mapping is live in the id tracker. The skip was an
optimisation, not a contract; remove it so the field indexes get
the same registration they'd get from the standard clear_payload
path.
Also collapses the debug env override to an inline cfg!()-gated
check at the struct literal — the helper with one-time logging and
multi-value matching was disproportionate for a debug-only escape
hatch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* make test_move_points_to_copy_on_write work with copy-on-write
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* test: reproduce MatchAny(any=[]) strict-mode rejection on integer index
Adds an openapi test that demonstrates the strict-mode bug where
`match: {"any": []}` on an integer-indexed payload field is rejected with:
Bad request: Index required but not found for "<field>" of one of
the following types: [keyword, uuid]
even though the field is indexed as integer. Root cause is that an empty
`any` list deserializes as `AnyVariants::Strings(empty)` (untagged enum;
Strings variant listed first), and strict mode infers a keyword/uuid
index requirement from the variant tag — ignoring that the list is
empty (i.e. a no-op condition).
The test covers:
- Baseline no-op semantics with and without indexes (passes today).
- Reproduction under strict mode for `must`, `must_not`, and a
FormulaQuery FieldCondition (currently failing; will pass once
empty `any`/`except` no longer requires an index).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: treat empty MatchAny/MatchExcept as no-op in strict-mode index check
An empty `match: {"any": []}` (or `{"except": []}`) is a no-op: `any: []`
matches nothing and `except: []` excludes nothing, regardless of the
field's data type. Because an empty list cannot carry type information it
deserializes as the keyword `AnyVariants::Strings(empty)` variant, which
previously caused strict mode to demand a keyword/uuid index and reject
the request with:
Index required but not found for "<field>" of one of the following
types: [keyword, uuid]
even on a field indexed as integer.
Fix:
- `infer_index_from_any_variants` returns no required index for an empty
variant set.
- `Extractor::update_from_condition` skips a condition whose required
index set is empty, so a no-op condition is never reported as needing
an index (this also covers the FormulaQuery condition path).
This makes the openapi reproduction in test_match_any_empty.py pass.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Strict mode: add `max_disk_usage_percent`
Mirrors `max_resident_memory_percent`: rejects disk-consuming update ops
(upsert, set/overwrite payload, update vectors) when the filesystem hosting
Qdrant storage is filled above the configured percentage. Delete-style ops
remain allowed so callers can free disk.
Disk usage is sampled via `statvfs` and TTL-cached for 5s (same cadence as
the resident-memory reader) so high-RPS request paths don't hammer the
syscall. Reader is keyed by path in `common::disk_usage` and returns `None`
on stat failure — callers (the strict-mode check) treat `None` as "skip",
matching the memory-check behaviour.
Plumbing follows the existing pattern: field on `StrictModeConfig` (+
output/diff/Hash), gRPC proto field `22`, validation 1..=100, REST/proto
conversions, and the hook into `check_strict_mode_toc_batch` alongside the
memory check (both guarded by `any_consumes_memory`).
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix CI: Windows disk_usage test + e2e WAL config
- `missing_path_returns_none` panicked on Windows because
`GetDiskFreeSpaceEx` succeeds for non-existent paths (it resolves up to
the containing drive). Relax the assertion to "must not panic; if a
value is returned it must be well-formed". The contract we care about
(None on failure) is platform-defined, not something we can portably
force.
- e2e test failed at batch 0 with "WAL buffer size exceeds available disk
space": Qdrant's existing per-shard `DiskUsageWatcher` enforces
`free >= 2 * wal_capacity_mb` and the default WAL didn't fit in the
50 MB tmpfs. Bump tmpfs to 200 MB and shrink `wal_capacity_mb` to 1 MB
(same pattern as `test_low_disk.py`) so our strict-mode gate is the
one that fires, not the WAL pre-check. Raise the gate threshold to
50% to match the larger headroom.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add Rust edge example: add-named-vector
Port `lib/edge/python/examples/add-named-vector.py` to Rust under
`lib/edge/publish/examples/src/bin/add-named-vector.rs`.
Re-export `VectorNameOperations`, `CreateVectorName`, `DeleteVectorName`,
`VectorNameConfig`, `DenseVectorConfig`, and `SparseVectorConfig` from
`qdrant_edge` so the public Rust API can create/delete named vectors
without reaching into the internal `shard` crate.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fmt
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix stale snapshot transfer recovery comment on failure/cancellation
The transfer status "comment" prefers the destination-side recovery
progress over the sender task status. Recovery progress was tracked in
`active_recoveries` and only removed via `finish_shard_recovery`, which
was called on the success path only. If `recover_shard_snapshot` returned
early - clearing the local shard, downloading the snapshot, checksum
mismatch, or cancellation (the future is spawned with
`spawn_cancel_on_drop`) - the entry leaked.
A leaked entry keeps reporting its last stage with an ever-growing
elapsed time (computed live from a fixed `Instant`), so subsequent
transfers for the same shard show stale timings until the next recovery
overwrites the entry or the node restarts.
Replace the manual start/finish pair with an RAII `ShardRecoveryGuard`
that removes the progress entry on drop, covering every exit path
including early returns and cancellation. The guard only removes its own
entry (checked via `Arc::ptr_eq`) so it never clobbers a newer recovery.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Extract recovery tracking into dedicated recovery_guard module
Move ShardRecoveryGuard out of the large shard_holder/mod.rs into a
dedicated recovery_guard.rs, and replace the verbose
`Arc<Mutex<HashMap<ShardId, Arc<Mutex<RecoveryProgress>>>>>` type with a
dedicated `ActiveRecoveries` struct that encapsulates start/comment/
set_stage/remove operations. No behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Bind recovery stage updates to the guard instance, not shard id
`ActiveRecoveries::remove` is pointer-guarded so an unwinding recovery
cannot delete a newer recovery's entry, but stage updates still resolved
the progress entry by shard id. If recovery A is still unwinding after
recovery B started for the same shard, A's late `set_stage` would mutate
B's progress and resurrect incorrect transfer comments.
Thread the guard's own progress handle (`RecoveryProgressHandle`) through
`recover_shard_snapshot_impl` -> `Collection::restore_shard_snapshot` ->
`ShardHolder::restore_shard_snapshot`, so the Unpacking/Restoring stages
(and Downloading, via `ShardRecoveryGuard::set_stage`) update that exact
recovery's progress. Direct API recoveries, which are not tracked as
transfer-side recoveries, pass `None`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Allow too_many_arguments on recover_shard_snapshot_impl
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Migrate InvertedIndexCompressedMmap to UniversalRead
* Rehaul search_scratch.rs (was scores_memory_pool.rs)
- Use `typed_arena::Arena` instead of `bumpalo::Bump`.
Reason: bump don't drop.
Drawback: `Arena::new()` is not free, and `Arena::clear()` doesn't
exist; but this commit has workarounds.
- Use names that make more sense.
* Misc fixes
* InvertedIndexCompressedMmap: explicit S type parameter
* load with any universal io
* Don't populate with sequential advice
* use appropriate fs for `exists`
* use `read_whole_via` more
* TODO
* clear ram cache after read_whole
* Oversample with approx facet
* return early for limit=0
* Document distributed limit flow on Collection::facet
Add an ASCII diagram showing how the facet limit is oversampled once on
the entry node and how peer nodes enter via facet_internal without
re-oversampling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [ai] TQDT in the API
* [ai] unify new sparse error for TQDT
* Rename to `turbo4`
* Add `Turbo4` to comments and doc strings.
* Also validate named sparse vector creation
* Abort resharding before we abort transfer
* Test resharding-down abort converges when a peer is killed mid-abort
---------
Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
`dir.try_lock()` resolves to the inherent `fs_err`/`std` `File::try_lock`,
which Rust's stdlib gates to a fixed target list that excludes
`target_os = "android"` (1.89+). On Android it returns
`ErrorKind::Unsupported` ("try_lock() not supported"), so opening a WAL —
and thus creating/loading a shard via qdrant-edge — fails.
Dispatch explicitly to `fs4::FileExt::try_lock` on the underlying
`std::fs::File`, which issues a direct `flock(LOCK_EX | LOCK_NB)` syscall
that Android supports. This restores the pre-#8770 behavior. UFCS is needed
because fs4's trait method collides by name with the inherent one (which
otherwise wins method resolution).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
