`cargo audit` flags rand 0.7.3 as unsound (RUSTSEC-2026-0097), and
permutation_iterator 0.1.2 is its sole importer. The crate is
unmaintained, so the finding is permanent for as long as we depend on
it.
Everything we used it for is "pick k distinct random indices out of n",
which is exactly `rand::seq::index::sample` from the workspace rand.
Switch the three src call sites and the two benches over, and drop the
dependency. 11 crates leave Cargo.lock.
Also fix a comment in quantile.rs claiming the permutation was
deterministic per count: the old crate keyed itself from thread_rng on
every call, so it never was.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The app telemetry was the only user of the sys-info crate, while segment
already depends on sysinfo for cgroup-aware memory accounting. Read the
distribution id/version via sysinfo statics, and the disk size fallback
via common::disk_usage, so the whole sys-info crate (and its bundled C
sources) drops out of the build. sysinfo is hoisted to a workspace
dependency, shared by the root crate and segment.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add CachedBlobFile: cached reads + write-through appends for object stores
Combine a DiskCache mirror (reads) with a BlobFile remote handle (appends)
into CachedBlobFile/CachedBlobFs, the appendable universal-IO citizen for
object stores. Appends perform the remote mutation inline and are durable
at Ok: a native write-offset append in AppendMode::Native (with a soft
limit on appends per object), or a whole-object rewrite in
AppendMode::Rewrite for stores without native append. After a successful
append the mirror length is advanced without extra IO; appended blocks
fault in from the remote on first read.
The multipart UploadPartCopy rewrite path (prefix >= 5 MiB) and the
rewrite-required error classification are left as todo!() pending the
AsyncRewrite backend capability.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Backend-advertised AppendMethod; reactive appended-block cap recovery
Replace CachedBlobFile's stored AppendMode with AsyncAppend::supported_append:
the backend advertises Native or PartialUpload, and append takes a matching
AppendRequest variant, rejecting the ones it does not support. The multipart
UploadPartCopy todo moves into the S3 backend's PartialUpload arm.
Drop the native_appends soft-limit counter: it is per-handle in-memory state
that resets on every restart, so it can never be the correctness mechanism
and persisting it would not make it authoritative either. The store is the
authority: hitting its appended-block cap now surfaces as the new
UniversalIoError::AppendRewriteRequired (S3 400 TooManyParts), and
CachedBlobFile recovers with a whole-object rewrite. Unrecognized errors
stay hard errors instead of silently triggering rewrites.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Per-store append strategies; server-side rewrites for plain S3 and GCS
Replace the single AppendContext struct with an enum of strategy objects,
one per store capability, each owning its append logic:
- NativeAppend: the signed write-offset PutObject (S3 Express, MinIO
AiStor; AwsConfig::native_append declares it for AiStor-like endpoints,
s3_express implies it).
- PartCopyAppend: plain S3 — appends land as one atomic multipart rewrite
whose prefix parts are server-side UploadPartCopy requests; nothing but
the appended data crosses the network. object_store keeps such
provider-specific calls out of its portable surface, so the requests are
hand-signed like the native append.
- ComposeAppend: GCS — the appended data is uploaded as a temporary
neighbor object and composed onto the destination server-side,
conditional on the observed generation (a real compare-and-swap).
AppendMethod is replaced by AppendSupport, which tells the caller the only
thing it needs: when the store takes a direct append. Always (native, and
compose: no part minimums, no block cap), AboveThreshold (part-copy: the
copied prefix lands as non-last multipart parts, >= 5 MiB each), or Never.
CachedBlobFile drops its hardcoded MIN_COPY_PREFIX and rewrites locally
only below the backend-advertised threshold; AppendRequest::Rewrite now
means only "append and rebuild as a single blob" — the appended-block cap
recovery.
The append module is split one file per strategy, with a shared
SignedRequestContext transport and a test-only HTTP stub.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* DiskCache tracks the remote object's etag
Seeded from the new known_etag open extra (OpenExtra::with_known_etag),
refreshed from FileInfo on schedule_reopen, and settable directly for
callers that mutate the remote out of band.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove AppendRequest enum; appended-block cap recovery moves into the backend
AsyncAppend::append takes plain (path, offset, data). A native S3 store
that rejects an append with TooManyParts now falls back to the part-copy
rewrite inside the dispatcher, instead of surfacing AppendRewriteRequired
to CachedBlobFile for a second Rewrite request. The Rewrite variant was
handled identically to Append everywhere except that one native path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Escalate to download+rewrite when the store rejects a part-copy rewrite
The cap-recovery rewrite is chosen by the store's returned error, not a
client-side threshold: a part-copy attempt rejected with EntityTooSmall
(typed as UniversalIoError::AppendEntityTooSmall, parsed from the S3
error <Code>) falls back to downloading the sub-part-minimum prefix and
PUTting the whole object back, guarded by a prefix-length offset check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix S3 Express appends: zonal endpoint + s3express SigV4 service
Hand-issued appends targeted the standard endpoint and signed as "s3",
so every append to a directory bucket got 404 NoSuchBucket, masked as
AppendOffsetConflict by the 404 mapping. Derive the zonal
{bucket}.s3express-{az}.{region} base from the mandatory --{az}--x-s3
bucket suffix (mirroring object_store's private derivation), carry the
SigV4 service name in SignedRequestContext, and treat a 404 as a
conflict only for NoSuchKey or bodiless responses — NoSuchBucket stays
a loud error guarding the endpoint derivation. extract_xml_tag moves up
to the context module and now tolerates tag attributes and
pretty-printed bodies.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Server-side etag precondition on appends; BlobFile loses UniversalAppend
AsyncAppend::append carries an expected_etag that S3 part-copy rewrites
attach as x-amz-copy-source-if-match (412 -> AppendEtagMismatch, a new
typed error) and download_rewrite checks against the GET's own etag;
native write-offset PUTs and GCS compose ignore it. BlobFile appends
only through the inherent etag-aware append_bytes now — CachedBlobFile
calls it directly with its DiskCache-tracked etag — and BlobFs's
mutating ops become inherent, delegated from CachedBlobFs, per the
standing TODOs. The append conformance battery runs over the
CachedBlobFs stack, via new direct constructors that share one backend.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drop unfulfilled too_many_arguments expectation
rewrite_parts has exactly seven parameters — at the clippy threshold,
not over it — so the lint never fires and the expect fails CI under
-D unfulfilled-lint-expectations.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A raw point can carry its payload as the byte blob it is stored as, mirroring
`PointStructRaw.raw_payload` on the internal gRPC API. The blob travels from the
sending node into the receiving node's WAL untouched, so the sender never parses
the payload it read and neither node builds a protobuf value tree for it.
It is parsed exactly once, where the operation is unpacked for apply
(`process_point_operation`), because that is the first place the parsed form is
actually needed: `set_full_payload` goes through the payload index, which cannot
be updated from bytes. The gRPC boundary therefore only checks the encoding tag
and rejects a point that sets both payload fields, the way the enclosing request
already rejects both `points` and `raw_points`.
Moving the parse onto the apply path makes its error classification load-bearing,
so a malformed blob is reported as `OperationError::MalformedPayloadBlob` — the
payload sibling of `MalformedVectorBlob`, mapped to `CollectionError::BadInput`
for the same reason: a bad blob that reached the WAL has to be skipped on replay
instead of crash-looping recovery.
Three consequences of the blob living that long are handled explicitly rather
than by convention:
- `decode_payload_raw` takes the blob only once it has parsed, so a failure
leaves the point holding it instead of holding neither representation.
- `upsert_points_raw` and `sync_points_raw` refuse a point that still carries a
blob. They read the parsed payload, so such a point would otherwise be stored
with no payload at all, and a `debug_assert!` would not catch it in release.
- `is_equal_to` compares blob to stored blob as bytes. A differing encoding costs
a redundant upsert on sync, never a skipped one.
The `raw_payload_transfer` bench measures the trade, per 100-point batch (one
transfer batch) at payloads of ~200 B / ~700 B / ~7 KB:
- Sender, storage bytes to wire: 16x / 37x / 113x faster. This is where the whole
win is — no parse of the blob that was read, no value tree built.
- WAL encode: 5x / 11x / 25x faster, writing a byte string instead of a map.
- Receiver, wire to applicable point: 1.09x / 1.10x / 1.06x. Near neutral, as it
swaps walking a prost value tree for a JSON parse.
- Wire bytes: ~6% smaller. WAL bytes: 10-32% *larger*, because the blob is JSON
while a parsed payload is written as a compact CBOR map.
The WAL growth is accepted rather than fixed: decoding earlier to win those bytes
back costs a second full deserialization, and would leave the receiving side with
a `payload_raw` that is never populated. Making the blob itself compact belongs in
the payload storage encoding (`RawPayloadEncoding` is the extension point for it),
not here.
Two flags, both off by default and both sender-only (nodes accept raw points and
raw payloads regardless), read where the transfer batch is prepared:
- `transfer_raw_points` transfers every collection as raw points, not only those
whose vector storage would drift in a decode-encode round-trip.
- `transfer_raw_payloads` ships the blob a raw read hands out; without it the
prepared batch decodes it back into the parsed payload, and the wire message is
exactly what it is today.
Neither is enabled by `all`: a node only accepts them once it runs a version that
understands them, so they can only be switched on a release later. Nothing
enforces that yet — the transfer has no peer-version gate.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Add edge-tool: CLI for creating, seeding, optimizing, and uploading local edge collections
Mirrors the style of lib/edge/tools/shard_update and shard_query: `create` builds a
minimal EdgeShard on disk (dense/sparse vectors, quantization presets including
turbo4, payload indexes, target segment count), `upsert` seeds it with random points
matching its live schema, `optimize` runs the shard optimizers, and `upload` pushes
the resulting directory to S3/GCS. Useful for quickly spinning up test collections
without a running Qdrant server, then promoting them to object storage.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* edge-tool: initialize feature flags, enable serverless_compatible, fix --sparse ambiguity
Initialize the global feature-flag OnceLock at startup (with serverless_compatible
set, cascading write_segment_manifest/append_only_mutations/compact_bitmask/
append_only_storages) so runs no longer spam "Feature flags not initialized!" and
collections are created in the serverless-compatible format.
Also splits --sparse into a plain boolean flag plus a repeatable --sparse-name:
clap's optional-value parsing for the old `--sparse [NAME]` form silently
swallowed a following positional PATH as the sparse vector's name whenever
--sparse was the last flag before it (e.g. `create --dense 1024 --sparse ./col`).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* edge-tool: fix --sparse=NAME to require_equals instead of a separate flag
The --sparse/--sparse-name split from the previous commit lost the ability to
name a sparse vector with --sparse itself. Restore a single --sparse[=NAME]
flag, but with require_equals(true): clap then only binds a value via
--sparse=NAME, never via a following bare token, so it stays safe next to the
trailing PATH positional in every position (bare --sparse, --sparse=NAME, or
multiple --sparse=NAME occurrences) without reintroducing the ambiguity that
made --sparse swallow PATH as its value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* edge-tool: remove --segments from create, it has no effect there
EdgeOptimizersConfig::default_segment_number only feeds MergeOptimizer as a
merge-down ceiling (reduce segment count when it exceeds the target); unlike
the main collection's LocalShard::build_local, EdgeShard::new never loops to
pre-create N appendable segments. A freshly created collection always starts
at exactly 1 segment, so passing --segments to `create` was silently a no-op.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* edge-tool: add --indexing-threshold-kb to create
Unlike --segments (removed previously), the indexing threshold is a parameter
IndexingOptimizer actually consults on every optimize() run: segments larger
than it get an HNSW index built. Verified end-to-end (create with a 1KB
threshold, upsert 2000 points, optimize) that it produces an hnsw-indexed
segment where it would otherwise stay plain.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* edge-tool: add --clean to upload, wiping the destination prefix first
Lists every object under DESTINATION and deletes it via ObjectStore::delete_stream
before uploading, so re-uploading a collection recreated with a different shape
(different segment UUIDs) doesn't leave the old segment's files behind.
Verified against the local S3 proxy: uploaded one collection, then a second,
differently-shaped one to the same prefix without --clean (29 objects, stale
leftovers from the first); re-uploading the second with --clean correctly
dropped it back to exactly its own 19 files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Minor version bump of the edge packages from `0.7.2` to `0.8.0`.
- `lib/edge/python/Cargo.toml`: `qdrant-edge-py` 0.7.2 -> 0.8.0
- `lib/edge/publish/amalgamate.py`: `VERSION` constant bumped
(`qdrant-edge` on crates.io)
- `lib/edge/ffi/Cargo.toml`: `qdrant-edge-ffi` bumped, kept in sync with the
other two as its header comment requires
- `lib/edge/publish/ast-grep-rules.yaml`: inline package version comment
updated (it was stale at 0.7.1)
- `Cargo.lock`: regenerated version entries
Follows the same pattern as #9252.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: `UpdateOnlySegment` / `UpdateOnlyEdgeShard` batch writer skeleton
Mirror image of the read-only pair, for the serverless updater: a
shard/segment whose public surface is writes only, built for batches of
many tiny operations against remote, append-only storage.
Implemented:
* `UpdateOnlySegment<S>` with a deliberately narrow open — id tracker,
payload storage and one storage per named vector, all cold. No vector
index, no quantized vectors, no payload index on the segments the
writer only reads from.
* `SegmentUpdateView`, the shared home of resolution logic, generic over
the component traits (`VectorDataStorageRead` is a `VectorDataRead`
without the index, so a segment that opens no index can produce the
view). Batched `locate_points` / `point_versions` /
`read_stored_points`.
* `UpdateOnlyEdgeShard<S>::apply_batch`: fold the batch to one entry per
point, locate the points, read only the ones that cannot be resolved
from the batch alone, materialize `FullyQualifiedPoint`s, append them
and tombstone the slots they replace.
`todo!()`, pending the append-only components on the roadmap (appendable
`DynamicStoredFlags` and `ChunkedVectors`, an appendable payload
blobstore and field indexes): `store_points`, `tombstone_points`,
`flush`, and creating the first appendable segment.
Filter-selected operations, point sync, conditional upserts and the
schema-level operations are rejected up front rather than silently
skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: codespell implementor → implementer in SegmentUpdateView docs
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs: trim update-only writer docstrings to guarantees
Less verbose throughout: state each function's contract — ordering,
absent-value behavior, preconditions, durability — and drop narration
about where types are used or why alternatives were rejected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: fold SegmentUpdateView into UpdateOnlySegment as inherent methods
The view was premature: it had exactly one producer, and its trait
bounds bought an unexercised option. Resolution (locate / versions /
read raw) now lives as inherent methods on UpdateOnlySegment, still
generic over the backend. A shared view can be extracted when a second
producer appears, e.g. batched CoW moves out of regular segments.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: split edge update_only batch.rs into a module
Pure move: mutation.rs (PointMutation fold + materialize), plan.rs
(UpdateBatchPlan operation intake), tests.rs. PointUpdates::new/push
narrowed to pub(super).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: parallel per-segment batch reads + tombstone every copy of a point
locate_points and read_stored_points visit segments in parallel on a
dedicated edge-update rayon pool (build_search_pool generalized to
build_segment_pool with a thread-name prefix).
locate_points now keeps every slot a point occupies, not just the
newest copy: a rewrite or delete retires all of them. Tombstoning only
the newest slot would let an older duplicate left by an interrupted
move outlive the point — and resurrect it after a delete.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: point_versions returns a map keyed by internal id
The id tracker's batch read is keyed by internal id already; returning
AHashMap drops the positions_of reverse-lookup adapter. Absent key =
unwritten slot, defaulted to version 0 at the caller.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: accept a deferred threshold when opening UpdateOnlySegment
Groundwork for an external rebuilder working the same directory: the
cutoff loads slots at or above it into the appendable id tracker's
deferred track (same appendable-only filter as ReadOnlySegment). It
hides nothing from the writer — resolution runs WithDeferred, so every
point still locates at its latest slot. The edge shard passes None
until the rebuilder coordination exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: preview_batch — resolve a batch without writing anything
apply_batch and the new preview_batch share one resolution stage
(resolve_batch: locate, read, materialize into per-point PointActions),
so a dry-run reports exactly what an apply would do. Plus
segment_configs(): per-segment configs with the write target marked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: prefetched + parallel segment opens for the update-only writer
UpdateOnlySegment::open now mirrors ReadOnlySegment::open: a
per-segment CachedFs primed by preopen, config parsed once and handed
to open_via. The edge shard opens segments in parallel on its pool,
keeping fail-hard semantics. With Populate::No throughout, prefetches
transfer no data-file content — only configs, the id tracker and the
deleted flags, whose opens consume them whole anyway.
Also: ReadOnlyAppendableIdTracker::preopen now tolerates the
not-yet-created mappings/versions files of an empty appendable segment,
matching its open's contract — previously unreachable because followers
skip appendable segments on error, while the writer must open them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: edge-shard-update — dry-run batch upserts against a shard
Counterpart of edge-shard-query for the write path: opens an
UpdateOnlyEdgeShard over a local directory or S3/GCS object storage,
generates random points shaped by the shard's own schema (segment
config + payload-index schema), and logs what applying them would do —
locations, versions, actions, tombstones — via preview_batch. Nothing
is written: the write half is still todo!().
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: box PointAction::Store to appease clippy::large_enum_variant
A resolved point is ~384 bytes while every other variant is empty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: adapt to dev's dead-code sweep (#10030)
Restore NamedVectors::remove_ref — removed as dead on dev, but the
batch fold's DeleteVectors arm is now its first caller. Drop the
allow(dead_code) on segment::update_only (no longer needed) and switch
the writer's unread fs field to expect(dead_code), per the new
ast-grep rule.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: pin per-shard search pool to a core
* fix: plumb search_pool_core through bindings
* feat: expose search_pool_core in python bindings
* fix: validate search pool core before pinning
* chore: trim comments
dependabot[bot]