mirror of
https://github.com/qdrant/qdrant.git
synced 2026-09-27 08:27:40 -05:00
edge-docs-diff
24
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xzfc
|
19d4da9988 |
UIO tracing and visualizer (#10742)
* Add uio_trace module and visualizer * Handle object store, not just gRPC * Improve visualizer * Improve visualizer [2] |
||
|
Andrey Vasnetsov
|
8f56a945f1 |
Add shared DiskCache statistics and latency histogram (#10637)
* [AI] Add shared disk cache statistics and latency histogram * [AI] Document disk cache statistics observer identity * [AI] Simplify disk cache statistics by removing pipeline error counters * [AI] Limit disk cache statistics to remote fetches and trim redundant tests * [AI] Close remote append handle before reload statistics snapshot |
||
|
Luis CossíoandClaude Fable 5
|
690d92e751 |
[updater] genericize fs to use UniversalAppendFs (#10451)
* introduce UniversalAppendFs helper * AI: migrate to UniversalAppendFs bound Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * don't use it in Gridstore --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
qdrant-cloud-bot
|
ca1113a8fa |
Rename ReadOnlyEdgeShard::refresh to live_reload (#10444)
Align the Edge follower API with the segment-level LiveReload naming used everywhere else, including the module, lock, and docs. |
||
|
Andrey VasnetsovandClaude Opus 5
|
81e9fb3e75 |
[UpdateOnly] Honor upsert update_mode in the batch writer (#10236)
* [UpdateOnly] Honor upsert update_mode in the batch writer The writer rejected every `UpsertPointsConditional`. Accept the ones whose condition is empty — `insert_only` and `update_only` — since existence is the whole gate they need, and locating a batch's points already answers it. The gate is evaluated per mutation at its position in the fold, so an `insert_only` upsert sees a point an earlier operation of the same batch created, matching a leader that resolves each operation only after the ones before it were applied. A conditional upsert may therefore not discard the mutations it follows. Rejecting an upsert also means never reading the point it would have overwritten: `needs_stored_point` asks whether the first mutation that applies to an existing point discards it, so an `insert_only` batch pays nothing for the ids that are already taken. A conditional upsert carrying a real filter is still rejected — evaluating one needs payload indexes the writer never fetches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] State contracts in the update-mode docstrings Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [UpdateOnly] Trim the update-mode diff Drop `--update-mode` from edge-shard-update: the modes are covered by unit and end-to-end tests, and the flag cost a wrapper enum, a conversion and a parameter threaded through both run paths. The tool still reports rejected points, which the exhaustive match requires. Inline `always_applies` into its one caller, drop the two test-batch wrappers over `conditional_batch`, drop the `update_only` fold test whose truth table two other tests already assert, and shorten two over-long comment blocks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Fable 5
|
5f8cef9ebd |
[UpdateOnly] Writer over object storage (#10214)
* Drop the vestigial UniversalWrite bound from the update-only writer Neither writer kind performs in-place writes: AppendableSegment is built on UniversalAppend, and DeleteOnlySegment tombstones via whole-mask atomic_save (UniversalWriteFileOps), which UniversalAppend's supertrait already carries. The bound is a leftover from the DiskIdTracker-based iterations that mutated the deleted mask in place. With it gone, UpdateOnlyEdgeShard::apply_batch is instantiable with the object-store-appendable CachedBlobFile. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * edge-shard-update: --apply writes the batch, over object storage too Open the object-storage backends through CachedBlobFs/CachedBlobFile instead of the read-only DiskCacheFs handle, so the shard is appendable in both modes, and add --apply: generate the same schema-derived batch and run apply_batch instead of preview_batch. Dry run stays the default and the generation is shared, so the preview cannot drift from what an apply would do. AwsConfig::native_append is exposed as --native-append for AiStor/RustFS-style endpoints; the Cached* types join io_bridge_object_store's re-export of the io_bridge stack. Applying to a leader-produced shard currently fails with a clean refusal — its appendable segment's payload storage was created in mutable mode, which the append-only writer rejects — the known segment-bootstrap gap, next in line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * CachedBlobFile: latency tracing for append_bytes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * UpdateOnlyEdgeShard: sequential batches through one writer Writers open once at shard open, next to the lookup segments they resume from. apply_batch hands the writer back on success, live-reloading the lookup half of every segment the batch wrote to (new LookupSegment::live_reload, mirroring the read-only segment's); on error the writer is consumed, since its lookups may no longer describe the durable state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * edge-shard-update: --interactive mode, sequential batches on one writer After each applied batch, prompt on stdin for the next round's ids and apply them through the writer apply_batch handed back — no shard re-open — with op-num (and seed) incremented per round. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * CachedBlobFile: create the missing object on an offset-0 rewrite append The caller-side rewrite path (part-copy S3 stores below the direct-append threshold) validated the offset against the mirror length, whose initialization HEAD-requests the remote and surfaced NotFound for an object that does not exist yet. Direct-append backends (GCS compose, native append) already create the object on an offset-0 append; the rewrite now reads a missing remote as length zero so its whole-object PUT does the same, and a non-zero offset against a missing object reports an offset conflict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * UpdateBatchOutcome: per-point records of retired slots Each applied point now carries a PointApplyRecord: what happened to it (stored/deleted/skipped/missing) and which slots it vacated where — tombstoned per segment, or superseded in place for the old write-target copy of a stored point. Built in the same loop that decides tombstone-vs-supersede, so the report cannot drift from the writes. edge-shard-update logs one line per point after the applied summary, telling a fresh insert from an overwrite and naming the segments the old copies were deleted from. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Fable 5
|
06ffcb881f |
Add CachedBlobFile: cached reads + write-through appends for object stores (#10206)
* Add CachedBlobFile: cached reads + write-through appends for object stores Combine a DiskCache mirror (reads) with a BlobFile remote handle (appends) into CachedBlobFile/CachedBlobFs, the appendable universal-IO citizen for object stores. Appends perform the remote mutation inline and are durable at Ok: a native write-offset append in AppendMode::Native (with a soft limit on appends per object), or a whole-object rewrite in AppendMode::Rewrite for stores without native append. After a successful append the mirror length is advanced without extra IO; appended blocks fault in from the remote on first read. The multipart UploadPartCopy rewrite path (prefix >= 5 MiB) and the rewrite-required error classification are left as todo!() pending the AsyncRewrite backend capability. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Backend-advertised AppendMethod; reactive appended-block cap recovery Replace CachedBlobFile's stored AppendMode with AsyncAppend::supported_append: the backend advertises Native or PartialUpload, and append takes a matching AppendRequest variant, rejecting the ones it does not support. The multipart UploadPartCopy todo moves into the S3 backend's PartialUpload arm. Drop the native_appends soft-limit counter: it is per-handle in-memory state that resets on every restart, so it can never be the correctness mechanism and persisting it would not make it authoritative either. The store is the authority: hitting its appended-block cap now surfaces as the new UniversalIoError::AppendRewriteRequired (S3 400 TooManyParts), and CachedBlobFile recovers with a whole-object rewrite. Unrecognized errors stay hard errors instead of silently triggering rewrites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Per-store append strategies; server-side rewrites for plain S3 and GCS Replace the single AppendContext struct with an enum of strategy objects, one per store capability, each owning its append logic: - NativeAppend: the signed write-offset PutObject (S3 Express, MinIO AiStor; AwsConfig::native_append declares it for AiStor-like endpoints, s3_express implies it). - PartCopyAppend: plain S3 — appends land as one atomic multipart rewrite whose prefix parts are server-side UploadPartCopy requests; nothing but the appended data crosses the network. object_store keeps such provider-specific calls out of its portable surface, so the requests are hand-signed like the native append. - ComposeAppend: GCS — the appended data is uploaded as a temporary neighbor object and composed onto the destination server-side, conditional on the observed generation (a real compare-and-swap). AppendMethod is replaced by AppendSupport, which tells the caller the only thing it needs: when the store takes a direct append. Always (native, and compose: no part minimums, no block cap), AboveThreshold (part-copy: the copied prefix lands as non-last multipart parts, >= 5 MiB each), or Never. CachedBlobFile drops its hardcoded MIN_COPY_PREFIX and rewrites locally only below the backend-advertised threshold; AppendRequest::Rewrite now means only "append and rebuild as a single blob" — the appended-block cap recovery. The append module is split one file per strategy, with a shared SignedRequestContext transport and a test-only HTTP stub. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * DiskCache tracks the remote object's etag Seeded from the new known_etag open extra (OpenExtra::with_known_etag), refreshed from FileInfo on schedule_reopen, and settable directly for callers that mutate the remote out of band. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Remove AppendRequest enum; appended-block cap recovery moves into the backend AsyncAppend::append takes plain (path, offset, data). A native S3 store that rejects an append with TooManyParts now falls back to the part-copy rewrite inside the dispatcher, instead of surfacing AppendRewriteRequired to CachedBlobFile for a second Rewrite request. The Rewrite variant was handled identically to Append everywhere except that one native path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Escalate to download+rewrite when the store rejects a part-copy rewrite The cap-recovery rewrite is chosen by the store's returned error, not a client-side threshold: a part-copy attempt rejected with EntityTooSmall (typed as UniversalIoError::AppendEntityTooSmall, parsed from the S3 error <Code>) falls back to downloading the sub-part-minimum prefix and PUTting the whole object back, guarded by a prefix-length offset check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix S3 Express appends: zonal endpoint + s3express SigV4 service Hand-issued appends targeted the standard endpoint and signed as "s3", so every append to a directory bucket got 404 NoSuchBucket, masked as AppendOffsetConflict by the 404 mapping. Derive the zonal {bucket}.s3express-{az}.{region} base from the mandatory --{az}--x-s3 bucket suffix (mirroring object_store's private derivation), carry the SigV4 service name in SignedRequestContext, and treat a 404 as a conflict only for NoSuchKey or bodiless responses — NoSuchBucket stays a loud error guarding the endpoint derivation. extract_xml_tag moves up to the context module and now tolerates tag attributes and pretty-printed bodies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Server-side etag precondition on appends; BlobFile loses UniversalAppend AsyncAppend::append carries an expected_etag that S3 part-copy rewrites attach as x-amz-copy-source-if-match (412 -> AppendEtagMismatch, a new typed error) and download_rewrite checks against the GET's own etag; native write-offset PUTs and GCS compose ignore it. BlobFile appends only through the inherent etag-aware append_bytes now — CachedBlobFile calls it directly with its DiskCache-tracked etag — and BlobFs's mutating ops become inherent, delegated from CachedBlobFs, per the standing TODOs. The append conformance battery runs over the CachedBlobFs stack, via new direct constructors that share one backend. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Drop unfulfilled too_many_arguments expectation rewrite_parts has exactly seven parameters — at the clippy threshold, not over it — so the lint never fires and the expect fails CI under -D unfulfilled-lint-expectations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Daniel Boros
|
638f8aad63 |
edge-tool: bound upload and upsert memory, skip the WAL, fix generated payload paths (#10189)
* fix: bound edge-tool resource use and fix generated payload paths * fix: keep sibling array elements when merging generated payload paths |
||
|
Andrey VasnetsovandClaude Sonnet 5
|
ca20151659 |
Add edge-tool: CLI for creating, seeding, optimizing, and uploading local edge collections (#10159)
* Add edge-tool: CLI for creating, seeding, optimizing, and uploading local edge collections Mirrors the style of lib/edge/tools/shard_update and shard_query: `create` builds a minimal EdgeShard on disk (dense/sparse vectors, quantization presets including turbo4, payload indexes, target segment count), `upsert` seeds it with random points matching its live schema, `optimize` runs the shard optimizers, and `upload` pushes the resulting directory to S3/GCS. Useful for quickly spinning up test collections without a running Qdrant server, then promoting them to object storage. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: initialize feature flags, enable serverless_compatible, fix --sparse ambiguity Initialize the global feature-flag OnceLock at startup (with serverless_compatible set, cascading write_segment_manifest/append_only_mutations/compact_bitmask/ append_only_storages) so runs no longer spam "Feature flags not initialized!" and collections are created in the serverless-compatible format. Also splits --sparse into a plain boolean flag plus a repeatable --sparse-name: clap's optional-value parsing for the old `--sparse [NAME]` form silently swallowed a following positional PATH as the sparse vector's name whenever --sparse was the last flag before it (e.g. `create --dense 1024 --sparse ./col`). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: fix --sparse=NAME to require_equals instead of a separate flag The --sparse/--sparse-name split from the previous commit lost the ability to name a sparse vector with --sparse itself. Restore a single --sparse[=NAME] flag, but with require_equals(true): clap then only binds a value via --sparse=NAME, never via a following bare token, so it stays safe next to the trailing PATH positional in every position (bare --sparse, --sparse=NAME, or multiple --sparse=NAME occurrences) without reintroducing the ambiguity that made --sparse swallow PATH as its value. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: remove --segments from create, it has no effect there EdgeOptimizersConfig::default_segment_number only feeds MergeOptimizer as a merge-down ceiling (reduce segment count when it exceeds the target); unlike the main collection's LocalShard::build_local, EdgeShard::new never loops to pre-create N appendable segments. A freshly created collection always starts at exactly 1 segment, so passing --segments to `create` was silently a no-op. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: add --indexing-threshold-kb to create Unlike --segments (removed previously), the indexing threshold is a parameter IndexingOptimizer actually consults on every optimize() run: segments larger than it get an HNSW index built. Verified end-to-end (create with a 1KB threshold, upsert 2000 points, optimize) that it produces an hnsw-indexed segment where it would otherwise stay plain. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * edge-tool: add --clean to upload, wiping the destination prefix first Lists every object under DESTINATION and deletes it via ObjectStore::delete_stream before uploading, so re-uploading a collection recreated with a different shape (different segment UUIDs) doesn't leave the old segment's files behind. Verified against the local S3 proxy: uploaded one collection, then a second, differently-shaped one to the same prefix without --clean (29 objects, stale leftovers from the first); re-uploading the second with --clean correctly dropped it back to exactly its own 19 files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
9f3c07b00d |
feat: UpdateOnlySegment / UpdateOnlyEdgeShard batch writer skeleton (#10021)
* feat: `UpdateOnlySegment` / `UpdateOnlyEdgeShard` batch writer skeleton Mirror image of the read-only pair, for the serverless updater: a shard/segment whose public surface is writes only, built for batches of many tiny operations against remote, append-only storage. Implemented: * `UpdateOnlySegment<S>` with a deliberately narrow open — id tracker, payload storage and one storage per named vector, all cold. No vector index, no quantized vectors, no payload index on the segments the writer only reads from. * `SegmentUpdateView`, the shared home of resolution logic, generic over the component traits (`VectorDataStorageRead` is a `VectorDataRead` without the index, so a segment that opens no index can produce the view). Batched `locate_points` / `point_versions` / `read_stored_points`. * `UpdateOnlyEdgeShard<S>::apply_batch`: fold the batch to one entry per point, locate the points, read only the ones that cannot be resolved from the batch alone, materialize `FullyQualifiedPoint`s, append them and tombstone the slots they replace. `todo!()`, pending the append-only components on the roadmap (appendable `DynamicStoredFlags` and `ChunkedVectors`, an appendable payload blobstore and field indexes): `store_points`, `tombstone_points`, `flush`, and creating the first appendable segment. Filter-selected operations, point sync, conditional upserts and the schema-level operations are rejected up front rather than silently skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: codespell implementor → implementer in SegmentUpdateView docs Co-authored-by: Cursor <cursoragent@cursor.com> * docs: trim update-only writer docstrings to guarantees Less verbose throughout: state each function's contract — ordering, absent-value behavior, preconditions, durability — and drop narration about where types are used or why alternatives were rejected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: fold SegmentUpdateView into UpdateOnlySegment as inherent methods The view was premature: it had exactly one producer, and its trait bounds bought an unexercised option. Resolution (locate / versions / read raw) now lives as inherent methods on UpdateOnlySegment, still generic over the backend. A shared view can be extracted when a second producer appears, e.g. batched CoW moves out of regular segments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: split edge update_only batch.rs into a module Pure move: mutation.rs (PointMutation fold + materialize), plan.rs (UpdateBatchPlan operation intake), tests.rs. PointUpdates::new/push narrowed to pub(super). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: parallel per-segment batch reads + tombstone every copy of a point locate_points and read_stored_points visit segments in parallel on a dedicated edge-update rayon pool (build_search_pool generalized to build_segment_pool with a thread-name prefix). locate_points now keeps every slot a point occupies, not just the newest copy: a rewrite or delete retires all of them. Tombstoning only the newest slot would let an older duplicate left by an interrupted move outlive the point — and resurrect it after a delete. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: point_versions returns a map keyed by internal id The id tracker's batch read is keyed by internal id already; returning AHashMap drops the positions_of reverse-lookup adapter. Absent key = unwritten slot, defaulted to version 0 at the caller. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: accept a deferred threshold when opening UpdateOnlySegment Groundwork for an external rebuilder working the same directory: the cutoff loads slots at or above it into the appendable id tracker's deferred track (same appendable-only filter as ReadOnlySegment). It hides nothing from the writer — resolution runs WithDeferred, so every point still locates at its latest slot. The edge shard passes None until the rebuilder coordination exists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: preview_batch — resolve a batch without writing anything apply_batch and the new preview_batch share one resolution stage (resolve_batch: locate, read, materialize into per-point PointActions), so a dry-run reports exactly what an apply would do. Plus segment_configs(): per-segment configs with the write target marked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: prefetched + parallel segment opens for the update-only writer UpdateOnlySegment::open now mirrors ReadOnlySegment::open: a per-segment CachedFs primed by preopen, config parsed once and handed to open_via. The edge shard opens segments in parallel on its pool, keeping fail-hard semantics. With Populate::No throughout, prefetches transfer no data-file content — only configs, the id tracker and the deleted flags, whose opens consume them whole anyway. Also: ReadOnlyAppendableIdTracker::preopen now tolerates the not-yet-created mappings/versions files of an empty appendable segment, matching its open's contract — previously unreachable because followers skip appendable segments on error, while the writer must open them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: edge-shard-update — dry-run batch upserts against a shard Counterpart of edge-shard-query for the write path: opens an UpdateOnlyEdgeShard over a local directory or S3/GCS object storage, generates random points shaped by the shard's own schema (segment config + payload-index schema), and logs what applying them would do — locations, versions, actions, tombstones — via preview_batch. Nothing is written: the write half is still todo!(). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: box PointAction::Store to appease clippy::large_enum_variant A resolved point is ~384 bytes while every other variant is empty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: adapt to dev's dead-code sweep (#10030) Restore NamedVectors::remove_ref — removed as dead on dev, but the batch fold's DeleteVectors arm is now its first caller. Drop the allow(dead_code) on segment::update_only (no longer needed) and switch the writer's unread fs field to expect(dead_code), per the new ast-grep rule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Andrey VasnetsovandClaude Opus 4.8
|
8888046cd3 |
Add usage skill file for edge-shard-query tool (#9836)
Standalone usage reference for `edge-shard-query`: backends (S3 / GCS / uio-grpc), connection and tuning flags, the scroll / search / search-sparse sub-commands, filtering, and the live-reload diff mode. Written to stand on its own, so it can be shared as a link without also sharing the source. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Fable 5
|
a5a0ecc5e9 |
Add search-sparse sub-command to edge-shard-query tool (#9834)
Sparse nearest-neighbour search over a ReadOnlyEdgeShard, reusing the same SearchRequest path as dense search with VectorInternal::Sparse. The query vector is accepted as a JSON object or an index:value pair list, sorted and validated before use. --vector is required (no random fallback: the sparse vocabulary is not recoverable from the shard config), and --hnsw-ef is omitted since it does not apply to the sparse index. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Fable 5
|
43e3d6ea8d |
[UIO] Request-specific load profile for read-only opens (#9797)
* Introduce request-specific LoadProfile with per-component placement A read-only shard opened for one known request (the serverless cold-start path) doesn't have to warm components the request will never touch. LoadProfile captures that from the request: warm components keep the persisted-config placement, everything else is parked cold. All placement decisions live in one place, so the memory placement of a whole segment under a profile is reviewable in one file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Thread LoadProfile through the read-only segment open ReadOnlySegment::open takes an optional profile; first_preopen and open_via resolve it into per-component populate overrides so the opens make the same placement decisions the prefetches did. Pinned components that materialize on open regardless (quantized RAM storage kinds, the immutable-RAM sparse index) and appendable components ignore the override; the HNSW graph and immutable payload indexes demote fully. Config reloads follow the new config alone and pass no override. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Open ReadOnlyEdgeShard under a request-derived load profile ReadOnlyEdgeShard::open takes an optional LoadProfile, applies it to every segment open and keeps it so segments discovered by a later refresh load with the same placement. ScrollRequestInternal and CoreSearchRequest gain load_profile() constructors, and edge-shard-query builds the request before the open and passes its profile (opt out with --no-load-profile). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Demote pinned quantized vectors and sparse index under a cold profile Within the immutable layout the quantized RAM and mmap loaders share the on-disk format — only how the data is brought into memory differs — and the immutable-RAM sparse index has the same lazy mmap open low-memory mode already downgrades to. So a cold populate override now demotes the effective placement itself (Memory::with_populate_override, shared with the HNSW residency mapping) instead of only skipping cache priming: a pinned quantized storage opens the mmap kind cold, and a pinned sparse index opens as Mmap, so neither reads its data on a cold start. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Add LoadProfile::merge for composite queries A composite query runs multiple core requests — e.g. a hybrid search runs one core search per vector, each with its own filter. Its profile is the union of its parts': merge extends the warm sets and ORs the payload-storage flag, so a component either part needs warm stays warm. The union is sound because every placement method is monotone in the warm sets (growing them only turns "park cold" into "keep configured placement"), so the merged profile dominates each input; and minimal, warming nothing no part asked for. Combine profiles with reduce, not fold: merge's identity element is the coldest profile (empty warm sets), the opposite of passing no profile at all — deliberately no empty()/Default constructor exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Adapt vanished-segment test to the profile-aware open signature The test landed on dev (#9777) after the load-profile signature change was written, so the rebase left its ReadOnlySegment::open calls without the new load_profile argument. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Defer vector index open entirely under a cold load profile A cold placement is not enough for the vector index on remote backends: GraphLinksView requires the whole links file as one contiguous slice, and the disk cache can only lend a borrowed slice once every block is locally present — so even a Cold HNSW open mirrors the entire links_compressed.bin (8.2 MB / 1.1 s per segment in the serverless cold-start trace), plus the unconditional graph.bin metadata read. The only way not to fetch the index is not to open it. LoadProfile::vector_index_placement is replaced by vector_index_deferred: a vector the request never scores now gets a DeferredVectorIndex — a new VectorIndexReadEnum variant holding the open arguments (an owned clone of the segment's raw backend, path, config, shared component handles) and a OnceLock. Nothing is opened or prefetched for it at segment open. Per-method policy of the deferred variant: - search, fill_idf_statistics and populate open the index on first use (with the cold placement the profile chose), so the profile contract holds: a request the profile did not predict still works, just pays the open then; - is_index reports true without opening (deferral only ever wraps a real HNSW or sparse index; plain opens no files and is never deferred); - telemetry, indexed_vector_count and sizes answer conservative defaults rather than trigger a remote fetch for a statistic. Tests: deleting the vector_index directory before an open under a scroll profile leaves open, filtered reads and payload reads working — proof that nothing of the index is read — while a search surfaces the missing files; and a segment opened under a scroll profile answers searches identically to an eagerly opened one via the transparent first-use open. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Make --vector optional in edge-shard-query: random query after open Omitting --vector on the search sub-command now searches with a random vector. The request is still built before the shard opens — the load profile only needs the vector name, not its values — with an empty placeholder; once the shard is open, fill_random_vector reads the dimension of the queried vector from the derived shard config and fills in uniform-random f32s (with a clear error if the named vector is not in the config). The vector is generated once, so live-reload iterations re-run the identical random query and the printed diffs stay meaningful. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Scope index deferral to the HNSW graph via a lazy OnceLock load Replace the DeferredVectorIndex wrapper (and the VectorIndexReadEnum:: Deferred variant) with deferral inside ReadOnlyHNSWIndex itself: the graph lives in a OnceLock (same first-wins arbitration as ReadOnlyRoaringFlags::bitmap) alongside the retained raw backend and residency, and loads on first use with a cold placement. The config read stays eager — one tiny, absence-tolerated file — so telemetry, is_on_disk and indexed_vector_count report real values where the Deferred arms answered with hard defaults. The sparse index needs no deferral: its mmap open reads lazily, with only small JSON metadata eager. A profile that never scores the vector now passes a cold placement override (LoadProfile:: vector_index_placement) into the eager open_sparse, which demotes ImmutableRam to the lazy Mmap open like low-memory mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Express HNSW graph deferral as a populate override, not a bool param Replace the `deferred: bool` on the read-only HNSW open/preopen (and the VectorIndexReadEnum pass-through) with the same `populate_override: Option<Populate>` every other component takes. A cold *override* defers the graph load — graph_deferred() mirrors the cold-override match of open_sparse — while a config-derived cold placement (or the low-memory clamp) keeps the eager load, since only a request-specific override carries the "never scored" prediction. With dense and sparse now consuming the same signal, LoadProfile::vector_index_deferred is gone: a single vector_index_placement() serves both index kinds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Serialize the deferred graph load via once_cell's get_or_try_init Loading outside the lock (std OnceLock's fallible init is still unstable) let a search burst on a deferred vector fetch the whole graph once per thread. Swap the cell for once_cell::sync::OnceCell: the fallible load runs inside the cell's lock, concurrent first users block on the one load, and a failed load leaves the cell empty so the next caller retries. Addresses https://github.com/qdrant/qdrant/pull/9797#discussion_r3573727836 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Fable 5
|
8f54076cbe |
Add uio-grpc backend and key-triggered live-reload to edge-shard-query (#9795)
* Add uio-grpc backend and key-triggered live-reload to edge-shard-query --backend uio-grpc opens the shard directly over a running Qdrant peer's StorageRead gRPC service (public gRPC endpoint, api-key aware), addressed by --collection/--shard-id — no object storage involved. The UioGrpcSource backend existed since #9634 but was never wired into the tool's CLI. --bucket is now per-backend optional (required for aws/gcs), and the default cache dir is scoped by collection/shard for uio-grpc, whose mirror has no distinguishing key prefix. --live-reload-key runs the same watch loop as --live-reload with each reload triggered by pressing Enter instead of a timer — easier when stepping through a debug scenario. Timer mode is unchanged; the two flags are mutually exclusive. Closed stdin ends the loop gracefully, so the flag cannot busy-loop on piped input (and is documented as incompatible with @- stdin request arguments). Verified end-to-end against a live instance started with QDRANT__FEATURE_FLAGS__WRITE_SEGMENT_MANIFEST=true: scroll over uio-grpc returns all points with payloads, timer mode picks up an upsert + delete as +/- diff lines, and key mode fires one reload per Enter and exits cleanly on EOF. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Log StorageRead not-found gRPC failures at debug level A read-only follower routinely probes files the writer creates lazily (e.g. the mutable id tracker's mappings and versions before the first flush), so every uio-grpc follower poll spammed INFO logs like: gRPC /qdrant.StorageRead/FileLength failed with NotFound "File not found: .../mutable_id_tracker.versions" NotFound on /qdrant.StorageRead/* now logs at debug; NotFound on all other services (missing collection etc.) stays at info. The service is matched by parsing the path's service component and comparing it to the tonic-generated storage_read_server::SERVICE_NAME constant rather than a hard-coded path string. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Fable 5
|
a43abbd3b8 |
Add --live-reload watch mode to edge-shard-query (#9791)
With --live-reload <SECONDS> the tool keeps running after the first answer: every interval it refreshes the ReadOnlyEdgeShard from object storage, re-runs the same request, and prints the difference against the previous results — `+` id appeared, `-` id disappeared, `~ old -> new` for changed content (payload/vector/score/version). Each cycle prints a summary line; pure reordering prints nothing. The request is parsed once into a PreparedRequest (ScrollRequest / SearchRequest are Clone), so every iteration re-runs the identical request and the filter/vector parsing and logging no longer repeat. The first run prints the full result set in the existing format. A failed refresh is logged and retried next tick — the shard keeps serving its previous state, so a transiently unreachable bucket does not kill the watch loop. Status lines go to stderr via log, diff rows to stdout, keeping stdout machine-consumable. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Daniel Boros
|
78688733cb | feat: support S3 Express One Zone buckets in the S3 bridge (#9757) | ||
|
Andrey VasnetsovandClaude Opus 4.8
|
02cacfe192 |
debug logging of the s3 connection (#9597)
* debug logging of the s3 connection * refactor(io_bridge): make S3 latency logs opt-in and low-overhead Move the blob-backend latency traces onto a dedicated `io_bridge::latency` log target at `trace` level, so they are silent by default and can be toggled as one group at runtime without a rebuild (e.g. `RUST_LOG=io_bridge::latency=trace`). Guard the timing `Instant::now()` behind `log_enabled!` so there is no overhead when the target is disabled. Also add `list_files` timing, and switch the shard_query CLI logger to millisecond timestamp resolution. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Opus 4.8
|
8311ab62a9 |
feat(edge): add --search-threads to shard_query tool (#9736)
Add a `--search-threads` option to the edge-shard-query tool so the number of threads in the shard's search thread pool can be specified. When set, it builds an EdgeConfig with `max_search_threads` and passes it to `ReadOnlyEdgeShard::open`, overriding the CPU-derived default used for both parallel segment reads at open and running searches. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Fable 5
|
729dc6af43 |
Make EdgeConfig tunables optional with a layered fallback chain (#9714)
Tunable EdgeConfig parameters (on_disk_payload, hnsw_config, optimizers) are now Option, and every tunable resolves through the fallback chain provided -> persisted -> derived from segments -> default when loading an existing shard. Leaving a parameter unspecified keeps the shard as it is; an explicit value overwrites it and existing segments converge to it through the optimizers. vectors/sparse_vectors are excluded from overwrite semantics: an empty map inherits the persisted/segment-derived definitions, a non-empty map is validated for compatibility against the loaded segments (size, distance, multivector, datatype, sparse modifier) and fails the load on mismatch. The derived layer folds over all segments in UUID order instead of taking an arbitrary first segment, so a plain appendable segment (which carries no HNSW parameters) can never mask an indexed segment's actual build parameters. Previously a lost edge_config.json could resolve unspecified HNSW params to compiled-in defaults and silently trigger a full re-index via ConfigMismatchOptimizer. The read-only follower accepts an optional config on open: provided tunables are applied once over the segment-derived config (vectors always come from the segments), and refresh re-derives from segments alone. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Opus 4.8
|
5dec49802e |
Split io_bridge core out of io_bridge_object_store; add uio backend (#9634)
* Split io_bridge core out of io_bridge_object_store; add uio backend Separate the backend-agnostic sync<->async bridge from the object-store backends so a second backend can plug in, and add a uio-client backend. - io_bridge (new): the bridge core (AsyncRead, BridgeRuntime, BlobFile, BlobFs, pipeline), with no object_store/tonic deps. BridgeRuntime::block_on is now pub so sibling crates can drive it. - io_bridge_object_store (slimmed): keeps BlobBackend + S3/GCS/Azure. The blanket `AsyncRead for Arc<S>` would now break the orphan rule, so it moves onto a local newtype `ObjectStoreSource<S>`. Re-exports the bridge core. - io_bridge_uio (new): UioSource implements AsyncRead over uio_client::Client, pinned to one (collection, shard_id) replica. Streams gRPC chunks into the aligned buffer; read_from = FileLength + tail stream; writes are Unsupported (read-only service). The client is built lazily on first use inside the bridge runtime, because tonic's connect_lazy captures the reactor at construction and would panic in the runtime-free AsyncRead::open. - uio-client: add sync connect_lazy, read_bytes_stream_raw (BoxStream<Bytes>), NOT_FOUND -> UniversalIoError::NotFound mapping, and an api-key auth interceptor (UioConfig::api_key) injecting the `api-key` gRPC header. - common: add UniversalKind::Uio. - Consumers (segment, edge-shard-query): Arc<AmazonS3> -> ObjectStoreSource<_>. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Rename uio crates/types to uio-grpc Clarify that this "uio" backend is the gRPC transport variant (the "uio" shorthand for Universal IO is used elsewhere in the tree and is left alone). - crate uio-client -> uio-grpc-client (dir lib/uio-client -> lib/uio-grpc-client) - crate io_bridge_uio -> io_bridge_uio_grpc - UniversalKind::Uio -> UniversalKind::UioGrpc - UioConfig/UioSource/UioFile/UioFs -> UioGrpc{Config,Source,File,Fs} - doc/import references updated accordingly Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Opus 4.8
|
33771fb6c1 |
Generalize edge S3 scroll tool into edge-shard-query (#9598)
Turn the single-purpose `edge-s3-scroll` tool into a sub-command-based `edge-shard-query` that can run different read requests against a ReadOnlyEdgeShard opened over object storage. - Add `scroll` and `search` sub-commands; shared connection/cache args live at the top level. - Accept arbitrary payload filters as JSON via `--filter` (curl `--data` style: literal JSON, `@file`, or `@-` for stdin), parsed straight into the filter DSL. Keep `--filter-key`/`--filter-value` as a shortcut. - `search` accepts a query vector (JSON array, comma list, or `@file`/`@-`), named vector, offset, score threshold, and HNSW params. - Rename the package/binary/dir (`s3_scroll` -> `shard_query`) since the tool is no longer scroll-only nor S3-only. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Luis Cossío
|
a3c313b9a6 |
Add s3_proxy.sh tool to simulate an S3 bucket locally (#9589)
|
||
|
Andrey VasnetsovandClaude Opus 4.8
|
f70a1462fb |
feat: read-only edge shard follower (ReadOnlyEdgeShard) (#9529)
* feat: read-only edge shard follower (ReadOnlyEdgeShard) Add a read-only follower of EdgeShard, mirroring the segment-level ReadOnlySegment + live_reload design one level up at the shard. A leader process owns a read-write EdgeShard; one or more followers open the same on-disk directory as a ReadOnlyEdgeShard, serve reads, and periodically refresh() to pick up the leader's flushed writes and optimizations. Shared read logic lives once in a crate-internal EdgeReadView<H>, generic over a ReadSegmentHandle: the follower is monomorphized over the concrete ReadOnlySegment<S> (no dynamic dispatch), while the read-write shard's heterogeneous Segment/ProxySegment holder uses the LockedSegment enum (the only dyn ReadSegmentEntry left, mirroring LockedSegment::get_read). EdgeShardRead is the public read API: it exposes the read methods (search/query/retrieve/count/facet/scroll/info/...) as default methods, requiring implementors only to provide read_segments() + config_snapshot(). EdgeShard keeps its inherent read methods for backwards compatibility. Segment discovery is injected via a SegmentEnumerator (temporary seam): LocalSegmentEnumerator scans the segments/ directory for the local/mmap case; S3 followers supply their own. This will be replaced by an on-disk segment manifest in follow-up work. shard: add LockedSegment::get_read_arc(); split retrieve_blocking into a handle-collecting wrapper + generic retrieve_over; generalize the private _read_points over the segment type (public signatures unchanged); remove the now-unused read_points_locked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor * feat: manifest-based segment loading for ReadOnlyEdgeShard Replace the read-only follower's directory-scan discovery with the segment manifest from the leader, the proper mechanism the SegmentEnumerator seam was a placeholder for. shard: add SegmentsManifest::load so readers can read manifest.json. edge (leader): EdgeShard now writes segments/manifest.json (gated by the write_segment_manifest feature flag), initialized from the live segment set on new/load and refreshed after optimize() swaps segments. edge (follower): ManifestSegmentEnumerator reads the manifest's `active` segments instead of scanning, and open_mmap uses it. It requires a manifest and errors when none is present (no silent scan fallback); discover by scanning explicitly via open() with a LocalSegmentEnumerator instead. Since the manifest only reports ready segments (and future versions will mark segments retiring/under-construction rather than removing them from active immediately), the read path no longer races with the leader, so the defensive `is_transient_open_error` skip handling is removed — a failure to open a reported segment is now a genuine error and propagates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor * feat: edge-s3-scroll experiment binary for ReadOnlyEdgeShard over S3 Adds a standalone `edge-s3-scroll` binary that opens a ReadOnlyEdgeShard directly over an S3 (or S3-compatible) bucket and runs a single scroll request with a hard-coded filter, for experimentation. Takes the bucket endpoint, credentials and key prefix as CLI flags/env vars, builds an S3-backed BlobFs, and discovers segments via a custom enumerator that reads the segment manifest over object storage. The edge_config.json is fetched to a local temp dir because ReadOnlyEdgeShard::open reads config from the local filesystem. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * filter config in s3 scroll tool * fix: read segment manifest from new location after dev rebase dev #9564 moved the segment manifest next to (rather than inside) the segments/ directory and named it segments_manifest.json. Adapt the read-only follower enumerators accordingly so the follower reads the manifest where the leader now writes it: - ManifestSegmentEnumerator reads via segment_manifest_path() (shard root) while still resolving segment dirs under segments/<uuid>. - The S3 scroll tool's enumerator mirrors the same layout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: read-only edge shard works over object storage without edge_config Make ReadOnlyEdgeShard self-sufficient over arbitrary backends (S3/GCS), and harden the read-only segment loading it depends on. - ReadOnlyEdgeShard derives its config from the segments via EdgeConfig::from_segment_config instead of requiring edge_config.json (open and refresh both derive); a follower never has that file. - ReadOnlyEdgeShard::open(fs, path) no longer takes an enumerator: a read-only follower always discovers segments through the manifest. open_with_enumerator remains as a pub(crate) seam for tests. - ManifestSegmentEnumerator is generic over UniversalReadFs (reads the manifest via read_json_via), so it works over local mmap or a blob/S3 backend; the tool's bespoke enumerator is removed. - Read-only mutable ID tracker tolerates absent mappings/versions files (they are not written while empty), matching MutableIdTracker::open: files are opened lazily and NotFound is treated as empty, avoiding an extra exists() round-trip on object storage. edge-s3-scroll experiment tool: - Supports AWS S3 / S3-compatible and GCS backends (--backend), with a hard-coded-free --filter-key/--filter-value scroll filter. - Reads segment data through a DiskCache so remote blocks are fetched once and served from a local mirror afterwards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: remove unused SegmentsManifest::load Its only caller (edge's ManifestSegmentEnumerator) now reads the manifest via read_json_via over UniversalReadFs, so the local-only load helper is dead. Drop it and its now-unused read_json/Path imports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
xzfc
|
d8383861b2 |
Test edge examples on CI (#8318)
* lib/edge/publish/cargo: improve target directory discovery
Don't hardcode workspace root, ask cargo instead.
* Fix warning in qdrant-edge amalgamation
`cargo check` in `lib/edge/publish` produces the following warning:
warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending`
* lib/edge/publish/examples: fix compiler warnings
* refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs
To match python examples that have the same constant in `common.py`.
* Use `lib/edge/data` dir for both Python and Rust examples
Before this commit, `lib/edge/publish/data` and `lib/edge/python/data`
were used. I'd like Python and Rust to use the same prepared resources
in the future.
* ❗Add lib/edge/Justfile with recipes to build/run examples
I constantly keep forgetting how to compile and run edge stuff.
Also, this would be used in CI in subsequent commits.
* ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes
Also now we can rename `creates-io-readme.md` to `README.md` like
reasonable people.
* ❗doc: Add qdrant-edge-py README.md
Since the previous commit removed qdrant-edge-py README.md, we need to
add something in its place instead.
The new README is adapted from Rust edge README.
And the previous README wasn't good anyway, see [1], it mentions maturin
which is irrelevant for users.
[1]: https://pypi.org/project/qdrant-edge-py/0.5.0/
* GHA: Add "Setup Qdrant" composite action
We need running Qdrant instance to test edge examples on CI. We can't
use docker for this because we want to run tests on Windows and macOS.
OTOH, this action downloads binaries for all platforms provided in
https://github.com/qdrant/qdrant/releases/tag/v1.17.0
* ❗GHA: Use single worflow for python and rust edge examples
Merge workflows to reuse build cache. Also, modernize it by using `uv`,
and recently introduced `setup-qdrant` and Justfile. Also, make it run
on three platforms (only on merges).
* ❗GHA: edge-{py,rust}-release: also run examples before publishing
Use similar approach as in the previous commit. But note that this time
we build/test also on ARM Linux and Windows.
|