mirror of
https://github.com/qdrant/qdrant.git
synced 2026-07-23 11:11:00 -05:00
f70a1462fb
* feat: read-only edge shard follower (ReadOnlyEdgeShard) Add a read-only follower of EdgeShard, mirroring the segment-level ReadOnlySegment + live_reload design one level up at the shard. A leader process owns a read-write EdgeShard; one or more followers open the same on-disk directory as a ReadOnlyEdgeShard, serve reads, and periodically refresh() to pick up the leader's flushed writes and optimizations. Shared read logic lives once in a crate-internal EdgeReadView<H>, generic over a ReadSegmentHandle: the follower is monomorphized over the concrete ReadOnlySegment<S> (no dynamic dispatch), while the read-write shard's heterogeneous Segment/ProxySegment holder uses the LockedSegment enum (the only dyn ReadSegmentEntry left, mirroring LockedSegment::get_read). EdgeShardRead is the public read API: it exposes the read methods (search/query/retrieve/count/facet/scroll/info/...) as default methods, requiring implementors only to provide read_segments() + config_snapshot(). EdgeShard keeps its inherent read methods for backwards compatibility. Segment discovery is injected via a SegmentEnumerator (temporary seam): LocalSegmentEnumerator scans the segments/ directory for the local/mmap case; S3 followers supply their own. This will be replaced by an on-disk segment manifest in follow-up work. shard: add LockedSegment::get_read_arc(); split retrieve_blocking into a handle-collecting wrapper + generic retrieve_over; generalize the private _read_points over the segment type (public signatures unchanged); remove the now-unused read_points_locked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor * feat: manifest-based segment loading for ReadOnlyEdgeShard Replace the read-only follower's directory-scan discovery with the segment manifest from the leader, the proper mechanism the SegmentEnumerator seam was a placeholder for. shard: add SegmentsManifest::load so readers can read manifest.json. edge (leader): EdgeShard now writes segments/manifest.json (gated by the write_segment_manifest feature flag), initialized from the live segment set on new/load and refreshed after optimize() swaps segments. edge (follower): ManifestSegmentEnumerator reads the manifest's `active` segments instead of scanning, and open_mmap uses it. It requires a manifest and errors when none is present (no silent scan fallback); discover by scanning explicitly via open() with a LocalSegmentEnumerator instead. Since the manifest only reports ready segments (and future versions will mark segments retiring/under-construction rather than removing them from active immediately), the read path no longer races with the leader, so the defensive `is_transient_open_error` skip handling is removed — a failure to open a reported segment is now a genuine error and propagates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor * feat: edge-s3-scroll experiment binary for ReadOnlyEdgeShard over S3 Adds a standalone `edge-s3-scroll` binary that opens a ReadOnlyEdgeShard directly over an S3 (or S3-compatible) bucket and runs a single scroll request with a hard-coded filter, for experimentation. Takes the bucket endpoint, credentials and key prefix as CLI flags/env vars, builds an S3-backed BlobFs, and discovers segments via a custom enumerator that reads the segment manifest over object storage. The edge_config.json is fetched to a local temp dir because ReadOnlyEdgeShard::open reads config from the local filesystem. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * filter config in s3 scroll tool * fix: read segment manifest from new location after dev rebase dev #9564 moved the segment manifest next to (rather than inside) the segments/ directory and named it segments_manifest.json. Adapt the read-only follower enumerators accordingly so the follower reads the manifest where the leader now writes it: - ManifestSegmentEnumerator reads via segment_manifest_path() (shard root) while still resolving segment dirs under segments/<uuid>. - The S3 scroll tool's enumerator mirrors the same layout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: read-only edge shard works over object storage without edge_config Make ReadOnlyEdgeShard self-sufficient over arbitrary backends (S3/GCS), and harden the read-only segment loading it depends on. - ReadOnlyEdgeShard derives its config from the segments via EdgeConfig::from_segment_config instead of requiring edge_config.json (open and refresh both derive); a follower never has that file. - ReadOnlyEdgeShard::open(fs, path) no longer takes an enumerator: a read-only follower always discovers segments through the manifest. open_with_enumerator remains as a pub(crate) seam for tests. - ManifestSegmentEnumerator is generic over UniversalReadFs (reads the manifest via read_json_via), so it works over local mmap or a blob/S3 backend; the tool's bespoke enumerator is removed. - Read-only mutable ID tracker tolerates absent mappings/versions files (they are not written while empty), matching MutableIdTracker::open: files are opened lazily and NotFound is treated as empty, avoiding an extra exists() round-trip on object storage. edge-s3-scroll experiment tool: - Supports AWS S3 / S3-compatible and GCS backends (--backend), with a hard-coded-free --filter-key/--filter-value scroll filter. - Reads segment data through a DiskCache so remote blocks are fetched once and served from a local mirror afterwards. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: remove unused SegmentsManifest::load Its only caller (edge's ManifestSegmentEnumerator) now reads the manifest via read_json_via over UniversalReadFs, so the local-only load helper is dead. Drop it and its now-unused read_json/Path imports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
29 lines
450 B
Plaintext
29 lines
450 B
Plaintext
/target
|
|
.idea
|
|
.vscode
|
|
/.devcontainer
|
|
/storage
|
|
/storage-model
|
|
/snapshots
|
|
/consensus_test_logs
|
|
/tests/consensus_test_logs
|
|
/config/local
|
|
/tls
|
|
/static
|
|
*.log
|
|
.qdrant-initialized
|
|
/.hypothesis
|
|
**/__pycache__
|
|
tests/storage-compat/compatibility.tar
|
|
tests/storage-compat/full-snapshot.snapshot.gz
|
|
tests/storage-compat/storage.tar.bz2
|
|
venv
|
|
.venv
|
|
.env
|
|
.python-version
|
|
.claude/
|
|
/docs/plans/
|
|
|
|
# Generated by lib/edge/python examples
|
|
lib/edge/python/examples/data/
|