Expand the regression suite to the full {optimizer on/off} x {restarts
on/off} matrix via a shared `smoke` helper, and retain storage on panic
(with seed) through a `StorageGuard` for postmortem repro. Op counts
unified to 8000; the optimizer+restarts cell runs at OP_NUM/4 to avoid
being the suite long pole. Module skipped on Windows (too slow).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* model_testing: add QueryFusion op (prefetch + fusion coverage)
The model tester's Query op only issued a top-level Nearest query, leaving
the Query API's prefetch + fusion path untested. QueryFusion issues 1-3
independent Nearest prefetch sources fused with RRF or DBSF, plus an
optional outer num filter.
Fusion ranking is score-based and approximate, so the oracle is upper-bound
only (same convention as Search/Query): every result must be some prefetch's
candidate and pass the outer filter, and the result size can't exceed the
outer-filtered candidate union capped at limit. Scores and order are not
checked. Single prefetch level (no nesting).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* model_testing: address review on QueryFusion
- assert no duplicate ids before collapsing fusion results into a set, so
an engine that fails to dedup overlapping prefetch hits is caught
- log full prefetch descriptors (vector_name, limit, filter_num) in the
trace instead of just vector names
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(edge): load ReadOnlyEdgeShard segments in parallel
Open each segment on a dedicated thread during initial open and refresh,
reducing follower startup time for shards with many segments.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(edge): resolve clippy type_complexity in parallel segment load
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(edge): run per-segment reads on a configurable thread pool
Replace the per-segment sequential read loops and the spawn-a-thread-per-segment
loader with a single fixed-size rayon thread pool owned by each shard.
- Add EdgeConfig::max_search_threads (Option<usize>, None = CPU-derived default
matching the core search runtime via common::defaults::search_thread_count).
- Build a long-lived pool in EdgeShard and ReadOnlyEdgeShard; reuse it for
parallel segment loading on open/refresh instead of std::thread::spawn.
- Add EdgeReadView::par_map_segments as the single seam that runs per-segment
work on the pool; use it in search, scroll, count, facet and rescore-formula.
Each task mints its own HardwareCounterCell from the shared accumulator.
- Expose max_search_threads through the builder and the Python binding (+ stub).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(edge): return error instead of panicking on search pool creation
ThreadPoolBuilder::build() can fail (e.g. thread spawn / resource exhaustion).
This runs during EdgeShard open/load and ReadOnlyEdgeShard follower open, so a
transient failure must not abort the process. Propagate it as an OperationError
through the existing OperationResult-returning constructors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [AI] fix: bound random-sample pre-allocation by available point count
`LocalShard::scroll_randomly` pre-allocated its result set with
`HashSet::with_capacity(limit)`, where `limit` is the client-supplied request
limit and is unbounded by default (`StrictModeConfig::max_query_limit` is `None`
unless an operator sets it). A very large limit made the up-front allocation
fail and abort the process (`handle_alloc_error`), which cannot be caught and
returned as an error.
The sample loop inserts at most `min(limit, total points available across
segments)` points, so bound the pre-allocation to what is actually available.
Capacity is unchanged for normal requests; only the pathological case is bounded.
Same class as the search top-k fix (#4321 / #4328) and the in-flight edge fix
(#9374). Adds a regression test.
* [AI] fix: cap random-sample preallocation by filtered candidate count
Review feedback. `available_point_count_without_deferred()` is filter-unaware,
so `availability.iter().sum()` could still drive `HashSet::with_capacity()` to
reserve a buffer proportional to the full collection on a highly selective
filtered request, re-opening the allocator-abort path on large collections.
The sampling loop only ever inserts points read into `segments_reads`, which is
filter-aware and already capped at `limit` per segment, so bound the
preallocation by that candidate count instead.
* Update lib/collection/src/shards/local_shard/scroll.rs
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
---------
Co-authored-by: Andrey Vasnetsov <vasnetsov93@gmail.com>
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
* ci(windows): skip slow facet sampling filter iter test
The 200k-point fixture is too slow for Windows CI runners.
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci(windows): gate N_FILTER_ITER behind same cfg as test
Avoid dead_code warning when the sampling filter iter test is skipped.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: add openapi regression test for values_count on missing fields
Reproduces https://github.com/qdrant/qdrant/issues/9586 where points with
a missing payload field are not matched by values_count filters that should
treat the count as 0.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test: use module fixture for values_count missing field test
Follow the standard openapi test structure with setup/teardown fixture
instead of explicit drop_collection calls in the test body.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: treat missing field as count 0 for values_count filter (#9607)
`FieldCondition::check_empty()` ignored the `values_count` condition, so a
point with a missing payload field was never matched by a `values_count`
filter. The desired semantics (and the existing `ValuesCount::check_empty`)
treat a missing field as having a value count of 0.
Forward the empty check to `values_count.check_empty()` so bounds like
`lt: 1`, `gte: 0` and `lte: 0` correctly match points whose field is absent.
Fixes#9586
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After `--package qdrant`, cargo scopes `--bin` to the qdrant crate only.
Pass `--package edge-shard-query --bin edge-shard-query` so the edge tool
is actually produced for collection/upload.
Co-authored-by: Cursor <cursoragent@cursor.com>
Without an explicit qdrant package, cargo applies --features service_debug
to every -p target including edge-shard-query, which does not define it.
Co-authored-by: Cursor <cursoragent@cursor.com>
edge-shard-query does not define service_debug; use qdrant/service_debug
when building qdrant bins and edge tools in one cargo invocation.
Co-authored-by: Cursor <cursoragent@cursor.com>
Use the perf profile instead of release, merge qdrant and edge tool builds
into one cargo invocation, and add sccache so shared deps compile once and
rustc outputs are cached across runs.
Co-authored-by: Cursor <cursoragent@cursor.com>
Turn the single-purpose `edge-s3-scroll` tool into a sub-command-based
`edge-shard-query` that can run different read requests against a
ReadOnlyEdgeShard opened over object storage.
- Add `scroll` and `search` sub-commands; shared connection/cache args
live at the top level.
- Accept arbitrary payload filters as JSON via `--filter` (curl
`--data` style: literal JSON, `@file`, or `@-` for stdin), parsed
straight into the filter DSL. Keep `--filter-key`/`--filter-value`
as a shortcut.
- `search` accepts a query vector (JSON array, comma list, or
`@file`/`@-`), named vector, offset, score threshold, and HNSW
params.
- Rename the package/binary/dir (`s3_scroll` -> `shard_query`) since
the tool is no longer scroll-only nor S3-only.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Debug requirement on UserData (#9588) left two impl blocks without
the trait bound, breaking compilation on dev.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add a `Debug` supertrait bound to the `UserData` marker trait and propagate
it through the universal-io read pipelines and the read APIs built on them.
This lets pipeline user-data (request ids, point ids, internal read metadata)
be formatted for diagnostics. Wrapper types that carry user data through the
pipelines (`RemoteMeta`, gridstore `ReadMeta`, hashmap `Entry`) gain `Debug`,
and the `U: UserData` bound is threaded through `read_vectors`/`read_payloads`/
`read_values`/`iter_vectors` and their implementors in gridstore and segment.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf: read whole objects in a single GET instead of HEAD+GET
* feat: read object tail from offset to EOF in a single GET
Generalize the whole-object single-GET read so a tail read from a known
offset also avoids the separate len()/HEAD round-trip. The backend
primitive becomes `read_from(path, from)`, issuing an open-ended
`GetRange::Offset(from)` GET (or a plain GET for from == 0) and reporting
the object's total size from the response; `read_whole` is now a thin
wrapper over it.
The owned blob pipeline's `schedule_whole` no longer HEADs the remote to
size a tail read. An offset at/past EOF is an unsatisfiable range (HTTP
416) rather than an empty body, so the buffer builder disambiguates with
a single len() only on the error path, yielding an empty read when the
tail is genuinely empty. The disk cache's reopen prefiller is made
tolerant of that empty tail so it no longer truncates the local mirror.
Also split the now-hard-to-follow simple_disk_cache `file.rs` into a
`file/` module (type/state, init state machine, reopen, read surface)
and document the FromScratch vs Prefiller init sources.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply the existing "model testing" label alongside "bug" when the nightly
workflow opens or updates a failure issue.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add NumericIndexValue
* MapConditionChecker: get rid of `F: Fn() -> bool` bound
To put this type into the upcoming `ConditionCheckerEnum`, it should be
nameable.
* filter_context: `Box<dyn ConditionChecker>` -> `OptimizedFilter`
Removes one level of dyn indirection, so faster checks.
* NullConditionChecker: merge IsEmpty/IsNull checkers into one
So, less variants in the upcoming `ConditionCheckerEnum`.
DeleteByFilter remains broken: WAL replay can resurrect filter-deleted
points (#9575). Re-mask the op in FORCE_OFF and document the issue.
Co-authored-by: Cursor <cursoragent@cursor.com>
The flushing/reload-durability bug that caused post-reload count
mismatches for live filtered deletes has been fixed, so DeleteByFilter
no longer needs to be masked off in the model tester swarm config.
Remove it from FORCE_OFF and update the related doc comments.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add the edge-s3-scroll tool (lib/edge/tools/s3_scroll) to the debug
tools build/upload. It's a standalone workspace crate without the
qdrant `service_debug` feature, so it's built in a separate `cargo build
-p edge-s3-scroll` invocation and then collected/uploaded alongside the
rest.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: add nightly model_testing workflow
Run the model_testing binary nightly on dev with two sequential passes
(async-scorer and without io_uring), reusing a single built binary.
Note: a temporary pull_request trigger is included to test-drive the
workflow on the PR; it is to be removed before merge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: fix checkout ref for pull_request event
github.ref_name resolves to <pr>/merge on pull_request, which checkout
treats as a branch and fails to fetch. Use the default ref outside the
nightly schedule.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: drop mold/clang linker from model_testing workflow
free-disk-space (large-packages) removes the llvm/clang packages, so the
clang linker used for mold no longer exists on the runner. Build time is
dominated by the long model-testing runs, so the default linker is fine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: random seed, constants, both-run, backtrace, dispatch inputs, failure issue
- Random seed per run (logged for repro), shared by both passes
- OP_NUM/SHARD_COUNT/RESTART_PROBABILITY/ID_POOL as env constants
- no-io_uring run uses if: !cancelled() so both passes always run
- RUST_BACKTRACE=1 for debuggable panics
- workflow_dispatch inputs to override op_num/seed
- rust-cache saves on the nightly (schedule) run, not the never-matching dev ref
- open/update a GitHub issue on scheduled failures
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: run model_testing with 2 shards by default
Exercises the cross-shard reload/WAL-replay path (the 2-shard-only revert
class) instead of single-shard only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: bump model_testing OP_NUM to 200000
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: remove temporary pull_request trigger from model_testing workflow
Test-drive validated on the PR; the nightly now runs only on schedule and
workflow_dispatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Build the debug tools for x86_64-unknown-linux-musl so the resulting
binaries are statically linked and run on any Linux host regardless of
distro / libc. Mirrors the musl setup used in release-artifacts.yml
(apt deps + taiki-e/setup-cross-toolchain-action) and drops the
gnu-specific mold linker hack, which the cross toolchain replaces.
Cache is keyed per target so the musl artifacts persist across runs
without colliding with gnu caches.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Recent AWS CLI versions add default CRC32 integrity checksums on
uploads, which GCS's S3-compatible UploadPart rejects with
SignatureDoesNotMatch. This only affected the larger debug binaries that
go through multipart upload. Opt out via AWS_REQUEST_CHECKSUM_CALCULATION
/ AWS_RESPONSE_CHECKSUM_VALIDATION=when_required.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: read-only edge shard follower (ReadOnlyEdgeShard)
Add a read-only follower of EdgeShard, mirroring the segment-level
ReadOnlySegment + live_reload design one level up at the shard. A leader
process owns a read-write EdgeShard; one or more followers open the same
on-disk directory as a ReadOnlyEdgeShard, serve reads, and periodically
refresh() to pick up the leader's flushed writes and optimizations.
Shared read logic lives once in a crate-internal EdgeReadView<H>, generic
over a ReadSegmentHandle: the follower is monomorphized over the concrete
ReadOnlySegment<S> (no dynamic dispatch), while the read-write shard's
heterogeneous Segment/ProxySegment holder uses the LockedSegment enum
(the only dyn ReadSegmentEntry left, mirroring LockedSegment::get_read).
EdgeShardRead is the public read API: it exposes the read methods
(search/query/retrieve/count/facet/scroll/info/...) as default methods,
requiring implementors only to provide read_segments() + config_snapshot().
EdgeShard keeps its inherent read methods for backwards compatibility.
Segment discovery is injected via a SegmentEnumerator (temporary seam):
LocalSegmentEnumerator scans the segments/ directory for the local/mmap
case; S3 followers supply their own. This will be replaced by an on-disk
segment manifest in follow-up work.
shard: add LockedSegment::get_read_arc(); split retrieve_blocking into a
handle-collecting wrapper + generic retrieve_over; generalize the private
_read_points over the segment type (public signatures unchanged); remove
the now-unused read_points_locked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor
* feat: manifest-based segment loading for ReadOnlyEdgeShard
Replace the read-only follower's directory-scan discovery with the segment
manifest from the leader, the proper mechanism the SegmentEnumerator seam
was a placeholder for.
shard: add SegmentsManifest::load so readers can read manifest.json.
edge (leader): EdgeShard now writes segments/manifest.json (gated by the
write_segment_manifest feature flag), initialized from the live segment
set on new/load and refreshed after optimize() swaps segments.
edge (follower): ManifestSegmentEnumerator reads the manifest's `active`
segments instead of scanning, and open_mmap uses it. It requires a manifest
and errors when none is present (no silent scan fallback); discover by
scanning explicitly via open() with a LocalSegmentEnumerator instead.
Since the manifest only reports ready segments (and future versions will
mark segments retiring/under-construction rather than removing them from
active immediately), the read path no longer races with the leader, so the
defensive `is_transient_open_error` skip handling is removed — a failure to
open a reported segment is now a genuine error and propagates.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor
* feat: edge-s3-scroll experiment binary for ReadOnlyEdgeShard over S3
Adds a standalone `edge-s3-scroll` binary that opens a ReadOnlyEdgeShard
directly over an S3 (or S3-compatible) bucket and runs a single scroll
request with a hard-coded filter, for experimentation.
Takes the bucket endpoint, credentials and key prefix as CLI flags/env
vars, builds an S3-backed BlobFs, and discovers segments via a custom
enumerator that reads the segment manifest over object storage. The
edge_config.json is fetched to a local temp dir because
ReadOnlyEdgeShard::open reads config from the local filesystem.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* filter config in s3 scroll tool
* fix: read segment manifest from new location after dev rebase
dev #9564 moved the segment manifest next to (rather than inside) the
segments/ directory and named it segments_manifest.json. Adapt the
read-only follower enumerators accordingly so the follower reads the
manifest where the leader now writes it:
- ManifestSegmentEnumerator reads via segment_manifest_path() (shard root)
while still resolving segment dirs under segments/<uuid>.
- The S3 scroll tool's enumerator mirrors the same layout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: read-only edge shard works over object storage without edge_config
Make ReadOnlyEdgeShard self-sufficient over arbitrary backends (S3/GCS),
and harden the read-only segment loading it depends on.
- ReadOnlyEdgeShard derives its config from the segments via
EdgeConfig::from_segment_config instead of requiring edge_config.json
(open and refresh both derive); a follower never has that file.
- ReadOnlyEdgeShard::open(fs, path) no longer takes an enumerator: a
read-only follower always discovers segments through the manifest.
open_with_enumerator remains as a pub(crate) seam for tests.
- ManifestSegmentEnumerator is generic over UniversalReadFs (reads the
manifest via read_json_via), so it works over local mmap or a blob/S3
backend; the tool's bespoke enumerator is removed.
- Read-only mutable ID tracker tolerates absent mappings/versions files
(they are not written while empty), matching MutableIdTracker::open:
files are opened lazily and NotFound is treated as empty, avoiding an
extra exists() round-trip on object storage.
edge-s3-scroll experiment tool:
- Supports AWS S3 / S3-compatible and GCS backends (--backend), with a
hard-coded-free --filter-key/--filter-value scroll filter.
- Reads segment data through a DiskCache so remote blocks are fetched
once and served from a local mirror afterwards.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: remove unused SegmentsManifest::load
Its only caller (edge's ManifestSegmentEnumerator) now reads the manifest
via read_json_via over UniversalReadFs, so the local-only load helper is
dead. Drop it and its now-unused read_json/Path imports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-add the push-to-dev trigger so debug tools are rebuilt automatically
on every merge to dev, while keeping manual dispatch with a branch
input. The branch used for checkout and the object prefix falls back to
the pushed branch when no input is given.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: build debug tools and publish to GHCR via ORAS
Adds a `Build debug tools` workflow that compiles the `service_debug`
helper binaries (wal_inspector, wal_pop, segment_inspector,
schema_generator, model_testing) and pushes them to GHCR as an OCI
artifact using ORAS.
These are debug-only tools built between releases and are intentionally
kept out of the release artifacts. Publishing them as an OCI artifact
gives a versioned, browsable, registry-backed location alongside our
Docker images instead of burying them in per-run CI artifacts.
Pull with:
oras pull ghcr.io/qdrant/qdrant/debug-tools:dev
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: upload debug tools to S3 instead of GHCR
Switch the debug-tools workflow from publishing an OCI artifact via ORAS
to uploading the `service_debug` binaries to S3 with public-read, giving
plain HTTPS download links and no client tooling beyond curl.
Binaries are uploaded under a moving `<branch>/` prefix (latest) and an
immutable `<branch>-<sha>/` prefix (history); direct links are written
to the job summary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: target GCS (S3-compatible) instead of AWS S3
Point the debug-tools upload at GCS via its S3-compatible XML API using
AWS-style HMAC interoperability keys. Drops per-object ACLs (GCS buckets
use uniform access; public read is granted via IAM) and uses GCS
path-style public URLs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: set debug-tools bucket to qdrant-debug
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci: run debug-tools manually with a branch input
Drop the push trigger so the workflow only runs on explicit dispatch
from the UI, and add a `branch` input selecting which branch to build.
The chosen branch (and its actual HEAD commit) form the object prefix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* hold init_guard during reopen
* add `from` parameter to `OwnedReadPipeline::schedule_whole`
* add `OwnedReadPipeline::into_inner` to extract inner file
* implement proper background populate on `reopen`
* clippy
* coderabbit
* explicit owned uring pipeline destructuring
* refactor to join local and remote into one state
* @ffuugoo's nits
Follow-up to #9530 and #9558.
The shard-level segment manifest was written inside the `segments/`
directory (`segments/manifest.json`). Older versions of Qdrant choke on
an unknown file inside `segments/`, so move it next to the directory as
`segments_manifest.json` instead.
- `SEGMENT_MANIFEST_FILE` is now `segments_manifest.json` and
`segment_manifest_path()` points at the shard root.
- The manifest is added to `ShardDataFiles` so clear/move handle it.
- Snapshots write the manifest to the snapshot root (next to
`segments/`), and restore/partial-snapshot loaders no longer need to
skip it inside `segments/`.
Co-authored-by: Cursor <cursoragent@cursor.com>
Follow-up to #9530. When the `write_segment_manifest` feature flag is
enabled, a shard maintains a `segments/manifest.json` listing its
segments so out-of-process readers can discover them without scanning
the filesystem. That manifest was not included in shard snapshots.
Include the segment manifest in the snapshot when the shard maintains
one, capturing it from the live segment holder before proxying (proxies
preserve the wrapped segments' UUIDs, which are the directories written
into the snapshot, so the manifest matches the snapshot contents).
On restore, the snapshot's `segments/manifest.json` is skipped while
restoring segment directories in place; the manifest is regenerated from
the loaded segments when the segment holder is built. The partial
snapshot manifest loader also skips this file.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(optimizer): defer source segment destruction until durable
Optimizations copy-on-write move points out of their source segments in
memory, and WAL replay can only re-derive those moves from the sources'
on-disk pre-images. Dropping the sources immediately in finish_optimization
is unsafe: the moved copies may still sit unflushed in appendable segments,
so a restart before they are persisted loses the points ("No point with id"
during replay).
Replace the immediate proxy.drop_data() with a generic post-flush action
mechanism on SegmentHolder:
- register_post_flush_action(ready_at, ack_pin, action) queues a retryable
closure (FnMut returning PostFlushOutcome) to run once a flush proves its
data durable.
- flush_all runs every action whose ready_at is covered by the durable
waterline, and caps the returned version (and thus the WAL acknowledge) at
the minimum ack_pin of the actions still pending, so every operation the
not-yet-cleaned data contradicts (deletions in particular) stays replayable
until the files are gone.
- finish_optimization registers each source's drop via register_segment_drop,
pinned at the source's persisted version.
A crash before an action runs loads the old files next to their replacement;
load-time deduplication resolves the overlap.
Robustness:
- LockedSegment::try_drop_data hands the segment back on a StillInUse failure
(data untouched) with a short timeout, so a failed drop is retried on a
later flush instead of leaking its ack pin or blocking the flush for up to
an hour; drop_data keeps its long timeout for callers without a retry path.
- run_ready_post_flush_actions records the ack-pin floor of the actions it is
running (briefly out of the queue) so a concurrent flush on the background
early-return path cannot advance the WAL acknowledge past them.
Optimizer tests that assert source files are gone now flush_all first to run
the deferred action before counting dirs / asserting. New SegmentHolder unit
tests cover the retry, hard-failure, and in-flight-pin-visibility paths.
The perf caveat (a fresh appendable segment reporting persistent_version()
== 0 drags the waterline down) is documented inline as a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* capture knowledge
* Update lib/shard/src/segment_holder/mod.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
