* HNSWIndex::build(): add `inline_vectors` arg
Let the caller decide whether to use `inline_vectors` format.
* SegmentBuilder::build: finalize GraphInline vector storage
Instead of old "graph-with-vectors plus regular vector storage",
keep only the graph-with-vectors storage.
* Gate the GraphInline segment build behind a feature flag
* Rename `wal_keep_from` to `wal_ack_pin`
The parameter pins the WAL acknowledge, the new name says so.
* Support pinning the WAL acknowledge from multiple places at once
The WAL acknowledge had a single pin slot, a shared `AtomicU64` any second
user would have clobbered. Replace it with `WalAckPins`, holding any number
of live pins. The flush worker never acknowledges at or past the lowest of
them.
Taking a pin hands out a `WalAckPinGuard` that releases when dropped, so the
queue proxy shard no longer has to release it by hand. The registry holds its
pins weakly, which means dropping the guard is all it takes and a queue proxy
lost to an unwind can no longer stall the WAL acknowledge forever.
* Tests for the WAL acknowledge pins
* Debug assert that set does not move version backwards
* Update tests
universal_map_index::UniversalMapIndex does not exist; the on-disk
variant is on_disk_map_index::OnDiskMapIndex (the third MapIndex
variant). Repointed the three rustdoc links accordingly.
`PostingList::upsert` ends in `propagate_max_next_weight_to_the_left`, whose doc
comment states "If an entry has a weight larger than `max_next_weight`, the
propagation stops". It never stopped — the loop always walked the entire prefix.
Record ids ascend during an upload, so every insert lands at the end, walks the
whole list, and growing a posting list to length L costs O(L²). On the NeurIPS
2023 sparse base set (MS MARCO / SPLADE) the hottest dimension appears in ~66% of
documents, so at 1M points its posting list holds 660k elements and every new
point rewrites all of them.
Two changes to `PostingList`:
* restore the early exit — every entry satisfies the same recurrence the loop
walks, `max_next_weight[i] = max(max_next_weight[i + 1], weight[i + 1])`, so
once an entry already holds the value being written, every entry to its left is
correct too;
* add an append fast path — when the incoming record id is past the last stored
id, push directly instead of binary searching a list that can hold millions of
elements.
Neither changes what is stored. An index grown by `upsert` stays identical to one
built by `InvertedIndexBuilder`, `max_next_weight` included, which is what a
segment reload depends on; searches return identical results.
Building an `InvertedIndexRam` one point at a time, SPLADE vectors:
points before after
100k 2,216/s 124,016/s
1M 250/s 114,422/s
Uploading 1M of those points to a single node: 72.2s -> 23.7s with default
collection settings, and 839.8s -> 29.5s with `indexing_threshold: 0`, which
stops segments converting and is the documented way to speed up a bulk load.
Claude-Session: https://claude.ai/code/session_01HXjykBsMZaRNrP17o2fuEJ
Co-authored-by: Andrey Vasnetsov <andrey@qdrant.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-commit `<branch>-<sha>` copies accumulate a full set of binaries
for every dev push and are never fetched.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: expose ReshardingStage in internal telemetry
Useful for cluster-manager operations.
Non-breaking change for existing API.
The current field values look stable enough,
so it's not worrisome to keep them stable.
Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com>
* fix: address comment, also document uuid for symmetry
Already returned, just document it in OpenAPI schema.
Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com>
---------
Signed-off-by: Anton Antonov <anton.synd.antonov@gmail.com>
* shard: persist proxy pending changes on flush, stop holding back WAL ack
Hook the pending changes component into the proxy segment's flush: the
proxy flusher first persists the buffered operations into the pending
changes log, then passes the flush along to the wrapped segment. The
proxy's `persistent_version` now covers what the log durably holds on
top of what the wrapped segment persisted itself.
That is what lifts the WAL cap proxies imposed so far. `flush_all`
compares each segment's version against its persistent version; a
proxy used to report only the wrapped segment's persisted version while
its own version climbed with every buffered operation, so the WAL could
never be acknowledged past the point the proxy was created at, and a
restart replayed all of it — potentially very expensive operations,
such as an update by filter, all over again. With the buffered state
durable on disk the generic rule acknowledges the full version, and a
restart recovers it from the log instead.
Dropping a proxy's data drops the component first, which waits for any
in-flight pending changes flusher so it cannot append to the segment
directory while that is being deleted.
Update the proxy flush test to the new semantics, add a segment holder
test asserting the acknowledged version advances past a proxied delete,
and update the ack pin rationale in `finish_optimization`: the pin is
still needed after the proxies leave the holder, it just snapshots a
persistent version that now includes the log.
* Persist wrapped segment before pending changes
Prevents raising version of proxy segment too early
* collection: test crash recovery through persisted proxy changes
End-to-end test of the persisted pending changes: wrap every segment of
a local shard in a proxy, delete points so the deletes are only
buffered, flush, and assert the acknowledgeable version covers them.
Then acknowledge the WAL up to that version, drop the shard without
ever propagating the proxies, and load it again: the deletes are gone
from the WAL and must come back through the pending changes logs.
The delete under test is deliberately not the last WAL entry, as the
acknowledge never passes the last entry and that one is always
replayed.
* test: cover untested pending proxy changes paths
* test: pin persisted proxy changes findings
* Fix bad merge
* Remove corrupt length test, we cannot detect if last entry was corrupt
Remove a test that asserts an entry that isn't the last cannot have a
corrupt length. We cannot reliably detect whether the invalid length was
the last entry or not, because nothing else tells us how many entries we
expect in the file. At the same time we don't expect random bit flips.
So I removed the test.
* Simulate segments flush to clear pending changes log file
* Update test, also assert proxy segment version
* Fix bad merge
* Fix blocked test
* Enable necessary feature flags in tests (2/2)
---------
Co-authored-by: timvisee <tim@visee.me>
* segment: move proxy pending change types into segment crate
Move the types describing the changes a proxy segment buffers — point
deletes (`ProxyDeletedPoint`), payload index changes (`ProxyIndexChange`,
`ProxyIndexChanges`) and vector name changes (`IntendedVector`,
`ProxyVectorNameChanges`) — from `shard::proxy_segment` into a new
`segment::pending_changes` module.
Pure move, no behavior change: the proxy segment re-exports them from
their old location. Having them in the segment crate lets both the proxy
segment and the segment load path share them, in preparation for
persisting pending proxy changes to disk and replaying them on restart.
* segment: add PendingChange describing a persisted proxy operation
Add the `PendingChange` enum with one variant per operation type a proxy
segment buffers — point delete, payload index change, vector name
change — each carrying the operation version it was issued with. This
is the shape in which pending proxy changes are persisted to disk.
Derive serde on it and on the buffered change types it embeds, so
entries can be serialized into a log file and read back. `PartialEq` on
those types lets a persisted batch be matched against the in-memory
pending buffer after a flush.
* segment: add PendingChanges component persisting proxy changes to a log
Add `PendingChanges`, the component that manages the operations a proxy
segment buffers for one proxy layer, and persists them to disk so they
no longer only live in memory.
It keeps the same per-type buffers the proxy segment served its reads
from (point deletes, payload index changes, vector name changes), plus a
single registration-ordered buffer of everything not yet persisted.
`flusher()` writes that buffer into an append-only log file inside the
wrapped segment's directory: `pending_changes.log` for the inner most
proxy layer, with the layer number as a suffix for each layer above it.
Appends follow the mutable ID tracker: all new entries are serialized
into one buffer and written with a single call on an append-mode file,
then fsynced, so a crash can only leave a torn entry at the very end.
Loading truncates such an entry — its operations were never durable and
thus never acknowledged in the WAL — but fails hard on a malformed entry
in the middle, which cannot be explained by a torn append.
The component tracks the highest operation version the log covers.
Every registered operation at or below it is either durable in the log
or was a no-op that does not need recovery; a flusher advances it to the
proxy's version even when there is nothing to write. The pending buffer
is deliberately not cleared when the proxy propagates its changes to
the wrapped segment, as that only makes them durable once the wrapped
segment flushes. Replaying an entry twice is a version-gated no-op.
A log file left behind by a previous proxy on the same segment is
adopted by `open()`: new entries are appended after it and its highest
version is taken over, while its entries are not loaded into the
buffers as they are already applied to the segment. `load()` also
reconstructs the buffers, for callers that do want the buffered state.
* segment: replay persisted pending proxy changes onto a segment on load
Add `recover_pending_changes`, to be called when a segment is loaded on
restart, before regular WAL replay. If the segment directory holds
pending changes log files, the proxies that wrote them did not
propagate their buffered state into the segment before the process
stopped. Instead of reconstructing the proxies, replay all logged
operations directly onto the segment: inner most proxy layer first,
each file in append order, through the regular version-gated segment
operations (`apply_change`). Entries the segment already applied are
silently skipped, so a stale file is harmless.
The segment is force-flushed before the files are removed; a crash in
between merely replays the files once more.
* segment: test PendingChanges component
Cover the pending changes component: registering and flushing each
operation type and reconstructing the buffers from the log, log file
naming per proxy layer and gap-tolerant listing, covering the proxy
version without entries, operations registered while a flusher is
captured, flushers of a dropped component, torn-tail truncation versus
mid-file corruption, adoption of an existing log, and replaying logs
onto a real segment: fresh, stale (already applied), multi-layer, and
vector name changes.
* segment: include pending changes logs in segment snapshots
Register the pending changes log files of a segment in its snapshot:
add them to `snapshot_files` next to the segment state and version
files, existence-guarded, and to the segment manifest as unversioned
files. Full, partial and streamed snapshots therefore all carry them.
The recovery side needs no changes: a restored segment is loaded like
any other, which replays and removes the logs.
* shard: back proxy segment pending changes by PendingChanges component
Replace the proxy segment's separate `deleted_points`, `changed_indexes`
and `changed_vector_names` fields with a single `PendingChanges`
component. Reads keep going through the same per-type buffers, now
behind accessors; writes go through the component's `register_*`
methods, which additionally queue every operation for persistence.
Opening the component is fallible, as it adopts a pending changes log
a previous proxy may have left in the wrapped segment's directory, so
`UnsyncedProxySegment::new` now returns a result. Wrapping another proxy
opens the next proxy layer up, writing to its own dedicated log file.
No behavior change yet: the proxy still flushes and reports persistence
exactly as before, nothing is written to the log.
* shard: persist proxy pending changes on flush, stop holding back WAL ack
Hook the pending changes component into the proxy segment's flush: the
proxy flusher first persists the buffered operations into the pending
changes log, then passes the flush along to the wrapped segment. The
proxy's `persistent_version` now covers what the log durably holds on
top of what the wrapped segment persisted itself.
That is what lifts the WAL cap proxies imposed so far. `flush_all`
compares each segment's version against its persistent version; a
proxy used to report only the wrapped segment's persisted version while
its own version climbed with every buffered operation, so the WAL could
never be acknowledged past the point the proxy was created at, and a
restart replayed all of it — potentially very expensive operations,
such as an update by filter, all over again. With the buffered state
durable on disk the generic rule acknowledges the full version, and a
restart recovers it from the log instead.
Dropping a proxy's data drops the component first, which waits for any
in-flight pending changes flusher so it cannot append to the segment
directory while that is being deleted.
Update the proxy flush test to the new semantics, add a segment holder
test asserting the acknowledged version advances past a proxied delete,
and update the ack pin rationale in `finish_optimization`: the pin is
still needed after the proxies leave the holder, it just snapshots a
persistent version that now includes the log.
* shard: propagate proxy changes when unwrapping on optimizer cancel
When an optimization is cancelled or fails, `unwrap_proxy` puts the
wrapped segments back into the segment holder. Propagate the changes
buffered in each proxy into its wrapped segment first, as the snapshot
unproxy path already does, instead of dropping them with the proxy.
The pending changes log is deliberately left in place when unwrapping:
deleting it before the wrapped segment has flushed the propagated
changes would not be crash safe. It is cleaned up on restart and when
the segment directory is dropped, and a new proxy on the same segment
adopts and appends to it; replaying a stale file is safe because all
operations are version gated.
* shard: test persisted proxy pending changes
Test the proxy segment against its persisted pending changes: buffered
changes survive dropping the proxy without propagation and are replayed
onto the segment when it is loaded again; unwrapping leaves the log in
place and a new proxy on the same segment adopts and appends to it;
layered proxies each persist into their own log file and a restart
replays both; and a persisted log is part of the segment manifest and
snapshot.
* collection, edge: recover persisted proxy changes on segment load
Replay the pending changes logs left behind by proxy segments onto each
segment when a shard loads its segments, right after consistency
repair and before the payload index rebuild, vector name reconciliation
and WAL replay. Proxy state that made it to disk no longer holds back
the WAL acknowledge, so this is where it must be recovered from.
Proxies are not reconstructed: the segment holder starts with plain
segments carrying the replayed operations, and the logs are removed
once the segment flushed them.
* collection: test crash recovery through persisted proxy changes
End-to-end test of the persisted pending changes: wrap every segment of
a local shard in a proxy, delete points so the deletes are only
buffered, flush, and assert the acknowledgeable version covers them.
Then acknowledge the WAL up to that version, drop the shard without
ever propagating the proxies, and load it again: the deletes are gone
from the WAL and must come back through the pending changes logs.
The delete under test is deliberately not the last WAL entry, as the
acknowledge never passes the last entry and that one is always
replayed.
* segment: make replaying persisted proxy changes on load an explicit mode
Add `PersistedProxyChanges` to state whether persisted pending proxy
changes are replayed onto a segment when it is loaded. `Replay`, the
default, recovers them and removes the logs as before. `Ignore` leaves
both the segment and the log files untouched and logs at debug level
that replaying was skipped; it is for segment files that mirror those
of another writer, where replaying would make the local copy diverge
from what the writer's manifest describes.
All callers pass `Replay` for now, no behavior change.
* collection: do not replay persisted proxy changes on partial snapshot recovery
Partial snapshots are recovered by read replicas in a read/write
segregation setup. A read replica must not mutate its segments, so it
cannot replay the persisted proxy segment changes on load and must
ignore them instead: its segment files are a local copy of the writer's
that must stay a faithful mirror of them, as later partial snapshots are
diffed against what the writer's manifest describes. Replaying would
mutate the segment files and remove the logs, making the copy diverge.
Thread the replay mode through `LocalShard::load` as a dedicated
`PersistedProxyChanges` argument, derived from the recovery type:
`RecoveryType::Full` replays as before, `RecoveryType::Partial` ignores
the persisted changes and leaves the logs in place. Regular shard loads
replay.
Extend the crash recovery test with an ignoring load first: the delete
under test must not come back and the logs must survive, before a
replaying load recovers it.
* Persist wrapped segment before pending changes
Prevents raising version of proxy segment too early
* Fix comment
* Fix crash window, only ready optimized segment after propagating changes
The optimizer renamed a newly built segment into segments_path and wrote
its version file before finish_optimization propagated the proxies'
buffered changes into it. A crash in that window left the segment
restart-loadable but stale, permanently losing or resurrecting points.
Defer the version file save until finish_optimization has fully
reconciled proxy changes into the segment, including the post-swap dedup
pass, so it stays invisible to restart and snapshot recovery until then.
SegmentBuilder::build() gains a `ready` flag; load_segment gains
`ignore_missing_version` for the one caller reloading before that point.
Incidentally also closes the crash-unsafe cancellation-orphan cleanup
gap noted in #9217, since a cancelled build is discarded on restart the
same way.
* Force flush optimized segment, otherwise we may lose proxy changes
* Don't force flush after replay, defer deleting log files until flush
* Include persisted proxy changes log file in segment manifest
* Add random ID to proxy log files, prevent instance conflicts
* Rename proxy log file, always include level
* Delete proxy log file on unproxy, defer until next flush cycle
* Fix truncation
* Reformat
* Lock persisted segments behind runtime feature flag
* Enable necessary feature flags in tests
* Fix linters
* Measure and persist document length in the mutable text index
BM25 length normalization needs the total token count per point, which nothing
stored: `point_to_tokens_count` is the distinct count, and its meaning is fixed
by the user-visible `values_count` filter.
- `doc_len` is measured in `add_many`, the only place that still sees every
token, and persisted in the stored record alongside the tokens.
- It is a parameter on `index_str_tokens` and `MutableInvertedIndexBuilder::add`,
never derived. Without phrase matching the stored tokens are sorted and
deduplicated, and the index is rebuilt from those records on every segment
open, so a derived length would degrade to the distinct-term count on restart.
- Array boundary sentinels are discounted by inserted count, not by value:
`tokenize_doc` does not strip that character from user text the way
`tokenize_query` does, so a payload containing it has those tokens indexed
and they must be counted.
- `MutableInvertedIndex` gains `point_to_doc_len` and a running `total_tokens`,
maintained across add, overwrite and remove, so `avgdl` is a division rather
than a scan. `set_doc_len` is the only writer, so an absent length means the
same thing on every path: the slot is zeroed, never left stale.
- Recording is gated behind `TextIndexParams::scoring()`, a private const for
now. Nothing can ask for a ranked query yet, so recording on every text index
would rebuild every collection to produce data no query can reach. State
lives in the data rather than in a flag: `point_to_doc_len` is an `Option`
the way `point_to_doc` already encodes positions, and `add_many` asks the
index whether it records lengths rather than asking the config.
- `StoredDocument::doc_len` is an `Option` skipped on write when absent, so a
non-scoring index writes byte-identical records to today's and a legacy
record reads back as `None`. A document whose tokens were all filtered is a
real `Some(0)`, and stays distinguishable from one that was never measured.
The immutable and on-disk backends drop the value for now and grow their own
sidecar next.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Address text index document length review feedback
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: generall <andrey@vasnetsov.com>
A hung harness_no_restarts run blocked ubuntu CI for ~50m with only SLOW
markers and no failure dump. Kill tests after five slow-timeout periods,
and print stage/op breadcrumbs to stdout so the timeout failure includes
seed, storage path, and the last op/stage that never returned.
* [AI] Add shared disk cache statistics and latency histogram
* [AI] Document disk cache statistics observer identity
* [AI] Simplify disk cache statistics by removing pipeline error counters
* [AI] Limit disk cache statistics to remote fetches and trim redundant tests
* [AI] Close remote append handle before reload statistics snapshot
test_recover_from_snapshot_2 and test_upload_snapshot_2 start snapshot
recovery on a freshly joined peer as soon as it lists the collection. The
collection appears once the creation entry is applied, while the peer is
still replaying the rest of the raft log, including the removal of the
killed peer. Recovery then decides which other replicas to remove or mark
dead from that stale local view and drops a healthy replica, leaving a
shard with a single replica.
Add a helper that waits until all peers share the same commit index and
have no pending operations, and use it in both tests before recovering.
Also fix a misleading comment in the recovery replica cleanup branch.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* Send the queue proxy batch as a pre-encoded gRPC body
The parent commits moved the WAL read, the operation clone and the request build
off the async runtime. Two passes over the batch were left on it.
Measured cost of each synchronous pass over a 26.2 MiB send batch (800 ops x 30
points x 256 dims), release build:
pass 1 clone WAL operations 72.4 ms moved by the parent commits
pass 2 build gRPC request 22.2 ms moved by the parent commits
pass 3 clone request to send 65.7 ms on the async runtime
pass 4 protobuf encode (tonic) 41.4 ms on the async runtime
Pass 3 is there because `with_points_client` takes `impl Fn` and the channel pool
calls that closure once per attempt, so each attempt needs its own owned message.
Pass 4 runs inside `poll_next`: for a unary call tonic encodes the whole message
in a single synchronous `encode_item`, so a worker is blocked for the full 41 ms,
once per attempt.
Encoding the batch up front removes both. The generated client cannot take a
pre-encoded body, `update_batch` is typed `impl IntoRequest<UpdateBatchInternal>`,
but all it does is pick a codec and a path and call `Grpc::unary`, and we already
build the client ourselves from a pooled channel. `update_batch_pre_encoded` does
the same three things with a codec that writes the encoded bytes through and
decodes the response with prost.
What stays on the runtime is the copy of the encoded body into tonic's send
buffer: 14.9 ms for 26.2 MiB under jemalloc, nearly all of it faulting in freshly
mapped pages rather than the copy itself (0.9 ms when the allocator hands back
warm pages). Retries share the same refcounted bytes instead of cloning and
re-encoding, so they drop with it.
on the runtime before 107.2 ms per attempt
on the runtime after 14.9 ms per attempt
Bypassing the generated client means the RPC path and message types no longer
follow the proto automatically, so a test checks them against the compiled
descriptor set.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZjWYpKeYdGpKiLeEZU9oc
* Hand the pre-encoded update batch a configured Grpc, take the service name from the generated code
`update_batch_pre_encoded` took a bare channel plus a `max_decoding_message_size`
argument, and its only caller passed `usize::MAX`. Every other internal client
applies that limit inside its `with_*_client` helper, so do the same: `with_grpc`
hands out the `tonic::client::Grpc` the generated clients wrap, already
configured, and the argument goes away.
The service half of the RPC identity now comes from the generated
`points_internal_server::SERVICE_NAME` instead of a second literal. Only the
method name and the path literal remain hand-written, still pinned to the
descriptor set by the test.
`PreEncodedMessage::encode` uses `encode_to_vec`: one pass instead of a separate
`encoded_len` call, and no `expect`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Adapt pre-encoding to the build/forward split from #10599
`forward_update_batch` takes `Arc<UpdateBatchInternal>` and encodes it
once on the blocking pool, so the channel pool's attempts share the
bytes. The queue proxy keeps the built request in that `Arc` across
`BATCH_RETRIES` and for the per-operation isolation path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@qdrant.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Share the text index post-tokenization indexing path
Extract `MutableInvertedIndex::index_str_tokens` so the write path and
read-only live reload cannot drift apart, and collapse the two identical
on-disk file listings into one.
The extracted helper gates the ordered document on `point_to_doc.is_some()`
rather than re-reading `config.phrase_matching`. Equivalent, since
`point_to_doc` is built from that flag, and it skips a needless clone if the
two ever disagree.
No behavior change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Cover the text index live reload path
`ReadOnlyAppendableFullTextIndex::live_reload` had no direct test: it replays
stored documents through the same post-tokenization indexing as the write path,
but nothing pinned that down.
Asserts the incremental reload lands on the same state as a fresh
`open_appendable` after a writer deletes one point and appends two, over both
`phrase_matching` values. The phrase leg matters because adjacency depends on
the ordered document being indexed, not just the token set.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Apply suggestion from @timvisee
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
Every failure to open a segment was reported the same way: one warning,
same wording, segment dropped, shard serves without it. Two very
different things land there.
The manifest is superset-biased, so it may list a segment the leader has
not finalized yet or has already removed. Both arrive as `FileNotFound`,
both fix themselves once the follower catches up, and both are routine.
Anything else is a segment that should have loaded and did not. The shard
opens without it and answers queries over a subset of its data, returning
success to the client. In a recent load test this produced 667 warnings
indistinguishable from ordinary leader churn.
Report the first at debug and the second at error. The manifest does not
need re-reading to tell them apart, so this costs nothing on the open
path.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
`map_get_err` is handed the key it was reading, but only the `NotFound`
arm kept it; every other error boxed the underlying failure and dropped
the key on the floor.
Callers that report such a failure are then unable to say what it was
reading. The read-only segment open is the clearest case: it logs one
warning per skipped segment, so a failure anywhere among a segment's
objects — state file, id tracker, payload storage, per-vector storage
and index, payload indexes — produces the same line, naming only the
segment uuid. A recent load test hit exactly this: 667 warnings, all
byte-identical, none of them saying which file failed.
Give the `S3` variant an explicit `path` beside its source error, rather
than folding the key into the message, so the object stays a field
callers can read. It is optional because most construction sites are not
about one particular object — a short or overlapping read from the
scatter buffer, the append context's protocol errors — and those keep
using `s3()` unchanged. `s3_at()` sets it, and the object-store read
surface (every `map_get_err` caller, plus `list_files` and `exists`) now
does.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(uio): don't fetch a zero-length object on a populating async open
The `PreferBackground`/`Blocking` arm of the disk cache's `open_async` built
its prefetch range by hand as `0..len`. For a zero-length object that is
`0..0`, which `object_store` rejects client-side with
`InvalidGetRange::Inconsistent` ("Range started at 0 and ended at 0") rather
than answering with an empty body.
The failure is not contained to the file: a single empty object anywhere
under a segment prefix fails the whole segment open, and a read-only
follower then logs "skipping unloadable segment" and serves the shard
without it — silently returning results computed over a subset of the data.
It is also deterministic, so the segment stays dropped on every retry.
The sync path already handles this correctly via `schedule_whole`
(`read_from_into_byte_buffer` disambiguates the unsatisfiable-range error
with a `len` call and yields an empty buffer); the async arm was the only
place assembling the range itself. Create the local mirror first and skip
the fetch entirely when there is nothing to read.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci: stop using the minio/mc image, which no longer exists
`minio/mc` has been withdrawn from Docker Hub — pulling it now fails with
"repository does not exist or may require 'docker login'". The readiness
loop ran it 60 times with output redirected to /dev/null, so the pull
error was invisible and the job failed as "rustfs did not become ready in
60s", pointing at the wrong component. Bucket creation used the same
image and would have failed next.
Use the AWS CLI that ships with the runner image instead: no third-party
container to pull for either step. The readiness probe stays an
authenticated call (`s3api list-buckets`), so it still waits out
credential setup and not just the port opening, and it now reports the
final failure instead of swallowing it.
Also pin the rustfs service image by digest. That was not the cause here
— rc.5 and rc.6 both work — but this workflow already pins its actions by
SHA, and a service tracking `latest` is how a green suite turns red with
no change to the repo.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
External->internal id resolution is the expensive half of estimating a
`has_id` filter, and the query API's rescore stage turns every prefetch
into exactly such a filter. Two paths paid for it more than once:
- `search_vectors_plain` re-estimated the filter its caller had already
estimated to pick the plain strategy, so every segment resolved the
whole id list twice. It now takes the estimation as an argument, and
the dispatcher makes it once for every strategy.
- Formula rescoring resolved the prefetched ids one at a time, which the
disk-resident id tracker turns into an unpipelined block read per
point, instead of the batched single pass it offers.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Reject snapshot upload without collection config before loading it
The raw IO error from `CollectionConfigInternal::load` embedded the
server-side temporary path in the API response. Check for the file first
and return a fixed bad-input error instead.
Part of #10553
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A3o9eWSZNMa6WAs5F2HMZC
* Fix missing-config check to run before restore_snapshot loads config
The path-leak guard lived after Collection::restore_snapshot, but that
function already calls CollectionConfigInternal::load and surfaced the
temp path as a 500. Require a regular config.json file before loading,
and cover a directory-shaped config entry in the openapi test.
* Use a valid empty TAR in the missing-config snapshot upload test
Avoid depending on malformed-archive handling; exercise the missing
collection-config path with a real TAR that has no entries.
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
* Move queue proxy WAL read off the async runtime
`read_wal_batch()` read up to MAX_BATCH_BYTES (32 MiB) from the WAL and
deserialized it synchronously on the async runtime. That runtime also serves
all internal gRPC, including the health check the transport channel pool uses
to decide whether a peer is alive, so during the queue-replay phase of a
snapshot shard transfer the sender periodically stopped answering internal
requests for the duration of a 32 MiB disk read plus decode.
Measured on a 3-node cluster with no CPU/memory/IO limits and ~30% free RAM:
the sender's health-check p99 went from 3-6 ms during the download phase to
81-93 ms during replay, across three separate transfers, while a peer probed
at the same instant stayed flat at 3-5 ms.
Take the lock with `lock_owned().await` before `spawn_blocking` rather than
`blocking_lock()` inside it, so a blocking-pool thread is only ever occupied
by the read and never by waiting for a concurrent writer. Lock scope is
unchanged - the mutex was already held across the whole synchronous read.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZjWYpKeYdGpKiLeEZU9oc
* remove unwanted comment
* Build the queue proxy send batch off the async runtime too
The previous commit moved the WAL read to the blocking pool, but measuring it
showed no improvement: the sender's health-check p99 during queue replay stayed
at ~90-100 ms. perf on the patched build explained why - optimizer/HNSW load is
ambient (~90% of CPU in both the download and replay phases, so not what makes
replay special), while one general-runtime worker burns 7.7% of all CPU during
replay against ~0.5% during download. That worker is doing the *send* half of
the loop, which is still synchronous:
- transfer_operations_batch() clones every operation in the batch
- forward_update_batch() converts each one into its gRPC representation
Both are full passes over up to MAX_BATCH_BYTES (32 MiB) of point data, on the
runtime that also answers internal health checks.
Move both to the blocking pool. WalBatch now holds its operations behind an Arc
so the batch can be shared into a blocking task without being copied first, and
the gRPC request construction is split out of forward_update_batch into
RemoteShard::build_update_batch_request so it can be spawned. The extracted
function keeps the original body and indentation, so the diff is the move plus
the plumbing rather than a reindent.
forward_update_batch has exactly one caller (the queue proxy), so this adds no
blocking-pool hop to the normal replication path.
Still on the runtime and not addressed here: the per-attempt clone of the
request inside with_points_client, and tonic's own protobuf encoding.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZjWYpKeYdGpKiLeEZU9oc
* Use single clone, don't iterate manually
* Methods are droppable, don't hang on spawn_blocking with full runtime
* Build the WAL transfer batch once, reuse it across retries
The batch was deep cloned on every send so the original operations stayed
available for retries and for the one-by-one isolation path. Instead, move the
operations into the gRPC request once and retry on the prebuilt request, which
`with_points_client` already clones per attempt.
Stripping WAL indices and setting the force flag now happens in the WAL read
task, so no per-operation work runs on the async runtime.
Drop the pre-1.14.1 fallback that transferred operations individually, all
peers support batched updates by now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@qdrant.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: timvisee <tim@visee.me>
Thread a `Populate` through the disk-resident id tracker's open paths
(`DiskMappingReader`, `DiskIdTracker::open`, `ReadOnlyDiskIdTracker`,
`ReadOnlyIdTrackerEnum`) so a `cached` placement primes the page cache with the
mapping files on load instead of leaving them to page in on demand. The
populate is derived from the segment config's placement at load time, clamped
by low-memory mode, in both the writable segment open and the read-only one.
The update-only lookup path keeps its transfer-nothing policy, and the
build-time open stays cold: the built segment is reloaded anyway.
`cached` is no longer rejected by validation.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Add `id_tracker: { memory: cold | pinned }` to CollectionParams,
CollectionParamsDiff and CreateCollection (REST + gRPC `IdTrackerParams`),
mirroring `payload: { memory }`. `cold` builds the disk-resident id tracker,
`pinned` the in-RAM immutable one. Unset keeps the current behavior: the
`serverless_compatible` feature flag decides.
The requested placement is persisted as an optional `id_tracker_memory` on
SegmentConfig (skipped when unset, so existing configs are unchanged); the
segment builder resolves it through `SegmentConfig::id_tracker_memory_placement`
instead of reading the feature flag directly.
The config mismatch optimizer rebuilds non-appendable segments whose effective
placement differs from the requested one. Appendable segments are skipped: they
always use the mutable tracker and get the current config when indexed.
`cached` is rejected by validation: the disk mapping reader has no
populate-on-open path.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix: do not claim an unfinished operation when flushing
A flush pass can capture a segment between the separately locked steps of
one update operation. Persisting it under that operation's version marks
the segment clean while the rest is still in memory, so every later pass
skips it and the WAL acknowledge moves past the operation.
Clamp what a flush claims to the last fully applied operation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Fix rustfmt in alias_mapping test after merging dev
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: qdrant-cloud-bot <111755117+qdrant-cloud-bot@users.noreply.github.com>
tools/coverage.sh was split into tools/unit-test-coverage.sh and
tools/integration-test-coverage.sh in #6414, but DEVELOPMENT.md still
pointed at the old path and the new script's own usage header kept the
old name. Updated both to the unit-test script.
* Fix MMR pagination with offsets
(cherry picked from commit 6a778bbde8)
* Clamp MMR selection capacity at the candidate count
Move the guard into `maximal_marginal_relevance`, which every MMR caller
routes through, so the collection, edge and local-shard paths are covered
instead of just the collection one. The collection-level limit is then
plainly `limit + offset`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit a63d744131)
---------
Co-authored-by: mikemikimike <13286568797@163.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
xzfc