Tolerate not-yet-ready collection upserts in test_rejoin_cluster and give
JWT snapshot uploads more headroom while still bounding auth-rejection hangs.
* feat: global quota API
Memory and disk are node-wide resources, so configuring their thresholds
per collection through strict mode makes little sense. Move them behind a
single cluster-wide `QuotaManager`.
The quota config is seeded from `storage.quotas` in the settings (and so
from env vars), overridden by `quota.json` in the storage directory, and
updated cluster-wide through a new `SetQuotaConfig` consensus operation
which rewrites that file on every peer. Raft snapshots carry it too, so a
peer that joins by snapshot picks it up.
Quotas are enforced wherever the strict mode memory and disk checks used
to run, but no longer gated behind `strict_mode.enabled`: a value set in
an enabled strict mode config still wins per resource, the quota is the
default. Rejections name both the condition that tripped and the config
that governs it.
`GET /quotas` reports the config plus current utilization to global read
users; `PUT /quotas` replaces it for global manage users.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: cover the quota endpoints in the API consistency checks
`test_all_rest_endpoints_are_covered` and the OpenAPI endpoint count both
break on any new REST endpoint. Add `GET`/`PUT /quotas` to `ACTION_ACCESS`
with their JWT access tests, and bump the expected API count. The quota
endpoints stay out of `REST_ENDPOINT_WHITELIST`: that list is for
data-plane endpoints reported per-endpoint in metrics.
Also add a Raft snapshot CBOR compatibility test — snapshots are exchanged
between peers of different versions during a rolling upgrade, so
`quota_config` must be absent-tolerant in both directions.
Review feedback: persist through `SaveOnDisk`, which already implements the
write-before-swap protocol this was doing by hand; validate the config at
both persistence boundaries, since a hand-edited quota file or a config
arriving through consensus does not pass the REST handler's validation, and
a `0%` limit would reject every update forever.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: assert seeding a quota from invalid settings persists nothing
Follow-up to review feedback claiming `SaveOnDisk::load_or_init` writes the
init value before it is validated. It does not — only `SaveOnDisk::new`
persists — but the property matters: were seeding to persist first, invalid
settings would leave a `quota.json` that fails validation on every
subsequent start, and the node could only be recovered by deleting it by
hand. Pin it down with a test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: make QuotaManager the single reader of memory and disk
The quota checks measured memory and disk themselves, while the optimizer
and the WAL disk watcher each called `fs4::available_space` behind their
own ad-hoc caches. Fold all of it into QuotaManager: it owns the readings,
the freshness policy, and the limits they are compared against.
Moves the module to `lib/shard`, since the optimizer sits below `storage`
and has to reach it; `storage::quota` re-exports it, so consensus, the
`/quotas` API and StorageConfig are unchanged. The manager is installed as
a process singleton by TableOfContent, ahead of loading any collection.
- Callers hand in QuotaLimits overrides instead of a StrictModeConfig, and
an override can now only tighten. A collection-level admin could raise
`max_disk_usage_percent` past a cluster-wide limit that needed global
manage rights to set; ties resolve to the quota so the rejection names
the knob that actually has to change.
- Measurements are cached for 5s, but a reading at or above its limit is
never reused: a rejected client retries, and freeing the resource has to
take effect on the next request rather than a TTL later.
- `fits_on_disk` sizes an optimization against physical free space only,
never the configured limits. Optimizations are what free a full disk, so
the quota must not be what stops one.
- `percent_of` widens to u128 instead of saturating the multiply, which
under-reported utilization (failing open) above ~184 PB.
- StorageConfig::quotas is optional; absent means no quota is enforced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: don't recover dead replicas onto a node at a resource limit
Recovering a dead replica pulls a whole copy of its shard onto this node.
If it is already at its memory or disk quota that transfer cannot finish,
and starting it only pushes the node further past the limit. Skip it and
reconsider on a later sync, once the resource frees up.
Adds QuotaManager::check_capacity for work that lands bytes here without
being an update. Unlike fits_on_disk the configured limits do apply:
taking on a replica is not what frees a full node, so there is no deadlock
to avoid by letting it through.
The check is hoisted out of the per-shard loop because a node over its
limit re-measures on every call, so checking per dead shard would cost a
statvfs each. It is free when no quota is configured.
Also trims the comments across the quota module, which had grown well past
what the code needs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: drop trivial and duplicated quota tests
Six tests removed, ~140 lines, with no loss of coverage:
- a_rejection_names_the_knob_that_has_to_change asserted that a format!
contains its own literals; the message is covered end-to-end by the
override test and by test_global_quota.py.
- a_node_over_its_quota_has_no_capacity_to_take_on_a_replica was 30 lines
for check_capacity, a one-line delegation to check_update the test above
it already calls.
- a_rejecting_measurement_is_never_served_from_the_cache duplicated the
meter test, which proves the same rule with an injected reader instead
of inferring it from the real filesystem.
- free_space_is_reported_without_enforcing_anything covered a one-line
accessor, and its point is what the fits_on_disk test is for.
- The two resolve tests and the three meter tests each collapse into one.
DiskFit::Unknown keeps its coverage as two lines inside the fits_on_disk
test rather than its own fixture. The snapshot compat pair becomes one
test: the second only asserted cluster_metadata.is_empty(), which says
nothing about quotas — the real check was the deserialize, now an expect
that states it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: re-measure free space as the disk fills, and drop a Windows-only assert
Two CI failures, both from this branch.
e2e test_low_disk: the DiskUsageWatcher I replaced escalated to checking
on every call once free space fell below 512 MB. Folding it into the quota
manager lost that — available_bytes passed no limit, so a reading was
reused for the full 5s however little space was left. On a disk filling as
fast as that test fills it, 5s blind is enough to actually run out and the
WAL write dies instead of returning "No space left on device".
available_bytes now takes a watch_below level and never reuses a reading
under it, which is what the old ladder was expressing. The watcher passes
max(min_free, 512 MB), so the escalation point is back; above it the 5s
cache still costs fewer syscalls than the old 128-call ladder. fits_on_disk
gets the same rule by passing required_bytes, so a merge that does not fit
re-checks rather than sitting on a stale sample.
Windows: fits_on_disk on a missing path was asserted to be Unknown, but
GetDiskFreeSpaceEx resolves up to the containing drive and succeeds — as
common::disk_usage's own test documents. Dropped; the branch is a two-line
else and is not portably reachable.
Also renames an_optimization_is_sized_against_the_disk_not_the_quota, which
needed explaining to be understood.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: report the quota config in telemetry
Reads it from the quota manager rather than the settings, so it is the
config the node is actually enforcing: a peer that missed a consensus
update reports what it is applying, not what the cluster agreed on.
Gated on global access, the same access `GET /quotas` requires, and left
out of `PeerTelemetry` — a quota is per-node state, so each peer reports
its own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: regenerate OpenAPI, and cover the quota in the telemetry key sets
Two CI failures from the previous commit.
Referencing QuotaConfig from TelemetryData moves its definition earlier
in `components/schemas`, because TelemetryData is generated ahead of
QuotaStatus. Regenerated rather than hand-patched, so the schema is a
pure move.
test_telemetry_detail asserts the exact set of top-level telemetry keys.
The quota is reported at every level, including 0 — it is three scalars,
it is the default the endpoint serves, and it is what explains an update
being rejected — so both key sets gain it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: remove `max_disk_usage_percent` from strict mode
Disk is a node-wide resource, so a per-collection percentage of it never
meant anything a caller could act on: the limit describes how full the
*node* is, and which collection the write happens to target has nothing
to do with it. The global quota is where it belongs.
It shipped in 1.18.2 without documentation, so this drops it outright
rather than deprecating. Removal is soft in every direction: StrictModeConfig
has no `deny_unknown_fields`, so a client still sending it gets it ignored
rather than a 400, and the same struct deserializes the persisted collection
config, so collections created on 1.18.2+ keep loading. Proto field 22 is
reserved so the number is never reused.
The e2e test becomes a quota test — the fixture and the timing are the
interesting parts and they carry over unchanged; only how the threshold is
configured differs.
`max_resident_memory_percent` was documented and stays for now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: enforce the strict mode memory limit outside the quota
`max_resident_memory_percent` was folded into the quota as an override,
which meant the quota check had to know about strict mode, and retiring
the setting would mean unpicking `EffectiveLimit` and `LimitSource` from
the resolution logic.
It is now a check of its own in `verification/mod.rs`, next to the strict
mode checks it belongs with, borrowing only the measurement from the quota
manager — which stays the node's single reader of process memory, so both
checks still share one reading. Deleting the setting later is deleting one
function and its one caller.
`QuotaManager::check_update` takes no arguments and consults the quota
alone. A collection can still only tighten the limit for itself, because
its own check runs in addition rather than in place of the quota's, and
each rejection now names the config that has to change without having to
carry a `LimitSource` to say so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: enforce the quota on the update path, not in strict mode
The quota check sat inside `check_strict_mode_toc_batch` only because that
was the one place holding the collection's strict mode config. It doesn't
need one any more, and the placement had a real cost: coverage depended on
each handler remembering to ask for a strict mode check, and four of the
internal update RPCs do — `sync_internal`, which moves the most bytes onto
a node, does not.
It now runs in `Collection::update_from_client` and `update_from_peer`,
which every update passes through. `update_from_client` checks ahead of the
shard split, so an operation is accepted or refused whole rather than
landing on some shards and being refused by others.
Classification moves with it, from ~10 `consumes_memory` impls on request
DTOs to one exhaustive `CollectionUpdateOperations::consumes_quota`. The
internal enum has variants — raw upserts, conditional upserts, the syncs —
that have no client-facing request type, so per-DTO impls structurally
could not classify them.
Shard-transfer syncs stay excluded, as they are today: a transfer is sized
up once before it starts, and refusing its batches partway abandons work
that is nearly done only for it to restart from the beginning.
Index and named-vector creation reach shards through consensus, past this
check — a peer must not refuse what the cluster agreed to — so they keep
their pre-consensus check, now against the quota manager directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: deprecate `max_resident_memory_percent` in strict mode
Same reason the disk threshold went: memory is node-wide, so a
per-collection percentage of it caps how full the *node* is, which has
nothing to do with which collection is being written to. The node-wide
quota caps it once for everything.
Unlike the disk threshold this one shipped documented, in 1.18.0, so it
keeps working — as a limit a collection can tighten for itself, never lift
— and gets the usual markers: `#[deprecated]` on both Rust structs,
`[deprecated = true]` on proto field 21, and `deprecated: true` in the
OpenAPI schema, which schemars derives from the attribute.
The note names 1.21 as the removal. Recording a version matters here: the
audit in docs/plans/overdue-deprecations.md found that this repo has never
written a removal deadline down, and members of the 1.15.0 deprecation
batch are still in tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: reconcile the quota readers with #9891#9891 landed effective (cgroup) figures in telemetry while this branch was
making QuotaManager the single reader of memory and disk. Two collisions,
neither of which git sees.
`segment::utils::mem::total_memory_bytes` is now a shared accessor with a
5s TTL, so a cgroup resize is picked up. The quota module had its own
`OnceLock` copy that froze the value at startup — exactly what #9891 set
out to fix — so it delegates to the shared one instead.
Telemetry's new `disk_size` called `common::disk_usage::disk_usage`
directly. That reader lost its TTL cache on this branch when the caching
moved into the quota manager's meter, so it would have taken an uncached
`statvfs` on every telemetry request, and it put a second disk reader back
in the tree. It goes through `QuotaManager::disk_capacity_bytes` now,
sharing the reading the quota check already takes. Verified it still
reports the storage filesystem, matching `df`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: split the quota manager by what each half does
`manager.rs` had grown to 450 lines holding three separate jobs: owning
the config and its file, taking the readings, and comparing one against
the other.
- `manager/store.rs` — the `Store` enum, `QUOTA_CONFIG_FILE`, and config
validation, which is now the store's own business rather than something
every caller has to remember to do first.
- `manager/measure.rs` — every reading, and `DiskFit`. The "nothing else
calls `statvfs` or reads process RSS" claim is now checkable by looking
at one file.
- `manager/enforce.rs` — `check_update` / `check_capacity` and the
threshold comparison.
- `manager/mod.rs` — the struct, its construction, and the config
accessors: what a reader needs to see first.
Tests move with their subject. No behaviour change: `set_config` used to
validate before delegating to the store, and now the store validates on
write, which is the same order of operations from the outside.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: count copy-on-write deletes toward the quota
Dropping a vector or a payload key does not free anything on its own:
copy-on-write rewrites the point to produce the version without that
field, so storage grows first and is only reclaimed once the optimizer
gets to it. Gating those as if they were reclaiming space let a full node
keep taking writes that make it fuller.
Deleting whole points stays exempt. That is the one operation that has to
work on a node at its limit, or there is no way back under it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: say that the quota's reported usage is per node
`GET /quotas` returns one cluster-wide config and one set of utilization
figures, which reads as though both describe the cluster. They do not:
memory and disk are node-local, so `usage` is whatever the peer that
served the request is seeing, and a peer under its limit says nothing
about the others.
Also corrects `resident_memory_percent`, which claimed to be a share of
total system memory. It is a share of the memory available to the
process, which under a cgroup is the limit rather than the host's RAM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: treat a node over its quota as a failed replica, not a bad request
A quota rejection described the request as invalid (400) and was classified
non-transient, which is how the replica set recognises errors that every
replica would produce alike. A quota is the opposite: the input is fine and
the answer depends on which machine you ask. On the default `wait=false`
path that combination silently dropped the write — `update.rs` only
deactivates transient failures when nothing completed — leaving the replica
Active and permanently missing data its co-replicas had.
It is now `InsufficientStorage`, transient, HTTP 507 / gRPC
`ResourceExhausted`. So a node that is out of room is handled like one that
is offline:
- last active replica, or every replica over quota: nothing could take the
write, and the client is told the cluster is out of room.
- more than one replica: the full node is deactivated through the same path
a dead peer takes, and the update stands if enough replicas accepted it.
`check_capacity` already keeps recovery off that node until it has room.
The check also moves off `update_from_client`, which applied the
coordinator's own limit to the whole operation even when it held no replica
of the shards being written. Each replica set now gates its own local write
and records the refusal as a failure of this peer, so a node only ever
answers for itself.
`ResourceExhausted` is shared with rate limiting, and the reverse conversion
mapped it straight to `RateLimitExceeded` — a forwarded rejection came back
as 429. Statuses now carry a marker so the two stay distinguishable across
the wire.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: report quota pressure per node, and across the cluster
A quota is node-local, so finding out which node has hit one meant asking
each of them in turn — and nothing at all showed up in monitoring.
`/metrics` gains a `quota_exceeded` gauge for the local node. It is emitted
only while the quota is enabled: with it off the value would be a constant
0 that says nothing about the node, and an alert built on it would go quiet
rather than fire if someone disabled the quota. Telemetry's `quota` field
carries the same verdict alongside the config, since that is where the
metric is derived from.
`GET /quotas` now answers for the whole cluster. A new `GetQuotaUsage` RPC
on the internal `QdrantInternal` service returns what one peer is using,
and the handler fans it out to every known peer in parallel. Peers that do
not answer are left out rather than failing the request — the nodes that
are out of room are exactly the ones most likely to time out, and a partial
answer still names them. Outside distributed mode the field is absent
rather than a map of one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: report the quota metric per resource
`quota_exceeded` was one flag for the whole node, which does not say what
to go and fix — disk is freed by deleting or optimizing, memory by
unloading. It now carries a `resource` label:
quota_exceeded{resource="memory"} 0
quota_exceeded{resource="disk"} 1
A resource with no limit gets no series at all, for the same reason the
metric is absent while the quota is disabled: a series that can never reach
1 reads as healthy and would quietly carry an alert that cannot fire.
`QuotaManager::exceeded` returns the per-resource verdict, with `None` for
a resource this node does not cap. Telemetry reports the same breakdown,
since the metric is derived from it. The peer usage RPC keeps a single
flag — it sits next to both percentages, so it only has to answer "is this
peer refusing writes".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: drop a no-op error conversion the linter caught
`check_global_access` already returns a `StorageError`, so mapping it
through `StorageError::from` converted the type to itself and tripped
`clippy::useless_conversion`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: hold a tripped quota until usage clears a release margin
A resource resting on its limit crosses it in both directions on the noise
between two readings, and each crossing is expensive: the node refuses a
write, its replica is deactivated, usage dips, recovery starts sending a
whole shard copy back, and the arriving data pushes it over again. The loop
sustains itself, and every lap costs a shard transfer.
A limit now trips at its configured value but only clears once usage has
fallen 5 percentage points below it, so the crossing has to be real. The
margin is floored at 1%, since a limit smaller than the margin would
otherwise be impossible to fall back under and would strand the node.
The verdict is carried on the manager rather than recomputed, which makes
it the thing reporting shows: expect `exceeded` to be set while the
utilization next to it is already back under the limit. Rejections say so
too, rather than claiming a limit that is no longer exceeded:
Disk usage is at 87% of total capacity. It reached the configured limit
of 90% and has to fall below 85% before this node takes writes again.
Changing the config clears the verdicts. New limits are a deliberate act,
and should not be held back by the margin of a limit that no longer exists.
Both resources are now evaluated on every check instead of stopping at the
first failure, so a verdict is never left behind reporting a reading that
has since been superseded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: make the quota release margin configurable
5 points is a guess about how noisy a deployment's usage is, which is not
something one number can be right about: a node whose disk moves in
gigabyte steps needs a wider margin than one that creeps, and an operator
who wants the old flip-on-every-reading behaviour should be able to ask
for it.
`release_margin_percent` joins the rest of the quota config, so it seeds
from `QDRANT__STORAGE__QUOTAS__RELEASE_MARGIN_PERCENT`, replicates through
consensus, and changes with `PUT /quotas`. Defaults to 5 and is filled in
when a request omits it, so it always answers with the margin actually in
force rather than leaving the caller to assume one. `0` releases as soon as
usage is back under the limit.
`QuotaConfig` grows a hand-written `Default` for it, since deriving one
would have quietly defaulted the margin to 0 and disabled the hysteresis
for anyone constructing a config in code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: leave the release margin unset by default, and hold verdicts in atomics
`release_margin_percent` is `null` unless someone sets it, rather than
materialising 5 into every config. A quota written today then does not pin a
number a later release may want to revise, and `{"enabled": false}` still
round-trips as itself. `QuotaConfig::limits` resolves it, next to `enabled`,
so enforcement never sees the unset case.
The verdicts move from a `Mutex<QuotaExceeded>` to one `AtomicBool` per
resource. They are judged independently and nothing reads them as a pair, so
the lock only added contention to the path every update takes; a verdict
that races a concurrent check is re-decided by the next one from a fresh
reading.
That also drops the tri-state. Only "was this over its limit" has to survive
between checks — whether a resource is enforced at all follows from the
config and the reading, so it is derived when reporting rather than stored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: drop a comment arguing with a design that was never here
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* remove unused iter_offsets
* Replace MultivectorOffsetsStorage::iter_offsets with callback-based for_each_offset
First step of removing the iterator-returning read API (whose deep,
composable generic types blow up mangled symbol size). Convert the
offsets read from an iterator to a callback the caller pushes into:
- trait method iter_offsets -> for_each_offset(ids, FnMut(usize, MultivectorOffset))
returning common::universal_io::Result<()>
- Mmap impl now uses the callback read_batch (drops one read_iter use)
- Ram / Chunked impls push into the callback; Chunked still goes through
iter_vectors for now (converted in a later step)
- the single caller (for_each_in_multi_batch) passes a closure
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Implement read_batch directly on ReadPipeline, not via read_iter
read_batch now drives the pipeline itself (refill-then-wait loop, like
read_multi_iter) and invokes the callback per result, instead of
consuming the iterator returned by read_iter. A step toward removing the
iterator-returning read API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove the ReadMulti RPC from the StorageRead gRPC service
ReadMulti was the only real consumer of UniversalRead::read_multi (which
itself relies on read_multi_iter). Removing the RPC end-to-end clears the
path to dropping that read API. StorageReadService keeps all its other
RPCs (ListFiles, FileExists, FileLength, ReadBytes, ReadBytesStream,
ReadWhole, ReadBatch).
- proto: drop `rpc ReadMulti` + ReadMulti{Entry,Request,Response}
- regenerated lib/api + uio-client generated code; drop ReadMulti
validation rules in lib/api/build.rs
- tonic: delete the read_multi handler + its 2 tests
- uio-client: delete Client::read_multi, the mock-server impl, and 2 tests
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove UniversalRead::read_multi
Its only real consumer was the StorageRead ReadMulti gRPC handler (removed
in the previous commit); the two wrapper forwarders had no callers. Drop
the trait method and both forwarders (typed/read_only), and remove the
io_uring test that only existed to compare read_multi vs read_multi_iter
(read_multi_iter stays covered by the other tests). Another step toward
removing the iterator-returning read API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drive ReadPipeline directly in gridstore read_from_pages
Replace the read_multi_iter call in Pages::read_from_pages with a direct
pipeline loop (refill-then-drain), scheduling each multi-page read on its
own page file. Behavior unchanged; another step toward removing the
iterator-returning read_multi_iter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drive ReadPipeline directly in gridstore read_batch_from_pages
Replace the second read_multi_iter call (in Pages::read_batch_from_pages)
with a direct pipeline loop, scheduling each (ReadMeta, page, range) on its
own page file and propagating errors via GridstoreError. Single/multi-page
buffering and out-of-order reassembly are unchanged. No more read_multi_iter
in gridstore.
Measured overhead on warm mmap (both paths are zero-copy borrows): ~0.3 ns
per read of fixed control cost, flat across read sizes — well under 0.1% of
a real payload read.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drive ReadPipeline directly in on-disk postings with_posting_views
Replace read_iter in OnDiskPostings::with_posting_views with a direct
pipeline loop. wait_bytemuck yields a file-borrowed Cow, so postings are
still stored zero-copy in raw_postings (a read_batch swap would have forced
an owned copy of every posting list per query on the mmap backend).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Read on-disk posting headers via read_batch, drop the HeadersBatch iterator
headers_iter now reads headers with the callback read_batch API: each header
is parsed (copied) out of the read bytes, so nothing borrows the file past the
read — no pipeline needed. Since the read is now eager, HeadersBatch holds the
collected Vec<HeaderResult> directly instead of a Box<dyn Iterator>, dropping
the boxing, the dynamic dispatch, and the struct's lifetime parameter.
with_posting_views takes the Vec and still pipelines the posting reads.
Removes the last read_iter use in this file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Use read_batch in simple_disk_cache populate_from
populate_from reads one byte per block only to fault blocks into the local
cache, discarding the bytes — a no-op-callback read_batch fits exactly. Drives
the same DiskCachePipeline as before; one less read_iter caller.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Implement read_iter directly on ReadPipeline, not via read_multi_iter
read_iter now drives the pipeline itself (refill-then-wait loop, mirroring
read_bytes_iter) instead of mapping its ranges onto self and calling
read_multi_iter. Same signature and iterator contract, so all callers are
unchanged. Leaves iter_vectors as read_multi_iter's only remaining caller.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "Drive ReadPipeline directly in on-disk postings with_posting_views"
This reverts commit c02c41f17b.
* Add callback-based for_each_vector next to iter_vectors
for_each_vector drives the ReadPipeline directly across chunk files and
invokes a fallible callback per flattened multi-vector, returning
OperationResult, instead of returning an iterator built on read_multi_iter.
Callers will migrate onto it so iter_vectors (read_multi_iter's last caller)
can be removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Make dense for_each_in_batch / for_each_in_dense_batch fallible
Thread OperationResult up the dense batch-read path so io_uring read
errors propagate instead of being .expect()ed deep inside the storage.
The for_each_in_dense_batch scorer path (custom/metric query scorers)
now carries the Result to the infallible score() boundary where it is
.expect()ed; read_vectors keeps its () signature and .expect()s locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Convert dense read_vectors to for_each_vector
The read-only and appendable dense storage read_vectors impls drove
ChunkedVectors::iter_vectors directly; switch them to the callback-based
for_each_vector and .expect() the result at the (infallible) read_vectors
boundary. Removes the last dense-path iter_vectors callers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Convert quantized for_each_offset to for_each_vector + OperationResult
The two chunked MultivectorOffsetsStorage impls drove iter_vectors to read
the offset table; switch them to the callback-based for_each_vector.
for_each_vector returns OperationResult, so upgrade the for_each_offset
trait (and all four impls) from universal_io::Result to OperationResult
(the universal_io -> Operation direction, via ?). No error downgrade.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Convert EncodedStorage/EncodedVectors iter_batch to callback for_each_batch
The two chunked-mmap EncodedStorage impls drove ChunkedVectors::iter_vectors
to back iter_batch. Replace the iterator-returning iter_batch on both the
EncodedStorage and EncodedVectors traits (quantization crate) with a callback
for_each_batch(FnMut(usize, &[u8])), and switch the chunked impls to
for_each_vector. The callback is infallible: the chunked impls .expect() the
read internally, matching iter_vectors' prior panic-on-read-error behavior,
so no OperationError is downgraded. Scorers and the multivector readers adopt
the callback; the accumulating multivector path owns (to_vec) only when it
must buffer across components.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Convert multivector read paths to for_each_vector
The read_only multivector free fn chained two iter_vectors (offsets feeding
vectors) - the recursive iterator nesting behind the worst symbol bloat.
Replace it with a callback for_each_vector that resolves the per-point
offsets into a Vec first, then drives ChunkedVectors::for_each_vector over
the flattened vectors. Migrate both multivector storages' read_vectors and
for_each_in_batch_multi accordingly, .expect()ing at their infallible
boundaries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove read_multi_iter and ChunkedVectors::iter_vectors
With every caller migrated to callback-based for_each_vector/for_each_batch,
delete the last iterator-returning multi-read APIs: ChunkedVectors::iter_vectors
(segment) and the read_multi_iter trait method plus its mmap override, the
TypedStorage/ReadOnly wrapper forwarders, and the two io_uring unit tests.
These deeply-nested monomorphized iterator types (read_multi_iter feeding
read_multi_iter) produced >1 MiB mangled drop_in_place symbols that overflowed
the macOS ld symbol-name limit; the callback rewrite eliminates them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Pass owned Cow through for_each_vector/for_each_batch to avoid a copy
The callback-based readers handed the callback a borrowed `&[u8]`/`&[T]`,
forcing the quantized multivector batch read to `to_vec()` each sub-vector
into its per-point buffer. But io_uring-like backends already return a freshly
owned buffer per read (`ACow::Owned`), so that was a redundant second copy.
Change `ChunkedVectorsRead::for_each_vector` and the `EncodedStorage` /
`EncodedVectors` `for_each_batch` callbacks to receive `Cow<[..]>` by value.
The buffering path now `into_owned()`s it — a move when the backend returned
owned (the case this path targets), a copy only for a borrowed Cow (mmap),
which never reaches this path. Immediate-use callers (scorers,
score_point_max_similarity, the dense/multivector readers) just deref the Cow;
the dense readers drop their now-redundant `Cow::Borrowed` wraps.
Also clarifies the multivector reader: `SubVectorOwner`/`owners`/
`sub_vector_offsets` naming, docs, and a corrected comment noting the per-point
buffer is what makes regrouping order-independent under out-of-order completion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drop the removed ReadMulti JWT access test; fix clippy unwrap_or_default
The ReadMulti StorageRead RPC was removed earlier in this branch, so the
consensus JWT-access test (and its registry entry) for it must go too. Also
switch the multivector buffer's `or_insert_with(SmallVec::new)` to
`or_default()` per clippy::unwrap_or_default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add MmapFile::read_batch
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* ci: parallelize consensus tests with pytest-xdist
Enable pytest-xdist for consensus_tests to run tests across multiple
workers in parallel, significantly reducing CI wall time (~20min → ~5-7min).
Changes:
- Add `-n auto --dist=loadfile` to the consensus test pytest invocation
- Remove hardcoded port_seed from tests that don't need fixed ports for
restart/rejoin (test_order_by, test_consensus_compaction,
test_named_vector_crud, test_listener_node)
- Give test_cluster_rejoin its own PORT_SEED=15000 to avoid port
conflicts with auth tests (PORT_SEED=10000)
- Derive restart ports from killed PeerProcess objects instead of
hardcoded arithmetic where possible
- Add xdist_group("auth") marker to auth test files to ensure they
run on the same worker (they share PORT_SEED=10000)
Made-with: Cursor
* fix: remove remaining hardcoded port_seed=20000 causing parallel test conflicts
8 test files were using port_seed=20000 as a positional argument to
start_cluster(), which was missed in the initial change. When running
in parallel with pytest-xdist, multiple workers would try to bind to
the same port range (20000-20x02), causing port conflicts and cascading
test failures.
Also remove port_seed=23000 from test_snapshot_recovery_kill.py since
it doesn't need fixed ports for restart.
Made-with: Cursor
* fix: use saved port for restart in test_two_follower_nodes_down
The test was restarting killed peers on hardcoded ports (20200/20100)
that previously matched port_seed=20000. After switching to random
ports, the restart ports no longer match the original peer ports,
causing raft state URI mismatches and peer startup failures.
Save the p2p_port from the killed PeerProcess and reuse it for restart.
Made-with: Cursor
* Reuse p2p ports when restarting killed peers in consensus tests
When a peer is killed and restarted with random ports, it gets a new
consensus URI. The cluster needs a Raft operation to update this URI,
which under CPU contention from parallel test workers can exceed the
30-second timeout. Fix by capturing each peer's p2p_port before killing
and reusing it on restart, so the URI stays the same and no consensus
update is needed.
Made-with: Cursor
* A few improvements for parallel runs (#8731)
* fix: make auth tests' PORT_SEED per-worker to avoid port collisions
* ci: improve failure visibility for parallel consensus tests
* Three small changes to make hangs, interleaved output, and coverage runs behave predictably under pytest-xdist
* fix: two test bugs surfaced by parallel runs and revert drop PR_SET_PDEATHSIG helper
* fix: wait for count convergence in test_triple_replication
* fix: clean leaked peer processes at test start
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add empty placeholder vector storage types for named vector CRUD
Introduce EmptyDenseVectorStorage and EmptySparseVectorStorage as
placeholder storages for newly created named vectors on immutable
segments. These report all vectors as deleted, consume no disk space,
and are reconstructed from segment config on load via the new
VectorStorageType::Empty and SparseVectorStorageType::Empty variants.
Key design decisions:
- is_on_disk is derived from original user config, not hardcoded
- MultiVectorConfig is preserved for multi-vector support
- Config mismatch optimizer skips Empty storage to avoid false rebuilds
- Quantization delegates normally (handles 0 vectors gracefully)
- get_vector includes debug_assert to catch unexpected access
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] segment-level operations for creating and deleting anmed vectors
* [AI] implement named vector creation and deleting in proxy segment
* [AI] Step 3: Proxy Segment Handling for Named Vector Operations
* [AI] implement for Edge
* [AI] implement consensus operations for named vector operations
* [AI] refactor VectorNameConfig, remove VectorNameConfigInternal
* [AI] handle vector schema inconsistency in raft snapshot recovery
* [AI] rest + grpc API
* [AI] clippy
* [AI] generate openAPI schema
* fmt
* ci fixes
* [AI] fix jwt access test
* [AI] nop operation for awaiting of consensus-commited update ops
* [AI] move vector name operations into points service
* [AI] implement internal api for vector name operations
* [AI] change collection-level config along with segment level operation
* [AI] vector schema reconceliation instead of error
* fmt
* missing compile-time option
* [AI] integration test
* [AI] fix missing JWT tests
* [AI] remove NOP
* [AI] openapi test
* [AI] fix initialization of mutable segment
* [AI] more simple integration tests
* fmt
* [AI] make cluster test a bit harder
* [AI] make test less flacky
* [AI] rabbit comments
* [AI] check params compatibility before writing vector config
* [AI] make sure to register vector storages in structure payload index
* [AI] vector name validation
* lower vector length validation to 200 chars to account for prefix in filename
* [AI] proxy segment: prevent stale data leak through optimization
* fmt
* [AI] filter out removed vectors from proxy response
* [AI] handle vector name in proxy
* fmt
* adjust proxy info based on dropped vectors
* [AI] proxy segment: update filters to correct has_vector condition
* fmt
* clippy
* Fix consensus snapshot applicaiton for vector schema
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add optional `api` field to audit log events
Add a new `api` field to audit log entries that records the API method
path (REST path or gRPC method name). Controlled by the `log_api` audit
config option. For denied auth requests, `api` is always logged and
`method` is omitted since there is no internal operation name available.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix formatting
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* deconstruct
* fix: test edge case
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* Skip audit logging for telemetry endpoints
Telemetry access checks (telemetry_memory, telemetry_requests,
telemetry_cluster, cluster_telemetry) fire on every metrics scrape and
produce very noisy audit logs with no security-relevant signal.
Switch all telemetry callers to use `auth.unlogged_access()` to bypass
audit logging, consistent with the existing metrics and prepare_data
handlers that already did this.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* trigger CI
* feat: add some tests
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* restrict recovery from snapshot folder
* fmt
* clippy
* Extend test, path must be inside snapshots directory
* In release builds, hide detailed tar error message to not leak contents
* Change from bad request to forbidden
* fix(auth_tests): place snapshot in peer snapshots dir for recover_collection_snapshot
The recover from snapshot API now requires file:// paths to be inside the
configured snapshots directory. Write the test snapshot into the peer's
snapshots directory instead of a temp file so the request is allowed.
Made-with: Cursor
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Cursor Agent <agent@cursor.com>
* Add progress_tracker.rs
* Pass progress tracker around
* Populate progress tracker with actual data
* Expose progress on `/collections/{name}/optimizations` endpoint
* make jwt with payload filter fail
* remove `whole` access requirement
* remove unnecessary mut
* fmt
* adjust test
* fix test
* remove more tests
* fix test
* fix test again
* fix test again
* Replace deprecated PayloadConstraint with JSON Value placeholder
* Fix expect message
---------
Co-authored-by: timvisee <tim@visee.me>
* allow to read related issues with collection-level access
* refactor to use requirements
* patch test
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* gRPC API for Distance Matrix
* generate API docs
* apply correct limit to sample
* add JWT tests
* pluralitiy
* fix COO naming
* drops COO as it trips codespell and is redundant
* clarify docs
* rename to FacetRequestInternal
* add rest endpoint
* fix correctness by fetching the whole list of values
* fix mmap map index variant
Also removes test for sorted output, for now
* add ytt spec
* fix clippy
* use hashmap inside of local shard
* rename operation to `facet`, add access test
* whitelist endpoint
* change api
* make limit optional
* universal-query: Grouping REST API
* add API docs
* bump API count
* add jwt validation test
* fix API
* better test
* stay on CoreSearchRequest where possible
* use existing scoring helper
* push broken test to illustrate issue
* track best score per point_id to enable sort on payload
* add example test for query discover with groups
* track ScoredPoint instead of PointId
* add default values for smoother ux
* universal-query: lookup_from implementation
* pushin test demonstrating broken equivalence
* more tests and fixes
* validate fusion not use with 'using' field
* cleanup collection info propagation
* do not clone the query in CollectionQueryResolveRequest
* cleanup
* address last code review
* separate access tests from validation tests
* bring changes from rbac-integration-tests review fixes
* bring changes from rbac-payload-access-tests
* update from base branch
* use const instead of magic number
* move COLL_NAME
* force usage of kill_all_processes()
* add payload constraint spec
* test that payload constraint filters queries
* fmt
* make rest check only for 403 status
* move file to own folder
* improve error message
* update spec for collection cluster info
* fix: use `.get()` instead of direct access
* use const instead of magic number
qdrant-cloud-bot