mirror of
https://github.com/qdrant/qdrant.git
synced 2026-09-21 13:37:46 -05:00
edge-docs-diff
547
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
0ff1c4e2b7 |
Send the queue proxy batch as a pre-encoded gRPC body (#10617)
* Send the queue proxy batch as a pre-encoded gRPC body The parent commits moved the WAL read, the operation clone and the request build off the async runtime. Two passes over the batch were left on it. Measured cost of each synchronous pass over a 26.2 MiB send batch (800 ops x 30 points x 256 dims), release build: pass 1 clone WAL operations 72.4 ms moved by the parent commits pass 2 build gRPC request 22.2 ms moved by the parent commits pass 3 clone request to send 65.7 ms on the async runtime pass 4 protobuf encode (tonic) 41.4 ms on the async runtime Pass 3 is there because `with_points_client` takes `impl Fn` and the channel pool calls that closure once per attempt, so each attempt needs its own owned message. Pass 4 runs inside `poll_next`: for a unary call tonic encodes the whole message in a single synchronous `encode_item`, so a worker is blocked for the full 41 ms, once per attempt. Encoding the batch up front removes both. The generated client cannot take a pre-encoded body, `update_batch` is typed `impl IntoRequest<UpdateBatchInternal>`, but all it does is pick a codec and a path and call `Grpc::unary`, and we already build the client ourselves from a pooled channel. `update_batch_pre_encoded` does the same three things with a codec that writes the encoded bytes through and decodes the response with prost. What stays on the runtime is the copy of the encoded body into tonic's send buffer: 14.9 ms for 26.2 MiB under jemalloc, nearly all of it faulting in freshly mapped pages rather than the copy itself (0.9 ms when the allocator hands back warm pages). Retries share the same refcounted bytes instead of cloning and re-encoding, so they drop with it. on the runtime before 107.2 ms per attempt on the runtime after 14.9 ms per attempt Bypassing the generated client means the RPC path and message types no longer follow the proto automatically, so a test checks them against the compiled descriptor set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZjWYpKeYdGpKiLeEZU9oc * Hand the pre-encoded update batch a configured Grpc, take the service name from the generated code `update_batch_pre_encoded` took a bare channel plus a `max_decoding_message_size` argument, and its only caller passed `usize::MAX`. Every other internal client applies that limit inside its `with_*_client` helper, so do the same: `with_grpc` hands out the `tonic::client::Grpc` the generated clients wrap, already configured, and the argument goes away. The service half of the RPC identity now comes from the generated `points_internal_server::SERVICE_NAME` instead of a second literal. Only the method name and the path literal remain hand-written, still pinned to the descriptor set by the test. `PreEncodedMessage::encode` uses `encode_to_vec`: one pass instead of a separate `encoded_len` call, and no `expect`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Adapt pre-encoding to the build/forward split from #10599 `forward_update_batch` takes `Arc<UpdateBatchInternal>` and encodes it once on the blocking pool, so the channel pool's attempts share the bytes. The queue proxy keeps the built request in that `Arc` across `BATCH_RETRIES` and for the per-operation isolation path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: generall <andrey@qdrant.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Fable 5.1
|
95eb3c3f79 |
Support cached id tracker memory placement (#10598)
Thread a `Populate` through the disk-resident id tracker's open paths (`DiskMappingReader`, `DiskIdTracker::open`, `ReadOnlyDiskIdTracker`, `ReadOnlyIdTrackerEnum`) so a `cached` placement primes the page cache with the mapping files on load instead of leaving them to page in on demand. The populate is derived from the segment config's placement at load time, clamped by low-memory mode, in both the writable segment open and the read-only one. The update-only lookup path keeps its transfer-nothing policy, and the build-time open stays cold: the built segment is reloaded anyway. `cached` is no longer rejected by validation. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Fable 5.1
|
81bb80a5d3 |
Expose id tracker memory placement in collection config (#10597)
Add `id_tracker: { memory: cold | pinned }` to CollectionParams,
CollectionParamsDiff and CreateCollection (REST + gRPC `IdTrackerParams`),
mirroring `payload: { memory }`. `cold` builds the disk-resident id tracker,
`pinned` the in-RAM immutable one. Unset keeps the current behavior: the
`serverless_compatible` feature flag decides.
The requested placement is persisted as an optional `id_tracker_memory` on
SegmentConfig (skipped when unset, so existing configs are unchanged); the
segment builder resolves it through `SegmentConfig::id_tracker_memory_placement`
instead of reading the feature flag directly.
The config mismatch optimizer rebuilds non-appendable segments whose effective
placement differs from the requested one. Appendable segments are skipped: they
always use the mutable tracker and get the current config when indexed.
`cached` is rejected by validation: the disk mapping reader has no
populate-on-open path.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
|
||
|
陈志谦
|
f0019e9bad |
[AI] docs: fix typos and duplicated words across config help and rustdoc (#10484)
- 'mising' -> 'missing' in .github/review-rules.md (flagged by the
repo's own codespell config)
- 'Do not create segments larger this size' -> 'larger than this size'
in config.yaml, the optimizer builder/diff sources and the grpc
proto + generated rust comment
- 'bigger then' -> 'bigger than' in the query scorer rustdoc
- three duplicated-word rustdoc fixes ('override in in', 'any of of',
'and and')
|
||
|
Kumar ShivenduandClaude Opus 5
|
83fe47c90a |
feat: add a dedicated min operator to score formulas (#10296)
Follow-up to #10287, which added `max`. Expressing a minimum still required spelling out `(a + b - |a - b|) / 2`, the sign flip of the max identity — drop the `neg` and you silently get a maximum instead. It also only works for two operands and mentions each one twice, so the scorer walks every sub-tree twice per candidate point. The pair is what makes clamping expressible: {"max": [0.0, {"min": [1.0, "$score"]}]} `min` mirrors `max` throughout, and both guard helpers introduced in #10287 already took an `operator: &str`, so they are reused unchanged: an empty operand list is rejected at parse time rather than folding to +infinity, and the Edge FFI rejects it at construction time. The result needs no `is_finite` check, since `min` cannot produce a non-finite value from finite inputs. The unindexed-field walker shares one arm for `Max | Min` as the bodies are identical, with a test pinning `min` separately so a later split cannot silently drop it. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Kumar ShivenduandClaude Opus 5
|
e2d42462fa |
feat: add a dedicated max operator to score formulas (#10287)
* feat: add a dedicated max operator to score formulas
Expressing a maximum in a score formula required spelling out the
arithmetic identity `(a + b + |a - b|) / 2`. That is easy to get wrong
(the `/ 2` is load-bearing), only works for two operands, and mentions
each operand twice, so the scorer evaluates every sub-tree twice per
candidate point.
`max` is variadic, mirroring `sum` and `mult`:
{"max": ["$score", {"mult": [0.5, "popularity"]}]}
Unlike `sum` and `mult`, `max` has no identity element for the empty
case, so an empty operand list is rejected at parse time rather than
folding to -infinity and scoring every point with a non-finite value.
The check lives in `ExpressionInternal::parse_and_convert`, which every
entry point passes through, and the Edge FFI additionally rejects it at
construction time to match how that crate validates elsewhere.
The result needs no `is_finite` check: unlike `log10`, `exp`, `div`,
`sqrt` and `pow`, `max` cannot produce a non-finite value from finite
inputs, so it follows the existing `sum` convention.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test: cover max error propagation and datetime operands
An operand that fails must fail the whole expression rather than being
passed over in favour of a finite sibling. Covered with the failure both
before and after the finite operand: `mult` short-circuits on zero and
so can skip evaluating later operands, and this pins down that `max`
must not grow a similar shortcut that would swallow an error.
Also covers `max` over datetime operands, which reach the scorer through
a separate conversion to seconds, so that "score by whichever timestamp
is newer" is verified rather than assumed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Tim ViséeandRoman Titov
|
ec3589f8b2 |
Reject .. in collection names (#10242)
* Reject dot segments in collection names Collection names are used as directory components on disk (storage/collections/<name>, snapshots/<name>). Apply the same plain-file-name predicate already used for snapshot names, so names like ".." and "." no longer resolve outside the parent directory. Applied to the legacy validator too: dot segments were never usable as directory names, so no existing collection can be locked out by this. * Fix collection name validation test on Windows, backslash disallowed * Apply suggestions from code review Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> --------- Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> |
||
|
Andrey VasnetsovandClaude Fable 5
|
e32d3fbf89 |
Add acosh expression to formula query (#10231)
Unary inverse hyperbolic cosine, parallel to sqrt/ln/exp/log10, in REST, gRPC, and edge (FFI + Python) interfaces. Inputs below 1 produce the same NonFiniteNumber error as an invalid sqrt or ln. Closes #10186 Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ivan PleshkovandClaude Opus 5
|
86b9330628 |
transfer: send raw payloads, behind feature flags (#10066)
A raw point can carry its payload as the byte blob it is stored as, mirroring `PointStructRaw.raw_payload` on the internal gRPC API. The blob travels from the sending node into the receiving node's WAL untouched, so the sender never parses the payload it read and neither node builds a protobuf value tree for it. It is parsed exactly once, where the operation is unpacked for apply (`process_point_operation`), because that is the first place the parsed form is actually needed: `set_full_payload` goes through the payload index, which cannot be updated from bytes. The gRPC boundary therefore only checks the encoding tag and rejects a point that sets both payload fields, the way the enclosing request already rejects both `points` and `raw_points`. Moving the parse onto the apply path makes its error classification load-bearing, so a malformed blob is reported as `OperationError::MalformedPayloadBlob` — the payload sibling of `MalformedVectorBlob`, mapped to `CollectionError::BadInput` for the same reason: a bad blob that reached the WAL has to be skipped on replay instead of crash-looping recovery. Three consequences of the blob living that long are handled explicitly rather than by convention: - `decode_payload_raw` takes the blob only once it has parsed, so a failure leaves the point holding it instead of holding neither representation. - `upsert_points_raw` and `sync_points_raw` refuse a point that still carries a blob. They read the parsed payload, so such a point would otherwise be stored with no payload at all, and a `debug_assert!` would not catch it in release. - `is_equal_to` compares blob to stored blob as bytes. A differing encoding costs a redundant upsert on sync, never a skipped one. The `raw_payload_transfer` bench measures the trade, per 100-point batch (one transfer batch) at payloads of ~200 B / ~700 B / ~7 KB: - Sender, storage bytes to wire: 16x / 37x / 113x faster. This is where the whole win is — no parse of the blob that was read, no value tree built. - WAL encode: 5x / 11x / 25x faster, writing a byte string instead of a map. - Receiver, wire to applicable point: 1.09x / 1.10x / 1.06x. Near neutral, as it swaps walking a prost value tree for a JSON parse. - Wire bytes: ~6% smaller. WAL bytes: 10-32% *larger*, because the blob is JSON while a parsed payload is written as a compact CBOR map. The WAL growth is accepted rather than fixed: decoding earlier to win those bytes back costs a second full deserialization, and would leave the receiving side with a `payload_raw` that is never populated. Making the blob itself compact belongs in the payload storage encoding (`RawPayloadEncoding` is the extension point for it), not here. Two flags, both off by default and both sender-only (nodes accept raw points and raw payloads regardless), read where the transfer batch is prepared: - `transfer_raw_points` transfers every collection as raw points, not only those whose vector storage would drift in a decode-encode round-trip. - `transfer_raw_payloads` ships the blob a raw read hands out; without it the prepared batch decodes it back into the parsed payload, and the wire message is exactly what it is today. Neither is enabled by `all`: a node only accepts them once it runs a version that understands them, so they can only be switched on a release later. Nothing enforces that yet — the transfer has no peer-version gate. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
GuTS805
|
ad58f4b9e7 |
fix(api): reject empty dense vectors in gRPC Vector::Dense validation (#10143)
Sparse and MultiDense variants already reject empty vectors via validate_sparse_vector_impl / validate_multi_vector_by_length, but Dense always returned Ok(()). Reuse the existing validate_non_empty_dense helper from the REST validation module for consistency. |
||
|
|
0a6cb3b4cf |
[Raw payloads]: read payload as stored bytes in retrieve_raw (#10040)
* segment: read payload as stored in retrieve_raw `retrieve_raw` already hands back vectors as stored; let the caller ask for the payload the same way, so a reader that only relocates a point parses nothing. `RawPayloadFormat` states what the caller wants — no payload, parsed, or as stored — and replaces the `WithPayload` argument, which could express a key selection that a raw read cannot serve anyway. [`MaybeRawPayload`] states what came back, which can differ from the request in one direction only: a payload storage that keeps payloads parsed cannot answer `Raw` with a blob, and now says so instead of encoding a payload for a reader that would parse it straight back. The raw path reaches the blobstore through `read_payloads_maybe_raw`, mirroring `read_payloads` down the payload storage and payload index traits, so it keeps the batched read. Every caller asks for `Parsed`, so this changes no behaviour: the copy-on-write move and the sync comparison need the parsed payload anyway, and the shard transfer switches over with the feature flag that ships the blob to another node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * segment: always hand out the stored payload blob from retrieve_raw Review follow-up: instead of telling `retrieve_raw` in which form to return the payload, it always returns it as stored and a caller that needs the parsed form decodes it itself. - Drop `RawPayloadFormat` and the payload parameter it replaced: no production caller ever asked for anything but the whole payload, and a selector cannot be applied to an opaque blob anyway. - Drop `MaybeRawPayload` / `MaybeRawPayloadRef`: only `InMemoryPayloadStorage` could produce the parsed variant, and no segment can be built with that storage (`PayloadStorageType` is `Mmap` or `InRamMmap`, both blobstore-backed). `SegmentRecordRaw` carries a plain `Option<RawPayload>`. - `PayloadStorageRead::read_payloads_maybe_raw` becomes `read_payloads_raw` and hands out `Option<&[u8]>`. The in-memory storage keeps payloads parsed, so it encodes on read, producing the bytes an on-disk storage would have written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * api: decode a received raw payload with the shared decoder `decode_payload` at the gRPC boundary matched on the encoding and parsed the blob itself, duplicating `RawPayload::decode`. Add the inbound conversion from the wire type and let the one decoder do the reading, so another encoding has a single place to be taught. The conversion also rejects an encoding number no variant maps to, which prost would otherwise hand out as the default encoding — a blob from a node that writes payloads some other way must not be read as JSON. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * simplification --------- Co-authored-by: Ivan Pleshkov <ivan.pleshkov@qdrant.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
0805d3f422 |
Add /profiler/consensus_lag to measure apply lag between peers (#10090)
* Add /profiler/consensus_lag to measure apply lag between peers Raft commit index advances on a peer whose apply loop is stalled, so the existing signals - `raft_info.commit` and the `all_nodes_have_same_commit` test helper - report a stuck peer as healthy. Nothing exposes how long a peer has been behind at *applying* entries, which is what shard transfer's `await_consensus_sync` barrier actually waits on. Each peer now keeps a ring of the last 32 entries it applied, stamped with its own wall clock and the time that entry took to apply. The ring is in memory on ConsensusManager, not in Persistent, so the on-disk format is untouched. `/profiler/consensus_lag` collects those rings from every peer over a new internal RPC and lines them up on the entry indices they share. Each entry is measured from whichever peer applied it first, so a lag is never negative; the peer that is first can differ per entry, so the baseline is per entry rather than a single chosen peer. Entries only one peer still remembers are excluded, otherwise a peer would be measured against itself. A peer stalled part-way through an entry keeps healthy lag statistics - everything it did apply, it applied on time - so the report carries `behind_entries` and `newest_applied_age_ms` alongside, which is what actually exposes the stall. Peers that fail or time out are listed rather than failing the request: a partial answer is more useful than none when the point is to find a peer that stopped answering. The endpoint follows `/profiler/slow_requests`: manage access, and outside OpenAPI, so no endpoint-count or ACTION_ACCESS guard applies. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Move applied-entry log into its own module Keeps the new code out of files that are already large. The ring, its entry type and the snapshot served over RPC move to `content_manager/consensus/applied_log.rs`, alongside the other consensus internals; `ConsensusManager` is left with a field, an accessor and the one `record` call in the apply loop. The grpc encoding moves next to the decoding it mirrors, in `common/consensus_lag.rs`, leaving the internal service handler three lines instead of thirty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Test that a consensus stall is still in the report after the peer catches up * Take each peer's applied index from consensus state, not its ring --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: tellet-q <elena.dubrovina@qdrant.com> |
||
|
Andrey VasnetsovandClaude Opus 5
|
aa6c5d8403 |
Global quota API (#10035)
* feat: global quota API Memory and disk are node-wide resources, so configuring their thresholds per collection through strict mode makes little sense. Move them behind a single cluster-wide `QuotaManager`. The quota config is seeded from `storage.quotas` in the settings (and so from env vars), overridden by `quota.json` in the storage directory, and updated cluster-wide through a new `SetQuotaConfig` consensus operation which rewrites that file on every peer. Raft snapshots carry it too, so a peer that joins by snapshot picks it up. Quotas are enforced wherever the strict mode memory and disk checks used to run, but no longer gated behind `strict_mode.enabled`: a value set in an enabled strict mode config still wins per resource, the quota is the default. Rejections name both the condition that tripped and the config that governs it. `GET /quotas` reports the config plus current utilization to global read users; `PUT /quotas` replaces it for global manage users. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: cover the quota endpoints in the API consistency checks `test_all_rest_endpoints_are_covered` and the OpenAPI endpoint count both break on any new REST endpoint. Add `GET`/`PUT /quotas` to `ACTION_ACCESS` with their JWT access tests, and bump the expected API count. The quota endpoints stay out of `REST_ENDPOINT_WHITELIST`: that list is for data-plane endpoints reported per-endpoint in metrics. Also add a Raft snapshot CBOR compatibility test — snapshots are exchanged between peers of different versions during a rolling upgrade, so `quota_config` must be absent-tolerant in both directions. Review feedback: persist through `SaveOnDisk`, which already implements the write-before-swap protocol this was doing by hand; validate the config at both persistence boundaries, since a hand-edited quota file or a config arriving through consensus does not pass the REST handler's validation, and a `0%` limit would reject every update forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: assert seeding a quota from invalid settings persists nothing Follow-up to review feedback claiming `SaveOnDisk::load_or_init` writes the init value before it is validated. It does not — only `SaveOnDisk::new` persists — but the property matters: were seeding to persist first, invalid settings would leave a `quota.json` that fails validation on every subsequent start, and the node could only be recovered by deleting it by hand. Pin it down with a test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: make QuotaManager the single reader of memory and disk The quota checks measured memory and disk themselves, while the optimizer and the WAL disk watcher each called `fs4::available_space` behind their own ad-hoc caches. Fold all of it into QuotaManager: it owns the readings, the freshness policy, and the limits they are compared against. Moves the module to `lib/shard`, since the optimizer sits below `storage` and has to reach it; `storage::quota` re-exports it, so consensus, the `/quotas` API and StorageConfig are unchanged. The manager is installed as a process singleton by TableOfContent, ahead of loading any collection. - Callers hand in QuotaLimits overrides instead of a StrictModeConfig, and an override can now only tighten. A collection-level admin could raise `max_disk_usage_percent` past a cluster-wide limit that needed global manage rights to set; ties resolve to the quota so the rejection names the knob that actually has to change. - Measurements are cached for 5s, but a reading at or above its limit is never reused: a rejected client retries, and freeing the resource has to take effect on the next request rather than a TTL later. - `fits_on_disk` sizes an optimization against physical free space only, never the configured limits. Optimizations are what free a full disk, so the quota must not be what stops one. - `percent_of` widens to u128 instead of saturating the multiply, which under-reported utilization (failing open) above ~184 PB. - StorageConfig::quotas is optional; absent means no quota is enforced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: don't recover dead replicas onto a node at a resource limit Recovering a dead replica pulls a whole copy of its shard onto this node. If it is already at its memory or disk quota that transfer cannot finish, and starting it only pushes the node further past the limit. Skip it and reconsider on a later sync, once the resource frees up. Adds QuotaManager::check_capacity for work that lands bytes here without being an update. Unlike fits_on_disk the configured limits do apply: taking on a replica is not what frees a full node, so there is no deadlock to avoid by letting it through. The check is hoisted out of the per-shard loop because a node over its limit re-measures on every call, so checking per dead shard would cost a statvfs each. It is free when no quota is configured. Also trims the comments across the quota module, which had grown well past what the code needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: drop trivial and duplicated quota tests Six tests removed, ~140 lines, with no loss of coverage: - a_rejection_names_the_knob_that_has_to_change asserted that a format! contains its own literals; the message is covered end-to-end by the override test and by test_global_quota.py. - a_node_over_its_quota_has_no_capacity_to_take_on_a_replica was 30 lines for check_capacity, a one-line delegation to check_update the test above it already calls. - a_rejecting_measurement_is_never_served_from_the_cache duplicated the meter test, which proves the same rule with an injected reader instead of inferring it from the real filesystem. - free_space_is_reported_without_enforcing_anything covered a one-line accessor, and its point is what the fits_on_disk test is for. - The two resolve tests and the three meter tests each collapse into one. DiskFit::Unknown keeps its coverage as two lines inside the fits_on_disk test rather than its own fixture. The snapshot compat pair becomes one test: the second only asserted cluster_metadata.is_empty(), which says nothing about quotas — the real check was the deserialize, now an expect that states it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: re-measure free space as the disk fills, and drop a Windows-only assert Two CI failures, both from this branch. e2e test_low_disk: the DiskUsageWatcher I replaced escalated to checking on every call once free space fell below 512 MB. Folding it into the quota manager lost that — available_bytes passed no limit, so a reading was reused for the full 5s however little space was left. On a disk filling as fast as that test fills it, 5s blind is enough to actually run out and the WAL write dies instead of returning "No space left on device". available_bytes now takes a watch_below level and never reuses a reading under it, which is what the old ladder was expressing. The watcher passes max(min_free, 512 MB), so the escalation point is back; above it the 5s cache still costs fewer syscalls than the old 128-call ladder. fits_on_disk gets the same rule by passing required_bytes, so a merge that does not fit re-checks rather than sitting on a stale sample. Windows: fits_on_disk on a missing path was asserted to be Unknown, but GetDiskFreeSpaceEx resolves up to the containing drive and succeeds — as common::disk_usage's own test documents. Dropped; the branch is a two-line else and is not portably reachable. Also renames an_optimization_is_sized_against_the_disk_not_the_quota, which needed explaining to be understood. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: report the quota config in telemetry Reads it from the quota manager rather than the settings, so it is the config the node is actually enforcing: a peer that missed a consensus update reports what it is applying, not what the cluster agreed on. Gated on global access, the same access `GET /quotas` requires, and left out of `PeerTelemetry` — a quota is per-node state, so each peer reports its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: regenerate OpenAPI, and cover the quota in the telemetry key sets Two CI failures from the previous commit. Referencing QuotaConfig from TelemetryData moves its definition earlier in `components/schemas`, because TelemetryData is generated ahead of QuotaStatus. Regenerated rather than hand-patched, so the schema is a pure move. test_telemetry_detail asserts the exact set of top-level telemetry keys. The quota is reported at every level, including 0 — it is three scalars, it is the default the endpoint serves, and it is what explains an update being rejected — so both key sets gain it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: remove `max_disk_usage_percent` from strict mode Disk is a node-wide resource, so a per-collection percentage of it never meant anything a caller could act on: the limit describes how full the *node* is, and which collection the write happens to target has nothing to do with it. The global quota is where it belongs. It shipped in 1.18.2 without documentation, so this drops it outright rather than deprecating. Removal is soft in every direction: StrictModeConfig has no `deny_unknown_fields`, so a client still sending it gets it ignored rather than a 400, and the same struct deserializes the persisted collection config, so collections created on 1.18.2+ keep loading. Proto field 22 is reserved so the number is never reused. The e2e test becomes a quota test — the fixture and the timing are the interesting parts and they carry over unchanged; only how the threshold is configured differs. `max_resident_memory_percent` was documented and stays for now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: enforce the strict mode memory limit outside the quota `max_resident_memory_percent` was folded into the quota as an override, which meant the quota check had to know about strict mode, and retiring the setting would mean unpicking `EffectiveLimit` and `LimitSource` from the resolution logic. It is now a check of its own in `verification/mod.rs`, next to the strict mode checks it belongs with, borrowing only the measurement from the quota manager — which stays the node's single reader of process memory, so both checks still share one reading. Deleting the setting later is deleting one function and its one caller. `QuotaManager::check_update` takes no arguments and consults the quota alone. A collection can still only tighten the limit for itself, because its own check runs in addition rather than in place of the quota's, and each rejection now names the config that has to change without having to carry a `LimitSource` to say so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: enforce the quota on the update path, not in strict mode The quota check sat inside `check_strict_mode_toc_batch` only because that was the one place holding the collection's strict mode config. It doesn't need one any more, and the placement had a real cost: coverage depended on each handler remembering to ask for a strict mode check, and four of the internal update RPCs do — `sync_internal`, which moves the most bytes onto a node, does not. It now runs in `Collection::update_from_client` and `update_from_peer`, which every update passes through. `update_from_client` checks ahead of the shard split, so an operation is accepted or refused whole rather than landing on some shards and being refused by others. Classification moves with it, from ~10 `consumes_memory` impls on request DTOs to one exhaustive `CollectionUpdateOperations::consumes_quota`. The internal enum has variants — raw upserts, conditional upserts, the syncs — that have no client-facing request type, so per-DTO impls structurally could not classify them. Shard-transfer syncs stay excluded, as they are today: a transfer is sized up once before it starts, and refusing its batches partway abandons work that is nearly done only for it to restart from the beginning. Index and named-vector creation reach shards through consensus, past this check — a peer must not refuse what the cluster agreed to — so they keep their pre-consensus check, now against the quota manager directly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: deprecate `max_resident_memory_percent` in strict mode Same reason the disk threshold went: memory is node-wide, so a per-collection percentage of it caps how full the *node* is, which has nothing to do with which collection is being written to. The node-wide quota caps it once for everything. Unlike the disk threshold this one shipped documented, in 1.18.0, so it keeps working — as a limit a collection can tighten for itself, never lift — and gets the usual markers: `#[deprecated]` on both Rust structs, `[deprecated = true]` on proto field 21, and `deprecated: true` in the OpenAPI schema, which schemars derives from the attribute. The note names 1.21 as the removal. Recording a version matters here: the audit in docs/plans/overdue-deprecations.md found that this repo has never written a removal deadline down, and members of the 1.15.0 deprecation batch are still in tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: reconcile the quota readers with #9891 #9891 landed effective (cgroup) figures in telemetry while this branch was making QuotaManager the single reader of memory and disk. Two collisions, neither of which git sees. `segment::utils::mem::total_memory_bytes` is now a shared accessor with a 5s TTL, so a cgroup resize is picked up. The quota module had its own `OnceLock` copy that froze the value at startup — exactly what #9891 set out to fix — so it delegates to the shared one instead. Telemetry's new `disk_size` called `common::disk_usage::disk_usage` directly. That reader lost its TTL cache on this branch when the caching moved into the quota manager's meter, so it would have taken an uncached `statvfs` on every telemetry request, and it put a second disk reader back in the tree. It goes through `QuotaManager::disk_capacity_bytes` now, sharing the reading the quota check already takes. Verified it still reports the storage filesystem, matching `df`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: split the quota manager by what each half does `manager.rs` had grown to 450 lines holding three separate jobs: owning the config and its file, taking the readings, and comparing one against the other. - `manager/store.rs` — the `Store` enum, `QUOTA_CONFIG_FILE`, and config validation, which is now the store's own business rather than something every caller has to remember to do first. - `manager/measure.rs` — every reading, and `DiskFit`. The "nothing else calls `statvfs` or reads process RSS" claim is now checkable by looking at one file. - `manager/enforce.rs` — `check_update` / `check_capacity` and the threshold comparison. - `manager/mod.rs` — the struct, its construction, and the config accessors: what a reader needs to see first. Tests move with their subject. No behaviour change: `set_config` used to validate before delegating to the store, and now the store validates on write, which is the same order of operations from the outside. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: count copy-on-write deletes toward the quota Dropping a vector or a payload key does not free anything on its own: copy-on-write rewrites the point to produce the version without that field, so storage grows first and is only reclaimed once the optimizer gets to it. Gating those as if they were reclaiming space let a full node keep taking writes that make it fuller. Deleting whole points stays exempt. That is the one operation that has to work on a node at its limit, or there is no way back under it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: say that the quota's reported usage is per node `GET /quotas` returns one cluster-wide config and one set of utilization figures, which reads as though both describe the cluster. They do not: memory and disk are node-local, so `usage` is whatever the peer that served the request is seeing, and a peer under its limit says nothing about the others. Also corrects `resident_memory_percent`, which claimed to be a share of total system memory. It is a share of the memory available to the process, which under a cgroup is the limit rather than the host's RAM. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: treat a node over its quota as a failed replica, not a bad request A quota rejection described the request as invalid (400) and was classified non-transient, which is how the replica set recognises errors that every replica would produce alike. A quota is the opposite: the input is fine and the answer depends on which machine you ask. On the default `wait=false` path that combination silently dropped the write — `update.rs` only deactivates transient failures when nothing completed — leaving the replica Active and permanently missing data its co-replicas had. It is now `InsufficientStorage`, transient, HTTP 507 / gRPC `ResourceExhausted`. So a node that is out of room is handled like one that is offline: - last active replica, or every replica over quota: nothing could take the write, and the client is told the cluster is out of room. - more than one replica: the full node is deactivated through the same path a dead peer takes, and the update stands if enough replicas accepted it. `check_capacity` already keeps recovery off that node until it has room. The check also moves off `update_from_client`, which applied the coordinator's own limit to the whole operation even when it held no replica of the shards being written. Each replica set now gates its own local write and records the refusal as a failure of this peer, so a node only ever answers for itself. `ResourceExhausted` is shared with rate limiting, and the reverse conversion mapped it straight to `RateLimitExceeded` — a forwarded rejection came back as 429. Statuses now carry a marker so the two stay distinguishable across the wire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: report quota pressure per node, and across the cluster A quota is node-local, so finding out which node has hit one meant asking each of them in turn — and nothing at all showed up in monitoring. `/metrics` gains a `quota_exceeded` gauge for the local node. It is emitted only while the quota is enabled: with it off the value would be a constant 0 that says nothing about the node, and an alert built on it would go quiet rather than fire if someone disabled the quota. Telemetry's `quota` field carries the same verdict alongside the config, since that is where the metric is derived from. `GET /quotas` now answers for the whole cluster. A new `GetQuotaUsage` RPC on the internal `QdrantInternal` service returns what one peer is using, and the handler fans it out to every known peer in parallel. Peers that do not answer are left out rather than failing the request — the nodes that are out of room are exactly the ones most likely to time out, and a partial answer still names them. Outside distributed mode the field is absent rather than a map of one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: report the quota metric per resource `quota_exceeded` was one flag for the whole node, which does not say what to go and fix — disk is freed by deleting or optimizing, memory by unloading. It now carries a `resource` label: quota_exceeded{resource="memory"} 0 quota_exceeded{resource="disk"} 1 A resource with no limit gets no series at all, for the same reason the metric is absent while the quota is disabled: a series that can never reach 1 reads as healthy and would quietly carry an alert that cannot fire. `QuotaManager::exceeded` returns the per-resource verdict, with `None` for a resource this node does not cap. Telemetry reports the same breakdown, since the metric is derived from it. The peer usage RPC keeps a single flag — it sits next to both percentages, so it only has to answer "is this peer refusing writes". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: drop a no-op error conversion the linter caught `check_global_access` already returns a `StorageError`, so mapping it through `StorageError::from` converted the type to itself and tripped `clippy::useless_conversion`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: hold a tripped quota until usage clears a release margin A resource resting on its limit crosses it in both directions on the noise between two readings, and each crossing is expensive: the node refuses a write, its replica is deactivated, usage dips, recovery starts sending a whole shard copy back, and the arriving data pushes it over again. The loop sustains itself, and every lap costs a shard transfer. A limit now trips at its configured value but only clears once usage has fallen 5 percentage points below it, so the crossing has to be real. The margin is floored at 1%, since a limit smaller than the margin would otherwise be impossible to fall back under and would strand the node. The verdict is carried on the manager rather than recomputed, which makes it the thing reporting shows: expect `exceeded` to be set while the utilization next to it is already back under the limit. Rejections say so too, rather than claiming a limit that is no longer exceeded: Disk usage is at 87% of total capacity. It reached the configured limit of 90% and has to fall below 85% before this node takes writes again. Changing the config clears the verdicts. New limits are a deliberate act, and should not be held back by the margin of a limit that no longer exists. Both resources are now evaluated on every check instead of stopping at the first failure, so a verdict is never left behind reporting a reading that has since been superseded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: make the quota release margin configurable 5 points is a guess about how noisy a deployment's usage is, which is not something one number can be right about: a node whose disk moves in gigabyte steps needs a wider margin than one that creeps, and an operator who wants the old flip-on-every-reading behaviour should be able to ask for it. `release_margin_percent` joins the rest of the quota config, so it seeds from `QDRANT__STORAGE__QUOTAS__RELEASE_MARGIN_PERCENT`, replicates through consensus, and changes with `PUT /quotas`. Defaults to 5 and is filled in when a request omits it, so it always answers with the margin actually in force rather than leaving the caller to assume one. `0` releases as soon as usage is back under the limit. `QuotaConfig` grows a hand-written `Default` for it, since deriving one would have quietly defaulted the margin to 0 and disabled the hysteresis for anyone constructing a config in code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: leave the release margin unset by default, and hold verdicts in atomics `release_margin_percent` is `null` unless someone sets it, rather than materialising 5 into every config. A quota written today then does not pin a number a later release may want to revise, and `{"enabled": false}` still round-trips as itself. `QuotaConfig::limits` resolves it, next to `enabled`, so enforcement never sees the unset case. The verdicts move from a `Mutex<QuotaExceeded>` to one `AtomicBool` per resource. They are judged independently and nothing reads them as a pair, so the lock only added contention to the path every update takes; a verdict that races a concurrent check is re-decided by the next one from a fresh reading. That also drops the tri-state. Only "was this over its limit" has to survive between checks — whether a resource is enforced at all follows from the config and the reading, so it is derived when reporting rather than stored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: drop a comment arguing with a design that was never here Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jojii
|
ec1aec5f4c |
Payload bytes gRPC (#9949)
* Prepare GRPC for raw payload bytes * Make RawPayload a separate protobuf message |
||
|
|
8a7325ad5d |
Remove deprecated search endpoints from OpenAPI, deprecate them in gRPC (#9982)
* Remove deprecated search/recommend/discover endpoints from OpenAPI Remove deprecated REST API endpoint definitions from the OpenAPI generator. These endpoints were deprecated in v1.13.3 (`f4ced2567`, #5907, 2025-01-30) in favor of the universal `/points/query` endpoint: - POST /points/search - POST /points/search/batch - POST /points/search/groups - POST /points/recommend - POST /points/recommend/batch - POST /points/recommend/groups - POST /points/discover - POST /points/discover/batch Also removes the corresponding request types from the schema generator and updates the expected API count in the consistency check. Co-authored-by: Cursor <cursoragent@cursor.com> * Migrate OpenAPI integration tests to /points/query The deprecated /points/search, /points/recommend and /points/discover endpoints (along with their /batch and /groups variants) were removed from the OpenAPI spec, which caused validation failures in the Python integration test harness. This commit migrates the affected tests to the universal /points/query endpoint: - Delete tests dedicated to the deprecated endpoints: test_recommend.py, test_discover.py, test_multicollection_reco.py, test_recommendation_multivector.py - Refactor remaining tests to call /points/query (and /query/batch, /query/groups), translating request bodies (vector -> query / using, positive/negative -> query.recommend, target/context -> query.discover) and unwrapping the new result.points response shape. - Drop equivalence assertions against the now-removed legacy endpoints. Co-authored-by: Cursor <cursoragent@cursor.com> * Relax non-empty assertions in migrated recommend/discover tests The previous migration added `len(...) > 0` assertions to tests that previously only checked equivalence between the deprecated and new API. These assertions are too strict because the parametrized `query_filter` cases legitimately produce empty result sets. Drop the `> 0` assertion and rely on `request_with_validation` to verify the response is well-formed and HTTP OK. Co-authored-by: Cursor <cursoragent@cursor.com> * Migrate remaining OpenAPI tests off deprecated search endpoints Tests added to dev after the original migration was written still call /points/search and /points/recommend/groups through `request_with_validation`, which resolves the endpoint against the OpenAPI spec and therefore breaks once the endpoint is not in the spec: - test_turbo4_storage.py, test_sparse_idf_corpus.py, test_validation.py: translate /points/search to /points/query (vector{name,vector} -> query + using, result -> result.points). - test_group.py: drop the /points/recommend/groups half of the lookup_from validation test in favour of the query equivalent. test_sparse_idf_corpus.py's test_query_api_supports_idf_corpus goes away: with the helper on /points/query every test in the file now exercises what it asserted. Also record why test_recommend_group cannot assert on its groups: it uses every point in the collection as a recommend example, so all of them are excluded and the result is legitimately empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Regenerate openapi.json without the deprecated search endpoints Drops the 8 deprecated paths and the request schemas that only they referenced: Search/Recommend/Discover request (+Batch, +Groups) types and their exclusive dependencies (NamedVector, NamedSparseVector, NamedVectorStruct, UsingVector, RecommendExample, ContextExamplePair). Regenerated output is a strict subset of the previous spec, and every remaining $ref still resolves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Deprecate the search/recommend/discover RPCs in gRPC The REST counterparts have carried `deprecated: true` since v1.13.3 and are now gone from the OpenAPI spec, while the gRPC RPCs never got any deprecation annotation at all. Mark all 8 with `option deprecated = true` so generated clients warn, and point each doc comment at its `Query` replacement. tonic puts `#[deprecated]` on the generated client methods only; the server trait gets the doc comment alone, so our own `impl` is unaffected. The RPCs keep serving traffic — this is annotation only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Restore the deleted recommend/discover suites on /points/query The earlier migration deleted these four files outright, but the query-side tests it left behind are all shallow smoke tests (`len(result) > 0`, `"points" in result[0]`). The deleted ones carried invariants with no query-API equivalent anywhere, so deleting them was a real loss of coverage rather than de-duplication: - test_recommend.py: default strategy equals average_vector; batch results identical to sequential singles across six request shapes; best_score with only negatives yields all-negative scores; best_score with a single positive orders identically to a nearest query; raw vectors as examples equal ids as examples. - test_discover.py: context-only scores are all <= 0; target-only orders identically to a nearest query but scores differently; with a fixed context the integer part of the score is stable while the decimal part moves, and vice versa with a fixed target; batch equals singles; lookup_from by id equals by vector. - test_multicollection_reco.py: cross-collection lookup_from, plus wrong-vector-size, unknown-collection and unknown-vector rejections. - test_recommendation_multivector.py: the same recommend invariants over a max_sim multivector collection, which the query suite never covered. Only test_recommend_missing_lookup_from_collection_with_raw_vector is dropped as genuinely redundant — test_query.py's test_query_missing_lookup_from_collection covers query, query/batch and prefetch. Two request-shape differences the translation had to absorb: - Giving no examples at all is 422 (a RecommendInput validation rule), where the legacy API reported 400 from the query itself. A malformed example, such as an empty vector, is still 400. - DiscoverInput requires the `context` key and accepts only an explicit null to mean "no context", so target-only discover must spell it out. The legacy API let it be omitted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Andrey VasnetsovandClaude Fable 5
|
446d140c2d |
Slice filtering condition: sliced scroll / deterministic sampling (#9899)
* feat: slice filtering condition for sliced scroll and deterministic sampling
Add a `slice` filter condition selecting points where
`stable_hash(point_id) % total == index`. The hash is SipHash-2-4 with a
zero key over canonical id bytes (8 LE bytes for numeric ids, 16 RFC 4122
bytes for UUIDs) — a frozen public contract, independent of the internal
resharding ring hash, reproducible by clients to predict membership.
For a fixed `total`, slices are disjoint and cover all points, enabling
parallel scroll streams (ES sliced-scroll style) and reproducible sampling
that composes with any other filter condition.
- REST: `{"slice": {"total": N, "index": R}}`; gRPC: `SliceCondition` in
the condition oneof (tag 8)
- Evaluated per point via id_tracker external-id lookup; no payload index
needed; cardinality estimated as `points / total` with no primary clause
- `total >= 1` enforced by NonZeroU32 at parse time, `index < total` by
validation in both REST and gRPC paths
- Hash contract locked by test vectors independently reproduced with a
reference SipHash-2-4 implementation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* tests: minimal OpenAPI test for slice filter condition
Scrolls all slices of a fixed total over numeric + UUID ids asserting
disjointness and full coverage, checks must_not inversion, and pins the
two rejection paths (422 for index >= total, 400 for total = 0). Requests
and responses are validated against the regenerated OpenAPI spec by the
test harness.
Note: the spec cannot itself reject total = 0 client-side — the Condition
anyOf falls through to the permissive Filter schema, as with any invalid
condition — so rejection is asserted via the server response.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Ivan Pleshkov
|
9a20fc49e4 |
[TQDT] TQ roundtrip: raw vectors in grpc, WAL and apply internal operation (#9813)
* raw vector grpc apply are you happy fmt clean up are you happy clippy review remarks review remarks * fix after rebase |
||
|
|
1d4d6f02da |
Per-query IDF corpus for sparse vector search (#9661)
* Add per-query IDF corpus for sparse vector search
Let the caller choose, per query, which population sparse IDF statistics
are computed over. `params.idf` is either `"global"` (default, unchanged
behavior) or `{"corpus": <filter>}`, where the corpus filter is
independent of - and usually broader than - the retrieval filter.
Decoupling the two keeps the score scale stable when the retrieval
filter tightens: term importance is measured against a population the
user names, not against whatever subset the filter happens to select.
Design decisions:
- Corpus grammar is restricted to a conjunction (`must`) of `match`
conditions on payload fields; loosening later is backward compatible.
- Strict mode validates the corpus filter like a read filter
(unindexed fields rejected).
- `idf` on a vector without the IDF modifier is a validation error,
never silently ignored.
- An empty corpus yields degenerate but corpus-scoped scores (smoothed
IDF over N=0), never a fallback to global statistics - in multi-tenant
collections a fallback would leak term statistics across tenants.
Implementation:
- QueryContext IDF stats are keyed by corpus, so one batch can mix
requests with different corpora.
- Statistics come from the sparse index: df(term) is counted over the
query terms' posting lists only, never by scanning stored vectors.
Small corpora (under ~1/32 of the segment, by cardinality estimate)
are kept as a sorted id list galloping through posting lists via
skip_to; large ones as a dense membership mask filled streaming from
the filtered-points iterator. A misestimated small corpus degrades
into the mask.
- Exposed uniformly: REST (`params.idf`), gRPC (`IdfParams` message),
edge python bindings; OpenAPI schema regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply rustfmt
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix clippy manual_is_multiple_of in sparse IDF corpus test.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Allow any filter as IDF corpus
Drop the must+match grammar restriction on the corpus filter. A
restriction enforced only as a validation step over the full Filter
type buys nothing; if a narrower corpus syntax is ever wanted, it
should be a dedicated API-level type instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix build: add memory field to SparseIndexConfig in idf corpus test
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
|
5593bc5564 |
Add optional last-modified timestamp to ListedFile and CachedFs FileInfo (#9803)
* Add optional last-modified timestamp to ListedFile and CachedFs FileInfo Filled where the listing backend exposes one: local filesystems (entry metadata) and object stores (ObjectMeta::last_modified). The uio-grpc backend reports None since the RPC does not carry mtimes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Carry last-modified over the StorageRead ListFiles RPC Extend ListFilesEntry with an optional google.protobuf.Timestamp, fill it on the server from the listing metadata, and convert it back to SystemTime in the uio-grpc client. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Update lib/common/io_bridge_object_store/src/source.rs Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Fix missing SystemTime import in io_bridge_object_store The CodeRabbit-suggested last_modified mapping used SystemTime::from without importing std::time::SystemTime, breaking compile and CI. Co-authored-by: Cursor <cursoragent@cursor.com> * Retrigger CI after flaky integration-tests-consensus timeout The compile fix is in; the prior run failed on an unrelated 30s timeout in test_collection_recovery, not on PR changes. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Tim Visée
|
7d4f70bb19 |
Add time to gRPC responses (#9733)
* Add missing time response in some gRPC APIs, make consistent with REST * Don't use destructor |
||
|
|
ab0d3ecc62 |
Add unified memory: cold|cached|pinned placement parameter for collection components (#9684)
* Add unified `memory: cold|cached|pinned` placement parameter for collection components
Introduce a single `memory` parameter that controls how each collection
component's data is held in RAM, replacing the inconsistent zoo of
`on_disk` / `always_ram` / `on_disk_payload` flags:
- `cold`: not pre-loaded from disk, cached with usage
- `cached`: pre-populated into page cache on load, evictable under pressure
- `pinned`: materialized on heap, never evicted by cache pressure
The parameter is available on dense vectors, HNSW config, all quantization
configs, the sparse index, all payload field index types, and payload
storage (as a new `payload: { memory }` sub-object on collection params).
When set, it overrides the deprecated legacy flag; when unset, behavior is
unchanged. Legacy flags are marked deprecated (Rust + proto) but keep
working; conflicts are resolved in favor of `memory` with a warning.
New capabilities enabled by the tri-state model:
- HNSW graph links can be pinned (first production caller of the existing
`GraphLinksResidency::Pinned`)
- sparse mmap index, quantized vectors and on-disk payload field indexes
gain a `cached` tier (mmap + populate on open)
`pinned` is rejected by API validation for components without a heap
variant (dense vector storage, payload storage). Low-memory mode degrades
placements at load time via `Memory::clamp_to_low_memory`, matching the
existing `prefer_disk`/`skip_populate` behavior. Effective-placement
comparison in the config-mismatch optimizer avoids spurious rebuilds when
the same placement is expressed through the new parameter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix gpu-gated tests for the new `memory` field
CI clippy runs with --all-features, which compiles the gpu-gated tests
that were missed locally: add the `memory` field to config literals and
allow deprecated placement params, same as in the rest of the tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add OpenAPI tests for memory placement, keep sparse config downgrade-clean
- OpenAPI tests: create/update collections with `memory` on every component,
assert the parameters are echoed in collection info, assert legacy-only
collections expose no new fields, and assert `pinned` is rejected (422)
for dense vector storage and payload storage on both create and update.
- Persist only the explicitly requested `memory` parameter in
`sparse_index_config.json` instead of the legacy-resolved placement, so
configurations using only the deprecated `on_disk` flag keep byte-identical
files that older Qdrant versions load without unknown fields.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Validate collection meta ops at construction, not only in the API layer
The `memory: pinned` rejection for dense vectors and payload storage
lived in `Validate` impls on the internal request types, which only ran
through the REST actix extractor. gRPC validates just the proto message,
so a gRPC client could persist `pinned` where it is not supported and
have it silently treated as `cached`.
Run the derived validation in `CreateCollectionOperation::new` and
`UpdateCollectionOperation::new` instead: the constructors are the
common chokepoint for all API paths, before the operation is proposed
to consensus. This covers every validator on these types, not just the
`memory` checks, and keeps consensus-apply unaffected so mixed-version
clusters never reject already-committed operations.
`UpdateCollectionOperation::new` becomes fallible; `remove_replica` now
uses `new_empty` since it carries no user config. Regression tests drive
the gRPC conversion path and assert `InvalidArgument` for `pinned` on
create and update, with `cold`/`cached` accepted as a control.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
Andrey VasnetsovandClaude Fable 5
|
efab63d024 |
Add prefix matching option to keyword index (#9683)
* Add prefix matching option to keyword index
Introduce an opt-in `prefix` option for the keyword payload index and a
new `match: { "prefix": ... }` filter condition, enabling efficient
byte-wise prefix filtering over keyword values (e.g. URL prefixes,
web-ui value autocompletion via facet + prefix filter).
Index side: a new `prefix_index.bin` file stores a sorted, front-coded
key dictionary with a resident block index (cumulative counts per
block); it is an ordered view over the keys of `values_to_points.bin`
and stores no postings. Presence of the file signals prefix support at
load time, so legacy segments load unchanged and enabling the option
goes through the standard incompatible-schema rebuild. The mutable
variant keeps an in-RAM ordered key set (not persisted), the immutable
variant builds a sorted key vector at load, and the on-disk variant
reads the dictionary lazily (block index resident, 1-2 block reads per
prefix lookup; reader is generic over UniversalRead).
Query side: prefix conditions are served from the dictionary when
available (filter + cardinality estimation from per-block aggregates),
from the forward index as per-point checks, and degrade to the payload
full-scan fallback otherwise - same execution model as other match
conditions. Strict mode (`unindexed_filtering_*`) rejects prefix
queries on fields without a prefix-enabled keyword index via a new
KeywordPrefix capability.
HNSW payload blocks: prefix-enabled indexes additionally emit prefix
blocks for heavy branching trie nodes (single-child chains collapsed to
their longest common prefix, one block per distinct point set, emitted
largest-first) so filtered search with prefix conditions gets navigable
subgraphs without rebuilding the same subset repeatedly.
API: `prefix` flag on KeywordIndexParams (REST bool, gRPC empty
message for extensibility), `prefix` variant in the Match oneof, edge
python bindings, regenerated OpenAPI spec.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Split prefix index into a dedicated module, fix clippy in tests
Reorganize the flat prefix_index.rs / prefix_read.rs into a
map_index/prefix_index/ module: format.rs (on-disk layout primitives),
writer.rs, reader.rs (PrefixIndex), map_read.rs (StrMapIndexPrefixRead
with per-variant impls) and tests.rs, with a file-format diagram and a
read-path walkthrough in the module docs. No logic changes.
Also fix clippy --all-targets complaints in test code: replace a
wildcard Match arm with an exhaustive list and a field-reassign-with-
default with a struct literal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add OpenAPI test for prefix match and snapshot file-tracking test
- tests/openapi/test_prefix_match.py: index-less fallback, prefix index
creation with schema echo, scroll/count parity against ground truth,
facet + prefix filter (the autocompletion flow), strict-mode rejection
without the prefix capability.
- test_prefix_index_file_tracking: `prefix_index.bin` is listed in
`files()` / `immutable_files()` exactly when built with the option.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Replace hand-rolled varint parsing with bytemuck Pod records
Per review: the prefix index format now uses fixed-size little-endian
Pod records (BlockEntry 24 B, KeyEntry 12 B, Header 40 B) written with
bytemuck::bytes_of and read back by copy via pod_read_unaligned — no
manual varint encode/decode, no alignment requirement, one shared
read_record helper. Costs ~9 bytes per key on disk versus LEB128; the
raw key bytes dominate dictionary size, so the simplification wins.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fetch the whole candidate block range with a single storage read
Candidate key blocks of a prefix lookup are contiguous in the file, so
enumerate them from one ranged read instead of one read per block; the
over-read versus the exact key range is bounded by the two boundary
blocks. Block decoding is split into a storage-free helper reused by
the per-block path of stats estimation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Align prefix payload blocks with the geo index granularity principle
Geo's large_hashes emits only the smallest geohash regions above the
threshold — a disjoint antichain, never a parent nested with its
children. Prefix payload blocks now follow the same rule: a heavy
collapsed trie node is emitted only if nothing heavy is nested inside
it, counting both deeper qualifying prefixes and single heavy values
(which already get their own exact-match blocks). Emitted blocks are
therefore mutually disjoint and disjoint from exact-value blocks; no
near-collection-sized ancestor subgraphs, no reliance on the HNSW
connectivity check to skip nested duplicates.
Implemented as a `covered` flag propagated through the existing
LCP-interval scan, still one O(total key bytes) pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Document block wire format and unaligned-read rationale in decode_block
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
|
80c9454141 |
[UIO] Include file size in UniversalReadFileOps::list_files (#9675)
* [AI + manual] Include file size in `UniversalReadFileOps::list_files` * Use dedicated `ListedFile` struct instead of `(PathBuf, u64)` pair Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
8c8a72d120 |
Remove read_multi_iter to fix macOS linker symbol overflow (#9643)
* remove unused iter_offsets
* Replace MultivectorOffsetsStorage::iter_offsets with callback-based for_each_offset
First step of removing the iterator-returning read API (whose deep,
composable generic types blow up mangled symbol size). Convert the
offsets read from an iterator to a callback the caller pushes into:
- trait method iter_offsets -> for_each_offset(ids, FnMut(usize, MultivectorOffset))
returning common::universal_io::Result<()>
- Mmap impl now uses the callback read_batch (drops one read_iter use)
- Ram / Chunked impls push into the callback; Chunked still goes through
iter_vectors for now (converted in a later step)
- the single caller (for_each_in_multi_batch) passes a closure
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Implement read_batch directly on ReadPipeline, not via read_iter
read_batch now drives the pipeline itself (refill-then-wait loop, like
read_multi_iter) and invokes the callback per result, instead of
consuming the iterator returned by read_iter. A step toward removing the
iterator-returning read API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove the ReadMulti RPC from the StorageRead gRPC service
ReadMulti was the only real consumer of UniversalRead::read_multi (which
itself relies on read_multi_iter). Removing the RPC end-to-end clears the
path to dropping that read API. StorageReadService keeps all its other
RPCs (ListFiles, FileExists, FileLength, ReadBytes, ReadBytesStream,
ReadWhole, ReadBatch).
- proto: drop `rpc ReadMulti` + ReadMulti{Entry,Request,Response}
- regenerated lib/api + uio-client generated code; drop ReadMulti
validation rules in lib/api/build.rs
- tonic: delete the read_multi handler + its 2 tests
- uio-client: delete Client::read_multi, the mock-server impl, and 2 tests
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove UniversalRead::read_multi
Its only real consumer was the StorageRead ReadMulti gRPC handler (removed
in the previous commit); the two wrapper forwarders had no callers. Drop
the trait method and both forwarders (typed/read_only), and remove the
io_uring test that only existed to compare read_multi vs read_multi_iter
(read_multi_iter stays covered by the other tests). Another step toward
removing the iterator-returning read API.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drive ReadPipeline directly in gridstore read_from_pages
Replace the read_multi_iter call in Pages::read_from_pages with a direct
pipeline loop (refill-then-drain), scheduling each multi-page read on its
own page file. Behavior unchanged; another step toward removing the
iterator-returning read_multi_iter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drive ReadPipeline directly in gridstore read_batch_from_pages
Replace the second read_multi_iter call (in Pages::read_batch_from_pages)
with a direct pipeline loop, scheduling each (ReadMeta, page, range) on its
own page file and propagating errors via GridstoreError. Single/multi-page
buffering and out-of-order reassembly are unchanged. No more read_multi_iter
in gridstore.
Measured overhead on warm mmap (both paths are zero-copy borrows): ~0.3 ns
per read of fixed control cost, flat across read sizes — well under 0.1% of
a real payload read.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drive ReadPipeline directly in on-disk postings with_posting_views
Replace read_iter in OnDiskPostings::with_posting_views with a direct
pipeline loop. wait_bytemuck yields a file-borrowed Cow, so postings are
still stored zero-copy in raw_postings (a read_batch swap would have forced
an owned copy of every posting list per query on the mmap backend).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Read on-disk posting headers via read_batch, drop the HeadersBatch iterator
headers_iter now reads headers with the callback read_batch API: each header
is parsed (copied) out of the read bytes, so nothing borrows the file past the
read — no pipeline needed. Since the read is now eager, HeadersBatch holds the
collected Vec<HeaderResult> directly instead of a Box<dyn Iterator>, dropping
the boxing, the dynamic dispatch, and the struct's lifetime parameter.
with_posting_views takes the Vec and still pipelines the posting reads.
Removes the last read_iter use in this file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Use read_batch in simple_disk_cache populate_from
populate_from reads one byte per block only to fault blocks into the local
cache, discarding the bytes — a no-op-callback read_batch fits exactly. Drives
the same DiskCachePipeline as before; one less read_iter caller.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Implement read_iter directly on ReadPipeline, not via read_multi_iter
read_iter now drives the pipeline itself (refill-then-wait loop, mirroring
read_bytes_iter) instead of mapping its ranges onto self and calling
read_multi_iter. Same signature and iterator contract, so all callers are
unchanged. Leaves iter_vectors as read_multi_iter's only remaining caller.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "Drive ReadPipeline directly in on-disk postings with_posting_views"
This reverts commit
|
||
|
xzfc
|
596e8c7c01 | Cleanup FieldCondition validation (#9557) | ||
|
Tim Visée
|
b9154713d7 |
Fix empty min_should with non-zero min_count matching everything (#9401)
* Empty match any with non-zero min count matches nothing * Update description * Validate that min_count is greater than 0 |
||
|
|
8c487a17e8 |
feat(bm25): explicit Disabled stemmer; deprecate language: "none" hack (#9376)
* feat(bm25): add explicit Disabled stemmer; deprecate language hack
Adds a `Disabled` variant to `StemmingAlgorithm` (`stemmer: {"type": "none"}`)
so stemming can be turned off explicitly in both the main engine and Edge,
instead of relying on the undocumented `language: "none"` footgun that
silently disabled both stemming and stopwords.
For language-neutral text processing the supported setup is now:
1. set the stemmer to disabled, and
2. configure an empty stopword set.
The main engine still tolerates unsupported languages (so existing
`language: "none"` configs keep working on upgrade) but now logs a
deprecation warning pointing users to the explicit setup. Edge continues
to reject unsupported languages, and now has a real way to disable stemming.
Refs: https://github.com/qdrant/qdrant/issues/9289
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(edge-py): handle Disabled stemmer in python bindings; fix openapi schema
- Handle the new StemmingAlgorithm::Disabled variant in the qdrant-edge-py
bindings (FromPyObject/IntoPyObject/Repr) and add a DisabledStemmer pyclass
plus its .pyi stub entry.
- Match generator output for the StemmingAlgorithm OpenAPI schema (plain $ref
in anyOf) so docs/redoc/master/openapi.json stays consistent.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openapi): regenerate StemmingAlgorithm schema with generator output
Ran tools/generate_openapi_models.sh so docs/redoc/master/openapi.json
exactly matches generator output: DisabledStemmerParams/NoStemmer are placed
after SnowballLanguage, and the StemmingAlgorithm anyOf entry is a plain $ref
(the schema2openapi step flattens the allOf+description wrapper).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): avoid wildcard enum match arm in bm25 sparse_len helper
clippy --all-targets flags `other => panic!()` as wildcard_enum_match_arm;
match the Dense/MultiDense variants explicitly instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: issues
* fix: log::warn as call once
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
|
||
|
Tim Viséeandcoderabbitai[bot]
|
463a305404 |
Add routing token for deterministic read routes (#9338)
* Add routing token structure * Implement routing token in read operation executor as per design doc * Add TODO to glue routing token to user requests * Implement routing header for REST API * Source routing token from request, not from JWT token * Implement routing token in gRPC API * Add test * Review remarks Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Use lower case header name to prevent panic * Rename header to X-Qdrant-Route-Affinity * Assert routing consistency in test on all peers --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
||
|
Marcelo Machuca
|
24f5391645 |
fix(api): validate geo coordinate ranges in gRPC FieldCondition (#9347)
`FieldCondition::validate` checked geo polygon *shape* (point count and closure) but never the *coordinate ranges* of any geo sub-condition. A gRPC filter with latitude outside [-90, 90] or longitude outside [-180, 180] (in geo_bounding_box, geo_radius, or geo_polygon) passed validation, reached the geo index, and panicked during geohash encoding, which expects pre-validated input. Reject out-of-range coordinates at the API boundary for all three geo sub-conditions, mirroring `segment::types::GeoPoint::validate`. The bounds are duplicated because the `api` crate does not depend on `segment`. Includes a regression test covering geo_radius, geo_bounding_box, and a well-formed (shape-valid) geo_polygon with out-of-range coordinates. |
||
|
Djole
|
25a4d4906d |
fix: validate hnsw_ef search parameter (#9320)
* fix: validate hnsw_ef search parameter * chore: regenerate openapi spec |
||
|
Tim Viséeandxzfc
|
7041b81f76 |
Use assert_matches! (#9231)
* Use assert_matches! * Add trailing commas * Use more assert_matches! Also, drop now redundant `expected blah but got blah` messages because `assert_matches!` will print these. * Use debug_assert_matches! --------- Co-authored-by: xzfc <xzfcpw@gmail.com> |
||
|
Marcelo Machuca
|
4d00a5e628 |
fix(api): reject empty multi-vector in flattened vectors_count path (#9308)
`validate_multi_vector_len(N, &[])` with N > 0 previously returned Ok: it passes the `vectors_count != 0` check, an empty `flatten_dense_vector` clears the size check, and `0.is_multiple_of(N)` is true, so it falls into the Ok branch. The unvalidated value then reaches `convert_to_plain_multi_vector`, where `dim = data.len() / vectors_count = 0`, the divisibility check `dim * vectors_count != data.len()` (0 == 0) passes, and `data.into_iter().chunks(0)` panics (itertools asserts the chunk size is non-zero). This is reachable from a single malformed gRPC Upsert via the deprecated `vectors_count` field. Add an `is_empty()` guard to `validate_multi_vector_len`, mirroring the sibling `validate_multi_vector_by_length`, so empty flattened data is rejected with a clear validation error before any conversion. Also add a defensive `vectors_count == 0 || data.is_empty()` guard at the top of `convert_to_plain_multi_vector`, which aligns it with the already-guarded `MultiDenseVectorInternal::try_from_flatten` and closes the same panic on the internal node-to-node `sync` path (where validation is log-only and `SyncPoints.points` is not validated). This completes the empty-vector hardening started in #9070, which covered the REST side only and did not touch the gRPC flattened `vectors_count` form. Includes a regression test covering both the rejected (empty) and accepted (consistent multivector) cases. |
||
|
Marcelo Machuca
|
3ce151632a |
fix(api): validate geo polygon in gRPC FieldCondition (#9309)
`impl Validate for grpc::FieldCondition` only checked that at least one condition field was set; it never validated the contents of a geo polygon. A `FieldCondition` carrying a malformed `geo_polygon` (empty exterior, fewer than 4 points, or an unclosed exterior/interior ring) therefore passed validation, and the invalid shape later reached the geo index and panicked. Recurse into the polygon's existing validator (`geo_polygon.validate()?`), which already rejects these shapes (it is covered by `test_geo_polygon`), so the request is rejected with a clean validation error instead. Includes a regression test asserting a FieldCondition with an empty polygon exterior is rejected while a well-formed polygon still passes. |
||
|
Marcelo Machuca
|
9edddf6e3b | fix(api): return formula default validation errors (#9271) | ||
|
qdrant-cloud-botandCursor
|
c150bd5caa |
Strict mode: add max_disk_usage_percent (#9212)
* Strict mode: add `max_disk_usage_percent` Mirrors `max_resident_memory_percent`: rejects disk-consuming update ops (upsert, set/overwrite payload, update vectors) when the filesystem hosting Qdrant storage is filled above the configured percentage. Delete-style ops remain allowed so callers can free disk. Disk usage is sampled via `statvfs` and TTL-cached for 5s (same cadence as the resident-memory reader) so high-RPS request paths don't hammer the syscall. Reader is keyed by path in `common::disk_usage` and returns `None` on stat failure — callers (the strict-mode check) treat `None` as "skip", matching the memory-check behaviour. Plumbing follows the existing pattern: field on `StrictModeConfig` (+ output/diff/Hash), gRPC proto field `22`, validation 1..=100, REST/proto conversions, and the hook into `check_strict_mode_toc_batch` alongside the memory check (both guarded by `any_consumes_memory`). Co-authored-by: Cursor <cursoragent@cursor.com> * Fix CI: Windows disk_usage test + e2e WAL config - `missing_path_returns_none` panicked on Windows because `GetDiskFreeSpaceEx` succeeds for non-existent paths (it resolves up to the containing drive). Relax the assertion to "must not panic; if a value is returned it must be well-formed". The contract we care about (None on failure) is platform-defined, not something we can portably force. - e2e test failed at batch 0 with "WAL buffer size exceeds available disk space": Qdrant's existing per-shard `DiskUsageWatcher` enforces `free >= 2 * wal_capacity_mb` and the default WAL didn't fit in the 50 MB tmpfs. Bump tmpfs to 200 MB and shrink `wal_capacity_mb` to 1 MB (same pattern as `test_low_disk.py`) so our strict-mode gate is the one that fires, not the WAL pre-check. Raise the gate threshold to 50% to match the larger headroom. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Jojii
|
7a8703f166 |
[TQDT] API (#9172)
* [ai] TQDT in the API * [ai] unify new sparse error for TQDT * Rename to `turbo4` * Add `Turbo4` to comments and doc strings. * Also validate named sparse vector creation |
||
|
Roman Titov
|
53a8562521 |
Upgrade tonic to v0.14.6 (#9139)
|
||
|
|
26aeb9c0b2 |
docs: refresh prevent_unoptimized description (#9133)
The previous description was outdated: it claimed that enabling this option "blocks updates at the request level" until segments are re-optimized. In practice the implementation uses "deferred points": new points written to large unoptimized segments are persisted but excluded from read/search results until the segments are optimized. Updates are not blocked; only `wait=true` clients are made to wait for the deferred points to become visible. Update this in the REST schema (via `OptimizersConfig` / `OptimizersConfigDiff`), in the gRPC proto, in the edge config docstrings, and regenerate the OpenAPI bundle via `tools/generate_openapi_models.sh`. Co-authored-by: Cursor Agent <agent@cursor.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
xzfc
|
faf2714f4b | Warn on clippy::wildcard_enum_match_arm (#9096) | ||
|
Tim Visée
|
ff4e41ed88 |
Fix empty vector panic (#9070)
* Add test * Validate empty vector name * Update test assertions |
||
|
Ivan Pleshkov
|
491712424d |
tq remove data fit option (#8943)
* tq disable data fit option * remove any mention in grpc |
||
|
Andrey VasnetsovandClaude Opus 4.7
|
fd9bc02696 |
Add ordering parameter to gRPC create/delete vector name (#8926)
Match the REST API by adding an optional `WriteOrdering` field to `CreateVectorNameRequest` and `DeleteVectorNameRequest`, and propagate it through the tonic handlers and remote-shard forwarding paths. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Jojii
|
d3ad1ac988 |
API Adjustments for TQ (#8914)
* API Adjustments for TQ * Clippy |
||
|
Daniel Boros
|
068fbc1426 | feat: add internal shard level storage api (#8778) | ||
|
Arnaud Gourlayandtimvisee
|
354bbb35e6 |
Delete unused code (#8771)
* Delete unused code * restore initialize_global * drop BadShardSelection * Remove now obsolete allow(dead_code) attributes * Remove more dead code --------- Co-authored-by: timvisee <tim@visee.me> |
||
|
Arnaud Gourlay
|
4cb494d3b6 | Fix missing validation on adding named vector (#8776) | ||
|
Daniel Boros
|
bf13d43816 | feat/internal-grpc-auth (#8676) | ||
|
|
9686c8f952 |
low ram strict mode (#8715)
* [AI] strict mode parameter for limiting update requests if ram usage is over threshold * opanAPI update * [AI] end-to-end test * fmt * Fix e2e test: memory rejection check broken by string truncation UnexpectedResponse.__str__() truncates the raw response body, cutting off the `max_resident_memory_percent` hint at the end of the error message. Use `resident memory usage` instead, which appears early enough to survive the truncation. Made-with: Cursor * add grpc validation * test check_resident_memory --------- Co-authored-by: Cursor Agent <agent@cursor.com> Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com> |
||
|
Jojii
|
f51c334554 |
Fix grpc stop words (#8728)
* Fix stopwords always being lowered in grpc path only * [ai] add integration test, stopwords grpc vs rest |
||
|
|
8c72d3b12f |
API Changes for TurboQuant (#8686)
* [ai + manual] API changes for TQ
* CI
* Add TurboQuant types to Python type stubs
Made-with: Cursor
* Revert "Add TurboQuant types to Python type stubs"
This reverts commit
|