Follow-up to #9530 and #9558.
The shard-level segment manifest was written inside the `segments/`
directory (`segments/manifest.json`). Older versions of Qdrant choke on
an unknown file inside `segments/`, so move it next to the directory as
`segments_manifest.json` instead.
- `SEGMENT_MANIFEST_FILE` is now `segments_manifest.json` and
`segment_manifest_path()` points at the shard root.
- The manifest is added to `ShardDataFiles` so clear/move handle it.
- Snapshots write the manifest to the snapshot root (next to
`segments/`), and restore/partial-snapshot loaders no longer need to
skip it inside `segments/`.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: feature-flagged segment manifest for LocalShard
Add an on-disk segment manifest (`segments/manifest.json`) that lists a
shard's segments and their state, so out-of-process readers (e.g. a
read-only follower, possibly over object storage) can discover segments
without scanning the filesystem. Gated by the new `write_segment_manifest`
feature flag (off by default).
shard: define the structure + helpers (`SegmentsManifest`,
`SegmentManifestState`, `from_segment_holder`) in a new `segment_manifest`
module, plus the `SEGMENT_MANIFEST_FILE` constant and path helper. The
manifest is a flat `{ "<uuid>": "<state>" }` map; only `active` is written
today, with `under_construction`/`retiring` defined so the format can be
extended without breaking compatibility.
collection: LocalShard owns the writing logic. The manifest is persisted
via `SaveOnDisk<SegmentsManifest>`, initialized from the live segment set
on load/build and refreshed by the optimization worker whenever the
segment set changes (the helper re-derives from the holder and no-ops when
unchanged). No changes to `lib/shard/src/optimize.rs` internals.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor
* feat: make append_only_mutations a proper feature flag
Replace the debug-only `QDRANT_APPEND_ONLY_MUTATIONS=1` env-var escape
hatch in segment construction with a `FeatureFlags::append_only_mutations`
flag, so it works in release builds and is configurable like the other
flags (config / `QDRANT__FEATURE_FLAGS__APPEND_ONLY_MUTATIONS`).
Deliberately left out of `FeatureFlags::all()`: it changes mutation
semantics and `all` is enabled in dev and e2e configs, so it stays
explicit opt-in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor
* upd openapi schema
* feat: register segments in manifest via must-use token, holder builder
Wire segment-manifest maintenance through the segment lifecycle so a
newly created segment is registered as soon as it exists on disk, and
construction can't silently skip it:
- build_segment now returns a #[must_use] NewSegmentToken carrying the
new segment UUID; the lint forces callers to register or drop it.
- SegmentHolder owns the manifest and reconciles it on sync; new
segments are registered ASAP (even before being added to the holder)
via the token, before they can receive writes.
- SegmentHolderBuilder is the only way to obtain a shard's holder; its
build() wires up the manifest, so it can't be forgotten. init/set
manifest helpers are now private / test-only.
- Optimization registers the optimized segment before dropping the
superseded segments' data; deletion is intentionally lenient.
- Document the consistency assumptions on SegmentsManifest: it is a
superset-biased view that may list not-yet-finalized or already-deleted
segments, which readers must tolerate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(bm25): add explicit Disabled stemmer; deprecate language hack
Adds a `Disabled` variant to `StemmingAlgorithm` (`stemmer: {"type": "none"}`)
so stemming can be turned off explicitly in both the main engine and Edge,
instead of relying on the undocumented `language: "none"` footgun that
silently disabled both stemming and stopwords.
For language-neutral text processing the supported setup is now:
1. set the stemmer to disabled, and
2. configure an empty stopword set.
The main engine still tolerates unsupported languages (so existing
`language: "none"` configs keep working on upgrade) but now logs a
deprecation warning pointing users to the explicit setup. Edge continues
to reject unsupported languages, and now has a real way to disable stemming.
Refs: https://github.com/qdrant/qdrant/issues/9289
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(edge-py): handle Disabled stemmer in python bindings; fix openapi schema
- Handle the new StemmingAlgorithm::Disabled variant in the qdrant-edge-py
bindings (FromPyObject/IntoPyObject/Repr) and add a DisabledStemmer pyclass
plus its .pyi stub entry.
- Match generator output for the StemmingAlgorithm OpenAPI schema (plain $ref
in anyOf) so docs/redoc/master/openapi.json stays consistent.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openapi): regenerate StemmingAlgorithm schema with generator output
Ran tools/generate_openapi_models.sh so docs/redoc/master/openapi.json
exactly matches generator output: DisabledStemmerParams/NoStemmer are placed
after SnowballLanguage, and the StemmingAlgorithm anyOf entry is a plain $ref
(the schema2openapi step flattens the allOf+description wrapper).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): avoid wildcard enum match arm in bm25 sparse_len helper
clippy --all-targets flags `other => panic!()` as wildcard_enum_match_arm;
match the Dense/MultiDense variants explicitly instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: issues
* fix: log::warn as call once
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
Previously all three Kubernetes health endpoints shared the same generic
description ("An endpoint for health checking used in Kubernetes."), which did
not convey what each one actually guarantees.
- /healthz and /livez: clarify they are pure liveness checks (200 once the HTTP
API is up), do not inspect data/shards/consensus, and are identical to each
other.
- /readyz: clarify it is a readiness probe that waits out pending data
operations (consensus catch-up + shard health in distributed mode) before
reporting ready, and document the real 200 ("all shards are ready") and 503
("some shards are not ready") responses.
Documentation-only change (OpenAPI descriptions/examples + added 503 response
doc). No runtime behavior changes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Strict mode: add `max_disk_usage_percent`
Mirrors `max_resident_memory_percent`: rejects disk-consuming update ops
(upsert, set/overwrite payload, update vectors) when the filesystem hosting
Qdrant storage is filled above the configured percentage. Delete-style ops
remain allowed so callers can free disk.
Disk usage is sampled via `statvfs` and TTL-cached for 5s (same cadence as
the resident-memory reader) so high-RPS request paths don't hammer the
syscall. Reader is keyed by path in `common::disk_usage` and returns `None`
on stat failure — callers (the strict-mode check) treat `None` as "skip",
matching the memory-check behaviour.
Plumbing follows the existing pattern: field on `StrictModeConfig` (+
output/diff/Hash), gRPC proto field `22`, validation 1..=100, REST/proto
conversions, and the hook into `check_strict_mode_toc_batch` alongside the
memory check (both guarded by `any_consumes_memory`).
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix CI: Windows disk_usage test + e2e WAL config
- `missing_path_returns_none` panicked on Windows because
`GetDiskFreeSpaceEx` succeeds for non-existent paths (it resolves up to
the containing drive). Relax the assertion to "must not panic; if a
value is returned it must be well-formed". The contract we care about
(None on failure) is platform-defined, not something we can portably
force.
- e2e test failed at batch 0 with "WAL buffer size exceeds available disk
space": Qdrant's existing per-shard `DiskUsageWatcher` enforces
`free >= 2 * wal_capacity_mb` and the default WAL didn't fit in the
50 MB tmpfs. Bump tmpfs to 200 MB and shrink `wal_capacity_mb` to 1 MB
(same pattern as `test_low_disk.py`) so our strict-mode gate is the
one that fires, not the WAL pre-check. Raise the gate threshold to
50% to match the larger headroom.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* [ai] TQDT in the API
* [ai] unify new sparse error for TQDT
* Rename to `turbo4`
* Add `Turbo4` to comments and doc strings.
* Also validate named sparse vector creation
The previous description was outdated: it claimed that enabling this
option "blocks updates at the request level" until segments are
re-optimized. In practice the implementation uses "deferred points":
new points written to large unoptimized segments are persisted but
excluded from read/search results until the segments are optimized.
Updates are not blocked; only `wait=true` clients are made to wait for
the deferred points to become visible. Update this in the REST schema
(via `OptimizersConfig` / `OptimizersConfigDiff`), in the gRPC proto,
in the edge config docstrings, and regenerate the OpenAPI bundle via
`tools/generate_openapi_models.sh`.
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Initial commit
* Edits
* Change non-descriptive “here” link texts
* Review feedback
* Add Edge code snippet, Web UI section with screenshot, mention autit logging, mention NVIDIA and AMD GPU support
* Resize Web UI image; Move to end of Features section
* Restore Web UI section in README
Reintroduce Web UI section to README with visual aid.
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
`StrictModeConfigOutput` is an output-only type that is never validated,
so remove its dead `#[validate(...)]` field attributes.
`ShardSnapshotRecover` does need validation: add the missing `Validate`
derive and switch the actix handler from `web::Json` to `valid::Json`
so the SHA256 checksum constraint is actually enforced.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* [AI] strict mode parameter for limiting update requests if ram usage is over threshold
* opanAPI update
* [AI] end-to-end test
* fmt
* Fix e2e test: memory rejection check broken by string truncation
UnexpectedResponse.__str__() truncates the raw response body, cutting
off the `max_resident_memory_percent` hint at the end of the error
message. Use `resident memory usage` instead, which appears early
enough to survive the truncation.
Made-with: Cursor
* add grpc validation
* test check_resident_memory
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Add empty placeholder vector storage types for named vector CRUD
Introduce EmptyDenseVectorStorage and EmptySparseVectorStorage as
placeholder storages for newly created named vectors on immutable
segments. These report all vectors as deleted, consume no disk space,
and are reconstructed from segment config on load via the new
VectorStorageType::Empty and SparseVectorStorageType::Empty variants.
Key design decisions:
- is_on_disk is derived from original user config, not hardcoded
- MultiVectorConfig is preserved for multi-vector support
- Config mismatch optimizer skips Empty storage to avoid false rebuilds
- Quantization delegates normally (handles 0 vectors gracefully)
- get_vector includes debug_assert to catch unexpected access
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] segment-level operations for creating and deleting anmed vectors
* [AI] implement named vector creation and deleting in proxy segment
* [AI] Step 3: Proxy Segment Handling for Named Vector Operations
* [AI] implement for Edge
* [AI] implement consensus operations for named vector operations
* [AI] refactor VectorNameConfig, remove VectorNameConfigInternal
* [AI] handle vector schema inconsistency in raft snapshot recovery
* [AI] rest + grpc API
* [AI] clippy
* [AI] generate openAPI schema
* fmt
* ci fixes
* [AI] fix jwt access test
* [AI] nop operation for awaiting of consensus-commited update ops
* [AI] move vector name operations into points service
* [AI] implement internal api for vector name operations
* [AI] change collection-level config along with segment level operation
* [AI] vector schema reconceliation instead of error
* fmt
* missing compile-time option
* [AI] integration test
* [AI] fix missing JWT tests
* [AI] remove NOP
* [AI] openapi test
* [AI] fix initialization of mutable segment
* [AI] more simple integration tests
* fmt
* [AI] make cluster test a bit harder
* [AI] make test less flacky
* [AI] rabbit comments
* [AI] check params compatibility before writing vector config
* [AI] make sure to register vector storages in structure payload index
* [AI] vector name validation
* lower vector length validation to 200 chars to account for prefix in filename
* [AI] proxy segment: prevent stale data leak through optimization
* fmt
* [AI] filter out removed vectors from proxy response
* [AI] handle vector name in proxy
* fmt
* adjust proxy info based on dropped vectors
* [AI] proxy segment: update filters to correct has_vector condition
* fmt
* clippy
* Fix consensus snapshot applicaiton for vector schema
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributors frequently open PRs against master instead of dev.
Add prominent callouts and a root-level CONTRIBUTING.md so the
branch requirement is visible at every step of the contribution flow.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* Skip broken tests
* Add rocksdb dropper
This is a temporary tool. It will be removed later in this PR.
* [automated] Drop rocksdb
This commit is made by running tools/rocksdb/drop.sh
* Touch-up after ast-grep
The previous automated commit removed items, but not their comments.
Also, some blocks are left with only one item.
Also, ast-grep can't handle macros like `vec![]`.
This commit completes the job.
* Remove RocksDB dropper
* Remove mentions of rocksdb feature in CI
* Fix clippy warnings
These `FIXME` comments added previously in this PR by ast-grep by
"peeling" cfg_attr like this:
-#[cfg_attr(not(feature = "rocksdb"), expect(...))]
+#[expect(...)] // FIXME(rocksdb): ...
This commit removes these allow/expect attributes and fixes clippy
lints.
* Remove leftover rocksdb-related code
Removed:
- Cargo.toml: rocksdb cargo feature flag and dependencies.
- code: rocksdb-related items that was not under feature flag.
- flags.rs: rocksdb-related qdrant feature flags.
Disabled:
- Some benchmarks because these do not compile now.
* Print warning if rocksdb leftovers found in snapshots
* Remove Clone from tokenizer
* Regenerate openapi.json
* feat: add search_max_batchsize to strict mode config
* added test case for search_max_batchsize
* Changes for fixing CI issue dure openapi
* Modify check_strict_mode_batch
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Add deferred point count to UpdateQueueInfo
* Hide if prevent_unoptimized is false
* Clippy
* Openapi
* Iterate over appendable segments only
* Only calculate deferred point count if prevent_unoptimized is true
* Rebase fixes
* Codespell
* Don't account for deferred points in some places
# Conflicts:
# lib/collection/src/shards/local_shard/scroll.rs
* Add in QueryContext
* Cover more places
* Coderabbit review remarks
* Properly count amount of deleted deferred points (#8386)
* Properly count amount of deleted deferred points
* Prevent double-counting of the same point
* Remove hints to estimations
* Properly handle counts in ProxySegment
* Add tests and fix deleted point count issue
* Adjusts tests + fix issues
* Separte fields for deferred points in telemetry (SegmentInfo)
* Remove deferred_points_count()
* openapi
* Fix test by manually calculating visible points
* Adjust ProxySegment test to revertion of SegmentInfo
* Throw error if collection was not found in telemetry (e2e Test)
* Update lib/segment/src/segment/segment_ops.rs
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Make new fields in API optional
* Don't take range if no deferred point exist
* Review remarks
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Add `deferred_internal_id` field to `SegmentInfo` so it is exposed
through segment telemetry. The field is optional and skipped from
serialization when `None`. It is excluded from anonymization via
`#[anonymize(false)]`.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* [manual] make untagged enum for consistent API for FeedbackStrategy
* add #[validate(nested)]
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* add rest and grpc interfaces
Also handle inference for this query
* follow refactor from base branch
* renaming from Ms Cooper
* get started on validation testing
* more validations
* test equivalence with query when less than 2 feedback elements
* rename feedback query to relevance_feedback query
* fix extraction of context pairs
* rename feedback vector to example
* make coderabbit happier
* upd test
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* weighted rrf implementation
* test
* fmt
* fix edge
* validate number of sources and number of weights
* do not partial match
* upd schema
* review fixes
* update formula
* remove calcualtions from tests
* update comment, because AI have OCD
* fmt
* Update queue info into CollectionInfo
* basic test
* fix
* cleaner
* Update lib/collection/src/shards/local_shard/mod.rs
Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
* fix the fix
* rename to op_num
* skip_serializing_if just in case
* first step
* test queue length with staging feature
* gate integration test based on binary feature
---------
Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
* introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert
* add test
* upd dockstring
* require resharding once all peers have updated version
* use service error
* fix clippy again
* wait for same version before resharding in tests
