* [ai] TQDT in the API
* [ai] unify new sparse error for TQDT
* Rename to `turbo4`
* Add `Turbo4` to comments and doc strings.
* Also validate named sparse vector creation
* fix: validate vector dimensions before WAL write for async upserts
When upserting points with wait=false (the default), dimension
mismatches were silently discarded during background processing.
The API returned 200 "acknowledged" but the points were never stored,
causing silent data loss with no error feedback to the user.
This adds an early dimension validation check in do_upsert_points()
that runs before the operation is written to WAL. This ensures that
dimension errors are returned to the client regardless of the wait
parameter, matching the behavior of wait=true.
The validation handles all vector types:
- Dense single vectors
- Multi-dense vectors
- Named vectors (dense, multi-dense, sparse)
- Sparse vectors are skipped (no fixed dimension)
Closes#9039
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor: move vector dimension validation into dedicated module
Extract validate_vector_dimensions and helper functions from update.rs
into src/common/validate_vectors.rs for better code organization.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: update shard update test for early dimension validation
The test expected a shard-level error message, but now dimension
mismatches are caught before reaching the shards. Update the assertion
to accept either the early validation error or the shard-level error.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: assert actual dimension error message in shard update test
Check for the descriptive error ("Vector dimension error: expected dim: 4, got 3")
rather than the generic shard failure wrapper.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add integration test for `match: {except: []}` with integer index
Regression test for https://github.com/qdrant/qdrant/issues/9050
An empty `except` list (NOT IN []) should always match all points that
have the field, both with and without a payload index. Currently the
integer (and keyword) indexed path incorrectly returns zero results
because:
1. serde deserializes `except: []` as `AnyVariants::Strings([])` (first
variant of the untagged enum)
2. The map index filter_impl returns `iter::empty()` for empty
cross-type variant, instead of matching everything
The test covers:
- except: [] without any index (baseline, currently works)
- except: [] with an integer index (currently broken)
- except: [] with a keyword index (currently broken)
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix `match: {except: []}` returning zero results with payload index
Fixes https://github.com/qdrant/qdrant/issues/9050
Root cause: `except: []` deserializes as `AnyVariants::Strings([])`
due to the untagged serde enum trying `Strings` first. On an integer
index, the filter_impl matched `Except + Strings(empty)` and returned
`iter::empty()` (zero results). The same issue existed symmetrically
on keyword indexes with `Integers(empty)` and on UUID indexes with
`Integers(empty)`.
The fix: when the cross-type variant reaches the Except branch (e.g.
Strings on an integer index, Integers on a keyword/UUID index), return
`None` unconditionally — regardless of whether the set is empty. This
delegates to the fallback condition checker, which already handles
`except: []` correctly by matching all values.
The `estimate_cardinality_impl` functions had the same bug (returning
`CardinalityEstimation::exact(0)` for the empty cross-type case) and
are fixed the same way.
Affected index types: integer, keyword (str), UUID.
Bool index is NOT affected — it already returns `None` for all
Any/Except conditions.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Expand match-except test to cover all index types and cross-type filters
Extends the regression test for #9050 to comprehensively cover:
- Integer, keyword, and UUID field indexes
- Empty except list (the original bug) for each index type
- Non-empty except list with matching types (normal filtering)
- Cross-type except values (e.g. strings on integer index, integers on
keyword/UUID index) — type mismatch should exclude nothing → all match
- Exclude-all: listing every value should return zero results
- All scenarios tested both with and without a payload index
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Raw requests already use QDRANT_HOST, but bypass request_with_validation
and missed QDRANT_HOST_HEADERS. Without these headers, tests cannot run
behind host-based reverse proxies or mocks.
These tests rely on the collection size stats cache being refreshed to
detect that a size limit has been exceeded. Without wait=true, upsert
operations are written to WAL and acknowledged immediately without being
applied to segments. When the cache refreshes, it reads segment data
which may not yet reflect the pending WAL operations, causing the size
check to see stale values and not reject the request.
Adding wait=true ensures operations are applied to segments before the
response returns, so the cache refresh sees the correct sizes.
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The documented format `2023-02-08T10:49` was not accepted because the
parser only had `%Y-%m-%d %H:%M` (space separator) but was missing
`%Y-%m-%dT%H:%M` (T separator). Add the missing format variant and
tests.
Closes#8718
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* Add empty placeholder vector storage types for named vector CRUD
Introduce EmptyDenseVectorStorage and EmptySparseVectorStorage as
placeholder storages for newly created named vectors on immutable
segments. These report all vectors as deleted, consume no disk space,
and are reconstructed from segment config on load via the new
VectorStorageType::Empty and SparseVectorStorageType::Empty variants.
Key design decisions:
- is_on_disk is derived from original user config, not hardcoded
- MultiVectorConfig is preserved for multi-vector support
- Config mismatch optimizer skips Empty storage to avoid false rebuilds
- Quantization delegates normally (handles 0 vectors gracefully)
- get_vector includes debug_assert to catch unexpected access
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] segment-level operations for creating and deleting anmed vectors
* [AI] implement named vector creation and deleting in proxy segment
* [AI] Step 3: Proxy Segment Handling for Named Vector Operations
* [AI] implement for Edge
* [AI] implement consensus operations for named vector operations
* [AI] refactor VectorNameConfig, remove VectorNameConfigInternal
* [AI] handle vector schema inconsistency in raft snapshot recovery
* [AI] rest + grpc API
* [AI] clippy
* [AI] generate openAPI schema
* fmt
* ci fixes
* [AI] fix jwt access test
* [AI] nop operation for awaiting of consensus-commited update ops
* [AI] move vector name operations into points service
* [AI] implement internal api for vector name operations
* [AI] change collection-level config along with segment level operation
* [AI] vector schema reconceliation instead of error
* fmt
* missing compile-time option
* [AI] integration test
* [AI] fix missing JWT tests
* [AI] remove NOP
* [AI] openapi test
* [AI] fix initialization of mutable segment
* [AI] more simple integration tests
* fmt
* [AI] make cluster test a bit harder
* [AI] make test less flacky
* [AI] rabbit comments
* [AI] check params compatibility before writing vector config
* [AI] make sure to register vector storages in structure payload index
* [AI] vector name validation
* lower vector length validation to 200 chars to account for prefix in filename
* [AI] proxy segment: prevent stale data leak through optimization
* fmt
* [AI] filter out removed vectors from proxy response
* [AI] handle vector name in proxy
* fmt
* adjust proxy info based on dropped vectors
* [AI] proxy segment: update filters to correct has_vector condition
* fmt
* clippy
* Fix consensus snapshot applicaiton for vector schema
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix empty sparse vector name validation
* Reorder validation to check empty sparse name before duplicate name
---------
Co-authored-by: leohenon <77656081+lhenon999@users.noreply.github.com>
* feat: add search_max_batchsize to strict mode config
* added test case for search_max_batchsize
* Changes for fixing CI issue dure openapi
* Modify check_strict_mode_batch
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Don't account for deferred points in some places
# Conflicts:
# lib/collection/src/shards/local_shard/scroll.rs
* Add in QueryContext
* Cover more places
* Coderabbit review remarks
* Properly count amount of deleted deferred points (#8386)
* Properly count amount of deleted deferred points
* Prevent double-counting of the same point
* Remove hints to estimations
* Properly handle counts in ProxySegment
* Add tests and fix deleted point count issue
* Adjusts tests + fix issues
* Separte fields for deferred points in telemetry (SegmentInfo)
* Remove deferred_points_count()
* openapi
* Fix test by manually calculating visible points
* Adjust ProxySegment test to revertion of SegmentInfo
* Throw error if collection was not found in telemetry (e2e Test)
* Update lib/segment/src/segment/segment_ops.rs
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Make new fields in API optional
* Don't take range if no deferred point exist
* Review remarks
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* restrict recovery from snapshot folder
* fmt
* clippy
* Extend test, path must be inside snapshots directory
* In release builds, hide detailed tar error message to not leak contents
* Change from bad request to forbidden
* fix(auth_tests): place snapshot in peer snapshots dir for recover_collection_snapshot
The recover from snapshot API now requires file:// paths to be inside the
configured snapshots directory. Write the test snapshot into the peer's
snapshots directory instead of a temp file so the request is allowed.
Made-with: Cursor
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Cursor Agent <agent@cursor.com>
* [manual] make untagged enum for consistent API for FeedbackStrategy
* add #[validate(nested)]
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Add integration test to assert all queued updates are also transferred
* Plunge update queue in stream records transfers
* Migrate existing plunger usages to new plunge helper
* Skip test if not compiled with staging flag
* Only send delay operation when staging feature is enabled
* Reformat
* Fix review remarks
* add rest and grpc interfaces
Also handle inference for this query
* follow refactor from base branch
* renaming from Ms Cooper
* get started on validation testing
* more validations
* test equivalence with query when less than 2 feedback elements
* rename feedback query to relevance_feedback query
* fix extraction of context pairs
* rename feedback vector to example
* make coderabbit happier
* upd test
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* handle `score_threshold` in Formula queries
* AI: add openapi test
prompt: Upload 4 points with a numeric payload, use the payload as the
score, and set a score threshold. We can assert which ids should be in
the result and which shouldn't
* weighted rrf implementation
* test
* fmt
* fix edge
* validate number of sources and number of weights
* do not partial match
* upd schema
* review fixes
* update formula
* remove calcualtions from tests
* update comment, because AI have OCD
* fmt
* Update queue info into CollectionInfo
* basic test
* fix
* cleaner
* Update lib/collection/src/shards/local_shard/mod.rs
Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
* fix the fix
* rename to op_num
* skip_serializing_if just in case
* first step
* test queue length with staging feature
* gate integration test based on binary feature
---------
Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
* introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert
* add test
* upd dockstring
* require resharding once all peers have updated version
* use service error
* fix clippy again
* wait for same version before resharding in tests
* feat: Add enable_hnsw option for payload field indexes
Add optional enable_hnsw parameter to all payload index types to control
whether additional HNSW graph links are built for each indexed field.
- Add enable_hnsw field to all 8 payload index param types
- Update gRPC proto definitions and conversions
- Update OpenAPI schema
- Modify HNSW graph builder to respect enable_hnsw flag
- Add enable_hnsw() helper methods to PayloadSchemaParams and PayloadFieldSchema
- Update all tests to include new field (default: None)
When enable_hnsw is true and payload_M > 0, additional HNSW links will
be built for the payload field. Default value is true for backward compatibility.
* Fix Some format problems
* fix: address comment problem
---------
Co-authored-by: EC2 Default User <ec2-user@ip-10-78-171-148.ec2.internal>
* Add progress_tracker.rs
* Pass progress tracker around
* Populate progress tracker with actual data
* Expose progress on `/collections/{name}/optimizations` endpoint
* Add vector count per vector-name to metrics API
* Add to metrics API
* Improve TinyMap::get_or_insert_default and add tests
* Minor improvements
* Update openapi
* Review remarks
* Remove `collection_vectors` since it can be calculated manually
