35 Commits

Author SHA1 Message Date
Andrey Vasnetsov
8a7325ad5d Remove deprecated search endpoints from OpenAPI, deprecate them in gRPC (#9982)
* Remove deprecated search/recommend/discover endpoints from OpenAPI

Remove deprecated REST API endpoint definitions from the OpenAPI
generator. These endpoints were deprecated in v1.13.3 (`f4ced2567`,
#5907, 2025-01-30) in favor of the universal `/points/query` endpoint:

- POST /points/search
- POST /points/search/batch
- POST /points/search/groups
- POST /points/recommend
- POST /points/recommend/batch
- POST /points/recommend/groups
- POST /points/discover
- POST /points/discover/batch

Also removes the corresponding request types from the schema generator
and updates the expected API count in the consistency check.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Migrate OpenAPI integration tests to /points/query

The deprecated /points/search, /points/recommend and /points/discover
endpoints (along with their /batch and /groups variants) were removed
from the OpenAPI spec, which caused validation failures in the Python
integration test harness.

This commit migrates the affected tests to the universal /points/query
endpoint:

- Delete tests dedicated to the deprecated endpoints:
  test_recommend.py, test_discover.py, test_multicollection_reco.py,
  test_recommendation_multivector.py
- Refactor remaining tests to call /points/query (and /query/batch,
  /query/groups), translating request bodies (vector -> query / using,
  positive/negative -> query.recommend, target/context -> query.discover)
  and unwrapping the new result.points response shape.
- Drop equivalence assertions against the now-removed legacy endpoints.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Relax non-empty assertions in migrated recommend/discover tests

The previous migration added `len(...) > 0` assertions to tests that
previously only checked equivalence between the deprecated and new
API. These assertions are too strict because the parametrized
`query_filter` cases legitimately produce empty result sets.

Drop the `> 0` assertion and rely on `request_with_validation` to
verify the response is well-formed and HTTP OK.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Migrate remaining OpenAPI tests off deprecated search endpoints

Tests added to dev after the original migration was written still call
/points/search and /points/recommend/groups through
`request_with_validation`, which resolves the endpoint against the
OpenAPI spec and therefore breaks once the endpoint is not in the spec:

- test_turbo4_storage.py, test_sparse_idf_corpus.py, test_validation.py:
  translate /points/search to /points/query (vector{name,vector} ->
  query + using, result -> result.points).
- test_group.py: drop the /points/recommend/groups half of the
  lookup_from validation test in favour of the query equivalent.

test_sparse_idf_corpus.py's test_query_api_supports_idf_corpus goes
away: with the helper on /points/query every test in the file now
exercises what it asserted.

Also record why test_recommend_group cannot assert on its groups: it
uses every point in the collection as a recommend example, so all of
them are excluded and the result is legitimately empty.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Regenerate openapi.json without the deprecated search endpoints

Drops the 8 deprecated paths and the request schemas that only they
referenced: Search/Recommend/Discover request (+Batch, +Groups) types
and their exclusive dependencies (NamedVector, NamedSparseVector,
NamedVectorStruct, UsingVector, RecommendExample, ContextExamplePair).

Regenerated output is a strict subset of the previous spec, and every
remaining $ref still resolves.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Deprecate the search/recommend/discover RPCs in gRPC

The REST counterparts have carried `deprecated: true` since v1.13.3 and
are now gone from the OpenAPI spec, while the gRPC RPCs never got any
deprecation annotation at all. Mark all 8 with `option deprecated = true`
so generated clients warn, and point each doc comment at its `Query`
replacement.

tonic puts `#[deprecated]` on the generated client methods only; the
server trait gets the doc comment alone, so our own `impl` is unaffected.
The RPCs keep serving traffic — this is annotation only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Restore the deleted recommend/discover suites on /points/query

The earlier migration deleted these four files outright, but the
query-side tests it left behind are all shallow smoke tests
(`len(result) > 0`, `"points" in result[0]`). The deleted ones carried
invariants with no query-API equivalent anywhere, so deleting them was a
real loss of coverage rather than de-duplication:

- test_recommend.py: default strategy equals average_vector; batch
  results identical to sequential singles across six request shapes;
  best_score with only negatives yields all-negative scores; best_score
  with a single positive orders identically to a nearest query; raw
  vectors as examples equal ids as examples.
- test_discover.py: context-only scores are all <= 0; target-only orders
  identically to a nearest query but scores differently; with a fixed
  context the integer part of the score is stable while the decimal part
  moves, and vice versa with a fixed target; batch equals singles;
  lookup_from by id equals by vector.
- test_multicollection_reco.py: cross-collection lookup_from, plus
  wrong-vector-size, unknown-collection and unknown-vector rejections.
- test_recommendation_multivector.py: the same recommend invariants over
  a max_sim multivector collection, which the query suite never covered.

Only test_recommend_missing_lookup_from_collection_with_raw_vector is
dropped as genuinely redundant — test_query.py's
test_query_missing_lookup_from_collection covers query, query/batch and
prefetch.

Two request-shape differences the translation had to absorb:

- Giving no examples at all is 422 (a RecommendInput validation rule),
  where the legacy API reported 400 from the query itself. A malformed
  example, such as an empty vector, is still 400.
- DiscoverInput requires the `context` key and accepts only an explicit
  null to mean "no context", so target-only discover must spell it out.
  The legacy API let it be omitted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-27 18:09:43 +02:00
qdrant-cloud-bot
be82eceb40 Fix flaky strict mode collection size tests (#8922)
These tests rely on the collection size stats cache being refreshed to
detect that a size limit has been exceeded. Without wait=true, upsert
operations are written to WAL and acknowledged immediately without being
applied to segments. When the cache refreshes, it reads segment data
which may not yet reflect the pending WAL operations, causing the size
check to see stale values and not reject the request.

Adding wait=true ensures operations are applied to segments before the
response returns, so the cache refresh sees the correct sizes.

Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-06 13:14:06 +02:00
Bhagirath Kapdi
fb704e5d13 Feature/strict mode search max batchsize (#8469)
* feat: add search_max_batchsize to strict mode config

* added test case for search_max_batchsize

* Changes for fixing CI issue dure openapi

* Modify check_strict_mode_batch

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2026-03-26 16:26:09 +01:00
Arnaud Gourlay
bc13f01680 [strict-mode] Add max number of payload index count (#7222)
* [strict-mode] Add max number of payload index count

* improve docs and validation for StrictModeConfig

* fix from coderabbit

* test blocking access to payload indexes

* polish test
2025-09-15 13:12:40 +02:00
Luis Cossío
6919e3ae5b [phrase matching] expose phrase condition (#6670)
* add `"match": { "phrase": ... }` condition

* gen grpc and openapi

* [phrase matching] expose `phrase_matching` flag in rest and grpc (#6620)

* expose setting in rest and grpc

* phrase matching openapi test

* regen openapi

* [phrase matching] Text index fixes (#6730)

* allow rocksdb-based immutable text index

* fix repeated-token phrases

* fmt

* Update OpenAPI spec

---------

Co-authored-by: timvisee <tim@visee.me>

* add repeated word case in openapi test

* [phrase match | strict mode] Allow phrase condition when enabled in index (#6749)

* allow phrase filter when index is present

* prettier error message

* clippppppy

---------

Co-authored-by: timvisee <tim@visee.me>
2025-06-24 16:56:36 -04:00
Arnaud Gourlay
2bba1017d9 Strict mode allows fullscan for multitenant payload index (#6498)
* Strict mode allows fullscan for multitenant payload index

* different error for missing payload index in multitenant case

* handle payload_m

* add simple test

* handle hnsw.m not set

* add another simple test

* fix test

* fallback to global HSNW config

* new error status code

* do not block on global HNSW and improver error reporting

* clearer error reporting

* review fixes

* fix test error messages

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-05-22 10:42:47 +02:00
Arnaud Gourlay
0903e920bd Fix UnindexedField infra to handle lookup&range index requirements (#6496)
* Fix UnindexedField infra to handle lookup&range index requirements

* more generic approach

* better test

* restore

* small unit test for sanity

* review nits

* Do not recommend parametrized index

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2025-05-20 16:06:01 +02:00
Andrey Vasnetsov
b92a322eb1 Strict mode error is now 400 instead of 403 (#6560)
* strcit mode error is now 400 instead of 403

* fix test
2025-05-20 15:19:36 +02:00
Arnaud Gourlay
1bdf1f34ae Strict mode detects full scan on query (#6473) 2025-05-05 15:59:13 +02:00
Arnaud Gourlay
b8698cf88b Fix strict mode unindexed group_by path (#6363)
* Fix strict mode unindexed group_by path

* check index schema for matching support

* Handle disabled lookup on integer index
2025-05-02 15:19:50 +02:00
Luis Cossío
56addec5be [rate limits] consider length of multivectors for query cost (#6356)
* consider length of multivectors for rate limiter

* add openapi test

* organize imports

* better cost estimation for sparse vectors

* restore groups openapi test

* address @timvisee's review

* Fix consensus test
2025-04-22 14:10:21 -04:00
Arnaud Gourlay
8d54eba487 Fix strict mode query limit for distance matrix API (#6354) 2025-04-10 09:52:25 +02:00
Arnaud Gourlay
e617b9fa20 Fix strict mode query limit for group APIs (#6353) 2025-04-09 20:24:23 +02:00
Arnaud Gourlay
932ac134e8 Read rate limiter charge per filter condition (#6333) 2025-04-07 19:01:54 +02:00
Arnaud Gourlay
d46c4bf099 Rate limit retrieve based on number of ids (#6327)
* Rate limit retrieve based on number of ids

* fmt

* Better test
2025-04-07 11:31:42 +02:00
Luis Cossío
f9a537174c [strict mode | score boosting] Strict mode for formula queries (#6317)
* move problems.rs to collection crate

* change type on datetime_key expression

* extract from expression

* check for unindexed fields in formula

* clippy

* dedup code

* Only implement verification for `CollectionQuery` types (#6319)

* verify CollectionQuery's Prefetch

* remove implementation for REST query

* remove implementation for REST query groups

* add basic openapi test
2025-04-04 13:52:01 -03:00
Arnaud Gourlay
fd9215858e Rate limite recommendation based on number of examples (#6324) 2025-04-04 15:53:14 +02:00
Arnaud Gourlay
1a0862ebd3 Write rate limit batch update per point count (#6152) 2025-03-11 16:10:07 +01:00
Arnaud Gourlay
58039f9f2e Unset strict config fields (#6120) 2025-03-10 15:32:21 +01:00
Arnaud Gourlay
1a7e2701d7 Add strict mode for max points count (#5967)
* Add strict mode for max points count

* clarify estimated nature
2025-02-12 11:10:51 +01:00
Arnaud Gourlay
d4e0aaa0d7 Retry-After on rate limiter hit for REST API (#5917)
* Retry-After on rate limiter hit for REST API

* fix tests

* Ceil retry after seconds number so clients don't retry too early

* Relax retry after time requirement a bit in case CI machines are slow

* Add rate limit test, test retry after if we replanish within a second

* Report time to wait in rate limit error

---------

Co-authored-by: timvisee <tim@visee.me>
2025-02-10 11:27:18 +01:00
n0x29a
1796c76e35 Strict mode sparsevector (#5841)
* Strict mode sparse vectors
* Update gRPC docs
* Use deny_unknown_fields (#5868)

---------

Co-authored-by: n0x29a <n0x29a@example.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2025-01-23 17:34:38 +00:00
Arnaud Gourlay
4dee0c5bf4 Strict mode multivector (#5757)
* strict mode for multivector

* simplify impl

* regen docs
2025-01-20 15:53:35 +01:00
Arnaud Gourlay
580c9f249a Migrate strict mode payload test for mmap (#5793) 2025-01-14 13:22:50 +01:00
Jojii
3e96fae333 [Strict mode] limits for filter and conditions (#5754)
* Add limits for filter and conditions

* clippy

* Review remarks + nested condition test

* Fix opnapi specs

* Improve error message by giving info about limits and usage
2025-01-14 11:07:27 +01:00
Jojii
0391891037 Merge pull request #5729
* Consider filter-cardinality as cost in update ratelimiter

* Use OperationEffectArea and improve rate limit error message

* Clippy

* Fix test
2025-01-08 17:12:10 +00:00
Arnaud Gourlay
430c068f79 Reshape read rate limiter and fix disabled mode (#5683) 2024-12-20 16:56:17 +01:00
Arnaud Gourlay
1c1a0d0b39 Fix rate limiting of internal update operations (#5653)
* Fix rate limiting of internal update operations

* code review

* write_rate_limiter turned Option<Mutex> and fix disabling mode

* Update TODO tag

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
2024-12-20 12:59:07 +01:00
Arnaud Gourlay
229fa94f69 Strict mode for payload storage (#5588)
* Strict mode for payload storage

* Don't increment counter multiple times per request

* Add (loose) integration tests for payload storage limit

* Minor improvements

* minor renaming

* Update lib/api/src/grpc/proto/collections.proto

Co-authored-by: Tim Visée <tim+github@visee.me>

---------

Co-authored-by: jojii <jojii@gmx.net>
Co-authored-by: Jojii <15957865+JojiiOfficial@users.noreply.github.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
2024-12-17 17:33:46 +01:00
Arnaud Gourlay
2a2a7033db Rate limit requests per minute (#5597)
* Rate limit requests per minute

* rename to remove time unit for API
2024-12-10 00:26:37 +01:00
Arnaud Gourlay
1e21a4a179 Rate limiting for shard operations (#5582)
* Rate limiting for shard operations

* address all review comments in one go
2024-12-06 11:02:18 +01:00
Jojii
42fd2e27e2 Strict mode max collection vector size (#5501)
* Strict mode config: Max collection size

* api specs

* Add tests + set/update payload check

* Improve function names and add comments

* rename config to separate vectors and payload

* fix tests

* Adjust configs docs

* add benchmark

* improve performance by caching shard info

* add bench for size_info() and fix tests

* Also limit the batch-size for vector updates (#5508)

* Also limit the batch-size for vector updates

* clippy

* add lost commit

* Load cache on collection initialization

* add unit type to parameter name

* fix renaming in test

* clearer error message

* fix test

* review remarks

* remove unused function for now

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2024-12-05 17:28:09 +01:00
Jojii
5417428951 Add max upsert batch size strict mode option (#5485) 2024-11-19 19:06:57 +01:00
Predrag Knezevic
9d67c6325b Test collections unique per openapi test module (#5384)
Collection name under test is equal to the test module name, without `.py` suffix.

* Helps by debugging/tracing failed tests and find relevant logs lines in qdrant log files
* Opens up a possibility to run tests in parallel, given that there are no data sharing
  between test modules

Change details:
* defined module scoped `collection_name` fixture in `conftest.py`
* removed `collection_name` module variable
* each test signature modified to declare the dependency to `collection_name` fixture
* `@pytest.mark.parametrize` migrated to `@pytest-cases.parametrize` in cases when
  `collection_name` was used as the value
2024-11-06 20:09:31 +01:00
Jojii
27883dbb41 Add strict mode config to update collection API + Populate strict mode in API (#5187)
* add strict mode to update collection API

* update API specs

* clippy

* Update lib/storage/src/content_manager/toc/collection_meta_ops.rs

* Strict mode integration tests (#5189)

* Add integration tests for strict mode

* add full update test

* Apply suggestions from code review

Co-authored-by: Tim Visée <tim+github@visee.me>

* Adjust error messages

* improve checking of error response

---------

Co-authored-by: Tim Visée <tim+github@visee.me>

* prefer explicit structu conversion

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
2024-10-14 10:27:00 +02:00