Commit Graph

34 Commits

Author SHA1 Message Date
qdrant-cloud-bot
be82eceb40 Fix flaky strict mode collection size tests (#8922)
These tests rely on the collection size stats cache being refreshed to
detect that a size limit has been exceeded. Without wait=true, upsert
operations are written to WAL and acknowledged immediately without being
applied to segments. When the cache refreshes, it reads segment data
which may not yet reflect the pending WAL operations, causing the size
check to see stale values and not reject the request.

Adding wait=true ensures operations are applied to segments before the
response returns, so the cache refresh sees the correct sizes.

Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-06 13:14:06 +02:00
Bhagirath Kapdi
fb704e5d13 Feature/strict mode search max batchsize (#8469)
* feat: add search_max_batchsize to strict mode config

* added test case for search_max_batchsize

* Changes for fixing CI issue dure openapi

* Modify check_strict_mode_batch

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2026-03-26 16:26:09 +01:00
Arnaud Gourlay
bc13f01680 [strict-mode] Add max number of payload index count (#7222)
* [strict-mode] Add max number of payload index count

* improve docs and validation for StrictModeConfig

* fix from coderabbit

* test blocking access to payload indexes

* polish test
2025-09-15 13:12:40 +02:00
Luis Cossío
6919e3ae5b [phrase matching] expose phrase condition (#6670)
* add `"match": { "phrase": ... }` condition

* gen grpc and openapi

* [phrase matching] expose `phrase_matching` flag in rest and grpc (#6620)

* expose setting in rest and grpc

* phrase matching openapi test

* regen openapi

* [phrase matching] Text index fixes (#6730)

* allow rocksdb-based immutable text index

* fix repeated-token phrases

* fmt

* Update OpenAPI spec

---------

Co-authored-by: timvisee <tim@visee.me>

* add repeated word case in openapi test

* [phrase match | strict mode] Allow phrase condition when enabled in index (#6749)

* allow phrase filter when index is present

* prettier error message

* clippppppy

---------

Co-authored-by: timvisee <tim@visee.me>
2025-06-24 16:56:36 -04:00
Arnaud Gourlay
2bba1017d9 Strict mode allows fullscan for multitenant payload index (#6498)
* Strict mode allows fullscan for multitenant payload index

* different error for missing payload index in multitenant case

* handle payload_m

* add simple test

* handle hnsw.m not set

* add another simple test

* fix test

* fallback to global HSNW config

* new error status code

* do not block on global HNSW and improver error reporting

* clearer error reporting

* review fixes

* fix test error messages

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-05-22 10:42:47 +02:00
Arnaud Gourlay
0903e920bd Fix UnindexedField infra to handle lookup&range index requirements (#6496)
* Fix UnindexedField infra to handle lookup&range index requirements

* more generic approach

* better test

* restore

* small unit test for sanity

* review nits

* Do not recommend parametrized index

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2025-05-20 16:06:01 +02:00
Andrey Vasnetsov
b92a322eb1 Strict mode error is now 400 instead of 403 (#6560)
* strcit mode error is now 400 instead of 403

* fix test
2025-05-20 15:19:36 +02:00
Arnaud Gourlay
1bdf1f34ae Strict mode detects full scan on query (#6473) 2025-05-05 15:59:13 +02:00
Arnaud Gourlay
b8698cf88b Fix strict mode unindexed group_by path (#6363)
* Fix strict mode unindexed group_by path

* check index schema for matching support

* Handle disabled lookup on integer index
2025-05-02 15:19:50 +02:00
Luis Cossío
56addec5be [rate limits] consider length of multivectors for query cost (#6356)
* consider length of multivectors for rate limiter

* add openapi test

* organize imports

* better cost estimation for sparse vectors

* restore groups openapi test

* address @timvisee's review

* Fix consensus test
2025-04-22 14:10:21 -04:00
Arnaud Gourlay
8d54eba487 Fix strict mode query limit for distance matrix API (#6354) 2025-04-10 09:52:25 +02:00
Arnaud Gourlay
e617b9fa20 Fix strict mode query limit for group APIs (#6353) 2025-04-09 20:24:23 +02:00
Arnaud Gourlay
932ac134e8 Read rate limiter charge per filter condition (#6333) 2025-04-07 19:01:54 +02:00
Arnaud Gourlay
d46c4bf099 Rate limit retrieve based on number of ids (#6327)
* Rate limit retrieve based on number of ids

* fmt

* Better test
2025-04-07 11:31:42 +02:00
Luis Cossío
f9a537174c [strict mode | score boosting] Strict mode for formula queries (#6317)
* move problems.rs to collection crate

* change type on datetime_key expression

* extract from expression

* check for unindexed fields in formula

* clippy

* dedup code

* Only implement verification for `CollectionQuery` types (#6319)

* verify CollectionQuery's Prefetch

* remove implementation for REST query

* remove implementation for REST query groups

* add basic openapi test
2025-04-04 13:52:01 -03:00
Arnaud Gourlay
fd9215858e Rate limite recommendation based on number of examples (#6324) 2025-04-04 15:53:14 +02:00
Arnaud Gourlay
1a0862ebd3 Write rate limit batch update per point count (#6152) 2025-03-11 16:10:07 +01:00
Arnaud Gourlay
58039f9f2e Unset strict config fields (#6120) 2025-03-10 15:32:21 +01:00
Arnaud Gourlay
1a7e2701d7 Add strict mode for max points count (#5967)
* Add strict mode for max points count

* clarify estimated nature
2025-02-12 11:10:51 +01:00
Arnaud Gourlay
d4e0aaa0d7 Retry-After on rate limiter hit for REST API (#5917)
* Retry-After on rate limiter hit for REST API

* fix tests

* Ceil retry after seconds number so clients don't retry too early

* Relax retry after time requirement a bit in case CI machines are slow

* Add rate limit test, test retry after if we replanish within a second

* Report time to wait in rate limit error

---------

Co-authored-by: timvisee <tim@visee.me>
2025-02-10 11:27:18 +01:00
n0x29a
1796c76e35 Strict mode sparsevector (#5841)
* Strict mode sparse vectors
* Update gRPC docs
* Use deny_unknown_fields (#5868)

---------

Co-authored-by: n0x29a <n0x29a@example.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2025-01-23 17:34:38 +00:00
Arnaud Gourlay
4dee0c5bf4 Strict mode multivector (#5757)
* strict mode for multivector

* simplify impl

* regen docs
2025-01-20 15:53:35 +01:00
Arnaud Gourlay
580c9f249a Migrate strict mode payload test for mmap (#5793) 2025-01-14 13:22:50 +01:00
Jojii
3e96fae333 [Strict mode] limits for filter and conditions (#5754)
* Add limits for filter and conditions

* clippy

* Review remarks + nested condition test

* Fix opnapi specs

* Improve error message by giving info about limits and usage
2025-01-14 11:07:27 +01:00
Jojii
0391891037 Merge pull request #5729
* Consider filter-cardinality as cost in update ratelimiter

* Use OperationEffectArea and improve rate limit error message

* Clippy

* Fix test
2025-01-08 17:12:10 +00:00
Arnaud Gourlay
430c068f79 Reshape read rate limiter and fix disabled mode (#5683) 2024-12-20 16:56:17 +01:00
Arnaud Gourlay
1c1a0d0b39 Fix rate limiting of internal update operations (#5653)
* Fix rate limiting of internal update operations

* code review

* write_rate_limiter turned Option<Mutex> and fix disabling mode

* Update TODO tag

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
2024-12-20 12:59:07 +01:00
Arnaud Gourlay
229fa94f69 Strict mode for payload storage (#5588)
* Strict mode for payload storage

* Don't increment counter multiple times per request

* Add (loose) integration tests for payload storage limit

* Minor improvements

* minor renaming

* Update lib/api/src/grpc/proto/collections.proto

Co-authored-by: Tim Visée <tim+github@visee.me>

---------

Co-authored-by: jojii <jojii@gmx.net>
Co-authored-by: Jojii <15957865+JojiiOfficial@users.noreply.github.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
2024-12-17 17:33:46 +01:00
Arnaud Gourlay
2a2a7033db Rate limit requests per minute (#5597)
* Rate limit requests per minute

* rename to remove time unit for API
2024-12-10 00:26:37 +01:00
Arnaud Gourlay
1e21a4a179 Rate limiting for shard operations (#5582)
* Rate limiting for shard operations

* address all review comments in one go
2024-12-06 11:02:18 +01:00
Jojii
42fd2e27e2 Strict mode max collection vector size (#5501)
* Strict mode config: Max collection size

* api specs

* Add tests + set/update payload check

* Improve function names and add comments

* rename config to separate vectors and payload

* fix tests

* Adjust configs docs

* add benchmark

* improve performance by caching shard info

* add bench for size_info() and fix tests

* Also limit the batch-size for vector updates (#5508)

* Also limit the batch-size for vector updates

* clippy

* add lost commit

* Load cache on collection initialization

* add unit type to parameter name

* fix renaming in test

* clearer error message

* fix test

* review remarks

* remove unused function for now

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2024-12-05 17:28:09 +01:00
Jojii
5417428951 Add max upsert batch size strict mode option (#5485) 2024-11-19 19:06:57 +01:00
Predrag Knezevic
9d67c6325b Test collections unique per openapi test module (#5384)
Collection name under test is equal to the test module name, without `.py` suffix.

* Helps by debugging/tracing failed tests and find relevant logs lines in qdrant log files
* Opens up a possibility to run tests in parallel, given that there are no data sharing
  between test modules

Change details:
* defined module scoped `collection_name` fixture in `conftest.py`
* removed `collection_name` module variable
* each test signature modified to declare the dependency to `collection_name` fixture
* `@pytest.mark.parametrize` migrated to `@pytest-cases.parametrize` in cases when
  `collection_name` was used as the value
2024-11-06 20:09:31 +01:00
Jojii
27883dbb41 Add strict mode config to update collection API + Populate strict mode in API (#5187)
* add strict mode to update collection API

* update API specs

* clippy

* Update lib/storage/src/content_manager/toc/collection_meta_ops.rs

* Strict mode integration tests (#5189)

* Add integration tests for strict mode

* add full update test

* Apply suggestions from code review

Co-authored-by: Tim Visée <tim+github@visee.me>

* Adjust error messages

* improve checking of error response

---------

Co-authored-by: Tim Visée <tim+github@visee.me>

* prefer explicit structu conversion

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
2024-10-14 10:27:00 +02:00