33 Commits

Author SHA1 Message Date
Bhagirath Kapdi
fb704e5d13 Feature/strict mode search max batchsize (#8469)
* feat: add search_max_batchsize to strict mode config

* added test case for search_max_batchsize

* Changes for fixing CI issue dure openapi

* Modify check_strict_mode_batch

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2026-03-26 16:26:09 +01:00
Arnaud Gourlay
bc13f01680 [strict-mode] Add max number of payload index count (#7222)
* [strict-mode] Add max number of payload index count

* improve docs and validation for StrictModeConfig

* fix from coderabbit

* test blocking access to payload indexes

* polish test
2025-09-15 13:12:40 +02:00
Luis Cossío
6919e3ae5b [phrase matching] expose phrase condition (#6670)
* add `"match": { "phrase": ... }` condition

* gen grpc and openapi

* [phrase matching] expose `phrase_matching` flag in rest and grpc (#6620)

* expose setting in rest and grpc

* phrase matching openapi test

* regen openapi

* [phrase matching] Text index fixes (#6730)

* allow rocksdb-based immutable text index

* fix repeated-token phrases

* fmt

* Update OpenAPI spec

---------

Co-authored-by: timvisee <tim@visee.me>

* add repeated word case in openapi test

* [phrase match | strict mode] Allow phrase condition when enabled in index (#6749)

* allow phrase filter when index is present

* prettier error message

* clippppppy

---------

Co-authored-by: timvisee <tim@visee.me>
2025-06-24 16:56:36 -04:00
Arnaud Gourlay
2bba1017d9 Strict mode allows fullscan for multitenant payload index (#6498)
* Strict mode allows fullscan for multitenant payload index

* different error for missing payload index in multitenant case

* handle payload_m

* add simple test

* handle hnsw.m not set

* add another simple test

* fix test

* fallback to global HSNW config

* new error status code

* do not block on global HNSW and improver error reporting

* clearer error reporting

* review fixes

* fix test error messages

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-05-22 10:42:47 +02:00
Arnaud Gourlay
0903e920bd Fix UnindexedField infra to handle lookup&range index requirements (#6496)
* Fix UnindexedField infra to handle lookup&range index requirements

* more generic approach

* better test

* restore

* small unit test for sanity

* review nits

* Do not recommend parametrized index

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2025-05-20 16:06:01 +02:00
Andrey Vasnetsov
b92a322eb1 Strict mode error is now 400 instead of 403 (#6560)
* strcit mode error is now 400 instead of 403

* fix test
2025-05-20 15:19:36 +02:00
Arnaud Gourlay
1bdf1f34ae Strict mode detects full scan on query (#6473) 2025-05-05 15:59:13 +02:00
Arnaud Gourlay
b8698cf88b Fix strict mode unindexed group_by path (#6363)
* Fix strict mode unindexed group_by path

* check index schema for matching support

* Handle disabled lookup on integer index
2025-05-02 15:19:50 +02:00
Luis Cossío
56addec5be [rate limits] consider length of multivectors for query cost (#6356)
* consider length of multivectors for rate limiter

* add openapi test

* organize imports

* better cost estimation for sparse vectors

* restore groups openapi test

* address @timvisee's review

* Fix consensus test
2025-04-22 14:10:21 -04:00
Arnaud Gourlay
8d54eba487 Fix strict mode query limit for distance matrix API (#6354) 2025-04-10 09:52:25 +02:00
Arnaud Gourlay
e617b9fa20 Fix strict mode query limit for group APIs (#6353) 2025-04-09 20:24:23 +02:00
Arnaud Gourlay
932ac134e8 Read rate limiter charge per filter condition (#6333) 2025-04-07 19:01:54 +02:00
Arnaud Gourlay
d46c4bf099 Rate limit retrieve based on number of ids (#6327)
* Rate limit retrieve based on number of ids

* fmt

* Better test
2025-04-07 11:31:42 +02:00
Luis Cossío
f9a537174c [strict mode | score boosting] Strict mode for formula queries (#6317)
* move problems.rs to collection crate

* change type on datetime_key expression

* extract from expression

* check for unindexed fields in formula

* clippy

* dedup code

* Only implement verification for `CollectionQuery` types (#6319)

* verify CollectionQuery's Prefetch

* remove implementation for REST query

* remove implementation for REST query groups

* add basic openapi test
2025-04-04 13:52:01 -03:00
Arnaud Gourlay
fd9215858e Rate limite recommendation based on number of examples (#6324) 2025-04-04 15:53:14 +02:00
Arnaud Gourlay
1a0862ebd3 Write rate limit batch update per point count (#6152) 2025-03-11 16:10:07 +01:00
Arnaud Gourlay
58039f9f2e Unset strict config fields (#6120) 2025-03-10 15:32:21 +01:00
Arnaud Gourlay
1a7e2701d7 Add strict mode for max points count (#5967)
* Add strict mode for max points count

* clarify estimated nature
2025-02-12 11:10:51 +01:00
Arnaud Gourlay
d4e0aaa0d7 Retry-After on rate limiter hit for REST API (#5917)
* Retry-After on rate limiter hit for REST API

* fix tests

* Ceil retry after seconds number so clients don't retry too early

* Relax retry after time requirement a bit in case CI machines are slow

* Add rate limit test, test retry after if we replanish within a second

* Report time to wait in rate limit error

---------

Co-authored-by: timvisee <tim@visee.me>
2025-02-10 11:27:18 +01:00
n0x29a
1796c76e35 Strict mode sparsevector (#5841)
* Strict mode sparse vectors
* Update gRPC docs
* Use deny_unknown_fields (#5868)

---------

Co-authored-by: n0x29a <n0x29a@example.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2025-01-23 17:34:38 +00:00
Arnaud Gourlay
4dee0c5bf4 Strict mode multivector (#5757)
* strict mode for multivector

* simplify impl

* regen docs
2025-01-20 15:53:35 +01:00
Arnaud Gourlay
580c9f249a Migrate strict mode payload test for mmap (#5793) 2025-01-14 13:22:50 +01:00
Jojii
3e96fae333 [Strict mode] limits for filter and conditions (#5754)
* Add limits for filter and conditions

* clippy

* Review remarks + nested condition test

* Fix opnapi specs

* Improve error message by giving info about limits and usage
2025-01-14 11:07:27 +01:00
Jojii
0391891037 Merge pull request #5729
* Consider filter-cardinality as cost in update ratelimiter

* Use OperationEffectArea and improve rate limit error message

* Clippy

* Fix test
2025-01-08 17:12:10 +00:00
Arnaud Gourlay
430c068f79 Reshape read rate limiter and fix disabled mode (#5683) 2024-12-20 16:56:17 +01:00
Arnaud Gourlay
1c1a0d0b39 Fix rate limiting of internal update operations (#5653)
* Fix rate limiting of internal update operations

* code review

* write_rate_limiter turned Option<Mutex> and fix disabling mode

* Update TODO tag

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
2024-12-20 12:59:07 +01:00
Arnaud Gourlay
229fa94f69 Strict mode for payload storage (#5588)
* Strict mode for payload storage

* Don't increment counter multiple times per request

* Add (loose) integration tests for payload storage limit

* Minor improvements

* minor renaming

* Update lib/api/src/grpc/proto/collections.proto

Co-authored-by: Tim Visée <tim+github@visee.me>

---------

Co-authored-by: jojii <jojii@gmx.net>
Co-authored-by: Jojii <15957865+JojiiOfficial@users.noreply.github.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
2024-12-17 17:33:46 +01:00
Arnaud Gourlay
2a2a7033db Rate limit requests per minute (#5597)
* Rate limit requests per minute

* rename to remove time unit for API
2024-12-10 00:26:37 +01:00
Arnaud Gourlay
1e21a4a179 Rate limiting for shard operations (#5582)
* Rate limiting for shard operations

* address all review comments in one go
2024-12-06 11:02:18 +01:00
Jojii
42fd2e27e2 Strict mode max collection vector size (#5501)
* Strict mode config: Max collection size

* api specs

* Add tests + set/update payload check

* Improve function names and add comments

* rename config to separate vectors and payload

* fix tests

* Adjust configs docs

* add benchmark

* improve performance by caching shard info

* add bench for size_info() and fix tests

* Also limit the batch-size for vector updates (#5508)

* Also limit the batch-size for vector updates

* clippy

* add lost commit

* Load cache on collection initialization

* add unit type to parameter name

* fix renaming in test

* clearer error message

* fix test

* review remarks

* remove unused function for now

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2024-12-05 17:28:09 +01:00
Jojii
5417428951 Add max upsert batch size strict mode option (#5485) 2024-11-19 19:06:57 +01:00
Predrag Knezevic
9d67c6325b Test collections unique per openapi test module (#5384)
Collection name under test is equal to the test module name, without `.py` suffix.

* Helps by debugging/tracing failed tests and find relevant logs lines in qdrant log files
* Opens up a possibility to run tests in parallel, given that there are no data sharing
  between test modules

Change details:
* defined module scoped `collection_name` fixture in `conftest.py`
* removed `collection_name` module variable
* each test signature modified to declare the dependency to `collection_name` fixture
* `@pytest.mark.parametrize` migrated to `@pytest-cases.parametrize` in cases when
  `collection_name` was used as the value
2024-11-06 20:09:31 +01:00
Jojii
27883dbb41 Add strict mode config to update collection API + Populate strict mode in API (#5187)
* add strict mode to update collection API

* update API specs

* clippy

* Update lib/storage/src/content_manager/toc/collection_meta_ops.rs

* Strict mode integration tests (#5189)

* Add integration tests for strict mode

* add full update test

* Apply suggestions from code review

Co-authored-by: Tim Visée <tim+github@visee.me>

* Adjust error messages

* improve checking of error response

---------

Co-authored-by: Tim Visée <tim+github@visee.me>

* prefer explicit structu conversion

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
2024-10-14 10:27:00 +02:00