Commit Graph
31 Commits
Author SHA1 Message Date
Luis Cossíoandtimvisee ad936a463c [phrase matching] expose phrase condition (#6670)
* add `"match": { "phrase": ... }` condition

* gen grpc and openapi

* [phrase matching] expose `phrase_matching` flag in rest and grpc (#6620)

* expose setting in rest and grpc

* phrase matching openapi test

* regen openapi

* [phrase matching] Text index fixes (#6730)

* allow rocksdb-based immutable text index

* fix repeated-token phrases

* fmt

* Update OpenAPI spec

---------

Co-authored-by: timvisee <tim@visee.me>

* add repeated word case in openapi test

* [phrase match | strict mode] Allow phrase condition when enabled in index (#6749)

* allow phrase filter when index is present

* prettier error message

* clippppppy

---------

Co-authored-by: timvisee <tim@visee.me>
2025-07-17 13:25:12 +02:00
Arnaud Gourlayandgenerall dd97253fe9 Strict mode allows fullscan for multitenant payload index (#6498)
* Strict mode allows fullscan for multitenant payload index

* different error for missing payload index in multitenant case

* handle payload_m

* add simple test

* handle hnsw.m not set

* add another simple test

* fix test

* fallback to global HSNW config

* new error status code

* do not block on global HNSW and improver error reporting

* clearer error reporting

* review fixes

* fix test error messages

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-05-22 23:26:16 +02:00
Arnaud GourlayandLuis Cossío bbc4beb9ec Fix UnindexedField infra to handle lookup&range index requirements (#6496)
* Fix UnindexedField infra to handle lookup&range index requirements

* more generic approach

* better test

* restore

* small unit test for sanity

* review nits

* Do not recommend parametrized index

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2025-05-22 23:24:10 +02:00
Andrey Vasnetsov 7872c182f1 Strict mode error is now 400 instead of 403 (#6560)
* strcit mode error is now 400 instead of 403

* fix test
2025-05-22 23:22:16 +02:00
Arnaud Gourlay 00e7ffa9b3 Strict mode detects full scan on query (#6473) 2025-05-22 22:59:16 +02:00
Arnaud Gourlay 1da478c897 Fix strict mode unindexed group_by path (#6363)
* Fix strict mode unindexed group_by path

* check index schema for matching support

* Handle disabled lookup on integer index
2025-05-22 22:56:24 +02:00
Luis Cossío a304a51f84 [rate limits] consider length of multivectors for query cost (#6356)
* consider length of multivectors for rate limiter

* add openapi test

* organize imports

* better cost estimation for sparse vectors

* restore groups openapi test

* address @timvisee's review

* Fix consensus test
2025-05-22 22:46:39 +02:00
Arnaud Gourlay b1b27c088a Fix strict mode query limit for distance matrix API (#6354) 2025-04-20 23:58:02 +02:00
Arnaud Gourlay e4400c6051 Fix strict mode query limit for group APIs (#6353) 2025-04-20 23:55:48 +02:00
Arnaud Gourlay 841048bbd2 Read rate limiter charge per filter condition (#6333) 2025-04-20 23:53:17 +02:00
Arnaud Gourlay db7dcda044 Rate limit retrieve based on number of ids (#6327)
* Rate limit retrieve based on number of ids

* fmt

* Better test
2025-04-20 23:52:54 +02:00
Luis Cossío 317f1e0dbe [strict mode | score boosting] Strict mode for formula queries (#6317)
* move problems.rs to collection crate

* change type on datetime_key expression

* extract from expression

* check for unindexed fields in formula

* clippy

* dedup code

* Only implement verification for `CollectionQuery` types (#6319)

* verify CollectionQuery's Prefetch

* remove implementation for REST query

* remove implementation for REST query groups

* add basic openapi test
2025-04-20 23:52:36 +02:00
Arnaud Gourlay 6518a053c9 Rate limite recommendation based on number of examples (#6324) 2025-04-20 23:52:11 +02:00
Arnaud Gourlay f5e35f9805 Write rate limit batch update per point count (#6152) 2025-03-21 11:43:26 +01:00
Arnaud Gourlay aa10df0fe4 Unset strict config fields (#6120) 2025-03-21 11:42:46 +01:00
Arnaud Gourlay cb1f3220df Add strict mode for max points count (#5967)
* Add strict mode for max points count

* clarify estimated nature
2025-02-17 10:20:50 +01:00
Arnaud Gourlayandtimvisee 27d5b39aee Retry-After on rate limiter hit for REST API (#5917)
* Retry-After on rate limiter hit for REST API

* fix tests

* Ceil retry after seconds number so clients don't retry too early

* Relax retry after time requirement a bit in case CI machines are slow

* Add rate limit test, test retry after if we replanish within a second

* Report time to wait in rate limit error

---------

Co-authored-by: timvisee <tim@visee.me>
2025-02-11 14:41:44 +01:00
00933d07ae Strict mode sparsevector (#5841)
* Strict mode sparse vectors
* Update gRPC docs
* Use deny_unknown_fields (#5868)

---------

Co-authored-by: n0x29a <n0x29a@example.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2025-01-27 18:25:34 +01:00
Arnaud Gourlay 0140eb83fc Strict mode multivector (#5757)
* strict mode for multivector

* simplify impl

* regen docs
2025-01-23 14:28:01 +01:00
Arnaud Gourlay bd882fde54 Migrate strict mode payload test for mmap (#5793) 2025-01-16 10:50:46 +01:00
Jojii b870f96513 [Strict mode] limits for filter and conditions (#5754)
* Add limits for filter and conditions

* clippy

* Review remarks + nested condition test

* Fix opnapi specs

* Improve error message by giving info about limits and usage
2025-01-16 10:50:36 +01:00
Jojii cb8667a7ae Merge pull request #5729
* Consider filter-cardinality as cost in update ratelimiter

* Use OperationEffectArea and improve rate limit error message

* Clippy

* Fix test
2025-01-16 10:49:14 +01:00
Arnaud Gourlay 3d3f8c104e Reshape read rate limiter and fix disabled mode (#5683) 2025-01-08 14:08:41 +01:00
Arnaud GourlayandTim Visée d3f2b8e79c Fix rate limiting of internal update operations (#5653)
* Fix rate limiting of internal update operations

* code review

* write_rate_limiter turned Option<Mutex> and fix disabling mode

* Update TODO tag

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
2025-01-08 14:08:29 +01:00
947e79e054 Strict mode for payload storage (#5588)
* Strict mode for payload storage

* Don't increment counter multiple times per request

* Add (loose) integration tests for payload storage limit

* Minor improvements

* minor renaming

* Update lib/api/src/grpc/proto/collections.proto

Co-authored-by: Tim Visée <tim+github@visee.me>

---------

Co-authored-by: jojii <jojii@gmx.net>
Co-authored-by: Jojii <15957865+JojiiOfficial@users.noreply.github.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
2025-01-08 14:07:24 +01:00
Arnaud Gourlay 993121b3ea Rate limit requests per minute (#5597)
* Rate limit requests per minute

* rename to remove time unit for API
2025-01-08 14:01:28 +01:00
Arnaud Gourlay 3c19eea9ca Rate limiting for shard operations (#5582)
* Rate limiting for shard operations

* address all review comments in one go
2024-12-09 11:18:50 +01:00
JojiiandArnaud Gourlay 0702854477 Strict mode max collection vector size (#5501)
* Strict mode config: Max collection size

* api specs

* Add tests + set/update payload check

* Improve function names and add comments

* rename config to separate vectors and payload

* fix tests

* Adjust configs docs

* add benchmark

* improve performance by caching shard info

* add bench for size_info() and fix tests

* Also limit the batch-size for vector updates (#5508)

* Also limit the batch-size for vector updates

* clippy

* add lost commit

* Load cache on collection initialization

* add unit type to parameter name

* fix renaming in test

* clearer error message

* fix test

* review remarks

* remove unused function for now

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2024-12-09 11:18:33 +01:00
Jojii 7b83223c8c Add max upsert batch size strict mode option (#5485) 2024-12-09 10:48:49 +01:00
Predrag Knezevic 96a5a2bce8 Test collections unique per openapi test module (#5384)
Collection name under test is equal to the test module name, without `.py` suffix.

* Helps by debugging/tracing failed tests and find relevant logs lines in qdrant log files
* Opens up a possibility to run tests in parallel, given that there are no data sharing
  between test modules

Change details:
* defined module scoped `collection_name` fixture in `conftest.py`
* removed `collection_name` module variable
* each test signature modified to declare the dependency to `collection_name` fixture
* `@pytest.mark.parametrize` migrated to `@pytest-cases.parametrize` in cases when
  `collection_name` was used as the value
2024-11-08 11:27:05 +01:00
a0de9509d7 Add strict mode config to update collection API + Populate strict mode in API (#5187)
* add strict mode to update collection API

* update API specs

* clippy

* Update lib/storage/src/content_manager/toc/collection_meta_ops.rs

* Strict mode integration tests (#5189)

* Add integration tests for strict mode

* add full update test

* Apply suggestions from code review

Co-authored-by: Tim Visée <tim+github@visee.me>

* Adjust error messages

* improve checking of error response

---------

Co-authored-by: Tim Visée <tim+github@visee.me>

* prefer explicit structu conversion

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-08 10:59:32 +01:00