Commit Graph

126 Commits

Author SHA1 Message Date
Jojii
8da4c22cae Make indexed-only-excluded-points metric default to 0 (#7649)
* Make indexed_only_excluded in metrics default to 0

* Fix tests
2025-12-02 09:36:41 +01:00
AMIR
4f13b994e3 Add list shard keys API (#7615) 2025-11-28 21:50:09 -03:00
Arnaud Gourlay
adc3bb8540 Fix Openapi schema for DELETE issues (#7634) 2025-11-28 14:50:34 +01:00
Arnaud Gourlay
b7da071199 Clarify multivector inline storage warning (#7523)
* Clarify multivector inline storage warning

* do what I actually wanted to do in first place

* fix test
2025-11-12 15:52:04 +01:00
Jojii
f6b715bd62 Metrics for snapshots (#7497)
* Add atomic counter for running snapshots

* Refactor collection telemetry + export prometheus metric

* Handle collection in ToC + current recovery measurements

* Fix openapi tests

* Fix tests

* Add snapshot metrics for streaming and partial snapshots.

* Fix typo in metrics name

* Adjust metrics names

---------

Co-authored-by: timvisee <tim@visee.me>
2025-11-11 15:00:11 +01:00
Andrey Vasnetsov
925d5127c8 remove init_from (#7454)
* wip: remove init_from

* remove init_from tests

* upd schema

* rm test
2025-10-29 17:37:02 +01:00
Andrey Vasnetsov
c1af720d0d remove lock api (#7449)
* remove lock api

* fix api count + remove tests

* remove another test

* remove more tests
2025-10-29 17:36:54 +01:00
Jojii
48d61f1719 Metrics vectors by name per collection (#7441)
* Add vector count per vector-name to metrics API

* Add to metrics API

* Improve TinyMap::get_or_insert_default and add tests

* Minor improvements

* Update openapi

* Review remarks

* Remove `collection_vectors` since it can be calculated manually
2025-10-29 17:00:53 +01:00
xzfc
95b4bbf978 Rename HnswConfig::{copy_vectors -> inline_storage} (#7389) 2025-10-13 10:08:06 +00:00
xzfc
780d406ff8 Add CollectionInfo::warnings field (#7293)
* refactor: swap DiffConfig parmeters

was: diff.update(&config)
now: config.update(diff)

* refactor: introduce DiffConfig::update_opt

* feat: add CollectionInfo::configuration_status field

* rename ConfigurationStatus to CollectionWarning
2025-09-29 18:17:03 +00:00
Andrey Vasnetsov
bff9cb1a50 fix strict mode validation for nested filters (#7264)
* fix strict mode validation for nested filters

* fix test
2025-09-16 12:03:54 +02:00
Arnaud Gourlay
e79100a2f2 Fix phrase match to not match with unknown tokens (#7252)
* Fix phrase match to not match with unknown tokens

* add tests

* spelling
2025-09-16 11:16:17 +02:00
Arnaud Gourlay
bc13f01680 [strict-mode] Add max number of payload index count (#7222)
* [strict-mode] Add max number of payload index count

* improve docs and validation for StrictModeConfig

* fix from coderabbit

* test blocking access to payload indexes

* polish test
2025-09-15 13:12:40 +02:00
Arnaud Gourlay
f30e7f7273 Fix missing REST API validation for point batch update (#7219)
* Fix missing REST API validation for point batch update

* use a valid vector
2025-09-05 18:25:37 +02:00
Andrey Vasnetsov
e744bd6014 implement custom collection metadata (#7123)
* implement custom collection metadata

* persist metadata change

* Also unset a key in the test

---------

Co-authored-by: timvisee <tim@visee.me>
2025-08-26 11:49:17 +02:00
Andrey Vasnetsov
bf60e96619 filter text any (#7100)
* internal implementation for filter any

* implement api for match `text_any`

* allow naive implementation of text match any

* dedup after kmerge + renames

* improve `expected_should_estimation`

* congruence test

* openapi test

* setup index in each test

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2025-08-24 12:09:05 +02:00
Andrey Vasnetsov
c188f5b7c4 Update if (#7006)
* implement condition parameter for upsert operation

* fmt

* update api schema

* implement conditional update for update-vectors op

* rename to update_filter for consistency

* add tests

* explicilty ignore strict mode for conditional updates

* rabbit review fix

* Also assert vector element before, confirm we don't normalize

* Simplify filter creation using new helper

* Rename merge_with_ids to with_point_ids

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
2025-08-15 15:47:43 +02:00
Jojii
290f9e2675 Always load InferenceService (#7050)
* Always load InferenceService

* Add BM25 openapi test

* Add warning to development.yaml config (#7051)
2025-08-13 16:59:42 +02:00
Tim Visée
6f4069e711 Preprocess query vectors, sort sparse vectors (#6895)
* Add test to assert behavior of unsorted sparse query

* Preprocess sparse vectors

* Fix typo
2025-07-17 16:11:44 +02:00
Tim Visée
cd1a31448d Fix broken is_empty filter on new points (#6882)
* Add test to catch bug

Bug: <https://github.com/qdrant/qdrant/issues/6880>

* Fix loading null index from wrong path

* Simplify test

* Also bump total point count when removing a point

* Add debug assertion, ensure we don't create null index in selector

* Update bug link

* Remove unused import

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-07-16 14:25:09 +02:00
Luis Cossío
6307510476 [MMR] expose as query variant in rest and grpc (#6797)
* expose mmr in rest and grpc

* fix referenced vectors in mmr queries

* rename to MmrInput in grpc for consistency

* mmr is now an optional parameter of nearest query

* fix score_threshold at local shard

* fix conversion of rest -> collection

* fix and refactor planned query

* nits

* rename `lambda` to `diversity` in interface

* handle score_threshold within mmr calculation

* add openapi test

* fix candidate limit at collection level

* candidate_limit -> candidates_limit

* finish rename to candidates_limit

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-07-11 10:39:12 -04:00
Andrey Vasnetsov
f47e499825 fix check for pure primary condition filter (#6840)
* fix check for pure primary condition filter

* avoid cloning has_id for primary condition

* fix test

* add test

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2025-07-10 11:40:11 +02:00
Arnaud Gourlay
5c54f437ab Deleted leftover test openapi collections (#6788) 2025-07-02 10:50:23 +02:00
Luis Cossío
6919e3ae5b [phrase matching] expose phrase condition (#6670)
* add `"match": { "phrase": ... }` condition

* gen grpc and openapi

* [phrase matching] expose `phrase_matching` flag in rest and grpc (#6620)

* expose setting in rest and grpc

* phrase matching openapi test

* regen openapi

* [phrase matching] Text index fixes (#6730)

* allow rocksdb-based immutable text index

* fix repeated-token phrases

* fmt

* Update OpenAPI spec

---------

Co-authored-by: timvisee <tim@visee.me>

* add repeated word case in openapi test

* [phrase match | strict mode] Allow phrase condition when enabled in index (#6749)

* allow phrase filter when index is present

* prettier error message

* clippppppy

---------

Co-authored-by: timvisee <tim@visee.me>
2025-06-24 16:56:36 -04:00
Roman Titov
b63f19b9b3 Add partial snapshot create requests count and recovery timestamp to telemetry (#6545)
Co-authored-by: generall <andrey@vasnetsov.com>
2025-05-22 15:56:52 +02:00
Arnaud Gourlay
2bba1017d9 Strict mode allows fullscan for multitenant payload index (#6498)
* Strict mode allows fullscan for multitenant payload index

* different error for missing payload index in multitenant case

* handle payload_m

* add simple test

* handle hnsw.m not set

* add another simple test

* fix test

* fallback to global HSNW config

* new error status code

* do not block on global HNSW and improver error reporting

* clearer error reporting

* review fixes

* fix test error messages

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-05-22 10:42:47 +02:00
Arnaud Gourlay
0903e920bd Fix UnindexedField infra to handle lookup&range index requirements (#6496)
* Fix UnindexedField infra to handle lookup&range index requirements

* more generic approach

* better test

* restore

* small unit test for sanity

* review nits

* Do not recommend parametrized index

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2025-05-20 16:06:01 +02:00
Andrey Vasnetsov
b92a322eb1 Strict mode error is now 400 instead of 403 (#6560)
* strcit mode error is now 400 instead of 403

* fix test
2025-05-20 15:19:36 +02:00
Arnaud Gourlay
db31bd2e93 Always overwrite existing payload on upsert (#6533)
* Always overwrite existing payload on upsert

* Add REST tests
2025-05-15 13:15:36 +02:00
Arnaud Gourlay
fc64334e29 Validate IntegerIndexParams (#6494)
* Validate IntegerIndexParams

* focus
2025-05-07 16:35:57 +02:00
Arnaud Gourlay
1bdf1f34ae Strict mode detects full scan on query (#6473) 2025-05-05 15:59:13 +02:00
Arnaud Gourlay
b8698cf88b Fix strict mode unindexed group_by path (#6363)
* Fix strict mode unindexed group_by path

* check index schema for matching support

* Handle disabled lookup on integer index
2025-05-02 15:19:50 +02:00
Arnaud Gourlay
1552409e8e Log context of openapi validation crash (#6448) 2025-04-28 13:55:12 +02:00
Arnaud Gourlay
b4a52548a2 Fix warning regarding missing wait param in openapi tests (#6434) 2025-04-25 10:59:07 +02:00
Arnaud Gourlay
fe9d6bd1db Fix flaky test formula (#6429) 2025-04-24 20:57:20 +02:00
Luis Cossío
56addec5be [rate limits] consider length of multivectors for query cost (#6356)
* consider length of multivectors for rate limiter

* add openapi test

* organize imports

* better cost estimation for sparse vectors

* restore groups openapi test

* address @timvisee's review

* Fix consensus test
2025-04-22 14:10:21 -04:00
Kumar Shivendu
d3d639ea9d Improve telemetry logic and test (#6399)
* Improve telemetry logic and test

* Parametrize telemetry test

* Consistency hash peeer ID across telemetry

* clean test

* Use Option in segments telemetry

* updat openapi spec

* Avoid test failure on change in order of params
2025-04-18 15:31:37 +02:00
Andrey Vasnetsov
c57d748dd4 Telemetry improvements (#6390)
* allow shard states in anonymize telemetry (with hashed peer ids)

* introduce level3 and level4 for telemetry, where level4 = everything, incl. segments info

* upd openapi

* fix tests

* expose vector count & size stats on shard level to avoid reading of segments

* fix spelling

* upd schema

* fix tests

* Use unwrap_or_default

* [#6390] skip serializing shard details in Level2 (#6398)

* skip serializing shard details in Level2

* upd openapi

---------

Co-authored-by: generall <andrey@vasnetsov.com>

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
2025-04-17 23:32:10 +02:00
Luis Cossío
634e5064ec [query api] don't propagate offset into prefetches (#6357)
* don't propagate offset into prefetches

* I want to see CI with that change

* Move import into platform specific scope

* edit test to make sure offset is not propagated

* fix planned query test

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: timvisee <tim@visee.me>
2025-04-11 14:11:21 +02:00
Arnaud Gourlay
8d54eba487 Fix strict mode query limit for distance matrix API (#6354) 2025-04-10 09:52:25 +02:00
Arnaud Gourlay
e617b9fa20 Fix strict mode query limit for group APIs (#6353) 2025-04-09 20:24:23 +02:00
Arnaud Gourlay
932ac134e8 Read rate limiter charge per filter condition (#6333) 2025-04-07 19:01:54 +02:00
Arnaud Gourlay
d46c4bf099 Rate limit retrieve based on number of ids (#6327)
* Rate limit retrieve based on number of ids

* fmt

* Better test
2025-04-07 11:31:42 +02:00
Luis Cossío
f9a537174c [strict mode | score boosting] Strict mode for formula queries (#6317)
* move problems.rs to collection crate

* change type on datetime_key expression

* extract from expression

* check for unindexed fields in formula

* clippy

* dedup code

* Only implement verification for `CollectionQuery` types (#6319)

* verify CollectionQuery's Prefetch

* remove implementation for REST query

* remove implementation for REST query groups

* add basic openapi test
2025-04-04 13:52:01 -03:00
Arnaud Gourlay
7723d91427 Document Multivector Euclidean scoring discrepancy (#6320) 2025-04-04 15:54:28 +02:00
Arnaud Gourlay
fd9215858e Rate limite recommendation based on number of examples (#6324) 2025-04-04 15:53:14 +02:00
Andrey Vasnetsov
dad1496d50 decrease collection probability (#6224) 2025-03-21 14:55:37 +01:00
Luis Cossío
4149e6d2b3 [score boosting] Error on unexpected type (#6187)
* helper for getting payload value

* Error instead of silent default

* fix clippy

* fix openapi test
2025-03-18 12:21:36 -03:00
Arnaud Gourlay
1a0862ebd3 Write rate limit batch update per point count (#6152) 2025-03-11 16:10:07 +01:00
Arnaud Gourlay
58039f9f2e Unset strict config fields (#6120) 2025-03-10 15:32:21 +01:00