Commit Graph

544 Commits

Author SHA1 Message Date
Roman Titov
a08c89082c Track payload index schema version (#6819) 2025-08-06 16:35:43 +02:00
Roman Titov
563194febf Fix test_create_shard_key_read_availability flakiness (#6902) 2025-08-06 16:14:13 +02:00
Kumar Shivendu
a4220fdd84 Partial snapshots update recovery ts (#6980)
* Update recovery ts despite empty partial snapshots

* fmt

* apply suggestions with some improvements
2025-08-06 10:48:03 +02:00
Arnaud Gourlay
33d15fd452 Bump Python integration test deps for Dependabot (#6979) 2025-08-05 18:05:36 +02:00
tellet-q
f87e505439 Refactor bash tests: low-ram (#6911)
* Refactor low-ran bash test

* Address review
2025-08-05 13:05:39 +02:00
Roman Titov
e8625ba903 Avoid creating partial snapshot if snapshot manifest contains no changes (#6924) 2025-07-29 14:18:57 +02:00
Roman Titov
22e15b9f40 Update collection-level payload index schema after recovering partial snapshot (#6886) 2025-07-29 12:25:24 +02:00
Andrey Vasnetsov
2c8c0a91fd wait-for instead of validate (#6899) 2025-07-24 08:37:23 +02:00
Tim Visée
6f4069e711 Preprocess query vectors, sort sparse vectors (#6895)
* Add test to assert behavior of unsorted sparse query

* Preprocess sparse vectors

* Fix typo
2025-07-17 16:11:44 +02:00
Tim Visée
cd1a31448d Fix broken is_empty filter on new points (#6882)
* Add test to catch bug

Bug: <https://github.com/qdrant/qdrant/issues/6880>

* Fix loading null index from wrong path

* Simplify test

* Also bump total point count when removing a point

* Add debug assertion, ensure we don't create null index in selector

* Update bug link

* Remove unused import

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

---------

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-07-16 14:25:09 +02:00
tellet-q
f84e1f1f4a Refactor bash tests: low-disk (#6845)
* Add e2e_tests pytest test suite

* Remove old bash test
2025-07-14 12:42:06 +02:00
Luis Cossío
6307510476 [MMR] expose as query variant in rest and grpc (#6797)
* expose mmr in rest and grpc

* fix referenced vectors in mmr queries

* rename to MmrInput in grpc for consistency

* mmr is now an optional parameter of nearest query

* fix score_threshold at local shard

* fix conversion of rest -> collection

* fix and refactor planned query

* nits

* rename `lambda` to `diversity` in interface

* handle score_threshold within mmr calculation

* add openapi test

* fix candidate limit at collection level

* candidate_limit -> candidates_limit

* finish rename to candidates_limit

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-07-11 10:39:12 -04:00
Andrey Vasnetsov
f47e499825 fix check for pure primary condition filter (#6840)
* fix check for pure primary condition filter

* avoid cloning has_id for primary condition

* fix test

* add test

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2025-07-10 11:40:11 +02:00
Roman Titov
68a653649e Wait until replicas are activated, when creating custom shards (#6817)
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: timvisee <tim@visee.me>
2025-07-08 12:18:53 +02:00
Roman Titov
3ff0a34821 Create custom shards in Initializing state instead of Active (#6778) 2025-07-02 17:53:42 +02:00
Arnaud Gourlay
5c54f437ab Deleted leftover test openapi collections (#6788) 2025-07-02 10:50:23 +02:00
Luis Cossío
6919e3ae5b [phrase matching] expose phrase condition (#6670)
* add `"match": { "phrase": ... }` condition

* gen grpc and openapi

* [phrase matching] expose `phrase_matching` flag in rest and grpc (#6620)

* expose setting in rest and grpc

* phrase matching openapi test

* regen openapi

* [phrase matching] Text index fixes (#6730)

* allow rocksdb-based immutable text index

* fix repeated-token phrases

* fmt

* Update OpenAPI spec

---------

Co-authored-by: timvisee <tim@visee.me>

* add repeated word case in openapi test

* [phrase match | strict mode] Allow phrase condition when enabled in index (#6749)

* allow phrase filter when index is present

* prettier error message

* clippppppy

---------

Co-authored-by: timvisee <tim@visee.me>
2025-06-24 16:56:36 -04:00
Tim Visée
f79ad06ae9 build(deps): bump urllib3 from 2.4.0 to 2.5.0 in /tests (#6721)
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.4.0 to 2.5.0.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.4.0...2.5.0)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-version: 2.5.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-19 10:07:19 +02:00
Roman Titov
a87a4e0d79 Track version of mutable files in vector and payload storage (#6652) 2025-06-17 19:00:11 +02:00
Roman Titov
d0f8a552bd Tweak partial snapshot recover and recover_from response (#6610) 2025-06-12 13:34:04 +02:00
Tim Visée
bc7636bb86 build(deps): bump requests from 2.32.3 to 2.32.4 in /tests (#6662)
Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.32.4.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](https://github.com/psf/requests/compare/v2.32.3...v2.32.4)

---
updated-dependencies:
- dependency-name: requests
  dependency-version: 2.32.4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-11 10:30:20 +02:00
Roman Titov
a04d887030 Simpler partial snapshots implementation based on immutable_files (#6599) 2025-06-09 10:55:26 +02:00
Roman Titov
b63f19b9b3 Add partial snapshot create requests count and recovery timestamp to telemetry (#6545)
Co-authored-by: generall <andrey@vasnetsov.com>
2025-05-22 15:56:52 +02:00
Roman Titov
4a4c194acd Add partial snapshot recovery and search locks (#6497)
Co-authored-by: generall <andrey@vasnetsov.com>
2025-05-22 14:49:19 +02:00
Arnaud Gourlay
2bba1017d9 Strict mode allows fullscan for multitenant payload index (#6498)
* Strict mode allows fullscan for multitenant payload index

* different error for missing payload index in multitenant case

* handle payload_m

* add simple test

* handle hnsw.m not set

* add another simple test

* fix test

* fallback to global HSNW config

* new error status code

* do not block on global HNSW and improver error reporting

* clearer error reporting

* review fixes

* fix test error messages

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-05-22 10:42:47 +02:00
Kumar Shivendu
f8fd25ce03 Fix force delete peer flaky test (#6568)
* Fix force delete peer flaky test

* Temporarily only run force delete peer test

* Run till failure

* Add mutable transfer field

* Introduce wait_with_return and use it to solve TOCTOU

* Revert "Run till failure"

This reverts commit 53319a19c8.

* Revert "Temporarily only run force delete peer test"

This reverts commit 5ea47e30a6.

* Simplify the test instead of introducing wait_with_return

* revert changes to utils
2025-05-21 20:55:09 +05:30
Arnaud Gourlay
0903e920bd Fix UnindexedField infra to handle lookup&range index requirements (#6496)
* Fix UnindexedField infra to handle lookup&range index requirements

* more generic approach

* better test

* restore

* small unit test for sanity

* review nits

* Do not recommend parametrized index

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2025-05-20 16:06:01 +02:00
Andrey Vasnetsov
b92a322eb1 Strict mode error is now 400 instead of 403 (#6560)
* strcit mode error is now 400 instead of 403

* fix test
2025-05-20 15:19:36 +02:00
Tim Visée
defa3af944 Wait when putting collection metadata (#6559) 2025-05-20 13:45:08 +02:00
Tim Visée
1d3ff46d57 Don't wait on shard transfer progress, just wait on transfer count (#6550) 2025-05-20 09:26:05 +02:00
Arnaud Gourlay
db31bd2e93 Always overwrite existing payload on upsert (#6533)
* Always overwrite existing payload on upsert

* Add REST tests
2025-05-15 13:15:36 +02:00
Kumar Shivendu
db514674ce Abort transfer on force peer delete and add test for bug (#6505)
* Test for reproducing force delete peer bug

* drop seed port

* use larger transfers to ensure killing before transfer is completed

* simplify test cases

* kill and wait for same commit before force delete

* Don't kill to avoid breaking consesnsus before force delete peer

* Abort transfer on force peer delete (#6507)

* Abort transfer on force peer delete

* ToC changes to abort internally

* Abort shard transfer using consensus not just locally

* fmt

* fix fmt

* Abort transfers internally when shard is being removed

* remove unused imports
2025-05-15 15:29:25 +05:30
n0x29a
73e7b17ee8 Refactor: move HardwareUsage to the Usage map (#6532)
---------

Co-authored-by: jojii <jojii@gmx.net>
2025-05-14 21:01:49 +02:00
Roman Titov
83cc409ccb Add recover_partial_snapshot_from endpoint (#6468) 2025-05-14 19:21:36 +02:00
Kumar Shivendu
e12de1472d Add a test for recovery mode (#6389)
* Add a test for recovery mode

* use fewer points for faster test

* fix failing test

* Sleep to ensure that qdrant does trigger a transfer
2025-05-09 14:15:13 +05:30
Arnaud Gourlay
fc64334e29 Validate IntegerIndexParams (#6494)
* Validate IntegerIndexParams

* focus
2025-05-07 16:35:57 +02:00
Arnaud Gourlay
1bdf1f34ae Strict mode detects full scan on query (#6473) 2025-05-05 15:59:13 +02:00
Arnaud Gourlay
b8698cf88b Fix strict mode unindexed group_by path (#6363)
* Fix strict mode unindexed group_by path

* check index schema for matching support

* Handle disabled lookup on integer index
2025-05-02 15:19:50 +02:00
Kumar Shivendu
a8d3132814 Fix point comparison in test (#6474)
* Fix point comparison in test

* keep using scroll

* drop unused imports
2025-04-30 19:44:56 +05:30
Kumar Shivendu
9c723aa480 Fix snapshot test flakiness (#6440)
* Temporarily run fewer tests in CI to reproduce flakiness

* disable other integration test

* log collection topology if request fails

* Run in CI until failure

* update tests

* Catch failures and log

* improve logging for search result comparison flakiness

* fix test

* Simplify test

* Use scroll instead of search and simplify existing code

* Trigger CI

* new util function

* debug again

* print collection topology on mismatch

* wait for deletion of flag after recovery is done

* use wait_for

* Revert integration-tests.yml
2025-04-29 17:10:44 +02:00
Arnaud Gourlay
1552409e8e Log context of openapi validation crash (#6448) 2025-04-28 13:55:12 +02:00
Kumar Shivendu
c8a960604a Introduce coverage reports for integration tests (#6414)
* Introduce coverage reports for integration tests

* Install cargo-llvm-cov

* Use multiline script

* Explicityl setup COVERAGE env var

* fix integration tests and log generated data

* fix ls path

* upload lcov file to GH artifacts

* integration profraw dynamic filename

* Fix llvm profile filename template

* Use interrupt instead of kill and merge consensus test results into same file

* Drop upload artifact stage

* install llvm-cov

* upload as artifact and export coverage files

* try simplifying workflow

* Migrate coverage generation to existing dedicated gh workflow

* trigger on coverage related branches

* Build only if qdrant binary with cov doesnt exist

* Use valid yaml

* include mode in profraw filename

* split coverage workflow into parallel jobs

* add poetry version to env

* log poetry version to install

* clean up integration test workflow

* Simplify comments
2025-04-25 16:42:41 +05:30
Arnaud Gourlay
d3e46acb35 Regen and update poetry lock (#6435) 2025-04-25 12:05:02 +02:00
Arnaud Gourlay
b4a52548a2 Fix warning regarding missing wait param in openapi tests (#6434) 2025-04-25 10:59:07 +02:00
dependabot[bot]
6df42bef79 build(deps): bump h11 from 0.14.0 to 0.16.0 in /tests (#6430)
Bumps [h11](https://github.com/python-hyper/h11) from 0.14.0 to 0.16.0.
- [Commits](https://github.com/python-hyper/h11/compare/v0.14.0...v0.16.0)

---
updated-dependencies:
- dependency-name: h11
  dependency-version: 0.16.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-04-25 10:30:55 +02:00
Arnaud Gourlay
d4f1f57cb6 Debug assert a missing column family or not loaded payload index (#6193)
* assertion on non built index

* Remove old custome storage for gridstore
2025-04-24 23:52:21 +02:00
Arnaud Gourlay
fe9d6bd1db Fix flaky test formula (#6429) 2025-04-24 20:57:20 +02:00
Roman Titov
438fcf86cb Add partial snapshot API endpoints (#6304) 2025-04-24 13:08:13 +02:00
Arnaud Gourlay
159b2c2865 Storage compatibility tests v1.14.0 (#6422) 2025-04-23 16:19:20 +02:00
Luis Cossío
56addec5be [rate limits] consider length of multivectors for query cost (#6356)
* consider length of multivectors for rate limiter

* add openapi test

* organize imports

* better cost estimation for sparse vectors

* restore groups openapi test

* address @timvisee's review

* Fix consensus test
2025-04-22 14:10:21 -04:00