Commit Graph

544 Commits

Author SHA1 Message Date
tellet-q
8d50f0a845 Add test for snapshot backpressure freeze (#8197)
* Add test for snapshot backpressure freeze
2026-02-24 15:04:59 +01:00
Arnaud Gourlay
a6c35be384 Fix Docker build for local compatibility tests (#8191) 2026-02-20 16:06:02 +01:00
Arnaud Gourlay
ab1917d634 Storage compatibility v1.17.0 (#8189) 2026-02-20 13:30:24 +01:00
Andrey Vasnetsov
6e79f8bf02 Untagged Enum for FeedbackStrategy (#8171)
* [manual] make untagged enum for consistent API for FeedbackStrategy

* add #[validate(nested)]

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2026-02-18 16:55:47 +01:00
Andrey Vasnetsov
345621f447 Expose shard snapshot API (#8132)
* [AI] generate openapi schema for stream download of shard snapshot

* bump number of APIs

* cover API with access test

* fix test

* fix test
2026-02-16 13:34:44 +01:00
Arnaud Gourlay
5ffc6a45a8 Remove update queue opnum from CollectionInfo (#8127) 2026-02-13 17:36:54 +01:00
Arnaud Gourlay
134ffde18f uv dependencies upgrade (#8124) 2026-02-13 12:05:30 +01:00
Roman Titov
05011e81cd Ensure WAL and shard clocks consistency when creating shard snapshot (#8104) 2026-02-13 11:27:02 +01:00
Arnaud Gourlay
db4aa33e6e Update Python dependencies: urllib3, protobuf, werkzeug, pyasn1 (#8119)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: agourlay <606963+agourlay@users.noreply.github.com>
2026-02-13 08:21:17 +01:00
Andrey Vasnetsov
af7608b101 Ignore cacert for internal requests if not configured (#8099)
* ignore cert verification if not configured

* also ignore hostnames

* script for manually testing tls clsuter
2026-02-12 10:59:38 +01:00
Tim Visée
c63f9eef29 Fix stream records transfer data race, losing pending updates (#8103)
* Add integration test to assert all queued updates are also transferred

* Plunge update queue in stream records transfers

* Migrate existing plunger usages to new plunge helper

* Skip test if not compiled with staging flag

* Only send delay operation when staging feature is enabled

* Reformat

* Fix review remarks
2026-02-11 17:18:44 +01:00
Luis Cossío
a43780941b [relevance feedback] add rest and grpc interfaces (#7399)
* add rest and grpc interfaces

Also handle inference for this query

* follow refactor from base branch

* renaming from Ms Cooper

* get started on validation testing

* more validations

* test equivalence with query when less than 2 feedback elements

* rename feedback query to relevance_feedback query

* fix extraction of context pairs

* rename feedback vector to example

* make coderabbit happier

* upd test

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2026-02-11 16:05:11 +01:00
tellet-q
3ba62e37b6 Add a test to check if creating snapshots blocks other operations (#7510)
* Add a test to check if creating snapshots blocks other operations
2026-02-11 12:20:47 +01:00
tellet-q
9b3939e008 Add test for consensus' freeze during streaming snapshot (#8075) 2026-02-11 10:58:36 +01:00
Luis Cossío
a87c22f56b fix: handle score_threshold in Formula queries (#8097)
* handle `score_threshold` in Formula queries

* AI: add openapi test

prompt: Upload 4 points with a numeric payload, use the payload as the
score, and set a score threshold. We can assert which ids should be in
the result and which shouldn't
2026-02-10 17:28:27 -03:00
Andrey Vasnetsov
4437edb775 Weighted rrf (#8063)
* weighted rrf implementation

* test

* fmt

* fix edge

* validate number of sources and number of weights

* do not partial match

* upd schema

* review fixes

* update formula

* remove calcualtions from tests

* update comment, because AI have OCD

* fmt
2026-02-06 19:36:27 +01:00
tellet-q
75ba1de122 Add more stages for shard transfer profiling (#8027)
* Improve stages reporting

* Remove transferring records info from snapshot transfer's comment

* Add detailed profiling for ReshardingStreamRecords

* Address AI review comment

* Remove some nesting

---------

Co-authored-by: timvisee <tim@visee.me>
2026-02-06 10:55:23 +01:00
Ivan Pleshkov
510b0c090a Update queue status in metrics and telemetry (#8060)
* update queue status in metrics and telemetry

* fix python tests

* update openapi
2026-02-05 11:37:19 +01:00
Arnaud Gourlay
9f09dbfd3b Update queue info into CollectionInfo (#8010)
* Update queue info into CollectionInfo

* basic test

* fix

* cleaner

* Update lib/collection/src/shards/local_shard/mod.rs

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>

* fix the fix

* rename to op_num

* skip_serializing_if just in case

* first step

* test queue length with staging feature

* gate integration test based on binary feature

---------

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
2026-01-30 17:23:53 +01:00
Tim Visée
36824b9aff Use local telemetry for peer version check, not cluster level telemetry (#8016) 2026-01-30 13:18:03 +01:00
Andrey Vasnetsov
15361e2fb8 more reliable version check (#8002) 2026-01-27 17:07:31 +01:00
tellet-q
00039e1537 Track shard transfer stages and duration (#8001)
* Track shard transfer stages and duration
2026-01-27 15:32:17 +01:00
Andrey Vasnetsov
81e7ab72fe introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert (#7963)
* introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert

* add test

* upd dockstring

* require resharding once all peers have updated version

* use service error

* fix clippy again

* wait for same version before resharding in tests
2026-01-27 00:39:25 +01:00
xzfc
d7a0f2f882 Update GET collections/{}/optimizations (#7946)
* refactor(SegmentOptimizer::name): make it `&'static`

* refactor(TrackerStatus): add is_running()

* refactor: add TrackerSegmentInfo

* feat: rework optimizations endpoint

* feat(collections/{}/optimizations): trim spaces
2026-01-26 18:03:52 +01:00
Daniel Boros
1c20cb5090 feat/test-timeout (#7920)
* feat: add timeout tests

* fix: timeout test

* fix: api urls

* fix: test

* feat: add staging api def

* feat: add parallel executor for tests

* fix: imports

* fix: ci/cd

* rollback openapi schema change

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2026-01-20 13:50:02 +01:00
Andrey Vasnetsov
28215e6125 Validate vector name in matrix request (#7923)
* validate `using` param

* fix slop
2026-01-16 12:20:40 +01:00
Kumar Shivendu
dd2f623ae7 Test behaviour when having no shards in custom sharding (#7867)
* Test behaviour when having no shards in custom sharding

* minor suggestion
2026-01-12 20:49:41 +05:30
Sapphire
a494452524 Add enable_hnsw option for payload field schema (#7887)
* feat: Add enable_hnsw option for payload field indexes

Add optional enable_hnsw parameter to all payload index types to control
whether additional HNSW graph links are built for each indexed field.

- Add enable_hnsw field to all 8 payload index param types
- Update gRPC proto definitions and conversions
- Update OpenAPI schema
- Modify HNSW graph builder to respect enable_hnsw flag
- Add enable_hnsw() helper methods to PayloadSchemaParams and PayloadFieldSchema
- Update all tests to include new field (default: None)

When enable_hnsw is true and payload_M > 0, additional HNSW links will
be built for the payload field. Default value is true for backward compatibility.

* Fix Some format problems

* fix: address comment problem

---------

Co-authored-by: EC2 Default User <ec2-user@ip-10-78-171-148.ec2.internal>
2026-01-09 11:39:44 +01:00
tellet-q
c0b70f1eae Add a test for recovery after kill during Partial (#7762)
* Add a test for recovery after kill during Partial

* Address AI review

* simplify test

* Merge pull request #7829

* introduce a new state

* switch to ManualRecovery for user-initiated snapshot operations

* fmt

* fix test and regen api

* test fix: instead of force recovery on update failed snapshots now re…

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2026-01-02 14:33:47 +01:00
Andrey Vasnetsov
dba26be155 Merge pull request #7835
* alt api key

* rm typo

* fix key length check logic

* differnt dirs for tests

* fix tests

* rm unused
2026-01-02 13:18:53 +01:00
Luis Cossío
b44596f477 allow collection-level jwt access (#7758) 2025-12-29 14:29:02 -06:00
Luis Cossío
85a5b61b98 [cluster telemetry] Expose endpoint (#7729)
* expose cluster telemetry endpoint

* add endpoint to openapi

* add jwt test

* remove dead_code expectation

* fix consistency check

* fix jwt access test

* use response(reference(..)) in openapi
2025-12-29 09:37:38 -06:00
tellet-q
2ae05890c5 Move test dependencies to tests (#7793)
* Move pyproject and uv.lock to tests folder

* Make tests working-dir agnostic

* Fix test

* Address review
2025-12-17 15:50:28 +01:00
Tim Visée
019632332d Add test for broken WAL delta after stream records abort (#7787)
* Add test to reproduce broken WAL delta after aborting stream records

* Add staging env var to slow down stream records transfers for test

* Tweak test formatting and utilities a bit

* Add comment to test, link to PR describing bug

* Update test so it still succeeds with patched behavior

* Fix broken WAL delta after stream records abort (#7791)

* Make set_replica_state async

* Add function called when active state of local replica changes

* Add snapshot for newest clocks

* Bump newest clocks snapshot on replica deactivation

* Use newest clocks snapshot during recovery

* Add enum for specifying whether to take or clear clocks snapshot

* Store clock snapshot inside clock map, removing extra file

This greatly simplifies state handling. It also prevent any kind of
desynchronization because all newest clocks are always persisted
atomically.

* Immediately persist clocks after taking snapshot

* Always update snapshot, only take if missing

* Take clock snapshots through each shard flavor, including proxies

* Propagate dedicated functions for taking and clearing clocks snapshot

* Only persist clocks immediately if changed on snapshot/clear

* Simplify recovery point logic, always take clocks snapshot if exists

* Remove unwrap

* Fix typo

* Fix doc comment

* Transfer driver is async, use Tokio sleep

* Reduce visibility
2025-12-17 12:33:00 +01:00
Tim Visée
b0bb7ef604 Migrate Python to uv (#7790)
* Move pyproject.toml to root

* Migrate pyproject.toml from Poetry to uv

* Update GH workflows

* Update test script, doc and nix to use uv

* Use latest uv

* Fix uv.lock

* Cleanup shell.nix

* Cleanup

- Explicit `uv sync` is not required, `uv run` will install deps
  automatically.
- We don't provide a python package, so the `[build-system]` section
  is not needed.

* Fix UV_VERSION inconsistency

---------

Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
2025-12-16 16:45:13 +01:00
Arnaud Gourlay
6b6981cef0 Bump misc. integration test deps (#7746) 2025-12-11 13:25:37 +01:00
tellet-q
82ca36ed37 Download archives once (#7711) 2025-12-10 10:04:19 +01:00
Andrey Vasnetsov
12815e81ff ignore shards with no active replicas (#7707) 2025-12-09 16:55:25 +01:00
xzfc
e617a4b777 Ignore deprecation warnings for schemathesis (#7730) 2025-12-09 00:16:37 +00:00
xzfc
77c71712db Indexing progress (#7625)
* Add progress_tracker.rs

* Pass progress tracker around

* Populate progress tracker with actual data

* Expose progress on `/collections/{name}/optimizations` endpoint
2025-12-09 00:15:36 +00:00
Roman Titov
7967c12b4a Fix test_collection_recovery test (#7700) 2025-12-05 13:20:37 +01:00
Roman Titov
4c9b843db4 Mark *new* local replicas as locally-disabled, when recovering Raft snapshot (#7684)
* Remove faulty mark-as-dead condition when recovering Raft snapshot 😅

* Mark new local replicas as locally-disabled when recovering Raft snapshot

* Add test
2025-12-03 18:12:07 +01:00
Tim Visée
a6a8fc50d9 build(deps): bump werkzeug from 3.1.3 to 3.1.4 in /tests (#7671)
Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.1.3 to 3.1.4.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/werkzeug/compare/3.1.3...3.1.4)

---
updated-dependencies:
- dependency-name: werkzeug
  dependency-version: 3.1.4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-12-02 16:33:24 +01:00
Jojii
8da4c22cae Make indexed-only-excluded-points metric default to 0 (#7649)
* Make indexed_only_excluded in metrics default to 0

* Fix tests
2025-12-02 09:36:41 +01:00
tellet-q
4fe1a1b068 Long running e2e test on many collections (#7576)
* Add test for many collections

* Reduce number of collections
2025-12-01 11:57:40 +01:00
Arnaud Gourlay
225caa8e5e Test storage compat. v1.6.1 (#7648) 2025-12-01 11:21:50 +01:00
AMIR
4f13b994e3 Add list shard keys API (#7615) 2025-11-28 21:50:09 -03:00
Arnaud Gourlay
adc3bb8540 Fix Openapi schema for DELETE issues (#7634) 2025-11-28 14:50:34 +01:00
Arnaud Gourlay
155465f154 Document storage compatibility test for v1.15.0 (#7601)
* Enable storage compatibility test for v1.15.0

* proper explanation
2025-11-26 12:11:19 +01:00
Arnaud Gourlay
0f2e415a72 Storage compat. tests all supported versions (#7566)
* Storage compat. tests all supported versions

* v1.15.0 creates problems

* less parallelism to not blow the disk

* Add xdist group (#7568)

* Add session fixture and xdist group

* Increase number of workers back to 4

* Revert caching

* Remove folders with docker_client

* Fix --dist=loadgroup

* Disable v1.15.0

---------

Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>
2025-11-24 12:45:26 +01:00