Commit Graph

648 Commits

Author SHA1 Message Date
Tim Visée
4ab6d2ee0f Bump version to 1.17.0 (#8160)
* Bump version to 1.17.0

* Update missed cherry picks

* Add OpenAPI spec for v1.17.x
2026-02-19 13:35:11 +01:00
Andrey Vasnetsov
9152fbd206 missing timeout for payload index ops (#8176) 2026-02-19 10:59:10 +01:00
Andrey Vasnetsov
da98a8eec4 Untagged Enum for FeedbackStrategy (#8171)
* [manual] make untagged enum for consistent API for FeedbackStrategy

* add #[validate(nested)]

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2026-02-18 17:21:06 +01:00
Andrey Vasnetsov
d16c62c76c Expose shard snapshot API (#8132)
* [AI] generate openapi schema for stream download of shard snapshot

* bump number of APIs

* cover API with access test

* fix test

* fix test
2026-02-17 11:00:47 +01:00
generall
5f9c3ca589 add missing timeout parameter for delete points API 2026-02-16 10:17:49 +01:00
Arnaud Gourlay
cc3f578b5c Remove update queue opnum from CollectionInfo (#8127) 2026-02-16 10:17:29 +01:00
Andrey Vasnetsov
a6b99fd583 distributed optimization info (#8120)
* [AI] initial implementation

* [AI] review fixes

* [manaul] some small fix

* [AI] fix issue about serializing optional fields

* upd schema

* [manual] review fixes

* [manual] only ask updatable replicas about optimizations
2026-02-16 10:17:28 +01:00
Andrey Vasnetsov
1e158632b6 Fix failed requests count reporting (#7944)
* add grpc status into grpc meterics similar to rest

* rm *_responses_fail_total metric

* upd openapi defs for grpc
2026-02-13 09:55:38 +01:00
Andrey Vasnetsov
27d0ff9c55 RFC: guideline for AI contributions (#8076) 2026-02-13 09:55:37 +01:00
Luis Cossío
bc6246df5a [relevance feedback] add rest and grpc interfaces (#7399)
* add rest and grpc interfaces

Also handle inference for this query

* follow refactor from base branch

* renaming from Ms Cooper

* get started on validation testing

* more validations

* test equivalence with query when less than 2 feedback elements

* rename feedback query to relevance_feedback query

* fix extraction of context pairs

* rename feedback vector to example

* make coderabbit happier

* upd test

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2026-02-13 09:55:37 +01:00
Andrey Vasnetsov
037600e8af Weighted rrf (#8063)
* weighted rrf implementation

* test

* fmt

* fix edge

* validate number of sources and number of weights

* do not partial match

* upd schema

* review fixes

* update formula

* remove calcualtions from tests

* update comment, because AI have OCD

* fmt
2026-02-10 00:04:33 +01:00
Ivan Pleshkov
78207079dd Update queue status in metrics and telemetry (#8060)
* update queue status in metrics and telemetry

* fix python tests

* update openapi
2026-02-10 00:01:27 +01:00
Arnaud Gourlay
aeeafe4a0e Update queue info into CollectionInfo (#8010)
* Update queue info into CollectionInfo

* basic test

* fix

* cleaner

* Update lib/collection/src/shards/local_shard/mod.rs

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>

* fix the fix

* rename to op_num

* skip_serializing_if just in case

* first step

* test queue length with staging feature

* gate integration test based on binary feature

---------

Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
2026-02-09 23:39:24 +01:00
Andrey Vasnetsov
9099a7ac1b introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert (#7963)
* introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert

* add test

* upd dockstring

* require resharding once all peers have updated version

* use service error

* fix clippy again

* wait for same version before resharding in tests
2026-02-09 23:28:45 +01:00
xzfc
8fb203eeb5 Update GET collections/{}/optimizations (#7946)
* refactor(SegmentOptimizer::name): make it `&'static`

* refactor(TrackerStatus): add is_running()

* refactor: add TrackerSegmentInfo

* feat: rework optimizations endpoint

* feat(collections/{}/optimizations): trim spaces
2026-02-09 23:28:03 +01:00
Andrey Vasnetsov
b2ad969b11 in ram single mmap file (#7971)
* WIP: introduce new vector store type

* handling of InRamMmap

* fmt

* feature-flag

* fmt

* Use if else

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>

* Update lib/common/common/src/flags.rs

Co-authored-by: Tim Visée <tim+github@visee.me>

* also choose madvise for single-file in-ram-mmap

* simplify generics

* gpu fix

* fix bug

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
2026-02-09 23:23:45 +01:00
xzfc
6abd5a1776 Enforce segment UUIDs (#7958)
* Swap docstrings for segment_ids/segment_uuids

Terse internal docs, detailed user-facing docs, not vice versa.

* Enforce segment UUIDs

* Export upcoming segment UUID to telemetry

* Add `optimize_for_test` wrapper

* Tune log message

Reason: the UUID is now guaranteed.

* Improve argument naming/docs
2026-02-09 23:20:44 +01:00
Kumar Shivendu
14dbd5e4cc Expose segment UUIDs in telemetry response (#7735)
* Add segment ID in telemetry response

* Migrate optimization log segment IDS to use UUIDs

* fmt

* clippy

* Bring back proxy segment SegmentEntry impl

* update openapi spec

* Bring back segment ID

* update comment

* Precompute segment uuid from filesyste while creating segment

* Update return types

* Update OpenAPI spec

* remove segment_ prefix from uuid field and use inline block

* Update OpenAPI spec

* Trigger CI

* clippy

* Improve comments to differentiate between segment id and uuid

* Remove duplicate function that shadowed segment entry function

---------

Co-authored-by: timvisee <tim@visee.me>
2026-02-09 23:16:07 +01:00
Luis Cossío
262ae08636 [cluster telemetry] Include shards information (#7886)
* make some existing fields optional

particularly, RemoteShardTelemetry's `searches` and `updates`, and
LocalShardTelemetry's `optimizations`

* Include shards information in cluster telemetry

* upd openapi

* nits

* only include partial snapshot info if non-empty

* fixes
2026-02-09 23:12:01 +01:00
Andrey Vasnetsov
fdceb11a18 implement a new configuration for read fan-out delay (#7929)
* implement a new configuration for read fan-out delay

* upd schema

* Update read_fan_out_delay_ms comment

---------

Co-authored-by: timvisee <tim@visee.me>
2026-02-09 23:10:13 +01:00
xzfc
d9d2801c71 Expose planned optimizations in API (#7784)
* Expose pending optimizations in API

* Address review comments
2026-02-09 23:07:34 +01:00
Andrey Vasnetsov
ee44157406 Prevent unoptimized updates (#7643)
* wip: wait for optimization before applying update

* implement api parameter

* nits

* fix deadloack when no optimizers are running

* release handle mutex

* Derive PartialEq

* Remove unused function

* fix missing kb conversion in threshold

* Update lib/api/src/grpc/proto/collections.proto

Co-authored-by: Tim Visée <tim+github@visee.me>

* sync comment change

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
2026-02-09 23:04:32 +01:00
Daniel Boros
b85176a11a feat/shard-ops-timeout (#7750)
* feat: add shard-ops timeout

* feat: add shard-ops timeout

* feat: add missing timeout

* fix: format

* fix: type serialization

* fix: bad wal flush logic

* feat: add seperate wait_time

* fix: linter issues

* fix: linter

* chore: make select more readable

* fix: linter issues

* chore: remove comment

* chore: adjust comment

* chore: update definition

* chore: remove extra wait_timeout -> make timeout global

* chore: remove wait_timeout from point_ops

* chore: remove wait_timeout from grpc

* refactor: make shard_ops more readable

* chore: remove timeout from update_all

* feat: propagate wait_timeout error

* fix: clippy recommendation

* chore: update wait timeout

* refactor: update merge_successful_update_results

* fix: tests inconsistency

* fix: revert global ord, partialord for UpdateStatus

* fix: integration test

* fix: res ingoring

* fix: review requests

* fix: clippy

* chore: revert back futures unordered

* fix: resolve response status

* force timeout for remote operations

* fix: pr reviews

* fix: switch to duration

* Bump OpenAPI spec

* feat: add serde_as

* fix: timeout statu condition

* Minor tweaks

* Use tokio timeout directly rather than deadline

* Revert schema generator permissions

* Remove obsolete semi-colon

---------

Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: timvisee <tim@visee.me>
2026-02-09 23:02:15 +01:00
Andrey Vasnetsov
983e10a2ef report resharding enabled flag (#7895)
* report if resharding is enabled in telemetry

* fmt
2026-02-09 22:55:35 +01:00
Sapphire
37de1f219d Add enable_hnsw option for payload field schema (#7887)
* feat: Add enable_hnsw option for payload field indexes

Add optional enable_hnsw parameter to all payload index types to control
whether additional HNSW graph links are built for each indexed field.

- Add enable_hnsw field to all 8 payload index param types
- Update gRPC proto definitions and conversions
- Update OpenAPI schema
- Modify HNSW graph builder to respect enable_hnsw flag
- Add enable_hnsw() helper methods to PayloadSchemaParams and PayloadFieldSchema
- Update all tests to include new field (default: None)

When enable_hnsw is true and payload_M > 0, additional HNSW links will
be built for the payload field. Default value is true for backward compatibility.

* Fix Some format problems

* fix: address comment problem

---------

Co-authored-by: EC2 Default User <ec2-user@ip-10-78-171-148.ec2.internal>
2026-02-09 22:53:47 +01:00
Luis Cossío
144cbd3f37 [cluster telemetry] Include peers versions (#7848)
* wrap details in another struct, less Options

* include version in peers info

* update openapi

* include app telemetry in internal service

* move debug assertion after check
2026-02-09 22:49:34 +01:00
tellet-q
30a34642b9 Add a test for recovery after kill during Partial (#7762)
* Add a test for recovery after kill during Partial

* Address AI review

* simplify test

* Merge pull request #7829

* introduce a new state

* switch to ManualRecovery for user-initiated snapshot operations

* fmt

* fix test and regen api

* test fix: instead of force recovery on update failed snapshots now re…

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2026-02-09 22:35:29 +01:00
Luis Cossío
45c560b5bc [cluster telemetry] Expose endpoint (#7729)
* expose cluster telemetry endpoint

* add endpoint to openapi

* add jwt test

* remove dead_code expectation

* fix consistency check

* fix jwt access test

* use response(reference(..)) in openapi
2026-02-09 22:31:32 +01:00
Luis Cossío
c9574abe74 [cluster telemetry] Aggregate telemetries from peers (#7696)
* aggregate telemetries

* upd openapi

* expect dead code

* Pick just the highest telemetry

* Tweak transfer selection

---------

Co-authored-by: timvisee <tim@visee.me>
2026-02-09 22:31:26 +01:00
Luis Cossío
80658e9855 [cluster telemetry] Internal service conversions (#7631)
* implement conversions

* clippy

* rename `ReshardStage` -> `ReshardingStage`

* impl conversions to grpc

* upd response in proto

* convert TelemetryData

* add peer info

* upd openapi

* add conversion from TelemetryData to PeerTelemetry

* review nit

* re-style

* make `method` optional

* change comments to reference `ReshardingStreamRecords`

* missing nits
2026-02-09 22:29:35 +01:00
Tim Visée
f937d0260f Add security caution banner to quick start command (#7815)
* Show simple security banner in quick start guide

* Swap sentence

* Refer to new secure your instance section

* Shorten text to fit on single line
2025-12-24 10:59:03 +01:00
tellet-q
f1d54ca889 Move test dependencies to tests (#7793)
* Move pyproject and uv.lock to tests folder

* Make tests working-dir agnostic

* Fix test

* Address review
2025-12-18 17:29:18 +01:00
Tim Visée
b234b55a4c Migrate Python to uv (#7790)
* Move pyproject.toml to root

* Migrate pyproject.toml from Poetry to uv

* Update GH workflows

* Update test script, doc and nix to use uv

* Use latest uv

* Fix uv.lock

* Cleanup shell.nix

* Cleanup

- Explicit `uv sync` is not required, `uv run` will install deps
  automatically.
- We don't provide a python package, so the `[build-system]` section
  is not needed.

* Fix UV_VERSION inconsistency

---------

Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
2025-12-18 17:28:50 +01:00
Daniel Boros
568f574429 feat/update-timeout (#7694)
* feat: add timeout to update params

* feat: update timeout params

* feat: add timeout for grpc

* fix: add missing timeout params

* fix: clippy warnings

* feat: update openapi definition

* chore: format

* fix: redoc master.json

* feat: add missing timeout to grpc api

* fix: missing timeout

* chore: update comments
2025-12-18 17:27:29 +01:00
xzfc
6c0d10abe7 Indexing progress (#7625)
* Add progress_tracker.rs

* Pass progress tracker around

* Populate progress tracker with actual data

* Expose progress on `/collections/{name}/optimizations` endpoint
2025-12-18 17:27:18 +01:00
AMIR
8eec2046d7 Add list shard keys API (#7615) 2025-12-03 10:20:40 +01:00
Arnaud Gourlay
9bfeed0292 Fix Openapi schema for DELETE issues (#7634) 2025-12-03 10:20:34 +01:00
Arnaud Gourlay
70b8623bb1 Add timeout to cancel metrics and telemetry calls (#7579)
* Add timeout to cancel metrics and telemetry calls

* relax constraints

* Configure timeout default in OpenAPI schema

---------

Co-authored-by: timvisee <tim@visee.me>
2025-11-25 12:56:45 +01:00
Tim Visée
4da01d375d Actively migrate away from RocksDB for vector/payload storage and index (#7551)
* Actively migrate away from RocksDB for vector/payload storage and index

* Update OpenAPI spec
2025-11-25 11:15:15 +01:00
Tim Visée
0d79528705 Bump version to 1.16.0 (#7535)
* Bump version to 1.16.0

* Update missed cherry picks

* Update redoc to 1.16.0
2025-11-17 12:10:37 +01:00
Andrey Vasnetsov
adcda00405 avoid docstring comments for enum fields to make openapi nicer (#7538)
* avoid docstring comments for enum fields to make openapi nicer

* add more spaces
2025-11-14 20:45:33 +01:00
Andrey Vasnetsov
ca902efde3 allow initial_state in API (#7528) 2025-11-14 12:31:07 +01:00
Kyamran
931db002a5 Fix docs/DEVELOPMENT.md (#7518) 2025-11-14 12:31:07 +01:00
Andrey Vasnetsov
17995c073f Mark legacy output vector fields deprecated (#7183)
* mark legacy output vector fields deprecated

* grpc docs

* Also mark legacy vector input fields as deprecated

---------

Co-authored-by: timvisee <tim@visee.me>
2025-11-14 12:31:07 +01:00
Jojii
0f191bfcf6 Metrics for snapshots (#7497)
* Add atomic counter for running snapshots

* Refactor collection telemetry + export prometheus metric

* Handle collection in ToC + current recovery measurements

* Fix openapi tests

* Fix tests

* Add snapshot metrics for streaming and partial snapshots.

* Fix typo in metrics name

* Adjust metrics names

---------

Co-authored-by: timvisee <tim@visee.me>
2025-11-14 12:31:06 +01:00
Tim Visée
6b39ff1353 Add ReadActive replica state, use in some stream records transfers (#7474)
* Add ReadActive replica state

* Add consensus function to switch from Partial to ReadActive and confirm

* On stream records to a different shard ID, switch to ReadActive first

* Update OpenAPI spec

* In stream records peer sync, require all peers to be up-to-date

* Review fixes

* Rename ReadActive to ActiveRead

* update openapi spec

* explicit state checking functions

* typo

* update fallback routing with usage of a new state

---------

Co-authored-by: KShivendu <kshivendu1@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
2025-11-14 12:31:04 +01:00
Ivan Pleshkov
f8b607c2ab Enable appendable quantization by default (#7508)
* enable appendable quantization by default

* update openapi
2025-11-14 12:31:04 +01:00
Andrey Vasnetsov
6257371413 remove init_from (#7454)
* wip: remove init_from

* remove init_from tests

* upd schema

* rm test
2025-11-14 12:30:16 +01:00
Andrey Vasnetsov
2a423d1f11 remove lock api (#7449)
* remove lock api

* fix api count + remove tests

* remove another test

* remove more tests
2025-11-14 12:30:16 +01:00
Tim Visée
2e239d5b58 Skip RocksDB for mutable payload index and payload storage (#7433)
* For mutable payload index, don't create with RocksDB anymore

* For payload storage, don't create with RocksDB anymore

* Add missing dot

* Update OpenAPI spec
2025-11-14 12:30:16 +01:00