Commit Graph

768 Commits

Author SHA1 Message Date
Kumar Shivendu
c0f580bddd Shard key initial state (#7371)
* Allow specifying initial state while creating shard key

* Dont wait for Active state if initial_state is set

* Generate gRPC docs and OpenAPI spec

* Apply suggestions

* Skip from scheamrs not serde

* Add integration test

* minor suggestions
2025-10-13 17:58:34 +05:30
Daniel Boros
3c6f26eced feat: handle 429 properly on infer svc (#7377)
* feat: handle 429 properly

* feat: change to rate limit error

* feat: handle http_date

* chore: remove unsued imports

* chore: make it more readable
2025-10-13 11:04:53 +02:00
Andrey Vasnetsov
96821e0376 edge retrieve api (#7344)
* refactor: move RecordInternal into Shard crate

* refactor: move retrieve_blocking into Shard crate

* implement retrieve method + move some structures into dedicated files
2025-10-06 19:59:43 +02:00
xzfc
23a0a0e8b8 DiffConfig: replace serde-based impl with explicit impl (#7294)
* refactor: replace serde-based DiffConfig implementation with macro

* refactor: macroexpand impl_diff_config

* refactor: drop `merge` dependency as unused
2025-09-29 18:55:50 +00:00
xzfc
780d406ff8 Add CollectionInfo::warnings field (#7293)
* refactor: swap DiffConfig parmeters

was: diff.update(&config)
now: config.update(diff)

* refactor: introduce DiffConfig::update_opt

* feat: add CollectionInfo::configuration_status field

* rename ConfigurationStatus to CollectionWarning
2025-09-29 18:17:03 +00:00
xzfc
a0d62330c7 Use fs-err (#7319) 2025-09-29 12:47:10 +00:00
dependabot[bot]
ae0e4e4305 build(deps): bump anyhow from 1.0.99 to 1.0.100 (#7287)
* build(deps): bump anyhow from 1.0.99 to 1.0.100

Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.99 to 1.0.100.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.99...1.0.100)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-version: 1.0.100
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Inline formatting argument

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: timvisee <tim@visee.me>
2025-09-23 10:30:52 +02:00
Arnaud Gourlay
8a31160297 Always run Telemetry on a blocking thread (#7269)
* Always run Telemetry on a blocking thread

* run Telemetry on general runtime instead of the blocking Actix one

* Polish test
2025-09-19 10:28:09 +02:00
Arnaud Gourlay
360d7bc26d Clippy 1.90 (#7253)
* Fix Clippy 1.90

* fmt
2025-09-15 20:12:11 +02:00
Arnaud Gourlay
bc13f01680 [strict-mode] Add max number of payload index count (#7222)
* [strict-mode] Add max number of payload index count

* improve docs and validation for StrictModeConfig

* fix from coderabbit

* test blocking access to payload indexes

* polish test
2025-09-15 13:12:40 +02:00
Kyamran
4c4a7cdbfd Add unified raft version (#7251) 2025-09-15 11:08:26 +02:00
Arnaud Gourlay
a45897568d Fix missing REST API validations misc. (#7220)
* Fix missing REST API validations misc.

* add validation for deleted_threshold

* backport vacuum_min_vector_number validation from gRPC
2025-09-08 12:48:14 +02:00
Arnaud Gourlay
83da3c4ee7 Bump http 0.2.12 (#7212)
* Bump http 0.2.12

* remove unused deps
2025-09-04 15:48:17 +02:00
xzfc
193306f2d7 Fixup remaining PeerId anonymization (#7207) 2025-09-04 09:54:39 +02:00
Roman Titov
6cb54ca380 Use Path instead of str in ConsensusWal::new (#7205) 2025-09-03 11:21:53 +02:00
dependabot[bot]
9b1bbac534 build(deps): bump url from 2.5.4 to 2.5.7 (#7144)
Bumps [url](https://github.com/servo/rust-url) from 2.5.4 to 2.5.7.
- [Release notes](https://github.com/servo/rust-url/releases)
- [Commits](https://github.com/servo/rust-url/commits)

---
updated-dependencies:
- dependency-name: url
  dependency-version: 2.5.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-26 13:20:57 +02:00
Andrey Vasnetsov
e744bd6014 implement custom collection metadata (#7123)
* implement custom collection metadata

* persist metadata change

* Also unset a key in the test

---------

Co-authored-by: timvisee <tim@visee.me>
2025-08-26 11:49:17 +02:00
Roman Titov
0bfa8f61d5 Move SaveOnDisk into common crate (#7105) 2025-08-20 17:06:47 +02:00
dependabot[bot]
599a6e42ad build(deps): bump anyhow from 1.0.98 to 1.0.99 (#7099) 2025-08-20 10:07:30 +00:00
dependabot[bot]
05e6488f23 build(deps): bump async-trait from 0.1.88 to 0.1.89 (#7085)
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.88 to 0.1.89.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.88...0.1.89)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-version: 0.1.89
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-19 09:44:39 +02:00
Andrey Vasnetsov
c188f5b7c4 Update if (#7006)
* implement condition parameter for upsert operation

* fmt

* update api schema

* implement conditional update for update-vectors op

* rename to update_filter for consistency

* add tests

* explicilty ignore strict mode for conditional updates

* rabbit review fix

* Also assert vector element before, confirm we don't normalize

* Simplify filter creation using new helper

* Rename merge_with_ids to with_point_ids

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
2025-08-15 15:47:43 +02:00
Arnaud Gourlay
255dd51e5a Fix Clippy 1.89 (#6981) 2025-08-05 18:06:04 +02:00
Roman Titov
e8625ba903 Avoid creating partial snapshot if snapshot manifest contains no changes (#6924) 2025-07-29 14:18:57 +02:00
Tim Visée
a5869beb99 Deprecations for Qdrant 1.15.0 (#6892)
* Deprecate init_from

* Mark memmap_threshold as deprecated

* Mark locks API as deprecated

* Mark RBAC collection access payload filter as deprecated

* Allow deprecations in tests and benchmarks
2025-07-17 16:10:45 +02:00
Kumar Shivendu
e14a003404 Fix bug that causes all replicas to die if node is restarted during resharding (#6800)
* Fix bug that causes all replicas to die if node is restarted during resharding

* fix recursive async problem

* Avoid write lock unless required

* avoid using &mut when & is sufficient

* Remove is_in_progress since check_abort_resharding exists (#6806)

* On resharding abort, only abort transfers related to current operation

* Fix resharding dead replicas improvements (#6881)

* Apply suggestions to fix resharding dead replicas

* fmt

---------

Co-authored-by: timvisee <tim@visee.me>
2025-07-16 17:51:30 +05:30
Tim Visée
5ba72b96b3 Change retain to extract_if where we depend on retain result (#6802) 2025-07-08 16:39:01 +02:00
Roman Titov
68a653649e Wait until replicas are activated, when creating custom shards (#6817)
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: timvisee <tim@visee.me>
2025-07-08 12:18:53 +02:00
Kumar Shivendu
8b5b244521 Use peer_id instead of default_peer_id (#6815) 2025-07-08 13:49:12 +05:30
Luis Cossío
137d71ada8 [MMR] resolve at collection level (#6786)
* add mmr as a ScoringQuery

* add mmr to CollectionQueryRequest

* handle mmr after collecting from shards

* use unwrap_or_else

* include mmr as a vector query

* ordering for MMR is None

* fix score threshold

* fmt + todo comment

* remove unused function

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-07-07 17:18:44 -04:00
Kumar Shivendu
d123b4fb84 Introduce new config to set default peer ID (#6783)
* Introduce new config to set default peer ID

* Add unit test to check persistence of default peer ID

* add missing update to main.rs

* fix clippy

* Log if using default peer ID

* Use non default peer id

* Move default_peer_id config to cluster config

* update main.rs

* minor var name refactor

* Add validation for default peer ID range

* Fix validation and ensure we dont generate Peer ID 0 randomly

* fmt
2025-07-04 17:35:37 +05:30
Roman Titov
3ff0a34821 Create custom shards in Initializing state instead of Active (#6778) 2025-07-02 17:53:42 +02:00
Andrey Vasnetsov
c11b96301b remove max indexing threads from config (#6760)
* remove max_optimization_threads from config.yaml

* fmt
2025-06-26 10:26:09 +02:00
xzfc
706571d8b4 Configurable HNSW healing threshold (#6756) 2025-06-25 10:22:18 +00:00
Tim Visée
4ce0c722f7 Feature flag RocksDB dependency (#6646)
* Flag index selector

* Flag take_database_snapshot

* Flag field index builder

* Flag bool index

* Flag full text index

* Flag geo index

* Flag map index

* Feature flag numeric index

* Flag skip_rocksdb

* Flag payload storage

* Flag segment

* Flag builder

* Flag backup

* Make RocksDB crate optional

* Tweak feature flag gate

* Disable default segment crate features from all dependents

* Handle flag in collection crate correctly

* Flag payload storage types in tests

* Temporarily disable on-disk check for sparse vectors

* Flag sparse vector RocksDB in GPU code

* Fix compilation error since recent merge

* Flag payload storage types in new function

* remove todo

* Explicitly implement default to more clearly state difference

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-06-11 14:29:30 +02:00
Andrey Vasnetsov
59c034f08e Fix default replication factor (#6632) 2025-06-04 10:04:37 +02:00
Andrey Vasnetsov
a95f997bda register payload index fields on recovering snapshot (#6582) 2025-05-22 17:36:28 +02:00
Roman Titov
4a4c194acd Add partial snapshot recovery and search locks (#6497)
Co-authored-by: generall <andrey@vasnetsov.com>
2025-05-22 14:49:19 +02:00
Andrey Vasnetsov
b92a322eb1 Strict mode error is now 400 instead of 403 (#6560)
* strcit mode error is now 400 instead of 403

* fix test
2025-05-20 15:19:36 +02:00
Andrey Vasnetsov
dc203e11b0 Add setting to limit max number of collections (#6548)
* add setting to limit max number of colelctions

* upd openapi

* Add max_collections to config

---------

Co-authored-by: timvisee <tim@visee.me>
2025-05-20 13:40:56 +02:00
Tim Visée
7c43f25544 Use more write buffers, and flush them (#6514)
* Explicitly flush buffered writers so that we catch errors

* Buffer IO in more places

* Tweaks

* Apply review suggestions
2025-05-12 17:37:51 +02:00
Tim Visée
833f310c3d Buffer all file IO when used as reader or writer (#6513) 2025-05-09 15:48:50 +02:00
Tim Visée
064fc24778 Don't batch queries if having more segments than search threads (#6478)
* Don't batch query if there's more segments than CPUs, limit threads

* Don't batch query if there's more segments than search threads

* Link to pull request

* Add note that we only consider local segments and global search threads
2025-05-07 12:57:48 +02:00
Roman Titov
438fcf86cb Add partial snapshot API endpoints (#6304) 2025-04-24 13:08:13 +02:00
Kumar Shivendu
d3d639ea9d Improve telemetry logic and test (#6399)
* Improve telemetry logic and test

* Parametrize telemetry test

* Consistency hash peeer ID across telemetry

* clean test

* Use Option in segments telemetry

* updat openapi spec

* Avoid test failure on change in order of params
2025-04-18 15:31:37 +02:00
Andrey Vasnetsov
c57d748dd4 Telemetry improvements (#6390)
* allow shard states in anonymize telemetry (with hashed peer ids)

* introduce level3 and level4 for telemetry, where level4 = everything, incl. segments info

* upd openapi

* fix tests

* expose vector count & size stats on shard level to avoid reading of segments

* fix spelling

* upd schema

* fix tests

* Use unwrap_or_default

* [#6390] skip serializing shard details in Level2 (#6398)

* skip serializing shard details in Level2

* upd openapi

---------

Co-authored-by: generall <andrey@vasnetsov.com>

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
2025-04-17 23:32:10 +02:00
Tim Visée
1370e01d58 Use ahash for maps/sets holding point IDs, offsets or similar (#6388) 2025-04-17 23:11:35 +02:00
dependabot[bot]
8cd8c84b88 build(deps): bump anyhow from 1.0.97 to 1.0.98 (#6374) 2025-04-14 20:06:16 +00:00
Luis Cossío
ca22191322 refactor QueryEnum::Nearest for using NamedQuery (#6349)
* refactor QueryEnum::Nearest for using NamedQuery

* use explicit `NamedQuery::from`
2025-04-10 09:53:31 -04:00
Luis Cossío
f9a537174c [strict mode | score boosting] Strict mode for formula queries (#6317)
* move problems.rs to collection crate

* change type on datetime_key expression

* extract from expression

* check for unindexed fields in formula

* clippy

* dedup code

* Only implement verification for `CollectionQuery` types (#6319)

* verify CollectionQuery's Prefetch

* remove implementation for REST query

* remove implementation for REST query groups

* add basic openapi test
2025-04-04 13:52:01 -03:00
Luis Cossío
5670a7c917 sum_scores recommendation strategy (#6256)
* Add in rest and grpc

* add to QueryEnum

* implement Query trait

* connect to scorer creation

* upd tests

* additional changes

* fmt

* gen openapi and grpc docs

* coderabbit fix

* add changes in async scorer

* test sum_scores in more places, refactor to remove repetition
2025-04-03 09:01:39 +02:00