upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-09-29 01:17:56 -05:00
Code Issues Packages Projects Releases Wiki Activity
1,077 Commits 451 Branches 119 Tags
consensus-bootstrap-fail
Commit Graph
234 Commits
Author SHA1 Message Date
Andrey Vasnetsov 880f8aadb1 add panic if we save things wrong 2023-01-05 16:00:09 +01:00
Andrey Vasnetsov 30a432690e add random fail for test 2023-01-05 12:32:10 +01:00
Andrey Vasnetsov bd05a69a5f Id mapper inconsistency (#1302) 
* always flush wal

* always flush wal fix

* always flush wal fmt

* flush wal during background flush

* async wal flush

* use id-tracker internal id for next-id instead of vector storage

* add flush order and recovery comment

fix merge bug

* longer timeout in test
 2022-12-25 22:36:31 +01:00
Andrey Vasnetsov 926a058c73 Fixing missed leader inconsistency (#1286) 
* WIP: introduce local state

* WIP: sync local state with consensus in idle

* fmt

* rm extensive debug

* update triple replication test

* update triple replication test

* rm debug logs

* fix check for established leader + only sync local if no proposals

* remove unused file

* test compatible with python 3.8

* test compatible with python 3.8

* longer wait for consensus

* longer wait for consensus

* extra sleep in test

* remove extra sleep

* Fixing missed leader inconsistency - transfers (#1298)

* explicit request timeout

* explicit request timeout

* explicit request timeout

* explicit request timeout

* explicit request timeout

* prevent double handelling of the transfer termination

* kill the process

* log on inconsistency

* log on inconsistency

* log on inconsistency

* debug

* revert debug in test

* forward updates to partial shards, abort transactions on dead node report

* fmt

* disable retry of transfer, if the transfer was cancelled + allow predictable ports in test

* fix import
 2022-12-23 00:45:04 +01:00
Andrey Vasnetsov a2c5de586b Recommendation with other collection (#1283) 
* WIP: move recommendation API out of collection

* recomendations with external source of vectors

* fix test

* fmt

* fixes + test

* review fixes
 2022-12-22 09:29:36 +01:00
dependabot[bot] 16a4d54885 Bump num_cpus from 1.14.0 to 1.15.0 (#1296) 
Bumps [num_cpus](https://github.com/seanmonstar/num_cpus) from 1.14.0 to 1.15.0.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases)
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.14.0...v1.15.0)

---
updated-dependencies:
- dependency-name: num_cpus
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-20 19:35:10 +01:00
dependabot[bot] e57f87a7ba Bump prost from 0.11.3 to 0.11.5 (#1295) 
Bumps [prost](https://github.com/tokio-rs/prost) from 0.11.3 to 0.11.5.
- [Release notes](https://github.com/tokio-rs/prost/releases)
- [Commits](https://github.com/tokio-rs/prost/compare/v0.11.3...v0.11.5)

---
updated-dependencies:
- dependency-name: prost
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-20 19:34:42 +01:00
Arnaud Gourlay e3c6bb8a53 update tonic 0.8 & prost 0.11 (#1285) 
* update tonic 0.8 & prost 0.11

* install protoc via package manager
 2022-12-19 15:30:33 +01:00
dependabot[bot] b76f671ac4 Bump async-trait from 0.1.59 to 0.1.60 (#1291) 
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.59 to 0.1.60.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.59...0.1.60)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-19 14:16:12 +01:00
dependabot[bot] 1c70a19d5a Bump anyhow from 1.0.66 to 1.0.68 (#1287) 
Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.66 to 1.0.68.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.66...1.0.68)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-19 14:15:07 +01:00
Arnaud Gourlay b6c37d72f2 Explicitly flush consensus WAL after (#1282) 
* panic on wal flush

* use latest wal

* try different flushing after truncate scheme

* pull latest wal branch rev

* use latest wal with fix
 2022-12-19 13:21:00 +01:00
Andrey Vasnetsov aeba32ba3e Shard build refactoring (#1280) 
* refactor some code of shard creation and optimizers

* Shard build refactoring debug (#1279)

* add debig code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* debug code

* refactor stop-checking during the optimization

* remove debug logs

* improve delete-renaming schema

* fmt

* move collection file removing to the async task

* rename check_optimization_stopped into more general check_process_stopped
 2022-12-14 10:11:46 +01:00
Andrey Vasnetsov 49c3e37061 add parameter that defines a source of truth for snapshot recovery (#1262) 
* add parameter that defines a source of truth for snapshot recovery

* review fixes
 2022-12-06 13:53:03 +01:00
dependabot[bot] c1caf271e5 Bump tokio from 1.22.0 to 1.23.0 (#1267) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.22.0 to 1.23.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.22.0...tokio-1.23.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-12-06 13:24:31 +01:00
Andrey Vasnetsov 4523ff5746 Peer id in snapshot name (#1260) 
* Add peer id into the snapshot name to distinguish snapshots from different ndoes

* fmt
 2022-12-06 12:31:58 +01:00
Andrey Vasnetsov 0925d02667 include backtrace for service errors (#1256) 2022-12-06 12:31:38 +01:00
Arnaud Gourlay 0cc6903a6d Update wal to revision 90471ad7 (#1253) 2022-12-01 23:05:16 +01:00
dependabot[bot] 4e4ccc83e6 Bump async-trait from 0.1.58 to 0.1.59 (#1250) 
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.58 to 0.1.59.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.58...0.1.59)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-11-29 20:30:01 +01:00
Arnaud Gourlay 8e82d1fb66 Bump wal to latest revision 94edd02 (#1246) 2022-11-29 12:03:51 +01:00
Arnaud Gourlay 52254755a0 Improve thread naming (#1240) 
* improve local shard thread naming

* setup proper runtime for toc
 2022-11-24 13:38:45 +01:00
Andrey Vasnetsov 0e42672525 fix snapshot recovery - fix proper deactivation messages (#1238) 2022-11-23 13:47:14 +01:00
dependabot[bot] 18a67686df Bump tokio from 1.21.2 to 1.22.0 (#1234) 
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.21.2 to 1.22.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.21.2...tokio-1.22.0)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-11-21 20:28:57 +01:00
Andrey Vasnetsov 8cc7f67f1b spawn tokio task to prevent blocking execution thread (#1226) 2022-11-18 13:35:57 +01:00
Arnaud Gourlay 6266d9bcdc enable compression for gRPC (#1186) 
* enable compression for gRPC

* add benchmarks

* fix grpc benchmark

* discardResponseBodies to reduce memory usage

* improve & split benchmarks

* do not use gzip for inter node communication as benchmarks are not conclusive
 2022-11-18 13:34:49 +01:00
Andrey VasnetsovandArnaud Gourlay 772326f848 Recover distributed collection (#1214) 
* WIP: shards recovering API: download snapshot

* fmt

* snapshot recovery API

* fmt

* snapshot recovery integration test + fixes

* review fixes

* review fixes 2

* Update lib/storage/src/content_manager/snapshots/recover.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-11-16 10:55:23 +01:00
Andrey Vasnetsov dad331fc26 fix colletion params update (#1208) 2022-11-09 13:10:54 +01:00
dependabot[bot] 8504cf502a Bump num_cpus from 1.13.1 to 1.14.0 (#1203) 2022-11-09 08:31:47 +00:00
Arnaud Gourlayallcontributors[bot]Andrey Vasnetsov
 		98fc5b2fa4 0.11.1 cherry picks (#1193) 
* Complete feature road-map

* docs: add erare-humanum as a contributor for code (#1181)

* docs: update README.md [skip ci]

* docs: update .all-contributorsrc [skip ci]

Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>

* update rust version and fix clippy complains (#1190)

* handle collections transition from single-node to cluster (#1189)

* Update to v0.11.1 (#1191)

* update to v0.11.1

* also default version

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

* consensus_state to consensus_state

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
 2022-11-07 11:31:12 +01:00
Andrey Vasnetsov 383100ebb6 remove suggesting changes in replications on replication factor change (#1177) 2022-10-31 09:57:22 +01:00
Andrey Vasnetsov a0b6fa81b7 Consensus q n a (#1169) 
* Expand comments and minor refactoring for raft state

* fmt

* add comments to consensus.rs
 2022-10-30 20:34:09 +01:00
Arnaud GourlayEgor IvkovAndrey Vasnetsov
 		286ca9f5e4 0.11.0 cherry picks (#1174) 
* implement concern-factor, forbig disabling the last node, improve tes… (#1168)

* implement concern-factor, forbig disabling the last node, improve test stability

* upd docs

* improve test_recover_dead_node

* rename to write_consistency_factor

* Fix bug: applied index higher than commit on restart (#1172)

* create collection with default status = dead + await for activation on create_collecton_op_submit

* fmt

* fix unit tests

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>

* upd integration tests

* fix transaction handling

* upd protobuf

* upd to v0.11.0

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>
 2022-10-27 10:16:21 +02:00
Andrey Vasnetsov 071d1948b0 update and improve openapi definitions 2022-10-24 16:04:31 +02:00
Ivan PleshkovandAndrey Vasnetsov 516dcd7020 Telemetry level of detail (#1049) 
* telemetry level of detail

* rename duration aggregator

* are you happy fmt

* move total searches sum

* separate levels

* optional bucket size

* search telemetry improvements

* separate web telemetry into methods

* tonic telemetry methods

* merge optimizations

* are you happy fmt

* better rounding

* qdrant configs on level 1

* provide collection params

* add peers count

* collection points count

* update openapi

* use pattern in actix telemetry

* are you happy fmt

* merge dev

* are you happy fmt

* fix merge conflicts

* update openapi

* fix build

* are you happy fmt

* add exact searches statistics

* process replica set

* update openapi

* fix wrong name

* fix naming

* fix unwrap

* review

* fmt

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-10-23 00:48:55 +02:00
Egor Ivkov a96f92cbfe Record message send failures and stop consensus on service error (#1152) 
* Record message send failures and stop consensus on service error

* Generate openapi

* Review fixes: log first error and remove on success
 2022-10-22 23:23:37 +02:00
Arnaud Gourlay faa4008938 Enable replication in consensus test (#1140) 2022-10-20 11:36:22 +02:00
Andrey Vasnetsov 75563b20b4 Consensus fault tollerance improvements (#1149) 
* consensus fault tollerance improvements

* fmt
 2022-10-20 11:22:22 +02:00
dependabot[bot] 7d9ca67ea9 Bump async-trait from 0.1.57 to 0.1.58 (#1146) 
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.57 to 0.1.58.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.57...0.1.58)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-10-20 08:55:54 +02:00
Andrey Vasnetsov 123ae1607d Await for collection created for sure (#1144) 
* run plunger operation after user request to ensure that the majority of the consensus actually applied previous operation

* fmt

* fix test

* ensure all operations are unique in consensus queue

* add extra timeout for CI tests

* fix timout param in delete peer + do not explicilty await on peer removal

* fmt
 2022-10-19 17:59:16 +02:00
Egor Ivkov d3849d712b Allow peer force removal (#1139) 
* Allow peer force removal

* Remove shards
 2022-10-19 17:01:13 +02:00
Andrey Vasnetsov 2b4ab27506 Write lock for user requests (#1143) 
* lock update operations only

* fmt

* fix

* lock openapi integration test
 2022-10-19 12:57:52 +02:00
Andrey Vasnetsov 7f8c088711 Replication transfer fixes (#1142) 
* fix validation and task assignment during replica transfer

* fmt

* upd openapi

* extend integration test
 2022-10-19 11:38:26 +02:00
Ivan PleshkovArnaud GourlayAndrey Vasnetsov
 		f50f1f4c71 #1119 disable updating feature (#1126) 
* disable updating feature

* customize error message

* update namings

* Update src/actix/api/service_api.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* fix build

* update api

* update api and allow deletion

* update openapi

* replace Bad Request with Forbidden

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-10-18 12:58:38 +02:00
Andrey Vasnetsov ef7b4ac2a7 propagate payload index to temporary segment during optimization (#1133) 2022-10-18 10:49:37 +02:00
Arnaud GourlayandAndrey Vasnetsov 9324be1d60 [Replication] Add replicas (#1085) 
* Add replicas to ReplicaSet

* unproxify shard & miscs

* no exlusive write locking while adding replicas

* on_optimizer_config_update on Shard with async. recursion

* no exlusive write locking while removing replicas

* shortcut replica set propagation #1101

* remove unused field

* make RequestShardTransfer callback sync.

* promote local & remote to replica state

* fixes for replica sync api (#1123)

* code review

* fix replica set update - fail only if all failed

* Add replica redesign (#1131)

* refactor shard/mod.rs

* wip

* fmt

* it compiles

* temporary disable replica placemeant change on replication factor

* fmt

* finish todos

* small refactoring

* remove change::add

* replica-set -> shard-replica-set

* fmt

* upd openapi

* fix finish transfer logic

* fix existing integration tests

* shard transfer validation

* fmt

* review fixes

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-10-17 20:12:10 +02:00
Andrey Vasnetsov b275d452a8 shard replica manipulation API (#1121) 2022-10-12 15:19:43 +02:00
Andrey Vasnetsov 62cc0880b8 Replica set snapshot (#1118) 
* replication set loading + snapshoting

* test + fixes

* fix typo

* fmt

* remove duplicated
 2022-10-12 12:19:08 +02:00
Andrey Vasnetsov 7a39ad4967 ensure consequent entries in consensus wal (#1108) 
* ensure consequent entries in consensus wal

* rollback consensus docker-compose

* rollback consensus docker-compose

* clippy

* add wal index debug assert
 2022-10-08 20:31:21 +02:00
Egor Ivkov ca01b8ac8e Fix: Do not rely on empty message for is_leader_established flag (#1103) 
* Fix: Do not rely on empty message for `is_leader_established` flag

* Record last consensus update

* Update OpenAPI
 2022-10-05 18:10:33 +03:00
Egor Ivkov 4a3dd91530 Fix minor todos in for ReplicaSet (#1091) 2022-10-04 18:34:28 +03:00
dependabot[bot] 814b0ebcb4 Bump schemars from 0.8.10 to 0.8.11 (#1089) 
Bumps [schemars](https://github.com/GREsau/schemars) from 0.8.10 to 0.8.11.
- [Release notes](https://github.com/GREsau/schemars/releases)
- [Changelog](https://github.com/GREsau/schemars/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GREsau/schemars/compare/v0.8.10...v0.8.11)

---
updated-dependencies:
- dependency-name: schemars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2022-10-04 11:17:22 +02:00