upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-09-28 08:58:05 -05:00
Code Issues Packages Projects Releases Wiki Activity
1,077 Commits 437 Branches 119 Tags
consensus-bootstrap-fail
Commit Graph
89 Commits
Author SHA1 Message Date
Andrey Vasnetsov bd05a69a5f Id mapper inconsistency (#1302) 
* always flush wal

* always flush wal fix

* always flush wal fmt

* flush wal during background flush

* async wal flush

* use id-tracker internal id for next-id instead of vector storage

* add flush order and recovery comment

fix merge bug

* longer timeout in test
 2022-12-25 22:36:31 +01:00
Andrey Vasnetsov 926a058c73 Fixing missed leader inconsistency (#1286) 
* WIP: introduce local state

* WIP: sync local state with consensus in idle

* fmt

* rm extensive debug

* update triple replication test

* update triple replication test

* rm debug logs

* fix check for established leader + only sync local if no proposals

* remove unused file

* test compatible with python 3.8

* test compatible with python 3.8

* longer wait for consensus

* longer wait for consensus

* extra sleep in test

* remove extra sleep

* Fixing missed leader inconsistency - transfers (#1298)

* explicit request timeout

* explicit request timeout

* explicit request timeout

* explicit request timeout

* explicit request timeout

* prevent double handelling of the transfer termination

* kill the process

* log on inconsistency

* log on inconsistency

* log on inconsistency

* debug

* revert debug in test

* forward updates to partial shards, abort transactions on dead node report

* fmt

* disable retry of transfer, if the transfer was cancelled + allow predictable ports in test

* fix import
 2022-12-23 00:45:04 +01:00
Andrey Vasnetsov 38444d2cc7 Fix consensus freezes (#1241) 
* do not await for replica deactivation if there is a logical error

* use async mutex in raft service + integration test for deadlock

* fmt
 2022-11-24 15:36:03 +01:00
Andrey Vasnetsov 0e42672525 fix snapshot recovery - fix proper deactivation messages (#1238) 2022-11-23 13:47:14 +01:00
Arnaud Gourlay 6266d9bcdc enable compression for gRPC (#1186) 
* enable compression for gRPC

* add benchmarks

* fix grpc benchmark

* discardResponseBodies to reduce memory usage

* improve & split benchmarks

* do not use gzip for inter node communication as benchmarks are not conclusive
 2022-11-18 13:34:49 +01:00
Andrey VasnetsovandArnaud Gourlay 772326f848 Recover distributed collection (#1214) 
* WIP: shards recovering API: download snapshot

* fmt

* snapshot recovery API

* fmt

* snapshot recovery integration test + fixes

* review fixes

* review fixes 2

* Update lib/storage/src/content_manager/snapshots/recover.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-11-16 10:55:23 +01:00
Arnaud Gourlay fc77e3e511 Use systematically assert_http_ok for better error reporting (#1183) 
* Use systematically assert_http_ok for better error reporting

* extraction assertion to use it outside of pytest
 2022-11-01 11:19:13 +01:00
Arnaud GourlayEgor IvkovAndrey Vasnetsov
 		286ca9f5e4 0.11.0 cherry picks (#1174) 
* implement concern-factor, forbig disabling the last node, improve tes… (#1168)

* implement concern-factor, forbig disabling the last node, improve test stability

* upd docs

* improve test_recover_dead_node

* rename to write_consistency_factor

* Fix bug: applied index higher than commit on restart (#1172)

* create collection with default status = dead + await for activation on create_collecton_op_submit

* fmt

* fix unit tests

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>

* upd integration tests

* fix transaction handling

* upd protobuf

* upd to v0.11.0

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>
 2022-10-27 10:16:21 +02:00
Andrey Vasnetsov 21754ef2ef report indexed payload points in info api + other fixes (#1164) 
* report indexed payload points in info api + other fixes

* rollback debug changes

* clippy

* clippy
 2022-10-24 09:43:02 +02:00
Andrey Vasnetsov 387b52d7fa propagate peer url to consensus if specified uri is does not match a … (#1156) 
* propagate peer url to consensus if specified uri is does not match a stored one + tests

* review fixes
 2022-10-23 00:07:39 +02:00
Arnaud Gourlay faa4008938 Enable replication in consensus test (#1140) 2022-10-20 11:36:22 +02:00
Andrey Vasnetsov 123ae1607d Await for collection created for sure (#1144) 
* run plunger operation after user request to ensure that the majority of the consensus actually applied previous operation

* fmt

* fix test

* ensure all operations are unique in consensus queue

* add extra timeout for CI tests

* fix timout param in delete peer + do not explicilty await on peer removal

* fmt
 2022-10-19 17:59:16 +02:00
Andrey Vasnetsov 7f8c088711 Replication transfer fixes (#1142) 
* fix validation and task assignment during replica transfer

* fmt

* upd openapi

* extend integration test
 2022-10-19 11:38:26 +02:00
Andrey Vasnetsov 05ee1d76b8 replication integration test (#1138) 
* replication integration test

* review fix
 2022-10-18 17:01:34 +02:00
Arnaud GourlayandAndrey Vasnetsov 9324be1d60 [Replication] Add replicas (#1085) 
* Add replicas to ReplicaSet

* unproxify shard & miscs

* no exlusive write locking while adding replicas

* on_optimizer_config_update on Shard with async. recursion

* no exlusive write locking while removing replicas

* shortcut replica set propagation #1101

* remove unused field

* make RequestShardTransfer callback sync.

* promote local & remote to replica state

* fixes for replica sync api (#1123)

* code review

* fix replica set update - fail only if all failed

* Add replica redesign (#1131)

* refactor shard/mod.rs

* wip

* fmt

* it compiles

* temporary disable replica placemeant change on replication factor

* fmt

* finish todos

* small refactoring

* remove change::add

* replica-set -> shard-replica-set

* fmt

* upd openapi

* fix finish transfer logic

* fix existing integration tests

* shard transfer validation

* fmt

* review fixes

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2022-10-17 20:12:10 +02:00
Andrey Vasnetsov 35ae83f3fc validate geo coordinate before adding it to the payload index (#1125) 
* validate geo coordinate before adding it to the payload index

* revert changes in api test
 2022-10-13 09:14:11 +02:00
Andrey Vasnetsov 7a39ad4967 ensure consequent entries in consensus wal (#1108) 
* ensure consequent entries in consensus wal

* rollback consensus docker-compose

* rollback consensus docker-compose

* clippy

* add wal index debug assert
 2022-10-08 20:31:21 +02:00
Arnaud Gourlay 75515f7c73 propagate proper error in case of partial update error across shards (#1041) 
* propagate proper error in case of partial update error across shards

* add consensus test

* less duplication by leveraging existing error infra.

* no need for generic error handling

* fmt 1.66.0-nightly
 2022-09-22 19:37:59 +02:00
Andrey VasnetsovandArnaud Gourlay dc07b01e1f remove deprecated fields from API (#1030) 
* remove depricated fields from API

* fmt

* upd openapi and integration tests

* fix grpc test

* regenerate storage reference data

* improve docs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-09-19 13:51:03 +02:00
Ivan Pleshkov 4d3b4b132d More multivector api updates (#1025) 
* rest multivector api updates

* update openapi

* update grpc doc
 2022-09-16 12:25:42 +04:00
Ivan Pleshkov 22fd3a3963 minor doc and api in tests updates (#1022) 2022-09-15 13:37:38 +04:00
Andrey Vasnetsov be38254ee8 small refactoring (#746) 
* small refactoring

* fix tests
 2022-09-01 12:36:28 +02:00
Arnaud Gourlay 03c10fc310 Batch recommendation API (#954) 2022-08-22 09:41:22 +02:00
Andrey VasnetsovandArnaud Gourlay a2acca0345 Segment batch search (#813) 
* batch search benchmark

* collect filter iterator in indexed search

* fmt

* fix

* fix

* fmt

* use new tempfile create

* auto batching

* Clippy fixes

* REST, gRPC and internal APIs

* fix bugs & less duplication

* two steps payload retrieval mechanism & fix duplication

* add proxy_segment implementation & tests

* add gRPC docs

* remove unused code (#950)

* only filter ids within a batch

* add more equivalence tests

* add integration test search vs batch

* assert more search options in tests

* cleanup assertions

* fix offset panic

* rename search batch API

* openapi spec

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2022-08-18 14:48:17 +02:00
Andrey Vasnetsov 993e94eaef V0.9.1 (#939) 
* minor fix + upd version to 0.9.1

* fmt

* fix clippy

* fix test
 2022-08-10 21:15:40 +02:00
Andrey Vasnetsov 9f10a734fd a third attempt to implement peer removal + integration test (#917) 
* a third attempt to implement peer removal + integration test

* fix test

* review updates
 2022-08-05 16:17:33 +02:00
Andrey Vasnetsov 88c672475e Fix integration test (#903) 
* fix integration test problems + overflow

* fmt
 2022-08-03 16:45:34 +02:00
Andrey Vasnetsov e911faf8b3 wait for leader election before submitting the consensus operation (#897) 2022-08-03 10:59:45 +02:00
Andrey Vasnetsov 71a8bfbfdf Shard transfer from (#880) 
* explicitly specify the source of shard transfer

* fmt

* fix integration test
 2022-08-01 20:40:23 +02:00
Arnaud Gourlay 672cde0fee Storage compatibility for HNSW graph file (#875) 
* Storage compatibility for HNSW graph file

* create collection with a lower indexing_threshold_kb to avoid manual config. change
 2022-08-01 13:53:22 +02:00
Arnaud Gourlay 320b220b01 Move shard API (#869) 
* Move shard API

* add missing collection_name to openapi spec

* add basic shard transfer info to collection cluster API

* improve testing
 2022-07-28 20:14:15 +02:00
Arnaud Gourlay 7a8b04db6f [Sharding] Temporary shards for receiving transfer (#829) 
Temporary shards for receiving transfer
 2022-07-21 10:53:32 +02:00
Arnaud Gourlay 9b8b5b6376 [sharding] Collection cluster info API (#822) 
* Collection cluster info API

* better output format

* update openapi spec

* local shard info reports number of points

* rebase
 2022-07-18 17:57:34 +02:00
Egor Ivkov ef3c6c5e15 Fair hash ring (#816) 
* Fair hash ring

* Python test processes removal fix
 2022-07-14 19:27:53 +02:00
Arnaud Gourlay 28f4339ea1 Fix gRPC port for consensus tests (#800) 
* Fix gRPC port for consensus tests

* python fmt
 2022-07-12 10:11:40 +02:00
Egor Ivkov 75008d8d57 Kill qdrant processes after each python test (#782) 2022-07-05 15:01:17 +03:00
Egor Ivkov 8f8f55f993 Learners switch to voters test (#753) 2022-06-28 11:28:11 +02:00
Andrey Vasnetsov 83c21450c3 Pagination (#743) 
* offset param for search and recomendation

* test and fix for pagination

* clippy + fmt

* upd schemas

* fix grpc requests in test
 2022-06-27 13:22:02 +02:00
Arnaud Gourlay 6f8b145b5e test many collections through consensus (#731) 2022-06-24 10:19:00 +02:00
Arnaud Gourlay e8cf5937f1 refresh storage compat. data (#734) 2022-06-23 14:57:47 +02:00
Andrey Vasnetsov b66fa3b39d use indexes in basic api test (#732) 2022-06-23 14:04:24 +02:00
Egor Ivkov 17a6b25e29 Add new peers as learners (#710) 
* Add new peers as learners

* Fix Storage impl

* test added

* Test fixes
 2022-06-22 19:50:27 +03:00
Arnaud Gourlay fa5a501e25 Remove last sleep consensus test (#720) 
* remove sleep by polling peer addition

* cleanup
 2022-06-21 17:59:59 +02:00
Arnaud Gourlay 81589c23c4 harden consensus test and debugging (#704) 2022-06-15 19:13:54 +02:00
Egor Ivkov 533fd208b6 Move out consensus logic from ToC (#675) 2022-06-15 16:18:26 +03:00
Arnaud Gourlay beef4d0185 Consensus tests use cluster API instead of sleep (#674) 2022-06-09 20:49:41 +02:00
Egor Ivkov 1d36910b93 Test shards with restart and add volumes (#654) 2022-06-02 12:37:21 +03:00
Andrey Vasnetsov 2601c017de Smarter defaults (#637) 
* auto segments number

* auto segments number

* replace vector number limits with vector size limits

* fmt
 2022-06-01 18:09:38 +02:00
Egor Ivkov 4b83464061 Skip consensus init on restart (#638) 
* Skip consensus init on restart

* Added test

* CI
 2022-06-01 11:38:23 +03:00
Andrey Vasnetsov 3d505f5ae5 gRPC int payload (#564) 
* use custom message for json representation in gRPC

* fmt
 2022-05-30 19:10:37 +02:00