Commit Graph
68 Commits
Author SHA1 Message Date
Andrey Vasnetsov 30a432690e add random fail for test 2023-01-05 12:32:10 +01:00
Andrey Vasnetsov 926a058c73 Fixing missed leader inconsistency (#1286)
* WIP: introduce local state

* WIP: sync local state with consensus in idle

* fmt

* rm extensive debug

* update triple replication test

* update triple replication test

* rm debug logs

* fix check for established leader + only sync local if no proposals

* remove unused file

* test compatible with python 3.8

* test compatible with python 3.8

* longer wait for consensus

* longer wait for consensus

* extra sleep in test

* remove extra sleep

* Fixing missed leader inconsistency - transfers (#1298)

* explicit request timeout

* explicit request timeout

* explicit request timeout

* explicit request timeout

* explicit request timeout

* prevent double handelling of the transfer termination

* kill the process

* log on inconsistency

* log on inconsistency

* log on inconsistency

* debug

* revert debug in test

* forward updates to partial shards, abort transactions on dead node report

* fmt

* disable retry of transfer, if the transfer was cancelled + allow predictable ports in test

* fix import
2022-12-23 00:45:04 +01:00
Andrey Vasnetsov a2c5de586b Recommendation with other collection (#1283)
* WIP: move recommendation API out of collection

* recomendations with external source of vectors

* fix test

* fmt

* fixes + test

* review fixes
2022-12-22 09:29:36 +01:00
Arnaud Gourlay 584cdfa801 update all dependencies Cargo.lock (#1293) 2022-12-20 10:30:49 +01:00
Arnaud Gourlay e3c6bb8a53 update tonic 0.8 & prost 0.11 (#1285)
* update tonic 0.8 & prost 0.11

* install protoc via package manager
2022-12-19 15:30:33 +01:00
Andrey Vasnetsov d05b2def10 allow set different M param for default HNSW and payload-related links (#1257)
* allow set different M param for default HNSW and payload-related links

* review fixes
2022-12-06 13:53:42 +01:00
Andrey Vasnetsov 04434f02b6 Update points by filter (#1249)
* allow to set remove payload by selector

* fmt
2022-12-01 09:16:07 +01:00
Andrey Vasnetsov 9f36ac60ee Full payload update (#1245)
* implement api to fully overwrite payload of the point

* fmt
2022-11-28 13:46:58 +01:00
Ivan PleshkovandAndrey Vasnetsov ac9538d5af Hnsw links memmap (#1211)
* add on disk key

* remove obsolete graph initialization

* remove obsolete max level

* update openapi

* graph links trait

* use mmap option

* same format for ram and mmap

* fix segment unit tests

* are you happy fmt

* are you happy clippy

* fix ci and add mmap test

* review fixes

* remove unused try-from

* fix version compatibility

* avoid loading from disk during conversion

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2022-11-25 11:47:52 +01:00
Arnaud Gourlay 6266d9bcdc enable compression for gRPC (#1186)
* enable compression for gRPC

* add benchmarks

* fix grpc benchmark

* discardResponseBodies to reduce memory usage

* improve & split benchmarks

* do not use gzip for inter node communication as benchmarks are not conclusive
2022-11-18 13:34:49 +01:00
286ca9f5e4 0.11.0 cherry picks (#1174)
* implement concern-factor, forbig disabling the last node, improve tes… (#1168)

* implement concern-factor, forbig disabling the last node, improve test stability

* upd docs

* improve test_recover_dead_node

* rename to write_consistency_factor

* Fix bug: applied index higher than commit on restart (#1172)

* create collection with default status = dead + await for activation on create_collecton_op_submit

* fmt

* fix unit tests

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>

* upd integration tests

* fix transaction handling

* upd protobuf

* upd to v0.11.0

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>
2022-10-27 10:16:21 +02:00
Andrey Vasnetsov 21754ef2ef report indexed payload points in info api + other fixes (#1164)
* report indexed payload points in info api + other fixes

* rollback debug changes

* clippy

* clippy
2022-10-24 09:43:02 +02:00
Ivan PleshkovandAndrey Vasnetsov 44626bc80a exact search parameter (#1150)
* exact search parameter

* exact search test

* decrease attempts count because of test speed

* minor review fixes

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2022-10-20 15:39:21 +04:00
Egor Ivkov 9a3fad44b2 Revert "Revert replication changes (#1055)" (#1056)
This reverts commit a2ed0cf5db.
2022-09-23 16:46:06 +03:00
Egor IvkovandAndrey Vasnetsov a2ed0cf5db Revert replication changes (#1055)
* Revert "Replication set `Change` enum"

This reverts commit 28e8487542.

* Revert changes (#1053)

* Revert "Generate API - replication follow up (#1043)"

This reverts commit 183ea9e5e3.

* Revert "Replication related API changes (#1037)"

This reverts commit d690b700ff.

* Revert "Persist replica state (#1035)"

This reverts commit 31c3ae482d.

* Revert "[replication] Account for replicas in shard distribution (#1009)"

This reverts commit e332d482b0.

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2022-09-23 16:33:07 +03:00
Egor Ivkov 183ea9e5e3 Generate API - replication follow up (#1043)
Also removes `JsonSchema` from structs not exposed in API
2022-09-22 14:53:00 +03:00
Egor Ivkov d690b700ff Replication related API changes (#1037)
* Replication related API changes

* minor proto file fix
2022-09-21 18:49:47 +03:00
Andrey VasnetsovandArnaud Gourlay dc07b01e1f remove deprecated fields from API (#1030)
* remove depricated fields from API

* fmt

* upd openapi and integration tests

* fix grpc test

* regenerate storage reference data

* improve docs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2022-09-19 13:51:03 +02:00
Andrey Vasnetsov 03d08236a8 better struct for fts (#1026) 2022-09-16 14:07:50 +02:00
Ivan PleshkovandAndrey Vasnetsov f6b2186193 [WIP] Many named vectors per point (#958)
* many named vectors per point (segment-level)

* operation result for dim function

* beautifulized vector name

* fix naming bug

* segment version migration

* fmt

* add segment tests

* are you happy clippy

* fix build

* [WIP] many named vectors per point (collection-level) (#975)

* config and search

* fix placeholders for proxy segment move

* remove VectorType from collection

* are you happy fmt

* vectors in grps messages

* create collections with vectors

* segment holder fixes

* are you happy fmt

* remove default vector name placeholders

* are you happy fmt

* are you happy clippy

* fix build

* fix web api

* are you happy clippy

* are you happy fmt

* record vector&vectors

* openapi update

* fix openapi integration tests

* segment builder fix todo

* vector names for update from segment

* remove unwrap

* backward compatibility

* upd openapi

* backward compatible PointStruct

* upd openapi

* fix record back-comp

* fmt

* vector configuration backward compatibility

* fix vetor storage size estimation

* fmt

* multi-vec segment test + index test

* fmt

* api integration tests

* [WIP] Named vectors struct (#1002)

* move to separate file

* named vectors as struct

* use cow

* fix build

* keys iterator

* avoid copy in PointStruct -> get_vectors

* avoid another copy

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2022-09-11 20:59:23 +02:00
24ac939b4a Sync Points API (#985)
* shard sync operation

* fnt

* tests

* fix test

* Update lib/collection/src/collection_manager/segments_updater.rs

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>

* Update lib/collection/src/collection_manager/segments_updater.rs

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>

* match payload after vector only

* Update lib/collection/src/collection_manager/segments_updater.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2022-09-05 15:09:39 +02:00
Andrey Vasnetsov c3bc5b3873 rename grpc fields + re-generate docs (#980) 2022-09-01 18:59:47 +02:00
Andrey Vasnetsov b9eee55a9f Full text search (#963)
* allow additional params for payload field index

* fmt

* wip: full text index building

* fmt

* text search request

* text search request

* full text index persitance and loading

* fmt

* enable fts index in mapping

* clippy

* fix tests + add integration test

* review fixes: extend payload index test

* revert incedental change
2022-09-01 12:50:12 +02:00
Arnaud Gourlay 03c10fc310 Batch recommendation API (#954) 2022-08-22 09:41:22 +02:00
Andrey VasnetsovandArnaud Gourlay a2acca0345 Segment batch search (#813)
* batch search benchmark

* collect filter iterator in indexed search

* fmt

* fix

* fix

* fmt

* use new tempfile create

* auto batching

* Clippy fixes

* REST, gRPC and internal APIs

* fix bugs & less duplication

* two steps payload retrieval mechanism & fix duplication

* add proxy_segment implementation & tests

* add gRPC docs

* remove unused code (#950)

* only filter ids within a batch

* add more equivalence tests

* add integration test search vs batch

* assert more search options in tests

* cleanup assertions

* fix offset panic

* rename search batch API

* openapi spec

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2022-08-18 14:48:17 +02:00
Egor Ivkov a4e760af34 AddPeer in one consensus operation (#923)
* AddPeer in one consensus operation
2022-08-09 17:00:38 +03:00
Andrey Vasnetsov 00e965fc96 Configuration change on peer removal (#914)
* propose configuration change along with removal of the peer

* fmt
2022-08-04 14:47:49 +02:00
Andrey VasnetsovandArnaud Gourlay 60c7131cc6 Indexed vectors in collection info (#879)
* include number of indexed points into the info api

* fmt

* await for a certain level of indexing progressing before promoting shard

* log before waiting

* await for index fixes

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2022-08-03 16:19:19 +02:00
Andrey Vasnetsov 826165cfcf fix enum conversion in gRPC (#898) 2022-08-03 09:48:30 +02:00
Andrey Vasnetsov dfe2d591a5 Clever channel timeout (#851)
* active connection liveness check for channels

* clippy
2022-07-25 18:42:56 +02:00
Egor IvkovandAndrey Vasnetsov fd70b662d1 Recreate transport channels on network error (#845)
* Recreate transport channels on network error

* fmt

* Recreate channel upd (#847)

* recoverable channels

* recoverable channels

* retry on specific response error codes

* rename: ChannelsToUri -> ChannelPool

* rename: get_last_created -> get_created_at

* fmt

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2022-07-25 12:22:16 +02:00
Arnaud Gourlay 7a8b04db6f [Sharding] Temporary shards for receiving transfer (#829)
Temporary shards for receiving transfer
2022-07-21 10:53:32 +02:00
Andrey Vasnetsov 123332c867 Full snapshot (#824)
* full snapshot rest api

* openapi for snapshots

* openapi snapshot api

* fmt + clippy

* fix recovery + implement grpc methods

* fmt

* same aliases to full storage snapshot

* fmt
2022-07-18 17:11:23 +02:00
Egor Ivkov 026bd040b0 Add import formatting rules (#820)
* Add import formatting rules

* Review fix: update rusty hook
2022-07-15 15:42:25 +03:00
Andrey Vasnetsov 098bc4c751 Count api (#777)
* count api

* test for approx counting

* fmt + clippy

* unit test

* add warning
2022-07-05 09:33:38 +02:00
Arnaud Gourlay b23d3c7243 gRPC API for snapshot (#772) 2022-07-04 19:14:40 +02:00
Andrey Vasnetsov e983b07a15 Parallel hnsw building (#773)
* parallel hnsw building

* improve hnsw payload blocks condition

* update indexing optimizer condition

* fmt
2022-07-03 01:14:05 +02:00
Andrey Vasnetsov 92c05bb56f set max number of open files for rocksdb 2022-06-28 02:22:08 +02:00
Andrey Vasnetsov 850e937c2a Storage points tracking refactoring (#750)
* segment refactoring

* rm points iterator

* fmt
2022-06-27 15:17:09 +02:00
Andrey Vasnetsov 83c21450c3 Pagination (#743)
* offset param for search and recomendation

* test and fix for pagination

* clippy + fmt

* upd schemas

* fix grpc requests in test
2022-06-27 13:22:02 +02:00
Egor Ivkov e12b91e10c Promote learner (#735)
* Promote learner

Also fixes ERROR voters not found

* Review fixes

* Review: Find learner first
2022-06-24 11:27:58 +03:00
c15981092a [WIP] [real-time index] Implement payloadstorage for structpayloadindex (#642)
* [real-time index] Extend FieldIndex enum and StructPayloadIndex with method from PayloadStorage

* [real-time index] add missing remove_point methods

* [real-time index] add new index to FieldIndex enum

* fix compile

* are you happy fmt

* merge load and remove

* fix test generics

* decrement points count

* remove from histogram

* simplify histogram usage

* [real-time index] remove old tests and fix clippy warnings

* histogram: method to derive range by size (#657)

* [real-time index] add histogram based payload_blocks implementation.

* payload blocks

* fmt

* clippy

* [real-time index] refactor Segment to use PayloadIndex instead of PayloadStorage.

* fix tests

* fmt

* clippy

* rename indexes

* remove redundent params

* add struct payload deletion test + fix delete payload in map index

* remove payload threshold

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2022-06-06 17:14:20 +02:00
Andrey Vasnetsov f92535aa39 update qdrant.rs 2022-06-02 19:15:35 +02:00
Andrey Vasnetsov 2601c017de Smarter defaults (#637)
* auto segments number

* auto segments number

* replace vector number limits with vector size limits

* fmt
2022-06-01 18:09:38 +02:00
Ivan Pleshkov 708a5b211a fix handwrited comment in qdrant.rs (#653) 2022-06-01 17:45:14 +02:00
Andrey Vasnetsov 1b458780eb On disk payload storage (#634)
* implement on-disk payload storage

* fmt + clippy

* config param for on-disk payload storage

* upd openapi definitions

* add integration test with on-disk payload

* fix clippy

* review fixes

* fmt
2022-06-01 17:23:34 +02:00
Andrey Vasnetsov 3d505f5ae5 gRPC int payload (#564)
* use custom message for json representation in gRPC

* fmt
2022-05-30 19:10:37 +02:00
Arnaud Gourlay 0a33e00647 Pool gRPC transport channels (#584)
* Pool gRPC transport channels

* code review: do not use an extra layer of Arc

* code review: better naming
2022-05-13 14:44:06 +02:00
Andrey VasnetsovandArnaud Gourlay 5800319edb refactor distances + add score_threshold + fix negative euclid distance (#569)
* refactor distances + add score_threshold + fix negative euclid distance

* generate docs

* fix clippy

* Update lib/segment/src/spaces/tools.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2022-05-09 17:32:48 +02:00
Egor Ivkov f3fc921e51 Derive this peer uri if not supplied (#546) 2022-05-03 16:17:06 +03:00