Commit Graph

617 Commits

Author SHA1 Message Date
Andrey Vasnetsov
a2acca0345 Segment batch search (#813)
* batch search benchmark

* collect filter iterator in indexed search

* fmt

* fix

* fix

* fmt

* use new tempfile create

* auto batching

* Clippy fixes

* REST, gRPC and internal APIs

* fix bugs & less duplication

* two steps payload retrieval mechanism & fix duplication

* add proxy_segment implementation & tests

* add gRPC docs

* remove unused code (#950)

* only filter ids within a batch

* add more equivalence tests

* add integration test search vs batch

* assert more search options in tests

* cleanup assertions

* fix offset panic

* rename search batch API

* openapi spec

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2022-08-18 14:48:17 +02:00
Andrey Vasnetsov
993e94eaef V0.9.1 (#939)
* minor fix + upd version to 0.9.1

* fmt

* fix clippy

* fix test
2022-08-10 21:15:40 +02:00
Arnaud Gourlay
935e9a21aa update v0.9.0 (#919)
* update v0.9.0

* update gRPC docs

* update abort transfer
2022-08-08 11:13:54 +02:00
Andrey Vasnetsov
abbd3bcbf1 Fix transfer abort on restart (#921)
* await for consensus leader before submitting transfer abort + api

* fmt
2022-08-06 14:53:35 +02:00
Andrey Vasnetsov
60c7131cc6 Indexed vectors in collection info (#879)
* include number of indexed points into the info api

* fmt

* await for a certain level of indexing progressing before promoting shard

* log before waiting

* await for index fixes

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2022-08-03 16:19:19 +02:00
Egor Ivkov
0a634453e3 Peer removal API (#894)
* Remove peer api

* Keep the same cluster status api for compatibility
2022-08-02 15:56:41 +03:00
Ivan Pleshkov
76d9584dda openapi for optimizators telemetry (#893) 2022-08-02 15:04:19 +04:00
Andrey Vasnetsov
71a8bfbfdf Shard transfer from (#880)
* explicitly specify the source of shard transfer

* fmt

* fix integration test
2022-08-01 20:40:23 +02:00
Andrey Vasnetsov
c53f19c0dd Disable mmap threshold (#877)
* allow to disable mmap by not specifying mmap threshold

* fmt

* smart default max_segment_size

* upd openapi for max_segment_size
2022-08-01 15:18:40 +02:00
Arnaud Gourlay
dc4cccde3d Prevent Vector size zero (#876) 2022-07-31 13:03:53 +02:00
Ivan Pleshkov
ddbd190aaa Grpc telemetry (#873)
* grpc telemetry

* fix avg calc

* separate telemetry collectors

* update openapi

* use parking lot for actix

* are you happy fmt

* fix miskate in naming
2022-07-30 12:33:15 +04:00
Egor Ivkov
95019c74ca Stop consensus on error (#872) 2022-07-29 18:17:41 +03:00
Arnaud Gourlay
320b220b01 Move shard API (#869)
* Move shard API

* add missing collection_name to openapi spec

* add basic shard transfer info to collection cluster API

* improve testing
2022-07-28 20:14:15 +02:00
Andrey Vasnetsov
bae041ecb4 upd version to 0.8.6 2022-07-26 19:28:53 +02:00
Ivan Pleshkov
78d1fa85e3 cluster status telemetry (#863) 2022-07-26 18:52:46 +04:00
Andrey Vasnetsov
314f139183 upd v0.8.5 (#852) 2022-07-25 20:05:07 +02:00
Ivan Pleshkov
42e930ab8f Segment telemetry (#814)
* segment telemetry

* anonymize trait

* fix build

* are you happy fmt

* anonimyze implementations

* sliding window avg (#826)

* Actix web telemetry (#828)

* actix web telemetry

* small as move

* use tokio mutex instead of std

* add comments

* are you happy fmt

* use u16 as http status code

* telemetry structs rename

* fix build

* using parking lot mutex

* telemetry web api (#842)

* telemetry web api

* telemetry openapi (#843)

* use async mutex for telemetry collector

* use tokio mutex for telemetry collector

* are you happy fmt
2022-07-22 19:27:07 +04:00
Ivan Pleshkov
34e3149605 update docs for list full snapshots requests (#841) 2022-07-22 13:11:24 +04:00
Andrey Vasnetsov
89da942cc8 upd tags in openapi 2022-07-18 18:12:40 +02:00
Arnaud Gourlay
9b8b5b6376 [sharding] Collection cluster info API (#822)
* Collection cluster info API

* better output format

* update openapi spec

* local shard info reports number of points

* rebase
2022-07-18 17:57:34 +02:00
Andrey Vasnetsov
123332c867 Full snapshot (#824)
* full snapshot rest api

* openapi for snapshots

* openapi snapshot api

* fmt + clippy

* fix recovery + implement grpc methods

* fmt

* same aliases to full storage snapshot

* fmt
2022-07-18 17:11:23 +02:00
Andrey Vasnetsov
763628178b upd version to v0.8.4 (#781) 2022-07-05 11:13:03 +02:00
Andrey Vasnetsov
098bc4c751 Count api (#777)
* count api

* test for approx counting

* fmt + clippy

* unit test

* add warning
2022-07-05 09:33:38 +02:00
Arnaud Gourlay
b23d3c7243 gRPC API for snapshot (#772) 2022-07-04 19:14:40 +02:00
Andrey Vasnetsov
fac87018c4 Snapshoting API (#764)
* wip: rest api for snapshots

* fmt

* fix tests

* fmt

* wip: collection snapshoting and recovery

* fmt

* remote shard snapshots

* remote shard test

* fmt

* extend proxy snapshot test + fix double read lock

* fmt + clippy

* openapi schema

* Update openapi/openapi-snapshots.ytt.yaml

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update openapi/openapi-snapshots.ytt.yaml

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update src/main.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* Update src/main.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* reduce writes on snapshots location

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2022-07-01 13:21:57 +02:00
Andrey Vasnetsov
9abb73a431 v0.8.3 (#755)
* v0.8.3

* update rest schema
2022-06-28 11:51:29 +02:00
Andrey Vasnetsov
83c21450c3 Pagination (#743)
* offset param for search and recomendation

* test and fix for pagination

* clippy + fmt

* upd schemas

* fix grpc requests in test
2022-06-27 13:22:02 +02:00
Andrey Vasnetsov
c6405efb1f v0.8.2 (#740) 2022-06-24 13:14:48 +02:00
Egor Ivkov
e12b91e10c Promote learner (#735)
* Promote learner

Also fixes ERROR voters not found

* Review fixes

* Review: Find learner first
2022-06-24 11:27:58 +03:00
Andrey Vasnetsov
5a9378d3b6 upd readme and logo (#719)
* upd readme and logo

* fix typo

* commerce link
2022-06-21 17:52:56 +02:00
Andrey Vasnetsov
9871cace14 upd version to 0.8.1 (#708) 2022-06-16 15:30:46 +02:00
Andrey Vasnetsov
ddf940c491 allow server selection in OpenAPI schema (#682) 2022-06-14 09:54:47 +02:00
Andrey Vasnetsov
63b868233e release v0.8.0 (#667)
* release v0.8.0

* upd roadmap
2022-06-08 12:47:10 +02:00
Andrey Vasnetsov
fc2f579ec6 smarter default number of shards (#668)
* smarter default number of shards

* upd openapi
2022-06-07 16:59:10 +02:00
Andrey Vasnetsov
a264a8262a fix typos 2022-06-07 15:16:23 +02:00
Gabriel Velo
c15981092a [WIP] [real-time index] Implement payloadstorage for structpayloadindex (#642)
* [real-time index] Extend FieldIndex enum and StructPayloadIndex with method from PayloadStorage

* [real-time index] add missing remove_point methods

* [real-time index] add new index to FieldIndex enum

* fix compile

* are you happy fmt

* merge load and remove

* fix test generics

* decrement points count

* remove from histogram

* simplify histogram usage

* [real-time index] remove old tests and fix clippy warnings

* histogram: method to derive range by size (#657)

* [real-time index] add histogram based payload_blocks implementation.

* payload blocks

* fmt

* clippy

* [real-time index] refactor Segment to use PayloadIndex instead of PayloadStorage.

* fix tests

* fmt

* clippy

* rename indexes

* remove redundent params

* add struct payload deletion test + fix delete payload in map index

* remove payload threshold

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2022-06-06 17:14:20 +02:00
Egor Ivkov
9ddaa1c749 Add leader_id and role to cluster API (#660) 2022-06-03 22:37:50 +02:00
Andrey Vasnetsov
7c254a0de0 preserve fields order in OpenAPI schema (#655) 2022-06-02 19:12:30 +02:00
Andrey Vasnetsov
2601c017de Smarter defaults (#637)
* auto segments number

* auto segments number

* replace vector number limits with vector size limits

* fmt
2022-06-01 18:09:38 +02:00
Andrey Vasnetsov
1b458780eb On disk payload storage (#634)
* implement on-disk payload storage

* fmt + clippy

* config param for on-disk payload storage

* upd openapi definitions

* add integration test with on-disk payload

* fix clippy

* review fixes

* fmt
2022-06-01 17:23:34 +02:00
Andrey Vasnetsov
c286bde562 Cluster api (#631)
* cluster status API

* add cluster api to OpenAPI specification
2022-05-27 14:52:51 +02:00
Andrey Vasnetsov
036c186e0d Better error reporting in enums (#587)
* remove conflicting deprecated apis

* implement custom JsonSchema for the PointInsertOperations

* fmt

* clippy

* revert unnesessare changes
2022-05-23 13:28:17 +02:00
Andrey Vasnetsov
5800319edb refactor distances + add score_threshold + fix negative euclid distance (#569)
* refactor distances + add score_threshold + fix negative euclid distance

* generate docs

* fix clippy

* Update lib/segment/src/spaces/tools.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2022-05-09 17:32:48 +02:00
trean
06668a1fd1 changed code background color in redoc (#561) 2022-05-05 14:31:50 +02:00
Arnaud Gourlay
2ff03c20fd fix typo in openapi spec (#558) 2022-05-05 07:39:57 +02:00
Andrey Vasnetsov
7d12bfcaa5 Update README.md 2022-04-13 10:50:45 +02:00
Andrey Vasnetsov
d8ed9b2b2b v0.7.0 - upd version and docs (#462)
* upd version and docs

* fix clippy
2022-04-11 18:40:35 +02:00
Egor Ivkov
ee4cd768ea Allows specifying collection meta operation commit timeout in API (#444)
* Allows specifying collection meta operation commit timeout in API

Also restructures a part of gRPC collection API

* Review: link to dev docs

* Review: Timeout error description

Also committed generated protobuf code that somehow was left out

* Review: seconds in API and minor proto fixes
2022-04-08 14:42:38 +03:00
Andrey Vasnetsov
ce21abf033 Values count condition (#439)
* add values_count condition

* fmt

* fix tests and clippy

* fmt
2022-04-05 14:31:53 +02:00
Andrey Vasnetsov
b07428f620 Is empty condition (#423)
* is-empty condition

* fmt

* better assert

* fmt
2022-04-03 16:55:51 +02:00