Commit Graph

62 Commits

Author SHA1 Message Date
Gabriel Velo
c15981092a [WIP] [real-time index] Implement payloadstorage for structpayloadindex (#642)
* [real-time index] Extend FieldIndex enum and StructPayloadIndex with method from PayloadStorage

* [real-time index] add missing remove_point methods

* [real-time index] add new index to FieldIndex enum

* fix compile

* are you happy fmt

* merge load and remove

* fix test generics

* decrement points count

* remove from histogram

* simplify histogram usage

* [real-time index] remove old tests and fix clippy warnings

* histogram: method to derive range by size (#657)

* [real-time index] add histogram based payload_blocks implementation.

* payload blocks

* fmt

* clippy

* [real-time index] refactor Segment to use PayloadIndex instead of PayloadStorage.

* fix tests

* fmt

* clippy

* rename indexes

* remove redundent params

* add struct payload deletion test + fix delete payload in map index

* remove payload threshold

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2022-06-06 17:14:20 +02:00
Andrey Vasnetsov
1b458780eb On disk payload storage (#634)
* implement on-disk payload storage

* fmt + clippy

* config param for on-disk payload storage

* upd openapi definitions

* add integration test with on-disk payload

* fix clippy

* review fixes

* fmt
2022-06-01 17:23:34 +02:00
Andrey Vasnetsov
c286bde562 Cluster api (#631)
* cluster status API

* add cluster api to OpenAPI specification
2022-05-27 14:52:51 +02:00
Andrey Vasnetsov
036c186e0d Better error reporting in enums (#587)
* remove conflicting deprecated apis

* implement custom JsonSchema for the PointInsertOperations

* fmt

* clippy

* revert unnesessare changes
2022-05-23 13:28:17 +02:00
Andrey Vasnetsov
5800319edb refactor distances + add score_threshold + fix negative euclid distance (#569)
* refactor distances + add score_threshold + fix negative euclid distance

* generate docs

* fix clippy

* Update lib/segment/src/spaces/tools.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2022-05-09 17:32:48 +02:00
Andrey Vasnetsov
6af867e037 fix and extend interation test for scrolling over points (#494) 2022-04-20 09:52:59 +02:00
Egor Ivkov
ee4cd768ea Allows specifying collection meta operation commit timeout in API (#444)
* Allows specifying collection meta operation commit timeout in API

Also restructures a part of gRPC collection API

* Review: link to dev docs

* Review: Timeout error description

Also committed generated protobuf code that somehow was left out

* Review: seconds in API and minor proto fixes
2022-04-08 14:42:38 +03:00
Andrey Vasnetsov
b07428f620 Is empty condition (#423)
* is-empty condition

* fmt

* better assert

* fmt
2022-04-03 16:55:51 +02:00
Andrey Vasnetsov
adc1f4ad97 Bool filter (#421)
* bool match condition

* use generic values for match requests

* fmt

* upd grpc interface

* upd grpc docs
2022-04-03 16:08:34 +02:00
Andrey Vasnetsov
6fc8d0affe upd openAPI schema names (#416) 2022-03-29 14:12:52 +02:00
Andrey Vasnetsov
8dcb900113 fix intergation test: freeze python deps on working versions 2022-03-28 23:56:08 +02:00
Arnaud Gourlay
2a81fc9142 Remove sled for alias mappings (#402)
* improve alias tests

* introduce new alias persistence and remove Sled

* introduce FileStorageError

* use new error in AliasMapping

* aliases are kept in memory and saved to disk when modified

* better naming

* make alias renaming atomic

* fmt

* use correct alias and docs

* code review: cleaner error conversion
2022-03-23 14:21:04 +01:00
Arnaud Gourlay
cfed400240 gitignore __pycache__/ (#401) 2022-03-21 17:39:40 +01:00
Gabriel Velo
f69a7b740f json as payload (#306)
add json as payload
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2022-03-21 07:09:10 -03:00
Andrey Vasnetsov
20c1e0f20d upd naming in api (#361) 2022-03-03 13:27:51 +01:00
Andrey Vasnetsov
c3cfc93574 fix PayloadSelector enum for better openapi compatibility (#358) 2022-03-02 15:01:27 +01:00
Andrey Vasnetsov
bdc3194b13 allow to return vectors and payload with recommendation request #353 (#357)
* allow to return vectors and payload with recommendation request #353

* fmt

* unify `with_vector` comment line

* upd openapi
2022-03-02 10:42:58 +01:00
Andrey Vasnetsov
0b9463a674 reproduce with integration test and fix #309
* reproduce with integration test

* create new update channel each time we spin up a new update worker
2022-02-15 10:21:46 +01:00
Andrey Vasnetsov
f5feff8080 Validated integration tests - #232
* wip: re-implement http tests using openapi validation library

* collection delete api

* payload operations test

* payload delete operations test

* payload index creation and removing test

* uuid operations test

* schema consistency test

* add tests into pipeline

* chmod +x

* faster local runs with docker cache

* upd instruction
2022-02-11 16:43:47 +01:00
Arnaud Gourlay
80652c5624 clean openapi setup (#288) 2022-02-07 07:59:09 +01:00
Andrey Vasnetsov
4a5cf094a4 fix in create index api: post -> put (#272) 2022-02-01 16:09:38 +01:00
Andrey Vasnetsov
efc0e4643d add missiong wait for collection index operation (#270) 2022-01-31 17:30:15 +01:00
Andrey Vasnetsov
65787f7f55 UUID as point id (#265)
* wip: u64 -> u128 + serialization tests

* breaking: use more flexible structure for saving point ids

* replace u64 external id type with enum

* update openapi definitions for uuid + fix retrieve point api + bash script tests
2022-01-31 13:18:07 +01:00
Andrey Vasnetsov
a2f35044c9 enable with_vector flag in api #50 (#267)
* enable `with_vector` flag in api #50

* fix readme
2022-01-31 11:31:16 +01:00
Tim Eggert
559e7a8055 Delete Points By Filter API #39 (#250)
* Delete Points By Filter API #39

* make delete_by_filter part of existing delete query + fix merge issues #39

* apply fmt

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2022-01-25 16:22:18 +01:00
Andrey Vasnetsov
9440143af0 rename point update api structure #208 (#251) 2022-01-24 19:32:14 +01:00
Andrey Vasnetsov
d51a70fa93 add openapi validation during generation #208 (#248)
* add openapi validation during generation #208

* fix: POST -> PUT in point update api implementation and docs #208

* fix: openapi structure exposure

* fix: api usage in stress test
2022-01-24 17:33:57 +01:00
Andrey Vasnetsov
02f53f2ef0 fix: openapi specs validation #208 (#246)
* fix: openapi specs validation #208

* fix: upd api descriptions
2022-01-24 11:46:14 +01:00
Arnaud Gourlay
97cb5091bc Split Points API #208 (#221)
Split Points API #208
2022-01-24 07:54:47 +01:00
Andrey Vasnetsov
898b692f39 [WIP] Limit segment size #135 (#195)
* add parameters to optimizer config

* benchmark search speed in different segment sizes

* use constructor for FilteredScorer

* * Implement benchmarks for HNSW index search with different number of
  stored points
* Fix minor issue in HNSW graph edge assignment
* Update profiler with call-graph report generation
* Add profiling guide
* Add HNSW graph statistics test function (debug inly)

* limit resulting segment size in merge optimizer

* fix clippy

* stop the music

* fix clippy once again

* fmt once again
2022-01-18 15:49:35 +01:00
Andrey Vasnetsov
38ee907af6 minor style fixes 2022-01-04 01:02:50 +01:00
Daniil
1ad529c315 Allow to include vector into search result (#176)
* feat(#50): include vector into search result

allow to specify 'with_vector' parameter in search api to get search results vector data

* test(#50): fix tests

* chore(#50): apply cargo fmt

* chore(#50): update api docs

run tools/generate_openapi_models.sh

Co-authored-by: Daniil Sunyaev <dasforrum@gmail.com>
2022-01-03 17:12:01 +01:00
Andrey Vasnetsov
f4a0511385 Add OpenAPI version selector (#171)
* add OpenAPI version selector

* fix typo
2022-01-03 11:34:09 +01:00
Andrey Vasnetsov
d7148126ce Split collection update API into several endpoints (#126)
* split storage operation structures #32

* cargo fmt #32

* split collection update api into several endpoints #32

* cargo fmt #32

* fix tonic-related code with new structures

* upd alias structures

* use ytt teplate engine for OpenAPI Endpoint schema generation
2021-12-13 09:38:33 +01:00
Andrey Vasnetsov
b73b641403 Update api descriptions 37 (#125)
* upd API description in OpenAPI #37

* upd openapi
2021-11-28 12:45:55 +01:00
Andrey Vasnetsov
e7e13162c2 add openblas patch to the root package 2021-10-27 23:02:41 +02:00
Andrey Vasnetsov
be1474d3ed upd version to 0.4.1 2021-10-25 23:03:59 +02:00
Andrey Vasnetsov
29830f953d default payload return is null + update openAPI 2021-10-24 18:47:58 +02:00
Andrey Vasnetsov
3a7485aa8d update OpenAPI schema 2021-10-24 18:21:17 +02:00
Andrey Vasnetsov
b4b39f8f75 v0.3.6 2021-08-30 00:12:59 +02:00
Andrey Vasnetsov
d5d8d2c407 fix serde deserialisation issue in WAL 2021-07-11 21:22:22 +02:00
trean
d30e6fa8ee Implementation of points scroll API #38 (#40)
* WIP: filtered points iterator #38

* add paginated filtered point request function #38

* add scroll api + openapi definitions #38

* fix openapi #38
2021-06-20 15:30:12 +02:00
Andrey Vasnetsov
cfc5beeac7 add payload schema to collection info + indexing fixes 2021-06-13 22:31:09 +02:00
Andrey Vasnetsov
3616631300 Filtrable hnsw (#26)
* raw points scorer

* raw point scorer for memmap storage

* search interface prepare

* graph binary saving + store PointOffsetId as u32

* WIP: entry points

* connect new link method

* update libs + search layer method + visited list + search context + update rust

* implement Euclid metric + always use MinHeap for priority queue

* small refactor

* search for 0 level entry

* update visited pool to be lock free and thread safe

* use ef_construct from graph layer struct + limit visited links to M

* add metric pre-processing before on vector upsert

* old hnsw heuristic

* save hnsw graph for export

* search method + tests

* small fixes

* add benchmark and profiler

* build time optimizations

* use SeaHash

* remove unsed benchmark

* merge hnsw graph function

* WIP:HNSW index build function

* HNSW build_index with additional indexing

* refactor fixtures

* graph save and load test

* test and fixes for filterable HNSW

* enable hnsw index for query planning

* fix cardinality estimation tests + remove query planner as class

* small refactor

* store full copy of collection settings with collection + allow partial override on creation #16

* API for updating collection parameters #16

* refactor: move collection error -> types

* report collection status in info API #17

* update OpenAPI Schema
2021-05-30 17:14:42 +02:00
Andrey Vasnetsov
dba7c9e558 fix filter query cardinality estimation 2021-05-04 20:12:51 +02:00
Andrey Vasnetsov
904d23c7cc fix openapi definitions 2021-04-06 14:13:41 +02:00
Andrey Vasnetsov
bb94656568 fix openapi 2021-04-06 14:13:41 +02:00
Andrey Vasnetsov
5231787ecd update OpenAPI definitions 2021-04-06 14:13:41 +02:00
Andrey Vasnetsov
ff2b1bfe60 Describe wait param in OpenAPI 2021-02-09 17:11:16 +01:00
Andrey Vasnetsov
44485277d0 hot-fix documentation 2021-01-25 00:13:37 +01:00