Commit Graph

3618 Commits

Author SHA1 Message Date
Konstantin
12e25089cb Actix update (#55)
* Updated actix to 4.0.0-beta.8

* Refactored search, scroll, update and collection operation APIs to be async
2021-07-05 00:20:16 +02:00
Konstantin
93e0fb5c2c [CLIPPY] Fix the last portion of rules and enable CI check (#53)
* [CLIPPY] Fixed the warning for references of the user defined types

* [CLIPPY] Fix module naming issue

* [CLIPPY] Fix the last set of warnings and enable clippy check during CI

* Moved cargo fmt and cargo clippy into it's own action
2021-07-03 13:12:21 +02:00
Konstantin
0e1a6e1750 [Clippy] Fix a range of warnings (#52) 2021-07-02 17:51:54 +02:00
Konstantin
d796c9da42 Avoid useless vector copy during scoring (#51)
* Avoid vector copy during scoring

* Fixing ptr_arg clippy rules for &[VectorElementType]
2021-07-02 15:17:04 +02:00
Konstantin
b253b03ae1 [CLIPPY] Fix File::write and option_env! (#49)
* Replace File::write to write_all to ensure the whole vector is written

* [CLIPPY] Replace option_env! with env!
2021-07-02 10:41:01 +02:00
Konstantin
a667747369 Applied and enforced rust fmt code formatting tool (#48)
* Apply cargo fmt command

* Enabled cargo fmt on build
2021-06-29 10:17:50 +02:00
Konstantin
840bd1fc70 Avoid cloning ScoredPointOffset when peeking the top scores (#44) 2021-06-28 00:55:59 +02:00
trean
d30e6fa8ee Implementation of points scroll API #38 (#40)
* WIP: filtered points iterator #38

* add paginated filtered point request function #38

* add scroll api + openapi definitions #38

* fix openapi #38
2021-06-20 15:30:12 +02:00
Andrey Vasnetsov
b4a2d0eb1e Update Tokio to the latest version (#36)
* update tokio in collection crate #18

* update tokio in main and storage crates #18
2021-06-18 13:48:37 +02:00
Andrey Vasnetsov
cfc5beeac7 add payload schema to collection info + indexing fixes 2021-06-13 22:31:09 +02:00
Andrey Vasnetsov
e5cb4ee9fe fix slow HNSW search + fix looping in mmap optimizer + make segment removal atomic 2021-06-06 22:20:45 +02:00
Andrey Vasnetsov
0514721912 add debug logging for HNSW build + small refactoring 2021-05-31 14:48:07 +02:00
Andrey Vasnetsov
3616631300 Filtrable hnsw (#26)
* raw points scorer

* raw point scorer for memmap storage

* search interface prepare

* graph binary saving + store PointOffsetId as u32

* WIP: entry points

* connect new link method

* update libs + search layer method + visited list + search context + update rust

* implement Euclid metric + always use MinHeap for priority queue

* small refactor

* search for 0 level entry

* update visited pool to be lock free and thread safe

* use ef_construct from graph layer struct + limit visited links to M

* add metric pre-processing before on vector upsert

* old hnsw heuristic

* save hnsw graph for export

* search method + tests

* small fixes

* add benchmark and profiler

* build time optimizations

* use SeaHash

* remove unsed benchmark

* merge hnsw graph function

* WIP:HNSW index build function

* HNSW build_index with additional indexing

* refactor fixtures

* graph save and load test

* test and fixes for filterable HNSW

* enable hnsw index for query planning

* fix cardinality estimation tests + remove query planner as class

* small refactor

* store full copy of collection settings with collection + allow partial override on creation #16

* API for updating collection parameters #16

* refactor: move collection error -> types

* report collection status in info API #17

* update OpenAPI Schema
2021-05-30 17:14:42 +02:00
Andrey Vasnetsov
c83ddec2cb refactor for PR #15 2021-05-27 00:54:44 +02:00
Joan Fontanals
a633e625f3 allow set full payload from serde_json Values (#15)
* allow set full payload from serde_json Values

* move payloadinterface and variant to segment

* expose direct json string in segment

* refactor extract function

* handle from payloadinterface

* add test on filter nested

* update lib/segment/src/segment.rs

Co-authored-by: Andrey Vasnetsov <vasnetsov93@gmail.com>

* use payloadinterface in collection

Co-authored-by: Andrey Vasnetsov <vasnetsov93@gmail.com>
2021-05-26 23:59:57 +02:00
Andrey Vasnetsov
dba7c9e558 fix filter query cardinality estimation 2021-05-04 20:12:51 +02:00
Andrey Vasnetsov
e9b86cdd46 rename operations 2021-04-06 14:13:41 +02:00
Andrey Vasnetsov
46ba12a198 update readme + change filter structure 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
e1e201a657 cargo fix unused imports 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
e0636b4920 optimized segment building in a separate directory 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
398da04aad add indexing optimizer + enable mmap and structure index rebuilding + fix issues 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
725c33aab2 endpoint option to manage indexes 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
dcbebac80c empty condition is same as no condition 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
0a91d3ae06 struct payload index segment test 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
cd2acd5dac test for cardinality estimation 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
5dd3c1935e WIP: struct payload index texts 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
328f445439 implement index filter 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
c50482d53f WIP: query_points in struct payload index 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
fe44c4e00e WIP: FieldCondition 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
7787f07d57 WIP: PayloadFieldIndex for FieldIndex 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
0dd48e0acb WIP: should, must and must_not estimators 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
4f798062bd WIP: payload save and load 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
c3fe443643 WIP 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
cb9f41a0df implement payload index builder + column index for keywords and integers 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
e5d7ac7721 extend payload storage interface + start functions for creating struct index 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
9fc12658da number field index 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
5a2985c6f7 refactor condition checking for better column storage support 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
62768a8193 recommender API 2021-01-24 23:38:59 +01:00
Andrey Vasnetsov
5c2afdc5b9 geo-radius + update wiki 2021-01-24 17:16:06 +01:00
Andrey Vasnetsov
69dc0f3bef fix docker build 2021-01-16 12:11:24 +01:00
Andrey Vasnetsov
8a85c10934 use BLAS for vector dot production 2021-01-04 22:59:22 +01:00
Andrey Vasnetsov
bd7ed06bfe add benchmark for vector similarity speed comparasion 2021-01-03 01:25:41 +01:00
Andrey Vasnetsov
10161f05e5 fix vector upsert on WAL reconstrution + use f32 for vector storage 2020-12-13 20:54:18 +01:00
Andrey Vasnetsov
0909b70c2c replace sled with rocksdb 2020-12-13 00:26:55 +01:00
Andrey Vasnetsov
eab0ac83ba attempt to replace sled for storing vectors 2020-12-08 16:55:09 +01:00
Andrey Vasnetsov
44c6c0a65c attempt better memory usage for in-memory vector storage 2020-12-08 00:26:10 +01:00
Andrey Vasnetsov
4ba31ef328 fix WAL indexes + stress test 2020-12-02 00:23:29 +01:00
Andrey Vasnetsov
5040278c16 gracefully stop updater on collection descruction 2020-10-27 22:28:54 +01:00
Andrey Vasnetsov
5f4b5850dd use separate runtime for each collection 2020-10-27 21:14:12 +01:00
Andrey Vasnetsov
d8b0e6c283 dockerfile + examples 2020-10-27 19:57:39 +01:00