Commit Graph

34 Commits

Author SHA1 Message Date
Anton Kaliaev
e196ce025b fix clippy warnings (#175)
* fix clippy warnings

- doc links
- explicit deref
- if instead of match for single bool condition
- combine similar match branches

* revert removal of transmute

* return Some when vec is not empty
2022-01-03 17:28:21 +01:00
Ivan Pleshkov
bcaa160ad8 Remove dyn Metric from vector storage and use generics (#163)
* Remove dyn Metric from vector storage and use generics

* upd benchmark code

* fix benchmark usage

* fmt

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2021-12-31 09:38:33 +01:00
Marcin Puc
c603f0075e Add various refactorings (#118) 2021-11-10 21:32:25 +01:00
Andrey Vasnetsov
bf3d8c2575 data consistency fixes and updates (#112)
* update segment version after completed update only

* more stable updates: check pre-existing points on update, fail recovery, WAL proper ack. check_unprocessed_points WIP

* switch to async channel

* perform update operations in a separate thread (#111)

* perform update operations in a separate thread

* ordered sending update signal

* locate a segment merging versioning bug

* rename id_mapper -> id_tracker

* per-record versioning

* clippy fixes

* cargo fmt

* rm limit of open files

* fail recovery test

* cargo fmt

* wait for worker stops befor dropping the runtime
2021-10-24 18:10:39 +02:00
Alexander Galibey
f3e8194310 Remove AtomicRefCell wrapper for condition checker (#84) 2021-08-03 10:35:55 +02:00
Konstantin
93e0fb5c2c [CLIPPY] Fix the last portion of rules and enable CI check (#53)
* [CLIPPY] Fixed the warning for references of the user defined types

* [CLIPPY] Fix module naming issue

* [CLIPPY] Fix the last set of warnings and enable clippy check during CI

* Moved cargo fmt and cargo clippy into it's own action
2021-07-03 13:12:21 +02:00
Konstantin
0e1a6e1750 [Clippy] Fix a range of warnings (#52) 2021-07-02 17:51:54 +02:00
Konstantin
a667747369 Applied and enforced rust fmt code formatting tool (#48)
* Apply cargo fmt command

* Enabled cargo fmt on build
2021-06-29 10:17:50 +02:00
Andrey Vasnetsov
3616631300 Filtrable hnsw (#26)
* raw points scorer

* raw point scorer for memmap storage

* search interface prepare

* graph binary saving + store PointOffsetId as u32

* WIP: entry points

* connect new link method

* update libs + search layer method + visited list + search context + update rust

* implement Euclid metric + always use MinHeap for priority queue

* small refactor

* search for 0 level entry

* update visited pool to be lock free and thread safe

* use ef_construct from graph layer struct + limit visited links to M

* add metric pre-processing before on vector upsert

* old hnsw heuristic

* save hnsw graph for export

* search method + tests

* small fixes

* add benchmark and profiler

* build time optimizations

* use SeaHash

* remove unsed benchmark

* merge hnsw graph function

* WIP:HNSW index build function

* HNSW build_index with additional indexing

* refactor fixtures

* graph save and load test

* test and fixes for filterable HNSW

* enable hnsw index for query planning

* fix cardinality estimation tests + remove query planner as class

* small refactor

* store full copy of collection settings with collection + allow partial override on creation #16

* API for updating collection parameters #16

* refactor: move collection error -> types

* report collection status in info API #17

* update OpenAPI Schema
2021-05-30 17:14:42 +02:00
Andrey Vasnetsov
e0636b4920 optimized segment building in a separate directory 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
398da04aad add indexing optimizer + enable mmap and structure index rebuilding + fix issues 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
725c33aab2 endpoint option to manage indexes 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
5dd3c1935e WIP: struct payload index texts 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
c3fe443643 WIP 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
9fc12658da number field index 2021-04-06 14:12:46 +02:00
Andrey Vasnetsov
0909b70c2c replace sled with rocksdb 2020-12-13 00:26:55 +01:00
Andrey Vasnetsov
162698ac11 collection loader + fixes and load test 2020-10-04 16:49:28 +02:00
Andrey Vasnetsov
7495fd1d81 storage type selection in optimizer 2020-09-29 13:15:18 +02:00
Andrey Vasnetsov
8125611b6f WIP: memmap vector storage 2020-09-22 11:39:48 +02:00
Andrey Vasnetsov
c9a3766fcf simple persistance for vector storage 2020-09-21 21:41:21 +02:00
Andrey Vasnetsov
e51d8bfec5 WIP: persistace 2020-09-20 20:59:58 +02:00
Andrey Vasnetsov
b135c329c8 WIP: implement table of content 2020-09-13 00:10:40 +02:00
Andrey Vasnetsov
57dcaad499 refactor segment optimizer 2020-08-31 23:23:29 +02:00
Andrey Vasnetsov
8404de7594 WIP vacuum optimizer: segment rebuilding option is required 2020-08-17 01:36:11 +02:00
Andrey Vasnetsov
f39b42152b implemented proxy segment + removed wipe operation due to complications in proxy segment implementation 2020-08-16 16:52:13 +02:00
Andrey Vasnetsov
9effaec4ae go with copy of all deleted ids into query, replace refcell with atomic refcell 2020-08-16 14:13:57 +02:00
Andrey Vasnetsov
358f2a6609 add id filter 2020-08-15 21:57:50 +02:00
Andrey Vasnetsov
ef343499b8 add is_appendable flag for segment 2020-08-15 15:24:41 +02:00
Andrey Vasnetsov
c924103166 fix cargo fix 2020-07-27 19:26:08 +02:00
Andrey Vasnetsov
7178468e7b cargo fix 2020-07-27 19:22:51 +02:00
Andrey Vasnetsov
5215e7996e parallel segments searcher + test 2020-07-18 17:17:23 +02:00
Andrey Vasnetsov
29f75a0ecc refactor distance, start segment manager implementation 2020-07-13 00:18:10 +02:00
Andrey Vasnetsov
1a11bfe89d remove explicit storage of deleted points 2020-07-11 17:38:35 +02:00
Andrey Vasnetsov
03c86e7f65 add simple segment builder 2020-07-06 23:50:21 +02:00