Commit Graph

1780 Commits

Author SHA1 Message Date
Ivan Pleshkov
883a46ac8e are you happy clippy? 2022-01-04 23:13:59 +03:00
Ivan Pleshkov
8daacbd160 are you happy fmt? 2022-01-04 18:19:34 +03:00
Ivan Pleshkov
55efa601ad restore copied call 2022-01-04 18:09:49 +03:00
Ivan Pleshkov
1e5670399f Merge branch 'master' into remove-blas 2022-01-04 18:05:39 +03:00
Ivan Pleshkov
8a644a5ced bugfix 2022-01-04 17:48:33 +03:00
Ivan Pleshkov
cd2f334271 horizontal sum and test 2022-01-04 16:30:36 +03:00
Andrey Vasnetsov
3b2b1bbf93 use seeded rng for stochastic test 2022-01-04 11:25:24 +01:00
Ivan Pleshkov
70e376bf5e add avx2 implementation 2022-01-04 00:32:13 +03:00
Ivan Pleshkov
297f54141d remove blas 2022-01-03 22:16:27 +03:00
Anton Kaliaev
cb0a1b6bbc use map_or instead of map & unwrap_or (#172)
https://rust-lang.github.io/rust-clippy/master/index.html#map_unwrap_or
2022-01-03 17:28:56 +01:00
Anton Kaliaev
298685102c add missing commas (#173) 2022-01-03 17:28:46 +01:00
Anton Kaliaev
57fa65072f use copied instead of cloned (#174)
* use copied instead of cloned

https://rust-lang.github.io/rust-clippy/master/index.html#cloned_instead_of_copied

* use copied instead of cloned
2022-01-03 17:28:36 +01:00
Anton Kaliaev
e196ce025b fix clippy warnings (#175)
* fix clippy warnings

- doc links
- explicit deref
- if instead of match for single bool condition
- combine similar match branches

* revert removal of transmute

* return Some when vec is not empty
2022-01-03 17:28:21 +01:00
Daniil
1ad529c315 Allow to include vector into search result (#176)
* feat(#50): include vector into search result

allow to specify 'with_vector' parameter in search api to get search results vector data

* test(#50): fix tests

* chore(#50): apply cargo fmt

* chore(#50): update api docs

run tools/generate_openapi_models.sh

Co-authored-by: Daniil Sunyaev <dasforrum@gmail.com>
2022-01-03 17:12:01 +01:00
Ivan Pleshkov
bcaa160ad8 Remove dyn Metric from vector storage and use generics (#163)
* Remove dyn Metric from vector storage and use generics

* upd benchmark code

* fix benchmark usage

* fmt

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2021-12-31 09:38:33 +01:00
Arnaud Gourlay
b3fbd0f1c8 rocksdb 0.17.0 (#149) 2021-12-15 09:53:36 +01:00
Arnaud Gourlay
072454ee68 Fix benches compilation for TestRawScorerProducer::new (#142) 2021-12-13 17:16:15 +01:00
Andrey Vasnetsov
617b97d3f7 add comments for segment entitites (#136)
* add comments for segment entitites

* fmt

* cargo fmt
2021-12-09 11:06:25 +01:00
Andrey Vasnetsov
5e551cb810 fix new version of clippy 2021-12-05 19:24:04 +01:00
Andrey Vasnetsov
8f4acc41f0 replace incompatible hashlib (#132) 2021-11-27 16:11:01 +01:00
Anton V
c922e16d2b Upgrade rust version (#127)
* - update rust version in Dockerfile
- use rust edition 2021

* - update rust edition for libs

* remove *.lock from gitignore
2021-11-24 11:26:19 +03:00
Andrey Vasnetsov
77bc6899ec fix refactoring (#124) 2021-11-21 19:32:18 +01:00
Marcin Puc
c1a790a7af Add more refactorings (#122) 2021-11-12 19:16:00 +01:00
Marcin Puc
fdcc4fa674 Fix ndarray imports in segment benches (#121) 2021-11-11 10:42:27 +01:00
Marcin Puc
c603f0075e Add various refactorings (#118) 2021-11-10 21:32:25 +01:00
Andrey Vasnetsov
29830f953d default payload return is null + update openAPI 2021-10-24 18:47:58 +02:00
Andrey Vasnetsov
bf3d8c2575 data consistency fixes and updates (#112)
* update segment version after completed update only

* more stable updates: check pre-existing points on update, fail recovery, WAL proper ack. check_unprocessed_points WIP

* switch to async channel

* perform update operations in a separate thread (#111)

* perform update operations in a separate thread

* ordered sending update signal

* locate a segment merging versioning bug

* rename id_mapper -> id_tracker

* per-record versioning

* clippy fixes

* cargo fmt

* rm limit of open files

* fail recovery test

* cargo fmt

* wait for worker stops befor dropping the runtime
2021-10-24 18:10:39 +02:00
Andrey Vasnetsov
ab58a41a11 fix seeded rng in caused blinking test 2021-10-12 22:05:16 +02:00
HaiCheViet
f55e5aa7b7 Features/filter payload (#104)
* update more test

* update fmt

* reduce non usecode and update docker version

* update commend code

* update name filter

* renames and minor fixes

* fix linter

Co-authored-by: hai che <haiche@jobhop.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Andrey Vasnetsov <vasnetsov93@gmail.com>
2021-10-12 11:07:36 +02:00
Andrey Vasnetsov
370d16d878 use seeded random number generator in search graph tests 2021-10-11 20:41:04 +02:00
Andrey Vasnetsov
a7c865c24c use custom openblas fork wit implemented dynamic arch flag (#106)
* use custom openblas fork wit implemented dynamic arch flag

* add comment
2021-10-06 19:20:57 +03:00
Konstantin
20d454e8e0 Exposed update_colletions operation over grpc (#89) 2021-09-05 14:17:40 +02:00
Andrey Vasnetsov
6cab7f2a0b dynamic arch (#79)
* dynamic arch

* fix fmt
2021-08-29 23:31:55 +02:00
Alexander Galibey
f3e8194310 Remove AtomicRefCell wrapper for condition checker (#84) 2021-08-03 10:35:55 +02:00
Andrey Vasnetsov
0bd0a1da42 fix new clippy suggestions 2021-07-30 23:49:02 +02:00
Daniil Naumetc
dd304495f5 Small cosmetics (#66)
* some small cosmetics

* fixed "expect" linting

* fix formation
2021-07-15 15:25:44 +02:00
Andrey Vasnetsov
2bfcac726e disable debug test 2021-07-11 21:54:49 +02:00
Andrey Vasnetsov
d5d8d2c407 fix serde deserialisation issue in WAL 2021-07-11 21:22:22 +02:00
Konstantin G
f70be9bb35 Own search runtime out of the async scope 2021-07-05 23:07:45 +01:00
Konstantin
93e0fb5c2c [CLIPPY] Fix the last portion of rules and enable CI check (#53)
* [CLIPPY] Fixed the warning for references of the user defined types

* [CLIPPY] Fix module naming issue

* [CLIPPY] Fix the last set of warnings and enable clippy check during CI

* Moved cargo fmt and cargo clippy into it's own action
2021-07-03 13:12:21 +02:00
Konstantin
0e1a6e1750 [Clippy] Fix a range of warnings (#52) 2021-07-02 17:51:54 +02:00
Konstantin
d796c9da42 Avoid useless vector copy during scoring (#51)
* Avoid vector copy during scoring

* Fixing ptr_arg clippy rules for &[VectorElementType]
2021-07-02 15:17:04 +02:00
Konstantin
b253b03ae1 [CLIPPY] Fix File::write and option_env! (#49)
* Replace File::write to write_all to ensure the whole vector is written

* [CLIPPY] Replace option_env! with env!
2021-07-02 10:41:01 +02:00
Konstantin
a667747369 Applied and enforced rust fmt code formatting tool (#48)
* Apply cargo fmt command

* Enabled cargo fmt on build
2021-06-29 10:17:50 +02:00
Konstantin
840bd1fc70 Avoid cloning ScoredPointOffset when peeking the top scores (#44) 2021-06-28 00:55:59 +02:00
trean
d30e6fa8ee Implementation of points scroll API #38 (#40)
* WIP: filtered points iterator #38

* add paginated filtered point request function #38

* add scroll api + openapi definitions #38

* fix openapi #38
2021-06-20 15:30:12 +02:00
Andrey Vasnetsov
cfc5beeac7 add payload schema to collection info + indexing fixes 2021-06-13 22:31:09 +02:00
Andrey Vasnetsov
e5cb4ee9fe fix slow HNSW search + fix looping in mmap optimizer + make segment removal atomic 2021-06-06 22:20:45 +02:00
Andrey Vasnetsov
0514721912 add debug logging for HNSW build + small refactoring 2021-05-31 14:48:07 +02:00
Andrey Vasnetsov
3616631300 Filtrable hnsw (#26)
* raw points scorer

* raw point scorer for memmap storage

* search interface prepare

* graph binary saving + store PointOffsetId as u32

* WIP: entry points

* connect new link method

* update libs + search layer method + visited list + search context + update rust

* implement Euclid metric + always use MinHeap for priority queue

* small refactor

* search for 0 level entry

* update visited pool to be lock free and thread safe

* use ef_construct from graph layer struct + limit visited links to M

* add metric pre-processing before on vector upsert

* old hnsw heuristic

* save hnsw graph for export

* search method + tests

* small fixes

* add benchmark and profiler

* build time optimizations

* use SeaHash

* remove unsed benchmark

* merge hnsw graph function

* WIP:HNSW index build function

* HNSW build_index with additional indexing

* refactor fixtures

* graph save and load test

* test and fixes for filterable HNSW

* enable hnsw index for query planning

* fix cardinality estimation tests + remove query planner as class

* small refactor

* store full copy of collection settings with collection + allow partial override on creation #16

* API for updating collection parameters #16

* refactor: move collection error -> types

* report collection status in info API #17

* update OpenAPI Schema
2021-05-30 17:14:42 +02:00