Arnaud Gourlay and Andrey Vasnetsov
c5558ab22b
Probabilistic search sampling for better limit ( #1199 )
...
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
2022-11-16 20:53:58 +01:00
dependabot[bot]
96a99935e9
Bump memmap2 from 0.5.7 to 0.5.8 ( #1210 )
...
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs ) from 0.5.7 to 0.5.8.
- [Release notes](https://github.com/RazrFalcon/memmap2-rs/releases )
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.5.7...v0.5.8 )
---
updated-dependencies:
- dependency-name: memmap2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-11-09 21:51:07 +01:00
dependabot[bot]
8504cf502a
Bump num_cpus from 1.13.1 to 1.14.0 ( #1203 )
2022-11-09 08:31:47 +00:00
dependabot[bot]
50a6c10c1d
Bump ordered-float from 3.3.0 to 3.4.0 ( #1204 )
...
Bumps [ordered-float](https://github.com/reem/rust-ordered-float ) from 3.3.0 to 3.4.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases )
- [Commits](https://github.com/reem/rust-ordered-float/compare/v3.3.0...v3.4.0 )
---
updated-dependencies:
- dependency-name: ordered-float
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-11-09 08:42:10 +01:00
Andrey Vasnetsov
7e4424ab57
Cosine dist zero vectors ( #1198 )
...
* skip pre-processing of zero-length vector for cosine distance
* fmt
2022-11-08 17:36:20 +01:00
Ivan Pleshkov
d555a244c2
Hnsw links storage ( #1171 )
...
* hnsw compact links storage
* dump links memsize
* are you happy fmt
* fix build
* calculate capacity and add shrink
* fmt
* remove garbage
* fix unit tests
* default values for graph layers
* remove obsolete fields
* add comments
* more comments
* use one flattened links structure
* refactor and add unit tests
* move merge into graph builder
* fix messages in test_hnsw_graph_properties
2022-11-08 13:42:50 +04:00
dependabot[bot]
558d12b255
Bump pprof from 0.10.1 to 0.11.0 ( #1188 )
...
Bumps [pprof](https://github.com/tikv/pprof-rs ) from 0.10.1 to 0.11.0.
- [Release notes](https://github.com/tikv/pprof-rs/releases )
- [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/tikv/pprof-rs/commits )
---
updated-dependencies:
- dependency-name: pprof
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-11-07 14:41:44 +01:00
98fc5b2fa4
0.11.1 cherry picks ( #1193 )
...
* Complete feature road-map
* docs: add erare-humanum as a contributor for code (#1181 )
* docs: update README.md [skip ci]
* docs: update .all-contributorsrc [skip ci]
Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
* update rust version and fix clippy complains (#1190 )
* handle collections transition from single-node to cluster (#1189 )
* Update to v0.11.1 (#1191 )
* update to v0.11.1
* also default version
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
* consensus_state to consensus_state
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
2022-11-07 11:31:12 +01:00
Ivan Pleshkov and Andrey Vasnetsov
017df52528
Use less ram for id tracker ( #1176 )
...
* use less ram for id tracker
* are you happy clippy
* use vec for internals
* use versions for internal ids
* keys test
* Use less ram for id tracker fixes (#1182 )
* WIP: internal_to_version
* fmt
* fix unit tests
* add comment
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com >
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
2022-10-31 10:44:00 +04:00
erare-humanum and Andrey Vasnetsov
93a4499859
Update: unmaintained crate memmap -> memmap2 ( #559 ) ( #1160 )
...
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
2022-10-30 11:50:27 +01:00
Andrey Vasnetsov
071d1948b0
update and improve openapi definitions
2022-10-24 16:04:31 +02:00
Andrey Vasnetsov
ab19576c5d
Force sync vector storage offset ( #1110 )
...
* simplify vector storage `update` semantic, always assume inplace vector replacement
* fmt
2022-10-24 12:34:26 +02:00
Andrey Vasnetsov
54e0857708
ignore points with missing expernal ids instead of panic ( #1109 )
2022-10-24 12:33:43 +02:00
Arnaud Gourlay
1ff09516a0
limit set to zero returns an empty result ( #1083 )
2022-10-24 12:32:46 +02:00
Andrey Vasnetsov
21754ef2ef
report indexed payload points in info api + other fixes ( #1164 )
...
* report indexed payload points in info api + other fixes
* rollback debug changes
* clippy
* clippy
2022-10-24 09:43:02 +02:00
Ivan Pleshkov and Andrey Vasnetsov
516dcd7020
Telemetry level of detail ( #1049 )
...
* telemetry level of detail
* rename duration aggregator
* are you happy fmt
* move total searches sum
* separate levels
* optional bucket size
* search telemetry improvements
* separate web telemetry into methods
* tonic telemetry methods
* merge optimizations
* are you happy fmt
* better rounding
* qdrant configs on level 1
* provide collection params
* add peers count
* collection points count
* update openapi
* use pattern in actix telemetry
* are you happy fmt
* merge dev
* are you happy fmt
* fix merge conflicts
* update openapi
* fix build
* are you happy fmt
* add exact searches statistics
* process replica set
* update openapi
* fix wrong name
* fix naming
* fix unwrap
* review
* fmt
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
2022-10-23 00:48:55 +02:00
Ivan Pleshkov and Andrey Vasnetsov
44626bc80a
exact search parameter ( #1150 )
...
* exact search parameter
* exact search test
* decrease attempts count because of test speed
* minor review fixes
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
2022-10-20 15:39:21 +04:00
Andrey Vasnetsov
bf48066145
Full reassign payload on upsert ( #1148 )
...
* set_full_payload on upsert
* fmt
* fix clippy
2022-10-20 10:26:33 +02:00
dependabot[bot]
2bd7a36006
Bump ordered-float from 3.2.0 to 3.3.0 ( #1147 )
...
Bumps [ordered-float](https://github.com/reem/rust-ordered-float ) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases )
- [Commits](https://github.com/reem/rust-ordered-float/compare/v3.2.0...v3.3.0 )
---
updated-dependencies:
- dependency-name: ordered-float
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-20 08:56:18 +02:00
Andrey Vasnetsov
ef7b4ac2a7
propagate payload index to temporary segment during optimization ( #1133 )
2022-10-18 10:49:37 +02:00
Arnaud Gourlay and Andrey Vasnetsov
9324be1d60
[Replication] Add replicas ( #1085 )
...
* Add replicas to ReplicaSet
* unproxify shard & miscs
* no exlusive write locking while adding replicas
* on_optimizer_config_update on Shard with async. recursion
* no exlusive write locking while removing replicas
* shortcut replica set propagation #1101
* remove unused field
* make RequestShardTransfer callback sync.
* promote local & remote to replica state
* fixes for replica sync api (#1123 )
* code review
* fix replica set update - fail only if all failed
* Add replica redesign (#1131 )
* refactor shard/mod.rs
* wip
* fmt
* it compiles
* temporary disable replica placemeant change on replication factor
* fmt
* finish todos
* small refactoring
* remove change::add
* replica-set -> shard-replica-set
* fmt
* upd openapi
* fix finish transfer logic
* fix existing integration tests
* shard transfer validation
* fmt
* review fixes
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
2022-10-17 20:12:10 +02:00
Andrey Vasnetsov
35ae83f3fc
validate geo coordinate before adding it to the payload index ( #1125 )
...
* validate geo coordinate before adding it to the payload index
* revert changes in api test
2022-10-13 09:14:11 +02:00
dependabot[bot]
d0789f3a90
Bump uuid from 1.1.2 to 1.2.1 ( #1111 )
...
Bumps [uuid](https://github.com/uuid-rs/uuid ) from 1.1.2 to 1.2.1.
- [Release notes](https://github.com/uuid-rs/uuid/releases )
- [Commits](https://github.com/uuid-rs/uuid/compare/1.1.2...1.2.1 )
---
updated-dependencies:
- dependency-name: uuid
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-11 08:25:24 +02:00
dependabot[bot]
414ba89e9b
Bump ordered-float from 3.1.0 to 3.2.0 ( #1090 )
...
Bumps [ordered-float](https://github.com/reem/rust-ordered-float ) from 3.1.0 to 3.2.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases )
- [Commits](https://github.com/reem/rust-ordered-float/compare/v3.1.0...v3.2.0 )
---
updated-dependencies:
- dependency-name: ordered-float
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-04 11:17:59 +02:00
dependabot[bot]
814b0ebcb4
Bump schemars from 0.8.10 to 0.8.11 ( #1089 )
...
Bumps [schemars](https://github.com/GREsau/schemars ) from 0.8.10 to 0.8.11.
- [Release notes](https://github.com/GREsau/schemars/releases )
- [Changelog](https://github.com/GREsau/schemars/blob/master/CHANGELOG.md )
- [Commits](https://github.com/GREsau/schemars/compare/v0.8.10...v0.8.11 )
---
updated-dependencies:
- dependency-name: schemars
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-04 11:17:22 +02:00
Andrey Vasnetsov
c333e95019
fix the filter-based points remove operation in proxy segment ( #1050 )
...
* fix the filter-based points remove operation in proxy segment
* propagate primary clause in proxy segment
2022-09-24 01:04:51 +02:00
dependabot[bot]
3bfd81d6c6
Bump ordered-float from 3.0.0 to 3.1.0 ( #1039 )
...
Bumps [ordered-float](https://github.com/reem/rust-ordered-float ) from 3.0.0 to 3.1.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases )
- [Commits](https://github.com/reem/rust-ordered-float/compare/v3.0.0...v3.1.0 )
---
updated-dependencies:
- dependency-name: ordered-float
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-21 10:45:46 +02:00
Andrey Vasnetsov
03d08236a8
better struct for fts ( #1026 )
2022-09-16 14:07:50 +02:00
dependabot[bot]
519c4db868
Bump semver from 1.0.13 to 1.0.14 ( #1024 )
...
Bumps [semver](https://github.com/dtolnay/semver ) from 1.0.13 to 1.0.14.
- [Release notes](https://github.com/dtolnay/semver/releases )
- [Commits](https://github.com/dtolnay/semver/compare/1.0.13...1.0.14 )
---
updated-dependencies:
- dependency-name: semver
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-15 21:56:32 +02:00
Andrey Vasnetsov
19ac5b4750
change service error into field error ( #1023 )
2022-09-15 15:24:00 +02:00
Ivan Pleshkov and Andrey Vasnetsov
8c6507987b
Integrate tiny vec to named vectors ( #1007 )
...
* integration
* are you happy fmt
* are you happy clippy
* implement partial eq
* is that it?
* small renaming
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
2022-09-15 14:45:48 +04:00
Ivan Pleshkov
22fd3a3963
minor doc and api in tests updates ( #1022 )
2022-09-15 13:37:38 +04:00
dependabot[bot]
1508ce83be
Bump criterion from 0.3.6 to 0.4.0 ( #1004 )
...
Bumps [criterion](https://github.com/bheisler/criterion.rs ) from 0.3.6 to 0.4.0.
- [Release notes](https://github.com/bheisler/criterion.rs/releases )
- [Changelog](https://github.com/bheisler/criterion.rs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/bheisler/criterion.rs/compare/0.3.6...0.4.0 )
---
updated-dependencies:
- dependency-name: criterion
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-13 10:18:56 +02:00
Andrey Vasnetsov
ba26e2f85e
Faster filtered scroll ( #1003 )
...
* faster filtered scroll for low cardinality filters
* add test
* scroll strategy heuristics
2022-09-12 17:55:56 +02:00
Andrey Vasnetsov and Ivan Pleshkov
4ae6552379
tiny-map + benchmark ( #1001 )
...
* tiny-map + benchmark
* are you happy clippy
* lower default capacity
* fmt
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com >
2022-09-12 01:56:47 +02:00
Ivan Pleshkov and Andrey Vasnetsov
f6b2186193
[WIP] Many named vectors per point ( #958 )
...
* many named vectors per point (segment-level)
* operation result for dim function
* beautifulized vector name
* fix naming bug
* segment version migration
* fmt
* add segment tests
* are you happy clippy
* fix build
* [WIP] many named vectors per point (collection-level) (#975 )
* config and search
* fix placeholders for proxy segment move
* remove VectorType from collection
* are you happy fmt
* vectors in grps messages
* create collections with vectors
* segment holder fixes
* are you happy fmt
* remove default vector name placeholders
* are you happy fmt
* are you happy clippy
* fix build
* fix web api
* are you happy clippy
* are you happy fmt
* record vector&vectors
* openapi update
* fix openapi integration tests
* segment builder fix todo
* vector names for update from segment
* remove unwrap
* backward compatibility
* upd openapi
* backward compatible PointStruct
* upd openapi
* fix record back-comp
* fmt
* vector configuration backward compatibility
* fix vetor storage size estimation
* fmt
* multi-vec segment test + index test
* fmt
* api integration tests
* [WIP] Named vectors struct (#1002 )
* move to separate file
* named vectors as struct
* use cow
* fix build
* keys iterator
* avoid copy in PointStruct -> get_vectors
* avoid another copy
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
2022-09-11 20:59:23 +02:00
24ac939b4a
Sync Points API ( #985 )
...
* shard sync operation
* fnt
* tests
* fix test
* Update lib/collection/src/collection_manager/segments_updater.rs
Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com >
* Update lib/collection/src/collection_manager/segments_updater.rs
Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com >
* match payload after vector only
* Update lib/collection/src/collection_manager/segments_updater.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com >
Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com >
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com >
2022-09-05 15:09:39 +02:00
Andrey Vasnetsov
b9eee55a9f
Full text search ( #963 )
...
* allow additional params for payload field index
* fmt
* wip: full text index building
* fmt
* text search request
* text search request
* full text index persitance and loading
* fmt
* enable fts index in mapping
* clippy
* fix tests + add integration test
* review fixes: extend payload index test
* revert incedental change
2022-09-01 12:50:12 +02:00
Andrey Vasnetsov
be38254ee8
small refactoring ( #746 )
...
* small refactoring
* fix tests
2022-09-01 12:36:28 +02:00
dependabot[bot]
4dd9d53b8d
Bump geo from 0.22.1 to 0.23.0 ( #964 )
...
Bumps [geo](https://github.com/georust/geo ) from 0.22.1 to 0.23.0.
- [Release notes](https://github.com/georust/geo/releases )
- [Changelog](https://github.com/georust/geo/blob/main/CHANGES.md )
- [Commits](https://github.com/georust/geo/compare/geo-0.22.1...geo-0.23.0 )
---
updated-dependencies:
- dependency-name: geo
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-08-29 12:13:32 +02:00
Arnaud Gourlay
95dcc333b0
properly format doc links for cargo doc ( #956 )
2022-08-23 09:09:34 +02:00
Ivan Pleshkov
f9fb0777a0
Wrap rocksdb column usages ( #951 )
...
* wrap rocksdb column usages
* remove obsolete comments
* are you happy clippy
2022-08-22 09:41:08 +02:00
Andrey Vasnetsov and Arnaud Gourlay
a2acca0345
Segment batch search ( #813 )
...
* batch search benchmark
* collect filter iterator in indexed search
* fmt
* fix
* fix
* fmt
* use new tempfile create
* auto batching
* Clippy fixes
* REST, gRPC and internal APIs
* fix bugs & less duplication
* two steps payload retrieval mechanism & fix duplication
* add proxy_segment implementation & tests
* add gRPC docs
* remove unused code (#950 )
* only filter ids within a batch
* add more equivalence tests
* add integration test search vs batch
* assert more search options in tests
* cleanup assertions
* fix offset panic
* rename search batch API
* openapi spec
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com >
2022-08-18 14:48:17 +02:00
f357bd5d9b
Allow to flush segment in separate thread ( #927 )
...
* allow to flush segment in separate thread
* flush as separate function (#928 )
* flush as separate function
* review suggestion
* reduce locks during vector scoring
* fmt
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
* don't run background flush twice
* Update lib/segment/src/segment.rs
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
* increase flush interval
* Update lib/segment/src/segment.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com >
* are you happy fmt
* test background flush
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com >
2022-08-15 13:47:52 +04:00
Arnaud Gourlay
38c8097fc8
Clippy next ( #941 )
...
* Clippy derive_partial_eq_without_eq
* Clippy explicit_auto_deref
* Clippy single_match
* Clippy manual_find_map
* Clippy unnecessary_to_owned
* Clippy derive_partial_eq_without_eq
* Clippy get_first
2022-08-12 09:38:31 +02:00
Russ Cam
8e1f2ca353
Use tempfile ( #922 )
...
This commit replaces tempdir with tempfile.
tempdir is archived.
Closes #544
2022-08-10 09:39:21 +02:00
dependabot[bot] and Arnaud Gourlay
9713267358
Bump rocksdb from 0.18.0 to 0.19.0 ( #920 )
...
* Bump rocksdb from 0.18.0 to 0.19.0
Bumps [rocksdb](https://github.com/rust-rocksdb/rust-rocksdb ) from 0.18.0 to 0.19.0.
- [Release notes](https://github.com/rust-rocksdb/rust-rocksdb/releases )
- [Changelog](https://github.com/rust-rocksdb/rust-rocksdb/blob/master/CHANGELOG.md )
- [Commits](https://github.com/rust-rocksdb/rust-rocksdb/compare/v0.18.0...v0.19.0 )
---
updated-dependencies:
- dependency-name: rocksdb
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
* iterator_cf returns a Result
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com >
2022-08-08 11:12:22 +02:00
Andrey Vasnetsov
70317d1bbc
check for segment integrity by checking the version, skip on failure ( #904 )
2022-08-03 17:39:13 +02:00
Andrey Vasnetsov
88c672475e
Fix integration test ( #903 )
...
* fix integration test problems + overflow
* fmt
2022-08-03 16:45:34 +02:00
Andrey Vasnetsov
826165cfcf
fix enum conversion in gRPC ( #898 )
2022-08-03 09:48:30 +02:00
