Andrey Vasnetsov
167fb518da
Fix recover snapshot peer ( #1534 )
...
* allow to recover distributed snapshot on local intance + allow recover non-existing collections
* async snapshot recovery
* fix tests
* be polite earlier
2023-03-07 17:03:01 +01:00
Ivan Pleshkov and Andrey Vasnetsov
5174388ab4
integrate quantized data to storages ( #1311 )
...
* integrate quantized data to storages
* revert gitignore
* are you happy clippy
* quantize in optimizer
* provide flag
* fix segfault
* skip quantization flag, update scores
* use quantization flag
* are you happy fmt
* use quantization flag
* quantized search test
* are you happy fmt
* refactor test, refactor scorer choosing
* are you happy fmt
* run quantization on segment builder
* decrease testing parameters
* simplify segment
* update version
* remove use_quantization flag
* provide quantization config
* quantization version up
* euclid dist
* add euclid test
* saveload
* fix initialization bugs
* quantization lib version up
* fix arm build
* refactor scorer selecting
* quant lib version up
* are you happy fmt
* are you happy fmt
* are you happy clippy
* add save/load test for simple storage
* add comments
* quantiles
* quantization mmap
* remove f32
* mmap test
* fix mmap slice
* fix mmap test
* use chunks for quantization storage
* fix build
* are you happy fmt
* update quantization library
* update quantization lib
* update quantization lib
* integrate api changes
* are you happy fmt
* change quantization api
* additional checks in tests
* update quantization version
* fix unit tests
* add quantization to storage config
* use quantization for all cardinality search cases
* Integrate quantization suggestions 2 (#1520 )
* review api
* wip: refactor quantization integrations
* wip: refactor quantization integrations
* wip: fmt
* include quantization into snapshot
* fmt
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
2023-03-03 17:46:17 +01:00
Andrey Vasnetsov
45b441a896
manage number of working htreads a bit better ( #1485 )
2023-02-21 15:32:29 +01:00
Roman Titov
cf04b1a92b
Add /cluster/recover API endpoint ( #1451 )
...
- Add required internal `raft::raw_node::RawNode::report_snapshot` calls
- Add methods to propagate `raft::raw_node::RawNode::request_snapshot` call to the consensus thread
- Add `/cluster/recover` API endpooint
- Add `test_collection_recovery.py` integration test (disabled on CI; documents current, incorrect behavior)
2023-02-15 04:19:11 +10:00
Andrey Vasnetsov
99eaabc31a
remove deprecate point ids param in gprc ( #1305 )
...
* remove deprecate point ids param in gprc
* upd docs
2023-02-06 13:37:18 +01:00
Roman Titov and Andrey Vasnetsov
a50e7bdcd4
Add read_consistency parameter to the APIs ( #1371 ) ( #1407 )
...
* WIP: Add `read_consistency` parameter to the APIs
* WIP: Add `read_consistency` parameter to the APIs
TODO:
- Add documentation
* `cargo fmt`
* Add gRPC documentation
* Add OpenAPI documentation
* Cleanup
* fixup! Add OpenAPI documentation
* fixup! Add gRPC documentation
Who would have known there's `generate_grpc_docs.sh`!? 🥲 🙈 🤦♀️
* generate openapi
* Fix `read_consistency` query parameter deserialization
* Further improve `read_consistency` query parameter deserialization
* `cargo clippy`
* Fix `Payload` comparison during read operation result resolving
* Fix grammar
* rename `read_consistency` -> `consistency` and add integration test
* use majority for test
* fix tests
* Fix tests
* fixup! Fix tests
Apply the same fix to `ScoredPoint`
* Remove an `unwrap`
* fixup! Fix tests
Gotta love those negative conditions, or how a missed `!` can ruin your day... 🤦♀️
* Make internal API calls strictly "local-shard only"
* Implement a few basic traits for `ResolverRecord`
* fixup! Implement a few basic traits for `ResolverRecord`
* Revert "Make internal API calls strictly "local-shard only""
This reverts commit 25378e61ffb566bea49bandrey@vasnetsov.com >
2023-02-03 01:12:28 +01:00
Andrey Vasnetsov
e3066b96fc
allow to overwrite max number of cpus with env variable ( #1425 )
2023-02-02 14:44:03 +01:00
Arnaud Gourlay
2d9e015905
Forward write request according to write consistency ( #1405 )
...
* Forward write request according to write consistency
* use gRPC internal update API by making shard_id optional
* improve and test
* mark leader peer as failed in case of service error during forward
* forward write ordering param
2023-01-30 15:02:21 +01:00
Andrey Vasnetsov
9826e0338c
add init state to replica set ( #1404 )
...
* add init state to replica set
* update openapi
* disallow activating shard from non-initialize state
* fmt
* remove useless operation confirm on collection creation
* clippy fixes
* fix test
2023-01-30 14:58:34 +01:00
Arnaud Gourlay
4c0f849749
Clippy rust 1.67 ( #1406 )
...
* inline format! args
* inline format! args
* explicit lifetime could be elided
* fmt
2023-01-26 17:48:52 +01:00
Andrey Vasnetsov
8bc8f63988
Telemetry collection ( #1401 )
...
* sending telemetry requests
* report panics
* fmt
* remove extra runtime
* fix telemetry lock release
2023-01-26 00:49:18 +01:00
Arnaud Gourlay
6a14e527c5
Merge remaining runtimes ( #1398 )
...
* Collections share a single optimizer runtime
* Merge remaining runtimes
* complete optimizer runtime renaming
* Give general_runtime ownership to toc to avoid shutdown issues
2023-01-25 13:43:46 +01:00
Andrey Vasnetsov
f356455b07
add write consistency param to rest and grpc api ( #1397 )
...
* add write consistency param to rest and grpc api
* fix grpcurl + refactor grpc for better extencibility
* upd grpc docs
* fix conversion
2023-01-25 11:49:01 +01:00
Arnaud Gourlay and Andrey Vasnetsov
287ac89369
Collections share a single optimizer runtime ( #1396 )
...
* Collections share a single optimizer runtime
* review fixes
* rename: optimizer_runtime -> update_runtime
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
2023-01-25 08:05:50 +01:00
Andrey Vasnetsov
61d970ef90
Init collection from ( #1364 )
...
* WIP
* WIP: add shards selection
* fmt
* WIP: collection init from other collection
* fmt
* fix bugs
* add collection write lock + use collection manager runtime instead of search runtime
* fmt
* integration test
* better names ?
* payload indexes transfer
2023-01-22 01:16:55 +01:00
Arnaud Gourlay
86b8ff6b7f
Delete snapshots API ( #1377 )
...
* Delete snapshots API
* remove unecessary plurals
2023-01-20 19:28:56 +01:00
Arnaud Gourlay
6f284da22e
List aliases API (rest & gRPC) ( #1363 )
...
* List aliases API (rest & gRPC)
* rework api to avoid conflicts
2023-01-19 07:59:02 +01:00
Arnaud Gourlay
5156fa8966
Stop on fail consensus recovery ( #1347 )
2023-01-17 15:45:45 +01:00
Arnaud Gourlay
1454c0d41c
Retry recovering from known peers ( #1338 )
2023-01-13 10:40:16 +01:00
Arnaud Gourlay
b1bf1e87e1
Add logging to node recovery ( #1337 )
2023-01-12 17:31:28 +01:00
Hozan
f5628f025d
Add the option to load alternative configuration file ( #1328 )
...
* add the option to load alternative configuration file
* minor fix & pass cargo clippy warning
2023-01-10 17:34:57 +01:00
Roman Titov
2ed11d544b
Add configuration option to select "memory advice" used for memmapped storage
...
- Add a simple abstraction/wrapper over `memmap2::Advice` and a global switch
to select "memory advice" used for memmapped storage to the `segment` crate
- Add `mmap_advice` configuration option to the `storage::types::StorageConfig`
- Implement `search-points` benchmark
2023-01-06 21:02:57 +10:00
Andrey Vasnetsov
926a058c73
Fixing missed leader inconsistency ( #1286 )
...
* WIP: introduce local state
* WIP: sync local state with consensus in idle
* fmt
* rm extensive debug
* update triple replication test
* update triple replication test
* rm debug logs
* fix check for established leader + only sync local if no proposals
* remove unused file
* test compatible with python 3.8
* test compatible with python 3.8
* longer wait for consensus
* longer wait for consensus
* extra sleep in test
* remove extra sleep
* Fixing missed leader inconsistency - transfers (#1298 )
* explicit request timeout
* explicit request timeout
* explicit request timeout
* explicit request timeout
* explicit request timeout
* prevent double handelling of the transfer termination
* kill the process
* log on inconsistency
* log on inconsistency
* log on inconsistency
* debug
* revert debug in test
* forward updates to partial shards, abort transactions on dead node report
* fmt
* disable retry of transfer, if the transfer was cancelled + allow predictable ports in test
* fix import
2022-12-23 00:45:04 +01:00
Andrey Vasnetsov
a2c5de586b
Recommendation with other collection ( #1283 )
...
* WIP: move recommendation API out of collection
* recomendations with external source of vectors
* fix test
* fmt
* fixes + test
* review fixes
2022-12-22 09:29:36 +01:00
Arnaud Gourlay
e3c6bb8a53
update tonic 0.8 & prost 0.11 ( #1285 )
...
* update tonic 0.8 & prost 0.11
* install protoc via package manager
2022-12-19 15:30:33 +01:00
Arnaud Gourlay
d77debf122
Clippy for Rust 1.66 ( #1284 )
2022-12-16 10:53:51 +01:00
Arnaud Gourlay
4b482e7630
do not block grpc async threads ( #1252 )
2022-12-02 11:59:39 +01:00
Andrey Vasnetsov
04434f02b6
Update points by filter ( #1249 )
...
* allow to set remove payload by selector
* fmt
2022-12-01 09:16:07 +01:00
Andrey Vasnetsov
9f36ac60ee
Full payload update ( #1245 )
...
* implement api to fully overwrite payload of the point
* fmt
2022-11-28 13:46:58 +01:00
Andrey Vasnetsov
38444d2cc7
Fix consensus freezes ( #1241 )
...
* do not await for replica deactivation if there is a logical error
* use async mutex in raft service + integration test for deadlock
* fmt
2022-11-24 15:36:03 +01:00
Andrey Vasnetsov
803ad06abe
Extend telemetry ( #1230 )
...
* WIP: extend telemetry api for faster cluster manager
* fmt
* clippy fix
2022-11-21 10:26:54 +01:00
Arnaud Gourlay
6266d9bcdc
enable compression for gRPC ( #1186 )
...
* enable compression for gRPC
* add benchmarks
* fix grpc benchmark
* discardResponseBodies to reduce memory usage
* improve & split benchmarks
* do not use gzip for inter node communication as benchmarks are not conclusive
2022-11-18 13:34:49 +01:00
Andrey Vasnetsov and Arnaud Gourlay
772326f848
Recover distributed collection ( #1214 )
...
* WIP: shards recovering API: download snapshot
* fmt
* snapshot recovery API
* fmt
* snapshot recovery integration test + fixes
* review fixes
* review fixes 2
* Update lib/storage/src/content_manager/snapshots/recover.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com >
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com >
2022-11-16 10:55:23 +01:00
Andrey Vasnetsov
dad331fc26
fix colletion params update ( #1208 )
2022-11-09 13:10:54 +01:00
98fc5b2fa4
0.11.1 cherry picks ( #1193 )
...
* Complete feature road-map
* docs: add erare-humanum as a contributor for code (#1181 )
* docs: update README.md [skip ci]
* docs: update .all-contributorsrc [skip ci]
Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
* update rust version and fix clippy complains (#1190 )
* handle collections transition from single-node to cluster (#1189 )
* Update to v0.11.1 (#1191 )
* update to v0.11.1
* also default version
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
* consensus_state to consensus_state
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>
2022-11-07 11:31:12 +01:00
Arnaud Gourlay
87b820894c
enable HTTP compression middleware ( #1184 )
2022-11-01 11:10:46 +01:00
Andrey Vasnetsov
a0b6fa81b7
Consensus q n a ( #1169 )
...
* Expand comments and minor refactoring for raft state
* fmt
* add comments to consensus.rs
2022-10-30 20:34:09 +01:00
286ca9f5e4
0.11.0 cherry picks ( #1174 )
...
* implement concern-factor, forbig disabling the last node, improve tes… (#1168 )
* implement concern-factor, forbig disabling the last node, improve test stability
* upd docs
* improve test_recover_dead_node
* rename to write_consistency_factor
* Fix bug: applied index higher than commit on restart (#1172 )
* create collection with default status = dead + await for activation on create_collecton_op_submit
* fmt
* fix unit tests
Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com >
* upd integration tests
* fix transaction handling
* upd protobuf
* upd to v0.11.0
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
Co-authored-by: Egor Ivkov <e.o.ivkov@gmail.com >
2022-10-27 10:16:21 +02:00
Ivan Pleshkov and Andrey Vasnetsov
516dcd7020
Telemetry level of detail ( #1049 )
...
* telemetry level of detail
* rename duration aggregator
* are you happy fmt
* move total searches sum
* separate levels
* optional bucket size
* search telemetry improvements
* separate web telemetry into methods
* tonic telemetry methods
* merge optimizations
* are you happy fmt
* better rounding
* qdrant configs on level 1
* provide collection params
* add peers count
* collection points count
* update openapi
* use pattern in actix telemetry
* are you happy fmt
* merge dev
* are you happy fmt
* fix merge conflicts
* update openapi
* fix build
* are you happy fmt
* add exact searches statistics
* process replica set
* update openapi
* fix wrong name
* fix naming
* fix unwrap
* review
* fmt
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
2022-10-23 00:48:55 +02:00
Andrey Vasnetsov
387b52d7fa
propagate peer url to consensus if specified uri is does not match a … ( #1156 )
...
* propagate peer url to consensus if specified uri is does not match a stored one + tests
* review fixes
2022-10-23 00:07:39 +02:00
Egor Ivkov
a96f92cbfe
Record message send failures and stop consensus on service error ( #1152 )
...
* Record message send failures and stop consensus on service error
* Generate openapi
* Review fixes: log first error and remove on success
2022-10-22 23:23:37 +02:00
Andrey Vasnetsov
658b185b24
exclude / from actix logger middleware ( #1157 )
2022-10-21 09:28:22 +02:00
Andrey Vasnetsov
75563b20b4
Consensus fault tollerance improvements ( #1149 )
...
* consensus fault tollerance improvements
* fmt
2022-10-20 11:22:22 +02:00
Andrey Vasnetsov
123ae1607d
Await for collection created for sure ( #1144 )
...
* run plunger operation after user request to ensure that the majority of the consensus actually applied previous operation
* fmt
* fix test
* ensure all operations are unique in consensus queue
* add extra timeout for CI tests
* fix timout param in delete peer + do not explicilty await on peer removal
* fmt
2022-10-19 17:59:16 +02:00
Egor Ivkov
d3849d712b
Allow peer force removal ( #1139 )
...
* Allow peer force removal
* Remove shards
2022-10-19 17:01:13 +02:00
Andrey Vasnetsov
7f8c088711
Replication transfer fixes ( #1142 )
...
* fix validation and task assignment during replica transfer
* fmt
* upd openapi
* extend integration test
2022-10-19 11:38:26 +02:00
f50f1f4c71
#1119 disable updating feature ( #1126 )
...
* disable updating feature
* customize error message
* update namings
* Update src/actix/api/service_api.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com >
* fix build
* update api
* update api and allow deletion
* update openapi
* replace Bad Request with Forbidden
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com >
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
2022-10-18 12:58:38 +02:00
Arnaud Gourlay and Andrey Vasnetsov
9324be1d60
[Replication] Add replicas ( #1085 )
...
* Add replicas to ReplicaSet
* unproxify shard & miscs
* no exlusive write locking while adding replicas
* on_optimizer_config_update on Shard with async. recursion
* no exlusive write locking while removing replicas
* shortcut replica set propagation #1101
* remove unused field
* make RequestShardTransfer callback sync.
* promote local & remote to replica state
* fixes for replica sync api (#1123 )
* code review
* fix replica set update - fail only if all failed
* Add replica redesign (#1131 )
* refactor shard/mod.rs
* wip
* fmt
* it compiles
* temporary disable replica placemeant change on replication factor
* fmt
* finish todos
* small refactoring
* remove change::add
* replica-set -> shard-replica-set
* fmt
* upd openapi
* fix finish transfer logic
* fix existing integration tests
* shard transfer validation
* fmt
* review fixes
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com >
2022-10-17 20:12:10 +02:00
Andrey Vasnetsov
b275d452a8
shard replica manipulation API ( #1121 )
2022-10-12 15:19:43 +02:00
Egor Ivkov
03ec804bef
Adds panic hook ( #1106 )
2022-10-07 19:17:08 +03:00
Andrey Vasnetsov