* Move `DenseVector`/`MultiDenseVector` from `api` to `segment`
* Move `OrderByInterface` from `api` to `segment`
Reason: it's used in `edge` which shouldn't depend on `api`.
* Make `shard` -> `api` dependency optional
* Remove `api` from the amalgamation
* Don't install protoc in edge Actions
* [manual] generate snapshot download link with respect of enabled tls
* Simplify http(s) selection
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* download tar
* compute sha256 for stream download
* wip: propagate unpacking into down to the logic, todo: validation
* unpacked snapshot validation
* Minor tweaks
* Fix typo
* validation during unpack
* cancellation token
* update docstring
* remove redundant dep
* Rearrange unpack functions
- Rename `safe_unpack.rs` into `tar_unpack.rs` so it would be listed
near `tar_ext.rs` in IDEs.
- Replace calls like `ar = open_snapshot_archive(…); safe_unpack(ar, …);`
with a single call to `tar_unpack_file(…)`.
- Put calls to `Archive::new(); Archive::set_overwrite(false);` inside
`tar_unpack_reader` (was `safe_unpack`). So, now it is the only place
that does `set_overwrite`.
* Let clippy complain if tar::Archive::unpack used
* Mock snapshot download URL
Instead of downloading from storage.googleapis.com every time the test
runs, put small snapshot file to the repo.
The snapshot file is created using this command:
curl -s \
https://storage.googleapis.com/qdrant-benchmark-snapshots/test-shard.snapshot \
| tar \
--delete segments/4ea958d8-0b64-4312-9a53-0cd857e93535.tar \
--delete segments/65ac6276-8cca-4f5c-b767-9722190cee8b.tar \
> lib/storage/src/content_manager/snapshots/test-shard.snapshot
File contents:
$ tar tf lib/storage/src/content_manager/snapshots/test-shard.snapshot
wal/
wal/closed-255
newest_clocks.json
replica_state.json
shard_config.json
$ du -sh lib/storage/src/content_manager/snapshots/test-shard.snapshot
12K lib/storage/src/content_manager/snapshots/test-shard.snapshot
$ sha256sum < lib/storage/src/content_manager/snapshots/test-shard.snapshot
5d94eac5c1ede3994a28bc406120046c37370d5d45b489a0d2252531b4e3e1f2 -
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* sync function to ask to stop workers
fmt
update blocking_ask_workers_to_stop
* graceful shard stop on ShardReplicaSet changes
* make stop consuming
* implement graceful stop for replica set
* graceful shard stop in start_resharding_unchecked
* stop_gracefully fn on shard_holder
* collection graceful stop on delete
* explain when it is ok to shutdown
* fix tests and ensure blocking callsa are not called from async runtime
* fmt
* Annotate cancel safety for do_recover_from_snapshot function
* Minor comment nit
* make drop not wait on stop thread
* graceful stop in more tests
* graceful stop in more tests
* avoid background thread is shard was stopped gracefully
* annotate cancel safety
* fmt
* More cancel safety annotations
* Add two spawns to ensure cancel safety
---------
Co-authored-by: timvisee <tim@visee.me>
* Fix non visible points during snapshots
* Update lib/shard/src/segment_holder/mod.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* code review from mr.rabbit
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Retain more closed segments
* Add back rocksdb
* Update WalOptions/WalConfig across the code
* Use NonZeroUsize
* Expose via APIs
* Update gRPC docs
* Fix stoarge compat test
* recompile openapi.json with rocksdb
* default wal retain closed fn
* update openapi.json
* Use qdrant/wal latest commit and remove from config.yaml
* Deprecate init_from
* Mark memmap_threshold as deprecated
* Mark locks API as deprecated
* Mark RBAC collection access payload filter as deprecated
* Allow deprecations in tests and benchmarks
* deprecate inserting order_value in payload, use the one in the struct instead
* fix for test
* dedup by (order_value, id) tuple
* Scroll dedup test must also consider order value
* Update assert message, include order value
---------
Co-authored-by: timvisee <tim@visee.me>
* Propagate shard key mapping wrapper deeper
* Also bump reverse shard mapping when applying shard keys from snapshot
* Also apply new shard key mappings to existing replicas
* Apply shard key to replica set through apply_state directly
* Set both new shard key mappings through utility function
* Set shard key mappings directly during directory creation
* Rename get_key to key
* Update comment
* Measure update operations hardware IO
* Add support for distributed setups
* also measure update_local
* Add consensus tests for HW metrics of update operations
* add test for upserting without waiting
* Disable HW usage reporting when not waiting for update API
* Review remarks
* Fix resharding collecting hw measurements
* Fix metric type
* New struct HardwareData for better accumulation
* Ensure we always apply CPU multiplier
* Apply suggestions from code review
* Update src/actix/api/update_api.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Fix assert_with_upper_bound_error threshold calculation.
* Clarifying why we don't measure shard cleanup
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Bump Rust edition to 2024
* gen is a reserved keyword now
* Remove ref mut on references
* Mark extern C as unsafe
* Wrap unsafe function bodies in unsafe block
* Geo hash implements Copy, don't reference but pass by value instead
* Replace secluded self import with parent
* Update execute_cluster_read_operation with new match semantics
* Fix lifetime issue
* Replace map_or with is_none_or
* set_var is unsafe now
* Reformat
* rename cpu_budget -> resource_budget
* clippy
* add io budget to resources
* fmt
* move budget structures into a separate file
* add extend permit function
* dont extend existing permit
* switch from IO to CPU permit
* do not release resource before aquiring an extension
* fmt
* Review remarks
* Improve resource permit number assertion
* Make resource permit replace_with only acquire extra needed permits
* Remove obsolete drop implementation
* allocate IO budget same as CPU
* review fixes
---------
Co-authored-by: timvisee <tim@visee.me>
* bump and migrate to rand 0.9.0
also bump rand_distr to 0.5.0 to match it
* Migrate AVX2 and SSE implementations
* Remove unused thread_rng placeholders
* More random migrations
* Migrate GPU tests
* bump seed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Measure read io for payload storage
* Add Hardware Counter to update functions
* Fix tests and benches
* Rename (some) *_measured functions back to original
* Add UUID to collection...
...and recreate collection, when applying Raft snapshot, if UUID of collection is different
* fixup! Add UUID to collection...
Remove UUID field from gRPC and exclude it from OpenAPI spec 🤡
* fixup! fixup! Add UUID to collection...
Always generate collection UUID 🤦♀️
* Raft snapshot recreate collection no expose UUID (#5452)
* separate colleciton config structure from API
* fmt
* Update lib/collection/src/operations/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Send current replica state with disable replica proposals
* Don't be strict about from state if in shard transfer related state
* Link to PR, update formatting
* Fix typo
* Fix test compilation failures
* Provide peer state when deactivating leader, remove unnecessary TODO
* ReplicaState is Copy
* include document & image objects into grpc API
* introduce image and object to rest api
* minor refactoring
* rename Vector -> VectorInternal
* decompose vector data structures
* add schema
* fmt
* grpc docs
* fix conversion
* fix clippy
* fix another conversion
* rename VectorInput -> VectorInputInternal
* replace grpc TryFrom with async functions
* fmt
* replace rest TryFrom with async functions
* add image and object into query rest
* separate inference related conversions
* move json-related conversions into a separate file
* move vector-related transformations into a separate file
* move more vector related-conversions into dedicated module
* Add and fix missing timeout in the retrieve APIs
* no need for extra timeout handling as single ops are doing it
* don't surface timeout too high in shard transfer infra
* add todos for future work
* update timeout after first query
* Distance matrix API internals
* minor code review
* use new Query sample instead of manual samplig + retrieve
* do not crash if the point does not contain the specific vector
* Use iterative approach in the absence of HasVector condition
* add tests
* drop some code
* Drop JsonPathString
* Fix test_remove_key
Drop failing tests:
- Deleting array indices is not idempotent, so we don't support it.
- Empty JSONPath is not supported.
* Make json_path::path() non-generic
* Remove references to JsonPathV2
* Drop JsonPathInterface
* Move json_path::v2 code into json_path
* Drop validate_not_empty
* Drop JsonPath::head() as being unused
* Replace path() with JsonPath::new()
* Restore comments
* Move tests to json_path
* Use json() consistently in tests
* Replace many into calls with Into trait
---------
Co-authored-by: timvisee <tim@visee.me>
* minor conversion improvement
* use NamedVectors in update_vectors
* remove merge from VectorStruct
* rename Multi -> Named in vector struct
* add multi-dense vectors option into VectorStruct
* generate openapi
* rename VectorStruct -> VectorStructInternal
* add conversion for anonymous multivec in grpc
* renames for BatchVectorStruct
* implement multi-dense for batch
* allow multi-dense in batch upserts
* test and fixes
* universal-query: add with_lookup to query API definition
* fix consistency doc check
* lookup_from and not lookup_with
* clean import
* improve doc for lookup location wrt using
* Suggestions for descriptions
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
