* Adding logging to disk note to config.yaml.
* Update config/config.yaml
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add SnapshotsStorageConfig enum(Local or S3) and deserialize implementation
* [refactor] use snapshots_config instead of s3_config
* update config
* add AWS official`aws-sdk-s3`
* implement store_file() WITHOUT error handling
* implement list_snapshots
* implement delete_snapshot
* run `cargo +nightly fmt`
* delete println
* implement get_stored_file
* Add error handlings
* Refactor AWS S3 configuration and error handling
* fix bugs
* create an empty test file
* fix `alias_test.rs` for StorageConfig type
* tempolary delete some test and try s3 test
* Update integration-tests.yml to use snap instead of apt-get for installing yq
* Update integration-tests.yml to use sudo when installing yq
* add sudo
* make (full/non-full) snapshots downloadable
* debug
* small fix
* Add S3 endpoint URL configuration option
* fix
* fix
* debug
* fix endpoint
* update to http://127.0.0.1:9000/
* update
* fix
* fix `#[get("/collections/{collection}/shards/{shard}/snapshots/{snapshot}")]` for s3
* put original tests back
* refactor
* small fix (delete println & echo)
* use object_store and refactor
* create snapshot_storage_ops and implement
* Refactor get_appropriate_chunk_size function to adjust chunk size based on service limits and file size
* cargo +nightly fmt --all
* make it more abstract
* Refactor SnapshotsStorageConfig deserialization in SnapShotsConfig
* small update
* small fix
* Update dependencies in Cargo.lock
* Update minio image to satantime/minio-server
* Refactor snapshot storage paths in snapshots_manager.rs and snapshot_storage_ops.rs
* Fix issue with downloaded file size not matching expected size in download_snapshot function
* add flush
* Use Streaming instead of donloading once
* apply `cargo +nightly fmt --all`
* Fix issue with opening file in SnapshotStream::LocalFS variant
* Fix error handling in SnapshotStream::LocalFS variant
* Add integration test for Shard Snapshot API with S3 storage (#7)
* Add API key to HTTP client
* Add API key field to snapshot recovery requests
* Add API key to channel service
* Provide API key when doing snapshot transfer
* Configure API key header name constant in a central place
* Reformat
* Update OpenAPI spec
* Remove suffixed spaces from configuration file
* Allow to specify HTTP headers in some consensus test utility functions
* Add snapshot transfer test with configured API key
* Use random API key in test
* Fix compilation errors and clippy warnings
* Move peer metadata type around
* Expose peer metadata in channel service
* Use WAL delta transfer by default for recovery, if all nodes are 1.8+
* Add check for missing metadata, assume versionless if we have less
* Use user configured shard transfer method, fall back to WAL delta/stream
* Minor improvements
* Update variable name
* Move CPU count function to common, fix wrong CPU count in visited list
* Change default number of rayon threads to 8
* Use CPU budget and CPU permits for optimizer tasks to limit utilization
* Respect configured thread limits, use new sane defaults in config
* Fix spelling issues
* Fix test compilation error
* Improve breaking if there is no CPU budget
* Block optimizations until CPU budget, fix potentially getting stuck
Our optimization worker now blocks until CPU budget is available to
perform the task.
Fix potential issue where optimization worker could get stuck. This
would happen if no optimization task is started because there's no
available CPU budget. This ensures the worker is woken up again to
retry.
* Utilize n-1 CPUs with optimization tasks
* Better handle situations where CPU budget is drained
* Dynamically scale rayon CPU count based on CPU size
* Fix incorrect default for max_indexing_threads conversion
* Respect max_indexing_threads for collection
* Make max_indexing_threads optional, use none to set no limit
* Update property documentation and comments
* Property max_optimization_threads is per shard, not per collection
* If we reached shard optimization limit, skip further checks
* Add remaining TODOs
* Fix spelling mistake
* Align gRPC comment blocks
* Fix compilation errors since last rebase
* Make tests aware of CPU budget
* Use new CPU budget calculation function everywhere
* Make CPU budget configurable in settings, move static budget to common
* Do not use static CPU budget, instance it and pass it through
* Update CPU budget description
* Move heuristic into defaults
* Fix spelling issues
* Move cpu_budget property to a better place
* Move some things around
* Minor review improvements
* Use range match statement for CPU count heuristics
* Systems with 1 or 2 CPUs do not keep cores unallocated by default
* Fix compilation errors since last rebase
* Update lib/segment/src/types.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/storage/src/content_manager/toc/transfer.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Rename cpu_budget to optimizer_cpu_budget
* Update OpenAPI specification
* Require at least half of the desired CPUs for optimizers
This prevents running optimizations with just one CPU, which could be
very slow.
* Don't use wildcard in CPU heuristic match statements
* Rename cpu_budget setting to optimizer_cpu_budget
* Update CPU budget comments
* Spell acquire correctly
* Change if-else into match
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Rename max_rayon_threads to num_rayon_threads, add explanation
* Explain limit in update handler
* Remove numbers for automatic selection of indexing threads
* Inline max_workers variable
* Remove CPU budget from ShardTransferConsensus trait, it is in collection
* small allow(dead_code) => cfg(test)
* Remove now obsolete lazy_static
* Fix incorrect CPU calculation in CPU saturation test
* Make waiting for CPU budget async, don't block current thread
* Prevent deadlock on optimizer signal channel
Do not block the optimization worker task anymore to wait for CPU budget
to be available. That prevents our optimizer signal channel from being
drained, blocking incoming updates because the cannot send another
optimizer signal. Now, prevent blocking this task all together and
retrigger the optimizers separately when CPU budget is available again.
* Fix incorrect CPU calculation in optimization cancel test
* Rename CPU budget wait function to notify
* Detach API changes from CPU saturation internals
This allows us to merge into a patch version of Qdrant. We can
reintroduce the API changes in the upcoming minor release to make all of
it fully functional.
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Do not select ourselves when finding auto shard transfer source
* Limit automatic shard transfers with basic transfer count check
* Extract shard recovery transfer request logic to separate function
* Add global shard transfer tracker structure to allow rate limiting
* Count and limit incoming and outgoing shard transfers separately
* Make automatic shard transfer limit configurable
* Move shard transfer tracker from global to collection level
* Comment out new config parameters
* Fix incorrect comment
* Fix missing space in log message
* Fix negated condition
* Remove logic for requesting shard transfer on replica state change
* Check shard transfer limits in consensus sync, use consensus state
Instead of bothering with tracking proposed shard transfers, this now
purely relies on state that is already in consensus. Each time we sync
consensus, we request the maximum number of shard transfers up to the
limit at that time.
* Remove now obsolete shard transfer tracker
* Revert now obsolete changes
* Improve transfer IO counting comment, revert now obsolete code
* Fix typos
* Fix flipped variables, don't take self reference on Copyable type
* Update lib/collection/src/collection/shard_transfer.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Handle incoming/outgoing transfer counts separately, don't tuple it
* Improve loop for counting incoming/outgoing transfers
* Remove unused test function
* Add consensus tests for automatic shard transfer limits
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Remove debug lines from new test
* Reorder last test a bit to resolve flakyness
* We can have one more transfer for recovery on the other alive node
* Attempt to reduce test flakyness, more points and more frequent polling
* Update config/config.yaml
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Explicitly note default shard transfer limit in configuration
* Use default for shard transfer IO everywhere
* Rename transfer limit check function to be more explicit
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Add config option to set default automatic shard transfer method
* Also use configured shard transfer method if user doesn't specify
This is not for automatic shard transfers, but when a user initiates a
transfer manually.
* Improve comment for shard transfer method configuration
* Fix test compilation
* Read-only API keys
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Correct placement of OpenAPI security
Place regex dep with actix/tonic
* Read-only API keys
* Replace with pytests
* API Key tests run on the same job
* Drop allow dead-code
* Rename setting key
* Containerized tests
* No special config files
* DRY
* refactor: re-use can_write method
* refactor: replace static by constants
* refactor: get PID from `$!`
* refactor: use explicit brackets on boolean condition
* style: fix identation
* small fixes + account for new APIs
* specify security in openapi
* small fix + chmod for .sh testfile
* add best-efford check for api consistency
---------
Co-authored-by: Amr Hassan <amr.hassan@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* add update_concurrency to control the parallelism of updating shards
* reformat the files
* reformat
* add update_concurrency control to shard updates
* use unwrap_or
* test using buffer_ordered
* use buffered (testing)
* Pre-allocate space for update futures
* use NonZeroUsize
* linter
* use NonZeroUsize in the test
* consensus test for update_concurrency
* update config commit
---------
Co-authored-by: Di Zhao <diz@twitter.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
* Issue 1905: Configurable location for the tmp snapshot files
* Apply suggestions from code review
Co-authored-by: Tim Visée <tim+github@visee.me>
* fix code review suggestions
* clippy fix
* Propagate temp path, use configured dir for snapshot creation
* Use real temp dir in snapshot tests
* Mention default temporary snapshot file path in configuration
* Use temp everywhere rather than a mix of temp and tmp
* Use consistent naming for temporary snapshot directories
* Extract logic for temporary storage path into toc method
* Resolve clippy warnings
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Migrate from OpenSSL to Rustls
* Add TTL based certificate rotation for Rustls in actix
See: <https://github.com/qdrant/qdrant/pull/1865#issuecomment-1539752859>
* Update last update time when certificate is replaced
* Update error message
* Add option to disable certificate TTL, add TTL validation
* Cleanup
* Update comment for service.enable_tls, also enables TLS for gRPC
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Move with_buf_read into certificate_helpers, it is used there
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Remove level of scoping
* Add TLS enabled/disabled log messages
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* feat: use 0 as disable for indexing and memmap
* amend: edit test, update doc comments, remove validation
* fix: use constant instead of literal, reword doc comments
* test: update test for loop and timeout
* fix: remove validation on OptimizerConfigDiff and fix test_validatioin.py
* fix: remove validation from build.rs, remove internal hack from doccomments
* fix: update grpc docs
* revert max_optimization_threads validation in test
* update config comments and minor test change
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* feat: embed default config file at compile time
* Add `test_no_config_files` and supporting `find_config_files` parameter
* Error if `RUN_MODE` set but can't find config files
* feat: warn if config files not found
* remove find_config_files flag
* Add test to deserialize just the embedded default config
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: timvisee <tim@visee.me>
* feat: built in api-key auth
* doc: Sending an api-key over an unencrypted channel is insecure.
* feat: constant time comparision for api-keys
* use `option_layer`
* Change file mode of some .sh files to make them executable
* Remove whitespace on empty lines from YAML configurations
* Fix unused import warning on Windows
* Add user client certificate validation capabiltiy
* Add integration test for client side certificate validation
* Add server side ca certificate to verify
* Use store builder for client side certificate
* Use a trust store
* Fix config for TLS test
* Fix test, remove mTLS for external grpc endpoint
* Fix config comment
* Remove useless commit
* Fix style
* Fix internodal TLS test
* Simplify setting SSL verify mode
---------
Co-authored-by: timvisee <tim@visee.me>
