* feat: `io_uring` setting to control which components use the io_uring backend
A few components have both an mmap and an io_uring variant reading the very
same files: the immutable dense vector storages, the single-file TurboQuant
storage, and the mmap payload storage. Until now the choice was a side effect
of `async_scorer` — a vector-search knob — plus, for the payload storage, a
feature flag that was parked off because io_uring is ~2x slower than mmap when
the data fits the page cache (#9310, #9409).
Add `storage.performance.io_uring`, optional, with two modes:
- unset (default): unchanged behaviour. The vector storages keep following
`async_scorer`; the payload storage stays on mmap.
- `disabled`: no component uses io_uring.
- `auto`: a component uses io_uring when its memory placement is `cold` (data
is left on disk, so reads hit the disk and there is something to gain), its
feature flag allows it, and the kernel supports io_uring. Components meant to
sit in RAM keep using mmap.
The decision lives in one place, `segment::common::io_uring::use_io_uring`, so
the openers no longer each reach for the async-scorer global. Kernel support is
now probed up front through `is_io_uring_supported()` instead of opening a file
and falling back on error.
`async_payload_storage` now defaults to on: it no longer decides anything by
itself, it only lifts the ban, and the payload storage no longer follows
`async_scorer` at all — so turning it on cannot silently move an existing
`async_scorer: true` deployment onto the slower path.
Which backend a component ended up on depends on the config, the placement and
the kernel at once, so report it in `SegmentInfo`: `vector_data[name].io_backend`
and `payload_storage_io_backend`, both `"mmap" | "io_uring"`, absent for
components that have no such choice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Trim comments, drop trivial tests
Two tests were only restating their own implementation: `test_mode_round_trip`
round-tripped the encode/decode pair next to it, and `test_io_uring_config`
checked that serde deserializes a two-variant enum. The mode matrix test stays,
it is the one that pins the semantics.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Flatten `IoBackend` in OpenAPI, derive `JsonSchema` for `IoUringMode`
Per-variant doc comments on a plain string enum make schemars emit a `oneOf`
of anonymous single-value objects instead of a flat `enum`. Move the variant
descriptions into the enum doc, as `Memory` and friends already do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Update lib/segment/src/vector_storage/turbo/turbo_vector_storage.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Update lib/segment/src/types.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Update lib/segment/src/types.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Update lib/segment/src/types.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Update lib/segment/src/types.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* upd openapi schema
* Update lib/common/common/src/flags.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Require kernel io_uring support in the async-scorer fallback
`use_io_uring` returned `get_async_scorer()` verbatim when the `io_uring`
setting is unset, so an enabled async scorer on a kernel without io_uring
opened the io_uring storage, failed, and fell back to mmap with an error
log per segment. Gate that branch on `is_io_uring_supported()` too, like
`Auto` already is, so the component just stays on mmap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* upd openapi schema
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Support memory placement in service-level storage config defaults
Follow-up to #9684: `storage.payload.memory` and
`storage.collection.vectors.memory` set service-wide placement defaults
for newly created collections, deprecating `storage.on_disk_payload` and
`storage.collection.vectors.on_disk`.
Defaults resolve as: request `memory` > request legacy flag > service
`memory` > service legacy flag; exactly one level is filled to avoid
spurious memory-vs-legacy mismatch warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Mark hnsw_index.on_disk deprecated in config.yaml, document memory option
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* [ai] Add config boolean to disable URL based snapshot restore
* Merge if-statement
* Comment-out config option by default
* [ai] Also block partial snapshots from remote URLs
* [ai] Only run clock consistency test when staging feature is present
* [ai] Add integration test
* initial implementation
* internal logging type
* wrap more places into auth function
* [manual refactor] use &Auth in .toc fucntion - minimise direct .access()
* [manual refactor] fmt
* [manual refactor] remove unannotated .access in creation of snapshots
* [manual refactor] remove unannotated .access in cluster telemetry
* [manual refactor] remove unannotated .access
* [manual refactor] do not log /metrics api access
* [manual refactor] do not run the service if audit logging failed to init
* [manual refactor] make auditable operation names for point and cluster updates
* [manual refactor] remove excess cloning of auth object
* [manual refactor] fmt
* [AI] instead of manual writing into the file, use tracing_appender crate
* [AI] simplify configuration to match internal crate
* fmt
* [manual refactor] cover staging options
* [AI] refactor x-forwarded-for handelling
* [manual refactor] use consts for x-forwarded-for + recover handelling for RFC case in tonic
* Update src/tonic/forwarded.rs
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* [manual] review fix: use FORWARDED in actix
* [manual] review fix: do not log strict mode checks
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* remove default explicit config for strict mode
* Don't prefix commented out properties with space to remain consistent
* Don't crash test if strict mode config is not set at all
---------
Co-authored-by: timvisee <tim@visee.me>
* Deprecate init_from
* Mark memmap_threshold as deprecated
* Mark locks API as deprecated
* Mark RBAC collection access payload filter as deprecated
* Allow deprecations in tests and benchmarks
* Add magic all flag, if set all features will be enabled
* Report enabled feature flags on start, if any is enabled
* Remove pub visibility from all, add comment it's only applied on init
* Don't check all flag in is_empty
* Rename mutable ID tracker mappings file
* Make new ID tracker the default, add new variant for RocksDB tracker
* In mutable ID tracker, only list files if they exist on disk
* Don't use the new mutable ID tracker yet
* Feature flag usage of new mutable ID tracker
* Simplify RocksDB check a bit
* Rename both the mutable ID tracker files
* Add global feature flags structure, load on start
* Feature flag use of new shard key format
* Enable feature flag for new shard key format by default in dev builds
* Prevent unwrapping everywhere; Single place for initialization (#6198)
* Disable shard key feature flag by default for now, it causes problems
---------
Co-authored-by: Jojii <15957865+JojiiOfficial@users.noreply.github.com>
* Add `resharding_enabled` option to `ClusterConfig`
If `resharding_enabled` is *not* set, `StartResharding` API returns an error
* Enable `resharding_enabled` option in development config
* fixup! Add `resharding_enabled` option to `ClusterConfig`
Fix tests
* fixup! Enable `resharding_enabled` option in development config
😅
* fixup! Enable `resharding_enabled` option in development config
🙄🙄🙄
* copy-paste blob_storage
* get rid of copied utils
* fix clippy
* fmt
* audit blob_store manifest
* move `Payload` into `fixtures.rs`
* fix clippy
* put csv in google storage
* ignore hm test
* fix codespell
* improve files test
* test different block sizes
* remove unused deps
* replace StdRng with SmallRng for faster tests
* make the default `on_disk_payload` match the default config value
* remove duplicate impl, update docstrings
* change default `PayloadStorageType` too
* gen openapi
* add HwMeasurementAcc and API helpers+types
* add config to API to toggle hardware reporting in API responses
* add Default impl for HwMeasurementAcc
* comments
* review remarks
* include usage into API schema
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Adding logging to disk note to config.yaml.
* Update config/config.yaml
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add SnapshotsStorageConfig enum(Local or S3) and deserialize implementation
* [refactor] use snapshots_config instead of s3_config
* update config
* add AWS official`aws-sdk-s3`
* implement store_file() WITHOUT error handling
* implement list_snapshots
* implement delete_snapshot
* run `cargo +nightly fmt`
* delete println
* implement get_stored_file
* Add error handlings
* Refactor AWS S3 configuration and error handling
* fix bugs
* create an empty test file
* fix `alias_test.rs` for StorageConfig type
* tempolary delete some test and try s3 test
* Update integration-tests.yml to use snap instead of apt-get for installing yq
* Update integration-tests.yml to use sudo when installing yq
* add sudo
* make (full/non-full) snapshots downloadable
* debug
* small fix
* Add S3 endpoint URL configuration option
* fix
* fix
* debug
* fix endpoint
* update to http://127.0.0.1:9000/
* update
* fix
* fix `#[get("/collections/{collection}/shards/{shard}/snapshots/{snapshot}")]` for s3
* put original tests back
* refactor
* small fix (delete println & echo)
* use object_store and refactor
* create snapshot_storage_ops and implement
* Refactor get_appropriate_chunk_size function to adjust chunk size based on service limits and file size
* cargo +nightly fmt --all
* make it more abstract
* Refactor SnapshotsStorageConfig deserialization in SnapShotsConfig
* small update
* small fix
* Update dependencies in Cargo.lock
* Update minio image to satantime/minio-server
* Refactor snapshot storage paths in snapshots_manager.rs and snapshot_storage_ops.rs
* Fix issue with downloaded file size not matching expected size in download_snapshot function
* add flush
* Use Streaming instead of donloading once
* apply `cargo +nightly fmt --all`
* Fix issue with opening file in SnapshotStream::LocalFS variant
* Fix error handling in SnapshotStream::LocalFS variant
* Add integration test for Shard Snapshot API with S3 storage (#7)
* Add API key to HTTP client
* Add API key field to snapshot recovery requests
* Add API key to channel service
* Provide API key when doing snapshot transfer
* Configure API key header name constant in a central place
* Reformat
* Update OpenAPI spec
* Remove suffixed spaces from configuration file
* Allow to specify HTTP headers in some consensus test utility functions
* Add snapshot transfer test with configured API key
* Use random API key in test
* Fix compilation errors and clippy warnings
* Move peer metadata type around
* Expose peer metadata in channel service
* Use WAL delta transfer by default for recovery, if all nodes are 1.8+
* Add check for missing metadata, assume versionless if we have less
* Use user configured shard transfer method, fall back to WAL delta/stream
* Minor improvements
* Update variable name
* Move CPU count function to common, fix wrong CPU count in visited list
* Change default number of rayon threads to 8
* Use CPU budget and CPU permits for optimizer tasks to limit utilization
* Respect configured thread limits, use new sane defaults in config
* Fix spelling issues
* Fix test compilation error
* Improve breaking if there is no CPU budget
* Block optimizations until CPU budget, fix potentially getting stuck
Our optimization worker now blocks until CPU budget is available to
perform the task.
Fix potential issue where optimization worker could get stuck. This
would happen if no optimization task is started because there's no
available CPU budget. This ensures the worker is woken up again to
retry.
* Utilize n-1 CPUs with optimization tasks
* Better handle situations where CPU budget is drained
* Dynamically scale rayon CPU count based on CPU size
* Fix incorrect default for max_indexing_threads conversion
* Respect max_indexing_threads for collection
* Make max_indexing_threads optional, use none to set no limit
* Update property documentation and comments
* Property max_optimization_threads is per shard, not per collection
* If we reached shard optimization limit, skip further checks
* Add remaining TODOs
* Fix spelling mistake
* Align gRPC comment blocks
* Fix compilation errors since last rebase
* Make tests aware of CPU budget
* Use new CPU budget calculation function everywhere
* Make CPU budget configurable in settings, move static budget to common
* Do not use static CPU budget, instance it and pass it through
* Update CPU budget description
* Move heuristic into defaults
* Fix spelling issues
* Move cpu_budget property to a better place
* Move some things around
* Minor review improvements
* Use range match statement for CPU count heuristics
* Systems with 1 or 2 CPUs do not keep cores unallocated by default
* Fix compilation errors since last rebase
* Update lib/segment/src/types.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/storage/src/content_manager/toc/transfer.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Rename cpu_budget to optimizer_cpu_budget
* Update OpenAPI specification
* Require at least half of the desired CPUs for optimizers
This prevents running optimizations with just one CPU, which could be
very slow.
* Don't use wildcard in CPU heuristic match statements
* Rename cpu_budget setting to optimizer_cpu_budget
* Update CPU budget comments
* Spell acquire correctly
* Change if-else into match
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Rename max_rayon_threads to num_rayon_threads, add explanation
* Explain limit in update handler
* Remove numbers for automatic selection of indexing threads
* Inline max_workers variable
* Remove CPU budget from ShardTransferConsensus trait, it is in collection
* small allow(dead_code) => cfg(test)
* Remove now obsolete lazy_static
* Fix incorrect CPU calculation in CPU saturation test
* Make waiting for CPU budget async, don't block current thread
* Prevent deadlock on optimizer signal channel
Do not block the optimization worker task anymore to wait for CPU budget
to be available. That prevents our optimizer signal channel from being
drained, blocking incoming updates because the cannot send another
optimizer signal. Now, prevent blocking this task all together and
retrigger the optimizers separately when CPU budget is available again.
* Fix incorrect CPU calculation in optimization cancel test
* Rename CPU budget wait function to notify
* Detach API changes from CPU saturation internals
This allows us to merge into a patch version of Qdrant. We can
reintroduce the API changes in the upcoming minor release to make all of
it fully functional.
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Do not select ourselves when finding auto shard transfer source
* Limit automatic shard transfers with basic transfer count check
* Extract shard recovery transfer request logic to separate function
* Add global shard transfer tracker structure to allow rate limiting
* Count and limit incoming and outgoing shard transfers separately
* Make automatic shard transfer limit configurable
* Move shard transfer tracker from global to collection level
* Comment out new config parameters
* Fix incorrect comment
* Fix missing space in log message
* Fix negated condition
* Remove logic for requesting shard transfer on replica state change
* Check shard transfer limits in consensus sync, use consensus state
Instead of bothering with tracking proposed shard transfers, this now
purely relies on state that is already in consensus. Each time we sync
consensus, we request the maximum number of shard transfers up to the
limit at that time.
* Remove now obsolete shard transfer tracker
* Revert now obsolete changes
* Improve transfer IO counting comment, revert now obsolete code
* Fix typos
* Fix flipped variables, don't take self reference on Copyable type
* Update lib/collection/src/collection/shard_transfer.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Handle incoming/outgoing transfer counts separately, don't tuple it
* Improve loop for counting incoming/outgoing transfers
* Remove unused test function
* Add consensus tests for automatic shard transfer limits
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Remove debug lines from new test
* Reorder last test a bit to resolve flakyness
* We can have one more transfer for recovery on the other alive node
* Attempt to reduce test flakyness, more points and more frequent polling
* Update config/config.yaml
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Explicitly note default shard transfer limit in configuration
* Use default for shard transfer IO everywhere
* Rename transfer limit check function to be more explicit
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Add config option to set default automatic shard transfer method
* Also use configured shard transfer method if user doesn't specify
This is not for automatic shard transfers, but when a user initiates a
transfer manually.
* Improve comment for shard transfer method configuration
* Fix test compilation
* Read-only API keys
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Correct placement of OpenAPI security
Place regex dep with actix/tonic
* Read-only API keys
* Replace with pytests
* API Key tests run on the same job
* Drop allow dead-code
* Rename setting key
* Containerized tests
* No special config files
* DRY
* refactor: re-use can_write method
* refactor: replace static by constants
* refactor: get PID from `$!`
* refactor: use explicit brackets on boolean condition
* style: fix identation
* small fixes + account for new APIs
* specify security in openapi
* small fix + chmod for .sh testfile
* add best-efford check for api consistency
---------
Co-authored-by: Amr Hassan <amr.hassan@gmail.com>
Co-authored-by: generall <andrey@vasnetsov.com>
