* Introduce EdgeShardConfig for edge shard
- Add EdgeShardConfig and EdgeOptimizersConfig in lib/edge/src/config.rs
- Segment config (vector_data, sparse_vector_data, payload_storage_type)
- Global hnsw_config and per-vector HNSW in segment config
- Optimizer params: deleted_threshold, vacuum_min_vector_number,
default_segment_number, max_segment_size, indexing_threshold,
prevent_unoptimized (excludes memmap_threshold, flush_interval_sec,
max_optimization_threads)
- Persist/load as edge_config.json in shard path
- EdgeShard uses RwLock<EdgeShardConfig>; load() accepts Option<EdgeShardConfig>,
falls back to file or infer from segments; compatibility checked on load
- load_with_segment_config() for backward compatibility (SegmentConfig -> EdgeShardConfig)
- optimize() uses EdgeShardConfig for hnsw and optimizer thresholds
- Public methods: set_hnsw_config(), set_vector_hnsw_config(), set_optimizers_config()
(update and persist)
- Python and examples use load_with_segment_config with existing config API
Made-with: Cursor
* Refactor EdgeShardConfig: user-facing params only, config module
- Replace SegmentConfig inside EdgeShardConfig with user-facing fields:
- on_disk_payload (bool) instead of payload_storage_type
- vectors: HashMap<VectorNameBuf, EdgeVectorParams> with on_disk per vector,
no per-vector quantization; global quantization_config only
- sparse_vectors: HashMap<VectorNameBuf, EdgeSparseVectorParams> with on_disk
- EdgeVectorParams / EdgeSparseVectorParams use on_disk (bool) instead of
storage_type; conversion to VectorDataConfig/SparseVectorDataConfig in
to_segment_config()
- Add config module: mod.rs, optimizers.rs, vectors.rs, shard.rs
- from_segment_config(&SegmentConfig) fills all inferrable params
- to_segment_config() builds SegmentConfig for segments and optimize()
- load_with_segment_config takes Option<SegmentConfig>, uses from_segment_config
Made-with: Cursor
* Move optimizer threshold helpers to shard crate
- Add get_number_segments, get_indexing_threshold_kb, get_max_segment_size_kb,
get_deferred_points_threshold_bytes in shard::optimizers::config
- Collection OptimizersConfig and edge EdgeOptimizersConfig delegate to these
- Single place for threshold logic; collection and edge use shard helpers
Made-with: Cursor
* Use destructuring in config conversions to avoid missing new fields
- EdgeVectorParams: destructure VectorDataConfig in from_*, destructure self in to_vector_data_config
- EdgeSparseVectorParams: destructure SparseVectorDataConfig and SparseIndexConfig in from_*, destructure self in to_sparse_vector_data_config
- EdgeShardConfig: destructure SegmentConfig in from_segment_config, destructure self in to_segment_config
Adding new fields to source structs will now cause compile errors until conversions are updated.
Made-with: Cursor
* refactor: centralize on_disk_payload→payload_storage_type, on_disk→storage_type, and appendable quantization logic
- PayloadStorageType::from_on_disk_payload(bool) in segment (Mmap/InRamMmap)
- VectorStorageType::from_on_disk(bool) in segment (ChunkedMmap/InRamChunkedMmap)
- QuantizationConfig::for_appendable_segment(Option<&Self>) in segment (feature flag + supports_appendable)
- collection: use from_on_disk_payload in non-rocksdb branch
- edge shard/vectors: use new helpers; remove duplicated conditionals
- shard optimizers: use from_on_disk and for_appendable_segment
Made-with: Cursor
* refactor(edge): use EdgeShardConfig directly, drop segment_config
- Add plain_segment_config() for create_appendable_segment (no HNSW)
- Add segment_optimizer_config() built from EdgeShardConfig for blocking optimizers
- Add vector_data_config(name) for query/MMR
- build_blocking_optimizers: use segment_optimizer_config() instead of SegmentConfig
- create_appendable_segment: use plain_segment_config()
- search/query: use config().vectors and vector_data_config() instead of segment_config()
- Remove segment_config() from EdgeShardConfig and EdgeShard
- Add to_plain_vector_data_config on EdgeVectorParams
Made-with: Cursor
* [manual] review changes
* refactor(edge-py): wrap EdgeShardConfig, add EdgeVectorParams/EdgeSparseVectorParams
- PyEdgeConfig now wraps EdgeShardConfig (vectors, sparse_vectors, on_disk_payload, etc.)
- PyEdgeVectorParams / PyEdgeSparseVectorParams wrap edge config types
- PyEdgeOptimizersConfig for optional optimizer settings
- EdgeShard.load() uses EdgeShardConfig; edge::config made pub for Python crate
- cargo fmt + clippy (remove map_identity)
Made-with: Cursor
* refactor(edge-py): simplify config API, remove unused Py* types, add EdgeConfig
- Remove unused PyPayloadStorageType, PyVectorDataConfig, PyVectorStorageType,
PySparseVectorDataConfig, PySparseVectorStorageType from Python bindings
- Move PyEdgeOptimizersConfig to lib/edge/python/src/config/optimizers.rs
- Update qdrant_edge.pyi: EdgeConfig with vectors/sparse_vectors,
EdgeVectorParams, EdgeSparseVectorParams, EdgeOptimizersConfig
- Update examples (common.py, repr.py) to use new config API
- Run cargo fmt
Made-with: Cursor
* [manual] review changes
* [manual] review changes
* [manual] fix test
* Address CodeRabbit review comments for PR 8322 (#8324)
* Address CodeRabbit review comments for PR 8322
- Python examples: explicit imports (repr.py, common.py) and new EdgeConfig API
- HnswIndexConfig: add max_indexing_threads param and property in .pyi and Rust bindings
- EdgeConfig: make vectors optional for sparse-only configs; validate at least one of vectors/sparse_vectors
- EdgeShardConfig::load: use try_exists(), propagate I/O errors
- from_segment_config: infer hnsw_config from per-vector HNSW when all agree
- EdgeShard setters: atomic clone-mutate-save-then-replace; persist config save errors
- Segment compat: prefix vector name in error messages; resolve None datatype to Float32
- max_indexing_threads: preserve 0 (auto) sentinel in trait default; remove per-optimizer overrides
- SegmentOptimizerConfig:🆕 build plain and optimizer maps in single pass
- config_mismatch_optimizer tests: use VectorNameBuf::from() instead of .into()
- vectors.rs: doc updates for per-vector quantization
Made-with: Cursor
* Address @generall review: SaveOnDisk for config, resolve num_rayon_threads in optimizer
- Use SaveOnDisk<EdgeShardConfig> for EdgeShard config (generall: 'We have SaveOnDisk struct for this')
- Create via SaveOnDisk::new() after resolving config; setters use .write() for atomic persist
- set_vector_hnsw_config: clone then mutate then write (fallible setter)
- max_indexing_threads: resolve 0 (auto) via num_rayon_threads inside impl (generall: 'proper solution would be to resolve num_rayon_threads inside the optimizer impl')
- max_indexing_threads_sentinel_aware() now returns Some(num_rayon_threads(raw)) so callers get actual thread count
Made-with: Cursor
* [manual] reorganize num_rayon_threads -> get_num_indexing_threads to better account per-vector configuration
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* update docstring and pyi
* fmt
* fmt
* clipy
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* Move `DenseVector`/`MultiDenseVector` from `api` to `segment`
* Move `OrderByInterface` from `api` to `segment`
Reason: it's used in `edge` which shouldn't depend on `api`.
* Make `shard` -> `api` dependency optional
* Remove `api` from the amalgamation
* Don't install protoc in edge Actions
* [AI] Imeplement timeout for reading snapshot stream
* remove debug code
* [no-AI] fix error type and cancellation test(not the one which actually found the problem)
* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
* [manual] generate snapshot download link with respect of enabled tls
* Simplify http(s) selection
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* initial implementation
* internal logging type
* wrap more places into auth function
* [manual refactor] use &Auth in .toc fucntion - minimise direct .access()
* [manual refactor] fmt
* [manual refactor] remove unannotated .access in creation of snapshots
* [manual refactor] remove unannotated .access in cluster telemetry
* [manual refactor] remove unannotated .access
* [manual refactor] do not log /metrics api access
* [manual refactor] do not run the service if audit logging failed to init
* [manual refactor] make auditable operation names for point and cluster updates
* [manual refactor] remove excess cloning of auth object
* [manual refactor] fmt
* [AI] instead of manual writing into the file, use tracing_appender crate
* [AI] simplify configuration to match internal crate
* fmt
* [manual refactor] cover staging options
* [AI] refactor x-forwarded-for handelling
* [manual refactor] use consts for x-forwarded-for + recover handelling for RFC case in tonic
* Update src/tonic/forwarded.rs
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* [manual] review fix: use FORWARDED in actix
* [manual] review fix: do not log strict mode checks
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* download tar
* compute sha256 for stream download
* wip: propagate unpacking into down to the logic, todo: validation
* unpacked snapshot validation
* Minor tweaks
* Fix typo
* validation during unpack
* cancellation token
* update docstring
* remove redundant dep
* Rearrange unpack functions
- Rename `safe_unpack.rs` into `tar_unpack.rs` so it would be listed
near `tar_ext.rs` in IDEs.
- Replace calls like `ar = open_snapshot_archive(…); safe_unpack(ar, …);`
with a single call to `tar_unpack_file(…)`.
- Put calls to `Archive::new(); Archive::set_overwrite(false);` inside
`tar_unpack_reader` (was `safe_unpack`). So, now it is the only place
that does `set_overwrite`.
* Let clippy complain if tar::Archive::unpack used
* Mock snapshot download URL
Instead of downloading from storage.googleapis.com every time the test
runs, put small snapshot file to the repo.
The snapshot file is created using this command:
curl -s \
https://storage.googleapis.com/qdrant-benchmark-snapshots/test-shard.snapshot \
| tar \
--delete segments/4ea958d8-0b64-4312-9a53-0cd857e93535.tar \
--delete segments/65ac6276-8cca-4f5c-b767-9722190cee8b.tar \
> lib/storage/src/content_manager/snapshots/test-shard.snapshot
File contents:
$ tar tf lib/storage/src/content_manager/snapshots/test-shard.snapshot
wal/
wal/closed-255
newest_clocks.json
replica_state.json
shard_config.json
$ du -sh lib/storage/src/content_manager/snapshots/test-shard.snapshot
12K lib/storage/src/content_manager/snapshots/test-shard.snapshot
$ sha256sum < lib/storage/src/content_manager/snapshots/test-shard.snapshot
5d94eac5c1ede3994a28bc406120046c37370d5d45b489a0d2252531b4e3e1f2 -
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert
* add test
* upd dockstring
* require resharding once all peers have updated version
* use service error
* fix clippy again
* wait for same version before resharding in tests
* Remove call to Archive::set_sync
Also, this method was the last remaining part of our `tar-rs` fork,
so we can switch to the upstream version now.
* Do syncfs
* Handle operations when having no shards with custom sharding
* Return error instead of acknowledge
* fmt
* clippy
* shorter msg
* Return collection payload schema if exists even without any shards
* minor clippy fix
* Replace `Option<Segment>` with `enum LoadSegmentOutcome`
* Replace some Path/PathBuf with str/String
* Rename field Segment::{current_path -> segment_path}
* safe_delete
* Add a test for recovery after kill during Partial
* Address AI review
* simplify test
* Merge pull request #7829
* introduce a new state
* switch to ManualRecovery for user-initiated snapshot operations
* fmt
* fix test and regen api
* test fix: instead of force recovery on update failed snapshots now re…
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* easy fix with version check
* Link to PR
* Fix get transfers with source/target pairs, also fix dead replica aborts
* Use new source/target functions elsewhere
* Flip shard and peer arguments
* Fix incorrect check, from should be to
* Inline more format arguments
---------
Co-authored-by: timvisee <tim@visee.me>
* Initial plan
* Update timeout error handling to use milliseconds for better precision
Co-authored-by: agourlay <606963+agourlay@users.noreply.github.com>
* Use Duration type and debug format for timeout errors
Changed timeout() methods to accept Duration directly instead of u128 milliseconds.
This uses Rust's built-in Duration debug formatting ({:?}) which provides clean output:
- Sub-second: "500ms"
- Exact seconds: "1s"
- Mixed: "2.5s"
Updated all call sites to pass Duration directly instead of timeout.as_millis().
Co-authored-by: agourlay <606963+agourlay@users.noreply.github.com>
* fmt/clippy
* make it consistent
* Inline some arguments and imports
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: agourlay <606963+agourlay@users.noreply.github.com>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
Co-authored-by: timvisee <tim@visee.me>
* Remove faulty mark-as-dead condition when recovering Raft snapshot 😅
* Mark new local replicas as locally-disabled when recovering Raft snapshot
* Add test
* sync function to ask to stop workers
fmt
update blocking_ask_workers_to_stop
* graceful shard stop on ShardReplicaSet changes
* make stop consuming
* implement graceful stop for replica set
* graceful shard stop in start_resharding_unchecked
* stop_gracefully fn on shard_holder
* collection graceful stop on delete
* explain when it is ok to shutdown
* fix tests and ensure blocking callsa are not called from async runtime
* fmt
* Annotate cancel safety for do_recover_from_snapshot function
* Minor comment nit
* make drop not wait on stop thread
* graceful stop in more tests
* graceful stop in more tests
* avoid background thread is shard was stopped gracefully
* annotate cancel safety
* fmt
* More cancel safety annotations
* Add two spawns to ensure cancel safety
---------
Co-authored-by: timvisee <tim@visee.me>
* After clearing WAL, flush segment
* Add debug log when WAL is cleared
* Clear WAL on consensus snapshot after writing state, truncate on start
* Apply consensus snapshot offset
* Fix off by one error
* Tweak debug assertion message
* Change WAL reconciliation condition, and fully clear WAL in this case
* Add debug assertion to prove Raft index and snapshot index are equal
* Add documentation to resolve bot nit
* Return error on WAL clear failure
* Fix typo
* Remove unused truncate functions
* Prematurely abort blocking task in `spawn_cancel_on_drop` on drop
These tasks are intended to be cancellable. Now we prematurely abort the
task if the future was dropped before the task is executed.
* Prematurely abort blocking task in `spawn_cancel_on_token` on cancel
These tasks are intended to be cancellable. Now we prematurely abort the
task if the cancellation token is triggered before the task is executed.
* Prematurely abort blocking task for fetching telemetry
* Prematurely abort stoppable task on drop, all are safe to abort early
* Make `move_dir` either move everything, or nothing at all
That is with the exception of file IO errors in which case data may be
partially moved.
Before this PR it was possible for the new target directory to be
created without moving all data into it. Now we either do all, or
nothing.
* Prematurely abort task for creating full snapshot
It is fine to either create it, or not at all.
* Prematurely abort blocking task for waiting on consensus leader
* Prematurely abort blocking cardinality estimation and shard info tasks
* Prematurely abort blocking point deduplication task
* Prematurely abort blocking task for checking available disk space
* Prematurely abort blocking shard read operations
All shard read operations, such as retrieve, scroll, facets and more can
be safely aborted prematurely.
Related to: <https://github.com/qdrant/qdrant/pull/7530>
* Prematurely abort blocking task for waiting on replica state
* Prematurely abort blocking task for waiting on transfer replica states
* Prematurely abort blocking task for loading segment
This can safely be aborted before the task is started
* Prematurely abort blocking task waiting for replica states
* Prematurely abort blocking task for creating snapshot file
Safe because it aborts before writing any snapshot files to disk
* simple integration test for tenant promotion
* add cleanup after promotion
* When switching to ReadActive state, set it for the correct peer ID
* fix test
---------
Co-authored-by: timvisee <tim@visee.me>
* make jwt with payload filter fail
* remove `whole` access requirement
* remove unnecessary mut
* fmt
* adjust test
* fix test
* remove more tests
* fix test
* fix test again
* fix test again
* Replace deprecated PayloadConstraint with JSON Value placeholder
* Fix expect message
---------
Co-authored-by: timvisee <tim@visee.me>
