mirror of
https://github.com/qdrant/qdrant.git
synced 2026-07-30 06:30:57 -05:00
4d23f32681
* Introduce EdgeShardConfig for edge shard
- Add EdgeShardConfig and EdgeOptimizersConfig in lib/edge/src/config.rs
- Segment config (vector_data, sparse_vector_data, payload_storage_type)
- Global hnsw_config and per-vector HNSW in segment config
- Optimizer params: deleted_threshold, vacuum_min_vector_number,
default_segment_number, max_segment_size, indexing_threshold,
prevent_unoptimized (excludes memmap_threshold, flush_interval_sec,
max_optimization_threads)
- Persist/load as edge_config.json in shard path
- EdgeShard uses RwLock<EdgeShardConfig>; load() accepts Option<EdgeShardConfig>,
falls back to file or infer from segments; compatibility checked on load
- load_with_segment_config() for backward compatibility (SegmentConfig -> EdgeShardConfig)
- optimize() uses EdgeShardConfig for hnsw and optimizer thresholds
- Public methods: set_hnsw_config(), set_vector_hnsw_config(), set_optimizers_config()
(update and persist)
- Python and examples use load_with_segment_config with existing config API
Made-with: Cursor
* Refactor EdgeShardConfig: user-facing params only, config module
- Replace SegmentConfig inside EdgeShardConfig with user-facing fields:
- on_disk_payload (bool) instead of payload_storage_type
- vectors: HashMap<VectorNameBuf, EdgeVectorParams> with on_disk per vector,
no per-vector quantization; global quantization_config only
- sparse_vectors: HashMap<VectorNameBuf, EdgeSparseVectorParams> with on_disk
- EdgeVectorParams / EdgeSparseVectorParams use on_disk (bool) instead of
storage_type; conversion to VectorDataConfig/SparseVectorDataConfig in
to_segment_config()
- Add config module: mod.rs, optimizers.rs, vectors.rs, shard.rs
- from_segment_config(&SegmentConfig) fills all inferrable params
- to_segment_config() builds SegmentConfig for segments and optimize()
- load_with_segment_config takes Option<SegmentConfig>, uses from_segment_config
Made-with: Cursor
* Move optimizer threshold helpers to shard crate
- Add get_number_segments, get_indexing_threshold_kb, get_max_segment_size_kb,
get_deferred_points_threshold_bytes in shard::optimizers::config
- Collection OptimizersConfig and edge EdgeOptimizersConfig delegate to these
- Single place for threshold logic; collection and edge use shard helpers
Made-with: Cursor
* Use destructuring in config conversions to avoid missing new fields
- EdgeVectorParams: destructure VectorDataConfig in from_*, destructure self in to_vector_data_config
- EdgeSparseVectorParams: destructure SparseVectorDataConfig and SparseIndexConfig in from_*, destructure self in to_sparse_vector_data_config
- EdgeShardConfig: destructure SegmentConfig in from_segment_config, destructure self in to_segment_config
Adding new fields to source structs will now cause compile errors until conversions are updated.
Made-with: Cursor
* refactor: centralize on_disk_payload→payload_storage_type, on_disk→storage_type, and appendable quantization logic
- PayloadStorageType::from_on_disk_payload(bool) in segment (Mmap/InRamMmap)
- VectorStorageType::from_on_disk(bool) in segment (ChunkedMmap/InRamChunkedMmap)
- QuantizationConfig::for_appendable_segment(Option<&Self>) in segment (feature flag + supports_appendable)
- collection: use from_on_disk_payload in non-rocksdb branch
- edge shard/vectors: use new helpers; remove duplicated conditionals
- shard optimizers: use from_on_disk and for_appendable_segment
Made-with: Cursor
* refactor(edge): use EdgeShardConfig directly, drop segment_config
- Add plain_segment_config() for create_appendable_segment (no HNSW)
- Add segment_optimizer_config() built from EdgeShardConfig for blocking optimizers
- Add vector_data_config(name) for query/MMR
- build_blocking_optimizers: use segment_optimizer_config() instead of SegmentConfig
- create_appendable_segment: use plain_segment_config()
- search/query: use config().vectors and vector_data_config() instead of segment_config()
- Remove segment_config() from EdgeShardConfig and EdgeShard
- Add to_plain_vector_data_config on EdgeVectorParams
Made-with: Cursor
* [manual] review changes
* refactor(edge-py): wrap EdgeShardConfig, add EdgeVectorParams/EdgeSparseVectorParams
- PyEdgeConfig now wraps EdgeShardConfig (vectors, sparse_vectors, on_disk_payload, etc.)
- PyEdgeVectorParams / PyEdgeSparseVectorParams wrap edge config types
- PyEdgeOptimizersConfig for optional optimizer settings
- EdgeShard.load() uses EdgeShardConfig; edge::config made pub for Python crate
- cargo fmt + clippy (remove map_identity)
Made-with: Cursor
* refactor(edge-py): simplify config API, remove unused Py* types, add EdgeConfig
- Remove unused PyPayloadStorageType, PyVectorDataConfig, PyVectorStorageType,
PySparseVectorDataConfig, PySparseVectorStorageType from Python bindings
- Move PyEdgeOptimizersConfig to lib/edge/python/src/config/optimizers.rs
- Update qdrant_edge.pyi: EdgeConfig with vectors/sparse_vectors,
EdgeVectorParams, EdgeSparseVectorParams, EdgeOptimizersConfig
- Update examples (common.py, repr.py) to use new config API
- Run cargo fmt
Made-with: Cursor
* [manual] review changes
* [manual] review changes
* [manual] fix test
* Address CodeRabbit review comments for PR 8322 (#8324)
* Address CodeRabbit review comments for PR 8322
- Python examples: explicit imports (repr.py, common.py) and new EdgeConfig API
- HnswIndexConfig: add max_indexing_threads param and property in .pyi and Rust bindings
- EdgeConfig: make vectors optional for sparse-only configs; validate at least one of vectors/sparse_vectors
- EdgeShardConfig::load: use try_exists(), propagate I/O errors
- from_segment_config: infer hnsw_config from per-vector HNSW when all agree
- EdgeShard setters: atomic clone-mutate-save-then-replace; persist config save errors
- Segment compat: prefix vector name in error messages; resolve None datatype to Float32
- max_indexing_threads: preserve 0 (auto) sentinel in trait default; remove per-optimizer overrides
- SegmentOptimizerConfig:🆕 build plain and optimizer maps in single pass
- config_mismatch_optimizer tests: use VectorNameBuf::from() instead of .into()
- vectors.rs: doc updates for per-vector quantization
Made-with: Cursor
* Address @generall review: SaveOnDisk for config, resolve num_rayon_threads in optimizer
- Use SaveOnDisk<EdgeShardConfig> for EdgeShard config (generall: 'We have SaveOnDisk struct for this')
- Create via SaveOnDisk::new() after resolving config; setters use .write() for atomic persist
- set_vector_hnsw_config: clone then mutate then write (fallible setter)
- max_indexing_threads: resolve 0 (auto) via num_rayon_threads inside impl (generall: 'proper solution would be to resolve num_rayon_threads inside the optimizer impl')
- max_indexing_threads_sentinel_aware() now returns Some(num_rayon_threads(raw)) so callers get actual thread count
Made-with: Cursor
* [manual] reorganize num_rayon_threads -> get_num_indexing_threads to better account per-vector configuration
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* update docstring and pyi
* fmt
* fmt
* clipy
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
49 lines
1.8 KiB
Python
49 lines
1.8 KiB
Python
# Common helper items for the examples.
|
|
# See lib/edge/publish/examples/src/lib.rs for the equivalent Rust helpers.
|
|
|
|
import os
|
|
import shutil
|
|
import uuid
|
|
|
|
from qdrant_edge import (
|
|
Distance,
|
|
EdgeConfig,
|
|
EdgeShard,
|
|
EdgeVectorParams,
|
|
Point,
|
|
UpdateOperation,
|
|
)
|
|
|
|
DATA_DIRECTORY = os.path.join(os.path.dirname(__file__), "data")
|
|
|
|
def load_new_shard():
|
|
print("---- Load shard ----")
|
|
|
|
# Clear and recreate data directory
|
|
if os.path.exists(DATA_DIRECTORY):
|
|
shutil.rmtree(DATA_DIRECTORY)
|
|
|
|
os.makedirs(DATA_DIRECTORY)
|
|
|
|
# Load Qdrant Edge shard
|
|
config = EdgeConfig(
|
|
vectors=EdgeVectorParams(size=4, distance=Distance.Dot),
|
|
)
|
|
|
|
return EdgeShard(DATA_DIRECTORY, config)
|
|
|
|
|
|
def fill_dummy_data(shard: EdgeShard):
|
|
shard.update(UpdateOperation.upsert_points([
|
|
Point(1, [0.05, 0.61, 0.76, 0.74], {"color": "red", "city": ["Moscow", "Berlin"]}),
|
|
Point(2, [0.19, 0.81, 0.75, 0.11], {"color": "red", "city": "Mexico"}),
|
|
Point(3, [0.36, 0.55, 0.47, 0.94], {"color": "blue", "city": ["Berlin", "Barcelona"]}),
|
|
Point(4, [0.12, 0.34, 0.56, 0.78], {"color": "green", "city": "Lisbon", "rating": 4.5}),
|
|
Point(5, [0.88, 0.12, 0.33, 0.44], {"color": "yellow", "city": ["Paris"], "active": True}),
|
|
Point(6, [0.21, 0.22, 0.23, 0.24], {"color": "blue", "city": "Tokyo", "tags": ["night", "food"]}),
|
|
Point(7, [0.99, 0.01, 0.50, 0.50], {"color": "red", "city": ["New York", "Boston"], "visits": 7}),
|
|
Point(8, [0.10, 0.20, 0.30, 0.40], {"color": "blue", "city": "Seoul", "meta": {"source": "import"}}),
|
|
Point(9, [0.45, 0.55, 0.65, 0.75], {"color": "green", "city": ["Berlin"], "score": 0.92}),
|
|
Point(10, [0.01, 0.02, 0.03, 0.04], {"color": "yellow", "city": None, "featured": False}),
|
|
]))
|