* Add Rust edge example: add-named-vector
Port `lib/edge/python/examples/add-named-vector.py` to Rust under
`lib/edge/publish/examples/src/bin/add-named-vector.rs`.
Re-export `VectorNameOperations`, `CreateVectorName`, `DeleteVectorName`,
`VectorNameConfig`, `DenseVectorConfig`, and `SparseVectorConfig` from
`qdrant_edge` so the public Rust API can create/delete named vectors
without reaching into the internal `shard` crate.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fmt
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* [ai] TQDT in the API
* [ai] unify new sparse error for TQDT
* Rename to `turbo4`
* Add `Turbo4` to comments and doc strings.
* Also validate named sparse vector creation
* ci(windows): skip IO-heavy tests that aren't OS-specific
On the Windows CI runner, several tests are 3-25x slower than on Ubuntu
purely due to slow filesystem IO. These tests exercise platform-agnostic
logic (optimizer, snapshot, WAL recovery, dedup, deferred points) and
are fully covered by the Linux and macOS jobs.
Mark them with `#[cfg_attr(target_os = "windows", ignore = "...")]` so:
- Windows CI skips them and finishes faster.
- They're still listed and runnable via `cargo test -- --ignored`
on Windows for local debugging.
Based on JUnit timings from CI run 26462785436 (PR #8827), this should
save ~5 minutes wall-clock on the Windows job, taking it closer to the
~13min Ubuntu and ~9min macOS jobs (currently 20m24s).
Tests affected:
- lib/wal: check_wal, check_last_index, check_clear, check_reopen,
check_truncate, check_prefix_truncate, test_prefix_truncate_parametric
- lib/edge/optimize: full tests module
- lib/segment deferred-point tests: read_operations,
dense_segment_combinations, sparse, facets
- lib/collection: snapshot_test, points_dedup, wal_recovery,
collection_test::test_ordered_read_api, snapshot_recovery_test
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert(ci/windows): keep WAL and WAL-recovery tests on Windows
Reviewer correctly pointed out that WAL is mmap-backed and has
substantial Windows-specific code paths:
- Different segment allocation (fs4 vs rustix::ftruncate)
- Windows-specific delete_windows() with mmap-drop + retry loop
- Windows-specific sync_all() because directory fsync is unavailable
- Windows-specific lock proxy file (directories aren't lockable)
So those tests genuinely need Windows coverage. Reverted skips for:
- lib/wal/src/lib.rs: all check_* tests and test_prefix_truncate_parametric
- lib/collection/src/tests/wal_recovery_test.rs: all three tests
Still skipped on Windows (no OS-specific code in their production paths):
- lib/edge/optimize.rs (no cfg(windows) in source)
- lib/segment deferred-point tests (segment/ has no cfg(windows))
- lib/collection snapshot/dedup tests (collection/ has no cfg(windows))
- lib/collection integration snapshot_recovery + ordered_read_api
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert(ci/windows): keep collection integration and snapshot_test
Per reviewer request, keep running these on Windows:
- lib/collection/tests/integration/* (snapshot_recovery_test,
collection_test::test_ordered_read_api)
- lib/collection/src/tests/snapshot_test.rs
These exercise higher-level collection/snapshot behavior that benefits
from cross-platform validation.
Remaining Windows skips (production code has no cfg(windows) branches):
- lib/edge/src/optimize.rs: 14 optimizer tests
- lib/segment/src/segment/tests/mod.rs: 4 deferred-point tests
- lib/collection/src/tests/points_dedup.rs: 2 dedup tests
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci(windows): also skip HNSW/quantization integration tests
Per reviewer, also skip these segment integration test modules on Windows:
- hnsw_quantized_search_test::* (25 tests)
- multivector_filtrable_hnsw_test::* (rstest cases)
- multivector_quantization_test::* (rstest cases)
- byte_storage_quantization_test::* (rstest cases)
- payload_index_test::test_struct_payload_index_nested_fields
These exercise pure HNSW/quantization correctness on top of standard
segment IO that is already covered by tests we keep running on Windows.
Adds ~930s of sequential time to the Windows skip list, bringing the
expected wall-clock saving from ~3 min to ~8-10 min.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(edge): expose WAL options via EdgeShard::load_with_wal_options
Adds a sibling method to EdgeShard::load that accepts custom
WalOptions, alongside the existing load() which keeps using
default_wal_options().
Motivation: embedded/mobile deployments (e.g. Flutter plugins on
iOS/Android) need much smaller WAL segments than the 32 MiB default —
typically 4 MiB. On filesystems with sparse files (APFS/ext4/F2FS)
the physical footprint is small, but the visible/reported size is
the full segment capacity, which is surfaced by iCloud backup, OS
size pickers, etc.
Changes:
* New EdgeShard::load_with_wal_options(path, config, wal_options).
* EdgeShard::load delegates to it with default_wal_options() — no
behavior change for existing callers.
* ensure_dirs_and_open_wal now takes WalOptions explicitly; called
from both EdgeShard::new (with default) and load_with_wal_options.
* WalOptions re-exported from edge crate root.
* Two regression tests in lib/edge/tests/wal_options.rs.
WAL options are intentionally a runtime parameter, not part of
EdgeConfig (which is persisted to edge_config.json) — different
processes may legitimately open the same shard with different WAL
options.
* test(edge): verify mismatching WAL options on reload preserve data
Addresses upstream review feedback (qdrant/qdrant#9067): demonstrate
that reloading an existing shard with WAL options different from the
ones it was created with is safe and preserves all previously written
points.
Two new tests in lib/edge/tests/wal_options.rs:
* reload_with_smaller_wal_capacity_after_upsert:
create shard with default 32 MiB WAL -> upsert point 42 ->
drop -> reload with 4 MiB WAL options -> point 42 still readable ->
upsert point 43 under smaller WAL -> count == 2.
* reload_with_larger_wal_capacity_after_upsert (symmetric):
create -> reload with 4 MiB -> upsert point 100 -> drop ->
reload with default 32 MiB -> point 100 readable -> upsert
point 101 -> count == 2.
Both pass. The WAL crate does not validate segment_capacity on
reload — existing segments on disk keep their original size,
subsequent appends honor the runtime options. This confirms the
PR description's claim that WalOptions is a runtime hint, not
persisted shard state.
* style(edge): cargo fmt for wal_options.rs mismatch tests
* refactor(edge): replace load_with_wal_options with builder-based options
Addresses upstream feedback (timvisee, generall): the sibling
constructor pattern doesn't scale as runtime configurability grows.
Replaces `EdgeShard::load_with_wal_options(path, config, wal_options)`
with `EdgeShard::load_with_options(path, config, EdgeShardOptions)`,
where `EdgeShardOptions` is a builder-style runtime-options struct.
* `EdgeShardOptions::new().with_wal_options(opts)` is the equivalent
of the old call.
* Adding new runtime options later (e.g. wal flush interval, initial
indexing threshold) is now an additive change — new builder method
on `EdgeShardOptions`, no new constructor variant on `EdgeShard`.
* `EdgeShardOptions` is intentionally not persisted (no
Serialize/Deserialize). The reload-mismatch tests above confirm
that WAL options are a runtime hint, not shard identity, so they
don't belong in `EdgeConfig` (which is persisted to
`edge_config.json`).
* `EdgeShard::load(path, config)` unchanged for existing callers —
delegates to `load_with_options(..., EdgeShardOptions::default())`.
Tests in `lib/edge/tests/wal_options.rs` migrated to the new shape;
all four still pass:
test load_with_options_accepts_custom_wal_capacity ... ok
test load_still_works_with_default_wal_options ... ok
test reload_with_smaller_wal_capacity_after_upsert ... ok
test reload_with_larger_wal_capacity_after_upsert ... ok
* refactor(edge): add fluent builders; move wal_options into EdgeConfig
Replaces the EdgeShardOptions side-channel with a single EdgeConfig that
carries wal_options inline, and introduces fluent builders for the three
user-facing config types.
* WalOptions now derives Clone/PartialEq/Eq/Serialize/Deserialize so it
can live inside EdgeConfig and round-trip through edge_config.json.
* EdgeConfig.wal_options (Option<WalOptions>) replaces EdgeShardOptions.
EdgeShard::load_with_options is folded into EdgeShard::load; both new
and load drive the WAL from config.wal_options.unwrap_or_default().
* New builders/ module hosts EdgeConfigBuilder, EdgeVectorParamsBuilder,
and EdgeSparseVectorParamsBuilder. Each builder has explicit per-field
storage and constructs its target via an exhaustive struct literal in
build(), so adding a field to the target forces a compile error in the
builder.
* publish example switched to the new builder API.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style(edge): cargo fmt after builder refactor
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous description was outdated: it claimed that enabling this
option "blocks updates at the request level" until segments are
re-optimized. In practice the implementation uses "deferred points":
new points written to large unoptimized segments are persisted but
excluded from read/search results until the segments are optimized.
Updates are not blocked; only `wait=true` clients are made to wait for
the deferred points to become visible. Update this in the REST schema
(via `OptimizersConfig` / `OptimizersConfigDiff`), in the gRPC proto,
in the edge config docstrings, and regenerate the OpenAPI bundle via
`tools/generate_openapi_models.sh`.
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add empty placeholder vector storage types for named vector CRUD
Introduce EmptyDenseVectorStorage and EmptySparseVectorStorage as
placeholder storages for newly created named vectors on immutable
segments. These report all vectors as deleted, consume no disk space,
and are reconstructed from segment config on load via the new
VectorStorageType::Empty and SparseVectorStorageType::Empty variants.
Key design decisions:
- is_on_disk is derived from original user config, not hardcoded
- MultiVectorConfig is preserved for multi-vector support
- Config mismatch optimizer skips Empty storage to avoid false rebuilds
- Quantization delegates normally (handles 0 vectors gracefully)
- get_vector includes debug_assert to catch unexpected access
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [AI] segment-level operations for creating and deleting anmed vectors
* [AI] implement named vector creation and deleting in proxy segment
* [AI] Step 3: Proxy Segment Handling for Named Vector Operations
* [AI] implement for Edge
* [AI] implement consensus operations for named vector operations
* [AI] refactor VectorNameConfig, remove VectorNameConfigInternal
* [AI] handle vector schema inconsistency in raft snapshot recovery
* [AI] rest + grpc API
* [AI] clippy
* [AI] generate openAPI schema
* fmt
* ci fixes
* [AI] fix jwt access test
* [AI] nop operation for awaiting of consensus-commited update ops
* [AI] move vector name operations into points service
* [AI] implement internal api for vector name operations
* [AI] change collection-level config along with segment level operation
* [AI] vector schema reconceliation instead of error
* fmt
* missing compile-time option
* [AI] integration test
* [AI] fix missing JWT tests
* [AI] remove NOP
* [AI] openapi test
* [AI] fix initialization of mutable segment
* [AI] more simple integration tests
* fmt
* [AI] make cluster test a bit harder
* [AI] make test less flacky
* [AI] rabbit comments
* [AI] check params compatibility before writing vector config
* [AI] make sure to register vector storages in structure payload index
* [AI] vector name validation
* lower vector length validation to 200 chars to account for prefix in filename
* [AI] proxy segment: prevent stale data leak through optimization
* fmt
* [AI] filter out removed vectors from proxy response
* [AI] handle vector name in proxy
* fmt
* adjust proxy info based on dropped vectors
* [AI] proxy segment: update filters to correct has_vector condition
* fmt
* clippy
* Fix consensus snapshot applicaiton for vector schema
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(amalgamate): strip `mod gpu`
Fixes `cargo +nightly fmt` in `lib/edge/publish`. Previously we had to
run `cargo +nightly fmt -p examples` to avoid the error.
* fix(Justfile): unset VIRTUAL_ENV
Otherwise it causes problems when running that Justfile from other
(unrelated) venvs. Particularly, `uv pip install` and `maturin develop`
will mess with cached venvs created by `uv run --script` which sounds
insane.
* Fix edge sparse vector search panic on score postprocessing
The distance lookup for score postprocessing only checked dense vector
configs, causing a panic when searching sparse vectors. Fall back to
Distance::Dot for sparse vectors, matching the full server behavior.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix formatting
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Address PR review: return error instead of panic, fix example
- Replace panic! with OperationError::service_error for unknown vector
names in edge search, avoiding process crash on bad client input
- Update stale "panics" comments and add assertions in sparse-search example
Made-with: Cursor
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <agent@cursor.com>
* Refactor: `SearchSegmentEntry` for search ops
* Move `has_deferrend_points_method`
to `SearchSegmentEntry`
* Reorg imports
* Renamings per review
* fixup! Renamings per review
* `StorageSegmentEntry` trait
It contains methods dealing with storage and syncronization
* Move index methods from `ReadSegmentEntry`
to `SegmentEntry`
* Add `_concurrent` suffix to some methods
* move field index methods into NonAppendableSegmentEntry
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Don't account for deferred points in some places
# Conflicts:
# lib/collection/src/shards/local_shard/scroll.rs
* Add in QueryContext
* Cover more places
* Coderabbit review remarks
* Properly count amount of deleted deferred points (#8386)
* Properly count amount of deleted deferred points
* Prevent double-counting of the same point
* Remove hints to estimations
* Properly handle counts in ProxySegment
* Add tests and fix deleted point count issue
* Adjusts tests + fix issues
* Separte fields for deferred points in telemetry (SegmentInfo)
* Remove deferred_points_count()
* openapi
* Fix test by manually calculating visible points
* Adjust ProxySegment test to revertion of SegmentInfo
* Throw error if collection was not found in telemetry (e2e Test)
* Update lib/segment/src/segment/segment_ops.rs
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Make new fields in API optional
* Don't take range if no deferred point exist
* Review remarks
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* refactor(edge): split EdgeShard load into new() and load()
- new(path, config): create edge shard only when path has no existing segments
- load(path, config?): load from existing files; config optional (load from
edge_config.json or infer from segments)
- Helpers: has_existing_segments, ensure_dirs_and_open_wal, resolve_initial_config,
load_segments, ensure_appendable_segment
- Python: EdgeShard.create_new(path, config); __init__ still calls load()
- Examples use new() for creation and load(path, None) for reopen
- Error instead of panic on config mismatch; explicit load/create in Python
- fix: use OptimizerThresholds.deferred_internal_id for dev compatibility
Made-with: Cursor
* rollback threshold changes
* fix(edge): address dancixx review comments
- Persist inferred config in load() when edge_config.json does not exist
(SaveOnDisk::new only saves when file exists)
- Python: add #[new] delegating to load() for backward compatibility
- qdrant_edge.pyi: Optional return types for deleted_threshold,
vacuum_min_vector_number, default_segment_number; add __init__ doc
Made-with: Cursor
* Revert "fix(edge): address dancixx review comments"
This reverts commit 370e21a6c74c41210e1658f89814395cb86ed81c.
* fix: optional returns
* fix: save config it is not exist
* fix: save data on disk always
* fix: python examples
* chore: remove edge.close()
* fix: wal lock
* fix: rename of EdgeShardConfig -> EdgeConfig
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* refactor(edge): let edge re-export segment/shard/sparse
* chore(ast-grep-rules): handle path like `::shard::blah::Blah`
This commit was required during the refactoring. It turns out we don't
need it right now, but it might be useful in the future. Or not.
* refactor(edge-rs): Rehaul public API
* lib/edge/publish/cargo: improve target directory discovery
Don't hardcode workspace root, ask cargo instead.
* Fix warning in qdrant-edge amalgamation
`cargo check` in `lib/edge/publish` produces the following warning:
warning: type `PointerUpdates` is more private than the item `gridstore::tracker::Tracker::<S>::write_pending`
* lib/edge/publish/examples: fix compiler warnings
* refactor: lib/edge/publish/examples: put DATA_DIRECTORY into lib.rs
To match python examples that have the same constant in `common.py`.
* Use `lib/edge/data` dir for both Python and Rust examples
Before this commit, `lib/edge/publish/data` and `lib/edge/python/data`
were used. I'd like Python and Rust to use the same prepared resources
in the future.
* ❗Add lib/edge/Justfile with recipes to build/run examples
I constantly keep forgetting how to compile and run edge stuff.
Also, this would be used in CI in subsequent commits.
* ❗doc: lib/edge/README.md: Combine Rust and Python Edge readmes
Also now we can rename `creates-io-readme.md` to `README.md` like
reasonable people.
* ❗doc: Add qdrant-edge-py README.md
Since the previous commit removed qdrant-edge-py README.md, we need to
add something in its place instead.
The new README is adapted from Rust edge README.
And the previous README wasn't good anyway, see [1], it mentions maturin
which is irrelevant for users.
[1]: https://pypi.org/project/qdrant-edge-py/0.5.0/
* GHA: Add "Setup Qdrant" composite action
We need running Qdrant instance to test edge examples on CI. We can't
use docker for this because we want to run tests on Windows and macOS.
OTOH, this action downloads binaries for all platforms provided in
https://github.com/qdrant/qdrant/releases/tag/v1.17.0
* ❗GHA: Use single worflow for python and rust edge examples
Merge workflows to reuse build cache. Also, modernize it by using `uv`,
and recently introduced `setup-qdrant` and Justfile. Also, make it run
on three platforms (only on merges).
* ❗GHA: edge-{py,rust}-release: also run examples before publishing
Use similar approach as in the previous commit. But note that this time
we build/test also on ARM Linux and Windows.
* Add ignore_deferred flag for internal scroll API
* Ignore deferred points in ProxyShard::update
* Use enum instead of bool as type in function parameters
* Use consistent parameter name
* fmt
* Apply suggestions from code review
Co-authored-by: Tim Visée <tim+github@visee.me>
* More explicit naming
* Add DeferredBehavior to `count()` and disable filtering in stream_records
* Don't filter `retrieve()` everywhere
* Use consistent filter behavior in read operations
* Add TODO for ignored parameter in remote_shard
* Rebase fixes
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Adjust points selection for deferred points update
* adjust proxy segment implementation
* simplify
* use simpler proxy impl
* stick to Entry API
* renaming to stay closer to the original
* two passes and simpler impl.
* fmt
* fmt
* use predefined deferred internal id
* calculate deferred point id
* move deferred check to the entry
* fix after rebase
* fmt
* fix tests
* review remarks
* fix tests
* codespell fix
* are you happy clippy
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Introduce EdgeShardConfig for edge shard
- Add EdgeShardConfig and EdgeOptimizersConfig in lib/edge/src/config.rs
- Segment config (vector_data, sparse_vector_data, payload_storage_type)
- Global hnsw_config and per-vector HNSW in segment config
- Optimizer params: deleted_threshold, vacuum_min_vector_number,
default_segment_number, max_segment_size, indexing_threshold,
prevent_unoptimized (excludes memmap_threshold, flush_interval_sec,
max_optimization_threads)
- Persist/load as edge_config.json in shard path
- EdgeShard uses RwLock<EdgeShardConfig>; load() accepts Option<EdgeShardConfig>,
falls back to file or infer from segments; compatibility checked on load
- load_with_segment_config() for backward compatibility (SegmentConfig -> EdgeShardConfig)
- optimize() uses EdgeShardConfig for hnsw and optimizer thresholds
- Public methods: set_hnsw_config(), set_vector_hnsw_config(), set_optimizers_config()
(update and persist)
- Python and examples use load_with_segment_config with existing config API
Made-with: Cursor
* Refactor EdgeShardConfig: user-facing params only, config module
- Replace SegmentConfig inside EdgeShardConfig with user-facing fields:
- on_disk_payload (bool) instead of payload_storage_type
- vectors: HashMap<VectorNameBuf, EdgeVectorParams> with on_disk per vector,
no per-vector quantization; global quantization_config only
- sparse_vectors: HashMap<VectorNameBuf, EdgeSparseVectorParams> with on_disk
- EdgeVectorParams / EdgeSparseVectorParams use on_disk (bool) instead of
storage_type; conversion to VectorDataConfig/SparseVectorDataConfig in
to_segment_config()
- Add config module: mod.rs, optimizers.rs, vectors.rs, shard.rs
- from_segment_config(&SegmentConfig) fills all inferrable params
- to_segment_config() builds SegmentConfig for segments and optimize()
- load_with_segment_config takes Option<SegmentConfig>, uses from_segment_config
Made-with: Cursor
* Move optimizer threshold helpers to shard crate
- Add get_number_segments, get_indexing_threshold_kb, get_max_segment_size_kb,
get_deferred_points_threshold_bytes in shard::optimizers::config
- Collection OptimizersConfig and edge EdgeOptimizersConfig delegate to these
- Single place for threshold logic; collection and edge use shard helpers
Made-with: Cursor
* Use destructuring in config conversions to avoid missing new fields
- EdgeVectorParams: destructure VectorDataConfig in from_*, destructure self in to_vector_data_config
- EdgeSparseVectorParams: destructure SparseVectorDataConfig and SparseIndexConfig in from_*, destructure self in to_sparse_vector_data_config
- EdgeShardConfig: destructure SegmentConfig in from_segment_config, destructure self in to_segment_config
Adding new fields to source structs will now cause compile errors until conversions are updated.
Made-with: Cursor
* refactor: centralize on_disk_payload→payload_storage_type, on_disk→storage_type, and appendable quantization logic
- PayloadStorageType::from_on_disk_payload(bool) in segment (Mmap/InRamMmap)
- VectorStorageType::from_on_disk(bool) in segment (ChunkedMmap/InRamChunkedMmap)
- QuantizationConfig::for_appendable_segment(Option<&Self>) in segment (feature flag + supports_appendable)
- collection: use from_on_disk_payload in non-rocksdb branch
- edge shard/vectors: use new helpers; remove duplicated conditionals
- shard optimizers: use from_on_disk and for_appendable_segment
Made-with: Cursor
* refactor(edge): use EdgeShardConfig directly, drop segment_config
- Add plain_segment_config() for create_appendable_segment (no HNSW)
- Add segment_optimizer_config() built from EdgeShardConfig for blocking optimizers
- Add vector_data_config(name) for query/MMR
- build_blocking_optimizers: use segment_optimizer_config() instead of SegmentConfig
- create_appendable_segment: use plain_segment_config()
- search/query: use config().vectors and vector_data_config() instead of segment_config()
- Remove segment_config() from EdgeShardConfig and EdgeShard
- Add to_plain_vector_data_config on EdgeVectorParams
Made-with: Cursor
* [manual] review changes
* refactor(edge-py): wrap EdgeShardConfig, add EdgeVectorParams/EdgeSparseVectorParams
- PyEdgeConfig now wraps EdgeShardConfig (vectors, sparse_vectors, on_disk_payload, etc.)
- PyEdgeVectorParams / PyEdgeSparseVectorParams wrap edge config types
- PyEdgeOptimizersConfig for optional optimizer settings
- EdgeShard.load() uses EdgeShardConfig; edge::config made pub for Python crate
- cargo fmt + clippy (remove map_identity)
Made-with: Cursor
* refactor(edge-py): simplify config API, remove unused Py* types, add EdgeConfig
- Remove unused PyPayloadStorageType, PyVectorDataConfig, PyVectorStorageType,
PySparseVectorDataConfig, PySparseVectorStorageType from Python bindings
- Move PyEdgeOptimizersConfig to lib/edge/python/src/config/optimizers.rs
- Update qdrant_edge.pyi: EdgeConfig with vectors/sparse_vectors,
EdgeVectorParams, EdgeSparseVectorParams, EdgeOptimizersConfig
- Update examples (common.py, repr.py) to use new config API
- Run cargo fmt
Made-with: Cursor
* [manual] review changes
* [manual] review changes
* [manual] fix test
* Address CodeRabbit review comments for PR 8322 (#8324)
* Address CodeRabbit review comments for PR 8322
- Python examples: explicit imports (repr.py, common.py) and new EdgeConfig API
- HnswIndexConfig: add max_indexing_threads param and property in .pyi and Rust bindings
- EdgeConfig: make vectors optional for sparse-only configs; validate at least one of vectors/sparse_vectors
- EdgeShardConfig::load: use try_exists(), propagate I/O errors
- from_segment_config: infer hnsw_config from per-vector HNSW when all agree
- EdgeShard setters: atomic clone-mutate-save-then-replace; persist config save errors
- Segment compat: prefix vector name in error messages; resolve None datatype to Float32
- max_indexing_threads: preserve 0 (auto) sentinel in trait default; remove per-optimizer overrides
- SegmentOptimizerConfig:🆕 build plain and optimizer maps in single pass
- config_mismatch_optimizer tests: use VectorNameBuf::from() instead of .into()
- vectors.rs: doc updates for per-vector quantization
Made-with: Cursor
* Address @generall review: SaveOnDisk for config, resolve num_rayon_threads in optimizer
- Use SaveOnDisk<EdgeShardConfig> for EdgeShard config (generall: 'We have SaveOnDisk struct for this')
- Create via SaveOnDisk::new() after resolving config; setters use .write() for atomic persist
- set_vector_hnsw_config: clone then mutate then write (fallible setter)
- max_indexing_threads: resolve 0 (auto) via num_rayon_threads inside impl (generall: 'proper solution would be to resolve num_rayon_threads inside the optimizer impl')
- max_indexing_threads_sentinel_aware() now returns Some(num_rayon_threads(raw)) so callers get actual thread count
Made-with: Cursor
* [manual] reorganize num_rayon_threads -> get_num_indexing_threads to better account per-vector configuration
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* update docstring and pyi
* fmt
* fmt
* clipy
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
Change pub mod to mod for count, facet, info, optimize, query,
retrieve, scroll, search, snapshots, and update in lib/edge.
The public API (EdgeShard, ShardInfo) is unchanged.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* amalgamate.py: replace `env!("CARGO_PKG_VERSION")`
Otherwise segment thinks it's v0.0.0 which breaks examples.
* Fix lib/edge/python/examples/repr.py
It it broken by d9393acac6 (#7933), which renamed `SegmentConfig` →
`EdgeConfig` and removed `index` parameter from `VectorDataConfig` by
hardcoding it to `Plain`.
Before:
$ python lib/edge/python/examples/repr.py
NameError: name 'SegmentConfig' is not defined
After:
$ python lib/edge/python/examples/repr.py
EdgeConfig(vector_data={"": VectorDataConfig(size=128, ...)}, ...)
* gitignore output of Rust edge examples
* Restore `lib/edge/python/examples/.gitignore`
I'm not sure why it's removed in 65312a1e82e6ae4d/#7522.
* Add `prepare_facet_snapshot.sh` for `facet_test.rs`
The script is based on steps mentioned in #8045.
* Remove `edge-cli.rs` example
It was added in 895318913afbad4a/#7140 and seems incomplete.
* Move `DenseVector`/`MultiDenseVector` from `api` to `segment`
* Move `OrderByInterface` from `api` to `segment`
Reason: it's used in `edge` which shouldn't depend on `api`.
* Make `shard` -> `api` dependency optional
* Remove `api` from the amalgamation
* Don't install protoc in edge Actions
* amalgamate.py: specify encoding="utf-8"
For legacy platforms that still default to cp1252.
* amalgamate.py: use `shutil.which`
In case if `ast-grep` is a cmd wrapper.
* amalgamate.py: write rules to temporary file
Windows runner don't like inline-rules.
* amalgamate.py: Add decription to package
Required for publishing to crates.io.
* amalgamate.py: package build scripts for `common` and `segment`
Also, add more comments.
* amalgamate.py: keep generated module order deterministic
Not an issue, but nice to have. Suggested by CodeRabbit.
* edge-rust-release.yml: also check on arm machines
* edge-rust-release.yml: Use cargo build instead of cargo check
To catch linker errors.
* edge-rust-release.yml: disable fail-fast (for easier debugging)
* edge-rust-release.yml: add --no-verify to cargo publish
The previous job `edge-rust-check` already builds it, no need to build
it again in the `publish` job.
* Fixups of amalgamator
Fix issues that break `qdrant-edge` build process:
- `use … as segment;` - this causes `ast-grep` rules to replace wrong
paths. So, rename to avoid collisions.
- `#[macro_use]` and `extern crate` required be in the top-level
`lib.rs`.
- `format!("…", crate::something::…)` - `ast-grep` can't fix paths
inside macros. Fixed by moving `crate::something::…` out of the macro.
* Add lib/edge/publish workspace and amalgamation script
* Move `lib/edge/examples` into `lib/edge/publish/` workspace
And fix them to use the generated `qdrant-edge` crate.
* Add github workflow
* Cleanup `qdrant-edge` public API
Removes empty modules. Checked by `cargo doc`.
