Replace the mixed read interface (internal types, explicit parameter
enumeration, ad-hoc custom types) with edge-owned request structs, one
file per request in src/requests/. Each has a new() constructor taking
only the required parameters, a no-macro fluent builder in src/builders/,
and a From conversion into the internal request type in
requests/conversions/, grouped by request type. Conversions construct
and destructure with full field lists, so a parameter added on either
side fails compilation instead of being silently dropped.
The old reexport aliases (ScrollRequest = ScrollRequestInternal, etc.)
are replaced by the edge types under the same names; retrieve() takes a
RetrieveRequest instead of a parameter triple. Python bindings wrap the
edge types, and the published Rust examples use the builders.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* refactor: group edge crate internals into edge_shard and read_view modules
Restructure lib/edge/src so the top level only contains the modules that
form the public crate surface. Implementation files move under the type
they implement:
- edge_shard/: the EdgeShard struct with its load/config-resolution
helpers (previously inlined in lib.rs), plus optimize, shard_read,
snapshots, and update
- read_view/: the EdgeShardRead/ReadSegmentHandle traits and
EdgeReadView (previously read_view.rs), plus the per-operation impl
files count, facet, grouping, info, matrix, query, retrieve, scroll,
and search; build_search_pool (previously pool.rs) is folded into
read_view/mod.rs next to par_map_segments, the seam it powers
lib.rs is now a thin facade of module declarations and re-exports. The
public API is unchanged: all previously exported names resolve exactly
as before, verified against the edge tests, the python bindings,
edge-shard-query, and the regenerated publish amalgamation with its
examples.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: trim EdgeShardRead surface and split read_view/read_only modules
Trait surface:
- Drop query_scroll and rescore_with_formula from EdgeShardRead and the
EdgeShard inherent wrappers; only the internal query pipeline used
them, via the pub(crate) EdgeReadView methods that remain.
- Hide the snapshot plumbing (read_segments, search_pool, plus
config_snapshot/path providers) in a crate-private ReadViewProvider
trait. EdgeShardRead now declares only user-facing methods and is
implemented for every provider through a blanket impl, so the
plumbing is not callable from user code (verified with a negative
compile test; a private supertrait alone leaves supertrait methods
callable through generic bounds). An empty sealed marker supertrait
keeps the trait unimplementable downstream.
- config_snapshot and path stay public: used by edge-shard-query and
the python bindings.
Module layout:
- read_view/: mod.rs keeps EdgeReadView and build_search_pool;
ReadSegmentHandle moves to handle.rs, the trait machinery to
shard_read.rs, and the nine per-operation impl files into ops/.
- read_only/: the follower's ReadViewProvider impl moves out of mod.rs
into shard_read.rs, mirroring edge_shard/shard_read.rs.
Verified: edge tests, python bindings, edge-shard-query, regenerated
publish amalgamation with all examples compiling and running.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat: segment manifest optimizing state with lease
* fix: clippy
* fix: exhaustive manifest state matching
* fix: merge manifest rebuilds under the write lock
* refactor: named state predicates, drop redundant enumerator test
Review follow-up: move the enumerator's filter into
SegmentManifestState::is_usable, name the preserving predicate
is_optimizer_mark, delete the enumerator test that re-tested serde.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat: read-only edge grouping/matrix search + object-storage read path
Add query_groups (group by a payload field) and search_matrix (single-shard distance matrix over a random sample) to the read-only edge shard's EdgeShardRead API, in new grouping and matrix modules plus edge test helpers.
Make the object-storage read path available outside tests: drop the #[cfg(test)] gate on the BlobFile UniversalReadExt impl and move io_bridge_object_store/object_store to segment's normal dependencies, so a ReadOnlyEdgeShard can serve segments read from S3.
* Share group-by building blocks between server and edge
Move GroupsAggregator, group candidate query shaping (is-empty filter,
group_by payload selector, prefetch limit scaling) and result-order
derivation into shard::grouping / shard::query, so the collection and
edge grouping implementations cannot silently diverge.
Edge grouping now handles multi-valued group keys, u64 keys, wildcard
group_by paths, prefetch limits and score-ordered groups the same way
as the server.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Drive group-by through a shared sans-IO state machine
Extract the multi-request collect/fill loop into
shard::grouping::GroupByDriver: next_request() yields shaped backend
queries, add_points() advances the state, distill() returns the groups.
Query execution stays with the caller, so the async server path and the
sync edge path drive the same machine, and the request shaping helpers
become private to shard::grouping.
Edge now uses the same request budget (5 collect + 5 fill requests) and
per-request candidates limit (groups * group_size, computed inside the
driver) as the server, replacing its single 4x-oversampled request.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* refactor: replace flush_all sync bool with FlushMode enum
SegmentHolder::flush_all took two adjacent bools (sync, force), and call
sites read as bare literal pairs like flush_all(true, false). Swapping
the arguments compiles and silently changes flush semantics: a swapped
pair at the snapshot site would make snapshots skip flushing entirely
when a background flush is running.
Introduce FlushMode { Sync, Background } for the first parameter so the
pair is no longer transposable and the behavior is named at each call
site. The force flag stays a bool since it feeds the
SegmentEntry::flusher(force) trait in lib/segment. No behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* review: exhaustive match on FlushMode instead of equality check
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Tunable EdgeConfig parameters (on_disk_payload, hnsw_config, optimizers)
are now Option, and every tunable resolves through the fallback chain
provided -> persisted -> derived from segments -> default when loading
an existing shard. Leaving a parameter unspecified keeps the shard as it
is; an explicit value overwrites it and existing segments converge to it
through the optimizers.
vectors/sparse_vectors are excluded from overwrite semantics: an empty
map inherits the persisted/segment-derived definitions, a non-empty map
is validated for compatibility against the loaded segments (size,
distance, multivector, datatype, sparse modifier) and fails the load on
mismatch.
The derived layer folds over all segments in UUID order instead of
taking an arbitrary first segment, so a plain appendable segment (which
carries no HNSW parameters) can never mask an indexed segment's actual
build parameters. Previously a lost edge_config.json could resolve
unspecified HNSW params to compiled-in defaults and silently trigger a
full re-index via ConfigMismatchOptimizer.
The read-only follower accepts an optional config on open: provided
tunables are applied once over the segment-derived config (vectors
always come from the segments), and refresh re-derives from segments
alone.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix: unblock optimizer after deleting a named vector
Deleting a named vector could permanently block the config-mismatch
optimizer. The source-superset check in SegmentBuilder::update cancelled
every rebuild that found the deleted vector still in old segment files,
and each retry cancelled again, so optimizations got stuck forever.
Removing the check (as in #9609) would fix delete but reintroduce data
loss for the CreateVectorName race. Instead, tell the two cases apart
with the live collection schema: prune a source vector that is gone from
the schema (a real deletion), but cancel when it is still present (a
freshly created vector this optimizer has not yet seen). This is safe
because the schema is persisted before the op reaches segments, and the
live schema is read after the source segments are frozen.
The live set covers dense and sparse vectors, since a segment stores both
together. When no live source is wired in, the conservative always-cancel
behavior is kept.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: wire live vector names into edge optimizers
Deleting a named vector left the edge path with the pre-fix behavior:
segment_optimizer_config hardcoded live_vector_names to None, so a merge
touching a segment that still carried the deleted vector cancelled, and
EdgeShard::optimize() propagated the cancellation as a hard error forever.
Share the shard config behind an Arc and hand the blocking optimizers a
provider that reads the current vector names on every call. Same safety
argument as the server wiring: update() holds the segments read guard
across both the segment application and the config update, so any name a
frozen source segment carries is visible to the live read.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: share vector-name enumeration via CollectionParams::vector_names
The optimizer's live-schema set and the WAL-recovery valid-name set are
the same dense+sparse enumeration and must stay in lockstep; a drift
between them would reintroduce a wrong prune/cancel decision. Replace
the private helper in optimizers_builder and the inline block in WAL
recovery with a single CollectionParams method.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: drop SegmentOptimizer::live_vector_names forwarding hop
The default trait method only forwarded to the config getter and had a
single caller; ShardOptimizationStrategy now reads the config directly,
removing one layer of indirection.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(edge): load ReadOnlyEdgeShard segments in parallel
Open each segment on a dedicated thread during initial open and refresh,
reducing follower startup time for shards with many segments.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(edge): resolve clippy type_complexity in parallel segment load
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(edge): run per-segment reads on a configurable thread pool
Replace the per-segment sequential read loops and the spawn-a-thread-per-segment
loader with a single fixed-size rayon thread pool owned by each shard.
- Add EdgeConfig::max_search_threads (Option<usize>, None = CPU-derived default
matching the core search runtime via common::defaults::search_thread_count).
- Build a long-lived pool in EdgeShard and ReadOnlyEdgeShard; reuse it for
parallel segment loading on open/refresh instead of std::thread::spawn.
- Add EdgeReadView::par_map_segments as the single seam that runs per-segment
work on the pool; use it in search, scroll, count, facet and rescore-formula.
Each task mints its own HardwareCounterCell from the shared accumulator.
- Expose max_search_threads through the builder and the Python binding (+ stub).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(edge): return error instead of panicking on search pool creation
ThreadPoolBuilder::build() can fail (e.g. thread spawn / resource exhaustion).
This runs during EdgeShard open/load and ReadOnlyEdgeShard follower open, so a
transient failure must not abort the process. Propagate it as an OperationError
through the existing OperationResult-returning constructors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: read-only edge shard follower (ReadOnlyEdgeShard)
Add a read-only follower of EdgeShard, mirroring the segment-level
ReadOnlySegment + live_reload design one level up at the shard. A leader
process owns a read-write EdgeShard; one or more followers open the same
on-disk directory as a ReadOnlyEdgeShard, serve reads, and periodically
refresh() to pick up the leader's flushed writes and optimizations.
Shared read logic lives once in a crate-internal EdgeReadView<H>, generic
over a ReadSegmentHandle: the follower is monomorphized over the concrete
ReadOnlySegment<S> (no dynamic dispatch), while the read-write shard's
heterogeneous Segment/ProxySegment holder uses the LockedSegment enum
(the only dyn ReadSegmentEntry left, mirroring LockedSegment::get_read).
EdgeShardRead is the public read API: it exposes the read methods
(search/query/retrieve/count/facet/scroll/info/...) as default methods,
requiring implementors only to provide read_segments() + config_snapshot().
EdgeShard keeps its inherent read methods for backwards compatibility.
Segment discovery is injected via a SegmentEnumerator (temporary seam):
LocalSegmentEnumerator scans the segments/ directory for the local/mmap
case; S3 followers supply their own. This will be replaced by an on-disk
segment manifest in follow-up work.
shard: add LockedSegment::get_read_arc(); split retrieve_blocking into a
handle-collecting wrapper + generic retrieve_over; generalize the private
_read_points over the segment type (public signatures unchanged); remove
the now-unused read_points_locked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor
* feat: manifest-based segment loading for ReadOnlyEdgeShard
Replace the read-only follower's directory-scan discovery with the segment
manifest from the leader, the proper mechanism the SegmentEnumerator seam
was a placeholder for.
shard: add SegmentsManifest::load so readers can read manifest.json.
edge (leader): EdgeShard now writes segments/manifest.json (gated by the
write_segment_manifest feature flag), initialized from the live segment
set on new/load and refreshed after optimize() swaps segments.
edge (follower): ManifestSegmentEnumerator reads the manifest's `active`
segments instead of scanning, and open_mmap uses it. It requires a manifest
and errors when none is present (no silent scan fallback); discover by
scanning explicitly via open() with a LocalSegmentEnumerator instead.
Since the manifest only reports ready segments (and future versions will
mark segments retiring/under-construction rather than removing them from
active immediately), the read path no longer races with the leader, so the
defensive `is_transient_open_error` skip handling is removed — a failure to
open a reported segment is now a genuine error and propagates.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Emj8TFxdtrf3K32eWhGgor
* feat: edge-s3-scroll experiment binary for ReadOnlyEdgeShard over S3
Adds a standalone `edge-s3-scroll` binary that opens a ReadOnlyEdgeShard
directly over an S3 (or S3-compatible) bucket and runs a single scroll
request with a hard-coded filter, for experimentation.
Takes the bucket endpoint, credentials and key prefix as CLI flags/env
vars, builds an S3-backed BlobFs, and discovers segments via a custom
enumerator that reads the segment manifest over object storage. The
edge_config.json is fetched to a local temp dir because
ReadOnlyEdgeShard::open reads config from the local filesystem.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* filter config in s3 scroll tool
* fix: read segment manifest from new location after dev rebase
dev #9564 moved the segment manifest next to (rather than inside) the
segments/ directory and named it segments_manifest.json. Adapt the
read-only follower enumerators accordingly so the follower reads the
manifest where the leader now writes it:
- ManifestSegmentEnumerator reads via segment_manifest_path() (shard root)
while still resolving segment dirs under segments/<uuid>.
- The S3 scroll tool's enumerator mirrors the same layout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: read-only edge shard works over object storage without edge_config
Make ReadOnlyEdgeShard self-sufficient over arbitrary backends (S3/GCS),
and harden the read-only segment loading it depends on.
- ReadOnlyEdgeShard derives its config from the segments via
EdgeConfig::from_segment_config instead of requiring edge_config.json
(open and refresh both derive); a follower never has that file.
- ReadOnlyEdgeShard::open(fs, path) no longer takes an enumerator: a
read-only follower always discovers segments through the manifest.
open_with_enumerator remains as a pub(crate) seam for tests.
- ManifestSegmentEnumerator is generic over UniversalReadFs (reads the
manifest via read_json_via), so it works over local mmap or a blob/S3
backend; the tool's bespoke enumerator is removed.
- Read-only mutable ID tracker tolerates absent mappings/versions files
(they are not written while empty), matching MutableIdTracker::open:
files are opened lazily and NotFound is treated as empty, avoiding an
extra exists() round-trip on object storage.
edge-s3-scroll experiment tool:
- Supports AWS S3 / S3-compatible and GCS backends (--backend), with a
hard-coded-free --filter-key/--filter-value scroll filter.
- Reads segment data through a DiskCache so remote blocks are fetched
once and served from a local mirror afterwards.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: remove unused SegmentsManifest::load
Its only caller (edge's ManifestSegmentEnumerator) now reads the manifest
via read_json_via over UniversalReadFs, so the local-only load helper is
dead. Drop it and its now-unused read_json/Path imports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(edge): expose WAL options via EdgeShard::load_with_wal_options
Adds a sibling method to EdgeShard::load that accepts custom
WalOptions, alongside the existing load() which keeps using
default_wal_options().
Motivation: embedded/mobile deployments (e.g. Flutter plugins on
iOS/Android) need much smaller WAL segments than the 32 MiB default —
typically 4 MiB. On filesystems with sparse files (APFS/ext4/F2FS)
the physical footprint is small, but the visible/reported size is
the full segment capacity, which is surfaced by iCloud backup, OS
size pickers, etc.
Changes:
* New EdgeShard::load_with_wal_options(path, config, wal_options).
* EdgeShard::load delegates to it with default_wal_options() — no
behavior change for existing callers.
* ensure_dirs_and_open_wal now takes WalOptions explicitly; called
from both EdgeShard::new (with default) and load_with_wal_options.
* WalOptions re-exported from edge crate root.
* Two regression tests in lib/edge/tests/wal_options.rs.
WAL options are intentionally a runtime parameter, not part of
EdgeConfig (which is persisted to edge_config.json) — different
processes may legitimately open the same shard with different WAL
options.
* test(edge): verify mismatching WAL options on reload preserve data
Addresses upstream review feedback (qdrant/qdrant#9067): demonstrate
that reloading an existing shard with WAL options different from the
ones it was created with is safe and preserves all previously written
points.
Two new tests in lib/edge/tests/wal_options.rs:
* reload_with_smaller_wal_capacity_after_upsert:
create shard with default 32 MiB WAL -> upsert point 42 ->
drop -> reload with 4 MiB WAL options -> point 42 still readable ->
upsert point 43 under smaller WAL -> count == 2.
* reload_with_larger_wal_capacity_after_upsert (symmetric):
create -> reload with 4 MiB -> upsert point 100 -> drop ->
reload with default 32 MiB -> point 100 readable -> upsert
point 101 -> count == 2.
Both pass. The WAL crate does not validate segment_capacity on
reload — existing segments on disk keep their original size,
subsequent appends honor the runtime options. This confirms the
PR description's claim that WalOptions is a runtime hint, not
persisted shard state.
* style(edge): cargo fmt for wal_options.rs mismatch tests
* refactor(edge): replace load_with_wal_options with builder-based options
Addresses upstream feedback (timvisee, generall): the sibling
constructor pattern doesn't scale as runtime configurability grows.
Replaces `EdgeShard::load_with_wal_options(path, config, wal_options)`
with `EdgeShard::load_with_options(path, config, EdgeShardOptions)`,
where `EdgeShardOptions` is a builder-style runtime-options struct.
* `EdgeShardOptions::new().with_wal_options(opts)` is the equivalent
of the old call.
* Adding new runtime options later (e.g. wal flush interval, initial
indexing threshold) is now an additive change — new builder method
on `EdgeShardOptions`, no new constructor variant on `EdgeShard`.
* `EdgeShardOptions` is intentionally not persisted (no
Serialize/Deserialize). The reload-mismatch tests above confirm
that WAL options are a runtime hint, not shard identity, so they
don't belong in `EdgeConfig` (which is persisted to
`edge_config.json`).
* `EdgeShard::load(path, config)` unchanged for existing callers —
delegates to `load_with_options(..., EdgeShardOptions::default())`.
Tests in `lib/edge/tests/wal_options.rs` migrated to the new shape;
all four still pass:
test load_with_options_accepts_custom_wal_capacity ... ok
test load_still_works_with_default_wal_options ... ok
test reload_with_smaller_wal_capacity_after_upsert ... ok
test reload_with_larger_wal_capacity_after_upsert ... ok
* refactor(edge): add fluent builders; move wal_options into EdgeConfig
Replaces the EdgeShardOptions side-channel with a single EdgeConfig that
carries wal_options inline, and introduces fluent builders for the three
user-facing config types.
* WalOptions now derives Clone/PartialEq/Eq/Serialize/Deserialize so it
can live inside EdgeConfig and round-trip through edge_config.json.
* EdgeConfig.wal_options (Option<WalOptions>) replaces EdgeShardOptions.
EdgeShard::load_with_options is folded into EdgeShard::load; both new
and load drive the WAL from config.wal_options.unwrap_or_default().
* New builders/ module hosts EdgeConfigBuilder, EdgeVectorParamsBuilder,
and EdgeSparseVectorParamsBuilder. Each builder has explicit per-field
storage and constructs its target via an exhaustive struct literal in
build(), so adding a field to the target forces a compile error in the
builder.
* publish example switched to the new builder API.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style(edge): cargo fmt after builder refactor
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Refactor: `SearchSegmentEntry` for search ops
* Move `has_deferrend_points_method`
to `SearchSegmentEntry`
* Reorg imports
* Renamings per review
* fixup! Renamings per review
* `StorageSegmentEntry` trait
It contains methods dealing with storage and syncronization
* Move index methods from `ReadSegmentEntry`
to `SegmentEntry`
* Add `_concurrent` suffix to some methods
* move field index methods into NonAppendableSegmentEntry
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* refactor(edge): split EdgeShard load into new() and load()
- new(path, config): create edge shard only when path has no existing segments
- load(path, config?): load from existing files; config optional (load from
edge_config.json or infer from segments)
- Helpers: has_existing_segments, ensure_dirs_and_open_wal, resolve_initial_config,
load_segments, ensure_appendable_segment
- Python: EdgeShard.create_new(path, config); __init__ still calls load()
- Examples use new() for creation and load(path, None) for reopen
- Error instead of panic on config mismatch; explicit load/create in Python
- fix: use OptimizerThresholds.deferred_internal_id for dev compatibility
Made-with: Cursor
* rollback threshold changes
* fix(edge): address dancixx review comments
- Persist inferred config in load() when edge_config.json does not exist
(SaveOnDisk::new only saves when file exists)
- Python: add #[new] delegating to load() for backward compatibility
- qdrant_edge.pyi: Optional return types for deleted_threshold,
vacuum_min_vector_number, default_segment_number; add __init__ doc
Made-with: Cursor
* Revert "fix(edge): address dancixx review comments"
This reverts commit 370e21a6c74c41210e1658f89814395cb86ed81c.
* fix: optional returns
* fix: save config it is not exist
* fix: save data on disk always
* fix: python examples
* chore: remove edge.close()
* fix: wal lock
* fix: rename of EdgeShardConfig -> EdgeConfig
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Daniel Boros <dancixx@gmail.com>
* refactor(edge): let edge re-export segment/shard/sparse
* chore(ast-grep-rules): handle path like `::shard::blah::Blah`
This commit was required during the refactoring. It turns out we don't
need it right now, but it might be useful in the future. Or not.
* refactor(edge-rs): Rehaul public API
* Introduce EdgeShardConfig for edge shard
- Add EdgeShardConfig and EdgeOptimizersConfig in lib/edge/src/config.rs
- Segment config (vector_data, sparse_vector_data, payload_storage_type)
- Global hnsw_config and per-vector HNSW in segment config
- Optimizer params: deleted_threshold, vacuum_min_vector_number,
default_segment_number, max_segment_size, indexing_threshold,
prevent_unoptimized (excludes memmap_threshold, flush_interval_sec,
max_optimization_threads)
- Persist/load as edge_config.json in shard path
- EdgeShard uses RwLock<EdgeShardConfig>; load() accepts Option<EdgeShardConfig>,
falls back to file or infer from segments; compatibility checked on load
- load_with_segment_config() for backward compatibility (SegmentConfig -> EdgeShardConfig)
- optimize() uses EdgeShardConfig for hnsw and optimizer thresholds
- Public methods: set_hnsw_config(), set_vector_hnsw_config(), set_optimizers_config()
(update and persist)
- Python and examples use load_with_segment_config with existing config API
Made-with: Cursor
* Refactor EdgeShardConfig: user-facing params only, config module
- Replace SegmentConfig inside EdgeShardConfig with user-facing fields:
- on_disk_payload (bool) instead of payload_storage_type
- vectors: HashMap<VectorNameBuf, EdgeVectorParams> with on_disk per vector,
no per-vector quantization; global quantization_config only
- sparse_vectors: HashMap<VectorNameBuf, EdgeSparseVectorParams> with on_disk
- EdgeVectorParams / EdgeSparseVectorParams use on_disk (bool) instead of
storage_type; conversion to VectorDataConfig/SparseVectorDataConfig in
to_segment_config()
- Add config module: mod.rs, optimizers.rs, vectors.rs, shard.rs
- from_segment_config(&SegmentConfig) fills all inferrable params
- to_segment_config() builds SegmentConfig for segments and optimize()
- load_with_segment_config takes Option<SegmentConfig>, uses from_segment_config
Made-with: Cursor
* Move optimizer threshold helpers to shard crate
- Add get_number_segments, get_indexing_threshold_kb, get_max_segment_size_kb,
get_deferred_points_threshold_bytes in shard::optimizers::config
- Collection OptimizersConfig and edge EdgeOptimizersConfig delegate to these
- Single place for threshold logic; collection and edge use shard helpers
Made-with: Cursor
* Use destructuring in config conversions to avoid missing new fields
- EdgeVectorParams: destructure VectorDataConfig in from_*, destructure self in to_vector_data_config
- EdgeSparseVectorParams: destructure SparseVectorDataConfig and SparseIndexConfig in from_*, destructure self in to_sparse_vector_data_config
- EdgeShardConfig: destructure SegmentConfig in from_segment_config, destructure self in to_segment_config
Adding new fields to source structs will now cause compile errors until conversions are updated.
Made-with: Cursor
* refactor: centralize on_disk_payload→payload_storage_type, on_disk→storage_type, and appendable quantization logic
- PayloadStorageType::from_on_disk_payload(bool) in segment (Mmap/InRamMmap)
- VectorStorageType::from_on_disk(bool) in segment (ChunkedMmap/InRamChunkedMmap)
- QuantizationConfig::for_appendable_segment(Option<&Self>) in segment (feature flag + supports_appendable)
- collection: use from_on_disk_payload in non-rocksdb branch
- edge shard/vectors: use new helpers; remove duplicated conditionals
- shard optimizers: use from_on_disk and for_appendable_segment
Made-with: Cursor
* refactor(edge): use EdgeShardConfig directly, drop segment_config
- Add plain_segment_config() for create_appendable_segment (no HNSW)
- Add segment_optimizer_config() built from EdgeShardConfig for blocking optimizers
- Add vector_data_config(name) for query/MMR
- build_blocking_optimizers: use segment_optimizer_config() instead of SegmentConfig
- create_appendable_segment: use plain_segment_config()
- search/query: use config().vectors and vector_data_config() instead of segment_config()
- Remove segment_config() from EdgeShardConfig and EdgeShard
- Add to_plain_vector_data_config on EdgeVectorParams
Made-with: Cursor
* [manual] review changes
* refactor(edge-py): wrap EdgeShardConfig, add EdgeVectorParams/EdgeSparseVectorParams
- PyEdgeConfig now wraps EdgeShardConfig (vectors, sparse_vectors, on_disk_payload, etc.)
- PyEdgeVectorParams / PyEdgeSparseVectorParams wrap edge config types
- PyEdgeOptimizersConfig for optional optimizer settings
- EdgeShard.load() uses EdgeShardConfig; edge::config made pub for Python crate
- cargo fmt + clippy (remove map_identity)
Made-with: Cursor
* refactor(edge-py): simplify config API, remove unused Py* types, add EdgeConfig
- Remove unused PyPayloadStorageType, PyVectorDataConfig, PyVectorStorageType,
PySparseVectorDataConfig, PySparseVectorStorageType from Python bindings
- Move PyEdgeOptimizersConfig to lib/edge/python/src/config/optimizers.rs
- Update qdrant_edge.pyi: EdgeConfig with vectors/sparse_vectors,
EdgeVectorParams, EdgeSparseVectorParams, EdgeOptimizersConfig
- Update examples (common.py, repr.py) to use new config API
- Run cargo fmt
Made-with: Cursor
* [manual] review changes
* [manual] review changes
* [manual] fix test
* Address CodeRabbit review comments for PR 8322 (#8324)
* Address CodeRabbit review comments for PR 8322
- Python examples: explicit imports (repr.py, common.py) and new EdgeConfig API
- HnswIndexConfig: add max_indexing_threads param and property in .pyi and Rust bindings
- EdgeConfig: make vectors optional for sparse-only configs; validate at least one of vectors/sparse_vectors
- EdgeShardConfig::load: use try_exists(), propagate I/O errors
- from_segment_config: infer hnsw_config from per-vector HNSW when all agree
- EdgeShard setters: atomic clone-mutate-save-then-replace; persist config save errors
- Segment compat: prefix vector name in error messages; resolve None datatype to Float32
- max_indexing_threads: preserve 0 (auto) sentinel in trait default; remove per-optimizer overrides
- SegmentOptimizerConfig:🆕 build plain and optimizer maps in single pass
- config_mismatch_optimizer tests: use VectorNameBuf::from() instead of .into()
- vectors.rs: doc updates for per-vector quantization
Made-with: Cursor
* Address @generall review: SaveOnDisk for config, resolve num_rayon_threads in optimizer
- Use SaveOnDisk<EdgeShardConfig> for EdgeShard config (generall: 'We have SaveOnDisk struct for this')
- Create via SaveOnDisk::new() after resolving config; setters use .write() for atomic persist
- set_vector_hnsw_config: clone then mutate then write (fallible setter)
- max_indexing_threads: resolve 0 (auto) via num_rayon_threads inside impl (generall: 'proper solution would be to resolve num_rayon_threads inside the optimizer impl')
- max_indexing_threads_sentinel_aware() now returns Some(num_rayon_threads(raw)) so callers get actual thread count
Made-with: Cursor
* [manual] reorganize num_rayon_threads -> get_num_indexing_threads to better account per-vector configuration
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
* update docstring and pyi
* fmt
* fmt
* clipy
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
Change pub mod to mod for count, facet, info, optimize, query,
retrieve, scroll, search, snapshots, and update in lib/edge.
The public API (EdgeShard, ShardInfo) is unchanged.
Made-with: Cursor
Co-authored-by: Cursor Agent <agent@cursor.com>
* Fixups of amalgamator
Fix issues that break `qdrant-edge` build process:
- `use … as segment;` - this causes `ast-grep` rules to replace wrong
paths. So, rename to avoid collisions.
- `#[macro_use]` and `extern crate` required be in the top-level
`lib.rs`.
- `format!("…", crate::something::…)` - `ast-grep` can't fix paths
inside macros. Fixed by moving `crate::something::…` out of the macro.
* Add lib/edge/publish workspace and amalgamation script
* Move `lib/edge/examples` into `lib/edge/publish/` workspace
And fix them to use the generated `qdrant-edge` crate.
* Add github workflow
* Cleanup `qdrant-edge` public API
Removes empty modules. Checked by `cargo doc`.
* Split `SegmentEntry`
Add `ImmutableSegmentEntry` for operations that can be applied to
immutable segments, and make the `SegmentEntry` as it subtrait.
* Rename to `NonAppendableSegmentEntry`
It differs semantically from `ImmutableSegmentEntry` by allowing point
deletion. Move point deletion to the trait too.
* Fix docstring
* use NonAppendableSegmentEntry where possible
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Replace `Option<Segment>` with `enum LoadSegmentOutcome`
* Replace some Path/PathBuf with str/String
* Rename field Segment::{current_path -> segment_path}
* safe_delete
* Timeout on point retrieve when acquiring segment holder
* review: less radical and cleaner
* highly embarassing I hope no one sees this!
* coderabbit
* no manual impl for Serialize and Hash
* new feedback query
* construct query scorer
* simplify formula
* better comment
* make it easier to add other feedback formulas
* rename query -> target
* rename in async raw scorer
* rename formula -> strategy
* better comments
* rename to Simple strategy
* clippyyyyy
* oops, reenable conditional compilation
* Fundamental refactor to precompute pair score
* rename FeedbackScorer -> FeedbackQuery
This type implements `Query` trait, not `Scorer`.`
