mirror of
https://github.com/qdrant/qdrant.git
synced 2026-07-26 20:51:08 -05:00
797779a159
* feat(edge): expose WAL options via EdgeShard::load_with_wal_options Adds a sibling method to EdgeShard::load that accepts custom WalOptions, alongside the existing load() which keeps using default_wal_options(). Motivation: embedded/mobile deployments (e.g. Flutter plugins on iOS/Android) need much smaller WAL segments than the 32 MiB default — typically 4 MiB. On filesystems with sparse files (APFS/ext4/F2FS) the physical footprint is small, but the visible/reported size is the full segment capacity, which is surfaced by iCloud backup, OS size pickers, etc. Changes: * New EdgeShard::load_with_wal_options(path, config, wal_options). * EdgeShard::load delegates to it with default_wal_options() — no behavior change for existing callers. * ensure_dirs_and_open_wal now takes WalOptions explicitly; called from both EdgeShard::new (with default) and load_with_wal_options. * WalOptions re-exported from edge crate root. * Two regression tests in lib/edge/tests/wal_options.rs. WAL options are intentionally a runtime parameter, not part of EdgeConfig (which is persisted to edge_config.json) — different processes may legitimately open the same shard with different WAL options. * test(edge): verify mismatching WAL options on reload preserve data Addresses upstream review feedback (qdrant/qdrant#9067): demonstrate that reloading an existing shard with WAL options different from the ones it was created with is safe and preserves all previously written points. Two new tests in lib/edge/tests/wal_options.rs: * reload_with_smaller_wal_capacity_after_upsert: create shard with default 32 MiB WAL -> upsert point 42 -> drop -> reload with 4 MiB WAL options -> point 42 still readable -> upsert point 43 under smaller WAL -> count == 2. * reload_with_larger_wal_capacity_after_upsert (symmetric): create -> reload with 4 MiB -> upsert point 100 -> drop -> reload with default 32 MiB -> point 100 readable -> upsert point 101 -> count == 2. Both pass. The WAL crate does not validate segment_capacity on reload — existing segments on disk keep their original size, subsequent appends honor the runtime options. This confirms the PR description's claim that WalOptions is a runtime hint, not persisted shard state. * style(edge): cargo fmt for wal_options.rs mismatch tests * refactor(edge): replace load_with_wal_options with builder-based options Addresses upstream feedback (timvisee, generall): the sibling constructor pattern doesn't scale as runtime configurability grows. Replaces `EdgeShard::load_with_wal_options(path, config, wal_options)` with `EdgeShard::load_with_options(path, config, EdgeShardOptions)`, where `EdgeShardOptions` is a builder-style runtime-options struct. * `EdgeShardOptions::new().with_wal_options(opts)` is the equivalent of the old call. * Adding new runtime options later (e.g. wal flush interval, initial indexing threshold) is now an additive change — new builder method on `EdgeShardOptions`, no new constructor variant on `EdgeShard`. * `EdgeShardOptions` is intentionally not persisted (no Serialize/Deserialize). The reload-mismatch tests above confirm that WAL options are a runtime hint, not shard identity, so they don't belong in `EdgeConfig` (which is persisted to `edge_config.json`). * `EdgeShard::load(path, config)` unchanged for existing callers — delegates to `load_with_options(..., EdgeShardOptions::default())`. Tests in `lib/edge/tests/wal_options.rs` migrated to the new shape; all four still pass: test load_with_options_accepts_custom_wal_capacity ... ok test load_still_works_with_default_wal_options ... ok test reload_with_smaller_wal_capacity_after_upsert ... ok test reload_with_larger_wal_capacity_after_upsert ... ok * refactor(edge): add fluent builders; move wal_options into EdgeConfig Replaces the EdgeShardOptions side-channel with a single EdgeConfig that carries wal_options inline, and introduces fluent builders for the three user-facing config types. * WalOptions now derives Clone/PartialEq/Eq/Serialize/Deserialize so it can live inside EdgeConfig and round-trip through edge_config.json. * EdgeConfig.wal_options (Option<WalOptions>) replaces EdgeShardOptions. EdgeShard::load_with_options is folded into EdgeShard::load; both new and load drive the WAL from config.wal_options.unwrap_or_default(). * New builders/ module hosts EdgeConfigBuilder, EdgeVectorParamsBuilder, and EdgeSparseVectorParamsBuilder. Each builder has explicit per-field storage and constructs its target via an exhaustive struct literal in build(), so adding a field to the target forces a compile error in the builder. * publish example switched to the new builder API. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style(edge): cargo fmt after builder refactor Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>