upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-07-30 06:30:57 -05:00
Code Issues Packages Projects Releases Wiki Activity
6,603 Commits 337 Branches 116 Tags
dev
Commit Graph

3 Commits

Author SHA1 Message Date
Andrey Vasnetsov
fb681b7d9d Lazy roaring flags bitmap and bool index counts (#9749) 
* [AI] make ReadOnlyRoaringFlags bitmap and bool index counts lazy

Opening a read-only segment scanned every flags file end to end:
`ReadOnlyRoaringFlags::open` materialized the whole RoaringBitmap via
`iter_ones()`. Every payload field carries a null index, so this was paid
per field per segment, for bitmaps most queries never touch.

Make the bitmap a `OnceLock`, filled by a scan on first access. Open now
reads only the tiny status file. `ReadOnlyBoolIndex`'s three eager count
fields collapse into one lazily-derived, cached `BoolCounts`; its
`live_reload` refreshes them in place when present and leaves them unset
otherwise, so reloading an index nothing queries stays scan-free.

Propagate the resulting `OperationResult` through `RoaringFlagsRead`,
`PayloadFieldIndexRead::count_indexed_points`, `FieldIndexRead`,
`PayloadIndexRead::{indexed_points, get_telemetry_data}`, `build_info` /
`build_telemetry` and `SegmentEntry::{info, get_telemetry_data}`, out
into shard, edge and collection.

`ram_usage_bytes` stays infallible: an unmaterialized bitmap holds no
RAM, so it reports 0 via the new `bitmap_if_materialized`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* [AI] correct `preopen` comment: `open` no longer scans the flags file

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* [AI] fix edge examples for fallible `info()`

`EdgeShardRead::info` now returns `OperationResult<ShardInfo>`. The
examples live in their own workspace (lib/edge/publish), so the main
`cargo check --workspace` never saw them.

Every call site sits in `fn main() -> Result<(), Box<dyn Error>>`, so
propagate with `?`. `bm25-search` compiled either way but would have
printed the `Result` rather than the `ShardInfo`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-09 17:48:40 +02:00
Andrey Vasnetsov
729dc6af43 Make EdgeConfig tunables optional with a layered fallback chain (#9714) 
Tunable EdgeConfig parameters (on_disk_payload, hnsw_config, optimizers)
are now Option, and every tunable resolves through the fallback chain
provided -> persisted -> derived from segments -> default when loading
an existing shard. Leaving a parameter unspecified keeps the shard as it
is; an explicit value overwrites it and existing segments converge to it
through the optimizers.

vectors/sparse_vectors are excluded from overwrite semantics: an empty
map inherits the persisted/segment-derived definitions, a non-empty map
is validated for compatibility against the loaded segments (size,
distance, multivector, datatype, sparse modifier) and fails the load on
mismatch.

The derived layer folds over all segments in UUID order instead of
taking an arbitrary first segment, so a plain appendable segment (which
carries no HNSW parameters) can never mask an indexed segment's actual
build parameters. Previously a lost edge_config.json could resolve
unspecified HNSW params to compiled-in defaults and silently trigger a
full re-index via ConfigMismatchOptimizer.

The read-only follower accepts an optional config on open: provided
tunables are applied once over the segment-derived config (vectors
always come from the segments), and refresh re-derives from segments
alone.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-07 11:45:02 +02:00
Sasha Denisov
797779a159 feat(edge): expose WAL options via EdgeShard::load_with_wal_options (#9067) 
* feat(edge): expose WAL options via EdgeShard::load_with_wal_options

Adds a sibling method to EdgeShard::load that accepts custom
WalOptions, alongside the existing load() which keeps using
default_wal_options().

Motivation: embedded/mobile deployments (e.g. Flutter plugins on
iOS/Android) need much smaller WAL segments than the 32 MiB default —
typically 4 MiB. On filesystems with sparse files (APFS/ext4/F2FS)
the physical footprint is small, but the visible/reported size is
the full segment capacity, which is surfaced by iCloud backup, OS
size pickers, etc.

Changes:

* New EdgeShard::load_with_wal_options(path, config, wal_options).
* EdgeShard::load delegates to it with default_wal_options() — no
  behavior change for existing callers.
* ensure_dirs_and_open_wal now takes WalOptions explicitly; called
  from both EdgeShard::new (with default) and load_with_wal_options.
* WalOptions re-exported from edge crate root.
* Two regression tests in lib/edge/tests/wal_options.rs.

WAL options are intentionally a runtime parameter, not part of
EdgeConfig (which is persisted to edge_config.json) — different
processes may legitimately open the same shard with different WAL
options.

* test(edge): verify mismatching WAL options on reload preserve data

Addresses upstream review feedback (qdrant/qdrant#9067): demonstrate
that reloading an existing shard with WAL options different from the
ones it was created with is safe and preserves all previously written
points.

Two new tests in lib/edge/tests/wal_options.rs:

* reload_with_smaller_wal_capacity_after_upsert:
  create shard with default 32 MiB WAL -> upsert point 42 ->
  drop -> reload with 4 MiB WAL options -> point 42 still readable ->
  upsert point 43 under smaller WAL -> count == 2.

* reload_with_larger_wal_capacity_after_upsert (symmetric):
  create -> reload with 4 MiB -> upsert point 100 -> drop ->
  reload with default 32 MiB -> point 100 readable -> upsert
  point 101 -> count == 2.

Both pass. The WAL crate does not validate segment_capacity on
reload — existing segments on disk keep their original size,
subsequent appends honor the runtime options. This confirms the
PR description's claim that WalOptions is a runtime hint, not
persisted shard state.

* style(edge): cargo fmt for wal_options.rs mismatch tests

* refactor(edge): replace load_with_wal_options with builder-based options

Addresses upstream feedback (timvisee, generall): the sibling
constructor pattern doesn't scale as runtime configurability grows.

Replaces `EdgeShard::load_with_wal_options(path, config, wal_options)`
with `EdgeShard::load_with_options(path, config, EdgeShardOptions)`,
where `EdgeShardOptions` is a builder-style runtime-options struct.

* `EdgeShardOptions::new().with_wal_options(opts)` is the equivalent
  of the old call.
* Adding new runtime options later (e.g. wal flush interval, initial
  indexing threshold) is now an additive change — new builder method
  on `EdgeShardOptions`, no new constructor variant on `EdgeShard`.
* `EdgeShardOptions` is intentionally not persisted (no
  Serialize/Deserialize). The reload-mismatch tests above confirm
  that WAL options are a runtime hint, not shard identity, so they
  don't belong in `EdgeConfig` (which is persisted to
  `edge_config.json`).
* `EdgeShard::load(path, config)` unchanged for existing callers —
  delegates to `load_with_options(..., EdgeShardOptions::default())`.

Tests in `lib/edge/tests/wal_options.rs` migrated to the new shape;
all four still pass:

  test load_with_options_accepts_custom_wal_capacity ... ok
  test load_still_works_with_default_wal_options ... ok
  test reload_with_smaller_wal_capacity_after_upsert ... ok
  test reload_with_larger_wal_capacity_after_upsert ... ok

* refactor(edge): add fluent builders; move wal_options into EdgeConfig

Replaces the EdgeShardOptions side-channel with a single EdgeConfig that
carries wal_options inline, and introduces fluent builders for the three
user-facing config types.

* WalOptions now derives Clone/PartialEq/Eq/Serialize/Deserialize so it
  can live inside EdgeConfig and round-trip through edge_config.json.
* EdgeConfig.wal_options (Option<WalOptions>) replaces EdgeShardOptions.
  EdgeShard::load_with_options is folded into EdgeShard::load; both new
  and load drive the WAL from config.wal_options.unwrap_or_default().
* New builders/ module hosts EdgeConfigBuilder, EdgeVectorParamsBuilder,
  and EdgeSparseVectorParamsBuilder. Each builder has explicit per-field
  storage and constructs its target via an exhaustive struct literal in
  build(), so adding a field to the target forces a compile error in the
  builder.
* publish example switched to the new builder API.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* style(edge): cargo fmt after builder refactor

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 17:06:49 +02:00