602 Commits

Author	SHA1	Message	Date
Andrey Vasnetsov	74f3e85b94	Bump version to 1.19.0 (#10084) ... * Bump version to 1.19.0 * Update missed cherry picks * Add OpenAPI spec for v1.19.x Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 13:48:45 +02:00
Andrey Vasnetsov	c34acc1a36	Global quota API (#10035) ... * feat: global quota API Memory and disk are node-wide resources, so configuring their thresholds per collection through strict mode makes little sense. Move them behind a single cluster-wide `QuotaManager`. The quota config is seeded from `storage.quotas` in the settings (and so from env vars), overridden by `quota.json` in the storage directory, and updated cluster-wide through a new `SetQuotaConfig` consensus operation which rewrites that file on every peer. Raft snapshots carry it too, so a peer that joins by snapshot picks it up. Quotas are enforced wherever the strict mode memory and disk checks used to run, but no longer gated behind `strict_mode.enabled`: a value set in an enabled strict mode config still wins per resource, the quota is the default. Rejections name both the condition that tripped and the config that governs it. `GET /quotas` reports the config plus current utilization to global read users; `PUT /quotas` replaces it for global manage users. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: cover the quota endpoints in the API consistency checks `test_all_rest_endpoints_are_covered` and the OpenAPI endpoint count both break on any new REST endpoint. Add `GET`/`PUT /quotas` to `ACTION_ACCESS` with their JWT access tests, and bump the expected API count. The quota endpoints stay out of `REST_ENDPOINT_WHITELIST`: that list is for data-plane endpoints reported per-endpoint in metrics. Also add a Raft snapshot CBOR compatibility test — snapshots are exchanged between peers of different versions during a rolling upgrade, so `quota_config` must be absent-tolerant in both directions. Review feedback: persist through `SaveOnDisk`, which already implements the write-before-swap protocol this was doing by hand; validate the config at both persistence boundaries, since a hand-edited quota file or a config arriving through consensus does not pass the REST handler's validation, and a `0%` limit would reject every update forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: assert seeding a quota from invalid settings persists nothing Follow-up to review feedback claiming `SaveOnDisk::load_or_init` writes the init value before it is validated. It does not — only `SaveOnDisk::new` persists — but the property matters: were seeding to persist first, invalid settings would leave a `quota.json` that fails validation on every subsequent start, and the node could only be recovered by deleting it by hand. Pin it down with a test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: make QuotaManager the single reader of memory and disk The quota checks measured memory and disk themselves, while the optimizer and the WAL disk watcher each called `fs4::available_space` behind their own ad-hoc caches. Fold all of it into QuotaManager: it owns the readings, the freshness policy, and the limits they are compared against. Moves the module to `lib/shard`, since the optimizer sits below `storage` and has to reach it; `storage::quota` re-exports it, so consensus, the `/quotas` API and StorageConfig are unchanged. The manager is installed as a process singleton by TableOfContent, ahead of loading any collection. - Callers hand in QuotaLimits overrides instead of a StrictModeConfig, and an override can now only tighten. A collection-level admin could raise `max_disk_usage_percent` past a cluster-wide limit that needed global manage rights to set; ties resolve to the quota so the rejection names the knob that actually has to change. - Measurements are cached for 5s, but a reading at or above its limit is never reused: a rejected client retries, and freeing the resource has to take effect on the next request rather than a TTL later. - `fits_on_disk` sizes an optimization against physical free space only, never the configured limits. Optimizations are what free a full disk, so the quota must not be what stops one. - `percent_of` widens to u128 instead of saturating the multiply, which under-reported utilization (failing open) above ~184 PB. - StorageConfig::quotas is optional; absent means no quota is enforced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: don't recover dead replicas onto a node at a resource limit Recovering a dead replica pulls a whole copy of its shard onto this node. If it is already at its memory or disk quota that transfer cannot finish, and starting it only pushes the node further past the limit. Skip it and reconsider on a later sync, once the resource frees up. Adds QuotaManager::check_capacity for work that lands bytes here without being an update. Unlike fits_on_disk the configured limits do apply: taking on a replica is not what frees a full node, so there is no deadlock to avoid by letting it through. The check is hoisted out of the per-shard loop because a node over its limit re-measures on every call, so checking per dead shard would cost a statvfs each. It is free when no quota is configured. Also trims the comments across the quota module, which had grown well past what the code needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: drop trivial and duplicated quota tests Six tests removed, ~140 lines, with no loss of coverage: - a_rejection_names_the_knob_that_has_to_change asserted that a format! contains its own literals; the message is covered end-to-end by the override test and by test_global_quota.py. - a_node_over_its_quota_has_no_capacity_to_take_on_a_replica was 30 lines for check_capacity, a one-line delegation to check_update the test above it already calls. - a_rejecting_measurement_is_never_served_from_the_cache duplicated the meter test, which proves the same rule with an injected reader instead of inferring it from the real filesystem. - free_space_is_reported_without_enforcing_anything covered a one-line accessor, and its point is what the fits_on_disk test is for. - The two resolve tests and the three meter tests each collapse into one. DiskFit::Unknown keeps its coverage as two lines inside the fits_on_disk test rather than its own fixture. The snapshot compat pair becomes one test: the second only asserted cluster_metadata.is_empty(), which says nothing about quotas — the real check was the deserialize, now an expect that states it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: re-measure free space as the disk fills, and drop a Windows-only assert Two CI failures, both from this branch. e2e test_low_disk: the DiskUsageWatcher I replaced escalated to checking on every call once free space fell below 512 MB. Folding it into the quota manager lost that — available_bytes passed no limit, so a reading was reused for the full 5s however little space was left. On a disk filling as fast as that test fills it, 5s blind is enough to actually run out and the WAL write dies instead of returning "No space left on device". available_bytes now takes a watch_below level and never reuses a reading under it, which is what the old ladder was expressing. The watcher passes max(min_free, 512 MB), so the escalation point is back; above it the 5s cache still costs fewer syscalls than the old 128-call ladder. fits_on_disk gets the same rule by passing required_bytes, so a merge that does not fit re-checks rather than sitting on a stale sample. Windows: fits_on_disk on a missing path was asserted to be Unknown, but GetDiskFreeSpaceEx resolves up to the containing drive and succeeds — as common::disk_usage's own test documents. Dropped; the branch is a two-line else and is not portably reachable. Also renames an_optimization_is_sized_against_the_disk_not_the_quota, which needed explaining to be understood. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: report the quota config in telemetry Reads it from the quota manager rather than the settings, so it is the config the node is actually enforcing: a peer that missed a consensus update reports what it is applying, not what the cluster agreed on. Gated on global access, the same access `GET /quotas` requires, and left out of `PeerTelemetry` — a quota is per-node state, so each peer reports its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: regenerate OpenAPI, and cover the quota in the telemetry key sets Two CI failures from the previous commit. Referencing QuotaConfig from TelemetryData moves its definition earlier in `components/schemas`, because TelemetryData is generated ahead of QuotaStatus. Regenerated rather than hand-patched, so the schema is a pure move. test_telemetry_detail asserts the exact set of top-level telemetry keys. The quota is reported at every level, including 0 — it is three scalars, it is the default the endpoint serves, and it is what explains an update being rejected — so both key sets gain it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: remove `max_disk_usage_percent` from strict mode Disk is a node-wide resource, so a per-collection percentage of it never meant anything a caller could act on: the limit describes how full the *node* is, and which collection the write happens to target has nothing to do with it. The global quota is where it belongs. It shipped in 1.18.2 without documentation, so this drops it outright rather than deprecating. Removal is soft in every direction: StrictModeConfig has no `deny_unknown_fields`, so a client still sending it gets it ignored rather than a 400, and the same struct deserializes the persisted collection config, so collections created on 1.18.2+ keep loading. Proto field 22 is reserved so the number is never reused. The e2e test becomes a quota test — the fixture and the timing are the interesting parts and they carry over unchanged; only how the threshold is configured differs. `max_resident_memory_percent` was documented and stays for now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: enforce the strict mode memory limit outside the quota `max_resident_memory_percent` was folded into the quota as an override, which meant the quota check had to know about strict mode, and retiring the setting would mean unpicking `EffectiveLimit` and `LimitSource` from the resolution logic. It is now a check of its own in `verification/mod.rs`, next to the strict mode checks it belongs with, borrowing only the measurement from the quota manager — which stays the node's single reader of process memory, so both checks still share one reading. Deleting the setting later is deleting one function and its one caller. `QuotaManager::check_update` takes no arguments and consults the quota alone. A collection can still only tighten the limit for itself, because its own check runs in addition rather than in place of the quota's, and each rejection now names the config that has to change without having to carry a `LimitSource` to say so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: enforce the quota on the update path, not in strict mode The quota check sat inside `check_strict_mode_toc_batch` only because that was the one place holding the collection's strict mode config. It doesn't need one any more, and the placement had a real cost: coverage depended on each handler remembering to ask for a strict mode check, and four of the internal update RPCs do — `sync_internal`, which moves the most bytes onto a node, does not. It now runs in `Collection::update_from_client` and `update_from_peer`, which every update passes through. `update_from_client` checks ahead of the shard split, so an operation is accepted or refused whole rather than landing on some shards and being refused by others. Classification moves with it, from ~10 `consumes_memory` impls on request DTOs to one exhaustive `CollectionUpdateOperations::consumes_quota`. The internal enum has variants — raw upserts, conditional upserts, the syncs — that have no client-facing request type, so per-DTO impls structurally could not classify them. Shard-transfer syncs stay excluded, as they are today: a transfer is sized up once before it starts, and refusing its batches partway abandons work that is nearly done only for it to restart from the beginning. Index and named-vector creation reach shards through consensus, past this check — a peer must not refuse what the cluster agreed to — so they keep their pre-consensus check, now against the quota manager directly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: deprecate `max_resident_memory_percent` in strict mode Same reason the disk threshold went: memory is node-wide, so a per-collection percentage of it caps how full the *node* is, which has nothing to do with which collection is being written to. The node-wide quota caps it once for everything. Unlike the disk threshold this one shipped documented, in 1.18.0, so it keeps working — as a limit a collection can tighten for itself, never lift — and gets the usual markers: `#[deprecated]` on both Rust structs, `[deprecated = true]` on proto field 21, and `deprecated: true` in the OpenAPI schema, which schemars derives from the attribute. The note names 1.21 as the removal. Recording a version matters here: the audit in docs/plans/overdue-deprecations.md found that this repo has never written a removal deadline down, and members of the 1.15.0 deprecation batch are still in tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: reconcile the quota readers with #9891 #9891 landed effective (cgroup) figures in telemetry while this branch was making QuotaManager the single reader of memory and disk. Two collisions, neither of which git sees. `segment::utils::mem::total_memory_bytes` is now a shared accessor with a 5s TTL, so a cgroup resize is picked up. The quota module had its own `OnceLock` copy that froze the value at startup — exactly what #9891 set out to fix — so it delegates to the shared one instead. Telemetry's new `disk_size` called `common::disk_usage::disk_usage` directly. That reader lost its TTL cache on this branch when the caching moved into the quota manager's meter, so it would have taken an uncached `statvfs` on every telemetry request, and it put a second disk reader back in the tree. It goes through `QuotaManager::disk_capacity_bytes` now, sharing the reading the quota check already takes. Verified it still reports the storage filesystem, matching `df`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: split the quota manager by what each half does `manager.rs` had grown to 450 lines holding three separate jobs: owning the config and its file, taking the readings, and comparing one against the other. - `manager/store.rs` — the `Store` enum, `QUOTA_CONFIG_FILE`, and config validation, which is now the store's own business rather than something every caller has to remember to do first. - `manager/measure.rs` — every reading, and `DiskFit`. The "nothing else calls `statvfs` or reads process RSS" claim is now checkable by looking at one file. - `manager/enforce.rs` — `check_update` / `check_capacity` and the threshold comparison. - `manager/mod.rs` — the struct, its construction, and the config accessors: what a reader needs to see first. Tests move with their subject. No behaviour change: `set_config` used to validate before delegating to the store, and now the store validates on write, which is the same order of operations from the outside. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: count copy-on-write deletes toward the quota Dropping a vector or a payload key does not free anything on its own: copy-on-write rewrites the point to produce the version without that field, so storage grows first and is only reclaimed once the optimizer gets to it. Gating those as if they were reclaiming space let a full node keep taking writes that make it fuller. Deleting whole points stays exempt. That is the one operation that has to work on a node at its limit, or there is no way back under it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: say that the quota's reported usage is per node `GET /quotas` returns one cluster-wide config and one set of utilization figures, which reads as though both describe the cluster. They do not: memory and disk are node-local, so `usage` is whatever the peer that served the request is seeing, and a peer under its limit says nothing about the others. Also corrects `resident_memory_percent`, which claimed to be a share of total system memory. It is a share of the memory available to the process, which under a cgroup is the limit rather than the host's RAM. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: treat a node over its quota as a failed replica, not a bad request A quota rejection described the request as invalid (400) and was classified non-transient, which is how the replica set recognises errors that every replica would produce alike. A quota is the opposite: the input is fine and the answer depends on which machine you ask. On the default `wait=false` path that combination silently dropped the write — `update.rs` only deactivates transient failures when nothing completed — leaving the replica Active and permanently missing data its co-replicas had. It is now `InsufficientStorage`, transient, HTTP 507 / gRPC `ResourceExhausted`. So a node that is out of room is handled like one that is offline: - last active replica, or every replica over quota: nothing could take the write, and the client is told the cluster is out of room. - more than one replica: the full node is deactivated through the same path a dead peer takes, and the update stands if enough replicas accepted it. `check_capacity` already keeps recovery off that node until it has room. The check also moves off `update_from_client`, which applied the coordinator's own limit to the whole operation even when it held no replica of the shards being written. Each replica set now gates its own local write and records the refusal as a failure of this peer, so a node only ever answers for itself. `ResourceExhausted` is shared with rate limiting, and the reverse conversion mapped it straight to `RateLimitExceeded` — a forwarded rejection came back as 429. Statuses now carry a marker so the two stay distinguishable across the wire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: report quota pressure per node, and across the cluster A quota is node-local, so finding out which node has hit one meant asking each of them in turn — and nothing at all showed up in monitoring. `/metrics` gains a `quota_exceeded` gauge for the local node. It is emitted only while the quota is enabled: with it off the value would be a constant 0 that says nothing about the node, and an alert built on it would go quiet rather than fire if someone disabled the quota. Telemetry's `quota` field carries the same verdict alongside the config, since that is where the metric is derived from. `GET /quotas` now answers for the whole cluster. A new `GetQuotaUsage` RPC on the internal `QdrantInternal` service returns what one peer is using, and the handler fans it out to every known peer in parallel. Peers that do not answer are left out rather than failing the request — the nodes that are out of room are exactly the ones most likely to time out, and a partial answer still names them. Outside distributed mode the field is absent rather than a map of one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: report the quota metric per resource `quota_exceeded` was one flag for the whole node, which does not say what to go and fix — disk is freed by deleting or optimizing, memory by unloading. It now carries a `resource` label: quota_exceeded{resource="memory"} 0 quota_exceeded{resource="disk"} 1 A resource with no limit gets no series at all, for the same reason the metric is absent while the quota is disabled: a series that can never reach 1 reads as healthy and would quietly carry an alert that cannot fire. `QuotaManager::exceeded` returns the per-resource verdict, with `None` for a resource this node does not cap. Telemetry reports the same breakdown, since the metric is derived from it. The peer usage RPC keeps a single flag — it sits next to both percentages, so it only has to answer "is this peer refusing writes". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: drop a no-op error conversion the linter caught `check_global_access` already returns a `StorageError`, so mapping it through `StorageError::from` converted the type to itself and tripped `clippy::useless_conversion`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: hold a tripped quota until usage clears a release margin A resource resting on its limit crosses it in both directions on the noise between two readings, and each crossing is expensive: the node refuses a write, its replica is deactivated, usage dips, recovery starts sending a whole shard copy back, and the arriving data pushes it over again. The loop sustains itself, and every lap costs a shard transfer. A limit now trips at its configured value but only clears once usage has fallen 5 percentage points below it, so the crossing has to be real. The margin is floored at 1%, since a limit smaller than the margin would otherwise be impossible to fall back under and would strand the node. The verdict is carried on the manager rather than recomputed, which makes it the thing reporting shows: expect `exceeded` to be set while the utilization next to it is already back under the limit. Rejections say so too, rather than claiming a limit that is no longer exceeded: Disk usage is at 87% of total capacity. It reached the configured limit of 90% and has to fall below 85% before this node takes writes again. Changing the config clears the verdicts. New limits are a deliberate act, and should not be held back by the margin of a limit that no longer exists. Both resources are now evaluated on every check instead of stopping at the first failure, so a verdict is never left behind reporting a reading that has since been superseded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: make the quota release margin configurable 5 points is a guess about how noisy a deployment's usage is, which is not something one number can be right about: a node whose disk moves in gigabyte steps needs a wider margin than one that creeps, and an operator who wants the old flip-on-every-reading behaviour should be able to ask for it. `release_margin_percent` joins the rest of the quota config, so it seeds from `QDRANT__STORAGE__QUOTAS__RELEASE_MARGIN_PERCENT`, replicates through consensus, and changes with `PUT /quotas`. Defaults to 5 and is filled in when a request omits it, so it always answers with the margin actually in force rather than leaving the caller to assume one. `0` releases as soon as usage is back under the limit. `QuotaConfig` grows a hand-written `Default` for it, since deriving one would have quietly defaulted the margin to 0 and disabled the hysteresis for anyone constructing a config in code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: leave the release margin unset by default, and hold verdicts in atomics `release_margin_percent` is `null` unless someone sets it, rather than materialising 5 into every config. A quota written today then does not pin a number a later release may want to revise, and `{"enabled": false}` still round-trips as itself. `QuotaConfig::limits` resolves it, next to `enabled`, so enforcement never sees the unset case. The verdicts move from a `Mutex<QuotaExceeded>` to one `AtomicBool` per resource. They are judged independently and nothing reads them as a pair, so the lock only added contention to the path every update takes; a verdict that races a concurrent check is re-decided by the next one from a fresh reading. That also drops the tri-state. Only "was this over its limit" has to survive between checks — whether a resource is enforced at all follows from the config and the reading, so it is derived when reporting rather than stored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: drop a comment arguing with a design that was never here Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 11:18:45 +02:00
xzfc	3d4f9c2f69	cargo +nightly fmt (#10057)	2026-08-04 11:18:45 +02:00
xzfc	efd3bfd617	bitpacking_ordered: batched reads (#10038) ... * bitpacking_ordered: batched reads * Replace ranges with pairs * "unsufficient" -> "insufficient" --------- Co-authored-by: Luis Cossío <luis.cossio@outlook.com>	2026-08-04 11:18:45 +02:00
Luis Cossío	22ac6bcb91	remove Clone implementation for MmapFile (#10047)	2026-08-04 11:18:45 +02:00
Luis Cossío	ed3325a115	[UIO] 2-stage DiskCache::reopen (#10031) ... * AI: implement 2 stage reopen * manual: simplification refactor * AI: simplify further * upd trait interface * use closure instead of `&CachedReadFs` * open a new remote for the tail fetch * rename to `cached_file_info` * only resize after fallible op * use consistent remote openoptions	2026-08-04 11:18:44 +02:00
xzfc	ca3fe18f16	Remove dead code (#10030) ... * Remove dead code * Remove unused dependencies * `allow(dead_code)` -> `expect(dead_code)` * ast-grep: rule-tests/*-test.yml => tests/*-test.yml For brevity. * ast-grep: forbid allow(dead_code)	2026-08-04 11:18:44 +02:00
xzfc	8e2e6b5f4c	chore: bench_cache (#10028)	2026-08-04 11:18:44 +02:00
Andrey Vasnetsov	a11f8bb4ae	feat: io_uring setting to control which components use the io_uring backend (#10008) ... * feat: `io_uring` setting to control which components use the io_uring backend A few components have both an mmap and an io_uring variant reading the very same files: the immutable dense vector storages, the single-file TurboQuant storage, and the mmap payload storage. Until now the choice was a side effect of `async_scorer` — a vector-search knob — plus, for the payload storage, a feature flag that was parked off because io_uring is ~2x slower than mmap when the data fits the page cache (#9310, #9409). Add `storage.performance.io_uring`, optional, with two modes: - unset (default): unchanged behaviour. The vector storages keep following `async_scorer`; the payload storage stays on mmap. - `disabled`: no component uses io_uring. - `auto`: a component uses io_uring when its memory placement is `cold` (data is left on disk, so reads hit the disk and there is something to gain), its feature flag allows it, and the kernel supports io_uring. Components meant to sit in RAM keep using mmap. The decision lives in one place, `segment::common::io_uring::use_io_uring`, so the openers no longer each reach for the async-scorer global. Kernel support is now probed up front through `is_io_uring_supported()` instead of opening a file and falling back on error. `async_payload_storage` now defaults to on: it no longer decides anything by itself, it only lifts the ban, and the payload storage no longer follows `async_scorer` at all — so turning it on cannot silently move an existing `async_scorer: true` deployment onto the slower path. Which backend a component ended up on depends on the config, the placement and the kernel at once, so report it in `SegmentInfo`: `vector_data[name].io_backend` and `payload_storage_io_backend`, both `"mmap" | "io_uring"`, absent for components that have no such choice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Trim comments, drop trivial tests Two tests were only restating their own implementation: `test_mode_round_trip` round-tripped the encode/decode pair next to it, and `test_io_uring_config` checked that serde deserializes a two-variant enum. The mode matrix test stays, it is the one that pins the semantics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Flatten `IoBackend` in OpenAPI, derive `JsonSchema` for `IoUringMode` Per-variant doc comments on a plain string enum make schemars emit a `oneOf` of anonymous single-value objects instead of a flat `enum`. Move the variant descriptions into the enum doc, as `Memory` and friends already do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Update lib/segment/src/vector_storage/turbo/turbo_vector_storage.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Update lib/segment/src/types.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * upd openapi schema * Update lib/common/common/src/flags.rs Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com> * Require kernel io_uring support in the async-scorer fallback `use_io_uring` returned `get_async_scorer()` verbatim when the `io_uring` setting is unset, so an enabled async scorer on a kernel without io_uring opened the io_uring storage, failed, and fell back to mmap with an error log per segment. Gate that branch on `is_io_uring_supported()` too, like `Auto` already is, so the component just stays on mmap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * upd openapi schema --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>	2026-08-04 11:18:44 +02:00
Arnaud Gourlay	47f5a57c5b	tests: skip clear_ram_cache eviction test on tmpfs (#10015) ... POSIX_FADV_DONTNEED cannot evict pages of a tmpfs file: the page cache is the backing store, so there is nothing to drop them to. On systems where /tmp is tmpfs (Ubuntu 24.10+ default) the test fails with all pages still resident. Detect tmpfs via statfs and skip. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-04 11:18:44 +02:00
Andrey Vasnetsov	5819d224da	fix: evict page cache of files that are mapped twice (#9984) ... * fix: evict page cache of files that are mapped twice `clear_cache()` on a `memory: cold` vector storage was a silent no-op: after optimization the whole storage stayed resident in the page cache. `MmapFile` opened with `need_sequential` holds two mappings of the same file (`MADV_RANDOM` + `MADV_SEQUENTIAL`), and `MADV_PAGEOUT` skips any page carrying more than one page-table reference. Any page faulted through both mappings was therefore never reclaimed, no matter which mapping was advised. Quantization is one way to get there: it reads the raw vectors through the sequential mapping while `populate_vector_storages()` populated the random one, so with quantization enabled `matrix.dat` stayed 100% cached after the build, and without it the same build evicted down to ~1%. `POSIX_FADV_DONTNEED` alone does not help either, as it skips pages with any page-table reference. So zap the page tables of both mappings first (`MADV_DONTNEED` on a shared file mapping only drops the PTEs; the data stays in the page cache and refaults on the next access) and then evict through the file. Dirty pages are still kept, exactly as before — `MADV_PAGEOUT` did not write back filesystem pages either — so callers that flush first, like `SegmentBuilder::build`, get a complete eviction. Also affects the payload storage (gridstore pages), the disk id tracker reader and quantized multivector offsets, which open with `need_sequential` too. Measured on a 200k x 256 build with quantization: `matrix.dat` 100% -> 0.0% resident, whole segment 67% -> 2.4%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: satisfy clippy::cast_lossless in the eviction test Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * MADV_DONTNEED should not be safe * Document why Madviseable::clear_cache might not be enough --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: xzfc <xzfcpw@gmail.com>	2026-08-04 11:18:43 +02:00
Tim Visée	71d99b144c	Add Logstore and Blobstore wrapper (#9673) ... * Gridstore: introduce storage operating mode in config Add a mode field to the gridstore config, selecting between the dynamic mode (current behavior, the default) and the upcoming serverless mode. The mode is specified through StorageOptions on creation, persisted in config.json, and read back first when opening so the correct variant can be selected automatically. Configs written before this field existed deserialize as dynamic. For now, selecting the serverless mode returns an error; the variant itself is added in follow-up commits. * Gridstore: move dynamic implementation into dedicated module Mechanical move of the current Gridstore implementation into gridstore/dynamic.rs as DynamicGridstore. The public Gridstore struct becomes a thin wrapper holding a mode variant enum, propagating every call into the selected variant. For now the enum only has the dynamic variant; the serverless variant is added in follow-up commits. No logic changes to the dynamic implementation itself: only visibility, the config parameter now passed into open (the wrapper reads it first to select the mode), and open_or_create staying on the wrapper. * Gridstore: add serverless tracker Add the append-only mapping tracker for the serverless storage mode. The tracker file is a plain array of 16-byte mapping entries without any header: the number of mappings is defined by the exact file length, and the entry index is the point offset. The file starts empty and only ever grows by appending, existing bytes are never rewritten. Mappings must be set in monotonically increasing point offset order; skipped offsets are backfilled as zeroed entries which decode as None. New mappings are buffered in memory and appended with a single write per flush. A flush with a stale target is a no-op so bytes are never written twice. A torn trailing entry (file length not a multiple of the entry size) is ignored when reading and truncated away when opening writable. Unlike the dynamic tracker, the file is read and written directly with positional file IO instead of memory mapping, as serverless environments do not handle memory mapped files well. * Gridstore: add serverless storage variant Add the append-only gridstore variant for serverless deployments, which restrict IO to appending to files: existing bytes can never be rewritten, and IO is expensive so as few files as possible are used. The variant stores all value data in a single page file next to the serverless tracker and the storage config, three files in total. Both data files start empty and only ever grow by appending; there is no preallocation, no used-block bitmask and no gap/region bookkeeping. Values are appended at put time at the next block aligned offset, with the zero padding included in the write so it lands exactly at the end of the file. Mappings are buffered and appended to the tracker with a single write per flush, after the page file is synced, so a mapping on disk never points at data that is not durable. Values cannot be updated or deleted, and must be put at monotonically increasing point offsets; violations are rejected before any data is written. Files are read and written directly, never memory mapped. The mode is selected through StorageOptions on creation and picked up automatically from the persisted config when opening. * Gridstore: serverless support in reader and view Extend the read-only GridstoreReader and the GridstoreView with the serverless mode, keeping both public types unchanged: like the writable Gridstore they now hold a mode variant internally, selected automatically from the persisted config when opening. The serverless reader holds the tracker and page directly and reads the files positionally, without memory mapping. A live reload re-reads the mapping count from the exact tracker file length (there is no size header), ignoring a torn trailing entry, and never truncates as it is read-only. Value reads always go directly to the file, so newly appended data is readable without remapping anything. * Gridstore: document storage operating modes * Gridstore: review fixes for the serverless mode Hardening and cleanup from a review pass over the new serverless storage variant: - Batch the reader side iteration like the writer already did, instead of materializing tracker mappings for the full range in one go, which could transiently allocate gigabytes on large storages. - Recover the append cursors when a positional write fails partway: truncate the file back to the tracked length so a retried append or flush never rewrites bytes that already landed in the file. - Validate page addressability before appending value data, a rejected put must not grow the page file. - Cross-check tracker and page consistency when opening: mappings that reference value data past the end of the page file (e.g. after a partial copy or restore) now fail fast instead of surfacing as opaque read errors per point. - Reject value pointers into any page other than page 0 on the serverless read path with PageNotFound, matching the dynamic mode contract, instead of silently reading from a wrong location. - Refresh the reported storage size on reader live reload even when no new mappings were flushed, unflushed value data may have grown the page file already. - Validate configs read from disk: a corrupt config with zero sized blocks, pages or regions is now rejected when opening instead of panicking on a division by zero later. - Classify rejected serverless puts as UnsupportedOperation, consistent with rejected deletes, so they don't surface as user-facing validation errors at the segment level. - Deduplicate the compression dispatch into Compression::compress and Compression::decompress, and the serverless file create/open patterns into shared direct IO helpers, so the two modes and files can't silently drift apart. * Gridstore: cover both operating modes in mode-agnostic tests Parameterize the gridstore tests that exercise mode-agnostic behavior over both the dynamic and serverless mode with rstest, using a single and bulk put/get roundtrips, storage files, basic persistence, corrupt config rejection, batched read congruence, reader live reload, and the different block sizes. Mode specific expectations branch inside the tests: expected file names, storage size semantics (whole blocks vs exactly packed bytes), value pointer layout (page spill over vs a single packed page), and gaps (created by deletes in dynamic mode, by skipped puts in serverless mode). Dynamic-only internals assertions are kept behind a mode check. Tests around updates, deletes, page spanning, block reuse and other dynamic-only behavior intentionally stay dynamic; the serverless specific format invariants remain covered by the dedicated serverless tests. * Gridstore: port serverless specific tests from sibling branch Source the serverless specific test cases that the serverless-gridstore-updates branch added, adapted to the dedicated variant implemented here (distinct file names, headerless tracker with 16 byte entries, a single packed page without trailing padding, and rejected re-puts): - writes only ever append: tracker and page files only grow and previously written bytes stay byte-for-byte untouched - new mappings land exactly at the end of the tracker file, which always covers the exact number of mappings - mapping gaps are zero-padded on disk and survive reopening - values are packed back to back at block aligned offsets, the page file ends exactly at the last value - serverless mode never creates nor reports block flag files - a flusher persists exactly the mappings that existed at its creation, later puts stay pending - a config claiming the wrong mode fails loudly in both directions instead of loading the incompatible file format of the other mode Tests around their mode switching, page spanning and tolerated deletes don't apply to this design and are intentionally not ported. * Gridstore: test serverless production risk scenarios Add tests for the operational aspects that matter before serverless mode goes to production, each covering a scenario that wasn't evaluated yet: - Replayed puts of already persisted offsets (a WAL redo after a crash where the flush completed but was never acknowledged) are rejected without appending anything, and max_point_offset is the exact offset a replay must resume at. - The accepted crash case of a tracker file extended with zeroed bytes: the entries count as permanent None mappings, can never be put again, and the storage stays consistent and writable past them. - The read-only reader never modifies the files: opening over a torn tracker tail, reading, iterating and live reloading leave both files byte-for-byte untouched. - A multi-round put/flush/reopen cycle always exposes exactly the flushed prefix, with the mapping count matching the exact tracker file length and unflushed offsets reusable. - An append beyond the maximum addressable block offset is rejected before writing anything, keeping retried puts from growing the page file unboundedly. * Gridstore: rename serverless mode to append-only, split into module Rename the mode after its defining characteristic instead of its deployment target: files only ever grow, existing bytes are never rewritten. Renames Mode::Serverless to Mode::AppendOnly (persisted as "mode": "append_only") and the on-disk file names to append_only_tracker.dat and append_only_page_0.dat. The serverless deployment motivation stays in the documentation. Also split the single 2300 line serverless.rs into an append_only module with dedicated files for the storage, page, view, reader and tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Use universal IO in Gridstore * Include upstream preopen logic in new Gridstore variant * Gridstore: buffer append-only value writes until flush In append-only mode, put previously wrote the value data to the page file right away, one write operation per put, while mappings were already buffered and batch persisted on flush. Buffer value writes the same way: both the value and its mapping now only land on disk once a flush cycle executes. This batches all new value data into a single write operation per flush, which is significantly more efficient on S3 based storage where every write is a costly operation. A flush now performs exactly two writes: one appending all buffered value data to the page file, one appending all pending mappings to the tracker file, in that order, so a mapping on disk never points at value data that is not durable. The page mirrors the tracker's pending mechanism: an in-memory buffer that is byte for byte the next append (zero padding between block aligned values included), a watermark captured at flusher creation so puts made during a flush stay buffered, a stale-flush no-op guard so appended bytes are never written twice, and truncate-back recovery on failed writes. Reads transparently serve buffered values from memory. As a side effect, a crash between flushes now leaves nothing on disk at all, where the write-through approach left orphaned value bytes in the page file. The buffered data is held in memory until the next flush, bounded by the flush cadence. Universal IO filesystem handles are now required to be Send + Sync, so the flusher closure can carry one to grow the page file at flush time; all existing backends already satisfied this. * Gridstore: rename inner DynamicGridstore to Gridstore The dynamic variant keeps the Gridstore name; the outer dispatching type will be renamed to Blobstore in a follow-up. Until then the inner type is referred to as dynamic::Gridstore to distinguish it from the outer type. * Gridstore: rename append-only variant to Arenastore The append-only variant stores all value data in a single ever-growing page, allocating space by appending, hence: arena store. * Gridstore: rename outer storage type to Blobstore The outer type dispatching between the two storage variants is now called Blobstore, being more generic than Gridstore. This frees up the Gridstore name, which now exclusively refers to the dynamic mode variant, next to Arenastore for the append-only variant. Storage components keep using the outer type, so they now use Blobstore. The gridstore crate name, GridstoreError, and the persisted names (config.json mode, payload config storage_type) are unchanged. * Gridstore: split Gridstore and Arenastore into dedicated modules The outer module is now blobstore, matching the Blobstore type it defines. The two storage variants each get their own submodule: the dynamic Gridstore moves from dynamic.rs into gridstore/ with its reader and view extracted from the shared files, mirroring the arenastore/ module (previously append_only/) which already had this layout. * Rename gridstore crate to blobstore The crate is named after the outer Blobstore storage type it provides. The gridstore name lives on in the dynamic mode variant. GridstoreError and the persisted names (config.json mode, payload config storage_type) are unchanged. * Arenastore: pack values back to back across multiple pages Drop the block alignment from the append-only mode: values are packed byte to byte, without blocks, and the tracker offset is now a plain byte offset within the page. Blocks and regions are dynamic mode concepts; their page size constraints no longer apply to append-only configs. Bring back support for multiple pages. Once appending a value would grow the current page beyond the configured page size, a new page is started, bounding the size of and the number of appends to each file: object stores like S3 Express limit the number of appends per object. A value larger than the page size gets a page of its own; values never span pages. A rollover creates the new, empty page file at put time; the value data itself stays buffered until the next flush, which appends to each touched page with a single write, using per-page watermarks captured at flusher creation. The reader scans for consecutively numbered page files when opening, validates the most recent mappings against them, and adopts pages created since on a live reload. * Blobstore: rename dynamic mode to mutable Rename Mode::Dynamic to Mode::Mutable, and the persisted config value with it: config.json now writes "mode": "mutable". There is no compatibility alias for "dynamic", released versions never wrote the mode field (a missing field still defaults to mutable), only unreleased storages did. The Gridstore type and module names for the mutable variant are unchanged. * Fix Edge compilation due to package rename * Review remarks * Extract Gridstore preopen into module * Rename Arenastore files * Use universal IO for append operations * Rename GridstoreError to BlobstoreError The error type belongs to the Blobstore crate and is shared by both the Gridstore and Arenastore variants, so it follows the crate naming. Also update the user-facing error messages that referred to the old name. * Split config into per-variant types * Rename Arenastore to Logstore Rename the Arenastore type to Logstore, including the reader, view, config, module and variant names. The storage file names follow: log_page_{n}.dat and log_tracker.dat. The persisted mode tag stays "append_only". * Move bitmask module into the Gridstore variant The bitmask tracks free blocks, which only exists in the mutable mode. Move the module from the crate root into the Gridstore variant that owns it. It stays re-exported at the crate root because the bitmask benchmark needs a public path. * Move pages module into the Gridstore variant Like the bitmask, the block based pages module is only used by the mutable mode. Move it from the crate root into the Gridstore variant that owns it. The Logstore variant has its own page implementation. * Use universal IO for every Logstore operation Replace the direct_io module with universal IO in the append-only tracker, making the whole Logstore go through a universal IO backend bounded by UniversalRead and UniversalAppend: - The tracker is generic over the backend now. Reads go through UniversalRead with the caller's access pattern, flushes land as one atomic append with the same offset compare-and-swap recovery as the pages: a retried append after a lost acknowledgement is adopted instead of appended twice. A torn trailing entry is still truncated away on writable open, through a fresh handle since shrinking is not supported through an open one. - The reader now schedules a prefetch for the tracker file too, it no longer bypasses the backend. - The config write, clear and wipe use the backend file operations instead of local filesystem calls, matching the Gridstore variant. * Batch reads in Logstore read_values Apply the same batching logic as the Gridstore variant: resolve all mappings first, then fetch the value data, both through the backend's read pipeline so async backends can serve the reads in parallel. The tracker gains a batched lookup mirroring the mutable tracker's iter, serving pending mappings and out of range point offsets directly from memory. The pages gain a batched value read; unflushed values are served from the in-memory buffers, and since values never span pages each value is a single read without reassembly. Like in the Gridstore variant, the callback may now be invoked in a different order than the requested point offsets. * Better describe logstore live reload ordering * use enum for options, swap `*Options`<->`*Config` naming * don't wrap enum in struct * ditch unused `StorageConfig`, make deserialization more ergonomic * rename `*Options`->`*Config` * make `preopen` non-blocking * fixup! ditch unused `StorageConfig`, make deserialization more ergonomic * fixup! use enum for options, swap `*Options`<->`*Config` naming * fixup! don't wrap enum in struct * fix rebase * use `populate` param in Logstore * test: failing repro of stale page after live reload across rollover A reader that live-reloads between a page rollover and the following flush adopts the new, still empty page. The previous page is then no longer the last one and is never reloaded again, so the tail that the next flush appends to it stays invisible to the reader forever: value pointer at byte 100 with length 100 is out of range AppendOnlyPages::live_reload only reloads the last held page, assuming earlier pages never change once a newer page exists. But the rollover creates the new page file eagerly at put time, while the previous page's buffered tail only lands at the next flush (see test_rollover_writes_no_value_data_before_flush), so a page can keep growing on disk after its successor exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: reload all pages that grew * use Fs in `open_or_create` * fix: publish tracker mappings only after the pages reload `AppendOnlyTracker::live_reload` observed the mapping count and made it visible in one step, before `LogstoreReader::live_reload` reloaded the pages. Every failure path in the page reload -- `list_files`, reopening a grown page, opening an adopted one, the truncation check -- therefore left the reader with mappings referencing value data it never loaded, so reads in the new offset range fail until a later reload happens to succeed. The edge refresh loop keeps a segment whose reload failed, expecting it to keep serving its pre-refresh state, which it then does not. Split observing from publishing: `reload_count` refreshes the handle and returns the count as a `PendingReload` token, `commit_reload` publishes it. The reader still observes the tracker first, as the writer persists pages before the mappings referencing them, but only commits once the pages are loaded. Reopening without committing is harmless: reads stay bounded by the unchanged count, and the bytes below it never change. A partial failure inside the page reload needs no unwinding, pages running ahead of the tracker is the safe direction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: batch the value reads in Logstore iteration `LogstoreView::iter_range`, the path behind `Logstore::iter` and `LogstoreReader::iter`, fetched the mappings for the whole range with a single read but then read the values themselves one at a time, serially. Gridstore routes its `iter` through `read_values` and pipelines both stages, so a full scan of an append-only storage was the one read path without batching -- one blocking round trip per value on the object store backends this variant exists for. It is reached by payload storage iteration and by the payload index build, which scans every payload. Feed the pointers into `read_batch_values` instead, keeping the single contiguous tracker read, which is better than the per-offset pipeline scheduling Gridstore does on that side. Values are now delivered through the read pipeline, so the callback may be invoked out of order, as it already could be for Gridstore's `iter` and for `read_values` in both variants. Both segment callers are order independent. Tests that happened to rely on the mmap backend completing reads in scheduling order now sort before comparing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: don't run the failed-page-reload test on Windows The test shrinks a page file out of band to make the page reload fail, but Windows refuses to resize a file while the reader holds it mapped, which it does by construction here: "the requested operation cannot be performed on a file with a user-mapped section open". The panic is on the injection itself, the code under test never runs. There is no portable injection. Truncating a page the reader holds is what the check under test detects, so the mapping cannot be avoided; failing the adopted page open instead needs a listed but unopenable file, and `local_list_files` descends into matching directories rather than listing them; failing the directory listing needs the storage directory removed, which Windows also refuses while pages are mapped. The storage itself is fine on Windows, its append path grows mapped pages there and every other Logstore test passes. The logic under test is platform independent and stays covered elsewhere, with the tracker half of the guarantee pinned by `test_live_reload`, which runs on every target. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Luis Cossío <luis.cossio@outlook.com>	2026-08-04 11:18:43 +02:00
xzfc	4645c498a9	Cleanup UniversalRead methods interface (#9934) ... * use common::generic_consts::{Random, Sequential}; * UniversalRead::read_batch: generic over E * UniversalRead::read_batch: pass `AccessPattern` as ZST arg * UniversalRead::read_bytes_iter: pass `AccessPattern` as ZST arg * UniversalRead::read_iter: pass `AccessPattern` as ZST arg * UniversalRead::read: pass `AccessPattern` as ZST arg * UniversalRead::read_bytes: pass `AccessPattern` as ZST arg	2026-08-04 11:18:42 +02:00
xzfc	a4a805acac	UioResult (#9933)	2026-08-04 11:18:42 +02:00
xzfc	8b7b7faee6	Batched ConditionChecker (#9740) ... * ConditionChecker::check_batched: trait + ConditionCheckerEnum * check_batched for OptimizedFilter * OnDiskPointToValues::values_iter_batch: improve performance * OnDiskPointToValues::values_iter_batch: update interface Accept bitvec, call on every point, pass UserData. * check_batched for geo index * geo_index tests: add same_geo_index_between_points_with_dups_test Catches broken load_from_on_disk/for_all_points_values. * check_batched for map index * check_batched for numeric index * check_batched for full-text index * tests for check_batched * Review fixups - default_check_batched: avoid running pred twice at boundary - condition_checker: use explicit match over matches!/if-else - Partitioner: use assert! over debug_assert! Co-authored-by: Luis Cossío <luis.cossio@outlook.com> * ConditionChecker::check_batched: &mut self -> &self * ConditionChecker::check_batched: make mandatory --------- Co-authored-by: Luis Cossío <luis.cossio@outlook.com>	2026-08-04 11:18:42 +02:00
Luis Cossío	eaa505ecdc	[io-bridge] split large reads into unordered chunks (#9896) ... * read large files in unordered chunks * use S3 error * use a vec of ranges to track scattering * self nits * use less concurrent chunks	2026-08-04 11:18:42 +02:00
Arnaud Gourlay	5191fc4f11	Tests: use SmallRng for RNG-bound test data generation in common (#9888) ... The persisted_hashmap and disk_cache tests generate their datasets with StdRng (ChaCha12 in rand 0.10). String keys draw one random_range call per character, so the crypto generator dominated the test runtime: test_k_str_* drop from ~3.3s to ~1.4s each with SmallRng (Xoshiro256++). Assertions are self-consistency checks (write then read back), so the changed sequences carry no retuning risk. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-04 11:18:40 +02:00
Luis Cossío	f189ef5cf3	Misc nits (#9894) ... * use `WithVector::is_enabled` * suppress unused var lint * fix non linux "useless mut" lint	2026-08-04 11:18:40 +02:00
Arnaud Gourlay	44cd82a1d1	Benches: use SmallRng instead of ChaCha12-based generators (#9887) ... * Benches: use SmallRng instead of ChaCha12-based generators All benchmarks used StdRng or rand::rng() (ThreadRng), both backed by the ChaCha12 block cipher in rand 0.10. Benchmarks do not need crypto-strength randomness, and several draw random values inside the timed closure, so cipher work was included in the measurement itself. Switch every bench target to SmallRng (Xoshiro256++), and key the HNSW graph cache and sparse index cache by RNG algorithm so stale caches built from the old generator are not reused against newly generated vectors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Benches: replace free-function rand::random with local SmallRng Addresses review: rand::random draws from the thread RNG (ChaCha12), including inside the timed loop of the pq score benchmark. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-04 11:18:40 +02:00
Andrey Vasnetsov	bd0048ea78	DiskIdTracker: RAM-resident is_uuid stored-bitmask sidecar + module split (#9878) ... * DiskIdTracker: RAM-resident is_uuid stored-bitmask sidecar + module split Move the is_uuid flags out of the i2e file into a separate id_tracker.is_uuid file in the compact StoredBitmask format (#9871), loaded whole into RAM as a RoaringBitmap on open and prefetched in preopen, so slot decoding never reads the flag from disk. Bump the on-disk format version to 2 (DiskIdTracker is unreleased; no migration). Add StoredBitmask::read_ones() in common to normalize any stored encoding into a bitmap of set positions, with logical_len validated against the u32 position space at open. Restructure disk_id_tracker: read_only.rs becomes read_only/{mod, lifecycle,live_reload,id_tracker_read} mirroring the immutable tracker, and reader.rs becomes reader/{mod,lifecycle,lookup,iter}. Also unbox the DiskMappingsRef iterators (impl Iterator instead of Box<dyn>), and make IdTrackerRead::iter_internal_versions fallible so the read-only disk tracker propagates storage errors instead of silently truncating; its implementation now reads the versions file in one pass (cleanup-on-open drains it anyway). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Split disk_id_tracker mod.rs into lifecycle / read / write files Mirror the read_only/ layout: mod.rs keeps the struct and resident-RAM helpers, lifecycle.rs the build/open paths, id_tracker_read.rs the DiskMappingsSource + IdTrackerRead impls, id_tracker.rs the mutable IdTracker impl. Code moved verbatim; only imports and a field doc touched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Tighten disk_id_tracker docstrings around guarantees State contracts (residency, laziness, error semantics, atomicity) instead of narrating which structures hold or use what. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-04 11:17:04 +02:00
Andrey Vasnetsov	eafec09a1e	Compact stored bitmask for on-disk field index deleted masks (#9871) ... * Compact stored bitmask for on-disk field index deleted masks Add StoredBitmask: a compact persisted bitmask written and read as a whole. The payload is a roaring bitmap of whichever bit value is the minority (mostly-0 and mostly-1 masks both stay tiny), falling back to raw dense bits when roaring would not be smaller, so the file is never larger than the dense representation. Files are replaced atomically via UniversalWriteFileOps::atomic_save; there is no in-place mutation. Use it for the write-once "no values" masks of the on-disk numeric, geo, map and full-text indexes, replacing the raw dense bitslice files sized at point_count/8 bytes regardless of content. Writing the new format is gated by the compact_bitmask feature flag (default off; enabled by `all` and `serverless_compatible`). Reading is format-agnostic regardless of the flag: the compact deleted_mask.bin is tried first, then the index-specific legacy file, so old segments stay readable and flag-off builds produce byte-identical legacy files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Split save_bitmask into named helpers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Regenerate OpenAPI spec for compact_bitmask feature flag Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review findings on stored bitmask - Avoid u64 overflow in the payload bound check when opening a mask with a corrupted payload_len. - Reject roaring payload positions beyond logical_len at read time, enforcing the BitmaskContent range contract for corrupted files. - Remove the opposite-format mask file after a successful save, so a rebuild with a flipped compact_bitmask flag can't leave a stale compact file shadowing the fresh legacy one (or an orphaned legacy file next to a compact one). - Make the compact-open numeric test tolerate builds that already wrote the compact format. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-04 11:17:03 +02:00
Luis Cossío	b2031d8aac	[UIO] Increase PHF prefetch (#9842) ... * increase phf prefetch * no duplicate comment Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com> --------- Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>	2026-08-04 11:17:03 +02:00
Tim Visée	7333e6092a	Universal IO: append to file (#9720) ... * universal_io: add UniversalAppend for atomic single-operation appends Growing a file previously took a separate set_len + reopen + write dance (bypassing universal_io and leaving a zero-filled window on crash), and there was no way to express appends for backends without random-offset writes. UniversalAppend::append grows the file by writing at the current end of file in one atomic grow+write operation and returns the offset at which the data landed; append_batch lands multiple buffers contiguously in as few operations as the backend allows. flusher() moves from UniversalWrite into a new UniversalFlush supertrait so append-only handles can require it without duplicating the method. Local backends, both single-syscall: - MmapFile appends through a dedicated O_APPEND fd (every write(2) / writev(2) is an atomic grow+write at EOF), then remaps via reopen(). Its flusher also fdatasyncs after appends, since msync alone does not persist file-size metadata. - IoUringFile appends via pwritev2(RWF_APPEND). O_APPEND is not an option there: on Linux, pwrite on an O_APPEND fd appends regardless of the given offset, which would break positioned writes on clones sharing the fd. Concurrent appenders are out of contract (single logical writer); object-store backends surface the new AppendOffsetConflict error and recover via reopen() + retry. * io_bridge: add AsyncWrite/AsyncAppend and appendable BlobFile Add the write-side backend traits reserved next to AsyncRead: AsyncWrite (create/remove/save) powers a UniversalWriteFileOps impl on BlobFs (create-or-truncate put, delete, atomic whole-object save; directory ops are no-ops), and AsyncAppend — a single-request append where the offset must equal the current object size, acting as a compare-and-swap token — powers UniversalAppend on BlobFile. BlobFile caches the object size across appends (one HEAD for N appends; a missing object counts as empty so the first append creates it), concatenates batches into a single request, and drops the cache on reopen() — the documented recovery path after AppendOffsetConflict. Its flusher is a no-op: appends are durable once the backend acknowledges them. * io_bridge_object_store: native single-request S3 append object_store has no append support, so issue the PutObject + x-amz-write-offset-bytes request ourselves, reusing the store's credential chain (AmazonS3::credentials) and object_store's SigV4 AwsAuthorizer, which signs every header present on the request — no hand-rolled signing and no direct reqwest dependency. The offset doubles as a compare-and-swap token: a mismatch (400 InvalidWriteOffset, or 412 on some S3-compatibles) maps to AppendOffsetConflict. The write-offset append API exists on AWS S3 Express One Zone directory buckets and compatible stores (e.g. MinIO AiStor) — plain S3 Standard buckets reject it, and real Express zonal endpoints / session auth are not verified yet; MinIO-AiStor-compatible stores are the primary target for now. GCS and Azure sources simply do not implement AsyncAppend. ObjectStoreSource carries an AppendContext (HTTP client + object URL base + signing region) built per backend from its config, and gains a generic AsyncWrite impl (single-put create/save, delete). A test-only multi-request CAS emulation over InMemory exercises the BlobFile append stack hermetically; an end-to-end flow against a real append-capable store is gated behind S3_APPEND_INTEGRATION_TEST=1. * simple_disk_cache: write-through UniversalAppend for DiskCache Append to the remote (the single grow+write operation), then write the same bytes through into the local mirror so tail reads do not re-fetch what was just uploaded. LocalState::append_local keeps the fetched bitmap accurate: blocks fully covered by the appended range are marked fetched, and the pre-append partial tail block — which resize() drops because set_len zero-fills its gap — is re-marked only when its prefix was already fetched. If the mirror turns out stale (the remote grew behind our back), append falls back to resize-only and lazy fetches heal the gap on the next read. Writeable opens are now allowed on DiskCacheFs solely to enable append; DiskCache still never implements UniversalWrite. The writeable flag propagates to the remote handle, which is opened buffered instead of O_DIRECT: appends write through the page cache, which O_DIRECT reads on the same fd would fight (and IoUringFile rejects appends on prevent_caching handles). The remote-immutability docs are relaxed to append-only with an immutable prefix, matching what reopen() already assumed. Includes a full-stack composition test: DiskCache write-through over BlobFile offset tracking over an in-memory object store. * io_bridge_object_store: build the append HTTP client lazily Opening a source from an AwsConfig eagerly built the reqwest client (TLS setup, connection pool) even when append was never used. Keep the AppendContext construction to pure config (allow_http flag, object URL base, signing region) and build the client on first append instead, cached in an Arc<OnceLock> shared across clones of the source — and thus across the file handles opened from it. Sources that never append now pay nothing; client-construction errors surface on the first append instead of at open. * universal_io: test that append grows the regular file on disk The conformance suite reads appended bytes back through universal-io handles; also assert the underlying regular file itself — created outside universal_io, verified with plain fs reads — for both local backends. * Mention why we use custom HTTP client, object_store crate has no support * io_bridge_object_store: reject appends unconfirmed by the size header A store without write-offset support may accept the signed PutObject as a plain put — replacing the object with just the appended bytes — and return 2xx (community MinIO did exactly this before 2025-05, commit minio/minio@6d18dba9). The old success path fabricated the new length when x-amz-object-size was missing, so the destruction stayed invisible while every subsequent append repeated it. Require the x-amz-object-size response header (returned by AWS and MinIO AiStor appends) for any append at offset > 0 and fail loudly without it. Offset-0 appends are equivalent to a whole-object write, so they remain valid either way — a misconfigured store now fails on the second append instead of never. * io_bridge_object_store: honor endpoint/region env vars for appends With AwsCredentials::Default the store is built via AmazonS3Builder::from_env, which honors AWS_ENDPOINT_URL_S3, AWS_ENDPOINT_URL, AWS_ENDPOINT, AWS_REGION and AWS_DEFAULT_REGION — but append_context derived the append URL and SigV4 region only from the typed config fields. An env-configured deployment would read from one host while signing and sending appends to https://{bucket}.s3.us-east-1.amazonaws.com. Resolve the append endpoint and region the same way build_store does: explicit config first, then (default credential chain only) the same environment variables, with AWS_ENDPOINT_URL_S3 taking precedence as in from_env. The resolution is a pure function over an injected env lookup so the test does not touch process-global environment state. * simple_disk_cache: delegate the append flusher to the remote DiskCache's UniversalFlush impl was an unconditional no-op, justified by object-store appends being durable on acknowledgement — but the impl is generic over any appendable remote, and for local remotes (MmapFile, IoUringFile, exactly the compositions the tests instantiate) that silently dropped the fdatasync the UniversalAppend contract requires: append, flush Ok, power loss, appended bytes gone. Delegate to the remote's flusher once the cache is materialized: local remotes get their sync, object-store flushers remain no-ops, and a never-materialized cache has made no appends so a no-op stays correct. * io_bridge_object_store: retry transient append failures The append RPC was a single unretried HTTP attempt, while every other request in this stack goes through object_store's retry layer — a routine transient 503 SlowDown or connection reset failed the append hard where a concurrent read would have silently recovered. Retry connection errors, 5xx and 429 up to three attempts with a short linear backoff, re-signing per attempt (the SigV4 signature embeds the request date). Retrying is safe because the write offset is a compare-and-swap; the one ambiguity — an attempt that landed but whose acknowledgement was lost — surfaces as a write-offset conflict on the retry, which is reconciled with a HEAD: under the single-writer contract, an object size of exactly offset + data_len proves the tail is ours, so the append reports success instead of a spurious conflict (whose reopen-and-retry recovery would duplicate the record). * universal_io: forward TypedStorage::flusher for any UniversalFlush The flusher forwarding lived in TypedStorage's S: UniversalWrite impl block, so append-only storages (DiskCache, BlobFile — UniversalAppend + UniversalFlush but not UniversalWrite) offered append through the wrapper while the durability flusher the append contract mandates was unreachable without going through .inner. Move it to an S: UniversalFlush block: UniversalWrite implies UniversalFlush, so existing callers resolve unchanged, and duplicating the method instead would have hit E0592 on backends implementing both — the very ambiguity UniversalFlush was extracted to avoid. * io_bridge_object_store: surface unbuildable append requests as errors Request building could panic on two reachable paths: url accepts URIs the http crate rejects (IPv6 zone identifiers, URIs beyond u16::MAX bytes), and AppendContext::new is public so the object URL base is not guaranteed to be a base URL. Both expects become S3Config errors, so a configuration edge case fails the append instead of panicking the thread driving the bridge runtime. * simple_disk_cache: don't fail appends the remote already committed append_impl committed to the remote first and returned Err when the subsequent local-mirror update failed (e.g. ENOSPC on the cache volume) — indistinguishable from "nothing was appended", so a retrying caller would duplicate the record on the remote. The mirror is cache maintenance, not part of the append: on a failed write-through, log and degrade to bare growth so lazy fetches heal the unmarked blocks (safe — blocks are only marked fetched after their bytes landed). Only an unresizable mirror still surfaces an error, and the UniversalAppend contract now documents that an append Err does not guarantee nothing was appended: reopen() and re-check the length before retrying. * universal_io: bounds-check positioned io_uring writes against EOF The UniversalAppend contract states that UniversalWrite::write beyond the end-of-file fails and append is the only growth path — mmap enforces it, but IoUringFile's write/write_batch/write_multi were unchecked pwrites that silently extended the file with a zero-filled hole, inflating subsequent append offsets. Check every positioned write against the file length (fstat once per call), matching mmap's OutOfBounds semantics, and generalize the regression test to run on both local backends including the batched path. * io_bridge: reject appends on handles opened without writeable BlobFs::open dropped OpenOptions entirely, so a BlobFile opened with writeable: false still accepted appends — mmap and DiskCache enforce the writeable requirement, the blob backend silently didn't, and a stray append through a nominally read-only handle would mutate a shared object. Thread OpenOptions::writeable into BlobFile and reject appends with PermissionDenied when it is unset, mirroring the other backends. Directly-constructed handles (BlobFile::new/open, which take no OpenOptions) remain writeable. With every backend now enforcing the flag, the UniversalAppend contract drops its 'where the backend enforces open modes' hedge. * simple_disk_cache: answer empty appends from the mirror An empty append still went through remote.append_batch, so it returned the remote's live end-of-file — which can diverge from what this handle's len() and reads observe when the remote grew behind our back — while leaving the stale mirror unhealed (unlike a non-empty append in the same state, which resizes). Accept empty appends early: return the mirror length without touching the remote at all, keeping the answer consistent with the handle's own view. The trait contract now spells out that empty appends return the handle's view of the end of file without growth I/O. * universal_io: grow the mmap in place after appends Every mmap append ended in a full reopen(): an open+fstat+close by path just to learn the new length, and — on the non-Linux fallback, which rebuilds the mapping with the open-time populate flag — a re-population of the ENTIRE file per append, making appends O(file size) for handles opened with Populate::Blocking. Populating after an append is pointless anyway: we just touched the data we wrote. Extract the remap machinery into remap_to() (reopen() keeps its exact semantics, populate included) and add grow_mapping(): a stat-free grow that never re-populates. Appends learn the new length from a single fstat on the already-open O_APPEND fd — kept rather than trusting the mapping length, which is stale exactly in the externally-grown-remote scenario the disk cache heals through lazy fetches (the foreign-growth tests catch the difference). The mirror's resize() passes the length it just set_len'd, dropping its stat round-trip entirely. Per small append this is write+fstat+mremap, down from write+open+fstat+close+mremap, with no populate anywhere. * universal_io: share the mmap append fd across clones The flusher captured the per-clone append_file at flusher-creation time, so a flusher obtained from a sibling clone — or created before the handle's first append — msynced the shared mapping's data pages but skipped the fdatasync that persists the appended file size: a half-persist where a crash loses the acknowledged tail even though a flusher ran after the appends (writer thread + long-lived flush-worker clone is exactly the natural WAL shape). Store the fd in an Arc<OnceLock> shared by all clones and read it at flush time instead of capture time: any clone's append makes every handle's flusher sync the size metadata, whatever the clone/flusher creation order. Initialization races between clones keep exactly one fd. No hot-path cost: reads and positioned writes never touch the cell, and the append path pays one atomic load next to its syscalls. The interior mutability also lets append_fd take &self. * universal_io: document the clone remap hazard truthfully The remap SAFETY comment claimed moving is safe "since we are holding &mut self" — which says nothing about clones: they share the mapping but keep their own raw ptr/len copies, so after a moving (or, on non-Linux, replacing) remap a sibling clone's next read dereferences an unmapped address. The trait contract understated the same hazard as a concurrent-read constraint, while the UB persists after append returns. State the contract once on MmapFile (clones must reopen before reading after any growth; a stale clone read is undefined behavior, not a stale view), correct the SAFETY argument to rely on it explicitly, annotate the as_bytes unsafe blocks that depend on it, and sharpen the UniversalAppend contract bullet accordingly. Making clones structurally safe (resolving ptr/len through the shared Arc) is deliberately left as a separate change. * universal_io: share the vectored append machinery between backends The IOV_MAX-chunking / EINTR-retry / WriteZero / advance_slices loop existed twice — as local_file_ops::write_all_vectored (mmap) and inlined around pwritev2 in IoUringFile::append_slices — along with a verbatim collect/cast/filter-empties preamble in both append_batch impls. Two copies of subtle short-write handling introduced by one branch will diverge the first time only one of them gets a fix. Add an io::Write adapter whose write_vectored issues pwritev2(RWF_APPEND), letting the io_uring append delegate to the shared write_all_vectored, and hoist the slice collection into local_file_ops::collect_append_slices. IOV_MAX becomes private to the one function enforcing it. No behavior change; the existing conformance tests (including the beyond-IOV_MAX batch) cover both backends through the shared path. * io_bridge_object_store: add s3_express to the test config helper The AwsConfig struct gained the s3_express field; update the resolve-endpoint test helper accordingly. * universal_io: run the append conformance suite over the S3 stack Promote the backend-generic UniversalAppend battery (offsets, batches across IOV_MAX, empty appends, read-after-append, reopen visibility, flusher) from a private test into universal_io::conformance, exposed under the testing feature so backend crates can run the identical suite. mmap and io_uring keep running it as before; the object-store bridge now runs it too, over BlobFs/BlobFile with the in-memory offset-CAS append emulation — so local file system and S3 append behavior are asserted by the same test. The real write-offset RPC remains covered by the gated test_native_append_flow integration test. * Swap order * universal_io: disambiguate the io_uring crate import The import reorder dropped the leading `::`, making `io_uring` ambiguous with this very module (pulled into scope by the `use super::*` glob) and breaking the build. * io_bridge: don't materialize the mock object on rejected appends MutableMockSource::append called get_or_insert_with before validating the offset, so a rejected stale append against a missing object left an empty entry behind (exists() flipping true) — a fidelity gap versus the real backends, where a rejected append has no side effects. Check the offset against the current length first and only materialize the buffer on a match. * universal_io: disambiguate the io_uring crate import Restore the leading `::` on the io_uring crate import — without it the name is ambiguous with this very module, which the `use super::*` glob pulls into scope, and the crate fails to compile. Matches the sibling files (pool.rs, runtime.rs), which already import via `::io_uring`. * io_bridge_object_store: treat 404 under a nonzero append offset as a conflict A missing object while the handle expected a nonzero end-of-file is a stale view (the object was deleted behind our back) — the same situation as an offset mismatch, with the same reopen-and-retry recovery, and it is exactly what the in-memory emulation and the io_bridge mock already report. The RPC path mapped every 404 to NotFound instead, so the three implementations disagreed on the same logical case. Keep NotFound for offset-0 appends, where a 404 is a genuine missing-target error (e.g. a missing bucket) that retrying cannot heal. * Preallocate vector * universal_io: disallow appends through the disk cache Appends must go directly to the backing storage (mmap, io_uring, S3) — the disk cache is strictly read-only again. Remove DiskCache's UniversalAppend/UniversalFlush impls and the mirror write-through machinery (LocalState::append_local), reject writeable opens at DiskCacheFs::open, and drop the writeable/prevent_caching plumbing that existed solely for cached appends, restoring read-only remote handles. Attempting to append through the cache is now a compile-time error (the trait impl no longer exists), and opening a cached handle writeable is rejected at runtime, covered by a test in each backend variant. * universal_io: make append idempotent via caller-supplied offset Append now takes the byte offset where the data must land: append(offset, data) -> Result<()>. Every backend validates that the offset equals the current end of file before writing (mmap and io_uring fstat the fd, object stores validate server-side via x-amz-write-offset-bytes); on mismatch nothing is written and the append fails with AppendOffsetConflict. Retrying an already-landed append therefore conflicts instead of appending twice, and recovery is re-deriving the offset from len(). BlobFile no longer tracks the object length locally; the store's own offset check is the compare-and-swap. * Review remarks * Validate file length in S3 append response * Fix linting, we don't mind a large enum variant on index builder * universal_io: conformance-test stale-handle append conflict recovery Promote the two-handle conflict scenario from the in-memory BlobFile test into the backend-generic conformance battery: a second writeable handle grows the file, the stale handle's append conflicts cleanly (the offset check runs against the file, not the handle's view), and the contract's documented recovery — reopen, re-check the length, append at the real end — lands the data exactly once. Now exercised over mmap, io_uring, and the object-store stack instead of only the in-memory emulation. * io_bridge_object_store: stub-server tests for append response handling The native append's HTTP state machine was only exercised by the gated live-store integration test (S3_APPEND_INTEGRATION_TEST=1), so none of its branches ran in CI. Cover them hermetically against a minimal local HTTP stub — one connection per canned response, no new dependencies: - the signed write-offset PUT, and the new-size validation on success (matching, mismatching, unparseable, and absent size headers — the absent case at offset zero and past it); - conflict mapping for 400 InvalidWriteOffset, 412, and 404 under a nonzero offset, with 404 at offset zero staying NotFound, and a 400 without the conflict code staying a plain error; - 429/5xx retries re-sending the same offset, giving up after MAX_ATTEMPTS, and the lost-acknowledgement reconciliation via HEAD (accepted when the object ends at offset + len, rejected otherwise); - the status + body excerpt on unexpected failures. * simple_disk_cache: statically assert the cache stays read-only Disallowing appends through the disk cache made them a compile-time error by removing the impls; pin that with assert_not_impl_any so the UniversalAppend/UniversalFlush/UniversalWrite impls cannot quietly return. Runtime rejection of writeable opens stays covered per backend variant.	2026-08-04 11:17:03 +02:00
Arnaud Gourlay	29836c65bd	Miscellaneous cleanups (#9849)	2026-08-04 11:17:03 +02:00
Luis Cossío	c00506bf46	[DiskCache] Dedup overlapping remote requests (#9838) ... * [AI] piggyback on in-flight requests * cleanup tests * Insert in-flight fetch through Slab's VacantEntry (#9840) Replace the peek-key-then-insert pattern with slab's vacant_entry() API: inserting through the reserved entry guarantees the fetch lands on the key the remote read was tagged with, instead of relying on nothing touching the slab between the peek and the insert. A failed remote schedule still leaves no trace, as VacantEntry allocates nothing until insert. get_or_init_remote_pipeline now takes the field instead of &mut self so the VacantEntry can hold a disjoint borrow of in_flight across the schedule call. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-04 11:17:03 +02:00
Luis Cossío	c7e0deed7c	exhaustive match on async-like backends (#9837)	2026-08-04 11:17:03 +02:00
Andrey Vasnetsov	ebcf5241e6	Add optional block-index sidecar to accelerate on-disk binary searches (#9810) ... * Add optional block-index sidecar to accelerate on-disk binary searches Binary search directly over an unpopulated mmap costs O(log n) random reads scattered across the whole file — one page fault (or remote range read) per probe on high-latency storage. Introduce SortedBlockIndex: an optional sidecar file storing the first element of every 16KiB block of a sorted on-disk array, read whole into RAM at open. A lookup becomes an in-RAM partition_point over the block firsts plus a single contiguous read of one block, bisected in RAM. Wire it into the three remaining scatter-probe sites: - on-disk numeric index: binary_search_pairs over data.bin - on-disk geo index: counts_of_hash over counts_per_hash.bin - on-disk geo index: all_points start boundary over points_map.bin The sidecar is backward and forward compatible: absent (old segments) or failing validation, readers fall back to the existing plain binary search; old code simply ignores the extra file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Validate block-index sidecar snapshot recovery and preopen coverage - snapshot round-trip test (Regular + Streamable): the sidecars written by the on-disk numeric and geo indexes are collected via files() into the snapshot tar and restored byte-for-byte, with correct query results on the reloaded segment - preopen tests for both indexes: after schedule_prefetch, open loads the sidecar from the CachedFs prefetch pool even when the file is unlinked from disk; an absent sidecar neither fails preopen nor open (fallback) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-04 11:17:02 +02:00
Andrey Vasnetsov	68395b4a9b	[UIO] Request-specific load profile for read-only opens (#9797) ... * Introduce request-specific LoadProfile with per-component placement A read-only shard opened for one known request (the serverless cold-start path) doesn't have to warm components the request will never touch. LoadProfile captures that from the request: warm components keep the persisted-config placement, everything else is parked cold. All placement decisions live in one place, so the memory placement of a whole segment under a profile is reviewable in one file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Thread LoadProfile through the read-only segment open ReadOnlySegment::open takes an optional profile; first_preopen and open_via resolve it into per-component populate overrides so the opens make the same placement decisions the prefetches did. Pinned components that materialize on open regardless (quantized RAM storage kinds, the immutable-RAM sparse index) and appendable components ignore the override; the HNSW graph and immutable payload indexes demote fully. Config reloads follow the new config alone and pass no override. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Open ReadOnlyEdgeShard under a request-derived load profile ReadOnlyEdgeShard::open takes an optional LoadProfile, applies it to every segment open and keeps it so segments discovered by a later refresh load with the same placement. ScrollRequestInternal and CoreSearchRequest gain load_profile() constructors, and edge-shard-query builds the request before the open and passes its profile (opt out with --no-load-profile). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Demote pinned quantized vectors and sparse index under a cold profile Within the immutable layout the quantized RAM and mmap loaders share the on-disk format — only how the data is brought into memory differs — and the immutable-RAM sparse index has the same lazy mmap open low-memory mode already downgrades to. So a cold populate override now demotes the effective placement itself (Memory::with_populate_override, shared with the HNSW residency mapping) instead of only skipping cache priming: a pinned quantized storage opens the mmap kind cold, and a pinned sparse index opens as Mmap, so neither reads its data on a cold start. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BzbiZFVHooxKVoJX4k6Ais * Add LoadProfile::merge for composite queries A composite query runs multiple core requests — e.g. a hybrid search runs one core search per vector, each with its own filter. Its profile is the union of its parts': merge extends the warm sets and ORs the payload-storage flag, so a component either part needs warm stays warm. The union is sound because every placement method is monotone in the warm sets (growing them only turns "park cold" into "keep configured placement"), so the merged profile dominates each input; and minimal, warming nothing no part asked for. Combine profiles with reduce, not fold: merge's identity element is the coldest profile (empty warm sets), the opposite of passing no profile at all — deliberately no empty()/Default constructor exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Adapt vanished-segment test to the profile-aware open signature The test landed on dev (#9777) after the load-profile signature change was written, so the rebase left its ReadOnlySegment::open calls without the new load_profile argument. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Defer vector index open entirely under a cold load profile A cold placement is not enough for the vector index on remote backends: GraphLinksView requires the whole links file as one contiguous slice, and the disk cache can only lend a borrowed slice once every block is locally present — so even a Cold HNSW open mirrors the entire links_compressed.bin (8.2 MB / 1.1 s per segment in the serverless cold-start trace), plus the unconditional graph.bin metadata read. The only way not to fetch the index is not to open it. LoadProfile::vector_index_placement is replaced by vector_index_deferred: a vector the request never scores now gets a DeferredVectorIndex — a new VectorIndexReadEnum variant holding the open arguments (an owned clone of the segment's raw backend, path, config, shared component handles) and a OnceLock. Nothing is opened or prefetched for it at segment open. Per-method policy of the deferred variant: - search, fill_idf_statistics and populate open the index on first use (with the cold placement the profile chose), so the profile contract holds: a request the profile did not predict still works, just pays the open then; - is_index reports true without opening (deferral only ever wraps a real HNSW or sparse index; plain opens no files and is never deferred); - telemetry, indexed_vector_count and sizes answer conservative defaults rather than trigger a remote fetch for a statistic. Tests: deleting the vector_index directory before an open under a scroll profile leaves open, filtered reads and payload reads working — proof that nothing of the index is read — while a search surfaces the missing files; and a segment opened under a scroll profile answers searches identically to an eagerly opened one via the transparent first-use open. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Make --vector optional in edge-shard-query: random query after open Omitting --vector on the search sub-command now searches with a random vector. The request is still built before the shard opens — the load profile only needs the vector name, not its values — with an empty placeholder; once the shard is open, fill_random_vector reads the dimension of the queried vector from the derived shard config and fills in uniform-random f32s (with a clear error if the named vector is not in the config). The vector is generated once, so live-reload iterations re-run the identical random query and the printed diffs stay meaningful. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Scope index deferral to the HNSW graph via a lazy OnceLock load Replace the DeferredVectorIndex wrapper (and the VectorIndexReadEnum:: Deferred variant) with deferral inside ReadOnlyHNSWIndex itself: the graph lives in a OnceLock (same first-wins arbitration as ReadOnlyRoaringFlags::bitmap) alongside the retained raw backend and residency, and loads on first use with a cold placement. The config read stays eager — one tiny, absence-tolerated file — so telemetry, is_on_disk and indexed_vector_count report real values where the Deferred arms answered with hard defaults. The sparse index needs no deferral: its mmap open reads lazily, with only small JSON metadata eager. A profile that never scores the vector now passes a cold placement override (LoadProfile:: vector_index_placement) into the eager open_sparse, which demotes ImmutableRam to the lazy Mmap open like low-memory mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Express HNSW graph deferral as a populate override, not a bool param Replace the `deferred: bool` on the read-only HNSW open/preopen (and the VectorIndexReadEnum pass-through) with the same `populate_override: Option<Populate>` every other component takes. A cold *override* defers the graph load — graph_deferred() mirrors the cold-override match of open_sparse — while a config-derived cold placement (or the low-memory clamp) keeps the eager load, since only a request-specific override carries the "never scored" prediction. With dense and sparse now consuming the same signal, LoadProfile::vector_index_deferred is gone: a single vector_index_placement() serves both index kinds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Serialize the deferred graph load via once_cell's get_or_try_init Loading outside the lock (std OnceLock's fallible init is still unstable) let a search burst on a deferred vector fetch the whole graph once per thread. Swap the cell for once_cell::sync::OnceCell: the fallible load runs inside the cell's lock, concurrent first users block on the one load, and a failed load leaves the cell empty so the next caller retries. Addresses https://github.com/qdrant/qdrant/pull/9797#discussion_r3573727836 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-04 11:17:01 +02:00
Andrey Vasnetsov	e1a4f9867d	Fix live-reload staleness of in-place-mutated files on caching backends (#9812) ... * Mutate same-operation slots in place in append-only mode With append_only_mutations enabled, every mutation of an existing point clones it to a fresh internal id. Shard-level updates decompose one point write into several SegmentEntry steps (upsert_with_payload issues upsert_point plus set_full_payload/clear_payload), so a single upsert burned one slot per step, leaving a chain of immediately-dead clones. A slot whose version already equals op_num was written by an earlier step of the current operation. It cannot be durable yet — the segment write lock is held across the whole operation, so no flush (and no read-only follower) can have observed it, and versions flush last so a crash discards it and WAL replay re-applies the whole operation. Mutate such slots in place: one operation now allocates exactly one slot regardless of how many steps it decomposes into. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stamp payload storage version in write_point_parts overwrite_payload mutates payload storage, but the fused write never bumped version_tracker's payload version — the old CoW path did, via set_full_payload. The segment manifest would stamp payload files with a stale version, letting a partial snapshot skip payload storage that contains the moved point's row, so the restored id tracker would point at an offset with no payload. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Resync ReadOnlyRoaringFlags from disk on live-reload The flags file is preallocated to power-of-two capacity and mutated in place within its length, which the held handle's reopen() — an append-only-growth contract on caching backends — never picks up: bits the writer changes inside already-cached DiskCache blocks stayed stale forever on shard followers. Drop the `impl LiveReload for ReadOnlyRoaringFlags` altogether: this storage holds arbitrary flags with no notion of points, so a point-delta interface (deleted/new points) did not belong here — open never applied deleted points either. Replace it with an inherent live_reload(fs) that opens a fresh StoredBitSlice (a fresh open always mirrors the current remote bytes), resyncs the materialized bitmap from it, and swaps the handle. The on-disk flags are the sole source of truth. To make refresh-by-fresh-open safe while the old handle is still alive, every DiskCache open now mirrors into a uniquely-named local file (.{pid}-{counter} suffix) and removes it on drop. Mirrors were already truncated on every open (block validity is in-memory only), so the stable name carried no state. Also add trace logging for live-reload consumed mapping changes and pending deltas. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Resync immutable id tracker deleted bitmap via fresh handle on live-reload The immutable tracker's id_tracker.deleted file is a fixed-size bitmap whose bits the writer flips in place — its length never changes — so the held handle's reopen(), an append-only-growth contract on caching backends, never picked it up: the pre-deletion state cached at open was served forever and live-reload never reported deletions on shard followers. live_reload now takes the fs, opens a fresh StoredBitSlice (a fresh open always mirrors the current remote bytes), diffs it against the tracker's current state — mappings already reflect every previously reported deletion, so they are the baseline — and swaps the handle in. ReadOnlyIdTrackerEnum and the segment-level reload pass the fs through; the appendable and disk-resident variants keep their no-arg reloads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Resync disk id tracker deleted bitmap via fresh handle on live-reload Same staleness as the immutable tracker: the deleted file is a fixed-size bitmap mutated in place, so reopen() — append-only-growth on caching backends — served the pre-deletion state forever. live_reload now takes the fs, opens a fresh StoredBitSlice, and swaps it into deleted_file, so the per-point get_bit lookups read fresh state from then on too. The deleted_full take/diff/set baseline logic is unchanged. The enum's DiskResident arm passes the fs through. The regression test now covers both trackers over DiskCacheFs; the disk leg was verified to fail under the old reopen-based behavior. The disk tracker's versions file staleness is a separate, still-open case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Introduce header-stateless ReadOnlyTracker for gridstore live-reload The gridstore tracker file is preallocated and mutated in place (header rewrites, slot updates), which the reader's held handle never picked up on caching backends: Tracker::live_reload read the header through the stale handle — before reopen(), and reopen() would not have refreshed cached blocks anyway — concluded "no new pointers", and never reloaded pages. New points' payloads read as empty on shard followers. Replace the reader's tracker with a dedicated ReadOnlyTracker that holds nothing but the storage handle: reads don't need header state (slot addressing is positional, unwritten slots read as None in the zero-initialized file) and readers have no pending-updates buffer. Its live_reload opens a fresh handle and swaps it in. Tracker::live_reload is deleted. A new TrackerRead trait (max_point_offset/get/iter) is implemented by both the writable Tracker and ReadOnlyTracker, and GridstoreView is generic over it, so writer and reader share the read logic. max_point_offset reads the stored header count as plain data through the current handle (fresh as of the last reload) rather than deriving it from slot capacity: sparse vector storage builds total_vector_count from it, and the capacity of the preallocated file (1MB -> 65536 slots) would inflate every follower segment's sparse count. The method is now fallible; consumers propagate the error. GridstoreReader::live_reload refreshes tracker and pages unconditionally — the tracker carries no reliable cheap change signal. Making this incremental again is a deferred follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Re-open last held chunk on chunked-vectors live-reload Chunk files are preallocated to full size, so appended vectors are in-place writes within the existing file length. On caching backends a held handle keeps serving blocks fetched earlier — and a block fetched near the old tail extends into then-unwritten space — so vectors appended into that block read back as stale bytes after a reload. Mirror Pages::live_reload: on a len change (the status file is read through a fresh open every reload, so it stays a reliable gate), drop and re-open the last held chunk — the only one that can have gained vectors — alongside adopting newly created chunk files. Earlier, fully-filled chunks keep their handles; their contents cannot change. Covers dense, multi-dense, and quantized-chunked storages, which all delegate to ChunkedVectorsRead::live_reload. Avoiding the whole-chunk refetch per reload is a deferred follow-up, together with the analogous gridstore pages/tracker case. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix lint: fs_err::create_dir_all in tests, drop unnecessary cast CI clippy runs with --all-targets and disallows std::fs methods in favor of fs_err; the new live-reload regression tests used std::fs::create_dir_all directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Tombstone-only deletes on non-appendable segments in append-only mode The tombstone-only delete path existed but was gated on is_append_only(), which requires the segment to be appendable. Deletes landing on non-appendable segments — notably the CoW update deleting the point's old copy — still went through delete_point_internal, which clears the payload row in place before the id-tracker drop. Clearing the payload destroys committed state of an offset that stays visible to live-reload followers until the drop is flushed: a follower refreshing in that window resolves the point through its (unchanged) id-tracker view and reads an empty payload — observed as a one-refresh {} payload flicker on a point update. Writer-side flush ordering cannot close the window, since the follower samples the id tracker and the payload storage at different instants; the versions commit protocol protects inserts exactly because the marker is read first, and deletes have no marker. This is the same failure class for which vector-deletion propagation was disabled (see the comment in delete_point_internal). Gate deletes on the new is_append_only_delete() — append_only_mutations alone, no appendability requirement: tombstoning needs nothing from the segment but the id tracker. Normal deployments keep eager payload clearing and prompt space reuse; clone-based mutations keep requiring appendability via is_append_only(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: chunked vectors reload --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Daniel Boros <dancixx@gmail.com>	2026-08-04 11:17:01 +02:00
Andrey Vasnetsov	f201849fac	[UIO] implement ReadOnlyQuantizedVectors::preopen (#9781) ... * [UIO] implement ReadOnlyQuantizedVectors::preopen Schedule background prefetch of the quantization config, per-method metadata, quantized data and multivector offsets, wired into the segment's first_preopen for every dense vector with quantization configured. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * additional stage for quantized config preopen * better quantized preopening * Partially populate the first quantized vector in preopen The load reads the first vector off quantized.data (and off the first chunk in the chunked layout) to validate the stored vector size — on a cold open that was a round-trip. Use Populate::Partial (#9769) to prefetch exactly that prefix; the exact size comes from the quantized config the preopen already read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Restore the layout-probing quantized preopen Recovers the pre-rewrite design: preopen never reads the quantization config — it probes both layout candidates against the listing snapshot (a segment only contains the layout its real config selects), derives warmth from the segment-side quantization config (placement resolution incl. cached and low-memory), iterates storage types exhaustively for layout coverage, and schedules the config for open to parse once off the parked handle — no threading, no second preopen stage. Keeps the first-vector partial populate, with the size now derived from the segment config via construct_vector_parameters, and keeps VectorStorageType::is_pinned. Restores the full preopen test matrix and OneshotFile::preopen. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review fixes --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Luis Cossío <luis.cossio@outlook.com>	2026-08-04 11:17:01 +02:00
Andrey Vasnetsov	bbc2694954	Add optional last-modified timestamp to ListedFile and CachedFs FileInfo (#9803) ... * Add optional last-modified timestamp to ListedFile and CachedFs FileInfo Filled where the listing backend exposes one: local filesystems (entry metadata) and object stores (ObjectMeta::last_modified). The uio-grpc backend reports None since the RPC does not carry mtimes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Carry last-modified over the StorageRead ListFiles RPC Extend ListFilesEntry with an optional google.protobuf.Timestamp, fill it on the server from the listing metadata, and convert it back to SystemTime in the uio-grpc client. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Update lib/common/io_bridge_object_store/src/source.rs Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Fix missing SystemTime import in io_bridge_object_store The CodeRabbit-suggested last_modified mapping used SystemTime::from without importing std::time::SystemTime, breaking compile and CI. Co-authored-by: Cursor <cursoragent@cursor.com> * Retrigger CI after flaky integration-tests-consensus timeout The compile fix is in; the prior run failed on an unrelated 30s timeout in test_collection_recovery, not on PR changes. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: root <111755117+qdrant-cloud-bot@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>	2026-08-04 11:17:01 +02:00
Andrey Vasnetsov	9e6892e272	Preserve not-found classification on read-only reload/open paths (#9777) ... Follow-up to #9763: an audit of the component live-reload and read-only open paths found four places where a not-found error lost its structured classification before reaching OperationError::FileNotFound: - MmapFile::reopen opened the file with a bare `?`, producing an unstructured Io(NotFound) — and this is the central call on the local follower's live-reload path, so no mid-reload segment removal would ever have classified on the mmap backend. Wrapped with extract_not_found. - BlockCacheFs::open and CachedSlice::reopen had the same bare-`?` pattern over CachedSlice::open's io::Result. Both wrapped. - ReadOnlySegment::open_via hand-rolled a service_error for a missing version file. The version file is written last, so its absence is exactly the vanished-mid-open signal; now FileNotFound { path }. - ReadOnlyRoaringFlags::read_status_len masked NotFound internally, so live_reload silently kept a stale len if the status file vanished mid-reload. The raw reader now propagates; open masks it explicitly with ok_not_found() (works on OperationResult since #9763), and live_reload requires the file. Everything else audited came back correct: lazily-created files (mutable id tracker, mutable-index gridstore) keep masking absence, optional-component probes at open keep ok_not_found, and the object-store bridge already maps missing objects to structured NotFound. New test vanished_segment_classifies_not_found pins both legs end-to-end: opening a missing segment directory and live-reloading a segment whose directory was removed both classify via is_not_found(). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-04 11:17:00 +02:00
Luis Cossío	3aaa127682	[UIO] Add and use Populate::Partial variant (#9769) ... * add (and handle) `Populate::Partial` * use partial populate * no OOB error	2026-08-04 11:17:00 +02:00
Andrey Vasnetsov	1bd3ee9b98	[UIO] implement VectorStorageReadEnum::preopen (#9780) ... * [UIO] implement VectorStorageReadEnum::preopen Schedule background prefetch of every file the read-only vector storages open, wired into the segment's first_preopen between the id tracker and the payload indexes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * polish PR - fully populate deleted bitslices - don't use `.ok_not_found()` - section comments * don't overwrite `populate` in `CachedFs` --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Luis Cossío <luis.cossio@outlook.com>	2026-08-04 11:17:00 +02:00
Luis Cossío	516c59a824	[UIO] implement ReadOnlyFullTextIndex::preopen (#9765) ... * prepare universalhashmap for partial populate * [AI] implement `ReadOnlyFullTextIndex::preopen`	2026-08-04 11:17:00 +02:00
Daniel Boros	31d8a023a5	feat: support S3 Express One Zone buckets in the S3 bridge (#9757)	2026-08-04 11:17:00 +02:00
Andrey Vasnetsov	e87c603b00	perf(universal-io): serve known file length from CachedFs snapshot on plain open (#9741) ... `CachedFs::cache_file_info` already snapshots every file's size from the `list_files` result, but that size was only threaded into opens on the prefetch path (`schedule_prefetch` -> `with_known_len`). Files opened directly through the fallback `open` — e.g. segments the loader opens lazily rather than prefetching — forwarded the caller's `OpenExtra` unchanged, so `known_len` was `None` and the backend later issued a remote `len`/HEAD to size the file. Thread the snapshot size into the fallback open too: when a snapshot exists and lists the path, apply `with_known_len(info.size)`. This lets `DiskCacheFs::open` go straight to `State::Ready`, eliminating a remote metadata round-trip per lazily-opened file on blob backends (S3). No-op when no snapshot was taken or for backends where `with_known_len` is meaningless. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-04 11:16:59 +02:00
Andrey Vasnetsov	ce3b1ceed8	debug logging of the s3 connection (#9597) ... * debug logging of the s3 connection * refactor(io_bridge): make S3 latency logs opt-in and low-overhead Move the blob-backend latency traces onto a dedicated `io_bridge::latency` log target at `trace` level, so they are silent by default and can be toggled as one group at runtime without a rebuild (e.g. `RUST_LOG=io_bridge::latency=trace`). Guard the timing `Instant::now()` behind `log_enabled!` so there is no overhead when the target is disabled. Also add `list_files` timing, and switch the shard_query CLI logger to millisecond timestamp resolution. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-08-04 11:16:59 +02:00
xzfc	fcb1c3db8a	Refactor: add DeletedBitVec (#9738)	2026-08-04 11:16:59 +02:00
Luis Cossío	2417721102	Pass file size to DiskCacheFs::open (#9730)	2026-08-04 11:16:59 +02:00
Luis Cossío	b5a12f1234	[AI + manual] cleanup (#9727) ... - get rid of `from_file` abstractions - renames: - `CachedFs` to be the implementation - `CachedReadFs` to be the trait	2026-08-04 11:16:59 +02:00
Andrey Vasnetsov	428e196f7d	CachedReadFs: prefetch-backed read-only segment opens (#9712) ... * Add CachedReadFs: prefetch-backed read-only universal-io filesystem Snapshots the file listing at construction and serves opens from explicitly prefetched handles (take-once, shared across clones via Arc<Mutex>). A non-prefetched open falls back to a direct open on the inner filesystem, panicking in debug builds and warning in release. CachedFile is a transparent wrapper needed to satisfy the bidirectional UniversalReadFs<File = Self> pinning, following the ReadOnly pattern. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Serve read-only segment opens from CachedReadFs prefetch pool ReadOnlySegment::open builds a per-segment CachedReadFs: the files known in advance (version.info, segment.json) are scheduled before the listing snapshot is taken so their fetch overlaps the listing round-trip, then every remaining listed file is scheduled, running all fetches in parallel instead of serializing them inside component opens. Existence checks and format-detection probes are answered from the snapshot without touching the inner filesystem. Stored handles are taken out of the pool via the new CachedReadFs::take_file, which returns the raw inner file — component types stay over plain S, and CachedFile exists only transiently inside open-read-discard helpers (read_json_via etc.) through the trait impl. The read-only open path takes &CachedReadFs<S::Fs> concretely; storing wrappers gained from-file constructors (StoredBitSlice::from_file, UniversalHashMap::from_file, ReadOnly::from_file, gridstore Tracker::open_cached / Pages::open_cached, read_chunks_cached). Snapshot-less, CachedReadFs is a passthrough to the inner filesystem — used by reload paths, writable build paths that reuse the on-disk index opens, and tests, all of which keep their previous behavior. Components that retain a filesystem for later reloads store the raw inner backend (CachedReadFs::inner), never the stale snapshot. Also: local_list_files now recurses into subdirectories, matching the flat key-prefix semantics of object-store listings; the immutable id tracker probes its defining file via exists (free on the snapshot) instead of a probe-open that would consume the take-once handle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Group imports per nightly rustfmt Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix sparse search bench for open_ro over CachedReadFs CI clippy runs --all-targets; the bench target was missed by the --tests sweep. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Match listing prefixes by path component, not by string The recursive local_list_files compared whole path strings; on Windows a joined prefix mixes `/` and `\` (`shard\index/chunk_`) while walked entry paths use `\` throughout, so nothing ever matched (broke list_files_returns_paths_relative_to_shard_dir on Windows CI). Match the entry name at the prefix's final position against the prefix's final component instead, then walk matched directories exhaustively — same semantics, separator-agnostic. Apply the same component-based matching to the CachedReadFs snapshot filter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * do not cache everything * fmt * dont unwrap files_info * fix clippy * relax debug assertion for now * [AI] refactor into extension trait, relax Fs<->File requirement (#9725) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Luis Cossío <luis.cossio@outlook.com>	2026-08-04 11:16:58 +02:00
Arnaud Gourlay	bc5ff2acd7	Remove dead code (#9719)	2026-08-04 11:16:58 +02:00
Arnaud Gourlay	a11a2ec6f1	Fix Clippy 1.97 (#9716) ... * Remove from_iter_instead_of_collect from workspace lints The lint was removed from clippy (beta) and now triggers renamed_and_removed_lints warnings in every crate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix clippy::chunks_exact_to_as_chunks Replace chunks_exact with a constant chunk size by as_chunks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix clippy::needless_late_init Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix clippy::useless_borrows_in_formatting Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix clippy::uninlined_format_args Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix clippy::for_kv_map Iterate map values directly instead of discarding keys. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Allow clippy::result_large_err on QueueProxyShard::new_from_version The Err variant intentionally hands the LocalShard back to the caller. Same pattern as the existing allow on ForwardProxyShard::new. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Allow clippy::result_unit_err on wait_for_consensus_commit Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-04 11:16:58 +02:00
Andrey Vasnetsov	30f00477c6	Add disk-resident id tracker (#9657) ... * Add disk-resident id tracker Introduce a disk-resident id tracker family that keeps the point-id mapping on disk instead of loading it into RAM, so resident memory no longer scales with total point count. This removes the last component of a segment that forces a full RAM load, enabling object-storage / edge followers and cutting RAM for regular deployments that opt in. New on-disk mapping format (`disk_id_tracker/on_disk_format.rs`): random-access `id_tracker.i2e` (internal->external) + `id_tracker.e2i` (external->internal as sorted num/uuid runs with a resident sparse block index). `id_tracker.versions` and `id_tracker.deleted` are reused unchanged. Trackers (sharing a lazy `DiskMappingReader` core and a `DiskMappingsSource` trait): - `DiskIdTracker` — writable, deletion-only; a new `IdTrackerEnum` variant used by regular segments, keeping deleted+versions resident and the mapping on disk. - `ReadOnlyDiskIdTracker` — read-only live-reload mirror for followers; per-point `get_bit` deletion checks on read-by-id, full deleted set materialized lazily. Selection: created when the `serverless_compatible` feature flag is set (in `segment_builder`); loaded by attempting each format in `ReadOnlyIdTrackerEnum:: detect_and_load` (no per-file `exists` round-trips) and by file-presence detection unified in `IdTrackerFormat`. `PointMappingsRefEnum` is generic over the read backend so the disk variant is a concrete `DiskMappingsRef` (no trait object). The `DiskMappingsSource` read surface returns `OperationResult`; errors are only swallowed at the infallible `IdTrackerRead` boundary. `ImmutableIdTracker` is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style: apply rustfmt Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix clippy: collapse nested if in DiskIdTracker::drop Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Align on-disk id tracker headers and sections to 16 bytes Pad the i2e header to 32 bytes and the e2i header to 48 bytes, and zero-pad between e2i sections so every section starts on a 16-byte boundary. This keeps the files mmap+transmute-friendly: the u128 arrays (i2e slots, e2i uuid sparse index) need 16-byte alignment in Rust. Add on_disk_sections_are_aligned test pinning the invariant and the store/parse padding agreement via exact file-length checks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-04 11:16:57 +02:00
Daniel Boros	26d2629df8	fix: resolve AWS S3 default credentials from env (IRSA); object_store 0.14 (#9692) ... * chore: update object_store to 0.14.0 * fix: resolve AWS S3 default credentials from the environment The default path built the client with AmazonS3Builder::new(), which reads nothing, so object_store's resolver always fell through to EC2 IMDS (the node role) — AWS_WEB_IDENTITY_TOKEN_FILE / AWS_ROLE_ARN (EKS IRSA), ECS and EKS Pod Identity were silently ignored. Seed the builder with from_env() for the default chain so web identity (IRSA), container credentials and AWS_* keys are honoured; explicit bucket/region/endpoint from config still override anything from_env picks up. Static credentials keep using new() with the provided keys.	2026-08-04 11:16:57 +02:00
Andrey Vasnetsov	b185c46435	[UIO] Split UniversalReadFileOps into read and write traits (#9682) ... UniversalReadFileOps mixed read-side operations (from_context, list_files, exists) with mutating ones (create, create_dir, remove, remove_dir, atomic_save). Move the mutating operations to a new UniversalWriteFileOps subtrait, mirroring UniversalRead/UniversalWrite. - UniversalWrite now requires Fs: UniversalWriteFileOps, so generic consumers (gridstore) keep reaching write ops through S::Fs. - ReadOnlyFs drops its runtime-erroring write stubs: read-only is now a compile-time property. - DiskCacheFs implements the write side only when the remote fs does. - BlobFs and the async AsyncRead trait are read-only: the async write methods are removed from AsyncRead and its backends (object store, uio-grpc) along with the tests that exercised them. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-04 11:16:57 +02:00
Luis Cossío	c5c46d4184	[DiskCache] refactor State into single enum (#9651) ... * [AI + manual] refactor State into single enum * use `let .. else` * use explicit matches	2026-08-04 11:16:57 +02:00
Luis Cossío	09f74749b2	[UIO] Include file size in UniversalReadFileOps::list_files (#9675) ... * [AI + manual] Include file size in `UniversalReadFileOps::list_files` * Use dedicated `ListedFile` struct instead of `(PathBuf, u64)` pair Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: generall <andrey@vasnetsov.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-08-04 11:16:57 +02:00
xzfc	bc2c40d93c	Tidy up Debug impls (#9653)	2026-08-04 11:16:56 +02:00