mirror of
https://github.com/qdrant/qdrant.git
synced 2026-09-25 15:37:42 -05:00
Arnaud GourlayandClaude Opus 5.5
9b51625164
* Expose document lengths from the text index read surface `doc_len` and `total_tokens` on `FullTextIndexRead`, backed by the `InvertedIndex` accessors deferred from the two previous steps. Nothing reads them yet, so the recording gate still keeps every answer absent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix up the document length accessors after review - drop the bounds check in read_point_to_doc_len: the deletion mask the caller already consults establishes the bounds, and `len()` is an fstat on io_uring and a blocking HEAD on object storage, once per scored point - pin the deletion mask in the on-disk total: the agreement test cannot catch it, since its deletions predate `create` and leave zeroes on disk - correct three doc comments. `total_tokens` is not the converse of `doc_len`, `remove` is exactly the invalidation hook the comments claimed did not exist, and "live points" is wrong under append-only deletion Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Make the document length accessors agree and bill their reads - `doc_len` answers the same for the same data on every backend. The on-disk one now bounds by the point space `open` already computes and reports `Some(0)` for a point it holds no tokens for, instead of letting the storage placement decide between `Some(0)` and `None` - both accessors take a `HardwareCounterCell`, so the per-point reads and the whole-sidecar total are measured like every other IO on the trait - `ImmutableInvertedIndex` maintains `total_tokens` rather than summing on every call: `From<MutableInvertedIndex>` already has the counter, the on-disk conversion already walks every element to mask it, and `remove` is the only mutation afterwards Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * State the settled population rule on the total tokens accessor The comment described the interim disagreement between the in-RAM and on-disk counts, which #10675 removes. Describe the rule instead: both counts cover the documents still held that carry at least one indexed token, so the ratio does not depend on the storage placement. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Read document lengths only in batches A per-point doc_len invites one read per point, a round trip each when the on-disk index sits on a slow or remote disk. Both InvertedIndex and FullTextIndexRead now expose doc_len_batch alone: the in-RAM backends answer from their vector, and the on-disk one answers out-of-range and inactive points without IO and sends the remaining reads through a single read_batch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>