mirror of
https://github.com/qdrant/qdrant.git
synced 2026-09-29 09:27:53 -05:00
Arnaud GourlayandClaude Opus 5
c296898355
* Carry document length into the immutable and on-disk text indexes The mutable index records `doc_len`. The immutable and on-disk backends dropped it on conversion. They now carry it and persist it as a sidecar, so a length survives an optimizer run and a restart. - `ImmutableInvertedIndex` gains `point_to_doc_len`, parallel to `point_to_tokens_count` and zeroed wherever that vector is, so summing it never counts a deleted document. - `OnDiskInvertedIndex` writes `point_to_doc_len.dat`, only when the index records lengths. `files()` lists it only when it exists, so a snapshot carries it exactly when there is one. - Deletions are masked on load, not at build time. The file is written once and a point deleted later through the id-tracker is zeroed in `TryFrom<&OnDiskInvertedIndex>`. No segment total is stored, since it could only be summed after that masking. - A sidecar shorter than the counts is treated as absent. It can only come from a partially copied file set, and padding it would give every point past the truncation a zero that reads like a real length. - A missing sidecar on an index that should have one makes `new_mmap` report the index absent, which routes into the existing rebuild from payload. The check lives there rather than in `OnDiskInvertedIndex::open` because the read-only stack never builds and would drop the field instead. - `FullTextMmapIndexBuilder::add_many` reached below `index_str_tokens` and so declared no length at all. It now tokenizes through the shared helper and measures like the gridstore path. Recording is still gated behind `TextIndexParams::scoring()`, so none of this is written today. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Wipe the text index directory instead of its listed files `files()` lists the `doc_len` sidecar only when the index loaded it, so a truncated sidecar is omitted. `wipe` then deleted the listed files, failed to remove the now non-empty directory, discarded that error and returned success, leaving the directory and a stale sidecar behind. Remove the directory itself. Each field owns its own `{field}-text` directory, so this no longer depends on `files()` being an accurate inventory, and real removal errors propagate rather than being swallowed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Drop the fully qualified visibility paths in the text index `pub(in crate::index::field_index::full_text_index)` is tedious to read and to keep correct, and it grants nothing here: `inverted_index` and `mutable_text_index` are private modules of `full_text_index`, so their types are not nameable from outside it whatever the field visibility says. Plain `pub` is no wider in practice. Widening `Storage` surfaced two `private_interfaces` warnings, since `ZerocopyPostingValue` and `PostingListHeader` in `types.rs` were still behind the long path; those move too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Harden the document length sidecar after review - check for the sidecar before opening the index rather than after. The open populates the whole file set, so the first start after scoring is enabled would fault in every segment's postings only to discard them - treat a sidecar that covers more points than the index as untrustworthy, not just one that covers fewer, and warn with both counts the way `SortedBlockIndex::open` does. A longer one used to be accepted and then silently truncated when materialized - unlink a stale sidecar when a build records no lengths. It was the only file here that could outlive the build that wrote it, and `open` would have read it as this build's - say why an index is being rebuilt from payload instead of leaving a silent full re-index announced at debug level - read `phrase_matching` from the config in the mmap builder, like the other two callers of the same helper pair, so the two halves of the sentinel rule cannot drift apart - assert the length invariant the writers actually maintain, and correct what `files()` and the field comment claim Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Address review on the document length sidecar - shrink the document length vector when it is handed over to the immutable index, it arrives with the mutable index's doubling capacity - narrow the immutable index fields to pub(super), with a test-only accessor for the one reader outside the module - let wipe propagate a missing index directory instead of treating it as success, nothing reaches it with the directory already gone Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>