mirror of
https://github.com/qdrant/qdrant.git
synced 2026-07-31 07:00:56 -05:00
570546c23e
* Fix phrase matching crossing string-array element boundaries
When indexing a string array like ["quick", "brown"], the index
concatenated tokens from all elements into a single flat document.
A phrase query for "quick brown" found the tokens consecutively,
producing false matches.
Insert a sentinel token ("\x00") between array elements during
indexing. It is registered as a normal vocab token that occupies a
position in the document, so phrase windows cannot span it. No
tokenizer ever produces this string, so it never appears in a query.
Closes #8937
Co-authored-by: Cursor <cursoragent@cursor.com>
* Strip null bytes from tokenizer output
Ensure no tokenizer can produce tokens containing '\0', which is
reserved as the array-boundary sentinel for phrase matching.
Covers all tokenizer paths:
- process_token_cow (used by Word, Whitespace, Prefix, Multilingual)
- Japanese tokenizer (has its own inline processing)
Co-authored-by: Cursor <cursoragent@cursor.com>
* Revert "Strip null bytes from tokenizer output"
This reverts commit fe6ea7a955.
* filter sentinel token from query
* Preallocate vector
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Tim Visée <tim+github@visee.me>