qdrant

mirror of https://github.com/qdrant/qdrant.git synced 2026-07-27 05:01:08 -05:00

Files

eltu aa0f49798e Full-Text Index ASCII Folding (Normalization) (#7408 )

* Add ASCII folding to tokenization process

Introduced an optional ASCII folding feature within the `TokensProcessor` to normalize non-ASCII characters to their ASCII equivalents. Updated tests and documentation to reflect the changes.

* Refactor tokenization code for improved readability and maintainability

Reorganized and reformatted the tokenization module, including `TokensProcessor` initialization and ASCII folding mappings for better clarity. Updated tests to align with the changes.

* Update test cases to reflect optional tokenizer settings changes

Adjusted `ascii_folding`, `lowercase`, and `phrase_matching` settings in tests to `None` where applicable, aligning with updates in tokenizer configuration defaults.

* address review remarks

* fix codespell

* thx coderabbit

* Don't copy tokens that are already ASCII

* Shrink folded string to fit

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>

2025-10-16 13:53:01 +02:00

grpc

Full-Text Index ASCII Folding (Normalization) (#7408 )

2025-10-16 13:53:01 +02:00

imgs

Add docs about coverage reports (#5958 )

2025-02-08 01:22:23 +01:00

redoc

Full-Text Index ASCII Folding (Normalization) (#7408 )

2025-10-16 13:53:01 +02:00

roadmap

Update roadmap (#6947 )