eltu aa0f49798e Full-Text Index ASCII Folding (Normalization) (#7408) ...

* Add ASCII folding to tokenization process

Introduced an optional ASCII folding feature within the `TokensProcessor` to normalize non-ASCII characters to their ASCII equivalents. Updated tests and documentation to reflect the changes.

* Refactor tokenization code for improved readability and maintainability

Reorganized and reformatted the tokenization module, including `TokensProcessor` initialization and ASCII folding mappings for better clarity. Updated tests to align with the changes.

* Update test cases to reflect optional tokenizer settings changes

Adjusted `ascii_folding`, `lowercase`, and `phrase_matching` settings in tests to `None` where applicable, aligning with updates in tokenizer configuration defaults.

* address review remarks

* fix codespell

* thx coderabbit

* Don't copy tokens that are already ASCII

* Shrink folded string to fit

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>

2025-10-16 13:53:01 +02:00

..

src

Full-Text Index ASCII Folding (Normalization) (#7408)

2025-10-16 13:53:01 +02:00

build.rs

Use fs-err (#7319)

2025-09-29 12:47:10 +00:00

Cargo.toml

Bump dev version to v1.15.5-dev (#7154)

2025-08-27 10:54:15 +02:00