Andrey Vasnetsov 5d2ec88276 Second thoughts on CJK tokenizer (#2260) ...

* charabia tokenizer for CJK, init

* add charabia to .proto

* generate grpc docs

* rename charabia to multilingual

* disable charabia default features

* ignore stopwords and separators in tokenizer

* fix rebase conflict

* fix test

* fix test

* Bump `charabia` crate version...

...and enable `charabia` features that does not require any additional dependencies by default

* Expose optional `charabia` features in the `segment` crate

* Expose optional `charabia` features in the `qdrant` crate

* fix codespell

* more regression tests

* more regression tests

* make language-specific tests feature-flagged

---------

Co-authored-by: Anatolii Smolianinov <zarkonesmall@gmail.com>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>

2023-07-19 14:59:40 +02:00

..

docs.md

Second thoughts on CJK tokenizer (#2260)

2023-07-19 14:59:40 +02:00