* Add new tokenizer for multilingual
* Make codespell happy
* improve handling of when to apply stemming
* Fix stemming implementation
* Fix Japanese language detection. Add tests for MultilingualV2
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/japanese.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/japanese.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/segment/src/index/field_index/full_text_index/tokenizers/multilingual.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Make init() private
* move model into segment, filter non-alphanumeric tokens
* fmt
* exclude punctuation from tokenizer result
* fm
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: generall <andrey@vasnetsov.com>
