mirror of
https://github.com/qdrant/qdrant.git
synced 2026-07-23 11:11:00 -05:00
edc299b0cb
* Add new tokenizer for multilingual * Make codespell happy * improve handling of when to apply stemming * Fix stemming implementation * Fix Japanese language detection. Add tests for MultilingualV2 * Update lib/segment/src/index/field_index/full_text_index/tokenizers/japanese.rs Co-authored-by: Luis Cossío <luis.cossio@qdrant.com> * Update lib/segment/src/index/field_index/full_text_index/tokenizers/japanese.rs Co-authored-by: Luis Cossío <luis.cossio@qdrant.com> * Update lib/segment/src/index/field_index/full_text_index/tokenizers/multilingual.rs Co-authored-by: Luis Cossío <luis.cossio@qdrant.com> * Make init() private * move model into segment, filter non-alphanumeric tokens * fmt * exclude punctuation from tokenizer result * fm --------- Co-authored-by: Luis Cossío <luis.cossio@qdrant.com> Co-authored-by: generall <andrey@vasnetsov.com>