* chore: Add missing type hints in functions
* add missing import, small type refactor
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* feat: Added a toggle to disable stemmer in bm25
* refactor: Refactored how to disable stemming in bm25
* refactor: Refactored the way of disabling stemmer in bm25
* new: Added english fallback if language = None
* tests: Added test case for disable stemmer
* fix: Fix language to be only string
* tests: Updated bm25 toggle stemmer tests
* refactor: fix stopwords type
* fix: fix param propagation in parallel embed in bm25
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* feat: Added multi gpu support for text embedding
* feat: Add support for multi-gpu for special text models
* fix: Fix lazy_load to load the model to child processes when parallel is not none
* feat: Added lazy_load and multi-gpu to colbert
* feat: Add lazy_load and multi gpu to image models
* feat: Support lazy_load and multi-gpu to sparse models (except BM25)
* fix: Fixed BM25 not working
* refactor: Remove redundant GPUParallelProcessor
* refactor: Refactor _embed_*_parallel
* feat: Add cuda argument
refactor: Refactor how worker assign device
* fix: Fix if providers and cuda are None
* fix: Fix providers and cuda are none
* WIP: Multi gpu support review (#361)
* WIP: review
* wip: review
* refactor: refactor images
* refactor: refactor sparse
* refactor: refactor late interaction
* add model loading
* add tests
* fix: uncomment models in tests
* fix: fix variable declaration order
* fix: fix device id assignment
* tests: add multi gpu tests
* fix: fix device id assignment for sparse embeddings
* tests: update multi gpu tests
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* refactor: remove redundant declarations
* fix: rollback redundant changes
* fix: remove num workers device ids dep, fix type hint
* fix: fix post process for sparse models
* fix: remove redundant model loading
* new: add lazy load and new gpu support to cross encoders
* fix: add rerankers to multi gpu tests
* fix: unlock multilingual test
* fix: fix gpu test with cross encoder
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* Tf-idf fix:
Remove punctuation properly + lowercase
* Test fix
* Test fix
* Test fix
* Redudant accum removal
* Redudant accum removal
* 0.878890950070688
* Latest changes with
Average recall: 0.8915516690721613
* Support of special characters
* Removal of debug print
* Type annotations
* Type annotation support for python 3.8
* Type annotation support for python 3.8
* wip: remove model dir
* fix: update pytest run
* wip: disable some tests not used atm
* wip: disable some tests in ci
* wip: add debug print
* fix: fix ci, remove models after usage
* fix: fix bm25 deletion
* fix: remove redundant ci commands
* Fix: Normalize tokens to lowercase before checking stopwords in BM25
* Test: Normalize tokens to lowercase before checking stopwords in BM25
* Test Fix test_multilanguage: in "Je suis au lit", the "Je" should be skipped because it in the stopwords.
* chore: apply ruff
---------
Co-authored-by: H4-8ZSI <H4-8ZSI@EXAMPLE.COM>
* MiniLM fix
* Added MiniLM to text embedding
Fixed MiniLM source destination
Black + isort for repo
* Fixed model all-MiniLM-L6-v2 description
Recomputed canonical vector for all-MiniLM-L6-v2 in test
---------
Co-authored-by: d.rudenko <dimitriyrudenk@gmail.com>
* Add SPLADE v1
* WIP SPLADE Export errors
* add ONNX model to HF hub and use that
* Update sentences in Converting_SPLADE_to_ONNX.ipynb
* Remove unnecessary files and directories
* Rename var in TextEmbedding class to use EMBEDDING_MODEL_TYPE
* Add SPLADE to list of text embeddings
* Add SPLADE model support for text embedding
* Fix deprecation warning in embedding.py
* Add test for batch embedding with sparse embeddings
* Refactor import statement in test_sparse_embeddings.py
* Rename nbs
* Update vocab size in SPLADE model
* Fix canonical vector lookup in test_text_onnx_embeddings.py
* review refactoring
* restore list_supported_models in OnnxTextEmbedding
* Remove unused method _preprocess_onnx_input() in SpladePP class
* Update SPLADE_PP_en_v1 source in splade_pp.py
* Refactor onnx_model.py to change base model behavior
* extend tests to sparse values as well as indicies
* chore: pre-commit hooks
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Anush008 <anushshetty90@gmail.com>
