* chore: Add missing type hints in functions
* add missing import, small type refactor
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* Merge master
* rerank_pairs interface + parallelism support
* remove test notebook
* Removed unused code
* New tests for cross encoders and new interface
* Importing Self fix. We will need it for mypy support in newer versions
* Removed Self typing
* Removed non-needed changes from text
* Isort + black
* wip: start reviewing (#420)
Co-authored-by: Dmitrii Ogn <dimitriy_rudenko@mail.ru>
* Test fix
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder_base.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Test for parallel processing + bugfix of PosixPath passing
* Removed non-needed import and added docstring
* Typing fix + argument passing
* Test parametrization
Moved to selected models set to test
* Run base test on all models
* Typing fix + improvement of input_names check
* nit: fix post process, update docstring, update tokenize, remove redundant imports
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* feat: Added jina reranker models
* chore: Added jina reranker canonical score values
* chore: added rounding of the output for easier reproducability
* chore: Added jina reranker models in batch test
* chore: remove redundant np.round
* chore: test only <1gb files in local
* chore: Updated docs to add rerankers
* fix: recompute canonical values with fp16
* new: extend NOTICE with jina reranker v2
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* feat: Added multi gpu support for text embedding
* feat: Add support for multi-gpu for special text models
* fix: Fix lazy_load to load the model to child processes when parallel is not none
* feat: Added lazy_load and multi-gpu to colbert
* feat: Add lazy_load and multi gpu to image models
* feat: Support lazy_load and multi-gpu to sparse models (except BM25)
* fix: Fixed BM25 not working
* refactor: Remove redundant GPUParallelProcessor
* refactor: Refactor _embed_*_parallel
* feat: Add cuda argument
refactor: Refactor how worker assign device
* fix: Fix if providers and cuda are None
* fix: Fix providers and cuda are none
* WIP: Multi gpu support review (#361)
* WIP: review
* wip: review
* refactor: refactor images
* refactor: refactor sparse
* refactor: refactor late interaction
* add model loading
* add tests
* fix: uncomment models in tests
* fix: fix variable declaration order
* fix: fix device id assignment
* tests: add multi gpu tests
* fix: fix device id assignment for sparse embeddings
* tests: update multi gpu tests
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* refactor: remove redundant declarations
* fix: rollback redundant changes
* fix: remove num workers device ids dep, fix type hint
* fix: fix post process for sparse models
* fix: remove redundant model loading
* new: add lazy load and new gpu support to cross encoders
* fix: add rerankers to multi gpu tests
* fix: unlock multilingual test
* fix: fix gpu test with cross encoder
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
