* chore: Add missing type hints in functions
* add missing import, small type refactor
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* new: Add mypy and pyright deps
* chore: Add any to kwargs
* chore: Add any to args
* add missing kwargs
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* HF sources for all models
* Specific_model_path model path support
* Fix hf download
* fix: rollback incorrect model replacement
* refactor: remove redundant type imports
* refactor: replace List with list
* fix: remove redundant param in late interaction text embedding
* Update fastembed/common/model_management.py
* fix: rollback post process onnx output
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* fix: Fix minilm paraphrase by adding it to pool models
* tests: Updated minilm paraphrase canonical vector
* chore: Added a warning message for updating the model
* chore: Added version where model will be removed
* new: Added jina embedding v3
* refactor: Changed dim to int value
* new: Updated notice
* new: Extended text embedding with query embed and passage embed
* fix: Fix lazy load in query and passage embed
* tests: Added test for multitask embeddings
* nit: Remove cache dir from tests
* tests: Updated tests
* improve: Improve task selection
* fix: Fix ci
* fix: Update fastembed/text/multitask_embedding.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/text/multitask_embedding.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* fix: Pass task id using kwargs to parallel processor
* tests: Added test for task assignment
* prefer enums over ints
* tests: Added test for parallel
* improve: Updated model description
* fix: Fix ci
* fix: Fix ci
* refactor: Refactor query_embed and passage_embed
* tests: Added task propagation to parallel
* refactor: Set default task as retrieval passage
* chore: Update default task in tests
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* Merge master
* rerank_pairs interface + parallelism support
* remove test notebook
* Removed unused code
* New tests for cross encoders and new interface
* Importing Self fix. We will need it for mypy support in newer versions
* Removed Self typing
* Removed non-needed changes from text
* Isort + black
* wip: start reviewing (#420)
Co-authored-by: Dmitrii Ogn <dimitriy_rudenko@mail.ru>
* Test fix
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder_base.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Test for parallel processing + bugfix of PosixPath passing
* Removed non-needed import and added docstring
* Typing fix + argument passing
* Test parametrization
Moved to selected models set to test
* Run base test on all models
* Typing fix + improvement of input_names check
* nit: fix post process, update docstring, update tokenize, remove redundant imports
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* WIP: Added jina clip text embedding
* WIP: Added preprocess for jina clip
* WIP: Added jina clip vision (not sure if it works yet)
* improve: Improved mean pooling if the output doesnt have seq length
* fix: Fixed jina clip text
* nit
* fix: Fixed jina clip image preprocessor
* fix: Fix type hints
new: added resize2square
* tests: Add jina clip vision test case
* nit
* refactor: Update fastembed/image/transform/operators.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* fix: Fix indentation
* refactor: Refactored how we call padding for image
* fix: Fix pad to image when resized size larger than new square canvas
* refactor: minor refactor
* refactor: Refactor some functions in preprocess image
* fix: Fix to pad image with specified fill color
* refactor: Change resize to classmethod
* fix: Fix jina clip text v1
* fix: fix pad to square for some rectangular images (#421)
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* feat: Added a toggle to disable stemmer in bm25
* refactor: Refactored how to disable stemming in bm25
* refactor: Refactored the way of disabling stemmer in bm25
* new: Added english fallback if language = None
* tests: Added test case for disable stemmer
* fix: Fix language to be only string
* tests: Updated bm25 toggle stemmer tests
* refactor: fix stopwords type
* fix: fix param propagation in parallel embed in bm25
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* chore: Remove typing hints of Python less than 3.9
* chore: Removed optional from cache as it cannot be undefined
* improve: Turned off progress bar of huggingface models if cached
This causes an incompatibility with spaCy because no matching
versions of numpy can be found when using both libraries at once.
It looks like the tests pass again without the numpy<2 requirement
so there is no reason to keep this requirement anymore.
* feat: Added jina reranker models
* chore: Added jina reranker canonical score values
* chore: added rounding of the output for easier reproducability
* chore: Added jina reranker models in batch test
* chore: remove redundant np.round
* chore: test only <1gb files in local
* chore: Updated docs to add rerankers
* fix: recompute canonical values with fp16
* new: extend NOTICE with jina reranker v2
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* fix: Fix return of onnx_embed
* fix: Fix type hint of start method in worker class
* fix: Fix not passing kwargs in _preprocess_onnx_input and tokenize as base class
* fix: Fix not passing kwargs in _preprocess_onnx_input as base class
* fix: change tokenize in simpleTokenizer to classmethod
* chore: Changed query argument to Iterable to match base class
* chore: changed mask token id and pad token id to be int
* review suggestions (#398)
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* improve: Changed the way we are adding the query and document markers in colbert
* fix: Truncate the inout_ids and attention_mask when adding query and document markers to original input length
* fix: Fix broadcast issue
* chore: Remove redundant if condition
* nit
* refactor (#397)
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* Add rerankers to the list of supported models (Supported_Models.ipynb)
* fix: update imports
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* feat: Added support for jina-colbert-v2
* chore: Generalized query marker and document marker
* nit: remove github action on dispatch
* chore: updated license
* fix: Fix attention mask to be all 1 in xlmrobertatokenizer
* feat: Added class for JinaColbertV2
* feat: Added jina colbert
* chore: Change tolerance of the test
* chore: Changed encoding of attention mask to 1 to be only in queries
* chore: Changed the replacable token to be ' @' as its considered as one token
* chore: Removed redundant functions
* chore: Updated supported models docs
* nit: Remove print statement
* nit: visual stuff
* fix: Fix dimention of jina colbert in description
* fix: canonical query and document values for jina colbert
* bump version to 0.4.0
* py-rust-stemmers support instead of snowball stemmer and pystemmer
* py-rust-stemmers support instead of snowball stemmer and pystemmer
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* fix: Fix type when calling parallelworkerpool
parallelworkerpool accepts num_workers as int only
* fix: Fix type OnnxOutputContext.
OnnxOutputContext takes attention_mask and input_ids as optional while we cannot assign none to ndarray
* fix: Fix progress might not be bool and might be literal 0
* feat: Added multi gpu support for text embedding
* feat: Add support for multi-gpu for special text models
* fix: Fix lazy_load to load the model to child processes when parallel is not none
* feat: Added lazy_load and multi-gpu to colbert
* feat: Add lazy_load and multi gpu to image models
* feat: Support lazy_load and multi-gpu to sparse models (except BM25)
* fix: Fixed BM25 not working
* refactor: Remove redundant GPUParallelProcessor
* refactor: Refactor _embed_*_parallel
* feat: Add cuda argument
refactor: Refactor how worker assign device
* fix: Fix if providers and cuda are None
* fix: Fix providers and cuda are none
* WIP: Multi gpu support review (#361)
* WIP: review
* wip: review
* refactor: refactor images
* refactor: refactor sparse
* refactor: refactor late interaction
* add model loading
* add tests
* fix: uncomment models in tests
* fix: fix variable declaration order
* fix: fix device id assignment
* tests: add multi gpu tests
* fix: fix device id assignment for sparse embeddings
* tests: update multi gpu tests
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* refactor: remove redundant declarations
* fix: rollback redundant changes
* fix: remove num workers device ids dep, fix type hint
* fix: fix post process for sparse models
* fix: remove redundant model loading
* new: add lazy load and new gpu support to cross encoders
* fix: add rerankers to multi gpu tests
* fix: unlock multilingual test
* fix: fix gpu test with cross encoder
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
d.rudenko