* chore: Add missing type hints in functions
* add missing import, small type refactor
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* fix: Fix minilm paraphrase by adding it to pool models
* tests: Updated minilm paraphrase canonical vector
* chore: Added a warning message for updating the model
* chore: Added version where model will be removed
* new: Added jina embedding v3
* refactor: Changed dim to int value
* new: Updated notice
* new: Extended text embedding with query embed and passage embed
* fix: Fix lazy load in query and passage embed
* tests: Added test for multitask embeddings
* nit: Remove cache dir from tests
* tests: Updated tests
* improve: Improve task selection
* fix: Fix ci
* fix: Update fastembed/text/multitask_embedding.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/text/multitask_embedding.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* fix: Pass task id using kwargs to parallel processor
* tests: Added test for task assignment
* prefer enums over ints
* tests: Added test for parallel
* improve: Updated model description
* fix: Fix ci
* fix: Fix ci
* refactor: Refactor query_embed and passage_embed
* tests: Added task propagation to parallel
* refactor: Set default task as retrieval passage
* chore: Update default task in tests
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* Merge master
* rerank_pairs interface + parallelism support
* remove test notebook
* Removed unused code
* New tests for cross encoders and new interface
* Importing Self fix. We will need it for mypy support in newer versions
* Removed Self typing
* Removed non-needed changes from text
* Isort + black
* wip: start reviewing (#420)
Co-authored-by: Dmitrii Ogn <dimitriy_rudenko@mail.ru>
* Test fix
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder_base.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Test for parallel processing + bugfix of PosixPath passing
* Removed non-needed import and added docstring
* Typing fix + argument passing
* Test parametrization
Moved to selected models set to test
* Run base test on all models
* Typing fix + improvement of input_names check
* nit: fix post process, update docstring, update tokenize, remove redundant imports
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* WIP: Added jina clip text embedding
* WIP: Added preprocess for jina clip
* WIP: Added jina clip vision (not sure if it works yet)
* improve: Improved mean pooling if the output doesnt have seq length
* fix: Fixed jina clip text
* nit
* fix: Fixed jina clip image preprocessor
* fix: Fix type hints
new: added resize2square
* tests: Add jina clip vision test case
* nit
* refactor: Update fastembed/image/transform/operators.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* fix: Fix indentation
* refactor: Refactored how we call padding for image
* fix: Fix pad to image when resized size larger than new square canvas
* refactor: minor refactor
* refactor: Refactor some functions in preprocess image
* fix: Fix to pad image with specified fill color
* refactor: Change resize to classmethod
* fix: Fix jina clip text v1
* fix: fix pad to square for some rectangular images (#421)
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* feat: Added a toggle to disable stemmer in bm25
* refactor: Refactored how to disable stemming in bm25
* refactor: Refactored the way of disabling stemmer in bm25
* new: Added english fallback if language = None
* tests: Added test case for disable stemmer
* fix: Fix language to be only string
* tests: Updated bm25 toggle stemmer tests
* refactor: fix stopwords type
* fix: fix param propagation in parallel embed in bm25
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* chore: Remove typing hints of Python less than 3.9
* chore: Removed optional from cache as it cannot be undefined
* improve: Turned off progress bar of huggingface models if cached
* feat: Added jina reranker models
* chore: Added jina reranker canonical score values
* chore: added rounding of the output for easier reproducability
* chore: Added jina reranker models in batch test
* chore: remove redundant np.round
* chore: test only <1gb files in local
* chore: Updated docs to add rerankers
* fix: recompute canonical values with fp16
* new: extend NOTICE with jina reranker v2
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* feat: Added support for jina-colbert-v2
* chore: Generalized query marker and document marker
* nit: remove github action on dispatch
* chore: updated license
* fix: Fix attention mask to be all 1 in xlmrobertatokenizer
* feat: Added class for JinaColbertV2
* feat: Added jina colbert
* chore: Change tolerance of the test
* chore: Changed encoding of attention mask to 1 to be only in queries
* chore: Changed the replacable token to be ' @' as its considered as one token
* chore: Removed redundant functions
* chore: Updated supported models docs
* nit: Remove print statement
* nit: visual stuff
* fix: Fix dimention of jina colbert in description
* fix: canonical query and document values for jina colbert
* feat: Added multi gpu support for text embedding
* feat: Add support for multi-gpu for special text models
* fix: Fix lazy_load to load the model to child processes when parallel is not none
* feat: Added lazy_load and multi-gpu to colbert
* feat: Add lazy_load and multi gpu to image models
* feat: Support lazy_load and multi-gpu to sparse models (except BM25)
* fix: Fixed BM25 not working
* refactor: Remove redundant GPUParallelProcessor
* refactor: Refactor _embed_*_parallel
* feat: Add cuda argument
refactor: Refactor how worker assign device
* fix: Fix if providers and cuda are None
* fix: Fix providers and cuda are none
* WIP: Multi gpu support review (#361)
* WIP: review
* wip: review
* refactor: refactor images
* refactor: refactor sparse
* refactor: refactor late interaction
* add model loading
* add tests
* fix: uncomment models in tests
* fix: fix variable declaration order
* fix: fix device id assignment
* tests: add multi gpu tests
* fix: fix device id assignment for sparse embeddings
* tests: update multi gpu tests
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* refactor: remove redundant declarations
* fix: rollback redundant changes
* fix: remove num workers device ids dep, fix type hint
* fix: fix post process for sparse models
* fix: remove redundant model loading
* new: add lazy load and new gpu support to cross encoders
* fix: add rerankers to multi gpu tests
* fix: unlock multilingual test
* fix: fix gpu test with cross encoder
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* Tf-idf fix:
Remove punctuation properly + lowercase
* Test fix
* Test fix
* Test fix
* Redudant accum removal
* Redudant accum removal
* 0.878890950070688
* Latest changes with
Average recall: 0.8915516690721613
* Support of special characters
* Removal of debug print
* Type annotations
* Type annotation support for python 3.8
* Type annotation support for python 3.8
* wip: remove model dir
* fix: update pytest run
* wip: disable some tests not used atm
* wip: disable some tests in ci
* wip: add debug print
* fix: fix ci, remove models after usage
* fix: fix bm25 deletion
* fix: remove redundant ci commands
* Fix: Normalize tokens to lowercase before checking stopwords in BM25
* Test: Normalize tokens to lowercase before checking stopwords in BM25
* Test Fix test_multilanguage: in "Je suis au lit", the "Je" should be skipped because it in the stopwords.
* chore: apply ruff
---------
Co-authored-by: H4-8ZSI <H4-8ZSI@EXAMPLE.COM>
* Fix to avoid overfloat and get rid of model_max_length
* Fixes for max_length vs model_max_length logic
Jupter warning disabled
* Support of https://github.com/jwodder/versioningit/issues/48
* Update fastembed/common/preprocessor_utils.py
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* Initial commit for opened images support
* Additional tests for image embeddings
* Added selfish logo test as requests input
* Isort for image tests
* Support of multilanguage for bm25
Tests for french
* Tests refactoring
* PR requested changes
* Nomic-embeddings-support
* Jina models moved to pooled-normalized embeddings
* Canonical vector for nomic-ai/nomic-embed-text-v1.5-Q
* Moved all nomics to pooled_embeddings
---------
Co-authored-by: d.rudenko <dimitriyrudenk@gmail.com>
* MiniLM fix
* Added MiniLM to text embedding
Fixed MiniLM source destination
Black + isort for repo
* Fixed model all-MiniLM-L6-v2 description
Recomputed canonical vector for all-MiniLM-L6-v2 in test
---------
Co-authored-by: d.rudenko <dimitriyrudenk@gmail.com>
* feat: add support for SOTA german embedding model with long context length jinaai/jina-embeddings-v2-base-de
* Fix jina de model weight
---------
Co-authored-by: George <panchuk.george@outlook.com>
* Resnet support added
* Tests fixed
Shapes matching for Resnet50-onnx
Example of Resnet50 to onnx conversion (basic)
* Removed optional conversion from PIL to np.ndarray and now it it's made default
Fixed test accordingly
* Refactoring of pil2ndarray
* Partial support of convnext preprocessing
Resize logic
* normalize canonical value
* Style changes for review
* new: update resnet repo
---------
Co-authored-by: d.rudenko <dimitriyrudenk@gmail.com>
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* Add SPLADE v1
* WIP SPLADE Export errors
* add ONNX model to HF hub and use that
* Update sentences in Converting_SPLADE_to_ONNX.ipynb
* Remove unnecessary files and directories
* Rename var in TextEmbedding class to use EMBEDDING_MODEL_TYPE
* Add SPLADE to list of text embeddings
* Add SPLADE model support for text embedding
* Fix deprecation warning in embedding.py
* Add test for batch embedding with sparse embeddings
* Refactor import statement in test_sparse_embeddings.py
* Rename nbs
* Update vocab size in SPLADE model
* Fix canonical vector lookup in test_text_onnx_embeddings.py
* review refactoring
* restore list_supported_models in OnnxTextEmbedding
* Remove unused method _preprocess_onnx_input() in SpladePP class
* Update SPLADE_PP_en_v1 source in splade_pp.py
* Refactor onnx_model.py to change base model behavior
* extend tests to sparse values as well as indicies
* chore: pre-commit hooks
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Anush008 <anushshetty90@gmail.com>
