* feat: Added support for jina-colbert-v2
* chore: Generalized query marker and document marker
* nit: remove github action on dispatch
* chore: updated license
* fix: Fix attention mask to be all 1 in xlmrobertatokenizer
* feat: Added class for JinaColbertV2
* feat: Added jina colbert
* chore: Change tolerance of the test
* chore: Changed encoding of attention mask to 1 to be only in queries
* chore: Changed the replacable token to be ' @' as its considered as one token
* chore: Removed redundant functions
* chore: Updated supported models docs
* nit: Remove print statement
* nit: visual stuff
* fix: Fix dimention of jina colbert in description
* fix: canonical query and document values for jina colbert
* feat: Added multi gpu support for text embedding
* feat: Add support for multi-gpu for special text models
* fix: Fix lazy_load to load the model to child processes when parallel is not none
* feat: Added lazy_load and multi-gpu to colbert
* feat: Add lazy_load and multi gpu to image models
* feat: Support lazy_load and multi-gpu to sparse models (except BM25)
* fix: Fixed BM25 not working
* refactor: Remove redundant GPUParallelProcessor
* refactor: Refactor _embed_*_parallel
* feat: Add cuda argument
refactor: Refactor how worker assign device
* fix: Fix if providers and cuda are None
* fix: Fix providers and cuda are none
* WIP: Multi gpu support review (#361)
* WIP: review
* wip: review
* refactor: refactor images
* refactor: refactor sparse
* refactor: refactor late interaction
* add model loading
* add tests
* fix: uncomment models in tests
* fix: fix variable declaration order
* fix: fix device id assignment
* tests: add multi gpu tests
* fix: fix device id assignment for sparse embeddings
* tests: update multi gpu tests
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* refactor: remove redundant declarations
* fix: rollback redundant changes
* fix: remove num workers device ids dep, fix type hint
* fix: fix post process for sparse models
* fix: remove redundant model loading
* new: add lazy load and new gpu support to cross encoders
* fix: add rerankers to multi gpu tests
* fix: unlock multilingual test
* fix: fix gpu test with cross encoder
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* Tf-idf fix:
Remove punctuation properly + lowercase
* Test fix
* Test fix
* Test fix
* Redudant accum removal
* Redudant accum removal
* 0.878890950070688
* Latest changes with
Average recall: 0.8915516690721613
* Support of special characters
* Removal of debug print
* Type annotations
* Type annotation support for python 3.8
* Type annotation support for python 3.8
* wip: remove model dir
* fix: update pytest run
* wip: disable some tests not used atm
* wip: disable some tests in ci
* wip: add debug print
* fix: fix ci, remove models after usage
* fix: fix bm25 deletion
* fix: remove redundant ci commands
* Fix: Normalize tokens to lowercase before checking stopwords in BM25
* Test: Normalize tokens to lowercase before checking stopwords in BM25
* Test Fix test_multilanguage: in "Je suis au lit", the "Je" should be skipped because it in the stopwords.
* chore: apply ruff
---------
Co-authored-by: H4-8ZSI <H4-8ZSI@EXAMPLE.COM>
* Fix to avoid overfloat and get rid of model_max_length
* Fixes for max_length vs model_max_length logic
Jupter warning disabled
* Support of https://github.com/jwodder/versioningit/issues/48
* Update fastembed/common/preprocessor_utils.py
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* Initial commit for opened images support
* Additional tests for image embeddings
* Added selfish logo test as requests input
* Isort for image tests
* Support of multilanguage for bm25
Tests for french
* Tests refactoring
* PR requested changes
* Nomic-embeddings-support
* Jina models moved to pooled-normalized embeddings
* Canonical vector for nomic-ai/nomic-embed-text-v1.5-Q
* Moved all nomics to pooled_embeddings
---------
Co-authored-by: d.rudenko <dimitriyrudenk@gmail.com>
* MiniLM fix
* Added MiniLM to text embedding
Fixed MiniLM source destination
Black + isort for repo
* Fixed model all-MiniLM-L6-v2 description
Recomputed canonical vector for all-MiniLM-L6-v2 in test
---------
Co-authored-by: d.rudenko <dimitriyrudenk@gmail.com>
* feat: add support for SOTA german embedding model with long context length jinaai/jina-embeddings-v2-base-de
* Fix jina de model weight
---------
Co-authored-by: George <panchuk.george@outlook.com>
* Resnet support added
* Tests fixed
Shapes matching for Resnet50-onnx
Example of Resnet50 to onnx conversion (basic)
* Removed optional conversion from PIL to np.ndarray and now it it's made default
Fixed test accordingly
* Refactoring of pil2ndarray
* Partial support of convnext preprocessing
Resize logic
* normalize canonical value
* Style changes for review
* new: update resnet repo
---------
Co-authored-by: d.rudenko <dimitriyrudenk@gmail.com>
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* Add SPLADE v1
* WIP SPLADE Export errors
* add ONNX model to HF hub and use that
* Update sentences in Converting_SPLADE_to_ONNX.ipynb
* Remove unnecessary files and directories
* Rename var in TextEmbedding class to use EMBEDDING_MODEL_TYPE
* Add SPLADE to list of text embeddings
* Add SPLADE model support for text embedding
* Fix deprecation warning in embedding.py
* Add test for batch embedding with sparse embeddings
* Refactor import statement in test_sparse_embeddings.py
* Rename nbs
* Update vocab size in SPLADE model
* Fix canonical vector lookup in test_text_onnx_embeddings.py
* review refactoring
* restore list_supported_models in OnnxTextEmbedding
* Remove unused method _preprocess_onnx_input() in SpladePP class
* Update SPLADE_PP_en_v1 source in splade_pp.py
* Refactor onnx_model.py to change base model behavior
* extend tests to sparse values as well as indicies
* chore: pre-commit hooks
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Anush008 <anushshetty90@gmail.com>
