* Migration of models to dataclasses
* Model description file
* Test fix
* kw_only support
* Multitask embeddings test fix
* list_supported_models type fix
* Dim fix for sparsemodels
* Dim fix for sparsemodels
* Dim fix for sparsemodels (x2)
* Model management type fix
* Interface docstring fixes
* Mypy fixes
* Typing fix again
* Typing fix again
* Special cast to SparseModelDescription
* Special cast to SparseModelDescription
* Special cast to SparseModelDescription
* typing fix for colpali
* typing fix for colpali
* typing fix for colpali
* typing fix for colpali
* Let's try generic typing for ModelManagment
* wip: dataclass idea, small fixes (#475)
* wip: dataclass idea, small fixes
* fix: fix exception message in base model description
* remove custom model descriptions
* make license, description and size in gb mandatory in model description
* fix: introduce _list_supported_models which returns model description objects
* test: add test for list supported models
* fix: fix list supported models usage in tests
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* wip: design draft
* Operators fix
* Fix model inputs
* Import from fastembed.late_interaction_multimodal
* Fixed method misspelling
* Tests, which do not run in CI
Docstring improvements
* Fix tests
* Bump colpali to version v1.3
* Remove colpali v1.2
* Remove colpali v1.2 from tests
* partial fix of change requests:
descriptions
docs
black
* query_max_length
* black colpali
* Added comment for EMPTY_TEXT_PLACEHOLDER
* Review fixes
* Removed redundant VISUAL_PROMPT_PREFIX
* type fix + model info
* new: add specific model path to colpali
* fix: revert accidental renaming
* fix: remove max_length from encode_batch
* refactoring: remove redundant QUERY_MAX_LENGTH variable
* refactoring: remove redundant document marker token id
* fix: fix type hints, fix tests, handle single image path embed, rename model, update description
* license: add gemma to NOTICE
* fix: do not run colpali test in ci
* fix: fix colpali test
---------
Co-authored-by: d.rudenko <dmitrii.rudenko@qdrant.com>
* new: Added type stub
* chore: Updated stubs
* chore: device_id type hint
* chore: add -> none to init without args
* new: Added workflow type check
* chore: Revert added type checkers
* chore: Added type hints
* fix: Fix generic type
* fix: Fix generic type
* new: Add type hints for parallel processor
* fix: Revert queue sub type as its not supported
* fix: Revert queue sub type as its not supported
* fix: Fixed type hints
* chore: Updated type hints
* fix: Update task id to be public
* chore: Updated type hints
* chore: Updated type hints
* fix: minor reverts in parallel processor and onnx text model
* chore: Add missing type hints in functions
* add missing import, small type refactor
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* fix: Fix minilm paraphrase by adding it to pool models
* tests: Updated minilm paraphrase canonical vector
* chore: Added a warning message for updating the model
* chore: Added version where model will be removed
* new: Added jina embedding v3
* refactor: Changed dim to int value
* new: Updated notice
* new: Extended text embedding with query embed and passage embed
* fix: Fix lazy load in query and passage embed
* tests: Added test for multitask embeddings
* nit: Remove cache dir from tests
* tests: Updated tests
* improve: Improve task selection
* fix: Fix ci
* fix: Update fastembed/text/multitask_embedding.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/text/multitask_embedding.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* fix: Pass task id using kwargs to parallel processor
* tests: Added test for task assignment
* prefer enums over ints
* tests: Added test for parallel
* improve: Updated model description
* fix: Fix ci
* fix: Fix ci
* refactor: Refactor query_embed and passage_embed
* tests: Added task propagation to parallel
* refactor: Set default task as retrieval passage
* chore: Update default task in tests
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* Merge master
* rerank_pairs interface + parallelism support
* remove test notebook
* Removed unused code
* New tests for cross encoders and new interface
* Importing Self fix. We will need it for mypy support in newer versions
* Removed Self typing
* Removed non-needed changes from text
* Isort + black
* wip: start reviewing (#420)
Co-authored-by: Dmitrii Ogn <dimitriy_rudenko@mail.ru>
* Test fix
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder_base.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Test for parallel processing + bugfix of PosixPath passing
* Removed non-needed import and added docstring
* Typing fix + argument passing
* Test parametrization
Moved to selected models set to test
* Run base test on all models
* Typing fix + improvement of input_names check
* nit: fix post process, update docstring, update tokenize, remove redundant imports
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* WIP: Added jina clip text embedding
* WIP: Added preprocess for jina clip
* WIP: Added jina clip vision (not sure if it works yet)
* improve: Improved mean pooling if the output doesnt have seq length
* fix: Fixed jina clip text
* nit
* fix: Fixed jina clip image preprocessor
* fix: Fix type hints
new: added resize2square
* tests: Add jina clip vision test case
* nit
* refactor: Update fastembed/image/transform/operators.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* fix: Fix indentation
* refactor: Refactored how we call padding for image
* fix: Fix pad to image when resized size larger than new square canvas
* refactor: minor refactor
* refactor: Refactor some functions in preprocess image
* fix: Fix to pad image with specified fill color
* refactor: Change resize to classmethod
* fix: Fix jina clip text v1
* fix: fix pad to square for some rectangular images (#421)
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* feat: Added a toggle to disable stemmer in bm25
* refactor: Refactored how to disable stemming in bm25
* refactor: Refactored the way of disabling stemmer in bm25
* new: Added english fallback if language = None
* tests: Added test case for disable stemmer
* fix: Fix language to be only string
* tests: Updated bm25 toggle stemmer tests
* refactor: fix stopwords type
* fix: fix param propagation in parallel embed in bm25
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* chore: Remove typing hints of Python less than 3.9
* chore: Removed optional from cache as it cannot be undefined
* improve: Turned off progress bar of huggingface models if cached
* feat: Added jina reranker models
* chore: Added jina reranker canonical score values
* chore: added rounding of the output for easier reproducability
* chore: Added jina reranker models in batch test
* chore: remove redundant np.round
* chore: test only <1gb files in local
* chore: Updated docs to add rerankers
* fix: recompute canonical values with fp16
* new: extend NOTICE with jina reranker v2
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* feat: Added support for jina-colbert-v2
* chore: Generalized query marker and document marker
* nit: remove github action on dispatch
* chore: updated license
* fix: Fix attention mask to be all 1 in xlmrobertatokenizer
* feat: Added class for JinaColbertV2
* feat: Added jina colbert
* chore: Change tolerance of the test
* chore: Changed encoding of attention mask to 1 to be only in queries
* chore: Changed the replacable token to be ' @' as its considered as one token
* chore: Removed redundant functions
* chore: Updated supported models docs
* nit: Remove print statement
* nit: visual stuff
* fix: Fix dimention of jina colbert in description
* fix: canonical query and document values for jina colbert
* feat: Added multi gpu support for text embedding
* feat: Add support for multi-gpu for special text models
* fix: Fix lazy_load to load the model to child processes when parallel is not none
* feat: Added lazy_load and multi-gpu to colbert
* feat: Add lazy_load and multi gpu to image models
* feat: Support lazy_load and multi-gpu to sparse models (except BM25)
* fix: Fixed BM25 not working
* refactor: Remove redundant GPUParallelProcessor
* refactor: Refactor _embed_*_parallel
* feat: Add cuda argument
refactor: Refactor how worker assign device
* fix: Fix if providers and cuda are None
* fix: Fix providers and cuda are none
* WIP: Multi gpu support review (#361)
* WIP: review
* wip: review
* refactor: refactor images
* refactor: refactor sparse
* refactor: refactor late interaction
* add model loading
* add tests
* fix: uncomment models in tests
* fix: fix variable declaration order
* fix: fix device id assignment
* tests: add multi gpu tests
* fix: fix device id assignment for sparse embeddings
* tests: update multi gpu tests
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* refactor: remove redundant declarations
* fix: rollback redundant changes
* fix: remove num workers device ids dep, fix type hint
* fix: fix post process for sparse models
* fix: remove redundant model loading
* new: add lazy load and new gpu support to cross encoders
* fix: add rerankers to multi gpu tests
* fix: unlock multilingual test
* fix: fix gpu test with cross encoder
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* Tf-idf fix:
Remove punctuation properly + lowercase
* Test fix
* Test fix
* Test fix
* Redudant accum removal
* Redudant accum removal
* 0.878890950070688
* Latest changes with
Average recall: 0.8915516690721613
* Support of special characters
* Removal of debug print
* Type annotations
* Type annotation support for python 3.8
* Type annotation support for python 3.8
* wip: remove model dir
* fix: update pytest run
* wip: disable some tests not used atm
* wip: disable some tests in ci
* wip: add debug print
* fix: fix ci, remove models after usage
* fix: fix bm25 deletion
* fix: remove redundant ci commands
* Fix: Normalize tokens to lowercase before checking stopwords in BM25
* Test: Normalize tokens to lowercase before checking stopwords in BM25
* Test Fix test_multilanguage: in "Je suis au lit", the "Je" should be skipped because it in the stopwords.
* chore: apply ruff
---------
Co-authored-by: H4-8ZSI <H4-8ZSI@EXAMPLE.COM>
* Fix to avoid overfloat and get rid of model_max_length
* Fixes for max_length vs model_max_length logic
Jupter warning disabled
* Support of https://github.com/jwodder/versioningit/issues/48
* Update fastembed/common/preprocessor_utils.py
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* Initial commit for opened images support
* Additional tests for image embeddings
* Added selfish logo test as requests input
* Isort for image tests
* Support of multilanguage for bm25
Tests for french
* Tests refactoring
* PR requested changes
* Nomic-embeddings-support
* Jina models moved to pooled-normalized embeddings
* Canonical vector for nomic-ai/nomic-embed-text-v1.5-Q
* Moved all nomics to pooled_embeddings
---------
Co-authored-by: d.rudenko <dimitriyrudenk@gmail.com>
* MiniLM fix
* Added MiniLM to text embedding
Fixed MiniLM source destination
Black + isort for repo
* Fixed model all-MiniLM-L6-v2 description
Recomputed canonical vector for all-MiniLM-L6-v2 in test
---------
Co-authored-by: d.rudenko <dimitriyrudenk@gmail.com>
* feat: add support for SOTA german embedding model with long context length jinaai/jina-embeddings-v2-base-de
* Fix jina de model weight
---------
Co-authored-by: George <panchuk.george@outlook.com>
* Resnet support added
* Tests fixed
Shapes matching for Resnet50-onnx
Example of Resnet50 to onnx conversion (basic)
* Removed optional conversion from PIL to np.ndarray and now it it's made default
Fixed test accordingly
* Refactoring of pil2ndarray
* Partial support of convnext preprocessing
Resize logic
* normalize canonical value
* Style changes for review
* new: update resnet repo
---------
Co-authored-by: d.rudenko <dimitriyrudenk@gmail.com>
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* Add SPLADE v1
* WIP SPLADE Export errors
* add ONNX model to HF hub and use that
* Update sentences in Converting_SPLADE_to_ONNX.ipynb
* Remove unnecessary files and directories
* Rename var in TextEmbedding class to use EMBEDDING_MODEL_TYPE
* Add SPLADE to list of text embeddings
* Add SPLADE model support for text embedding
* Fix deprecation warning in embedding.py
* Add test for batch embedding with sparse embeddings
* Refactor import statement in test_sparse_embeddings.py
* Rename nbs
* Update vocab size in SPLADE model
* Fix canonical vector lookup in test_text_onnx_embeddings.py
* review refactoring
* restore list_supported_models in OnnxTextEmbedding
* Remove unused method _preprocess_onnx_input() in SpladePP class
* Update SPLADE_PP_en_v1 source in splade_pp.py
* Refactor onnx_model.py to change base model behavior
* extend tests to sparse values as well as indicies
* chore: pre-commit hooks
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Anush008 <anushshetty90@gmail.com>
