* Migration of models to dataclasses
* Model description file
* Test fix
* kw_only support
* Multitask embeddings test fix
* list_supported_models type fix
* Dim fix for sparsemodels
* Dim fix for sparsemodels
* Dim fix for sparsemodels (x2)
* Model management type fix
* Interface docstring fixes
* Mypy fixes
* Typing fix again
* Typing fix again
* Special cast to SparseModelDescription
* Special cast to SparseModelDescription
* Special cast to SparseModelDescription
* typing fix for colpali
* typing fix for colpali
* typing fix for colpali
* typing fix for colpali
* Let's try generic typing for ModelManagment
* wip: dataclass idea, small fixes (#475)
* wip: dataclass idea, small fixes
* fix: fix exception message in base model description
* remove custom model descriptions
* make license, description and size in gb mandatory in model description
* fix: introduce _list_supported_models which returns model description objects
* test: add test for list supported models
* fix: fix list supported models usage in tests
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* new: Add mypy type checker
* fix: fix mypy command
* fix: fix indentation for mypy
* fix: do not install redundant optional groups cuz python3.13 does not support onnx
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* new: Add missing type hints
* refactor: Removed type ignore
* fix: fix mypy complaints
* fix: remove redundant type coercion, fix skip list type
* new: more precise type for sparse embedding inference, a small revert for parallel processor
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* wip: design draft
* Operators fix
* Fix model inputs
* Import from fastembed.late_interaction_multimodal
* Fixed method misspelling
* Tests, which do not run in CI
Docstring improvements
* Fix tests
* Bump colpali to version v1.3
* Remove colpali v1.2
* Remove colpali v1.2 from tests
* partial fix of change requests:
descriptions
docs
black
* query_max_length
* black colpali
* Added comment for EMPTY_TEXT_PLACEHOLDER
* Review fixes
* Removed redundant VISUAL_PROMPT_PREFIX
* type fix + model info
* new: add specific model path to colpali
* fix: revert accidental renaming
* fix: remove max_length from encode_batch
* refactoring: remove redundant QUERY_MAX_LENGTH variable
* refactoring: remove redundant document marker token id
* fix: fix type hints, fix tests, handle single image path embed, rename model, update description
* license: add gemma to NOTICE
* fix: do not run colpali test in ci
* fix: fix colpali test
---------
Co-authored-by: d.rudenko <dmitrii.rudenko@qdrant.com>
* new: Added type stub
* chore: Updated stubs
* chore: device_id type hint
* chore: add -> none to init without args
* new: Added workflow type check
* chore: Revert added type checkers
* chore: Added type hints
* fix: Fix generic type
* fix: Fix generic type
* new: Add type hints for parallel processor
* fix: Revert queue sub type as its not supported
* fix: Revert queue sub type as its not supported
* fix: Fixed type hints
* chore: Updated type hints
* fix: Update task id to be public
* chore: Updated type hints
* chore: Updated type hints
* fix: minor reverts in parallel processor and onnx text model
* chore: Add missing type hints in functions
* add missing import, small type refactor
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* new: Add mypy and pyright deps
* chore: Add any to kwargs
* chore: Add any to args
* add missing kwargs
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* HF sources for all models
* Specific_model_path model path support
* Fix hf download
* fix: rollback incorrect model replacement
* refactor: remove redundant type imports
* refactor: replace List with list
* fix: remove redundant param in late interaction text embedding
* Update fastembed/common/model_management.py
* fix: rollback post process onnx output
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* fix: Fix minilm paraphrase by adding it to pool models
* tests: Updated minilm paraphrase canonical vector
* chore: Added a warning message for updating the model
* chore: Added version where model will be removed
* new: Added jina embedding v3
* refactor: Changed dim to int value
* new: Updated notice
* new: Extended text embedding with query embed and passage embed
* fix: Fix lazy load in query and passage embed
* tests: Added test for multitask embeddings
* nit: Remove cache dir from tests
* tests: Updated tests
* improve: Improve task selection
* fix: Fix ci
* fix: Update fastembed/text/multitask_embedding.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/text/multitask_embedding.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* fix: Pass task id using kwargs to parallel processor
* tests: Added test for task assignment
* prefer enums over ints
* tests: Added test for parallel
* improve: Updated model description
* fix: Fix ci
* fix: Fix ci
* refactor: Refactor query_embed and passage_embed
* tests: Added task propagation to parallel
* refactor: Set default task as retrieval passage
* chore: Update default task in tests
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* Merge master
* rerank_pairs interface + parallelism support
* remove test notebook
* Removed unused code
* New tests for cross encoders and new interface
* Importing Self fix. We will need it for mypy support in newer versions
* Removed Self typing
* Removed non-needed changes from text
* Isort + black
* wip: start reviewing (#420)
Co-authored-by: Dmitrii Ogn <dimitriy_rudenko@mail.ru>
* Test fix
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Update fastembed/rerank/cross_encoder/text_cross_encoder_base.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* Test for parallel processing + bugfix of PosixPath passing
* Removed non-needed import and added docstring
* Typing fix + argument passing
* Test parametrization
Moved to selected models set to test
* Run base test on all models
* Typing fix + improvement of input_names check
* nit: fix post process, update docstring, update tokenize, remove redundant imports
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* WIP: Added jina clip text embedding
* WIP: Added preprocess for jina clip
* WIP: Added jina clip vision (not sure if it works yet)
* improve: Improved mean pooling if the output doesnt have seq length
* fix: Fixed jina clip text
* nit
* fix: Fixed jina clip image preprocessor
* fix: Fix type hints
new: added resize2square
* tests: Add jina clip vision test case
* nit
* refactor: Update fastembed/image/transform/operators.py
Co-authored-by: George <george.panchuk@qdrant.tech>
* fix: Fix indentation
* refactor: Refactored how we call padding for image
* fix: Fix pad to image when resized size larger than new square canvas
* refactor: minor refactor
* refactor: Refactor some functions in preprocess image
* fix: Fix to pad image with specified fill color
* refactor: Change resize to classmethod
* fix: Fix jina clip text v1
* fix: fix pad to square for some rectangular images (#421)
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* feat: Added a toggle to disable stemmer in bm25
* refactor: Refactored how to disable stemming in bm25
* refactor: Refactored the way of disabling stemmer in bm25
* new: Added english fallback if language = None
* tests: Added test case for disable stemmer
* fix: Fix language to be only string
* tests: Updated bm25 toggle stemmer tests
* refactor: fix stopwords type
* fix: fix param propagation in parallel embed in bm25
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* chore: Remove typing hints of Python less than 3.9
* chore: Removed optional from cache as it cannot be undefined
* improve: Turned off progress bar of huggingface models if cached
This causes an incompatibility with spaCy because no matching
versions of numpy can be found when using both libraries at once.
It looks like the tests pass again without the numpy<2 requirement
so there is no reason to keep this requirement anymore.
* feat: Added jina reranker models
* chore: Added jina reranker canonical score values
* chore: added rounding of the output for easier reproducability
* chore: Added jina reranker models in batch test
* chore: remove redundant np.round
* chore: test only <1gb files in local
* chore: Updated docs to add rerankers
* fix: recompute canonical values with fp16
* new: extend NOTICE with jina reranker v2
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* fix: Fix return of onnx_embed
* fix: Fix type hint of start method in worker class
* fix: Fix not passing kwargs in _preprocess_onnx_input and tokenize as base class
* fix: Fix not passing kwargs in _preprocess_onnx_input as base class
* fix: change tokenize in simpleTokenizer to classmethod
* chore: Changed query argument to Iterable to match base class
* chore: changed mask token id and pad token id to be int
* review suggestions (#398)
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* improve: Changed the way we are adding the query and document markers in colbert
* fix: Truncate the inout_ids and attention_mask when adding query and document markers to original input length
* fix: Fix broadcast issue
* chore: Remove redundant if condition
* nit
* refactor (#397)
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* Add rerankers to the list of supported models (Supported_Models.ipynb)
* fix: update imports
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
