This causes an incompatibility with spaCy because no matching
versions of numpy can be found when using both libraries at once.
It looks like the tests pass again without the numpy<2 requirement
so there is no reason to keep this requirement anymore.
* feat: Added jina reranker models
* chore: Added jina reranker canonical score values
* chore: added rounding of the output for easier reproducability
* chore: Added jina reranker models in batch test
* chore: remove redundant np.round
* chore: test only <1gb files in local
* chore: Updated docs to add rerankers
* fix: recompute canonical values with fp16
* new: extend NOTICE with jina reranker v2
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* fix: Fix return of onnx_embed
* fix: Fix type hint of start method in worker class
* fix: Fix not passing kwargs in _preprocess_onnx_input and tokenize as base class
* fix: Fix not passing kwargs in _preprocess_onnx_input as base class
* fix: change tokenize in simpleTokenizer to classmethod
* chore: Changed query argument to Iterable to match base class
* chore: changed mask token id and pad token id to be int
* review suggestions (#398)
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* improve: Changed the way we are adding the query and document markers in colbert
* fix: Truncate the inout_ids and attention_mask when adding query and document markers to original input length
* fix: Fix broadcast issue
* chore: Remove redundant if condition
* nit
* refactor (#397)
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* Add rerankers to the list of supported models (Supported_Models.ipynb)
* fix: update imports
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* feat: Added support for jina-colbert-v2
* chore: Generalized query marker and document marker
* nit: remove github action on dispatch
* chore: updated license
* fix: Fix attention mask to be all 1 in xlmrobertatokenizer
* feat: Added class for JinaColbertV2
* feat: Added jina colbert
* chore: Change tolerance of the test
* chore: Changed encoding of attention mask to 1 to be only in queries
* chore: Changed the replacable token to be ' @' as its considered as one token
* chore: Removed redundant functions
* chore: Updated supported models docs
* nit: Remove print statement
* nit: visual stuff
* fix: Fix dimention of jina colbert in description
* fix: canonical query and document values for jina colbert
* bump version to 0.4.0
* py-rust-stemmers support instead of snowball stemmer and pystemmer
* py-rust-stemmers support instead of snowball stemmer and pystemmer
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* fix: Fix type when calling parallelworkerpool
parallelworkerpool accepts num_workers as int only
* fix: Fix type OnnxOutputContext.
OnnxOutputContext takes attention_mask and input_ids as optional while we cannot assign none to ndarray
* fix: Fix progress might not be bool and might be literal 0
* feat: Added multi gpu support for text embedding
* feat: Add support for multi-gpu for special text models
* fix: Fix lazy_load to load the model to child processes when parallel is not none
* feat: Added lazy_load and multi-gpu to colbert
* feat: Add lazy_load and multi gpu to image models
* feat: Support lazy_load and multi-gpu to sparse models (except BM25)
* fix: Fixed BM25 not working
* refactor: Remove redundant GPUParallelProcessor
* refactor: Refactor _embed_*_parallel
* feat: Add cuda argument
refactor: Refactor how worker assign device
* fix: Fix if providers and cuda are None
* fix: Fix providers and cuda are none
* WIP: Multi gpu support review (#361)
* WIP: review
* wip: review
* refactor: refactor images
* refactor: refactor sparse
* refactor: refactor late interaction
* add model loading
* add tests
* fix: uncomment models in tests
* fix: fix variable declaration order
* fix: fix device id assignment
* tests: add multi gpu tests
* fix: fix device id assignment for sparse embeddings
* tests: update multi gpu tests
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* refactor: remove redundant declarations
* fix: rollback redundant changes
* fix: remove num workers device ids dep, fix type hint
* fix: fix post process for sparse models
* fix: remove redundant model loading
* new: add lazy load and new gpu support to cross encoders
* fix: add rerankers to multi gpu tests
* fix: unlock multilingual test
* fix: fix gpu test with cross encoder
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* chore: Updating docs of installing fastembed-gpu
* chore: updated common issues
* chore: Update gcp setup
* chore: Updated an example of installing cuda12 and cudnn9 on ubuntu 22.04
* rephrasing
* update readme
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* fix: Fix deadlock when child gets kill -9 sig
* chore: Better cleanup for resources
* chore: changed place of processes.clear
* fix: Added cancle_join_thread for emergency shutdown
* Tf-idf fix:
Remove punctuation properly + lowercase
* Test fix
* Test fix
* Test fix
* Redudant accum removal
* Redudant accum removal
* 0.878890950070688
* Latest changes with
Average recall: 0.8915516690721613
* Support of special characters
* Removal of debug print
* Type annotations
* Type annotation support for python 3.8
* Type annotation support for python 3.8
* wip: remove model dir
* fix: update pytest run
* wip: disable some tests not used atm
* wip: disable some tests in ci
* wip: add debug print
* fix: fix ci, remove models after usage
* fix: fix bm25 deletion
* fix: remove redundant ci commands
* Fix: Normalize tokens to lowercase before checking stopwords in BM25
* Test: Normalize tokens to lowercase before checking stopwords in BM25
* Test Fix test_multilanguage: in "Je suis au lit", the "Je" should be skipped because it in the stopwords.
* chore: apply ruff
---------
Co-authored-by: H4-8ZSI <H4-8ZSI@EXAMPLE.COM>
* Fix to avoid overfloat and get rid of model_max_length
* Fixes for max_length vs model_max_length logic
Jupter warning disabled
* Support of https://github.com/jwodder/versioningit/issues/48
* Update fastembed/common/preprocessor_utils.py
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* Description of text embedding models, fix for consistency
* fixed misplacing of one description
---------
Co-authored-by: Evgeniya Sukhodolskaya <evgeniya.sukhodolskaya@tum.de>
* Initial commit for opened images support
* Additional tests for image embeddings
* Added selfish logo test as requests input
* Isort for image tests
* Support of multilanguage for bm25
Tests for french
* Tests refactoring
* PR requested changes
* Nomic-embeddings-support
* Jina models moved to pooled-normalized embeddings
* Canonical vector for nomic-ai/nomic-embed-text-v1.5-Q
* Moved all nomics to pooled_embeddings
---------
Co-authored-by: d.rudenko <dimitriyrudenk@gmail.com>
