* feat: Added support for jina-colbert-v2
* chore: Generalized query marker and document marker
* nit: remove github action on dispatch
* chore: updated license
* fix: Fix attention mask to be all 1 in xlmrobertatokenizer
* feat: Added class for JinaColbertV2
* feat: Added jina colbert
* chore: Change tolerance of the test
* chore: Changed encoding of attention mask to 1 to be only in queries
* chore: Changed the replacable token to be ' @' as its considered as one token
* chore: Removed redundant functions
* chore: Updated supported models docs
* nit: Remove print statement
* nit: visual stuff
* fix: Fix dimention of jina colbert in description
* fix: canonical query and document values for jina colbert
* bump version to 0.4.0
* py-rust-stemmers support instead of snowball stemmer and pystemmer
* py-rust-stemmers support instead of snowball stemmer and pystemmer
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* fix: Fix type when calling parallelworkerpool
parallelworkerpool accepts num_workers as int only
* fix: Fix type OnnxOutputContext.
OnnxOutputContext takes attention_mask and input_ids as optional while we cannot assign none to ndarray
* fix: Fix progress might not be bool and might be literal 0
* feat: Added multi gpu support for text embedding
* feat: Add support for multi-gpu for special text models
* fix: Fix lazy_load to load the model to child processes when parallel is not none
* feat: Added lazy_load and multi-gpu to colbert
* feat: Add lazy_load and multi gpu to image models
* feat: Support lazy_load and multi-gpu to sparse models (except BM25)
* fix: Fixed BM25 not working
* refactor: Remove redundant GPUParallelProcessor
* refactor: Refactor _embed_*_parallel
* feat: Add cuda argument
refactor: Refactor how worker assign device
* fix: Fix if providers and cuda are None
* fix: Fix providers and cuda are none
* WIP: Multi gpu support review (#361)
* WIP: review
* wip: review
* refactor: refactor images
* refactor: refactor sparse
* refactor: refactor late interaction
* add model loading
* add tests
* fix: uncomment models in tests
* fix: fix variable declaration order
* fix: fix device id assignment
* tests: add multi gpu tests
* fix: fix device id assignment for sparse embeddings
* tests: update multi gpu tests
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* refactor: remove redundant declarations
* fix: rollback redundant changes
* fix: remove num workers device ids dep, fix type hint
* fix: fix post process for sparse models
* fix: remove redundant model loading
* new: add lazy load and new gpu support to cross encoders
* fix: add rerankers to multi gpu tests
* fix: unlock multilingual test
* fix: fix gpu test with cross encoder
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* chore: Updating docs of installing fastembed-gpu
* chore: updated common issues
* chore: Update gcp setup
* chore: Updated an example of installing cuda12 and cudnn9 on ubuntu 22.04
* rephrasing
* update readme
---------
Co-authored-by: George Panchuk <george.panchuk@qdrant.tech>
* fix: Fix deadlock when child gets kill -9 sig
* chore: Better cleanup for resources
* chore: changed place of processes.clear
* fix: Added cancle_join_thread for emergency shutdown
* Tf-idf fix:
Remove punctuation properly + lowercase
* Test fix
* Test fix
* Test fix
* Redudant accum removal
* Redudant accum removal
* 0.878890950070688
* Latest changes with
Average recall: 0.8915516690721613
* Support of special characters
* Removal of debug print
* Type annotations
* Type annotation support for python 3.8
* Type annotation support for python 3.8
* wip: remove model dir
* fix: update pytest run
* wip: disable some tests not used atm
* wip: disable some tests in ci
* wip: add debug print
* fix: fix ci, remove models after usage
* fix: fix bm25 deletion
* fix: remove redundant ci commands
* Fix: Normalize tokens to lowercase before checking stopwords in BM25
* Test: Normalize tokens to lowercase before checking stopwords in BM25
* Test Fix test_multilanguage: in "Je suis au lit", the "Je" should be skipped because it in the stopwords.
* chore: apply ruff
---------
Co-authored-by: H4-8ZSI <H4-8ZSI@EXAMPLE.COM>
* Fix to avoid overfloat and get rid of model_max_length
* Fixes for max_length vs model_max_length logic
Jupter warning disabled
* Support of https://github.com/jwodder/versioningit/issues/48
* Update fastembed/common/preprocessor_utils.py
---------
Co-authored-by: George <george.panchuk@qdrant.tech>
* Description of text embedding models, fix for consistency
* fixed misplacing of one description
---------
Co-authored-by: Evgeniya Sukhodolskaya <evgeniya.sukhodolskaya@tum.de>
* Initial commit for opened images support
* Additional tests for image embeddings
* Added selfish logo test as requests input
* Isort for image tests
* Support of multilanguage for bm25
Tests for french
* Tests refactoring
* PR requested changes
* Nomic-embeddings-support
* Jina models moved to pooled-normalized embeddings
* Canonical vector for nomic-ai/nomic-embed-text-v1.5-Q
* Moved all nomics to pooled_embeddings
---------
Co-authored-by: d.rudenko <dimitriyrudenk@gmail.com>
