upstream-archive/fastembed
1
0
Fork 0
mirror of https://github.com/qdrant/fastembed.git synced 2026-07-23 11:20:51 -05:00
Code Issues Packages Projects Releases Wiki Activity
408 Commits 38 Branches 59 Tags
gpu-runtime-debug
Commit Graph

408 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
George Panchuk
d9e85fe670 debug: try out with markers 2024-05-01 10:51:32 +02:00
George Panchuk
383e12c760 fix: fix type hint for 3.8 2024-04-26 21:45:24 +02:00
George Panchuk
0d6801d960 fix: update poetry.lock 2024-04-26 21:45:24 +02:00
George Panchuk
bff1b67637 new: add gpu support, allow users to override providers 2024-04-26 21:45:24 +02:00
George
5cde2898bc new: remove slurm environment variables (#213) 2024-04-26 21:45:02 +02:00
Anush
ab7a99a748 feat: Quantized models (#201) 
* feat: Quantized models

* refactor: use model_file for GCS

* refactoring: refactor model downloading (#209)

* refactoring: refactor model downloading

* refactor: update docstring

Co-authored-by: Anush <anushshetty90@gmail.com>

* Update fastembed/common/model_management.py

Co-authored-by: George <george.panchuk@qdrant.tech>

* fix: model_file for Snowflake models

---------

Co-authored-by: George <george.panchuk@qdrant.tech>
 2024-04-26 17:27:16 +02:00
Anush
466886a317 ci: Schedule python-tests.yml (#211) 
* ci: Schedule python-tests.yml

* ci: use emojis

* ci: Bump action versions python-tests.yml

* ci: python-tests.yml
 2024-04-26 10:28:37 +05:30
Anush
cc4112d859 feat: Snowflake models (#207) 
* feat: Snowflake models

* Added snowflake/snowflake-arctic-embed-m

* docs: snowflake/snowflake-arctic-embed-m
 2024-04-19 19:00:10 +05:30
Nirant
864217a7d9 Re-run of identical hardware and generate graphs (#205) 2024-04-18 11:13:18 +05:30
Andrew Green
9bad44368e Workaround for running on SLURM (#198) 
* Workaround for running on SLURM

onnxruntime would usually get the number of threads from OMP_NUM_THREADS, but that isn't set on SLURM which handles the number of threads differently.

This addition tries to figure out if we're running under SLURM, and if so sets the session options accordingly using SLURM environment variables instead.

Tested working with latest versions of onnxruntime and fastembed on slurm 23.02.07

* onnxruntime requires number of threads be an integer

Caused by me having mis-matched version from my machine and the slurm cluster :(

* `os.getenv` returns None for unset environment variables, fix logic

Instead of empty string, which I thought it did
 2024-04-15 19:14:11 +05:30
dependabot[bot]
a5cab7a31a build(deps): bump idna from 3.6 to 3.7 (#195) 
Bumps [idna](https://github.com/kjd/idna) from 3.6 to 3.7.
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](https://github.com/kjd/idna/compare/v3.6...v3.7)

---
updated-dependencies:
- dependency-name: idna
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nirant <NirantK@users.noreply.github.com>
 2024-04-15 06:04:54 +05:30
Anush
e55c145924 chore: Exclude unused model repo files (#196) 
* chore: Exclude unused model files

* fix: blob pattern
 2024-04-12 20:15:02 +05:30
dependabot[bot]
ad02f60eea build(deps-dev): bump pillow from 10.2.0 to 10.3.0 (#186) 
Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.2.0 to 10.3.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](https://github.com/python-pillow/Pillow/compare/10.2.0...10.3.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nirant <NirantK@users.noreply.github.com>
 2024-04-08 16:21:32 +05:30
Nirant
2fcec07f1b Add Python version to the ISSUE_TEMPLATE (#188) 
* Add Python version

* Update .github/ISSUE_TEMPLATE/bug-report.yml

Co-authored-by: George <george.panchuk@qdrant.tech>

---------

Co-authored-by: George <george.panchuk@qdrant.tech>
 2024-04-08 16:05:29 +05:30
George
f340a73a3e refactoring: update binary quantization notebook (#180) 2024-04-02 14:45:27 +05:30
Nirant
335f673f3a Update bug-report.yml 2024-04-02 13:51:51 +05:30
Nirant
ee7ba0a536 Update bug-report.yml 2024-04-02 13:51:27 +05:30
George
912e95e5c8 fix: remove model archive if model extraction was not finished correctly (#179) 2024-04-01 20:11:32 +02:00
Nirant
c09909773e Add Execution Counts (#177) 
* Update notebooks

* Update FastEmbed usage across docs

* Refactor code for better readability and maintainability

* Clean outputs

* Change dataset

* Add numbers inline in output

* Remove inline outputs since I used :memory:

* Fix syntax error in Hindi_Tamil_RAG_with_Navarasa7B.ipynb
v0.2.6 		2024-04-01 20:48:26 +05:30
Nirant
3d2254d215 Bump version to 0.2.6 in pyproject.toml (#175) 2024-04-01 17:16:07 +05:30
George
e11f66bb55 refactoring: update imports in notebooks (#173) 
* new: simplify imports

* refactoring: update import

* refactoring: update imports in notebooks

* fix: fix notebook output

* Re-run notebook with revised imports

---------

Co-authored-by: Nirant Kasliwal <nirant.bits@gmail.com>
 2024-04-01 17:02:13 +05:30
Nirant
d6f9ace425 Update size_in_GB for BAAI/bge-small-en-v1.5 model (#176) 2024-04-01 16:40:45 +05:30
George
8b7b8476a5 fix: fix model sizes in supported models lists (#167) 
* fix: fix model sizes in supported models lists

* fix: remove redundant comment

* fix: fix test

* fix: update supported models notebook

* Consistentcy around quantization in supported_onnx_models

---------

Co-authored-by: Nirant <NirantK@users.noreply.github.com>
Co-authored-by: Nirant Kasliwal <nirant.bits@gmail.com>
 2024-04-01 16:18:49 +05:30
George
ae35a96bcb new: simplify imports (#171) 
* new: simplify imports

* refactoring: update import
 2024-04-01 12:58:26 +05:30
George
25671ec349 Update ruff (#172) 
* refactoring: reduce max line-length

* new: update ruff

---------

Co-authored-by: Nirant <NirantK@users.noreply.github.com>
 2024-04-01 12:55:15 +05:30
George
ce98631b9a Fix spladepp parallelism (#169) 
* fix: add get_worker_class implementation to spladepp

* fix: add tests for parallel embed for spladepp
 2024-04-01 10:22:24 +05:30
George
0a4ed42b58 fix: unify existing patterns, remove redundant (#168) 2024-03-30 14:40:19 +05:30
Nirant
e3d2e1dc44 Hybrid Search Tutorial (#165) 
* Re-organize docs

* Rename notebooks

* Move nbs

* Working Sparse and Dense Search

* Add RRF

* Refactor code to improve performance and readability

* Add ESCI label for the RRF results

* Update docs/examples/Hybrid_Search.ipynb

Co-authored-by: Anush  <anushshetty90@gmail.com>

* Update docs/examples/Hybrid_Search.ipynb

Co-authored-by: Anush  <anushshetty90@gmail.com>

* Remove unnecessary code and update vector format

---------

Co-authored-by: Anush <anushshetty90@gmail.com>
 2024-03-29 21:10:14 +05:30
Nirant
62c21b0237 Add misspelled version of SPLADE++ model for English (#161) 2024-03-22 21:06:03 +05:30
Anush
d791f38704 chore: case-insensitive model_management.py (#160) 2024-03-22 21:02:31 +05:30
Nirant Kasliwal
c651b2b539 Update SPLADE notebook with new sections 2024-03-22 15:18:34 +05:30
Yuvraj Wale
bc1e23849b feat: support mixedbread-ai/mxbai-embed-large-v1 (#158) 
* add: support mixedbread-ai/mxbai-embed-large-v1

* refactor: canonical vector

* Update fastembed/text/onnx_embedding.py

---------

Co-authored-by: Nirant <NirantK@users.noreply.github.com>
 2024-03-22 06:23:54 +05:30
Nirant
4db4839995 [PyPi Publish] Bump version to 0.2.5 in pyproject.toml (#156) 
* Bump version to 0.2.5 in pyproject.toml

* chore: case insensitive check (#157)
v0.2.5 		2024-03-20 19:11:44 +05:30
Nirant
96a2a9097a Fix model name typo + Add SPLADE notebook (#155) 
* Rename model + Add SPLADE notebook

* Update docs/examples/SPLADE_with_FastEmbed.ipynb

Co-authored-by: Anush  <anushshetty90@gmail.com>

* Update docs/examples/SPLADE_with_FastEmbed.ipynb

Co-authored-by: Anush  <anushshetty90@gmail.com>

* Update docs/examples/SPLADE_with_FastEmbed.ipynb

Co-authored-by: Anush  <anushshetty90@gmail.com>

* Update CANONICAL_COLUMN_VALUES in test_sparse_embeddings.py

---------

Co-authored-by: Anush <anushshetty90@gmail.com>
 2024-03-20 18:15:53 +05:30
Nirant
256b2265d5 Move CONTRIBUTING.md + Add Test for Adding New Models (#154) 
* Move CONTRIBUTING.md + Ad Test for Adding New Models

* Update CONTRIBUTING.md

Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>

---------

Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>
 2024-03-18 19:58:02 +05:30
Nirant
1e91c8d165 Fix Issue Template forms (#152) 
* Add CONTRIBUTING.md file with guidelines for contributing to FastEmbed

* Add code linting and pre-commit info to CONTRIBUTING

* Update CONTRIBUTING.md

Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>

* Add bug/new model issue template and move CONTRIBUTING.md

* Re-organize issue templates

---------

Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>
 2024-03-14 15:11:23 +05:30
Nirant
a761f456b2 Add import statement for version debugging (#151) 2024-03-14 15:02:15 +05:30
Nirant
287e19c494 Add CONTRIBUTING.md file with guidelines for contributing to FastEmbed (#150) 
* Add CONTRIBUTING.md file with guidelines for contributing to FastEmbed

* Add code linting and pre-commit info to CONTRIBUTING

* Update CONTRIBUTING.md

Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>

* Add bug/new model issue template and move CONTRIBUTING.md

---------

Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>
 2024-03-14 14:50:48 +05:30
Anush
6a94994038 Release v0.2.4 (#149) v0.2.4 2024-03-13 23:56:01 +05:30
Andrey Vasnetsov
361f674e47 avoid changing output dimentionality for a single input (#148) 2024-03-13 23:38:51 +05:30
Nirant
041a606285 Merge pull request #146 from qdrant:v0.2.3 
Publish to PyPi with SPLADE models
v0.2.3 		2024-03-13 18:17:41 +05:30
Nirant Kasliwal
5b937c29f6 Update poetry install command to exclude docs 2024-03-13 18:15:51 +05:30
Nirant Kasliwal
8d368889c0 Update version and add pre-commit dependency 2024-03-13 18:07:31 +05:30
Nirant
d817da2e01 Add Splade v1 (#144) 
* Add SPLADE v1

* WIP SPLADE Export errors

* add ONNX model to HF hub and use that

* Update sentences in Converting_SPLADE_to_ONNX.ipynb

* Remove unnecessary files and directories

* Rename var in TextEmbedding class to use EMBEDDING_MODEL_TYPE

* Add SPLADE to list of text embeddings

* Add SPLADE model support for text embedding

* Fix deprecation warning in embedding.py

* Add test for batch embedding with sparse embeddings

* Refactor import statement in test_sparse_embeddings.py

* Rename nbs

* Update vocab size in SPLADE model

* Fix canonical vector lookup in test_text_onnx_embeddings.py

* review refactoring

* restore list_supported_models in OnnxTextEmbedding

* Remove unused method _preprocess_onnx_input() in SpladePP class

* Update SPLADE_PP_en_v1 source in splade_pp.py

* Refactor onnx_model.py to change base model behavior

* extend tests to sparse values as well as indicies

* chore: pre-commit hooks

---------

Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Anush008 <anushshetty90@gmail.com>
 2024-03-13 18:04:44 +05:30
Artem Daineko
68635efa3f Fix link to Optimus in docs (#143) 2024-03-10 11:45:44 +05:30
Nirant
9ffde58df6 Getting Started Improvements (#138) 
* Update FastEmbed README.md

* Rewrite GettingStarted to use TextEmbedding instead of DefaultEmbedding

* Improve grammar

* Update Getting Started.ipynb with model information and document format
 2024-03-07 17:01:11 +05:30
Nirant Kasliwal
5603fbe1fb Fix naming typo 2024-03-06 14:30:54 +05:30
Nirant Kasliwal
9ed4486d9c Rename typo in notebook 2024-03-06 14:24:48 +05:30
Nirant Kasliwal
e36a39c388 Update author information in notebook 2024-03-06 14:18:48 +05:30
Nirant Kasliwal
97c359c1b9 Add Navrasa LLM download and explanation sections 2024-03-06 14:17:41 +05:30