upstream-archive/fastembed
1
0
Fork 0
mirror of https://github.com/qdrant/fastembed.git synced 2026-10-03 19:37:46 -05:00
Code Issues Packages Projects Releases Wiki Activity
419 Commits 38 Branches 61 Tags
clip-export
Commit Graph
419 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nirant 27e70794b3 Merge branch 'main' into clip-export 2024-05-05 06:17:21 +05:30
Andrey Vasnetsov 6ecab6d40d bump v0.2.7 v0.2.7 2024-05-03 16:33:06 +00:00
GeorgeNirant KasliwalNirant
 		d8c592032b new: allow users to override providers (#214) 
* new: add gpu support, allow users to override providers

* fix: update poetry.lock

* fix: fix type hint for 3.8

* [readme] Remove similar work

* [README] Add GPU support for FastEmbed library

* [README]  Add device check

* fix: revert changes to pyproject and lock, update readme

* Update poetry.lock

* new: add type alias for providers, add explicit providers to embeddings

---------

Co-authored-by: Nirant Kasliwal <nirant.bits@gmail.com>
Co-authored-by: Nirant <NirantK@users.noreply.github.com>
 2024-05-03 18:31:22 +02:00
GeorgeandAndrey Vasnetsov da603b8b7d new: add release instructions (#231) 
* new: add release instructions

* review fixes

---------

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
 2024-05-03 18:30:54 +02:00
Andrey Vasnetsov 562d604375 Merge pull request #223 from Waffleboy/main 
[Bugfix] Allow user to pick local mode only so huggingface does not do a network call and timeout
 2024-05-03 18:29:54 +02:00
Andrey Vasnetsov 8b1a98a6a3 Merge pull request #230 from qdrant/update-tokenizers 
new: update tokenizers
 2024-05-03 18:26:09 +02:00
Andrey Vasnetsov 3c9b147e0a version 2024-05-03 16:21:33 +00:00
George Panchuk 6abd415f4a fix: add local_files_only to sparse, formatting, refactor 2024-05-03 17:49:10 +02:00
George Panchuk 8184acbb39 new: update tokenizers 2024-05-03 16:57:45 +02:00
George 47cf7f9f92 new: add gpu package into workflow (#228) 
* new: add gpu package into workflow

* remove gpu tag
 2024-05-03 16:30:23 +02:00
generall f7896c81f3 do not ship poetry.lock with the repo, as package users wont have it anyway 2024-05-03 13:00:53 +02:00
Thiru 4a59d09248 Allow user to pick local mode only so huggingface does not do a network call and timeout 2024-05-02 00:25:24 +08:00
George Panchuk 7b51486fcf fix: clean output 2024-04-30 19:26:52 +02:00
George Panchuk 5f9a29fe34 new: add clip exporter 2024-04-30 18:59:25 +02:00
Arun 432da42c11 fix links (#215) 2024-04-27 22:24:29 +05:30
George 5cde2898bc new: remove slurm environment variables (#213) 2024-04-26 21:45:02 +02:00
AnushandGeorge ab7a99a748 feat: Quantized models (#201) 
* feat: Quantized models

* refactor: use model_file for GCS

* refactoring: refactor model downloading (#209)

* refactoring: refactor model downloading

* refactor: update docstring

Co-authored-by: Anush <anushshetty90@gmail.com>

* Update fastembed/common/model_management.py

Co-authored-by: George <george.panchuk@qdrant.tech>

* fix: model_file for Snowflake models

---------

Co-authored-by: George <george.panchuk@qdrant.tech>
 2024-04-26 17:27:16 +02:00
Anush 466886a317 ci: Schedule python-tests.yml (#211) 
* ci: Schedule python-tests.yml

* ci: use emojis

* ci: Bump action versions python-tests.yml

* ci: python-tests.yml
 2024-04-26 10:28:37 +05:30
Anush cc4112d859 feat: Snowflake models (#207) 
* feat: Snowflake models

* Added snowflake/snowflake-arctic-embed-m

* docs: snowflake/snowflake-arctic-embed-m
 2024-04-19 19:00:10 +05:30
Nirant 864217a7d9 Re-run of identical hardware and generate graphs (#205) 2024-04-18 11:13:18 +05:30
Andrew Green 9bad44368e Workaround for running on SLURM (#198) 
* Workaround for running on SLURM

onnxruntime would usually get the number of threads from OMP_NUM_THREADS, but that isn't set on SLURM which handles the number of threads differently.

This addition tries to figure out if we're running under SLURM, and if so sets the session options accordingly using SLURM environment variables instead.

Tested working with latest versions of onnxruntime and fastembed on slurm 23.02.07

* onnxruntime requires number of threads be an integer

Caused by me having mis-matched version from my machine and the slurm cluster :(

* `os.getenv` returns None for unset environment variables, fix logic

Instead of empty string, which I thought it did
 2024-04-15 19:14:11 +05:30
dependabot[bot]andNirant a5cab7a31a build(deps): bump idna from 3.6 to 3.7 (#195) 
Bumps [idna](https://github.com/kjd/idna) from 3.6 to 3.7.
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](https://github.com/kjd/idna/compare/v3.6...v3.7)

---
updated-dependencies:
- dependency-name: idna
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nirant <NirantK@users.noreply.github.com>
 2024-04-15 06:04:54 +05:30
Anush e55c145924 chore: Exclude unused model repo files (#196) 
* chore: Exclude unused model files

* fix: blob pattern
 2024-04-12 20:15:02 +05:30
dependabot[bot]andNirant ad02f60eea build(deps-dev): bump pillow from 10.2.0 to 10.3.0 (#186) 
Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.2.0 to 10.3.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](https://github.com/python-pillow/Pillow/compare/10.2.0...10.3.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nirant <NirantK@users.noreply.github.com>
 2024-04-08 16:21:32 +05:30
NirantandGeorge 2fcec07f1b Add Python version to the ISSUE_TEMPLATE (#188) 
* Add Python version

* Update .github/ISSUE_TEMPLATE/bug-report.yml

Co-authored-by: George <george.panchuk@qdrant.tech>

---------

Co-authored-by: George <george.panchuk@qdrant.tech>
 2024-04-08 16:05:29 +05:30
George f340a73a3e refactoring: update binary quantization notebook (#180) 2024-04-02 14:45:27 +05:30
Nirant 335f673f3a Update bug-report.yml 2024-04-02 13:51:51 +05:30
Nirant ee7ba0a536 Update bug-report.yml 2024-04-02 13:51:27 +05:30
George 912e95e5c8 fix: remove model archive if model extraction was not finished correctly (#179) 2024-04-01 20:11:32 +02:00
Nirant c09909773e Add Execution Counts (#177) 
* Update notebooks

* Update FastEmbed usage across docs

* Refactor code for better readability and maintainability

* Clean outputs

* Change dataset

* Add numbers inline in output

* Remove inline outputs since I used :memory:

* Fix syntax error in Hindi_Tamil_RAG_with_Navarasa7B.ipynb
v0.2.6 		2024-04-01 20:48:26 +05:30
Nirant 3d2254d215 Bump version to 0.2.6 in pyproject.toml (#175) 2024-04-01 17:16:07 +05:30
GeorgeandNirant Kasliwal e11f66bb55 refactoring: update imports in notebooks (#173) 
* new: simplify imports

* refactoring: update import

* refactoring: update imports in notebooks

* fix: fix notebook output

* Re-run notebook with revised imports

---------

Co-authored-by: Nirant Kasliwal <nirant.bits@gmail.com>
 2024-04-01 17:02:13 +05:30
Nirant d6f9ace425 Update size_in_GB for BAAI/bge-small-en-v1.5 model (#176) 2024-04-01 16:40:45 +05:30
GeorgeNirantNirant Kasliwal
 		8b7b8476a5 fix: fix model sizes in supported models lists (#167) 
* fix: fix model sizes in supported models lists

* fix: remove redundant comment

* fix: fix test

* fix: update supported models notebook

* Consistentcy around quantization in supported_onnx_models

---------

Co-authored-by: Nirant <NirantK@users.noreply.github.com>
Co-authored-by: Nirant Kasliwal <nirant.bits@gmail.com>
 2024-04-01 16:18:49 +05:30
George ae35a96bcb new: simplify imports (#171) 
* new: simplify imports

* refactoring: update import
 2024-04-01 12:58:26 +05:30
GeorgeandNirant 25671ec349 Update ruff (#172) 
* refactoring: reduce max line-length

* new: update ruff

---------

Co-authored-by: Nirant <NirantK@users.noreply.github.com>
 2024-04-01 12:55:15 +05:30
George ce98631b9a Fix spladepp parallelism (#169) 
* fix: add get_worker_class implementation to spladepp

* fix: add tests for parallel embed for spladepp
 2024-04-01 10:22:24 +05:30
George 0a4ed42b58 fix: unify existing patterns, remove redundant (#168) 2024-03-30 14:40:19 +05:30
NirantandAnush e3d2e1dc44 Hybrid Search Tutorial (#165) 
* Re-organize docs

* Rename notebooks

* Move nbs

* Working Sparse and Dense Search

* Add RRF

* Refactor code to improve performance and readability

* Add ESCI label for the RRF results

* Update docs/examples/Hybrid_Search.ipynb

Co-authored-by: Anush  <anushshetty90@gmail.com>

* Update docs/examples/Hybrid_Search.ipynb

Co-authored-by: Anush  <anushshetty90@gmail.com>

* Remove unnecessary code and update vector format

---------

Co-authored-by: Anush <anushshetty90@gmail.com>
 2024-03-29 21:10:14 +05:30
Nirant 62c21b0237 Add misspelled version of SPLADE++ model for English (#161) 2024-03-22 21:06:03 +05:30
Anush d791f38704 chore: case-insensitive model_management.py (#160) 2024-03-22 21:02:31 +05:30
Nirant Kasliwal c651b2b539 Update SPLADE notebook with new sections 2024-03-22 15:18:34 +05:30
Yuvraj WaleandNirant bc1e23849b feat: support mixedbread-ai/mxbai-embed-large-v1 (#158) 
* add: support mixedbread-ai/mxbai-embed-large-v1

* refactor: canonical vector

* Update fastembed/text/onnx_embedding.py

---------

Co-authored-by: Nirant <NirantK@users.noreply.github.com>
 2024-03-22 06:23:54 +05:30
Nirant 4db4839995 [PyPi Publish] Bump version to 0.2.5 in pyproject.toml (#156) 
* Bump version to 0.2.5 in pyproject.toml

* chore: case insensitive check (#157)
v0.2.5 		2024-03-20 19:11:44 +05:30
NirantandAnush 96a2a9097a Fix model name typo + Add SPLADE notebook (#155) 
* Rename model + Add SPLADE notebook

* Update docs/examples/SPLADE_with_FastEmbed.ipynb

Co-authored-by: Anush  <anushshetty90@gmail.com>

* Update docs/examples/SPLADE_with_FastEmbed.ipynb

Co-authored-by: Anush  <anushshetty90@gmail.com>

* Update docs/examples/SPLADE_with_FastEmbed.ipynb

Co-authored-by: Anush  <anushshetty90@gmail.com>

* Update CANONICAL_COLUMN_VALUES in test_sparse_embeddings.py

---------

Co-authored-by: Anush <anushshetty90@gmail.com>
 2024-03-20 18:15:53 +05:30
NirantandKumar Shivendu 256b2265d5 Move CONTRIBUTING.md + Add Test for Adding New Models (#154) 
* Move CONTRIBUTING.md + Ad Test for Adding New Models

* Update CONTRIBUTING.md

Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>

---------

Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>
 2024-03-18 19:58:02 +05:30
NirantandKumar Shivendu 1e91c8d165 Fix Issue Template forms (#152) 
* Add CONTRIBUTING.md file with guidelines for contributing to FastEmbed

* Add code linting and pre-commit info to CONTRIBUTING

* Update CONTRIBUTING.md

Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>

* Add bug/new model issue template and move CONTRIBUTING.md

* Re-organize issue templates

---------

Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>
 2024-03-14 15:11:23 +05:30
Nirant a761f456b2 Add import statement for version debugging (#151) 2024-03-14 15:02:15 +05:30
NirantandKumar Shivendu 287e19c494 Add CONTRIBUTING.md file with guidelines for contributing to FastEmbed (#150) 
* Add CONTRIBUTING.md file with guidelines for contributing to FastEmbed

* Add code linting and pre-commit info to CONTRIBUTING

* Update CONTRIBUTING.md

Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>

* Add bug/new model issue template and move CONTRIBUTING.md

---------

Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>
 2024-03-14 14:50:48 +05:30
Anush 6a94994038 Release v0.2.4 (#149) v0.2.4 2024-03-13 23:56:01 +05:30