{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "ExecuteTime": { "end_time": "2024-03-30T11:18:52.052764Z", "start_time": "2024-03-30T11:18:52.039616Z" } }, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "\n", "from fastembed import SparseTextEmbedding, TextEmbedding" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Supported Text Embedding Models" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
modeldimdescriptionsize_in_GB
0BAAI/bge-small-en-v1.5384Fast and Default English model0.067
1BAAI/bge-small-zh-v1.5512Fast and recommended Chinese model0.090
2sentence-transformers/all-MiniLM-L6-v2384Sentence Transformer model, MiniLM-L6-v20.090
3snowflake/snowflake-arctic-embed-xs384Based on all-MiniLM-L6-v2 model with only 22m ...0.090
4jinaai/jina-embeddings-v2-small-en512English embedding model supporting 8192 sequen...0.120
5snowflake/snowflake-arctic-embed-s384Based on infloat/e5-small-unsupervised, does n...0.130
6BAAI/bge-small-en384Fast English model0.130
7BAAI/bge-base-en-v1.5768Base English model, v1.50.210
8sentence-transformers/paraphrase-multilingual-...384Sentence Transformer model, paraphrase-multili...0.220
9BAAI/bge-base-en768Base English model0.420
10snowflake/snowflake-arctic-embed-m768Based on intfloat/e5-base-unsupervised model, ...0.430
11jinaai/jina-embeddings-v2-base-en768English embedding model supporting 8192 sequen...0.520
12nomic-ai/nomic-embed-text-v17688192 context length english model0.520
13nomic-ai/nomic-embed-text-v1.57688192 context length english model0.520
14snowflake/snowflake-arctic-embed-m-long768Based on nomic-ai/nomic-embed-text-v1-unsuperv...0.540
15mixedbread-ai/mxbai-embed-large-v11024MixedBread Base sentence embedding model, does...0.640
16sentence-transformers/paraphrase-multilingual-...768Sentence-transformers model for tasks like clu...1.000
17snowflake/snowflake-arctic-embed-l1024Based on intfloat/e5-large-unsupervised, large...1.020
18BAAI/bge-large-en-v1.51024Large English model, v1.51.200
19thenlper/gte-large1024Large general text embeddings model1.200
20intfloat/multilingual-e5-large1024Multilingual model, e5-large. Recommend using ...2.240
\n", "
" ], "text/plain": [ " model dim \\\n", "0 BAAI/bge-small-en-v1.5 384 \n", "1 BAAI/bge-small-zh-v1.5 512 \n", "2 sentence-transformers/all-MiniLM-L6-v2 384 \n", "3 snowflake/snowflake-arctic-embed-xs 384 \n", "4 jinaai/jina-embeddings-v2-small-en 512 \n", "5 snowflake/snowflake-arctic-embed-s 384 \n", "6 BAAI/bge-small-en 384 \n", "7 BAAI/bge-base-en-v1.5 768 \n", "8 sentence-transformers/paraphrase-multilingual-... 384 \n", "9 BAAI/bge-base-en 768 \n", "10 snowflake/snowflake-arctic-embed-m 768 \n", "11 jinaai/jina-embeddings-v2-base-en 768 \n", "12 nomic-ai/nomic-embed-text-v1 768 \n", "13 nomic-ai/nomic-embed-text-v1.5 768 \n", "14 snowflake/snowflake-arctic-embed-m-long 768 \n", "15 mixedbread-ai/mxbai-embed-large-v1 1024 \n", "16 sentence-transformers/paraphrase-multilingual-... 768 \n", "17 snowflake/snowflake-arctic-embed-l 1024 \n", "18 BAAI/bge-large-en-v1.5 1024 \n", "19 thenlper/gte-large 1024 \n", "20 intfloat/multilingual-e5-large 1024 \n", "\n", " description size_in_GB \n", "0 Fast and Default English model 0.067 \n", "1 Fast and recommended Chinese model 0.090 \n", "2 Sentence Transformer model, MiniLM-L6-v2 0.090 \n", "3 Based on all-MiniLM-L6-v2 model with only 22m ... 0.090 \n", "4 English embedding model supporting 8192 sequen... 0.120 \n", "5 Based on infloat/e5-small-unsupervised, does n... 0.130 \n", "6 Fast English model 0.130 \n", "7 Base English model, v1.5 0.210 \n", "8 Sentence Transformer model, paraphrase-multili... 0.220 \n", "9 Base English model 0.420 \n", "10 Based on intfloat/e5-base-unsupervised model, ... 0.430 \n", "11 English embedding model supporting 8192 sequen... 0.520 \n", "12 8192 context length english model 0.520 \n", "13 8192 context length english model 0.520 \n", "14 Based on nomic-ai/nomic-embed-text-v1-unsuperv... 0.540 \n", "15 MixedBread Base sentence embedding model, does... 0.640 \n", "16 Sentence-transformers model for tasks like clu... 1.000 \n", "17 Based on intfloat/e5-large-unsupervised, large... 1.020 \n", "18 Large English model, v1.5 1.200 \n", "19 Large general text embeddings model 1.200 \n", "20 Multilingual model, e5-large. Recommend using ... 2.240 " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "supported_models = (\n", " pd.DataFrame(TextEmbedding.list_supported_models())\n", " .sort_values(\"size_in_GB\")\n", " .drop(columns=\"sources\")\n", " .reset_index(drop=True)\n", ")\n", "supported_models" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Supported Sparse Text Embedding Models" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "ExecuteTime": { "end_time": "2024-03-30T11:19:01.564291Z", "start_time": "2024-03-30T11:19:01.538768Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
modelvocab_sizedescriptionsize_in_GBsources
0prithvida/Splade_PP_en_v130522Misspelled version of the model. Retained for ...0.532{'hf': 'Qdrant/SPLADE_PP_en_v1'}
1prithivida/Splade_PP_en_v130522Independent Implementation of SPLADE++ Model f...0.532{'hf': 'Qdrant/SPLADE_PP_en_v1'}
\n", "
" ], "text/plain": [ " model vocab_size \\\n", "0 prithvida/Splade_PP_en_v1 30522 \n", "1 prithivida/Splade_PP_en_v1 30522 \n", "\n", " description size_in_GB \\\n", "0 Misspelled version of the model. Retained for ... 0.532 \n", "1 Independent Implementation of SPLADE++ Model f... 0.532 \n", "\n", " sources \n", "0 {'hf': 'Qdrant/SPLADE_PP_en_v1'} \n", "1 {'hf': 'Qdrant/SPLADE_PP_en_v1'} " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.DataFrame(SparseTextEmbedding.list_supported_models())" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.8.18 ('base')", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" }, "orig_nbformat": 4, "vscode": { "interpreter": { "hash": "c4a27af61e455bc18dcf16f5867a2ff0402fa12b01dd0f6ce3a79ae73ad15e91" } } }, "nbformat": 4, "nbformat_minor": 2 }