mirror of
https://github.com/qdrant/fastembed.git
synced 2026-07-30 14:41:04 -05:00
e3d2e1dc44
* Re-organize docs * Rename notebooks * Move nbs * Working Sparse and Dense Search * Add RRF * Refactor code to improve performance and readability * Add ESCI label for the RRF results * Update docs/examples/Hybrid_Search.ipynb Co-authored-by: Anush <anushshetty90@gmail.com> * Update docs/examples/Hybrid_Search.ipynb Co-authored-by: Anush <anushshetty90@gmail.com> * Remove unnecessary code and update vector format --------- Co-authored-by: Anush <anushshetty90@gmail.com>
454 lines
12 KiB
Plaintext
454 lines
12 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Setup: Install Dependencies, Imports & Download Embeddings"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!pip install matplotlib tqdm pandas numpy --quiet"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"metadata": {
|
|
"id": "WBVTItUX4yyr"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import numpy as np\n",
|
|
"import pandas as pd\n",
|
|
"from tqdm import tqdm"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 👨🏾💻 Code Walkthrough\n",
|
|
"Here's an explanation of the code structure provided:\n",
|
|
"\n",
|
|
"1. **Loading Data**: OpenAI embeddings are loaded from a parquet files (we can load upto 1M embedding) and concatenated into one array.\n",
|
|
"2. **Binary Conversion**: A new array with the same shape is initialized with zeros, and the positive values in the original vectors are set to 1.\n",
|
|
"3. **Accuracy Function**: The accuracy function compares original vectors with binary vectors for a given index, limit, and oversampling rate. The comparison is done using dot products and logical XOR, sorting the results, and measuring the intersection.\n",
|
|
"4. **Testing**: The accuracy is tested for different oversampling rates (1, 2, 4), revealing a correctness of ~0.96 for an oversampling of 4.\n",
|
|
"\n",
|
|
"\n",
|
|
"## 💿 Loading Data"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/",
|
|
"height": 250
|
|
},
|
|
"id": "REJpFqkG7EG2",
|
|
"outputId": "7a43c0ae-fbcc-45fe-fd58-bfe691297b22"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 26/26 [00:10<00:00, 2.45it/s]\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"(1000000, 1536)"
|
|
]
|
|
},
|
|
"execution_count": 13,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"def get_openai_vectors(force_download: bool = False):\n",
|
|
" res = []\n",
|
|
" for i in tqdm(range(26)):\n",
|
|
" if force_download:\n",
|
|
" !wget https://huggingface.co/api/datasets/KShivendu/dbpedia-entities-openai-1M/parquet/KShivendu--dbpedia-entities-openai-1M/train/{i}.parquet\n",
|
|
" df = pd.read_parquet(f\"{i}.parquet\", engine=\"pyarrow\")\n",
|
|
" res.append(np.stack(df.openai))\n",
|
|
" del df\n",
|
|
"\n",
|
|
" openai_vectors = np.concatenate(res)\n",
|
|
" del res\n",
|
|
" return openai_vectors\n",
|
|
"\n",
|
|
"\n",
|
|
"openai_vectors = get_openai_vectors(force_download=False)\n",
|
|
"openai_vectors.shape"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## ㆓ Binary Conversion\n",
|
|
"\n",
|
|
"Here, we will use 0 as the threshold for the binary conversion. All values greater than 0 will be set to 1, and others will remain 0. This is a simple and effective way to convert continuous values into binary values for OpenAI embeddings."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"metadata": {
|
|
"id": "0JM2-Bj2Jkab"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"openai_bin = np.zeros_like(openai_vectors, dtype=np.int8)\n",
|
|
"openai_bin[openai_vectors > 0] = 1"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 🎯 Accuracy Function\n",
|
|
"\n",
|
|
"We will use the accuracy function to compare the original vectors with the binary vectors for a given index, limit, and oversampling rate. The comparison is done using dot products and logical XOR, sorting the results, and measuring the intersection."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"metadata": {
|
|
"id": "FqshI-GlIERd"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def accuracy(idx, limit: int, oversampling: int):\n",
|
|
" scores = np.dot(openai_vectors, openai_vectors[idx])\n",
|
|
" dot_results = np.argsort(scores)[-limit:][::-1]\n",
|
|
"\n",
|
|
" bin_scores = 1536 - np.logical_xor(openai_bin, openai_bin[idx]).sum(axis=1)\n",
|
|
" bin_results = np.argsort(bin_scores)[-(limit * oversampling) :][::-1]\n",
|
|
"\n",
|
|
" return len(set(dot_results).intersection(set(bin_results))) / limit"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## 📊 Results"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "qtzUlq_sFTRf",
|
|
"outputId": "17fe04ea-4f73-4a57-990b-180f1c04b472"
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" 0%| | 0/4 [00:00<?, ?it/s]"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'sampling_rate': 1, 'limit': 10, 'recall': 0.8}\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 2/2 [00:33<00:00, 16.98s/it]\n",
|
|
" 25%|██▌ | 1/4 [00:33<01:41, 33.96s/it]"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'sampling_rate': 1, 'limit': 100, 'recall': 0.708}\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": []
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'sampling_rate': 2, 'limit': 10, 'recall': 0.95}\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 2/2 [00:32<00:00, 16.38s/it]\n",
|
|
" 50%|█████ | 2/4 [01:06<01:06, 33.26s/it]"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'sampling_rate': 2, 'limit': 100, 'recall': 0.877}\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": []
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'sampling_rate': 3, 'limit': 10, 'recall': 0.96}\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 2/2 [00:32<00:00, 16.49s/it]\n",
|
|
" 75%|███████▌ | 3/4 [01:39<00:33, 33.13s/it]"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'sampling_rate': 3, 'limit': 100, 'recall': 0.937}\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": []
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'sampling_rate': 5, 'limit': 10, 'recall': 0.9800000000000001}\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"100%|██████████| 2/2 [00:32<00:00, 16.47s/it]\n",
|
|
"100%|██████████| 4/4 [02:12<00:00, 33.17s/it]"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'sampling_rate': 5, 'limit': 100, 'recall': 0.977}\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"number_of_samples = 10\n",
|
|
"limits = [10, 100]\n",
|
|
"sampling_rate = [1, 2, 3, 5]\n",
|
|
"results = []\n",
|
|
"\n",
|
|
"\n",
|
|
"def mean_accuracy(number_of_samples, limit, sampling_rate):\n",
|
|
" return np.mean([accuracy(i, limit=limit, oversampling=sampling_rate) for i in range(number_of_samples)])\n",
|
|
"\n",
|
|
"\n",
|
|
"for i in tqdm(sampling_rate):\n",
|
|
" for j in tqdm(limits):\n",
|
|
" result = {\"sampling_rate\": i, \"limit\": j, \"recall\": mean_accuracy(number_of_samples, j, i)}\n",
|
|
" print(result)\n",
|
|
" results.append(result)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>sampling_rate</th>\n",
|
|
" <th>limit</th>\n",
|
|
" <th>recall</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>1</td>\n",
|
|
" <td>10</td>\n",
|
|
" <td>0.800</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>1</td>\n",
|
|
" <td>100</td>\n",
|
|
" <td>0.708</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>2</td>\n",
|
|
" <td>10</td>\n",
|
|
" <td>0.950</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>2</td>\n",
|
|
" <td>100</td>\n",
|
|
" <td>0.877</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>3</td>\n",
|
|
" <td>10</td>\n",
|
|
" <td>0.960</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>5</th>\n",
|
|
" <td>3</td>\n",
|
|
" <td>100</td>\n",
|
|
" <td>0.937</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>6</th>\n",
|
|
" <td>5</td>\n",
|
|
" <td>10</td>\n",
|
|
" <td>0.980</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>7</th>\n",
|
|
" <td>5</td>\n",
|
|
" <td>100</td>\n",
|
|
" <td>0.977</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>"
|
|
],
|
|
"text/plain": [
|
|
" sampling_rate limit recall\n",
|
|
"0 1 10 0.800\n",
|
|
"1 1 100 0.708\n",
|
|
"2 2 10 0.950\n",
|
|
"3 2 100 0.877\n",
|
|
"4 3 10 0.960\n",
|
|
"5 3 100 0.937\n",
|
|
"6 5 10 0.980\n",
|
|
"7 5 100 0.977"
|
|
]
|
|
},
|
|
"execution_count": 19,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"results = pd.DataFrame(results)\n",
|
|
"results"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"| sampling_rate | limit | accuracy |\n",
|
|
"|---------------|-------|----------|\n",
|
|
"| 1 | 10 | 0.800 |\n",
|
|
"| 1 | 100 | 0.708 |\n",
|
|
"| 2 | 10 | 0.950 |\n",
|
|
"| 2 | 100 | 0.877 |\n",
|
|
"| 4 | 10 | 0.970 |\n",
|
|
"| 4 | 100 | 0.956 |\n",
|
|
"| 8 | 10 | 0.990 |\n",
|
|
"| 8 | 100 | 0.990 |\n",
|
|
"| 16 | 10 | 1.000 |\n",
|
|
"| 16 | 100 | 0.998 |"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"machine_shape": "hm",
|
|
"provenance": []
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.17"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|