diff --git a/docs/experimental/Accuracy_vs_SamplingRate.png b/docs/experimental/Accuracy_vs_SamplingRate.png new file mode 100644 index 0000000..7e27e1e Binary files /dev/null and b/docs/experimental/Accuracy_vs_SamplingRate.png differ diff --git a/docs/experimental/Binary Quantisation.ipynb b/docs/experimental/Binary Quantisation.ipynb new file mode 100644 index 0000000..6885e88 --- /dev/null +++ b/docs/experimental/Binary Quantisation.ipynb @@ -0,0 +1,509 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Binary Quantization of OpenAI Embeddings: A Leap in Retrieval Efficiency\n", + "\n", + "In the world of large-scale data retrieval and processing, efficiency is crucial. With the exponential growth of data, the ability to retrieve information quickly and accurately can significantly affect system performance. This blog post explores a technique known as binary quantization applied to OpenAI embeddings, demonstrating how it can enhance retrieval latency by a factor of 20 or more.\n", + "\n", + "## What Are OpenAI Embeddings?\n", + "OpenAI embeddings are numerical representations of textual information. They transform text into a vector space where semantically similar texts are mapped close together. This mathematical representation enables computers to understand and process human language more effectively.\n", + "\n", + "## Binary Quantization\n", + "Binary quantization is a method which converts continuous numerical values into binary values (0 or 1). It simplifies the data structure, allowing faster computations. Here's a brief overview of the binary quantization process applied to OpenAI embeddings:\n", + "\n", + "1. Load Embeddings: OpenAI embeddings are loaded from parquet files.\n", + "2. Binary Transformation: The continuous-valued vectors are converted into binary form. Values greater than 0 are set to 1, and others remain 0.\n", + "3. Comparison & Retrieval: Binary vectors are used for comparison using logical XOR operations and other efficient algorithms." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup: Install Dependencies, Imports & Download Embeddings" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install matplotlib tqdm pandas numpy --quiet" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "id": "WBVTItUX4yyr" + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import pandas as pd\n", + "import matplotlib.pyplot as plt\n", + "from tqdm import tqdm" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Download Embeddings" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Code Walkthrough\n", + "Here's an explanation of the code structure provided:\n", + "\n", + "1. Loading Data: OpenAI embeddings are loaded from a parquet files (we can load upto 1M embedding) and concatenated into one array.\n", + "2. Binary Conversion: A new array with the same shape is initialized with zeros, and the positive values in the original vectors are set to 1.\n", + "3. Accuracy Function: The accuracy function compares original vectors with binary vectors for a given index, limit, and oversampling rate. The comparison is done using dot products and logical XOR, sorting the results, and measuring the intersection.\n", + "4. Testing: The accuracy is tested for different oversampling rates (1, 2, 4), revealing a correctness of 0.9697 for an oversampling of 4.\n", + "\n", + "\n", + "### 💿 Loading Data" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 250 + }, + "id": "REJpFqkG7EG2", + "outputId": "7a43c0ae-fbcc-45fe-fd58-bfe691297b22" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 26/26 [00:12<00:00, 2.02it/s]\n" + ] + }, + { + "data": { + "text/plain": [ + "(1000000, 1536)" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def get_openai_vectors(force_download: bool = False):\n", + " res = []\n", + " for i in tqdm(range(26)):\n", + " if force_download:\n", + " !wget https://huggingface.co/api/datasets/KShivendu/dbpedia-entities-openai-1M/parquet/KShivendu--dbpedia-entities-openai-1M/train/{i}.parquet\n", + " df = pd.read_parquet(f\"{i}.parquet\", engine=\"pyarrow\")\n", + " res.append(np.stack(df.openai))\n", + " del df\n", + "\n", + " openai_vectors = np.concatenate(res)\n", + " del res\n", + " return openai_vectors\n", + "\n", + "\n", + "openai_vectors = get_openai_vectors(force_download=False)\n", + "openai_vectors.shape" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Binary Conversion\n", + "\n", + "Here, we will use 0 as the threshold for the binary conversion. All values greater than 0 will be set to 1, and others will remain 0. This is a simple and effective way to convert continuous values into binary values for OpenAI embeddings." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "id": "0JM2-Bj2Jkab" + }, + "outputs": [], + "source": [ + "openai_bin = np.zeros_like(openai_vectors, dtype=np.int8)\n", + "openai_bin[openai_vectors > 0] = 1" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Accuracy Function\n", + "\n", + "We will use the accuracy function to compare the original vectors with the binary vectors for a given index, limit, and oversampling rate. The comparison is done using dot products and logical XOR, sorting the results, and measuring the intersection." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "id": "FqshI-GlIERd" + }, + "outputs": [], + "source": [ + "def accuracy(idx, limit: int, oversampling: int):\n", + " scores = np.dot(openai_vectors, openai_vectors[idx])\n", + " dot_results = np.argsort(scores)[-limit:][::-1]\n", + "\n", + " bin_scores = 1536 - np.logical_xor(openai_bin, openai_bin[idx]).sum(axis=1)\n", + " bin_results = np.argsort(bin_scores)[-(limit * oversampling) :][::-1]\n", + "\n", + " return len(set(dot_results).intersection(set(bin_results))) / limit" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 📊 Results" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "qtzUlq_sFTRf", + "outputId": "17fe04ea-4f73-4a57-990b-180f1c04b472" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " 0%| | 0/5 [00:00, ?it/s]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'sampling_rate': 1, 'limit': 10, 'accuracy': 0.8}\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 2/2 [00:44<00:00, 22.11s/it]\n", + " 20%|██ | 1/5 [00:44<02:56, 44.23s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'sampling_rate': 1, 'limit': 100, 'accuracy': 0.708}\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'sampling_rate': 2, 'limit': 10, 'accuracy': 0.95}\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 2/2 [00:45<00:00, 22.55s/it]\n", + " 40%|████ | 2/5 [01:29<02:14, 44.74s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'sampling_rate': 2, 'limit': 100, 'accuracy': 0.877}\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'sampling_rate': 4, 'limit': 10, 'accuracy': 0.97}\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 2/2 [00:44<00:00, 22.27s/it]\n", + " 60%|██████ | 3/5 [02:13<01:29, 44.65s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'sampling_rate': 4, 'limit': 100, 'accuracy': 0.9560000000000001}\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'sampling_rate': 8, 'limit': 10, 'accuracy': 0.99}\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 2/2 [00:43<00:00, 21.89s/it]\n", + " 80%|████████ | 4/5 [02:57<00:44, 44.30s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'sampling_rate': 8, 'limit': 100, 'accuracy': 0.99}\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'sampling_rate': 16, 'limit': 10, 'accuracy': 1.0}\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 2/2 [00:44<00:00, 22.10s/it]\n", + "100%|██████████| 5/5 [03:41<00:00, 44.37s/it]" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'sampling_rate': 16, 'limit': 100, 'accuracy': 0.998}\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\n" + ] + } + ], + "source": [ + "number_of_samples = 10\n", + "limits = [10, 100]\n", + "sampling_rate = [1, 2, 4, 8, 16]\n", + "results = []\n", + "\n", + "def mean_accuracy(number_of_samples, limit, sampling_rate):\n", + " return np.mean([accuracy(i, limit=limit, oversampling=sampling_rate) for i in range(number_of_samples)])\n", + "\n", + "for i in tqdm(sampling_rate):\n", + " for j in tqdm(limits):\n", + " result = {\"sampling_rate\": i, \"limit\": j, \"accuracy\": mean_accuracy(number_of_samples, j, i)}\n", + " print(result)\n", + " results.append(result)" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
| \n", + " | sampling_rate | \n", + "limit | \n", + "accuracy | \n", + "
|---|---|---|---|
| 0 | \n", + "1 | \n", + "10 | \n", + "0.800 | \n", + "
| 1 | \n", + "1 | \n", + "100 | \n", + "0.708 | \n", + "
| 2 | \n", + "2 | \n", + "10 | \n", + "0.950 | \n", + "
| 3 | \n", + "2 | \n", + "100 | \n", + "0.877 | \n", + "
| 4 | \n", + "4 | \n", + "10 | \n", + "0.970 | \n", + "
| 5 | \n", + "4 | \n", + "100 | \n", + "0.956 | \n", + "
| 6 | \n", + "8 | \n", + "10 | \n", + "0.990 | \n", + "
| 7 | \n", + "8 | \n", + "100 | \n", + "0.990 | \n", + "
| 8 | \n", + "16 | \n", + "10 | \n", + "1.000 | \n", + "
| 9 | \n", + "16 | \n", + "100 | \n", + "0.998 | \n", + "