{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup: Install Dependencies, Imports & Download Embeddings" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "!pip install matplotlib tqdm pandas numpy --quiet" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "id": "WBVTItUX4yyr" }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "from tqdm import tqdm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 👨🏾‍💻 Code Walkthrough\n", "Here's an explanation of the code structure provided:\n", "\n", "1. **Loading Data**: OpenAI embeddings are loaded from a parquet files (we can load upto 1M embedding) and concatenated into one array.\n", "2. **Binary Conversion**: A new array with the same shape is initialized with zeros, and the positive values in the original vectors are set to 1.\n", "3. **Accuracy Function**: The accuracy function compares original vectors with binary vectors for a given index, limit, and oversampling rate. The comparison is done using dot products and logical XOR, sorting the results, and measuring the intersection.\n", "4. **Testing**: The accuracy is tested for different oversampling rates (1, 2, 4), revealing a correctness of ~0.96 for an oversampling of 4.\n", "\n", "\n", "## 💿 Loading Data" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 250 }, "id": "REJpFqkG7EG2", "outputId": "7a43c0ae-fbcc-45fe-fd58-bfe691297b22" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████| 26/26 [00:10<00:00, 2.45it/s]\n" ] }, { "data": { "text/plain": [ "(1000000, 1536)" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def get_openai_vectors(force_download: bool = False):\n", " res = []\n", " for i in tqdm(range(26)):\n", " if force_download:\n", " !wget https://huggingface.co/api/datasets/KShivendu/dbpedia-entities-openai-1M/parquet/KShivendu--dbpedia-entities-openai-1M/train/{i}.parquet\n", " df = pd.read_parquet(f\"{i}.parquet\", engine=\"pyarrow\")\n", " res.append(np.stack(df.openai))\n", " del df\n", "\n", " openai_vectors = np.concatenate(res)\n", " del res\n", " return openai_vectors\n", "\n", "\n", "openai_vectors = get_openai_vectors(force_download=False)\n", "openai_vectors.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ㆓ Binary Conversion\n", "\n", "Here, we will use 0 as the threshold for the binary conversion. All values greater than 0 will be set to 1, and others will remain 0. This is a simple and effective way to convert continuous values into binary values for OpenAI embeddings." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "id": "0JM2-Bj2Jkab" }, "outputs": [], "source": [ "openai_bin = np.zeros_like(openai_vectors, dtype=np.int8)\n", "openai_bin[openai_vectors > 0] = 1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 🎯 Accuracy Function\n", "\n", "We will use the accuracy function to compare the original vectors with the binary vectors for a given index, limit, and oversampling rate. The comparison is done using dot products and logical XOR, sorting the results, and measuring the intersection." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "id": "FqshI-GlIERd" }, "outputs": [], "source": [ "def accuracy(idx, limit: int, oversampling: int):\n", " scores = np.dot(openai_vectors, openai_vectors[idx])\n", " dot_results = np.argsort(scores)[-limit:][::-1]\n", "\n", " bin_scores = 1536 - np.logical_xor(openai_bin, openai_bin[idx]).sum(axis=1)\n", " bin_results = np.argsort(bin_scores)[-(limit * oversampling) :][::-1]\n", "\n", " return len(set(dot_results).intersection(set(bin_results))) / limit" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 📊 Results" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "qtzUlq_sFTRf", "outputId": "17fe04ea-4f73-4a57-990b-180f1c04b472" }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " 0%| | 0/4 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sampling_ratelimitrecall
01100.800
111000.708
22100.950
321000.877
43100.960
531000.937
65100.980
751000.977
\n", "" ], "text/plain": [ " sampling_rate limit recall\n", "0 1 10 0.800\n", "1 1 100 0.708\n", "2 2 10 0.950\n", "3 2 100 0.877\n", "4 3 10 0.960\n", "5 3 100 0.937\n", "6 5 10 0.980\n", "7 5 100 0.977" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "results = pd.DataFrame(results)\n", "results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "| sampling_rate | limit | accuracy |\n", "|---------------|-------|----------|\n", "| 1 | 10 | 0.800 |\n", "| 1 | 100 | 0.708 |\n", "| 2 | 10 | 0.950 |\n", "| 2 | 100 | 0.877 |\n", "| 4 | 10 | 0.970 |\n", "| 4 | 100 | 0.956 |\n", "| 8 | 10 | 0.990 |\n", "| 8 | 100 | 0.990 |\n", "| 16 | 10 | 1.000 |\n", "| 16 | 100 | 0.998 |" ] } ], "metadata": { "colab": { "machine_shape": "hm", "provenance": [] }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.17" } }, "nbformat": 4, "nbformat_minor": 0 }