Hybrid Search Tutorial (#165)

* Re-organize docs

* Rename notebooks

* Move nbs

* Working Sparse and Dense Search

* Add RRF

* Refactor code to improve performance and readability

* Add ESCI label for the RRF results

* Update docs/examples/Hybrid_Search.ipynb

Co-authored-by: Anush  <anushshetty90@gmail.com>

* Update docs/examples/Hybrid_Search.ipynb

Co-authored-by: Anush  <anushshetty90@gmail.com>

* Remove unnecessary code and update vector format

---------

Co-authored-by: Anush <anushshetty90@gmail.com>
This commit is contained in:
Nirant
2024-03-29 21:10:14 +05:30
committed by GitHub
parent 62c21b0237
commit e3d2e1dc44
5 changed files with 1146 additions and 17 deletions

File diff suppressed because it is too large Load Diff

View File

@@ -3,22 +3,7 @@
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Binary Quantization of OpenAI Embedding\n",
"---\n",
"\n",
"In the world of large-scale data retrieval and processing, efficiency is crucial. With the exponential growth of data, the ability to retrieve information quickly and accurately can significantly affect system performance. This blog post explores a technique known as binary quantization applied to OpenAI embeddings, demonstrating how it can enhance **retrieval latency by 20x** or more.\n",
"\n",
"## What Are OpenAI Embeddings?\n",
"OpenAI embeddings are numerical representations of textual information. They transform text into a vector space where semantically similar texts are mapped close together. This mathematical representation enables computers to understand and process human language more effectively.\n",
"\n",
"## Binary Quantization\n",
"Binary quantization is a method which converts continuous numerical values into binary values (0 or 1). It simplifies the data structure, allowing faster computations. Here's a brief overview of the binary quantization process applied to OpenAI embeddings:\n",
"\n",
"1. **Load Embeddings**: OpenAI embeddings are loaded from parquet files.\n",
"2. **Binary Transformation**: The continuous valued vectors are converted into binary form. Here, values greater than 0 are set to 1, and others remain 0.\n",
"3. **Comparison & Retrieval**: Binary vectors are used for comparison using logical XOR operations and other efficient algorithms."
]
"source": []
},
{
"cell_type": "markdown",

View File

@@ -4,7 +4,20 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Binary Quantization with Qdrant\n",
"# Binary Quantization with Qdrant & OpenAI Embedding\n",
"\n",
"---\n",
"In the world of large-scale data retrieval and processing, efficiency is crucial. With the exponential growth of data, the ability to retrieve information quickly and accurately can significantly affect system performance. This blog post explores a technique known as binary quantization applied to OpenAI embeddings, demonstrating how it can enhance **retrieval latency by 20x** or more.\n",
"\n",
"## What Are OpenAI Embeddings?\n",
"OpenAI embeddings are numerical representations of textual information. They transform text into a vector space where semantically similar texts are mapped close together. This mathematical representation enables computers to understand and process human language more effectively.\n",
"\n",
"## Binary Quantization\n",
"Binary quantization is a method which converts continuous numerical values into binary values (0 or 1). It simplifies the data structure, allowing faster computations. Here's a brief overview of the binary quantization process applied to OpenAI embeddings:\n",
"\n",
"1. **Load Embeddings**: OpenAI embeddings are loaded from parquet files.\n",
"2. **Binary Transformation**: The continuous valued vectors are converted into binary form. Here, values greater than 0 are set to 1, and others remain 0.\n",
"3. **Comparison & Retrieval**: Binary vectors are used for comparison using logical XOR operations and other efficient algorithms.\n",
"\n",
"Binary Quantization is a promising approach to improve retrieval speeds and reduce memory footprint of vector search engines. In this notebook we will show how to use Qdrant to perform binary quantization of vectors and perform fast similarity search on the resulting index.\n",
"\n",