mirror of
https://github.com/qdrant/fastembed.git
synced 2026-07-23 11:20:51 -05:00
Hybrid Search Tutorial (#165)
* Re-organize docs * Rename notebooks * Move nbs * Working Sparse and Dense Search * Add RRF * Refactor code to improve performance and readability * Add ESCI label for the RRF results * Update docs/examples/Hybrid_Search.ipynb Co-authored-by: Anush <anushshetty90@gmail.com> * Update docs/examples/Hybrid_Search.ipynb Co-authored-by: Anush <anushshetty90@gmail.com> * Remove unnecessary code and update vector format --------- Co-authored-by: Anush <anushshetty90@gmail.com>
This commit is contained in:
1131
docs/examples/Hybrid_Search.ipynb
Normal file
1131
docs/examples/Hybrid_Search.ipynb
Normal file
File diff suppressed because it is too large
Load Diff
@@ -3,22 +3,7 @@
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Binary Quantization of OpenAI Embedding\n",
|
||||
"---\n",
|
||||
"\n",
|
||||
"In the world of large-scale data retrieval and processing, efficiency is crucial. With the exponential growth of data, the ability to retrieve information quickly and accurately can significantly affect system performance. This blog post explores a technique known as binary quantization applied to OpenAI embeddings, demonstrating how it can enhance **retrieval latency by 20x** or more.\n",
|
||||
"\n",
|
||||
"## What Are OpenAI Embeddings?\n",
|
||||
"OpenAI embeddings are numerical representations of textual information. They transform text into a vector space where semantically similar texts are mapped close together. This mathematical representation enables computers to understand and process human language more effectively.\n",
|
||||
"\n",
|
||||
"## Binary Quantization\n",
|
||||
"Binary quantization is a method which converts continuous numerical values into binary values (0 or 1). It simplifies the data structure, allowing faster computations. Here's a brief overview of the binary quantization process applied to OpenAI embeddings:\n",
|
||||
"\n",
|
||||
"1. **Load Embeddings**: OpenAI embeddings are loaded from parquet files.\n",
|
||||
"2. **Binary Transformation**: The continuous valued vectors are converted into binary form. Here, values greater than 0 are set to 1, and others remain 0.\n",
|
||||
"3. **Comparison & Retrieval**: Binary vectors are used for comparison using logical XOR operations and other efficient algorithms."
|
||||
]
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
|
||||
@@ -4,7 +4,20 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Binary Quantization with Qdrant\n",
|
||||
"# Binary Quantization with Qdrant & OpenAI Embedding\n",
|
||||
"\n",
|
||||
"---\n",
|
||||
"In the world of large-scale data retrieval and processing, efficiency is crucial. With the exponential growth of data, the ability to retrieve information quickly and accurately can significantly affect system performance. This blog post explores a technique known as binary quantization applied to OpenAI embeddings, demonstrating how it can enhance **retrieval latency by 20x** or more.\n",
|
||||
"\n",
|
||||
"## What Are OpenAI Embeddings?\n",
|
||||
"OpenAI embeddings are numerical representations of textual information. They transform text into a vector space where semantically similar texts are mapped close together. This mathematical representation enables computers to understand and process human language more effectively.\n",
|
||||
"\n",
|
||||
"## Binary Quantization\n",
|
||||
"Binary quantization is a method which converts continuous numerical values into binary values (0 or 1). It simplifies the data structure, allowing faster computations. Here's a brief overview of the binary quantization process applied to OpenAI embeddings:\n",
|
||||
"\n",
|
||||
"1. **Load Embeddings**: OpenAI embeddings are loaded from parquet files.\n",
|
||||
"2. **Binary Transformation**: The continuous valued vectors are converted into binary form. Here, values greater than 0 are set to 1, and others remain 0.\n",
|
||||
"3. **Comparison & Retrieval**: Binary vectors are used for comparison using logical XOR operations and other efficient algorithms.\n",
|
||||
"\n",
|
||||
"Binary Quantization is a promising approach to improve retrieval speeds and reduce memory footprint of vector search engines. In this notebook we will show how to use Qdrant to perform binary quantization of vectors and perform fast similarity search on the resulting index.\n",
|
||||
"\n",
|
||||
Reference in New Issue
Block a user