upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-08-06 01:50:57 -05:00
Code Issues Packages Projects Releases Wiki Activity
5,465 Commits 346 Branches 118 Tags
94de0e2d2dcf997bf26bec34dad407d8de9a9abb
Commit Graph

140 Commits

Author SHA1 Message Date
xzfc
dd3102baeb U8 quantization for sparse vector index (#4514) 
* U8 quantization for sparse index

* Spelling
 2024-06-24 14:12:29 +00:00
xzfc
cde39fb8a2 Move build_index out of VectorIndex (#4490) 
* Move build_index out of VectorIndex

* Build index in HNSWIndex::open()

* Introduce HnswIndexOpenArgs

* Proper deletion

* Improve tests

* HNSW::open(): add warn, comment and assert

* Revert to making up the config if it does not exist
 2024-06-24 14:44:34 +02:00
xzfc
05ff867ff3 Integrate compressed posting list (attempt 2) (#4453) 
* Rename InvertedIndex* -> InvertedIndexCompressed*

* Extract method VectorIndexEnum::fill_idf_statistics

* Extend VectorIndexEnum with new variants

* Introduce sparse::InvertedIndex::Version

* Replace SparseVectorIndexVersion -> InvertedIndex::Version

* Introduce sparse_vector_index::OpenArgs

* SparseVectorIndex::open: do not build index if directory is empty

Otherwise it would build the index twice since `SegmentBuilder::build()`
calls `::open()`, then `::build_index()`. This restores the old (<=v1.9)
behavior.

* Renames
 2024-06-17 14:41:38 +02:00
xzfc
409fb73885 Fix: Ignore non-existing postings when deleting old vector (#4472) 2024-06-14 16:13:01 +00:00
Andrey Vasnetsov
3e4c6dc0d3 Fix inplace updates for sparse index (#4375) 
* include old vector into update function and clean posting lists accordingly

* add integration test

* fix counter

* also remove old vector if the insertion is empty

* clippy

* borrow once

* fix max_next_weight correcton on delete + test

* vector index responsible for updating vector storage

* review fixes

* add debug assert
 2024-06-12 12:13:49 +02:00
xzfc
d2ef556237 Make sparse vectors generic (#4364) 
* Make sparse vector generic

* Generic search_context tests
 2024-06-10 15:04:32 +00:00
xzfc
f4c7ba15af Sparse: misc fixes (#4361) 
* Borrow cows
* Hide sparse_vector_fixture under feature="testing"
 2024-05-30 20:58:00 +00:00
xzfc
cd808afe3c Improve performance of compressed posting list (#4350) 
* Improve performance of compressed posting list

* Merge `compressed_idx` and `remainders_idx`
 2024-05-30 13:30:24 +00:00
xzfc
f026fc8a42 Recreate sparse vector index (#4313) 
* Recreate sparse vector index

* Rename index file; add simple migration
 2024-05-29 17:04:17 +00:00
dependabot[bot]
6ebd22d487 Bump parking_lot from 0.12.2 to 0.12.3 (#4337) 
Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.12.2 to 0.12.3.
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.12.2...0.12.3)

---
updated-dependencies:
- dependency-name: parking_lot
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-05-28 09:02:20 +02:00
Luis Cossío
b14d0b2463 universal-query: user-facing gRPC and REST request types (#4303) 
* add grpc `QueryPoints`

* add rest types

* move to REST types to `api::rest::schema`, decouple `OrderByInterface` and `RecommendStrategy`

* openapi changes

* add score_threshold param

* decouple `ShardKeySelector`, add shard key param

* remove commented-out code

* new codespell
 2024-05-24 08:23:12 -04:00
Arnaud Gourlay
cbcdefb310 Cleanup unused dependencies (#4306) 2024-05-23 10:06:44 +02:00
xzfc
93a42b4e2d Compressed posting lists (#4253) 2024-05-22 18:18:20 +00:00
xzfc
ccf7f1d24f InvertedIndexImmutableRam and index migrations (#4220) 
* Move StorageVersion from segment crate to common/io

* Refine StorageVersion API

* Move methods from SparseVectorDataConfig to enum SparseIndexType

* Introduce InvertedIndexImmutableRam

* Add migrate

* Don't migrate
 2024-05-16 06:47:47 +00:00
Luis Cossío
bfa72bb6d8 universal-query: Add query() to ShardOperation trait (#4210) 
* add `query` to shard trait

* add missing conversions for query

* update grpc docs

* Query response has intermediate results

* add ShardQueryResponse description

* move pub use to the top, keep only one way of reaching reexports
 2024-05-15 09:36:46 -04:00
xzfc
4f3239f27c Misc fixes (#4212) 
* SparseVectorIndex: replace pub with feature-gated getters

* Fixup dev-dependencies

* Fixup rust doc comments
 2024-05-10 08:16:28 +00:00
xzfc
6f738ca95f Update sparse vector benchmarks (#4163) 2024-05-07 20:34:37 +00:00
xzfc
cb31c533bf Introduce Cargo feature "testing" (#4192) 2024-05-07 16:14:46 +00:00
dependabot[bot]
699d83036d Bump parking_lot from 0.12.1 to 0.12.2 (#4136) 2024-04-29 22:45:18 +02:00
Andrey Vasnetsov
c173a9f5e5 Sparse idf dot (#4126) 
* introduce QueryContext, which accumulates runtime info needed for executing search

* fmt

* propagate query context into segment internals

* [WIP] prepare idf stats for search query context

* Split SparseVector and RemmapedSparseVector to guarantee we will not mix them up on the type level

* implement filling of the query context with IDF statistics

* implement re-weighting of the sparse query with idf

* fmt

* update idf param only if explicitly specified (more consistent with diff param update

* replace idf bool with modifier enum, improve further extensibility

* test and fixes

* Update lib/collection/src/operations/types.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* review fixes

* fmt

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2024-04-29 14:54:14 +02:00
Arnaud Gourlay
abcd2af537 Median based TopK for sparse vectors scoring (#4037) 
* Median based TopK for sparse vectors scoring

* add test with identical scores
 2024-04-19 13:27:21 +02:00
Arnaud Gourlay
c6e7c6f2b2 Fix average sparse vector recommendation (#4047) 
* Fix average sparse vector recommendation

* Revert "Fix average sparse vector recommendation"

This reverts commit 186c51209c.

* sort within combine (Ivan's approach)

* Revert "sort within combine (Ivan's approach)"

This reverts commit 28d560cc56.

* simpler approach without mutation
 2024-04-17 13:53:59 +02:00
Arnaud Gourlay
4ff2fb855b promote Validator to workspace (#4013) 2024-04-11 12:01:27 +02:00
dependabot[bot]
1c1b74ec99 Bump tempfile from 3.10.0 to 3.10.1 (#3763) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.10.0 to 3.10.1.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.10.0...v3.10.1)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-05 10:39:48 +01:00
Arnaud Gourlay
ab90b03aa0 Remove unused proptest on Sparse crate (#3667) 2024-02-22 11:35:22 +01:00
Arnaud Gourlay
722d905010 Optimize sparse vector index build time (#3605) 
* Optimize sparse vector index build time

* rework index builder to reuse posting list builder

* remove unecessary change

* make it shorter
 2024-02-20 12:01:42 +01:00
dependabot[bot]
b7165b5a81 Bump tempfile from 3.9.0 to 3.10.0 (#3596) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.9.0 to 3.10.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.9.0...v3.10.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-02-13 08:22:15 +01:00
Arnaud Gourlay
d7d5c9bb43 Pool larger memory batches for sparse vector search (#3560) 
* Pool larger memory batches for sparse vector search

* rework naming

* refine naming

* address great comments

* bump ADVANCE_BATCH_SIZE to use power of 2

* 16k performs bad - trying 8k

* 10k offers the best tradeoff
 2024-02-12 14:58:03 +01:00
Arnaud Gourlay
eec5022831 Optimize sparse search by advancing postings without binary search (#3527) 
* Optimize search by advancing posting list with binary search

* better name for offset
 2024-02-06 16:04:45 +01:00
Arnaud Gourlay
86ea2ae6cd Optimize sparse vector search with batched scoring (#3464) 
* Optimize sparse vector search with batched scoring

* replace roaring bitmap by id generation

* compute intersections in Vec instead of Map

* cargo update ahash@0.8.7 --precise 0.8.5

* let rustc handle the pattern match optimization

* profiling done

* remove Option and filter out zero scores

* restore integration test assertion

* filter based on min score to avoid work

* decrease batch size and cleanup posting creation

* remove outdated tests

* remove dead test - I know where to find it if I need it
 2024-02-05 15:47:11 +01:00
Roman Titov
b8627c5e8e Add clock_tag field to update operations (#3408) 2024-02-01 11:44:13 +01:00
Tim Visée
06712e8f4d Move common dependencies to workspace (#3490) 2024-01-31 09:01:38 +01:00
dependabot[bot]
1fc3a22f75 Bump itertools from 0.12.0 to 0.12.1 (#3486) 
Bumps [itertools](https://github.com/rust-itertools/itertools) from 0.12.0 to 0.12.1.
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.12.0...v0.12.1)

---
updated-dependencies:
- dependency-name: itertools
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-01-30 10:30:07 +01:00
dependabot[bot]
8e5587c390 Bump memmap2 from 0.9.3 to 0.9.4 (#3484) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.9.3 to 0.9.4.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.9.3...v0.9.4)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-01-30 09:42:42 +01:00
Arnaud Gourlay
7368e0a79b PostingElement does not need to be Copy (#3479) 2024-01-29 18:04:14 +01:00
Arnaud Gourlay
a20bb6ce5a Remove unused dependencies (#3435) 2024-01-22 15:36:38 +01:00
Arnaud Gourlay
3df6b60e14 Optimize sparse search reaching single posting list (#3424) 2024-01-19 14:04:07 +01:00
Arnaud Gourlay
7fde4ad32c Optimize sparse search by deleting empty postings (#3413) 2024-01-18 09:12:59 +01:00
Arnaud Gourlay
cfb3eb1470 Optimize sparse vector for real life distribution (#3393) 2024-01-17 11:02:55 +01:00
Arnaud Gourlay
511ffbc75a Handle empty sparse vector search (#3368) 2024-01-10 11:28:53 +01:00
Arnaud Gourlay
b3cbf521ab Optimize inverted index search by tracking min record ids (#3347) 
* single pass for min record_id with look-ahead

* Optimize inverted index by tracking min record ids in BinaryHeap
 2024-01-10 08:25:20 +01:00
dependabot[bot]
4b08e7310e Bump tempfile from 3.8.1 to 3.9.0 (#3353) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.8.1 to 3.9.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.8.1...v3.9.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-01-09 12:17:26 +01:00
Arnaud Gourlay
2d093e6bcd Check cancellation after filtering condition to speedup filtered sparse search (#3348) 2024-01-08 12:58:01 +01:00
Arnaud Gourlay
c49aa37116 Optimize sparse search by precomputing once the query info per posting interator (#3327) 2024-01-08 10:27:54 +01:00
Arnaud Gourlay
3b9b5126cc Large sparse vectors in test & bench (#3274) 2024-01-04 09:31:34 +01:00
dependabot[bot]
c1210186b0 Bump memmap2 from 0.9.2 to 0.9.3 (#3283) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.9.2 to 0.9.3.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.9.2...v0.9.3)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-01-02 11:36:14 +01:00
Arnaud Gourlay
3de1fc4a8b Optimize sparse inverted index advance (#3275) 2023-12-25 13:36:27 +00:00
Arnaud Gourlay
2cd426dccf Fix clippy 1.75 (#3270) 2023-12-25 13:26:21 +00:00
dependabot[bot]
71fb667c93 Bump ordered-float from 4.1.1 to 4.2.0 (#3242) 
Bumps [ordered-float](https://github.com/reem/rust-ordered-float) from 4.1.1 to 4.2.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases)
- [Commits](https://github.com/reem/rust-ordered-float/compare/v4.1.1...v4.2.0)

---
updated-dependencies:
- dependency-name: ordered-float
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-19 20:08:01 +01:00
dependabot[bot]
f1de3b9586 Bump memmap2 from 0.9.0 to 0.9.2 (#3239) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.9.0 to 0.9.2.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.9.0...v0.9.2)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-19 16:00:45 +00:00