upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-07-31 15:11:35 -05:00
Code Issues Packages Projects Releases Wiki Activity
2,826 Commits 343 Branches 116 Tags
83b93b951002a75845bc00ec00b50dafae86bfc8
Commit Graph

85 Commits

Author SHA1 Message Date
xzfc
d2ef556237 Make sparse vectors generic (#4364) 
* Make sparse vector generic

* Generic search_context tests
 2024-06-10 15:04:32 +00:00
xzfc
f4c7ba15af Sparse: misc fixes (#4361) 
* Borrow cows
* Hide sparse_vector_fixture under feature="testing"
 2024-05-30 20:58:00 +00:00
xzfc
cd808afe3c Improve performance of compressed posting list (#4350) 
* Improve performance of compressed posting list

* Merge `compressed_idx` and `remainders_idx`
 2024-05-30 13:30:24 +00:00
xzfc
f026fc8a42 Recreate sparse vector index (#4313) 
* Recreate sparse vector index

* Rename index file; add simple migration
 2024-05-29 17:04:17 +00:00
dependabot[bot]
6ebd22d487 Bump parking_lot from 0.12.2 to 0.12.3 (#4337) 
Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.12.2 to 0.12.3.
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Amanieu/parking_lot/compare/0.12.2...0.12.3)

---
updated-dependencies:
- dependency-name: parking_lot
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-05-28 09:02:20 +02:00
Luis Cossío
b14d0b2463 universal-query: user-facing gRPC and REST request types (#4303) 
* add grpc `QueryPoints`

* add rest types

* move to REST types to `api::rest::schema`, decouple `OrderByInterface` and `RecommendStrategy`

* openapi changes

* add score_threshold param

* decouple `ShardKeySelector`, add shard key param

* remove commented-out code

* new codespell
 2024-05-24 08:23:12 -04:00
Arnaud Gourlay
cbcdefb310 Cleanup unused dependencies (#4306) 2024-05-23 10:06:44 +02:00
xzfc
93a42b4e2d Compressed posting lists (#4253) 2024-05-22 18:18:20 +00:00
xzfc
ccf7f1d24f InvertedIndexImmutableRam and index migrations (#4220) 
* Move StorageVersion from segment crate to common/io

* Refine StorageVersion API

* Move methods from SparseVectorDataConfig to enum SparseIndexType

* Introduce InvertedIndexImmutableRam

* Add migrate

* Don't migrate
 2024-05-16 06:47:47 +00:00
Luis Cossío
bfa72bb6d8 universal-query: Add query() to ShardOperation trait (#4210) 
* add `query` to shard trait

* add missing conversions for query

* update grpc docs

* Query response has intermediate results

* add ShardQueryResponse description

* move pub use to the top, keep only one way of reaching reexports
 2024-05-15 09:36:46 -04:00
xzfc
4f3239f27c Misc fixes (#4212) 
* SparseVectorIndex: replace pub with feature-gated getters

* Fixup dev-dependencies

* Fixup rust doc comments
 2024-05-10 08:16:28 +00:00
xzfc
6f738ca95f Update sparse vector benchmarks (#4163) 2024-05-07 20:34:37 +00:00
xzfc
cb31c533bf Introduce Cargo feature "testing" (#4192) 2024-05-07 16:14:46 +00:00
dependabot[bot]
699d83036d Bump parking_lot from 0.12.1 to 0.12.2 (#4136) 2024-04-29 22:45:18 +02:00
Andrey Vasnetsov
c173a9f5e5 Sparse idf dot (#4126) 
* introduce QueryContext, which accumulates runtime info needed for executing search

* fmt

* propagate query context into segment internals

* [WIP] prepare idf stats for search query context

* Split SparseVector and RemmapedSparseVector to guarantee we will not mix them up on the type level

* implement filling of the query context with IDF statistics

* implement re-weighting of the sparse query with idf

* fmt

* update idf param only if explicitly specified (more consistent with diff param update

* replace idf bool with modifier enum, improve further extensibility

* test and fixes

* Update lib/collection/src/operations/types.rs

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>

* review fixes

* fmt

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
 2024-04-29 14:54:14 +02:00
Arnaud Gourlay
abcd2af537 Median based TopK for sparse vectors scoring (#4037) 
* Median based TopK for sparse vectors scoring

* add test with identical scores
 2024-04-19 13:27:21 +02:00
Arnaud Gourlay
c6e7c6f2b2 Fix average sparse vector recommendation (#4047) 
* Fix average sparse vector recommendation

* Revert "Fix average sparse vector recommendation"

This reverts commit 186c51209c.

* sort within combine (Ivan's approach)

* Revert "sort within combine (Ivan's approach)"

This reverts commit 28d560cc56.

* simpler approach without mutation
 2024-04-17 13:53:59 +02:00
Arnaud Gourlay
4ff2fb855b promote Validator to workspace (#4013) 2024-04-11 12:01:27 +02:00
dependabot[bot]
1c1b74ec99 Bump tempfile from 3.10.0 to 3.10.1 (#3763) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.10.0 to 3.10.1.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.10.0...v3.10.1)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-05 10:39:48 +01:00
Arnaud Gourlay
ab90b03aa0 Remove unused proptest on Sparse crate (#3667) 2024-02-22 11:35:22 +01:00
Arnaud Gourlay
722d905010 Optimize sparse vector index build time (#3605) 
* Optimize sparse vector index build time

* rework index builder to reuse posting list builder

* remove unecessary change

* make it shorter
 2024-02-20 12:01:42 +01:00
dependabot[bot]
b7165b5a81 Bump tempfile from 3.9.0 to 3.10.0 (#3596) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.9.0 to 3.10.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.9.0...v3.10.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-02-13 08:22:15 +01:00
Arnaud Gourlay
d7d5c9bb43 Pool larger memory batches for sparse vector search (#3560) 
* Pool larger memory batches for sparse vector search

* rework naming

* refine naming

* address great comments

* bump ADVANCE_BATCH_SIZE to use power of 2

* 16k performs bad - trying 8k

* 10k offers the best tradeoff
 2024-02-12 14:58:03 +01:00
Arnaud Gourlay
eec5022831 Optimize sparse search by advancing postings without binary search (#3527) 
* Optimize search by advancing posting list with binary search

* better name for offset
 2024-02-06 16:04:45 +01:00
Arnaud Gourlay
86ea2ae6cd Optimize sparse vector search with batched scoring (#3464) 
* Optimize sparse vector search with batched scoring

* replace roaring bitmap by id generation

* compute intersections in Vec instead of Map

* cargo update ahash@0.8.7 --precise 0.8.5

* let rustc handle the pattern match optimization

* profiling done

* remove Option and filter out zero scores

* restore integration test assertion

* filter based on min score to avoid work

* decrease batch size and cleanup posting creation

* remove outdated tests

* remove dead test - I know where to find it if I need it
 2024-02-05 15:47:11 +01:00
Roman Titov
b8627c5e8e Add clock_tag field to update operations (#3408) 2024-02-01 11:44:13 +01:00
Tim Visée
06712e8f4d Move common dependencies to workspace (#3490) 2024-01-31 09:01:38 +01:00
dependabot[bot]
1fc3a22f75 Bump itertools from 0.12.0 to 0.12.1 (#3486) 
Bumps [itertools](https://github.com/rust-itertools/itertools) from 0.12.0 to 0.12.1.
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.12.0...v0.12.1)

---
updated-dependencies:
- dependency-name: itertools
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-01-30 10:30:07 +01:00
dependabot[bot]
8e5587c390 Bump memmap2 from 0.9.3 to 0.9.4 (#3484) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.9.3 to 0.9.4.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.9.3...v0.9.4)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-01-30 09:42:42 +01:00
Arnaud Gourlay
7368e0a79b PostingElement does not need to be Copy (#3479) 2024-01-29 18:04:14 +01:00
Arnaud Gourlay
a20bb6ce5a Remove unused dependencies (#3435) 2024-01-22 15:36:38 +01:00
Arnaud Gourlay
3df6b60e14 Optimize sparse search reaching single posting list (#3424) 2024-01-19 14:04:07 +01:00
Arnaud Gourlay
7fde4ad32c Optimize sparse search by deleting empty postings (#3413) 2024-01-18 09:12:59 +01:00
Arnaud Gourlay
cfb3eb1470 Optimize sparse vector for real life distribution (#3393) 2024-01-17 11:02:55 +01:00
Arnaud Gourlay
511ffbc75a Handle empty sparse vector search (#3368) 2024-01-10 11:28:53 +01:00
Arnaud Gourlay
b3cbf521ab Optimize inverted index search by tracking min record ids (#3347) 
* single pass for min record_id with look-ahead

* Optimize inverted index by tracking min record ids in BinaryHeap
 2024-01-10 08:25:20 +01:00
dependabot[bot]
4b08e7310e Bump tempfile from 3.8.1 to 3.9.0 (#3353) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.8.1 to 3.9.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.8.1...v3.9.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-01-09 12:17:26 +01:00
Arnaud Gourlay
2d093e6bcd Check cancellation after filtering condition to speedup filtered sparse search (#3348) 2024-01-08 12:58:01 +01:00
Arnaud Gourlay
c49aa37116 Optimize sparse search by precomputing once the query info per posting interator (#3327) 2024-01-08 10:27:54 +01:00
Arnaud Gourlay
3b9b5126cc Large sparse vectors in test & bench (#3274) 2024-01-04 09:31:34 +01:00
dependabot[bot]
c1210186b0 Bump memmap2 from 0.9.2 to 0.9.3 (#3283) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.9.2 to 0.9.3.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.9.2...v0.9.3)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-01-02 11:36:14 +01:00
Arnaud Gourlay
3de1fc4a8b Optimize sparse inverted index advance (#3275) 2023-12-25 13:36:27 +00:00
Arnaud Gourlay
2cd426dccf Fix clippy 1.75 (#3270) 2023-12-25 13:26:21 +00:00
dependabot[bot]
71fb667c93 Bump ordered-float from 4.1.1 to 4.2.0 (#3242) 
Bumps [ordered-float](https://github.com/reem/rust-ordered-float) from 4.1.1 to 4.2.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases)
- [Commits](https://github.com/reem/rust-ordered-float/compare/v4.1.1...v4.2.0)

---
updated-dependencies:
- dependency-name: ordered-float
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-19 20:08:01 +01:00
dependabot[bot]
f1de3b9586 Bump memmap2 from 0.9.0 to 0.9.2 (#3239) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.9.0 to 0.9.2.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.9.0...v0.9.2)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-19 16:00:45 +00:00
Ivan Pleshkov
84f9011537 sparse vectors with large indices support (#3230) 
* sparse vectors with large indices support

* fix unit tests

* add tracker file to snapshot

* fix unit tests

* are you happy clippy

* fix config loading

* test with max value

* fix python tests

* fix consensus tests

* review remarks; acc test for inverted index
 2023-12-16 00:02:01 +00:00
Arnaud Gourlay
8d11a3fbc6 Faster sparse vectors plain search (#3231) 
* Faster sparse vectors plain search

* no comments
 2023-12-15 14:50:06 +00:00
Ivan Pleshkov
c054773296 Load non-appendable ram sparse vector index from file (#3154) 
* internal sparse vector segment configs

update openapi

fix build

provide correct index type to fixtures

internal sparse vector segment configs

update openapi

fix build

load ram sparse index from file

fix spelling

add test

spelling error

use appendable flag to decide load or not

fix conflict

dont flush mmap when file is empty

* fix build

* set list of files for snapshot

* fix snapshots test

* fix codespell

* simple save-load unit test

* refactor InvertedIndex::files

* fix unit tests

* add debug_assert

* test all persistence loading cases
 2023-12-06 17:36:08 +01:00
Ivan Pleshkov
7d57fb8859 Sparse vectors advanced search api support (#3128) 
* sparse vectors query scorer

plain search test without internals

fix sparse_vector_index_plain_search

fix async scorer build

fix unit tests

are you happy fmt

sparse index with full query support

fix benches, use exact flag to force plain search

add alias and comment to exact flag

fix tests

recommendations TODOs fix

add simple test

add todo

refactor index

test vs comparison dense and sparse discovery

are you happy fmt

propogate error from avg negatives

reuse filtered points list

rollback api changes

better discovery test

are you happy fmt

rollback openapi

fix build

fix tests

review remark recommendations

review remarks

review remarks

codespell

integrate plain search

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

fix comment of plain search

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

fix tests

* simplify search query function
 2023-12-05 20:32:23 +01:00
Arnaud Gourlay
07b977706c Plain search by ids in parse vector inverted index (#3161) 2023-12-05 16:12:57 +01:00