upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-08-03 16:40:54 -05:00
Code Issues Packages Projects Releases Wiki Activity
2,354 Commits 353 Branches 116 Tags
f514b296415d8e2fe6ecf670aff4f7b05769fa99
Commit Graph

64 Commits

Author SHA1 Message Date
dependabot[bot]
b59145cec0 Bump tempfile from 3.9.0 to 3.10.0 (#3596) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.9.0 to 3.10.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.9.0...v3.10.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-05 17:06:54 +01:00
Arnaud Gourlay
02004c8720 Pool larger memory batches for sparse vector search (#3560) 
* Pool larger memory batches for sparse vector search

* rework naming

* refine naming

* address great comments

* bump ADVANCE_BATCH_SIZE to use power of 2

* 16k performs bad - trying 8k

* 10k offers the best tradeoff
 2024-03-05 17:06:10 +01:00
Arnaud Gourlay
a51275b476 Optimize sparse search by advancing postings without binary search (#3527) 
* Optimize search by advancing posting list with binary search

* better name for offset
 2024-03-05 17:01:42 +01:00
Arnaud Gourlay
7d6a0972e8 Optimize sparse vector search with batched scoring (#3464) 
* Optimize sparse vector search with batched scoring

* replace roaring bitmap by id generation

* compute intersections in Vec instead of Map

* cargo update ahash@0.8.7 --precise 0.8.5

* let rustc handle the pattern match optimization

* profiling done

* remove Option and filter out zero scores

* restore integration test assertion

* filter based on min score to avoid work

* decrease batch size and cleanup posting creation

* remove outdated tests

* remove dead test - I know where to find it if I need it
 2024-03-05 16:57:50 +01:00
Roman Titov
99b750fcfa Add clock_tag field to update operations (#3408) 2024-03-05 16:57:21 +01:00
Tim Visée
71bcbf4bb3 Move common dependencies to workspace (#3490) 2024-03-05 16:56:33 +01:00
dependabot[bot]
14bdcafffc Bump itertools from 0.12.0 to 0.12.1 (#3486) 
Bumps [itertools](https://github.com/rust-itertools/itertools) from 0.12.0 to 0.12.1.
- [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-itertools/itertools/compare/v0.12.0...v0.12.1)

---
updated-dependencies:
- dependency-name: itertools
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-05 16:55:58 +01:00
dependabot[bot]
bd0208dd1d Bump memmap2 from 0.9.3 to 0.9.4 (#3484) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.9.3 to 0.9.4.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.9.3...v0.9.4)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-05 16:55:51 +01:00
Arnaud Gourlay
abdb651f7a PostingElement does not need to be Copy (#3479) 2024-03-05 16:55:31 +01:00
Arnaud Gourlay
b26b87c8d5 Remove unused dependencies (#3435) 2024-03-05 16:49:09 +01:00
Arnaud Gourlay
64a024ba51 Optimize sparse search reaching single posting list (#3424) 2024-03-05 16:49:04 +01:00
Arnaud Gourlay
c6fc008f0e Optimize sparse search by deleting empty postings (#3413) 2024-03-05 16:48:35 +01:00
Arnaud Gourlay
3c87260a01 Optimize sparse vector for real life distribution (#3393) 2024-03-05 16:48:21 +01:00
Arnaud Gourlay
7ceaf73a46 Handle empty sparse vector search (#3368) 2024-03-05 16:46:27 +01:00
Arnaud Gourlay
f92a9af6d8 Optimize inverted index search by tracking min record ids (#3347) 
* single pass for min record_id with look-ahead

* Optimize inverted index by tracking min record ids in BinaryHeap
 2024-03-05 16:46:23 +01:00
dependabot[bot]
861714196b Bump tempfile from 3.8.1 to 3.9.0 (#3353) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.8.1 to 3.9.0.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/compare/v3.8.1...v3.9.0)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-05 16:46:10 +01:00
Arnaud Gourlay
3bcb7a5122 Check cancellation after filtering condition to speedup filtered sparse search (#3348) 2024-03-05 16:45:24 +01:00
Arnaud Gourlay
da8eed2af0 Optimize sparse search by precomputing once the query info per posting interator (#3327) 2024-03-05 16:45:15 +01:00
Arnaud Gourlay
d88d654873 Large sparse vectors in test & bench (#3274) 2024-03-05 16:44:07 +01:00
dependabot[bot]
3b659b4ee2 Bump memmap2 from 0.9.2 to 0.9.3 (#3283) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.9.2 to 0.9.3.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.9.2...v0.9.3)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-05 16:43:39 +01:00
Arnaud Gourlay
b67a6d7f5c Optimize sparse inverted index advance (#3275) 2024-03-05 16:42:17 +01:00
Arnaud Gourlay
680574347f Fix clippy 1.75 (#3270) 2024-01-29 15:23:22 +01:00
dependabot[bot]
2224002c86 Bump ordered-float from 4.1.1 to 4.2.0 (#3242) 
Bumps [ordered-float](https://github.com/reem/rust-ordered-float) from 4.1.1 to 4.2.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases)
- [Commits](https://github.com/reem/rust-ordered-float/compare/v4.1.1...v4.2.0)

---
updated-dependencies:
- dependency-name: ordered-float
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-19 19:31:36 +00:00
dependabot[bot]
700f2bbbf9 Bump memmap2 from 0.9.0 to 0.9.2 (#3239) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.9.0 to 0.9.2.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.9.0...v0.9.2)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-19 19:29:29 +00:00
Ivan Pleshkov
772a8a75d9 sparse vectors with large indices support (#3230) 
* sparse vectors with large indices support

* fix unit tests

* add tracker file to snapshot

* fix unit tests

* are you happy clippy

* fix config loading

* test with max value

* fix python tests

* fix consensus tests

* review remarks; acc test for inverted index
 2023-12-19 19:28:13 +00:00
Arnaud Gourlay
2fdab78789 Faster sparse vectors plain search (#3231) 
* Faster sparse vectors plain search

* no comments
 2023-12-19 19:28:03 +00:00
Ivan Pleshkov
8558399b4e Load non-appendable ram sparse vector index from file (#3154) 
* internal sparse vector segment configs

update openapi

fix build

provide correct index type to fixtures

internal sparse vector segment configs

update openapi

fix build

load ram sparse index from file

fix spelling

add test

spelling error

use appendable flag to decide load or not

fix conflict

dont flush mmap when file is empty

* fix build

* set list of files for snapshot

* fix snapshots test

* fix codespell

* simple save-load unit test

* refactor InvertedIndex::files

* fix unit tests

* add debug_assert

* test all persistence loading cases
 2023-12-06 22:16:30 +01:00
Ivan Pleshkov
d5a705e0fa Sparse vectors advanced search api support (#3128) 
* sparse vectors query scorer

plain search test without internals

fix sparse_vector_index_plain_search

fix async scorer build

fix unit tests

are you happy fmt

sparse index with full query support

fix benches, use exact flag to force plain search

add alias and comment to exact flag

fix tests

recommendations TODOs fix

add simple test

add todo

refactor index

test vs comparison dense and sparse discovery

are you happy fmt

propogate error from avg negatives

reuse filtered points list

rollback api changes

better discovery test

are you happy fmt

rollback openapi

fix build

fix tests

review remark recommendations

review remarks

review remarks

codespell

integrate plain search

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

fix comment of plain search

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

fix tests

* simplify search query function
 2023-12-06 17:20:42 +01:00
Arnaud Gourlay
67b464174f Plain search by ids in parse vector inverted index (#3161) 2023-12-06 17:20:18 +01:00
Arnaud Gourlay
b0bd9b2bf9 Mmap InvertedIndex uses Sequential madvice (#3092) 
* Mmap InvertedIndex uses Sequential madvice

* use madvice Normal

* fix new bench compile after rebase
 2023-12-06 17:18:56 +01:00
Arnaud Gourlay
82c08843b4 Fix type for SparseVector converter (#3136) 2023-12-06 17:18:48 +01:00
Ivan Pleshkov
286caac3a4 Sparse index persistence (#3131) 
* sparse index persistence

* fix tests

* remove defaults
 2023-12-06 17:18:38 +01:00
Arnaud Gourlay
85c96cdd9a Integration tests sparse vector validation (#3122) 
* Integration test for sparse vector validations

* cleanup

* clean again
 2023-12-06 17:16:12 +01:00
Arnaud Gourlay
2fae4e491d Plain search for inverted index (#3113) 2023-12-06 17:15:48 +01:00
Ivan Pleshkov
47690244bc Sparse vector grpc definition (#3070) 
* Sparse vector grpc definition

remove obsolete use

update grpc docs

update openapi

more convertion fixes

update grpc docs

* validations

* review renamings

* add todos
 2023-12-06 17:11:06 +01:00
Arnaud Gourlay
bbc0f12dbc Fix validation of unicity of sparse index (#3072) 2023-12-06 17:09:08 +01:00
Arnaud Gourlay
a09a630367 Accept empty sparse vectors (#3057) 
* Accept empty sparse vectors

* sparse vector index is responsbile for filtering empty vectors
 2023-12-06 17:09:01 +01:00
dependabot[bot]
cac1e4c429 Bump schemars from 0.8.15 to 0.8.16 (#3058) 
Bumps [schemars](https://github.com/GREsau/schemars) from 0.8.15 to 0.8.16.
- [Release notes](https://github.com/GREsau/schemars/releases)
- [Changelog](https://github.com/GREsau/schemars/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GREsau/schemars/compare/v0.8.15...v0.8.16)

---
updated-dependencies:
- dependency-name: schemars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-06 17:07:45 +01:00
Arnaud Gourlay
de390f51bf Better heuristic for sparse vector pruning (#3048) 2023-12-06 17:07:23 +01:00
Arnaud Gourlay
05ea913324 Merge scoring and overlap check for sparse vector (#3047) 2023-12-06 17:07:08 +01:00
Ivan Pleshkov
86153086ec Sparse vectors api preparations (#3038) 
* Sparse vectors api preparations

generic transpose

remove features

unnecessary local variable

add snake case

* distance in config function
 2023-12-06 17:06:25 +01:00
Arnaud Gourlay
73d013da3c Remove sorted constraint on sparse vectors (#3030) 
* Remove sorted constraint on sparse vectors

* make sure to sort the query as well
 2023-12-06 17:05:04 +01:00
Arnaud Gourlay
6d0a48b2dc Deactivate sparse index pruning for negative query vectors (#3024) 2023-12-06 17:04:27 +01:00
Arnaud Gourlay
bc68840558 Improve sparse index pruning (#3011) 
* Improve sparse index pruning

* add test for promote longest

* document and test possible underpruning
 2023-12-06 17:04:01 +01:00
Arnaud Gourlay
c2dd19ae8c Benchmark sparse index build (#2984) 
* Benchmark sparse index build

* add vector count assertion
 2023-12-06 17:03:30 +01:00
Arnaud Gourlay
28fa2eedf8 Add fallback to plain search for sparse vectors (#2963) 2023-12-06 17:03:21 +01:00
Arnaud Gourlay
8f844b6635 Fix vector count in sparse index (#2992) 
* Fix vector count in sparse index

* simplify naming

* rely on max id for count
 2023-12-06 17:02:36 +01:00
dependabot[bot]
f502fdfa0d Bump memmap2 from 0.7.1 to 0.9.0 (#2999) 
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.7.1 to 0.9.0.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.7.1...v0.9.0)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-06 17:01:51 +01:00
dependabot[bot]
eaf51c3a8e Bump tempfile from 3.8.0 to 3.8.1 (#2996) 
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.8.0 to 3.8.1.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/commits)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-12-06 17:01:41 +01:00
Arnaud Gourlay
da991a505f Sparse vector validation (#2965) 
* Sparse vector validation

* cheaper indices uniqueness validation and tests
 2023-12-06 16:59:25 +01:00