Commit Graph

140 Commits

Author SHA1 Message Date
Ivan Pleshkov
84f9011537 sparse vectors with large indices support (#3230)
* sparse vectors with large indices support

* fix unit tests

* add tracker file to snapshot

* fix unit tests

* are you happy clippy

* fix config loading

* test with max value

* fix python tests

* fix consensus tests

* review remarks; acc test for inverted index
2023-12-16 00:02:01 +00:00
Arnaud Gourlay
8d11a3fbc6 Faster sparse vectors plain search (#3231)
* Faster sparse vectors plain search

* no comments
2023-12-15 14:50:06 +00:00
Ivan Pleshkov
c054773296 Load non-appendable ram sparse vector index from file (#3154)
* internal sparse vector segment configs

update openapi

fix build

provide correct index type to fixtures

internal sparse vector segment configs

update openapi

fix build

load ram sparse index from file

fix spelling

add test

spelling error

use appendable flag to decide load or not

fix conflict

dont flush mmap when file is empty

* fix build

* set list of files for snapshot

* fix snapshots test

* fix codespell

* simple save-load unit test

* refactor InvertedIndex::files

* fix unit tests

* add debug_assert

* test all persistence loading cases
2023-12-06 17:36:08 +01:00
Ivan Pleshkov
7d57fb8859 Sparse vectors advanced search api support (#3128)
* sparse vectors query scorer

plain search test without internals

fix sparse_vector_index_plain_search

fix async scorer build

fix unit tests

are you happy fmt

sparse index with full query support

fix benches, use exact flag to force plain search

add alias and comment to exact flag

fix tests

recommendations TODOs fix

add simple test

add todo

refactor index

test vs comparison dense and sparse discovery

are you happy fmt

propogate error from avg negatives

reuse filtered points list

rollback api changes

better discovery test

are you happy fmt

rollback openapi

fix build

fix tests

review remark recommendations

review remarks

review remarks

codespell

integrate plain search

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

fix comment of plain search

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

fix tests

* simplify search query function
2023-12-05 20:32:23 +01:00
Arnaud Gourlay
07b977706c Plain search by ids in parse vector inverted index (#3161) 2023-12-05 16:12:57 +01:00
Arnaud Gourlay
f39c95c34b Mmap InvertedIndex uses Sequential madvice (#3092)
* Mmap InvertedIndex uses Sequential madvice

* use madvice Normal

* fix new bench compile after rebase
2023-12-04 14:55:35 +01:00
Arnaud Gourlay
73bf1c05fe Fix type for SparseVector converter (#3136) 2023-12-04 12:54:01 +01:00
Ivan Pleshkov
d272bf4b90 Sparse index persistence (#3131)
* sparse index persistence

* fix tests

* remove defaults
2023-12-04 12:02:53 +01:00
Arnaud Gourlay
0b3efb2bc2 Integration tests sparse vector validation (#3122)
* Integration test for sparse vector validations

* cleanup

* clean again
2023-12-01 14:08:42 +01:00
Arnaud Gourlay
ae68067c28 Plain search for inverted index (#3113) 2023-11-30 15:29:53 +01:00
Ivan Pleshkov
4f7afa46c3 Sparse vector grpc definition (#3070)
* Sparse vector grpc definition

remove obsolete use

update grpc docs

update openapi

more convertion fixes

update grpc docs

* validations

* review renamings

* add todos
2023-11-23 14:01:29 +01:00
Arnaud Gourlay
8cad65b218 Fix validation of unicity of sparse index (#3072) 2023-11-21 12:08:51 +01:00
Arnaud Gourlay
bbc7b37123 Accept empty sparse vectors (#3057)
* Accept empty sparse vectors

* sparse vector index is responsbile for filtering empty vectors
2023-11-21 11:27:12 +01:00
dependabot[bot]
c9b9646432 Bump schemars from 0.8.15 to 0.8.16 (#3058)
Bumps [schemars](https://github.com/GREsau/schemars) from 0.8.15 to 0.8.16.
- [Release notes](https://github.com/GREsau/schemars/releases)
- [Changelog](https://github.com/GREsau/schemars/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GREsau/schemars/compare/v0.8.15...v0.8.16)

---
updated-dependencies:
- dependency-name: schemars
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-21 08:21:30 +01:00
Arnaud Gourlay
d69b38a05d Better heuristic for sparse vector pruning (#3048) 2023-11-20 15:58:10 +01:00
Arnaud Gourlay
e7fa2e0810 Merge scoring and overlap check for sparse vector (#3047) 2023-11-20 15:43:48 +01:00
Ivan Pleshkov
ee23eda42c Sparse vectors api preparations (#3038)
* Sparse vectors api preparations

generic transpose

remove features

unnecessary local variable

add snake case

* distance in config function
2023-11-17 16:22:10 +01:00
Arnaud Gourlay
e386f58695 Remove sorted constraint on sparse vectors (#3030)
* Remove sorted constraint on sparse vectors

* make sure to sort the query as well
2023-11-17 14:01:08 +01:00
Arnaud Gourlay
3a0353ede0 Deactivate sparse index pruning for negative query vectors (#3024) 2023-11-16 10:51:02 +01:00
Arnaud Gourlay
04052b7722 Improve sparse index pruning (#3011)
* Improve sparse index pruning

* add test for promote longest

* document and test possible underpruning
2023-11-15 14:35:42 +01:00
Arnaud Gourlay
7a140b69da Benchmark sparse index build (#2984)
* Benchmark sparse index build

* add vector count assertion
2023-11-14 20:01:50 +01:00
Arnaud Gourlay
6266a438f0 Add fallback to plain search for sparse vectors (#2963) 2023-11-14 19:58:52 +01:00
Arnaud Gourlay
46c5b2a30c Fix vector count in sparse index (#2992)
* Fix vector count in sparse index

* simplify naming

* rely on max id for count
2023-11-14 17:31:18 +01:00
dependabot[bot]
629e71c08d Bump memmap2 from 0.7.1 to 0.9.0 (#2999)
Bumps [memmap2](https://github.com/RazrFalcon/memmap2-rs) from 0.7.1 to 0.9.0.
- [Changelog](https://github.com/RazrFalcon/memmap2-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/RazrFalcon/memmap2-rs/compare/v0.7.1...v0.9.0)

---
updated-dependencies:
- dependency-name: memmap2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-14 08:59:46 +01:00
dependabot[bot]
7331952054 Bump tempfile from 3.8.0 to 3.8.1 (#2996)
Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.8.0 to 3.8.1.
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Stebalien/tempfile/commits)

---
updated-dependencies:
- dependency-name: tempfile
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-14 07:19:29 +01:00
Arnaud Gourlay
bc5d0976a1 Sparse vector validation (#2965)
* Sparse vector validation

* cheaper indices uniqueness validation and tests
2023-11-10 16:42:10 +01:00
Tim Visée
9394fc1757 Some micro optimizations in reserve/resize usage (#2967)
* In HNSW graph layer building, infer right capacity on collect

* Resize with empty vector, use constant function rather than clone

* Use resize_with in other places as well
2023-11-10 09:42:14 +01:00
Arnaud Gourlay
1ae1f483d4 Benchmark sparse filter plain search (#2947)
* Sparse filter plain search

* code review
2023-11-09 09:57:43 +01:00
Arnaud Gourlay
186c906d6d Faster search in sparse inverted index with unstable sort (#2937) 2023-11-07 14:51:21 +01:00
Arnaud Gourlay
ec2dc36719 SparseVectorIndex implements VectorIndex (#2900)
* SparseVector implements VectorIndex

* dedicated telemetry

* conflict

* easy code reviews

* simplify tracking indexed points count

* move telemetry conversion to sparse file

* move max_result_count out of inverted index trait

* unify sparse vector fixtures

* simpler conversion

* add todo regarding OOM potential

* reuse check deleted from raw scorer with TODO

* change new to open to handle mmap index
2023-11-03 16:23:05 +01:00
Arnaud Gourlay
0fa6688764 Document and test underfetching with sparse vectors (#2908)
* Document and test underfetching with sparse vectors

* deduplicate test point ids
2023-11-02 15:30:43 +01:00
Arnaud Gourlay
8a05462c6d Raw scorer for sparse vectors (#2915)
* Raw scorer for sparse vectors

* sparse vector storage trait

* use get sparse vector fn

* remove dedicated types

---------

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
2023-11-02 14:59:18 +01:00
Arnaud Gourlay
460832cc70 Rename sparse vector values (#2892) 2023-10-31 11:52:45 +01:00
Arnaud Gourlay
97ffee476e Simple sparse vector storage (#2806)
* Simple sparse vector storage

* better todo
2023-10-30 11:27:36 +01:00
Arnaud Gourlay
ecaff66f4c Sparse vector index integration (#2835)
* inverted index as trait

* introduce sparse vector index integration

* sparse vectors search cancellation

* clippy

* expose helper for search sparse

* code review
2023-10-18 12:03:04 +02:00
Arnaud Gourlay
4df28d250d Upsert operation on Sparse RAM index (#2826)
* Upsert operaiton on Sparse RAM index

* code review Ivan

* fix insertion of posting not at the end

* cleaner shortcut of upsert
2023-10-17 12:10:27 +02:00
Arnaud Gourlay
fc0b77636a Sparse index persistence (#2718)
* Sparse index persistence

* code review

* remove pub visibility
2023-10-04 14:39:52 +02:00
Arnaud Gourlay
5bb80fbc77 Move ScoredPointOffset into common (#2734) 2023-09-28 12:11:29 +02:00
Roman Titov
abf533671a Refactor lib/common to support adding sub-crates to "common" (#2551) 2023-09-20 13:13:32 +02:00
Arnaud Gourlay
5fe82cb6d7 Sparse vector index (#2318)
* Sparse vector index

* better name

* fill properly postings

* conflict

* easy reviews

* use unstable sort

* use optional instead of bool flag

* clearer score type and future direction

* NonZeroUsize length for FixedLengthPriorityQueue

* rename id to record_id

* cleanup dimension inverted index

* standard Iterator

* code review

* Shift comment

* early return if no matching posting lists

* sprinkle iterators

* privatize the means of production

---------

Co-authored-by: timvisee <tim@visee.me>
2023-09-15 16:30:11 +02:00