Commit Graph

51 Commits

Author SHA1 Message Date
Arnaud Gourlay
6705f81732 Optimize sparse vector index build time (#3605)
* Optimize sparse vector index build time

* rework index builder to reuse posting list builder

* remove unecessary change

* make it shorter
2024-03-05 17:09:03 +01:00
Arnaud Gourlay
02004c8720 Pool larger memory batches for sparse vector search (#3560)
* Pool larger memory batches for sparse vector search

* rework naming

* refine naming

* address great comments

* bump ADVANCE_BATCH_SIZE to use power of 2

* 16k performs bad - trying 8k

* 10k offers the best tradeoff
2024-03-05 17:06:10 +01:00
Arnaud Gourlay
a51275b476 Optimize sparse search by advancing postings without binary search (#3527)
* Optimize search by advancing posting list with binary search

* better name for offset
2024-03-05 17:01:42 +01:00
Arnaud Gourlay
7d6a0972e8 Optimize sparse vector search with batched scoring (#3464)
* Optimize sparse vector search with batched scoring

* replace roaring bitmap by id generation

* compute intersections in Vec instead of Map

* cargo update ahash@0.8.7 --precise 0.8.5

* let rustc handle the pattern match optimization

* profiling done

* remove Option and filter out zero scores

* restore integration test assertion

* filter based on min score to avoid work

* decrease batch size and cleanup posting creation

* remove outdated tests

* remove dead test - I know where to find it if I need it
2024-03-05 16:57:50 +01:00
Arnaud Gourlay
abdb651f7a PostingElement does not need to be Copy (#3479) 2024-03-05 16:55:31 +01:00
Arnaud Gourlay
64a024ba51 Optimize sparse search reaching single posting list (#3424) 2024-03-05 16:49:04 +01:00
Arnaud Gourlay
c6fc008f0e Optimize sparse search by deleting empty postings (#3413) 2024-03-05 16:48:35 +01:00
Arnaud Gourlay
3c87260a01 Optimize sparse vector for real life distribution (#3393) 2024-03-05 16:48:21 +01:00
Arnaud Gourlay
7ceaf73a46 Handle empty sparse vector search (#3368) 2024-03-05 16:46:27 +01:00
Arnaud Gourlay
f92a9af6d8 Optimize inverted index search by tracking min record ids (#3347)
* single pass for min record_id with look-ahead

* Optimize inverted index by tracking min record ids in BinaryHeap
2024-03-05 16:46:23 +01:00
Arnaud Gourlay
3bcb7a5122 Check cancellation after filtering condition to speedup filtered sparse search (#3348) 2024-03-05 16:45:24 +01:00
Arnaud Gourlay
da8eed2af0 Optimize sparse search by precomputing once the query info per posting interator (#3327) 2024-03-05 16:45:15 +01:00
Arnaud Gourlay
d88d654873 Large sparse vectors in test & bench (#3274) 2024-03-05 16:44:07 +01:00
Arnaud Gourlay
b67a6d7f5c Optimize sparse inverted index advance (#3275) 2024-03-05 16:42:17 +01:00
Arnaud Gourlay
680574347f Fix clippy 1.75 (#3270) 2024-01-29 15:23:22 +01:00
Ivan Pleshkov
772a8a75d9 sparse vectors with large indices support (#3230)
* sparse vectors with large indices support

* fix unit tests

* add tracker file to snapshot

* fix unit tests

* are you happy clippy

* fix config loading

* test with max value

* fix python tests

* fix consensus tests

* review remarks; acc test for inverted index
2023-12-19 19:28:13 +00:00
Arnaud Gourlay
2fdab78789 Faster sparse vectors plain search (#3231)
* Faster sparse vectors plain search

* no comments
2023-12-19 19:28:03 +00:00
Ivan Pleshkov
8558399b4e Load non-appendable ram sparse vector index from file (#3154)
* internal sparse vector segment configs

update openapi

fix build

provide correct index type to fixtures

internal sparse vector segment configs

update openapi

fix build

load ram sparse index from file

fix spelling

add test

spelling error

use appendable flag to decide load or not

fix conflict

dont flush mmap when file is empty

* fix build

* set list of files for snapshot

* fix snapshots test

* fix codespell

* simple save-load unit test

* refactor InvertedIndex::files

* fix unit tests

* add debug_assert

* test all persistence loading cases
2023-12-06 22:16:30 +01:00
Ivan Pleshkov
d5a705e0fa Sparse vectors advanced search api support (#3128)
* sparse vectors query scorer

plain search test without internals

fix sparse_vector_index_plain_search

fix async scorer build

fix unit tests

are you happy fmt

sparse index with full query support

fix benches, use exact flag to force plain search

add alias and comment to exact flag

fix tests

recommendations TODOs fix

add simple test

add todo

refactor index

test vs comparison dense and sparse discovery

are you happy fmt

propogate error from avg negatives

reuse filtered points list

rollback api changes

better discovery test

are you happy fmt

rollback openapi

fix build

fix tests

review remark recommendations

review remarks

review remarks

codespell

integrate plain search

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

fix comment of plain search

Update lib/collection/src/recommendations.rs

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>

fix tests

* simplify search query function
2023-12-06 17:20:42 +01:00
Arnaud Gourlay
67b464174f Plain search by ids in parse vector inverted index (#3161) 2023-12-06 17:20:18 +01:00
Arnaud Gourlay
b0bd9b2bf9 Mmap InvertedIndex uses Sequential madvice (#3092)
* Mmap InvertedIndex uses Sequential madvice

* use madvice Normal

* fix new bench compile after rebase
2023-12-06 17:18:56 +01:00
Arnaud Gourlay
82c08843b4 Fix type for SparseVector converter (#3136) 2023-12-06 17:18:48 +01:00
Ivan Pleshkov
286caac3a4 Sparse index persistence (#3131)
* sparse index persistence

* fix tests

* remove defaults
2023-12-06 17:18:38 +01:00
Arnaud Gourlay
85c96cdd9a Integration tests sparse vector validation (#3122)
* Integration test for sparse vector validations

* cleanup

* clean again
2023-12-06 17:16:12 +01:00
Arnaud Gourlay
2fae4e491d Plain search for inverted index (#3113) 2023-12-06 17:15:48 +01:00
Ivan Pleshkov
47690244bc Sparse vector grpc definition (#3070)
* Sparse vector grpc definition

remove obsolete use

update grpc docs

update openapi

more convertion fixes

update grpc docs

* validations

* review renamings

* add todos
2023-12-06 17:11:06 +01:00
Arnaud Gourlay
bbc0f12dbc Fix validation of unicity of sparse index (#3072) 2023-12-06 17:09:08 +01:00
Arnaud Gourlay
a09a630367 Accept empty sparse vectors (#3057)
* Accept empty sparse vectors

* sparse vector index is responsbile for filtering empty vectors
2023-12-06 17:09:01 +01:00
Arnaud Gourlay
de390f51bf Better heuristic for sparse vector pruning (#3048) 2023-12-06 17:07:23 +01:00
Arnaud Gourlay
05ea913324 Merge scoring and overlap check for sparse vector (#3047) 2023-12-06 17:07:08 +01:00
Ivan Pleshkov
86153086ec Sparse vectors api preparations (#3038)
* Sparse vectors api preparations

generic transpose

remove features

unnecessary local variable

add snake case

* distance in config function
2023-12-06 17:06:25 +01:00
Arnaud Gourlay
73d013da3c Remove sorted constraint on sparse vectors (#3030)
* Remove sorted constraint on sparse vectors

* make sure to sort the query as well
2023-12-06 17:05:04 +01:00
Arnaud Gourlay
6d0a48b2dc Deactivate sparse index pruning for negative query vectors (#3024) 2023-12-06 17:04:27 +01:00
Arnaud Gourlay
bc68840558 Improve sparse index pruning (#3011)
* Improve sparse index pruning

* add test for promote longest

* document and test possible underpruning
2023-12-06 17:04:01 +01:00
Arnaud Gourlay
c2dd19ae8c Benchmark sparse index build (#2984)
* Benchmark sparse index build

* add vector count assertion
2023-12-06 17:03:30 +01:00
Arnaud Gourlay
28fa2eedf8 Add fallback to plain search for sparse vectors (#2963) 2023-12-06 17:03:21 +01:00
Arnaud Gourlay
8f844b6635 Fix vector count in sparse index (#2992)
* Fix vector count in sparse index

* simplify naming

* rely on max id for count
2023-12-06 17:02:36 +01:00
Arnaud Gourlay
da991a505f Sparse vector validation (#2965)
* Sparse vector validation

* cheaper indices uniqueness validation and tests
2023-12-06 16:59:25 +01:00
Tim Visée
0d0f8459d7 Some micro optimizations in reserve/resize usage (#2967)
* In HNSW graph layer building, infer right capacity on collect

* Resize with empty vector, use constant function rather than clone

* Use resize_with in other places as well
2023-12-06 16:58:43 +01:00
Arnaud Gourlay
ee5fe63019 Benchmark sparse filter plain search (#2947)
* Sparse filter plain search

* code review
2023-12-06 16:57:21 +01:00
Arnaud Gourlay
dfb70c8621 Faster search in sparse inverted index with unstable sort (#2937) 2023-12-06 16:56:24 +01:00
Arnaud Gourlay
d5f98d8205 SparseVectorIndex implements VectorIndex (#2900)
* SparseVector implements VectorIndex

* dedicated telemetry

* conflict

* easy code reviews

* simplify tracking indexed points count

* move telemetry conversion to sparse file

* move max_result_count out of inverted index trait

* unify sparse vector fixtures

* simpler conversion

* add todo regarding OOM potential

* reuse check deleted from raw scorer with TODO

* change new to open to handle mmap index
2023-12-06 16:54:15 +01:00
Arnaud Gourlay
5c3c9012b7 Document and test underfetching with sparse vectors (#2908)
* Document and test underfetching with sparse vectors

* deduplicate test point ids
2023-12-06 16:52:43 +01:00
Arnaud Gourlay
cbf1811eaa Raw scorer for sparse vectors (#2915)
* Raw scorer for sparse vectors

* sparse vector storage trait

* use get sparse vector fn

* remove dedicated types

---------

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
2023-12-06 16:52:33 +01:00
Arnaud Gourlay
1bafacd6f8 Rename sparse vector values (#2892) 2023-12-06 16:50:43 +01:00
Arnaud Gourlay
cc69ec7fec Simple sparse vector storage (#2806)
* Simple sparse vector storage

* better todo
2023-12-06 16:49:31 +01:00
Arnaud Gourlay
935b759dbe Sparse vector index integration (#2835)
* inverted index as trait

* introduce sparse vector index integration

* sparse vectors search cancellation

* clippy

* expose helper for search sparse

* code review
2023-12-06 16:44:32 +01:00
Arnaud Gourlay
600835b668 Upsert operation on Sparse RAM index (#2826)
* Upsert operaiton on Sparse RAM index

* code review Ivan

* fix insertion of posting not at the end

* cleaner shortcut of upsert
2023-12-06 16:44:15 +01:00
Arnaud Gourlay
c00e496e47 Sparse index persistence (#2718)
* Sparse index persistence

* code review

* remove pub visibility
2023-10-06 12:44:01 +02:00
Arnaud Gourlay
0d4a373659 Move ScoredPointOffset into common (#2734) 2023-10-06 12:41:51 +02:00