Commit Graph

14 Commits

Author SHA1 Message Date
Arnaud Gourlay
bc5d0976a1 Sparse vector validation (#2965)
* Sparse vector validation

* cheaper indices uniqueness validation and tests
2023-11-10 16:42:10 +01:00
Tim Visée
9394fc1757 Some micro optimizations in reserve/resize usage (#2967)
* In HNSW graph layer building, infer right capacity on collect

* Resize with empty vector, use constant function rather than clone

* Use resize_with in other places as well
2023-11-10 09:42:14 +01:00
Arnaud Gourlay
1ae1f483d4 Benchmark sparse filter plain search (#2947)
* Sparse filter plain search

* code review
2023-11-09 09:57:43 +01:00
Arnaud Gourlay
186c906d6d Faster search in sparse inverted index with unstable sort (#2937) 2023-11-07 14:51:21 +01:00
Arnaud Gourlay
ec2dc36719 SparseVectorIndex implements VectorIndex (#2900)
* SparseVector implements VectorIndex

* dedicated telemetry

* conflict

* easy code reviews

* simplify tracking indexed points count

* move telemetry conversion to sparse file

* move max_result_count out of inverted index trait

* unify sparse vector fixtures

* simpler conversion

* add todo regarding OOM potential

* reuse check deleted from raw scorer with TODO

* change new to open to handle mmap index
2023-11-03 16:23:05 +01:00
Arnaud Gourlay
0fa6688764 Document and test underfetching with sparse vectors (#2908)
* Document and test underfetching with sparse vectors

* deduplicate test point ids
2023-11-02 15:30:43 +01:00
Arnaud Gourlay
8a05462c6d Raw scorer for sparse vectors (#2915)
* Raw scorer for sparse vectors

* sparse vector storage trait

* use get sparse vector fn

* remove dedicated types

---------

Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
2023-11-02 14:59:18 +01:00
Arnaud Gourlay
460832cc70 Rename sparse vector values (#2892) 2023-10-31 11:52:45 +01:00
Arnaud Gourlay
97ffee476e Simple sparse vector storage (#2806)
* Simple sparse vector storage

* better todo
2023-10-30 11:27:36 +01:00
Arnaud Gourlay
ecaff66f4c Sparse vector index integration (#2835)
* inverted index as trait

* introduce sparse vector index integration

* sparse vectors search cancellation

* clippy

* expose helper for search sparse

* code review
2023-10-18 12:03:04 +02:00
Arnaud Gourlay
4df28d250d Upsert operation on Sparse RAM index (#2826)
* Upsert operaiton on Sparse RAM index

* code review Ivan

* fix insertion of posting not at the end

* cleaner shortcut of upsert
2023-10-17 12:10:27 +02:00
Arnaud Gourlay
fc0b77636a Sparse index persistence (#2718)
* Sparse index persistence

* code review

* remove pub visibility
2023-10-04 14:39:52 +02:00
Arnaud Gourlay
5bb80fbc77 Move ScoredPointOffset into common (#2734) 2023-09-28 12:11:29 +02:00
Arnaud Gourlay
5fe82cb6d7 Sparse vector index (#2318)
* Sparse vector index

* better name

* fill properly postings

* conflict

* easy reviews

* use unstable sort

* use optional instead of bool flag

* clearer score type and future direction

* NonZeroUsize length for FixedLengthPriorityQueue

* rename id to record_id

* cleanup dimension inverted index

* standard Iterator

* code review

* Shift comment

* early return if no matching posting lists

* sprinkle iterators

* privatize the means of production

---------

Co-authored-by: timvisee <tim@visee.me>
2023-09-15 16:30:11 +02:00