* Add HardwareCounterCell and counting for sparse plain search
* Add measurement for indexed sparse vector
* add tests for calculations
* move SegmentQueryContent higher in the call stack
* move hardware counters inside the query context
* fix clippy
* Fix applying hardware measurements and add test for this
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* initial structures for reading mmap text index
* creation of text index mmap
* WIP create mmap inverted index
* use same file writing technique as in MmapHashmap
* Save points_to_tokens_count in two files
* clippy
* fill slice with iterator
* review fixes
* fix after review
* implement loading of MmapInvertedIndex
* WIP: prepare for converting ChunkReader into trait
* fmt
* WIP: prepare ChunkReader to be a trai
* WIP: replace ChunkReaderImpl with trait
* implement chunk reader for mmap view
* rollback ChunkReader trait and remove CompressedMmapPostingListView
* move ImmutableInvertedIndex and MutableInvertedIndex into separate files
* impl ops for MmapInvertedIndex
* make mmap hashmap value param generic (#5042)
* make mmap hashmap value param generic
* test + fixes
* Update lib/common/common/src/mmap_hashmap.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Update lib/common/common/src/mmap_hashmap.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* fixups
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* test and fix immutable to mmap conversion
* fix case 4 and refactor MmapBitSlice::create
* more tests
* fmt
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* refactor: introduce AdviceSetting for open_read_mmap/open_write_mmap
* perf: use Advice::Normal in a few places
* perf: use MADV_DONTNEED after reading
* perf: use POSIX_FADV_DONTNEED after reading
* Move build_index out of VectorIndex
* Build index in HNSWIndex::open()
* Introduce HnswIndexOpenArgs
* Proper deletion
* Improve tests
* HNSW::open(): add warn, comment and assert
* Revert to making up the config if it does not exist
* Rename InvertedIndex* -> InvertedIndexCompressed*
* Extract method VectorIndexEnum::fill_idf_statistics
* Extend VectorIndexEnum with new variants
* Introduce sparse::InvertedIndex::Version
* Replace SparseVectorIndexVersion -> InvertedIndex::Version
* Introduce sparse_vector_index::OpenArgs
* SparseVectorIndex::open: do not build index if directory is empty
Otherwise it would build the index twice since `SegmentBuilder::build()`
calls `::open()`, then `::build_index()`. This restores the old (<=v1.9)
behavior.
* Renames
* include old vector into update function and clean posting lists accordingly
* add integration test
* fix counter
* also remove old vector if the insertion is empty
* clippy
* borrow once
* fix max_next_weight correcton on delete + test
* vector index responsible for updating vector storage
* review fixes
* add debug assert
* introduce QueryContext, which accumulates runtime info needed for executing search
* fmt
* propagate query context into segment internals
* [WIP] prepare idf stats for search query context
* Split SparseVector and RemmapedSparseVector to guarantee we will not mix them up on the type level
* implement filling of the query context with IDF statistics
* implement re-weighting of the sparse query with idf
* fmt
* update idf param only if explicitly specified (more consistent with diff param update
* replace idf bool with modifier enum, improve further extensibility
* test and fixes
* Update lib/collection/src/operations/types.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* review fixes
* fmt
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Pool larger memory batches for sparse vector search
* rework naming
* refine naming
* address great comments
* bump ADVANCE_BATCH_SIZE to use power of 2
* 16k performs bad - trying 8k
* 10k offers the best tradeoff
* Optimize sparse vector search with batched scoring
* replace roaring bitmap by id generation
* compute intersections in Vec instead of Map
* cargo update ahash@0.8.7 --precise 0.8.5
* let rustc handle the pattern match optimization
* profiling done
* remove Option and filter out zero scores
* restore integration test assertion
* filter based on min score to avoid work
* decrease batch size and cleanup posting creation
* remove outdated tests
* remove dead test - I know where to find it if I need it
* sparse vectors with large indices support
* fix unit tests
* add tracker file to snapshot
* fix unit tests
* are you happy clippy
* fix config loading
* test with max value
* fix python tests
* fix consensus tests
* review remarks; acc test for inverted index
* internal sparse vector segment configs
update openapi
fix build
provide correct index type to fixtures
internal sparse vector segment configs
update openapi
fix build
load ram sparse index from file
fix spelling
add test
spelling error
use appendable flag to decide load or not
fix conflict
dont flush mmap when file is empty
* fix build
* set list of files for snapshot
* fix snapshots test
* fix codespell
* simple save-load unit test
* refactor InvertedIndex::files
* fix unit tests
* add debug_assert
* test all persistence loading cases
* sparse vectors query scorer
plain search test without internals
fix sparse_vector_index_plain_search
fix async scorer build
fix unit tests
are you happy fmt
sparse index with full query support
fix benches, use exact flag to force plain search
add alias and comment to exact flag
fix tests
recommendations TODOs fix
add simple test
add todo
refactor index
test vs comparison dense and sparse discovery
are you happy fmt
propogate error from avg negatives
reuse filtered points list
rollback api changes
better discovery test
are you happy fmt
rollback openapi
fix build
fix tests
review remark recommendations
review remarks
review remarks
codespell
integrate plain search
Update lib/collection/src/recommendations.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Update lib/collection/src/recommendations.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Update lib/collection/src/recommendations.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
fix comment of plain search
Update lib/collection/src/recommendations.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
fix tests
* simplify search query function
