* implement mmap sparse vector storage
* add to VectorStorageEnum
* clippy
* add tests, fix both simple and mmap storages
* smol correction on total_vector_count
* add sparse storage type to config
* fix reading config without storage type
* generate openapi
* use blob_store by path
* hidden setting to enable new storage
* validate existing path in `BlobStore::open()`
* use new dir for each sparse vector name
* fix and rename `max_point_offset`
Plus some extra refactors
* add storage compat test, to always check both storages work
* fix opening of storage + other misc fixes
* FIX!!!
`Unset` operations in the Tracker weren't updating the
`next_pointer_id`. So, when reopening the storage, those points wouldn't
get marked as deleted in the bitslice, thus creating the illusion that
they should exist, when they did not.
* refactor naming from `iter_*` to `for_each_*`
* fix checking for BlobStore existance
* fix typo
* fix error message
* better docs for open_or_create
* fix after rebase
* Add HardwareCounterCell and counting for sparse plain search
* Add measurement for indexed sparse vector
* add tests for calculations
* move SegmentQueryContent higher in the call stack
* move hardware counters inside the query context
* fix clippy
* Fix applying hardware measurements and add test for this
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* initial structures for reading mmap text index
* creation of text index mmap
* WIP create mmap inverted index
* use same file writing technique as in MmapHashmap
* Save points_to_tokens_count in two files
* clippy
* fill slice with iterator
* review fixes
* fix after review
* implement loading of MmapInvertedIndex
* WIP: prepare for converting ChunkReader into trait
* fmt
* WIP: prepare ChunkReader to be a trai
* WIP: replace ChunkReaderImpl with trait
* implement chunk reader for mmap view
* rollback ChunkReader trait and remove CompressedMmapPostingListView
* move ImmutableInvertedIndex and MutableInvertedIndex into separate files
* impl ops for MmapInvertedIndex
* make mmap hashmap value param generic (#5042)
* make mmap hashmap value param generic
* test + fixes
* Update lib/common/common/src/mmap_hashmap.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Update lib/common/common/src/mmap_hashmap.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* fixups
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* test and fix immutable to mmap conversion
* fix case 4 and refactor MmapBitSlice::create
* more tests
* fmt
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* refactor: introduce AdviceSetting for open_read_mmap/open_write_mmap
* perf: use Advice::Normal in a few places
* perf: use MADV_DONTNEED after reading
* perf: use POSIX_FADV_DONTNEED after reading
* Move build_index out of VectorIndex
* Build index in HNSWIndex::open()
* Introduce HnswIndexOpenArgs
* Proper deletion
* Improve tests
* HNSW::open(): add warn, comment and assert
* Revert to making up the config if it does not exist
* Rename InvertedIndex* -> InvertedIndexCompressed*
* Extract method VectorIndexEnum::fill_idf_statistics
* Extend VectorIndexEnum with new variants
* Introduce sparse::InvertedIndex::Version
* Replace SparseVectorIndexVersion -> InvertedIndex::Version
* Introduce sparse_vector_index::OpenArgs
* SparseVectorIndex::open: do not build index if directory is empty
Otherwise it would build the index twice since `SegmentBuilder::build()`
calls `::open()`, then `::build_index()`. This restores the old (<=v1.9)
behavior.
* Renames
* include old vector into update function and clean posting lists accordingly
* add integration test
* fix counter
* also remove old vector if the insertion is empty
* clippy
* borrow once
* fix max_next_weight correcton on delete + test
* vector index responsible for updating vector storage
* review fixes
* add debug assert
* add `query` to shard trait
* add missing conversions for query
* update grpc docs
* Query response has intermediate results
* add ShardQueryResponse description
* move pub use to the top, keep only one way of reaching reexports
* introduce QueryContext, which accumulates runtime info needed for executing search
* fmt
* propagate query context into segment internals
* [WIP] prepare idf stats for search query context
* Split SparseVector and RemmapedSparseVector to guarantee we will not mix them up on the type level
* implement filling of the query context with IDF statistics
* implement re-weighting of the sparse query with idf
* fmt
* update idf param only if explicitly specified (more consistent with diff param update
* replace idf bool with modifier enum, improve further extensibility
* test and fixes
* Update lib/collection/src/operations/types.rs
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* review fixes
* fmt
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Pool larger memory batches for sparse vector search
* rework naming
* refine naming
* address great comments
* bump ADVANCE_BATCH_SIZE to use power of 2
* 16k performs bad - trying 8k
* 10k offers the best tradeoff
* Optimize sparse vector search with batched scoring
* replace roaring bitmap by id generation
* compute intersections in Vec instead of Map
* cargo update ahash@0.8.7 --precise 0.8.5
* let rustc handle the pattern match optimization
* profiling done
* remove Option and filter out zero scores
* restore integration test assertion
* filter based on min score to avoid work
* decrease batch size and cleanup posting creation
* remove outdated tests
* remove dead test - I know where to find it if I need it
