* Add congruence test for full text payload index
* Report what index variants we're testing
* Use string references
* deterministic keywords
* linting
* Use truncate instead of drain
* Don't check cardinality on deletion, indices behave differently
* Include index type in iterator
* Fix test compilation
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Fix unecessary unsafe in Clippy 1.87
* bump Rust version for intrinsics safe API
* ignore until we can bump the MSRV with cargo-chef
* remove unecessary change
* mark clearly as TODO
* Test for reproducing force delete peer bug
* drop seed port
* use larger transfers to ensure killing before transfer is completed
* simplify test cases
* kill and wait for same commit before force delete
* Don't kill to avoid breaking consesnsus before force delete peer
* Abort transfer on force peer delete (#6507)
* Abort transfer on force peer delete
* ToC changes to abort internally
* Abort shard transfer using consensus not just locally
* fmt
* fix fmt
* Abort transfers internally when shard is being removed
* remove unused imports
* take max_point_offset into consideration
* add null index even if index selector is for rocksdb
* fmt
* better cardinality estimation, check deleted inside filtered iter
* fix count indexed points, add primary conditions to estimator
* use len for count_indexed_points, adjust test
* remove deleted check from filtered iter
* cleaner removal
* Make gridstore Advise::Random and add function for sequential read
* Reduce duplicate code in gridstore
* use sequential implementation in iterator
* actually use the sequential read
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Add storage enum to geo index
* Construct immutable geo index from mmap geo index
* Load immutable geo index from mmap storage if on-disk is false
* When immutable geo index is loaded into memory, clear mmap cache
* Adjust immutable geo index from mmap loading, consider point deletions
* Rename get_db_wrapper to db_wrapper
* Add immutable mmap geo index test, currently still failing
* Properly load geo index
* Add congruence test
* Also test with deletions
* Fix clipping warnings
* Add storage enum to full text index
* Add compressed mmap posting list iterator
* Don't check second condition and return early if possible
* Use helper function for creating empty iterator
* Construct immutable inverted index from mmap inverted index
* Construct immutable text index from mmap text index
* Load immutable text index from mmap storage if on-disk is false
* Match implementation for mutable to immutable inverted index, drop empty
* Point tokens should be none if empty
* Fix chunk reader iterator not buffering remaining postings correctly
* Add immutable mmap full text index to congruence test
* Debug assert deleted points have no tokens
* Add another test
* Fix point deletion regression in immutable full text index
* Use consistent function naming
* Restructure full text index opening and loading, make it consistent
* When immutable full text index is loaded into memory, clear mmap cache
* Remove init from immutable text index
* Prevent potential panic
* avoid panic
* Fix custom iterator size reporting, add test for it
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Refactor out LinksContainer
This commit extracts Vec<PointOffsetType> and heuristic related
functions from graph_layers_builder.rs into a new file.
No significant changes in logic are made.
* Optimize LinksContainer
* Add illustration
* Fixups
* Add more comments and rename `score` to `cached_score`
* Add RAM wrapper to mmap numeric index
* Add RAM wrapper to mmap map index
* Fix test compilation
* Merge RAM wrappers into existing immutable indices
Do this to minimize code duplication since the majority of the code is
exactly the same.
* Minor tweaks
* Remove unnecessary pub visibility, remove unused assertion
* Test immutable numeric index on mmap
* In numeric index test, remove some points
* Add map index test
* Restructure index opening and loading, make it consistent
* don't pre-collect intermediate mapping
* review nits
* Make init unreachable for immutable and mmap, clear must consume index
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Don't batch query if there's more segments than CPUs, limit threads
* Don't batch query if there's more segments than search threads
* Link to pull request
* Add note that we only consider local segments and global search threads
* Temporarily run fewer tests in CI to reproduce flakiness
* disable other integration test
* log collection topology if request fails
* Run in CI until failure
* update tests
* Catch failures and log
* improve logging for search result comparison flakiness
* fix test
* Simplify test
* Use scroll instead of search and simplify existing code
* Trigger CI
* new util function
* debug again
* print collection topology on mismatch
* wait for deletion of flag after recovery is done
* use wait_for
* Revert integration-tests.yml
