* Don't use ID but current point offset when optimizing dense mmap storage
This change now matches the implementation in all other storage types.
* Remove now obsolete point offsets in update_from point iterator
* Fix test
* rename to FacetRequestInternal
* add rest endpoint
* fix correctness by fetching the whole list of values
* fix mmap map index variant
Also removes test for sorted output, for now
* add ytt spec
* fix clippy
* use hashmap inside of local shard
* rename operation to `facet`, add access test
* whitelist endpoint
* change api
* make limit optional
* refactor: introduce AdviceSetting for open_read_mmap/open_write_mmap
* perf: use Advice::Normal in a few places
* perf: use MADV_DONTNEED after reading
* perf: use POSIX_FADV_DONTNEED after reading
* make mmap_type independent from segment structures
* make bitvec and thiserror workspace dependencies
* move mmap_type into common/memory subcrate
* fmt
* Migrate uuid index to mmap
* add uuid numeric index
* select correct index + tenants
* Only serialize if necessary
* select correct index
* reset Cargo.toml
* review: use only map index for uuid for now
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* add force_ram parameter to chuncked mmap vector storage
* enable mlocked mmap vector storage on unix by default
* regen openapi
* add mlock on creation of chunck
* minor unrelated renaming
* rollback changes in LockedChunkedMmap
* fmt
* make AppendableMmapDenseVectorStorage generic of storage type
* make AppendableMmapMultiDenseVectorStorage generic of storage type
* implement initialization of InRamChunkedMmap
* implement MultiDenseAppendableInRam and variations
* enable InRamChunkedMmap for multivectors
* use same CHUNK_SIZE for mmap and regular chuncked vectors
* enable InRamChunkedMmap by default
* fix tests
* rollback usage of InRamChunkedMmap by default
* review changes
* add assertion on chunk_capacity [skip-ci]
* refactor: let SegmentBuilder::update take unlocked segments
* style: split long lines
* refactor: introduce TestSegments
* test: add tests for mmap indices
* Extend FieldIndexBuilder with mmap indices
* Introduce PayloadFieldIndex::files()
* Create and delete index dirs
* Update index_selector and index_builder_selector
* Distance matrix API internals
* minor code review
* use new Query sample instead of manual samplig + retrieve
* do not crash if the point does not contain the specific vector
* Use iterative approach in the absence of HasVector condition
* add tests
* define mmap map index
add point to values
add mmap hash map
are you happy fmt
use new mmap hashmap methods
build index
saturating_sub
are you happy clippy
fix tests build
* integrate facets for mmap index
* mmap tests
* fix ci
* review remarks
* review remarks
* faceting in segment
* Add segment integration test
* nits
* count from filtered stream, not value->points map directly
* drop AtomicRef from fn signature
* count only unique values per point
* use entry in hashmap
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* mmap numeric index
dynamic numeric index
remove pub from fields
add mmap enum
remove vec<u8> as number
get_value without self
mmap save load
trait for conversion from P into T
save-load points-to-values
fix unit tests
are you happy clippy
fix ci
apply changes from #4743
NumericIndex: add new_mmap() method
remove mmapvalue trait from immutable
small refactor
implement from_referenced for str
are you happy clippy
deleted flags by idx instead of pairs
* fix build after rebase
* review remarks
* tests macos
* increase deleted count while deletion
* add UuidIndex
* fix clippy
* rebase to dev
* update api docs
* don't use wrapper type for Uuid index
* rebase to `dev`
* remove existence checking
* rename UuidPayloadKeyType => UuidIntType
* apply review changes
* rebase to dev
* post-rebase fixes
* Improve estimation
* Improve naming
* Apply suggestions from code review
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* use u128 in histogram and improve uuid sorting
* Also allow defragmentation for completely random Uuids
---------
Co-authored-by: generall <andrey@vasnetsov.com>
