* map index: fix reachable code marked as unreachable
* plumber work to get mmap text index to interfaces
* test: add fixture for mmap text index, always create mmap segment
* various fixes
- ensure dir is created for mmap
- implement is_on_disk() for text index
- invert deleted condition for filter in mmap inverted index
* update grpc docs and openapi
* implement return of files
* review nit
* fix after rebase
---------
Co-authored-by: generall <andrey@vasnetsov.com>
This uses `TempDir` and comes with its caveats, e.g. the temporary segment
directory will not be deleted if the process exits. This should not be a
problem in practice, as all temporary segments get deleted when the shard is
loaded (see PR #2319).
Fixes#2978.
* Shrink to fit immutable map index after construction
* Simplify mutable map index loading a bit
* Preallocate value to points container because we know the exact size
* refactor inverted index into trait
* create immutable and mutable versions of text index
* migrate to full text index enum
* fix clippy
* review remarks
* packed geohash
* refactor
* are you happy codespell
* review remarks
* better iterator
* iterator with only one field
* encode db key as smolstr
* Update geohash_starts_with
---------
Co-authored-by: xzfc <xzfcpw@gmail.com>
* initial structures for reading mmap text index
* creation of text index mmap
* WIP create mmap inverted index
* use same file writing technique as in MmapHashmap
* Save points_to_tokens_count in two files
* clippy
* fill slice with iterator
* review fixes
* fix after review
* implement loading of MmapInvertedIndex
* WIP: prepare for converting ChunkReader into trait
* fmt
* WIP: prepare ChunkReader to be a trai
* WIP: replace ChunkReaderImpl with trait
* implement chunk reader for mmap view
* rollback ChunkReader trait and remove CompressedMmapPostingListView
* move ImmutableInvertedIndex and MutableInvertedIndex into separate files
* impl ops for MmapInvertedIndex
* make mmap hashmap value param generic (#5042)
* make mmap hashmap value param generic
* test + fixes
* Update lib/common/common/src/mmap_hashmap.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Update lib/common/common/src/mmap_hashmap.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* fixups
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* test and fix immutable to mmap conversion
* fix case 4 and refactor MmapBitSlice::create
* more tests
* fmt
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* discard unused token ids to slim down immutable text index
* clippy
* review: use custom types
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* move CompressedPostingList into a separate file
* decouple posting list from visitor using the reader abstraction
* refactor comressed_posting
* separate iterator into a new file + include diagram
* fmt
* decouple CompressedPostingList from PostingList
* decouple compression
* delete unused iter
* review: use correct names in diagrma
* review: fix comments
* review: code fixes
* fmt
* add tests
* hide compression logic
* move FacetIndex into facet_index.rs
* add support for integer facets
* add support for uuid facets
* use separate internal structure
* rename FacetValue::Keyword into FacetValue::String in REST
* fix after rebase
* rename keyword -> string for internal data structures
* implement MatchAny and expect for filter on UUID mmap index
* implement MatchAny and expect for cardinality estimation on UUID mmap index
* refactor PayloadFieldIndex filter for handling incorrect empty query parsing case
* Don't use ID but current point offset when optimizing dense mmap storage
This change now matches the implementation in all other storage types.
* Remove now obsolete point offsets in update_from point iterator
* Fix test
* rename to FacetRequestInternal
* add rest endpoint
* fix correctness by fetching the whole list of values
* fix mmap map index variant
Also removes test for sorted output, for now
* add ytt spec
* fix clippy
* use hashmap inside of local shard
* rename operation to `facet`, add access test
* whitelist endpoint
* change api
* make limit optional
* refactor: introduce AdviceSetting for open_read_mmap/open_write_mmap
* perf: use Advice::Normal in a few places
* perf: use MADV_DONTNEED after reading
* perf: use POSIX_FADV_DONTNEED after reading
* make mmap_type independent from segment structures
* make bitvec and thiserror workspace dependencies
* move mmap_type into common/memory subcrate
* fmt
