* include document & image objects into grpc API
* introduce image and object to rest api
* minor refactoring
* rename Vector -> VectorInternal
* decompose vector data structures
* add schema
* fmt
* grpc docs
* fix conversion
* fix clippy
* fix another conversion
* rename VectorInput -> VectorInputInternal
* replace grpc TryFrom with async functions
* fmt
* replace rest TryFrom with async functions
* add image and object into query rest
* separate inference related conversions
* move json-related conversions into a separate file
* move vector-related transformations into a separate file
* move more vector related-conversions into dedicated module
* define mmap geo index
fix compilation
deleted flags
load new mmap
geo index tests
fix tests
fix build after rebase
add files list
* refactor get_stored_sub_regions output type
* review remanings
* mmap geo index api
* fix after rebase
* define mmap geo index
fix compilation
deleted flags
load new mmap
geo index tests
fix tests
fix build after rebase
add files list
* are you happy fmt
* refactor get_stored_sub_regions output type
* repr(C) for geohash
* Replace manual flattening with flat map iterator
* review remanings
* remove panics
* add files for snapshot
* rename Dynamic into InMemory
* are you happy fmt
* review renames
* Use copied rather than cloned
* minor review refactoring + comments
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
* tar_ext::BuilderExt: generalize and add drop workaround
- Generalize: instead of taking `std::fs::File`, take
`impl Write + Seek`.
- Add a workaround for drop: `tar::Builder<W>::drop` might panic if the
underlying writer panics when using in async context.
* Drop unnecessary write_fn usage
* tar_ext::BuilderExt: support both seekable and streaming variants
* tar_ext::BuilderExt: support both owned/borrowed variants
* map index: fix reachable code marked as unreachable
* plumber work to get mmap text index to interfaces
* test: add fixture for mmap text index, always create mmap segment
* various fixes
- ensure dir is created for mmap
- implement is_on_disk() for text index
- invert deleted condition for filter in mmap inverted index
* update grpc docs and openapi
* implement return of files
* review nit
* fix after rebase
---------
Co-authored-by: generall <andrey@vasnetsov.com>
This uses `TempDir` and comes with its caveats, e.g. the temporary segment
directory will not be deleted if the process exits. This should not be a
problem in practice, as all temporary segments get deleted when the shard is
loaded (see PR #2319).
Fixes#2978.
* Shrink to fit immutable map index after construction
* Simplify mutable map index loading a bit
* Preallocate value to points container because we know the exact size
* refactor inverted index into trait
* create immutable and mutable versions of text index
* migrate to full text index enum
* fix clippy
* review remarks
* packed geohash
* refactor
* are you happy codespell
* review remarks
* better iterator
* iterator with only one field
* encode db key as smolstr
* Update geohash_starts_with
---------
Co-authored-by: xzfc <xzfcpw@gmail.com>
* initial structures for reading mmap text index
* creation of text index mmap
* WIP create mmap inverted index
* use same file writing technique as in MmapHashmap
* Save points_to_tokens_count in two files
* clippy
* fill slice with iterator
* review fixes
* fix after review
* implement loading of MmapInvertedIndex
* WIP: prepare for converting ChunkReader into trait
* fmt
* WIP: prepare ChunkReader to be a trai
* WIP: replace ChunkReaderImpl with trait
* implement chunk reader for mmap view
* rollback ChunkReader trait and remove CompressedMmapPostingListView
* move ImmutableInvertedIndex and MutableInvertedIndex into separate files
* impl ops for MmapInvertedIndex
* make mmap hashmap value param generic (#5042)
* make mmap hashmap value param generic
* test + fixes
* Update lib/common/common/src/mmap_hashmap.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Update lib/common/common/src/mmap_hashmap.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* fixups
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* test and fix immutable to mmap conversion
* fix case 4 and refactor MmapBitSlice::create
* more tests
* fmt
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* discard unused token ids to slim down immutable text index
* clippy
* review: use custom types
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* move CompressedPostingList into a separate file
* decouple posting list from visitor using the reader abstraction
* refactor comressed_posting
* separate iterator into a new file + include diagram
* fmt
* decouple CompressedPostingList from PostingList
* decouple compression
* delete unused iter
* review: use correct names in diagrma
* review: fix comments
* review: code fixes
* fmt
* add tests
* hide compression logic
