* Strict mode config: Max collection size
* api specs
* Add tests + set/update payload check
* Improve function names and add comments
* rename config to separate vectors and payload
* fix tests
* Adjust configs docs
* add benchmark
* improve performance by caching shard info
* add bench for size_info() and fix tests
* Also limit the batch-size for vector updates (#5508)
* Also limit the batch-size for vector updates
* clippy
* add lost commit
* Load cache on collection initialization
* add unit type to parameter name
* fix renaming in test
* clearer error message
* fix test
* review remarks
* remove unused function for now
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* implement mmap sparse vector storage
* add to VectorStorageEnum
* clippy
* add tests, fix both simple and mmap storages
* smol correction on total_vector_count
* add sparse storage type to config
* fix reading config without storage type
* generate openapi
* use blob_store by path
* hidden setting to enable new storage
* validate existing path in `BlobStore::open()`
* use new dir for each sparse vector name
* fix and rename `max_point_offset`
Plus some extra refactors
* add storage compat test, to always check both storages work
* fix opening of storage + other misc fixes
* FIX!!!
`Unset` operations in the Tracker weren't updating the
`next_pointer_id`. So, when reopening the storage, those points wouldn't
get marked as deleted in the bitslice, thus creating the illusion that
they should exist, when they did not.
* refactor naming from `iter_*` to `for_each_*`
* fix checking for BlobStore existance
* fix typo
* fix error message
* better docs for open_or_create
* fix after rebase
* copy-paste blob_storage
* get rid of copied utils
* fix clippy
* fmt
* audit blob_store manifest
* move `Payload` into `fixtures.rs`
* fix clippy
* put csv in google storage
* ignore hm test
* fix codespell
* improve files test
* test different block sizes
* remove unused deps
* replace StdRng with SmallRng for faster tests
* Add UUID to collection...
...and recreate collection, when applying Raft snapshot, if UUID of collection is different
* fixup! Add UUID to collection...
Remove UUID field from gRPC and exclude it from OpenAPI spec 🤡
* fixup! fixup! Add UUID to collection...
Always generate collection UUID 🤦♀️
* Raft snapshot recreate collection no expose UUID (#5452)
* separate colleciton config structure from API
* fmt
* Update lib/collection/src/operations/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
* make the default `on_disk_payload` match the default config value
* remove duplicate impl, update docstrings
* change default `PayloadStorageType` too
* gen openapi
* add HwMeasurementAcc and API helpers+types
* add config to API to toggle hardware reporting in API responses
* add Default impl for HwMeasurementAcc
* comments
* review remarks
* include usage into API schema
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Track disk usage for each of the vectors and segments
* Dont extend vector_data field
* Remove comment and improve var name
* Introduce vectors_size_bytes and rename available_size_in_bytes
* Introduce vectors_size_bytes and rename available_size_in_bytes
* fix linter
* Avoid converting to u64
* calculate vectors_size_bytes for proxy segments
* Fix OpenAPI spec
* fix typo
* Restructure WAL delta logic, similar to what we had before the patch
* Tweak comments to elaborate on data race fix
* Refactor recovered function, not snapshot specific anymore
* Remove partial snapshot replica set state from schema
* include document & image objects into grpc API
* introduce image and object to rest api
* minor refactoring
* rename Vector -> VectorInternal
* decompose vector data structures
* add schema
* fmt
* grpc docs
* fix conversion
* fix clippy
* fix another conversion
* rename VectorInput -> VectorInputInternal
* replace grpc TryFrom with async functions
* fmt
* replace rest TryFrom with async functions
* add image and object into query rest
* separate inference related conversions
* move json-related conversions into a separate file
* move vector-related transformations into a separate file
* move more vector related-conversions into dedicated module
* Report CPU endianness in telemetry output
* Make endianness optional, don't report if anonymizing telemetry
* Update OpenAPI definition
* Switch back to single compile time endianness check
* Update OpenAPI specification
* Remove cpu-endian dependency
* De-anonymize CPU endianness
---------
Co-authored-by: xzfc <xzfcpw@gmail.com>
* define mmap geo index
fix compilation
deleted flags
load new mmap
geo index tests
fix tests
fix build after rebase
add files list
* refactor get_stored_sub_regions output type
* review remanings
* mmap geo index api
* fix after rebase
* map index: fix reachable code marked as unreachable
* plumber work to get mmap text index to interfaces
* test: add fixture for mmap text index, always create mmap segment
* various fixes
- ensure dir is created for mmap
- implement is_on_disk() for text index
- invert deleted condition for filter in mmap inverted index
* update grpc docs and openapi
* implement return of files
* review nit
* fix after rebase
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Track number of points optimized and expose in telemetry
* refactor
* openapi specs
* remove dbg
* Return num points optimized from optimize() func
* fmt
* fix
* fix type in tests
* Store total points indexed on shard level instead of optimization level
* fmt
* fix test
* trigger ci
* fix openapi schema
* review fixes
* fmt
* improvements and fix test
* review fixes
* use const for indexing optimizer name
* fmt
* return segment id from optimize() func
* review fixes
* fix
* fix
* fik
* minor var name improvement
* Use Option<usize> to return segment id
* Use segment ID type rather than ambiguous usize
* fix test
* avoid intermediate check
* review fixes
* Rename total_indexed_points to total_optimized_points
* Update openapi schema
* optimize() should return number of points in new segment instead of segment id
* add else condition
* take read lock
* fmt
* remove flaky assert
* Count points on new segment without locking
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
* Expose cluster telemetry in metadata
* Only expose metadata in telemetry for details level 1 and up
* Update OpenAPI specification
* move metadata into cluster telemetry
* regen openapi
* use Option instead of raw hashmap
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Expose shard status in telemetry API
* fmt
* Drop segment lock before using async fetching shard status
* Use Self in From implementation for ShardStatus to CollectionStatus mapping
* Improve comments
* Remove redundant clone
* Update openapi specs
* Isolate function for shard status
* Fix compiler error
* Avoid adding dedicated function for shard status
* review fixes
* define missing var
* lint err
* comment
* comment
* refactor
* improve comments
* Improve comments
* fix lint and update openapi specs
* improve comment
* gRPC API for Distance Matrix
* generate API docs
* apply correct limit to sample
* add JWT tests
* pluralitiy
* fix COO naming
* drops COO as it trips codespell and is redundant
* clarify docs
* move FacetIndex into facet_index.rs
* add support for integer facets
* add support for uuid facets
* use separate internal structure
* rename FacetValue::Keyword into FacetValue::String in REST
* fix after rebase
* rename to FacetRequestInternal
* add rest endpoint
* fix correctness by fetching the whole list of values
* fix mmap map index variant
Also removes test for sorted output, for now
* add ytt spec
* fix clippy
* use hashmap inside of local shard
* rename operation to `facet`, add access test
* whitelist endpoint
* change api
* make limit optional
