* include document & image objects into grpc API
* introduce image and object to rest api
* minor refactoring
* rename Vector -> VectorInternal
* decompose vector data structures
* add schema
* fmt
* grpc docs
* fix conversion
* fix clippy
* fix another conversion
* rename VectorInput -> VectorInputInternal
* replace grpc TryFrom with async functions
* fmt
* replace rest TryFrom with async functions
* add image and object into query rest
* separate inference related conversions
* move json-related conversions into a separate file
* move vector-related transformations into a separate file
* move more vector related-conversions into dedicated module
* Report CPU endianness in telemetry output
* Make endianness optional, don't report if anonymizing telemetry
* Update OpenAPI definition
* Switch back to single compile time endianness check
* Update OpenAPI specification
* Remove cpu-endian dependency
* De-anonymize CPU endianness
---------
Co-authored-by: xzfc <xzfcpw@gmail.com>
* define mmap geo index
fix compilation
deleted flags
load new mmap
geo index tests
fix tests
fix build after rebase
add files list
* refactor get_stored_sub_regions output type
* review remanings
* mmap geo index api
* fix after rebase
* map index: fix reachable code marked as unreachable
* plumber work to get mmap text index to interfaces
* test: add fixture for mmap text index, always create mmap segment
* various fixes
- ensure dir is created for mmap
- implement is_on_disk() for text index
- invert deleted condition for filter in mmap inverted index
* update grpc docs and openapi
* implement return of files
* review nit
* fix after rebase
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Track number of points optimized and expose in telemetry
* refactor
* openapi specs
* remove dbg
* Return num points optimized from optimize() func
* fmt
* fix
* fix type in tests
* Store total points indexed on shard level instead of optimization level
* fmt
* fix test
* trigger ci
* fix openapi schema
* review fixes
* fmt
* improvements and fix test
* review fixes
* use const for indexing optimizer name
* fmt
* return segment id from optimize() func
* review fixes
* fix
* fix
* fik
* minor var name improvement
* Use Option<usize> to return segment id
* Use segment ID type rather than ambiguous usize
* fix test
* avoid intermediate check
* review fixes
* Rename total_indexed_points to total_optimized_points
* Update openapi schema
* optimize() should return number of points in new segment instead of segment id
* add else condition
* take read lock
* fmt
* remove flaky assert
* Count points on new segment without locking
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
* Expose cluster telemetry in metadata
* Only expose metadata in telemetry for details level 1 and up
* Update OpenAPI specification
* move metadata into cluster telemetry
* regen openapi
* use Option instead of raw hashmap
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Expose shard status in telemetry API
* fmt
* Drop segment lock before using async fetching shard status
* Use Self in From implementation for ShardStatus to CollectionStatus mapping
* Improve comments
* Remove redundant clone
* Update openapi specs
* Isolate function for shard status
* Fix compiler error
* Avoid adding dedicated function for shard status
* review fixes
* define missing var
* lint err
* comment
* comment
* refactor
* improve comments
* Improve comments
* fix lint and update openapi specs
* improve comment
* gRPC API for Distance Matrix
* generate API docs
* apply correct limit to sample
* add JWT tests
* pluralitiy
* fix COO naming
* drops COO as it trips codespell and is redundant
* clarify docs
* move FacetIndex into facet_index.rs
* add support for integer facets
* add support for uuid facets
* use separate internal structure
* rename FacetValue::Keyword into FacetValue::String in REST
* fix after rebase
* rename to FacetRequestInternal
* add rest endpoint
* fix correctness by fetching the whole list of values
* fix mmap map index variant
Also removes test for sorted output, for now
* add ytt spec
* fix clippy
* use hashmap inside of local shard
* rename operation to `facet`, add access test
* whitelist endpoint
* change api
* make limit optional
* Migrate uuid index to mmap
* add uuid numeric index
* select correct index + tenants
* Only serialize if necessary
* select correct index
* reset Cargo.toml
* review: use only map index for uuid for now
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* add force_ram parameter to chuncked mmap vector storage
* enable mlocked mmap vector storage on unix by default
* regen openapi
* add mlock on creation of chunck
* minor unrelated renaming
* rollback changes in LockedChunkedMmap
* fmt
* make AppendableMmapDenseVectorStorage generic of storage type
* make AppendableMmapMultiDenseVectorStorage generic of storage type
* implement initialization of InRamChunkedMmap
* implement MultiDenseAppendableInRam and variations
* enable InRamChunkedMmap for multivectors
* use same CHUNK_SIZE for mmap and regular chuncked vectors
* enable InRamChunkedMmap by default
* fix tests
* rollback usage of InRamChunkedMmap by default
* review changes
* add assertion on chunk_capacity [skip-ci]
* add UuidIndex
* fix clippy
* rebase to dev
* update api docs
* don't use wrapper type for Uuid index
* rebase to `dev`
* remove existence checking
* rename UuidPayloadKeyType => UuidIntType
* apply review changes
* rebase to dev
* post-rebase fixes
* Improve estimation
* Improve naming
* Apply suggestions from code review
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* use u128 in histogram and improve uuid sorting
* Also allow defragmentation for completely random Uuids
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Refactor: Convert app and cluster info metrics from counters to gauges
- Change app_info, app_status_recovery_mode, and cluster_enabled metrics from COUNTER to GAUGE
- Update metric values to use boolean as f64 for consistency and clarity
- Improve metric type accuracy for better representation of state information
* fix: Update docs regarding metrics type
* openapi: Update app_info metric type
* metrics: Update test to check for the right metric type
* example for CollectionsResponse
* points_example for ScrollResult
* example for SnapshotDescription
* fix
* example for AliasDescription
* fix: Style nit: add newline
* fix time and id
* fix order_value
* upd openapi
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* expose random sampling query
* fix random query rescoring
* only ignore offset when there are no prefetches
* fix offset and with_payload/with_vector
* gen grpc docs
* make tests less flaky
* gen openapi schema
* Remove suffixed spaces
---------
Co-authored-by: timvisee <tim@visee.me>
* sorting
* migrate tests and move logic into SegmentBuilder
* add test and improve implementation
* improve code
* review
* code review improvements
* add index building to test
* Do not clone ranges
* Resolve clippy warnings due to recent PR on dev
* review suggestions
* Defragmentation in api (#4684)
* add tenant config to api
* deduplicate used defragmentation keys
* rename is_tenant to is_primary
* use all values to defrag key
* rename is_primary -> is_tenant
* update schema
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: timvisee <tim@visee.me>
* Revert "Merge pull request #4560"
This reverts commit 5b0536df39.
* point id examples
* payload example
* VectorStruct example
* fmt
* ShardKey example
* OrderValue example
* fix
* example for SeqNumberType, ScoreType
* universal-query: Grouping REST API
* add API docs
* bump API count
* add jwt validation test
* fix API
* better test
* stay on CoreSearchRequest where possible
* use existing scoring helper
* push broken test to illustrate issue
* track best score per point_id to enable sort on payload
* add example test for query discover with groups
* track ScoredPoint instead of PointId
* add default values for smoother ux
* schemars example for vectors in scored point
* schemars example for payload in scored point
* Using json! in schemars example for payload in scored point
* fix
* schemars example for shard_key in scored point
* schemars example for order_by in scored point
* schemars example for id, version, score in scored point
* schemars example for id, payload, vector, shard_key, order_value in …
* point id examples
* remove id_example in schema.rs
* schemars example for vector, named_sparse_vector, named_vector
* schemars example for order_by
* schemars example for OrderByInterface
* schemars example for Fusion
* schemars example for MultiDenseVector,DenseVector in VectorInput
* schemars example for QueryRequest
* schemars example for QueryRequestBatch
* schemars example for QueryResponse
* schemars example for FusionQuery, OrderByQuery, ContextQuery, Discove…
* example for WithLookup, ContextPair, ContextInput, DiscoverInput, Rec…
* fix
* Add resharding status to API
* Report human readable resharding status in cluster info
* Fix delete stage not showing shard IDs
* Update OpenAPI spec
* Temporarily hide resharding operations in cluster info if empty
We'll enable it again to always be included when releasing resharding
* Remove resharding details from cluster info in gRPC until release
---------
Co-authored-by: timvisee <tim@visee.me>
* universal-query: Add validations to query input
* update openapi spec
* use raw request to by-pass local validations
* add validation for order_by and factorize
