* [ai] Replace manual into mappings with Into::into
* Reformat
* [ai] Use implicit .iter
* Don't iterate over keys too
* [ai] Replace unwrap_or
* Reformat
* [ai] Use as_deref and then_some
* [ai] Use more to_string
* [ai] Use explicitly typed into conversions
* Reformat
* [ai] More explicit into conversions
* Reformat
* Move `DenseVector`/`MultiDenseVector` from `api` to `segment`
* Move `OrderByInterface` from `api` to `segment`
Reason: it's used in `edge` which shouldn't depend on `api`.
* Make `shard` -> `api` dependency optional
* Remove `api` from the amalgamation
* Don't install protoc in edge Actions
* [manual] make untagged enum for consistent API for FeedbackStrategy
* add #[validate(nested)]
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* add rest and grpc interfaces
Also handle inference for this query
* follow refactor from base branch
* renaming from Ms Cooper
* get started on validation testing
* more validations
* test equivalence with query when less than 2 feedback elements
* rename feedback query to relevance_feedback query
* fix extraction of context pairs
* rename feedback vector to example
* make coderabbit happier
* upd test
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* weighted rrf implementation
* test
* fmt
* fix edge
* validate number of sources and number of weights
* do not partial match
* upd schema
* review fixes
* update formula
* remove calcualtions from tests
* update comment, because AI have OCD
* fmt
* introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert
* add test
* upd dockstring
* require resharding once all peers have updated version
* use service error
* fix clippy again
* wait for same version before resharding in tests
* reorder arguments of scroll_by
* add slow request log for scroll request
* add slow request log for facet request
* add slow request log for count request
* add slow request log for retrieve request
* Remove obsolete clone
* generic implementation of Loggable for Arc
* Refactor scroll by arguments (#7296)
* separate internal scroll operation from the one used by user
* fmt
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add ASCII folding to tokenization process
Introduced an optional ASCII folding feature within the `TokensProcessor` to normalize non-ASCII characters to their ASCII equivalents. Updated tests and documentation to reflect the changes.
* Refactor tokenization code for improved readability and maintainability
Reorganized and reformatted the tokenization module, including `TokensProcessor` initialization and ASCII folding mappings for better clarity. Updated tests to align with the changes.
* Update test cases to reflect optional tokenizer settings changes
Adjusted `ascii_folding`, `lowercase`, and `phrase_matching` settings in tests to `None` where applicable, aligning with updates in tokenizer configuration defaults.
* address review remarks
* fix codespell
* thx coderabbit
* Don't copy tokens that are already ASCII
* Shrink folded string to fit
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
* wip: generalization trait for queries
* implement generalization for point operations
* fmt
* log priority queue
* wip: SlowRequestsListener
* fmt
* fix clippy
* simplify generalization
* fmt
* implement collection of requests profiles for update API
* implement API for viewing slow requests log
* add collection name to update worker
* add datetime to log
* fmt
* probabilistic counter of unique requests
* rename
* compute hash before converting into json value
* move logable out of generalizable
* fmt
* log query request
* fmt
* some fixes
* move measurement into local shard
* fmt
* upd openapi (not important)
* For enum variants, has discriminant
* Make SearchParams Copy
* Hash 0.0 and -0.0 the same
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Correctly hash enum variants and float values
* Hash through ordered float instead
* Fix priority queue not keeping longest request for hash
* SearchParams implements Copy
* Fix clippy warning
* Add unordered_hash_unique
* skip serialization if none
* Use OrderedFloat for hashing a float
* Use OrderedFloat for hashing a float
* only log updates if they are performed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* allow custom K parameter for RRF
* generate grpc docs and openapi
* use tagged type approach for parametrized fusions
* use params approach in grpc
* simplify api structure
* upd schema
* nits
* rest: parameterized rrf as query variant
* consistency in doc comments
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* make bm25 parameters compatible with default fastembed params
* bm25 params in openapi schema
* fmt
* improve bm25 config deserialization
* more explicit docstring for disabling english
* expose mmr in rest and grpc
* fix referenced vectors in mmr queries
* rename to MmrInput in grpc for consistency
* mmr is now an optional parameter of nearest query
* fix score_threshold at local shard
* fix conversion of rest -> collection
* fix and refactor planned query
* nits
* rename `lambda` to `diversity` in interface
* handle score_threshold within mmr calculation
* add openapi test
* fix candidate limit at collection level
* candidate_limit -> candidates_limit
* finish rename to candidates_limit
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* move problems.rs to collection crate
* change type on datetime_key expression
* extract from expression
* check for unindexed fields in formula
* clippy
* dedup code
* Only implement verification for `CollectionQuery` types (#6319)
* verify CollectionQuery's Prefetch
* remove implementation for REST query
* remove implementation for REST query groups
* add basic openapi test
* Add in rest and grpc
* add to QueryEnum
* implement Query trait
* connect to scorer creation
* upd tests
* additional changes
* fmt
* gen openapi and grpc docs
* coderabbit fix
* add changes in async scorer
* test sum_scores in more places, refactor to remove repetition
* use f64 as internal expression score
* allow datetime as expression
* add test cases
* fix schema, generate openapi and grpc docs
* fmt
* clippy
* fix test precision
* graceful handling of too large numbers, more tests
* rename to DatetimeExpression for consistency
* use datetime's timestamp as seconds
* capture variable name when parsing
* homogenize DateTime into Datetime
* use interface distinction between datetime strings and payload keys
* fix rebase
* fix after rebase
* Add decay expressions
* remove code duplication in conversion
* test and fix lambda roundtrip
* Remove debug print in decay midpoint validation
* Improve error messages for decay midpoint and scale validation
* use similar lambda for gauss and exp decays
* avoid abs before squaring
* add comment about decay range
* add debug assertion
* Add payload index filtering IO measurements for some indices
* Add payload index metric to api and telemetry
* Also account for index access overhead
* Review remarks
* Anonymize new HardwareUsage field
* Fix tests
* Anonymize ShardKey in ReplicaSetTelemetry
* Anonymize vector names in CollectionConfigInternal
* no-op: impl HardwareUsage for HardwareUsage (used in HardwareTelemetry)
* no-op: impl Anonymize for OptimizersStatus (used in OptimizerTelemetry and CollectionsAggregatedTelemetry)
* no-op: impl Anonymize for TrackerStatus in (used in TrackerTelemetry)
* no-op: Anonymize VectorParams in VectorsConfig
* no-op: Copy fields explicitly in OperationDurationStatistics
* no-op: `.clone().map(|x| x.anonymize())` -> `.anonymize()`
* no-op: impl Anonymize for ConsensusThreadStatus (used in PeerInfo and ClusterStatusTelemetry)
* create REST, CollectionQuery, and ShardQuery structures. Parse all variables and conditions
* move stuff around, rename segment Expression to ParsedExpression
* rename to MultExpression
* add Div in REST, use unique geo_distance tag
* review fixes
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Allow max-optimization-thread config to be set to null on update
* add new field. Support both, but prepare for deprecation.
* better openapi
* only introduce change in OptimizersConfigDiff
* move to `api::rest::schema`
* update openapi
* improve test, fix diff to config conversion
* upd grpc docs
* clippy
* remove schemars from common common
---------
Co-authored-by: Gulshan Kumar <kumargu@amazon.com>
