Commit Graph

376 Commits

Author SHA1 Message Date
Luis Cossío
697c797fa4 add formula to internal grpc api (#6057) 2025-02-26 10:27:53 -03:00
Luis Cossío
64ff5f0af5 [score boosting] add more expressions (#6054)
* sqrt, pow, exp, log, ln, abs

* upd openapi and grpc

* handle undefined math as negative infinity
2025-02-26 09:31:50 -03:00
Arnaud Gourlay
69a9150bbd Struct deconstruction in conversions (#6066)
* Struct deconstruction in conversions

* payload ops conversions

* inference conversions

* RecordInternal conversion

* shard operation conversions
2025-02-26 12:42:10 +01:00
Arnaud Gourlay
fbf8dfd350 Struct deconstruction in gRPC conversions (#6062)
* Struct deconstruction in gRPC conversions

* explicit conversion
2025-02-26 09:44:25 +01:00
Tim Visée
3e536347e1 Bump Rust edition to 2024 (#6042)
* Bump Rust edition to 2024

* gen is a reserved keyword now

* Remove ref mut on references

* Mark extern C as unsafe

* Wrap unsafe function bodies in unsafe block

* Geo hash implements Copy, don't reference but pass by value instead

* Replace secluded self import with parent

* Update execute_cluster_read_operation with new match semantics

* Fix lifetime issue

* Replace map_or with is_none_or

* set_var is unsafe now

* Reformat
2025-02-25 11:21:25 +01:00
Luis Cossío
b494771e61 [score boosting] add gRPC structures (#6035)
* add grpc structures

* grpc docs

* make by_zero_default optional
2025-02-24 10:44:46 -03:00
Luis Cossío
08cf86c421 [score boosting] Connect formula to query (#6024)
* finish connecting formula query

* upd openapi
2025-02-21 09:36:18 -03:00
Luis Cossío
c2312d0e50 [score boosting] create query structures and parse them (#5990)
* create REST, CollectionQuery, and ShardQuery structures. Parse all variables and conditions

* move stuff around, rename segment Expression to ParsedExpression

* rename to MultExpression

* add Div in REST, use unique geo_distance tag

* review fixes

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2025-02-21 09:03:25 -03:00
Tim Visée
7a56fd143b Fix mix ups in read/write rate limit settings (#6029) 2025-02-20 10:15:32 +01:00
Arnaud Gourlay
d88e76f90d Fix Clippy 1.85 (#6011)
* Fix Clippy 1.85

* fix false positive
2025-02-18 10:41:28 +01:00
Arnaud Gourlay
1a7e2701d7 Add strict mode for max points count (#5967)
* Add strict mode for max points count

* clarify estimated nature
2025-02-12 11:10:51 +01:00
Kumar Shivendu
9d56dcfa7a fix typo (#5960) 2025-02-08 17:05:53 +05:30
Jojii
bae4c08a2a Measure IO reads for vectors (#5912) 2025-02-03 11:56:43 +01:00
Jojii
5a321d6f09 HwCounter rename IO metrics (#5898)
* Rename io metrics and add vector_io_write

* Chore: Cleanup obsolete TODOs and simplify retrieving HardwareCounterCells (#5899)
2025-01-30 10:23:35 +01:00
Luis Cossío
af74d1b96a bump and migrate to rand 0.9.0 (#5892)
* bump and migrate to rand 0.9.0

also bump rand_distr to 0.5.0 to match it

* Migrate AVX2 and SSE implementations

* Remove unused thread_rng placeholders

* More random migrations

* Migrate GPU tests

* bump seed

---------

Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2025-01-28 16:19:11 +01:00
xzfc
d90a68137b Add VectorName type alias (#5763)
* Add VectorName/VectorNameBuf type aliases [1/2]

* Add VectorName/VectorNameBuf type aliases [2/2]
2025-01-24 01:29:01 +00:00
n0x29a
1796c76e35 Strict mode sparsevector (#5841)
* Strict mode sparse vectors
* Update gRPC docs
* Use deny_unknown_fields (#5868)

---------

Co-authored-by: n0x29a <n0x29a@example.com>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2025-01-23 17:34:38 +00:00
xzfc
12137d2264 Fix cargo doc warnings (#5854) 2025-01-23 09:37:20 +01:00
Arnaud Gourlay
4dee0c5bf4 Strict mode multivector (#5757)
* strict mode for multivector

* simplify impl

* regen docs
2025-01-20 15:53:35 +01:00
Tim Visée
47de3955db Fix deleted threshold validation in gRPC, should be in range [0.0, 1.0] (#5810) 2025-01-15 15:39:32 +01:00
Tim Visée
7d57ee485f Resharding: expose direction in gRPC API (#5794)
* Expose resharding direction in gRPC API

* Trim whitespaces
2025-01-14 14:18:51 +01:00
Jojii
3e96fae333 [Strict mode] limits for filter and conditions (#5754)
* Add limits for filter and conditions

* clippy

* Review remarks + nested condition test

* Fix opnapi specs

* Improve error message by giving info about limits and usage
2025-01-14 11:07:27 +01:00
Jojii
9ad886691e IO metrics in hw counter types (#5769)
* io metrics in hw counter types

* Add new metrics to `new_with`
2025-01-10 18:28:34 +01:00
Tim Visée
306f4a88ce Resharding: expose operations in cluster info (#4599)
* Expose resharding operations in cluster info

* Fix collection info conversion to gRPC
2025-01-10 11:50:31 +01:00
Tim Visée
db80432f8b Filter out empty filter conditions from gRPC (#5690)
* Filter out empty filter conditions from gRPC rather than error

* Link pull request
2024-12-24 21:36:56 +00:00
Roman Titov
8beb1106a6 Abort scale-down resharding on update error (#5670) 2024-12-24 13:58:32 +01:00
Tim Visée
2f7cde3d45 Fix collection creation validation, disallow replication factor 0 (#5643)
* Validate default collection config in settings

* Validate shard number, replication/write consistency factor in gRPC

* Cleanup

* Add more validation rules

* Also validate strict mode config in gRPC CollectionConfig

* Update OpenAPI specification
2024-12-20 12:07:58 +01:00
Luis Cossío
daef7134e2 Enable resetting max_optimization_threads to automatic (#5634)
* Allow max-optimization-thread config to be set to null on update

* add new field. Support both, but prepare for deprecation.

* better openapi

* only introduce change in OptimizersConfigDiff

* move to `api::rest::schema`

* update openapi

* improve test, fix diff to config conversion

* upd grpc docs

* clippy

* remove schemars from common common

---------

Co-authored-by: Gulshan Kumar <kumargu@amazon.com>
2024-12-18 17:17:47 -03:00
Arnaud Gourlay
229fa94f69 Strict mode for payload storage (#5588)
* Strict mode for payload storage

* Don't increment counter multiple times per request

* Add (loose) integration tests for payload storage limit

* Minor improvements

* minor renaming

* Update lib/api/src/grpc/proto/collections.proto

Co-authored-by: Tim Visée <tim+github@visee.me>

---------

Co-authored-by: jojii <jojii@gmx.net>
Co-authored-by: Jojii <15957865+JojiiOfficial@users.noreply.github.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
2024-12-17 17:33:46 +01:00
Luis Cossío
14c1979a47 Integrate mmap bool index (#5571)
* add on_disk option for bool index

* test that all files are covered

* generate openapi and docs

* clippy

* remove `populate` changes

* use `walkdir` crate

* Apply clippy suggestions

---------

Co-authored-by: timvisee <tim@visee.me>
2024-12-16 20:35:10 -03:00
Tim Visée
7d52c95253 Simplify gRPC validation (#5650) 2024-12-16 12:46:43 +01:00
Jojii
e0d0e233d8 Timeout aware hardware counter (#5555)
* Make hardware counting timeout aware

* improve test

* rebuild everything

* fmt

* post-rebase fixes

* upd tests

* fix tests

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-12-10 12:12:36 +01:00
Arnaud Gourlay
2a2a7033db Rate limit requests per minute (#5597)
* Rate limit requests per minute

* rename to remove time unit for API
2024-12-10 00:26:37 +01:00
Arnaud Gourlay
1e21a4a179 Rate limiting for shard operations (#5582)
* Rate limiting for shard operations

* address all review comments in one go
2024-12-06 11:02:18 +01:00
Redouan El Rhazouani
0c55bcc53d Optimize JSON-Proto Conversion, Improve Memory Allocation, and Add Unit Tests (#5591)
* Optimize JSON-Proto conversion and add unit tests
 * mproved memory allocation by eliminating raw loops
 * Enhanced readability by reducing nesting and restructuring logic for clarity
 * Added tests to assert correctness and reliability of the JSON-Proto conversion process

* Resolve clippy issues
2024-12-06 10:43:58 +01:00
Jojii
42fd2e27e2 Strict mode max collection vector size (#5501)
* Strict mode config: Max collection size

* api specs

* Add tests + set/update payload check

* Improve function names and add comments

* rename config to separate vectors and payload

* fix tests

* Adjust configs docs

* add benchmark

* improve performance by caching shard info

* add bench for size_info() and fix tests

* Also limit the batch-size for vector updates (#5508)

* Also limit the batch-size for vector updates

* clippy

* add lost commit

* Load cache on collection initialization

* add unit type to parameter name

* fix renaming in test

* clearer error message

* fix test

* review remarks

* remove unused function for now

---------

Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
2024-12-05 17:28:09 +01:00
Luis Cossío
5003b85719 Friendlier untagged enum errors (#5580)
* add friendlier errors for untagged enums

* alias with_vector and with_vectors in query api
2024-12-03 17:53:22 -06:00
Luis Cossío
ed668c636f improve QueryInterface error message (#5546) 2024-11-28 16:26:37 -06:00
Arnaud Gourlay
36d07a5444 Return complete error for sparse vector validation (#5536)
* Return complete error for sparse vector validation

* remove unecessary path

* update test to show new value
2024-11-28 12:27:12 +01:00
Jojii
5417428951 Add max upsert batch size strict mode option (#5485) 2024-11-19 19:06:57 +01:00
Jojii
55b193a2c8 Add hardware metrics to /metrics and /telemetry API (#5408)
* add hardware metrics to /telemetry and /metrics

* move all_hw_metrics function
2024-11-19 15:08:04 +01:00
Luis Cossío
b895bcd872 Transfer blob_store into this repo (#5436)
* copy-paste blob_storage

* get rid of copied utils

* fix clippy

* fmt

* audit blob_store manifest

* move `Payload` into `fixtures.rs`

* fix clippy

* put csv in google storage

* ignore hm test

* fix codespell

* improve files test

* test different block sizes

* remove unused deps

* replace StdRng with SmallRng for faster tests
2024-11-18 12:43:52 -06:00
Jojii
b83ace82f0 Per collection hardware measurements (#5453)
* Add HwMeasurementCollector

* Add hardware reporting to TOC + RequestHwCounter

* Pass HwMeasurementAcc by reference + Update accumulation

* Update tests and benchmarks

* update REST API

* Update gRPC API

* codespell

* Adjust internal API

* improve docs

* introduce drain to the HwMeasurementAcc

* fmt

* use drain to report to the collection counter

* implement hw metrics drain for internal and external queries

* fix drinage

* refactor rest models: move away from grpc crate

* fmt

* implement usage reporting to collection acc for rest api

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-17 12:46:01 +01:00
n0x29a
7833fba05f allow inference for Documents in points (grpc + rest) (#5447)
* allow inference for Documents in points (grpc + rest)
* move validation to documents & images

---------

Co-authored-by: n0x29a <n0x29a@example.com>
Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-15 10:31:05 +01:00
George
15656e9110 Make model name field mandatory in inference structures (#5431)
* Make model name field mandatory in inference structures

* fix batch tests
2024-11-12 17:00:27 +01:00
George
ccc5551701 update Image.image type to Value instead of string (#5403)
* update Image.image type to Value instead of string

* apply cargo fmt

* fix tests
2024-11-08 20:50:27 +01:00
Jojii
7b677fb7fa Hardware counting for quantization (#5369)
* add cpu measurement for quantization

* clippy

* fix tests

* discard hardware counters in benchmarks

* Forwarding hardware counter in distributed setup (#5371)

* make hardware counter available in distributed setup

* clippy

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-06 16:23:02 +01:00
Jojii
e35ec150df Add hardware usage to grpc API (#5333)
* Add hardware usage to grpc API

* generate grpc docs
2024-11-05 16:15:39 +01:00
n0x29a
717594bdf5 Inference Service, query requests (batch processing) (#5329)
* Inference Service, query requests (batch processing)

---------

Co-authored-by: n0x29a <n0x29a@example.com>
Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-04 14:19:37 +01:00
Jojii
70646af7a8 Add hardware measurements in API (#5306)
* add HwMeasurementAcc and API helpers+types

* add config to API to toggle hardware reporting in API responses

* add Default impl for HwMeasurementAcc

* comments

* review remarks

* include usage into API schema

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-10-29 09:53:22 +01:00