Commit Graph

605 Commits

Author SHA1 Message Date
Jojii
13cabb32e4 Add hardware usage to grpc API (#5333)
* Add hardware usage to grpc API

* generate grpc docs
2024-11-08 11:25:36 +01:00
n0x29a
097508d07d Inference Service, query requests (batch processing) (#5329)
* Inference Service, query requests (batch processing)

---------

Co-authored-by: n0x29a <n0x29a@example.com>
Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-08 11:23:22 +01:00
Jojii
c1d0a8d61e Populate hardware counter to REST API (#5308)
* populate hardware counter

* make consume semantic explicit

* Merge pull request #5328

* add hardware info to more endpoints

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-08 11:19:41 +01:00
Andrey Vasnetsov
c71b008e69 Reinit consensus (#5265)
* option to clear consensus state while preserving peer id

* compact or clear consensus WAL on re-init

* Add test on reinit

* minor changes

* more points check suggestion

---------

Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
2024-11-08 11:19:36 +01:00
Jojii
cc807cbfbc Add hardware measurements in API (#5306)
* add HwMeasurementAcc and API helpers+types

* add config to API to toggle hardware reporting in API responses

* add Default impl for HwMeasurementAcc

* comments

* review remarks

* include usage into API schema

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-08 11:19:25 +01:00
Arnaud Gourlay
b2a7f01236 Report memory usage in metrics & telemetry (#5266)
* Report memory usage in metrics & telemetry

* Fenster reparatur

* fix error handling & openapi spec

* follow naming convention

* show memory stats on level1

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-08 11:15:43 +01:00
n0x29a
672740540e Inference Service, query requests (rest) (#5295)
* Inference Service, query requests (rest)

---------

Co-authored-by: n0x29a <n0x29a@example.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2024-11-08 11:12:26 +01:00
n0x29a
62380a742a Inference Service, billing-id (#5287)
* Inference Service, token

---------

Co-authored-by: n0x29a <n0x29a@example.com>
2024-11-08 11:11:36 +01:00
n0x29a
2789f7f0e1 Inference for image (#5267)
* Inference Service for Images

* Inference Service for Images

---------

Co-authored-by: n0x29a <n0x29a@example.com>
2024-11-08 11:09:48 +01:00
Roman Titov
903327cf24 Refactor ConsensusOpWal (#5217)
* Refactor `ConsensusOpWal`

* fixup! Refactor `ConsensusOpWal`

I hate `clippy::comparison_chain` and hope it dies 🤬

* fixup! Refactor `ConsensusOpWal`

Tweak max size condition in `ConsensusOpWal::entries`

* Add documentation to `ConsensusOpWal::entries` method

Co-authored-by: Tim Visée <tim+github@visee.me>

* Compact consensus WAL (#5221)

* WIP: Compact `ConsensusOpWal`

* fixup! WIP: Compact `ConsensusOpWal`

* Compact consensus WAL

* fixup! Compact consensus WAL

- Rename `min_entries_to_truncate` to `min_entries_to_compact`
- Do not compact if `compact_wal_entries`/`min_entries_to_compact` is 0
- Avoid underflow when calculating if we should compact WAL

* fixup! fixup! Compact consensus WAL

Rename `min_entries_to_truncate` to `min_entries_to_compact` (lol)

* review comments

---------

Co-authored-by: generall <andrey@vasnetsov.com>

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: generall <andrey@vasnetsov.com>
2024-11-08 11:09:33 +01:00
Tim Visée
6f51df0124 Make some or's more lazy (#5255)
* In config merge, lazily replace overrides versus proactively taking out

* Add helper function to replace option if value is some

* Make some eager ors more lazy

* Do not aggressively take out
2024-11-08 11:08:35 +01:00
Arnaud Gourlay
f4c17524c4 Do not create errors eagerly (#5254) 2024-11-08 11:07:47 +01:00
n0x29a
cf8f22fbe7 Inference Service Integration (#5240)
* Inference Service Integration

* Update src/common/inference/service.rs

Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>

* Inference Service Integration

* review fixes

---------

Co-authored-by: n0x29a <n0x29a@example.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
2024-11-08 11:06:55 +01:00
xzfc
02050108a0 Use SnapshotStream both for S3 and streaming snapshots (#5238)
* Use SnapshotStream both for S3 and streaming snapshots

* Set Content-Disposition header for streaming shard snapshots
2024-11-08 11:05:47 +01:00
xzfc
ba93831683 Fix shard snapshot operations for s3 (#5227)
* Fix shard snapshot operations for s3

* tests/shard-snapshot-api.sh: Improve error reporting

* Drop SnapshotStorageManager::get_shard_snapshot_path

* Fix s3 test on CI
2024-11-08 11:02:37 +01:00
Roman Titov
e6ba988baa Persist and propagate origin peer when bootstrapping new nodes (#5147) 2024-11-08 10:58:45 +01:00
Roman Titov
83c1dc987e When initializing new cluster, propose origin peer to consensus (#5141)
Co-authored-by: Tim Visée <tim+github@visee.me>
2024-11-08 10:58:03 +01:00
Kumar Shivendu
751b4adced Dont consider resharding shards as unhealthy for /readyz (#5204)
* Dont consider resharding shards as unhealthy for /readyz

* clippy
2024-11-08 10:55:20 +01:00
Andrey Vasnetsov
1d0ee7ea32 Inference interface in REST and gRPC (#5165)
* include document & image objects into grpc API

* introduce image and object to rest api

* minor refactoring

* rename Vector -> VectorInternal

* decompose vector data structures

* add schema

* fmt

* grpc docs

* fix conversion

* fix clippy

* fix another conversion

* rename VectorInput -> VectorInputInternal

* replace grpc TryFrom with async functions

* fmt

* replace rest TryFrom with async functions

* add image and object into query rest

* separate inference related conversions

* move json-related conversions into a separate file

* move vector-related transformations into a separate file

* move more vector related-conversions into dedicated module
2024-11-08 10:55:15 +01:00
Tim Visée
347c1945b1 Report CPU endianness in telemetry output (#5180)
* Report CPU endianness in telemetry output

* Make endianness optional, don't report if anonymizing telemetry

* Update OpenAPI definition

* Switch back to single compile time endianness check

* Update OpenAPI specification

* Remove cpu-endian dependency

* De-anonymize CPU endianness

---------

Co-authored-by: xzfc <xzfcpw@gmail.com>
2024-11-08 10:54:46 +01:00
Tim Visée
76e167a069 Add wait parameter to cluster metadata API (#5174)
* Add blocking implementation of update cluster metadata

* Expose wait parameter to cluster metadata API

* Refactor shard transfer dispatcher into toc dispatcher

* Reformat
2024-10-08 13:37:05 +02:00
xzfc
89cce79dac Support SnapshotFormat::Streamable, add REST endpoint (#5178) 2024-10-08 13:15:29 +02:00
Roman Titov
180d09a68f Instantly self-elect Raft leader, when cluster contains single peer (#5173)
* When starting first peer of a new cluster, tick a few times to establish leader instantly

* Instantly self-elect itself as Raft leader, when cluster contains single peer
2024-10-08 13:14:51 +02:00
Jojii
7830506a1b remove obsolete timeout (#5142) 2024-10-08 13:10:56 +02:00
Jojii
05b8dca540 [Strict mode] Merge StrictMode config types (#5137)
* merge StrictModeConfig and StrictModeConfigDiff into one type

* review remarks
2024-10-08 13:10:33 +02:00
Jojii
9e1ccf6237 [Strict mode] migrate toc_new -> toc (#5120)
* Replace all remaining calls of toc() with toc_new

* mirgate toc_new -> toc
2024-10-08 13:10:01 +02:00
Jojii
0b236d8107 [Strict mode] default config (#5037)
* add default config for strict mode

* add default values as example to config/config.yaml

* improve implementation

* update grpc docs
2024-10-08 13:09:56 +02:00
Jojii
b7c5f73590 [Strict mode] GRPC verification (#5003)
* first grpc implementations

* make usage of 'scroll' more clear

* introduce 'CheckedTocProvider' trait

* implement more grpc types

* more grpc implementations

* rename StrictModeCheckedProvider to StrictModeCheckedTocProvider

* don't use deprecated feature for UncheckedTocProvider
2024-10-08 13:09:49 +02:00
Jojii
50c06177f4 [Strict mode] Finish REST requests (#4990)
* add strict mode to more requests

* add timeout check from request query

* unify timeout checking

* adjust functions not requiring strict mode

* add missing checks

* add Prefetch check for query api

* review remarks
2024-10-08 13:09:45 +02:00
Roman Titov
0f5b1caa26 Refactor local_shard_points::delete_points API into cleanup_shard API (#5024)
Co-authored-by: timvisee <tim@visee.me>
2024-09-23 16:04:03 +02:00
Roman Titov
f3987bec38 Properly preserve full error chain, when handle_committed_entries returns an error (#5095) 2024-09-17 15:30:34 +02:00
xzfc
bab137f4d8 Unpack snapshots with sync (#5075)
* refactor: Simplify open_snapshot_archive_with_validation

* feat: tar::Archive::set_sync(true)
2024-09-17 15:29:23 +02:00
Roman Titov
90fbb88297 Log underlying error, when consensus is stopped due to one (#5066) 2024-09-17 15:28:33 +02:00
Roman Titov
9ab060b565 Split operations into update_all/update_existing during resharding (#4928)
Co-authored-by: generall <andrey@vasnetsov.com>
2024-09-17 15:28:29 +02:00
Andrey Vasnetsov
52700c52d8 custom key to hide kwt dashboard (#5056) 2024-09-17 15:27:12 +02:00
Tim Visée
41ea2bc75c Expose cluster metadata in telemetry (#5019)
* Expose cluster telemetry in metadata

* Only expose metadata in telemetry for details level 1 and up

* Update OpenAPI specification

* move metadata into cluster telemetry

* regen openapi

* use Option instead of raw hashmap

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-09-17 15:26:26 +02:00
Jojii
1527119899 [Strict-mode] Request verification implementation (#4944)
* implement verification for more types and improve trait

* implement StrictModeVerification for update requests

* more impls

* rebase

* add strict mode for more requests

* improve checking

* exhaustive match

* review remarks
2024-09-17 15:26:13 +02:00
Arnaud Gourlay
ba57d63200 gRPC API for Distance Matrix (#5011)
* gRPC API for Distance Matrix

* generate API docs

* apply correct limit to sample

* add JWT tests

* pluralitiy

* fix COO naming

* drops COO as it trips codespell and is redundant

* clarify docs
2024-09-17 15:26:02 +02:00
Jojii
5335e9365e [Strict-mode] Verification check in API (#4935)
* verification check in API

* review remarks

* minor review fixes [skip-ci]

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-09-17 15:25:56 +02:00
Sebastiaan van Steenis
f4d82f75f6 Add logs for JWT RBAC configuration (#4581) 2024-09-17 15:25:11 +02:00
Arnaud Gourlay
708b941f0d Create Telemetry client only once (#4992) 2024-09-17 15:24:50 +02:00
Jojii
3377530c62 [Strict-Mode] Basic implementation (#4887)
* add CollectionRequestVerification

* add to api

* rebase

* improve implementation

* implement strict mode for SearchRequest+Batch

* improve code + fix Clippy

* improve error handling

* restructure StrictModeVerification trait

* generate docs

* check `enabled` option

* review remarks

* rename StrictModeConfigDiff in grpc

* use missing payload detection from issue api

* performance improvement

* decouple extractor from issues (#4945)

* some review remarks

* don't default to empty functions in StrictModeVerification trait

* update openapi

* filter_limit => query_limit

* replace discovery_max_context_size and recommend_max_examples with max_input_examples

* review remarks

* review fix: include possible index types into error message

* review remarks

---------

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: generall <andrey@vasnetsov.com>
2024-08-29 15:24:11 +02:00
Roman Titov
8374ec09cf Cleanup actix::helpers module (#4951) 2024-08-28 15:18:04 +02:00
Andrey Vasnetsov
287f287bbd Implement better handling of UUID index (#4961)
* rename keyword -> string for internal data structures

* implement MatchAny and expect for filter on UUID mmap index

* implement MatchAny and expect for cardinality estimation on UUID mmap index

* refactor PayloadFieldIndex filter for handling incorrect empty query parsing case
2024-08-27 01:16:44 +02:00
Andrey Vasnetsov
9573162ffe include peers list into telemetry (#4947) 2024-08-27 01:14:45 +02:00
Roman Titov
9e8ba48bf1 Add local shard points get/scroll/count/delete HTTP API (#4843) 2024-08-27 01:13:31 +02:00
Luis Cossío
5f5fd25d64 Add grpc endpoint for Facet (#4933)
* add grpc endpoint

* gen grpc docs
2024-08-27 01:13:12 +02:00
Luis Cossío
45f6e67ed1 fix and futureproof mismatch of index parametrization and type (#4929) 2024-08-27 01:12:01 +02:00
Srubin Sethu Madhavan
bf743a6e08 log errors when wait=false (#4926)
Co-authored-by: Srubin Sethu Madhavan <ssethumadhavan@hubspot.com>
2024-08-27 01:09:53 +02:00
Jojii
217b87e76c Wrap handler body in helpers::time (#4901)
* wrap handler body in helpers::time

* remove unnecessary async closures

* fix clippy

* remove more async closures

* avoid another async {}

* remove more async {}

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-08-27 01:09:32 +02:00