d91beef0fd314250d8d9b94de86dfea019a8bd96
18
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Palash Debnath
|
5a615d2c66 |
feat(workers): package headless GPU nodes (#1638) (#1648)
Closes #1638.\n\nPackages headless GPU workers with durable enrollment, bounded artifact handling, cross-platform lifecycle cleanup, and regression coverage. Incorporates CodeRabbit, Greptile, CodeQL, and platform-CI findings before merge. |
||
|
velixio
|
ee35d2389e | fix: require remote voice render parity | ||
|
velixio
|
1fda5bdf96 | fix: preserve voice identity on remote workers | ||
|
Palash DebnathandClaude Fable 5
|
f83b7371c3 |
docs: remote-workers guide speaks VoiceStudio (#1543)
CodeRabbit's last-cycle Minor on #1540, applied as the immediate follow-up the docs-sync rule prescribes: three OmniVoice mentions in docs/remote-workers.md now carry the product's name. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Giuseppe Rojas
|
cd54113173 |
fix(security): close server-mode admin bypasses (#1525)
* fix(security): require keys for remote admin actions * fix(frontend): guard unavailable scrollIntoView * docs: link changelog to PR 1525 * fix(security): align PIN-only discovery policy * fix(security): preserve strict sidecar boundary * fix(security): normalize remote API keys * fix(auth): normalize credential fallback order |
||
|
Palash DebnathandClaude Opus 5
|
a310141114 |
feat(workers): join from the app, share by QR, and a status-bar Compute control (#1516)
* feat(workers): join from the app, share by QR, and a status-bar Compute control Remote workers shipped with a hole in the middle: the control plane could mint join codes, and on the other machine there was nothing to paste them into. Becoming a worker meant launching with OMNIVOICE_WORKER_MODE and OMNIVOICE_WORKER_TOKEN in the environment and relaunching — on the machine that is usually the least convenient one to configure by hand. Backend - GET /workers/agent, POST /workers/agent/join, POST /workers/agent/enabled. Join redeems a code and starts the agent live; no restart. - Worker mode now persists in settings as well as the environment (env still wins, and the panel is told so it can disable a switch it cannot honour), and it is written only after a join that actually worked — a failed enrolment must not have the app retrying on every launch. - The endpoint carried by the redeemed code is remembered. Without that a machine that joined from the UI came back up enrolled but with nowhere to dial, and the only fix was OMNIVOICE_WORKER_ENDPOINT. UI - "Lend this machine's GPU": paste the code, Join. Once joined it offers a switch rather than another code, because the pinned certificate survives. - <OneTimeSecret/> renders join codes and connection strings as a QR next to the text, with a live expiry countdown, and is used by both halves. QR generation is best-effort: a string past the format's capacity still shows the code and Copy, because losing the QR is a degraded share and losing the only copy of a one-time secret is data loss. - Status-bar Compute control: pick local or a machine, flip the feature, mint a join code — without opening Settings. Absent entirely until the user has opted in or enrolled something. - Remote workers now reads as a device list: status dot, address, latency, live task meter, resident models, last seen; housekeeping actions revealed on hover; a three-step empty state. - Approve is on the row. A worker could connect, sit there labelled "Not approved" and never be usable, with no way out of it in the UI. Fixes found on the way - Status dots and menu surfaces in the GPU picker were painted from fixed Tailwind palette classes (bg-emerald-400, text-amber-400, hover:bg-white/5), so on Midnight or Catppuccin they showed Gruvbox colours next to the theme's own. Both controls now paint from themed --color-* tokens, shared in computeTarget.jsx along with the JSON wrapper all three copies duplicated. - Button funnels every child into one <span>, so an icon passed as a child renders glued to its label — the flex gap only applies to the `leading` slot. Six buttons across these panels were affected. - InboundNodePanel passed `variant="warning"` to Badge, which takes `tone`; the "on your network" warning rendered as an ordinary neutral pill. Docs updated in the same change (docs/remote-workers.md): the join flow, the QR, the status-bar control, and the new environment variable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(changelog): stamp the remote-workers entries with their PR ref Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(workers): a join is not done until the control plane accepts it Review findings on #1516: - Greptile P1: `start()` only SCHEDULES the dial-out loop, so a control plane that rejected this worker — expired token, wrong address, a server that never answers — looked identical to a successful join. The route persisted worker mode, reported success, and the machine retried forever on every launch. The agent now signals first registration, and join waits for it before persisting anything. - CodeRabbit: a failed REJOIN left the machine unable to reconnect to the control plane it was already serving, because pinning the new certificate overwrites the old one on disk. Snapshot the pinned certificate, endpoint and setting up front, and restore them (and the running agent) when the join fails. - CodeRabbit: join and the enable toggle awaited stop()/start() with no exclusion, so two concurrent requests could interleave their pairs and have `start()` return early — reporting success for a control plane it never dialled. Both now hold one lifecycle lock. - CodeRabbit: with OMNIVOICE_WORKER_MODE set, the toggle still started or stopped the agent and wrote a setting the rest of the app ignores, contradicting the env_pinned status it reports. It now answers 409 and says which variable is in charge. - CodeRabbit: the QR code kept encoding the previous secret until the new one finished encoding, so the code on screen could disagree with the text beside it. CI: regenerated tests/fixtures/api_routes.txt for the three /workers/agent routes. Tests: a join the control plane never accepts is a 409 that persists nothing and leaves no agent dialling; a failed rejoin restores the previous certificate, endpoint and setting; an env-pinned machine refuses the toggle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(workers): the environment pin governs joining too, not just the toggle CodeRabbit, #1516: - join_control_plane skipped the env_pinned guard set_agent_enabled enforces, and joining is precisely what ENABLES worker mode: under OMNIVOICE_WORKER_MODE it wrote a setting nothing consults, and with the variable pinned off it handed back a machine that reported a successful join and lent nothing. One shared guard now covers both routes. - Two of the three rollback assertions could not fail before the fix (nothing wrote those settings on the failure path). The test now pins the behaviour only the rollback produces: the previous enrollment is dialling again, rather than left stopped until someone notices. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Codex
|
5818e19137 | fix(worker): secure inbound node connections | ||
|
velixio
|
090cc37144 |
fix(memory): release the model before emptying the cache, not after
The shared voice model's unload emptied the allocator caches and *then* dropped the reference. That frees nothing: the weights are still reachable when gc.collect() runs, empty_cache() only returns blocks the allocator already considered free, and the reference drops a moment later into a cache nothing will flush again. The unload logs success, the engine leaves the registry, and nvidia-smi does not move. Six modules open-coded the same two lines. Exactly one had them inverted -- OmniVoiceBackend.unload, which is the path the engine-registry idle sweep reaches, which is the sweep a headless worker node runs. So every unload a user could trigger from the UI worked, and the one that runs unattended on a machine lending its GPU held 3.6 GB indefinitely. Found on hardware: the sweep fired on schedule, logged "Released 1 idle engine(s)", and VRAM stayed flat at 3656 MiB for the next two minutes. Replace all six with model_manager.unload_shared_model(), which clears the reference, drops the clone-prompt side cache, then frees -- in that order, in one place. Two callers gain the side-cache drop they were missing (/system/flush-memory and the shutdown path), which is the same defect one step down: an unload that kept the encoded reference tensors belonging to the model it had just released. A source guard asserts nothing outside model_manager assigns the shared reference, so the next caller cannot reintroduce the ordering. It caught the sixth site while being written. Also give the AudioSeal watermark models the bargain every other model in the app already makes: they loaded on the first embed and stayed resident for the life of the process. CPU-resident, so this is system RAM rather than VRAM, and the machines that notice are the ones running batches. The error text on a failing unload changes with the ordering. "Could not be unloaded, retry after the current generation finishes" was accurate when the cache flush ran first and aborted before the release; now the release has already happened and only the flush can fail, so it says that instead of sending the user to repeat work that is done. |
||
|
velixio
|
b8fb5a14c2 |
feat(workers): let the idle-unload timings be shortened for testing
Watching a ten-minute rule take effect means waiting ten minutes, so it tends not to get watched. Both numbers are now env-tunable: OMNIVOICE_ENGINE_IDLE_UNLOAD_SECONDS and OMNIVOICE_IDLE_SWEEP_SECONDS. They are documented as a pair, because shortening only the threshold still means waiting a full sweep interval to see it fire — which reads as a broken sweep and sends you looking for a bug that is not there. Unparseable values and anything below the floor are ignored with a warning rather than honoured. A zero threshold would hand back a model the instant it went idle and reload it for the very next request, which is worse than the behaviour being tuned. |
||
|
velixio
|
33714b2fe0 |
fix(workers): make Disconnect hold, and stop a bad paste from replacing a good key
Two more found on hardware. Disconnect ended the session and the panel redialled two seconds later, so the log read disconnected and connected in the same breath and the button appeared to do nothing. A kicked key now sits out for a minute — long enough that the disconnect is real and the person notices, short enough that it is plainly not a revocation, which stays a separate and permanent action. The docs now say which of the two buttons does which. Re-pasting a connection string for an already-connected machine saved the new string and then short-circuited on the existing session, so a wrong key reported success, kept running on the old connection, and only failed after a restart — by which point nothing pointed back at the paste that caused it. The live session is now torn down before the new one is dialled. |
||
|
velixio
|
2b6f49c596 |
feat(workers): make inbound mode reachable — settings, endpoints, docs
Wires the two transport halves into something a user can actually turn on. Two independent switches, deliberately not one. "Accept connections" makes this machine a node others dial; "saved connections" are the nodes this panel dials out to. A workstation with a GPU that also drives jobs on a second box does both, so neither implies the other. Binding stays on 127.0.0.1 until someone explicitly widens it, and widening is its own field rather than a flag riding along with the enable toggle. With no encryption that boundary is the difference between a credential on one machine and a credential on a network, so it is never crossed as a side effect. The API reports `exposed` so the UI can say which side of it the user is on. Saved nodes are redialled only after the control plane is up, since the connector hands frames to its servicer. Failing to listen records the reason rather than leaving the feature looking enabled while it quietly accepts nothing. Docs say plainly that this mode is unencrypted, that the connection string is a password crossing the network in the clear, and that dial-out remains the better choice when one machine is enough. The Security section no longer implies its TLS guarantees cover both modes. |
||
|
velixio
|
aa1d739843 |
feat(workers): dubbing goes remote, and the protocol stops lying to old workers
The remote-GPU line, verified on hardware rather than asserted. **Dubbing renders on the worker.** dub_generate.py dispatches the coarse `dub_segments` operation through the gateway, following the audiobook pattern: per-unit local fallback after consecutive remote failures, one aggregated notice rather than one per segment. A 40-minute dub that loses its worker at segment 200 degrades instead of producing 200 error rows. **An out-of-date worker is now refused by name.** This was the worst defect in the plan and it was silent: an un-upgraded worker registered cleanly, then ignored `inputs` and rendered a clone with NO reference audio — returned as success. A plausible wrong result with nothing anywhere to surface it. Workers now declare features, and one missing them is turned away with the features named and `no task was run`. Verified live: a worker one commit behind was correctly refused. **"Offline" and "cannot run this" are different facts.** Asking a live worker for an engine it lacks answered "is offline or cannot be reached. Wake the selected worker" — while that worker reported ready, one free slot and 3.6 ms latency. The user was sent to wake a machine that was already awake. The scheduler now distinguishes absent from present-but- incapable, and names the engine rather than the operation, because the engine is the thing a user can install. **An engine with no catalog entry is no longer hidden.** A `repo_ids` non-emptiness check had been implemented as a runtime filter, so a worker silently refused to advertise any engine lacking a models.yaml entry — which is four registered engines, including CosyVoice. Users with those already installed would have lost remote support with only a log line. Empty `repo_ids` now means "not downloadable here", never "not runnable". **And a script so this stops being done by hand.** scripts/verify-remote-worker.sh runs the per-phase acceptance checks against a live worker, non-destructively. Its preconditions are the mistakes that cost the most time: exactly one listener on the control port (two instances silently shared it), and never detecting the worker with a pgrep pattern that matches the ssh shell running it. Its first real run found the dubbing picker claiming remote placement. That turned out to be the CHECK being stale, not the picker — the port had landed since it was written. It now asserts self-consistency instead: the picker may claim remote only for an operation the control plane actually advertises as remotely producible, which cannot rot the next time an op is ported. Backend 5291 passed, frontend 1812 passed. Acceptance script: no automated failures across Phases 4-8 on an RTX 4090. Four checks remain MANUAL by design — true airplane mode, concurrent downloads, killing a worker mid-audiobook, and the model-list UI — and are reported as unverified rather than passed. |
||
|
velixio
|
b7caa494eb |
feat(workers): remote downloads, audiobook chapters, and one port that stays honest
Five workstreams that finish the remote-GPU line, plus the test hole that let a broken signature reach a commit. **Downloads go through the normal path** (Phase 5). Rather than a second remote-only route, the existing Models install flow became target-aware, so a model landing on a worker uses the same code, the same progress events and the same UI as a local one. Progress rows key on (target, repo_id) — the aggregator keyed on bare repo_id, so the same model downloading here and on a worker at once collapsed into one row that told the user nothing true about either. **Audiobooks render chapter by chapter on the worker** (Phase 8), with per-chapter local fallback and ONE aggregated notice. The failure that shape exists to prevent: a remote GPU that sleeps at chapter 40 of 200 must not turn a working book into 160 rows of PROGRESS_LEASE_EXPIRED. Dictation is deliberately NOT ported — it runs ASR per utterance inside a live WebSocket loop, and paying queue admission plus a round trip there would spend the one thing that route is for. **Dubbing stays local, and says so** (Phase 7). The coarse worker operation is not finished, so the picker still reports dubbing as local rather than showing a green remote chip over work this machine is doing. What could not wait is the in-loop OOM retry: it sniffed the error string and flushed the *local* CUDA cache, which under remote execution is the wrong machine's GPU entirely. That is fixed now, before the path that would have exercised it exists. **Two instances can no longer share the control plane.** A second VoiceStudio silently bound the same worker port and coexisted, so remote workers landed on whichever process won the race — a session that registers with one instance and appears dead to the other. This produced hours of misdiagnosis during hardware testing and would hit any user with the app open twice. The second instance now keeps running locally and explains the conflict instead of quietly competing. **And the hole that allowed all this to be missable.** gpu_gateway called Scheduler.submit(pinned_worker_id=...) one commit before that parameter existed. Every remote generation raised TypeError; 5236 tests passed anyway, because nothing exercised the gateway against the real scheduler. tests/test_gpu_gateway_scheduler_contract.py now runs that path for real and binds every gateway→dependency call signature. Verified by renaming the parameter away and watching both tests fail with the original error. Gallery previews also fall back to a local render when a downloaded clip cannot be decoded, rather than yielding silence. Backend 5274 passed, frontend 1812 passed. Not yet verified on hardware: Phases 4, 5, 6, 7, 8. Only the TTS path and its artifact transport have been proven on a real GPU. |
||
|
velixio
|
bda169c900 |
feat(workers): pin work to the chosen GPU, and say when its model is missing
Three phases that only make sense together: a job that names a worker, a worker that reports honestly what it can actually run, and the small defects that made both lie. **Pinning** (Phase 1). `pinned_worker_id` is now honoured in both places that choose a worker — `eligible_workers` and `select_worker` build independent lists, so applying it to one silently leaked work onto whichever machine was least busy. The pin persists across a restart via an additive column, deliberately not alembic (justified in the code, per the precedent already in db.py): quitting mid-render used to drop it without a word. `max_attempts=1` was rejected as the mechanism — it makes the FIRST failure terminal, including the penalty-free ones a stale advisory view produces routinely. Cancel now actually reaches the worker. `WorkerServicer.cancel` had zero callers, so cancelling released the slot while the GPU thread kept running, and a late result could resurrect the task as COMPLETED — `commit_result` assigned that state directly, bypassing the transition table where CANCELLED is terminal by construction. **Honest capabilities** (Phase 4). A worker now probes whether weights are actually present, and a job stops BEFORE dispatch with a typed 409 naming the model and the machine, instead of failing mid-task. The probe fails OPEN: `is_cached`/`cache_is_complete` cannot see a user-managed clone outside the HF layout, so only a positive "absent" refuses. Refusing an engine that works today would break the compatibility promise. `pool.supports` deliberately still ignores `downloaded` — had it not, the scheduler would drop the worker and answer with a terminal NO_CAPABLE_WORKER, which tells the user to check their install when the truth is one download away. The frontend no longer offers "Report this bug" for that state; it offers the download. Catalog tags resolve against the TARGET's OS/arch/backend, not this machine's. From a Mac control plane, a CUDA worker's model list was showing the mlx-community repos it cannot run and hiding the ones it needs. **And the quiet ones** (Phase 0 leftovers): a model's human label rides its own proto field so renaming it cannot orphan breaker history; an empty model_id no longer forks the capacity slot key into two slots for one model; the idle sweep cannot evict an engine out from under a live LOCAL render. Verified on real hardware, which is the only verification that has ever caught anything here: 2025 characters, default settings, routed to an RTX 4090 over the wire — 100% GPU utilisation on the remote box, 119.6 s of 24 kHz audio returned in 16.6 s, 5.7 MB delivered out of band through the artifact path rather than the control stream. Backend 5259 passed, frontend 1808 passed. |
||
|
velixio
|
fe467d15e7 |
fix(workers): stop evicting workers for pinging at the interval we set
Every enrolled worker sat at connected=False against a healthy control
plane, and the control-plane log showed no Register call arriving at all.
The worker's own log said only "connecting", then nothing.
The cause was on our side of the handshake. The client sends an HTTP/2
ping every 25 s to keep its long-lived Control RPC alive through NAT —
an interval the control plane itself configures. But the server kept
gRPC's default enforcement policy, which permits two idle pings and then
answers ENHANCE_YOUR_CALM:
GOAWAY received; Error code: 11; Debug Text: too_many_pings
So the control plane hung up on every worker for obeying the keepalive
the control plane asked for. Idle workers were hit hardest, because a
session with no traffic is exactly the case the ping exists to protect.
Fixed by accepting the interval this protocol configures: a 20 s minimum
still rate-limits an abusive peer, while removing the idle-ping count
ceiling stops a healthy session dying of its own liveness mechanism.
This is a whole-fleet fix, not a per-enrollment one.
Worth recording what this was NOT, because it looked exactly like it:
TLS pin-on-first-use was the obvious suspect, since a control plane that
regenerated its certificate on restart would strand every enrolled
worker with no useful error. Disproved — the live certificate
fingerprint and the pinned copy on the remote worker match exactly, and
the certificate survives restarts. Enrollment was never involved.
Verified live against a remote worker: the session now establishes where
previously nothing reached the server. It is not yet stable — it drops
after ~17 s and advertises zero engines — but that is a separate defect
being tracked on its own, and this fix is a prerequisite for reaching it.
|
||
|
velixio
|
c643706d07 |
feat(workers): make a remote GPU actually run a task, end to end
Selecting a remote worker repainted a badge and nothing else. The cause was not subtle: `scheduler.submit` had no production caller, and `routing.decide()` was read only by the status endpoint that paints the header. Remote execution was a complete, tested pipeline with no producer at its head. This adds the producer and fixes the defects that made the pipeline unable to carry a real job: - Nothing routed to the scheduler. Adds `POST /workers/tasks` (loopback-gated, **development-only** until the gateway lands) and `Scheduler.wait`, backed by per-task futures rather than the unregisterable `on_change` listener list. - Every task over two minutes died. No worker ever sent `TaskProgress`, so the 120s progress lease expired mid-render — including during the cold model load, which happens after `TaskStarted`. Workers now report progress and emit a keepalive, bounded by the phase's absolute budget so it renews the lease without deleting the only enforced bound in the system. - The executor rebuilt its engine per task (`return cls()`), so every job paid a cold load. Engines now share one instance cache with the router, resolved by the assignment's engine — never `get_active_tts_backend()`, which returns the worker machine's own Settings preference and would silently run the wrong engine. - One lease expiry took a worker offline permanently: parked slots were never reclaimed. Parks now expire on a TTL, and are deliberately NOT reconciled against the worker's own load report — at a ceiling of one the only task such a worker can report is the wedged one, so "busy" would drop the park and the next idle heartbeat would hand out a slot with a live GPU thread (#730/#1190). - A worker that dropped and reconnected mid-render had every liveness frame discarded: task frames were fenced on the live session epoch, which bumps on every reconnect, while the worker echoes the ref stamped at dispatch. The control plane then expired a task whose GPU was still rendering, and swallowed the failure report when it went wrong. Fenced per attempt instead. - A result from one worker could commit another's task, after which the owner's real delivery arrived as a duplicate and its audio was discarded. "Unknown attempt" and "another worker's attempt" are no longer the same answer. - An oversized result was a poison pill, re-sent identically on every reconnect and permanently disconnecting the worker. It is now a terminal `RESULT_TOO_LARGE`, which is also classified — it was falling through to TRANSIENT and retrying a re-render that could never fit. - `_store_inline` joined the artifact directory with worker-supplied ids, and `os.path.join` discards its prefix on an absolute component. Paths are now minted control-plane-side and resolved through `core.path_security`. - Remote synthesis bypassed `mark_synthetic`, and the guard that exists to catch exactly that walked only `backend/api` and `backend/services` — so it stayed green while a fourth unmarked producer shipped. Marking moved to the worker's tensor stage; the guard now walks `backend/worker` too. Also adds pre-rendered voice previews (`services/gallery.py`), so browsing the gallery no longer needs a GPU or a downloaded model. The manifest is verified against the updater's release key already baked into the binary; a fresh install hears voices without downloading 2.4GB first, and everything falls back to local rendering when the gallery is unreachable. Verified on hardware, not just in CI: 1728 characters submitted to an RTX 4090 returned 105.94s of 24kHz audio in 23.9s, committed and served from the artifact store. Not yet done, and deliberately not claimed: the keepalive fix cannot be exercised end-to-end on fast hardware, because any job long enough to reach the 120s lease produces audio past the 8MiB inline cap. Chunked `UploadResult` has to land first. Pinning to the worker the user chose is also still absent, so "Remote" reaches a remote GPU but not necessarily the one on the badge. |
||
|
velixio
|
4f4d9c6e3e |
refactor(workers): give Remote workers its own System entry; ignore remote/
Remote workers was nested under Sharing, which reads backwards: everything in Sharing is about letting something else reach THIS machine (a remote backend, an MCP client, a share PIN), while remote workers sends work OUT to machines you own. It is now its own System entry. Docs-sync: every "Settings → Sharing → Remote workers" reference is updated — the guide, the changelog, the two API error messages that tell a user where to generate a token, and the agent's not-enrolled error. Also ignores remote/ (local goal docs, review briefs, council reports) and repoints the code comments that cited remote/goal_v2.md at the shipped docs/remote-workers.md, so no committed file references a path that is not in the repo. |
||
|
velixio
|
43de1c794c |
feat(workers): remote GPU workers over a versioned gRPC protocol
Send individual jobs to GPUs on your other machines while everything else stays local. Opt-in, off by default: with the toggle off there is no listening socket, no certificate and no background loop. Design follows remote/goal_v2.md, the council-revised goal doc. The decisions that shaped the code, and why: * A disconnect is an unknown outcome, not a failure. The original design reassigned on disconnect while also describing the case where the worker had already finished — following both guarantees duplicate execution. An attempt now holds a grace window; a worker returning inside it commits its result and no second attempt is ever made. * At-least-once execution, exactly-once result commit. The result is persisted BEFORE it is acknowledged, so a crash between the two cannot silently lose a finished render. * Deadlines are phased (accept -> model load -> execute -> deliver) and liveness is a progress lease. The old fixed 30s execution budget was two orders of magnitude below what this product actually does; silence is the failure signal, not slowness. * Capacity is derived from free VRAM, never configured: a static value corrupts output under torch.compile thread affinity (#315) and aborts the process on small cards (#567). * A circuit breaker replaces the reliability-score/quarantine machinery, which had no recovery path (no probation workload exists in a TTS product) and penalised consumer networks for existing. * Identity is a keypair the worker generates and never sends. A server-assigned id is a name, not an authenticator, so revocation of one would be theatre. Enrollment tokens are single-use and carry the control plane's certificate fingerprint for pin-on-first-use. Adds the domain core, scheduler, durable task store, gRPC transport, worker agent, management API, Settings panel, and docs. Protobuf reserves the tenant/trace/usage fields a hosted control plane would need, since adding them later means upgrading a whole fleet. Includes tests for the failure paths that matter: duplicate delivery, stale-session fencing, reconnect reconciliation, grace expiry, breaker attribution, and a real end-to-end TLS round trip. |