Two phases of the remote-GPU plan, landing together because neither is
useful alone: on a 4090 any render long enough to exercise the progress
lease also outgrows the 8 MiB message cap, so a gateway that routes work
remotely without an artifact transport just moves where the failure
happens.
**The gateway** (`services/gpu_gateway.py`) is the single owner of GPU
calling, model status, downloads and model load, for both targets —
`prewarm`, `run`, `status`, `download`. prewarm and run stay separate
because collapsing them loses the two-phase load/generate budget split
(#1033/#1037) that the worker protocol already mirrors. Admission moves
in here too: the old `check_gpu_admission` call read *local* pool stats,
so under Remote it would 429 on local saturation while the remote GPU
sat idle.
**Artifacts** now move out of band above a negotiated threshold. Bytes
land in an attempt-scoped `.part` file, are verified against a declared
sha256, and are renamed into place only on an explicit last chunk — a
transfer that arrives short, reordered, or simply stops commits nothing.
A resume rehashes what is already on disk, or the digest would attest
only to the tail, which is the exact case a resume exists to protect.
Two failure modes found while verifying this, both fixed with
mutation-checked regressions:
* an oversized payload with no session (mid-reconnect, or a control
plane too old to serve UploadResult) has nowhere to go. It must not
enter `_pending` — an over-cap frame is re-sent on every reconnect,
killing the session each time and stranding every other task — but
it must stay retryable, unlike the size gate's TERMINAL verdict:
nothing about the render is wrong, only the route to it.
* the upload resume loop was bounded by "did the offset change", which
a receiver alternating between two byte counts satisfies forever.
The worker is single-slot by default, so that is not one lost upload
but the machine, doing nothing else, until someone restarts it.
Bounded by a round count instead.
The control stream is split into control and bulk queues so the
heartbeat this whole liveness model rests on cannot queue behind a
payload — `result_json` has no size cliff to catch it, and the next bulk
message added to the protocol would have reintroduced the stall
silently.
Live streaming stays on the control plane and now says so once per
socket: that route exists to put audio in the user's ear before the
sentence finishes, and paying queue admission plus a round trip per
utterance would spend the one thing it is for. Silence would have been
worse than the limit — the header badge would read "gpu2" while this
machine did all the work.
Backend 5236 passed, frontend 1807 passed. End-to-end verification on
real hardware has NOT been re-run since these changes; the CHANGELOG
claim for the Synthesize button waits on that.
Selecting a remote worker repainted a badge and nothing else. The cause was
not subtle: `scheduler.submit` had no production caller, and `routing.decide()`
was read only by the status endpoint that paints the header. Remote execution
was a complete, tested pipeline with no producer at its head.
This adds the producer and fixes the defects that made the pipeline unable to
carry a real job:
- Nothing routed to the scheduler. Adds `POST /workers/tasks` (loopback-gated,
**development-only** until the gateway lands) and `Scheduler.wait`, backed by
per-task futures rather than the unregisterable `on_change` listener list.
- Every task over two minutes died. No worker ever sent `TaskProgress`, so the
120s progress lease expired mid-render — including during the cold model
load, which happens after `TaskStarted`. Workers now report progress and
emit a keepalive, bounded by the phase's absolute budget so it renews the
lease without deleting the only enforced bound in the system.
- The executor rebuilt its engine per task (`return cls()`), so every job paid
a cold load. Engines now share one instance cache with the router, resolved
by the assignment's engine — never `get_active_tts_backend()`, which returns
the worker machine's own Settings preference and would silently run the
wrong engine.
- One lease expiry took a worker offline permanently: parked slots were never
reclaimed. Parks now expire on a TTL, and are deliberately NOT reconciled
against the worker's own load report — at a ceiling of one the only task such
a worker can report is the wedged one, so "busy" would drop the park and the
next idle heartbeat would hand out a slot with a live GPU thread (#730/#1190).
- A worker that dropped and reconnected mid-render had every liveness frame
discarded: task frames were fenced on the live session epoch, which bumps on
every reconnect, while the worker echoes the ref stamped at dispatch. The
control plane then expired a task whose GPU was still rendering, and swallowed
the failure report when it went wrong. Fenced per attempt instead.
- A result from one worker could commit another's task, after which the owner's
real delivery arrived as a duplicate and its audio was discarded. "Unknown
attempt" and "another worker's attempt" are no longer the same answer.
- An oversized result was a poison pill, re-sent identically on every reconnect
and permanently disconnecting the worker. It is now a terminal
`RESULT_TOO_LARGE`, which is also classified — it was falling through to
TRANSIENT and retrying a re-render that could never fit.
- `_store_inline` joined the artifact directory with worker-supplied ids, and
`os.path.join` discards its prefix on an absolute component. Paths are now
minted control-plane-side and resolved through `core.path_security`.
- Remote synthesis bypassed `mark_synthetic`, and the guard that exists to
catch exactly that walked only `backend/api` and `backend/services` — so it
stayed green while a fourth unmarked producer shipped. Marking moved to the
worker's tensor stage; the guard now walks `backend/worker` too.
Also adds pre-rendered voice previews (`services/gallery.py`), so browsing the
gallery no longer needs a GPU or a downloaded model. The manifest is verified
against the updater's release key already baked into the binary; a fresh
install hears voices without downloading 2.4GB first, and everything falls back
to local rendering when the gallery is unreachable.
Verified on hardware, not just in CI: 1728 characters submitted to an RTX 4090
returned 105.94s of 24kHz audio in 23.9s, committed and served from the
artifact store.
Not yet done, and deliberately not claimed: the keepalive fix cannot be
exercised end-to-end on fast hardware, because any job long enough to reach the
120s lease produces audio past the 8MiB inline cap. Chunked `UploadResult` has
to land first. Pinning to the worker the user chose is also still absent, so
"Remote" reaches a remote GPU but not necessarily the one on the badge.
Palash Debnath