The remote-GPU line, verified on hardware rather than asserted.
**Dubbing renders on the worker.** dub_generate.py dispatches the coarse
`dub_segments` operation through the gateway, following the audiobook
pattern: per-unit local fallback after consecutive remote failures, one
aggregated notice rather than one per segment. A 40-minute dub that loses
its worker at segment 200 degrades instead of producing 200 error rows.
**An out-of-date worker is now refused by name.** This was the worst
defect in the plan and it was silent: an un-upgraded worker registered
cleanly, then ignored `inputs` and rendered a clone with NO reference
audio — returned as success. A plausible wrong result with nothing
anywhere to surface it. Workers now declare features, and one missing
them is turned away with the features named and `no task was run`.
Verified live: a worker one commit behind was correctly refused.
**"Offline" and "cannot run this" are different facts.** Asking a live
worker for an engine it lacks answered "is offline or cannot be reached.
Wake the selected worker" — while that worker reported ready, one free
slot and 3.6 ms latency. The user was sent to wake a machine that was
already awake. The scheduler now distinguishes absent from present-but-
incapable, and names the engine rather than the operation, because the
engine is the thing a user can install.
**An engine with no catalog entry is no longer hidden.** A `repo_ids`
non-emptiness check had been implemented as a runtime filter, so a worker
silently refused to advertise any engine lacking a models.yaml entry —
which is four registered engines, including CosyVoice. Users with those
already installed would have lost remote support with only a log line.
Empty `repo_ids` now means "not downloadable here", never "not runnable".
**And a script so this stops being done by hand.**
scripts/verify-remote-worker.sh runs the per-phase acceptance checks
against a live worker, non-destructively. Its preconditions are the
mistakes that cost the most time: exactly one listener on the control
port (two instances silently shared it), and never detecting the worker
with a pgrep pattern that matches the ssh shell running it.
Its first real run found the dubbing picker claiming remote placement.
That turned out to be the CHECK being stale, not the picker — the port
had landed since it was written. It now asserts self-consistency instead:
the picker may claim remote only for an operation the control plane
actually advertises as remotely producible, which cannot rot the next
time an op is ported.
Backend 5291 passed, frontend 1812 passed. Acceptance script: no
automated failures across Phases 4-8 on an RTX 4090. Four checks remain
MANUAL by design — true airplane mode, concurrent downloads, killing a
worker mid-audiobook, and the model-list UI — and are reported as
unverified rather than passed.
Five workstreams that finish the remote-GPU line, plus the test hole that
let a broken signature reach a commit.
**Downloads go through the normal path** (Phase 5). Rather than a second
remote-only route, the existing Models install flow became target-aware,
so a model landing on a worker uses the same code, the same progress
events and the same UI as a local one. Progress rows key on
(target, repo_id) — the aggregator keyed on bare repo_id, so the same
model downloading here and on a worker at once collapsed into one row
that told the user nothing true about either.
**Audiobooks render chapter by chapter on the worker** (Phase 8), with
per-chapter local fallback and ONE aggregated notice. The failure that
shape exists to prevent: a remote GPU that sleeps at chapter 40 of 200
must not turn a working book into 160 rows of PROGRESS_LEASE_EXPIRED.
Dictation is deliberately NOT ported — it runs ASR per utterance inside a
live WebSocket loop, and paying queue admission plus a round trip there
would spend the one thing that route is for.
**Dubbing stays local, and says so** (Phase 7). The coarse worker
operation is not finished, so the picker still reports dubbing as local
rather than showing a green remote chip over work this machine is doing.
What could not wait is the in-loop OOM retry: it sniffed the error string
and flushed the *local* CUDA cache, which under remote execution is the
wrong machine's GPU entirely. That is fixed now, before the path that
would have exercised it exists.
**Two instances can no longer share the control plane.** A second
VoiceStudio silently bound the same worker port and coexisted, so remote
workers landed on whichever process won the race — a session that
registers with one instance and appears dead to the other. This produced
hours of misdiagnosis during hardware testing and would hit any user with
the app open twice. The second instance now keeps running locally and
explains the conflict instead of quietly competing.
**And the hole that allowed all this to be missable.** gpu_gateway called
Scheduler.submit(pinned_worker_id=...) one commit before that parameter
existed. Every remote generation raised TypeError; 5236 tests passed
anyway, because nothing exercised the gateway against the real scheduler.
tests/test_gpu_gateway_scheduler_contract.py now runs that path for real
and binds every gateway→dependency call signature. Verified by renaming
the parameter away and watching both tests fail with the original error.
Gallery previews also fall back to a local render when a downloaded clip
cannot be decoded, rather than yielding silence.
Backend 5274 passed, frontend 1812 passed.
Not yet verified on hardware: Phases 4, 5, 6, 7, 8. Only the TTS path and
its artifact transport have been proven on a real GPU.
Two phases of the remote-GPU plan, landing together because neither is
useful alone: on a 4090 any render long enough to exercise the progress
lease also outgrows the 8 MiB message cap, so a gateway that routes work
remotely without an artifact transport just moves where the failure
happens.
**The gateway** (`services/gpu_gateway.py`) is the single owner of GPU
calling, model status, downloads and model load, for both targets —
`prewarm`, `run`, `status`, `download`. prewarm and run stay separate
because collapsing them loses the two-phase load/generate budget split
(#1033/#1037) that the worker protocol already mirrors. Admission moves
in here too: the old `check_gpu_admission` call read *local* pool stats,
so under Remote it would 429 on local saturation while the remote GPU
sat idle.
**Artifacts** now move out of band above a negotiated threshold. Bytes
land in an attempt-scoped `.part` file, are verified against a declared
sha256, and are renamed into place only on an explicit last chunk — a
transfer that arrives short, reordered, or simply stops commits nothing.
A resume rehashes what is already on disk, or the digest would attest
only to the tail, which is the exact case a resume exists to protect.
Two failure modes found while verifying this, both fixed with
mutation-checked regressions:
* an oversized payload with no session (mid-reconnect, or a control
plane too old to serve UploadResult) has nowhere to go. It must not
enter `_pending` — an over-cap frame is re-sent on every reconnect,
killing the session each time and stranding every other task — but
it must stay retryable, unlike the size gate's TERMINAL verdict:
nothing about the render is wrong, only the route to it.
* the upload resume loop was bounded by "did the offset change", which
a receiver alternating between two byte counts satisfies forever.
The worker is single-slot by default, so that is not one lost upload
but the machine, doing nothing else, until someone restarts it.
Bounded by a round count instead.
The control stream is split into control and bulk queues so the
heartbeat this whole liveness model rests on cannot queue behind a
payload — `result_json` has no size cliff to catch it, and the next bulk
message added to the protocol would have reintroduced the stall
silently.
Live streaming stays on the control plane and now says so once per
socket: that route exists to put audio in the user's ear before the
sentence finishes, and paying queue admission plus a round trip per
utterance would spend the one thing it is for. Silence would have been
worse than the limit — the header badge would read "gpu2" while this
machine did all the work.
Backend 5236 passed, frontend 1807 passed. End-to-end verification on
real hardware has NOT been re-run since these changes; the CHANGELOG
claim for the Synthesize button waits on that.
Selecting a remote worker repainted a badge and nothing else. The cause was
not subtle: `scheduler.submit` had no production caller, and `routing.decide()`
was read only by the status endpoint that paints the header. Remote execution
was a complete, tested pipeline with no producer at its head.
This adds the producer and fixes the defects that made the pipeline unable to
carry a real job:
- Nothing routed to the scheduler. Adds `POST /workers/tasks` (loopback-gated,
**development-only** until the gateway lands) and `Scheduler.wait`, backed by
per-task futures rather than the unregisterable `on_change` listener list.
- Every task over two minutes died. No worker ever sent `TaskProgress`, so the
120s progress lease expired mid-render — including during the cold model
load, which happens after `TaskStarted`. Workers now report progress and
emit a keepalive, bounded by the phase's absolute budget so it renews the
lease without deleting the only enforced bound in the system.
- The executor rebuilt its engine per task (`return cls()`), so every job paid
a cold load. Engines now share one instance cache with the router, resolved
by the assignment's engine — never `get_active_tts_backend()`, which returns
the worker machine's own Settings preference and would silently run the
wrong engine.
- One lease expiry took a worker offline permanently: parked slots were never
reclaimed. Parks now expire on a TTL, and are deliberately NOT reconciled
against the worker's own load report — at a ceiling of one the only task such
a worker can report is the wedged one, so "busy" would drop the park and the
next idle heartbeat would hand out a slot with a live GPU thread (#730/#1190).
- A worker that dropped and reconnected mid-render had every liveness frame
discarded: task frames were fenced on the live session epoch, which bumps on
every reconnect, while the worker echoes the ref stamped at dispatch. The
control plane then expired a task whose GPU was still rendering, and swallowed
the failure report when it went wrong. Fenced per attempt instead.
- A result from one worker could commit another's task, after which the owner's
real delivery arrived as a duplicate and its audio was discarded. "Unknown
attempt" and "another worker's attempt" are no longer the same answer.
- An oversized result was a poison pill, re-sent identically on every reconnect
and permanently disconnecting the worker. It is now a terminal
`RESULT_TOO_LARGE`, which is also classified — it was falling through to
TRANSIENT and retrying a re-render that could never fit.
- `_store_inline` joined the artifact directory with worker-supplied ids, and
`os.path.join` discards its prefix on an absolute component. Paths are now
minted control-plane-side and resolved through `core.path_security`.
- Remote synthesis bypassed `mark_synthetic`, and the guard that exists to
catch exactly that walked only `backend/api` and `backend/services` — so it
stayed green while a fourth unmarked producer shipped. Marking moved to the
worker's tensor stage; the guard now walks `backend/worker` too.
Also adds pre-rendered voice previews (`services/gallery.py`), so browsing the
gallery no longer needs a GPU or a downloaded model. The manifest is verified
against the updater's release key already baked into the binary; a fresh
install hears voices without downloading 2.4GB first, and everything falls back
to local rendering when the gallery is unreachable.
Verified on hardware, not just in CI: 1728 characters submitted to an RTX 4090
returned 105.94s of 24kHz audio in 23.9s, committed and served from the
artifact store.
Not yet done, and deliberately not claimed: the keepalive fix cannot be
exercised end-to-end on fast hardware, because any job long enough to reach the
120s lease produces audio past the 8MiB inline cap. Chunked `UploadResult` has
to land first. Pinning to the worker the user chose is also still absent, so
"Remote" reaches a remote GPU but not necessarily the one on the badge.
Send individual jobs to GPUs on your other machines while everything else
stays local. Opt-in, off by default: with the toggle off there is no
listening socket, no certificate and no background loop.
Design follows remote/goal_v2.md, the council-revised goal doc. The
decisions that shaped the code, and why:
* A disconnect is an unknown outcome, not a failure. The original design
reassigned on disconnect while also describing the case where the worker
had already finished — following both guarantees duplicate execution. An
attempt now holds a grace window; a worker returning inside it commits
its result and no second attempt is ever made.
* At-least-once execution, exactly-once result commit. The result is
persisted BEFORE it is acknowledged, so a crash between the two cannot
silently lose a finished render.
* Deadlines are phased (accept -> model load -> execute -> deliver) and
liveness is a progress lease. The old fixed 30s execution budget was two
orders of magnitude below what this product actually does; silence is
the failure signal, not slowness.
* Capacity is derived from free VRAM, never configured: a static value
corrupts output under torch.compile thread affinity (#315) and aborts
the process on small cards (#567).
* A circuit breaker replaces the reliability-score/quarantine machinery,
which had no recovery path (no probation workload exists in a TTS
product) and penalised consumer networks for existing.
* Identity is a keypair the worker generates and never sends. A
server-assigned id is a name, not an authenticator, so revocation of one
would be theatre. Enrollment tokens are single-use and carry the control
plane's certificate fingerprint for pin-on-first-use.
Adds the domain core, scheduler, durable task store, gRPC transport,
worker agent, management API, Settings panel, and docs. Protobuf reserves
the tenant/trace/usage fields a hosted control plane would need, since
adding them later means upgrading a whole fleet.
Includes tests for the failure paths that matter: duplicate delivery,
stale-session fencing, reconnect reconciliation, grace expiry, breaker
attribution, and a real end-to-end TLS round trip.
Palash Debnath