Five workstreams that finish the remote-GPU line, plus the test hole that
let a broken signature reach a commit.
**Downloads go through the normal path** (Phase 5). Rather than a second
remote-only route, the existing Models install flow became target-aware,
so a model landing on a worker uses the same code, the same progress
events and the same UI as a local one. Progress rows key on
(target, repo_id) — the aggregator keyed on bare repo_id, so the same
model downloading here and on a worker at once collapsed into one row
that told the user nothing true about either.
**Audiobooks render chapter by chapter on the worker** (Phase 8), with
per-chapter local fallback and ONE aggregated notice. The failure that
shape exists to prevent: a remote GPU that sleeps at chapter 40 of 200
must not turn a working book into 160 rows of PROGRESS_LEASE_EXPIRED.
Dictation is deliberately NOT ported — it runs ASR per utterance inside a
live WebSocket loop, and paying queue admission plus a round trip there
would spend the one thing that route is for.
**Dubbing stays local, and says so** (Phase 7). The coarse worker
operation is not finished, so the picker still reports dubbing as local
rather than showing a green remote chip over work this machine is doing.
What could not wait is the in-loop OOM retry: it sniffed the error string
and flushed the *local* CUDA cache, which under remote execution is the
wrong machine's GPU entirely. That is fixed now, before the path that
would have exercised it exists.
**Two instances can no longer share the control plane.** A second
VoiceStudio silently bound the same worker port and coexisted, so remote
workers landed on whichever process won the race — a session that
registers with one instance and appears dead to the other. This produced
hours of misdiagnosis during hardware testing and would hit any user with
the app open twice. The second instance now keeps running locally and
explains the conflict instead of quietly competing.
**And the hole that allowed all this to be missable.** gpu_gateway called
Scheduler.submit(pinned_worker_id=...) one commit before that parameter
existed. Every remote generation raised TypeError; 5236 tests passed
anyway, because nothing exercised the gateway against the real scheduler.
tests/test_gpu_gateway_scheduler_contract.py now runs that path for real
and binds every gateway→dependency call signature. Verified by renaming
the parameter away and watching both tests fail with the original error.
Gallery previews also fall back to a local render when a downloaded clip
cannot be decoded, rather than yielding silence.
Backend 5274 passed, frontend 1812 passed.
Not yet verified on hardware: Phases 4, 5, 6, 7, 8. Only the TTS path and
its artifact transport have been proven on a real GPU.
Three phases that only make sense together: a job that names a worker,
a worker that reports honestly what it can actually run, and the small
defects that made both lie.
**Pinning** (Phase 1). `pinned_worker_id` is now honoured in both places
that choose a worker — `eligible_workers` and `select_worker` build
independent lists, so applying it to one silently leaked work onto
whichever machine was least busy. The pin persists across a restart via
an additive column, deliberately not alembic (justified in the code, per
the precedent already in db.py): quitting mid-render used to drop it
without a word. `max_attempts=1` was rejected as the mechanism — it makes
the FIRST failure terminal, including the penalty-free ones a stale
advisory view produces routinely.
Cancel now actually reaches the worker. `WorkerServicer.cancel` had zero
callers, so cancelling released the slot while the GPU thread kept
running, and a late result could resurrect the task as COMPLETED —
`commit_result` assigned that state directly, bypassing the transition
table where CANCELLED is terminal by construction.
**Honest capabilities** (Phase 4). A worker now probes whether weights
are actually present, and a job stops BEFORE dispatch with a typed 409
naming the model and the machine, instead of failing mid-task. The probe
fails OPEN: `is_cached`/`cache_is_complete` cannot see a user-managed
clone outside the HF layout, so only a positive "absent" refuses.
Refusing an engine that works today would break the compatibility
promise. `pool.supports` deliberately still ignores `downloaded` — had it
not, the scheduler would drop the worker and answer with a terminal
NO_CAPABLE_WORKER, which tells the user to check their install when the
truth is one download away. The frontend no longer offers "Report this
bug" for that state; it offers the download.
Catalog tags resolve against the TARGET's OS/arch/backend, not this
machine's. From a Mac control plane, a CUDA worker's model list was
showing the mlx-community repos it cannot run and hiding the ones it
needs.
**And the quiet ones** (Phase 0 leftovers): a model's human label rides
its own proto field so renaming it cannot orphan breaker history; an
empty model_id no longer forks the capacity slot key into two slots for
one model; the idle sweep cannot evict an engine out from under a live
LOCAL render.
Verified on real hardware, which is the only verification that has ever
caught anything here: 2025 characters, default settings, routed to an
RTX 4090 over the wire — 100% GPU utilisation on the remote box, 119.6 s
of 24 kHz audio returned in 16.6 s, 5.7 MB delivered out of band through
the artifact path rather than the control stream.
Backend 5259 passed, frontend 1808 passed.
Palash Debnath