Rendering 1126 previews ran one clip at a time, and only the first of a
clip's five stages is on the GPU: render, then watermark embed, MP3 encode,
decode, and detection. The card idled through four CPU stages per clip.
The embed and detection are neural forward passes that ran inline on the
event loop, so they held it for the whole clip -- concurrency would have
queued behind a busy loop and bought nothing. They now go through
asyncio.to_thread, which is what makes threads the right tool here: torch
releases the GIL inside those passes, so there is no second model copy and no
IPC for the tensors. Clips then build --jobs at a time (4 by default, 1
restores the old serial behaviour) under a semaphore, because every clip in
flight holds decoded audio.
A lost watermark still stops the entire run rather than only its own clip.
--resume now also adopts MP3s already on disk. The manifest is written once,
at the end, so a run interrupted at clip 900 left 900 correct files that
--resume could not see and re-rendered every one of them. Everything an entry
needs -- sha256, byte length, duration, featured flag -- is recoverable from
the file and the catalog, so recover it.
Also add a watermark preflight. Every clip was already verified individually,
but only after the first full render, and the message blamed the bitrate when
the cause can be unrelated to audio: on a host without python3-dev, AudioSeal's
forward pass dies inside Inductor, embed_watermark catches it, and the clip is
returned unmarked. Two seconds up front, with the actual cause named. It also
warms the lazy generator/detector globals single-threaded, before --jobs fans
out.
Selecting a remote worker repainted a badge and nothing else. The cause was
not subtle: `scheduler.submit` had no production caller, and `routing.decide()`
was read only by the status endpoint that paints the header. Remote execution
was a complete, tested pipeline with no producer at its head.
This adds the producer and fixes the defects that made the pipeline unable to
carry a real job:
- Nothing routed to the scheduler. Adds `POST /workers/tasks` (loopback-gated,
**development-only** until the gateway lands) and `Scheduler.wait`, backed by
per-task futures rather than the unregisterable `on_change` listener list.
- Every task over two minutes died. No worker ever sent `TaskProgress`, so the
120s progress lease expired mid-render — including during the cold model
load, which happens after `TaskStarted`. Workers now report progress and
emit a keepalive, bounded by the phase's absolute budget so it renews the
lease without deleting the only enforced bound in the system.
- The executor rebuilt its engine per task (`return cls()`), so every job paid
a cold load. Engines now share one instance cache with the router, resolved
by the assignment's engine — never `get_active_tts_backend()`, which returns
the worker machine's own Settings preference and would silently run the
wrong engine.
- One lease expiry took a worker offline permanently: parked slots were never
reclaimed. Parks now expire on a TTL, and are deliberately NOT reconciled
against the worker's own load report — at a ceiling of one the only task such
a worker can report is the wedged one, so "busy" would drop the park and the
next idle heartbeat would hand out a slot with a live GPU thread (#730/#1190).
- A worker that dropped and reconnected mid-render had every liveness frame
discarded: task frames were fenced on the live session epoch, which bumps on
every reconnect, while the worker echoes the ref stamped at dispatch. The
control plane then expired a task whose GPU was still rendering, and swallowed
the failure report when it went wrong. Fenced per attempt instead.
- A result from one worker could commit another's task, after which the owner's
real delivery arrived as a duplicate and its audio was discarded. "Unknown
attempt" and "another worker's attempt" are no longer the same answer.
- An oversized result was a poison pill, re-sent identically on every reconnect
and permanently disconnecting the worker. It is now a terminal
`RESULT_TOO_LARGE`, which is also classified — it was falling through to
TRANSIENT and retrying a re-render that could never fit.
- `_store_inline` joined the artifact directory with worker-supplied ids, and
`os.path.join` discards its prefix on an absolute component. Paths are now
minted control-plane-side and resolved through `core.path_security`.
- Remote synthesis bypassed `mark_synthetic`, and the guard that exists to
catch exactly that walked only `backend/api` and `backend/services` — so it
stayed green while a fourth unmarked producer shipped. Marking moved to the
worker's tensor stage; the guard now walks `backend/worker` too.
Also adds pre-rendered voice previews (`services/gallery.py`), so browsing the
gallery no longer needs a GPU or a downloaded model. The manifest is verified
against the updater's release key already baked into the binary; a fresh
install hears voices without downloading 2.4GB first, and everything falls back
to local rendering when the gallery is unreachable.
Verified on hardware, not just in CI: 1728 characters submitted to an RTX 4090
returned 105.94s of 24kHz audio in 23.9s, committed and served from the
artifact store.
Not yet done, and deliberately not claimed: the keepalive fix cannot be
exercised end-to-end on fast hardware, because any job long enough to reach the
120s lease produces audio past the 8MiB inline cap. Chunked `UploadResult` has
to land first. Pinning to the worker the user chose is also still absent, so
"Remote" reaches a remote GPU but not necessarily the one on the badge.
velixio