A cold engine emits no execution evidence while it loads weights and compiles,
so the Gateway's attempt lease expires mid-load, the attempt is fenced, and the
next attempt pays the same cost — a loop that never produces audio. Loading
every READY model before the socket accepts work moves that cost to startup,
where preflight already expects to wait, so the first Execute begins inference
immediately. A prewarm failure is reported rather than fatal, and --no-prewarm
restores the previous behavior.
Rendering 1126 previews ran one clip at a time, and only the first of a
clip's five stages is on the GPU: render, then watermark embed, MP3 encode,
decode, and detection. The card idled through four CPU stages per clip.
The embed and detection are neural forward passes that ran inline on the
event loop, so they held it for the whole clip -- concurrency would have
queued behind a busy loop and bought nothing. They now go through
asyncio.to_thread, which is what makes threads the right tool here: torch
releases the GIL inside those passes, so there is no second model copy and no
IPC for the tensors. Clips then build --jobs at a time (4 by default, 1
restores the old serial behaviour) under a semaphore, because every clip in
flight holds decoded audio.
A lost watermark still stops the entire run rather than only its own clip.
--resume now also adopts MP3s already on disk. The manifest is written once,
at the end, so a run interrupted at clip 900 left 900 correct files that
--resume could not see and re-rendered every one of them. Everything an entry
needs -- sha256, byte length, duration, featured flag -- is recoverable from
the file and the catalog, so recover it.
Also add a watermark preflight. Every clip was already verified individually,
but only after the first full render, and the message blamed the bitrate when
the cause can be unrelated to audio: on a host without python3-dev, AudioSeal's
forward pass dies inside Inductor, embed_watermark catches it, and the clip is
returned unmarked. Two seconds up front, with the actual cause named. It also
warms the lazy generator/detector globals single-threaded, before --jobs fans
out.
Fake-engine/fake-inventory tests over a real UDS gRPC server plus a fast
direct-executor path: health/capabilities shape and version identity, the
Go-preflight port passing with a READY model and failing closed without
one, only-READY-counts semantics, digest stability/sensitivity, execute
happy path (manifest checksum matches the written WAV), deadline
enforcement, cancel race with idempotent dispositions, slot exhaustion,
duplicate attempts, URL/relative handle rejection, checksum mismatch, and
the input/model-load/inference/GPU/storage failure classification.
RuntimeAdapterService over a private Unix-domain socket (default
/run/voicestudio/runtime.sock, VOICE_STUDIO_RUNTIME_SOCKET override; no
HTTP, no TCP, no database, no outbound network):
- Health/GetCapabilities read one RuntimeContext, so runtime/adapter
versions are identical across both calls by construction. Devices come
from torch (CUDA per-GPU / MPS / CPU with system RAM as capacity);
models come from the tts_backend engine registry + hf_revisions pinned
revisions, digest-pinned via a cached sha256 snapshot digest. READY is
explicit: probe passed, snapshot complete, digest computed — a
loading/installed/failed model is reported truthfully, never READY.
- Execute streams started -> bounded progress -> exactly one terminal
event, validates attempt identity, approved model digest, typed bounded
parameters, and LOCAL absolute-path handles (URL-shaped handles are
invalid input, never fetched), runs the engine on a worker thread,
enforces the request deadline, and writes the output WAV atomically
with a size/sha256/duration manifest plus raw measurements.
- Stable RTA_* failure codes map onto RuntimeFailureClass: input,
model-load, inference, GPU-resource, local-storage, canceled, crash.
- Cancel is idempotent by attempt id (ACCEPTED / ALREADY_TERMINAL /
NOT_FOUND) against a bounded attempt registry.
- python -m backend.runtime_adapter serves; --selfcheck starts a temp
socket and runs a port of internal/gateway/preflight.go's checks
against itself (verified passing on this host: 1 device, 2 ready
digest-pinned models).
Vendor api/proto/voicestudio/runtime/v1/runtime_adapter.proto from vssaas
byte-identically into backend/runtime_adapter/, generate the grpcio stubs
into gen/ (committed, same policy and import fixup as
backend/worker/protocol/gen/), and add the drift test that regenerates
into a tmpdir and diffs.
After the ordering fix, unloading the model on a 4090 still left the GPU at
1238 MiB with torch reporting 8.5 MB allocated and 803 MB reserved -- and no
number of Flush Memory presses moved it. A segment dump said why: ONE 803 MB
segment, 794.7 MB of it inactive-but-split, pinned by a single live block of
8,519,680 bytes.
That is cuBLAS's default workspace. It is taken from the caching allocator on
first use, so it lands inside whatever segment the model load had just grown,
and it is held for the life of the cuBLAS handle. empty_cache() can only
return segments that are entirely free, so one 8.5 MB block kept three
quarters of a gigabyte from ever reaching the driver again. On a machine
lending its GPU that is the difference between an idle node costing 470 MiB
and costing 1.2 GB.
free_vram() now clears the workspaces before emptying the cache, on the
unload paths only -- the next cuBLAS call re-takes one, which is cheap but
not something to pay per generate. The binding is private
(torch._C._cuda_clearCublasWorkspaces), so it is optional by construction: a
build without it keeps today's behaviour rather than failing an unload.
Found by adding reserved-vs-allocated to /system/flush-memory in 642513d2.
Allocated alone reads near zero after an unload, which is exactly why this
hid for so long -- every diagnostic we had agreed the memory was free.
memory_allocated counts live tensors only, so after an unload it reads
near zero while nvidia-smi still shows gigabytes. That gap is the whole
substance of every "flush says it worked, the GPU says it didn't" report,
and the endpoint was reporting only the half that looks good.
memory_reserved is what the caching allocator holds from the driver; the
remainder between that and the driver's own figure is the CUDA context and
kernel workspaces, which nothing in-process can hand back.
The shared voice model's unload emptied the allocator caches and *then*
dropped the reference. That frees nothing: the weights are still reachable
when gc.collect() runs, empty_cache() only returns blocks the allocator
already considered free, and the reference drops a moment later into a cache
nothing will flush again. The unload logs success, the engine leaves the
registry, and nvidia-smi does not move.
Six modules open-coded the same two lines. Exactly one had them inverted --
OmniVoiceBackend.unload, which is the path the engine-registry idle sweep
reaches, which is the sweep a headless worker node runs. So every unload a
user could trigger from the UI worked, and the one that runs unattended on a
machine lending its GPU held 3.6 GB indefinitely. Found on hardware: the
sweep fired on schedule, logged "Released 1 idle engine(s)", and VRAM stayed
flat at 3656 MiB for the next two minutes.
Replace all six with model_manager.unload_shared_model(), which clears the
reference, drops the clone-prompt side cache, then frees -- in that order,
in one place. Two callers gain the side-cache drop they were missing
(/system/flush-memory and the shutdown path), which is the same defect one
step down: an unload that kept the encoded reference tensors belonging to the
model it had just released.
A source guard asserts nothing outside model_manager assigns the shared
reference, so the next caller cannot reintroduce the ordering. It caught the
sixth site while being written.
Also give the AudioSeal watermark models the bargain every other model in the
app already makes: they loaded on the first embed and stayed resident for the
life of the process. CPU-resident, so this is system RAM rather than VRAM,
and the machines that notice are the ones running batches.
The error text on a failing unload changes with the ordering. "Could not be
unloaded, retry after the current generation finishes" was accurate when the
cache flush ran first and aborted before the release; now the release has
already happened and only the flush can fail, so it says that instead of
sending the user to repeat work that is done.
The startup preload exists so the first generate feels instant for the person
sitting in front of the app. A machine lending its GPU has nobody sitting
there, so it was several GB of VRAM held from boot against a request that may
never arrive — and the idle sweep could not reclaim it, because the sweep owns
the worker executor's engines while this is the default local model.
Measured on gpu2: a node that had run nothing still sat at 2.4 GB, and an idle
unload after a real job returned it to exactly that floor rather than below it.
Worker-mode processes now load on first request and release when idle, which is
what a node should do. A machine that is both a desktop app and a worker keeps
the warm-up — there is a real user there and the point stands.
Watching a ten-minute rule take effect means waiting ten minutes, so it tends
not to get watched. Both numbers are now env-tunable:
OMNIVOICE_ENGINE_IDLE_UNLOAD_SECONDS and OMNIVOICE_IDLE_SWEEP_SECONDS.
They are documented as a pair, because shortening only the threshold still
means waiting a full sweep interval to see it fire — which reads as a broken
sweep and sends you looking for a bug that is not there.
Unparseable values and anything below the floor are ignored with a warning
rather than honoured. A zero threshold would hand back a model the instant it
went idle and reload it for the very next request, which is worse than the
behaviour being tuned.
The ten-minute idle sweep lived inside the dial-out agent. A node that only
accepts inbound connections never starts that agent — on gpu2 it fails outright
with 'Set OMNIVOICE_WORKER_ENDPOINT' — so a machine lending its GPU to panels
that dial IN held several GB of weights forever. That is precisely the cost the
sweep exists to avoid, and it was silently missing in the mode most likely to
be a shared box.
The loop moves to module scope and both transports use it. Inbound starts it
when the listener starts and cancels it when the listener stops, and passes a
callback that re-advertises capabilities to every attached panel, so a control
plane's view of what is resident does not go stale the moment it becomes
useful. Local behaviour is unchanged: nothing sweeps unless a worker role runs.
Found by clicking Synthesize in the desktop UI — the one path nothing had
exercised.
task_store.stage_input mints inputs/<digest><ext>, a path rather than a bare
name. The node ran safe_filename over it, which rejects anything nested, so
every real clone input was refused, the dispatch failed, and the scheduler
retried about eighteen times a second while the 4090 sat idle and the user
watched a spinner.
The wire id is now hashed into a directory name rather than used as one. That
accepts any id the protocol allows while leaving placement entirely ours to
decide, which is the property the check was really buying. The declared
filename is still required to be a bare name, and a hostile one is still
refused outright — covered by its own test so the containment cannot be traded
away later to fix some future rejection.
Every earlier test used a flat id like 'ref-1' and so never met the shape
production emits.
Found on hardware. The queue was built once per connection and reused across
reconnects, so a frame a dying session left behind became the FIRST frame of
the next attach. The node requires a registration there, aborted the call, and
the two span at full speed — session epoch 2445 inside one second, the node
logging 'Locally aborted' on repeat, and the panel reporting the machine
offline while the connection list showed it connected.
Two more found on hardware.
Disconnect ended the session and the panel redialled two seconds later, so the
log read disconnected and connected in the same breath and the button appeared
to do nothing. A kicked key now sits out for a minute — long enough that the
disconnect is real and the person notices, short enough that it is plainly not
a revocation, which stays a separate and permanent action. The docs now say
which of the two buttons does which.
Re-pasting a connection string for an already-connected machine saved the new
string and then short-circuited on the existing session, so a wrong key
reported success, kept running on the old connection, and only failed after a
restart — by which point nothing pointed back at the paste that caused it. The
live session is now torn down before the new one is dialled.
Found on hardware. FetchResult seeked to request.size_bytes as though it were
a resume point, but that field is the artifact's total size — so every fetch
started at end-of-file, yielded no chunks, and failed with 'the result ended
before its final chunk' while the finished render sat on the node's disk.
ArtifactRef carries no resume field, so resumption is a protocol addition
rather than a reinterpreted one, and the fetch now always starts at zero.
Every earlier test drove publish and stage directly and never called
FetchResult with a populated ref, which is exactly why this survived them.
Found on hardware. The job ran on the GPU machine and the audio never arrived:
'gpu2 finished the job but its audio did not arrive.'
Both artifact directions were built and neither was wired. A result reported by
a dialled node is only staged on that node's disk — nothing pushes it, because
the node cannot call us — so the commit recorded an artifact path that had
never been written. Inputs had the mirror problem: nothing sent them, so a
clone would have failed on a reference file that was never delivered.
Results are now pulled when the frame naming them arrives, and inputs are
pushed before the assignment rather than alongside it, because the executor
asks for them as soon as it starts and an assignment that overtakes its own
reference audio fails on a file that is merely late.
A fetch that fails is not a silent loss: no artifact is recorded, the task
fails naming the machine, and the node keeps its copy because nothing
acknowledges a result we could not fetch.
Found on hardware. The Attach handler started the read pump and the outbound
loop but never the heartbeat loop that the outbound path starts inside
_connect_once. So a node registered, went silent, was declared dead about
ninety seconds later, reconnected, and flapped forever — and in between, work
aimed at it fell back to the local machine with 'gpu2 is offline', while the
panel had shown it ready at 3.4 ms moments earlier.
Every end-to-end test in this file finished inside three seconds, comfortably
within the grace window that hid it. The regression test therefore asserts on
the emitted heartbeat frames themselves rather than on liveness, and shortens
the advertised interval so it does that in two seconds instead of twenty.
Found on hardware. With the listener bound to 0.0.0.0 — which is what sharing
a GPU across a network requires — the issued string came out as
ovnode://...@0.0.0.0:7444. That is a legal bind and a meaningless destination,
so it would have failed on the far end with a connection error naming nothing,
and the person who pasted it had no way to tell a bad string from a firewall.
The string is now built from an advertised address rather than the bind: for a
wildcard bind, the source address the routing table would use to leave this
machine, found with a connected UDP socket that sends no packets and needs no
DNS. An explicitly typed bind is advertised verbatim, because someone who
entered a specific address meant it.
Adds the panel that makes inbound mode usable: a toggle to accept connections,
a bind field that says which side of "only this machine" you are on, per-person
connection strings with a copy button, the live list of who is connected with a
disconnect button, and a paste box for joining someone else's GPU.
Placed behind the existing Remote workers toggle rather than beside it. "Off
means off" is this feature's stated contract, and a second switch that stayed
live underneath would be exactly the surprise that promise exists to prevent.
Headless machines that only lend a GPU set OMNIVOICE_INBOUND_NODE and never see
this panel.
The unencrypted warning appears where it becomes true, not buried in a doc:
next to the bind field once it points beyond this machine, naming the address,
and again under every freshly issued connection string. The remove-access
confirm says the others stay connected, since that is the only place a user
learns keys are per person rather than one switch for everybody.
All 36 strings are translated into all 20 non-English locales in this change,
with the {{address}}, {{label}}, {{count}} and {{when}} placeholders verified
programmatically against en.json before writing — a dropped token is the exact
bug the parity test was built for, and en-only keys would have passed CI
silently while every other language read English.
Two existing WorkersPanel rename tests queried the only textbox on the page.
That was incidental, not intentional; they now name the field they mean.
Wires the two transport halves into something a user can actually turn on.
Two independent switches, deliberately not one. "Accept connections" makes this
machine a node others dial; "saved connections" are the nodes this panel dials
out to. A workstation with a GPU that also drives jobs on a second box does
both, so neither implies the other.
Binding stays on 127.0.0.1 until someone explicitly widens it, and widening is
its own field rather than a flag riding along with the enable toggle. With no
encryption that boundary is the difference between a credential on one machine
and a credential on a network, so it is never crossed as a side effect. The
API reports `exposed` so the UI can say which side of it the user is on.
Saved nodes are redialled only after the control plane is up, since the
connector hands frames to its servicer. Failing to listen records the reason
rather than leaving the feature looking enabled while it quietly accepts
nothing.
Docs say plainly that this mode is unencrypted, that the connection string is a
password crossing the network in the clear, and that dial-out remains the
better choice when one machine is enough. The Security section no longer
implies its TLS guarantees cover both modes.
Completes the inbound path. The panel opens NodeService.Attach with its key in
call metadata, answers the node's register frame, and then runs the ordinary
control-plane loops against the dialled stream — the same _read_loop and
_ping_loop the outbound path uses, so assignments, cancels, results and
reconciliation all behave identically. Only who opened the socket changed.
Registration is shared rather than copied: the body of Register is now
establish_session, reached from both roads. A second copy of session issue,
capability application and in-flight reconciliation is a second thing to keep
in step forever, and the half that gets forgotten is always reconciliation.
The version and feature gates run on the inbound road too — skipping them would
let an out-of-date node register cleanly and then ignore task inputs, which is
how a clone with no reference audio once came back reported as success.
Artifacts invert with the transport: the panel pushes inputs before it assigns,
and pulls results after. Both directions verify the declared sha256 and refuse
a stream that ends without its final chunk, because a truncated file renamed
into place and called done is the failure the upload path was already hardened
against.
Two things the end-to-end tests found, neither visible from unit tests:
* Every Attach built a fresh client with an empty worker id, so the challenge
signature could never match after first enrollment — inbound could connect
once and never reconnect. The id is now kept per panel key, because each
panel keeps its own registry and the same machine is a different worker id
to each of them.
* A node that has lost the id a panel gave it could prove possession of its
key and still be refused forever, with no way back except deleting it from
both sides. It is now re-adopted on proof of key possession, narrowly: the
public key must already be the one enrolled, so this can never admit a new
key. Covered by a test that forges a valid self-signature from a different
keypair and asserts it is refused.
Remote workers connect outbound: the node dials the control plane, spends an
enrollment token, pins a certificate. That stays the default and is unchanged.
It is also structurally 1:1 — a worker process holds one endpoint, one pinned
certificate and one worker id — so a second person wanting the same GPU box has
to get shell access to it, repoint the start script at their own address and
restart, which disconnects whoever was using it. Sharing a GPU requires root on
it and evicts the incumbent, and no amount of UI work fixes that, because the
constraint is the shape of the connection.
This adds the other arrangement: the node listens, and any panel holding a key
connects to it, concurrently, with no shell access to the machine.
* NodeService mirrors WorkerService. Transport roles invert; message roles do
not — the node still sends WorkerMessage and the panel still sends
ServerMessage, so every state machine on both sides is untouched. Register
folds into the stream as the first exchange and reuses the existing
request/response messages rather than growing parallel ones.
* Keys are per panel, not per node. Revoking one person leaves everyone else
connected; a shared key would be revoked by nobody and leave no record of
who used it. Stored hashed, compared in constant time against every key so
the reply time is not an oracle, and the plaintext exists exactly once.
* Failed authentication is throttled per source address, so one stale
bookmark cannot lock out a different panel.
* A connection log records every attach, refusal and disconnect, and any
session can be kicked. That is what replaces per-job approval, which would
make a shared GPU unusable and train people to click yes.
* Artifacts invert too: the panel pushes inputs before assigning, and fetches
results after. The node stages both under one contained directory and
trusts no id or filename off the wire.
Runs in plaintext by deliberate decision, recorded with its accepted risk in
docs/adr/inbound-node-mode.md, and scoped there to LAN and self-hosted use —
never a fleet transport, which goal_v2 B2/B5.2 still require to dial out.
Off by default, and bound to 127.0.0.1 until someone explicitly widens it.
Three conflicts, all additive on both sides — resolved by keeping both
rather than choosing, since either side's entries were real shipped work:
* CHANGELOG.md — remote-GPU entries against branding, IndexTTS 2.5 and
the recording-input work
* setup/download.py — the per-target progress reset against main's
active-install tracking; both belong in the same finally block
* docs/features.yaml — the remote-worker and model docs against
docs/branding.md
Backend 5349 passed, frontend 1871 passed. `bun install --frozen-lockfile`
reports no changes, so the Docker build sees the same tree CI does.
The remote-GPU line, verified on hardware rather than asserted.
**Dubbing renders on the worker.** dub_generate.py dispatches the coarse
`dub_segments` operation through the gateway, following the audiobook
pattern: per-unit local fallback after consecutive remote failures, one
aggregated notice rather than one per segment. A 40-minute dub that loses
its worker at segment 200 degrades instead of producing 200 error rows.
**An out-of-date worker is now refused by name.** This was the worst
defect in the plan and it was silent: an un-upgraded worker registered
cleanly, then ignored `inputs` and rendered a clone with NO reference
audio — returned as success. A plausible wrong result with nothing
anywhere to surface it. Workers now declare features, and one missing
them is turned away with the features named and `no task was run`.
Verified live: a worker one commit behind was correctly refused.
**"Offline" and "cannot run this" are different facts.** Asking a live
worker for an engine it lacks answered "is offline or cannot be reached.
Wake the selected worker" — while that worker reported ready, one free
slot and 3.6 ms latency. The user was sent to wake a machine that was
already awake. The scheduler now distinguishes absent from present-but-
incapable, and names the engine rather than the operation, because the
engine is the thing a user can install.
**An engine with no catalog entry is no longer hidden.** A `repo_ids`
non-emptiness check had been implemented as a runtime filter, so a worker
silently refused to advertise any engine lacking a models.yaml entry —
which is four registered engines, including CosyVoice. Users with those
already installed would have lost remote support with only a log line.
Empty `repo_ids` now means "not downloadable here", never "not runnable".
**And a script so this stops being done by hand.**
scripts/verify-remote-worker.sh runs the per-phase acceptance checks
against a live worker, non-destructively. Its preconditions are the
mistakes that cost the most time: exactly one listener on the control
port (two instances silently shared it), and never detecting the worker
with a pgrep pattern that matches the ssh shell running it.
Its first real run found the dubbing picker claiming remote placement.
That turned out to be the CHECK being stale, not the picker — the port
had landed since it was written. It now asserts self-consistency instead:
the picker may claim remote only for an operation the control plane
actually advertises as remotely producible, which cannot rot the next
time an op is ported.
Backend 5291 passed, frontend 1812 passed. Acceptance script: no
automated failures across Phases 4-8 on an RTX 4090. Four checks remain
MANUAL by design — true airplane mode, concurrent downloads, killing a
worker mid-audiobook, and the model-list UI — and are reported as
unverified rather than passed.
Five workstreams that finish the remote-GPU line, plus the test hole that
let a broken signature reach a commit.
**Downloads go through the normal path** (Phase 5). Rather than a second
remote-only route, the existing Models install flow became target-aware,
so a model landing on a worker uses the same code, the same progress
events and the same UI as a local one. Progress rows key on
(target, repo_id) — the aggregator keyed on bare repo_id, so the same
model downloading here and on a worker at once collapsed into one row
that told the user nothing true about either.
**Audiobooks render chapter by chapter on the worker** (Phase 8), with
per-chapter local fallback and ONE aggregated notice. The failure that
shape exists to prevent: a remote GPU that sleeps at chapter 40 of 200
must not turn a working book into 160 rows of PROGRESS_LEASE_EXPIRED.
Dictation is deliberately NOT ported — it runs ASR per utterance inside a
live WebSocket loop, and paying queue admission plus a round trip there
would spend the one thing that route is for.
**Dubbing stays local, and says so** (Phase 7). The coarse worker
operation is not finished, so the picker still reports dubbing as local
rather than showing a green remote chip over work this machine is doing.
What could not wait is the in-loop OOM retry: it sniffed the error string
and flushed the *local* CUDA cache, which under remote execution is the
wrong machine's GPU entirely. That is fixed now, before the path that
would have exercised it exists.
**Two instances can no longer share the control plane.** A second
VoiceStudio silently bound the same worker port and coexisted, so remote
workers landed on whichever process won the race — a session that
registers with one instance and appears dead to the other. This produced
hours of misdiagnosis during hardware testing and would hit any user with
the app open twice. The second instance now keeps running locally and
explains the conflict instead of quietly competing.
**And the hole that allowed all this to be missable.** gpu_gateway called
Scheduler.submit(pinned_worker_id=...) one commit before that parameter
existed. Every remote generation raised TypeError; 5236 tests passed
anyway, because nothing exercised the gateway against the real scheduler.
tests/test_gpu_gateway_scheduler_contract.py now runs that path for real
and binds every gateway→dependency call signature. Verified by renaming
the parameter away and watching both tests fail with the original error.
Gallery previews also fall back to a local render when a downloaded clip
cannot be decoded, rather than yielding silence.
Backend 5274 passed, frontend 1812 passed.
Not yet verified on hardware: Phases 4, 5, 6, 7, 8. Only the TTS path and
its artifact transport have been proven on a real GPU.
Three phases that only make sense together: a job that names a worker,
a worker that reports honestly what it can actually run, and the small
defects that made both lie.
**Pinning** (Phase 1). `pinned_worker_id` is now honoured in both places
that choose a worker — `eligible_workers` and `select_worker` build
independent lists, so applying it to one silently leaked work onto
whichever machine was least busy. The pin persists across a restart via
an additive column, deliberately not alembic (justified in the code, per
the precedent already in db.py): quitting mid-render used to drop it
without a word. `max_attempts=1` was rejected as the mechanism — it makes
the FIRST failure terminal, including the penalty-free ones a stale
advisory view produces routinely.
Cancel now actually reaches the worker. `WorkerServicer.cancel` had zero
callers, so cancelling released the slot while the GPU thread kept
running, and a late result could resurrect the task as COMPLETED —
`commit_result` assigned that state directly, bypassing the transition
table where CANCELLED is terminal by construction.
**Honest capabilities** (Phase 4). A worker now probes whether weights
are actually present, and a job stops BEFORE dispatch with a typed 409
naming the model and the machine, instead of failing mid-task. The probe
fails OPEN: `is_cached`/`cache_is_complete` cannot see a user-managed
clone outside the HF layout, so only a positive "absent" refuses.
Refusing an engine that works today would break the compatibility
promise. `pool.supports` deliberately still ignores `downloaded` — had it
not, the scheduler would drop the worker and answer with a terminal
NO_CAPABLE_WORKER, which tells the user to check their install when the
truth is one download away. The frontend no longer offers "Report this
bug" for that state; it offers the download.
Catalog tags resolve against the TARGET's OS/arch/backend, not this
machine's. From a Mac control plane, a CUDA worker's model list was
showing the mlx-community repos it cannot run and hiding the ones it
needs.
**And the quiet ones** (Phase 0 leftovers): a model's human label rides
its own proto field so renaming it cannot orphan breaker history; an
empty model_id no longer forks the capacity slot key into two slots for
one model; the idle sweep cannot evict an engine out from under a live
LOCAL render.
Verified on real hardware, which is the only verification that has ever
caught anything here: 2025 characters, default settings, routed to an
RTX 4090 over the wire — 100% GPU utilisation on the remote box, 119.6 s
of 24 kHz audio returned in 16.6 s, 5.7 MB delivered out of band through
the artifact path rather than the control stream.
Backend 5259 passed, frontend 1808 passed.
Every enrolled worker sat at connected=False against a healthy control
plane, and the control-plane log showed no Register call arriving at all.
The worker's own log said only "connecting", then nothing.
The cause was on our side of the handshake. The client sends an HTTP/2
ping every 25 s to keep its long-lived Control RPC alive through NAT —
an interval the control plane itself configures. But the server kept
gRPC's default enforcement policy, which permits two idle pings and then
answers ENHANCE_YOUR_CALM:
GOAWAY received; Error code: 11; Debug Text: too_many_pings
So the control plane hung up on every worker for obeying the keepalive
the control plane asked for. Idle workers were hit hardest, because a
session with no traffic is exactly the case the ping exists to protect.
Fixed by accepting the interval this protocol configures: a 20 s minimum
still rate-limits an abusive peer, while removing the idle-ping count
ceiling stops a healthy session dying of its own liveness mechanism.
This is a whole-fleet fix, not a per-enrollment one.
Worth recording what this was NOT, because it looked exactly like it:
TLS pin-on-first-use was the obvious suspect, since a control plane that
regenerated its certificate on restart would strand every enrolled
worker with no useful error. Disproved — the live certificate
fingerprint and the pinned copy on the remote worker match exactly, and
the certificate survives restarts. Enrollment was never involved.
Verified live against a remote worker: the session now establishes where
previously nothing reached the server. It is not yet stable — it drops
after ~17 s and advertises zero engines — but that is a separate defect
being tracked on its own, and this fix is a prerequisite for reaching it.
Two phases of the remote-GPU plan, landing together because neither is
useful alone: on a 4090 any render long enough to exercise the progress
lease also outgrows the 8 MiB message cap, so a gateway that routes work
remotely without an artifact transport just moves where the failure
happens.
**The gateway** (`services/gpu_gateway.py`) is the single owner of GPU
calling, model status, downloads and model load, for both targets —
`prewarm`, `run`, `status`, `download`. prewarm and run stay separate
because collapsing them loses the two-phase load/generate budget split
(#1033/#1037) that the worker protocol already mirrors. Admission moves
in here too: the old `check_gpu_admission` call read *local* pool stats,
so under Remote it would 429 on local saturation while the remote GPU
sat idle.
**Artifacts** now move out of band above a negotiated threshold. Bytes
land in an attempt-scoped `.part` file, are verified against a declared
sha256, and are renamed into place only on an explicit last chunk — a
transfer that arrives short, reordered, or simply stops commits nothing.
A resume rehashes what is already on disk, or the digest would attest
only to the tail, which is the exact case a resume exists to protect.
Two failure modes found while verifying this, both fixed with
mutation-checked regressions:
* an oversized payload with no session (mid-reconnect, or a control
plane too old to serve UploadResult) has nowhere to go. It must not
enter `_pending` — an over-cap frame is re-sent on every reconnect,
killing the session each time and stranding every other task — but
it must stay retryable, unlike the size gate's TERMINAL verdict:
nothing about the render is wrong, only the route to it.
* the upload resume loop was bounded by "did the offset change", which
a receiver alternating between two byte counts satisfies forever.
The worker is single-slot by default, so that is not one lost upload
but the machine, doing nothing else, until someone restarts it.
Bounded by a round count instead.
The control stream is split into control and bulk queues so the
heartbeat this whole liveness model rests on cannot queue behind a
payload — `result_json` has no size cliff to catch it, and the next bulk
message added to the protocol would have reintroduced the stall
silently.
Live streaming stays on the control plane and now says so once per
socket: that route exists to put audio in the user's ear before the
sentence finishes, and paying queue admission plus a round trip per
utterance would spend the one thing it is for. Silence would have been
worse than the limit — the header badge would read "gpu2" while this
machine did all the work.
Backend 5236 passed, frontend 1807 passed. End-to-end verification on
real hardware has NOT been re-run since these changes; the CHANGELOG
claim for the Synthesize button waits on that.
Selecting a remote worker repainted a badge and nothing else. The cause was
not subtle: `scheduler.submit` had no production caller, and `routing.decide()`
was read only by the status endpoint that paints the header. Remote execution
was a complete, tested pipeline with no producer at its head.
This adds the producer and fixes the defects that made the pipeline unable to
carry a real job:
- Nothing routed to the scheduler. Adds `POST /workers/tasks` (loopback-gated,
**development-only** until the gateway lands) and `Scheduler.wait`, backed by
per-task futures rather than the unregisterable `on_change` listener list.
- Every task over two minutes died. No worker ever sent `TaskProgress`, so the
120s progress lease expired mid-render — including during the cold model
load, which happens after `TaskStarted`. Workers now report progress and
emit a keepalive, bounded by the phase's absolute budget so it renews the
lease without deleting the only enforced bound in the system.
- The executor rebuilt its engine per task (`return cls()`), so every job paid
a cold load. Engines now share one instance cache with the router, resolved
by the assignment's engine — never `get_active_tts_backend()`, which returns
the worker machine's own Settings preference and would silently run the
wrong engine.
- One lease expiry took a worker offline permanently: parked slots were never
reclaimed. Parks now expire on a TTL, and are deliberately NOT reconciled
against the worker's own load report — at a ceiling of one the only task such
a worker can report is the wedged one, so "busy" would drop the park and the
next idle heartbeat would hand out a slot with a live GPU thread (#730/#1190).
- A worker that dropped and reconnected mid-render had every liveness frame
discarded: task frames were fenced on the live session epoch, which bumps on
every reconnect, while the worker echoes the ref stamped at dispatch. The
control plane then expired a task whose GPU was still rendering, and swallowed
the failure report when it went wrong. Fenced per attempt instead.
- A result from one worker could commit another's task, after which the owner's
real delivery arrived as a duplicate and its audio was discarded. "Unknown
attempt" and "another worker's attempt" are no longer the same answer.
- An oversized result was a poison pill, re-sent identically on every reconnect
and permanently disconnecting the worker. It is now a terminal
`RESULT_TOO_LARGE`, which is also classified — it was falling through to
TRANSIENT and retrying a re-render that could never fit.
- `_store_inline` joined the artifact directory with worker-supplied ids, and
`os.path.join` discards its prefix on an absolute component. Paths are now
minted control-plane-side and resolved through `core.path_security`.
- Remote synthesis bypassed `mark_synthetic`, and the guard that exists to
catch exactly that walked only `backend/api` and `backend/services` — so it
stayed green while a fourth unmarked producer shipped. Marking moved to the
worker's tensor stage; the guard now walks `backend/worker` too.
Also adds pre-rendered voice previews (`services/gallery.py`), so browsing the
gallery no longer needs a GPU or a downloaded model. The manifest is verified
against the updater's release key already baked into the binary; a fresh
install hears voices without downloading 2.4GB first, and everything falls back
to local rendering when the gallery is unreachable.
Verified on hardware, not just in CI: 1728 characters submitted to an RTX 4090
returned 105.94s of 24kHz audio in 23.9s, committed and served from the
artifact store.
Not yet done, and deliberately not claimed: the keepalive fix cannot be
exercised end-to-end on fast hardware, because any job long enough to reach the
120s lease produces audio past the 8MiB inline cap. Chunked `UploadResult` has
to land first. Pinning to the worker the user chose is also still absent, so
"Remote" reaches a remote GPU but not necessarily the one on the badge.
Adds a GPU target picker to the header: Local, or one of the machines you
enrolled. Exactly one is active at a time; other connected workers are
standby and receive nothing.
The selection is the user's, not the scheduler's. The engine underneath can
rank many workers and the hosted platform will need that, but a desktop app
is better served by a choice you can predict and explain: "your worker is
offline, this ran locally" is a sentence, "least-busy ranking preferred the
laptop" is not. Picking an offline machine is allowed on purpose — you
choose your desktop, then go and switch it on.
`routing.decide()` is the single answer to "where does the next job run",
shared by the badge and (soon) the generation path, so the badge cannot
claim something the router will not do. It shows the RESOLVED answer rather
than the stored choice: pick your desktop, let it sleep, and the chip reads
Local with the reason, while the menu still shows your desktop selected.
Connection latency is now real. `latency_ms` existed but nothing measured
it — the protocol had Ping with no reply — so it was always zero. Adds Pong
(additive, field 12) and times the round trip on the control plane's
MONOTONIC clock, so an NTP step or a sleep/wake cannot produce a nonsense
reading, and no worker timestamp is trusted. Reported as a median of five
samples and withheld until a second sample exists: the first round trip
after connect lands while the worker is still importing torch, which
measured 139 ms on loopback and, averaged, carried that for a minute.
This is CONNECTION latency, not time-to-result. It is shown as information,
never as a routing input — RTT is milliseconds where inference is seconds,
so ranking on it would optimise noise.
Also fixes a bug the picker exposed: worker config was read from the pool,
which caches the row handed to it at connect time. Renaming a CONNECTED
worker updated the database and the API kept serving the old name until it
reconnected — same for priority and enable/disable. Config now comes from
the database and liveness from the pool, never the reverse, and writers
refresh the live copy so the scheduler's logs do not use a stale name.
Adds worker rename (the backend already supported it; no UI called it),
worker address as seen by the control plane rather than self-reported, and
ready/busy/offline status behind the header dot.
The Settings panel posted a JSON *string* with no content type, so FastAPI
refused every write with a 422 ("Input should be a valid dictionary"). It
also read `.enabled` straight off apiFetch's return value — but apiFetch
resolves to a raw Response, not parsed JSON, and does not throw on 4xx. So
the panel could never have shown a worker even once the 422 was fixed, and
no HTTP error ever reached a catch block.
All three now go through one request() helper: it sets the content type,
checks res.ok, parses, and raises FastAPI's `detail` so the user reads
"Remote workers are turned off." rather than a status code.
Why the tests missed it: they mocked apiFetch as if it returned parsed data,
so they agreed with the mock instead of the client. The mock now returns a
Response-shaped object, and the assertions check the wire shape — method,
Content-Type, parsed body — because a was-it-called assertion cannot see a
missing header.
Three endpoints had no test at all (/enabled, /resume, /tasks/{id}/cancel);
/enabled is the one that broke. All nine are covered now, including the
string-body 422 itself.
An enrollment token carries the endpoint a worker will dial, but
default_endpoint() read the CONFIGURED port rather than the bound one. Start
on any other port and every token points somewhere nothing is listening —
the worker retries forever against a dead address with backoff, so it looks
like a network problem rather than a wrong number.
Found by running the feature end to end on a non-default port, which is also
the second bug in this seam: the first was advertising a .local hostname
gRPC's resolver cannot resolve. Both were about what the token tells a
worker to dial, so both now have regression tests.
Remote workers was nested under Sharing, which reads backwards: everything
in Sharing is about letting something else reach THIS machine (a remote
backend, an MCP client, a share PIN), while remote workers sends work OUT
to machines you own. It is now its own System entry.
Docs-sync: every "Settings → Sharing → Remote workers" reference is
updated — the guide, the changelog, the two API error messages that tell a
user where to generate a token, and the agent's not-enrolled error.
Also ignores remote/ (local goal docs, review briefs, council reports) and
repoints the code comments that cited remote/goal_v2.md at the shipped
docs/remote-workers.md, so no committed file references a path that is not
in the repo.
Send individual jobs to GPUs on your other machines while everything else
stays local. Opt-in, off by default: with the toggle off there is no
listening socket, no certificate and no background loop.
Design follows remote/goal_v2.md, the council-revised goal doc. The
decisions that shaped the code, and why:
* A disconnect is an unknown outcome, not a failure. The original design
reassigned on disconnect while also describing the case where the worker
had already finished — following both guarantees duplicate execution. An
attempt now holds a grace window; a worker returning inside it commits
its result and no second attempt is ever made.
* At-least-once execution, exactly-once result commit. The result is
persisted BEFORE it is acknowledged, so a crash between the two cannot
silently lose a finished render.
* Deadlines are phased (accept -> model load -> execute -> deliver) and
liveness is a progress lease. The old fixed 30s execution budget was two
orders of magnitude below what this product actually does; silence is
the failure signal, not slowness.
* Capacity is derived from free VRAM, never configured: a static value
corrupts output under torch.compile thread affinity (#315) and aborts
the process on small cards (#567).
* A circuit breaker replaces the reliability-score/quarantine machinery,
which had no recovery path (no probation workload exists in a TTS
product) and penalised consumer networks for existing.
* Identity is a keypair the worker generates and never sends. A
server-assigned id is a name, not an authenticator, so revocation of one
would be theatre. Enrollment tokens are single-use and carry the control
plane's certificate fingerprint for pin-on-first-use.
Adds the domain core, scheduler, durable task store, gRPC transport,
worker agent, management API, Settings panel, and docs. Protobuf reserves
the tenant/trace/usage fields a hosted control plane would need, since
adding them later means upgrading a whole fleet.
Includes tests for the failure paths that matter: duplicate delivery,
stale-session fencing, reconnect reconciliation, grace expiry, breaker
attribution, and a real end-to-end TLS round trip.
velixio