Commit Graph
2 Commits
Author SHA1 Message Date
velixio b7caa494eb feat(workers): remote downloads, audiobook chapters, and one port that stays honest
Five workstreams that finish the remote-GPU line, plus the test hole that
let a broken signature reach a commit.

**Downloads go through the normal path** (Phase 5). Rather than a second
remote-only route, the existing Models install flow became target-aware,
so a model landing on a worker uses the same code, the same progress
events and the same UI as a local one. Progress rows key on
(target, repo_id) — the aggregator keyed on bare repo_id, so the same
model downloading here and on a worker at once collapsed into one row
that told the user nothing true about either.

**Audiobooks render chapter by chapter on the worker** (Phase 8), with
per-chapter local fallback and ONE aggregated notice. The failure that
shape exists to prevent: a remote GPU that sleeps at chapter 40 of 200
must not turn a working book into 160 rows of PROGRESS_LEASE_EXPIRED.
Dictation is deliberately NOT ported — it runs ASR per utterance inside a
live WebSocket loop, and paying queue admission plus a round trip there
would spend the one thing that route is for.

**Dubbing stays local, and says so** (Phase 7). The coarse worker
operation is not finished, so the picker still reports dubbing as local
rather than showing a green remote chip over work this machine is doing.
What could not wait is the in-loop OOM retry: it sniffed the error string
and flushed the *local* CUDA cache, which under remote execution is the
wrong machine's GPU entirely. That is fixed now, before the path that
would have exercised it exists.

**Two instances can no longer share the control plane.** A second
VoiceStudio silently bound the same worker port and coexisted, so remote
workers landed on whichever process won the race — a session that
registers with one instance and appears dead to the other. This produced
hours of misdiagnosis during hardware testing and would hit any user with
the app open twice. The second instance now keeps running locally and
explains the conflict instead of quietly competing.

**And the hole that allowed all this to be missable.** gpu_gateway called
Scheduler.submit(pinned_worker_id=...) one commit before that parameter
existed. Every remote generation raised TypeError; 5236 tests passed
anyway, because nothing exercised the gateway against the real scheduler.
tests/test_gpu_gateway_scheduler_contract.py now runs that path for real
and binds every gateway→dependency call signature. Verified by renaming
the parameter away and watching both tests fail with the original error.

Gallery previews also fall back to a local render when a downloaded clip
cannot be decoded, rather than yielding silence.

Backend 5274 passed, frontend 1812 passed.

Not yet verified on hardware: Phases 4, 5, 6, 7, 8. Only the TTS path and
its artifact transport have been proven on a real GPU.
2026-08-11 16:53:33 +05:30
velixio 43de1c794c feat(workers): remote GPU workers over a versioned gRPC protocol
Send individual jobs to GPUs on your other machines while everything else
stays local. Opt-in, off by default: with the toggle off there is no
listening socket, no certificate and no background loop.

Design follows remote/goal_v2.md, the council-revised goal doc. The
decisions that shaped the code, and why:

* A disconnect is an unknown outcome, not a failure. The original design
  reassigned on disconnect while also describing the case where the worker
  had already finished — following both guarantees duplicate execution. An
  attempt now holds a grace window; a worker returning inside it commits
  its result and no second attempt is ever made.
* At-least-once execution, exactly-once result commit. The result is
  persisted BEFORE it is acknowledged, so a crash between the two cannot
  silently lose a finished render.
* Deadlines are phased (accept -> model load -> execute -> deliver) and
  liveness is a progress lease. The old fixed 30s execution budget was two
  orders of magnitude below what this product actually does; silence is
  the failure signal, not slowness.
* Capacity is derived from free VRAM, never configured: a static value
  corrupts output under torch.compile thread affinity (#315) and aborts
  the process on small cards (#567).
* A circuit breaker replaces the reliability-score/quarantine machinery,
  which had no recovery path (no probation workload exists in a TTS
  product) and penalised consumer networks for existing.
* Identity is a keypair the worker generates and never sends. A
  server-assigned id is a name, not an authenticator, so revocation of one
  would be theatre. Enrollment tokens are single-use and carry the control
  plane's certificate fingerprint for pin-on-first-use.

Adds the domain core, scheduler, durable task store, gRPC transport,
worker agent, management API, Settings panel, and docs. Protobuf reserves
the tenant/trace/usage fields a hosted control plane would need, since
adding them later means upgrading a whole fleet.

Includes tests for the failure paths that matter: duplicate delivery,
stale-session fencing, reconnect reconciliation, grace expiry, breaker
attribution, and a real end-to-end TLS round trip.
2026-08-10 14:18:42 +05:30