d91beef0fd314250d8d9b94de86dfea019a8bd96
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 debpalash
|
037a5689de | fix(worker): address legacy transport review findings | ||
|
velixio
|
c643706d07 |
feat(workers): make a remote GPU actually run a task, end to end
Selecting a remote worker repainted a badge and nothing else. The cause was not subtle: `scheduler.submit` had no production caller, and `routing.decide()` was read only by the status endpoint that paints the header. Remote execution was a complete, tested pipeline with no producer at its head. This adds the producer and fixes the defects that made the pipeline unable to carry a real job: - Nothing routed to the scheduler. Adds `POST /workers/tasks` (loopback-gated, **development-only** until the gateway lands) and `Scheduler.wait`, backed by per-task futures rather than the unregisterable `on_change` listener list. - Every task over two minutes died. No worker ever sent `TaskProgress`, so the 120s progress lease expired mid-render — including during the cold model load, which happens after `TaskStarted`. Workers now report progress and emit a keepalive, bounded by the phase's absolute budget so it renews the lease without deleting the only enforced bound in the system. - The executor rebuilt its engine per task (`return cls()`), so every job paid a cold load. Engines now share one instance cache with the router, resolved by the assignment's engine — never `get_active_tts_backend()`, which returns the worker machine's own Settings preference and would silently run the wrong engine. - One lease expiry took a worker offline permanently: parked slots were never reclaimed. Parks now expire on a TTL, and are deliberately NOT reconciled against the worker's own load report — at a ceiling of one the only task such a worker can report is the wedged one, so "busy" would drop the park and the next idle heartbeat would hand out a slot with a live GPU thread (#730/#1190). - A worker that dropped and reconnected mid-render had every liveness frame discarded: task frames were fenced on the live session epoch, which bumps on every reconnect, while the worker echoes the ref stamped at dispatch. The control plane then expired a task whose GPU was still rendering, and swallowed the failure report when it went wrong. Fenced per attempt instead. - A result from one worker could commit another's task, after which the owner's real delivery arrived as a duplicate and its audio was discarded. "Unknown attempt" and "another worker's attempt" are no longer the same answer. - An oversized result was a poison pill, re-sent identically on every reconnect and permanently disconnecting the worker. It is now a terminal `RESULT_TOO_LARGE`, which is also classified — it was falling through to TRANSIENT and retrying a re-render that could never fit. - `_store_inline` joined the artifact directory with worker-supplied ids, and `os.path.join` discards its prefix on an absolute component. Paths are now minted control-plane-side and resolved through `core.path_security`. - Remote synthesis bypassed `mark_synthetic`, and the guard that exists to catch exactly that walked only `backend/api` and `backend/services` — so it stayed green while a fourth unmarked producer shipped. Marking moved to the worker's tensor stage; the guard now walks `backend/worker` too. Also adds pre-rendered voice previews (`services/gallery.py`), so browsing the gallery no longer needs a GPU or a downloaded model. The manifest is verified against the updater's release key already baked into the binary; a fresh install hears voices without downloading 2.4GB first, and everything falls back to local rendering when the gallery is unreachable. Verified on hardware, not just in CI: 1728 characters submitted to an RTX 4090 returned 105.94s of 24kHz audio in 23.9s, committed and served from the artifact store. Not yet done, and deliberately not claimed: the keepalive fix cannot be exercised end-to-end on fast hardware, because any job long enough to reach the 120s lease produces audio past the 8MiB inline cap. Chunked `UploadResult` has to land first. Pinning to the worker the user chose is also still absent, so "Remote" reaches a remote GPU but not necessarily the one on the badge. |
||
|
velixio
|
43de1c794c |
feat(workers): remote GPU workers over a versioned gRPC protocol
Send individual jobs to GPUs on your other machines while everything else stays local. Opt-in, off by default: with the toggle off there is no listening socket, no certificate and no background loop. Design follows remote/goal_v2.md, the council-revised goal doc. The decisions that shaped the code, and why: * A disconnect is an unknown outcome, not a failure. The original design reassigned on disconnect while also describing the case where the worker had already finished — following both guarantees duplicate execution. An attempt now holds a grace window; a worker returning inside it commits its result and no second attempt is ever made. * At-least-once execution, exactly-once result commit. The result is persisted BEFORE it is acknowledged, so a crash between the two cannot silently lose a finished render. * Deadlines are phased (accept -> model load -> execute -> deliver) and liveness is a progress lease. The old fixed 30s execution budget was two orders of magnitude below what this product actually does; silence is the failure signal, not slowness. * Capacity is derived from free VRAM, never configured: a static value corrupts output under torch.compile thread affinity (#315) and aborts the process on small cards (#567). * A circuit breaker replaces the reliability-score/quarantine machinery, which had no recovery path (no probation workload exists in a TTS product) and penalised consumer networks for existing. * Identity is a keypair the worker generates and never sends. A server-assigned id is a name, not an authenticator, so revocation of one would be theatre. Enrollment tokens are single-use and carry the control plane's certificate fingerprint for pin-on-first-use. Adds the domain core, scheduler, durable task store, gRPC transport, worker agent, management API, Settings panel, and docs. Protobuf reserves the tenant/trace/usage fields a hosted control plane would need, since adding them later means upgrading a whole fleet. Includes tests for the failure paths that matter: duplicate delivery, stale-session fencing, reconnect reconciliation, grace expiry, breaker attribution, and a real end-to-end TLS round trip. |