1 Commits

Include renames

Author	SHA1	Message	Date
velixio	c643706d07	feat(workers): make a remote GPU actually run a task, end to end ... Selecting a remote worker repainted a badge and nothing else. The cause was not subtle: `scheduler.submit` had no production caller, and `routing.decide()` was read only by the status endpoint that paints the header. Remote execution was a complete, tested pipeline with no producer at its head. This adds the producer and fixes the defects that made the pipeline unable to carry a real job: - Nothing routed to the scheduler. Adds `POST /workers/tasks` (loopback-gated, **development-only** until the gateway lands) and `Scheduler.wait`, backed by per-task futures rather than the unregisterable `on_change` listener list. - Every task over two minutes died. No worker ever sent `TaskProgress`, so the 120s progress lease expired mid-render — including during the cold model load, which happens after `TaskStarted`. Workers now report progress and emit a keepalive, bounded by the phase's absolute budget so it renews the lease without deleting the only enforced bound in the system. - The executor rebuilt its engine per task (`return cls()`), so every job paid a cold load. Engines now share one instance cache with the router, resolved by the assignment's engine — never `get_active_tts_backend()`, which returns the worker machine's own Settings preference and would silently run the wrong engine. - One lease expiry took a worker offline permanently: parked slots were never reclaimed. Parks now expire on a TTL, and are deliberately NOT reconciled against the worker's own load report — at a ceiling of one the only task such a worker can report is the wedged one, so "busy" would drop the park and the next idle heartbeat would hand out a slot with a live GPU thread (#730/#1190). - A worker that dropped and reconnected mid-render had every liveness frame discarded: task frames were fenced on the live session epoch, which bumps on every reconnect, while the worker echoes the ref stamped at dispatch. The control plane then expired a task whose GPU was still rendering, and swallowed the failure report when it went wrong. Fenced per attempt instead. - A result from one worker could commit another's task, after which the owner's real delivery arrived as a duplicate and its audio was discarded. "Unknown attempt" and "another worker's attempt" are no longer the same answer. - An oversized result was a poison pill, re-sent identically on every reconnect and permanently disconnecting the worker. It is now a terminal `RESULT_TOO_LARGE`, which is also classified — it was falling through to TRANSIENT and retrying a re-render that could never fit. - `_store_inline` joined the artifact directory with worker-supplied ids, and `os.path.join` discards its prefix on an absolute component. Paths are now minted control-plane-side and resolved through `core.path_security`. - Remote synthesis bypassed `mark_synthetic`, and the guard that exists to catch exactly that walked only `backend/api` and `backend/services` — so it stayed green while a fourth unmarked producer shipped. Marking moved to the worker's tensor stage; the guard now walks `backend/worker` too. Also adds pre-rendered voice previews (`services/gallery.py`), so browsing the gallery no longer needs a GPU or a downloaded model. The manifest is verified against the updater's release key already baked into the binary; a fresh install hears voices without downloading 2.4GB first, and everything falls back to local rendering when the gallery is unreachable. Verified on hardware, not just in CI: 1728 characters submitted to an RTX 4090 returned 105.94s of 24kHz audio in 23.9s, committed and served from the artifact store. Not yet done, and deliberately not claimed: the keepalive fix cannot be exercised end-to-end on fast hardware, because any job long enough to reach the 120s lease produces audio past the 8MiB inline cap. Chunked `UploadResult` has to land first. Pinning to the worker the user chose is also still absent, so "Remote" reaches a remote GPU but not necessarily the one on the badge.	2026-08-11 07:16:04 +05:30