d91beef0fd314250d8d9b94de86dfea019a8bd96
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 velixio
|
aa1d739843 |
feat(workers): dubbing goes remote, and the protocol stops lying to old workers
The remote-GPU line, verified on hardware rather than asserted. **Dubbing renders on the worker.** dub_generate.py dispatches the coarse `dub_segments` operation through the gateway, following the audiobook pattern: per-unit local fallback after consecutive remote failures, one aggregated notice rather than one per segment. A 40-minute dub that loses its worker at segment 200 degrades instead of producing 200 error rows. **An out-of-date worker is now refused by name.** This was the worst defect in the plan and it was silent: an un-upgraded worker registered cleanly, then ignored `inputs` and rendered a clone with NO reference audio — returned as success. A plausible wrong result with nothing anywhere to surface it. Workers now declare features, and one missing them is turned away with the features named and `no task was run`. Verified live: a worker one commit behind was correctly refused. **"Offline" and "cannot run this" are different facts.** Asking a live worker for an engine it lacks answered "is offline or cannot be reached. Wake the selected worker" — while that worker reported ready, one free slot and 3.6 ms latency. The user was sent to wake a machine that was already awake. The scheduler now distinguishes absent from present-but- incapable, and names the engine rather than the operation, because the engine is the thing a user can install. **An engine with no catalog entry is no longer hidden.** A `repo_ids` non-emptiness check had been implemented as a runtime filter, so a worker silently refused to advertise any engine lacking a models.yaml entry — which is four registered engines, including CosyVoice. Users with those already installed would have lost remote support with only a log line. Empty `repo_ids` now means "not downloadable here", never "not runnable". **And a script so this stops being done by hand.** scripts/verify-remote-worker.sh runs the per-phase acceptance checks against a live worker, non-destructively. Its preconditions are the mistakes that cost the most time: exactly one listener on the control port (two instances silently shared it), and never detecting the worker with a pgrep pattern that matches the ssh shell running it. Its first real run found the dubbing picker claiming remote placement. That turned out to be the CHECK being stale, not the picker — the port had landed since it was written. It now asserts self-consistency instead: the picker may claim remote only for an operation the control plane actually advertises as remotely producible, which cannot rot the next time an op is ported. Backend 5291 passed, frontend 1812 passed. Acceptance script: no automated failures across Phases 4-8 on an RTX 4090. Four checks remain MANUAL by design — true airplane mode, concurrent downloads, killing a worker mid-audiobook, and the model-list UI — and are reported as unverified rather than passed. |
||
|
velixio
|
b7caa494eb |
feat(workers): remote downloads, audiobook chapters, and one port that stays honest
Five workstreams that finish the remote-GPU line, plus the test hole that let a broken signature reach a commit. **Downloads go through the normal path** (Phase 5). Rather than a second remote-only route, the existing Models install flow became target-aware, so a model landing on a worker uses the same code, the same progress events and the same UI as a local one. Progress rows key on (target, repo_id) — the aggregator keyed on bare repo_id, so the same model downloading here and on a worker at once collapsed into one row that told the user nothing true about either. **Audiobooks render chapter by chapter on the worker** (Phase 8), with per-chapter local fallback and ONE aggregated notice. The failure that shape exists to prevent: a remote GPU that sleeps at chapter 40 of 200 must not turn a working book into 160 rows of PROGRESS_LEASE_EXPIRED. Dictation is deliberately NOT ported — it runs ASR per utterance inside a live WebSocket loop, and paying queue admission plus a round trip there would spend the one thing that route is for. **Dubbing stays local, and says so** (Phase 7). The coarse worker operation is not finished, so the picker still reports dubbing as local rather than showing a green remote chip over work this machine is doing. What could not wait is the in-loop OOM retry: it sniffed the error string and flushed the *local* CUDA cache, which under remote execution is the wrong machine's GPU entirely. That is fixed now, before the path that would have exercised it exists. **Two instances can no longer share the control plane.** A second VoiceStudio silently bound the same worker port and coexisted, so remote workers landed on whichever process won the race — a session that registers with one instance and appears dead to the other. This produced hours of misdiagnosis during hardware testing and would hit any user with the app open twice. The second instance now keeps running locally and explains the conflict instead of quietly competing. **And the hole that allowed all this to be missable.** gpu_gateway called Scheduler.submit(pinned_worker_id=...) one commit before that parameter existed. Every remote generation raised TypeError; 5236 tests passed anyway, because nothing exercised the gateway against the real scheduler. tests/test_gpu_gateway_scheduler_contract.py now runs that path for real and binds every gateway→dependency call signature. Verified by renaming the parameter away and watching both tests fail with the original error. Gallery previews also fall back to a local render when a downloaded clip cannot be decoded, rather than yielding silence. Backend 5274 passed, frontend 1812 passed. Not yet verified on hardware: Phases 4, 5, 6, 7, 8. Only the TTS path and its artifact transport have been proven on a real GPU. |
||
|
velixio
|
c643706d07 |
feat(workers): make a remote GPU actually run a task, end to end
Selecting a remote worker repainted a badge and nothing else. The cause was not subtle: `scheduler.submit` had no production caller, and `routing.decide()` was read only by the status endpoint that paints the header. Remote execution was a complete, tested pipeline with no producer at its head. This adds the producer and fixes the defects that made the pipeline unable to carry a real job: - Nothing routed to the scheduler. Adds `POST /workers/tasks` (loopback-gated, **development-only** until the gateway lands) and `Scheduler.wait`, backed by per-task futures rather than the unregisterable `on_change` listener list. - Every task over two minutes died. No worker ever sent `TaskProgress`, so the 120s progress lease expired mid-render — including during the cold model load, which happens after `TaskStarted`. Workers now report progress and emit a keepalive, bounded by the phase's absolute budget so it renews the lease without deleting the only enforced bound in the system. - The executor rebuilt its engine per task (`return cls()`), so every job paid a cold load. Engines now share one instance cache with the router, resolved by the assignment's engine — never `get_active_tts_backend()`, which returns the worker machine's own Settings preference and would silently run the wrong engine. - One lease expiry took a worker offline permanently: parked slots were never reclaimed. Parks now expire on a TTL, and are deliberately NOT reconciled against the worker's own load report — at a ceiling of one the only task such a worker can report is the wedged one, so "busy" would drop the park and the next idle heartbeat would hand out a slot with a live GPU thread (#730/#1190). - A worker that dropped and reconnected mid-render had every liveness frame discarded: task frames were fenced on the live session epoch, which bumps on every reconnect, while the worker echoes the ref stamped at dispatch. The control plane then expired a task whose GPU was still rendering, and swallowed the failure report when it went wrong. Fenced per attempt instead. - A result from one worker could commit another's task, after which the owner's real delivery arrived as a duplicate and its audio was discarded. "Unknown attempt" and "another worker's attempt" are no longer the same answer. - An oversized result was a poison pill, re-sent identically on every reconnect and permanently disconnecting the worker. It is now a terminal `RESULT_TOO_LARGE`, which is also classified — it was falling through to TRANSIENT and retrying a re-render that could never fit. - `_store_inline` joined the artifact directory with worker-supplied ids, and `os.path.join` discards its prefix on an absolute component. Paths are now minted control-plane-side and resolved through `core.path_security`. - Remote synthesis bypassed `mark_synthetic`, and the guard that exists to catch exactly that walked only `backend/api` and `backend/services` — so it stayed green while a fourth unmarked producer shipped. Marking moved to the worker's tensor stage; the guard now walks `backend/worker` too. Also adds pre-rendered voice previews (`services/gallery.py`), so browsing the gallery no longer needs a GPU or a downloaded model. The manifest is verified against the updater's release key already baked into the binary; a fresh install hears voices without downloading 2.4GB first, and everything falls back to local rendering when the gallery is unreachable. Verified on hardware, not just in CI: 1728 characters submitted to an RTX 4090 returned 105.94s of 24kHz audio in 23.9s, committed and served from the artifact store. Not yet done, and deliberately not claimed: the keepalive fix cannot be exercised end-to-end on fast hardware, because any job long enough to reach the 120s lease produces audio past the 8MiB inline cap. Chunked `UploadResult` has to land first. Pinning to the worker the user chose is also still absent, so "Remote" reaches a remote GPU but not necessarily the one on the badge. |
||
|
velixio
|
9eb1ec7591 |
feat(workers): choose where jobs run, and show whether that machine is well
Adds a GPU target picker to the header: Local, or one of the machines you enrolled. Exactly one is active at a time; other connected workers are standby and receive nothing. The selection is the user's, not the scheduler's. The engine underneath can rank many workers and the hosted platform will need that, but a desktop app is better served by a choice you can predict and explain: "your worker is offline, this ran locally" is a sentence, "least-busy ranking preferred the laptop" is not. Picking an offline machine is allowed on purpose — you choose your desktop, then go and switch it on. `routing.decide()` is the single answer to "where does the next job run", shared by the badge and (soon) the generation path, so the badge cannot claim something the router will not do. It shows the RESOLVED answer rather than the stored choice: pick your desktop, let it sleep, and the chip reads Local with the reason, while the menu still shows your desktop selected. Connection latency is now real. `latency_ms` existed but nothing measured it — the protocol had Ping with no reply — so it was always zero. Adds Pong (additive, field 12) and times the round trip on the control plane's MONOTONIC clock, so an NTP step or a sleep/wake cannot produce a nonsense reading, and no worker timestamp is trusted. Reported as a median of five samples and withheld until a second sample exists: the first round trip after connect lands while the worker is still importing torch, which measured 139 ms on loopback and, averaged, carried that for a minute. This is CONNECTION latency, not time-to-result. It is shown as information, never as a routing input — RTT is milliseconds where inference is seconds, so ranking on it would optimise noise. Also fixes a bug the picker exposed: worker config was read from the pool, which caches the row handed to it at connect time. Renaming a CONNECTED worker updated the database and the API kept serving the old name until it reconnected — same for priority and enable/disable. Config now comes from the database and liveness from the pool, never the reverse, and writers refresh the live copy so the scheduler's logs do not use a stale name. Adds worker rename (the backend already supported it; no UI called it), worker address as seen by the control plane rather than self-reported, and ready/busy/offline status behind the header dot. |