VoiceStudio

1,912 Commits 24 Branches 39 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
Palash Debnath	5a615d2c66	feat(workers): package headless GPU nodes (#1638 ) (#1648 ) Closes #1638.\n\nPackages headless GPU workers with durable enrollment, bounded artifact handling, cross-platform lifecycle cleanup, and regression coverage. Incorporates CodeRabbit, Greptile, CodeQL, and platform-CI findings before merge.	2026-08-24 16:32:56 +05:30
velixio	aa1d739843	feat(workers): dubbing goes remote, and the protocol stops lying to old workers The remote-GPU line, verified on hardware rather than asserted. Dubbing renders on the worker. dub_generate.py dispatches the coarse `dub_segments` operation through the gateway, following the audiobook pattern: per-unit local fallback after consecutive remote failures, one aggregated notice rather than one per segment. A 40-minute dub that loses its worker at segment 200 degrades instead of producing 200 error rows. An out-of-date worker is now refused by name. This was the worst defect in the plan and it was silent: an un-upgraded worker registered cleanly, then ignored `inputs` and rendered a clone with NO reference audio — returned as success. A plausible wrong result with nothing anywhere to surface it. Workers now declare features, and one missing them is turned away with the features named and `no task was run`. Verified live: a worker one commit behind was correctly refused. "Offline" and "cannot run this" are different facts. Asking a live worker for an engine it lacks answered "is offline or cannot be reached. Wake the selected worker" — while that worker reported ready, one free slot and 3.6 ms latency. The user was sent to wake a machine that was already awake. The scheduler now distinguishes absent from present-but- incapable, and names the engine rather than the operation, because the engine is the thing a user can install. An engine with no catalog entry is no longer hidden. A `repo_ids` non-emptiness check had been implemented as a runtime filter, so a worker silently refused to advertise any engine lacking a models.yaml entry — which is four registered engines, including CosyVoice. Users with those already installed would have lost remote support with only a log line. Empty `repo_ids` now means "not downloadable here", never "not runnable". And a script so this stops being done by hand. scripts/verify-remote-worker.sh runs the per-phase acceptance checks against a live worker, non-destructively. Its preconditions are the mistakes that cost the most time: exactly one listener on the control port (two instances silently shared it), and never detecting the worker with a pgrep pattern that matches the ssh shell running it. Its first real run found the dubbing picker claiming remote placement. That turned out to be the CHECK being stale, not the picker — the port had landed since it was written. It now asserts self-consistency instead: the picker may claim remote only for an operation the control plane actually advertises as remotely producible, which cannot rot the next time an op is ported. Backend 5291 passed, frontend 1812 passed. Acceptance script: no automated failures across Phases 4-8 on an RTX 4090. Four checks remain MANUAL by design — true airplane mode, concurrent downloads, killing a worker mid-audiobook, and the model-list UI — and are reported as unverified rather than passed.	2026-08-11 17:39:46 +05:30
velixio	b54cd28403	feat(workers): one gateway for GPU calls, and results too big for the wire Two phases of the remote-GPU plan, landing together because neither is useful alone: on a 4090 any render long enough to exercise the progress lease also outgrows the 8 MiB message cap, so a gateway that routes work remotely without an artifact transport just moves where the failure happens. The gateway (`services/gpu_gateway.py`) is the single owner of GPU calling, model status, downloads and model load, for both targets — `prewarm`, `run`, `status`, `download`. prewarm and run stay separate because collapsing them loses the two-phase load/generate budget split (#1033/#1037) that the worker protocol already mirrors. Admission moves in here too: the old `check_gpu_admission` call read local pool stats, so under Remote it would 429 on local saturation while the remote GPU sat idle. Artifacts now move out of band above a negotiated threshold. Bytes land in an attempt-scoped `.part` file, are verified against a declared sha256, and are renamed into place only on an explicit last chunk — a transfer that arrives short, reordered, or simply stops commits nothing. A resume rehashes what is already on disk, or the digest would attest only to the tail, which is the exact case a resume exists to protect. Two failure modes found while verifying this, both fixed with mutation-checked regressions: * an oversized payload with no session (mid-reconnect, or a control plane too old to serve UploadResult) has nowhere to go. It must not enter `_pending` — an over-cap frame is re-sent on every reconnect, killing the session each time and stranding every other task — but it must stay retryable, unlike the size gate's TERMINAL verdict: nothing about the render is wrong, only the route to it. * the upload resume loop was bounded by "did the offset change", which a receiver alternating between two byte counts satisfies forever. The worker is single-slot by default, so that is not one lost upload but the machine, doing nothing else, until someone restarts it. Bounded by a round count instead. The control stream is split into control and bulk queues so the heartbeat this whole liveness model rests on cannot queue behind a payload — `result_json` has no size cliff to catch it, and the next bulk message added to the protocol would have reintroduced the stall silently. Live streaming stays on the control plane and now says so once per socket: that route exists to put audio in the user's ear before the sentence finishes, and paying queue admission plus a round trip per utterance would spend the one thing it is for. Silence would have been worse than the limit — the header badge would read "gpu2" while this machine did all the work. Backend 5236 passed, frontend 1807 passed. End-to-end verification on real hardware has NOT been re-run since these changes; the CHANGELOG claim for the Synthesize button waits on that.	2026-08-11 13:30:26 +05:30

Author

SHA1

Message

Date

Palash Debnath

5a615d2c66

feat(workers): package headless GPU nodes (#1638 ) (#1648 )

Closes #1638.\n\nPackages headless GPU workers with durable enrollment, bounded artifact handling, cross-platform lifecycle cleanup, and regression coverage. Incorporates CodeRabbit, Greptile, CodeQL, and platform-CI findings before merge.

2026-08-24 16:32:56 +05:30

velixio

aa1d739843

feat(workers): dubbing goes remote, and the protocol stops lying to old workers

The remote-GPU line, verified on hardware rather than asserted.

**Dubbing renders on the worker.** dub_generate.py dispatches the coarse
`dub_segments` operation through the gateway, following the audiobook
pattern: per-unit local fallback after consecutive remote failures, one
aggregated notice rather than one per segment. A 40-minute dub that loses
its worker at segment 200 degrades instead of producing 200 error rows.

**An out-of-date worker is now refused by name.** This was the worst
defect in the plan and it was silent: an un-upgraded worker registered
cleanly, then ignored `inputs` and rendered a clone with NO reference
audio — returned as success. A plausible wrong result with nothing
anywhere to surface it. Workers now declare features, and one missing
them is turned away with the features named and `no task was run`.
Verified live: a worker one commit behind was correctly refused.

**"Offline" and "cannot run this" are different facts.** Asking a live
worker for an engine it lacks answered "is offline or cannot be reached.
Wake the selected worker" — while that worker reported ready, one free
slot and 3.6 ms latency. The user was sent to wake a machine that was
already awake. The scheduler now distinguishes absent from present-but-
incapable, and names the engine rather than the operation, because the
engine is the thing a user can install.

**An engine with no catalog entry is no longer hidden.** A `repo_ids`
non-emptiness check had been implemented as a runtime filter, so a worker
silently refused to advertise any engine lacking a models.yaml entry —
which is four registered engines, including CosyVoice. Users with those
already installed would have lost remote support with only a log line.
Empty `repo_ids` now means "not downloadable here", never "not runnable".

**And a script so this stops being done by hand.**
scripts/verify-remote-worker.sh runs the per-phase acceptance checks
against a live worker, non-destructively. Its preconditions are the
mistakes that cost the most time: exactly one listener on the control
port (two instances silently shared it), and never detecting the worker
with a pgrep pattern that matches the ssh shell running it.

Its first real run found the dubbing picker claiming remote placement.
That turned out to be the CHECK being stale, not the picker — the port
had landed since it was written. It now asserts self-consistency instead:
the picker may claim remote only for an operation the control plane
actually advertises as remotely producible, which cannot rot the next
time an op is ported.

Backend 5291 passed, frontend 1812 passed. Acceptance script: no
automated failures across Phases 4-8 on an RTX 4090. Four checks remain
MANUAL by design — true airplane mode, concurrent downloads, killing a
worker mid-audiobook, and the model-list UI — and are reported as
unverified rather than passed.

2026-08-11 17:39:46 +05:30

velixio

b54cd28403

feat(workers): one gateway for GPU calls, and results too big for the wire

Two phases of the remote-GPU plan, landing together because neither is
useful alone: on a 4090 any render long enough to exercise the progress
lease also outgrows the 8 MiB message cap, so a gateway that routes work
remotely without an artifact transport just moves where the failure
happens.

**The gateway** (`services/gpu_gateway.py`) is the single owner of GPU
calling, model status, downloads and model load, for both targets —
`prewarm`, `run`, `status`, `download`. prewarm and run stay separate
because collapsing them loses the two-phase load/generate budget split
(#1033/#1037) that the worker protocol already mirrors. Admission moves
in here too: the old `check_gpu_admission` call read *local* pool stats,
so under Remote it would 429 on local saturation while the remote GPU
sat idle.

**Artifacts** now move out of band above a negotiated threshold. Bytes
land in an attempt-scoped `.part` file, are verified against a declared
sha256, and are renamed into place only on an explicit last chunk — a
transfer that arrives short, reordered, or simply stops commits nothing.
A resume rehashes what is already on disk, or the digest would attest
only to the tail, which is the exact case a resume exists to protect.

Two failure modes found while verifying this, both fixed with
mutation-checked regressions:

  * an oversized payload with no session (mid-reconnect, or a control
    plane too old to serve UploadResult) has nowhere to go. It must not
    enter `_pending` — an over-cap frame is re-sent on every reconnect,
    killing the session each time and stranding every other task — but
    it must stay retryable, unlike the size gate's TERMINAL verdict:
    nothing about the render is wrong, only the route to it.

  * the upload resume loop was bounded by "did the offset change", which
    a receiver alternating between two byte counts satisfies forever.
    The worker is single-slot by default, so that is not one lost upload
    but the machine, doing nothing else, until someone restarts it.
    Bounded by a round count instead.

The control stream is split into control and bulk queues so the
heartbeat this whole liveness model rests on cannot queue behind a
payload — `result_json` has no size cliff to catch it, and the next bulk
message added to the protocol would have reintroduced the stall
silently.

Live streaming stays on the control plane and now says so once per
socket: that route exists to put audio in the user's ear before the
sentence finishes, and paying queue admission plus a round trip per
utterance would spend the one thing it is for. Silence would have been
worse than the limit — the header badge would read "gpu2" while this
machine did all the work.

Backend 5236 passed, frontend 1807 passed. End-to-end verification on
real hardware has NOT been re-run since these changes; the CHANGELOG
claim for the Synthesize button waits on that.

2026-08-11 13:30:26 +05:30