d91beef0fd314250d8d9b94de86dfea019a8bd96
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Palash Debnath
|
5a615d2c66 |
feat(workers): package headless GPU nodes (#1638) (#1648)
Closes #1638.\n\nPackages headless GPU workers with durable enrollment, bounded artifact handling, cross-platform lifecycle cleanup, and regression coverage. Incorporates CodeRabbit, Greptile, CodeQL, and platform-CI findings before merge. |
||
|
velixio
|
bda169c900 |
feat(workers): pin work to the chosen GPU, and say when its model is missing
Three phases that only make sense together: a job that names a worker, a worker that reports honestly what it can actually run, and the small defects that made both lie. **Pinning** (Phase 1). `pinned_worker_id` is now honoured in both places that choose a worker — `eligible_workers` and `select_worker` build independent lists, so applying it to one silently leaked work onto whichever machine was least busy. The pin persists across a restart via an additive column, deliberately not alembic (justified in the code, per the precedent already in db.py): quitting mid-render used to drop it without a word. `max_attempts=1` was rejected as the mechanism — it makes the FIRST failure terminal, including the penalty-free ones a stale advisory view produces routinely. Cancel now actually reaches the worker. `WorkerServicer.cancel` had zero callers, so cancelling released the slot while the GPU thread kept running, and a late result could resurrect the task as COMPLETED — `commit_result` assigned that state directly, bypassing the transition table where CANCELLED is terminal by construction. **Honest capabilities** (Phase 4). A worker now probes whether weights are actually present, and a job stops BEFORE dispatch with a typed 409 naming the model and the machine, instead of failing mid-task. The probe fails OPEN: `is_cached`/`cache_is_complete` cannot see a user-managed clone outside the HF layout, so only a positive "absent" refuses. Refusing an engine that works today would break the compatibility promise. `pool.supports` deliberately still ignores `downloaded` — had it not, the scheduler would drop the worker and answer with a terminal NO_CAPABLE_WORKER, which tells the user to check their install when the truth is one download away. The frontend no longer offers "Report this bug" for that state; it offers the download. Catalog tags resolve against the TARGET's OS/arch/backend, not this machine's. From a Mac control plane, a CUDA worker's model list was showing the mlx-community repos it cannot run and hiding the ones it needs. **And the quiet ones** (Phase 0 leftovers): a model's human label rides its own proto field so renaming it cannot orphan breaker history; an empty model_id no longer forks the capacity slot key into two slots for one model; the idle sweep cannot evict an engine out from under a live LOCAL render. Verified on real hardware, which is the only verification that has ever caught anything here: 2025 characters, default settings, routed to an RTX 4090 over the wire — 100% GPU utilisation on the remote box, 119.6 s of 24 kHz audio returned in 16.6 s, 5.7 MB delivered out of band through the artifact path rather than the control stream. Backend 5259 passed, frontend 1808 passed. |
||
|
velixio
|
b54cd28403 |
feat(workers): one gateway for GPU calls, and results too big for the wire
Two phases of the remote-GPU plan, landing together because neither is
useful alone: on a 4090 any render long enough to exercise the progress
lease also outgrows the 8 MiB message cap, so a gateway that routes work
remotely without an artifact transport just moves where the failure
happens.
**The gateway** (`services/gpu_gateway.py`) is the single owner of GPU
calling, model status, downloads and model load, for both targets —
`prewarm`, `run`, `status`, `download`. prewarm and run stay separate
because collapsing them loses the two-phase load/generate budget split
(#1033/#1037) that the worker protocol already mirrors. Admission moves
in here too: the old `check_gpu_admission` call read *local* pool stats,
so under Remote it would 429 on local saturation while the remote GPU
sat idle.
**Artifacts** now move out of band above a negotiated threshold. Bytes
land in an attempt-scoped `.part` file, are verified against a declared
sha256, and are renamed into place only on an explicit last chunk — a
transfer that arrives short, reordered, or simply stops commits nothing.
A resume rehashes what is already on disk, or the digest would attest
only to the tail, which is the exact case a resume exists to protect.
Two failure modes found while verifying this, both fixed with
mutation-checked regressions:
* an oversized payload with no session (mid-reconnect, or a control
plane too old to serve UploadResult) has nowhere to go. It must not
enter `_pending` — an over-cap frame is re-sent on every reconnect,
killing the session each time and stranding every other task — but
it must stay retryable, unlike the size gate's TERMINAL verdict:
nothing about the render is wrong, only the route to it.
* the upload resume loop was bounded by "did the offset change", which
a receiver alternating between two byte counts satisfies forever.
The worker is single-slot by default, so that is not one lost upload
but the machine, doing nothing else, until someone restarts it.
Bounded by a round count instead.
The control stream is split into control and bulk queues so the
heartbeat this whole liveness model rests on cannot queue behind a
payload — `result_json` has no size cliff to catch it, and the next bulk
message added to the protocol would have reintroduced the stall
silently.
Live streaming stays on the control plane and now says so once per
socket: that route exists to put audio in the user's ear before the
sentence finishes, and paying queue admission plus a round trip per
utterance would spend the one thing it is for. Silence would have been
worse than the limit — the header badge would read "gpu2" while this
machine did all the work.
Backend 5236 passed, frontend 1807 passed. End-to-end verification on
real hardware has NOT been re-run since these changes; the CHANGELOG
claim for the Synthesize button waits on that.
|
||
|
velixio
|
43de1c794c |
feat(workers): remote GPU workers over a versioned gRPC protocol
Send individual jobs to GPUs on your other machines while everything else stays local. Opt-in, off by default: with the toggle off there is no listening socket, no certificate and no background loop. Design follows remote/goal_v2.md, the council-revised goal doc. The decisions that shaped the code, and why: * A disconnect is an unknown outcome, not a failure. The original design reassigned on disconnect while also describing the case where the worker had already finished — following both guarantees duplicate execution. An attempt now holds a grace window; a worker returning inside it commits its result and no second attempt is ever made. * At-least-once execution, exactly-once result commit. The result is persisted BEFORE it is acknowledged, so a crash between the two cannot silently lose a finished render. * Deadlines are phased (accept -> model load -> execute -> deliver) and liveness is a progress lease. The old fixed 30s execution budget was two orders of magnitude below what this product actually does; silence is the failure signal, not slowness. * Capacity is derived from free VRAM, never configured: a static value corrupts output under torch.compile thread affinity (#315) and aborts the process on small cards (#567). * A circuit breaker replaces the reliability-score/quarantine machinery, which had no recovery path (no probation workload exists in a TTS product) and penalised consumer networks for existing. * Identity is a keypair the worker generates and never sends. A server-assigned id is a name, not an authenticator, so revocation of one would be theatre. Enrollment tokens are single-use and carry the control plane's certificate fingerprint for pin-on-first-use. Adds the domain core, scheduler, durable task store, gRPC transport, worker agent, management API, Settings panel, and docs. Protobuf reserves the tenant/trace/usage fields a hosted control plane would need, since adding them later means upgrading a whole fleet. Includes tests for the failure paths that matter: duplicate delivery, stale-session fencing, reconnect reconciliation, grace expiry, breaker attribution, and a real end-to-end TLS round trip. |