Three phases that only make sense together: a job that names a worker,
a worker that reports honestly what it can actually run, and the small
defects that made both lie.
**Pinning** (Phase 1). `pinned_worker_id` is now honoured in both places
that choose a worker — `eligible_workers` and `select_worker` build
independent lists, so applying it to one silently leaked work onto
whichever machine was least busy. The pin persists across a restart via
an additive column, deliberately not alembic (justified in the code, per
the precedent already in db.py): quitting mid-render used to drop it
without a word. `max_attempts=1` was rejected as the mechanism — it makes
the FIRST failure terminal, including the penalty-free ones a stale
advisory view produces routinely.
Cancel now actually reaches the worker. `WorkerServicer.cancel` had zero
callers, so cancelling released the slot while the GPU thread kept
running, and a late result could resurrect the task as COMPLETED —
`commit_result` assigned that state directly, bypassing the transition
table where CANCELLED is terminal by construction.
**Honest capabilities** (Phase 4). A worker now probes whether weights
are actually present, and a job stops BEFORE dispatch with a typed 409
naming the model and the machine, instead of failing mid-task. The probe
fails OPEN: `is_cached`/`cache_is_complete` cannot see a user-managed
clone outside the HF layout, so only a positive "absent" refuses.
Refusing an engine that works today would break the compatibility
promise. `pool.supports` deliberately still ignores `downloaded` — had it
not, the scheduler would drop the worker and answer with a terminal
NO_CAPABLE_WORKER, which tells the user to check their install when the
truth is one download away. The frontend no longer offers "Report this
bug" for that state; it offers the download.
Catalog tags resolve against the TARGET's OS/arch/backend, not this
machine's. From a Mac control plane, a CUDA worker's model list was
showing the mlx-community repos it cannot run and hiding the ones it
needs.
**And the quiet ones** (Phase 0 leftovers): a model's human label rides
its own proto field so renaming it cannot orphan breaker history; an
empty model_id no longer forks the capacity slot key into two slots for
one model; the idle sweep cannot evict an engine out from under a live
LOCAL render.
Verified on real hardware, which is the only verification that has ever
caught anything here: 2025 characters, default settings, routed to an
RTX 4090 over the wire — 100% GPU utilisation on the remote box, 119.6 s
of 24 kHz audio returned in 16.6 s, 5.7 MB delivered out of band through
the artifact path rather than the control stream.
Backend 5259 passed, frontend 1808 passed.
Send individual jobs to GPUs on your other machines while everything else
stays local. Opt-in, off by default: with the toggle off there is no
listening socket, no certificate and no background loop.
Design follows remote/goal_v2.md, the council-revised goal doc. The
decisions that shaped the code, and why:
* A disconnect is an unknown outcome, not a failure. The original design
reassigned on disconnect while also describing the case where the worker
had already finished — following both guarantees duplicate execution. An
attempt now holds a grace window; a worker returning inside it commits
its result and no second attempt is ever made.
* At-least-once execution, exactly-once result commit. The result is
persisted BEFORE it is acknowledged, so a crash between the two cannot
silently lose a finished render.
* Deadlines are phased (accept -> model load -> execute -> deliver) and
liveness is a progress lease. The old fixed 30s execution budget was two
orders of magnitude below what this product actually does; silence is
the failure signal, not slowness.
* Capacity is derived from free VRAM, never configured: a static value
corrupts output under torch.compile thread affinity (#315) and aborts
the process on small cards (#567).
* A circuit breaker replaces the reliability-score/quarantine machinery,
which had no recovery path (no probation workload exists in a TTS
product) and penalised consumer networks for existing.
* Identity is a keypair the worker generates and never sends. A
server-assigned id is a name, not an authenticator, so revocation of one
would be theatre. Enrollment tokens are single-use and carry the control
plane's certificate fingerprint for pin-on-first-use.
Adds the domain core, scheduler, durable task store, gRPC transport,
worker agent, management API, Settings panel, and docs. Protobuf reserves
the tenant/trace/usage fields a hosted control plane would need, since
adding them later means upgrading a whole fleet.
Includes tests for the failure paths that matter: duplicate delivery,
stale-session fencing, reconnect reconciliation, grace expiry, breaker
attribution, and a real end-to-end TLS round trip.
Palash Debnath