velixio
43de1c794c
Send individual jobs to GPUs on your other machines while everything else stays local. Opt-in, off by default: with the toggle off there is no listening socket, no certificate and no background loop. Design follows remote/goal_v2.md, the council-revised goal doc. The decisions that shaped the code, and why: * A disconnect is an unknown outcome, not a failure. The original design reassigned on disconnect while also describing the case where the worker had already finished — following both guarantees duplicate execution. An attempt now holds a grace window; a worker returning inside it commits its result and no second attempt is ever made. * At-least-once execution, exactly-once result commit. The result is persisted BEFORE it is acknowledged, so a crash between the two cannot silently lose a finished render. * Deadlines are phased (accept -> model load -> execute -> deliver) and liveness is a progress lease. The old fixed 30s execution budget was two orders of magnitude below what this product actually does; silence is the failure signal, not slowness. * Capacity is derived from free VRAM, never configured: a static value corrupts output under torch.compile thread affinity (#315) and aborts the process on small cards (#567). * A circuit breaker replaces the reliability-score/quarantine machinery, which had no recovery path (no probation workload exists in a TTS product) and penalised consumer networks for existing. * Identity is a keypair the worker generates and never sends. A server-assigned id is a name, not an authenticator, so revocation of one would be theatre. Enrollment tokens are single-use and carry the control plane's certificate fingerprint for pin-on-first-use. Adds the domain core, scheduler, durable task store, gRPC transport, worker agent, management API, Settings panel, and docs. Protobuf reserves the tenant/trace/usage fields a hosted control plane would need, since adding them later means upgrading a whole fleet. Includes tests for the failure paths that matter: duplicate delivery, stale-session fencing, reconnect reconciliation, grace expiry, breaker attribution, and a real end-to-end TLS round trip.
23 lines
759 B
Python
23 lines
759 B
Python
"""Injectable clock.
|
|
|
|
Every deadline, lease, grace window, and cooldown in this package takes an
|
|
optional ``now``. The obvious spelling — ``now or time.time()`` — is a trap:
|
|
``0.0`` is falsy, so a caller that pins time at the epoch silently gets the
|
|
wall clock instead. That makes tests lie (they pass while measuring real time)
|
|
and would make any future replay or simulation harness quietly wrong.
|
|
|
|
One helper, used everywhere, so the mistake cannot recur.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import time
|
|
from typing import Optional
|
|
|
|
|
|
def resolve(now: Optional[float] = None) -> float:
|
|
"""Return ``now`` when supplied — including ``0.0`` — else the wall clock."""
|
|
return time.time() if now is None else float(now)
|
|
|
|
|
|
__all__ = ["resolve"]
|