velixio
43de1c794c
Send individual jobs to GPUs on your other machines while everything else stays local. Opt-in, off by default: with the toggle off there is no listening socket, no certificate and no background loop. Design follows remote/goal_v2.md, the council-revised goal doc. The decisions that shaped the code, and why: * A disconnect is an unknown outcome, not a failure. The original design reassigned on disconnect while also describing the case where the worker had already finished — following both guarantees duplicate execution. An attempt now holds a grace window; a worker returning inside it commits its result and no second attempt is ever made. * At-least-once execution, exactly-once result commit. The result is persisted BEFORE it is acknowledged, so a crash between the two cannot silently lose a finished render. * Deadlines are phased (accept -> model load -> execute -> deliver) and liveness is a progress lease. The old fixed 30s execution budget was two orders of magnitude below what this product actually does; silence is the failure signal, not slowness. * Capacity is derived from free VRAM, never configured: a static value corrupts output under torch.compile thread affinity (#315) and aborts the process on small cards (#567). * A circuit breaker replaces the reliability-score/quarantine machinery, which had no recovery path (no probation workload exists in a TTS product) and penalised consumer networks for existing. * Identity is a keypair the worker generates and never sends. A server-assigned id is a name, not an authenticator, so revocation of one would be theatre. Enrollment tokens are single-use and carry the control plane's certificate fingerprint for pin-on-first-use. Adds the domain core, scheduler, durable task store, gRPC transport, worker agent, management API, Settings panel, and docs. Protobuf reserves the tenant/trace/usage fields a hosted control plane would need, since adding them later means upgrading a whole fleet. Includes tests for the failure paths that matter: duplicate delivery, stale-session fencing, reconnect reconciliation, grace expiry, breaker attribution, and a real end-to-end TLS round trip.
82 lines
2.4 KiB
Python
82 lines
2.4 KiB
Python
#!/usr/bin/env python3
|
|
"""Regenerate the worker protocol stubs from ``worker_v1.proto``.
|
|
|
|
uv run python scripts/gen_worker_protocol.py
|
|
|
|
The generated files are committed so that neither the installer, the frozen
|
|
build, nor Docker needs ``protoc`` — only developers changing the ``.proto`` do.
|
|
``tests/test_worker_protocol_gen.py`` regenerates into a temporary directory
|
|
and fails if the committed output has drifted, so a forgotten regeneration is a
|
|
red test rather than a runtime import error.
|
|
|
|
The one post-processing step is the import fixup: ``protoc`` emits
|
|
``import worker_v1_pb2`` in the gRPC stub, which only resolves if the output
|
|
directory happens to be on ``sys.path``. Rewriting it to a relative import lets
|
|
the package be imported normally.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import re
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
_REPO = Path(__file__).resolve().parent.parent
|
|
_PROTO_DIR = _REPO / "backend" / "worker" / "protocol"
|
|
_OUT_DIR = _PROTO_DIR / "gen"
|
|
_PROTO = _PROTO_DIR / "worker_v1.proto"
|
|
|
|
_INIT = '''"""Generated protocol stubs — DO NOT EDIT.
|
|
|
|
Regenerate with ``uv run python scripts/gen_worker_protocol.py`` after any
|
|
change to ``../worker_v1.proto``.
|
|
"""
|
|
'''
|
|
|
|
|
|
def generate(out_dir: Path) -> int:
|
|
"""Run protoc into ``out_dir``. Returns protoc's exit code."""
|
|
from grpc_tools import protoc # noqa: PLC0415 — dev-only dependency
|
|
|
|
out_dir.mkdir(parents=True, exist_ok=True)
|
|
code = protoc.main(
|
|
[
|
|
"protoc",
|
|
f"-I{_PROTO_DIR}",
|
|
f"--python_out={out_dir}",
|
|
f"--pyi_out={out_dir}",
|
|
f"--grpc_python_out={out_dir}",
|
|
str(_PROTO),
|
|
]
|
|
)
|
|
if code != 0:
|
|
return code
|
|
_fix_imports(out_dir)
|
|
(out_dir / "__init__.py").write_text(_INIT, encoding="utf-8")
|
|
return 0
|
|
|
|
|
|
def _fix_imports(out_dir: Path) -> None:
|
|
"""Make protoc's flat sibling import work inside a package."""
|
|
stub = out_dir / "worker_v1_pb2_grpc.py"
|
|
if not stub.exists():
|
|
return
|
|
text = stub.read_text(encoding="utf-8")
|
|
text = re.sub(
|
|
r"^import (\w+_pb2) as (\w+)$",
|
|
r"from . import \1 as \2",
|
|
text,
|
|
flags=re.M,
|
|
)
|
|
stub.write_text(text, encoding="utf-8")
|
|
|
|
|
|
def main() -> int:
|
|
code = generate(_OUT_DIR)
|
|
if code == 0:
|
|
print(f"Generated {_OUT_DIR.relative_to(_REPO)}")
|
|
return code
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|