velixio
aa1d739843
The remote-GPU line, verified on hardware rather than asserted. **Dubbing renders on the worker.** dub_generate.py dispatches the coarse `dub_segments` operation through the gateway, following the audiobook pattern: per-unit local fallback after consecutive remote failures, one aggregated notice rather than one per segment. A 40-minute dub that loses its worker at segment 200 degrades instead of producing 200 error rows. **An out-of-date worker is now refused by name.** This was the worst defect in the plan and it was silent: an un-upgraded worker registered cleanly, then ignored `inputs` and rendered a clone with NO reference audio — returned as success. A plausible wrong result with nothing anywhere to surface it. Workers now declare features, and one missing them is turned away with the features named and `no task was run`. Verified live: a worker one commit behind was correctly refused. **"Offline" and "cannot run this" are different facts.** Asking a live worker for an engine it lacks answered "is offline or cannot be reached. Wake the selected worker" — while that worker reported ready, one free slot and 3.6 ms latency. The user was sent to wake a machine that was already awake. The scheduler now distinguishes absent from present-but- incapable, and names the engine rather than the operation, because the engine is the thing a user can install. **An engine with no catalog entry is no longer hidden.** A `repo_ids` non-emptiness check had been implemented as a runtime filter, so a worker silently refused to advertise any engine lacking a models.yaml entry — which is four registered engines, including CosyVoice. Users with those already installed would have lost remote support with only a log line. Empty `repo_ids` now means "not downloadable here", never "not runnable". **And a script so this stops being done by hand.** scripts/verify-remote-worker.sh runs the per-phase acceptance checks against a live worker, non-destructively. Its preconditions are the mistakes that cost the most time: exactly one listener on the control port (two instances silently shared it), and never detecting the worker with a pgrep pattern that matches the ssh shell running it. Its first real run found the dubbing picker claiming remote placement. That turned out to be the CHECK being stale, not the picker — the port had landed since it was written. It now asserts self-consistency instead: the picker may claim remote only for an operation the control plane actually advertises as remotely producible, which cannot rot the next time an op is ported. Backend 5291 passed, frontend 1812 passed. Acceptance script: no automated failures across Phases 4-8 on an RTX 4090. Four checks remain MANUAL by design — true airplane mode, concurrent downloads, killing a worker mid-audiobook, and the model-list UI — and are reported as unverified rather than passed.
133 lines
4.6 KiB
Python
133 lines
4.6 KiB
Python
"""Deadline policy, including the drift guard against model_manager.
|
|
|
|
The original goal doc's 2s/30s/35s example was off by one to two orders of
|
|
magnitude for this product. These tests pin the corrected behaviour and, more
|
|
importantly, keep it tied to the single source of truth for execution budgets
|
|
so the two cannot silently diverge.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from worker import deadlines
|
|
from worker.deadlines import Deadlines, Operation, for_task
|
|
|
|
|
|
def test_execution_budget_matches_model_manager_exactly():
|
|
"""Drift guard.
|
|
|
|
``model_manager.generate_timeout_s`` is THE execution budget for local
|
|
synthesis. If remote tasks computed their own, a change there would leave
|
|
remote work being killed at a different time than local work — the class of
|
|
bug that produced "exceeded 300s" reports when only two call sites were
|
|
updated (#1190/#1202).
|
|
"""
|
|
from services import model_manager
|
|
|
|
for text in ("", "short", "x" * 1200, "x" * 5000, "x" * 50_000):
|
|
assert deadlines._base_execution_seconds(text) == pytest.approx(
|
|
model_manager.generate_timeout_s(text)
|
|
)
|
|
|
|
|
|
def test_fallback_formula_matches_when_model_manager_is_unavailable(monkeypatch):
|
|
"""The control plane may run without torch on the path; the fallback must
|
|
still agree with the real formula."""
|
|
from services import model_manager
|
|
|
|
real = model_manager.generate_timeout_s("x" * 4000)
|
|
|
|
def _boom(*a, **k):
|
|
raise ImportError("no torch here")
|
|
|
|
monkeypatch.setattr(model_manager, "generate_timeout_s", _boom)
|
|
assert deadlines._base_execution_seconds("x" * 4000) == pytest.approx(real)
|
|
|
|
|
|
def test_accept_is_generous_enough_for_a_busy_worker():
|
|
"""The old 2s accept would time out against a worker mid-inference holding
|
|
the GIL, then penalise it for being busy."""
|
|
d = for_task("tts")
|
|
assert d.accept_seconds >= 10
|
|
|
|
|
|
def test_cold_load_gets_far_more_than_the_old_30s():
|
|
d = for_task("tts", model_resident=False, model_downloaded=False)
|
|
assert d.model_load_seconds >= 1200
|
|
|
|
|
|
def test_resident_model_gets_a_short_load_budget():
|
|
warm = for_task("tts", model_resident=True)
|
|
cold = for_task("tts", model_resident=False, model_downloaded=True)
|
|
undownloaded = for_task("tts", model_resident=False, model_downloaded=False)
|
|
|
|
assert warm.model_load_seconds < cold.model_load_seconds < undownloaded.model_load_seconds
|
|
|
|
|
|
def test_execution_scales_with_input_length():
|
|
short = for_task("tts", text="hello")
|
|
long = for_task("tts", text="x" * 50_000)
|
|
assert long.execution_seconds > short.execution_seconds
|
|
|
|
|
|
def test_dub_gets_far_longer_than_dictation():
|
|
"""Inference duration varies by orders of magnitude; one scheme cannot fit."""
|
|
assert for_task("dub").execution_seconds > for_task("dictation").execution_seconds * 10
|
|
|
|
|
|
def test_long_operations_get_a_longer_grace_window():
|
|
"""Losing a 40-minute dub to a 45-second Wi-Fi blip and redoing it from
|
|
zero is the expensive mistake."""
|
|
assert for_task("dub").grace_seconds > for_task("dictation").grace_seconds
|
|
|
|
|
|
def test_reconnect_backoff_grace_and_progress_lease_are_strictly_ordered():
|
|
from worker.transport.client import _MAX_BACKOFF_SECONDS
|
|
|
|
for operation in Operation:
|
|
budget = for_task(operation.value)
|
|
assert _MAX_BACKOFF_SECONDS < budget.grace_seconds < budget.progress_lease_seconds
|
|
|
|
|
|
def test_media_length_scales_operations_measured_in_seconds():
|
|
short = for_task("dub", input_seconds=10)
|
|
long = for_task("dub", input_seconds=3600)
|
|
assert long.execution_seconds > short.execution_seconds
|
|
|
|
|
|
def test_progress_lease_is_shorter_than_execution():
|
|
"""Silence is the failure signal, not slowness — so the lease must be able
|
|
to fire well inside a long job."""
|
|
d = for_task("dub")
|
|
assert d.progress_lease_seconds < d.execution_seconds
|
|
|
|
|
|
def test_unknown_operation_falls_back_instead_of_raising():
|
|
assert Operation.coerce("no-such-op") is Operation.TTS
|
|
assert for_task("no-such-op").execution_seconds > 0
|
|
|
|
|
|
def test_deadlines_serialize_for_the_wire():
|
|
payload = for_task("tts").to_dict()
|
|
assert set(payload) == {
|
|
"accept_seconds",
|
|
"model_load_seconds",
|
|
"execution_seconds",
|
|
"progress_lease_seconds",
|
|
"result_delivery_seconds",
|
|
"grace_seconds",
|
|
}
|
|
assert all(isinstance(v, int) for v in payload.values())
|
|
|
|
|
|
def test_total_is_the_sum_of_the_phases():
|
|
d = Deadlines(
|
|
accept_seconds=1,
|
|
model_load_seconds=2,
|
|
execution_seconds=4,
|
|
progress_lease_seconds=99,
|
|
result_delivery_seconds=8,
|
|
grace_seconds=99,
|
|
)
|
|
assert d.total_seconds == 15
|