velixio
aa1d739843
The remote-GPU line, verified on hardware rather than asserted. **Dubbing renders on the worker.** dub_generate.py dispatches the coarse `dub_segments` operation through the gateway, following the audiobook pattern: per-unit local fallback after consecutive remote failures, one aggregated notice rather than one per segment. A 40-minute dub that loses its worker at segment 200 degrades instead of producing 200 error rows. **An out-of-date worker is now refused by name.** This was the worst defect in the plan and it was silent: an un-upgraded worker registered cleanly, then ignored `inputs` and rendered a clone with NO reference audio — returned as success. A plausible wrong result with nothing anywhere to surface it. Workers now declare features, and one missing them is turned away with the features named and `no task was run`. Verified live: a worker one commit behind was correctly refused. **"Offline" and "cannot run this" are different facts.** Asking a live worker for an engine it lacks answered "is offline or cannot be reached. Wake the selected worker" — while that worker reported ready, one free slot and 3.6 ms latency. The user was sent to wake a machine that was already awake. The scheduler now distinguishes absent from present-but- incapable, and names the engine rather than the operation, because the engine is the thing a user can install. **An engine with no catalog entry is no longer hidden.** A `repo_ids` non-emptiness check had been implemented as a runtime filter, so a worker silently refused to advertise any engine lacking a models.yaml entry — which is four registered engines, including CosyVoice. Users with those already installed would have lost remote support with only a log line. Empty `repo_ids` now means "not downloadable here", never "not runnable". **And a script so this stops being done by hand.** scripts/verify-remote-worker.sh runs the per-phase acceptance checks against a live worker, non-destructively. Its preconditions are the mistakes that cost the most time: exactly one listener on the control port (two instances silently shared it), and never detecting the worker with a pgrep pattern that matches the ssh shell running it. Its first real run found the dubbing picker claiming remote placement. That turned out to be the CHECK being stale, not the picker — the port had landed since it was written. It now asserts self-consistency instead: the picker may claim remote only for an operation the control plane actually advertises as remotely producible, which cannot rot the next time an op is ported. Backend 5291 passed, frontend 1812 passed. Acceptance script: no automated failures across Phases 4-8 on an RTX 4090. Four checks remain MANUAL by design — true airplane mode, concurrent downloads, killing a worker mid-audiobook, and the model-list UI — and are reported as unverified rather than passed.
114 lines
3.8 KiB
Python
114 lines
3.8 KiB
Python
"""Error taxonomy: would retrying somewhere else help, and who pays for it.
|
|
|
|
The scheduler needs two independent answers per failure, and conflating them is
|
|
what produces the two worst behaviours: a poison task that rotates through
|
|
every worker in the fleet, and a healthy fleet that quarantines itself for
|
|
being busy.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from worker import errors
|
|
from worker.errors import ErrorClass, WorkerError, from_exception, from_reason
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"cls,retryable",
|
|
[
|
|
(ErrorClass.TRANSIENT, True),
|
|
(ErrorClass.CAPABILITY, True),
|
|
(ErrorClass.CAPACITY, True),
|
|
(ErrorClass.TIMEOUT, True),
|
|
(ErrorClass.TERMINAL, False),
|
|
(ErrorClass.PROTOCOL, False),
|
|
],
|
|
)
|
|
def test_retryability_matrix(cls, retryable):
|
|
assert cls.retryable is retryable
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"cls,charges",
|
|
[
|
|
(ErrorClass.TRANSIENT, True),
|
|
(ErrorClass.TIMEOUT, True),
|
|
# Declining work you cannot or should not take is not misbehaviour.
|
|
(ErrorClass.CAPACITY, False),
|
|
(ErrorClass.CAPABILITY, False),
|
|
(ErrorClass.TERMINAL, False),
|
|
(ErrorClass.PROTOCOL, False),
|
|
],
|
|
)
|
|
def test_chargeability_matrix(cls, charges):
|
|
assert cls.charges_worker is charges
|
|
|
|
|
|
def test_retryable_and_chargeable_are_independent():
|
|
"""Capability failures retry elsewhere yet never charge the worker — the
|
|
two questions must not collapse into one flag."""
|
|
assert ErrorClass.CAPABILITY.retryable is True
|
|
assert ErrorClass.CAPABILITY.charges_worker is False
|
|
|
|
|
|
def test_protocol_codes_classify():
|
|
assert errors.classify_code("WORKER_AT_CAPACITY") is ErrorClass.CAPACITY
|
|
assert errors.classify_code("INSUFFICIENT_MEMORY") is ErrorClass.CAPABILITY
|
|
assert errors.classify_code("EXECUTION_TIMEOUT") is ErrorClass.TIMEOUT
|
|
assert errors.classify_code("STALE_EPOCH") is ErrorClass.PROTOCOL
|
|
assert errors.classify_code("INVALID_TASK_PARAMS") is ErrorClass.TERMINAL
|
|
|
|
|
|
def test_docs_taxonomy_keys_are_reused_not_reinvented():
|
|
"""A failure that already has a user-facing hint keeps it on the wire."""
|
|
assert errors.classify_code("HF_AUTH_FAILED") is ErrorClass.TERMINAL
|
|
assert errors.classify_code("BROKEN_VENV") is ErrorClass.CAPABILITY
|
|
assert errors.classify_code("VIDEO_DOWNLOAD_NETWORK") is ErrorClass.TRANSIENT
|
|
|
|
|
|
def test_unknown_codes_default_to_retryable():
|
|
"""One wasted retry beats permanently failing work that would have run."""
|
|
assert errors.classify_code("SOMETHING_NEW") is ErrorClass.TRANSIENT
|
|
|
|
|
|
def test_secrets_never_reach_the_wire():
|
|
"""The worker is a remote machine whose logs the user may never read, so
|
|
scrubbing has to happen before the error is sent, not at display time."""
|
|
err = from_reason("failed with token hf_ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789")
|
|
assert "hf_ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789" not in err.message
|
|
|
|
|
|
def test_hint_is_carried_when_the_taxonomy_has_one():
|
|
err = from_reason("boom", code="HF_AUTH_FAILED")
|
|
assert err.error_class is ErrorClass.TERMINAL
|
|
assert err.hint
|
|
|
|
|
|
def test_every_protocol_code_has_an_actionable_hint():
|
|
from core.failure import _HINTS
|
|
|
|
assert errors._PROTOCOL_CODES.keys() <= _HINTS.keys()
|
|
assert all(_HINTS[code].strip() for code in errors._PROTOCOL_CODES)
|
|
|
|
|
|
def test_from_exception_never_produces_an_empty_message():
|
|
"""Empty str(e) was the root of the 'unknown error' reports (#122/#63)."""
|
|
|
|
class Silent(Exception):
|
|
pass
|
|
|
|
assert from_exception(Silent()).message
|
|
|
|
|
|
def test_wire_shape_is_complete():
|
|
payload = WorkerError(
|
|
error_class=ErrorClass.TIMEOUT, code="EXECUTION_TIMEOUT", message="m", hint="h"
|
|
).to_dict()
|
|
assert payload == {
|
|
"error_class": "timeout",
|
|
"code": "EXECUTION_TIMEOUT",
|
|
"message": "m",
|
|
"hint": "h",
|
|
"retryable": True,
|
|
}
|