5302170688
* fix(engines): contain SystemExit at the pool boundary — a CLI-shaped dependency killed the backend (#1133) Auto-report #1133 (8GB M1, v0.3.21, engine mlx-audio, exit code 1 at 21s uptime) carried the whole story in its stderr tail: mlx-audio's Kokoro pipeline uses misaki's G2P, whose __init__ runs spacy.cli.download() IN PROCESS when en_core_web_sm is missing. spaCy's downloader is written as a CLI: with no pip in the venv (uv-managed venvs ship none), its error printer calls sys.exit(1). SystemExit is not an Exception, so every except Exception on the path waved it through; it rode the executor future into the event loop, where uvicorn treats SystemExit as "shut down" — backend dead. Class fix, not a spacy special-case: _contain_system_exit() wraps every callable dispatched through run_on_gpu_pool_guarded (all engine loads AND generates funnel through it, #1033) and asr_backend.run_transcribe_guarded, converting SystemExit into a RuntimeError that names the real failure mode. Any engine dependency written as a CLI is now covered on both the TTS and ASR sides. Not done here (follow-up candidates): pre-provisioning en_core_web_sm for the Kokoro/mlx-audio path so the download never triggers, and/or shipping pip into the managed venv. Both are provisioning decisions; this PR makes the failure survivable and honest first. Tests: SystemExit from a pool job -> RuntimeError naming SystemExit(code), executor still usable afterwards; same for the transcribe guard. Both fail with the containment reverted. Full suite: 3016 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(engines): containment helper moves to a leaf module (CodeQL cyclic-import) utils/containment is stdlib-only, so model_manager and asr_backend both import it at module top with no cycle — the call-time back-import CodeQL flagged is gone. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: mergetest <nizam4103@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
33 lines
1.5 KiB
Python
33 lines
1.5 KiB
Python
"""Process-survival containment for engine/library code.
|
|
|
|
Leaf module (stdlib-only) so both services.model_manager and
|
|
services.asr_backend can import it at module top without a cycle.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
|
|
def contain_system_exit(fn, what: str):
|
|
"""Wrap a pool job so library code calling ``sys.exit()`` cannot kill the app.
|
|
|
|
Real case (#1133): mlx-audio's Kokoro pipeline uses misaki's G2P, which
|
|
runs ``spacy.cli.download()`` IN-PROCESS on first use; spaCy's CLI error
|
|
printer responds to a missing pip (uv-managed venvs ship none) with
|
|
``sys.exit(1)``. ``except Exception`` never catches SystemExit, so it rode
|
|
the executor future into the event loop — where uvicorn treats SystemExit
|
|
as "shut down", killing the whole backend 21 s after start. Any engine
|
|
dependency written as a CLI can do this; containing it at the dispatch
|
|
boundary covers every load, generate, and transcribe.
|
|
"""
|
|
def wrapped():
|
|
try:
|
|
return fn()
|
|
except SystemExit as e: # noqa: PERF203 — the whole point
|
|
raise RuntimeError(
|
|
f"{what}: engine code tried to exit the process "
|
|
f"(SystemExit {e.code}) — contained. This usually means an "
|
|
f"engine dependency failed to auto-install something (e.g. a "
|
|
f"spaCy model needing pip); see the backend log above this "
|
|
f"line for the real error."
|
|
) from e
|
|
return wrapped
|