Paolo Antinori
47c51698c9
* feat(engines): add omnivoice-subprocess, a crash-isolated TTS engine The default in-process OmniVoice engine runs on the GPU ThreadPoolExecutor. When a generate or load exceeds its execution budget the pool is "reset", but the abandoned worker thread cannot be killed (Python cannot interrupt a native torch/MPS call), so it keeps holding the device until it finishes on its own and later synths queue behind it and hang. The reset restores pool capacity but not the device. This is the residual root cause behind the closed #730 and #1190: the messaging/reset mitigations address the symptom, not the device-holding zombie. Add an opt-in `omnivoice-subprocess` engine that runs the same model in a child process via SubprocessBackend. A child process can be hard-killed: on a recv-timeout the watchdog calls proc.kill(), reclaiming VRAM/device, and the next request transparently respawns a fresh sidecar. The in-process engine remains the default, so existing users see no change; this is an opt-in for unattended / scheduled / reaction-triggered synthesis where a stuck job must self-recover instead of hanging until a manual restart. Base-class and mitigation changes that ship with it: - SubprocessBackend.generate() now consumes non-terminal {"op":"progress"} frames a sidecar emits during a cold load (previously the first cold generate after spawn failed, then worked on retry). Additive: engines that reply with audio directly are unaffected. - recv_timeout_s is overridable per engine (default 60s unchanged); the new engine sets it to the generate budget so a long-but-valid synth is not falsely killed while a wedged one still is. - make_room_before_generate(): free idle GPU memory before a warm, heavy generate. The cold-load path already evicted; the warm path skipped it, so a long synth on a VRAM-tight MPS box could contend its way into the budget. Verified end-to-end against the live model (cold / warm / recovery-after-kill) and under a sustained + concurrent-pressure soak: killed-worker recovery 5/5, chunked long text 9/9, no memory leak. * Address review: install_hint + move make_room into get_model - Add `omnivoice-subprocess` to `_INSTALL_HINTS`; the test_install_hints_cover_all_registered_backends gate requires every registered backend to carry one (this was the CI failure). - Move the warm-generate VRAM eviction out of the /generate and /v1/audio/speech routes and into get_model()'s warm-return path, so EVERY native TTS generate is covered (REST, WS TTS, dub, batch, audiobook), not just the two REST routes. Drops the now-redundant per-route wiring. (Greptile P1: the per-route placement missed the other generation surfaces.) * Address review: drop dead long-text eviction path; log probe failure - _should_make_room_for_generate: the long-text headroom boost became dead code once the eviction moved into get_model() (which has no text), so the long-text branch never fired. Removed the text param, the long-text threshold/multiplier branch, and the now-unused _env_float helper. The core RAM-tight gate (the part that matters on a starved box) is unchanged. - Log the available_memory probe failure at debug instead of silently swallowing it (CodeRabbit: silent swallow breaks the debug trail). - Tests updated for the text-agnostic policy. * fix(engines): stop subprocess generate() self-deadlock on 1-worker pools SubprocessBackend.generate() acquires a GPU-pool slot for accounting, but /v1/audio/speech and /generate dispatch backend.generate() via run_on_gpu_pool_guarded, i.e. already ON a pool worker. On a 1-worker pool (MPS) the inner pool.submit queued behind the very job running it and slot_future.result(timeout=10) raised before the sidecar ever spawned, so omnivoice-subprocess (and every other subprocess engine on MPS) surfaced the in-process 300s-abandon instead of synthesizing. Skip the slot acquisition when current_thread() is already a gpu-pool worker; the outer guard already accounts for the slot. Direct callers (off the pool) still acquire one. Regression test added (generate on a pool worker). * Address review: reword slot-skip comment (fixes watermark-coverage CI) + simplify - The slot-skip comment said "dispatch backend.generate() via", and test_watermark_route_coverage's _SYNTH_CALL regex matches the literal backend.generate( anywhere in a module, so it counted subprocess_backend.py as a synthesis producer that must reference mark_synthetic (it doesn't — the routes apply mark_synthetic; the engine sits below the chokepoint, like tts_backend.py). Reworded to "dispatch generate() via". - Fold in the simplify refinement: single negated predicate, import+pool moved into the acquire branch.
106 lines
4.2 KiB
Python
106 lines
4.2 KiB
Python
"""omnivoice-subprocess: the resident OmniVoice TTS engine in a crash-isolated
|
|
sidecar process (#730/#1190).
|
|
|
|
The default ``omnivoice`` engine runs in-process on the GPU ``ThreadPoolExecutor``.
|
|
When a generate or load there exceeds its execution budget the pool is "reset"
|
|
but the abandoned worker *thread* cannot be killed (Python cannot interrupt a
|
|
native torch/MPS call), so it holds the MPS device until it finishes on its
|
|
own, and every later synth contends with the zombie and hangs.
|
|
|
|
This engine runs the SAME OmniVoice model in a child process via
|
|
:class:`SubprocessBackend`. A child process CAN be hard-killed: on a recv
|
|
timeout the parent's watchdog calls ``proc.kill()``, reclaiming the child's
|
|
VRAM/device, and the next request transparently respawns a fresh sidecar. That
|
|
is the one thing the in-process engine structurally cannot do.
|
|
|
|
OPT-IN (Settings -> Engines, or ``OMNIVOICE_TTS_BACKEND=omnivoice-subprocess``);
|
|
the in-process ``omnivoice`` stays the default so existing users see no change.
|
|
|
|
Tradeoff vs the in-process engine: identical model and quality, a little extra
|
|
per-call overhead (one stdio round-trip), and it does not carry the native
|
|
advanced-parameter surface (``t_shift`` / ``layer_penalty_factor`` /
|
|
``position_temperature`` / ``class_temperature``) or parent-side seed
|
|
determinism, because the generic ``backend.generate`` path does not forward
|
|
those. Acceptable for unattended / reaction-triggered use where reliability
|
|
matters more than those controls.
|
|
|
|
Unlike IndexTTS / dots.tts / Supertonic-3, this sidecar runs under the PARENT
|
|
interpreter (``venv_python() -> sys.executable``): the goal here is crash
|
|
isolation, not dependency isolation, and the OmniVoice engine uses the host's
|
|
own pins.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
import os
|
|
import sys
|
|
from pathlib import Path
|
|
from typing import TYPE_CHECKING
|
|
|
|
from services.subprocess_backend import SubprocessBackend
|
|
|
|
if TYPE_CHECKING:
|
|
import torch # noqa: F401
|
|
|
|
logger = logging.getLogger("omnivoice.omnivoice_subprocess")
|
|
|
|
|
|
class OmniVoiceSubprocessBackend(SubprocessBackend):
|
|
"""The resident OmniVoice model in a killable sidecar process."""
|
|
|
|
id = "omnivoice-subprocess"
|
|
display_name = "OmniVoice (subprocess-isolated, killable on timeout)"
|
|
_DEFAULT_SAMPLE_RATE = 24000
|
|
gpu_compat = ("cuda", "mps", "cpu")
|
|
# Match OmniVoiceBackend: the measured floor below which a render that
|
|
# should take seconds runs for minutes (the #1226/#1222 4 GB reports).
|
|
min_vram_gb = 6.0
|
|
|
|
@classmethod
|
|
def is_available(cls) -> tuple[bool, str]:
|
|
# Same probe as OmniVoiceBackend: the package must be importable. The
|
|
# interpreter is the parent's own (sys.executable), so there is no
|
|
# separate venv to validate.
|
|
try:
|
|
import omnivoice.models.omnivoice # noqa: F401
|
|
except Exception as e:
|
|
return False, f"omnivoice package missing: {e}"
|
|
return True, "ready"
|
|
|
|
@classmethod
|
|
def venv_python(cls) -> Path:
|
|
# Same interpreter as the parent: this engine isolates for crash
|
|
# recovery, not dependency pins, so it needs no dedicated venv.
|
|
return Path(sys.executable)
|
|
|
|
@classmethod
|
|
def sidecar_script(cls) -> Path:
|
|
return Path(__file__).resolve().parent / "main.py"
|
|
|
|
@property
|
|
def recv_timeout_s(self) -> float:
|
|
"""Override the base 60s recv timeout.
|
|
|
|
Aligns the kill deadline with the generate budget: a long-but-valid
|
|
OmniVoice synth (which can take tens of seconds) is not falsely killed,
|
|
while a genuinely wedged one is hard-killed and its VRAM reclaimed at
|
|
the deadline. That reclaim is the concrete behavior the in-process
|
|
engine lacks (it abandons but never frees the device).
|
|
"""
|
|
try:
|
|
return max(30.0, float(os.environ.get("OMNIVOICE_SIDECAR_RECV_TIMEOUT_S", "300")))
|
|
except (ValueError, TypeError):
|
|
return 300.0
|
|
|
|
@property
|
|
def sample_rate(self) -> int:
|
|
return self._DEFAULT_SAMPLE_RATE
|
|
|
|
@property
|
|
def supported_languages(self) -> list[str]:
|
|
# OmniVoice advertises 600+ zero-shot; "multi" is the honest tag.
|
|
return ["multi"]
|
|
|
|
|
|
__all__ = ["OmniVoiceSubprocessBackend"]
|