Paolo Antinori
47c51698c9
* feat(engines): add omnivoice-subprocess, a crash-isolated TTS engine The default in-process OmniVoice engine runs on the GPU ThreadPoolExecutor. When a generate or load exceeds its execution budget the pool is "reset", but the abandoned worker thread cannot be killed (Python cannot interrupt a native torch/MPS call), so it keeps holding the device until it finishes on its own and later synths queue behind it and hang. The reset restores pool capacity but not the device. This is the residual root cause behind the closed #730 and #1190: the messaging/reset mitigations address the symptom, not the device-holding zombie. Add an opt-in `omnivoice-subprocess` engine that runs the same model in a child process via SubprocessBackend. A child process can be hard-killed: on a recv-timeout the watchdog calls proc.kill(), reclaiming VRAM/device, and the next request transparently respawns a fresh sidecar. The in-process engine remains the default, so existing users see no change; this is an opt-in for unattended / scheduled / reaction-triggered synthesis where a stuck job must self-recover instead of hanging until a manual restart. Base-class and mitigation changes that ship with it: - SubprocessBackend.generate() now consumes non-terminal {"op":"progress"} frames a sidecar emits during a cold load (previously the first cold generate after spawn failed, then worked on retry). Additive: engines that reply with audio directly are unaffected. - recv_timeout_s is overridable per engine (default 60s unchanged); the new engine sets it to the generate budget so a long-but-valid synth is not falsely killed while a wedged one still is. - make_room_before_generate(): free idle GPU memory before a warm, heavy generate. The cold-load path already evicted; the warm path skipped it, so a long synth on a VRAM-tight MPS box could contend its way into the budget. Verified end-to-end against the live model (cold / warm / recovery-after-kill) and under a sustained + concurrent-pressure soak: killed-worker recovery 5/5, chunked long text 9/9, no memory leak. * Address review: install_hint + move make_room into get_model - Add `omnivoice-subprocess` to `_INSTALL_HINTS`; the test_install_hints_cover_all_registered_backends gate requires every registered backend to carry one (this was the CI failure). - Move the warm-generate VRAM eviction out of the /generate and /v1/audio/speech routes and into get_model()'s warm-return path, so EVERY native TTS generate is covered (REST, WS TTS, dub, batch, audiobook), not just the two REST routes. Drops the now-redundant per-route wiring. (Greptile P1: the per-route placement missed the other generation surfaces.) * Address review: drop dead long-text eviction path; log probe failure - _should_make_room_for_generate: the long-text headroom boost became dead code once the eviction moved into get_model() (which has no text), so the long-text branch never fired. Removed the text param, the long-text threshold/multiplier branch, and the now-unused _env_float helper. The core RAM-tight gate (the part that matters on a starved box) is unchanged. - Log the available_memory probe failure at debug instead of silently swallowing it (CodeRabbit: silent swallow breaks the debug trail). - Tests updated for the text-agnostic policy. * fix(engines): stop subprocess generate() self-deadlock on 1-worker pools SubprocessBackend.generate() acquires a GPU-pool slot for accounting, but /v1/audio/speech and /generate dispatch backend.generate() via run_on_gpu_pool_guarded, i.e. already ON a pool worker. On a 1-worker pool (MPS) the inner pool.submit queued behind the very job running it and slot_future.result(timeout=10) raised before the sidecar ever spawned, so omnivoice-subprocess (and every other subprocess engine on MPS) surfaced the in-process 300s-abandon instead of synthesizing. Skip the slot acquisition when current_thread() is already a gpu-pool worker; the outer guard already accounts for the slot. Direct callers (off the pool) still acquire one. Regression test added (generate on a pool worker). * Address review: reword slot-skip comment (fixes watermark-coverage CI) + simplify - The slot-skip comment said "dispatch backend.generate() via", and test_watermark_route_coverage's _SYNTH_CALL regex matches the literal backend.generate( anywhere in a module, so it counted subprocess_backend.py as a synthesis producer that must reference mark_synthetic (it doesn't — the routes apply mark_synthetic; the engine sits below the chokepoint, like tts_backend.py). Reworded to "dispatch generate() via". - Fold in the simplify refinement: single negated predicate, import+pool moved into the acquire branch.
3.4 KiB
3.4 KiB
OmniVoice (subprocess-isolated) Engine
The omnivoice-subprocess engine runs the same resident OmniVoice model as
the default omnivoice engine, but in a crash-isolated child process so a
wedged generation can be hard-killed and its VRAM/device reclaimed.
Why this engine exists
The default omnivoice engine runs in-process on the GPU worker pool. On
VRAM-tight machines (Apple Silicon MPS especially) a heavy generation or model
load can exceed its execution budget. When that happens the worker is
"abandoned" but cannot be killed (Python cannot interrupt a native torch /
MPS call), so it keeps holding the GPU device until it finishes on its own, and
every later synth queues behind it and hangs (#730 / #1190).
omnivoice-subprocess runs the model in a child process spawned via the same
SubprocessBackend primitive used by IndexTTS, Supertonic-3, and dots.tts. A
child process can be hard-killed: on a timeout the parent kills it
(proc.kill()), freeing its VRAM/device, and the next request transparently
respawns a fresh sidecar. That is the one thing the in-process engine
structurally cannot do.
When to use it
- Unattended / scheduled / reaction-triggered synthesis where a stuck job must recover on its own instead of hanging until a manual restart.
- VRAM-starved MPS hosts that hit the abandoned-worker cascade.
For interactive single-shot use on a machine with comfortable VRAM, the default
in-process omnivoice engine is faster (no stdio round-trip) and remains the
default.
Selecting it
- Settings -> Engines, or
OMNIVOICE_TTS_BACKEND=omnivoice-subprocess
It is opt-in; the in-process engine stays the default, so existing setups see no change.
Platform support
- CUDA, MPS, and CPU (same as the in-process OmniVoice engine).
- No extra install. Unlike IndexTTS / dots.tts / Supertonic-3, this sidecar
runs under OmniVoice's own interpreter, because the goal here is crash
isolation, not dependency isolation. If the default
omnivoiceengine works for you, this one is ready too.
Tradeoffs vs the in-process omnivoice engine
- Identical model and output quality.
- Slightly higher per-call latency (one stdio round-trip per synth).
- A wedged generation is killed and recovered at the recv-timeout deadline
(
OMNIVOICE_SIDECAR_RECV_TIMEOUT_S, default 300s, aligned with the generate budget) instead of hanging indefinitely. - It does not carry the native advanced-parameter surface
(
t_shift/layer_penalty_factor/position_temperature/class_temperature) or parent-side seed determinism, because the generic engine path does not forward those. For plain voice-clone and design synthesis this is a non-issue. - The recv-timeout deadline is per call and assumes the route's text chunking:
/generateand/v1/audio/speechsplit long text into pieces of at mostmax_chunk_charsbefore calling the engine, so each call stays short. A single very long unchunkedgenerate()can exceed the deadline and be killed; that is the watchdog working as intended, not a hang.
Tuning
| Env var | Default | Purpose |
|---|---|---|
OMNIVOICE_SIDECAR_RECV_TIMEOUT_S |
300 |
Seconds to wait for a synth frame before hard-killing the sidecar (floored at 30s). |
OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S |
300 |
Idle seconds before the sidecar is reaped to free its VRAM (shared with all subprocess engines). |