* feat(engines): add omnivoice-subprocess, a crash-isolated TTS engine
The default in-process OmniVoice engine runs on the GPU ThreadPoolExecutor.
When a generate or load exceeds its execution budget the pool is "reset", but
the abandoned worker thread cannot be killed (Python cannot interrupt a native
torch/MPS call), so it keeps holding the device until it finishes on its own
and later synths queue behind it and hang. The reset restores pool capacity
but not the device. This is the residual root cause behind the closed#730
and #1190: the messaging/reset mitigations address the symptom, not the
device-holding zombie.
Add an opt-in `omnivoice-subprocess` engine that runs the same model in a
child process via SubprocessBackend. A child process can be hard-killed: on a
recv-timeout the watchdog calls proc.kill(), reclaiming VRAM/device, and the
next request transparently respawns a fresh sidecar. The in-process engine
remains the default, so existing users see no change; this is an opt-in for
unattended / scheduled / reaction-triggered synthesis where a stuck job must
self-recover instead of hanging until a manual restart.
Base-class and mitigation changes that ship with it:
- SubprocessBackend.generate() now consumes non-terminal {"op":"progress"}
frames a sidecar emits during a cold load (previously the first cold
generate after spawn failed, then worked on retry). Additive: engines that
reply with audio directly are unaffected.
- recv_timeout_s is overridable per engine (default 60s unchanged); the new
engine sets it to the generate budget so a long-but-valid synth is not
falsely killed while a wedged one still is.
- make_room_before_generate(): free idle GPU memory before a warm, heavy
generate. The cold-load path already evicted; the warm path skipped it, so a
long synth on a VRAM-tight MPS box could contend its way into the budget.
Verified end-to-end against the live model (cold / warm / recovery-after-kill)
and under a sustained + concurrent-pressure soak: killed-worker recovery 5/5,
chunked long text 9/9, no memory leak.
* Address review: install_hint + move make_room into get_model
- Add `omnivoice-subprocess` to `_INSTALL_HINTS`; the
test_install_hints_cover_all_registered_backends gate requires every
registered backend to carry one (this was the CI failure).
- Move the warm-generate VRAM eviction out of the /generate and
/v1/audio/speech routes and into get_model()'s warm-return path, so EVERY
native TTS generate is covered (REST, WS TTS, dub, batch, audiobook), not
just the two REST routes. Drops the now-redundant per-route wiring.
(Greptile P1: the per-route placement missed the other generation surfaces.)
* Address review: drop dead long-text eviction path; log probe failure
- _should_make_room_for_generate: the long-text headroom boost became dead
code once the eviction moved into get_model() (which has no text), so the
long-text branch never fired. Removed the text param, the long-text
threshold/multiplier branch, and the now-unused _env_float helper. The core
RAM-tight gate (the part that matters on a starved box) is unchanged.
- Log the available_memory probe failure at debug instead of silently
swallowing it (CodeRabbit: silent swallow breaks the debug trail).
- Tests updated for the text-agnostic policy.
* fix(engines): stop subprocess generate() self-deadlock on 1-worker pools
SubprocessBackend.generate() acquires a GPU-pool slot for accounting, but
/v1/audio/speech and /generate dispatch backend.generate() via
run_on_gpu_pool_guarded, i.e. already ON a pool worker. On a 1-worker pool
(MPS) the inner pool.submit queued behind the very job running it and
slot_future.result(timeout=10) raised before the sidecar ever spawned, so
omnivoice-subprocess (and every other subprocess engine on MPS) surfaced the
in-process 300s-abandon instead of synthesizing.
Skip the slot acquisition when current_thread() is already a gpu-pool worker;
the outer guard already accounts for the slot. Direct callers (off the pool)
still acquire one. Regression test added (generate on a pool worker).
* Address review: reword slot-skip comment (fixes watermark-coverage CI) + simplify
- The slot-skip comment said "dispatch backend.generate() via", and
test_watermark_route_coverage's _SYNTH_CALL regex matches the literal
backend.generate( anywhere in a module, so it counted subprocess_backend.py
as a synthesis producer that must reference mark_synthetic (it doesn't — the
routes apply mark_synthetic; the engine sits below the chokepoint, like
tts_backend.py). Reworded to "dispatch generate() via".
- Fold in the simplify refinement: single negated predicate, import+pool
moved into the acquire branch.
Paolo Antinori
47c51698c9