fix(tts): bound + reset the GPU pool on a hung generate so it can't brick the backend (#730 class) (#851)

* fix(tts): bound + reset the GPU pool on a hung generate so it can't brick the backend (#730 class)

A GPU job that wedges on some Windows+CUDA setups occupies its worker
forever — run_in_executor can't cancel the thread — so on the 1–2 worker
pools we ship, one stuck job starves every other request and the next
action surfaces as the misleading "Can't reach the local backend" even
though the process is alive.

ASR/dub/model-load already bound+reset the pool on hang (#730). The TTS
**generate** paths (generation.py, tts_stream.py) were the last unguarded
GPU dispatch — and the residual on-main reports (#850 #802 #755 #723 #721,
plus the 0.3.7 generate cohort) all fail on generate:start (audio).

- model_manager: add run_on_gpu_pool_guarded() + GpuJobTimeoutError, a
  generalized version of the ASR guard so every GPU dispatch shares one
  bound+reset recovery path. Env-tunable via OMNIVOICE_GENERATE_TIMEOUT_S
  (default 300s).
- generation.py: route both inference branches + the reference-clip
  transcribe through the guard; map a timeout to an actionable 503.
- tts_stream.py: same guard on the streaming path (timeout → error frame).
- test_generate_timeout_730: fail-before/pass-after regression (timeout
  resets pool + restores capacity, happy path, env override, no-reset exec).
- docs + CHANGELOG: extend troubleshooting §14 to cover generate; document
  the new env var.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(tts): extend the GPU-pool hang guard to batch/dub/archetype/openai-compat generate (#730 class)

The generate-hang class wasn't only in Studio + streaming: batch generate,
the dub per-segment + preview generate, archetype preview render, and the
OpenAI-compat /v1/audio/speech path all dispatched the TTS model to the GPU
pool with no wall-clock bound either. Any one of them wedging on a
Windows+CUDA hang starves the pool and bricks the backend the same way.

Route all of them through run_on_gpu_pool_guarded so the whole class is
closed — a hung generate anywhere resets the pool and returns an actionable
timeout instead of a dead backend. Batch/dub recover per-segment on a fresh
worker; drop the now-dead loop/_gpu_pool/asyncio locals ruff flagged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -50,6 +50,22 @@ The bundled TTS model package (`pyproject.toml`) is versioned independently.
 
 ### Fixed
 
+- **A hung TTS generate can no longer brick the backend ("Can't reach the local
+  backend").** A GPU job that wedges on some Windows + CUDA setups occupies its
+  worker forever — Python can't cancel the thread — so on the 1–2 worker pools we
+  ship, one stuck job starved every other request and the next action surfaced as
+  the misleading "Can't reach the local backend" even though the process was
+  alive. ASR/dub/model-load already bounded and reset the pool on hang (#730); but
+  **every generate path** — Studio synthesis, the streaming path, batch, the dub
+  per-segment + preview render, archetype previews, and the OpenAI-compatible
+  `/v1/audio/speech` API — was still an unguarded GPU dispatch, and the residual
+  reports all failed on `generate:start (audio)`. Every one is now bounded by the
+  same wall-clock guard
+  (`OMNIVOICE_GENERATE_TIMEOUT_S`, default 300s) that abandons the wedged worker
+  and rebuilds the pool, so capacity is restored automatically and you get an
+  actionable timeout instead of a dead backend. Closes the whole class of
+  GPU-job-hang reports (#850, #802, #755, #723, #721, and the 0.3.7 cohort).
+
 - **The "TRANSLATION FAILED" banner now dismisses and clears itself.** The Dub
   translation-error banner used to be sticky — it survived a successful re-try and
   never went away. It now has a close (×), auto-clears on the next corrective

backend/api/routers/archetypes.py

@@ -18,7 +18,6 @@ Design notes
 """
 from __future__ import annotations
 
-import asyncio
 import hashlib
 import logging
 import os
@@ -137,7 +136,7 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
     from api.routers.generation import (  # noqa: WPS433 — intentional lazy import
         get_model,
         _run_inference,
-        _gpu_pool,
+        run_on_gpu_pool_guarded,
         _safe_torchaudio_save,
     )
 
@@ -147,8 +146,6 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
         language = None
     text = (a.get("sample_script") or "").strip() or _FALLBACK_SCRIPT
 
-    loop = asyncio.get_running_loop()
-
     def _infer(seed: int):
         return _run_inference(
             model,          # _model
@@ -171,14 +168,18 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
             "broadcast",    # effect_preset
         )
 
-    audio_tensor = await loop.run_in_executor(_gpu_pool, _infer, _PREVIEW_SEED)
+    # Bounded + pool-reset on hang so a wedged preview render can't starve the
+    # GPU pool and brick the backend (#730 class).
+    audio_tensor = await run_on_gpu_pool_guarded(
+        lambda: _infer(_PREVIEW_SEED), what="Archetype preview generate")
     if _is_unusable_audio(audio_tensor):
         # Blank OR a degenerate tonal buzz — retry once on a different seed to
         # step off the bad diffusion trajectory. Static message only: the
         # archetype id is request-derived (CodeQL log-injection); the seed is a
         # module constant, safe to log.
         logger.warning("Archetype rendered unusable at seed %d — retrying once", _PREVIEW_SEED)
-        audio_tensor = await loop.run_in_executor(_gpu_pool, _infer, _PREVIEW_SEED + 1)
+        audio_tensor = await run_on_gpu_pool_guarded(
+            lambda: _infer(_PREVIEW_SEED + 1), what="Archetype preview generate")
     if _is_unusable_audio(audio_tensor):
         raise RuntimeError("the voice engine returned no audible audio for this archetype")
 

backend/api/routers/batch.py

@@ -142,7 +142,7 @@ async def _run_batch_pipeline(job_id: str, job: dict):
     _set_progress(job, "transcribe", 0)
 
     from services.asr_backend import get_active_asr_backend
-    from services.model_manager import _gpu_pool, _cpu_pool
+    from services.model_manager import _gpu_pool, _cpu_pool, run_on_gpu_pool_guarded
     from services.segmentation import (
         segment_transcript, assign_speakers_heuristic,
     )
@@ -316,7 +316,9 @@ async def _run_batch_pipeline(job_id: str, job: dict):
                     return torch.zeros(1, int(dur * sr))
 
             try:
-                audio_tensor = await loop.run_in_executor(_gpu_pool, _gen)
+                # Bounded + pool-reset on hang so a wedged batch segment can't
+                # starve the GPU pool and brick the backend (#730 class).
+                audio_tensor = await run_on_gpu_pool_guarded(_gen, what="Batch generate")
 
                 # Fit to slot
                 target_samples_seg = int(seg_duration * sr)

backend/api/routers/dub_generate.py

@@ -11,7 +11,7 @@ from core.db import db_conn
 from core.config import DUB_DIR, VOICES_DIR, dub_seg_path
 from core.tasks import task_manager
 from schemas.requests import DubRequest
-from services.model_manager import get_model, _gpu_pool
+from services.model_manager import get_model, _gpu_pool, run_on_gpu_pool_guarded
 from services.audio_dsp import apply_mastering, normalize_audio, apply_effects_chain, get_effect_chain
 from services.audio_io import atomic_save_wav, _safe_torchaudio_save
 from services.ffmpeg_utils import (
@@ -541,10 +541,14 @@ async def dub_generate(job_id: str, req: DubRequest):
                 # where dur_s is the slot hint.
                 _dur_for_tts = seg_duration if strategy == "strict_slot" else None
 
-                audio_tensor = await loop.run_in_executor(
-                    _gpu_pool, _gen,
-                    seg.text, seg_lang, seg_instruct, _dur_for_tts,
-                    _num_step, req.guidance_scale, seg_speed, seg_profile, seg_effect_preset,
+                # Bounded + pool-reset on hang so a wedged dub segment can't
+                # starve the GPU pool and brick the backend (#730 class).
+                audio_tensor = await run_on_gpu_pool_guarded(
+                    lambda: _gen(
+                        seg.text, seg_lang, seg_instruct, _dur_for_tts,
+                        _num_step, req.guidance_scale, seg_speed, seg_profile, seg_effect_preset,
+                    ),
+                    what="Dub generate",
                 )
                 _t_tts += time.perf_counter() - _t_tts_0
 
@@ -1152,8 +1156,9 @@ async def preview_segment(job_id: str, req: SegmentPreviewRequest):
         )
         return normalize_audio(mastered, target_dBFS=-2.0)
 
-    loop = asyncio.get_running_loop()
-    audio_tensor = await loop.run_in_executor(_gpu_pool, _gen)
+    # Bounded + pool-reset on hang so a wedged preview generate can't starve the
+    # GPU pool and brick the backend (#730 class).
+    audio_tensor = await run_on_gpu_pool_guarded(_gen, what="Dub preview generate")
 
     sr = getattr(_model, "sampling_rate", 24000)
     buf = io.BytesIO()

backend/api/routers/generation.py

@@ -15,7 +15,10 @@ from fastapi.responses import StreamingResponse
 import sqlite3
 from core.db import db_conn, ensure_schema
 from core.config import OUTPUTS_DIR, VOICES_DIR
-from services.model_manager import get_model, _gpu_pool
+import functools
+from services.model_manager import (
+    get_model, _gpu_pool, run_on_gpu_pool_guarded, GpuJobTimeoutError,
+)
 from services.audio_io import _safe_torchaudio_save
 from core import event_bus
 from omnivoice.utils.voice_design import heal_design_instruct
@@ -565,9 +568,19 @@ async def generate_speech(
     # fallback behaves exactly as before.
     if ref_audio_path and not ref_text:
         from services.asr_backend import transcribe_reference
-        ref_text = await asyncio.get_running_loop().run_in_executor(
-            _gpu_pool, transcribe_reference, ref_audio_path
-        )
+        # Same #730 hang risk as any whisperx transcribe — bound + reset the pool
+        # so a wedged reference transcribe can't brick the backend. This path is
+        # best-effort (transcribe_reference returns None on failure → the model's
+        # built-in ASR fallback), so a timeout degrades to None rather than
+        # failing the whole generate.
+        try:
+            ref_text = await run_on_gpu_pool_guarded(
+                functools.partial(transcribe_reference, ref_audio_path),
+                what="Reference transcribe",
+            )
+        except GpuJobTimeoutError as e:
+            logger.warning("reference transcribe hung (%s); using model ASR fallback", e)
+            ref_text = None
 
     # #526: materialize a concrete seed when none was supplied (and no profile
     # pinned one) so the take is reproducible and we can hand it back via the
@@ -611,24 +624,32 @@ async def generate_speech(
     try:
         loop = asyncio.get_running_loop()
         if _backend is not None:
-            audio_tensor = await loop.run_in_executor(
-                _gpu_pool, _run_backend_inference,
-                _backend, text, language, ref_audio_path, ref_text, instruct,
-                duration, num_step, guidance_scale, speed, denoise,
-                postprocess_output, used_seed, effect_preset,
-                max_chunk_chars, crossfade_ms,
+            # Bounded + pool-reset on hang so a wedged generate can't starve the
+            # GPU pool and brick the backend ("can't reach backend", #730 class).
+            audio_tensor = await run_on_gpu_pool_guarded(
+                functools.partial(
+                    _run_backend_inference,
+                    _backend, text, language, ref_audio_path, ref_text, instruct,
+                    duration, num_step, guidance_scale, speed, denoise,
+                    postprocess_output, used_seed, effect_preset,
+                    max_chunk_chars, crossfade_ms,
+                ),
+                what="TTS generate",
             )
             # Read after generation: engines with lazy model loading report
             # their real rate only once weights are up.
             sample_rate = _backend.sample_rate
         else:
-            audio_tensor = await loop.run_in_executor(
-                _gpu_pool, _run_inference,
-                _model, text, language, ref_audio_path, ref_text, instruct, duration,
-                num_step, guidance_scale, speed, t_shift, denoise,
-                postprocess_output, layer_penalty_factor, position_temperature,
-                class_temperature, used_seed, effect_preset,
-                max_chunk_chars, crossfade_ms,
+            audio_tensor = await run_on_gpu_pool_guarded(
+                functools.partial(
+                    _run_inference,
+                    _model, text, language, ref_audio_path, ref_text, instruct, duration,
+                    num_step, guidance_scale, speed, t_shift, denoise,
+                    postprocess_output, layer_penalty_factor, position_temperature,
+                    class_temperature, used_seed, effect_preset,
+                    max_chunk_chars, crossfade_ms,
+                ),
+                what="TTS generate",
             )
             sample_rate = _model.sampling_rate
         # Invisible AudioSeal provenance watermark on the final audio. Embedding
@@ -708,6 +729,12 @@ async def generate_speech(
         )
     except HTTPException:
         raise
+    except GpuJobTimeoutError as e:
+        # A wedged GPU generate — the pool was already reset to restore capacity
+        # (#730 class). Report the actionable timeout instead of the misleading
+        # "can't reach backend" the frontend shows when the pool starves.
+        logger.error("Generate timed out: %s", e)
+        raise HTTPException(status_code=503, detail=str(e)) from e
     except ValueError as e:
         logger.error("Validation failed: %s", e)
         raise HTTPException(status_code=400, detail=str(e)) from e

backend/api/routers/openai_compat.py

@@ -22,7 +22,6 @@ from __future__ import annotations
 import io
 import logging
 import os
-import asyncio
 import tempfile
 from typing import Literal, Optional
 
@@ -30,7 +29,7 @@ from fastapi import APIRouter, File, Form, HTTPException, UploadFile
 from fastapi.responses import StreamingResponse
 from pydantic import BaseModel, Field
 
-from services.model_manager import _gpu_pool
+from services.model_manager import _gpu_pool, run_on_gpu_pool_guarded
 
 logger = logging.getLogger("omnivoice.openai_compat")
 
@@ -313,8 +312,10 @@ async def create_speech(req: SpeechRequest):
             kw["voice"] = voice
 
     try:
-        loop = asyncio.get_running_loop()
-        wav, sr = await loop.run_in_executor(_gpu_pool, _run_tts, backend, req.input, kw)
+        # Bounded + pool-reset on hang so a wedged TTS request can't starve the
+        # GPU pool and brick the backend (#730 class).
+        wav, sr = await run_on_gpu_pool_guarded(
+            lambda: _run_tts(backend, req.input, kw), what="OpenAI TTS generate")
     except Exception as e:
         logger.exception("OpenAI TTS failed: %s", e)
         raise HTTPException(status_code=500, detail=str(e))

backend/api/routers/tts_stream.py

@@ -182,8 +182,8 @@ async def ws_tts(websocket: WebSocket):
                     sentences = [text]
 
                 # Run generation in the GPU pool
-                from services.model_manager import _gpu_pool
-                loop = asyncio.get_running_loop()
+                import functools
+                from services.model_manager import run_on_gpu_pool_guarded
 
                 def _generate(sentence_text):
                     from services.audio_dsp import apply_mastering, normalize_audio
@@ -204,8 +204,13 @@ async def ws_tts(websocket: WebSocket):
                 started = False
 
                 for sentence in sentences:
-                    wav_tensor, sr = await loop.run_in_executor(
-                        _gpu_pool, _generate, sentence
+                    # Bounded + pool-reset on hang so a wedged generate can't
+                    # starve the GPU pool and brick the backend (#730 class). On
+                    # timeout GpuJobTimeoutError propagates to the handler below,
+                    # which sends an actionable error frame.
+                    wav_tensor, sr = await run_on_gpu_pool_guarded(
+                        functools.partial(_generate, sentence),
+                        what="TTS generate",
                     )
 
                     if not started:

backend/services/model_manager.py

@@ -198,6 +198,68 @@ def __getattr__(name: str):
         return _get_gpu_pool()
     raise AttributeError(f"module 'services.model_manager' has no attribute {name!r}")
 
+
+# ── GPU-job timeout guard (#730 class; residual #850/#802/#755 …) ─────
+# A blocking GPU job that wedges on a Windows+CUDA hang keeps occupying its
+# worker forever — run_in_executor can't cancel the thread. With a 1–2 worker
+# pool that starves *every* other request, so the next user action surfaces as
+# the misleading "Can't reach the local backend" even though the process is
+# alive. ASR/dub/model-load already bound+reset on hang (run_transcribe_guarded,
+# _reset_pool_on_wedge, _load_model_with_timeout); the TTS **generate** paths
+# (generation.py, tts_stream.py) were the last unguarded dispatch — and the
+# residual on-main reports all fail on generate:start (audio). This is the same
+# guard generalised so every GPU dispatch shares one recovery path.
+GPU_JOB_TIMEOUT_S = float(os.environ.get("OMNIVOICE_GENERATE_TIMEOUT_S", "300.0"))
+
+
+class GpuJobTimeoutError(TimeoutError):
+    """A GPU-pool job exceeded its wall-clock bound and was abandoned.
+
+    The backend is alive — the job was too heavy for the available compute
+    (most often a VRAM-starved GPU). Pool capacity is restored automatically by
+    resetting the pool; the message carries the durable fix.
+    """
+
+
+async def run_on_gpu_pool_guarded(fn, *, what: str = "GPU job",
+                                  timeout: float = GPU_JOB_TIMEOUT_S,
+                                  executor=None):
+    """Run blocking ``fn`` on the GPU pool with a hard wall-clock bound.
+
+    On timeout, ``reset()`` the pool (abandon the wedged worker so the next
+    submit gets a fresh one) and raise :class:`GpuJobTimeoutError`. ``fn`` must
+    be a zero-arg callable — wrap args with ``functools.partial`` at the call
+    site. Deliberately mirrors ``asr_backend.run_transcribe_guarded`` so every
+    GPU dispatch shares one bound+recover path (#730 class). Executors without
+    ``reset`` (a plain ThreadPoolExecutor in tests) still get the bound + error.
+    """
+    loop = asyncio.get_running_loop()
+    ex = executor if executor is not None else _get_gpu_pool()
+    fut = loop.run_in_executor(ex, fn)
+    try:
+        return await asyncio.wait_for(fut, timeout=timeout)
+    except asyncio.TimeoutError:
+        _reset = getattr(ex, "reset", None)
+        if callable(_reset):
+            try:
+                _reset()
+                logger.warning(
+                    "%s exceeded %.0fs — abandoned the GPU-pool worker to "
+                    "restore capacity (#730).", what, timeout,
+                )
+            except Exception:
+                logger.exception("GPU pool reset after %s timeout failed", what)
+        raise GpuJobTimeoutError(
+            f"{what} exceeded {timeout:.0f}s and was abandoned — the backend is "
+            "running, but the job was too heavy for the available compute. Most "
+            "often the GPU is VRAM-starved (a resident model and this job contend "
+            "for memory). Capacity was restored automatically; for a durable fix "
+            "try shorter text, a lighter engine, or set the engine to CPU in "
+            "Settings → Models. (Raise OMNIVOICE_GENERATE_TIMEOUT_S for very long "
+            "single generations.)"
+        )
+
+
 model = None  # type: ignore
 _model_lock = asyncio.Lock()
 _last_used = time.time()

docs/install/troubleshooting.md

@@ -309,21 +309,25 @@ files land where the app looks.)
 
 **Linked issue:** [#622](https://github.com/debpalash/OmniVoice-Studio/issues/622)
 
-## 14. "Can't reach the local backend" *during* transcription / dubbing
+## 14. "Can't reach the local backend" *during* generation / transcription / dubbing
 
 **Symptom:** the app worked at startup (you reached the main menu and the model
-loaded), but the moment you **dub a video, transcribe, or dictate**, it spins for
-a long time and then shows **"Can't reach the local backend."** The backend log
-ends right after a line like `whisperx transcribing …tmpXXXX.wav` with nothing
-after it — i.e. the backend is **alive**, the *transcription* is what stalled.
+loaded), but the moment you **generate audio, dub a video, transcribe, or
+dictate**, it spins for a long time and then shows **"Can't reach the local
+backend."** The backend log ends right after a line like `whisperx transcribing
+…tmpXXXX.wav` (or a generate) with nothing after it — i.e. the backend is
+**alive**, the GPU *job* is what stalled.
 
 **Cause:** this is **not** a connection, download, or "network mirror" problem —
-the backend started fine. The ASR model (WhisperX/faster-whisper **large-v3**) is
-too heavy for the available compute and the transcribe call runs for minutes,
-which the UI surfaces as an unreachable backend. The usual trigger is **VRAM
-starvation on NVIDIA**: the resident TTS model and a large ASR model contend for
-memory on an 8 GB-class GPU (the log shows e.g. `GPU pool sized … 7.0 GB free`).
-CPU-only machines hit the same wall on long clips.
+the backend started fine. A GPU job (a **generate** on the TTS model, or an ASR
+transcribe with WhisperX/faster-whisper **large-v3**) is too heavy for the
+available compute and runs for minutes; because it wedges its GPU-pool worker,
+every *other* request — including the next generate and the health check — is
+starved, which the UI surfaces as an unreachable backend. The usual trigger is
+**VRAM starvation on NVIDIA**: models contend for memory on an 8 GB-class GPU
+(the log shows e.g. `GPU pool sized … 7.0 GB free`). CPU-only machines hit the
+same wall on long clips. This is the same root cause whether the last thing you
+did was `generate:start (audio)`, a dub, or a dictation.
 
 > There is **no "Network → Restricted/Global mirror" toggle** in Settings — that
 > control (the footer/Sharing **Network** button) is for **LAN sharing**, not
@@ -340,10 +344,14 @@ CPU-only machines hit the same wall on long clips.
 4. **Test with a 10-second clip** first — if that returns quickly, it confirms a
    compute/VRAM limit rather than a true hang.
 
-Newer builds **bound** whole-file transcription: instead of hanging, it now fails
-after a timeout with this exact guidance. Tune the bound with
-`OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S` (seconds; default 300) — **raise** it for
-very long single files, **lower** it to fail faster on a small machine.
+Newer builds **bound** every GPU job — whole-file transcription **and** TTS
+generation: instead of hanging forever and starving the backend, a wedged job now
+fails after a timeout with this exact guidance, and the worker pool is reset so
+capacity is restored automatically (no app restart needed). Tune the bounds with
+`OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S` (transcription) and
+`OMNIVOICE_GENERATE_TIMEOUT_S` (generation) — both in seconds, default 300.
+**Raise** them for very long single files/generations, **lower** them to fail
+faster on a small machine.
 
 ## Dub: "translation engine needs the optional … package"
 

tests/test_generate_timeout_730.py

@@ -0,0 +1,113 @@
+"""Regression (#730 class; residual #850/#802/#755): a wedged GPU **generate**
+must not brick the backend.
+
+Before this fix, the TTS generate paths (`generation.py`, `tts_stream.py`)
+dispatched to the GPU pool with no wall-clock bound and no recovery — unlike
+ASR/dub/model-load, which already bound+reset on hang. On the 1–2 worker pools
+we ship, one wedged generate (a Windows+CUDA hang) occupied its worker forever,
+starving every other request so the next user action surfaced as the misleading
+"Can't reach the local backend" even though the process was alive.
+
+`run_on_gpu_pool_guarded` gives every GPU dispatch the same bound+reset recovery:
+on timeout it abandons the wedged worker (pool `reset()`), restoring capacity,
+and raises `GpuJobTimeoutError` with an actionable message.
+"""
+from __future__ import annotations
+
+import asyncio
+import sys
+import threading
+
+import pytest
+
+
+@pytest.fixture
+def model_manager(monkeypatch):
+    for mod_name in ("core.config", "services.model_manager"):
+        if getattr(sys.modules.get(mod_name), "__file__", None) is None:
+            sys.modules.pop(mod_name, None)
+    import services.model_manager as mm
+    return mm
+
+
+def test_guard_times_out_resets_pool_and_restores_capacity(model_manager):
+    mm = model_manager
+    pool = mm._ResilientGpuPool()
+
+    release = threading.Event()
+
+    def _hang():  # a wedged generate that never returns on its own
+        release.wait(2.0)
+        return "late"
+
+    try:
+        # Force the inner pool to exist so we can prove reset() drops it.
+        assert pool._live_pool() is not None
+        assert pool._pool is not None
+
+        with pytest.raises(mm.GpuJobTimeoutError, match="abandoned"):
+            asyncio.run(
+                mm.run_on_gpu_pool_guarded(_hang, what="TTS generate",
+                                           timeout=0.2, executor=pool)
+            )
+
+        # The wedged worker was abandoned: the inner pool is dropped so the next
+        # submit builds a fresh one instead of queueing behind the hang.
+        assert pool._pool is None
+
+        # Capacity is genuinely restored — a follow-up job runs on a new worker
+        # even while the orphaned one is still stuck.
+        result = asyncio.run(
+            mm.run_on_gpu_pool_guarded(lambda: "ok", what="TTS generate",
+                                       timeout=5.0, executor=pool)
+        )
+        assert result == "ok"
+    finally:
+        release.set()  # let the orphaned worker exit immediately
+
+
+def test_guard_happy_path_returns_value(model_manager):
+    mm = model_manager
+    pool = mm._ResilientGpuPool()
+    try:
+        result = asyncio.run(
+            mm.run_on_gpu_pool_guarded(lambda: 42, what="TTS generate",
+                                       timeout=5.0, executor=pool)
+        )
+        assert result == 42
+    finally:
+        pool.shutdown(wait=False)
+
+
+def test_guard_timeout_env_default(model_manager, monkeypatch):
+    """The generate bound is env-overridable (parity with the ASR bound)."""
+    import importlib
+    monkeypatch.setenv("OMNIVOICE_GENERATE_TIMEOUT_S", "123.5")
+    mm = importlib.reload(model_manager)
+    try:
+        assert mm.GPU_JOB_TIMEOUT_S == 123.5
+    finally:
+        monkeypatch.delenv("OMNIVOICE_GENERATE_TIMEOUT_S", raising=False)
+        importlib.reload(mm)
+
+
+def test_guard_without_reset_still_bounds(model_manager):
+    """A plain executor (no `reset`, e.g. in other call sites/tests) still gets
+    the wall-clock bound + actionable error — reset is best-effort, not required.
+    """
+    from concurrent.futures import ThreadPoolExecutor
+    mm = model_manager
+    ex = ThreadPoolExecutor(max_workers=1)
+    release = threading.Event()
+
+    def _hang():
+        release.wait(2.0)
+
+    try:
+        with pytest.raises(mm.GpuJobTimeoutError):
+            asyncio.run(
+                mm.run_on_gpu_pool_guarded(_hang, timeout=0.2, executor=ex)
+            )
+    finally:
+        release.set()
+        ex.shutdown(wait=False)