`_reset_gpu_pool()` fires on a model-load timeout to recover a wedged worker —
it shut the ThreadPoolExecutor down and rebuilt a fresh one on next access. But
several request handlers (generation, dub_generate, dub_core, dub_translate,
openai_compat) did a *module-level* `from services.model_manager import
_gpu_pool`, capturing the executor object at import time. After a reset those
references pointed at the dead pool, so the next generate/dub/transcribe/
translate raised `RuntimeError: cannot schedule new futures after shutdown` —
surfacing as a 500 or "Can't reach the local backend" (#589#599).
Make `_gpu_pool` a single long-lived `_ResilientGpuPool` wrapper (a
concurrent.futures.Executor) whose *inner* ThreadPoolExecutor is swapped:
- every submit() resolves the live pool, and a submit that races a shutdown
rebuilds once and retries, so a stale captured reference self-heals;
- `_reset_gpu_pool()` now drops only the inner pool (fresh worker on retry)
while preserving the wrapper identity every importer holds;
- pool sizing stays lazy, so we still probe the device after torch's lazy
import (the reason for the original __getattr__ indirection).
Fixes the whole class — all importers share one wrapper, module-level or
function-level. Regression tests cover stale-ref-survives-reset, identity
stability, submit-after-inner-shutdown self-heal, and asyncio.run_in_executor
compatibility; updated the load-timeout test to the new reset semantics.
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Windows users reported 'create demo voice runs indefinitely, no audio, no
error'. Root cause: the first /generate triggers OmniVoice.from_pretrained()
which downloads multi-GB weights via the legacy LFS path (HF_HUB_DISABLE_XET=1),
with NO timeout anywhere. A stalled socket (proxy/firewall/AV) blocks the GPU-
pool worker forever inside get_model() -- before the try/except that would
surface an error -- and the frontend /generate fetch had no abort, so the
spinner spun forever with no toast.
- backend/main.py: set HF_HUB_ETAG_TIMEOUT=15 + HF_HUB_DOWNLOAD_TIMEOUT=30
(per-read timeout: resets on each chunk, so slow-but-progressing downloads
are never punished; only a dead socket trips it). Set before hf import.
- model_manager: get_model()/preload_model() now load via _load_model_with_timeout(),
an asyncio.wait_for backstop (OMNIVOICE_MODEL_LOAD_TIMEOUT, default 1200s) that
drops the poisoned GPU pool and raises a clear, actionable RuntimeError so a
retry gets a fresh worker instead of queueing behind the wedged one.
- useTTS.js: AbortController backstop on /generate so the UI never spins forever
even if the backend is unreachable; friendly timeout toast.
- tests: watchdog raises + resets pool + releases lock; env/floor parsing.
Cross-platform (no OS-specific behavior); backward-compatible with installed
models; local-first preserved.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
