`_reset_gpu_pool()` fires on a model-load timeout to recover a wedged worker —
it shut the ThreadPoolExecutor down and rebuilt a fresh one on next access. But
several request handlers (generation, dub_generate, dub_core, dub_translate,
openai_compat) did a *module-level* `from services.model_manager import
_gpu_pool`, capturing the executor object at import time. After a reset those
references pointed at the dead pool, so the next generate/dub/transcribe/
translate raised `RuntimeError: cannot schedule new futures after shutdown` —
surfacing as a 500 or "Can't reach the local backend" (#589#599).
Make `_gpu_pool` a single long-lived `_ResilientGpuPool` wrapper (a
concurrent.futures.Executor) whose *inner* ThreadPoolExecutor is swapped:
- every submit() resolves the live pool, and a submit that races a shutdown
rebuilds once and retries, so a stale captured reference self-heals;
- `_reset_gpu_pool()` now drops only the inner pool (fresh worker on retry)
while preserving the wrapper identity every importer holds;
- pool sizing stays lazy, so we still probe the device after torch's lazy
import (the reason for the original __getattr__ indirection).
Fixes the whole class — all importers share one wrapper, module-level or
function-level. Regression tests cover stale-ref-survives-reset, identity
stability, submit-after-inner-shutdown self-heal, and asyncio.run_in_executor
compatibility; updated the load-timeout test to the new reset semantics.
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
