fix(tts): stop stranding the TTS model on CPU after a dub abort (#1191)

`offload_tts_for_asr()` moves the TTS model to CPU to make VRAM room for
WhisperX, but its partner `restore_tts_after_asr()` was only reachable on
the dub-transcribe success path. Any abort, terminal error, or client
disconnect skipped it, and `get_model()` never re-checked placement — so
EVERY subsequent /generate ran on CPU (10-50x slower, CPU pegged) until
the ~15-minute idle unload happened to fire. Reported as "speed varies by
time of day"; it is fully deterministic.

Two independent guarantees:

- Balance the pair at the call site: gen()'s `finally` now pays the
  restore debt on every exit path, chained off the ASR unload so the two
  never contend for VRAM (and fire-and-forget, since the finally also
  runs under GeneratorExit where awaiting is illegal).
- Self-heal placement (the class fix): `get_model()` verifies the model
  is on the resolved target device and moves it back if not, so a future
  unbalanced offload path cannot strand it either. Cheapest-first probe —
  one parameter check on the hot path; unified memory is exempt (its
  offload releases the model rather than moving it).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
debpalash
2026-07-20 14:17:47 +05:30
co-authored by Claude Opus 4.8
parent 6750c59791
commit 0f70744105
5 changed files with 455 additions and 6 deletions
+12 -1
View File
@@ -5,7 +5,7 @@ should leave alone. Everything here applies to the current release; numbers
marked "measured" come from `scripts/bench_pipeline.py` on a 16 GB Apple
Silicon M2 — your hardware will differ, but the *ratios* hold.
## First: the three classic causes of "it got slow"
## First: the classic causes of "it got slow"
Before touching any knob, check these — they account for most slowness reports:
@@ -43,6 +43,17 @@ Before touching any knob, check these — they account for most slowness reports
the badge (full text on hover).
Note: **GPU acceleration on Windows is NVIDIA/CUDA-only** — AMD and Intel
GPUs run CPU-only there (see [Windows install notes](install/windows.md)).
5. **You aborted a dub earlier (fixed in v0.3.23).** Dubbing moves the TTS
model to CPU to free VRAM for the ASR model, then moves it back when the
transcription finishes. Before v0.3.23 that move-back only ran on the fully
successful path, so cancelling a dub, hitting a dub error, or closing the
tab mid-transcription left the TTS model stranded on CPU — and **every**
later generation ran there, 10-50x slower with the CPU pegged, until the
~15-minute idle unload happened to fire. Restarting the backend cleared it,
which made it look random or time-of-day related (#1191). Since v0.3.23 the
move-back runs on every exit path, *and* each generation verifies the model
is on the expected device and moves it back itself — so no future code path
can strand it again. If you are on an older build, restart the backend.
## What a generation actually spends time on