fix(tts): stop stranding the TTS model on CPU after a dub abort (#1191)
`offload_tts_for_asr()` moves the TTS model to CPU to make VRAM room for WhisperX, but its partner `restore_tts_after_asr()` was only reachable on the dub-transcribe success path. Any abort, terminal error, or client disconnect skipped it, and `get_model()` never re-checked placement — so EVERY subsequent /generate ran on CPU (10-50x slower, CPU pegged) until the ~15-minute idle unload happened to fire. Reported as "speed varies by time of day"; it is fully deterministic. Two independent guarantees: - Balance the pair at the call site: gen()'s `finally` now pays the restore debt on every exit path, chained off the ASR unload so the two never contend for VRAM (and fire-and-forget, since the finally also runs under GeneratorExit where awaiting is illegal). - Self-heal placement (the class fix): `get_model()` verifies the model is on the resolved target device and moves it back if not, so a future unbalanced offload path cannot strand it either. Cheapest-first probe — one parameter check on the hot path; unified memory is exempt (its offload releases the model rather than moving it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
6750c59791
commit
0f70744105
+12
-1
@@ -5,7 +5,7 @@ should leave alone. Everything here applies to the current release; numbers
|
||||
marked "measured" come from `scripts/bench_pipeline.py` on a 16 GB Apple
|
||||
Silicon M2 — your hardware will differ, but the *ratios* hold.
|
||||
|
||||
## First: the three classic causes of "it got slow"
|
||||
## First: the classic causes of "it got slow"
|
||||
|
||||
Before touching any knob, check these — they account for most slowness reports:
|
||||
|
||||
@@ -43,6 +43,17 @@ Before touching any knob, check these — they account for most slowness reports
|
||||
the badge (full text on hover).
|
||||
Note: **GPU acceleration on Windows is NVIDIA/CUDA-only** — AMD and Intel
|
||||
GPUs run CPU-only there (see [Windows install notes](install/windows.md)).
|
||||
5. **You aborted a dub earlier (fixed in v0.3.23).** Dubbing moves the TTS
|
||||
model to CPU to free VRAM for the ASR model, then moves it back when the
|
||||
transcription finishes. Before v0.3.23 that move-back only ran on the fully
|
||||
successful path, so cancelling a dub, hitting a dub error, or closing the
|
||||
tab mid-transcription left the TTS model stranded on CPU — and **every**
|
||||
later generation ran there, 10-50x slower with the CPU pegged, until the
|
||||
~15-minute idle unload happened to fire. Restarting the backend cleared it,
|
||||
which made it look random or time-of-day related (#1191). Since v0.3.23 the
|
||||
move-back runs on every exit path, *and* each generation verifies the model
|
||||
is on the expected device and moves it back itself — so no future code path
|
||||
can strand it again. If you are on an older build, restart the backend.
|
||||
|
||||
## What a generation actually spends time on
|
||||
|
||||
|
||||
Reference in New Issue
Block a user