214 lines
13 KiB
Markdown
214 lines
13 KiB
Markdown
# Performance guide
|
|
|
|
Where the time goes when OmniVoice feels slow, what you can tune, and what you
|
|
should leave alone. Everything here applies to the current release; numbers
|
|
marked "measured" come from `scripts/bench_pipeline.py` on a 16 GB Apple
|
|
Silicon M2 — your hardware will differ, but the *ratios* hold.
|
|
|
|
## First: the classic causes of "it got slow"
|
|
|
|
Before touching any knob, check these — they account for most slowness reports:
|
|
|
|
1. **A voice profile with an empty Transcript field.** Cloning needs the
|
|
reference clip's transcript. If the profile doesn't have one, the app runs a
|
|
full Whisper transcription of the clip — and before v0.3.15 it did that on
|
|
**every single generate** (the "TTS got much slower after updating, CPU
|
|
pegged at 100%" regression, #1032). Since v0.3.15 the auto-transcription
|
|
runs once and is saved onto the profile, but a profile that still has an
|
|
empty transcript (e.g. imported or hand-edited data) keeps paying an ASR
|
|
pass per generation. **Fix:** open the voice's editor and check the
|
|
Transcript box — if it's empty, type or paste what the reference clip says
|
|
(or just generate once on v0.3.15+ and confirm the box filled itself in).
|
|
2. **The first generation after a (re)start is always the slowest.** Model
|
|
weights load lazily (~8 s), CUDA builds torch.compile kernels, Apple Silicon
|
|
warms Metal kernels. Judge speed from the *second* generation onward.
|
|
3. **Memory pressure.** On a 16 GB unified-memory machine, a browser with 40
|
|
tabs next to a dub means the OS pages the model in and out — or kills the
|
|
backend outright ("Can't reach the local backend"). Check Settings →
|
|
Models for what's resident, and Settings → Performance for free RAM. See
|
|
[Flush caches / Unload resident model](#flush-caches--unload-resident-model)
|
|
for freeing memory without a restart.
|
|
4. **You're generating on CPU without realizing it.** A driver update, a
|
|
CUDA/torch mismatch, or simply running on hardware with no supported GPU
|
|
path silently drops you to CPU — everything works, just several times
|
|
slower. Three places tell you the truth:
|
|
- **Settings → Performance → Device & compute** shows the live compute
|
|
device (`cuda` / `mps` / `cpu`), a "GPU active" badge, and RAM/VRAM
|
|
readouts.
|
|
- **Settings → About → Run self-check** (the `/system/diagnose` endpoint)
|
|
warns explicitly: *"cpu (no GPU acceleration detected)"* with a hint
|
|
about drivers.
|
|
- **Settings → Engines** shows a routing badge per engine — "GPU active",
|
|
"CPU fallback", or "CPU" — with the *reason* shown as small text under
|
|
the badge (full text on hover).
|
|
Note: **GPU acceleration on Windows is NVIDIA/CUDA-only** — AMD and Intel
|
|
GPUs run CPU-only there (see [Windows install notes](install/windows.md)).
|
|
5. **You aborted a dub earlier (fixed in v0.3.23).** Dubbing moves the TTS
|
|
model to CPU to free VRAM for the ASR model, then moves it back when the
|
|
transcription finishes. Before v0.3.23 that move-back only ran on the fully
|
|
successful path, so cancelling a dub, hitting a dub error, or closing the
|
|
tab mid-transcription left the TTS model stranded on CPU — and **every**
|
|
later generation ran there, 10-50x slower with the CPU pegged, until the
|
|
~15-minute idle unload happened to fire. Restarting the backend cleared it,
|
|
which made it look random or time-of-day related (#1191). Since v0.3.23 the
|
|
move-back runs on every exit path, *and* each generation verifies the model
|
|
is on the expected device and moves it back itself — so no future code path
|
|
can strand it again. If you are on an older build, restart the backend.
|
|
|
|
## What a generation actually spends time on
|
|
|
|
For a cloned voice, one generation is: encode the reference clip (~0.4 s,
|
|
measured; cached after the first use for the voices you reuse — a dub's
|
|
per-line clips are each used once, so there's nothing for a cache to save
|
|
there) → synthesize (the bulk; scales with output length) → post-process
|
|
(mastering, watermark; fractions of a second). Long texts are split into
|
|
chunks synthesized sequentially — time scales roughly linearly with text
|
|
length.
|
|
|
|
For a dub, the stages are: audio extraction + vocal separation (one-time,
|
|
minutes for long videos) → transcription (on the best accelerator available —
|
|
Apple Silicon uses MLX since v0.3.21, NVIDIA uses CUDA; CPU-only installs fall
|
|
back to the processor) → translation (parallel, 6 concurrent requests for LLM
|
|
providers) → per-segment synthesis (sequential, the bulk of the time) →
|
|
mixing and export (mostly stream-copied, fast).
|
|
|
|
## Knobs you can actually turn
|
|
|
|
All of these are environment variables read by the backend at start. Set them
|
|
in `~/.config/omnivoice/env` (created by the installer) or your shell profile.
|
|
None of them are required — the defaults are chosen for the common case.
|
|
|
|
| Variable | Default | What it does |
|
|
|---|---|---|
|
|
| `OMNIVOICE_IDLE_TIMEOUT_S` | `900` | Seconds of idle before the TTS model unloads to free memory. Raise it (e.g. `3600`) if you generate in bursts and dislike the ~8 s reload; lower it on tight-memory machines. |
|
|
| `OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S` | `300` | Same idea for sidecar engines (IndexTTS-2 etc.). |
|
|
| `OMNIVOICE_LLM_CONCURRENCY` | `6` | Parallel LLM translation calls during a dub. Raise for a fast API endpoint, lower if your provider rate-limits. |
|
|
| `OMNIVOICE_GPU_WORKERS` | auto | Concurrent generations on the GPU. Auto-sized from free VRAM (1 worker per 5 GB, max 4); MPS and CPU always get 1. **Do not raise this on ≤10 GB cards or Apple Silicon** — two concurrent jobs over-committing VRAM is exactly the crash class (#567) the auto-sizing exists to prevent. |
|
|
| `OMNIVOICE_CPU_POOL` | `min(8, cores)` | Thread pool for CPU-side work (translation dispatch, audio I/O). |
|
|
| `OMNIVOICE_SINGLE_ENGINE_RESIDENT` | `1` | Keep only one TTS engine in memory at a time. Set `0` on 32 GB+ machines to keep several engines warm across switches. |
|
|
| `OMNIVOICE_UNIFIED_OFFLOAD_HEADROOM_GB` | `6` | On unified memory (Apple Silicon): if free RAM is below this when a dub needs the transcription model, the TTS model is fully released first (it reloads on the next generation). Raise to be more aggressive about freeing, lower on 32 GB+ machines to avoid the reload. |
|
|
| `OMNIVOICE_INDEXTTS_FP16` | `1` | IndexTTS half-precision. Leave on. |
|
|
| `OMNIVOICE_ASR_VRAM_PREFLIGHT` | `1` | Downgrade transcription precision instead of crashing when VRAM is short (CUDA). Leave on. |
|
|
| `OMNIVOICE_GENERATE_TIMEOUT_S` | `300` | Abandon a generation after this many seconds **of actual compute** — the clock starts when a GPU worker picks the job up, never while it waits in line. It's a floor, not a ceiling: the budget grows with the text (+1 s per 40 characters past the first 1200), so long inputs rarely need this raised. |
|
|
| `OMNIVOICE_GPU_QUEUE_TIMEOUT_S` | `1800` | How long a job may sit in the GPU queue before it's reported as a saturated pool (a retryable condition — nothing ran). Waiting is normal on 1-worker machines; lower this only if you'd rather fail fast than queue. |
|
|
|
|
**torch.compile** is probe-based, not platform-based: it's attempted only
|
|
where the runtime check says it can work (a CUDA device with Triton importable
|
|
and a supported GPU architecture) and skipped automatically everywhere else —
|
|
MPS, CPU, and the typical Windows install (Triton ships no Windows wheel).
|
|
The one user-facing control is Settings → Performance → "Disable
|
|
torch.compile" (shown on Windows), for the rare setup where a partial Triton
|
|
install makes the probe pass but the compile attempt itself crash — see
|
|
[Windows install notes](install/windows.md).
|
|
|
|
## Flush caches / Unload resident model
|
|
|
|
This is the feature the VRAM-starved timeout error ("TTS generate ran for more
|
|
than 300s … Flush caches / Unload the resident model") points at. It frees
|
|
RAM/VRAM **without restarting the app**, and it never loses data — an
|
|
unloaded model simply reloads lazily (~8 s) on the next generation.
|
|
|
|
One thing Flush **can't** free: the job that just timed out. An abandoned
|
|
generation cannot be killed from Python — its thread runs to completion and
|
|
holds its VRAM until it does, so a Flush (or a retry) issued seconds after a
|
|
timeout is competing with a job that is still on the device. Wait for it to
|
|
drain, or restart the backend, and then Flush.
|
|
|
|
**Where it lives:**
|
|
|
|
- **Top toolbar → Flush** (the button next to the model-status badge). The
|
|
dropdown lists every model currently in memory — the TTS model, its
|
|
co-loaded ASR, the diarization pipeline, and any resident engines or
|
|
sidecars — with its device and VRAM use, and a per-model **Unload** button
|
|
where unloading is possible (WhisperX is released together with the TTS
|
|
model, so it has no button of its own). An engine left resident after you
|
|
switched away from it is marked *"not active — safe to unload"*. Below the
|
|
list are the two bulk actions:
|
|
- **Flush caches** — runs a multi-pass garbage collection and releases the
|
|
accelerator's cached memory (CUDA/MPS/XPU `empty_cache`). Models stay
|
|
loaded, so there's no reload cost; this recovers cache/fragmentation
|
|
memory only.
|
|
- **Unload all + flush** — the above **plus** fully unloads the resident
|
|
TTS model. Frees the most memory; the next generation pays the ~8 s
|
|
reload.
|
|
- **Settings → Models** — rows whose weights are resident right now show an
|
|
"In memory" badge with the same per-model **Unload** button.
|
|
|
|
**From a script** (the local API on port 3900), the same operations:
|
|
|
|
```bash
|
|
curl -X POST "http://127.0.0.1:3900/system/flush-memory" # flush caches
|
|
curl -X POST "http://127.0.0.1:3900/system/flush-memory?unload_model=true" # + unload TTS model
|
|
curl "http://127.0.0.1:3900/model/loaded" # what's resident
|
|
# unload one model — ids: tts | diarization | sidecar:<id> | sidecars
|
|
curl -X POST "http://127.0.0.1:3900/model/unload/tts"
|
|
```
|
|
|
|
**When to use it:**
|
|
|
|
- **After a VRAM-starved 503 timeout** — a resident model and your generate
|
|
were contending for GPU memory. Unload all + flush, then retry.
|
|
- **Before a dub on a tight-memory machine** — transcription needs room the
|
|
resident TTS model is holding (on Apple Silicon the app does this
|
|
automatically, see `OMNIVOICE_UNIFIED_OFFLOAD_HEADROOM_GB` above).
|
|
- **After switching engines** — with `OMNIVOICE_SINGLE_ENGINE_RESIDENT=0`,
|
|
or for sidecar engines, the previous engine can stay in memory; the
|
|
dropdown shows it and marks it safe to unload.
|
|
- **Mid batch-run on a small GPU** — an occasional
|
|
`POST /system/flush-memory` between jobs keeps cache growth from
|
|
starving later generations.
|
|
|
|
**When it won't help:** many generate errors are *not* memory problems, and
|
|
their messages say so explicitly ("the Flush button won't help here") —
|
|
missing env vars, network failures during a model download, a broken native
|
|
component. Believe the message; Flush only fixes memory contention. Also
|
|
note the app already frees memory on its own when idle
|
|
(`OMNIVOICE_IDLE_TIMEOUT_S`) — Flush is for when you need the memory *now*,
|
|
between jobs.
|
|
|
|
If the timeout error keeps recurring even right after an unload, see
|
|
[troubleshooting §14](install/troubleshooting.md#14-cant-reach-the-local-backend-during-generation--transcription--dubbing)
|
|
— the same starvation class has more remedies there (smaller ASR model,
|
|
CPU ASR, the crash-isolated ASR engine).
|
|
|
|
## Platform notes
|
|
|
|
- **Apple Silicon**: everything runs on the GPU via MPS/MLX. One generation at
|
|
a time by design — unified memory means TTS and ASR compete for the same
|
|
RAM, and the app actively unloads one to make room for the other on 16 GB
|
|
machines. More RAM directly improves dub throughput (fewer unload/reload
|
|
cycles).
|
|
- **NVIDIA**: fp16 + torch.compile on by default. ≥16 GB VRAM parallelizes up
|
|
to 3-4 concurrent generations (API/batch workloads); ≤10 GB deliberately
|
|
serializes.
|
|
- **CPU-only**: expect ~2x slower than MPS, more against CUDA. Prefer the
|
|
smaller/faster engines (see Settings → Engines) and short reference clips.
|
|
|
|
## Measuring instead of guessing
|
|
|
|
`scripts/bench_pipeline.py` (repo checkouts) profiles each stage one at a
|
|
time, memory-safely — it refuses to start a stage without enough free RAM,
|
|
and unloads models between stages:
|
|
|
|
```bash
|
|
# stop the app first — a running backend holds a model and skews numbers
|
|
uv run python scripts/bench_pipeline.py # everything
|
|
uv run python scripts/bench_pipeline.py tts clone # just these stages
|
|
```
|
|
|
|
If you report a performance issue, pasting its table (plus your platform and
|
|
RAM/VRAM) turns a guessing game into a bisect.
|
|
|
|
## Things that look like knobs but aren't
|
|
|
|
- **Deleting and re-adding a voice** doesn't speed anything up; the reference
|
|
encode is cached per file for voices you reuse. (A dub's per-line reference
|
|
clips are the deliberate exception — each is a distinct clip used once, so
|
|
there's nothing for a cache to save.)
|
|
- **Killing the backend between generations** makes everything slower — you
|
|
pay the model load every time. The idle timeout already frees memory when
|
|
it's genuinely idle.
|
|
- **`OMNIVOICE_PRELOAD_TTS_ASR`** exists for a legacy in-process Whisper
|
|
fallback; enabling it costs memory on every start and speeds up nothing on
|
|
a default install.
|