perf(dub): single-use per-segment refs no longer evict the prompts a dub reuses; add docs/performance.md (#1132)

* perf(dub): single-use per-segment refs no longer evict the prompts a dub reuses; add docs/performance.md

The scan-resistance fix:

A dub cuts a distinct reference clip per segment (Wave 3.2 / #486 — each line
clones its own source delivery) and falls back to the per-speaker clone for
segments under 3 s. Both paths flow through the voice-clone prompt cache — an
LRU of 8. Streaming hundreds of one-shot per-segment clips through that LRU
evicts the per-speaker and locked-profile prompts that every fallback segment
reuses, so the speaker ref was re-encoded (~0.4 s each, measured with
scripts/bench_pipeline.py) again and again across the render.

Note what this deliberately does NOT do: the bench's "166 misses vs 2 speakers"
framing suggested keying refs per speaker — but per-segment refs are the
intentional prosody-matching feature, and the re-transcription behind them is
the #1004 correctness fix. Their encode cost is the price of the feature, not
waste. The waste was only the eviction side-effect, and that's what this
removes: _get_clone_prompt(store=False) still reads the cache (a hit is free)
but never inserts, and the dub loop marks exactly the segment-scoped refs
(auto-seg: bindings and auto: bindings resolved to a segment clip) as
single-use. Per-speaker, locked-profile, and preview refs cache as before.

cache_ref is popped in generate_with_cached_ref before the model call — the
model's generate() has an explicit signature and would TypeError — and unknown
engines ignore it (**kw adapters).

The doc:

docs/performance.md is the first performance documentation in the repo — none
of the ~15 perf env vars appeared anywhere in docs/, the Performance panel's
only control is Windows-only, and slowness reports (#1032) arrived as mysteries
instead of settings checks. Covers the three classic causes of "it got slow",
where generation/dub time goes, every knob with defaults and warnings (raising
OMNIVOICE_GPU_WORKERS on a small GPU is the #567 crash, not a speedup), platform
notes, and how to run the bench so reports carry numbers. Linked from README's
install section.

Tests: store=False semantics (encodes, never inserts, still reads), the flood
scenario end to end (a speaker prompt stays warm through 3x the cache cap of
one-shots), and the pop contract (cache_ref never reaches the model). Full
suite: 2974 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs,dub: review round — qualify the per-file cache claim; note the OOM-retry tradeoff

- CodeRabbit: docs/performance.md's "the reference encode is cached per file"
  now carves out the dub's per-line clips (single-use by design — nothing for
  a cache to save).
- Greptile P2 (OOM retry re-encodes a single-use ref): acknowledged in a code
  comment as deliberate — caching the retry's ref would reintroduce the
  eviction this flag prevents, to optimize a path that only runs after an OOM
  already cost seconds.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(performance): probe-based torch.compile wording; honest accelerator + cache claims (review)

Greptile's repeated OOM-retry finding is deliberately skipped: retaining the
prompt across the retry would require passing prompt objects through the
adapter protocol (backend.generate takes paths), to save 0.4s on a path that
only runs after an OOM already cost seconds — the tradeoff is documented at
the call site.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <nizam4103@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

CHANGELOG.md

@@ -8,8 +8,14 @@ The bundled TTS model package (`pyproject.toml`) is versioned independently.
 
 ## [Unreleased]
 
+### Added
+
+- **A performance guide, at last.** [docs/performance.md](docs/performance.md) explains where generation and dubbing time actually goes, the three classic causes of "it got slow" (an empty Transcript field on a voice profile chief among them), every tuning knob the backend reads — none of which were documented anywhere — and which settings to leave alone (raising `OMNIVOICE_GPU_WORKERS` on a small GPU is how you get the crash the default exists to prevent). Includes how to run the built-in profiler so a slowness report can carry numbers instead of vibes.
+
 ### Fixed
 
+- **Dubbing kept re-studying the same speaker's voice, hundreds of times per video.** Each dubbed line clones from a clip of its own source audio (that's what makes deliveries match), and lines too short to clone from fall back to a per-speaker sample. But the app's memory for already-studied voices only holds 8 — and a long dub streams *hundreds* of one-shot per-line clips through it, each pushing out the per-speaker samples that every other line needs. Result: the speaker sample was re-studied (~0.4 s, measured) over and over. One-shot clips are now studied without displacing anything, so the per-speaker samples stay warm for the whole dub. Nothing about the audio changes — same clips, same voices, less repeated work. (#1132)
+
 - **Clicking "Install" on an engine right after opening Settings could silently do nothing.** When the Engines page opens, it quietly checks each installable engine for an in-flight install to re-attach to. If you clicked Install while that check was still running, your click's status update was thrown away to keep requests orderly — so no progress panel, no error, no retry, just nothing (the install itself *did* start in the background; the UI simply never showed it). Fast machines usually won the race, which is why this mostly showed up as a once-in-a-while CI test failure. The Install click's update can no longer be dropped — it politely waits out the startup check instead. (#1131)
 
 - **Cloning re-listened to your reference clip for every chunk of text — now it listens once.** Before OmniVoice can speak in a cloned voice it has to *encode* the reference clip you gave it. That encode was being redone on **every single piece of the job**: long text is split into chunks, and each chunk re-encoded the same reference from scratch; so did each `[pause]` span, and each chapter segment of an audiobook. A cache to prevent exactly this was written a while back — and then quietly bypassed on the path the Generate button actually takes, so for several releases it only ever helped the API. It's now wired into every path. Measured on an M2, one encode costs **0.4 seconds**, so this gives back roughly **3–4 seconds on a long paragraph** and **about a minute on a 166-segment audiobook** — the same voice, the same audio out, just without listening to your reference clip 166 times. As a bonus, `preprocess_prompt` on the OpenAI-compatible endpoint now actually does something; it was being accepted and silently discarded. (#1130)

README.md

@@ -203,6 +203,8 @@ Pick your OS and follow the guide end-to-end:
 - 🐧 **Linux** — [docs/install/linux.md](docs/install/linux.md)
 - 🐳 **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
 
+Feels slow? [docs/performance.md](docs/performance.md) covers where generation time actually goes, the tuning knobs, and the three classic causes of "it got slow".
+
 > Coming from **[CorentinJ/Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning)** (now archived)? There's a dedicated migration guide: [docs/migration/real-time-voice-cloning.md](docs/migration/real-time-voice-cloning.md).
 
 <details>

backend/api/routers/dub_generate.py

@@ -445,6 +445,13 @@ async def dub_generate(job_id: str, req: DubRequest):
                 ref_audio = None
                 ref_text = None
                 used_seed = None
+                # Per-segment refs are a distinct file per segment, each used
+                # exactly once in this render — telling the prompt cache to
+                # store them would evict the per-speaker / locked-profile
+                # prompts that every OTHER segment reuses (LRU of 8 vs
+                # potentially hundreds of segment clips). cache_ref=False =
+                # "encode it, don't let it displace anything".
+                ref_single_use = False
 
                 # Auto-clones extracted from the source video during prepare
                 # (see services/speaker_clone.py) live at job["speaker_clones"]
@@ -458,6 +465,7 @@ async def dub_generate(job_id: str, req: DubRequest):
                     if info:
                         ref_audio = info.get("ref_audio")
                         ref_text = info.get("ref_text")
+                        ref_single_use = True
                     profile_id = None  # prevent the voice_profiles lookup below
 
                 elif profile_id and profile_id.startswith("auto:"):
@@ -474,6 +482,7 @@ async def dub_generate(job_id: str, req: DubRequest):
                     if seg_ref:
                         ref_audio = seg_ref.get("ref_audio")
                         ref_text = seg_ref.get("ref_text")
+                        ref_single_use = True
                     else:
                         key = profile_id[len("auto:"):]
                         clones = job.get("speaker_clones") or {}
@@ -517,6 +526,7 @@ async def dub_generate(job_id: str, req: DubRequest):
                     audio_out = backend.generate(
                         text=text, language=lang if lang != "Auto" else None,
                         ref_audio=ref_audio, ref_text=ref_text,
+                        cache_ref=not ref_single_use,
                         instruct=instruct_str if instruct_str else None,
                         duration=dur_s, num_step=nstep, guidance_scale=cfg,
                         speed=spd, denoise=True, postprocess_output=True,
@@ -561,9 +571,15 @@ async def dub_generate(job_id: str, req: DubRequest):
                         nstep, retry_steps,
                     )
                     try:
+                        # An OOM retry on a single-use ref pays the reference
+                        # encode a second time (~0.4s) — deliberate: caching it
+                        # would reintroduce the eviction this flag exists to
+                        # prevent, to optimize a path that only runs after an
+                        # OOM already cost seconds.
                         audio_out = backend.generate(
                             text=text, language=lang if lang != "Auto" else None,
                             ref_audio=ref_audio, ref_text=ref_text,
+                            cache_ref=not ref_single_use,
                             instruct=instruct_str if instruct_str else None,
                             duration=dur_s, num_step=retry_steps, guidance_scale=cfg,
                             speed=spd, denoise=True, postprocess_output=True,

backend/services/tts_backend.py

@@ -314,10 +314,21 @@ def _clone_prompt_key(ref_audio: str, ref_text, preprocess_prompt: bool = True):
     return (os.path.abspath(ref_audio), mtime, ref_text or "", bool(preprocess_prompt))
 
 
-def _get_clone_prompt(model, ref_audio: str, ref_text, preprocess_prompt: bool = True):
+def _get_clone_prompt(
+    model, ref_audio: str, ref_text, preprocess_prompt: bool = True, *,
+    store: bool = True,
+):
     """Return a cached/precomputed ``VoiceClonePrompt`` for
     (ref_audio, ref_text, preprocess_prompt), or ``None`` to fall back to the
-    inline ref path. Never raises."""
+    inline ref path. Never raises.
+
+    ``store=False`` still *reads* the cache (a hit is free) but never inserts:
+    it exists for single-use references — a dub's per-segment ref clips are each
+    a distinct file used exactly once, and inserting a stream of them into an
+    LRU of 8 evicts the per-speaker and locked-profile prompts that ARE reused.
+    Every short segment falling back to its speaker ref then re-encodes it
+    (~0.4 s each, measured). Scan-resistance, not a second cache policy.
+    """
     try:
         key = _clone_prompt_key(ref_audio, ref_text, preprocess_prompt)
     except Exception:
@@ -336,6 +347,8 @@ def _get_clone_prompt(model, ref_audio: str, ref_text, preprocess_prompt: bool =
     except Exception as e:  # noqa: BLE001 — fall back, never break synthesis
         logger.warning("voice-clone prompt precompute failed; using inline ref: %s", e)
         return None
+    if not store:
+        return prompt
     with _prompt_cache_lock:
         _prompt_cache[key] = prompt
         _prompt_cache.move_to_end(key)
@@ -364,10 +377,18 @@ def generate_with_cached_ref(model, *, ref_audio, ref_text, **gen_kw):
     synthesize exactly as before. A latency optimization must never be able to turn
     a generation that would have succeeded into an error.
     """
+    # cache_ref=False marks a single-use reference (a dub's per-segment clips):
+    # look the cache up, but never insert — see _get_clone_prompt(store=). MUST
+    # be popped: the model's generate() has an explicit signature and would
+    # TypeError on an unknown kwarg.
+    cache_ref = bool(gen_kw.pop("cache_ref", True))
     # Stays in gen_kw too: the model needs it on the inline branch, and it is inert
     # on the prompt branch (that prompt is already encoded).
     preprocess_prompt = bool(gen_kw.get("preprocess_prompt", True))
-    prompt = _get_clone_prompt(model, ref_audio, ref_text, preprocess_prompt) if ref_audio else None
+    prompt = (
+        _get_clone_prompt(model, ref_audio, ref_text, preprocess_prompt, store=cache_ref)
+        if ref_audio else None
+    )
     if prompt is not None:
         try:
             return model.generate(voice_clone_prompt=prompt, **gen_kw)
@@ -465,6 +486,9 @@ class OmniVoiceBackend(TTSBackend):
         # used to be dropped on the floor here — the API accepted it and gen_kw
         # never carried it, so it silently did nothing.
         gen_kw["preprocess_prompt"] = bool(kw.get("preprocess_prompt", True))
+        # Single-use reference hint (dub per-segment clips) — see
+        # generate_with_cached_ref, which pops it before the model sees it.
+        gen_kw["cache_ref"] = bool(kw.get("cache_ref", True))
         # The cached-reference path lives in generate_with_cached_ref, shared with
         # the native callers. Deliberately NOT a second copy: this logic living in
         # one place here and a subtly different one there is exactly how the cache

docs/performance.md

@@ -0,0 +1,110 @@
+# Performance guide
+
+Where the time goes when OmniVoice feels slow, what you can tune, and what you
+should leave alone. Everything here applies to the current release; numbers
+marked "measured" come from `scripts/bench_pipeline.py` on a 16 GB Apple
+Silicon M2 — your hardware will differ, but the *ratios* hold.
+
+## First: the three classic causes of "it got slow"
+
+Before touching any knob, check these — they account for most slowness reports:
+
+1. **A voice profile with an empty Transcript field.** Cloning needs the
+   reference clip's transcript. If the profile doesn't have one, the app
+   transcribes the clip — since v0.3.15 that happens **once** and is saved onto
+   the profile, but a profile that somehow keeps an empty transcript (e.g.
+   imported data) pays an ASR pass per generation. Open the voice's editor and
+   confirm the Transcript box shows text.
+2. **The first generation after a (re)start is always the slowest.** Model
+   weights load lazily (~8 s), CUDA builds torch.compile kernels, Apple Silicon
+   warms Metal kernels. Judge speed from the *second* generation onward.
+3. **Memory pressure.** On a 16 GB unified-memory machine, a browser with 40
+   tabs next to a dub means the OS pages the model in and out — or kills the
+   backend outright ("Can't reach the local backend"). Check Settings →
+   Models for what's resident, and Settings → Performance for free RAM.
+
+## What a generation actually spends time on
+
+For a cloned voice, one generation is: encode the reference clip (~0.4 s,
+measured; cached after the first use for the voices you reuse — a dub's
+per-line clips are each used once, so there's nothing for a cache to save
+there) → synthesize (the bulk; scales with output length) → post-process
+(mastering, watermark; fractions of a second). Long texts are split into
+chunks synthesized sequentially — time scales roughly linearly with text
+length.
+
+For a dub, the stages are: audio extraction + vocal separation (one-time,
+minutes for long videos) → transcription (on the best accelerator available —
+Apple Silicon uses MLX since v0.3.21, NVIDIA uses CUDA; CPU-only installs fall
+back to the processor) → translation (parallel, 6 concurrent requests for LLM
+providers) → per-segment synthesis (sequential, the bulk of the time) →
+mixing and export (mostly stream-copied, fast).
+
+## Knobs you can actually turn
+
+All of these are environment variables read by the backend at start. Set them
+in `~/.config/omnivoice/env` (created by the installer) or your shell profile.
+None of them are required — the defaults are chosen for the common case.
+
+| Variable | Default | What it does |
+|---|---|---|
+| `OMNIVOICE_IDLE_TIMEOUT_S` | `900` | Seconds of idle before the TTS model unloads to free memory. Raise it (e.g. `3600`) if you generate in bursts and dislike the ~8 s reload; lower it on tight-memory machines. |
+| `OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S` | `300` | Same idea for sidecar engines (IndexTTS-2 etc.). |
+| `OMNIVOICE_LLM_CONCURRENCY` | `6` | Parallel LLM translation calls during a dub. Raise for a fast API endpoint, lower if your provider rate-limits. |
+| `OMNIVOICE_GPU_WORKERS` | auto | Concurrent generations on the GPU. Auto-sized from free VRAM (1 worker per 5 GB, max 4); MPS and CPU always get 1. **Do not raise this on ≤10 GB cards or Apple Silicon** — two concurrent jobs over-committing VRAM is exactly the crash class (#567) the auto-sizing exists to prevent. |
+| `OMNIVOICE_CPU_POOL` | `min(8, cores)` | Thread pool for CPU-side work (translation dispatch, audio I/O). |
+| `OMNIVOICE_SINGLE_ENGINE_RESIDENT` | `1` | Keep only one TTS engine in memory at a time. Set `0` on 32 GB+ machines to keep several engines warm across switches. |
+| `OMNIVOICE_UNIFIED_OFFLOAD_HEADROOM_GB` | `6` | On unified memory (Apple Silicon): if free RAM is below this when a dub needs the transcription model, the TTS model is fully released first (it reloads on the next generation). Raise to be more aggressive about freeing, lower on 32 GB+ machines to avoid the reload. |
+| `OMNIVOICE_INDEXTTS_FP16` | `1` | IndexTTS half-precision. Leave on. |
+| `OMNIVOICE_ASR_VRAM_PREFLIGHT` | `1` | Downgrade transcription precision instead of crashing when VRAM is short (CUDA). Leave on. |
+| `OMNIVOICE_GENERATE_TIMEOUT_S` | `300` | Abandon a generation after this many seconds. Raise for very long single generations on slow hardware. |
+
+**torch.compile** is probe-based, not platform-based: it's attempted only
+where the runtime check says it can work (a CUDA device with Triton importable
+and a supported GPU architecture) and skipped automatically everywhere else —
+MPS, CPU, and the typical Windows install (Triton ships no Windows wheel).
+The one user-facing control is Settings → Performance → "Disable
+torch.compile" (shown on Windows), for the rare setup where a partial Triton
+install makes the probe pass but the compile attempt itself crash — see
+[Windows install notes](install/windows.md).
+
+## Platform notes
+
+- **Apple Silicon**: everything runs on the GPU via MPS/MLX. One generation at
+  a time by design — unified memory means TTS and ASR compete for the same
+  RAM, and the app actively unloads one to make room for the other on 16 GB
+  machines. More RAM directly improves dub throughput (fewer unload/reload
+  cycles).
+- **NVIDIA**: fp16 + torch.compile on by default. ≥16 GB VRAM parallelizes up
+  to 3-4 concurrent generations (API/batch workloads); ≤10 GB deliberately
+  serializes.
+- **CPU-only**: expect ~2x slower than MPS, more against CUDA. Prefer the
+  smaller/faster engines (see Settings → Engines) and short reference clips.
+
+## Measuring instead of guessing
+
+`scripts/bench_pipeline.py` (repo checkouts) profiles each stage one at a
+time, memory-safely — it refuses to start a stage without enough free RAM,
+and unloads models between stages:
+
+```bash
+# stop the app first — a running backend holds a model and skews numbers
+uv run python scripts/bench_pipeline.py            # everything
+uv run python scripts/bench_pipeline.py tts clone  # just these stages
+```
+
+If you report a performance issue, pasting its table (plus your platform and
+RAM/VRAM) turns a guessing game into a bisect.
+
+## Things that look like knobs but aren't
+
+- **Deleting and re-adding a voice** doesn't speed anything up; the reference
+  encode is cached per file for voices you reuse. (A dub's per-line reference
+  clips are the deliberate exception — each is a distinct clip used once, so
+  there's nothing for a cache to save.)
+- **Killing the backend between generations** makes everything slower — you
+  pay the model load every time. The idle timeout already frees memory when
+  it's genuinely idle.
+- **`OMNIVOICE_PRELOAD_TTS_ASR`** exists for a legacy in-process Whisper
+  fallback; enabling it costs memory on every start and speeds up nothing on
+  a default install.

tests/test_clone_prompt_cache.py

@@ -110,3 +110,45 @@ def test_clear_empties_cache(tmp_path):
     assert len(tb._prompt_cache) == 1
     tb.clear_clone_prompt_cache()
     assert len(tb._prompt_cache) == 0
+
+
+# ── Single-use references (store=False): dub per-segment clips ───────────────
+#
+# A dub cuts a distinct reference clip per segment (Wave 3.2 prosody matching),
+# each used exactly once. Inserting a stream of hundreds of those into an LRU
+# of 8 evicts the per-speaker / locked-profile prompts every OTHER segment
+# reuses — so each short segment falling back to its speaker ref re-encoded it
+# (~0.4 s each, measured on an M2). store=False is the scan-resistance: encode,
+# use, don't displace anything.
+
+
+def test_store_false_encodes_but_never_inserts(tmp_path):
+    m = _StubModel()
+    ref = _wav(tmp_path)
+    p = tb._get_clone_prompt(m, ref, "one-shot", store=False)
+    assert p is not None and m.calls == 1
+    assert len(tb._prompt_cache) == 0, "single-use prompt was inserted into the LRU"
+
+
+def test_store_false_still_reads_the_cache(tmp_path):
+    """A hit is free — store=False only skips the insert, not the lookup."""
+    m = _StubModel()
+    ref = _wav(tmp_path)
+    tb._get_clone_prompt(m, ref, "hi")               # cached normally
+    tb._get_clone_prompt(m, ref, "hi", store=False)  # must hit, not re-encode
+    assert m.calls == 1
+
+
+def test_single_use_flood_does_not_evict_reused_prompts(tmp_path):
+    """The dub scenario end to end: a per-speaker ref stays warm through a
+    flood of per-segment one-shots far larger than the cache cap."""
+    m = _StubModel()
+    speaker_ref = _wav(tmp_path, "speaker.wav")
+    tb._get_clone_prompt(m, speaker_ref, "speaker")          # encode #1, cached
+    for i in range(tb._PROMPT_CACHE_MAX * 3):                # the flood
+        tb._get_clone_prompt(m, _wav(tmp_path, f"seg{i}.wav"), "seg", store=False)
+    before = m.calls
+    tb._get_clone_prompt(m, speaker_ref, "speaker")          # short-segment fallback
+    assert m.calls == before, (
+        "the speaker prompt was evicted by single-use segment refs and re-encoded"
+    )

tests/test_clone_prompt_wiring.py

@@ -223,3 +223,34 @@ def test_audiobook_native_synth_encodes_reference_once_per_voice(ref_wav):
         f"reference re-encoded {m.encodes}x across 12 audiobook segments — "
         "a book would pay this hundreds of times"
     )
+
+
+def test_cache_ref_false_is_popped_and_skips_the_insert(ref_wav):
+    """The dub loop marks per-segment refs cache_ref=False. Two contracts:
+    the flag must never reach model.generate (explicit signature → TypeError
+    on the real model), and the encoded prompt must not enter the LRU (a dub's
+    flood of one-shot clips would evict the per-speaker prompts that ARE
+    reused — the scan-resistance this exists for)."""
+    from services.tts_backend import generate_with_cached_ref
+
+    class _RejectsUnknownKwargs(_StubModel):
+        def generate(self, **kw):
+            assert "cache_ref" not in kw, "cache_ref leaked through to the model"
+            return super().generate(**kw)
+
+    m = _RejectsUnknownKwargs()
+    generate_with_cached_ref(
+        m, ref_audio=ref_wav, ref_text="hello",
+        text="One-shot segment.", language=None, instruct=None,
+        duration=None, speed=1.0, cache_ref=False,
+    )
+    assert m.encodes == 1
+    assert len(_tb()._prompt_cache) == 0, "single-use ref was inserted into the LRU"
+
+    # Default (cache_ref absent) still caches — the /generate & audiobook paths.
+    generate_with_cached_ref(
+        m, ref_audio=ref_wav, ref_text="hello",
+        text="Reused voice.", language=None, instruct=None,
+        duration=None, speed=1.0,
+    )
+    assert len(_tb()._prompt_cache) == 1