The re-transcribe added for the (ref_audio, ref_text) mismatch fix is
dispatched with a bare run_in_executor(_gpu_pool, ...), and refine_ref_text
calls asr_backend.transcribe() directly. Its try/except catches a raised
error but not a *hang* — a wedged whisperx/CTranslate2 transcribe (#730)
holds the GPU-pool worker forever. On a <=10 GB card the pool is 1 worker,
so that starves every later GPU job into the misleading "can't reach the
local backend", and there's no ping on the await so the EventSource drops.
Route both refine dispatches (per-speaker and per-segment) through the same
run_transcribe_guarded the rest of dub_core.py already uses (the chunk loop
and the whole-file "Dub" transcribe). On timeout it resets the pool and
raises ASRTimeoutError; keep the original clones, matching refine_ref_text's
own "failure is a strict no-op" fallback.
Adds a repro test: refine_ref_texts dispatched raw is unbounded on a hang;
through the guard it times out and falls back to the original ref_text.
Co-authored-by: stronghamjji <289942360+stronghamjji@users.noreply.github.com>
extract_speaker_clones/extract_segment_refs pair each audio slice (cut
at ASR segment timestamps) with that segment's own `text` field, on
the assumption the two agree. They routinely don't — Whisper (and
friends) frequently drift on segment boundaries: a trailing word
audible in [start, end] but missing from text, or vice versa. When the
(ref_audio, ref_text) pair disagrees, zero-shot TTS prompt-priming
breaks down and the clone can emit the mismatched reference text
itself instead of the target-language line it was asked to speak —
reported with an exceptionally clear root-cause diagnosis and a
working A/B repro (matched pair: clean on the first try; mismatched
pair: wrong language 6/6 times).
Fix (as proposed in the report): re-transcribe each written reference
clip via the already-loaded, already-warm active ASR backend and use
that transcript as ref_text — this guarantees the pair matches by
construction, independent of whether the original segment text was
ever right. Falls back to the original text on any re-transcribe
failure or empty result — never a regression from current behavior,
only ever a fix.
New services.speaker_clone.refine_ref_text (single clip, unit-testable
against a duck-typed fake ASR backend) and refine_ref_texts (batch —
one executor round-trip per whole clones/seg_clones dict rather than
one per reference). Wired into dub_core.py's two clone-extraction call
sites, routed through _gpu_pool to match the established convention
for ASR-backend calls (the model is mid-lifecycle: TTS is offloaded,
ASR is loaded and exclusive, right where the existing per-chunk
transcribe calls already run on this same pool).
Tests: tests/test_speaker_clone_purity.py — 6 new cases covering the
mismatch-correction path, ASR-failure fallback, empty-transcript
fallback, no-backend no-op, and batch behavior (one failing entry
doesn't affect the others). Full backend suite: 2412 passed, 0 failed.
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(dub): the speaker-count hint is honored on every diarization path + clone-purity guard
The dub "Speakers" count reached _diarize() and then died on 3 of its 4
branches, so setting it changed nothing, speakers blended, and auto-clones
were cut from mixed-speaker audio ("made up" voices):
- FunASR inline-turns shortcut returned before the hint was ever consulted
→ now an explicit num_speakers routes the job through pyannote (the one
engine that honors an exact count); turns stay the fast path only when no
hint is set, and remain the fallback (with an honest "hint ignored"
warning) when pyannote can't load or crashes mid-run.
- pyannote-unavailable fallback used a hardcoded 2-speaker silence-gap
heuristic → assign_speakers_heuristic now takes num_speakers and cycles N
labels on gap boundaries (1 → single speaker; None → legacy alternation),
and the existing diarization warning says the hint is only approximately
honored.
- pyannote-crash fallback dropped the hint the same way → same treatment.
No branch drops the hint silently anymore: every degraded path extends the
existing `warning` SSE payload (detail + a machine-readable speaker_hint
field) that the frontend already renders.
Parity + purity:
- POST /dub/transcribe/{job_id} (the CLI's endpoint) gains the same clamped
num_speakers query param, forwarded to pyannote and the heuristic; the
omnivoice-dub CLI gains --speakers N.
- Clone-purity guard: _pick_reference_slices rejects sub-1.5s slices, prefers
slices not temporally adjacent (<0.3s) to another speaker's turn (scoring
preference, not a hard filter), and extract_speaker_clones skips extraction
entirely when labels came from the heuristic (labels_source kwarg threaded
from _diarize; missing kwarg keeps the old behavior) — with a user-facing
warning pointing at Settings → Models → pyannote.
Tests: fail-before/pass-after coverage in tests/test_speaker_hint.py (all
four _diarize branches driven through the real SSE stream), clone-purity
guards in tests/test_speaker_clone_purity.py, heuristic hint semantics in
tests/test_segmentation.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(changelog): add the speaker-hint + clone-purity fix under [Unreleased] (#952)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Palash Debnath