extract_speaker_clones/extract_segment_refs pair each audio slice (cut
at ASR segment timestamps) with that segment's own `text` field, on
the assumption the two agree. They routinely don't — Whisper (and
friends) frequently drift on segment boundaries: a trailing word
audible in [start, end] but missing from text, or vice versa. When the
(ref_audio, ref_text) pair disagrees, zero-shot TTS prompt-priming
breaks down and the clone can emit the mismatched reference text
itself instead of the target-language line it was asked to speak —
reported with an exceptionally clear root-cause diagnosis and a
working A/B repro (matched pair: clean on the first try; mismatched
pair: wrong language 6/6 times).
Fix (as proposed in the report): re-transcribe each written reference
clip via the already-loaded, already-warm active ASR backend and use
that transcript as ref_text — this guarantees the pair matches by
construction, independent of whether the original segment text was
ever right. Falls back to the original text on any re-transcribe
failure or empty result — never a regression from current behavior,
only ever a fix.
New services.speaker_clone.refine_ref_text (single clip, unit-testable
against a duck-typed fake ASR backend) and refine_ref_texts (batch —
one executor round-trip per whole clones/seg_clones dict rather than
one per reference). Wired into dub_core.py's two clone-extraction call
sites, routed through _gpu_pool to match the established convention
for ASR-backend calls (the model is mid-lifecycle: TTS is offloaded,
ASR is loaded and exclusive, right where the existing per-chunk
transcribe calls already run on this same pool).
Tests: tests/test_speaker_clone_purity.py — 6 new cases covering the
mismatch-correction path, ASR-failure fallback, empty-transcript
fallback, no-backend no-op, and batch behavior (one failing entry
doesn't affect the others). Full backend suite: 2412 passed, 0 failed.
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(dub): the speaker-count hint is honored on every diarization path + clone-purity guard
The dub "Speakers" count reached _diarize() and then died on 3 of its 4
branches, so setting it changed nothing, speakers blended, and auto-clones
were cut from mixed-speaker audio ("made up" voices):
- FunASR inline-turns shortcut returned before the hint was ever consulted
→ now an explicit num_speakers routes the job through pyannote (the one
engine that honors an exact count); turns stay the fast path only when no
hint is set, and remain the fallback (with an honest "hint ignored"
warning) when pyannote can't load or crashes mid-run.
- pyannote-unavailable fallback used a hardcoded 2-speaker silence-gap
heuristic → assign_speakers_heuristic now takes num_speakers and cycles N
labels on gap boundaries (1 → single speaker; None → legacy alternation),
and the existing diarization warning says the hint is only approximately
honored.
- pyannote-crash fallback dropped the hint the same way → same treatment.
No branch drops the hint silently anymore: every degraded path extends the
existing `warning` SSE payload (detail + a machine-readable speaker_hint
field) that the frontend already renders.
Parity + purity:
- POST /dub/transcribe/{job_id} (the CLI's endpoint) gains the same clamped
num_speakers query param, forwarded to pyannote and the heuristic; the
omnivoice-dub CLI gains --speakers N.
- Clone-purity guard: _pick_reference_slices rejects sub-1.5s slices, prefers
slices not temporally adjacent (<0.3s) to another speaker's turn (scoring
preference, not a hard filter), and extract_speaker_clones skips extraction
entirely when labels came from the heuristic (labels_source kwarg threaded
from _diarize; missing kwarg keeps the old behavior) — with a user-facing
warning pointing at Settings → Models → pyannote.
Tests: fail-before/pass-after coverage in tests/test_speaker_hint.py (all
four _diarize branches driven through the real SSE stream), clone-purity
guards in tests/test_speaker_clone_purity.py, heuristic hint semantics in
tests/test_segmentation.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(changelog): add the speaker-hint + clone-purity fix under [Unreleased] (#952)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Cut each long-enough dub segment's clone reference from the isolated vocals
at that segment's own timestamps, so the dub of each line carries the
prosody/emotion of its source line — finer than one reference per speaker.
Reimplemented from the clean-room spec (pyvideotrans per-line ref idea); our
design delta is a quality floor with fallback.
- services/speaker_clone.py: extract_segment_refs() keyed by segment id;
reference transcript is the SOURCE text (text_original), since the vocals
slice is source-language audio. Floor at MIN_SEGMENT_REF_DURATION_S=3.0
(not the per-speaker 5.0, which most dialogue lines fall under) — shorter
lines are omitted and fall back to the per-speaker clone, so it's a strict
improvement, never a regression.
- dub_core: run extraction at transcribe (per_segment_refs query param,
default on), store job['segment_clones'], default each unassigned
segment's profile_id to 'auto-seg:{id}' when it has its own ref, else the
existing 'auto:{speaker}'. Forcing per-speaker (per_segment_refs=false)
is supported for long-form consistency.
- dub_generate _gen: resolve 'auto-seg:' from segment_clones, ahead of the
per-speaker 'auto:' path. profile_id is already a fingerprint field, so
flipping the mode re-dubs automatically (no _GEN_INPUT_FIELDS change).
7 pure tests over a synthetic vocals wav (own-ref for long lines,
short-line omission/fallback, source-text transcript, bounds clamping,
floor boundary). Pipeline wiring validated in CI.
Spec 4 / parity program Wave 3.2.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Autofixes the genuine lint behind the CodeQL py/unused-import and
py/unused-local-variable note-level alerts — actually removing the dead
code rather than dismissing it. 68 safe fixes via 'ruff check --select
F401,F841 --fix' across 29 backend files (dead stdlib/symbol imports like
io/sys/json/torch/typing.Optional and unused locals). Only ruff's safe
fixes applied — the 9 'unsafe' fixes and the audio_dsp numpy availability
import were left untouched.
Not touched: empty-except (needs per-site judgement, not autofixable);
frontend js/unused-local-variable (eslint no-unused-vars has no autofix);
the loopback-low-risk path/log/stack-trace alerts (real, left visible).
Verified: full tests/ suite unchanged at 601 passed (the 2 test_supertonic3
failures are pre-existing on main, local .venv state, green in CI).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Palash Debnath