fix(voice): free-text instruct now filtered before every generate/save call (#1010) (#1018)

buildDesignInstruct() already keeps Studio's design/clone generate
calls (useTTS.js) from round-tripping a 400 "Unsupported instruct
items" — it filters free-text against the active engine's supported
vocabulary client-side, with a toast instead of a failed request. Three
other call sites built their own instruct string directly and skipped
it entirely:

- handleSegmentPreview (Dub tab's per-segment preview) — instruct comes
  straight from segment/preset data; a preset's raw attrs merged with a
  free-text style field can carry phrases outside the vocabulary.
- handleSaveProfile / handleSaveHistoryAsProfile — both always create a
  kind='clone' profile; the backend only sanitizes instruct on save for
  kind='design' (see profiles.py's heal_design_instruct branch), so a
  clone profile could silently persist an unusable instruct and then
  400 every single time it's later used to generate.

All three now filter through the same buildDesignInstruct({}, instruct)
call useTTS.js's own clone path already uses, with the same
unsupported/duplicate-item toasts.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Palash Debnath
2026-07-09 01:47:06 +05:30
committed by GitHub
co-authored by mergetest Claude Fable 5
parent 7f8a42ce51
commit 33379890ad
2 changed files with 35 additions and 3 deletions
+1
View File
@@ -19,6 +19,7 @@ The bundled TTS model package (`pyproject.toml`) is versioned independently.
- **Cross-language dub no longer speaks the source-language reference line verbatim.** Auto-generated speaker clones pair an audio slice with the ASR segment's own text field, assuming the two agree — but ASR segment text and its timestamps routinely drift (a trailing word audible in the clip but missing from the text, or vice versa). A mismatched (reference audio, reference text) pair breaks zero-shot TTS prompt priming badly enough that the clone can emit the reference text itself instead of the target-language line it was asked to speak. Each reference clip is now re-transcribed after it's written, so the pair matches by construction — reported with an exceptionally clear root-cause diagnosis and a working A/B repro. (#1004)
- **Voice Gallery errors now say what actually went wrong.** "Use voice", "Preview", search, upload, save, delete, and trim in the Gallery all showed the same hardcoded guess ("the engine may be loading") on ANY failure — a 500, a validation error, a genuinely unrelated bug — discarding the real, already-clean backend error message in the process. Every one of those now shows the actual error.
- **Voice cloning on mlx-audio's CSM model no longer crashes with an opaque "list index out of range".** `MLXAudioBackend.generate()` read `voice`/`ref_audio`/`language`/`speed` from its kwargs but silently dropped `ref_text` — CSM only builds its cloning context when both `ref_audio` and `ref_text` are present, so cloning on this engine could never have worked as shipped. Reported with the exact root cause and a working fix. (#1012, #1013)
- **A dub segment's free-text style tags no longer 400 the segment preview.** A validator-safe instruct builder already keeps Studio and Clone generation from round-tripping a 400 on unsupported free-text (a preset's raw attrs, an old profile's stray descriptive phrase) — but the Dub tab's segment preview, and saving a profile from a clone or from history, built their instruct strings directly and skipped it. Same guard now applies everywhere an instruct string is sent. (#1010)
## [0.3.12] — 2026-07-08
+34 -3
View File
@@ -11,7 +11,11 @@ import { generateSpeech, audioUrlWithCacheBust } from '../api/generate';
import { apiFetch } from '../api/client';
import { playBlobAudio } from '../utils/media';
import { PRESETS } from '../utils/constants';
import { instructToFormValue, mergeDescribedAttrs } from '../utils/voiceInstruct';
import {
instructToFormValue,
mergeDescribedAttrs,
buildDesignInstruct,
} from '../utils/voiceInstruct';
import { askConfirm } from '../utils/dialog';
import { toast } from 'react-hot-toast';
import { recordValueMoment } from '../utils/donationMoments';
@@ -55,7 +59,12 @@ export default function useProfiles({ loadHistory, loadProfiles }) {
const safeBlob = new Blob([arrBuf], { type: refAudio.type });
formData.append('ref_audio', safeBlob, refAudio.name || 'profile.wav');
formData.append('ref_text', refText);
formData.append('instruct', instruct);
// #1010: the backend only sanitizes instruct on save for kind='design'
// profiles — a clone profile (this call always creates kind='clone')
// would silently persist an unsupported free-text instruct and then
// 400 every single time it's used to generate. Filter here too.
const { instruct: safeInst } = buildDesignInstruct({}, instruct);
formData.append('instruct', safeInst);
formData.append('language', language);
try {
await createProfile(formData);
@@ -204,6 +213,25 @@ export default function useProfiles({ loadHistory, loadProfiles }) {
fin_prof = '';
}
// #1010: this instruct string comes straight from segment/preset data,
// never through the validator-safe builder — a preset's raw attrs or a
// free-text style field can carry phrases outside the active engine's
// supported instruct vocabulary, 400ing instead of previewing. Same
// client-side guard useTTS.js already applies to the clone path.
if (fin_inst) {
const { instruct: safeInst, unsupported, duplicates } = buildDesignInstruct({}, fin_inst);
if (unsupported.length) {
toast(t('tts_errors.ignored_unsupported', { items: unsupported.join(', ') }), {
icon: '⚠️',
});
}
if (duplicates.length) {
toast(t('tts_errors.ignored_duplicate', { items: duplicates.join(', ') }), {
icon: '⚠️',
});
}
fin_inst = safeInst;
}
if (fin_prof) formData.append('profile_id', fin_prof);
if (fin_inst) formData.append('instruct', fin_inst);
const fin_lang = seg.target_lang || dubLang;
@@ -245,7 +273,10 @@ export default function useProfiles({ loadHistory, loadProfiles }) {
: item.text
: '';
formData.append('ref_text', extractedText);
formData.append('instruct', item.instruct || '');
// #1010: same guard as handleSaveProfile — this always creates a
// kind='clone' profile, which the backend never sanitizes on save.
const { instruct: safeHistInst } = buildDesignInstruct({}, item.instruct || '');
formData.append('instruct', safeHistInst);
formData.append('language', item.language || 'Auto');
if (item.seed !== undefined && item.seed !== null) {
formData.append('seed', item.seed);