upstream-archive/VoiceStudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(dub): realtime dub preview (#1769)

Browse Source 
Opt-in live preview for dub segments: edits debounce into a streamed /ws/tts synthesis played through the chunk player, with cancellation preserved through buffered playback. Maintainer fixes: /ws/tts added to the backend ticket allowlist (feature was dead off-loopback), handshake failures surface a toast, loopback-only plaintext refusal reverted to keep the documented remote-GPU setup working, PCM16 decode hardened. Thanks @mvanhorn!
This commit is contained in:
Matt Van Horn
2026-09-03 18:27:07 +05:30
committed by GitHub
parent ac287c612f
commit 999345de41
39 changed files with 1147 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+2
View File
@@ -26,6 +26,7 @@ the frozen-backend fallback mirror it for their toolchains.
- Preview builds now stay newer than Stable even when automatic post-release version bumps are disabled (#1762)
- CosyVoice setup guidance now separates downloaded model files from the runtime that makes the engine available (#1761)
- MCP tools can now keep audio out of agent context by returning files and accepting base-path-confined file inputs (#1760) — thanks @agudmund!
- Hear a dub line as you type it — an opt-in live preview streams TTS for the edited segment (#1769) — thanks @mvanhorn!
- Studio gains a Convert method: re-say any clip in one of your saved voices, speech to speech, fully local (#1765) — thanks @mvanhorn!
- Hardsub video export gains an opt-in karaoke word-highlight caption style (#1764) — thanks @mvanhorn!
- The audiobook player now shows the chapter text and highlights the word being narrated (#1766) — thanks @mvanhorn!
@@ -40,6 +41,7 @@ the frozen-backend fallback mirror it for their toolchains.
- Studio's new Convert method turns a dropped or recorded clip into an existing voice profile's voice, with optional source-duration matching (#1765) — thanks @mvanhorn!
- Hardsub export can now burn karaoke word-highlight captions: an opt-in Line | Karaoke control renders a word-timed ASS sweep from timings persisted at transcription, with an even-split fallback for older jobs and translated tracks, plus a `GET /dub/ass/{job_id}` sidecar (#1764) — thanks @mvanhorn!
- Windows releases now include an independently updatable per-user MSI that installs and uninstalls without elevation (#1713)
- Dub segments can now stream live TTS while you edit a translated line — opt-in toggle, existing `/ws/tts` socket, shared generation admission, exports still render at full quality (#1769) — thanks @mvanhorn!
- Engine status and diagnostic bundles now record loaded execution provider, device, precision, fallback stage, accelerator identity, runtime versions, and parent-process memory visibility (#1717)
### Docs
backend/services/admin_sessions.py
+4 -1
View File
@@ -33,8 +33,11 @@ WS_TICKET_PREFIX = "ovs_ws_ticket_"
_TOKEN_BYTES = 32
_ENCODED_TOKEN_LENGTH = 43
_TOKEN_BODY_RE = re.compile(rf"^[A-Za-z0-9_-]{{{_ENCODED_TOKEN_LENGTH}}}$")
# Every ticketed WebSocket route. The first-party mirror is ``ALLOWED_WS_PATHS``
# in frontend/src/api/authSession.ts — a route missing here mints a 422 and the
# UI consumer fails silently (#1769 added /ws/tts for the live dub preview).
_ALLOWED_WS_PATHS = frozenset(
{"/ws/events", "/ws/transcribe", "/v1/audio/transcriptions/stream"}
{"/ws/events", "/ws/transcribe", "/ws/tts", "/v1/audio/transcriptions/stream"}
)
_ADMIN_CAPABILITIES = frozenset({"consume", "admin"})
_KEY_GENERATION_INFO = b"omnivoice-admin-key-generation-v1"
docs/ROADMAP.md
+1 -1
View File
@@ -213,7 +213,7 @@ None on the critical path to world-class. All are answers to real demand.
| Voice memory across projects | ⏳ | After Phase 4 |
| Context-aware pipeline (video frames → pipeline decisions) | ⏳ | After Phase 4 |
| On-device learning from corrections (user edits → LoRA) | ⏳ | Research only; possibly Phase 5+ |
| Real-time dub preview (stream TTS as you edit) | ⏳ | After Phase 4.1 |
| Real-time dub preview (stream TTS as you edit) | ✅ | Shipped 2026-09-02 (#1769) — opt-in "Live preview" toggle on the dub segment table streams the edited line over `/ws/tts` with its CAST voice; export path unchanged. |
### 🧪 Quality track _(🟡 underway)_
docs/api-auth.md
+4 -2
View File
@@ -204,7 +204,8 @@ ws://gpu-box:3900/ws/transcribe?api_key=<key>
That URL form is retained for non-browser compatibility only. The first-party
UI never constructs it. A bearer administrator session first calls
`POST /api/auth/ws-ticket` and puts only the returned `ws_ticket` in the URL.
Tickets are scoped to `/ws/transcribe` or `/ws/events`, expire after 30 seconds,
Tickets are scoped to one of `/ws/transcribe`, `/ws/events` or `/ws/tts` (the
live dub preview stream), expire after 30 seconds,
return the same bounded `expires_in`/`expires_at` pair, and are consumed
atomically at most once. Same-origin UI WebSockets use the
HttpOnly session cookie and must pass exact `Origin` validation; `null`, missing,
@@ -338,7 +339,8 @@ drives exact-Origin checks and the session cookie's `Secure` attribute.
For a public path prefix such as `/studio`, either strip that prefix before
forwarding or configure the ASGI `root_path` to the same value. WebSocket ticket
validation removes only that trusted, configured prefix; it never accepts an
arbitrary path merely because it ends in `/ws/events` or `/ws/transcribe`.
arbitrary path merely because it ends in `/ws/events`, `/ws/transcribe` or
`/ws/tts`.
## Status codes
frontend/src/api/authCredentialHygiene.test.js
+2 -1
View File
@@ -81,13 +81,14 @@ describe('administrator credential hygiene static guard', () => {
expect(violations, 'WebSocket URLs may contain ws_ticket, never a master key').toEqual([]);
});
it('keeps both WebSocket consumers behind the authenticated URL boundary', () => {
it('keeps every WebSocket consumer behind the authenticated URL boundary', () => {
const constructors = sources()
.filter(({ source }) => source.includes('new WebSocket('))
.map(({ file, source }) => ({ file, authenticated: source.includes('authenticatedWsUrl') }));
expect(constructors).toEqual([
{ file: 'components/CaptureWidget.jsx', authenticated: true },
{ file: 'hooks/useDubLivePreview.js', authenticated: true },
{ file: 'hooks/useRealtimeEvents.js', authenticated: true },
]);
});
frontend/src/api/authSession.ts
+10 -1
View File
@@ -379,7 +379,10 @@ export async function revokeAdminSession(
}
}
const ALLOWED_WS_PATHS = new Set(['/ws/events', '/ws/transcribe']);
// Mirrors the backend ticket allowlist (`_ALLOWED_WS_PATHS` in
// backend/services/admin_sessions.py). A path listed here but not there mints
// a 422, and the consumer fails silently — keep the two in lockstep.
const ALLOWED_WS_PATHS = new Set(['/ws/events', '/ws/transcribe', '/ws/tts']);
const LOGICAL_WS_ORIGIN = 'http://omnivoice.invalid';
function websocketTarget(path: string, apiBase: string): { url: URL; logicalPath: string } {
@@ -426,6 +429,12 @@ export async function requestWebSocketTicket(
const { logicalPath } = websocketTarget(path, base);
const session = getAdminSession(base, { storage, now });
if (!session) throw new AuthSessionError(401);
// Deliberately no plaintext (`ws:`) refusal: the documented remote-GPU setup
// is plain HTTP over a Tailscale/WireGuard tailnet (docs/remote-gpu.md), and
// the bearer session that mints this ticket already crossed that same
// transport. A one-use, 30 s, path-bound ticket adds no exposure the session
// lacks; refusing it would only cut /ws/events and /ws/transcribe off for
// every remote-backend user.
let response: Response;
const controller = new AbortController();
frontend/src/api/authSessionWsTransport.test.ts
+70
View File
@@ -0,0 +1,70 @@
import { beforeEach, describe, expect, it, vi } from 'vitest';
import { ADMIN_SESSION_STORAGE_KEY, AuthSessionError, authenticatedWsUrl } from './authSession';
const SESSION = `ovs_admin_session_${'A'.repeat(43)}`;
const TICKET = `ovs_ws_ticket_${'B'.repeat(43)}`;
const NOW_SECONDS = 1_800_000_000;
const response = () =>
new Response(JSON.stringify({ ticket: TICKET, expires_at: NOW_SECONDS + 30 }), {
status: 201,
headers: { 'content-type': 'application/json' },
});
const storeSession = (apiBase: string) => {
sessionStorage.setItem(
ADMIN_SESSION_STORAGE_KEY,
JSON.stringify({ token: SESSION, expiresAt: NOW_SECONDS + 3600, apiBase }),
);
};
describe('ticketed WebSocket transport', () => {
beforeEach(() => sessionStorage.clear());
it('mints a path-bound /ws/tts ticket for the live dub preview (#1769)', async () => {
const apiBase = 'https://gpu.test:3900';
storeSession(apiBase);
const fetchImpl = vi.fn().mockResolvedValue(response());
await expect(
authenticatedWsUrl('/ws/tts', { apiBase, fetchImpl, now: () => NOW_SECONDS * 1000 }),
).resolves.toBe(`wss://gpu.test:3900/ws/tts?ws_ticket=${TICKET}`);
expect(JSON.parse(fetchImpl.mock.calls[0][1].body)).toEqual({ path: '/ws/tts' });
});
it.each([
'http://gpu-box.your-tailnet.ts.net:3900', // docs/remote-gpu.md tailnet flow
'http://192.168.1.20:3900', // LAN Docker host
'http://127.0.0.2:3900',
])('keeps ticketing the plaintext non-loopback bases the docs support: %s', async (apiBase) => {
// The bearer session that mints the ticket already crossed this same
// transport; refusing plaintext here would only cut /ws/events and
// /ws/transcribe off for every documented remote-backend user.
storeSession(apiBase);
const fetchImpl = vi.fn().mockResolvedValue(response());
await expect(
authenticatedWsUrl('/ws/tts', { apiBase, fetchImpl, now: () => NOW_SECONDS * 1000 }),
).resolves.toBe(`${apiBase.replace('http:', 'ws:')}/ws/tts?ws_ticket=${TICKET}`);
});
it('returns a credential-free URL when no admin session exists (loopback desktop)', async () => {
const fetchImpl = vi.fn();
await expect(
authenticatedWsUrl('/ws/tts', { apiBase: 'http://127.0.0.1:3900', fetchImpl }),
).resolves.toBe('ws://127.0.0.1:3900/ws/tts');
expect(fetchImpl).not.toHaveBeenCalled();
});
it('refuses paths outside the ticket allowlist before any network call', async () => {
const apiBase = 'https://gpu.test:3900';
storeSession(apiBase);
const fetchImpl = vi.fn().mockResolvedValue(response());
await expect(
authenticatedWsUrl('/ws/anything', { apiBase, fetchImpl, now: () => NOW_SECONDS * 1000 }),
).rejects.toBeInstanceOf(AuthSessionError);
expect(fetchImpl).not.toHaveBeenCalled();
});
});
frontend/src/components/DubSegmentRow.jsx
+63 -32
View File
@@ -13,6 +13,7 @@ import {
Minus,
Plus,
Sparkles,
Volume2,
} from 'lucide-react';
import { formatTime } from '../utils/format';
import { LANG_CODES } from '../utils/languages';
@@ -89,6 +90,10 @@ function DubSegmentRow({
onSeek,
timelineSelected,
hasOverlap,
liveEnabled,
liveActive,
onLiveEdit,
onLiveToggle,
}) {
const { t } = useTranslation();
const textInputRef = useRef(null);
@@ -412,38 +417,60 @@ function DubSegmentRow({
)}
<span className="seg-text-col">
<input
ref={textInputRef}
className="input-base segment-input"
value={seg.text}
onChange={(e) => onEditField(seg.id, 'text', e.target.value)}
onKeyDown={handleTextKeyDown}
onKeyUp={captureCursor}
onSelect={captureCursor}
onClick={(e) => {
e.stopPropagation();
captureCursor(e);
}}
disabled={disabled}
title={
seg.translate_error
? t('segment.translate_error_title', { error: seg.translate_error })
: seg.translate_degraded
? t('segment.translate_degraded_title', { reason: seg.translate_degraded })
: overBudget
? t('segment.budget_title', {
pct: Math.round((seg.text.length / seg.text_original.length) * 100),
})
: t('segment.text_title')
}
style={
overBudget
? { background: 'rgba(250,189,47,0.10)' }
: seg.translate_error
? { background: 'rgba(251,73,52,0.10)' }
: undefined
}
/>
<span className="flex items-center gap-[2px] min-w-0">
<input
ref={textInputRef}
className="input-base segment-input"
value={seg.text}
onChange={(e) => {
onEditField(seg.id, 'text', e.target.value);
// Live dub preview (opt-in): debounce-stream the edited line.
if (liveEnabled) onLiveEdit?.(seg, e.target.value);
}}
onKeyDown={handleTextKeyDown}
onKeyUp={captureCursor}
onSelect={captureCursor}
onClick={(e) => {
e.stopPropagation();
captureCursor(e);
}}
disabled={disabled}
title={
seg.translate_error
? t('segment.translate_error_title', { error: seg.translate_error })
: seg.translate_degraded
? t('segment.translate_degraded_title', { reason: seg.translate_degraded })
: overBudget
? t('segment.budget_title', {
pct: Math.round((seg.text.length / seg.text_original.length) * 100),
})
: t('segment.text_title')
}
style={
overBudget
? { background: 'rgba(250,189,47,0.10)' }
: seg.translate_error
? { background: 'rgba(251,73,52,0.10)' }
: undefined
}
/>
{liveEnabled && (
// Tiny live-preview speaker: pulses while this line streams;
// click streams the current text now / stops the active stream.
<button
type="button"
className={`seg-live-btn${liveActive ? ' seg-live-btn--on' : ''}`}
disabled={disabled}
title={t(liveActive ? 'segment.live_stop_title' : 'segment.live_play_title')}
onClick={(e) => {
e.stopPropagation();
onLiveToggle?.(seg);
}}
>
<Volume2 size={9} />
</button>
)}
</span>
{seg.text_original && seg.text_original !== seg.text && (
<span className="text-[0.52rem] text-[#6b6657] flex items-center gap-[3px] px-[2px] overflow-hidden">
<span className="opacity-80 uppercase font-semibold text-[0.48rem] text-[#7c6f64]">
@@ -627,5 +654,9 @@ export default memo(
prev.canMergePrev === next.canMergePrev &&
prev.profiles === next.profiles &&
prev.speakerClones === next.speakerClones &&
prev.liveEnabled === next.liveEnabled &&
prev.liveActive === next.liveActive &&
prev.onLiveEdit === next.onLiveEdit &&
prev.onLiveToggle === next.onLiveToggle &&
prev.idx === next.idx,
);
frontend/src/components/DubSegmentTable.jsx
+34
View File
@@ -5,6 +5,7 @@ import DubSegmentRow from './DubSegmentRow';
import { Table, Select } from '../ui';
import { useAppStore } from '../store';
import { visibleMergeAvailability } from '../utils/segmentParts';
import useDubLivePreview from '../hooks/useDubLivePreview';
const BASE_ROW_HEIGHT = 48;
const ROW_HEIGHT_WITH_ORIG = 62;
@@ -51,6 +52,14 @@ export default function DubSegmentTable({
// timeupdate tick.
const currentSegId = useAppStore((s) => s.dubCurrentSegId);
// Opt-in live dub preview (default off): editing a row's translated text
// streams that line over /ws/tts. Suspended while a dub generation runs —
// the pipeline already owns the TTS admission slot.
const livePreviewOn = useAppStore((s) => s.dubLivePreview);
const setDubLivePreview = useAppStore((s) => s.setDubLivePreview);
const liveEnabled = livePreviewOn && !disabled;
const { liveSegId, onLiveEdit, onLiveToggle } = useDubLivePreview({ enabled: liveEnabled });
// Imperative handle for react-window v2 so we can auto-scroll the row
// containing the playhead into view. (The scroll effect itself lives
// below the `filtered` memo so it can depend on it without TDZ.)
@@ -162,6 +171,10 @@ export default function DubSegmentTable({
segments,
currentSegId,
timelineSelectedId,
liveEnabled,
liveSegId,
onLiveEdit,
onLiveToggle,
}),
[
filtered,
@@ -186,6 +199,10 @@ export default function DubSegmentTable({
segments,
currentSegId,
timelineSelectedId,
liveEnabled,
liveSegId,
onLiveEdit,
onLiveToggle,
],
);
@@ -215,6 +232,10 @@ export default function DubSegmentTable({
segments: segs,
currentSegId: curId,
timelineSelectedId: tlSel,
liveEnabled: liveOn,
liveSegId: liveId,
onLiveEdit: liveEdit,
onLiveToggle: liveToggle,
}) => {
const seg = fl[index];
if (!seg) return null;
@@ -261,6 +282,10 @@ export default function DubSegmentTable({
canMergePrev={canMergePrev}
onDirect={direct}
onSeek={seek}
liveEnabled={liveOn}
liveActive={liveOn && liveId === seg.id}
onLiveEdit={liveEdit}
onLiveToggle={liveToggle}
/>
);
},
@@ -290,6 +315,15 @@ export default function DubSegmentTable({
searchPlaceholder={t('segment.search_placeholder')}
meta={meta}
>
<label className="dub-live-toggle" title={t('dub.live_preview_title')}>
<input
type="checkbox"
className="accent-[var(--color-brand)]"
checked={!!livePreviewOn}
onChange={(e) => setDubLivePreview(e.target.checked)}
/>
{t('dub.live_preview')}
</label>
{speakers.length > 1 && (
<Select
size="sm"
frontend/src/hooks/useDubLivePreview.js
+285
View File
@@ -0,0 +1,285 @@
/**
* useDubLivePreview — live-as-you-edit dub audio (ROADMAP: "Real-time dub
* preview (stream TTS as you edit)").
*
* When the opt-in `dubLivePreview` pref is on and the user edits a segment's
* translated text, the edit is debounced (~400 ms) and the line is streamed
* over the EXISTING `/ws/tts` WebSocket (binary PCM16 sentence chunks) with
* the segment's CAST voice, playing progressively through the same Web Audio
* chunk player the streaming /generate preview uses. A new keystroke or a
* segment change closes the previous socket first, so sockets never pile up.
*
* Boundaries (deliberate):
* • Admission — every stream runs inside `withTtsInflight`, the same
* process-wide chokepoint /generate holds, so a live preview can never
* race a running generation (busy → the standard localized toast). The
* backend side already serializes on the GPU pool.
* • Nothing is persisted: `/ws/tts` audio is ear-only. Export still goes
* through the full-quality generate path (`finalizeTtsBeforeExport`),
* and the incremental re-dub flow is untouched.
* • Voice resolution goes through `segmentGenInputs` — the SAME expansion
* the dub generate body uses (`preset:` → instruct) — so the preview
* voice cannot disagree with what Generate would render. No CAST voice
* at all → skip with an actionable "assign a voice" toast.
* • Local-first: `/ws/tts` streams on this machine by design (the route
* itself refuses remote workers); no new outbound calls.
*/
import { useCallback, useEffect, useRef, useState } from 'react';
import { toast } from 'react-hot-toast';
import { useTranslation } from 'react-i18next';
import { API } from '../api/client';
import { authenticatedWsUrl } from '../api/authSession';
import { withTtsInflight, TtsGenerationBusyError } from '../api/generate';
import { createStreamingChunkPlayer, supportsStreamingPreview } from '../utils/streamingTts';
import { segmentGenInputs } from '../utils/segments';
import { useAppStore } from '../store';
/** Keystroke → stream debounce. Long enough to skip mid-word churn, short
* enough to feel live. Exported for the fake-timer tests. */
export const LIVE_PREVIEW_DEBOUNCE_MS = 400;
/** Same-message toast throttle so a typing burst can't stack toasts. */
const TOAST_THROTTLE_MS = 4000;
export default function useDubLivePreview({ enabled }) {
const { t } = useTranslation();
const [liveSegId, setLiveSegId] = useState(null);
const enabledRef = useRef(enabled);
const timerRef = useRef(null);
const sessionRef = useRef(null); // { segId, ws, player, abort, admission }
const intentRef = useRef(0);
const lastToastRef = useRef({ key: '', at: 0 });
const throttledToast = useCallback((key, show) => {
const now = Date.now();
const last = lastToastRef.current;
if (last.key === key && now - last.at < TOAST_THROTTLE_MS) return;
lastToastRef.current = { key, at: now };
show();
}, []);
/** Close the live socket and silence its player. Resolves the stream's
* admission promise, so `withTtsInflight` releases the in-flight slot. */
const abortSession = useCallback(() => {
if (timerRef.current) {
clearTimeout(timerRef.current);
timerRef.current = null;
}
const session = sessionRef.current;
if (!session) return Promise.resolve();
session.abort();
// The admission promise settles only after withTtsInflight has released
// the slot — awaiting it means the next stream can't trip over our own
// in-flight count.
return session.admission ?? Promise.resolve();
}, []);
/** One socket lifetime: connect → send → play chunks → done/error/close. */
const streamOnce = useCallback(
(session, payload) =>
new Promise((resolve) => {
if (session.settled) {
resolve();
return;
}
const stopPlayer = () => {
const player = session.player;
session.player = null;
try {
player?.fail();
} catch {
/* teardown must never throw into the admission chain */
}
};
const settle = ({ keepPlayer = false } = {}) => {
if (session.settled) return;
session.settled = true;
if (!keepPlayer) stopPlayer();
try {
session.ws?.close();
} catch {
/* already closed */
}
if (!keepPlayer && sessionRef.current === session) sessionRef.current = null;
setLiveSegId((cur) => (cur === session.segId ? null : cur));
resolve();
};
session.abort = () => {
if (!session.settled) {
settle();
return;
}
stopPlayer();
if (sessionRef.current === session) sessionRef.current = null;
};
(async () => {
let endpoint;
try {
// Same one-use-ticket boundary as /ws/events and /ws/transcribe.
endpoint = await authenticatedWsUrl('/ws/tts', { apiBase: API });
} catch (err) {
// Say so — a silently dead toggle is the one failure the user
// can't diagnose (a backend that refuses the ticket looks exactly
// like "the feature does nothing").
throttledToast('live-connect', () =>
toast.error(t('tts_errors.error_prefix', { message: err?.message || '' })),
);
settle();
return;
}
if (session.settled) return;
let ws;
try {
ws = new WebSocket(endpoint);
} catch {
settle();
return;
}
ws.binaryType = 'arraybuffer';
session.ws = ws;
ws.onopen = () => ws.send(JSON.stringify(payload));
ws.onmessage = (event) => {
if (session.settled) return;
if (typeof event.data !== 'string') {
session.player?.appendPcm16Bytes(event.data);
return;
}
let msg;
try {
msg = JSON.parse(event.data);
} catch {
return;
}
if (msg.type === 'start') {
session.player = createStreamingChunkPlayer({
label: payload.text,
sampleRate: msg.sample_rate,
onDone: () => {
session.player = null;
if (!session.settled) {
session.abort();
} else if (sessionRef.current === session) {
sessionRef.current = null;
}
},
});
} else if (msg.type === 'done') {
// Release network/admission state now, but retain the player in
// sessionRef until its buffered tail ends. A later edit, toggle,
// or unmount can therefore still silence obsolete audio.
const keepPlayer = Boolean(session.player);
session.player?.finalize();
settle({ keepPlayer });
} else if (msg.type === 'error') {
throttledToast('live-error', () =>
toast.error(t('tts_errors.error_prefix', { message: msg.detail || '' })),
);
settle();
}
// "routing" frames are advisory (local-stream notice) — ignored.
};
ws.onerror = () => settle();
ws.onclose = () => settle();
})();
}),
[t, throttledToast],
);
const startStream = useCallback(
async (seg, text, intent) => {
if (intent !== intentRef.current) return;
if (!enabledRef.current || !supportsStreamingPreview()) return;
if (!text || !text.trim()) {
void abortSession();
return;
}
// Same voice expansion as the dub generate body (#281 helper): a
// `preset:` id becomes instruct text, everything else is a profile id.
const inputs = segmentGenInputs({ ...seg, text });
if (!inputs.profile_id && !inputs.instruct) {
void abortSession();
throttledToast('live-no-voice', () =>
toast(t('dub.live_preview_pick_voice'), { icon: '🎤' }),
);
return;
}
const payload = { text, speed: inputs.speed || 1.0 };
if (inputs.profile_id) payload.voice = inputs.profile_id;
if (inputs.instruct) payload.instruct = inputs.instruct;
const lang = inputs.target_lang || useAppStore.getState().dubLang;
if (lang && lang !== 'Auto') payload.language = lang;
await abortSession();
if (!enabledRef.current || intent !== intentRef.current) return;
const session = { segId: seg.id, ws: null, player: null, settled: false };
// Pre-stream abort: admission can refuse (busy) before streamOnce ever
// installs the socket-aware settle — the session must still clear its
// refs so the row indicator can't stick on.
session.abort = () => {
session.settled = true;
if (sessionRef.current === session) sessionRef.current = null;
setLiveSegId((cur) => (cur === session.segId ? null : cur));
};
sessionRef.current = session;
setLiveSegId(seg.id);
session.admission = withTtsInflight(() => streamOnce(session, payload)).catch((err) => {
session.abort();
if (err instanceof TtsGenerationBusyError) {
throttledToast('live-busy', () =>
toast(t('tts_errors.generation_in_progress'), { icon: '⏳' }),
);
}
});
},
[abortSession, streamOnce, t, throttledToast],
);
/** Text-input edit: close the previous socket NOW, re-stream after the
* debounce. A segment switch mid-type goes through the same path. */
const onLiveEdit = useCallback(
(seg, text) => {
if (!enabledRef.current) return;
const intent = ++intentRef.current;
void abortSession();
timerRef.current = setTimeout(() => {
timerRef.current = null;
void startStream(seg, text, intent);
}, LIVE_PREVIEW_DEBOUNCE_MS);
},
[abortSession, startStream],
);
/** Row speaker button: stream this line's current text now, or stop it. */
const onLiveToggle = useCallback(
(seg) => {
const intent = ++intentRef.current;
if (sessionRef.current?.segId === seg.id) {
void abortSession();
return;
}
void startStream(seg, seg.text, intent);
},
[abortSession, startStream],
);
// Toggle off / unmount: nothing may keep streaming. The ref keeps the
// debounce/stream callbacks stable across toggles (row memo identity).
useEffect(() => {
enabledRef.current = enabled;
if (!enabled) {
intentRef.current += 1;
void abortSession();
}
}, [enabled, abortSession]);
useEffect(
() => () => {
intentRef.current += 1;
void abortSession();
},
[abortSession],
);
return { liveSegId, onLiveEdit, onLiveToggle, stop: abortSession };
}
frontend/src/i18n/locales/ar.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "فشلت ترجمة {{langs}}، لذا تم تخطي عمليات الدبلجة هذه لتجنب إنشاء مسار بلغة خاطئة.",
"manage_languages": "إدارة اللغات",
"languages_done": "مكتمل",
"languages_pending": "قيد الانتظار"
"languages_pending": "قيد الانتظار",
"live_preview": "معاينة حية",
"live_preview_title": "استمع إلى السطر أثناء تحريره — يبثّ المقطع المحرَّر تحويل النص إلى كلام بصوت CAST الخاص به بعد توقّف قصير. تشغيل فقط: لا يُحفظ أي شيء، ويظل التصدير يُصيَّر بالجودة الكاملة.",
"live_preview_pick_voice": "عيِّن صوت CAST لهذا السطر أو لمتحدثه لسماع المعاينة الحية."
},
"glossary": {
"title": "مسرد",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "الصوت مناسب داخل الفتحة.",
"fit_ratio_title": "يمثل صوت تحويل النص إلى كلام (TTS) {{pct}}% من الفتحة.",
"qc_verify": "تحقق",
"qc_verify_title": "تم سماع ASR للتمرير الثاني: \"{{heard}}\" - أعد الاستماع أو إعادة دبلجة هذا السطر."
"qc_verify_title": "تم سماع ASR للتمرير الثاني: \"{{heard}}\" - أعد الاستماع أو إعادة دبلجة هذا السطر.",
"live_play_title": "بثّ هذا السطر الآن (معاينة حية)",
"live_stop_title": "إيقاف المعاينة الحية"
},
"timeline": {
"track_label": "الخط الزمني للمقاطع",
frontend/src/i18n/locales/de.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "Die Übersetzung für {{langs}} ist fehlgeschlagen. Diese Synchronfassungen wurden übersprungen, damit keine Spur in der falschen Sprache entsteht.",
"manage_languages": "Sprachen verwalten",
"languages_done": "Fertig",
"languages_pending": "Ausstehend"
"languages_pending": "Ausstehend",
"live_preview": "Live-Vorschau",
"live_preview_title": "Zeile beim Bearbeiten anhören — das bearbeitete Segment streamt nach kurzer Pause TTS mit seiner CAST-Stimme. Nur Wiedergabe: nichts wird gespeichert, der Export rendert weiterhin in voller Qualität.",
"live_preview_pick_voice": "Weise dieser Zeile oder ihrem Sprecher eine CAST-Stimme zu, um die Live-Vorschau zu hören."
},
"glossary": {
"title": "Glossar",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "Audio passt in den Steckplatz.",
"fit_ratio_title": "TTS-Audio macht {{pct}} % des Slots aus.",
"qc_verify": "Überprüfen",
"qc_verify_title": "ASR im zweiten Durchgang hörte: „{{heard}}“ – hören Sie sich diese Zeile noch einmal an oder überspielen Sie sie erneut."
"qc_verify_title": "ASR im zweiten Durchgang hörte: „{{heard}}“ – hören Sie sich diese Zeile noch einmal an oder überspielen Sie sie erneut.",
"live_play_title": "Diese Zeile jetzt streamen (Live-Vorschau)",
"live_stop_title": "Live-Vorschau stoppen"
},
"timeline": {
"track_label": "Segment-Zeitleiste",
frontend/src/i18n/locales/en.json
+7 -2
View File
@@ -1395,7 +1395,10 @@
"languages_selected_other": "{{count}} languages selected",
"manage_languages": "Manage languages",
"languages_done": "Done",
"languages_pending": "Pending"
"languages_pending": "Pending",
"live_preview": "Live preview",
"live_preview_title": "Hear a line as you edit it — the edited segment streams TTS with its CAST voice after a short pause. Playback only: nothing is saved, and export still renders at full quality.",
"live_preview_pick_voice": "Assign a CAST voice to this line or its speaker to hear the live preview."
},
"glossary": {
"title": "Glossary",
@@ -1498,7 +1501,9 @@
"plan_impossible": "Won't fit +{{seconds}}s",
"plan_impossible_title": "Predicted speech ≈{{est}}s vs {{avail}}s available — about {{seconds}}s more than any fitting can absorb, so the audio would be trimmed. Shorten the text before generating.",
"plan_apply": "Use shorter rewrite",
"plan_apply_title": "Replace this line with the suggested shorter rewrite: \"{{text}}\""
"plan_apply_title": "Replace this line with the suggested shorter rewrite: \"{{text}}\"",
"live_play_title": "Stream this line now (live preview)",
"live_stop_title": "Stop the live preview"
},
"timeline": {
"track_label": "Segment timeline",
frontend/src/i18n/locales/es.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "Falló la traducción de {{langs}}; esos doblajes se omitieron para evitar pistas en el idioma incorrecto.",
"manage_languages": "Gestionar idiomas",
"languages_done": "Listo",
"languages_pending": "Pendiente"
"languages_pending": "Pendiente",
"live_preview": "Vista previa en vivo",
"live_preview_title": "Escucha una línea mientras la editas: el segmento editado transmite TTS con su voz de CAST tras una breve pausa. Solo reproducción: no se guarda nada y la exportación se renderiza a calidad completa.",
"live_preview_pick_voice": "Asigna una voz de CAST a esta línea o a su hablante para oír la vista previa en vivo."
},
"glossary": {
"title": "Glosario",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "El audio encaja dentro de la ranura.",
"fit_ratio_title": "El audio TTS es el {{pct}}% de la ranura.",
"qc_verify": "verificar",
"qc_verify_title": "Se escuchó ASR en el segundo paso: \"{{heard}}\": vuelva a escuchar o doblar esta línea."
"qc_verify_title": "Se escuchó ASR en el segundo paso: \"{{heard}}\": vuelva a escuchar o doblar esta línea.",
"live_play_title": "Transmitir esta línea ahora (vista previa en vivo)",
"live_stop_title": "Detener la vista previa en vivo"
},
"timeline": {
"track_label": "Línea de tiempo de segmentos",
frontend/src/i18n/locales/fr.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "La traduction a échoué pour {{langs}} ; ces doublages ont été ignorés afin d’éviter une piste dans la mauvaise langue.",
"manage_languages": "Gérer les langues",
"languages_done": "Terminé",
"languages_pending": "En attente"
"languages_pending": "En attente",
"live_preview": "Aperçu en direct",
"live_preview_title": "Écoutez une ligne pendant que vous la modifiez : le segment modifié diffuse la synthèse vocale avec sa voix CAST après une courte pause. Lecture seule : rien n'est enregistré et l'export reste rendu en pleine qualité.",
"live_preview_pick_voice": "Attribuez une voix CAST à cette ligne ou à son locuteur pour entendre l'aperçu en direct."
},
"glossary": {
"title": "Glossaire",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "L'audio s'adapte à l'intérieur de la fente.",
"fit_ratio_title": "L'audio TTS représente {{pct}}% de l'emplacement.",
"qc_verify": "Vérifier",
"qc_verify_title": "ASR de deuxième passage entendu : \"{{heard}}\" – réécoutez ou redoublez cette ligne."
"qc_verify_title": "ASR de deuxième passage entendu : \"{{heard}}\" – réécoutez ou redoublez cette ligne.",
"live_play_title": "Diffuser cette ligne maintenant (aperçu en direct)",
"live_stop_title": "Arrêter l'aperçu en direct"
},
"timeline": {
"track_label": "Chronologie des segments",
frontend/src/i18n/locales/hi.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "{{langs}} के लिए अनुवाद विफल रहा—गलत भाषा का ट्रैक बनने से रोकने के लिए उन डब को छोड़ दिया गया।",
"manage_languages": "भाषाएँ प्रबंधित करें",
"languages_done": "पूर्ण",
"languages_pending": "बाकी"
"languages_pending": "बाकी",
"live_preview": "लाइव पूर्वावलोकन",
"live_preview_title": "संपादन करते समय पंक्ति सुनें — संपादित सेगमेंट थोड़े विराम के बाद अपनी CAST आवाज़ के साथ TTS स्ट्रीम करता है। केवल प्लेबैक: कुछ भी सहेजा नहीं जाता और निर्यात पूरी गुणवत्ता में ही रेंडर होता है।",
"live_preview_pick_voice": "लाइव पूर्वावलोकन सुनने के लिए इस पंक्ति या इसके वक्ता को CAST आवाज़ असाइन करें।"
},
"glossary": {
"title": "शब्दावली",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "ऑडियो स्लॉट के अंदर फ़िट हो जाता है।",
"fit_ratio_title": "टीटीएस ऑडियो स्लॉट का {{pct}}% है।",
"qc_verify": "सत्यापित करें",
"qc_verify_title": "सेकेंड-पास एएसआर ने सुना: \"{{heard}}\" - इस पंक्ति को दोबारा सुनें या दोबारा डब करें।"
"qc_verify_title": "सेकेंड-पास एएसआर ने सुना: \"{{heard}}\" - इस पंक्ति को दोबारा सुनें या दोबारा डब करें।",
"live_play_title": "यह पंक्ति अभी स्ट्रीम करें (लाइव पूर्वावलोकन)",
"live_stop_title": "लाइव पूर्वावलोकन रोकें"
},
"timeline": {
"track_label": "सेगमेंट टाइमलाइन",
frontend/src/i18n/locales/id.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "Terjemahan untuk {{langs}} gagal; sulih suara tersebut dilewati agar tidak menghasilkan trek dalam bahasa yang salah.",
"manage_languages": "Kelola bahasa",
"languages_done": "Selesai",
"languages_pending": "Tertunda"
"languages_pending": "Tertunda",
"live_preview": "Pratinjau langsung",
"live_preview_title": "Dengarkan baris saat Anda mengeditnya — segmen yang diedit menstreaming TTS dengan suara CAST-nya setelah jeda singkat. Hanya pemutaran: tidak ada yang disimpan, dan ekspor tetap dirender dalam kualitas penuh.",
"live_preview_pick_voice": "Tetapkan suara CAST ke baris ini atau pembicaranya untuk mendengar pratinjau langsung."
},
"glossary": {
"title": "Glosarium",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "Audio pas di dalam slot.",
"fit_ratio_title": "Audio TTS adalah {{pct}}% dari slot.",
"qc_verify": "Verifikasi",
"qc_verify_title": "ASR lintasan kedua mendengar: \"{{heard}}\" — dengarkan ulang atau sulih suara ulang baris ini."
"qc_verify_title": "ASR lintasan kedua mendengar: \"{{heard}}\" — dengarkan ulang atau sulih suara ulang baris ini.",
"live_play_title": "Streaming baris ini sekarang (pratinjau langsung)",
"live_stop_title": "Hentikan pratinjau langsung"
},
"timeline": {
"track_label": "Lini masa segmen",
frontend/src/i18n/locales/it.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "La traduzione non è riuscita per {{langs}}; i relativi doppiaggi sono stati ignorati per evitare una traccia nella lingua sbagliata.",
"manage_languages": "Gestisci lingue",
"languages_done": "Completate",
"languages_pending": "In attesa"
"languages_pending": "In attesa",
"live_preview": "Anteprima dal vivo",
"live_preview_title": "Ascolta una riga mentre la modifichi: il segmento modificato trasmette il TTS con la sua voce CAST dopo una breve pausa. Solo riproduzione: nulla viene salvato e l'esportazione resta a piena qualità.",
"live_preview_pick_voice": "Assegna una voce CAST a questa riga o al suo parlante per sentire l'anteprima dal vivo."
},
"glossary": {
"title": "Glossario",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "L'audio si inserisce all'interno dello slot.",
"fit_ratio_title": "L'audio TTS è il {{pct}}% dello slot.",
"qc_verify": "Verifica",
"qc_verify_title": "ASR di secondo passaggio ha sentito: \"{{heard}}\" — riascolta o ri-doppia questa riga."
"qc_verify_title": "ASR di secondo passaggio ha sentito: \"{{heard}}\" — riascolta o ri-doppia questa riga.",
"live_play_title": "Trasmetti questa riga ora (anteprima dal vivo)",
"live_stop_title": "Ferma l'anteprima dal vivo"
},
"timeline": {
"track_label": "Timeline dei segmenti",
frontend/src/i18n/locales/ja.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "{{langs}} の翻訳に失敗したため、誤った言語のトラックが生成されないよう該当する吹き替えをスキップしました。",
"manage_languages": "言語を管理",
"languages_done": "完了",
"languages_pending": "保留中"
"languages_pending": "保留中",
"live_preview": "ライブプレビュー",
"live_preview_title": "編集中のセリフをそのまま試聴 — 編集したセグメントは少し置いてから CAST ボイスで TTS をストリーミングします。再生のみ: 何も保存されず、書き出しは引き続きフル品質でレンダリングされます。",
"live_preview_pick_voice": "ライブプレビューを聴くには、この行または話者に CAST ボイスを割り当ててください。"
},
"glossary": {
"title": "用語集",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "オーディオはスロット内に収まります。",
"fit_ratio_title": "TTS オーディオはスロットの {{pct}}% です。",
"qc_verify": "検証する",
"qc_verify_title": "2 パス目の ASR は次のように聞きました: 「{{heard}}」 — この行をもう一度聞くか、再ダビングします。"
"qc_verify_title": "2 パス目の ASR は次のように聞きました: 「{{heard}}」 — この行をもう一度聞くか、再ダビングします。",
"live_play_title": "この行を今すぐストリーミング(ライブプレビュー)",
"live_stop_title": "ライブプレビューを停止"
},
"timeline": {
"track_label": "セグメントタイムライン",
frontend/src/i18n/locales/ko.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "{{langs}} 번역에 실패하여 잘못된 언어의 트랙이 생성되지 않도록 해당 더빙을 건너뛰었습니다.",
"manage_languages": "언어 관리",
"languages_done": "완료",
"languages_pending": "대기 중"
"languages_pending": "대기 중",
"live_preview": "실시간 미리듣기",
"live_preview_title": "편집 중인 문장을 바로 들어보세요 — 편집한 세그먼트는 잠시 후 CAST 보이스로 TTS를 스트리밍합니다. 재생 전용: 아무것도 저장되지 않으며 내보내기는 여전히 최고 품질로 렌더링됩니다.",
"live_preview_pick_voice": "실시간 미리듣기를 들으려면 이 줄이나 화자에게 CAST 보이스를 지정하세요."
},
"glossary": {
"title": "용어집",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "오디오는 슬롯 안에 맞습니다.",
"fit_ratio_title": "TTS 오디오는 슬롯의 {{pct}}%입니다.",
"qc_verify": "확인",
"qc_verify_title": "두 번째 패스 ASR 수신: \"{{heard}}\" — 이 라인을 다시 듣거나 다시 더빙합니다."
"qc_verify_title": "두 번째 패스 ASR 수신: \"{{heard}}\" — 이 라인을 다시 듣거나 다시 더빙합니다.",
"live_play_title": "이 줄 지금 스트리밍(실시간 미리듣기)",
"live_stop_title": "실시간 미리듣기 중지"
},
"timeline": {
"track_label": "세그먼트 타임라인",
frontend/src/i18n/locales/nl.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "De vertaling voor {{langs}} is mislukt. Deze nasynchronisaties zijn overgeslagen om een track in de verkeerde taal te voorkomen.",
"manage_languages": "Talen beheren",
"languages_done": "Gereed",
"languages_pending": "In behandeling"
"languages_pending": "In behandeling",
"live_preview": "Livevoorbeeld",
"live_preview_title": "Hoor een regel terwijl je die bewerkt — het bewerkte segment streamt na een korte pauze TTS met zijn CAST-stem. Alleen afspelen: er wordt niets opgeslagen en exporteren rendert nog steeds op volledige kwaliteit.",
"live_preview_pick_voice": "Wijs een CAST-stem toe aan deze regel of de spreker om het livevoorbeeld te horen."
},
"glossary": {
"title": "Woordenlijst",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "Audio past in de sleuf.",
"fit_ratio_title": "TTS-audio is {{pct}}% van de sleuf.",
"qc_verify": "Verifieer",
"qc_verify_title": "Tweede passage ASR hoorde: \"{{heard}}\" — luister of kopieer deze regel opnieuw."
"qc_verify_title": "Tweede passage ASR hoorde: \"{{heard}}\" — luister of dub deze regel opnieuw.",
"live_play_title": "Deze regel nu streamen (livevoorbeeld)",
"live_stop_title": "Livevoorbeeld stoppen"
},
"timeline": {
"track_label": "Segmenttijdlijn",
frontend/src/i18n/locales/pl.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "Tłumaczenie dla {{langs}} nie powiodło się. Te dubbingi pominięto, aby nie utworzyć ścieżki w niewłaściwym języku.",
"manage_languages": "Zarządzaj językami",
"languages_done": "Gotowe",
"languages_pending": "Oczekujące"
"languages_pending": "Oczekujące",
"live_preview": "Podgląd na żywo",
"live_preview_title": "Słuchaj linii podczas edycji — edytowany segment po krótkiej pauzie streamuje TTS z przypisanym głosem CAST. Tylko odtwarzanie: nic nie jest zapisywane, a eksport nadal renderuje się w pełnej jakości.",
"live_preview_pick_voice": "Przypisz głos CAST tej linii lub jej mówcy, aby usłyszeć podgląd na żywo."
},
"glossary": {
"title": "Glosariusz",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "Dźwięk mieści się w gnieździe.",
"fit_ratio_title": "Dźwięk TTS zajmuje {{pct}}% szczeliny.",
"qc_verify": "Zweryfikuj",
"qc_verify_title": "ASR drugiego przejścia usłyszał: „{{heard}}” — posłuchaj ponownie lub ponownie dubuj tę linijkę."
"qc_verify_title": "ASR drugiego przejścia usłyszał: „{{heard}}” — posłuchaj ponownie lub ponownie dubuj tę linijkę.",
"live_play_title": "Streamuj tę linię teraz (podgląd na żywo)",
"live_stop_title": "Zatrzymaj podgląd na żywo"
},
"timeline": {
"track_label": "Oś czasu segmentów",
frontend/src/i18n/locales/pt.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "A tradução de {{langs}} falhou; essas dublagens foram ignoradas para evitar uma faixa no idioma errado.",
"manage_languages": "Gerenciar idiomas",
"languages_done": "Concluído",
"languages_pending": "Pendente"
"languages_pending": "Pendente",
"live_preview": "Prévia ao vivo",
"live_preview_title": "Ouça uma linha enquanto a edita — o segmento editado transmite TTS com sua voz do CAST após uma breve pausa. Apenas reprodução: nada é salvo e a exportação continua renderizando em qualidade total.",
"live_preview_pick_voice": "Atribua uma voz do CAST a esta linha ou ao seu falante para ouvir a prévia ao vivo."
},
"glossary": {
"title": "Glossário",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "O áudio cabe dentro do slot.",
"fit_ratio_title": "O áudio TTS é {{pct}}% do slot.",
"qc_verify": "Verifique",
"qc_verify_title": "ASR de segunda passagem ouvido: \"{{heard}}\" — ouça novamente ou duble esta linha."
"qc_verify_title": "ASR de segunda passagem ouvido: \"{{heard}}\" — ouça novamente ou duble esta linha.",
"live_play_title": "Transmitir esta linha agora (prévia ao vivo)",
"live_stop_title": "Parar a prévia ao vivo"
},
"timeline": {
"track_label": "Linha do tempo de segmentos",
frontend/src/i18n/locales/ru.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "Не удалось перевести {{langs}}. Эти дорожки пропущены, чтобы не создать озвучку на неверном языке.",
"manage_languages": "Управление языками",
"languages_done": "Готово",
"languages_pending": "Ожидает"
"languages_pending": "Ожидает",
"live_preview": "Живой предпросмотр",
"live_preview_title": "Слушайте строку прямо во время правки — изменённый сегмент после короткой паузы стримит TTS голосом из CAST. Только воспроизведение: ничего не сохраняется, экспорт по-прежнему рендерится в полном качестве.",
"live_preview_pick_voice": "Назначьте голос CAST этой строке или её спикеру, чтобы услышать живой предпросмотр."
},
"glossary": {
"title": "Глоссарий",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "Аудио поместилось внутри слота.",
"fit_ratio_title": "Звук TTS занимает {{pct}}% слота.",
"qc_verify": "Проверять",
"qc_verify_title": "При втором проходе ASR услышал: «{{heard}}» — прослушайте или перезапишите эту строку."
"qc_verify_title": "При втором проходе ASR услышал: «{{heard}}» — прослушайте или перезапишите эту строку.",
"live_play_title": "Стримить эту строку сейчас (живой предпросмотр)",
"live_stop_title": "Остановить живой предпросмотр"
},
"timeline": {
"track_label": "Шкала сегментов",
frontend/src/i18n/locales/sv.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "Översättningen misslyckades för {{langs}}. Dessa dubbningar hoppades över för att undvika ett spår på fel språk.",
"manage_languages": "Hantera språk",
"languages_done": "Klart",
"languages_pending": "Väntar"
"languages_pending": "Väntar",
"live_preview": "Live-förhandsvisning",
"live_preview_title": "Hör en rad medan du redigerar den — det redigerade segmentet strömmar TTS med sin CAST-röst efter en kort paus. Endast uppspelning: inget sparas och exporten renderas fortfarande i full kvalitet.",
"live_preview_pick_voice": "Tilldela en CAST-röst till den här raden eller dess talare för att höra live-förhandsvisningen."
},
"glossary": {
"title": "Ordlista",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "Ljudet passar in i öppningen.",
"fit_ratio_title": "TTS-ljud är {{pct}}% av kortplatsen.",
"qc_verify": "Verifiera",
"qc_verify_title": "Second-pass ASR hörde: \"{{heard}}\" — lyssna igen eller dubba om den här raden."
"qc_verify_title": "Second-pass ASR hörde: \"{{heard}}\" — lyssna igen eller dubba om den här raden.",
"live_play_title": "Strömma den här raden nu (live-förhandsvisning)",
"live_stop_title": "Stoppa live-förhandsvisningen"
},
"timeline": {
"track_label": "Segmenttidslinje",
frontend/src/i18n/locales/th.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "การแปลสำหรับ {{langs}} ล้มเหลว จึงข้ามเสียงพากย์เหล่านั้นเพื่อป้องกันไม่ให้สร้างแทร็กผิดภาษา",
"manage_languages": "จัดการภาษา",
"languages_done": "เสร็จแล้ว",
"languages_pending": "รอดำเนินการ"
"languages_pending": "รอดำเนินการ",
"live_preview": "พรีวิวสด",
"live_preview_title": "ฟังบรรทัดขณะแก้ไข — เซกเมนต์ที่แก้ไขจะสตรีม TTS ด้วยเสียง CAST ของมันหลังหยุดพิมพ์ครู่หนึ่ง เป็นการเล่นเสียงเท่านั้น: ไม่มีการบันทึกใด ๆ และการส่งออกยังคงเรนเดอร์ที่คุณภาพเต็ม",
"live_preview_pick_voice": "กำหนดเสียง CAST ให้บรรทัดนี้หรือผู้พูดของบรรทัดเพื่อฟังพรีวิวสด"
},
"glossary": {
"title": "อภิธานศัพท์",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "เสียงพอดีกับช่อง",
"fit_ratio_title": "เสียง TTS คือ {{pct}}% ของช่อง",
"qc_verify": "ตรวจสอบ",
"qc_verify_title": "ได้ยิน ASR ครั้งที่สอง: \"{{heard}}\" — ฟังซ้ำหรือพากย์บรรทัดนี้ใหม่"
"qc_verify_title": "ได้ยิน ASR ครั้งที่สอง: \"{{heard}}\" — ฟังซ้ำหรือพากย์บรรทัดนี้ใหม่",
"live_play_title": "สตรีมบรรทัดนี้ทันที (พรีวิวสด)",
"live_stop_title": "หยุดพรีวิวสด"
},
"timeline": {
"track_label": "ไทม์ไลน์ของเซกเมนต์",
frontend/src/i18n/locales/tr.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "{{langs}} için çeviri başarısız oldu; yanlış dilde parça oluşmaması için bu dublajlar atlandı.",
"manage_languages": "Dilleri yönet",
"languages_done": "Tamamlandı",
"languages_pending": "Bekliyor"
"languages_pending": "Bekliyor",
"live_preview": "Canlı önizleme",
"live_preview_title": "Bir satırı düzenlerken dinleyin — düzenlenen segment kısa bir duraklamadan sonra CAST sesiyle TTS akışı yapar. Yalnızca oynatma: hiçbir şey kaydedilmez ve dışa aktarma yine tam kalitede işlenir.",
"live_preview_pick_voice": "Canlı önizlemeyi duymak için bu satıra veya konuşmacısına bir CAST sesi atayın."
},
"glossary": {
"title": "Sözlük",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "Ses yuvanın içine sığar.",
"fit_ratio_title": "TTS sesi yuvanın %{{pct}} kadarıdır.",
"qc_verify": "Doğrula",
"qc_verify_title": "İkinci geçiş ASR şunu duydu: \"{{heard}}\" — bu satırı yeniden dinleyin veya yeniden dublajlayın."
"qc_verify_title": "İkinci geçiş ASR şunu duydu: \"{{heard}}\" — bu satırı yeniden dinleyin veya yeniden dublajlayın.",
"live_play_title": "Bu satırı şimdi akıt (canlı önizleme)",
"live_stop_title": "Canlı önizlemeyi durdur"
},
"timeline": {
"track_label": "Segment zaman çizelgesi",
frontend/src/i18n/locales/uk.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "Не вдалося перекласти {{langs}}. Ці доріжки пропущено, щоб не створити дубляж неправильною мовою.",
"manage_languages": "Керувати мовами",
"languages_done": "Готово",
"languages_pending": "Очікує"
"languages_pending": "Очікує",
"live_preview": "Прослуховування наживо",
"live_preview_title": "Слухайте рядок під час редагування — змінений сегмент після короткої паузи стрімить TTS голосом із CAST. Лише відтворення: нічого не зберігається, експорт і далі рендериться в повній якості.",
"live_preview_pick_voice": "Призначте голос CAST цьому рядку або його мовцеві, щоб почути прослуховування наживо."
},
"glossary": {
"title": "Глосарій",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "Аудіо вміщується в слот.",
"fit_ratio_title": "Аудіо TTS займає {{pct}}% слота.",
"qc_verify": "Підтвердити",
"qc_verify_title": "ASR другого проходу почула: \"{{heard}}\" — переслухайте або дублюйте цей рядок."
"qc_verify_title": "ASR другого проходу почула: \"{{heard}}\" — переслухайте або дублюйте цей рядок.",
"live_play_title": "Стрімити цей рядок зараз (наживо)",
"live_stop_title": "Зупинити прослуховування наживо"
},
"timeline": {
"track_label": "Шкала сегментів",
frontend/src/i18n/locales/vi.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "Không thể dịch {{langs}}; các bản lồng tiếng đó đã được bỏ qua để tránh tạo bản nhạc sai ngôn ngữ.",
"manage_languages": "Quản lý ngôn ngữ",
"languages_done": "Hoàn tất",
"languages_pending": "Đang chờ"
"languages_pending": "Đang chờ",
"live_preview": "Nghe thử trực tiếp",
"live_preview_title": "Nghe từng dòng ngay khi chỉnh sửa — phân đoạn vừa sửa sẽ stream TTS bằng giọng CAST của nó sau một khoảng dừng ngắn. Chỉ phát lại: không lưu gì cả, và xuất video vẫn kết xuất ở chất lượng đầy đủ.",
"live_preview_pick_voice": "Gán giọng CAST cho dòng này hoặc người nói của nó để nghe thử trực tiếp."
},
"glossary": {
"title": "Thuật ngữ",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "Âm thanh vừa vặn bên trong khe cắm.",
"fit_ratio_title": "Âm thanh TTS chiếm {{pct}}% dung lượng khe.",
"qc_verify": "Xác minh",
"qc_verify_title": "Đã nghe thấy ASR vượt qua lần thứ hai: \"{{heard}}\" — nghe lại hoặc lồng tiếng lại dòng này."
"qc_verify_title": "Đã nghe thấy ASR vượt qua lần thứ hai: \"{{heard}}\" — nghe lại hoặc lồng tiếng lại dòng này.",
"live_play_title": "Stream dòng này ngay (nghe thử trực tiếp)",
"live_stop_title": "Dừng nghe thử trực tiếp"
},
"timeline": {
"track_label": "Dòng thời gian phân đoạn",
frontend/src/i18n/locales/zh-CN.json
+7 -2
View File
@@ -1101,7 +1101,10 @@
"multi_lang_skipped": "{{langs}} 翻译失败——已跳过这些配音，以免生成语言错误的音轨。",
"manage_languages": "管理语言",
"languages_done": "已完成",
"languages_pending": "待处理"
"languages_pending": "待处理",
"live_preview": "实时预览",
"live_preview_title": "边编辑边试听 — 停顿片刻后,所编辑的片段会用其 CAST 音色流式合成语音。仅播放:不会保存任何内容,导出仍按完整质量渲染。",
"live_preview_pick_voice": "为该行或其说话人指定一个 CAST 音色,即可收听实时预览。"
},
"glossary": {
"title": "术语表",
@@ -1198,7 +1201,9 @@
"fit_audio_title": "音频适合插槽内。",
"fit_ratio_title": "TTS 音频占插槽的 {{pct}}%。",
"qc_verify": "验证",
"qc_verify_title": "第二遍 ASR 听到：“{{heard}}” — 重新聆听或重新配音此台词。"
"qc_verify_title": "第二遍 ASR 听到：“{{heard}}” — 重新聆听或重新配音此台词。",
"live_play_title": "立即流式播放该行(实时预览)",
"live_stop_title": "停止实时预览"
},
"timeline": {
"track_label": "片段时间轴",
frontend/src/i18n/locales/zh-TW.json
+7 -2
View File
@@ -1142,7 +1142,10 @@
"multi_lang_skipped": "{{langs}} 翻譯失敗；已略過這些配音，以免產生語言錯誤的音軌。",
"manage_languages": "管理語言",
"languages_done": "已完成",
"languages_pending": "待處理"
"languages_pending": "待處理",
"live_preview": "即時預覽",
"live_preview_title": "邊編輯邊試聽 — 停頓片刻後,所編輯的片段會以其 CAST 聲音串流合成語音。僅播放:不會儲存任何內容,匯出仍以完整品質算繪。",
"live_preview_pick_voice": "為此行或其說話者指定一個 CAST 聲音,即可聆聽即時預覽。"
},
"glossary": {
"title": "詞彙表",
@@ -1239,7 +1242,9 @@
"fit_audio_title": "音訊適合插槽內。",
"fit_ratio_title": "TTS 音訊佔插槽的 {{pct}}%。",
"qc_verify": "驗證",
"qc_verify_title": "第二遍 ASR 聽到：「{{heard}}」 — 重新聆聽或重新配音此台詞。"
"qc_verify_title": "第二遍 ASR 聽到：「{{heard}}」 — 重新聆聽或重新配音此台詞。",
"live_play_title": "立即串流此行(即時預覽)",
"live_stop_title": "停止即時預覽"
},
"timeline": {
"track_label": "片段時間軸",
frontend/src/index.css
+26
View File
@@ -4077,6 +4077,32 @@ html[data-window='widget'] body:has(.capture-pill) {
display: inline-flex; align-items: center; justify-content: center;
}
/* Live dub preview (opt-in, /ws/tts as-you-edit streaming) */
.dub-live-toggle {
display: inline-flex; align-items: center; gap: 4px;
font-size: 0.62rem; color: var(--chrome-fg-muted);
cursor: pointer; white-space: nowrap; user-select: none;
}
.dub-live-toggle:hover { color: var(--chrome-fg); }
.seg-live-btn {
width: 18px; height: 18px; flex-shrink: 0;
display: inline-flex; align-items: center; justify-content: center;
background: transparent; border: 1px solid var(--chrome-border);
color: var(--chrome-fg-muted); border-radius: var(--chrome-radius-pill);
cursor: pointer; padding: 0;
transition: color var(--dur-fast), border-color var(--dur-fast);
}
.seg-live-btn:hover { color: var(--chrome-fg); border-color: var(--chrome-border-strong); }
.seg-live-btn--on {
color: var(--color-brand);
border-color: var(--color-brand);
animation: seg-live-pulse 1.1s ease-in-out infinite;
}
@keyframes seg-live-pulse {
0%, 100% { opacity: 1; }
50% { opacity: 0.35; }
}
/* Row selected from the waveform timeline (#280, item 3) — distinct from
multi-select (checkbox) and from the playhead row. */
frontend/src/store/index.ts
+1
View File
@@ -164,6 +164,7 @@ export const useAppStore = create<AppStore>()(
timingStrategy: s.timingStrategy,
fitOptions: s.fitOptions,
voiceMatch: s.voiceMatch,
dubLivePreview: s.dubLivePreview,
// "What's new" affordance (feat/safe-updates) — remembering which
// version's notes were seen only works if it survives restarts.
whatsNewSeenVersion: s.whatsNewSeenVersion,
frontend/src/store/prefsSlice.ts
+12
View File
@@ -135,6 +135,16 @@ export interface PrefsSlice {
/** Dub voice-identity mode — see the VoiceMatch type doc. */
voiceMatch: VoiceMatch;
/**
* Opt-in live dub preview (default OFF). When on, editing a segment's
* translated text streams TTS for that line over the existing `/ws/tts`
* socket (debounced, local playback only) so the user hears the edit
* without pressing Generate. Never persisted as job audio — export still
* goes through the full-quality generate path.
*/
dubLivePreview: boolean;
setDubLivePreview: (on: boolean) => void;
/**
* Last app version whose release notes the user has seen (feat/safe-updates).
* `null` = never recorded (fresh install / pre-feature profile): the first
@@ -293,6 +303,7 @@ export const createPrefsSlice: StateCreator<PrefsSlice, [], [], PrefsSlice> = (s
timingStrategy: 'strict_slot',
fitOptions: null,
voiceMatch: 'per_line',
dubLivePreview: false,
whatsNewSeenVersion: null,
dismissedNotificationIds: [],
aecEnabled: false,
@@ -318,6 +329,7 @@ export const createPrefsSlice: StateCreator<PrefsSlice, [], [], PrefsSlice> = (s
setTimingStrategy: (s) => set({ timingStrategy: s }),
setFitOptions: (o) => set({ fitOptions: o }),
setVoiceMatch: (m) => set({ voiceMatch: m }),
setDubLivePreview: (on) => set({ dubLivePreview: on }),
setWhatsNewSeenVersion: (v) => set({ whatsNewSeenVersion: v }),
dismissNotification: (id) =>
set((s) => ({
frontend/src/test/dubLivePreview.test.jsx
+385
View File
@@ -0,0 +1,385 @@
/**
* Live dub preview (ROADMAP: "Real-time dub preview (stream TTS as you edit)").
*
* The contract under test:
* • an edit streams over /ws/tts only after the ~400 ms debounce, with the
* segment's CAST voice resolved through segmentGenInputs;
* • a new keystroke or a segment switch closes the previous socket — no
* socket pile-up;
* • every stream runs inside the process-wide withTtsInflight admission the
* /generate and Convert paths share (busy → localized toast, no socket);
* • a segment with no CAST voice never opens a socket and gets an
* actionable "assign a voice" toast;
* • binary PCM frames reach the chunk player; `done` finalizes it; abort
* silences it. Nothing here persists job audio.
*/
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { renderHook, act } from '@testing-library/react';
const { mocks } = vi.hoisted(() => {
const players = [];
return {
mocks: {
state: { ttsInflight: 0, dubLang: 'Auto' },
authenticatedWsUrl: vi.fn(async () => 'ws://test/ws/tts?ws_ticket=one-use'),
players,
createStreamingChunkPlayer: vi.fn((options = {}) => {
const player = {
appendPcm16Base64: vi.fn(),
appendPcm16Bytes: vi.fn(),
finalize: vi.fn(),
fail: vi.fn(),
onDone: options.onDone,
};
players.push(player);
return player;
}),
toast: Object.assign(vi.fn(), { error: vi.fn() }),
},
};
});
vi.mock('../api/client', () => ({
API: 'http://test',
apiUrl: (p) => p,
apiFetch: vi.fn(),
apiJson: vi.fn(),
}));
vi.mock('../api/authSession', () => ({ authenticatedWsUrl: mocks.authenticatedWsUrl }));
vi.mock('../utils/generatePreflight', () => ({ warnIfEngineUnderProvisioned: vi.fn() }));
vi.mock('../utils/streamingTts', () => ({
createStreamingChunkPlayer: mocks.createStreamingChunkPlayer,
supportsStreamingPreview: () => true,
}));
vi.mock('react-hot-toast', () => ({ toast: mocks.toast }));
vi.mock('react-i18next', () => ({ useTranslation: () => ({ t: (k) => k }) }));
vi.mock('../store', () => ({
useAppStore: {
getState: () => ({
...mocks.state,
addTtsInflight: (d) => {
mocks.state.ttsInflight = Math.max(0, mocks.state.ttsInflight + d);
},
}),
},
}));
import useDubLivePreview, { LIVE_PREVIEW_DEBOUNCE_MS } from '../hooks/useDubLivePreview';
class FakeWebSocket {
static instances = [];
constructor(url) {
this.url = url;
this.readyState = 0;
this.sent = [];
this.closed = false;
this.binaryType = 'blob';
FakeWebSocket.instances.push(this);
}
send(data) {
this.sent.push(data);
}
close() {
if (this.closed) return;
this.closed = true;
this.readyState = 3;
this.onclose?.({ code: 1000 });
}
open() {
this.readyState = 1;
this.onopen?.();
}
}
const SEG = { id: 'seg-1', text: 'Hola mundo', profile_id: 'prof-1', target_lang: 'es' };
const renderPreview = (enabled = true) =>
renderHook(({ on }) => useDubLivePreview({ enabled: on }), { initialProps: { on: enabled } });
/** Type into a segment, run the debounce, and flush the async connect. */
const editAndSettle = async (result, seg, text) => {
act(() => result.current.onLiveEdit(seg, text));
await act(async () => {
await vi.advanceTimersByTimeAsync(LIVE_PREVIEW_DEBOUNCE_MS);
});
};
beforeEach(() => {
vi.useFakeTimers();
FakeWebSocket.instances = [];
mocks.players.length = 0;
mocks.state.ttsInflight = 0;
mocks.state.dubLang = 'Auto';
mocks.authenticatedWsUrl.mockClear();
mocks.createStreamingChunkPlayer.mockClear();
mocks.toast.mockClear();
mocks.toast.error.mockClear();
vi.stubGlobal('WebSocket', FakeWebSocket);
});
afterEach(() => {
vi.useRealTimers();
vi.unstubAllGlobals();
});
describe('useDubLivePreview', () => {
it('streams only after the debounce, with the CAST voice and one-use ticket', async () => {
const { result } = renderPreview();
act(() => result.current.onLiveEdit(SEG, 'Hola mundo'));
await act(async () => {
await vi.advanceTimersByTimeAsync(LIVE_PREVIEW_DEBOUNCE_MS - 1);
});
expect(FakeWebSocket.instances).toHaveLength(0);
await act(async () => {
await vi.advanceTimersByTimeAsync(1);
});
expect(mocks.authenticatedWsUrl).toHaveBeenCalledWith('/ws/tts', { apiBase: 'http://test' });
expect(FakeWebSocket.instances).toHaveLength(1);
const ws = FakeWebSocket.instances[0];
expect(ws.url).toContain('ws_ticket=one-use');
expect(ws.binaryType).toBe('arraybuffer');
act(() => ws.open());
expect(JSON.parse(ws.sent[0])).toEqual({
text: 'Hola mundo',
voice: 'prof-1',
speed: 1.0,
language: 'es',
});
});
it('plays binary PCM frames through the chunk player and finalizes on done', async () => {
const { result } = renderPreview();
await editAndSettle(result, SEG, 'Hola mundo');
const ws = FakeWebSocket.instances[0];
act(() => ws.open());
expect(mocks.state.ttsInflight).toBe(1); // admission held while streaming
act(() => ws.onmessage({ data: JSON.stringify({ type: 'start', sample_rate: 24000 }) }));
expect(mocks.createStreamingChunkPlayer).toHaveBeenCalledWith(
expect.objectContaining({ sampleRate: 24000 }),
);
const frame = new Int16Array([100, -100, 500]).buffer;
act(() => ws.onmessage({ data: frame }));
expect(mocks.players[0].appendPcm16Bytes).toHaveBeenCalledWith(frame);
await act(async () => {
ws.onmessage({ data: JSON.stringify({ type: 'done', duration_s: 1.2 }) });
await vi.advanceTimersByTimeAsync(0);
});
expect(mocks.players[0].finalize).toHaveBeenCalled();
// done ≠ abort: the buffered tail keeps playing, it is not torn down.
expect(mocks.players[0].fail).not.toHaveBeenCalled();
expect(mocks.state.ttsInflight).toBe(0); // admission released
expect(result.current.liveSegId).toBe(null);
});
it('can still silence a finalized buffered tail when a newer edit arrives', async () => {
const { result } = renderPreview();
await editAndSettle(result, SEG, 'Old line');
const ws = FakeWebSocket.instances[0];
act(() => ws.open());
act(() => ws.onmessage({ data: JSON.stringify({ type: 'start', sample_rate: 24000 }) }));
await act(async () => {
ws.onmessage({ data: JSON.stringify({ type: 'done' }) });
await vi.advanceTimersByTimeAsync(0);
});
expect(mocks.players[0].finalize).toHaveBeenCalled();
expect(mocks.players[0].fail).not.toHaveBeenCalled();
act(() => result.current.onLiveEdit(SEG, 'New line'));
expect(mocks.players[0].fail).toHaveBeenCalledOnce();
});
it('closes the previous socket on the next keystroke', async () => {
const { result } = renderPreview();
await editAndSettle(result, SEG, 'Hola mun');
const first = FakeWebSocket.instances[0];
act(() => first.open());
act(() => first.onmessage({ data: JSON.stringify({ type: 'start', sample_rate: 24000 }) }));
await editAndSettle(result, SEG, 'Hola mundo');
expect(first.closed).toBe(true);
expect(mocks.players[0].fail).toHaveBeenCalled(); // the stale audio is silenced
expect(FakeWebSocket.instances).toHaveLength(2);
expect(FakeWebSocket.instances[1].closed).toBe(false);
expect(mocks.state.ttsInflight).toBe(1); // exactly one admission slot held
});
it('closes the previous socket when the user moves to another segment', async () => {
const { result } = renderPreview();
await editAndSettle(result, SEG, 'Hola mundo');
const first = FakeWebSocket.instances[0];
act(() => first.open());
const other = { id: 'seg-2', text: 'Adiós', profile_id: 'prof-2' };
await editAndSettle(result, other, 'Adiós amigo');
expect(first.closed).toBe(true);
const second = FakeWebSocket.instances[1];
act(() => second.open());
expect(JSON.parse(second.sent[0]).voice).toBe('prof-2');
expect(result.current.liveSegId).toBe('seg-2');
});
it('does not start an older intent when teardown synchronously queues a newer edit', async () => {
const { result } = renderPreview();
await editAndSettle(result, SEG, 'First line');
const first = FakeWebSocket.instances[0];
act(() => first.open());
act(() => first.onmessage({ data: JSON.stringify({ type: 'start', sample_rate: 24000 }) }));
const older = { id: 'seg-2', text: 'Older', profile_id: 'prof-2' };
const latest = { id: 'seg-3', text: 'Latest', profile_id: 'prof-3' };
mocks.players[0].fail.mockImplementationOnce(() => {
result.current.onLiveEdit(latest, latest.text);
});
await act(async () => {
result.current.onLiveToggle(older);
await vi.advanceTimersByTimeAsync(0);
});
// The older toggle was superseded while it awaited admission teardown.
// It must not open a socket during the latest edit's debounce window.
expect(FakeWebSocket.instances).toHaveLength(1);
await act(async () => {
await vi.advanceTimersByTimeAsync(LIVE_PREVIEW_DEBOUNCE_MS);
});
expect(FakeWebSocket.instances).toHaveLength(2);
const latestSocket = FakeWebSocket.instances[1];
act(() => latestSocket.open());
expect(JSON.parse(latestSocket.sent[0])).toEqual(
expect.objectContaining({ text: 'Latest', voice: 'prof-3' }),
);
});
it('skips segments with no CAST voice and says how to fix it', async () => {
const { result } = renderPreview();
await editAndSettle(result, { id: 'seg-3', text: 'Sin voz' }, 'Sin voz');
expect(FakeWebSocket.instances).toHaveLength(0);
expect(mocks.toast).toHaveBeenCalledWith('dub.live_preview_pick_voice', expect.anything());
});
it('expands legacy preset: voices to instruct text instead of skipping them', async () => {
const { result } = renderPreview();
const seg = { id: 'seg-4', text: 'Preset line', profile_id: 'preset:narrator' };
await editAndSettle(result, seg, 'Preset line');
// Same segmentGenInputs expansion the dub generate body uses: the request
// must carry instruct text, never a profile id the backend can't resolve.
expect(FakeWebSocket.instances).toHaveLength(1);
const ws = FakeWebSocket.instances[0];
act(() => ws.open());
const payload = JSON.parse(ws.sent[0]);
expect(payload.voice).toBeUndefined();
expect(payload.instruct).toContain('male');
});
it('respects the process-wide TTS admission: busy → toast, no socket', async () => {
mocks.state.ttsInflight = 1; // a /generate or Convert is running
const { result } = renderPreview();
await editAndSettle(result, SEG, 'Hola mundo');
expect(FakeWebSocket.instances).toHaveLength(0);
expect(mocks.toast).toHaveBeenCalledWith(
'tts_errors.generation_in_progress',
expect.anything(),
);
expect(mocks.state.ttsInflight).toBe(1); // untouched — we never admitted
expect(result.current.liveSegId).toBe(null);
});
it('does nothing while the pref is off, and stops the stream when toggled off', async () => {
const { result, rerender } = renderPreview(false);
await editAndSettle(result, SEG, 'Hola mundo');
expect(FakeWebSocket.instances).toHaveLength(0);
rerender({ on: true });
await editAndSettle(result, SEG, 'Hola mundo');
expect(FakeWebSocket.instances).toHaveLength(1);
act(() => FakeWebSocket.instances[0].open());
await act(async () => {
rerender({ on: false });
await vi.advanceTimersByTimeAsync(0);
});
expect(FakeWebSocket.instances[0].closed).toBe(true);
expect(mocks.state.ttsInflight).toBe(0);
});
it('surfaces backend error frames and releases admission', async () => {
const { result } = renderPreview();
await editAndSettle(result, SEG, 'Hola mundo');
const ws = FakeWebSocket.instances[0];
act(() => ws.open());
await act(async () => {
ws.onmessage({ data: JSON.stringify({ type: 'error', detail: 'engine cannot run' }) });
await vi.advanceTimersByTimeAsync(0);
});
expect(mocks.toast.error).toHaveBeenCalledWith('tts_errors.error_prefix');
expect(ws.closed).toBe(true);
expect(mocks.state.ttsInflight).toBe(0);
});
it('tells the user and releases admission when the ticket handshake fails', async () => {
// A backend whose ticket allowlist lacks /ws/tts answers 422; that must
// not look like "the toggle does nothing" (the #1769 first-cut symptom).
mocks.authenticatedWsUrl.mockRejectedValueOnce(new Error('ticket refused'));
const { result } = renderPreview();
await editAndSettle(result, SEG, 'Hola mundo');
await act(async () => {
await vi.advanceTimersByTimeAsync(0);
});
expect(FakeWebSocket.instances).toHaveLength(0);
expect(mocks.toast.error).toHaveBeenCalledWith('tts_errors.error_prefix');
expect(mocks.state.ttsInflight).toBe(0);
expect(result.current.liveSegId).toBe(null);
});
it('releases admission when WebSocket construction throws synchronously', async () => {
vi.stubGlobal(
'WebSocket',
class ThrowingWebSocket {
constructor() {
throw new Error('constructor failed');
}
},
);
const { result } = renderPreview();
await editAndSettle(result, SEG, 'Hola mundo');
await act(async () => {
await vi.advanceTimersByTimeAsync(0);
});
expect(mocks.state.ttsInflight).toBe(0);
expect(result.current.liveSegId).toBe(null);
});
it('row speaker toggle streams the current text immediately and stops on second press', async () => {
const { result } = renderPreview();
await act(async () => {
result.current.onLiveToggle(SEG);
await vi.advanceTimersByTimeAsync(0);
});
expect(FakeWebSocket.instances).toHaveLength(1);
expect(result.current.liveSegId).toBe('seg-1');
await act(async () => {
result.current.onLiveToggle(SEG);
await vi.advanceTimersByTimeAsync(0);
});
expect(FakeWebSocket.instances[0].closed).toBe(true);
expect(result.current.liveSegId).toBe(null);
});
it('cleans up the socket on unmount', async () => {
const { result, unmount } = renderPreview();
await editAndSettle(result, SEG, 'Hola mundo');
act(() => FakeWebSocket.instances[0].open());
unmount();
expect(FakeWebSocket.instances[0].closed).toBe(true);
await act(async () => {
await vi.advanceTimersByTimeAsync(0);
});
expect(mocks.state.ttsInflight).toBe(0);
});
});
frontend/src/test/streamingTts.test.js
+42
View File
@@ -17,6 +17,7 @@ const {
supportsStreamingPreview,
resolveRemoteTtsTarget,
decodePcm16Base64,
pcm16BytesToFloat32,
peaksFromChunkList,
StreamingPreviewError,
shouldFallbackToClassic,
@@ -162,6 +163,47 @@ describe('decodePcm16Base64', () => {
});
});
describe('pcm16BytesToFloat32', () => {
it('decodes a raw binary frame (the /ws/tts shape) identically to base64', () => {
const samples = [0, 16384, -16384, 32767, -32768];
const raw = pcm16BytesToFloat32(new Int16Array(samples).buffer);
const viaB64 = decodePcm16Base64(b64Pcm(samples));
expect(Array.from(raw)).toEqual(Array.from(viaB64));
});
it('decodes a view carved at an odd byteOffset instead of throwing', () => {
const backing = new Uint8Array(5);
backing.set(new Uint8Array(new Int16Array([-12345]).buffer), 1);
const out = pcm16BytesToFloat32(backing.subarray(1, 3));
expect(out.length).toBe(1);
expect(out[0]).toBeCloseTo(-12345 / 32768, 5);
});
it('respects a Uint8Array view with a nonzero byteOffset', () => {
const backing = new Uint8Array(8);
backing.set(new Uint8Array(new Int16Array([12345]).buffer), 2);
const out = pcm16BytesToFloat32(backing.subarray(2, 4));
expect(out.length).toBe(1);
expect(out[0]).toBeCloseTo(12345 / 32768, 5);
});
});
describe('createStreamingChunkPlayer.appendPcm16Bytes', () => {
it('schedules raw /ws/tts frames on the same gapless timeline as base64 chunks', () => {
const player = createStreamingChunkPlayer({ label: 'live', sampleRate: 24000 });
const ctx = FakeAudioContext.instances.at(-1);
const frame = new Int16Array(Array.from({ length: 2400 }, (_, i) => (i % 100) * 50));
player.appendPcm16Bytes(frame.buffer);
player.appendPcm16Bytes(frame.buffer);
expect(ctx.started.length).toBe(2);
// Second chunk starts exactly one chunk-duration later — gapless.
expect(ctx.started[1].startedAt.when).toBeCloseTo(2400 / 24000, 5);
player.fail();
});
});
describe('peaksFromChunkList', () => {
it('returns normalized peaks across chunk boundaries', () => {
const quiet = new Float32Array(1000).fill(0.1);
frontend/src/utils/streamingTts.js
+45 -29
View File
@@ -86,15 +86,26 @@ export class StreamingPreviewError extends Error {
export const shouldFallbackToClassic = (error) =>
error instanceof StreamingPreviewError && !error.retryable && !error.terminal;
/** Raw PCM16 bytes (ArrayBuffer / Uint8Array) → Float32 samples in [-1, 1].
* The binary-WebSocket twin of decodePcm16Base64 (`/ws/tts` sends raw
* frames, not NDJSON base64). Exported for tests. */
export const pcm16BytesToFloat32 = (data) => {
let bytes = data instanceof Uint8Array ? data : new Uint8Array(data);
// An Int16Array view must start on an even byte; a view carved at an odd
// offset (sliced framing buffers) would throw a RangeError, so copy it.
if (bytes.byteOffset % 2) bytes = bytes.slice();
const pcm = new Int16Array(bytes.buffer, bytes.byteOffset, bytes.byteLength >> 1);
const out = new Float32Array(pcm.length);
for (let i = 0; i < pcm.length; i++) out[i] = pcm[i] / 32768;
return out;
};
/** base64 → Int16 PCM → Float32 samples in [-1, 1]. Exported for tests. */
export const decodePcm16Base64 = (b64) => {
const bin = atob(b64);
const bytes = new Uint8Array(bin.length);
for (let i = 0; i < bin.length; i++) bytes[i] = bin.charCodeAt(i);
const pcm = new Int16Array(bytes.buffer, 0, bytes.length >> 1);
const out = new Float32Array(pcm.length);
for (let i = 0; i < pcm.length; i++) out[i] = pcm[i] / 32768;
return out;
return pcm16BytesToFloat32(bytes);
};
/**
@@ -259,33 +270,38 @@ export const createStreamingChunkPlayer = ({ label, sampleRate, crossfadeMs = 0,
}
}, 250);
// Schedule one decoded Float32 chunk for gapless playback. Shared by the
// NDJSON (base64) and binary-WebSocket (raw PCM16 frame) append shapes.
const appendSamples = (data) => {
if (finished) return;
if (!data.length) return;
const i = chunks.length;
chunks.push(data);
starts.push(i === 0 ? 0 : starts[i - 1] + dur(i - 1) - fadeFor(i));
totalDuration = starts[i] + dur(i);
const when = anchor + (starts[i] - baseOffset);
if (i > 0 && when < ctx.currentTime - 0.01) {
// Underrun: playback drained the buffered chunks before this one
// arrived. Re-anchor so the new chunk starts now (the playhead sat at
// the buffered edge) instead of clipping its head.
baseOffset = starts[i];
anchor = ctx.currentTime + 0.02;
scheduleChunk(i, 0, false);
} else {
scheduleChunk(i);
}
session.update({
duration: totalDuration,
peaks: peaksFromChunkList(chunks),
});
};
return {
/** Append one base64 PCM16 chunk and schedule it for gapless playback. */
appendPcm16Base64: (b64) => {
if (finished) return;
const data = decodePcm16Base64(b64);
if (!data.length) return;
const i = chunks.length;
chunks.push(data);
starts.push(i === 0 ? 0 : starts[i - 1] + dur(i - 1) - fadeFor(i));
totalDuration = starts[i] + dur(i);
const when = anchor + (starts[i] - baseOffset);
if (i > 0 && when < ctx.currentTime - 0.01) {
// Underrun: playback drained the buffered chunks before this one
// arrived. Re-anchor so the new chunk starts now (the playhead sat at
// the buffered edge) instead of clipping its head.
baseOffset = starts[i];
anchor = ctx.currentTime + 0.02;
scheduleChunk(i, 0, false);
} else {
scheduleChunk(i);
}
session.update({
duration: totalDuration,
peaks: peaksFromChunkList(chunks),
});
},
appendPcm16Base64: (b64) => appendSamples(decodePcm16Base64(b64)),
/** Append one raw PCM16 binary frame (ArrayBuffer) — the `/ws/tts` shape. */
appendPcm16Bytes: (buf) => appendSamples(pcm16BytesToFloat32(buf)),
/** All chunks received: freeze the duration and retitle the bar. */
finalize: ({ label: finalLabel } = {}) => {
if (finished) return;
tests/test_admin_sessions.py
+13
View File
@@ -374,6 +374,19 @@ def test_ticket_is_scoped_to_normalized_path(store: AdminSessionStore):
assert store.consume_ws_ticket(ticket.token, "/ws/transcribe", MASTER) is None
def test_ticket_covers_every_first_party_ws_route(store: AdminSessionStore):
"""Backend allowlist must carry every route the UI mints tickets for
(authSession.ts ALLOWED_WS_PATHS); a missing route fails silently in the
UI — #1769's live dub preview over /ws/tts was the first casualty."""
session = store.issue(MASTER)
for path in ("/ws/events", "/ws/transcribe", "/ws/tts"):
wrong = "/ws/events" if path != "/ws/events" else "/ws/tts"
ticket = store.issue_ws_ticket(session.token, path, MASTER)
assert store.consume_ws_ticket(ticket.token, wrong, MASTER) is None
ticket = store.issue_ws_ticket(session.token, path, MASTER)
assert store.consume_ws_ticket(ticket.token, path, MASTER) is not None
@pytest.mark.parametrize(
"path",
[
tests/test_auth_session_api.py
+1 -1
View File
@@ -459,7 +459,7 @@ def test_legacy_cookie_migration_fails_without_exact_origin(origin):
@pytest.mark.parametrize(
"path",
["/ws/transcribe", "/v1/audio/transcriptions/stream"],
["/ws/events", "/ws/transcribe", "/ws/tts", "/v1/audio/transcriptions/stream"],
)
def test_session_can_mint_path_bound_ws_ticket(path):
client = _client()