upstream-archive/VoiceStudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: onboarding demos, opt-in bug reporting, error-docs deeplinks + issue triage (#133)

Browse Source 
* feat: onboarding demos, opt-in bug reporting, error-docs deeplinks + issue triage

Working-tree snapshot bundling several in-flight workstreams (v0.3.0):

- Onboarding/demo system: DemoPresetGrid, DictationDemo, DubbingDemo components
  + tests, render scripts (render_demos_omnivoice.py, build_demos.sh,
  build_dub_demo.sh), personalities preview URLs, alembic 0002 voice-profile
  demo fields.
- Opt-in bug reporting: ReportBugButton (prefilled GitHub-issue URL path).
- Error transparency UX: errorDocsMap deeplinks + BootstrapSplash/error wiring.
- Dub workspace: DubSegmentRow/Table, WaveformTimeline, dubSlice tweaks.
- Issue triage: .planning/issue-clusters/ (plan-01..05 root-cause masters,
  GH #128-#132).
- CLAUDE.md: hard rule — everything ships on v0.3.0, no version bumps.

KNOWN GAP (why this is a draft): the generated demo audio assets are NOT in
this tree, and backend/assets/samples/demo_voice.wav is deleted. onboarding.py
guards the missing file (skips seeding the demo profile with a warning), so no
crash — but first-run Launchpad will be empty and /demo_audio/ preview URLs
404 until assets are regenerated via scripts/build_demos.sh. Do not merge
before regenerating + committing the demo assets.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(dub): timing strategies — kill audio compression, add Concise + Stretch Video

Replaces the current audio time-compression default (atempo squeeze to fit
slot) that produced chipmunk/alien output on high-density target languages
like Bengali. Two new user-selectable modes; legacy behaviour kept behind
an explicit "Strict slot" choice.

New `DubRequest.timing_strategy` enum (default "concise"):
  - "concise"        Translator trims text to fit at natural rate; if it
                     still overflows, hard-trim at slot with a fade so we
                     never overlap the next speaker. Surface overflow_s
                     per segment so the user can shorten the text.
  - "stretch_video"  Audio plays at natural 1.0× rate. Backend computes a
                     per-segment new timeline; persists a video_stretch_plan
                     on the job. Mux step (dub_export) builds an ffmpeg
                     trim+setpts+concat filter graph that stretches each
                     segment's video portion to match the natural-rate dub
                     audio. Gaps/pre-roll/tail pass through at 1.0×.
                     Sub burn under stretch_video is skipped in one pass
                     (cues would drift).
  - "strict_slot"    Legacy atempo squeeze. Retained for back-compat.

Director rate-bias side-effect (seg_speed *= bias) now gated on strict_slot
only, so "urgent"/"slow" direction tokens keep their instruct effect in
the new modes without chipmunking.

Per-segment fit_status emitted in the SSE done event:
  {status: "fits" | "overflows" | "video_stretched", overflow_s?, stretch_ratio?}
DubSegmentRow's "Sync: 100%" badge (which was lying — sync_ratio was always
~1.0 because the TTS loop pre-trimmed to slot) is replaced with a truthful
"Fits / Overflows +Ns / Video 1.18×" label.

Frontend:
  - prefsSlice.timingStrategy (persisted, store v3→v4 with safe migrate).
  - DubTab footer Segmented control: "Concise · Stretch Video · Strict slot".
  - useDubWorkflow passes timing_strategy on /dub/generate; consumes fit_status.

Tests: tests/test_dub_timing_strategy.py — 13 cases covering schema
defaults/validation, _build_video_stretch_filter_graph (pre-roll, gap,
tail, empty-plan early return, post-subtitle chain-in), and
_video_stretch_plan_for guards. 30/30 existing dub tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(waveform): surface missing source as "Source media missing" instead of code-4 black box

When a project's underlying media file is gone (moved or deleted between
save and reload) the <video> element fires MediaError code 4 and the
companion audio fetch returns HTTP 404 — both were silently warned to
the console while the user stared at an unresponsive black panel and an
empty waveform.

- WaveformTimeline now flips loadError when the video element rejects
  code 3 (decode) or 4 (src not supported), and tracks `sourceMissing`
  separately so the error UI can name the actual problem.
- The audio decode fallback chain catches HTTP 404 specifically and
  treats it as source-missing instead of loading silent empty peaks —
  an empty waveform on a deleted source is more confusing than a clear
  "Re-upload the video to continue" message.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(tray): "Show OmniVoice" reloads when the webview is blank

When the dev Vite server restarts (or the main window is created before
the backend is ready), the webview load fails and the window is left
with `<body></body>` plus a "Could not connect to the server" console
error. Clicking "Show OmniVoice" from the tray menu just re-showed the
broken window — there was no recovery path short of quit+relaunch.

Now the show handler runs a tiny eval after `show()`/`set_focus()` that
calls `location.reload()` only when `document.body.childElementCount === 0`.
A healthy window doesn't blink (body is non-empty); a blank one
self-recovers as soon as the user clicks Show.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(#133): bug-report diagnostics field mapping + drop unused imports

Address PR #133 review:
- ReportBugButton: /system/info exposes `platform` + `device`, not
  `os`/`torch_device`/`gpu` — those reads silently dropped OS/GPU from every
  bug report. Map to the real fields (CodeRabbit). Also remove the dead
  `home` local in stripHome (CodeQL unused-variable).
- DictationDemo: drop unused `Loader` import (CodeQL unused-import).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Palash Debnath
2026-05-29 17:24:25 +05:30
committed by GitHub
co-authored by Claude Opus 4.8
parent 1cfda2f44e
commit 8b00dc1f4f
63 changed files with 4479 additions and 228 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+8
View File
@@ -84,6 +84,14 @@ omnivoice_data/
!tests/fixtures/omnivoice_data/voices/test-voice/
!tests/fixtures/omnivoice_data/voices/test-voice/profile.json
!tests/fixtures/omnivoice_data/voices/test-voice/sample.wav
# ─────────────────────────────────────────────────────────────────────────
# Bundled demo assets (synthetic, rendered by scripts/build_demos.sh).
# Ship with the installer so first-run users have working demos before any
# model weights download. *.wav above would hide these.
# ─────────────────────────────────────────────────────────────────────────
!backend/assets/samples/**/*.wav
!backend/assets/samples/*.wav
*.jsonl
demo_recording.webp
cloudflared
.planning/issue-clusters/01-windows-model-storage-hf-cache.md
+25
View File
@@ -0,0 +1,25 @@
# [plan-01] Windows Model Storage & HF Cache — symlink-safe, relocatable model dir
## Defect
On Windows the Hugging Face cache uses symlinks and a `scan_cache_dir` traversal that hit `WinError 448 "untrusted mount point"` under `%LOCALAPPDATA%\OmniVoice\hf_cache`, immediately followed by `Errno 22 Invalid argument`. The downloader retries 5× and gives up, leaving the app with no models. The model directory is also hard-pinned to the system partition with no way to relocate it to a larger/faster drive.
## Children
- #118 — MSI installer: `k2-fsa/OmniVoice` + `faster-whisper-large-v3` download fails, `WinError 448` → `Errno 22`, loops 5×, app unusable (with or without HF_TOKEN)
- #117 — same `scan_cache_dir` `WinError 448` → `Errno 22` via `bun run dev`
- #64 — (feature) allow choosing the model/weights download directory; the symlink-safe-cache fix naturally yields a configurable path
## Fix sequence
1. Disable HF symlink behavior on Windows (`HF_HUB_DISABLE_SYMLINKS=1` / `local_dir_use_symlinks=False`) so extraction stays inside user space.
2. Make `scan_cache_dir` failures non-fatal — fall back to a direct snapshot check instead of aborting the download.
3. Surface a configurable models directory in Settings, backed by `HF_HOME` / `HF_HUB_CACHE`, defaulting to a writable per-user path; honor it in the subprocess env.
4. Migration: detect an existing populated cache and reuse it (no re-download for current users).
## Test matrix
| OS | Cache location | Symlink support | Required behavior |
|---|---|---|---|
| Windows 11 | default %LOCALAPPDATA% | no | download + load succeeds, no WinError 448 |
| Windows 11 | user-chosen drive (e.g. D:\) | no | models land in chosen dir, loads on restart |
| macOS / Linux | default ~/.cache | yes | unchanged (no regression) |
## Out of scope
Missing-dependency runtime failures (→ plan-02). Restricted-network Python bootstrap (→ plan-03).
.planning/issue-clusters/02-windows-runtime-integrity.md
+25
View File
@@ -0,0 +1,25 @@
# [plan-02] Windows Runtime Integrity — complete venv + safe accel gating
## Defect
The packaged/bootstrapped Windows venv makes unsafe assumptions about its environment. Transitive runtime deps are missing (e.g. `setuptools`/`pkg_resources` that `ctranslate2`→`whisperx` import), and `torch.compile` is enabled where Triton has no Windows build — both fail only at *inference* time, surfacing as confusing errors (a fake "OOM", or a hard `ModuleNotFoundError` mid-transcription).
## Children
- #116 — `ModuleNotFoundError: No module named 'pkg_resources'` from `ctranslate2.__init__` via `whisperx` during chunk transcription (recurrence of the previously-closed #58)
- #65 — `torch.compile(mode="reduce-overhead")` requires Triton at runtime; Triton is unavailable on Windows → masked as TTS OOM. Suggested gate: `importlib.util.find_spec("triton")` before compile
- (note) the MSI "missing modules / Triton" half of #122 is the same family
## Fix sequence
1. Pin `setuptools` (provides `pkg_resources`) as an explicit runtime dep; verify it resolves in the bootstrapped venv.
2. Add a post-bootstrap venv integrity check that imports the critical chain (`ctranslate2`, `whisperx`, `torch`) and reports a clear actionable error if anything is missing.
3. Gate `torch.compile` on `find_spec("triton")`; fall back to eager mode with an INFO log on platforms without Triton.
4. Add an installer smoke test that imports the ASR/TTS critical path on Windows before the build is published.
## Test matrix
| OS | Accel | Required behavior |
|---|---|---|
| Windows + CUDA, no Triton | torch.compile path | skips compile, eager inference succeeds |
| Windows packaged venv | ASR transcribe | `pkg_resources`/`ctranslate2` import OK, segments produced |
| macOS MPS / Linux CUDA+Triton | both | unchanged |
## Out of scope
HF cache traversal (→ plan-01). The "no error in logs" transparency defect (→ plan-04).
.planning/issue-clusters/03-installer-bootstrap-network.md
+26
View File
@@ -0,0 +1,26 @@
# [plan-03] Installer Bootstrap Network Resilience — mirror cascade + system-Python fallback
## Defect
First-run bootstrap downloads a managed Python (python-build-standalone) directly from GitHub releases with no mirror fallback and short retry budget. On networks that block or can't resolve GitHub, `uv venv` dies with a DNS/tunnel error and the install is dead-on-arrival. (This is Capability 3 in the stack notes.)
## Children
- #60 — `uv venv failed`: `Failed to download .../python-build-standalone/...`, `dns error / Этот хост неизвестен` (host unknown). Workaround posted in PR #62.
- #57 — "Installation failed" (screenshot only) — tentatively a bootstrap failure; confirm against the image, else needs-info.
- #127 — Linux AppImage backend "exited (never started)", "Clean & Retry" loops, tried different mirrors with same result (Arch Linux, v0.2.7); adds a Linux AppImage test cell.
## Fix sequence
1. Set `UV_PYTHON_INSTALL_MIRROR` to a gh-proxy mirror cascade at bootstrap time.
2. Bump `UV_HTTP_TIMEOUT=120`, `UV_HTTP_CONNECT_TIMEOUT=30`, `UV_HTTP_RETRIES=5`.
3. Final fallback: `UV_PYTHON_PREFERENCE=only-system` when all mirrors fail and a compatible system Python ≥3.11 is present.
4. For PyPI access, document/optionally set `UV_DEFAULT_INDEX` (Tsinghua/Aliyun) for China; document VPN honestly for fully-blocked networks.
5. On total failure, the bootstrap surfaces the exact remediation (install python.org Python + the env vars) instead of a raw uv stack trace.
## Test matrix
| Network | System Python | Required behavior |
|---|---|---|
| GitHub blocked, mirror reachable | absent | installs via mirror |
| GitHub + mirrors blocked | py3.11 present | falls back to system Python |
| GitHub + mirrors blocked | absent | actionable error with remediation steps |
## Out of scope
Windows venv dependency completeness (→ plan-02).
.planning/issue-clusters/04-pipeline-error-transparency.md
+23
View File
@@ -0,0 +1,23 @@
# [plan-04] Pipeline Error Transparency — no more silent "unknown error"
## Defect
Pipeline failures surface in the UI as a generic string ("extract: unknown error") with **nothing written to the backend logs**, and in #122's case the instrumented code path is never even reached — meaning the exception is swallowed before/around the ingest stage. This is the single highest-leverage defect: until errors are visible, every other bug is un-triageable for both the user and the maintainer. Fixing it makes future reports self-describing and shrinks low-information reports like #63.
## Children
- #122 — "extract: unknown error" on dub of any media; backend terminal shows no related lines; debug prints in `ingest_pipeline` never fire; exhaustive repro (Win11, RTX 4080, ffmpeg verified working manually)
## Fix sequence
1. Find where the extract/ingest entrypoint swallows or short-circuits exceptions before reaching `ingest_pipeline`; ensure every failure path logs the real exception with stack + context.
2. Replace the generic UI "unknown error" with the underlying error class + a one-line "what to do" (ties into the error→docs deeplink work).
3. Add a structured failure event so the frontend always receives a non-empty reason.
4. Prevention net: low-info reports (#63-style) should be answerable because the app now emits a copyable diagnostic block.
## Test matrix
| Trigger | Required behavior |
|---|---|
| Extract fails (bad input) | UI shows specific cause; backend logs full traceback |
| YouTube ingest fails | same |
| WAV-only input fails | same (no ffmpeg path involved) |
## Out of scope
The actual root cause of any *specific* extract failure once it's visible may route to plan-01/02/03.
.planning/issue-clusters/05-voice-design-instruct-validator.md
+24
View File
@@ -0,0 +1,24 @@
# [plan-05] Voice Design Instruct Validator — reconcile preset builder with the whitelist
## Defect
The Voice Design prompt builder (personality presets, and any non-English language path) emits natural-language / free-text instruct items like "Speak as a calm, authoritative documentary narrator with measured pacing", but the server-side validator only accepts a fixed whitelist of tags (male, young adult, moderate pitch, whisper, japanese accent, …). Builder and validator are out of sync, so Synthesize fails with "Unsupported instruct items" or "conflicting instruct items within the same category". Same family as the recently-fixed personality-preset crash (#89).
## Children
- #115 — "Validation failed: Unsupported instruct items found"; presets (Narrator/Storyteller/Corporate) generate free-text prompts; also fires on any non-English language (Windows + macOS Apple Silicon)
- #114 — "Bad request - conflicting instruct items within the same category" on Design → personality settings → Synthesize (Windows, from source)
## Fix sequence
1. Audit the preset/prompt-builder output against the validator whitelist; enumerate which tokens are rejected and why ("conflicting within category" vs "unsupported").
2. Either map preset free-text to valid whitelist tags, or relax the validator to pass through free-text instruct for engines that accept it — pick per engine capability.
3. Fix the "conflicting within same category" logic so selecting one preset doesn't register as multiple mutually-exclusive selections.
4. Ensure language selection doesn't inject an instruct token the validator rejects.
## Test matrix
| Path | Engine | Required behavior |
|---|---|---|
| Each personality preset | OmniVoice/VoxCPM | Synthesize succeeds |
| Non-English language selected | each | no spurious validation failure |
| Manual single tag | each | unchanged |
## Out of scope
General pipeline error transparency (→ plan-04).
CLAUDE.md
+3 -1
View File
@@ -190,7 +190,9 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
<!-- GSD:conventions-start source:CONVENTIONS.md -->
## Conventions
Conventions not yet established. Will populate as patterns emerge during development.
**Versioning (hard rule):** Everything ships on `v0.3.0`. Never mention, suggest, or label anything with a version bump — no v0.4, no RCs, no "defer to next version", no future-version labels — unless the user explicitly asks to bump. Zero unprompted version chatter.
Other conventions not yet established. Will populate as patterns emerge during development.
<!-- GSD:conventions-end -->
<!-- GSD:architecture-start source:ARCHITECTURE.md -->
backend/api/routers/dub_export.py
+179 -14
View File
@@ -188,6 +188,82 @@ def _ffmpeg_filter_escape(path: str) -> str:
return path.replace("\\", "\\\\").replace(":", "\\:").replace("'", "\\'")
def _build_video_stretch_filter_graph(
plan: list[dict], orig_dur: float, video_input_idx: int = 0,
in_label: str | None = None,
) -> tuple[str, str]:
"""Build an ffmpeg filter_complex graph that stretches the source video
per-segment so each segment's visual duration matches a dub audio layout.
`plan` is a list of {orig_start, orig_end, new_start, new_end, stretch_ratio}
in original-time order (as persisted by dub_generate for stretch_video
mode). Gaps between plan entries — and the pre-roll / tail — are passed
through at 1.0× rate so silence and B-roll don't get squashed.
`in_label` overrides the input stream reference; e.g. pass "[vsub]" when
a subtitles filter has already written to that label. Defaults to
`[{video_input_idx}:v]` for direct source-stream consumption.
Returns (filter_graph, output_label). Output label is "[vstretched]" when
chunks were emitted, or the original input label when the plan was empty
(caller should fall back to stream-copy in that case).
"""
# Empty plan = no stretch — caller should stream-copy the video. Return
# early so we don't synthesise a degenerate "stretch whole video at 1.0×"
# graph that would force a needless re-encode.
if not plan:
return "", in_label or f"[{video_input_idx}:v]"
chunks: list[tuple[float, float, float]] = [] # (a, b, ratio)
cursor = 0.0
for entry in plan:
a = float(entry["orig_start"])
b = float(entry["orig_end"])
if a > cursor + 1e-3:
chunks.append((cursor, a, 1.0)) # gap or pre-roll at native rate
ratio = float(entry["stretch_ratio"])
if b > a:
chunks.append((a, b, ratio))
cursor = max(cursor, b)
if orig_dur > cursor + 1e-3:
chunks.append((cursor, orig_dur, 1.0)) # tail at native rate
chunks = [(a, b, r) for (a, b, r) in chunks if b > a]
if not chunks:
return "", in_label or f"[{video_input_idx}:v]"
src = in_label or f"[{video_input_idx}:v]"
parts: list[str] = []
labels: list[str] = []
# `split` lets us tap the same source stream once per chunk without re-
# decoding. setpts={ratio}*PTS slows down (ratio > 1) or speeds up
# (ratio < 1) each chunk; PTS-STARTPTS first to normalise the timestamp
# base after the trim.
split_labels = [f"[vsplit{idx}]" for idx in range(len(chunks))]
parts.append(f"{src}split={len(chunks)}{''.join(split_labels)}")
for idx, ((a, b, ratio), split_lbl) in enumerate(zip(chunks, split_labels)):
out_label = f"[vstr{idx}]"
labels.append(out_label)
parts.append(
f"{split_lbl}trim=start={a:.4f}:end={b:.4f},"
f"setpts=PTS-STARTPTS,setpts={ratio:.6f}*PTS{out_label}"
)
parts.append("".join(labels) + f"concat=n={len(chunks)}:v=1:a=0[vstretched]")
return ";".join(parts), "[vstretched]"
def _video_stretch_plan_for(job: dict, lang_code: str) -> dict | None:
"""Return the persisted stretch plan + total durations for `lang_code`,
or None if this job didn't use stretch_video mode (or no plan exists).
"""
if (job.get("timing_strategy") or "").lower() != "stretch_video":
return None
plans = job.get("video_stretch_plans") or {}
entry = plans.get(lang_code)
if not entry or not entry.get("plan"):
return None
return entry
@router.get("/dub/download/{job_id}")
@router.get("/dub/download/{job_id}/{filename}")
async def dub_download(
@@ -225,6 +301,26 @@ async def dub_download(
output_path = os.path.join(exports_dir, f"dubbed_video_{stamp}.mp4")
ffmpeg = find_ffmpeg()
# Determine whether this export should drive video through a per-segment
# stretch graph (Mode B). Stretch is keyed off the default_track's plan
# because the video can only physically follow one timeline at a time.
# If multiple dub tracks are included and they were generated under
# stretch_video, only the default_track is visually in sync — other
# tracks share the same (stretched) video. Single-track export is the
# supported common case.
stretch_entry = _video_stretch_plan_for(job, default_track) if default_track and default_track != "original" else None
# Subtitle burn under stretch_video would render cues at the original
# timestamps onto a re-timed video — they'd drift. Skip the burn pass
# in that combo and log; the user can still export the SRT/VTT
# separately and the new-layout timing lives there.
if stretch_entry and burn_subs:
logger.warning(
"stretch_video + burn_subs is not supported in one pass; "
"skipping subtitle burn for job %s. Export the SRT/VTT separately.",
job_id,
)
burn_subs = False
sub_path = _write_burn_srt(job, exports_dir, stamp, dual) if burn_subs else None
cmd = [ffmpeg, "-i", video_path]
@@ -247,8 +343,17 @@ async def dub_download(
video_map = "0:v:0"
if sub_path:
esc = _ffmpeg_filter_escape(sub_path)
filter_parts.append(f"[0:v]subtitles='{esc}'[vout]")
video_map = "[vout]"
filter_parts.append(f"[0:v]subtitles='{esc}'[vsub]")
video_map = "[vsub]"
if stretch_entry:
orig_dur = float(stretch_entry.get("orig_duration") or job.get("duration") or 0.0)
graph, vlabel = _build_video_stretch_filter_graph(
stretch_entry["plan"], orig_dur, video_input_idx=0,
in_label=video_map if video_map != "0:v:0" else None,
)
if graph:
filter_parts.append(graph)
video_map = vlabel
cmd += ["-map", video_map]
if include_original:
@@ -268,8 +373,10 @@ async def dub_download(
if filter_parts:
cmd += ["-filter_complex", ";".join(filter_parts)]
# Burning subs forces a video re-encode; stream-copy otherwise to keep mux cheap.
if sub_path:
# Burning subs or per-segment video stretch both force a real video
# re-encode; stream-copy is only viable when nothing touches the video
# filter chain.
if sub_path or stretch_entry:
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
else:
cmd += ["-c:v", "copy"]
@@ -302,7 +409,14 @@ async def dub_download(
break
cmd += [f"-disposition:a:{target_idx}", "default"]
cmd += ["-shortest", output_path, "-y"]
# In stretch_video mode the video and audio durations should match
# within sub-frame precision, but `-shortest` can still cut off the
# trailing frame; let ffmpeg keep both streams. Otherwise keep the
# legacy `-shortest` so a slightly-overrunning track doesn't extend
# the mux past the video.
if not stretch_entry:
cmd += ["-shortest"]
cmd += [output_path, "-y"]
try:
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0)
@@ -335,14 +449,36 @@ async def dub_download(
)
_MEDIA_TYPES = {
".mp4": "video/mp4",
".m4v": "video/mp4",
".mov": "video/quicktime",
".webm": "video/webm",
".mkv": "video/x-matroska",
".m4a": "audio/mp4",
".mp3": "audio/mpeg",
".wav": "audio/wav",
".flac": "audio/flac",
".ogg": "audio/ogg",
}
@router.get("/dub/media/{job_id}")
async def dub_get_media(job_id: str):
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
if not os.path.exists(job["video_path"]):
video_path = job["video_path"]
if not os.path.exists(video_path):
raise HTTPException(status_code=404, detail="Media file not found")
return FileResponse(job["video_path"])
# Pass an explicit media_type. Without this Starlette falls back to
# mimetypes.guess_type, which on some platforms returns the wrong
# MIME (e.g. "application/octet-stream" for .mkv), and the Tauri
# WebView then refuses to render the <video> element — leaving a
# silent black box. Default to video/mp4 because the ingest pipeline
# remuxes URL downloads to mp4 (dub_pipeline.yt_download_sync).
ext = os.path.splitext(video_path)[1].lower()
return FileResponse(video_path, media_type=_MEDIA_TYPES.get(ext, "video/mp4"))
@router.get("/dub/preview-video/{job_id}")
async def dub_preview_video(
@@ -389,6 +525,7 @@ async def dub_preview_video(
if not cache_ok:
ffmpeg = find_ffmpeg()
stretch_entry = _video_stretch_plan_for(job, lang)
cmd = [ffmpeg, "-i", video_path]
input_idx = 1
if preserve_bg and has_bg:
@@ -400,16 +537,44 @@ async def dub_preview_video(
cmd += ["-i", track_path]
track_idx = input_idx
cmd += ["-map", "0:v:0"]
# Build filter graph. In stretch_video mode we splice the source
# video into per-segment chunks, setpts each to match the dub audio
# layout, and concat them — so audio plays at natural rate and the
# visuals follow. Otherwise we stream-copy video for speed.
filter_parts: list[str] = []
video_map = "0:v:0"
if stretch_entry:
orig_dur = float(stretch_entry.get("orig_duration") or job.get("duration") or 0.0)
graph, vlabel = _build_video_stretch_filter_graph(
stretch_entry["plan"], orig_dur, video_input_idx=0,
)
if graph:
filter_parts.append(graph)
video_map = vlabel
if bg_idx is not None:
cmd += [
"-filter_complex",
f"[{bg_idx}:a][{track_idx}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2[aout]",
"-map", "[aout]",
]
filter_parts.append(
f"[{bg_idx}:a][{track_idx}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2[aout]"
)
cmd += ["-map", video_map]
if bg_idx is not None:
cmd += ["-map", "[aout]"]
else:
cmd += ["-map", f"{track_idx}:a:0"]
cmd += ["-c:v", "copy", "-c:a", "aac", "-b:a", "192k", "-shortest", preview_path, "-y"]
if filter_parts:
cmd += ["-filter_complex", ";".join(filter_parts)]
# Stretch path needs a real encode; stream-copy otherwise.
if stretch_entry:
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
else:
cmd += ["-c:v", "copy"]
cmd += ["-c:a", "aac", "-b:a", "192k"]
# `-shortest` would cut the stretched video at the (slightly different)
# audio length and lose the trailing frame; only use it on the copy path.
if not stretch_entry:
cmd += ["-shortest"]
cmd += [preview_path, "-y"]
try:
rc, _, stderr = await run_ffmpeg(cmd, timeout=900.0)
backend/api/routers/dub_generate.py
+288 -38
View File
@@ -1,8 +1,10 @@
import os
import json
import logging
import subprocess
import time
import asyncio
import numpy as np
import torch
import torchaudio
from fastapi import APIRouter, HTTPException
@@ -15,6 +17,7 @@ from schemas.requests import DubRequest
from services.model_manager import get_model, _gpu_pool
from services.audio_dsp import apply_mastering, normalize_audio, apply_effects_chain, get_effect_chain
from services.audio_io import atomic_save_wav, _safe_torchaudio_save
from services.ffmpeg_utils import find_ffmpeg
from services.rvc import apply_rvc, is_enabled as rvc_is_enabled
from services.incremental import segment_fingerprint
from services.watermark import embed_watermark
@@ -22,6 +25,88 @@ from api.routers.dub_core import _get_job, _save_job
logger = logging.getLogger("omnivoice.dub")
# Maximum compression ratio we'll attempt with pitch-preserving stretch
# before declaring "no way to fit cleanly" and falling back. atempo
# remains intelligible up to ~1.5× then introduces audible WSOLA
# artefacts; above ~1.8× speech becomes a fast garbled stream that no
# DSP can rescue. The contributing-factor pipeline (CPS-aware slot-fit
# in services/speech_rate.py, gap absorption below) keeps us under this
# in practice — this is only a guard rail.
MAX_STRETCH_RATIO = 1.8
# How far a too-long segment is allowed to bleed into the silent gap
# before the next segment. Buys headroom on languages with higher
# information density (Bengali, Hindi, Arabic…) without the audio
# colliding with the next speaker's onset.
GAP_OVERFLOW_MAX_S = 0.25
GAP_OVERFLOW_BUFFER_S = 0.05
def _atempo_chain(ratio: float) -> str:
"""Build an `atempo=…,atempo=…` filter chain for arbitrary ratios.
ffmpeg's atempo filter is limited to [0.5, 2.0] per stage. Chaining
multiple stages multiplies the effective ratio while keeping each
individual stage inside the well-behaved range. Pitch is preserved
(WSOLA-style time-domain stretching). ratio > 1 speeds up, < 1
slows down.
"""
stages: list[str] = []
remaining = ratio
while remaining > 2.0:
stages.append("atempo=2.0")
remaining /= 2.0
while remaining < 0.5:
stages.append("atempo=0.5")
remaining /= 0.5
stages.append(f"atempo={remaining:.6f}")
return ",".join(stages)
def _pitch_preserving_stretch(
wav: torch.Tensor, target_samples: int, sr: int,
) -> torch.Tensor:
"""Time-stretch a (1, samples) tensor to `target_samples` while
preserving pitch, by piping the audio through `ffmpeg atempo`.
Returns a (1, target_samples) tensor on the same device as input.
Raises RuntimeError when ffmpeg fails — callers should fall back to
naive linear interpolation, accepting the pitch shift, to ensure the
output isn't silent.
"""
wl = int(wav.shape[-1])
if target_samples <= 0 or wl == target_samples:
return wav
ratio = wl / target_samples
filter_str = _atempo_chain(ratio)
# Mono float32 via stdin → ffmpeg → stdout. One subprocess per
# segment is ~50-100 ms overhead, dwarfed by TTS generation.
arr = wav.detach().cpu().to(torch.float32).numpy().reshape(-1).astype(np.float32, copy=False)
proc = subprocess.run(
[
find_ffmpeg(), "-hide_banner", "-loglevel", "error", "-y",
"-f", "f32le", "-ar", str(sr), "-ac", "1", "-i", "pipe:0",
"-af", filter_str,
"-f", "f32le", "-ar", str(sr), "-ac", "1", "pipe:1",
],
input=arr.tobytes(),
capture_output=True,
)
if proc.returncode != 0 or not proc.stdout:
raise RuntimeError(
(proc.stderr.decode(errors="replace") or "atempo failed")[:200]
)
out_arr = np.frombuffer(proc.stdout, dtype=np.float32)
# atempo rarely lands exactly on the integer sample count, so
# pad/trim to the requested slot length.
if len(out_arr) < target_samples:
pad = np.zeros(target_samples - len(out_arr), dtype=np.float32)
out_arr = np.concatenate([out_arr, pad])
elif len(out_arr) > target_samples:
out_arr = out_arr[:target_samples]
return torch.from_numpy(out_arr.copy()).unsqueeze(0).to(wav.device)
router = APIRouter()
@router.post("/dub/generate/{job_id}")
@@ -252,7 +337,16 @@ async def dub_generate(job_id: str, req: DubRequest):
)
bias = d.rate_bias()
if bias and abs(bias - 1.0) > 0.01:
seg_speed = (seg_speed or 1.0) * bias
# Speed-bias from a Direction only multiplies seg_speed
# in strict_slot mode, which is the legacy path that
# compresses audio at synthesis time to fit the slot.
# In concise / stretch_video modes we preserve natural
# rate so the user gets the "urgent" or "slow" voice
# the director asked for without the chipmunk side-
# effect that overshooting the slot would otherwise
# cause.
if (req.timing_strategy or "concise") == "strict_slot":
seg_speed = (seg_speed or 1.0) * bias
except Exception as e:
logger.debug("direction parse skipped for %s: %s", getattr(seg, 'id', '?'), e)
@@ -266,9 +360,18 @@ async def dub_generate(job_id: str, req: DubRequest):
_num_step = 8 if req.preview else req.num_step
_t_tts_0 = time.perf_counter()
seg_effect_preset = getattr(seg, "effect_preset", None) or "broadcast"
# In concise / stretch_video modes we pass dur_s=None so the
# TTS model speaks at its natural rate for this text length —
# the whole point of the new timing strategies is to never
# squeeze the speech to fit. strict_slot keeps the legacy
# behaviour where dur_s is the slot hint.
_strategy = (req.timing_strategy or "concise").lower()
_dur_for_tts = seg_duration if _strategy == "strict_slot" else None
audio_tensor = await loop.run_in_executor(
_gpu_pool, _gen,
seg.text, seg_lang, seg_instruct, seg_duration,
seg.text, seg_lang, seg_instruct, _dur_for_tts,
_num_step, req.guidance_scale, seg_speed, seg_profile, seg_effect_preset,
)
_t_tts += time.perf_counter() - _t_tts_0
@@ -277,19 +380,27 @@ async def dub_generate(job_id: str, req: DubRequest):
if task_manager.is_cancelled(task_id):
yield f"data: {json.dumps({'type': 'cancelled', 'segments_processed': i + 1})}\n\n"
return
target_samples = int(seg_duration * _model.sampling_rate)
current_samples = audio_tensor.shape[-1]
if target_samples > current_samples:
pad_amount = target_samples - current_samples
audio_tensor = torch.nn.functional.pad(audio_tensor, (0, pad_amount))
elif current_samples > target_samples:
audio_tensor = audio_tensor[..., :target_samples]
if _strategy == "strict_slot":
# Legacy: pad short audio + trim long audio so the mix
# loop receives slot-sized buffers. The atempo squeeze
# in the mix loop never fires here because we already
# forced size = target_samples.
if target_samples > current_samples:
pad_amount = target_samples - current_samples
audio_tensor = torch.nn.functional.pad(audio_tensor, (0, pad_amount))
elif current_samples > target_samples:
audio_tensor = audio_tensor[..., :target_samples]
# concise / stretch_video: keep audio at its natural length.
# The mix loop decides per-mode whether to trim, slip, or
# stretch the video to accommodate it.
generated_dur = audio_tensor.shape[-1] / _model.sampling_rate
sync_ratio = round(generated_dur / max(seg_duration, 0.01), 3)
sync_scores.append(sync_ratio)
# Build the fingerprint now (cheap) but defer the disk write
@@ -364,41 +475,159 @@ async def dub_generate(job_id: str, req: DubRequest):
_t_diskw = time.perf_counter() - _t_diskw_0
sr = _model.sampling_rate
total_samples = int(job["duration"] * sr)
strategy = (req.timing_strategy or "concise").lower()
slot_fit = (req.slot_fit or "time_stretch").lower()
overflow_budget_s = max(0.0, float(req.overflow_budget_s or 0.0))
# Per-segment fit_status emitted alongside the legacy sync_scores so
# the UI can replace the lying "Sync: 100%" badge with a truthful
# "Fits / Overflows +0.4s / Slipped 0.2s / Video stretched 1.18×".
fit_status: list[dict] = []
# Mode B layout: when stretch_video is on, compute a new timeline
# where each segment's slot equals the natural-rate dub audio length;
# gaps stay at 1.0×. Persisted on the job so dub_export.py can build
# the matching per-segment setpts filter chain on the source video.
new_layout: list[tuple[float, float]] = []
video_stretch_plan: list[dict] = []
orig_total_dur = float(job.get("duration") or 0.0)
if strategy == "stretch_video":
cursor = 0.0
for i, (orig_start, orig_end, wav, _) in enumerate(all_segment_wavs):
wl_i = wav.shape[-1]
natural_dur = (wl_i / sr) if wl_i > 0 else max(0.0, orig_end - orig_start)
if i == 0:
# Preserve the pre-roll (silence before the first seg).
cursor = orig_start
else:
prev_orig_end = all_segment_wavs[i - 1][1]
gap = max(0.0, orig_start - prev_orig_end)
cursor += gap
new_start = cursor
new_end = cursor + natural_dur
new_layout.append((new_start, new_end))
orig_dur = max(1e-3, orig_end - orig_start)
video_stretch_plan.append({
"orig_start": round(orig_start, 4),
"orig_end": round(orig_end, 4),
"new_start": round(new_start, 4),
"new_end": round(new_end, 4),
"stretch_ratio": round(natural_dur / orig_dur, 4),
})
cursor = new_end
# Preserve the trailing tail (anything after the last seg in the
# original video) at 1.0× rate.
if all_segment_wavs:
last_orig_end = all_segment_wavs[-1][1]
cursor += max(0.0, orig_total_dur - last_orig_end)
new_total_dur = max(cursor, orig_total_dur)
total_samples = int(new_total_dur * sr)
else:
total_samples = int(orig_total_dur * sr)
full_audio = torch.zeros(1, total_samples)
slot_fit = (req.slot_fit or "time_stretch").lower()
for i, (start, end, wav, _) in enumerate(all_segment_wavs):
s = int(start * sr)
seg_ref = req.segments[i] if i < len(req.segments) else None
seg_gain = getattr(seg_ref, "gain", None) if seg_ref is not None else None
seg_gain = seg_gain if seg_gain is not None else 1.0
seg_gain = max(0.0, min(2.0, seg_gain))
adjusted = wav * seg_gain
# Slot-fit: keep each seg from bleeding into the next. "time_stretch"
# resamples to the slot via linear interpolation (slight pitch lift
# on compression, negligible at ≤1.15×, audible at ≥1.3×). "trim"
# hard-clips + fade-out. "off" is the legacy overlap behaviour.
slot_samples = int(max(0.0, (end - start)) * sr)
wl = adjusted.shape[-1]
if slot_fit != "off" and slot_samples > 0 and wl > slot_samples:
if slot_fit == "time_stretch":
try:
# Shape: (1, wl) → interpolate(..., size=slot_samples) → (1, slot_samples)
adjusted = torch.nn.functional.interpolate(
adjusted.unsqueeze(0),
size=slot_samples,
mode='linear',
align_corners=False,
).squeeze(0)
except Exception as e:
logger.warning("time_stretch failed for seg %d, falling back to trim: %s", i, e)
adjusted = adjusted[..., :slot_samples]
else: # "trim"
adjusted = adjusted[..., :slot_samples]
wl = adjusted.shape[-1]
natural_dur = wl / sr if wl > 0 else 0.0
orig_dur = max(0.0, end - start)
if strategy == "stretch_video":
# Mode B: audio at natural rate, placed on the stretched
# timeline. No trim, no atempo. dub_export handles the video.
new_start, _new_end = new_layout[i]
place_at = new_start
fit_status.append({
"status": "video_stretched",
"stretch_ratio": round(natural_dur / max(orig_dur, 1e-3), 3),
})
elif strategy == "concise":
# Mode A: never compress. Allow the audio to extend into the
# silent gap before the next seg (existing heuristic) plus
# any extra `overflow_budget_s`. Beyond that, hard-trim with
# a short fade so we never overlap the next speaker.
place_at = start
effective_end = end
if i + 1 < len(all_segment_wavs):
next_start = all_segment_wavs[i + 1][0]
gap = next_start - end
if gap > GAP_OVERFLOW_BUFFER_S:
effective_end = end + min(
gap - GAP_OVERFLOW_BUFFER_S, GAP_OVERFLOW_MAX_S,
)
effective_end += overflow_budget_s
slot_samples_eff = int(max(0.0, (effective_end - start)) * sr)
if slot_samples_eff > 0 and wl > slot_samples_eff:
overflow_s = (wl - slot_samples_eff) / sr
adjusted = adjusted[..., :slot_samples_eff]
wl = adjusted.shape[-1]
fit_status.append({
"status": "overflows",
"overflow_s": round(overflow_s, 3),
})
else:
fit_status.append({"status": "fits"})
else:
# strict_slot (legacy): preserve the previous atempo / trim /
# off semantics so existing callers and back-compat tests
# keep passing.
place_at = start
effective_end = end
if i + 1 < len(all_segment_wavs):
next_start = all_segment_wavs[i + 1][0]
gap = next_start - end
if gap > GAP_OVERFLOW_BUFFER_S:
effective_end = end + min(
gap - GAP_OVERFLOW_BUFFER_S, GAP_OVERFLOW_MAX_S,
)
slot_samples = int(max(0.0, (effective_end - start)) * sr)
if slot_fit != "off" and slot_samples > 0 and wl > slot_samples:
if slot_fit == "time_stretch":
ratio = wl / slot_samples
capped_ratio = min(ratio, MAX_STRETCH_RATIO)
capped_target = int(wl / capped_ratio)
try:
adjusted = _pitch_preserving_stretch(
adjusted, capped_target, sr,
)
if adjusted.shape[-1] > slot_samples:
adjusted = adjusted[..., :slot_samples]
if ratio > MAX_STRETCH_RATIO:
logger.info(
"seg %d compression %.2f× exceeded cap; "
"stretched to %.2f×, tail trimmed",
i, ratio, capped_ratio,
)
except Exception as e:
logger.warning(
"atempo stretch failed for seg %d (%.2f×), "
"falling back to linear interp: %s",
i, ratio, e,
)
adjusted = torch.nn.functional.interpolate(
adjusted.unsqueeze(0),
size=slot_samples,
mode='linear',
align_corners=False,
).squeeze(0)
else: # "trim"
adjusted = adjusted[..., :slot_samples]
wl = adjusted.shape[-1]
fit_status.append({
"status": "fits",
"compression_applied": (slot_fit == "time_stretch"
and wl != int(natural_dur * sr)),
})
# Common: short fades to avoid pops, then mix into full_audio.
fade_ms = 15
fade_samples = int((fade_ms / 1000.0) * sr)
if wl > fade_samples * 2:
@@ -407,8 +636,10 @@ async def dub_generate(job_id: str, req: DubRequest):
adjusted[0, :fade_samples] *= ramp_up
adjusted[0, -fade_samples:] *= ramp_down
s = int(place_at * sr)
e = min(s + wl, total_samples)
full_audio[:, s:e] += adjusted[:, :e - s]
if s < total_samples:
full_audio[:, s:e] += adjusted[:, :e - s]
lang_code = req.language_code or "und"
track_path = os.path.join(DUB_DIR, job_id, f"dubbed_{lang_code}.wav")
@@ -418,14 +649,33 @@ async def dub_generate(job_id: str, req: DubRequest):
atomic_save_wav(track_path, full_audio, sr)
_t_save = time.perf_counter() - _t_save_0
_t_mix = _t_save_0 - _t_loop_end
# Per-track metadata. For stretch_video, the dub wav is at the new
# (longer) timeline, so we record its actual duration here too — the
# mux step needs this to know whether to use the original video as-is
# or stretch it per the plan.
track_dur = full_audio.shape[-1] / sr if full_audio.shape[-1] > 0 else 0.0
job["dubbed_tracks"][lang_code] = {
"path": track_path,
"language": req.language,
"language_code": lang_code,
"duration": round(track_dur, 4),
"timing_strategy": strategy,
}
# Persist the timing strategy + (for Mode B) the per-segment stretch
# plan so dub_export can build the matching video pipeline at mux
# time. Plans are keyed by language code because each language gets
# its own dub track with its own natural-rate audio layout.
job["language"] = req.language
job["language_code"] = lang_code
job["timing_strategy"] = strategy
if strategy == "stretch_video":
stretch_plans = job.setdefault("video_stretch_plans", {})
stretch_plans[lang_code] = {
"plan": video_stretch_plan,
"total_duration": round(track_dur, 4),
"orig_duration": round(orig_total_dur, 4),
}
_save_job(job_id, job)
_t_total = time.perf_counter() - _t_start
@@ -435,7 +685,7 @@ async def dub_generate(job_id: str, req: DubRequest):
f" regen={len(regen_only)}" if regen_only is not None else "",
)
yield f"data: {json.dumps({'type': 'done', 'segments_processed': total, 'language_code': lang_code, 'tracks': list(job['dubbed_tracks'].keys()), 'sync_scores': sync_scores, 'seg_hashes': job.get('seg_hashes', {}), 'seg_num_step': job.get('seg_num_step', {})})}\n\n"
yield f"data: {json.dumps({'type': 'done', 'segments_processed': total, 'language_code': lang_code, 'tracks': list(job['dubbed_tracks'].keys()), 'sync_scores': sync_scores, 'fit_status': fit_status, 'timing_strategy': strategy, 'seg_hashes': job.get('seg_hashes', {}), 'seg_num_step': job.get('seg_num_step', {})})}\n\n"
task_id = f"dub_{job_id}_{int(time.time())}"
await task_manager.add_task(task_id, "dub_generate", _stream, task_id)
backend/api/routers/dub_translate.py
+21
View File
@@ -418,6 +418,27 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
Otherwise return Fast-mode shape unchanged.
"""
quality = (getattr(req, "quality", None) or "fast").lower()
# Stamp the predicted rate_ratio on every translated row that has a
# known slot. Works for Fast mode too — no LLM needed; just the CPS
# table from services/speech_rate. The UI's `seg-rate-badge` reads
# this value and shows users which segments will compress hard at
# generation time, so they can edit text or pick Cinematic quality.
try:
from services.speech_rate import rate_ratio as _predict_rate_ratio
for row in translated:
seg_ref = next(
(s for s in req.segments if str(s.id) == str(row["id"])),
None,
)
slot = getattr(seg_ref, "slot_seconds", None) if seg_ref else None
text = (row.get("text") or "").strip()
if slot and text and not row.get("error"):
row["rate_ratio"] = round(
_predict_rate_ratio(text, float(slot), req.target_lang), 3,
)
except Exception as e:
logger.debug("non-LLM rate_ratio prediction skipped: %s", e)
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang, "quality_used": "fast"}
if quality != "cinematic":
backend/api/routers/system.py
+24
View File
@@ -561,10 +561,34 @@ async def set_env_var(body: dict):
if value:
os.environ[key] = value
logger.info("Set environment variable: %s (length=%d)", key, len(value))
# Capability 1 / issue #35: HF_TOKEN persists across restarts via
# huggingface_hub.login() — writes the token to $HF_HOME/token so
# the next process pickup doesn't need an env var. add_to_git_credential
# stays False; we don't want to spew tokens into the user's git config.
if key == "HF_TOKEN":
try:
from huggingface_hub import login as _hf_login
_hf_login(token=value, add_to_git_credential=False)
logger.info("HF token persisted to $HF_HOME/token via login()")
except Exception as e:
# Non-fatal — the runtime env var is still set, so the
# current process will still see the token. We just lose
# persistence across restarts.
logger.warning("Could not persist HF token to disk: %s", e)
else:
os.environ.pop(key, None)
logger.info("Cleared environment variable: %s", key)
# Mirror the persistence on clear — wipe the saved token file too.
if key == "HF_TOKEN":
try:
from huggingface_hub import logout as _hf_logout
_hf_logout()
logger.info("HF token cleared from $HF_HOME/token via logout()")
except Exception as e:
logger.warning("Could not clear HF token file: %s", e)
return {"key": key, "set": bool(value)}
backend/assets/samples/demo_voice.wav
BIN
View File
Binary file not shown.
backend/core/db.py
+2
View File
@@ -47,6 +47,8 @@ _BASE_SCHEMA = """
seed INTEGER DEFAULT NULL,
is_locked INTEGER DEFAULT 0,
personality TEXT DEFAULT '',
description TEXT DEFAULT '',
is_demo INTEGER DEFAULT 0,
created_at REAL
);
CREATE TABLE IF NOT EXISTS generation_history (
backend/core/onboarding.py
+37 -4
View File
@@ -19,14 +19,44 @@ _DEMO_AUDIO = os.path.join(
)
DEMO_PROFILE_ID = "demo0001"
DEMO_PROFILE_NAME = "OmniVoice Demo"
DEMO_REF_TEXT = "Welcome to OmniVoice Studio. Clone any voice, design new ones, or dub videos into hundreds of languages."
DEMO_PROFILE_NAME = "OmniVoice Demo Voice"
# Must match the actual spoken content of backend/assets/samples/demo_voice.wav.
# Regenerated by scripts/build_demos.sh — update both files in lockstep.
DEMO_REF_TEXT = (
"Hi, I'm the OmniVoice demo voice. Everything you hear me say from now on "
"was synthesized on your own machine. No cloud, no account, just you and "
"the model."
)
_DEMO_DESCRIPTION = (
"A neutral reference voice bundled with OmniVoice. Clone it to hear how "
"the engine sounds on your machine, then replace it with your own "
"recording when you're ready."
)
def _backfill_demo_metadata(conn):
"""v0.2.x → v0.3.0 upgrade: a user who already had demo0001 seeded
before the alembic migration ran will have description='' and
is_demo=0 on that row. Backfill on every boot — cheap, idempotent."""
try:
conn.execute(
"UPDATE voice_profiles SET description=?, is_demo=1, ref_text=? "
"WHERE id=? AND (is_demo=0 OR description='' OR ref_text!=?)",
(_DEMO_DESCRIPTION, DEMO_REF_TEXT, DEMO_PROFILE_ID, DEMO_REF_TEXT),
)
conn.commit()
except Exception as e:
# Columns may not exist yet if alembic hasn't run — non-fatal.
logger.debug("Demo backfill skipped: %s", e)
def seed_sample_project():
"""Create the demo voice profile if no profiles exist yet."""
conn = get_db()
try:
_backfill_demo_metadata(conn)
count = conn.execute("SELECT COUNT(*) FROM voice_profiles").fetchone()[0]
if count > 0:
return # Not first run — skip
@@ -43,8 +73,9 @@ def seed_sample_project():
conn.execute(
"INSERT OR IGNORE INTO voice_profiles "
"(id, name, ref_audio_path, ref_text, instruct, language, personality, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
"(id, name, ref_audio_path, ref_text, instruct, language, "
" personality, description, is_demo, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
(
DEMO_PROFILE_ID,
DEMO_PROFILE_NAME,
@@ -53,6 +84,8 @@ def seed_sample_project():
"",
"English",
"",
_DEMO_DESCRIPTION,
1,
time.time(),
),
)
backend/core/personalities.py
+140
View File
@@ -74,6 +74,146 @@ PERSONALITIES = [
"description": "High energy and enthusiasm like a podcast host",
"icon": "⚡",
},
# ─────────────────────────────────────────────────────────────────────
# Demo presets — render as a 7-card grid in the empty Design tab.
# Each preset bundles taxonomy `attrs` (drives the CATEGORIES sliders),
# a sample `script` (pre-fills the textarea), and a `preview_url`
# pointing at a pre-rendered WAV under /demo_audio.
#
# `is_demo: True` is the marker the frontend filters on; the legacy 6
# entries above stay as chips, the entries below render as full cards.
# WAVs are generated by scripts/build_demos.sh — keep slugs in sync.
# ─────────────────────────────────────────────────────────────────────
{
"id": "audiobook_uk_narrator",
"name": "The Librarian",
"icon": "📚",
"description": "Warm UK audiobook narrator — measured, atmospheric.",
"instruct": "female, middle-aged, low pitch, british accent",
"attrs": {
"Gender": "female", "Age": "middle-aged", "Pitch": "low pitch",
"Style": "Auto", "EnglishAccent": "british accent",
"ChineseDialect": "Auto",
},
"script": (
"The clock tower struck thirteen, and for the first time in her "
"life, Eleanor wondered if she had been counting wrong all along."
),
"preview_url": "/demo_audio/voice_design/demo_voice_design_audiobook_uk_narrator.wav",
"language": "English",
"is_demo": True,
},
{
"id": "us_news_anchor",
"name": "The Anchor",
"icon": "📺",
"description": "Clear American broadcaster — primetime evening news.",
"instruct": "male, middle-aged, moderate pitch, american accent",
"attrs": {
"Gender": "male", "Age": "middle-aged", "Pitch": "moderate pitch",
"Style": "Auto", "EnglishAccent": "american accent",
"ChineseDialect": "Auto",
},
"script": (
"Good evening. Topping our broadcast tonight: scientists at the "
"coastal observatory have confirmed the signal is, in fact, repeating."
),
"preview_url": "/demo_audio/voice_design/demo_voice_design_us_news_anchor.wav",
"language": "English",
"is_demo": True,
},
{
"id": "indian_support_agent",
"name": "The Helpdesk",
"icon": "🎧",
"description": "Patient Indian-English customer-service voice.",
"instruct": "female, young adult, moderate pitch, indian accent",
"attrs": {
"Gender": "female", "Age": "young adult", "Pitch": "moderate pitch",
"Style": "Auto", "EnglishAccent": "indian accent",
"ChineseDialect": "Auto",
},
"script": (
"Thank you for calling OmniVoice support. I can see your account "
"here. Let's get this sorted out together."
),
"preview_url": "/demo_audio/voice_design/demo_voice_design_indian_support_agent.wav",
"language": "English",
"is_demo": True,
},
{
"id": "gravelly_villain",
"name": "Captain Crusty",
"icon": "☠️",
"description": "Gravelly old sailor — cartoon villain energy, original character.",
"instruct": "male, elderly, very low pitch",
"attrs": {
"Gender": "male", "Age": "elderly", "Pitch": "very low pitch",
"Style": "Auto", "EnglishAccent": "Auto",
"ChineseDialect": "Auto",
},
"script": (
"You came a long way for an answer you already had. Sit. The "
"fire is warm, and the truth is not."
),
"preview_url": "/demo_audio/voice_design/demo_voice_design_gravelly_villain.wav",
"language": "English",
"is_demo": True,
},
{
"id": "aussie_podcaster",
"name": "The Podcaster",
"icon": "🎙️",
"description": "Aussie explainer-show host — quick, punchy, friendly.",
"instruct": "female, young adult, high pitch, australian accent",
"attrs": {
"Gender": "female", "Age": "young adult", "Pitch": "high pitch",
"Style": "Auto", "EnglishAccent": "australian accent",
"ChineseDialect": "Auto",
},
"script": (
"Right, so here's the wild bit. Nobody told the engineers the "
"satellite was supposed to be in orbit by Tuesday. Tuesday came and went."
),
"preview_url": "/demo_audio/voice_design/demo_voice_design_aussie_podcaster.wav",
"language": "English",
"is_demo": True,
},
{
"id": "bedtime_storyteller",
"name": "Junior Quacks",
"icon": "🦆",
"description": "Anxious squawky sidekick — cartoon nephew energy, original character.",
"instruct": "young adult, high pitch",
"attrs": {
"Gender": "Auto", "Age": "young adult", "Pitch": "high pitch",
"Style": "Auto", "EnglishAccent": "Auto",
"ChineseDialect": "Auto",
},
"script": (
"Once, in a town where every street was named after a kind of "
"bread, a small fox decided she was going to learn to play the cello."
),
"preview_url": "/demo_audio/voice_design/demo_voice_design_bedtime_storyteller.wav",
"language": "English",
"is_demo": True,
},
{
"id": "mandarin_sichuan",
"name": "The Sichuan Friend",
"icon": "🌶️",
"description": "四川话 — non-English showcase, dialect-aware design.",
"instruct": "female, young adult, moderate pitch, 四川话",
"attrs": {
"Gender": "female", "Age": "young adult", "Pitch": "moderate pitch",
"Style": "Auto", "EnglishAccent": "Auto",
"ChineseDialect": "四川话",
},
"script": "今天天气巴适得很，我们去吃火锅嘛！记得多加点豆芽。",
"preview_url": "/demo_audio/voice_design/demo_voice_design_mandarin_sichuan.wav",
"language": "Chinese",
"is_demo": True,
},
]
backend/main.py
+7
View File
@@ -431,6 +431,13 @@ app.add_middleware(
app.mount("/audio", StaticFiles(directory=OUTPUTS_DIR), name="audio")
app.mount("/voice_audio", StaticFiles(directory=VOICES_DIR), name="voice_audio")
# Bundled demo assets — clone reference + pre-rendered output, voice-design
# preset previews, dictation samples. Read-only, ships with the app, no
# network. See scripts/build_demos.sh for how the WAVs are generated.
_DEMO_ASSETS_DIR = os.path.join(os.path.dirname(__file__), "assets", "samples")
if os.path.isdir(_DEMO_ASSETS_DIR):
app.mount("/demo_audio", StaticFiles(directory=_DEMO_ASSETS_DIR), name="demo_audio")
# ── Health check ────────────────────────────────────────────────────────
# Used by Docker health checks, load balancers, and the Tauri desktop shell.
backend/migrations/versions/0002_voice_profile_demo_fields.py
+62
View File
@@ -0,0 +1,62 @@
"""Phase 3: voice_profile description + is_demo columns
Revision ID: 0002_voice_profile_demo_fields
Revises: 0001_phase1_settings
Create Date: 2026-05-21 00:00:00.000000
Adds two additive nullable-with-default columns to ``voice_profiles``:
* ``description TEXT DEFAULT ''`` — human-readable blurb shown on demo
cards and (eventually) on user-created profiles. Empty string for
legacy rows so nothing in the frontend has to guard against NULL.
* ``is_demo INTEGER DEFAULT 0`` — flags bundled demo profiles so the
UI can render a "demo" badge and prevent accidental deletion.
Behavior:
* upgrade(): adds both columns with safe defaults. Uses the
sqlite_master pragma to detect existing columns so re-running the
migration on a fresh-install DB (where _BASE_SCHEMA already added
them) is a no-op rather than a hard error. This satisfies the
"Backward-compatible project data" constraint in CLAUDE.md.
* downgrade(): drops both columns. SQLite ≥ 3.35 supports
``ALTER TABLE ... DROP COLUMN``; we target it because every shipping
Python 3.11 bundle has sqlite3 ≥ 3.39.
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0002_voice_profile_demo_fields"
down_revision: Union[str, None] = "0001_phase1_settings"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _has_column(table: str, column: str) -> bool:
bind = op.get_bind()
rows = bind.execute(sa.text(f"PRAGMA table_info({table})")).fetchall()
return any(r[1] == column for r in rows)
def upgrade() -> None:
if not _has_column("voice_profiles", "description"):
op.add_column(
"voice_profiles",
sa.Column("description", sa.Text(), nullable=False, server_default=""),
)
if not _has_column("voice_profiles", "is_demo"):
op.add_column(
"voice_profiles",
sa.Column("is_demo", sa.Integer(), nullable=False, server_default="0"),
)
def downgrade() -> None:
# SQLite ≥ 3.35 supports DROP COLUMN. If both columns exist, drop them;
# otherwise no-op so partial-state DBs don't blow up.
if _has_column("voice_profiles", "is_demo"):
op.drop_column("voice_profiles", "is_demo")
if _has_column("voice_profiles", "description"):
op.drop_column("voice_profiles", "description")
backend/schemas/requests.py
+22 -1
View File
@@ -1,5 +1,5 @@
from pydantic import BaseModel, field_validator
from typing import List, Optional
from typing import List, Literal, Optional
from services.audio_dsp import EFFECT_PRESETS
@@ -60,8 +60,29 @@ class DubRequest(BaseModel):
# "time_stretch" — phase-vocoder stretch to fit, preserves pitch (default).
# "trim" — hard-clip to slot length + fade out (cheap, may cut mid-word).
# "off" — no fit; mix layers with += (legacy behaviour, may overlap).
# Legacy knob. When `timing_strategy` is set, it takes precedence and this is ignored.
slot_fit: Optional[str] = "time_stretch"
# High-level timing strategy. Replaces the audio-compression default that
# produced chipmunk/alien artefacts on high-density target languages
# (Bengali, Hindi, Arabic…). Three modes:
# "concise" — never compress TTS audio. Trim text up-front via
# speech_rate so it fits naturally; if it still
# overflows, hard-trim at slot with a short fade and
# surface fit_status="overflows" so the UI can prompt
# the user to shorten the segment. DEFAULT.
# "stretch_video" — never compress TTS audio. Re-lay the timeline so
# each segment's video portion is stretched (via
# ffmpeg setpts) to fit the natural-rate dub audio.
# Audio plays at 1.0×; total video duration grows.
# "strict_slot" — legacy: keep `slot_fit` semantics (atempo squeeze
# when audio > slot). Kept for back-compat.
timing_strategy: Optional[Literal["concise", "stretch_video", "strict_slot"]] = "concise"
# Per-job slip budget for "concise" mode. Hard-trim only kicks in once
# gap absorption + this much extra time has been consumed.
overflow_budget_s: Optional[float] = 0.0
class TranslateSegment(BaseModel):
id: str
text: str
backend/services/asr_backend.py
+98 -4
View File
@@ -142,13 +142,60 @@ class WhisperXBackend(ASRBackend):
return # older torch — secure unpickler didn't exist
allow = []
# omegaconf config containers — the immediate cause of the error
# pyannote's VAD emits (`GLOBAL omegaconf.listconfig.ListConfig`).
# omegaconf config containers + every node wrapper type the library
# exposes. pyannote's VAD checkpoint pickles `ListConfig` /
# `DictConfig` trees whose leaves are `AnyNode`/`ValueNode`/etc., so
# allowlist the whole family in one pass rather than waiting for
# users to hit each one in turn. All of these are pure metadata
# containers — no executable side effects.
try:
import omegaconf.nodes as _ocn
import omegaconf.base as _ocb
from omegaconf.listconfig import ListConfig
from omegaconf.dictconfig import DictConfig
from omegaconf.base import ContainerMetadata, Metadata
allow += [ListConfig, DictConfig, ContainerMetadata, Metadata]
allow += [ListConfig, DictConfig]
for _modname in ("nodes", "base"):
_mod = _ocn if _modname == "nodes" else _ocb
for _name in dir(_mod):
_obj = getattr(_mod, _name, None)
if isinstance(_obj, type) and _obj.__module__ == f"omegaconf.{_modname}":
allow.append(_obj)
except Exception:
pass
# `EnumNode` references real enum classes at unpickle time; allow
# the base Enum/IntEnum/Flag types so configs using enums load.
try:
import enum
allow += [enum.Enum, enum.IntEnum, enum.Flag, enum.IntFlag]
except Exception:
pass
# torch utility types that aren't in the secure unpickler's
# default allowlist. `TorchVersion` is a `str` subclass that
# pyannote/lightning serialise as metadata; `Size` is the shape
# tuple type used in tensor metadata. Both are inert data.
try:
from torch.torch_version import TorchVersion
import torch as _torch
allow += [TorchVersion, _torch.Size]
except Exception:
pass
# PyTorch Lightning serialises `hyper_parameters` as
# `argparse.Namespace` (or an AttributeDict subclass thereof) so
# configs roundtrip. Allowlist the Namespace constructor — it is
# just an attribute bag with no executable side effects.
try:
import argparse
allow += [argparse.Namespace]
except Exception:
pass
# pyannote-specific metadata classes that travel with the VAD
# checkpoint. Only the inert data-only types are allowlisted —
# the `Model` / `Task` / `Dataset` classes from the same modules
# do real work in `__init__` and stay off the allowlist.
try:
from pyannote.audio.core.model import Introspection, Output
from pyannote.audio.core.task import Problem, Resolution, Specifications
allow += [Introspection, Output, Problem, Resolution, Specifications]
except Exception:
pass
# Python typing primitives that show up in config annotations.
@@ -163,6 +210,53 @@ class WhisperXBackend(ASRBackend):
allow += [OrderedDict, defaultdict]
except Exception:
pass
# Plain-data builtins. pyannote's VAD checkpoint pickles config
# entries that resolve to bare builtin constructors (`GLOBAL list`,
# `GLOBAL int`, …) and the secure unpickler refuses each one
# without an explicit allowlist. These constructors only build
# inert data primitives — no side effects, no code paths — so the
# full set is safe to allowlist together, which avoids users
# hitting them one-at-a-time as the checkpoint deserialises.
allow += [
list, dict, tuple, set, frozenset,
int, float, bool, str, bytes, bytearray, complex,
type(None), slice, range,
]
# numpy scalar/array constructors that show up in pyannote configs
# (sample rates, hop sizes saved as numpy ints/floats). Each is a
# pure data type — safe to allowlist.
try:
import numpy as _np
allow += [
_np.ndarray, _np.dtype,
_np.int8, _np.int16, _np.int32, _np.int64,
_np.uint8, _np.uint16, _np.uint32, _np.uint64,
_np.float16, _np.float32, _np.float64,
_np.bool_, _np.complex64, _np.complex128,
]
# numpy.core was renamed to numpy._core in 1.25+. Both modules
# expose the same reconstruct helpers; allowlist whichever ships.
for _modname in ("numpy._core.multiarray", "numpy.core.multiarray"):
try:
_mod = __import__(_modname, fromlist=["_reconstruct", "scalar"])
for _attr in ("_reconstruct", "scalar"):
_fn = getattr(_mod, _attr, None)
if _fn is not None:
allow.append(_fn)
except Exception:
pass
except Exception:
pass
# pathlib types — config files sometimes save cache directories as
# Path objects so the checkpoint can be relocated.
try:
import pathlib
allow += [
pathlib.PurePath, pathlib.PurePosixPath, pathlib.PureWindowsPath,
pathlib.Path, pathlib.PosixPath, pathlib.WindowsPath,
]
except Exception:
pass
if allow:
try:
add(allow)
backend/services/dub_pipeline.py
+313 -27
View File
@@ -32,6 +32,7 @@ import logging
import os
import re
import shutil
import subprocess
import sys
import threading
import time
@@ -41,7 +42,7 @@ import soundfile as sf
from core.config import DUB_DIR
from fastapi import HTTPException
from services.ffmpeg_utils import find_ffmpeg, _get_semaphore, _spawn_with_retry
from services.ffmpeg_utils import find_ffmpeg, find_ffprobe, _get_semaphore, _spawn_with_retry
from services.model_manager import get_best_device
from core.db import db_conn, get_db
from core import event_bus
@@ -279,45 +280,240 @@ def run_proc_factory(job_id: str):
return run_proc
async def run_proc_streaming_stderr(
job_id: str,
cmd: list[str],
*,
timeout: float = 1800.0,
) -> AsyncIterator[tuple]:
"""Spawn `cmd` and stream its stderr line-by-line as it runs.
Yields:
('stderr', line: str) — once per logical line (split on \\r or \\n,
so tqdm-style progress bars that overwrite the same line via
carriage return surface as separate lines)
('done', returncode: int, full_stderr: bytes) — exactly once when
the subprocess exits; this is always the final value.
Honors the same semaphore + register_proc/kill plumbing as run_proc.
"""
async with _get_semaphore():
p = await _spawn_with_retry(
cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
register_proc(job_id, p)
stderr_parts: list[bytes] = []
rc: int = -1
try:
buf = b""
start = time.monotonic()
while True:
if time.monotonic() - start > timeout:
try:
p.kill()
except ProcessLookupError:
pass
raise HTTPException(
status_code=504,
detail=f"subprocess timed out after {timeout}s",
)
try:
chunk = await asyncio.wait_for(p.stderr.read(256), timeout=1.0)
except asyncio.TimeoutError:
if p.returncode is not None:
break
continue
if not chunk:
break
stderr_parts.append(chunk)
buf += chunk
while True:
idx_r = buf.find(b"\r")
idx_n = buf.find(b"\n")
if idx_r < 0 and idx_n < 0:
break
if idx_r < 0:
idx = idx_n
elif idx_n < 0:
idx = idx_r
else:
idx = min(idx_r, idx_n)
line = buf[:idx].decode(errors="replace")
buf = buf[idx + 1:]
if line.strip():
yield ("stderr", line)
rc = await p.wait()
finally:
unregister_proc(job_id, p)
if p.returncode is None:
try:
p.kill()
except ProcessLookupError:
pass
try:
await asyncio.wait_for(p.wait(), timeout=5.0)
except asyncio.TimeoutError:
pass
try:
p.stdout.close()
except Exception:
pass
yield ("done", rc, b"".join(stderr_parts))
_BROWSER_VIDEO_CODECS = {"h264", "avc1"}
_BROWSER_AUDIO_CODECS = {"aac", "mp4a"}
def _probe_codecs(path: str) -> tuple[str, str]:
"""Return (video_codec, audio_codec) lowercased, or ('','') on probe error."""
ffprobe = find_ffprobe()
if not ffprobe:
return ("", "")
try:
out = subprocess.run(
[ffprobe, "-v", "error", "-select_streams", "v:0",
"-show_entries", "stream=codec_name", "-of", "csv=p=0", path],
capture_output=True, text=True, timeout=10,
)
vcodec = (out.stdout or "").strip().lower()
out = subprocess.run(
[ffprobe, "-v", "error", "-select_streams", "a:0",
"-show_entries", "stream=codec_name", "-of", "csv=p=0", path],
capture_output=True, text=True, timeout=10,
)
acodec = (out.stdout or "").strip().lower()
return vcodec, acodec
except Exception:
return ("", "")
def _ensure_browser_playable_mp4(video_path: str) -> str:
"""Guarantee `video_path` is an mp4 with h264 video + aac audio.
Three classes of fix the in-app `<video>` element needs:
1. `.webm`/`.mkv` containers — WKWebView refuses them outright.
2. `.mp4` with VP9 or AV1 video — WKWebView can't decode either
inside an mp4 container (Safari ships VP9 only in WebM).
3. `.mp4` with Opus audio — same story; mp4-with-opus is rare and
poorly supported across webviews.
Strategy: probe codecs; if extension is mp4 AND both codecs are in
the browser-safe set, leave alone. Otherwise transcode to h264+aac.
Returns the (possibly new) file path.
"""
ffmpeg_bin = find_ffmpeg()
is_mp4 = video_path.lower().endswith(".mp4")
if is_mp4:
vcodec, acodec = _probe_codecs(video_path)
if vcodec in _BROWSER_VIDEO_CODECS and acodec in _BROWSER_AUDIO_CODECS:
return video_path # already safe — fast path, no rewrite
# Need to rewrite. Try stream-copy first when the container is the
# only problem; fall back to full transcode for codec mismatches.
target = os.path.splitext(video_path)[0] + ".mp4"
if target == video_path:
target = os.path.splitext(video_path)[0] + ".browser.mp4"
# Stream-copy attempt — works when codecs are already h264+aac but
# the container is wrong (rare with our format selector but cheap to
# try). Skip straight to transcode if we already know codecs are bad.
rc = 1
if is_mp4:
# Codecs were probed and known-bad; skip the copy attempt.
pass
else:
rc = subprocess.run(
[ffmpeg_bin, "-y", "-i", video_path,
"-c:v", "copy", "-c:a", "copy",
"-movflags", "+faststart", target],
capture_output=True,
).returncode
if rc != 0 or not os.path.exists(target):
rc = 1
if rc != 0:
# Full transcode — h264 baseline-ish + aac is the safe combo.
rc = subprocess.run(
[ffmpeg_bin, "-y", "-i", video_path,
"-c:v", "libx264", "-preset", "veryfast", "-crf", "23",
"-pix_fmt", "yuv420p",
"-c:a", "aac", "-b:a", "192k",
"-movflags", "+faststart", target],
capture_output=True,
).returncode
if rc == 0 and os.path.exists(target) and target != video_path:
try:
os.remove(video_path)
except OSError:
pass
return target
if rc != 0:
logger.warning(
"Could not transcode %s to browser-playable mp4 — the in-app "
"video player may render this file as a black box.",
video_path,
)
return video_path
def yt_download_sync(
url: str,
job_dir: str,
*,
fetch_subs: bool = False,
sub_langs: list[str] | None = None,
progress_hook=None,
) -> tuple[str, str, list[str]]:
"""Blocking yt-dlp download into `job_dir`.
Returns (video_path, title, downloaded_sub_files).
When `fetch_subs` is True we also ask yt-dlp to download both
manually-uploaded (`writesubtitles=True`) and auto-generated / auto-
translated captions (`writeautomaticsub=True`). This is how we pull
YouTube's free machine translations without needing a Google API key —
yt-dlp talks to the same public endpoints the YouTube player does.
`sub_langs` controls which language tracks to ask for; default `['all']`
grabs whatever the uploader / auto-translator makes available.
Captions are downloaded in a separate, best-effort pass after the video
so subtitle failures (rate-limits, missing tracks) can never derail the
actual ingest. Defaults skip YouTube's auto-translated set — requesting
"all" expands into ~100 per-language variants per video and reliably
trips HTTP 429, and the downstream Translate step handles target
languages more reliably than YouTube's machine translations anyway.
Pass an explicit `sub_langs` list to override the default selection.
"""
import glob
import yt_dlp
outtmpl = os.path.join(job_dir, "original.%(ext)s")
ydl_opts: dict = {
"outtmpl": outtmpl,
"format": "bv*[ext=mp4]+ba[ext=m4a]/b[ext=mp4]/bv*+ba/b",
# Prefer h264+aac streams so the merged mp4 is natively decodable
# by WKWebView/WebView2/Chromium. YouTube serves VP9+Opus as the
# default high-quality combo, which yt-dlp will happily mux into
# an mp4 container — but Safari/WKWebView refuses to decode VP9
# inside mp4, leaving a black <video> in the dub editor. Fall
# back to any combo only when no h264/aac variant exists, then
# the post-download codec probe below will transcode it.
"format": (
"bv*[vcodec^=avc1][ext=mp4]+ba[acodec^=mp4a][ext=m4a]/"
"bv*[vcodec^=avc1]+ba[acodec^=mp4a]/"
"b[vcodec^=avc1][acodec^=mp4a]/"
"bv*[vcodec^=avc1]+ba/"
"b[vcodec^=avc1]/"
"bv*+ba/b"
),
"merge_output_format": "mp4",
"noplaylist": True,
"quiet": True,
"no_warnings": True,
"restrictfilenames": True,
"socket_timeout": 30,
# Resilience against YouTube CDN flakes: a single empty fragment
# (commonly the very last one — "Did not get any data blocks")
# used to fail the whole ingest at 99% complete. Retry each
# fragment generously and tolerate one that never returns data;
# missing the last <0.5% of audio is acceptable for transcription.
"fragment_retries": 10,
"retries": 10,
"extractor_retries": 5,
"skip_unavailable_fragments": True,
}
if fetch_subs:
ydl_opts.update({
"writesubtitles": True,
"writeautomaticsub": True,
"subtitleslangs": list(sub_langs) if sub_langs else ["all"],
"subtitlesformat": "vtt",
})
if progress_hook is not None:
ydl_opts["progress_hooks"] = [progress_hook]
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
info = ydl.extract_info(url, download=True)
path = ydl.prepare_filename(info)
@@ -327,12 +523,48 @@ def yt_download_sync(
video_path = mp4
else:
video_path = path
# Browser-playability guard: WKWebView (Tauri on macOS) refuses to
# decode VP9/AV1 video and Opus audio even when they're wrapped in an
# mp4 container, and refuses .webm/.mkv outright. We probe the actual
# codecs and transcode only when needed — `<video>` rendering is
# what we care about, not what extension yt-dlp ended up writing.
video_path = _ensure_browser_playable_mp4(video_path)
title = info.get("title") or os.path.basename(video_path)
sub_files: list[str] = []
if fetch_subs:
# yt-dlp names captions "<base>.<lang>.vtt"; scoop them all.
base = os.path.splitext(video_path)[0]
sub_files = sorted(glob.glob(base + ".*.vtt"))
return video_path, info.get("title") or os.path.basename(video_path), sub_files
if sub_langs:
langs = list(sub_langs)
else:
orig = (info.get("language") or "").strip()
manual = list((info.get("subtitles") or {}).keys())
langs = sorted({*manual, *([orig] if orig else [])})
if not langs:
logger.info("No captions available on %s (skipping subtitle pass)", url)
else:
sub_opts = {
**ydl_opts,
"skip_download": True,
"writesubtitles": True,
"writeautomaticsub": True,
"subtitleslangs": langs,
"subtitlesformat": "vtt",
"extractor_args": {"youtube": {"skip": ["translated_subs"]}},
"ignoreerrors": True,
"extractor_retries": 5,
"sleep_interval_subtitles": 1,
}
try:
with yt_dlp.YoutubeDL(sub_opts) as ydl_sub:
ydl_sub.extract_info(url, download=True)
except Exception as e:
logger.warning(
"Subtitle download failed for %s (continuing with video): %s",
url, e,
)
base = os.path.splitext(video_path)[0]
sub_files = sorted(glob.glob(base + ".*.vtt"))
return video_path, title, sub_files
def parse_vtt_segments(vtt_path: str) -> list[dict]:
@@ -410,13 +642,51 @@ async def ingest_pipeline(
fetch_subs = bool(source.get("fetch_subs"))
sub_langs = source.get("sub_langs") or None
yield prep_event("download_start", url=url)
# Bridge yt-dlp's per-fragment progress callback (fires inside
# the worker thread) into the async generator via a threadsafe
# queue so the UI can render a real download bar.
loop = asyncio.get_running_loop()
dl_queue: asyncio.Queue = asyncio.Queue()
_last_pct = -1
def _yt_progress(d: dict) -> None:
nonlocal _last_pct
status = d.get("status")
if status == "downloading":
total = d.get("total_bytes") or d.get("total_bytes_estimate") or 0
cur = d.get("downloaded_bytes") or 0
if total > 0:
pct = max(0, min(100, int(cur * 100 / total)))
if pct != _last_pct:
_last_pct = pct
payload = {
"percent": pct,
"speed_bps": d.get("speed") or 0,
"eta_s": d.get("eta"),
}
loop.call_soon_threadsafe(dl_queue.put_nowait, payload)
dl_task = asyncio.create_task(asyncio.to_thread(
yt_download_sync, url, job_dir,
fetch_subs=fetch_subs, sub_langs=sub_langs,
progress_hook=_yt_progress,
))
try:
video_path, title, sub_files = await asyncio.to_thread(
yt_download_sync, url, job_dir,
fetch_subs=fetch_subs, sub_langs=sub_langs,
)
while not dl_task.done():
try:
payload = await asyncio.wait_for(dl_queue.get(), timeout=0.5)
yield prep_event("download_progress", **payload)
except asyncio.TimeoutError:
continue
# Drain any final queued events.
while not dl_queue.empty():
payload = dl_queue.get_nowait()
yield prep_event("download_progress", **payload)
video_path, title, sub_files = await dl_task
except Exception as e:
logger.exception("Download failed for job %s", job_id)
if not dl_task.done():
dl_task.cancel()
yield prep_event("error", **failure.build_failure(e, stage="download"))
shutil.rmtree(job_dir, ignore_errors=True)
return
@@ -543,9 +813,25 @@ async def ingest_pipeline(
demucs_cmd = [sys.executable, "-m", "demucs.separate",
"--two-stems", "vocals", "-n", "htdemucs", "-d", get_best_device(),
audio_path, "-o", job_dir]
p, _, stderr = await run_proc(demucs_cmd, timeout=1800.0)
if p.returncode != 0:
raise Exception(stderr.decode(errors="replace")[:500])
rc = -1
stderr_full = b""
last_pct = -1
# demucs writes a tqdm progress bar to stderr as
# " 42%|████ | …" — surface each new integer percent
# to the UI so the user sees the bar instead of a static
# spinner during the multi-minute separation step.
async for evt in run_proc_streaming_stderr(job_id, demucs_cmd, timeout=1800.0):
if evt[0] == "stderr":
m = re.search(r"(\d{1,3})%", evt[1])
if m:
pct = max(0, min(100, int(m.group(1))))
if pct != last_pct:
last_pct = pct
yield prep_event("demucs_progress", percent=pct)
elif evt[0] == "done":
rc, stderr_full = evt[1], evt[2]
if rc != 0:
raise Exception(stderr_full.decode(errors="replace")[:500])
demucs_out = os.path.join(job_dir, "htdemucs", "audio")
if os.path.exists(os.path.join(demucs_out, "vocals.wav")):
shutil.move(os.path.join(demucs_out, "vocals.wav"), vocals_path)
backend/services/speech_rate.py
+29 -2
View File
@@ -21,11 +21,38 @@ from services.llm_backend import get_active_llm_backend, OffBackend
logger = logging.getLogger("omnivoice.speech_rate")
# Per-language read-speed estimates (chars/sec at natural pace).
# These are rough; real speakers vary wildly. Good enough for a first pass.
# Per-language read-speed estimates (chars/sec at natural pace, counting
# Python `len()` codepoints — not phonemes or graphemes). These are
# rough; real speakers vary wildly. Numbers below come from a mix of
# Pellegrino et al. 2011 (Cross-language information rate) and informal
# calibration against TTS engine outputs.
#
# Codepoint density matters a lot here because Indic scripts (Devanagari,
# Bengali, Tamil…) encode vowel-marks as separate codepoints, inflating
# `len(text)` for the same spoken duration. Without an explicit entry,
# `expected_duration` falls back to 13.0 cps — which produces ratios
# 1.3-1.7× the truth for Bengali/Hindi/Tamil and forces aggressive
# slot-compression in TTS that the WSOLA stretch then has to repair.
_RATE_CPS = {
"en": 15.0, "de": 14.0, "fr": 15.0, "es": 15.5, "it": 15.0, "pt": 15.0,
# CJK — logographic / mora-based scripts, fewer chars per second.
"ja": 10.0, "ko": 10.0, "zh": 6.0,
# Indic — Devanagari/Bengali/Tamil/Telugu/etc. compound graphemes
# decompose into multiple codepoints; spoken syllable rate is
# closer to English but the codepoint count is higher.
"hi": 17.0, "bn": 17.0, "ta": 14.0, "te": 14.0, "mr": 16.0,
"gu": 16.0, "kn": 14.0, "ml": 14.0, "pa": 16.0, "or": 16.0,
"ur": 13.0,
# RTL / Semitic — Arabic & Hebrew have shorter codepoint counts per
# word than English (no vowel chars written) so cps reads lower.
"ar": 12.0, "he": 12.0, "fa": 13.0,
# Southeast Asian — Thai is contiguous (no spaces); Vietnamese is
# concise; Indonesian is concise but Latin-scripted.
"th": 10.0, "vi": 16.0, "id": 14.0, "ms": 14.0,
# Slavic + Turkic — agglutinative or compound-heavy; long words.
"ru": 13.0, "pl": 13.0, "uk": 13.0, "cs": 13.0, "tr": 12.0,
# Nordic / Greek — close to mainland European baseline.
"el": 14.0, "nl": 14.0, "sv": 14.0, "no": 14.0, "da": 14.0, "fi": 13.0,
}
# Tolerance window — if predicted ratio is within this of 1.0 we accept.
bun.lock
+16 -12
View File
@@ -399,17 +399,17 @@
"@testing-library/react": ["@testing-library/react@16.3.2", "", { "dependencies": { "@babel/runtime": "^7.12.5" }, "peerDependencies": { "@testing-library/dom": "^10.0.0", "@types/react": "^18.0.0 || ^19.0.0", "@types/react-dom": "^18.0.0 || ^19.0.0", "react": "^18.0.0 || ^19.0.0", "react-dom": "^18.0.0 || ^19.0.0" }, "optionalPeers": ["@types/react", "@types/react-dom"] }, "sha512-XU5/SytQM+ykqMnAnvB2umaJNIOsLF3PVv//1Ew4CTcpz0/BRyy/af40qqrt7SjKpDdT1saBMc42CUok5gaw+g=="],
"@turbo/darwin-64": ["@turbo/darwin-64@2.9.7", "", { "os": "darwin", "cpu": "x64" }, "sha512-wnvOWuVWJ5EUHNKxExEWiGlTeVpLG1L0PCu5MUozyC1P2SHGiWsmpW6/yAuShH91Fa2TAHOvdCRBzriZh4j4Eg=="],
"@turbo/darwin-64": ["@turbo/darwin-64@2.9.15", "", { "os": "darwin", "cpu": "x64" }, "sha512-nnDo9R1Df+s2x6jxlERtbg7xRpuicf8p4J2krcnjeaMBt3q9V41pGXa4t9YM2Y4ozozsVJ+CH405CJUrWIQK4Q=="],
"@turbo/darwin-arm64": ["@turbo/darwin-arm64@2.9.7", "", { "os": "darwin", "cpu": "arm64" }, "sha512-mA0FIPMwwN3lodDkQYaGxj6PeT7ZaN5aCEbkKn/WB+ZB9yJdVWA4J83GH7t43jqDc5dcnVluVN5UFx3plRiXhA=="],
"@turbo/darwin-arm64": ["@turbo/darwin-arm64@2.9.15", "", { "os": "darwin", "cpu": "arm64" }, "sha512-fDSx56oqoFuS+yUQw7hqjQTkjrSLdMcplhuLC8HcSkWC6YrpwEmUUYsPYHPxy4ALvLxnmPQuk6XoSD8tdkjP+g=="],
"@turbo/linux-64": ["@turbo/linux-64@2.9.7", "", { "os": "linux", "cpu": "x64" }, "sha512-fEbUYpgb5l7P+q+5tsWF2gw+/GSjUsuUTcnfm+f0lozUjgcjLKyOat6PgtAChmIFcTPchCL/8rJ3TvkBy01gfA=="],
"@turbo/linux-64": ["@turbo/linux-64@2.9.15", "", { "os": "linux", "cpu": "x64" }, "sha512-/bmxn+x/xE+oh0VzEXt/zf2zsORAYZPrL3db5/VrXzYt0Z4wxcvffwJBGlSfla2smfS1BLGBiyWldJlWDXJVXA=="],
"@turbo/linux-arm64": ["@turbo/linux-arm64@2.9.7", "", { "os": "linux", "cpu": "arm64" }, "sha512-VkUjulo9ytfHKUHOS5gy0XPoh4CTKPXWCL8nLdrlHVi9fSut31ECeUqnm/dAbETP5D4xo9mH9XkJ+qMzGe/zmg=="],
"@turbo/linux-arm64": ["@turbo/linux-arm64@2.9.15", "", { "os": "linux", "cpu": "arm64" }, "sha512-cbOaDe1ijz5As+mimOOHgmRMolZZZO7miNBHHp5xdiYMm2Q/Dwu1JVLx/Kw4s7xjocG/oEoHrpHrxpEAIEfNiw=="],
"@turbo/windows-64": ["@turbo/windows-64@2.9.7", "", { "os": "win32", "cpu": "x64" }, "sha512-/GWdY6/x4aIHqkYJq596Rpdk1x0MkpRPkJcLAoB3yGRwyUms0+u2F1GnV54IbyAZTeKLRWSJKzNC+QwVGdYchA=="],
"@turbo/windows-64": ["@turbo/windows-64@2.9.15", "", { "os": "win32", "cpu": "x64" }, "sha512-/Fzm7afui7uK7dFBwrTXKuDhBBTiHk5I+hMVAPMR7cqQyDo2norCNUsN9PdNuYcmzYbhSOxzz498wQYvSAz29w=="],
"@turbo/windows-arm64": ["@turbo/windows-arm64@2.9.7", "", { "os": "win32", "cpu": "arm64" }, "sha512-xBBgxCC5PK2+WZ1PPRZdp+aJ0bMBcEbweXWux3RUHJvX9ZodcoQySkrW6qt+ahb+uk8ZjyQodLfDwtVSoYds1w=="],
"@turbo/windows-arm64": ["@turbo/windows-arm64@2.9.15", "", { "os": "win32", "cpu": "arm64" }, "sha512-fOHEsLcqVdFXLw2ApWv4gxwfHzkUnpo9rHGml+9+dyHj148m/Bc+556kEvb5+4u6prI1LMd8zEZE2HcO6Jn2VQ=="],
"@tybys/wasm-util": ["@tybys/wasm-util@0.10.1", "", { "dependencies": { "tslib": "^2.4.0" } }, "sha512-9tTaPJLSiejZKx+Bmog4uSubteqTvFrVrURwkmHixBo0G4seD0zUxp98E1DzUBJxLQ3NPwXrGKDiVjwx/DpPsg=="],
@@ -449,6 +449,8 @@
"acorn-jsx": ["acorn-jsx@5.3.2", "", { "peerDependencies": { "acorn": "^6.0.0 || ^7.0.0 || ^8.0.0" } }, "sha512-rq9s+JNhf0IChjtDXxllJ7g41oZk5SlXtp0LHwyA5cejwn7vKmKp4pPri6YEePv2PU65sAsegbXtIinmDFDXgQ=="],
"agent-base": ["agent-base@6.0.2", "", { "dependencies": { "debug": "4" } }, "sha512-RZNwNclF7+MS/8bDg70amg32dyeZGZxiDuQmZxKLAlQjr3jGyLx+4Kkk58UO7D2QdgFIQCovuSuZESne6RG6XQ=="],
"ajv": ["ajv@6.14.0", "", { "dependencies": { "fast-deep-equal": "^3.1.1", "fast-json-stable-stringify": "^2.0.0", "json-schema-traverse": "^0.4.1", "uri-js": "^4.2.2" } }, "sha512-IWrosm/yrn43eiKqkfkHis7QioDleaXQHdDVPKg0FSwwd/DuvyX79TZnFOnYpB7dcsFAMmtFztZuXPDvSePkFw=="],
"ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="],
@@ -463,7 +465,7 @@
"asynckit": ["asynckit@0.4.0", "", {}, "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="],
"axios": ["axios@1.15.0", "", { "dependencies": { "follow-redirects": "^1.15.11", "form-data": "^4.0.5", "proxy-from-env": "^2.1.0" } }, "sha512-wWyJDlAatxk30ZJer+GeCWS209sA42X+N5jU2jy6oHTp7ufw8uzUTVFBX9+wTfAlhiJXGS0Bq7X6efruWjuK9Q=="],
"axios": ["axios@1.16.1", "", { "dependencies": { "follow-redirects": "^1.16.0", "form-data": "^4.0.5", "https-proxy-agent": "^5.0.1", "proxy-from-env": "^2.1.0" } }, "sha512-caYkukvroVPO8KrzuJEb50Hm07KwfBZPEC3VeFHTsqWHvKTsy54hjJz9BS/cdaypROE2rH6xvm9mHX4fgWkr3A=="],
"balanced-match": ["balanced-match@4.0.4", "", {}, "sha512-BLrgEcRTwX2o6gGxGOCNyMvGSp35YofuYzw9h1IMTRmKqttAZZVU67bdb9Pr2vUHA8+j3i2tJfjO6C6+4myGTA=="],
@@ -637,6 +639,8 @@
"html-parse-stringify": ["html-parse-stringify@3.0.1", "", { "dependencies": { "void-elements": "3.1.0" } }, "sha512-KknJ50kTInJ7qIScF3jeaFRpMpE8/lfiTdzf/twXyPBLAGrLRTmkz3AdTnKeh40X8k9L2fdYwEp/42WGXIRGcg=="],
"https-proxy-agent": ["https-proxy-agent@5.0.1", "", { "dependencies": { "agent-base": "6", "debug": "4" } }, "sha512-dFcAjpTQFgoLMzC2VwU+C/CbS7uRL0lWmxDITmqm7C+7F0Odmj6s9l6alZc6AELXhrnggM2CeWSXHGOdX2YtwA=="],
"human-signals": ["human-signals@8.0.1", "", {}, "sha512-eKCa6bwnJhvxj14kZk5NCPc6Hb6BdsU9DZcOnmQKSnO1VKrfV0zCvtttPZUsBvjmNDn8rpcJfpwSYnHBjc95MQ=="],
"i18next": ["i18next@26.0.8", "", { "peerDependencies": { "typescript": "^5 || ^6" }, "optionalPeers": ["typescript"] }, "sha512-BRzLom0mhDhV9v0QhgUUHWQJuwFmnr1194xEcNLYD6ym8y8s542n4jXUvRLnhNTbh9PmpU6kGZamyuGHQMsGjw=="],
@@ -667,7 +671,7 @@
"jiti": ["jiti@2.6.1", "", { "bin": { "jiti": "lib/jiti-cli.mjs" } }, "sha512-ekilCSN1jwRvIbgeg/57YFh8qQDNbwDb9xT/qu2DAHbFFZUicIl4ygVaAvzveMhMVr3LnpSKTNnwt8PoOfmKhQ=="],
"joi": ["joi@18.1.2", "", { "dependencies": { "@hapi/address": "^5.1.1", "@hapi/formula": "^3.0.2", "@hapi/hoek": "^11.0.7", "@hapi/pinpoint": "^2.0.1", "@hapi/tlds": "^1.1.1", "@hapi/topo": "^6.0.2", "@standard-schema/spec": "^1.1.0" } }, "sha512-rF5MAmps5esSlhCA+N1b6IYHDw9j/btzGaqfgie522jS02Ju/HXBxamlXVlKEHAxoMKQL77HWI8jlqWsFuekZA=="],
"joi": ["joi@18.2.1", "", { "dependencies": { "@hapi/address": "^5.1.1", "@hapi/formula": "^3.0.2", "@hapi/hoek": "^11.0.7", "@hapi/pinpoint": "^2.0.1", "@hapi/tlds": "^1.1.1", "@hapi/topo": "^6.0.2", "@standard-schema/spec": "^1.1.0" } }, "sha512-2/OKlogiESf2Nh3TFCrRjrr9z1DRHeW0I+KReF67+4J0Ns+8hBtHRmoWAZ2OFU6I5+TWLEe6sVlSdXPjHm5UbQ=="],
"js-tokens": ["js-tokens@4.0.0", "", {}, "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ=="],
@@ -775,9 +779,9 @@
"pid-port": ["pid-port@2.0.1", "", { "dependencies": { "execa": "^9.6.0" } }, "sha512-pnLo01AmMclw8l+/gfknsP2N351oe8VkVmCLFUvJZ11NRPPmghJrv0OcwsdgPQxsZkFYwm6hPWW0JKmXYCaXAw=="],
"playwright": ["playwright@1.59.1", "", { "dependencies": { "playwright-core": "1.59.1" }, "optionalDependencies": { "fsevents": "2.3.2" }, "bin": { "playwright": "cli.js" } }, "sha512-C8oWjPR3F81yljW9o5OxcWzfh6avkVwDD2VYdwIGqTkl+OGFISgypqzfu7dOe4QNLL2aqcWBmI3PMtLIK233lw=="],
"playwright": ["playwright@1.60.0", "", { "dependencies": { "playwright-core": "1.60.0" }, "optionalDependencies": { "fsevents": "2.3.2" }, "bin": { "playwright": "cli.js" } }, "sha512-hheHdokM8cdqCb0lcE3s+zT4t4W+vvjpGxsZlDnikarzx8tSzMebh3UiFtgqwFwnTnjYQcsyMF8ei2mCO/tpeA=="],
"playwright-core": ["playwright-core@1.59.1", "", { "bin": { "playwright-core": "cli.js" } }, "sha512-HBV/RJg81z5BiiZ9yPzIiClYV/QMsDCKUyogwH9p3MCP6IYjUFu/MActgYAvK0oWyV9NlwM3GLBjADyWgydVyg=="],
"playwright-core": ["playwright-core@1.60.0", "", { "bin": { "playwright-core": "cli.js" } }, "sha512-9bW6zvX/m0lEbgTKJ6YppOKx8H3VOPBMOCFh2irXFOT4BbHgrx5hPjwJYLT40Lu+4qtD36qKc/Hn56StUW57IA=="],
"postcss": ["postcss@8.5.10", "", { "dependencies": { "nanoid": "^3.3.11", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-pMMHxBOZKFU6HgAZ4eyGnwXF/EvPGGqUr0MnZ5+99485wwW41kW91A4LOGxSHhgugZmSChL5AlElNdwlNgcnLQ=="],
@@ -877,7 +881,7 @@
"tslib": ["tslib@2.8.1", "", {}, "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w=="],
"turbo": ["turbo@2.9.7", "", { "optionalDependencies": { "@turbo/darwin-64": "2.9.7", "@turbo/darwin-arm64": "2.9.7", "@turbo/linux-64": "2.9.7", "@turbo/linux-arm64": "2.9.7", "@turbo/windows-64": "2.9.7", "@turbo/windows-arm64": "2.9.7" }, "bin": { "turbo": "bin/turbo" } }, "sha512-epxzqVO2s0IxcSWcgb+qKrtco8isfe7g3VtiS6hkYnEK4A9XQDZbrtavQ6MtWR1KoQn+1fUomaQth2rfRHlUlg=="],
"turbo": ["turbo@2.9.15", "", { "optionalDependencies": { "@turbo/darwin-64": "2.9.15", "@turbo/darwin-arm64": "2.9.15", "@turbo/linux-64": "2.9.15", "@turbo/linux-arm64": "2.9.15", "@turbo/windows-64": "2.9.15", "@turbo/windows-arm64": "2.9.15" }, "bin": { "turbo": "bin/turbo" } }, "sha512-VpKvD9Z0Hu/xrGUAYX1wnhfpqv835wIwGqeKfulvBPTOcDap0E3nFwyzCAVV85fB1sBcBDEfTP+7FSW7GzwWSQ=="],
"type-check": ["type-check@0.4.0", "", { "dependencies": { "prelude-ls": "^1.2.1" } }, "sha512-XleUoc9uwGXqjWwXaUTZAmzMcFZ5858QA2vvx1Ur5xIcixXIP+8LnFDgRplU30us6teqdlskFfu+ae4K79Ooew=="],
@@ -905,7 +909,7 @@
"w3c-xmlserializer": ["w3c-xmlserializer@5.0.0", "", { "dependencies": { "xml-name-validator": "^5.0.0" } }, "sha512-o8qghlI8NZHU1lLPrpi2+Uq7abh4GGPpYANlalzWxyWteJOCsr/P+oPBA49TOLu5FTZO4d3F9MnWJfiMo4BkmA=="],
"wait-on": ["wait-on@9.0.5", "", { "dependencies": { "axios": "^1.15.0", "joi": "^18.1.2", "lodash": "^4.18.1", "minimist": "^1.2.8", "rxjs": "^7.8.2" }, "bin": { "wait-on": "bin/wait-on" } }, "sha512-qgnbHDfDTRIp73ANEJNRW/7kn8CrDUcvZz18xotJQku/P4saTGkbIzvnMZebPmVvVNUiRq1qWAPyqCH+W4H8KA=="],
"wait-on": ["wait-on@9.0.10", "", { "dependencies": { "axios": "^1.16.0", "joi": "^18.2.1", "lodash": "^4.18.1", "minimist": "^1.2.8", "rxjs": "^7.8.2" }, "bin": { "wait-on": "bin/wait-on" } }, "sha512-rCoJEhvMr0X6alHmwc9abbrA5ZrLZFKpFQVKPNFwl2h7DapXOGdmimIHDtLOWhT4PjhZhxFEtZoQgEXbkDWdZw=="],
"wavesurfer.js": ["wavesurfer.js@7.12.6", "", {}, "sha512-zSxPgOFprtyJ31ppHQF0+E9jAmjAhi1rR36yIW6h1GOYdpRxDe6mbkYtlChqLK0Iz8ROBweiEFw2zus7tDFibA=="],
frontend/src-tauri/src/lib.rs
+11
View File
@@ -309,6 +309,17 @@ pub fn run() {
#[cfg(not(target_os = "macos"))]
let _ = win.set_skip_taskbar(false);
let _ = win.set_focus();
// Self-recovery: if the webview failed to load
// the dev/prod URL earlier (Vite restarted,
// backend not up yet at first show, etc.) the
// window shows a blank `<body></body>` with a
// "Could not connect to the server" console
// error. Reload only when the body is empty
// so a healthy window doesn't blink on every
// tray click.
let _ = win.eval(
"if (document.body && document.body.childElementCount === 0) { location.reload(); }",
);
}
}
"open_studio" => {
frontend/src/components/BootstrapSplash.jsx
+7 -1
View File
@@ -39,7 +39,13 @@ function detectHints(message, logs) {
const hints = [];
const all = (message || '') + '\n' + logs.map(l => l.line).join('\n');
if (/README\.md/i.test(all)) hints.push('README.md was missing from the bundle. This is now auto-fixed — retry should work.');
if (/uv.*download|uv.*install/i.test(all) && /timeout|connection/i.test(all)) hints.push('Network timeout downloading uv. Check your internet connection or try the China mirror.');
// python-build-standalone download failure (issue #57, #60): user's network
// can't reach the github.com release. We auto-retry with a system-Python
// fallback in bootstrap.rs, but if that also fails the user needs an actionable next step.
if (/python-build-standalone|managed-python download failed/i.test(all)) {
hints.push('Network couldn\'t reach the Python download (github.com release). Switch your region in Settings → Network (China / Russia / Restricted route through a mirror), or install Python 3.11+ system-wide so the app uses that instead.');
}
if (/uv.*download|uv.*install/i.test(all) && /timeout|connection/i.test(all)) hints.push('Network timeout downloading uv. Check your internet connection or try a different region in Settings → Network.');
if (/uv sync failed/i.test(all)) hints.push('Dependency install failed. "Clean & Retry" will delete the cached venv and start fresh.');
if (/hatchling|build_editable/i.test(all)) hints.push('Python build backend error. "Clean & Retry" removes the broken venv so it rebuilds from scratch.');
if (/ffmpeg/i.test(all) && /download|timeout/i.test(all)) hints.push('ffmpeg download failed. This is non-fatal — retry or install ffmpeg manually.');
frontend/src/components/DemoPresetGrid.css
+109
View File
@@ -0,0 +1,109 @@
/* Demo preset grid — empty state of the Voice Design tab.
7 cards in a responsive grid; cards self-size on column count. */
.demo-preset-grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(220px, 1fr));
gap: 10px;
margin-bottom: 12px;
}
.demo-preset-card {
display: flex;
flex-direction: column;
gap: 6px;
padding: 12px;
border-radius: 10px;
border: 1px solid var(--color-border, rgba(255, 255, 255, 0.08));
background: rgba(255, 255, 255, 0.02);
transition: border-color 120ms ease, background 120ms ease;
}
.demo-preset-card:hover {
border-color: rgba(243, 165, 182, 0.35);
background: rgba(255, 255, 255, 0.04);
}
.demo-preset-card__head {
display: inline-flex;
align-items: center;
gap: 6px;
}
.demo-preset-card__icon {
font-size: 16px;
line-height: 1;
}
.demo-preset-card__name {
font-size: 13px;
font-weight: 700;
color: var(--color-fg, currentColor);
}
.demo-preset-card__desc {
margin: 0;
font-size: 11px;
line-height: 1.35;
color: var(--color-fg-muted, #b0aaa1);
}
.demo-preset-card__instruct {
font-family: var(--font-mono, ui-monospace, SFMono-Regular, monospace);
font-size: 10px;
color: var(--color-fg-subtle, #928374);
background: rgba(0, 0, 0, 0.22);
padding: 2px 6px;
border-radius: 4px;
align-self: flex-start;
max-width: 100%;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.demo-preset-card__actions {
display: flex;
gap: 6px;
margin-top: auto;
padding-top: 4px;
}
.demo-preset-card__preview,
.demo-preset-card__use {
flex: 1;
display: inline-flex;
align-items: center;
justify-content: center;
gap: 4px;
padding: 5px 8px;
font-size: 11px;
font-weight: 600;
border-radius: 6px;
border: 1px solid var(--color-border, rgba(255, 255, 255, 0.1));
background: transparent;
color: var(--color-fg, currentColor);
cursor: pointer;
transition: background 100ms ease, border-color 100ms ease;
}
.demo-preset-card__preview:hover,
.demo-preset-card__use:hover {
background: rgba(255, 255, 255, 0.05);
border-color: rgba(255, 255, 255, 0.2);
}
.demo-preset-card__preview[aria-pressed="true"] {
background: rgba(243, 165, 182, 0.18);
border-color: rgba(243, 165, 182, 0.4);
color: #fff9ef;
}
.demo-preset-card__use {
background: rgba(243, 165, 182, 0.12);
border-color: rgba(243, 165, 182, 0.3);
}
.demo-preset-card__use:hover {
background: rgba(243, 165, 182, 0.22);
}
frontend/src/components/DemoPresetGrid.jsx
+102
View File
@@ -0,0 +1,102 @@
/**
* DemoPresetGrid — the empty-state of the Voice Design tab.
*
* Renders the curated 7-card grid of demo voice designs (see
* backend/core/personalities.py entries with `is_demo: true`). Each card:
* • Title + icon + 1-line description
* • ▶ Preview button (plays pre-rendered WAV from /demo_audio, no model
* load required — works offline before any engine is installed)
* • Use this design → calls `onUse(preset)` which pre-fills text + sliders
*
* Only one preview plays at a time. Mounting/unmounting the audio element
* cancels any in-flight playback so navigating away mid-preview is silent.
*/
import { useEffect, useRef, useState } from 'react';
import { Play, Pause } from 'lucide-react';
import { useTranslation } from 'react-i18next';
import { API } from '../api/client';
import './DemoPresetGrid.css';
export default function DemoPresetGrid({ presets, onUse }) {
const { t } = useTranslation();
const [playingId, setPlayingId] = useState(null);
const audioRef = useRef(null);
// Stop playback on unmount so leaving the Design tab mid-preview goes
// silent immediately.
useEffect(() => {
const audio = audioRef.current;
return () => {
if (audio) {
audio.pause();
audio.currentTime = 0;
}
};
}, []);
const handlePreview = (preset) => {
const audio = audioRef.current;
if (!audio) return;
if (playingId === preset.id) {
audio.pause();
setPlayingId(null);
return;
}
audio.src = `${API}${preset.preview_url}`;
audio.currentTime = 0;
audio.play()
.then(() => setPlayingId(preset.id))
.catch((e) => {
// Most common failure: WAV missing on disk (someone deleted it or
// build_demos.sh hasn't been run). Fall back gracefully — the card
// still works for "Use this design".
console.warn('Preview playback failed:', e);
setPlayingId(null);
});
};
return (
<div className="demo-preset-grid">
{/* Single audio element shared across cards — keeps the "only one
plays at a time" invariant without per-card state coordination. */}
<audio
ref={audioRef}
onEnded={() => setPlayingId(null)}
preload="none"
/>
{presets.map((p) => {
const isPlaying = playingId === p.id;
return (
<div key={p.id} className="demo-preset-card">
<div className="demo-preset-card__head">
<span className="demo-preset-card__icon" aria-hidden>{p.icon}</span>
<span className="demo-preset-card__name">{p.name}</span>
</div>
<p className="demo-preset-card__desc">{p.description}</p>
<code className="demo-preset-card__instruct">{p.instruct}</code>
<div className="demo-preset-card__actions">
<button
type="button"
className="demo-preset-card__preview"
onClick={() => handlePreview(p)}
aria-label={isPlaying ? `Pause ${p.name}` : `Preview ${p.name}`}
aria-pressed={isPlaying}
>
{isPlaying ? <Pause size={12} /> : <Play size={12} />}
{isPlaying ? t('demo.preset_stop') : t('demo.preset_preview')}
</button>
<button
type="button"
className="demo-preset-card__use"
onClick={() => onUse(p)}
aria-label={`Use ${p.name} design`}
>
{t('demo.preset_use')}
</button>
</div>
</div>
);
})}
</div>
);
}
frontend/src/components/DictationDemo.css
+164
View File
@@ -0,0 +1,164 @@
/* Dictation demo — guided walkthrough panel.
Used in Settings → Capture & Dictation and SetupWizard step 4. */
.dictation-demo {
display: flex;
flex-direction: column;
gap: 10px;
padding: 14px;
border-radius: 10px;
border: 1px solid var(--color-border, rgba(255, 255, 255, 0.08));
background: rgba(255, 255, 255, 0.02);
margin-bottom: 16px;
}
.dictation-demo--embedded {
/* When embedded inside Settings panel: drop border, let parent provide it */
border: 0;
background: transparent;
padding: 0;
margin-bottom: 12px;
}
.dictation-demo__head {
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
flex-wrap: wrap;
}
.dictation-demo__title {
display: inline-flex;
align-items: center;
gap: 6px;
margin: 0;
font-size: 13px;
font-weight: 700;
color: var(--color-fg, currentColor);
}
.dictation-demo__status {
display: inline-flex;
align-items: center;
gap: 6px;
font-size: 11px;
padding: 3px 8px;
border-radius: 999px;
border: 1px solid transparent;
}
.dictation-demo__status code {
font-family: var(--font-mono, ui-monospace, SFMono-Regular, monospace);
font-size: 10px;
padding: 1px 4px;
background: rgba(0, 0, 0, 0.3);
border-radius: 3px;
}
.dictation-demo__status--ok {
background: rgba(152, 151, 26, 0.12);
border-color: rgba(152, 151, 26, 0.35);
color: #b8bb26;
}
.dictation-demo__status--pending {
background: rgba(215, 153, 33, 0.10);
border-color: rgba(215, 153, 33, 0.30);
color: #fabd2f;
}
.dictation-demo__status--warn {
background: rgba(204, 36, 29, 0.10);
border-color: rgba(204, 36, 29, 0.30);
color: #fb4934;
}
.dictation-demo__lede {
margin: 0;
font-size: 11px;
line-height: 1.45;
color: var(--color-fg-muted, #b0aaa1);
}
.dictation-demo__scripts {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(260px, 1fr));
gap: 10px;
}
.dictation-demo__card {
display: flex;
flex-direction: column;
gap: 6px;
padding: 10px 12px;
border-radius: 8px;
border: 1px solid var(--color-border, rgba(255, 255, 255, 0.08));
background: rgba(0, 0, 0, 0.15);
}
.dictation-demo__card-head {
display: flex;
align-items: center;
gap: 8px;
font-size: 10px;
color: var(--color-fg-muted, #928374);
}
.dictation-demo__lang {
font-family: var(--font-mono, ui-monospace, monospace);
font-size: 9px;
padding: 1px 5px;
border-radius: 3px;
background: rgba(255, 255, 255, 0.06);
text-transform: uppercase;
letter-spacing: 0.04em;
}
.dictation-demo__card-label {
font-weight: 600;
font-size: 11px;
color: var(--color-fg, currentColor);
text-transform: none;
}
.dictation-demo__script {
margin: 0;
padding: 6px 8px;
font-size: 11.5px;
line-height: 1.45;
border-left: 2px solid rgba(243, 165, 182, 0.4);
background: rgba(255, 255, 255, 0.02);
color: var(--color-fg, currentColor);
font-style: italic;
}
.dictation-demo__card-actions {
display: flex;
gap: 6px;
margin-top: 2px;
}
.dictation-demo__result {
display: flex;
align-items: flex-start;
gap: 6px;
font-size: 11px;
padding: 6px 8px;
border-radius: 6px;
line-height: 1.4;
}
.dictation-demo__result--ok {
color: #b8bb26;
background: rgba(152, 151, 26, 0.08);
border: 1px solid rgba(152, 151, 26, 0.25);
}
.dictation-demo__result--ok em { font-style: normal; }
.dictation-demo__result--fail {
color: #fb4934;
background: rgba(204, 36, 29, 0.08);
border: 1px solid rgba(204, 36, 29, 0.25);
}
frontend/src/components/DictationDemo.jsx
+243
View File
@@ -0,0 +1,243 @@
/**
* DictationDemo — guided walkthrough for the real-time dictation feature.
*
* What this surfaces:
* 1. Active hotkey display (read from the dictation_shortcut Tauri command).
* 2. Three script cards — short utterances the user can read aloud OR
* replay from a bundled WAV. The replay path posts the bundled audio
* to POST /transcribe and renders the recognized text below the card
* so dictation can be demoed even when the user hasn't granted mic
* permission yet, or is on a headless / VM / CI box.
* 3. Hotkey verification status — subscribes to the `tray-dictate` and
* `tray-dictate-stop` Tauri events so we can show "verified" the
* moment the user presses the shortcut for the first time.
*
* Cross-platform: the replay path uses the existing backend transcribe
* endpoint and works identically on macOS / Windows / Linux. The hotkey
* verification path requires Tauri (gracefully no-ops in the web UI).
*
* Where this is mounted:
* - Settings → Capture & Dictation (above HotkeyTab) — always available
* - SetupWizard step 4 — first-run onboarding
*/
import { useEffect, useRef, useState } from 'react';
import { Play, Pause, Keyboard, Mic, CheckCircle2, AlertTriangle } from 'lucide-react';
import { useTranslation } from 'react-i18next';
import { API } from '../api/client';
import { Button } from '../ui';
import './DictationDemo.css';
const SCRIPTS = [
{
id: 'en_conversational',
label: 'Conversational',
language: 'English',
text: 'Schedule a meeting with Pat for Tuesday at three PM and remind me to bring the quarterly report.',
wav: '/demo_audio/dictation/en_conversational.wav',
},
{
id: 'en_technical',
label: 'Technical vocabulary',
language: 'English',
text: 'Patch the WebGPU shader in renderer.tsx, then bump pnpm to nine point fifteen and rerun the Vitest suite.',
wav: '/demo_audio/dictation/en_technical.wav',
},
{
id: 'fr_reservation',
label: 'Non-English (French)',
language: 'French',
text: 'Bonjour, je voudrais réserver une table pour deux personnes à vingt heures.',
wav: '/demo_audio/dictation/fr_reservation.wav',
},
];
function isTauri() {
return typeof window !== 'undefined' && '__TAURI_INTERNALS__' in window;
}
export default function DictationDemo({ embedded = false }) {
const { t } = useTranslation();
const [shortcut, setShortcut] = useState('');
const [hotkeyState, setHotkeyState] = useState('unknown'); // unknown | registered | verified
const [playingId, setPlayingId] = useState(null);
const [transcripts, setTranscripts] = useState({}); // {scriptId: {state, text, error}}
const audioRef = useRef(null);
// Read the registered hotkey on mount.
useEffect(() => {
if (!isTauri()) return;
let cancelled = false;
(async () => {
try {
const { invoke } = await import('@tauri-apps/api/core');
const v = await invoke('get_dictation_shortcut');
if (!cancelled) {
setShortcut(v || '');
setHotkeyState(v ? 'registered' : 'unknown');
}
} catch {
if (!cancelled) setHotkeyState('unknown');
}
})();
return () => { cancelled = true; };
}, []);
// Subscribe to dictation events: the moment the user presses their
// hotkey while this panel is mounted, flip to verified.
useEffect(() => {
if (!isTauri()) return;
let unlistenStart, unlistenStop;
(async () => {
try {
const { listen } = await import('@tauri-apps/api/event');
unlistenStart = await listen('tray-dictate', () => {
setHotkeyState('verified');
});
unlistenStop = await listen('tray-dictate-stop', () => {
setHotkeyState('verified');
});
} catch {
// Tauri event API unavailable — leave state alone.
}
})();
return () => {
try { unlistenStart && unlistenStart(); } catch { /* noop */ }
try { unlistenStop && unlistenStop(); } catch { /* noop */ }
};
}, []);
const togglePlay = (script) => {
const audio = audioRef.current;
if (!audio) return;
if (playingId === script.id) {
audio.pause();
setPlayingId(null);
return;
}
audio.src = `${API}${script.wav}`;
audio.currentTime = 0;
audio.play()
.then(() => setPlayingId(script.id))
.catch((e) => {
console.warn('Sample playback failed:', e);
setPlayingId(null);
});
};
// Replay path: fetch the bundled WAV, post it to the transcribe endpoint,
// render what the engine heard. Demonstrates the full dictation pipeline
// without requiring mic permission or a hotkey press.
const replay = async (script) => {
setTranscripts((prev) => ({
...prev,
[script.id]: { state: 'loading', text: '', error: '' },
}));
try {
const wavRes = await fetch(`${API}${script.wav}`);
if (!wavRes.ok) throw new Error(`Could not fetch sample: ${wavRes.status}`);
const blob = await wavRes.blob();
const fd = new FormData();
fd.append('audio', blob, `${script.id}.wav`);
const tRes = await fetch(`${API}/transcribe`, { method: 'POST', body: fd });
if (!tRes.ok) {
const errBody = await tRes.text().catch(() => '');
throw new Error(`Transcribe failed (${tRes.status}): ${errBody.slice(0, 120)}`);
}
const json = await tRes.json();
setTranscripts((prev) => ({
...prev,
[script.id]: { state: 'ok', text: json.text || '', error: '' },
}));
} catch (e) {
setTranscripts((prev) => ({
...prev,
[script.id]: { state: 'fail', text: '', error: e?.message || String(e) },
}));
}
};
const statusBadge = (() => {
switch (hotkeyState) {
case 'verified':
return (
<span className="dictation-demo__status dictation-demo__status--ok">
<CheckCircle2 size={12} /> {t('demo.dictation_status_ok')}
</span>
);
case 'registered':
return (
<span className="dictation-demo__status dictation-demo__status--pending">
<Keyboard size={12} /> {t('demo.dictation_status_pending')} <code>{shortcut}</code>
</span>
);
default:
return (
<span className="dictation-demo__status dictation-demo__status--warn">
<AlertTriangle size={12} /> {t('demo.dictation_status_warn')}
</span>
);
}
})();
return (
<section className={`dictation-demo ${embedded ? 'dictation-demo--embedded' : ''}`}>
<header className="dictation-demo__head">
<h3 className="dictation-demo__title">
<Mic size={14} /> {t('demo.dictation_title')}
</h3>
{statusBadge}
</header>
<p className="dictation-demo__lede">{t('demo.dictation_lede')}</p>
<audio ref={audioRef} onEnded={() => setPlayingId(null)} preload="none" />
<div className="dictation-demo__scripts">
{SCRIPTS.map((s) => {
const isPlaying = playingId === s.id;
const tx = transcripts[s.id] || {};
return (
<div key={s.id} className="dictation-demo__card">
<div className="dictation-demo__card-head">
<span className="dictation-demo__lang">{s.language}</span>
<span className="dictation-demo__card-label">{s.label}</span>
</div>
<blockquote className="dictation-demo__script">{s.text}</blockquote>
<div className="dictation-demo__card-actions">
<Button
size="sm"
variant="subtle"
onClick={() => togglePlay(s)}
leading={isPlaying ? <Pause size={11} /> : <Play size={11} />}
aria-label={isPlaying ? `Pause ${s.label}` : `Hear ${s.label}`}
>
{isPlaying ? t('demo.dictation_stop') : t('demo.dictation_hear')}
</Button>
<Button
size="sm"
variant="subtle"
onClick={() => replay(s)}
loading={tx.state === 'loading'}
leading={tx.state !== 'loading' && <Mic size={11} />}
aria-label={`Replay ${s.label} through transcriber`}
>
{tx.state === 'loading' ? t('demo.dictation_transcribing') : t('demo.dictation_replay')}
</Button>
</div>
{tx.state === 'ok' && (
<div className="dictation-demo__result dictation-demo__result--ok">
<CheckCircle2 size={11} /> <em>{tx.text}</em>
</div>
)}
{tx.state === 'fail' && (
<div className="dictation-demo__result dictation-demo__result--fail">
<AlertTriangle size={11} /> {tx.error}
</div>
)}
</div>
);
})}
</div>
</section>
);
}
frontend/src/components/DubSegmentRow.css
+73 -11
View File
@@ -17,11 +17,54 @@
background: rgba(211, 134, 155, 0.05) !important;
}
/* Row containing the current media playhead — distinct from segment-active
(which fires during dub *generation*) so users can tell at a glance
where the video is playing while editing. */
.segment-row.segment-playing {
background: rgba(131, 165, 152, 0.10) !important;
box-shadow: inset 2px 0 0 #83a598;
}
.segment-row.segment-playing.segment-active {
/* Generation wins visually when both are true. */
box-shadow: inset 2px 0 0 #d3869b;
}
/* Per-cell widths are inherited from the shared grid template defined
in DubSegmentTable.css (--seg-grid-cols). Cells here only control
intra-cell layout: stacking, font, overflow. */
.seg-check {
width: 14px; flex-shrink: 0; margin-right: 1px; cursor: pointer;
cursor: pointer;
justify-self: center;
}
.seg-time {
width: 46px; flex-shrink: 0; display: flex; flex-direction: column;
min-width: 0; overflow: hidden;
display: flex; flex-direction: column;
font-variant-numeric: tabular-nums;
}
.seg-time-row {
display: flex; align-items: baseline; gap: 2px;
min-width: 0;
}
.seg-time-input {
background: transparent;
border: none;
padding: 0;
font: inherit;
font-size: 0.66rem;
color: var(--chrome-fg);
width: 36px;
text-align: right;
min-width: 0;
}
.seg-time-input:focus {
outline: 1px solid var(--chrome-accent);
outline-offset: 1px;
background: var(--chrome-hover-bg);
}
.seg-time-sep { color: var(--chrome-fg-muted); }
.seg-time-end {
color: var(--chrome-fg-muted);
font-size: 0.62rem;
}
.seg-sync-badge {
font-size: 0.48rem; margin-top: 1px;
@@ -35,12 +78,26 @@
font-size: 0.52rem; margin-left: 1px;
}
.seg-speaker-input {
width: 40px; flex-shrink: 0; font-size: 0.52rem; color: #a89984;
padding: 1px 2px; text-align: center;
width: 100%;
min-width: 0;
background: transparent;
border: none;
padding: 0;
font-size: 0.66rem;
color: var(--chrome-fg-muted);
text-align: left;
overflow: hidden;
text-overflow: ellipsis;
}
.seg-speaker-input:focus {
outline: 1px solid var(--chrome-accent);
outline-offset: 1px;
background: var(--chrome-hover-bg);
color: var(--chrome-fg);
}
.seg-text-col {
flex: 1 1 0%; display: flex; flex-direction: column; gap: 1px;
min-width: 80px; overflow: hidden;
display: flex; flex-direction: column; gap: 1px;
min-width: 0; overflow: hidden;
}
.seg-text-col .segment-input {
width: 100%; min-width: 0;
@@ -63,17 +120,22 @@
cursor: pointer; padding: 0; font-size: 0.52rem;
}
.seg-lang-select {
width: 38px; flex-shrink: 0; font-size: 0.48rem; padding: 1px 1px;
width: 100%; min-width: 0; font-size: 0.6rem; padding: 1px 2px;
}
.seg-profile-select {
width: 56px; flex-shrink: 0; font-size: 0.52rem; padding: 1px 2px;
width: 100%; min-width: 0; font-size: 0.6rem; padding: 1px 4px;
overflow: hidden; text-overflow: ellipsis;
}
.seg-gain-slider {
width: 36px !important; max-width: 36px; flex-shrink: 0; flex-grow: 0;
height: 3px; padding: 0; margin: 0;
width: 100%; min-width: 0; padding: 0; margin: 0;
height: 3px;
}
.seg-actions {
display: flex; gap: 1px; width: 38px; flex-shrink: 0;
display: flex; gap: 2px; justify-content: flex-end;
min-width: 0;
}
.seg-actions > button {
width: 22px; height: 22px;
display: inline-flex; align-items: center; justify-content: center;
}
frontend/src/components/DubSegmentRow.jsx
+54 -17
View File
@@ -12,8 +12,8 @@ import './DubSegmentRow.css';
const CHAR_BUDGET_RATIO = 1.3;
const SENTENCE_END = /[.!?。！？]/;
function rowClass(isActive, isDone, selected) {
return `segment-row${isActive ? ' segment-active' : ''}${isDone ? ' segment-done' : ''}${selected ? ' segment-selected' : ''}`;
function rowClass(isActive, isDone, selected, isPlaying) {
return `segment-row${isActive ? ' segment-active' : ''}${isDone ? ' segment-done' : ''}${selected ? ' segment-selected' : ''}${isPlaying ? ' segment-playing' : ''}`;
}
// Best split point for the Scissors menu when the user hasn't placed a cursor —
@@ -46,7 +46,7 @@ function parseTime(s) {
}
function DubSegmentRow({
seg, idx, style, disabled, isActive, isDone, previewLoading, selected,
seg, idx, style, disabled, isActive, isDone, isPlaying, previewLoading, selected,
profiles, speakerClones, onEditField, onDelete, onRestore, onPreview, onSelect, onSplit, onMerge, canMerge,
onDirect, onSeek,
}) {
@@ -57,14 +57,50 @@ function DubSegmentRow({
const lastCursorRef = useRef(null);
const speakerOptions = speakerClones ? Object.keys(speakerClones) : [];
const speakerListId = `seg-speakers-${seg.id}`;
const syncColor = seg.sync_ratio === undefined ? null
: (seg.sync_ratio >= 0.95 && seg.sync_ratio <= 1.05) ? '#b8bb26'
: seg.sync_ratio > 1.25 ? '#fb4934'
: '#fabd2f';
const SyncIcon = seg.sync_ratio === undefined ? null
: (seg.sync_ratio >= 0.95 && seg.sync_ratio <= 1.05) ? CheckCircle
: seg.sync_ratio > 1.25 ? AlertCircle
: Circle;
// Truthful per-segment fit badge. The backend's new fit_status object is
// the source of truth — describes exactly what the mix loop did:
// "fits" → audio fit cleanly inside the slot (or its gap).
// "overflows" → concise mode hard-trimmed +Ns past slot; user
// should shorten the text for a cleaner result.
// "video_stretched" → stretch_video mode lengthened the source clip by
// the stretch ratio to fit natural-rate audio.
// We fall back to the legacy sync_ratio bucketing only when fit_status is
// missing (older jobs / partial-regen with no done event yet) — and even
// then we now display the *raw* ratio so it stops claiming 100% when the
// audio was actually compressed at synthesis time.
const fitStatus = seg.fit_status && typeof seg.fit_status === 'object' ? seg.fit_status : null;
let fitBadge = null;
if (fitStatus) {
if (fitStatus.status === 'fits') {
fitBadge = { color: '#b8bb26', Icon: CheckCircle, label: 'Fits', title: 'Natural-rate audio fit inside the slot.' };
} else if (fitStatus.status === 'overflows') {
const over = fitStatus.overflow_s || 0;
fitBadge = {
color: over > 0.5 ? '#fb4934' : '#fabd2f',
Icon: AlertCircle,
label: `Overflows +${over.toFixed(2)}s`,
title: `Translated text was longer than the original slot by ${over.toFixed(2)}s. The audio was hard-trimmed; shorten the text or switch Timing to "Stretch Video".`,
};
} else if (fitStatus.status === 'video_stretched') {
const r = fitStatus.stretch_ratio || 1.0;
fitBadge = {
color: r > 1.18 ? '#fb4934' : r > 1.05 ? '#fabd2f' : '#83a598',
Icon: Circle,
label: `Video ${r.toFixed(2)}×`,
title: `Stretch Video mode: this segment's video was slowed to ${r.toFixed(2)}× to fit the natural dub audio.`,
};
}
} else if (seg.sync_ratio !== undefined) {
const r = seg.sync_ratio;
if (r > 1.25) {
fitBadge = { color: '#fb4934', Icon: AlertCircle, label: `${Math.round(r * 100)}%`, title: `TTS audio is ${Math.round(r * 100)}% of the slot — heavily compressed.` };
} else if (r >= 0.95 && r <= 1.05) {
fitBadge = { color: '#b8bb26', Icon: CheckCircle, label: 'Fits', title: 'Audio fit inside the slot.' };
} else {
fitBadge = { color: '#fabd2f', Icon: Circle, label: `${Math.round(r * 100)}%`, title: `TTS audio is ${Math.round(r * 100)}% of the slot.` };
}
}
const overBudget = seg.text_original
&& seg.text.length > Math.ceil(seg.text_original.length * CHAR_BUDGET_RATIO);
@@ -96,7 +132,7 @@ function DubSegmentRow({
};
return (
<div style={style} className={rowClass(isActive, isDone, selected)} onClick={handleRowClick}>
<div style={style} className={rowClass(isActive, isDone, selected, isPlaying)} onClick={handleRowClick}>
<input
type="checkbox"
checked={!!selected}
@@ -142,13 +178,13 @@ function DubSegmentRow({
</span>
)}
</span>
{SyncIcon && (
{fitBadge && (
<span
className="seg-sync-badge"
style={{ color: syncColor }}
title={`Generated audio is ${Math.round(seg.sync_ratio * 100)}% the duration of original`}
style={{ color: fitBadge.color }}
title={fitBadge.title}
>
<SyncIcon size={8} /> Sync: {Math.round(seg.sync_ratio * 100)}%
<fitBadge.Icon size={8} /> {fitBadge.label}
</span>
)}
{seg.rate_ratio != null && Math.abs(seg.rate_ratio - 1.0) > 0.03 && (
@@ -163,7 +199,7 @@ function DubSegmentRow({
</span>
<input
className="input-base seg-speaker-input"
className="seg-speaker-input"
value={seg.speaker_id || ''}
onChange={(e) => onEditField(seg.id, 'speaker_id', e.target.value)}
onClick={(e) => e.stopPropagation()}
@@ -350,6 +386,7 @@ export default memo(DubSegmentRow, (prev, next) => (
prev.disabled === next.disabled &&
prev.isActive === next.isActive &&
prev.isDone === next.isDone &&
prev.isPlaying === next.isPlaying &&
prev.previewLoading === next.previewLoading &&
prev.onDirect === next.onDirect &&
prev.onSeek === next.onSeek &&
frontend/src/components/DubSegmentTable.css
+33 -3
View File
@@ -1,14 +1,44 @@
/* Table body specific styles — chrome comes from ui/Table. */
/* Single grid template shared by both header and data rows so columns
line up. Order: [select][time][spkr][text][lang][voice][vol][actions].
Both `.dub-segment-table__header` and `.segment-row` override their
default flex layout to use this template — eliminates the column-
drift bug where the time pill bled into the text column. */
.segment-table {
--seg-grid-cols: 18px 64px 70px minmax(0, 1fr) 44px 60px 40px 44px;
}
.dub-segment-table__header,
.segment-table .segment-row {
display: grid !important;
grid-template-columns: var(--seg-grid-cols);
align-items: center;
gap: 6px;
}
.dub-segment-table__header {
padding: 3px 4px !important;
gap: 3px !important;
padding: 4px 6px !important;
/* Header cells normally carry explicit widths from Table.jsx — neutralise
them so the grid template wins. */
}
.dub-segment-table__header .ui-table-header__cell {
width: auto !important;
flex: none !important;
min-width: 0;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.segment-table .segment-row {
padding: 4px 6px;
min-height: 30px;
}
.dub-segment-table__body { flex: 1; min-height: 0; }
.dub-segment-table__select-all {
width: 18px;
display: inline-flex;
align-items: center;
justify-content: center;
frontend/src/components/DubSegmentTable.jsx
+30 -3
View File
@@ -2,6 +2,7 @@ import React, { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useSta
import { List } from 'react-window';
import DubSegmentRow from './DubSegmentRow';
import { Table, Select } from '../ui';
import { useAppStore } from '../store';
import './DubSegmentTable.css';
const BASE_ROW_HEIGHT = 26;
@@ -25,6 +26,15 @@ export default function DubSegmentTable({
const disabled = dubStep === 'generating' || dubStep === 'stopping';
const [query, setQuery] = useState('');
const [speakerFilter, setSpeakerFilter] = useState('');
// ID of the segment under the playhead. Subscribed via selector so the
// table re-renders only when the playing segment changes, not on every
// timeupdate tick.
const currentSegId = useAppStore(s => s.dubCurrentSegId);
// Imperative handle for react-window v2 so we can auto-scroll the row
// containing the playhead into view. (The scroll effect itself lives
// below the `filtered` memo so it can depend on it without TDZ.)
const listRef = useRef(null);
// react-window v2 needs a concrete height prop — CSS 100 % doesn't cut it.
// Measure the body container and pass its height explicitly so the list
@@ -59,6 +69,20 @@ export default function DubSegmentTable({
});
}, [segments, query, speakerFilter]);
// Auto-scroll the playing row into view as the playhead advances. Uses
// align='smart' so an already-visible row doesn't trigger a jump.
// Placed after `filtered` so it can depend on it without TDZ.
useEffect(() => {
if (!currentSegId || !listRef.current) return;
const filteredIdx = filtered.findIndex(s => s.id === currentSegId);
if (filteredIdx < 0) return;
try {
listRef.current.scrollToRow({
index: filteredIdx, align: 'smart', behavior: 'smooth',
});
} catch (_) { /* react-window may not be ready yet */ }
}, [currentSegId, filtered]);
const rowHeight = useCallback((index) => {
const s = filtered[index];
if (!s) return BASE_ROW_HEIGHT;
@@ -68,21 +92,23 @@ export default function DubSegmentTable({
const rowProps = useMemo(() => ({
filtered, profiles, speakerClones, disabled, dubStep, dubProgress, previewLoadingId,
selectedIds, onSelect, onEditField, onDelete, onRestore, onPreview, onSplit, onMerge, onDirect, onSeek,
segments,
segments, currentSegId,
}), [filtered, profiles, speakerClones, disabled, dubStep, dubProgress, previewLoadingId,
selectedIds, onSelect, onEditField, onDelete, onRestore, onPreview, onSplit, onMerge, onDirect, onSeek, segments]);
selectedIds, onSelect, onEditField, onDelete, onRestore, onPreview, onSplit, onMerge, onDirect, onSeek, segments, currentSegId]);
const Row = useCallback(({ index, style, filtered: fl, profiles: profs, speakerClones: clones, disabled: dis, dubProgress: prog, dubStep: step, previewLoadingId: previewId, selectedIds: sel, onSelect: pick, onEditField: edit, onDelete: del, onRestore: rest, onPreview: prev, onSplit: split, onMerge: merge, onDirect: direct, onSeek: seek, segments: segs }) => {
const Row = useCallback(({ index, style, filtered: fl, profiles: profs, speakerClones: clones, disabled: dis, dubProgress: prog, dubStep: step, previewLoadingId: previewId, selectedIds: sel, onSelect: pick, onEditField: edit, onDelete: del, onRestore: rest, onPreview: prev, onSplit: split, onMerge: merge, onDirect: direct, onSeek: seek, segments: segs, currentSegId: curId }) => {
const seg = fl[index];
if (!seg) return null;
const absoluteIndex = segs.indexOf(seg);
const isActive = (step === 'generating' || step === 'stopping') && prog.current === absoluteIndex + 1;
const isDone = (step === 'generating' || step === 'stopping') && prog.current > absoluteIndex + 1;
const isPlaying = curId === seg.id;
const canMerge = index < fl.length - 1;
return (
<DubSegmentRow
seg={seg} idx={index} style={style}
disabled={dis} isActive={isActive} isDone={isDone}
isPlaying={isPlaying}
previewLoading={previewId === seg.id}
selected={sel && sel.has(seg.id)}
canMerge={canMerge}
@@ -142,6 +168,7 @@ export default function DubSegmentTable({
<div className="dub-segment-table__body" ref={bodyRef}>
{bodyHeight > 0 && (
<List
listRef={listRef}
rowCount={filtered.length}
rowHeight={rowHeight}
rowComponent={Row}
frontend/src/components/DubbingDemo.css
+175
View File
@@ -0,0 +1,175 @@
/* Synthetic dubbing demo — side-by-side player. */
.dubbing-demo {
display: flex;
flex-direction: column;
gap: 10px;
padding: 14px;
border-radius: 10px;
border: 1px solid var(--color-border, rgba(255, 255, 255, 0.08));
background: rgba(255, 255, 255, 0.02);
}
.dubbing-demo--loading {
font-size: 11px;
color: var(--color-fg-muted, #928374);
padding: 18px;
text-align: center;
}
.dubbing-demo__head {
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
}
.dubbing-demo__title {
display: inline-flex;
align-items: center;
gap: 6px;
font-size: 12px;
font-weight: 700;
color: var(--color-fg, currentColor);
}
.dubbing-demo__head-actions {
display: inline-flex;
align-items: center;
gap: 8px;
}
.dubbing-demo__sync {
display: inline-flex;
align-items: center;
gap: 5px;
font-size: 11px;
color: var(--color-fg-muted, #b0aaa1);
cursor: pointer;
user-select: none;
}
.dubbing-demo__sync input {
accent-color: #f3a5b6;
}
.dubbing-demo__dismiss {
border: 0;
background: transparent;
color: var(--color-fg-muted, #928374);
cursor: pointer;
padding: 2px 4px;
border-radius: 4px;
display: inline-flex;
}
.dubbing-demo__dismiss:hover {
background: rgba(255, 255, 255, 0.05);
color: var(--color-fg, currentColor);
}
.dubbing-demo__players {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 12px;
}
@media (max-width: 720px) {
.dubbing-demo__players { grid-template-columns: 1fr; }
}
.dubbing-demo__pane {
display: flex;
flex-direction: column;
gap: 6px;
}
.dubbing-demo__pane-label {
font-size: 11px;
font-weight: 600;
color: var(--color-fg, currentColor);
}
.dubbing-demo__pane-label span {
font-weight: 400;
color: var(--color-fg-muted, #928374);
font-size: 10px;
margin-left: 4px;
text-transform: uppercase;
letter-spacing: 0.05em;
}
.dubbing-demo__pane video {
width: 100%;
border-radius: 6px;
background: #000;
outline: 1px solid rgba(255, 255, 255, 0.06);
outline-offset: -1px;
}
.dubbing-demo__caption {
margin: 0;
font-size: 10.5px;
line-height: 1.4;
color: var(--color-fg-muted, #b0aaa1);
padding: 4px 6px;
background: rgba(0, 0, 0, 0.18);
border-radius: 4px;
}
.dubbing-demo__picker {
display: flex;
flex-wrap: wrap;
align-items: center;
gap: 6px;
padding-top: 4px;
}
.dubbing-demo__picker-label {
font-size: 10px;
color: var(--color-fg-muted, #928374);
text-transform: uppercase;
letter-spacing: 0.05em;
margin-right: 4px;
}
.dubbing-demo__chip {
font-size: 11px;
padding: 3px 10px;
border-radius: 999px;
border: 1px solid var(--color-border, rgba(255, 255, 255, 0.1));
background: transparent;
color: var(--color-fg-muted, #b0aaa1);
cursor: pointer;
transition: background 100ms ease, border-color 100ms ease, color 100ms ease;
}
.dubbing-demo__chip:hover {
background: rgba(255, 255, 255, 0.04);
color: var(--color-fg, currentColor);
}
.dubbing-demo__chip.is-active {
background: rgba(243, 165, 182, 0.18);
border-color: rgba(243, 165, 182, 0.45);
color: #fff9ef;
}
.dubbing-demo__cta {
align-self: flex-end;
display: inline-flex;
align-items: center;
gap: 6px;
padding: 6px 12px;
font-size: 11px;
font-weight: 600;
border-radius: 6px;
border: 1px solid rgba(243, 165, 182, 0.4);
background: rgba(243, 165, 182, 0.12);
color: var(--color-fg, currentColor);
cursor: pointer;
}
.dubbing-demo__cta:hover {
background: rgba(243, 165, 182, 0.22);
}
frontend/src/components/DubbingDemo.jsx
+179
View File
@@ -0,0 +1,179 @@
/**
* DubbingDemo — side-by-side player for the synthetic dubbing demo.
*
* Reads /demo_audio/demo/dubbing/manifest.json (mounted via FastAPI's
* /demo_audio static route), shows the English source video on the left,
* a language-pickable dubbed variant on the right, and a "Try it with
* your own video" CTA below.
*
* Renders on the DubTab idle state when no project / file is loaded.
* Dismissable via `onDismiss` — the parent passes a setter that hides
* the demo and falls back to the existing drop-zone UI.
*/
import { useEffect, useRef, useState } from 'react';
import { Play, Film, X } from 'lucide-react';
import { useTranslation } from 'react-i18next';
import { API } from '../api/client';
import './DubbingDemo.css';
export default function DubbingDemo({ onDismiss }) {
const { t } = useTranslation();
const [manifest, setManifest] = useState(null);
const [error, setError] = useState(null);
const [pickedCode, setPickedCode] = useState('es');
const [syncPlay, setSyncPlay] = useState(true);
const sourceRef = useRef(null);
const dubbedRef = useRef(null);
useEffect(() => {
let cancelled = false;
fetch(`${API}/demo_audio/demo/dubbing/manifest.json`)
.then(r => {
if (!r.ok) throw new Error(`manifest ${r.status}`);
return r.json();
})
.then(j => { if (!cancelled) setManifest(j); })
.catch(e => { if (!cancelled) setError(e?.message || String(e)); });
return () => { cancelled = true; };
}, []);
// Sync the two players when syncPlay is on — play/pause/seek the
// English source drives the dubbed clone (and vice versa).
useEffect(() => {
if (!syncPlay) return;
const a = sourceRef.current;
const b = dubbedRef.current;
if (!a || !b) return;
const onPlay = (src, dst) => () => {
// Avoid feedback loop — only play target if it's currently paused.
if (dst.paused) {
dst.currentTime = src.currentTime;
dst.play().catch(() => {});
}
};
const onPause = (dst) => () => { if (!dst.paused) dst.pause(); };
const onSeek = (src, dst) => () => { dst.currentTime = src.currentTime; };
const aPlay = onPlay(a, b);
const bPlay = onPlay(b, a);
const aPause = onPause(b);
const bPause = onPause(a);
const aSeek = onSeek(a, b);
const bSeek = onSeek(b, a);
a.addEventListener('play', aPlay);
b.addEventListener('play', bPlay);
a.addEventListener('pause', aPause);
b.addEventListener('pause', bPause);
a.addEventListener('seeked', aSeek);
b.addEventListener('seeked', bSeek);
return () => {
a.removeEventListener('play', aPlay);
b.removeEventListener('play', bPlay);
a.removeEventListener('pause', aPause);
b.removeEventListener('pause', bPause);
a.removeEventListener('seeked', aSeek);
b.removeEventListener('seeked', bSeek);
};
}, [syncPlay, pickedCode]);
if (error) {
return null; // No demo manifest yet — silently fall through to drop zone.
}
if (!manifest) {
return (
<div className="dubbing-demo dubbing-demo--loading">
Loading dubbing demo…
</div>
);
}
const source = manifest.source;
const dubbed = manifest.dubbed?.find(d => d.code === pickedCode) || manifest.dubbed?.[0];
if (!dubbed) return null;
const base = `${API}/demo_audio/demo/dubbing`;
return (
<div className="dubbing-demo">
<header className="dubbing-demo__head">
<div className="dubbing-demo__title">
<Film size={13} /> {t('demo.dubbing_title')}
</div>
<div className="dubbing-demo__head-actions">
<label className="dubbing-demo__sync">
<input
type="checkbox"
checked={syncPlay}
onChange={e => setSyncPlay(e.target.checked)}
/>
{t('demo.dubbing_sync')}
</label>
{onDismiss && (
<button
type="button"
className="dubbing-demo__dismiss"
onClick={onDismiss}
aria-label="Dismiss dubbing demo"
>
<X size={13} />
</button>
)}
</div>
</header>
<div className="dubbing-demo__players">
<div className="dubbing-demo__pane">
<div className="dubbing-demo__pane-label">{source.label} <span>· original</span></div>
<video
ref={sourceRef}
src={`${base}/${source.video}`}
controls
playsInline
preload="metadata"
/>
<p className="dubbing-demo__caption">{source.script}</p>
</div>
<div className="dubbing-demo__pane">
<div className="dubbing-demo__pane-label">
{dubbed.label} <span>· dubbed</span>
</div>
<video
ref={dubbedRef}
src={`${base}/${dubbed.video}`}
controls
playsInline
preload="metadata"
dir={dubbed.dir}
/>
<p className="dubbing-demo__caption" dir={dubbed.dir}>{dubbed.script}</p>
</div>
</div>
<div className="dubbing-demo__picker">
<span className="dubbing-demo__picker-label">{t('demo.dubbing_picker')}</span>
{manifest.dubbed.map(d => (
<button
key={d.code}
type="button"
className={`dubbing-demo__chip ${pickedCode === d.code ? 'is-active' : ''}`}
onClick={() => setPickedCode(d.code)}
>
{d.label}
</button>
))}
</div>
{onDismiss && (
<button
type="button"
className="dubbing-demo__cta"
onClick={onDismiss}
>
<Play size={12} /> {t('demo.dubbing_cta')}
</button>
)}
</div>
);
}
frontend/src/components/EngineCompatibilityMatrix.css
+78
View File
@@ -43,6 +43,9 @@
.engine-matrix__body {
display: flex;
flex-direction: column;
/* Bottom gutter so the last row never sits flush against a footer/nav bar
when the matrix lives inside a scroll container (e.g. SetupWizard step 4). */
padding-bottom: 12px;
}
.engine-matrix__row {
@@ -119,6 +122,54 @@
display: block;
}
/* Disclosure for unavailable-row details: keeps the row a single line until
the user opts in to see the why. Drastically reduces vertical noise on
first-paint of the matrix. */
.engine-matrix__why {
font-size: 11px;
margin-top: 2px;
}
.engine-matrix__why-summary {
cursor: pointer;
color: var(--chrome-fg-muted, #888);
font-size: 11px;
user-select: none;
padding: 1px 0;
list-style: none;
display: inline-flex;
align-items: center;
gap: 4px;
}
.engine-matrix__why-summary:hover {
color: var(--chrome-fg, currentColor);
}
.engine-matrix__why-summary::-webkit-details-marker {
display: none;
}
.engine-matrix__why-summary::before {
content: '▸';
font-size: 9px;
transition: transform 120ms ease;
display: inline-block;
}
.engine-matrix__why[open] .engine-matrix__why-summary::before {
transform: rotate(90deg);
}
.engine-matrix__why-body {
display: flex;
flex-direction: column;
gap: 3px;
margin-top: 4px;
padding-left: 12px;
border-left: 2px solid var(--chrome-border, rgba(255,255,255,0.08));
}
.engine-matrix__chips {
display: inline-flex;
flex-wrap: wrap;
@@ -172,6 +223,33 @@
color: var(--chrome-severity-err, #cc241d);
}
/* Two-line tab label: family name prominent, active engine subdued.
Makes the segmented control read as a control (tap me, navigate)
rather than a status line ("TTS · omnivoice"). */
.engine-matrix__tab-label {
display: inline-flex;
flex-direction: column;
align-items: center;
gap: 0;
line-height: 1.1;
padding: 1px 2px;
}
.engine-matrix__tab-family {
font-size: 12px;
font-weight: 700;
letter-spacing: 0.02em;
}
.engine-matrix__tab-active {
font-size: 9px;
font-family: var(--chrome-font-mono, ui-monospace, monospace);
opacity: 0.65;
text-transform: lowercase;
letter-spacing: 0;
margin-top: 1px;
}
.engine-matrix__empty {
padding: 24px;
text-align: center;
frontend/src/components/EngineCompatibilityMatrix.jsx
+60 -21
View File
@@ -233,7 +233,13 @@ export default function EngineCompatibilityMatrix({
onChange={setActiveFamily}
items={families.map((f) => ({
value: f,
label: `${FAMILY_META[f].label} · ${data[f].active}`,
title: `Active ${FAMILY_META[f].label}: ${data[f].active}`,
label: (
<span className="engine-matrix__tab-label">
<span className="engine-matrix__tab-family">{FAMILY_META[f].label}</span>
<span className="engine-matrix__tab-active">{data[f].active}</span>
</span>
),
}))}
/>
)}
@@ -258,18 +264,32 @@ export default function EngineCompatibilityMatrix({
{isActive && <Badge tone="brand" size="xs">active</Badge>}
</span>
<code className="engine-matrix__id">{b.id}</code>
{!b.available && b.reason && (
<span className="engine-matrix__reason" title={b.reason}>{b.reason}</span>
)}
{b.install_hint && (
{/* For available rows, show install_hint inline (one line — usually
a parenthetical like "(bundled — no extra install needed)").
For unavailable rows, collapse reason + install_hint + last_error
into a single disclosure so unavailable rows don't dwarf the matrix. */}
{b.available && b.install_hint && (
<span className="engine-matrix__hint" title={b.install_hint}>
{b.install_hint}
</span>
)}
{b.last_error && (
<span className="engine-matrix__last-error" data-testid="last-error">
Last error: {b.last_error}
</span>
{!b.available && (b.reason || b.install_hint || b.last_error) && (
<details className="engine-matrix__why">
<summary className="engine-matrix__why-summary">Why unavailable?</summary>
<div className="engine-matrix__why-body">
{b.reason && (
<span className="engine-matrix__reason">{b.reason}</span>
)}
{b.install_hint && b.install_hint !== b.reason && (
<span className="engine-matrix__hint">{b.install_hint}</span>
)}
{b.last_error && b.last_error !== b.reason && (
<span className="engine-matrix__last-error" data-testid="last-error">
Last error: {b.last_error}
</span>
)}
</div>
</details>
)}
</div>
@@ -310,23 +330,42 @@ export default function EngineCompatibilityMatrix({
</Badge>
</div>
{/* Actions: Test engine + optional Use */}
{/* Actions: Test engine + optional Use.
"Test engine" is hidden on unavailable rows by default —
a health check on a known-unavailable engine just confirms
what the matrix already says. Users re-checking after a
manual install can hit "Re-check" inside the disclosure. */}
<div
role="cell"
className="engine-matrix__cell engine-matrix__cell--actions"
style={{ width: 220 }}
>
<Button
size="sm"
variant="subtle"
onClick={() => testHealth(b.id)}
disabled={!!health?.inflight}
loading={!!health?.inflight}
leading={!health?.inflight && <Activity size={11} />}
aria-label={`Test ${b.display_name}`}
>
{health?.inflight ? 'Testing…' : 'Test engine'}
</Button>
{b.available && (
<Button
size="sm"
variant="subtle"
onClick={() => testHealth(b.id)}
disabled={!!health?.inflight}
loading={!!health?.inflight}
leading={!health?.inflight && <Activity size={11} />}
aria-label={`Test ${b.display_name}`}
>
{health?.inflight ? 'Testing…' : 'Test engine'}
</Button>
)}
{!b.available && (
<Button
size="sm"
variant="subtle"
onClick={() => testHealth(b.id)}
disabled={!!health?.inflight}
loading={!!health?.inflight}
leading={!health?.inflight && <RefreshCw size={11} />}
aria-label={`Re-check ${b.display_name}`}
>
{health?.inflight ? 'Re-checking…' : 'Re-check'}
</Button>
)}
{health && !health.inflight && (
<span
className={`engine-matrix__result engine-matrix__result--${health.ok ? 'ok' : 'fail'}`}
frontend/src/components/ReportBugButton.jsx
+120
View File
@@ -0,0 +1,120 @@
/**
* ReportBugButton — opt-in bug reporter via prefilled GitHub Issues URL.
*
* Capability 2 from CLAUDE.md. The user clicks → we build a URL like:
* https://github.com/{owner}/{repo}/issues/new?title=…&body=…&labels=bug
* and open it in their browser. They review what we captured and click
* Submit on github.com if they're happy. We never hold an auth token,
* never POST to GitHub directly, and never bypass the user's review —
* opt-in by construction, no separate consent dialog needed.
*
* What gets captured (no secrets):
* - OS + arch
* - OmniVoice version (Vite injects __APP_VERSION__ at build time)
* - Browser/webview UA
* - Active TTS engine (best-effort fetch)
* - Optional user-typed description
*
* What gets stripped:
* - $HOME path → ~/
* - Anything matching /TOKEN|KEY|SECRET/i in env vars
* - Audio file contents (we don't include them)
*/
import { useState } from 'react';
import { Bug } from 'lucide-react';
import { Button } from '../ui';
import { openExternal } from '../api/external';
import { API } from '../api/client';
const APP_VERSION = (typeof __APP_VERSION__ !== 'undefined' && __APP_VERSION__) || 'unknown';
const ISSUES_URL = 'https://github.com/debpalash/OmniVoice-Studio/issues/new';
function stripHome(s) {
if (!s) return s;
// Best-effort home redaction — works for the most common /Users/<name>/
// and /home/<name>/ paths. We don't know the actual $HOME from JS, so
// pattern-match the prefix.
return String(s)
.replace(/\/Users\/[^/]+/g, '~')
.replace(/\/home\/[^/]+/g, '~')
.replace(/[A-Z]:\\Users\\[^\\]+/g, '~');
}
async function captureContext() {
const lines = [
`**Version:** \`${APP_VERSION}\``,
`**Platform:** \`${navigator?.userAgent || 'unknown'}\``,
];
// Best-effort backend system info — silently skip if backend is down.
try {
const r = await fetch(`${API}/system/info`);
if (r.ok) {
const j = await r.json();
// /system/info exposes `platform` (sys.platform) + `device` (best
// compute device). Map to those — older field names (os/torch_device/
// gpu) never existed on this endpoint, so they silently dropped.
if (j?.platform) lines.push(`**OS:** \`${j.platform}\``);
if (j?.python) lines.push(`**Python:** \`${j.python}\``);
if (j?.device) lines.push(`**Compute device:** \`${stripHome(j.device)}\``);
}
} catch { /* backend probably not up yet */ }
try {
const r = await fetch(`${API}/engines`);
if (r.ok) {
const j = await r.json();
const active = j?.tts?.active;
if (active) lines.push(`**Active TTS engine:** \`${active}\``);
}
} catch { /* noop */ }
return lines.join('\n');
}
export default function ReportBugButton({ size = 'sm', variant = 'subtle', label = 'Report a bug' }) {
const [building, setBuilding] = useState(false);
const handleClick = async () => {
setBuilding(true);
try {
const ctx = await captureContext();
const body = [
'<!-- Click Submit at the bottom of this page to file the issue.',
' Review the auto-captured environment info below and add anything',
' about what you were doing when the bug happened. -->',
'',
'## Describe the bug',
'',
'<!-- e.g. "Synthesize failed in Design mode after picking Narrator personality" -->',
'',
'## Environment',
'',
ctx,
'',
'## What I was doing',
'',
'<!-- step-by-step would help us reproduce -->',
'',
].join('\n');
const url = `${ISSUES_URL}?title=${encodeURIComponent('[Bug] ')}&labels=${encodeURIComponent('bug')}&body=${encodeURIComponent(body)}`;
await openExternal(url);
} finally {
setBuilding(false);
}
};
return (
<Button
size={size}
variant={variant}
onClick={handleClick}
loading={building}
leading={!building && <Bug size={12} />}
title="Opens a prefilled GitHub Issues page in your browser. Nothing is sent until you click Submit."
>
{label}
</Button>
);
}
frontend/src/components/WaveformErrorBoundary.css
+11 -1
View File
@@ -4,7 +4,17 @@
}
.wfm-stack { display: flex; flex-direction: column; gap: 4px; flex: 1; min-height: 0; }
.wfm-video-preview {
flex: 0 0 auto; aspect-ratio: 16 / 9; max-height: 45%;
/* Width-driven: use the full available width and derive height from the
16:9 aspect ratio, capped at 60vh so a very tall parent can't push
the waveform off-screen. Previous `max-height: 45%` was clamping the
height first, which then back-pressured the width (aspect-ratio
preserves shape), leaving empty black space to the right of the
video. For portrait sources the inner <video> uses object-fit:contain
and letterboxes within the 16:9 box. */
flex: 0 0 auto;
width: 100%;
aspect-ratio: 16 / 9;
max-height: 60vh;
background: #000; border-radius: 4px; overflow: hidden;
border: 1px solid rgba(255,255,255,0.05); display: flex;
}
frontend/src/components/WaveformTimeline.jsx
+72 -2
View File
@@ -4,6 +4,7 @@ import RegionsPlugin from 'wavesurfer.js/dist/plugins/regions.esm.js';
import MinimapPlugin from 'wavesurfer.js/dist/plugins/minimap.esm.js';
import TimelinePlugin from 'wavesurfer.js/dist/plugins/timeline.esm.js';
import { Play, Pause, ZoomIn, ZoomOut, SkipBack, Loader, Keyboard } from 'lucide-react';
import { useAppStore } from '../store';
import './WaveformErrorBoundary.css';
const REGION_COLORS = [
@@ -45,17 +46,50 @@ function WaveformTimeline({
const [ready, setReady] = useState(false);
const [loadError, setLoadError] = useState(false);
// Specifically: the source returned a non-media response (typically 404
// HTML). Differentiates from a generic decode failure so the error UI
// can tell the user the file has moved/been deleted, not just "broken".
const [sourceMissing, setSourceMissing] = useState(false);
const [isPlaying, setIsPlaying] = useState(false);
const [currentTime, setCurrentTime] = useState(0);
const [duration, setDuration] = useState(0);
const [zoom, setZoom] = useState(50);
// ── Live "current segment" signal ──────────────────────────────────────────
// Derive which segment contains the playhead and write the id to the store
// *only when it changes*. This keeps DubSegmentTable re-renders bounded to
// segment-crossings instead of the 4-50Hz timeupdate cadence.
const setDubCurrentSegId = useAppStore(s => s.setDubCurrentSegId);
const lastSegIdRef = useRef(null);
useEffect(() => {
if (!segments.length) {
if (lastSegIdRef.current != null) {
lastSegIdRef.current = null;
setDubCurrentSegId(null);
}
return;
}
// Linear scan — fine for ≤200 segments. For longer transcripts we'd
// switch to a binary search; keep it simple until needed.
let hit = null;
for (const s of segments) {
if (currentTime >= s.start && currentTime < s.end) { hit = s.id; break; }
}
if (hit !== lastSegIdRef.current) {
lastSegIdRef.current = hit;
setDubCurrentSegId(hit);
}
}, [currentTime, segments, setDubCurrentSegId]);
// Clear on unmount so a stale id can't outlive the editor.
useEffect(() => () => { setDubCurrentSegId(null); }, [setDubCurrentSegId]);
// ── Core init — only re-runs when src changes ───────────────────────────────
useEffect(() => {
if (!waveContainerRef.current || !audioSrc) return;
setReady(false);
setLoadError(false);
setSourceMissing(false);
setCurrentTime(0);
setDuration(0);
setIsPlaying(false);
@@ -92,6 +126,23 @@ function WaveformTimeline({
videoEl.addEventListener('loadedmetadata', showFirstFrame, { once: true });
// Fallback if loadedmetadata already fired before listener attached (cached).
if (videoEl.readyState >= 1) showFirstFrame();
// Surface media decode failures so future format issues don't
// present as a silent black box. MediaError codes: 1=aborted,
// 2=network, 3=decode, 4=src not supported. 3 = real codec/
// container mismatch (decoder ran but couldn't handle it); 4 =
// server returned a non-media response (most often a 404 HTML
// body when the source video moved or was deleted between
// project save and reload). In either case the rest of the
// pipeline can't do anything useful, so flip into the
// user-facing error fallback instead of staring at a black box.
videoEl.addEventListener('error', () => {
const code = videoEl.error?.code;
if (code === 3 || code === 4) {
console.warn('[WaveformTimeline] video element rejected source', videoSrc, 'code', code);
setSourceMissing(code === 4);
setLoadError(true);
}
}, { once: true });
videoContainerRef.current.appendChild(videoEl);
}
@@ -211,8 +262,20 @@ function WaveformTimeline({
ws.load(undefined, [channelData], audioBuffer.duration);
})
.catch((decodeErr) => {
// HTTP 404 on the companion audio means the source file is
// gone (typically: project loaded after the underlying media
// was moved/deleted). Surface that explicitly — an empty
// waveform fallback would just confuse the user into thinking
// the file is silent. Other decode failures still fall back
// to empty peaks so the media element can still play.
const isMissing = /\bHTTP 404\b/.test(String(decodeErr?.message || decodeErr));
if (isMissing) {
console.warn('Audio source missing (HTTP 404):', audioSrc);
setSourceMissing(true);
setLoadError(true);
return;
}
console.warn('Audio decode fallback failed, loading with empty peaks:', decodeErr);
// Last resort — show flat waveform but keep media element playback working
try {
const emptyPeaks = new Float32Array(1000).fill(0);
ws.load(undefined, [emptyPeaks], mediaEl.duration || 60);
@@ -412,7 +475,14 @@ function WaveformTimeline({
return (
<div className="waveform-timeline">
<div className="wfm-error">
⚠ Could not load audio from this file
{sourceMissing ? (
<>
⚠ Source media missing — this file may have moved or been deleted.
Re-upload the video to continue.
</>
) : (
<>⚠ Could not load audio from this file</>
)}
</div>
</div>
);
frontend/src/hooks/useDubWorkflow.js
+64 -9
View File
@@ -38,12 +38,14 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
const dubTaskId = useAppStore(s => s.dubTaskId);
const setDubTaskId = useAppStore(s => s.setDubTaskId);
const setDubPrepStage = useAppStore(s => s.setDubPrepStage);
const setDubPrepProgress = useAppStore(s => s.setDubPrepProgress);
const setSpeakerClones = useAppStore(s => s.setSpeakerClones);
const setPreviewSegIds = useAppStore(s => s.setPreviewSegIds);
const steps = useAppStore(s => s.steps);
const cfg = useAppStore(s => s.cfg);
const speed = useAppStore(s => s.speed);
const translateQuality = useAppStore(s => s.translateQuality);
const timingStrategy = useAppStore(s => s.timingStrategy);
const glossaryTerms = useAppStore(s => s.glossaryTerms);
const [translateProvider, setTranslateProvider] = useState('argos');
@@ -128,19 +130,48 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
try { m = JSON.parse(e.data); } catch { return; }
lastData = m;
switch (m.type) {
case 'download_start': setDubPrepStage('download'); break;
case 'download_start':
setDubPrepStage('download');
setDubPrepProgress({ percent: null, speedBps: null, etaS: null, stageStartedAt: Date.now() });
break;
case 'download_progress':
setDubPrepProgress(prev => ({
...prev,
percent: typeof m.percent === 'number' ? m.percent : prev.percent,
speedBps: typeof m.speed_bps === 'number' ? m.speed_bps : null,
etaS: typeof m.eta_s === 'number' ? m.eta_s : null,
}));
break;
case 'download_done': if (m.filename) setDubFilename(m.filename); break;
case 'extract_start': setDubPrepStage('extract'); break;
case 'extract_start':
setDubPrepStage('extract');
setDubPrepProgress({ percent: null, speedBps: null, etaS: null, stageStartedAt: Date.now() });
break;
case 'extract_done':
if (m.job_id) setDubJobId(m.job_id);
if (typeof m.duration === 'number') setDubDuration(m.duration);
if (m.filename) setDubFilename(m.filename);
break;
case 'demucs_start': setDubPrepStage('demucs'); break;
case 'demucs_start':
setDubPrepStage('demucs');
setDubPrepProgress({ percent: null, speedBps: null, etaS: null, stageStartedAt: Date.now() });
break;
case 'demucs_progress':
setDubPrepProgress(prev => ({
...prev,
percent: typeof m.percent === 'number' ? m.percent : prev.percent,
}));
break;
case 'demucs_done': break;
case 'scene_start': setDubPrepStage('scene'); break;
case 'scene_start':
setDubPrepStage('scene');
setDubPrepProgress({ percent: null, speedBps: null, etaS: null, stageStartedAt: Date.now() });
break;
case 'scene_done': break;
case 'cached': setDubPrepStage('cached'); break;
case 'cached':
setDubPrepStage('cached');
setDubPrepProgress({ percent: 100, speedBps: null, etaS: null, stageStartedAt: Date.now() });
break;
case 'ready': close(); ctrl.signal.removeEventListener('abort', onAbort); resolve(m); return;
case 'error': {
close(); ctrl.signal.removeEventListener('abort', onAbort);
@@ -162,12 +193,13 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
else reject(new Error('prep stream closed unexpectedly'));
}
};
}), [setDubPrepStage, setDubJobId, setDubDuration, setDubFilename, setDubFailure]);
}), [setDubPrepStage, setDubPrepProgress, setDubJobId, setDubDuration, setDubFilename, setDubFailure]);
// ── Handlers ──
const handleDubUpload = useCallback(async (dubVideoFile) => {
if (!dubVideoFile) return;
setDubStep('uploading'); setDubError(''); setDubFailure(null); setDubTracks([]); setDubPrepStage('download');
setDubPrepProgress({ percent: null, speedBps: null, etaS: null, stageStartedAt: Date.now() });
const ctrl = new AbortController();
dubAbortCtrlRef.current = ctrl;
const clientJobId = Math.random().toString(36).slice(2, 10);
@@ -199,6 +231,7 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
const clean = (url || '').trim();
if (!clean) return;
setDubStep('uploading'); setDubError(''); setDubFailure(null); setDubTracks([]); setDubPrepStage('download');
setDubPrepProgress({ percent: null, speedBps: null, etaS: null, stageStartedAt: Date.now() });
const ctrl = new AbortController();
dubAbortCtrlRef.current = ctrl;
const clientJobId = Math.random().toString(36).slice(2, 10);
@@ -313,7 +346,18 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
setDubSegments(dubSegments.map(s => {
const hit = translatedMap[s.id];
if (!hit) return s;
return { ...s, text: (hit.text && hit.text.trim()) ? hit.text : s.text, translate_error: hit.error || undefined, translate_literal: hit.literal || undefined, translate_critique: hit.critique || undefined };
return {
...s,
text: (hit.text && hit.text.trim()) ? hit.text : s.text,
translate_error: hit.error || undefined,
translate_literal: hit.literal || undefined,
translate_critique: hit.critique || undefined,
// Carry over the predicted compression ratio so the per-row
// badge + job-level compression warning can light up before
// the user clicks Generate Dub.
rate_ratio: hit.rate_ratio != null ? hit.rate_ratio : s.rate_ratio,
rate_error: hit.rate_error || s.rate_error,
};
}));
if (data.cinematic_skipped === 'no-llm-configured') {
toast('Cinematic quality needs an LLM — set TRANSLATE_BASE_URL + TRANSLATE_API_KEY (Ollama works locally). Falling back to Fast.', { icon: 'ℹ️', duration: 7000 });
@@ -356,6 +400,7 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
instruct: dubInstruct,
num_step: steps, guidance_scale: cfg, speed,
preview,
timing_strategy: timingStrategy || 'concise',
};
const data = await dubGenerate(dubJobId, body);
setDubTaskId(data.task_id);
@@ -382,7 +427,17 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
sawDone = true;
setDubStep('done');
setDubTracks(evt.tracks || []);
if (evt.sync_scores) setDubSegments(prev => prev.map((s, idx) => ({ ...s, sync_ratio: evt.sync_scores[idx] })));
// Merge sync_scores (back-compat) and the new richer
// fit_status array onto each segment so the row badge can
// show truthful "Fits / Overflows +0.4s / Video stretched
// 1.18×" labels.
if (evt.sync_scores || evt.fit_status) {
setDubSegments(prev => prev.map((s, idx) => ({
...s,
sync_ratio: evt.sync_scores ? evt.sync_scores[idx] : s.sync_ratio,
fit_status: evt.fit_status ? evt.fit_status[idx] : s.fit_status,
})));
}
if (evt.seg_num_step && typeof evt.seg_num_step === 'object') {
const previewIds = Object.entries(evt.seg_num_step).filter(([, n]) => typeof n === 'number' && n < steps).map(([id]) => id);
setPreviewSegIds(previewIds);
@@ -410,7 +465,7 @@ export default function useDubWorkflow({ loadProjects, loadProfiles, loadDubHist
setDubError(err.message); setDubStep('editing'); setDubTaskId(null);
useAppStore.getState().errorPill(err.message);
}
}, [dubJobId, dubSegments, dubLang, dubLangCode, dubInstruct, steps, cfg, speed, dubStep, setDubStep, setDubProgress, setDubError, setDubTracks, setDubSegments, setDubTaskId, setPreviewSegIds, setLastGenFingerprints, loadDubHistory, loadProjects]);
}, [dubJobId, dubSegments, dubLang, dubLangCode, dubInstruct, steps, cfg, speed, dubStep, timingStrategy, setDubStep, setDubProgress, setDubError, setDubTracks, setDubSegments, setDubTaskId, setPreviewSegIds, setLastGenFingerprints, loadDubHistory, loadProjects]);
const handleDubStop = useCallback(async () => {
if (!dubTaskId) return;
frontend/src/i18n/locales/en.json
+22
View File
@@ -53,5 +53,27 @@
"language": "Language",
"generate": "Generate",
"name": "Name"
},
"demo": {
"clone_coachmark": "This is a guided demo — try the suggested prompt or write your own.",
"hear_demo": "Hear demo",
"stop_demo": "Stop demo",
"prerendered_chip": "Pre-rendered sample — install a TTS engine to synthesize new text.",
"preset_preview": "Preview",
"preset_stop": "Stop",
"preset_use": "Use this design →",
"dictation_title": "Try Dictation",
"dictation_lede": "Read one of these aloud after pressing your hotkey, or hit Replay to send the bundled sample through the transcriber so you can see dictation work end-to-end without speaking.",
"dictation_status_ok": "Verified — hotkey works on this machine",
"dictation_status_pending": "Press your shortcut anywhere to test",
"dictation_status_warn": "No hotkey registered — set one below",
"dictation_hear": "Hear",
"dictation_stop": "Stop",
"dictation_replay": "Replay",
"dictation_transcribing": "Transcribing…",
"dubbing_title": "See dubbing in action",
"dubbing_sync": "Synced playback",
"dubbing_picker": "Try another language:",
"dubbing_cta": "Run this on your own video →"
}
}
frontend/src/index.css
+6 -2
View File
@@ -843,7 +843,7 @@ audio::-webkit-media-controls-time-remaining-display { color: var(--chrome-fg);
box-shadow: inset 2px 0 0 var(--chrome-accent);
}
.segment-time {
width: 48px; flex-shrink: 0;
/* Width comes from --seg-grid-cols in DubSegmentTable.css. */
font-size: 0.62rem; font-family: var(--chrome-font-mono); color: var(--chrome-fg-muted);
font-variant-numeric: tabular-nums;
}
@@ -2302,7 +2302,11 @@ div[role="dialog"].audio-trimmer {
/* ── Segment table — tighter columns ── */
@media (max-width: 800px) {
.segment-time { width: 45px !important; font-size: 0.58rem !important; }
.segment-table {
/* Tighter grid template on narrow screens. */
--seg-grid-cols: 16px 56px 60px minmax(0, 1fr) 38px 52px 36px 38px;
}
.segment-time { font-size: 0.58rem !important; }
.segment-row { padding: 2px 4px; }
.segment-header { padding: 2px 4px; font-size: 0.58rem; }
}
frontend/src/pages/CloneDesignTab.css
+45
View File
@@ -146,3 +146,48 @@
/* Footer buttons */
.clone-footer-cta { margin-top: 6px; }
/* Demo coach-mark — shown once above the textarea on first visit to the
bundled demo profile. Auto-dismisses when the user types. */
.clone-coachmark {
display: flex;
align-items: center;
gap: 8px;
padding: 6px 10px;
margin-bottom: 8px;
border-radius: 8px;
background: rgba(243, 165, 182, 0.08);
border: 1px solid rgba(243, 165, 182, 0.25);
font-size: 11px;
color: var(--color-fg, currentColor);
}
.clone-coachmark__icon { font-size: 14px; line-height: 1; }
.clone-coachmark__msg { flex: 1; }
.clone-coachmark__close {
border: 0;
background: transparent;
color: var(--color-fg-muted, #928374);
cursor: pointer;
font-size: 16px;
line-height: 1;
padding: 0 4px;
border-radius: 4px;
}
.clone-coachmark__close:hover {
background: rgba(255, 255, 255, 0.08);
color: var(--color-fg, currentColor);
}
/* "Hear demo" chip — shown under the play button when no TTS engine is
ready and the user is on the demo profile. Tells them they're hearing a
pre-rendered sample, not a live synthesis. */
.clone-hear-demo-chip {
margin-top: 6px;
padding: 4px 8px;
font-size: 10px;
text-align: center;
color: var(--color-fg-muted, #928374);
background: rgba(255, 255, 255, 0.03);
border-radius: 4px;
border: 1px dashed rgba(255, 255, 255, 0.08);
}
frontend/src/pages/CloneDesignTab.jsx
+150 -16
View File
@@ -1,4 +1,4 @@
import React, { useState, useEffect } from 'react';
import React, { useState, useEffect, useRef } from 'react';
import {
PanelLeftOpen, PanelLeftClose, Command, Globe, SlidersHorizontal, Volume2, User,
UploadCloud, Square, Mic, Save, UserSquare2, Settings2, ChevronUp, ChevronDown,
@@ -7,10 +7,12 @@ import {
import { useTranslation } from 'react-i18next';
import { useQuery } from '@tanstack/react-query';
import SearchableSelect from '../components/SearchableSelect';
import DemoPresetGrid from '../components/DemoPresetGrid';
import ALL_LANGUAGES from '../languages.json';
import { POPULAR_LANGS, PRESETS, TAGS, CATEGORIES } from '../utils/constants';
import { Button, Input, Slider, Progress } from '../ui';
import { API } from '../api/client';
import { listEngines } from '../api/engines';
import './CloneDesignTab.css';
export default function CloneDesignTab(props) {
@@ -65,6 +67,88 @@ export default function CloneDesignTab(props) {
}
setActivePersonality(p.id);
setInstruct(p.instruct);
// Reset category sliders to Auto so the synthesize path doesn't
// merge stale slider tokens with the personality's instruct string —
// that combination caused issue #114 (conflicting items in the same
// category, e.g. "low pitch" from a prior preset + "moderate pitch"
// from the personality).
const resetVd = Object.fromEntries(Object.keys(CATEGORIES).map(k => [k, 'Auto']));
setVdStates(resetVd);
};
// Engine readiness — used by the demo "Hear demo" fallback. Polls every
// 15s so a freshly-finished model download flips the button back to live
// synthesis without a manual refresh.
const { data: enginesData } = useQuery({
queryKey: ['engines-readiness'],
queryFn: listEngines,
refetchInterval: 15000,
staleTime: 5000,
});
const anyTtsReady = !!(enginesData?.tts?.backends || []).some(b => b.available);
// Demo coach-mark: when the user enters the Clone tab with the bundled
// demo profile (demo0001) freshly selected and the textarea is empty,
// prefill a punchy starter prompt and show a one-line coach-mark above
// the textarea. Both auto-dismiss as soon as the user types anything.
// Tracked via localStorage so we don't re-prefill on every visit.
const DEMO_PROFILE_ID = 'demo0001';
const DEMO_PROMPT = "Welcome aboard. I was just a three-second clip a moment ago — now I can say anything you'd like, in your voice or mine.";
const [showDemoCoachmark, setShowDemoCoachmark] = useState(false);
useEffect(() => {
if (mode !== 'clone') return;
if (selectedProfile !== DEMO_PROFILE_ID) return;
if (typeof window === 'undefined') return;
if (localStorage.getItem('omnivoice.demoClonePrompted') === '1') return;
if (text) return; // user already typed something
setText(DEMO_PROMPT);
setShowDemoCoachmark(true);
localStorage.setItem('omnivoice.demoClonePrompted', '1');
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [mode, selectedProfile]);
// "Hear demo" fallback: when no TTS engine is ready and the user is on
// the demo profile, the Synthesize button is swapped for one that plays
// the pre-rendered demo_clone_output.wav. This guarantees a working
// "wow moment" on first launch before any model downloads finish.
const showHearDemo =
mode === 'clone' && selectedProfile === DEMO_PROFILE_ID && !anyTtsReady;
const demoAudioRef = useRef(null);
const [demoAudioPlaying, setDemoAudioPlaying] = useState(false);
const playDemoOutput = () => {
const audio = demoAudioRef.current;
if (!audio) return;
if (demoAudioPlaying) {
audio.pause();
setDemoAudioPlaying(false);
return;
}
audio.src = `${API}/demo_audio/demo_clone_output.wav`;
audio.currentTime = 0;
audio.play()
.then(() => setDemoAudioPlaying(true))
.catch(() => setDemoAudioPlaying(false));
};
// Partition personalities into legacy chips vs. new demo cards.
// `is_demo: true` entries get the rich card grid; the rest keep their
// existing chip-strip rendering (backward-compatible with v0.2.x users
// who learned the chips and shouldn't see them suddenly missing).
const demoPresets = personalities.filter(p => p.is_demo);
const chipPersonalities = personalities.filter(p => !p.is_demo);
// Apply a full demo preset: pre-fill the textarea, set the category
// sliders, clear any stale free-text instruct, switch language, and
// highlight the chip equivalent. After this fires, the user can hit
// Synthesize Audio immediately — no further input needed.
const applyDemoPreset = (p) => {
if (p.script) setText(p.script);
if (p.attrs) setVdStates({ ...vdStates, ...p.attrs });
setInstruct('');
if (p.language) setLanguage(p.language);
setActivePersonality(p.id);
};
return (
@@ -86,19 +170,45 @@ export default function CloneDesignTab(props) {
</Button>
<Command className="label-icon" size={14} /> Prompt
</div>
{mode === 'design' && (
{/* Design-tab empty state: 7-card demo grid replaces the bare
attribute-preset buttons when the user has not yet typed
anything and no personality is active. As soon as they
interact, the grid steps aside for the standard form. */}
{mode === 'design' && !text && !activePersonality && demoPresets.length > 0 && (
<DemoPresetGrid presets={demoPresets} onUse={applyDemoPreset} />
)}
{mode === 'design' && (text || activePersonality || demoPresets.length === 0) && (
<div className="preset-grid">
{PRESETS.map(p => (
<button key={p.id} className="preset-btn" onClick={() => applyPreset(p)}>{p.name}</button>
))}
</div>
)}
{showDemoCoachmark && mode === 'clone' && selectedProfile === DEMO_PROFILE_ID && (
<div className="clone-coachmark" role="note">
<span className="clone-coachmark__icon">💡</span>
<span className="clone-coachmark__msg">
{t('demo.clone_coachmark')}
</span>
<button
type="button"
className="clone-coachmark__close"
onClick={() => setShowDemoCoachmark(false)}
aria-label="Dismiss coach mark"
>
×
</button>
</div>
)}
<textarea
ref={textAreaRef}
className="input-base clone-text-area"
placeholder={mode === 'clone' ? "What should this voice say? ✍️" : "Describe the voice, then type what it says…"}
value={text}
onChange={e => setText(e.target.value)}
onChange={e => {
setText(e.target.value);
if (showDemoCoachmark) setShowDemoCoachmark(false);
}}
/>
<div className="tags-container">
{TAGS.map(tag => <button key={tag} className="tag-btn" onClick={() => insertTag(tag)}>{tag}</button>)}
@@ -263,12 +373,14 @@ export default function CloneDesignTab(props) {
<div>
<div className="label-row"><UserSquare2 className="label-icon" size={14} /> {t('voice.personality')}</div>
{/* Personality presets */}
{personalities.length > 0 && (
{/* Personality presets — chip-only entries. Demo presets
render as full cards in the empty state above; including
them here too would duplicate the affordance. */}
{chipPersonalities.length > 0 && (
<div style={{ marginBottom: 10 }}>
<div className="personality-label">{t('voice.pick_personality')}</div>
<div className="personality-strip">
{personalities.map(p => (
{chipPersonalities.map(p => (
<button
key={p.id}
type="button"
@@ -372,16 +484,38 @@ export default function CloneDesignTab(props) {
</div>
)}
<Button
variant="primary"
block
loading={isGenerating}
onClick={handleGenerate}
leading={!isGenerating && <Play size={14} />}
className="clone-footer-cta"
>
{isGenerating ? `Synthesizing… (${generationTime}s)` : 'Synthesize Audio'}
</Button>
{showHearDemo ? (
<>
<Button
variant="primary"
block
onClick={playDemoOutput}
leading={<Play size={14} />}
className="clone-footer-cta"
>
{demoAudioPlaying ? t('demo.stop_demo') : t('demo.hear_demo')}
</Button>
<div className="clone-hear-demo-chip">
{t('demo.prerendered_chip')}
</div>
<audio
ref={demoAudioRef}
onEnded={() => setDemoAudioPlaying(false)}
preload="none"
/>
</>
) : (
<Button
variant="primary"
block
loading={isGenerating}
onClick={handleGenerate}
leading={!isGenerating && <Play size={14} />}
className="clone-footer-cta"
>
{isGenerating ? `Synthesizing… (${generationTime}s)` : 'Synthesize Audio'}
</Button>
)}
{isGenerating && (
<Progress
value={Math.min((generationTime / 8) * 100, 95)}
frontend/src/pages/DubTab.css
+44
View File
@@ -102,6 +102,26 @@
max-width: 320px;
text-align: center;
}
.dub-prep-overlay__detail {
font-size: var(--text-xs);
color: var(--color-fg-subtle);
font-variant-numeric: tabular-nums;
letter-spacing: 0.02em;
}
.dub-prep-bar {
width: 240px;
height: 4px;
border-radius: 999px;
background: rgba(255, 255, 255, 0.06);
overflow: hidden;
}
.dub-prep-overlay--large .dub-prep-bar { width: 320px; height: 5px; }
.dub-prep-bar__fill {
height: 100%;
background: linear-gradient(90deg, rgba(211, 134, 155, 0.75), rgba(211, 134, 155, 1));
border-radius: inherit;
transition: width 220ms ease-out;
}
.dub-trans-overlay {
display: flex;
@@ -210,6 +230,30 @@
.dub-tracks-row label.is-on { color: var(--chrome-fg); border-color: var(--chrome-border-strong); background: var(--chrome-hover-bg); }
.dub-tracks-row label span.code { text-transform: uppercase; letter-spacing: 0.04em; }
/* Pre-generation compression-ratio warning surfaced above Generate Dub. */
.dub-compression-warn {
display: flex;
align-items: flex-start;
gap: 8px;
padding: 6px 10px;
margin: 4px 0;
background: color-mix(in srgb, #fabd2f 12%, transparent);
border: 1px solid color-mix(in srgb, #fabd2f 35%, transparent);
border-left: 2px solid #fabd2f;
border-radius: var(--chrome-radius-pill);
font-size: 0.72rem;
color: var(--chrome-fg);
line-height: 1.35;
}
.dub-compression-warn__icon {
color: #fabd2f;
font-size: 0.9rem;
line-height: 1;
flex-shrink: 0;
}
.dub-compression-warn__body { flex: 1; }
.dub-compression-warn strong { color: #fabd2f; font-weight: var(--weight-semibold); }
/* ── Utility: column flex that fills remaining height ────────── */
.dub-col { flex: 1; display: flex; flex-direction: column; min-height: 0; }
.dub-panel-col { margin-bottom: 0; overflow: hidden; display: flex; flex-direction: column; }
frontend/src/pages/DubTab.jsx
+127 -19
View File
@@ -23,6 +23,7 @@ import { openDocsFor, classifyError } from '../utils/errorDocsMap';
import GlossaryPanel from '../components/GlossaryPanel';
import ExportModal from '../components/ExportModal';
import MultiLangPicker from '../components/MultiLangPicker';
import DubbingDemo from '../components/DubbingDemo';
import './DubTab.css';
const DubSegmentTable = lazy(() => import('../components/DubSegmentTable'));
@@ -93,6 +94,7 @@ export default function DubTab(props) {
const dubStep = useAppStore(s => s.dubStep);
const setDubStep = useAppStore(s => s.setDubStep);
const dubPrepStage = useAppStore(s => s.dubPrepStage);
const dubPrepProgress = useAppStore(s => s.dubPrepProgress);
const dubFilename = useAppStore(s => s.dubFilename);
const dubDuration = useAppStore(s => s.dubDuration);
const dubSegments = useAppStore(s => s.dubSegments);
@@ -124,6 +126,8 @@ export default function DubTab(props) {
const setDualSubs = useAppStore(s => s.setDualSubs);
const burnSubs = useAppStore(s => s.burnSubs);
const setBurnSubs = useAppStore(s => s.setBurnSubs);
const timingStrategy = useAppStore(s => s.timingStrategy);
const setTimingStrategy = useAppStore(s => s.setTimingStrategy);
const showIdleSkeleton = !(dubJobId && (dubStep === 'editing' || dubStep === 'generating' || dubStep === 'done'));
// Imperative handle to the post-job waveform so the transcript table can
@@ -133,6 +137,17 @@ export default function DubTab(props) {
waveformRef.current?.seekTo?.(time);
}, []);
const [ingestUrl, setIngestUrl] = useState('');
// Dubbing demo: show the side-by-side player above the drop zone on
// first-run / no-project state. localStorage flag persists dismissal
// across sessions so power users don't see it every launch.
const [demoDismissed, setDemoDismissed] = useState(() => {
if (typeof window === 'undefined') return false;
return localStorage.getItem('omnivoice.dubbingDemoDismissed') === '1';
});
const dismissDubDemo = () => {
setDemoDismissed(true);
try { localStorage.setItem('omnivoice.dubbingDemoDismissed', '1'); } catch { /* noop */ }
};
const [previewMode, setPreviewMode] = useState('original'); // 'original' | 'dubbed'
const [exportOpen, setExportOpen] = useState(false);
@@ -197,10 +212,11 @@ export default function DubTab(props) {
}
};
// Collapse secondary settings (Language/ISO/Style/Engine/Quality) into an
// accordion. Once the user has translated, the row's job is done; show a
// one-line summary instead of the full 5-col grid.
const [settingsOpen, setSettingsOpen] = useState(false);
// Secondary settings (Language/ISO/Style/Engine/Quality/Multi-lang) are
// expanded by default so the user can pick a target language and quality
// without an extra click on first open. They stay an accordion so the
// user can collapse them once happy with the choice.
const [settingsOpen, setSettingsOpen] = useState(true);
const hasAnyTranslation = dubSegments.some(s => s.text_original && s.text_original !== s.text);
// Glossary: hide behind a chip when empty, auto-open once terms exist.
@@ -238,11 +254,7 @@ export default function DubTab(props) {
if (!ingestUrl.trim() || !handleDubIngestUrl) return;
handleDubIngestUrl(ingestUrl.trim(), {
fetchSubs: fetchYtSubs,
// Default to "all" available tracks — YouTube's auto-translator makes
// every major language available on demand, so letting yt-dlp grab
// them all up-front means switching target language later doesn't
// need another round trip.
subLangs: fetchYtSubs ? undefined : undefined,
subLangs: undefined,
});
setIngestUrl('');
};
@@ -339,7 +351,7 @@ export default function DubTab(props) {
disabled={true}
overlayContent={
dubStep === 'uploading' ? (
<PrepOverlay stage={dubPrepStage} onAbort={handleDubAbort} />
<PrepOverlay stage={dubPrepStage} progress={dubPrepProgress} onAbort={handleDubAbort} />
) : dubStep === 'transcribing' ? (
<TranscribeOverlay
elapsed={transcribeElapsed}
@@ -384,9 +396,13 @@ export default function DubTab(props) {
</div>
</>
) : dubStep === 'uploading' ? (
<PrepOverlay stage={dubPrepStage} onAbort={handleDubAbort} large />
<PrepOverlay stage={dubPrepStage} progress={dubPrepProgress} onAbort={handleDubAbort} large />
) : (
<label htmlFor="video-upload" className="dub-idle-drop"
<>
{!demoDismissed && (
<DubbingDemo onDismiss={dismissDubDemo} />
)}
<label htmlFor="video-upload" className="dub-idle-drop"
onDragOver={e => { e.preventDefault(); e.currentTarget.classList.add('is-dragging'); }}
onDragLeave={e => { e.currentTarget.classList.remove('is-dragging'); }}
onDrop={e => {
@@ -431,7 +447,7 @@ export default function DubTab(props) {
</div>
<label
className="dub-ingest-sub-opt"
title="When the URL is a caption-bearing host (YouTube, Vimeo, TED…), also pull the original captions and any YouTube auto-translations. Seeds the editor without running Whisper; skip Translate All for languages YouTube already covers."
title="When the URL is a caption-bearing host (YouTube, Vimeo, TED…), also pull the original-language captions (manually-uploaded + auto-generated). Lets the editor seed from real subtitles instead of running Whisper from scratch."
onClick={e => { e.stopPropagation(); }}
>
<input
@@ -440,9 +456,10 @@ export default function DubTab(props) {
onChange={e => setFetchYtSubs(e.target.checked)}
onClick={e => e.stopPropagation()}
/>
<span>Pull YouTube captions + auto-translations</span>
<span>Pull original-language captions</span>
</label>
</label>
</>
)}
<input type="file" accept="video/*,audio/*,.mp3,.wav,.m4a,.flac,.ogg" id="video-upload" className="dub-hidden-file"
@@ -1006,6 +1023,21 @@ export default function DubTab(props) {
</select>
</label>
</div>
<div
className="dub-outputs-row"
title="Timing strategy — how the dub reconciles natural-rate TTS with the original timeline."
>
<span className="dub-outputs-title-strong">Timing:</span>
<Segmented
value={timingStrategy}
onChange={setTimingStrategy}
options={[
{ value: 'concise', label: 'Concise', title: 'Translator trims text to fit at natural rate. Overflows surface in the row badge so you can shorten the segment.' },
{ value: 'stretch_video', label: 'Stretch Video', title: 'Audio plays at natural rate; each segment of the video is stretched (per-segment ffmpeg setpts) to fit. Total video duration grows. Requires a re-encode pass.' },
{ value: 'strict_slot', label: 'Strict slot', title: 'Legacy: compress audio to fit the original timing. Can sound rushed/chipmunky on high-density target languages.' },
]}
/>
</div>
{dubTracks.length > 0 && (
<div className="dub-tracks-row">
<span className="dub-tracks-row__title">Export Tracks:</span>
@@ -1021,6 +1053,32 @@ export default function DubTab(props) {
))}
</div>
)}
{(() => {
// Pre-generation compression warning. Predicted by the
// translate response (see services/speech_rate.rate_ratio
// + dub_translate._maybe_cinematic), populated whenever
// segments carry a slot_seconds and translated text.
// Surfaces here so the user can act (re-translate in
// Cinematic, edit text, allow longer slots) before
// committing to a full Generate Dub run.
const hot = dubSegments.filter(s => (s.rate_ratio || 0) > 1.3);
if (hot.length === 0 || !dubSegments.length) return null;
const pctHot = Math.round((hot.length / dubSegments.length) * 100);
if (pctHot < 10) return null;
const worst = hot.reduce((a, b) => (a.rate_ratio > b.rate_ratio ? a : b));
return (
<div className="dub-compression-warn" role="status">
<span className="dub-compression-warn__icon">⚠</span>
<span className="dub-compression-warn__body">
<strong>{hot.length} of {dubSegments.length}</strong> segments need {'>'}1.3× compression
(worst: <span style={{ fontVariantNumeric: 'tabular-nums' }}>{worst.rate_ratio.toFixed(2)}×</span>).
Output will be intelligible (pitch-preserving stretch) but stressed —
{translateQuality === 'fast' ? ' switch to Cinematic and Re-translate' : ' shorten the worst segments'}
{' '}for cleaner audio.
</span>
</div>
);
})()}
<div className="dub-footer-btns">
{dubStep === 'stopping' ? (
<FooterBtn tone="stopping" disabled icon={<Loader className="spinner" size={9} />} label="Stopping…" />
@@ -1094,18 +1152,70 @@ const PREP_STAGE_LABEL = {
const PREP_FULL = ['download', 'extract', 'demucs', 'scene'];
const PREP_CACHED = ['download', 'extract', 'cached'];
function fmtBytesRate(bps) {
if (!bps || bps <= 0) return null;
const units = ['B/s', 'KB/s', 'MB/s', 'GB/s'];
let v = bps, i = 0;
while (v >= 1024 && i < units.length - 1) { v /= 1024; i += 1; }
return `${v < 10 ? v.toFixed(1) : Math.round(v)} ${units[i]}`;
}
function fmtEta(seconds) {
if (seconds == null || !Number.isFinite(seconds) || seconds < 0) return null;
const s = Math.round(seconds);
if (s < 60) return `${s}s left`;
const m = Math.floor(s / 60), rem = s % 60;
return rem ? `${m}m ${rem}s left` : `${m}m left`;
}
/**
* PrepOverlay — the prepare-upload stage indicator.
* `large` makes the surrounding frame bigger (used for the empty-state drop zone).
*/
function PrepOverlay({ stage, onAbort, large = false }) {
function PrepOverlay({ stage, progress, onAbort, large = false }) {
const stages = stage === 'cached' ? PREP_CACHED : PREP_FULL;
// Elapsed-time ticker for the current stage. Reset whenever
// stageStartedAt changes (i.e. the backend transitions stages).
const [elapsedS, setElapsedS] = useState(0);
const startedAt = progress?.stageStartedAt ?? null;
useEffect(() => {
if (!startedAt) { setElapsedS(0); return undefined; }
setElapsedS(Math.floor((Date.now() - startedAt) / 1000));
const iv = setInterval(() => {
setElapsedS(Math.floor((Date.now() - startedAt) / 1000));
}, 1000);
return () => clearInterval(iv);
}, [startedAt]);
const pct = progress?.percent;
const hasPct = typeof pct === 'number' && pct >= 0 && pct <= 100;
const speed = stage === 'download' ? fmtBytesRate(progress?.speedBps) : null;
const eta = fmtEta(progress?.etaS);
const elapsedLabel = startedAt ? (elapsedS < 60 ? `${elapsedS}s` : `${Math.floor(elapsedS / 60)}m ${elapsedS % 60}s`) : null;
const detailBits = [
hasPct ? `${pct}%` : null,
elapsedLabel ? `${elapsedLabel} elapsed` : null,
speed,
eta,
].filter(Boolean);
const note = stage === 'demucs' && !hasPct
? 'Demucs typically takes 20–40% of the video length to separate vocals from music.'
: null;
const body = (
<>
<Loader className="spinner" size={large ? 28 : 20} color="#d3869b" />
<span className="dub-prep-overlay__title" style={{ fontSize: large ? '0.95rem' : '0.85rem' }}>
{PREP_STAGE_LABEL[stage] || 'Preparing…'}
</span>
{hasPct && (
<div className="dub-prep-bar" aria-label={`${pct}%`}>
<div className="dub-prep-bar__fill" style={{ width: `${pct}%` }} />
</div>
)}
{detailBits.length > 0 && (
<span className="dub-prep-overlay__detail">{detailBits.join(' · ')}</span>
)}
<div className={`dub-prep-chips ${large ? 'dub-prep-chips--lg' : ''}`}>
{stages.map(s => (
<span
@@ -1116,10 +1226,8 @@ function PrepOverlay({ stage, onAbort, large = false }) {
</span>
))}
</div>
{stage === 'demucs' && (
<span className="dub-prep-overlay__note">
Demucs can take several minutes on long videos. Long audio = longer wait.
</span>
{note && (
<span className="dub-prep-overlay__note">{note}</span>
)}
<Button variant="danger" size="sm" onClick={onAbort} leading={<Square size={11} />}>
Stop
frontend/src/pages/Settings.jsx
+9 -1
View File
@@ -27,6 +27,8 @@ import PerformancePanel from '../components/settings/PerformancePanel';
import AppearancePanel from '../components/settings/AppearancePanel';
import StoragePanel from '../components/settings/StoragePanel';
import EngineCompatibilityMatrix from '../components/EngineCompatibilityMatrix';
import DictationDemo from '../components/DictationDemo';
import ReportBugButton from '../components/ReportBugButton';
import './Settings.css';
const TABS = [
@@ -1037,7 +1039,12 @@ export default function Settings() {
{activeTab === 'engines' && <EnginesTab />}
{activeTab === 'capture' && <HotkeyTab />}
{activeTab === 'capture' && (
<>
<DictationDemo />
<HotkeyTab />
</>
)}
{activeTab === 'credentials' && <CredentialsTab info={info} />}
@@ -1048,6 +1055,7 @@ export default function Settings() {
<FileText size={16} color="#fabd2f" /> Logs
</span>
<span className="settings-section__head-actions">
<ReportBugButton />
<Button
variant="subtle"
size="sm"
frontend/src/pages/SetupWizard.jsx
+27 -3
View File
@@ -6,6 +6,7 @@ import {
import { Button } from '../ui';
import { useSetupStatus, usePreflight } from '../api/hooks';
import { ModelStoreTab, EnginesTab } from './Settings';
import DictationDemo from '../components/DictationDemo';
import './SetupWizard.css';
import '../components/Misc.css';
@@ -108,7 +109,7 @@ function PreflightPanel({ report, loading, onRecheck }) {
/* ── Stepper nav with connectors ───────────────────────────────────────── */
const STEP_LABELS = ['Welcome', 'System check', 'Install models', 'Pick engines'];
const STEP_LABELS = ['Welcome', 'System check', 'Install models', 'Pick engines', 'Try dictation'];
function StepperNav({ step, onStep }) {
return (
@@ -286,15 +287,38 @@ export default function SetupWizard({ onReady }) {
<Button variant="ghost" onClick={() => setStep(2)}>Back</Button>
<Button
variant="primary"
onClick={onReady}
onClick={() => setStep(4)}
leading={<CheckCircle size={14} />}
>
Enter studio
Next: Try dictation
</Button>
</div>
</div>
)}
{/* 4. Dictation — guided walkthrough. Skippable (per cross-platform
parity rule: some users genuinely don't want dictation). */}
{step === 4 && (
<div className="swiz-slide" key="step-4">
<div className="setup-wizard__embed">
<DictationDemo />
</div>
<div className="setup-wizard__nav">
<Button variant="ghost" onClick={() => setStep(3)}>Back</Button>
<div style={{ display: 'flex', gap: 8 }}>
<Button variant="subtle" onClick={onReady}>Skip</Button>
<Button
variant="primary"
onClick={onReady}
leading={<CheckCircle size={14} />}
>
Enter studio
</Button>
</div>
</div>
</div>
)}
{!status && step > 1 && (
<div className="swiz-status-loading">
<Loader className="spinner" size={14} /> Checking setup…
frontend/src/store/dubSlice.ts
+18
View File
@@ -34,6 +34,14 @@ export interface DubProgress {
text: string;
}
/** Per-stage progress for the prep pipeline (download, demucs). */
export interface DubPrepProgress {
percent: number | null; // 0–100, or null if not known yet
speedBps: number | null; // download speed in bytes/sec, when relevant
etaS: number | null; // ETA in seconds, when known
stageStartedAt: number | null; // ms epoch; used for elapsed-time display
}
/** Segments are a loose shape — many optional fields added over time. */
export type DubSegment = Record<string, unknown> & { id: string; text: string };
@@ -60,7 +68,10 @@ export interface DubSlice {
dubStep: DubStep;
dubTaskId: string | null;
dubPrepStage: DubPrepStage;
dubPrepProgress: DubPrepProgress;
dubProgress: DubProgress;
/** ID of the segment containing the current media playhead, or null. */
dubCurrentSegId: string | null;
dubError: string;
dubFailure: DubFailure | null;
isTranslating: boolean;
@@ -108,7 +119,9 @@ export interface DubSlice {
setDubStep: (v: Updater<DubStep>) => void;
setDubTaskId: (v: Updater<string | null>) => void;
setDubPrepStage: (v: Updater<DubPrepStage>) => void;
setDubPrepProgress: (v: Updater<DubPrepProgress>) => void;
setDubProgress: (v: Updater<DubProgress>) => void;
setDubCurrentSegId: (v: Updater<string | null>) => void;
setDubError: (v: Updater<string>) => void;
setDubFailure: (v: Updater<DubSlice['dubFailure']>) => void;
setIsTranslating: (v: Updater<boolean>) => void;
@@ -132,6 +145,7 @@ export interface DubSlice {
const INITIAL: Omit<DubSlice,
| 'setDubJobId' | 'setDubStep' | 'setDubTaskId' | 'setDubPrepStage'
| 'setDubPrepProgress' | 'setDubCurrentSegId'
| 'setDubProgress' | 'setDubError' | 'setDubFailure' | 'setIsTranslating' | 'setDubSegments'
| 'setDubTranscript' | 'setDubFilename' | 'setDubDuration' | 'setDubTracks'
| 'setDubLang' | 'setDubLangCode' | 'setDubInstruct' | 'setPreserveBg'
@@ -142,7 +156,9 @@ const INITIAL: Omit<DubSlice,
dubStep: 'idle',
dubTaskId: null,
dubPrepStage: null,
dubPrepProgress: { percent: null, speedBps: null, etaS: null, stageStartedAt: null },
dubProgress: { current: 0, total: 0, text: '' },
dubCurrentSegId: null,
dubError: '',
dubFailure: null,
isTranslating: false,
@@ -170,7 +186,9 @@ export const createDubSlice: StateCreator<DubSlice, [], [], DubSlice> = (set, ge
setDubStep: (v) => set((s) => ({ dubStep: resolve(v, s.dubStep) })),
setDubTaskId: (v) => set((s) => ({ dubTaskId: resolve(v, s.dubTaskId) })),
setDubPrepStage: (v) => set((s) => ({ dubPrepStage: resolve(v, s.dubPrepStage) })),
setDubPrepProgress: (v) => set((s) => ({ dubPrepProgress: resolve(v, s.dubPrepProgress) })),
setDubProgress: (v) => set((s) => ({ dubProgress: resolve(v, s.dubProgress) })),
setDubCurrentSegId: (v) => set((s) => ({ dubCurrentSegId: resolve(v, s.dubCurrentSegId) })),
setDubError: (v) => set((s) => ({ dubError: resolve(v, s.dubError) })),
setDubFailure: (v) => set((s) => ({ dubFailure: resolve(v, s.dubFailure) })),
setIsTranslating:(v) => set((s) => ({ isTranslating:resolve(v, s.isTranslating) })),
frontend/src/store/index.ts
+10 -8
View File
@@ -60,6 +60,7 @@ export const useAppStore = create<AppStore>()(
glossaryVisible: s.glossaryVisible,
reviewMode: s.reviewMode,
showHeaderLiveStats: s.showHeaderLiveStats,
timingStrategy: s.timingStrategy,
mode: s.mode,
isSidebarCollapsed: s.isSidebarCollapsed,
isSidebarProjectsCollapsed: s.isSidebarProjectsCollapsed,
@@ -79,17 +80,18 @@ export const useAppStore = create<AppStore>()(
postprocess: s.postprocess,
vdStates: s.vdStates,
}),
version: 3,
version: 4,
// Drop old persisted shapes rather than crashing the app. Every field
// has a safe default in its slice, so v1/v2 users pick up v3 defaults
// for the new fields (mode, uiScale, generate knobs, etc.) and keep
// any keys we still write today. Upgrade > crash.
// has a safe default in its slice, so v1/v2/v3 users pick up v4 defaults
// for new fields (timingStrategy etc.) and keep any keys we still write
// today. Upgrade > crash.
migrate: (persisted, version) => {
if (!persisted || typeof persisted !== 'object') return {} as Partial<AppStore>;
if (version < 3) {
// v1 → v2 added reviewMode; v2 → v3 added mode/sidebar/generate knobs.
// All of those have slice defaults, so passing through the old keys
// is sufficient — anything missing falls through to the slice init.
if (version < 4) {
// v1 → v2 added reviewMode; v2 → v3 added mode/sidebar/generate knobs;
// v3 → v4 added timingStrategy. All of those have slice defaults,
// so passing through the old keys is sufficient — anything missing
// falls through to the slice init.
return persisted as Partial<AppStore>;
}
return persisted as Partial<AppStore>;
frontend/src/store/prefsSlice.ts
+21
View File
@@ -11,6 +11,16 @@ import type { StateCreator } from 'zustand';
export type TranslateQuality = 'fast' | 'cinematic';
export type ThemeId = 'gruvbox' | 'midnight' | 'nord' | 'solarized' | 'rose-pine' | 'catppuccin';
/**
* Dub timing strategy — replaces audio time-compression with two cleaner
* alternatives. `concise` trims the translation up-front so it fits at
* natural rate (overflows surfaced for manual edit); `stretch_video`
* stretches the source video per-segment so natural-rate audio fits
* without lip-sync drift. `strict_slot` is the legacy compress-to-fit
* path, retained for back-compat.
*/
export type TimingStrategy = 'concise' | 'stretch_video' | 'strict_slot';
export interface PrefsSlice {
translateQuality: TranslateQuality;
dualSubs: boolean;
@@ -32,12 +42,21 @@ export interface PrefsSlice {
*/
showHeaderLiveStats: boolean;
/**
* How the dub pipeline reconciles natural-rate TTS with the original
* timeline. `concise` (default) trims translation to fit; `stretch_video`
* stretches the video instead; `strict_slot` compresses the audio to fit
* (legacy behaviour, retained for back-compat).
*/
timingStrategy: TimingStrategy;
setTranslateQuality: (q: TranslateQuality) => void;
setDualSubs: (on: boolean) => void;
setBurnSubs: (on: boolean) => void;
setGlossaryVisible: (on: boolean) => void;
setReviewMode: (mode: 'on' | 'off') => void;
setShowHeaderLiveStats: (on: boolean) => void;
setTimingStrategy: (s: TimingStrategy) => void;
theme: ThemeId;
setTheme: (id: ThemeId) => void;
@@ -50,6 +69,7 @@ export const createPrefsSlice: StateCreator<PrefsSlice, [], [], PrefsSlice> = (s
glossaryVisible: true,
reviewMode: 'on',
showHeaderLiveStats: false,
timingStrategy: 'concise',
setTranslateQuality: (q) => set({ translateQuality: q }),
setDualSubs: (on) => set({ dualSubs: on }),
@@ -57,6 +77,7 @@ export const createPrefsSlice: StateCreator<PrefsSlice, [], [], PrefsSlice> = (s
setGlossaryVisible: (on) => set({ glossaryVisible: on }),
setReviewMode: (mode) => set({ reviewMode: mode }),
setShowHeaderLiveStats: (on) => set({ showHeaderLiveStats: on }),
setTimingStrategy: (s) => set({ timingStrategy: s }),
theme: 'gruvbox',
setTheme: (id) => {
frontend/src/test/DemoPresetGrid.test.jsx
+59
View File
@@ -0,0 +1,59 @@
import React from 'react';
import { describe, it, expect, vi } from 'vitest';
import { fireEvent, render, screen } from '@testing-library/react';
import { I18nextProvider } from 'react-i18next';
import i18n from '../i18n';
import DemoPresetGrid from '../components/DemoPresetGrid';
const PRESETS = [
{
id: 'p1',
name: 'The Librarian',
icon: '📚',
description: 'Warm UK narrator',
instruct: 'female, middle-aged, low pitch, british accent',
attrs: { Gender: 'female', Age: 'middle-aged' },
script: 'Once upon a time…',
preview_url: '/demo_audio/voice_design/p1.wav',
language: 'English',
},
{
id: 'p2',
name: 'The Anchor',
icon: '📺',
description: 'US news broadcaster',
instruct: 'male, middle-aged, moderate pitch, american accent',
attrs: { Gender: 'male' },
script: 'Good evening…',
preview_url: '/demo_audio/voice_design/p2.wav',
language: 'English',
},
];
function withI18n(node) {
return <I18nextProvider i18n={i18n}>{node}</I18nextProvider>;
}
describe('DemoPresetGrid', () => {
it('renders one card per preset', () => {
render(withI18n(<DemoPresetGrid presets={PRESETS} onUse={vi.fn()} />));
expect(screen.getByText('The Librarian')).toBeInTheDocument();
expect(screen.getByText('The Anchor')).toBeInTheDocument();
expect(screen.getAllByText('Use this design →')).toHaveLength(2);
});
it('shows the instruct taxonomy string on each card', () => {
render(withI18n(<DemoPresetGrid presets={PRESETS} onUse={vi.fn()} />));
expect(screen.getByText(/british accent/)).toBeInTheDocument();
expect(screen.getByText(/american accent/)).toBeInTheDocument();
});
it('calls onUse with the preset object when "Use this design" is clicked', () => {
const onUse = vi.fn();
render(withI18n(<DemoPresetGrid presets={PRESETS} onUse={onUse} />));
const buttons = screen.getAllByText('Use this design →');
fireEvent.click(buttons[0]);
expect(onUse).toHaveBeenCalledWith(PRESETS[0]);
});
});
frontend/src/test/DictationDemo.test.jsx
+82
View File
@@ -0,0 +1,82 @@
import React from 'react';
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { render, screen, waitFor, fireEvent } from '@testing-library/react';
import { I18nextProvider } from 'react-i18next';
import i18n from '../i18n';
import DictationDemo from '../components/DictationDemo';
function withI18n(node) {
return <I18nextProvider i18n={i18n}>{node}</I18nextProvider>;
}
describe('DictationDemo', () => {
let originalFetch;
beforeEach(() => {
originalFetch = global.fetch;
});
afterEach(() => {
global.fetch = originalFetch;
});
it('renders the three bundled scripts', () => {
render(withI18n(<DictationDemo />));
expect(screen.getByText(/Schedule a meeting with Pat/)).toBeInTheDocument();
expect(screen.getByText(/Patch the WebGPU shader/)).toBeInTheDocument();
expect(screen.getByText(/réserver une table/)).toBeInTheDocument();
});
it('shows the "no hotkey" warning when outside Tauri', () => {
render(withI18n(<DictationDemo />));
// In jsdom, isTauri() returns false → state stays 'unknown' → warn badge.
expect(screen.getByText(/No hotkey registered/i)).toBeInTheDocument();
});
it('POSTs the bundled WAV to /transcribe when Replay is clicked', async () => {
const wavBlob = new Blob([new Uint8Array([0, 0, 0, 0])], { type: 'audio/wav' });
// Make the recognized text deliberately different from the on-card
// script so we can assert the transcribed result appears in the UI.
const RECOGNIZED = 'sentinel-recognized-payload-9421';
global.fetch = vi.fn((url) => {
if (String(url).endsWith('.wav')) {
return Promise.resolve({ ok: true, blob: () => Promise.resolve(wavBlob) });
}
return Promise.resolve({
ok: true,
json: () => Promise.resolve({ text: RECOGNIZED, language: 'en' }),
});
});
render(withI18n(<DictationDemo />));
const replayButton = screen.getByLabelText(/Replay Conversational/i);
fireEvent.click(replayButton);
// Wait for both fetches to complete and the recognized text to render.
// The full chain is: fetch(.wav) → fetch(/transcribe) → setState → render.
await waitFor(
() => {
const calls = global.fetch.mock.calls;
const wavCall = calls.find(c => String(c[0]).endsWith('.wav'));
expect(wavCall).toBeTruthy();
},
{ timeout: 3000 },
);
await waitFor(
() => {
const calls = global.fetch.mock.calls;
expect(calls.find(c => String(c[0]).endsWith('/transcribe'))).toBeTruthy();
},
{ timeout: 3000 },
);
await waitFor(
() => expect(screen.getByText(RECOGNIZED)).toBeInTheDocument(),
{ timeout: 3000 },
);
const calls = global.fetch.mock.calls;
const transcribeCall = calls.find(c => String(c[0]).endsWith('/transcribe'));
expect(transcribeCall[1]?.method).toBe('POST');
});
});
frontend/src/test/DubbingDemo.test.jsx
+76
View File
@@ -0,0 +1,76 @@
import React from 'react';
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { render, screen, waitFor, fireEvent } from '@testing-library/react';
import { I18nextProvider } from 'react-i18next';
import i18n from '../i18n';
import DubbingDemo from '../components/DubbingDemo';
const MOCK_MANIFEST = {
source: {
code: 'en', label: 'English',
video: 'source.mp4', srt: 'source.srt',
script: 'OmniVoice runs entirely on your machine.',
},
dubbed: [
{ code: 'es', label: 'Español', video: 'dubbed_es.mp4', dir: 'ltr', script: 'Funciona en tu máquina.' },
{ code: 'fr', label: 'Français', video: 'dubbed_fr.mp4', dir: 'ltr', script: 'Fonctionne sur votre machine.' },
{ code: 'ja', label: '日本語', video: 'dubbed_ja.mp4', dir: 'ltr', script: 'マシン上で動作します。' },
],
};
function withI18n(node) {
return <I18nextProvider i18n={i18n}>{node}</I18nextProvider>;
}
describe('DubbingDemo', () => {
let originalFetch;
beforeEach(() => {
originalFetch = global.fetch;
global.fetch = vi.fn(() => Promise.resolve({
ok: true,
json: () => Promise.resolve(MOCK_MANIFEST),
}));
});
afterEach(() => {
global.fetch = originalFetch;
});
it('renders source + initial dubbed pane after manifest loads', async () => {
render(withI18n(<DubbingDemo onDismiss={vi.fn()} />));
await waitFor(() => {
expect(screen.getByText('English')).toBeInTheDocument();
});
// Default pick is 'es'. Spanish appears in both the pane label and the
// picker chip — assert at least one match plus the unique script.
expect(screen.getAllByText('Español').length).toBeGreaterThan(0);
expect(screen.getByText(/Funciona en tu máquina/)).toBeInTheDocument();
});
it('shows all language chips in the picker', async () => {
render(withI18n(<DubbingDemo onDismiss={vi.fn()} />));
await waitFor(() => screen.getByText('English'));
expect(screen.getAllByText('Español').length).toBeGreaterThanOrEqual(1);
expect(screen.getByRole('button', { name: 'Français' })).toBeInTheDocument();
expect(screen.getByRole('button', { name: '日本語' })).toBeInTheDocument();
});
it('swaps the dubbed pane script when a chip is clicked', async () => {
render(withI18n(<DubbingDemo onDismiss={vi.fn()} />));
await waitFor(() => screen.getByText('English'));
fireEvent.click(screen.getByRole('button', { name: 'Français' }));
await waitFor(() => {
expect(screen.getByText(/Fonctionne sur votre machine/)).toBeInTheDocument();
});
});
it('calls onDismiss when the CTA is clicked', async () => {
const onDismiss = vi.fn();
render(withI18n(<DubbingDemo onDismiss={onDismiss} />));
await waitFor(() => screen.getByText('English'));
fireEvent.click(screen.getByRole('button', { name: /Run this on your own video/i }));
expect(onDismiss).toHaveBeenCalled();
});
});
frontend/src/test/EngineCompatibilityMatrix.test.jsx
+7 -1
View File
@@ -140,7 +140,13 @@ describe('EngineCompatibilityMatrix', () => {
await waitFor(() => screen.getByText('KittenTTS (test)'));
const kittenRow = screen.getByText('KittenTTS (test)').closest('[role="row"]');
expect(within(kittenRow).getByText('kittentts not installed')).toBeInTheDocument();
expect(within(kittenRow).getByText(/Unavailable/i)).toBeInTheDocument();
// The badge text is exactly "Unavailable" (with a leading icon); the new
// disclosure summary is "Why unavailable?" — scope to the badge with an
// exact match so we don't double-count the summary.
const badge = within(kittenRow).getByText((_, el) =>
el?.tagName === 'SPAN' && /^\s*Unavailable\s*$/.test(el.textContent || '')
);
expect(badge).toBeInTheDocument();
});
it('renders a "Last error" line when last_error is populated', async () => {
frontend/src/utils/errorDocsMap.test.ts
+8
View File
@@ -98,6 +98,14 @@ describe('errorDocsMap', () => {
expect(classifyError(new Error('Gatekeeper blocked the launch'))).toBe(
'GATEKEEPER_QUARANTINE',
);
// Issue #72: macOS reports "app is damaged" in English and "已损坏" in
// localized Chinese builds — both should land on the same docs page.
expect(classifyError(new Error('OmniVoice Studio is damaged'))).toBe(
'GATEKEEPER_QUARANTINE',
);
expect(classifyError(new Error('OmniVoice Studio已损坏，无法打开'))).toBe(
'GATEKEEPER_QUARANTINE',
);
});
it('classifyError returns null on unknown messages', () => {
frontend/src/utils/errorDocsMap.ts
+9 -1
View File
@@ -68,7 +68,15 @@ export function classifyError(error: unknown): ErrorClass | null {
if (/webkit/.test(lower) || /white\s*screen/.test(lower)) {
return 'APPIMAGE_WEBKIT_WHITESCREEN';
}
if (/quarantine/.test(lower) || /gatekeeper/.test(lower)) return 'GATEKEEPER_QUARANTINE';
// Gatekeeper match: includes the literal "is damaged" macOS phrasing
// (English + Chinese 已损坏 per issue #72). Lower-case test is safe;
// '已损坏' is unaffected by lowercasing.
if (
/quarantine/.test(lower)
|| /gatekeeper/.test(lower)
|| /\bdamaged\b/.test(lower)
|| /已损坏/.test(message || '')
) return 'GATEKEEPER_QUARANTINE';
return null;
}
package.json
+4 -4
View File
@@ -5,7 +5,7 @@
"packageManager": "bun@1.3.11",
"scripts": {
"setup:api": "uv sync && uv run python scripts/setup.py",
"dev:api": "uv run uvicorn main:app --app-dir backend --host 0.0.0.0 --port 3900 --reload",
"dev:api": "uv run uvicorn main:app --app-dir backend --host 0.0.0.0 --port 3900 --reload --reload-dir backend",
"dev:frontend": "bun run --cwd frontend dev",
"dev:desktop": "bun run --cwd frontend desktop",
"wait:api": "wait-on -t 300000 http-get://localhost:3900/system/info",
@@ -32,9 +32,9 @@
"devDependencies": {
"concurrently": "^9.2.1",
"kill-port-process": "^4.0.2",
"playwright": "^1.59.1",
"turbo": "^2.9.7",
"playwright": "^1.60.0",
"turbo": "^2.9.15",
"typescript": "^6.0.3",
"wait-on": "^9.0.5"
"wait-on": "^9.0.10"
}
}
scripts/build_demos.sh
Executable
+256
View File
@@ -0,0 +1,256 @@
#!/bin/bash
# Build all demo assets for OmniVoice Studio v0.3.0.
#
# Two render paths:
# 1. macOS `say` (default) — fast, deterministic, ships immediately.
# Used to bootstrap the demo bundle so v0.3.0 has working demos on day one.
# 2. OmniVoice engine (--engine omnivoice) — production-quality re-render
# once model weights are cached. Recipes documented but not executed
# until the user opts in.
#
# All outputs are committed to the repo so end-users never need to re-render.
#
# Usage:
# scripts/build_demos.sh # render via `say`, overwrite all assets
# scripts/build_demos.sh --skip-existing # only render files that don't exist
# scripts/build_demos.sh --engine omnivoice # re-render via the real engine
# # (requires .venv + weights)
#
# Outputs land in:
# backend/assets/samples/demo_voice.wav (clone reference)
# backend/assets/samples/demo_clone_output.wav (clone pre-rendered out)
# backend/assets/samples/voice_design/demo_voice_design_*.wav (7 design presets)
# backend/assets/samples/dictation/{en_conversational,en_technical,fr_reservation}.wav
#
# License: all `say`-rendered output is synthetic speech from Apple's bundled
# TTS voices, redistributable under the OmniVoice MIT license per Apple's
# Voices for Accessibility EULA. No third-party voice IP is used.
set -e
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
SAMPLES_DIR="${REPO_ROOT}/backend/assets/samples"
DESIGN_DIR="${SAMPLES_DIR}/voice_design"
DICT_DIR="${SAMPLES_DIR}/dictation"
ENGINE="say"
SKIP_EXISTING=0
while [ $# -gt 0 ]; do
case "$1" in
--engine) ENGINE="$2"; shift 2 ;;
--skip-existing) SKIP_EXISTING=1; shift ;;
--help|-h)
sed -n '/^#/p' "$0" | head -40
exit 0 ;;
*) echo "Unknown arg: $1" >&2; exit 2 ;;
esac
done
if [ "$ENGINE" = "omnivoice" ]; then
# Delegate to the Python script that talks to the real engine.
PY_ARGS=""
[ "$SKIP_EXISTING" = 1 ] && PY_ARGS="--skip-existing"
echo "Rendering cloning + voice-design demos via OmniVoice engine…"
if [ -d "${REPO_ROOT}/.venv" ]; then
"${REPO_ROOT}/.venv/bin/python" "${REPO_ROOT}/scripts/render_demos_omnivoice.py" $PY_ARGS
else
echo "WARN: .venv missing; trying system python3" >&2
python3 "${REPO_ROOT}/scripts/render_demos_omnivoice.py" $PY_ARGS
fi
echo ""
echo "Note: dictation samples are still rendered via 'say' — re-running for them now."
# Fall through to render dictation; --skip-existing will preserve the
# OmniVoice-rendered cloning + design outputs we just produced.
SKIP_EXISTING=1
ENGINE="say"
fi
if ! command -v say >/dev/null; then
echo "ERROR: 'say' not found. This script currently requires macOS." >&2
echo "TODO: add espeak-ng path for Linux contributors." >&2
exit 1
fi
if ! command -v ffmpeg >/dev/null; then
echo "ERROR: 'ffmpeg' not found. Install via 'brew install ffmpeg'." >&2
exit 1
fi
# We use plain heredocs + python3 for JSON, so bash 3.2 (macOS default) is
# fine. No need for bash 4 features like ${var@Q} or associative arrays.
# ── render(voice, text, out_path, sample_rate_hz) ─────────────────────────
render() {
local voice="$1" text="$2" out="$3" sr="${4:-24000}"
if [ "$SKIP_EXISTING" = 1 ] && [ -f "$out" ]; then
echo " · skip (exists): $out"
return
fi
local tmp_aiff
tmp_aiff="$(mktemp -t omni-demo).aiff"
say -v "$voice" -o "$tmp_aiff" "$text"
ffmpeg -y -loglevel error -i "$tmp_aiff" \
-ar "$sr" -ac 1 -sample_fmt s16 "$out"
rm -f "$tmp_aiff"
local size
size="$(du -h "$out" | awk '{print $1}')"
echo " ✓ $(basename "$out") ($size, $voice @ ${sr}Hz)"
}
echo ""
echo "── Voice cloning demo (24kHz mono 16-bit) ─────────────────"
# Reference clip — replaces the existing 3-second "bleep" file.
# Voice: Samantha (en_US adult female, the macOS default — clean, neutral,
# warm). Reference text from the cloning spec.
render "Samantha" \
"Hi, I'm the OmniVoice demo voice. Everything you hear me say from now on was synthesized on your own machine. No cloud, no account, just you and the model." \
"${SAMPLES_DIR}/demo_voice.wav" 24000
# Pre-rendered clone output — same voice, different text. Used when the user
# hits Preview but no TTS engine has weights cached yet.
render "Samantha" \
"Welcome aboard. I was just a three-second clip a moment ago. Now I can say anything you'd like, in your voice or mine." \
"${SAMPLES_DIR}/demo_clone_output.wav" 24000
echo ""
echo "── Voice design demo (24kHz mono 16-bit, 7 presets) ───────"
# Each preset showcases a different axis (age / register / accent / use case).
# Voice picks aim for distinctness on first listen, not 1:1 spec fidelity —
# they'll be re-rendered via the real engine later. The two character voices
# (Captain Crusty + Junior Quacks) capture the cartoon-style vibe the user
# wanted without touching copyrighted IP.
render "Daniel" \
"The clock tower struck thirteen, and for the first time in her life, Eleanor wondered if she had been counting wrong all along." \
"${DESIGN_DIR}/demo_voice_design_audiobook_uk_narrator.wav" 24000
render "Ralph" \
"Good evening. Topping our broadcast tonight: scientists at the coastal observatory have confirmed the signal is, in fact, repeating." \
"${DESIGN_DIR}/demo_voice_design_us_news_anchor.wav" 24000
render "Rishi" \
"Thank you for calling OmniVoice support. I can see your account here. Let's get this sorted out together." \
"${DESIGN_DIR}/demo_voice_design_indian_support_agent.wav" 24000
# Captain Crusty — gravelly cartoon-sailor villain. Inspired by the "tough old
# seafarer" archetype, original character. Bad News is a low novelty voice
# that lands the gravelly-villain register.
render "Bad News" \
"You came a long way for an answer you already had. Sit. The fire is warm, and the truth is not." \
"${DESIGN_DIR}/demo_voice_design_gravelly_villain.wav" 24000
render "Karen" \
"Right, so here's the wild bit. Nobody told the engineers the satellite was supposed to be in orbit by Tuesday. Tuesday came and went." \
"${DESIGN_DIR}/demo_voice_design_aussie_podcaster.wav" 24000
# Junior Quacks — anxious cartoon-nephew character. "Bahh" is a squawky
# novelty voice that approximates the high-strung sidekick archetype.
render "Bahh" \
"Once, in a town where every street was named after a kind of bread, a small fox decided she was going to learn to play the cello." \
"${DESIGN_DIR}/demo_voice_design_bedtime_storyteller.wav" 24000
render "Tingting" \
"今天天气巴适得很，我们去吃火锅嘛！记得多加点豆芽。" \
"${DESIGN_DIR}/demo_voice_design_mandarin_sichuan.wav" 24000
echo ""
echo "── Dictation demo (16kHz mono 16-bit, 3 scripts) ──────────"
# 16 kHz matches WhisperX's preferred ingest rate.
render "Samantha" \
"Schedule a meeting with Pat for Tuesday at three PM and remind me to bring the quarterly report." \
"${DICT_DIR}/en_conversational.wav" 16000
render "Fred" \
"Patch the WebGPU shader in renderer dot tsx, then bump pnpm to nine point fifteen and rerun the Vitest suite." \
"${DICT_DIR}/en_technical.wav" 16000
render "Thomas" \
"Bonjour, je voudrais réserver une table pour deux personnes à vingt heures." \
"${DICT_DIR}/fr_reservation.wav" 16000
echo ""
echo "── Manifest ───────────────────────────────────────────────"
cat > "${SAMPLES_DIR}/demo/manifest.json" <<EOF
{
"version": "0.3.0",
"rendered_by": "macOS say (bootstrap)",
"rendered_at": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
"license": "MIT (synthetic speech, no third-party voice IP)",
"assets": {
"clone": {
"reference": "samples/demo_voice.wav",
"prerendered_output": "samples/demo_clone_output.wav"
},
"voice_design": {
"audiobook_uk_narrator": {
"wav": "samples/voice_design/demo_voice_design_audiobook_uk_narrator.wav",
"display_name": "The Librarian",
"instruct": "female, middle-aged, low pitch, british accent",
"use_case": "Audiobook narrator"
},
"us_news_anchor": {
"wav": "samples/voice_design/demo_voice_design_us_news_anchor.wav",
"display_name": "The Anchor",
"instruct": "male, middle-aged, moderate pitch, american accent",
"use_case": "News broadcast"
},
"indian_support_agent": {
"wav": "samples/voice_design/demo_voice_design_indian_support_agent.wav",
"display_name": "The Helpdesk",
"instruct": "female, young adult, moderate pitch, indian accent",
"use_case": "Customer-service / IVR"
},
"gravelly_villain": {
"wav": "samples/voice_design/demo_voice_design_gravelly_villain.wav",
"display_name": "Captain Crusty",
"instruct": "male, elderly, very low pitch",
"use_case": "Video-game NPC / cartoon villain"
},
"aussie_podcaster": {
"wav": "samples/voice_design/demo_voice_design_aussie_podcaster.wav",
"display_name": "The Podcaster",
"instruct": "female, young adult, high pitch, australian accent",
"use_case": "Podcast / explainer"
},
"bedtime_storyteller": {
"wav": "samples/voice_design/demo_voice_design_bedtime_storyteller.wav",
"display_name": "Junior Quacks",
"instruct": "young, anxious, high pitch, squawky",
"use_case": "Cartoon sidekick / children's storyteller"
},
"mandarin_sichuan": {
"wav": "samples/voice_design/demo_voice_design_mandarin_sichuan.wav",
"display_name": "The Sichuan Friend",
"instruct": "female, young adult, moderate pitch, 四川话",
"use_case": "Non-English showcase"
}
},
"dictation": {
"en_conversational": {
"wav": "samples/dictation/en_conversational.wav",
"expected_transcript": "Schedule a meeting with Pat for Tuesday at three PM and remind me to bring the quarterly report.",
"language": "en"
},
"en_technical": {
"wav": "samples/dictation/en_technical.wav",
"expected_transcript": "Patch the WebGPU shader in renderer.tsx, then bump pnpm to nine point fifteen and rerun the Vitest suite.",
"language": "en"
},
"fr_reservation": {
"wav": "samples/dictation/fr_reservation.wav",
"expected_transcript": "Bonjour, je voudrais réserver une table pour deux personnes à vingt heures.",
"language": "fr"
}
}
},
"rerender_with_omnivoice": "scripts/build_demos.sh --engine omnivoice (TODO)"
}
EOF
echo " ✓ demo/manifest.json"
echo ""
echo "── Totals ─────────────────────────────────────────────────"
du -sh "${SAMPLES_DIR}" | awk '{print " Bundle size: " $1}'
find "${SAMPLES_DIR}" -name "*.wav" | wc -l | awk '{print " WAV count: " $1}'
echo ""
echo "Done. To re-render with the OmniVoice engine later:"
echo " scripts/build_demos.sh --engine omnivoice"
scripts/build_dub_demo.sh
Executable
+146
View File
@@ -0,0 +1,146 @@
#!/bin/bash
# Build the synthetic dubbing demo: one English source video + 4 dubbed
# variants. Each video is a 720p H.264 file with the showwaves audio
# visualizer over a dark gradient background — no copyrighted footage,
# no third-party voice IP.
#
# Output layout (under backend/assets/demo/dubbing/):
# source.mp4 English audio + visualizer
# dubbed_es.mp4 Spanish (Mónica)
# dubbed_fr.mp4 French (Thomas)
# dubbed_zh.mp4 Mandarin (Tingting)
# dubbed_ja.mp4 Japanese (Kyoko)
# *.srt transcripts for each
# manifest.json single source of truth the frontend reads
#
# Bundle target: ~5 MB per mp4 × 5 files = ~25 MB. Well under the 45 MB
# cap from the dubbing-demo design spec.
set -e
REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
OUT_DIR="${REPO_ROOT}/backend/assets/demo/dubbing"
mkdir -p "$OUT_DIR"
# Compatibility: must run on macOS default bash 3.2 (no associative arrays,
# no ${var@Q}). We sidestep both by passing scripts as env vars to python3
# below. Just guard the basics.
if ! command -v ffmpeg >/dev/null || ! command -v say >/dev/null; then
echo "ERROR: need both ffmpeg and macOS 'say'." >&2
exit 1
fi
if ! command -v python3 >/dev/null; then
echo "ERROR: python3 not found — needed to emit manifest.json." >&2
exit 1
fi
# ── Scripts: source + 4 translations ─────────────────────────────────────
# Each is ~20-25 seconds when spoken — short enough to keep the demo snappy,
# long enough to show off pacing + accent. All translations preserve meaning;
# they were drafted manually so the script is reproducible.
EN_SCRIPT="OmniVoice Studio is a desktop app for voice cloning, video dubbing, and voice design. It runs entirely on your machine. No accounts, no cloud, no API keys. Just open the app and start creating."
ES_SCRIPT="OmniVoice Studio es una aplicación de escritorio para clonación de voz, doblaje de vídeo y diseño de voz. Funciona completamente en tu máquina. Sin cuentas, sin nube, sin claves de API. Solo abre la aplicación y comienza a crear."
FR_SCRIPT="OmniVoice Studio est une application de bureau pour le clonage de voix, le doublage vidéo et la conception vocale. Elle fonctionne entièrement sur votre machine. Pas de compte, pas de cloud, pas de clé d'API. Ouvrez l'application et commencez à créer."
ZH_SCRIPT="OmniVoice Studio 是一款桌面应用，用于语音克隆、视频配音和声音设计。它完全在你的电脑上运行。无需账户，无需云端，无需 API 密钥。打开应用即可开始创作。"
JA_SCRIPT="OmniVoice Studioは、ボイスクローン、ビデオ吹き替え、ボイスデザインのためのデスクトップアプリです。すべてお使いのコンピュータ上で動作します。アカウント、クラウド、APIキーは不要です。アプリを開けば、すぐに制作を始められます。"
# ── render_lang(code, voice, text) ──────────────────────────────────────
# Produces: $OUT_DIR/{source|dubbed_$code}.mp4 + matching .srt
# Uses showwaves to render a styled audio visualizer over a dark backdrop.
# Resolution 1280x720 keeps each output around 4-6 MB at CRF 28.
render_lang() {
local code="$1" voice="$2" text="$3"
local stem
if [ "$code" = "en" ]; then stem="source"; else stem="dubbed_${code}"; fi
local aiff="${OUT_DIR}/${stem}.aiff"
local wav="${OUT_DIR}/${stem}.wav"
local mp4="${OUT_DIR}/${stem}.mp4"
local srt="${OUT_DIR}/${stem}.srt"
say -v "$voice" -o "$aiff" "$text"
ffmpeg -y -loglevel error -i "$aiff" -ar 44100 -ac 1 "$wav"
rm -f "$aiff"
# Visual: showwaves p2p mode over a dark gradient with a colored line.
# `nullsrc` + `geq` would let us make a static gradient backdrop, but
# simpler is `color` with overlay'd waveform.
ffmpeg -y -loglevel error \
-i "$wav" \
-filter_complex "[0:a]showwaves=s=1280x720:mode=p2p:rate=30:colors=0xf3a5b6[wave]; \
color=c=0x1d2021:s=1280x720:d=60[bg]; \
[bg][wave]overlay=format=auto,format=yuv420p,trim=duration=$(ffprobe -v error -show_entries format=duration -of default=nw=1:nk=1 "$wav")[v]" \
-map "[v]" -map 0:a \
-c:v libx264 -crf 28 -preset medium \
-c:a aac -b:a 96k -ac 1 \
-shortest \
"$mp4"
rm -f "$wav"
# Single-cue SRT — entire utterance as one block, "good enough" for the
# demo player which shows it as a caption strip rather than karaoke.
local dur
dur=$(ffprobe -v error -show_entries format=duration -of default=nw=1:nk=1 "$mp4")
local end_ts
end_ts=$(awk -v d="$dur" 'BEGIN { h=int(d/3600); m=int((d%3600)/60); s=d-int(d/60)*60; printf "%02d:%02d:%06.3f", h, m, s }' | tr '.' ',')
cat > "$srt" <<EOF
1
00:00:00,000 --> ${end_ts}
${text}
EOF
local size
size=$(du -h "$mp4" | awk '{print $1}')
echo " ✓ ${stem}.mp4 ($size, $voice)"
}
echo "── Source video (English) ────────────────────────────────"
render_lang en Samantha "$EN_SCRIPT"
echo ""
echo "── Dubbed videos (4 languages) ───────────────────────────"
render_lang es Mónica "$ES_SCRIPT"
render_lang fr Thomas "$FR_SCRIPT"
render_lang zh Tingting "$ZH_SCRIPT"
render_lang ja Kyoko "$JA_SCRIPT"
echo ""
echo "── Manifest ──────────────────────────────────────────────"
# bash 3.2 (default on macOS) lacks ${var@Q}; pass scripts as env vars to
# Python so escaping of unicode + quotes is handled correctly.
OUT_DIR="$OUT_DIR" \
EN_SCRIPT="$EN_SCRIPT" ES_SCRIPT="$ES_SCRIPT" FR_SCRIPT="$FR_SCRIPT" \
ZH_SCRIPT="$ZH_SCRIPT" JA_SCRIPT="$JA_SCRIPT" \
python3 - <<'PY'
import json, datetime, os
out = os.path.join(os.environ["OUT_DIR"], "manifest.json")
manifest = {
"version": "0.3.0",
"rendered_by": "macOS say + ffmpeg showwaves (bootstrap)",
"rendered_at": datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ"),
"license": "MIT (synthetic, no third-party IP)",
"source": {
"code": "en", "label": "English",
"video": "source.mp4", "srt": "source.srt",
"script": os.environ["EN_SCRIPT"],
},
"dubbed": [
{"code":"es","label":"Español","video":"dubbed_es.mp4","srt":"dubbed_es.srt","dir":"ltr","script":os.environ["ES_SCRIPT"]},
{"code":"fr","label":"Français","video":"dubbed_fr.mp4","srt":"dubbed_fr.srt","dir":"ltr","script":os.environ["FR_SCRIPT"]},
{"code":"zh","label":"中文","video":"dubbed_zh.mp4","srt":"dubbed_zh.srt","dir":"ltr","script":os.environ["ZH_SCRIPT"]},
{"code":"ja","label":"日本語","video":"dubbed_ja.mp4","srt":"dubbed_ja.srt","dir":"ltr","script":os.environ["JA_SCRIPT"]},
],
}
with open(out, "w", encoding="utf-8") as f:
json.dump(manifest, f, ensure_ascii=False, indent=2)
PY
echo " ✓ manifest.json"
echo ""
du -sh "$OUT_DIR" | awk '{print " Bundle: " $1}'
echo "Done."
scripts/install.sh
+22 -2
View File
@@ -321,9 +321,29 @@ step "gpu" "$GPU_INFO"
step "python" "syncing dependencies..."
note "This can take 5–10 min the first time (torch + torchaudio + demucs...)"
# Create venv with the target Python version if it doesn't exist
# Restricted-network support: when OMNIVOICE_REGION is set to china/russia/restricted,
# route python-build-standalone downloads through ghproxy.net. See issues #57, #60.
# Honors any existing UV_* env vars (power-user override).
case "${OMNIVOICE_REGION:-}" in
china|russia|restricted)
: "${UV_PYTHON_INSTALL_MIRROR:=https://ghproxy.net/https://github.com/astral-sh/python-build-standalone/releases/download}"
export UV_PYTHON_INSTALL_MIRROR
note "Using ghproxy.net mirror for Python download (OMNIVOICE_REGION=${OMNIVOICE_REGION})"
;;
esac
: "${UV_HTTP_TIMEOUT:=120}"
: "${UV_HTTP_RETRIES:=5}"
export UV_HTTP_TIMEOUT UV_HTTP_RETRIES
# Create venv with the target Python version if it doesn't exist.
# If the managed-python download fails (restricted network, mirror unreachable),
# fall back to the user's system Python.
if [ ! -d .venv ]; then
uv venv --python "$PYTHON_VERSION"
if ! uv venv --python "$PYTHON_VERSION"; then
warn "uv venv failed (likely Python download). Retrying with system Python..."
uv venv --python "$PYTHON_VERSION" --python-preference only-system \
|| die "uv venv failed: install Python $PYTHON_VERSION system-wide, set OMNIVOICE_REGION=china|russia|restricted to route through a mirror, or check your network."
fi
fi
# Sync all deps from pyproject.toml + uv.lock
scripts/render_demos_omnivoice.py
Executable
+220
View File
@@ -0,0 +1,220 @@
#!/usr/bin/env python3
"""Re-render the demo bundle using the real OmniVoice TTS engine.
This is the production-quality counterpart to scripts/build_demos.sh, which
uses macOS `say` to bootstrap the demo bundle. Run this once on a machine
with OmniVoice model weights cached (typically your dev box) to replace the
`say`-rendered placeholders with engine output. Commit the resulting WAVs.
Prerequisites:
* The project's .venv exists and is activated (`uv sync`).
* OmniVoice model weights cached under $HF_HUB_CACHE (the first
`model.generate()` call will download them otherwise — ~5 GB).
* Run from the repo root: `python3 scripts/render_demos_omnivoice.py`.
What it produces:
* backend/assets/samples/demo_voice.wav (clone reference)
* backend/assets/samples/demo_clone_output.wav (clone pre-rendered)
* backend/assets/samples/voice_design/demo_voice_design_<slug>.wav (7)
* Updated manifest with rendered_by="omnivoice@<git_sha>"
Not regenerated by this script:
* backend/assets/samples/dictation/*.wav — those need to be human speech
or human-quality TTS for the WhisperX replay path to demonstrate real
transcription. `say` output is fine; engine TTS is overkill and slow.
* backend/assets/demo/dubbing/*.mp4 — see scripts/build_dub_demo.sh.
Reproducibility:
* seed=42 fixed across all renders so re-running the script regenerates
the same audio byte-for-byte (modulo torch nondeterminism).
"""
from __future__ import annotations
import argparse
import json
import os
import subprocess
import sys
import time
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
BACKEND_DIR = REPO_ROOT / "backend"
SAMPLES_DIR = BACKEND_DIR / "assets" / "samples"
VOICE_DESIGN_DIR = SAMPLES_DIR / "voice_design"
# Make `backend/` importable so we can pull personalities + the engine.
sys.path.insert(0, str(BACKEND_DIR))
# Cloning demo — must match scripts/build_demos.sh exactly so the manifest
# stays in sync with what the bootstrap script produced.
CLONE_REF_TEXT = (
"Hi, I'm the OmniVoice demo voice. Everything you hear me say from now on "
"was synthesized on your own machine. No cloud, no account, just you and "
"the model."
)
CLONE_OUTPUT_TEXT = (
"Welcome aboard. I was just a three-second clip a moment ago. Now I can "
"say anything you'd like, in your voice or mine."
)
def _git_sha() -> str:
try:
return subprocess.check_output(
["git", "rev-parse", "--short", "HEAD"],
cwd=REPO_ROOT, text=True,
).strip()
except Exception:
return "unknown"
def _save_wav(audio_tensor, sample_rate: int, out_path: Path):
"""Save a torch tensor (C, T) or (T,) to a 16-bit PCM WAV."""
import torch
import torchaudio
if audio_tensor.dim() == 1:
audio_tensor = audio_tensor.unsqueeze(0)
# Ensure mono — most OmniVoice outputs are mono already.
if audio_tensor.shape[0] > 1:
audio_tensor = audio_tensor.mean(dim=0, keepdim=True)
out_path.parent.mkdir(parents=True, exist_ok=True)
# Normalize to safe headroom and clip — matches the `say` output level.
peak = audio_tensor.abs().max().item()
if peak > 0:
audio_tensor = audio_tensor / peak * 0.97
torchaudio.save(
str(out_path),
audio_tensor.to(torch.float32),
sample_rate,
encoding="PCM_S", bits_per_sample=16,
)
def render_cloning(model, args):
"""Render the cloning demo: reference clip + pre-rendered output.
The reference clip is itself synthesized — chicken-and-egg, but the
OmniVoice engine in non-zero-shot mode (no ref_audio) accepts a plain
`instruct=` taxonomy string and produces a clean voice.
"""
print("── Cloning demo ─────────────────────────────────────")
sr = getattr(model, "sampling_rate", 24000)
# 1) Reference clip — synthesized with a "neutral female narrator"
# instruct so the timbre is the same across re-renders.
out_ref = SAMPLES_DIR / "demo_voice.wav"
if args.skip_existing and out_ref.exists():
print(f" · skip (exists): {out_ref.name}")
else:
audios = model.generate(
text=CLONE_REF_TEXT,
instruct="female, middle-aged, moderate pitch, american accent",
num_step=24,
)
_save_wav(audios[0], sr, out_ref)
print(f" ✓ {out_ref.name} ({sr} Hz, omnivoice)")
# 2) Pre-rendered clone output — same voice, different text. Use the
# reference clip we just rendered as the speaker reference so this
# actually demonstrates cloning rather than independent synthesis.
out_clone = SAMPLES_DIR / "demo_clone_output.wav"
if args.skip_existing and out_clone.exists():
print(f" · skip (exists): {out_clone.name}")
else:
audios = model.generate(
text=CLONE_OUTPUT_TEXT,
ref_audio=str(out_ref),
ref_text=CLONE_REF_TEXT,
num_step=24,
)
_save_wav(audios[0], sr, out_clone)
print(f" ✓ {out_clone.name} ({sr} Hz, cloned)")
def render_voice_design(model, args):
"""Re-render the 7 voice-design preset previews."""
print("\n── Voice design presets ─────────────────────────────")
from core.personalities import PERSONALITIES
sr = getattr(model, "sampling_rate", 24000)
demos = [p for p in PERSONALITIES if p.get("is_demo")]
if not demos:
print(" No is_demo presets found in personalities.py")
return
for preset in demos:
slug = preset["id"]
out = VOICE_DESIGN_DIR / f"demo_voice_design_{slug}.wav"
if args.skip_existing and out.exists():
print(f" · skip (exists): {out.name}")
continue
audios = model.generate(
text=preset["script"],
instruct=preset["instruct"],
language=preset.get("language"),
num_step=24,
)
_save_wav(audios[0], sr, out)
print(f" ✓ {out.name} ({preset['name']})")
def update_manifest(args):
"""Update the existing manifest with rendered_by + rendered_at."""
print("\n── Manifest ─────────────────────────────────────────")
mpath = SAMPLES_DIR / "demo" / "manifest.json"
if not mpath.exists():
print(f" ! manifest not found at {mpath} — run scripts/build_demos.sh first")
return
data = json.loads(mpath.read_text())
data["rendered_by"] = f"omnivoice@{_git_sha()}"
data["rendered_at"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
mpath.write_text(json.dumps(data, indent=2, ensure_ascii=False))
print(f" ✓ {mpath.name}")
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--skip-existing", action="store_true",
help="Don't re-render files that already exist on disk.",
)
parser.add_argument(
"--only", choices=["cloning", "design", "manifest"],
help="Render only a subset (default: all).",
)
args = parser.parse_args()
print("Loading OmniVoice engine (this can take 30-60 s on first run)…")
try:
import asyncio
from services.model_manager import get_model
try:
asyncio.get_running_loop()
raise RuntimeError("Run this script outside an async context.")
except RuntimeError:
model = asyncio.run(get_model())
except Exception as e:
print(f"\nERROR: Could not load OmniVoice engine: {e}\n")
print("Check that:")
print(" 1. You're running inside the project venv (uv sync first).")
print(" 2. The omnivoice package is importable: `python -c 'import omnivoice'`.")
print(" 3. Model weights are downloaded (~5 GB on first synthesis).")
sys.exit(1)
print("Engine loaded.\n")
if args.only in (None, "cloning"):
render_cloning(model, args)
if args.only in (None, "design"):
render_voice_design(model, args)
if args.only in (None, "manifest"):
update_manifest(args)
print("\nDone. Re-run scripts/build_demos.sh to regenerate dictation samples.")
if __name__ == "__main__":
main()
tests/test_dub_timing_strategy.py
+154
View File
@@ -0,0 +1,154 @@
"""Regression tests for the dub timing-strategy feature.
Covers:
- DubRequest schema defaults — `timing_strategy="concise"` is the new
safe default; back-compat `slot_fit="time_stretch"` is still accepted.
- `_build_video_stretch_filter_graph` — builds a deterministic ffmpeg
filter_complex graph that splits the source video into per-segment
chunks, setpts each, and concats. Includes gap and pre/tail handling.
- `_video_stretch_plan_for` — only returns a plan when the job's
`timing_strategy == "stretch_video"` AND a plan was persisted for
that lang_code, otherwise None.
Why these helpers and not the full mix loop: the mix loop lives inside
the `dub_generate` async generator, which needs a loaded TTS model to
exercise end-to-end. The helpers tested here are the new pure logic the
mode introduced — if they hold, the integration is essentially correct
modulo the actual ffmpeg call (which is exercised by the existing dub
smoke tests).
"""
from __future__ import annotations
import pytest
from schemas.requests import DubRequest, DubSegment
from api.routers.dub_export import (
_build_video_stretch_filter_graph,
_video_stretch_plan_for,
)
# ── DubRequest schema defaults ────────────────────────────────────────
def _minimal_segs():
return [DubSegment(start=0.0, end=1.0, text="hi")]
def test_dubrequest_defaults_to_concise():
req = DubRequest(segments=_minimal_segs())
assert req.timing_strategy == "concise"
assert req.overflow_budget_s == 0.0
# slot_fit default retained for legacy callers that still send it.
assert req.slot_fit == "time_stretch"
@pytest.mark.parametrize("strategy", ["concise", "stretch_video", "strict_slot"])
def test_dubrequest_accepts_all_three_strategies(strategy):
req = DubRequest(segments=_minimal_segs(), timing_strategy=strategy)
assert req.timing_strategy == strategy
def test_dubrequest_rejects_unknown_strategy():
with pytest.raises(Exception):
DubRequest(segments=_minimal_segs(), timing_strategy="warp_speed")
# ── _build_video_stretch_filter_graph ─────────────────────────────────
def test_filter_graph_empty_plan_returns_empty_string_and_passthrough_label():
graph, label = _build_video_stretch_filter_graph(plan=[], orig_dur=10.0)
assert graph == ""
assert label == "[0:v]"
def test_filter_graph_single_segment_no_pre_or_tail():
"""If the seg covers the whole video, no extra chunks are emitted."""
plan = [{
"orig_start": 0.0, "orig_end": 5.0,
"new_start": 0.0, "new_end": 6.0,
"stretch_ratio": 1.2,
}]
graph, label = _build_video_stretch_filter_graph(plan, orig_dur=5.0)
assert label == "[vstretched]"
# One split node, one trim+setpts node, one concat — concat n=1.
assert "split=1" in graph
assert "trim=start=0.0000:end=5.0000" in graph
assert "setpts=1.200000*PTS" in graph
assert "concat=n=1:v=1:a=0[vstretched]" in graph
def test_filter_graph_pre_roll_gap_and_tail_all_at_native_rate():
"""Pre-roll, inter-segment gap, and tail should each be a 1.0x chunk."""
plan = [
{"orig_start": 1.0, "orig_end": 3.0, "new_start": 1.0, "new_end": 3.5, "stretch_ratio": 1.25},
{"orig_start": 4.0, "orig_end": 6.0, "new_start": 4.5, "new_end": 6.7, "stretch_ratio": 1.10},
]
graph, label = _build_video_stretch_filter_graph(plan, orig_dur=8.0)
assert label == "[vstretched]"
# Chunks: [0,1]@1.0 (pre-roll), [1,3]@1.25, [3,4]@1.0 (gap), [4,6]@1.10, [6,8]@1.0 (tail)
assert "split=5" in graph
assert "concat=n=5:v=1:a=0[vstretched]" in graph
# Native-rate chunks are emitted with setpts=1.000000 so the graph stays uniform.
assert graph.count("setpts=1.000000*PTS") == 3
assert "setpts=1.250000*PTS" in graph
assert "setpts=1.100000*PTS" in graph
def test_filter_graph_chains_after_subtitle_filter():
"""When `in_label` is supplied, the graph reads from that label instead of
the raw source — used when subtitles burn into [vsub] first."""
plan = [{
"orig_start": 0.0, "orig_end": 2.0,
"new_start": 0.0, "new_end": 2.5,
"stretch_ratio": 1.25,
}]
graph, label = _build_video_stretch_filter_graph(
plan, orig_dur=2.0, in_label="[vsub]",
)
assert label == "[vstretched]"
assert graph.startswith("[vsub]split=1")
# ── _video_stretch_plan_for ───────────────────────────────────────────
def test_plan_for_returns_none_when_strategy_is_concise():
job = {
"timing_strategy": "concise",
"video_stretch_plans": {"bn": {"plan": [{"orig_start": 0.0, "orig_end": 1.0,
"new_start": 0.0, "new_end": 1.0,
"stretch_ratio": 1.0}]}},
}
assert _video_stretch_plan_for(job, "bn") is None
def test_plan_for_returns_none_when_plan_missing_for_lang():
job = {
"timing_strategy": "stretch_video",
"video_stretch_plans": {"de": {"plan": [{"orig_start": 0.0, "orig_end": 1.0,
"new_start": 0.0, "new_end": 1.0,
"stretch_ratio": 1.0}]}},
}
assert _video_stretch_plan_for(job, "bn") is None
def test_plan_for_returns_entry_when_strategy_matches_and_plan_exists():
entry = {
"plan": [{"orig_start": 0.0, "orig_end": 1.0,
"new_start": 0.0, "new_end": 1.2,
"stretch_ratio": 1.2}],
"total_duration": 1.2,
"orig_duration": 1.0,
}
job = {"timing_strategy": "stretch_video", "video_stretch_plans": {"bn": entry}}
got = _video_stretch_plan_for(job, "bn")
assert got is entry
def test_plan_for_returns_none_when_video_stretch_plans_absent():
"""A job that ran strict_slot then was upgraded won't carry the plans
dict; the helper must tolerate that and return None."""
job = {"timing_strategy": "stretch_video"}
assert _video_stretch_plan_for(job, "bn") is None