Compare commits

..
14 Commits
Author SHA1 Message Date
fd7d20fe1e release: freeze v0.3.10 — version bump, lockfiles, changelog (#954)
package.json (source of truth) + the three mirrors -> 0.3.10, in lockstep;
Cargo.lock/uv.lock/bun.lock regenerated (one line each; bun --frozen-lockfile
verified). CHANGELOG [Unreleased] -> [0.3.10] — 2026-07-05 with the release
headline; nine fixes since v0.3.9, mostly same-day field-report turnarounds.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 00:57:17 +05:30
27ff5846c2 fix(audiobook): chapter render no longer crashes on mixed 1-D/2-D audio chunks (#897) (#953)
* fix(audiobook): chapter render no longer crashes on mixed 1-D/2-D audio chunks (#897)

Root cause: synthesize_chapter (backend/services/audiobook.py) built
inter-span pause silence as bare 1-D torch.zeros(n) while every real
engine's synth returns (1, samples) per the TTSBackend.generate contract
(OmniVoice's model.generate(...)[0] included) — so the chapter's final
hard concat in chunked_tts.concatenate_audio_chunks hit
torch.cat with mixed ranks and died with
'RuntimeError: Tensors must have same number of dimensions: got 1 and 2'.
Any chapter containing a [pause] span (Stories/audiobook longform)
crashed; existing tests missed it because their stub synth returned 1-D.
The crossfade branch had the same latent bug for mixed-rank chunks.

Fix, both layers:
- concatenate_audio_chunks now normalizes chunk shapes before any cat
  (_normalize_chunk_shapes): lower-rank chunks gain leading singleton
  dims to the highest rank present, then singleton channel dims
  broadcast to the widest channel count (mono follows stereo). Covers
  both the hard-cut and crossfade branches; homogeneous input passes
  through untouched, so all-1-D / all-2-D callers keep their exact
  output shapes. No future backend's output rank can re-break the join.
- synthesize_chapter materializes silence AFTER the loop, matching the
  rendered audio's channel dims / dtype / device — the same pattern
  generation.py's _render_with_pauses already uses for the single-shot
  path — so the data is rank-consistent at the source too. A
  silence-only chapter stays 1-D float32 as before.

Regression tests: mixed-rank hard-cut (both orders), mixed-rank
crossfade, mono->stereo broadcast, all-1-D/all-2-D shape stability, a
2-D-engine + [pause] chapter through synthesize_chapter (the exact #897
scenario), and a spy asserting the parts reaching the concat are
rank-homogeneous. All fail before the fix with the reported error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): add the audiobook pause-span concat fix under [Unreleased] (#953)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 00:40:35 +05:30
d14b37fab2 fix(dub): the speaker-count hint is honored on every diarization path + clone-purity guard (#952)
* fix(dub): the speaker-count hint is honored on every diarization path + clone-purity guard

The dub "Speakers" count reached _diarize() and then died on 3 of its 4
branches, so setting it changed nothing, speakers blended, and auto-clones
were cut from mixed-speaker audio ("made up" voices):

- FunASR inline-turns shortcut returned before the hint was ever consulted
  → now an explicit num_speakers routes the job through pyannote (the one
  engine that honors an exact count); turns stay the fast path only when no
  hint is set, and remain the fallback (with an honest "hint ignored"
  warning) when pyannote can't load or crashes mid-run.
- pyannote-unavailable fallback used a hardcoded 2-speaker silence-gap
  heuristic → assign_speakers_heuristic now takes num_speakers and cycles N
  labels on gap boundaries (1 → single speaker; None → legacy alternation),
  and the existing diarization warning says the hint is only approximately
  honored.
- pyannote-crash fallback dropped the hint the same way → same treatment.

No branch drops the hint silently anymore: every degraded path extends the
existing `warning` SSE payload (detail + a machine-readable speaker_hint
field) that the frontend already renders.

Parity + purity:
- POST /dub/transcribe/{job_id} (the CLI's endpoint) gains the same clamped
  num_speakers query param, forwarded to pyannote and the heuristic; the
  omnivoice-dub CLI gains --speakers N.
- Clone-purity guard: _pick_reference_slices rejects sub-1.5s slices, prefers
  slices not temporally adjacent (<0.3s) to another speaker's turn (scoring
  preference, not a hard filter), and extract_speaker_clones skips extraction
  entirely when labels came from the heuristic (labels_source kwarg threaded
  from _diarize; missing kwarg keeps the old behavior) — with a user-facing
  warning pointing at Settings → Models → pyannote.

Tests: fail-before/pass-after coverage in tests/test_speaker_hint.py (all
four _diarize branches driven through the real SSE stream), clone-purity
guards in tests/test_speaker_clone_purity.py, heuristic hint semantics in
tests/test_segmentation.py.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): add the speaker-hint + clone-purity fix under [Unreleased] (#952)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 22:53:47 +05:30
edf86c1800 fix(ui): dub timeline boxes can no longer flash invisible during playback (#373 class, completes #381) (#951)
* fix(ui): dub timeline boxes can no longer flash invisible during playback (#373 class, completes #381)

Root cause: the segment lane animated an inline
`transform: translateX(-scrollLeft)` on every playback tick, so Chromium
promoted it to a compositor layer — and on some Windows GPU/WebView2
driver combos, composited semi-transparent paints (the 0.45-alpha box
fills) flash invisible/visible while the layer moves, settling only when
paused. PR #381 removed `will-change` and raised the alpha, which only
dampened the symptom; the animated transform kept the lane composited.

Fix the class — no composited translucent paints on the lane, ever:

- Position boxes in pure layout: the lane transform is gone; each box's
  `left` is start·pxPerSec − scrollLeft (viewport coordinates). The
  virtualization window already derives from the same scrollLeft, so
  windowing stays consistent by construction. The selfScroll WebKit
  fallback keeps lane coordinates (its viewport is a real scroll
  container), unchanged.
- Belt-and-braces: REGION_COLORS are now fully opaque — each entry
  pre-blends the old 45% tint against the surface behind the lane
  (`--chrome-bg`, the .studio-panel background) via color-mix, which is
  pixel-identical to the previous alpha compositing (0.45·tint + 0.55·bg)
  in every theme, with zero alpha.

Regression tests (fail on pre-fix code): lane carries no transform at
rest and after a scroll update, box lefts are viewport-relative for a
scrolled view, no double-shift in the selfScroll fallback, and every
REGION_COLORS entry is alpha-free with the 45% ratio preserved.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): add the timeline-box compositor fix under [Unreleased] (#951)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 22:41:16 +05:30
437a995a0f fix(translate): Cinematic/Autofit can no longer invent dialogue — divergence guard + pinned temperature (#950)
Root cause (v0.3.9 field report): the refine paths had no-op output guards.
Cinematic's ADAPT step only checked _looks_like_target_script, which returns
True unconditionally for every Latin-script target (no _SCRIPT_RANGES entry)
— so any non-empty LLM reply (hallucinated dialogue, refusals, commentary,
or the REFLECT critique itself) shipped as the dub line. Autofit's
adjust_for_slot accepted ANY non-empty reply, and its best-candidate tracker
(closest rate_ratio to 1.0) actively selected the most-padded output, while
_EXPAND_PROMPT invited invention with no ceiling. Both call paths also ran
at the provider-default temperature 1.0, unlike the working Fast path which
pins 0.2.

The fix, class-level:
- Shared divergence guard translator.refine_output_ok (length window
  0.4–2.5x, env-tunable via OMNIVOICE_REFINE_RATIO_MIN/MAX, with an
  absolute cap for short references; target-script check; critique-echo
  detection). Rejected ADAPT output degrades to the literal with
  error="adapt-diverged" (wrong-script keeps its adapt-wrong-script:<lang>
  marker), riding the existing degradation machinery unchanged.
- Autofit validates every reply against the ORIGINAL input text (divergence
  compounds across attempts otherwise); rejected candidates are discarded
  (attempt burned, graceful degradation to the input preserved) with
  error="fit-diverged"; lines under 15% of their slot skip LLM expansion
  entirely (fit-skip-short) — they could only "fill" the slot with
  fabricated dialogue.
- temperature=0.2 pinned on the cinematic (_chat) and fit (llm.chat) calls;
  chat/chat_messages gained an optional temperature param that is only sent
  when set, so refinement/director/glossary callers keep provider defaults.
- Prompts hardened: ADAPT forbids introducing facts/names/dialogue not in
  the source line; EXPAND forbids inventing information and more than
  doubling the line.
- speech_rate strict-mode docstring made honest: strict changes only the
  upper tolerance bound; expansion still runs (now guard-bounded).

Fail-before/pass-after regression tests for the reported bugs (10x runaway
ADAPT on an es target, critique echo, hallucinated slot-fill expansion,
refusal replies, tiny-line expansion skip, pinned temperature) plus the
previously-untested wrong-script fallback and legit-output acceptance.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 22:28:37 +05:30
03eb8f7d04 docs(readme): community table lists the Discord server's real channels (#949)
The table described aspirational channels (#showcase/#help/#feature-requests/#dev)
that don't exist on the server; it now matches reality (#announcements,
#releases+#changelog, #issues and #ideas forums, #discuss-ideas, #general).

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 21:39:12 +05:30
beca8cfea8 fix(launcher): replace a stale old-version backend instead of attaching to it (#947)
* fix(launcher): replace a stale old-version backend instead of attaching to it

After an update, an orphaned backend from the PREVIOUS version can survive
holding the port. It still answers /system/info, so both attach paths
(lib.rs launch bootstrap + bootstrap.rs retry) treated it as "already
serving OmniVoice — attaching" and the new UI silently ran OLD backend
code: every fix in the update appeared to change nothing. Reported on
Discord as "a bound port which blocked the newer versions"; the app
already knew how to kill_orphan_on_port on both Unix (lsof) and Windows
(netstat) — it just never applied it to a healthy-but-stale backend.

The attach decision now compares versions: running_backend_version()
reads app_version from /system/info (string-sniff, no new deps), and
same_app_version() compares BASE versions (pre-release -N suffix
stripped, so a preview build 0.3.10-4 still attaches to its 0.3.10
backend). Same version → attach exactly as before. Different or missing
version → the orphan is killed and the bundled backend spawns. Foreign
processes keep the existing port_in_use take-ownership path; the
post-spawn health polls are untouched (we spawned that backend
ourselves).

Rust unit tests cover the /system/info parse shape and the
match/preview/stale/unversioned decisions; 51 pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): add the stale-backend port-reclaim fix under [Unreleased] (#947)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 20:40:42 +05:30
32103ad7b7 docs(readme): charm + organization overhaul (Opal-style) — collapsibles + OpenAI-compatible API section (#945)
* docs(readme): charm + organization overhaul (Opal-style)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(readme): restore inventory-exact feature names (docs-drift guard)

The charm pass sentence-cased five bold leads in the collapsed feature
list; scripts/check-docs-drift.py greps for the inventory's exact
title-case names. Restored: Vocal Isolation, Speaker Diarization,
Batch Queue, AI Watermark, GPU Auto-Detect.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(readme): dubbing screenshot shows a real completed dub (37 segs, EN→BN)

Replaces the empty drop-zone shot with the populated editor — video +
waveform + cast, 37 Bengali segment rows, DUB COMPLETE banner — captured
live from the v0.3.9 app; caption updated to match.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 20:36:34 +05:30
46c4445258 fix(ui): relative timestamps no longer render as "20617d ago" (seconds-vs-ms class) (#946)
Backend rows store timestamps as Unix SECONDS (time.time() REAL columns:
generation_history, dub_history, exports, longform jobs, projects), while
frontend-local records carry milliseconds (Date.now() story projects) or ISO
strings (transcriptions). Projects/OmniDrive fed the seconds straight into a
millisecond-based diff (fmtTime), so every generation-history card rendered
as ~1970 ("20617d ago") — and, because the same raw value drove the recency
sort, history items also sank to the bottom of the drive.

Fix the class, not the label: a single shared, unit-tolerant normalizer
(frontend/src/utils/relativeTime.js) now backs every relative-time call site.

- toMillis(ts): numbers < 1e12 are seconds (×1000), >= 1e12 already ms; ISO
  and numeric strings parse; Date instances pass through; null/0/undefined/
  garbage -> null. Backend storage format is untouched (backward compat).
- timeAgo(ts): "—" for missing stamps (never an epoch age), "just now" for
  future stamps within 1 min of clock skew, s/m/h/d buckets, short absolute
  date beyond 7 days.
- absoluteTime(ts): unit-safe tooltip text, '' when missing (no more
  "Jan 1 1970" titles on null rows).

Converted call sites: pages/Projects.jsx (drop local fmtTime + per-source
*1000 juggling; normalize ts once so sort and label agree), components/
Sidebar.jsx + components/WorkspaceProjects.jsx (drop duplicated local
timeAgo copies and caller-side *1000), pages/BatchQueue.jsx (drop local
formatAge; missing created_at used to render an epoch date), pages/
Transcriptions.jsx + components/TranscriptionPicker.jsx (parse via
toMillis, keep their i18n labels; unparseable stamps no longer render
"Invalid Date").

Tests (fail-before/pass-after): utils/relativeTime.test.js covers seconds/
ms/ISO/numeric-string/Date inputs, null/0 -> "—", clock-skew "just now",
and the 1970 regression (a seconds stamp from today must not render as
thousands of days ago); test/ProjectsRelativeTime.test.jsx guards the
OmniDrive wiring end-to-end (seconds created_at renders "2h ago", null
renders "—", mixed-unit sort orders by real recency). Full frontend suite:
106 files / 843 tests green; oxlint, oxfmt, typecheck:ci, node:test green.

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 20:03:41 +05:30
8bbab3fcc3 fix(translate): Dub LLM engine runs on the configured LLM provider (new dub_translation skill) (#944)
* fix(translate): the Dub LLM engine now runs on the configured LLM provider

Picking "LLM (OpenAI-compatible)" in the Dub tab read only the raw
TRANSLATE_* env vars — completely bypassing Settings → LLM Providers, so a
provider the user had configured AND tested in-app silently didn't power
the engine (empty key → raw 401 per segment). The Cinematic refiner was
already rewired through LLM Skills (#910/#912); this closes the gap for
direct LLM translation:

* new "dub_translation" LLM skill (Settings → LLM Skills) — per-skill
  provider override → global active provider, same resolution as every
  other skill; disabled == unconfigured, no new degradation modes
* the provider=openai branch resolves through resolve_skill_client();
  TRANSLATE_BASE_URL/TRANSLATE_API_KEY/TRANSLATE_MODEL stay working as
  the power-user override (env-only setups see zero behavior change,
  except the stale gpt-3.5-turbo default is now gpt-4o-mini, matching
  the cinematic path)
* per-segment calls are now bounded by the LLM timeout (45s default via
  OMNIVOICE_LLM_TIMEOUT) instead of the SDK's 600s default
* fully unconfigured → an up-front actionable 400 naming Settings → LLM
  Providers / LLM Skills instead of a per-segment 401
* provider-store keys are resolved into the error scrubber so a provider
  echoing the key can't leak it (parity with the env-key scrub)
* translation_engines registry: honest notes + a configured/configured_via
  stamp on LLM entries so the Engine dropdown can show ready-vs-needs-setup
  before the user clicks Translate

Tests: 4 new (skills-resolved client wins with its model+timeout; 400s
name the right settings page for no_provider vs disabled; env fallback
keeps working incl. TRANSLATE_MODEL); skills registry coverage updated;
existing openai-branch tests routed deterministically through the env
branch via the shared fake helper.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): add the dub-translation provider wiring under [Unreleased] (#944)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 19:22:12 +05:30
9ff5f38cb3 docs(readme): cross-promote the maker's other local-first projects (Opal, memxt) (#943)
Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 19:09:46 +05:30
359c70bc21 docs(readme): use the Launchpad screenshot as the hero image (#942)
Swap the static social-preview banner for the live v0.3.9 Launchpad shot
and drop the now-duplicate Launchpad row from the gallery (shown once).

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 18:56:21 +05:30
0579ec91ad docs(readme): Opal-style restyle + fresh v0.3.9 screenshots (#937)
- Emoji section headers with explicit <a id> anchors. Emoji breaks
  GitHub's auto-generated heading slugs, so every in-page nav target keeps
  a stable explicit anchor (verified all href="#..." resolve).
- Refresh the screenshot gallery. The prior set was from April, predating
  the launchpad / settings / dictation UI overhaul, so it misrepresented
  the app. Captured fresh at retina from the live v0.3.9 UI and led the
  gallery with the new Launchpad home: launchpad, studio, voice design,
  voice gallery, dubbing, engine-compatibility matrix, model store,
  embedded API reference (Scalar), and the in-app changelog reader.
- Fix the stale engine count (11 -> 14 TTS engines) in the comparison
  table, FAQ, and roadmap to match the engine table + backend registry.
- Use <kbd> keycaps for the dictation shortcut (Opal detail).

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 18:38:47 +05:30
b6ec4e23f3 fix(engines): snapshot lazy registry keys so /engines can't 500 under concurrency (#940)
* fix(engines): snapshot lazy registry keys so /engines can't 500 under concurrency

`list_backends()` runs in a FastAPI threadpool and iterates the lazy TTS/ASR
registries via `items()` → `__iter__`, which held a *live* `dict.__iter__(self)`
open across each engine's slow `is_available()` probe. Meanwhile the lazy
`__getitem__` resolves a deferred entry by mutating the dict (`self[key] = cls`).
A second concurrent `/engines` request (or any ASR op) materializing the lazy
`faster-whisper-isolated` entry therefore changed the dict size mid-iteration:

    RuntimeError: dictionary changed size during iteration
      asr_backend.py:1729 list_backends → _REGISTRY.items()
      asr_backend.py:1665 __iter__ → for k in dict.__iter__(self)

Both `_LazyRegistry` (TTS) and `_LazyASRRegistry` (ASR) now snapshot their live
keys up front with `list(dict.__iter__(self))` — consumed atomically under the
GIL — so a concurrent lazy insert can no longer trip the iteration. The slow
per-engine probes then run over the snapshot, not the live iterator.

Deterministic fail-before/pass-after regression for both registries:
tests/backend/services/test_lazy_registry_concurrency.py.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(changelog): add the /engines concurrency fix under [Unreleased] (#940)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: mergetest <test@local>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 16:53:57 +05:30
60 changed files with 2414 additions and 402 deletions
+15
View File
@@ -6,6 +6,21 @@ The format is loosely based on [Keep a Changelog](https://keepachangelog.com/).
Versions track the desktop app (`tauri.conf.json` + `frontend/src-tauri/Cargo.toml`).
The bundled TTS model package (`pyproject.toml`) is versioned independently.
## [0.3.10] — 2026-07-05
The listening release — nine fixes in twenty-four hours, almost all driven by your v0.3.9 field reports (several with same-day turnaround). The dubbing pipeline stops lying: **Cinematic and Autofit can no longer invent dialogue**, the **speaker count you set is honored on every path** (and auto-cloning stops fabricating voices from guessed labels), and the timeline stops flashing invisible on Windows. Audiobook chapters with pauses render again. And one fix everyone should want: **updating can no longer leave you secretly running the old version** — a leftover backend from a previous install holding the port is now detected and replaced at launch. Plus: the Dub tab's LLM engine finally runs on the provider you configured in Settings, history timestamps stop reading "20617d ago", and the Engines page can't crash under concurrent load.
### Fixed
- **Audiobook/Stories chapters with a `[pause]` no longer fail to render.** Pause spans were built as 1-D silence while every TTS engine returns 2-D audio, so the chapter concatenation crashed with `Tensors must have same number of dimensions` — any chapter containing a pause failed on every attempt (reported with a precise trace in #897). Silence now matches the rendered audio's shape at the source, and the chunk concatenator defensively normalizes mixed ranks (including honest mono→stereo broadcast) so no engine can re-trigger the class. (#953)
- **Cinematic and Autofit dubbing can no longer invent dialogue.** The refine and slot-fit passes accepted any non-empty LLM reply for Latin-script languages — hallucinated lines, refusals, or the critique itself could ship as the dub. Every reply is now checked against the original line (length window, target script, critique echo — tunable via `OMNIVOICE_REFINE_RATIO_MIN/MAX`), rejected output falls back to the literal translation with an `adapt-diverged`/`fit-diverged` marker, lines too short to honestly fill their slot skip LLM expansion entirely, and both passes pin `temperature=0.2` like the Fast path. (#950)
- **Dub timeline boxes can no longer flash invisible during playback.** On some Windows GPU/WebView2 driver combos the segment boxes under the video vanished and reappeared while playing (first reported in #373; the earlier fix was incomplete) — the timeline lane still animated a CSS transform every playback tick, keeping the translucent boxes on a composited layer that the driver mis-painted. Boxes are now positioned in pure layout with fully opaque theme-aware fills (pixel-identical colors), removing the glitch class on every platform. (#951)
- **The dub "Speakers" count now actually does something — on every path.** The hint only reached pyannote; the common fallbacks silently ignored it (the no-diarization heuristic was hardcoded to alternate two speakers, and the FunASR shortcut never consulted it). The heuristic now cycles the requested count, an explicit count routes through pyannote when available, every path that can only approximate (or must ignore) the setting says so in a visible warning, and the legacy endpoint + a new CLI `--speakers` flag accept it too. Auto voice-cloning also stops fabricating voices from guessed labels: reference slices under 1.5s are rejected, slices bordering another speaker's turn are avoided, and cloning is skipped with an honest warning when speaker labels came from the gap heuristic instead of real diarization. (#952)
- **Settings → Engines can no longer 500 under concurrent loads.** The lazy TTS/ASR engine registries held a *live* dictionary iterator open across each engine's `is_available()` probe while `list_backends()` ran in a FastAPI threadpool — so a second concurrent `/engines` request materializing a lazy engine entry (`self[key] = cls`) mutated the dict mid-iteration and crashed the request with `RuntimeError: dictionary changed size during iteration`. Both registries now snapshot their keys before iterating (atomic under the GIL), immune to a concurrent insert; regression-tested for TTS and ASR. (#940)
- **The Dub tab's LLM translation engine now runs on your configured LLM provider.** Picking "LLM (OpenAI-compatible)" silently required three hand-set environment variables even when a provider was already configured and tested in Settings → LLM Providers; it now resolves through a new "Dub translation" LLM skill (route it to any provider — remote or local — in Settings → LLM Skills, independently of Cinematic refinement), keeps the `TRANSLATE_*` env vars as a power-user override, bounds every call with the LLM timeout instead of the SDK's 600-second default, tells the Engine dropdown whether the engine is actually ready (and via which provider), and — when nothing is configured — returns a clear pointer to Settings → LLM Providers instead of a raw 401 per segment. (#944)
- **Timestamps no longer show "20617d ago" in OmniDrive/Projects.** The backend stores record times in Unix seconds while some views assumed milliseconds, so generation-history cards rendered as ~1970 ("20617d ago") and sorted last; every relative-time label (OmniDrive, sidebar history, dub projects, batch queue, transcriptions) now goes through one unit-tolerant formatter, and records missing a timestamp show "—" instead of an epoch age.
- **Updating can no longer leave you secretly running the old version.** If a backend from a previous version was still holding the port (an orphan that survived an update), the new app "attached" to it because it answered health checks — so every fix in the update appeared to change nothing (the reported "bound port blocked the newer version"). The launcher now compares the running backend's version against the app before attaching: same version attaches as before, a stale one is killed and the bundled backend is started in its place — on macOS, Windows, and Linux. (#947)
## [0.3.9] — 2026-07-04
The dictation release — and a deep reliability pass driven by live-testing the entire app. **Dictation is rebuilt end-to-end**: instant feedback with a live waveform, words that commit about half a second after you stop speaking, clean punctuation, and text insertion that never lies about success. **LLM providers get one-click connection testing** with real diagnostics and model discovery, in all 21 languages. The app now **always opens maximized**, bottom buttons **can't hide under the footer** at small window sizes, and a wave of "out of memory / can't reach the backend / stuck at preparing" reports were traced to their real causes and fixed — including the silent VRAM crash on 8 GB cards, dead-IPC startup hangs after a Windows BSOD, and misleading error labels. Intel-Mac support status is now stated honestly, Confucius4-TTS is validated end-to-end, and Parakeet — roughly 20× faster than the default transcriber on CPU — is unlocked for every machine.
+225 -147
View File
@@ -10,6 +10,7 @@
<a href="#why-ovs">Why OVS</a> ·
<a href="#tts-engines">TTS Engines</a> ·
<a href="#asr-engines">ASR Engines</a> ·
<a href="#openai-api">API</a> ·
<a href="#sponsors">Sponsors</a> ·
<a href="#sponsor--donate">Donate</a> ·
<a href="#contributing">Contributing</a> ·
@@ -31,11 +32,21 @@
<br/>
<div align="center">
<img src=".github/assets/social-preview.png" alt="OmniVoice Studio — The open-source ElevenLabs alternative" width="100%"/>
<img src="docs/screenshot-launchpad.png" alt="OmniVoice Studio — Launchpad" width="100%"/>
</div>
> **Your voice is the most personal data you have. So why rent it back from a cloud?** Every mainstream voice tool ships your audio to someone else's server and bills you monthly for the privilege. OmniVoice Studio flips that: clone, design, dub, and dictate on your own hardware — 646 languages, no meter running, nothing leaving your machine.
<div align="center">
| 🔑 No API keys | 🙅 No accounts | ☁️ No cloud | 💳 No subscription |
|:---:|:---:|:---:|:---:|
| nothing to paste in | nothing to sign up for | your audio stays home | it's your computer |
</div>
> [!WARNING]
> **OmniVoice Studio is in active beta.** Things may break between releases. For the latest features and fixes, clone the repo and run from source rather than using pre-built installers. Bug reports and PRs are very welcome [open an issue](https://github.com/debpalash/OmniVoice-Studio/issues) or [join Discord](https://discord.gg/bzQavDfVV9).
> **OmniVoice Studio is in active beta.** Things may break between releases — for the latest features and fixes, clone the repo and run from source rather than the pre-built installers. Bug reports and PRs are very welcome: [open an issue](https://github.com/debpalash/OmniVoice-Studio/issues) or [join Discord](https://discord.gg/bzQavDfVV9).
<div align="center">
<br/>
@@ -47,7 +58,68 @@
<br/>
## Features
<a id="screenshots"></a>
## 📸 See it in action
<table>
<tr>
<td align="center" width="50%">
<img src="docs/screenshot-studio.png" alt="Studio" width="100%"/>
<br/><b>Studio</b><br/>
<sub>Generate &amp; clone in one workspace — a 3-second clip mirrors any voice, 646 languages, zero-shot.</sub>
</td>
<td align="center" width="50%">
<img src="docs/screenshot-design.png" alt="Voice Design" width="100%"/>
<br/><b>Voice Design</b><br/>
<sub>Build new voices from scratch — gender, age, accent, pitch, emotion, dialect.</sub>
</td>
</tr>
<tr>
<td align="center">
<img src="docs/screenshot-gallery.png" alt="Voice Gallery" width="100%"/>
<br/><b>Voice Gallery</b><br/>
<sub>Browse ready-made archetype voices with language filters — or build your own library.</sub>
</td>
<td align="center">
<img src="docs/screenshot-dub.png" alt="Video Dubbing" width="100%"/>
<br/><b>Video Dubbing</b><br/>
<sub>A real dub, end to end: 37 segments transcribed, translated to Bengali, re-voiced, and timed — ready to export as MP4.</sub>
</td>
</tr>
<tr>
<td align="center">
<img src="docs/screenshot-engines.png" alt="Settings — Engines" width="100%"/>
<br/><b>Settings → Engines</b><br/>
<sub>The engine compatibility matrix — 14 TTS engines with per-engine GPU preflight, no silent CPU fallback.</sub>
</td>
<td align="center">
<img src="docs/screenshot-settings.png" alt="Settings — Models" width="100%"/>
<br/><b>Settings → Models</b><br/>
<sub>One-click model store — auto-detects your platform (CUDA / MPS / CPU) and recommends the right models.</sub>
</td>
</tr>
<tr>
<td align="center">
<img src="docs/screenshot-openapi.png" alt="Settings — API Reference" width="100%"/>
<br/><b>API Reference</b><br/>
<sub>The full local REST API, embedded — every endpoint documented with copy-paste client snippets.</sub>
</td>
<td align="center">
<img src="docs/screenshot-updates.png" alt="Settings — What's New" width="100%"/>
<br/><b>What's New</b><br/>
<sub>In-app changelog reader — see exactly what shipped in each release without leaving the app.</sub>
</td>
</tr>
</table>
---
<a id="features"></a>
## ✨ Features
The eight headliners — and twelve more waiting under the fold.
<table>
<tr>
@@ -75,76 +147,44 @@
</td>
<td align="center" valign="top">
<h3>⌨️ Dictation Widget</h3>
<p><code>⌘+⇧+Space</code> from <b>any app</b>.<br/>Transcribes, auto-pastes, disappears.</p>
<p><kbd>⌘</kbd>+<kbd>⇧</kbd>+<kbd>Space</kbd> from <b>any app</b>.<br/>Transcribes, auto-pastes, disappears.</p>
</td>
<td align="center" valign="top">
<h3>🔊 Vocal Isolation</h3>
<p>Demucs-powered. Splits speech<br/>from music, <b>keeps the background</b>.</p>
</td>
<td align="center" valign="top">
<h3>👥 Speaker Diarization</h3>
<p>Pyannote + WhisperX.<br/><b>Auto-identifies</b> who said what.</p>
</td>
</tr>
<tr>
<td align="center" valign="top">
<h3>📦 Batch Queue</h3>
<p>Drop <b>50 videos</b>, walk away.<br/>Progress bars per job.</p>
</td>
<td align="center" valign="top">
<h3>🤖 MCP Server</h3>
<p>Use OmniVoice from <b>Claude</b>,<br/>Cursor, or any MCP client.</p>
</td>
<td align="center" valign="top">
<h3>🛡️ AI Watermark</h3>
<p>AudioSeal (Meta). <b>Invisible</b>,<br/>survives compression.</p>
</td>
<td align="center" valign="top">
<h3>🔬 Diagnostics</h3>
<p>Self-check, error journal,<br/>scrubbed <b>diagnostic bundle</b>.</p>
</td>
</tr>
<tr>
<td align="center" valign="top">
<h3>🔐 100% Local</h3>
<p>No keys, no cloud, no accounts.<br/><b>Your machine only</b>.</p>
</td>
<td align="center" valign="top">
<h3>⚡ GPU Auto-Detect</h3>
<p>CUDA · MPS · ROCm · CPU.<br/>≤8 GB? <b>Auto-offloads</b>.</p>
</td>
<td align="center" valign="top">
<h3>🧩 Extensible</h3>
<p>Subclass <code>TTSbackend</code>,<br/>add any engine in <b>~50 lines</b>.</p>
</td>
<td align="center" valign="top">
<h3>🧭 Engine Routing</h3>
<p>Preflight GPU check per engine.<br/><b>No silent CPU fallback</b>.</p>
</td>
</tr>
<tr>
<td align="center" valign="top">
<h3>🎒 Portable Personas</h3>
<p>Export voices as <code>.ovsvoice</code><br/>bundles — identity + <b>watermark</b>.</p>
</td>
<td align="center" valign="top">
<h3>♾️ Unlimited TTS</h3>
<p>Sentence-chunked generation.<br/><b>No length cap</b>. Streaming via WS.</p>
</td>
<td align="center" valign="top">
<h3>🌐 Remote Backend</h3>
<p>Point UI at a remote server.<br/>Tailscale-friendly. <b>Bearer auth</b>.</p>
</td>
<td align="center" valign="top">
<h3>🧠 Dictation + LLM</h3>
<p>Local LLM cleanup of transcripts.<br/>Optional echo <b>cancellation</b>.</p>
<h3>🤖 MCP Server</h3>
<p>Use OmniVoice from <b>Claude</b>,<br/>Cursor, or any MCP client.</p>
</td>
</tr>
</table>
<details>
<summary><b>…and 12 more</b> — isolation, diarization, batch, watermarking, diagnostics, and friends</summary>
<br/>
- 🔊 **Vocal Isolation** — Demucs-powered: splits speech from music and keeps the background bed.
- 👥 **Speaker Diarization** — Pyannote + WhisperX auto-identify who said what.
- 📦 **Batch Queue** — drop 50 videos, walk away; per-job progress bars.
- 🛡️ **AI Watermark** — AudioSeal (Meta): invisible, survives compression.
- 🔬 **Diagnostics** — self-check suite, error journal, scrubbed diagnostic bundles.
-**GPU Auto-Detect** — CUDA · MPS · ROCm · CPU; ≤8 GB VRAM auto-offloads.
- 🧭 **Engine routing** — preflight GPU check per engine; no silent CPU fallback.
- 🧩 **Extensible** — subclass `TTSBackend`, add any engine in ~50 lines.
- 🎒 **Portable personas** — export voices as `.ovsvoice` bundles: identity + watermark.
- ♾️ **Unlimited TTS** — sentence-chunked generation, no length cap, streaming via WebSocket.
- 🌐 **Remote backend** — point the UI at a remote server; Tailscale-friendly, bearer auth.
- 🧠 **Dictation + LLM** — local-LLM cleanup of transcripts, optional echo cancellation.
</details>
---
## Quickstart
<a id="quickstart"></a>
## ⚡ Quickstart
<div align="center">
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
@@ -157,14 +197,19 @@
<sub><b>Intel Macs are not supported for the local backend:</b> the app UI installs, but the Python backend cannot run because PyTorch no longer ships Intel-Mac (x86_64) wheels (<a href="https://github.com/debpalash/OmniVoice-Studio/issues/889">#889</a>) — see <a href="docs/install/macos.md">docs/install/macos.md</a>.</sub>
</div>
Per-OS install guides — pick yours and follow it end-to-end:
Pick your OS and follow the guide end-to-end:
- **macOS** — [docs/install/macos.md](docs/install/macos.md)
- **Windows** — [docs/install/windows.md](docs/install/windows.md)
- **Linux** — [docs/install/linux.md](docs/install/linux.md)
- **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
- 🍎 **macOS** — [docs/install/macos.md](docs/install/macos.md)
- 🪟 **Windows** — [docs/install/windows.md](docs/install/windows.md)
- 🐧 **Linux** — [docs/install/linux.md](docs/install/linux.md)
- 🐳 **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
Stuck? Run the built-in self-check first — **Settings → About → "Run
<details>
<summary><b>🧰 Stuck? Self-checks, tokens &amp; restricted networks</b></summary>
<br/>
Run the built-in self-check first — **Settings → About → "Run
self-check"** in the app, or `uv run python backend/main.py --diagnose` from
a checkout (`--deep` also test-loads the active engine). Then see
[docs/install/troubleshooting.md](docs/install/troubleshooting.md) for the
@@ -179,57 +224,13 @@ diarization-specific gating, see
speed, the ⚡ fast-download (Xet) status, and restricted-network / mirror
options, see [docs/downloading-models.md](docs/downloading-models.md).
## Screenshots
<table>
<tr>
<td align="center" width="50%">
<img src="docs/screenshot-clone.png" alt="Voice Clone" width="100%"/>
<br/><b>Voice Clone</b><br/>
<sub>Drop a 3-second clip → mirror any voice. 646 languages, zero-shot.</sub>
</td>
<td align="center" width="50%">
<img src="docs/screenshot-design.png" alt="Voice Design" width="100%"/>
<br/><b>Voice Design</b><br/>
<sub>Build new voices from scratch — gender, age, accent, pitch, style.</sub>
</td>
</tr>
<tr>
<td align="center">
<img src="docs/screenshot-dub.png" alt="Video Dubbing" width="100%"/>
<br/><b>Video Dubbing</b><br/>
<sub>Upload or paste a YouTube URL. Transcribe, translate, re-voice, export.</sub>
</td>
<td align="center">
<img src="docs/screenshot-gallery.png" alt="Voice Gallery" width="100%"/>
<br/><b>Voice Gallery</b><br/>
<sub>Search YouTube, browse categories, download clips, build your library.</sub>
</td>
</tr>
<tr>
<td align="center">
<img src="docs/screenshot-settings.png" alt="Settings — Models" width="100%"/>
<br/><b>Settings → Models</b><br/>
<sub>15 models. One-click install. Auto-detects your platform (CUDA / MPS / CPU).</sub>
</td>
<td align="center">
<img src="docs/screenshot-libraryprojects.png" alt="Projects" width="100%"/>
<br/><b>Projects</b><br/>
<sub>Dub projects, voice profiles, generation history, exports — all searchable.</sub>
</td>
</tr>
<tr>
<td align="center" colspan="2">
<img src="docs/screenshot-logs.png" alt="Settings — Logs" width="100%"/>
<br/><b>Settings → Logs</b><br/>
<sub>Live backend, frontend, and Tauri runtime logs. Filter, refresh, clear.</sub>
</td>
</tr>
</table>
</details>
---
## Why OVS?
<a id="why-ovs"></a>
## 💡 Why OmniVoice?
ElevenLabs charges **$5$330/mo** and processes your audio on their servers. OmniVoice Studio runs **on your hardware, with no usage limits.**
@@ -245,13 +246,13 @@ ElevenLabs charges **$5$330/mo** and processes your audio on their servers. O
| **API Keys** | Required | Not needed |
| **GPU Support** | N/A (cloud) | CUDA · Apple Silicon · ROCm · CPU |
| **Desktop App** | ❌ | ✅ macOS · Windows · Linux |
| **TTS Engines** | 1 | **11** (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, IndexTTS 2, OmniVoice GGUF, Supertonic 3) |
| **TTS Engines** | 1 | **14** (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS) |
| **ASR Engines** | 1 | **9** (WhisperX, Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet, Moonshine, FunASR, isolated Faster-Whisper, sherpa-onnx live dictation) |
| **MCP Server** | ❌ | ✅ Use from Claude, Cursor, any MCP client |
| **Self-check** | ❌ | ✅ Diagnostics suite, error journal, scrubbed debug bundles |
| **Customizable** | ❌ Closed | ✅ Fork it, extend it, ship it |
OmniVoice Studio gives you professional-grade AI tools without the subscription or the cloud.
Professional-grade voice AI, minus the subscription and the cloud.
<div align="center">
<br/>
@@ -262,7 +263,7 @@ OmniVoice Studio gives you professional-grade AI tools without the subscription
---
## System Requirements
## 🖥️ System Requirements
| | **Minimum** | **Recommended** |
|---|---|---|
@@ -279,9 +280,16 @@ OmniVoice Studio gives you professional-grade AI tools without the subscription
> [!IMPORTANT]
> **macOS Intel (x86_64) is unsupported for the local backend:** the app UI installs, but the Python backend cannot run because PyTorch no longer ships Intel-Mac wheels ([#889](https://github.com/debpalash/OmniVoice-Studio/issues/889)). Intel-Mac users can still point the UI at a remote backend on another machine — see [docs/install/macos.md](docs/install/macos.md).
### TTS Engines
<a id="tts-engines"></a>
OmniVoice ships a multi-engine TTS backend. The default engine (OmniVoice) is always available; additional engines are opt-in and auto-detected. Switch engines in **Settings → TTS Engine** or via the `OMNIVOICE_TTS_BACKEND` env var.
### 🗣️ TTS Engines
**14 engines, one picker.** OmniVoice (default, 600+ languages) is always available; CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, and Sherpa-ONNX are opt-in and auto-detected — plus six lazy-installed heavyweights (IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS). Switch in **Settings → TTS Engine** or via the `OMNIVOICE_TTS_BACKEND` env var.
<details>
<summary><b>📊 The full matrix</b> — 14 engines × platform × clone/instruct × license</summary>
<br/>
| Engine | Languages | Clone | Instruct | Linux | macOS ARM | Windows | License |
|--------|:---------:|:-----:|:--------:|:-----:|:---------:|:-------:|:-------:|
@@ -304,9 +312,18 @@ OmniVoice ships a multi-engine TTS backend. The default engine (OmniVoice) is al
>
> **MOSS-TTS-v1.5** (8B, ~16 GB weights) and **dots.tts** (2B, ~9 GB weights) are heavyweight opt-in engines that run in their own isolated venv from a local clone — see [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) and [dots.tts](docs/engines/dots-tts.md). Neither claims Apple-Silicon **MPS** (upstream is CUDA/CPU only; on a Mac they run on CPU). dots.tts upstream is Linux/macOS only — no Windows path. **Confucius4-TTS** (14-language cross-lingual zero-shot cloning) is similar — its own Python 3.10 venv from a clone; CUDA recommended, CPU validated end-to-end (slow, ~17× realtime; no MPS — tested slower than CPU); see [Confucius4-TTS](docs/engines/confucius4-tts.md).
### ASR Engines
</details>
OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictation, video dubbing, and subtitle generation — all fully local. **WhisperX** is the cross-platform default; the rest are opt-in and auto-detected. Switch in **Settings → ASR Engine** or via the `OMNIVOICE_ASR_BACKEND` env var.
<a id="asr-engines"></a>
### 🎧 ASR Engines
**9 engines, all fully local** — they power dictation, video dubbing, and subtitles. **WhisperX** is the cross-platform default (~100 languages, word-level timing); the rest are opt-in and auto-detected. Switch in **Settings → ASR Engine** or via the `OMNIVOICE_ASR_BACKEND` env var.
<details>
<summary><b>📊 The full lineup</b> — 9 engines, what each is best at, and compute-type notes</summary>
<br/>
| Engine | `OMNIVOICE_ASR_BACKEND` | Languages | Best for |
|--------|-------------------------|:---------:|----------|
@@ -324,9 +341,11 @@ OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictatio
> **GPU without efficient float16?** On older NVIDIA GPUs (Maxwell/Pascal, GTX 16xx) or after a CTranslate2/cuDNN mismatch, the CTranslate2 ASR engines (WhisperX, Faster-Whisper) can't run `float16` and OmniVoice automatically retries on `int8` — no config needed. If transcription still fails, pin the compute type with the `ASR_COMPUTE_TYPE` env var (escape hatch): `ASR_COMPUTE_TYPE=int8` (or `float32` for CPU). Set it to `int8` and restart the backend.
</details>
---
## Architecture
## 🏗️ Architecture
```
┌─────────────────────────────────────────────────────────────┐
@@ -344,11 +363,50 @@ OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictatio
CUDA / MPS / ROCm / CPU (auto-detected + routed)
```
<a id="openai-api"></a>
## 🔌 OpenAI-compatible API
Already have a script, agent, or tool that speaks OpenAI's audio API? Point it at `http://localhost:3900/v1` — no key needed, no code changes. The backend ships a drop-in surface for the audio endpoints, wired to whichever TTS/ASR engine you have active (and yes, `voice` accepts your cloned voice-profile IDs).
| Endpoint | What it does |
|---|---|
| `POST /v1/audio/speech` | TTS — text in; `mp3` / `wav` / `flac` / `opus` / `pcm` out. `tts-1` / `tts-1-hd` map to your active engine; OpenAI voice names (`alloy`, …) are accepted. |
| `POST /v1/audio/transcriptions` | STT — audio file in; `json`, `text`, `verbose_json`, `srt`, or `vtt` out. `whisper-1` maps to your active ASR engine. |
| `GET /v1/audio/voices` | OmniVoice extension — lists every voice profile and engine, so clients can discover your clones. |
```sh
curl http://localhost:3900/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"model": "tts-1", "voice": "alloy", "input": "Generated on my own hardware.", "response_format": "wav"}' \
--output speech.wav
```
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:3900/v1", api_key="none") # any string works — nothing checks it
result = client.audio.transcriptions.create(model="whisper-1", file=open("clip.wav", "rb"))
print(result.text)
```
Want the whole surface (100+ endpoints)? The full REST API reference is embedded in the app — **Settings → OpenAPI Reference** (Scalar-powered), or the `{}` button in the footer.
---
## Roadmap
## 🗺️ Roadmap
### ✅ Shipped
### 🔜 Up Next
- 🎬 **Lip-sync v2** — visual speech timing with wav2lip
- 🌐 **Hosted Demo** — try OmniVoice without installing anything
- 🔌 **Plugin Marketplace** — community-contributed TTS engines and effects
- 🎵 **Real-time Voice Changer** — live microphone transformation during calls
<details>
<summary><b>✅ Everything shipped so far</b> — the receipts, by category</summary>
<br/>
| Category | Features |
|----------|----------|
@@ -359,7 +417,7 @@ OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictatio
| **Multi-Lang** | Multi-language batch picker, batch dubbing queue with sequential GPU execution |
| **Diarization** | Pyannote ML diarization, auto speaker clone extraction, per-speaker voice assignment |
| **ASR** | 9 engines (WhisperX, Faster-Whisper, isolated Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet TDT, Moonshine, FunASR/SenseVoice, sherpa-onnx live dictation), crash-isolated subprocess backend |
| **TTS** | 11 engines (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, + lazy: IndexTTS 2, OmniVoice GGUF, Supertonic 3), engine routing with GPU preflight |
| **TTS** | 14 engines (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, + lazy: IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS), engine routing with GPU preflight |
| **Infra** | Docker deployment, CUDA/MPS/ROCm auto-detect, cuDNN 8 compat, VRAM-aware model offloading, engine routing (no silent CPU fallback), diagnostics suite & error journal, restricted-network mirror support |
| **AI Provenance** | AudioSeal invisible watermarking (SynthID-like), video logo overlay, watermark detection API |
| **UX** | Undo/redo, keyboard shortcuts, drag-and-drop, session persistence, glassmorphism design system, UI scale fix for Linux/WebKitGTK |
@@ -372,16 +430,13 @@ OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictatio
| **Remote Backend** | Point the desktop UI at a remote backend URL with bearer auth (Tailscale-documented) |
| **Reliability** | Stall watchdog on bootstrap splash, per-engine GPU compatibility matrix, actionable errors for non-executable engine binaries, setuptools auto-repair |
### 🔜 Up Next
- 🎬 **Lip-sync v2** — visual speech timing with wav2lip
- 🌐 **Hosted Demo** — try OmniVoice without installing anything
- 🔌 **Plugin Marketplace** — community-contributed TTS engines and effects
- 🎵 **Real-time Voice Changer** — live microphone transformation during calls
</details>
---
## Sponsor / Donate
<a id="sponsor--donate"></a>
## 💜 Sponsor / Donate
OmniVoice Studio is built by one developer using Claude Code and AI agents — and the agent bills are real. Over the last three months I've spent thousands of dollars on Claude subscriptions to keep the features shipping, the bugs fixed, and your issues answered. If OmniVoice has created value for you, helping cover those bills means I can keep developing full-time.
@@ -402,7 +457,9 @@ OmniVoice Studio is built by one developer using Claude Code and AI agents — a
</div>
### Sponsors
<a id="sponsors"></a>
### 🌟 Sponsors
OmniVoice is **free** and **AGPL-3.0** — no paid tier, no SaaS revenue. Sponsors keep development going, and in return get a logo slot here, in the app, and (for top tiers) on the project website. It's a thank-you, never a paywall. **[See tiers & become a sponsor →](SPONSORS.md)**
@@ -420,29 +477,37 @@ OmniVoice is **free** and **AGPL-3.0** — no paid tier, no SaaS revenue. Sponso
---
## Community
## 💬 Community
<div align="center">
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/💬_Discord-Join_Community-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="Join Discord" /></a>
<br/>
<sub>We respond to setup questions within hours, not days.</sub>
</div>
<details>
<summary><b>What happens in there</b></summary>
<br/>
| Channel | What happens there |
|---------|--------------------|
| `#showcase` | Members share their dubs, clones, and voice designs |
| `#help` | Setup issues, GPU troubleshooting, model questions |
| `#feature-requests` | Vote on what gets built next |
| `#dev` | Architecture discussions, PR reviews, engine integrations |
| `#announcements` | Release notes, breaking changes, early access |
| `#announcements` | Release news and the big moments — new versions land here first |
| `#releases` + `#changelog` | Every build and exactly what's inside it |
| `#issues` | Bug reports as forum posts — triaged straight into GitHub issues |
| `#ideas` | Feature requests, discussed and voted on |
| `#discuss-ideas` | Design talk before things get built |
| `#general` | Setup help, GPU troubleshooting, and showing off your dubs |
**[→ Join the Discord](https://discord.gg/bzQavDfVV9)** — we respond to setup questions within hours, not days.
</details>
---
## Contributing
<a id="contributing"></a>
We welcome contributions of all kinds — bug fixes, new TTS engine adapters, UI improvements, docs, and translations.
## 🤝 Contributing
Yes please — bug fixes, new TTS engine adapters, UI improvements, docs, translations. All of it.
- 📖 Read the **[Contributing Guide](CONTRIBUTING.md)** for setup, code style, and PR workflow
- 🐛 Browse [good first issues](https://github.com/debpalash/OmniVoice-Studio/labels/good%20first%20issue)
@@ -450,7 +515,7 @@ We welcome contributions of all kinds — bug fixes, new TTS engine adapters, UI
---
## FAQ
## FAQ
<details>
<summary><b>Is this really as good as ElevenLabs?</b></summary>
@@ -485,12 +550,14 @@ Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are availab
<details>
<summary><b>Can I add my own TTS engine?</b></summary>
<br/>
Yes. OmniVoice uses a <b>built-in backend registry</b>. To add an engine in ~50 lines, subclass <code>TTSBackend</code> in <code>backend/services/tts_backend.py</code> and add it to the <code>_REGISTRY</code> dictionary. Eleven engines are built in: OmniVoice, CosyVoice 3, GPT-SoVITS, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, KittenTTS, Sherpa-ONNX, plus lazy-registered IndexTTS 2, OmniVoice GGUF, and Supertonic 3. See the <a href="#tts-engines">TTS Engines</a> section for details.
Yes. OmniVoice uses a <b>built-in backend registry</b>. To add an engine in ~50 lines, subclass <code>TTSBackend</code> in <code>backend/services/tts_backend.py</code> and add it to the <code>_REGISTRY</code> dictionary. Fourteen engines are built in: OmniVoice, CosyVoice 3, GPT-SoVITS, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, KittenTTS, Sherpa-ONNX, plus lazy-registered IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, and Confucius4-TTS. See the <a href="#tts-engines">TTS Engines</a> section for details.
</details>
---
## License
<a id="license"></a>
## 📜 License
OmniVoice Studio is free and open-source software under the [**GNU Affero General Public License v3.0 (AGPL-3.0)**](https://www.gnu.org/licenses/agpl-3.0.html).
@@ -502,7 +569,7 @@ The bundled `omnivoice/` TTS model by Han Zhu remains Apache-2.0 upstream. See [
---
## Acknowledgments
## 🙏 Acknowledgments
OmniVoice Studio is built on the shoulders of exceptional open-source work:
@@ -521,6 +588,17 @@ OmniVoice Studio is built on the shoulders of exceptional open-source work:
---
## 🧰 More local open-source from the maker
Like the local-first philosophy? It runs in the family:
| Project | What it is |
|---------|------------|
| [**Opal**](https://github.com/debpalash/Opal) 💠 | **Play everything.** The evolved media player for the next decades of entertainment — video, anime, comics, torrents, Jellyfin/Plex, with local AI built in. |
| [**memxt**](https://github.com/debpalash/memxt) 🧠 | **The fastest benchmarked open-source AI memory system.** 100% local memory for AI agents, with MCP support. |
---
<div align="center">
<br/>
+223 -58
View File
@@ -375,6 +375,31 @@ _CHUNK_TRANSCRIBE_ATTEMPTS = max(1, int(os.environ.get("OMNIVOICE_TRANSCRIBE_CHU
_sse_event = dub_pipeline.sse_event
_prep_event_helper = dub_pipeline.prep_event # alias; we keep the module-local _prep_event below for the inline one-liner shape
#: User-facing warning emitted when auto voice cloning is skipped because the
#: speaker labels came from the silence-gap heuristic (see _diarize /
#: extract_speaker_clones — gap-based labels routinely mix two people's audio
#: into one reference, which is how "made up" clone voices happen).
CLONE_SKIP_HEURISTIC_MSG = (
"auto voice cloning skipped: speaker labels are gap-based estimates — "
"set up diarization (Settings → Models → pyannote) for per-speaker clones"
)
def _clamp_num_speakers(value) -> Optional[int]:
"""Clamp the user's speaker-count hint to a sane 120 range.
Shared by the SSE and legacy transcribe endpoints so the two can't drift.
None / non-int / out-of-range → None (auto-detect), so a bad query string
can never break a diarization call.
"""
if value is None:
return None
try:
value = int(value)
except (TypeError, ValueError):
return None
return value if 1 <= value <= 20 else None
@router.get("/dub/transcribe-stream/{job_id}")
async def dub_transcribe_stream(
@@ -393,15 +418,14 @@ async def dub_transcribe_stream(
pyannote auto-detects the count — but its auto-detect can collapse a
multi-speaker clip to a single speaker (issue #274). When the user knows
the exact count, supplying it forces pyannote to return that many speakers.
On paths that can't honor the hint exactly (inline ASR turns, the
silence-gap heuristic) it is never silently dropped: the heuristic cycles
the requested count and a `warning` SSE event tells the user how far the
labels can be trusted.
"""
# Clamp to a sane range; ignore anything non-positive / absurd so a bad
# query string can never break the diarization call. None → auto-detect.
if num_speakers is not None:
try:
num_speakers = int(num_speakers)
num_speakers = num_speakers if 1 <= num_speakers <= 20 else None
except (TypeError, ValueError):
num_speakers = None
num_speakers = _clamp_num_speakers(num_speakers)
job = _get_job(job_id)
@@ -630,7 +654,11 @@ async def dub_transcribe_stream(
snap_segment_starts(chunk_segs, audio_np, sr)
except Exception as e:
logger.warning("onset alignment skipped for chunk %d: %s", i, e)
chunk_segs = assign_speakers_heuristic(chunk_segs)
# Provisional per-chunk labels for the streaming UI only — the
# final diarization pass below overwrites them. Honor the user's
# speaker-count hint here too so the interim view doesn't flip
# between 2 and N speakers.
chunk_segs = assign_speakers_heuristic(chunk_segs, num_speakers)
for s in chunk_segs:
s["id"] = f"s{next_seg_id:05x}"
s["text_original"] = s.get("text", "")
@@ -684,33 +712,110 @@ async def dub_transcribe_stream(
return
def _diarize():
"""Returns (segments, warning_payload_or_None).
"""Returns (segments, warning_payload_or_None, labels_source).
`labels_source` records where the speaker labels came from —
`"pyannote"` | `"turns"` | `"heuristic"` — so downstream
auto-clone extraction can refuse to cut reference audio from
gap-based estimates (a mixed-speaker reference is how "made up"
clone voices happen).
`warning_payload` is a structured dict
`{detail, error_class, docs_url}` whenever we silently fell back
to the silence-gap heuristic (no HF_TOKEN, model unavailable,
license not accepted, or pyannote raised). The heuristic only
detects speaker turns from >1.2s silences, so a rapid-fire
man↔woman exchange will read as one speaker. Issue #78 — we
attach an `error_class` so the front-end's errorDocsMap can
render a "See docs" deeplink instead of a dead-end toast.
license not accepted, or pyannote raised) — or whenever the
user's `num_speakers` hint could not be honored exactly. The
heuristic only detects speaker turns from >1.2s silences, so a
rapid-fire man↔woman exchange will read as one speaker. Issue
#78 — we attach an `error_class` so the front-end's errorDocsMap
can render a "See docs" deeplink instead of a dead-end toast.
"""
# The active ASR backend already diarized inline (FunASR cam++):
# use its speaker turns directly and skip pyannote entirely (#182).
if asr_speaker_turns:
logger.info("Using inline ASR diarization (%d turns); skipping pyannote.", len(asr_speaker_turns))
assigned = assign_speakers_from_turns(all_segments, asr_speaker_turns)
# #486: split any segment that spans two speakers' turns at the
# word boundary (single-speaker segments pass through unchanged).
return resplit_segments_by_turns(assigned, all_words, asr_speaker_turns), None
from services.model_manager import (
DIARIZATION_ERR_LICENSE,
DIARIZATION_ERR_NO_TOKEN,
)
from core import error_docs_map
diar_pipe, err_sentinel = get_diarization_pipeline(return_error=True)
def _hint_suffix() -> str:
"""Honest caveat appended to heuristic-fallback warnings when a
multi-speaker hint is set: the count is now honored, but the
heuristic can't attribute voices. (A hint of 1 IS fully
honored — one label — so it needs no caveat.)"""
if not num_speakers or num_speakers < 2:
return ""
return (
f" Your speaker-count setting ({num_speakers}) is only "
f"approximately honored: the heuristic cycles "
f"{num_speakers} speaker labels on silence gaps instead "
f"of recognizing voices, so lines may be attributed to "
f"the wrong speaker."
)
def _use_turns(crash: Exception | None = None, err_sentinel=None):
"""Label from the ASR backend's inline speaker turns; warn when
that means the user's explicit count can't be enforced."""
logger.info(
"Using inline ASR diarization (%d turns)%s.",
len(asr_speaker_turns),
"" if crash else "; skipping pyannote",
)
assigned = assign_speakers_from_turns(all_segments, asr_speaker_turns)
# #486: split any segment that spans two speakers' turns at the
# word boundary (single-speaker segments pass through unchanged).
resplit = resplit_segments_by_turns(assigned, all_words, asr_speaker_turns)
if not num_speakers:
return resplit, None, "turns"
error_class = (
"HF_AUTH_FAILED"
if err_sentinel == DIARIZATION_ERR_NO_TOKEN
else "PYANNOTE_LICENSE_REQUIRED"
)
if crash:
detail = (
f"Speaker diarization crashed mid-run "
f"({type(crash).__name__}); falling back to the ASR "
f"engine's built-in speaker turns. Speaker-count hint "
f"ignored: the detected count may differ from the "
f"{num_speakers} you set."
)
else:
detail = (
f"Speaker-count hint ignored: pyannote diarization is "
f"unavailable, so the ASR engine's built-in speaker "
f"turns were used and the detected count may differ "
f"from the {num_speakers} you set. Set up diarization "
f"(Settings → Models → pyannote) to enforce an exact "
f"speaker count."
)
return resplit, {
"detail": detail,
"error_class": error_class,
"docs_url": error_docs_map.lookup(error_class),
"speaker_hint": {"requested": num_speakers, "status": "ignored"},
}, "turns"
# The active ASR backend already diarized inline (FunASR cam++):
# its turns are the fast path and skip pyannote entirely (#182) —
# but ONLY when the user didn't set an explicit speaker count.
# Inline turns are labeled per-30s-chunk and can't be forced to N
# speakers, so a set num_speakers prefers pyannote — the one
# engine that honors an exact count. When pyannote can't load,
# the turns are still the best labels available; use them and say
# so instead of silently eating the hint.
diar_pipe = None
err_sentinel = None
if asr_speaker_turns:
if num_speakers:
diar_pipe, err_sentinel = get_diarization_pipeline(return_error=True)
if not diar_pipe:
return _use_turns(err_sentinel=err_sentinel)
logger.info(
"num_speakers=%d set: preferring pyannote over %d inline "
"ASR turns (only pyannote honors an exact count).",
num_speakers, len(asr_speaker_turns),
)
else:
diar_pipe, err_sentinel = get_diarization_pipeline(return_error=True)
if not diar_pipe:
# Phase 1 AUTH-01: ask the resolver (App → Env → HF-CLI),
# not just the env var. This is the #35 fix — users who
@@ -761,13 +866,20 @@ async def dub_transcribe_stream(
f"heuristic; rapid speaker turns may be merged."
)
error_class = "PYANNOTE_LICENSE_REQUIRED"
warning = {
"detail": detail + _hint_suffix(),
"error_class": error_class,
"docs_url": error_docs_map.lookup(error_class),
}
if num_speakers:
warning["speaker_hint"] = {
"requested": num_speakers,
"status": "approximate" if num_speakers > 1 else "honored",
}
return (
assign_speakers_heuristic(all_segments),
{
"detail": detail,
"error_class": error_class,
"docs_url": error_docs_map.lookup(error_class),
},
assign_speakers_heuristic(all_segments, num_speakers),
warning,
"heuristic",
)
try:
# Pass the user's speaker-count hint through to pyannote when
@@ -782,9 +894,14 @@ async def dub_transcribe_stream(
assigned = assign_speakers_from_diarization(all_segments, diar)
# #486: split any segment that spans two speakers' turns at the
# word boundary (single-speaker segments pass through unchanged).
return resplit_segments_by_diarization(assigned, all_words, diar), None
return resplit_segments_by_diarization(assigned, all_words, diar), None, "pyannote"
except Exception as e:
logger.error(f"Diarization failed: {e}")
# Inline ASR turns beat the silence-gap heuristic as a crash
# fallback (this path is reachable with turns present since a
# set num_speakers routes turns-jobs through pyannote).
if asr_speaker_turns:
return _use_turns(crash=e)
# Mid-run failure — classify against the same sentinels so a
# post-load 401 (rare but possible after a token rotation)
# still gets the right docs deeplink.
@@ -795,36 +912,50 @@ async def dub_transcribe_stream(
if err_class_post == DIARIZATION_ERR_LICENSE
else "PYANNOTE_LICENSE_REQUIRED" # LOAD failures land here too
)
warning = {
"detail": (
f"Speaker diarization crashed mid-run "
f"({type(e).__name__}); falling back to a silence-gap "
f"heuristic. Rapid speaker turns may be merged."
+ _hint_suffix()
),
"error_class": error_class,
"docs_url": error_docs_map.lookup(error_class),
}
if num_speakers:
warning["speaker_hint"] = {
"requested": num_speakers,
"status": "approximate" if num_speakers > 1 else "honored",
}
return (
assign_speakers_heuristic(all_segments),
{
"detail": (
f"Speaker diarization crashed mid-run "
f"({type(e).__name__}); falling back to a silence-gap "
f"heuristic. Rapid speaker turns may be merged."
),
"error_class": error_class,
"docs_url": error_docs_map.lookup(error_class),
},
assign_speakers_heuristic(all_segments, num_speakers),
warning,
"heuristic",
)
fut_diar = loop.run_in_executor(_gpu_pool, _diarize)
final_segs = None
diar_warning = None
labels_source = "heuristic"
while True:
done, pending = await asyncio.wait([fut_diar], timeout=5.0)
if done:
final_segs, diar_warning = done.pop().result()
final_segs, diar_warning, labels_source = done.pop().result()
break
yield _sse_event("ping", {})
if diar_warning:
logger.warning("diarization fallback: %s", diar_warning.get("detail"))
yield _sse_event("warning", {
payload = {
"detail": diar_warning.get("detail"),
"source": "diarization",
"error_class": diar_warning.get("error_class"),
"docs_url": diar_warning.get("docs_url"),
})
}
# Machine-readable trail of what happened to the user's
# speaker-count hint (the `detail` text carries the human story).
if diar_warning.get("speaker_hint"):
payload["speaker_hint"] = diar_warning["speaker_hint"]
yield _sse_event("warning", payload)
job["segments"] = final_segs
@@ -836,17 +967,37 @@ async def dub_transcribe_stream(
try:
from services.speaker_clone import extract_speaker_clones, auto_profile_id
vocals_for_clone = job.get("vocals_path") or asr_audio_target
fut_clones = loop.run_in_executor(
_cpu_pool, extract_speaker_clones,
vocals_for_clone, final_segs, os.path.dirname(vocals_for_clone),
)
clones = None
while True:
done, pending = await asyncio.wait([fut_clones], timeout=5.0)
if done:
clones = done.pop().result()
break
yield _sse_event("ping", {})
clones = {}
if labels_source == "heuristic":
# Clone-purity guard: heuristic labels are silence-gap
# estimates, not voice identity — a per-speaker reference cut
# from them routinely concatenates two people's audio and the
# clone sounds "made up". Skip auto-clones and say so instead
# of shipping bad ones. (extract_speaker_clones enforces the
# same guard internally; this branch exists to surface the
# warning to the user.)
logger.info(
"auto speaker clones skipped (labels_source=heuristic, job=%s)",
job_id,
)
yield _sse_event("warning", {
"detail": CLONE_SKIP_HEURISTIC_MSG,
"source": "speaker_clone",
})
else:
fut_clones = loop.run_in_executor(
_cpu_pool, lambda: extract_speaker_clones(
vocals_for_clone, final_segs,
os.path.dirname(vocals_for_clone),
labels_source=labels_source,
),
)
while True:
done, pending = await asyncio.wait([fut_clones], timeout=5.0)
if done:
clones = done.pop().result()
break
yield _sse_event("ping", {})
# Wave 3.2: per-segment clone refs. Cut each long-enough segment's
# own reference from the vocals so the dub of each line matches the
# prosody of its source line. Short lines fall back to the
@@ -960,7 +1111,14 @@ async def dub_transcribe_stream(
@router.post("/dub/transcribe/{job_id}")
async def dub_transcribe(job_id: str):
async def dub_transcribe(job_id: str, num_speakers: Optional[int] = None):
"""Legacy synchronous transcribe (kept for the headless CLI).
`num_speakers` mirrors the SSE endpoint's query param (same 120 clamp):
an exact speaker count forwarded to pyannote, or cycled by the silence-gap
heuristic when pyannote is unavailable. None → auto-detect.
"""
num_speakers = _clamp_num_speakers(num_speakers)
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
@@ -1026,13 +1184,20 @@ async def dub_transcribe(job_id: str):
if diar_pipe:
try:
diar_target = job.get("vocals_path") or job.get("audio_path")
diarization = diar_pipe(diar_target)
# Same hint pass-through as the SSE endpoint (#274): omit the
# kwarg entirely when unset so we don't depend on it existing
# in every pyannote build.
if num_speakers:
logger.info("Diarizing with num_speakers=%d (user hint)", num_speakers)
diarization = diar_pipe(diar_target, num_speakers=num_speakers)
else:
diarization = diar_pipe(diar_target)
segments = assign_speakers_from_diarization(segments, diarization)
except Exception as e:
logger.error(f"Pyannote diarization failed during inference: {e}. Falling back to heuristic.")
segments = assign_speakers_heuristic(segments)
segments = assign_speakers_heuristic(segments, num_speakers)
else:
segments = assign_speakers_heuristic(segments)
segments = assign_speakers_heuristic(segments, num_speakers)
# Previously ran `segment_for_subtitles(segments)` here. Removed 2026-04-21 —
# that splitter enforces Netflix's 17 CPS reading-speed ceiling which
+57 -7
View File
@@ -308,14 +308,63 @@ async def dub_translate(req: TranslateRequest):
# first is fine — the refine LLM is a separate network provider.
return await _maybe_cinematic(translated, req, src_lang, loop)
# OpenAI / Ollama Local LLM Translation
# LLM translation — resolves through the LLM Skills registry: per-skill
# "Dub translation" override → global active provider (Settings → LLM
# Providers). The keys users configure + test in the app now actually
# power this engine; the raw TRANSLATE_* env vars stay working as a
# power-user override so pre-skills setups see zero behavior change.
if provider == "openai":
base_url = os.environ.get("TRANSLATE_BASE_URL")
model_name = os.environ.get("TRANSLATE_MODEL", "gpt-3.5-turbo")
from openai import OpenAI
# max_retries=0: a 429 + long Retry-After must not let one segment's
# SDK call sleep+retry and blow the overall translate wall time.
client = OpenAI(base_url=base_url, api_key=api_key or "local", max_retries=0)
from services import llm_skills
llm_timeout = llm_skills._default_timeout()
handle = None
try:
handle = llm_skills.resolve_skill_client("dub_translation")
except Exception: # noqa: BLE001 — resolution must never 500 a translate
logger.exception("dub_translation skill resolution failed; trying env fallback")
if handle is not None:
client = handle.client
model_name = handle.model
llm_timeout = handle.timeout
# The provider-store key never touches env; resolve it so the
# error scrubber below can redact it if a provider echoes it.
try:
from services import llm_providers
api_key = llm_providers.resolve_api_key(
llm_skills.effective_provider("dub_translation")) or api_key
except Exception: # noqa: BLE001 — scrub-key resolution is best-effort
pass
elif os.environ.get("TRANSLATE_BASE_URL") or api_key:
# Legacy env-only setup (no provider configured in-app).
from openai import OpenAI
# max_retries=0: a 429 + long Retry-After must not let one segment's
# SDK call sleep+retry and blow the overall translate wall time.
client = OpenAI(base_url=os.environ.get("TRANSLATE_BASE_URL"),
api_key=api_key or "local", max_retries=0)
model_name = os.environ.get("TRANSLATE_MODEL", "gpt-4o-mini")
else:
# Nothing configured anywhere — name the exact next step instead
# of letting an empty key surface as a raw 401 per segment.
try:
reason = llm_skills.resolve_skill("dub_translation").reason
except Exception: # noqa: BLE001
reason = None
if reason == "disabled":
friendly = (
"The LLM translation engine is turned off — enable the "
"'Dub translation' skill in Settings → LLM Skills, or "
"pick another engine in the Engine dropdown."
)
else:
friendly = (
"The LLM translation engine has no provider configured. "
"Add and test one in Settings → LLM Providers (it powers "
"this engine; route it per-skill in Settings → LLM "
"Skills), or set TRANSLATE_BASE_URL + TRANSLATE_API_KEY "
"+ TRANSLATE_MODEL. Or pick another engine in the "
"Engine dropdown."
)
return JSONResponse(status_code=400, content={"error": friendly})
def _build_prompt(src_code: str, tgt_code: str) -> str:
"""Build a system prompt that resists hallucinations on small
@@ -377,6 +426,7 @@ async def dub_translate(req: TranslateRequest):
res = client.chat.completions.create(
model=model_name,
temperature=0.2, # less drift than default 1.0
timeout=llm_timeout, # bound per call (OMNIVOICE_LLM_TIMEOUT, 45s default)
messages=[
{"role": "system", "content": sys_for_attempt},
{"role": "user", "content": seg.text},
+1 -1
View File
@@ -24,7 +24,7 @@ from pathlib import Path
# tests/test_app_version.py::test_all_version_files_in_lockstep and bumped by
# release.yml's version-bump job, so it stays equal to
# pyproject/tauri.conf/Cargo/package.json.
_FALLBACK_VERSION = "0.3.9"
_FALLBACK_VERSION = "0.3.10"
def _fallback_version() -> str:
+7 -1
View File
@@ -1662,7 +1662,13 @@ class _LazyASRRegistry(dict):
def __iter__(self):
seen = set()
for k in dict.__iter__(self):
# Snapshot the live keys before yielding — see _LazyRegistry.__iter__ in
# tts_backend.py. A concurrent lazy __getitem__ inserts into self, and
# list_backends() runs in a FastAPI threadpool, so a *live* dict iterator
# held open across the per-engine is_available() probes would raise
# "dictionary changed size during iteration". list() consumes it
# atomically under the GIL, closing the window.
for k in list(dict.__iter__(self)):
seen.add(k)
yield k
for k in self._LAZY:
+21 -8
View File
@@ -104,9 +104,10 @@ def synthesize_chapter(
):
"""Render a chapter's spans to one waveform via an injected ``synth``.
``synth(text, voice_id, speed)`` returns a 1-D float32 audio tensor for a
span of text in the given voice (``speed`` may be ``None`` for the engine
default). Long spans are split with the ``chunked_tts`` splitter and
``synth(text, voice_id, speed)`` returns a float32 audio tensor 1-D
``(samples,)`` or ``(channels, samples)``; real engines emit ``(1, samples)``
per the ``TTSBackend`` contract (#897) — for a span of text in the given
voice (``speed`` may be ``None`` for the engine default). Long spans are split with the ``chunked_tts`` splitter and
crossfaded; inter-span ``pause_ms_after`` becomes silence. ``lexicon`` (when
given) respells each span's text before chunking so the engine pronounces
tricky words correctly; a ``None``/empty lexicon is a no-op pass-through.
@@ -118,23 +119,35 @@ def synthesize_chapter(
from services.chunked_tts import concatenate_audio_chunks, split_text_into_chunks
from services.pronunciation import apply_lexicon
parts: list = []
items: list = [] # ("a", tensor) for audio, ("s", n_samples) for silence
for span in spans:
if span.text:
chunks = split_text_into_chunks(apply_lexicon(span.text, lexicon))
rendered = [synth(c, span.voice_id, span.speed) for c in chunks]
rendered = [r for r in rendered if r is not None and getattr(r, "numel", lambda: 0)()]
if len(rendered) == 1:
parts.append(rendered[0])
items.append(("a", rendered[0]))
elif rendered:
parts.append(concatenate_audio_chunks(rendered, sample_rate, crossfade_ms=crossfade_ms))
items.append(("a", concatenate_audio_chunks(rendered, sample_rate, crossfade_ms=crossfade_ms)))
if span.pause_ms_after > 0:
n = int(sample_rate * span.pause_ms_after / 1000.0)
if n > 0:
parts.append(torch.zeros(n, dtype=torch.float32))
items.append(("s", n))
if not parts:
if not items:
return torch.zeros(0, dtype=torch.float32), 0.0
# Engines return (1, samples) per the TTSBackend contract while a bare
# zeros(n) is 1-D — mixing the two crashed the final concat (#897). So
# materialize inter-span silence AFTER the loop, matching the rendered
# audio's channel dims / dtype / device (same pattern as generation.py's
# _render_with_pauses). A silence-only chapter stays 1-D float32 as before.
ref = next((t for kind, t in items if kind == "a"), None)
parts: list = [
val if kind == "a"
else (torch.zeros(val, dtype=torch.float32) if ref is None
else torch.zeros(*ref.shape[:-1], val, dtype=ref.dtype, device=ref.device))
for kind, val in items
]
# Hard-concat spans + silences (crossfading silence would bleed the gap).
audio = parts[0] if len(parts) == 1 else concatenate_audio_chunks(parts, sample_rate, crossfade_ms=0)
return audio, audio.shape[-1] / float(sample_rate)
+32 -2
View File
@@ -183,14 +183,43 @@ def _safe_hard_cut(segment: str, max_chars: int) -> int:
return cut
def _normalize_chunk_shapes(chunks: list) -> list:
"""Coerce mixed-rank / mixed-channel chunks to one concat-compatible shape.
Engines return ``(1, samples)`` per the ``TTSBackend.generate`` contract,
but silence buffers and some model paths hand over bare ``(samples,)``
tensors ``torch.cat`` then dies with "Tensors must have same number of
dimensions" (#897). Promote lower-rank chunks with leading singleton dims
to the highest rank present, then broadcast singleton channel dims up to
the widest channel count (mono follows stereo). Rank-homogeneous,
channel-homogeneous input is returned untouched, so all-1-D / all-2-D
callers keep their exact output shape; a genuine channel conflict
(e.g. 2 vs 3 channels) still raises, which is the honest outcome.
"""
target = max(c.dim() for c in chunks)
if any(c.dim() != target for c in chunks):
promoted = []
for c in chunks:
while c.dim() < target:
c = c.unsqueeze(0)
promoted.append(c)
chunks = promoted
if target > 1:
lead = tuple(max(c.shape[i] for c in chunks) for i in range(target - 1))
chunks = [c if tuple(c.shape[:-1]) == lead else c.expand(*lead, -1)
for c in chunks]
return chunks
def concatenate_audio_chunks(chunks: list, sample_rate: int,
crossfade_ms: int = DEFAULT_CROSSFADE_MS):
"""Join per-chunk waveforms with a linear crossfade on the sample axis.
``chunks`` are torch tensors as returned by the engine (1-D, or N-D with
samples on the last axis matching what ``_render_with_pauses`` handles).
Crossfade overlap is clamped to the shorter neighbor; ``crossfade_ms=0``
is a hard concat.
Mixed ranks / mono-vs-multichannel chunks are normalized to one shape
first (#897), so no producer can crash the concat. Crossfade overlap is
clamped to the shorter neighbor; ``crossfade_ms=0`` is a hard concat.
"""
import torch
@@ -199,6 +228,7 @@ def concatenate_audio_chunks(chunks: list, sample_rate: int,
return torch.zeros(1, dtype=torch.float32)
if len(chunks) == 1:
return chunks[0]
chunks = _normalize_chunk_shapes(chunks)
crossfade_samples = int(sample_rate * crossfade_ms / 1000)
result = chunks[0]
+17 -3
View File
@@ -54,9 +54,12 @@ class LLMBackend(ABC):
def model_name(self) -> str: ...
@abstractmethod
def chat(self, *, system: str, user: str, timeout: Optional[float] = None) -> str:
def chat(self, *, system: str, user: str, timeout: Optional[float] = None,
temperature: Optional[float] = None) -> str:
"""One-shot chat completion. Returns the assistant content string.
Raises on failure callers decide whether to fallback gracefully.
``temperature`` is only sent to the provider when set callers that
leave it None keep the provider default (existing behavior).
"""
@@ -132,32 +135,43 @@ class OpenAICompatBackend(LLMBackend):
self._client = OpenAI(max_retries=0, **kw)
return self._client
def chat(self, *, system: str, user: str, timeout: Optional[float] = None) -> str:
def chat(self, *, system: str, user: str, timeout: Optional[float] = None,
temperature: Optional[float] = None) -> str:
return self.chat_messages(
messages=[
{"role": "system", "content": system},
{"role": "user", "content": user},
],
timeout=timeout,
temperature=temperature,
)
def chat_messages(self, *, messages: list[dict], timeout: Optional[float] = None) -> str:
def chat_messages(self, *, messages: list[dict], timeout: Optional[float] = None,
temperature: Optional[float] = None) -> str:
"""One-shot completion over a full message list.
Additive surface for callers that need structured few-shot turns
(dictation refinement, Wave 2.1) small local models pattern-match
and echo inline examples, so examples must arrive as prior chat
turns, not inside the system prompt.
``temperature`` is only forwarded when set (Cinematic/Autofit pin 0.2
the provider default of 1.0 makes local models drift and invent);
every other caller leaves it None and keeps the provider default.
"""
if timeout is None:
try:
timeout = float(os.environ.get("OMNIVOICE_LLM_TIMEOUT", "45"))
except ValueError:
timeout = 45.0
kw = {}
if temperature is not None:
kw["temperature"] = temperature
res = self._get_client().chat.completions.create(
model=self.model_name,
timeout=timeout,
messages=messages,
**kw,
)
return (res.choices[0].message.content or "").strip()
+5 -2
View File
@@ -5,8 +5,10 @@ the Settings → LLM Skills panel can (a) toggle it and (b) route it to a
specific provider (a local Ollama/LM Studio vs a remote key) instead of
everything riding the one global active provider.
The five consumption points today:
The six consumption points today:
dub_translation api/routers/dub_translate.py (the Dub tab's direct
"LLM" translation engine; provider=openai branch)
cinematic_translation services/translator.py (Cinematic + Autofit
REFLECT/ADAPT rewrite; dub_translate quality gate)
slot_fitting services/speech_rate.py (trim/expand a line to its
@@ -68,8 +70,9 @@ def _skill(sid: str) -> LLMSkill:
# Display order in the settings panel: the dub pipeline first (translation →
# fit → glossary → direction), then dictation.
# refine → fit → glossary → direction), then dictation.
_SKILLS: tuple[LLMSkill, ...] = (
_skill("dub_translation"),
_skill("cinematic_translation"),
_skill("slot_fitting"),
_skill("glossary_extract"),
+33 -6
View File
@@ -532,14 +532,41 @@ def assign_speakers_from_turns(
return segments
def assign_speakers_heuristic(segments: List[dict]) -> List[dict]:
"""Two-speaker alternation based on silence gaps."""
current = 1
def assign_speakers_heuristic(
segments: List[dict], num_speakers: Optional[int] = None
) -> List[dict]:
"""Silence-gap speaker assignment (used when no diarization model runs).
Base signal: a gap > SPEAKER_GAP seconds between consecutive segments is
treated as a speaker change. Without a ``num_speakers`` hint this keeps
the legacy behavior alternate between exactly two labels. With a hint:
* ``num_speakers=1`` every segment gets ``"Speaker 1"``.
* ``num_speakers>=2`` labels round-robin across N speakers at each
gap boundary, so the user's requested count is represented instead of
being silently capped at 2.
Limits (be honest with callers): this honors the *count*, not voice
identity. The rotation order is arbitrary (a returning speaker gets the
next label in the cycle, not their own), rapid exchanges with no
> SPEAKER_GAP pause still collapse into one label, and N is an upper
bound audio with fewer gap boundaries than N yields fewer labels.
Real per-speaker attribution needs pyannote (or an inline-diarizing ASR
backend); callers should warn the user accordingly (see dub_core).
Invalid hints (non-int, < 1) fall back to the legacy two-speaker cycle.
"""
try:
n = int(num_speakers) if num_speakers is not None else 2
except (TypeError, ValueError):
n = 2
if n < 1:
n = 2
current = 0 # zero-based rotation index; rendered one-based below
last_end = 0.0
for i, s in enumerate(segments):
if i > 0 and (s["start"] - last_end) > SPEAKER_GAP:
current = 2 if current == 1 else 1
s["speaker_id"] = f"Speaker {current}"
if i > 0 and n > 1 and (s["start"] - last_end) > SPEAKER_GAP:
current = (current + 1) % n
s["speaker_id"] = f"Speaker {current + 1}"
last_end = s["end"]
return segments
+101 -13
View File
@@ -42,11 +42,25 @@ IDEAL_REF_DURATION_S = 8.0 # target window — long enough for prosody, short e
# is the empirical floor below which our zero-shot clone gets unstable.
MIN_SEGMENT_REF_DURATION_S = 3.0
# Clone-purity guards (speaker-hint fix): a per-speaker reference cut from
# mislabeled or boundary-adjacent audio mixes two people's voices and the
# resulting clone sounds "made up".
# * A slice below MIN_SLICE_DURATION_S is too short to be a reliable
# single-speaker sample (and diarization boundary jitter dominates it).
# * A slice whose edges come within ADJACENT_TURN_GUARD_S of a *different*
# speaker's turn risks bleeding that speaker's audio across the imprecise
# boundary — deprioritized (scoring preference, not a hard filter, so
# extraction still succeeds on dense dialogue).
MIN_SLICE_DURATION_S = 1.5
ADJACENT_TURN_GUARD_S = 0.3
def extract_speaker_clones(
vocals_path: str,
segments: list[dict],
out_dir: str,
*,
labels_source: str | None = None,
) -> dict[str, dict]:
"""Build a per-speaker reference sample from `vocals_path` + `segments`.
@@ -63,7 +77,20 @@ def extract_speaker_clones(
Speakers whose segments total < MIN_REF_DURATION_S are skipped we'd
rather fall back to the default TTS voice than ship a bad clone.
``labels_source`` records where the ``speaker_id`` labels came from
(``"pyannote"`` | ``"turns"`` | ``"heuristic"``; ``None`` = unknown,
treated as trusted for backward compatibility). ``"heuristic"`` labels
are silence-gap *estimates*, not voice identity a reference cut from
them routinely concatenates two people's audio, so extraction is skipped
entirely (the caller warns the user and falls back to the default voice).
"""
if labels_source == "heuristic":
logger.info(
"speaker_clone: skipping auto-clone extraction — speaker labels "
"are gap-based heuristic estimates, not voice identity"
)
return {}
if not vocals_path or not os.path.exists(vocals_path):
logger.info("speaker_clone: no vocals track at %s; skipping", vocals_path)
return {}
@@ -88,7 +115,12 @@ def extract_speaker_clones(
out: dict[str, dict] = {}
for speaker_id, items in by_speaker.items():
chosen = _pick_reference_slices(items)
chosen = _pick_reference_slices(
items,
speaker_id=speaker_id,
all_segments=segments,
labels_source=labels_source,
)
if not chosen:
logger.info(
"speaker_clone: %s has <%ss of usable audio; will fall back to default voice",
@@ -194,31 +226,87 @@ def extract_segment_refs(
# ── Internals ───────────────────────────────────────────────────────────────
def _pick_reference_slices(items: list[tuple[int, dict]]) -> list[tuple[int, dict]]:
def _adjacent_to_other_speaker(
seg: dict, speaker_id: str, all_segments: list[dict] | None
) -> bool:
"""True when `seg`'s edges come within ADJACENT_TURN_GUARD_S of (or
overlap) a segment attributed to a *different* speaker a boundary where
imprecise diarization timestamps risk bleeding the other voice into the
reference slice."""
if not all_segments:
return False
s0 = float(seg.get("start", 0.0))
s1 = float(seg.get("end", 0.0))
for other in all_segments:
if other is seg:
continue
if (other.get("speaker_id") or "Speaker 1") == speaker_id:
continue
o0 = float(other.get("start", 0.0))
o1 = float(other.get("end", 0.0))
# Signed gap between the two spans; negative = overlap.
if max(o0 - s1, s0 - o1) < ADJACENT_TURN_GUARD_S:
return True
return False
def _pick_reference_slices(
items: list[tuple[int, dict]],
*,
speaker_id: str | None = None,
all_segments: list[dict] | None = None,
labels_source: str | None = None,
) -> list[tuple[int, dict]]:
"""Select the subset of a speaker's segments to use as reference audio.
Strategy: take the single longest segment; if it's short, accumulate the
next longest ones in original order until we clear IDEAL_REF_DURATION_S.
Cap at MAX_REF_DURATION_S. Return [] if we can't reach MIN_REF_DURATION_S.
Strategy: rank candidates clean-first (not temporally adjacent to a
different speaker's turn — see ``_adjacent_to_other_speaker``), longest
first within each tier, and accumulate until IDEAL_REF_DURATION_S is
cleared. Adjacency is a scoring preference, NOT a hard filter on dense
dialogue where every slice borders another speaker, extraction still
succeeds using the adjacent ones. Two hard guards protect clone purity:
* slices shorter than MIN_SLICE_DURATION_S are rejected outright
(boundary jitter dominates them, so they're the likeliest to carry a
second speaker's audio);
* ``labels_source="heuristic"`` returns [] gap-based labels are not
voice identity, so no slice of them is safe to clone from.
Cap at MAX_REF_DURATION_S. Return [] if we can't reach
MIN_REF_DURATION_S. When ``all_segments``/``speaker_id`` are not
provided (legacy callers), adjacency scoring degrades to duration-only
the pre-guard behavior.
"""
if not items:
return []
if labels_source == "heuristic":
return []
if speaker_id is None:
speaker_id = items[0][1].get("speaker_id") or "Speaker 1"
# Longest-first candidates. Keep original indices so we can preserve order.
by_dur = sorted(
def _dur(pair) -> float:
return max(0.0, float(pair[1].get("end", 0.0)) - float(pair[1].get("start", 0.0)))
# Rank: clean (non-adjacent) before adjacent, longest first within each
# tier. Keep original indices so we can restore transcript order below.
ranked = sorted(
items,
key=lambda pair: (pair[1].get("end", 0.0) - pair[1].get("start", 0.0)),
reverse=True,
key=lambda pair: (
_adjacent_to_other_speaker(pair[1], speaker_id, all_segments),
-_dur(pair),
),
)
picked: list[tuple[int, dict]] = []
total = 0.0
for idx, seg in by_dur:
dur = max(0.0, float(seg.get("end", 0.0)) - float(seg.get("start", 0.0)))
if dur <= 0.0:
for idx, seg in ranked:
dur = _dur((idx, seg))
if dur < MIN_SLICE_DURATION_S:
continue
if total + dur > MAX_REF_DURATION_S and picked:
break
# Ranking is no longer duration-monotonic, so a later (shorter or
# adjacent) slice may still fit — skip, don't stop.
continue
picked.append((idx, seg))
total += dur
if total >= IDEAL_REF_DURATION_S:
+60 -10
View File
@@ -18,6 +18,10 @@ import logging
from typing import Iterable, Optional
from services.llm_backend import get_active_llm_backend, OffBackend
# Shared LLM-output divergence guard (length window + target-script +
# critique-echo). Lives in translator; translator never imports this module,
# so there is no import cycle.
from services.translator import refine_output_ok
logger = logging.getLogger("omnivoice.speech_rate")
@@ -91,9 +95,15 @@ _EXPAND_PROMPT = """\
You are a dubbing writer. The user will give you a translated line + the exact
time slot it must fit. The current line is TOO SHORT add natural filler or
gently flesh out the thought while keeping the meaning the same. Aim for a
reading duration that matches the slot.
reading duration that matches the slot. Never invent new information, names,
or dialogue that is not already in the line; do not more than double the line.
Reply with ONLY the new line. No quotes, no commentary."""
# Below this predicted rate ratio a line can never honestly fill its slot —
# any LLM "expansion" that far would be fabricated dialogue. Skip the expand
# pass entirely and keep the short line (slot-aware TTS absorbs the silence).
_MIN_EXPANDABLE_RATIO = 0.15
def adjust_for_slot(
text: str,
@@ -107,17 +117,36 @@ def adjust_for_slot(
Falls back to the input text if the LLM is off or the loop gives up.
``strict`` (Autofit mode) caps the accepted upper bound at 1.0 instead of
``TOL_HIGH`` i.e. the line must fit *within* the slot, never overrun it
so the target-language reading time can't exceed the segment and push the
video timing out. A too-short line is still accepted down to ``TOL_LOW`` (we
don't pad just to fill silence). Best-effort: after ``MAX_ATTEMPTS`` it
returns the closest candidate seen, so a stubborn line degrades gracefully.
``strict`` (Autofit mode) changes exactly one thing: the accepted upper
bound is 1.0 instead of ``TOL_HIGH`` the line must fit *within* the
slot, never overrun it so the target-language reading time can't exceed
the segment and push the video timing out. Lines under ``TOL_LOW`` still
go through the LLM expand pass in strict mode too (same as loose mode);
padding is bounded by the divergence guard below, and a line under
``_MIN_EXPANDABLE_RATIO`` is never expanded at all it could only "fill"
the slot with fabricated dialogue, so it stays short. Best-effort: after
``MAX_ATTEMPTS`` it returns the closest candidate seen, so a stubborn line
degrades gracefully.
Every LLM reply is validated with ``translator.refine_output_ok`` against
the ORIGINAL input ``text`` (not the previous candidate divergence
compounds across attempts otherwise). A reply that fails the guard is
discarded: the attempt is burned, ``current``/``best`` stay put, and if
nothing valid ever came back the input text is returned with
``error="fit-diverged"`` a hallucinating model can no longer invent the
dub line (v0.3.9 field report).
"""
tol_high = 1.0 if strict else TOL_HIGH
initial_ratio = rate_ratio(text, slot_seconds, target_lang)
if TOL_LOW <= initial_ratio <= tol_high:
return {"text": text, "rate_ratio": initial_ratio, "attempts": 0}
if initial_ratio < _MIN_EXPANDABLE_RATIO:
return {
"text": text,
"rate_ratio": initial_ratio,
"attempts": 0,
"error": "fit-skip-short",
}
from services import llm_skills
# `active=` forwards this module's (monkeypatch-able) name so the
@@ -133,6 +162,7 @@ def adjust_for_slot(
current = text
best = (current, initial_ratio)
diverged = False
for attempt in range(1, MAX_ATTEMPTS + 1):
r = rate_ratio(current, slot_seconds, target_lang)
if TOL_LOW <= r <= tol_high:
@@ -150,23 +180,43 @@ def adjust_for_slot(
user_lines.append(f"Source line (for meaning): {source_text}")
try:
next_text = llm.chat(system=system, user="\n".join(user_lines))
next_text = llm.chat(
system=system, user="\n".join(user_lines),
temperature=0.2, # pinned like the Fast path — default 1.0 drifts/invents
)
except Exception as e:
logger.warning("speech-rate attempt %d failed: %s", attempt, e)
return {"text": best[0], "rate_ratio": best[1], "attempts": attempt - 1, "error": str(e)}
if next_text and next_text.strip():
current = next_text.strip()
candidate = next_text.strip()
# Divergence guard — validate against the ORIGINAL text, not
# `current`: each accepted reply becomes the next prompt's input,
# so per-step checks would let drift compound across attempts.
ok, reason = refine_output_ok(text, candidate, target_lang)
if not ok:
diverged = True
logger.warning(
"speech-rate attempt %d rejected (%s) — discarding candidate",
attempt, reason,
)
continue # attempt burned; current/best untouched
current = candidate
new_r = rate_ratio(current, slot_seconds, target_lang)
# Keep the best candidate seen so far in case we exhaust retries.
if abs(new_r - 1.0) < abs(best[1] - 1.0):
best = (current, new_r)
return {
out = {
"text": best[0],
"rate_ratio": best[1],
"attempts": MAX_ATTEMPTS,
}
# Every usable reply diverged and the input text survived unchanged —
# surface it on the row (rate_error in dub_translate, like fit-budget).
if diverged and best[0] == text:
out["error"] = "fit-diverged"
return out
def adjust_many(pairs: Iterable[tuple[str, float, str, Optional[str]]]) -> list[dict]:
+35 -5
View File
@@ -92,9 +92,11 @@ REGISTRY: dict[str, dict] = {
"category": "llm",
"needs_key": True,
"notes": (
"Any OpenAI-compatible endpoint: GPT-4/5 (OpenAI), Claude (via OpenRouter), "
"Gemini (OpenAI-compat mode), DeepSeek, Qwen, Ollama, LM Studio. "
"Set TRANSLATE_BASE_URL + TRANSLATE_API_KEY + TRANSLATE_MODEL."
"Uses the LLM provider you configure in Settings → LLM Providers "
"(route it via the 'Dub translation' skill in Settings → LLM Skills): "
"GPT (OpenAI), Claude (via OpenRouter), Gemini, DeepSeek, Qwen, "
"Ollama, LM Studio. Power-user env override: TRANSLATE_BASE_URL + "
"TRANSLATE_API_KEY + TRANSLATE_MODEL."
),
},
}
@@ -137,17 +139,45 @@ def install_command(engine: "str | dict | None") -> str | None:
return f"uv pip install {pkg}" if pkg else None
def _llm_configured() -> tuple[bool, "str | None"]:
"""Whether the LLM translation engine has something to call, and via what.
Resolution mirrors the translate-time path in dub_translate.py: the
"dub_translation" LLM skill (per-skill override active provider from
Settings LLM Providers) first, then the TRANSLATE_* env override. Lets
the Engine dropdown say "ready via <provider>" / "needs setup" up front
instead of a per-segment failure after the user clicks Translate.
"""
try:
from services import llm_skills
res = llm_skills.resolve_skill("dub_translation")
if res.ready and res.provider is not None:
return True, res.provider.display_name
except Exception: # noqa: BLE001 — a probe must never break list_engines()
logger.debug("dub_translation skill probe failed", exc_info=True)
if os.environ.get("TRANSLATE_BASE_URL") or os.environ.get("TRANSLATE_API_KEY"):
return True, "env"
return False, None
def list_engines() -> list[dict]:
"""Return a UI-ready list with per-engine availability stamped in."""
out = []
for e in REGISTRY.values():
installed, reason = _probe(e)
out.append({
entry = {
**e,
"installed": installed,
"availability_reason": reason,
"install_command": install_command(e),
})
}
# LLM engines additionally need a provider/key — surface configured-ness
# so the UI can distinguish "importable" from "actually ready to call".
if e.get("category") == "llm":
configured, via = _llm_configured()
entry["configured"] = configured
entry["configured_via"] = via
out.append(entry)
return out
+127 -17
View File
@@ -59,8 +59,9 @@ _ADAPT_PROMPT = """\
You are a cinematic dubbing writer. Rewrite the literal translation using the
editor's critique so it sounds natural, in-character, and fits the speaker's
time slot. Keep meaning faithful but prefer native idiom over word-for-word
accuracy. The output MUST be written in the same target language and script
as the literal translation never switch language or transliterate.
accuracy. Never introduce facts, names, or dialogue that are not present in
the source line. The output MUST be written in the same target language and
script as the literal translation never switch language or transliterate.
Reply ONLY with the adapted translation no quotes, no headers, no code
fences, no commentary."""
@@ -93,6 +94,107 @@ def _looks_like_target_script(text: str, code: str, threshold: float = 0.5) -> b
return (inside / len(letters)) >= threshold
# ── Divergence guard (shared with speech_rate's Autofit fit pass) ────────────
# For every Latin-script target `_looks_like_target_script` passes ANY text
# unconditionally (no `_SCRIPT_RANGES` entry), so it was the only — and for
# es/de/fr/… a no-op — gate on the ADAPT/fit LLM output. These checks close
# that gap for the whole class: runaway length (hallucinated dialogue,
# refusals, commentary) and the REFLECT critique echoed back as the "line".
_SHORT_REF_CHARS = 20 # below this, a length *ratio* is meaningless
_SHORT_REF_ABS_SLACK = 120 # …use an absolute cap instead: ref + this many chars
def _refine_ratio_bounds() -> tuple[float, float]:
"""Accepted ``len(candidate)/len(reference)`` window for LLM refine output.
Anything outside is treated as divergence and the caller degrades to its
input text. Defaults [0.4, 2.5]; env-tunable like the cinematic budget."""
try:
lo = float(os.environ.get("OMNIVOICE_REFINE_RATIO_MIN", "0.4"))
except ValueError:
lo = 0.4
try:
hi = float(os.environ.get("OMNIVOICE_REFINE_RATIO_MAX", "2.5"))
except ValueError:
hi = 2.5
return lo, hi
def _norm_overlap_text(s: str) -> str:
return " ".join(s.lower().split())
def _echoes_critique(candidate: str, critique: str) -> bool:
"""True when the "adaptation" is really the REFLECT critique leaking through.
Deterministic on purpose (no fuzzy matching): exact match after
case/whitespace normalization; containment the full critique inside the
candidate always counts, the candidate inside the critique only when it
covers most of it (critiques legitimately quote short phrases from the
line); or >0.8 token-set overlap.
"""
c = _norm_overlap_text(candidate)
k = _norm_overlap_text(critique)
if not c or not k:
return False
if c == k:
return True
if k in c: # critique embedded in the output
return True
if c in k and len(c) >= 0.6 * len(k): # output ≈ a big chunk of the critique
return True
ct, kt = set(c.split()), set(k.split())
union = ct | kt
return bool(union) and len(ct & kt) / len(union) > 0.8
def refine_output_ok(
reference: str,
candidate: str,
target_lang: str,
*,
critique: str | None = None,
max_ratio: float | None = None,
) -> tuple[bool, str | None]:
"""Sanity-check one LLM refine output against the text it was rewriting.
Shared by the Cinematic ADAPT step here and by ``speech_rate``'s Autofit
fit pass (speech_rate imports this; translator never imports speech_rate,
so there is no cycle). Returns ``(ok, reason)`` ``reason`` is ``None``
when ok, otherwise a short machine-readable tag for logs/error mapping.
Checks, in order:
script candidate must look like the target language's script
(``_looks_like_target_script``; Latin-script targets pass, as before);
length ``len(candidate)/len(reference)`` must sit inside
[``OMNIVOICE_REFINE_RATIO_MIN``, ``OMNIVOICE_REFINE_RATIO_MAX``]
(default 0.42.5; ``max_ratio`` overrides the upper bound). References
shorter than ~20 chars use an absolute cap (reference + 120 chars)
instead a two-word line legitimately doubles or halves;
critique echo the candidate must not be the critique itself.
"""
cand = (candidate or "").strip()
ref = (reference or "").strip()
if not cand:
return False, "empty"
if not _looks_like_target_script(cand, target_lang):
return False, f"wrong-script:{target_lang}"
lo, hi = _refine_ratio_bounds()
if max_ratio is not None:
hi = max_ratio
if ref:
if len(ref) < _SHORT_REF_CHARS:
if len(cand) > len(ref) + _SHORT_REF_ABS_SLACK:
return False, f"length-abs:{len(cand)}>{len(ref)}+{_SHORT_REF_ABS_SLACK}"
else:
ratio = len(cand) / len(ref)
if not (lo <= ratio <= hi):
return False, f"length-ratio:{ratio:.2f}"
if critique and _echoes_critique(cand, critique):
return False, "critique-echo"
return True, None
# The LLM Skills registry entry this pipeline resolves through — lets the
# user disable Cinematic/Autofit's LLM use or route it to a specific provider
# (Settings → LLM Skills) independently of the other LLM features.
@@ -173,6 +275,7 @@ def _chat(client, *, system: str, user: str) -> str:
res = client.chat.completions.create(
model=_llm_model(),
timeout=_llm_timeout(),
temperature=0.2, # pinned like the Fast path — default 1.0 drifts/invents
messages=[
{"role": "system", "content": system},
{"role": "user", "content": user},
@@ -274,21 +377,28 @@ def cinematic_refine_sync(
}
final = (adapted or "").strip() or literal_text
# Refuse adaptations that drifted off the target script (e.g. local LLM
# rewrote a Devanagari line in Latin/German). Caller still gets the
# critique so the UI can show what happened, but the live text falls
# back to the literal translation rather than corrupting the dub.
if final is not literal_text and not _looks_like_target_script(final, target_lang):
logger.warning(
"cinematic adapt produced wrong-script output for %s — falling back to literal",
target_lang,
)
return {
"text": literal_text,
"literal": literal_text,
"critique": critique,
"error": f"adapt-wrong-script:{target_lang}",
}
# Refuse adaptations that diverged from the line they were rewriting:
# wrong script (e.g. a local LLM rewrote a Devanagari line in
# Latin/German), runaway length (hallucinated dialogue, refusals,
# commentary — the script check alone passes ANY text for Latin-script
# targets), or the critique echoed back as the "adaptation". Caller still
# gets the critique so the UI can show what happened, but the live text
# falls back to the literal translation rather than corrupting the dub.
if final is not literal_text:
ok, reason = refine_output_ok(literal_text, final, target_lang, critique=critique)
if not ok:
logger.warning(
"cinematic adapt diverged for %s (%s) — falling back to literal",
target_lang, reason,
)
wrong_script = (reason or "").startswith("wrong-script")
return {
"text": literal_text,
"literal": literal_text,
"critique": critique,
"error": (f"adapt-wrong-script:{target_lang}" if wrong_script
else "adapt-diverged"),
}
return {
"text": final,
"literal": literal_text,
+7 -1
View File
@@ -1280,7 +1280,13 @@ class _LazyRegistry(dict):
# effect on every list_backends() call — we keep iteration light
# and let the caller's __getitem__ trigger the import.
seen: set[str] = set()
for k in dict.__iter__(self):
# Snapshot the live keys before yielding. A concurrent thread's lazy
# __getitem__ inserts into self (self[key] = cls), and list_backends()
# runs in a FastAPI threadpool — so holding a *live* dict iterator open
# across the per-engine is_available() probes would raise
# "dictionary changed size during iteration". list() consumes the
# iterator atomically under the GIL, closing that window.
for k in list(dict.__iter__(self)):
seen.add(k)
yield k
for k in _LAZY_REGISTRY:
+1 -1
View File
@@ -16,7 +16,7 @@
},
"frontend": {
"name": "omnivoice-studio",
"version": "0.3.9",
"version": "0.3.10",
"dependencies": {
"@fontsource-variable/inter": "^5.2.8",
"@fontsource-variable/source-serif-4": "^5.2.9",
Binary file not shown.

Before

Width:  |  Height:  |  Size: 187 KiB

After

Width:  |  Height:  |  Size: 331 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 134 KiB

After

Width:  |  Height:  |  Size: 420 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 342 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 106 KiB

After

Width:  |  Height:  |  Size: 420 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 358 KiB

After

Width:  |  Height:  |  Size: 426 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 278 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 295 KiB

After

Width:  |  Height:  |  Size: 347 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 183 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 666 KiB

+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "omnivoice-studio",
"version": "0.3.9",
"version": "0.3.10",
"private": true,
"license": "AGPL-3.0-only",
"type": "module",
+1 -1
View File
@@ -2941,7 +2941,7 @@ dependencies = [
[[package]]
name = "omnivoice-studio"
version = "0.3.9"
version = "0.3.10"
dependencies = [
"arboard",
"dirs-next",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "omnivoice-studio"
version = "0.3.9"
version = "0.3.10"
description = "OmniVoice Studio AI voice cloning & dubbing desktop app"
authors = ["Debpalash"]
license = "AGPL-3.0-only"
+75 -1
View File
@@ -34,11 +34,55 @@ pub fn port_in_use(port: u16) -> bool {
pub fn backend_healthy(port: u16) -> bool {
let url = format!("http://127.0.0.1:{}/system/info", port);
match ureq_get_with_timeout(&url, Duration::from_millis(500)) {
Ok(body) => body.contains("\"model_checkpoint\"") || body.contains("\"data_dir\""),
Ok(body) => is_omnivoice_body(&body),
Err(_) => false,
}
}
fn is_omnivoice_body(body: &str) -> bool {
body.contains("\"model_checkpoint\"") || body.contains("\"data_dir\"")
}
/// The `app_version` reported by the OmniVoice backend at :port.
/// `None` when nothing OmniVoice answers there (port free, or a foreign
/// process). `Some("")` when it IS our backend but predates the
/// `app_version` field — callers treat that as stale.
pub fn running_backend_version(port: u16) -> Option<String> {
let url = format!("http://127.0.0.1:{}/system/info", port);
let body = ureq_get_with_timeout(&url, Duration::from_millis(500)).ok()?;
if !is_omnivoice_body(&body) {
return None;
}
Some(parse_app_version(&body).unwrap_or_default())
}
/// Extract `"app_version": "X"` from a /system/info body. String-sniff on one
/// field (consistent with `backend_healthy`) — no JSON dependency needed.
fn parse_app_version(body: &str) -> Option<String> {
let key = "\"app_version\"";
let rest = &body[body.find(key)? + key.len()..];
let rest = rest[rest.find(':')? + 1..].trim_start();
let rest = rest.strip_prefix('"')?;
Some(rest[..rest.find('"')?].to_string())
}
/// Whether a running backend's version matches THIS app build, comparing
/// **base** versions (any `-N` pre-release suffix stripped from both sides) so
/// a preview build `0.3.10-4` still attaches to its `0.3.10` backend.
///
/// Why this exists (the "bound port blocked the newer version" report): an
/// orphaned backend from a *previous* version keeps answering health checks
/// after an update, so "healthy" alone made the new UI silently attach to old
/// backend code — every fix in the update appeared to change nothing. A
/// version-mismatched (or unversioned) OmniVoice responder is stale by
/// definition; callers kill it and spawn the bundled backend instead.
pub fn same_app_version(running: &str) -> bool {
fn base(v: &str) -> &str {
v.split('-').next().unwrap_or(v).trim()
}
!running.is_empty() && base(running) == base(env!("CARGO_PKG_VERSION"))
}
fn ureq_get_with_timeout(url: &str, timeout: Duration) -> Result<String, String> {
let url = url.strip_prefix("http://").ok_or("only http:// supported")?;
let (host_port, path) = match url.find('/') {
@@ -365,4 +409,34 @@ mod tests {
assert!(diag.contains("Interpreter present on disk: false"));
assert!(diag.contains("Clean & Retry"), "must give an actionable hint");
}
// ── stale-backend detection (the "bound port blocked the newer version"
// report: a healthy orphan from a previous version must NOT be
// attached to) ─────────────────────────────────────────────────────
#[test]
fn parse_app_version_reads_system_info_shape() {
let body = r#"{"app_version":"0.3.9","data_dir":"/x","model_checkpoint":"k2"}"#;
assert_eq!(parse_app_version(body).as_deref(), Some("0.3.9"));
// whitespace after the colon is fine
assert_eq!(
parse_app_version(r#"{ "app_version" : "1.2.3" }"#).as_deref(),
Some("1.2.3")
);
// pre-app_version backends and foreign bodies yield None
assert_eq!(parse_app_version(r#"{"data_dir":"/x"}"#), None);
assert_eq!(parse_app_version("<html>not json</html>"), None);
}
#[test]
fn same_app_version_matches_current_build_and_rejects_stale() {
let ours = env!("CARGO_PKG_VERSION");
assert!(same_app_version(ours), "own version must attach");
// preview stamp of the same base still attaches
assert!(same_app_version(&format!("{}-7", ours)));
// a different (older) release is stale
assert!(!same_app_version("0.0.1"));
// unversioned (pre-app_version backend) is stale by definition
assert!(!same_app_version(""));
}
}
+23 -4
View File
@@ -145,10 +145,29 @@ pub fn retry_bootstrap(app: tauri::AppHandle, state: tauri::State<'_, BootstrapS
set_stage(&stage_handle, BootstrapStage::Ready);
return;
}
if crate::backend::backend_healthy(backend_port()) {
log::info!("Port {} already serving OmniVoice backend — attaching", backend_port());
set_stage(&stage_handle, BootstrapStage::Ready);
return;
match crate::backend::running_backend_version(backend_port()) {
Some(v) if crate::backend::same_app_version(&v) => {
log::info!(
"Port {} already serving OmniVoice backend v{} — attaching",
backend_port(), v
);
set_stage(&stage_handle, BootstrapStage::Ready);
return;
}
Some(v) => {
// A healthy-but-stale backend from a previous version (the
// classic post-update orphan). Attaching would silently run
// OLD backend code under the new UI — replace it instead.
log::warn!(
"Port {} serves a stale OmniVoice backend (v{} != app v{}) — replacing it",
backend_port(),
if v.is_empty() { "<unknown>" } else { v.as_str() },
env!("CARGO_PKG_VERSION"),
);
crate::backend::kill_orphan_on_port(backend_port());
std::thread::sleep(Duration::from_millis(500));
}
None => {}
}
if crate::backend::port_in_use(backend_port()) {
log::warn!("Port {} in use — taking ownership", backend_port());
+24 -7
View File
@@ -657,13 +657,30 @@ pub fn run() {
set_stage(&stage_handle, BootstrapStage::AwaitingSetup);
return;
}
if backend::backend_healthy(backend_port()) {
log::info!(
"Port {} already serving OmniVoice backend — attaching",
backend_port()
);
set_stage(&stage_handle, BootstrapStage::Ready);
return;
match backend::running_backend_version(backend_port()) {
Some(v) if backend::same_app_version(&v) => {
log::info!(
"Port {} already serving OmniVoice backend v{} — attaching",
backend_port(), v
);
set_stage(&stage_handle, BootstrapStage::Ready);
return;
}
Some(v) => {
// Healthy-but-stale backend from a previous version —
// the post-update orphan that made new installs run
// old backend code. Replace it (see backend.rs
// same_app_version for the full story).
log::warn!(
"Port {} serves a stale OmniVoice backend (v{} != app v{}) — replacing it",
backend_port(),
if v.is_empty() { "<unknown>" } else { v.as_str() },
env!("CARGO_PKG_VERSION"),
);
backend::kill_orphan_on_port(backend_port());
std::thread::sleep(Duration::from_millis(500));
}
None => {}
}
if backend::port_in_use(backend_port()) {
log::warn!(
+21 -8
View File
@@ -27,11 +27,15 @@ const fmt = (t) => {
* SegmentTrack custom DOM segment editor lane for the dub timeline (#280).
*
* Replaces the WaveSurfer Regions plugin in the editing path. Renders one
* absolutely-positioned box per segment inside a lane whose horizontal
* position is derived from a single {pxPerSec, scrollLeft} source (read off
* WaveSurfer's wrapper by the parent), so boxes stay pixel-aligned with the
* waveform across zoom/scroll/resize. Virtualized by TIME only the boxes
* inside the visible window (+ buffer) are mounted.
* absolutely-positioned box per segment, each placed in PURE LAYOUT from a
* single {pxPerSec, scrollLeft} source (read off WaveSurfer's wrapper by the
* parent), so boxes stay pixel-aligned with the waveform across
* zoom/scroll/resize. The lane itself must NEVER be transform-animated: a
* lane translateX'd on every playback tick gets promoted to a compositor
* layer by Chromium, and composited semi-transparent paints flash
* invisible/visible on some Windows GPU/WebView2 drivers (#373; #381 only
* dampened it). Virtualized by TIME only the boxes inside the visible
* window (+ buffer) are mounted.
*
* Props:
* segments sorted-by-start segment array (store shape)
@@ -465,6 +469,13 @@ export default function SegmentTrack({
const innerWidth = Math.max(viewWidth, Math.ceil(duration * pxPerSec));
const playheadX = currentTime * pxPerSec - effScroll;
const windowed = effSegments.slice(lo, hi);
// Scroll offset baked into each box's `left` (viewport coordinates) instead
// of a `translateX` on the lane an animated lane transform is composited
// by Chromium and flashes on some Windows GPU/WebView2 drivers (#373). In
// the selfScroll fallback the viewport is a real scroll container, so boxes
// stay in lane coordinates (offset 0). The virtualization window above
// derives from the same effScroll, so both stay consistent by construction.
const laneShift = selfScroll ? 0 : effScroll;
return (
<div
@@ -489,14 +500,16 @@ export default function SegmentTrack({
aria-orientation="horizontal"
title={t('timeline.keyboard_hint')}
style={{
width: innerWidth,
transform: selfScroll ? undefined : `translateX(${-effScroll}px)`,
// selfScroll needs the full content width for the native
// scrollbar range; in synced mode the lane is a static
// viewport-sized strip (NO transform see laneShift above).
width: selfScroll ? innerWidth : '100%',
}}
>
{windowed.map((s) => {
const sid = String(s.id);
const idx = indexById.get(sid) ?? 0;
const left = s.start * pxPerSec;
const left = s.start * pxPerSec - laneShift;
const width = Math.max(2, (s.end - s.start) * pxPerSec);
const isSel = selectedId != null && String(selectedId) === sid;
const isFocus = focusId === sid;
@@ -171,6 +171,46 @@ describe('SegmentTrack — keyboard', () => {
});
});
describe('SegmentTrack — compositor-safe positioning (#373)', () => {
const baseProps = {
segments: SEGS,
pxPerSec: 100,
duration: 10,
currentTime: 0,
onsets: [],
};
it('the lane never carries a transform — at rest and after a scroll update', () => {
// Regression guard for #373: an animated `translateX` on the lane gets
// composited by Chromium during playback and its boxes flash on some
// Windows GPU/WebView2 drivers. Any reintroduction must fail here.
const { rerender } = render(<SegmentTrack {...baseProps} scrollLeft={0} />);
expect(screen.getByRole('listbox').style.transform).toBe('');
// Simulate a WaveSurfer autoscroll tick during playback.
rerender(<SegmentTrack {...baseProps} scrollLeft={250} />);
expect(screen.getByRole('listbox').style.transform).toBe('');
});
it('boxes are laid out in viewport coordinates: left = start·pxPerSec scrollLeft', () => {
setup({ scrollLeft: 250 });
expect(box(0).style.left).toBe('-250px'); // start 0
expect(box(1).style.left).toBe('50px'); // start 3 300 250
expect(box(2).style.left).toBe('250px'); // start 5 500 250
});
it('selfScroll fallback keeps lane coordinates — viewport scroll must not double-shift boxes', () => {
setup({ selfScroll: true });
const lane = screen.getByRole('listbox');
const viewport = lane.parentElement;
// jsdom has no layout: stub the scroll position, then fire the event the
// component listens to.
Object.defineProperty(viewport, 'scrollLeft', { value: 120, configurable: true });
fireEvent.scroll(viewport);
expect(lane.style.transform).toBe('');
expect(box(1).style.left).toBe('300px'); // lane coords: 3s · 100px/s
});
});
describe('SegmentTrack — pointer + selection', () => {
it('pointerdown selects the segment (table sync)', () => {
const { onSelectSeg } = setup();
+4 -20
View File
@@ -30,6 +30,7 @@ import WaveformPlayer from './WaveformPlayer';
import { useAppStore } from '../store';
import { useTranslation } from 'react-i18next';
import { askConfirm } from '../utils/dialog';
import { absoluteTime, timeAgo } from '../utils/relativeTime';
const SIDEBAR_TABS = [
{ id: 'projects', icon: FolderOpen, accent: '#b8bb26' },
@@ -71,20 +72,6 @@ function LazyWaveformPlayer({ height = 36, className = '', ...rest }) {
return <div ref={holderRef} className={className} style={{ height }} aria-hidden="true" />;
}
function timeAgo(ms) {
const diff = Date.now() - ms;
if (!isFinite(diff) || diff < 0) return '';
const s = Math.floor(diff / 1000);
if (s < 60) return `${s}s ago`;
const m = Math.floor(s / 60);
if (m < 60) return `${m}m ago`;
const h = Math.floor(m / 60);
if (h < 24) return `${h}h ago`;
const d = Math.floor(h / 24);
if (d < 7) return `${d}d ago`;
return new Date(ms).toLocaleDateString([], { month: 'short', day: 'numeric' });
}
export default function Sidebar(props) {
const {
availableTabs = ['projects', 'history', 'downloads'],
@@ -311,11 +298,8 @@ export default function Sidebar(props) {
<span className="history-kind history-kind--audio">
<Film size={9} /> {t('sidebar.dub_label')}
</span>
<span
className="history-meta"
title={new Date(proj.updated_at * 1000).toLocaleString()}
>
{timeAgo(proj.updated_at * 1000)}
<span className="history-meta" title={absoluteTime(proj.updated_at)}>
{timeAgo(proj.updated_at)}
</span>
</div>
<div className="history-title">{proj.name}</div>
@@ -772,7 +756,7 @@ export default function Sidebar(props) {
>
<KindIcon size={9} /> {item.mode}
</span>
<span className="history-meta">{timeAgo(item.created_at * 1000)}</span>
<span className="history-meta">{timeAgo(item.created_at)}</span>
</div>
<div className="history-title">{item.filename}</div>
<div className="history-subtitle">
@@ -2,6 +2,7 @@ import React, { useEffect, useMemo, useState } from 'react';
import { useTranslation } from 'react-i18next';
import { Mic, Search, Clock, Languages } from 'lucide-react';
import { Dialog, Input } from '../ui';
import { toMillis } from '../utils/relativeTime';
import { loadTranscriptions, TRANSCRIPTION_EVENT } from '../utils/transcriptionsStore';
/**
@@ -27,10 +28,12 @@ export default function TranscriptionPicker({ open, onClose, onPick }) {
}, [open]);
// Relative time, host-locale absolute fallback; null on unparseable timestamp.
// toMillis keeps this unit-safe (ISO strings today; seconds/ms tolerated).
const formatTime = (iso) => {
const d = new Date(iso);
if (Number.isNaN(d.getTime())) return null;
const diff = Date.now() - d.getTime();
const ms = toMillis(iso);
if (ms == null) return null;
const d = new Date(ms);
const diff = Date.now() - ms;
if (diff < 60000) return t('transcriptions.just_now');
if (diff < 3600000) return t('transcriptions.m_ago', { count: Math.floor(diff / 60000) });
if (diff < 86400000) return t('transcriptions.h_ago', { count: Math.floor(diff / 3600000) });
+3 -20
View File
@@ -10,21 +10,7 @@ import React, { useMemo, useState } from 'react';
import { useTranslation } from 'react-i18next';
import { Search, Film, FolderOpen, Trash2, Save, Pencil, Check, X } from 'lucide-react';
import { Button } from '../ui';
// Local copy of the sidebar's relative-time formatter (small, self-contained).
function timeAgo(ms) {
const diff = Date.now() - ms;
if (!isFinite(diff) || diff < 0) return '';
const s = Math.floor(diff / 1000);
if (s < 60) return `${s}s ago`;
const m = Math.floor(s / 60);
if (m < 60) return `${m}m ago`;
const h = Math.floor(m / 60);
if (h < 24) return `${h}h ago`;
const d = Math.floor(h / 24);
if (d < 7) return `${d}d ago`;
return new Date(ms).toLocaleDateString([], { month: 'short', day: 'numeric' });
}
import { absoluteTime, timeAgo } from '../utils/relativeTime';
export default function WorkspaceProjects({
projects = [],
@@ -105,11 +91,8 @@ export default function WorkspaceProjects({
<span className="history-kind history-kind--audio">
<Film size={9} /> {t('sidebar.dub_label')}
</span>
<span
className="history-meta"
title={new Date(proj.updated_at * 1000).toLocaleString()}
>
{timeAgo(proj.updated_at * 1000)}
<span className="history-meta" title={absoluteTime(proj.updated_at)}>
{timeAgo(proj.updated_at)}
</span>
</div>
{editingId === proj.id ? (
+2
View File
@@ -370,6 +370,8 @@
"llmskills_open_providers": "Configure providers",
"llmskills_load_failed": "Failed to load LLM skills",
"llmskills_save_failed": "Failed to save",
"llmskills_dub_translation_name": "Dub translation",
"llmskills_dub_translation_desc": "Translates dub lines directly with your configured LLM provider (the Dub tab's 'LLM' engine). Off: the LLM engine is unavailable — pick another engine.",
"llmskills_cinematic_translation_name": "Cinematic & Autofit translation",
"llmskills_cinematic_translation_desc": "Rewrites each translated line for natural, in-character delivery (and Autofit's time budget). Off: dubs use the Fast translation result.",
"llmskills_slot_fitting_name": "Speech-rate slot fitting",
+3 -13
View File
@@ -26,6 +26,7 @@ import BatchAddDialog from '../components/BatchAddDialog';
import toast from 'react-hot-toast';
import { toastErrorWithReport } from '../utils/errorToast';
import { recordValueMoment } from '../utils/donationMoments';
import { absoluteTime, timeAgo } from '../utils/relativeTime';
/**
* BatchQueue UI for the /batch/* dubbing pipeline.
@@ -261,7 +262,7 @@ function JobCard({ job, onCancel, onDelete, t }) {
const st = STATUS_TONE[job.status] || STATUS_TONE.queued;
const StIcon = st.icon;
const ageLabel = formatAge((Date.now() / 1000 - (job.created_at || 0)) * 1000);
const ageLabel = timeAgo(job.created_at);
const duration =
job.finished_at && job.started_at ? Math.max(0, job.finished_at - job.started_at) : null;
@@ -287,7 +288,7 @@ function JobCard({ job, onCancel, onDelete, t }) {
<span className="batch-queue__card-spacer flex-1" />
<span
className="batch-queue__card-age text-[var(--text-xs)] text-fg-subtle [font-variant-numeric:tabular-nums]"
title={new Date((job.created_at || 0) * 1000).toLocaleString()}
title={absoluteTime(job.created_at)}
>
{ageLabel}
</span>
@@ -394,17 +395,6 @@ function JobCard({ job, onCancel, onDelete, t }) {
);
}
function formatAge(ms) {
if (!isFinite(ms) || ms < 0) return '—';
const s = Math.floor(ms / 1000);
if (s < 60) return `${s}s ago`;
const m = Math.floor(s / 60);
if (m < 60) return `${m}m ago`;
const h = Math.floor(m / 60);
if (h < 24) return `${h}h ago`;
return new Date(Date.now() - ms).toLocaleDateString();
}
function formatDuration(secs) {
if (secs < 60) return `${secs.toFixed(1)}s`;
const m = Math.floor(secs / 60);
+16 -21
View File
@@ -18,6 +18,7 @@ import {
BookOpen,
} from 'lucide-react';
import { apiFetch } from '../api/client';
import { timeAgo, toMillis } from '../utils/relativeTime';
import { loadTranscriptions, TRANSCRIPTION_EVENT } from '../utils/transcriptionsStore';
import { audioUrl } from '../api/generate';
import { playBlobAudio } from '../utils/media';
@@ -41,18 +42,6 @@ import { playBlobAudio } from '../utils/media';
* this page stays in sync with the Sidebar and Launchpad automatically.
*/
function fmtTime(ts) {
if (!ts) return '';
const d = typeof ts === 'number' ? ts : Date.parse(ts);
if (!Number.isFinite(d)) return '';
const diff = Date.now() - d;
const s = Math.floor(diff / 1000);
if (s < 60) return `${s}s ago`;
if (s < 3600) return `${Math.floor(s / 60)}m ago`;
if (s < 86400) return `${Math.floor(s / 3600)}h ago`;
return `${Math.floor(s / 86400)}d ago`;
}
function fmtDuration(sec) {
if (!sec) return '';
const n = Number(sec);
@@ -177,7 +166,10 @@ export default function Projects({
}, []);
// Normalise every source into a common shape so the filter + search +
// sort pipeline is identical regardless of origin.
// sort pipeline is identical regardless of origin. Timestamps arrive in
// mixed units (backend rows: Unix seconds; story projects: Date.now() ms;
// transcriptions: ISO strings) toMillis() normalizes them all to epoch
// ms (or 0 when missing) so both the sort and timeAgo() stay unit-safe.
const items = useMemo(() => {
const list = [];
for (const p of studioProjects) {
@@ -186,7 +178,7 @@ export default function Projects({
id: p.id,
title: p.name || p.video_path?.split('/').pop() || p.id,
subtitle: fmtDuration(p.duration),
ts: (p.updated_at || p.created_at || 0) * 1000,
ts: toMillis(p.updated_at || p.created_at) ?? 0,
accent: '#fe8019',
Icon: Film,
onClick: () => onOpenDub?.(p.id),
@@ -205,7 +197,7 @@ export default function Projects({
]
.filter(Boolean)
.join(' · '),
ts: sp.updatedAt || 0,
ts: toMillis(sp.updatedAt) ?? 0,
accent: '#83a598',
Icon: BookOpen,
onClick: () => onOpenStory?.(sp.id),
@@ -218,7 +210,7 @@ export default function Projects({
id: pr.id,
title: pr.name || pr.id,
subtitle: kind === 'design' ? t('projects.designed_voice') : t('projects.cloned_voice'),
ts: (pr.updated_at || pr.created_at || 0) * 1000,
ts: toMillis(pr.updated_at || pr.created_at) ?? 0,
accent: kind === 'design' ? '#8ec07c' : '#d3869b',
Icon: kind === 'design' ? Wand2 : Fingerprint,
onClick: () => onOpenProfile?.(pr.id),
@@ -230,7 +222,10 @@ export default function Projects({
id: h.filename || h.id || String(Math.random()),
title: (h.text || h.prompt || h.filename || t('projects.generated_audio')).slice(0, 80),
subtitle: h.language || h.voice || '',
ts: h.timestamp || h.created_at || 0,
// #epoch-bug regression site: generation_history rows carry created_at
// in Unix SECONDS feeding them to a ms-based diff rendered every
// history card as "20617d ago" (1970) and sorted them last.
ts: toMillis(h.timestamp || h.created_at) ?? 0,
accent: '#f3a5b6',
Icon: Music,
onClick: undefined,
@@ -242,7 +237,7 @@ export default function Projects({
id: e.path || e.id,
title: e.path?.split('/').pop() || e.filename || t('projects.export'),
subtitle: e.mode || '',
ts: (e.created_at || 0) * 1000,
ts: toMillis(e.created_at) ?? 0,
accent: '#fabd2f',
Icon: Download,
onClick: () => e.path && onRevealExport?.(e.path),
@@ -261,7 +256,7 @@ export default function Projects({
]
.filter(Boolean)
.join(' · '),
ts: (j.created_at || 0) * 1000,
ts: toMillis(j.created_at) ?? 0,
accent: '#d3869b',
Icon: BookMarked,
onClick: () => j.output && playRenderInApp(audioUrl(j.output)),
@@ -275,7 +270,7 @@ export default function Projects({
subtitle: [tr.language, tr.duration_s ? `${Math.round(tr.duration_s)}s` : '']
.filter(Boolean)
.join(' · '),
ts: tr.timestamp ? Date.parse(tr.timestamp) : 0,
ts: toMillis(tr.timestamp) ?? 0,
accent: '#83a598',
Icon: FileText,
onClick: () => {
@@ -410,7 +405,7 @@ export default function Projects({
trailing={
<span className="inline-flex items-center gap-[3px] [font-family:var(--chrome-font-mono)]">
<Clock size={10} />
{fmtTime(it.ts)}
{timeAgo(it.ts)}
</span>
}
onClick={it.onClick}
+7 -3
View File
@@ -12,6 +12,7 @@ import { useTranslation } from 'react-i18next';
import { Mic, Copy, Trash2, Search, Clock, Languages, FileText, Download } from 'lucide-react';
import { Button } from '../ui';
import { toast } from 'react-hot-toast';
import { toMillis } from '../utils/relativeTime';
import {
loadTranscriptions,
TRANSCRIPTIONS_KEY,
@@ -108,10 +109,13 @@ export default function TranscriptionsPage() {
toast.success(t('transcriptions.exported'));
}, [transcriptions]);
// toMillis keeps this unit-safe (ISO strings today; seconds/ms tolerated)
// and guards unparseable stamps, which used to render "Invalid Date".
const formatTime = (iso) => {
const d = new Date(iso);
const now = new Date();
const diff = now - d;
const ms = toMillis(iso);
if (ms == null) return null;
const d = new Date(ms);
const diff = Date.now() - ms;
if (diff < 60000) return t('transcriptions.just_now');
if (diff < 3600000) return t('transcriptions.m_ago', { count: Math.floor(diff / 60000) });
if (diff < 86400000) return t('transcriptions.h_ago', { count: Math.floor(diff / 3600000) });
@@ -0,0 +1,64 @@
import React from 'react';
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { render, screen } from '@testing-library/react';
// Regression guard for the "20617d ago" epoch bug: /history rows carry
// created_at as Unix SECONDS (backend time.time()). Projects/OmniDrive fed
// them to a ms-based diff, so every generation-history card rendered as
// ~1970 ("20617d ago") and sorted to the bottom. The fix funnels every
// timestamp through utils/relativeTime.toMillis().
vi.mock('../utils/media', () => ({ playBlobAudio: vi.fn() }));
vi.mock('../api/generate', () => ({ audioUrl: (f) => `http://test.local/audio/${f}` }));
vi.mock('../api/client', () => ({
apiFetch: vi.fn(async () => ({ json: async () => ({ jobs: [] }) })),
}));
import Projects from '../pages/Projects';
describe('Projects — relative timestamps (seconds-vs-ms epoch class)', () => {
beforeEach(() => {
localStorage.clear();
});
afterEach(() => {
vi.restoreAllMocks();
});
it('renders a seconds created_at from today as hours ago, not "20617d ago"', async () => {
const history = [
{
id: 'h1',
filename: 'gen1.wav',
text: 'Hello from today',
created_at: Date.now() / 1000 - 7200, // 2h ago, in Unix SECONDS
},
];
render(<Projects history={history} />);
expect(await screen.findByText('2h ago')).toBeInTheDocument();
// The literal pre-fix rendering: tens of thousands of days.
expect(screen.queryByText(/\d{3,}d ago/)).toBeNull();
});
it('renders a dash for records with a missing timestamp', async () => {
const history = [{ id: 'h2', filename: 'gen2.wav', text: 'No stamp', created_at: null }];
render(<Projects history={history} />);
expect(await screen.findByText('No stamp')).toBeInTheDocument();
expect(screen.getByText('—')).toBeInTheDocument();
});
it('sorts seconds-stamped records among ms-stamped ones by real recency', async () => {
const nowS = Date.now() / 1000;
render(
<Projects
history={[{ id: 'h3', filename: 'g3.wav', text: 'Newest gen', created_at: nowS - 60 }]}
storyProjects={[{ id: 's1', name: 'Old story', updatedAt: Date.now() - 3 * 86400e3 }]}
/>,
);
const titles = (await screen.findAllByText(/Newest gen|Old story/)).map((el) => el.textContent);
// Pre-fix, the seconds stamp sorted as ~0 and sank below the story.
expect(titles).toEqual(['Newest gen', 'Old story']);
});
});
+88
View File
@@ -0,0 +1,88 @@
/**
* Shared, unit-tolerant timestamp normalization + relative-time formatting.
*
* Root cause of the "20617d ago" bug class: the backend stores timestamps as
* Unix SECONDS (`time.time()` REAL columns: generation_history.created_at,
* dub_history, exports, longform jobs, projects), while frontend-local
* records carry MILLISECONDS (`Date.now()` story projects) or ISO strings
* (transcriptions). Any formatter that assumes one unit renders the other as
* ~1970 ("20617d ago") or an epoch date ("Jan 1, 1970"). Every timestamp
* render must funnel through toMillis() so no individual view can regress.
*
* Do NOT change the backend's stored format existing user DBs hold seconds.
*/
// Numeric timestamps below this are seconds, at/above it milliseconds.
// 1e12 ms = Sep 2001; 1e12 s = year 33658 — unambiguous for real data.
const MS_THRESHOLD = 1e12;
const SHORT_DATE_OPTS = { month: 'short', day: 'numeric' };
/**
* Normalize any timestamp shape to epoch milliseconds, or null when missing/
* unparseable. Accepts: Unix seconds (float), epoch ms, numeric strings of
* either, ISO/date strings, Date instances. 0 / null / undefined / '' are
* treated as "missing" (0 is this codebase's missing-timestamp sentinel
* never a real 1970 record).
*/
export function toMillis(ts) {
if (ts == null || ts === 0 || ts === '') return null;
if (ts instanceof Date) {
const t = ts.getTime();
return Number.isNaN(t) ? null : t;
}
if (typeof ts === 'string') {
const trimmed = ts.trim();
if (!trimmed) return null;
// Numeric strings ("1751600000", "1751600000000.5") are Unix stamps,
// not date strings — Date.parse would reject or misread them.
const asNum = Number(trimmed);
if (Number.isFinite(asNum)) return toMillis(asNum);
const parsed = Date.parse(trimmed);
return Number.isNaN(parsed) ? null : parsed;
}
if (typeof ts !== 'number' || !Number.isFinite(ts) || ts <= 0) return null;
return Math.round(ts < MS_THRESHOLD ? ts * 1000 : ts);
}
// Future stamps within this window are clock skew, not data corruption.
const CLOCK_SKEW_MS = 60_000;
/**
* Relative-time label for record timestamps (any unit see toMillis).
* - missing/unparseable "—" (never "20617d ago" / epoch dates)
* - future within 1 min (clock skew) "just now"
* - <60s "Ns ago"; <60m "Nm ago"; <24h "Nh ago"; <7d "Nd ago"
* - older (or far-future, i.e. bad clock) short absolute date
*/
export function timeAgo(ts) {
const ms = toMillis(ts);
if (ms == null) return '—';
const diff = Date.now() - ms;
if (diff < -CLOCK_SKEW_MS) return shortDate(ms);
if (diff < 0) return 'just now';
const s = Math.floor(diff / 1000);
if (s < 60) return `${s}s ago`;
const m = Math.floor(s / 60);
if (m < 60) return `${m}m ago`;
const h = Math.floor(m / 60);
if (h < 24) return `${h}h ago`;
const d = Math.floor(h / 24);
if (d < 7) return `${d}d ago`;
return shortDate(ms);
}
/** Absolute host-locale datetime for tooltips; '' when missing/unparseable. */
export function absoluteTime(ts) {
const ms = toMillis(ts);
return ms == null ? '' : new Date(ms).toLocaleString();
}
function shortDate(ms) {
const d = new Date(ms);
const opts =
d.getFullYear() === new Date().getFullYear()
? SHORT_DATE_OPTS
: { ...SHORT_DATE_OPTS, year: 'numeric' };
return d.toLocaleDateString([], opts);
}
+96
View File
@@ -0,0 +1,96 @@
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
import { toMillis, timeAgo, absoluteTime } from './relativeTime';
// Regression suite for the "20617d ago" bug class: backend rows store Unix
// SECONDS (time.time()); formatters that assumed epoch MILLISECONDS rendered
// every record as ~1970 ("20617d ago" ≈ 56.5 years) or an epoch date.
// Frozen "now": 2026-07-04T12:00:00Z.
const NOW_MS = Date.UTC(2026, 6, 4, 12, 0, 0);
const NOW_S = NOW_MS / 1000;
beforeEach(() => {
vi.useFakeTimers();
vi.setSystemTime(NOW_MS);
});
afterEach(() => {
vi.useRealTimers();
});
describe('toMillis', () => {
it('converts Unix seconds (backend time.time() floats) to ms', () => {
expect(toMillis(NOW_S - 7200)).toBe(NOW_MS - 7200 * 1000);
expect(toMillis(1751600000.123)).toBe(1751600000123);
});
it('passes epoch milliseconds through unchanged', () => {
expect(toMillis(NOW_MS)).toBe(NOW_MS);
expect(toMillis(1751600000123)).toBe(1751600000123);
});
it('parses ISO strings', () => {
expect(toMillis('2026-07-04T11:00:00.000Z')).toBe(NOW_MS - 3600 * 1000);
});
it('treats numeric strings as Unix stamps, not dates', () => {
expect(toMillis(String(NOW_S))).toBe(NOW_MS);
expect(toMillis(String(NOW_MS))).toBe(NOW_MS);
});
it('accepts Date instances', () => {
expect(toMillis(new Date(NOW_MS))).toBe(NOW_MS);
expect(toMillis(new Date('garbage'))).toBeNull();
});
it('returns null for missing/degenerate values', () => {
for (const v of [null, undefined, 0, '', ' ', NaN, Infinity, -5, 'not a date']) {
expect(toMillis(v)).toBeNull();
}
});
});
describe('timeAgo', () => {
it('EPOCH REGRESSION: a seconds timestamp from today never renders as thousands of days ago', () => {
// Pre-fix, 2h-ago-in-seconds fed to a ms diff → "20617d ago" (1970).
const label = timeAgo(NOW_S - 7200);
expect(label).toBe('2h ago');
expect(label).not.toMatch(/\d{3,}d ago/);
});
it('renders identical output for seconds and milliseconds inputs', () => {
expect(timeAgo(NOW_S - 90)).toBe('1m ago');
expect(timeAgo(NOW_MS - 90 * 1000)).toBe('1m ago');
expect(timeAgo(NOW_S - 3 * 86400)).toBe('3d ago');
expect(timeAgo(NOW_MS - 3 * 86400 * 1000)).toBe('3d ago');
});
it('parses ISO strings', () => {
expect(timeAgo('2026-07-04T11:59:30.000Z')).toBe('30s ago');
});
it('renders a dash for null/0/undefined — never an epoch age', () => {
expect(timeAgo(null)).toBe('—');
expect(timeAgo(0)).toBe('—');
expect(timeAgo(undefined)).toBe('—');
expect(timeAgo('')).toBe('—');
});
it('treats future stamps within clock skew (<1 min ahead) as "just now"', () => {
expect(timeAgo(NOW_S + 30)).toBe('just now');
expect(timeAgo(NOW_MS + 59 * 1000)).toBe('just now');
});
it('falls back to an absolute date beyond 7 days (and for far-future stamps)', () => {
expect(timeAgo(NOW_S - 30 * 86400)).not.toMatch(/ago/);
expect(timeAgo(NOW_S + 3600)).not.toMatch(/ago/);
});
});
describe('absoluteTime', () => {
it('formats any unit and is empty for missing values (no "Jan 1, 1970" tooltips)', () => {
expect(absoluteTime(NOW_S)).toBe(new Date(NOW_MS).toLocaleString());
expect(absoluteTime(NOW_MS)).toBe(new Date(NOW_MS).toLocaleString());
expect(absoluteTime(null)).toBe('');
expect(absoluteTime(0)).toBe('');
});
});
+19 -7
View File
@@ -19,14 +19,26 @@ const GRID_SNAP_MAX_PX_PER_SEC = 40;
// Segment box palette — was WaveformTimeline's region palette; lives here so
// both the track and any legend can share it without circular imports.
//
// FULLY OPAQUE by design (#373): these used to be `rgba(…, 0.45)` and relied
// on alpha compositing over the panel behind the track — and on some Windows
// GPU/WebView2 drivers, semi-transparent paints on the (formerly
// transform-animated) lane flashed invisible during playback. Each entry now
// pre-blends the same 45% tint against the surface behind the lane
// (`--chrome-bg`, the .studio-panel background) via color-mix, which computes
// the identical pixels (0.45·tint + 0.55·bg) with zero alpha — and stays
// theme-aware because the variable resolves per [data-theme]. Do NOT
// reintroduce alpha here; guarded by timeline.test.js.
const opaqueTint = (r, g, b) =>
`color-mix(in srgb, rgb(${r} ${g} ${b}) 45%, var(--chrome-bg, #0f1011))`;
export const REGION_COLORS = [
'rgba(211,134,155,0.45)',
'rgba(131,165,152,0.45)',
'rgba(184,187,38,0.45)',
'rgba(250,189,47,0.45)',
'rgba(142,192,124,0.45)',
'rgba(254,128,25,0.45)',
'rgba(104,157,106,0.45)',
opaqueTint(211, 134, 155),
opaqueTint(131, 165, 152),
opaqueTint(184, 187, 38),
opaqueTint(250, 189, 47),
opaqueTint(142, 192, 124),
opaqueTint(254, 128, 25),
opaqueTint(104, 157, 106),
];
/**
+27
View File
@@ -2,6 +2,7 @@ import { describe, it, expect } from 'vitest';
import {
MIN_SEG_DUR,
MAX_OVERLAP,
REGION_COLORS,
visibleSegmentRange,
snapTime,
snapCandidates,
@@ -225,3 +226,29 @@ describe('nearestOnset', () => {
expect(nearestOnset(1, [])).toBeNull();
});
});
describe('REGION_COLORS — opaque paint guard (#373)', () => {
// Semi-transparent box fills flash on some Windows GPU/WebView2 drivers
// when the lane gets composited. Every palette entry must be fully opaque:
// no alpha-carrying color syntax anywhere in the value.
it('no entry carries an alpha channel', () => {
expect(REGION_COLORS.length).toBeGreaterThan(0);
for (const color of REGION_COLORS) {
expect(color).not.toMatch(/rgba\(|hsla\(|transparent/i); // legacy alpha fns
expect(color).not.toMatch(/\/\s*(?:0?\.\d+|\d+%)/); // modern `… / alpha` syntax
for (const hex of color.match(/#[0-9a-fA-F]+/g) ?? []) {
expect([4, 7]).toContain(hex.length); // #rgb / #rrggbb only — no alpha digits
}
}
});
it('color-mix mixes only opaque inputs and preserves the original 45% tint ratio', () => {
for (const color of REGION_COLORS) {
const m = color.match(
/^color-mix\(in srgb, rgb\(\d+ \d+ \d+\) (\d+)%, var\(--chrome-bg, (#[0-9a-fA-F]{6})\)\)$/,
);
expect(m, `unexpected palette entry shape: ${color}`).not.toBeNull();
expect(m[1]).toBe('45'); // same visual weight the 0.45-alpha fills had
}
});
});
+7 -1
View File
@@ -115,6 +115,11 @@ def main(argv=None):
ap.add_argument("--target", required=True, help="Target language ISO code (e.g. de, ja, es).")
ap.add_argument("--voice", help="Voice profile ID to apply to every segment.")
ap.add_argument("--quality", choices=("fast", "cinematic"), default="fast")
ap.add_argument(
"--speakers", type=int, default=None, metavar="N",
help="Exact number of speakers in the source (1-20). Forwarded to "
"diarization so distinct voices don't blend; omit to auto-detect.",
)
ap.add_argument("--glossary", help="Path to a JSON glossary: [{source,target,note}]")
ap.add_argument("--api", default=os.environ.get("OMNIVOICE_API", "http://localhost:8000"))
args = ap.parse_args(argv)
@@ -155,7 +160,8 @@ def main(argv=None):
# 3. Transcribe (sync).
_log("→ transcribing…")
tx = _post(api, f"/dub/transcribe/{job_id}", {})
speakers_q = f"?num_speakers={args.speakers}" if args.speakers else ""
tx = _post(api, f"/dub/transcribe/{job_id}{speakers_q}", {})
segs = tx.get("segments", [])
_log(f"{len(segs)} segment(s), source={tx.get('source_lang')}")
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "omnivoice"
version = "0.3.9"
version = "0.3.10"
description = "OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models"
readme = "README.md"
# Free and open-source under the GNU Affero General Public License v3 (see
@@ -0,0 +1,38 @@
"""Regression: the lazy TTS/ASR registries must not raise "dictionary changed
size during iteration" when a lazy ``__getitem__`` inserts a resolved key while
another caller iterates the ``/engines`` 500 seen in production logs.
FastAPI runs ``list_backends()`` in a threadpool, so two concurrent ``/engines``
requests race: one iterates ``_REGISTRY.items()`` (which held a *live* dict
iterator open across the slow per-engine ``is_available()`` probes) while the
other materializes the lazy entry via ``__getitem__`` (``self[key] = cls``).
The insert then tripped the open iterator. ``__iter__`` now snapshots the live
keys up front (``list(dict.__iter__(self))``, atomic under the GIL), so a
concurrent insert can no longer trip the iteration.
The tests drive the exact crash site (``__iter__``) deterministically: begin
iterating, insert mid-iteration, then drain. Pre-fix this raises on the drain;
post-fix it completes.
"""
def _assert_iter_survives_concurrent_insert(reg):
it = iter(reg) # the generator items()/list_backends() drives
first = next(it) # first yield → the real-key snapshot is taken here
reg["zzz-concurrent-insert"] = object() # a concurrent lazy insert, mid-iteration
drained = [first, *it] # must NOT raise "dictionary changed size during iteration"
assert first in drained
def test_tts_lazy_registry_iter_survives_concurrent_insert():
from services.tts_backend import _LazyRegistry
reg = _LazyRegistry({"omnivoice": object(), "b": object(), "c": object()})
_assert_iter_survives_concurrent_insert(reg)
def test_asr_lazy_registry_iter_survives_concurrent_insert():
from services.asr_backend import _LazyASRRegistry
reg = _LazyASRRegistry({"whisperx": object(), "faster-whisper": object()})
_assert_iter_survives_concurrent_insert(reg)
+39
View File
@@ -152,6 +152,45 @@ def test_synthesize_empty_spans_is_silent():
assert dur == 0.0
def test_synthesize_chapter_2d_engine_output_with_pause():
# #897 regression: real engines return (1, samples) (TTSBackend contract)
# while inter-span pause silence was built as 1-D zeros — the final hard
# concat crashed a longform chapter render with
# "RuntimeError: Tensors must have same number of dimensions: got 1 and 2".
torch = pytest.importorskip("torch")
sr = 16000
def synth(text, voice_id, speed=None):
return torch.ones(1, 1000, dtype=torch.float32)
plan = parse_audiobook_script("First. [pause 1s] Second.", default_voice="v")
audio, dur = synthesize_chapter(plan.chapters[0].spans, synth, sr)
assert audio.shape == (1, 1000 + sr + 1000)
assert dur == pytest.approx((2000 + sr) / sr)
def test_synthesize_chapter_silence_rank_matches_engine_output(monkeypatch):
# The producer half of the #897 fix: pause silence is materialized with
# the rendered audio's rank, so the parts reaching the final concat are
# rank-homogeneous even without concatenate_audio_chunks' normalization.
torch = pytest.importorskip("torch")
import services.chunked_tts as ct
seen: list[list[int]] = []
real = ct.concatenate_audio_chunks
def spy(chunks, sr, crossfade_ms=50):
seen.append([c.dim() for c in chunks])
return real(chunks, sr, crossfade_ms=crossfade_ms)
monkeypatch.setattr(ct, "concatenate_audio_chunks", spy)
plan = parse_audiobook_script("First. [pause 500ms] Second.", default_voice="v")
synthesize_chapter(plan.chapters[0].spans,
lambda t, v, s=None: torch.ones(1, 100), 16000)
assert seen, "the stitcher never reached the concat"
assert all(d == 2 for dims in seen for d in dims)
def test_parse_applies_ssml_lite_prosody():
plan = parse_audiobook_script("[slow]hush[/slow] normal [spell]USA[/spell]")
spans = plan.chapters[0].spans
+77 -1
View File
@@ -14,10 +14,23 @@ import pytest
class _FakeTrimmer:
"""Minimal non-Off LLM stand-in that trims any line to a short fixed length
so the fit loop converges deterministically."""
def chat(self, *, system, user, timeout=None):
def chat(self, *, system, user, timeout=None, temperature=None):
return "A" * 14 # ratio 0.93 for slot 1.0s @ en 15 cps → inside [0.92, 1.0]
class _FakeChatter:
"""Non-Off LLM stand-in that always returns one fixed reply; counts calls."""
def __init__(self, reply: str):
self.reply = reply
self.calls = 0
self.last_temperature = None
def chat(self, *, system, user, timeout=None, temperature=None):
self.calls += 1
self.last_temperature = temperature
return self.reply
@pytest.fixture
def speech_rate(monkeypatch):
from services import speech_rate as _sr
@@ -53,3 +66,66 @@ def test_strict_within_tolerance_is_noop(speech_rate):
res = speech_rate.adjust_for_slot("A" * 15, slot_seconds=1.0, target_lang="en", strict=True)
assert res["attempts"] == 0
assert res["rate_ratio"] == pytest.approx(1.0, abs=0.01)
# ── Divergence guard (v0.3.9 field report: Autofit invented dialogue) ───────
def test_expand_hallucination_rejected_output_stays_input(monkeypatch):
"""The reported bug: a short line over a long slot invited the LLM to
fabricate a slot-filling wall of dialogue, and the accept step took ANY
non-empty reply (the best-tracker then favored the most-padded one). The
guard must discard it and keep the input text."""
from services import speech_rate as _sr
text = "B" * 24 # 1.6s @ en 15cps over a 10s slot → ratio 0.16
fake = _FakeChatter("A" * 145) # a "perfect" slot fill = ~6× the input line
monkeypatch.setattr(_sr, "get_active_llm_backend", lambda: fake)
res = _sr.adjust_for_slot(text, slot_seconds=10.0, target_lang="en", strict=True)
assert res["text"] == text # hallucination never adopted
assert res.get("error") == "fit-diverged"
assert fake.calls == _sr.MAX_ATTEMPTS # attempts burned, loop still bounded
def test_refusal_reply_rejected_keeps_input(monkeypatch):
"""A refusal/commentary reply must not replace a long line just because
its length happens to land near the slot budget."""
from services import speech_rate as _sr
text = "C" * 60 # 4s over a 1s slot → heavy trim ask
fake = _FakeChatter("Sorry, I cannot.") # 16 chars ≈ the 15-char slot budget
monkeypatch.setattr(_sr, "get_active_llm_backend", lambda: fake)
res = _sr.adjust_for_slot(text, slot_seconds=1.0, target_lang="en")
assert res["text"] == text
assert res.get("error") == "fit-diverged"
def test_legitimate_expansion_accepted(monkeypatch):
from services import speech_rate as _sr
text = "D" * 40 # ratio 0.67 over a 4s slot @ en
fake = _FakeChatter("E" * 57) # ~1.4× the line → ratio 0.95, honest fill
monkeypatch.setattr(_sr, "get_active_llm_backend", lambda: fake)
res = _sr.adjust_for_slot(text, slot_seconds=4.0, target_lang="en", strict=True)
assert res["text"] == "E" * 57
assert "error" not in res
def test_tiny_line_skips_llm_expansion(monkeypatch):
"""A line under 15% of its slot can never honestly fill it — the LLM must
not even be asked (it could only fabricate), and the line stays short."""
from services import speech_rate as _sr
fake = _FakeChatter("A" * 150)
monkeypatch.setattr(_sr, "get_active_llm_backend", lambda: fake)
res = _sr.adjust_for_slot("Hi.", slot_seconds=10.0, target_lang="en", strict=True)
assert res["text"] == "Hi."
assert res["attempts"] == 0
assert res.get("error") == "fit-skip-short"
assert fake.calls == 0
def test_fit_llm_call_pins_low_temperature(monkeypatch):
"""The fit pass pins temperature=0.2 like the Fast translate path — the
provider default of 1.0 is part of what made expansions drift."""
from services import speech_rate as _sr
fake = _FakeChatter("A" * 14)
monkeypatch.setattr(_sr, "get_active_llm_backend", lambda: fake)
_sr.adjust_for_slot("A" * 16, slot_seconds=1.0, target_lang="en", strict=True)
assert fake.last_temperature == 0.2
+46
View File
@@ -133,6 +133,52 @@ def test_single_and_empty_chunk_lists():
assert concatenate_audio_chunks([], sr).shape[-1] == 1 # silent guard
# ── Mixed-rank chunks (#897) ─────────────────────────────────────────────────
# Engines emit (1, samples) per the TTSBackend contract while silence buffers
# were bare (samples,) — torch.cat crashed longform chapter renders with
# "RuntimeError: Tensors must have same number of dimensions: got 1 and 2".
def test_mixed_rank_chunks_hard_cut():
out = concatenate_audio_chunks([torch.ones(1, 300), torch.zeros(200)],
24000, crossfade_ms=0)
assert out.shape == (1, 500)
def test_mixed_rank_chunks_hard_cut_1d_first():
out = concatenate_audio_chunks([torch.zeros(200), torch.ones(1, 300)],
24000, crossfade_ms=0)
assert out.shape == (1, 500)
def test_mixed_rank_chunks_crossfade():
sr = 1000
out = concatenate_audio_chunks([torch.ones(1, 500), torch.zeros(500)],
sr, crossfade_ms=100) # 100-sample overlap
assert out.shape == (1, 900)
def test_mono_chunk_broadcasts_to_stereo():
# Honest channel handling: a mono chunk follows the widest channel count.
out = concatenate_audio_chunks([torch.ones(2, 100), torch.zeros(1, 50)],
24000, crossfade_ms=0)
assert out.shape == (2, 150)
out = concatenate_audio_chunks([torch.ones(2, 100), torch.zeros(60)],
24000, crossfade_ms=0) # 1-D mono, too
assert out.shape == (2, 160)
def test_all_1d_output_stays_1d():
out = concatenate_audio_chunks([torch.ones(100), torch.ones(100)],
24000, crossfade_ms=0)
assert out.dim() == 1 and out.shape[-1] == 200
def test_all_2d_output_stays_2d():
out = concatenate_audio_chunks([torch.ones(1, 100), torch.ones(1, 100)],
24000, crossfade_ms=0)
assert out.shape == (1, 200)
# ── Endpoint integration (stubbed engine, pattern from test_generate_engine) ─
def _tts_mod():
+4 -1
View File
@@ -87,10 +87,13 @@ def test_cinematic_refine_sync_injects_dialect_into_prompts(monkeypatch):
from services import translator
captured_systems = []
# Distinct reflect/adapt replies — an adapt identical to the critique is
# (correctly) rejected by the divergence guard as a critique echo.
replies = iter(["usa voseo rioplatense", "vos sos muy listo"])
def fake_chat(client, *, system, user):
captured_systems.append(system)
return "vos sos muy listo"
return next(replies)
monkeypatch.setattr(translator, "_llm_client", lambda: object())
monkeypatch.setattr(translator, "_chat", fake_chat)
+132 -1
View File
@@ -326,12 +326,21 @@ async def test_argos_fast_stamps_rate_ratio(monkeypatch):
def _install_fake_openai(monkeypatch, *, content="hola mundo", raises=None):
"""Register a fake `openai.OpenAI` whose chat.completions.create returns
`content` (or raises `raises`). Accepts the max_retries kwarg the code adds."""
`content` (or raises `raises`). Accepts the max_retries kwarg the code adds.
Also sets TRANSLATE_API_KEY so provider="openai" requests deterministically
take the legacy env-fallback branch since the provider-wiring fix, a fully
unconfigured LLM engine 400s up front instead of reaching a client (the
skills-resolved path has its own tests below). Returns the recorded
chat.completions.create kwargs for call-shape assertions."""
import sys
import types
calls = []
class _Completions:
def create(self, **kw):
calls.append(kw)
if raises is not None:
raise raises
msg = type("M", (), {"content": content})
@@ -349,6 +358,8 @@ def _install_fake_openai(monkeypatch, *, content="hola mundo", raises=None):
mod = types.ModuleType("openai")
mod.OpenAI = _FakeClient
monkeypatch.setitem(sys.modules, "openai", mod)
monkeypatch.setenv("TRANSLATE_API_KEY", "sk-test-env")
return calls
# ── P1: the Autofit fit pass must be bounded by the cinematic wall-clock ─────
@@ -414,3 +425,123 @@ async def test_openai_segment_error_is_scrubbed(monkeypatch):
assert "sk-LEAKLEAKLEAKLEAKLEAK12345" not in seg["error"]
assert "/Users/bob" not in seg["error"]
assert "***REDACTED***" in seg["error"]
# ── provider="openai" resolves through the LLM Providers/Skills system ───────
# The engine used to read only the TRANSLATE_* env vars, so a provider the user
# configured + tested in Settings → LLM Providers silently didn't power it.
class _RecordingLLMClient:
"""Minimal OpenAI-compatible fake that records create() kwargs."""
def __init__(self, content):
self.calls = []
outer = self
class _Completions:
def create(self, **kw):
outer.calls.append(kw)
msg = type("M", (), {"content": content})
choice = type("C", (), {"message": msg})
return type("R", (), {"choices": [choice]})
self.chat = type("Chat", (), {"completions": _Completions()})()
@pytest.mark.asyncio
async def test_openai_uses_provider_configured_in_settings(monkeypatch):
"""A ready dub_translation skill (Settings → LLM Providers) powers the
engine its client, its model, its timeout with no env vars set."""
import types
from api.routers import dub_translate
from schemas.requests import TranslateRequest, TranslateSegment
from services import llm_skills
for var in ("TRANSLATE_API_KEY", "TRANSLATE_BASE_URL", "TRANSLATE_MODEL"):
monkeypatch.delenv(var, raising=False)
fake = _RecordingLLMClient("hallo welt")
handle = types.SimpleNamespace(
client=fake, model="provider-model", provider_id="groq", timeout=7.0)
monkeypatch.setattr(llm_skills, "resolve_skill_client", lambda sid: handle)
req = TranslateRequest(
segments=[TranslateSegment(id="s1", text="Hello")],
target_lang="de", provider="openai", source_lang="en",
)
resp = await dub_translate.dub_translate(req)
assert resp["translated"][0]["text"] == "hallo welt"
assert fake.calls, "the skills-resolved client was not used"
assert fake.calls[0]["model"] == "provider-model"
assert fake.calls[0]["timeout"] == 7.0
@pytest.mark.asyncio
async def test_openai_unconfigured_400_names_llm_providers(monkeypatch):
"""Nothing configured anywhere → an up-front actionable 400 pointing at
Settings LLM Providers, not a raw per-segment 401."""
import types
from api.routers import dub_translate
from schemas.requests import TranslateRequest, TranslateSegment
from services import llm_skills
for var in ("TRANSLATE_API_KEY", "TRANSLATE_BASE_URL", "TRANSLATE_MODEL"):
monkeypatch.delenv(var, raising=False)
monkeypatch.setattr(llm_skills, "resolve_skill_client", lambda sid: None)
monkeypatch.setattr(
llm_skills, "resolve_skill",
lambda sid: types.SimpleNamespace(reason="no_provider"))
req = TranslateRequest(
segments=[TranslateSegment(id="s1", text="Hello")],
target_lang="es", provider="openai", source_lang="en",
)
resp = await dub_translate.dub_translate(req)
assert resp.status_code == 400
assert b"LLM Providers" in resp.body
@pytest.mark.asyncio
async def test_openai_disabled_skill_400_names_llm_skills(monkeypatch):
"""A deliberately disabled dub_translation skill names the Skills page."""
import types
from api.routers import dub_translate
from schemas.requests import TranslateRequest, TranslateSegment
from services import llm_skills
for var in ("TRANSLATE_API_KEY", "TRANSLATE_BASE_URL", "TRANSLATE_MODEL"):
monkeypatch.delenv(var, raising=False)
monkeypatch.setattr(llm_skills, "resolve_skill_client", lambda sid: None)
monkeypatch.setattr(
llm_skills, "resolve_skill",
lambda sid: types.SimpleNamespace(reason="disabled"))
req = TranslateRequest(
segments=[TranslateSegment(id="s1", text="Hello")],
target_lang="es", provider="openai", source_lang="en",
)
resp = await dub_translate.dub_translate(req)
assert resp.status_code == 400
assert b"LLM Skills" in resp.body
@pytest.mark.asyncio
async def test_openai_env_fallback_still_works(monkeypatch):
"""Legacy env-only setups (no provider in the app) keep working unchanged,
including TRANSLATE_MODEL selection."""
from api.routers import dub_translate
from schemas.requests import TranslateRequest, TranslateSegment
from services import llm_skills
calls = _install_fake_openai(monkeypatch, content="hola mundo")
monkeypatch.setenv("TRANSLATE_MODEL", "env-model")
monkeypatch.setattr(llm_skills, "resolve_skill_client", lambda sid: None)
req = TranslateRequest(
segments=[TranslateSegment(id="s1", text="Hello")],
target_lang="es", provider="openai", source_lang="en",
)
resp = await dub_translate.dub_translate(req)
assert resp["translated"][0]["text"] == "hola mundo"
assert calls and calls[0]["model"] == "env-model"
+6 -4
View File
@@ -62,8 +62,8 @@ def _activate_groq(store):
def test_all_skills_cover_every_consumption_point(skills):
assert [s.id for s in skills.all_skills()] == [
"cinematic_translation", "slot_fitting", "glossary_extract",
"direction_parse", "dictation_refinement",
"dub_translation", "cinematic_translation", "slot_fitting",
"glossary_extract", "direction_parse", "dictation_refinement",
]
for s in skills.all_skills():
assert s.name_key == f"settings.llmskills_{s.id}_name"
@@ -255,9 +255,11 @@ def test_disabled_slot_fitting_returns_no_llm_marker(skills, store, monkeypatch)
class _Fake:
id = "openai-compat"
def chat(self, **kw):
return "short"
# A plausible trim (0.93 ratio, within the divergence guard's
# length window vs the 30-char input) so the enabled path converges.
return "x" * 14
monkeypatch.setattr(speech_rate, "get_active_llm_backend", lambda: _Fake())
long_text = "x" * 400 # far over any slot → forces the LLM branch
long_text = "x" * 30 # 2× over the slot → forces the LLM branch
assert "error" not in speech_rate.adjust_for_slot(
long_text, slot_seconds=1.0, target_lang="en")
skills.configure_skill("slot_fitting", enabled=False)
+48
View File
@@ -295,6 +295,54 @@ class TestSpeakerAssignment:
assert out[0]["speaker_id"] == out[1]["speaker_id"]
assert out[2]["speaker_id"] != out[1]["speaker_id"]
@staticmethod
def _gapped_segs(n, dur=2.0, gap=2.0):
"""n segments, each separated by a > SPEAKER_GAP silence."""
segs = []
t = 0.0
for i in range(n):
segs.append({"start": t, "end": t + dur, "text": f"s{i}", "id": str(i)})
t += dur + gap
return segs
def test_heuristic_hint_three_speakers_cycles_three_labels(self):
# Speaker-hint fix: num_speakers=3 must yield 3 distinct labels on
# alternating-gap audio. Pre-fix the heuristic hardcoded 2 speakers
# and silently ignored the hint.
out = assign_speakers_heuristic(self._gapped_segs(6), num_speakers=3)
labels = [s["speaker_id"] for s in out]
assert labels == [
"Speaker 1", "Speaker 2", "Speaker 3",
"Speaker 1", "Speaker 2", "Speaker 3",
]
assert len(set(labels)) == 3
def test_heuristic_hint_one_speaker_single_label(self):
out = assign_speakers_heuristic(self._gapped_segs(4), num_speakers=1)
assert {s["speaker_id"] for s in out} == {"Speaker 1"}
def test_heuristic_none_hint_preserves_legacy_two_speaker_alternation(self):
out = assign_speakers_heuristic(self._gapped_segs(4), num_speakers=None)
labels = [s["speaker_id"] for s in out]
assert labels == ["Speaker 1", "Speaker 2", "Speaker 1", "Speaker 2"]
@pytest.mark.parametrize("bad", [0, -3, "not-a-number"])
def test_heuristic_invalid_hint_falls_back_to_legacy(self, bad):
out = assign_speakers_heuristic(self._gapped_segs(4), num_speakers=bad)
labels = [s["speaker_id"] for s in out]
assert labels == ["Speaker 1", "Speaker 2", "Speaker 1", "Speaker 2"]
def test_heuristic_hint_no_gaps_keeps_one_speaker(self):
# N is an upper bound, not a quota: back-to-back speech with no
# > SPEAKER_GAP pause stays one speaker even with a hint of 3.
segs = [
{"start": 0.0, "end": 2.0, "text": "a", "id": "1"},
{"start": 2.1, "end": 4.0, "text": "b", "id": "2"},
{"start": 4.2, "end": 6.0, "text": "c", "id": "3"},
]
out = assign_speakers_heuristic(segs, num_speakers=3)
assert {s["speaker_id"] for s in out} == {"Speaker 1"}
def test_diarization_uses_overlap_weighted_assignment(self):
# Build a fake diarization with two overlapping turns for the same seg;
# the one with more overlap should win, not the one at midpoint.
+126
View File
@@ -0,0 +1,126 @@
"""Clone-purity guards in services.speaker_clone (speaker-hint fix).
A per-speaker auto-clone reference cut from mislabeled or boundary-adjacent
audio mixes two people's voices — the field-reported "made up" clone voices.
These tests pin the three guards:
* per-slice minimum duration (MIN_SLICE_DURATION_S),
* non-adjacency scoring preference (ADJACENT_TURN_GUARD_S) a preference,
never a hard filter,
* labels_source="heuristic" skips extraction entirely.
Pure tests over a synthetic vocals wav no model, no main import.
"""
import numpy as np
import pytest
import soundfile as sf
from services.speaker_clone import (
ADJACENT_TURN_GUARD_S,
MIN_REF_DURATION_S,
MIN_SLICE_DURATION_S,
_pick_reference_slices,
extract_speaker_clones,
)
SR = 16000
@pytest.fixture
def vocals(tmp_path):
# 60 s of non-silent audio so every segment slice has content.
path = tmp_path / "vocals.wav"
sf.write(str(path), np.float32(np.sin(np.linspace(0, 18000, 60 * SR))), SR)
return str(path)
def _seg(start, end, speaker="Speaker 1", text="hello there"):
return {"start": start, "end": end, "speaker_id": speaker, "text": text}
class TestPickReferenceSlices:
def test_rejects_slices_below_minimum_duration(self):
# Six 1.0 s fragments total 6 s (> MIN_REF_DURATION_S) — pre-fix they
# were all picked; now every one is under MIN_SLICE_DURATION_S so the
# speaker yields no reference at all (default voice beats a bad clone).
items = [(i, _seg(i * 3.0, i * 3.0 + 1.0)) for i in range(6)]
assert MIN_SLICE_DURATION_S > 1.0 # test premise
assert 6 * 1.0 > MIN_REF_DURATION_S # pre-fix these WOULD have passed
assert _pick_reference_slices(items) == []
def test_prefers_slice_not_adjacent_to_other_speaker(self):
# Two equal-length candidates for Speaker 1; the first is 0.1 s away
# from a Speaker 2 turn (< ADJACENT_TURN_GUARD_S), the second is far
# from everyone. The clean one must win the ranking.
adjacent = _seg(0.0, 8.0, "Speaker 1")
other = _seg(8.1, 10.0, "Speaker 2")
clean = _seg(20.0, 28.0, "Speaker 1")
all_segments = [adjacent, other, clean]
items = [(0, adjacent), (2, clean)]
chosen = _pick_reference_slices(
items, speaker_id="Speaker 1", all_segments=all_segments,
)
assert [seg for _, seg in chosen] == [clean]
def test_adjacency_is_a_preference_not_a_hard_filter(self):
# Dense dialogue: every Speaker 1 slice borders a Speaker 2 turn.
# Extraction must still succeed using the adjacent slices.
s1a = _seg(0.0, 6.0, "Speaker 1")
s2a = _seg(6.1, 8.0, "Speaker 2")
s1b = _seg(8.2, 12.0, "Speaker 1")
all_segments = [s1a, s2a, s1b]
items = [(0, s1a), (2, s1b)]
chosen = _pick_reference_slices(
items, speaker_id="Speaker 1", all_segments=all_segments,
)
assert chosen, "dense dialogue must still produce a reference"
def test_heuristic_labels_source_returns_nothing(self):
items = [(0, _seg(0.0, 8.0))]
assert _pick_reference_slices(items, labels_source="heuristic") == []
def test_legacy_call_without_kwargs_still_picks_long_slice(self):
# Backward compat: positional-only invocation (the pre-fix signature)
# keeps working and picks the long slice.
long_seg = _seg(0.0, 8.0)
chosen = _pick_reference_slices([(0, long_seg)])
assert [seg for _, seg in chosen] == [long_seg]
def test_overlapping_other_speaker_counts_as_adjacent(self):
# Negative gap (overlap) must also be flagged — that is the worst
# mixed-audio case of all.
overlapped = _seg(0.0, 8.0, "Speaker 1")
other = _seg(4.0, 6.0, "Speaker 2")
clean = _seg(20.0, 28.0, "Speaker 1")
chosen = _pick_reference_slices(
[(0, overlapped), (2, clean)],
speaker_id="Speaker 1",
all_segments=[overlapped, other, clean],
)
assert [seg for _, seg in chosen] == [clean]
class TestExtractSpeakerClones:
def test_heuristic_labels_source_skips_extraction(self, tmp_path, vocals):
segs = [_seg(0.0, 8.0), _seg(10.0, 18.0, "Speaker 2")]
out = extract_speaker_clones(
vocals, segs, str(tmp_path), labels_source="heuristic",
)
assert out == {}
@pytest.mark.parametrize("source", [None, "pyannote", "turns"])
def test_trusted_labels_still_extract(self, tmp_path, vocals, source):
# None (legacy caller, missing kwarg) and real diarization sources
# keep the current behavior: clones are produced.
segs = [_seg(0.0, 8.0), _seg(10.0, 18.0, "Speaker 2")]
kwargs = {} if source is None else {"labels_source": source}
out = extract_speaker_clones(vocals, segs, str(tmp_path), **kwargs)
assert set(out) == {"Speaker 1", "Speaker 2"}
for info in out.values():
assert info["duration"] >= MIN_REF_DURATION_S
def test_adjacency_guard_constant_sane(self):
# The guard must stay tighter than the heuristic's own gap threshold,
# or every real turn boundary would be flagged.
from services.segmentation import SPEAKER_GAP
assert 0 < ADJACENT_TURN_GUARD_S < SPEAKER_GAP
+306
View File
@@ -0,0 +1,306 @@
"""The dub speaker-count hint must be honored on EVERY diarization path.
Field report (v0.3.9): setting the dub "Speakers" count changed nothing the
`?num_speakers=` hint reached `_diarize()` and then died on 3 of its 4
branches (FunASR inline-turns shortcut, pyannote-unavailable heuristic
fallback, pyannote-crash heuristic fallback). Speakers blended and auto-clones
were cut from mixed-speaker audio.
These tests drive `dub_transcribe_stream`'s async generator directly (the
established pattern in test_dub_transcribe.py no TestClient, no GPU, no
pyannote) and pin:
* an explicit num_speakers routes a turns-capable job through pyannote
(with the hint) instead of the inline-turns shortcut;
* when a branch cannot honor the hint exactly, a `warning` SSE event says so
honestly instead of dropping it silently;
* heuristic labels skip auto voice-clone extraction (clone-purity guard)
with their own warning;
* the legacy POST /dub/transcribe/{job_id} endpoint exposes the same
(clamped) num_speakers parameter.
"""
from __future__ import annotations
import asyncio
import struct
import wave
from pathlib import Path
from unittest.mock import MagicMock
import pytest
from api.routers import dub_core as dc
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _make_wav(path: Path, seconds: float = 1.0, sr: int = 16000) -> None:
n = int(seconds * sr)
with wave.open(str(path), "wb") as wf:
wf.setnchannels(1)
wf.setsampwidth(2)
wf.setframerate(sr)
wf.writeframes(struct.pack(f"<{n}h", *([0] * n)))
class _FakeASR:
"""Minimal ASR backend; `turns` simulates FunASR-style inline diarization."""
id = "fake"
def __init__(self, turns=None):
self._turns = turns or []
def ensure_loaded(self):
pass
def transcribe(self, path, *, word_timestamps=True):
return {
"chunks": [
{"text": "Hello there my friend.", "timestamp": (0.0, 0.4)},
{"text": "I am doing quite well today.", "timestamp": (0.5, 0.9)},
],
"segments": self._turns,
"language": "en",
}
def unload(self):
pass
class _FakeTurn:
def __init__(self, start, end):
self.start = start
self.end = end
class _FakeDiar:
def itertracks(self, yield_label=True):
yield _FakeTurn(0.0, 0.45), None, "SPEAKER_00"
yield _FakeTurn(0.45, 1.0), None, "SPEAKER_01"
class _RecordingPipe:
"""Stands in for the pyannote pipeline; records the num_speakers kwarg."""
def __init__(self, crash=False):
self.calls: list = []
self._crash = crash
def __call__(self, path, num_speakers=None):
self.calls.append(num_speakers)
if self._crash:
raise RuntimeError("pyannote exploded: simulated")
return _FakeDiar()
def _wire_stream(tmp_path, monkeypatch, *, job_id, asr, diar_pipeline):
"""Common monkeypatching for driving the stream happy path end-to-end."""
audio = tmp_path / "a.wav"
_make_wav(audio, seconds=1.0)
dc._dub_jobs[job_id] = {
"audio_path": str(audio), "vocals_path": None, "scene_cuts": [],
}
fake_model = MagicMock()
fake_model._asr_pipe = MagicMock()
async def _ok_model():
return fake_model
monkeypatch.setattr(dc, "get_model", _ok_model)
monkeypatch.setattr(
"services.asr_backend.get_active_asr_backend", lambda *a, **k: asr,
)
monkeypatch.setattr(dc, "get_diarization_pipeline", diar_pipeline)
monkeypatch.setattr(dc, "offload_tts_for_asr", lambda *a, **k: None)
monkeypatch.setattr(dc, "restore_tts_after_asr", lambda *a, **k: None)
monkeypatch.setattr(dc, "_save_job", lambda *a, **k: None)
# Keep the no-token branch deterministic (never read this machine's HF creds).
monkeypatch.setattr("services.token_resolver.resolve", lambda *a, **k: None)
def _run_stream(job_id, num_speakers=None) -> str:
async def _collect():
resp = await dc.dub_transcribe_stream(job_id, num_speakers=num_speakers)
parts = []
async for chunk in resp.body_iterator:
parts.append(chunk.decode() if isinstance(chunk, (bytes, bytearray)) else str(chunk))
return "".join(parts)
try:
return asyncio.run(_collect())
finally:
dc._dub_jobs.pop(job_id, None)
_TURNS = [
{"start": 0.0, "end": 0.45, "speaker": "Speaker 1"},
{"start": 0.5, "end": 0.9, "speaker": "Speaker 2"},
]
_NO_PIPE = lambda return_error=False: (None, "NO_TOKEN") if return_error else None # noqa: E731
# ---------------------------------------------------------------------------
# Branch 1 — FunASR inline-turns shortcut must not eat the hint
# ---------------------------------------------------------------------------
def test_hint_routes_turns_job_through_pyannote(tmp_path, monkeypatch):
"""With inline ASR turns present AND pyannote loadable, an explicit
num_speakers must skip the turns shortcut and reach pyannote as a kwarg.
Pre-fix the shortcut returned early and the hint was never consulted."""
pipe = _RecordingPipe()
_wire_stream(
tmp_path, monkeypatch, job_id="t_hint_pyannote",
asr=_FakeASR(turns=_TURNS),
diar_pipeline=lambda return_error=False: (pipe, None) if return_error else pipe,
)
body = _run_stream("t_hint_pyannote", num_speakers=2)
assert pipe.calls == [2], f"pyannote must be called once with the hint: {pipe.calls}"
assert "event: final" in body, body
# The hint was fully honored — no hint warning may fire.
assert "Speaker-count hint ignored" not in body, body
def test_no_hint_keeps_turns_fast_path(tmp_path, monkeypatch):
"""Without a hint, the inline-turns shortcut stays the fast path: pyannote
is never invoked and no warning fires."""
pipe = _RecordingPipe()
_wire_stream(
tmp_path, monkeypatch, job_id="t_nohint_turns",
asr=_FakeASR(turns=_TURNS),
diar_pipeline=lambda return_error=False: (pipe, None) if return_error else pipe,
)
body = _run_stream("t_nohint_turns", num_speakers=None)
assert pipe.calls == [], "turns fast path must skip pyannote when no hint is set"
assert "event: final" in body, body
assert "event: warning" not in body, body
def test_hint_with_turns_but_no_pyannote_warns_hint_ignored(tmp_path, monkeypatch):
"""Turns present, hint set, pyannote unavailable: keep the turns (best
labels available) but tell the user the hint was ignored never silence."""
_wire_stream(
tmp_path, monkeypatch, job_id="t_hint_turns_warn",
asr=_FakeASR(turns=_TURNS), diar_pipeline=_NO_PIPE,
)
body = _run_stream("t_hint_turns_warn", num_speakers=3)
assert "event: warning" in body, body
assert "Speaker-count hint ignored" in body, body
assert "may differ from the 3 you set" in body, body
# Turns are trusted labels — the clone-purity guard must NOT fire.
assert "auto voice cloning skipped" not in body, body
# ---------------------------------------------------------------------------
# Branches 2+3 — heuristic fallbacks: hint threaded + honest warning + clone guard
# ---------------------------------------------------------------------------
def test_heuristic_fallback_warns_approximate_and_skips_clones(tmp_path, monkeypatch):
"""No turns, no pyannote, hint set: the heuristic cycles the requested
count (approximate honesty), the warning says exactly that, and auto
voice-clone extraction is skipped because gap-based labels are estimates."""
_wire_stream(
tmp_path, monkeypatch, job_id="t_hint_heur",
asr=_FakeASR(turns=[]), diar_pipeline=_NO_PIPE,
)
body = _run_stream("t_hint_heur", num_speakers=3)
assert "event: warning" in body, body
assert "only approximately honored" in body, body
assert "cycles 3 speaker labels" in body, body
# Clone-purity guard fires with its own warning.
assert dc.CLONE_SKIP_HEURISTIC_MSG in body, body
assert "event: final" in body, body
def test_heuristic_fallback_without_hint_keeps_legacy_warning(tmp_path, monkeypatch):
"""No hint → the pre-existing fallback warning is unchanged (no hint text),
but the clone-purity guard still protects against gap-based labels."""
_wire_stream(
tmp_path, monkeypatch, job_id="t_nohint_heur",
asr=_FakeASR(turns=[]), diar_pipeline=_NO_PIPE,
)
body = _run_stream("t_nohint_heur", num_speakers=None)
assert "event: warning" in body, body
assert "silence-gap" in body, body
assert "approximately honored" not in body, body
assert dc.CLONE_SKIP_HEURISTIC_MSG in body, body
def test_pyannote_crash_with_turns_falls_back_to_turns_not_heuristic(tmp_path, monkeypatch):
"""When the hint routes a turns-job through pyannote and pyannote crashes
mid-run, the inline turns are the fallback (better than the heuristic) and
the warning says the hint was ignored."""
pipe = _RecordingPipe(crash=True)
_wire_stream(
tmp_path, monkeypatch, job_id="t_crash_turns",
asr=_FakeASR(turns=_TURNS),
diar_pipeline=lambda return_error=False: (pipe, None) if return_error else pipe,
)
body = _run_stream("t_crash_turns", num_speakers=2)
assert pipe.calls == [2], pipe.calls
assert "crashed mid-run" in body, body
assert "built-in speaker turns" in body, body
assert "Speaker-count hint ignored" in body, body
# Turns labels → clones stay allowed.
assert "auto voice cloning skipped" not in body, body
# ---------------------------------------------------------------------------
# Shared clamp + legacy endpoint parity
# ---------------------------------------------------------------------------
class TestClampNumSpeakers:
@pytest.mark.parametrize("raw,expected", [
(None, None), (1, 1), (5, 5), (20, 20),
(0, None), (-2, None), (21, None), ("7", 7), ("junk", None),
])
def test_clamp(self, raw, expected):
assert dc._clamp_num_speakers(raw) == expected
def test_legacy_transcribe_endpoint_accepts_num_speakers():
"""POST /dub/transcribe/{job_id} (the CLI's endpoint) exposes the same
optional num_speakers query parameter as the SSE stream. Pre-fix the
legacy route had no way to express a speaker count at all."""
import inspect
from typing import Optional
sig = inspect.signature(dc.dub_transcribe)
assert "num_speakers" in sig.parameters, "legacy endpoint lost num_speakers"
p = sig.parameters["num_speakers"]
assert p.default is None, "num_speakers must be optional (auto-detect default)"
assert p.annotation == Optional[int]
# And the stream endpoint still has it too (parity in both directions).
stream_sig = inspect.signature(dc.dub_transcribe_stream)
assert "num_speakers" in stream_sig.parameters
def test_cli_exposes_speakers_flag(capsys):
"""omnivoice-dub --speakers N must parse and be forwarded as the
num_speakers query param on the legacy transcribe endpoint."""
import importlib.util
cli_path = Path(__file__).resolve().parents[1] / "omnivoice" / "cli" / "dub.py"
spec = importlib.util.spec_from_file_location("_omnivoice_cli_dub", cli_path)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
# Parse-level: --help lists the flag (argparse would reject an unknown one).
with pytest.raises(SystemExit):
mod.main(["--help"])
assert "--speakers" in capsys.readouterr().out
# Wiring-level: the flag is forwarded as the num_speakers query param.
src = cli_path.read_text()
assert "num_speakers={args.speakers}" in src
+95
View File
@@ -115,6 +115,101 @@ def test_cinematic_reflect_failure_returns_literal(monkeypatch):
assert "reflect" in res.get("error", "")
# ── Divergence guard (v0.3.9 field report: hallucinated dub lines) ─────────
def _mock_chain(monkeypatch, reflect: str, adapt: str):
"""Wire cinematic_refine_sync to a mocked 2-step REFLECT→ADAPT chain."""
responses = iter([reflect, adapt])
monkeypatch.setattr(tr, "_llm_client", lambda: MagicMock())
monkeypatch.setattr(tr, "_chat", lambda client, *, system, user: next(responses))
def test_cinematic_adapt_runaway_length_falls_back_to_literal(monkeypatch):
"""The reported bug: for a Latin-script target the script check passes ANY
text, so a hallucinated wall of dialogue used to ship as the dub line."""
literal = "¿Cómo estás hoy, amigo mío?"
_mock_chain(monkeypatch, "fine but a bit stiff", "Hola amigo. " * 30) # ~13× the literal
res = tr.cinematic_refine_sync(
"How are you doing today, my friend?", literal,
source_lang="en", target_lang="es",
)
assert res["text"] == literal
assert res.get("error") == "adapt-diverged"
assert res["critique"] == "fine but a bit stiff" # UI still sees what happened
def test_cinematic_adapt_critique_echo_rejected(monkeypatch):
"""ADAPT returning the critique itself must not become the dub line."""
literal = "¿Cómo estás hoy, amigo mío? Hace mucho que no te veo por aquí."
critique = (
"The literal translation reads stiff and does not fit the slot; "
"prefer a shorter, more idiomatic phrasing with warmer tone."
)
_mock_chain(monkeypatch, critique, critique) # adapt echoes the critique verbatim
res = tr.cinematic_refine_sync(
"How are you doing today, my friend? Long time no see.", literal,
source_lang="en", target_lang="es",
)
assert res["text"] == literal
assert res.get("error") == "adapt-diverged"
def test_cinematic_sane_adaptation_accepted(monkeypatch):
"""A faithful, idiomatic rewrite passes every guard untouched."""
literal = "¿Cómo estás hoy, amigo mío? Hace mucho que no te veo."
adapted = "¿Qué tal, amigo? ¡Cuánto tiempo sin verte!"
_mock_chain(monkeypatch, "a bit formal; contract it", adapted)
res = tr.cinematic_refine_sync(
"How are you doing today, my friend? Long time no see.", literal,
source_lang="en", target_lang="es",
)
assert res["text"] == adapted
assert "error" not in res
def test_cinematic_adapt_wrong_script_falls_back_to_literal(monkeypatch):
"""ADAPT output off the target script degrades to the literal with the
script-specific marker (the pre-existing fallback, previously untested)."""
literal = "नमस्ते मेरे दोस्त, आप कैसे हैं?"
_mock_chain(monkeypatch, "solid but wordy", "This is English, not Hindi, sorry.")
res = tr.cinematic_refine_sync(
"Hello my friend, how are you?", literal,
source_lang="en", target_lang="hi",
)
assert res["text"] == literal
assert res.get("error") == "adapt-wrong-script:hi"
def test_chat_pins_low_temperature(monkeypatch):
"""Cinematic reflect/adapt must pin temperature like the Fast path does —
the provider default of 1.0 is what let local models drift into invention."""
client = MagicMock()
client.chat.completions.create.return_value = MagicMock(
choices=[MagicMock(message=MagicMock(content="ok"))])
monkeypatch.setattr(tr, "_llm_model", lambda: "test-model")
assert tr._chat(client, system="s", user="u") == "ok"
assert client.chat.completions.create.call_args.kwargs["temperature"] == 0.2
def test_refine_guard_short_reference_uses_absolute_cap():
# A 2-word line legitimately triples — the ratio window must not apply.
ok, _ = tr.refine_output_ok("¡No!", "¡Claro que no, jamás!", "es")
assert ok
# …but a wall of text after a 2-word line is still divergence.
ok, reason = tr.refine_output_ok("¡No!", "x" * 200, "es")
assert not ok and reason.startswith("length-abs")
def test_refine_guard_ratio_env_override(monkeypatch):
literal = "¿Cómo estás hoy, amigo mío?"
ok, reason = tr.refine_output_ok(literal, literal * 4, "es")
assert not ok and reason.startswith("length-ratio")
monkeypatch.setenv("OMNIVOICE_REFINE_RATIO_MAX", "5.0")
ok, _ = tr.refine_output_ok(literal, literal * 4, "es")
assert ok # ceiling raised via env, mirroring the _cinematic_budget pattern
# ── Cinematic pass wall-clock budget (#stall follow-up) ────────────────────
def test_cinematic_budget_degrades_slow_segments_to_literal(monkeypatch):
Generated
+1 -1
View File
@@ -3207,7 +3207,7 @@ wheels = [
[[package]]
name = "omnivoice"
version = "0.3.9"
version = "0.3.10"
source = { editable = "." }
dependencies = [
{ name = "accelerate" },