Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fd7d20fe1e | ||
|
|
27ff5846c2 | ||
|
|
d14b37fab2 | ||
|
|
edf86c1800 | ||
|
|
437a995a0f | ||
|
|
03eb8f7d04 | ||
|
|
beca8cfea8 | ||
|
|
32103ad7b7 | ||
|
|
46c4445258 | ||
|
|
8bbab3fcc3 | ||
|
|
9ff5f38cb3 | ||
|
|
359c70bc21 | ||
|
|
0579ec91ad | ||
|
|
b6ec4e23f3 |
@@ -6,6 +6,21 @@ The format is loosely based on [Keep a Changelog](https://keepachangelog.com/).
|
||||
Versions track the desktop app (`tauri.conf.json` + `frontend/src-tauri/Cargo.toml`).
|
||||
The bundled TTS model package (`pyproject.toml`) is versioned independently.
|
||||
|
||||
## [0.3.10] — 2026-07-05
|
||||
|
||||
The listening release — nine fixes in twenty-four hours, almost all driven by your v0.3.9 field reports (several with same-day turnaround). The dubbing pipeline stops lying: **Cinematic and Autofit can no longer invent dialogue**, the **speaker count you set is honored on every path** (and auto-cloning stops fabricating voices from guessed labels), and the timeline stops flashing invisible on Windows. Audiobook chapters with pauses render again. And one fix everyone should want: **updating can no longer leave you secretly running the old version** — a leftover backend from a previous install holding the port is now detected and replaced at launch. Plus: the Dub tab's LLM engine finally runs on the provider you configured in Settings, history timestamps stop reading "20617d ago", and the Engines page can't crash under concurrent load.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Audiobook/Stories chapters with a `[pause]` no longer fail to render.** Pause spans were built as 1-D silence while every TTS engine returns 2-D audio, so the chapter concatenation crashed with `Tensors must have same number of dimensions` — any chapter containing a pause failed on every attempt (reported with a precise trace in #897). Silence now matches the rendered audio's shape at the source, and the chunk concatenator defensively normalizes mixed ranks (including honest mono→stereo broadcast) so no engine can re-trigger the class. (#953)
|
||||
- **Cinematic and Autofit dubbing can no longer invent dialogue.** The refine and slot-fit passes accepted any non-empty LLM reply for Latin-script languages — hallucinated lines, refusals, or the critique itself could ship as the dub. Every reply is now checked against the original line (length window, target script, critique echo — tunable via `OMNIVOICE_REFINE_RATIO_MIN/MAX`), rejected output falls back to the literal translation with an `adapt-diverged`/`fit-diverged` marker, lines too short to honestly fill their slot skip LLM expansion entirely, and both passes pin `temperature=0.2` like the Fast path. (#950)
|
||||
- **Dub timeline boxes can no longer flash invisible during playback.** On some Windows GPU/WebView2 driver combos the segment boxes under the video vanished and reappeared while playing (first reported in #373; the earlier fix was incomplete) — the timeline lane still animated a CSS transform every playback tick, keeping the translucent boxes on a composited layer that the driver mis-painted. Boxes are now positioned in pure layout with fully opaque theme-aware fills (pixel-identical colors), removing the glitch class on every platform. (#951)
|
||||
- **The dub "Speakers" count now actually does something — on every path.** The hint only reached pyannote; the common fallbacks silently ignored it (the no-diarization heuristic was hardcoded to alternate two speakers, and the FunASR shortcut never consulted it). The heuristic now cycles the requested count, an explicit count routes through pyannote when available, every path that can only approximate (or must ignore) the setting says so in a visible warning, and the legacy endpoint + a new CLI `--speakers` flag accept it too. Auto voice-cloning also stops fabricating voices from guessed labels: reference slices under 1.5s are rejected, slices bordering another speaker's turn are avoided, and cloning is skipped with an honest warning when speaker labels came from the gap heuristic instead of real diarization. (#952)
|
||||
- **Settings → Engines can no longer 500 under concurrent loads.** The lazy TTS/ASR engine registries held a *live* dictionary iterator open across each engine's `is_available()` probe while `list_backends()` ran in a FastAPI threadpool — so a second concurrent `/engines` request materializing a lazy engine entry (`self[key] = cls`) mutated the dict mid-iteration and crashed the request with `RuntimeError: dictionary changed size during iteration`. Both registries now snapshot their keys before iterating (atomic under the GIL), immune to a concurrent insert; regression-tested for TTS and ASR. (#940)
|
||||
- **The Dub tab's LLM translation engine now runs on your configured LLM provider.** Picking "LLM (OpenAI-compatible)" silently required three hand-set environment variables even when a provider was already configured and tested in Settings → LLM Providers; it now resolves through a new "Dub translation" LLM skill (route it to any provider — remote or local — in Settings → LLM Skills, independently of Cinematic refinement), keeps the `TRANSLATE_*` env vars as a power-user override, bounds every call with the LLM timeout instead of the SDK's 600-second default, tells the Engine dropdown whether the engine is actually ready (and via which provider), and — when nothing is configured — returns a clear pointer to Settings → LLM Providers instead of a raw 401 per segment. (#944)
|
||||
- **Timestamps no longer show "20617d ago" in OmniDrive/Projects.** The backend stores record times in Unix seconds while some views assumed milliseconds, so generation-history cards rendered as ~1970 ("20617d ago") and sorted last; every relative-time label (OmniDrive, sidebar history, dub projects, batch queue, transcriptions) now goes through one unit-tolerant formatter, and records missing a timestamp show "—" instead of an epoch age.
|
||||
- **Updating can no longer leave you secretly running the old version.** If a backend from a previous version was still holding the port (an orphan that survived an update), the new app "attached" to it because it answered health checks — so every fix in the update appeared to change nothing (the reported "bound port blocked the newer version"). The launcher now compares the running backend's version against the app before attaching: same version attaches as before, a stale one is killed and the bundled backend is started in its place — on macOS, Windows, and Linux. (#947)
|
||||
|
||||
## [0.3.9] — 2026-07-04
|
||||
|
||||
The dictation release — and a deep reliability pass driven by live-testing the entire app. **Dictation is rebuilt end-to-end**: instant feedback with a live waveform, words that commit about half a second after you stop speaking, clean punctuation, and text insertion that never lies about success. **LLM providers get one-click connection testing** with real diagnostics and model discovery, in all 21 languages. The app now **always opens maximized**, bottom buttons **can't hide under the footer** at small window sizes, and a wave of "out of memory / can't reach the backend / stuck at preparing" reports were traced to their real causes and fixed — including the silent VRAM crash on 8 GB cards, dead-IPC startup hangs after a Windows BSOD, and misleading error labels. Intel-Mac support status is now stated honestly, Confucius4-TTS is validated end-to-end, and Parakeet — roughly 20× faster than the default transcriber on CPU — is unlocked for every machine.
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
<a href="#why-ovs">Why OVS</a> ·
|
||||
<a href="#tts-engines">TTS Engines</a> ·
|
||||
<a href="#asr-engines">ASR Engines</a> ·
|
||||
<a href="#openai-api">API</a> ·
|
||||
<a href="#sponsors">Sponsors</a> ·
|
||||
<a href="#sponsor--donate">Donate</a> ·
|
||||
<a href="#contributing">Contributing</a> ·
|
||||
@@ -31,11 +32,21 @@
|
||||
<br/>
|
||||
|
||||
<div align="center">
|
||||
<img src=".github/assets/social-preview.png" alt="OmniVoice Studio — The open-source ElevenLabs alternative" width="100%"/>
|
||||
<img src="docs/screenshot-launchpad.png" alt="OmniVoice Studio — Launchpad" width="100%"/>
|
||||
</div>
|
||||
|
||||
> **Your voice is the most personal data you have. So why rent it back from a cloud?** Every mainstream voice tool ships your audio to someone else's server and bills you monthly for the privilege. OmniVoice Studio flips that: clone, design, dub, and dictate on your own hardware — 646 languages, no meter running, nothing leaving your machine.
|
||||
|
||||
<div align="center">
|
||||
|
||||
| 🔑 No API keys | 🙅 No accounts | ☁️ No cloud | 💳 No subscription |
|
||||
|:---:|:---:|:---:|:---:|
|
||||
| nothing to paste in | nothing to sign up for | your audio stays home | it's your computer |
|
||||
|
||||
</div>
|
||||
|
||||
> [!WARNING]
|
||||
> **OmniVoice Studio is in active beta.** Things may break between releases. For the latest features and fixes, clone the repo and run from source rather than using pre-built installers. Bug reports and PRs are very welcome — [open an issue](https://github.com/debpalash/OmniVoice-Studio/issues) or [join Discord](https://discord.gg/bzQavDfVV9).
|
||||
> **OmniVoice Studio is in active beta.** Things may break between releases — for the latest features and fixes, clone the repo and run from source rather than the pre-built installers. Bug reports and PRs are very welcome: [open an issue](https://github.com/debpalash/OmniVoice-Studio/issues) or [join Discord](https://discord.gg/bzQavDfVV9).
|
||||
|
||||
<div align="center">
|
||||
<br/>
|
||||
@@ -47,7 +58,68 @@
|
||||
|
||||
<br/>
|
||||
|
||||
## Features
|
||||
<a id="screenshots"></a>
|
||||
|
||||
## 📸 See it in action
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td align="center" width="50%">
|
||||
<img src="docs/screenshot-studio.png" alt="Studio" width="100%"/>
|
||||
<br/><b>Studio</b><br/>
|
||||
<sub>Generate & clone in one workspace — a 3-second clip mirrors any voice, 646 languages, zero-shot.</sub>
|
||||
</td>
|
||||
<td align="center" width="50%">
|
||||
<img src="docs/screenshot-design.png" alt="Voice Design" width="100%"/>
|
||||
<br/><b>Voice Design</b><br/>
|
||||
<sub>Build new voices from scratch — gender, age, accent, pitch, emotion, dialect.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-gallery.png" alt="Voice Gallery" width="100%"/>
|
||||
<br/><b>Voice Gallery</b><br/>
|
||||
<sub>Browse ready-made archetype voices with language filters — or build your own library.</sub>
|
||||
</td>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-dub.png" alt="Video Dubbing" width="100%"/>
|
||||
<br/><b>Video Dubbing</b><br/>
|
||||
<sub>A real dub, end to end: 37 segments transcribed, translated to Bengali, re-voiced, and timed — ready to export as MP4.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-engines.png" alt="Settings — Engines" width="100%"/>
|
||||
<br/><b>Settings → Engines</b><br/>
|
||||
<sub>The engine compatibility matrix — 14 TTS engines with per-engine GPU preflight, no silent CPU fallback.</sub>
|
||||
</td>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-settings.png" alt="Settings — Models" width="100%"/>
|
||||
<br/><b>Settings → Models</b><br/>
|
||||
<sub>One-click model store — auto-detects your platform (CUDA / MPS / CPU) and recommends the right models.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-openapi.png" alt="Settings — API Reference" width="100%"/>
|
||||
<br/><b>API Reference</b><br/>
|
||||
<sub>The full local REST API, embedded — every endpoint documented with copy-paste client snippets.</sub>
|
||||
</td>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-updates.png" alt="Settings — What's New" width="100%"/>
|
||||
<br/><b>What's New</b><br/>
|
||||
<sub>In-app changelog reader — see exactly what shipped in each release without leaving the app.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
---
|
||||
|
||||
<a id="features"></a>
|
||||
|
||||
## ✨ Features
|
||||
|
||||
The eight headliners — and twelve more waiting under the fold.
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
@@ -75,76 +147,44 @@
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>⌨️ Dictation Widget</h3>
|
||||
<p><code>⌘+⇧+Space</code> from <b>any app</b>.<br/>Transcribes, auto-pastes, disappears.</p>
|
||||
<p><kbd>⌘</kbd>+<kbd>⇧</kbd>+<kbd>Space</kbd> from <b>any app</b>.<br/>Transcribes, auto-pastes, disappears.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🔊 Vocal Isolation</h3>
|
||||
<p>Demucs-powered. Splits speech<br/>from music, <b>keeps the background</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>👥 Speaker Diarization</h3>
|
||||
<p>Pyannote + WhisperX.<br/><b>Auto-identifies</b> who said what.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
<h3>📦 Batch Queue</h3>
|
||||
<p>Drop <b>50 videos</b>, walk away.<br/>Progress bars per job.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🤖 MCP Server</h3>
|
||||
<p>Use OmniVoice from <b>Claude</b>,<br/>Cursor, or any MCP client.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🛡️ AI Watermark</h3>
|
||||
<p>AudioSeal (Meta). <b>Invisible</b>,<br/>survives compression.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🔬 Diagnostics</h3>
|
||||
<p>Self-check, error journal,<br/>scrubbed <b>diagnostic bundle</b>.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
<h3>🔐 100% Local</h3>
|
||||
<p>No keys, no cloud, no accounts.<br/><b>Your machine only</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>⚡ GPU Auto-Detect</h3>
|
||||
<p>CUDA · MPS · ROCm · CPU.<br/>≤8 GB? <b>Auto-offloads</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🧩 Extensible</h3>
|
||||
<p>Subclass <code>TTSbackend</code>,<br/>add any engine in <b>~50 lines</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🧭 Engine Routing</h3>
|
||||
<p>Preflight GPU check per engine.<br/><b>No silent CPU fallback</b>.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
<h3>🎒 Portable Personas</h3>
|
||||
<p>Export voices as <code>.ovsvoice</code><br/>bundles — identity + <b>watermark</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>♾️ Unlimited TTS</h3>
|
||||
<p>Sentence-chunked generation.<br/><b>No length cap</b>. Streaming via WS.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🌐 Remote Backend</h3>
|
||||
<p>Point UI at a remote server.<br/>Tailscale-friendly. <b>Bearer auth</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🧠 Dictation + LLM</h3>
|
||||
<p>Local LLM cleanup of transcripts.<br/>Optional echo <b>cancellation</b>.</p>
|
||||
<h3>🤖 MCP Server</h3>
|
||||
<p>Use OmniVoice from <b>Claude</b>,<br/>Cursor, or any MCP client.</p>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
<details>
|
||||
<summary><b>…and 12 more</b> — isolation, diarization, batch, watermarking, diagnostics, and friends</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
- 🔊 **Vocal Isolation** — Demucs-powered: splits speech from music and keeps the background bed.
|
||||
- 👥 **Speaker Diarization** — Pyannote + WhisperX auto-identify who said what.
|
||||
- 📦 **Batch Queue** — drop 50 videos, walk away; per-job progress bars.
|
||||
- 🛡️ **AI Watermark** — AudioSeal (Meta): invisible, survives compression.
|
||||
- 🔬 **Diagnostics** — self-check suite, error journal, scrubbed diagnostic bundles.
|
||||
- ⚡ **GPU Auto-Detect** — CUDA · MPS · ROCm · CPU; ≤8 GB VRAM auto-offloads.
|
||||
- 🧭 **Engine routing** — preflight GPU check per engine; no silent CPU fallback.
|
||||
- 🧩 **Extensible** — subclass `TTSBackend`, add any engine in ~50 lines.
|
||||
- 🎒 **Portable personas** — export voices as `.ovsvoice` bundles: identity + watermark.
|
||||
- ♾️ **Unlimited TTS** — sentence-chunked generation, no length cap, streaming via WebSocket.
|
||||
- 🌐 **Remote backend** — point the UI at a remote server; Tailscale-friendly, bearer auth.
|
||||
- 🧠 **Dictation + LLM** — local-LLM cleanup of transcripts, optional echo cancellation.
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Quickstart
|
||||
<a id="quickstart"></a>
|
||||
|
||||
## ⚡ Quickstart
|
||||
|
||||
<div align="center">
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
|
||||
@@ -157,14 +197,19 @@
|
||||
<sub><b>Intel Macs are not supported for the local backend:</b> the app UI installs, but the Python backend cannot run because PyTorch no longer ships Intel-Mac (x86_64) wheels (<a href="https://github.com/debpalash/OmniVoice-Studio/issues/889">#889</a>) — see <a href="docs/install/macos.md">docs/install/macos.md</a>.</sub>
|
||||
</div>
|
||||
|
||||
Per-OS install guides — pick yours and follow it end-to-end:
|
||||
Pick your OS and follow the guide end-to-end:
|
||||
|
||||
- **macOS** — [docs/install/macos.md](docs/install/macos.md)
|
||||
- **Windows** — [docs/install/windows.md](docs/install/windows.md)
|
||||
- **Linux** — [docs/install/linux.md](docs/install/linux.md)
|
||||
- **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
|
||||
- 🍎 **macOS** — [docs/install/macos.md](docs/install/macos.md)
|
||||
- 🪟 **Windows** — [docs/install/windows.md](docs/install/windows.md)
|
||||
- 🐧 **Linux** — [docs/install/linux.md](docs/install/linux.md)
|
||||
- 🐳 **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
|
||||
|
||||
Stuck? Run the built-in self-check first — **Settings → About → "Run
|
||||
<details>
|
||||
<summary><b>🧰 Stuck? Self-checks, tokens & restricted networks</b></summary>
|
||||
|
||||
<br/>
|
||||
|
||||
Run the built-in self-check first — **Settings → About → "Run
|
||||
self-check"** in the app, or `uv run python backend/main.py --diagnose` from
|
||||
a checkout (`--deep` also test-loads the active engine). Then see
|
||||
[docs/install/troubleshooting.md](docs/install/troubleshooting.md) for the
|
||||
@@ -179,57 +224,13 @@ diarization-specific gating, see
|
||||
speed, the ⚡ fast-download (Xet) status, and restricted-network / mirror
|
||||
options, see [docs/downloading-models.md](docs/downloading-models.md).
|
||||
|
||||
## Screenshots
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td align="center" width="50%">
|
||||
<img src="docs/screenshot-clone.png" alt="Voice Clone" width="100%"/>
|
||||
<br/><b>Voice Clone</b><br/>
|
||||
<sub>Drop a 3-second clip → mirror any voice. 646 languages, zero-shot.</sub>
|
||||
</td>
|
||||
<td align="center" width="50%">
|
||||
<img src="docs/screenshot-design.png" alt="Voice Design" width="100%"/>
|
||||
<br/><b>Voice Design</b><br/>
|
||||
<sub>Build new voices from scratch — gender, age, accent, pitch, style.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-dub.png" alt="Video Dubbing" width="100%"/>
|
||||
<br/><b>Video Dubbing</b><br/>
|
||||
<sub>Upload or paste a YouTube URL. Transcribe, translate, re-voice, export.</sub>
|
||||
</td>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-gallery.png" alt="Voice Gallery" width="100%"/>
|
||||
<br/><b>Voice Gallery</b><br/>
|
||||
<sub>Search YouTube, browse categories, download clips, build your library.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-settings.png" alt="Settings — Models" width="100%"/>
|
||||
<br/><b>Settings → Models</b><br/>
|
||||
<sub>15 models. One-click install. Auto-detects your platform (CUDA / MPS / CPU).</sub>
|
||||
</td>
|
||||
<td align="center">
|
||||
<img src="docs/screenshot-libraryprojects.png" alt="Projects" width="100%"/>
|
||||
<br/><b>Projects</b><br/>
|
||||
<sub>Dub projects, voice profiles, generation history, exports — all searchable.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" colspan="2">
|
||||
<img src="docs/screenshot-logs.png" alt="Settings — Logs" width="100%"/>
|
||||
<br/><b>Settings → Logs</b><br/>
|
||||
<sub>Live backend, frontend, and Tauri runtime logs. Filter, refresh, clear.</sub>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Why OVS?
|
||||
<a id="why-ovs"></a>
|
||||
|
||||
## 💡 Why OmniVoice?
|
||||
|
||||
ElevenLabs charges **$5–$330/mo** and processes your audio on their servers. OmniVoice Studio runs **on your hardware, with no usage limits.**
|
||||
|
||||
@@ -245,13 +246,13 @@ ElevenLabs charges **$5–$330/mo** and processes your audio on their servers. O
|
||||
| **API Keys** | Required | Not needed |
|
||||
| **GPU Support** | N/A (cloud) | CUDA · Apple Silicon · ROCm · CPU |
|
||||
| **Desktop App** | ❌ | ✅ macOS · Windows · Linux |
|
||||
| **TTS Engines** | 1 | **11** (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, IndexTTS 2, OmniVoice GGUF, Supertonic 3) |
|
||||
| **TTS Engines** | 1 | **14** (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS) |
|
||||
| **ASR Engines** | 1 | **9** (WhisperX, Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet, Moonshine, FunASR, isolated Faster-Whisper, sherpa-onnx live dictation) |
|
||||
| **MCP Server** | ❌ | ✅ Use from Claude, Cursor, any MCP client |
|
||||
| **Self-check** | ❌ | ✅ Diagnostics suite, error journal, scrubbed debug bundles |
|
||||
| **Customizable** | ❌ Closed | ✅ Fork it, extend it, ship it |
|
||||
|
||||
OmniVoice Studio gives you professional-grade AI tools without the subscription or the cloud.
|
||||
Professional-grade voice AI, minus the subscription and the cloud.
|
||||
|
||||
<div align="center">
|
||||
<br/>
|
||||
@@ -262,7 +263,7 @@ OmniVoice Studio gives you professional-grade AI tools without the subscription
|
||||
|
||||
---
|
||||
|
||||
## System Requirements
|
||||
## 🖥️ System Requirements
|
||||
|
||||
| | **Minimum** | **Recommended** |
|
||||
|---|---|---|
|
||||
@@ -279,9 +280,16 @@ OmniVoice Studio gives you professional-grade AI tools without the subscription
|
||||
> [!IMPORTANT]
|
||||
> **macOS Intel (x86_64) is unsupported for the local backend:** the app UI installs, but the Python backend cannot run because PyTorch no longer ships Intel-Mac wheels ([#889](https://github.com/debpalash/OmniVoice-Studio/issues/889)). Intel-Mac users can still point the UI at a remote backend on another machine — see [docs/install/macos.md](docs/install/macos.md).
|
||||
|
||||
### TTS Engines
|
||||
<a id="tts-engines"></a>
|
||||
|
||||
OmniVoice ships a multi-engine TTS backend. The default engine (OmniVoice) is always available; additional engines are opt-in and auto-detected. Switch engines in **Settings → TTS Engine** or via the `OMNIVOICE_TTS_BACKEND` env var.
|
||||
### 🗣️ TTS Engines
|
||||
|
||||
**14 engines, one picker.** OmniVoice (default, 600+ languages) is always available; CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, and Sherpa-ONNX are opt-in and auto-detected — plus six lazy-installed heavyweights (IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS). Switch in **Settings → TTS Engine** or via the `OMNIVOICE_TTS_BACKEND` env var.
|
||||
|
||||
<details>
|
||||
<summary><b>📊 The full matrix</b> — 14 engines × platform × clone/instruct × license</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| Engine | Languages | Clone | Instruct | Linux | macOS ARM | Windows | License |
|
||||
|--------|:---------:|:-----:|:--------:|:-----:|:---------:|:-------:|:-------:|
|
||||
@@ -304,9 +312,18 @@ OmniVoice ships a multi-engine TTS backend. The default engine (OmniVoice) is al
|
||||
>
|
||||
> **MOSS-TTS-v1.5** (8B, ~16 GB weights) and **dots.tts** (2B, ~9 GB weights) are heavyweight opt-in engines that run in their own isolated venv from a local clone — see [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) and [dots.tts](docs/engines/dots-tts.md). Neither claims Apple-Silicon **MPS** (upstream is CUDA/CPU only; on a Mac they run on CPU). dots.tts upstream is Linux/macOS only — no Windows path. **Confucius4-TTS** (14-language cross-lingual zero-shot cloning) is similar — its own Python 3.10 venv from a clone; CUDA recommended, CPU validated end-to-end (slow, ~17× realtime; no MPS — tested slower than CPU); see [Confucius4-TTS](docs/engines/confucius4-tts.md).
|
||||
|
||||
### ASR Engines
|
||||
</details>
|
||||
|
||||
OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictation, video dubbing, and subtitle generation — all fully local. **WhisperX** is the cross-platform default; the rest are opt-in and auto-detected. Switch in **Settings → ASR Engine** or via the `OMNIVOICE_ASR_BACKEND` env var.
|
||||
<a id="asr-engines"></a>
|
||||
|
||||
### 🎧 ASR Engines
|
||||
|
||||
**9 engines, all fully local** — they power dictation, video dubbing, and subtitles. **WhisperX** is the cross-platform default (~100 languages, word-level timing); the rest are opt-in and auto-detected. Switch in **Settings → ASR Engine** or via the `OMNIVOICE_ASR_BACKEND` env var.
|
||||
|
||||
<details>
|
||||
<summary><b>📊 The full lineup</b> — 9 engines, what each is best at, and compute-type notes</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| Engine | `OMNIVOICE_ASR_BACKEND` | Languages | Best for |
|
||||
|--------|-------------------------|:---------:|----------|
|
||||
@@ -324,9 +341,11 @@ OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictatio
|
||||
|
||||
> **GPU without efficient float16?** On older NVIDIA GPUs (Maxwell/Pascal, GTX 16xx) or after a CTranslate2/cuDNN mismatch, the CTranslate2 ASR engines (WhisperX, Faster-Whisper) can't run `float16` and OmniVoice automatically retries on `int8` — no config needed. If transcription still fails, pin the compute type with the `ASR_COMPUTE_TYPE` env var (escape hatch): `ASR_COMPUTE_TYPE=int8` (or `float32` for CPU). Set it to `int8` and restart the backend.
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
## 🏗️ Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
@@ -344,11 +363,50 @@ OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictatio
|
||||
CUDA / MPS / ROCm / CPU (auto-detected + routed)
|
||||
```
|
||||
|
||||
<a id="openai-api"></a>
|
||||
|
||||
## 🔌 OpenAI-compatible API
|
||||
|
||||
Already have a script, agent, or tool that speaks OpenAI's audio API? Point it at `http://localhost:3900/v1` — no key needed, no code changes. The backend ships a drop-in surface for the audio endpoints, wired to whichever TTS/ASR engine you have active (and yes, `voice` accepts your cloned voice-profile IDs).
|
||||
|
||||
| Endpoint | What it does |
|
||||
|---|---|
|
||||
| `POST /v1/audio/speech` | TTS — text in; `mp3` / `wav` / `flac` / `opus` / `pcm` out. `tts-1` / `tts-1-hd` map to your active engine; OpenAI voice names (`alloy`, …) are accepted. |
|
||||
| `POST /v1/audio/transcriptions` | STT — audio file in; `json`, `text`, `verbose_json`, `srt`, or `vtt` out. `whisper-1` maps to your active ASR engine. |
|
||||
| `GET /v1/audio/voices` | OmniVoice extension — lists every voice profile and engine, so clients can discover your clones. |
|
||||
|
||||
```sh
|
||||
curl http://localhost:3900/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "tts-1", "voice": "alloy", "input": "Generated on my own hardware.", "response_format": "wav"}' \
|
||||
--output speech.wav
|
||||
```
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:3900/v1", api_key="none") # any string works — nothing checks it
|
||||
|
||||
result = client.audio.transcriptions.create(model="whisper-1", file=open("clip.wav", "rb"))
|
||||
print(result.text)
|
||||
```
|
||||
|
||||
Want the whole surface (100+ endpoints)? The full REST API reference is embedded in the app — **Settings → OpenAPI Reference** (Scalar-powered), or the `{}` button in the footer.
|
||||
|
||||
---
|
||||
|
||||
## Roadmap
|
||||
## 🗺️ Roadmap
|
||||
|
||||
### ✅ Shipped
|
||||
### 🔜 Up Next
|
||||
|
||||
- 🎬 **Lip-sync v2** — visual speech timing with wav2lip
|
||||
- 🌐 **Hosted Demo** — try OmniVoice without installing anything
|
||||
- 🔌 **Plugin Marketplace** — community-contributed TTS engines and effects
|
||||
- 🎵 **Real-time Voice Changer** — live microphone transformation during calls
|
||||
|
||||
<details>
|
||||
<summary><b>✅ Everything shipped so far</b> — the receipts, by category</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| Category | Features |
|
||||
|----------|----------|
|
||||
@@ -359,7 +417,7 @@ OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictatio
|
||||
| **Multi-Lang** | Multi-language batch picker, batch dubbing queue with sequential GPU execution |
|
||||
| **Diarization** | Pyannote ML diarization, auto speaker clone extraction, per-speaker voice assignment |
|
||||
| **ASR** | 9 engines (WhisperX, Faster-Whisper, isolated Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet TDT, Moonshine, FunASR/SenseVoice, sherpa-onnx live dictation), crash-isolated subprocess backend |
|
||||
| **TTS** | 11 engines (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, + lazy: IndexTTS 2, OmniVoice GGUF, Supertonic 3), engine routing with GPU preflight |
|
||||
| **TTS** | 14 engines (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, + lazy: IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS), engine routing with GPU preflight |
|
||||
| **Infra** | Docker deployment, CUDA/MPS/ROCm auto-detect, cuDNN 8 compat, VRAM-aware model offloading, engine routing (no silent CPU fallback), diagnostics suite & error journal, restricted-network mirror support |
|
||||
| **AI Provenance** | AudioSeal invisible watermarking (SynthID-like), video logo overlay, watermark detection API |
|
||||
| **UX** | Undo/redo, keyboard shortcuts, drag-and-drop, session persistence, glassmorphism design system, UI scale fix for Linux/WebKitGTK |
|
||||
@@ -372,16 +430,13 @@ OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictatio
|
||||
| **Remote Backend** | Point the desktop UI at a remote backend URL with bearer auth (Tailscale-documented) |
|
||||
| **Reliability** | Stall watchdog on bootstrap splash, per-engine GPU compatibility matrix, actionable errors for non-executable engine binaries, setuptools auto-repair |
|
||||
|
||||
### 🔜 Up Next
|
||||
|
||||
- 🎬 **Lip-sync v2** — visual speech timing with wav2lip
|
||||
- 🌐 **Hosted Demo** — try OmniVoice without installing anything
|
||||
- 🔌 **Plugin Marketplace** — community-contributed TTS engines and effects
|
||||
- 🎵 **Real-time Voice Changer** — live microphone transformation during calls
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Sponsor / Donate
|
||||
<a id="sponsor--donate"></a>
|
||||
|
||||
## 💜 Sponsor / Donate
|
||||
|
||||
OmniVoice Studio is built by one developer using Claude Code and AI agents — and the agent bills are real. Over the last three months I've spent thousands of dollars on Claude subscriptions to keep the features shipping, the bugs fixed, and your issues answered. If OmniVoice has created value for you, helping cover those bills means I can keep developing full-time.
|
||||
|
||||
@@ -402,7 +457,9 @@ OmniVoice Studio is built by one developer using Claude Code and AI agents — a
|
||||
|
||||
</div>
|
||||
|
||||
### Sponsors
|
||||
<a id="sponsors"></a>
|
||||
|
||||
### 🌟 Sponsors
|
||||
|
||||
OmniVoice is **free** and **AGPL-3.0** — no paid tier, no SaaS revenue. Sponsors keep development going, and in return get a logo slot here, in the app, and (for top tiers) on the project website. It's a thank-you, never a paywall. **[See tiers & become a sponsor →](SPONSORS.md)**
|
||||
|
||||
@@ -420,29 +477,37 @@ OmniVoice is **free** and **AGPL-3.0** — no paid tier, no SaaS revenue. Sponso
|
||||
|
||||
---
|
||||
|
||||
## Community
|
||||
## 💬 Community
|
||||
|
||||
<div align="center">
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/💬_Discord-Join_Community-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="Join Discord" /></a>
|
||||
<br/>
|
||||
<sub>We respond to setup questions within hours, not days.</sub>
|
||||
</div>
|
||||
|
||||
<details>
|
||||
<summary><b>What happens in there</b></summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| Channel | What happens there |
|
||||
|---------|--------------------|
|
||||
| `#showcase` | Members share their dubs, clones, and voice designs |
|
||||
| `#help` | Setup issues, GPU troubleshooting, model questions |
|
||||
| `#feature-requests` | Vote on what gets built next |
|
||||
| `#dev` | Architecture discussions, PR reviews, engine integrations |
|
||||
| `#announcements` | Release notes, breaking changes, early access |
|
||||
| `#announcements` | Release news and the big moments — new versions land here first |
|
||||
| `#releases` + `#changelog` | Every build and exactly what's inside it |
|
||||
| `#issues` | Bug reports as forum posts — triaged straight into GitHub issues |
|
||||
| `#ideas` | Feature requests, discussed and voted on |
|
||||
| `#discuss-ideas` | Design talk before things get built |
|
||||
| `#general` | Setup help, GPU troubleshooting, and showing off your dubs |
|
||||
|
||||
**[→ Join the Discord](https://discord.gg/bzQavDfVV9)** — we respond to setup questions within hours, not days.
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
<a id="contributing"></a>
|
||||
|
||||
We welcome contributions of all kinds — bug fixes, new TTS engine adapters, UI improvements, docs, and translations.
|
||||
## 🤝 Contributing
|
||||
|
||||
Yes please — bug fixes, new TTS engine adapters, UI improvements, docs, translations. All of it.
|
||||
|
||||
- 📖 Read the **[Contributing Guide](CONTRIBUTING.md)** for setup, code style, and PR workflow
|
||||
- 🐛 Browse [good first issues](https://github.com/debpalash/OmniVoice-Studio/labels/good%20first%20issue)
|
||||
@@ -450,7 +515,7 @@ We welcome contributions of all kinds — bug fixes, new TTS engine adapters, UI
|
||||
|
||||
---
|
||||
|
||||
## FAQ
|
||||
## ❓ FAQ
|
||||
|
||||
<details>
|
||||
<summary><b>Is this really as good as ElevenLabs?</b></summary>
|
||||
@@ -485,12 +550,14 @@ Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are availab
|
||||
<details>
|
||||
<summary><b>Can I add my own TTS engine?</b></summary>
|
||||
<br/>
|
||||
Yes. OmniVoice uses a <b>built-in backend registry</b>. To add an engine in ~50 lines, subclass <code>TTSBackend</code> in <code>backend/services/tts_backend.py</code> and add it to the <code>_REGISTRY</code> dictionary. Eleven engines are built in: OmniVoice, CosyVoice 3, GPT-SoVITS, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, KittenTTS, Sherpa-ONNX, plus lazy-registered IndexTTS 2, OmniVoice GGUF, and Supertonic 3. See the <a href="#tts-engines">TTS Engines</a> section for details.
|
||||
Yes. OmniVoice uses a <b>built-in backend registry</b>. To add an engine in ~50 lines, subclass <code>TTSBackend</code> in <code>backend/services/tts_backend.py</code> and add it to the <code>_REGISTRY</code> dictionary. Fourteen engines are built in: OmniVoice, CosyVoice 3, GPT-SoVITS, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, KittenTTS, Sherpa-ONNX, plus lazy-registered IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, and Confucius4-TTS. See the <a href="#tts-engines">TTS Engines</a> section for details.
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
<a id="license"></a>
|
||||
|
||||
## 📜 License
|
||||
|
||||
OmniVoice Studio is free and open-source software under the [**GNU Affero General Public License v3.0 (AGPL-3.0)**](https://www.gnu.org/licenses/agpl-3.0.html).
|
||||
|
||||
@@ -502,7 +569,7 @@ The bundled `omnivoice/` TTS model by Han Zhu remains Apache-2.0 upstream. See [
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgments
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
OmniVoice Studio is built on the shoulders of exceptional open-source work:
|
||||
|
||||
@@ -521,6 +588,17 @@ OmniVoice Studio is built on the shoulders of exceptional open-source work:
|
||||
|
||||
---
|
||||
|
||||
## 🧰 More local open-source from the maker
|
||||
|
||||
Like the local-first philosophy? It runs in the family:
|
||||
|
||||
| Project | What it is |
|
||||
|---------|------------|
|
||||
| [**Opal**](https://github.com/debpalash/Opal) 💠 | **Play everything.** The evolved media player for the next decades of entertainment — video, anime, comics, torrents, Jellyfin/Plex, with local AI built in. |
|
||||
| [**memxt**](https://github.com/debpalash/memxt) 🧠 | **The fastest benchmarked open-source AI memory system.** 100% local memory for AI agents, with MCP support. |
|
||||
|
||||
---
|
||||
|
||||
<div align="center">
|
||||
|
||||
<br/>
|
||||
|
||||
@@ -375,6 +375,31 @@ _CHUNK_TRANSCRIBE_ATTEMPTS = max(1, int(os.environ.get("OMNIVOICE_TRANSCRIBE_CHU
|
||||
_sse_event = dub_pipeline.sse_event
|
||||
_prep_event_helper = dub_pipeline.prep_event # alias; we keep the module-local _prep_event below for the inline one-liner shape
|
||||
|
||||
#: User-facing warning emitted when auto voice cloning is skipped because the
|
||||
#: speaker labels came from the silence-gap heuristic (see _diarize /
|
||||
#: extract_speaker_clones — gap-based labels routinely mix two people's audio
|
||||
#: into one reference, which is how "made up" clone voices happen).
|
||||
CLONE_SKIP_HEURISTIC_MSG = (
|
||||
"auto voice cloning skipped: speaker labels are gap-based estimates — "
|
||||
"set up diarization (Settings → Models → pyannote) for per-speaker clones"
|
||||
)
|
||||
|
||||
|
||||
def _clamp_num_speakers(value) -> Optional[int]:
|
||||
"""Clamp the user's speaker-count hint to a sane 1–20 range.
|
||||
|
||||
Shared by the SSE and legacy transcribe endpoints so the two can't drift.
|
||||
None / non-int / out-of-range → None (auto-detect), so a bad query string
|
||||
can never break a diarization call.
|
||||
"""
|
||||
if value is None:
|
||||
return None
|
||||
try:
|
||||
value = int(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return value if 1 <= value <= 20 else None
|
||||
|
||||
|
||||
@router.get("/dub/transcribe-stream/{job_id}")
|
||||
async def dub_transcribe_stream(
|
||||
@@ -393,15 +418,14 @@ async def dub_transcribe_stream(
|
||||
pyannote auto-detects the count — but its auto-detect can collapse a
|
||||
multi-speaker clip to a single speaker (issue #274). When the user knows
|
||||
the exact count, supplying it forces pyannote to return that many speakers.
|
||||
On paths that can't honor the hint exactly (inline ASR turns, the
|
||||
silence-gap heuristic) it is never silently dropped: the heuristic cycles
|
||||
the requested count and a `warning` SSE event tells the user how far the
|
||||
labels can be trusted.
|
||||
"""
|
||||
# Clamp to a sane range; ignore anything non-positive / absurd so a bad
|
||||
# query string can never break the diarization call. None → auto-detect.
|
||||
if num_speakers is not None:
|
||||
try:
|
||||
num_speakers = int(num_speakers)
|
||||
num_speakers = num_speakers if 1 <= num_speakers <= 20 else None
|
||||
except (TypeError, ValueError):
|
||||
num_speakers = None
|
||||
num_speakers = _clamp_num_speakers(num_speakers)
|
||||
|
||||
job = _get_job(job_id)
|
||||
|
||||
@@ -630,7 +654,11 @@ async def dub_transcribe_stream(
|
||||
snap_segment_starts(chunk_segs, audio_np, sr)
|
||||
except Exception as e:
|
||||
logger.warning("onset alignment skipped for chunk %d: %s", i, e)
|
||||
chunk_segs = assign_speakers_heuristic(chunk_segs)
|
||||
# Provisional per-chunk labels for the streaming UI only — the
|
||||
# final diarization pass below overwrites them. Honor the user's
|
||||
# speaker-count hint here too so the interim view doesn't flip
|
||||
# between 2 and N speakers.
|
||||
chunk_segs = assign_speakers_heuristic(chunk_segs, num_speakers)
|
||||
for s in chunk_segs:
|
||||
s["id"] = f"s{next_seg_id:05x}"
|
||||
s["text_original"] = s.get("text", "")
|
||||
@@ -684,33 +712,110 @@ async def dub_transcribe_stream(
|
||||
return
|
||||
|
||||
def _diarize():
|
||||
"""Returns (segments, warning_payload_or_None).
|
||||
"""Returns (segments, warning_payload_or_None, labels_source).
|
||||
|
||||
`labels_source` records where the speaker labels came from —
|
||||
`"pyannote"` | `"turns"` | `"heuristic"` — so downstream
|
||||
auto-clone extraction can refuse to cut reference audio from
|
||||
gap-based estimates (a mixed-speaker reference is how "made up"
|
||||
clone voices happen).
|
||||
|
||||
`warning_payload` is a structured dict
|
||||
`{detail, error_class, docs_url}` whenever we silently fell back
|
||||
to the silence-gap heuristic (no HF_TOKEN, model unavailable,
|
||||
license not accepted, or pyannote raised). The heuristic only
|
||||
detects speaker turns from >1.2s silences, so a rapid-fire
|
||||
man↔woman exchange will read as one speaker. Issue #78 — we
|
||||
attach an `error_class` so the front-end's errorDocsMap can
|
||||
render a "See docs" deeplink instead of a dead-end toast.
|
||||
license not accepted, or pyannote raised) — or whenever the
|
||||
user's `num_speakers` hint could not be honored exactly. The
|
||||
heuristic only detects speaker turns from >1.2s silences, so a
|
||||
rapid-fire man↔woman exchange will read as one speaker. Issue
|
||||
#78 — we attach an `error_class` so the front-end's errorDocsMap
|
||||
can render a "See docs" deeplink instead of a dead-end toast.
|
||||
"""
|
||||
# The active ASR backend already diarized inline (FunASR cam++):
|
||||
# use its speaker turns directly and skip pyannote entirely (#182).
|
||||
if asr_speaker_turns:
|
||||
logger.info("Using inline ASR diarization (%d turns); skipping pyannote.", len(asr_speaker_turns))
|
||||
assigned = assign_speakers_from_turns(all_segments, asr_speaker_turns)
|
||||
# #486: split any segment that spans two speakers' turns at the
|
||||
# word boundary (single-speaker segments pass through unchanged).
|
||||
return resplit_segments_by_turns(assigned, all_words, asr_speaker_turns), None
|
||||
|
||||
from services.model_manager import (
|
||||
DIARIZATION_ERR_LICENSE,
|
||||
DIARIZATION_ERR_NO_TOKEN,
|
||||
)
|
||||
from core import error_docs_map
|
||||
|
||||
diar_pipe, err_sentinel = get_diarization_pipeline(return_error=True)
|
||||
def _hint_suffix() -> str:
|
||||
"""Honest caveat appended to heuristic-fallback warnings when a
|
||||
multi-speaker hint is set: the count is now honored, but the
|
||||
heuristic can't attribute voices. (A hint of 1 IS fully
|
||||
honored — one label — so it needs no caveat.)"""
|
||||
if not num_speakers or num_speakers < 2:
|
||||
return ""
|
||||
return (
|
||||
f" Your speaker-count setting ({num_speakers}) is only "
|
||||
f"approximately honored: the heuristic cycles "
|
||||
f"{num_speakers} speaker labels on silence gaps instead "
|
||||
f"of recognizing voices, so lines may be attributed to "
|
||||
f"the wrong speaker."
|
||||
)
|
||||
|
||||
def _use_turns(crash: Exception | None = None, err_sentinel=None):
|
||||
"""Label from the ASR backend's inline speaker turns; warn when
|
||||
that means the user's explicit count can't be enforced."""
|
||||
logger.info(
|
||||
"Using inline ASR diarization (%d turns)%s.",
|
||||
len(asr_speaker_turns),
|
||||
"" if crash else "; skipping pyannote",
|
||||
)
|
||||
assigned = assign_speakers_from_turns(all_segments, asr_speaker_turns)
|
||||
# #486: split any segment that spans two speakers' turns at the
|
||||
# word boundary (single-speaker segments pass through unchanged).
|
||||
resplit = resplit_segments_by_turns(assigned, all_words, asr_speaker_turns)
|
||||
if not num_speakers:
|
||||
return resplit, None, "turns"
|
||||
error_class = (
|
||||
"HF_AUTH_FAILED"
|
||||
if err_sentinel == DIARIZATION_ERR_NO_TOKEN
|
||||
else "PYANNOTE_LICENSE_REQUIRED"
|
||||
)
|
||||
if crash:
|
||||
detail = (
|
||||
f"Speaker diarization crashed mid-run "
|
||||
f"({type(crash).__name__}); falling back to the ASR "
|
||||
f"engine's built-in speaker turns. Speaker-count hint "
|
||||
f"ignored: the detected count may differ from the "
|
||||
f"{num_speakers} you set."
|
||||
)
|
||||
else:
|
||||
detail = (
|
||||
f"Speaker-count hint ignored: pyannote diarization is "
|
||||
f"unavailable, so the ASR engine's built-in speaker "
|
||||
f"turns were used and the detected count may differ "
|
||||
f"from the {num_speakers} you set. Set up diarization "
|
||||
f"(Settings → Models → pyannote) to enforce an exact "
|
||||
f"speaker count."
|
||||
)
|
||||
return resplit, {
|
||||
"detail": detail,
|
||||
"error_class": error_class,
|
||||
"docs_url": error_docs_map.lookup(error_class),
|
||||
"speaker_hint": {"requested": num_speakers, "status": "ignored"},
|
||||
}, "turns"
|
||||
|
||||
# The active ASR backend already diarized inline (FunASR cam++):
|
||||
# its turns are the fast path and skip pyannote entirely (#182) —
|
||||
# but ONLY when the user didn't set an explicit speaker count.
|
||||
# Inline turns are labeled per-30s-chunk and can't be forced to N
|
||||
# speakers, so a set num_speakers prefers pyannote — the one
|
||||
# engine that honors an exact count. When pyannote can't load,
|
||||
# the turns are still the best labels available; use them and say
|
||||
# so instead of silently eating the hint.
|
||||
diar_pipe = None
|
||||
err_sentinel = None
|
||||
if asr_speaker_turns:
|
||||
if num_speakers:
|
||||
diar_pipe, err_sentinel = get_diarization_pipeline(return_error=True)
|
||||
if not diar_pipe:
|
||||
return _use_turns(err_sentinel=err_sentinel)
|
||||
logger.info(
|
||||
"num_speakers=%d set: preferring pyannote over %d inline "
|
||||
"ASR turns (only pyannote honors an exact count).",
|
||||
num_speakers, len(asr_speaker_turns),
|
||||
)
|
||||
else:
|
||||
diar_pipe, err_sentinel = get_diarization_pipeline(return_error=True)
|
||||
if not diar_pipe:
|
||||
# Phase 1 AUTH-01: ask the resolver (App → Env → HF-CLI),
|
||||
# not just the env var. This is the #35 fix — users who
|
||||
@@ -761,13 +866,20 @@ async def dub_transcribe_stream(
|
||||
f"heuristic; rapid speaker turns may be merged."
|
||||
)
|
||||
error_class = "PYANNOTE_LICENSE_REQUIRED"
|
||||
warning = {
|
||||
"detail": detail + _hint_suffix(),
|
||||
"error_class": error_class,
|
||||
"docs_url": error_docs_map.lookup(error_class),
|
||||
}
|
||||
if num_speakers:
|
||||
warning["speaker_hint"] = {
|
||||
"requested": num_speakers,
|
||||
"status": "approximate" if num_speakers > 1 else "honored",
|
||||
}
|
||||
return (
|
||||
assign_speakers_heuristic(all_segments),
|
||||
{
|
||||
"detail": detail,
|
||||
"error_class": error_class,
|
||||
"docs_url": error_docs_map.lookup(error_class),
|
||||
},
|
||||
assign_speakers_heuristic(all_segments, num_speakers),
|
||||
warning,
|
||||
"heuristic",
|
||||
)
|
||||
try:
|
||||
# Pass the user's speaker-count hint through to pyannote when
|
||||
@@ -782,9 +894,14 @@ async def dub_transcribe_stream(
|
||||
assigned = assign_speakers_from_diarization(all_segments, diar)
|
||||
# #486: split any segment that spans two speakers' turns at the
|
||||
# word boundary (single-speaker segments pass through unchanged).
|
||||
return resplit_segments_by_diarization(assigned, all_words, diar), None
|
||||
return resplit_segments_by_diarization(assigned, all_words, diar), None, "pyannote"
|
||||
except Exception as e:
|
||||
logger.error(f"Diarization failed: {e}")
|
||||
# Inline ASR turns beat the silence-gap heuristic as a crash
|
||||
# fallback (this path is reachable with turns present since a
|
||||
# set num_speakers routes turns-jobs through pyannote).
|
||||
if asr_speaker_turns:
|
||||
return _use_turns(crash=e)
|
||||
# Mid-run failure — classify against the same sentinels so a
|
||||
# post-load 401 (rare but possible after a token rotation)
|
||||
# still gets the right docs deeplink.
|
||||
@@ -795,36 +912,50 @@ async def dub_transcribe_stream(
|
||||
if err_class_post == DIARIZATION_ERR_LICENSE
|
||||
else "PYANNOTE_LICENSE_REQUIRED" # LOAD failures land here too
|
||||
)
|
||||
warning = {
|
||||
"detail": (
|
||||
f"Speaker diarization crashed mid-run "
|
||||
f"({type(e).__name__}); falling back to a silence-gap "
|
||||
f"heuristic. Rapid speaker turns may be merged."
|
||||
+ _hint_suffix()
|
||||
),
|
||||
"error_class": error_class,
|
||||
"docs_url": error_docs_map.lookup(error_class),
|
||||
}
|
||||
if num_speakers:
|
||||
warning["speaker_hint"] = {
|
||||
"requested": num_speakers,
|
||||
"status": "approximate" if num_speakers > 1 else "honored",
|
||||
}
|
||||
return (
|
||||
assign_speakers_heuristic(all_segments),
|
||||
{
|
||||
"detail": (
|
||||
f"Speaker diarization crashed mid-run "
|
||||
f"({type(e).__name__}); falling back to a silence-gap "
|
||||
f"heuristic. Rapid speaker turns may be merged."
|
||||
),
|
||||
"error_class": error_class,
|
||||
"docs_url": error_docs_map.lookup(error_class),
|
||||
},
|
||||
assign_speakers_heuristic(all_segments, num_speakers),
|
||||
warning,
|
||||
"heuristic",
|
||||
)
|
||||
|
||||
fut_diar = loop.run_in_executor(_gpu_pool, _diarize)
|
||||
final_segs = None
|
||||
diar_warning = None
|
||||
labels_source = "heuristic"
|
||||
while True:
|
||||
done, pending = await asyncio.wait([fut_diar], timeout=5.0)
|
||||
if done:
|
||||
final_segs, diar_warning = done.pop().result()
|
||||
final_segs, diar_warning, labels_source = done.pop().result()
|
||||
break
|
||||
yield _sse_event("ping", {})
|
||||
if diar_warning:
|
||||
logger.warning("diarization fallback: %s", diar_warning.get("detail"))
|
||||
yield _sse_event("warning", {
|
||||
payload = {
|
||||
"detail": diar_warning.get("detail"),
|
||||
"source": "diarization",
|
||||
"error_class": diar_warning.get("error_class"),
|
||||
"docs_url": diar_warning.get("docs_url"),
|
||||
})
|
||||
}
|
||||
# Machine-readable trail of what happened to the user's
|
||||
# speaker-count hint (the `detail` text carries the human story).
|
||||
if diar_warning.get("speaker_hint"):
|
||||
payload["speaker_hint"] = diar_warning["speaker_hint"]
|
||||
yield _sse_event("warning", payload)
|
||||
|
||||
job["segments"] = final_segs
|
||||
|
||||
@@ -836,17 +967,37 @@ async def dub_transcribe_stream(
|
||||
try:
|
||||
from services.speaker_clone import extract_speaker_clones, auto_profile_id
|
||||
vocals_for_clone = job.get("vocals_path") or asr_audio_target
|
||||
fut_clones = loop.run_in_executor(
|
||||
_cpu_pool, extract_speaker_clones,
|
||||
vocals_for_clone, final_segs, os.path.dirname(vocals_for_clone),
|
||||
)
|
||||
clones = None
|
||||
while True:
|
||||
done, pending = await asyncio.wait([fut_clones], timeout=5.0)
|
||||
if done:
|
||||
clones = done.pop().result()
|
||||
break
|
||||
yield _sse_event("ping", {})
|
||||
clones = {}
|
||||
if labels_source == "heuristic":
|
||||
# Clone-purity guard: heuristic labels are silence-gap
|
||||
# estimates, not voice identity — a per-speaker reference cut
|
||||
# from them routinely concatenates two people's audio and the
|
||||
# clone sounds "made up". Skip auto-clones and say so instead
|
||||
# of shipping bad ones. (extract_speaker_clones enforces the
|
||||
# same guard internally; this branch exists to surface the
|
||||
# warning to the user.)
|
||||
logger.info(
|
||||
"auto speaker clones skipped (labels_source=heuristic, job=%s)",
|
||||
job_id,
|
||||
)
|
||||
yield _sse_event("warning", {
|
||||
"detail": CLONE_SKIP_HEURISTIC_MSG,
|
||||
"source": "speaker_clone",
|
||||
})
|
||||
else:
|
||||
fut_clones = loop.run_in_executor(
|
||||
_cpu_pool, lambda: extract_speaker_clones(
|
||||
vocals_for_clone, final_segs,
|
||||
os.path.dirname(vocals_for_clone),
|
||||
labels_source=labels_source,
|
||||
),
|
||||
)
|
||||
while True:
|
||||
done, pending = await asyncio.wait([fut_clones], timeout=5.0)
|
||||
if done:
|
||||
clones = done.pop().result()
|
||||
break
|
||||
yield _sse_event("ping", {})
|
||||
# Wave 3.2: per-segment clone refs. Cut each long-enough segment's
|
||||
# own reference from the vocals so the dub of each line matches the
|
||||
# prosody of its source line. Short lines fall back to the
|
||||
@@ -960,7 +1111,14 @@ async def dub_transcribe_stream(
|
||||
|
||||
|
||||
@router.post("/dub/transcribe/{job_id}")
|
||||
async def dub_transcribe(job_id: str):
|
||||
async def dub_transcribe(job_id: str, num_speakers: Optional[int] = None):
|
||||
"""Legacy synchronous transcribe (kept for the headless CLI).
|
||||
|
||||
`num_speakers` mirrors the SSE endpoint's query param (same 1–20 clamp):
|
||||
an exact speaker count forwarded to pyannote, or cycled by the silence-gap
|
||||
heuristic when pyannote is unavailable. None → auto-detect.
|
||||
"""
|
||||
num_speakers = _clamp_num_speakers(num_speakers)
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
@@ -1026,13 +1184,20 @@ async def dub_transcribe(job_id: str):
|
||||
if diar_pipe:
|
||||
try:
|
||||
diar_target = job.get("vocals_path") or job.get("audio_path")
|
||||
diarization = diar_pipe(diar_target)
|
||||
# Same hint pass-through as the SSE endpoint (#274): omit the
|
||||
# kwarg entirely when unset so we don't depend on it existing
|
||||
# in every pyannote build.
|
||||
if num_speakers:
|
||||
logger.info("Diarizing with num_speakers=%d (user hint)", num_speakers)
|
||||
diarization = diar_pipe(diar_target, num_speakers=num_speakers)
|
||||
else:
|
||||
diarization = diar_pipe(diar_target)
|
||||
segments = assign_speakers_from_diarization(segments, diarization)
|
||||
except Exception as e:
|
||||
logger.error(f"Pyannote diarization failed during inference: {e}. Falling back to heuristic.")
|
||||
segments = assign_speakers_heuristic(segments)
|
||||
segments = assign_speakers_heuristic(segments, num_speakers)
|
||||
else:
|
||||
segments = assign_speakers_heuristic(segments)
|
||||
segments = assign_speakers_heuristic(segments, num_speakers)
|
||||
|
||||
# Previously ran `segment_for_subtitles(segments)` here. Removed 2026-04-21 —
|
||||
# that splitter enforces Netflix's 17 CPS reading-speed ceiling which
|
||||
|
||||
@@ -308,14 +308,63 @@ async def dub_translate(req: TranslateRequest):
|
||||
# first is fine — the refine LLM is a separate network provider.
|
||||
return await _maybe_cinematic(translated, req, src_lang, loop)
|
||||
|
||||
# OpenAI / Ollama Local LLM Translation
|
||||
# LLM translation — resolves through the LLM Skills registry: per-skill
|
||||
# "Dub translation" override → global active provider (Settings → LLM
|
||||
# Providers). The keys users configure + test in the app now actually
|
||||
# power this engine; the raw TRANSLATE_* env vars stay working as a
|
||||
# power-user override so pre-skills setups see zero behavior change.
|
||||
if provider == "openai":
|
||||
base_url = os.environ.get("TRANSLATE_BASE_URL")
|
||||
model_name = os.environ.get("TRANSLATE_MODEL", "gpt-3.5-turbo")
|
||||
from openai import OpenAI
|
||||
# max_retries=0: a 429 + long Retry-After must not let one segment's
|
||||
# SDK call sleep+retry and blow the overall translate wall time.
|
||||
client = OpenAI(base_url=base_url, api_key=api_key or "local", max_retries=0)
|
||||
from services import llm_skills
|
||||
|
||||
llm_timeout = llm_skills._default_timeout()
|
||||
handle = None
|
||||
try:
|
||||
handle = llm_skills.resolve_skill_client("dub_translation")
|
||||
except Exception: # noqa: BLE001 — resolution must never 500 a translate
|
||||
logger.exception("dub_translation skill resolution failed; trying env fallback")
|
||||
if handle is not None:
|
||||
client = handle.client
|
||||
model_name = handle.model
|
||||
llm_timeout = handle.timeout
|
||||
# The provider-store key never touches env; resolve it so the
|
||||
# error scrubber below can redact it if a provider echoes it.
|
||||
try:
|
||||
from services import llm_providers
|
||||
api_key = llm_providers.resolve_api_key(
|
||||
llm_skills.effective_provider("dub_translation")) or api_key
|
||||
except Exception: # noqa: BLE001 — scrub-key resolution is best-effort
|
||||
pass
|
||||
elif os.environ.get("TRANSLATE_BASE_URL") or api_key:
|
||||
# Legacy env-only setup (no provider configured in-app).
|
||||
from openai import OpenAI
|
||||
# max_retries=0: a 429 + long Retry-After must not let one segment's
|
||||
# SDK call sleep+retry and blow the overall translate wall time.
|
||||
client = OpenAI(base_url=os.environ.get("TRANSLATE_BASE_URL"),
|
||||
api_key=api_key or "local", max_retries=0)
|
||||
model_name = os.environ.get("TRANSLATE_MODEL", "gpt-4o-mini")
|
||||
else:
|
||||
# Nothing configured anywhere — name the exact next step instead
|
||||
# of letting an empty key surface as a raw 401 per segment.
|
||||
try:
|
||||
reason = llm_skills.resolve_skill("dub_translation").reason
|
||||
except Exception: # noqa: BLE001
|
||||
reason = None
|
||||
if reason == "disabled":
|
||||
friendly = (
|
||||
"The LLM translation engine is turned off — enable the "
|
||||
"'Dub translation' skill in Settings → LLM Skills, or "
|
||||
"pick another engine in the Engine dropdown."
|
||||
)
|
||||
else:
|
||||
friendly = (
|
||||
"The LLM translation engine has no provider configured. "
|
||||
"Add and test one in Settings → LLM Providers (it powers "
|
||||
"this engine; route it per-skill in Settings → LLM "
|
||||
"Skills), or set TRANSLATE_BASE_URL + TRANSLATE_API_KEY "
|
||||
"+ TRANSLATE_MODEL. Or pick another engine in the "
|
||||
"Engine dropdown."
|
||||
)
|
||||
return JSONResponse(status_code=400, content={"error": friendly})
|
||||
|
||||
def _build_prompt(src_code: str, tgt_code: str) -> str:
|
||||
"""Build a system prompt that resists hallucinations on small
|
||||
@@ -377,6 +426,7 @@ async def dub_translate(req: TranslateRequest):
|
||||
res = client.chat.completions.create(
|
||||
model=model_name,
|
||||
temperature=0.2, # less drift than default 1.0
|
||||
timeout=llm_timeout, # bound per call (OMNIVOICE_LLM_TIMEOUT, 45s default)
|
||||
messages=[
|
||||
{"role": "system", "content": sys_for_attempt},
|
||||
{"role": "user", "content": seg.text},
|
||||
|
||||
@@ -24,7 +24,7 @@ from pathlib import Path
|
||||
# tests/test_app_version.py::test_all_version_files_in_lockstep and bumped by
|
||||
# release.yml's version-bump job, so it stays equal to
|
||||
# pyproject/tauri.conf/Cargo/package.json.
|
||||
_FALLBACK_VERSION = "0.3.9"
|
||||
_FALLBACK_VERSION = "0.3.10"
|
||||
|
||||
|
||||
def _fallback_version() -> str:
|
||||
|
||||
@@ -1662,7 +1662,13 @@ class _LazyASRRegistry(dict):
|
||||
|
||||
def __iter__(self):
|
||||
seen = set()
|
||||
for k in dict.__iter__(self):
|
||||
# Snapshot the live keys before yielding — see _LazyRegistry.__iter__ in
|
||||
# tts_backend.py. A concurrent lazy __getitem__ inserts into self, and
|
||||
# list_backends() runs in a FastAPI threadpool, so a *live* dict iterator
|
||||
# held open across the per-engine is_available() probes would raise
|
||||
# "dictionary changed size during iteration". list() consumes it
|
||||
# atomically under the GIL, closing the window.
|
||||
for k in list(dict.__iter__(self)):
|
||||
seen.add(k)
|
||||
yield k
|
||||
for k in self._LAZY:
|
||||
|
||||
@@ -104,9 +104,10 @@ def synthesize_chapter(
|
||||
):
|
||||
"""Render a chapter's spans to one waveform via an injected ``synth``.
|
||||
|
||||
``synth(text, voice_id, speed)`` returns a 1-D float32 audio tensor for a
|
||||
span of text in the given voice (``speed`` may be ``None`` for the engine
|
||||
default). Long spans are split with the ``chunked_tts`` splitter and
|
||||
``synth(text, voice_id, speed)`` returns a float32 audio tensor — 1-D
|
||||
``(samples,)`` or ``(channels, samples)``; real engines emit ``(1, samples)``
|
||||
per the ``TTSBackend`` contract (#897) — for a span of text in the given
|
||||
voice (``speed`` may be ``None`` for the engine default). Long spans are split with the ``chunked_tts`` splitter and
|
||||
crossfaded; inter-span ``pause_ms_after`` becomes silence. ``lexicon`` (when
|
||||
given) respells each span's text before chunking so the engine pronounces
|
||||
tricky words correctly; a ``None``/empty lexicon is a no-op pass-through.
|
||||
@@ -118,23 +119,35 @@ def synthesize_chapter(
|
||||
from services.chunked_tts import concatenate_audio_chunks, split_text_into_chunks
|
||||
from services.pronunciation import apply_lexicon
|
||||
|
||||
parts: list = []
|
||||
items: list = [] # ("a", tensor) for audio, ("s", n_samples) for silence
|
||||
for span in spans:
|
||||
if span.text:
|
||||
chunks = split_text_into_chunks(apply_lexicon(span.text, lexicon))
|
||||
rendered = [synth(c, span.voice_id, span.speed) for c in chunks]
|
||||
rendered = [r for r in rendered if r is not None and getattr(r, "numel", lambda: 0)()]
|
||||
if len(rendered) == 1:
|
||||
parts.append(rendered[0])
|
||||
items.append(("a", rendered[0]))
|
||||
elif rendered:
|
||||
parts.append(concatenate_audio_chunks(rendered, sample_rate, crossfade_ms=crossfade_ms))
|
||||
items.append(("a", concatenate_audio_chunks(rendered, sample_rate, crossfade_ms=crossfade_ms)))
|
||||
if span.pause_ms_after > 0:
|
||||
n = int(sample_rate * span.pause_ms_after / 1000.0)
|
||||
if n > 0:
|
||||
parts.append(torch.zeros(n, dtype=torch.float32))
|
||||
items.append(("s", n))
|
||||
|
||||
if not parts:
|
||||
if not items:
|
||||
return torch.zeros(0, dtype=torch.float32), 0.0
|
||||
# Engines return (1, samples) per the TTSBackend contract while a bare
|
||||
# zeros(n) is 1-D — mixing the two crashed the final concat (#897). So
|
||||
# materialize inter-span silence AFTER the loop, matching the rendered
|
||||
# audio's channel dims / dtype / device (same pattern as generation.py's
|
||||
# _render_with_pauses). A silence-only chapter stays 1-D float32 as before.
|
||||
ref = next((t for kind, t in items if kind == "a"), None)
|
||||
parts: list = [
|
||||
val if kind == "a"
|
||||
else (torch.zeros(val, dtype=torch.float32) if ref is None
|
||||
else torch.zeros(*ref.shape[:-1], val, dtype=ref.dtype, device=ref.device))
|
||||
for kind, val in items
|
||||
]
|
||||
# Hard-concat spans + silences (crossfading silence would bleed the gap).
|
||||
audio = parts[0] if len(parts) == 1 else concatenate_audio_chunks(parts, sample_rate, crossfade_ms=0)
|
||||
return audio, audio.shape[-1] / float(sample_rate)
|
||||
|
||||
@@ -183,14 +183,43 @@ def _safe_hard_cut(segment: str, max_chars: int) -> int:
|
||||
return cut
|
||||
|
||||
|
||||
def _normalize_chunk_shapes(chunks: list) -> list:
|
||||
"""Coerce mixed-rank / mixed-channel chunks to one concat-compatible shape.
|
||||
|
||||
Engines return ``(1, samples)`` per the ``TTSBackend.generate`` contract,
|
||||
but silence buffers and some model paths hand over bare ``(samples,)``
|
||||
tensors — ``torch.cat`` then dies with "Tensors must have same number of
|
||||
dimensions" (#897). Promote lower-rank chunks with leading singleton dims
|
||||
to the highest rank present, then broadcast singleton channel dims up to
|
||||
the widest channel count (mono follows stereo). Rank-homogeneous,
|
||||
channel-homogeneous input is returned untouched, so all-1-D / all-2-D
|
||||
callers keep their exact output shape; a genuine channel conflict
|
||||
(e.g. 2 vs 3 channels) still raises, which is the honest outcome.
|
||||
"""
|
||||
target = max(c.dim() for c in chunks)
|
||||
if any(c.dim() != target for c in chunks):
|
||||
promoted = []
|
||||
for c in chunks:
|
||||
while c.dim() < target:
|
||||
c = c.unsqueeze(0)
|
||||
promoted.append(c)
|
||||
chunks = promoted
|
||||
if target > 1:
|
||||
lead = tuple(max(c.shape[i] for c in chunks) for i in range(target - 1))
|
||||
chunks = [c if tuple(c.shape[:-1]) == lead else c.expand(*lead, -1)
|
||||
for c in chunks]
|
||||
return chunks
|
||||
|
||||
|
||||
def concatenate_audio_chunks(chunks: list, sample_rate: int,
|
||||
crossfade_ms: int = DEFAULT_CROSSFADE_MS):
|
||||
"""Join per-chunk waveforms with a linear crossfade on the sample axis.
|
||||
|
||||
``chunks`` are torch tensors as returned by the engine (1-D, or N-D with
|
||||
samples on the last axis — matching what ``_render_with_pauses`` handles).
|
||||
Crossfade overlap is clamped to the shorter neighbor; ``crossfade_ms=0``
|
||||
is a hard concat.
|
||||
Mixed ranks / mono-vs-multichannel chunks are normalized to one shape
|
||||
first (#897), so no producer can crash the concat. Crossfade overlap is
|
||||
clamped to the shorter neighbor; ``crossfade_ms=0`` is a hard concat.
|
||||
"""
|
||||
import torch
|
||||
|
||||
@@ -199,6 +228,7 @@ def concatenate_audio_chunks(chunks: list, sample_rate: int,
|
||||
return torch.zeros(1, dtype=torch.float32)
|
||||
if len(chunks) == 1:
|
||||
return chunks[0]
|
||||
chunks = _normalize_chunk_shapes(chunks)
|
||||
|
||||
crossfade_samples = int(sample_rate * crossfade_ms / 1000)
|
||||
result = chunks[0]
|
||||
|
||||
@@ -54,9 +54,12 @@ class LLMBackend(ABC):
|
||||
def model_name(self) -> str: ...
|
||||
|
||||
@abstractmethod
|
||||
def chat(self, *, system: str, user: str, timeout: Optional[float] = None) -> str:
|
||||
def chat(self, *, system: str, user: str, timeout: Optional[float] = None,
|
||||
temperature: Optional[float] = None) -> str:
|
||||
"""One-shot chat completion. Returns the assistant content string.
|
||||
Raises on failure — callers decide whether to fallback gracefully.
|
||||
``temperature`` is only sent to the provider when set — callers that
|
||||
leave it None keep the provider default (existing behavior).
|
||||
"""
|
||||
|
||||
|
||||
@@ -132,32 +135,43 @@ class OpenAICompatBackend(LLMBackend):
|
||||
self._client = OpenAI(max_retries=0, **kw)
|
||||
return self._client
|
||||
|
||||
def chat(self, *, system: str, user: str, timeout: Optional[float] = None) -> str:
|
||||
def chat(self, *, system: str, user: str, timeout: Optional[float] = None,
|
||||
temperature: Optional[float] = None) -> str:
|
||||
return self.chat_messages(
|
||||
messages=[
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
],
|
||||
timeout=timeout,
|
||||
temperature=temperature,
|
||||
)
|
||||
|
||||
def chat_messages(self, *, messages: list[dict], timeout: Optional[float] = None) -> str:
|
||||
def chat_messages(self, *, messages: list[dict], timeout: Optional[float] = None,
|
||||
temperature: Optional[float] = None) -> str:
|
||||
"""One-shot completion over a full message list.
|
||||
|
||||
Additive surface for callers that need structured few-shot turns
|
||||
(dictation refinement, Wave 2.1) — small local models pattern-match
|
||||
and echo inline examples, so examples must arrive as prior chat
|
||||
turns, not inside the system prompt.
|
||||
|
||||
``temperature`` is only forwarded when set (Cinematic/Autofit pin 0.2
|
||||
— the provider default of 1.0 makes local models drift and invent);
|
||||
every other caller leaves it None and keeps the provider default.
|
||||
"""
|
||||
if timeout is None:
|
||||
try:
|
||||
timeout = float(os.environ.get("OMNIVOICE_LLM_TIMEOUT", "45"))
|
||||
except ValueError:
|
||||
timeout = 45.0
|
||||
kw = {}
|
||||
if temperature is not None:
|
||||
kw["temperature"] = temperature
|
||||
res = self._get_client().chat.completions.create(
|
||||
model=self.model_name,
|
||||
timeout=timeout,
|
||||
messages=messages,
|
||||
**kw,
|
||||
)
|
||||
return (res.choices[0].message.content or "").strip()
|
||||
|
||||
|
||||
@@ -5,8 +5,10 @@ the Settings → LLM Skills panel can (a) toggle it and (b) route it to a
|
||||
specific provider (a local Ollama/LM Studio vs a remote key) instead of
|
||||
everything riding the one global active provider.
|
||||
|
||||
The five consumption points today:
|
||||
The six consumption points today:
|
||||
|
||||
dub_translation — api/routers/dub_translate.py (the Dub tab's direct
|
||||
"LLM" translation engine; provider=openai branch)
|
||||
cinematic_translation — services/translator.py (Cinematic + Autofit
|
||||
REFLECT/ADAPT rewrite; dub_translate quality gate)
|
||||
slot_fitting — services/speech_rate.py (trim/expand a line to its
|
||||
@@ -68,8 +70,9 @@ def _skill(sid: str) -> LLMSkill:
|
||||
|
||||
|
||||
# Display order in the settings panel: the dub pipeline first (translation →
|
||||
# fit → glossary → direction), then dictation.
|
||||
# refine → fit → glossary → direction), then dictation.
|
||||
_SKILLS: tuple[LLMSkill, ...] = (
|
||||
_skill("dub_translation"),
|
||||
_skill("cinematic_translation"),
|
||||
_skill("slot_fitting"),
|
||||
_skill("glossary_extract"),
|
||||
|
||||
@@ -532,14 +532,41 @@ def assign_speakers_from_turns(
|
||||
return segments
|
||||
|
||||
|
||||
def assign_speakers_heuristic(segments: List[dict]) -> List[dict]:
|
||||
"""Two-speaker alternation based on silence gaps."""
|
||||
current = 1
|
||||
def assign_speakers_heuristic(
|
||||
segments: List[dict], num_speakers: Optional[int] = None
|
||||
) -> List[dict]:
|
||||
"""Silence-gap speaker assignment (used when no diarization model runs).
|
||||
|
||||
Base signal: a gap > SPEAKER_GAP seconds between consecutive segments is
|
||||
treated as a speaker change. Without a ``num_speakers`` hint this keeps
|
||||
the legacy behavior — alternate between exactly two labels. With a hint:
|
||||
|
||||
* ``num_speakers=1`` → every segment gets ``"Speaker 1"``.
|
||||
* ``num_speakers>=2`` → labels round-robin across N speakers at each
|
||||
gap boundary, so the user's requested count is represented instead of
|
||||
being silently capped at 2.
|
||||
|
||||
Limits (be honest with callers): this honors the *count*, not voice
|
||||
identity. The rotation order is arbitrary (a returning speaker gets the
|
||||
next label in the cycle, not their own), rapid exchanges with no
|
||||
> SPEAKER_GAP pause still collapse into one label, and N is an upper
|
||||
bound — audio with fewer gap boundaries than N yields fewer labels.
|
||||
Real per-speaker attribution needs pyannote (or an inline-diarizing ASR
|
||||
backend); callers should warn the user accordingly (see dub_core).
|
||||
Invalid hints (non-int, < 1) fall back to the legacy two-speaker cycle.
|
||||
"""
|
||||
try:
|
||||
n = int(num_speakers) if num_speakers is not None else 2
|
||||
except (TypeError, ValueError):
|
||||
n = 2
|
||||
if n < 1:
|
||||
n = 2
|
||||
current = 0 # zero-based rotation index; rendered one-based below
|
||||
last_end = 0.0
|
||||
for i, s in enumerate(segments):
|
||||
if i > 0 and (s["start"] - last_end) > SPEAKER_GAP:
|
||||
current = 2 if current == 1 else 1
|
||||
s["speaker_id"] = f"Speaker {current}"
|
||||
if i > 0 and n > 1 and (s["start"] - last_end) > SPEAKER_GAP:
|
||||
current = (current + 1) % n
|
||||
s["speaker_id"] = f"Speaker {current + 1}"
|
||||
last_end = s["end"]
|
||||
return segments
|
||||
|
||||
|
||||
@@ -42,11 +42,25 @@ IDEAL_REF_DURATION_S = 8.0 # target window — long enough for prosody, short e
|
||||
# is the empirical floor below which our zero-shot clone gets unstable.
|
||||
MIN_SEGMENT_REF_DURATION_S = 3.0
|
||||
|
||||
# Clone-purity guards (speaker-hint fix): a per-speaker reference cut from
|
||||
# mislabeled or boundary-adjacent audio mixes two people's voices and the
|
||||
# resulting clone sounds "made up".
|
||||
# * A slice below MIN_SLICE_DURATION_S is too short to be a reliable
|
||||
# single-speaker sample (and diarization boundary jitter dominates it).
|
||||
# * A slice whose edges come within ADJACENT_TURN_GUARD_S of a *different*
|
||||
# speaker's turn risks bleeding that speaker's audio across the imprecise
|
||||
# boundary — deprioritized (scoring preference, not a hard filter, so
|
||||
# extraction still succeeds on dense dialogue).
|
||||
MIN_SLICE_DURATION_S = 1.5
|
||||
ADJACENT_TURN_GUARD_S = 0.3
|
||||
|
||||
|
||||
def extract_speaker_clones(
|
||||
vocals_path: str,
|
||||
segments: list[dict],
|
||||
out_dir: str,
|
||||
*,
|
||||
labels_source: str | None = None,
|
||||
) -> dict[str, dict]:
|
||||
"""Build a per-speaker reference sample from `vocals_path` + `segments`.
|
||||
|
||||
@@ -63,7 +77,20 @@ def extract_speaker_clones(
|
||||
|
||||
Speakers whose segments total < MIN_REF_DURATION_S are skipped — we'd
|
||||
rather fall back to the default TTS voice than ship a bad clone.
|
||||
|
||||
``labels_source`` records where the ``speaker_id`` labels came from
|
||||
(``"pyannote"`` | ``"turns"`` | ``"heuristic"``; ``None`` = unknown,
|
||||
treated as trusted for backward compatibility). ``"heuristic"`` labels
|
||||
are silence-gap *estimates*, not voice identity — a reference cut from
|
||||
them routinely concatenates two people's audio, so extraction is skipped
|
||||
entirely (the caller warns the user and falls back to the default voice).
|
||||
"""
|
||||
if labels_source == "heuristic":
|
||||
logger.info(
|
||||
"speaker_clone: skipping auto-clone extraction — speaker labels "
|
||||
"are gap-based heuristic estimates, not voice identity"
|
||||
)
|
||||
return {}
|
||||
if not vocals_path or not os.path.exists(vocals_path):
|
||||
logger.info("speaker_clone: no vocals track at %s; skipping", vocals_path)
|
||||
return {}
|
||||
@@ -88,7 +115,12 @@ def extract_speaker_clones(
|
||||
out: dict[str, dict] = {}
|
||||
|
||||
for speaker_id, items in by_speaker.items():
|
||||
chosen = _pick_reference_slices(items)
|
||||
chosen = _pick_reference_slices(
|
||||
items,
|
||||
speaker_id=speaker_id,
|
||||
all_segments=segments,
|
||||
labels_source=labels_source,
|
||||
)
|
||||
if not chosen:
|
||||
logger.info(
|
||||
"speaker_clone: %s has <%ss of usable audio; will fall back to default voice",
|
||||
@@ -194,31 +226,87 @@ def extract_segment_refs(
|
||||
# ── Internals ───────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _pick_reference_slices(items: list[tuple[int, dict]]) -> list[tuple[int, dict]]:
|
||||
def _adjacent_to_other_speaker(
|
||||
seg: dict, speaker_id: str, all_segments: list[dict] | None
|
||||
) -> bool:
|
||||
"""True when `seg`'s edges come within ADJACENT_TURN_GUARD_S of (or
|
||||
overlap) a segment attributed to a *different* speaker — a boundary where
|
||||
imprecise diarization timestamps risk bleeding the other voice into the
|
||||
reference slice."""
|
||||
if not all_segments:
|
||||
return False
|
||||
s0 = float(seg.get("start", 0.0))
|
||||
s1 = float(seg.get("end", 0.0))
|
||||
for other in all_segments:
|
||||
if other is seg:
|
||||
continue
|
||||
if (other.get("speaker_id") or "Speaker 1") == speaker_id:
|
||||
continue
|
||||
o0 = float(other.get("start", 0.0))
|
||||
o1 = float(other.get("end", 0.0))
|
||||
# Signed gap between the two spans; negative = overlap.
|
||||
if max(o0 - s1, s0 - o1) < ADJACENT_TURN_GUARD_S:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _pick_reference_slices(
|
||||
items: list[tuple[int, dict]],
|
||||
*,
|
||||
speaker_id: str | None = None,
|
||||
all_segments: list[dict] | None = None,
|
||||
labels_source: str | None = None,
|
||||
) -> list[tuple[int, dict]]:
|
||||
"""Select the subset of a speaker's segments to use as reference audio.
|
||||
|
||||
Strategy: take the single longest segment; if it's short, accumulate the
|
||||
next longest ones in original order until we clear IDEAL_REF_DURATION_S.
|
||||
Cap at MAX_REF_DURATION_S. Return [] if we can't reach MIN_REF_DURATION_S.
|
||||
Strategy: rank candidates clean-first (not temporally adjacent to a
|
||||
different speaker's turn — see ``_adjacent_to_other_speaker``), longest
|
||||
first within each tier, and accumulate until IDEAL_REF_DURATION_S is
|
||||
cleared. Adjacency is a scoring preference, NOT a hard filter — on dense
|
||||
dialogue where every slice borders another speaker, extraction still
|
||||
succeeds using the adjacent ones. Two hard guards protect clone purity:
|
||||
|
||||
* slices shorter than MIN_SLICE_DURATION_S are rejected outright
|
||||
(boundary jitter dominates them, so they're the likeliest to carry a
|
||||
second speaker's audio);
|
||||
* ``labels_source="heuristic"`` returns [] — gap-based labels are not
|
||||
voice identity, so no slice of them is safe to clone from.
|
||||
|
||||
Cap at MAX_REF_DURATION_S. Return [] if we can't reach
|
||||
MIN_REF_DURATION_S. When ``all_segments``/``speaker_id`` are not
|
||||
provided (legacy callers), adjacency scoring degrades to duration-only —
|
||||
the pre-guard behavior.
|
||||
"""
|
||||
if not items:
|
||||
return []
|
||||
if labels_source == "heuristic":
|
||||
return []
|
||||
if speaker_id is None:
|
||||
speaker_id = items[0][1].get("speaker_id") or "Speaker 1"
|
||||
|
||||
# Longest-first candidates. Keep original indices so we can preserve order.
|
||||
by_dur = sorted(
|
||||
def _dur(pair) -> float:
|
||||
return max(0.0, float(pair[1].get("end", 0.0)) - float(pair[1].get("start", 0.0)))
|
||||
|
||||
# Rank: clean (non-adjacent) before adjacent, longest first within each
|
||||
# tier. Keep original indices so we can restore transcript order below.
|
||||
ranked = sorted(
|
||||
items,
|
||||
key=lambda pair: (pair[1].get("end", 0.0) - pair[1].get("start", 0.0)),
|
||||
reverse=True,
|
||||
key=lambda pair: (
|
||||
_adjacent_to_other_speaker(pair[1], speaker_id, all_segments),
|
||||
-_dur(pair),
|
||||
),
|
||||
)
|
||||
|
||||
picked: list[tuple[int, dict]] = []
|
||||
total = 0.0
|
||||
for idx, seg in by_dur:
|
||||
dur = max(0.0, float(seg.get("end", 0.0)) - float(seg.get("start", 0.0)))
|
||||
if dur <= 0.0:
|
||||
for idx, seg in ranked:
|
||||
dur = _dur((idx, seg))
|
||||
if dur < MIN_SLICE_DURATION_S:
|
||||
continue
|
||||
if total + dur > MAX_REF_DURATION_S and picked:
|
||||
break
|
||||
# Ranking is no longer duration-monotonic, so a later (shorter or
|
||||
# adjacent) slice may still fit — skip, don't stop.
|
||||
continue
|
||||
picked.append((idx, seg))
|
||||
total += dur
|
||||
if total >= IDEAL_REF_DURATION_S:
|
||||
|
||||
@@ -18,6 +18,10 @@ import logging
|
||||
from typing import Iterable, Optional
|
||||
|
||||
from services.llm_backend import get_active_llm_backend, OffBackend
|
||||
# Shared LLM-output divergence guard (length window + target-script +
|
||||
# critique-echo). Lives in translator; translator never imports this module,
|
||||
# so there is no import cycle.
|
||||
from services.translator import refine_output_ok
|
||||
|
||||
logger = logging.getLogger("omnivoice.speech_rate")
|
||||
|
||||
@@ -91,9 +95,15 @@ _EXPAND_PROMPT = """\
|
||||
You are a dubbing writer. The user will give you a translated line + the exact
|
||||
time slot it must fit. The current line is TOO SHORT — add natural filler or
|
||||
gently flesh out the thought while keeping the meaning the same. Aim for a
|
||||
reading duration that matches the slot.
|
||||
reading duration that matches the slot. Never invent new information, names,
|
||||
or dialogue that is not already in the line; do not more than double the line.
|
||||
Reply with ONLY the new line. No quotes, no commentary."""
|
||||
|
||||
# Below this predicted rate ratio a line can never honestly fill its slot —
|
||||
# any LLM "expansion" that far would be fabricated dialogue. Skip the expand
|
||||
# pass entirely and keep the short line (slot-aware TTS absorbs the silence).
|
||||
_MIN_EXPANDABLE_RATIO = 0.15
|
||||
|
||||
|
||||
def adjust_for_slot(
|
||||
text: str,
|
||||
@@ -107,17 +117,36 @@ def adjust_for_slot(
|
||||
|
||||
Falls back to the input text if the LLM is off or the loop gives up.
|
||||
|
||||
``strict`` (Autofit mode) caps the accepted upper bound at 1.0 instead of
|
||||
``TOL_HIGH`` — i.e. the line must fit *within* the slot, never overrun it —
|
||||
so the target-language reading time can't exceed the segment and push the
|
||||
video timing out. A too-short line is still accepted down to ``TOL_LOW`` (we
|
||||
don't pad just to fill silence). Best-effort: after ``MAX_ATTEMPTS`` it
|
||||
returns the closest candidate seen, so a stubborn line degrades gracefully.
|
||||
``strict`` (Autofit mode) changes exactly one thing: the accepted upper
|
||||
bound is 1.0 instead of ``TOL_HIGH`` — the line must fit *within* the
|
||||
slot, never overrun it — so the target-language reading time can't exceed
|
||||
the segment and push the video timing out. Lines under ``TOL_LOW`` still
|
||||
go through the LLM expand pass in strict mode too (same as loose mode);
|
||||
padding is bounded by the divergence guard below, and a line under
|
||||
``_MIN_EXPANDABLE_RATIO`` is never expanded at all — it could only "fill"
|
||||
the slot with fabricated dialogue, so it stays short. Best-effort: after
|
||||
``MAX_ATTEMPTS`` it returns the closest candidate seen, so a stubborn line
|
||||
degrades gracefully.
|
||||
|
||||
Every LLM reply is validated with ``translator.refine_output_ok`` against
|
||||
the ORIGINAL input ``text`` (not the previous candidate — divergence
|
||||
compounds across attempts otherwise). A reply that fails the guard is
|
||||
discarded: the attempt is burned, ``current``/``best`` stay put, and if
|
||||
nothing valid ever came back the input text is returned with
|
||||
``error="fit-diverged"`` — a hallucinating model can no longer invent the
|
||||
dub line (v0.3.9 field report).
|
||||
"""
|
||||
tol_high = 1.0 if strict else TOL_HIGH
|
||||
initial_ratio = rate_ratio(text, slot_seconds, target_lang)
|
||||
if TOL_LOW <= initial_ratio <= tol_high:
|
||||
return {"text": text, "rate_ratio": initial_ratio, "attempts": 0}
|
||||
if initial_ratio < _MIN_EXPANDABLE_RATIO:
|
||||
return {
|
||||
"text": text,
|
||||
"rate_ratio": initial_ratio,
|
||||
"attempts": 0,
|
||||
"error": "fit-skip-short",
|
||||
}
|
||||
|
||||
from services import llm_skills
|
||||
# `active=` forwards this module's (monkeypatch-able) name so the
|
||||
@@ -133,6 +162,7 @@ def adjust_for_slot(
|
||||
|
||||
current = text
|
||||
best = (current, initial_ratio)
|
||||
diverged = False
|
||||
for attempt in range(1, MAX_ATTEMPTS + 1):
|
||||
r = rate_ratio(current, slot_seconds, target_lang)
|
||||
if TOL_LOW <= r <= tol_high:
|
||||
@@ -150,23 +180,43 @@ def adjust_for_slot(
|
||||
user_lines.append(f"Source line (for meaning): {source_text}")
|
||||
|
||||
try:
|
||||
next_text = llm.chat(system=system, user="\n".join(user_lines))
|
||||
next_text = llm.chat(
|
||||
system=system, user="\n".join(user_lines),
|
||||
temperature=0.2, # pinned like the Fast path — default 1.0 drifts/invents
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("speech-rate attempt %d failed: %s", attempt, e)
|
||||
return {"text": best[0], "rate_ratio": best[1], "attempts": attempt - 1, "error": str(e)}
|
||||
|
||||
if next_text and next_text.strip():
|
||||
current = next_text.strip()
|
||||
candidate = next_text.strip()
|
||||
# Divergence guard — validate against the ORIGINAL text, not
|
||||
# `current`: each accepted reply becomes the next prompt's input,
|
||||
# so per-step checks would let drift compound across attempts.
|
||||
ok, reason = refine_output_ok(text, candidate, target_lang)
|
||||
if not ok:
|
||||
diverged = True
|
||||
logger.warning(
|
||||
"speech-rate attempt %d rejected (%s) — discarding candidate",
|
||||
attempt, reason,
|
||||
)
|
||||
continue # attempt burned; current/best untouched
|
||||
current = candidate
|
||||
new_r = rate_ratio(current, slot_seconds, target_lang)
|
||||
# Keep the best candidate seen so far in case we exhaust retries.
|
||||
if abs(new_r - 1.0) < abs(best[1] - 1.0):
|
||||
best = (current, new_r)
|
||||
|
||||
return {
|
||||
out = {
|
||||
"text": best[0],
|
||||
"rate_ratio": best[1],
|
||||
"attempts": MAX_ATTEMPTS,
|
||||
}
|
||||
# Every usable reply diverged and the input text survived unchanged —
|
||||
# surface it on the row (rate_error in dub_translate, like fit-budget).
|
||||
if diverged and best[0] == text:
|
||||
out["error"] = "fit-diverged"
|
||||
return out
|
||||
|
||||
|
||||
def adjust_many(pairs: Iterable[tuple[str, float, str, Optional[str]]]) -> list[dict]:
|
||||
|
||||
@@ -92,9 +92,11 @@ REGISTRY: dict[str, dict] = {
|
||||
"category": "llm",
|
||||
"needs_key": True,
|
||||
"notes": (
|
||||
"Any OpenAI-compatible endpoint: GPT-4/5 (OpenAI), Claude (via OpenRouter), "
|
||||
"Gemini (OpenAI-compat mode), DeepSeek, Qwen, Ollama, LM Studio. "
|
||||
"Set TRANSLATE_BASE_URL + TRANSLATE_API_KEY + TRANSLATE_MODEL."
|
||||
"Uses the LLM provider you configure in Settings → LLM Providers "
|
||||
"(route it via the 'Dub translation' skill in Settings → LLM Skills): "
|
||||
"GPT (OpenAI), Claude (via OpenRouter), Gemini, DeepSeek, Qwen, "
|
||||
"Ollama, LM Studio. Power-user env override: TRANSLATE_BASE_URL + "
|
||||
"TRANSLATE_API_KEY + TRANSLATE_MODEL."
|
||||
),
|
||||
},
|
||||
}
|
||||
@@ -137,17 +139,45 @@ def install_command(engine: "str | dict | None") -> str | None:
|
||||
return f"uv pip install {pkg}" if pkg else None
|
||||
|
||||
|
||||
def _llm_configured() -> tuple[bool, "str | None"]:
|
||||
"""Whether the LLM translation engine has something to call, and via what.
|
||||
|
||||
Resolution mirrors the translate-time path in dub_translate.py: the
|
||||
"dub_translation" LLM skill (per-skill override → active provider from
|
||||
Settings → LLM Providers) first, then the TRANSLATE_* env override. Lets
|
||||
the Engine dropdown say "ready via <provider>" / "needs setup" up front
|
||||
instead of a per-segment failure after the user clicks Translate.
|
||||
"""
|
||||
try:
|
||||
from services import llm_skills
|
||||
res = llm_skills.resolve_skill("dub_translation")
|
||||
if res.ready and res.provider is not None:
|
||||
return True, res.provider.display_name
|
||||
except Exception: # noqa: BLE001 — a probe must never break list_engines()
|
||||
logger.debug("dub_translation skill probe failed", exc_info=True)
|
||||
if os.environ.get("TRANSLATE_BASE_URL") or os.environ.get("TRANSLATE_API_KEY"):
|
||||
return True, "env"
|
||||
return False, None
|
||||
|
||||
|
||||
def list_engines() -> list[dict]:
|
||||
"""Return a UI-ready list with per-engine availability stamped in."""
|
||||
out = []
|
||||
for e in REGISTRY.values():
|
||||
installed, reason = _probe(e)
|
||||
out.append({
|
||||
entry = {
|
||||
**e,
|
||||
"installed": installed,
|
||||
"availability_reason": reason,
|
||||
"install_command": install_command(e),
|
||||
})
|
||||
}
|
||||
# LLM engines additionally need a provider/key — surface configured-ness
|
||||
# so the UI can distinguish "importable" from "actually ready to call".
|
||||
if e.get("category") == "llm":
|
||||
configured, via = _llm_configured()
|
||||
entry["configured"] = configured
|
||||
entry["configured_via"] = via
|
||||
out.append(entry)
|
||||
return out
|
||||
|
||||
|
||||
|
||||
@@ -59,8 +59,9 @@ _ADAPT_PROMPT = """\
|
||||
You are a cinematic dubbing writer. Rewrite the literal translation using the
|
||||
editor's critique so it sounds natural, in-character, and fits the speaker's
|
||||
time slot. Keep meaning faithful but prefer native idiom over word-for-word
|
||||
accuracy. The output MUST be written in the same target language and script
|
||||
as the literal translation — never switch language or transliterate.
|
||||
accuracy. Never introduce facts, names, or dialogue that are not present in
|
||||
the source line. The output MUST be written in the same target language and
|
||||
script as the literal translation — never switch language or transliterate.
|
||||
Reply ONLY with the adapted translation — no quotes, no headers, no code
|
||||
fences, no commentary."""
|
||||
|
||||
@@ -93,6 +94,107 @@ def _looks_like_target_script(text: str, code: str, threshold: float = 0.5) -> b
|
||||
return (inside / len(letters)) >= threshold
|
||||
|
||||
|
||||
# ── Divergence guard (shared with speech_rate's Autofit fit pass) ────────────
|
||||
# For every Latin-script target `_looks_like_target_script` passes ANY text
|
||||
# unconditionally (no `_SCRIPT_RANGES` entry), so it was the only — and for
|
||||
# es/de/fr/… a no-op — gate on the ADAPT/fit LLM output. These checks close
|
||||
# that gap for the whole class: runaway length (hallucinated dialogue,
|
||||
# refusals, commentary) and the REFLECT critique echoed back as the "line".
|
||||
|
||||
_SHORT_REF_CHARS = 20 # below this, a length *ratio* is meaningless
|
||||
_SHORT_REF_ABS_SLACK = 120 # …use an absolute cap instead: ref + this many chars
|
||||
|
||||
|
||||
def _refine_ratio_bounds() -> tuple[float, float]:
|
||||
"""Accepted ``len(candidate)/len(reference)`` window for LLM refine output.
|
||||
Anything outside is treated as divergence and the caller degrades to its
|
||||
input text. Defaults [0.4, 2.5]; env-tunable like the cinematic budget."""
|
||||
try:
|
||||
lo = float(os.environ.get("OMNIVOICE_REFINE_RATIO_MIN", "0.4"))
|
||||
except ValueError:
|
||||
lo = 0.4
|
||||
try:
|
||||
hi = float(os.environ.get("OMNIVOICE_REFINE_RATIO_MAX", "2.5"))
|
||||
except ValueError:
|
||||
hi = 2.5
|
||||
return lo, hi
|
||||
|
||||
|
||||
def _norm_overlap_text(s: str) -> str:
|
||||
return " ".join(s.lower().split())
|
||||
|
||||
|
||||
def _echoes_critique(candidate: str, critique: str) -> bool:
|
||||
"""True when the "adaptation" is really the REFLECT critique leaking through.
|
||||
|
||||
Deterministic on purpose (no fuzzy matching): exact match after
|
||||
case/whitespace normalization; containment — the full critique inside the
|
||||
candidate always counts, the candidate inside the critique only when it
|
||||
covers most of it (critiques legitimately quote short phrases from the
|
||||
line); or >0.8 token-set overlap.
|
||||
"""
|
||||
c = _norm_overlap_text(candidate)
|
||||
k = _norm_overlap_text(critique)
|
||||
if not c or not k:
|
||||
return False
|
||||
if c == k:
|
||||
return True
|
||||
if k in c: # critique embedded in the output
|
||||
return True
|
||||
if c in k and len(c) >= 0.6 * len(k): # output ≈ a big chunk of the critique
|
||||
return True
|
||||
ct, kt = set(c.split()), set(k.split())
|
||||
union = ct | kt
|
||||
return bool(union) and len(ct & kt) / len(union) > 0.8
|
||||
|
||||
|
||||
def refine_output_ok(
|
||||
reference: str,
|
||||
candidate: str,
|
||||
target_lang: str,
|
||||
*,
|
||||
critique: str | None = None,
|
||||
max_ratio: float | None = None,
|
||||
) -> tuple[bool, str | None]:
|
||||
"""Sanity-check one LLM refine output against the text it was rewriting.
|
||||
|
||||
Shared by the Cinematic ADAPT step here and by ``speech_rate``'s Autofit
|
||||
fit pass (speech_rate imports this; translator never imports speech_rate,
|
||||
so there is no cycle). Returns ``(ok, reason)`` — ``reason`` is ``None``
|
||||
when ok, otherwise a short machine-readable tag for logs/error mapping.
|
||||
|
||||
Checks, in order:
|
||||
• script — candidate must look like the target language's script
|
||||
(``_looks_like_target_script``; Latin-script targets pass, as before);
|
||||
• length — ``len(candidate)/len(reference)`` must sit inside
|
||||
[``OMNIVOICE_REFINE_RATIO_MIN``, ``OMNIVOICE_REFINE_RATIO_MAX``]
|
||||
(default 0.4–2.5; ``max_ratio`` overrides the upper bound). References
|
||||
shorter than ~20 chars use an absolute cap (reference + 120 chars)
|
||||
instead — a two-word line legitimately doubles or halves;
|
||||
• critique echo — the candidate must not be the critique itself.
|
||||
"""
|
||||
cand = (candidate or "").strip()
|
||||
ref = (reference or "").strip()
|
||||
if not cand:
|
||||
return False, "empty"
|
||||
if not _looks_like_target_script(cand, target_lang):
|
||||
return False, f"wrong-script:{target_lang}"
|
||||
lo, hi = _refine_ratio_bounds()
|
||||
if max_ratio is not None:
|
||||
hi = max_ratio
|
||||
if ref:
|
||||
if len(ref) < _SHORT_REF_CHARS:
|
||||
if len(cand) > len(ref) + _SHORT_REF_ABS_SLACK:
|
||||
return False, f"length-abs:{len(cand)}>{len(ref)}+{_SHORT_REF_ABS_SLACK}"
|
||||
else:
|
||||
ratio = len(cand) / len(ref)
|
||||
if not (lo <= ratio <= hi):
|
||||
return False, f"length-ratio:{ratio:.2f}"
|
||||
if critique and _echoes_critique(cand, critique):
|
||||
return False, "critique-echo"
|
||||
return True, None
|
||||
|
||||
|
||||
# The LLM Skills registry entry this pipeline resolves through — lets the
|
||||
# user disable Cinematic/Autofit's LLM use or route it to a specific provider
|
||||
# (Settings → LLM Skills) independently of the other LLM features.
|
||||
@@ -173,6 +275,7 @@ def _chat(client, *, system: str, user: str) -> str:
|
||||
res = client.chat.completions.create(
|
||||
model=_llm_model(),
|
||||
timeout=_llm_timeout(),
|
||||
temperature=0.2, # pinned like the Fast path — default 1.0 drifts/invents
|
||||
messages=[
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
@@ -274,21 +377,28 @@ def cinematic_refine_sync(
|
||||
}
|
||||
|
||||
final = (adapted or "").strip() or literal_text
|
||||
# Refuse adaptations that drifted off the target script (e.g. local LLM
|
||||
# rewrote a Devanagari line in Latin/German). Caller still gets the
|
||||
# critique so the UI can show what happened, but the live text falls
|
||||
# back to the literal translation rather than corrupting the dub.
|
||||
if final is not literal_text and not _looks_like_target_script(final, target_lang):
|
||||
logger.warning(
|
||||
"cinematic adapt produced wrong-script output for %s — falling back to literal",
|
||||
target_lang,
|
||||
)
|
||||
return {
|
||||
"text": literal_text,
|
||||
"literal": literal_text,
|
||||
"critique": critique,
|
||||
"error": f"adapt-wrong-script:{target_lang}",
|
||||
}
|
||||
# Refuse adaptations that diverged from the line they were rewriting:
|
||||
# wrong script (e.g. a local LLM rewrote a Devanagari line in
|
||||
# Latin/German), runaway length (hallucinated dialogue, refusals,
|
||||
# commentary — the script check alone passes ANY text for Latin-script
|
||||
# targets), or the critique echoed back as the "adaptation". Caller still
|
||||
# gets the critique so the UI can show what happened, but the live text
|
||||
# falls back to the literal translation rather than corrupting the dub.
|
||||
if final is not literal_text:
|
||||
ok, reason = refine_output_ok(literal_text, final, target_lang, critique=critique)
|
||||
if not ok:
|
||||
logger.warning(
|
||||
"cinematic adapt diverged for %s (%s) — falling back to literal",
|
||||
target_lang, reason,
|
||||
)
|
||||
wrong_script = (reason or "").startswith("wrong-script")
|
||||
return {
|
||||
"text": literal_text,
|
||||
"literal": literal_text,
|
||||
"critique": critique,
|
||||
"error": (f"adapt-wrong-script:{target_lang}" if wrong_script
|
||||
else "adapt-diverged"),
|
||||
}
|
||||
return {
|
||||
"text": final,
|
||||
"literal": literal_text,
|
||||
|
||||
@@ -1280,7 +1280,13 @@ class _LazyRegistry(dict):
|
||||
# effect on every list_backends() call — we keep iteration light
|
||||
# and let the caller's __getitem__ trigger the import.
|
||||
seen: set[str] = set()
|
||||
for k in dict.__iter__(self):
|
||||
# Snapshot the live keys before yielding. A concurrent thread's lazy
|
||||
# __getitem__ inserts into self (self[key] = cls), and list_backends()
|
||||
# runs in a FastAPI threadpool — so holding a *live* dict iterator open
|
||||
# across the per-engine is_available() probes would raise
|
||||
# "dictionary changed size during iteration". list() consumes the
|
||||
# iterator atomically under the GIL, closing that window.
|
||||
for k in list(dict.__iter__(self)):
|
||||
seen.add(k)
|
||||
yield k
|
||||
for k in _LAZY_REGISTRY:
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
},
|
||||
"frontend": {
|
||||
"name": "omnivoice-studio",
|
||||
"version": "0.3.9",
|
||||
"version": "0.3.10",
|
||||
"dependencies": {
|
||||
"@fontsource-variable/inter": "^5.2.8",
|
||||
"@fontsource-variable/source-serif-4": "^5.2.9",
|
||||
|
||||
|
Before Width: | Height: | Size: 187 KiB After Width: | Height: | Size: 331 KiB |
|
Before Width: | Height: | Size: 134 KiB After Width: | Height: | Size: 420 KiB |
|
After Width: | Height: | Size: 342 KiB |
|
Before Width: | Height: | Size: 106 KiB After Width: | Height: | Size: 420 KiB |
|
Before Width: | Height: | Size: 358 KiB After Width: | Height: | Size: 426 KiB |
|
After Width: | Height: | Size: 278 KiB |
|
Before Width: | Height: | Size: 295 KiB After Width: | Height: | Size: 347 KiB |
|
After Width: | Height: | Size: 183 KiB |
|
After Width: | Height: | Size: 666 KiB |
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "omnivoice-studio",
|
||||
"version": "0.3.9",
|
||||
"version": "0.3.10",
|
||||
"private": true,
|
||||
"license": "AGPL-3.0-only",
|
||||
"type": "module",
|
||||
|
||||
@@ -2941,7 +2941,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "omnivoice-studio"
|
||||
version = "0.3.9"
|
||||
version = "0.3.10"
|
||||
dependencies = [
|
||||
"arboard",
|
||||
"dirs-next",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "omnivoice-studio"
|
||||
version = "0.3.9"
|
||||
version = "0.3.10"
|
||||
description = "OmniVoice Studio – AI voice cloning & dubbing desktop app"
|
||||
authors = ["Debpalash"]
|
||||
license = "AGPL-3.0-only"
|
||||
|
||||
@@ -34,11 +34,55 @@ pub fn port_in_use(port: u16) -> bool {
|
||||
pub fn backend_healthy(port: u16) -> bool {
|
||||
let url = format!("http://127.0.0.1:{}/system/info", port);
|
||||
match ureq_get_with_timeout(&url, Duration::from_millis(500)) {
|
||||
Ok(body) => body.contains("\"model_checkpoint\"") || body.contains("\"data_dir\""),
|
||||
Ok(body) => is_omnivoice_body(&body),
|
||||
Err(_) => false,
|
||||
}
|
||||
}
|
||||
|
||||
fn is_omnivoice_body(body: &str) -> bool {
|
||||
body.contains("\"model_checkpoint\"") || body.contains("\"data_dir\"")
|
||||
}
|
||||
|
||||
/// The `app_version` reported by the OmniVoice backend at :port.
|
||||
/// `None` when nothing OmniVoice answers there (port free, or a foreign
|
||||
/// process). `Some("")` when it IS our backend but predates the
|
||||
/// `app_version` field — callers treat that as stale.
|
||||
pub fn running_backend_version(port: u16) -> Option<String> {
|
||||
let url = format!("http://127.0.0.1:{}/system/info", port);
|
||||
let body = ureq_get_with_timeout(&url, Duration::from_millis(500)).ok()?;
|
||||
if !is_omnivoice_body(&body) {
|
||||
return None;
|
||||
}
|
||||
Some(parse_app_version(&body).unwrap_or_default())
|
||||
}
|
||||
|
||||
/// Extract `"app_version": "X"` from a /system/info body. String-sniff on one
|
||||
/// field (consistent with `backend_healthy`) — no JSON dependency needed.
|
||||
fn parse_app_version(body: &str) -> Option<String> {
|
||||
let key = "\"app_version\"";
|
||||
let rest = &body[body.find(key)? + key.len()..];
|
||||
let rest = rest[rest.find(':')? + 1..].trim_start();
|
||||
let rest = rest.strip_prefix('"')?;
|
||||
Some(rest[..rest.find('"')?].to_string())
|
||||
}
|
||||
|
||||
/// Whether a running backend's version matches THIS app build, comparing
|
||||
/// **base** versions (any `-N` pre-release suffix stripped from both sides) so
|
||||
/// a preview build `0.3.10-4` still attaches to its `0.3.10` backend.
|
||||
///
|
||||
/// Why this exists (the "bound port blocked the newer version" report): an
|
||||
/// orphaned backend from a *previous* version keeps answering health checks
|
||||
/// after an update, so "healthy" alone made the new UI silently attach to old
|
||||
/// backend code — every fix in the update appeared to change nothing. A
|
||||
/// version-mismatched (or unversioned) OmniVoice responder is stale by
|
||||
/// definition; callers kill it and spawn the bundled backend instead.
|
||||
pub fn same_app_version(running: &str) -> bool {
|
||||
fn base(v: &str) -> &str {
|
||||
v.split('-').next().unwrap_or(v).trim()
|
||||
}
|
||||
!running.is_empty() && base(running) == base(env!("CARGO_PKG_VERSION"))
|
||||
}
|
||||
|
||||
fn ureq_get_with_timeout(url: &str, timeout: Duration) -> Result<String, String> {
|
||||
let url = url.strip_prefix("http://").ok_or("only http:// supported")?;
|
||||
let (host_port, path) = match url.find('/') {
|
||||
@@ -365,4 +409,34 @@ mod tests {
|
||||
assert!(diag.contains("Interpreter present on disk: false"));
|
||||
assert!(diag.contains("Clean & Retry"), "must give an actionable hint");
|
||||
}
|
||||
|
||||
// ── stale-backend detection (the "bound port blocked the newer version"
|
||||
// report: a healthy orphan from a previous version must NOT be
|
||||
// attached to) ─────────────────────────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn parse_app_version_reads_system_info_shape() {
|
||||
let body = r#"{"app_version":"0.3.9","data_dir":"/x","model_checkpoint":"k2"}"#;
|
||||
assert_eq!(parse_app_version(body).as_deref(), Some("0.3.9"));
|
||||
// whitespace after the colon is fine
|
||||
assert_eq!(
|
||||
parse_app_version(r#"{ "app_version" : "1.2.3" }"#).as_deref(),
|
||||
Some("1.2.3")
|
||||
);
|
||||
// pre-app_version backends and foreign bodies yield None
|
||||
assert_eq!(parse_app_version(r#"{"data_dir":"/x"}"#), None);
|
||||
assert_eq!(parse_app_version("<html>not json</html>"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_app_version_matches_current_build_and_rejects_stale() {
|
||||
let ours = env!("CARGO_PKG_VERSION");
|
||||
assert!(same_app_version(ours), "own version must attach");
|
||||
// preview stamp of the same base still attaches
|
||||
assert!(same_app_version(&format!("{}-7", ours)));
|
||||
// a different (older) release is stale
|
||||
assert!(!same_app_version("0.0.1"));
|
||||
// unversioned (pre-app_version backend) is stale by definition
|
||||
assert!(!same_app_version(""));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -145,10 +145,29 @@ pub fn retry_bootstrap(app: tauri::AppHandle, state: tauri::State<'_, BootstrapS
|
||||
set_stage(&stage_handle, BootstrapStage::Ready);
|
||||
return;
|
||||
}
|
||||
if crate::backend::backend_healthy(backend_port()) {
|
||||
log::info!("Port {} already serving OmniVoice backend — attaching", backend_port());
|
||||
set_stage(&stage_handle, BootstrapStage::Ready);
|
||||
return;
|
||||
match crate::backend::running_backend_version(backend_port()) {
|
||||
Some(v) if crate::backend::same_app_version(&v) => {
|
||||
log::info!(
|
||||
"Port {} already serving OmniVoice backend v{} — attaching",
|
||||
backend_port(), v
|
||||
);
|
||||
set_stage(&stage_handle, BootstrapStage::Ready);
|
||||
return;
|
||||
}
|
||||
Some(v) => {
|
||||
// A healthy-but-stale backend from a previous version (the
|
||||
// classic post-update orphan). Attaching would silently run
|
||||
// OLD backend code under the new UI — replace it instead.
|
||||
log::warn!(
|
||||
"Port {} serves a stale OmniVoice backend (v{} != app v{}) — replacing it",
|
||||
backend_port(),
|
||||
if v.is_empty() { "<unknown>" } else { v.as_str() },
|
||||
env!("CARGO_PKG_VERSION"),
|
||||
);
|
||||
crate::backend::kill_orphan_on_port(backend_port());
|
||||
std::thread::sleep(Duration::from_millis(500));
|
||||
}
|
||||
None => {}
|
||||
}
|
||||
if crate::backend::port_in_use(backend_port()) {
|
||||
log::warn!("Port {} in use — taking ownership", backend_port());
|
||||
|
||||
@@ -657,13 +657,30 @@ pub fn run() {
|
||||
set_stage(&stage_handle, BootstrapStage::AwaitingSetup);
|
||||
return;
|
||||
}
|
||||
if backend::backend_healthy(backend_port()) {
|
||||
log::info!(
|
||||
"Port {} already serving OmniVoice backend — attaching",
|
||||
backend_port()
|
||||
);
|
||||
set_stage(&stage_handle, BootstrapStage::Ready);
|
||||
return;
|
||||
match backend::running_backend_version(backend_port()) {
|
||||
Some(v) if backend::same_app_version(&v) => {
|
||||
log::info!(
|
||||
"Port {} already serving OmniVoice backend v{} — attaching",
|
||||
backend_port(), v
|
||||
);
|
||||
set_stage(&stage_handle, BootstrapStage::Ready);
|
||||
return;
|
||||
}
|
||||
Some(v) => {
|
||||
// Healthy-but-stale backend from a previous version —
|
||||
// the post-update orphan that made new installs run
|
||||
// old backend code. Replace it (see backend.rs
|
||||
// same_app_version for the full story).
|
||||
log::warn!(
|
||||
"Port {} serves a stale OmniVoice backend (v{} != app v{}) — replacing it",
|
||||
backend_port(),
|
||||
if v.is_empty() { "<unknown>" } else { v.as_str() },
|
||||
env!("CARGO_PKG_VERSION"),
|
||||
);
|
||||
backend::kill_orphan_on_port(backend_port());
|
||||
std::thread::sleep(Duration::from_millis(500));
|
||||
}
|
||||
None => {}
|
||||
}
|
||||
if backend::port_in_use(backend_port()) {
|
||||
log::warn!(
|
||||
|
||||
@@ -27,11 +27,15 @@ const fmt = (t) => {
|
||||
* SegmentTrack — custom DOM segment editor lane for the dub timeline (#280).
|
||||
*
|
||||
* Replaces the WaveSurfer Regions plugin in the editing path. Renders one
|
||||
* absolutely-positioned box per segment inside a lane whose horizontal
|
||||
* position is derived from a single {pxPerSec, scrollLeft} source (read off
|
||||
* WaveSurfer's wrapper by the parent), so boxes stay pixel-aligned with the
|
||||
* waveform across zoom/scroll/resize. Virtualized by TIME — only the boxes
|
||||
* inside the visible window (+ buffer) are mounted.
|
||||
* absolutely-positioned box per segment, each placed in PURE LAYOUT from a
|
||||
* single {pxPerSec, scrollLeft} source (read off WaveSurfer's wrapper by the
|
||||
* parent), so boxes stay pixel-aligned with the waveform across
|
||||
* zoom/scroll/resize. The lane itself must NEVER be transform-animated: a
|
||||
* lane translateX'd on every playback tick gets promoted to a compositor
|
||||
* layer by Chromium, and composited semi-transparent paints flash
|
||||
* invisible/visible on some Windows GPU/WebView2 drivers (#373; #381 only
|
||||
* dampened it). Virtualized by TIME — only the boxes inside the visible
|
||||
* window (+ buffer) are mounted.
|
||||
*
|
||||
* Props:
|
||||
* segments sorted-by-start segment array (store shape)
|
||||
@@ -465,6 +469,13 @@ export default function SegmentTrack({
|
||||
const innerWidth = Math.max(viewWidth, Math.ceil(duration * pxPerSec));
|
||||
const playheadX = currentTime * pxPerSec - effScroll;
|
||||
const windowed = effSegments.slice(lo, hi);
|
||||
// Scroll offset baked into each box's `left` (viewport coordinates) instead
|
||||
// of a `translateX` on the lane — an animated lane transform is composited
|
||||
// by Chromium and flashes on some Windows GPU/WebView2 drivers (#373). In
|
||||
// the selfScroll fallback the viewport is a real scroll container, so boxes
|
||||
// stay in lane coordinates (offset 0). The virtualization window above
|
||||
// derives from the same effScroll, so both stay consistent by construction.
|
||||
const laneShift = selfScroll ? 0 : effScroll;
|
||||
|
||||
return (
|
||||
<div
|
||||
@@ -489,14 +500,16 @@ export default function SegmentTrack({
|
||||
aria-orientation="horizontal"
|
||||
title={t('timeline.keyboard_hint')}
|
||||
style={{
|
||||
width: innerWidth,
|
||||
transform: selfScroll ? undefined : `translateX(${-effScroll}px)`,
|
||||
// selfScroll needs the full content width for the native
|
||||
// scrollbar range; in synced mode the lane is a static
|
||||
// viewport-sized strip (NO transform — see laneShift above).
|
||||
width: selfScroll ? innerWidth : '100%',
|
||||
}}
|
||||
>
|
||||
{windowed.map((s) => {
|
||||
const sid = String(s.id);
|
||||
const idx = indexById.get(sid) ?? 0;
|
||||
const left = s.start * pxPerSec;
|
||||
const left = s.start * pxPerSec - laneShift;
|
||||
const width = Math.max(2, (s.end - s.start) * pxPerSec);
|
||||
const isSel = selectedId != null && String(selectedId) === sid;
|
||||
const isFocus = focusId === sid;
|
||||
|
||||
@@ -171,6 +171,46 @@ describe('SegmentTrack — keyboard', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('SegmentTrack — compositor-safe positioning (#373)', () => {
|
||||
const baseProps = {
|
||||
segments: SEGS,
|
||||
pxPerSec: 100,
|
||||
duration: 10,
|
||||
currentTime: 0,
|
||||
onsets: [],
|
||||
};
|
||||
|
||||
it('the lane never carries a transform — at rest and after a scroll update', () => {
|
||||
// Regression guard for #373: an animated `translateX` on the lane gets
|
||||
// composited by Chromium during playback and its boxes flash on some
|
||||
// Windows GPU/WebView2 drivers. Any reintroduction must fail here.
|
||||
const { rerender } = render(<SegmentTrack {...baseProps} scrollLeft={0} />);
|
||||
expect(screen.getByRole('listbox').style.transform).toBe('');
|
||||
// Simulate a WaveSurfer autoscroll tick during playback.
|
||||
rerender(<SegmentTrack {...baseProps} scrollLeft={250} />);
|
||||
expect(screen.getByRole('listbox').style.transform).toBe('');
|
||||
});
|
||||
|
||||
it('boxes are laid out in viewport coordinates: left = start·pxPerSec − scrollLeft', () => {
|
||||
setup({ scrollLeft: 250 });
|
||||
expect(box(0).style.left).toBe('-250px'); // start 0
|
||||
expect(box(1).style.left).toBe('50px'); // start 3 → 300 − 250
|
||||
expect(box(2).style.left).toBe('250px'); // start 5 → 500 − 250
|
||||
});
|
||||
|
||||
it('selfScroll fallback keeps lane coordinates — viewport scroll must not double-shift boxes', () => {
|
||||
setup({ selfScroll: true });
|
||||
const lane = screen.getByRole('listbox');
|
||||
const viewport = lane.parentElement;
|
||||
// jsdom has no layout: stub the scroll position, then fire the event the
|
||||
// component listens to.
|
||||
Object.defineProperty(viewport, 'scrollLeft', { value: 120, configurable: true });
|
||||
fireEvent.scroll(viewport);
|
||||
expect(lane.style.transform).toBe('');
|
||||
expect(box(1).style.left).toBe('300px'); // lane coords: 3s · 100px/s
|
||||
});
|
||||
});
|
||||
|
||||
describe('SegmentTrack — pointer + selection', () => {
|
||||
it('pointerdown selects the segment (table sync)', () => {
|
||||
const { onSelectSeg } = setup();
|
||||
|
||||
@@ -30,6 +30,7 @@ import WaveformPlayer from './WaveformPlayer';
|
||||
import { useAppStore } from '../store';
|
||||
import { useTranslation } from 'react-i18next';
|
||||
import { askConfirm } from '../utils/dialog';
|
||||
import { absoluteTime, timeAgo } from '../utils/relativeTime';
|
||||
|
||||
const SIDEBAR_TABS = [
|
||||
{ id: 'projects', icon: FolderOpen, accent: '#b8bb26' },
|
||||
@@ -71,20 +72,6 @@ function LazyWaveformPlayer({ height = 36, className = '', ...rest }) {
|
||||
return <div ref={holderRef} className={className} style={{ height }} aria-hidden="true" />;
|
||||
}
|
||||
|
||||
function timeAgo(ms) {
|
||||
const diff = Date.now() - ms;
|
||||
if (!isFinite(diff) || diff < 0) return '';
|
||||
const s = Math.floor(diff / 1000);
|
||||
if (s < 60) return `${s}s ago`;
|
||||
const m = Math.floor(s / 60);
|
||||
if (m < 60) return `${m}m ago`;
|
||||
const h = Math.floor(m / 60);
|
||||
if (h < 24) return `${h}h ago`;
|
||||
const d = Math.floor(h / 24);
|
||||
if (d < 7) return `${d}d ago`;
|
||||
return new Date(ms).toLocaleDateString([], { month: 'short', day: 'numeric' });
|
||||
}
|
||||
|
||||
export default function Sidebar(props) {
|
||||
const {
|
||||
availableTabs = ['projects', 'history', 'downloads'],
|
||||
@@ -311,11 +298,8 @@ export default function Sidebar(props) {
|
||||
<span className="history-kind history-kind--audio">
|
||||
<Film size={9} /> {t('sidebar.dub_label')}
|
||||
</span>
|
||||
<span
|
||||
className="history-meta"
|
||||
title={new Date(proj.updated_at * 1000).toLocaleString()}
|
||||
>
|
||||
{timeAgo(proj.updated_at * 1000)}
|
||||
<span className="history-meta" title={absoluteTime(proj.updated_at)}>
|
||||
{timeAgo(proj.updated_at)}
|
||||
</span>
|
||||
</div>
|
||||
<div className="history-title">{proj.name}</div>
|
||||
@@ -772,7 +756,7 @@ export default function Sidebar(props) {
|
||||
>
|
||||
<KindIcon size={9} /> {item.mode}
|
||||
</span>
|
||||
<span className="history-meta">{timeAgo(item.created_at * 1000)}</span>
|
||||
<span className="history-meta">{timeAgo(item.created_at)}</span>
|
||||
</div>
|
||||
<div className="history-title">{item.filename}</div>
|
||||
<div className="history-subtitle">
|
||||
|
||||
@@ -2,6 +2,7 @@ import React, { useEffect, useMemo, useState } from 'react';
|
||||
import { useTranslation } from 'react-i18next';
|
||||
import { Mic, Search, Clock, Languages } from 'lucide-react';
|
||||
import { Dialog, Input } from '../ui';
|
||||
import { toMillis } from '../utils/relativeTime';
|
||||
import { loadTranscriptions, TRANSCRIPTION_EVENT } from '../utils/transcriptionsStore';
|
||||
|
||||
/**
|
||||
@@ -27,10 +28,12 @@ export default function TranscriptionPicker({ open, onClose, onPick }) {
|
||||
}, [open]);
|
||||
|
||||
// Relative time, host-locale absolute fallback; null on unparseable timestamp.
|
||||
// toMillis keeps this unit-safe (ISO strings today; seconds/ms tolerated).
|
||||
const formatTime = (iso) => {
|
||||
const d = new Date(iso);
|
||||
if (Number.isNaN(d.getTime())) return null;
|
||||
const diff = Date.now() - d.getTime();
|
||||
const ms = toMillis(iso);
|
||||
if (ms == null) return null;
|
||||
const d = new Date(ms);
|
||||
const diff = Date.now() - ms;
|
||||
if (diff < 60000) return t('transcriptions.just_now');
|
||||
if (diff < 3600000) return t('transcriptions.m_ago', { count: Math.floor(diff / 60000) });
|
||||
if (diff < 86400000) return t('transcriptions.h_ago', { count: Math.floor(diff / 3600000) });
|
||||
|
||||
@@ -10,21 +10,7 @@ import React, { useMemo, useState } from 'react';
|
||||
import { useTranslation } from 'react-i18next';
|
||||
import { Search, Film, FolderOpen, Trash2, Save, Pencil, Check, X } from 'lucide-react';
|
||||
import { Button } from '../ui';
|
||||
|
||||
// Local copy of the sidebar's relative-time formatter (small, self-contained).
|
||||
function timeAgo(ms) {
|
||||
const diff = Date.now() - ms;
|
||||
if (!isFinite(diff) || diff < 0) return '';
|
||||
const s = Math.floor(diff / 1000);
|
||||
if (s < 60) return `${s}s ago`;
|
||||
const m = Math.floor(s / 60);
|
||||
if (m < 60) return `${m}m ago`;
|
||||
const h = Math.floor(m / 60);
|
||||
if (h < 24) return `${h}h ago`;
|
||||
const d = Math.floor(h / 24);
|
||||
if (d < 7) return `${d}d ago`;
|
||||
return new Date(ms).toLocaleDateString([], { month: 'short', day: 'numeric' });
|
||||
}
|
||||
import { absoluteTime, timeAgo } from '../utils/relativeTime';
|
||||
|
||||
export default function WorkspaceProjects({
|
||||
projects = [],
|
||||
@@ -105,11 +91,8 @@ export default function WorkspaceProjects({
|
||||
<span className="history-kind history-kind--audio">
|
||||
<Film size={9} /> {t('sidebar.dub_label')}
|
||||
</span>
|
||||
<span
|
||||
className="history-meta"
|
||||
title={new Date(proj.updated_at * 1000).toLocaleString()}
|
||||
>
|
||||
{timeAgo(proj.updated_at * 1000)}
|
||||
<span className="history-meta" title={absoluteTime(proj.updated_at)}>
|
||||
{timeAgo(proj.updated_at)}
|
||||
</span>
|
||||
</div>
|
||||
{editingId === proj.id ? (
|
||||
|
||||
@@ -370,6 +370,8 @@
|
||||
"llmskills_open_providers": "Configure providers",
|
||||
"llmskills_load_failed": "Failed to load LLM skills",
|
||||
"llmskills_save_failed": "Failed to save",
|
||||
"llmskills_dub_translation_name": "Dub translation",
|
||||
"llmskills_dub_translation_desc": "Translates dub lines directly with your configured LLM provider (the Dub tab's 'LLM' engine). Off: the LLM engine is unavailable — pick another engine.",
|
||||
"llmskills_cinematic_translation_name": "Cinematic & Autofit translation",
|
||||
"llmskills_cinematic_translation_desc": "Rewrites each translated line for natural, in-character delivery (and Autofit's time budget). Off: dubs use the Fast translation result.",
|
||||
"llmskills_slot_fitting_name": "Speech-rate slot fitting",
|
||||
|
||||
@@ -26,6 +26,7 @@ import BatchAddDialog from '../components/BatchAddDialog';
|
||||
import toast from 'react-hot-toast';
|
||||
import { toastErrorWithReport } from '../utils/errorToast';
|
||||
import { recordValueMoment } from '../utils/donationMoments';
|
||||
import { absoluteTime, timeAgo } from '../utils/relativeTime';
|
||||
|
||||
/**
|
||||
* BatchQueue — UI for the /batch/* dubbing pipeline.
|
||||
@@ -261,7 +262,7 @@ function JobCard({ job, onCancel, onDelete, t }) {
|
||||
const st = STATUS_TONE[job.status] || STATUS_TONE.queued;
|
||||
const StIcon = st.icon;
|
||||
|
||||
const ageLabel = formatAge((Date.now() / 1000 - (job.created_at || 0)) * 1000);
|
||||
const ageLabel = timeAgo(job.created_at);
|
||||
|
||||
const duration =
|
||||
job.finished_at && job.started_at ? Math.max(0, job.finished_at - job.started_at) : null;
|
||||
@@ -287,7 +288,7 @@ function JobCard({ job, onCancel, onDelete, t }) {
|
||||
<span className="batch-queue__card-spacer flex-1" />
|
||||
<span
|
||||
className="batch-queue__card-age text-[var(--text-xs)] text-fg-subtle [font-variant-numeric:tabular-nums]"
|
||||
title={new Date((job.created_at || 0) * 1000).toLocaleString()}
|
||||
title={absoluteTime(job.created_at)}
|
||||
>
|
||||
{ageLabel}
|
||||
</span>
|
||||
@@ -394,17 +395,6 @@ function JobCard({ job, onCancel, onDelete, t }) {
|
||||
);
|
||||
}
|
||||
|
||||
function formatAge(ms) {
|
||||
if (!isFinite(ms) || ms < 0) return '—';
|
||||
const s = Math.floor(ms / 1000);
|
||||
if (s < 60) return `${s}s ago`;
|
||||
const m = Math.floor(s / 60);
|
||||
if (m < 60) return `${m}m ago`;
|
||||
const h = Math.floor(m / 60);
|
||||
if (h < 24) return `${h}h ago`;
|
||||
return new Date(Date.now() - ms).toLocaleDateString();
|
||||
}
|
||||
|
||||
function formatDuration(secs) {
|
||||
if (secs < 60) return `${secs.toFixed(1)}s`;
|
||||
const m = Math.floor(secs / 60);
|
||||
|
||||
@@ -18,6 +18,7 @@ import {
|
||||
BookOpen,
|
||||
} from 'lucide-react';
|
||||
import { apiFetch } from '../api/client';
|
||||
import { timeAgo, toMillis } from '../utils/relativeTime';
|
||||
import { loadTranscriptions, TRANSCRIPTION_EVENT } from '../utils/transcriptionsStore';
|
||||
import { audioUrl } from '../api/generate';
|
||||
import { playBlobAudio } from '../utils/media';
|
||||
@@ -41,18 +42,6 @@ import { playBlobAudio } from '../utils/media';
|
||||
* this page stays in sync with the Sidebar and Launchpad automatically.
|
||||
*/
|
||||
|
||||
function fmtTime(ts) {
|
||||
if (!ts) return '';
|
||||
const d = typeof ts === 'number' ? ts : Date.parse(ts);
|
||||
if (!Number.isFinite(d)) return '';
|
||||
const diff = Date.now() - d;
|
||||
const s = Math.floor(diff / 1000);
|
||||
if (s < 60) return `${s}s ago`;
|
||||
if (s < 3600) return `${Math.floor(s / 60)}m ago`;
|
||||
if (s < 86400) return `${Math.floor(s / 3600)}h ago`;
|
||||
return `${Math.floor(s / 86400)}d ago`;
|
||||
}
|
||||
|
||||
function fmtDuration(sec) {
|
||||
if (!sec) return '';
|
||||
const n = Number(sec);
|
||||
@@ -177,7 +166,10 @@ export default function Projects({
|
||||
}, []);
|
||||
|
||||
// Normalise every source into a common shape so the filter + search +
|
||||
// sort pipeline is identical regardless of origin.
|
||||
// sort pipeline is identical regardless of origin. Timestamps arrive in
|
||||
// mixed units (backend rows: Unix seconds; story projects: Date.now() ms;
|
||||
// transcriptions: ISO strings) — toMillis() normalizes them all to epoch
|
||||
// ms (or 0 when missing) so both the sort and timeAgo() stay unit-safe.
|
||||
const items = useMemo(() => {
|
||||
const list = [];
|
||||
for (const p of studioProjects) {
|
||||
@@ -186,7 +178,7 @@ export default function Projects({
|
||||
id: p.id,
|
||||
title: p.name || p.video_path?.split('/').pop() || p.id,
|
||||
subtitle: fmtDuration(p.duration),
|
||||
ts: (p.updated_at || p.created_at || 0) * 1000,
|
||||
ts: toMillis(p.updated_at || p.created_at) ?? 0,
|
||||
accent: '#fe8019',
|
||||
Icon: Film,
|
||||
onClick: () => onOpenDub?.(p.id),
|
||||
@@ -205,7 +197,7 @@ export default function Projects({
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join(' · '),
|
||||
ts: sp.updatedAt || 0,
|
||||
ts: toMillis(sp.updatedAt) ?? 0,
|
||||
accent: '#83a598',
|
||||
Icon: BookOpen,
|
||||
onClick: () => onOpenStory?.(sp.id),
|
||||
@@ -218,7 +210,7 @@ export default function Projects({
|
||||
id: pr.id,
|
||||
title: pr.name || pr.id,
|
||||
subtitle: kind === 'design' ? t('projects.designed_voice') : t('projects.cloned_voice'),
|
||||
ts: (pr.updated_at || pr.created_at || 0) * 1000,
|
||||
ts: toMillis(pr.updated_at || pr.created_at) ?? 0,
|
||||
accent: kind === 'design' ? '#8ec07c' : '#d3869b',
|
||||
Icon: kind === 'design' ? Wand2 : Fingerprint,
|
||||
onClick: () => onOpenProfile?.(pr.id),
|
||||
@@ -230,7 +222,10 @@ export default function Projects({
|
||||
id: h.filename || h.id || String(Math.random()),
|
||||
title: (h.text || h.prompt || h.filename || t('projects.generated_audio')).slice(0, 80),
|
||||
subtitle: h.language || h.voice || '',
|
||||
ts: h.timestamp || h.created_at || 0,
|
||||
// #epoch-bug regression site: generation_history rows carry created_at
|
||||
// in Unix SECONDS — feeding them to a ms-based diff rendered every
|
||||
// history card as "20617d ago" (1970) and sorted them last.
|
||||
ts: toMillis(h.timestamp || h.created_at) ?? 0,
|
||||
accent: '#f3a5b6',
|
||||
Icon: Music,
|
||||
onClick: undefined,
|
||||
@@ -242,7 +237,7 @@ export default function Projects({
|
||||
id: e.path || e.id,
|
||||
title: e.path?.split('/').pop() || e.filename || t('projects.export'),
|
||||
subtitle: e.mode || '',
|
||||
ts: (e.created_at || 0) * 1000,
|
||||
ts: toMillis(e.created_at) ?? 0,
|
||||
accent: '#fabd2f',
|
||||
Icon: Download,
|
||||
onClick: () => e.path && onRevealExport?.(e.path),
|
||||
@@ -261,7 +256,7 @@ export default function Projects({
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join(' · '),
|
||||
ts: (j.created_at || 0) * 1000,
|
||||
ts: toMillis(j.created_at) ?? 0,
|
||||
accent: '#d3869b',
|
||||
Icon: BookMarked,
|
||||
onClick: () => j.output && playRenderInApp(audioUrl(j.output)),
|
||||
@@ -275,7 +270,7 @@ export default function Projects({
|
||||
subtitle: [tr.language, tr.duration_s ? `${Math.round(tr.duration_s)}s` : '']
|
||||
.filter(Boolean)
|
||||
.join(' · '),
|
||||
ts: tr.timestamp ? Date.parse(tr.timestamp) : 0,
|
||||
ts: toMillis(tr.timestamp) ?? 0,
|
||||
accent: '#83a598',
|
||||
Icon: FileText,
|
||||
onClick: () => {
|
||||
@@ -410,7 +405,7 @@ export default function Projects({
|
||||
trailing={
|
||||
<span className="inline-flex items-center gap-[3px] [font-family:var(--chrome-font-mono)]">
|
||||
<Clock size={10} />
|
||||
{fmtTime(it.ts)}
|
||||
{timeAgo(it.ts)}
|
||||
</span>
|
||||
}
|
||||
onClick={it.onClick}
|
||||
|
||||
@@ -12,6 +12,7 @@ import { useTranslation } from 'react-i18next';
|
||||
import { Mic, Copy, Trash2, Search, Clock, Languages, FileText, Download } from 'lucide-react';
|
||||
import { Button } from '../ui';
|
||||
import { toast } from 'react-hot-toast';
|
||||
import { toMillis } from '../utils/relativeTime';
|
||||
import {
|
||||
loadTranscriptions,
|
||||
TRANSCRIPTIONS_KEY,
|
||||
@@ -108,10 +109,13 @@ export default function TranscriptionsPage() {
|
||||
toast.success(t('transcriptions.exported'));
|
||||
}, [transcriptions]);
|
||||
|
||||
// toMillis keeps this unit-safe (ISO strings today; seconds/ms tolerated)
|
||||
// and guards unparseable stamps, which used to render "Invalid Date".
|
||||
const formatTime = (iso) => {
|
||||
const d = new Date(iso);
|
||||
const now = new Date();
|
||||
const diff = now - d;
|
||||
const ms = toMillis(iso);
|
||||
if (ms == null) return null;
|
||||
const d = new Date(ms);
|
||||
const diff = Date.now() - ms;
|
||||
if (diff < 60000) return t('transcriptions.just_now');
|
||||
if (diff < 3600000) return t('transcriptions.m_ago', { count: Math.floor(diff / 60000) });
|
||||
if (diff < 86400000) return t('transcriptions.h_ago', { count: Math.floor(diff / 3600000) });
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
import React from 'react';
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
import { render, screen } from '@testing-library/react';
|
||||
|
||||
// Regression guard for the "20617d ago" epoch bug: /history rows carry
|
||||
// created_at as Unix SECONDS (backend time.time()). Projects/OmniDrive fed
|
||||
// them to a ms-based diff, so every generation-history card rendered as
|
||||
// ~1970 ("20617d ago") and sorted to the bottom. The fix funnels every
|
||||
// timestamp through utils/relativeTime.toMillis().
|
||||
|
||||
vi.mock('../utils/media', () => ({ playBlobAudio: vi.fn() }));
|
||||
vi.mock('../api/generate', () => ({ audioUrl: (f) => `http://test.local/audio/${f}` }));
|
||||
vi.mock('../api/client', () => ({
|
||||
apiFetch: vi.fn(async () => ({ json: async () => ({ jobs: [] }) })),
|
||||
}));
|
||||
|
||||
import Projects from '../pages/Projects';
|
||||
|
||||
describe('Projects — relative timestamps (seconds-vs-ms epoch class)', () => {
|
||||
beforeEach(() => {
|
||||
localStorage.clear();
|
||||
});
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('renders a seconds created_at from today as hours ago, not "20617d ago"', async () => {
|
||||
const history = [
|
||||
{
|
||||
id: 'h1',
|
||||
filename: 'gen1.wav',
|
||||
text: 'Hello from today',
|
||||
created_at: Date.now() / 1000 - 7200, // 2h ago, in Unix SECONDS
|
||||
},
|
||||
];
|
||||
render(<Projects history={history} />);
|
||||
|
||||
expect(await screen.findByText('2h ago')).toBeInTheDocument();
|
||||
// The literal pre-fix rendering: tens of thousands of days.
|
||||
expect(screen.queryByText(/\d{3,}d ago/)).toBeNull();
|
||||
});
|
||||
|
||||
it('renders a dash for records with a missing timestamp', async () => {
|
||||
const history = [{ id: 'h2', filename: 'gen2.wav', text: 'No stamp', created_at: null }];
|
||||
render(<Projects history={history} />);
|
||||
|
||||
expect(await screen.findByText('No stamp')).toBeInTheDocument();
|
||||
expect(screen.getByText('—')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('sorts seconds-stamped records among ms-stamped ones by real recency', async () => {
|
||||
const nowS = Date.now() / 1000;
|
||||
render(
|
||||
<Projects
|
||||
history={[{ id: 'h3', filename: 'g3.wav', text: 'Newest gen', created_at: nowS - 60 }]}
|
||||
storyProjects={[{ id: 's1', name: 'Old story', updatedAt: Date.now() - 3 * 86400e3 }]}
|
||||
/>,
|
||||
);
|
||||
|
||||
const titles = (await screen.findAllByText(/Newest gen|Old story/)).map((el) => el.textContent);
|
||||
// Pre-fix, the seconds stamp sorted as ~0 and sank below the story.
|
||||
expect(titles).toEqual(['Newest gen', 'Old story']);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,88 @@
|
||||
/**
|
||||
* Shared, unit-tolerant timestamp normalization + relative-time formatting.
|
||||
*
|
||||
* Root cause of the "20617d ago" bug class: the backend stores timestamps as
|
||||
* Unix SECONDS (`time.time()` → REAL columns: generation_history.created_at,
|
||||
* dub_history, exports, longform jobs, projects…), while frontend-local
|
||||
* records carry MILLISECONDS (`Date.now()` — story projects) or ISO strings
|
||||
* (transcriptions). Any formatter that assumes one unit renders the other as
|
||||
* ~1970 ("20617d ago") or an epoch date ("Jan 1, 1970"). Every timestamp
|
||||
* render must funnel through toMillis() so no individual view can regress.
|
||||
*
|
||||
* Do NOT change the backend's stored format — existing user DBs hold seconds.
|
||||
*/
|
||||
|
||||
// Numeric timestamps below this are seconds, at/above it milliseconds.
|
||||
// 1e12 ms = Sep 2001; 1e12 s = year 33658 — unambiguous for real data.
|
||||
const MS_THRESHOLD = 1e12;
|
||||
|
||||
const SHORT_DATE_OPTS = { month: 'short', day: 'numeric' };
|
||||
|
||||
/**
|
||||
* Normalize any timestamp shape to epoch milliseconds, or null when missing/
|
||||
* unparseable. Accepts: Unix seconds (float), epoch ms, numeric strings of
|
||||
* either, ISO/date strings, Date instances. 0 / null / undefined / '' are
|
||||
* treated as "missing" (0 is this codebase's missing-timestamp sentinel —
|
||||
* never a real 1970 record).
|
||||
*/
|
||||
export function toMillis(ts) {
|
||||
if (ts == null || ts === 0 || ts === '') return null;
|
||||
if (ts instanceof Date) {
|
||||
const t = ts.getTime();
|
||||
return Number.isNaN(t) ? null : t;
|
||||
}
|
||||
if (typeof ts === 'string') {
|
||||
const trimmed = ts.trim();
|
||||
if (!trimmed) return null;
|
||||
// Numeric strings ("1751600000", "1751600000000.5") are Unix stamps,
|
||||
// not date strings — Date.parse would reject or misread them.
|
||||
const asNum = Number(trimmed);
|
||||
if (Number.isFinite(asNum)) return toMillis(asNum);
|
||||
const parsed = Date.parse(trimmed);
|
||||
return Number.isNaN(parsed) ? null : parsed;
|
||||
}
|
||||
if (typeof ts !== 'number' || !Number.isFinite(ts) || ts <= 0) return null;
|
||||
return Math.round(ts < MS_THRESHOLD ? ts * 1000 : ts);
|
||||
}
|
||||
|
||||
// Future stamps within this window are clock skew, not data corruption.
|
||||
const CLOCK_SKEW_MS = 60_000;
|
||||
|
||||
/**
|
||||
* Relative-time label for record timestamps (any unit — see toMillis).
|
||||
* - missing/unparseable → "—" (never "20617d ago" / epoch dates)
|
||||
* - future within 1 min (clock skew) → "just now"
|
||||
* - <60s → "Ns ago"; <60m → "Nm ago"; <24h → "Nh ago"; <7d → "Nd ago"
|
||||
* - older (or far-future, i.e. bad clock) → short absolute date
|
||||
*/
|
||||
export function timeAgo(ts) {
|
||||
const ms = toMillis(ts);
|
||||
if (ms == null) return '—';
|
||||
const diff = Date.now() - ms;
|
||||
if (diff < -CLOCK_SKEW_MS) return shortDate(ms);
|
||||
if (diff < 0) return 'just now';
|
||||
const s = Math.floor(diff / 1000);
|
||||
if (s < 60) return `${s}s ago`;
|
||||
const m = Math.floor(s / 60);
|
||||
if (m < 60) return `${m}m ago`;
|
||||
const h = Math.floor(m / 60);
|
||||
if (h < 24) return `${h}h ago`;
|
||||
const d = Math.floor(h / 24);
|
||||
if (d < 7) return `${d}d ago`;
|
||||
return shortDate(ms);
|
||||
}
|
||||
|
||||
/** Absolute host-locale datetime for tooltips; '' when missing/unparseable. */
|
||||
export function absoluteTime(ts) {
|
||||
const ms = toMillis(ts);
|
||||
return ms == null ? '' : new Date(ms).toLocaleString();
|
||||
}
|
||||
|
||||
function shortDate(ms) {
|
||||
const d = new Date(ms);
|
||||
const opts =
|
||||
d.getFullYear() === new Date().getFullYear()
|
||||
? SHORT_DATE_OPTS
|
||||
: { ...SHORT_DATE_OPTS, year: 'numeric' };
|
||||
return d.toLocaleDateString([], opts);
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
|
||||
import { toMillis, timeAgo, absoluteTime } from './relativeTime';
|
||||
|
||||
// Regression suite for the "20617d ago" bug class: backend rows store Unix
|
||||
// SECONDS (time.time()); formatters that assumed epoch MILLISECONDS rendered
|
||||
// every record as ~1970 ("20617d ago" ≈ 56.5 years) or an epoch date.
|
||||
|
||||
// Frozen "now": 2026-07-04T12:00:00Z.
|
||||
const NOW_MS = Date.UTC(2026, 6, 4, 12, 0, 0);
|
||||
const NOW_S = NOW_MS / 1000;
|
||||
|
||||
beforeEach(() => {
|
||||
vi.useFakeTimers();
|
||||
vi.setSystemTime(NOW_MS);
|
||||
});
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
describe('toMillis', () => {
|
||||
it('converts Unix seconds (backend time.time() floats) to ms', () => {
|
||||
expect(toMillis(NOW_S - 7200)).toBe(NOW_MS - 7200 * 1000);
|
||||
expect(toMillis(1751600000.123)).toBe(1751600000123);
|
||||
});
|
||||
|
||||
it('passes epoch milliseconds through unchanged', () => {
|
||||
expect(toMillis(NOW_MS)).toBe(NOW_MS);
|
||||
expect(toMillis(1751600000123)).toBe(1751600000123);
|
||||
});
|
||||
|
||||
it('parses ISO strings', () => {
|
||||
expect(toMillis('2026-07-04T11:00:00.000Z')).toBe(NOW_MS - 3600 * 1000);
|
||||
});
|
||||
|
||||
it('treats numeric strings as Unix stamps, not dates', () => {
|
||||
expect(toMillis(String(NOW_S))).toBe(NOW_MS);
|
||||
expect(toMillis(String(NOW_MS))).toBe(NOW_MS);
|
||||
});
|
||||
|
||||
it('accepts Date instances', () => {
|
||||
expect(toMillis(new Date(NOW_MS))).toBe(NOW_MS);
|
||||
expect(toMillis(new Date('garbage'))).toBeNull();
|
||||
});
|
||||
|
||||
it('returns null for missing/degenerate values', () => {
|
||||
for (const v of [null, undefined, 0, '', ' ', NaN, Infinity, -5, 'not a date']) {
|
||||
expect(toMillis(v)).toBeNull();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('timeAgo', () => {
|
||||
it('EPOCH REGRESSION: a seconds timestamp from today never renders as thousands of days ago', () => {
|
||||
// Pre-fix, 2h-ago-in-seconds fed to a ms diff → "20617d ago" (1970).
|
||||
const label = timeAgo(NOW_S - 7200);
|
||||
expect(label).toBe('2h ago');
|
||||
expect(label).not.toMatch(/\d{3,}d ago/);
|
||||
});
|
||||
|
||||
it('renders identical output for seconds and milliseconds inputs', () => {
|
||||
expect(timeAgo(NOW_S - 90)).toBe('1m ago');
|
||||
expect(timeAgo(NOW_MS - 90 * 1000)).toBe('1m ago');
|
||||
expect(timeAgo(NOW_S - 3 * 86400)).toBe('3d ago');
|
||||
expect(timeAgo(NOW_MS - 3 * 86400 * 1000)).toBe('3d ago');
|
||||
});
|
||||
|
||||
it('parses ISO strings', () => {
|
||||
expect(timeAgo('2026-07-04T11:59:30.000Z')).toBe('30s ago');
|
||||
});
|
||||
|
||||
it('renders a dash for null/0/undefined — never an epoch age', () => {
|
||||
expect(timeAgo(null)).toBe('—');
|
||||
expect(timeAgo(0)).toBe('—');
|
||||
expect(timeAgo(undefined)).toBe('—');
|
||||
expect(timeAgo('')).toBe('—');
|
||||
});
|
||||
|
||||
it('treats future stamps within clock skew (<1 min ahead) as "just now"', () => {
|
||||
expect(timeAgo(NOW_S + 30)).toBe('just now');
|
||||
expect(timeAgo(NOW_MS + 59 * 1000)).toBe('just now');
|
||||
});
|
||||
|
||||
it('falls back to an absolute date beyond 7 days (and for far-future stamps)', () => {
|
||||
expect(timeAgo(NOW_S - 30 * 86400)).not.toMatch(/ago/);
|
||||
expect(timeAgo(NOW_S + 3600)).not.toMatch(/ago/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('absoluteTime', () => {
|
||||
it('formats any unit and is empty for missing values (no "Jan 1, 1970" tooltips)', () => {
|
||||
expect(absoluteTime(NOW_S)).toBe(new Date(NOW_MS).toLocaleString());
|
||||
expect(absoluteTime(NOW_MS)).toBe(new Date(NOW_MS).toLocaleString());
|
||||
expect(absoluteTime(null)).toBe('');
|
||||
expect(absoluteTime(0)).toBe('');
|
||||
});
|
||||
});
|
||||
@@ -19,14 +19,26 @@ const GRID_SNAP_MAX_PX_PER_SEC = 40;
|
||||
|
||||
// Segment box palette — was WaveformTimeline's region palette; lives here so
|
||||
// both the track and any legend can share it without circular imports.
|
||||
//
|
||||
// FULLY OPAQUE by design (#373): these used to be `rgba(…, 0.45)` and relied
|
||||
// on alpha compositing over the panel behind the track — and on some Windows
|
||||
// GPU/WebView2 drivers, semi-transparent paints on the (formerly
|
||||
// transform-animated) lane flashed invisible during playback. Each entry now
|
||||
// pre-blends the same 45% tint against the surface behind the lane
|
||||
// (`--chrome-bg`, the .studio-panel background) via color-mix, which computes
|
||||
// the identical pixels (0.45·tint + 0.55·bg) with zero alpha — and stays
|
||||
// theme-aware because the variable resolves per [data-theme]. Do NOT
|
||||
// reintroduce alpha here; guarded by timeline.test.js.
|
||||
const opaqueTint = (r, g, b) =>
|
||||
`color-mix(in srgb, rgb(${r} ${g} ${b}) 45%, var(--chrome-bg, #0f1011))`;
|
||||
export const REGION_COLORS = [
|
||||
'rgba(211,134,155,0.45)',
|
||||
'rgba(131,165,152,0.45)',
|
||||
'rgba(184,187,38,0.45)',
|
||||
'rgba(250,189,47,0.45)',
|
||||
'rgba(142,192,124,0.45)',
|
||||
'rgba(254,128,25,0.45)',
|
||||
'rgba(104,157,106,0.45)',
|
||||
opaqueTint(211, 134, 155),
|
||||
opaqueTint(131, 165, 152),
|
||||
opaqueTint(184, 187, 38),
|
||||
opaqueTint(250, 189, 47),
|
||||
opaqueTint(142, 192, 124),
|
||||
opaqueTint(254, 128, 25),
|
||||
opaqueTint(104, 157, 106),
|
||||
];
|
||||
|
||||
/**
|
||||
|
||||
@@ -2,6 +2,7 @@ import { describe, it, expect } from 'vitest';
|
||||
import {
|
||||
MIN_SEG_DUR,
|
||||
MAX_OVERLAP,
|
||||
REGION_COLORS,
|
||||
visibleSegmentRange,
|
||||
snapTime,
|
||||
snapCandidates,
|
||||
@@ -225,3 +226,29 @@ describe('nearestOnset', () => {
|
||||
expect(nearestOnset(1, [])).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('REGION_COLORS — opaque paint guard (#373)', () => {
|
||||
// Semi-transparent box fills flash on some Windows GPU/WebView2 drivers
|
||||
// when the lane gets composited. Every palette entry must be fully opaque:
|
||||
// no alpha-carrying color syntax anywhere in the value.
|
||||
it('no entry carries an alpha channel', () => {
|
||||
expect(REGION_COLORS.length).toBeGreaterThan(0);
|
||||
for (const color of REGION_COLORS) {
|
||||
expect(color).not.toMatch(/rgba\(|hsla\(|transparent/i); // legacy alpha fns
|
||||
expect(color).not.toMatch(/\/\s*(?:0?\.\d+|\d+%)/); // modern `… / alpha` syntax
|
||||
for (const hex of color.match(/#[0-9a-fA-F]+/g) ?? []) {
|
||||
expect([4, 7]).toContain(hex.length); // #rgb / #rrggbb only — no alpha digits
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it('color-mix mixes only opaque inputs and preserves the original 45% tint ratio', () => {
|
||||
for (const color of REGION_COLORS) {
|
||||
const m = color.match(
|
||||
/^color-mix\(in srgb, rgb\(\d+ \d+ \d+\) (\d+)%, var\(--chrome-bg, (#[0-9a-fA-F]{6})\)\)$/,
|
||||
);
|
||||
expect(m, `unexpected palette entry shape: ${color}`).not.toBeNull();
|
||||
expect(m[1]).toBe('45'); // same visual weight the 0.45-alpha fills had
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -115,6 +115,11 @@ def main(argv=None):
|
||||
ap.add_argument("--target", required=True, help="Target language ISO code (e.g. de, ja, es).")
|
||||
ap.add_argument("--voice", help="Voice profile ID to apply to every segment.")
|
||||
ap.add_argument("--quality", choices=("fast", "cinematic"), default="fast")
|
||||
ap.add_argument(
|
||||
"--speakers", type=int, default=None, metavar="N",
|
||||
help="Exact number of speakers in the source (1-20). Forwarded to "
|
||||
"diarization so distinct voices don't blend; omit to auto-detect.",
|
||||
)
|
||||
ap.add_argument("--glossary", help="Path to a JSON glossary: [{source,target,note}]")
|
||||
ap.add_argument("--api", default=os.environ.get("OMNIVOICE_API", "http://localhost:8000"))
|
||||
args = ap.parse_args(argv)
|
||||
@@ -155,7 +160,8 @@ def main(argv=None):
|
||||
|
||||
# 3. Transcribe (sync).
|
||||
_log("→ transcribing…")
|
||||
tx = _post(api, f"/dub/transcribe/{job_id}", {})
|
||||
speakers_q = f"?num_speakers={args.speakers}" if args.speakers else ""
|
||||
tx = _post(api, f"/dub/transcribe/{job_id}{speakers_q}", {})
|
||||
segs = tx.get("segments", [])
|
||||
_log(f" ✓ {len(segs)} segment(s), source={tx.get('source_lang')}")
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "omnivoice"
|
||||
version = "0.3.9"
|
||||
version = "0.3.10"
|
||||
description = "OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models"
|
||||
readme = "README.md"
|
||||
# Free and open-source under the GNU Affero General Public License v3 (see
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
"""Regression: the lazy TTS/ASR registries must not raise "dictionary changed
|
||||
size during iteration" when a lazy ``__getitem__`` inserts a resolved key while
|
||||
another caller iterates — the ``/engines`` 500 seen in production logs.
|
||||
|
||||
FastAPI runs ``list_backends()`` in a threadpool, so two concurrent ``/engines``
|
||||
requests race: one iterates ``_REGISTRY.items()`` (which held a *live* dict
|
||||
iterator open across the slow per-engine ``is_available()`` probes) while the
|
||||
other materializes the lazy entry via ``__getitem__`` (``self[key] = cls``).
|
||||
The insert then tripped the open iterator. ``__iter__`` now snapshots the live
|
||||
keys up front (``list(dict.__iter__(self))``, atomic under the GIL), so a
|
||||
concurrent insert can no longer trip the iteration.
|
||||
|
||||
The tests drive the exact crash site (``__iter__``) deterministically: begin
|
||||
iterating, insert mid-iteration, then drain. Pre-fix this raises on the drain;
|
||||
post-fix it completes.
|
||||
"""
|
||||
|
||||
|
||||
def _assert_iter_survives_concurrent_insert(reg):
|
||||
it = iter(reg) # the generator items()/list_backends() drives
|
||||
first = next(it) # first yield → the real-key snapshot is taken here
|
||||
reg["zzz-concurrent-insert"] = object() # a concurrent lazy insert, mid-iteration
|
||||
drained = [first, *it] # must NOT raise "dictionary changed size during iteration"
|
||||
assert first in drained
|
||||
|
||||
|
||||
def test_tts_lazy_registry_iter_survives_concurrent_insert():
|
||||
from services.tts_backend import _LazyRegistry
|
||||
|
||||
reg = _LazyRegistry({"omnivoice": object(), "b": object(), "c": object()})
|
||||
_assert_iter_survives_concurrent_insert(reg)
|
||||
|
||||
|
||||
def test_asr_lazy_registry_iter_survives_concurrent_insert():
|
||||
from services.asr_backend import _LazyASRRegistry
|
||||
|
||||
reg = _LazyASRRegistry({"whisperx": object(), "faster-whisper": object()})
|
||||
_assert_iter_survives_concurrent_insert(reg)
|
||||
@@ -152,6 +152,45 @@ def test_synthesize_empty_spans_is_silent():
|
||||
assert dur == 0.0
|
||||
|
||||
|
||||
def test_synthesize_chapter_2d_engine_output_with_pause():
|
||||
# #897 regression: real engines return (1, samples) (TTSBackend contract)
|
||||
# while inter-span pause silence was built as 1-D zeros — the final hard
|
||||
# concat crashed a longform chapter render with
|
||||
# "RuntimeError: Tensors must have same number of dimensions: got 1 and 2".
|
||||
torch = pytest.importorskip("torch")
|
||||
sr = 16000
|
||||
|
||||
def synth(text, voice_id, speed=None):
|
||||
return torch.ones(1, 1000, dtype=torch.float32)
|
||||
|
||||
plan = parse_audiobook_script("First. [pause 1s] Second.", default_voice="v")
|
||||
audio, dur = synthesize_chapter(plan.chapters[0].spans, synth, sr)
|
||||
assert audio.shape == (1, 1000 + sr + 1000)
|
||||
assert dur == pytest.approx((2000 + sr) / sr)
|
||||
|
||||
|
||||
def test_synthesize_chapter_silence_rank_matches_engine_output(monkeypatch):
|
||||
# The producer half of the #897 fix: pause silence is materialized with
|
||||
# the rendered audio's rank, so the parts reaching the final concat are
|
||||
# rank-homogeneous even without concatenate_audio_chunks' normalization.
|
||||
torch = pytest.importorskip("torch")
|
||||
import services.chunked_tts as ct
|
||||
|
||||
seen: list[list[int]] = []
|
||||
real = ct.concatenate_audio_chunks
|
||||
|
||||
def spy(chunks, sr, crossfade_ms=50):
|
||||
seen.append([c.dim() for c in chunks])
|
||||
return real(chunks, sr, crossfade_ms=crossfade_ms)
|
||||
|
||||
monkeypatch.setattr(ct, "concatenate_audio_chunks", spy)
|
||||
plan = parse_audiobook_script("First. [pause 500ms] Second.", default_voice="v")
|
||||
synthesize_chapter(plan.chapters[0].spans,
|
||||
lambda t, v, s=None: torch.ones(1, 100), 16000)
|
||||
assert seen, "the stitcher never reached the concat"
|
||||
assert all(d == 2 for dims in seen for d in dims)
|
||||
|
||||
|
||||
def test_parse_applies_ssml_lite_prosody():
|
||||
plan = parse_audiobook_script("[slow]hush[/slow] normal [spell]USA[/spell]")
|
||||
spans = plan.chapters[0].spans
|
||||
|
||||
@@ -14,10 +14,23 @@ import pytest
|
||||
class _FakeTrimmer:
|
||||
"""Minimal non-Off LLM stand-in that trims any line to a short fixed length
|
||||
so the fit loop converges deterministically."""
|
||||
def chat(self, *, system, user, timeout=None):
|
||||
def chat(self, *, system, user, timeout=None, temperature=None):
|
||||
return "A" * 14 # ratio 0.93 for slot 1.0s @ en 15 cps → inside [0.92, 1.0]
|
||||
|
||||
|
||||
class _FakeChatter:
|
||||
"""Non-Off LLM stand-in that always returns one fixed reply; counts calls."""
|
||||
def __init__(self, reply: str):
|
||||
self.reply = reply
|
||||
self.calls = 0
|
||||
self.last_temperature = None
|
||||
|
||||
def chat(self, *, system, user, timeout=None, temperature=None):
|
||||
self.calls += 1
|
||||
self.last_temperature = temperature
|
||||
return self.reply
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def speech_rate(monkeypatch):
|
||||
from services import speech_rate as _sr
|
||||
@@ -53,3 +66,66 @@ def test_strict_within_tolerance_is_noop(speech_rate):
|
||||
res = speech_rate.adjust_for_slot("A" * 15, slot_seconds=1.0, target_lang="en", strict=True)
|
||||
assert res["attempts"] == 0
|
||||
assert res["rate_ratio"] == pytest.approx(1.0, abs=0.01)
|
||||
|
||||
|
||||
# ── Divergence guard (v0.3.9 field report: Autofit invented dialogue) ───────
|
||||
|
||||
|
||||
def test_expand_hallucination_rejected_output_stays_input(monkeypatch):
|
||||
"""The reported bug: a short line over a long slot invited the LLM to
|
||||
fabricate a slot-filling wall of dialogue, and the accept step took ANY
|
||||
non-empty reply (the best-tracker then favored the most-padded one). The
|
||||
guard must discard it and keep the input text."""
|
||||
from services import speech_rate as _sr
|
||||
text = "B" * 24 # 1.6s @ en 15cps over a 10s slot → ratio 0.16
|
||||
fake = _FakeChatter("A" * 145) # a "perfect" slot fill = ~6× the input line
|
||||
monkeypatch.setattr(_sr, "get_active_llm_backend", lambda: fake)
|
||||
res = _sr.adjust_for_slot(text, slot_seconds=10.0, target_lang="en", strict=True)
|
||||
assert res["text"] == text # hallucination never adopted
|
||||
assert res.get("error") == "fit-diverged"
|
||||
assert fake.calls == _sr.MAX_ATTEMPTS # attempts burned, loop still bounded
|
||||
|
||||
|
||||
def test_refusal_reply_rejected_keeps_input(monkeypatch):
|
||||
"""A refusal/commentary reply must not replace a long line just because
|
||||
its length happens to land near the slot budget."""
|
||||
from services import speech_rate as _sr
|
||||
text = "C" * 60 # 4s over a 1s slot → heavy trim ask
|
||||
fake = _FakeChatter("Sorry, I cannot.") # 16 chars ≈ the 15-char slot budget
|
||||
monkeypatch.setattr(_sr, "get_active_llm_backend", lambda: fake)
|
||||
res = _sr.adjust_for_slot(text, slot_seconds=1.0, target_lang="en")
|
||||
assert res["text"] == text
|
||||
assert res.get("error") == "fit-diverged"
|
||||
|
||||
|
||||
def test_legitimate_expansion_accepted(monkeypatch):
|
||||
from services import speech_rate as _sr
|
||||
text = "D" * 40 # ratio 0.67 over a 4s slot @ en
|
||||
fake = _FakeChatter("E" * 57) # ~1.4× the line → ratio 0.95, honest fill
|
||||
monkeypatch.setattr(_sr, "get_active_llm_backend", lambda: fake)
|
||||
res = _sr.adjust_for_slot(text, slot_seconds=4.0, target_lang="en", strict=True)
|
||||
assert res["text"] == "E" * 57
|
||||
assert "error" not in res
|
||||
|
||||
|
||||
def test_tiny_line_skips_llm_expansion(monkeypatch):
|
||||
"""A line under 15% of its slot can never honestly fill it — the LLM must
|
||||
not even be asked (it could only fabricate), and the line stays short."""
|
||||
from services import speech_rate as _sr
|
||||
fake = _FakeChatter("A" * 150)
|
||||
monkeypatch.setattr(_sr, "get_active_llm_backend", lambda: fake)
|
||||
res = _sr.adjust_for_slot("Hi.", slot_seconds=10.0, target_lang="en", strict=True)
|
||||
assert res["text"] == "Hi."
|
||||
assert res["attempts"] == 0
|
||||
assert res.get("error") == "fit-skip-short"
|
||||
assert fake.calls == 0
|
||||
|
||||
|
||||
def test_fit_llm_call_pins_low_temperature(monkeypatch):
|
||||
"""The fit pass pins temperature=0.2 like the Fast translate path — the
|
||||
provider default of 1.0 is part of what made expansions drift."""
|
||||
from services import speech_rate as _sr
|
||||
fake = _FakeChatter("A" * 14)
|
||||
monkeypatch.setattr(_sr, "get_active_llm_backend", lambda: fake)
|
||||
_sr.adjust_for_slot("A" * 16, slot_seconds=1.0, target_lang="en", strict=True)
|
||||
assert fake.last_temperature == 0.2
|
||||
|
||||
@@ -133,6 +133,52 @@ def test_single_and_empty_chunk_lists():
|
||||
assert concatenate_audio_chunks([], sr).shape[-1] == 1 # silent guard
|
||||
|
||||
|
||||
# ── Mixed-rank chunks (#897) ─────────────────────────────────────────────────
|
||||
# Engines emit (1, samples) per the TTSBackend contract while silence buffers
|
||||
# were bare (samples,) — torch.cat crashed longform chapter renders with
|
||||
# "RuntimeError: Tensors must have same number of dimensions: got 1 and 2".
|
||||
|
||||
def test_mixed_rank_chunks_hard_cut():
|
||||
out = concatenate_audio_chunks([torch.ones(1, 300), torch.zeros(200)],
|
||||
24000, crossfade_ms=0)
|
||||
assert out.shape == (1, 500)
|
||||
|
||||
|
||||
def test_mixed_rank_chunks_hard_cut_1d_first():
|
||||
out = concatenate_audio_chunks([torch.zeros(200), torch.ones(1, 300)],
|
||||
24000, crossfade_ms=0)
|
||||
assert out.shape == (1, 500)
|
||||
|
||||
|
||||
def test_mixed_rank_chunks_crossfade():
|
||||
sr = 1000
|
||||
out = concatenate_audio_chunks([torch.ones(1, 500), torch.zeros(500)],
|
||||
sr, crossfade_ms=100) # 100-sample overlap
|
||||
assert out.shape == (1, 900)
|
||||
|
||||
|
||||
def test_mono_chunk_broadcasts_to_stereo():
|
||||
# Honest channel handling: a mono chunk follows the widest channel count.
|
||||
out = concatenate_audio_chunks([torch.ones(2, 100), torch.zeros(1, 50)],
|
||||
24000, crossfade_ms=0)
|
||||
assert out.shape == (2, 150)
|
||||
out = concatenate_audio_chunks([torch.ones(2, 100), torch.zeros(60)],
|
||||
24000, crossfade_ms=0) # 1-D mono, too
|
||||
assert out.shape == (2, 160)
|
||||
|
||||
|
||||
def test_all_1d_output_stays_1d():
|
||||
out = concatenate_audio_chunks([torch.ones(100), torch.ones(100)],
|
||||
24000, crossfade_ms=0)
|
||||
assert out.dim() == 1 and out.shape[-1] == 200
|
||||
|
||||
|
||||
def test_all_2d_output_stays_2d():
|
||||
out = concatenate_audio_chunks([torch.ones(1, 100), torch.ones(1, 100)],
|
||||
24000, crossfade_ms=0)
|
||||
assert out.shape == (1, 200)
|
||||
|
||||
|
||||
# ── Endpoint integration (stubbed engine, pattern from test_generate_engine) ─
|
||||
|
||||
def _tts_mod():
|
||||
|
||||
@@ -87,10 +87,13 @@ def test_cinematic_refine_sync_injects_dialect_into_prompts(monkeypatch):
|
||||
from services import translator
|
||||
|
||||
captured_systems = []
|
||||
# Distinct reflect/adapt replies — an adapt identical to the critique is
|
||||
# (correctly) rejected by the divergence guard as a critique echo.
|
||||
replies = iter(["usa voseo rioplatense", "vos sos muy listo"])
|
||||
|
||||
def fake_chat(client, *, system, user):
|
||||
captured_systems.append(system)
|
||||
return "vos sos muy listo"
|
||||
return next(replies)
|
||||
|
||||
monkeypatch.setattr(translator, "_llm_client", lambda: object())
|
||||
monkeypatch.setattr(translator, "_chat", fake_chat)
|
||||
|
||||
@@ -326,12 +326,21 @@ async def test_argos_fast_stamps_rate_ratio(monkeypatch):
|
||||
|
||||
def _install_fake_openai(monkeypatch, *, content="hola mundo", raises=None):
|
||||
"""Register a fake `openai.OpenAI` whose chat.completions.create returns
|
||||
`content` (or raises `raises`). Accepts the max_retries kwarg the code adds."""
|
||||
`content` (or raises `raises`). Accepts the max_retries kwarg the code adds.
|
||||
|
||||
Also sets TRANSLATE_API_KEY so provider="openai" requests deterministically
|
||||
take the legacy env-fallback branch — since the provider-wiring fix, a fully
|
||||
unconfigured LLM engine 400s up front instead of reaching a client (the
|
||||
skills-resolved path has its own tests below). Returns the recorded
|
||||
chat.completions.create kwargs for call-shape assertions."""
|
||||
import sys
|
||||
import types
|
||||
|
||||
calls = []
|
||||
|
||||
class _Completions:
|
||||
def create(self, **kw):
|
||||
calls.append(kw)
|
||||
if raises is not None:
|
||||
raise raises
|
||||
msg = type("M", (), {"content": content})
|
||||
@@ -349,6 +358,8 @@ def _install_fake_openai(monkeypatch, *, content="hola mundo", raises=None):
|
||||
mod = types.ModuleType("openai")
|
||||
mod.OpenAI = _FakeClient
|
||||
monkeypatch.setitem(sys.modules, "openai", mod)
|
||||
monkeypatch.setenv("TRANSLATE_API_KEY", "sk-test-env")
|
||||
return calls
|
||||
|
||||
|
||||
# ── P1: the Autofit fit pass must be bounded by the cinematic wall-clock ─────
|
||||
@@ -414,3 +425,123 @@ async def test_openai_segment_error_is_scrubbed(monkeypatch):
|
||||
assert "sk-LEAKLEAKLEAKLEAKLEAK12345" not in seg["error"]
|
||||
assert "/Users/bob" not in seg["error"]
|
||||
assert "***REDACTED***" in seg["error"]
|
||||
|
||||
|
||||
# ── provider="openai" resolves through the LLM Providers/Skills system ───────
|
||||
# The engine used to read only the TRANSLATE_* env vars, so a provider the user
|
||||
# configured + tested in Settings → LLM Providers silently didn't power it.
|
||||
|
||||
|
||||
class _RecordingLLMClient:
|
||||
"""Minimal OpenAI-compatible fake that records create() kwargs."""
|
||||
|
||||
def __init__(self, content):
|
||||
self.calls = []
|
||||
outer = self
|
||||
|
||||
class _Completions:
|
||||
def create(self, **kw):
|
||||
outer.calls.append(kw)
|
||||
msg = type("M", (), {"content": content})
|
||||
choice = type("C", (), {"message": msg})
|
||||
return type("R", (), {"choices": [choice]})
|
||||
|
||||
self.chat = type("Chat", (), {"completions": _Completions()})()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openai_uses_provider_configured_in_settings(monkeypatch):
|
||||
"""A ready dub_translation skill (Settings → LLM Providers) powers the
|
||||
engine — its client, its model, its timeout — with no env vars set."""
|
||||
import types
|
||||
from api.routers import dub_translate
|
||||
from schemas.requests import TranslateRequest, TranslateSegment
|
||||
from services import llm_skills
|
||||
|
||||
for var in ("TRANSLATE_API_KEY", "TRANSLATE_BASE_URL", "TRANSLATE_MODEL"):
|
||||
monkeypatch.delenv(var, raising=False)
|
||||
|
||||
fake = _RecordingLLMClient("hallo welt")
|
||||
handle = types.SimpleNamespace(
|
||||
client=fake, model="provider-model", provider_id="groq", timeout=7.0)
|
||||
monkeypatch.setattr(llm_skills, "resolve_skill_client", lambda sid: handle)
|
||||
|
||||
req = TranslateRequest(
|
||||
segments=[TranslateSegment(id="s1", text="Hello")],
|
||||
target_lang="de", provider="openai", source_lang="en",
|
||||
)
|
||||
resp = await dub_translate.dub_translate(req)
|
||||
assert resp["translated"][0]["text"] == "hallo welt"
|
||||
assert fake.calls, "the skills-resolved client was not used"
|
||||
assert fake.calls[0]["model"] == "provider-model"
|
||||
assert fake.calls[0]["timeout"] == 7.0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openai_unconfigured_400_names_llm_providers(monkeypatch):
|
||||
"""Nothing configured anywhere → an up-front actionable 400 pointing at
|
||||
Settings → LLM Providers, not a raw per-segment 401."""
|
||||
import types
|
||||
from api.routers import dub_translate
|
||||
from schemas.requests import TranslateRequest, TranslateSegment
|
||||
from services import llm_skills
|
||||
|
||||
for var in ("TRANSLATE_API_KEY", "TRANSLATE_BASE_URL", "TRANSLATE_MODEL"):
|
||||
monkeypatch.delenv(var, raising=False)
|
||||
monkeypatch.setattr(llm_skills, "resolve_skill_client", lambda sid: None)
|
||||
monkeypatch.setattr(
|
||||
llm_skills, "resolve_skill",
|
||||
lambda sid: types.SimpleNamespace(reason="no_provider"))
|
||||
|
||||
req = TranslateRequest(
|
||||
segments=[TranslateSegment(id="s1", text="Hello")],
|
||||
target_lang="es", provider="openai", source_lang="en",
|
||||
)
|
||||
resp = await dub_translate.dub_translate(req)
|
||||
assert resp.status_code == 400
|
||||
assert b"LLM Providers" in resp.body
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openai_disabled_skill_400_names_llm_skills(monkeypatch):
|
||||
"""A deliberately disabled dub_translation skill names the Skills page."""
|
||||
import types
|
||||
from api.routers import dub_translate
|
||||
from schemas.requests import TranslateRequest, TranslateSegment
|
||||
from services import llm_skills
|
||||
|
||||
for var in ("TRANSLATE_API_KEY", "TRANSLATE_BASE_URL", "TRANSLATE_MODEL"):
|
||||
monkeypatch.delenv(var, raising=False)
|
||||
monkeypatch.setattr(llm_skills, "resolve_skill_client", lambda sid: None)
|
||||
monkeypatch.setattr(
|
||||
llm_skills, "resolve_skill",
|
||||
lambda sid: types.SimpleNamespace(reason="disabled"))
|
||||
|
||||
req = TranslateRequest(
|
||||
segments=[TranslateSegment(id="s1", text="Hello")],
|
||||
target_lang="es", provider="openai", source_lang="en",
|
||||
)
|
||||
resp = await dub_translate.dub_translate(req)
|
||||
assert resp.status_code == 400
|
||||
assert b"LLM Skills" in resp.body
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openai_env_fallback_still_works(monkeypatch):
|
||||
"""Legacy env-only setups (no provider in the app) keep working unchanged,
|
||||
including TRANSLATE_MODEL selection."""
|
||||
from api.routers import dub_translate
|
||||
from schemas.requests import TranslateRequest, TranslateSegment
|
||||
from services import llm_skills
|
||||
|
||||
calls = _install_fake_openai(monkeypatch, content="hola mundo")
|
||||
monkeypatch.setenv("TRANSLATE_MODEL", "env-model")
|
||||
monkeypatch.setattr(llm_skills, "resolve_skill_client", lambda sid: None)
|
||||
|
||||
req = TranslateRequest(
|
||||
segments=[TranslateSegment(id="s1", text="Hello")],
|
||||
target_lang="es", provider="openai", source_lang="en",
|
||||
)
|
||||
resp = await dub_translate.dub_translate(req)
|
||||
assert resp["translated"][0]["text"] == "hola mundo"
|
||||
assert calls and calls[0]["model"] == "env-model"
|
||||
|
||||
@@ -62,8 +62,8 @@ def _activate_groq(store):
|
||||
|
||||
def test_all_skills_cover_every_consumption_point(skills):
|
||||
assert [s.id for s in skills.all_skills()] == [
|
||||
"cinematic_translation", "slot_fitting", "glossary_extract",
|
||||
"direction_parse", "dictation_refinement",
|
||||
"dub_translation", "cinematic_translation", "slot_fitting",
|
||||
"glossary_extract", "direction_parse", "dictation_refinement",
|
||||
]
|
||||
for s in skills.all_skills():
|
||||
assert s.name_key == f"settings.llmskills_{s.id}_name"
|
||||
@@ -255,9 +255,11 @@ def test_disabled_slot_fitting_returns_no_llm_marker(skills, store, monkeypatch)
|
||||
class _Fake:
|
||||
id = "openai-compat"
|
||||
def chat(self, **kw):
|
||||
return "short"
|
||||
# A plausible trim (0.93 ratio, within the divergence guard's
|
||||
# length window vs the 30-char input) so the enabled path converges.
|
||||
return "x" * 14
|
||||
monkeypatch.setattr(speech_rate, "get_active_llm_backend", lambda: _Fake())
|
||||
long_text = "x" * 400 # far over any slot → forces the LLM branch
|
||||
long_text = "x" * 30 # 2× over the slot → forces the LLM branch
|
||||
assert "error" not in speech_rate.adjust_for_slot(
|
||||
long_text, slot_seconds=1.0, target_lang="en")
|
||||
skills.configure_skill("slot_fitting", enabled=False)
|
||||
|
||||
@@ -295,6 +295,54 @@ class TestSpeakerAssignment:
|
||||
assert out[0]["speaker_id"] == out[1]["speaker_id"]
|
||||
assert out[2]["speaker_id"] != out[1]["speaker_id"]
|
||||
|
||||
@staticmethod
|
||||
def _gapped_segs(n, dur=2.0, gap=2.0):
|
||||
"""n segments, each separated by a > SPEAKER_GAP silence."""
|
||||
segs = []
|
||||
t = 0.0
|
||||
for i in range(n):
|
||||
segs.append({"start": t, "end": t + dur, "text": f"s{i}", "id": str(i)})
|
||||
t += dur + gap
|
||||
return segs
|
||||
|
||||
def test_heuristic_hint_three_speakers_cycles_three_labels(self):
|
||||
# Speaker-hint fix: num_speakers=3 must yield 3 distinct labels on
|
||||
# alternating-gap audio. Pre-fix the heuristic hardcoded 2 speakers
|
||||
# and silently ignored the hint.
|
||||
out = assign_speakers_heuristic(self._gapped_segs(6), num_speakers=3)
|
||||
labels = [s["speaker_id"] for s in out]
|
||||
assert labels == [
|
||||
"Speaker 1", "Speaker 2", "Speaker 3",
|
||||
"Speaker 1", "Speaker 2", "Speaker 3",
|
||||
]
|
||||
assert len(set(labels)) == 3
|
||||
|
||||
def test_heuristic_hint_one_speaker_single_label(self):
|
||||
out = assign_speakers_heuristic(self._gapped_segs(4), num_speakers=1)
|
||||
assert {s["speaker_id"] for s in out} == {"Speaker 1"}
|
||||
|
||||
def test_heuristic_none_hint_preserves_legacy_two_speaker_alternation(self):
|
||||
out = assign_speakers_heuristic(self._gapped_segs(4), num_speakers=None)
|
||||
labels = [s["speaker_id"] for s in out]
|
||||
assert labels == ["Speaker 1", "Speaker 2", "Speaker 1", "Speaker 2"]
|
||||
|
||||
@pytest.mark.parametrize("bad", [0, -3, "not-a-number"])
|
||||
def test_heuristic_invalid_hint_falls_back_to_legacy(self, bad):
|
||||
out = assign_speakers_heuristic(self._gapped_segs(4), num_speakers=bad)
|
||||
labels = [s["speaker_id"] for s in out]
|
||||
assert labels == ["Speaker 1", "Speaker 2", "Speaker 1", "Speaker 2"]
|
||||
|
||||
def test_heuristic_hint_no_gaps_keeps_one_speaker(self):
|
||||
# N is an upper bound, not a quota: back-to-back speech with no
|
||||
# > SPEAKER_GAP pause stays one speaker even with a hint of 3.
|
||||
segs = [
|
||||
{"start": 0.0, "end": 2.0, "text": "a", "id": "1"},
|
||||
{"start": 2.1, "end": 4.0, "text": "b", "id": "2"},
|
||||
{"start": 4.2, "end": 6.0, "text": "c", "id": "3"},
|
||||
]
|
||||
out = assign_speakers_heuristic(segs, num_speakers=3)
|
||||
assert {s["speaker_id"] for s in out} == {"Speaker 1"}
|
||||
|
||||
def test_diarization_uses_overlap_weighted_assignment(self):
|
||||
# Build a fake diarization with two overlapping turns for the same seg;
|
||||
# the one with more overlap should win, not the one at midpoint.
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
"""Clone-purity guards in services.speaker_clone (speaker-hint fix).
|
||||
|
||||
A per-speaker auto-clone reference cut from mislabeled or boundary-adjacent
|
||||
audio mixes two people's voices — the field-reported "made up" clone voices.
|
||||
These tests pin the three guards:
|
||||
|
||||
* per-slice minimum duration (MIN_SLICE_DURATION_S),
|
||||
* non-adjacency scoring preference (ADJACENT_TURN_GUARD_S) — a preference,
|
||||
never a hard filter,
|
||||
* labels_source="heuristic" skips extraction entirely.
|
||||
|
||||
Pure tests over a synthetic vocals wav — no model, no main import.
|
||||
"""
|
||||
import numpy as np
|
||||
import pytest
|
||||
import soundfile as sf
|
||||
|
||||
from services.speaker_clone import (
|
||||
ADJACENT_TURN_GUARD_S,
|
||||
MIN_REF_DURATION_S,
|
||||
MIN_SLICE_DURATION_S,
|
||||
_pick_reference_slices,
|
||||
extract_speaker_clones,
|
||||
)
|
||||
|
||||
SR = 16000
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def vocals(tmp_path):
|
||||
# 60 s of non-silent audio so every segment slice has content.
|
||||
path = tmp_path / "vocals.wav"
|
||||
sf.write(str(path), np.float32(np.sin(np.linspace(0, 18000, 60 * SR))), SR)
|
||||
return str(path)
|
||||
|
||||
|
||||
def _seg(start, end, speaker="Speaker 1", text="hello there"):
|
||||
return {"start": start, "end": end, "speaker_id": speaker, "text": text}
|
||||
|
||||
|
||||
class TestPickReferenceSlices:
|
||||
def test_rejects_slices_below_minimum_duration(self):
|
||||
# Six 1.0 s fragments total 6 s (> MIN_REF_DURATION_S) — pre-fix they
|
||||
# were all picked; now every one is under MIN_SLICE_DURATION_S so the
|
||||
# speaker yields no reference at all (default voice beats a bad clone).
|
||||
items = [(i, _seg(i * 3.0, i * 3.0 + 1.0)) for i in range(6)]
|
||||
assert MIN_SLICE_DURATION_S > 1.0 # test premise
|
||||
assert 6 * 1.0 > MIN_REF_DURATION_S # pre-fix these WOULD have passed
|
||||
assert _pick_reference_slices(items) == []
|
||||
|
||||
def test_prefers_slice_not_adjacent_to_other_speaker(self):
|
||||
# Two equal-length candidates for Speaker 1; the first is 0.1 s away
|
||||
# from a Speaker 2 turn (< ADJACENT_TURN_GUARD_S), the second is far
|
||||
# from everyone. The clean one must win the ranking.
|
||||
adjacent = _seg(0.0, 8.0, "Speaker 1")
|
||||
other = _seg(8.1, 10.0, "Speaker 2")
|
||||
clean = _seg(20.0, 28.0, "Speaker 1")
|
||||
all_segments = [adjacent, other, clean]
|
||||
items = [(0, adjacent), (2, clean)]
|
||||
chosen = _pick_reference_slices(
|
||||
items, speaker_id="Speaker 1", all_segments=all_segments,
|
||||
)
|
||||
assert [seg for _, seg in chosen] == [clean]
|
||||
|
||||
def test_adjacency_is_a_preference_not_a_hard_filter(self):
|
||||
# Dense dialogue: every Speaker 1 slice borders a Speaker 2 turn.
|
||||
# Extraction must still succeed using the adjacent slices.
|
||||
s1a = _seg(0.0, 6.0, "Speaker 1")
|
||||
s2a = _seg(6.1, 8.0, "Speaker 2")
|
||||
s1b = _seg(8.2, 12.0, "Speaker 1")
|
||||
all_segments = [s1a, s2a, s1b]
|
||||
items = [(0, s1a), (2, s1b)]
|
||||
chosen = _pick_reference_slices(
|
||||
items, speaker_id="Speaker 1", all_segments=all_segments,
|
||||
)
|
||||
assert chosen, "dense dialogue must still produce a reference"
|
||||
|
||||
def test_heuristic_labels_source_returns_nothing(self):
|
||||
items = [(0, _seg(0.0, 8.0))]
|
||||
assert _pick_reference_slices(items, labels_source="heuristic") == []
|
||||
|
||||
def test_legacy_call_without_kwargs_still_picks_long_slice(self):
|
||||
# Backward compat: positional-only invocation (the pre-fix signature)
|
||||
# keeps working and picks the long slice.
|
||||
long_seg = _seg(0.0, 8.0)
|
||||
chosen = _pick_reference_slices([(0, long_seg)])
|
||||
assert [seg for _, seg in chosen] == [long_seg]
|
||||
|
||||
def test_overlapping_other_speaker_counts_as_adjacent(self):
|
||||
# Negative gap (overlap) must also be flagged — that is the worst
|
||||
# mixed-audio case of all.
|
||||
overlapped = _seg(0.0, 8.0, "Speaker 1")
|
||||
other = _seg(4.0, 6.0, "Speaker 2")
|
||||
clean = _seg(20.0, 28.0, "Speaker 1")
|
||||
chosen = _pick_reference_slices(
|
||||
[(0, overlapped), (2, clean)],
|
||||
speaker_id="Speaker 1",
|
||||
all_segments=[overlapped, other, clean],
|
||||
)
|
||||
assert [seg for _, seg in chosen] == [clean]
|
||||
|
||||
|
||||
class TestExtractSpeakerClones:
|
||||
def test_heuristic_labels_source_skips_extraction(self, tmp_path, vocals):
|
||||
segs = [_seg(0.0, 8.0), _seg(10.0, 18.0, "Speaker 2")]
|
||||
out = extract_speaker_clones(
|
||||
vocals, segs, str(tmp_path), labels_source="heuristic",
|
||||
)
|
||||
assert out == {}
|
||||
|
||||
@pytest.mark.parametrize("source", [None, "pyannote", "turns"])
|
||||
def test_trusted_labels_still_extract(self, tmp_path, vocals, source):
|
||||
# None (legacy caller, missing kwarg) and real diarization sources
|
||||
# keep the current behavior: clones are produced.
|
||||
segs = [_seg(0.0, 8.0), _seg(10.0, 18.0, "Speaker 2")]
|
||||
kwargs = {} if source is None else {"labels_source": source}
|
||||
out = extract_speaker_clones(vocals, segs, str(tmp_path), **kwargs)
|
||||
assert set(out) == {"Speaker 1", "Speaker 2"}
|
||||
for info in out.values():
|
||||
assert info["duration"] >= MIN_REF_DURATION_S
|
||||
|
||||
def test_adjacency_guard_constant_sane(self):
|
||||
# The guard must stay tighter than the heuristic's own gap threshold,
|
||||
# or every real turn boundary would be flagged.
|
||||
from services.segmentation import SPEAKER_GAP
|
||||
assert 0 < ADJACENT_TURN_GUARD_S < SPEAKER_GAP
|
||||
@@ -0,0 +1,306 @@
|
||||
"""The dub speaker-count hint must be honored on EVERY diarization path.
|
||||
|
||||
Field report (v0.3.9): setting the dub "Speakers" count changed nothing — the
|
||||
`?num_speakers=` hint reached `_diarize()` and then died on 3 of its 4
|
||||
branches (FunASR inline-turns shortcut, pyannote-unavailable heuristic
|
||||
fallback, pyannote-crash heuristic fallback). Speakers blended and auto-clones
|
||||
were cut from mixed-speaker audio.
|
||||
|
||||
These tests drive `dub_transcribe_stream`'s async generator directly (the
|
||||
established pattern in test_dub_transcribe.py — no TestClient, no GPU, no
|
||||
pyannote) and pin:
|
||||
|
||||
* an explicit num_speakers routes a turns-capable job through pyannote
|
||||
(with the hint) instead of the inline-turns shortcut;
|
||||
* when a branch cannot honor the hint exactly, a `warning` SSE event says so
|
||||
honestly instead of dropping it silently;
|
||||
* heuristic labels skip auto voice-clone extraction (clone-purity guard)
|
||||
with their own warning;
|
||||
* the legacy POST /dub/transcribe/{job_id} endpoint exposes the same
|
||||
(clamped) num_speakers parameter.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import struct
|
||||
import wave
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from api.routers import dub_core as dc
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _make_wav(path: Path, seconds: float = 1.0, sr: int = 16000) -> None:
|
||||
n = int(seconds * sr)
|
||||
with wave.open(str(path), "wb") as wf:
|
||||
wf.setnchannels(1)
|
||||
wf.setsampwidth(2)
|
||||
wf.setframerate(sr)
|
||||
wf.writeframes(struct.pack(f"<{n}h", *([0] * n)))
|
||||
|
||||
|
||||
class _FakeASR:
|
||||
"""Minimal ASR backend; `turns` simulates FunASR-style inline diarization."""
|
||||
|
||||
id = "fake"
|
||||
|
||||
def __init__(self, turns=None):
|
||||
self._turns = turns or []
|
||||
|
||||
def ensure_loaded(self):
|
||||
pass
|
||||
|
||||
def transcribe(self, path, *, word_timestamps=True):
|
||||
return {
|
||||
"chunks": [
|
||||
{"text": "Hello there my friend.", "timestamp": (0.0, 0.4)},
|
||||
{"text": "I am doing quite well today.", "timestamp": (0.5, 0.9)},
|
||||
],
|
||||
"segments": self._turns,
|
||||
"language": "en",
|
||||
}
|
||||
|
||||
def unload(self):
|
||||
pass
|
||||
|
||||
|
||||
class _FakeTurn:
|
||||
def __init__(self, start, end):
|
||||
self.start = start
|
||||
self.end = end
|
||||
|
||||
|
||||
class _FakeDiar:
|
||||
def itertracks(self, yield_label=True):
|
||||
yield _FakeTurn(0.0, 0.45), None, "SPEAKER_00"
|
||||
yield _FakeTurn(0.45, 1.0), None, "SPEAKER_01"
|
||||
|
||||
|
||||
class _RecordingPipe:
|
||||
"""Stands in for the pyannote pipeline; records the num_speakers kwarg."""
|
||||
|
||||
def __init__(self, crash=False):
|
||||
self.calls: list = []
|
||||
self._crash = crash
|
||||
|
||||
def __call__(self, path, num_speakers=None):
|
||||
self.calls.append(num_speakers)
|
||||
if self._crash:
|
||||
raise RuntimeError("pyannote exploded: simulated")
|
||||
return _FakeDiar()
|
||||
|
||||
|
||||
def _wire_stream(tmp_path, monkeypatch, *, job_id, asr, diar_pipeline):
|
||||
"""Common monkeypatching for driving the stream happy path end-to-end."""
|
||||
audio = tmp_path / "a.wav"
|
||||
_make_wav(audio, seconds=1.0)
|
||||
dc._dub_jobs[job_id] = {
|
||||
"audio_path": str(audio), "vocals_path": None, "scene_cuts": [],
|
||||
}
|
||||
|
||||
fake_model = MagicMock()
|
||||
fake_model._asr_pipe = MagicMock()
|
||||
|
||||
async def _ok_model():
|
||||
return fake_model
|
||||
|
||||
monkeypatch.setattr(dc, "get_model", _ok_model)
|
||||
monkeypatch.setattr(
|
||||
"services.asr_backend.get_active_asr_backend", lambda *a, **k: asr,
|
||||
)
|
||||
monkeypatch.setattr(dc, "get_diarization_pipeline", diar_pipeline)
|
||||
monkeypatch.setattr(dc, "offload_tts_for_asr", lambda *a, **k: None)
|
||||
monkeypatch.setattr(dc, "restore_tts_after_asr", lambda *a, **k: None)
|
||||
monkeypatch.setattr(dc, "_save_job", lambda *a, **k: None)
|
||||
# Keep the no-token branch deterministic (never read this machine's HF creds).
|
||||
monkeypatch.setattr("services.token_resolver.resolve", lambda *a, **k: None)
|
||||
|
||||
|
||||
def _run_stream(job_id, num_speakers=None) -> str:
|
||||
async def _collect():
|
||||
resp = await dc.dub_transcribe_stream(job_id, num_speakers=num_speakers)
|
||||
parts = []
|
||||
async for chunk in resp.body_iterator:
|
||||
parts.append(chunk.decode() if isinstance(chunk, (bytes, bytearray)) else str(chunk))
|
||||
return "".join(parts)
|
||||
|
||||
try:
|
||||
return asyncio.run(_collect())
|
||||
finally:
|
||||
dc._dub_jobs.pop(job_id, None)
|
||||
|
||||
|
||||
_TURNS = [
|
||||
{"start": 0.0, "end": 0.45, "speaker": "Speaker 1"},
|
||||
{"start": 0.5, "end": 0.9, "speaker": "Speaker 2"},
|
||||
]
|
||||
|
||||
_NO_PIPE = lambda return_error=False: (None, "NO_TOKEN") if return_error else None # noqa: E731
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Branch 1 — FunASR inline-turns shortcut must not eat the hint
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_hint_routes_turns_job_through_pyannote(tmp_path, monkeypatch):
|
||||
"""With inline ASR turns present AND pyannote loadable, an explicit
|
||||
num_speakers must skip the turns shortcut and reach pyannote as a kwarg.
|
||||
Pre-fix the shortcut returned early and the hint was never consulted."""
|
||||
pipe = _RecordingPipe()
|
||||
_wire_stream(
|
||||
tmp_path, monkeypatch, job_id="t_hint_pyannote",
|
||||
asr=_FakeASR(turns=_TURNS),
|
||||
diar_pipeline=lambda return_error=False: (pipe, None) if return_error else pipe,
|
||||
)
|
||||
body = _run_stream("t_hint_pyannote", num_speakers=2)
|
||||
|
||||
assert pipe.calls == [2], f"pyannote must be called once with the hint: {pipe.calls}"
|
||||
assert "event: final" in body, body
|
||||
# The hint was fully honored — no hint warning may fire.
|
||||
assert "Speaker-count hint ignored" not in body, body
|
||||
|
||||
|
||||
def test_no_hint_keeps_turns_fast_path(tmp_path, monkeypatch):
|
||||
"""Without a hint, the inline-turns shortcut stays the fast path: pyannote
|
||||
is never invoked and no warning fires."""
|
||||
pipe = _RecordingPipe()
|
||||
_wire_stream(
|
||||
tmp_path, monkeypatch, job_id="t_nohint_turns",
|
||||
asr=_FakeASR(turns=_TURNS),
|
||||
diar_pipeline=lambda return_error=False: (pipe, None) if return_error else pipe,
|
||||
)
|
||||
body = _run_stream("t_nohint_turns", num_speakers=None)
|
||||
|
||||
assert pipe.calls == [], "turns fast path must skip pyannote when no hint is set"
|
||||
assert "event: final" in body, body
|
||||
assert "event: warning" not in body, body
|
||||
|
||||
|
||||
def test_hint_with_turns_but_no_pyannote_warns_hint_ignored(tmp_path, monkeypatch):
|
||||
"""Turns present, hint set, pyannote unavailable: keep the turns (best
|
||||
labels available) but tell the user the hint was ignored — never silence."""
|
||||
_wire_stream(
|
||||
tmp_path, monkeypatch, job_id="t_hint_turns_warn",
|
||||
asr=_FakeASR(turns=_TURNS), diar_pipeline=_NO_PIPE,
|
||||
)
|
||||
body = _run_stream("t_hint_turns_warn", num_speakers=3)
|
||||
|
||||
assert "event: warning" in body, body
|
||||
assert "Speaker-count hint ignored" in body, body
|
||||
assert "may differ from the 3 you set" in body, body
|
||||
# Turns are trusted labels — the clone-purity guard must NOT fire.
|
||||
assert "auto voice cloning skipped" not in body, body
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Branches 2+3 — heuristic fallbacks: hint threaded + honest warning + clone guard
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_heuristic_fallback_warns_approximate_and_skips_clones(tmp_path, monkeypatch):
|
||||
"""No turns, no pyannote, hint set: the heuristic cycles the requested
|
||||
count (approximate honesty), the warning says exactly that, and auto
|
||||
voice-clone extraction is skipped because gap-based labels are estimates."""
|
||||
_wire_stream(
|
||||
tmp_path, monkeypatch, job_id="t_hint_heur",
|
||||
asr=_FakeASR(turns=[]), diar_pipeline=_NO_PIPE,
|
||||
)
|
||||
body = _run_stream("t_hint_heur", num_speakers=3)
|
||||
|
||||
assert "event: warning" in body, body
|
||||
assert "only approximately honored" in body, body
|
||||
assert "cycles 3 speaker labels" in body, body
|
||||
# Clone-purity guard fires with its own warning.
|
||||
assert dc.CLONE_SKIP_HEURISTIC_MSG in body, body
|
||||
assert "event: final" in body, body
|
||||
|
||||
|
||||
def test_heuristic_fallback_without_hint_keeps_legacy_warning(tmp_path, monkeypatch):
|
||||
"""No hint → the pre-existing fallback warning is unchanged (no hint text),
|
||||
but the clone-purity guard still protects against gap-based labels."""
|
||||
_wire_stream(
|
||||
tmp_path, monkeypatch, job_id="t_nohint_heur",
|
||||
asr=_FakeASR(turns=[]), diar_pipeline=_NO_PIPE,
|
||||
)
|
||||
body = _run_stream("t_nohint_heur", num_speakers=None)
|
||||
|
||||
assert "event: warning" in body, body
|
||||
assert "silence-gap" in body, body
|
||||
assert "approximately honored" not in body, body
|
||||
assert dc.CLONE_SKIP_HEURISTIC_MSG in body, body
|
||||
|
||||
|
||||
def test_pyannote_crash_with_turns_falls_back_to_turns_not_heuristic(tmp_path, monkeypatch):
|
||||
"""When the hint routes a turns-job through pyannote and pyannote crashes
|
||||
mid-run, the inline turns are the fallback (better than the heuristic) and
|
||||
the warning says the hint was ignored."""
|
||||
pipe = _RecordingPipe(crash=True)
|
||||
_wire_stream(
|
||||
tmp_path, monkeypatch, job_id="t_crash_turns",
|
||||
asr=_FakeASR(turns=_TURNS),
|
||||
diar_pipeline=lambda return_error=False: (pipe, None) if return_error else pipe,
|
||||
)
|
||||
body = _run_stream("t_crash_turns", num_speakers=2)
|
||||
|
||||
assert pipe.calls == [2], pipe.calls
|
||||
assert "crashed mid-run" in body, body
|
||||
assert "built-in speaker turns" in body, body
|
||||
assert "Speaker-count hint ignored" in body, body
|
||||
# Turns labels → clones stay allowed.
|
||||
assert "auto voice cloning skipped" not in body, body
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shared clamp + legacy endpoint parity
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestClampNumSpeakers:
|
||||
@pytest.mark.parametrize("raw,expected", [
|
||||
(None, None), (1, 1), (5, 5), (20, 20),
|
||||
(0, None), (-2, None), (21, None), ("7", 7), ("junk", None),
|
||||
])
|
||||
def test_clamp(self, raw, expected):
|
||||
assert dc._clamp_num_speakers(raw) == expected
|
||||
|
||||
|
||||
def test_legacy_transcribe_endpoint_accepts_num_speakers():
|
||||
"""POST /dub/transcribe/{job_id} (the CLI's endpoint) exposes the same
|
||||
optional num_speakers query parameter as the SSE stream. Pre-fix the
|
||||
legacy route had no way to express a speaker count at all."""
|
||||
import inspect
|
||||
from typing import Optional
|
||||
|
||||
sig = inspect.signature(dc.dub_transcribe)
|
||||
assert "num_speakers" in sig.parameters, "legacy endpoint lost num_speakers"
|
||||
p = sig.parameters["num_speakers"]
|
||||
assert p.default is None, "num_speakers must be optional (auto-detect default)"
|
||||
assert p.annotation == Optional[int]
|
||||
|
||||
# And the stream endpoint still has it too (parity in both directions).
|
||||
stream_sig = inspect.signature(dc.dub_transcribe_stream)
|
||||
assert "num_speakers" in stream_sig.parameters
|
||||
|
||||
|
||||
def test_cli_exposes_speakers_flag(capsys):
|
||||
"""omnivoice-dub --speakers N must parse and be forwarded as the
|
||||
num_speakers query param on the legacy transcribe endpoint."""
|
||||
import importlib.util
|
||||
|
||||
cli_path = Path(__file__).resolve().parents[1] / "omnivoice" / "cli" / "dub.py"
|
||||
spec = importlib.util.spec_from_file_location("_omnivoice_cli_dub", cli_path)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
|
||||
# Parse-level: --help lists the flag (argparse would reject an unknown one).
|
||||
with pytest.raises(SystemExit):
|
||||
mod.main(["--help"])
|
||||
assert "--speakers" in capsys.readouterr().out
|
||||
|
||||
# Wiring-level: the flag is forwarded as the num_speakers query param.
|
||||
src = cli_path.read_text()
|
||||
assert "num_speakers={args.speakers}" in src
|
||||
@@ -115,6 +115,101 @@ def test_cinematic_reflect_failure_returns_literal(monkeypatch):
|
||||
assert "reflect" in res.get("error", "")
|
||||
|
||||
|
||||
# ── Divergence guard (v0.3.9 field report: hallucinated dub lines) ─────────
|
||||
|
||||
|
||||
def _mock_chain(monkeypatch, reflect: str, adapt: str):
|
||||
"""Wire cinematic_refine_sync to a mocked 2-step REFLECT→ADAPT chain."""
|
||||
responses = iter([reflect, adapt])
|
||||
monkeypatch.setattr(tr, "_llm_client", lambda: MagicMock())
|
||||
monkeypatch.setattr(tr, "_chat", lambda client, *, system, user: next(responses))
|
||||
|
||||
|
||||
def test_cinematic_adapt_runaway_length_falls_back_to_literal(monkeypatch):
|
||||
"""The reported bug: for a Latin-script target the script check passes ANY
|
||||
text, so a hallucinated wall of dialogue used to ship as the dub line."""
|
||||
literal = "¿Cómo estás hoy, amigo mío?"
|
||||
_mock_chain(monkeypatch, "fine but a bit stiff", "Hola amigo. " * 30) # ~13× the literal
|
||||
res = tr.cinematic_refine_sync(
|
||||
"How are you doing today, my friend?", literal,
|
||||
source_lang="en", target_lang="es",
|
||||
)
|
||||
assert res["text"] == literal
|
||||
assert res.get("error") == "adapt-diverged"
|
||||
assert res["critique"] == "fine but a bit stiff" # UI still sees what happened
|
||||
|
||||
|
||||
def test_cinematic_adapt_critique_echo_rejected(monkeypatch):
|
||||
"""ADAPT returning the critique itself must not become the dub line."""
|
||||
literal = "¿Cómo estás hoy, amigo mío? Hace mucho que no te veo por aquí."
|
||||
critique = (
|
||||
"The literal translation reads stiff and does not fit the slot; "
|
||||
"prefer a shorter, more idiomatic phrasing with warmer tone."
|
||||
)
|
||||
_mock_chain(monkeypatch, critique, critique) # adapt echoes the critique verbatim
|
||||
res = tr.cinematic_refine_sync(
|
||||
"How are you doing today, my friend? Long time no see.", literal,
|
||||
source_lang="en", target_lang="es",
|
||||
)
|
||||
assert res["text"] == literal
|
||||
assert res.get("error") == "adapt-diverged"
|
||||
|
||||
|
||||
def test_cinematic_sane_adaptation_accepted(monkeypatch):
|
||||
"""A faithful, idiomatic rewrite passes every guard untouched."""
|
||||
literal = "¿Cómo estás hoy, amigo mío? Hace mucho que no te veo."
|
||||
adapted = "¿Qué tal, amigo? ¡Cuánto tiempo sin verte!"
|
||||
_mock_chain(monkeypatch, "a bit formal; contract it", adapted)
|
||||
res = tr.cinematic_refine_sync(
|
||||
"How are you doing today, my friend? Long time no see.", literal,
|
||||
source_lang="en", target_lang="es",
|
||||
)
|
||||
assert res["text"] == adapted
|
||||
assert "error" not in res
|
||||
|
||||
|
||||
def test_cinematic_adapt_wrong_script_falls_back_to_literal(monkeypatch):
|
||||
"""ADAPT output off the target script degrades to the literal with the
|
||||
script-specific marker (the pre-existing fallback, previously untested)."""
|
||||
literal = "नमस्ते मेरे दोस्त, आप कैसे हैं?"
|
||||
_mock_chain(monkeypatch, "solid but wordy", "This is English, not Hindi, sorry.")
|
||||
res = tr.cinematic_refine_sync(
|
||||
"Hello my friend, how are you?", literal,
|
||||
source_lang="en", target_lang="hi",
|
||||
)
|
||||
assert res["text"] == literal
|
||||
assert res.get("error") == "adapt-wrong-script:hi"
|
||||
|
||||
|
||||
def test_chat_pins_low_temperature(monkeypatch):
|
||||
"""Cinematic reflect/adapt must pin temperature like the Fast path does —
|
||||
the provider default of 1.0 is what let local models drift into invention."""
|
||||
client = MagicMock()
|
||||
client.chat.completions.create.return_value = MagicMock(
|
||||
choices=[MagicMock(message=MagicMock(content="ok"))])
|
||||
monkeypatch.setattr(tr, "_llm_model", lambda: "test-model")
|
||||
assert tr._chat(client, system="s", user="u") == "ok"
|
||||
assert client.chat.completions.create.call_args.kwargs["temperature"] == 0.2
|
||||
|
||||
|
||||
def test_refine_guard_short_reference_uses_absolute_cap():
|
||||
# A 2-word line legitimately triples — the ratio window must not apply.
|
||||
ok, _ = tr.refine_output_ok("¡No!", "¡Claro que no, jamás!", "es")
|
||||
assert ok
|
||||
# …but a wall of text after a 2-word line is still divergence.
|
||||
ok, reason = tr.refine_output_ok("¡No!", "x" * 200, "es")
|
||||
assert not ok and reason.startswith("length-abs")
|
||||
|
||||
|
||||
def test_refine_guard_ratio_env_override(monkeypatch):
|
||||
literal = "¿Cómo estás hoy, amigo mío?"
|
||||
ok, reason = tr.refine_output_ok(literal, literal * 4, "es")
|
||||
assert not ok and reason.startswith("length-ratio")
|
||||
monkeypatch.setenv("OMNIVOICE_REFINE_RATIO_MAX", "5.0")
|
||||
ok, _ = tr.refine_output_ok(literal, literal * 4, "es")
|
||||
assert ok # ceiling raised via env, mirroring the _cinematic_budget pattern
|
||||
|
||||
|
||||
# ── Cinematic pass wall-clock budget (#stall follow-up) ────────────────────
|
||||
|
||||
def test_cinematic_budget_degrades_slow_segments_to_literal(monkeypatch):
|
||||
|
||||