diff --git a/CHANGELOG.md b/CHANGELOG.md index 9fd09cdc..94943ebc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,8 +22,10 @@ the frozen-backend fallback mirror it for their toolchains. ### Docs - The READMEs now lead with download buttons and a three-step first-clone walkthrough, and a new benchmarks page anchors measured per-engine/per-device numbers on the in-repo harness (#1555) +- Every engine now has its own guide — 21 new pages under docs/engines plus an index covering all 16 TTS and 11 ASR engines, linked from both READMEs (#1556) ### Fixed +- The crash-isolated ASR sidecar and its download preflight now agree on which model to load — setting the shared faster-whisper model variable applies to both variants instead of the sidecar quietly using a different one (#1556) - "Ready" now requires the deep health probe (a working database-backed route), not just the identity probe — a backend whose install broke underneath can no longer be announced up while every real request fails (#1548) - Supervisor restarts after repeat crashes now back off (immediate, then 5s, then 15s) instead of respawning back-to-back, so a tight crash loop can't burn the whole restart budget in seconds (#1548) diff --git a/README.md b/README.md index 797222ab..adf39184 100644 --- a/README.md +++ b/README.md @@ -218,7 +218,7 @@ Professional-grade voice AI, minus the subscription and the cloud. Convinced? [C ### 🗣️ TTS Engines -**16 engines, one picker.** VoiceStudio (default, 600+ languages) is always available; seven more are opt-in and auto-detected (CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX), plus eight lazy-installed opt-ins (IndexTTS 2.5, OmniVoice GGUF, OmniVoice subprocess, PocketTTS, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS). Switch in **Model Catalogue → Engines** — or from anywhere with Ctrl/Cmd+E; the choice applies everywhere synthesis happens. +**16 engines, one picker.** VoiceStudio (default, 600+ languages) is always available; seven more are opt-in and auto-detected (CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX), plus eight lazy-installed opt-ins (IndexTTS 2.5, OmniVoice GGUF, OmniVoice subprocess, PocketTTS, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS). Switch in **Model Catalogue → Engines** — or from anywhere with Ctrl/Cmd+E; the choice applies everywhere synthesis happens. **Every engine has its own guide: [docs/engines](docs/engines/README.md).**
📊 The full matrix — 16 engines × platform × clone/instruct × license @@ -228,15 +228,15 @@ Professional-grade voice AI, minus the subscription and the cloud. Convinced? [C | Engine | Languages | Clone | Instruct | Linux | macOS ARM | Windows | License | |--------|:---------:|:-----:|:--------:|:-----:|:---------:|:-------:|:-------:| | **VoiceStudio** (default, powered by k2-fsa/OmniVoice) | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Built-in | -| **CosyVoice 3** | 9 + 18 dialects | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 | +| **CosyVoice 3** | 9 + 18 dialects | ✅ | ✅ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 | | **GPT-SoVITS** | 5 | ✅ | — | ✅ CUDA/CPU | — | ✅ CUDA/CPU | MIT | | **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 | | **MOSS-TTS-Nano** | 20 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 | | **KittenTTS** | English | — | — | ✅ CPU | ✅ CPU | ✅ CPU | MIT | | **MLX-Audio** (Kokoro, Qwen3-TTS, CSM, Dia, …) | Multi | Varies | Varies | ❌ | ✅ Native | ❌ | Varies | | **Sherpa-ONNX** | 20+ | — | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 | -| **IndexTTS 2.5** ⚡ | ZH · EN · JA · ES · AR | ✅ | — | ✅ CUDA | — | ✅ CUDA | Bilibili model license¹ | -| **OmniVoice GGUF** ⚡ | 600+ | ✅ | ✅ | ✅ CPU | ✅ CPU | ✅ CPU | Built-in | +| **IndexTTS 2.5** ⚡ | ZH · EN · JA · ES · AR | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Bilibili model license¹ | +| **OmniVoice GGUF** ⚡ | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS/CPU | ✅ CUDA/CPU | Built-in | | **OmniVoice (subprocess)** ⚡² | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Built-in | | **PocketTTS** ⚡ (Kyutai) | EN · FR · DE · PT · IT · ES | ✅ | — | ✅ CPU | ✅ CPU | ✅ CPU | CC-BY-4.0 (gated)³ | | **Supertonic 3** ⚡ | 31 | — | — | ✅ CPU | ✅ CPU | ✅ CPU | OpenRAIL-M | @@ -275,7 +275,7 @@ another machine, set `OMNIVOICE_GPTSOVITS_URL` to its credential-free ### 🎧 ASR Engines -**11 engines** — they power dictation, video dubbing, and subtitles. **WhisperX** is the cross-platform default (~100 languages, word-level timing); the rest are opt-in and auto-detected. Switch in **Model Catalogue → Engines**. Ten run fully on-device; the eleventh (OpenAI-compatible) is an optional remote client for Qwen3-ASR or any compatible server. +**11 engines** — they power dictation, video dubbing, and subtitles. **WhisperX** is the cross-platform default (~100 languages, word-level timing); the rest are opt-in and auto-detected. Switch in **Model Catalogue → Engines**. Ten run fully on-device; the eleventh (OpenAI-compatible) is an optional remote client for Qwen3-ASR or any compatible server. **Per-engine guides: [docs/engines](docs/engines/README.md).**
📊 The full lineup — 11 engines, what each is best at, and compute-type notes diff --git a/README_CN.md b/README_CN.md index e62d187a..58b833d9 100644 --- a/README_CN.md +++ b/README_CN.md @@ -180,8 +180,8 @@ Hugging Face Token 的配置见 | **API 密钥** | 需要账号 | 本地流程不需要 | | **GPU 支持** | 不适用(云端) | CUDA · Apple Silicon · ROCm(Linux)· CPU | | **桌面应用** | ❌ | ✅ macOS · Windows · Linux | -| **TTS 引擎** | 1 | **14** — [完整矩阵](#tts-engines) | -| **ASR 引擎** | 1 | **10** — [完整阵容](#asr-engines) | +| **TTS 引擎** | 1 | **16** — [完整矩阵](#tts-engines) | +| **ASR 引擎** | 1 | **11** — [完整阵容](#asr-engines) | | **MCP 服务器** | ❌ | ✅ 可从 Claude、Cursor 及任何 MCP 客户端使用 | | **自检** | ❌ | ✅ 诊断套件、错误日志、脱敏调试包 | | **可定制** | ❌ 闭源 | ✅ 随你 Fork、扩展、发布 | @@ -221,10 +221,10 @@ Hugging Face Token 的配置见 ### 🗣️ TTS 引擎 -**14 个引擎,一个选择器。** VoiceStudio(默认,支持 600+ 语言)始终可用;另有七个引擎可选装并自动检测(CosyVoice 3、GPT-SoVITS、VoxCPM2、MOSS-TTS-Nano、KittenTTS、MLX-Audio、Sherpa-ONNX),外加六个按需延迟安装的重量级引擎(IndexTTS 2.5、OmniVoice GGUF、Supertonic 3、MOSS-TTS-v1.5、dots.tts、Confucius4-TTS)。在 **设置 → TTS 引擎** 中切换;所选引擎将应用于所有语音合成场景。 +**16 个引擎,一个选择器。** VoiceStudio(默认,支持 600+ 语言)始终可用;另有七个引擎可选装并自动检测(CosyVoice 3、GPT-SoVITS、VoxCPM2、MOSS-TTS-Nano、KittenTTS、MLX-Audio、Sherpa-ONNX),外加八个按需延迟安装的引擎(IndexTTS 2.5、OmniVoice GGUF、OmniVoice 子进程版、PocketTTS、Supertonic 3、MOSS-TTS-v1.5、dots.tts、Confucius4-TTS)。在 **设置 → TTS 引擎** 中切换;所选引擎将应用于所有语音合成场景。**每个引擎都有独立指南:[docs/engines](docs/engines/README.md)(英文)。**
-📊 完整矩阵——14 个引擎 × 平台 × 克隆/指令 × 许可证 +📊 完整矩阵——16 个引擎 × 平台 × 克隆/指令 × 许可证
@@ -261,10 +261,10 @@ Hugging Face Token 的配置见 ### 🎧 ASR 引擎 -**10 个引擎**——它们驱动听写、视频配音和字幕。**WhisperX** 是跨平台的默认引擎(约 100 种语言,词级时间对齐);其余引擎均为可选装并自动检测。在 **设置 → 引擎** 中切换。九个完全在本地设备上运行;第十个(OpenAI 兼容)是可选的远程客户端,可用于 Qwen3-ASR 或任何兼容的服务器。 +**11 个引擎**——它们驱动听写、视频配音和字幕。**WhisperX** 是跨平台的默认引擎(约 100 种语言,词级时间对齐);其余引擎均为可选装并自动检测。在 **设置 → 引擎** 中切换。十个完全在本地设备上运行;第十一个(OpenAI 兼容)是可选的远程客户端,可用于 Qwen3-ASR 或任何兼容的服务器。
-📊 完整阵容——10 个引擎、各自的强项与计算类型说明 +📊 完整阵容——11 个引擎、各自的强项与计算类型说明
@@ -281,7 +281,7 @@ Hugging Face Token 的配置见 | **sherpa-onnx**(实时听写) | `sherpa-onnx-asr` | 25 种欧洲语言 + 90+ | 实时、快于实时的听写——小体积流式/离线 ONNX 模型(Parakeet TDT v3/v2、流式 Zipformer 与 Paraformer、Whisper Tiny),CPU 运行,macOS / Windows / Linux 表现完全一致。在 **设置 → 语音** 中按模型选择。 | | **OpenAI 兼容** ⚠️ 远程 | `openai-compat-asr` | 取决于服务器 | 当下通往 **Qwen3-ASR** 的路径(自托管服务器,无需等 transformers 支持)、任何 OpenAI 兼容的转录端点,或 OpenAI 官方 API——无需安装,在 **设置 → 引擎**(ASR 标签页)中配置并测试连接。音频会离开你的设备,发送到你指定的任何服务器;参见 [docs/engines/openai-compatible-asr.md](docs/engines/openai-compatible-asr.md)。 | -> Whisper 系列引擎覆盖约 100 种语言;**FunASR / SenseVoice** 额外提供一条多语言一体化路径,内置语音活动检测与行内说话人分离。**sherpa-onnx** 驱动实时听写的模型选择器——你边说,文字边出现。每个引擎都在本地设备上运行——无需 API 密钥,无需云端。 +> Whisper 系列引擎覆盖约 100 种语言;**FunASR / SenseVoice** 额外提供一条多语言一体化路径,内置语音活动检测与行内说话人分离。**sherpa-onnx** 驱动实时听写的模型选择器——你边说,文字边出现。除可选的 OpenAI 兼容远程客户端外,所有引擎都在本地设备上运行——无需 API 密钥,无需云端。 > **GPU 不支持高效 float16?** 在较老的 NVIDIA GPU(Maxwell/Pascal、GTX 16xx)上,或在 CTranslate2/cuDNN 版本不匹配之后,CTranslate2 系 ASR 引擎(WhisperX、Faster-Whisper)无法运行 `float16`,VoiceStudio 会自动改用 `int8` 重试——无需配置。如果转录仍然失败,可用 `ASR_COMPUTE_TYPE` 环境变量固定计算类型(逃生舱口):`ASR_COMPUTE_TYPE=int8`(CPU 用 `float32`)。将其设为 `int8` 并重启后端。 diff --git a/backend/engines/_asr_sidecar/main.py b/backend/engines/_asr_sidecar/main.py index db6ebf7a..d0c04c58 100644 --- a/backend/engines/_asr_sidecar/main.py +++ b/backend/engines/_asr_sidecar/main.py @@ -85,7 +85,16 @@ def _get_model(): global _model if _model is None: from faster_whisper import WhisperModel - name = os.environ.get("ASR_MODEL_FW", "large-v3") + # Same weights as in-process faster-whisper: ASR_MODEL_FASTER selects + # for BOTH variants, ASR_MODEL_FW stays as a sidecar-only override. + # Before this, the sidecar read only ASR_MODEL_FW while the download + # preflight read ASR_MODEL_FASTER — set one and the other variant (or + # the preflight) quietly used a different model. + name = ( + os.environ.get("ASR_MODEL_FW") + or os.environ.get("ASR_MODEL_FASTER") + or "large-v3" + ) try: import torch device = "cuda" if torch.cuda.is_available() else "cpu" diff --git a/backend/services/asr_backend.py b/backend/services/asr_backend.py index 4e317ddc..8afafbdf 100644 --- a/backend/services/asr_backend.py +++ b/backend/services/asr_backend.py @@ -2325,7 +2325,7 @@ _INSTALL_HINTS: dict[str, str] = { "mac-ARM source installs since 0.3.22. Parakeet TDT v3 on the GPU via " "MLX: 25 European languages, word timestamps, ~2 GB unified memory.)" ), - "moonshine": "pip install useful-moonshine (edge/CPU-optimized ASR)", + "moonshine": "uv pip install moonshine-onnx (or moonshine-voice; edge/CPU-optimized ASR)", "funasr": "pip install funasr (SenseVoiceSmall + FSMN-VAD; CUDA or CPU)", "sherpa-onnx-asr": "uv add sherpa-onnx (ONNX live dictation; CPU, cross-platform)", "openai-compat-asr": ( @@ -3120,10 +3120,18 @@ def _offline_asr_repo(backend_id: str | None = None) -> str | None: bid = backend_id or active_backend_id() if bid == "whisperx": return _fw_repo(os.environ.get("ASR_MODEL_WHISPERX", "large-v3")) - if bid in ("faster-whisper", "faster-whisper-isolated"): - # The crash-isolated sidecar loads the SAME CT2 weights as in-process - # faster-whisper (it reuses the ASR_MODEL_FASTER selection). + if bid == "faster-whisper": return _fw_repo(os.environ.get("ASR_MODEL_FASTER", _FASTER_WHISPER_DEFAULT)) + if bid == "faster-whisper-isolated": + # Mirror the sidecar's own resolution (_asr_sidecar/main.py): + # ASR_MODEL_FW is a sidecar-only override, otherwise the shared + # ASR_MODEL_FASTER selection applies — so the preflight can never + # download a different repo than the sidecar will load. + return _fw_repo( + os.environ.get("ASR_MODEL_FW") + or os.environ.get("ASR_MODEL_FASTER") + or _FASTER_WHISPER_DEFAULT + ) if bid == "mlx-whisper": return os.environ.get("ASR_MODEL", _MLX_MODEL_DEFAULT) if bid == "parakeet-mlx": diff --git a/backend/services/tts_backend.py b/backend/services/tts_backend.py index 75486c5d..f50418aa 100644 --- a/backend/services/tts_backend.py +++ b/backend/services/tts_backend.py @@ -1092,7 +1092,8 @@ class KittenTTSBackend(TTSBackend): - English only - Much faster + much smaller install - Preset voice is chosen via `extras["voice"]` (defaults to "Jasper"). Any + Preset voice is chosen via `extras["voice"]` (defaults to DEFAULT_VOICE, + "expr-voice-2-f"). Any `ref_audio` / `instruct` / `language` arg is ignored with a log line so the common call-site doesn't need to know which engine it's talking to. """ diff --git a/docs/engines/README.md b/docs/engines/README.md new file mode 100644 index 00000000..534d6891 --- /dev/null +++ b/docs/engines/README.md @@ -0,0 +1,57 @@ +# Engine guides + +One page per engine: what it's for, what it needs, how to enable it, and its +quirks. Select engines in **Model Catalogue → Engines** (or quick-switch with +Ctrl/Cmd+E), or pin one with +`OMNIVOICE_TTS_BACKEND` / `OMNIVOICE_ASR_BACKEND`. + +Measured speed/VRAM numbers live in [benchmarks](../benchmarks.md); what each +engine can do expressively in [expressive-speech](../expressive-speech.md); +sidecar disk footprints in [disk-usage](disk-usage.md); the bar a new engine +must clear in [engine-acceptance](../engine-acceptance.md). + +New to VoiceStudio? Install the app first — [macOS](../install/macos.md) +(first launch needs the one-time right-click → **Open** Gatekeeper +approval), [Windows](../install/windows.md), [Linux](../install/linux.md), +[Docker](../install/docker.md). + +## Text-to-speech + +| Engine | Guide | Runs on | Cloning | Enabled by | +|---|---|---|---|---| +| VoiceStudio (OmniVoice) — **default** | [omnivoice](omnivoice.md) | CUDA · MPS · CPU | ✅ | installed by default | +| VoxCPM2 | [voxcpm2](voxcpm2.md) | CUDA · MPS · CPU | ✅ + voice design | `pip install "voxcpm>=2.0.3"` | +| MOSS-TTS-Nano | [moss-tts-nano](moss-tts-nano.md) | CUDA · CPU | ✅ (ref only) | clone + `uv pip install -e .` | +| KittenTTS | [kittentts](kittentts.md) | CPU | — (8 preset voices) | `pip install kittentts` | +| MLX-Audio (Kokoro, CSM, Dia, …) | [mlx-audio](mlx-audio.md) | Apple Silicon | model-dependent | `pip install mlx-audio` | +| CosyVoice 3 | [cosyvoice](cosyvoice.md) | CUDA · CPU | ✅ | clone + requirements | +| GPT-SoVITS | [gpt-sovits](gpt-sovits.md) | external server | ✅ | its own API server | +| Sherpa-ONNX | [sherpa-onnx](sherpa-onnx.md) | CUDA · CPU | — | `pip install sherpa-onnx` + model dir | +| IndexTTS 2.5 | [indextts](indextts.md) | CUDA · CPU | ✅ + emotion | one-click sidecar install | +| OmniVoice GGUF | [omnivoice-gguf](omnivoice-gguf.md) | CUDA · MPS · CPU | ✅ | bundled binary | +| Supertonic-3 | [supertonic3](supertonic3.md) | CPU | — (7 preset voices) | `uv sync --extra supertonic` + license | +| MOSS-TTS-v1.5 (8B) | [moss-tts-v15](moss-tts-v15.md) | CUDA · CPU | ✅ | clone + env var | +| dots.tts (2B) | [dots-tts](dots-tts.md) | CUDA · CPU (not Windows) | ✅ | clone + env var | +| OmniVoice (subprocess) | [omnivoice-subprocess](omnivoice-subprocess.md) | CUDA · MPS · CPU | ✅ | opt-in pick, no install | +| PocketTTS (Kyutai) | [pockettts](pockettts.md) | CPU (not Intel Mac) | ✅ | `uv sync --extra pockettts` + license | +| Confucius4-TTS | [confucius4-tts](confucius4-tts.md) | CUDA · CPU | ✅ | clone + env var | + +## Speech-to-text + +| Engine | Guide | Runs on | Best at | Enabled by | +|---|---|---|---|---| +| WhisperX | [whisperx](whisperx.md) | CUDA · CPU | dubbing (word timestamps + diarization) | installed by default | +| Faster-Whisper | [faster-whisper](faster-whisper.md) | CUDA · CPU | general transcription | installed by default | +| Faster-Whisper (isolated) | [faster-whisper-isolated](faster-whisper-isolated.md) | CUDA · CPU | unattended batches | opt-in pick | +| MLX Whisper | [mlx-whisper](mlx-whisper.md) | Apple Silicon | Mac default | `pip install mlx-whisper` | +| PyTorch Whisper | [pytorch-whisper](pytorch-whisper.md) | CUDA · MPS · CPU | ROCm hosts | installed by default | +| Parakeet TDT (NeMo) | [nemo-parakeet](nemo-parakeet.md) | CUDA · CPU | 25 languages, fast CPU | separate venv (never the app's) | +| Parakeet TDT (MLX) | [parakeet-mlx](parakeet-mlx.md) | Apple Silicon | dictation, 25 EU languages | default on mac-ARM source installs | +| Moonshine | [moonshine](moonshine.md) | CPU | edge/low-power, no timestamps | `pip install` (see guide) | +| FunASR (SenseVoice) | [funasr](funasr.md) | CUDA · CPU | 50+ languages, inline diarization | `pip install funasr` | +| Sherpa-ONNX dictation | [sherpa-onnx-asr](sherpa-onnx-asr.md) | CPU | live streaming dictation | curated model download | +| OpenAI-compatible (remote) | [openai-compatible-asr](openai-compatible-asr.md) | network | offloading to a server (audio leaves the machine) | Model Catalogue | + +Speaker diarization is not an engine registry of its own — the dub pipeline +uses pyannote (HF-gated; see [diarization](../features/diarization.md)) and +FunASR can diarize inline with its `cam++` speaker model. diff --git a/docs/engines/faster-whisper-isolated.md b/docs/engines/faster-whisper-isolated.md new file mode 100644 index 00000000..e48d506d --- /dev/null +++ b/docs/engines/faster-whisper-isolated.md @@ -0,0 +1,61 @@ +# VoiceStudio — Faster-Whisper (Crash-Isolated) Engine + +The same CTranslate2 Whisper engine as [faster-whisper](faster-whisper.md), +run in a **separate child process** ("sidecar"). CTranslate2's GPU teardown +can segfault — the endemic faster-whisper crash — and a hung or crashed +transcribe in-process takes the whole backend down with it. Isolated, the +child can crash or be force-killed to reclaim a hung transcribe and its VRAM +while the backend stays up +([#730](https://github.com/debpalash/VoiceStudio/issues/730)). + +There is nothing extra to install: the sidecar reuses the app's own venv — +only the process boundary is new. + +## Selecting it + +- **Model Catalogue → Engines**, ASR tab → **Use** on the crash-isolated row, or +- pin it with `OMNIVOICE_ASR_BACKEND=faster-whisper-isolated`. + +It is never picked by auto-detect — it's an explicit opt-in escape hatch. + +## Best at + +- **Long batch runs** where one bad file must not kill the backend. +- Machines where in-process faster-whisper has crashed or hung before: + a sidecar crash fails only that job, and the next transcribe respawns a + fresh sidecar automatically. + +## Platform support + +Same as faster-whisper: CUDA float16 or CPU int8 on macOS, Windows, and +Linux. The sidecar picks cuda/cpu itself and walks the same +float16 → int8_float16 → int8 degrade chain on GPUs without efficient fp16 +([#551](https://github.com/debpalash/VoiceStudio/issues/551)). + +## Model selection + +- `ASR_MODEL_FASTER` — the shared model selection, same as the in-process + engine: set it once and both variants load the same weights. +- `ASR_MODEL_FW` — optional sidecar-only override; when set it wins over + `ASR_MODEL_FASTER` for this engine. Default `large-v3`. +- `ASR_COMPUTE_TYPE` — optional: pin the sidecar to one CTranslate2 compute + type instead of the automatic degrade chain. + +Weights download on first load — see +[downloading-models](../downloading-models.md). + +## Trade-offs and quirks + +- **Slightly slower per call** than in-process faster-whisper (IPC overhead); + the model stays warm inside the sidecar between calls, so the cost is per + request, not per chunk of audio. +- Word timestamps are Whisper-native (±100–300 ms) — no forced alignment. + For dubbing lip-sync, use [whisperx](whisperx.md) or + [mlx-whisper](mlx-whisper.md). +- If the sidecar dies mid-transcription the job fails with a clear + "sidecar crashed" error and the backend stays up — retry to respawn. +- **cuDNN 8 is still required on CUDA** — same CTranslate2 requirement as the + in-process engine. It's checked up front so a missing cuDNN 8 shows as + "unavailable" in Model Catalogue → Engines instead of a sidecar that + silently fails every transcribe + ([#1371](https://github.com/debpalash/VoiceStudio/issues/1371)). diff --git a/docs/engines/faster-whisper.md b/docs/engines/faster-whisper.md new file mode 100644 index 00000000..35023d5f --- /dev/null +++ b/docs/engines/faster-whisper.md @@ -0,0 +1,70 @@ +# VoiceStudio — Faster-Whisper Engine + +Faster-Whisper runs Whisper on CTranslate2 — the same transcription core +WhisperX uses, **without** the wav2vec2 forced-alignment pass. It's the safe +cross-platform fallback when whisperx isn't installed, and the capture/dictation +fallback on non-Apple machines. + +## Selecting it + +- **Model Catalogue → Engines**, ASR tab → **Use** on the Faster-Whisper row, or +- pin it with `OMNIVOICE_ASR_BACKEND=faster-whisper`. + +Auto-detect only picks it when [whisperx](whisperx.md) is unavailable. + +## Best at + +- **Subtitles, dictation buffers, and batch transcription** where Whisper's + native word timing (±100–300 ms) is good enough. +- For dubbing lip-sync, prefer [whisperx](whisperx.md) (or + [mlx-whisper](mlx-whisper.md) on Apple Silicon) — their forced alignment is + an order of magnitude tighter on word boundaries. + +## Platform support + +- **CUDA** — float16, with automatic degradation (below). +- **CPU** — int8 on macOS, Windows, and Linux. +- **Apple Silicon GPU / ROCm** — not supported: CTranslate2 has no Metal or + HIP build, so those hosts run on CPU + ([#1529](https://github.com/debpalash/VoiceStudio/issues/1529)); auto-detect + routes them to mlx-whisper / pytorch-whisper instead. + +## Model selection + +`ASR_MODEL_FASTER` — default `Systran/faster-whisper-large-v3`. Accepts the +size aliases (`tiny` … `large-v3`, `distil-large-v3`) or any CTranslate2 +Whisper repo on HF. Weights download on first load — see +[downloading-models](../downloading-models.md). + +Segments are cleaned up by faster-whisper's built-in Silero VAD before +transcription. + +## Degradation chains + +- GPUs without efficient fp16 (older Maxwell/Pascal, GTX 16xx, or a + CTranslate2/cuDNN mismatch) fail at model construction with a compute-type + error; the engine walks float16 → int8_float16 → int8 instead of failing + every chunk ([#551](https://github.com/debpalash/VoiceStudio/issues/551)). +- A CUDA out-of-memory falls back to CPU (slower, same model and accuracy) — + flushing the resident TTS model frees VRAM for GPU-speed ASR + ([#255](https://github.com/debpalash/VoiceStudio/issues/255)). + +## Quirks + +- **cuDNN 8 required on CUDA** — a missing cuDNN 8 would fast-fail the whole + process, so the engine checks up front and reports itself unavailable + instead ([#1371](https://github.com/debpalash/VoiceStudio/issues/1371)). + pytorch-whisper covers that case on torch's bundled cuDNN 9. +- On some hardened Linux kernels the CTranslate2 native library is rejected + with "cannot enable executable stack" (an OSError, not an ImportError) — + reported as unavailable rather than crashing engine selection + ([#692](https://github.com/debpalash/VoiceStudio/issues/692)). +- CTranslate2's GPU teardown can rarely segfault the process at unload. If + you hit that, switch to the crash-isolated variant — + [faster-whisper-isolated](faster-whisper-isolated.md) + ([#730](https://github.com/debpalash/VoiceStudio/issues/730)). +- Transcribes are time-bounded: `OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S` + (default 120 s per dub chunk) and `OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S` + (default 300 s whole-file). + +Speed comparisons across engines live in [performance](../performance.md). diff --git a/docs/engines/funasr.md b/docs/engines/funasr.md new file mode 100644 index 00000000..89a60c10 --- /dev/null +++ b/docs/engines/funasr.md @@ -0,0 +1,62 @@ +# VoiceStudio — FunASR (SenseVoice) Engine + +FunASR drives Alibaba's SenseVoiceSmall with FSMN-VAD: an all-in-one +multilingual pipeline — transcription with punctuation and inverse text +normalization across **50+ languages**, plus optional **inline speaker +diarization** via the cam++ speaker model. It's the opt-in alternative to +WhisperX ([#182](https://github.com/debpalash/VoiceStudio/issues/182)); +WhisperX remains the cross-platform default. + +## Selecting it + +- Install it into the app venv: `uv pip install funasr`. +- Then **Model Catalogue → Engines**, ASR tab → **Use** on the FunASR row, or + `OMNIVOICE_ASR_BACKEND=funasr`. + +Auto-detect never picks it; it's an explicit opt-in. + +## Best at + +- **Multi-speaker transcription without any HuggingFace token.** This is the + only ASR engine with diarization built in: cam++ labels each sentence + (`Speaker 1`, `Speaker 2`, ...) in the same pass — no gated pyannote + model, no license click-through. Compare + [diarization](../features/diarization.md) for the pyannote/WhisperX route + and what each buys you. +- **Broad language coverage** beyond Whisper's strongest languages, with + punctuation included. + +## Not suited for + +- **Lip-sync dubbing** — FunASR returns sentence-level timestamps, not + word-level ones. Use [whisperx](whisperx.md) / + [mlx-whisper](mlx-whisper.md) when word timing matters. + +## Platform support + +CUDA or CPU, on macOS, Windows, and Linux. + +## Model selection + +| Variable | Default | Role | +| --- | --- | --- | +| `ASR_MODEL_FUNASR` | `iic/SenseVoiceSmall` | main ASR model | +| `ASR_FUNASR_VAD` | `fsmn-vad` | VAD segmentation model | +| `ASR_FUNASR_SPK` | `cam++` | speaker model; set to empty (`ASR_FUNASR_SPK=`) to disable diarization and use the dub pipeline's pyannote/heuristic path instead | + +Weights download on first load (through FunASR's own model hub) — see +[downloading-models](../downloading-models.md). + +## Quirks + +- With the speaker model enabled, long recordings are transcribed in **one + call** and split by FunASR's internal VAD — cam++ assigns speaker cluster + IDs per call, so this is what keeps "Speaker 1" meaning the same person + across the whole file. +- The engine runs with `spk_mode="vad_segment"`: FunASR 1.3.1's default + (`punc_segment`) requires a separate punctuation model and crashes when + SenseVoice is loaded without one. +- SenseVoice's rich-token markup (language/emotion/event tags around the + text) is stripped from the output automatically. +- Language detection is automatic (`language: auto`); the detected language + is reported per file. diff --git a/docs/engines/gpt-sovits.md b/docs/engines/gpt-sovits.md new file mode 100644 index 00000000..0aa1f36d --- /dev/null +++ b/docs/engines/gpt-sovits.md @@ -0,0 +1,78 @@ +# VoiceStudio — GPT-SoVITS Engine + +GPT-SoVITS (RVC-Boss) is one of the most popular open-source voice-cloning +systems (57k+ GitHub stars, MIT-licensed). It does zero-shot and few-shot +cloning with excellent naturalness in Chinese, English, Japanese, Cantonese, +and Korean, and it is very fast (RTF ~0.014 on suitable hardware). + +Unlike VoiceStudio's other engines, GPT-SoVITS does not run inside the app. +It ships as a standalone API server, and VoiceStudio connects to it over +HTTP. + +## When to pick it + +- You already run (or want to run) a GPT-SoVITS server, e.g. with few-shot + fine-tuned voices. +- You need fast, natural cloning in zh/en/ja/yue/ko. + +## Setup + +1. Install and start the GPT-SoVITS API server (upstream project): + + ```bash + cd GPT-SoVITS + python api_v2.py -a 127.0.0.1 -p 9880 -c GPT_SoVITS/configs/tts_infer.yaml + ``` + +2. Select the engine via **Model Catalogue → Engines** or + `OMNIVOICE_TTS_BACKEND=gpt-sovits`. + +VoiceStudio marks the engine available only when the server responds +(2-second reachability probe). + +## Configuration + +| Variable | Default | Meaning | +| --- | --- | --- | +| `OMNIVOICE_GPTSOVITS_URL` | `http://127.0.0.1:9880` | API server URL | +| `OMNIVOICE_TRUSTED_NETWORKS` | (unset) | Required to allow a non-loopback server | + +**Remote servers:** by default VoiceStudio only talks to loopback addresses +— part of the local-first guarantee. To point at a server on another +machine (e.g. a GPU box on your LAN), add its network to +`OMNIVOICE_TRUSTED_NETWORKS`; otherwise the connection is refused as an +untrusted endpoint. + +Prefer `https://` (or a private tunnel such as Tailscale/WireGuard) for any +non-loopback server: with plain `http://` the text you synthesize and the +audio that comes back cross the network unencrypted. VoiceStudio does not +disable certificate verification, so a TLS endpoint needs a certificate the +system trusts. + +## Behaviour notes + +- Output is 32 kHz mono (server output is resampled if needed). +- Cloning passes your reference clip path and optional transcript to the + server; the reference path must be readable **by the server process**, so + remote servers need the clip on their own filesystem. +- Speed control is forwarded as the server's `speed_factor`. +- The GPU is whatever the GPT-SoVITS server itself uses (CUDA preferred); + VoiceStudio's side is just an HTTP client. + +## Known limits + +- Five languages only; for broader coverage use + [OmniVoice](omnivoice.md) ([languages.md](../languages.md)). +- No voice design; server availability is your responsibility — if the + server stops, generations fail with a "server not reachable" error. + +## Troubleshooting + +- "GPT-SoVITS server not reachable": start the server with the command + above, or fix `OMNIVOICE_GPTSOVITS_URL`. +- "endpoint is outside loopback or OMNIVOICE_TRUSTED_NETWORKS": see + Configuration above. +- Other issues: [install/troubleshooting.md](../install/troubleshooting.md). + +See also: [benchmarks.md](../benchmarks.md), +[expressive-speech.md](../expressive-speech.md). diff --git a/docs/engines/kittentts.md b/docs/engines/kittentts.md new file mode 100644 index 00000000..943a30fe --- /dev/null +++ b/docs/engines/kittentts.md @@ -0,0 +1,75 @@ +# VoiceStudio — KittenTTS Engine + +KittenTTS (KittenML) is the lightweight English "flash" tier: a 25–80 MB +ONNX model with 8 preset voices that runs realtime on any CPU — no torch, no +CUDA, no GPU of any kind. Use it when you just need quick English narration +(voiceovers, demo reads, short phrases) with no reference sample. + +## When to pick it + +- English-only content where speed and a tiny install matter more than + cloning. +- Machines with no usable GPU. + +The trade-off against [OmniVoice](omnivoice.md): no voice cloning, English +only — but a much faster and much smaller install. + +## Setup + +```bash +pip install kittentts +``` + +Then select the engine via **Model Catalogue → Engines** or +`OMNIVOICE_TTS_BACKEND=kittentts`. + +## Voices + +Eight preset voices, four male/female pairs: + +```text +expr-voice-2-m expr-voice-2-f (default: expr-voice-2-f) +expr-voice-3-m expr-voice-3-f +expr-voice-4-m expr-voice-4-f +expr-voice-5-m expr-voice-5-f +``` + +An unknown voice id logs an info message and falls back to the default. + +## Model selection + +| Variable | Default | Meaning | +| --- | --- | --- | +| `OMNIVOICE_KITTENTTS_MODEL` | `KittenML/kitten-tts-mini-0.8` | HuggingFace checkpoint to load | + +The ~80 MB model downloads from HuggingFace on first use (retried once on a +flaky connection). See [downloading-models.md](../downloading-models.md). + +## Behaviour notes + +- Output is 24 kHz mono. +- CPU-only by design — the ONNX graph has no CUDA/MPS path. +- Non-English `language` values are ignored with a log line pointing at + OmniVoice; reference audio is likewise ignored (no cloning). +- **Long-input hardening + ([#1173](https://github.com/debpalash/VoiceStudio/issues/1173)):** the + shipped ONNX graph has a hard 512-token cap, and phonemization can expand + text massively (digits especially). VoiceStudio pre-measures every chunk + with the model's own tokenizer and splits oversized chunks at word + boundaries, so long or digit-heavy inputs no longer abort inside + onnxruntime with an opaque "invalid expand shape" error. + +## Known limits + +- English only; no cloning, no voice design, no emotion controls + (see [expressive-speech.md](../expressive-speech.md)). +- Preset voices only — speed is the one knob. + +## Troubleshooting + +- Engine unavailable: `pip install kittentts` into VoiceStudio's Python + environment and restart. +- Other issues: [install/troubleshooting.md](../install/troubleshooting.md). + +See also: [benchmarks.md](../benchmarks.md), +[disk usage](disk-usage.md). diff --git a/docs/engines/mlx-audio.md b/docs/engines/mlx-audio.md new file mode 100644 index 00000000..07a40e97 --- /dev/null +++ b/docs/engines/mlx-audio.md @@ -0,0 +1,76 @@ +# VoiceStudio — MLX-Audio Engine (Apple Silicon) + +MLX-Audio (Blaizzy/mlx-audio) wraps 14+ TTS engines — Kokoro, CSM, Dia, +Qwen3-TTS, Chatterbox, MeloTTS, OuteTTS, and more — behind a single adapter +that runs on Apple's MLX framework. It is **Apple Silicon only**: the engine +is not shipped on Linux, Windows, or Intel Macs, and a stray wheel on those +platforms never reports as available +([#390](https://github.com/debpalash/VoiceStudio/issues/390)). + +## When to pick it + +- You're on an M-series Mac and want small, fast models tuned for it. +- You want one of the specific hosted models (Kokoro for small multilingual, + CSM for cloning, Qwen3-TTS for voice design, Dia for dialogue, …). + +## Setup + +```bash +pip install mlx-audio +``` + +Then select the engine via **Model Catalogue → Engines** or +`OMNIVOICE_TTS_BACKEND=mlx-audio`. + +## Model selection + +One backend hosts many models. The curated set: + +| Key | Model | Niche | +| --- | --- | --- | +| `kokoro` (default) | `mlx-community/Kokoro-82M-bf16` | small multilingual | +| `csm` | `mlx-community/csm-1b-8bit` | voice cloning | +| `qwen3-tts` | `mlx-community/Qwen3-TTS-12Hz-1.7B-VoiceDesign-4bit` | voice design | +| `dia` | `mlx-community/Dia-1.6B` | dialogue | +| `chatterbox` | `mlx-community/Chatterbox-TTS-4bit` | expressive | +| `melotts` | `mlx-community/MeloTTS-English-v3-MLX` | lightweight VITS | +| `outetts` | `mlx-community/Llama-OuteTTS-1.0-1B-4bit` | LM-based | + +Pick a model in the **Model Catalogue → Engines** curated picker +([#981](https://github.com/debpalash/VoiceStudio/issues/981)) or set +`OMNIVOICE_MLX_AUDIO_MODEL` to either a curated key (`kokoro`) or any full +HF repo id. The env var overrides the persisted UI choice. + +## Behaviour notes + +- Output is 24 kHz mono for most hosted models. +- **Cloning works only with the `csm` model** — it is the only curated model + confirmed to accept a reference clip. Other models silently ignore + reference audio, so the engine reports cloning support only when CSM is + selected (dub/batch jobs gate on this). +- Voice design (text description → voice) is available through the + Qwen3-TTS VoiceDesign model. +- Language support is per-model (Kokoro ~8 languages, others vary). An + unsupported language for Kokoro produces a clear error naming what it + does support ([#977](https://github.com/debpalash/VoiceStudio/issues/977)) + — leave language on Auto or switch to a multilingual engine. + +## Platform notes + +This engine is exempt from cross-platform parity as a platform-only +capability behind explicit opt-in: it exists only where Apple's MLX runtime +exists. On any other platform the engine picker shows it unavailable with +the reason. + +## Troubleshooting + +- Unavailable on an M-series Mac: `pip install mlx-audio` into + VoiceStudio's Python environment; in a packaged app build, MLX's native + libraries may fail to load — the engine reports unavailable rather than + crashing. +- Other issues: [install/troubleshooting.md](../install/troubleshooting.md). + +See also: [benchmarks.md](../benchmarks.md), +[languages.md](../languages.md), +[downloading-models.md](../downloading-models.md), +[disk usage](disk-usage.md). diff --git a/docs/engines/mlx-whisper.md b/docs/engines/mlx-whisper.md new file mode 100644 index 00000000..e4fddd92 --- /dev/null +++ b/docs/engines/mlx-whisper.md @@ -0,0 +1,57 @@ +# VoiceStudio — MLX Whisper Engine + +MLX Whisper runs Whisper on the Apple Silicon GPU via MLX. It exists because +CTranslate2 (whisperx / faster-whisper) has **no Metal build** — on a Mac +those engines transcribe on the CPU no matter what GPU is present. Measured +on an M2 with whisper-large-v3, one 30 s dub chunk: **90.4 s on WhisperX +(CPU) vs 20.5 s on MLX (GPU)** — which is why auto-detect picks MLX Whisper +on every Apple Silicon machine +([#1127](https://github.com/debpalash/VoiceStudio/issues/1127)). + +## Selecting it + +- Nothing to do on Apple Silicon — auto-detect prefers it there. +- Or explicitly: **Model Catalogue → Engines**, ASR tab → **Use**, or + `OMNIVOICE_ASR_BACKEND=mlx-whisper`. + +## Best at + +- **Dubbing on a Mac** — it layers the same wav2vec2 forced alignment + WhisperX uses on top of the GPU transcription, so word timing (±10–30 ms) + and therefore lip-sync accuracy are unchanged. Same model, same alignment, + ~4x the speed. +- **Dictation/capture** — the capture path automatically swaps in + `mlx-community/whisper-large-v3-turbo` (~5x faster than large-v3) unless a + sherpa dictation model or [parakeet-mlx](parakeet-mlx.md) is preferred. + +## Platform support + +**Apple Silicon only.** A shared platform gate refuses Linux, Windows, and +Intel Macs before any package import, so a stray `mlx-whisper` wheel on the +wrong platform never reports itself available +([#390](https://github.com/debpalash/VoiceStudio/issues/390)). All other +platforms use the CUDA/CPU engines instead. + +## Model selection + +- `ASR_MODEL` — default `mlx-community/whisper-large-v3-mlx`. Any MLX-format + Whisper repo works. Weights download on first load — see + [downloading-models](../downloading-models.md). +- `OMNIVOICE_ALIGN_DEVICE` — force the wav2vec2 aligner's device. The aligner + runs on MPS when it can and falls back to CPU; languages without a bundled + aligner (~20 major languages have one) keep Whisper's native word + timestamps. + +## Quirks + +- Audio is decoded through VoiceStudio's validated ffmpeg rather than the + bare `ffmpeg` PATH lookup mlx-whisper would do on its own — a clean + from-source install with no system ffmpeg works fine + ([#479](https://github.com/debpalash/VoiceStudio/issues/479)). +- The model is warmed into unified memory in the background, so the first + transcribe after startup doesn't pay the load cost. +- In a packaged app, a native MLX library that fails to load is reported as + "unavailable" (with fallback to another engine) rather than crashing the + engine list. + +Speed comparisons across engines live in [performance](../performance.md). diff --git a/docs/engines/moonshine.md b/docs/engines/moonshine.md new file mode 100644 index 00000000..02d85f59 --- /dev/null +++ b/docs/engines/moonshine.md @@ -0,0 +1,48 @@ +# VoiceStudio — Moonshine Engine + +Moonshine is an edge-optimized ASR family built for CPU-only machines. +Unlike Whisper it processes variable-length audio (no padding everything to +30 s), which keeps latency low on short clips — sub-200 ms class on capture +buffers. It's the lightest local option for quick transcription on hardware +where even int8 whisper-large is too slow. + +## Selecting it + +- Install one of the runtimes into the app venv: + `uv pip install moonshine-onnx` (lighter, tried first) or + `moonshine-voice`. +- Then **Model Catalogue → Engines**, ASR tab → **Use** on the Moonshine row, + or `OMNIVOICE_ASR_BACKEND=moonshine`. + +Auto-detect never picks it; it's an explicit opt-in. + +## Best at + +- **Quick notes and short-clip transcription on low-power CPU machines.** +- Environments where a sub-1 GB footprint matters more than word timing or + language coverage. + +## Not suited for + +- **Dubbing.** Output is plain text as a **single segment spanning the whole + file — no word or segment timestamps** — so there's nothing for lip-sync + or subtitle timing to work with. Use a Whisper-family engine or + [sherpa-onnx-asr](sherpa-onnx-asr.md) for those jobs. +- Multilingual work: results report English; for broad language coverage use + [whisperx](whisperx.md) or [funasr](funasr.md). + +## Platform support + +CPU only, by design — macOS, Windows, and Linux. It claims no GPU. + +## Model selection + +`ASR_MODEL_MOONSHINE` — default `moonshine/base`. Weights download on first +load — see [downloading-models](../downloading-models.md). + +## Quirks + +- The engine tries `moonshine_onnx` first and falls back to + `moonshine_voice` — installing either one is enough. +- Segment bounds are synthesized from the audio duration (start 0, end = + file length), since the model reports none. diff --git a/docs/engines/moss-tts-nano.md b/docs/engines/moss-tts-nano.md new file mode 100644 index 00000000..286891a5 --- /dev/null +++ b/docs/engines/moss-tts-nano.md @@ -0,0 +1,78 @@ +# VoiceStudio — MOSS-TTS-Nano Engine + +MOSS-TTS-Nano (OpenMOSS) is the low-resource, broad-language pick: a +100M-parameter autoregressive codec LM that runs realtime on a 4-core CPU — +no GPU required — with native 48 kHz output and 20 languages under an +Apache-2.0 license. It fills the "runs on a fanless laptop" tier while still +covering languages like Arabic, Hebrew, Persian, Korean, and Turkish. + +## When to pick it + +- CPU-only or low-power hardware, but you still need cloning and non-English + coverage. +- Your language is among: Chinese, English, German, Spanish, French, + Japanese, Italian, Hebrew, Korean, Russian, Persian, Arabic, Polish, + Portuguese, Czech, Danish, Swedish, Hungarian, Greek, Turkish. + +## Setup + +The package is **not on PyPI** — install it from the upstream repo into +VoiceStudio's Python environment: + +```bash +git clone https://github.com/OpenMOSS/MOSS-TTS-Nano.git +cd MOSS-TTS-Nano +uv pip install -e . +``` + +Then select the engine via **Model Catalogue → Engines** or +`OMNIVOICE_TTS_BACKEND=moss-tts-nano`. + +## Model selection + +| Variable | Default | Meaning | +| --- | --- | --- | +| `OMNIVOICE_MOSS_TTS_MODEL` | `OpenMOSS-Team/MOSS-TTS-Nano` | HuggingFace checkpoint to load | + +The first use downloads the weights (retried once on a truncated download). +See [downloading-models.md](../downloading-models.md). + +## Behaviour notes + +- **Cloning is reference-only**: pass a reference clip. Style instructions, + preset speakers, and speed control are not supported and are silently + ignored, so mixed-engine call sites keep working. +- The model emits 48 kHz stereo; VoiceStudio downmixes to mono, matching the + rest of the pipeline (the dub mixer treats TTS output as mono per + segment). +- Runs on CPU or CUDA. + +## Upstream is unpinned + +The upstream repo is installed straight from git with no pinned release, and +the model class it exports has changed before +([#1287](https://github.com/debpalash/VoiceStudio/issues/1287)). VoiceStudio +therefore verifies that a usable model class actually exists — not just that +the package imports — before reporting the engine as ready. If the engine +shows unavailable with a "does not expose a usable model class" message, +pull the latest upstream and re-run `uv pip install -e .`, or open an issue +with the version you have. + +## Known limits + +- No voice design, no instruct, no speed control — cloning from a reference + clip only. +- Quality sits below the large engines; see + [benchmarks.md](../benchmarks.md). + +## Troubleshooting + +- "moss_tts_nano package not installed": run the clone + `uv pip install -e .` + steps above. +- Entry-point errors after an upstream update: see "Upstream is unpinned" + above. +- General issues: [install/troubleshooting.md](../install/troubleshooting.md). + +See also: [languages.md](../languages.md), +[expressive-speech.md](../expressive-speech.md), +[disk usage](disk-usage.md). diff --git a/docs/engines/nemo-parakeet.md b/docs/engines/nemo-parakeet.md new file mode 100644 index 00000000..097be4f6 --- /dev/null +++ b/docs/engines/nemo-parakeet.md @@ -0,0 +1,58 @@ +# VoiceStudio — Parakeet TDT (NVIDIA NeMo) Engine + +NVIDIA's Parakeet TDT via the NeMo toolkit: a FastConformer encoder with a +Token-and-Duration Transducer decoder. It beats Whisper large-v3 on English +benchmarks (~6% WER) and supports **25 (mostly European) languages** with +automatic language detection. The 0.6B model is fast even on CPU — measured +RTF 0.08–0.23 on an Apple Silicon M2 CPU (2026-07-02), ~20x faster than +faster-whisper large-v3 int8 on the same host. + +## Do not install NeMo into the app venv + +`nemo_toolkit`'s ASR extras pin `transformers>=4.57,<4.58`, which conflicts +with VoiceStudio's own `transformers>=5.3` requirement and **will break the +backend** (ImportError on startup) if installed into the shared venv. There +is currently no safe in-app install path for this engine; in-app isolation +is tracked separately. + +If you want the Parakeet models without a separate environment, use these +instead — same model family, no NeMo dependency: + +- **Apple Silicon:** [parakeet-mlx](parakeet-mlx.md) (installed by default on + mac-ARM source installs). +- **Any platform, CPU:** [sherpa-onnx-asr](sherpa-onnx-asr.md) — its default + dictation model is an int8 ONNX export of Parakeet TDT v3. + +## Selecting it + +Only meaningful if you've set up `nemo_toolkit[asr]` in a **separate, +dedicated Python environment** that runs the backend: + +- **Model Catalogue → Engines**, ASR tab → **Use** on the Parakeet TDT row, or +- `OMNIVOICE_ASR_BACKEND=nemo-parakeet`. + +Auto-detect never picks it; it's an explicit opt-in. + +## Best at + +- **English and European-language transcription** where WER matters more + than word-level subtitle timing. +- **CPU-only hosts** — faster than realtime without any GPU. + +## Platform support + +CUDA or CPU (the old hard CUDA gate was removed — see the RTF numbers +above). Availability is a pure dependency check on `nemo.collections.asr`. + +## Model selection + +`ASR_MODEL_NEMO` — default `nvidia/parakeet-tdt-0.6b-v3`. Weights download +on first load — see [downloading-models](../downloading-models.md). + +## Quirks + +- Output is a **single segment** for the whole file (NeMo doesn't VAD-split + like Whisper), with word timestamps when the model exposes them — fine for + dictation and plain transcripts, not ideal for long-form subtitles. +- The detected language isn't exposed cleanly by NeMo, so results report + `en` regardless of the actual (auto-detected) language. diff --git a/docs/engines/omnivoice-gguf.md b/docs/engines/omnivoice-gguf.md new file mode 100644 index 00000000..5010f522 --- /dev/null +++ b/docs/engines/omnivoice-gguf.md @@ -0,0 +1,83 @@ +# VoiceStudio — OmniVoice GGUF Engine + +OmniVoice GGUF runs the same OmniVoice model as the [default +engine](omnivoice.md), but through a bundled native binary +(`bin/omnivoice-tts-`) loading quantized GGUF weights. It is +hardware-adaptive: a probe picks the quantization that fits your machine, so +small GPUs and CPU-only hosts get a working OmniVoice instead of a paging, +timing-out one. + +## When to pick it + +- Your GPU is below the default engine's 6 GB VRAM floor. +- CPU-only machines that still want OmniVoice's voice and language coverage. +- You want generation isolated in a separate process (a crash or leak never + takes the app down — each generation spawns the binary fresh). + +## Quantization selection + +Weights come from the `Serveurperso/OmniVoice-GGUF` HuggingFace repo, pinned +to an exact revision. The hardware probe selects: + +| Hardware | Quant | Approx. VRAM use | +| --- | --- | --- | +| 12 GB+ VRAM | BF16 | ~1.6 GB (quality-first) | +| 4–12 GB VRAM | Q8_0 | ~945 MB (recommended balance) | +| 1–4 GB VRAM | Q4_K_M | ~659 MB (minimal footprint) | +| CPU-only | Q4_K_M | RAM-bound, latency-tolerable | + +You can override the selection from Settings; overrides are allow-listed +against the same table (an F32 reference quant, ~3.2 GB, is override-only). + +## Setup + +Nothing to install: installer and CI builds bundle the binary for your +platform. Select the engine via **Model Catalogue → Engines** or +`OMNIVOICE_TTS_BACKEND=omnivoice-gguf`. The quant weights download on first +use (see [downloading-models.md](../downloading-models.md)) — install them +ahead of time from **Model Catalogue → Models** if you want the first +generation to be quick; a long first render is the download, not a hang. + +**Source checkouts:** the repo ships zero-byte placeholders in `bin/` — real +binaries come from CI or the installer. The engine detects a placeholder and +reports unavailable with instructions +([#1172](https://github.com/debpalash/VoiceStudio/issues/1172)) instead of +failing at spawn time; build one with +`scripts/build-omnivoice-tts.sh --platform ` or use the default +in-process engine. + +## Integrity and self-healing + +Before reporting ready, the engine: + +- verifies the binary against the SHA-256 manifest (`bin/checksums.sha256`); +- detects macOS Gatekeeper quarantine and prints the exact + `xattr -cr '/Applications/VoiceStudio.app'` fix; +- restores a missing execute bit (a git clone or zip extract on POSIX can + drop `+x`, which used to surface as a permission error mislabeled as + out-of-memory — [#437](https://github.com/debpalash/VoiceStudio/issues/437)). + The chmod runs only after the SHA check confirms it's the right file. + +## Behaviour notes + +- Output is 24 kHz mono — same model, same rate as in-process OmniVoice. +- Cloning from a reference clip (with optional transcript) and style + instructions are supported; no voice design. +- Same multilingual surface as OmniVoice ([languages.md](../languages.md)). +- Because generation runs in another process, the app's own GPU counters + don't see its allocations — diagnostics label it accordingly. + +| Variable | Default | Meaning | +| --- | --- | --- | +| `OMNIVOICE_GGUF_GENERATE_TIMEOUT_S` | (generous built-in) | Per-generation timeout for the spawned binary | + +## Troubleshooting + +- "GGUF binary missing": this build doesn't bundle the runtime for your + platform — use the default engine. +- Checksum mismatch or quarantine messages: follow the printed fix, or + reinstall. +- Other issues: [install/troubleshooting.md](../install/troubleshooting.md). + +See also: [benchmarks.md](../benchmarks.md), +[performance.md](../performance.md), [disk usage](disk-usage.md). diff --git a/docs/engines/omnivoice.md b/docs/engines/omnivoice.md new file mode 100644 index 00000000..90b60055 --- /dev/null +++ b/docs/engines/omnivoice.md @@ -0,0 +1,74 @@ +# VoiceStudio — OmniVoice Engine (default) + +OmniVoice (k2-fsa/OmniVoice) is VoiceStudio's default TTS engine — the one a +fresh install uses without any configuration. It does zero-shot voice cloning +across 600+ languages and outputs 24 kHz mono audio. Voice cloning, dubbing, +and dictation all run on it out of the box. + +## When to pick it + +- You want cloning plus the broadest language coverage (see + [languages.md](../languages.md)). +- You have a GPU (CUDA or Apple Silicon MPS) with ~6 GB VRAM or more. +- You just installed VoiceStudio — it's already selected. + +For low-VRAM or CPU-only machines, the +[OmniVoice GGUF](omnivoice-gguf.md) variant runs the same model through a +quantized native binary with a much smaller memory footprint. + +## Requirements + +- Runs on CUDA, MPS (Apple Silicon), or CPU — auto-detected. +- Recommended VRAM floor: **6 GB** on a dedicated GPU. This is the only + engine with a measured floor: on 4 GB cards (GTX 1650 Ti, Quadro P2000 — + issues [#1226](https://github.com/debpalash/VoiceStudio/issues/1226) / + [#1222](https://github.com/debpalash/VoiceStudio/issues/1222)) the driver + pages to system RAM and a render that should take seconds runs for minutes + until the compute budget kills it. The UI warns before you wait; nothing + hard-blocks, since short inputs can still fit. +- No extra install — the model ships with the app and downloads its weights + on first use (see [downloading-models.md](../downloading-models.md)). + +## Selecting the engine + +OmniVoice is the default, so normally there is nothing to do. If you switched +away and want it back: + +- **Model Catalogue → Engines**, or +- set `OMNIVOICE_TTS_BACKEND=omnivoice`. + +The env var overrides the persisted UI choice. + +## Behaviour notes + +- Weights load lazily on first use and are shared with the rest of the app + (dubbing, dictation) — the model is never double-loaded. +- On CUDA the model runs fp16 with `torch.compile`; a speech recognizer is + co-loaded for the cloning path. +- Output is 24 kHz mono; the shared mastering chain (highpass + compressor) + is tuned for this rate and applied automatically. +- Cloning takes a short reference clip (`ref_audio`); an optional transcript + of the clip improves conditioning. + +## Known limits + +- No voice design from a text description — use [VoxCPM2](voxcpm2.md) for + that. +- Below the 6 GB VRAM floor, expect very slow renders or budget timeouts; + prefer [OmniVoice GGUF](omnivoice-gguf.md) or a CPU engine such as + [PocketTTS](pockettts.md). + +## Troubleshooting + +- "Too heavy for the available compute" on a small GPU: see the VRAM floor + above — switch to OmniVoice GGUF or close other GPU apps. +- First generation is slow: the first call downloads multi-GB weights. To + keep the first render quick, install the model ahead of time from + **Model Catalogue → Models** — a long first generate is almost always the + download, not a hang. +- General install issues: [install/troubleshooting.md](../install/troubleshooting.md). + +See also: [benchmarks.md](../benchmarks.md), +[performance.md](../performance.md), +[expressive-speech.md](../expressive-speech.md), +[disk usage](disk-usage.md). diff --git a/docs/engines/parakeet-mlx.md b/docs/engines/parakeet-mlx.md new file mode 100644 index 00000000..22e43a71 --- /dev/null +++ b/docs/engines/parakeet-mlx.md @@ -0,0 +1,58 @@ +# VoiceStudio — Parakeet TDT v3 (MLX) Engine + +NVIDIA's Parakeet TDT v3 on the Apple Silicon GPU, via the small pure-Python +`parakeet-mlx` package. It gives Macs the Parakeet tier CUDA/CPU users get +through NeMo or sherpa-onnx: **25 European languages**, word timestamps from +the TDT decoder itself (no wav2vec2 alignment pass needed), ~1.2 GB download, +~2 GB unified memory, dictation-grade speed on the GPU. + +Unlike [nemo-parakeet](nemo-parakeet.md) it needs no `nemo_toolkit` (whose +transformers pin conflicts with the app's) — it is **installed by default on +Apple Silicon source installs since 0.3.22**. + +## Selecting it + +- **Model Catalogue → Engines**, ASR tab → **Use** on the Parakeet TDT v3 + (MLX) row, or `OMNIVOICE_ASR_BACKEND=parakeet-mlx`. +- **Dictation prefers it automatically**: once the model weights are + installed (Model Catalogue → Models — the auto-pick never triggers a + download), live dictation/capture uses it whenever your system language is + one of the 25 covered European languages. Other languages keep the + multilingual Whisper engine, so dictation coverage never regresses. + +## Best at + +- **Live dictation on a Mac** — TDT decoding is fast enough for the capture + path, at Parakeet's better-than-Whisper English WER. +- **European-language transcription** with word timestamps at a fraction of + whisper-large-v3's memory and compute. + +For languages outside the 25 (CJK, Arabic, ...), use +[mlx-whisper](mlx-whisper.md) instead. + +## Platform support + +**Apple Silicon only** — the same shared MLX platform gate as mlx-whisper +refuses Linux, Windows, and Intel Macs before any import +([#390](https://github.com/debpalash/VoiceStudio/issues/390)). It runs on the +unified-memory GPU; there is no CPU tier. + +## Model selection + +`ASR_MODEL_PARAKEET_MLX` — default `mlx-community/parakeet-tdt-0.6b-v3`. +Weights download on first load — see +[downloading-models](../downloading-models.md). + +## Quirks + +- Long files are processed in 120 s chunks internally to bound unified-memory + use; short dictation buffers and dub chunks are unaffected. +- Parakeet v3 auto-detects among its 25 languages but doesn't expose the + pick, so the reported language is the one you requested (or none) — it is + never hardcoded to English. +- Word timestamps are merged from the decoder's subword tokens — good for + subtitles and dictation; for lip-sync-critical dubbing the wav2vec2-aligned + engines ([mlx-whisper](mlx-whisper.md), [whisperx](whisperx.md)) remain the + accuracy tier. + +Speed comparisons across engines live in [performance](../performance.md). diff --git a/docs/engines/pockettts.md b/docs/engines/pockettts.md new file mode 100644 index 00000000..9084074a --- /dev/null +++ b/docs/engines/pockettts.md @@ -0,0 +1,82 @@ +# VoiceStudio — PocketTTS Engine + +PocketTTS (kyutai-labs/pocket-tts, 100M parameters) is the fastest-CPU-render +pick: small, low-latency, CPU-only, with zero-shot voice cloning from a +reference clip. It covers six languages — English, French, German, +Portuguese, Italian, Spanish — with one model per language, and measures +roughly 8–9x real-time on an Apple M3 Pro. + +It complements the quality engines: where they fall back to CPU, PocketTTS +is built for it. CPU-only is deliberate — upstream observes no GPU speedup +for this model. + +## When to pick it + +- CPU-only machines that need fast rendering *and* voice cloning. +- Latency-sensitive use (dictation-style, short utterances) in one of the + six languages. + +## Setup + +1. Install the optional dependency: + + ```bash + uv sync --extra pockettts + ``` + + (Or enable it from **Model Catalogue → Engines**.) + +2. **Accept the license in-app** + ([#1306](https://github.com/debpalash/VoiceStudio/issues/1306)). The code + is MIT and the weights are CC-BY-4.0, but the weights are **gated on + HuggingFace** behind an access agreement with an acceptable-use clause. + VoiceStudio surfaces this before first use: the engine stays unavailable + until you review and accept in **Model Catalogue → Engines → PocketTTS**. + You also need HuggingFace access to the gated repo (see + [downloading-models.md](../downloading-models.md) for token setup). + +3. Select the engine via **Model Catalogue → Engines** or + `OMNIVOICE_TTS_BACKEND=pockettts`. + +## Platform notes + +- Works on Linux, Windows, macOS Apple Silicon — CPU only everywhere. +- **Not available on Intel Macs**: the required PyTorch version has no + macOS x86_64 wheel. The engine reports this plainly instead of failing + mid-install. + +## Behaviour notes + +- Output is 24 kHz mono. +- Six languages, one model per language, chosen by the `language` you + request; cloning takes a short reference clip. +- Runs in a crash-isolated sidecar process (parent Python environment): a + wedged generation is hard-killed by a watchdog and its memory reclaimed — + something an in-process engine cannot do. +- The first use downloads the gated weights; the sidecar heartbeats + progress during the download so the watchdog doesn't fire. + +| Variable | Default | Meaning | +| --- | --- | --- | +| `OMNIVOICE_POCKETTTS_RECV_TIMEOUT_S` | `600` | Sidecar response deadline in seconds (min 30; cold loads download weights) | + +## Known limits + +- No voice design, no emotion controls + (see [expressive-speech.md](../expressive-speech.md)). +- Six languages only — for broader coverage use + [OmniVoice](omnivoice.md) ([languages.md](../languages.md)). +- Revoking the license acceptance takes effect immediately, without a + restart — subsequent generations refuse. + +## Troubleshooting + +- "pocket_tts package not installed": run the `uv sync` above. +- "license not accepted": open **Model Catalogue → Engines → PocketTTS** + and review/accept. +- Timeouts on a slow connection: raise + `OMNIVOICE_POCKETTTS_RECV_TIMEOUT_S` for the first (download-heavy) run. +- Other issues: [install/troubleshooting.md](../install/troubleshooting.md). + +See also: [benchmarks.md](../benchmarks.md), +[performance.md](../performance.md), [disk usage](disk-usage.md). diff --git a/docs/engines/pytorch-whisper.md b/docs/engines/pytorch-whisper.md new file mode 100644 index 00000000..7623911d --- /dev/null +++ b/docs/engines/pytorch-whisper.md @@ -0,0 +1,61 @@ +# VoiceStudio — PyTorch Whisper Engine + +Whisper through the plain `transformers` pipeline, riding torch itself. No +extra install — transformers ships with the app — and because it runs on +torch's own stack (including torch's bundled cuDNN 9), it works on machines +where the CTranslate2 engines can't load. It is also the engine that +genuinely uses **AMD ROCm** GPUs, so auto-detect picks it on ROCm hosts +([#1529](https://github.com/debpalash/VoiceStudio/issues/1529)). + +## Selecting it + +- **Model Catalogue → Engines**, ASR tab → **Use** on the PyTorch Whisper + row, or `OMNIVOICE_ASR_BACKEND=pytorch-whisper`. +- Auto-detect picks it on ROCm, and as the last resort everywhere else. + +## Best at + +- **ROCm dubbing/transcription** — the only Whisper engine that uses the HIP + GPU (CTranslate2 has no HIP build, MLX is Apple-only). +- **Rescue engine** when whisperx/faster-whisper can't load — e.g. the + missing-cuDNN-8 case + ([#255](https://github.com/debpalash/VoiceStudio/issues/255)) — since it + needs neither CTranslate2 nor cuDNN 8. + +For lip-sync-grade word timing prefer [whisperx](whisperx.md) or +[mlx-whisper](mlx-whisper.md); this engine returns the pipeline's own word +timestamps. + +## Platform support + +CUDA, Apple Silicon (MPS), ROCm (HIP), and CPU — wherever torch runs, on +macOS, Windows, and Linux. + +## Model selection + +`OMNIVOICE_PYTORCH_ASR_MODEL` — default `openai/whisper-large-v3-turbo`. Any +transformers-format Whisper repo works. Weights download on first load — see +[downloading-models](../downloading-models.md). + +## VRAM preflight + +whisper-large-v3-turbo needs roughly 3.2 GiB before generation adds its +workspace; loading it onto a nearly-full card "succeeds" and then the first +transcribe OOMs with zero segments. So on CUDA the engine checks free VRAM +against a 5 GB budget before loading and uses the CPU instead when the card +is too full (flush the TTS model to restore GPU-speed ASR). Disable with +`OMNIVOICE_ASR_VRAM_PREFLIGHT=0`. + +## Quirks + +- If the pipeline fails to import (`AutoFeatureExtractor` errors), the cause + is either an incomplete transformers install or a torch/torchvision + version mismatch — the error message names the exact reinstall command; + the trio has to move together at the pinned versions + ([#549](https://github.com/debpalash/VoiceStudio/issues/549), + [#1376](https://github.com/debpalash/VoiceStudio/issues/1376)). +- Transcribes are time-bounded like every local engine: + `OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S` (default 120 s per dub chunk), + `OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S` (default 300 s whole-file). + +Speed comparisons across engines live in [performance](../performance.md). diff --git a/docs/engines/sherpa-onnx-asr.md b/docs/engines/sherpa-onnx-asr.md new file mode 100644 index 00000000..765f5553 --- /dev/null +++ b/docs/engines/sherpa-onnx-asr.md @@ -0,0 +1,71 @@ +# VoiceStudio — Sherpa-ONNX Dictation Engine + +The k2-fsa/sherpa-onnx ONNX runtime as a **live dictation** engine: small +int8 models that transcribe faster than realtime on CPU, with identical +behavior on macOS (arm64 + x86_64), Windows, and Linux — no CUDA dependency. +Streaming models emit partial text frame-by-frame as you speak; offline +models re-transcribe a growing buffer on a short cadence, so you see live +partials either way. + +## Selecting it + +- Ensure `sherpa-onnx` is installed (`uv add sherpa-onnx` on source installs). +- Pick a dictation model in the app (Model Catalogue → Models lists the + curated set below), or **Model Catalogue → Engines**, ASR tab → **Use**, or + pin `OMNIVOICE_ASR_BACKEND=sherpa-onnx-asr`. +- `OMNIVOICE_SHERPA_ASR_MODEL` selects the model — default + `sherpa-parakeet-tdt-v3`. + +## Best at + +- **Live dictation on CPU** — the whole point of this engine. Fast partials, + automatic endpointing on silence, no GPU required. +- It also honors the regular offline `transcribe` contract, so any of its + models can transcribe a file — plain text, single segment, no word + timestamps, which makes it a dictation/notes tool rather than a dubbing + engine. + +## The 7 curated models + +| Id | Type | Languages | Download | +| --- | --- | --- | --- | +| `sherpa-parakeet-tdt-v3` (default) | offline | 25 European languages | 0.67 GB | +| `sherpa-parakeet-tdt-v2` | offline | English | 0.66 GB | +| `sherpa-zipformer-bilingual-zh-en` | streaming | Chinese + English | 0.20 GB | +| `sherpa-paraformer-bilingual-zh-en` | streaming | Chinese + English | 0.24 GB | +| `sherpa-zipformer-en-20m` | streaming | English | 0.044 GB | +| `sherpa-zipformer-zh-14m` | streaming | Chinese | 0.025 GB | +| `sherpa-whisper-tiny` | offline | 90+ languages (auto-detect) | 0.104 GB | + +Sizes are measured on-disk download sizes. Weights are int8 ONNX checkpoints +that download on first use through the same HF cache as everything else — +see [downloading-models](../downloading-models.md). Peak RAM for the 0.6B +Parakeets is noticeably higher than their download size (onnxruntime's arena +allocator holds onto freed blocks). + +## Platform support + +CPU on every platform, by the strict cross-platform default-parity rule. +`OMNIVOICE_SHERPA_ASR_PROVIDER` can override the ONNX provider on a verified +GPU build, but the default never diverges. + +## Tuning + +- `OMNIVOICE_SHERPA_ASR_THREADS` — decode threads (default 2; the 0.6B + Parakeets automatically use up to 4 when the host has the cores, so decode + keeps ahead of the speaker). +- `OMNIVOICE_DICTATION_ENDPOINT_R1` / `OMNIVOICE_DICTATION_ENDPOINT_R2` — + streaming endpoint rules in seconds (defaults 1.0 / 0.6: text commits + ~0.6 s after you stop speaking). Applied without a restart. + +## Quirks + +- The recognizer is **pre-warmed in the background** so the first dictation + session doesn't pay the 1.3–2.5 s ONNX session load + ([#888](https://github.com/debpalash/VoiceStudio/issues/888)); it's then + shared warm across sessions. +- On Apple Silicon, installing the [parakeet-mlx](parakeet-mlx.md) model + makes dictation prefer the GPU Parakeet automatically for the 25 covered + languages; an explicitly selected sherpa model still wins. +- The offline `transcribe` path reports `language: auto` — per-file language + detection is only meaningful for the Whisper Tiny model. diff --git a/docs/engines/sherpa-onnx.md b/docs/engines/sherpa-onnx.md new file mode 100644 index 00000000..6cd09aab --- /dev/null +++ b/docs/engines/sherpa-onnx.md @@ -0,0 +1,75 @@ +# VoiceStudio — Sherpa-ONNX Engine + +Sherpa-ONNX (k2-fsa/sherpa-onnx) is a unified C++ ONNX runtime that wraps +20+ TTS model families (VITS, MeloTTS, Piper, Kokoro, Matcha, and more) +behind one API, with pre-built wheels for Linux, Windows, and macOS (x86 and +ARM). You bring the model: point VoiceStudio at any downloaded sherpa-onnx +TTS model directory. + +## When to pick it + +- You want a specific community model (e.g. a Piper or VITS voice for your + language) that no other engine hosts. +- You need a dependable CPU engine with optional CUDA acceleration. + +## Setup + +1. Install the runtime: + + ```bash + pip install sherpa-onnx + ``` + +2. Download a TTS model from the + [sherpa-onnx releases](https://github.com/k2-fsa/sherpa-onnx/releases) + and unpack it somewhere permanent. + +3. Point VoiceStudio at the model directory and restart: + + ```bash + export OMNIVOICE_SHERPA_MODEL=/path/to/model-dir + ``` + +4. Select the engine via **Model Catalogue → Engines** or + `OMNIVOICE_TTS_BACKEND=sherpa-onnx`. + +The directory must contain `model.onnx` and `tokens.txt`. Sherpa-ONNX ships +no bundled default model, so the engine reports unavailable — with the +reason — until `OMNIVOICE_SHERPA_MODEL` points at a valid directory. (Before +this gate, selecting the engine unconfigured produced a failure mislabeled +as out-of-memory — +[#919](https://github.com/debpalash/VoiceStudio/issues/919).) + +## Configuration + +| Variable | Default | Meaning | +| --- | --- | --- | +| `OMNIVOICE_SHERPA_MODEL` | (unset) | Directory containing `model.onnx` + `tokens.txt` | + +## Behaviour notes + +- Output defaults to 22.05 kHz (the VITS default); once a model is loaded, + its own sample rate is used. +- CPU is the universal baseline; the CUDA onnxruntime provider is available + on Linux/Windows installs. +- **No cloning**: voices come from the model itself. Multi-speaker VITS + models select a voice by numeric speaker id; speed is supported. +- Languages depend entirely on the model you download. + +## Known limits + +- One model at a time — switching models means changing + `OMNIVOICE_SHERPA_MODEL` and restarting. +- No voice design, no reference-audio cloning, no emotion controls + (see [expressive-speech.md](../expressive-speech.md)). + +## Troubleshooting + +- "OMNIVOICE_SHERPA_MODEL not set" / "No model.onnx in …": follow Setup + above — the variable must point at the *unpacked* model directory, not + the archive. +- Other issues: [install/troubleshooting.md](../install/troubleshooting.md). + +See also: [benchmarks.md](../benchmarks.md), +[languages.md](../languages.md), +[disk usage](disk-usage.md). diff --git a/docs/engines/supertonic3.md b/docs/engines/supertonic3.md new file mode 100644 index 00000000..f1db1744 --- /dev/null +++ b/docs/engines/supertonic3.md @@ -0,0 +1,76 @@ +# VoiceStudio — Supertonic-3 Engine + +Supertonic-3 (Supertone Inc.) is a ~99M-parameter ONNX TTS engine covering +31 languages with 7 preset voices at native 44.1 kHz. It is CPU-only by +design — pure ONNX Runtime on the CPU execution provider, with no CUDA or +MPS path in the upstream SDK — and runs in its own sidecar process so +crashes and cold init never block the rest of VoiceStudio. + +## When to pick it + +- Broad language coverage on machines with no usable GPU. +- Preset-voice narration at a higher sample rate than the default engine. + +## Setup + +1. Install the optional dependency into VoiceStudio's environment: + + ```bash + uv sync --extra supertonic + ``` + + (Or enable it from **Model Catalogue → Engines**, which installs the + pinned `supertonic` wheel for you.) + +2. **Accept the license in-app.** First use is gated behind an explicit + acceptance dialog: the inference SDK is MIT, but the model weights are + **OpenRAIL-M**, which carries use restrictions. The engine stays + unavailable until you review and accept in **Model Catalogue → Engines → + Supertonic-3**. + +3. Select the engine via **Model Catalogue → Engines** or + `OMNIVOICE_TTS_BACKEND=supertonic3`. + +The first synthesis cold-downloads ~400 MB of model weights, pinned to an +exact HuggingFace revision SHA so the bytes match what the SDK was validated +against. See [downloading-models.md](../downloading-models.md). + +## Voices + +Seven preset voices are surfaced: `M1` (default), `M3`, `M4`, `M5`, `F3`, +`F4`, `F5`. The SDK itself accepts the full `M1`–`M5` / `F1`–`F5` set if a +caller passes one explicitly; unknown ids fall back to the default with a +log line. + +## Behaviour notes + +- Output is 44.1 kHz mono. +- Runs as a long-lived sidecar in the parent Python environment (its + dependencies — onnxruntime, numpy, soundfile — already match + VoiceStudio's pins); subsequent calls reuse the warm ONNX session. +- `speed` is clamped to 0.7–2.0; quality steps clamp to 5–12. +- Language is an ISO 639-1 code; Auto engages the SDK's multilingual + fallback. + +## Known limits + +- **No cloning and no voice design** — preset voices only. Dub/batch jobs + that need cloning won't select it. +- CPU-only: hardware acceleration is a property of the upstream SDK, not a + VoiceStudio limitation. +- OpenRAIL-M weights are not covered by VoiceStudio's blanket + commercial-use statement — review the model license terms in the + acceptance dialog. + +## Troubleshooting + +- "supertonic package not installed": run the `uv sync` above or enable + from the Model Catalogue. +- "license not accepted": open **Model Catalogue → Engines → Supertonic-3** + and accept. +- Other issues: [install/troubleshooting.md](../install/troubleshooting.md). + +See also: [benchmarks.md](../benchmarks.md), +[languages.md](../languages.md), +[expressive-speech.md](../expressive-speech.md), +[disk usage](disk-usage.md). diff --git a/docs/engines/voxcpm2.md b/docs/engines/voxcpm2.md new file mode 100644 index 00000000..556c9b5d --- /dev/null +++ b/docs/engines/voxcpm2.md @@ -0,0 +1,76 @@ +# VoiceStudio — VoxCPM2 Engine + +VoxCPM2 (OpenBMB) is the studio-quality option: native 48 kHz output, +zero-shot voice cloning, and — uniquely among VoiceStudio's engines — +**voice design**: creating a synthetic voice from a text description +("young female, warm tone, British accent") with no reference audio at all. + +## When to pick it + +- You want voice design without a reference clip. +- You want the highest output sample rate (48 kHz vs OmniVoice's 24 kHz). +- Your language is among its 30 supported languages: Arabic, Burmese, + Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, + Hindi, Indonesian, Italian, Japanese, Khmer, Korean, Lao, Malay, + Norwegian, Polish, Portuguese, Russian, Spanish, Swahili, Swedish, + Tagalog, Thai, Turkish, Vietnamese. + +## Requirements + +- Python ≥ 3.10, PyTorch ≥ 2.5. +- CUDA ≥ 12 recommended for full speed; MPS (Apple Silicon) and CPU also + work. + +## Setup + +Install the package into VoiceStudio's Python environment: + +```bash +pip install "voxcpm>=2.0.3" +``` + +That is a version **floor**, not a pin — an older install still works, but +the engine logs an upgrade hint at load time. Then select the engine via +**Model Catalogue → Engines** or `OMNIVOICE_TTS_BACKEND=voxcpm2`. + +## Model selection + +| Variable | Default | Meaning | +| --- | --- | --- | +| `OMNIVOICE_VOXCPM_MODEL` | `openbmb/VoxCPM2` | HuggingFace checkpoint to load | + +The first use downloads a multi-GB checkpoint from HuggingFace. A download +interrupted near the end used to abort the load outright +([#1224](https://github.com/debpalash/VoiceStudio/issues/1224)); the load is +now retried once with a fresh client. See +[downloading-models.md](../downloading-models.md). + +## Behaviour notes + +- **Voice design:** provide a description and no reference audio. +- **Cloning:** the reference clip is prepared before use (edge-silence trim + and length cap) so dead air in a raw clip doesn't condition the output; on + any prep problem the raw clip is used as-is. +- **Style instructions** are passed as an inline prefix to the text. +- VoxCPM2 emits mastered, studio-grade audio, so VoiceStudio **skips its + shared mastering chain** (which is tuned for 24 kHz engines) — only benign + loudness normalization applies. +- A trailing-silence guard trims long near-silent tails from generations, + keeping a short natural tail. + +## Known limits + +- Slower than the lightweight CPU engines — see + [benchmarks.md](../benchmarks.md) and [performance.md](../performance.md). +- Language coverage is 30 languages; for anything else use the default + [OmniVoice](omnivoice.md) engine ([languages.md](../languages.md)). + +## Troubleshooting + +- Engine shows unavailable: the `voxcpm` package isn't installed — run the + `pip install` above and restart VoiceStudio. +- Repeated first-download failures: check connectivity/HF access, then see + [install/troubleshooting.md](../install/troubleshooting.md). + +See also: [expressive-speech.md](../expressive-speech.md), +[disk usage](disk-usage.md). diff --git a/docs/engines/whisperx.md b/docs/engines/whisperx.md new file mode 100644 index 00000000..95c7a628 --- /dev/null +++ b/docs/engines/whisperx.md @@ -0,0 +1,80 @@ +# VoiceStudio — WhisperX Engine + +WhisperX is the default ASR engine on CUDA and plain-CPU hosts: faster-whisper +(CTranslate2) transcription plus a **wav2vec2 forced-alignment** pass that +snaps word boundaries to ±10–30 ms (Whisper's own timestamps are ±100–300 ms). +That word timing is what dubbing lip-sync depends on, which is why auto-detect +prefers it wherever CTranslate2 can use the GPU. + +## Selecting it + +- **Model Catalogue → Engines**, ASR tab → **Use** on the WhisperX row, or +- pin it with `OMNIVOICE_ASR_BACKEND=whisperx` (the env var always wins over + the Settings pick; with neither set, auto-detect chooses per-hardware). + +## Best at + +- **Dubbing** — the forced alignment is the accuracy tier lip-sync needs. +- **Batch transcription** with word-level subtitles. +- Multi-speaker work: it pairs with pyannote speaker diarization — see + [diarization](../features/diarization.md). + +## Platform support + +| Host | What happens | +| --- | --- | +| NVIDIA CUDA | GPU, float16 (degrades automatically, see below) | +| CPU (any OS) | int8 — works, but slow for large-v3 | +| Apple Silicon | CPU only — CTranslate2 has no Metal build, so auto-detect prefers [mlx-whisper](mlx-whisper.md) there ([#1127](https://github.com/debpalash/VoiceStudio/issues/1127)) | +| AMD ROCm | CPU only — CTranslate2 has no HIP build, so auto-detect prefers [pytorch-whisper](pytorch-whisper.md) there ([#1529](https://github.com/debpalash/VoiceStudio/issues/1529)) | + +## Model selection + +- `ASR_MODEL_WHISPERX` — default `large-v3`. Accepts the usual size aliases + (`tiny` … `large-v3`, `distil-large-v3`) or a full HF repo id. Weights + download on first load — see [downloading-models](../downloading-models.md). +- `OMNIVOICE_ALIGN_DEVICE` — force the wav2vec2 aligner's device. Aligners + exist for ~20 major languages; other languages keep Whisper's native word + timestamps instead of failing. + +## VRAM preflight and degradation + +Loading fp16 large-v3 onto a nearly-full 8 GB card dies as a *native* CUDA +abort — no Python exception, the whole backend goes down +([#723](https://github.com/debpalash/VoiceStudio/issues/723)). So before every +load the engine checks free VRAM against per-compute-type budgets +(float16 5.0 GB, int8_float16 3.5 GB, int8 3.0 GB, scaled down for smaller +models) and degrades the compute type — or falls to CPU int8 — instead of +starting a load that would kill the process. Disable with +`OMNIVOICE_ASR_VRAM_PREFLIGHT=0`. + +Two more fallback chains run at load time: + +- GPUs without efficient fp16 (older Maxwell/Pascal, GTX 16xx) raise a + compute-type error — the engine retries int8_float16, then int8 + ([#551](https://github.com/debpalash/VoiceStudio/issues/551)). +- A genuine CUDA OOM retries on CPU int8, so dubbing still completes + (slower, same model and accuracy). + +## Quirks + +- **cuDNN 8 required on CUDA.** CTranslate2 links cuDNN 8; if it's missing the + process fast-fails with no traceback, so the engine is reported unavailable + up front and selection falls through to pytorch-whisper, which uses torch's + own cuDNN 9 ([#1371](https://github.com/debpalash/VoiceStudio/issues/1371)). +- On some hardened Linux kernels CTranslate2's native library is rejected with + "cannot enable executable stack" — reported as unavailable, not a crash + ([#692](https://github.com/debpalash/VoiceStudio/issues/692)). +- A partially-installed environment (interrupted sync, antivirus quarantine) + can break WhisperX's deep import chain (whisperx → pyannote → + lightning_fabric). The engine is then reported unavailable with a repair + hint — reinstall, or `uv sync --reinstall` on a source checkout + ([#1185](https://github.com/debpalash/VoiceStudio/issues/1185)). +- Audio is decoded through VoiceStudio's validated ffmpeg, not a bare `ffmpeg` + PATH lookup ([#479](https://github.com/debpalash/VoiceStudio/issues/479)). +- Transcribes are time-bounded: each dub chunk by + `OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S` (default 120 s), whole files by + `OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S` (default 300 s). Raise them for very + long files on slow hardware. + +Speed comparisons across engines live in [performance](../performance.md).