docs(engines): a guide for every engine + index; fix two engine-metadata bugs (#1556)

* docs(engines): a guide for every engine + index; fix two engine-metadata bugs

21 new pages under docs/engines/ (10 TTS, 10 ASR, index README) — every
registered engine now has one: what it's for, platform support, model env
vars, quirks with issue refs. Linked from both READMEs' engine sections.

Code fixes found while verifying facts against the registries:
- KittenTTS docstring claimed default voice 'Jasper'; the code default is
  expr-voice-2-f
- the isolated-ASR sidecar read only ASR_MODEL_FW while the download
  preflight read ASR_MODEL_FASTER — set one and the other quietly used a
  different model; both now resolve ASR_MODEL_FW-override → ASR_MODEL_FASTER
- moonshine's install hint named 'useful-moonshine', a package the backend
  never imports; now moonshine-onnx / moonshine-voice

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(changelog): entries for the engine guides + sidecar model fix (#1556)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(engines): second-harvest fixes — TLS guidance, matrix/code alignment, CN counts

- README matrix aligned to gpu_compat (the code is the source of truth):
  CosyVoice macOS is CPU not MPS, IndexTTS and GGUF gain their real
  CUDA/CPU/MPS cells
- gpt-sovits guide: prefer https/tunnel for non-loopback servers, plaintext
  warning; first-use download guidance on both OmniVoice pages
- preflight empty-env fallback matches the sidecar (ASR_MODEL_FASTER='' no
  longer resolves a different repo)
- nano installs via uv pip; kitten log level wording; index links install
  guides incl. the Gatekeeper step; README_CN engine counts 16/11

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(readme-cn): the all-engines-local claim now excludes the remote client

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Palash Debnath
2026-08-15 03:55:59 +00:00
committed by GitHub
co-authored by Claude Fable 5
parent b79ba9bd3b
commit 030d5ea01f
27 changed files with 1494 additions and 18 deletions
+2
View File
@@ -22,8 +22,10 @@ the frozen-backend fallback mirror it for their toolchains.
### Docs
- The READMEs now lead with download buttons and a three-step first-clone walkthrough, and a new benchmarks page anchors measured per-engine/per-device numbers on the in-repo harness (#1555)
- Every engine now has its own guide — 21 new pages under docs/engines plus an index covering all 16 TTS and 11 ASR engines, linked from both READMEs (#1556)
### Fixed
- The crash-isolated ASR sidecar and its download preflight now agree on which model to load — setting the shared faster-whisper model variable applies to both variants instead of the sidecar quietly using a different one (#1556)
- "Ready" now requires the deep health probe (a working database-backed route), not just the identity probe — a backend whose install broke underneath can no longer be announced up while every real request fails (#1548)
- Supervisor restarts after repeat crashes now back off (immediate, then 5s, then 15s) instead of respawning back-to-back, so a tight crash loop can't burn the whole restart budget in seconds (#1548)
+5 -5
View File
@@ -218,7 +218,7 @@ Professional-grade voice AI, minus the subscription and the cloud. Convinced? [C
### 🗣️ TTS Engines
**16 engines, one picker.** VoiceStudio (default, 600+ languages) is always available; seven more are opt-in and auto-detected (CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX), plus eight lazy-installed opt-ins (IndexTTS 2.5, OmniVoice GGUF, OmniVoice subprocess, PocketTTS, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS). Switch in **Model Catalogue → Engines** — or from anywhere with <kbd>Ctrl</kbd>/<kbd>Cmd</kbd>+<kbd>E</kbd>; the choice applies everywhere synthesis happens.
**16 engines, one picker.** VoiceStudio (default, 600+ languages) is always available; seven more are opt-in and auto-detected (CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX), plus eight lazy-installed opt-ins (IndexTTS 2.5, OmniVoice GGUF, OmniVoice subprocess, PocketTTS, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS). Switch in **Model Catalogue → Engines** — or from anywhere with <kbd>Ctrl</kbd>/<kbd>Cmd</kbd>+<kbd>E</kbd>; the choice applies everywhere synthesis happens. **Every engine has its own guide: [docs/engines](docs/engines/README.md).**
<details>
<summary><b>📊 The full matrix</b> — 16 engines × platform × clone/instruct × license</summary>
@@ -228,15 +228,15 @@ Professional-grade voice AI, minus the subscription and the cloud. Convinced? [C
| Engine | Languages | Clone | Instruct | Linux | macOS ARM | Windows | License |
|--------|:---------:|:-----:|:--------:|:-----:|:---------:|:-------:|:-------:|
| **VoiceStudio** (default, powered by k2-fsa/OmniVoice) | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Built-in |
| **CosyVoice 3** | 9 + 18 dialects | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
| **CosyVoice 3** | 9 + 18 dialects | ✅ | ✅ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **GPT-SoVITS** | 5 | ✅ | — | ✅ CUDA/CPU | — | ✅ CUDA/CPU | MIT |
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
| **MOSS-TTS-Nano** | 20 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **KittenTTS** | English | — | — | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
| **MLX-Audio** (Kokoro, Qwen3-TTS, CSM, Dia, …) | Multi | Varies | Varies | ❌ | ✅ Native | ❌ | Varies |
| **Sherpa-ONNX** | 20+ | — | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **IndexTTS 2.5** ⚡ | ZH · EN · JA · ES · AR | ✅ | — | ✅ CUDA | — | ✅ CUDA | Bilibili model license¹ |
| **OmniVoice GGUF** ⚡ | 600+ | ✅ | ✅ | ✅ CPU | ✅ CPU | ✅ CPU | Built-in |
| **IndexTTS 2.5** ⚡ | ZH · EN · JA · ES · AR | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Bilibili model license¹ |
| **OmniVoice GGUF** ⚡ | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS/CPU | ✅ CUDA/CPU | Built-in |
| **OmniVoice (subprocess)** ⚡² | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Built-in |
| **PocketTTS** ⚡ (Kyutai) | EN · FR · DE · PT · IT · ES | ✅ | — | ✅ CPU | ✅ CPU | ✅ CPU | CC-BY-4.0 (gated)³ |
| **Supertonic 3** ⚡ | 31 | — | — | ✅ CPU | ✅ CPU | ✅ CPU | OpenRAIL-M |
@@ -275,7 +275,7 @@ another machine, set `OMNIVOICE_GPTSOVITS_URL` to its credential-free
### 🎧 ASR Engines
**11 engines** — they power dictation, video dubbing, and subtitles. **WhisperX** is the cross-platform default (~100 languages, word-level timing); the rest are opt-in and auto-detected. Switch in **Model Catalogue → Engines**. Ten run fully on-device; the eleventh (OpenAI-compatible) is an optional remote client for Qwen3-ASR or any compatible server.
**11 engines** — they power dictation, video dubbing, and subtitles. **WhisperX** is the cross-platform default (~100 languages, word-level timing); the rest are opt-in and auto-detected. Switch in **Model Catalogue → Engines**. Ten run fully on-device; the eleventh (OpenAI-compatible) is an optional remote client for Qwen3-ASR or any compatible server. **Per-engine guides: [docs/engines](docs/engines/README.md).**
<details>
<summary><b>📊 The full lineup</b> — 11 engines, what each is best at, and compute-type notes</summary>
+7 -7
View File
@@ -180,8 +180,8 @@ Hugging Face Token 的配置见
| **API 密钥** | 需要账号 | 本地流程不需要 |
| **GPU 支持** | 不适用(云端) | CUDA · Apple Silicon · ROCmLinux)· CPU |
| **桌面应用** | ❌ | ✅ macOS · Windows · Linux |
| **TTS 引擎** | 1 | **14** — [完整矩阵](#tts-engines) |
| **ASR 引擎** | 1 | **10** — [完整阵容](#asr-engines) |
| **TTS 引擎** | 1 | **16** — [完整矩阵](#tts-engines) |
| **ASR 引擎** | 1 | **11** — [完整阵容](#asr-engines) |
| **MCP 服务器** | ❌ | ✅ 可从 Claude、Cursor 及任何 MCP 客户端使用 |
| **自检** | ❌ | ✅ 诊断套件、错误日志、脱敏调试包 |
| **可定制** | ❌ 闭源 | ✅ 随你 Fork、扩展、发布 |
@@ -221,10 +221,10 @@ Hugging Face Token 的配置见
### 🗣️ TTS 引擎
**14 个引擎,一个选择器。** VoiceStudio(默认,支持 600+ 语言)始终可用;另有七个引擎可选装并自动检测(CosyVoice 3、GPT-SoVITS、VoxCPM2、MOSS-TTS-Nano、KittenTTS、MLX-Audio、Sherpa-ONNX),外加个按需延迟安装的重量级引擎(IndexTTS 2.5、OmniVoice GGUF、Supertonic 3、MOSS-TTS-v1.5、dots.tts、Confucius4-TTS)。在 **设置 → TTS 引擎** 中切换;所选引擎将应用于所有语音合成场景。
**16 个引擎,一个选择器。** VoiceStudio(默认,支持 600+ 语言)始终可用;另有七个引擎可选装并自动检测(CosyVoice 3、GPT-SoVITS、VoxCPM2、MOSS-TTS-Nano、KittenTTS、MLX-Audio、Sherpa-ONNX),外加个按需延迟安装的引擎(IndexTTS 2.5、OmniVoice GGUF、OmniVoice 子进程版、PocketTTS、Supertonic 3、MOSS-TTS-v1.5、dots.tts、Confucius4-TTS)。在 **设置 → TTS 引擎** 中切换;所选引擎将应用于所有语音合成场景。**每个引擎都有独立指南:[docs/engines](docs/engines/README.md)(英文)。**
<details>
<summary><b>📊 完整矩阵</b>——14 个引擎 × 平台 × 克隆/指令 × 许可证</summary>
<summary><b>📊 完整矩阵</b>——16 个引擎 × 平台 × 克隆/指令 × 许可证</summary>
<br/>
@@ -261,10 +261,10 @@ Hugging Face Token 的配置见
### 🎧 ASR 引擎
**10 个引擎**——它们驱动听写、视频配音和字幕。**WhisperX** 是跨平台的默认引擎(约 100 种语言,词级时间对齐);其余引擎均为可选装并自动检测。在 **设置 → 引擎** 中切换。个完全在本地设备上运行;第十个(OpenAI 兼容)是可选的远程客户端,可用于 Qwen3-ASR 或任何兼容的服务器。
**11 个引擎**——它们驱动听写、视频配音和字幕。**WhisperX** 是跨平台的默认引擎(约 100 种语言,词级时间对齐);其余引擎均为可选装并自动检测。在 **设置 → 引擎** 中切换。个完全在本地设备上运行;第十个(OpenAI 兼容)是可选的远程客户端,可用于 Qwen3-ASR 或任何兼容的服务器。
<details>
<summary><b>📊 完整阵容</b>——10 个引擎、各自的强项与计算类型说明</summary>
<summary><b>📊 完整阵容</b>——11 个引擎、各自的强项与计算类型说明</summary>
<br/>
@@ -281,7 +281,7 @@ Hugging Face Token 的配置见
| **sherpa-onnx**(实时听写) | `sherpa-onnx-asr` | 25 种欧洲语言 + 90+ | 实时、快于实时的听写——小体积流式/离线 ONNX 模型(Parakeet TDT v3/v2、流式 Zipformer 与 Paraformer、Whisper Tiny),CPU 运行,macOS / Windows / Linux 表现完全一致。在 **设置 → 语音** 中按模型选择。 |
| **OpenAI 兼容** ⚠️ 远程 | `openai-compat-asr` | 取决于服务器 | 当下通往 **Qwen3-ASR** 的路径(自托管服务器,无需等 transformers 支持)、任何 OpenAI 兼容的转录端点,或 OpenAI 官方 API——无需安装,在 **设置 → 引擎**(ASR 标签页)中配置并测试连接。音频会离开你的设备,发送到你指定的任何服务器;参见 [docs/engines/openai-compatible-asr.md](docs/engines/openai-compatible-asr.md)。 |
> Whisper 系列引擎覆盖约 100 种语言;**FunASR / SenseVoice** 额外提供一条多语言一体化路径,内置语音活动检测与行内说话人分离。**sherpa-onnx** 驱动实时听写的模型选择器——你边说,文字边出现。每个引擎都在本地设备上运行——无需 API 密钥,无需云端。
> Whisper 系列引擎覆盖约 100 种语言;**FunASR / SenseVoice** 额外提供一条多语言一体化路径,内置语音活动检测与行内说话人分离。**sherpa-onnx** 驱动实时听写的模型选择器——你边说,文字边出现。除可选的 OpenAI 兼容远程客户端外,所有引擎都在本地设备上运行——无需 API 密钥,无需云端。
> **GPU 不支持高效 float16** 在较老的 NVIDIA GPUMaxwell/Pascal、GTX 16xx)上,或在 CTranslate2/cuDNN 版本不匹配之后,CTranslate2 系 ASR 引擎(WhisperX、Faster-Whisper)无法运行 `float16`VoiceStudio 会自动改用 `int8` 重试——无需配置。如果转录仍然失败,可用 `ASR_COMPUTE_TYPE` 环境变量固定计算类型(逃生舱口):`ASR_COMPUTE_TYPE=int8`CPU 用 `float32`)。将其设为 `int8` 并重启后端。
+10 -1
View File
@@ -85,7 +85,16 @@ def _get_model():
global _model
if _model is None:
from faster_whisper import WhisperModel
name = os.environ.get("ASR_MODEL_FW", "large-v3")
# Same weights as in-process faster-whisper: ASR_MODEL_FASTER selects
# for BOTH variants, ASR_MODEL_FW stays as a sidecar-only override.
# Before this, the sidecar read only ASR_MODEL_FW while the download
# preflight read ASR_MODEL_FASTER — set one and the other variant (or
# the preflight) quietly used a different model.
name = (
os.environ.get("ASR_MODEL_FW")
or os.environ.get("ASR_MODEL_FASTER")
or "large-v3"
)
try:
import torch
device = "cuda" if torch.cuda.is_available() else "cpu"
+12 -4
View File
@@ -2325,7 +2325,7 @@ _INSTALL_HINTS: dict[str, str] = {
"mac-ARM source installs since 0.3.22. Parakeet TDT v3 on the GPU via "
"MLX: 25 European languages, word timestamps, ~2 GB unified memory.)"
),
"moonshine": "pip install useful-moonshine (edge/CPU-optimized ASR)",
"moonshine": "uv pip install moonshine-onnx (or moonshine-voice; edge/CPU-optimized ASR)",
"funasr": "pip install funasr (SenseVoiceSmall + FSMN-VAD; CUDA or CPU)",
"sherpa-onnx-asr": "uv add sherpa-onnx (ONNX live dictation; CPU, cross-platform)",
"openai-compat-asr": (
@@ -3120,10 +3120,18 @@ def _offline_asr_repo(backend_id: str | None = None) -> str | None:
bid = backend_id or active_backend_id()
if bid == "whisperx":
return _fw_repo(os.environ.get("ASR_MODEL_WHISPERX", "large-v3"))
if bid in ("faster-whisper", "faster-whisper-isolated"):
# The crash-isolated sidecar loads the SAME CT2 weights as in-process
# faster-whisper (it reuses the ASR_MODEL_FASTER selection).
if bid == "faster-whisper":
return _fw_repo(os.environ.get("ASR_MODEL_FASTER", _FASTER_WHISPER_DEFAULT))
if bid == "faster-whisper-isolated":
# Mirror the sidecar's own resolution (_asr_sidecar/main.py):
# ASR_MODEL_FW is a sidecar-only override, otherwise the shared
# ASR_MODEL_FASTER selection applies — so the preflight can never
# download a different repo than the sidecar will load.
return _fw_repo(
os.environ.get("ASR_MODEL_FW")
or os.environ.get("ASR_MODEL_FASTER")
or _FASTER_WHISPER_DEFAULT
)
if bid == "mlx-whisper":
return os.environ.get("ASR_MODEL", _MLX_MODEL_DEFAULT)
if bid == "parakeet-mlx":
+2 -1
View File
@@ -1092,7 +1092,8 @@ class KittenTTSBackend(TTSBackend):
- English only
- Much faster + much smaller install
Preset voice is chosen via `extras["voice"]` (defaults to "Jasper"). Any
Preset voice is chosen via `extras["voice"]` (defaults to DEFAULT_VOICE,
"expr-voice-2-f"). Any
`ref_audio` / `instruct` / `language` arg is ignored with a log line so
the common call-site doesn't need to know which engine it's talking to.
"""
+57
View File
@@ -0,0 +1,57 @@
# Engine guides
One page per engine: what it's for, what it needs, how to enable it, and its
quirks. Select engines in **Model Catalogue → Engines** (or quick-switch with
<kbd>Ctrl</kbd>/<kbd>Cmd</kbd>+<kbd>E</kbd>), or pin one with
`OMNIVOICE_TTS_BACKEND` / `OMNIVOICE_ASR_BACKEND`.
Measured speed/VRAM numbers live in [benchmarks](../benchmarks.md); what each
engine can do expressively in [expressive-speech](../expressive-speech.md);
sidecar disk footprints in [disk-usage](disk-usage.md); the bar a new engine
must clear in [engine-acceptance](../engine-acceptance.md).
New to VoiceStudio? Install the app first — [macOS](../install/macos.md)
(first launch needs the one-time right-click → **Open** Gatekeeper
approval), [Windows](../install/windows.md), [Linux](../install/linux.md),
[Docker](../install/docker.md).
## Text-to-speech
| Engine | Guide | Runs on | Cloning | Enabled by |
|---|---|---|---|---|
| VoiceStudio (OmniVoice) — **default** | [omnivoice](omnivoice.md) | CUDA · MPS · CPU | ✅ | installed by default |
| VoxCPM2 | [voxcpm2](voxcpm2.md) | CUDA · MPS · CPU | ✅ + voice design | `pip install "voxcpm>=2.0.3"` |
| MOSS-TTS-Nano | [moss-tts-nano](moss-tts-nano.md) | CUDA · CPU | ✅ (ref only) | clone + `uv pip install -e .` |
| KittenTTS | [kittentts](kittentts.md) | CPU | — (8 preset voices) | `pip install kittentts` |
| MLX-Audio (Kokoro, CSM, Dia, …) | [mlx-audio](mlx-audio.md) | Apple Silicon | model-dependent | `pip install mlx-audio` |
| CosyVoice 3 | [cosyvoice](cosyvoice.md) | CUDA · CPU | ✅ | clone + requirements |
| GPT-SoVITS | [gpt-sovits](gpt-sovits.md) | external server | ✅ | its own API server |
| Sherpa-ONNX | [sherpa-onnx](sherpa-onnx.md) | CUDA · CPU | — | `pip install sherpa-onnx` + model dir |
| IndexTTS 2.5 | [indextts](indextts.md) | CUDA · CPU | ✅ + emotion | one-click sidecar install |
| OmniVoice GGUF | [omnivoice-gguf](omnivoice-gguf.md) | CUDA · MPS · CPU | ✅ | bundled binary |
| Supertonic-3 | [supertonic3](supertonic3.md) | CPU | — (7 preset voices) | `uv sync --extra supertonic` + license |
| MOSS-TTS-v1.5 (8B) | [moss-tts-v15](moss-tts-v15.md) | CUDA · CPU | ✅ | clone + env var |
| dots.tts (2B) | [dots-tts](dots-tts.md) | CUDA · CPU (not Windows) | ✅ | clone + env var |
| OmniVoice (subprocess) | [omnivoice-subprocess](omnivoice-subprocess.md) | CUDA · MPS · CPU | ✅ | opt-in pick, no install |
| PocketTTS (Kyutai) | [pockettts](pockettts.md) | CPU (not Intel Mac) | ✅ | `uv sync --extra pockettts` + license |
| Confucius4-TTS | [confucius4-tts](confucius4-tts.md) | CUDA · CPU | ✅ | clone + env var |
## Speech-to-text
| Engine | Guide | Runs on | Best at | Enabled by |
|---|---|---|---|---|
| WhisperX | [whisperx](whisperx.md) | CUDA · CPU | dubbing (word timestamps + diarization) | installed by default |
| Faster-Whisper | [faster-whisper](faster-whisper.md) | CUDA · CPU | general transcription | installed by default |
| Faster-Whisper (isolated) | [faster-whisper-isolated](faster-whisper-isolated.md) | CUDA · CPU | unattended batches | opt-in pick |
| MLX Whisper | [mlx-whisper](mlx-whisper.md) | Apple Silicon | Mac default | `pip install mlx-whisper` |
| PyTorch Whisper | [pytorch-whisper](pytorch-whisper.md) | CUDA · MPS · CPU | ROCm hosts | installed by default |
| Parakeet TDT (NeMo) | [nemo-parakeet](nemo-parakeet.md) | CUDA · CPU | 25 languages, fast CPU | separate venv (never the app's) |
| Parakeet TDT (MLX) | [parakeet-mlx](parakeet-mlx.md) | Apple Silicon | dictation, 25 EU languages | default on mac-ARM source installs |
| Moonshine | [moonshine](moonshine.md) | CPU | edge/low-power, no timestamps | `pip install` (see guide) |
| FunASR (SenseVoice) | [funasr](funasr.md) | CUDA · CPU | 50+ languages, inline diarization | `pip install funasr` |
| Sherpa-ONNX dictation | [sherpa-onnx-asr](sherpa-onnx-asr.md) | CPU | live streaming dictation | curated model download |
| OpenAI-compatible (remote) | [openai-compatible-asr](openai-compatible-asr.md) | network | offloading to a server (audio leaves the machine) | Model Catalogue |
Speaker diarization is not an engine registry of its own — the dub pipeline
uses pyannote (HF-gated; see [diarization](../features/diarization.md)) and
FunASR can diarize inline with its `cam++` speaker model.
+61
View File
@@ -0,0 +1,61 @@
# VoiceStudio — Faster-Whisper (Crash-Isolated) Engine
The same CTranslate2 Whisper engine as [faster-whisper](faster-whisper.md),
run in a **separate child process** ("sidecar"). CTranslate2's GPU teardown
can segfault — the endemic faster-whisper crash — and a hung or crashed
transcribe in-process takes the whole backend down with it. Isolated, the
child can crash or be force-killed to reclaim a hung transcribe and its VRAM
while the backend stays up
([#730](https://github.com/debpalash/VoiceStudio/issues/730)).
There is nothing extra to install: the sidecar reuses the app's own venv —
only the process boundary is new.
## Selecting it
- **Model Catalogue → Engines**, ASR tab → **Use** on the crash-isolated row, or
- pin it with `OMNIVOICE_ASR_BACKEND=faster-whisper-isolated`.
It is never picked by auto-detect — it's an explicit opt-in escape hatch.
## Best at
- **Long batch runs** where one bad file must not kill the backend.
- Machines where in-process faster-whisper has crashed or hung before:
a sidecar crash fails only that job, and the next transcribe respawns a
fresh sidecar automatically.
## Platform support
Same as faster-whisper: CUDA float16 or CPU int8 on macOS, Windows, and
Linux. The sidecar picks cuda/cpu itself and walks the same
float16 → int8_float16 → int8 degrade chain on GPUs without efficient fp16
([#551](https://github.com/debpalash/VoiceStudio/issues/551)).
## Model selection
- `ASR_MODEL_FASTER` — the shared model selection, same as the in-process
engine: set it once and both variants load the same weights.
- `ASR_MODEL_FW` — optional sidecar-only override; when set it wins over
`ASR_MODEL_FASTER` for this engine. Default `large-v3`.
- `ASR_COMPUTE_TYPE` — optional: pin the sidecar to one CTranslate2 compute
type instead of the automatic degrade chain.
Weights download on first load — see
[downloading-models](../downloading-models.md).
## Trade-offs and quirks
- **Slightly slower per call** than in-process faster-whisper (IPC overhead);
the model stays warm inside the sidecar between calls, so the cost is per
request, not per chunk of audio.
- Word timestamps are Whisper-native (±100300 ms) — no forced alignment.
For dubbing lip-sync, use [whisperx](whisperx.md) or
[mlx-whisper](mlx-whisper.md).
- If the sidecar dies mid-transcription the job fails with a clear
"sidecar crashed" error and the backend stays up — retry to respawn.
- **cuDNN 8 is still required on CUDA** — same CTranslate2 requirement as the
in-process engine. It's checked up front so a missing cuDNN 8 shows as
"unavailable" in Model Catalogue → Engines instead of a sidecar that
silently fails every transcribe
([#1371](https://github.com/debpalash/VoiceStudio/issues/1371)).
+70
View File
@@ -0,0 +1,70 @@
# VoiceStudio — Faster-Whisper Engine
Faster-Whisper runs Whisper on CTranslate2 — the same transcription core
WhisperX uses, **without** the wav2vec2 forced-alignment pass. It's the safe
cross-platform fallback when whisperx isn't installed, and the capture/dictation
fallback on non-Apple machines.
## Selecting it
- **Model Catalogue → Engines**, ASR tab → **Use** on the Faster-Whisper row, or
- pin it with `OMNIVOICE_ASR_BACKEND=faster-whisper`.
Auto-detect only picks it when [whisperx](whisperx.md) is unavailable.
## Best at
- **Subtitles, dictation buffers, and batch transcription** where Whisper's
native word timing (±100300 ms) is good enough.
- For dubbing lip-sync, prefer [whisperx](whisperx.md) (or
[mlx-whisper](mlx-whisper.md) on Apple Silicon) — their forced alignment is
an order of magnitude tighter on word boundaries.
## Platform support
- **CUDA** — float16, with automatic degradation (below).
- **CPU** — int8 on macOS, Windows, and Linux.
- **Apple Silicon GPU / ROCm** — not supported: CTranslate2 has no Metal or
HIP build, so those hosts run on CPU
([#1529](https://github.com/debpalash/VoiceStudio/issues/1529)); auto-detect
routes them to mlx-whisper / pytorch-whisper instead.
## Model selection
`ASR_MODEL_FASTER` — default `Systran/faster-whisper-large-v3`. Accepts the
size aliases (`tiny``large-v3`, `distil-large-v3`) or any CTranslate2
Whisper repo on HF. Weights download on first load — see
[downloading-models](../downloading-models.md).
Segments are cleaned up by faster-whisper's built-in Silero VAD before
transcription.
## Degradation chains
- GPUs without efficient fp16 (older Maxwell/Pascal, GTX 16xx, or a
CTranslate2/cuDNN mismatch) fail at model construction with a compute-type
error; the engine walks float16 → int8_float16 → int8 instead of failing
every chunk ([#551](https://github.com/debpalash/VoiceStudio/issues/551)).
- A CUDA out-of-memory falls back to CPU (slower, same model and accuracy) —
flushing the resident TTS model frees VRAM for GPU-speed ASR
([#255](https://github.com/debpalash/VoiceStudio/issues/255)).
## Quirks
- **cuDNN 8 required on CUDA** — a missing cuDNN 8 would fast-fail the whole
process, so the engine checks up front and reports itself unavailable
instead ([#1371](https://github.com/debpalash/VoiceStudio/issues/1371)).
pytorch-whisper covers that case on torch's bundled cuDNN 9.
- On some hardened Linux kernels the CTranslate2 native library is rejected
with "cannot enable executable stack" (an OSError, not an ImportError) —
reported as unavailable rather than crashing engine selection
([#692](https://github.com/debpalash/VoiceStudio/issues/692)).
- CTranslate2's GPU teardown can rarely segfault the process at unload. If
you hit that, switch to the crash-isolated variant —
[faster-whisper-isolated](faster-whisper-isolated.md)
([#730](https://github.com/debpalash/VoiceStudio/issues/730)).
- Transcribes are time-bounded: `OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S`
(default 120 s per dub chunk) and `OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S`
(default 300 s whole-file).
Speed comparisons across engines live in [performance](../performance.md).
+62
View File
@@ -0,0 +1,62 @@
# VoiceStudio — FunASR (SenseVoice) Engine
FunASR drives Alibaba's SenseVoiceSmall with FSMN-VAD: an all-in-one
multilingual pipeline — transcription with punctuation and inverse text
normalization across **50+ languages**, plus optional **inline speaker
diarization** via the cam++ speaker model. It's the opt-in alternative to
WhisperX ([#182](https://github.com/debpalash/VoiceStudio/issues/182));
WhisperX remains the cross-platform default.
## Selecting it
- Install it into the app venv: `uv pip install funasr`.
- Then **Model Catalogue → Engines**, ASR tab → **Use** on the FunASR row, or
`OMNIVOICE_ASR_BACKEND=funasr`.
Auto-detect never picks it; it's an explicit opt-in.
## Best at
- **Multi-speaker transcription without any HuggingFace token.** This is the
only ASR engine with diarization built in: cam++ labels each sentence
(`Speaker 1`, `Speaker 2`, ...) in the same pass — no gated pyannote
model, no license click-through. Compare
[diarization](../features/diarization.md) for the pyannote/WhisperX route
and what each buys you.
- **Broad language coverage** beyond Whisper's strongest languages, with
punctuation included.
## Not suited for
- **Lip-sync dubbing** — FunASR returns sentence-level timestamps, not
word-level ones. Use [whisperx](whisperx.md) /
[mlx-whisper](mlx-whisper.md) when word timing matters.
## Platform support
CUDA or CPU, on macOS, Windows, and Linux.
## Model selection
| Variable | Default | Role |
| --- | --- | --- |
| `ASR_MODEL_FUNASR` | `iic/SenseVoiceSmall` | main ASR model |
| `ASR_FUNASR_VAD` | `fsmn-vad` | VAD segmentation model |
| `ASR_FUNASR_SPK` | `cam++` | speaker model; set to empty (`ASR_FUNASR_SPK=`) to disable diarization and use the dub pipeline's pyannote/heuristic path instead |
Weights download on first load (through FunASR's own model hub) — see
[downloading-models](../downloading-models.md).
## Quirks
- With the speaker model enabled, long recordings are transcribed in **one
call** and split by FunASR's internal VAD — cam++ assigns speaker cluster
IDs per call, so this is what keeps "Speaker 1" meaning the same person
across the whole file.
- The engine runs with `spk_mode="vad_segment"`: FunASR 1.3.1's default
(`punc_segment`) requires a separate punctuation model and crashes when
SenseVoice is loaded without one.
- SenseVoice's rich-token markup (language/emotion/event tags around the
text) is stripped from the output automatically.
- Language detection is automatic (`language: auto`); the detected language
is reported per file.
+78
View File
@@ -0,0 +1,78 @@
# VoiceStudio — GPT-SoVITS Engine
GPT-SoVITS (RVC-Boss) is one of the most popular open-source voice-cloning
systems (57k+ GitHub stars, MIT-licensed). It does zero-shot and few-shot
cloning with excellent naturalness in Chinese, English, Japanese, Cantonese,
and Korean, and it is very fast (RTF ~0.014 on suitable hardware).
Unlike VoiceStudio's other engines, GPT-SoVITS does not run inside the app.
It ships as a standalone API server, and VoiceStudio connects to it over
HTTP.
## When to pick it
- You already run (or want to run) a GPT-SoVITS server, e.g. with few-shot
fine-tuned voices.
- You need fast, natural cloning in zh/en/ja/yue/ko.
## Setup
1. Install and start the GPT-SoVITS API server (upstream project):
```bash
cd GPT-SoVITS
python api_v2.py -a 127.0.0.1 -p 9880 -c GPT_SoVITS/configs/tts_infer.yaml
```
2. Select the engine via **Model Catalogue → Engines** or
`OMNIVOICE_TTS_BACKEND=gpt-sovits`.
VoiceStudio marks the engine available only when the server responds
(2-second reachability probe).
## Configuration
| Variable | Default | Meaning |
| --- | --- | --- |
| `OMNIVOICE_GPTSOVITS_URL` | `http://127.0.0.1:9880` | API server URL |
| `OMNIVOICE_TRUSTED_NETWORKS` | (unset) | Required to allow a non-loopback server |
**Remote servers:** by default VoiceStudio only talks to loopback addresses
— part of the local-first guarantee. To point at a server on another
machine (e.g. a GPU box on your LAN), add its network to
`OMNIVOICE_TRUSTED_NETWORKS`; otherwise the connection is refused as an
untrusted endpoint.
Prefer `https://` (or a private tunnel such as Tailscale/WireGuard) for any
non-loopback server: with plain `http://` the text you synthesize and the
audio that comes back cross the network unencrypted. VoiceStudio does not
disable certificate verification, so a TLS endpoint needs a certificate the
system trusts.
## Behaviour notes
- Output is 32 kHz mono (server output is resampled if needed).
- Cloning passes your reference clip path and optional transcript to the
server; the reference path must be readable **by the server process**, so
remote servers need the clip on their own filesystem.
- Speed control is forwarded as the server's `speed_factor`.
- The GPU is whatever the GPT-SoVITS server itself uses (CUDA preferred);
VoiceStudio's side is just an HTTP client.
## Known limits
- Five languages only; for broader coverage use
[OmniVoice](omnivoice.md) ([languages.md](../languages.md)).
- No voice design; server availability is your responsibility — if the
server stops, generations fail with a "server not reachable" error.
## Troubleshooting
- "GPT-SoVITS server not reachable": start the server with the command
above, or fix `OMNIVOICE_GPTSOVITS_URL`.
- "endpoint is outside loopback or OMNIVOICE_TRUSTED_NETWORKS": see
Configuration above.
- Other issues: [install/troubleshooting.md](../install/troubleshooting.md).
See also: [benchmarks.md](../benchmarks.md),
[expressive-speech.md](../expressive-speech.md).
+75
View File
@@ -0,0 +1,75 @@
# VoiceStudio — KittenTTS Engine
KittenTTS (KittenML) is the lightweight English "flash" tier: a 2580 MB
ONNX model with 8 preset voices that runs realtime on any CPU — no torch, no
CUDA, no GPU of any kind. Use it when you just need quick English narration
(voiceovers, demo reads, short phrases) with no reference sample.
## When to pick it
- English-only content where speed and a tiny install matter more than
cloning.
- Machines with no usable GPU.
The trade-off against [OmniVoice](omnivoice.md): no voice cloning, English
only — but a much faster and much smaller install.
## Setup
```bash
pip install kittentts
```
Then select the engine via **Model Catalogue → Engines** or
`OMNIVOICE_TTS_BACKEND=kittentts`.
## Voices
Eight preset voices, four male/female pairs:
```text
expr-voice-2-m expr-voice-2-f (default: expr-voice-2-f)
expr-voice-3-m expr-voice-3-f
expr-voice-4-m expr-voice-4-f
expr-voice-5-m expr-voice-5-f
```
An unknown voice id logs an info message and falls back to the default.
## Model selection
| Variable | Default | Meaning |
| --- | --- | --- |
| `OMNIVOICE_KITTENTTS_MODEL` | `KittenML/kitten-tts-mini-0.8` | HuggingFace checkpoint to load |
The ~80 MB model downloads from HuggingFace on first use (retried once on a
flaky connection). See [downloading-models.md](../downloading-models.md).
## Behaviour notes
- Output is 24 kHz mono.
- CPU-only by design — the ONNX graph has no CUDA/MPS path.
- Non-English `language` values are ignored with a log line pointing at
OmniVoice; reference audio is likewise ignored (no cloning).
- **Long-input hardening
([#1173](https://github.com/debpalash/VoiceStudio/issues/1173)):** the
shipped ONNX graph has a hard 512-token cap, and phonemization can expand
text massively (digits especially). VoiceStudio pre-measures every chunk
with the model's own tokenizer and splits oversized chunks at word
boundaries, so long or digit-heavy inputs no longer abort inside
onnxruntime with an opaque "invalid expand shape" error.
## Known limits
- English only; no cloning, no voice design, no emotion controls
(see [expressive-speech.md](../expressive-speech.md)).
- Preset voices only — speed is the one knob.
## Troubleshooting
- Engine unavailable: `pip install kittentts` into VoiceStudio's Python
environment and restart.
- Other issues: [install/troubleshooting.md](../install/troubleshooting.md).
See also: [benchmarks.md](../benchmarks.md),
[disk usage](disk-usage.md).
+76
View File
@@ -0,0 +1,76 @@
# VoiceStudio — MLX-Audio Engine (Apple Silicon)
MLX-Audio (Blaizzy/mlx-audio) wraps 14+ TTS engines — Kokoro, CSM, Dia,
Qwen3-TTS, Chatterbox, MeloTTS, OuteTTS, and more — behind a single adapter
that runs on Apple's MLX framework. It is **Apple Silicon only**: the engine
is not shipped on Linux, Windows, or Intel Macs, and a stray wheel on those
platforms never reports as available
([#390](https://github.com/debpalash/VoiceStudio/issues/390)).
## When to pick it
- You're on an M-series Mac and want small, fast models tuned for it.
- You want one of the specific hosted models (Kokoro for small multilingual,
CSM for cloning, Qwen3-TTS for voice design, Dia for dialogue, …).
## Setup
```bash
pip install mlx-audio
```
Then select the engine via **Model Catalogue → Engines** or
`OMNIVOICE_TTS_BACKEND=mlx-audio`.
## Model selection
One backend hosts many models. The curated set:
| Key | Model | Niche |
| --- | --- | --- |
| `kokoro` (default) | `mlx-community/Kokoro-82M-bf16` | small multilingual |
| `csm` | `mlx-community/csm-1b-8bit` | voice cloning |
| `qwen3-tts` | `mlx-community/Qwen3-TTS-12Hz-1.7B-VoiceDesign-4bit` | voice design |
| `dia` | `mlx-community/Dia-1.6B` | dialogue |
| `chatterbox` | `mlx-community/Chatterbox-TTS-4bit` | expressive |
| `melotts` | `mlx-community/MeloTTS-English-v3-MLX` | lightweight VITS |
| `outetts` | `mlx-community/Llama-OuteTTS-1.0-1B-4bit` | LM-based |
Pick a model in the **Model Catalogue → Engines** curated picker
([#981](https://github.com/debpalash/VoiceStudio/issues/981)) or set
`OMNIVOICE_MLX_AUDIO_MODEL` to either a curated key (`kokoro`) or any full
HF repo id. The env var overrides the persisted UI choice.
## Behaviour notes
- Output is 24 kHz mono for most hosted models.
- **Cloning works only with the `csm` model** — it is the only curated model
confirmed to accept a reference clip. Other models silently ignore
reference audio, so the engine reports cloning support only when CSM is
selected (dub/batch jobs gate on this).
- Voice design (text description → voice) is available through the
Qwen3-TTS VoiceDesign model.
- Language support is per-model (Kokoro ~8 languages, others vary). An
unsupported language for Kokoro produces a clear error naming what it
does support ([#977](https://github.com/debpalash/VoiceStudio/issues/977))
— leave language on Auto or switch to a multilingual engine.
## Platform notes
This engine is exempt from cross-platform parity as a platform-only
capability behind explicit opt-in: it exists only where Apple's MLX runtime
exists. On any other platform the engine picker shows it unavailable with
the reason.
## Troubleshooting
- Unavailable on an M-series Mac: `pip install mlx-audio` into
VoiceStudio's Python environment; in a packaged app build, MLX's native
libraries may fail to load — the engine reports unavailable rather than
crashing.
- Other issues: [install/troubleshooting.md](../install/troubleshooting.md).
See also: [benchmarks.md](../benchmarks.md),
[languages.md](../languages.md),
[downloading-models.md](../downloading-models.md),
[disk usage](disk-usage.md).
+57
View File
@@ -0,0 +1,57 @@
# VoiceStudio — MLX Whisper Engine
MLX Whisper runs Whisper on the Apple Silicon GPU via MLX. It exists because
CTranslate2 (whisperx / faster-whisper) has **no Metal build** — on a Mac
those engines transcribe on the CPU no matter what GPU is present. Measured
on an M2 with whisper-large-v3, one 30 s dub chunk: **90.4 s on WhisperX
(CPU) vs 20.5 s on MLX (GPU)** — which is why auto-detect picks MLX Whisper
on every Apple Silicon machine
([#1127](https://github.com/debpalash/VoiceStudio/issues/1127)).
## Selecting it
- Nothing to do on Apple Silicon — auto-detect prefers it there.
- Or explicitly: **Model Catalogue → Engines**, ASR tab → **Use**, or
`OMNIVOICE_ASR_BACKEND=mlx-whisper`.
## Best at
- **Dubbing on a Mac** — it layers the same wav2vec2 forced alignment
WhisperX uses on top of the GPU transcription, so word timing (±1030 ms)
and therefore lip-sync accuracy are unchanged. Same model, same alignment,
~4x the speed.
- **Dictation/capture** — the capture path automatically swaps in
`mlx-community/whisper-large-v3-turbo` (~5x faster than large-v3) unless a
sherpa dictation model or [parakeet-mlx](parakeet-mlx.md) is preferred.
## Platform support
**Apple Silicon only.** A shared platform gate refuses Linux, Windows, and
Intel Macs before any package import, so a stray `mlx-whisper` wheel on the
wrong platform never reports itself available
([#390](https://github.com/debpalash/VoiceStudio/issues/390)). All other
platforms use the CUDA/CPU engines instead.
## Model selection
- `ASR_MODEL` — default `mlx-community/whisper-large-v3-mlx`. Any MLX-format
Whisper repo works. Weights download on first load — see
[downloading-models](../downloading-models.md).
- `OMNIVOICE_ALIGN_DEVICE` — force the wav2vec2 aligner's device. The aligner
runs on MPS when it can and falls back to CPU; languages without a bundled
aligner (~20 major languages have one) keep Whisper's native word
timestamps.
## Quirks
- Audio is decoded through VoiceStudio's validated ffmpeg rather than the
bare `ffmpeg` PATH lookup mlx-whisper would do on its own — a clean
from-source install with no system ffmpeg works fine
([#479](https://github.com/debpalash/VoiceStudio/issues/479)).
- The model is warmed into unified memory in the background, so the first
transcribe after startup doesn't pay the load cost.
- In a packaged app, a native MLX library that fails to load is reported as
"unavailable" (with fallback to another engine) rather than crashing the
engine list.
Speed comparisons across engines live in [performance](../performance.md).
+48
View File
@@ -0,0 +1,48 @@
# VoiceStudio — Moonshine Engine
Moonshine is an edge-optimized ASR family built for CPU-only machines.
Unlike Whisper it processes variable-length audio (no padding everything to
30 s), which keeps latency low on short clips — sub-200 ms class on capture
buffers. It's the lightest local option for quick transcription on hardware
where even int8 whisper-large is too slow.
## Selecting it
- Install one of the runtimes into the app venv:
`uv pip install moonshine-onnx` (lighter, tried first) or
`moonshine-voice`.
- Then **Model Catalogue → Engines**, ASR tab → **Use** on the Moonshine row,
or `OMNIVOICE_ASR_BACKEND=moonshine`.
Auto-detect never picks it; it's an explicit opt-in.
## Best at
- **Quick notes and short-clip transcription on low-power CPU machines.**
- Environments where a sub-1 GB footprint matters more than word timing or
language coverage.
## Not suited for
- **Dubbing.** Output is plain text as a **single segment spanning the whole
file — no word or segment timestamps** — so there's nothing for lip-sync
or subtitle timing to work with. Use a Whisper-family engine or
[sherpa-onnx-asr](sherpa-onnx-asr.md) for those jobs.
- Multilingual work: results report English; for broad language coverage use
[whisperx](whisperx.md) or [funasr](funasr.md).
## Platform support
CPU only, by design — macOS, Windows, and Linux. It claims no GPU.
## Model selection
`ASR_MODEL_MOONSHINE` — default `moonshine/base`. Weights download on first
load — see [downloading-models](../downloading-models.md).
## Quirks
- The engine tries `moonshine_onnx` first and falls back to
`moonshine_voice` — installing either one is enough.
- Segment bounds are synthesized from the audio duration (start 0, end =
file length), since the model reports none.
+78
View File
@@ -0,0 +1,78 @@
# VoiceStudio — MOSS-TTS-Nano Engine
MOSS-TTS-Nano (OpenMOSS) is the low-resource, broad-language pick: a
100M-parameter autoregressive codec LM that runs realtime on a 4-core CPU —
no GPU required — with native 48 kHz output and 20 languages under an
Apache-2.0 license. It fills the "runs on a fanless laptop" tier while still
covering languages like Arabic, Hebrew, Persian, Korean, and Turkish.
## When to pick it
- CPU-only or low-power hardware, but you still need cloning and non-English
coverage.
- Your language is among: Chinese, English, German, Spanish, French,
Japanese, Italian, Hebrew, Korean, Russian, Persian, Arabic, Polish,
Portuguese, Czech, Danish, Swedish, Hungarian, Greek, Turkish.
## Setup
The package is **not on PyPI** — install it from the upstream repo into
VoiceStudio's Python environment:
```bash
git clone https://github.com/OpenMOSS/MOSS-TTS-Nano.git
cd MOSS-TTS-Nano
uv pip install -e .
```
Then select the engine via **Model Catalogue → Engines** or
`OMNIVOICE_TTS_BACKEND=moss-tts-nano`.
## Model selection
| Variable | Default | Meaning |
| --- | --- | --- |
| `OMNIVOICE_MOSS_TTS_MODEL` | `OpenMOSS-Team/MOSS-TTS-Nano` | HuggingFace checkpoint to load |
The first use downloads the weights (retried once on a truncated download).
See [downloading-models.md](../downloading-models.md).
## Behaviour notes
- **Cloning is reference-only**: pass a reference clip. Style instructions,
preset speakers, and speed control are not supported and are silently
ignored, so mixed-engine call sites keep working.
- The model emits 48 kHz stereo; VoiceStudio downmixes to mono, matching the
rest of the pipeline (the dub mixer treats TTS output as mono per
segment).
- Runs on CPU or CUDA.
## Upstream is unpinned
The upstream repo is installed straight from git with no pinned release, and
the model class it exports has changed before
([#1287](https://github.com/debpalash/VoiceStudio/issues/1287)). VoiceStudio
therefore verifies that a usable model class actually exists — not just that
the package imports — before reporting the engine as ready. If the engine
shows unavailable with a "does not expose a usable model class" message,
pull the latest upstream and re-run `uv pip install -e .`, or open an issue
with the version you have.
## Known limits
- No voice design, no instruct, no speed control — cloning from a reference
clip only.
- Quality sits below the large engines; see
[benchmarks.md](../benchmarks.md).
## Troubleshooting
- "moss_tts_nano package not installed": run the clone + `uv pip install -e .`
steps above.
- Entry-point errors after an upstream update: see "Upstream is unpinned"
above.
- General issues: [install/troubleshooting.md](../install/troubleshooting.md).
See also: [languages.md](../languages.md),
[expressive-speech.md](../expressive-speech.md),
[disk usage](disk-usage.md).
+58
View File
@@ -0,0 +1,58 @@
# VoiceStudio — Parakeet TDT (NVIDIA NeMo) Engine
NVIDIA's Parakeet TDT via the NeMo toolkit: a FastConformer encoder with a
Token-and-Duration Transducer decoder. It beats Whisper large-v3 on English
benchmarks (~6% WER) and supports **25 (mostly European) languages** with
automatic language detection. The 0.6B model is fast even on CPU — measured
RTF 0.080.23 on an Apple Silicon M2 CPU (2026-07-02), ~20x faster than
faster-whisper large-v3 int8 on the same host.
## Do not install NeMo into the app venv
`nemo_toolkit`'s ASR extras pin `transformers>=4.57,<4.58`, which conflicts
with VoiceStudio's own `transformers>=5.3` requirement and **will break the
backend** (ImportError on startup) if installed into the shared venv. There
is currently no safe in-app install path for this engine; in-app isolation
is tracked separately.
If you want the Parakeet models without a separate environment, use these
instead — same model family, no NeMo dependency:
- **Apple Silicon:** [parakeet-mlx](parakeet-mlx.md) (installed by default on
mac-ARM source installs).
- **Any platform, CPU:** [sherpa-onnx-asr](sherpa-onnx-asr.md) — its default
dictation model is an int8 ONNX export of Parakeet TDT v3.
## Selecting it
Only meaningful if you've set up `nemo_toolkit[asr]` in a **separate,
dedicated Python environment** that runs the backend:
- **Model Catalogue → Engines**, ASR tab → **Use** on the Parakeet TDT row, or
- `OMNIVOICE_ASR_BACKEND=nemo-parakeet`.
Auto-detect never picks it; it's an explicit opt-in.
## Best at
- **English and European-language transcription** where WER matters more
than word-level subtitle timing.
- **CPU-only hosts** — faster than realtime without any GPU.
## Platform support
CUDA or CPU (the old hard CUDA gate was removed — see the RTF numbers
above). Availability is a pure dependency check on `nemo.collections.asr`.
## Model selection
`ASR_MODEL_NEMO` — default `nvidia/parakeet-tdt-0.6b-v3`. Weights download
on first load — see [downloading-models](../downloading-models.md).
## Quirks
- Output is a **single segment** for the whole file (NeMo doesn't VAD-split
like Whisper), with word timestamps when the model exposes them — fine for
dictation and plain transcripts, not ideal for long-form subtitles.
- The detected language isn't exposed cleanly by NeMo, so results report
`en` regardless of the actual (auto-detected) language.
+83
View File
@@ -0,0 +1,83 @@
# VoiceStudio — OmniVoice GGUF Engine
OmniVoice GGUF runs the same OmniVoice model as the [default
engine](omnivoice.md), but through a bundled native binary
(`bin/omnivoice-tts-<platform>`) loading quantized GGUF weights. It is
hardware-adaptive: a probe picks the quantization that fits your machine, so
small GPUs and CPU-only hosts get a working OmniVoice instead of a paging,
timing-out one.
## When to pick it
- Your GPU is below the default engine's 6 GB VRAM floor.
- CPU-only machines that still want OmniVoice's voice and language coverage.
- You want generation isolated in a separate process (a crash or leak never
takes the app down — each generation spawns the binary fresh).
## Quantization selection
Weights come from the `Serveurperso/OmniVoice-GGUF` HuggingFace repo, pinned
to an exact revision. The hardware probe selects:
| Hardware | Quant | Approx. VRAM use |
| --- | --- | --- |
| 12 GB+ VRAM | BF16 | ~1.6 GB (quality-first) |
| 412 GB VRAM | Q8_0 | ~945 MB (recommended balance) |
| 14 GB VRAM | Q4_K_M | ~659 MB (minimal footprint) |
| CPU-only | Q4_K_M | RAM-bound, latency-tolerable |
You can override the selection from Settings; overrides are allow-listed
against the same table (an F32 reference quant, ~3.2 GB, is override-only).
## Setup
Nothing to install: installer and CI builds bundle the binary for your
platform. Select the engine via **Model Catalogue → Engines** or
`OMNIVOICE_TTS_BACKEND=omnivoice-gguf`. The quant weights download on first
use (see [downloading-models.md](../downloading-models.md)) — install them
ahead of time from **Model Catalogue → Models** if you want the first
generation to be quick; a long first render is the download, not a hang.
**Source checkouts:** the repo ships zero-byte placeholders in `bin/` — real
binaries come from CI or the installer. The engine detects a placeholder and
reports unavailable with instructions
([#1172](https://github.com/debpalash/VoiceStudio/issues/1172)) instead of
failing at spawn time; build one with
`scripts/build-omnivoice-tts.sh --platform <slug>` or use the default
in-process engine.
## Integrity and self-healing
Before reporting ready, the engine:
- verifies the binary against the SHA-256 manifest (`bin/checksums.sha256`);
- detects macOS Gatekeeper quarantine and prints the exact
`xattr -cr '/Applications/VoiceStudio.app'` fix;
- restores a missing execute bit (a git clone or zip extract on POSIX can
drop `+x`, which used to surface as a permission error mislabeled as
out-of-memory — [#437](https://github.com/debpalash/VoiceStudio/issues/437)).
The chmod runs only after the SHA check confirms it's the right file.
## Behaviour notes
- Output is 24 kHz mono — same model, same rate as in-process OmniVoice.
- Cloning from a reference clip (with optional transcript) and style
instructions are supported; no voice design.
- Same multilingual surface as OmniVoice ([languages.md](../languages.md)).
- Because generation runs in another process, the app's own GPU counters
don't see its allocations — diagnostics label it accordingly.
| Variable | Default | Meaning |
| --- | --- | --- |
| `OMNIVOICE_GGUF_GENERATE_TIMEOUT_S` | (generous built-in) | Per-generation timeout for the spawned binary |
## Troubleshooting
- "GGUF binary missing": this build doesn't bundle the runtime for your
platform — use the default engine.
- Checksum mismatch or quarantine messages: follow the printed fix, or
reinstall.
- Other issues: [install/troubleshooting.md](../install/troubleshooting.md).
See also: [benchmarks.md](../benchmarks.md),
[performance.md](../performance.md), [disk usage](disk-usage.md).
+74
View File
@@ -0,0 +1,74 @@
# VoiceStudio — OmniVoice Engine (default)
OmniVoice (k2-fsa/OmniVoice) is VoiceStudio's default TTS engine — the one a
fresh install uses without any configuration. It does zero-shot voice cloning
across 600+ languages and outputs 24 kHz mono audio. Voice cloning, dubbing,
and dictation all run on it out of the box.
## When to pick it
- You want cloning plus the broadest language coverage (see
[languages.md](../languages.md)).
- You have a GPU (CUDA or Apple Silicon MPS) with ~6 GB VRAM or more.
- You just installed VoiceStudio — it's already selected.
For low-VRAM or CPU-only machines, the
[OmniVoice GGUF](omnivoice-gguf.md) variant runs the same model through a
quantized native binary with a much smaller memory footprint.
## Requirements
- Runs on CUDA, MPS (Apple Silicon), or CPU — auto-detected.
- Recommended VRAM floor: **6 GB** on a dedicated GPU. This is the only
engine with a measured floor: on 4 GB cards (GTX 1650 Ti, Quadro P2000 —
issues [#1226](https://github.com/debpalash/VoiceStudio/issues/1226) /
[#1222](https://github.com/debpalash/VoiceStudio/issues/1222)) the driver
pages to system RAM and a render that should take seconds runs for minutes
until the compute budget kills it. The UI warns before you wait; nothing
hard-blocks, since short inputs can still fit.
- No extra install — the model ships with the app and downloads its weights
on first use (see [downloading-models.md](../downloading-models.md)).
## Selecting the engine
OmniVoice is the default, so normally there is nothing to do. If you switched
away and want it back:
- **Model Catalogue → Engines**, or
- set `OMNIVOICE_TTS_BACKEND=omnivoice`.
The env var overrides the persisted UI choice.
## Behaviour notes
- Weights load lazily on first use and are shared with the rest of the app
(dubbing, dictation) — the model is never double-loaded.
- On CUDA the model runs fp16 with `torch.compile`; a speech recognizer is
co-loaded for the cloning path.
- Output is 24 kHz mono; the shared mastering chain (highpass + compressor)
is tuned for this rate and applied automatically.
- Cloning takes a short reference clip (`ref_audio`); an optional transcript
of the clip improves conditioning.
## Known limits
- No voice design from a text description — use [VoxCPM2](voxcpm2.md) for
that.
- Below the 6 GB VRAM floor, expect very slow renders or budget timeouts;
prefer [OmniVoice GGUF](omnivoice-gguf.md) or a CPU engine such as
[PocketTTS](pockettts.md).
## Troubleshooting
- "Too heavy for the available compute" on a small GPU: see the VRAM floor
above — switch to OmniVoice GGUF or close other GPU apps.
- First generation is slow: the first call downloads multi-GB weights. To
keep the first render quick, install the model ahead of time from
**Model Catalogue → Models** — a long first generate is almost always the
download, not a hang.
- General install issues: [install/troubleshooting.md](../install/troubleshooting.md).
See also: [benchmarks.md](../benchmarks.md),
[performance.md](../performance.md),
[expressive-speech.md](../expressive-speech.md),
[disk usage](disk-usage.md).
+58
View File
@@ -0,0 +1,58 @@
# VoiceStudio — Parakeet TDT v3 (MLX) Engine
NVIDIA's Parakeet TDT v3 on the Apple Silicon GPU, via the small pure-Python
`parakeet-mlx` package. It gives Macs the Parakeet tier CUDA/CPU users get
through NeMo or sherpa-onnx: **25 European languages**, word timestamps from
the TDT decoder itself (no wav2vec2 alignment pass needed), ~1.2 GB download,
~2 GB unified memory, dictation-grade speed on the GPU.
Unlike [nemo-parakeet](nemo-parakeet.md) it needs no `nemo_toolkit` (whose
transformers pin conflicts with the app's) — it is **installed by default on
Apple Silicon source installs since 0.3.22**.
## Selecting it
- **Model Catalogue → Engines**, ASR tab → **Use** on the Parakeet TDT v3
(MLX) row, or `OMNIVOICE_ASR_BACKEND=parakeet-mlx`.
- **Dictation prefers it automatically**: once the model weights are
installed (Model Catalogue → Models — the auto-pick never triggers a
download), live dictation/capture uses it whenever your system language is
one of the 25 covered European languages. Other languages keep the
multilingual Whisper engine, so dictation coverage never regresses.
## Best at
- **Live dictation on a Mac** — TDT decoding is fast enough for the capture
path, at Parakeet's better-than-Whisper English WER.
- **European-language transcription** with word timestamps at a fraction of
whisper-large-v3's memory and compute.
For languages outside the 25 (CJK, Arabic, ...), use
[mlx-whisper](mlx-whisper.md) instead.
## Platform support
**Apple Silicon only** — the same shared MLX platform gate as mlx-whisper
refuses Linux, Windows, and Intel Macs before any import
([#390](https://github.com/debpalash/VoiceStudio/issues/390)). It runs on the
unified-memory GPU; there is no CPU tier.
## Model selection
`ASR_MODEL_PARAKEET_MLX` — default `mlx-community/parakeet-tdt-0.6b-v3`.
Weights download on first load — see
[downloading-models](../downloading-models.md).
## Quirks
- Long files are processed in 120 s chunks internally to bound unified-memory
use; short dictation buffers and dub chunks are unaffected.
- Parakeet v3 auto-detects among its 25 languages but doesn't expose the
pick, so the reported language is the one you requested (or none) — it is
never hardcoded to English.
- Word timestamps are merged from the decoder's subword tokens — good for
subtitles and dictation; for lip-sync-critical dubbing the wav2vec2-aligned
engines ([mlx-whisper](mlx-whisper.md), [whisperx](whisperx.md)) remain the
accuracy tier.
Speed comparisons across engines live in [performance](../performance.md).
+82
View File
@@ -0,0 +1,82 @@
# VoiceStudio — PocketTTS Engine
PocketTTS (kyutai-labs/pocket-tts, 100M parameters) is the fastest-CPU-render
pick: small, low-latency, CPU-only, with zero-shot voice cloning from a
reference clip. It covers six languages — English, French, German,
Portuguese, Italian, Spanish — with one model per language, and measures
roughly 89x real-time on an Apple M3 Pro.
It complements the quality engines: where they fall back to CPU, PocketTTS
is built for it. CPU-only is deliberate — upstream observes no GPU speedup
for this model.
## When to pick it
- CPU-only machines that need fast rendering *and* voice cloning.
- Latency-sensitive use (dictation-style, short utterances) in one of the
six languages.
## Setup
1. Install the optional dependency:
```bash
uv sync --extra pockettts
```
(Or enable it from **Model Catalogue → Engines**.)
2. **Accept the license in-app**
([#1306](https://github.com/debpalash/VoiceStudio/issues/1306)). The code
is MIT and the weights are CC-BY-4.0, but the weights are **gated on
HuggingFace** behind an access agreement with an acceptable-use clause.
VoiceStudio surfaces this before first use: the engine stays unavailable
until you review and accept in **Model Catalogue → Engines → PocketTTS**.
You also need HuggingFace access to the gated repo (see
[downloading-models.md](../downloading-models.md) for token setup).
3. Select the engine via **Model Catalogue → Engines** or
`OMNIVOICE_TTS_BACKEND=pockettts`.
## Platform notes
- Works on Linux, Windows, macOS Apple Silicon — CPU only everywhere.
- **Not available on Intel Macs**: the required PyTorch version has no
macOS x86_64 wheel. The engine reports this plainly instead of failing
mid-install.
## Behaviour notes
- Output is 24 kHz mono.
- Six languages, one model per language, chosen by the `language` you
request; cloning takes a short reference clip.
- Runs in a crash-isolated sidecar process (parent Python environment): a
wedged generation is hard-killed by a watchdog and its memory reclaimed —
something an in-process engine cannot do.
- The first use downloads the gated weights; the sidecar heartbeats
progress during the download so the watchdog doesn't fire.
| Variable | Default | Meaning |
| --- | --- | --- |
| `OMNIVOICE_POCKETTTS_RECV_TIMEOUT_S` | `600` | Sidecar response deadline in seconds (min 30; cold loads download weights) |
## Known limits
- No voice design, no emotion controls
(see [expressive-speech.md](../expressive-speech.md)).
- Six languages only — for broader coverage use
[OmniVoice](omnivoice.md) ([languages.md](../languages.md)).
- Revoking the license acceptance takes effect immediately, without a
restart — subsequent generations refuse.
## Troubleshooting
- "pocket_tts package not installed": run the `uv sync` above.
- "license not accepted": open **Model Catalogue → Engines → PocketTTS**
and review/accept.
- Timeouts on a slow connection: raise
`OMNIVOICE_POCKETTTS_RECV_TIMEOUT_S` for the first (download-heavy) run.
- Other issues: [install/troubleshooting.md](../install/troubleshooting.md).
See also: [benchmarks.md](../benchmarks.md),
[performance.md](../performance.md), [disk usage](disk-usage.md).
+61
View File
@@ -0,0 +1,61 @@
# VoiceStudio — PyTorch Whisper Engine
Whisper through the plain `transformers` pipeline, riding torch itself. No
extra install — transformers ships with the app — and because it runs on
torch's own stack (including torch's bundled cuDNN 9), it works on machines
where the CTranslate2 engines can't load. It is also the engine that
genuinely uses **AMD ROCm** GPUs, so auto-detect picks it on ROCm hosts
([#1529](https://github.com/debpalash/VoiceStudio/issues/1529)).
## Selecting it
- **Model Catalogue → Engines**, ASR tab → **Use** on the PyTorch Whisper
row, or `OMNIVOICE_ASR_BACKEND=pytorch-whisper`.
- Auto-detect picks it on ROCm, and as the last resort everywhere else.
## Best at
- **ROCm dubbing/transcription** — the only Whisper engine that uses the HIP
GPU (CTranslate2 has no HIP build, MLX is Apple-only).
- **Rescue engine** when whisperx/faster-whisper can't load — e.g. the
missing-cuDNN-8 case
([#255](https://github.com/debpalash/VoiceStudio/issues/255)) — since it
needs neither CTranslate2 nor cuDNN 8.
For lip-sync-grade word timing prefer [whisperx](whisperx.md) or
[mlx-whisper](mlx-whisper.md); this engine returns the pipeline's own word
timestamps.
## Platform support
CUDA, Apple Silicon (MPS), ROCm (HIP), and CPU — wherever torch runs, on
macOS, Windows, and Linux.
## Model selection
`OMNIVOICE_PYTORCH_ASR_MODEL` — default `openai/whisper-large-v3-turbo`. Any
transformers-format Whisper repo works. Weights download on first load — see
[downloading-models](../downloading-models.md).
## VRAM preflight
whisper-large-v3-turbo needs roughly 3.2 GiB before generation adds its
workspace; loading it onto a nearly-full card "succeeds" and then the first
transcribe OOMs with zero segments. So on CUDA the engine checks free VRAM
against a 5 GB budget before loading and uses the CPU instead when the card
is too full (flush the TTS model to restore GPU-speed ASR). Disable with
`OMNIVOICE_ASR_VRAM_PREFLIGHT=0`.
## Quirks
- If the pipeline fails to import (`AutoFeatureExtractor` errors), the cause
is either an incomplete transformers install or a torch/torchvision
version mismatch — the error message names the exact reinstall command;
the trio has to move together at the pinned versions
([#549](https://github.com/debpalash/VoiceStudio/issues/549),
[#1376](https://github.com/debpalash/VoiceStudio/issues/1376)).
- Transcribes are time-bounded like every local engine:
`OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S` (default 120 s per dub chunk),
`OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S` (default 300 s whole-file).
Speed comparisons across engines live in [performance](../performance.md).
+71
View File
@@ -0,0 +1,71 @@
# VoiceStudio — Sherpa-ONNX Dictation Engine
The k2-fsa/sherpa-onnx ONNX runtime as a **live dictation** engine: small
int8 models that transcribe faster than realtime on CPU, with identical
behavior on macOS (arm64 + x86_64), Windows, and Linux — no CUDA dependency.
Streaming models emit partial text frame-by-frame as you speak; offline
models re-transcribe a growing buffer on a short cadence, so you see live
partials either way.
## Selecting it
- Ensure `sherpa-onnx` is installed (`uv add sherpa-onnx` on source installs).
- Pick a dictation model in the app (Model Catalogue → Models lists the
curated set below), or **Model Catalogue → Engines**, ASR tab → **Use**, or
pin `OMNIVOICE_ASR_BACKEND=sherpa-onnx-asr`.
- `OMNIVOICE_SHERPA_ASR_MODEL` selects the model — default
`sherpa-parakeet-tdt-v3`.
## Best at
- **Live dictation on CPU** — the whole point of this engine. Fast partials,
automatic endpointing on silence, no GPU required.
- It also honors the regular offline `transcribe` contract, so any of its
models can transcribe a file — plain text, single segment, no word
timestamps, which makes it a dictation/notes tool rather than a dubbing
engine.
## The 7 curated models
| Id | Type | Languages | Download |
| --- | --- | --- | --- |
| `sherpa-parakeet-tdt-v3` (default) | offline | 25 European languages | 0.67 GB |
| `sherpa-parakeet-tdt-v2` | offline | English | 0.66 GB |
| `sherpa-zipformer-bilingual-zh-en` | streaming | Chinese + English | 0.20 GB |
| `sherpa-paraformer-bilingual-zh-en` | streaming | Chinese + English | 0.24 GB |
| `sherpa-zipformer-en-20m` | streaming | English | 0.044 GB |
| `sherpa-zipformer-zh-14m` | streaming | Chinese | 0.025 GB |
| `sherpa-whisper-tiny` | offline | 90+ languages (auto-detect) | 0.104 GB |
Sizes are measured on-disk download sizes. Weights are int8 ONNX checkpoints
that download on first use through the same HF cache as everything else —
see [downloading-models](../downloading-models.md). Peak RAM for the 0.6B
Parakeets is noticeably higher than their download size (onnxruntime's arena
allocator holds onto freed blocks).
## Platform support
CPU on every platform, by the strict cross-platform default-parity rule.
`OMNIVOICE_SHERPA_ASR_PROVIDER` can override the ONNX provider on a verified
GPU build, but the default never diverges.
## Tuning
- `OMNIVOICE_SHERPA_ASR_THREADS` — decode threads (default 2; the 0.6B
Parakeets automatically use up to 4 when the host has the cores, so decode
keeps ahead of the speaker).
- `OMNIVOICE_DICTATION_ENDPOINT_R1` / `OMNIVOICE_DICTATION_ENDPOINT_R2`
streaming endpoint rules in seconds (defaults 1.0 / 0.6: text commits
~0.6 s after you stop speaking). Applied without a restart.
## Quirks
- The recognizer is **pre-warmed in the background** so the first dictation
session doesn't pay the 1.32.5 s ONNX session load
([#888](https://github.com/debpalash/VoiceStudio/issues/888)); it's then
shared warm across sessions.
- On Apple Silicon, installing the [parakeet-mlx](parakeet-mlx.md) model
makes dictation prefer the GPU Parakeet automatically for the 25 covered
languages; an explicitly selected sherpa model still wins.
- The offline `transcribe` path reports `language: auto` — per-file language
detection is only meaningful for the Whisper Tiny model.
+75
View File
@@ -0,0 +1,75 @@
# VoiceStudio — Sherpa-ONNX Engine
Sherpa-ONNX (k2-fsa/sherpa-onnx) is a unified C++ ONNX runtime that wraps
20+ TTS model families (VITS, MeloTTS, Piper, Kokoro, Matcha, and more)
behind one API, with pre-built wheels for Linux, Windows, and macOS (x86 and
ARM). You bring the model: point VoiceStudio at any downloaded sherpa-onnx
TTS model directory.
## When to pick it
- You want a specific community model (e.g. a Piper or VITS voice for your
language) that no other engine hosts.
- You need a dependable CPU engine with optional CUDA acceleration.
## Setup
1. Install the runtime:
```bash
pip install sherpa-onnx
```
2. Download a TTS model from the
[sherpa-onnx releases](https://github.com/k2-fsa/sherpa-onnx/releases)
and unpack it somewhere permanent.
3. Point VoiceStudio at the model directory and restart:
```bash
export OMNIVOICE_SHERPA_MODEL=/path/to/model-dir
```
4. Select the engine via **Model Catalogue → Engines** or
`OMNIVOICE_TTS_BACKEND=sherpa-onnx`.
The directory must contain `model.onnx` and `tokens.txt`. Sherpa-ONNX ships
no bundled default model, so the engine reports unavailable — with the
reason — until `OMNIVOICE_SHERPA_MODEL` points at a valid directory. (Before
this gate, selecting the engine unconfigured produced a failure mislabeled
as out-of-memory —
[#919](https://github.com/debpalash/VoiceStudio/issues/919).)
## Configuration
| Variable | Default | Meaning |
| --- | --- | --- |
| `OMNIVOICE_SHERPA_MODEL` | (unset) | Directory containing `model.onnx` + `tokens.txt` |
## Behaviour notes
- Output defaults to 22.05 kHz (the VITS default); once a model is loaded,
its own sample rate is used.
- CPU is the universal baseline; the CUDA onnxruntime provider is available
on Linux/Windows installs.
- **No cloning**: voices come from the model itself. Multi-speaker VITS
models select a voice by numeric speaker id; speed is supported.
- Languages depend entirely on the model you download.
## Known limits
- One model at a time — switching models means changing
`OMNIVOICE_SHERPA_MODEL` and restarting.
- No voice design, no reference-audio cloning, no emotion controls
(see [expressive-speech.md](../expressive-speech.md)).
## Troubleshooting
- "OMNIVOICE_SHERPA_MODEL not set" / "No model.onnx in …": follow Setup
above — the variable must point at the *unpacked* model directory, not
the archive.
- Other issues: [install/troubleshooting.md](../install/troubleshooting.md).
See also: [benchmarks.md](../benchmarks.md),
[languages.md](../languages.md),
[disk usage](disk-usage.md).
+76
View File
@@ -0,0 +1,76 @@
# VoiceStudio — Supertonic-3 Engine
Supertonic-3 (Supertone Inc.) is a ~99M-parameter ONNX TTS engine covering
31 languages with 7 preset voices at native 44.1 kHz. It is CPU-only by
design — pure ONNX Runtime on the CPU execution provider, with no CUDA or
MPS path in the upstream SDK — and runs in its own sidecar process so
crashes and cold init never block the rest of VoiceStudio.
## When to pick it
- Broad language coverage on machines with no usable GPU.
- Preset-voice narration at a higher sample rate than the default engine.
## Setup
1. Install the optional dependency into VoiceStudio's environment:
```bash
uv sync --extra supertonic
```
(Or enable it from **Model Catalogue → Engines**, which installs the
pinned `supertonic` wheel for you.)
2. **Accept the license in-app.** First use is gated behind an explicit
acceptance dialog: the inference SDK is MIT, but the model weights are
**OpenRAIL-M**, which carries use restrictions. The engine stays
unavailable until you review and accept in **Model Catalogue → Engines →
Supertonic-3**.
3. Select the engine via **Model Catalogue → Engines** or
`OMNIVOICE_TTS_BACKEND=supertonic3`.
The first synthesis cold-downloads ~400 MB of model weights, pinned to an
exact HuggingFace revision SHA so the bytes match what the SDK was validated
against. See [downloading-models.md](../downloading-models.md).
## Voices
Seven preset voices are surfaced: `M1` (default), `M3`, `M4`, `M5`, `F3`,
`F4`, `F5`. The SDK itself accepts the full `M1``M5` / `F1``F5` set if a
caller passes one explicitly; unknown ids fall back to the default with a
log line.
## Behaviour notes
- Output is 44.1 kHz mono.
- Runs as a long-lived sidecar in the parent Python environment (its
dependencies — onnxruntime, numpy, soundfile — already match
VoiceStudio's pins); subsequent calls reuse the warm ONNX session.
- `speed` is clamped to 0.72.0; quality steps clamp to 512.
- Language is an ISO 639-1 code; Auto engages the SDK's multilingual
fallback.
## Known limits
- **No cloning and no voice design** — preset voices only. Dub/batch jobs
that need cloning won't select it.
- CPU-only: hardware acceleration is a property of the upstream SDK, not a
VoiceStudio limitation.
- OpenRAIL-M weights are not covered by VoiceStudio's blanket
commercial-use statement — review the model license terms in the
acceptance dialog.
## Troubleshooting
- "supertonic package not installed": run the `uv sync` above or enable
from the Model Catalogue.
- "license not accepted": open **Model Catalogue → Engines → Supertonic-3**
and accept.
- Other issues: [install/troubleshooting.md](../install/troubleshooting.md).
See also: [benchmarks.md](../benchmarks.md),
[languages.md](../languages.md),
[expressive-speech.md](../expressive-speech.md),
[disk usage](disk-usage.md).
+76
View File
@@ -0,0 +1,76 @@
# VoiceStudio — VoxCPM2 Engine
VoxCPM2 (OpenBMB) is the studio-quality option: native 48 kHz output,
zero-shot voice cloning, and — uniquely among VoiceStudio's engines —
**voice design**: creating a synthetic voice from a text description
("young female, warm tone, British accent") with no reference audio at all.
## When to pick it
- You want voice design without a reference clip.
- You want the highest output sample rate (48 kHz vs OmniVoice's 24 kHz).
- Your language is among its 30 supported languages: Arabic, Burmese,
Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew,
Hindi, Indonesian, Italian, Japanese, Khmer, Korean, Lao, Malay,
Norwegian, Polish, Portuguese, Russian, Spanish, Swahili, Swedish,
Tagalog, Thai, Turkish, Vietnamese.
## Requirements
- Python ≥ 3.10, PyTorch ≥ 2.5.
- CUDA ≥ 12 recommended for full speed; MPS (Apple Silicon) and CPU also
work.
## Setup
Install the package into VoiceStudio's Python environment:
```bash
pip install "voxcpm>=2.0.3"
```
That is a version **floor**, not a pin — an older install still works, but
the engine logs an upgrade hint at load time. Then select the engine via
**Model Catalogue → Engines** or `OMNIVOICE_TTS_BACKEND=voxcpm2`.
## Model selection
| Variable | Default | Meaning |
| --- | --- | --- |
| `OMNIVOICE_VOXCPM_MODEL` | `openbmb/VoxCPM2` | HuggingFace checkpoint to load |
The first use downloads a multi-GB checkpoint from HuggingFace. A download
interrupted near the end used to abort the load outright
([#1224](https://github.com/debpalash/VoiceStudio/issues/1224)); the load is
now retried once with a fresh client. See
[downloading-models.md](../downloading-models.md).
## Behaviour notes
- **Voice design:** provide a description and no reference audio.
- **Cloning:** the reference clip is prepared before use (edge-silence trim
and length cap) so dead air in a raw clip doesn't condition the output; on
any prep problem the raw clip is used as-is.
- **Style instructions** are passed as an inline prefix to the text.
- VoxCPM2 emits mastered, studio-grade audio, so VoiceStudio **skips its
shared mastering chain** (which is tuned for 24 kHz engines) — only benign
loudness normalization applies.
- A trailing-silence guard trims long near-silent tails from generations,
keeping a short natural tail.
## Known limits
- Slower than the lightweight CPU engines — see
[benchmarks.md](../benchmarks.md) and [performance.md](../performance.md).
- Language coverage is 30 languages; for anything else use the default
[OmniVoice](omnivoice.md) engine ([languages.md](../languages.md)).
## Troubleshooting
- Engine shows unavailable: the `voxcpm` package isn't installed — run the
`pip install` above and restart VoiceStudio.
- Repeated first-download failures: check connectivity/HF access, then see
[install/troubleshooting.md](../install/troubleshooting.md).
See also: [expressive-speech.md](../expressive-speech.md),
[disk usage](disk-usage.md).
+80
View File
@@ -0,0 +1,80 @@
# VoiceStudio — WhisperX Engine
WhisperX is the default ASR engine on CUDA and plain-CPU hosts: faster-whisper
(CTranslate2) transcription plus a **wav2vec2 forced-alignment** pass that
snaps word boundaries to ±1030 ms (Whisper's own timestamps are ±100300 ms).
That word timing is what dubbing lip-sync depends on, which is why auto-detect
prefers it wherever CTranslate2 can use the GPU.
## Selecting it
- **Model Catalogue → Engines**, ASR tab → **Use** on the WhisperX row, or
- pin it with `OMNIVOICE_ASR_BACKEND=whisperx` (the env var always wins over
the Settings pick; with neither set, auto-detect chooses per-hardware).
## Best at
- **Dubbing** — the forced alignment is the accuracy tier lip-sync needs.
- **Batch transcription** with word-level subtitles.
- Multi-speaker work: it pairs with pyannote speaker diarization — see
[diarization](../features/diarization.md).
## Platform support
| Host | What happens |
| --- | --- |
| NVIDIA CUDA | GPU, float16 (degrades automatically, see below) |
| CPU (any OS) | int8 — works, but slow for large-v3 |
| Apple Silicon | CPU only — CTranslate2 has no Metal build, so auto-detect prefers [mlx-whisper](mlx-whisper.md) there ([#1127](https://github.com/debpalash/VoiceStudio/issues/1127)) |
| AMD ROCm | CPU only — CTranslate2 has no HIP build, so auto-detect prefers [pytorch-whisper](pytorch-whisper.md) there ([#1529](https://github.com/debpalash/VoiceStudio/issues/1529)) |
## Model selection
- `ASR_MODEL_WHISPERX` — default `large-v3`. Accepts the usual size aliases
(`tiny``large-v3`, `distil-large-v3`) or a full HF repo id. Weights
download on first load — see [downloading-models](../downloading-models.md).
- `OMNIVOICE_ALIGN_DEVICE` — force the wav2vec2 aligner's device. Aligners
exist for ~20 major languages; other languages keep Whisper's native word
timestamps instead of failing.
## VRAM preflight and degradation
Loading fp16 large-v3 onto a nearly-full 8 GB card dies as a *native* CUDA
abort — no Python exception, the whole backend goes down
([#723](https://github.com/debpalash/VoiceStudio/issues/723)). So before every
load the engine checks free VRAM against per-compute-type budgets
(float16 5.0 GB, int8_float16 3.5 GB, int8 3.0 GB, scaled down for smaller
models) and degrades the compute type — or falls to CPU int8 — instead of
starting a load that would kill the process. Disable with
`OMNIVOICE_ASR_VRAM_PREFLIGHT=0`.
Two more fallback chains run at load time:
- GPUs without efficient fp16 (older Maxwell/Pascal, GTX 16xx) raise a
compute-type error — the engine retries int8_float16, then int8
([#551](https://github.com/debpalash/VoiceStudio/issues/551)).
- A genuine CUDA OOM retries on CPU int8, so dubbing still completes
(slower, same model and accuracy).
## Quirks
- **cuDNN 8 required on CUDA.** CTranslate2 links cuDNN 8; if it's missing the
process fast-fails with no traceback, so the engine is reported unavailable
up front and selection falls through to pytorch-whisper, which uses torch's
own cuDNN 9 ([#1371](https://github.com/debpalash/VoiceStudio/issues/1371)).
- On some hardened Linux kernels CTranslate2's native library is rejected with
"cannot enable executable stack" — reported as unavailable, not a crash
([#692](https://github.com/debpalash/VoiceStudio/issues/692)).
- A partially-installed environment (interrupted sync, antivirus quarantine)
can break WhisperX's deep import chain (whisperx → pyannote →
lightning_fabric). The engine is then reported unavailable with a repair
hint — reinstall, or `uv sync --reinstall` on a source checkout
([#1185](https://github.com/debpalash/VoiceStudio/issues/1185)).
- Audio is decoded through VoiceStudio's validated ffmpeg, not a bare `ffmpeg`
PATH lookup ([#479](https://github.com/debpalash/VoiceStudio/issues/479)).
- Transcribes are time-bounded: each dub chunk by
`OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S` (default 120 s), whole files by
`OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S` (default 300 s). Raise them for very
long files on slow hardware.
Speed comparisons across engines live in [performance](../performance.md).