docs(engines): a guide for every engine + index; fix two engine-metadata bugs (#1556)
* docs(engines): a guide for every engine + index; fix two engine-metadata bugs 21 new pages under docs/engines/ (10 TTS, 10 ASR, index README) — every registered engine now has one: what it's for, platform support, model env vars, quirks with issue refs. Linked from both READMEs' engine sections. Code fixes found while verifying facts against the registries: - KittenTTS docstring claimed default voice 'Jasper'; the code default is expr-voice-2-f - the isolated-ASR sidecar read only ASR_MODEL_FW while the download preflight read ASR_MODEL_FASTER — set one and the other quietly used a different model; both now resolve ASR_MODEL_FW-override → ASR_MODEL_FASTER - moonshine's install hint named 'useful-moonshine', a package the backend never imports; now moonshine-onnx / moonshine-voice Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(changelog): entries for the engine guides + sidecar model fix (#1556) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(engines): second-harvest fixes — TLS guidance, matrix/code alignment, CN counts - README matrix aligned to gpu_compat (the code is the source of truth): CosyVoice macOS is CPU not MPS, IndexTTS and GGUF gain their real CUDA/CPU/MPS cells - gpt-sovits guide: prefer https/tunnel for non-loopback servers, plaintext warning; first-use download guidance on both OmniVoice pages - preflight empty-env fallback matches the sidecar (ASR_MODEL_FASTER='' no longer resolves a different repo) - nano installs via uv pip; kitten log level wording; index links install guides incl. the Gatekeeper step; README_CN engine counts 16/11 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(readme-cn): the all-engines-local claim now excludes the remote client Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
b79ba9bd3b
commit
030d5ea01f
@@ -22,8 +22,10 @@ the frozen-backend fallback mirror it for their toolchains.
|
||||
|
||||
### Docs
|
||||
- The READMEs now lead with download buttons and a three-step first-clone walkthrough, and a new benchmarks page anchors measured per-engine/per-device numbers on the in-repo harness (#1555)
|
||||
- Every engine now has its own guide — 21 new pages under docs/engines plus an index covering all 16 TTS and 11 ASR engines, linked from both READMEs (#1556)
|
||||
|
||||
### Fixed
|
||||
- The crash-isolated ASR sidecar and its download preflight now agree on which model to load — setting the shared faster-whisper model variable applies to both variants instead of the sidecar quietly using a different one (#1556)
|
||||
- "Ready" now requires the deep health probe (a working database-backed route), not just the identity probe — a backend whose install broke underneath can no longer be announced up while every real request fails (#1548)
|
||||
- Supervisor restarts after repeat crashes now back off (immediate, then 5s, then 15s) instead of respawning back-to-back, so a tight crash loop can't burn the whole restart budget in seconds (#1548)
|
||||
|
||||
|
||||
@@ -218,7 +218,7 @@ Professional-grade voice AI, minus the subscription and the cloud. Convinced? [C
|
||||
|
||||
### 🗣️ TTS Engines
|
||||
|
||||
**16 engines, one picker.** VoiceStudio (default, 600+ languages) is always available; seven more are opt-in and auto-detected (CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX), plus eight lazy-installed opt-ins (IndexTTS 2.5, OmniVoice GGUF, OmniVoice subprocess, PocketTTS, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS). Switch in **Model Catalogue → Engines** — or from anywhere with <kbd>Ctrl</kbd>/<kbd>Cmd</kbd>+<kbd>E</kbd>; the choice applies everywhere synthesis happens.
|
||||
**16 engines, one picker.** VoiceStudio (default, 600+ languages) is always available; seven more are opt-in and auto-detected (CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX), plus eight lazy-installed opt-ins (IndexTTS 2.5, OmniVoice GGUF, OmniVoice subprocess, PocketTTS, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS). Switch in **Model Catalogue → Engines** — or from anywhere with <kbd>Ctrl</kbd>/<kbd>Cmd</kbd>+<kbd>E</kbd>; the choice applies everywhere synthesis happens. **Every engine has its own guide: [docs/engines](docs/engines/README.md).**
|
||||
|
||||
<details>
|
||||
<summary><b>📊 The full matrix</b> — 16 engines × platform × clone/instruct × license</summary>
|
||||
@@ -228,15 +228,15 @@ Professional-grade voice AI, minus the subscription and the cloud. Convinced? [C
|
||||
| Engine | Languages | Clone | Instruct | Linux | macOS ARM | Windows | License |
|
||||
|--------|:---------:|:-----:|:--------:|:-----:|:---------:|:-------:|:-------:|
|
||||
| **VoiceStudio** (default, powered by k2-fsa/OmniVoice) | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Built-in |
|
||||
| **CosyVoice 3** | 9 + 18 dialects | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **CosyVoice 3** | 9 + 18 dialects | ✅ | ✅ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **GPT-SoVITS** | 5 | ✅ | — | ✅ CUDA/CPU | — | ✅ CUDA/CPU | MIT |
|
||||
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MOSS-TTS-Nano** | 20 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **KittenTTS** | English | — | — | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
|
||||
| **MLX-Audio** (Kokoro, Qwen3-TTS, CSM, Dia, …) | Multi | Varies | Varies | ❌ | ✅ Native | ❌ | Varies |
|
||||
| **Sherpa-ONNX** | 20+ | — | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **IndexTTS 2.5** ⚡ | ZH · EN · JA · ES · AR | ✅ | — | ✅ CUDA | — | ✅ CUDA | Bilibili model license¹ |
|
||||
| **OmniVoice GGUF** ⚡ | 600+ | ✅ | ✅ | ✅ CPU | ✅ CPU | ✅ CPU | Built-in |
|
||||
| **IndexTTS 2.5** ⚡ | ZH · EN · JA · ES · AR | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Bilibili model license¹ |
|
||||
| **OmniVoice GGUF** ⚡ | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS/CPU | ✅ CUDA/CPU | Built-in |
|
||||
| **OmniVoice (subprocess)** ⚡² | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Built-in |
|
||||
| **PocketTTS** ⚡ (Kyutai) | EN · FR · DE · PT · IT · ES | ✅ | — | ✅ CPU | ✅ CPU | ✅ CPU | CC-BY-4.0 (gated)³ |
|
||||
| **Supertonic 3** ⚡ | 31 | — | — | ✅ CPU | ✅ CPU | ✅ CPU | OpenRAIL-M |
|
||||
@@ -275,7 +275,7 @@ another machine, set `OMNIVOICE_GPTSOVITS_URL` to its credential-free
|
||||
|
||||
### 🎧 ASR Engines
|
||||
|
||||
**11 engines** — they power dictation, video dubbing, and subtitles. **WhisperX** is the cross-platform default (~100 languages, word-level timing); the rest are opt-in and auto-detected. Switch in **Model Catalogue → Engines**. Ten run fully on-device; the eleventh (OpenAI-compatible) is an optional remote client for Qwen3-ASR or any compatible server.
|
||||
**11 engines** — they power dictation, video dubbing, and subtitles. **WhisperX** is the cross-platform default (~100 languages, word-level timing); the rest are opt-in and auto-detected. Switch in **Model Catalogue → Engines**. Ten run fully on-device; the eleventh (OpenAI-compatible) is an optional remote client for Qwen3-ASR or any compatible server. **Per-engine guides: [docs/engines](docs/engines/README.md).**
|
||||
|
||||
<details>
|
||||
<summary><b>📊 The full lineup</b> — 11 engines, what each is best at, and compute-type notes</summary>
|
||||
|
||||
+7
-7
@@ -180,8 +180,8 @@ Hugging Face Token 的配置见
|
||||
| **API 密钥** | 需要账号 | 本地流程不需要 |
|
||||
| **GPU 支持** | 不适用(云端) | CUDA · Apple Silicon · ROCm(Linux)· CPU |
|
||||
| **桌面应用** | ❌ | ✅ macOS · Windows · Linux |
|
||||
| **TTS 引擎** | 1 | **14** — [完整矩阵](#tts-engines) |
|
||||
| **ASR 引擎** | 1 | **10** — [完整阵容](#asr-engines) |
|
||||
| **TTS 引擎** | 1 | **16** — [完整矩阵](#tts-engines) |
|
||||
| **ASR 引擎** | 1 | **11** — [完整阵容](#asr-engines) |
|
||||
| **MCP 服务器** | ❌ | ✅ 可从 Claude、Cursor 及任何 MCP 客户端使用 |
|
||||
| **自检** | ❌ | ✅ 诊断套件、错误日志、脱敏调试包 |
|
||||
| **可定制** | ❌ 闭源 | ✅ 随你 Fork、扩展、发布 |
|
||||
@@ -221,10 +221,10 @@ Hugging Face Token 的配置见
|
||||
|
||||
### 🗣️ TTS 引擎
|
||||
|
||||
**14 个引擎,一个选择器。** VoiceStudio(默认,支持 600+ 语言)始终可用;另有七个引擎可选装并自动检测(CosyVoice 3、GPT-SoVITS、VoxCPM2、MOSS-TTS-Nano、KittenTTS、MLX-Audio、Sherpa-ONNX),外加六个按需延迟安装的重量级引擎(IndexTTS 2.5、OmniVoice GGUF、Supertonic 3、MOSS-TTS-v1.5、dots.tts、Confucius4-TTS)。在 **设置 → TTS 引擎** 中切换;所选引擎将应用于所有语音合成场景。
|
||||
**16 个引擎,一个选择器。** VoiceStudio(默认,支持 600+ 语言)始终可用;另有七个引擎可选装并自动检测(CosyVoice 3、GPT-SoVITS、VoxCPM2、MOSS-TTS-Nano、KittenTTS、MLX-Audio、Sherpa-ONNX),外加八个按需延迟安装的引擎(IndexTTS 2.5、OmniVoice GGUF、OmniVoice 子进程版、PocketTTS、Supertonic 3、MOSS-TTS-v1.5、dots.tts、Confucius4-TTS)。在 **设置 → TTS 引擎** 中切换;所选引擎将应用于所有语音合成场景。**每个引擎都有独立指南:[docs/engines](docs/engines/README.md)(英文)。**
|
||||
|
||||
<details>
|
||||
<summary><b>📊 完整矩阵</b>——14 个引擎 × 平台 × 克隆/指令 × 许可证</summary>
|
||||
<summary><b>📊 完整矩阵</b>——16 个引擎 × 平台 × 克隆/指令 × 许可证</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
@@ -261,10 +261,10 @@ Hugging Face Token 的配置见
|
||||
|
||||
### 🎧 ASR 引擎
|
||||
|
||||
**10 个引擎**——它们驱动听写、视频配音和字幕。**WhisperX** 是跨平台的默认引擎(约 100 种语言,词级时间对齐);其余引擎均为可选装并自动检测。在 **设置 → 引擎** 中切换。九个完全在本地设备上运行;第十个(OpenAI 兼容)是可选的远程客户端,可用于 Qwen3-ASR 或任何兼容的服务器。
|
||||
**11 个引擎**——它们驱动听写、视频配音和字幕。**WhisperX** 是跨平台的默认引擎(约 100 种语言,词级时间对齐);其余引擎均为可选装并自动检测。在 **设置 → 引擎** 中切换。十个完全在本地设备上运行;第十一个(OpenAI 兼容)是可选的远程客户端,可用于 Qwen3-ASR 或任何兼容的服务器。
|
||||
|
||||
<details>
|
||||
<summary><b>📊 完整阵容</b>——10 个引擎、各自的强项与计算类型说明</summary>
|
||||
<summary><b>📊 完整阵容</b>——11 个引擎、各自的强项与计算类型说明</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
@@ -281,7 +281,7 @@ Hugging Face Token 的配置见
|
||||
| **sherpa-onnx**(实时听写) | `sherpa-onnx-asr` | 25 种欧洲语言 + 90+ | 实时、快于实时的听写——小体积流式/离线 ONNX 模型(Parakeet TDT v3/v2、流式 Zipformer 与 Paraformer、Whisper Tiny),CPU 运行,macOS / Windows / Linux 表现完全一致。在 **设置 → 语音** 中按模型选择。 |
|
||||
| **OpenAI 兼容** ⚠️ 远程 | `openai-compat-asr` | 取决于服务器 | 当下通往 **Qwen3-ASR** 的路径(自托管服务器,无需等 transformers 支持)、任何 OpenAI 兼容的转录端点,或 OpenAI 官方 API——无需安装,在 **设置 → 引擎**(ASR 标签页)中配置并测试连接。音频会离开你的设备,发送到你指定的任何服务器;参见 [docs/engines/openai-compatible-asr.md](docs/engines/openai-compatible-asr.md)。 |
|
||||
|
||||
> Whisper 系列引擎覆盖约 100 种语言;**FunASR / SenseVoice** 额外提供一条多语言一体化路径,内置语音活动检测与行内说话人分离。**sherpa-onnx** 驱动实时听写的模型选择器——你边说,文字边出现。每个引擎都在本地设备上运行——无需 API 密钥,无需云端。
|
||||
> Whisper 系列引擎覆盖约 100 种语言;**FunASR / SenseVoice** 额外提供一条多语言一体化路径,内置语音活动检测与行内说话人分离。**sherpa-onnx** 驱动实时听写的模型选择器——你边说,文字边出现。除可选的 OpenAI 兼容远程客户端外,所有引擎都在本地设备上运行——无需 API 密钥,无需云端。
|
||||
|
||||
> **GPU 不支持高效 float16?** 在较老的 NVIDIA GPU(Maxwell/Pascal、GTX 16xx)上,或在 CTranslate2/cuDNN 版本不匹配之后,CTranslate2 系 ASR 引擎(WhisperX、Faster-Whisper)无法运行 `float16`,VoiceStudio 会自动改用 `int8` 重试——无需配置。如果转录仍然失败,可用 `ASR_COMPUTE_TYPE` 环境变量固定计算类型(逃生舱口):`ASR_COMPUTE_TYPE=int8`(CPU 用 `float32`)。将其设为 `int8` 并重启后端。
|
||||
|
||||
|
||||
@@ -85,7 +85,16 @@ def _get_model():
|
||||
global _model
|
||||
if _model is None:
|
||||
from faster_whisper import WhisperModel
|
||||
name = os.environ.get("ASR_MODEL_FW", "large-v3")
|
||||
# Same weights as in-process faster-whisper: ASR_MODEL_FASTER selects
|
||||
# for BOTH variants, ASR_MODEL_FW stays as a sidecar-only override.
|
||||
# Before this, the sidecar read only ASR_MODEL_FW while the download
|
||||
# preflight read ASR_MODEL_FASTER — set one and the other variant (or
|
||||
# the preflight) quietly used a different model.
|
||||
name = (
|
||||
os.environ.get("ASR_MODEL_FW")
|
||||
or os.environ.get("ASR_MODEL_FASTER")
|
||||
or "large-v3"
|
||||
)
|
||||
try:
|
||||
import torch
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
|
||||
@@ -2325,7 +2325,7 @@ _INSTALL_HINTS: dict[str, str] = {
|
||||
"mac-ARM source installs since 0.3.22. Parakeet TDT v3 on the GPU via "
|
||||
"MLX: 25 European languages, word timestamps, ~2 GB unified memory.)"
|
||||
),
|
||||
"moonshine": "pip install useful-moonshine (edge/CPU-optimized ASR)",
|
||||
"moonshine": "uv pip install moonshine-onnx (or moonshine-voice; edge/CPU-optimized ASR)",
|
||||
"funasr": "pip install funasr (SenseVoiceSmall + FSMN-VAD; CUDA or CPU)",
|
||||
"sherpa-onnx-asr": "uv add sherpa-onnx (ONNX live dictation; CPU, cross-platform)",
|
||||
"openai-compat-asr": (
|
||||
@@ -3120,10 +3120,18 @@ def _offline_asr_repo(backend_id: str | None = None) -> str | None:
|
||||
bid = backend_id or active_backend_id()
|
||||
if bid == "whisperx":
|
||||
return _fw_repo(os.environ.get("ASR_MODEL_WHISPERX", "large-v3"))
|
||||
if bid in ("faster-whisper", "faster-whisper-isolated"):
|
||||
# The crash-isolated sidecar loads the SAME CT2 weights as in-process
|
||||
# faster-whisper (it reuses the ASR_MODEL_FASTER selection).
|
||||
if bid == "faster-whisper":
|
||||
return _fw_repo(os.environ.get("ASR_MODEL_FASTER", _FASTER_WHISPER_DEFAULT))
|
||||
if bid == "faster-whisper-isolated":
|
||||
# Mirror the sidecar's own resolution (_asr_sidecar/main.py):
|
||||
# ASR_MODEL_FW is a sidecar-only override, otherwise the shared
|
||||
# ASR_MODEL_FASTER selection applies — so the preflight can never
|
||||
# download a different repo than the sidecar will load.
|
||||
return _fw_repo(
|
||||
os.environ.get("ASR_MODEL_FW")
|
||||
or os.environ.get("ASR_MODEL_FASTER")
|
||||
or _FASTER_WHISPER_DEFAULT
|
||||
)
|
||||
if bid == "mlx-whisper":
|
||||
return os.environ.get("ASR_MODEL", _MLX_MODEL_DEFAULT)
|
||||
if bid == "parakeet-mlx":
|
||||
|
||||
@@ -1092,7 +1092,8 @@ class KittenTTSBackend(TTSBackend):
|
||||
- English only
|
||||
- Much faster + much smaller install
|
||||
|
||||
Preset voice is chosen via `extras["voice"]` (defaults to "Jasper"). Any
|
||||
Preset voice is chosen via `extras["voice"]` (defaults to DEFAULT_VOICE,
|
||||
"expr-voice-2-f"). Any
|
||||
`ref_audio` / `instruct` / `language` arg is ignored with a log line so
|
||||
the common call-site doesn't need to know which engine it's talking to.
|
||||
"""
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
# Engine guides
|
||||
|
||||
One page per engine: what it's for, what it needs, how to enable it, and its
|
||||
quirks. Select engines in **Model Catalogue → Engines** (or quick-switch with
|
||||
<kbd>Ctrl</kbd>/<kbd>Cmd</kbd>+<kbd>E</kbd>), or pin one with
|
||||
`OMNIVOICE_TTS_BACKEND` / `OMNIVOICE_ASR_BACKEND`.
|
||||
|
||||
Measured speed/VRAM numbers live in [benchmarks](../benchmarks.md); what each
|
||||
engine can do expressively in [expressive-speech](../expressive-speech.md);
|
||||
sidecar disk footprints in [disk-usage](disk-usage.md); the bar a new engine
|
||||
must clear in [engine-acceptance](../engine-acceptance.md).
|
||||
|
||||
New to VoiceStudio? Install the app first — [macOS](../install/macos.md)
|
||||
(first launch needs the one-time right-click → **Open** Gatekeeper
|
||||
approval), [Windows](../install/windows.md), [Linux](../install/linux.md),
|
||||
[Docker](../install/docker.md).
|
||||
|
||||
## Text-to-speech
|
||||
|
||||
| Engine | Guide | Runs on | Cloning | Enabled by |
|
||||
|---|---|---|---|---|
|
||||
| VoiceStudio (OmniVoice) — **default** | [omnivoice](omnivoice.md) | CUDA · MPS · CPU | ✅ | installed by default |
|
||||
| VoxCPM2 | [voxcpm2](voxcpm2.md) | CUDA · MPS · CPU | ✅ + voice design | `pip install "voxcpm>=2.0.3"` |
|
||||
| MOSS-TTS-Nano | [moss-tts-nano](moss-tts-nano.md) | CUDA · CPU | ✅ (ref only) | clone + `uv pip install -e .` |
|
||||
| KittenTTS | [kittentts](kittentts.md) | CPU | — (8 preset voices) | `pip install kittentts` |
|
||||
| MLX-Audio (Kokoro, CSM, Dia, …) | [mlx-audio](mlx-audio.md) | Apple Silicon | model-dependent | `pip install mlx-audio` |
|
||||
| CosyVoice 3 | [cosyvoice](cosyvoice.md) | CUDA · CPU | ✅ | clone + requirements |
|
||||
| GPT-SoVITS | [gpt-sovits](gpt-sovits.md) | external server | ✅ | its own API server |
|
||||
| Sherpa-ONNX | [sherpa-onnx](sherpa-onnx.md) | CUDA · CPU | — | `pip install sherpa-onnx` + model dir |
|
||||
| IndexTTS 2.5 | [indextts](indextts.md) | CUDA · CPU | ✅ + emotion | one-click sidecar install |
|
||||
| OmniVoice GGUF | [omnivoice-gguf](omnivoice-gguf.md) | CUDA · MPS · CPU | ✅ | bundled binary |
|
||||
| Supertonic-3 | [supertonic3](supertonic3.md) | CPU | — (7 preset voices) | `uv sync --extra supertonic` + license |
|
||||
| MOSS-TTS-v1.5 (8B) | [moss-tts-v15](moss-tts-v15.md) | CUDA · CPU | ✅ | clone + env var |
|
||||
| dots.tts (2B) | [dots-tts](dots-tts.md) | CUDA · CPU (not Windows) | ✅ | clone + env var |
|
||||
| OmniVoice (subprocess) | [omnivoice-subprocess](omnivoice-subprocess.md) | CUDA · MPS · CPU | ✅ | opt-in pick, no install |
|
||||
| PocketTTS (Kyutai) | [pockettts](pockettts.md) | CPU (not Intel Mac) | ✅ | `uv sync --extra pockettts` + license |
|
||||
| Confucius4-TTS | [confucius4-tts](confucius4-tts.md) | CUDA · CPU | ✅ | clone + env var |
|
||||
|
||||
## Speech-to-text
|
||||
|
||||
| Engine | Guide | Runs on | Best at | Enabled by |
|
||||
|---|---|---|---|---|
|
||||
| WhisperX | [whisperx](whisperx.md) | CUDA · CPU | dubbing (word timestamps + diarization) | installed by default |
|
||||
| Faster-Whisper | [faster-whisper](faster-whisper.md) | CUDA · CPU | general transcription | installed by default |
|
||||
| Faster-Whisper (isolated) | [faster-whisper-isolated](faster-whisper-isolated.md) | CUDA · CPU | unattended batches | opt-in pick |
|
||||
| MLX Whisper | [mlx-whisper](mlx-whisper.md) | Apple Silicon | Mac default | `pip install mlx-whisper` |
|
||||
| PyTorch Whisper | [pytorch-whisper](pytorch-whisper.md) | CUDA · MPS · CPU | ROCm hosts | installed by default |
|
||||
| Parakeet TDT (NeMo) | [nemo-parakeet](nemo-parakeet.md) | CUDA · CPU | 25 languages, fast CPU | separate venv (never the app's) |
|
||||
| Parakeet TDT (MLX) | [parakeet-mlx](parakeet-mlx.md) | Apple Silicon | dictation, 25 EU languages | default on mac-ARM source installs |
|
||||
| Moonshine | [moonshine](moonshine.md) | CPU | edge/low-power, no timestamps | `pip install` (see guide) |
|
||||
| FunASR (SenseVoice) | [funasr](funasr.md) | CUDA · CPU | 50+ languages, inline diarization | `pip install funasr` |
|
||||
| Sherpa-ONNX dictation | [sherpa-onnx-asr](sherpa-onnx-asr.md) | CPU | live streaming dictation | curated model download |
|
||||
| OpenAI-compatible (remote) | [openai-compatible-asr](openai-compatible-asr.md) | network | offloading to a server (audio leaves the machine) | Model Catalogue |
|
||||
|
||||
Speaker diarization is not an engine registry of its own — the dub pipeline
|
||||
uses pyannote (HF-gated; see [diarization](../features/diarization.md)) and
|
||||
FunASR can diarize inline with its `cam++` speaker model.
|
||||
@@ -0,0 +1,61 @@
|
||||
# VoiceStudio — Faster-Whisper (Crash-Isolated) Engine
|
||||
|
||||
The same CTranslate2 Whisper engine as [faster-whisper](faster-whisper.md),
|
||||
run in a **separate child process** ("sidecar"). CTranslate2's GPU teardown
|
||||
can segfault — the endemic faster-whisper crash — and a hung or crashed
|
||||
transcribe in-process takes the whole backend down with it. Isolated, the
|
||||
child can crash or be force-killed to reclaim a hung transcribe and its VRAM
|
||||
while the backend stays up
|
||||
([#730](https://github.com/debpalash/VoiceStudio/issues/730)).
|
||||
|
||||
There is nothing extra to install: the sidecar reuses the app's own venv —
|
||||
only the process boundary is new.
|
||||
|
||||
## Selecting it
|
||||
|
||||
- **Model Catalogue → Engines**, ASR tab → **Use** on the crash-isolated row, or
|
||||
- pin it with `OMNIVOICE_ASR_BACKEND=faster-whisper-isolated`.
|
||||
|
||||
It is never picked by auto-detect — it's an explicit opt-in escape hatch.
|
||||
|
||||
## Best at
|
||||
|
||||
- **Long batch runs** where one bad file must not kill the backend.
|
||||
- Machines where in-process faster-whisper has crashed or hung before:
|
||||
a sidecar crash fails only that job, and the next transcribe respawns a
|
||||
fresh sidecar automatically.
|
||||
|
||||
## Platform support
|
||||
|
||||
Same as faster-whisper: CUDA float16 or CPU int8 on macOS, Windows, and
|
||||
Linux. The sidecar picks cuda/cpu itself and walks the same
|
||||
float16 → int8_float16 → int8 degrade chain on GPUs without efficient fp16
|
||||
([#551](https://github.com/debpalash/VoiceStudio/issues/551)).
|
||||
|
||||
## Model selection
|
||||
|
||||
- `ASR_MODEL_FASTER` — the shared model selection, same as the in-process
|
||||
engine: set it once and both variants load the same weights.
|
||||
- `ASR_MODEL_FW` — optional sidecar-only override; when set it wins over
|
||||
`ASR_MODEL_FASTER` for this engine. Default `large-v3`.
|
||||
- `ASR_COMPUTE_TYPE` — optional: pin the sidecar to one CTranslate2 compute
|
||||
type instead of the automatic degrade chain.
|
||||
|
||||
Weights download on first load — see
|
||||
[downloading-models](../downloading-models.md).
|
||||
|
||||
## Trade-offs and quirks
|
||||
|
||||
- **Slightly slower per call** than in-process faster-whisper (IPC overhead);
|
||||
the model stays warm inside the sidecar between calls, so the cost is per
|
||||
request, not per chunk of audio.
|
||||
- Word timestamps are Whisper-native (±100–300 ms) — no forced alignment.
|
||||
For dubbing lip-sync, use [whisperx](whisperx.md) or
|
||||
[mlx-whisper](mlx-whisper.md).
|
||||
- If the sidecar dies mid-transcription the job fails with a clear
|
||||
"sidecar crashed" error and the backend stays up — retry to respawn.
|
||||
- **cuDNN 8 is still required on CUDA** — same CTranslate2 requirement as the
|
||||
in-process engine. It's checked up front so a missing cuDNN 8 shows as
|
||||
"unavailable" in Model Catalogue → Engines instead of a sidecar that
|
||||
silently fails every transcribe
|
||||
([#1371](https://github.com/debpalash/VoiceStudio/issues/1371)).
|
||||
@@ -0,0 +1,70 @@
|
||||
# VoiceStudio — Faster-Whisper Engine
|
||||
|
||||
Faster-Whisper runs Whisper on CTranslate2 — the same transcription core
|
||||
WhisperX uses, **without** the wav2vec2 forced-alignment pass. It's the safe
|
||||
cross-platform fallback when whisperx isn't installed, and the capture/dictation
|
||||
fallback on non-Apple machines.
|
||||
|
||||
## Selecting it
|
||||
|
||||
- **Model Catalogue → Engines**, ASR tab → **Use** on the Faster-Whisper row, or
|
||||
- pin it with `OMNIVOICE_ASR_BACKEND=faster-whisper`.
|
||||
|
||||
Auto-detect only picks it when [whisperx](whisperx.md) is unavailable.
|
||||
|
||||
## Best at
|
||||
|
||||
- **Subtitles, dictation buffers, and batch transcription** where Whisper's
|
||||
native word timing (±100–300 ms) is good enough.
|
||||
- For dubbing lip-sync, prefer [whisperx](whisperx.md) (or
|
||||
[mlx-whisper](mlx-whisper.md) on Apple Silicon) — their forced alignment is
|
||||
an order of magnitude tighter on word boundaries.
|
||||
|
||||
## Platform support
|
||||
|
||||
- **CUDA** — float16, with automatic degradation (below).
|
||||
- **CPU** — int8 on macOS, Windows, and Linux.
|
||||
- **Apple Silicon GPU / ROCm** — not supported: CTranslate2 has no Metal or
|
||||
HIP build, so those hosts run on CPU
|
||||
([#1529](https://github.com/debpalash/VoiceStudio/issues/1529)); auto-detect
|
||||
routes them to mlx-whisper / pytorch-whisper instead.
|
||||
|
||||
## Model selection
|
||||
|
||||
`ASR_MODEL_FASTER` — default `Systran/faster-whisper-large-v3`. Accepts the
|
||||
size aliases (`tiny` … `large-v3`, `distil-large-v3`) or any CTranslate2
|
||||
Whisper repo on HF. Weights download on first load — see
|
||||
[downloading-models](../downloading-models.md).
|
||||
|
||||
Segments are cleaned up by faster-whisper's built-in Silero VAD before
|
||||
transcription.
|
||||
|
||||
## Degradation chains
|
||||
|
||||
- GPUs without efficient fp16 (older Maxwell/Pascal, GTX 16xx, or a
|
||||
CTranslate2/cuDNN mismatch) fail at model construction with a compute-type
|
||||
error; the engine walks float16 → int8_float16 → int8 instead of failing
|
||||
every chunk ([#551](https://github.com/debpalash/VoiceStudio/issues/551)).
|
||||
- A CUDA out-of-memory falls back to CPU (slower, same model and accuracy) —
|
||||
flushing the resident TTS model frees VRAM for GPU-speed ASR
|
||||
([#255](https://github.com/debpalash/VoiceStudio/issues/255)).
|
||||
|
||||
## Quirks
|
||||
|
||||
- **cuDNN 8 required on CUDA** — a missing cuDNN 8 would fast-fail the whole
|
||||
process, so the engine checks up front and reports itself unavailable
|
||||
instead ([#1371](https://github.com/debpalash/VoiceStudio/issues/1371)).
|
||||
pytorch-whisper covers that case on torch's bundled cuDNN 9.
|
||||
- On some hardened Linux kernels the CTranslate2 native library is rejected
|
||||
with "cannot enable executable stack" (an OSError, not an ImportError) —
|
||||
reported as unavailable rather than crashing engine selection
|
||||
([#692](https://github.com/debpalash/VoiceStudio/issues/692)).
|
||||
- CTranslate2's GPU teardown can rarely segfault the process at unload. If
|
||||
you hit that, switch to the crash-isolated variant —
|
||||
[faster-whisper-isolated](faster-whisper-isolated.md)
|
||||
([#730](https://github.com/debpalash/VoiceStudio/issues/730)).
|
||||
- Transcribes are time-bounded: `OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S`
|
||||
(default 120 s per dub chunk) and `OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S`
|
||||
(default 300 s whole-file).
|
||||
|
||||
Speed comparisons across engines live in [performance](../performance.md).
|
||||
@@ -0,0 +1,62 @@
|
||||
# VoiceStudio — FunASR (SenseVoice) Engine
|
||||
|
||||
FunASR drives Alibaba's SenseVoiceSmall with FSMN-VAD: an all-in-one
|
||||
multilingual pipeline — transcription with punctuation and inverse text
|
||||
normalization across **50+ languages**, plus optional **inline speaker
|
||||
diarization** via the cam++ speaker model. It's the opt-in alternative to
|
||||
WhisperX ([#182](https://github.com/debpalash/VoiceStudio/issues/182));
|
||||
WhisperX remains the cross-platform default.
|
||||
|
||||
## Selecting it
|
||||
|
||||
- Install it into the app venv: `uv pip install funasr`.
|
||||
- Then **Model Catalogue → Engines**, ASR tab → **Use** on the FunASR row, or
|
||||
`OMNIVOICE_ASR_BACKEND=funasr`.
|
||||
|
||||
Auto-detect never picks it; it's an explicit opt-in.
|
||||
|
||||
## Best at
|
||||
|
||||
- **Multi-speaker transcription without any HuggingFace token.** This is the
|
||||
only ASR engine with diarization built in: cam++ labels each sentence
|
||||
(`Speaker 1`, `Speaker 2`, ...) in the same pass — no gated pyannote
|
||||
model, no license click-through. Compare
|
||||
[diarization](../features/diarization.md) for the pyannote/WhisperX route
|
||||
and what each buys you.
|
||||
- **Broad language coverage** beyond Whisper's strongest languages, with
|
||||
punctuation included.
|
||||
|
||||
## Not suited for
|
||||
|
||||
- **Lip-sync dubbing** — FunASR returns sentence-level timestamps, not
|
||||
word-level ones. Use [whisperx](whisperx.md) /
|
||||
[mlx-whisper](mlx-whisper.md) when word timing matters.
|
||||
|
||||
## Platform support
|
||||
|
||||
CUDA or CPU, on macOS, Windows, and Linux.
|
||||
|
||||
## Model selection
|
||||
|
||||
| Variable | Default | Role |
|
||||
| --- | --- | --- |
|
||||
| `ASR_MODEL_FUNASR` | `iic/SenseVoiceSmall` | main ASR model |
|
||||
| `ASR_FUNASR_VAD` | `fsmn-vad` | VAD segmentation model |
|
||||
| `ASR_FUNASR_SPK` | `cam++` | speaker model; set to empty (`ASR_FUNASR_SPK=`) to disable diarization and use the dub pipeline's pyannote/heuristic path instead |
|
||||
|
||||
Weights download on first load (through FunASR's own model hub) — see
|
||||
[downloading-models](../downloading-models.md).
|
||||
|
||||
## Quirks
|
||||
|
||||
- With the speaker model enabled, long recordings are transcribed in **one
|
||||
call** and split by FunASR's internal VAD — cam++ assigns speaker cluster
|
||||
IDs per call, so this is what keeps "Speaker 1" meaning the same person
|
||||
across the whole file.
|
||||
- The engine runs with `spk_mode="vad_segment"`: FunASR 1.3.1's default
|
||||
(`punc_segment`) requires a separate punctuation model and crashes when
|
||||
SenseVoice is loaded without one.
|
||||
- SenseVoice's rich-token markup (language/emotion/event tags around the
|
||||
text) is stripped from the output automatically.
|
||||
- Language detection is automatic (`language: auto`); the detected language
|
||||
is reported per file.
|
||||
@@ -0,0 +1,78 @@
|
||||
# VoiceStudio — GPT-SoVITS Engine
|
||||
|
||||
GPT-SoVITS (RVC-Boss) is one of the most popular open-source voice-cloning
|
||||
systems (57k+ GitHub stars, MIT-licensed). It does zero-shot and few-shot
|
||||
cloning with excellent naturalness in Chinese, English, Japanese, Cantonese,
|
||||
and Korean, and it is very fast (RTF ~0.014 on suitable hardware).
|
||||
|
||||
Unlike VoiceStudio's other engines, GPT-SoVITS does not run inside the app.
|
||||
It ships as a standalone API server, and VoiceStudio connects to it over
|
||||
HTTP.
|
||||
|
||||
## When to pick it
|
||||
|
||||
- You already run (or want to run) a GPT-SoVITS server, e.g. with few-shot
|
||||
fine-tuned voices.
|
||||
- You need fast, natural cloning in zh/en/ja/yue/ko.
|
||||
|
||||
## Setup
|
||||
|
||||
1. Install and start the GPT-SoVITS API server (upstream project):
|
||||
|
||||
```bash
|
||||
cd GPT-SoVITS
|
||||
python api_v2.py -a 127.0.0.1 -p 9880 -c GPT_SoVITS/configs/tts_infer.yaml
|
||||
```
|
||||
|
||||
2. Select the engine via **Model Catalogue → Engines** or
|
||||
`OMNIVOICE_TTS_BACKEND=gpt-sovits`.
|
||||
|
||||
VoiceStudio marks the engine available only when the server responds
|
||||
(2-second reachability probe).
|
||||
|
||||
## Configuration
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `OMNIVOICE_GPTSOVITS_URL` | `http://127.0.0.1:9880` | API server URL |
|
||||
| `OMNIVOICE_TRUSTED_NETWORKS` | (unset) | Required to allow a non-loopback server |
|
||||
|
||||
**Remote servers:** by default VoiceStudio only talks to loopback addresses
|
||||
— part of the local-first guarantee. To point at a server on another
|
||||
machine (e.g. a GPU box on your LAN), add its network to
|
||||
`OMNIVOICE_TRUSTED_NETWORKS`; otherwise the connection is refused as an
|
||||
untrusted endpoint.
|
||||
|
||||
Prefer `https://` (or a private tunnel such as Tailscale/WireGuard) for any
|
||||
non-loopback server: with plain `http://` the text you synthesize and the
|
||||
audio that comes back cross the network unencrypted. VoiceStudio does not
|
||||
disable certificate verification, so a TLS endpoint needs a certificate the
|
||||
system trusts.
|
||||
|
||||
## Behaviour notes
|
||||
|
||||
- Output is 32 kHz mono (server output is resampled if needed).
|
||||
- Cloning passes your reference clip path and optional transcript to the
|
||||
server; the reference path must be readable **by the server process**, so
|
||||
remote servers need the clip on their own filesystem.
|
||||
- Speed control is forwarded as the server's `speed_factor`.
|
||||
- The GPU is whatever the GPT-SoVITS server itself uses (CUDA preferred);
|
||||
VoiceStudio's side is just an HTTP client.
|
||||
|
||||
## Known limits
|
||||
|
||||
- Five languages only; for broader coverage use
|
||||
[OmniVoice](omnivoice.md) ([languages.md](../languages.md)).
|
||||
- No voice design; server availability is your responsibility — if the
|
||||
server stops, generations fail with a "server not reachable" error.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- "GPT-SoVITS server not reachable": start the server with the command
|
||||
above, or fix `OMNIVOICE_GPTSOVITS_URL`.
|
||||
- "endpoint is outside loopback or OMNIVOICE_TRUSTED_NETWORKS": see
|
||||
Configuration above.
|
||||
- Other issues: [install/troubleshooting.md](../install/troubleshooting.md).
|
||||
|
||||
See also: [benchmarks.md](../benchmarks.md),
|
||||
[expressive-speech.md](../expressive-speech.md).
|
||||
@@ -0,0 +1,75 @@
|
||||
# VoiceStudio — KittenTTS Engine
|
||||
|
||||
KittenTTS (KittenML) is the lightweight English "flash" tier: a 25–80 MB
|
||||
ONNX model with 8 preset voices that runs realtime on any CPU — no torch, no
|
||||
CUDA, no GPU of any kind. Use it when you just need quick English narration
|
||||
(voiceovers, demo reads, short phrases) with no reference sample.
|
||||
|
||||
## When to pick it
|
||||
|
||||
- English-only content where speed and a tiny install matter more than
|
||||
cloning.
|
||||
- Machines with no usable GPU.
|
||||
|
||||
The trade-off against [OmniVoice](omnivoice.md): no voice cloning, English
|
||||
only — but a much faster and much smaller install.
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
pip install kittentts
|
||||
```
|
||||
|
||||
Then select the engine via **Model Catalogue → Engines** or
|
||||
`OMNIVOICE_TTS_BACKEND=kittentts`.
|
||||
|
||||
## Voices
|
||||
|
||||
Eight preset voices, four male/female pairs:
|
||||
|
||||
```text
|
||||
expr-voice-2-m expr-voice-2-f (default: expr-voice-2-f)
|
||||
expr-voice-3-m expr-voice-3-f
|
||||
expr-voice-4-m expr-voice-4-f
|
||||
expr-voice-5-m expr-voice-5-f
|
||||
```
|
||||
|
||||
An unknown voice id logs an info message and falls back to the default.
|
||||
|
||||
## Model selection
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `OMNIVOICE_KITTENTTS_MODEL` | `KittenML/kitten-tts-mini-0.8` | HuggingFace checkpoint to load |
|
||||
|
||||
The ~80 MB model downloads from HuggingFace on first use (retried once on a
|
||||
flaky connection). See [downloading-models.md](../downloading-models.md).
|
||||
|
||||
## Behaviour notes
|
||||
|
||||
- Output is 24 kHz mono.
|
||||
- CPU-only by design — the ONNX graph has no CUDA/MPS path.
|
||||
- Non-English `language` values are ignored with a log line pointing at
|
||||
OmniVoice; reference audio is likewise ignored (no cloning).
|
||||
- **Long-input hardening
|
||||
([#1173](https://github.com/debpalash/VoiceStudio/issues/1173)):** the
|
||||
shipped ONNX graph has a hard 512-token cap, and phonemization can expand
|
||||
text massively (digits especially). VoiceStudio pre-measures every chunk
|
||||
with the model's own tokenizer and splits oversized chunks at word
|
||||
boundaries, so long or digit-heavy inputs no longer abort inside
|
||||
onnxruntime with an opaque "invalid expand shape" error.
|
||||
|
||||
## Known limits
|
||||
|
||||
- English only; no cloning, no voice design, no emotion controls
|
||||
(see [expressive-speech.md](../expressive-speech.md)).
|
||||
- Preset voices only — speed is the one knob.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Engine unavailable: `pip install kittentts` into VoiceStudio's Python
|
||||
environment and restart.
|
||||
- Other issues: [install/troubleshooting.md](../install/troubleshooting.md).
|
||||
|
||||
See also: [benchmarks.md](../benchmarks.md),
|
||||
[disk usage](disk-usage.md).
|
||||
@@ -0,0 +1,76 @@
|
||||
# VoiceStudio — MLX-Audio Engine (Apple Silicon)
|
||||
|
||||
MLX-Audio (Blaizzy/mlx-audio) wraps 14+ TTS engines — Kokoro, CSM, Dia,
|
||||
Qwen3-TTS, Chatterbox, MeloTTS, OuteTTS, and more — behind a single adapter
|
||||
that runs on Apple's MLX framework. It is **Apple Silicon only**: the engine
|
||||
is not shipped on Linux, Windows, or Intel Macs, and a stray wheel on those
|
||||
platforms never reports as available
|
||||
([#390](https://github.com/debpalash/VoiceStudio/issues/390)).
|
||||
|
||||
## When to pick it
|
||||
|
||||
- You're on an M-series Mac and want small, fast models tuned for it.
|
||||
- You want one of the specific hosted models (Kokoro for small multilingual,
|
||||
CSM for cloning, Qwen3-TTS for voice design, Dia for dialogue, …).
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
pip install mlx-audio
|
||||
```
|
||||
|
||||
Then select the engine via **Model Catalogue → Engines** or
|
||||
`OMNIVOICE_TTS_BACKEND=mlx-audio`.
|
||||
|
||||
## Model selection
|
||||
|
||||
One backend hosts many models. The curated set:
|
||||
|
||||
| Key | Model | Niche |
|
||||
| --- | --- | --- |
|
||||
| `kokoro` (default) | `mlx-community/Kokoro-82M-bf16` | small multilingual |
|
||||
| `csm` | `mlx-community/csm-1b-8bit` | voice cloning |
|
||||
| `qwen3-tts` | `mlx-community/Qwen3-TTS-12Hz-1.7B-VoiceDesign-4bit` | voice design |
|
||||
| `dia` | `mlx-community/Dia-1.6B` | dialogue |
|
||||
| `chatterbox` | `mlx-community/Chatterbox-TTS-4bit` | expressive |
|
||||
| `melotts` | `mlx-community/MeloTTS-English-v3-MLX` | lightweight VITS |
|
||||
| `outetts` | `mlx-community/Llama-OuteTTS-1.0-1B-4bit` | LM-based |
|
||||
|
||||
Pick a model in the **Model Catalogue → Engines** curated picker
|
||||
([#981](https://github.com/debpalash/VoiceStudio/issues/981)) or set
|
||||
`OMNIVOICE_MLX_AUDIO_MODEL` to either a curated key (`kokoro`) or any full
|
||||
HF repo id. The env var overrides the persisted UI choice.
|
||||
|
||||
## Behaviour notes
|
||||
|
||||
- Output is 24 kHz mono for most hosted models.
|
||||
- **Cloning works only with the `csm` model** — it is the only curated model
|
||||
confirmed to accept a reference clip. Other models silently ignore
|
||||
reference audio, so the engine reports cloning support only when CSM is
|
||||
selected (dub/batch jobs gate on this).
|
||||
- Voice design (text description → voice) is available through the
|
||||
Qwen3-TTS VoiceDesign model.
|
||||
- Language support is per-model (Kokoro ~8 languages, others vary). An
|
||||
unsupported language for Kokoro produces a clear error naming what it
|
||||
does support ([#977](https://github.com/debpalash/VoiceStudio/issues/977))
|
||||
— leave language on Auto or switch to a multilingual engine.
|
||||
|
||||
## Platform notes
|
||||
|
||||
This engine is exempt from cross-platform parity as a platform-only
|
||||
capability behind explicit opt-in: it exists only where Apple's MLX runtime
|
||||
exists. On any other platform the engine picker shows it unavailable with
|
||||
the reason.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Unavailable on an M-series Mac: `pip install mlx-audio` into
|
||||
VoiceStudio's Python environment; in a packaged app build, MLX's native
|
||||
libraries may fail to load — the engine reports unavailable rather than
|
||||
crashing.
|
||||
- Other issues: [install/troubleshooting.md](../install/troubleshooting.md).
|
||||
|
||||
See also: [benchmarks.md](../benchmarks.md),
|
||||
[languages.md](../languages.md),
|
||||
[downloading-models.md](../downloading-models.md),
|
||||
[disk usage](disk-usage.md).
|
||||
@@ -0,0 +1,57 @@
|
||||
# VoiceStudio — MLX Whisper Engine
|
||||
|
||||
MLX Whisper runs Whisper on the Apple Silicon GPU via MLX. It exists because
|
||||
CTranslate2 (whisperx / faster-whisper) has **no Metal build** — on a Mac
|
||||
those engines transcribe on the CPU no matter what GPU is present. Measured
|
||||
on an M2 with whisper-large-v3, one 30 s dub chunk: **90.4 s on WhisperX
|
||||
(CPU) vs 20.5 s on MLX (GPU)** — which is why auto-detect picks MLX Whisper
|
||||
on every Apple Silicon machine
|
||||
([#1127](https://github.com/debpalash/VoiceStudio/issues/1127)).
|
||||
|
||||
## Selecting it
|
||||
|
||||
- Nothing to do on Apple Silicon — auto-detect prefers it there.
|
||||
- Or explicitly: **Model Catalogue → Engines**, ASR tab → **Use**, or
|
||||
`OMNIVOICE_ASR_BACKEND=mlx-whisper`.
|
||||
|
||||
## Best at
|
||||
|
||||
- **Dubbing on a Mac** — it layers the same wav2vec2 forced alignment
|
||||
WhisperX uses on top of the GPU transcription, so word timing (±10–30 ms)
|
||||
and therefore lip-sync accuracy are unchanged. Same model, same alignment,
|
||||
~4x the speed.
|
||||
- **Dictation/capture** — the capture path automatically swaps in
|
||||
`mlx-community/whisper-large-v3-turbo` (~5x faster than large-v3) unless a
|
||||
sherpa dictation model or [parakeet-mlx](parakeet-mlx.md) is preferred.
|
||||
|
||||
## Platform support
|
||||
|
||||
**Apple Silicon only.** A shared platform gate refuses Linux, Windows, and
|
||||
Intel Macs before any package import, so a stray `mlx-whisper` wheel on the
|
||||
wrong platform never reports itself available
|
||||
([#390](https://github.com/debpalash/VoiceStudio/issues/390)). All other
|
||||
platforms use the CUDA/CPU engines instead.
|
||||
|
||||
## Model selection
|
||||
|
||||
- `ASR_MODEL` — default `mlx-community/whisper-large-v3-mlx`. Any MLX-format
|
||||
Whisper repo works. Weights download on first load — see
|
||||
[downloading-models](../downloading-models.md).
|
||||
- `OMNIVOICE_ALIGN_DEVICE` — force the wav2vec2 aligner's device. The aligner
|
||||
runs on MPS when it can and falls back to CPU; languages without a bundled
|
||||
aligner (~20 major languages have one) keep Whisper's native word
|
||||
timestamps.
|
||||
|
||||
## Quirks
|
||||
|
||||
- Audio is decoded through VoiceStudio's validated ffmpeg rather than the
|
||||
bare `ffmpeg` PATH lookup mlx-whisper would do on its own — a clean
|
||||
from-source install with no system ffmpeg works fine
|
||||
([#479](https://github.com/debpalash/VoiceStudio/issues/479)).
|
||||
- The model is warmed into unified memory in the background, so the first
|
||||
transcribe after startup doesn't pay the load cost.
|
||||
- In a packaged app, a native MLX library that fails to load is reported as
|
||||
"unavailable" (with fallback to another engine) rather than crashing the
|
||||
engine list.
|
||||
|
||||
Speed comparisons across engines live in [performance](../performance.md).
|
||||
@@ -0,0 +1,48 @@
|
||||
# VoiceStudio — Moonshine Engine
|
||||
|
||||
Moonshine is an edge-optimized ASR family built for CPU-only machines.
|
||||
Unlike Whisper it processes variable-length audio (no padding everything to
|
||||
30 s), which keeps latency low on short clips — sub-200 ms class on capture
|
||||
buffers. It's the lightest local option for quick transcription on hardware
|
||||
where even int8 whisper-large is too slow.
|
||||
|
||||
## Selecting it
|
||||
|
||||
- Install one of the runtimes into the app venv:
|
||||
`uv pip install moonshine-onnx` (lighter, tried first) or
|
||||
`moonshine-voice`.
|
||||
- Then **Model Catalogue → Engines**, ASR tab → **Use** on the Moonshine row,
|
||||
or `OMNIVOICE_ASR_BACKEND=moonshine`.
|
||||
|
||||
Auto-detect never picks it; it's an explicit opt-in.
|
||||
|
||||
## Best at
|
||||
|
||||
- **Quick notes and short-clip transcription on low-power CPU machines.**
|
||||
- Environments where a sub-1 GB footprint matters more than word timing or
|
||||
language coverage.
|
||||
|
||||
## Not suited for
|
||||
|
||||
- **Dubbing.** Output is plain text as a **single segment spanning the whole
|
||||
file — no word or segment timestamps** — so there's nothing for lip-sync
|
||||
or subtitle timing to work with. Use a Whisper-family engine or
|
||||
[sherpa-onnx-asr](sherpa-onnx-asr.md) for those jobs.
|
||||
- Multilingual work: results report English; for broad language coverage use
|
||||
[whisperx](whisperx.md) or [funasr](funasr.md).
|
||||
|
||||
## Platform support
|
||||
|
||||
CPU only, by design — macOS, Windows, and Linux. It claims no GPU.
|
||||
|
||||
## Model selection
|
||||
|
||||
`ASR_MODEL_MOONSHINE` — default `moonshine/base`. Weights download on first
|
||||
load — see [downloading-models](../downloading-models.md).
|
||||
|
||||
## Quirks
|
||||
|
||||
- The engine tries `moonshine_onnx` first and falls back to
|
||||
`moonshine_voice` — installing either one is enough.
|
||||
- Segment bounds are synthesized from the audio duration (start 0, end =
|
||||
file length), since the model reports none.
|
||||
@@ -0,0 +1,78 @@
|
||||
# VoiceStudio — MOSS-TTS-Nano Engine
|
||||
|
||||
MOSS-TTS-Nano (OpenMOSS) is the low-resource, broad-language pick: a
|
||||
100M-parameter autoregressive codec LM that runs realtime on a 4-core CPU —
|
||||
no GPU required — with native 48 kHz output and 20 languages under an
|
||||
Apache-2.0 license. It fills the "runs on a fanless laptop" tier while still
|
||||
covering languages like Arabic, Hebrew, Persian, Korean, and Turkish.
|
||||
|
||||
## When to pick it
|
||||
|
||||
- CPU-only or low-power hardware, but you still need cloning and non-English
|
||||
coverage.
|
||||
- Your language is among: Chinese, English, German, Spanish, French,
|
||||
Japanese, Italian, Hebrew, Korean, Russian, Persian, Arabic, Polish,
|
||||
Portuguese, Czech, Danish, Swedish, Hungarian, Greek, Turkish.
|
||||
|
||||
## Setup
|
||||
|
||||
The package is **not on PyPI** — install it from the upstream repo into
|
||||
VoiceStudio's Python environment:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/OpenMOSS/MOSS-TTS-Nano.git
|
||||
cd MOSS-TTS-Nano
|
||||
uv pip install -e .
|
||||
```
|
||||
|
||||
Then select the engine via **Model Catalogue → Engines** or
|
||||
`OMNIVOICE_TTS_BACKEND=moss-tts-nano`.
|
||||
|
||||
## Model selection
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `OMNIVOICE_MOSS_TTS_MODEL` | `OpenMOSS-Team/MOSS-TTS-Nano` | HuggingFace checkpoint to load |
|
||||
|
||||
The first use downloads the weights (retried once on a truncated download).
|
||||
See [downloading-models.md](../downloading-models.md).
|
||||
|
||||
## Behaviour notes
|
||||
|
||||
- **Cloning is reference-only**: pass a reference clip. Style instructions,
|
||||
preset speakers, and speed control are not supported and are silently
|
||||
ignored, so mixed-engine call sites keep working.
|
||||
- The model emits 48 kHz stereo; VoiceStudio downmixes to mono, matching the
|
||||
rest of the pipeline (the dub mixer treats TTS output as mono per
|
||||
segment).
|
||||
- Runs on CPU or CUDA.
|
||||
|
||||
## Upstream is unpinned
|
||||
|
||||
The upstream repo is installed straight from git with no pinned release, and
|
||||
the model class it exports has changed before
|
||||
([#1287](https://github.com/debpalash/VoiceStudio/issues/1287)). VoiceStudio
|
||||
therefore verifies that a usable model class actually exists — not just that
|
||||
the package imports — before reporting the engine as ready. If the engine
|
||||
shows unavailable with a "does not expose a usable model class" message,
|
||||
pull the latest upstream and re-run `uv pip install -e .`, or open an issue
|
||||
with the version you have.
|
||||
|
||||
## Known limits
|
||||
|
||||
- No voice design, no instruct, no speed control — cloning from a reference
|
||||
clip only.
|
||||
- Quality sits below the large engines; see
|
||||
[benchmarks.md](../benchmarks.md).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- "moss_tts_nano package not installed": run the clone + `uv pip install -e .`
|
||||
steps above.
|
||||
- Entry-point errors after an upstream update: see "Upstream is unpinned"
|
||||
above.
|
||||
- General issues: [install/troubleshooting.md](../install/troubleshooting.md).
|
||||
|
||||
See also: [languages.md](../languages.md),
|
||||
[expressive-speech.md](../expressive-speech.md),
|
||||
[disk usage](disk-usage.md).
|
||||
@@ -0,0 +1,58 @@
|
||||
# VoiceStudio — Parakeet TDT (NVIDIA NeMo) Engine
|
||||
|
||||
NVIDIA's Parakeet TDT via the NeMo toolkit: a FastConformer encoder with a
|
||||
Token-and-Duration Transducer decoder. It beats Whisper large-v3 on English
|
||||
benchmarks (~6% WER) and supports **25 (mostly European) languages** with
|
||||
automatic language detection. The 0.6B model is fast even on CPU — measured
|
||||
RTF 0.08–0.23 on an Apple Silicon M2 CPU (2026-07-02), ~20x faster than
|
||||
faster-whisper large-v3 int8 on the same host.
|
||||
|
||||
## Do not install NeMo into the app venv
|
||||
|
||||
`nemo_toolkit`'s ASR extras pin `transformers>=4.57,<4.58`, which conflicts
|
||||
with VoiceStudio's own `transformers>=5.3` requirement and **will break the
|
||||
backend** (ImportError on startup) if installed into the shared venv. There
|
||||
is currently no safe in-app install path for this engine; in-app isolation
|
||||
is tracked separately.
|
||||
|
||||
If you want the Parakeet models without a separate environment, use these
|
||||
instead — same model family, no NeMo dependency:
|
||||
|
||||
- **Apple Silicon:** [parakeet-mlx](parakeet-mlx.md) (installed by default on
|
||||
mac-ARM source installs).
|
||||
- **Any platform, CPU:** [sherpa-onnx-asr](sherpa-onnx-asr.md) — its default
|
||||
dictation model is an int8 ONNX export of Parakeet TDT v3.
|
||||
|
||||
## Selecting it
|
||||
|
||||
Only meaningful if you've set up `nemo_toolkit[asr]` in a **separate,
|
||||
dedicated Python environment** that runs the backend:
|
||||
|
||||
- **Model Catalogue → Engines**, ASR tab → **Use** on the Parakeet TDT row, or
|
||||
- `OMNIVOICE_ASR_BACKEND=nemo-parakeet`.
|
||||
|
||||
Auto-detect never picks it; it's an explicit opt-in.
|
||||
|
||||
## Best at
|
||||
|
||||
- **English and European-language transcription** where WER matters more
|
||||
than word-level subtitle timing.
|
||||
- **CPU-only hosts** — faster than realtime without any GPU.
|
||||
|
||||
## Platform support
|
||||
|
||||
CUDA or CPU (the old hard CUDA gate was removed — see the RTF numbers
|
||||
above). Availability is a pure dependency check on `nemo.collections.asr`.
|
||||
|
||||
## Model selection
|
||||
|
||||
`ASR_MODEL_NEMO` — default `nvidia/parakeet-tdt-0.6b-v3`. Weights download
|
||||
on first load — see [downloading-models](../downloading-models.md).
|
||||
|
||||
## Quirks
|
||||
|
||||
- Output is a **single segment** for the whole file (NeMo doesn't VAD-split
|
||||
like Whisper), with word timestamps when the model exposes them — fine for
|
||||
dictation and plain transcripts, not ideal for long-form subtitles.
|
||||
- The detected language isn't exposed cleanly by NeMo, so results report
|
||||
`en` regardless of the actual (auto-detected) language.
|
||||
@@ -0,0 +1,83 @@
|
||||
# VoiceStudio — OmniVoice GGUF Engine
|
||||
|
||||
OmniVoice GGUF runs the same OmniVoice model as the [default
|
||||
engine](omnivoice.md), but through a bundled native binary
|
||||
(`bin/omnivoice-tts-<platform>`) loading quantized GGUF weights. It is
|
||||
hardware-adaptive: a probe picks the quantization that fits your machine, so
|
||||
small GPUs and CPU-only hosts get a working OmniVoice instead of a paging,
|
||||
timing-out one.
|
||||
|
||||
## When to pick it
|
||||
|
||||
- Your GPU is below the default engine's 6 GB VRAM floor.
|
||||
- CPU-only machines that still want OmniVoice's voice and language coverage.
|
||||
- You want generation isolated in a separate process (a crash or leak never
|
||||
takes the app down — each generation spawns the binary fresh).
|
||||
|
||||
## Quantization selection
|
||||
|
||||
Weights come from the `Serveurperso/OmniVoice-GGUF` HuggingFace repo, pinned
|
||||
to an exact revision. The hardware probe selects:
|
||||
|
||||
| Hardware | Quant | Approx. VRAM use |
|
||||
| --- | --- | --- |
|
||||
| 12 GB+ VRAM | BF16 | ~1.6 GB (quality-first) |
|
||||
| 4–12 GB VRAM | Q8_0 | ~945 MB (recommended balance) |
|
||||
| 1–4 GB VRAM | Q4_K_M | ~659 MB (minimal footprint) |
|
||||
| CPU-only | Q4_K_M | RAM-bound, latency-tolerable |
|
||||
|
||||
You can override the selection from Settings; overrides are allow-listed
|
||||
against the same table (an F32 reference quant, ~3.2 GB, is override-only).
|
||||
|
||||
## Setup
|
||||
|
||||
Nothing to install: installer and CI builds bundle the binary for your
|
||||
platform. Select the engine via **Model Catalogue → Engines** or
|
||||
`OMNIVOICE_TTS_BACKEND=omnivoice-gguf`. The quant weights download on first
|
||||
use (see [downloading-models.md](../downloading-models.md)) — install them
|
||||
ahead of time from **Model Catalogue → Models** if you want the first
|
||||
generation to be quick; a long first render is the download, not a hang.
|
||||
|
||||
**Source checkouts:** the repo ships zero-byte placeholders in `bin/` — real
|
||||
binaries come from CI or the installer. The engine detects a placeholder and
|
||||
reports unavailable with instructions
|
||||
([#1172](https://github.com/debpalash/VoiceStudio/issues/1172)) instead of
|
||||
failing at spawn time; build one with
|
||||
`scripts/build-omnivoice-tts.sh --platform <slug>` or use the default
|
||||
in-process engine.
|
||||
|
||||
## Integrity and self-healing
|
||||
|
||||
Before reporting ready, the engine:
|
||||
|
||||
- verifies the binary against the SHA-256 manifest (`bin/checksums.sha256`);
|
||||
- detects macOS Gatekeeper quarantine and prints the exact
|
||||
`xattr -cr '/Applications/VoiceStudio.app'` fix;
|
||||
- restores a missing execute bit (a git clone or zip extract on POSIX can
|
||||
drop `+x`, which used to surface as a permission error mislabeled as
|
||||
out-of-memory — [#437](https://github.com/debpalash/VoiceStudio/issues/437)).
|
||||
The chmod runs only after the SHA check confirms it's the right file.
|
||||
|
||||
## Behaviour notes
|
||||
|
||||
- Output is 24 kHz mono — same model, same rate as in-process OmniVoice.
|
||||
- Cloning from a reference clip (with optional transcript) and style
|
||||
instructions are supported; no voice design.
|
||||
- Same multilingual surface as OmniVoice ([languages.md](../languages.md)).
|
||||
- Because generation runs in another process, the app's own GPU counters
|
||||
don't see its allocations — diagnostics label it accordingly.
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `OMNIVOICE_GGUF_GENERATE_TIMEOUT_S` | (generous built-in) | Per-generation timeout for the spawned binary |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- "GGUF binary missing": this build doesn't bundle the runtime for your
|
||||
platform — use the default engine.
|
||||
- Checksum mismatch or quarantine messages: follow the printed fix, or
|
||||
reinstall.
|
||||
- Other issues: [install/troubleshooting.md](../install/troubleshooting.md).
|
||||
|
||||
See also: [benchmarks.md](../benchmarks.md),
|
||||
[performance.md](../performance.md), [disk usage](disk-usage.md).
|
||||
@@ -0,0 +1,74 @@
|
||||
# VoiceStudio — OmniVoice Engine (default)
|
||||
|
||||
OmniVoice (k2-fsa/OmniVoice) is VoiceStudio's default TTS engine — the one a
|
||||
fresh install uses without any configuration. It does zero-shot voice cloning
|
||||
across 600+ languages and outputs 24 kHz mono audio. Voice cloning, dubbing,
|
||||
and dictation all run on it out of the box.
|
||||
|
||||
## When to pick it
|
||||
|
||||
- You want cloning plus the broadest language coverage (see
|
||||
[languages.md](../languages.md)).
|
||||
- You have a GPU (CUDA or Apple Silicon MPS) with ~6 GB VRAM or more.
|
||||
- You just installed VoiceStudio — it's already selected.
|
||||
|
||||
For low-VRAM or CPU-only machines, the
|
||||
[OmniVoice GGUF](omnivoice-gguf.md) variant runs the same model through a
|
||||
quantized native binary with a much smaller memory footprint.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Runs on CUDA, MPS (Apple Silicon), or CPU — auto-detected.
|
||||
- Recommended VRAM floor: **6 GB** on a dedicated GPU. This is the only
|
||||
engine with a measured floor: on 4 GB cards (GTX 1650 Ti, Quadro P2000 —
|
||||
issues [#1226](https://github.com/debpalash/VoiceStudio/issues/1226) /
|
||||
[#1222](https://github.com/debpalash/VoiceStudio/issues/1222)) the driver
|
||||
pages to system RAM and a render that should take seconds runs for minutes
|
||||
until the compute budget kills it. The UI warns before you wait; nothing
|
||||
hard-blocks, since short inputs can still fit.
|
||||
- No extra install — the model ships with the app and downloads its weights
|
||||
on first use (see [downloading-models.md](../downloading-models.md)).
|
||||
|
||||
## Selecting the engine
|
||||
|
||||
OmniVoice is the default, so normally there is nothing to do. If you switched
|
||||
away and want it back:
|
||||
|
||||
- **Model Catalogue → Engines**, or
|
||||
- set `OMNIVOICE_TTS_BACKEND=omnivoice`.
|
||||
|
||||
The env var overrides the persisted UI choice.
|
||||
|
||||
## Behaviour notes
|
||||
|
||||
- Weights load lazily on first use and are shared with the rest of the app
|
||||
(dubbing, dictation) — the model is never double-loaded.
|
||||
- On CUDA the model runs fp16 with `torch.compile`; a speech recognizer is
|
||||
co-loaded for the cloning path.
|
||||
- Output is 24 kHz mono; the shared mastering chain (highpass + compressor)
|
||||
is tuned for this rate and applied automatically.
|
||||
- Cloning takes a short reference clip (`ref_audio`); an optional transcript
|
||||
of the clip improves conditioning.
|
||||
|
||||
## Known limits
|
||||
|
||||
- No voice design from a text description — use [VoxCPM2](voxcpm2.md) for
|
||||
that.
|
||||
- Below the 6 GB VRAM floor, expect very slow renders or budget timeouts;
|
||||
prefer [OmniVoice GGUF](omnivoice-gguf.md) or a CPU engine such as
|
||||
[PocketTTS](pockettts.md).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- "Too heavy for the available compute" on a small GPU: see the VRAM floor
|
||||
above — switch to OmniVoice GGUF or close other GPU apps.
|
||||
- First generation is slow: the first call downloads multi-GB weights. To
|
||||
keep the first render quick, install the model ahead of time from
|
||||
**Model Catalogue → Models** — a long first generate is almost always the
|
||||
download, not a hang.
|
||||
- General install issues: [install/troubleshooting.md](../install/troubleshooting.md).
|
||||
|
||||
See also: [benchmarks.md](../benchmarks.md),
|
||||
[performance.md](../performance.md),
|
||||
[expressive-speech.md](../expressive-speech.md),
|
||||
[disk usage](disk-usage.md).
|
||||
@@ -0,0 +1,58 @@
|
||||
# VoiceStudio — Parakeet TDT v3 (MLX) Engine
|
||||
|
||||
NVIDIA's Parakeet TDT v3 on the Apple Silicon GPU, via the small pure-Python
|
||||
`parakeet-mlx` package. It gives Macs the Parakeet tier CUDA/CPU users get
|
||||
through NeMo or sherpa-onnx: **25 European languages**, word timestamps from
|
||||
the TDT decoder itself (no wav2vec2 alignment pass needed), ~1.2 GB download,
|
||||
~2 GB unified memory, dictation-grade speed on the GPU.
|
||||
|
||||
Unlike [nemo-parakeet](nemo-parakeet.md) it needs no `nemo_toolkit` (whose
|
||||
transformers pin conflicts with the app's) — it is **installed by default on
|
||||
Apple Silicon source installs since 0.3.22**.
|
||||
|
||||
## Selecting it
|
||||
|
||||
- **Model Catalogue → Engines**, ASR tab → **Use** on the Parakeet TDT v3
|
||||
(MLX) row, or `OMNIVOICE_ASR_BACKEND=parakeet-mlx`.
|
||||
- **Dictation prefers it automatically**: once the model weights are
|
||||
installed (Model Catalogue → Models — the auto-pick never triggers a
|
||||
download), live dictation/capture uses it whenever your system language is
|
||||
one of the 25 covered European languages. Other languages keep the
|
||||
multilingual Whisper engine, so dictation coverage never regresses.
|
||||
|
||||
## Best at
|
||||
|
||||
- **Live dictation on a Mac** — TDT decoding is fast enough for the capture
|
||||
path, at Parakeet's better-than-Whisper English WER.
|
||||
- **European-language transcription** with word timestamps at a fraction of
|
||||
whisper-large-v3's memory and compute.
|
||||
|
||||
For languages outside the 25 (CJK, Arabic, ...), use
|
||||
[mlx-whisper](mlx-whisper.md) instead.
|
||||
|
||||
## Platform support
|
||||
|
||||
**Apple Silicon only** — the same shared MLX platform gate as mlx-whisper
|
||||
refuses Linux, Windows, and Intel Macs before any import
|
||||
([#390](https://github.com/debpalash/VoiceStudio/issues/390)). It runs on the
|
||||
unified-memory GPU; there is no CPU tier.
|
||||
|
||||
## Model selection
|
||||
|
||||
`ASR_MODEL_PARAKEET_MLX` — default `mlx-community/parakeet-tdt-0.6b-v3`.
|
||||
Weights download on first load — see
|
||||
[downloading-models](../downloading-models.md).
|
||||
|
||||
## Quirks
|
||||
|
||||
- Long files are processed in 120 s chunks internally to bound unified-memory
|
||||
use; short dictation buffers and dub chunks are unaffected.
|
||||
- Parakeet v3 auto-detects among its 25 languages but doesn't expose the
|
||||
pick, so the reported language is the one you requested (or none) — it is
|
||||
never hardcoded to English.
|
||||
- Word timestamps are merged from the decoder's subword tokens — good for
|
||||
subtitles and dictation; for lip-sync-critical dubbing the wav2vec2-aligned
|
||||
engines ([mlx-whisper](mlx-whisper.md), [whisperx](whisperx.md)) remain the
|
||||
accuracy tier.
|
||||
|
||||
Speed comparisons across engines live in [performance](../performance.md).
|
||||
@@ -0,0 +1,82 @@
|
||||
# VoiceStudio — PocketTTS Engine
|
||||
|
||||
PocketTTS (kyutai-labs/pocket-tts, 100M parameters) is the fastest-CPU-render
|
||||
pick: small, low-latency, CPU-only, with zero-shot voice cloning from a
|
||||
reference clip. It covers six languages — English, French, German,
|
||||
Portuguese, Italian, Spanish — with one model per language, and measures
|
||||
roughly 8–9x real-time on an Apple M3 Pro.
|
||||
|
||||
It complements the quality engines: where they fall back to CPU, PocketTTS
|
||||
is built for it. CPU-only is deliberate — upstream observes no GPU speedup
|
||||
for this model.
|
||||
|
||||
## When to pick it
|
||||
|
||||
- CPU-only machines that need fast rendering *and* voice cloning.
|
||||
- Latency-sensitive use (dictation-style, short utterances) in one of the
|
||||
six languages.
|
||||
|
||||
## Setup
|
||||
|
||||
1. Install the optional dependency:
|
||||
|
||||
```bash
|
||||
uv sync --extra pockettts
|
||||
```
|
||||
|
||||
(Or enable it from **Model Catalogue → Engines**.)
|
||||
|
||||
2. **Accept the license in-app**
|
||||
([#1306](https://github.com/debpalash/VoiceStudio/issues/1306)). The code
|
||||
is MIT and the weights are CC-BY-4.0, but the weights are **gated on
|
||||
HuggingFace** behind an access agreement with an acceptable-use clause.
|
||||
VoiceStudio surfaces this before first use: the engine stays unavailable
|
||||
until you review and accept in **Model Catalogue → Engines → PocketTTS**.
|
||||
You also need HuggingFace access to the gated repo (see
|
||||
[downloading-models.md](../downloading-models.md) for token setup).
|
||||
|
||||
3. Select the engine via **Model Catalogue → Engines** or
|
||||
`OMNIVOICE_TTS_BACKEND=pockettts`.
|
||||
|
||||
## Platform notes
|
||||
|
||||
- Works on Linux, Windows, macOS Apple Silicon — CPU only everywhere.
|
||||
- **Not available on Intel Macs**: the required PyTorch version has no
|
||||
macOS x86_64 wheel. The engine reports this plainly instead of failing
|
||||
mid-install.
|
||||
|
||||
## Behaviour notes
|
||||
|
||||
- Output is 24 kHz mono.
|
||||
- Six languages, one model per language, chosen by the `language` you
|
||||
request; cloning takes a short reference clip.
|
||||
- Runs in a crash-isolated sidecar process (parent Python environment): a
|
||||
wedged generation is hard-killed by a watchdog and its memory reclaimed —
|
||||
something an in-process engine cannot do.
|
||||
- The first use downloads the gated weights; the sidecar heartbeats
|
||||
progress during the download so the watchdog doesn't fire.
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `OMNIVOICE_POCKETTTS_RECV_TIMEOUT_S` | `600` | Sidecar response deadline in seconds (min 30; cold loads download weights) |
|
||||
|
||||
## Known limits
|
||||
|
||||
- No voice design, no emotion controls
|
||||
(see [expressive-speech.md](../expressive-speech.md)).
|
||||
- Six languages only — for broader coverage use
|
||||
[OmniVoice](omnivoice.md) ([languages.md](../languages.md)).
|
||||
- Revoking the license acceptance takes effect immediately, without a
|
||||
restart — subsequent generations refuse.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- "pocket_tts package not installed": run the `uv sync` above.
|
||||
- "license not accepted": open **Model Catalogue → Engines → PocketTTS**
|
||||
and review/accept.
|
||||
- Timeouts on a slow connection: raise
|
||||
`OMNIVOICE_POCKETTTS_RECV_TIMEOUT_S` for the first (download-heavy) run.
|
||||
- Other issues: [install/troubleshooting.md](../install/troubleshooting.md).
|
||||
|
||||
See also: [benchmarks.md](../benchmarks.md),
|
||||
[performance.md](../performance.md), [disk usage](disk-usage.md).
|
||||
@@ -0,0 +1,61 @@
|
||||
# VoiceStudio — PyTorch Whisper Engine
|
||||
|
||||
Whisper through the plain `transformers` pipeline, riding torch itself. No
|
||||
extra install — transformers ships with the app — and because it runs on
|
||||
torch's own stack (including torch's bundled cuDNN 9), it works on machines
|
||||
where the CTranslate2 engines can't load. It is also the engine that
|
||||
genuinely uses **AMD ROCm** GPUs, so auto-detect picks it on ROCm hosts
|
||||
([#1529](https://github.com/debpalash/VoiceStudio/issues/1529)).
|
||||
|
||||
## Selecting it
|
||||
|
||||
- **Model Catalogue → Engines**, ASR tab → **Use** on the PyTorch Whisper
|
||||
row, or `OMNIVOICE_ASR_BACKEND=pytorch-whisper`.
|
||||
- Auto-detect picks it on ROCm, and as the last resort everywhere else.
|
||||
|
||||
## Best at
|
||||
|
||||
- **ROCm dubbing/transcription** — the only Whisper engine that uses the HIP
|
||||
GPU (CTranslate2 has no HIP build, MLX is Apple-only).
|
||||
- **Rescue engine** when whisperx/faster-whisper can't load — e.g. the
|
||||
missing-cuDNN-8 case
|
||||
([#255](https://github.com/debpalash/VoiceStudio/issues/255)) — since it
|
||||
needs neither CTranslate2 nor cuDNN 8.
|
||||
|
||||
For lip-sync-grade word timing prefer [whisperx](whisperx.md) or
|
||||
[mlx-whisper](mlx-whisper.md); this engine returns the pipeline's own word
|
||||
timestamps.
|
||||
|
||||
## Platform support
|
||||
|
||||
CUDA, Apple Silicon (MPS), ROCm (HIP), and CPU — wherever torch runs, on
|
||||
macOS, Windows, and Linux.
|
||||
|
||||
## Model selection
|
||||
|
||||
`OMNIVOICE_PYTORCH_ASR_MODEL` — default `openai/whisper-large-v3-turbo`. Any
|
||||
transformers-format Whisper repo works. Weights download on first load — see
|
||||
[downloading-models](../downloading-models.md).
|
||||
|
||||
## VRAM preflight
|
||||
|
||||
whisper-large-v3-turbo needs roughly 3.2 GiB before generation adds its
|
||||
workspace; loading it onto a nearly-full card "succeeds" and then the first
|
||||
transcribe OOMs with zero segments. So on CUDA the engine checks free VRAM
|
||||
against a 5 GB budget before loading and uses the CPU instead when the card
|
||||
is too full (flush the TTS model to restore GPU-speed ASR). Disable with
|
||||
`OMNIVOICE_ASR_VRAM_PREFLIGHT=0`.
|
||||
|
||||
## Quirks
|
||||
|
||||
- If the pipeline fails to import (`AutoFeatureExtractor` errors), the cause
|
||||
is either an incomplete transformers install or a torch/torchvision
|
||||
version mismatch — the error message names the exact reinstall command;
|
||||
the trio has to move together at the pinned versions
|
||||
([#549](https://github.com/debpalash/VoiceStudio/issues/549),
|
||||
[#1376](https://github.com/debpalash/VoiceStudio/issues/1376)).
|
||||
- Transcribes are time-bounded like every local engine:
|
||||
`OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S` (default 120 s per dub chunk),
|
||||
`OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S` (default 300 s whole-file).
|
||||
|
||||
Speed comparisons across engines live in [performance](../performance.md).
|
||||
@@ -0,0 +1,71 @@
|
||||
# VoiceStudio — Sherpa-ONNX Dictation Engine
|
||||
|
||||
The k2-fsa/sherpa-onnx ONNX runtime as a **live dictation** engine: small
|
||||
int8 models that transcribe faster than realtime on CPU, with identical
|
||||
behavior on macOS (arm64 + x86_64), Windows, and Linux — no CUDA dependency.
|
||||
Streaming models emit partial text frame-by-frame as you speak; offline
|
||||
models re-transcribe a growing buffer on a short cadence, so you see live
|
||||
partials either way.
|
||||
|
||||
## Selecting it
|
||||
|
||||
- Ensure `sherpa-onnx` is installed (`uv add sherpa-onnx` on source installs).
|
||||
- Pick a dictation model in the app (Model Catalogue → Models lists the
|
||||
curated set below), or **Model Catalogue → Engines**, ASR tab → **Use**, or
|
||||
pin `OMNIVOICE_ASR_BACKEND=sherpa-onnx-asr`.
|
||||
- `OMNIVOICE_SHERPA_ASR_MODEL` selects the model — default
|
||||
`sherpa-parakeet-tdt-v3`.
|
||||
|
||||
## Best at
|
||||
|
||||
- **Live dictation on CPU** — the whole point of this engine. Fast partials,
|
||||
automatic endpointing on silence, no GPU required.
|
||||
- It also honors the regular offline `transcribe` contract, so any of its
|
||||
models can transcribe a file — plain text, single segment, no word
|
||||
timestamps, which makes it a dictation/notes tool rather than a dubbing
|
||||
engine.
|
||||
|
||||
## The 7 curated models
|
||||
|
||||
| Id | Type | Languages | Download |
|
||||
| --- | --- | --- | --- |
|
||||
| `sherpa-parakeet-tdt-v3` (default) | offline | 25 European languages | 0.67 GB |
|
||||
| `sherpa-parakeet-tdt-v2` | offline | English | 0.66 GB |
|
||||
| `sherpa-zipformer-bilingual-zh-en` | streaming | Chinese + English | 0.20 GB |
|
||||
| `sherpa-paraformer-bilingual-zh-en` | streaming | Chinese + English | 0.24 GB |
|
||||
| `sherpa-zipformer-en-20m` | streaming | English | 0.044 GB |
|
||||
| `sherpa-zipformer-zh-14m` | streaming | Chinese | 0.025 GB |
|
||||
| `sherpa-whisper-tiny` | offline | 90+ languages (auto-detect) | 0.104 GB |
|
||||
|
||||
Sizes are measured on-disk download sizes. Weights are int8 ONNX checkpoints
|
||||
that download on first use through the same HF cache as everything else —
|
||||
see [downloading-models](../downloading-models.md). Peak RAM for the 0.6B
|
||||
Parakeets is noticeably higher than their download size (onnxruntime's arena
|
||||
allocator holds onto freed blocks).
|
||||
|
||||
## Platform support
|
||||
|
||||
CPU on every platform, by the strict cross-platform default-parity rule.
|
||||
`OMNIVOICE_SHERPA_ASR_PROVIDER` can override the ONNX provider on a verified
|
||||
GPU build, but the default never diverges.
|
||||
|
||||
## Tuning
|
||||
|
||||
- `OMNIVOICE_SHERPA_ASR_THREADS` — decode threads (default 2; the 0.6B
|
||||
Parakeets automatically use up to 4 when the host has the cores, so decode
|
||||
keeps ahead of the speaker).
|
||||
- `OMNIVOICE_DICTATION_ENDPOINT_R1` / `OMNIVOICE_DICTATION_ENDPOINT_R2` —
|
||||
streaming endpoint rules in seconds (defaults 1.0 / 0.6: text commits
|
||||
~0.6 s after you stop speaking). Applied without a restart.
|
||||
|
||||
## Quirks
|
||||
|
||||
- The recognizer is **pre-warmed in the background** so the first dictation
|
||||
session doesn't pay the 1.3–2.5 s ONNX session load
|
||||
([#888](https://github.com/debpalash/VoiceStudio/issues/888)); it's then
|
||||
shared warm across sessions.
|
||||
- On Apple Silicon, installing the [parakeet-mlx](parakeet-mlx.md) model
|
||||
makes dictation prefer the GPU Parakeet automatically for the 25 covered
|
||||
languages; an explicitly selected sherpa model still wins.
|
||||
- The offline `transcribe` path reports `language: auto` — per-file language
|
||||
detection is only meaningful for the Whisper Tiny model.
|
||||
@@ -0,0 +1,75 @@
|
||||
# VoiceStudio — Sherpa-ONNX Engine
|
||||
|
||||
Sherpa-ONNX (k2-fsa/sherpa-onnx) is a unified C++ ONNX runtime that wraps
|
||||
20+ TTS model families (VITS, MeloTTS, Piper, Kokoro, Matcha, and more)
|
||||
behind one API, with pre-built wheels for Linux, Windows, and macOS (x86 and
|
||||
ARM). You bring the model: point VoiceStudio at any downloaded sherpa-onnx
|
||||
TTS model directory.
|
||||
|
||||
## When to pick it
|
||||
|
||||
- You want a specific community model (e.g. a Piper or VITS voice for your
|
||||
language) that no other engine hosts.
|
||||
- You need a dependable CPU engine with optional CUDA acceleration.
|
||||
|
||||
## Setup
|
||||
|
||||
1. Install the runtime:
|
||||
|
||||
```bash
|
||||
pip install sherpa-onnx
|
||||
```
|
||||
|
||||
2. Download a TTS model from the
|
||||
[sherpa-onnx releases](https://github.com/k2-fsa/sherpa-onnx/releases)
|
||||
and unpack it somewhere permanent.
|
||||
|
||||
3. Point VoiceStudio at the model directory and restart:
|
||||
|
||||
```bash
|
||||
export OMNIVOICE_SHERPA_MODEL=/path/to/model-dir
|
||||
```
|
||||
|
||||
4. Select the engine via **Model Catalogue → Engines** or
|
||||
`OMNIVOICE_TTS_BACKEND=sherpa-onnx`.
|
||||
|
||||
The directory must contain `model.onnx` and `tokens.txt`. Sherpa-ONNX ships
|
||||
no bundled default model, so the engine reports unavailable — with the
|
||||
reason — until `OMNIVOICE_SHERPA_MODEL` points at a valid directory. (Before
|
||||
this gate, selecting the engine unconfigured produced a failure mislabeled
|
||||
as out-of-memory —
|
||||
[#919](https://github.com/debpalash/VoiceStudio/issues/919).)
|
||||
|
||||
## Configuration
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `OMNIVOICE_SHERPA_MODEL` | (unset) | Directory containing `model.onnx` + `tokens.txt` |
|
||||
|
||||
## Behaviour notes
|
||||
|
||||
- Output defaults to 22.05 kHz (the VITS default); once a model is loaded,
|
||||
its own sample rate is used.
|
||||
- CPU is the universal baseline; the CUDA onnxruntime provider is available
|
||||
on Linux/Windows installs.
|
||||
- **No cloning**: voices come from the model itself. Multi-speaker VITS
|
||||
models select a voice by numeric speaker id; speed is supported.
|
||||
- Languages depend entirely on the model you download.
|
||||
|
||||
## Known limits
|
||||
|
||||
- One model at a time — switching models means changing
|
||||
`OMNIVOICE_SHERPA_MODEL` and restarting.
|
||||
- No voice design, no reference-audio cloning, no emotion controls
|
||||
(see [expressive-speech.md](../expressive-speech.md)).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- "OMNIVOICE_SHERPA_MODEL not set" / "No model.onnx in …": follow Setup
|
||||
above — the variable must point at the *unpacked* model directory, not
|
||||
the archive.
|
||||
- Other issues: [install/troubleshooting.md](../install/troubleshooting.md).
|
||||
|
||||
See also: [benchmarks.md](../benchmarks.md),
|
||||
[languages.md](../languages.md),
|
||||
[disk usage](disk-usage.md).
|
||||
@@ -0,0 +1,76 @@
|
||||
# VoiceStudio — Supertonic-3 Engine
|
||||
|
||||
Supertonic-3 (Supertone Inc.) is a ~99M-parameter ONNX TTS engine covering
|
||||
31 languages with 7 preset voices at native 44.1 kHz. It is CPU-only by
|
||||
design — pure ONNX Runtime on the CPU execution provider, with no CUDA or
|
||||
MPS path in the upstream SDK — and runs in its own sidecar process so
|
||||
crashes and cold init never block the rest of VoiceStudio.
|
||||
|
||||
## When to pick it
|
||||
|
||||
- Broad language coverage on machines with no usable GPU.
|
||||
- Preset-voice narration at a higher sample rate than the default engine.
|
||||
|
||||
## Setup
|
||||
|
||||
1. Install the optional dependency into VoiceStudio's environment:
|
||||
|
||||
```bash
|
||||
uv sync --extra supertonic
|
||||
```
|
||||
|
||||
(Or enable it from **Model Catalogue → Engines**, which installs the
|
||||
pinned `supertonic` wheel for you.)
|
||||
|
||||
2. **Accept the license in-app.** First use is gated behind an explicit
|
||||
acceptance dialog: the inference SDK is MIT, but the model weights are
|
||||
**OpenRAIL-M**, which carries use restrictions. The engine stays
|
||||
unavailable until you review and accept in **Model Catalogue → Engines →
|
||||
Supertonic-3**.
|
||||
|
||||
3. Select the engine via **Model Catalogue → Engines** or
|
||||
`OMNIVOICE_TTS_BACKEND=supertonic3`.
|
||||
|
||||
The first synthesis cold-downloads ~400 MB of model weights, pinned to an
|
||||
exact HuggingFace revision SHA so the bytes match what the SDK was validated
|
||||
against. See [downloading-models.md](../downloading-models.md).
|
||||
|
||||
## Voices
|
||||
|
||||
Seven preset voices are surfaced: `M1` (default), `M3`, `M4`, `M5`, `F3`,
|
||||
`F4`, `F5`. The SDK itself accepts the full `M1`–`M5` / `F1`–`F5` set if a
|
||||
caller passes one explicitly; unknown ids fall back to the default with a
|
||||
log line.
|
||||
|
||||
## Behaviour notes
|
||||
|
||||
- Output is 44.1 kHz mono.
|
||||
- Runs as a long-lived sidecar in the parent Python environment (its
|
||||
dependencies — onnxruntime, numpy, soundfile — already match
|
||||
VoiceStudio's pins); subsequent calls reuse the warm ONNX session.
|
||||
- `speed` is clamped to 0.7–2.0; quality steps clamp to 5–12.
|
||||
- Language is an ISO 639-1 code; Auto engages the SDK's multilingual
|
||||
fallback.
|
||||
|
||||
## Known limits
|
||||
|
||||
- **No cloning and no voice design** — preset voices only. Dub/batch jobs
|
||||
that need cloning won't select it.
|
||||
- CPU-only: hardware acceleration is a property of the upstream SDK, not a
|
||||
VoiceStudio limitation.
|
||||
- OpenRAIL-M weights are not covered by VoiceStudio's blanket
|
||||
commercial-use statement — review the model license terms in the
|
||||
acceptance dialog.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- "supertonic package not installed": run the `uv sync` above or enable
|
||||
from the Model Catalogue.
|
||||
- "license not accepted": open **Model Catalogue → Engines → Supertonic-3**
|
||||
and accept.
|
||||
- Other issues: [install/troubleshooting.md](../install/troubleshooting.md).
|
||||
|
||||
See also: [benchmarks.md](../benchmarks.md),
|
||||
[languages.md](../languages.md),
|
||||
[expressive-speech.md](../expressive-speech.md),
|
||||
[disk usage](disk-usage.md).
|
||||
@@ -0,0 +1,76 @@
|
||||
# VoiceStudio — VoxCPM2 Engine
|
||||
|
||||
VoxCPM2 (OpenBMB) is the studio-quality option: native 48 kHz output,
|
||||
zero-shot voice cloning, and — uniquely among VoiceStudio's engines —
|
||||
**voice design**: creating a synthetic voice from a text description
|
||||
("young female, warm tone, British accent") with no reference audio at all.
|
||||
|
||||
## When to pick it
|
||||
|
||||
- You want voice design without a reference clip.
|
||||
- You want the highest output sample rate (48 kHz vs OmniVoice's 24 kHz).
|
||||
- Your language is among its 30 supported languages: Arabic, Burmese,
|
||||
Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew,
|
||||
Hindi, Indonesian, Italian, Japanese, Khmer, Korean, Lao, Malay,
|
||||
Norwegian, Polish, Portuguese, Russian, Spanish, Swahili, Swedish,
|
||||
Tagalog, Thai, Turkish, Vietnamese.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python ≥ 3.10, PyTorch ≥ 2.5.
|
||||
- CUDA ≥ 12 recommended for full speed; MPS (Apple Silicon) and CPU also
|
||||
work.
|
||||
|
||||
## Setup
|
||||
|
||||
Install the package into VoiceStudio's Python environment:
|
||||
|
||||
```bash
|
||||
pip install "voxcpm>=2.0.3"
|
||||
```
|
||||
|
||||
That is a version **floor**, not a pin — an older install still works, but
|
||||
the engine logs an upgrade hint at load time. Then select the engine via
|
||||
**Model Catalogue → Engines** or `OMNIVOICE_TTS_BACKEND=voxcpm2`.
|
||||
|
||||
## Model selection
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `OMNIVOICE_VOXCPM_MODEL` | `openbmb/VoxCPM2` | HuggingFace checkpoint to load |
|
||||
|
||||
The first use downloads a multi-GB checkpoint from HuggingFace. A download
|
||||
interrupted near the end used to abort the load outright
|
||||
([#1224](https://github.com/debpalash/VoiceStudio/issues/1224)); the load is
|
||||
now retried once with a fresh client. See
|
||||
[downloading-models.md](../downloading-models.md).
|
||||
|
||||
## Behaviour notes
|
||||
|
||||
- **Voice design:** provide a description and no reference audio.
|
||||
- **Cloning:** the reference clip is prepared before use (edge-silence trim
|
||||
and length cap) so dead air in a raw clip doesn't condition the output; on
|
||||
any prep problem the raw clip is used as-is.
|
||||
- **Style instructions** are passed as an inline prefix to the text.
|
||||
- VoxCPM2 emits mastered, studio-grade audio, so VoiceStudio **skips its
|
||||
shared mastering chain** (which is tuned for 24 kHz engines) — only benign
|
||||
loudness normalization applies.
|
||||
- A trailing-silence guard trims long near-silent tails from generations,
|
||||
keeping a short natural tail.
|
||||
|
||||
## Known limits
|
||||
|
||||
- Slower than the lightweight CPU engines — see
|
||||
[benchmarks.md](../benchmarks.md) and [performance.md](../performance.md).
|
||||
- Language coverage is 30 languages; for anything else use the default
|
||||
[OmniVoice](omnivoice.md) engine ([languages.md](../languages.md)).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Engine shows unavailable: the `voxcpm` package isn't installed — run the
|
||||
`pip install` above and restart VoiceStudio.
|
||||
- Repeated first-download failures: check connectivity/HF access, then see
|
||||
[install/troubleshooting.md](../install/troubleshooting.md).
|
||||
|
||||
See also: [expressive-speech.md](../expressive-speech.md),
|
||||
[disk usage](disk-usage.md).
|
||||
@@ -0,0 +1,80 @@
|
||||
# VoiceStudio — WhisperX Engine
|
||||
|
||||
WhisperX is the default ASR engine on CUDA and plain-CPU hosts: faster-whisper
|
||||
(CTranslate2) transcription plus a **wav2vec2 forced-alignment** pass that
|
||||
snaps word boundaries to ±10–30 ms (Whisper's own timestamps are ±100–300 ms).
|
||||
That word timing is what dubbing lip-sync depends on, which is why auto-detect
|
||||
prefers it wherever CTranslate2 can use the GPU.
|
||||
|
||||
## Selecting it
|
||||
|
||||
- **Model Catalogue → Engines**, ASR tab → **Use** on the WhisperX row, or
|
||||
- pin it with `OMNIVOICE_ASR_BACKEND=whisperx` (the env var always wins over
|
||||
the Settings pick; with neither set, auto-detect chooses per-hardware).
|
||||
|
||||
## Best at
|
||||
|
||||
- **Dubbing** — the forced alignment is the accuracy tier lip-sync needs.
|
||||
- **Batch transcription** with word-level subtitles.
|
||||
- Multi-speaker work: it pairs with pyannote speaker diarization — see
|
||||
[diarization](../features/diarization.md).
|
||||
|
||||
## Platform support
|
||||
|
||||
| Host | What happens |
|
||||
| --- | --- |
|
||||
| NVIDIA CUDA | GPU, float16 (degrades automatically, see below) |
|
||||
| CPU (any OS) | int8 — works, but slow for large-v3 |
|
||||
| Apple Silicon | CPU only — CTranslate2 has no Metal build, so auto-detect prefers [mlx-whisper](mlx-whisper.md) there ([#1127](https://github.com/debpalash/VoiceStudio/issues/1127)) |
|
||||
| AMD ROCm | CPU only — CTranslate2 has no HIP build, so auto-detect prefers [pytorch-whisper](pytorch-whisper.md) there ([#1529](https://github.com/debpalash/VoiceStudio/issues/1529)) |
|
||||
|
||||
## Model selection
|
||||
|
||||
- `ASR_MODEL_WHISPERX` — default `large-v3`. Accepts the usual size aliases
|
||||
(`tiny` … `large-v3`, `distil-large-v3`) or a full HF repo id. Weights
|
||||
download on first load — see [downloading-models](../downloading-models.md).
|
||||
- `OMNIVOICE_ALIGN_DEVICE` — force the wav2vec2 aligner's device. Aligners
|
||||
exist for ~20 major languages; other languages keep Whisper's native word
|
||||
timestamps instead of failing.
|
||||
|
||||
## VRAM preflight and degradation
|
||||
|
||||
Loading fp16 large-v3 onto a nearly-full 8 GB card dies as a *native* CUDA
|
||||
abort — no Python exception, the whole backend goes down
|
||||
([#723](https://github.com/debpalash/VoiceStudio/issues/723)). So before every
|
||||
load the engine checks free VRAM against per-compute-type budgets
|
||||
(float16 5.0 GB, int8_float16 3.5 GB, int8 3.0 GB, scaled down for smaller
|
||||
models) and degrades the compute type — or falls to CPU int8 — instead of
|
||||
starting a load that would kill the process. Disable with
|
||||
`OMNIVOICE_ASR_VRAM_PREFLIGHT=0`.
|
||||
|
||||
Two more fallback chains run at load time:
|
||||
|
||||
- GPUs without efficient fp16 (older Maxwell/Pascal, GTX 16xx) raise a
|
||||
compute-type error — the engine retries int8_float16, then int8
|
||||
([#551](https://github.com/debpalash/VoiceStudio/issues/551)).
|
||||
- A genuine CUDA OOM retries on CPU int8, so dubbing still completes
|
||||
(slower, same model and accuracy).
|
||||
|
||||
## Quirks
|
||||
|
||||
- **cuDNN 8 required on CUDA.** CTranslate2 links cuDNN 8; if it's missing the
|
||||
process fast-fails with no traceback, so the engine is reported unavailable
|
||||
up front and selection falls through to pytorch-whisper, which uses torch's
|
||||
own cuDNN 9 ([#1371](https://github.com/debpalash/VoiceStudio/issues/1371)).
|
||||
- On some hardened Linux kernels CTranslate2's native library is rejected with
|
||||
"cannot enable executable stack" — reported as unavailable, not a crash
|
||||
([#692](https://github.com/debpalash/VoiceStudio/issues/692)).
|
||||
- A partially-installed environment (interrupted sync, antivirus quarantine)
|
||||
can break WhisperX's deep import chain (whisperx → pyannote →
|
||||
lightning_fabric). The engine is then reported unavailable with a repair
|
||||
hint — reinstall, or `uv sync --reinstall` on a source checkout
|
||||
([#1185](https://github.com/debpalash/VoiceStudio/issues/1185)).
|
||||
- Audio is decoded through VoiceStudio's validated ffmpeg, not a bare `ffmpeg`
|
||||
PATH lookup ([#479](https://github.com/debpalash/VoiceStudio/issues/479)).
|
||||
- Transcribes are time-bounded: each dub chunk by
|
||||
`OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S` (default 120 s), whole files by
|
||||
`OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S` (default 300 s). Raise them for very
|
||||
long files on slow hardware.
|
||||
|
||||
Speed comparisons across engines live in [performance](../performance.md).
|
||||
Reference in New Issue
Block a user