Palash DebnathandClaude Fable 5
030d5ea01f
* docs(engines): a guide for every engine + index; fix two engine-metadata bugs 21 new pages under docs/engines/ (10 TTS, 10 ASR, index README) — every registered engine now has one: what it's for, platform support, model env vars, quirks with issue refs. Linked from both READMEs' engine sections. Code fixes found while verifying facts against the registries: - KittenTTS docstring claimed default voice 'Jasper'; the code default is expr-voice-2-f - the isolated-ASR sidecar read only ASR_MODEL_FW while the download preflight read ASR_MODEL_FASTER — set one and the other quietly used a different model; both now resolve ASR_MODEL_FW-override → ASR_MODEL_FASTER - moonshine's install hint named 'useful-moonshine', a package the backend never imports; now moonshine-onnx / moonshine-voice Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(changelog): entries for the engine guides + sidecar model fix (#1556) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(engines): second-harvest fixes — TLS guidance, matrix/code alignment, CN counts - README matrix aligned to gpu_compat (the code is the source of truth): CosyVoice macOS is CPU not MPS, IndexTTS and GGUF gain their real CUDA/CPU/MPS cells - gpt-sovits guide: prefer https/tunnel for non-loopback servers, plaintext warning; first-use download guidance on both OmniVoice pages - preflight empty-env fallback matches the sidecar (ASR_MODEL_FASTER='' no longer resolves a different repo) - nano installs via uv pip; kitten log level wording; index links install guides incl. the Gatekeeper step; README_CN engine counts 16/11 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(readme-cn): the all-engines-local claim now excludes the remote client Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2.6 KiB
2.6 KiB
VoiceStudio — MLX Whisper Engine
MLX Whisper runs Whisper on the Apple Silicon GPU via MLX. It exists because CTranslate2 (whisperx / faster-whisper) has no Metal build — on a Mac those engines transcribe on the CPU no matter what GPU is present. Measured on an M2 with whisper-large-v3, one 30 s dub chunk: 90.4 s on WhisperX (CPU) vs 20.5 s on MLX (GPU) — which is why auto-detect picks MLX Whisper on every Apple Silicon machine (#1127).
Selecting it
- Nothing to do on Apple Silicon — auto-detect prefers it there.
- Or explicitly: Model Catalogue → Engines, ASR tab → Use, or
OMNIVOICE_ASR_BACKEND=mlx-whisper.
Best at
- Dubbing on a Mac — it layers the same wav2vec2 forced alignment WhisperX uses on top of the GPU transcription, so word timing (±10–30 ms) and therefore lip-sync accuracy are unchanged. Same model, same alignment, ~4x the speed.
- Dictation/capture — the capture path automatically swaps in
mlx-community/whisper-large-v3-turbo(~5x faster than large-v3) unless a sherpa dictation model or parakeet-mlx is preferred.
Platform support
Apple Silicon only. A shared platform gate refuses Linux, Windows, and
Intel Macs before any package import, so a stray mlx-whisper wheel on the
wrong platform never reports itself available
(#390). All other
platforms use the CUDA/CPU engines instead.
Model selection
ASR_MODEL— defaultmlx-community/whisper-large-v3-mlx. Any MLX-format Whisper repo works. Weights download on first load — see downloading-models.OMNIVOICE_ALIGN_DEVICE— force the wav2vec2 aligner's device. The aligner runs on MPS when it can and falls back to CPU; languages without a bundled aligner (~20 major languages have one) keep Whisper's native word timestamps.
Quirks
- Audio is decoded through VoiceStudio's validated ffmpeg rather than the
bare
ffmpegPATH lookup mlx-whisper would do on its own — a clean from-source install with no system ffmpeg works fine (#479). - The model is warmed into unified memory in the background, so the first transcribe after startup doesn't pay the load cost.
- In a packaged app, a native MLX library that fails to load is reported as "unavailable" (with fallback to another engine) rather than crashing the engine list.
Speed comparisons across engines live in performance.