VoiceStudio/README.md

OmniVoice Studio

The open-source ElevenLabs alternative.

Real-time dictation, zero-shot voice cloning, and cinematic video dubbing — all on your desktop.
Open-source, no API keys, fully local. 646 languages.

Quickstart · Features · Why OmniVoice Studio? · TTS Engines · ASR Engines · Contributing · Discord · 简体中文

_{macOS: first launch needs a one-time approval — right-click → Open (or System Settings → Privacy & Security → "Open Anyway" on macOS 15). No Terminal needed. Why?}

Warning

OmniVoice Studio is in active beta. Things may break between releases. For the latest features and fixes, clone the repo and run from source rather than using pre-built installers. Bug reports and PRs are very welcome — open an issue or join Discord.

_{Get setup help · Share your dubs · Vote on the roadmap · Early access to new engines}

Features

🎙️ Voice Cloning 3-second clip → mirror any voice. 646 languages, zero-shot.	🎨 Voice Design Gender, age, accent, pitch, speed, emotion, dialect — dial it in.	🎬 Video Dubbing YouTube URL or file → transcribe → translate → re-voice → MP4.
⌨️ Dictation Widget ⌘+⇧+Space from any app. Transcribes, auto-pastes, disappears.	🔊 Vocal Isolation Demucs-powered. Splits speech from music, keeps the background.	👥 Speaker Diarization Pyannote + WhisperX. Auto-identifies who said what.
📦 Batch Queue Drop 50 videos, walk away. Progress bars per job.	🤖 MCP Server Use OmniVoice from Claude, Cursor, or any MCP client.	🛡️ AI Watermark AudioSeal (Meta). Invisible, survives compression.
🔐 100% Local No keys, no cloud, no accounts. Your machine only.	⚡ GPU Auto-Detect CUDA · MPS · ROCm · CPU. ≤8 GB? Auto-offloads.	🧩 Extensible Subclass TTSBackend, add any engine in ~50 lines.

---

Quickstart

Per-OS install guides — pick yours and follow it end-to-end:

• macOS — docs/install/macos.md
• Windows — docs/install/windows.md
• Linux — docs/install/linux.md
• Docker — docs/install/docker.md · Docker Hub: palashdeb/omnivoice-studio

Stuck? Run the built-in self-check first — Settings → About → "Run self-check" in the app, or uv run python backend/main.py --diagnose from a checkout (--deep also test-loads the active engine). Then see docs/install/troubleshooting.md for the top 10 install errors. The in-app error UI deeplinks to those entries when something breaks at runtime, and Settings → About → "Save diagnostic bundle" packages scrubbed logs + the self-check report for bug reports.

For Hugging Face token setup, see docs/setup/huggingface-token.md. For diarization-specific gating, see docs/features/diarization.md. For download speed, the ⚡ fast-download (Xet) status, and restricted-network / mirror options, see docs/downloading-models.md.

Screenshots

Voice Clone Drop a 3-second clip → mirror any voice. 646 languages, zero-shot.	Voice Design Build new voices from scratch — gender, age, accent, pitch, style.
Video Dubbing Upload or paste a YouTube URL. Transcribe, translate, re-voice, export.	Voice Gallery Search YouTube, browse categories, download clips, build your library.
Settings → Models 15 models. One-click install. Auto-detects your platform (CUDA / MPS / CPU).	Projects Dub projects, voice profiles, generation history, exports — all searchable.
Settings → Logs Live backend, frontend, and Tauri runtime logs. Filter, refresh, clear.

---

Why OmniVoice Studio?

ElevenLabs charges $5–$330/mo and processes your audio on their servers. OmniVoice Studio runs on your hardware, with no usage limits.

	ElevenLabs	OmniVoice Studio
Pricing	$5–$330/mo, per-character billing	Free & open-source (AGPL-3.0) · Commercial license for proprietary use
Voice Cloning	✅ 3s clip	✅ 3s clip, zero-shot
Voice Design	✅ Gender, age	✅ Gender, age, accent, pitch, style, dialect
Languages	32	646
Video Dubbing	✅ Cloud-only	✅ Fully local
Data Privacy	Audio sent to cloud	Nothing leaves your machine
API Keys	Required	Not needed
GPU Support	N/A (cloud)	CUDA · Apple Silicon · ROCm · CPU
Desktop App	❌	✅ macOS · Windows · Linux
Customizable	❌ Closed	✅ Fork it, extend it, ship it

OmniVoice Studio gives you professional-grade AI tools without the subscription or the cloud.

Convinced? Come build with us.

---

System Requirements

	Minimum	Recommended
OS	Windows 10, macOS 12+, Ubuntu 20.04+	Any modern 64-bit OS
RAM	8 GB	16 GB+
VRAM (GPU)	4 GB (auto-offloads TTS to CPU)	8 GB+ (NVIDIA RTX 3060+)
Disk	10 GB free (models + cache)	20 GB+ SSD
Python	3.10+ (managed by uv)	3.11–3.12
GPU	Optional — CPU works	NVIDIA CUDA · Apple Silicon MPS · AMD ROCm

Tip

On GPUs with ≤8 GB VRAM, OmniVoice automatically offloads TTS to CPU during transcription — no config needed. A dedicated GPU is not required; the entire pipeline runs on CPU (just slower).

TTS Engines

OmniVoice ships a multi-engine TTS backend. The default engine (OmniVoice) is always available; additional engines are opt-in and auto-detected. Switch engines in Settings → TTS Engine or via the OMNIVOICE_TTS_BACKEND env var.

Engine	Languages	Clone	Instruct	Linux	macOS ARM	Windows	License
OmniVoice (default)	600+	✅	✅	✅ CUDA/CPU	✅ MPS	✅ CUDA/CPU	Built-in
CosyVoice 3	9 + 18 dialects	✅	✅	✅ CUDA/CPU	✅ MPS	✅ CUDA/CPU	Apache-2.0
MLX-Audio (Kokoro, Qwen3-TTS, CSM, Dia, …)	Multi	Varies	Varies	❌	✅ Native	❌	Varies
VoxCPM2	30	✅	✅	✅ CUDA/CPU	✅ MPS	✅ CUDA/CPU	Apache-2.0
MOSS-TTS-Nano	20	✅	❌	✅ CUDA/CPU	✅ CPU	✅ CUDA/CPU	Apache-2.0
KittenTTS	English	❌	❌	✅ CPU	✅ CPU	✅ CPU	MIT

CUDA = GPU-accelerated · MPS = Apple Silicon Metal · CPU = runs everywhere, slower for large models · KittenTTS and MOSS-TTS-Nano run realtime on CPU · MLX-Audio is Apple Silicon only.

ASR Engines

OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictation, video dubbing, and subtitle generation — all fully local. WhisperX is the cross-platform default; the rest are opt-in and auto-detected. Switch in Settings → ASR Engine or via the OMNIVOICE_ASR_BACKEND env var.

Engine	OMNIVOICE_ASR_BACKEND	Languages	Best for
WhisperX (default)	whisperx	~100	Dubbing & subtitles — word-level timing via wav2vec2 forced alignment
Faster-Whisper	faster-whisper	~100	Fast transcription on Linux / macOS / Windows (CTranslate2)
MLX Whisper	mlx-whisper	~100	Native Apple Silicon speed (Apple MLX / Metal)
PyTorch Whisper	pytorch-whisper	~100	CUDA / CPU fallback via 🤗 Transformers
Parakeet TDT	nemo-parakeet	English + 25 EU	SOTA English accuracy, auto language detection (NVIDIA NeMo, GPU only)
Moonshine	moonshine	English	Edge / low-latency, ONNX
FunASR	funasr	50+	All-in-one multilingual — built-in VAD + inline speaker diarization (SenseVoice)

Whisper-family engines cover ~100 languages; FunASR / SenseVoice adds an all-in-one multilingual path with built-in voice-activity detection and inline speaker diarization. Every engine runs on-device — no API keys, no cloud.

---

Architecture

┌─────────────────────────────────────────────────┐
│                  Frontend (React)                │
│  DubTab · VoicePreview · BatchQueue · Gallery    │
├─────────────────────────────────────────────────┤
│                Backend (FastAPI)                  │
│  97 API endpoints · SSE streaming · SQLite       │
├──────────┬──────────┬──────────┬────────────────┤
│ WhisperX │  Demucs  │OmniVoice │   Pyannote     │
│   ASR    │  Source  │   TTS    │  Diarization   │
│          │  Sep.    │          │                │
└──────────┴──────────┴──────────┴────────────────┘
        CUDA / MPS / ROCm / CPU (auto-detected)

---

Roadmap

✅ Shipped

Category	Features
Dubbing	Full pipeline (transcribe→translate→synthesize→mux), scene-aware splitting, lip-sync scoring, streaming TTS
Voice	Zero-shot cloning, voice design, A/B comparison, voice preview widget, gallery with favorites/tags
Audio	Demucs vocal isolation, per-segment gain, selective track export, stem/SRT/VTT/MP3 export
Multi-Lang	Multi-language batch picker, batch dubbing queue with sequential GPU execution
Diarization	Pyannote ML diarization, auto speaker clone extraction, per-speaker voice assignment
Infra	Docker deployment, CUDA/MPS/ROCm auto-detect, cuDNN 8 compat, VRAM-aware model offloading
AI Provenance	AudioSeal invisible watermarking (SynthID-like), video logo overlay, watermark detection API
UX	Undo/redo, keyboard shortcuts, drag-and-drop, session persistence, glassmorphism design system
Real-time Events	WebSocket event bus — instant sidebar refresh on data mutations, exponential backoff reconnect
State Management	Zustand store migration — uiSlice, pillSlice, dubSlice, generateSlice, prefsSlice, glossarySlice
Desktop	Cross-platform Tauri installers (macOS DMG, Windows MSI, Linux deb/AppImage), auto-update infrastructure
Windows Hardening	Cross-platform log paths, Triton workaround, HF symlink bypass, 300s health check timeout
Dictation	Global system-wide hotkey (⌘+⇧+Space), frameless floating widget, streaming ASR via WebSocket, auto-paste
Batch Pipeline	Full batch TTS: extract → transcribe → translate → generate → mix → export, with live progress tracking

🔜 Up Next

• 🎬 Lip-sync v2 — visual speech timing with wav2lip
• 📖 Audiobook Editor — chapter-aware long-form narration
• 🌐 Hosted Demo — try OmniVoice without installing anything
• 🔌 Plugin Marketplace — community-contributed TTS engines and effects

---

Community

Channel	What happens there
#showcase	Members share their dubs, clones, and voice designs
#help	Setup issues, GPU troubleshooting, model questions
#feature-requests	Vote on what gets built next
#dev	Architecture discussions, PR reviews, engine integrations
#announcements	Release notes, breaking changes, early access

→ Join the Discord — we respond to setup questions within hours, not days.

---

Contributing

We welcome contributions of all kinds — bug fixes, new TTS engine adapters, UI improvements, docs, and translations.

• 📖 Read the Contributing Guide for setup, code style, and PR workflow
• 🐛 Browse good first issues
• 💬 Join our Discord to discuss ideas or ask for help

---

FAQ

Is this really as good as ElevenLabs?

For voice cloning and dubbing, yes — OmniVoice uses a state-of-the-art diffusion TTS model with 646 languages (ElevenLabs supports 32). Quality is comparable for most use cases. Where ElevenLabs wins is in their polished cloud API and pre-made voice library. OmniVoice wins on privacy, cost, language coverage, and customizability.

Does it work on Apple Silicon (M1/M2/M3/M4)?

Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are available for faster transcription on Apple hardware.

How much VRAM do I need?

4 GB minimum. With ≤8 GB, the TTS model is automatically offloaded to CPU during transcription. With 8+ GB, everything runs on GPU simultaneously. No GPU at all? CPU mode works — just slower (~3× for TTS).

Can I use this commercially?

Yes — commercial use is free. OmniVoice Studio is free and open-source under the GNU AGPL-3.0. So personal, educational, research, and commercial / business use are all free: run it, sell the audio you make with it, dub your own or a client's videos, deploy it across your team. Because AGPL is a network copyleft license, if you modify OmniVoice Studio and make that modified version available to others over a network, you must offer those users the source of your modified version under the same AGPL terms. Want to embed OmniVoice in a closed-source or proprietary product without those obligations? A commercial license is available — see License.

What languages are supported?

646 languages for TTS via the OmniVoice model. Transcription (WhisperX) supports 99 languages. Translation coverage depends on the target language pair.

Can I add my own TTS engine?

Yes. OmniVoice uses a built-in backend registry. To add an engine in ~50 lines, subclass TTSBackend in backend/services/tts_backend.py and add it to the _REGISTRY dictionary at the bottom. Six engines are built in: OmniVoice, CosyVoice, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, and KittenTTS. See the TTS Engines section for details.

---

License

OmniVoice Studio is free and open-source software under the GNU Affero General Public License v3.0 (AGPL-3.0).

Free for any use — including commercial and internal business use. Run it, sell the audio you produce with it, dub your own or clients' videos, roll it out across your team — all free, no license needed. As a network copyleft license, AGPL adds one obligation: if you modify OmniVoice Studio and offer that modified version to others over a network, you must make the complete corresponding source of your modified version available to them under the same AGPL-3.0 terms.

A commercial license is available for organizations that want to embed OmniVoice Studio in a closed-source or proprietary product or service without the AGPL-3.0 copyleft obligations. Pricing tiers coming soon. Inquiries: OmniVoice@palash.dev.

The bundled omnivoice/ TTS model by Han Zhu remains Apache-2.0 upstream. See LICENSE for the full, binding terms.

---

Acknowledgments

OmniVoice Studio is built on the shoulders of exceptional open-source work:

Project	Role
OmniVoice (k2-fsa)	Zero-shot diffusion TTS engine — the core voice synthesis model
WhisperX	Word-level speech recognition and alignment
Demucs (Meta)	Music source separation for vocal isolation
Pyannote	Speaker diarization — who said what
CTranslate2	Optimized Transformer inference on CPU and GPU
AudioSeal (Meta)	Invisible neural audio watermarking for AI provenance
Tauri	Native desktop app framework

---

If you read this far, you're our kind of person.
⭐ Star this repo so others can find it too.
💬 Join the Discord to share what you build.