Carlos D. Escobar-Valbuena
fa9c7d43ca
* fix(mcp): drop unsupported FastMCP kwargs (mcp SDK >= 1.10)
The MCP server passes `version=` and `description=` to FastMCP(), but
neither kwarg exists on mcp >= 1.10 — the protocol version is now
managed internally and `description` was renamed to `instructions`.
Symptom on a fresh install (uv sync && pip install 'mcp[cli]'):
TypeError: FastMCP.__init__() got an unexpected keyword argument 'version'
Tested locally end-to-end:
- create_mcp_server() now constructs cleanly
- All 5 tools register and are listable via FastMCP.list_tools()
- generate_speech round-trip returns base64 WAV; ~24s server-side
for 4.2s of audio at steps=16 on Apple Silicon MPS
- pytest backend/ -x -q: 45 passed
* feat: bundle Claude Code agent skill at .claude/skills/omnivoice/
CLAUDE.md already invites contributions at .claude/skills/:
"No project skills found. Add skills to any of: .claude/skills/,
.agents/skills/, .cursor/skills/, .github/skills/, or .codex/skills/
with a SKILL.md index file."
But the existing .gitignore blanket-ignored .claude/ (line 41), making
the invited path un-trackable. This commit narrows the ignore so ad-hoc
Claude state stays out while deliberate skill bundles are tracked:
-.claude/
+.claude/*
+!.claude/skills/
+!.claude/skills/**
Once merged, any compatible agent client running
`npx skills add debpalash/OmniVoice-Studio` gets immediate context on:
- What the MCP server exposes (5 tools + 2 resources)
- When to pick OmniVoice vs other engines
- How to wire the stdio MCP server into a client config
- Backend lifecycle: start / health / stop scripts
- Common failure modes + fixes (port collision, model download stall,
missing HF_TOKEN, MPS fallback, voice-profile-not-found, etc.)
Conforms to Anthropic skill-creator conventions: frontmatter
description under 1024-char limit, body under 500 lines, references/
for detail, scripts/ for deterministic ops, no README/CHANGELOG
inside the skill, validates clean against quick_validate.py.
Verified locally that `npx skills list` discovers the bundled skill
automatically once cloned. End-to-end tested through MCP:
- generate_speech (English, demo voice, steps=16) -> 4.2 s WAV
- generate_speech (voice design via instruct only, steps=8) -> 6.3 s WAV
- generate_speech (Spanish, demo voice, steps=16) -> 2.8 s WAV
Depends on #112 (FastMCP API fix). Without it, every MCP tool call
fails with TypeError at server construction.
* feat(skill): add voice-clone end-to-end recipe + record-reference.sh helper
Two additions to the bundled skill, closing the gap where agents had no
procedural knowledge for creating a voice profile (the previous SKILL.md
said "use the UI or POST /profiles" but didn't include the recording +
trimming + verification workflow).
1. scripts/record-reference.sh — macOS-only helper that records a clean
reference clip with **audible** countdown + start/stop cues via
`say` + /System/Library/Sounds/Ping.aiff. Solves the buffering bug
where text-mode "speak now" prompts arrive after recording starts.
Captures a longer raw window then trims to ~10 sec of speech via
silenceremove + atrim. Plays back for verification. Prints the
next-step `curl` command for POST /profiles.
2. SKILL.md "Voice clone — end-to-end recipe" section (replaces the
stub one-liner). Covers:
- Path A: the bundled helper (one command, audible cues)
- Path B: manual ffmpeg flow if the helper doesn't fit
- POST /profiles multipart/form-data fields (required: name +
ref_audio; optional: ref_text, language, instruct, seed, personality)
- Reference clip quality factors that materially affect output
(single speaker, natural prosody, 3-10 sec sweet spot, ref_text
alignment, language correctness, loudness ≥ -15 dB peak)
Tested locally: recorded a 10-sec Spanish reference + 3-sec English
reference, created two profiles via the helper + curl flow, generated
14.1 sec of Spanish + 10.2 sec of English audio in the user's cloned
voice. Round-trip works end-to-end at steps=16 on Apple Silicon MPS.
Frontmatter description unchanged (860 chars, under the 1024 limit).
Body grew from ~120 to 169 lines (still well under the 500-line skill
ceiling).
* fix(skill): address P20 cross-review findings on PR #113
Adversarial multi-agent review (code + comment + silent-failure analyzers
on parallel reviewers) surfaced one blocker, one critical silent-failure
class, two medium-severity bugs, and two minor doc inaccuracies. All
addressed in this commit.
Blocker (cited 3x by both code-reviewer and comment-analyzer):
- SKILL.md linked references/engines-comparison.md three times (lines 44,
153, 160) but the file was never copied into the upstream skill tree.
+ Added the file (engine decision tree across OmniVoice / kokoro /
Voicebox / Edge TTS / ElevenLabs / cloud APIs).
Critical — record-reference.sh (was 4/10):
- Mic-permission silent failure: macOS denies the mic by sending a silent
stream; ffmpeg exits 0 with a valid silent WAV. The script printed
"✓ raw captured" and produced a degenerate reference clip that would
train a broken voice profile.
+ Parse mean_volume from volumedetect; exit 3 with a diagnostic
pointing the user to System Settings → Privacy → Microphone if
the recording is below -50 dB.
- afplay backgrounded with no exit check; if /System/Library/Sounds/*.aiff
is missing the user gets no audible cue.
+ beep() helper falls back to printf '\a' (terminal bell) when the
system sound file is missing.
- silenceremove silent corruption: silent input → near-empty output WAV,
exit 0.
+ ffprobe duration check after trim; exit 4 if < 2.0 sec.
- trap only covered EXIT; Ctrl-C / SIGTERM mid-recording leaked tmp file.
+ trap '...' EXIT INT TERM HUP.
- macOS guard ran after mktemp + trap.
+ Moved guard to first executable line.
- afplay verification swallowed stderr.
+ Drop 2>/dev/null; surface failure as a warning.
- Documented exit codes in header (0/2/3/4).
Medium — start-backend.sh (was 6/10):
- TOCTOU race: lsof check → uvicorn start could lose the port to another
process; only signal was a 60s health timeout.
+ Added `kill -0 $PID` check inside the probe loop; immediate exit 5
with log tail if uvicorn died.
- lsof check couldn't tell "stale us" from "third party" — same exit 3
for both.
+ ps -o command attribution; the message now tells the user whether
it's a stale uvicorn (suggest stop-backend.sh) or unknown process.
- Documented exit codes (0/2/3/4/5).
Medium — stop-backend.sh (was 7/10):
- No post-SIGKILL verification — script exited 0 even if process still
bound.
+ Added current_pids() helper; re-query after SIGKILL; exit 1 if still
bound, with lsof dump for diagnostics.
- 2>/dev/null || true on kill swallowed EPERM silently.
+ Capture stderr; classify EPERM vs ESRCH; exit 2 on EPERM with
actionable hint (try sudo).
- Documented exit codes (0/1/2).
Minor docs (comment-analyzer):
- SKILL.md line 120 claimed profiles persist as `<id>.wav`. Actual
backend (profiles.py:48-50) preserves uploaded extension.
+ Reworded to `<id>.<ext>` with explanation.
- mcp-setup.md line 68 cited HF cache path as Linux/macOS only.
Windows redirects via backend/core/config.py:38 to
%LOCALAPPDATA%\OmniVoice\hf_cache.
+ Added Windows row + reference to config.py.
Re-validated: all 6 files compile under set -euo pipefail; SKILL.md
frontmatter description stays at 860 chars (under 1024 cap); skill body
under 500 lines.
Diff: 6 files changed, ~+269/-47.
OmniVoice Studio
The open-source ElevenLabs alternative.
Real-time dictation, zero-shot voice cloning, and cinematic video dubbing — all on your desktop.
Open-source, no API keys, fully local. 646 languages.
Quickstart · Features · Why OmniVoice Studio? · TTS Engines · Contributing · Discord
-000?style=for-the-badge&logo=apple&logoColor=white){kind=icon}
-0078D4?style=for-the-badge&logo=windows&logoColor=white){kind=icon}
-FCC624?style=for-the-badge&logo=linux&logoColor=black){kind=icon}
Warning
OmniVoice Studio is in active beta. Things may break between releases. For the latest features and fixes, clone the repo and run from source rather than using pre-built installers. Bug reports and PRs are very welcome — open an issue or join Discord.
Get setup help · Share your dubs · Vote on the roadmap · Early access to new engines
Features
🎙️ Voice Cloning3-second clip → mirror any voice. |
🎨 Voice DesignGender, age, accent, pitch, speed, |
🎬 Video DubbingYouTube URL or file → transcribe → |
⌨️ Dictation Widget
|
🔊 Vocal IsolationDemucs-powered. Splits speech |
👥 Speaker DiarizationPyannote + WhisperX. |
📦 Batch QueueDrop 50 videos, walk away. |
🤖 MCP ServerUse OmniVoice from Claude, |
🛡️ AI WatermarkAudioSeal (Meta). Invisible, |
🔐 100% LocalNo keys, no cloud, no accounts. |
⚡ GPU Auto-DetectCUDA · MPS · ROCm · CPU. |
🧩 ExtensibleSubclass |
Quickstart
Per-OS install guides — pick yours and follow it end-to-end:
- macOS — docs/install/macos.md
- Windows — docs/install/windows.md
- Linux — docs/install/linux.md
- Docker — docs/install/docker.md
Stuck? See docs/install/troubleshooting.md for the top 10 install errors. The in-app error UI deeplinks to those entries when something breaks at runtime.
For Hugging Face token setup, see docs/setup/huggingface-token.md. For diarization-specific gating, see docs/features/diarization.md.
Screenshots
Voice Clone Drop a 3-second clip → mirror any voice. 646 languages, zero-shot. |
Voice Design Build new voices from scratch — gender, age, accent, pitch, style. |
Video Dubbing Upload or paste a YouTube URL. Transcribe, translate, re-voice, export. |
Voice Gallery Search YouTube, browse categories, download clips, build your library. |
Settings → Models 15 models. One-click install. Auto-detects your platform (CUDA / MPS / CPU). |
Projects Dub projects, voice profiles, generation history, exports — all searchable. |
Settings → Logs Live backend, frontend, and Tauri runtime logs. Filter, refresh, clear. |
|
Why OmniVoice Studio?
ElevenLabs charges $5–$330/mo and processes your audio on their servers. OmniVoice Studio runs on your hardware, with no usage limits.
| ElevenLabs | OmniVoice Studio | |
|---|---|---|
| Pricing | $5–$330/mo, per-character billing | Free for personal use · Commercial license for business |
| Voice Cloning | ✅ 3s clip | ✅ 3s clip, zero-shot |
| Voice Design | ✅ Gender, age | ✅ Gender, age, accent, pitch, style, dialect |
| Languages | 32 | 646 |
| Video Dubbing | ✅ Cloud-only | ✅ Fully local |
| Data Privacy | Audio sent to cloud | Nothing leaves your machine |
| API Keys | Required | Not needed |
| GPU Support | N/A (cloud) | CUDA · Apple Silicon · ROCm · CPU |
| Desktop App | ❌ | ✅ macOS · Windows · Linux |
| Customizable | ❌ Closed | ✅ Fork it, extend it, ship it |
OmniVoice Studio gives you professional-grade AI tools without the subscription or the cloud.
Convinced? Come build with us.
System Requirements
| Minimum | Recommended | |
|---|---|---|
| OS | Windows 10, macOS 12+, Ubuntu 20.04+ | Any modern 64-bit OS |
| RAM | 8 GB | 16 GB+ |
| VRAM (GPU) | 4 GB (auto-offloads TTS to CPU) | 8 GB+ (NVIDIA RTX 3060+) |
| Disk | 10 GB free (models + cache) | 20 GB+ SSD |
| Python | 3.10+ (managed by uv) |
3.11–3.12 |
| GPU | Optional — CPU works | NVIDIA CUDA · Apple Silicon MPS · AMD ROCm |
Tip
On GPUs with ≤8 GB VRAM, OmniVoice automatically offloads TTS to CPU during transcription — no config needed. A dedicated GPU is not required; the entire pipeline runs on CPU (just slower).
TTS Engines
OmniVoice ships a multi-engine TTS backend. The default engine (OmniVoice) is always available; additional engines are opt-in and auto-detected. Switch engines in Settings → TTS Engine or via the OMNIVOICE_TTS_BACKEND env var.
| Engine | Languages | Clone | Instruct | Linux | macOS ARM | Windows | License |
|---|---|---|---|---|---|---|---|
| OmniVoice (default) | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Built-in |
| CosyVoice 3 | 9 + 18 dialects | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
| MLX-Audio (Kokoro, Qwen3-TTS, CSM, Dia, …) | Multi | Varies | Varies | ❌ | ✅ Native | ❌ | Varies |
| VoxCPM2 | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
| MOSS-TTS-Nano | 20 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| KittenTTS | English | ❌ | ❌ | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
CUDA = GPU-accelerated · MPS = Apple Silicon Metal · CPU = runs everywhere, slower for large models · KittenTTS and MOSS-TTS-Nano run realtime on CPU · MLX-Audio is Apple Silicon only.
Architecture
┌─────────────────────────────────────────────────┐
│ Frontend (React) │
│ DubTab · VoicePreview · BatchQueue · Gallery │
├─────────────────────────────────────────────────┤
│ Backend (FastAPI) │
│ 97 API endpoints · SSE streaming · SQLite │
├──────────┬──────────┬──────────┬────────────────┤
│ WhisperX │ Demucs │OmniVoice │ Pyannote │
│ ASR │ Source │ TTS │ Diarization │
│ │ Sep. │ │ │
└──────────┴──────────┴──────────┴────────────────┘
CUDA / MPS / ROCm / CPU (auto-detected)
Roadmap
✅ Shipped
| Category | Features |
|---|---|
| Dubbing | Full pipeline (transcribe→translate→synthesize→mux), scene-aware splitting, lip-sync scoring, streaming TTS |
| Voice | Zero-shot cloning, voice design, A/B comparison, voice preview widget, gallery with favorites/tags |
| Audio | Demucs vocal isolation, per-segment gain, selective track export, stem/SRT/VTT/MP3 export |
| Multi-Lang | Multi-language batch picker, batch dubbing queue with sequential GPU execution |
| Diarization | Pyannote ML diarization, auto speaker clone extraction, per-speaker voice assignment |
| Infra | Docker deployment, CUDA/MPS/ROCm auto-detect, cuDNN 8 compat, VRAM-aware model offloading |
| AI Provenance | AudioSeal invisible watermarking (SynthID-like), video logo overlay, watermark detection API |
| UX | Undo/redo, keyboard shortcuts, drag-and-drop, session persistence, glassmorphism design system |
| Real-time Events | WebSocket event bus — instant sidebar refresh on data mutations, exponential backoff reconnect |
| State Management | Zustand store migration — uiSlice, pillSlice, dubSlice, generateSlice, prefsSlice, glossarySlice |
| Desktop | Cross-platform Tauri installers (macOS DMG, Windows MSI, Linux deb/AppImage), auto-update infrastructure |
| Windows Hardening | Cross-platform log paths, Triton workaround, HF symlink bypass, 300s health check timeout |
| Dictation | Global system-wide hotkey (⌘+⇧+Space), frameless floating widget, streaming ASR via WebSocket, auto-paste |
| Batch Pipeline | Full batch TTS: extract → transcribe → translate → generate → mix → export, with live progress tracking |
🔜 Up Next
- 🎬 Lip-sync v2 — visual speech timing with wav2lip
- 📖 Audiobook Editor — chapter-aware long-form narration
- 🌐 Hosted Demo — try OmniVoice without installing anything
- 🔌 Plugin Marketplace — community-contributed TTS engines and effects
Community
| Channel | What happens there |
|---|---|
#showcase |
Members share their dubs, clones, and voice designs |
#help |
Setup issues, GPU troubleshooting, model questions |
#feature-requests |
Vote on what gets built next |
#dev |
Architecture discussions, PR reviews, engine integrations |
#announcements |
Release notes, breaking changes, early access |
→ Join the Discord — we respond to setup questions within hours, not days.
Contributing
We welcome contributions of all kinds — bug fixes, new TTS engine adapters, UI improvements, docs, and translations.
- 📖 Read the Contributing Guide for setup, code style, and PR workflow
- 🐛 Browse good first issues
- 💬 Join our Discord to discuss ideas or ask for help
FAQ
Is this really as good as ElevenLabs?
For voice cloning and dubbing, yes — OmniVoice uses a state-of-the-art diffusion TTS model with 646 languages (ElevenLabs supports 32). Quality is comparable for most use cases. Where ElevenLabs wins is in their polished cloud API and pre-made voice library. OmniVoice wins on privacy, cost, language coverage, and customizability.
Does it work on Apple Silicon (M1/M2/M3/M4)?
Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are available for faster transcription on Apple hardware.
How much VRAM do I need?
4 GB minimum. With ≤8 GB, the TTS model is automatically offloaded to CPU during transcription. With 8+ GB, everything runs on GPU simultaneously. No GPU at all? CPU mode works — just slower (~3× for TTS).
Can I use this commercially?
Personal, educational, internal-team, and non-commercial use is free under FSL-1.1-ALv2. Building a competing product or service on top of OmniVoice Studio requires a commercial license — see License. Pricing tiers coming soon. Each release converts to Apache 2.0 two years after publication.
What languages are supported?
646 languages for TTS via the OmniVoice model. Transcription (WhisperX) supports 99 languages. Translation coverage depends on the target language pair.
Can I add my own TTS engine?
Yes. OmniVoice uses a built-in backend registry. To add an engine in ~50 lines, subclass
TTSBackend in backend/services/tts_backend.py and add it to the _REGISTRY dictionary at the bottom. Six engines are built in: OmniVoice, CosyVoice, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, and KittenTTS. See the TTS Engines section for details.
License
OmniVoice Studio is source-available under the Functional Source License (FSL-1.1-ALv2).
Free for personal, educational, research, internal team, and non-commercial use. Each release converts to Apache 2.0 automatically two years after publication.
Business / enterprise users building a competing product or service on top of OmniVoice Studio need a commercial license. Pricing tiers coming soon. For inquiries in the meantime, reach out at OmniVoice@palash.dev.
See LICENSE for the full terms.
Acknowledgments
OmniVoice Studio is built on the shoulders of exceptional open-source work:
| Project | Role |
|---|---|
| OmniVoice (k2-fsa) | Zero-shot diffusion TTS engine — the core voice synthesis model |
| WhisperX | Word-level speech recognition and alignment |
| Demucs (Meta) | Music source separation for vocal isolation |
| Pyannote | Speaker diarization — who said what |
| CTranslate2 | Optimized Transformer inference on CPU and GPU |
| AudioSeal (Meta) | Invisible neural audio watermarking for AI provenance |
| Tauri | Native desktop app framework |
If you read this far, you're our kind of person.
⭐ Star this repo so others can find it too.
💬 Join the Discord to share what you build.