feat: bundle Claude Code agent skill at .claude/skills/omnivoice/ (#113)
* fix(mcp): drop unsupported FastMCP kwargs (mcp SDK >= 1.10)
The MCP server passes `version=` and `description=` to FastMCP(), but
neither kwarg exists on mcp >= 1.10 — the protocol version is now
managed internally and `description` was renamed to `instructions`.
Symptom on a fresh install (uv sync && pip install 'mcp[cli]'):
TypeError: FastMCP.__init__() got an unexpected keyword argument 'version'
Tested locally end-to-end:
- create_mcp_server() now constructs cleanly
- All 5 tools register and are listable via FastMCP.list_tools()
- generate_speech round-trip returns base64 WAV; ~24s server-side
for 4.2s of audio at steps=16 on Apple Silicon MPS
- pytest backend/ -x -q: 45 passed
* feat: bundle Claude Code agent skill at .claude/skills/omnivoice/
CLAUDE.md already invites contributions at .claude/skills/:
"No project skills found. Add skills to any of: .claude/skills/,
.agents/skills/, .cursor/skills/, .github/skills/, or .codex/skills/
with a SKILL.md index file."
But the existing .gitignore blanket-ignored .claude/ (line 41), making
the invited path un-trackable. This commit narrows the ignore so ad-hoc
Claude state stays out while deliberate skill bundles are tracked:
-.claude/
+.claude/*
+!.claude/skills/
+!.claude/skills/**
Once merged, any compatible agent client running
`npx skills add debpalash/OmniVoice-Studio` gets immediate context on:
- What the MCP server exposes (5 tools + 2 resources)
- When to pick OmniVoice vs other engines
- How to wire the stdio MCP server into a client config
- Backend lifecycle: start / health / stop scripts
- Common failure modes + fixes (port collision, model download stall,
missing HF_TOKEN, MPS fallback, voice-profile-not-found, etc.)
Conforms to Anthropic skill-creator conventions: frontmatter
description under 1024-char limit, body under 500 lines, references/
for detail, scripts/ for deterministic ops, no README/CHANGELOG
inside the skill, validates clean against quick_validate.py.
Verified locally that `npx skills list` discovers the bundled skill
automatically once cloned. End-to-end tested through MCP:
- generate_speech (English, demo voice, steps=16) -> 4.2 s WAV
- generate_speech (voice design via instruct only, steps=8) -> 6.3 s WAV
- generate_speech (Spanish, demo voice, steps=16) -> 2.8 s WAV
Depends on #112 (FastMCP API fix). Without it, every MCP tool call
fails with TypeError at server construction.
* feat(skill): add voice-clone end-to-end recipe + record-reference.sh helper
Two additions to the bundled skill, closing the gap where agents had no
procedural knowledge for creating a voice profile (the previous SKILL.md
said "use the UI or POST /profiles" but didn't include the recording +
trimming + verification workflow).
1. scripts/record-reference.sh — macOS-only helper that records a clean
reference clip with **audible** countdown + start/stop cues via
`say` + /System/Library/Sounds/Ping.aiff. Solves the buffering bug
where text-mode "speak now" prompts arrive after recording starts.
Captures a longer raw window then trims to ~10 sec of speech via
silenceremove + atrim. Plays back for verification. Prints the
next-step `curl` command for POST /profiles.
2. SKILL.md "Voice clone — end-to-end recipe" section (replaces the
stub one-liner). Covers:
- Path A: the bundled helper (one command, audible cues)
- Path B: manual ffmpeg flow if the helper doesn't fit
- POST /profiles multipart/form-data fields (required: name +
ref_audio; optional: ref_text, language, instruct, seed, personality)
- Reference clip quality factors that materially affect output
(single speaker, natural prosody, 3-10 sec sweet spot, ref_text
alignment, language correctness, loudness ≥ -15 dB peak)
Tested locally: recorded a 10-sec Spanish reference + 3-sec English
reference, created two profiles via the helper + curl flow, generated
14.1 sec of Spanish + 10.2 sec of English audio in the user's cloned
voice. Round-trip works end-to-end at steps=16 on Apple Silicon MPS.
Frontmatter description unchanged (860 chars, under the 1024 limit).
Body grew from ~120 to 169 lines (still well under the 500-line skill
ceiling).
* fix(skill): address P20 cross-review findings on PR #113
Adversarial multi-agent review (code + comment + silent-failure analyzers
on parallel reviewers) surfaced one blocker, one critical silent-failure
class, two medium-severity bugs, and two minor doc inaccuracies. All
addressed in this commit.
Blocker (cited 3x by both code-reviewer and comment-analyzer):
- SKILL.md linked references/engines-comparison.md three times (lines 44,
153, 160) but the file was never copied into the upstream skill tree.
+ Added the file (engine decision tree across OmniVoice / kokoro /
Voicebox / Edge TTS / ElevenLabs / cloud APIs).
Critical — record-reference.sh (was 4/10):
- Mic-permission silent failure: macOS denies the mic by sending a silent
stream; ffmpeg exits 0 with a valid silent WAV. The script printed
"✓ raw captured" and produced a degenerate reference clip that would
train a broken voice profile.
+ Parse mean_volume from volumedetect; exit 3 with a diagnostic
pointing the user to System Settings → Privacy → Microphone if
the recording is below -50 dB.
- afplay backgrounded with no exit check; if /System/Library/Sounds/*.aiff
is missing the user gets no audible cue.
+ beep() helper falls back to printf '\a' (terminal bell) when the
system sound file is missing.
- silenceremove silent corruption: silent input → near-empty output WAV,
exit 0.
+ ffprobe duration check after trim; exit 4 if < 2.0 sec.
- trap only covered EXIT; Ctrl-C / SIGTERM mid-recording leaked tmp file.
+ trap '...' EXIT INT TERM HUP.
- macOS guard ran after mktemp + trap.
+ Moved guard to first executable line.
- afplay verification swallowed stderr.
+ Drop 2>/dev/null; surface failure as a warning.
- Documented exit codes in header (0/2/3/4).
Medium — start-backend.sh (was 6/10):
- TOCTOU race: lsof check → uvicorn start could lose the port to another
process; only signal was a 60s health timeout.
+ Added `kill -0 $PID` check inside the probe loop; immediate exit 5
with log tail if uvicorn died.
- lsof check couldn't tell "stale us" from "third party" — same exit 3
for both.
+ ps -o command attribution; the message now tells the user whether
it's a stale uvicorn (suggest stop-backend.sh) or unknown process.
- Documented exit codes (0/2/3/4/5).
Medium — stop-backend.sh (was 7/10):
- No post-SIGKILL verification — script exited 0 even if process still
bound.
+ Added current_pids() helper; re-query after SIGKILL; exit 1 if still
bound, with lsof dump for diagnostics.
- 2>/dev/null || true on kill swallowed EPERM silently.
+ Capture stderr; classify EPERM vs ESRCH; exit 2 on EPERM with
actionable hint (try sudo).
- Documented exit codes (0/1/2).
Minor docs (comment-analyzer):
- SKILL.md line 120 claimed profiles persist as `<id>.wav`. Actual
backend (profiles.py:48-50) preserves uploaded extension.
+ Reworded to `<id>.<ext>` with explanation.
- mcp-setup.md line 68 cited HF cache path as Linux/macOS only.
Windows redirects via backend/core/config.py:38 to
%LOCALAPPDATA%\OmniVoice\hf_cache.
+ Added Windows row + reference to config.py.
Re-validated: all 6 files compile under set -euo pipefail; SKILL.md
frontmatter description stays at 860 chars (under 1024 cap); skill body
under 500 lines.
Diff: 6 files changed, ~+269/-47.
This commit is contained in:
Carlos D. Escobar-Valbuena
@@ -0,0 +1,169 @@
|
||||
---
|
||||
name: omnivoice
|
||||
description: "Local TTS, voice cloning, voice design, and video dubbing via the OmniVoice Studio MCP server (open-source ElevenLabs alternative; nothing leaves the machine, runs on MPS/CUDA/CPU). Use when: (1) generating speech from text in any of 646 languages, (2) cloning a voice from a 3-second reference clip, (3) designing a voice by gender/age/accent/pitch/style, (4) dubbing a video into another language, (5) listing voice profiles or personality presets, (6) producing narration where privacy, cost, or absent API keys matter, (7) non-English narration where Edge TTS/kokoro fall short, (8) batch audio for blog posts or content pipelines. Triggers: 'omnivoice', 'voice clone', 'clone this voice', 'tts', 'narrate', 'generate speech', 'voice synthesis', 'dub video', 'voice design', 'local tts', 'multilingual voice', 'narrate this post', 'elevenlabs alternative'."
|
||||
---
|
||||
|
||||
# OmniVoice
|
||||
|
||||
## Overview
|
||||
|
||||
Generate audio locally via the OmniVoice Studio MCP server. Tools: `generate_speech`, `list_voices`, `list_personalities`, `list_languages`, `check_health`. Resources: `voice://{id}`, `history://recent`.
|
||||
|
||||
## Prerequisites — Backend Must Be Running
|
||||
|
||||
The MCP tools all hit `$OMNIVOICE_API_URL` (default `http://localhost:3900`). If the backend is down, every tool returns a connection error. Install + boot:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/debpalash/OmniVoice-Studio.git "$OMNIVOICE_HOME"
|
||||
cd "$OMNIVOICE_HOME"
|
||||
uv sync
|
||||
VIRTUAL_ENV="$(pwd)/.venv" uv pip install 'mcp[cli]'
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
```bash
|
||||
scripts/check-health.sh # exit 0 if up
|
||||
scripts/start-backend.sh # boot in background (MPS/CUDA auto-detected)
|
||||
```
|
||||
|
||||
First synthesis call lazy-downloads the `k2-fsa/OmniVoice` model (~2.4 GB) from HuggingFace — cached on subsequent boots.
|
||||
|
||||
## Task Index — Pick the Right Tool
|
||||
|
||||
| Task | Tool | Notes |
|
||||
|---|---|---|
|
||||
| Verify backend is up | `check_health` | Returns `{"status":"ok","device":"mps|cuda|cpu"}` |
|
||||
| Text → audio with a saved voice | `generate_speech(text, profile_id)` | Returns base64 WAV. `profile_id="demo0001"` is the bundled demo voice |
|
||||
| Text → audio without a clone (voice design) | `generate_speech(text, instruct="…")` | Omit `profile_id`; pass an `instruct` like `"warm middle-aged female narrator, calm pace"` |
|
||||
| Multilingual narration | `generate_speech(text, language="es")` | Any ISO 639 code or `"Auto"` |
|
||||
| List existing voices | `list_voices` | Returns id, name, type, personality |
|
||||
| List personality presets | `list_personalities` | Returns narrator / casual / news-anchor / etc. with their `instruct` strings |
|
||||
| List supported languages | `list_languages` | 646 total; returns 20 popular + the full count |
|
||||
|
||||
For non-trivial decisions (which engine to use, when to pick OmniVoice over kokoro / Edge TTS / ElevenLabs), see [references/engines-comparison.md](references/engines-comparison.md).
|
||||
|
||||
For MCP wiring details, backend lifecycle, troubleshooting, and a clean teardown, see [references/mcp-setup.md](references/mcp-setup.md).
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### 1. One-shot narration with the demo voice
|
||||
|
||||
```python
|
||||
# As called through the MCP client (your agent will do this for you):
|
||||
result = generate_speech(
|
||||
text="Hello — this is OmniVoice generating speech locally.",
|
||||
profile_id="demo0001",
|
||||
language="English",
|
||||
steps=16, # 8 = fast/draft · 16 = balanced · 32 = quality
|
||||
)
|
||||
# result is JSON with audio_id, generation_time_s, audio_duration_s, format, wav_base64
|
||||
```
|
||||
|
||||
Benchmark: 4.2 s of audio in ~24 s server-side on Apple Silicon MPS at 16 diffusion steps.
|
||||
|
||||
### 2. Save the WAV to disk and play
|
||||
|
||||
Tool returns base64 PCM WAV (16-bit, mono, 24 kHz). Decode + write:
|
||||
|
||||
```python
|
||||
import base64, json
|
||||
payload = json.loads(result_text) # parse JSON the tool returns
|
||||
open("out.wav","wb").write(base64.b64decode(payload["wav_base64"]))
|
||||
```
|
||||
|
||||
On macOS: `afplay out.wav`. Convert to MP3 with `ffmpeg -i out.wav -codec:a libmp3lame -b:a 128k out.mp3`.
|
||||
|
||||
### 3. Voice clone — end-to-end recipe
|
||||
|
||||
Cloning needs a 3-10 second reference clip the model will use as a speaker embedding. The MCP server does NOT expose profile creation — it only reads existing profiles. Two paths to create one:
|
||||
|
||||
**Path A — bundled helper (macOS, recommended for fresh clones):**
|
||||
|
||||
```bash
|
||||
scripts/record-reference.sh ~/Downloads/my-ref.wav 12 1
|
||||
# args: output_path raw_duration_sec mic_index
|
||||
# Default mic_index=1 (MacBook built-in); list devices via:
|
||||
# ffmpeg -f avfoundation -list_devices true -i ""
|
||||
```
|
||||
|
||||
The script gives **audible** countdown + start/stop cues via macOS `say` + `/System/Library/Sounds/Ping.aiff` so the user knows when to speak (terminal stdout is buffered — text "speak now" prompts arrive too late). It records a longer raw window, then trims to ~10 seconds of speech via `silenceremove + atrim`, plays back for verification, and prints the next-step `curl` command.
|
||||
|
||||
**Path B — manual:**
|
||||
|
||||
```bash
|
||||
# 1. Record (mono, 24 kHz native — matches model's internal rate)
|
||||
ffmpeg -f avfoundation -i ":1" -t 12 -ac 1 -ar 24000 raw.wav
|
||||
|
||||
# 2. Trim leading silence + take first 10 sec of speech
|
||||
ffmpeg -i raw.wav \
|
||||
-af "silenceremove=start_periods=1:start_silence=0.05:start_threshold=-40dB,atrim=end=10" \
|
||||
-ac 1 -ar 24000 ref.wav
|
||||
|
||||
# 3. Verify
|
||||
ffmpeg -i ref.wav -af volumedetect -f null - 2>&1 | grep volume # max should be > -20 dB
|
||||
afplay ref.wav
|
||||
```
|
||||
|
||||
**POST to /profiles** (multipart/form-data — required fields: `name`, `ref_audio`):
|
||||
|
||||
```bash
|
||||
curl -X POST http://127.0.0.1:3900/profiles \
|
||||
-F "name=carlos-clone" \
|
||||
-F "ref_audio=@ref.wav" \
|
||||
-F "ref_text=The exact text spoken in the clip" \
|
||||
-F "language=English" \
|
||||
| python3 -m json.tool
|
||||
# returns { "id": "abc12345", "name": "carlos-clone" }
|
||||
```
|
||||
|
||||
Once created, pass `profile_id` to `generate_speech` (via MCP) or directly via `POST /generate`. Profiles persist in SQLite + reference-audio files at `~/Library/Application Support/OmniVoice/voices/<id>.<ext>` (the backend preserves the uploaded extension — `.wav` if you uploaded a WAV, `.mp3` if MP3, etc.). State persists across backend restarts.
|
||||
|
||||
**Reference clip tips that materially affect quality:**
|
||||
|
||||
| Factor | Why it matters |
|
||||
|---|---|
|
||||
| Single speaker | Mixed speakers blur the embedding |
|
||||
| Clean speech, no music/noise | Model embeds the noise too |
|
||||
| Natural prosody (avoid pangrams) | Diffusion samples replicate prosody, not just timbre |
|
||||
| 3-10 sec is the sweet spot | < 3 s lacks information; > 10 s adds compute without quality gain |
|
||||
| Match `ref_text` to what's spoken | Improves alignment, especially on noisy refs |
|
||||
| `language` correct | Wrong language → cross-lingual transfer artifacts |
|
||||
| Loudness peak ≥ -15 dB | Quiet refs work but normalize poorly |
|
||||
|
||||
### 4. Voice design (no reference clip)
|
||||
|
||||
Skip `profile_id`; provide an `instruct` string describing the desired voice:
|
||||
|
||||
```python
|
||||
generate_speech(
|
||||
text="Welcome to the future of agentic systems.",
|
||||
instruct="warm middle-aged female narrator, calm authoritative pace, documentary style",
|
||||
)
|
||||
```
|
||||
|
||||
Get pre-made instructs via `list_personalities` and copy the one matching the brief (narrator, casual, news-anchor, etc.).
|
||||
|
||||
### 5. Video dubbing (web UI only)
|
||||
|
||||
The MCP server does not expose the dubbing endpoint. The full transcribe → translate → re-voice → mux pipeline lives behind the desktop UI (`bun run desktop` in `$OMNIVOICE_HOME`) and the `/dub/*` REST routes. When the user asks to dub a video, point them to the UI; surface this skill only for the synthesis primitives above.
|
||||
|
||||
## When NOT to use OmniVoice
|
||||
|
||||
- **Fast English-only narration on weak hardware** → `kokoro-tts` is ~10× smaller and 2× realtime on CPU (see [references/engines-comparison.md](references/engines-comparison.md))
|
||||
- **Lowest-friction one-off TTS** → Edge TTS needs no install or backend
|
||||
- **Highest possible quality regardless of cost** → ElevenLabs still wins on English narration polish; OmniVoice ties or wins on multilingual + cloning
|
||||
- **Real-time streaming dictation** → use the OmniVoice desktop widget (`⌘+⇧+Space`), not the MCP server
|
||||
|
||||
## Resources
|
||||
|
||||
- [references/engines-comparison.md](references/engines-comparison.md) — Decision tree across OmniVoice / kokoro / Voicebox / Edge TTS / ElevenLabs / cloud APIs
|
||||
- [references/mcp-setup.md](references/mcp-setup.md) — MCP wiring, backend lifecycle, env vars, troubleshooting
|
||||
- [scripts/check-health.sh](scripts/check-health.sh) — `curl /health`, exit 0/1
|
||||
- [scripts/start-backend.sh](scripts/start-backend.sh) — Start uvicorn on 127.0.0.1:3900 with health probe
|
||||
- [scripts/stop-backend.sh](scripts/stop-backend.sh) — Clean shutdown via `kill -TERM` on the bound PID
|
||||
- [scripts/record-reference.sh](scripts/record-reference.sh) — macOS-only: record + trim + verify a reference clip for cloning, with audible cues (`say` + system beeps) that bypass terminal output buffering
|
||||
|
||||
Backend Swagger / OpenAPI: `http://127.0.0.1:3900/docs` (when backend is up).
|
||||
|
||||
Upstream: github.com/debpalash/OmniVoice-Studio — FSL-1.1-ALv2 (free for personal/internal/non-commercial; auto-converts to Apache-2.0 two years after each release).
|
||||
@@ -0,0 +1,71 @@
|
||||
# TTS Engine Selection — Decision Tree
|
||||
|
||||
When to pick OmniVoice vs other engines available in this workspace. Match the user's constraint to the right column.
|
||||
|
||||
## Decision tree
|
||||
|
||||
```
|
||||
Is voice cloning required?
|
||||
├─ yes → OmniVoice (3-sec ref clip, zero-shot, 646 langs)
|
||||
└─ no →
|
||||
Is the language non-English?
|
||||
├─ yes → OmniVoice (646 langs) or Edge TTS (subset, cloud)
|
||||
└─ no (English) →
|
||||
Is privacy required (no cloud)?
|
||||
├─ yes →
|
||||
│ Is GPU available?
|
||||
│ ├─ yes (CUDA/MPS) → OmniVoice (best quality) or Voicebox
|
||||
│ └─ no (CPU only) → kokoro-tts (2× realtime CPU) or OmniVoice on CPU (slow)
|
||||
└─ no (cloud OK) →
|
||||
Is cost-no-object?
|
||||
├─ yes → ElevenLabs (best polish), then OpenAI TTS
|
||||
└─ no → Edge TTS (free, unofficial, MS Azure neural)
|
||||
```
|
||||
|
||||
## Full comparison
|
||||
|
||||
| Engine | Quality | Clone | Multilingual | Cost | Privacy | Setup | Best for |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **OmniVoice** | 8-9/10 | ✅ 3-sec ref | 646 langs | Free | Local | Bun + uv install | Multilingual, cloning, privacy-critical |
|
||||
| ElevenLabs | 9-10/10 | ✅ 3-sec ref | 32 langs | $5-330/mo | Cloud | API key | Best English polish, fastest cloud TTS |
|
||||
| Voicebox (Qwen3-TTS) | 8-9/10 | ✅ | Multi | Free | Local | Docker | Self-hosted alternative to OmniVoice |
|
||||
| Voicebox (LuxTTS) | 7/10 | ❌ | Multi | Free | Local | Docker | CPU at 150× realtime |
|
||||
| kokoro-tts | 7-8/10 | ❌ | Multi (limited) | Free | Local | pip | Fast English narration on CPU |
|
||||
| mlx-audio | 7-8/10 | varies | Multi | Free | Local | pip | Apple Silicon native, 14+ sub-engines |
|
||||
| Edge TTS | 7-8/10 | ❌ | 50+ | Free* | Cloud | pip | Zero-friction one-off |
|
||||
| OpenAI TTS | 8/10 | ❌ | Multi | $0.015/1k chars | Cloud | API key | Convenient, cheap-ish, good quality |
|
||||
| Google Cloud TTS | 8/10 | ❌ | Multi | $4/1M chars (WaveNet) | Cloud | GCP project | Large free tier (1M chars/mo) |
|
||||
|
||||
*Edge TTS is unofficial. Microsoft could block it at any time.
|
||||
|
||||
## When OmniVoice wins decisively
|
||||
|
||||
1. **Voice cloning** — 3-sec reference clip, zero-shot, no fine-tuning. ElevenLabs is the only competitor; OmniVoice is free and local.
|
||||
2. **Long-tail languages** — 646 supported. ElevenLabs covers 32; everything else fewer.
|
||||
3. **Privacy / regulatory** — Nothing leaves the machine. ElevenLabs and OpenAI ship audio to their servers.
|
||||
4. **No-API-key constraint** — Local-first. No accounts.
|
||||
5. **Bulk generation without metered cost** — ElevenLabs bills per character. OmniVoice is free at any volume.
|
||||
|
||||
## When OmniVoice loses
|
||||
|
||||
1. **Lowest-friction one-off TTS** — Backend install + ~3 GB model + uvicorn boot. Edge TTS or OpenAI TTS is one command.
|
||||
2. **Fast English narration on weak hardware** — kokoro-tts is ~30 MB vs OmniVoice's 2.4 GB and runs 2× realtime on CPU. Use kokoro for blog-narration batch jobs unless you need cloning.
|
||||
3. **Streaming real-time TTS** — OmniVoice is diffusion-based and not streaming. Use Edge TTS or cloud APIs for true streaming.
|
||||
4. **Apple Silicon-only specialized voices** — `mlx-audio` ships 14 engines (Kokoro, CSM, Dia, Qwen3-TTS, etc.) that may match a specific voice better.
|
||||
|
||||
## Composition with content pipelines
|
||||
|
||||
OmniVoice fits between visual asset generation and video assembly:
|
||||
|
||||
```
|
||||
research → narrative → visual assets → AUDIO (OmniVoice) → video assembly → distribution
|
||||
```
|
||||
|
||||
Default for blog-post audio narration:
|
||||
|
||||
- **English, no cloning needed, fast** → kokoro-tts (cheap CPU)
|
||||
- **English, want a specific cloned voice** → OmniVoice with a saved profile
|
||||
- **Non-English** → OmniVoice
|
||||
- **One-time, no install** → Edge TTS
|
||||
|
||||
For Remotion-based video pipelines that previously required ElevenLabs, OmniVoice closes the last cloud dependency — pair it with any local image/video generator for a fully self-hosted multimedia stack.
|
||||
@@ -0,0 +1,102 @@
|
||||
# OmniVoice MCP Setup, Lifecycle, Troubleshooting
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# Pick any location. The scripts in this skill default to ~/OmniVoice-Studio if
|
||||
# $OMNIVOICE_HOME is unset.
|
||||
export OMNIVOICE_HOME="${HOME}/OmniVoice-Studio"
|
||||
|
||||
git clone https://github.com/debpalash/OmniVoice-Studio.git "$OMNIVOICE_HOME"
|
||||
cd "$OMNIVOICE_HOME"
|
||||
uv sync # ~1.6 GB venv on darwin arm64
|
||||
VIRTUAL_ENV="$(pwd)/.venv" uv pip install 'mcp[cli]' # SDK not in their lockfile yet
|
||||
```
|
||||
|
||||
Any non-default install location works as long as `$OMNIVOICE_HOME` is set in the env that launches the MCP server.
|
||||
|
||||
## MCP Wiring
|
||||
|
||||
Drop into your MCP client config (Claude Desktop, Claude Code at `~/.claude.json`, Cursor, OpenClaw, etc.). Replace `<OMNIVOICE_HOME>` with the absolute path:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"omnivoice": {
|
||||
"type": "stdio",
|
||||
"command": "uv",
|
||||
"args": [
|
||||
"--directory", "<OMNIVOICE_HOME>",
|
||||
"run", "python", "-m", "backend.mcp_server"
|
||||
],
|
||||
"env": { "OMNIVOICE_API_URL": "http://localhost:3900" }
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Restart the MCP client. The server only starts at client launch — in-session edits do not hot-reload.
|
||||
|
||||
> **Note (mcp SDK ≥ 1.10):** If you see `TypeError: FastMCP.__init__() got an unexpected keyword argument 'version'`, your `OmniVoice-Studio` checkout is older than [debpalash/OmniVoice-Studio#112](https://github.com/debpalash/OmniVoice-Studio/pull/112). Either `git pull` once that PR lands, or apply the 3-line patch manually: replace `version="…", description=(…)` with `instructions=(…)` in `backend/mcp_server.py`.
|
||||
|
||||
## Backend Lifecycle
|
||||
|
||||
The MCP server needs the FastAPI backend running:
|
||||
|
||||
```bash
|
||||
# Foreground (logs in terminal)
|
||||
cd "$OMNIVOICE_HOME"
|
||||
uv run uvicorn main:app --app-dir backend --host 127.0.0.1 --port 3900
|
||||
|
||||
# Detached, via the helper script in this skill
|
||||
scripts/start-backend.sh # idempotent
|
||||
scripts/check-health.sh # exit 0/1
|
||||
scripts/stop-backend.sh # graceful SIGTERM
|
||||
```
|
||||
|
||||
`127.0.0.1` keeps the API local-only. The project's `package.json` defaults to `0.0.0.0` which exposes the API on all interfaces — wider than needed for personal use.
|
||||
|
||||
First boot runs alembic migrations on the SQLite settings DB at `<data_dir>/omnivoice.db`. Idempotent — safe to re-run.
|
||||
|
||||
First synthesis call lazy-downloads the `k2-fsa/OmniVoice` model (~2.4 GB) into the HuggingFace cache. Path varies by OS:
|
||||
|
||||
- **macOS / Linux**: `~/.cache/huggingface/hub/`
|
||||
- **Windows**: `%LOCALAPPDATA%\OmniVoice\hf_cache` (OmniVoice redirects via `backend/core/config.py` to keep the cache off the system drive root)
|
||||
|
||||
Cached on subsequent boots.
|
||||
|
||||
## Idle Behavior
|
||||
|
||||
`GET /system/info` exposes `idle_timeout_seconds: 900`. After 15 min of no synthesis, the diffusion model is evicted from GPU memory but the FastAPI server stays up. Next call pays ~5-10 s warm-up.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Var | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `OMNIVOICE_HOME` | `~/OmniVoice-Studio` | Where the OmniVoice Studio repo is cloned (used by scripts in this skill) |
|
||||
| `OMNIVOICE_API_URL` | `http://localhost:3900` | MCP server's target backend URL |
|
||||
| `OMNIVOICE_TTS_BACKEND` | `omnivoice` | Switch engine: `cosyvoice`, `mlx-audio`, `voxcpm2`, `moss-tts-nano`, `kittentts` |
|
||||
| `HF_TOKEN` | (none) | Only needed for gated pyannote diarization models — basic TTS does not require one |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---|---|---|
|
||||
| MCP tool returns connection error | Backend not running | `scripts/start-backend.sh` |
|
||||
| `address already in use` | Stale uvicorn on 3900 | `lsof -nP -iTCP:3900 -sTCP:LISTEN` → `kill -TERM <pid>` |
|
||||
| `FastMCP.__init__() got unexpected keyword argument 'version'` | mcp SDK ≥ 1.10 dropped `version`/`description`, checkout pre-dates [#112](https://github.com/debpalash/OmniVoice-Studio/pull/112) | Update the checkout or apply the 3-line patch manually |
|
||||
| First call hangs 5-10 min | Model download from HuggingFace | Watch `~/.cache/huggingface/hub/models--k2-fsa--OmniVoice/` grow |
|
||||
| `/health` returns 500 | Alembic migration failed | Inspect `<data_dir>/crash_log.txt` |
|
||||
| Voice profile not found | `profile_id` invalid or profile not yet created | `list_voices` first to get valid IDs |
|
||||
| `pyannote.audio` errors at startup | Missing `HF_TOKEN` for diarization | Only matters for dub pipeline; basic TTS unaffected |
|
||||
| Generation slow on Apple Silicon | Diffusion fell back to CPU | `/health` should return `"device":"mps"`. Lower `steps` from 16 → 8 for drafts |
|
||||
|
||||
## Clean teardown
|
||||
|
||||
```bash
|
||||
scripts/stop-backend.sh # graceful shutdown
|
||||
# Uninstall: rm -rf "$OMNIVOICE_HOME" ~/.cache/huggingface/hub/models--k2-fsa--OmniVoice
|
||||
# Remove the `omnivoice` entry from your MCP client config
|
||||
```
|
||||
|
||||
User profiles + history live in the platform data dir (`~/Library/Application Support/OmniVoice/` on macOS; `~/.local/share/OmniVoice/` on Linux). Preserve across reinstalls if you want to keep your saved voice profiles.
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
URL="${OMNIVOICE_API_URL:-http://127.0.0.1:3900}/health"
|
||||
if out=$(curl -sf --max-time 3 "$URL" 2>/dev/null); then
|
||||
echo "$out"
|
||||
exit 0
|
||||
fi
|
||||
echo "omnivoice backend not reachable at $URL" >&2
|
||||
exit 1
|
||||
+147
@@ -0,0 +1,147 @@
|
||||
#!/usr/bin/env bash
|
||||
# Record a clean reference clip for OmniVoice voice cloning.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/record-reference.sh [output.wav] [duration_seconds] [mic_index]
|
||||
#
|
||||
# Defaults:
|
||||
# output = ~/Downloads/omnivoice-ref.wav
|
||||
# duration = 12 seconds raw capture (trimmed to 10 sec of speech)
|
||||
# mic_index = 1 (typically MacBook built-in; run `ffmpeg -f avfoundation -list_devices true -i ""` to enumerate)
|
||||
#
|
||||
# Why this script exists:
|
||||
# - macOS Terminal buffers stdout — "speak now" prints arrive AFTER the recording finishes.
|
||||
# This script uses `say` (macOS TTS) + system sound beeps to give the user *audible* cues
|
||||
# that bypass terminal buffering.
|
||||
# - 24 kHz mono is what OmniVoice's diffusion model expects internally; recording natively at
|
||||
# that rate avoids a resample step.
|
||||
# - silenceremove + atrim crops the user's actual speech window out of a longer raw capture,
|
||||
# so the user doesn't have to time their start perfectly.
|
||||
#
|
||||
# Cross-platform note: macOS-only (relies on avfoundation, `say`, /System/Library/Sounds).
|
||||
# Linux equivalent would use `arecord` + `espeak` + `aplay`; not implemented here.
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 success
|
||||
# 2 environment problem (wrong OS, missing tool)
|
||||
# 3 user-action problem (mic permission denied — captured silence)
|
||||
# 4 verification failed (trimmed reference too short)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Platform guard FIRST — before mktemp / trap / anything else macOS-specific
|
||||
if [ "$(uname)" != "Darwin" ]; then
|
||||
echo "✗ this helper is macOS-only (uses avfoundation, say, /System/Library/Sounds)." >&2
|
||||
echo " Linux equivalent: arecord + espeak + aplay; not implemented." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if ! command -v ffmpeg >/dev/null 2>&1; then
|
||||
echo "✗ ffmpeg not found — install via brew install ffmpeg" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if ! command -v ffprobe >/dev/null 2>&1; then
|
||||
echo "✗ ffprobe not found — install via brew install ffmpeg" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
OUT="${1:-$HOME/Downloads/omnivoice-ref.wav}"
|
||||
DUR="${2:-12}"
|
||||
MIC="${3:-1}"
|
||||
|
||||
RAW="$(mktemp -t omnivoice-raw-XXXXX).wav"
|
||||
# Cover all common termination paths so the temp file never leaks
|
||||
trap 'rm -f "$RAW"' EXIT INT TERM HUP
|
||||
|
||||
# Beep helper — uses system sound if available, otherwise terminal bell.
|
||||
beep() {
|
||||
local snd="$1"
|
||||
if [ -f "$snd" ]; then
|
||||
afplay "$snd" &
|
||||
else
|
||||
printf '\a' >&2
|
||||
fi
|
||||
}
|
||||
|
||||
PING="/System/Library/Sounds/Ping.aiff"
|
||||
POP="/System/Library/Sounds/Pop.aiff"
|
||||
|
||||
echo "▶︎ Recording reference for OmniVoice voice cloning"
|
||||
echo " mic index: $MIC (run \`ffmpeg -f avfoundation -list_devices true -i \"\"\` to list)"
|
||||
echo " raw window: ${DUR}s · output: $OUT"
|
||||
echo
|
||||
|
||||
# Spoken instructions + countdown (heard in real time, bypasses terminal buffering)
|
||||
say -r 180 "Recording in three seconds. After the high beep, speak your reference phrase. Recording continues for ${DUR} seconds."
|
||||
sleep 0.3
|
||||
say -r 220 "three"; say -r 220 "two"; say -r 220 "one"
|
||||
|
||||
# Start cue
|
||||
beep "$PING"
|
||||
|
||||
# Capture
|
||||
ffmpeg -hide_banner -loglevel error -y -f avfoundation -i ":$MIC" -t "$DUR" -ac 1 -ar 24000 "$RAW"
|
||||
|
||||
# End cue
|
||||
beep "$POP"
|
||||
|
||||
echo "✓ raw captured"
|
||||
|
||||
# Sanity-check levels — bail loudly if the recording is silent (mic permission denied is the
|
||||
# most common cause: macOS gives the calling process a silent stream and ffmpeg returns 0).
|
||||
LEVELS=$(ffmpeg -hide_banner -i "$RAW" -af "volumedetect" -f null - 2>&1)
|
||||
echo "raw levels:"
|
||||
echo "$LEVELS" | grep -E "mean_volume|max_volume" | sed 's/^/ /'
|
||||
|
||||
MEAN=$(echo "$LEVELS" | grep -oE "mean_volume:[[:space:]]*-?[0-9.]+" | grep -oE "\-?[0-9.]+$" | head -1)
|
||||
if [ -z "$MEAN" ]; then
|
||||
echo "✗ could not parse audio levels — ffmpeg may have failed to record" >&2
|
||||
exit 3
|
||||
fi
|
||||
# Compare via awk (bash can't do floating-point natively)
|
||||
if awk -v m="$MEAN" 'BEGIN { exit !(m < -50) }'; then
|
||||
echo "✗ recording is silent (mean ${MEAN} dB < -50 dB)." >&2
|
||||
echo " Most likely: macOS denied microphone access to the calling process." >&2
|
||||
echo " Fix: System Settings → Privacy & Security → Microphone → enable for your terminal" >&2
|
||||
echo " (or for the agent harness). Then re-run this script." >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
# Trim leading silence + take first ~10 seconds of speech (or all of it if shorter)
|
||||
TARGET=10
|
||||
ffmpeg -hide_banner -loglevel error -y -i "$RAW" \
|
||||
-af "silenceremove=start_periods=1:start_silence=0.05:start_threshold=-40dB,atrim=end=${TARGET}" \
|
||||
-ac 1 -ar 24000 "$OUT"
|
||||
|
||||
# Verify the trim produced a usable reference (≥ 2.0s of speech)
|
||||
REF_DUR=$(ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "$OUT" 2>/dev/null || echo "0")
|
||||
if awk -v d="$REF_DUR" 'BEGIN { exit !(d < 2.0) }'; then
|
||||
echo "✗ trimmed reference is only ${REF_DUR}s (< 2s minimum)." >&2
|
||||
echo " silenceremove found very little speech above -40 dB threshold." >&2
|
||||
echo " Likely cause: you spoke too softly, or the mic captured mostly background noise." >&2
|
||||
echo " Try again, speak closer to the mic." >&2
|
||||
exit 4
|
||||
fi
|
||||
|
||||
echo "✓ trimmed reference: $OUT"
|
||||
ffmpeg -hide_banner -i "$OUT" -af "volumedetect" -f null - 2>&1 \
|
||||
| grep -E "Duration|mean_volume|max_volume" \
|
||||
| sed 's/^/ /'
|
||||
|
||||
# Auto-play back so the user can verify before POSTing.
|
||||
# Don't swallow stderr — if playback fails the user should know.
|
||||
echo
|
||||
echo "▶︎ playing reference for verification..."
|
||||
if ! afplay -v 3 "$OUT"; then
|
||||
echo "⚠ verification playback failed — afplay exited non-zero. The file may still be valid;" >&2
|
||||
echo " try \`afplay $OUT\` manually or open it in a player to confirm." >&2
|
||||
fi
|
||||
echo "✓ done"
|
||||
echo
|
||||
echo "Next: POST to /profiles to create a voice profile:"
|
||||
echo " curl -X POST http://127.0.0.1:3900/profiles \\"
|
||||
echo " -F \"name=my-voice\" \\"
|
||||
echo " -F \"ref_audio=@$OUT\" \\"
|
||||
echo " -F \"ref_text=<the exact text you spoke>\" \\"
|
||||
echo " -F \"language=English\""
|
||||
+72
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env bash
|
||||
# Start the OmniVoice FastAPI backend on 127.0.0.1:3900, detached, idempotent.
|
||||
# Honors $OMNIVOICE_HOME (default ~/OmniVoice-Studio).
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 success (already running, or freshly started + healthy within 60s)
|
||||
# 2 $OMNIVOICE_HOME doesn't exist
|
||||
# 3 port 3900 held but /health unresponsive (won't auto-kill — caller decides)
|
||||
# 4 backend started but /health didn't respond within 60s
|
||||
# 5 uvicorn process died during the wait
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
HOME_DIR="${OMNIVOICE_HOME:-$HOME/OmniVoice-Studio}"
|
||||
URL="${OMNIVOICE_API_URL:-http://127.0.0.1:3900}"
|
||||
LOG="$HOME_DIR/backend.log"
|
||||
|
||||
if [ ! -d "$HOME_DIR" ]; then
|
||||
echo "OMNIVOICE_HOME not found: $HOME_DIR" >&2
|
||||
echo "See references/mcp-setup.md for install steps." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Already up?
|
||||
if curl -sf --max-time 2 "$URL/health" >/dev/null 2>&1; then
|
||||
echo "already running: $URL"
|
||||
curl -sf "$URL/health"; echo
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Port held by something else? Identify it before refusing.
|
||||
BIND_PID="$(lsof -nP -iTCP:3900 -sTCP:LISTEN -t 2>/dev/null | head -1)"
|
||||
if [ -n "$BIND_PID" ]; then
|
||||
BIND_CMD="$(ps -o command= -p "$BIND_PID" 2>/dev/null || echo "?")"
|
||||
echo "port 3900 held by PID $BIND_PID but /health not responding — investigate before starting" >&2
|
||||
echo " bound process: $BIND_CMD" >&2
|
||||
case "$BIND_CMD" in
|
||||
*uvicorn*main:app*)
|
||||
echo " → looks like a stale uvicorn from a previous run; consider scripts/stop-backend.sh" >&2
|
||||
;;
|
||||
*)
|
||||
echo " → unknown process holds the port; resolve before re-running this script" >&2
|
||||
;;
|
||||
esac
|
||||
exit 3
|
||||
fi
|
||||
|
||||
cd "$HOME_DIR"
|
||||
nohup uv run uvicorn main:app --app-dir backend --host 127.0.0.1 --port 3900 \
|
||||
> "$LOG" 2>&1 &
|
||||
PID=$!
|
||||
echo "starting backend (PID $PID, log: $LOG)..."
|
||||
|
||||
# Wait up to 60s for /health, AND verify the child stays alive.
|
||||
# A dead child means immediate bind failure (port grabbed in the TOCTOU window above)
|
||||
# or an early crash — surface it instead of waiting the full timeout.
|
||||
for i in $(seq 1 30); do
|
||||
sleep 2
|
||||
if ! kill -0 "$PID" 2>/dev/null; then
|
||||
echo "✗ uvicorn (PID $PID) exited during startup — see $LOG" >&2
|
||||
tail -20 "$LOG" >&2 || true
|
||||
exit 5
|
||||
fi
|
||||
if curl -sf --max-time 2 "$URL/health" >/dev/null 2>&1; then
|
||||
curl -sf "$URL/health"; echo
|
||||
echo "ready after $((i*2))s"
|
||||
exit 0
|
||||
fi
|
||||
done
|
||||
|
||||
echo "backend did not respond on $URL/health within 60s — see $LOG" >&2
|
||||
exit 4
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
#!/usr/bin/env bash
|
||||
# Gracefully stop the OmniVoice backend bound to 127.0.0.1:3900.
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 stopped (or never running)
|
||||
# 1 process(es) still bound to 3900 after SIGTERM + SIGKILL escalation
|
||||
# 2 permission denied killing a bound process (EPERM — try sudo or a different account)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Helper: discover current PIDs bound to 3900 (recomputed each time to avoid stale data)
|
||||
current_pids() {
|
||||
lsof -nP -iTCP:3900 -sTCP:LISTEN -t 2>/dev/null || true
|
||||
}
|
||||
|
||||
PIDS="$(current_pids)"
|
||||
if [ -z "$PIDS" ]; then
|
||||
echo "no listener on 3900"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# SIGTERM phase — let processes shut down cleanly. Surface EPERM loudly so the user
|
||||
# knows when they can't actually stop the backend (wrong user, sandboxed process, etc.).
|
||||
EPERM_HIT=0
|
||||
for pid in $PIDS; do
|
||||
echo "kill -TERM $pid"
|
||||
if ! kill -TERM "$pid" 2>/tmp/.omnivoice-kill-err; then
|
||||
if grep -qi 'permitted\|denied' /tmp/.omnivoice-kill-err 2>/dev/null; then
|
||||
echo " ✗ EPERM — cannot signal PID $pid (different user / sandboxed)" >&2
|
||||
EPERM_HIT=1
|
||||
elif grep -qi 'no such process' /tmp/.omnivoice-kill-err 2>/dev/null; then
|
||||
: # benign — process already gone
|
||||
else
|
||||
cat /tmp/.omnivoice-kill-err >&2 || true
|
||||
fi
|
||||
fi
|
||||
done
|
||||
rm -f /tmp/.omnivoice-kill-err
|
||||
|
||||
if [ "$EPERM_HIT" -eq 1 ]; then
|
||||
echo "✗ at least one bound process refused SIGTERM (permission denied)." >&2
|
||||
echo " Try \`sudo $(realpath "$0")\` or stop the owning process manually." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Wait up to 10s for graceful exit (poll lsof, not the captured PID list — PIDs may have been
|
||||
# reaped or recycled by the kernel during this window).
|
||||
for i in $(seq 1 5); do
|
||||
sleep 2
|
||||
if [ -z "$(current_pids)" ]; then
|
||||
echo "stopped"
|
||||
exit 0
|
||||
fi
|
||||
done
|
||||
|
||||
# Escalate to SIGKILL on whatever is currently bound (re-query — don't trust stale PIDs).
|
||||
echo "still running after 10s — escalating to SIGKILL" >&2
|
||||
PIDS="$(current_pids)"
|
||||
for pid in $PIDS; do kill -KILL "$pid" 2>/dev/null || true; done
|
||||
sleep 1
|
||||
|
||||
# Verify the port is actually free now. If it's still held, the script failed its job.
|
||||
if [ -n "$(current_pids)" ]; then
|
||||
echo "✗ port 3900 STILL bound after SIGKILL:" >&2
|
||||
lsof -nP -iTCP:3900 -sTCP:LISTEN 2>&1 | head -3 >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "stopped (after SIGKILL)"
|
||||
exit 0
|
||||
+5
-1
@@ -38,7 +38,11 @@ Thumbs.db
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
# Editor / tool caches
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
.claude/
|
||||
# Ignore ad-hoc Claude Code state, but allow project-bundled skills
|
||||
# (CLAUDE.md invites `.claude/skills/<name>/SKILL.md`).
|
||||
.claude/*
|
||||
!.claude/skills/
|
||||
!.claude/skills/**
|
||||
/.cache*
|
||||
/.tmp/
|
||||
|
||||
|
||||
@@ -48,8 +48,7 @@ def create_mcp_server():
|
||||
FastMCP = _ensure_mcp()
|
||||
mcp = FastMCP(
|
||||
"OmniVoice Studio",
|
||||
version="0.3.0",
|
||||
description=(
|
||||
instructions=(
|
||||
"AI-agent interface for OmniVoice Studio — voice cloning, "
|
||||
"voice design, and video dubbing in 646 languages."
|
||||
),
|
||||
|
||||
Reference in New Issue
Block a user