Merge branch 'main' into fix/2163-gated-install-token
This commit is contained in:
shivsin25
@@ -0,0 +1,29 @@
|
||||
# VoiceStudio compatibility entry
|
||||
|
||||
The current cross-agent package is [voicestudio](../../../../skills/voicestudio/SKILL.md).
|
||||
For new installations use `npx skills add debpalash/VoiceStudio --skill voicestudio`.
|
||||
|
||||
Use the running backend at the user's configured address (default
|
||||
`http://localhost:3900`). Check `/health`, discover `/openapi.json` and
|
||||
`/v1/audio/voices`, then use the installed schema for speech, transcription,
|
||||
profiles, and jobs. The HTTP MCP endpoint is `/mcp`; discover tools from the
|
||||
connected server instead of assuming this older package's tool inventory.
|
||||
|
||||
Launch the installed Electron app if the backend is unavailable. For source
|
||||
development follow the checkout's Electron README. Existing helpers in
|
||||
`scripts/` support legacy source installations; inspect their environment
|
||||
and dependency assumptions before running them.
|
||||
|
||||
Model downloads and remote services require the user's choice. Never silently
|
||||
install models, promise fixed latency, or treat compatibility voice names as
|
||||
real provider voices. Validate saved audio and asynchronous job completion
|
||||
before reporting success. Protected backends require configured credentials;
|
||||
never disable authentication to make an example work.
|
||||
|
||||
Source and current setup documentation:
|
||||
https://github.com/debpalash/VoiceStudio
|
||||
|
||||
This archived entry is not an installable skill. Existing installations should
|
||||
remove the old `omnivoice` / `oss-maintainer` entries and install `voicestudio` /
|
||||
`voicestudio-maintainer` from the canonical repository. Legacy helpers remain
|
||||
for existing users; the Electron supervisor is the preferred launcher.
|
||||
@@ -1,172 +0,0 @@
|
||||
---
|
||||
name: omnivoice
|
||||
description: "Local TTS, voice cloning, voice design, and video dubbing via the VoiceStudio MCP server (open-source ElevenLabs alternative; nothing leaves the machine, runs on MPS/CUDA/CPU). Use when: (1) generating speech from text in any of 646 languages, (2) cloning a voice from a 3-second reference clip, (3) designing a voice by gender/age/accent/pitch/style, (4) dubbing a video into another language, (5) listing voice profiles or personality presets, (6) producing narration where privacy, cost, or absent API keys matter, (7) non-English narration where Edge TTS/kokoro fall short, (8) batch audio for blog posts or content pipelines. Triggers: 'omnivoice', 'voice clone', 'clone this voice', 'tts', 'narrate', 'generate speech', 'voice synthesis', 'dub video', 'voice design', 'local tts', 'multilingual voice', 'narrate this post', 'elevenlabs alternative'."
|
||||
---
|
||||
|
||||
# VoiceStudio
|
||||
|
||||
The canonical cross-agent package lives at `skills/omnivoice/SKILL.md`. This
|
||||
Claude-specific package retains the MCP lifecycle helpers and references.
|
||||
|
||||
## Overview
|
||||
|
||||
Generate audio locally via the VoiceStudio MCP server. Tools: `generate_speech`, `list_voices`, `list_personalities`, `list_languages`, `check_health`. Resources: `voice://{id}`, `history://recent`.
|
||||
|
||||
## Prerequisites — Backend Must Be Running
|
||||
|
||||
The MCP tools all hit `$OMNIVOICE_API_URL` (default `http://localhost:3900`). If the backend is down, every tool returns a connection error. Install + boot:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/debpalash/VoiceStudio.git "$OMNIVOICE_HOME"
|
||||
cd "$OMNIVOICE_HOME"
|
||||
uv sync
|
||||
VIRTUAL_ENV="$(pwd)/.venv" uv pip install 'mcp[cli]'
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
```bash
|
||||
scripts/check-health.sh # exit 0 if up
|
||||
scripts/start-backend.sh # boot in background (MPS/CUDA auto-detected)
|
||||
```
|
||||
|
||||
First synthesis call lazy-downloads the `k2-fsa/OmniVoice` model (~2.4 GB) from HuggingFace — cached on subsequent boots.
|
||||
|
||||
## Task Index — Pick the Right Tool
|
||||
|
||||
| Task | Tool | Notes |
|
||||
|---|---|---|
|
||||
| Verify backend is up | `check_health` | Returns `{"status":"ok","device":"mps|cuda|cpu"}` |
|
||||
| Text → audio with a saved voice | `generate_speech(text, profile_id)` | Returns base64 WAV. `profile_id="demo0001"` is the bundled demo voice |
|
||||
| Text → audio without a clone (voice design) | `generate_speech(text, instruct="…")` | Omit `profile_id`; pass an `instruct` like `"warm middle-aged female narrator, calm pace"` |
|
||||
| Multilingual narration | `generate_speech(text, language="es")` | Any ISO 639 code or `"Auto"` |
|
||||
| List existing voices | `list_voices` | Returns id, name, type, personality |
|
||||
| List personality presets | `list_personalities` | Returns narrator / casual / news-anchor / etc. with their `instruct` strings |
|
||||
| List supported languages | `list_languages` | 646 total; returns 20 popular + the full count |
|
||||
|
||||
For non-trivial decisions (which engine to use, when to pick VoiceStudio over kokoro / Edge TTS / ElevenLabs), see [references/engines-comparison.md](references/engines-comparison.md).
|
||||
|
||||
For MCP wiring details, backend lifecycle, troubleshooting, and a clean teardown, see [references/mcp-setup.md](references/mcp-setup.md).
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### 1. One-shot narration with the demo voice
|
||||
|
||||
```python
|
||||
# As called through the MCP client (your agent will do this for you):
|
||||
result = generate_speech(
|
||||
text="Hello — this is VoiceStudio generating speech locally.",
|
||||
profile_id="demo0001",
|
||||
language="English",
|
||||
steps=16, # 8 = fast/draft · 16 = balanced · 32 = quality
|
||||
)
|
||||
# result is JSON with audio_id, generation_time_s, audio_duration_s, format, wav_base64
|
||||
```
|
||||
|
||||
Benchmark: 4.2 s of audio in ~24 s server-side on Apple Silicon MPS at 16 diffusion steps.
|
||||
|
||||
### 2. Save the WAV to disk and play
|
||||
|
||||
Tool returns base64 PCM WAV (16-bit, mono, 24 kHz). Decode + write:
|
||||
|
||||
```python
|
||||
import base64, json
|
||||
payload = json.loads(result_text) # parse JSON the tool returns
|
||||
open("out.wav","wb").write(base64.b64decode(payload["wav_base64"]))
|
||||
```
|
||||
|
||||
On macOS: `afplay out.wav`. Convert to MP3 with `ffmpeg -i out.wav -codec:a libmp3lame -b:a 128k out.mp3`.
|
||||
|
||||
### 3. Voice clone — end-to-end recipe
|
||||
|
||||
Cloning needs a 3-10 second reference clip the model will use as a speaker embedding. The MCP server does NOT expose profile creation — it only reads existing profiles. Two paths to create one:
|
||||
|
||||
**Path A — bundled helper (macOS, recommended for fresh clones):**
|
||||
|
||||
```bash
|
||||
scripts/record-reference.sh ~/Downloads/my-ref.wav 12 1
|
||||
# args: output_path raw_duration_sec mic_index
|
||||
# Default mic_index=1 (MacBook built-in); list devices via:
|
||||
# ffmpeg -f avfoundation -list_devices true -i ""
|
||||
```
|
||||
|
||||
The script gives **audible** countdown + start/stop cues via macOS `say` + `/System/Library/Sounds/Ping.aiff` so the user knows when to speak (terminal stdout is buffered — text "speak now" prompts arrive too late). It records a longer raw window, then trims to ~10 seconds of speech via `silenceremove + atrim`, plays back for verification, and prints the next-step `curl` command.
|
||||
|
||||
**Path B — manual:**
|
||||
|
||||
```bash
|
||||
# 1. Record (mono, 24 kHz native — matches model's internal rate)
|
||||
ffmpeg -f avfoundation -i ":1" -t 12 -ac 1 -ar 24000 raw.wav
|
||||
|
||||
# 2. Trim leading silence + take first 10 sec of speech
|
||||
ffmpeg -i raw.wav \
|
||||
-af "silenceremove=start_periods=1:start_silence=0.05:start_threshold=-40dB,atrim=end=10" \
|
||||
-ac 1 -ar 24000 ref.wav
|
||||
|
||||
# 3. Verify
|
||||
ffmpeg -i ref.wav -af volumedetect -f null - 2>&1 | grep volume # max should be > -20 dB
|
||||
afplay ref.wav
|
||||
```
|
||||
|
||||
**POST to /profiles** (multipart/form-data — required fields: `name`, `ref_audio`):
|
||||
|
||||
```bash
|
||||
curl -X POST http://127.0.0.1:3900/profiles \
|
||||
-F "name=carlos-clone" \
|
||||
-F "ref_audio=@ref.wav" \
|
||||
-F "ref_text=The exact text spoken in the clip" \
|
||||
-F "language=English" \
|
||||
| python3 -m json.tool
|
||||
# returns { "id": "abc12345", "name": "carlos-clone" }
|
||||
```
|
||||
|
||||
Once created, pass `profile_id` to `generate_speech` (via MCP) or directly via `POST /generate`. Profiles persist in SQLite + reference-audio files at `~/Library/Application Support/OmniVoice/voices/<id>.<ext>` (the backend preserves the uploaded extension — `.wav` if you uploaded a WAV, `.mp3` if MP3, etc.). State persists across backend restarts.
|
||||
|
||||
**Reference clip tips that materially affect quality:**
|
||||
|
||||
| Factor | Why it matters |
|
||||
|---|---|
|
||||
| Single speaker | Mixed speakers blur the embedding |
|
||||
| Clean speech, no music/noise | Model embeds the noise too |
|
||||
| Natural prosody (avoid pangrams) | Diffusion samples replicate prosody, not just timbre |
|
||||
| 3-10 sec is the sweet spot | < 3 s lacks information; > 10 s adds compute without quality gain |
|
||||
| Match `ref_text` to what's spoken | Improves alignment, especially on noisy refs |
|
||||
| `language` correct | Wrong language → cross-lingual transfer artifacts |
|
||||
| Loudness peak ≥ -15 dB | Quiet refs work but normalize poorly |
|
||||
|
||||
### 4. Voice design (no reference clip)
|
||||
|
||||
Skip `profile_id`; provide an `instruct` string describing the desired voice:
|
||||
|
||||
```python
|
||||
generate_speech(
|
||||
text="Welcome to the future of agentic systems.",
|
||||
instruct="warm middle-aged female narrator, calm authoritative pace, documentary style",
|
||||
)
|
||||
```
|
||||
|
||||
Get pre-made instructs via `list_personalities` and copy the one matching the brief (narrator, casual, news-anchor, etc.).
|
||||
|
||||
### 5. Video dubbing (web UI only)
|
||||
|
||||
The MCP server does not expose the dubbing endpoint. The full transcribe → translate → re-voice → mux pipeline lives behind the desktop UI (`bun run desktop` in `$OMNIVOICE_HOME`) and the `/dub/*` REST routes. When the user asks to dub a video, point them to the UI; surface this skill only for the synthesis primitives above.
|
||||
|
||||
## When NOT to use VoiceStudio
|
||||
|
||||
- **Fast English-only narration on weak hardware** → `kokoro-tts` is ~10× smaller and 2× realtime on CPU (see [references/engines-comparison.md](references/engines-comparison.md))
|
||||
- **Lowest-friction one-off TTS** → Edge TTS needs no install or backend
|
||||
- **Highest possible quality regardless of cost** → ElevenLabs still wins on English narration polish; VoiceStudio ties or wins on multilingual + cloning
|
||||
- **Real-time streaming dictation** → use the VoiceStudio desktop widget (`⌘+⇧+Space`), not the MCP server
|
||||
|
||||
## Resources
|
||||
|
||||
- [references/engines-comparison.md](references/engines-comparison.md) — Decision tree across VoiceStudio / kokoro / Voicebox / Edge TTS / ElevenLabs / cloud APIs
|
||||
- [references/mcp-setup.md](references/mcp-setup.md) — MCP wiring, backend lifecycle, env vars, troubleshooting
|
||||
- [scripts/check-health.sh](scripts/check-health.sh) — `curl /health`, exit 0/1
|
||||
- [scripts/start-backend.sh](scripts/start-backend.sh) — Start uvicorn on 127.0.0.1:3900 with health probe
|
||||
- [scripts/stop-backend.sh](scripts/stop-backend.sh) — Clean shutdown via `kill -TERM` on the bound PID
|
||||
- [scripts/record-reference.sh](scripts/record-reference.sh) — macOS-only: record + trim + verify a reference clip for cloning, with audible cues (`say` + system beeps) that bypass terminal output buffering
|
||||
|
||||
Backend Swagger / OpenAPI: `http://127.0.0.1:3900/docs` (when backend is up).
|
||||
|
||||
Upstream: github.com/debpalash/VoiceStudio. The app uses AGPL-3.0-only; optional engines and downloaded models retain their own licenses. See `LICENSE-NOTICE.md` in the repository.
|
||||
+20
-10
@@ -56,7 +56,17 @@ bun install
|
||||
bun run dev
|
||||
```
|
||||
|
||||
This starts both services:
|
||||
This launches Electron with hot reload. Its runtime supervisor manages backend setup
|
||||
and startup; do not launch a second backend. See [Electron setup](../electron/README.md).
|
||||
|
||||
```bash
|
||||
bun run build # build Electron
|
||||
bun run start # launch the built Electron app
|
||||
bun run dist # package locally without publishing
|
||||
bun run dev:web # legacy browser UI + backend
|
||||
```
|
||||
|
||||
The legacy browser command starts both services:
|
||||
|
||||
| Service | URL | What it does |
|
||||
|---------|-----|---|
|
||||
@@ -71,29 +81,29 @@ cause doesn't scroll away with the terminal. The same death is also reported
|
||||
as a crash notice in the UI the next time the backend starts (see
|
||||
[docs/install/troubleshooting.md §14c](docs/install/troubleshooting.md)).
|
||||
|
||||
### Desktop App (Tauri)
|
||||
### Legacy Desktop App (Tauri)
|
||||
|
||||
```bash
|
||||
bun run desktop # dev: hot-reload Tauri shell + backend
|
||||
bun run desktop-prod # production: builds, bundles the backend, then launches
|
||||
bun run tauri # legacy dev: hot-reload Tauri shell + backend
|
||||
bun run tauri:desktop-prod # legacy production: builds, bundles the backend, then launches
|
||||
```
|
||||
|
||||
Both run `uv sync` first (so the Python backend env is set up) and start the
|
||||
backend automatically — you do **not** start it separately. Use the exact script
|
||||
names: there is no `desktop=prod` (note the **hyphen** in `desktop-prod`).
|
||||
`desktop-prod` is Windows-aware (auto-detects bash/git; see `scripts/desktop-prod.mjs`).
|
||||
names: there is no `desktop=prod` (note the **hyphen** in `tauri:desktop-prod`).
|
||||
`tauri:desktop-prod` is Windows-aware (auto-detects bash/git; see `scripts/desktop-prod.mjs`).
|
||||
|
||||
Requires [Rust](https://rustup.rs/) and platform-specific Tauri dependencies — see the [Tauri prerequisites](https://v2.tauri.app/start/prerequisites/).
|
||||
|
||||
After installing Rust with rustup (or `uv` with its installer), a terminal that
|
||||
was already open still has the old `PATH`. The desktop launchers (`bun desktop`,
|
||||
`bun desktop-prod`, `bun desktop-fresh`) detect this and add `~/.cargo/bin` /
|
||||
was already open still has the old `PATH`. The desktop launchers (`bun tauri`,
|
||||
`bun tauri:desktop-prod`, `bun tauri:desktop-fresh`) detect this and add `~/.cargo/bin` /
|
||||
`~/.local/bin` for that run, printing a one-line note; to make it permanent,
|
||||
open a new terminal, or on macOS/Linux load Cargo into the current one:
|
||||
|
||||
```bash
|
||||
source "$HOME/.cargo/env"
|
||||
bun desktop
|
||||
bun run tauri
|
||||
```
|
||||
|
||||
If Rust is genuinely not installed, the launchers stop up front with the
|
||||
@@ -310,7 +320,7 @@ that — the agent recalls the architecture, conventions, and your past findings
|
||||
instead of re-reading the tree each time. [**memxt**](https://github.com/debpalash/memxt)
|
||||
(100% local, MCP-based, built by this project's maintainer) exists for exactly
|
||||
this; any MCP memory server works. Pair it with the repo's agent skill —
|
||||
`npx skills add debpalash/omnivoice-studio` — so your agent knows the project's
|
||||
`npx skills add debpalash/VoiceStudio` — so your agent knows the project's
|
||||
hard rules from the first prompt.
|
||||
|
||||
## Quality gates your PR must pass
|
||||
|
||||
@@ -63,10 +63,9 @@ jobs:
|
||||
experimental: false
|
||||
- os: macos-14
|
||||
platform: darwin-arm64
|
||||
# Metal build path is unpublished upstream (Pitfall 1 in
|
||||
# 04-RESEARCH.md); experimental so a failed Metal build doesn't
|
||||
# block — the SPIKE-01 ADR records the in-process fallback.
|
||||
experimental: true
|
||||
# Apple Silicon Metal build compiles cleanly with -DGGML_METAL=ON
|
||||
# at the pinned SHA (#2105); non-experimental to catch regressions.
|
||||
experimental: false
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
name: Electron packaging rehearsal
|
||||
|
||||
# Explicitly artifact-only: no tag, schedule, release, or publishing permission.
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: electron-rehearsal-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
package:
|
||||
runs-on: ${{ matrix.runner }}
|
||||
timeout-minutes: 60
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- runner: ubuntu-24.04
|
||||
platform: linux
|
||||
arch: x64
|
||||
target: x86_64-unknown-linux-gnu
|
||||
flags: --linux --x64
|
||||
- runner: windows-2022
|
||||
platform: win32
|
||||
arch: x64
|
||||
target: x86_64-pc-windows-msvc
|
||||
flags: --win --x64
|
||||
- runner: macos-15
|
||||
platform: darwin
|
||||
arch: arm64
|
||||
target: aarch64-apple-darwin
|
||||
flags: --mac --arm64
|
||||
- runner: macos-15-intel
|
||||
platform: darwin
|
||||
arch: x64
|
||||
target: x86_64-apple-darwin
|
||||
flags: --mac --x64
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
env:
|
||||
VOICESTUDIO_RUST_TARGET: ${{ matrix.target }}
|
||||
VOICESTUDIO_UPDATE_CHANNEL: electron-preview-${{ matrix.platform }}-${{ matrix.arch }}
|
||||
CSC_IDENTITY_AUTO_DISCOVERY: 'false'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
- uses: oven-sh/setup-bun@v2
|
||||
with:
|
||||
bun-version: '1.4.2'
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
with:
|
||||
targets: ${{ matrix.target }}
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
workspaces: native/desktop-bridge -> target
|
||||
key: electron-${{ matrix.target }}
|
||||
- uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: '0.12.13'
|
||||
enable-cache: false
|
||||
- name: Linux native dependencies
|
||||
if: runner.os == 'Linux'
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y libasound2-dev libxdo-dev libxtst-dev libx11-dev libxkbcommon-dev libwayland-dev libssl-dev pkg-config xvfb
|
||||
- name: Bundle pinned uv for the host architecture
|
||||
run: |
|
||||
node --input-type=module <<'NODE'
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { mkdirSync, copyFileSync, chmodSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
const expected = process.env.VOICESTUDIO_RUST_TARGET;
|
||||
const targets = { 'linux-x64': 'x86_64-unknown-linux-gnu', 'win32-x64': 'x86_64-pc-windows-msvc', 'darwin-arm64': 'aarch64-apple-darwin', 'darwin-x64': 'x86_64-apple-darwin' };
|
||||
if (targets[`${process.platform}-${process.arch}`] !== expected) throw new Error('Runner architecture does not match package target');
|
||||
const source = execFileSync(process.platform === 'win32' ? 'where.exe' : 'which', ['uv'], { encoding: 'utf8' }).trim().split(/\r?\n/)[0];
|
||||
const dir = 'frontend/src-tauri/binaries';
|
||||
mkdirSync(dir, { recursive: true });
|
||||
const destination = join(dir, `uv-${expected}${process.platform === 'win32' ? '.exe' : ''}`);
|
||||
copyFileSync(source, destination);
|
||||
if (process.platform !== 'win32') chmodSync(destination, 0o755);
|
||||
NODE
|
||||
- name: Install locked dependencies
|
||||
run: bun install --frozen-lockfile
|
||||
- name: Validate and build Electron
|
||||
run: bun run check:electron
|
||||
- name: Package without publishing
|
||||
working-directory: electron
|
||||
run: |
|
||||
bun x electron-builder --config electron-builder.config.mjs ${{ matrix.flags }} --publish never
|
||||
node tests/packaging-contract.mjs --artifact
|
||||
node tests/update-package-contract.mjs --platform ${{ matrix.platform }} --arch ${{ matrix.arch }}
|
||||
- name: Packaged startup smoke test
|
||||
working-directory: electron
|
||||
run: |
|
||||
if [ "$RUNNER_OS" = Linux ]; then
|
||||
xvfb-run -a node tests/packaged-smoke.mjs --setup
|
||||
else
|
||||
node tests/packaged-smoke.mjs --setup
|
||||
fi
|
||||
- name: Save installers and updater metadata for review
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: electron-rehearsal-${{ matrix.platform }}-${{ matrix.arch }}
|
||||
retention-days: 14
|
||||
if-no-files-found: error
|
||||
path: |
|
||||
electron/release/VoiceStudio-Electron-*
|
||||
electron/release/electron-*.yml
|
||||
@@ -0,0 +1,242 @@
|
||||
name: Electron desktop release
|
||||
|
||||
# Builds are safe by default. Only an explicit publish dispatch exposes a release.
|
||||
on:
|
||||
push:
|
||||
tags: ['v*']
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
release_tag:
|
||||
description: "Existing version tag to package using this workflow from main (optional)"
|
||||
type: string
|
||||
default: ''
|
||||
publish:
|
||||
description: "Publish the tagged Electron release after all platforms pass"
|
||||
type: boolean
|
||||
default: false
|
||||
allow_unsigned:
|
||||
description: "Explicitly accept unsigned/unnotarized Electron installers and documented updater limitations"
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: electron-release-${{ inputs.release_tag || github.ref_name }}
|
||||
cancel-in-progress: false
|
||||
|
||||
env:
|
||||
RELEASE_REF: ${{ inputs.release_tag && format('refs/tags/{0}', inputs.release_tag) || github.ref }}
|
||||
|
||||
jobs:
|
||||
validate:
|
||||
# The transition tag is assembled after the manual Tauri draft succeeds.
|
||||
if: github.event_name == 'workflow_dispatch' || github.ref_name != vars.TAURI_SUNSET_TAG
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ env.RELEASE_REF }}
|
||||
- name: Require an exact version tag
|
||||
env:
|
||||
REF: ${{ env.RELEASE_REF }}
|
||||
WORKFLOW_REF: ${{ github.ref }}
|
||||
RELEASE_TAG_OVERRIDE: ${{ inputs.release_tag }}
|
||||
ALLOW_UNSIGNED: ${{ inputs.allow_unsigned }}
|
||||
DISPATCH_ACTOR: ${{ github.actor }}
|
||||
RERUN_ACTOR: ${{ github.triggering_actor }}
|
||||
OWNER: ${{ github.repository_owner }}
|
||||
run: |
|
||||
if [ "$ALLOW_UNSIGNED" = true ]; then
|
||||
test "$DISPATCH_ACTOR" = "$OWNER" && test "$RERUN_ACTOR" = "$OWNER" || {
|
||||
echo "Only the repository owner may accept unsigned installers"; exit 1;
|
||||
}
|
||||
fi
|
||||
if [ -n "$RELEASE_TAG_OVERRIDE" ]; then
|
||||
test "$WORKFLOW_REF" = refs/heads/main || { echo "Tag overrides require the workflow from main"; exit 1; }
|
||||
fi
|
||||
VERSION=$(node -p "require('./frontend/package.json').version")
|
||||
test "$REF" = "refs/tags/v$VERSION" || { echo "Select the exact version tag"; exit 1; }
|
||||
test "$(git rev-parse HEAD)" = "$(git rev-parse "$REF^{commit}")" || { echo "Checkout does not match the release tag"; exit 1; }
|
||||
package:
|
||||
needs: validate
|
||||
runs-on: ${{ matrix.runner }}
|
||||
timeout-minutes: 60
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- runner: ubuntu-24.04
|
||||
platform: linux
|
||||
arch: x64
|
||||
target: x86_64-unknown-linux-gnu
|
||||
flags: --linux --x64
|
||||
- runner: windows-2022
|
||||
platform: win32
|
||||
arch: x64
|
||||
target: x86_64-pc-windows-msvc
|
||||
flags: --win --x64
|
||||
- runner: macos-15
|
||||
platform: darwin
|
||||
arch: arm64
|
||||
target: aarch64-apple-darwin
|
||||
flags: --mac --arm64
|
||||
- runner: macos-15-intel
|
||||
platform: darwin
|
||||
arch: x64
|
||||
target: x86_64-apple-darwin
|
||||
flags: --mac --x64
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
env:
|
||||
VOICESTUDIO_RUST_TARGET: ${{ matrix.target }}
|
||||
VOICESTUDIO_UPDATE_CHANNEL: electron-stable-${{ matrix.platform }}-${{ matrix.arch }}
|
||||
CSC_IDENTITY_AUTO_DISCOVERY: 'false'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ env.RELEASE_REF }}
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
- uses: oven-sh/setup-bun@v2
|
||||
with:
|
||||
bun-version: '1.4.2'
|
||||
- uses: dtolnay/rust-toolchain@6bed0761d98439e5a578e2877258200ad565ba87 # stable
|
||||
with:
|
||||
targets: ${{ matrix.target }}
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
workspaces: native/desktop-bridge -> target
|
||||
key: electron-${{ matrix.target }}
|
||||
- uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: '0.12.13'
|
||||
enable-cache: false
|
||||
- name: Linux native dependencies
|
||||
if: runner.os == 'Linux'
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y libasound2-dev libxdo-dev libxtst-dev libx11-dev libxkbcommon-dev libwayland-dev libssl-dev pkg-config xvfb
|
||||
- name: Bundle pinned uv for the host architecture
|
||||
run: |
|
||||
node --input-type=module <<'NODE'
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { mkdirSync, copyFileSync, chmodSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
const expected = process.env.VOICESTUDIO_RUST_TARGET;
|
||||
const targets = { 'linux-x64': 'x86_64-unknown-linux-gnu', 'win32-x64': 'x86_64-pc-windows-msvc', 'darwin-arm64': 'aarch64-apple-darwin', 'darwin-x64': 'x86_64-apple-darwin' };
|
||||
if (targets[`${process.platform}-${process.arch}`] !== expected) throw new Error('Runner architecture does not match package target');
|
||||
const source = execFileSync(process.platform === 'win32' ? 'where.exe' : 'which', ['uv'], { encoding: 'utf8' }).trim().split(/\r?\n/)[0];
|
||||
const dir = 'frontend/src-tauri/binaries';
|
||||
mkdirSync(dir, { recursive: true });
|
||||
const destination = join(dir, `uv-${expected}${process.platform === 'win32' ? '.exe' : ''}`);
|
||||
copyFileSync(source, destination);
|
||||
if (process.platform !== 'win32') chmodSync(destination, 0o755);
|
||||
NODE
|
||||
- name: Install locked dependencies
|
||||
run: bun install --frozen-lockfile
|
||||
- name: Validate and build Electron
|
||||
run: bun run check:electron
|
||||
- name: Package without publishing
|
||||
env:
|
||||
CSC_LINK: ${{ secrets.ELECTRON_CSC_LINK }}
|
||||
CSC_KEY_PASSWORD: ${{ secrets.ELECTRON_CSC_KEY_PASSWORD }}
|
||||
working-directory: electron
|
||||
run: |
|
||||
# An empty CSC_LINK is interpreted as the working directory by the
|
||||
# signer. Omit absent credentials rather than passing empty strings.
|
||||
if [ -z "${CSC_LINK:-}" ]; then
|
||||
unset CSC_LINK CSC_KEY_PASSWORD
|
||||
fi
|
||||
bun x electron-builder --config electron-builder.config.mjs ${{ matrix.flags }} --publish never
|
||||
node tests/packaging-contract.mjs --artifact
|
||||
node tests/update-package-contract.mjs --platform ${{ matrix.platform }} --arch ${{ matrix.arch }}
|
||||
- name: Verify macOS signing and notarization before publication
|
||||
if: inputs.publish == true && inputs.allow_unsigned != true && matrix.platform == 'darwin'
|
||||
run: |
|
||||
APP=$(find electron/release -maxdepth 2 -name VoiceStudio.app -type d -print -quit)
|
||||
test -n "$APP"
|
||||
codesign --verify --deep --strict "$APP"
|
||||
spctl --assess --type execute --verbose=2 "$APP"
|
||||
- name: Verify Windows installer signature before publication
|
||||
if: inputs.publish == true && inputs.allow_unsigned != true && matrix.platform == 'win32'
|
||||
shell: pwsh
|
||||
run: |
|
||||
$installers = @(Get-ChildItem electron/release/VoiceStudio-Electron-*.exe)
|
||||
if ($installers.Count -eq 0) { throw "No installer to verify" }
|
||||
foreach ($installer in $installers) {
|
||||
$signature = Get-AuthenticodeSignature $installer.FullName
|
||||
if ($signature.Status -ne 'Valid') { throw "Installer signature is not trusted: $($installer.Name)" }
|
||||
}
|
||||
- name: Packaged startup smoke test
|
||||
working-directory: electron
|
||||
run: |
|
||||
if [ "$RUNNER_OS" = Linux ]; then
|
||||
xvfb-run -a node tests/packaged-smoke.mjs --setup
|
||||
else
|
||||
node tests/packaged-smoke.mjs --setup
|
||||
fi
|
||||
- name: Save installers and updater metadata for review
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: electron-release-${{ matrix.platform }}-${{ matrix.arch }}
|
||||
retention-days: 14
|
||||
if-no-files-found: error
|
||||
path: |
|
||||
electron/release/VoiceStudio-Electron-*
|
||||
electron/release/electron-*.yml
|
||||
|
||||
release:
|
||||
needs: package
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: write
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
TAG: ${{ inputs.release_tag || github.ref_name }}
|
||||
SUNSET_TAG: ${{ vars.TAURI_SUNSET_TAG }}
|
||||
PUBLISH: ${{ inputs.publish }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ env.RELEASE_REF }}
|
||||
- uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: electron-release-*
|
||||
merge-multiple: true
|
||||
path: release-assets
|
||||
- name: Validate all platforms before creating a release
|
||||
run: |
|
||||
python3 scripts/prepare_electron_release.py --assets release-assets --tag "$TAG"
|
||||
- name: Preserve the final Tauri updater feeds
|
||||
run: |
|
||||
test -n "$SUNSET_TAG" || { echo "Set TAURI_SUNSET_TAG before releasing"; exit 1; }
|
||||
# The transition tag already holds its own final Tauri feeds.
|
||||
# Later releases carry copies pointing to the immutable sunset payloads.
|
||||
gh release download "$SUNSET_TAG" --pattern latest.json --dir release-assets
|
||||
gh release download "$SUNSET_TAG" --pattern latest-user.json --dir release-assets
|
||||
python3 scripts/prepare_electron_release.py --assets release-assets --tag "$TAG" --sunset-tag "$SUNSET_TAG"
|
||||
- name: Disclose explicitly accepted unsigned artifacts
|
||||
if: inputs.allow_unsigned == true
|
||||
run: |
|
||||
cat >> release-assets/RELEASE_NOTES.md <<'EOF'
|
||||
|
||||
### Electron installer trust
|
||||
These Electron installers are unsigned or ad-hoc signed and are not Apple-notarized.
|
||||
Windows/macOS may show trust warnings. macOS automatic updates are unverified;
|
||||
use manual installer updates. Tauri updater signatures remain independently verified.
|
||||
EOF
|
||||
- name: Create or update draft
|
||||
run: |
|
||||
if ! gh release view "$TAG" >/dev/null 2>&1; then
|
||||
gh release create "$TAG" --verify-tag --draft --title "$TAG — VoiceStudio" --notes-file release-assets/RELEASE_NOTES.md
|
||||
fi
|
||||
test "$(gh release view "$TAG" --json isDraft --jq .isDraft)" = true || { echo "Refusing to replace a published release"; exit 1; }
|
||||
gh release edit "$TAG" --notes-file release-assets/RELEASE_NOTES.md
|
||||
find release-assets -maxdepth 1 -type f ! -name RELEASE_NOTES.md -print0 | xargs -0 gh release upload "$TAG" --clobber
|
||||
- name: Publish only when explicitly requested
|
||||
if: github.event_name == 'workflow_dispatch' && inputs.publish == true
|
||||
run: gh release edit "$TAG" --draft=false --latest
|
||||
@@ -27,27 +27,22 @@
|
||||
# each to surface PyInstaller/Tauri issues that never showed up locally on
|
||||
# macOS — iterate on CI.
|
||||
|
||||
name: Desktop Release
|
||||
name: Tauri sunset (manual only)
|
||||
|
||||
# Legacy workflow: run once on the final Tauri version tag.
|
||||
# Electron releases are owned by electron-release.yml.
|
||||
on:
|
||||
push:
|
||||
tags: ['v*']
|
||||
schedule:
|
||||
# 07:00 UTC daily — rolling `preview` prerelease from `main`. The
|
||||
# preview-gate job no-ops the matrix when main hasn't moved in a day.
|
||||
- cron: '0 7 * * *'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
draft:
|
||||
description: "Create as draft release (tag push only)"
|
||||
required: false
|
||||
description: "Keep the final Tauri release draft until Electron artifacts are ready"
|
||||
default: "true"
|
||||
publish_preview:
|
||||
description: "Publish a rolling 'preview' prerelease (updater Preview channel). Previews ALWAYS build from main — dispatching from any other branch fails the preview-gate."
|
||||
required: false
|
||||
description: "Legacy compatibility input; previews are retired"
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
|
||||
permissions:
|
||||
contents: write # needed to attach artifacts + updater manifest to GH Release
|
||||
|
||||
@@ -76,6 +71,16 @@ jobs:
|
||||
name: Tests (backend + frontend)
|
||||
runs-on: ubuntu-22.04
|
||||
steps:
|
||||
- name: Require the designated final Tauri tag
|
||||
env:
|
||||
SUNSET_TAG: ${{ vars.TAURI_SUNSET_TAG }}
|
||||
REF: ${{ github.ref }}
|
||||
PREVIEW: ${{ inputs.publish_preview }}
|
||||
run: |
|
||||
test -n "$SUNSET_TAG" || { echo "Set TAURI_SUNSET_TAG to the final v* tag first"; exit 1; }
|
||||
test "$REF" = "refs/tags/$SUNSET_TAG"
|
||||
test "$PREVIEW" != "true"
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Python 3.11
|
||||
@@ -476,6 +481,9 @@ jobs:
|
||||
fi
|
||||
{
|
||||
echo 'body<<RELEASE_BODY_EOF'
|
||||
echo '## Final Tauri update'
|
||||
echo 'VoiceStudio desktop is moving to Electron. This is the last Tauri release. Back up your data and install Electron separately: https://github.com/debpalash/VoiceStudio/blob/main/docs/electron-migration.md'
|
||||
echo
|
||||
echo "$BODY"
|
||||
echo 'RELEASE_BODY_EOF'
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
@@ -667,6 +675,7 @@ jobs:
|
||||
includeUpdaterJson: true
|
||||
|
||||
- name: Build + publish Electron desktop
|
||||
if: false # Electron is released independently by electron-release.yml.
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
@@ -973,7 +982,7 @@ jobs:
|
||||
# Writes SHA256SUMS-<label>.txt, attached to the release below. The
|
||||
# release-notes-checksums job puts every leg's file into the notes.
|
||||
- name: Compute SHA-256 checksums
|
||||
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
|
||||
if: startsWith(github.ref, 'refs/tags/v') && (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
|
||||
id: checksums
|
||||
shell: bash
|
||||
run: |
|
||||
@@ -1032,7 +1041,7 @@ jobs:
|
||||
# decision, so both belong to the single release-notes-checksums job
|
||||
# that runs after the whole matrix (see there for why).
|
||||
- name: Attach SHA256SUMS file
|
||||
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
|
||||
if: startsWith(github.ref, 'refs/tags/v') && (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
@@ -1058,7 +1067,7 @@ jobs:
|
||||
# missing, so a failed platform leaves the release a draft.
|
||||
release-notes-checksums:
|
||||
needs: [build, repair-updater-manifest, uninstall-scripts]
|
||||
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
|
||||
if: startsWith(github.ref, 'refs/tags/v') && (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
|
||||
runs-on: ubuntu-22.04
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
@@ -1067,6 +1076,7 @@ jobs:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
REPO: ${{ github.repository }}
|
||||
TAG: ${{ github.ref_name }}
|
||||
KEEP_DRAFT: ${{ inputs.draft }}
|
||||
steps:
|
||||
- name: Write every platform's checksums into the notes, then publish
|
||||
shell: bash
|
||||
@@ -1091,7 +1101,9 @@ jobs:
|
||||
done
|
||||
[ "$missing" = 0 ] || exit 1
|
||||
gh release edit "$TAG" --repo "$REPO" --notes-file "$WORK/notes.md"
|
||||
if [[ "$TAG" == *-* ]]; then
|
||||
if [[ "$KEEP_DRAFT" == "true" ]]; then
|
||||
echo "Final Tauri draft verified; Electron publication owns the transition."
|
||||
elif [[ "$TAG" == *-* ]]; then
|
||||
gh release edit "$TAG" --repo "$REPO" --draft=false --prerelease
|
||||
else
|
||||
gh release edit "$TAG" --repo "$REPO" --draft=false --latest
|
||||
@@ -1121,9 +1133,7 @@ jobs:
|
||||
# can leave the in-app updater with four 404 feeds.
|
||||
electron-publish-contract:
|
||||
needs: [build, preview-gate]
|
||||
if: >-
|
||||
needs.build.result == 'success' &&
|
||||
(needs.preview-gate.outputs.is_preview == 'true' || startsWith(github.ref, 'refs/tags/v'))
|
||||
if: false # Electron release workflow owns this contract.
|
||||
runs-on: ubuntu-22.04
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -1200,7 +1210,7 @@ jobs:
|
||||
|
||||
uninstall-scripts:
|
||||
needs: [build]
|
||||
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
|
||||
if: startsWith(github.ref, 'refs/tags/v') && (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
|
||||
runs-on: ubuntu-22.04
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
@@ -10,6 +10,49 @@ the frozen-backend fallback mirror it for their toolchains.
|
||||
|
||||
**Highlights**
|
||||
|
||||
- More reliable engine installation, transcription, and sidecar recovery (#2165, #2109, #2111)
|
||||
- Preserve subtitle text and legacy manuscript encodings across desktop and web (#2077, #2151, #2073)
|
||||
- Clearer setup guidance and media failure diagnostics (#2166, #2167)
|
||||
|
||||
### Fixed
|
||||
|
||||
- Repair CTranslate2 loading safely across ASR and translation, and retain the loaded Whisper model during CPU fallback (#2165) — thanks @guruthechosen!
|
||||
- Avoid pedalboard wheels that crash on unsupported CPU instructions (#2080) — thanks @D3nii!
|
||||
- Include cuDNN 8 compatibility libraries for CTranslate2 in CUDA containers (#2072) — thanks @basil-k-aji-dev!
|
||||
- Preserve audio reads, writes, and reference amplitude without TorchCodec (#2083) — thanks @Moep90!
|
||||
- Give isolated engines request-sized deadlines, validate timeout overrides, and distinguish hangs from crashes (#2109) (#2111) — thanks @SurefireStudios and @LMGXENON!
|
||||
- Keep dubbing streams alive during quiet steps and delay model cleanup until native refinement ends (#2138) — thanks @denemon!
|
||||
- Locate ffprobe beside ffmpeg without changing parent directory names (#2107) — thanks @kapelame!
|
||||
- Resample MLX output chunks to the declared rate before joining them (#2106) — thanks @kapelame!
|
||||
- Read database migration configuration on Chinese, Japanese, and Korean Windows (#2075) — thanks @kevin9327!
|
||||
- Preserve milliseconds and carry rounded subtitle timestamps across second boundaries (#2074) — thanks @kevin9327!
|
||||
- Decode UTF-16 and Windows-1252 subtitle and manuscript imports in Electron, web, and backend routes (#2073) — thanks @kevin9327!
|
||||
- Preserve numeric subtitle dialogue while recognizing mixed indexed and unindexed cues (#2151) — thanks @shivsin25!
|
||||
- Parse pasted WebVTT cues while separating metadata, identifiers, empty cues, and complete timing lines (#2077) — thanks @kevin9327!
|
||||
- Normalize Argos language aliases without silently changing Traditional Chinese to Simplified (#2143, #2152) — thanks @gyanu2507 and @rollroyces!
|
||||
- Clarify Blackwell import-crash diagnostics without blaming missing kernels (#2084) — thanks @Moep90!
|
||||
- Distinguish architecture preflight rejection from independent compile-stack failures (#2085) — thanks @Moep90!
|
||||
- Require the pinned Apple Silicon GGUF build to pass and document runtime preflight conditions (#2115) — thanks @LMGXENON and @martinezpl!
|
||||
- Correct the Windows Rustup installation command in tooling and documentation (#2066) — thanks @Rukhaam!
|
||||
- Show local setup guidance when remote native engine installation is unavailable (#2166)
|
||||
- Show scrubbed native error tails and exit codes for failed dubbing extraction (#2167)
|
||||
|
||||
### CI
|
||||
|
||||
- Handle missing Electron signing credentials and retry packaging fixes without moving release tags (#2157)
|
||||
|
||||
## [0.5.3] — 2026-09-17
|
||||
|
||||
**Highlights**
|
||||
|
||||
- The README is shorter, with a new Electron UI tour and refreshed screenshots (#2129)
|
||||
|
||||
- Support pages feature cleaner donation cards, with a workspace support shortcut and sponsor footer with hover cards and email inquiries (#2129)
|
||||
|
||||
- Integrations has a dedicated sidebar workspace with featured sponsors, searchable AI providers, and smooth sponsor-strip scrolling (#2129)
|
||||
|
||||
- Integrations now covers 100+ automation, communications, MCP, agent, developer, data, and productivity tools with config-driven detail pages (#2129)
|
||||
|
||||
- Electron now ships as a complete cross-platform VoiceStudio desktop app with local-first cloning, production workspaces, model packs, repair agents, native integrations, updates, parity checks, and the shared backend contracts required by those workflows (#1823)
|
||||
|
||||
- The Model Catalogue is one page: what you use now on top, then each family's engines and weights (#2013)
|
||||
@@ -19,12 +62,38 @@ the frozen-backend fallback mirror it for their toolchains.
|
||||
|
||||
### Changed
|
||||
|
||||
- Electron becomes the default source desktop, with artifact-only packaging rehearsals and a separate final Tauri update path (#2157)
|
||||
- Installable agent skills use current VoiceStudio names and Electron workflows (#2157)
|
||||
- README clarifies the Electron transition while keeping desktop contributions welcome (#2153) — thanks @cyberspace-cs!
|
||||
|
||||
- Electron first run uses four simple steps with model packs, optional advanced controls and skippable dictation setup (#2129)
|
||||
|
||||
- Model Catalogue is one page: a setup summary (speech, transcription, dictation, language model) on top, one TTS / ASR / LLM switch, and each family's downloadable weights listed under its engines; the separate Models pane and the Settings → Voice → Engines / Models signposts are gone, the models directory and voice previews moved to Settings → Storage and the HF mirror to Network (#2013)
|
||||
- The engine list is one line per engine (engine, device it runs on, status, one action) with a detail panel for everything else; each engine's weights install from its panel, so the separate weights list and recommendation card are gone (#2020)
|
||||
- CosyVoice 3 installs patched protobuf and transformers releases, clearing five security advisories (#2030, #2031)
|
||||
|
||||
### Fixed
|
||||
|
||||
- Keep demo playback aligned across languages, preserve worker GPU metrics, and restrict unsigned releases to owner dispatches (#2157)
|
||||
|
||||
- Desktop integration checks cover current dubbing safeguards, navigation, and the linked engine catalog (#2157)
|
||||
|
||||
- Tauri and Electron now share native dictation, watch-folder, and Wayland shortcut contracts; focused paste stays ordered and first-run uv stays pinned at 0.12.13 (#2122)
|
||||
- Dubbing demos synchronize playheads without simultaneous playback and let you open a sample in the editor (#2131)
|
||||
- macOS desktop sidebar clears the traffic lights, uses a narrower collapsed rail, and places notifications and device controls with more space (#2126)
|
||||
- Dubbing timelines keep short segments proportional, support zoom, and remove timestamp-confirmed duplicate ASR context (#2129)
|
||||
|
||||
- Dubbing translation shares the agent footer with live logs, validated output, cancellation and contextual retries (#2129)
|
||||
|
||||
- Agent dubbing translation saves a custom tone and adaptation prompt and preserves it during timing rewrites (#2129)
|
||||
|
||||
- Dubbing preserves original sound outside dialogue and mixes separated background only beneath replacement speech (#2129)
|
||||
|
||||
- Dubbing repairs missing speech caches, rejects incomplete output, avoids oversized speaker references, and fits full speech without early clipping (#2129)
|
||||
- Workspace sidebars have a working right-edge resize handle, allow 40% more width, remember their size, and keep video controls inside the preview (#2129)
|
||||
- Pressing Play while a video is loading starts playback when it is ready instead of reporting playback unavailable (#2129)
|
||||
- Video previews show their thumbnail before playback, including the source video in Dub (#2129)
|
||||
- Linux and Windows workspace headers consistently expand and collapse the sidebar, with the app logo at the top of the collapsed rail (#2129)
|
||||
- Stopping a process on macOS no longer fails with "Operation not permitted" when it was already exiting (#2032)
|
||||
- A YouTube link blocked by its "not a bot" check now says how to attach signed-in cookies in Dub, instead of quoting yt-dlp's command-line flags (#2036, #2034)
|
||||
- An engine that fails to start now says whether it timed out, crashed (with its exit code and last output) or answered wrongly, instead of "did not signal ready: None" (#2037, #2026)
|
||||
@@ -32,6 +101,10 @@ the frozen-backend fallback mirror it for their toolchains.
|
||||
- PyTorch Whisper runs on 6 GB NVIDIA cards instead of falling back to CPU, because its memory check now fits the model it loads (#2044, #2041)
|
||||
- MCP tools wait as long as the backend does, so a long transcription no longer fails at 120 s with an empty error (#2043, #2040)
|
||||
- Installing a gated model now sends your Hugging Face token on the fast download path too, so pyannote diarisation and gated engine weights stop failing with "401 Unauthorized" when the token is saved in Settings (#2173, #2163)
|
||||
- Generating on an older NVIDIA GPU (Tesla T4, and other pre-Ampere cards) no longer kills the backend on the first request — CUDA graphs are not captured below sm_80 (#2135)
|
||||
- "Disable torch.compile" in Settings → Performance now works on macOS and Linux, not only Windows; it was greyed out on the platforms that needed it (#2135)
|
||||
- Setting `TORCH_COMPILE_DISABLE=1` in the environment now actually disables torch.compile, for the in-process engine and engine subprocesses alike (#2135)
|
||||
- A backend killed by a native crash now leaves the faulting thread's stack in `backend_err.log` instead of exiting silently (#2135)
|
||||
|
||||
### CI
|
||||
|
||||
@@ -130,6 +203,8 @@ the frozen-backend fallback mirror it for their toolchains.
|
||||
|
||||
### Added
|
||||
|
||||
- Remote-worker metrics distinguish unavailable readings from zero and keep probes off the control loop (#2155)
|
||||
|
||||
- The audiobook result is now a synced-lyrics player: chapter text follows playback with the current word highlighted and click-to-seek, timed from the render's own chapter durations with a karaoke-style even split — no ASR pass, fully local (#1766) — thanks @mvanhorn!
|
||||
- The dub CAST strip expands into a project-level casting board: drag voice chips (clone profiles, design presets, Default) onto speaker rows — or pick from a keyboard listbox — writing the same per-speaker cast fields as the existing dropdowns (#1767) — thanks @mvanhorn!
|
||||
- Studio's new Convert method turns a dropped or recorded clip into an existing voice profile's voice, with optional source-duration matching (#1765) — thanks @mvanhorn!
|
||||
|
||||
@@ -1,502 +1,101 @@
|
||||
<div align="center">
|
||||
|
||||
<h3>NOTE: Electron Rewrite Ongoing: Please dont't create desktop app related issues and pr</h3>
|
||||
|
||||
<p><img src="docs/logo.png" alt="VoiceStudio logo" width="120" height="120" /></p>
|
||||
<img src="docs/logo.png" alt="VoiceStudio" width="88" />
|
||||
<h1>VoiceStudio</h1>
|
||||
<p>
|
||||
<a href="https://trendshift.io/repositories/28176?utm_source=repository-badge&utm_medium=badge&utm_campaign=badge-repository-28176" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/repositories/28176" alt="VoiceStudio ranking on Trendshift" width="220" height="48" /></a>
|
||||
</p>
|
||||
<p><sub>Previously OmniVoice-Studio</sub></p>
|
||||
<h3>Clone voices, dub video, dictate, and produce long-form audio on your own hardware.</h3>
|
||||
<p>16 TTS engines · 11 ASR engines · 646-language catalogue · macOS, Windows, Linux, and Docker</p>
|
||||
<p>No account, API key, subscription, or usage meter for the local workflow.</p>
|
||||
|
||||
<p><strong>Open source voice cloning and workflow engine. Build local.</strong></p>
|
||||
<p>
|
||||
<a href="#install">Install</a> ·
|
||||
<a href="#features">Features</a> ·
|
||||
<a href="#comparison">Compare</a> ·
|
||||
<a href="#requirements">Requirements</a> ·
|
||||
<a href="#hardware-recommendations">Hardware</a> ·
|
||||
<a href="#engines">Engines</a> ·
|
||||
<a href="#architecture">Architecture</a> ·
|
||||
<a href="#api">API</a> ·
|
||||
<a href="https://voicestudio.sh/?utm_source=github&utm_medium=readme&utm_campaign=project">Website</a> ·
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest">Download</a> ·
|
||||
<a href="#get-started">Get started</a> ·
|
||||
<a href="#documentation">Docs</a> ·
|
||||
<a href="#faq">FAQ</a> ·
|
||||
<a href="README_CN.md"><strong>简体中文</strong></a>
|
||||
<a href="https://discord.gg/bzQavDfVV9">Discord</a> ·
|
||||
<a href="README_CN.md">简体中文</a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/debpalash/VoiceStudio/ci.yml?branch=main&style=flat-square&label=CI" alt="CI status" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/stargazers"><img src="https://img.shields.io/github/stars/debpalash/VoiceStudio?style=flat-square&color=f59e0b" alt="GitHub stars" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases"><img src="https://img.shields.io/github/downloads/debpalash/VoiceStudio/total?style=flat-square&color=8b5cf6&label=downloads" alt="Total downloads" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/VoiceStudio?style=flat-square&color=10b981" alt="Latest release" /></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="AGPL-3.0 license" /></a>
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Discord-Community-5865F2?style=flat-square&logo=discord&logoColor=white" alt="Discord community" /></a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/badge/Download-macOS_·_Windows_·_Linux-10b981?style=for-the-badge" alt="Download VoiceStudio" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/debpalash/VoiceStudio/ci.yml?branch=main" alt="CI" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/VoiceStudio" alt="Latest release" /></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue" alt="AGPL-3.0" /></a>
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div align="center">
|
||||
<img src="docs/media/0.5.0/quick-switch.gif" alt="Switching TTS engines from the VoiceStudio status bar" width="100%" />
|
||||
</div>
|
||||
|
||||
|
||||
> [!WARNING]
|
||||
> **Active beta.** Use the [latest release](https://github.com/debpalash/VoiceStudio/releases/latest) for stable work. `main` contains the newest fixes and may change between releases. Report problems through [GitHub Issues](https://github.com/debpalash/VoiceStudio/issues).
|
||||
## Your voice. Your workflow.
|
||||
|
||||
## At a glance
|
||||
| Create | Produce | Connect |
|
||||
| :--- | :--- | :--- |
|
||||
| Clone a voice or design your own | Dub videos with timed speech | Local API & MCP for agents |
|
||||
| Dictate with a floating widget | Stories, audiobooks & batch jobs | Optional remote workers |
|
||||
|
||||
| | VoiceStudio |
|
||||
|---|---|
|
||||
| **Workflows** | Voice cloning and design, video dubbing, dictation, stories, audiobooks, batch generation |
|
||||
| **Language catalogue** | 646 TTS languages; actual coverage and quality depend on the selected engine |
|
||||
| **Engines** | 16 TTS · 11 ASR · switch in Model Catalogue or with <kbd>Ctrl</kbd>/<kbd>Cmd</kbd>+<kbd>E</kbd> |
|
||||
| **Platforms** | macOS 13.3+ on Apple Silicon · Windows 10/11 x64 · Linux x86_64 with glibc 2.39+ |
|
||||
| **Compute** | CUDA · Apple Silicon MPS/MLX · ROCm on Linux · CPU · optional remote workers |
|
||||
| **Interfaces** | Desktop app · local REST/SSE/WebSocket API · OpenAI-compatible audio API · MCP Server |
|
||||
| **Storage** | Voices, projects, settings, and outputs stay on the machine by default |
|
||||
| **License** | AGPL-3.0 application; downloaded models keep their upstream terms |
|
||||
Start with **VoiceStudio** (default, powered by k2-fsa/OmniVoice), or choose another engine. [Features & engine catalog](docs/feature-catalog.md).
|
||||
|
||||
The Voice workspace starts with three tabs: **From audio** for cloning, **By design** for creating a voice, and **Convert** for speech-to-speech conversion. Each tab displays its own workflow, with Synthesize Audio or Convert pinned below the scrolling form. The top-bar **Engines** panel combines engine selection, loaded models, and unload/flush controls; <kbd>Ctrl</kbd>/<kbd>Cmd</kbd>+<kbd>E</kbd> opens it. The searchable language picker shares Dubbing’s flags and language list layout, selects one output language, and retains Auto and the full cloning catalogue. Language options flow into multiple columns when space allows. Expand **Workspaces** in the sidebar to reveal navigation labels; Escape collapses it.
|
||||
Local workflows run on your hardware. Remote services are optional; usage analytics requires consent.
|
||||
|
||||
Dubbing starts with file upload or URL import and nearby language choices. Its **Projects** panel lists previous dubs so they can be reopened by clicking anywhere on a card; action buttons operate independently. Advanced import options include captions and optional YouTube sign-in. Dubbing places playback controls over the video with background blur and combines the waveform and timed transcript in one compact editing surface. Drag the zoomed waveform left or right to pan; click to seek. Translation language and ISO-code controls stay synchronized; Auto clears any previous language code and dialect. Transcript items group editable text, timing and status, and voice controls into three readable rows that wrap with the panel width. Output Options stays compact with the active settings shown in its summary; expand it to change output, timing, or voice matching. Transcript, glossary, and paste controls share a toolbar above the segment editor. Project details, workflow steps, and Generate/Verify/Export actions use an unfilled header.
|
||||
<details>
|
||||
<summary><strong>Explore the workspaces</strong> · Clone, dub, design & models</summary>
|
||||
|
||||
The Audiobook Script editor fills the available workspace beneath its markup toolbar; Voices and Book settings stay in their own tabs.
|
||||
<table>
|
||||
<tr>
|
||||
<td><img src="docs/media/electron/voice-cloning.png" alt="Electron voice cloning workspace with the bundled demo voice" width="100%" /></td>
|
||||
<td><img src="docs/media/electron/dubbing.png" alt="Electron video dubbing workspace" width="100%" /></td>
|
||||
</tr>
|
||||
<tr><td align="center">Voice cloning</td><td align="center">Video dubbing</td></tr>
|
||||
<tr>
|
||||
<td><img src="docs/media/electron/voice-design.png" alt="Describe a voice in the Electron voice design workspace" width="100%" /></td>
|
||||
<td><img src="docs/media/electron/models.png" alt="Install and manage local speech models" width="100%" /></td>
|
||||
</tr>
|
||||
<tr><td align="center">Voice design</td><td align="center">Local models</td></tr>
|
||||
</table>
|
||||
|
||||
Output settings use aligned rows; review status appears before the collapsible transcript and glossary. Glossary terms have labelled entry fields and an explicit edit action. Launchpad arranges recent files and saved voices side by side when space allows, with responsive card grids and visible Open actions.
|
||||
<img width="2628" height="1950" alt="VoiceStudio desktop workspace" src="https://github.com/user-attachments/assets/b474497d-a453-49a3-a2dd-f023ec6b7659" />
|
||||
|
||||
The casting board shows icon-based voice cards and searchable selectors for each speaker. Drag a card onto a speaker or choose a voice from that speaker’s menu.
|
||||
</details>
|
||||
|
||||
<a id="install"></a>
|
||||
## Get started
|
||||
|
||||
## Install
|
||||
Download from [Releases](https://github.com/debpalash/VoiceStudio/releases/latest), then follow your platform guide:
|
||||
|
||||
Download a package from the [latest release](https://github.com/debpalash/VoiceStudio/releases/latest), then follow the platform guide.
|
||||
**[macOS](docs/install/macos.md) · [Windows](docs/install/windows.md) · [Linux](docs/install/linux.md) · [Docker](docs/install/docker.md)**
|
||||
|
||||
| Platform | Package | Guide |
|
||||
|---|---|---|
|
||||
| macOS 13.3+ | Apple Silicon DMG | [Install on macOS](docs/install/macos.md) |
|
||||
| Windows 10/11 | x64 MSI; choose the current-user build when listed to install without admin access | [Install on Windows](docs/install/windows.md#install-pre-built-msi) |
|
||||
| Linux | AppImage, x86_64 with glibc 2.39+ | [Install on Linux](docs/install/linux.md) |
|
||||
| Docker | Linux/AMD64 images; CUDA, ROCm, CPU, and worker-only GPU profiles | [Run with Docker](docs/install/docker.md) |
|
||||
Open **Voice cloning**, choose a voice or add a clean reference recording, enter your text, and generate. Install the required model when prompted. Hardware needs vary by engine; see [performance](docs/performance.md).
|
||||
|
||||
First launch creates a managed Python environment and downloads the default model. Later launches reuse both.
|
||||
|
||||
> [!NOTE]
|
||||
> On macOS, first launch needs a one-time right-click, then **Open** approval. Intel Macs cannot run the local Python backend; use a [remote backend](docs/install/macos.md) instead.
|
||||
|
||||
### Quick Docker run
|
||||
|
||||
The published images are **`linux/amd64` only**. On Apple Silicon, use the
|
||||
[native macOS app](docs/install/macos.md) for GPU acceleration. ARM64 hosts
|
||||
should read the [architecture requirements](docs/install/docker.md#architecture)
|
||||
before pulling an image.
|
||||
|
||||
```bash
|
||||
docker run -d -p 127.0.0.1:3900:3900 -v omnivoice-data:/app/omnivoice_data --name voicestudio palashdeb/omnivoice-studio:stable
|
||||
```
|
||||
|
||||
### First voice
|
||||
|
||||
1. Launch VoiceStudio and open **Voice Cloning**.
|
||||
2. Add a clean voice sample. Three seconds works; 5 to 15 seconds usually gives a better prompt.
|
||||
3. Enter text, choose a language, then select **Generate**.
|
||||
|
||||
> [!TIP]
|
||||
> **Try without installing:** Run VoiceStudio in the cloud via the [Google Colab notebook](https://colab.research.google.com/github/debpalash/VoiceStudio/blob/main/notebooks/OmniVoice_Studio_Colab.ipynb). Explore audio quality comparisons in [benchmarks](docs/benchmarks.md) and prompt design tips in [expressive speech](docs/expressive-speech.md).
|
||||
|
||||
### Audio samples
|
||||
|
||||
Listen to sample outputs produced locally with VoiceStudio:
|
||||
|
||||
| Workflow | Prompt / Reference Audio | Generated Audio |
|
||||
|---|---|---|
|
||||
| **Voice Cloning** | [demo_voice.wav](backend/assets/samples/demo_voice.wav) | [demo_clone_output.wav](backend/assets/samples/demo_clone_output.wav) |
|
||||
| **Voice Design** (US News Anchor) | *"Clear, authoritative American broadcast tone"* | [demo_voice_design_us_news_anchor.wav](backend/assets/samples/voice_design/demo_voice_design_us_news_anchor.wav) |
|
||||
| **Voice Design** (UK Audiobook) | *"Warm, expressive British storytelling voice"* | [demo_voice_design_audiobook_uk_narrator.wav](backend/assets/samples/voice_design/demo_voice_design_audiobook_uk_narrator.wav) |
|
||||
| **Video Dubbing** (Multilingual) | [source.src.wav](backend/assets/samples/demo/dubbing/source.src.wav) | [Spanish](backend/assets/samples/demo/dubbing/dubbed_es.src.wav) · [French](backend/assets/samples/demo/dubbing/dubbed_fr.src.wav) · [Japanese](backend/assets/samples/demo/dubbing/dubbed_ja.src.wav) · [Chinese](backend/assets/samples/demo/dubbing/dubbed_zh.src.wav) |
|
||||
|
||||
### Run from source
|
||||
|
||||
Install the [development prerequisites](.github/CONTRIBUTING.md#development-setup) (Node 20+/Bun and Python 3.11+), then:
|
||||
<details>
|
||||
<summary><strong>Run the Electron preview from source</strong></summary>
|
||||
|
||||
```bash
|
||||
git clone https://github.com/debpalash/VoiceStudio.git
|
||||
cd VoiceStudio
|
||||
bun install
|
||||
bun run desktop
|
||||
bun run dev
|
||||
```
|
||||
|
||||
The desktop launcher configures Python dependencies on first run via `uv` automatically. Use `bun run dev` for the browser UI. See [Contributing](.github/CONTRIBUTING.md) for services, tests, and platform packages.
|
||||
See [Electron setup](electron/README.md) for prerequisites and backend configuration.
|
||||
|
||||
### If setup fails
|
||||
</details>
|
||||
|
||||
- Run **Settings → About → Run self-check** or `uv run python backend/main.py --diagnose --deep`.
|
||||
- Check [install troubleshooting](docs/install/troubleshooting.md).
|
||||
- Save a scrubbed diagnostic bundle from the app when opening an issue.
|
||||
- For slow generation, compare [measured benchmarks](docs/benchmarks.md) and [performance settings](docs/performance.md).
|
||||
|
||||
<a id="features"></a>
|
||||
|
||||
## Features
|
||||
|
||||
| Area | Included |
|
||||
|---|---|
|
||||
| **Voice Cloning** | Zero-shot synthesis from a short reference clip ([guide](docs/engines/README.md)) |
|
||||
| **Voice Design** | Create a voice from age, accent, pitch, style, and delivery instructions ([expressive speech](docs/expressive-speech.md)) |
|
||||
| **Video Dubbing** | Transcribe, translate, preserve speakers, synthesize, and export video; compact translation settings include track selection, and completed dubs flag timing issues for review ([export guide](docs/dubbing/export.md)) |
|
||||
| **Stories and audiobooks** | Multi-voice scripts · EPUB/PDF import · chapter rendering · `.m4b` export |
|
||||
| **[Dictation Widget](docs/features/dictation.md)** | System-wide shortcut, live transcription, optional local-LLM cleanup |
|
||||
| **Vocal Isolation** | Demucs speech/background separation |
|
||||
| **Speaker Diarization** | Pyannote and WhisperX speaker assignment ([guide](docs/features/diarization.md)) |
|
||||
| **Batch Queue** | Queue large sets of audio and video jobs with per-job progress, or watch a local folder for new videos |
|
||||
| **Model Catalogue** | Install, remove, select, and route TTS, ASR, and LLM models ([catalogue](docs/engines/README.md)) |
|
||||
| **Remote Model Downloads** | Install models on enrolled remote workers with live progress ([guide](docs/downloading-models.md)) |
|
||||
| **GPU Auto-Detect** | CUDA, MPS, ROCm, and CPU routing with per-engine checks ([performance](docs/performance.md)) |
|
||||
| **AI Watermark** | AudioSeal embedding and detection |
|
||||
| **MCP Server** | Synthesis and transcription tools for MCP clients ([guide](docs/mcp.md)) |
|
||||
| **Diagnostics** | Self-checks, error journal, logs, and scrubbed support bundles ([troubleshooting](docs/install/troubleshooting.md)) |
|
||||
| **Local-first** | Core creation stays local; network-backed features are explicit opt-ins |
|
||||
| **Extensible** | Registry-based TTS, ASR, and plugin interfaces ([acceptance](docs/engine-acceptance.md)) |
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td width="50%"><img src="docs/media/0.5.0/catalogue.png" alt="VoiceStudio Model Catalogue" width="100%" /></td>
|
||||
<td width="50%"><img src="docs/media/0.5.0/gallery-save.png" alt="Saving a gallery voice as a local profile" width="100%" /></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center"><sub>Model Catalogue: engine, device, and install state</sub></td>
|
||||
<td align="center"><sub>Gallery: save a shared voice as a local profile</sub></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
<a id="comparison"></a>
|
||||
|
||||
## Comparison
|
||||
|
||||
VoiceStudio trades managed cloud compute for local control. This is the practical difference:
|
||||
|
||||
| | **VoiceStudio** | **Typical hosted voice service** |
|
||||
|---|---|---|
|
||||
| **Best fit** | Private, offline, self-hosted, or high-volume work | Fast setup without local model management |
|
||||
| **Data path** | Local by default; remote features are opt-in | Audio and text are processed by the provider |
|
||||
| **Cost model** | Free software; you supply the hardware | Subscription, credits, or metered API use |
|
||||
| **Setup** | Install the app and model weights | Create an account and use the web app or API |
|
||||
| **Performance** | Depends on your engine and hardware | Provider manages compute and scaling |
|
||||
| **Offline use** | Yes, after required models are installed | Usually requires a network connection |
|
||||
| **Customization** | Source, engines, models, API, and routing are open | Limited to provider options |
|
||||
| **Maintenance** | You manage updates, disk, and compute | Provider manages infrastructure |
|
||||
|
||||
<a id="requirements"></a>
|
||||
|
||||
## Requirements
|
||||
|
||||
Requirements vary by engine. These values cover the default local workflow.
|
||||
|
||||
| | **Minimum** | **Recommended** |
|
||||
|---|---|---|
|
||||
| **OS** | Windows 10 x64 · macOS 13.3 Apple Silicon · Linux x86_64 with glibc 2.39+ | Current supported OS release |
|
||||
| **RAM** | 8 GB | 16 GB+ |
|
||||
| **Disk** | 10 GB free | 20 GB+ SSD |
|
||||
| **GPU** | Optional; CPU mode is supported | NVIDIA CUDA or Apple Silicon |
|
||||
| **VRAM** | 4 GB when using a GPU | 8 GB+; large optional engines need more |
|
||||
| **Python from source** | 3.11+ | 3.11 or 3.12 |
|
||||
|
||||
ROCm is Linux-only and opt-in. Windows AMD/Ryzen AI uses CPU. Systems with limited VRAM offload work to CPU when required. See [performance](docs/performance.md), [benchmarks](docs/benchmarks.md), and [engine disk usage](docs/engines/disk-usage.md).
|
||||
|
||||
<a id="hardware-recommendations"></a>
|
||||
|
||||
### Recommended stack by hardware
|
||||
|
||||
| Hardware | Recommended TTS | Recommended ASR | Why |
|
||||
|---|---|---|---|
|
||||
| **Apple Silicon (M1–M4)** | [MLX-Audio](docs/engines/mlx-audio.md) · [OmniVoice](docs/engines/omnivoice.md) (MPS) | [MLX Whisper](docs/engines/mlx-whisper.md) · [Parakeet MLX](docs/engines/parakeet-mlx.md) | Native unified memory, lowest latency on macOS |
|
||||
| **NVIDIA GPU (8 GB+ VRAM)** | [OmniVoice](docs/engines/omnivoice.md) · [CosyVoice 3](docs/engines/cosyvoice.md) | [WhisperX](docs/engines/whisperx.md) | High-fidelity zero-shot cloning, word timestamps, diarization |
|
||||
| **Low VRAM / CPU-only** | [PocketTTS](docs/engines/pockettts.md) · [Sherpa-ONNX](docs/engines/sherpa-onnx.md) · [KittenTTS](docs/engines/kittentts.md) | [Moonshine](docs/engines/moonshine.md) · [Faster-Whisper](docs/engines/faster-whisper.md) (`int8`) | Low memory footprint, optimized CPU inference |
|
||||
|
||||
<a id="engines"></a>
|
||||
|
||||
## Engines
|
||||
|
||||
Engine support is capability-specific. Check cloning, language, platform, memory, and license before choosing one. Full setup guides: [docs/engines](docs/engines/README.md).
|
||||
|
||||
<a id="tts-engines"></a>
|
||||
|
||||
### Text to speech
|
||||
|
||||
| Engine | Languages | Clone | Instruct | Linux | macOS ARM | Windows | License |
|
||||
|---|:---:|:---:|:---:|:---:|:---:|:---:|---|
|
||||
| [**VoiceStudio** (default, powered by k2-fsa/OmniVoice)](docs/engines/omnivoice.md) | 600+ | Yes | Yes | CUDA/CPU | MPS | CUDA/CPU | [AGPL-3.0](LICENSE) app · [Apache-2.0 code, CC-BY-NC weights](https://huggingface.co/k2-fsa/OmniVoice#license)³ |
|
||||
| [**CosyVoice 3**](docs/engines/cosyvoice.md) | 9 + 18 dialects | Yes | Yes | CUDA/CPU | CPU | CUDA/CPU | Apache-2.0 |
|
||||
| [**GPT-SoVITS**](docs/engines/gpt-sovits.md) | 5 | Yes | No | CUDA/CPU | No | CUDA/CPU | MIT |
|
||||
| [**VoxCPM2**](docs/engines/voxcpm2.md) | 30 | Yes | Yes | CUDA/CPU | MPS | CUDA/CPU | Apache-2.0 |
|
||||
| [**MOSS-TTS-Nano**](docs/engines/moss-tts-nano.md) | 20 | Yes | No | CUDA/CPU | CPU | CUDA/CPU | Apache-2.0 |
|
||||
| [**KittenTTS**](docs/engines/kittentts.md) | English | No | No | CPU | CPU | CPU | MIT |
|
||||
| [**MLX-Audio**](docs/engines/mlx-audio.md) | Model-dependent | Varies | Varies | No | MLX | No | Varies |
|
||||
| [**Sherpa-ONNX**](docs/engines/sherpa-onnx.md) | 20+ | No | No | CUDA/CPU | CPU | CUDA/CPU | Apache-2.0 |
|
||||
| [**IndexTTS 2.5** ⚡](docs/engines/indextts.md) | ZH · EN · JA · ES · AR | Yes | No | CUDA/CPU | CPU | CUDA/CPU | Bilibili model license¹ |
|
||||
| [**OmniVoice GGUF** ⚡](docs/engines/omnivoice-gguf.md) | 600+ | Yes | Yes | CUDA/CPU | MPS/CPU | CUDA/CPU | [AGPL-3.0](LICENSE) app · [review the derivative model terms](https://huggingface.co/Serveurperso/OmniVoice-GGUF#license)³ |
|
||||
| [**OmniVoice (subprocess; opt-in off MPS)** ⚡](docs/engines/omnivoice-subprocess.md) | 600+ | Yes | Yes | CUDA/CPU | MPS via default OmniVoice | CUDA/CPU | [AGPL-3.0](LICENSE) app · [Apache-2.0 code, CC-BY-NC weights](https://huggingface.co/k2-fsa/OmniVoice#license)³ |
|
||||
| [**PocketTTS** ⚡](docs/engines/pockettts.md) | EN · FR · DE · PT · IT · ES | Yes | No | CPU | CPU | CPU | CC-BY-4.0, gated² |
|
||||
| [**Supertonic 3** ⚡](docs/engines/supertonic3.md) | 31 | No | No | CPU | CPU | CPU | OpenRAIL-M |
|
||||
| [**MOSS-TTS-v1.5** ⚡](docs/engines/moss-tts-v15.md) | 31 | Yes | No | CUDA/CPU | CPU | CUDA/CPU | Apache-2.0 |
|
||||
| [**dots.tts** ⚡](docs/engines/dots-tts.md) | 24 | Yes | No | CUDA/CPU | CPU | No | Apache-2.0 |
|
||||
| [**Confucius4-TTS** ⚡](docs/engines/confucius4-tts.md) | 14 | Yes | No | CUDA/CPU | CPU | CUDA/CPU | Apache-2.0 |
|
||||
|
||||
⚡ Installed or registered on demand.
|
||||
|
||||
¹ IndexTTS 2.5 requires a separate written Bilibili license above 100 million monthly active users or RMB 1 billion annual revenue. Review the [model license](https://huggingface.co/IndexTeam/IndexTTS-2.5/blob/main/LICENSE).
|
||||
|
||||
² PocketTTS shows its gated-access and CC-BY-4.0 terms before first use.
|
||||
|
||||
³ The OmniVoice snapshot also includes an audio tokenizer under separate [Boson Higgs Audio 2 and Meta Llama community terms](https://huggingface.co/k2-fsa/OmniVoice/blob/main/audio_tokenizer/LICENSE). VoiceStudio's application license does not replace model or tokenizer terms.
|
||||
|
||||
Clone-less engines cannot preserve a reference speaker in dubbing or pinned-voice batch jobs. VoiceStudio rejects those jobs instead of silently changing engines. Heavy engines have separate memory and platform limits; check their engine guide first.
|
||||
|
||||
<a id="asr-engines"></a>
|
||||
|
||||
### Speech to text
|
||||
|
||||
| Engine | ID | Languages | Best fit |
|
||||
|---|---|:---:|---|
|
||||
| [**WhisperX** (default)](docs/engines/whisperx.md) | `whisperx` | ~100 | Dubbing, subtitles, word-level timing |
|
||||
| [**Faster-Whisper**](docs/engines/faster-whisper.md) | `faster-whisper` | ~100 | General cross-platform transcription |
|
||||
| [**Faster-Whisper (isolated)**](docs/engines/faster-whisper-isolated.md) | `faster-whisper-isolated` | ~100 | Crash-isolated batch transcription |
|
||||
| [**MLX Whisper**](docs/engines/mlx-whisper.md) | `mlx-whisper` | ~100 | Apple Silicon |
|
||||
| [**PyTorch Whisper**](docs/engines/pytorch-whisper.md) | `pytorch-whisper` | ~100 | CUDA, MPS, and CPU fallback |
|
||||
| [**Parakeet TDT**](docs/engines/nemo-parakeet.md) | `nemo-parakeet` | English + 25 EU | Fast CPU/CUDA transcription |
|
||||
| [**Parakeet TDT v3 (MLX)**](docs/engines/parakeet-mlx.md) | `parakeet-mlx` | 25 EU | Apple Silicon dictation and word timestamps |
|
||||
| [**Moonshine**](docs/engines/moonshine.md) | `moonshine` | English | Low-power, low-latency ONNX |
|
||||
| [**FunASR**](docs/engines/funasr.md) | `funasr` | 50+ | VAD and inline diarization |
|
||||
| [**sherpa-onnx** (live dictation)](docs/engines/sherpa-onnx-asr.md) | `sherpa-onnx-asr` | Model-dependent | Streaming CPU dictation |
|
||||
| [**OpenAI-compatible** ⚠️ configured server](docs/engines/openai-compatible-asr.md) | `openai-compat-asr` | Server-dependent | Local gigastt/Qwen3-ASR or a remote endpoint; audio goes only to that server |
|
||||
|
||||
WhisperX and Faster-Whisper retry with `int8` when efficient `float16` is unavailable. Pin `ASR_COMPUTE_TYPE=int8` or `float32` only if automatic selection still fails.
|
||||
|
||||
<a id="architecture"></a>
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
Tauri v2 desktop shell (Rust)
|
||||
│ IPC
|
||||
React + Vite UI
|
||||
│ HTTP · SSE · WebSocket on localhost:3900
|
||||
FastAPI backend
|
||||
├── TTS / ASR engine registries
|
||||
├── dubbing / audio / long-form pipelines
|
||||
├── OpenAI-compatible API and MCP server
|
||||
└── SQLite + Alembic → omnivoice_data/
|
||||
```
|
||||
|
||||
| Layer | Path | Responsibility |
|
||||
|---|---|---|
|
||||
| Desktop shell | `frontend/src-tauri/` | Window lifecycle, tray, shortcuts, updater, sidecar bootstrap |
|
||||
| Frontend | `frontend/src/` | React UI, Zustand state, API and event clients, i18n |
|
||||
| API | `backend/api/` | REST routes, schemas, auth boundaries, streaming |
|
||||
| Core services | `backend/services/` | Generation, dubbing, audio processing, persistence |
|
||||
| Engines | `backend/engines/` | Isolated and optional engine adapters |
|
||||
| Worker system | `backend/worker/` | Authenticated remote compute and job transport |
|
||||
| Data | `omnivoice_data/` | Projects, voices, settings, logs, and SQLite state |
|
||||
| Delivery | `scripts/`, `deploy/`, `.github/workflows/` | Development, packaging, containers, releases, CI |
|
||||
|
||||
### Network boundary
|
||||
|
||||
- The desktop talks to a loopback-only backend on `localhost:3900`.
|
||||
- Loopback API calls need no server key. Remote access requires a share PIN or API key.
|
||||
- Remote workers and OpenAI-compatible ASR are opt-in. Loopback ASR may use HTTP and keeps audio on the machine; non-loopback endpoints require HTTPS, and redirects are not followed.
|
||||
- Analytics is off until consent. If enabled, it sends allowlisted, content-free usage metadata. It never sends text, audio, file names, or projects.
|
||||
|
||||
<a id="api"></a>
|
||||
|
||||
## Local speech platform and OpenAI-compatible API
|
||||
|
||||
Point an OpenAI-compatible audio client at the local backend:
|
||||
|
||||
```diff
|
||||
- base_url="https://api.openai.com/v1"
|
||||
+ base_url="http://localhost:3900/v1"
|
||||
```
|
||||
|
||||
| Endpoint | Purpose |
|
||||
|---|---|
|
||||
| `POST /v1/audio/speech` | TTS to `mp3`, `opus`, `aac`, `flac`, `wav`, or `pcm`; select a profile with `voice` and an engine with `model` |
|
||||
| `POST /v1/audio/transcriptions` | STT to `json`, `text`, `verbose_json`, `srt`, or `vtt` |
|
||||
| `WS /v1/audio/transcriptions/stream` | Live PCM/WebM transcription with partial, utterance, and session-final events |
|
||||
| `GET /.well-known/voicestudio-speech` | Discover HTTP, WebSocket, MCP, and native dictation-control transports |
|
||||
| `GET /v1/audio/voices` | List local voice profiles and engines |
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:3900/v1", api_key="local")
|
||||
|
||||
with client.audio.speech.with_streaming_response.create(
|
||||
model="tts-1",
|
||||
voice="<profile-id>",
|
||||
input="Made on my own hardware.",
|
||||
response_format="wav",
|
||||
) as response:
|
||||
response.stream_to_file("speech.wav")
|
||||
```
|
||||
|
||||
```bash
|
||||
# Quick test via cURL
|
||||
curl http://localhost:3900/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "tts-1", "input": "Made on my own hardware.", "voice": "default", "response_format": "wav"}' \
|
||||
--output speech.wav
|
||||
```
|
||||
|
||||
The bundled Rust control sidecar lets Herdr, coding agents, VS Code, desktop apps,
|
||||
and TUIs trigger the system-wide dictation flow or reuse its native text
|
||||
insertion. See the [speech platform guide](docs/speech-platform.md). The full API
|
||||
reference is in **Settings → OpenAPI Reference**. For LAN, Tailscale, or proxy
|
||||
access, read [API authentication](docs/api-auth.md) before exposing the backend.
|
||||
|
||||
### Agent skills
|
||||
|
||||
Install the VoiceStudio skills for Claude Code, Codex, Cursor, and other [skills.sh](https://skills.sh)-compatible agents:
|
||||
|
||||
```bash
|
||||
npx skills add debpalash/VoiceStudio
|
||||
```
|
||||
|
||||
- `omnivoice`: synthesize speech and transcribe audio through local VoiceStudio.
|
||||
- `oss-maintainer`: the repository's open-source maintenance workflow.
|
||||
|
||||
### Model Context Protocol (MCP)
|
||||
|
||||
VoiceStudio mounts an MCP server at `http://localhost:3900/mcp` for Claude Desktop, Cursor, and AI agents:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicestudio": {
|
||||
"url": "http://localhost:3900/mcp"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For clients requiring stdio transport, use the bundled local shim (`docs/mcp.json`):
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicestudio": {
|
||||
"command": "python",
|
||||
"args": ["-m", "backend.mcp_shim"],
|
||||
"cwd": "/path/to/VoiceStudio"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
See the [MCP guide](docs/mcp.md) for tools (`generate_speech`, `clone_voice`, `transcribe`), file streaming modes, and client bindings.
|
||||
|
||||
### Google Colab
|
||||
|
||||
[](https://colab.research.google.com/github/debpalash/VoiceStudio/blob/main/notebooks/OmniVoice_Studio_Colab.ipynb)
|
||||
|
||||
The [notebook](notebooks/OmniVoice_Studio_Colab.ipynb) runs the app and web UI on a Colab GPU. Colab is remote compute, so uploaded audio and project data do not remain local to your machine.
|
||||
|
||||
<a id="documentation"></a>
|
||||
> **Electron is the primary desktop app.** The next desktop release ships Electron, with one final Tauri sunset update. Bug reports and contributions remain welcome; include the app version and whether you use Electron or Tauri.
|
||||
|
||||
## Documentation
|
||||
|
||||
| Need | Read |
|
||||
| Need | Start here |
|
||||
|---|---|
|
||||
| Install | [macOS](docs/install/macos.md) · [Windows](docs/install/windows.md) · [Linux](docs/install/linux.md) · [Docker](docs/install/docker.md) |
|
||||
| Fix setup | [Troubleshooting](docs/install/troubleshooting.md) · [model downloads](docs/downloading-models.md) · [Hugging Face token](docs/setup/huggingface-token.md) |
|
||||
| Choose an engine | [Engine guides](docs/engines/README.md) · [benchmarks](docs/benchmarks.md) · [expressive speech](docs/expressive-speech.md) |
|
||||
| Tune hardware | [Performance](docs/performance.md) · [remote workers](docs/remote-workers.md) |
|
||||
| Build integrations | [Speech platform](docs/speech-platform.md) · [Private production API](docs/production-private-api.md) · [API auth](docs/api-auth.md) · [MCP](docs/mcp.md) · [examples](examples/README.md) |
|
||||
| Build VoiceStudio | [Contributing](.github/CONTRIBUTING.md) · [engine acceptance](docs/engine-acceptance.md) |
|
||||
| Track changes | [Changelog](CHANGELOG.md) · [roadmap](docs/ROADMAP.md) · [latest release](https://github.com/debpalash/VoiceStudio/releases/latest) |
|
||||
| Remove everything | [Uninstall guide](docs/install/uninstall.md) |
|
||||
| Setup help | [Troubleshooting](docs/install/troubleshooting.md) · [Model downloads](docs/downloading-models.md) |
|
||||
| Models & audio quality | [Engine guides](docs/engines/README.md) · [Benchmarks](docs/benchmarks.md) |
|
||||
| Integrations | [Local API](docs/speech-platform.md) · [MCP](docs/mcp.md) · [Examples](examples/README.md) |
|
||||
| Development | [Contributing](.github/CONTRIBUTING.md) · [Electron](electron/README.md) · [Changelog](CHANGELOG.md) |
|
||||
|
||||
<a id="faq"></a>
|
||||
Agent skills: `npx skills add debpalash/VoiceStudio` — choose **voicestudio** for audio workflows or **voicestudio-maintainer** for repository maintenance.
|
||||
|
||||
## FAQ
|
||||
## Sponsors
|
||||
|
||||
<details>
|
||||
<summary><strong>Does it work on Apple Silicon and Intel Macs?</strong></summary>
|
||||
<a href="https://forms.gle/2PYCvd39hbwijzX37"><img src="docs/media/sponsor-slot.svg" alt="Your brand — apply for a featured VoiceStudio sponsor slot" width="640" /></a>
|
||||
|
||||
Apple Silicon is supported with MPS and MLX options. Intel Macs cannot run the local backend because current PyTorch wheels are unavailable; they can connect to a remote backend. See [macOS installation](docs/install/macos.md).
|
||||
</details>
|
||||
**Become a featured partner.** [Apply for a paid placement](https://forms.gle/2PYCvd39hbwijzX37) · [Email us](mailto:partner@voicestudio.sh)
|
||||
|
||||
<details>
|
||||
<summary><strong>How much VRAM do I need?</strong></summary>
|
||||
Support development: [Ko-fi](https://ko-fi.com/debpalash) · [PayPal](https://paypal.me/palashCoder) · [Sponsorship details](SPONSORS.md)
|
||||
|
||||
A GPU is optional. Use 4 GB VRAM as the minimum for accelerated work and 8 GB+ for the default multi-stage workflow. Large optional engines can require 12 to 16 GB or more. Check the [benchmarks](docs/benchmarks.md) and engine guide.
|
||||
</details>
|
||||
## License & responsible use
|
||||
|
||||
<details>
|
||||
<summary><strong>Why does a longer reference clip not always improve the clone?</strong></summary>
|
||||
|
||||
Cloning is zero-shot: the clip is a prompt, not training data. Use 5 to 15 seconds of one speaker, close to the microphone, without music, noise, or reverb. Match the tone and pace you want in the output. For training, see [data preparation](docs/data_preparation.md) and [training](docs/training.md).
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>Can I use generated audio commercially?</strong></summary>
|
||||
|
||||
VoiceStudio's application license does not restrict generated audio, but it does not grant rights under a model's separate terms. The default OmniVoice repository labels its pretrained weights CC-BY-NC and includes a tokenizer under separate community terms. Review the selected model terms before commercial use.
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>Does VoiceStudio collect data?</strong></summary>
|
||||
|
||||
Not unless you opt in. Analytics is off by default and skipping consent keeps it off. When enabled, the app sends allowlisted, content-free usage metadata. Text, audio, file names, voices, and projects are excluded. Change this at **Settings → Privacy**.
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>How do I remove VoiceStudio and its data?</strong></summary>
|
||||
|
||||
Use `scripts/uninstall.sh` on macOS/Linux or `scripts\uninstall.ps1` on Windows. Both show a dry run before deletion. See the [uninstall guide](docs/install/uninstall.md) for every path.
|
||||
</details>
|
||||
|
||||
## Community and contributing
|
||||
|
||||
- [GitHub Issues](https://github.com/debpalash/VoiceStudio/issues) for reproducible bugs and feature requests.
|
||||
- [Discord](https://discord.gg/bzQavDfVV9) for setup help and project discussion.
|
||||
- [Good first issues](https://github.com/debpalash/VoiceStudio/labels/good%20first%20issue) for a scoped starting point.
|
||||
- [Contributing guide](.github/CONTRIBUTING.md) for setup, tests, and pull requests.
|
||||
|
||||
<p align="center">
|
||||
<a href="https://star-history.com/#debpalash/VoiceStudio&Date">
|
||||
<img src="https://api.star-history.com/svg?repos=debpalash/VoiceStudio&type=Date" alt="Star History Chart" width="100%" />
|
||||
</a>
|
||||
</p>
|
||||
|
||||
## Support development
|
||||
|
||||
VoiceStudio is free and has no paid tier. Donations fund development and infrastructure.
|
||||
|
||||
[Ko-fi](https://ko-fi.com/debpalash) · [PayPal](https://paypal.me/palashCoder) · [Sponsorship details](SPONSORS.md)
|
||||
|
||||
## Responsible use and safety
|
||||
|
||||
VoiceStudio enables zero-shot voice cloning and speech generation on personal hardware. Please use it responsibly:
|
||||
- **Consent:** Only clone or synthesize voices with explicit permission from the speaker.
|
||||
- **Audio provenance:** VoiceStudio integrates [AudioSeal](https://github.com/facebookresearch/audioseal) imperceptible watermarking by default to detect and identify synthetic speech without altering sound quality.
|
||||
- **Local privacy:** For the default local workflow, audio recordings, transcripts, voices, and projects remain strictly on your local disk; data leaves your device only when you explicitly configure remote workers or external ASR endpoints.
|
||||
|
||||
## License
|
||||
|
||||
VoiceStudio is licensed under [AGPL-3.0](LICENSE). You may run it, modify it, and use it internally. The application license itself does not restrict selling generated audio, but downloaded model and tokenizer terms may. If you modify VoiceStudio and provide that modified version as a network service, AGPL requires you to offer the corresponding source under the same license. A commercial license for VoiceStudio-owned code is available for proprietary embedding; it does not relicense third-party models. Contact **VoiceStudio@palash.dev**. See [LICENSE-NOTICE.md](LICENSE-NOTICE.md) for the plain-language scope.
|
||||
|
||||
Optional engines and downloaded models retain their own licenses. The bundled `omnivoice/` Python code is Apache-2.0 upstream; the default downloaded weights and audio tokenizer use separate terms.
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
VoiceStudio builds on [OmniVoice](https://github.com/k2-fsa/OmniVoice), [WhisperX](https://github.com/m-bain/whisperX), [Demucs](https://github.com/facebookresearch/demucs), [Pyannote](https://github.com/pyannote/pyannote-audio), [CTranslate2](https://github.com/OpenNMT/CTranslate2), [AudioSeal](https://github.com/facebookresearch/audioseal), [Tauri](https://tauri.app), [Supertonic](https://huggingface.co/Supertone/supertonic-3), [Sherpa-ONNX](https://github.com/k2-fsa/sherpa-onnx), [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS), and [PocketTTS](https://kyutai.org).
|
||||
|
||||
<div align="center">
|
||||
<strong><a href="https://github.com/debpalash/VoiceStudio/releases/latest">Download VoiceStudio</a></strong> ·
|
||||
<a href="https://github.com/debpalash/VoiceStudio">Star the project</a> ·
|
||||
<a href="https://discord.gg/bzQavDfVV9">Join Discord</a>
|
||||
</div>
|
||||
[AGPL-3.0](LICENSE). Models have their own licenses; review them before commercial use. Clone voices only with permission. See [license details](LICENSE-NOTICE.md).
|
||||
|
||||
+53
-669
@@ -1,697 +1,81 @@
|
||||
*本文档是 [README.md](README.md) 的简体中文翻译；若与英文版有出入，以英文版为准。*
|
||||
|
||||
<div align="center">
|
||||
<img src="docs/logo.png" alt="VoiceStudio 徽标" width="120" height="120" />
|
||||
<img src="docs/logo.png" alt="VoiceStudio" width="88" />
|
||||
<h1>VoiceStudio</h1>
|
||||
<p><sub><em>原名 OmniVoice-Studio</em></sub></p>
|
||||
<h3>创造声音，讲述故事，文件始终属于你。♡</h3>
|
||||
<p>在一个开源桌面工作室里完成克隆、设计、配音、听写和有声书制作。<br/><b>默认本地优先。</b>没有订阅，也没有用量计费；联网服务始终由你主动选择。</p>
|
||||
|
||||
<p><strong>开源声音克隆与工作流引擎。在本地构建。</strong></p>
|
||||
<p>使用本地 AI 克隆声音、翻译配音、语音听写和制作有声书。</p>
|
||||
<p>
|
||||
<a href="#quickstart">快速开始</a> ·
|
||||
<a href="#features">功能</a> ·
|
||||
<a href="#why-voicestudio">为什么选择 VoiceStudio</a> ·
|
||||
<a href="#tts-engines">引擎</a> ·
|
||||
<a href="#openai-api">API</a> ·
|
||||
<a href="#sponsor--donate">捐赠</a> ·
|
||||
<a href="#contributing">参与贡献</a> ·
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest">下载</a> ·
|
||||
<a href="#开始使用">开始使用</a> ·
|
||||
<a href="#文档">文档</a> ·
|
||||
<a href="https://discord.gg/bzQavDfVV9">Discord</a> ·
|
||||
<a href="README.md"><strong>English</strong></a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/debpalash/VoiceStudio/ci.yml?branch=main&style=flat-square&label=CI" alt="CI 状态" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/stargazers"><img src="https://img.shields.io/github/stars/debpalash/VoiceStudio?style=flat-square&color=f59e0b" alt="Star 数" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/VoiceStudio?style=flat-square&color=10b981" alt="版本" /></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="许可证" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/issues"><img src="https://img.shields.io/github/issues/debpalash/VoiceStudio?style=flat-square&color=ef4444" alt="Issues" /></a>
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Discord-Join_Community-5865F2?style=flat-square&logo=discord&logoColor=white" alt="Discord" /></a>
|
||||
<a href="https://ko-fi.com/debpalash"><img src="https://img.shields.io/badge/Ko--fi-Support_Us-FF5E5B?style=flat-square&logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
|
||||
<a href="https://paypal.me/palashCoder"><img src="https://img.shields.io/badge/PayPal-Donate-00457C?style=flat-square&logo=paypal&logoColor=white" alt="PayPal" /></a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/badge/⬇_Download-macOS_·_Windows_·_Linux-10b981?style=for-the-badge" alt="下载最新版本" /></a>
|
||||
<a href="README.md">English</a>
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<br/>
|
||||
|
||||
|
||||
<div align="center">
|
||||
<img src="docs/media/0.5.0/quick-switch.gif" alt="VoiceStudio — 从状态栏快速切换 TTS 引擎" width="100%"/>
|
||||
</div>
|
||||
<p align="center"><sub>新 Electron 桌面界面，使用此分支及内置演示声音录制。正式发布版本的界面可能有所不同。</sub></p>
|
||||
|
||||
> **声音很私人，创作空间也应该真正属于你。** VoiceStudio 的核心流程运行在你的硬件上：克隆、设计、配音、听写，并以 646 种语言创作，不需要订阅，也没有用量计费。联网引擎和服务始终是清晰可见的可选项，而不是隐藏依赖。
|
||||
## 用 VoiceStudio 创作
|
||||
|
||||
> [!WARNING]
|
||||
> **活跃 Beta 阶段。** 各版本之间可能出现故障——如需最新修复，请从源码运行。非常欢迎 Bug 报告和 PR：[提交 Issue](https://github.com/debpalash/VoiceStudio/issues) 或 [加入 Discord](https://discord.gg/bzQavDfVV9)。
|
||||
- **声音克隆与设计**：上传参考录音，或用文字描述你想要的声音。
|
||||
- **视频配音**：转录、翻译、分配说话人，并编辑语音时间轴。
|
||||
- **语音听写**：通过悬浮录音组件录制、转录和复制文字。
|
||||
- **长篇创作**：制作多角色脚本、有声书和批量任务。
|
||||
- **模型管理**：选择语音合成与转录引擎、语言及计算设备。
|
||||
|
||||
<a id="quickstart"></a>
|
||||
本地工作流在你的硬件上运行。远程服务为可选功能；使用情况分析须经同意才会启用。
|
||||
|
||||
## ⚡ 快速开始
|
||||
<table>
|
||||
<tr>
|
||||
<td><img src="docs/media/electron/voice-cloning.png" alt="Electron 声音克隆工作区与内置演示声音" width="100%" /></td>
|
||||
<td><img src="docs/media/electron/dubbing.png" alt="Electron 视频配音工作区" width="100%" /></td>
|
||||
</tr>
|
||||
<tr><td align="center">声音克隆</td><td align="center">视频配音</td></tr>
|
||||
<tr>
|
||||
<td><img src="docs/media/electron/voice-design.png" alt="Electron 声音设计工作区" width="100%" /></td>
|
||||
<td><img src="docs/media/electron/models.png" alt="本地语音模型管理" width="100%" /></td>
|
||||
</tr>
|
||||
<tr><td align="center">声音设计</td><td align="center">本地模型</td></tr>
|
||||
</table>
|
||||
|
||||
<div align="center">
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="下载 macOS DMG" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="下载 Windows MSI" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="下载 Linux AppImage" /></a>
|
||||
<br/>
|
||||
<sub>三个按钮都会打开最新发布页——在资源列表中下载对应你系统的安装包。</sub><br/>
|
||||
<sub><b>macOS：</b>首次启动需要一次性批准——右键点击 → <b>打开</b>（macOS 15 上为 系统设置 → 隐私与安全性 → <b>“仍要打开”</b>）。无需终端。<a href="docs/install/macos.md#gatekeeper-quarantine">为什么？</a> · <b>Intel Mac：</b>不支持本地后端（<a href="https://github.com/debpalash/VoiceStudio/issues/889">#889</a>）——<a href="docs/install/macos.md">详情</a>。</sub>
|
||||
</div>
|
||||
## 开始使用
|
||||
|
||||
选择你的操作系统，按指南从头到尾操作：
|
||||
从 [Releases](https://github.com/debpalash/VoiceStudio/releases/latest) 下载，然后阅读对应平台的安装指南：
|
||||
|
||||
- 🍎 **macOS** — [docs/install/macos.md](docs/install/macos.md)
|
||||
- 🪟 **Windows** — [docs/install/windows.md](docs/install/windows.md)
|
||||
- 🐧 **Linux** — [docs/install/linux.md](docs/install/linux.md)
|
||||
- 🐳 **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
|
||||
**[macOS](docs/install/macos.md) · [Windows](docs/install/windows.md) · [Linux](docs/install/linux.md) · [Docker](docs/install/docker.md)**
|
||||
|
||||
打开声音克隆页面，选择已有声音或添加清晰的参考录音，输入文字并生成。按提示安装所需模型。硬件要求因引擎而异，详见[性能指南](docs/performance.md)。
|
||||
|
||||
**从源码运行 Electron 预览版：**
|
||||
|
||||
```bash
|
||||
# Docker 快速运行 (CPU / 本地环回模式)
|
||||
docker run -d -p 127.0.0.1:3900:3900 -v omnivoice-data:/app/omnivoice_data --name voicestudio palashdeb/omnivoice-studio:stable
|
||||
git clone https://github.com/debpalash/VoiceStudio.git
|
||||
cd VoiceStudio
|
||||
bun install
|
||||
cd electron
|
||||
bun run dev
|
||||
```
|
||||
|
||||
**三步克隆出你的第一个声音：**
|
||||
环境要求和后端配置见 [Electron 开发指南](electron/README.md)。项目仍在积极开发中，可通过 [GitHub Issues](https://github.com/debpalash/VoiceStudio/issues) 反馈问题。
|
||||
|
||||
1. **安装并启动。** 首次启动会自动搭建 Python 运行环境并下载模型权重——启动画面会逐步显示进度（仅首次，需要几分钟；之后即开即用）。
|
||||
2. 从启动台打开**语音克隆**，拖入任意声音的 **3 秒音频**。
|
||||
3. **输入一句话，点击生成。** 音频在你的设备上生成并保存，支持 646 种语言（商业使用前请审阅所选模型与分词器的许可条款）。
|
||||
## 文档
|
||||
|
||||
### 🎧 音频示例
|
||||
|
||||
在线试听 VoiceStudio 本地生成的实际音频样例：
|
||||
|
||||
| 工作流 | 提示词 / 参考音频 | 生成音频 |
|
||||
|---|---|---|
|
||||
| **声音克隆** | [demo_voice.wav](backend/assets/samples/demo_voice.wav) | [demo_clone_output.wav](backend/assets/samples/demo_clone_output.wav) |
|
||||
| **声音设计** (美语新闻主播) | *"清晰、权威的美国广播级音色"* | [demo_voice_design_us_news_anchor.wav](backend/assets/samples/voice_design/demo_voice_design_us_news_anchor.wav) |
|
||||
| **声音设计** (英式有声书) | *"温暖生动的英式故事讲述音色"* | [demo_voice_design_audiobook_uk_narrator.wav](backend/assets/samples/voice_design/demo_voice_design_audiobook_uk_narrator.wav) |
|
||||
| **视频配音** (多语种) | [source.src.wav](backend/assets/samples/demo/dubbing/source.src.wav) | [西班牙语](backend/assets/samples/demo/dubbing/dubbed_es.src.wav) · [法语](backend/assets/samples/demo/dubbing/dubbed_fr.src.wav) · [日语](backend/assets/samples/demo/dubbing/dubbed_ja.src.wav) · [中文](backend/assets/samples/demo/dubbing/dubbed_zh.src.wav) |
|
||||
|
||||
觉得慢？[docs/performance.md](docs/performance.md) 讲清了生成时间到底花在哪里、有哪些调优开关，以及“它变慢了”的三个经典原因。各引擎/设备的实测数据见 [docs/benchmarks.md](docs/benchmarks.md)。
|
||||
|
||||
> 正在从 **[CorentinJ/Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning)**（现已归档）迁移过来？我们有专门的迁移指南：[docs/migration/real-time-voice-cloning.md](docs/migration/real-time-voice-cloning.md)。
|
||||
|
||||
<details>
|
||||
<summary><b>🧰 卡住了？自检、Token 与受限网络</b></summary>
|
||||
|
||||
<br/>
|
||||
|
||||
先运行内置自检——在应用中打开 **设置 → 关于 → “运行自检”**，或在源码检出目录中执行
|
||||
`uv run python backend/main.py --diagnose`（加 `--deep` 还会实际加载当前引擎进行测试）。然后查看
|
||||
[docs/install/troubleshooting.md](docs/install/troubleshooting.md) 中排名前
|
||||
10 的安装错误。运行时出错时，应用内的错误界面会直接深链到对应条目；**设置 → 关于 →
|
||||
“保存诊断包”** 会把脱敏日志与自检报告打包，方便附在 Bug 报告里。
|
||||
|
||||
Hugging Face Token 的配置见
|
||||
[docs/setup/huggingface-token.md](docs/setup/huggingface-token.md)。说话人分离相关的模型访问门槛见
|
||||
[docs/features/diarization.md](docs/features/diarization.md)。下载速度、⚡ 快速下载（Xet）状态，以及受限网络 / 镜像选项见
|
||||
[docs/downloading-models.md](docs/downloading-models.md)。
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
<a id="features"></a>
|
||||
|
||||
## ✨ 功能
|
||||
|
||||
八大主打功能——折叠区里还有十二项等你展开。
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td align="center" width="25%">
|
||||
<h3>🎙️ 语音克隆</h3>
|
||||
<p>3 秒音频 → 复刻任何声音。<br/><b>646 种语言</b>，零样本。</p>
|
||||
</td>
|
||||
<td align="center" width="25%">
|
||||
<h3>🎨 声音设计</h3>
|
||||
<p>性别、年龄、口音、音高、语速、<br/>情感、方言——<b>随心调节</b>。</p>
|
||||
</td>
|
||||
<td align="center" width="25%">
|
||||
<h3>🎬 视频配音</h3>
|
||||
<p>YouTube 链接或文件 → 转录 →<br/>翻译 → 重新配音 → <b>MP4</b>。</p>
|
||||
</td>
|
||||
<td align="center" width="25%">
|
||||
<h3>📖 有声书编辑器</h3>
|
||||
<p>导入文本、EPUB 或 PDF。自动分章、<br/>响度归一、元数据。导出 <b>.m4b</b>。</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
<h3>🎭 故事模式</h3>
|
||||
<p>多声音编辑器。逐行分配声音、<br/>预览、<b>导出完整配音阵容</b>。</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>⌨️ 听写工具</h3>
|
||||
<p>在<b>任何应用</b>中按 <kbd>⌘</kbd>+<kbd>⇧</kbd>+<kbd>Space</kbd>。<br/>转录、自动粘贴、随即消失。</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🔐 本地优先</h3>
|
||||
<p>核心创作流程<br/><b>留在你的设备上</b>。</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🤖 MCP 服务器</h3>
|
||||
<p>从 <b>Claude</b>、Cursor 或<br/>任何 MCP 客户端使用 VoiceStudio。</p>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
<details>
|
||||
<summary><b>……还有 12 项</b>——人声分离、说话人分离、批量处理、水印、诊断等等</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
- 🔊 **人声分离** — 基于 Demucs：把语音从音乐中分离出来，同时保留背景音床。
|
||||
- 👥 **说话人分离** — Pyannote + WhisperX 自动识别谁说了什么。
|
||||
- 📦 **批量队列** — 拖入 50 个视频就可以走开；每个任务都有独立进度条。
|
||||
- 🛡️ **AI 水印** — AudioSeal（Meta）：不可见，且能在压缩后留存。
|
||||
- 🔬 **诊断** — 自检套件、错误日志、脱敏诊断包。
|
||||
- ⚡ **GPU 自动检测** — CUDA · MPS · ROCm（Linux，需手动开启）· CPU；显存 ≤8 GB 时自动卸载。
|
||||
- 🧭 **引擎路由** — 逐引擎 GPU 预检；绝不静默回退到 CPU。
|
||||
- 🧩 **可扩展** — 继承 `TTSBackend`，约 50 行代码即可接入任意引擎。
|
||||
- 🎒 **便携声音角色** — 将声音导出为 `.ovsvoice` 包：身份 + 水印。
|
||||
- ♾️ **无限长 TTS** — 按句分块生成，没有长度上限，可经 WebSocket 流式输出。
|
||||
- 🌐 **远程后端** — 让 UI 指向远程服务器；对 Tailscale 友好，支持 Bearer 认证。
|
||||
- 🧠 **听写 + LLM** — 用本地 LLM 润色转录文本，可选回声消除。
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
<a id="why-voicestudio"></a>
|
||||
|
||||
## 💡 为什么选择 VoiceStudio？
|
||||
|
||||
云端语音工具很方便，但工作流会依赖账号、用量计费和他人的基础设施。VoiceStudio 在你的硬件上提供完整工作室；只有你主动选择时，才会使用联网集成。
|
||||
|
||||
| | **ElevenLabs** | **VoiceStudio** |
|
||||
|---|---|---|
|
||||
| **价格** | 订阅与用量限制 | 免费且开源（AGPL-3.0）· 专有用途可选 [商业许可证](#license) |
|
||||
| **语音克隆** | ✅ 3 秒音频 | ✅ 3 秒音频，零样本 |
|
||||
| **声音设计** | ✅ 性别、年龄 | ✅ 性别、年龄、口音、音高、风格、方言 |
|
||||
| **有声书 / 故事** | ❌ | ✅ 完整有声书编辑器 + 多声音故事（EPUB/PDF 导入，.m4b 导出） |
|
||||
| **语言** | 取决于套餐和模型 | **646** |
|
||||
| **视频配音** | ✅ 仅云端 | ✅ 完全本地 |
|
||||
| **数据隐私** | 音频在远端处理 | 核心流程在本地运行；联网服务必须主动选择 |
|
||||
| **API 密钥** | 需要账号 | 本地流程不需要 |
|
||||
| **GPU 支持** | 不适用（云端） | CUDA · Apple Silicon · ROCm（Linux）· CPU |
|
||||
| **桌面应用** | ❌ | ✅ macOS · Windows · Linux |
|
||||
| **TTS 引擎** | 1 | **16** — [完整矩阵](#tts-engines) |
|
||||
| **ASR 引擎** | 1 | **11** — [完整阵容](#asr-engines) |
|
||||
| **MCP 服务器** | ❌ | ✅ 可从 Claude、Cursor 及任何 MCP 客户端使用 |
|
||||
| **自检** | ❌ | ✅ 诊断套件、错误日志、脱敏调试包 |
|
||||
| **可定制** | ❌ 闭源 | ✅ 随你 Fork、扩展、发布 |
|
||||
|
||||
专业级语音 AI，去掉订阅，也去掉云端。
|
||||
|
||||
<div align="center">
|
||||
<br/>
|
||||
<b>心动了？来和我们一起构建吧。</b><br/>
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Join_Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="加入 Discord" /></a>
|
||||
<br/><br/>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ 系统要求
|
||||
|
||||
| | **最低配置** | **推荐配置** |
|
||||
|---|---|---|
|
||||
| **操作系统** | Windows 10、macOS 12+（Apple Silicon）、Ubuntu 24.04+（glibc 2.39+） | 任意现代 64 位操作系统 |
|
||||
| **内存** | 8 GB | 16 GB+ |
|
||||
| **显存（GPU）** | 4 GB（自动将 TTS 卸载到 CPU） | 8 GB+（NVIDIA RTX 3060+） |
|
||||
| **硬盘** | 10 GB 可用空间（模型 + 缓存） | 20 GB+ SSD |
|
||||
| **Python** | 3.10+（由 `uv` 管理） | 3.11–3.12 |
|
||||
| **GPU** | 可选——CPU 也能跑 | NVIDIA CUDA · Apple Silicon MPS · AMD ROCm（仅 Linux） |
|
||||
|
||||
> [!TIP]
|
||||
> 对于显存 **≤8 GB** 的 GPU，VoiceStudio 会在转录期间自动将 TTS 卸载到 CPU——无需配置。不需要专用 GPU；整条流水线都可以在 CPU 上运行（只是慢一些）。
|
||||
|
||||
> [!NOTE]
|
||||
> **AMD GPU：** ROCm 加速**仅限 Linux 且需手动开启**——在首次运行的设置界面选择 **“AMD GPU (ROCm)”**，或设置 `OMNIVOICE_TORCH_VARIANT=rocm`（[docs/install/linux.md](docs/install/linux.md#amd-gpu-rocm)）。在 **Docker/Podman** 中请改用专门的 ROCm 镜像：`ghcr.io/debpalash/omnivoice-studio:rocm`（[docs/install/docker.md](docs/install/docker.md#pull-and-run-amd-gpu--rocm)）。**在 Windows 上，AMD GPU（含 Ryzen AI 核显）只能以 CPU 运行**：PyTorch 没有 Windows 版 ROCm 轮子，因此 Windows 上的 GPU 加速仅限 NVIDIA/CUDA（[docs/install/windows.md](docs/install/windows.md#gpu-support)）。
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **macOS Intel（x86_64）不支持本地后端：** 应用 UI 可以安装，但 Python 后端无法运行，因为 PyTorch 已不再发布 Intel Mac 轮子（[#889](https://github.com/debpalash/VoiceStudio/issues/889)）。Intel Mac 用户仍可让 UI 指向另一台机器上的远程后端——参见 [docs/install/macos.md](docs/install/macos.md)。
|
||||
|
||||
<a id="hardware-recommendations"></a>
|
||||
|
||||
### 💡 按硬件推荐引擎配置
|
||||
|
||||
| 硬件配置 | 推荐 TTS 引擎 | 推荐 ASR 语音识别 | 优势 |
|
||||
|---|---|---|---|
|
||||
| **Apple Silicon (M1–M4)** | [MLX-Audio](docs/engines/mlx-audio.md) · [OmniVoice](docs/engines/omnivoice.md) (MPS) | [MLX Whisper](docs/engines/mlx-whisper.md) · [Parakeet MLX](docs/engines/parakeet-mlx.md) | 原生统一内存，macOS 上延迟最低、性能最强 |
|
||||
| **NVIDIA 显卡 (8 GB+ 显存)** | [OmniVoice](docs/engines/omnivoice.md) · [CosyVoice 3](docs/engines/cosyvoice.md) | [WhisperX](docs/engines/whisperx.md) | 极致零样本克隆品质、字级时间戳对齐与说话人分离 |
|
||||
| **低显存 / 仅 CPU 设备** | [PocketTTS](docs/engines/pockettts.md) · [Sherpa-ONNX](docs/engines/sherpa-onnx.md) · [KittenTTS](docs/engines/kittentts.md) | [Moonshine](docs/engines/moonshine.md) · [Faster-Whisper](docs/engines/faster-whisper.md) (`int8`) | 超低内存占用，针对 CPU 指令集深度优化 |
|
||||
|
||||
<a id="tts-engines"></a>
|
||||
|
||||
### 🗣️ TTS 引擎
|
||||
|
||||
**16 个引擎，一个选择器。** VoiceStudio（默认，支持 600+ 语言）始终可用；另有七个引擎可选装并自动检测（CosyVoice 3、GPT-SoVITS、VoxCPM2、MOSS-TTS-Nano、KittenTTS、MLX-Audio、Sherpa-ONNX），外加八个按需延迟安装的引擎（IndexTTS 2.5、OmniVoice GGUF、OmniVoice 子进程版、PocketTTS、Supertonic 3、MOSS-TTS-v1.5、dots.tts、Confucius4-TTS）。在 **设置 → TTS 引擎** 中切换；所选引擎将应用于所有语音合成场景。**每个引擎都有独立指南：[docs/engines](docs/engines/README.md)（英文）。**
|
||||
|
||||
<details>
|
||||
<summary><b>📊 完整矩阵</b>——16 个引擎 × 平台 × 克隆/指令 × 许可证</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| 引擎 | 语言 | 克隆 | 指令 | Linux | macOS ARM | Windows | 许可证 |
|
||||
|--------|:---------:|:-----:|:--------:|:-----:|:---------:|:-------:|:-------:|
|
||||
| **VoiceStudio**（默认，由 k2-fsa/OmniVoice 驱动） | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | 内置 |
|
||||
| **CosyVoice 3** | 9 + 18 种方言 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **GPT-SoVITS** | 5 | ✅ | — | ✅ CUDA/CPU | — | ✅ CUDA/CPU | MIT |
|
||||
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MOSS-TTS-Nano** | 20 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **KittenTTS** | 英语 | — | — | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
|
||||
| **MLX-Audio**（Kokoro、Qwen3-TTS、CSM、Dia 等） | 多语言 | 因模型而异 | 因模型而异 | ❌ | ✅ 原生 | ❌ | 因模型而异 |
|
||||
| **Sherpa-ONNX** | 20+ | — | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **IndexTTS 2.5** ⚡ | 中文 · 英语 · 日语 · 西班牙语 · 阿拉伯语 | ✅ | — | ✅ CUDA | — | ✅ CUDA | Bilibili 模型许可¹ |
|
||||
| **OmniVoice GGUF** ⚡ | 600+ | ✅ | ✅ | ✅ CPU | ✅ CPU | ✅ CPU | 内置 |
|
||||
| **Supertonic 3** ⚡ | 31 | — | — | ✅ CPU | ✅ CPU | ✅ CPU | OpenRAIL-M |
|
||||
| **MOSS-TTS-v1.5** ⚡（8B） | 31 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **dots.tts** ⚡（2B） | 24 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ❌ | Apache-2.0 |
|
||||
| **Confucius4-TTS** ⚡ | 14 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
|
||||
¹ 若月活跃用户超过 1 亿，或年收入超过人民币 10 亿元，使用 IndexTTS 2.5
|
||||
前必须另行取得 Bilibili 的书面许可。启用可选边车前，请审阅其
|
||||
[模型许可](https://huggingface.co/IndexTeam/IndexTTS-2.5/blob/main/LICENSE)。
|
||||
|
||||
> **CUDA** = GPU 加速 · **MPS** = Apple Silicon Metal · **CPU** = 随处可运行，大模型较慢 · KittenTTS 和 MOSS-TTS-Nano 可在 CPU 上实时运行 · MLX-Audio 仅限 Apple Silicon · ⚡ = 延迟注册（首次使用时安装）
|
||||
>
|
||||
> **克隆**能力的意义不止于单段生成：视频配音（以及任何固定了声音的批量任务）需要参考音频克隆来保持说话人身份，因此把不支持克隆的引擎（KittenTTS、Sherpa-ONNX、Supertonic 3）设为当前引擎时，这些任务会在开始前就给出可操作的失败提示，而不是静默回退到 VoiceStudio。
|
||||
>
|
||||
> **MOSS-TTS-v1.5**（8B，约 16 GB）、**dots.tts**（2B，约 9 GB）和 **Confucius4-TTS** 是重量级可选引擎，从本地克隆在各自独立的 venv 中运行。三者均不支持 Apple Silicon MPS（在 Mac 上以 CPU 运行）；dots.tts 没有 Windows 路径；Confucius4 建议使用 CUDA（CPU 可用，约为实时时长的 17 倍）。详情：[MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) · [dots.tts](docs/engines/dots-tts.md) · [Confucius4-TTS](docs/engines/confucius4-tts.md)。
|
||||
|
||||
</details>
|
||||
|
||||
<a id="asr-engines"></a>
|
||||
|
||||
### 🎧 ASR 引擎
|
||||
|
||||
**11 个引擎**——它们驱动听写、视频配音和字幕。**WhisperX** 是跨平台的默认引擎（约 100 种语言，词级时间对齐）；其余引擎均为可选装并自动检测。在 **设置 → 引擎** 中切换。十个完全在本地设备上运行；第十一个（OpenAI 兼容）是可选的远程客户端，可用于 Qwen3-ASR 或任何兼容的服务器。
|
||||
|
||||
<details>
|
||||
<summary><b>📊 完整阵容</b>——11 个引擎、各自的强项与计算类型说明</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| 引擎 | `OMNIVOICE_ASR_BACKEND` | 语言 | 最适合 |
|
||||
|--------|-------------------------|:---------:|----------|
|
||||
| **WhisperX**（默认） | `whisperx` | ~100 | 配音与字幕——通过 wav2vec2 强制对齐实现词级时间对齐 |
|
||||
| **Faster-Whisper** | `faster-whisper` | ~100 | Linux / macOS / Windows 上的快速转录（CTranslate2） |
|
||||
| **Faster-Whisper（隔离）** | `faster-whisper-isolated` | ~100 | 与 Faster-Whisper 相同，但在子进程中崩溃隔离——ASR 崩溃不会拖垮整个应用 |
|
||||
| **MLX Whisper** | `mlx-whisper` | ~100 | Apple Silicon 原生速度（Apple MLX / Metal） |
|
||||
| **PyTorch Whisper** | `pytorch-whisper` | ~100 | 经 🤗 Transformers 的 CUDA / CPU 兜底方案（无需 cuDNN 8） |
|
||||
| **Parakeet TDT** | `nemo-parakeet` | 英语 + 25 种欧洲语言 | 即使在 CPU 上也能以约 10 倍实时速度达到 SOTA 精度，自动语言检测（NVIDIA NeMo，CUDA/CPU） |
|
||||
| **Moonshine** | `moonshine` | 英语 | 边缘设备 / 低延迟，ONNX |
|
||||
| **FunASR** | `funasr` | 50+ | 多语言一体化——内置 VAD + 行内说话人分离（SenseVoice） |
|
||||
| **sherpa-onnx**（实时听写） | `sherpa-onnx-asr` | 25 种欧洲语言 + 90+ | 实时、快于实时的听写——小体积流式/离线 ONNX 模型（Parakeet TDT v3/v2、流式 Zipformer 与 Paraformer、Whisper Tiny），CPU 运行，macOS / Windows / Linux 表现完全一致。在 **设置 → 语音** 中按模型选择。 |
|
||||
| **OpenAI 兼容** ⚠️ 远程 | `openai-compat-asr` | 取决于服务器 | 当下通往 **Qwen3-ASR** 的路径（自托管服务器，无需等 transformers 支持）、任何 OpenAI 兼容的转录端点，或 OpenAI 官方 API——无需安装，在 **设置 → 引擎**（ASR 标签页）中配置并测试连接。音频会离开你的设备，发送到你指定的任何服务器；参见 [docs/engines/openai-compatible-asr.md](docs/engines/openai-compatible-asr.md)。 |
|
||||
|
||||
> Whisper 系列引擎覆盖约 100 种语言；**FunASR / SenseVoice** 额外提供一条多语言一体化路径，内置语音活动检测与行内说话人分离。**sherpa-onnx** 驱动实时听写的模型选择器——你边说，文字边出现。除可选的 OpenAI 兼容远程客户端外，所有引擎都在本地设备上运行——无需 API 密钥，无需云端。
|
||||
|
||||
> **GPU 不支持高效 float16？** 在较老的 NVIDIA GPU（Maxwell/Pascal、GTX 16xx）上，或在 CTranslate2/cuDNN 版本不匹配之后，CTranslate2 系 ASR 引擎（WhisperX、Faster-Whisper）无法运行 `float16`，VoiceStudio 会自动改用 `int8` 重试——无需配置。如果转录仍然失败，可用 `ASR_COMPUTE_TYPE` 环境变量固定计算类型（逃生舱口）：`ASR_COMPUTE_TYPE=int8`（CPU 用 `float32`）。将其设为 `int8` 并重启后端。
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ 架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Frontend (React) │
|
||||
│ DubTab · VoiceConsole · Stories · Audiobook · Gallery │
|
||||
│ Dictation · BatchQueue · Diagnostics · MCP Client │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Backend (FastAPI) │
|
||||
│ 100+ API endpoints · SSE+WSS streaming · SQLite │
|
||||
├──────────┬──────────┬──────────┬──────────┬────────────────┤
|
||||
│ WhisperX │ Demucs │VoiceStudio │ Pyannote │ Engine Routing │
|
||||
│ (+7 ASR │ Source │ (+10 │ Diariz- │ ↳ GPU preflight │
|
||||
│ engines) │ Sep. │ TTS) │ ation │ ↳ No silent CPU │
|
||||
└──────────┴──────────┴──────────┴──────────┴────────────────┘
|
||||
CUDA / MPS / ROCm / CPU (auto-detected + routed)
|
||||
```
|
||||
|
||||
<a id="openai-api"></a>
|
||||
|
||||
## 🔌 OpenAI 兼容 API
|
||||
|
||||
已经有会说 OpenAI 音频 API 的脚本、智能体或工具？把它指向 `http://localhost:3900/v1` 即可——不需要密钥，也不用改代码。后端为音频端点内置了即插即用的兼容接口，直接接到你当前启用的 TTS/ASR 引擎（没错，`voice` 参数接受你克隆的声音配置 ID）。
|
||||
|
||||
| 端点 | 作用 |
|
||||
| 需求 | 链接 |
|
||||
|---|---|
|
||||
| `POST /v1/audio/speech` | TTS——输入文本；输出 `mp3` / `wav` / `flac` / `opus` / `pcm`。`tts-1` / `tts-1-hd` 映射到你当前启用的引擎；也接受 OpenAI 的声音名称（`alloy` 等）。 |
|
||||
| `POST /v1/audio/transcriptions` | STT——输入音频文件；输出 `json`、`text`、`verbose_json`、`srt` 或 `vtt`。`whisper-1` 映射到你当前启用的 ASR 引擎。 |
|
||||
| `GET /v1/audio/voices` | VoiceStudio 扩展——列出所有声音配置和引擎，客户端可据此发现你的克隆声音。 |
|
||||
| 安装帮助 | [故障排查](docs/install/troubleshooting.md) · [模型下载](docs/downloading-models.md) |
|
||||
| 模型与音质 | [引擎指南](docs/engines/README.md) · [基准测试](docs/benchmarks.md) |
|
||||
| 集成 | [本地 API](docs/speech-platform.md) · [MCP](docs/mcp.md) · [示例](examples/README.md) |
|
||||
| 参与开发 | [贡献指南](.github/CONTRIBUTING.md) · [Electron](electron/README.md) · [更新日志](CHANGELOG.md) |
|
||||
|
||||
```sh
|
||||
curl http://localhost:3900/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "tts-1", "voice": "alloy", "input": "Generated on my own hardware.", "response_format": "wav"}' \
|
||||
--output speech.wav
|
||||
```
|
||||
安装智能体技能：`npx skills add debpalash/VoiceStudio`
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:3900/v1", api_key="none") # any string works — nothing checks it
|
||||
## 支持 VoiceStudio
|
||||
|
||||
result = client.audio.transcriptions.create(model="whisper-1", file=open("clip.wav", "rb"))
|
||||
print(result.text)
|
||||
```
|
||||
[Ko-fi](https://ko-fi.com/debpalash) · [PayPal](https://paypal.me/palashCoder) · [赞助项目](SPONSORS.md) · [商务合作](mailto:partner@voicestudio.sh)
|
||||
|
||||
想要完整的接口（100+ 端点）？完整的 REST API 参考已内嵌在应用中——**设置 → OpenAPI 参考**（由 Scalar 驱动），或点击页脚的 `{}` 按钮。
|
||||
**让语音应用开发者看到你的品牌。** 了解应用底部栏、集成目录、文档和 README 的付费展示合作。[申请合作](https://forms.gle/2PYCvd39hbwijzX37)或[发送邮件](mailto:partner@voicestudio.sh)。
|
||||
|
||||
### 📓 在 Google Colab 上运行
|
||||
## 许可与负责任使用
|
||||
|
||||
[](https://colab.research.google.com/github/debpalash/VoiceStudio/blob/main/notebooks/OmniVoice_Studio_Colab.ipynb)
|
||||
|
||||
没有本地 GPU？官方笔记本（[notebooks/OmniVoice_Studio_Colab.ipynb](notebooks/OmniVoice_Studio_Colab.ipynb)）可在免费的 Colab T4 上启动完整应用（包含 Web 界面）：在笔记本内直接构建前端，用 uv 安装后端（复用 Colab 预装的 CUDA PyTorch），并通过 Colab 内置端口代理打开界面。无需第三方隧道，也无需任何 API 密钥。随后还有一套覆盖全部主要功能的 API 导览，全部可在笔记本内直接播放：多语言 TTS、声音克隆与声音设计、已保存的声音档案、语音转写、AI 水印检测、OpenAI 兼容 API、多角色故事、带章节的 m4b 有声书，以及一个附带人声分离音轨的迷你视频配音。
|
||||
|
||||
### 🤝 智能体技能（Agent Skills）
|
||||
|
||||
用一条命令教会你的 AI 智能体（Claude Code、Cursor、Codex 等）使用 VoiceStudio：
|
||||
|
||||
```sh
|
||||
npx skills add debpalash/omnivoice-studio
|
||||
```
|
||||
|
||||
内含两个 [skills](https://skills.sh)：**`omnivoice`**——让任何智能体通过你的本地安装进行语音合成与转录（包括你克隆的声音），免费且离线；以及 **`oss-maintainer`**——本项目所遵循的维护者方法论，适合任何用智能体运营自己开源项目的人。
|
||||
|
||||
### 🔌 模型上下文协议（MCP 服务器）
|
||||
|
||||
VoiceStudio 在 `http://localhost:3900/mcp` 挂载了 MCP 服务，可供 Claude Desktop、Cursor 与自主智能体调用：
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicestudio": {
|
||||
"url": "http://localhost:3900/mcp"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
对于需要 stdio 管道传输的客户端，请使用内置的本地桥接脚本（`docs/mcp.json`）：
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicestudio": {
|
||||
"command": "python",
|
||||
"args": ["-m", "backend.mcp_shim"],
|
||||
"cwd": "/path/to/VoiceStudio"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
支持 `generate_speech`、`clone_voice`、`transcribe` 等工具与流式文件输出模式，详见 [docs/mcp.md](docs/mcp.md)。
|
||||
|
||||
---
|
||||
|
||||
## 🗺️ 路线图
|
||||
|
||||
### 🔜 即将推出
|
||||
|
||||
- 🎬 **唇形同步 v2** — 使用 wav2lip 进行视觉语音时间对齐
|
||||
- 🌐 **在线演示** — 无需安装即可体验 VoiceStudio
|
||||
- 🔌 **插件市场** — 社区贡献的 TTS 引擎与特效
|
||||
- 🎵 **实时变声器** — 通话中的麦克风实时变声
|
||||
|
||||
<details>
|
||||
<summary><b>✅ 已经发布的一切</b>——按类别列出的“成绩单”</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| 分类 | 功能 |
|
||||
|----------|----------|
|
||||
| **长内容** | 有声书编辑器（文本/EPUB/PDF → 分章 .m4b）、Stories 多声音编辑器、两遍响度归一母带处理、渲染中断后的崩溃续渲、发音控制 + SSML-lite 韵律 |
|
||||
| **配音** | 完整流水线（转录→翻译→合成→封装）、场景感知分割、唇形同步评分、流式 TTS、逐说话人声音分配、Smart Fit 时长匹配 + 二次 QC、独立的配音主页 |
|
||||
| **声音** | 零样本克隆、声音设计、A/B 对比、声音预览控件、支持收藏/标签的声音库、便携声音角色包（`.ovsvoice`）、声音控制台工作区 |
|
||||
| **音频** | Demucs 人声分离、逐段增益、选择性音轨导出、分轨/SRT/VTT/MP3 导出、按句分块实现的无限长 TTS |
|
||||
| **多语言** | 多语言批量选择器、顺序 GPU 执行的批量配音队列 |
|
||||
| **说话人分离** | Pyannote 机器学习分离、自动说话人克隆提取、逐说话人声音分配 |
|
||||
| **ASR** | 9 个引擎（WhisperX、Faster-Whisper、隔离版 Faster-Whisper、MLX Whisper、PyTorch Whisper、Parakeet TDT、Moonshine、FunASR/SenseVoice、sherpa-onnx 实时听写）、崩溃隔离的子进程后端 |
|
||||
| **TTS** | 14 个引擎（VoiceStudio、CosyVoice 3、GPT-SoVITS、VoxCPM2、MOSS-TTS-Nano、KittenTTS、MLX-Audio、Sherpa-ONNX，+ 延迟安装：IndexTTS 2.5、OmniVoice GGUF、Supertonic 3、MOSS-TTS-v1.5、dots.tts、Confucius4-TTS）、带 GPU 预检的引擎路由 |
|
||||
| **基础设施** | Docker 部署、CUDA/MPS/ROCm 自动检测、cuDNN 8 兼容、显存感知模型卸载、引擎路由（绝不静默回退 CPU）、诊断套件与错误日志、受限网络镜像支持 |
|
||||
| **AI 溯源** | AudioSeal 不可见水印（类似 SynthID）、视频徽标叠加、水印检测 API |
|
||||
| **用户体验** | 撤销/重做、键盘快捷键、拖放、会话持久化、首次启动按屏幕推荐界面缩放，以及原生 WebKitGTK 缩放 |
|
||||
| **实时事件** | WebSocket 事件总线——数据变更时即时刷新侧边栏、指数退避重连 |
|
||||
| **状态管理** | Zustand 状态迁移——`uiSlice`、`pillSlice`、`dubSlice`、`generateSlice`、`prefsSlice`、`glossarySlice` |
|
||||
| **桌面** | 跨平台 Tauri 安装程序（macOS DMG——Apple Silicon；Intel 不支持本地后端，#889——Windows MSI、Linux deb/AppImage）、自动更新基础设施、单实例约束、关闭最小化到托盘、macOS Gatekeeper 修复 |
|
||||
| **听写** | 全局系统级热键（`⌘+⇧+Space`）、无边框浮动控件、WebSocket 流式 ASR、自动粘贴、可自定义热键、本地 LLM 转录润色 |
|
||||
| **批量流水线** | 完整批量 TTS：提取 → 转录 → 翻译 → 生成 → 混音 → 导出，带实时进度追踪 |
|
||||
| **MCP 服务器** | 让 VoiceStudio 成为 Claude、Cursor 及任何 MCP 客户端的本地 TTS/STT 提供方 |
|
||||
| **远程后端** | 让桌面 UI 指向远程后端 URL，支持 Bearer 认证（附 Tailscale 文档） |
|
||||
| **可靠性** | 启动开屏的卡死看门狗、逐引擎 GPU 兼容矩阵、引擎二进制不可执行时的可操作报错、setuptools 自动修复 |
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
<a id="sponsor--donate"></a>
|
||||
|
||||
## 💜 赞助 / 捐赠
|
||||
|
||||
VoiceStudio 由一位开发者使用 Claude Code 和 AI 智能体独立打造——而智能体账单是实打实的（过去三个月花了数千美元）。如果 VoiceStudio 为你创造了价值，帮忙分担一小部分账单，就能让开发保持全职推进。
|
||||
|
||||
<div align="center">
|
||||
|
||||
**本月智能体账单基金**
|
||||
|
||||
<img src="https://img.shields.io/badge/raised_%2410_of_%24200-5%25-EAB308?style=for-the-badge" alt="已筹 $10 / $200" />
|
||||
|
||||
<br/><br/>
|
||||
|
||||
<a href="https://ko-fi.com/debpalash"><img src="https://img.shields.io/badge/Ko--fi-Support_❤️-FF5E5B?style=for-the-badge&logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
|
||||
|
||||
<a href="https://paypal.me/palashCoder"><img src="https://img.shields.io/badge/PayPal-Donate-00457C?style=for-the-badge&logo=paypal&logoColor=white" alt="PayPal" /></a>
|
||||
|
||||
<br/>
|
||||
<sub>每一美元都直接用于支付智能体账单——让 VoiceStudio 的开发持续不断。</sub>
|
||||
|
||||
<br/><br/>
|
||||
|
||||
<sub><b>来自 VoiceStudio 作者的更多应用</b>——同样的本地优先理念：
|
||||
<a href="https://github.com/debpalash/Opal"><b>Opal</b> 💠</a>（播放一切——AI 时代的媒体播放器）·
|
||||
<a href="https://github.com/debpalash/memxt"><b>memxt</b> 🧠</a>（Claude Code 与编码智能体的本地记忆）。
|
||||
给它们点个 ⭐ 也是一种支持 → <a href="#more-from-the-maker">详见下文</a>。</sub>
|
||||
|
||||
</div>
|
||||
|
||||
<a id="sponsors"></a>
|
||||
|
||||
### 🌟 赞助商
|
||||
|
||||
VoiceStudio **免费**且采用 **AGPL-3.0** 许可——没有付费版，没有 SaaS 收入。赞助商让开发得以持续，作为回报，可以在这里、在应用内（顶级档位还包括项目官网）获得一个徽标位。这是一份感谢，绝不是付费墙。**[查看档位并成为赞助商 →](SPONSORS.md)**
|
||||
|
||||
<div align="center">
|
||||
|
||||
<!-- SPONSORS:START — logo slots are filled here as sponsors come aboard; see SPONSORS.md -->
|
||||
|
||||
**这里可以是你的徽标** — [成为赞助商](SPONSORS.md)
|
||||
|
||||
<!-- SPONSORS:END -->
|
||||
|
||||
</div>
|
||||
|
||||
<sub>💡 GitHub 也会在本仓库顶部显示一个 **Sponsor** 按钮，经由 <a href=".github/FUNDING.yml"><code>.github/FUNDING.yml</code></a> 指向相同的链接。</sub>
|
||||
|
||||
---
|
||||
|
||||
## 💬 社区
|
||||
|
||||
<div align="center">
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/💬_Discord-Join_Community-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="加入 Discord" /></a>
|
||||
<br/>
|
||||
<sub>设置类问题我们几小时内就会回复，而不是几天。</sub>
|
||||
</div>
|
||||
|
||||
<details>
|
||||
<summary><b>里面都在聊什么</b></summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| 频道 | 那里发生什么 |
|
||||
|---------|--------------------|
|
||||
| `#announcements` | 发布消息与重大时刻——新版本最先在这里公布 |
|
||||
| `#releases` + `#changelog` | 每一个构建，以及里面究竟有什么 |
|
||||
| `#issues` | 以论坛帖子形式提交的 Bug 报告——直接分诊进 GitHub Issues |
|
||||
| `#ideas` | 功能请求，供讨论与投票 |
|
||||
| `#discuss-ideas` | 动手之前的设计讨论 |
|
||||
| `#general` | 安装帮助、GPU 疑难排查，以及晒你的配音成果 |
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
<a id="contributing"></a>
|
||||
|
||||
## 🤝 参与贡献
|
||||
|
||||
非常欢迎——Bug 修复、新的 TTS 引擎适配器、UI 改进、文档、翻译。统统欢迎。
|
||||
|
||||
- 📖 阅读 **[贡献指南](.github/CONTRIBUTING.md)** 了解环境搭建、代码风格和 PR 工作流
|
||||
- 🐛 浏览 [good first issues](https://github.com/debpalash/VoiceStudio/labels/good%20first%20issue)
|
||||
- 💬 加入我们的 [Discord](https://discord.gg/bzQavDfVV9) 讨论想法或寻求帮助
|
||||
|
||||
---
|
||||
|
||||
## ❓ 常见问题
|
||||
|
||||
<details>
|
||||
<summary><b>真的能和 ElevenLabs 一样好吗？</b></summary>
|
||||
<br/>
|
||||
诚实的回答：<b>取决于你要做什么。</b>
|
||||
|
||||
<b>VoiceStudio 真正有竞争力的地方：</b>从干净的参考音频进行语音克隆（最先进的开源扩散 TTS）、语言覆盖（646 种语言对他们的 32 种），以及所有结构性优势——没有按字符计费、没有用量上限、音频不离开你的设备、完整的流水线可定制性（14 个 TTS 引擎、10 个 ASR 引擎、翻译方案随你选）。
|
||||
|
||||
<b>ElevenLabs 仍然领先的地方：</b>开箱即用的稳定性与打磨程度，尤其是英语 TTS。他们的单一模型经过深度调优；我们的质量取决于你选择的引擎、你的硬件，以及（对克隆而言）参考音频——干燥、近麦的音频比嘈杂或有回声的音频克隆效果好得多。
|
||||
|
||||
<b>具体到配音：</b>配音是一条链——转录 → 翻译 → 克隆 → 合成——在<i>你的</i>素材上，它只取决于最薄弱的一环。如果部分输出语无伦次，先检查片段表里的<i>原文</i>：当转录本身就错了，换一个 ASR 引擎或使用更干净的源音频——修复点通常在这里，而不是声音。
|
||||
|
||||
拿你的真实素材试试——免费，下载一次即可。许多用户直接用它替换了 ElevenLabs；也有人两个都留着。这两种结果我们都乐见。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>能在 Apple Silicon（M1/M2/M3/M4）上运行吗？</b></summary>
|
||||
<br/>
|
||||
可以。MPS 加速会被自动检测。在 Apple 硬件上，MLX 优化的 Whisper 模型可提供更快的转录速度。<b>不支持 Intel Mac</b>：应用 UI 可以安装，但本地 Python 后端无法运行，因为 PyTorch 已不再发布 Intel Mac 轮子（<a href="https://github.com/debpalash/VoiceStudio/issues/889">#889</a>）——Intel Mac 只能配合远程后端使用。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>需要多少显存？</b></summary>
|
||||
<br/>
|
||||
<b>最低 4 GB。</b> 显存 ≤8 GB 时，TTS 模型会在转录期间自动卸载到 CPU。8 GB 以上时，所有组件同时在 GPU 上运行。完全没有 GPU？CPU 模式也能用——只是慢一些（TTS 约慢 3 倍）。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>可以用于商业用途吗？</b></summary>
|
||||
<br/>
|
||||
<b>可以——商业使用免费</b>，基于 <a href="https://www.gnu.org/licenses/agpl-3.0.html">AGPL-3.0</a>：运行它、出售用它生成的音频、为客户的视频配音、在团队中部署。只有一项义务：如果你<b>修改</b>了 VoiceStudio 并通过网络向他人提供该修改版本，你必须依据相同条款分享修改后的源代码。想把它嵌入闭源产品？可获取商业许可证——参见<a href="#license">许可证</a>。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>支持哪些语言？</b></summary>
|
||||
<br/>
|
||||
通过 VoiceStudio 模型的 TTS 支持 646 种语言。转录（WhisperX）支持 99 种语言。翻译覆盖范围取决于目标语言对。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>可以添加自己的 TTS 引擎吗？</b></summary>
|
||||
<br/>
|
||||
可以。在 <code>backend/services/tts_backend.py</code> 中继承 <code>TTSBackend</code>，并将其添加到 <code>_REGISTRY</code> 字典中——约 50 行代码。十四个内置引擎均以此方式实现；参见 <a href="#tts-engines">TTS 引擎</a>。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>VoiceStudio 会收集我的任何数据吗？</b></summary>
|
||||
<br/>
|
||||
<b>除非你明确同意，否则不会。</b>首次运行时应用会<i>询问</i>你——一个页面、两个同等分量的按钮，没有预先勾选。在你回答“是”之前，VoiceStudio 什么都不发送：没有分析、没有遥测、没有账号、没有“回传”。跳过提问就等于“否”。无论如何，你的文本、音频、声音和项目永远不会离开你的设备。
|
||||
|
||||
如果你选择同意（也可随时在 <b>设置 → 隐私 → “帮助改进 VoiceStudio”</b> 中开关），发送的只是匿名、不含内容的使用统计：生成信息（引擎、语言、生成耗时、字符<i>数量</i>、错误<i>类型</i>），以及应用生命周期——一次安装信号、版本更新（版本号之间）、崩溃（错误类别和<i>分桶后的</i>运行时长，绝不含日志）、错误<i>类型</i>（有上限、去重），以及卸载时的一次告别信号。绝不包含你的文本、音频、文件名或任何可识别信息——这由代码中的属性白名单强制保证（<code>backend/core/analytics.py</code>），而不只是一句承诺。源码构建根本没有分析数据的接收端，因此根本不会询问。你自己的统计数字在 <b>设置 → 用量</b> 中查看，本地计算，不发送到任何地方。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>如何卸载它 / 删除它的所有数据？</b></summary>
|
||||
<br/>
|
||||
VoiceStudio 完全本地运行——卸载就是删除应用及其写入的文件夹（模型缓存、Python 环境、你的声音/项目、配置）。运行 <code>scripts/uninstall.sh</code>（macOS/Linux）或 <code>scripts\uninstall.ps1</code>（Windows）——它会先以干跑方式列出每个文件夹及其大小，加 <code>--yes</code> 才会真正删除。完整的各平台路径列表和应用移除步骤见 <a href="docs/install/uninstall.md"><b>docs/install/uninstall.md</b></a>。
|
||||
</details>
|
||||
|
||||
## 🛡️ 负责任使用与安全
|
||||
|
||||
VoiceStudio 在个人硬件上提供零样本语音克隆与语音创作能力。我们提倡负责任的技术使用：
|
||||
- **明确授权：** 严禁在未经说话人本人知情并明确授权的情况下克隆其声音。
|
||||
- **AI 溯源：** VoiceStudio 默认集成 [AudioSeal](https://github.com/facebookresearch/audioseal) 不可见神经音频水印，在完全不影响听感音质的前提下精准标记合成语音。
|
||||
- **本地隐私：** 默认本地工作流下，所有音频、声音档案、项目与转录文本始终保存在你的本地设备上；仅当你主动配置远程工作节点或第三方 ASR 端点时，相应数据才会传输到对应服务。
|
||||
|
||||
---
|
||||
|
||||
<a id="license"></a>
|
||||
|
||||
## 📜 许可证
|
||||
|
||||
VoiceStudio 是基于 [**GNU Affero 通用公共许可证 v3.0（AGPL-3.0）**](https://www.gnu.org/licenses/agpl-3.0.html) 的自由开源软件。
|
||||
|
||||
**可免费用于任何用途——包括商业和企业内部用途。** 运行它、出售用它生成的音频、为自己或客户的视频配音、在团队中推广——全部免费，无需许可证。作为一份**网络著佐权（copyleft）**许可证，AGPL 增加了一项义务：如果你**修改**了 VoiceStudio 并通过网络向他人提供该修改版本，你必须依据相同的 AGPL-3.0 条款向他们提供该修改版本的完整对应源代码。
|
||||
|
||||
希望将 VoiceStudio 嵌入**闭源或专有**产品或服务、又不受 AGPL-3.0 著佐权义务约束的组织，可获取**商业许可证**。**定价方案即将推出。** 咨询：**VoiceStudio@palash.dev**。
|
||||
|
||||
捆绑的 `omnivoice/` TTS 模型（作者 Han Zhu）在上游仍为 Apache-2.0 许可。完整且具约束力的条款请参见 [`LICENSE`](LICENSE)。
|
||||
|
||||
---
|
||||
|
||||
## 🙏 致谢
|
||||
|
||||
VoiceStudio 站在这些杰出开源工作的肩膀上：
|
||||
|
||||
| 项目 | 作用 |
|
||||
|---------|------|
|
||||
| [**VoiceStudio (k2-fsa)**](https://github.com/k2-fsa/OmniVoice) | 零样本扩散 TTS 引擎——核心语音合成模型 |
|
||||
| [**WhisperX**](https://github.com/m-bain/whisperX) | 词级别语音识别与时间对齐 |
|
||||
| [**Demucs (Meta)**](https://github.com/facebookresearch/demucs) | 音乐源分离，用于人声分离 |
|
||||
| [**Pyannote**](https://github.com/pyannote/pyannote-audio) | 说话人分离——谁说了什么 |
|
||||
| [**CTranslate2**](https://github.com/OpenNMT/CTranslate2) | CPU 和 GPU 上的优化 Transformer 推理 |
|
||||
| [**AudioSeal (Meta)**](https://github.com/facebookresearch/audioseal) | 用于 AI 溯源的不可见神经音频水印 |
|
||||
| [**Tauri**](https://tauri.app) | 原生桌面应用框架 |
|
||||
| [**Supertone / Supertonic 3**](https://huggingface.co/Supertone/supertonic-3) | ONNX TTS 引擎——31 种语言，CPU 高效 |
|
||||
| [**Sherpa-ONNX**](https://github.com/k2-fsa/sherpa-onnx) | 支持 WASM 的通用 TTS/ASR 运行时 |
|
||||
| [**GPT-SoVITS**](https://github.com/RVC-Boss/GPT-SoVITS) | 零样本 TTS 引擎——5 种语言，RTF 0.014 |
|
||||
|
||||
---
|
||||
|
||||
<a id="more-from-the-maker"></a>
|
||||
|
||||
## 🧰 来自同一作者的更多本地开源项目
|
||||
|
||||
喜欢这种本地优先的理念？它是一脉相承的——同一位作者，同一条准则：**你的数据只留在你的设备上。** 全部项目见 [palash.dev](https://palash.dev)。
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td align="center" width="50%" valign="top">
|
||||
<br/>
|
||||
<a href="https://github.com/debpalash/Opal"><img src="https://raw.githubusercontent.com/debpalash/Opal/main/assets/opal_logo.png" width="96" alt="Opal 徽标"/></a>
|
||||
<h3><a href="https://github.com/debpalash/Opal">Opal 💠</a></h3>
|
||||
<p><b>播放一切。</b>AI 时代的媒体播放器。</p>
|
||||
<p><sub>视频、动漫、漫画、种子、Jellyfin 和 Plex——一个播放器全部搞定，并内置本地 AI 记忆与上下文。使用 Zig 编写，支持 macOS 和 Windows。</sub></p>
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/Opal/stargazers"><img src="https://img.shields.io/github/stars/debpalash/Opal?style=flat-square&color=f59e0b" alt="Opal Star 数"/></a>
|
||||
<a href="https://palash.dev/opal"><img src="https://img.shields.io/badge/site-palash.dev%2Fopal-8b5cf6?style=flat-square" alt="Opal 官网"/></a>
|
||||
</p>
|
||||
</td>
|
||||
<td align="center" width="50%" valign="top">
|
||||
<br/>
|
||||
<a href="https://github.com/debpalash/memxt"><img src="https://raw.githubusercontent.com/debpalash/memxt/main/assets/logo-mark.svg" width="96" alt="memxt 徽标"/></a>
|
||||
<h3><a href="https://github.com/debpalash/memxt">memxt 🧠</a></h3>
|
||||
<p><b>经基准测试验证的最快开源 AI 记忆系统。</b></p>
|
||||
<p><sub>为 Claude Code 和编码智能体提供本地长期记忆——基于 SQLite + 嵌入向量的 MCP 服务器，100% 在你的设备上运行。你的智能体终于能记住昨天了。</sub></p>
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/memxt/stargazers"><img src="https://img.shields.io/github/stars/debpalash/memxt?style=flat-square&color=f59e0b" alt="memxt Star 数"/></a>
|
||||
<a href="https://github.com/debpalash/memxt#readme"><img src="https://img.shields.io/badge/docs-README-10b981?style=flat-square" alt="memxt 文档"/></a>
|
||||
</p>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
---
|
||||
|
||||
<div align="center">
|
||||
|
||||
<br/>
|
||||
|
||||
如果你读到了这里，你就是我们的同路人。<br/>
|
||||
**[⭐ 给这个仓库点个 Star](https://github.com/debpalash/VoiceStudio)**，让更多人能找到它。<br/>
|
||||
**[💬 加入 Discord](https://discord.gg/bzQavDfVV9)**，分享你的作品。<br/>
|
||||
**[❤️ 支持开发](https://ko-fi.com/debpalash)**——资助让 VoiceStudio 持续发布的 AI 智能体账单。
|
||||
|
||||
<br/>
|
||||
|
||||
<a href="https://star-history.com/#debpalash/VoiceStudio&Date">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=debpalash/VoiceStudio&type=Date&theme=dark" />
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=debpalash/VoiceStudio&type=Date" />
|
||||
<img alt="Star 历史" src="https://api.star-history.com/svg?repos=debpalash/VoiceStudio&type=Date&theme=dark" width="600" />
|
||||
</picture>
|
||||
</a>
|
||||
</div>
|
||||
应用采用 [AGPL-3.0](LICENSE) 许可。模型遵循各自的许可，商用前请确认其条款。克隆声音前须取得本人许可。详见[许可说明](LICENSE-NOTICE.md)。
|
||||
|
||||
+10
-6
@@ -1,25 +1,29 @@
|
||||
# Alembic configuration for VoiceStudio.
|
||||
# Run from anywhere: alembic -c <repo>/alembic.ini <command>
|
||||
# Default commands:
|
||||
# alembic upgrade head — apply all pending migrations
|
||||
# alembic revision -m "…" — create a new migration
|
||||
# alembic current — show current schema version
|
||||
# alembic upgrade head - apply all pending migrations
|
||||
# alembic revision -m "..." - create a new migration
|
||||
# alembic current - show current schema version
|
||||
#
|
||||
# Keep this file ASCII: alembic reads it in the locale code page, which on a
|
||||
# Chinese, Japanese or Korean Windows cannot decode UTF-8 punctuation
|
||||
# (tests/test_alembic_ini_locale.py).
|
||||
#
|
||||
# DB URL is resolved dynamically from core.config (env aware).
|
||||
# See backend/migrations/env.py.
|
||||
|
||||
[alembic]
|
||||
# %(here)s = this file's directory. Alembic resolves bare relative paths
|
||||
# against the process CWD, not the ini — and the app doesn't always start
|
||||
# against the process CWD, not the ini - and the app doesn't always start
|
||||
# from the repo root (`tauri dev` runs the backend with
|
||||
# cwd=frontend/src-tauri), which made startup migrations die with
|
||||
# "Path doesn't exist: backend/migrations" the first time one was pending.
|
||||
script_location = %(here)s/backend/migrations
|
||||
prepend_sys_path = %(here)s/backend
|
||||
# Split multi-path options on os.pathsep, not the legacy space/comma/colon
|
||||
# set — a colon-split would shred "C:\..." absolute paths on Windows.
|
||||
# set - a colon-split would shred "C:\..." absolute paths on Windows.
|
||||
path_separator = os
|
||||
# sqlalchemy.url is set programmatically in env.py — do NOT set it here.
|
||||
# sqlalchemy.url is set programmatically in env.py - do NOT set it here.
|
||||
sqlalchemy.url =
|
||||
|
||||
[loggers]
|
||||
|
||||
@@ -186,7 +186,8 @@ async def audiobook_import(file: UploadFile = File(...)) -> dict:
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=f"couldn't parse PDF: {e}")
|
||||
else:
|
||||
script = chapterize_plaintext(data.decode("utf-8", "ignore"))
|
||||
from services.text_upload import decode_text_upload
|
||||
script = chapterize_plaintext(decode_text_upload(data))
|
||||
if not script.strip():
|
||||
raise HTTPException(status_code=400, detail="no text found in the file")
|
||||
plan = parse_audiobook_script(script)
|
||||
|
||||
@@ -1030,11 +1030,14 @@ async def retry_batch_job(job_id: str):
|
||||
if not translation_engines.is_ready(provider):
|
||||
raise HTTPException(409, "Configure the selected translation provider before retrying")
|
||||
if provider == "argos" and job.get("source_lang"):
|
||||
status = await asyncio.to_thread(
|
||||
translation_engines.argos_pack_status,
|
||||
job["source_lang"],
|
||||
job["langs"],
|
||||
)
|
||||
try:
|
||||
status = await asyncio.to_thread(
|
||||
translation_engines.argos_pack_status,
|
||||
job["source_lang"],
|
||||
job["langs"],
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
if any(not pair["installed"] for pair in status["pairs"]):
|
||||
raise HTTPException(409, "Install the required Argos language packs before retrying")
|
||||
|
||||
|
||||
+108
-34
@@ -272,12 +272,10 @@ async def dub_import_srt(job_id: str, file: UploadFile = File(...)):
|
||||
raise HTTPException(status_code=400, detail=f"Could not read uploaded file: {e}") from e
|
||||
if not raw_bytes:
|
||||
raise HTTPException(status_code=400, detail="Uploaded SRT file is empty.")
|
||||
# Most SRT files are UTF-8 (with or without BOM); fall back to latin-1
|
||||
# so legacy Windows-encoded subs don't blow up the import.
|
||||
try:
|
||||
text = raw_bytes.decode("utf-8-sig")
|
||||
except UnicodeDecodeError:
|
||||
text = raw_bytes.decode("latin-1", errors="replace")
|
||||
# Most SRT files are UTF-8, but Windows subtitle tools also save UTF-16
|
||||
# (with a BOM) and Windows-1252; decode those instead of finding no cues.
|
||||
from services.text_upload import decode_text_upload
|
||||
text = decode_text_upload(raw_bytes)
|
||||
|
||||
from services.srt_parser import parse_srt
|
||||
result = parse_srt(text)
|
||||
@@ -869,9 +867,73 @@ _CHUNK_TRANSCRIBE_ATTEMPTS = max(1, int(os.environ.get("OMNIVOICE_TRANSCRIBE_CHU
|
||||
#: by Chrome's ~5 min no-response cap and by reverse-proxy idle timeouts,
|
||||
#: which the UI can only report as the generic "stream dropped" guess.
|
||||
ASR_LOAD_KEEPALIVE_S = float(os.environ.get("OMNIVOICE_ASR_LOAD_KEEPALIVE_S", "15.0"))
|
||||
#: Seconds between `ping` events while a post-transcript step runs on an
|
||||
#: executor (diarization, clone extraction, reference-text refinement, the
|
||||
#: ASR unload / TTS restore). #2108: the per-segment refinement re-runs ASR
|
||||
#: once per segment — 113 passes, ~19 min on an M1 Pro CPU — and was awaited
|
||||
#: bare, so the stream went byte-silent for the whole stretch, the webview
|
||||
#: severed it, and the UI could only say "stream ended early".
|
||||
POST_ASR_PING_S = 5.0
|
||||
|
||||
|
||||
_sse_event = dub_pipeline.sse_event
|
||||
|
||||
|
||||
class _ASRWorkLifetime:
|
||||
"""Keep model cleanup behind native work even after its waiter is cancelled."""
|
||||
|
||||
def __init__(self):
|
||||
import threading
|
||||
self._lock = threading.Lock()
|
||||
self._closed = threading.Event()
|
||||
self._cleaned = False
|
||||
|
||||
def run(self, fn):
|
||||
with self._lock:
|
||||
if self._closed.is_set():
|
||||
raise RuntimeError("Transcription stream has ended")
|
||||
return fn()
|
||||
|
||||
def stop(self):
|
||||
# Reject queued work before it can touch an unloaded model.
|
||||
self._closed.set()
|
||||
|
||||
def cleanup(self, fn):
|
||||
self.stop()
|
||||
with self._lock:
|
||||
if self._cleaned:
|
||||
return
|
||||
# Claim cleanup before invoking even a non-idempotent unload.
|
||||
self._cleaned = True
|
||||
return fn()
|
||||
|
||||
|
||||
async def _ping_while(fut):
|
||||
"""Yield `ping` events every POST_ASR_PING_S until ``fut`` settles.
|
||||
|
||||
Every await in the transcribe stream body that can outlast a few seconds
|
||||
goes through here so the connection never goes byte-silent. The result
|
||||
(or exception) stays on ``fut`` for the caller to read.
|
||||
|
||||
Leaving early — the client disconnected, or the body raised at a `yield` —
|
||||
cancels ``fut`` exactly as a bare ``await fut`` would have, so a wrapped
|
||||
run_transcribe_guarded still runs its abandon path instead of refining on
|
||||
after disconnection. Native threads are not cancelled by Future.cancel();
|
||||
_ASRWorkLifetime orders model cleanup behind their actual completion. Nothing is awaited in the finally: it also runs under GeneratorExit.
|
||||
A failure that lands after we left is still marked retrieved, so it cannot
|
||||
surface later as "exception was never retrieved" (CodeRabbit, #2138) —
|
||||
the same done-callback the TTS-load keepalive uses.
|
||||
"""
|
||||
fut.add_done_callback(lambda f: f.cancelled() or f.exception())
|
||||
try:
|
||||
while True:
|
||||
done, _ = await asyncio.wait({fut}, timeout=POST_ASR_PING_S)
|
||||
if done:
|
||||
return
|
||||
yield _sse_event("ping", {})
|
||||
finally:
|
||||
if not fut.done():
|
||||
fut.cancel()
|
||||
_prep_event_helper = dub_pipeline.prep_event # alias; we keep the module-local _prep_event below for the inline one-liner shape
|
||||
|
||||
#: User-facing warning emitted when auto voice cloning is skipped because the
|
||||
@@ -1045,6 +1107,7 @@ async def dub_transcribe_stream(
|
||||
# _gen_body parks the loaded backend here; the normal unload clears it;
|
||||
# gen()'s `finally` unloads whatever is still parked, on EVERY exit.
|
||||
_loaded_asr: dict = {"backend": None}
|
||||
_asr_work = _ASRWorkLifetime()
|
||||
# Same shape, same reason, for the TTS offload (#1191): offload_tts_for_asr()
|
||||
# moves the TTS model to CPU, and only _gen_body's success path moved it
|
||||
# back — so an abort/error/disconnect stranded it there, silently making
|
||||
@@ -1447,7 +1510,7 @@ async def dub_transcribe_stream(
|
||||
for _attempt in range(1, _CHUNK_TRANSCRIBE_ATTEMPTS + 1):
|
||||
# Run as a task and poll so pings keep the EventSource alive.
|
||||
task = asyncio.ensure_future(run_transcribe_guarded(
|
||||
_gpu_pool, _transcribe_chunk,
|
||||
_gpu_pool, lambda: _asr_work.run(_transcribe_chunk),
|
||||
what=f"Dub chunk {i + 1}/{chunks_n}",
|
||||
timeout=transcribe_timeout_s,
|
||||
timeout_env=transcribe_timeout_env,
|
||||
@@ -1855,16 +1918,10 @@ async def dub_transcribe_stream(
|
||||
"heuristic",
|
||||
)
|
||||
|
||||
fut_diar = loop.run_in_executor(_gpu_pool, _diarize)
|
||||
final_segs = None
|
||||
diar_warning = None
|
||||
labels_source = "heuristic"
|
||||
while True:
|
||||
done, pending = await asyncio.wait([fut_diar], timeout=5.0)
|
||||
if done:
|
||||
final_segs, diar_warning, labels_source = done.pop().result()
|
||||
break
|
||||
yield _sse_event("ping", {})
|
||||
fut_diar = loop.run_in_executor(_gpu_pool, lambda: _asr_work.run(_diarize))
|
||||
async for _ping in _ping_while(fut_diar):
|
||||
yield _ping
|
||||
final_segs, diar_warning, labels_source = fut_diar.result()
|
||||
if job.get("aborted") or task_manager.is_cancelled(job_id):
|
||||
yield _sse_event("aborted", {})
|
||||
return
|
||||
@@ -1882,6 +1939,8 @@ async def dub_transcribe_stream(
|
||||
payload["speaker_hint"] = diar_warning["speaker_hint"]
|
||||
yield _sse_event("warning", payload)
|
||||
|
||||
from services.segmentation import deduplicate_chunk_segments
|
||||
final_segs = deduplicate_chunk_segments(final_segs)
|
||||
job["segments"] = final_segs
|
||||
|
||||
# Auto-speaker-clone: sample each detected speaker's voice from the
|
||||
@@ -1929,12 +1988,9 @@ async def dub_transcribe_stream(
|
||||
labels_source=labels_source,
|
||||
),
|
||||
)
|
||||
while True:
|
||||
done, pending = await asyncio.wait([fut_clones], timeout=5.0)
|
||||
if done:
|
||||
clones = done.pop().result()
|
||||
break
|
||||
yield _sse_event("ping", {})
|
||||
async for _ping in _ping_while(fut_clones):
|
||||
yield _ping
|
||||
clones = fut_clones.result()
|
||||
if clones:
|
||||
from services.speaker_clone import refine_ref_texts
|
||||
# Bound the re-transcribe like every other ASR dispatch in
|
||||
@@ -1944,11 +2000,14 @@ async def dub_transcribe_stream(
|
||||
# and raises — keep the original (unrefined) clones, matching
|
||||
# refine_ref_text's own "failure is a strict no-op" fallback.
|
||||
try:
|
||||
clones = await run_transcribe_guarded(
|
||||
fut_refine = asyncio.ensure_future(run_transcribe_guarded(
|
||||
_gpu_pool,
|
||||
lambda: refine_ref_texts(clones, _asr_backend),
|
||||
lambda: _asr_work.run(lambda: refine_ref_texts(clones, _asr_backend)),
|
||||
what="Dub clone ref-text refine",
|
||||
)
|
||||
))
|
||||
async for _ping in _ping_while(fut_refine):
|
||||
yield _ping
|
||||
clones = fut_refine.result()
|
||||
except ASRTimeoutError as e:
|
||||
logger.warning(
|
||||
"clone ref-text refine timed out; keeping original ref_text: %s", e
|
||||
@@ -1971,23 +2030,29 @@ async def dub_transcribe_stream(
|
||||
# reviewers, on the first version of this fix).
|
||||
_seg_clone_dir = _safe_job_dir(job_id) or os.path.dirname(vocals_for_clone)
|
||||
os.makedirs(_seg_clone_dir, exist_ok=True)
|
||||
seg_clones = await loop.run_in_executor(
|
||||
fut_seg_refs = loop.run_in_executor(
|
||||
_cpu_pool, lambda: extract_segment_refs(
|
||||
vocals_for_clone, final_segs,
|
||||
_seg_clone_dir,
|
||||
seg_ids=seg_ids_for_clone,
|
||||
),
|
||||
)
|
||||
async for _ping in _ping_while(fut_seg_refs):
|
||||
yield _ping
|
||||
seg_clones = fut_seg_refs.result()
|
||||
if seg_clones:
|
||||
from services.speaker_clone import refine_ref_texts
|
||||
# Same guard as the per-speaker refine above (#730):
|
||||
# keep the original seg_clones on a wedge/timeout.
|
||||
try:
|
||||
seg_clones = await run_transcribe_guarded(
|
||||
fut_refine = asyncio.ensure_future(run_transcribe_guarded(
|
||||
_gpu_pool,
|
||||
lambda: refine_ref_texts(seg_clones, _asr_backend),
|
||||
lambda: _asr_work.run(lambda: refine_ref_texts(seg_clones, _asr_backend)),
|
||||
what="Dub segment ref-text refine",
|
||||
)
|
||||
))
|
||||
async for _ping in _ping_while(fut_refine):
|
||||
yield _ping
|
||||
seg_clones = fut_refine.result()
|
||||
except ASRTimeoutError as e:
|
||||
logger.warning(
|
||||
"segment ref-text refine timed out; keeping original ref_text: %s", e
|
||||
@@ -2042,13 +2107,19 @@ async def dub_transcribe_stream(
|
||||
# (CodeRabbit review, #1198 — normal-completion half).
|
||||
if _asr_backend:
|
||||
try:
|
||||
await loop.run_in_executor(_gpu_pool, _asr_backend.unload)
|
||||
fut_unload = loop.run_in_executor(_gpu_pool, lambda: _asr_work.cleanup(_asr_backend.unload))
|
||||
async for _ping in _ping_while(fut_unload):
|
||||
yield _ping
|
||||
fut_unload.result()
|
||||
except Exception as e:
|
||||
logger.warning("Failed to unload ASR backend: %s", e)
|
||||
# Unload attempted once — don't retry from gen()'s finally.
|
||||
_loaded_asr["backend"] = None
|
||||
|
||||
await loop.run_in_executor(_cpu_pool, restore_tts_after_asr)
|
||||
fut_restore = loop.run_in_executor(_cpu_pool, restore_tts_after_asr)
|
||||
async for _ping in _ping_while(fut_restore):
|
||||
yield _ping
|
||||
fut_restore.result()
|
||||
# Debt paid — don't make gen()'s finally repeat it.
|
||||
_tts_offloaded["v"] = False
|
||||
|
||||
@@ -2090,6 +2161,7 @@ async def dub_transcribe_stream(
|
||||
yield _sse_event("error", stream_failure("transcription_failed"))
|
||||
yield _sse_event("done", {})
|
||||
finally:
|
||||
_asr_work.stop()
|
||||
# Last-resort VRAM release (see _loaded_asr above): covers crashes,
|
||||
# early terminal-error returns, and client disconnects
|
||||
# (GeneratorExit bypasses the except, never this finally).
|
||||
@@ -2115,7 +2187,7 @@ async def dub_transcribe_stream(
|
||||
# (CodeRabbit review, #1198).
|
||||
try:
|
||||
_fut = asyncio.get_running_loop().run_in_executor(
|
||||
_gpu_pool, _b.unload
|
||||
_gpu_pool, lambda: _asr_work.cleanup(_b.unload)
|
||||
)
|
||||
# Restore the TTS model only AFTER the ASR weights are
|
||||
# freed — the same ordering the success path enforces, so
|
||||
@@ -2124,7 +2196,7 @@ async def dub_transcribe_stream(
|
||||
except RuntimeError:
|
||||
# No running loop (interpreter teardown) — best effort.
|
||||
try:
|
||||
_b.unload()
|
||||
_asr_work.cleanup(_b.unload)
|
||||
except Exception as e:
|
||||
logger.warning("Failed to unload ASR backend: %s", e)
|
||||
_submit_tts_restore()
|
||||
@@ -2294,6 +2366,8 @@ async def dub_transcribe(job_id: str, num_speakers: Optional[int] = None):
|
||||
raise
|
||||
if job.get("aborted"):
|
||||
raise HTTPException(status_code=499, detail="Transcription aborted")
|
||||
from services.segmentation import deduplicate_chunk_segments
|
||||
segments_result = deduplicate_chunk_segments(segments_result)
|
||||
job["segments"] = segments_result
|
||||
source_lang = job.get("source_lang")
|
||||
_save_job(job_id, job)
|
||||
|
||||
@@ -38,6 +38,31 @@ router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.api")
|
||||
|
||||
|
||||
async def _preserved_background(job: dict, job_id: str, lang: str, *, prepare: bool = True) -> str:
|
||||
"""All mixed preview/download paths share the same dialogue-only bed."""
|
||||
from services.dub_background import surgical_background
|
||||
|
||||
bed = _optional_dub_artifact(job.get("no_vocals_path"), job_id)
|
||||
source = _optional_dub_artifact(job.get("video_path"), job_id) or _optional_dub_artifact(job.get("audio_path"), job_id)
|
||||
if not bed or not source:
|
||||
raise HTTPException(status_code=409, detail={"code": "dub_background_unavailable", "message": "Original audio and background separation are required"})
|
||||
track = (job.get("dubbed_tracks") or {}).get(lang) or {}
|
||||
segments = track.get("source_segments") or job.get("segments") or []
|
||||
if not segments:
|
||||
raise HTTPException(status_code=409, detail={"code": "dub_background_unavailable", "message": "Dialogue timing is required"})
|
||||
if not prepare:
|
||||
return bed
|
||||
strategy = track.get("timing_strategy") or job.get("timing_strategy")
|
||||
plans = job.get("fit_plans" if strategy == "smart_fit" else "video_stretch_plans") or {}
|
||||
entry = (plans.get(lang) or {}) if strategy in {"smart_fit", "stretch_video"} else {}
|
||||
directory = os.path.join(_existing_job_dir_or_404(job_id), "exports")
|
||||
os.makedirs(directory, exist_ok=True)
|
||||
try:
|
||||
return await surgical_background(source, bed, directory, segments, entry.get("plan") or [], float(entry.get("orig_duration") or job.get("duration") or 0))
|
||||
except (ValueError, RuntimeError) as exc:
|
||||
raise HTTPException(status_code=409, detail={"code": "dub_background_unavailable", "message": str(exc)}) from exc
|
||||
|
||||
|
||||
def _unique_stamp() -> str:
|
||||
"""Return a short unique suffix like '20260415T142301-ab12cd34' for export files."""
|
||||
return f"{time.strftime('%Y%m%dT%H%M%S')}-{uuid.uuid4().hex[:8]}"
|
||||
@@ -45,6 +70,13 @@ def _unique_stamp() -> str:
|
||||
|
||||
_SAFE_LANG = re.compile(r"^[A-Za-z0-9_-]{1,32}$")
|
||||
|
||||
#: Seconds of silence on a `/tasks/stream` before a keepalive comment goes out.
|
||||
#: A task that is busy but quiet — ffmpeg on a long video, a slow TTS segment,
|
||||
#: a job queued behind another — leaves the stream byte-silent, and byte-silent
|
||||
#: SSE gets severed by the desktop webview, Chrome's ~5 min cap or a proxy's
|
||||
#: idle timeout (#1196, #2108). Comments are invisible to every consumer.
|
||||
TASK_STREAM_KEEPALIVE_S = 15.0
|
||||
|
||||
|
||||
def _job_dir_or_400(job_id: str) -> str:
|
||||
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", job_id or ""):
|
||||
@@ -248,14 +280,21 @@ async def stream_task(task_id: str, after_seq: int = 0):
|
||||
await task_manager.add_listener(task_id, q)
|
||||
try:
|
||||
while True:
|
||||
evt = await q.get()
|
||||
try:
|
||||
evt = await asyncio.wait_for(q.get(), timeout=TASK_STREAM_KEEPALIVE_S)
|
||||
except asyncio.TimeoutError:
|
||||
yield ": keepalive\n\n"
|
||||
continue
|
||||
if evt is None:
|
||||
break
|
||||
yield evt
|
||||
finally:
|
||||
await task_manager.remove_listener(task_id, q)
|
||||
|
||||
return StreamingResponse(_reader(), media_type="text/event-stream")
|
||||
return StreamingResponse(
|
||||
_reader(), media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache, no-transform", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/jobs")
|
||||
@@ -596,7 +635,7 @@ def _build_audio_export_cmd(
|
||||
# Mix the dubbed voice over the original background bed (same weights
|
||||
# as the video mux path) so ambience/music is preserved.
|
||||
cmd += ["-i", bg_path, "-filter_complex",
|
||||
bed_mix_filter("1:a", "0:a"),
|
||||
bed_mix_filter("1:a", "0:a", bed_gain=1.0),
|
||||
"-map", "[aout]"]
|
||||
cmd += codec
|
||||
cmd.append(out_path)
|
||||
@@ -691,7 +730,7 @@ async def dub_download(
|
||||
else:
|
||||
output_name = f"dubbed_audio_{stamp}.m4a"
|
||||
out_path = os.path.join(exports_dir, output_name)
|
||||
bg = _optional_dub_artifact(job.get("no_vocals_path"), job_id) if preserve_bg else None
|
||||
bg = await _preserved_background(job, job_id, lang_code) if preserve_bg else None
|
||||
cmd = _build_audio_export_cmd(ffmpeg, track_info["path"], bg, out_path, fmt)
|
||||
try:
|
||||
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0)
|
||||
@@ -839,17 +878,16 @@ async def dub_download(
|
||||
retimed_idx = input_idx
|
||||
input_idx += 1
|
||||
|
||||
bg_audio = _optional_dub_artifact(job.get("no_vocals_path"), job_id) if preserve_bg else None
|
||||
bg_idx = None
|
||||
if bg_audio and filtered_tracks:
|
||||
cmd += ["-i", bg_audio]
|
||||
bg_idx = input_idx
|
||||
input_idx += 1
|
||||
|
||||
tracks_to_process = []
|
||||
for lang_code, track_info in filtered_tracks.items():
|
||||
if preserve_bg:
|
||||
bg_audio = await _preserved_background(job, job_id, lang_code)
|
||||
cmd += ["-i", bg_audio]
|
||||
bg_idx = input_idx
|
||||
input_idx += 1
|
||||
cmd += ["-i", track_info["path"]]
|
||||
tracks_to_process.append({"lang_code": lang_code, "idx": input_idx, "info": track_info})
|
||||
tracks_to_process.append({"lang_code": lang_code, "idx": input_idx, "bg_idx": bg_idx, "info": track_info})
|
||||
input_idx += 1
|
||||
|
||||
filter_parts: list[str] = []
|
||||
@@ -918,7 +956,7 @@ async def dub_download(
|
||||
for i, t in enumerate(tracks_to_process):
|
||||
tail = f",apad=whole_dur={apad_dur:.4f}" if apad_dur else ""
|
||||
filter_parts.append(bed_mix_filter(
|
||||
f"{bg_idx}:a", f"{t['idx']}:a", out=f"aout{i}", tail=tail, uniq=str(i),
|
||||
f"{t['bg_idx']}:a", f"{t['idx']}:a", out=f"aout{i}", tail=tail, uniq=str(i), bed_gain=1.0,
|
||||
))
|
||||
t["out_label"] = f"[aout{i}]"
|
||||
for t in tracks_to_process:
|
||||
@@ -1119,7 +1157,7 @@ async def dub_preview_video(
|
||||
|
||||
video_path = _dub_artifact(job.get("video_path"), job_id, missing_detail="Source video missing")
|
||||
|
||||
bg_audio = _optional_dub_artifact(job.get("no_vocals_path"), job_id) if preserve_bg else None
|
||||
bg_audio = await _preserved_background(job, job_id, lang, prepare=request.method != "HEAD") if preserve_bg else None
|
||||
has_bg = bool(bg_audio)
|
||||
|
||||
# realpath-normalised + containment-checked inline BEFORE any filesystem
|
||||
@@ -1131,7 +1169,7 @@ async def dub_preview_video(
|
||||
if not exports_dir.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid job id")
|
||||
os.makedirs(exports_dir, exist_ok=True)
|
||||
bg_suffix = "bg" if (preserve_bg and has_bg) else "nobg"
|
||||
bg_suffix = "surgical_v2_" + Path(bg_audio).stem if (preserve_bg and has_bg) else "nobg"
|
||||
preview_path = os.path.realpath(
|
||||
os.path.join(exports_dir, f"preview_v2_{lang}_{bg_suffix}.mp4")
|
||||
)
|
||||
@@ -1270,7 +1308,7 @@ async def dub_preview_video(
|
||||
audio_map = f"{track_idx}:a:0"
|
||||
if bg_idx is not None:
|
||||
tail = f",apad=whole_dur={apad_dur:.4f}" if apad_dur else ""
|
||||
filter_parts.append(bed_mix_filter(f"{bg_idx}:a", f"{track_idx}:a", tail=tail))
|
||||
filter_parts.append(bed_mix_filter(f"{bg_idx}:a", f"{track_idx}:a", tail=tail, bed_gain=1.0))
|
||||
audio_map = "[aout]"
|
||||
elif apad_dur:
|
||||
filter_parts.append(f"[{track_idx}:a]apad=whole_dur={apad_dur:.4f}[aout]")
|
||||
@@ -1661,13 +1699,13 @@ async def dub_download_audio(
|
||||
exports_dir = os.path.join(job_dir, "exports")
|
||||
os.makedirs(exports_dir, exist_ok=True)
|
||||
|
||||
bg_audio = _optional_dub_artifact(job.get("no_vocals_path"), job_id) if preserve_bg else None
|
||||
bg_audio = await _preserved_background(job, job_id, lang_label) if preserve_bg else None
|
||||
if bg_audio:
|
||||
ffmpeg = find_ffmpeg()
|
||||
final_audio_path = os.path.join(exports_dir, f"mixed_dub_{stamp}.wav")
|
||||
cmd = [
|
||||
ffmpeg, "-i", bg_audio, "-i", wav_path,
|
||||
"-filter_complex", bed_mix_filter("0:a", "1:a"),
|
||||
"-filter_complex", bed_mix_filter("0:a", "1:a", bed_gain=1.0),
|
||||
"-map", "[aout]", "-c:a", "pcm_s16le", "-y", final_audio_path
|
||||
]
|
||||
try:
|
||||
@@ -1678,8 +1716,9 @@ async def dub_download_audio(
|
||||
raise Exception("ffmpeg mix produced no output file")
|
||||
wav_path = final_audio_path
|
||||
logger.info("Dub audio mix completed")
|
||||
except Exception:
|
||||
except Exception as exc:
|
||||
logger.exception("Failed to mix audio")
|
||||
raise HTTPException(status_code=500, detail={"code": "dub_background_unavailable", "message": "Could not preserve background audio"}) from exc
|
||||
|
||||
base_name = os.path.splitext(job.get('filename', 'audio'))[0]
|
||||
safe_name = ''.join(c for c in base_name if c.isalnum() or c in '-_ ').strip() or 'audio'
|
||||
@@ -1697,11 +1736,8 @@ async def dub_download_audio(
|
||||
|
||||
|
||||
def _format_srt_time(seconds):
|
||||
h = int(seconds // 3600)
|
||||
m = int((seconds % 3600) // 60)
|
||||
s = int(seconds % 60)
|
||||
ms = int((seconds % 1) * 1000)
|
||||
return f"{h:02d}:{m:02d}:{s:02d},{ms:03d}"
|
||||
from services.srt_parser import format_cue_timestamp
|
||||
return format_cue_timestamp(seconds, ",")
|
||||
|
||||
def _pick_subtitle_text(seg: dict, dual: bool) -> str:
|
||||
"""One line per subtitle cue, unless dual=true and an original exists.
|
||||
@@ -1785,11 +1821,8 @@ async def dub_export_srt(
|
||||
)
|
||||
|
||||
def _format_vtt_time(seconds):
|
||||
h = int(seconds // 3600)
|
||||
m = int((seconds % 3600) // 60)
|
||||
s = int(seconds % 60)
|
||||
ms = int((seconds % 1) * 1000)
|
||||
return f"{h:02d}:{m:02d}:{s:02d}.{ms:03d}"
|
||||
from services.srt_parser import format_cue_timestamp
|
||||
return format_cue_timestamp(seconds, ".")
|
||||
|
||||
@router.get("/dub/vtt/{job_id}")
|
||||
@router.get("/dub/vtt/{job_id}/{filename}")
|
||||
@@ -1947,20 +1980,23 @@ async def dub_download_mp3(
|
||||
os.makedirs(exports_dir, exist_ok=True)
|
||||
|
||||
source_path = wav_path
|
||||
bg_audio = _optional_dub_artifact(job.get("no_vocals_path"), job_id) if preserve_bg else None
|
||||
bg_audio = await _preserved_background(job, job_id, lang_label) if preserve_bg else None
|
||||
if bg_audio:
|
||||
mixed_path = os.path.join(exports_dir, f"mixed_mp3_{stamp}.wav")
|
||||
cmd_mix = [
|
||||
ffmpeg, "-i", bg_audio, "-i", wav_path,
|
||||
"-filter_complex", bed_mix_filter("0:a", "1:a"),
|
||||
"-filter_complex", bed_mix_filter("0:a", "1:a", bed_gain=1.0),
|
||||
"-map", "[aout]", "-c:a", "pcm_s16le", "-y", mixed_path
|
||||
]
|
||||
try:
|
||||
rc, _, _ = await run_ffmpeg(cmd_mix, timeout=900.0)
|
||||
if rc == 0 and os.path.exists(mixed_path) and os.path.getsize(mixed_path) > 0:
|
||||
source_path = mixed_path
|
||||
except Exception:
|
||||
else:
|
||||
raise RuntimeError("Background mixing failed")
|
||||
except Exception as exc:
|
||||
logger.exception("Failed to mix audio for MP3")
|
||||
raise HTTPException(status_code=500, detail={"code": "dub_background_unavailable", "message": "Could not preserve background audio"}) from exc
|
||||
|
||||
mp3_path = os.path.join(exports_dir, f"dubbed_{stamp}.mp3")
|
||||
# Accept '128', '192k' etc. — normalize to ffmpeg's 'Nk' form and clamp
|
||||
|
||||
@@ -30,7 +30,7 @@ from services.ffmpeg_utils import (
|
||||
)
|
||||
from services.rvc import apply_rvc, is_enabled as rvc_is_enabled
|
||||
from services.incremental import segment_fingerprint, fit_fingerprint
|
||||
from services.fit_planner import UNDERRUN_TOLERANCE, FitParams, plan_fit
|
||||
from services.fit_planner import FitParams, plan_fit
|
||||
from services.watermark import mark_synthetic
|
||||
from services.speaker_clone import auto_profile_id
|
||||
from services.segment_bundle import extract_segment_wavs
|
||||
@@ -39,16 +39,6 @@ from omnivoice.utils.voice_design import heal_design_instruct
|
||||
|
||||
logger = logging.getLogger("omnivoice.dub")
|
||||
|
||||
# Maximum compression ratio we'll attempt with pitch-preserving stretch
|
||||
# before declaring "no way to fit cleanly" and falling back. atempo
|
||||
# remains intelligible up to ~1.5× then introduces audible WSOLA
|
||||
# artefacts; above ~1.8× speech becomes a fast garbled stream that no
|
||||
# DSP can rescue. The contributing-factor pipeline (CPS-aware slot-fit
|
||||
# in services/speech_rate.py, gap absorption below) keeps us under this
|
||||
# in practice — this is only a guard rail.
|
||||
MAX_STRETCH_RATIO = 1.8
|
||||
|
||||
|
||||
class _RemoteDubBackend:
|
||||
"""Sample-rate carrier while Dubbing runs without local TTS weights."""
|
||||
|
||||
@@ -378,6 +368,12 @@ def forget_missing_ref_warnings(job_id: str) -> None:
|
||||
_MISSING_REF_WARNED.pop(str(job_id), None)
|
||||
|
||||
|
||||
|
||||
def _ref_within_limit(info) -> bool:
|
||||
from services.speaker_clone import MAX_REF_DURATION_S
|
||||
return bool(info) and float(info.get("duration") or 0.0) <= MAX_REF_DURATION_S
|
||||
|
||||
|
||||
def resolve_consistent_ref(job: dict, speaker_key: str, memo: dict | None = None):
|
||||
"""ONE clone reference for every segment of `speaker_key`.
|
||||
|
||||
@@ -386,8 +382,9 @@ def resolve_consistent_ref(job: dict, speaker_key: str, memo: dict | None = None
|
||||
the per-line path uses as its fallback;
|
||||
2. no speaker clone (heuristic diarization skips extraction entirely —
|
||||
the key case): a deterministic pick among that speaker's per-segment
|
||||
clips: longest clip ≥3 s, tie-break lowest segment id. Clips all
|
||||
shorter than 3 s degrade to "longest overall", same tie-break.
|
||||
clips: longest usable clip ≥3 s within the shared reference limit,
|
||||
tie-break lowest segment id. Short clips fall back to longest usable.
|
||||
Oversized references must never strand every short line for a speaker.
|
||||
|
||||
Returns the clone info dict ({"ref_audio", "ref_text", ...}) or None.
|
||||
Pure function of the job dict; `memo` (keyed by speaker_key) just avoids
|
||||
@@ -396,7 +393,11 @@ def resolve_consistent_ref(job: dict, speaker_key: str, memo: dict | None = None
|
||||
if memo is not None and speaker_key in memo:
|
||||
return memo[speaker_key]
|
||||
|
||||
from services.speaker_clone import MAX_REF_DURATION_S
|
||||
|
||||
ref = _find_speaker_clone(job.get("speaker_clones") or {}, speaker_key)
|
||||
if ref and float(ref.get("duration") or 0.0) > MAX_REF_DURATION_S:
|
||||
ref = None
|
||||
if ref is None:
|
||||
seg_clones = job.get("segment_clones") or {}
|
||||
candidates = []
|
||||
@@ -408,7 +409,8 @@ def resolve_consistent_ref(job: dict, speaker_key: str, memo: dict | None = None
|
||||
continue
|
||||
sid = str(row.get("id", ""))
|
||||
info = seg_clones.get(sid)
|
||||
if info and info.get("ref_audio"):
|
||||
if (info and info.get("ref_audio")
|
||||
and float(info.get("duration") or 0.0) <= MAX_REF_DURATION_S):
|
||||
candidates.append((sid, info))
|
||||
if candidates:
|
||||
usable = [
|
||||
@@ -452,6 +454,9 @@ def _remote_voice(job: dict, profile_id: str | None, seg_id, voice_match: str,
|
||||
info = ((job.get("segment_clones") or {}).get(str(seg_id))
|
||||
or _find_speaker_clone(job.get("speaker_clones") or {}, key))
|
||||
single_use = str(seg_id) in (job.get("segment_clones") or {})
|
||||
if not _ref_within_limit(info):
|
||||
info = resolve_consistent_ref(job, key, memo)
|
||||
single_use = False
|
||||
if info:
|
||||
ref_audio, ref_text = info.get("ref_audio"), info.get("ref_text")
|
||||
elif profile_id:
|
||||
@@ -701,8 +706,29 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
_wav_kind = (
|
||||
_kind_map.get(lang_code) if isinstance(_kind_map, dict) else job.get("seg_wav_kind")
|
||||
)
|
||||
if strategy != "strict_slot" and regen_only is not None and _wav_kind != "natural":
|
||||
if regen_only is not None and _wav_kind != "natural":
|
||||
regen_only = None
|
||||
# A partial rerun must repair absent or corrupt caches, including before
|
||||
# remote batching decides which lines need synthesis.
|
||||
if regen_only is not None:
|
||||
for index, segment in enumerate(req.segments):
|
||||
sid = seg_ids[index] if index < len(seg_ids) else f"seg_{index}"
|
||||
if sid in regen_only or not segment.text.strip():
|
||||
continue
|
||||
cache = _seg_lang_path(sid)
|
||||
if not os.path.exists(cache) and _legacy_seg_cache_ok(job, lang_code):
|
||||
for key in (sid, index):
|
||||
legacy = dub_seg_path(job_id, key)
|
||||
if os.path.exists(legacy):
|
||||
cache = legacy
|
||||
break
|
||||
try:
|
||||
info = torchaudio.info(cache)
|
||||
intact = info.num_frames > 0 and _cached_payload_intact(cache, info)
|
||||
except Exception:
|
||||
intact = False
|
||||
if not intact:
|
||||
regen_only.add(sid)
|
||||
# Manifest: stable segment id per current index. Per-segment WAVs are
|
||||
# named by stable id (dub_seg_path) so regen reuses the right audio after
|
||||
# reorder; index-keyed readers (preview/export) resolve via this manifest.
|
||||
@@ -955,7 +981,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
all_segment_wavs.append(
|
||||
(seg.start, seg.end, seg_wav_path, backend.sample_rate)
|
||||
)
|
||||
sync_scores.append(getattr(seg, 'sync_ratio', None) or 1.0)
|
||||
sync_scores.append(round(cached_info.num_frames / cached_info.sample_rate / max(seg_duration, 0.01), 3))
|
||||
_t_cache += time.perf_counter() - _t_cache_0
|
||||
continue
|
||||
|
||||
@@ -963,39 +989,20 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
if cached_sr != backend.sample_rate:
|
||||
import torchaudio.functional as AF
|
||||
cached_wav = AF.resample(cached_wav, cached_sr, backend.sample_rate)
|
||||
# strict_slot persists slot-sized buffers. Every other
|
||||
# strategy consumes natural-rate audio and lets the mix
|
||||
# loop fit it to the current timeline.
|
||||
if strategy == "strict_slot":
|
||||
target_samples = int(seg_duration * backend.sample_rate)
|
||||
current_samples = cached_wav.shape[-1]
|
||||
if target_samples > current_samples:
|
||||
cached_wav = torch.nn.functional.pad(cached_wav, (0, target_samples - current_samples))
|
||||
elif current_samples > target_samples:
|
||||
cached_wav = cached_wav[..., :target_samples]
|
||||
cached_ratio = round(cached_wav.shape[-1] / backend.sample_rate / max(seg_duration, 0.01), 3)
|
||||
all_segment_wavs.append(_store_mix_wav(seg.start, seg.end, cached_wav, backend.sample_rate, f"mix_{seg_id}"))
|
||||
try:
|
||||
del cached_wav
|
||||
except Exception:
|
||||
pass
|
||||
_release_audio_tensors()
|
||||
sync_scores.append(getattr(seg, 'sync_ratio', None) or 1.0)
|
||||
sync_scores.append(cached_ratio)
|
||||
_t_cache += time.perf_counter() - _t_cache_0
|
||||
continue
|
||||
except Exception as e:
|
||||
# Fall through to a silent placeholder if the cached WAV
|
||||
# is broken — cleaner than aborting the whole mix.
|
||||
yield f"data: {json.dumps({'type': 'warning', 'segment': i, 'message': f'cached seg lost, padding silence: {str(e)[:120]}'})}\n\n"
|
||||
sr = backend.sample_rate
|
||||
silence = torch.zeros(1, max(0, int(seg_duration * sr)))
|
||||
all_segment_wavs.append(_store_mix_wav(seg.start, seg.end, silence, sr, f"mix_{seg_id}"))
|
||||
try:
|
||||
del silence
|
||||
except Exception:
|
||||
pass
|
||||
_release_audio_tensors()
|
||||
sync_scores.append(1.0)
|
||||
continue
|
||||
except Exception:
|
||||
logger.exception("Dub cached segment could not be read: %s", seg_id)
|
||||
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'segment_id': seg_id, 'error_code': 'dub_speech_missing', 'error': 'Cached speech could not be read. Regenerate this segment before exporting.'})}\n\n"
|
||||
return
|
||||
|
||||
def _gen(text, lang, instruct_str, dur_s, nstep, cfg, spd, profile_id, effect_preset,
|
||||
*, execution_target="local", prepare_only=False, current_seg_id=None):
|
||||
@@ -1092,7 +1099,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
if selected_is_segment_speaker
|
||||
else None
|
||||
)
|
||||
if seg_ref:
|
||||
if _ref_within_limit(seg_ref):
|
||||
ref_audio = seg_ref.get("ref_audio")
|
||||
ref_text = seg_ref.get("ref_text")
|
||||
ref_single_use = True
|
||||
@@ -1100,7 +1107,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
auto = _find_speaker_clone(
|
||||
job.get("speaker_clones") or {}, key
|
||||
)
|
||||
if auto is None:
|
||||
if not _ref_within_limit(auto):
|
||||
# Short lines may have no line-specific clip.
|
||||
# Reuse this speaker's best source instead of
|
||||
# silently reverting to the engine default.
|
||||
@@ -1458,27 +1465,17 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
yield f"data: {json.dumps({'type': 'cancelled', 'segments_processed': i + 1})}\n\n"
|
||||
return
|
||||
|
||||
target_samples = int(seg_duration * backend.sample_rate)
|
||||
current_samples = audio_tensor.shape[-1]
|
||||
# Capture the real spoken duration before strict-slot padding
|
||||
# or trimming. This is the evidence used by Agent timing and
|
||||
# Capture the real spoken duration before assembly fitting.
|
||||
# This is the evidence used by Agent timing and
|
||||
# keeps sync badges truthful for every timing strategy.
|
||||
natural_generated_dur = current_samples / backend.sample_rate
|
||||
|
||||
if strategy == "strict_slot":
|
||||
# Legacy: pad short audio + trim long audio so the mix
|
||||
# loop receives slot-sized buffers. The atempo squeeze
|
||||
# in the mix loop never fires here because we already
|
||||
# forced size = target_samples.
|
||||
if target_samples > current_samples:
|
||||
pad_amount = target_samples - current_samples
|
||||
audio_tensor = torch.nn.functional.pad(audio_tensor, (0, pad_amount))
|
||||
elif current_samples > target_samples:
|
||||
audio_tensor = audio_tensor[..., :target_samples]
|
||||
# concise / stretch_video / smart_fit: keep audio at its
|
||||
# natural length. The mix loop decides per-mode whether to
|
||||
# trim, slip, stretch the video, or split audio/video
|
||||
# retiming (smart_fit) to accommodate it.
|
||||
# Keep the complete waveform in every cache. Fitting happens
|
||||
# once during assembly; pre-trimming here destroyed words before
|
||||
# the pitch-preserving stretcher could see them.
|
||||
if current_samples == 0 or not torch.isfinite(audio_tensor).all() or not torch.any(audio_tensor.abs() > 1e-6):
|
||||
raise ValueError("The speech engine returned empty or silent audio")
|
||||
|
||||
generated_dur = natural_generated_dur
|
||||
sync_ratio = round(generated_dur / max(seg_duration, 0.01), 3)
|
||||
@@ -1488,7 +1485,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
# Duration-planner calibration sample: this text length spoke
|
||||
# for this long at natural rate. Keyed by stable seg id and
|
||||
# merged into the per-language job map after the loop.
|
||||
if strategy != "strict_slot" and seg.text.strip() and generated_dur > 0:
|
||||
if seg.text.strip() and generated_dur > 0:
|
||||
_natural_dur_records[str(seg_id)] = {
|
||||
"chars": len(seg.text.strip()),
|
||||
"dur": round(generated_dur, 4),
|
||||
@@ -1526,15 +1523,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
if rvc_sr == backend.sample_rate:
|
||||
audio_tensor = rvc_wav
|
||||
|
||||
if strategy == "strict_slot":
|
||||
target_samples = int(seg_duration * backend.sample_rate)
|
||||
current_samples = audio_tensor.shape[-1]
|
||||
if target_samples > current_samples:
|
||||
audio_tensor = torch.nn.functional.pad(
|
||||
audio_tensor, (0, target_samples - current_samples)
|
||||
)
|
||||
elif current_samples > target_samples:
|
||||
audio_tensor = audio_tensor[..., :target_samples]
|
||||
except Exception as e:
|
||||
yield f"data: {json.dumps({'type': 'warning', 'segment': i, 'message': f'RVC skipped: {str(e)[:120]}'})}\n\n"
|
||||
|
||||
@@ -1581,10 +1569,9 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
from core.public_errors import stream_generation_failure
|
||||
|
||||
error_detail = stream_generation_failure(e)["detail"]
|
||||
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'error': error_detail})}\n\n"
|
||||
sr = backend.sample_rate
|
||||
all_segment_wavs.append(_store_mix_wav(seg.start, seg.end, torch.zeros(1, max(0, int(seg_duration * sr))), sr, f"mix_{seg_id}"))
|
||||
sync_scores.append(1.0)
|
||||
logger.exception("Dub generation failed for segment %s", seg_id)
|
||||
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'segment_id': seg_id, 'error': error_detail})}\n\n"
|
||||
return
|
||||
|
||||
_t_loop_end = time.perf_counter()
|
||||
|
||||
@@ -1731,19 +1718,16 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
seg_gain = max(0.0, min(2.0, seg_gain))
|
||||
try:
|
||||
wav = _load_entry_wav((start, end, wav_path, sr), sr)
|
||||
except Exception as e:
|
||||
# A WAV header can be readable while its payload is
|
||||
# truncated. Direct cache reuse deliberately defers the
|
||||
# decode to assembly, so preserve the old recovery contract
|
||||
# here: warn and fill this slot with silence instead of
|
||||
# aborting the entire dub.
|
||||
warning = {
|
||||
"type": "warning",
|
||||
"segment": i,
|
||||
"message": f"cached seg lost, padding silence: {str(e)[:120]}",
|
||||
}
|
||||
yield f"data: {json.dumps(warning)}\n\n"
|
||||
wav = torch.zeros(1, max(0, int((end - start) * sr)))
|
||||
except Exception:
|
||||
logger.exception("Dub assembly could not read segment %d", i)
|
||||
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'error_code': 'dub_speech_missing', 'error': 'A speech segment could not be read. Regenerate it before exporting.'})}\n\n"
|
||||
return
|
||||
from services.audio_dsp import trim_speech_padding
|
||||
if seg_ref is not None and seg_ref.text.strip():
|
||||
if wav.numel() == 0 or not torch.isfinite(wav).all() or not torch.any(wav.abs() > 1e-6):
|
||||
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'error_code': 'dub_speech_missing', 'error': 'A speech segment is empty or silent. Regenerate it before exporting.'})}\n\n"
|
||||
return
|
||||
wav = trim_speech_padding(wav, sr)
|
||||
adjusted = wav * seg_gain
|
||||
if adjusted.ndim == 2 and adjusted.shape[0] > 1:
|
||||
adjusted = adjusted.mean(dim=0, keepdim=True)
|
||||
@@ -1791,12 +1775,12 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
align_corners=False,
|
||||
).squeeze(0)
|
||||
wl = adjusted.shape[-1]
|
||||
# Residual overflow → hard-trim to the segment's new video
|
||||
# slot (fade below keeps the cut pop-free).
|
||||
# Never publish a complete track with speech discarded by
|
||||
# the fit caps. The user can shorten text or relax the caps.
|
||||
new_slot_samples = int(max(0.0, sf.new_end - sf.new_start) * sr)
|
||||
if new_slot_samples > 0 and wl > new_slot_samples:
|
||||
adjusted = adjusted[..., :new_slot_samples]
|
||||
wl = adjusted.shape[-1]
|
||||
if new_slot_samples > 0 and wl > new_slot_samples + int(sr * 0.02):
|
||||
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'error_code': 'dub_timing_overflow', 'error': 'Speech exceeds the fitting limits. Shorten the translation or choose Strict Slot or Stretch Video before exporting.'})}\n\n"
|
||||
return
|
||||
# Truthful per-segment verdict for the UI badge.
|
||||
entry = {"status": sf.status}
|
||||
if abs(sf.audio_rate - 1.0) > 1e-6:
|
||||
@@ -1816,8 +1800,8 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
elif strategy == "concise":
|
||||
# Mode A: never compress. Allow the audio to extend into the
|
||||
# silent gap before the next seg (existing heuristic) plus
|
||||
# any extra `overflow_budget_s`. Beyond that, hard-trim with
|
||||
# a short fade so we never overlap the next speaker.
|
||||
# any extra `overflow_budget_s`. Beyond that, require a
|
||||
# timing/text adjustment instead of discarding speech.
|
||||
place_at = start
|
||||
effective_end = end
|
||||
if i + 1 < len(all_segment_wavs):
|
||||
@@ -1831,47 +1815,25 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
slot_samples_eff = int(max(0.0, (effective_end - start)) * sr)
|
||||
if slot_samples_eff > 0 and wl > slot_samples_eff:
|
||||
overflow_s = (wl - slot_samples_eff) / sr
|
||||
adjusted = adjusted[..., :slot_samples_eff]
|
||||
wl = adjusted.shape[-1]
|
||||
fit_status.append({
|
||||
"status": "overflows",
|
||||
"overflow_s": round(overflow_s, 3),
|
||||
})
|
||||
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'segment_id': job['seg_order'][i], 'error_code': 'dub_timing_overflow', 'error': 'Speech exceeds its time slot. Shorten the translation or choose Strict Slot or Stretch Video before exporting.', 'overflow_s': round(overflow_s, 3)})}\n\n"
|
||||
return
|
||||
else:
|
||||
fit_status.append({"status": "fits"})
|
||||
|
||||
else:
|
||||
# strict_slot (legacy): preserve the previous atempo / trim /
|
||||
# off semantics so existing callers and back-compat tests
|
||||
# keep passing.
|
||||
# Strict Slot fits the complete speech to the original
|
||||
# slot. Explicit legacy trim/off choices remain available.
|
||||
place_at = start
|
||||
effective_end = end
|
||||
slowed_rate = None
|
||||
if i + 1 < len(all_segment_wavs):
|
||||
next_start = all_segment_wavs[i + 1][0]
|
||||
gap = next_start - end
|
||||
if gap > GAP_OVERFLOW_BUFFER_S:
|
||||
effective_end = end + min(
|
||||
gap - GAP_OVERFLOW_BUFFER_S, GAP_OVERFLOW_MAX_S,
|
||||
)
|
||||
slot_samples = int(max(0.0, (effective_end - start)) * sr)
|
||||
if slot_fit != "off" and slot_samples > 0 and wl > slot_samples:
|
||||
if slot_fit == "time_stretch":
|
||||
ratio = wl / slot_samples
|
||||
capped_ratio = min(ratio, MAX_STRETCH_RATIO)
|
||||
capped_target = int(wl / capped_ratio)
|
||||
try:
|
||||
adjusted = await _pitch_preserving_stretch(
|
||||
adjusted, capped_target, sr,
|
||||
adjusted, slot_samples, sr,
|
||||
)
|
||||
if adjusted.shape[-1] > slot_samples:
|
||||
adjusted = adjusted[..., :slot_samples]
|
||||
if ratio > MAX_STRETCH_RATIO:
|
||||
logger.info(
|
||||
"seg %d compression %.2f× exceeded cap; "
|
||||
"stretched to %.2f×, tail trimmed",
|
||||
i, ratio, capped_ratio,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"atempo stretch failed for seg %d (%.2f×), "
|
||||
@@ -1891,15 +1853,14 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
slot_fit == "time_stretch"
|
||||
and slot_samples > 0
|
||||
and wl > 0
|
||||
and wl < slot_samples * UNDERRUN_TOLERANCE
|
||||
and _underrun_min_rate() < 1.0 - 1e-6
|
||||
and wl < slot_samples
|
||||
):
|
||||
# Underrun fill (mirror of the compression above): the
|
||||
# dub finished early, leaving the on-screen mouth moving
|
||||
# over the thin under-speech bed residue — perceived as
|
||||
# dead air. Slow toward the slot, never below the floor.
|
||||
rate = max(wl / slot_samples, _underrun_min_rate())
|
||||
target = min(slot_samples, int(round(wl / rate)))
|
||||
rate = wl / slot_samples
|
||||
target = slot_samples
|
||||
try:
|
||||
adjusted = await _pitch_preserving_stretch(
|
||||
adjusted, target, sr,
|
||||
@@ -2000,6 +1961,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
"language_code": lang_code,
|
||||
"duration": round(track_dur, 4),
|
||||
"timing_strategy": strategy,
|
||||
"source_segments": [{"start": seg.start, "end": seg.end} for seg in req.segments],
|
||||
}
|
||||
|
||||
# Persist the timing strategy + (for Mode B) the per-segment stretch
|
||||
@@ -2043,12 +2005,9 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
"fit_fp": fit_fp,
|
||||
}
|
||||
job["dubbed_tracks"][lang_code]["fit_fp"] = fit_fp
|
||||
# Record what kind of per-segment WAVs are on disk so a later
|
||||
# smart_fit run knows whether partial regen / fit-only re-mix can
|
||||
# reuse them ("natural") or must regen once ("slotted"). Per-track
|
||||
# (P1.3) — each language renders under its own strategy; the flat
|
||||
# field stays in lock-step for older readers.
|
||||
_kind = "slotted" if strategy == "strict_slot" else "natural"
|
||||
# Every new cache preserves natural speech. Old slotted caches must
|
||||
# be regenerated once because their missing tails cannot be recovered.
|
||||
_kind = "natural"
|
||||
job.setdefault("seg_wav_kind_by_lang", {})[lang_code] = _kind
|
||||
job["seg_wav_kind"] = _kind
|
||||
_save_job(job_id, job)
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
import asyncio
|
||||
@@ -666,6 +667,7 @@ async def dub_translate(req: TranslateRequest):
|
||||
f"You are a professional dubbing translator. "
|
||||
f"Translate the user's text from {src_name} into "
|
||||
f"{tgt_name}.{script_clause}{dia_clause} "
|
||||
f"{translation_style_brief(req)} "
|
||||
f"Reply ONLY with the translated {tgt_name} text, do not "
|
||||
f"add quotes, notes, headers, explanations, or commentary."
|
||||
)
|
||||
@@ -737,7 +739,7 @@ async def dub_translate(req: TranslateRequest):
|
||||
source_lang=src_lang,
|
||||
target_lang=tgt_code,
|
||||
target_name=LANG_NAMES.get(tgt_code, tgt_code),
|
||||
extra_clause=context_extra,
|
||||
extra_clause="\n".join(filter(None, [context_extra, translation_style_brief(req)])),
|
||||
)
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.warning("reflect pass skipped for %s: %s",
|
||||
@@ -788,6 +790,28 @@ async def dub_translate(req: TranslateRequest):
|
||||
f"switch the Engine dropdown to another provider."
|
||||
)
|
||||
return JSONResponse(status_code=400, content={"error": friendly})
|
||||
# The package imports without its native dep; the *translator*
|
||||
# needs CTranslate2, whose library is rejected outright by kernels
|
||||
# that refuse an executable stack (#692). Repair it (a one-bit ELF
|
||||
# patch), and if that is impossible say so in one actionable 400
|
||||
# instead of the opaque 500 every segment used to produce.
|
||||
try:
|
||||
from core.execstack import ensure_ctranslate2_loadable
|
||||
|
||||
ensure_ctranslate2_loadable()
|
||||
except Exception as e: # noqa: BLE001 — repair must not block translation
|
||||
logger.debug("exec-stack repair unavailable (%s) — continuing", e)
|
||||
try:
|
||||
import argostranslate.translate # noqa: F401
|
||||
except Exception as e: # noqa: BLE001 — OSError here, not ImportError
|
||||
friendly = (
|
||||
f"The '{provider}' engine's CTranslate2 runtime could not be "
|
||||
"loaded in this backend."
|
||||
+ " Switch the Engine dropdown to NLLB (local) or an online "
|
||||
"provider, or reinstall the backend, then retry."
|
||||
)
|
||||
return JSONResponse(status_code=400, content={"error": friendly, "detail": {"code": "argos_runtime_unavailable", "message": friendly}})
|
||||
|
||||
target_codes = list(dict.fromkeys(
|
||||
seg.target_lang if seg.target_lang else req.target_lang
|
||||
for seg in req.segments
|
||||
@@ -1234,7 +1258,7 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
|
||||
target_lang=req.target_lang,
|
||||
glossary=req.glossary,
|
||||
directions=directions,
|
||||
dialect_hint=dialect_hint,
|
||||
dialect_hint="\n".join(filter(None, [dialect_hint, translation_style_brief(req)])),
|
||||
executor=_cpu_pool,
|
||||
)
|
||||
refined_by_id = {r["id"]: r for r in refined}
|
||||
@@ -1283,6 +1307,12 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
|
||||
}
|
||||
|
||||
|
||||
def translation_style_brief(req) -> str:
|
||||
instructions = (getattr(req, "translation_instructions", None) or "").strip()
|
||||
return ("User translation style brief (tone and wording only; preserve meaning, timing and output format): "
|
||||
+ json.dumps(instructions, ensure_ascii=False)) if instructions else ""
|
||||
|
||||
|
||||
@router.post("/dub/agent-fit")
|
||||
async def dub_agent_fit(req: AgentFitRequest):
|
||||
"""Rewrite rendered lines from real duration evidence.
|
||||
@@ -1319,7 +1349,7 @@ async def dub_agent_fit(req: AgentFitRequest):
|
||||
]
|
||||
budget = _cinematic_budget()
|
||||
try:
|
||||
call = adjust_for_measured_slot_many(items, executor=_cpu_pool)
|
||||
call = adjust_for_measured_slot_many(items, executor=_cpu_pool, translation_instructions=req.translation_instructions)
|
||||
rows = await asyncio.wait_for(call, timeout=budget) if budget and budget > 0 else await call
|
||||
except asyncio.TimeoutError:
|
||||
rows = {
|
||||
|
||||
@@ -21,12 +21,12 @@ import os
|
||||
import threading
|
||||
from time import perf_counter
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from huggingface_hub import utils as hf_utils
|
||||
from huggingface_hub.errors import HFValidationError
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from api.dependencies import require_admin, require_admin_action, require_desktop
|
||||
from api.dependencies import require_admin, require_admin_action, require_desktop, is_loopback
|
||||
from core import prefs
|
||||
from core.engine_licenses import LICENSE_GATED_ENGINES
|
||||
from services import tts_backend, asr_backend, llm_backend, translation_engines
|
||||
@@ -116,18 +116,29 @@ def _is_hf_repo_id(value: str) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
def _request_install_capability(payload, request):
|
||||
allowed = bool(request and request.client and is_loopback(request.client.host))
|
||||
result = dict(payload)
|
||||
result["backends"] = [dict(entry) for entry in payload["backends"]]
|
||||
for entry in result["backends"]:
|
||||
if entry.get("one_click_install") and not allowed:
|
||||
entry["one_click_install"] = False
|
||||
entry["local_install_required"] = True
|
||||
return result
|
||||
|
||||
|
||||
@router.get("/engines")
|
||||
def list_all_engines():
|
||||
def list_all_engines(request: Request):
|
||||
return {
|
||||
"tts": _family_payload("tts", tts_backend),
|
||||
"tts": _request_install_capability(_family_payload("tts", tts_backend), request),
|
||||
"asr": _family_payload("asr", asr_backend),
|
||||
"llm": _family_payload("llm", llm_backend),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/engines/tts")
|
||||
def list_tts_backends():
|
||||
return _family_payload("tts", tts_backend)
|
||||
def list_tts_backends(request: Request):
|
||||
return _request_install_capability(_family_payload("tts", tts_backend), request)
|
||||
|
||||
|
||||
@router.get(
|
||||
@@ -409,10 +420,10 @@ async def uninstall_translation_engine(engine_id: str):
|
||||
"/engines/audiocpp/runtime/install/status",
|
||||
dependencies=[Depends(require_admin)],
|
||||
)
|
||||
def audiocpp_runtime_install_status():
|
||||
def audiocpp_runtime_install_status(request: Request):
|
||||
from services import audiocpp_runtime_install
|
||||
|
||||
return audiocpp_runtime_install.status()
|
||||
return {**audiocpp_runtime_install.status(), "install_allowed": bool(request.client and is_loopback(request.client.host))}
|
||||
|
||||
|
||||
@router.post(
|
||||
@@ -485,7 +496,7 @@ def install_sidecar_engine(engine_id: str):
|
||||
"/engines/sidecar/{engine_id}/install/status",
|
||||
dependencies=[Depends(require_admin)],
|
||||
)
|
||||
def sidecar_install_status(engine_id: str):
|
||||
def sidecar_install_status(engine_id: str, request: Request = None):
|
||||
"""Step-by-step status of the sidecar install job (poll while running).
|
||||
|
||||
Shape: ``{engine_id, installed, managed, install_dir, job}`` where job is
|
||||
@@ -494,7 +505,7 @@ def sidecar_install_status(engine_id: str):
|
||||
"""
|
||||
from services import sidecar_install
|
||||
try:
|
||||
return sidecar_install.get_status(engine_id)
|
||||
return {**sidecar_install.get_status(engine_id), "install_allowed": bool(request and request.client and is_loopback(request.client.host))}
|
||||
except KeyError:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
|
||||
@@ -807,6 +807,7 @@ def _oom_friendly_reraise(e):
|
||||
def _generate_timeout_s(
|
||||
text: str,
|
||||
*,
|
||||
engine: object = None,
|
||||
execution_device=None,
|
||||
min_vram_gb=0.0,
|
||||
hardware_family=None,
|
||||
@@ -827,6 +828,7 @@ def _generate_timeout_s(
|
||||
from services.model_manager import generate_timeout_s
|
||||
return generate_timeout_s(
|
||||
text,
|
||||
engine=engine,
|
||||
execution_device=execution_device,
|
||||
min_vram_gb=min_vram_gb,
|
||||
hardware_family=hardware_family,
|
||||
@@ -1752,6 +1754,7 @@ async def generate_speech(
|
||||
what="TTS generate",
|
||||
timeout=_generate_timeout_s(
|
||||
text,
|
||||
engine=_backend,
|
||||
execution_device=_routing["effective_device"],
|
||||
min_vram_gb=_engine_min_vram_gb,
|
||||
hardware_family=_routing_hardware_family,
|
||||
@@ -2052,6 +2055,7 @@ async def generate_speech(
|
||||
min_vram_gb=_engine_min_vram_gb,
|
||||
timeout=_generate_timeout_s(
|
||||
text,
|
||||
engine=_backend,
|
||||
execution_device=_routing["effective_device"],
|
||||
min_vram_gb=_engine_min_vram_gb,
|
||||
hardware_family=_routing_hardware_family,
|
||||
@@ -2078,6 +2082,7 @@ async def generate_speech(
|
||||
min_vram_gb=_engine_min_vram_gb,
|
||||
timeout=_generate_timeout_s(
|
||||
text,
|
||||
engine=_backend,
|
||||
execution_device=_routing["effective_device"],
|
||||
min_vram_gb=_engine_min_vram_gb,
|
||||
hardware_family=_routing_hardware_family,
|
||||
@@ -2124,6 +2129,7 @@ async def generate_speech(
|
||||
# even after the v0.3.22 scaled budget shipped.
|
||||
timeout=_generate_timeout_s(
|
||||
chunk_text,
|
||||
engine=_backend,
|
||||
execution_device=_routing["effective_device"],
|
||||
min_vram_gb=_engine_min_vram_gb,
|
||||
hardware_family=_routing_hardware_family,
|
||||
@@ -2290,6 +2296,7 @@ async def generate_speech(
|
||||
_local_render, what="TTS generate",
|
||||
timeout=_generate_timeout_s(
|
||||
text,
|
||||
engine=_backend,
|
||||
execution_device=_routing["effective_device"],
|
||||
min_vram_gb=_engine_min_vram_gb,
|
||||
hardware_family=_routing_hardware_family,
|
||||
|
||||
@@ -721,17 +721,11 @@ def list_voices():
|
||||
|
||||
def _format_ts_srt(seconds: float) -> str:
|
||||
"""Format seconds as SRT timestamp: HH:MM:SS,mmm"""
|
||||
h = int(seconds // 3600)
|
||||
m = int((seconds % 3600) // 60)
|
||||
s = int(seconds % 60)
|
||||
ms = int((seconds % 1) * 1000)
|
||||
return f"{h:02d}:{m:02d}:{s:02d},{ms:03d}"
|
||||
from services.srt_parser import format_cue_timestamp
|
||||
return format_cue_timestamp(seconds, ",")
|
||||
|
||||
|
||||
def _format_ts_vtt(seconds: float) -> str:
|
||||
"""Format seconds as VTT timestamp: HH:MM:SS.mmm"""
|
||||
h = int(seconds // 3600)
|
||||
m = int((seconds % 3600) // 60)
|
||||
s = int(seconds % 60)
|
||||
ms = int((seconds % 1) * 1000)
|
||||
return f"{h:02d}:{m:02d}:{s:02d}.{ms:03d}"
|
||||
from services.srt_parser import format_cue_timestamp
|
||||
return format_cue_timestamp(seconds, ".")
|
||||
|
||||
@@ -156,7 +156,7 @@ def set_performance_profile(body: _PerformanceProfileBody):
|
||||
|
||||
|
||||
class _TorchCompileBody(BaseModel):
|
||||
enabled: bool = Field(..., description="True to set TORCH_COMPILE_DISABLE=1 on engine subprocesses")
|
||||
enabled: bool = Field(..., description="True to disable torch.compile (eager mode) for the engine")
|
||||
|
||||
|
||||
def _torch_compile_state() -> dict:
|
||||
@@ -170,15 +170,21 @@ def _torch_compile_state() -> dict:
|
||||
@router.get("/perf/torch-compile-disabled")
|
||||
def get_torch_compile_disabled():
|
||||
"""Return the current torch.compile-disabled toggle + the runtime platform.
|
||||
UI uses the platform to render the toggle disabled (with an explainer)
|
||||
on non-Windows hosts, since the OOM is Windows-specific (issue #65)."""
|
||||
|
||||
`platform` is still reported (clients may show it), but since #2135 the
|
||||
toggle is live on every host: it used to be rendered disabled off Windows
|
||||
on the assumption that only #65's Windows OOM needed it, which left the
|
||||
Linux/CUDA reporter of #2135 with no way to switch off the compile that
|
||||
was killing their backend.
|
||||
"""
|
||||
return _torch_compile_state()
|
||||
|
||||
|
||||
@router.put("/perf/torch-compile-disabled")
|
||||
def set_torch_compile_disabled(body: _TorchCompileBody):
|
||||
"""Persist the toggle. Honoured by `services.engine_env.build_engine_env()`
|
||||
which injects TORCH_COMPILE_DISABLE=1 on Windows when enabled."""
|
||||
(subprocess engines) and `services.engine_env.should_torch_compile()`
|
||||
(in-process), on every platform since #2135."""
|
||||
from services import settings_store
|
||||
|
||||
try:
|
||||
|
||||
@@ -337,6 +337,7 @@ async def convert_speech(
|
||||
what="Voice convert",
|
||||
timeout=_generate_timeout_s(
|
||||
text,
|
||||
engine=backend,
|
||||
execution_device=compute_profile["effective_device"],
|
||||
min_vram_gb=compute_profile["min_vram_gb"],
|
||||
hardware_family=compute_profile.get("runtime_hardware_family"),
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Native-crash diagnostics for the backend process (#2135).
|
||||
|
||||
A crash inside torch/CUDA — graph capture, a driver fault, an allocator abort —
|
||||
kills the interpreter below the level any ``except`` can reach. #2135's reporter
|
||||
saw exactly that: the backend "simply exited" mid-``/generate`` with no Python
|
||||
traceback, no HTTP response, and ``ConnectionRefused`` on the next ``/health``.
|
||||
There was nothing in the logs to diagnose because nothing in Python ever ran
|
||||
again.
|
||||
|
||||
``faulthandler`` installs handlers for the fatal signals (SIGSEGV, SIGABRT,
|
||||
SIGBUS, SIGFPE, SIGILL) that print every thread's Python stack to stderr on the
|
||||
way down. That is the difference between "the process vanished" and a named
|
||||
frame pointing at the engine call that killed it.
|
||||
|
||||
This is strictly a diagnostic: it does not prevent the crash, and it must never
|
||||
be the reason startup fails.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
_DISABLE_ENV = "OMNIVOICE_DISABLE_FAULTHANDLER"
|
||||
_TRUTHY = frozenset({"1", "true", "yes", "on"})
|
||||
|
||||
|
||||
def _disabled() -> bool:
|
||||
return os.environ.get(_DISABLE_ENV, "").strip().lower() in _TRUTHY
|
||||
|
||||
|
||||
def enable_fault_handler(stderr=None) -> bool:
|
||||
"""Arm fatal-signal tracebacks. Returns True when armed.
|
||||
|
||||
Call as early as possible — before torch is imported — so a crash during
|
||||
model load is covered too. Honours ``OMNIVOICE_DISABLE_FAULTHANDLER=1`` for
|
||||
hosts whose outer supervisor installs its own handlers.
|
||||
|
||||
Args:
|
||||
stderr: optional file object to write dumps to. Defaults to the real
|
||||
``sys.stderr`` (→ ``backend_err.log``). faulthandler keeps the
|
||||
underlying fd, so the object must stay open for the process
|
||||
lifetime.
|
||||
|
||||
Never raises: a frozen build with a detached stderr, or a platform without
|
||||
the signals, degrades to "no crash dump" rather than a failed boot.
|
||||
"""
|
||||
if _disabled():
|
||||
return False
|
||||
try:
|
||||
import faulthandler
|
||||
|
||||
# all_threads=True: the fatal frame is routinely on a GPU-pool or
|
||||
# compile worker, not whichever thread happens to take the signal.
|
||||
if stderr is not None:
|
||||
faulthandler.enable(file=stderr, all_threads=True)
|
||||
else:
|
||||
faulthandler.enable(all_threads=True)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
@@ -0,0 +1,245 @@
|
||||
"""Repair wheel-shipped shared libraries that request an executable stack.
|
||||
|
||||
Why this exists
|
||||
---------------
|
||||
CTranslate2 wheels up to and including 4.4.0 ship
|
||||
``ctranslate2.libs/libctranslate2-*.so`` with ``PT_GNU_STACK`` marked
|
||||
``RWE`` — a request for an executable stack. Linux kernels that refuse to
|
||||
grant it (hardened kernels, and mainline from 6.x onwards) fail the
|
||||
``dlopen`` outright::
|
||||
|
||||
ImportError: libctranslate2-d3638643.so.4.4.0: cannot enable executable
|
||||
stack as shared object requires: Invalid argument
|
||||
|
||||
Everything that links CTranslate2 dies with it: the **whisperx** and
|
||||
**faster-whisper** ASR engines (#692) *and* Argos translation, which is the
|
||||
default dub translation engine (``argostranslate.translate`` imports
|
||||
``ctranslate2``). #692 taught the ASR selector to fall through to another
|
||||
engine; it never fixed the library, so Linux users on Python 3.11 lost both
|
||||
engines. The pin is upstream and not ours to lift: whisperx 3.4.5 — the last
|
||||
release that supports Python 3.11, which is what ``.python-version``, CI and
|
||||
the installers use — requires ``ctranslate2<4.5.0``, and 4.5.0 is the first
|
||||
release whose ``.so`` drops the exec-stack request.
|
||||
|
||||
The flag is a single bit in the ELF program header, so we clear it in place
|
||||
rather than shipping a patched wheel or asking users for ``patchelf`` (which
|
||||
is not installed on a typical desktop). Inspection is a ~100-byte read with
|
||||
no imports, so :func:`ensure_ctranslate2_loadable` is cheap enough to call
|
||||
from an availability probe: it only rewrites a file when that file would
|
||||
otherwise refuse to load.
|
||||
|
||||
Everything here is a no-op off Linux (macOS/Windows have no such rejection)
|
||||
and handles malformed ELF data — a repair that cannot happen returns a reason, and the
|
||||
caller degrades exactly as it did before.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import glob
|
||||
import logging
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import threading
|
||||
|
||||
logger = logging.getLogger("omnivoice.execstack")
|
||||
|
||||
#: ELF segment type for the stack-permission marker, and its executable bit.
|
||||
_PT_GNU_STACK = 0x6474E551
|
||||
_PF_X = 0x1
|
||||
|
||||
#: Serialize in-process writes; flock also coordinates sidecar processes.
|
||||
_REPAIR_LOCK = threading.RLock()
|
||||
|
||||
|
||||
def _elf_header(fh) -> tuple[str, int, int, int, bool] | None:
|
||||
"""Return ``(endian_prefix, e_phoff, e_phentsize, e_phnum, is_64)`` or None.
|
||||
|
||||
None means "not an ELF file we understand" — which is a normal answer
|
||||
(a ``.so`` stub, a text file, a Mach-O), never an error.
|
||||
"""
|
||||
fh.seek(0)
|
||||
ident = fh.read(16)
|
||||
if len(ident) < 16 or ident[:4] != b"\x7fELF":
|
||||
return None
|
||||
if ident[4] not in (1, 2) or ident[5] not in (1, 2):
|
||||
return None
|
||||
is_64 = ident[4] == 2
|
||||
endian = "<" if ident[5] == 1 else ">"
|
||||
fh.seek(0, os.SEEK_END)
|
||||
size = fh.tell()
|
||||
header_size = 64 if is_64 else 52
|
||||
if size < header_size:
|
||||
return None
|
||||
fh.seek(0)
|
||||
header = fh.read(header_size)
|
||||
if len(header) != header_size:
|
||||
return None
|
||||
e_phoff = struct.unpack_from(endian + ("Q" if is_64 else "I"), header, 0x20 if is_64 else 0x1C)[0]
|
||||
e_phentsize, e_phnum = struct.unpack_from(endian + "HH", header, 0x36 if is_64 else 0x2A)
|
||||
if (not e_phnum or e_phoff < header_size or
|
||||
e_phentsize < (56 if is_64 else 32) or
|
||||
e_phoff + e_phentsize * e_phnum > size):
|
||||
return None
|
||||
# p_flags sits at a different offset per class (ELF64 puts it right after
|
||||
# p_type; ELF32 puts it last), so the caller needs the class too.
|
||||
return endian, e_phoff, e_phentsize, e_phnum, is_64
|
||||
|
||||
|
||||
def _gnu_stack_flags_offset(fh) -> tuple[int, int, str] | None:
|
||||
"""Locate the ``PT_GNU_STACK`` ``p_flags`` field.
|
||||
|
||||
Returns ``(file_offset, flags_value, endian_prefix)``, or None when the
|
||||
file is not an ELF or carries no such segment.
|
||||
"""
|
||||
parsed = _elf_header(fh)
|
||||
if parsed is None:
|
||||
return None
|
||||
endian, e_phoff, e_phentsize, e_phnum, is_64 = parsed
|
||||
flags_rel = 4 if is_64 else 24 # p_flags offset inside the program header
|
||||
for i in range(e_phnum):
|
||||
base = e_phoff + i * e_phentsize
|
||||
fh.seek(base)
|
||||
raw = fh.read(e_phentsize)
|
||||
if len(raw) < flags_rel + 4:
|
||||
continue
|
||||
(p_type,) = struct.unpack_from(endian + "I", raw, 0)
|
||||
if p_type != _PT_GNU_STACK:
|
||||
continue
|
||||
(p_flags,) = struct.unpack_from(endian + "I", raw, flags_rel)
|
||||
return base + flags_rel, p_flags, endian
|
||||
return None
|
||||
|
||||
|
||||
def has_execstack(path: str) -> bool | None:
|
||||
"""True when ``path`` requests an executable stack.
|
||||
|
||||
None when the question does not apply: unreadable, not an ELF, or no
|
||||
``PT_GNU_STACK`` segment.
|
||||
"""
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
found = _gnu_stack_flags_offset(fh)
|
||||
except OSError:
|
||||
return None
|
||||
if found is None:
|
||||
return None
|
||||
_offset, flags, _endian = found
|
||||
return bool(flags & _PF_X)
|
||||
|
||||
|
||||
def clear_execstack(path: str) -> tuple[bool, str]:
|
||||
"""Clear the executable-stack request on ``path``.
|
||||
|
||||
Returns ``(changed, detail)``. ``changed`` is False both when there was
|
||||
nothing to do and when the write was refused (a read-only bundle, for
|
||||
instance) — ``detail`` says which.
|
||||
"""
|
||||
try:
|
||||
# Lock and inspect the same descriptor we write: another process may
|
||||
# already have repaired it, or the wheel may have been replaced.
|
||||
with _REPAIR_LOCK, open(path, "r+b") as fh:
|
||||
# Use host capability, not an emulated target platform.
|
||||
if os.name == "posix":
|
||||
import fcntl
|
||||
fcntl.flock(fh, fcntl.LOCK_EX)
|
||||
found = _gnu_stack_flags_offset(fh)
|
||||
if found is None:
|
||||
return False, "no PT_GNU_STACK segment"
|
||||
offset, flags, endian = found
|
||||
if not flags & _PF_X:
|
||||
return False, "already non-executable"
|
||||
fh.seek(offset)
|
||||
fh.write(struct.pack(endian + "I", flags & ~_PF_X))
|
||||
fh.flush()
|
||||
os.fsync(fh.fileno())
|
||||
except OSError as e:
|
||||
return False, f"unreadable or not writable ({e.__class__.__name__})"
|
||||
return True, "cleared PT_GNU_STACK executable bit"
|
||||
|
||||
|
||||
def ctranslate2_library_paths() -> list[str]:
|
||||
"""Native libraries shipped with the installed ``ctranslate2`` wheel.
|
||||
|
||||
Found without importing ``ctranslate2`` — importing it is the very thing
|
||||
that fails when the exec-stack bit is set.
|
||||
"""
|
||||
import importlib.util
|
||||
|
||||
roots: list[str] = []
|
||||
try:
|
||||
spec = importlib.util.find_spec("ctranslate2")
|
||||
except (ImportError, ValueError): # pragma: no cover — defensive
|
||||
spec = None
|
||||
locations = list(getattr(spec, "submodule_search_locations", None) or []) if spec else []
|
||||
for pkg_dir in locations:
|
||||
roots.append(pkg_dir)
|
||||
roots.append(os.path.join(os.path.dirname(pkg_dir), "ctranslate2.libs"))
|
||||
# Frozen builds flatten the wheel into the bundle directory.
|
||||
meipass = getattr(sys, "_MEIPASS", None)
|
||||
if meipass:
|
||||
roots.append(meipass)
|
||||
roots.append(os.path.join(meipass, "ctranslate2.libs"))
|
||||
out: list[str] = []
|
||||
for root in roots:
|
||||
if not os.path.isdir(root):
|
||||
continue
|
||||
for pattern in ("libctranslate2*.so*", "libctranslate2*.dylib"):
|
||||
out.extend(sorted(glob.glob(os.path.join(root, pattern))))
|
||||
# Dedupe, preserving order.
|
||||
return list(dict.fromkeys(out))
|
||||
|
||||
|
||||
def ensure_ctranslate2_loadable() -> tuple[bool, str]:
|
||||
"""Make ``import ctranslate2`` possible on kernels that refuse exec stacks.
|
||||
|
||||
Returns ``(ok, detail)`` where ``ok`` is False only when a library needs
|
||||
the repair and could not get it — the caller should then report its
|
||||
engine unavailable with ``detail`` as the reason. The repair is idempotent. Re-probe on each call so installation or
|
||||
external repair takes effect without restarting.
|
||||
"""
|
||||
|
||||
result: tuple[bool, str]
|
||||
if sys.platform != "linux":
|
||||
# Only Linux rejects an exec-stack request at dlopen time.
|
||||
result = (True, "not applicable off Linux")
|
||||
else:
|
||||
libs = ctranslate2_library_paths()
|
||||
if not libs:
|
||||
result = (True, "no ctranslate2 library found")
|
||||
else:
|
||||
repaired: list[str] = []
|
||||
blocked: list[str] = []
|
||||
for lib in libs:
|
||||
if has_execstack(lib) is not True:
|
||||
continue
|
||||
changed, detail = clear_execstack(lib)
|
||||
if changed:
|
||||
repaired.append(os.path.basename(lib))
|
||||
logger.warning(
|
||||
"Repaired %s: %s — its executable-stack request is "
|
||||
"rejected by this kernel, which broke whisperx, "
|
||||
"faster-whisper and Argos translation (#692)",
|
||||
os.path.basename(lib), detail,
|
||||
)
|
||||
elif has_execstack(lib) is not False:
|
||||
blocked.append(f"{os.path.basename(lib)} ({detail})")
|
||||
if blocked:
|
||||
result = (
|
||||
False,
|
||||
"ctranslate2's native library requests an executable stack, "
|
||||
"which this kernel refuses, and it could not be patched: "
|
||||
+ "; ".join(blocked)
|
||||
+ ". Reinstall the backend on Python 3.12+ (which resolves "
|
||||
"ctranslate2 4.8+, without the exec-stack request), or run "
|
||||
"`patchelf --clear-execstack <library>` once.",
|
||||
)
|
||||
elif repaired:
|
||||
result = (True, "repaired " + ", ".join(repaired))
|
||||
else:
|
||||
result = (True, "no exec-stack request")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def reset_ctranslate2_cache() -> None:
|
||||
"""Compatibility hook; recoverable probe results are no longer cached."""
|
||||
+32
-1
@@ -10,6 +10,27 @@ from core import run_sentinel
|
||||
logger = logging.getLogger("omnivoice.tasks")
|
||||
|
||||
|
||||
def _stream_failure(update):
|
||||
"""Recognize terminal SSE failures, including generators that do not raise."""
|
||||
if isinstance(update, bytes):
|
||||
update = update.decode("utf-8", errors="replace")
|
||||
if not isinstance(update, str):
|
||||
return None
|
||||
lines = update.splitlines()
|
||||
try:
|
||||
payload = json.loads("\n".join(line[5:].strip() for line in lines if line.startswith("data:")))
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
if not isinstance(payload, dict):
|
||||
return None
|
||||
if payload.get("type") != "error" and not any(line.strip() == "event: error" for line in lines):
|
||||
return None
|
||||
detail = payload.get("reason") or payload.get("error") or payload.get("detail")
|
||||
if isinstance(detail, dict):
|
||||
detail = detail.get("message") or detail.get("reason")
|
||||
return detail if isinstance(detail, str) and detail else "Task failed"
|
||||
|
||||
|
||||
class TaskManager:
|
||||
"""In-memory task dispatcher with SQLite-backed metadata.
|
||||
|
||||
@@ -125,9 +146,19 @@ class TaskManager:
|
||||
except Exception: logger.exception("job_store.mark_cancelled failed")
|
||||
break
|
||||
await self._push_event(task_id, update)
|
||||
stream_error = _stream_failure(update)
|
||||
if stream_error is not None:
|
||||
t["status"] = "failed"
|
||||
t["error"] = stream_error
|
||||
try:
|
||||
job_store.mark_failed(task_id, stream_error)
|
||||
except Exception:
|
||||
logger.exception("job_store.mark_failed failed")
|
||||
await res.aclose()
|
||||
break
|
||||
elif inspect.iscoroutine(res):
|
||||
await res
|
||||
if t["status"] != "cancelled":
|
||||
if t["status"] not in {"cancelled", "failed"}:
|
||||
t["status"] = "done"
|
||||
try: job_store.mark_done(task_id)
|
||||
except Exception: logger.exception("job_store.mark_done failed")
|
||||
|
||||
@@ -24,7 +24,7 @@ from pathlib import Path
|
||||
# tests/test_app_version.py::test_all_version_files_in_lockstep and bumped by
|
||||
# release.yml's version-bump job, so it stays equal to
|
||||
# pyproject/tauri.conf/Cargo/package.json.
|
||||
_FALLBACK_VERSION = "0.5.2"
|
||||
_FALLBACK_VERSION = "0.5.3"
|
||||
|
||||
|
||||
def _fallback_version() -> str:
|
||||
|
||||
@@ -84,6 +84,10 @@ def _is_ct_error(msg):
|
||||
def _get_model():
|
||||
global _model
|
||||
if _model is None:
|
||||
from core.execstack import ensure_ctranslate2_loadable
|
||||
ok, detail = ensure_ctranslate2_loadable()
|
||||
if not ok:
|
||||
raise ImportError(f"faster-whisper cannot load CTranslate2: {detail}")
|
||||
from faster_whisper import WhisperModel
|
||||
# Same weights as in-process faster-whisper: ASR_MODEL_FASTER selects
|
||||
# for BOTH variants, ASR_MODEL_FW stays as a sidecar-only override.
|
||||
|
||||
@@ -25,6 +25,8 @@ runs under the Confucius4 venv — never imported by the parent), and
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from services.subprocess_backend import SubprocessBackend
|
||||
@@ -99,6 +101,19 @@ class Confucius4Backend(SubprocessBackend):
|
||||
from engines.confucius4.bootstrap import CONFUCIUS4_SIDECAR_SCRIPT
|
||||
return CONFUCIUS4_SIDECAR_SCRIPT
|
||||
|
||||
@property
|
||||
def recv_timeout_s(self) -> float:
|
||||
"""Receive timeout in seconds for the Confucius4 sidecar process (#2103)."""
|
||||
# Confucius4 is an LLM-based TTS (~17x realtime on CPU); synthesis legitimately
|
||||
# outruns the 60s class default. OMNIVOICE_CONFUCIUS4_RECV_TIMEOUT_S tunes it (#2103).
|
||||
try:
|
||||
v = float(os.environ.get("OMNIVOICE_CONFUCIUS4_RECV_TIMEOUT_S", "900"))
|
||||
except (ValueError, TypeError):
|
||||
return 900.0
|
||||
if not math.isfinite(v):
|
||||
return 900.0
|
||||
return max(30.0, v)
|
||||
|
||||
@property
|
||||
def sample_rate(self) -> int:
|
||||
return self._DEFAULT_SAMPLE_RATE
|
||||
|
||||
@@ -31,6 +31,8 @@ by the parent), and ``bootstrap.py`` (venv probe + lazy bootstrap).
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
@@ -121,6 +123,19 @@ class DotsTTSBackend(SubprocessBackend):
|
||||
from engines.dots_tts.bootstrap import DOTS_TTS_SIDECAR_SCRIPT
|
||||
return DOTS_TTS_SIDECAR_SCRIPT
|
||||
|
||||
@property
|
||||
def recv_timeout_s(self) -> float:
|
||||
"""Receive timeout in seconds for the dots.tts sidecar process (#2103)."""
|
||||
# dots.tts is a 2B autoregressive model; synthesis on CPU legitimately
|
||||
# outruns the 60s class default. OMNIVOICE_DOTS_TTS_RECV_TIMEOUT_S tunes it (#2103).
|
||||
try:
|
||||
v = float(os.environ.get("OMNIVOICE_DOTS_TTS_RECV_TIMEOUT_S", "900"))
|
||||
except (ValueError, TypeError):
|
||||
return 900.0
|
||||
if not math.isfinite(v):
|
||||
return 900.0
|
||||
return max(30.0, v)
|
||||
|
||||
# ── TTSBackend protocol ────────────────────────────────────────────────
|
||||
|
||||
@property
|
||||
|
||||
@@ -38,6 +38,8 @@ isolated engine venv.
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from services.subprocess_backend import SubprocessBackend
|
||||
@@ -127,6 +129,19 @@ class MossTTSV15Backend(SubprocessBackend):
|
||||
from engines.moss_tts_v15.bootstrap import MOSS_TTS_V15_SIDECAR_SCRIPT
|
||||
return MOSS_TTS_V15_SIDECAR_SCRIPT
|
||||
|
||||
@property
|
||||
def recv_timeout_s(self) -> float:
|
||||
"""Receive timeout in seconds for the MOSS-TTS-v1.5 sidecar process (#2103)."""
|
||||
# MOSS-TTS-v1.5 is an 8B model; synthesis legitimately outruns the
|
||||
# 60s class default. OMNIVOICE_MOSS_TTS_V15_RECV_TIMEOUT_S tunes it (#2103).
|
||||
try:
|
||||
v = float(os.environ.get("OMNIVOICE_MOSS_TTS_V15_RECV_TIMEOUT_S", "900"))
|
||||
except (ValueError, TypeError):
|
||||
return 900.0
|
||||
if not math.isfinite(v):
|
||||
return 900.0
|
||||
return max(30.0, v)
|
||||
|
||||
# ── TTSBackend protocol ────────────────────────────────────────────────
|
||||
|
||||
@property
|
||||
|
||||
@@ -113,11 +113,11 @@ This clears the quarantine xattr recursively — including on the bundled
|
||||
returns a clear error message pointing at this command rather than
|
||||
silently hanging on a Gatekeeper-killed spawn.
|
||||
|
||||
If the macOS Apple Silicon Metal build fails to materialize in Wave 1
|
||||
(no published `buildmetal.sh` in `omnivoice.cpp` per Pitfall 1), the
|
||||
GGUF engine is unavailable on Apple Silicon and the existing in-process
|
||||
`VoiceStudioBackend` remains the cloning default on that platform — no
|
||||
hard block, no error toast on launch.
|
||||
The macOS Apple Silicon Metal build compiles cleanly with `-DGGML_METAL=ON`
|
||||
at the pinned `omnivoice.cpp` SHA (#2105), enabling GPU-accelerated GGUF
|
||||
voice cloning when the packaged binary passes preflight and is permitted by
|
||||
macOS. Missing binaries, placeholders, or Gatekeeper rejection leave
|
||||
`VoiceStudioBackend` available as the in-process fallback.
|
||||
|
||||
## Smoke test
|
||||
|
||||
|
||||
@@ -35,6 +35,7 @@ Threat model (per Plan 03-01 frontmatter):
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
@@ -99,6 +100,19 @@ class Supertonic3Backend(SubprocessBackend):
|
||||
def sidecar_script(cls) -> Path:
|
||||
return SUPERTONIC3_SIDECAR_SCRIPT
|
||||
|
||||
@property
|
||||
def recv_timeout_s(self) -> float:
|
||||
"""Receive timeout in seconds for the Supertonic-3 sidecar process (#2103)."""
|
||||
# Supertonic-3 runs ONNX on CPU; cold load downloads ~400MB and long
|
||||
# synthesis benefits from more headroom than 60s. OMNIVOICE_SUPERTONIC3_RECV_TIMEOUT_S (#2103).
|
||||
try:
|
||||
v = float(os.environ.get("OMNIVOICE_SUPERTONIC3_RECV_TIMEOUT_S", "300"))
|
||||
except (ValueError, TypeError):
|
||||
return 300.0
|
||||
if not math.isfinite(v):
|
||||
return 300.0
|
||||
return max(30.0, v)
|
||||
|
||||
# ── availability ───────────────────────────────────────────────────
|
||||
|
||||
@classmethod
|
||||
|
||||
+17
-5
@@ -9,6 +9,13 @@ _backend_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
if _backend_dir not in sys.path:
|
||||
sys.path.insert(0, _backend_dir)
|
||||
|
||||
# #2135: arm fatal-signal tracebacks before anything heavy is imported, so a
|
||||
# native crash inside torch/CUDA leaves a named frame in backend_err.log
|
||||
# instead of a silently vanished process. See core/crash_diagnostics.py.
|
||||
from core.crash_diagnostics import enable_fault_handler # noqa: E402
|
||||
|
||||
enable_fault_handler()
|
||||
|
||||
# PyInstaller re-executes this entry module when the frozen backend binary is
|
||||
# launched. Nested operation supervisors therefore dispatch here, before math,
|
||||
# logging, FastAPI, torch, or any application initialization. Source launches
|
||||
@@ -629,11 +636,16 @@ def _phase_a_build_inner() -> None:
|
||||
# failure in that phase takes the whole backend down — the desktop app sits
|
||||
# on "starting backend" forever and /health stays 503.
|
||||
#
|
||||
# That is not a hypothetical version: RTX 50-series (Blackwell, sm_120)
|
||||
# users have no choice but to move off the pinned torch 2.8.0, which has no
|
||||
# sm_120 kernels, and the torch 2.9.x they land on brings torchaudio 2.9
|
||||
# with it. So the one group forced to upgrade hit a hard startup crash for
|
||||
# a line that does nothing (#1931).
|
||||
# That is not a hypothetical version: #1931 came from an sm_120 (Blackwell)
|
||||
# user whose torch import crashed on Windows and who fixed it by moving to
|
||||
# torch 2.9.1, which brings torchaudio 2.9 with it. Someone already working
|
||||
# around one problem then hit a hard startup crash on a line that does
|
||||
# nothing (#1931).
|
||||
#
|
||||
# The pin is not missing sm_120 kernels: torch 2.8.0 from the cu128 index
|
||||
# lists sm_120 in get_arch_list(). CU128_ARCHS in
|
||||
# tests/test_cuda_arch_compat.py records the same list, captured verbatim
|
||||
# from a real cu128 build in #1285.
|
||||
if hasattr(torchaudio, "set_audio_backend"):
|
||||
torchaudio.set_audio_backend("soundfile")
|
||||
from utils import hf_progress
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
from pydantic import BaseModel, field_validator
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
from typing import List, Literal, Optional
|
||||
|
||||
from services.audio_dsp import EFFECT_PRESETS
|
||||
@@ -89,9 +89,9 @@ class DubRequest(BaseModel):
|
||||
# (Bengali, Hindi, Arabic…). Three modes:
|
||||
# "concise" — never compress TTS audio. Trim text up-front via
|
||||
# speech_rate so it fits naturally; if it still
|
||||
# overflows, hard-trim at slot with a short fade and
|
||||
# surface fit_status="overflows" so the UI can prompt
|
||||
# the user to shorten the segment. DEFAULT.
|
||||
# overflows, fail without replacing the current track;
|
||||
# the user must shorten it or choose another fit mode.
|
||||
# DEFAULT.
|
||||
# "stretch_video" — never compress TTS audio. Re-lay the timeline so
|
||||
# each segment's video portion is stretched (via
|
||||
# ffmpeg setpts) to fit the natural-rate dub audio.
|
||||
@@ -100,13 +100,13 @@ class DubRequest(BaseModel):
|
||||
# mild pitch-preserving audio speed-up (≤1.2× alone,
|
||||
# ≤1.5× in hybrid) and a mild per-segment video
|
||||
# slow-down (≤2.0×), per services/fit_planner.py.
|
||||
# Residual overflow is trimmed and surfaced.
|
||||
# "strict_slot" — legacy: keep `slot_fit` semantics (atempo squeeze
|
||||
# when audio > slot). Kept for back-compat.
|
||||
# Residual overflow fails without discarding words.
|
||||
# "strict_slot" — pitch-preserving fit of the complete speech to
|
||||
# the original start/end; may sound faster or slower.
|
||||
timing_strategy: Optional[Literal["concise", "stretch_video", "strict_slot", "smart_fit"]] = "concise"
|
||||
|
||||
# Per-job slip budget for "concise" mode. Hard-trim only kicks in once
|
||||
# gap absorption + this much extra time has been consumed.
|
||||
# Per-job slip budget for "concise" mode. Overflow fails once gap
|
||||
# absorption + this much extra time has been consumed.
|
||||
overflow_budget_s: Optional[float] = 0.0
|
||||
|
||||
# Knob overrides for `smart_fit` (ignored by other strategies). Omitted
|
||||
@@ -161,6 +161,7 @@ class TranslateRequest(BaseModel):
|
||||
# voseo: "vos sos" instead of "tú eres"). Non-LLM providers (Argos, NLLB,
|
||||
# Google) can't honor it; the response then carries dialect_applied=false.
|
||||
dialect: Optional[str] = None
|
||||
translation_instructions: Optional[str] = Field(default=None, max_length=5000)
|
||||
# Two-stage LLM translation quality (provider="openai" only; MT engines
|
||||
# ignore both). None = default ON for the LLM engine.
|
||||
# auto_glossary — one up-front LLM pass over the full transcript extracts
|
||||
@@ -194,6 +195,7 @@ class AgentFitSegment(BaseModel):
|
||||
class AgentFitRequest(BaseModel):
|
||||
"""Revise only rendered lines that missed their exact timeline slot."""
|
||||
|
||||
translation_instructions: Optional[str] = Field(default=None, max_length=5000)
|
||||
segments: List[AgentFitSegment]
|
||||
target_lang: str
|
||||
|
||||
|
||||
@@ -167,6 +167,8 @@ async def run_transcribe_guarded(executor, fn, *, what: str = "ASR",
|
||||
immediately; running work calls it from the worker finalizer. Normal
|
||||
completion leaves cleanup with the caller.
|
||||
"""
|
||||
from services.inference_cancellation import InferenceCancellation
|
||||
cancellation = InferenceCancellation()
|
||||
loop = asyncio.get_running_loop()
|
||||
# Same SystemExit containment as the TTS pool (#1133 class): an ASR
|
||||
# dependency written as a CLI must not be able to shut the backend down.
|
||||
@@ -192,7 +194,8 @@ async def run_transcribe_guarded(executor, fn, *, what: str = "ASR",
|
||||
|
||||
def _job():
|
||||
try:
|
||||
return inner()
|
||||
with cancellation.activate():
|
||||
return inner()
|
||||
finally:
|
||||
with abandon_lock:
|
||||
abandon_state["finished"] = True
|
||||
@@ -204,6 +207,7 @@ async def run_transcribe_guarded(executor, fn, *, what: str = "ASR",
|
||||
fut = asyncio.wrap_future(concurrent_fut, loop=loop)
|
||||
|
||||
def _abandon() -> None:
|
||||
cancellation.cancel()
|
||||
cancelled_before_start = concurrent_fut.cancel()
|
||||
with abandon_lock:
|
||||
abandon_state["requested"] = True
|
||||
@@ -286,6 +290,26 @@ def _ctranslate2_cudnn_ok() -> tuple[bool, str]:
|
||||
return True, "ready"
|
||||
|
||||
|
||||
def _ctranslate2_execstack_ok() -> tuple[bool, str]:
|
||||
"""Make CTranslate2 importable on kernels that refuse an executable stack.
|
||||
|
||||
ctranslate2 ≤4.4.0 — the version whisperx 3.4.5 pins, and 3.4.5 is the
|
||||
newest release that supports the Python 3.11 we ship — marks its native
|
||||
library's stack ``RWE``. Kernels that refuse the request fail the dlopen
|
||||
with "cannot enable executable stack", killing whisperx, faster-whisper
|
||||
*and* Argos translation (#692). :mod:`core.execstack` clears that one bit
|
||||
in place, so call this BEFORE importing either engine; it cheaply rechecks the library and
|
||||
only writes when the library would otherwise refuse to load.
|
||||
"""
|
||||
try:
|
||||
from core.execstack import ensure_ctranslate2_loadable
|
||||
|
||||
return ensure_ctranslate2_loadable()
|
||||
except Exception as e: # noqa: BLE001 — a broken repair must not block ASR
|
||||
logger.debug("exec-stack repair unavailable (%s) — continuing", e)
|
||||
return True, "repair probe unavailable"
|
||||
|
||||
|
||||
def _decode_audio_16k_mono(audio_path: str):
|
||||
"""Decode `audio_path` to a 16 kHz mono float32 waveform using VoiceStudio's
|
||||
*validated* ffmpeg, instead of whisperx.load_audio's bare ``"ffmpeg"`` PATH
|
||||
@@ -657,6 +681,9 @@ class WhisperXBackend(ASRBackend):
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
ct2_ok, ct2_detail = _ctranslate2_execstack_ok()
|
||||
if not ct2_ok:
|
||||
return False, f"whisperx cannot load CTranslate2: {ct2_detail}"
|
||||
try:
|
||||
import whisperx # noqa: F401
|
||||
except ImportError as e:
|
||||
@@ -684,6 +711,14 @@ class WhisperXBackend(ASRBackend):
|
||||
# → speechbrain, or a stray k2_fsa redirect import aborts ASR on Windows
|
||||
# (#630/#611/#647). No-op on macOS/Linux and when speechbrain is absent.
|
||||
_harden_speechbrain_lazy_imports()
|
||||
# #692: repair CTranslate2's exec-stack request before the import that
|
||||
# would be rejected by it. Memoized, so this is free after the probe.
|
||||
ct2_ok, ct2_detail = _ctranslate2_execstack_ok()
|
||||
if not ct2_ok:
|
||||
# ImportError (not RuntimeError): this IS a native-import failure,
|
||||
# and the sentinel lets load_active_asr_backend degrade to the next
|
||||
# engine instead of failing ASR wholesale (#1185).
|
||||
raise ImportError(f"whisperx cannot load CTranslate2: {ct2_detail}")
|
||||
import whisperx
|
||||
# #723: re-check the CUDA pick against *currently free* VRAM — the TTS
|
||||
# model may have claimed the card since __init__. A too-big load dies
|
||||
@@ -1020,6 +1055,9 @@ class FasterWhisperBackend(ASRBackend):
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
ct2_ok, ct2_detail = _ctranslate2_execstack_ok()
|
||||
if not ct2_ok:
|
||||
return False, f"faster-whisper cannot load CTranslate2: {ct2_detail}"
|
||||
try:
|
||||
import faster_whisper # noqa: F401
|
||||
except ImportError as e:
|
||||
@@ -1034,6 +1072,9 @@ class FasterWhisperBackend(ASRBackend):
|
||||
def _ensure_model(self):
|
||||
if self._model is not None:
|
||||
return
|
||||
ct2_ok, ct2_detail = _ctranslate2_execstack_ok() # #692, see WhisperX
|
||||
if not ct2_ok:
|
||||
raise ImportError(f"faster-whisper cannot load CTranslate2: {ct2_detail}")
|
||||
from faster_whisper import WhisperModel
|
||||
# Device / compute-type auto-pick:
|
||||
# - CUDA present → GPU fp16
|
||||
@@ -1445,9 +1486,42 @@ class PyTorchWhisperBackend(ASRBackend):
|
||||
f"Underlying: {e}"
|
||||
) from e
|
||||
|
||||
#: Batch sizes to try on CUDA, largest first. The VRAM preflight only sizes
|
||||
#: the *weights*; generation adds an encoder/decoder workspace that scales
|
||||
#: with the batch, and `return_timestamps="word"` keeps every layer's
|
||||
#: cross-attention for the whole batch — gigabytes at batch 16. A card with
|
||||
#: room for the model can therefore still OOM at the first transcribe, which
|
||||
#: used to lose that chunk entirely (the dub retried the same batch size and
|
||||
#: gave up, leaving a hole in the transcript). Step down, then use CPU.
|
||||
_CUDA_BATCH_LADDER = (16, 4, 1)
|
||||
_CUDA_BATCH_LADDER_WORD_TS = (8, 2, 1)
|
||||
|
||||
@staticmethod
|
||||
def _is_oom(exc: BaseException) -> bool:
|
||||
try:
|
||||
import torch
|
||||
|
||||
if isinstance(exc, torch.cuda.OutOfMemoryError):
|
||||
return True
|
||||
except Exception: # noqa: BLE001 — classification must not raise
|
||||
pass
|
||||
return "out of memory" in str(exc).lower()
|
||||
|
||||
def _rebuild_on_cpu(self) -> None:
|
||||
"""Move the existing pipeline to CPU without resolving any model files."""
|
||||
import torch
|
||||
|
||||
# Keep the loaded checkpoint, tokenizer and feature extractor. Looking
|
||||
# up the default model here could download a different model offline.
|
||||
self._pipe.model.to(device="cpu", dtype=torch.float32)
|
||||
self._pipe.device = torch.device("cpu")
|
||||
try:
|
||||
torch.cuda.empty_cache()
|
||||
except Exception:
|
||||
pass # Some builds have no CUDA cache to release.
|
||||
|
||||
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
|
||||
import soundfile as sf
|
||||
import torch
|
||||
self._ensure_pipe()
|
||||
# #2039: libsndfile cannot open MP4/M4A (AAC), which /transcribe and
|
||||
# the MCP tool both accept. Those decode through the validated ffmpeg
|
||||
@@ -1460,15 +1534,67 @@ class PyTorchWhisperBackend(ASRBackend):
|
||||
audio_np, sr = _decode_audio_16k_mono(audio_path), 16000
|
||||
if audio_np.ndim > 1:
|
||||
audio_np = audio_np.mean(axis=1)
|
||||
bs = 16 if torch.cuda.is_available() else 2
|
||||
result = self._pipe(
|
||||
{"array": audio_np, "sampling_rate": sr},
|
||||
return_timestamps="word" if word_timestamps else True,
|
||||
chunk_length_s=15,
|
||||
batch_size=bs,
|
||||
)
|
||||
|
||||
def _run(batch_size: int):
|
||||
return self._pipe(
|
||||
{"array": audio_np, "sampling_rate": sr},
|
||||
return_timestamps="word" if word_timestamps else True,
|
||||
chunk_length_s=15,
|
||||
batch_size=batch_size,
|
||||
)
|
||||
|
||||
if self._on_cuda():
|
||||
ladder = (
|
||||
self._CUDA_BATCH_LADDER_WORD_TS if word_timestamps
|
||||
else self._CUDA_BATCH_LADDER
|
||||
)
|
||||
for i, bs in enumerate(ladder):
|
||||
try:
|
||||
result = _run(bs)
|
||||
break
|
||||
except Exception as e: # noqa: BLE001 — only OOM is retryable
|
||||
if not self._is_oom(e):
|
||||
raise
|
||||
try:
|
||||
import torch
|
||||
|
||||
torch.cuda.empty_cache()
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
if i + 1 < len(ladder):
|
||||
logger.warning(
|
||||
"PyTorch Whisper CUDA OOM at batch_size=%d — "
|
||||
"retrying at %d. Free VRAM (Flush models, close "
|
||||
"other GPU apps) for full-speed ASR.",
|
||||
bs, ladder[i + 1],
|
||||
)
|
||||
continue
|
||||
# Smallest batch still OOMs: finish on CPU rather than
|
||||
# return an empty chunk the caller cannot distinguish
|
||||
# from silence.
|
||||
logger.warning(
|
||||
"PyTorch Whisper CUDA OOM even at batch_size=1 — "
|
||||
"transcribing on CPU (slower, same model). Detail: %s", e,
|
||||
)
|
||||
self._rebuild_on_cpu()
|
||||
result = _run(2)
|
||||
else:
|
||||
result = _run(2)
|
||||
return result if isinstance(result, dict) else {"chunks": [], "raw": result}
|
||||
|
||||
def _on_cuda(self) -> bool:
|
||||
"""Whether the built pipeline actually sits on a CUDA device.
|
||||
|
||||
`torch.cuda.is_available()` is the wrong question: `_pick_device()` may
|
||||
have chosen CPU on a CUDA host (low free VRAM), and a CPU pipeline must
|
||||
not be handed a CUDA-sized batch.
|
||||
"""
|
||||
try:
|
||||
device = getattr(self._pipe, "device", None)
|
||||
return "cuda" in str(device).lower()
|
||||
except Exception: # noqa: BLE001
|
||||
return False
|
||||
|
||||
|
||||
# ── NeMo Parakeet TDT (NVIDIA — Open ASR Leaderboard SOTA, 25 langs) ────────
|
||||
|
||||
|
||||
@@ -186,6 +186,26 @@ def trim_trailing_silence(
|
||||
return audio_tensor[..., :end]
|
||||
|
||||
|
||||
|
||||
def trim_speech_padding(audio_tensor: torch.Tensor, sample_rate: int) -> torch.Tensor:
|
||||
"""Remove generated edge silence before timing, retaining 50 ms of context.
|
||||
|
||||
Never compress silence into the spoken slot or delete internal pauses.
|
||||
Silent/invalid outputs remain intact for the generation integrity guard.
|
||||
"""
|
||||
if audio_tensor.numel() == 0 or sample_rate <= 0:
|
||||
return audio_tensor
|
||||
envelope = audio_tensor.abs()
|
||||
if envelope.ndim > 1:
|
||||
envelope = envelope.amax(dim=tuple(range(envelope.ndim - 1)))
|
||||
voiced = torch.nonzero(envelope > 10 ** (-50 / 20))
|
||||
if voiced.numel() == 0:
|
||||
return audio_tensor
|
||||
margin = int(sample_rate * 0.05)
|
||||
start = max(0, int(voiced[0].item()) - margin)
|
||||
end = min(audio_tensor.shape[-1], int(voiced[-1].item()) + 1 + margin)
|
||||
return audio_tensor[..., start:end]
|
||||
|
||||
def apply_effects_chain(audio_tensor, sample_rate: int, chain: list[dict]) -> torch.Tensor:
|
||||
"""Apply a chain of named effects to an audio tensor.
|
||||
|
||||
|
||||
@@ -204,6 +204,42 @@ def _safe_torchaudio_save(
|
||||
fmt, e,
|
||||
)
|
||||
torchaudio.save(path_or_buf, tensor, sample_rate, format=fmt)
|
||||
except (ImportError, RuntimeError) as e:
|
||||
if isinstance(e, RuntimeError) and "could not load libtorchcodec" not in str(e).lower():
|
||||
raise _describe_write_failure(e, path_or_buf) from e
|
||||
# torchaudio >= 2.9 routes save() through TorchCodec, which needs
|
||||
# FFmpeg *shared libraries* on the system. Where those are absent the
|
||||
# write raises ImportError and every generation fails. #1931 guarded
|
||||
# set_audio_backend() against that torchaudio but left save() itself
|
||||
# unprotected; arm64 CUDA hosts reach it unavoidably, since torch
|
||||
# 2.8.0 publishes no aarch64 wheel. soundfile is already a locked
|
||||
# dependency and the tensor is normalized by this point, so hand it to
|
||||
# the audited sibling helper rather than failing the request.
|
||||
logger.warning(
|
||||
"torchaudio.save needs TorchCodec (%s); writing via soundfile", e
|
||||
)
|
||||
if hasattr(path_or_buf, "seek") and hasattr(path_or_buf, "truncate"):
|
||||
try:
|
||||
path_or_buf.seek(0)
|
||||
path_or_buf.truncate(0)
|
||||
except (OSError, io.UnsupportedOperation):
|
||||
pass # Non-seekable streams cannot be rewound; preserve fallback behavior.
|
||||
_subtype = {
|
||||
"wav": "FLOAT" if bits_per_sample == 32 else "PCM_16",
|
||||
"flac": "PCM_16",
|
||||
"ogg": "VORBIS",
|
||||
"mp3": "MPEG_LAYER_III",
|
||||
}.get(fmt, "PCM_16")
|
||||
try:
|
||||
_safe_soundfile_write(
|
||||
path_or_buf,
|
||||
tensor.transpose(0, 1).contiguous().numpy(),
|
||||
sample_rate,
|
||||
subtype=_subtype,
|
||||
format=fmt.upper(),
|
||||
)
|
||||
except Exception as e2:
|
||||
raise _describe_write_failure(e2, path_or_buf) from e2
|
||||
except Exception as e:
|
||||
# #1221: libsndfile reports OS-level write failures as a bare
|
||||
# "LibsndfileError: System error." — no path, no errno, nothing the
|
||||
@@ -267,6 +303,7 @@ def _safe_soundfile_write(
|
||||
sample_rate: int,
|
||||
*,
|
||||
subtype: str = "PCM_16",
|
||||
format: str | None = None,
|
||||
) -> None:
|
||||
"""Sibling helper for the one in-tree ``sf.write`` site.
|
||||
|
||||
@@ -284,6 +321,10 @@ def _safe_soundfile_write(
|
||||
subtype: Soundfile subtype string. ``"PCM_16"`` (default) for
|
||||
standard 16-bit PCM WAV; ``"PCM_24"``, ``"FLOAT"`` etc.
|
||||
also work.
|
||||
format: Container format (``"WAV"``, ``"FLAC"``, ``"OGG"``,
|
||||
``"MP3"``). ``None`` lets soundfile infer it from the path's
|
||||
extension — which it cannot do for a file-like object, so
|
||||
callers passing a buffer must name it.
|
||||
|
||||
Raises:
|
||||
ValueError: if the array is empty.
|
||||
@@ -321,7 +362,7 @@ def _safe_soundfile_write(
|
||||
samples = np.ascontiguousarray(samples)
|
||||
|
||||
_ensure_audio_parent(path)
|
||||
sf.write(path, samples, sample_rate, subtype=subtype)
|
||||
sf.write(path, samples, sample_rate, subtype=subtype, format=format)
|
||||
|
||||
|
||||
def atomic_save_wav(
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
"""Dialogue-only replacement beds: original outside speech, separated bed inside."""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
|
||||
from services.video_retime import expand_retime_chunks
|
||||
|
||||
RATE = 48000
|
||||
FADE_S = .01
|
||||
_locks: dict[str, asyncio.Lock] = {}
|
||||
|
||||
|
||||
def dialogue_intervals(segments: list[dict]) -> list[tuple[float, float]]:
|
||||
intervals = []
|
||||
for row in segments:
|
||||
a, b = float(row['start']), float(row['end'])
|
||||
if not math.isfinite(a) or not math.isfinite(b) or a < 0 or b <= a:
|
||||
raise ValueError('Invalid dialogue interval')
|
||||
intervals.append((a, b))
|
||||
merged: list[tuple[float, float]] = []
|
||||
for a, b in sorted(intervals):
|
||||
if merged and a <= merged[-1][1]:
|
||||
merged[-1] = (merged[-1][0], max(b, merged[-1][1]))
|
||||
else:
|
||||
merged.append((a, b))
|
||||
return merged
|
||||
|
||||
|
||||
def splice_background(original: str, separated: str, output: str, intervals: list[tuple[float, float]]) -> None:
|
||||
"""Stream in bounded memory; crossfades lie INSIDE dialogue intervals."""
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
|
||||
with sf.SoundFile(original) as src, sf.SoundFile(separated) as bed:
|
||||
if src.samplerate != bed.samplerate or src.channels != bed.channels:
|
||||
raise ValueError('Background inputs must have matching sample format')
|
||||
if bed.frames < src.frames - int(.1 * src.samplerate):
|
||||
raise ValueError('Separated background is incomplete')
|
||||
with sf.SoundFile(output, 'w', samplerate=src.samplerate, channels=src.channels, subtype='FLOAT') as out:
|
||||
offset = 0
|
||||
active = 0
|
||||
while True:
|
||||
wave = src.read(65536, dtype='float32', always_2d=True)
|
||||
if not len(wave):
|
||||
break
|
||||
background = bed.read(len(wave), dtype='float32', always_2d=True)
|
||||
if len(background) < len(wave):
|
||||
background = np.pad(background, ((0, len(wave)-len(background)), (0, 0)))
|
||||
times = np.arange(offset, offset + len(wave)) / src.samplerate
|
||||
mask = np.zeros(len(wave), dtype='float32')
|
||||
while active < len(intervals) and intervals[active][1] < times[0]:
|
||||
active += 1
|
||||
for a, b in intervals[active:]:
|
||||
if a > times[-1]:
|
||||
break
|
||||
fade = min(FADE_S, (b-a)/2)
|
||||
envelope = np.clip(np.minimum((times-a)/fade, (b-times)/fade), 0, 1)
|
||||
mask = np.maximum(mask, envelope)
|
||||
out.write(wave * (1-mask[:, None]) + background * mask[:, None])
|
||||
offset += len(wave)
|
||||
|
||||
|
||||
async def _checked(cmd: list[str]) -> None:
|
||||
rc, _, error = await run_ffmpeg(cmd, timeout=1800.0)
|
||||
if rc:
|
||||
raise RuntimeError('Could not preserve original background audio: ' + str(error)[-500:])
|
||||
|
||||
|
||||
async def surgical_background(source: str, separated: str, cache_dir: str, segments: list[dict], plan: list[dict], duration: float) -> str:
|
||||
for chunk in plan:
|
||||
ratio = float(chunk["stretch_ratio"])
|
||||
if not math.isfinite(ratio) or ratio <= 0:
|
||||
raise ValueError("Invalid background retiming ratio")
|
||||
intervals = dialogue_intervals(segments)
|
||||
if not intervals:
|
||||
raise ValueError('Dialogue timing is required to preserve original background audio')
|
||||
identity = [(p, os.stat(p).st_size, os.stat(p).st_mtime_ns) for p in (source, separated)]
|
||||
key = hashlib.sha256(json.dumps([1, identity, intervals, plan, duration], sort_keys=True).encode()).hexdigest()[:24]
|
||||
target = str(Path(cache_dir) / f'surgical_{key}.wav')
|
||||
async with _locks.setdefault(target, asyncio.Lock()):
|
||||
if os.path.isfile(target):
|
||||
return target
|
||||
ffmpeg = find_ffmpeg()
|
||||
with tempfile.TemporaryDirectory(prefix='.surgical-', dir=cache_dir) as tmp:
|
||||
original, bed, spliced = [str(Path(tmp)/name) for name in ('source.wav', 'bed.wav', 'spliced.wav')]
|
||||
for inp, out in ((source, original), (separated, bed)):
|
||||
await _checked([ffmpeg, '-y', '-i', inp, '-map', '0:a:0', '-vn', '-ar', str(RATE), '-ac', '2', '-c:a', 'pcm_f32le', out])
|
||||
await asyncio.to_thread(splice_background, original, bed, spliced, intervals)
|
||||
if plan and any(abs(float(p['stretch_ratio'])-1) > 1e-6 for p in plan):
|
||||
chunks = expand_retime_chunks(plan, duration)
|
||||
# Bound filter buffering for long projects; trim each batch's
|
||||
# input before splitting it among the chunk filters.
|
||||
batches = []
|
||||
for batch_index in range(0, len(chunks), 16):
|
||||
batch = chunks[batch_index:batch_index+16]
|
||||
origin = batch[0][0]
|
||||
filters = []
|
||||
for i, (a, b, ratio) in enumerate(batch):
|
||||
rate = 1 / ratio
|
||||
tempos = []
|
||||
while rate < .5:
|
||||
tempos.append('atempo=0.5')
|
||||
rate /= .5
|
||||
while rate > 2:
|
||||
tempos.append('atempo=2')
|
||||
rate /= 2
|
||||
tempos.append(f'atempo={rate:.9f}')
|
||||
length = (b-a)*ratio
|
||||
filters.append(f'[0:a]atrim=start={a-origin:.9f}:end={b-origin:.9f},asetpts=PTS-STARTPTS,' + ','.join(tempos) + f',apad,atrim=duration={length:.9f}[c{i}]')
|
||||
filters.append(''.join(f'[c{i}]' for i in range(len(batch))) + f'concat=n={len(batch)}:v=0:a=1[out]')
|
||||
script = Path(tmp)/'retime.txt'
|
||||
script.write_text(';'.join(filters))
|
||||
batch_name = f'batch{batch_index}.wav'
|
||||
output = str(Path(tmp)/batch_name)
|
||||
await _checked([ffmpeg, '-y', '-ss', str(origin), '-t', str(batch[-1][1]-origin), '-i', spliced, '-filter_complex_script', str(script), '-map', '[out]', '-c:a', 'pcm_f32le', output])
|
||||
batches.append(batch_name)
|
||||
listing = Path(tmp)/'concat.txt'
|
||||
listing.write_text(''.join(f"file '{name}'\n" for name in batches))
|
||||
retimed = str(Path(tmp)/'retimed.wav')
|
||||
await _checked([ffmpeg, '-y', '-f', 'concat', '-safe', '1', '-i', str(listing), '-c:a', 'copy', retimed])
|
||||
spliced = retimed
|
||||
os.replace(spliced, target)
|
||||
return target
|
||||
@@ -64,6 +64,23 @@ from core.logging_utils import log_safe
|
||||
|
||||
logger = logging.getLogger("omnivoice.dub_pipeline")
|
||||
|
||||
|
||||
def _media_process_error(tool: str, returncode: int, stderr: bytes, *, paths=()) -> str:
|
||||
"""Keep the actionable end of native diagnostics without leaking paths."""
|
||||
from core.scrub import scrub_text
|
||||
detail = stderr.decode(errors="replace")
|
||||
# Input/output may live outside home directories (mounted media, Windows
|
||||
# drive roots). Remove the exact command paths before generic scrubbing.
|
||||
for path in sorted((str(p) for p in paths if p), key=len, reverse=True):
|
||||
for variant in {path, path.replace("\\", "/"), path.replace("/", "\\")}:
|
||||
detail = detail.replace(variant, "[redacted path]")
|
||||
detail = scrub_text(detail).strip()
|
||||
tail = detail[-2000:]
|
||||
if len(detail) > 2000:
|
||||
tail = "…" + tail
|
||||
return f"{tool} exited with code {returncode}" + (f": {tail}" if tail else ". No diagnostic output.")
|
||||
|
||||
|
||||
# ── Module-level state ──────────────────────────────────────────────────────
|
||||
# These used to live in dub_core.py. The router now re-exports them for
|
||||
# backward compat during the transition.
|
||||
@@ -1326,7 +1343,7 @@ async def ingest_pipeline(
|
||||
"-ar", "16000", "-ac", "1", audio_path, "-y",
|
||||
])
|
||||
if p.returncode != 0:
|
||||
msg = (stderr.decode(errors="replace") or f"ffmpeg returned exit code {p.returncode}").strip()[:500]
|
||||
msg = _media_process_error("FFmpeg", p.returncode, stderr, paths=(video_path, audio_path, job_dir))
|
||||
raise Exception(msg)
|
||||
# Second, FULL-QUALITY extraction for source separation. audio.wav
|
||||
# is deliberately 16 kHz mono — that's what ASR wants — but Demucs
|
||||
@@ -1476,7 +1493,7 @@ async def ingest_pipeline(
|
||||
elif evt[0] == "done":
|
||||
rc, stderr_full = evt[1], evt[2]
|
||||
if rc != 0:
|
||||
raise Exception(stderr_full.decode(errors="replace")[:500])
|
||||
raise Exception(_media_process_error("Demucs", rc, stderr_full, paths=(audio_hq_path, audio_path, job_dir)))
|
||||
# Stems land under the INPUT's basename ("audio_hq" when the
|
||||
# full-quality extraction succeeded, "audio" on its fallback).
|
||||
demucs_out = os.path.join(
|
||||
|
||||
@@ -17,7 +17,6 @@ from __future__ import annotations
|
||||
import importlib.util
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger("omnivoice.engine_env")
|
||||
@@ -29,6 +28,53 @@ _TORCH_COMPILE_KEY = "perf.torch_compile_disabled"
|
||||
# (e.g. a brand-new architecture running through PTX forward-compat).
|
||||
_FORCE_COMPILE_ENV = "OMNIVOICE_FORCE_TORCH_COMPILE"
|
||||
|
||||
# #2135: the environment escape hatches that torch itself honours. `main.py`
|
||||
# sets TORCH_COMPILE_DISABLE/TORCHDYNAMO_DISABLE on win32, `build_engine_env`
|
||||
# injects TORCH_COMPILE_DISABLE into engine subprocesses, and
|
||||
# `docs/install/windows.md` tells users to export it — but the in-process gate
|
||||
# below never read them, so an operator who set the documented variable still
|
||||
# got a compiled model (and, on a cudagraph mode, a native crash they could not
|
||||
# turn off). Reading them here makes one knob mean one thing everywhere.
|
||||
_COMPILE_DISABLE_ENVS = (
|
||||
"TORCH_COMPILE_DISABLE",
|
||||
"TORCHDYNAMO_DISABLE",
|
||||
"TORCHINDUCTOR_DISABLE",
|
||||
)
|
||||
|
||||
_TRUTHY = frozenset({"1", "true", "yes", "on"})
|
||||
|
||||
|
||||
def _env_compile_disabled() -> Optional[str]:
|
||||
"""The name of the first set-and-truthy compile-disable env var, else None.
|
||||
|
||||
Mirrors torch's own reading of these variables so the app's decision and
|
||||
torch's behaviour cannot disagree — the state the reporter in #2135 hit,
|
||||
where the log said "torch.compile applied" while TORCH_COMPILE_DISABLE=1
|
||||
was exported.
|
||||
"""
|
||||
for name in _COMPILE_DISABLE_ENVS:
|
||||
if os.environ.get(name, "").strip().lower() in _TRUTHY:
|
||||
return name
|
||||
return None
|
||||
|
||||
|
||||
def _settings_db_path() -> str:
|
||||
"""The settings DB the compile toggle is actually read from (best-effort).
|
||||
|
||||
Logged alongside the toggle because #2135's reporter had three
|
||||
`omnivoice.db` files on the box and edited one the backend never opened;
|
||||
naming the path turns "the setting doesn't work" into a one-line diagnosis.
|
||||
"""
|
||||
try:
|
||||
from core.config import DB_PATH
|
||||
|
||||
from core.scrub import scrub_text
|
||||
|
||||
return scrub_text(str(DB_PATH))
|
||||
except Exception:
|
||||
return "<unknown>"
|
||||
|
||||
|
||||
# #278: set (with a reason) the first time torch.compile — or *running* the
|
||||
# compiled model — fails at runtime in this process. Once set, every later
|
||||
# load in the same session goes straight to eager instead of re-tripping the
|
||||
@@ -153,10 +199,12 @@ def mark_flashinfer_runtime_failure(reason: str) -> None:
|
||||
def _cuda_arch_supported_for_compile() -> "tuple[bool, str]":
|
||||
"""Check the GPU's architecture against this torch build's arch list.
|
||||
|
||||
New GPU architectures (e.g. Blackwell sm_120, issue #278) routinely break
|
||||
torch.compile/Triton before upstream support lands: the eager model runs
|
||||
via PTX forward-compat, but Inductor/Triton kernel compilation targets the
|
||||
new arch directly and fails mid-generation. If the device's arch tag is
|
||||
A new GPU architecture routinely breaks torch.compile/Triton before
|
||||
upstream support lands (issue #278): the eager model runs via PTX
|
||||
forward-compat, but Inductor/Triton kernel compilation targets the new arch
|
||||
directly and fails mid-generation. Blackwell sm_120 was that case; it no
|
||||
longer is on the pinned torch 2.8.0+cu128, where this probe can return
|
||||
supported; independent compiler/runtime failures still need eager fallback. If the device's arch tag is
|
||||
absent from this build's arch list we treat compile as unsupported and use
|
||||
eager. The comparison is delegated to ``core.device_caps.arch_unsupported``
|
||||
so it stays CUDA/ROCm-aware — a ROCm build lists ``gfx…`` names, and the
|
||||
@@ -251,6 +299,15 @@ def should_torch_compile(device: str) -> bool:
|
||||
"""
|
||||
if device != "cuda":
|
||||
return False
|
||||
# #2135: honoured before every other gate — an explicit env opt-out is the
|
||||
# user's most direct statement of intent, and it must hold on every
|
||||
# platform (the reporter was on Linux, where this used to be ignored).
|
||||
disabled_by = _env_compile_disabled()
|
||||
if disabled_by is not None:
|
||||
logger.info(
|
||||
"torch.compile skipped: %s is set — using eager mode.", disabled_by,
|
||||
)
|
||||
return False
|
||||
if importlib.util.find_spec("triton") is None:
|
||||
logger.info("torch.compile skipped: Triton unavailable — using eager mode.")
|
||||
return False
|
||||
@@ -258,8 +315,18 @@ def should_torch_compile(device: str) -> bool:
|
||||
from services import settings_store
|
||||
|
||||
if settings_store.get_text(_TORCH_COMPILE_KEY, "0") == "1":
|
||||
logger.info("torch.compile skipped: disabled in Settings (Performance).")
|
||||
logger.info(
|
||||
"torch.compile skipped: disabled in Settings (Performance) [%s].",
|
||||
_settings_db_path(),
|
||||
)
|
||||
return False
|
||||
# #2135: say which DB answered "not disabled". Without this the only
|
||||
# observable outcome of a toggle that never reached the running
|
||||
# backend is a log line saying compile was applied anyway.
|
||||
logger.debug(
|
||||
"torch.compile: %s not set in %s — compile remains eligible.",
|
||||
_TORCH_COMPILE_KEY, _settings_db_path(),
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("should_torch_compile: settings read failed; proceeding")
|
||||
if _compile_runtime_failure is not None:
|
||||
@@ -328,19 +395,31 @@ def build_engine_env(
|
||||
except Exception:
|
||||
logger.exception("build_engine_env: token resolver failed (non-fatal)")
|
||||
|
||||
# INST-12: TORCH_COMPILE_DISABLE on Windows when the user opted in.
|
||||
# The flag is a Windows-only escape hatch — torch.compile OOMs the same
|
||||
# Triton kernel cache differently on macOS/Linux, so injecting on those
|
||||
# platforms would just slow the engine for no gain. (The in-process
|
||||
# should_torch_compile() gate handles the automatic Triton-absence case;
|
||||
# the subprocess var stays user-driven by design — see test_perf_settings.)
|
||||
if sys.platform.startswith("win"):
|
||||
try:
|
||||
from services import settings_store
|
||||
# INST-12 (#65), widened to every platform by #2135: TORCH_COMPILE_DISABLE
|
||||
# when the user opted in. This was win32-only on the theory that
|
||||
# torch.compile only misbehaves on Windows (no Triton wheel). #2135 is the
|
||||
# counter-example — a Linux/CUDA host where compile crashes the engine —
|
||||
# and a Settings toggle that silently does nothing on the user's platform
|
||||
# is worse than no toggle at all. Cost when enabled on Linux/macOS is a
|
||||
# slower engine, which is exactly what the user asked for by enabling it.
|
||||
try:
|
||||
from services import settings_store
|
||||
|
||||
if settings_store.get_text(_TORCH_COMPILE_KEY, "0") == "1":
|
||||
env["TORCH_COMPILE_DISABLE"] = "1"
|
||||
except Exception:
|
||||
logger.exception("build_engine_env: torch_compile_disabled read failed")
|
||||
if settings_store.get_text(_TORCH_COMPILE_KEY, "0") == "1":
|
||||
env["TORCH_COMPILE_DISABLE"] = "1"
|
||||
except Exception:
|
||||
logger.exception("build_engine_env: torch_compile_disabled read failed")
|
||||
|
||||
# #2135: an env opt-out on the parent must reach the child too. Without
|
||||
# this a user who exported TORCH_COMPILE_DISABLE=1 got an eager parent and
|
||||
# a compiled sidecar — the inconsistency that made the flag look ignored.
|
||||
disabled_by = _env_compile_disabled()
|
||||
if disabled_by is not None:
|
||||
if env.get("TORCH_COMPILE_DISABLE") != "1":
|
||||
logger.debug(
|
||||
"build_engine_env: %s is set — disabling torch.compile in the "
|
||||
"engine subprocess too.", disabled_by,
|
||||
)
|
||||
env["TORCH_COMPILE_DISABLE"] = "1"
|
||||
|
||||
return env
|
||||
|
||||
@@ -89,6 +89,7 @@ def bed_mix_filter(
|
||||
duration: str = "longest",
|
||||
tail: str = "",
|
||||
uniq: str = "",
|
||||
bed_gain: float = BED_GAIN,
|
||||
) -> str:
|
||||
"""One ffmpeg filter chain mixing `voice_in` over `bed_in` at original level.
|
||||
|
||||
@@ -110,22 +111,22 @@ def bed_mix_filter(
|
||||
# Gains applied per input, amix reduced to a plain sum: levels are
|
||||
# exact for the whole timeline, including after either stream ends.
|
||||
return (
|
||||
f"[{bed_in}]aresample={BED_MIX_SAMPLE_RATE},{stereo},volume={BED_GAIN:g}[{b}];"
|
||||
f"[{bed_in}]aresample={BED_MIX_SAMPLE_RATE},{stereo},volume={bed_gain:g}[{b}];"
|
||||
f"[{voice_in}]aresample={BED_MIX_SAMPLE_RATE},{stereo},volume={VOICE_GAIN:g}[{v}];"
|
||||
f"[{b}][{v}]amix=inputs=2:duration={duration}:dropout_transition=2:"
|
||||
f"normalize=0,alimiter=level=false:limit=0.98{tail}[{out}]"
|
||||
f"normalize=0,alimiter=level=false:limit=0.98:latency=1{tail}[{out}]"
|
||||
)
|
||||
# Legacy ffmpeg (<5, no `normalize`): cancel amix's normalization with a
|
||||
# compensating multiply. Exact while both streams run; if one ends early
|
||||
# the tail is over-boosted into the limiter until the graph ends — a known
|
||||
# quirk accepted only on old ffmpeg, where the alternative is no export.
|
||||
total = BED_GAIN + VOICE_GAIN
|
||||
total = bed_gain + VOICE_GAIN
|
||||
return (
|
||||
f"[{bed_in}]aresample={BED_MIX_SAMPLE_RATE},{stereo}[{b}];"
|
||||
f"[{voice_in}]aresample={BED_MIX_SAMPLE_RATE},{stereo}[{v}];"
|
||||
f"[{b}][{v}]amix=inputs=2:duration={duration}:dropout_transition=2:"
|
||||
f"weights={BED_GAIN:g} {VOICE_GAIN:g},volume={total:g},"
|
||||
f"alimiter=level=false:limit=0.98{tail}[{out}]"
|
||||
f"weights={bed_gain:g} {VOICE_GAIN:g},volume={total:g},"
|
||||
f"alimiter=level=false:limit=0.98:latency=1{tail}[{out}]"
|
||||
)
|
||||
|
||||
|
||||
@@ -173,12 +174,12 @@ def _binary_runs(path: str) -> bool:
|
||||
if cached is not None:
|
||||
return cached
|
||||
try:
|
||||
subprocess.run(
|
||||
result = subprocess.run(
|
||||
[path, "-version"],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||
timeout=10, check=False,
|
||||
)
|
||||
ok = True
|
||||
ok = result.returncode == 0
|
||||
except (OSError, subprocess.TimeoutExpired, subprocess.SubprocessError) as e:
|
||||
logger.warning(
|
||||
"Rejecting non-runnable ffmpeg/ffprobe candidate %s: %s",
|
||||
@@ -308,8 +309,11 @@ def find_ffprobe():
|
||||
try:
|
||||
ffmpeg_path = find_ffmpeg()
|
||||
if ffmpeg_path:
|
||||
candidate = ffmpeg_path.replace("ffmpeg", "ffprobe")
|
||||
if os.path.isfile(candidate):
|
||||
candidate = os.path.join(
|
||||
os.path.dirname(ffmpeg_path),
|
||||
os.path.basename(ffmpeg_path).replace("ffmpeg", "ffprobe"),
|
||||
)
|
||||
if os.path.isfile(candidate) and _binary_runs(candidate):
|
||||
return candidate
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Request cancellation visible to killable sidecars on executor threads.
|
||||
|
||||
In-process native calls remain accounted for until they return; only sidecars
|
||||
can safely terminate early. A scope belongs to one job, never to a pool thread.
|
||||
"""
|
||||
from contextlib import contextmanager
|
||||
import threading
|
||||
|
||||
_local = threading.local()
|
||||
|
||||
|
||||
class InferenceCancellation:
|
||||
def __init__(self):
|
||||
self.cancelled = threading.Event()
|
||||
|
||||
def cancel(self):
|
||||
self.cancelled.set()
|
||||
|
||||
@contextmanager
|
||||
def activate(self):
|
||||
previous = getattr(_local, "scope", None)
|
||||
_local.scope = self
|
||||
try:
|
||||
if self.cancelled.is_set():
|
||||
raise RuntimeError("Inference request was cancelled before execution")
|
||||
yield
|
||||
finally:
|
||||
_local.scope = previous
|
||||
|
||||
|
||||
def current_cancellation():
|
||||
return getattr(_local, "scope", None)
|
||||
@@ -1,11 +1,12 @@
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import asyncio
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import queue
|
||||
import re
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import Executor, Future, ThreadPoolExecutor
|
||||
|
||||
from utils.containment import contain_system_exit
|
||||
@@ -527,6 +528,7 @@ def generate_timeout_s(
|
||||
text: "str | None", *, engine: object = None, execution_device: "str | None" = None,
|
||||
min_vram_gb: float = 0.0, hardware_family: "str | None" = None,
|
||||
vram_gb: "float | None" = None,
|
||||
_include_sidecar_grace: bool = True,
|
||||
) -> float:
|
||||
"""THE wall-clock execution budget for one synthesis job, scaled to input.
|
||||
|
||||
@@ -562,6 +564,7 @@ def generate_timeout_s(
|
||||
claiming the card is under-provisioned in user-facing diagnostics.
|
||||
"""
|
||||
base = GPU_JOB_TIMEOUT_S
|
||||
explicit_budget = _GENERATE_TIMEOUT_EXPLICIT or GPU_JOB_TIMEOUT_S != _CONFIGURED_GPU_JOB_TIMEOUT_S
|
||||
try:
|
||||
from core.device_caps import detect_host_caps
|
||||
caps = detect_host_caps()
|
||||
@@ -588,6 +591,7 @@ def generate_timeout_s(
|
||||
)
|
||||
if family == "cpu" and (cpu_explicit or not universal_override):
|
||||
base = CPU_JOB_TIMEOUT_S
|
||||
explicit_budget = cpu_explicit
|
||||
elif not universal_override and family in (
|
||||
"cuda", "rocm", "vulkan", "xpu",
|
||||
):
|
||||
@@ -610,7 +614,23 @@ def generate_timeout_s(
|
||||
# Device probing is advisory here; the configured universal bound is
|
||||
# still safe when a platform probe is unavailable during startup.
|
||||
pass
|
||||
return base + (max(0, len(text or "") - 1200) / 40.0)
|
||||
|
||||
# If the engine specifies its own sidecar receive timeout (e.g. SubprocessBackend
|
||||
# engines like Confucius, Dots, Moss, Supertonic), the outer execution budget
|
||||
# must not cut the sidecar off early (#2103). A bounded 5s grace period ensures
|
||||
# the sidecar's watchdog timer fires and surfaces its actionable timeout error
|
||||
# before the outer pool cancellation cuts it off.
|
||||
sidecar_grace = 0.0
|
||||
if not explicit_budget and engine is not None and hasattr(engine, "recv_timeout_s"):
|
||||
try:
|
||||
sidecar_timeout = float(engine.recv_timeout_s)
|
||||
if math.isfinite(sidecar_timeout) and sidecar_timeout > 0:
|
||||
base = max(base, sidecar_timeout)
|
||||
sidecar_grace = 5.0 if _include_sidecar_grace else 0.0
|
||||
except (TypeError, ValueError):
|
||||
pass # Invalid optional engine metadata cannot disable the outer guard.
|
||||
|
||||
return base + (max(0, len(text or "") - 1200) / 40.0) + sidecar_grace
|
||||
|
||||
|
||||
def _retry_after_estimate(stats: dict) -> float:
|
||||
@@ -726,6 +746,8 @@ async def run_on_gpu_pool_guarded(fn, *, what: str = "GPU job",
|
||||
finalizer. Normal completion never calls it. This lets request-owned temp
|
||||
files outlive abandoned workers without delaying ordinary requests (#1668).
|
||||
"""
|
||||
from services.inference_cancellation import InferenceCancellation
|
||||
cancellation = InferenceCancellation()
|
||||
loop = asyncio.get_running_loop()
|
||||
ex = executor if executor is not None else _get_gpu_pool()
|
||||
# Resolved at CALL time, not def time, so monkeypatching/reloading the
|
||||
@@ -768,7 +790,8 @@ async def run_on_gpu_pool_guarded(fn, *, what: str = "GPU job",
|
||||
except RuntimeError:
|
||||
pass # loop already closed (caller vanished) — still run the job
|
||||
try:
|
||||
return _inner()
|
||||
with cancellation.activate():
|
||||
return _inner()
|
||||
finally:
|
||||
# Idents are reused by the OS; a stale heartbeat under this ident
|
||||
# must not vouch for some future job on the same thread.
|
||||
@@ -783,6 +806,7 @@ async def run_on_gpu_pool_guarded(fn, *, what: str = "GPU job",
|
||||
fut = asyncio.wrap_future(concurrent_fut, loop=loop)
|
||||
|
||||
def _abandon() -> None:
|
||||
cancellation.cancel()
|
||||
# Keep the concurrent future so we can distinguish a job cancelled out
|
||||
# of the queue from a thread that Python cannot stop once it has begun.
|
||||
cancelled_before_start = concurrent_fut.cancel()
|
||||
@@ -1565,10 +1589,13 @@ def _is_compile_runtime_failure(exc: BaseException) -> bool:
|
||||
"""True when an exception originates in the torch.compile stack (Dynamo /
|
||||
Inductor / Triton / FX / CUDA-graph trees) rather than in the model itself.
|
||||
|
||||
#278: on GPU architectures Triton doesn't support yet (e.g. Blackwell
|
||||
sm_120), the compiled model dies mid-generation with errors like
|
||||
"Detected that you are using FX to symbolically trace a dynamo-optimized
|
||||
function" or an AssertionError out of torch/_inductor/cudagraph_trees.py.
|
||||
#278: an independent compile-stack failure can surface during generation
|
||||
as an AssertionError out of torch/_inductor/cudagraph_trees.py. An
|
||||
architecture missing from the running torch build's arch list is rejected
|
||||
earlier by should_torch_compile(), before this runtime fallback applies.
|
||||
#278 also quotes "Detected that you are using FX to symbolically trace a
|
||||
dynamo-optimized function"; Dynamo raises that on any device, CPU included,
|
||||
so it is a compile-stack error to catch here but never an arch signal.
|
||||
Walks the exception chain and checks (a) the exception type's module,
|
||||
(b) the message, (c) the traceback file paths — the cudagraph case is a
|
||||
bare AssertionError, so the traceback check is load-bearing.
|
||||
@@ -1793,6 +1820,64 @@ _TORCH_COMPILE_MODE = "reduce-overhead"
|
||||
# would not.
|
||||
_CUDAGRAPH_COMPILE_MODES = frozenset({"reduce-overhead", "max-autotune"})
|
||||
|
||||
# ── #2135: CUDA-graph capture needs Ampere or newer ─────────────────────────
|
||||
# On a Turing T4 (sm_75) the cudagraph mode above took the whole backend
|
||||
# process down on the first generate — no Python traceback, no HTTP response,
|
||||
# just a dead PID (the native capture aborts below the interpreter, so neither
|
||||
# the #278 eager fallback nor any `except` can see it). The graph *capture* is
|
||||
# the risky part, not Inductor: dropping to the non-cudagraph "default" mode
|
||||
# keeps the compiled kernels (and most of the speedup) while removing the
|
||||
# crash surface. Ampere (sm_80) is the floor because that is where the app has
|
||||
# actual passing evidence; anything older takes the conservative path.
|
||||
_CUDAGRAPH_MIN_CAPABILITY = (8, 0)
|
||||
# Escape hatch in the other direction, for operators benchmarking on old GPUs.
|
||||
_FORCE_CUDAGRAPH_ENV = "OMNIVOICE_FORCE_CUDAGRAPH"
|
||||
|
||||
|
||||
def _resolve_compile_mode() -> str:
|
||||
"""The ``torch.compile`` mode to use on this GPU (#2135).
|
||||
|
||||
Returns the configured cudagraph mode on Ampere+, and the non-cudagraph
|
||||
``"default"`` on older architectures where graph capture has been observed
|
||||
to abort the process. Fails *safe* (→ "default") only when we positively
|
||||
identify a pre-Ampere device; any probe error keeps the configured mode so
|
||||
a weird torch build doesn't silently lose the optimization.
|
||||
"""
|
||||
if _TORCH_COMPILE_MODE not in _CUDAGRAPH_COMPILE_MODES:
|
||||
return _TORCH_COMPILE_MODE
|
||||
if os.environ.get(_FORCE_CUDAGRAPH_ENV, "").strip().lower() in {"1", "true", "yes", "on"}:
|
||||
logger.warning(
|
||||
"%s=1 — keeping torch.compile mode %r on a GPU where CUDA-graph "
|
||||
"capture is not known-good (#2135).",
|
||||
_FORCE_CUDAGRAPH_ENV, _TORCH_COMPILE_MODE,
|
||||
)
|
||||
return _TORCH_COMPILE_MODE
|
||||
try:
|
||||
import torch
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
return _TORCH_COMPILE_MODE
|
||||
capability = torch.cuda.get_device_capability(0)
|
||||
except Exception:
|
||||
logger.debug("compile-mode capability probe failed; keeping %r",
|
||||
_TORCH_COMPILE_MODE, exc_info=True)
|
||||
return _TORCH_COMPILE_MODE
|
||||
if tuple(capability) >= _CUDAGRAPH_MIN_CAPABILITY:
|
||||
return _TORCH_COMPILE_MODE
|
||||
try:
|
||||
device_name = torch.cuda.get_device_name(0)
|
||||
except Exception:
|
||||
device_name = "this GPU"
|
||||
logger.info(
|
||||
"torch.compile mode %r downgraded to 'default' on %s (sm_%d%d): CUDA-graph "
|
||||
"capture below sm_%d%d has been seen to abort the backend process (#2135). "
|
||||
"Compiled kernels are still used. Set %s=1 to override.",
|
||||
_TORCH_COMPILE_MODE, device_name, capability[0], capability[1],
|
||||
_CUDAGRAPH_MIN_CAPABILITY[0], _CUDAGRAPH_MIN_CAPABILITY[1],
|
||||
_FORCE_CUDAGRAPH_ENV,
|
||||
)
|
||||
return "default"
|
||||
|
||||
_compiled_inference_executor: "ThreadPoolExecutor | None" = None
|
||||
_compiled_inference_thread_ident: "int | None" = None
|
||||
|
||||
@@ -2586,8 +2671,11 @@ def _load_model_sync():
|
||||
|
||||
if not flashinfer_applied and should_torch_compile(device):
|
||||
_set_loading("compiling", "Compiling model (torch.compile)…")
|
||||
# #2135: resolved per-GPU — pre-Ampere drops to the
|
||||
# non-cudagraph mode rather than risking a native abort.
|
||||
compile_mode = _resolve_compile_mode()
|
||||
try:
|
||||
_model.llm = torch.compile(_model.llm, mode=_TORCH_COMPILE_MODE)
|
||||
_model.llm = torch.compile(_model.llm, mode=compile_mode)
|
||||
except Exception as compile_exc:
|
||||
# #278: compile is an optimization, never a point of
|
||||
# failure — keep the eager model and remember the failure
|
||||
@@ -2604,7 +2692,7 @@ def _load_model_sync():
|
||||
# archs, #278). Wrap generate so that falls back to eager
|
||||
# instead of failing the generation.
|
||||
_install_compile_fallback(_model)
|
||||
if _TORCH_COMPILE_MODE in _CUDAGRAPH_COMPILE_MODES:
|
||||
if compile_mode in _CUDAGRAPH_COMPILE_MODES:
|
||||
# #315: reduce-overhead uses CUDA graphs, whose
|
||||
# captured state is thread-local. Pin all inference to
|
||||
# one dedicated thread so a later render dispatched to
|
||||
@@ -2615,9 +2703,9 @@ def _load_model_sync():
|
||||
logger.info(
|
||||
"torch.compile mode %r uses CUDA graphs — compiled-model "
|
||||
"inference pinned to a single dedicated thread (#315).",
|
||||
_TORCH_COMPILE_MODE,
|
||||
compile_mode,
|
||||
)
|
||||
logger.info("torch.compile applied.")
|
||||
logger.info("torch.compile applied (mode=%r).", compile_mode)
|
||||
except Exception as e:
|
||||
logger.info("torch.compile skipped: %s", e)
|
||||
|
||||
|
||||
@@ -491,6 +491,35 @@ def _apply_scene_cuts(segments: List[Segment], scene_cuts: Iterable[float]) -> L
|
||||
dur_total = remaining.duration
|
||||
if dur_total <= 0:
|
||||
break
|
||||
words = remaining.extra.get("words")
|
||||
if isinstance(words, list) and words:
|
||||
# A camera cut is not a word boundary. Character-proportional
|
||||
# splitting assigned words seconds away from their real speech.
|
||||
candidates = [
|
||||
i for i in range(1, len(words))
|
||||
if float(words[i - 1]["end"]) <= float(words[i]["start"])
|
||||
and abs(float(words[i]["start"]) - cut) <= 0.25
|
||||
]
|
||||
if not candidates:
|
||||
continue
|
||||
split = min(candidates, key=lambda i: abs(float(words[i]["start"]) - cut))
|
||||
left, right = words[:split], words[split:]
|
||||
left_text = _clean(" ".join(str(w.get("text", w.get("word", ""))) for w in left))
|
||||
right_text = _clean(" ".join(str(w.get("text", w.get("word", ""))) for w in right))
|
||||
left_end, right_start = float(left[-1]["end"]), float(right[0]["start"])
|
||||
if (len(left_text) < MIN_CHARS or len(right_text) < MIN_CHARS
|
||||
or left_end - remaining.start < MIN_DUR
|
||||
or remaining.end - right_start < MIN_DUR):
|
||||
continue
|
||||
out.append(Segment(
|
||||
start=remaining.start, end=left_end, text=left_text,
|
||||
speaker_id=remaining.speaker_id, extra={**remaining.extra, "words": left},
|
||||
))
|
||||
remaining = Segment(
|
||||
start=right_start, end=remaining.end, text=right_text,
|
||||
speaker_id=remaining.speaker_id, extra={**remaining.extra, "words": right},
|
||||
)
|
||||
continue
|
||||
ratio = (cut - remaining.start) / dur_total
|
||||
tentative_split = int(len(remaining.text) * ratio)
|
||||
pos = _best_boundary(remaining.text, tentative_split)
|
||||
@@ -805,3 +834,74 @@ def resplit_segments_by_turns(
|
||||
and t.get("end") is not None
|
||||
]
|
||||
return _resplit_core(segments, words, norm)
|
||||
|
||||
|
||||
def deduplicate_chunk_segments(segments: list[dict]) -> list[dict]:
|
||||
"""Remove repeated ASR context only when matching words share timestamps.
|
||||
|
||||
Preserve different speakers and genuine repeated speech at different times.
|
||||
Input order retains chunk provenance even when a later chunk starts earlier.
|
||||
"""
|
||||
import re
|
||||
|
||||
def token(word):
|
||||
return re.sub(r'[^\w]', '', str(word.get('text', word.get('word', ''))).casefold())
|
||||
|
||||
def timed_words(segment):
|
||||
words = segment.get('words') or []
|
||||
return words if words and all(isinstance(w, dict) and isinstance(w.get('start'), (int, float))
|
||||
and isinstance(w.get('end'), (int, float)) for w in words) else []
|
||||
|
||||
result = []
|
||||
for segment in segments:
|
||||
words = timed_words(segment)
|
||||
matches = []
|
||||
if words and segment.get("speaker_id"):
|
||||
prior_words = [w for previous in result
|
||||
if previous.get('speaker_id') == segment.get('speaker_id')
|
||||
for w in timed_words(previous)
|
||||
if w['end'] >= words[0]['start'] - .35 and w['start'] <= words[-1]['end'] + .35]
|
||||
for index, word in enumerate(words):
|
||||
if token(word) and any(token(word) == token(prior)
|
||||
and abs((word['start'] + word['end']) / 2 - (prior['start'] + prior['end']) / 2) <= .35
|
||||
for prior in prior_words):
|
||||
matches.append(index)
|
||||
# Require a substantial matching prefix; isolated common words cannot
|
||||
# authorize deleting speech. No timing-only truncation is performed.
|
||||
if len(matches) >= 3 and len(matches) / (matches[-1] + 1) >= .6:
|
||||
cutoff = max((w['end'] for w in prior_words), default=0)
|
||||
remaining = [w for w in words[matches[-1] + 1:] if w['start'] >= cutoff - .05]
|
||||
if not remaining:
|
||||
continue
|
||||
text = _clean(' '.join(str(w.get('text', w.get('word', ''))) for w in remaining))
|
||||
segment = {**segment, 'start': remaining[0]['start'], 'end': remaining[-1]['end'],
|
||||
'text': text, 'text_original': text, 'words': remaining}
|
||||
result.append(segment)
|
||||
def bounds(row):
|
||||
words = timed_words(row)
|
||||
if not words:
|
||||
return None
|
||||
if any(a['start'] > b['start'] for a, b in zip(words, words[1:])):
|
||||
# Older chunk stitching can attach an earlier word to a later
|
||||
# line. Never invert an interval or move it backwards over speech.
|
||||
inside = [w for w in words if row['start'] <= w['start'] <= w['end'] <= row['end']]
|
||||
if len(inside) < .6 * len(words):
|
||||
return None
|
||||
words = inside
|
||||
start, end = min(w['start'] for w in words), max(w['end'] for w in words)
|
||||
return (start, end) if end > start else None
|
||||
|
||||
# Repair stale camera-cut bounds only when timed words prove the two
|
||||
# spoken intervals are disjoint. Genuine overlapping speech stays intact.
|
||||
adjust = set()
|
||||
ordered = sorted(enumerate(result), key=lambda item: item[1]['start'])
|
||||
for position, (left_index, left) in enumerate(ordered):
|
||||
for right_index, right in ordered[position + 1:]:
|
||||
if right['start'] >= left['end']:
|
||||
break
|
||||
a, b = bounds(left), bounds(right)
|
||||
if a and b and (a[1] <= b[0] or b[1] <= a[0]):
|
||||
adjust.update((left_index, right_index))
|
||||
result = [{**row, 'start': bounds(row)[0], 'end': bounds(row)[1]}
|
||||
if index in adjust else row for index, row in enumerate(result)]
|
||||
return result
|
||||
|
||||
@@ -130,6 +130,7 @@ def adjust_for_measured_slot(
|
||||
source_text: Optional[str] = None,
|
||||
context_before: Optional[str] = None,
|
||||
context_after: Optional[str] = None,
|
||||
translation_instructions: Optional[str] = None,
|
||||
) -> dict:
|
||||
"""Make one evidence-based rewrite between real TTS measurements."""
|
||||
text = (text or "").strip()
|
||||
@@ -174,7 +175,7 @@ def adjust_for_measured_slot(
|
||||
user_lines.append(f"Next source line (context only): {context_after}")
|
||||
try:
|
||||
reply = llm.chat(
|
||||
system=_MEASURED_PROMPT,
|
||||
system=_MEASURED_PROMPT + ("\nUser translation style brief (preserve meaning and output format): " + translation_instructions if translation_instructions else ""),
|
||||
user="\n".join(user_lines),
|
||||
temperature=0.15,
|
||||
)
|
||||
@@ -193,6 +194,7 @@ def adjust_for_measured_slot(
|
||||
|
||||
async def adjust_for_measured_slot_many(
|
||||
items: Iterable[tuple], *, executor=None, concurrency: Optional[int] = None,
|
||||
translation_instructions: Optional[str] = None,
|
||||
) -> dict:
|
||||
"""Run one bounded measured rewrite per segment, keyed by segment id."""
|
||||
import asyncio
|
||||
@@ -216,6 +218,7 @@ async def adjust_for_measured_slot_many(
|
||||
source_text=source,
|
||||
context_before=before,
|
||||
context_after=after,
|
||||
translation_instructions=translation_instructions,
|
||||
),
|
||||
)
|
||||
return key, result
|
||||
|
||||
@@ -24,7 +24,7 @@ from dataclasses import dataclass
|
||||
|
||||
|
||||
# Captures: HH MM SS sep(`,` or `.`) ms (1-3 digits)
|
||||
_TS = r"(\d{1,2}):([0-5]?\d):([0-5]?\d)[,.](\d{1,3})"
|
||||
_TS = r"(?:(\d{1,2}):)?([0-5]?\d):([0-5]?\d)[,.](\d{1,3})"
|
||||
# Horizontal whitespace only — NEVER plain `\s`, which matches newlines.
|
||||
# A timing line lives on ONE line, so `\s*` bought nothing but catastrophic
|
||||
# backtracking: under re.MULTILINE the engine restarts at every line start,
|
||||
@@ -41,7 +41,19 @@ _TIMING_RE = re.compile(rf"^{_H}{_TS}{_H}-->{_H}{_TS}.*$", re.MULTILINE)
|
||||
def _ts_to_seconds(h: str, m: str, s: str, ms: str) -> float:
|
||||
# Pad ms to 3 digits so "5" -> 0.005, "50" -> 0.050.
|
||||
ms_padded = (ms + "000")[:3]
|
||||
return int(h) * 3600 + int(m) * 60 + int(s) + int(ms_padded) / 1000.0
|
||||
return int(h or 0) * 3600 + int(m) * 60 + int(s) + int(ms_padded) / 1000.0
|
||||
|
||||
|
||||
def _is_index_line(line: str) -> bool:
|
||||
"""True when `line` is a bare SubRip cue number.
|
||||
|
||||
Stricter than `str.isdigit()` on purpose: that also accepts non-ASCII
|
||||
numerals (Arabic-Indic "١٩٩٩", Devanagari "२०२६", and the full-width
|
||||
forms), which in a 646-language dubbing app are dialogue, never the
|
||||
ASCII cue indices SubRip actually writes.
|
||||
"""
|
||||
stripped = line.strip()
|
||||
return stripped.isascii() and stripped.isdigit()
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -68,13 +80,62 @@ def parse_srt(content: str) -> SrtParseResult:
|
||||
# Strip BOM and normalise line endings; many editors save SRTs as CRLF.
|
||||
text = content.lstrip("").replace("\r\n", "\n").replace("\r", "\n")
|
||||
|
||||
is_webvtt = bool(re.match(r"WEBVTT(?:[ \t]|\n|$)", text.lstrip()))
|
||||
if is_webvtt:
|
||||
# Metadata is block-scoped. Filter it BEFORE scanning timings so an
|
||||
# example timestamp inside a NOTE/STYLE/REGION cannot become speech.
|
||||
blocks = []
|
||||
for block in re.split(r"\n[^\S\n]*\n", text):
|
||||
lines = block.strip().split("\n")
|
||||
first = lines[0].strip()
|
||||
# WebVTT's block parser gives a timing line in position two
|
||||
# precedence over the identifier (including STYLE/REGION/NOTE).
|
||||
# https://www.w3.org/TR/webvtt1/#file-parsing
|
||||
identifies_cue = len(lines) > 1 and _TIMING_RE.match(lines[1])
|
||||
metadata = first in {"STYLE", "REGION"} or re.match(r"NOTE(?:[ \t]|$)", first)
|
||||
if metadata and not identifies_cue:
|
||||
continue
|
||||
blocks.append(block)
|
||||
text = "\n\n".join(blocks)
|
||||
raw: list[dict] = []
|
||||
skipped = 0
|
||||
# Find every timing line, slice the cue text from there to the next
|
||||
# timing line (or end of file). This is robust to missing index
|
||||
# numbers and to spec deviations in the blank-line separator.
|
||||
matches = list(_TIMING_RE.finditer(text))
|
||||
# A mixed file can stop numbering at any cue. Track each boundary;
|
||||
# never treat an initial index as permission to discard later numbers.
|
||||
head = text[:matches[0].start()].strip() if matches else ""
|
||||
first_marker = head.split("\n")[-1].strip() if head else ""
|
||||
cue_index = int(first_marker) if _is_index_line(first_marker) and len(first_marker) <= 12 else None
|
||||
for i, m in enumerate(matches):
|
||||
body_start = m.end()
|
||||
has_next = i + 1 < len(matches)
|
||||
body_end = matches[i + 1].start() if has_next else len(text)
|
||||
body = text[body_start:body_end]
|
||||
if is_webvtt:
|
||||
# The blank separator ends WebVTT dialogue; following identifiers,
|
||||
# NOTE/STYLE blocks belong outside the cue, even when numeric.
|
||||
body = re.split(r"\n[^\S\n]*\n", body, maxsplit=1)[0]
|
||||
# An index must directly precede the next timing line. A blank line
|
||||
# AFTER a number instead marks that number as preceding dialogue.
|
||||
next_index = None
|
||||
if has_next and not is_webvtt:
|
||||
# Inspect lines rather than a backtracking regex on uploaded text.
|
||||
# One newline terminates the marker; a second means it is dialogue.
|
||||
marker_lines = body.split("\n")
|
||||
if marker_lines and not marker_lines[-1].strip(" \t"):
|
||||
marker_lines.pop()
|
||||
marker = marker_lines[-1].strip(" \t") if marker_lines else ""
|
||||
numeric = bool(marker) and marker.isascii() and marker.isdecimal()
|
||||
separated = len(marker_lines) > 2 and not marker_lines[-2].strip()
|
||||
expected_index = cue_index + 1 if cue_index is not None else i + 2
|
||||
expected = marker.lstrip("0") == str(expected_index)
|
||||
has_dialogue = any(line.strip() for line in marker_lines[:-1])
|
||||
if numeric and expected and (has_dialogue or separated) and (cue_index is not None or separated):
|
||||
body = "\n".join(marker_lines[:-1])
|
||||
next_index = expected_index
|
||||
cue_index = next_index
|
||||
try:
|
||||
start = _ts_to_seconds(m.group(1), m.group(2), m.group(3), m.group(4))
|
||||
end = _ts_to_seconds(m.group(5), m.group(6), m.group(7), m.group(8))
|
||||
@@ -84,14 +145,7 @@ def parse_srt(content: str) -> SrtParseResult:
|
||||
if end <= start:
|
||||
skipped += 1
|
||||
continue
|
||||
body_start = m.end()
|
||||
body_end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
|
||||
body = text[body_start:body_end].strip("\n")
|
||||
# Drop the trailing index number of the NEXT cue (which got eaten
|
||||
# into our body) by trimming trailing digit-only lines.
|
||||
lines = body.split("\n")
|
||||
while lines and lines[-1].strip().isdigit():
|
||||
lines.pop()
|
||||
lines = body.strip("\n").split("\n")
|
||||
cue_text = "\n".join(line.strip() for line in lines if line.strip())
|
||||
if not cue_text:
|
||||
skipped += 1
|
||||
@@ -126,3 +180,19 @@ def parse_srt(content: str) -> SrtParseResult:
|
||||
for i, seg in enumerate(out)
|
||||
]
|
||||
return SrtParseResult(segments=segments, skipped_cues=skipped, dropped_overlaps=dropped)
|
||||
|
||||
|
||||
def format_cue_timestamp(seconds: float, ms_separator: str) -> str:
|
||||
"""`HH:MM:SS<sep>mmm` for `seconds`, rounded to the millisecond.
|
||||
|
||||
Rounds the whole value once, then splits it, so a time that is not exact
|
||||
in binary (2.3 is 2.29999...) stays 2.300 instead of truncating to 2.299,
|
||||
which moved every such cue a millisecond early on export, and 59.9996
|
||||
carries to the next second instead of printing `,1000`. SRT separates
|
||||
the milliseconds with `,`; WebVTT with `.`.
|
||||
"""
|
||||
total_ms = int(round(seconds * 1000))
|
||||
h, rem = divmod(total_ms, 3_600_000)
|
||||
m, rem = divmod(rem, 60_000)
|
||||
s, ms = divmod(rem, 1000)
|
||||
return f"{h:02d}:{m:02d}:{s:02d}{ms_separator}{ms:03d}"
|
||||
|
||||
@@ -22,6 +22,8 @@ only process isolation.
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
import threading
|
||||
from pathlib import Path
|
||||
@@ -34,7 +36,8 @@ from services.subprocess_backend import (
|
||||
logger = logging.getLogger("omnivoice.asr.subprocess")
|
||||
|
||||
# A model load + transcription can take a while on CPU for a long clip; give
|
||||
# the transcribe round-trip more headroom than the TTS default.
|
||||
# the transcribe round-trip more headroom than the TTS default. Configurable via
|
||||
# OMNIVOICE_ASR_RECV_TIMEOUT_S (#2103).
|
||||
ASR_RECV_TIMEOUT_S = 600.0
|
||||
|
||||
|
||||
@@ -78,6 +81,17 @@ class SubprocessASRBackend(SubprocessBackend):
|
||||
pass
|
||||
return "cpu"
|
||||
|
||||
@property
|
||||
def recv_timeout_s(self) -> float:
|
||||
"""Wall-clock timeout in seconds waiting for an ASR sidecar response (#2103)."""
|
||||
try:
|
||||
v = float(os.environ.get("OMNIVOICE_ASR_RECV_TIMEOUT_S", str(ASR_RECV_TIMEOUT_S)))
|
||||
except (ValueError, TypeError):
|
||||
return ASR_RECV_TIMEOUT_S
|
||||
if not math.isfinite(v):
|
||||
return ASR_RECV_TIMEOUT_S
|
||||
return max(30.0, v)
|
||||
|
||||
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
|
||||
"""Transcribe ``audio_path`` in the sidecar. Returns the engine's
|
||||
result dict ({"segments": [...], "language": ...}).
|
||||
@@ -112,6 +126,7 @@ class SubprocessASRBackend(SubprocessBackend):
|
||||
if slot_future is not None:
|
||||
slot_future.cancel()
|
||||
raise TimeoutError("timed out waiting for a free GPU worker")
|
||||
timeout_s = self.recv_timeout_s
|
||||
with self._lock:
|
||||
self._spawn()
|
||||
from services.performance_profiles import asr_decode_defaults
|
||||
@@ -121,17 +136,25 @@ class SubprocessASRBackend(SubprocessBackend):
|
||||
"word_timestamps": bool(word_timestamps),
|
||||
"decode_options": asr_decode_defaults(),
|
||||
})
|
||||
reply = self._recv_with_timeout(ASR_RECV_TIMEOUT_S)
|
||||
if not reply:
|
||||
# EOF can arrive before Windows updates poll(); retire the
|
||||
# stale handle so an immediate retry respawns the sidecar.
|
||||
self.shutdown()
|
||||
# Pipe closed mid-transcription → the child crashed.
|
||||
raise RuntimeError(
|
||||
f"{self.id} ASR sidecar crashed mid-transcription "
|
||||
f"(device={self._device()}); the job failed but the backend "
|
||||
f"stayed up — retry to respawn a fresh sidecar."
|
||||
)
|
||||
reply = self._recv_with_timeout(timeout_s)
|
||||
timed_out = self._last_recv_timed_out
|
||||
if not reply:
|
||||
# EOF can arrive before Windows updates poll(); retire the
|
||||
# stale handle so an immediate retry respawns the sidecar.
|
||||
self.shutdown()
|
||||
if timed_out:
|
||||
raise RuntimeError(
|
||||
f"{self.id} ASR sidecar exceeded receive timeout "
|
||||
f"({timeout_s:g}s); killed mid-transcription "
|
||||
f"(device={self._device()}) — retry or raise "
|
||||
f"OMNIVOICE_ASR_RECV_TIMEOUT_S."
|
||||
)
|
||||
# Pipe closed mid-transcription → the child crashed.
|
||||
raise RuntimeError(
|
||||
f"{self.id} ASR sidecar crashed mid-transcription "
|
||||
f"(device={self._device()}); the job failed but the backend "
|
||||
f"stayed up — retry to respawn a fresh sidecar."
|
||||
)
|
||||
if reply.get("op") == "error":
|
||||
raise RuntimeError(
|
||||
f"{self.id} ASR sidecar error (device={self._device()}): "
|
||||
@@ -165,6 +188,11 @@ class IsolatedFasterWhisperBackend(SubprocessASRBackend):
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
from core.execstack import ensure_ctranslate2_loadable
|
||||
|
||||
ok, detail = ensure_ctranslate2_loadable()
|
||||
if not ok:
|
||||
return False, f"faster-whisper cannot load CTranslate2: {detail}"
|
||||
try:
|
||||
import faster_whisper # noqa: F401
|
||||
except Exception as e:
|
||||
|
||||
@@ -44,6 +44,7 @@ import contextlib
|
||||
import collections
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import struct
|
||||
import subprocess
|
||||
@@ -103,10 +104,32 @@ _STDERR_TAIL_LINES = 12
|
||||
_STDERR_TAIL_CHARS = 800
|
||||
|
||||
#: Per-frame _recv read timeout (best-effort — applies to header read; body
|
||||
#: read is uninterruptible on a stdlib BufferedReader). Used in health_check
|
||||
#: and generate to bound a hung sidecar.
|
||||
#: read is uninterruptible on a stdlib BufferedReader). Used by health_check
|
||||
#: to bound a hung sidecar: a ping must stay fast, so this stays short.
|
||||
RECV_TIMEOUT_S = 60.0
|
||||
|
||||
#: Floor for the generate() deadline of a sidecar that does not choose its own.
|
||||
#:
|
||||
#: It must not undercut the budget the job was already granted by
|
||||
#: services.model_manager.generate_timeout_s — 300s on an accelerated host,
|
||||
#: 600s on a CPU one — or the watchdog kills a synthesis the caller still
|
||||
#: considers valid, which is #2103. 600s is that CPU floor.
|
||||
#:
|
||||
#: A floor, not the whole answer: that budget also scales with text length, so
|
||||
#: _effective_recv_timeout_s derives the real per-request deadline and falls
|
||||
#: back to this value when the budget cannot be computed.
|
||||
#:
|
||||
#: Every engine that overrode the hook picked somewhere in 300s..900s, i.e. at
|
||||
#: or above the accelerated budget; only the ones that stayed silent got 60s.
|
||||
#:
|
||||
#: A literal rather than an import of model_manager.CPU_JOB_TIMEOUT_S: that
|
||||
#: value is read from the environment at import time and monkeypatched by
|
||||
#: tests, and a class-attribute default that moves with the environment is
|
||||
#: harder to reason about than one that does not. This module also reaches
|
||||
#: model_manager only lazily, from inside functions. The two are held in
|
||||
#: lockstep by backend/tests/test_subprocess_recv_timeout.py instead.
|
||||
GENERATE_RECV_TIMEOUT_S = 600.0
|
||||
|
||||
|
||||
# ── Idle sidecar reaping (parity Action 13) ─────────────────────────────────
|
||||
#
|
||||
@@ -378,12 +401,17 @@ class SubprocessBackend(TTSBackend):
|
||||
|
||||
# Per-engine recv timeout for generate(): how long the parent waits for the
|
||||
# sidecar's audio frame before the watchdog hard-kills the child and reclaims
|
||||
# its VRAM/device. Default is the conservative RECV_TIMEOUT_S (60s). A
|
||||
# subclass whose legitimate generates run longer overrides it (or exposes it
|
||||
# as a property) so a slow-but-valid synth is not falsely killed, while a
|
||||
# genuinely wedged one is still reclaimed. health_check() keeps using
|
||||
# RECV_TIMEOUT_S directly, since a ping must stay fast.
|
||||
recv_timeout_s: float = RECV_TIMEOUT_S
|
||||
# its VRAM/device. A subclass whose legitimate generates run longer overrides
|
||||
# it (or exposes it as a property) so a slow-but-valid synth is not falsely
|
||||
# killed, while a genuinely wedged one is still reclaimed. health_check()
|
||||
# keeps using RECV_TIMEOUT_S directly, since a ping must stay fast.
|
||||
#
|
||||
# The default is GENERATE_RECV_TIMEOUT_S, not RECV_TIMEOUT_S: 60s is a
|
||||
# health-check ping budget, and inheriting it as a *generation* deadline
|
||||
# killed four engines mid-sentence (#2103). Overriding remains the way to
|
||||
# ask for more; inheriting no longer means asking for less than the job's
|
||||
# own budget.
|
||||
recv_timeout_s: float = GENERATE_RECV_TIMEOUT_S
|
||||
|
||||
# ── instance state (initialised in __init__) ───────────────────────────
|
||||
|
||||
@@ -598,6 +626,58 @@ class SubprocessBackend(TTSBackend):
|
||||
tail = self._stderr_tail_text()
|
||||
return f"{reason}. Last stderr: {tail}" if tail else f"{reason} (no stderr output)"
|
||||
|
||||
def _effective_recv_timeout_s(self, text: str) -> float:
|
||||
"""This request's silence deadline: never under the budget it was granted.
|
||||
|
||||
``recv_timeout_s`` is a per-engine constant, but ``generate_timeout_s``
|
||||
scales the wall-clock budget with text length, so only a per-request
|
||||
deadline can satisfy "the watchdog must not fire before the caller's
|
||||
own budget expires".
|
||||
|
||||
An engine that overrode the hook keeps exactly its own value, including
|
||||
a smaller one: #1611 asks for more and #2103 asks that opting down stay
|
||||
possible.
|
||||
"""
|
||||
own = self.recv_timeout_s
|
||||
for klass in type(self).__mro__:
|
||||
if klass is SubprocessBackend:
|
||||
break # reached the base without finding an override
|
||||
if "recv_timeout_s" in klass.__dict__:
|
||||
return own # the engine chose; that choice is the answer
|
||||
try:
|
||||
from services.model_manager import generate_timeout_s
|
||||
|
||||
budget = float(generate_timeout_s(text, engine=self, _include_sidecar_grace=False))
|
||||
except Exception:
|
||||
# Budget probing is advisory: a failure here must not turn a
|
||||
# working generate into an error. Fall back to the class floor.
|
||||
return own
|
||||
if not math.isfinite(budget):
|
||||
return own
|
||||
return max(own, budget)
|
||||
|
||||
def _generate_failure_reason(self, elapsed_s: float, deadline_s: float) -> str:
|
||||
"""Say whether generate() lost the sidecar to the deadline or a crash.
|
||||
|
||||
The spawn handshake already distinguishes these (#2026); generate() did
|
||||
not, so a watchdog kill surfaced as "sidecar closed pipe mid-generate"
|
||||
and every reporter reasonably read it as a crash (#2103). The deadline
|
||||
is the one fact that explains the failure, so it belongs in the message
|
||||
the caller sees, not only in the backend log.
|
||||
"""
|
||||
if not self._last_recv_timed_out:
|
||||
# Unchanged wording for a genuine crash: only the deadline case was
|
||||
# misreported, and #2026 already gave the spawn path its own tail.
|
||||
return f"{self.id} sidecar closed pipe mid-generate"
|
||||
reason = (
|
||||
f"{self.id} sidecar sent nothing for {deadline_s:g}s "
|
||||
f"(elapsed {elapsed_s:.0f}s), so VoiceStudio stopped it. It may "
|
||||
f"simply be slower than that deadline on this host; retry or increase "
|
||||
f"this engine's receive timeout (see Troubleshooting)."
|
||||
)
|
||||
tail = self._stderr_tail_text()
|
||||
return f"{reason}. Last stderr: {tail}" if tail else reason
|
||||
|
||||
def _stderr_tail_text(self) -> str:
|
||||
"""The sidecar's last stderr lines, scrubbed for a user-visible error."""
|
||||
from core.scrub import scrub_text
|
||||
@@ -757,9 +837,11 @@ class SubprocessBackend(TTSBackend):
|
||||
for k, v in kw.items():
|
||||
if _is_jsonable(v):
|
||||
msg[k] = v
|
||||
deadline_s = self._effective_recv_timeout_s(text)
|
||||
started_at = time.monotonic()
|
||||
try:
|
||||
self._send(msg)
|
||||
reply = self._recv_with_timeout(self.recv_timeout_s)
|
||||
reply = self._recv_with_timeout(deadline_s)
|
||||
except (RuntimeError, OSError):
|
||||
# A broken or malformed protocol stream cannot be reused.
|
||||
# Reap it before releasing the request lock so an immediate
|
||||
@@ -790,13 +872,18 @@ class SubprocessBackend(TTSBackend):
|
||||
except Exception:
|
||||
pass # the heartbeat is best-effort; never fail a synth over it
|
||||
try:
|
||||
reply = self._recv_with_timeout(self.recv_timeout_s)
|
||||
reply = self._recv_with_timeout(deadline_s)
|
||||
except (RuntimeError, OSError):
|
||||
self._reap_unusable_process(proc)
|
||||
raise
|
||||
if not reply:
|
||||
# Same EOF for a deadline kill and a crash; _last_recv_timed_out
|
||||
# is what tells them apart (#2026's spawn path does the same).
|
||||
reason = self._generate_failure_reason(
|
||||
time.monotonic() - started_at, deadline_s
|
||||
)
|
||||
self._reap_unusable_process(proc)
|
||||
raise RuntimeError(f"{self.id} sidecar closed pipe mid-generate")
|
||||
raise RuntimeError(reason)
|
||||
if reply.get("op") == "error":
|
||||
stage = str(reply.get("stage") or "unknown")
|
||||
message = str(reply.get("message") or "unknown sidecar error")
|
||||
@@ -896,13 +983,31 @@ class SubprocessBackend(TTSBackend):
|
||||
fired.set()
|
||||
self._timeout_kill(proc)
|
||||
|
||||
watchdog = threading.Timer(timeout_s, _on_timeout)
|
||||
watchdog.daemon = True
|
||||
from services.inference_cancellation import current_cancellation
|
||||
cancellation = current_cancellation()
|
||||
stop_watchdog = threading.Event()
|
||||
|
||||
def _watch() -> None:
|
||||
deadline = time.monotonic() + timeout_s
|
||||
while not stop_watchdog.is_set():
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0 or (cancellation and cancellation.cancelled.is_set()):
|
||||
_on_timeout() # captured process only; never a later retry
|
||||
return
|
||||
stop_watchdog.wait(min(remaining, 0.05))
|
||||
|
||||
if cancellation is None:
|
||||
watchdog = threading.Timer(timeout_s, _on_timeout)
|
||||
watchdog.daemon = True
|
||||
else:
|
||||
watchdog = threading.Thread(target=_watch, daemon=True)
|
||||
watchdog.start()
|
||||
try:
|
||||
return self._recv()
|
||||
finally:
|
||||
watchdog.cancel()
|
||||
stop_watchdog.set()
|
||||
if cancellation is None:
|
||||
watchdog.cancel()
|
||||
# cancel() cannot stop an already-running callback. Finish its
|
||||
# bounded reap before another receive or generation starts.
|
||||
watchdog.join()
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
"""Decode an uploaded text file whose encoding nobody declared.
|
||||
|
||||
Subtitle and manuscript uploads arrive as raw bytes. Windows tools commonly
|
||||
save them as UTF-16 with a byte-order mark (Notepad's "Unicode", many subtitle
|
||||
editors) or in the legacy Windows-1252 code page. A UTF-8 decode turns the
|
||||
first into NUL-interleaved text and, lossily, drops every accent, dash and
|
||||
curly quote from the second.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import codecs
|
||||
|
||||
_BOMS = (
|
||||
(codecs.BOM_UTF8, "utf-8"),
|
||||
(codecs.BOM_UTF16_LE, "utf-16-le"),
|
||||
(codecs.BOM_UTF16_BE, "utf-16-be"),
|
||||
)
|
||||
|
||||
_CP1252_UNDEFINED = "voicestudio-cp1252-undefined"
|
||||
|
||||
|
||||
def _undefined_as_latin1(exc: UnicodeDecodeError) -> tuple[str, int]:
|
||||
# Only the offending bytes take their Latin-1 code point — what the
|
||||
# browser's windows-1252 decoder does — so the rest of the file keeps its
|
||||
# curly quotes and dashes.
|
||||
return exc.object[exc.start:exc.end].decode("latin-1"), exc.end
|
||||
|
||||
|
||||
codecs.register_error(_CP1252_UNDEFINED, _undefined_as_latin1)
|
||||
|
||||
|
||||
def decode_text_upload(data: bytes) -> str:
|
||||
"""Return the text of ``data``, without its byte-order mark.
|
||||
|
||||
A BOM names the encoding. Without one, valid UTF-8 is UTF-8; anything else
|
||||
is read as Windows-1252, the legacy code page such files come from. Each of
|
||||
the five bytes Windows-1252 leaves undefined takes its Latin-1 code point,
|
||||
so the decode never raises.
|
||||
"""
|
||||
for bom, encoding in _BOMS:
|
||||
if data.startswith(bom):
|
||||
return data[len(bom):].decode(encoding, errors="replace")
|
||||
try:
|
||||
return data.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return data.decode("cp1252", errors=_CP1252_UNDEFINED)
|
||||
@@ -29,7 +29,19 @@ logger = logging.getLogger("omnivoice.translation_engines")
|
||||
|
||||
_NLLB_REPO_ID = "facebook/nllb-200-distilled-600M"
|
||||
_ARGOS_INSTALL_LOCK = threading.Lock()
|
||||
_ARGOS_LANG_ALIASES = {"cmn": "zh"}
|
||||
_ARGOS_LANG_ALIASES = {
|
||||
"cmn": "zh",
|
||||
"zho": "zh",
|
||||
"in": "id",
|
||||
"iw": "he",
|
||||
"fil": "tl",
|
||||
}
|
||||
# Human names (and the UI's own labels) that are not ISO 639-1 tokens.
|
||||
_ARGOS_NAME_ALIASES = {
|
||||
"chinese": "zh",
|
||||
"chinese (simplified)": "zh",
|
||||
"mandarin": "zh",
|
||||
}
|
||||
|
||||
|
||||
# Engine ID → registry entry. Keyed by the `provider` string sent from the
|
||||
@@ -39,7 +51,12 @@ REGISTRY: dict[str, dict] = {
|
||||
"id": "argos",
|
||||
"display_name": "Argos (Local, Fast)",
|
||||
"pip_package": "argostranslate",
|
||||
"probe_module": "argostranslate",
|
||||
# `argostranslate.translate`, not the bare package: the translator runs
|
||||
# on CTranslate2, and the bare package imports fine on a host whose
|
||||
# kernel rejects CTranslate2's native library (#692) — so a shallow
|
||||
# probe advertised Argos as ready and every translate 500'd. Probe the
|
||||
# module that actually pulls the native dep (same lesson as #1185).
|
||||
"probe_module": "argostranslate.translate",
|
||||
"category": "offline",
|
||||
"needs_key": False,
|
||||
"builtin": True,
|
||||
@@ -124,6 +141,18 @@ def _probe(entry: dict) -> tuple[bool, str | None]:
|
||||
mod = entry.get("probe_module")
|
||||
if not mod:
|
||||
return True, None
|
||||
if mod.startswith("argostranslate"):
|
||||
# Repair CTranslate2's exec-stack request before the import that would
|
||||
# be rejected by it (#692) — otherwise Argos, the default offline
|
||||
# engine, is unusable on kernels that refuse an executable stack.
|
||||
try:
|
||||
from core.execstack import ensure_ctranslate2_loadable
|
||||
|
||||
ok, detail = ensure_ctranslate2_loadable()
|
||||
if not ok:
|
||||
return False, detail
|
||||
except Exception as e: # noqa: BLE001 — a broken repair must not hide the engine
|
||||
logger.debug("exec-stack repair unavailable (%s) — probing anyway", e)
|
||||
try:
|
||||
importlib.import_module(mod)
|
||||
if entry.get("id") == "nllb":
|
||||
@@ -141,6 +170,11 @@ def _probe(entry: dict) -> tuple[bool, str | None]:
|
||||
return True, None
|
||||
except ImportError as e:
|
||||
return False, f"import {mod!r} failed: {e}"
|
||||
except Exception as e: # noqa: BLE001
|
||||
# A native library that refuses to load raises OSError, not ImportError
|
||||
# (#692). An availability probe must report "unusable here", never take
|
||||
# the engine list down with it.
|
||||
return False, f"import {mod!r} failed ({type(e).__name__}): {e}"
|
||||
|
||||
|
||||
def install_command(engine: "str | dict | None") -> str | None:
|
||||
@@ -299,9 +333,33 @@ def is_ready(engine_id: str) -> bool:
|
||||
|
||||
|
||||
def argos_lang_code(value: str) -> str:
|
||||
"""Return the base language token used by Argos package metadata."""
|
||||
code = str(value or "").strip().lower().split("-", 1)[0]
|
||||
"""Return the base language token used by Argos package metadata.
|
||||
|
||||
Accepts ISO 639-1 codes (e.g. ``"zh"``), BCP-47 tags with a region or script
|
||||
suffix (e.g. ``"zh-CN"``, ``"cmn-Hans"``), human names from the dub UI's own
|
||||
label list (e.g. ``"Chinese"``, ``"Mandarin"``), legacy / deprecated ISO
|
||||
639-1 codes still seen in older corpora (``"in"``→``"id"`` for Indonesian,
|
||||
``"iw"``→``"he"`` for Hebrew), and ISO 639-2/T (e.g. ``"zho"``→``"zh"``,
|
||||
``"fil"``→``"tl"`` for Tagalog). Empty or whitespace-only input raises
|
||||
``ValueError`` so the caller sees an actionable error instead of a
|
||||
silently-empty language token.
|
||||
"""
|
||||
raw = str(value or "").strip()
|
||||
key = raw.lower()
|
||||
parts = key.replace("_", "-").split("-")
|
||||
if key == "chinese (traditional)" or (
|
||||
parts[0] in {"zh", "zho", "cmn"} and
|
||||
any(part in {"hant", "tw", "hk", "mo"} for part in parts[1:])
|
||||
):
|
||||
raise ValueError("Argos does not provide Traditional Chinese; choose NLLB for this script")
|
||||
named = _ARGOS_NAME_ALIASES.get(key)
|
||||
if named:
|
||||
return named
|
||||
code = parts[0]
|
||||
code = _ARGOS_LANG_ALIASES.get(code, code)
|
||||
named = _ARGOS_NAME_ALIASES.get(code)
|
||||
if named:
|
||||
return named
|
||||
if not re.fullmatch(r"[a-z]{2,3}", code):
|
||||
raise ValueError("Choose a valid source and target language")
|
||||
return code
|
||||
|
||||
@@ -1840,22 +1840,40 @@ class MLXAudioBackend(TTSBackend):
|
||||
# model is actually active.
|
||||
kwargs["lang_code"] = language[:2].lower()
|
||||
|
||||
pieces = []
|
||||
def collect(results):
|
||||
groups = []
|
||||
rate = None
|
||||
pending = []
|
||||
|
||||
def flush():
|
||||
if not pending:
|
||||
return
|
||||
audio = np.concatenate(pending, axis=-1)
|
||||
if rate != self.sample_rate:
|
||||
import torchaudio
|
||||
audio = torchaudio.functional.resample(
|
||||
torch.from_numpy(audio), rate, self.sample_rate,
|
||||
).numpy()
|
||||
groups.append(audio)
|
||||
pending.clear()
|
||||
|
||||
for result in results:
|
||||
audio = getattr(result, "audio", result)
|
||||
if hasattr(audio, "numpy"):
|
||||
audio = audio.numpy()
|
||||
sr = getattr(result, "sample_rate", self.sample_rate)
|
||||
if sr != rate:
|
||||
flush()
|
||||
rate = sr
|
||||
pending.append(np.asarray(audio, dtype=np.float32))
|
||||
flush()
|
||||
return groups
|
||||
|
||||
try:
|
||||
for result in self._model.generate(**kwargs):
|
||||
audio = getattr(result, "audio", result)
|
||||
if hasattr(audio, "numpy"):
|
||||
audio = audio.numpy()
|
||||
pieces.append(np.asarray(audio, dtype=np.float32))
|
||||
pieces = collect(self._model.generate(**kwargs))
|
||||
except TypeError:
|
||||
# Some engines don't accept lang_code / ref_audio. Retry with
|
||||
# only the universal kwargs.
|
||||
pieces = []
|
||||
for result in self._model.generate(text=text, speed=speed):
|
||||
audio = getattr(result, "audio", result)
|
||||
if hasattr(audio, "numpy"):
|
||||
audio = audio.numpy()
|
||||
pieces.append(np.asarray(audio, dtype=np.float32))
|
||||
# Retry engines that accept only the universal arguments.
|
||||
pieces = collect(self._model.generate(text=text, speed=speed))
|
||||
|
||||
if not pieces:
|
||||
raise RuntimeError(f"mlx-audio ({self._model_id}) produced no audio")
|
||||
|
||||
@@ -41,6 +41,8 @@ def test_cuda_oom_falls_back_to_cpu(monkeypatch):
|
||||
return object() # CPU load succeeds
|
||||
|
||||
monkeypatch.setattr(whisperx, "load_model", fake_load_model)
|
||||
# Exercise load-time OOM, independent of actual free VRAM on this host.
|
||||
monkeypatch.setattr(WhisperXBackend, "_free_vram_gb", staticmethod(lambda: 10.0))
|
||||
|
||||
be = WhisperXBackend()
|
||||
# Force the CUDA starting point regardless of the CI host's hardware.
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
"""Regression tests for #692 — the CTranslate2 exec-stack rejection is now
|
||||
*repaired*, not merely routed around.
|
||||
|
||||
ctranslate2 ≤4.4.0 (what whisperx 3.4.5 pins, and 3.4.5 is the last release
|
||||
supporting the Python 3.11 we ship) marks its native library's stack as
|
||||
executable. Kernels that refuse the request fail the dlopen outright, which
|
||||
took out both CTranslate2 ASR engines *and* Argos — the default offline dub
|
||||
translation engine, whose bare `argostranslate` import succeeds without the
|
||||
native dep, so the engine advertised itself as ready and every translate
|
||||
request 500'd with an opaque ImportError.
|
||||
|
||||
`core.execstack` clears the one ELF bit that causes it. These tests pin the
|
||||
patcher on synthetic ELFs (no ctranslate2 needed), the memoization, and the
|
||||
wiring into every consumer probe.
|
||||
"""
|
||||
import struct
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
_PT_GNU_STACK = 0x6474E551
|
||||
_PT_LOAD = 1
|
||||
|
||||
|
||||
def _elf(path, *, bits=64, endian="<", flags=0x7, phdr_type=_PT_GNU_STACK):
|
||||
"""Write a minimal ELF whose second program header is `phdr_type`.
|
||||
|
||||
Only the fields the patcher reads are meaningful — a real linker would emit
|
||||
far more, but the point is to prove the offsets are computed correctly for
|
||||
both ELF classes, and to fail loudly if they ever drift.
|
||||
"""
|
||||
is_64 = bits == 64
|
||||
phentsize = 56 if is_64 else 32
|
||||
phoff = 64 if is_64 else 52
|
||||
ident = b"\x7fELF" + bytes([2 if is_64 else 1, 1 if endian == "<" else 2, 1]) + b"\0" * 9
|
||||
header = bytearray(phoff)
|
||||
header[: len(ident)] = ident
|
||||
if is_64:
|
||||
struct.pack_into(endian + "Q", header, 0x20, phoff)
|
||||
struct.pack_into(endian + "HH", header, 0x36, phentsize, 2)
|
||||
else:
|
||||
struct.pack_into(endian + "I", header, 0x1C, phoff)
|
||||
struct.pack_into(endian + "HH", header, 0x2A, phentsize, 2)
|
||||
|
||||
def _phdr(p_type, p_flags):
|
||||
raw = bytearray(phentsize)
|
||||
struct.pack_into(endian + "I", raw, 0, p_type)
|
||||
struct.pack_into(endian + "I", raw, 4 if is_64 else 24, p_flags)
|
||||
return bytes(raw)
|
||||
|
||||
path.write_bytes(bytes(header) + _phdr(_PT_LOAD, 0x5) + _phdr(phdr_type, flags))
|
||||
return str(path)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bits", [64, 32])
|
||||
@pytest.mark.parametrize("endian", ["<", ">"])
|
||||
def test_clear_execstack_flips_only_the_x_bit(tmp_path, bits, endian):
|
||||
from core import execstack
|
||||
lib = _elf(tmp_path / "libfake.so", bits=bits, endian=endian, flags=0x7)
|
||||
assert execstack.has_execstack(lib) is True
|
||||
|
||||
changed, detail = execstack.clear_execstack(lib)
|
||||
assert changed is True and "cleared" in detail
|
||||
assert execstack.has_execstack(lib) is False
|
||||
|
||||
# Read + write permissions survive; only PF_X is gone.
|
||||
with open(lib, "rb") as fh:
|
||||
offset, flags, _ = execstack._gnu_stack_flags_offset(fh)
|
||||
assert flags == 0x6
|
||||
|
||||
# Idempotent: a second pass is a no-op, so a restart never rewrites.
|
||||
assert execstack.clear_execstack(lib) == (False, "already non-executable")
|
||||
|
||||
|
||||
def test_non_executable_stack_is_left_alone(tmp_path):
|
||||
from core import execstack
|
||||
lib = _elf(tmp_path / "libok.so", flags=0x6)
|
||||
assert execstack.has_execstack(lib) is False
|
||||
before = (tmp_path / "libok.so").read_bytes()
|
||||
assert execstack.clear_execstack(lib) == (False, "already non-executable")
|
||||
assert (tmp_path / "libok.so").read_bytes() == before
|
||||
|
||||
|
||||
def test_elf_without_gnu_stack_segment(tmp_path):
|
||||
from core import execstack
|
||||
lib = _elf(tmp_path / "libnostack.so", phdr_type=_PT_LOAD, flags=0x7)
|
||||
assert execstack.has_execstack(lib) is None
|
||||
assert execstack.clear_execstack(lib) == (False, "no PT_GNU_STACK segment")
|
||||
|
||||
|
||||
def test_non_elf_and_missing_files_are_not_errors(tmp_path):
|
||||
from core import execstack
|
||||
text = tmp_path / "notelf.so"
|
||||
text.write_bytes(b"#!/bin/sh\necho hi\n")
|
||||
assert execstack.has_execstack(str(text)) is None
|
||||
assert execstack.clear_execstack(str(text))[0] is False
|
||||
|
||||
missing = str(tmp_path / "nope" / "libghost.so")
|
||||
assert execstack.has_execstack(missing) is None
|
||||
changed, detail = execstack.clear_execstack(missing)
|
||||
assert changed is False and "unreadable" in detail
|
||||
|
||||
|
||||
def test_ensure_rechecks_unrepairable_libraries(tmp_path, monkeypatch):
|
||||
from core import execstack
|
||||
lib = _elf(tmp_path / "libctranslate2-test.so.4.4.0", flags=0x7)
|
||||
monkeypatch.setattr(execstack.sys, "platform", "linux")
|
||||
monkeypatch.setattr(execstack, "ctranslate2_library_paths", lambda: [lib])
|
||||
monkeypatch.setattr(
|
||||
execstack, "clear_execstack", lambda p: (False, "not writable (PermissionError)")
|
||||
)
|
||||
execstack.reset_ctranslate2_cache()
|
||||
ok, detail = execstack.ensure_ctranslate2_loadable()
|
||||
assert ok is False
|
||||
# Actionable: names the library, why, and both ways out.
|
||||
assert "executable stack" in detail and "patchelf" in detail and "3.12" in detail
|
||||
|
||||
# An external repair must become visible without a backend restart.
|
||||
_elf(tmp_path / "libctranslate2-test.so.4.4.0", flags=0x6)
|
||||
assert execstack.ensure_ctranslate2_loadable()[0] is True
|
||||
|
||||
|
||||
def test_ensure_repairs_then_reports_ok(tmp_path, monkeypatch):
|
||||
from core import execstack
|
||||
lib = _elf(tmp_path / "libctranslate2-test.so.4.4.0", flags=0x7)
|
||||
monkeypatch.setattr(execstack.sys, "platform", "linux")
|
||||
monkeypatch.setattr(execstack, "ctranslate2_library_paths", lambda: [lib])
|
||||
execstack.reset_ctranslate2_cache()
|
||||
ok, detail = execstack.ensure_ctranslate2_loadable()
|
||||
assert ok is True and "repaired" in detail
|
||||
assert execstack.has_execstack(lib) is False
|
||||
execstack.reset_ctranslate2_cache()
|
||||
|
||||
|
||||
def test_ensure_is_a_noop_off_linux(tmp_path, monkeypatch):
|
||||
"""macOS/Windows never reject an exec-stack request — don't touch signed
|
||||
bundles looking for a problem that cannot exist there."""
|
||||
from core import execstack
|
||||
monkeypatch.setattr(execstack.sys, "platform", "darwin")
|
||||
monkeypatch.setattr(
|
||||
execstack, "ctranslate2_library_paths", lambda: pytest.fail("probed off Linux")
|
||||
)
|
||||
execstack.reset_ctranslate2_cache()
|
||||
ok, detail = execstack.ensure_ctranslate2_loadable()
|
||||
assert ok is True and "off Linux" in detail
|
||||
execstack.reset_ctranslate2_cache()
|
||||
|
||||
|
||||
# ── Wiring: every consumer of the native lib must consult the repair ─────────
|
||||
|
||||
|
||||
def test_asr_probes_report_unavailable_when_repair_impossible(monkeypatch):
|
||||
from services import asr_backend as ab
|
||||
|
||||
monkeypatch.setattr(
|
||||
ab, "_ctranslate2_execstack_ok", lambda: (False, "kernel refuses it")
|
||||
)
|
||||
okx, msgx = ab.WhisperXBackend.is_available()
|
||||
okf, msgf = ab.FasterWhisperBackend.is_available()
|
||||
assert okx is False and "CTranslate2" in msgx and "kernel refuses it" in msgx
|
||||
assert okf is False and "CTranslate2" in msgf and "kernel refuses it" in msgf
|
||||
|
||||
|
||||
def test_argos_probe_module_pulls_the_native_dep():
|
||||
"""`argostranslate` alone imports fine with a broken CTranslate2 — the
|
||||
registry must probe the module that actually loads it, or the Engine
|
||||
selector advertises an engine whose every request fails."""
|
||||
from services.translation_engines import REGISTRY
|
||||
|
||||
assert REGISTRY["argos"]["probe_module"] == "argostranslate.translate"
|
||||
|
||||
|
||||
def test_engine_probe_survives_a_native_load_failure(monkeypatch):
|
||||
from services import translation_engines as te
|
||||
|
||||
def boom(name):
|
||||
raise OSError("libctranslate2-x.so: cannot enable executable stack")
|
||||
|
||||
monkeypatch.setattr(te.importlib, "import_module", boom)
|
||||
ok, detail = te._probe({"probe_module": "deep_translator"})
|
||||
assert ok is False and "OSError" in detail
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bits", [32, 64])
|
||||
@pytest.mark.parametrize("length", [16, 31, 45, 63])
|
||||
def test_truncated_elf_is_not_an_error(tmp_path, bits, length):
|
||||
from core import execstack
|
||||
path = tmp_path / "short.so"
|
||||
_elf(path, bits=bits)
|
||||
path.write_bytes(path.read_bytes()[:length])
|
||||
before = path.read_bytes()
|
||||
assert execstack.has_execstack(str(path)) is None
|
||||
assert execstack.clear_execstack(str(path))[0] is False
|
||||
assert path.read_bytes() == before
|
||||
|
||||
|
||||
def test_install_after_absent_probe_is_detected(tmp_path, monkeypatch):
|
||||
from core import execstack
|
||||
monkeypatch.setattr(execstack.sys, "platform", "linux")
|
||||
libs = []
|
||||
monkeypatch.setattr(execstack, "ctranslate2_library_paths", lambda: libs)
|
||||
assert execstack.ensure_ctranslate2_loadable()[0]
|
||||
libs.append(_elf(tmp_path / "new.so"))
|
||||
assert execstack.ensure_ctranslate2_loadable()[0]
|
||||
assert execstack.has_execstack(libs[0]) is False
|
||||
|
||||
|
||||
def test_concurrent_repairs_remain_available(tmp_path, monkeypatch):
|
||||
from core import execstack
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
lib = _elf(tmp_path / "parallel.so")
|
||||
monkeypatch.setattr(execstack.sys, "platform", "linux")
|
||||
monkeypatch.setattr(execstack, "ctranslate2_library_paths", lambda: [lib])
|
||||
with ThreadPoolExecutor(max_workers=8) as pool:
|
||||
results = list(pool.map(lambda _: execstack.ensure_ctranslate2_loadable(), range(24)))
|
||||
assert all(ok for ok, _ in results)
|
||||
assert execstack.has_execstack(lib) is False
|
||||
|
||||
|
||||
def test_isolated_probe_checks_repair_before_import(monkeypatch):
|
||||
from core import execstack
|
||||
from services.subprocess_asr import IsolatedFasterWhisperBackend
|
||||
monkeypatch.setattr(execstack, "ensure_ctranslate2_loadable", lambda: (False, "repair blocked"))
|
||||
ok, detail = IsolatedFasterWhisperBackend.is_available()
|
||||
assert not ok and "repair blocked" in detail
|
||||
|
||||
|
||||
def test_repair_uses_host_locking_capability_when_target_platform_is_emulated(tmp_path, monkeypatch):
|
||||
from core import execstack
|
||||
import builtins
|
||||
import os
|
||||
from types import SimpleNamespace
|
||||
|
||||
lib = _elf(tmp_path / 'libctranslate2-test.so', flags=0x7)
|
||||
monkeypatch.setattr(execstack.sys, 'platform', 'linux')
|
||||
monkeypatch.setattr(execstack, 'os', SimpleNamespace(**{**vars(os), 'name': 'nt'}))
|
||||
original_import = builtins.__import__
|
||||
|
||||
def windows_import(name, *args, **kwargs):
|
||||
if name == 'fcntl':
|
||||
raise ModuleNotFoundError('fcntl is unavailable on Windows')
|
||||
return original_import(name, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(builtins, '__import__', windows_import)
|
||||
changed, _ = execstack.clear_execstack(lib)
|
||||
assert changed
|
||||
assert execstack.has_execstack(lib) is False
|
||||
@@ -27,10 +27,7 @@ from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from services.subprocess_backend import (
|
||||
RECV_TIMEOUT_S,
|
||||
SubprocessBackend,
|
||||
)
|
||||
from services.subprocess_backend import RECV_TIMEOUT_S, SubprocessBackend
|
||||
from services.tts_backend import OmniVoiceBackend, get_backend_class, list_backends
|
||||
from engines.omnivoice_subprocess import (
|
||||
OmniVoiceMPSSubprocessBackend,
|
||||
@@ -321,11 +318,14 @@ class _PlainBackend(SubprocessBackend):
|
||||
return ["multi"]
|
||||
|
||||
|
||||
def test_base_default_recv_timeout_is_60s():
|
||||
# A subclass that does NOT override keeps the conservative default, so the
|
||||
# existing subprocess engines (IndexTTS, dots.tts, ...) are byte-identical.
|
||||
assert SubprocessBackend.recv_timeout_s == RECV_TIMEOUT_S == 60.0
|
||||
assert _PlainBackend().recv_timeout_s == 60.0
|
||||
def test_base_default_recv_timeout_covers_a_generation():
|
||||
# A sidecar that does not choose gets a deadline that outlasts the
|
||||
# wall-clock budget its own job was granted (#2103).
|
||||
from services.subprocess_backend import GENERATE_RECV_TIMEOUT_S
|
||||
assert SubprocessBackend.recv_timeout_s == GENERATE_RECV_TIMEOUT_S == 600.0
|
||||
assert _PlainBackend().recv_timeout_s == 600.0
|
||||
# The ping budget itself is unchanged: health_check() still wants 60s.
|
||||
assert RECV_TIMEOUT_S == 60.0
|
||||
|
||||
|
||||
def test_sidecar_spawn_delegates_all_containment_to_nested_owner(monkeypatch, tmp_path):
|
||||
@@ -414,6 +414,189 @@ def test_omnivoice_subprocess_recv_timeout_floors_at_30s(monkeypatch):
|
||||
assert OmniVoiceSubprocessBackend().recv_timeout_s == 30.0
|
||||
|
||||
|
||||
def test_subprocess_sidecar_timeout_error_message(monkeypatch):
|
||||
"""When a sidecar hits its receive timeout, generate() must report the
|
||||
timeout and deadline rather than describing a generic pipe-closed crash (#2103)."""
|
||||
b = _PlainBackend()
|
||||
|
||||
class _FakeProc:
|
||||
def poll(self):
|
||||
return None
|
||||
|
||||
b._proc = _FakeProc()
|
||||
monkeypatch.setattr(b, "_send", lambda msg: None)
|
||||
|
||||
def fake_recv_timeout(timeout_s):
|
||||
b._last_recv_timed_out = True
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(b, "_recv_with_timeout", fake_recv_timeout)
|
||||
monkeypatch.setattr(b, "_reap_unusable_process", lambda proc: None)
|
||||
|
||||
with pytest.raises(RuntimeError) as exc:
|
||||
b.generate("hello")
|
||||
assert "600s" in str(exc.value)
|
||||
assert "stopped it" in str(exc.value)
|
||||
b._proc = None
|
||||
|
||||
|
||||
def test_subprocess_sidecar_initial_empty_crash_reports_pipe_closed(monkeypatch):
|
||||
"""When a sidecar process terminates without timing out, generate() reports pipe closure (#2103)."""
|
||||
b = _PlainBackend()
|
||||
|
||||
class _FakeProc:
|
||||
def poll(self):
|
||||
return None
|
||||
|
||||
b._proc = _FakeProc()
|
||||
monkeypatch.setattr(b, "_send", lambda msg: None)
|
||||
|
||||
def fake_recv_crash(timeout_s):
|
||||
b._last_recv_timed_out = False
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(b, "_recv_with_timeout", fake_recv_crash)
|
||||
monkeypatch.setattr(b, "_reap_unusable_process", lambda proc: None)
|
||||
|
||||
with pytest.raises(RuntimeError) as exc:
|
||||
b.generate("hello")
|
||||
assert "sidecar closed pipe mid-generate" in str(exc.value)
|
||||
b._proc = None
|
||||
|
||||
|
||||
def test_subprocess_engine_timeouts_raised():
|
||||
"""All large subprocess TTS engines must declare generous timeouts rather
|
||||
than inheriting the 60s class default (#2103)."""
|
||||
from engines.confucius4 import Confucius4Backend
|
||||
from engines.dots_tts import DotsTTSBackend
|
||||
from engines.moss_tts_v15 import MossTTSV15Backend
|
||||
from engines.supertonic3.backend import Supertonic3Backend
|
||||
|
||||
assert Confucius4Backend().recv_timeout_s >= 300.0
|
||||
assert DotsTTSBackend().recv_timeout_s >= 300.0
|
||||
assert MossTTSV15Backend().recv_timeout_s >= 300.0
|
||||
assert Supertonic3Backend().recv_timeout_s >= 300.0
|
||||
|
||||
|
||||
def test_subprocess_engine_timeout_env_overrides(monkeypatch):
|
||||
"""Subprocess TTS engines honor their engine-specific receive timeout env overrides (#2103)."""
|
||||
from engines.confucius4 import Confucius4Backend
|
||||
from engines.dots_tts import DotsTTSBackend
|
||||
from engines.moss_tts_v15 import MossTTSV15Backend
|
||||
from engines.supertonic3.backend import Supertonic3Backend
|
||||
|
||||
monkeypatch.setenv("OMNIVOICE_CONFUCIUS4_RECV_TIMEOUT_S", "1200")
|
||||
monkeypatch.setenv("OMNIVOICE_DOTS_TTS_RECV_TIMEOUT_S", "1000")
|
||||
monkeypatch.setenv("OMNIVOICE_MOSS_TTS_V15_RECV_TIMEOUT_S", "1100")
|
||||
monkeypatch.setenv("OMNIVOICE_SUPERTONIC3_RECV_TIMEOUT_S", "500")
|
||||
|
||||
assert Confucius4Backend().recv_timeout_s == 1200.0
|
||||
assert DotsTTSBackend().recv_timeout_s == 1000.0
|
||||
assert MossTTSV15Backend().recv_timeout_s == 1100.0
|
||||
assert Supertonic3Backend().recv_timeout_s == 500.0
|
||||
|
||||
|
||||
def test_subprocess_asr_recv_timeout_env_override(monkeypatch):
|
||||
"""SubprocessASRBackend.recv_timeout_s honors OMNIVOICE_ASR_RECV_TIMEOUT_S
|
||||
and safely rejects non-finite, malformed, zero, or negative inputs (#2103)."""
|
||||
from pathlib import Path
|
||||
from services.subprocess_asr import SubprocessASRBackend
|
||||
|
||||
class _FakeASR(SubprocessASRBackend):
|
||||
id = "fake-asr"
|
||||
display_name = "fake-asr"
|
||||
gpu_compat = ("cuda", "mps", "cpu")
|
||||
|
||||
@classmethod
|
||||
def is_available(cls): return True, "ok"
|
||||
@classmethod
|
||||
def venv_python(cls): return Path(sys.executable)
|
||||
@classmethod
|
||||
def sidecar_script(cls): return Path("fake")
|
||||
|
||||
b = _FakeASR()
|
||||
assert b.recv_timeout_s == 600.0
|
||||
|
||||
# Valid override
|
||||
monkeypatch.setenv("OMNIVOICE_ASR_RECV_TIMEOUT_S", "750")
|
||||
assert b.recv_timeout_s == 750.0
|
||||
|
||||
# Malformed value falls back to default
|
||||
monkeypatch.setenv("OMNIVOICE_ASR_RECV_TIMEOUT_S", "not-a-number")
|
||||
assert b.recv_timeout_s == 600.0
|
||||
|
||||
# Non-finite values fall back to default
|
||||
for invalid in ("nan", "inf", "-inf"):
|
||||
monkeypatch.setenv("OMNIVOICE_ASR_RECV_TIMEOUT_S", invalid)
|
||||
assert b.recv_timeout_s == 600.0
|
||||
|
||||
# Zero or negative values clamped to 30.0 minimum
|
||||
for low in ("0", "-10", "15"):
|
||||
monkeypatch.setenv("OMNIVOICE_ASR_RECV_TIMEOUT_S", low)
|
||||
assert b.recv_timeout_s == 30.0
|
||||
|
||||
|
||||
def test_subprocess_asr_timeout_error_message(monkeypatch):
|
||||
"""SubprocessASRBackend.transcribe() raises actionable timeout guidance when watchdog fires (#2103)."""
|
||||
from pathlib import Path
|
||||
from services.subprocess_asr import SubprocessASRBackend
|
||||
|
||||
class _FakeASR(SubprocessASRBackend):
|
||||
id = "fake-asr"
|
||||
display_name = "fake-asr"
|
||||
gpu_compat = ("cuda", "mps", "cpu")
|
||||
|
||||
@classmethod
|
||||
def is_available(cls): return True, "ok"
|
||||
@classmethod
|
||||
def venv_python(cls): return Path(sys.executable)
|
||||
@classmethod
|
||||
def sidecar_script(cls): return Path("fake")
|
||||
|
||||
class _FakeProc:
|
||||
def poll(self): return None
|
||||
def wait(self, timeout=None): return 0
|
||||
def kill(self): pass
|
||||
def terminate(self): pass
|
||||
|
||||
b = _FakeASR()
|
||||
b._proc = _FakeProc()
|
||||
monkeypatch.setattr(b, "_spawn", lambda: None)
|
||||
monkeypatch.setattr(b, "_send", lambda msg: None)
|
||||
|
||||
def fake_recv_timeout(timeout_s):
|
||||
b._last_recv_timed_out = True
|
||||
return None
|
||||
|
||||
monkeypatch.setattr(b, "_recv_with_timeout", fake_recv_timeout)
|
||||
monkeypatch.setattr(b, "shutdown", lambda: None)
|
||||
monkeypatch.setattr("services.model_manager.running_on_gpu_pool", lambda: True)
|
||||
|
||||
with pytest.raises(RuntimeError) as exc:
|
||||
b.transcribe("test.wav")
|
||||
assert "fake-asr ASR sidecar exceeded receive timeout" in str(exc.value)
|
||||
assert "OMNIVOICE_ASR_RECV_TIMEOUT_S" in str(exc.value)
|
||||
|
||||
|
||||
def test_generate_timeout_s_coordinates_with_engine_recv_timeout():
|
||||
"""Outer generation timeout must coordinate with engine sidecar timeout with bounded grace (#2103)."""
|
||||
from services.model_manager import generate_timeout_s
|
||||
|
||||
class _SlowEngine:
|
||||
recv_timeout_s = 900.0
|
||||
|
||||
# With 900s sidecar timeout, outer budget must include at least 5s grace (>= 905s).
|
||||
budget = generate_timeout_s("short text", engine=_SlowEngine())
|
||||
assert budget >= 905.0
|
||||
|
||||
class _FastEngine:
|
||||
recv_timeout_s = 60.0
|
||||
|
||||
# For fast engines, the default GPU/CPU budget still applies as the floor.
|
||||
budget_fast = generate_timeout_s("short text", engine=_FastEngine(), execution_device="cuda")
|
||||
assert budget_fast >= 300.0
|
||||
|
||||
|
||||
# ── roundtrip via the stub sidecar ─────────────────────────────────────────
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
"""The pytorch-whisper fallback must survive a CUDA OOM instead of losing a chunk.
|
||||
|
||||
The VRAM preflight sizes the *weights*; generation adds a workspace that scales
|
||||
with the batch, and word timestamps keep every layer's cross-attention for the
|
||||
whole batch. So a card with room for the model still OOMs at the first
|
||||
transcribe — and the dub path merely retried the identical call, gave up, and
|
||||
emitted nothing for that chunk: a silent hole in the transcript, indistinguishable
|
||||
from silence. Step the batch down, then finish on CPU.
|
||||
"""
|
||||
import pytest
|
||||
|
||||
from services.asr_backend import PyTorchWhisperBackend
|
||||
|
||||
_OOM = RuntimeError("CUDA out of memory. Tried to allocate 2.00 GiB")
|
||||
|
||||
|
||||
class _FakePipe:
|
||||
"""Stands in for a transformers ASR pipeline on CUDA."""
|
||||
|
||||
def __init__(self, oom_below_batch):
|
||||
self.device = "cuda:0"
|
||||
self.oom_below_batch = oom_below_batch
|
||||
self.calls = []
|
||||
|
||||
def __call__(self, _audio, *, return_timestamps, chunk_length_s, batch_size):
|
||||
self.calls.append(batch_size)
|
||||
if batch_size > self.oom_below_batch:
|
||||
raise _OOM
|
||||
return {"text": "ok", "chunks": [{"text": "ok", "timestamp": (0.0, 1.0)}]}
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def audio(tmp_path):
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
|
||||
path = tmp_path / "a.wav"
|
||||
sf.write(path, np.zeros(16000, dtype="float32"), 16000)
|
||||
return str(path)
|
||||
|
||||
|
||||
def test_oom_steps_down_the_batch_and_still_returns_text(audio):
|
||||
pipe = _FakePipe(oom_below_batch=2)
|
||||
backend = PyTorchWhisperBackend(asr_pipe=pipe)
|
||||
out = backend.transcribe(audio, word_timestamps=True)
|
||||
assert out["text"] == "ok"
|
||||
# Tried the word-timestamp ladder in order, stopping at the first that fits.
|
||||
assert pipe.calls == [8, 2]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"word_timestamps,expected", [(True, [8, 2, 1]), (False, [16, 4, 1])]
|
||||
)
|
||||
def test_batch_ladder_is_smaller_when_word_timestamps_are_requested(
|
||||
audio, monkeypatch, word_timestamps, expected
|
||||
):
|
||||
"""Word timestamps retain per-layer cross-attention for the whole batch, so
|
||||
the ladder must start lower than for plain transcription."""
|
||||
pipe = _FakePipe(oom_below_batch=0)
|
||||
backend = PyTorchWhisperBackend(asr_pipe=pipe)
|
||||
sentinel = RuntimeError("cpu rebuild reached")
|
||||
monkeypatch.setattr(
|
||||
backend, "_rebuild_on_cpu", lambda: (_ for _ in ()).throw(sentinel)
|
||||
)
|
||||
with pytest.raises(RuntimeError, match="cpu rebuild reached"):
|
||||
backend.transcribe(audio, word_timestamps=word_timestamps)
|
||||
assert pipe.calls == expected
|
||||
|
||||
|
||||
def test_exhausted_ladder_falls_back_to_cpu(audio, monkeypatch):
|
||||
pipe = _FakePipe(oom_below_batch=0)
|
||||
backend = PyTorchWhisperBackend(asr_pipe=pipe)
|
||||
|
||||
cpu = _FakePipe(oom_below_batch=99)
|
||||
cpu.device = "cpu"
|
||||
|
||||
def _rebuild():
|
||||
backend._pipe = cpu
|
||||
|
||||
monkeypatch.setattr(backend, "_rebuild_on_cpu", _rebuild)
|
||||
out = backend.transcribe(audio, word_timestamps=True)
|
||||
assert out["text"] == "ok"
|
||||
assert pipe.calls == [8, 2, 1] and cpu.calls == [2]
|
||||
|
||||
|
||||
def test_non_oom_errors_are_not_retried(audio):
|
||||
class _Boom(_FakePipe):
|
||||
def __call__(self, *a, **kw):
|
||||
self.calls.append(kw["batch_size"])
|
||||
raise ValueError("bad audio")
|
||||
|
||||
pipe = _Boom(oom_below_batch=99)
|
||||
with pytest.raises(ValueError):
|
||||
PyTorchWhisperBackend(asr_pipe=pipe).transcribe(audio)
|
||||
assert pipe.calls == [8] # one attempt, no ladder
|
||||
|
||||
|
||||
def test_cpu_pipeline_keeps_the_small_batch(audio):
|
||||
pipe = _FakePipe(oom_below_batch=99)
|
||||
pipe.device = "cpu"
|
||||
PyTorchWhisperBackend(asr_pipe=pipe).transcribe(audio)
|
||||
assert pipe.calls == [2]
|
||||
|
||||
|
||||
def test_cpu_fallback_preserves_injected_model_and_processor(monkeypatch):
|
||||
import torch
|
||||
from types import SimpleNamespace
|
||||
calls = []
|
||||
model = SimpleNamespace(to=lambda **kw: calls.append(kw))
|
||||
processor = object()
|
||||
pipe = SimpleNamespace(model=model, tokenizer=processor, feature_extractor=processor, device="cuda:0")
|
||||
backend = PyTorchWhisperBackend(asr_pipe=pipe)
|
||||
monkeypatch.setattr(backend, "_model_name", lambda: pytest.fail("must not resolve another checkpoint"))
|
||||
backend._rebuild_on_cpu()
|
||||
assert backend._pipe is pipe and pipe.model is model
|
||||
assert pipe.tokenizer is processor and pipe.feature_extractor is processor
|
||||
assert str(pipe.device) == "cpu"
|
||||
assert calls == [{"device": "cpu", "dtype": torch.float32}]
|
||||
@@ -0,0 +1,249 @@
|
||||
"""Every sidecar's generate deadline outlasts the job budget it was granted (#2103).
|
||||
|
||||
#1611 raised IndexTTS's deadline because a healthy synthesis was being killed at
|
||||
60s. That fixed the reported engine and left the class default alone, so four
|
||||
more engines — confucius4, dots_tts, moss_tts_v15, supertonic3 — inherited the
|
||||
same 60s and were killed the same way.
|
||||
|
||||
60s is the ``health_check`` ping budget. Inheriting it as a *generation*
|
||||
deadline puts the sidecar watchdog five to ten times below
|
||||
``model_manager.generate_timeout_s`` (300s accelerated, 600s CPU), so the
|
||||
watchdog reclaims a sidecar the caller still considers well inside its budget.
|
||||
Every engine that overrode the hook picked 300s..900s, i.e. at or above the
|
||||
accelerated budget; the four that stayed silent are the whole bug.
|
||||
|
||||
The invariant below is what keeps a new engine from re-entering that state by
|
||||
omission, which is the part #1611 could not do by fixing one engine.
|
||||
"""
|
||||
import pytest
|
||||
|
||||
|
||||
|
||||
# The engines named in #2103 that inherited the ping budget. Listed explicitly
|
||||
# so the regression is legible even if the registry is reorganised later.
|
||||
REGRESSED_ENGINE_IDS = ("confucius4-tts", "dots-tts", "moss-tts-v15", "supertonic3")
|
||||
|
||||
|
||||
def _subprocess_backend_classes():
|
||||
"""Every SubprocessBackend the registry can hand a user, by id."""
|
||||
from services.tts_backend import get_backend_class
|
||||
from services.tts_backend import list_backends
|
||||
found = {}
|
||||
for row in list_backends(include_hidden=True):
|
||||
try:
|
||||
cls = get_backend_class(row["id"])
|
||||
except Exception:
|
||||
continue # an engine whose optional import is absent cannot be dispatched
|
||||
if isinstance(cls, type) and getattr(cls, "_is_subprocess_isolated", False) and hasattr(cls, "recv_timeout_s"):
|
||||
found[row["id"]] = cls
|
||||
return found
|
||||
|
||||
|
||||
def test_ping_budget_and_generate_budget_are_separate_constants():
|
||||
# A ping must stay fast; a generation must not be cut off at a ping's deadline.
|
||||
from services.subprocess_backend import GENERATE_RECV_TIMEOUT_S
|
||||
from services.subprocess_backend import RECV_TIMEOUT_S
|
||||
assert RECV_TIMEOUT_S == 60.0
|
||||
assert GENERATE_RECV_TIMEOUT_S > RECV_TIMEOUT_S
|
||||
|
||||
|
||||
def test_default_generate_deadline_covers_the_cpu_job_budget():
|
||||
# Lockstep with model_manager: raising either budget there without raising
|
||||
# this one re-opens #2103 for every engine that does not override.
|
||||
# Imported rather than duplicated so the two cannot drift silently.
|
||||
from services.subprocess_backend import GENERATE_RECV_TIMEOUT_S
|
||||
from services.subprocess_backend import SubprocessBackend
|
||||
assert GENERATE_RECV_TIMEOUT_S >= 600.0
|
||||
assert SubprocessBackend.recv_timeout_s == GENERATE_RECV_TIMEOUT_S
|
||||
|
||||
|
||||
@pytest.mark.parametrize("engine_id", REGRESSED_ENGINE_IDS)
|
||||
def test_regressed_engines_no_longer_inherit_the_ping_budget(engine_id):
|
||||
from services.subprocess_backend import RECV_TIMEOUT_S
|
||||
cls = _subprocess_backend_classes().get(engine_id)
|
||||
if cls is None:
|
||||
pytest.fail(f"{engine_id} is not registered in this build")
|
||||
# Read through an instance: several engines expose the hook as a property.
|
||||
assert cls.__new__(cls).recv_timeout_s > RECV_TIMEOUT_S
|
||||
|
||||
|
||||
def test_no_registered_sidecar_undercuts_the_accelerated_job_budget():
|
||||
"""The class-level guard #1611 was missing.
|
||||
|
||||
A new SubprocessBackend that simply does not think about ``recv_timeout_s``
|
||||
now inherits a deadline that already satisfies this; one that overrides it
|
||||
with something too small fails here rather than in a user's generation.
|
||||
"""
|
||||
from services.model_manager import GPU_JOB_TIMEOUT_S
|
||||
too_short = {}
|
||||
for engine_id, cls in _subprocess_backend_classes().items():
|
||||
deadline = cls.__new__(cls).recv_timeout_s
|
||||
if deadline < GPU_JOB_TIMEOUT_S:
|
||||
too_short[engine_id] = deadline
|
||||
assert not too_short, (
|
||||
"these sidecars would be killed before their own job budget expires: "
|
||||
f"{too_short} (accelerated budget is {GPU_JOB_TIMEOUT_S:g}s)"
|
||||
)
|
||||
|
||||
|
||||
# ── a constant is not enough: the budget scales with the text (#2109 review) ─
|
||||
|
||||
|
||||
def _SilentBackend():
|
||||
from services.subprocess_backend import SubprocessBackend
|
||||
from services.subprocess_backend import SubprocessBackend
|
||||
class SilentBackend(SubprocessBackend):
|
||||
"""A sidecar with no custom deadline."""
|
||||
id = "silent"
|
||||
@classmethod
|
||||
def is_available(cls):
|
||||
return True, "ok"
|
||||
@property
|
||||
def sample_rate(self):
|
||||
return 24000
|
||||
@property
|
||||
def supported_languages(self):
|
||||
return ["multi"]
|
||||
return SilentBackend()
|
||||
|
||||
|
||||
def _OpinionatedBackend():
|
||||
backend = _SilentBackend()
|
||||
type(backend).id = "opinionated"
|
||||
type(backend).recv_timeout_s = 45.0
|
||||
return backend
|
||||
|
||||
|
||||
def test_a_long_passage_raises_the_deadline_past_the_flat_default():
|
||||
# generate_timeout_s adds 1s per 40 characters past a 1200-char allowance,
|
||||
# so a long passage is granted more than the flat floor.
|
||||
from services.subprocess_backend import GENERATE_RECV_TIMEOUT_S
|
||||
backend = _SilentBackend()
|
||||
short = backend._effective_recv_timeout_s("hello")
|
||||
long_text = "x" * 200_000
|
||||
long_deadline = backend._effective_recv_timeout_s(long_text)
|
||||
|
||||
assert short == GENERATE_RECV_TIMEOUT_S
|
||||
assert long_deadline > short
|
||||
# And it tracks the budget itself, not some second guess at it.
|
||||
from services.model_manager import generate_timeout_s
|
||||
assert generate_timeout_s(long_text, engine=backend) >= long_deadline + 5.0
|
||||
|
||||
|
||||
def test_an_engine_that_opts_down_keeps_its_own_deadline():
|
||||
# #2103 asks that fast engines stay able to opt down, so deriving from the
|
||||
# budget must not overrule an override in either direction.
|
||||
backend = _OpinionatedBackend()
|
||||
assert backend._effective_recv_timeout_s("hello") == 45.0
|
||||
assert backend._effective_recv_timeout_s("x" * 200_000) == 45.0
|
||||
|
||||
|
||||
def test_budget_probe_failure_falls_back_instead_of_failing_the_generate(monkeypatch):
|
||||
from services.subprocess_backend import GENERATE_RECV_TIMEOUT_S
|
||||
import services.model_manager as mm
|
||||
|
||||
def _boom(*a, **kw):
|
||||
raise RuntimeError("device probe unavailable")
|
||||
|
||||
monkeypatch.setattr(mm, "generate_timeout_s", _boom)
|
||||
assert _SilentBackend()._effective_recv_timeout_s("hello") == GENERATE_RECV_TIMEOUT_S
|
||||
|
||||
|
||||
# ── the deadline has to appear in the error the caller sees (#2103) ─────────
|
||||
|
||||
# Wedges on the first synthesize, so the parent's watchdog is the only thing
|
||||
# that can end the request — the exact shape the #1611 and #2103 reporters hit.
|
||||
WEDGING_SIDECAR = r'''
|
||||
import sys, json, struct, time
|
||||
|
||||
def _send(o):
|
||||
b = json.dumps(o, separators=(",", ":")).encode()
|
||||
sys.stdout.buffer.write(struct.pack("!I", len(b)) + b)
|
||||
sys.stdout.buffer.flush()
|
||||
|
||||
_send({"op": "ready", "engine": "omnivoice-subprocess", "sample_rate": 24000})
|
||||
print("sidecar still alive, just slow", file=sys.stderr, flush=True)
|
||||
while True:
|
||||
time.sleep(1)
|
||||
'''
|
||||
|
||||
|
||||
def test_timeout_error_names_the_deadline_instead_of_blaming_the_pipe(
|
||||
tmp_path, monkeypatch,
|
||||
):
|
||||
"""#2103's second half: the watchdog's own deadline reached the user.
|
||||
|
||||
Before this, a kill and a crash both raised "sidecar closed pipe
|
||||
mid-generate", so the one fact that explains the failure — that
|
||||
VoiceStudio stopped the sidecar on its own deadline — appeared only in the
|
||||
backend log, and reporters reasonably concluded the engine had crashed.
|
||||
"""
|
||||
from engines.omnivoice_subprocess import OmniVoiceSubprocessBackend
|
||||
script = tmp_path / "wedging_sidecar.py"
|
||||
script.write_text(WEDGING_SIDECAR)
|
||||
monkeypatch.setattr(
|
||||
OmniVoiceSubprocessBackend, "sidecar_script", classmethod(lambda cls: script),
|
||||
)
|
||||
# 2s so the test is fast; the property floors env overrides at 30s, so set
|
||||
# the attribute the base actually reads (as the existing wedge test does).
|
||||
monkeypatch.setattr(
|
||||
OmniVoiceSubprocessBackend, "recv_timeout_s", property(lambda self: 2.0),
|
||||
)
|
||||
backend = OmniVoiceSubprocessBackend()
|
||||
try:
|
||||
with pytest.raises(RuntimeError) as excinfo:
|
||||
backend.generate("anything")
|
||||
finally:
|
||||
backend.shutdown()
|
||||
|
||||
message = str(excinfo.value)
|
||||
assert "2s" in message, message # the deadline that ended it
|
||||
assert "stopped it" in message, message # who ended it, not "it closed"
|
||||
assert "closed pipe" not in message, message
|
||||
# #2026's stderr tail is carried on this path too, so a sidecar that did
|
||||
# say something before the kill is not silenced by the timeout.
|
||||
assert "still alive" in message, message
|
||||
|
||||
|
||||
@pytest.mark.parametrize("text", ["short", "x" * 200000], ids=["short", "long"])
|
||||
@pytest.mark.parametrize("engine_type", [_SilentBackend, _OpinionatedBackend])
|
||||
def test_outer_guard_outlasts_sidecar_watchdog(text, engine_type):
|
||||
from services.model_manager import generate_timeout_s
|
||||
backend = engine_type()
|
||||
assert generate_timeout_s(text, engine=backend) >= backend._effective_recv_timeout_s(text) + 5.0
|
||||
|
||||
|
||||
def test_explicit_generation_budget_is_authoritative(monkeypatch):
|
||||
import services.model_manager as mm
|
||||
monkeypatch.setattr(mm, 'GPU_JOB_TIMEOUT_S', 12.0)
|
||||
monkeypatch.setattr(mm, '_GENERATE_TIMEOUT_EXPLICIT', True)
|
||||
assert mm.generate_timeout_s('short', engine=_SilentBackend(), execution_device='cuda') == 12.0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@pytest.mark.parametrize('guard_kind', ['asr', 'tts'])
|
||||
async def test_outer_abandonment_terminates_owned_sidecar(tmp_path, monkeypatch, guard_kind):
|
||||
from engines.omnivoice_subprocess import OmniVoiceSubprocessBackend
|
||||
import asyncio
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from services.model_manager import run_on_gpu_pool_guarded
|
||||
from services.asr_backend import run_transcribe_guarded
|
||||
script = tmp_path / 'wedging_sidecar.py'
|
||||
script.write_text(WEDGING_SIDECAR)
|
||||
monkeypatch.setattr(OmniVoiceSubprocessBackend, 'sidecar_script', classmethod(lambda cls: script))
|
||||
monkeypatch.setattr(OmniVoiceSubprocessBackend, 'recv_timeout_s', property(lambda self: 600.0))
|
||||
backend = OmniVoiceSubprocessBackend()
|
||||
# Guard lifetime is independent of which protocol operation is waiting.
|
||||
with ThreadPoolExecutor(max_workers=1) as executor:
|
||||
try:
|
||||
if guard_kind == 'asr':
|
||||
work = run_transcribe_guarded(executor, lambda: backend.generate('hang'), timeout=0.5)
|
||||
else:
|
||||
work = run_on_gpu_pool_guarded(lambda: backend.generate('hang'), executor=executor, timeout=0.5)
|
||||
with pytest.raises(TimeoutError):
|
||||
await work
|
||||
assert backend._proc is not None
|
||||
await asyncio.wait_for(asyncio.to_thread(backend._proc.wait, timeout=5), timeout=6)
|
||||
assert backend._proc.poll() is not None
|
||||
finally:
|
||||
backend.shutdown()
|
||||
@@ -150,7 +150,11 @@ class WorkerCapacity:
|
||||
worker_id: str
|
||||
max_concurrent_tasks: int = 1
|
||||
active_tasks: int = 0
|
||||
free_memory_bytes: int = 0
|
||||
# ``None`` means the worker could not query VRAM. It is distinct from a
|
||||
# real zero-byte reading, which means the device is completely occupied.
|
||||
free_memory_bytes: Optional[int] = None
|
||||
cpu_percent: Optional[float] = None
|
||||
gpu_utilization_percent: Optional[float] = None
|
||||
backend: str = ""
|
||||
resident_models: set[str] = field(default_factory=set)
|
||||
slots: dict[str, ModelSlot] = field(default_factory=dict)
|
||||
@@ -287,6 +291,8 @@ class WorkerCapacity:
|
||||
available_slots: int,
|
||||
resident_models: Optional[set[str]] = None,
|
||||
free_memory_bytes: Optional[int] = None,
|
||||
cpu_percent: Optional[float] = None,
|
||||
gpu_utilization_percent: Optional[float] = None,
|
||||
now: Optional[float] = None,
|
||||
) -> None:
|
||||
"""Adopt a heartbeat snapshot. The worker is the source of truth for
|
||||
@@ -307,6 +313,10 @@ class WorkerCapacity:
|
||||
self.resident_models = set(resident_models)
|
||||
if free_memory_bytes is not None:
|
||||
self.free_memory_bytes = free_memory_bytes
|
||||
if cpu_percent is not None:
|
||||
self.cpu_percent = max(0.0, min(100.0, float(cpu_percent)))
|
||||
if gpu_utilization_percent is not None:
|
||||
self.gpu_utilization_percent = max(0.0, min(100.0, float(gpu_utilization_percent)))
|
||||
# Parks are released on a timer, and by the worker restarting — never
|
||||
# by the worker's own load report.
|
||||
#
|
||||
@@ -330,6 +340,9 @@ class WorkerCapacity:
|
||||
"zombie_tasks": self.zombie_tasks,
|
||||
"available_slots": self.available_slots,
|
||||
"resident_models": sorted(self.resident_models),
|
||||
"free_memory_bytes": self.free_memory_bytes,
|
||||
"cpu_percent": self.cpu_percent,
|
||||
"gpu_utilization_percent": self.gpu_utilization_percent,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -305,6 +305,8 @@ class WorkerPool:
|
||||
available_slots: int,
|
||||
resident_models: Optional[set[str]] = None,
|
||||
free_memory_bytes: Optional[int] = None,
|
||||
cpu_percent: Optional[float] = None,
|
||||
gpu_utilization_percent: Optional[float] = None,
|
||||
latency_ms: Optional[float] = None,
|
||||
now: Optional[float] = None,
|
||||
) -> Optional[ConnectedWorker]:
|
||||
@@ -319,6 +321,8 @@ class WorkerPool:
|
||||
available_slots=available_slots,
|
||||
resident_models=resident_models,
|
||||
free_memory_bytes=free_memory_bytes,
|
||||
cpu_percent=cpu_percent,
|
||||
gpu_utilization_percent=gpu_utilization_percent,
|
||||
)
|
||||
return worker
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -194,20 +194,22 @@ class RegisterResponse(_message.Message):
|
||||
def __init__(self, envelope: _Optional[_Union[Envelope, _Mapping]] = ..., worker_id: _Optional[str] = ..., session_token: _Optional[str] = ..., session_epoch: _Optional[int] = ..., protocol_version: _Optional[int] = ..., session_expires_at_unix: _Optional[int] = ..., heartbeat_interval_seconds: _Optional[int] = ..., authoritative_in_flight: _Optional[_Iterable[_Union[TaskRef, _Mapping]]] = ..., error: _Optional[_Union[Error, _Mapping]] = ...) -> None: ...
|
||||
|
||||
class Heartbeat(_message.Message):
|
||||
__slots__ = ("envelope", "active_tasks", "available_slots", "resident_models", "free_memory_bytes", "cpu_percent")
|
||||
__slots__ = ("envelope", "active_tasks", "available_slots", "resident_models", "free_memory_bytes", "cpu_percent", "gpu_utilization_percent")
|
||||
ENVELOPE_FIELD_NUMBER: _ClassVar[int]
|
||||
ACTIVE_TASKS_FIELD_NUMBER: _ClassVar[int]
|
||||
AVAILABLE_SLOTS_FIELD_NUMBER: _ClassVar[int]
|
||||
RESIDENT_MODELS_FIELD_NUMBER: _ClassVar[int]
|
||||
FREE_MEMORY_BYTES_FIELD_NUMBER: _ClassVar[int]
|
||||
CPU_PERCENT_FIELD_NUMBER: _ClassVar[int]
|
||||
GPU_UTILIZATION_PERCENT_FIELD_NUMBER: _ClassVar[int]
|
||||
envelope: Envelope
|
||||
active_tasks: int
|
||||
available_slots: int
|
||||
resident_models: _containers.RepeatedScalarFieldContainer[str]
|
||||
free_memory_bytes: int
|
||||
cpu_percent: float
|
||||
def __init__(self, envelope: _Optional[_Union[Envelope, _Mapping]] = ..., active_tasks: _Optional[int] = ..., available_slots: _Optional[int] = ..., resident_models: _Optional[_Iterable[str]] = ..., free_memory_bytes: _Optional[int] = ..., cpu_percent: _Optional[float] = ...) -> None: ...
|
||||
gpu_utilization_percent: float
|
||||
def __init__(self, envelope: _Optional[_Union[Envelope, _Mapping]] = ..., active_tasks: _Optional[int] = ..., available_slots: _Optional[int] = ..., resident_models: _Optional[_Iterable[str]] = ..., free_memory_bytes: _Optional[int] = ..., cpu_percent: _Optional[float] = ..., gpu_utilization_percent: _Optional[float] = ...) -> None: ...
|
||||
|
||||
class TaskAccepted(_message.Message):
|
||||
__slots__ = ("ref", "envelope")
|
||||
|
||||
@@ -228,11 +228,9 @@ message Heartbeat {
|
||||
uint32 active_tasks = 2;
|
||||
uint32 available_slots = 3;
|
||||
repeated string resident_models = 4;
|
||||
uint64 free_memory_bytes = 5;
|
||||
double cpu_percent = 6;
|
||||
// GPU utilisation is deliberately absent: unobtainable on Apple without
|
||||
// sudo powermetrics and absent on CUDA without a new NVML dependency.
|
||||
// Slots + queue depth are the load signals (goal_v2.md A11).
|
||||
optional uint64 free_memory_bytes = 5;
|
||||
optional double cpu_percent = 6;
|
||||
optional double gpu_utilization_percent = 7;
|
||||
}
|
||||
|
||||
message TaskAccepted { TaskRef ref = 1; Envelope envelope = 2; }
|
||||
|
||||
@@ -117,6 +117,13 @@ class Target:
|
||||
latency_ms: float = 0.0
|
||||
active_tasks: int = 0
|
||||
max_tasks: int = 0
|
||||
cpu_percent: Optional[float] = None
|
||||
free_memory_bytes: Optional[int] = None
|
||||
system_memory_bytes: int = 0
|
||||
cpu_count: int = 0
|
||||
gpu_name: str = ""
|
||||
gpu_memory_bytes: int = 0
|
||||
gpu_utilization_percent: Optional[float] = None
|
||||
|
||||
@property
|
||||
def is_local(self) -> bool:
|
||||
@@ -135,6 +142,13 @@ class Target:
|
||||
"latency_ms": round(self.latency_ms, 1),
|
||||
"active_tasks": self.active_tasks,
|
||||
"max_tasks": self.max_tasks,
|
||||
"cpu_percent": self.cpu_percent,
|
||||
"free_memory_bytes": self.free_memory_bytes,
|
||||
"system_memory_bytes": self.system_memory_bytes,
|
||||
"cpu_count": self.cpu_count,
|
||||
"gpu_name": self.gpu_name,
|
||||
"gpu_memory_bytes": self.gpu_memory_bytes,
|
||||
"gpu_utilization_percent": self.gpu_utilization_percent,
|
||||
}
|
||||
|
||||
|
||||
@@ -192,6 +206,8 @@ def list_targets(control_plane=None) -> list[Target]:
|
||||
pool = getattr(control_plane, "pool", None) if control_plane.running else None
|
||||
for record in enrolled:
|
||||
live = pool.get(record.id) if pool is not None else None
|
||||
host = record.host or {}
|
||||
gpu = (host.get("gpus") or [{}])[0]
|
||||
connected = live is not None and not live.stale()
|
||||
available, detail = _availability(record, live, pool)
|
||||
targets.append(
|
||||
@@ -207,6 +223,13 @@ def list_targets(control_plane=None) -> list[Target]:
|
||||
latency_ms=live.latency_ms if live else 0.0,
|
||||
active_tasks=live.capacity.active_tasks if live else 0,
|
||||
max_tasks=live.capacity.max_concurrent_tasks if live else 0,
|
||||
cpu_percent=live.capacity.cpu_percent if live else None,
|
||||
free_memory_bytes=live.capacity.free_memory_bytes if live else None,
|
||||
system_memory_bytes=int(host.get("system_memory_bytes") or 0),
|
||||
cpu_count=int(host.get("cpu_count") or 0),
|
||||
gpu_name=str(gpu.get("model") or ""),
|
||||
gpu_memory_bytes=int(gpu.get("memory_bytes") or 0),
|
||||
gpu_utilization_percent=live.capacity.gpu_utilization_percent if live else None,
|
||||
)
|
||||
)
|
||||
return targets
|
||||
|
||||
@@ -43,6 +43,8 @@ import platform
|
||||
import random
|
||||
import socket
|
||||
import sys
|
||||
import threading
|
||||
from concurrent.futures import Future
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Awaitable, Callable, Optional, Protocol
|
||||
|
||||
@@ -73,6 +75,33 @@ _FALLBACK_MODEL_LOAD_SECONDS = 1800.0
|
||||
# see _oversized_result_error for why that has to be a failure and not a retry.
|
||||
MAX_MESSAGE_BYTES = 8 * 1024 * 1024
|
||||
|
||||
|
||||
def _heartbeat_resources() -> tuple[Optional[float], Optional[int], Optional[float]]:
|
||||
"""Sample cheap host telemetry without making a heartbeat depend on CUDA."""
|
||||
cpu_percent = free_memory_bytes = gpu_utilization_percent = None
|
||||
try:
|
||||
import psutil
|
||||
|
||||
cpu_percent = float(psutil.cpu_percent(interval=None))
|
||||
except Exception:
|
||||
logger.debug("Could not sample worker CPU usage", exc_info=True)
|
||||
try:
|
||||
import torch
|
||||
|
||||
if torch.cuda.is_available():
|
||||
free_memory_bytes = int(torch.cuda.mem_get_info()[0])
|
||||
except Exception:
|
||||
logger.debug("Could not sample worker free VRAM", exc_info=True)
|
||||
try:
|
||||
import pynvml
|
||||
|
||||
pynvml.nvmlInit()
|
||||
handle = pynvml.nvmlDeviceGetHandleByIndex(0)
|
||||
gpu_utilization_percent = float(pynvml.nvmlDeviceGetUtilizationRates(handle).gpu)
|
||||
except Exception:
|
||||
logger.debug("Could not sample worker GPU usage", exc_info=True)
|
||||
return cpu_percent, free_memory_bytes, gpu_utilization_percent
|
||||
|
||||
# Room left for result_json, the ref, and protobuf framing when a payload does
|
||||
# ride inline. The inline decision is made on the payload alone, so without a
|
||||
# reserve a payload sized exactly at the frame cap would overflow it.
|
||||
@@ -339,6 +368,11 @@ class WorkerClient:
|
||||
self._running: dict[str, asyncio.Task] = {}
|
||||
self._keepalives: dict[str, asyncio.Task] = {}
|
||||
self._maintenance: set[asyncio.Task] = set()
|
||||
self._telemetry: tuple[Optional[float], Optional[int], Optional[float]] = (None, None, None)
|
||||
# A driver query can hang indefinitely. Keep that one query owned
|
||||
# rather than cancelling its awaiter and starting a fresh thread at
|
||||
# every heartbeat.
|
||||
self._telemetry_task: Optional[asyncio.Future] = None
|
||||
self._prewarms: dict[str, asyncio.Task] = {}
|
||||
self._prewarm_cancellations: dict[str, asyncio.Task] = {}
|
||||
self._epoch = 0
|
||||
@@ -683,10 +717,59 @@ class WorkerClient:
|
||||
async def _heartbeat_loop(self, interval: float) -> None:
|
||||
while True:
|
||||
await asyncio.sleep(interval)
|
||||
await self._refresh_telemetry()
|
||||
await self._send(self.heartbeat_message())
|
||||
|
||||
async def _refresh_telemetry(self) -> None:
|
||||
"""Publish completed samples and retain one non-blocking probe.
|
||||
|
||||
CUDA/NVML calls may wedge in a driver. A timed ``to_thread`` await
|
||||
only cancels the awaiter, leaving that thread alive; retaining this
|
||||
task prevents later heartbeats from accumulating more blocked probes.
|
||||
"""
|
||||
if not self._accepting_assignments or self._stop.is_set():
|
||||
return
|
||||
task = self._telemetry_task
|
||||
if task is not None and task.done():
|
||||
try:
|
||||
sampled = task.result()
|
||||
except Exception:
|
||||
logger.debug("Could not sample worker telemetry", exc_info=True)
|
||||
else:
|
||||
# A partial failed sample must not erase an independent last
|
||||
# good value. Presence on the heartbeat remains honest until
|
||||
# that individual metric can next be measured.
|
||||
self._telemetry = tuple(
|
||||
current if value is None else value
|
||||
for current, value in zip(self._telemetry, sampled)
|
||||
)
|
||||
self._telemetry_task = None
|
||||
|
||||
if self._telemetry_task is None:
|
||||
# Read-only driver probes cannot be interrupted. Keep one across
|
||||
# reconnects, outside assignment drain and the shared executor
|
||||
# (whose shutdown would otherwise wait forever for a wedged driver).
|
||||
result = Future()
|
||||
self._telemetry_task = asyncio.wrap_future(result)
|
||||
|
||||
def sample() -> None:
|
||||
try:
|
||||
result.set_result(_heartbeat_resources())
|
||||
except Exception:
|
||||
logger.debug("Could not sample worker telemetry", exc_info=True)
|
||||
result.set_result((None, None, None))
|
||||
|
||||
threading.Thread(
|
||||
target=sample, name="worker-telemetry-probe", daemon=True,
|
||||
).start()
|
||||
|
||||
def heartbeat_message(self) -> pb.WorkerMessage:
|
||||
"""Build the worker's current liveness/capacity frame."""
|
||||
cpu_percent, free_memory_bytes, gpu_utilization_percent = self._telemetry
|
||||
telemetry = {}
|
||||
if cpu_percent is not None: telemetry["cpu_percent"] = cpu_percent
|
||||
if free_memory_bytes is not None: telemetry["free_memory_bytes"] = free_memory_bytes
|
||||
if gpu_utilization_percent is not None: telemetry["gpu_utilization_percent"] = gpu_utilization_percent
|
||||
return pb.WorkerMessage(
|
||||
heartbeat=pb.Heartbeat(
|
||||
active_tasks=len(self._running),
|
||||
@@ -694,6 +777,7 @@ class WorkerClient:
|
||||
0, self.config.max_concurrent_tasks - len(self._running)
|
||||
),
|
||||
resident_models=self._resident_models(),
|
||||
**telemetry,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@@ -2068,7 +2068,9 @@ class WorkerServicer(pb_grpc.WorkerServiceServicer):
|
||||
active_tasks=active_tasks,
|
||||
available_slots=available_slots,
|
||||
resident_models=set(beat.resident_models),
|
||||
free_memory_bytes=beat.free_memory_bytes,
|
||||
free_memory_bytes=beat.free_memory_bytes if beat.HasField("free_memory_bytes") else None,
|
||||
cpu_percent=beat.cpu_percent if beat.HasField("cpu_percent") else None,
|
||||
gpu_utilization_percent=beat.gpu_utilization_percent if beat.HasField("gpu_utilization_percent") else None,
|
||||
)
|
||||
self._queue_heartbeat_touch(session)
|
||||
return
|
||||
|
||||
+6
-7
@@ -21,13 +21,12 @@ The pinned commit SHA for `omnivoice.cpp` lives in
|
||||
scripts/build-omnivoice-tts.sh --platform <slug> --commit-sha <40hex>
|
||||
```
|
||||
|
||||
See `.github/workflows/ci.yml` `build-omnivoice-tts` job for the CI
|
||||
matrix that produces these artifacts on every PR. The Apple Silicon
|
||||
slot (`macos-14`) is marked `continue-on-error: true` because the
|
||||
upstream `omnivoice.cpp` README does not publish a `buildmetal.sh`
|
||||
(Pitfall 1 in `04-RESEARCH.md`); a failed Metal build is documented
|
||||
and the macOS Apple Silicon cloning default falls back to the
|
||||
in-process `VoiceStudioBackend`.
|
||||
See `.github/workflows/build-omnivoice-tts.yml` `build-omnivoice-tts` job
|
||||
for the CI matrix that produces these artifacts. Apple Silicon (`macos-14`)
|
||||
builds cleanly with `-DGGML_METAL=ON` at the pinned SHA (#2105), enabling
|
||||
hardware-accelerated Metal inference when the packaged artifact passes binary
|
||||
preflight and macOS permits execution. Missing or blocked binaries retain the
|
||||
in-process `VoiceStudioBackend` fallback.
|
||||
|
||||
## Placeholder note
|
||||
|
||||
|
||||
@@ -75,7 +75,7 @@
|
||||
},
|
||||
"frontend": {
|
||||
"name": "omnivoice-studio",
|
||||
"version": "0.5.2",
|
||||
"version": "0.5.3",
|
||||
"dependencies": {
|
||||
"@fontsource-variable/inter": "^5.3.0",
|
||||
"@fontsource-variable/source-serif-4": "^5.3.0",
|
||||
@@ -2800,6 +2800,8 @@
|
||||
|
||||
"@types/keyv/@types/node": ["@types/node@24.13.3", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-Dh8vAsV36ig5wa9OX4pXvMc9D3Veibfw2wix0CUwYODLD8nkj9UsLjASr49nPg+2eKzxhBV+v7L8pXvT4e639Q=="],
|
||||
|
||||
"@types/react-dom/@types/react": ["@types/react@19.3.0", "", { "dependencies": { "csstype": "^3.2.2" } }, "sha512-N0rFCuH9YoxG9/m61l9MfpJKfmLOVU0em7ipIz6TRgSSkvReLB9vL85GB+yr8Bs5leqpvg96JSwF4ZS1s4viQg=="],
|
||||
|
||||
"@types/responselike/@types/node": ["@types/node@24.13.3", "", { "dependencies": { "undici-types": "~7.18.0" } }, "sha512-Dh8vAsV36ig5wa9OX4pXvMc9D3Veibfw2wix0CUwYODLD8nkj9UsLjASr49nPg+2eKzxhBV+v7L8pXvT4e639Q=="],
|
||||
|
||||
"@vitest/browser/pngjs": ["pngjs@7.0.0", "", {}, "sha512-LKWqWJRhstyYo9pGvgor/ivk2w94eSjE3RGVuzLGlr3NmD8bf7RcYGze1mNdEHRP6TRP6rMuDHk5t44hnTRyow=="],
|
||||
|
||||
@@ -103,6 +103,27 @@ RUN python3 -c "import os, torch, torchaudio, torchvision; \
|
||||
import torchvision.ops; torchvision.ops.nms; \
|
||||
print('torchvision C++ ops resolve against this torch')"
|
||||
|
||||
# CTranslate2 (WhisperX, faster-whisper) links cuDNN 8, but the CUDA base image
|
||||
# ships cuDNN 9, so libcudnn_ops_infer.so.8 is absent and loading it aborts the
|
||||
# backend process outright rather than raising (#1371). scripts/setup.py
|
||||
# side-loads the cuDNN 8 libraries for source installs; the image needs the same
|
||||
# shim or Docker users lose every CTranslate2 ASR engine (#2050).
|
||||
#
|
||||
# The target is derived from sys.prefix rather than hardcoded: backend/core/
|
||||
# cudnn8.py looks for <sys.prefix>/lib/pythonX.Y/site-packages/cudnn8_compat,
|
||||
# and sys.prefix differs between the conda-based CUDA image and the ROCm venv.
|
||||
# --no-deps keeps this to the cuDNN wheels alone, leaving the base image's torch
|
||||
# stack untouched. Skipped for ROCm, which does not use cuDNN.
|
||||
RUN if [ "$GPU_FLAVOR" = "cuda" ]; then \
|
||||
target="$(python3 -c "import os, sys; print(os.path.join(sys.prefix, 'lib', 'python%d.%d' % sys.version_info[:2], 'site-packages', 'cudnn8_compat'))")" && \
|
||||
uv pip install --python "$(command -v python3)" --no-cache --no-deps \
|
||||
--target "$target" nvidia-cudnn-cu12==8.9.7.29 && \
|
||||
python3 -c "import os, sys; d = os.path.join(sys.prefix, 'lib', 'python%d.%d' % sys.version_info[:2], 'site-packages', 'cudnn8_compat', 'nvidia', 'cudnn', 'lib'); \
|
||||
libs = [f for f in os.listdir(d) if '.so.8' in f]; \
|
||||
assert libs, 'cudnn8_compat installed but no .so.8 libraries in ' + d; \
|
||||
print('cuDNN 8 compat libraries: %d' % len(libs))"; \
|
||||
fi
|
||||
|
||||
# Copy application source
|
||||
COPY backend/ ./backend/
|
||||
COPY omnivoice/ ./omnivoice/
|
||||
|
||||
+62
-26
@@ -71,26 +71,18 @@ git tag vX.Y.Z
|
||||
git push origin vX.Y.Z
|
||||
```
|
||||
|
||||
The `Desktop Release` workflow fires on tag push. It builds four targets in parallel on GitHub Actions runners:
|
||||
`electron-release.yml` builds Linux x64, Windows x64, macOS arm64 and macOS
|
||||
x64 installers with updater metadata and packaged startup checks. Ordinary tag
|
||||
pushes create drafts; a tag-scoped manual dispatch with `publish=true` publishes
|
||||
after all four targets pass. Signing checks apply by default.
|
||||
|
||||
| Target | Runner | Artifact |
|
||||
|---|---|---|
|
||||
| macOS Apple Silicon | macos-14 | `.dmg` + updater `.app.tar.gz` |
|
||||
| macOS Intel | macos-13 | `.dmg` + updater `.app.tar.gz` |
|
||||
| Windows x64 | windows-2022 | `.msi`, machine-wide and per-user, each with its updater `.sig` |
|
||||
| Linux x64 | ubuntu-22.04 | `.AppImage` + updater `.AppImage.sig` |
|
||||
|
||||
Each runner signs the updater payload with the stored `TAURI_SIGNING_PRIVATE_KEY`, merges into a single `latest.json`, and attaches everything to the draft release.
|
||||
|
||||
Workflow runtime: **~20-40 minutes** (PyInstaller + four platform builds). Follow progress at:
|
||||
`https://github.com/debpalash/VoiceStudio/actions`
|
||||
|
||||
The release stays a draft while the platforms build. Once every platform, the
|
||||
updater-manifest repair and the uninstall scripts are done, the
|
||||
`release-notes-checksums` job writes all four platforms' checksums into the
|
||||
notes and publishes it, with no manual step. A failed platform leaves the
|
||||
release a draft, so nothing half-built goes public. Existing clients detect
|
||||
the update on their next launch.
|
||||
For the one-time transition tag, set `TAURI_SUNSET_TAG`, dispatch `release.yml`
|
||||
on that tag with `draft=true`, and wait for its final Tauri installers and signed
|
||||
updater feeds. Then dispatch `electron-release.yml` on the same tag. Automatic
|
||||
Electron builds are skipped for this tag to avoid racing the Tauri draft.
|
||||
Keep the release draft until both builds and their checks have passed.
|
||||
See [Electron transition](#electron-transition-next-desktop-release) below for
|
||||
signing requirements and the explicit owner-only unsigned exception.
|
||||
|
||||
## 5b. Deployment channels — all must ship (hard rule, owner-set 2026-07-16)
|
||||
|
||||
@@ -100,8 +92,9 @@ bug to fix immediately, not backlog.
|
||||
|
||||
| Channel | Source | Produced by | How to verify |
|
||||
|---|---|---|---|
|
||||
| GitHub Release: installers + signed `latest.json` (**Stable** updater channel) | the `vX.Y.Z` tag | `release.yml` on tag push | Release page has dmg (arm+intel), msi (machine-wide and per-user), AppImage, `latest.json` and `latest-user.json`; body = the CHANGELOG section (not the auto-generated fallback), followed by per-platform checksums and a **Contributors** avatar strip (owner + every PR author for the tag — the `contributors-strip` job) |
|
||||
| **Preview** updater channel (rolling `preview` prerelease) | **`main` only** | `release.yml` nightly cron / manual dispatch | preview `latest.json` uses main's version when it is ahead; otherwise it advances the stable patch, then appends `-N` so it semver-sorts above stable |
|
||||
| GitHub Release: Electron installers and updater manifests | the `vX.Y.Z` tag | `electron-release.yml`, explicit publish dispatch | All four platforms, Electron manifests, SHA256SUMS.txt, versioned CHANGELOG notes; retained Tauri feeds point to the final Tauri tag |
|
||||
| Final Tauri installers and signed updater feeds | `TAURI_SUNSET_TAG` | `release.yml`, manual dispatch only | Both macOS architectures, Windows system/user installers, Linux AppImage, signed `latest.json` and `latest-user.json` |
|
||||
| Desktop preview channel | frozen during transition | no scheduled publishing | Existing preview assets remain available; new desktop previews are paused |
|
||||
| GHCR CUDA image: `:X.Y.Z`, `:X.Y`, `:stable` | the tag | `docker.yml` on tag push | `docker manifest inspect ghcr.io/debpalash/omnivoice-studio:X.Y.Z` |
|
||||
| GHCR ROCm image: `:X.Y.Z-rocm`, `:X.Y-rocm`, `:stable-rocm` | the tag | `docker.yml` on tag push | same, with `-rocm` suffix |
|
||||
| Docker Hub mirror of **all** the above tags | the tag | `docker.yml` (gated on `DOCKERHUB_*` secrets) | tag list at hub.docker.com/r/palashdeb/omnivoice-studio/tags |
|
||||
@@ -109,11 +102,8 @@ bug to fix immediately, not backlog.
|
||||
| Rolling Docker previews: `:latest`, `:main`, `:rocm` | **`main` only** | `docker.yml` on every main push | tag timestamps move with main |
|
||||
|
||||
**Preview/RC policy:** there are no RC tags (beta cadence — see CLAUDE.md).
|
||||
The preview channel *is* the release candidate, and it **always builds from
|
||||
`main`** — the preview-gate in `release.yml` refuses `publish_preview` from
|
||||
any other branch, and the rolling Docker tags track `main` by construction.
|
||||
To get users testing a fix: merge to `main`, then cut a preview. Never a
|
||||
side-branch build.
|
||||
Rolling Docker previews always build from `main`. Desktop preview publication
|
||||
is paused during the Electron transition; never publish a side-branch preview.
|
||||
|
||||
## 6. Expect-to-fail-first-time on Windows and Linux
|
||||
|
||||
@@ -161,3 +151,49 @@ before Tauri uploads them again. A macOS retry also replaces that architecture's
|
||||
versionless updater archive. Other versions, sibling platforms, and updater
|
||||
manifests remain intact. Inventory or deletion permission/network failures stop
|
||||
the job instead of hiding an upload collision.
|
||||
|
||||
## Electron transition (next desktop release)
|
||||
|
||||
Electron is the primary desktop distribution. electron-release.yml builds Linux
|
||||
x64, Windows x64, macOS arm64 and macOS x64, checks packaged startup and updater
|
||||
artifacts, then creates a draft. Publishing requires a tag-scoped manual dispatch
|
||||
with publish=true. Tag pushes never publish automatically. electron-build.yml
|
||||
remains the artifact-only rehearsal; run it before tagging.
|
||||
|
||||
Set TAURI_SUNSET_TAG to the final Tauri version tag. Run the manual release.yml
|
||||
on that tag first; it rejects other refs. Automatic Tauri builds and scheduled
|
||||
previews are retired. Keep the transition release draft until Electron on the
|
||||
same tag completes. Electron requires the final signed latest.json and
|
||||
latest-user.json assets; subsequent releases copy those feeds without changing
|
||||
their immutable sunset payload URLs. Retain the sunset release and its assets.
|
||||
|
||||
Write versioned CHANGELOG notes before release. Review all four platform builds,
|
||||
checksums, signing requirements and docs/electron-migration.md. Existing Electron
|
||||
artifact names and app IDs remain stable for updater compatibility. This pipeline
|
||||
ships stable releases; rolling preview publication is paused during transition.
|
||||
|
||||
Preparation is not proof of cross-platform packaging, signing, migration, or a
|
||||
real installed update hop. Record those results before release. Keep Tauri source
|
||||
and shared assets until remaining Electron resource references are relocated.
|
||||
No tag, version bump, or publishing is authorized by workflow preparation alone.
|
||||
|
||||
Electron signing uses ELECTRON_CSC_LINK and ELECTRON_CSC_KEY_PASSWORD secrets.
|
||||
Without them rehearsal/draft artifacts are unsigned or ad-hoc signed. Publishing
|
||||
checks macOS signing/notarization and Windows Authenticode signatures by default.
|
||||
The owner may explicitly choose the existing unsigned-release policy by dispatching
|
||||
with `allow_unsigned=true` (both dispatch and rerun actors must be the repository owner); the release notes then disclose OS trust warnings and
|
||||
unverified macOS automatic updates. Never select this exception without the owner's
|
||||
choice. Tauri's signing keys do not sign Electron packages.
|
||||
|
||||
For the transition tag, automatic Electron release jobs are skipped. Build the
|
||||
manual Tauri sunset draft first, then dispatch Electron on the same tag after
|
||||
its signed updater feeds exist. Later tags build Electron automatically.
|
||||
|
||||
|
||||
If a packaging-workflow fix is needed after tagging, keep the release tag
|
||||
immutable. Merge and validate the workflow fix on main, then dispatch
|
||||
`electron-release.yml` from main with `release_tag=vX.Y.Z`. Validation and every
|
||||
packaging/release job check out that exact tag; only the workflow comes from
|
||||
main. Empty signing secrets are omitted from the builder environment so drafts
|
||||
and explicitly accepted unsigned builds do not interpret the working directory
|
||||
as a certificate. Publication still requires `publish=true` and the same guards.
|
||||
|
||||
+2
-2
@@ -39,7 +39,7 @@ VoiceStudio/
|
||||
│ │ └── setup/ first-run wizard, model download
|
||||
│ ├── core/ config, db, job queue, event bus, auth/CSRF, path security,
|
||||
│ │ opt-in analytics, version, diagnostics
|
||||
│ ├── services/ 85 modules of business logic — TTS, dubbing pipeline,
|
||||
│ ├── services/ 88 modules of business logic — TTS, dubbing pipeline,
|
||||
│ │ audio DSP, GPU gateway, engine routing, model lifecycle
|
||||
│ ├── engines/ per-engine adapters: indextts, supertonic3, confucius4,
|
||||
│ │ dots_tts, moss_tts_v15, pockettts, audiocpp,
|
||||
@@ -103,7 +103,7 @@ VoiceStudio/
|
||||
│
|
||||
├── .agents/skills/ ⟵ canonical skill copies (vite, fastapi-python), pinned by
|
||||
│ skills-lock.json — followed by path, never symlinked
|
||||
├── skills/ ⟵ skills this repo publishes (omnivoice, oss-maintainer)
|
||||
├── skills/ ⟵ skills this repo publishes (voicestudio, voicestudio-maintainer)
|
||||
│
|
||||
├── infra/ ⟵ edge/deploy workers (not the Docker deploy path)
|
||||
│ └── install-redirect/ voicestudio.sh/install — UA-sniffing installer worker
|
||||
|
||||
@@ -30,7 +30,7 @@ The integration shape is `VoiceStudioGGUFBackend(TTSBackend)` wrapping Phase 2's
|
||||
| License compatible with v0.3.x ship? | YES — Apache-2.0 (model) + MIT (runtime) | Both verified via HF model card + GitHub README. Same Apache-2.0 chain as the upstream model already shipping in v0.2.7. |
|
||||
| Runtime: llama.cpp / candle / custom? | CUSTOM (`omnivoice.cpp`, MIT) — does NOT load in vanilla llama.cpp | `gguf.architecture = "omnivoice-lm"` from HF API; README states "GGUF weights for omnivoice.cpp, a C++17/GGML port of VoiceStudio". |
|
||||
| Quant variants and footprints? | 4 quants × 2 files each (base + tokenizer): Q4_K_M (659 MB), Q8_0 (945 MB), BF16 (1.60 GB), F32 (3.19 GB) | HF `siblings` list confirms all 8 files; sizes from model card table. |
|
||||
| Cross-platform runtime fit? | Linux + Windows + macOS Intel YES via documented build scripts; macOS Apple Silicon Metal CONDITIONAL (no `buildmetal.sh` published, only feature mention) | `buildcpu.sh`, `buildcuda.sh`, `buildvulkan.sh`, `buildall.sh` listed; Metal in description only — Wave 1 Task 3 builds and verifies via `cmake -DGGML_METAL=ON` per A1. |
|
||||
| Cross-platform runtime fit? | Linux + Windows + macOS Intel YES via documented build scripts; macOS Apple Silicon Metal YES (builds clean via `cmake -DGGML_METAL=ON` at pinned SHA, #2105) | `buildcpu.sh`, `buildcuda.sh`, `buildvulkan.sh`, `buildall.sh` listed; Metal builds cleanly via `cmake -DGGML_METAL=ON` in CI and locally (#2105). |
|
||||
| Subprocess CLI fits Phase 2 `SubprocessBackend`? | YES | README shows `echo "Hello world." | ./build/omnivoice-tts --model … --codec … --lang … -o …` — line-oriented stdin + argv + output-file pattern is exactly what `SubprocessBackend` is designed for. |
|
||||
|
||||
### Pinned SHAs (filled in Wave 1 by Task 1)
|
||||
@@ -52,13 +52,13 @@ Both SHAs are mirrored in `backend/engines/omnivoice_gguf/quant_map.json` `_meta
|
||||
- Adds a maintained-by-others C++ runtime to the dependency graph (`omnivoice.cpp`, 42 stars at decision time).
|
||||
- Adds ~12-16 MB of platform binaries to the installer (must verify against Phase 3 mirror-timing baseline per Pitfall 6).
|
||||
- macOS code signing scope expands by 4 binaries (track via REL-05; same `xattr -cr` workaround as #54 applies in v0.3.x).
|
||||
- `omnivoice.cpp` README does not publish a macOS Metal build script — only `buildcpu.sh`, `buildcuda.sh`, `buildvulkan.sh`, `buildall.sh`. Apple Silicon Metal must be verified in Wave 1.
|
||||
- `omnivoice.cpp` README does not publish a standalone `buildmetal.sh` script; Apple Silicon Metal is built directly via `cmake -DGGML_METAL=ON` (#2105).
|
||||
|
||||
**Mitigations:**
|
||||
- Pin `omnivoice.cpp` by commit SHA (`886fc079838ca7400cb2b42b36e2a65aa1daabe8`); rebuild from pinned SHA in CI for all 4 target platforms.
|
||||
- Pin every quant file by commit SHA in `quant_map.json` (`361609388ae572a820d085185bbbe2a2aac4b30e`); shippable JSON so the table can update without an app release.
|
||||
- In-process `VoiceStudioBackend` remains as fallback if any GGUF step fails (probe, download, load, generate).
|
||||
- macOS Apple Silicon Metal build is verified in Wave 1 with explicit acceptance criteria; if blocked, downgrade SPIKE-01 default on macOS to in-process path and document in this ADR's "Status" line.
|
||||
- macOS Apple Silicon Metal build is verified via `cmake -DGGML_METAL=ON` (#2105); in-process `VoiceStudioBackend` remains available as general fallback.
|
||||
- SHA-256 checksums on bundled binaries (per GATE-05); verify at first launch and on every quant load.
|
||||
- Subprocess arg composition uses typed `Path` objects rooted in app directories; quant override UI is a dropdown over `quant_map.json` entries only (no freeform path input — supply-chain control analogous to INST-09).
|
||||
|
||||
|
||||
@@ -234,3 +234,33 @@ panels instead so neither editor becomes unusably small.
|
||||
from-source checkout.
|
||||
- **Installed it but still "needs install"** — restart the backend so Python
|
||||
picks up the newly-installed module.
|
||||
- **"The 'argos' engine's CTranslate2 runtime could not be loaded…"** — Argos
|
||||
translates on CTranslate2, and on Linux kernels that refuse an executable
|
||||
stack the CTranslate2 library shipped with Python 3.11 installs (4.4.0) is
|
||||
rejected outright. VoiceStudio repairs that library in place on first use; if
|
||||
it cannot (read-only install), the message names the fix — reinstall the
|
||||
backend on Python 3.12+, or run `patchelf --clear-execstack` on the library
|
||||
once — and NLLB stays available in the meantime
|
||||
([#692](https://github.com/debpalash/VoiceStudio/issues/692)).
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
SRT and WebVTT exports round each cue timestamp once to the nearest millisecond, including carry into the next second or minute. The OpenAI-compatible transcription exports use the same formatter.
|
||||
|
||||
|
||||
Paste translation accepts WebVTT files with hourless timestamps. Only timing records at line starts activate timestamp matching; timestamp-like text inside a sentence remains dialogue.
|
||||
|
||||
Subtitle import preserves numeric dialogue such as years and countdowns, including files mixing numbered and unnumbered cues. Cue numbers are removed only at identified cue boundaries.
|
||||
|
||||
|
||||
Argos accepts Chinese/Simplified Chinese names, Mandarin aliases, and language tags such as `zh-CN`. Traditional Chinese requests (`zh-TW`, `zh-Hant`, and the display name) are rejected explicitly; select NLLB for Traditional Chinese rather than silently receiving a different script.
|
||||
|
||||
Mixed or malformed SRT files can make a bare number indistinguishable from spoken dialogue. The importer removes numbering only when cue boundaries and sequential numbering support it; ambiguous nonsequential numbers are retained as text to avoid silent data loss. Standard indexed SRT and WebVTT exports avoid this ambiguity.
|
||||
|
||||
If the Argos native runtime cannot load, both desktop and browser clients show localized recovery guidance: reinstall the backend or select NLLB. The API returns the stable `argos_runtime_unavailable` error code without exposing native library paths.
|
||||
|
||||
WebVTT import separates metadata blocks from cue identifiers using the [WebVTT block-parsing rules](https://www.w3.org/TR/webvtt1/#file-parsing): a timing line immediately after an identifier makes a cue, even when that identifier is NOTE, STYLE, or REGION. Later timing examples inside metadata are ignored, and empty cues never borrow the next cue’s identifier as dialogue.
|
||||
|
||||
Dubbing transcription emits keepalives during quiet diarization, reference-refinement, and cleanup steps. Disconnecting stops queued model work; native calls already running retain their model until they finish, then cleanup restores TTS. Task streams also request that proxies disable buffering so keepalives reach the client promptly.
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
# Electron dubbing workspace
|
||||
|
||||
The idle workspace includes an original/dubbed demo comparison with compact
|
||||
player controls. Sync playheads aligns positions without starting both videos.
|
||||
Sample transcript edits are retained per language while the demo is mounted;
|
||||
they do not regenerate the prerecorded audio. Edit on the dubbed card imports
|
||||
that sample video into the normal upload/transcription and editing workflow.
|
||||
|
||||
Open Dub from the cloning sidebar or command search. Upload or drop audio/video, or explicitly submit a video URL;
|
||||
preparation completes before transcription starts. The editor shows source text,
|
||||
editable translated text, and per-segment voice/timing controls. Translation uses
|
||||
@@ -199,3 +205,92 @@ Below 40rem of workspace width, Dubbing stacks its controls above the editor wit
|
||||
Global speed/quality changes preserve explicit Dubbing production steps, including settings restored from projects. Unset steps continue to follow the backend's current preset.
|
||||
|
||||
NLLB resolves explicit FLORES language/script codes, unambiguous ISO-639-3 codes and common short aliases, including Traditional Chinese. Unsupported or script-ambiguous source, target or per-segment languages are rejected before model loading instead of silently translating to English.
|
||||
|
||||
### Speech integrity and timing
|
||||
|
||||
A failed, empty or unreadable speech segment stops generation before a new track
|
||||
replaces the previous output. Partial regeneration repairs missing or corrupt
|
||||
segment caches, including missing clips outside the requested changed-line list;
|
||||
it does not substitute silence and report completion. Auto speaker references
|
||||
exclude oversized clips when selecting a shared fallback, so short lines reuse a
|
||||
usable reference from their own speaker.
|
||||
|
||||
New segment caches retain the full generated speech in every timing mode. Strict
|
||||
Slot removes edge silence and fits the complete clip to its original start/end
|
||||
with pitch-preserving speed adjustment. Very long or short translations can still
|
||||
sound unnaturally fast or slow; shorten or expand the translation for natural
|
||||
pacing. Legacy clipped caches require regeneration once, because fitting cannot
|
||||
recover discarded words. Explicit legacy Trim and Off options retain their
|
||||
respective clipping and overlap behavior.
|
||||
|
||||
Concise and Smart Fit stop on unresolved overflow rather than publishing cut-off
|
||||
words. Shorten the translation, choose Strict Slot, or allow Stretch Video before
|
||||
retrying. Camera-cut segmentation uses nearby timed word boundaries when available
|
||||
and skips cuts inside speech that cannot be assigned safely. This improves phrase
|
||||
timing; it does not promise phoneme-level lip sync or correct inaccurate source
|
||||
transcripts automatically.
|
||||
|
||||
### Preserve sound outside dialogue
|
||||
|
||||
Background-preserving previews and audio/video exports keep the original stereo
|
||||
sound outside dialogue intervals, including audience reactions, music and ambience.
|
||||
Inside those intervals, they mix dubbed speech over the separated background, with
|
||||
10 ms transitions contained within the dialogue boundaries. Each generated language
|
||||
stores its source intervals; older tracks use their saved project intervals.
|
||||
Retimed modes also retime this background to follow the video. Ordinary Strict Slot
|
||||
and Concise modes keep the original background timeline.
|
||||
|
||||
Original media and a complete separated background are required. A missing or failed
|
||||
background mix stops export rather than silently exporting speech alone. Explicit
|
||||
speech-only export remains available. The preserved bed is cached locally and rebuilt
|
||||
when source files, dialogue intervals or the language's retiming plan change.
|
||||
Separation can still affect sounds overlapping dialogue; exact isolation from a
|
||||
single mixed recording is not guaranteed. Correct dialogue boundaries matter.
|
||||
|
||||
### Custom translation style
|
||||
|
||||
Select **Translate with agent**, then fill in **Translation style prompt** beside
|
||||
the translator controls. Describe tone, audience, formality, humor, idiom handling
|
||||
and how freely the dialogue should be adapted. For example: “Conversational Bengali
|
||||
for a young adult audience. Preserve jokes, adapt idioms naturally, keep names and
|
||||
numbers unchanged, and avoid stiff literal phrasing.”
|
||||
|
||||
The optional brief is saved with the project and restored after reopening. It applies
|
||||
to every selected target language and subsequent agent timing rewrites, using either
|
||||
a local CLI agent or the configured LLM. Meaning, timing, glossary and structured
|
||||
output requirements remain in effect. Clear the field to restore the default style;
|
||||
changing the brief does not retranslate existing segments until you run translation.
|
||||
The field accepts up to 5,000 characters and is locked while work is running.
|
||||
|
||||
### Translation activity footer
|
||||
|
||||
Translation and timing rewrites use the same footer area as Repair Agent. It opens
|
||||
with live CLI stdout/stderr in **Logs**; **Translations** shows original text beside
|
||||
validated translated output. Collapse **Details** to keep the status, language,
|
||||
elapsed time and Cancel action visible. Output from each language stays available
|
||||
until dismissed, the app reloads, or translation starts in another project. Logs are bounded to the
|
||||
latest 250,000 characters per run and are not saved into project files.
|
||||
|
||||
The counter tracks validated returned segments, not estimated progress. A CLI may
|
||||
stream logs while withholding its translation JSON until completion; API-backed
|
||||
translation currently returns one response, so its count updates when that response
|
||||
arrives. No fabricated percentage is shown. Errors remain visible, with Retry for
|
||||
failed translation work in the same project. API retries use failed segments when
|
||||
available; an incomplete CLI response requires retrying that language. Cancel stops
|
||||
the active translation and prevents late results from applying.
|
||||
|
||||
### Long timelines and duplicate ASR context
|
||||
|
||||
The timeline draws segments at their actual duration. Use Zoom in/out and Fit all
|
||||
above it to inspect short lines in long recordings; the zoomed view scrolls
|
||||
horizontally. Tiny overview bars cannot be accidentally dragged or resized. Click
|
||||
an overlap warning to zoom to the first affected segment. Nested overlaps are
|
||||
included in detection; simultaneous speakers are not automatically shifted apart.
|
||||
|
||||
After transcription, repeated chunk context is removed only when at least three
|
||||
matching words form a substantial prefix at matching timestamps for the same
|
||||
speaker. New words beyond that context remain. Stale segment boundaries are aligned
|
||||
to their own word timestamps only when those prove that the speech is disjoint.
|
||||
Existing translations/renders do not become correct merely by editing source text:
|
||||
changed lines must be translated and regenerated. Preserve the prior project when
|
||||
repairing an older transcript.
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
# macOS desktop shell
|
||||
|
||||
The expanded sidebar reserves space for the native traffic lights and app name.
|
||||
The collapsed sidebar is 64 px wide, with its toggle below the traffic lights
|
||||
and its right divider beginning below the 72 px header region.
|
||||
|
||||
Notifications appear at the top right, with space reserved before the bell.
|
||||
The notification menu opens downward and remains available while notification
|
||||
data loads. Settings uses an icon in the macOS sidebar footer; Local device
|
||||
sits beside it and opens the device and compute-target menu. The expanded
|
||||
sidebar retains the Local device label.
|
||||
|
||||
Windows and Linux retain their existing notification and device placement.
|
||||
|
||||
The notification control follows workspace headers in document order so their
|
||||
native drag regions cannot consume its mouse clicks. On macOS, run
|
||||
`node tests/native-bell-repro.mjs` from `electron/` against the dev renderer
|
||||
to verify a real system mouse click (requires Swift and Accessibility access).
|
||||
Browser automation alone bypasses native titlebar hit testing.
|
||||
@@ -0,0 +1,20 @@
|
||||
# Moving to Electron
|
||||
|
||||
The next desktop release uses Electron. Tauri receives one final sunset update;
|
||||
subsequent releases build only Electron. Existing Tauri downloads remain available.
|
||||
|
||||
1. Find your Tauri data directory in Settings and back up the entire directory
|
||||
while the app is closed. Keep reference audio stored outside it too.
|
||||
2. Install Electron for your platform. Keep Tauri and its data until verification.
|
||||
Do not run both apps against the same data directory.
|
||||
3. Check Electron's configured data location before generating. Use its supported
|
||||
storage/backend configuration to select the existing data directory.
|
||||
4. Verify voices, projects, history, and model locations. Generate a short test
|
||||
clip before removing the old app.
|
||||
|
||||
The shells share the backend, but shell preferences and credentials are not
|
||||
guaranteed to migrate. Recheck devices, shortcuts, theme, backend address and
|
||||
permissions. No automatic installer-to-installer migration is provided.
|
||||
|
||||
The final Tauri updater feeds retain signed Tauri payloads at immutable URLs.
|
||||
A Tauri updater must never receive an Electron installer.
|
||||
@@ -1,10 +1,10 @@
|
||||
# Electron compute and performance settings
|
||||
|
||||
Settings > Compute device exposes the existing device override, Windows torch.compile workaround, generation time budgets, and hardware readouts.
|
||||
Settings > Compute device exposes the existing device override, the torch.compile workaround, generation time budgets, and hardware readouts.
|
||||
|
||||
Device choices come from the backend's detected families plus Auto. The chosen preference and currently active family are displayed separately. Environment-pinned choices are disabled, an ignored unavailable override is explained, and a changed preference shows its actual restart requirement. Failed saves keep the last confirmed state. Nothing automatically restarts the backend or changes the active model.
|
||||
|
||||
The torch.compile workaround uses the same platform gate as Tauri: Windows can opt in; other platforms retain their working optimization. Generation budgets preserve separate GPU and CPU limits, validate the existing positive/21600-second range, and keep edits during refetches. An externally overridden budget reports that fact instead of implying the saved value will take effect after restart. Hardware RAM/VRAM readouts poll only while this view is mounted.
|
||||
The torch.compile workaround matches Tauri: since #2135 it is selectable on every platform, because the compile failures it works around are not Windows-only. Generation budgets preserve separate GPU and CPU limits, validate the existing positive/21600-second range, and keep edits during refetches. An externally overridden budget reports that fact instead of implying the saved value will take effect after restart. Hardware RAM/VRAM readouts poll only while this view is mounted.
|
||||
|
||||
During synthesis, the fixed-width primary action polls the existing model-status contract and names the active runtime phase: starting the AI runtime, loading weights, warming speech recognition, optimizing the model, generating, or receiving audio. Model-load percentage and elapsed time share the reserved status line, and the progress track switches from model loading to streamed audio delivery without moving the controls.
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@ Cancel stops the active process tree or download and returns to setup. Failure p
|
||||
|
||||
Verification: runtime regression tests cover consent gating, dependency changes, incomplete environments, failed repair, cancellation before download, and source replacement. `node electron/tests/packaged-smoke.mjs --setup` verifies the first-run view without initiating an installation. `--install` performs the explicit isolated runtime installation and verifies the packaged renderer's same-origin connection to its managed backend.
|
||||
|
||||
The first-run browser gate also survives a partial preflight payload without blanking, keeps navigation disabled until a complete passing report arrives, and presents required, installed and curated recommended models before the optional catalogue. Its compact 640 px layout leaves the model cards usable without horizontal overflow.
|
||||
First run has four steps: System check, Model packs, Privacy, and Enter studio. Model packs reuse the existing performance tiers and install only missing supported models after an explicit click. Installed state, remaining download size, disk checks and installation progress appear in one place. Advanced mode reveals individual models, engine tuning, privacy settings and recovery tools; failed system checks still expose recovery without enabling Advanced. Dictation and its permissions are optional on the final step and remain available in Settings. Navigation requires passing preflight, required models and the privacy choice; entering the studio opens the first-voice demo. On narrow windows the step navigation moves above the content and the footer stays visible.
|
||||
|
||||
A subsequent packaged launch also reused that installed runtime and started a managed backend. Electron discovers compatible Tauri default, custom and portable runtime locations and can reuse them without a download when the interpreter and both frozen dependency manifests match. Its location record distinguishes reused environments from custom runtimes Electron creates. Uninstall includes only an Electron-owned custom runtime; it never claims or removes a reused Tauri runtime.
|
||||
|
||||
@@ -73,3 +73,13 @@ Setup reserves its active operation before asynchronous compatibility checks. Re
|
||||
If the renderer remains empty after three bounded reloads, Electron paints an asset-independent localized recovery page. Its retry clears only Chromium's display cache, schedules a relaunch, and still shuts the managed backend down cleanly; voices, projects, models and preferences remain untouched.
|
||||
|
||||
Branding remains in a fixed native title row while onboarding status loads, installation runs or recovery needs retry. Runtime details scroll independently below it; installer phases wrap into two columns on narrow windows, so a growing progress/log surface cannot clip the wordmark or window controls.
|
||||
|
||||
On Windows and Linux, the collapsed workspace sidebar shows the VoiceStudio icon at the top. Use the toggle beside the page title to expand the sidebar. macOS retains its existing sidebar control.
|
||||
|
||||
Every main workspace header exposes the same sidebar toggle, including pages that automatically collapse the voice library at narrow widths. The control reflects the visible sidebar state and explicitly expands it for the current workspace.
|
||||
|
||||
The shared video player renders Vidstack's poster before playback, including the Dub source thumbnail, and hides it once playback starts. Play requests made while the video is loading wait for the provider to become ready, including timeline preview requests.
|
||||
|
||||
On Linux Wayland systems where Chromium logs `eglCreateImage failed` / `OzoneImageBacking` and video or window contents flicker, launch Electron with `--disable-gpu-compositing`. For source development, run `bun run dev:software-compositing` from `electron/`. This opt-in uses software window compositing while leaving backend CUDA inference available; it does not disable acceleration for other installations. It requires a full Electron restart, not a renderer reload. A refused connection to port 3903 instead means the development proxy is stopped; restart the Electron development process to restore it.
|
||||
|
||||
Secondary workspace sidebars resize from their right edge up to 40% wider than the previous limits (515 / 616 / 750 px by size), while reserving space for the main workspace. Widths are saved per size in the app profile’s local storage and restored on navigation and restart. Double-click the divider to reset the width; focus it and use arrow keys for keyboard resizing. Sidebar sections fill the resized width, and video controls adapt to the player width.
|
||||
|
||||
@@ -55,10 +55,12 @@ transcription.
|
||||
process, so the engine checks up front and reports itself unavailable
|
||||
instead ([#1371](https://github.com/debpalash/VoiceStudio/issues/1371)).
|
||||
pytorch-whisper covers that case on torch's bundled cuDNN 9.
|
||||
- On some hardened Linux kernels the CTranslate2 native library is rejected
|
||||
with "cannot enable executable stack" (an OSError, not an ImportError) —
|
||||
reported as unavailable rather than crashing engine selection
|
||||
([#692](https://github.com/debpalash/VoiceStudio/issues/692)).
|
||||
- On Linux kernels that refuse an executable stack, the CTranslate2 native
|
||||
library (4.4.0 and older) is rejected with "cannot enable executable stack"
|
||||
(an OSError, not an ImportError). VoiceStudio clears that ELF flag in place
|
||||
on first probe so the engine loads; if the file cannot be written it reports
|
||||
itself unavailable with the repair command rather than crashing engine
|
||||
selection ([#692](https://github.com/debpalash/VoiceStudio/issues/692)).
|
||||
- CTranslate2's GPU teardown can rarely segfault the process at unload. If
|
||||
you hit that, switch to the crash-isolated variant —
|
||||
[faster-whisper-isolated](faster-whisper-isolated.md)
|
||||
|
||||
@@ -43,7 +43,8 @@ HF repo id. The env var overrides the persisted UI choice.
|
||||
|
||||
## Behaviour notes
|
||||
|
||||
- Output is 24 kHz mono for most hosted models.
|
||||
- Output is 24 kHz mono. Results from models with a different native rate
|
||||
(such as Dia at 44.1 kHz) are resampled before stitching and export.
|
||||
- **Cloning works only with the `csm` model** — it is the only curated model
|
||||
confirmed to accept a reference clip. Other models silently ignore
|
||||
reference audio, so the engine reports cloning support only when CSM is
|
||||
@@ -74,3 +75,6 @@ See also: [benchmarks.md](../benchmarks.md),
|
||||
[languages.md](../languages.md),
|
||||
[downloading-models.md](../downloading-models.md),
|
||||
[disk usage](disk-usage.md).
|
||||
|
||||
Consecutive chunks with the same native sample rate are resampled together to
|
||||
preserve filter context at chunk boundaries; rate changes start a new group.
|
||||
|
||||
@@ -61,6 +61,12 @@ A 6 GB card with nothing else loaded runs the default model on the GPU
|
||||
([#2041](https://github.com/debpalash/VoiceStudio/issues/2041)). Disable
|
||||
the check with `OMNIVOICE_ASR_VRAM_PREFLIGHT=0`.
|
||||
|
||||
The preflight sizes the weights; the generation workspace grows with the
|
||||
batch on top. If a transcribe still hits a CUDA out-of-memory, the engine
|
||||
steps the batch down (16 → 4 → 1, or 8 → 2 → 1 with word timestamps) and
|
||||
finishes on the CPU rather than dropping the chunk — no silent holes in a
|
||||
dub transcript.
|
||||
|
||||
## Quirks
|
||||
|
||||
- If the pipeline fails to import (`AutoFeatureExtractor` errors), the cause
|
||||
|
||||
@@ -62,8 +62,12 @@ Two more fallback chains run at load time:
|
||||
process fast-fails with no traceback, so the engine is reported unavailable
|
||||
up front and selection falls through to pytorch-whisper, which uses torch's
|
||||
own cuDNN 9 ([#1371](https://github.com/debpalash/VoiceStudio/issues/1371)).
|
||||
- On some hardened Linux kernels CTranslate2's native library is rejected with
|
||||
"cannot enable executable stack" — reported as unavailable, not a crash
|
||||
- On Linux kernels that refuse an executable stack, CTranslate2's native
|
||||
library (4.4.0 and older — what whisperx 3.4.5 pins on Python 3.11) is
|
||||
rejected with "cannot enable executable stack". VoiceStudio now clears that
|
||||
one ELF flag in place on first probe and the engine loads normally; if the
|
||||
library cannot be written (a read-only bundle), the engine reports itself
|
||||
unavailable with the repair command instead of crashing
|
||||
([#692](https://github.com/debpalash/VoiceStudio/issues/692)).
|
||||
- A partially-installed environment (interrupted sync, antivirus quarantine)
|
||||
can break WhisperX's deep import chain (whisperx → pyannote →
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# Features and engines
|
||||
|
||||
Engine availability depends on installed models, hardware, and configured providers.
|
||||
|
||||
## Features
|
||||
|
||||
- **Voice Cloning**
|
||||
- **Voice Design**
|
||||
- **Video Dubbing**
|
||||
- **Dictation Widget**
|
||||
- **Vocal Isolation**
|
||||
- **Speaker Diarization**
|
||||
- **Batch Queue**
|
||||
- **MCP Server**
|
||||
- **AI Watermark**
|
||||
- **Local-first**
|
||||
- **GPU Auto-Detect**
|
||||
- **Remote Model Downloads**
|
||||
- **Extensible**
|
||||
|
||||
## Speech generation
|
||||
|
||||
- **VoiceStudio** (default, powered by k2-fsa/OmniVoice)
|
||||
- omnivoice-subprocess — [Guide](engines/omnivoice-subprocess.md)
|
||||
- CosyVoice 3 — [Guide](engines/cosyvoice.md)
|
||||
- KittenTTS
|
||||
- MLX-Audio
|
||||
- VoxCPM2
|
||||
- MOSS-TTS-Nano
|
||||
- gpt-sovits
|
||||
- sherpa-onnx
|
||||
- **IndexTTS 2.5** ⚡ — [Guide](engines/indextts.md)
|
||||
- omnivoice-gguf
|
||||
- supertonic3
|
||||
- **MOSS-TTS-v1.5** — [Guide](engines/moss-tts-v15.md)
|
||||
- **dots.tts** — [Guide](engines/dots-tts.md)
|
||||
- **Confucius4-TTS** — [Guide](engines/confucius4-tts.md)
|
||||
- pockettts
|
||||
- audiocpp — [Guide](engines/audio-cpp.md)
|
||||
|
||||
## Transcription
|
||||
|
||||
- **WhisperX** (default)
|
||||
- Faster-Whisper
|
||||
- MLX Whisper
|
||||
- PyTorch Whisper
|
||||
- Parakeet TDT
|
||||
- Parakeet TDT v3 (MLX)
|
||||
- Moonshine
|
||||
- FunASR
|
||||
- **sherpa-onnx** (live dictation)
|
||||
- **OpenAI-compatible** ⚠️ configured server
|
||||
@@ -1,3 +1,4 @@
|
||||
catalog: docs/feature-catalog.md
|
||||
# Canonical feature inventory — the single source of truth that the daily
|
||||
# docs-drift job (.github/workflows/docs-drift.yml) diffs against README.md,
|
||||
# docs/, and the engine registries via scripts/check-docs-drift.py.
|
||||
|
||||
@@ -53,8 +53,8 @@ that the app already runs the "fast" preset unless you override it via `/generat
|
||||
| dtype | `torch.float16` hardcoded for the `omnivoice` engine (`model_manager.py`) — correct for Turing (no bf16 tensor cores this generation). No env var override for this engine specifically (ASR engines have `ASR_COMPUTE_TYPE`; `dots_tts`/`indextts` have their own precision vars; `omnivoice` doesn't). |
|
||||
| Attention | `sdpa`, selected automatically since `flash_attn` isn't installed (`_supports_flash_attn_2=True` is declared but the package itself is absent) — safe on T4. |
|
||||
| int8 | No int8 path for this engine (ASR's CTranslate2 `int8` and `sherpa-onnx`'s int8 ONNX models are separate/unrelated). |
|
||||
| CUDA Graphs | No direct API usage in the app. Reachable indirectly via `torch.compile(mode="reduce-overhead")`, which the app attempts **by default** on this GPU (T4/sm_75 isn't in the framework's compile-exclusion list, unlike newer/Blackwell GPUs). The numbers above were measured with `TORCH_COMPILE_DISABLE=1` for a clean eager baseline. |
|
||||
| torch.compile | Attempted by default on T4 (see above) — not evaluated further here. |
|
||||
| CUDA Graphs | **Not used on T4 any more (#2135).** Reachable only indirectly via `torch.compile(mode="reduce-overhead")`, which the app used to attempt by default here — and which killed the backend process outright on the first `/generate` (no traceback, no HTTP response). The app now picks the compile mode per GPU and drops to the non-cudagraph `default` mode below sm_80. `OMNIVOICE_FORCE_CUDAGRAPH=1` restores the old behaviour for benchmarking. |
|
||||
| torch.compile | Still attempted on T4, in `default` mode — compiled Inductor kernels, no graph capture. Disable entirely with Settings → Performance → "Disable torch.compile" or `TORCH_COMPILE_DISABLE=1`. |
|
||||
|
||||
## VRAM
|
||||
|
||||
|
||||
@@ -419,3 +419,7 @@ Two paths are worth persisting across container restarts:
|
||||
[Pull and run (AMD GPU / ROCm)](#pull-and-run-amd-gpu--rocm) above for when
|
||||
to set one by hand.
|
||||
- More entries: [docs/install/troubleshooting.md](troubleshooting.md).
|
||||
|
||||
### CTranslate2 compatibility
|
||||
|
||||
CUDA images include an isolated cuDNN 8 compatibility library directory for WhisperX and faster-whisper alongside the base PyTorch cuDNN 9 runtime. The image build verifies the compatibility libraries exist. ROCm images skip this NVIDIA-only dependency.
|
||||
|
||||
+24
-7
@@ -1,5 +1,22 @@
|
||||
# VoiceStudio — Install on Linux
|
||||
|
||||
## Electron desktop (current)
|
||||
|
||||
From the repository root, install Bun and uv, then run:
|
||||
|
||||
```sh
|
||||
bun install
|
||||
bun run dev
|
||||
```
|
||||
|
||||
Use `bun run desktop-prod` to build and launch Electron, or `bun run dist`
|
||||
to create local installers without publishing. The app manages its backend.
|
||||
See [Electron setup](../../electron/README.md) and [migration notes](../electron-migration.md).
|
||||
|
||||
## Legacy Tauri installation and troubleshooting
|
||||
|
||||
The instructions below apply to the sunset Tauri app and existing Tauri installers.
|
||||
|
||||
This page is self-contained: follow it top to bottom and you'll end up with a
|
||||
working VoiceStudio install on a Debian / Ubuntu / Fedora / Arch host.
|
||||
|
||||
@@ -28,7 +45,7 @@ Everything above, plus the toolchain:
|
||||
`sudo dnf install python3.11` on Fedora, or already installed on Arch.
|
||||
- **Bun** — `curl -fsSL https://bun.sh/install | bash`.
|
||||
- **Rust / Cargo** — `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh` or via your package manager (e.g., `sudo apt install rustc cargo`).
|
||||
If you use rustup, reopen the shell or source `"$HOME/.cargo/env"` before running `bun run desktop-prod`.
|
||||
If you use rustup, reopen the shell or source `"$HOME/.cargo/env"` before running `bun run tauri:desktop-prod`.
|
||||
- **GTK/WebKit deps** for the Tauri shell:
|
||||
|
||||
```bash
|
||||
@@ -66,10 +83,10 @@ git clone https://github.com/debpalash/VoiceStudio.git
|
||||
cd VoiceStudio
|
||||
bun install
|
||||
source "$HOME/.cargo/env" # only needed in a shell opened before rustup finished
|
||||
bun desktop # development build with hot reload
|
||||
bun tauri # development build with hot reload
|
||||
```
|
||||
|
||||
Use `bun run desktop-prod` instead when you need to build and launch the
|
||||
Use `bun run tauri:desktop-prod` instead when you need to build and launch the
|
||||
production bundle. Both commands create the Python environment via `uv`, sync
|
||||
dependencies, and start the backend automatically; do not start the backend in
|
||||
a second terminal.
|
||||
@@ -87,7 +104,7 @@ pkg-config --exists \
|
||||
&& echo "Tauri system libraries are ready"
|
||||
```
|
||||
|
||||
`bun desktop` also checks the native `libxdo` linker input and GStreamer's
|
||||
`bun tauri` also checks the native `libxdo` linker input and GStreamer's
|
||||
`autoaudiosink` before starting. The latter is required even if you do not plan
|
||||
to record: WebKitGTK 2.52 aborts its renderer when a page creates an audio
|
||||
element without that plugin, which otherwise turns a running app blank. The
|
||||
@@ -256,7 +273,7 @@ If you are on v0.4.0 or older, either update or build from source:
|
||||
git clone https://github.com/debpalash/VoiceStudio.git
|
||||
cd VoiceStudio
|
||||
bun install
|
||||
bun run desktop-prod
|
||||
bun run tauri:desktop-prod
|
||||
```
|
||||
|
||||
Tracking issues: [#62](https://github.com/debpalash/VoiceStudio/issues/62),
|
||||
@@ -389,9 +406,9 @@ reinstall and left the CPU-only CUDA build in place).
|
||||
**2. Environment variable (existing installs / headless / source).** Set
|
||||
`OMNIVOICE_TORCH_VARIANT=rocm` before launching — the next bootstrap performs
|
||||
the same ROCm reinstall. Source installs honour it too:
|
||||
`OMNIVOICE_TORCH_VARIANT=rocm bun run desktop` swaps torch right after
|
||||
`OMNIVOICE_TORCH_VARIANT=rocm bun run tauri` swaps torch right after
|
||||
`uv sync` and launches the backend without re-syncing, so the wheel is not
|
||||
reverted on the next start (#1665). Without the variable, `bun run desktop`
|
||||
reverted on the next start (#1665). Without the variable, `bun run tauri`
|
||||
restores the lockfile's CUDA build — a hand-swapped ROCm wheel does not
|
||||
survive it. `OMNIVOICE_TORCH_INDEX=<url>` overrides the wheel
|
||||
index when you need a different ROCm version — e.g. AMD publishes newer
|
||||
|
||||
+19
-2
@@ -1,5 +1,22 @@
|
||||
# VoiceStudio — Install on macOS
|
||||
|
||||
## Electron desktop (current)
|
||||
|
||||
From the repository root, install Bun and uv, then run:
|
||||
|
||||
```sh
|
||||
bun install
|
||||
bun run dev
|
||||
```
|
||||
|
||||
Use `bun run desktop-prod` to build and launch Electron, or `bun run dist`
|
||||
to create local installers without publishing. The app manages its backend.
|
||||
See [Electron setup](../../electron/README.md) and [migration notes](../electron-migration.md).
|
||||
|
||||
## Legacy Tauri installation and troubleshooting
|
||||
|
||||
The instructions below apply to the sunset Tauri app and existing Tauri installers.
|
||||
|
||||
This page is self-contained: follow it top to bottom and you'll end up with a
|
||||
working VoiceStudio install on macOS (Apple Silicon).
|
||||
|
||||
@@ -36,7 +53,7 @@ Everything above, plus the toolchain:
|
||||
- **Python 3.11+** — `brew install python@3.11` (or use `pyenv` / the system Python if you already have ≥3.11).
|
||||
- **Bun** — `curl -fsSL https://bun.sh/install | bash`.
|
||||
- **Rust / Cargo** — `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh` or `brew install rust`.
|
||||
If you use rustup, reopen the terminal or source `"$HOME/.cargo/env"` before running `bun run desktop-prod`.
|
||||
If you use rustup, reopen the terminal or source `"$HOME/.cargo/env"` before running `bun run tauri:desktop-prod`.
|
||||
|
||||
FFmpeg/FFprobe and yt-dlp are **not** prerequisites on any install path: the
|
||||
app resolves them itself (a static build ships with the Python environment;
|
||||
@@ -63,7 +80,7 @@ Or manually:
|
||||
git clone https://github.com/debpalash/VoiceStudio.git
|
||||
cd VoiceStudio
|
||||
bun install
|
||||
bun run desktop-prod
|
||||
bun run tauri:desktop-prod
|
||||
```
|
||||
|
||||
The first launch builds the Tauri shell, creates the Python venv via `uv`,
|
||||
|
||||
+108
-17
@@ -248,6 +248,42 @@ peak memory footprint that exceeds free VRAM. Windows-only quirk.
|
||||
|
||||
**Linked issue:** [#65](https://github.com/debpalash/VoiceStudio/issues/65)
|
||||
|
||||
## 5a. Backend dies on the first `/generate` (older NVIDIA GPUs, e.g. Tesla T4)
|
||||
|
||||
**Symptom:** the backend starts fine, `/health` reports your GPU, the model
|
||||
preloads — and then the first generation request returns
|
||||
`RemoteDisconnected: Remote end closed connection without response`. Every call
|
||||
after it gets `ConnectionRefused`, because the backend process is gone. No
|
||||
Python traceback is printed.
|
||||
|
||||
**Cause:** `torch.compile(mode="reduce-overhead")` captures CUDA graphs. On
|
||||
pre-Ampere cards (Turing sm_75 / Volta sm_70 — the Tesla T4 on Google Colab is
|
||||
the common case) that capture can abort the process from inside the native CUDA
|
||||
library. It happens below the interpreter, so no `except` in the app can catch
|
||||
it and nothing is logged.
|
||||
|
||||
**Fix:** update — VoiceStudio now selects the compile mode per GPU and does not
|
||||
capture CUDA graphs below sm_80, so this should no longer happen. If you still
|
||||
see a crash in the generate path on any GPU, turn compilation off entirely:
|
||||
|
||||
- **In the app:** Settings → Performance → **"Disable torch.compile"**.
|
||||
- **From the CLI / from source:** `TORCH_COMPILE_DISABLE=1` before launching.
|
||||
This is honoured on every platform and by every engine, in-process or
|
||||
sidecar.
|
||||
|
||||
**Getting a traceback:** the backend now arms `faulthandler`, so a native crash
|
||||
writes the faulting thread's Python stack to `backend_err.log` on the way down.
|
||||
Include that stack when reporting — without it a native crash is unattributable.
|
||||
(`OMNIVOICE_DISABLE_FAULTHANDLER=1` turns it off.)
|
||||
|
||||
**Extra containment:** to keep a crashing engine from taking the API down with
|
||||
it, run the engine in a killable child process — select
|
||||
**OmniVoice (subprocess-isolated)** in Settings → Engines, or
|
||||
`OMNIVOICE_TTS_BACKEND=omnivoice-subprocess`. The parent then returns an HTTP
|
||||
error and respawns the sidecar instead of dying.
|
||||
|
||||
**Linked issue:** [#2135](https://github.com/debpalash/VoiceStudio/issues/2135)
|
||||
|
||||
## 5b. RTX 50-series (Blackwell, sm_120): backend crashes during `ml_imports`
|
||||
|
||||
**Symptom:** on an RTX 5070 / 5070 Ti / 5080 / 5090, the backend never becomes
|
||||
@@ -255,15 +291,29 @@ ready. The desktop app sits on "starting backend", `/health` returns 503, and
|
||||
`/startup/progress` shows `ml_imports` active. From source you see `import
|
||||
torch` die with a native access violation rather than a Python traceback.
|
||||
|
||||
**Cause:** VoiceStudio pins `torch 2.8.0`. That build carries no `sm_120`
|
||||
kernels, so on a Blackwell card the CUDA initializer faults inside the native
|
||||
library. This is not a VoiceStudio bug and no setting works around it — the
|
||||
wheel does not contain code for the GPU.
|
||||
**Cause:** not established. The pinned build is not missing Blackwell code:
|
||||
`torch 2.8.0+cu128` lists `sm_120` in `torch.cuda.get_arch_list()`, and that
|
||||
build imports and runs CUDA normally on some Blackwell cards. What is confirmed
|
||||
is that on the Windows setups in
|
||||
[#1931](https://github.com/debpalash/VoiceStudio/issues/1931) `import torch`
|
||||
faults inside the native library before Python can raise an error, and moving
|
||||
the torch trio to 2.9.x clears it.
|
||||
|
||||
**Fix:** move the whole torch trio to a build with `sm_120` kernels. They must
|
||||
move together — upgrading one past the ABI the others were built against gives
|
||||
you `RuntimeError: operator torchvision::nms does not exist`, which is the
|
||||
next section's problem instead.
|
||||
If `import torch` crashes for you, that crash *is* the symptom — skip straight
|
||||
to the fix below. Where torch does import, this shows what the build actually
|
||||
contains:
|
||||
|
||||
```bash
|
||||
uv run python -c "import torch; print(torch.__version__, torch.cuda.get_arch_list())"
|
||||
```
|
||||
|
||||
`sm_120` in that list means the kernels are present and the crash is elsewhere
|
||||
in the native init path. Either way the upgrade below is the known workaround.
|
||||
|
||||
**Fix:** move the whole torch trio to 2.9.x. They must move together —
|
||||
upgrading one past the ABI the others were built against gives you
|
||||
`RuntimeError: operator torchvision::nms does not exist`, which is the next
|
||||
section's problem instead.
|
||||
|
||||
Edit **both** pin lists, keeping them identical:
|
||||
|
||||
@@ -305,8 +355,10 @@ guarded now, so the upgrade path above is clean on a current checkout.
|
||||
**Keeping the change:** these are the repo's own pins, so a `git pull` that
|
||||
touches them will conflict or overwrite. Re-apply after updating until the
|
||||
default pin moves — the default cannot move for everyone until the newer torch
|
||||
is verified across the older GPUs VoiceStudio supports, since a build that adds
|
||||
`sm_120` can drop older architectures.
|
||||
is verified across the older GPUs VoiceStudio supports, since a newer build can
|
||||
drop older architectures. Which ones varies by torch release, not by the CUDA
|
||||
variant alone: the pinned 2.8.0+cu128 build reports `sm_70` first, while the
|
||||
cu128 arch list captured in #1285 still carried `sm_61`.
|
||||
|
||||
**Linked issue:** [#1931](https://github.com/debpalash/VoiceStudio/issues/1931)
|
||||
— thanks to the reporter for the full diagnosis, including the verification
|
||||
@@ -1055,14 +1107,16 @@ retrying.
|
||||
and the backend log ends inside the `ml_imports` phase — often with a native
|
||||
crash (exit code `0xffffffff` / `-1073741819`) rather than a Python traceback.
|
||||
|
||||
**Cause.** VoiceStudio pins `torch 2.8.0+cu128`, which ships no `sm_120`
|
||||
kernels. On an RTX 50-series card `import torch` dies natively, before any
|
||||
VoiceStudio code can classify it — which is why the app can only say the
|
||||
backend did not start. This is a property of the pinned build, not of your
|
||||
driver or your install.
|
||||
**Cause.** Not established. `torch 2.8.0+cu128` does contain Blackwell code:
|
||||
`sm_120` is in `torch.cuda.get_arch_list()`, and that build imports and runs
|
||||
CUDA normally on some Blackwell cards, so this is not simply a wheel without
|
||||
kernels for your GPU. What is confirmed is that on the Windows setups in
|
||||
[#1931](https://github.com/debpalash/VoiceStudio/issues/1931) `import torch`
|
||||
dies natively before any VoiceStudio code can classify it, which is why the app
|
||||
can only say the backend did not start. Moving to torch 2.9.x clears it for the
|
||||
users who hit it.
|
||||
|
||||
**Fix.** Move to a torch build that has Blackwell kernels. From a source
|
||||
checkout, in the project folder:
|
||||
**Fix.** Move to torch 2.9.x. From a source checkout, in the project folder:
|
||||
|
||||
1. Edit `pyproject.toml` → `[tool.uv] constraint-dependencies` and raise the
|
||||
torch constraint to `torch==2.9.1+cu128` (matching `torchaudio` /
|
||||
@@ -1112,3 +1166,40 @@ remove the app binary itself are in
|
||||
[docs/install/uninstall.md](uninstall.md).
|
||||
|
||||
**Linked issue:** [#1089](https://github.com/debpalash/VoiceStudio/issues/1089)
|
||||
|
||||
### Pedalboard illegal-instruction crashes
|
||||
|
||||
VoiceStudio pins pedalboard to `>=0.9.14,<0.9.21` while [upstream portable-wheel repair #466](https://github.com/spotify/pedalboard/pull/466) remains open. Re-sync the locked environment after updating from a build with newer affected wheels. This keeps the existing effects API floor while avoiding the reported Linux CPU import crash.
|
||||
|
||||
### TorchCodec unavailable
|
||||
|
||||
When torchaudio requires an unavailable TorchCodec installation, VoiceStudio writes through soundfile and reads reference audio through its FFmpeg fallback. Reference amplitude is normalized using the decoded sample representation, including 8-, 24-, and 32-bit PCM.
|
||||
|
||||
### Isolated engine timeouts
|
||||
|
||||
Generation has a separate deadline from health checks. Default sidecar deadlines scale with text length and the host execution budget; per-engine timeout overrides remain supported. The outer job guard includes time for sidecar termination and error reporting. A timeout identifies the deadline, while a closed pipe without a timeout indicates a crash.
|
||||
|
||||
Per-engine receive overrides include `OMNIVOICE_CONFUCIUS4_RECV_TIMEOUT_S`, `OMNIVOICE_DOTS_TTS_RECV_TIMEOUT_S`, `OMNIVOICE_MOSS_TTS_V15_RECV_TIMEOUT_S`, and `OMNIVOICE_SUPERTONIC3_RECV_TIMEOUT_S` (seconds). Invalid or non-finite values use the default; values below 30 seconds are raised to 30.
|
||||
|
||||
### FFprobe alongside FFmpeg
|
||||
|
||||
When FFprobe is not on PATH, VoiceStudio also checks beside the selected FFmpeg binary. Parent folders named `ffmpeg` remain unchanged; only the executable name becomes `ffprobe` (or `ffprobe.exe` on Windows).
|
||||
|
||||
### Subtitle and manuscript encodings
|
||||
|
||||
Electron and web imports accept UTF-8, UTF-16 with a byte-order mark, and Windows-1252 text. The same decoder rules apply to uploaded subtitles and audiobook manuscripts; legacy punctuation is preserved.
|
||||
|
||||
|
||||
### Compile fallback after startup
|
||||
|
||||
An architecture accepted by the torch.compile preflight may still encounter independent Dynamo, Inductor, Triton, or CUDA-graph runtime errors. VoiceStudio distinguishes those from GPU memory exhaustion and retries with eager execution; architecture support alone does not guarantee compilation succeeds.
|
||||
|
||||
### Native engine installation from a remote client
|
||||
|
||||
Sidecar and audio.cpp runtime installation is restricted to requests from the backend computer's loopback interface. An API key does not bypass this restriction. The catalogue now shows local setup guidance instead of offering a remote install that will be rejected. Open the backend through `localhost` on that computer, or follow the engine's setup guide there. In Docker, bridge-network requests may not be loopback even when the browser runs on the host; use the documented container setup rather than weakening the native-install gate.
|
||||
|
||||
### Dubbing extraction fails
|
||||
|
||||
Extraction errors show the FFmpeg exit code and the end of its diagnostics, with private paths scrubbed. Use the final error line to distinguish missing audio streams, unsupported inputs, permissions, or disk errors. A version banner alone does not identify the cause; include the final diagnostic and source format when reporting a failure.
|
||||
|
||||
Explicit generation budgets remain authoritative. If an outer TTS/ASR guard times out or its caller disconnects, the active sidecar receive kills and reaps its captured child; it cannot terminate a later retry. In-process inference keeps its existing lifetime accounting until the native call returns.
|
||||
|
||||
+38
-14
@@ -1,5 +1,22 @@
|
||||
# VoiceStudio — Install on Windows
|
||||
|
||||
## Electron desktop (current)
|
||||
|
||||
From the repository root, install Bun and uv, then run:
|
||||
|
||||
```sh
|
||||
bun install
|
||||
bun run dev
|
||||
```
|
||||
|
||||
Use `bun run desktop-prod` to build and launch Electron, or `bun run dist`
|
||||
to create local installers without publishing. The app manages its backend.
|
||||
See [Electron setup](../../electron/README.md) and [migration notes](../electron-migration.md).
|
||||
|
||||
## Legacy Tauri installation and troubleshooting
|
||||
|
||||
The instructions below apply to the sunset Tauri app and existing Tauri installers.
|
||||
|
||||
This page is self-contained: follow it top to bottom and you'll end up with a
|
||||
working VoiceStudio install on Windows 10 / 11 (x64).
|
||||
|
||||
@@ -20,7 +37,7 @@ by the app itself on first launch. No toolchain needed.
|
||||
Everything above, plus the toolchain:
|
||||
|
||||
- **Git for Windows** — `winget install --id Git.Git -e`. Needed for
|
||||
`git clone`, and it includes **Git Bash**, which `bun run desktop-prod`
|
||||
`git clone`, and it includes **Git Bash**, which `bun run tauri:desktop-prod`
|
||||
uses to run its build-and-launch script. Without it, `desktop-prod` stops
|
||||
with an error telling you to install it.
|
||||
- **Python 3.11+** — `winget install Python.Python.3.11` (or download from
|
||||
@@ -31,8 +48,8 @@ Everything above, plus the toolchain:
|
||||
with the **"Desktop development with C++"** workload checked.
|
||||
- **Bun** — `powershell -c "irm bun.sh/install.ps1 | iex"`.
|
||||
- **FFmpeg** — `winget install Gyan.FFmpeg`.
|
||||
- **Rust / Cargo** — `winget install Rust.Rustup` or download `rustup-init.exe` from [rustup.rs](https://rustup.rs/).
|
||||
After installing Rustup, close and reopen PowerShell before running `bun run desktop-prod`.
|
||||
- **Rust / Cargo** — `winget install Rustlang.Rustup` or download `rustup-init.exe` from [rustup.rs](https://rustup.rs/).
|
||||
After installing Rustup, close and reopen PowerShell before running `bun run tauri:desktop-prod`.
|
||||
|
||||
## GPU support on Windows
|
||||
|
||||
@@ -64,17 +81,17 @@ Or manually:
|
||||
git clone https://github.com/debpalash/VoiceStudio.git
|
||||
cd VoiceStudio
|
||||
bun install
|
||||
bun run desktop-prod
|
||||
bun run tauri:desktop-prod
|
||||
```
|
||||
|
||||
The first launch creates the Python venv via `uv`, syncs deps, and downloads
|
||||
model weights. The splash screen shows progress.
|
||||
|
||||
> **Note:** `bun run desktop-prod` runs a bash script under the hood. You can
|
||||
> **Note:** `bun run tauri:desktop-prod` runs a bash script under the hood. You can
|
||||
> launch it from PowerShell or cmd as shown — it finds Git Bash automatically
|
||||
> (installed with Git for Windows, see Prerequisites). If no Git Bash is
|
||||
> found, it prints instructions instead of failing silently. Alternatives
|
||||
> that don't need bash: `bun run desktop` (dev mode) or the pre-built MSI
|
||||
> that don't need bash: `bun run tauri` (dev mode) or the pre-built MSI
|
||||
> below.
|
||||
|
||||
## Install (pre-built MSI)
|
||||
@@ -274,21 +291,25 @@ synthesise call. On machines with <16 GB VRAM, that compile step can OOM
|
||||
failed`.
|
||||
|
||||
**The one-click fix:** open **Settings → Performance** in the app and toggle
|
||||
**"Disable torch.compile (Windows)"** on. That sets the
|
||||
`TORCH_COMPILE_DISABLE=1` env var on every engine subprocess VoiceStudio spawns,
|
||||
which falls back to the eager-mode kernel path. You'll lose a few percent of
|
||||
peak throughput in exchange for the engine actually loading.
|
||||
**"Disable torch.compile"** on. That sets the `TORCH_COMPILE_DISABLE=1` env var
|
||||
on every engine subprocess VoiceStudio spawns and forces the in-process engine
|
||||
to eager mode as well. You'll lose a few percent of peak throughput in exchange
|
||||
for the engine actually loading.
|
||||
|
||||
**From the CLI / from source:** set the env var manually before launching:
|
||||
|
||||
```powershell
|
||||
$env:TORCH_COMPILE_DISABLE = "1"
|
||||
bun run desktop-prod
|
||||
bun run tauri:desktop-prod
|
||||
```
|
||||
|
||||
This setting is a no-op on macOS and Linux (the OOM is Windows-specific —
|
||||
the `torch.compile` kernel cache behaves differently on the other platforms).
|
||||
Tracking issue: [#65](https://github.com/debpalash/VoiceStudio/issues/65).
|
||||
The OOM this section describes is Windows-specific, but the toggle itself works
|
||||
on **every** platform — it used to be greyed out elsewhere, which left Linux and
|
||||
macOS users with no way to switch off a `torch.compile` that was breaking their
|
||||
engine. Tracking issues:
|
||||
[#65](https://github.com/debpalash/VoiceStudio/issues/65) (this OOM) and
|
||||
[#2135](https://github.com/debpalash/VoiceStudio/issues/2135) (the same toggle
|
||||
on Linux/CUDA).
|
||||
|
||||
## Hugging Face token (optional but recommended)
|
||||
|
||||
@@ -329,3 +350,6 @@ This test-host preparation does not change installer privileges or user machines
|
||||
Verbose MSI logs are printed if installation or removal fails. The Windows CI
|
||||
job also rejects an invalid MSI and verifies policy absence, value types, account
|
||||
cleanup, and verbose failure logs using Windows PowerShell 5.1.
|
||||
|
||||
|
||||
Migration configuration is kept ASCII so Alembic can read it under Windows locale code pages as well as UTF-8. This applies to source installs and direct Alembic commands.
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
# Integration directory
|
||||
|
||||
Directory entries are illustrative, not paid sponsors, endorsements, or verified VoiceStudio integrations. Icons are bundled locally so viewing the catalog sends no logo requests to providers. Brand marks belong to their respective owners.
|
||||
|
||||
| Company | Official source | Icon source |
|
||||
|---|---|---|
|
||||
| Twilio | [Website](https://www.twilio.com) | Bundled site icon |
|
||||
| Plivo | [Website](https://www.plivo.com) | Bundled site icon |
|
||||
| Telnyx | [Website](https://telnyx.com) | Bundled site icon |
|
||||
| n8n | [Website](https://n8n.io) | Bundled site icon |
|
||||
| Zapier | [Website](https://zapier.com) | Bundled site icon |
|
||||
| Make | [Website](https://www.make.com) | Bundled generic mark |
|
||||
| GitHub | [Website](https://github.com) | Bundled site icon |
|
||||
| GitHub Container Registry | [Website](https://ghcr.io) | Bundled GitHub icon |
|
||||
| Docker | [Website](https://www.docker.com) | Bundled site icon |
|
||||
| Model Context Protocol | [Website](https://modelcontextprotocol.io) | Bundled site icon |
|
||||
| OpenAI Agents | [Guide](https://platform.openai.com/docs/guides/agents) | Bundled local mark |
|
||||
| Claude Code | [Guide](https://docs.anthropic.com/en/docs/claude-code) | Bundled site icon |
|
||||
| Codex CLI | [Repository](https://github.com/openai/codex) | Bundled local mark |
|
||||
| VoiceStudio API | [Repository](https://github.com/debpalash/VoiceStudio) | Bundled local mark |
|
||||
@@ -0,0 +1,19 @@
|
||||
# README media
|
||||
|
||||
Captured from the Electron renderer on Linux, September 16, 2026. These images show the development branch, not a claim about a published release. The browser capture runs the same renderer as Electron; native window decorations are excluded.
|
||||
|
||||
Only the bundled demo voice appears. Personal profiles, history, and projects are filtered from the capture context, and API mutations are blocked. The normal app and its local storage are left alone.
|
||||
|
||||
With the Electron development server running:
|
||||
|
||||
```bash
|
||||
CHROMIUM_PATH=/usr/bin/chromium node scripts/capture-readme-electron.mjs
|
||||
```
|
||||
|
||||
The script writes PNG screenshots here and prints the temporary WebM path. Convert that recording to the main GIF (replace `recording.webm` with that path):
|
||||
|
||||
```bash
|
||||
ffmpeg -y -ss 1 -i recording.webm -vf 'fps=8,scale=1120:-1:flags=lanczos,split[s0][s1];[s0]palettegen=stats_mode=diff[p];[s1][p]paletteuse=dither=bayer:bayer_scale=3' -loop 0 docs/media/electron/voicestudio.gif
|
||||
```
|
||||
|
||||
The README uses the GIF plus the cloning and dubbing screenshots. Design and model screenshots are captured as companion stills. The official logo and repository badges retain their existing assets.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 330 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 296 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 299 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 362 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 4.3 MiB |
@@ -0,0 +1,10 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="640" height="104" viewBox="0 0 640 104">
|
||||
<title>Your brand — partner with VoiceStudio</title>
|
||||
<desc>Apply for a paid featured placement in the app, integrations directory, and README.</desc>
|
||||
<rect x="1" y="1" width="638" height="102" rx="16" fill="#191420" stroke="#574051"/>
|
||||
<circle cx="52" cy="52" r="23" fill="#30212e" stroke="#d3869b" stroke-width="1.5"/>
|
||||
<text x="52" y="60" text-anchor="middle" fill="#f5eaf2" font-family="Arial,sans-serif" font-size="26" font-weight="700">?</text>
|
||||
<text x="94" y="43" fill="#f5eaf2" font-family="Arial,sans-serif" font-size="19" font-weight="600">Your brand, where people build with voice.</text>
|
||||
<text x="94" y="69" fill="#c0adbd" font-family="Arial,sans-serif" font-size="13">App placement · Integration page · README exposure</text>
|
||||
<path d="M590 61l18-18m-15 0h15v15" fill="none" stroke="#d3869b" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 982 B |
+14
-3
@@ -102,9 +102,20 @@ where the runtime check says it can work (a CUDA device with Triton importable
|
||||
and a supported GPU architecture) and skipped automatically everywhere else —
|
||||
MPS, CPU, and the typical Windows install (Triton ships no Windows wheel).
|
||||
The one user-facing control is Settings → Performance → "Disable
|
||||
torch.compile" (shown on Windows), for the rare setup where a partial Triton
|
||||
install makes the probe pass but the compile attempt itself crash — see
|
||||
[Windows install notes](install/windows.md).
|
||||
torch.compile", available on every platform, for the setup where the probe
|
||||
passes but the compile attempt itself misbehaves — a partial Triton install,
|
||||
or a GPU whose compiled kernels crash the engine. Setting
|
||||
`TORCH_COMPILE_DISABLE=1` (or `TORCHDYNAMO_DISABLE=1`) in the environment does
|
||||
the same thing and is honoured by both the in-process engine and every engine
|
||||
subprocess. See [Windows install notes](install/windows.md).
|
||||
|
||||
On CUDA the compile **mode** is chosen per GPU: Ampere (sm_80) and newer use
|
||||
`reduce-overhead`, which captures CUDA graphs; older cards (Turing/Volta, e.g.
|
||||
the Tesla T4) fall back to the plain `default` mode, because graph capture was
|
||||
observed to abort the whole backend process there
|
||||
([#2135](https://github.com/debpalash/VoiceStudio/issues/2135)). They still get
|
||||
compiled Inductor kernels. `OMNIVOICE_FORCE_CUDAGRAPH=1` restores the
|
||||
cudagraph mode if you want to benchmark it.
|
||||
|
||||
## Warnings before a slow generation
|
||||
|
||||
|
||||
@@ -396,3 +396,9 @@ would disrupt the machine or network, including airplane mode, simultaneous
|
||||
downloads, and stopping a worker during an audiobook, are printed as exact
|
||||
`MANUAL` steps and are never reported as passed automatically. A failed
|
||||
precondition or automated check exits non-zero.
|
||||
|
||||
Remote compute targets show available CPU/GPU usage and free VRAM. Unavailable
|
||||
metrics are omitted; a transient sampling failure retains the last successful
|
||||
reading. Telemetry runs off the control loop with at most one probe per worker
|
||||
client, retained across reconnects. Read-only probes never block task draining or
|
||||
shutdown; a stuck driver probe cannot accumulate more threads.
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
# Support page
|
||||
|
||||
The support page puts the monthly development goal, donation amounts, and Ko-fi / PayPal links first. Selecting an amount carries it into PayPal; Ko-fi lets you choose the amount on its own page. No checkout opens until you choose a provider.
|
||||
|
||||
Star and community links offer other ways to help. Sponsors remain visible. The Electron page uses the official VoiceStudio logo, a single donation panel, visible sponsor and Pro cards, and an icon grid for contact channels. No accordion hides those actions. Controls support keyboard navigation, and decorative interaction animations respect reduced-motion preferences.
|
||||
|
||||
Workspace headers link to Support immediately before Search. A sponsor footer sits below each workspace content area, outside its scrolling editor and above the agent dock. It reads the shared sponsor roster, uses themed hover/focus tooltips, and opens sponsor links in the system browser. With an empty roster, one combined “Your logo here” booking tile demonstrates the placement and opens the sponsorship message form. It prepares a mailto draft to partner@voicestudio.sh in the default email app, or copies the address; it never sends email itself.
|
||||
|
||||
The sponsor-bar remove control opens the Free vs Pro comparison on Support. The comparison lists the proposed Pro benefits: no telemetry, a hideable sponsor bar, a Pro badge, and advanced tools. Activation verification and the specific advanced-tool list are not configured; no Pro entitlement is inferred from donations or local preferences.
|
||||
|
||||
Sponsor tiles form a left-aligned, horizontally scrolling row with 1px gaps. The rightmost combined logo-plus tile opens the booking form.
|
||||
|
||||
Logo hover cards match their trigger tile width and grow vertically to fit their contents.
|
||||
|
||||
The footer chevron opens a searchable sponsor catalog above the strip. Cards show logos, names, tiers, and destination links; a booking card opens the email form. The panel scrolls within 60% of the viewport. Escape or the close button collapses it and returns focus to the chevron.
|
||||
|
||||
The expanded catalog is labeled Integrations. Entries from the sponsored roster display a Featured badge in their catalog card and hover card; the empty booking preview does not.
|
||||
|
||||
Ten voice-AI company examples populate the catalog and compact strip using locally bundled official icons. They carry a Directory example label, not Featured. Capabilities and source links are recorded in [the directory notes](integration-directory.md).
|
||||
+15
-17
@@ -1,19 +1,11 @@
|
||||
# VoiceStudio — Electron shell (preview)
|
||||
# VoiceStudio — Electron desktop app
|
||||
|
||||
An Electron rewrite of the desktop shell, built page by page. Today it ships
|
||||
**Voice cloning** only; the Tauri app in `frontend/` remains the product.
|
||||
Electron is the primary desktop app for voice cloning, stories, dubbing,
|
||||
transcription, voice design, and workflows. Tauri is retained only for its final
|
||||
sunset update; see [migration notes](../docs/electron-migration.md).
|
||||
|
||||
Both shells talk to the same local FastAPI backend (`backend/`, port 3900), so
|
||||
voices, history and installed engines are shared. Nothing leaves the machine.
|
||||
|
||||
The Electron UI follows T3 Code's styling foundation: shadcn Base UI Mira,
|
||||
Zinc light surfaces, near-black dark surfaces, blue actions, system fonts,
|
||||
compact controls, and translucent popovers/dialogs. Shared palette roles live in
|
||||
`src/renderer/src/styles/t3-theme.css`; app geometry and surface utilities live in
|
||||
`styles/globals.css`. The palette is adapted from
|
||||
[T3 Code](https://github.com/pingdotgg/t3code/blob/main/apps/web/src/index.css)
|
||||
under the MIT license (see `T3CODE-LICENSE.txt`).
|
||||
Light/dark switching is local; T3's theme editor and theme library are not included.
|
||||
The runtime supervisor manages the local FastAPI backend. Network integrations
|
||||
and remote workers require configuration; local generation stays on your machine.
|
||||
|
||||
## Stack
|
||||
|
||||
@@ -28,7 +20,7 @@ Light/dark switching is local; T3's theme editor and theme library are not inclu
|
||||
## Run it
|
||||
|
||||
```sh
|
||||
cd electron
|
||||
# From the repository root
|
||||
bun install
|
||||
bun run dev # electron-vite: main + preload + renderer with HMR
|
||||
```
|
||||
@@ -57,7 +49,7 @@ app-relative `/api/...`:
|
||||
|
||||
```sh
|
||||
bun run typecheck # tsgo, both projects
|
||||
bun run check # vp: format + lint + types
|
||||
bun run check:electron # types, tests, build, packaging contract
|
||||
bun run test # vitest (jsdom)
|
||||
bun run build # electron-vite build → out/
|
||||
bun run dist # + electron-builder → release/
|
||||
@@ -123,7 +115,7 @@ Appearance and General have direct routes and share a breadcrumb header, searcha
|
||||
sidebar, max-w-4xl scroll frame, grouped sections, and consistent setting rows.
|
||||
Sidebar active/hover surfaces use the shared T3 theme tokens.
|
||||
|
||||
The local palette library includes Signal, Canopy, Current, Hearth, and Orchid, with
|
||||
The local palette library includes VoiceStudio Original, Canopy, Current, Hearth, and Orchid, with
|
||||
upstream light/dark color definitions with VoiceStudio display names from T3 Code (MIT). Each appearance keeps
|
||||
its own selected palette. System mode follows live OS appearance changes; the
|
||||
sidebar toggle explicitly switches to light or dark. Choices persist under
|
||||
@@ -206,3 +198,9 @@ open a form in the browser; they do not publish a voice automatically.
|
||||
Saved voice editor > Export persona downloads a portable `.ovsvoice` bundle.
|
||||
Include voice clip controls whether the original reference accompanies the
|
||||
watermarked preview. Gallery > My Imports accepts the exported bundle again.
|
||||
|
||||
Workspace navigation groups Clone, Design, Profiles, and Gallery under Voice;
|
||||
Stories and Audiobook under Stories; and single/batch dubbing under Dubbing.
|
||||
The current workflow opens automatically. Group buttons can expand or collapse
|
||||
without navigating; the compact rail opens the same destinations in a flyout.
|
||||
Transcribe, Projects, Tools, and Integrations remain directly accessible.
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user