fix: integrate current main and finish review requirements for #2080
This commit is contained in:
Palash Debnath
@@ -0,0 +1,29 @@
|
||||
# VoiceStudio compatibility entry
|
||||
|
||||
The current cross-agent package is [voicestudio](../../../../skills/voicestudio/SKILL.md).
|
||||
For new installations use `npx skills add debpalash/VoiceStudio --skill voicestudio`.
|
||||
|
||||
Use the running backend at the user's configured address (default
|
||||
`http://localhost:3900`). Check `/health`, discover `/openapi.json` and
|
||||
`/v1/audio/voices`, then use the installed schema for speech, transcription,
|
||||
profiles, and jobs. The HTTP MCP endpoint is `/mcp`; discover tools from the
|
||||
connected server instead of assuming this older package's tool inventory.
|
||||
|
||||
Launch the installed Electron app if the backend is unavailable. For source
|
||||
development follow the checkout's Electron README. Existing helpers in
|
||||
`scripts/` support legacy source installations; inspect their environment
|
||||
and dependency assumptions before running them.
|
||||
|
||||
Model downloads and remote services require the user's choice. Never silently
|
||||
install models, promise fixed latency, or treat compatibility voice names as
|
||||
real provider voices. Validate saved audio and asynchronous job completion
|
||||
before reporting success. Protected backends require configured credentials;
|
||||
never disable authentication to make an example work.
|
||||
|
||||
Source and current setup documentation:
|
||||
https://github.com/debpalash/VoiceStudio
|
||||
|
||||
This archived entry is not an installable skill. Existing installations should
|
||||
remove the old `omnivoice` / `oss-maintainer` entries and install `voicestudio` /
|
||||
`voicestudio-maintainer` from the canonical repository. Legacy helpers remain
|
||||
for existing users; the Electron supervisor is the preferred launcher.
|
||||
@@ -1,172 +0,0 @@
|
||||
---
|
||||
name: omnivoice
|
||||
description: "Local TTS, voice cloning, voice design, and video dubbing via the VoiceStudio MCP server (open-source ElevenLabs alternative; nothing leaves the machine, runs on MPS/CUDA/CPU). Use when: (1) generating speech from text in any of 646 languages, (2) cloning a voice from a 3-second reference clip, (3) designing a voice by gender/age/accent/pitch/style, (4) dubbing a video into another language, (5) listing voice profiles or personality presets, (6) producing narration where privacy, cost, or absent API keys matter, (7) non-English narration where Edge TTS/kokoro fall short, (8) batch audio for blog posts or content pipelines. Triggers: 'omnivoice', 'voice clone', 'clone this voice', 'tts', 'narrate', 'generate speech', 'voice synthesis', 'dub video', 'voice design', 'local tts', 'multilingual voice', 'narrate this post', 'elevenlabs alternative'."
|
||||
---
|
||||
|
||||
# VoiceStudio
|
||||
|
||||
The canonical cross-agent package lives at `skills/omnivoice/SKILL.md`. This
|
||||
Claude-specific package retains the MCP lifecycle helpers and references.
|
||||
|
||||
## Overview
|
||||
|
||||
Generate audio locally via the VoiceStudio MCP server. Tools: `generate_speech`, `list_voices`, `list_personalities`, `list_languages`, `check_health`. Resources: `voice://{id}`, `history://recent`.
|
||||
|
||||
## Prerequisites — Backend Must Be Running
|
||||
|
||||
The MCP tools all hit `$OMNIVOICE_API_URL` (default `http://localhost:3900`). If the backend is down, every tool returns a connection error. Install + boot:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/debpalash/VoiceStudio.git "$OMNIVOICE_HOME"
|
||||
cd "$OMNIVOICE_HOME"
|
||||
uv sync
|
||||
VIRTUAL_ENV="$(pwd)/.venv" uv pip install 'mcp[cli]'
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
```bash
|
||||
scripts/check-health.sh # exit 0 if up
|
||||
scripts/start-backend.sh # boot in background (MPS/CUDA auto-detected)
|
||||
```
|
||||
|
||||
First synthesis call lazy-downloads the `k2-fsa/OmniVoice` model (~2.4 GB) from HuggingFace — cached on subsequent boots.
|
||||
|
||||
## Task Index — Pick the Right Tool
|
||||
|
||||
| Task | Tool | Notes |
|
||||
|---|---|---|
|
||||
| Verify backend is up | `check_health` | Returns `{"status":"ok","device":"mps|cuda|cpu"}` |
|
||||
| Text → audio with a saved voice | `generate_speech(text, profile_id)` | Returns base64 WAV. `profile_id="demo0001"` is the bundled demo voice |
|
||||
| Text → audio without a clone (voice design) | `generate_speech(text, instruct="…")` | Omit `profile_id`; pass an `instruct` like `"warm middle-aged female narrator, calm pace"` |
|
||||
| Multilingual narration | `generate_speech(text, language="es")` | Any ISO 639 code or `"Auto"` |
|
||||
| List existing voices | `list_voices` | Returns id, name, type, personality |
|
||||
| List personality presets | `list_personalities` | Returns narrator / casual / news-anchor / etc. with their `instruct` strings |
|
||||
| List supported languages | `list_languages` | 646 total; returns 20 popular + the full count |
|
||||
|
||||
For non-trivial decisions (which engine to use, when to pick VoiceStudio over kokoro / Edge TTS / ElevenLabs), see [references/engines-comparison.md](references/engines-comparison.md).
|
||||
|
||||
For MCP wiring details, backend lifecycle, troubleshooting, and a clean teardown, see [references/mcp-setup.md](references/mcp-setup.md).
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### 1. One-shot narration with the demo voice
|
||||
|
||||
```python
|
||||
# As called through the MCP client (your agent will do this for you):
|
||||
result = generate_speech(
|
||||
text="Hello — this is VoiceStudio generating speech locally.",
|
||||
profile_id="demo0001",
|
||||
language="English",
|
||||
steps=16, # 8 = fast/draft · 16 = balanced · 32 = quality
|
||||
)
|
||||
# result is JSON with audio_id, generation_time_s, audio_duration_s, format, wav_base64
|
||||
```
|
||||
|
||||
Benchmark: 4.2 s of audio in ~24 s server-side on Apple Silicon MPS at 16 diffusion steps.
|
||||
|
||||
### 2. Save the WAV to disk and play
|
||||
|
||||
Tool returns base64 PCM WAV (16-bit, mono, 24 kHz). Decode + write:
|
||||
|
||||
```python
|
||||
import base64, json
|
||||
payload = json.loads(result_text) # parse JSON the tool returns
|
||||
open("out.wav","wb").write(base64.b64decode(payload["wav_base64"]))
|
||||
```
|
||||
|
||||
On macOS: `afplay out.wav`. Convert to MP3 with `ffmpeg -i out.wav -codec:a libmp3lame -b:a 128k out.mp3`.
|
||||
|
||||
### 3. Voice clone — end-to-end recipe
|
||||
|
||||
Cloning needs a 3-10 second reference clip the model will use as a speaker embedding. The MCP server does NOT expose profile creation — it only reads existing profiles. Two paths to create one:
|
||||
|
||||
**Path A — bundled helper (macOS, recommended for fresh clones):**
|
||||
|
||||
```bash
|
||||
scripts/record-reference.sh ~/Downloads/my-ref.wav 12 1
|
||||
# args: output_path raw_duration_sec mic_index
|
||||
# Default mic_index=1 (MacBook built-in); list devices via:
|
||||
# ffmpeg -f avfoundation -list_devices true -i ""
|
||||
```
|
||||
|
||||
The script gives **audible** countdown + start/stop cues via macOS `say` + `/System/Library/Sounds/Ping.aiff` so the user knows when to speak (terminal stdout is buffered — text "speak now" prompts arrive too late). It records a longer raw window, then trims to ~10 seconds of speech via `silenceremove + atrim`, plays back for verification, and prints the next-step `curl` command.
|
||||
|
||||
**Path B — manual:**
|
||||
|
||||
```bash
|
||||
# 1. Record (mono, 24 kHz native — matches model's internal rate)
|
||||
ffmpeg -f avfoundation -i ":1" -t 12 -ac 1 -ar 24000 raw.wav
|
||||
|
||||
# 2. Trim leading silence + take first 10 sec of speech
|
||||
ffmpeg -i raw.wav \
|
||||
-af "silenceremove=start_periods=1:start_silence=0.05:start_threshold=-40dB,atrim=end=10" \
|
||||
-ac 1 -ar 24000 ref.wav
|
||||
|
||||
# 3. Verify
|
||||
ffmpeg -i ref.wav -af volumedetect -f null - 2>&1 | grep volume # max should be > -20 dB
|
||||
afplay ref.wav
|
||||
```
|
||||
|
||||
**POST to /profiles** (multipart/form-data — required fields: `name`, `ref_audio`):
|
||||
|
||||
```bash
|
||||
curl -X POST http://127.0.0.1:3900/profiles \
|
||||
-F "name=carlos-clone" \
|
||||
-F "ref_audio=@ref.wav" \
|
||||
-F "ref_text=The exact text spoken in the clip" \
|
||||
-F "language=English" \
|
||||
| python3 -m json.tool
|
||||
# returns { "id": "abc12345", "name": "carlos-clone" }
|
||||
```
|
||||
|
||||
Once created, pass `profile_id` to `generate_speech` (via MCP) or directly via `POST /generate`. Profiles persist in SQLite + reference-audio files at `~/Library/Application Support/OmniVoice/voices/<id>.<ext>` (the backend preserves the uploaded extension — `.wav` if you uploaded a WAV, `.mp3` if MP3, etc.). State persists across backend restarts.
|
||||
|
||||
**Reference clip tips that materially affect quality:**
|
||||
|
||||
| Factor | Why it matters |
|
||||
|---|---|
|
||||
| Single speaker | Mixed speakers blur the embedding |
|
||||
| Clean speech, no music/noise | Model embeds the noise too |
|
||||
| Natural prosody (avoid pangrams) | Diffusion samples replicate prosody, not just timbre |
|
||||
| 3-10 sec is the sweet spot | < 3 s lacks information; > 10 s adds compute without quality gain |
|
||||
| Match `ref_text` to what's spoken | Improves alignment, especially on noisy refs |
|
||||
| `language` correct | Wrong language → cross-lingual transfer artifacts |
|
||||
| Loudness peak ≥ -15 dB | Quiet refs work but normalize poorly |
|
||||
|
||||
### 4. Voice design (no reference clip)
|
||||
|
||||
Skip `profile_id`; provide an `instruct` string describing the desired voice:
|
||||
|
||||
```python
|
||||
generate_speech(
|
||||
text="Welcome to the future of agentic systems.",
|
||||
instruct="warm middle-aged female narrator, calm authoritative pace, documentary style",
|
||||
)
|
||||
```
|
||||
|
||||
Get pre-made instructs via `list_personalities` and copy the one matching the brief (narrator, casual, news-anchor, etc.).
|
||||
|
||||
### 5. Video dubbing (web UI only)
|
||||
|
||||
The MCP server does not expose the dubbing endpoint. The full transcribe → translate → re-voice → mux pipeline lives behind the desktop UI (`bun run desktop` in `$OMNIVOICE_HOME`) and the `/dub/*` REST routes. When the user asks to dub a video, point them to the UI; surface this skill only for the synthesis primitives above.
|
||||
|
||||
## When NOT to use VoiceStudio
|
||||
|
||||
- **Fast English-only narration on weak hardware** → `kokoro-tts` is ~10× smaller and 2× realtime on CPU (see [references/engines-comparison.md](references/engines-comparison.md))
|
||||
- **Lowest-friction one-off TTS** → Edge TTS needs no install or backend
|
||||
- **Highest possible quality regardless of cost** → ElevenLabs still wins on English narration polish; VoiceStudio ties or wins on multilingual + cloning
|
||||
- **Real-time streaming dictation** → use the VoiceStudio desktop widget (`⌘+⇧+Space`), not the MCP server
|
||||
|
||||
## Resources
|
||||
|
||||
- [references/engines-comparison.md](references/engines-comparison.md) — Decision tree across VoiceStudio / kokoro / Voicebox / Edge TTS / ElevenLabs / cloud APIs
|
||||
- [references/mcp-setup.md](references/mcp-setup.md) — MCP wiring, backend lifecycle, env vars, troubleshooting
|
||||
- [scripts/check-health.sh](scripts/check-health.sh) — `curl /health`, exit 0/1
|
||||
- [scripts/start-backend.sh](scripts/start-backend.sh) — Start uvicorn on 127.0.0.1:3900 with health probe
|
||||
- [scripts/stop-backend.sh](scripts/stop-backend.sh) — Clean shutdown via `kill -TERM` on the bound PID
|
||||
- [scripts/record-reference.sh](scripts/record-reference.sh) — macOS-only: record + trim + verify a reference clip for cloning, with audible cues (`say` + system beeps) that bypass terminal output buffering
|
||||
|
||||
Backend Swagger / OpenAPI: `http://127.0.0.1:3900/docs` (when backend is up).
|
||||
|
||||
Upstream: github.com/debpalash/VoiceStudio. The app uses AGPL-3.0-only; optional engines and downloaded models retain their own licenses. See `LICENSE-NOTICE.md` in the repository.
|
||||
+20
-10
@@ -56,7 +56,17 @@ bun install
|
||||
bun run dev
|
||||
```
|
||||
|
||||
This starts both services:
|
||||
This launches Electron with hot reload. Its runtime supervisor manages backend setup
|
||||
and startup; do not launch a second backend. See [Electron setup](../electron/README.md).
|
||||
|
||||
```bash
|
||||
bun run build # build Electron
|
||||
bun run start # launch the built Electron app
|
||||
bun run dist # package locally without publishing
|
||||
bun run dev:web # legacy browser UI + backend
|
||||
```
|
||||
|
||||
The legacy browser command starts both services:
|
||||
|
||||
| Service | URL | What it does |
|
||||
|---------|-----|---|
|
||||
@@ -71,29 +81,29 @@ cause doesn't scroll away with the terminal. The same death is also reported
|
||||
as a crash notice in the UI the next time the backend starts (see
|
||||
[docs/install/troubleshooting.md §14c](docs/install/troubleshooting.md)).
|
||||
|
||||
### Desktop App (Tauri)
|
||||
### Legacy Desktop App (Tauri)
|
||||
|
||||
```bash
|
||||
bun run desktop # dev: hot-reload Tauri shell + backend
|
||||
bun run desktop-prod # production: builds, bundles the backend, then launches
|
||||
bun run tauri # legacy dev: hot-reload Tauri shell + backend
|
||||
bun run tauri:desktop-prod # legacy production: builds, bundles the backend, then launches
|
||||
```
|
||||
|
||||
Both run `uv sync` first (so the Python backend env is set up) and start the
|
||||
backend automatically — you do **not** start it separately. Use the exact script
|
||||
names: there is no `desktop=prod` (note the **hyphen** in `desktop-prod`).
|
||||
`desktop-prod` is Windows-aware (auto-detects bash/git; see `scripts/desktop-prod.mjs`).
|
||||
names: there is no `desktop=prod` (note the **hyphen** in `tauri:desktop-prod`).
|
||||
`tauri:desktop-prod` is Windows-aware (auto-detects bash/git; see `scripts/desktop-prod.mjs`).
|
||||
|
||||
Requires [Rust](https://rustup.rs/) and platform-specific Tauri dependencies — see the [Tauri prerequisites](https://v2.tauri.app/start/prerequisites/).
|
||||
|
||||
After installing Rust with rustup (or `uv` with its installer), a terminal that
|
||||
was already open still has the old `PATH`. The desktop launchers (`bun desktop`,
|
||||
`bun desktop-prod`, `bun desktop-fresh`) detect this and add `~/.cargo/bin` /
|
||||
was already open still has the old `PATH`. The desktop launchers (`bun tauri`,
|
||||
`bun tauri:desktop-prod`, `bun tauri:desktop-fresh`) detect this and add `~/.cargo/bin` /
|
||||
`~/.local/bin` for that run, printing a one-line note; to make it permanent,
|
||||
open a new terminal, or on macOS/Linux load Cargo into the current one:
|
||||
|
||||
```bash
|
||||
source "$HOME/.cargo/env"
|
||||
bun desktop
|
||||
bun run tauri
|
||||
```
|
||||
|
||||
If Rust is genuinely not installed, the launchers stop up front with the
|
||||
@@ -310,7 +320,7 @@ that — the agent recalls the architecture, conventions, and your past findings
|
||||
instead of re-reading the tree each time. [**memxt**](https://github.com/debpalash/memxt)
|
||||
(100% local, MCP-based, built by this project's maintainer) exists for exactly
|
||||
this; any MCP memory server works. Pair it with the repo's agent skill —
|
||||
`npx skills add debpalash/omnivoice-studio` — so your agent knows the project's
|
||||
`npx skills add debpalash/VoiceStudio` — so your agent knows the project's
|
||||
hard rules from the first prompt.
|
||||
|
||||
## Quality gates your PR must pass
|
||||
|
||||
@@ -172,6 +172,12 @@ jobs:
|
||||
working-directory: frontend
|
||||
run: node --experimental-strip-types --no-warnings --test ../tests/frontend/*.test.mjs
|
||||
|
||||
# Electron used to be built only after a release started, so renderer,
|
||||
# preload and packaging regressions could pass the required PR gate.
|
||||
# Keep this command shared with release.yml through the root script.
|
||||
- name: Electron typecheck, tests and production contract
|
||||
run: bun run check:electron
|
||||
|
||||
# Production-bundle blank-screen gate. Everything above runs UN-minified
|
||||
# (dev server + Vitest/jsdom), so a crash that exists ONLY in the minified
|
||||
# release bundle — a TDZ reorder that throws before React mounts — passes
|
||||
@@ -186,6 +192,28 @@ jobs:
|
||||
- name: Production-bundle smoke — no blank screen
|
||||
working-directory: frontend
|
||||
run: bun run test:prod-bundle
|
||||
- name: Electron renderer workflow smokes
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
export OMNIVOICE_PORT=3999
|
||||
export VOICESTUDIO_UI_URL=http://localhost:3912
|
||||
export PLAYWRIGHT_CHANNEL=chromium
|
||||
bun run --cwd electron smoke:server > /tmp/voicestudio-electron-smoke.log 2>&1 &
|
||||
server_pid=$!
|
||||
trap 'kill "$server_pid" 2>/dev/null || true' EXIT
|
||||
for _ in {1..60}; do
|
||||
if curl --fail --silent --show-error "$VOICESTUDIO_UI_URL" >/dev/null; then
|
||||
break
|
||||
fi
|
||||
sleep 0.25
|
||||
done
|
||||
curl --fail --silent --show-error "$VOICESTUDIO_UI_URL" >/dev/null || {
|
||||
cat /tmp/voicestudio-electron-smoke.log
|
||||
exit 1
|
||||
}
|
||||
node electron/tests/playback-smoke.mjs
|
||||
node electron/tests/dub-smoke.mjs
|
||||
|
||||
# ── Cross-platform Tauri shell check ────────────────────────────────────
|
||||
# Catches platform-specific Rust regressions on PR (cfg(target_os=...)
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
name: Electron packaging rehearsal
|
||||
|
||||
# Explicitly artifact-only: no tag, schedule, release, or publishing permission.
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: electron-rehearsal-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
package:
|
||||
runs-on: ${{ matrix.runner }}
|
||||
timeout-minutes: 60
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- runner: ubuntu-24.04
|
||||
platform: linux
|
||||
arch: x64
|
||||
target: x86_64-unknown-linux-gnu
|
||||
flags: --linux --x64
|
||||
- runner: windows-2022
|
||||
platform: win32
|
||||
arch: x64
|
||||
target: x86_64-pc-windows-msvc
|
||||
flags: --win --x64
|
||||
- runner: macos-15
|
||||
platform: darwin
|
||||
arch: arm64
|
||||
target: aarch64-apple-darwin
|
||||
flags: --mac --arm64
|
||||
- runner: macos-15-intel
|
||||
platform: darwin
|
||||
arch: x64
|
||||
target: x86_64-apple-darwin
|
||||
flags: --mac --x64
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
env:
|
||||
VOICESTUDIO_RUST_TARGET: ${{ matrix.target }}
|
||||
VOICESTUDIO_UPDATE_CHANNEL: electron-preview-${{ matrix.platform }}-${{ matrix.arch }}
|
||||
CSC_IDENTITY_AUTO_DISCOVERY: 'false'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
- uses: oven-sh/setup-bun@v2
|
||||
with:
|
||||
bun-version: '1.4.2'
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
with:
|
||||
targets: ${{ matrix.target }}
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
workspaces: native/desktop-bridge -> target
|
||||
key: electron-${{ matrix.target }}
|
||||
- uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: '0.12.13'
|
||||
enable-cache: false
|
||||
- name: Linux native dependencies
|
||||
if: runner.os == 'Linux'
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y libasound2-dev libxdo-dev libxtst-dev libx11-dev libxkbcommon-dev libwayland-dev libssl-dev pkg-config xvfb
|
||||
- name: Bundle pinned uv for the host architecture
|
||||
run: |
|
||||
node --input-type=module <<'NODE'
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { mkdirSync, copyFileSync, chmodSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
const expected = process.env.VOICESTUDIO_RUST_TARGET;
|
||||
const targets = { 'linux-x64': 'x86_64-unknown-linux-gnu', 'win32-x64': 'x86_64-pc-windows-msvc', 'darwin-arm64': 'aarch64-apple-darwin', 'darwin-x64': 'x86_64-apple-darwin' };
|
||||
if (targets[`${process.platform}-${process.arch}`] !== expected) throw new Error('Runner architecture does not match package target');
|
||||
const source = execFileSync(process.platform === 'win32' ? 'where.exe' : 'which', ['uv'], { encoding: 'utf8' }).trim().split(/\r?\n/)[0];
|
||||
const dir = 'frontend/src-tauri/binaries';
|
||||
mkdirSync(dir, { recursive: true });
|
||||
const destination = join(dir, `uv-${expected}${process.platform === 'win32' ? '.exe' : ''}`);
|
||||
copyFileSync(source, destination);
|
||||
if (process.platform !== 'win32') chmodSync(destination, 0o755);
|
||||
NODE
|
||||
- name: Install locked dependencies
|
||||
run: bun install --frozen-lockfile
|
||||
- name: Validate and build Electron
|
||||
run: bun run check:electron
|
||||
- name: Package without publishing
|
||||
working-directory: electron
|
||||
run: |
|
||||
bun x electron-builder --config electron-builder.config.mjs ${{ matrix.flags }} --publish never
|
||||
node tests/packaging-contract.mjs --artifact
|
||||
node tests/update-package-contract.mjs --platform ${{ matrix.platform }} --arch ${{ matrix.arch }}
|
||||
- name: Packaged startup smoke test
|
||||
working-directory: electron
|
||||
run: |
|
||||
if [ "$RUNNER_OS" = Linux ]; then
|
||||
xvfb-run -a node tests/packaged-smoke.mjs --setup
|
||||
else
|
||||
node tests/packaged-smoke.mjs --setup
|
||||
fi
|
||||
- name: Save installers and updater metadata for review
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: electron-rehearsal-${{ matrix.platform }}-${{ matrix.arch }}
|
||||
retention-days: 14
|
||||
if-no-files-found: error
|
||||
path: |
|
||||
electron/release/VoiceStudio-Electron-*
|
||||
electron/release/electron-*.yml
|
||||
@@ -0,0 +1,242 @@
|
||||
name: Electron desktop release
|
||||
|
||||
# Builds are safe by default. Only an explicit publish dispatch exposes a release.
|
||||
on:
|
||||
push:
|
||||
tags: ['v*']
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
release_tag:
|
||||
description: "Existing version tag to package using this workflow from main (optional)"
|
||||
type: string
|
||||
default: ''
|
||||
publish:
|
||||
description: "Publish the tagged Electron release after all platforms pass"
|
||||
type: boolean
|
||||
default: false
|
||||
allow_unsigned:
|
||||
description: "Explicitly accept unsigned/unnotarized Electron installers and documented updater limitations"
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: electron-release-${{ inputs.release_tag || github.ref_name }}
|
||||
cancel-in-progress: false
|
||||
|
||||
env:
|
||||
RELEASE_REF: ${{ inputs.release_tag && format('refs/tags/{0}', inputs.release_tag) || github.ref }}
|
||||
|
||||
jobs:
|
||||
validate:
|
||||
# The transition tag is assembled after the manual Tauri draft succeeds.
|
||||
if: github.event_name == 'workflow_dispatch' || github.ref_name != vars.TAURI_SUNSET_TAG
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ env.RELEASE_REF }}
|
||||
- name: Require an exact version tag
|
||||
env:
|
||||
REF: ${{ env.RELEASE_REF }}
|
||||
WORKFLOW_REF: ${{ github.ref }}
|
||||
RELEASE_TAG_OVERRIDE: ${{ inputs.release_tag }}
|
||||
ALLOW_UNSIGNED: ${{ inputs.allow_unsigned }}
|
||||
DISPATCH_ACTOR: ${{ github.actor }}
|
||||
RERUN_ACTOR: ${{ github.triggering_actor }}
|
||||
OWNER: ${{ github.repository_owner }}
|
||||
run: |
|
||||
if [ "$ALLOW_UNSIGNED" = true ]; then
|
||||
test "$DISPATCH_ACTOR" = "$OWNER" && test "$RERUN_ACTOR" = "$OWNER" || {
|
||||
echo "Only the repository owner may accept unsigned installers"; exit 1;
|
||||
}
|
||||
fi
|
||||
if [ -n "$RELEASE_TAG_OVERRIDE" ]; then
|
||||
test "$WORKFLOW_REF" = refs/heads/main || { echo "Tag overrides require the workflow from main"; exit 1; }
|
||||
fi
|
||||
VERSION=$(node -p "require('./frontend/package.json').version")
|
||||
test "$REF" = "refs/tags/v$VERSION" || { echo "Select the exact version tag"; exit 1; }
|
||||
test "$(git rev-parse HEAD)" = "$(git rev-parse "$REF^{commit}")" || { echo "Checkout does not match the release tag"; exit 1; }
|
||||
package:
|
||||
needs: validate
|
||||
runs-on: ${{ matrix.runner }}
|
||||
timeout-minutes: 60
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- runner: ubuntu-24.04
|
||||
platform: linux
|
||||
arch: x64
|
||||
target: x86_64-unknown-linux-gnu
|
||||
flags: --linux --x64
|
||||
- runner: windows-2022
|
||||
platform: win32
|
||||
arch: x64
|
||||
target: x86_64-pc-windows-msvc
|
||||
flags: --win --x64
|
||||
- runner: macos-15
|
||||
platform: darwin
|
||||
arch: arm64
|
||||
target: aarch64-apple-darwin
|
||||
flags: --mac --arm64
|
||||
- runner: macos-15-intel
|
||||
platform: darwin
|
||||
arch: x64
|
||||
target: x86_64-apple-darwin
|
||||
flags: --mac --x64
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
env:
|
||||
VOICESTUDIO_RUST_TARGET: ${{ matrix.target }}
|
||||
VOICESTUDIO_UPDATE_CHANNEL: electron-stable-${{ matrix.platform }}-${{ matrix.arch }}
|
||||
CSC_IDENTITY_AUTO_DISCOVERY: 'false'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ env.RELEASE_REF }}
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
- uses: oven-sh/setup-bun@v2
|
||||
with:
|
||||
bun-version: '1.4.2'
|
||||
- uses: dtolnay/rust-toolchain@6bed0761d98439e5a578e2877258200ad565ba87 # stable
|
||||
with:
|
||||
targets: ${{ matrix.target }}
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
with:
|
||||
workspaces: native/desktop-bridge -> target
|
||||
key: electron-${{ matrix.target }}
|
||||
- uses: astral-sh/setup-uv@v6
|
||||
with:
|
||||
version: '0.12.13'
|
||||
enable-cache: false
|
||||
- name: Linux native dependencies
|
||||
if: runner.os == 'Linux'
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y libasound2-dev libxdo-dev libxtst-dev libx11-dev libxkbcommon-dev libwayland-dev libssl-dev pkg-config xvfb
|
||||
- name: Bundle pinned uv for the host architecture
|
||||
run: |
|
||||
node --input-type=module <<'NODE'
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { mkdirSync, copyFileSync, chmodSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
const expected = process.env.VOICESTUDIO_RUST_TARGET;
|
||||
const targets = { 'linux-x64': 'x86_64-unknown-linux-gnu', 'win32-x64': 'x86_64-pc-windows-msvc', 'darwin-arm64': 'aarch64-apple-darwin', 'darwin-x64': 'x86_64-apple-darwin' };
|
||||
if (targets[`${process.platform}-${process.arch}`] !== expected) throw new Error('Runner architecture does not match package target');
|
||||
const source = execFileSync(process.platform === 'win32' ? 'where.exe' : 'which', ['uv'], { encoding: 'utf8' }).trim().split(/\r?\n/)[0];
|
||||
const dir = 'frontend/src-tauri/binaries';
|
||||
mkdirSync(dir, { recursive: true });
|
||||
const destination = join(dir, `uv-${expected}${process.platform === 'win32' ? '.exe' : ''}`);
|
||||
copyFileSync(source, destination);
|
||||
if (process.platform !== 'win32') chmodSync(destination, 0o755);
|
||||
NODE
|
||||
- name: Install locked dependencies
|
||||
run: bun install --frozen-lockfile
|
||||
- name: Validate and build Electron
|
||||
run: bun run check:electron
|
||||
- name: Package without publishing
|
||||
env:
|
||||
CSC_LINK: ${{ secrets.ELECTRON_CSC_LINK }}
|
||||
CSC_KEY_PASSWORD: ${{ secrets.ELECTRON_CSC_KEY_PASSWORD }}
|
||||
working-directory: electron
|
||||
run: |
|
||||
# An empty CSC_LINK is interpreted as the working directory by the
|
||||
# signer. Omit absent credentials rather than passing empty strings.
|
||||
if [ -z "${CSC_LINK:-}" ]; then
|
||||
unset CSC_LINK CSC_KEY_PASSWORD
|
||||
fi
|
||||
bun x electron-builder --config electron-builder.config.mjs ${{ matrix.flags }} --publish never
|
||||
node tests/packaging-contract.mjs --artifact
|
||||
node tests/update-package-contract.mjs --platform ${{ matrix.platform }} --arch ${{ matrix.arch }}
|
||||
- name: Verify macOS signing and notarization before publication
|
||||
if: inputs.publish == true && inputs.allow_unsigned != true && matrix.platform == 'darwin'
|
||||
run: |
|
||||
APP=$(find electron/release -maxdepth 2 -name VoiceStudio.app -type d -print -quit)
|
||||
test -n "$APP"
|
||||
codesign --verify --deep --strict "$APP"
|
||||
spctl --assess --type execute --verbose=2 "$APP"
|
||||
- name: Verify Windows installer signature before publication
|
||||
if: inputs.publish == true && inputs.allow_unsigned != true && matrix.platform == 'win32'
|
||||
shell: pwsh
|
||||
run: |
|
||||
$installers = @(Get-ChildItem electron/release/VoiceStudio-Electron-*.exe)
|
||||
if ($installers.Count -eq 0) { throw "No installer to verify" }
|
||||
foreach ($installer in $installers) {
|
||||
$signature = Get-AuthenticodeSignature $installer.FullName
|
||||
if ($signature.Status -ne 'Valid') { throw "Installer signature is not trusted: $($installer.Name)" }
|
||||
}
|
||||
- name: Packaged startup smoke test
|
||||
working-directory: electron
|
||||
run: |
|
||||
if [ "$RUNNER_OS" = Linux ]; then
|
||||
xvfb-run -a node tests/packaged-smoke.mjs --setup
|
||||
else
|
||||
node tests/packaged-smoke.mjs --setup
|
||||
fi
|
||||
- name: Save installers and updater metadata for review
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: electron-release-${{ matrix.platform }}-${{ matrix.arch }}
|
||||
retention-days: 14
|
||||
if-no-files-found: error
|
||||
path: |
|
||||
electron/release/VoiceStudio-Electron-*
|
||||
electron/release/electron-*.yml
|
||||
|
||||
release:
|
||||
needs: package
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: write
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
TAG: ${{ inputs.release_tag || github.ref_name }}
|
||||
SUNSET_TAG: ${{ vars.TAURI_SUNSET_TAG }}
|
||||
PUBLISH: ${{ inputs.publish }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ env.RELEASE_REF }}
|
||||
- uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: electron-release-*
|
||||
merge-multiple: true
|
||||
path: release-assets
|
||||
- name: Validate all platforms before creating a release
|
||||
run: |
|
||||
python3 scripts/prepare_electron_release.py --assets release-assets --tag "$TAG"
|
||||
- name: Preserve the final Tauri updater feeds
|
||||
run: |
|
||||
test -n "$SUNSET_TAG" || { echo "Set TAURI_SUNSET_TAG before releasing"; exit 1; }
|
||||
# The transition tag already holds its own final Tauri feeds.
|
||||
# Later releases carry copies pointing to the immutable sunset payloads.
|
||||
gh release download "$SUNSET_TAG" --pattern latest.json --dir release-assets
|
||||
gh release download "$SUNSET_TAG" --pattern latest-user.json --dir release-assets
|
||||
python3 scripts/prepare_electron_release.py --assets release-assets --tag "$TAG" --sunset-tag "$SUNSET_TAG"
|
||||
- name: Disclose explicitly accepted unsigned artifacts
|
||||
if: inputs.allow_unsigned == true
|
||||
run: |
|
||||
cat >> release-assets/RELEASE_NOTES.md <<'EOF'
|
||||
|
||||
### Electron installer trust
|
||||
These Electron installers are unsigned or ad-hoc signed and are not Apple-notarized.
|
||||
Windows/macOS may show trust warnings. macOS automatic updates are unverified;
|
||||
use manual installer updates. Tauri updater signatures remain independently verified.
|
||||
EOF
|
||||
- name: Create or update draft
|
||||
run: |
|
||||
if ! gh release view "$TAG" >/dev/null 2>&1; then
|
||||
gh release create "$TAG" --verify-tag --draft --title "$TAG — VoiceStudio" --notes-file release-assets/RELEASE_NOTES.md
|
||||
fi
|
||||
test "$(gh release view "$TAG" --json isDraft --jq .isDraft)" = true || { echo "Refusing to replace a published release"; exit 1; }
|
||||
gh release edit "$TAG" --notes-file release-assets/RELEASE_NOTES.md
|
||||
find release-assets -maxdepth 1 -type f ! -name RELEASE_NOTES.md -print0 | xargs -0 gh release upload "$TAG" --clobber
|
||||
- name: Publish only when explicitly requested
|
||||
if: github.event_name == 'workflow_dispatch' && inputs.publish == true
|
||||
run: gh release edit "$TAG" --draft=false --latest
|
||||
+157
-18
@@ -27,27 +27,22 @@
|
||||
# each to surface PyInstaller/Tauri issues that never showed up locally on
|
||||
# macOS — iterate on CI.
|
||||
|
||||
name: Desktop Release
|
||||
name: Tauri sunset (manual only)
|
||||
|
||||
# Legacy workflow: run once on the final Tauri version tag.
|
||||
# Electron releases are owned by electron-release.yml.
|
||||
on:
|
||||
push:
|
||||
tags: ['v*']
|
||||
schedule:
|
||||
# 07:00 UTC daily — rolling `preview` prerelease from `main`. The
|
||||
# preview-gate job no-ops the matrix when main hasn't moved in a day.
|
||||
- cron: '0 7 * * *'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
draft:
|
||||
description: "Create as draft release (tag push only)"
|
||||
required: false
|
||||
description: "Keep the final Tauri release draft until Electron artifacts are ready"
|
||||
default: "true"
|
||||
publish_preview:
|
||||
description: "Publish a rolling 'preview' prerelease (updater Preview channel). Previews ALWAYS build from main — dispatching from any other branch fails the preview-gate."
|
||||
required: false
|
||||
description: "Legacy compatibility input; previews are retired"
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
|
||||
permissions:
|
||||
contents: write # needed to attach artifacts + updater manifest to GH Release
|
||||
|
||||
@@ -76,6 +71,16 @@ jobs:
|
||||
name: Tests (backend + frontend)
|
||||
runs-on: ubuntu-22.04
|
||||
steps:
|
||||
- name: Require the designated final Tauri tag
|
||||
env:
|
||||
SUNSET_TAG: ${{ vars.TAURI_SUNSET_TAG }}
|
||||
REF: ${{ github.ref }}
|
||||
PREVIEW: ${{ inputs.publish_preview }}
|
||||
run: |
|
||||
test -n "$SUNSET_TAG" || { echo "Set TAURI_SUNSET_TAG to the final v* tag first"; exit 1; }
|
||||
test "$REF" = "refs/tags/$SUNSET_TAG"
|
||||
test "$PREVIEW" != "true"
|
||||
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Python 3.11
|
||||
@@ -142,6 +147,9 @@ jobs:
|
||||
working-directory: frontend
|
||||
run: node --experimental-strip-types --no-warnings --test ../tests/frontend/*.test.mjs
|
||||
|
||||
- name: Electron typecheck, tests and production contract
|
||||
run: bun run check:electron
|
||||
|
||||
# Decide preview-vs-stable, and for nightly runs whether `main` actually
|
||||
# moved in the last day. Outputs gate the expensive matrix (`build`) and the
|
||||
# `preview-notes` job, so a no-commit night costs only this ~30s job.
|
||||
@@ -311,7 +319,7 @@ jobs:
|
||||
libwebkit2gtk-4.1-dev \
|
||||
build-essential curl wget file libxdo-dev libssl-dev \
|
||||
libayatana-appindicator3-dev librsvg2-dev \
|
||||
libasound2-dev ffmpeg
|
||||
libasound2-dev ffmpeg xvfb
|
||||
|
||||
# ── Frontend build ─────────────────────────────────────────────────
|
||||
- name: Cache bun deps
|
||||
@@ -344,7 +352,7 @@ jobs:
|
||||
- name: Bundle uv (${{ matrix.rust_target }})
|
||||
shell: bash
|
||||
env:
|
||||
UV_VERSION: "0.11.7"
|
||||
UV_VERSION: "0.12.13"
|
||||
TRIPLE: ${{ matrix.rust_target }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
@@ -473,6 +481,9 @@ jobs:
|
||||
fi
|
||||
{
|
||||
echo 'body<<RELEASE_BODY_EOF'
|
||||
echo '## Final Tauri update'
|
||||
echo 'VoiceStudio desktop is moving to Electron. This is the last Tauri release. Back up your data and install Electron separately: https://github.com/debpalash/VoiceStudio/blob/main/docs/electron-migration.md'
|
||||
echo
|
||||
echo "$BODY"
|
||||
echo 'RELEASE_BODY_EOF'
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
@@ -663,6 +674,84 @@ jobs:
|
||||
updaterJsonPreferNsis: false
|
||||
includeUpdaterJson: true
|
||||
|
||||
- name: Build + publish Electron desktop
|
||||
if: false # Electron is released independently by electron-release.yml.
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
RELEASE_TAG: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'preview' || github.ref_name }}
|
||||
IS_PREVIEW: ${{ needs.preview-gate.outputs.is_preview }}
|
||||
VOICESTUDIO_RUST_TARGET: ${{ matrix.rust_target }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
case "${{ matrix.rust_target }}" in
|
||||
aarch64-apple-darwin) ELECTRON_OS=darwin; ELECTRON_ARCH=arm64; FLAGS="--mac --arm64" ;;
|
||||
x86_64-apple-darwin) ELECTRON_OS=darwin; ELECTRON_ARCH=x64; FLAGS="--mac --x64" ;;
|
||||
x86_64-pc-windows-msvc) ELECTRON_OS=win32; ELECTRON_ARCH=x64; FLAGS="--win --x64" ;;
|
||||
x86_64-unknown-linux-gnu) ELECTRON_OS=linux; ELECTRON_ARCH=x64; FLAGS="--linux --x64" ;;
|
||||
*) echo "Unsupported Electron target: ${{ matrix.rust_target }}"; exit 1 ;;
|
||||
esac
|
||||
if [ "$IS_PREVIEW" = "true" ]; then
|
||||
export VOICESTUDIO_UPDATE_CHANNEL="electron-preview-${ELECTRON_OS}-${ELECTRON_ARCH}"
|
||||
else
|
||||
export VOICESTUDIO_UPDATE_CHANNEL="electron-stable-${ELECTRON_OS}-${ELECTRON_ARCH}"
|
||||
fi
|
||||
if [ -n "${APPLE_CERTIFICATE:-}" ]; then
|
||||
export CSC_LINK="$APPLE_CERTIFICATE"
|
||||
export CSC_KEY_PASSWORD="${APPLE_CERTIFICATE_PASSWORD:-}"
|
||||
fi
|
||||
|
||||
bun install --frozen-lockfile
|
||||
(
|
||||
cd electron
|
||||
bun run build
|
||||
node tests/packaging-contract.mjs
|
||||
bun x electron-builder \
|
||||
--config electron-builder.config.mjs $FLAGS --publish never
|
||||
node tests/packaging-contract.mjs --artifact
|
||||
if [ "$RUNNER_OS" = "Linux" ]; then
|
||||
xvfb-run -a node tests/packaged-smoke.mjs --setup
|
||||
else
|
||||
node tests/packaged-smoke.mjs --setup
|
||||
fi
|
||||
node tests/update-package-contract.mjs \
|
||||
--channel "$VOICESTUDIO_UPDATE_CHANNEL" \
|
||||
--platform "$ELECTRON_OS" \
|
||||
--arch "$ELECTRON_ARCH"
|
||||
)
|
||||
|
||||
# A rolling preview reuses one release. Remove only this platform /
|
||||
# architecture's older Electron artifacts before publishing the new
|
||||
# version; sibling matrix legs own different names and metadata.
|
||||
if [ "$IS_PREVIEW" = "true" ]; then
|
||||
gh release view "$RELEASE_TAG" --repo "$GITHUB_REPOSITORY" --json assets \
|
||||
--jq '.assets[].name' > electron-assets.txt
|
||||
case "${{ runner.os }}" in
|
||||
Windows) OS_TOKEN=win ;;
|
||||
macOS) OS_TOKEN=mac ;;
|
||||
Linux) OS_TOKEN=linux ;;
|
||||
esac
|
||||
while IFS= read -r asset; do
|
||||
case "$asset" in
|
||||
VoiceStudio-Electron-*-${OS_TOKEN}-${ELECTRON_ARCH}.*|${VOICESTUDIO_UPDATE_CHANNEL}*.yml)
|
||||
gh release delete-asset "$RELEASE_TAG" "$asset" --yes --repo "$GITHUB_REPOSITORY"
|
||||
;;
|
||||
esac
|
||||
done < electron-assets.txt
|
||||
fi
|
||||
|
||||
electron_artifact_count=0
|
||||
while IFS= read -r artifact; do
|
||||
gh release upload "$RELEASE_TAG" "$artifact" --clobber --repo "$GITHUB_REPOSITORY"
|
||||
electron_artifact_count=$((electron_artifact_count + 1))
|
||||
done < <(find electron/release -maxdepth 1 -type f \
|
||||
\( -name 'VoiceStudio-Electron-*' -o -name "${VOICESTUDIO_UPDATE_CHANNEL}*.yml" \) | sort)
|
||||
if [ "$electron_artifact_count" -eq 0 ]; then
|
||||
echo "FAIL — Electron build produced no publishable artifacts"
|
||||
find electron/release -maxdepth 1 -type f -print || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Build per-user Windows MSI
|
||||
if: runner.os == 'Windows'
|
||||
shell: bash
|
||||
@@ -893,7 +982,7 @@ jobs:
|
||||
# Writes SHA256SUMS-<label>.txt, attached to the release below. The
|
||||
# release-notes-checksums job puts every leg's file into the notes.
|
||||
- name: Compute SHA-256 checksums
|
||||
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
|
||||
if: startsWith(github.ref, 'refs/tags/v') && (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
|
||||
id: checksums
|
||||
shell: bash
|
||||
run: |
|
||||
@@ -914,6 +1003,10 @@ jobs:
|
||||
-o -name "*.msi" -o -name "*.msi.sig" \
|
||||
-o -name "*.AppImage" -o -name "*.AppImage.sig" \
|
||||
-o -name "*.deb" \) 2>/dev/null | sort)
|
||||
while IFS= read -r artifact; do
|
||||
ARTIFACTS+=("$artifact")
|
||||
done < <(find electron/release -maxdepth 1 -type f \
|
||||
\( -name 'VoiceStudio-Electron-*' -o -name 'electron-*.yml' \) 2>/dev/null | sort)
|
||||
|
||||
if [ ${#ARTIFACTS[@]} -eq 0 ]; then
|
||||
echo "FAIL — no artifacts found under $BUNDLE_DIR"
|
||||
@@ -948,7 +1041,7 @@ jobs:
|
||||
# decision, so both belong to the single release-notes-checksums job
|
||||
# that runs after the whole matrix (see there for why).
|
||||
- name: Attach SHA256SUMS file
|
||||
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
|
||||
if: startsWith(github.ref, 'refs/tags/v') && (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
@@ -974,7 +1067,7 @@ jobs:
|
||||
# missing, so a failed platform leaves the release a draft.
|
||||
release-notes-checksums:
|
||||
needs: [build, repair-updater-manifest, uninstall-scripts]
|
||||
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
|
||||
if: startsWith(github.ref, 'refs/tags/v') && (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
|
||||
runs-on: ubuntu-22.04
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
@@ -983,6 +1076,7 @@ jobs:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
REPO: ${{ github.repository }}
|
||||
TAG: ${{ github.ref_name }}
|
||||
KEEP_DRAFT: ${{ inputs.draft }}
|
||||
steps:
|
||||
- name: Write every platform's checksums into the notes, then publish
|
||||
shell: bash
|
||||
@@ -1007,7 +1101,9 @@ jobs:
|
||||
done
|
||||
[ "$missing" = 0 ] || exit 1
|
||||
gh release edit "$TAG" --repo "$REPO" --notes-file "$WORK/notes.md"
|
||||
if [[ "$TAG" == *-* ]]; then
|
||||
if [[ "$KEEP_DRAFT" == "true" ]]; then
|
||||
echo "Final Tauri draft verified; Electron publication owns the transition."
|
||||
elif [[ "$TAG" == *-* ]]; then
|
||||
gh release edit "$TAG" --repo "$REPO" --draft=false --prerelease
|
||||
else
|
||||
gh release edit "$TAG" --repo "$REPO" --draft=false --latest
|
||||
@@ -1031,6 +1127,49 @@ jobs:
|
||||
# This job runs once after the whole matrix as the single final writer:
|
||||
# it makes the manifest's linux signature agree with the .sig asset that
|
||||
# actually shipped, and refuses to leave a mismatch behind.
|
||||
# The matrix validates each Electron package before upload. This final read-only
|
||||
# check validates the other half of the contract: GitHub must actually serve
|
||||
# all four manifests and every payload they name. Without it a green release
|
||||
# can leave the in-app updater with four 404 feeds.
|
||||
electron-publish-contract:
|
||||
needs: [build, preview-gate]
|
||||
if: false # Electron release workflow owns this contract.
|
||||
runs-on: ubuntu-22.04
|
||||
permissions:
|
||||
contents: read
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
TAG: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'preview' || github.ref_name }}
|
||||
CHANNEL: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'preview' || 'stable' }}
|
||||
STABLE_TAG: ${{ needs.preview-gate.outputs.stable_tag }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Verify published Electron updater assets
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
WORK="$(mktemp -d)"
|
||||
mkdir -p "$WORK/manifests"
|
||||
gh release view "$TAG" --repo "$GITHUB_REPOSITORY" \
|
||||
--json tagName,isPrerelease,assets > "$WORK/release.json"
|
||||
gh release download "$TAG" --repo "$GITHUB_REPOSITORY" \
|
||||
--pattern "electron-${CHANNEL}-*.yml" --dir "$WORK/manifests"
|
||||
if [ "$CHANNEL" = "preview" ]; then
|
||||
VERSION=$(python3 scripts/stamp-preview-version.py \
|
||||
--package-json frontend/package.json \
|
||||
--stable-tag "$STABLE_TAG" \
|
||||
--run-number "${{ github.run_number }}")
|
||||
else
|
||||
VERSION=$(python3 -c 'import json; print(json.load(open("frontend/package.json"))["version"])')
|
||||
fi
|
||||
python3 scripts/check_electron_release_assets.py \
|
||||
--release-json "$WORK/release.json" \
|
||||
--manifest-dir "$WORK/manifests" \
|
||||
--channel "$CHANNEL" \
|
||||
--version "$VERSION"
|
||||
|
||||
repair-updater-manifest:
|
||||
needs: [build, preview-gate]
|
||||
runs-on: ubuntu-latest
|
||||
@@ -1071,7 +1210,7 @@ jobs:
|
||||
|
||||
uninstall-scripts:
|
||||
needs: [build]
|
||||
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
|
||||
if: startsWith(github.ref, 'refs/tags/v') && (github.event_name == 'push' || github.event_name == 'workflow_dispatch')
|
||||
runs-on: ubuntu-22.04
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
@@ -22,6 +22,8 @@ node_modules
|
||||
.turbo/
|
||||
bun.lockb
|
||||
frontend/src-tauri/target/
|
||||
electron/.tmp-native-target/
|
||||
native/**/target*/
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
# Secrets & env
|
||||
@@ -55,6 +57,7 @@ memxt.db-wal
|
||||
!.claude/agents/**
|
||||
/.cache*
|
||||
/.tmp/
|
||||
/.tmp-*
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
# Research clones — upstream repos used as reference, not shipped
|
||||
|
||||
+56
-1
@@ -10,6 +10,30 @@ the frozen-backend fallback mirror it for their toolchains.
|
||||
|
||||
**Highlights**
|
||||
|
||||
- Electron packaging can recover without changing a release tag
|
||||
|
||||
### CI
|
||||
|
||||
- Handle missing Electron signing credentials and retry packaging fixes without moving release tags (#2157)
|
||||
|
||||
### Fixed
|
||||
|
||||
- Avoid pedalboard wheels that crash on unsupported CPU instructions (#2080) — thanks @D3nii!
|
||||
|
||||
## [0.5.3] — 2026-09-17
|
||||
|
||||
**Highlights**
|
||||
|
||||
- The README is shorter, with a new Electron UI tour and refreshed screenshots (#2129)
|
||||
|
||||
- Support pages feature cleaner donation cards, with a workspace support shortcut and sponsor footer with hover cards and email inquiries (#2129)
|
||||
|
||||
- Integrations has a dedicated sidebar workspace with featured sponsors, searchable AI providers, and smooth sponsor-strip scrolling (#2129)
|
||||
|
||||
- Integrations now covers 100+ automation, communications, MCP, agent, developer, data, and productivity tools with config-driven detail pages (#2129)
|
||||
|
||||
- Electron now ships as a complete cross-platform VoiceStudio desktop app with local-first cloning, production workspaces, model packs, repair agents, native integrations, updates, parity checks, and the shared backend contracts required by those workflows (#1823)
|
||||
|
||||
- The Model Catalogue is one page: what you use now on top, then each family's engines and weights (#2013)
|
||||
- VoxCPM2 installs in one click into its own environment, with the CUDA build of PyTorch on NVIDIA GPUs (#2021)
|
||||
- MOSS-TTS-Nano installs in one click into its own environment, pinned to a reviewed upstream commit it works with (#2022)
|
||||
@@ -17,19 +41,48 @@ the frozen-backend fallback mirror it for their toolchains.
|
||||
|
||||
### Changed
|
||||
|
||||
- Electron becomes the default source desktop, with artifact-only packaging rehearsals and a separate final Tauri update path (#2157)
|
||||
- Installable agent skills use current VoiceStudio names and Electron workflows (#2157)
|
||||
- README clarifies the Electron transition while keeping desktop contributions welcome (#2153) — thanks @cyberspace-cs!
|
||||
|
||||
- Electron first run uses four simple steps with model packs, optional advanced controls and skippable dictation setup (#2129)
|
||||
|
||||
- Model Catalogue is one page: a setup summary (speech, transcription, dictation, language model) on top, one TTS / ASR / LLM switch, and each family's downloadable weights listed under its engines; the separate Models pane and the Settings → Voice → Engines / Models signposts are gone, the models directory and voice previews moved to Settings → Storage and the HF mirror to Network (#2013)
|
||||
- The engine list is one line per engine (engine, device it runs on, status, one action) with a detail panel for everything else; each engine's weights install from its panel, so the separate weights list and recommendation card are gone (#2020)
|
||||
- CosyVoice 3 installs patched protobuf and transformers releases, clearing five security advisories (#2030, #2031)
|
||||
|
||||
### Fixed
|
||||
|
||||
- Keep demo playback aligned across languages, preserve worker GPU metrics, and restrict unsigned releases to owner dispatches (#2157)
|
||||
|
||||
- Desktop integration checks cover current dubbing safeguards, navigation, and the linked engine catalog (#2157)
|
||||
|
||||
- Tauri and Electron now share native dictation, watch-folder, and Wayland shortcut contracts; focused paste stays ordered and first-run uv stays pinned at 0.12.13 (#2122)
|
||||
- Dubbing demos synchronize playheads without simultaneous playback and let you open a sample in the editor (#2131)
|
||||
- macOS desktop sidebar clears the traffic lights, uses a narrower collapsed rail, and places notifications and device controls with more space (#2126)
|
||||
- Dubbing timelines keep short segments proportional, support zoom, and remove timestamp-confirmed duplicate ASR context (#2129)
|
||||
|
||||
- Dubbing translation shares the agent footer with live logs, validated output, cancellation and contextual retries (#2129)
|
||||
|
||||
- Agent dubbing translation saves a custom tone and adaptation prompt and preserves it during timing rewrites (#2129)
|
||||
|
||||
- Dubbing preserves original sound outside dialogue and mixes separated background only beneath replacement speech (#2129)
|
||||
|
||||
- Dubbing repairs missing speech caches, rejects incomplete output, avoids oversized speaker references, and fits full speech without early clipping (#2129)
|
||||
- Workspace sidebars have a working right-edge resize handle, allow 40% more width, remember their size, and keep video controls inside the preview (#2129)
|
||||
- Pressing Play while a video is loading starts playback when it is ready instead of reporting playback unavailable (#2129)
|
||||
- Video previews show their thumbnail before playback, including the source video in Dub (#2129)
|
||||
- Linux and Windows workspace headers consistently expand and collapse the sidebar, with the app logo at the top of the collapsed rail (#2129)
|
||||
- Stopping a process on macOS no longer fails with "Operation not permitted" when it was already exiting (#2032)
|
||||
- A YouTube link blocked by its "not a bot" check now says how to attach signed-in cookies in Dub, instead of quoting yt-dlp's command-line flags (#2036, #2034)
|
||||
- An engine that fails to start now says whether it timed out, crashed (with its exit code and last output) or answered wrongly, instead of "did not signal ready: None" (#2037, #2026)
|
||||
- Transcribing an M4A file with PyTorch Whisper works, instead of failing with "Format not recognised" (#2042, #2039)
|
||||
- PyTorch Whisper runs on 6 GB NVIDIA cards instead of falling back to CPU, because its memory check now fits the model it loads (#2044, #2041)
|
||||
- MCP tools wait as long as the backend does, so a long transcription no longer fails at 120 s with an empty error (#2043, #2040)
|
||||
- First synthesis no longer dies with SIGILL on Linux/Docker CPUs that cannot run pedalboard 0.9.21+ wheels (#2052) — thanks @asartonev!
|
||||
- Generating on an older NVIDIA GPU (Tesla T4, and other pre-Ampere cards) no longer kills the backend on the first request — CUDA graphs are not captured below sm_80 (#2135)
|
||||
- "Disable torch.compile" in Settings → Performance now works on macOS and Linux, not only Windows; it was greyed out on the platforms that needed it (#2135)
|
||||
- Setting `TORCH_COMPILE_DISABLE=1` in the environment now actually disables torch.compile, for the in-process engine and engine subprocesses alike (#2135)
|
||||
- A backend killed by a native crash now leaves the faulting thread's stack in `backend_err.log` instead of exiting silently (#2135)
|
||||
|
||||
### CI
|
||||
|
||||
@@ -128,6 +181,8 @@ the frozen-backend fallback mirror it for their toolchains.
|
||||
|
||||
### Added
|
||||
|
||||
- Remote-worker metrics distinguish unavailable readings from zero and keep probes off the control loop (#2155)
|
||||
|
||||
- The audiobook result is now a synced-lyrics player: chapter text follows playback with the current word highlighted and click-to-seek, timed from the render's own chapter durations with a karaoke-style even split — no ASR pass, fully local (#1766) — thanks @mvanhorn!
|
||||
- The dub CAST strip expands into a project-level casting board: drag voice chips (clone profiles, design presets, Default) onto speaker rows — or pick from a keyboard listbox — writing the same per-speaker cast fields as the existing dropdowns (#1767) — thanks @mvanhorn!
|
||||
- Studio's new Convert method turns a dropped or recorded clip into an existing voice profile's voice, with optional source-duration matching (#1765) — thanks @mvanhorn!
|
||||
|
||||
@@ -1,502 +1,101 @@
|
||||
<div align="center">
|
||||
|
||||
<h3>NOTE: Electron Rewrite Ongoing: Please dont't create desktop app related issues and pr</h3>
|
||||
|
||||
<p><img src="docs/logo.png" alt="VoiceStudio logo" width="120" height="120" /></p>
|
||||
<img src="docs/logo.png" alt="VoiceStudio" width="88" />
|
||||
<h1>VoiceStudio</h1>
|
||||
<p>
|
||||
<a href="https://trendshift.io/repositories/28176?utm_source=repository-badge&utm_medium=badge&utm_campaign=badge-repository-28176" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/repositories/28176" alt="VoiceStudio ranking on Trendshift" width="220" height="48" /></a>
|
||||
</p>
|
||||
<p><sub>Previously OmniVoice-Studio</sub></p>
|
||||
<h3>Clone voices, dub video, dictate, and produce long-form audio on your own hardware.</h3>
|
||||
<p>16 TTS engines · 11 ASR engines · 646-language catalogue · macOS, Windows, Linux, and Docker</p>
|
||||
<p>No account, API key, subscription, or usage meter for the local workflow.</p>
|
||||
|
||||
<p><strong>Open source voice cloning and workflow engine. Build local.</strong></p>
|
||||
<p>
|
||||
<a href="#install">Install</a> ·
|
||||
<a href="#features">Features</a> ·
|
||||
<a href="#comparison">Compare</a> ·
|
||||
<a href="#requirements">Requirements</a> ·
|
||||
<a href="#hardware-recommendations">Hardware</a> ·
|
||||
<a href="#engines">Engines</a> ·
|
||||
<a href="#architecture">Architecture</a> ·
|
||||
<a href="#api">API</a> ·
|
||||
<a href="https://voicestudio.sh/?utm_source=github&utm_medium=readme&utm_campaign=project">Website</a> ·
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest">Download</a> ·
|
||||
<a href="#get-started">Get started</a> ·
|
||||
<a href="#documentation">Docs</a> ·
|
||||
<a href="#faq">FAQ</a> ·
|
||||
<a href="README_CN.md"><strong>简体中文</strong></a>
|
||||
<a href="https://discord.gg/bzQavDfVV9">Discord</a> ·
|
||||
<a href="README_CN.md">简体中文</a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/debpalash/VoiceStudio/ci.yml?branch=main&style=flat-square&label=CI" alt="CI status" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/stargazers"><img src="https://img.shields.io/github/stars/debpalash/VoiceStudio?style=flat-square&color=f59e0b" alt="GitHub stars" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases"><img src="https://img.shields.io/github/downloads/debpalash/VoiceStudio/total?style=flat-square&color=8b5cf6&label=downloads" alt="Total downloads" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/VoiceStudio?style=flat-square&color=10b981" alt="Latest release" /></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="AGPL-3.0 license" /></a>
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Discord-Community-5865F2?style=flat-square&logo=discord&logoColor=white" alt="Discord community" /></a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/badge/Download-macOS_·_Windows_·_Linux-10b981?style=for-the-badge" alt="Download VoiceStudio" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/debpalash/VoiceStudio/ci.yml?branch=main" alt="CI" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/VoiceStudio" alt="Latest release" /></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue" alt="AGPL-3.0" /></a>
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div align="center">
|
||||
<img src="docs/media/0.5.0/quick-switch.gif" alt="Switching TTS engines from the VoiceStudio status bar" width="100%" />
|
||||
</div>
|
||||
|
||||
|
||||
> [!WARNING]
|
||||
> **Active beta.** Use the [latest release](https://github.com/debpalash/VoiceStudio/releases/latest) for stable work. `main` contains the newest fixes and may change between releases. Report problems through [GitHub Issues](https://github.com/debpalash/VoiceStudio/issues).
|
||||
## Your voice. Your workflow.
|
||||
|
||||
## At a glance
|
||||
| Create | Produce | Connect |
|
||||
| :--- | :--- | :--- |
|
||||
| Clone a voice or design your own | Dub videos with timed speech | Local API & MCP for agents |
|
||||
| Dictate with a floating widget | Stories, audiobooks & batch jobs | Optional remote workers |
|
||||
|
||||
| | VoiceStudio |
|
||||
|---|---|
|
||||
| **Workflows** | Voice cloning and design, video dubbing, dictation, stories, audiobooks, batch generation |
|
||||
| **Language catalogue** | 646 TTS languages; actual coverage and quality depend on the selected engine |
|
||||
| **Engines** | 16 TTS · 11 ASR · switch in Model Catalogue or with <kbd>Ctrl</kbd>/<kbd>Cmd</kbd>+<kbd>E</kbd> |
|
||||
| **Platforms** | macOS 13.3+ on Apple Silicon · Windows 10/11 x64 · Linux x86_64 with glibc 2.39+ |
|
||||
| **Compute** | CUDA · Apple Silicon MPS/MLX · ROCm on Linux · CPU · optional remote workers |
|
||||
| **Interfaces** | Desktop app · local REST/SSE/WebSocket API · OpenAI-compatible audio API · MCP Server |
|
||||
| **Storage** | Voices, projects, settings, and outputs stay on the machine by default |
|
||||
| **License** | AGPL-3.0 application; downloaded models keep their upstream terms |
|
||||
Start with **VoiceStudio** (default, powered by k2-fsa/OmniVoice), or choose another engine. [Features & engine catalog](docs/feature-catalog.md).
|
||||
|
||||
The Voice workspace starts with three tabs: **From audio** for cloning, **By design** for creating a voice, and **Convert** for speech-to-speech conversion. Each tab displays its own workflow, with Synthesize Audio or Convert pinned below the scrolling form. The top-bar **Engines** panel combines engine selection, loaded models, and unload/flush controls; <kbd>Ctrl</kbd>/<kbd>Cmd</kbd>+<kbd>E</kbd> opens it. The searchable language picker shares Dubbing’s flags and language list layout, selects one output language, and retains Auto and the full cloning catalogue. Language options flow into multiple columns when space allows. Expand **Workspaces** in the sidebar to reveal navigation labels; Escape collapses it.
|
||||
Local workflows run on your hardware. Remote services are optional; usage analytics requires consent.
|
||||
|
||||
Dubbing starts with file upload or URL import and nearby language choices. Its **Projects** panel lists previous dubs so they can be reopened by clicking anywhere on a card; action buttons operate independently. Advanced import options include captions and optional YouTube sign-in. Dubbing places playback controls over the video with background blur and combines the waveform and timed transcript in one compact editing surface. Drag the zoomed waveform left or right to pan; click to seek. Translation language and ISO-code controls stay synchronized; Auto clears any previous language code and dialect. Transcript items group editable text, timing and status, and voice controls into three readable rows that wrap with the panel width. Output Options stays compact with the active settings shown in its summary; expand it to change output, timing, or voice matching. Transcript, glossary, and paste controls share a toolbar above the segment editor. Project details, workflow steps, and Generate/Verify/Export actions use an unfilled header.
|
||||
<details>
|
||||
<summary><strong>Explore the workspaces</strong> · Clone, dub, design & models</summary>
|
||||
|
||||
The Audiobook Script editor fills the available workspace beneath its markup toolbar; Voices and Book settings stay in their own tabs.
|
||||
<table>
|
||||
<tr>
|
||||
<td><img src="docs/media/electron/voice-cloning.png" alt="Electron voice cloning workspace with the bundled demo voice" width="100%" /></td>
|
||||
<td><img src="docs/media/electron/dubbing.png" alt="Electron video dubbing workspace" width="100%" /></td>
|
||||
</tr>
|
||||
<tr><td align="center">Voice cloning</td><td align="center">Video dubbing</td></tr>
|
||||
<tr>
|
||||
<td><img src="docs/media/electron/voice-design.png" alt="Describe a voice in the Electron voice design workspace" width="100%" /></td>
|
||||
<td><img src="docs/media/electron/models.png" alt="Install and manage local speech models" width="100%" /></td>
|
||||
</tr>
|
||||
<tr><td align="center">Voice design</td><td align="center">Local models</td></tr>
|
||||
</table>
|
||||
|
||||
Output settings use aligned rows; review status appears before the collapsible transcript and glossary. Glossary terms have labelled entry fields and an explicit edit action. Launchpad arranges recent files and saved voices side by side when space allows, with responsive card grids and visible Open actions.
|
||||
<img width="2628" height="1950" alt="VoiceStudio desktop workspace" src="https://github.com/user-attachments/assets/b474497d-a453-49a3-a2dd-f023ec6b7659" />
|
||||
|
||||
The casting board shows icon-based voice cards and searchable selectors for each speaker. Drag a card onto a speaker or choose a voice from that speaker’s menu.
|
||||
</details>
|
||||
|
||||
<a id="install"></a>
|
||||
## Get started
|
||||
|
||||
## Install
|
||||
Download from [Releases](https://github.com/debpalash/VoiceStudio/releases/latest), then follow your platform guide:
|
||||
|
||||
Download a package from the [latest release](https://github.com/debpalash/VoiceStudio/releases/latest), then follow the platform guide.
|
||||
**[macOS](docs/install/macos.md) · [Windows](docs/install/windows.md) · [Linux](docs/install/linux.md) · [Docker](docs/install/docker.md)**
|
||||
|
||||
| Platform | Package | Guide |
|
||||
|---|---|---|
|
||||
| macOS 13.3+ | Apple Silicon DMG | [Install on macOS](docs/install/macos.md) |
|
||||
| Windows 10/11 | x64 MSI; choose the current-user build when listed to install without admin access | [Install on Windows](docs/install/windows.md#install-pre-built-msi) |
|
||||
| Linux | AppImage, x86_64 with glibc 2.39+ | [Install on Linux](docs/install/linux.md) |
|
||||
| Docker | Linux/AMD64 images; CUDA, ROCm, CPU, and worker-only GPU profiles | [Run with Docker](docs/install/docker.md) |
|
||||
Open **Voice cloning**, choose a voice or add a clean reference recording, enter your text, and generate. Install the required model when prompted. Hardware needs vary by engine; see [performance](docs/performance.md).
|
||||
|
||||
First launch creates a managed Python environment and downloads the default model. Later launches reuse both.
|
||||
|
||||
> [!NOTE]
|
||||
> On macOS, first launch needs a one-time right-click, then **Open** approval. Intel Macs cannot run the local Python backend; use a [remote backend](docs/install/macos.md) instead.
|
||||
|
||||
### Quick Docker run
|
||||
|
||||
The published images are **`linux/amd64` only**. On Apple Silicon, use the
|
||||
[native macOS app](docs/install/macos.md) for GPU acceleration. ARM64 hosts
|
||||
should read the [architecture requirements](docs/install/docker.md#architecture)
|
||||
before pulling an image.
|
||||
|
||||
```bash
|
||||
docker run -d -p 127.0.0.1:3900:3900 -v omnivoice-data:/app/omnivoice_data --name voicestudio palashdeb/omnivoice-studio:stable
|
||||
```
|
||||
|
||||
### First voice
|
||||
|
||||
1. Launch VoiceStudio and open **Voice Cloning**.
|
||||
2. Add a clean voice sample. Three seconds works; 5 to 15 seconds usually gives a better prompt.
|
||||
3. Enter text, choose a language, then select **Generate**.
|
||||
|
||||
> [!TIP]
|
||||
> **Try without installing:** Run VoiceStudio in the cloud via the [Google Colab notebook](https://colab.research.google.com/github/debpalash/VoiceStudio/blob/main/notebooks/OmniVoice_Studio_Colab.ipynb). Explore audio quality comparisons in [benchmarks](docs/benchmarks.md) and prompt design tips in [expressive speech](docs/expressive-speech.md).
|
||||
|
||||
### Audio samples
|
||||
|
||||
Listen to sample outputs produced locally with VoiceStudio:
|
||||
|
||||
| Workflow | Prompt / Reference Audio | Generated Audio |
|
||||
|---|---|---|
|
||||
| **Voice Cloning** | [demo_voice.wav](backend/assets/samples/demo_voice.wav) | [demo_clone_output.wav](backend/assets/samples/demo_clone_output.wav) |
|
||||
| **Voice Design** (US News Anchor) | *"Clear, authoritative American broadcast tone"* | [demo_voice_design_us_news_anchor.wav](backend/assets/samples/voice_design/demo_voice_design_us_news_anchor.wav) |
|
||||
| **Voice Design** (UK Audiobook) | *"Warm, expressive British storytelling voice"* | [demo_voice_design_audiobook_uk_narrator.wav](backend/assets/samples/voice_design/demo_voice_design_audiobook_uk_narrator.wav) |
|
||||
| **Video Dubbing** (Multilingual) | [source.src.wav](backend/assets/samples/demo/dubbing/source.src.wav) | [Spanish](backend/assets/samples/demo/dubbing/dubbed_es.src.wav) · [French](backend/assets/samples/demo/dubbing/dubbed_fr.src.wav) · [Japanese](backend/assets/samples/demo/dubbing/dubbed_ja.src.wav) · [Chinese](backend/assets/samples/demo/dubbing/dubbed_zh.src.wav) |
|
||||
|
||||
### Run from source
|
||||
|
||||
Install the [development prerequisites](.github/CONTRIBUTING.md#development-setup) (Node 20+/Bun and Python 3.11+), then:
|
||||
<details>
|
||||
<summary><strong>Run the Electron preview from source</strong></summary>
|
||||
|
||||
```bash
|
||||
git clone https://github.com/debpalash/VoiceStudio.git
|
||||
cd VoiceStudio
|
||||
bun install
|
||||
bun run desktop
|
||||
bun run dev
|
||||
```
|
||||
|
||||
The desktop launcher configures Python dependencies on first run via `uv` automatically. Use `bun run dev` for the browser UI. See [Contributing](.github/CONTRIBUTING.md) for services, tests, and platform packages.
|
||||
See [Electron setup](electron/README.md) for prerequisites and backend configuration.
|
||||
|
||||
### If setup fails
|
||||
</details>
|
||||
|
||||
- Run **Settings → About → Run self-check** or `uv run python backend/main.py --diagnose --deep`.
|
||||
- Check [install troubleshooting](docs/install/troubleshooting.md).
|
||||
- Save a scrubbed diagnostic bundle from the app when opening an issue.
|
||||
- For slow generation, compare [measured benchmarks](docs/benchmarks.md) and [performance settings](docs/performance.md).
|
||||
|
||||
<a id="features"></a>
|
||||
|
||||
## Features
|
||||
|
||||
| Area | Included |
|
||||
|---|---|
|
||||
| **Voice Cloning** | Zero-shot synthesis from a short reference clip ([guide](docs/engines/README.md)) |
|
||||
| **Voice Design** | Create a voice from age, accent, pitch, style, and delivery instructions ([expressive speech](docs/expressive-speech.md)) |
|
||||
| **Video Dubbing** | Transcribe, translate, preserve speakers, synthesize, and export video; compact translation settings include track selection, and completed dubs flag timing issues for review ([export guide](docs/dubbing/export.md)) |
|
||||
| **Stories and audiobooks** | Multi-voice scripts · EPUB/PDF import · chapter rendering · `.m4b` export |
|
||||
| **[Dictation Widget](docs/features/dictation.md)** | System-wide shortcut, live transcription, optional local-LLM cleanup |
|
||||
| **Vocal Isolation** | Demucs speech/background separation |
|
||||
| **Speaker Diarization** | Pyannote and WhisperX speaker assignment ([guide](docs/features/diarization.md)) |
|
||||
| **Batch Queue** | Queue large sets of audio and video jobs with per-job progress, or watch a local folder for new videos |
|
||||
| **Model Catalogue** | Install, remove, select, and route TTS, ASR, and LLM models ([catalogue](docs/engines/README.md)) |
|
||||
| **Remote Model Downloads** | Install models on enrolled remote workers with live progress ([guide](docs/downloading-models.md)) |
|
||||
| **GPU Auto-Detect** | CUDA, MPS, ROCm, and CPU routing with per-engine checks ([performance](docs/performance.md)) |
|
||||
| **AI Watermark** | AudioSeal embedding and detection |
|
||||
| **MCP Server** | Synthesis and transcription tools for MCP clients ([guide](docs/mcp.md)) |
|
||||
| **Diagnostics** | Self-checks, error journal, logs, and scrubbed support bundles ([troubleshooting](docs/install/troubleshooting.md)) |
|
||||
| **Local-first** | Core creation stays local; network-backed features are explicit opt-ins |
|
||||
| **Extensible** | Registry-based TTS, ASR, and plugin interfaces ([acceptance](docs/engine-acceptance.md)) |
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td width="50%"><img src="docs/media/0.5.0/catalogue.png" alt="VoiceStudio Model Catalogue" width="100%" /></td>
|
||||
<td width="50%"><img src="docs/media/0.5.0/gallery-save.png" alt="Saving a gallery voice as a local profile" width="100%" /></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center"><sub>Model Catalogue: engine, device, and install state</sub></td>
|
||||
<td align="center"><sub>Gallery: save a shared voice as a local profile</sub></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
<a id="comparison"></a>
|
||||
|
||||
## Comparison
|
||||
|
||||
VoiceStudio trades managed cloud compute for local control. This is the practical difference:
|
||||
|
||||
| | **VoiceStudio** | **Typical hosted voice service** |
|
||||
|---|---|---|
|
||||
| **Best fit** | Private, offline, self-hosted, or high-volume work | Fast setup without local model management |
|
||||
| **Data path** | Local by default; remote features are opt-in | Audio and text are processed by the provider |
|
||||
| **Cost model** | Free software; you supply the hardware | Subscription, credits, or metered API use |
|
||||
| **Setup** | Install the app and model weights | Create an account and use the web app or API |
|
||||
| **Performance** | Depends on your engine and hardware | Provider manages compute and scaling |
|
||||
| **Offline use** | Yes, after required models are installed | Usually requires a network connection |
|
||||
| **Customization** | Source, engines, models, API, and routing are open | Limited to provider options |
|
||||
| **Maintenance** | You manage updates, disk, and compute | Provider manages infrastructure |
|
||||
|
||||
<a id="requirements"></a>
|
||||
|
||||
## Requirements
|
||||
|
||||
Requirements vary by engine. These values cover the default local workflow.
|
||||
|
||||
| | **Minimum** | **Recommended** |
|
||||
|---|---|---|
|
||||
| **OS** | Windows 10 x64 · macOS 13.3 Apple Silicon · Linux x86_64 with glibc 2.39+ | Current supported OS release |
|
||||
| **RAM** | 8 GB | 16 GB+ |
|
||||
| **Disk** | 10 GB free | 20 GB+ SSD |
|
||||
| **GPU** | Optional; CPU mode is supported | NVIDIA CUDA or Apple Silicon |
|
||||
| **VRAM** | 4 GB when using a GPU | 8 GB+; large optional engines need more |
|
||||
| **Python from source** | 3.11+ | 3.11 or 3.12 |
|
||||
|
||||
ROCm is Linux-only and opt-in. Windows AMD/Ryzen AI uses CPU. Systems with limited VRAM offload work to CPU when required. See [performance](docs/performance.md), [benchmarks](docs/benchmarks.md), and [engine disk usage](docs/engines/disk-usage.md).
|
||||
|
||||
<a id="hardware-recommendations"></a>
|
||||
|
||||
### Recommended stack by hardware
|
||||
|
||||
| Hardware | Recommended TTS | Recommended ASR | Why |
|
||||
|---|---|---|---|
|
||||
| **Apple Silicon (M1–M4)** | [MLX-Audio](docs/engines/mlx-audio.md) · [OmniVoice](docs/engines/omnivoice.md) (MPS) | [MLX Whisper](docs/engines/mlx-whisper.md) · [Parakeet MLX](docs/engines/parakeet-mlx.md) | Native unified memory, lowest latency on macOS |
|
||||
| **NVIDIA GPU (8 GB+ VRAM)** | [OmniVoice](docs/engines/omnivoice.md) · [CosyVoice 3](docs/engines/cosyvoice.md) | [WhisperX](docs/engines/whisperx.md) | High-fidelity zero-shot cloning, word timestamps, diarization |
|
||||
| **Low VRAM / CPU-only** | [PocketTTS](docs/engines/pockettts.md) · [Sherpa-ONNX](docs/engines/sherpa-onnx.md) · [KittenTTS](docs/engines/kittentts.md) | [Moonshine](docs/engines/moonshine.md) · [Faster-Whisper](docs/engines/faster-whisper.md) (`int8`) | Low memory footprint, optimized CPU inference |
|
||||
|
||||
<a id="engines"></a>
|
||||
|
||||
## Engines
|
||||
|
||||
Engine support is capability-specific. Check cloning, language, platform, memory, and license before choosing one. Full setup guides: [docs/engines](docs/engines/README.md).
|
||||
|
||||
<a id="tts-engines"></a>
|
||||
|
||||
### Text to speech
|
||||
|
||||
| Engine | Languages | Clone | Instruct | Linux | macOS ARM | Windows | License |
|
||||
|---|:---:|:---:|:---:|:---:|:---:|:---:|---|
|
||||
| [**VoiceStudio** (default, powered by k2-fsa/OmniVoice)](docs/engines/omnivoice.md) | 600+ | Yes | Yes | CUDA/CPU | MPS | CUDA/CPU | [AGPL-3.0](LICENSE) app · [Apache-2.0 code, CC-BY-NC weights](https://huggingface.co/k2-fsa/OmniVoice#license)³ |
|
||||
| [**CosyVoice 3**](docs/engines/cosyvoice.md) | 9 + 18 dialects | Yes | Yes | CUDA/CPU | CPU | CUDA/CPU | Apache-2.0 |
|
||||
| [**GPT-SoVITS**](docs/engines/gpt-sovits.md) | 5 | Yes | No | CUDA/CPU | No | CUDA/CPU | MIT |
|
||||
| [**VoxCPM2**](docs/engines/voxcpm2.md) | 30 | Yes | Yes | CUDA/CPU | MPS | CUDA/CPU | Apache-2.0 |
|
||||
| [**MOSS-TTS-Nano**](docs/engines/moss-tts-nano.md) | 20 | Yes | No | CUDA/CPU | CPU | CUDA/CPU | Apache-2.0 |
|
||||
| [**KittenTTS**](docs/engines/kittentts.md) | English | No | No | CPU | CPU | CPU | MIT |
|
||||
| [**MLX-Audio**](docs/engines/mlx-audio.md) | Model-dependent | Varies | Varies | No | MLX | No | Varies |
|
||||
| [**Sherpa-ONNX**](docs/engines/sherpa-onnx.md) | 20+ | No | No | CUDA/CPU | CPU | CUDA/CPU | Apache-2.0 |
|
||||
| [**IndexTTS 2.5** ⚡](docs/engines/indextts.md) | ZH · EN · JA · ES · AR | Yes | No | CUDA/CPU | CPU | CUDA/CPU | Bilibili model license¹ |
|
||||
| [**OmniVoice GGUF** ⚡](docs/engines/omnivoice-gguf.md) | 600+ | Yes | Yes | CUDA/CPU | MPS/CPU | CUDA/CPU | [AGPL-3.0](LICENSE) app · [review the derivative model terms](https://huggingface.co/Serveurperso/OmniVoice-GGUF#license)³ |
|
||||
| [**OmniVoice (subprocess; opt-in off MPS)** ⚡](docs/engines/omnivoice-subprocess.md) | 600+ | Yes | Yes | CUDA/CPU | MPS via default OmniVoice | CUDA/CPU | [AGPL-3.0](LICENSE) app · [Apache-2.0 code, CC-BY-NC weights](https://huggingface.co/k2-fsa/OmniVoice#license)³ |
|
||||
| [**PocketTTS** ⚡](docs/engines/pockettts.md) | EN · FR · DE · PT · IT · ES | Yes | No | CPU | CPU | CPU | CC-BY-4.0, gated² |
|
||||
| [**Supertonic 3** ⚡](docs/engines/supertonic3.md) | 31 | No | No | CPU | CPU | CPU | OpenRAIL-M |
|
||||
| [**MOSS-TTS-v1.5** ⚡](docs/engines/moss-tts-v15.md) | 31 | Yes | No | CUDA/CPU | CPU | CUDA/CPU | Apache-2.0 |
|
||||
| [**dots.tts** ⚡](docs/engines/dots-tts.md) | 24 | Yes | No | CUDA/CPU | CPU | No | Apache-2.0 |
|
||||
| [**Confucius4-TTS** ⚡](docs/engines/confucius4-tts.md) | 14 | Yes | No | CUDA/CPU | CPU | CUDA/CPU | Apache-2.0 |
|
||||
|
||||
⚡ Installed or registered on demand.
|
||||
|
||||
¹ IndexTTS 2.5 requires a separate written Bilibili license above 100 million monthly active users or RMB 1 billion annual revenue. Review the [model license](https://huggingface.co/IndexTeam/IndexTTS-2.5/blob/main/LICENSE).
|
||||
|
||||
² PocketTTS shows its gated-access and CC-BY-4.0 terms before first use.
|
||||
|
||||
³ The OmniVoice snapshot also includes an audio tokenizer under separate [Boson Higgs Audio 2 and Meta Llama community terms](https://huggingface.co/k2-fsa/OmniVoice/blob/main/audio_tokenizer/LICENSE). VoiceStudio's application license does not replace model or tokenizer terms.
|
||||
|
||||
Clone-less engines cannot preserve a reference speaker in dubbing or pinned-voice batch jobs. VoiceStudio rejects those jobs instead of silently changing engines. Heavy engines have separate memory and platform limits; check their engine guide first.
|
||||
|
||||
<a id="asr-engines"></a>
|
||||
|
||||
### Speech to text
|
||||
|
||||
| Engine | ID | Languages | Best fit |
|
||||
|---|---|:---:|---|
|
||||
| [**WhisperX** (default)](docs/engines/whisperx.md) | `whisperx` | ~100 | Dubbing, subtitles, word-level timing |
|
||||
| [**Faster-Whisper**](docs/engines/faster-whisper.md) | `faster-whisper` | ~100 | General cross-platform transcription |
|
||||
| [**Faster-Whisper (isolated)**](docs/engines/faster-whisper-isolated.md) | `faster-whisper-isolated` | ~100 | Crash-isolated batch transcription |
|
||||
| [**MLX Whisper**](docs/engines/mlx-whisper.md) | `mlx-whisper` | ~100 | Apple Silicon |
|
||||
| [**PyTorch Whisper**](docs/engines/pytorch-whisper.md) | `pytorch-whisper` | ~100 | CUDA, MPS, and CPU fallback |
|
||||
| [**Parakeet TDT**](docs/engines/nemo-parakeet.md) | `nemo-parakeet` | English + 25 EU | Fast CPU/CUDA transcription |
|
||||
| [**Parakeet TDT v3 (MLX)**](docs/engines/parakeet-mlx.md) | `parakeet-mlx` | 25 EU | Apple Silicon dictation and word timestamps |
|
||||
| [**Moonshine**](docs/engines/moonshine.md) | `moonshine` | English | Low-power, low-latency ONNX |
|
||||
| [**FunASR**](docs/engines/funasr.md) | `funasr` | 50+ | VAD and inline diarization |
|
||||
| [**sherpa-onnx** (live dictation)](docs/engines/sherpa-onnx-asr.md) | `sherpa-onnx-asr` | Model-dependent | Streaming CPU dictation |
|
||||
| [**OpenAI-compatible** ⚠️ configured server](docs/engines/openai-compatible-asr.md) | `openai-compat-asr` | Server-dependent | Local gigastt/Qwen3-ASR or a remote endpoint; audio goes only to that server |
|
||||
|
||||
WhisperX and Faster-Whisper retry with `int8` when efficient `float16` is unavailable. Pin `ASR_COMPUTE_TYPE=int8` or `float32` only if automatic selection still fails.
|
||||
|
||||
<a id="architecture"></a>
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
Tauri v2 desktop shell (Rust)
|
||||
│ IPC
|
||||
React + Vite UI
|
||||
│ HTTP · SSE · WebSocket on localhost:3900
|
||||
FastAPI backend
|
||||
├── TTS / ASR engine registries
|
||||
├── dubbing / audio / long-form pipelines
|
||||
├── OpenAI-compatible API and MCP server
|
||||
└── SQLite + Alembic → omnivoice_data/
|
||||
```
|
||||
|
||||
| Layer | Path | Responsibility |
|
||||
|---|---|---|
|
||||
| Desktop shell | `frontend/src-tauri/` | Window lifecycle, tray, shortcuts, updater, sidecar bootstrap |
|
||||
| Frontend | `frontend/src/` | React UI, Zustand state, API and event clients, i18n |
|
||||
| API | `backend/api/` | REST routes, schemas, auth boundaries, streaming |
|
||||
| Core services | `backend/services/` | Generation, dubbing, audio processing, persistence |
|
||||
| Engines | `backend/engines/` | Isolated and optional engine adapters |
|
||||
| Worker system | `backend/worker/` | Authenticated remote compute and job transport |
|
||||
| Data | `omnivoice_data/` | Projects, voices, settings, logs, and SQLite state |
|
||||
| Delivery | `scripts/`, `deploy/`, `.github/workflows/` | Development, packaging, containers, releases, CI |
|
||||
|
||||
### Network boundary
|
||||
|
||||
- The desktop talks to a loopback-only backend on `localhost:3900`.
|
||||
- Loopback API calls need no server key. Remote access requires a share PIN or API key.
|
||||
- Remote workers and OpenAI-compatible ASR are opt-in. Loopback ASR may use HTTP and keeps audio on the machine; non-loopback endpoints require HTTPS, and redirects are not followed.
|
||||
- Analytics is off until consent. If enabled, it sends allowlisted, content-free usage metadata. It never sends text, audio, file names, or projects.
|
||||
|
||||
<a id="api"></a>
|
||||
|
||||
## Local speech platform and OpenAI-compatible API
|
||||
|
||||
Point an OpenAI-compatible audio client at the local backend:
|
||||
|
||||
```diff
|
||||
- base_url="https://api.openai.com/v1"
|
||||
+ base_url="http://localhost:3900/v1"
|
||||
```
|
||||
|
||||
| Endpoint | Purpose |
|
||||
|---|---|
|
||||
| `POST /v1/audio/speech` | TTS to `mp3`, `opus`, `aac`, `flac`, `wav`, or `pcm`; select a profile with `voice` and an engine with `model` |
|
||||
| `POST /v1/audio/transcriptions` | STT to `json`, `text`, `verbose_json`, `srt`, or `vtt` |
|
||||
| `WS /v1/audio/transcriptions/stream` | Live PCM/WebM transcription with partial, utterance, and session-final events |
|
||||
| `GET /.well-known/voicestudio-speech` | Discover HTTP, WebSocket, MCP, and native dictation-control transports |
|
||||
| `GET /v1/audio/voices` | List local voice profiles and engines |
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:3900/v1", api_key="local")
|
||||
|
||||
with client.audio.speech.with_streaming_response.create(
|
||||
model="tts-1",
|
||||
voice="<profile-id>",
|
||||
input="Made on my own hardware.",
|
||||
response_format="wav",
|
||||
) as response:
|
||||
response.stream_to_file("speech.wav")
|
||||
```
|
||||
|
||||
```bash
|
||||
# Quick test via cURL
|
||||
curl http://localhost:3900/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "tts-1", "input": "Made on my own hardware.", "voice": "default", "response_format": "wav"}' \
|
||||
--output speech.wav
|
||||
```
|
||||
|
||||
The bundled Rust control sidecar lets Herdr, coding agents, VS Code, desktop apps,
|
||||
and TUIs trigger the system-wide dictation flow or reuse its native text
|
||||
insertion. See the [speech platform guide](docs/speech-platform.md). The full API
|
||||
reference is in **Settings → OpenAPI Reference**. For LAN, Tailscale, or proxy
|
||||
access, read [API authentication](docs/api-auth.md) before exposing the backend.
|
||||
|
||||
### Agent skills
|
||||
|
||||
Install the VoiceStudio skills for Claude Code, Codex, Cursor, and other [skills.sh](https://skills.sh)-compatible agents:
|
||||
|
||||
```bash
|
||||
npx skills add debpalash/VoiceStudio
|
||||
```
|
||||
|
||||
- `omnivoice`: synthesize speech and transcribe audio through local VoiceStudio.
|
||||
- `oss-maintainer`: the repository's open-source maintenance workflow.
|
||||
|
||||
### Model Context Protocol (MCP)
|
||||
|
||||
VoiceStudio mounts an MCP server at `http://localhost:3900/mcp` for Claude Desktop, Cursor, and AI agents:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicestudio": {
|
||||
"url": "http://localhost:3900/mcp"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For clients requiring stdio transport, use the bundled local shim (`docs/mcp.json`):
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicestudio": {
|
||||
"command": "python",
|
||||
"args": ["-m", "backend.mcp_shim"],
|
||||
"cwd": "/path/to/VoiceStudio"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
See the [MCP guide](docs/mcp.md) for tools (`generate_speech`, `clone_voice`, `transcribe`), file streaming modes, and client bindings.
|
||||
|
||||
### Google Colab
|
||||
|
||||
[](https://colab.research.google.com/github/debpalash/VoiceStudio/blob/main/notebooks/OmniVoice_Studio_Colab.ipynb)
|
||||
|
||||
The [notebook](notebooks/OmniVoice_Studio_Colab.ipynb) runs the app and web UI on a Colab GPU. Colab is remote compute, so uploaded audio and project data do not remain local to your machine.
|
||||
|
||||
<a id="documentation"></a>
|
||||
> **Electron is the primary desktop app.** The next desktop release ships Electron, with one final Tauri sunset update. Bug reports and contributions remain welcome; include the app version and whether you use Electron or Tauri.
|
||||
|
||||
## Documentation
|
||||
|
||||
| Need | Read |
|
||||
| Need | Start here |
|
||||
|---|---|
|
||||
| Install | [macOS](docs/install/macos.md) · [Windows](docs/install/windows.md) · [Linux](docs/install/linux.md) · [Docker](docs/install/docker.md) |
|
||||
| Fix setup | [Troubleshooting](docs/install/troubleshooting.md) · [model downloads](docs/downloading-models.md) · [Hugging Face token](docs/setup/huggingface-token.md) |
|
||||
| Choose an engine | [Engine guides](docs/engines/README.md) · [benchmarks](docs/benchmarks.md) · [expressive speech](docs/expressive-speech.md) |
|
||||
| Tune hardware | [Performance](docs/performance.md) · [remote workers](docs/remote-workers.md) |
|
||||
| Build integrations | [Speech platform](docs/speech-platform.md) · [Private production API](docs/production-private-api.md) · [API auth](docs/api-auth.md) · [MCP](docs/mcp.md) · [examples](examples/README.md) |
|
||||
| Build VoiceStudio | [Contributing](.github/CONTRIBUTING.md) · [engine acceptance](docs/engine-acceptance.md) |
|
||||
| Track changes | [Changelog](CHANGELOG.md) · [roadmap](docs/ROADMAP.md) · [latest release](https://github.com/debpalash/VoiceStudio/releases/latest) |
|
||||
| Remove everything | [Uninstall guide](docs/install/uninstall.md) |
|
||||
| Setup help | [Troubleshooting](docs/install/troubleshooting.md) · [Model downloads](docs/downloading-models.md) |
|
||||
| Models & audio quality | [Engine guides](docs/engines/README.md) · [Benchmarks](docs/benchmarks.md) |
|
||||
| Integrations | [Local API](docs/speech-platform.md) · [MCP](docs/mcp.md) · [Examples](examples/README.md) |
|
||||
| Development | [Contributing](.github/CONTRIBUTING.md) · [Electron](electron/README.md) · [Changelog](CHANGELOG.md) |
|
||||
|
||||
<a id="faq"></a>
|
||||
Agent skills: `npx skills add debpalash/VoiceStudio` — choose **voicestudio** for audio workflows or **voicestudio-maintainer** for repository maintenance.
|
||||
|
||||
## FAQ
|
||||
## Sponsors
|
||||
|
||||
<details>
|
||||
<summary><strong>Does it work on Apple Silicon and Intel Macs?</strong></summary>
|
||||
<a href="https://forms.gle/2PYCvd39hbwijzX37"><img src="docs/media/sponsor-slot.svg" alt="Your brand — apply for a featured VoiceStudio sponsor slot" width="640" /></a>
|
||||
|
||||
Apple Silicon is supported with MPS and MLX options. Intel Macs cannot run the local backend because current PyTorch wheels are unavailable; they can connect to a remote backend. See [macOS installation](docs/install/macos.md).
|
||||
</details>
|
||||
**Become a featured partner.** [Apply for a paid placement](https://forms.gle/2PYCvd39hbwijzX37) · [Email us](mailto:partner@voicestudio.sh)
|
||||
|
||||
<details>
|
||||
<summary><strong>How much VRAM do I need?</strong></summary>
|
||||
Support development: [Ko-fi](https://ko-fi.com/debpalash) · [PayPal](https://paypal.me/palashCoder) · [Sponsorship details](SPONSORS.md)
|
||||
|
||||
A GPU is optional. Use 4 GB VRAM as the minimum for accelerated work and 8 GB+ for the default multi-stage workflow. Large optional engines can require 12 to 16 GB or more. Check the [benchmarks](docs/benchmarks.md) and engine guide.
|
||||
</details>
|
||||
## License & responsible use
|
||||
|
||||
<details>
|
||||
<summary><strong>Why does a longer reference clip not always improve the clone?</strong></summary>
|
||||
|
||||
Cloning is zero-shot: the clip is a prompt, not training data. Use 5 to 15 seconds of one speaker, close to the microphone, without music, noise, or reverb. Match the tone and pace you want in the output. For training, see [data preparation](docs/data_preparation.md) and [training](docs/training.md).
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>Can I use generated audio commercially?</strong></summary>
|
||||
|
||||
VoiceStudio's application license does not restrict generated audio, but it does not grant rights under a model's separate terms. The default OmniVoice repository labels its pretrained weights CC-BY-NC and includes a tokenizer under separate community terms. Review the selected model terms before commercial use.
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>Does VoiceStudio collect data?</strong></summary>
|
||||
|
||||
Not unless you opt in. Analytics is off by default and skipping consent keeps it off. When enabled, the app sends allowlisted, content-free usage metadata. Text, audio, file names, voices, and projects are excluded. Change this at **Settings → Privacy**.
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><strong>How do I remove VoiceStudio and its data?</strong></summary>
|
||||
|
||||
Use `scripts/uninstall.sh` on macOS/Linux or `scripts\uninstall.ps1` on Windows. Both show a dry run before deletion. See the [uninstall guide](docs/install/uninstall.md) for every path.
|
||||
</details>
|
||||
|
||||
## Community and contributing
|
||||
|
||||
- [GitHub Issues](https://github.com/debpalash/VoiceStudio/issues) for reproducible bugs and feature requests.
|
||||
- [Discord](https://discord.gg/bzQavDfVV9) for setup help and project discussion.
|
||||
- [Good first issues](https://github.com/debpalash/VoiceStudio/labels/good%20first%20issue) for a scoped starting point.
|
||||
- [Contributing guide](.github/CONTRIBUTING.md) for setup, tests, and pull requests.
|
||||
|
||||
<p align="center">
|
||||
<a href="https://star-history.com/#debpalash/VoiceStudio&Date">
|
||||
<img src="https://api.star-history.com/svg?repos=debpalash/VoiceStudio&type=Date" alt="Star History Chart" width="100%" />
|
||||
</a>
|
||||
</p>
|
||||
|
||||
## Support development
|
||||
|
||||
VoiceStudio is free and has no paid tier. Donations fund development and infrastructure.
|
||||
|
||||
[Ko-fi](https://ko-fi.com/debpalash) · [PayPal](https://paypal.me/palashCoder) · [Sponsorship details](SPONSORS.md)
|
||||
|
||||
## Responsible use and safety
|
||||
|
||||
VoiceStudio enables zero-shot voice cloning and speech generation on personal hardware. Please use it responsibly:
|
||||
- **Consent:** Only clone or synthesize voices with explicit permission from the speaker.
|
||||
- **Audio provenance:** VoiceStudio integrates [AudioSeal](https://github.com/facebookresearch/audioseal) imperceptible watermarking by default to detect and identify synthetic speech without altering sound quality.
|
||||
- **Local privacy:** For the default local workflow, audio recordings, transcripts, voices, and projects remain strictly on your local disk; data leaves your device only when you explicitly configure remote workers or external ASR endpoints.
|
||||
|
||||
## License
|
||||
|
||||
VoiceStudio is licensed under [AGPL-3.0](LICENSE). You may run it, modify it, and use it internally. The application license itself does not restrict selling generated audio, but downloaded model and tokenizer terms may. If you modify VoiceStudio and provide that modified version as a network service, AGPL requires you to offer the corresponding source under the same license. A commercial license for VoiceStudio-owned code is available for proprietary embedding; it does not relicense third-party models. Contact **VoiceStudio@palash.dev**. See [LICENSE-NOTICE.md](LICENSE-NOTICE.md) for the plain-language scope.
|
||||
|
||||
Optional engines and downloaded models retain their own licenses. The bundled `omnivoice/` Python code is Apache-2.0 upstream; the default downloaded weights and audio tokenizer use separate terms.
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
VoiceStudio builds on [OmniVoice](https://github.com/k2-fsa/OmniVoice), [WhisperX](https://github.com/m-bain/whisperX), [Demucs](https://github.com/facebookresearch/demucs), [Pyannote](https://github.com/pyannote/pyannote-audio), [CTranslate2](https://github.com/OpenNMT/CTranslate2), [AudioSeal](https://github.com/facebookresearch/audioseal), [Tauri](https://tauri.app), [Supertonic](https://huggingface.co/Supertone/supertonic-3), [Sherpa-ONNX](https://github.com/k2-fsa/sherpa-onnx), [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS), and [PocketTTS](https://kyutai.org).
|
||||
|
||||
<div align="center">
|
||||
<strong><a href="https://github.com/debpalash/VoiceStudio/releases/latest">Download VoiceStudio</a></strong> ·
|
||||
<a href="https://github.com/debpalash/VoiceStudio">Star the project</a> ·
|
||||
<a href="https://discord.gg/bzQavDfVV9">Join Discord</a>
|
||||
</div>
|
||||
[AGPL-3.0](LICENSE). Models have their own licenses; review them before commercial use. Clone voices only with permission. See [license details](LICENSE-NOTICE.md).
|
||||
|
||||
+53
-669
@@ -1,697 +1,81 @@
|
||||
*本文档是 [README.md](README.md) 的简体中文翻译；若与英文版有出入，以英文版为准。*
|
||||
|
||||
<div align="center">
|
||||
<img src="docs/logo.png" alt="VoiceStudio 徽标" width="120" height="120" />
|
||||
<img src="docs/logo.png" alt="VoiceStudio" width="88" />
|
||||
<h1>VoiceStudio</h1>
|
||||
<p><sub><em>原名 OmniVoice-Studio</em></sub></p>
|
||||
<h3>创造声音，讲述故事，文件始终属于你。♡</h3>
|
||||
<p>在一个开源桌面工作室里完成克隆、设计、配音、听写和有声书制作。<br/><b>默认本地优先。</b>没有订阅，也没有用量计费；联网服务始终由你主动选择。</p>
|
||||
|
||||
<p><strong>开源声音克隆与工作流引擎。在本地构建。</strong></p>
|
||||
<p>使用本地 AI 克隆声音、翻译配音、语音听写和制作有声书。</p>
|
||||
<p>
|
||||
<a href="#quickstart">快速开始</a> ·
|
||||
<a href="#features">功能</a> ·
|
||||
<a href="#why-voicestudio">为什么选择 VoiceStudio</a> ·
|
||||
<a href="#tts-engines">引擎</a> ·
|
||||
<a href="#openai-api">API</a> ·
|
||||
<a href="#sponsor--donate">捐赠</a> ·
|
||||
<a href="#contributing">参与贡献</a> ·
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest">下载</a> ·
|
||||
<a href="#开始使用">开始使用</a> ·
|
||||
<a href="#文档">文档</a> ·
|
||||
<a href="https://discord.gg/bzQavDfVV9">Discord</a> ·
|
||||
<a href="README.md"><strong>English</strong></a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/debpalash/VoiceStudio/ci.yml?branch=main&style=flat-square&label=CI" alt="CI 状态" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/stargazers"><img src="https://img.shields.io/github/stars/debpalash/VoiceStudio?style=flat-square&color=f59e0b" alt="Star 数" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/VoiceStudio?style=flat-square&color=10b981" alt="版本" /></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="许可证" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/issues"><img src="https://img.shields.io/github/issues/debpalash/VoiceStudio?style=flat-square&color=ef4444" alt="Issues" /></a>
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Discord-Join_Community-5865F2?style=flat-square&logo=discord&logoColor=white" alt="Discord" /></a>
|
||||
<a href="https://ko-fi.com/debpalash"><img src="https://img.shields.io/badge/Ko--fi-Support_Us-FF5E5B?style=flat-square&logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
|
||||
<a href="https://paypal.me/palashCoder"><img src="https://img.shields.io/badge/PayPal-Donate-00457C?style=flat-square&logo=paypal&logoColor=white" alt="PayPal" /></a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/badge/⬇_Download-macOS_·_Windows_·_Linux-10b981?style=for-the-badge" alt="下载最新版本" /></a>
|
||||
<a href="README.md">English</a>
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<br/>
|
||||
|
||||
|
||||
<div align="center">
|
||||
<img src="docs/media/0.5.0/quick-switch.gif" alt="VoiceStudio — 从状态栏快速切换 TTS 引擎" width="100%"/>
|
||||
</div>
|
||||
<p align="center"><sub>新 Electron 桌面界面，使用此分支及内置演示声音录制。正式发布版本的界面可能有所不同。</sub></p>
|
||||
|
||||
> **声音很私人，创作空间也应该真正属于你。** VoiceStudio 的核心流程运行在你的硬件上：克隆、设计、配音、听写，并以 646 种语言创作，不需要订阅，也没有用量计费。联网引擎和服务始终是清晰可见的可选项，而不是隐藏依赖。
|
||||
## 用 VoiceStudio 创作
|
||||
|
||||
> [!WARNING]
|
||||
> **活跃 Beta 阶段。** 各版本之间可能出现故障——如需最新修复，请从源码运行。非常欢迎 Bug 报告和 PR：[提交 Issue](https://github.com/debpalash/VoiceStudio/issues) 或 [加入 Discord](https://discord.gg/bzQavDfVV9)。
|
||||
- **声音克隆与设计**：上传参考录音，或用文字描述你想要的声音。
|
||||
- **视频配音**：转录、翻译、分配说话人，并编辑语音时间轴。
|
||||
- **语音听写**：通过悬浮录音组件录制、转录和复制文字。
|
||||
- **长篇创作**：制作多角色脚本、有声书和批量任务。
|
||||
- **模型管理**：选择语音合成与转录引擎、语言及计算设备。
|
||||
|
||||
<a id="quickstart"></a>
|
||||
本地工作流在你的硬件上运行。远程服务为可选功能；使用情况分析须经同意才会启用。
|
||||
|
||||
## ⚡ 快速开始
|
||||
<table>
|
||||
<tr>
|
||||
<td><img src="docs/media/electron/voice-cloning.png" alt="Electron 声音克隆工作区与内置演示声音" width="100%" /></td>
|
||||
<td><img src="docs/media/electron/dubbing.png" alt="Electron 视频配音工作区" width="100%" /></td>
|
||||
</tr>
|
||||
<tr><td align="center">声音克隆</td><td align="center">视频配音</td></tr>
|
||||
<tr>
|
||||
<td><img src="docs/media/electron/voice-design.png" alt="Electron 声音设计工作区" width="100%" /></td>
|
||||
<td><img src="docs/media/electron/models.png" alt="本地语音模型管理" width="100%" /></td>
|
||||
</tr>
|
||||
<tr><td align="center">声音设计</td><td align="center">本地模型</td></tr>
|
||||
</table>
|
||||
|
||||
<div align="center">
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="下载 macOS DMG" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="下载 Windows MSI" /></a>
|
||||
<a href="https://github.com/debpalash/VoiceStudio/releases/latest"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="下载 Linux AppImage" /></a>
|
||||
<br/>
|
||||
<sub>三个按钮都会打开最新发布页——在资源列表中下载对应你系统的安装包。</sub><br/>
|
||||
<sub><b>macOS：</b>首次启动需要一次性批准——右键点击 → <b>打开</b>（macOS 15 上为 系统设置 → 隐私与安全性 → <b>“仍要打开”</b>）。无需终端。<a href="docs/install/macos.md#gatekeeper-quarantine">为什么？</a> · <b>Intel Mac：</b>不支持本地后端（<a href="https://github.com/debpalash/VoiceStudio/issues/889">#889</a>）——<a href="docs/install/macos.md">详情</a>。</sub>
|
||||
</div>
|
||||
## 开始使用
|
||||
|
||||
选择你的操作系统，按指南从头到尾操作：
|
||||
从 [Releases](https://github.com/debpalash/VoiceStudio/releases/latest) 下载，然后阅读对应平台的安装指南：
|
||||
|
||||
- 🍎 **macOS** — [docs/install/macos.md](docs/install/macos.md)
|
||||
- 🪟 **Windows** — [docs/install/windows.md](docs/install/windows.md)
|
||||
- 🐧 **Linux** — [docs/install/linux.md](docs/install/linux.md)
|
||||
- 🐳 **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
|
||||
**[macOS](docs/install/macos.md) · [Windows](docs/install/windows.md) · [Linux](docs/install/linux.md) · [Docker](docs/install/docker.md)**
|
||||
|
||||
打开声音克隆页面，选择已有声音或添加清晰的参考录音，输入文字并生成。按提示安装所需模型。硬件要求因引擎而异，详见[性能指南](docs/performance.md)。
|
||||
|
||||
**从源码运行 Electron 预览版：**
|
||||
|
||||
```bash
|
||||
# Docker 快速运行 (CPU / 本地环回模式)
|
||||
docker run -d -p 127.0.0.1:3900:3900 -v omnivoice-data:/app/omnivoice_data --name voicestudio palashdeb/omnivoice-studio:stable
|
||||
git clone https://github.com/debpalash/VoiceStudio.git
|
||||
cd VoiceStudio
|
||||
bun install
|
||||
cd electron
|
||||
bun run dev
|
||||
```
|
||||
|
||||
**三步克隆出你的第一个声音：**
|
||||
环境要求和后端配置见 [Electron 开发指南](electron/README.md)。项目仍在积极开发中，可通过 [GitHub Issues](https://github.com/debpalash/VoiceStudio/issues) 反馈问题。
|
||||
|
||||
1. **安装并启动。** 首次启动会自动搭建 Python 运行环境并下载模型权重——启动画面会逐步显示进度（仅首次，需要几分钟；之后即开即用）。
|
||||
2. 从启动台打开**语音克隆**，拖入任意声音的 **3 秒音频**。
|
||||
3. **输入一句话，点击生成。** 音频在你的设备上生成并保存，支持 646 种语言（商业使用前请审阅所选模型与分词器的许可条款）。
|
||||
## 文档
|
||||
|
||||
### 🎧 音频示例
|
||||
|
||||
在线试听 VoiceStudio 本地生成的实际音频样例：
|
||||
|
||||
| 工作流 | 提示词 / 参考音频 | 生成音频 |
|
||||
|---|---|---|
|
||||
| **声音克隆** | [demo_voice.wav](backend/assets/samples/demo_voice.wav) | [demo_clone_output.wav](backend/assets/samples/demo_clone_output.wav) |
|
||||
| **声音设计** (美语新闻主播) | *"清晰、权威的美国广播级音色"* | [demo_voice_design_us_news_anchor.wav](backend/assets/samples/voice_design/demo_voice_design_us_news_anchor.wav) |
|
||||
| **声音设计** (英式有声书) | *"温暖生动的英式故事讲述音色"* | [demo_voice_design_audiobook_uk_narrator.wav](backend/assets/samples/voice_design/demo_voice_design_audiobook_uk_narrator.wav) |
|
||||
| **视频配音** (多语种) | [source.src.wav](backend/assets/samples/demo/dubbing/source.src.wav) | [西班牙语](backend/assets/samples/demo/dubbing/dubbed_es.src.wav) · [法语](backend/assets/samples/demo/dubbing/dubbed_fr.src.wav) · [日语](backend/assets/samples/demo/dubbing/dubbed_ja.src.wav) · [中文](backend/assets/samples/demo/dubbing/dubbed_zh.src.wav) |
|
||||
|
||||
觉得慢？[docs/performance.md](docs/performance.md) 讲清了生成时间到底花在哪里、有哪些调优开关，以及“它变慢了”的三个经典原因。各引擎/设备的实测数据见 [docs/benchmarks.md](docs/benchmarks.md)。
|
||||
|
||||
> 正在从 **[CorentinJ/Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning)**（现已归档）迁移过来？我们有专门的迁移指南：[docs/migration/real-time-voice-cloning.md](docs/migration/real-time-voice-cloning.md)。
|
||||
|
||||
<details>
|
||||
<summary><b>🧰 卡住了？自检、Token 与受限网络</b></summary>
|
||||
|
||||
<br/>
|
||||
|
||||
先运行内置自检——在应用中打开 **设置 → 关于 → “运行自检”**，或在源码检出目录中执行
|
||||
`uv run python backend/main.py --diagnose`（加 `--deep` 还会实际加载当前引擎进行测试）。然后查看
|
||||
[docs/install/troubleshooting.md](docs/install/troubleshooting.md) 中排名前
|
||||
10 的安装错误。运行时出错时，应用内的错误界面会直接深链到对应条目；**设置 → 关于 →
|
||||
“保存诊断包”** 会把脱敏日志与自检报告打包，方便附在 Bug 报告里。
|
||||
|
||||
Hugging Face Token 的配置见
|
||||
[docs/setup/huggingface-token.md](docs/setup/huggingface-token.md)。说话人分离相关的模型访问门槛见
|
||||
[docs/features/diarization.md](docs/features/diarization.md)。下载速度、⚡ 快速下载（Xet）状态，以及受限网络 / 镜像选项见
|
||||
[docs/downloading-models.md](docs/downloading-models.md)。
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
<a id="features"></a>
|
||||
|
||||
## ✨ 功能
|
||||
|
||||
八大主打功能——折叠区里还有十二项等你展开。
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td align="center" width="25%">
|
||||
<h3>🎙️ 语音克隆</h3>
|
||||
<p>3 秒音频 → 复刻任何声音。<br/><b>646 种语言</b>，零样本。</p>
|
||||
</td>
|
||||
<td align="center" width="25%">
|
||||
<h3>🎨 声音设计</h3>
|
||||
<p>性别、年龄、口音、音高、语速、<br/>情感、方言——<b>随心调节</b>。</p>
|
||||
</td>
|
||||
<td align="center" width="25%">
|
||||
<h3>🎬 视频配音</h3>
|
||||
<p>YouTube 链接或文件 → 转录 →<br/>翻译 → 重新配音 → <b>MP4</b>。</p>
|
||||
</td>
|
||||
<td align="center" width="25%">
|
||||
<h3>📖 有声书编辑器</h3>
|
||||
<p>导入文本、EPUB 或 PDF。自动分章、<br/>响度归一、元数据。导出 <b>.m4b</b>。</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
<h3>🎭 故事模式</h3>
|
||||
<p>多声音编辑器。逐行分配声音、<br/>预览、<b>导出完整配音阵容</b>。</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>⌨️ 听写工具</h3>
|
||||
<p>在<b>任何应用</b>中按 <kbd>⌘</kbd>+<kbd>⇧</kbd>+<kbd>Space</kbd>。<br/>转录、自动粘贴、随即消失。</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🔐 本地优先</h3>
|
||||
<p>核心创作流程<br/><b>留在你的设备上</b>。</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🤖 MCP 服务器</h3>
|
||||
<p>从 <b>Claude</b>、Cursor 或<br/>任何 MCP 客户端使用 VoiceStudio。</p>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
<details>
|
||||
<summary><b>……还有 12 项</b>——人声分离、说话人分离、批量处理、水印、诊断等等</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
- 🔊 **人声分离** — 基于 Demucs：把语音从音乐中分离出来，同时保留背景音床。
|
||||
- 👥 **说话人分离** — Pyannote + WhisperX 自动识别谁说了什么。
|
||||
- 📦 **批量队列** — 拖入 50 个视频就可以走开；每个任务都有独立进度条。
|
||||
- 🛡️ **AI 水印** — AudioSeal（Meta）：不可见，且能在压缩后留存。
|
||||
- 🔬 **诊断** — 自检套件、错误日志、脱敏诊断包。
|
||||
- ⚡ **GPU 自动检测** — CUDA · MPS · ROCm（Linux，需手动开启）· CPU；显存 ≤8 GB 时自动卸载。
|
||||
- 🧭 **引擎路由** — 逐引擎 GPU 预检；绝不静默回退到 CPU。
|
||||
- 🧩 **可扩展** — 继承 `TTSBackend`，约 50 行代码即可接入任意引擎。
|
||||
- 🎒 **便携声音角色** — 将声音导出为 `.ovsvoice` 包：身份 + 水印。
|
||||
- ♾️ **无限长 TTS** — 按句分块生成，没有长度上限，可经 WebSocket 流式输出。
|
||||
- 🌐 **远程后端** — 让 UI 指向远程服务器；对 Tailscale 友好，支持 Bearer 认证。
|
||||
- 🧠 **听写 + LLM** — 用本地 LLM 润色转录文本，可选回声消除。
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
<a id="why-voicestudio"></a>
|
||||
|
||||
## 💡 为什么选择 VoiceStudio？
|
||||
|
||||
云端语音工具很方便，但工作流会依赖账号、用量计费和他人的基础设施。VoiceStudio 在你的硬件上提供完整工作室；只有你主动选择时，才会使用联网集成。
|
||||
|
||||
| | **ElevenLabs** | **VoiceStudio** |
|
||||
|---|---|---|
|
||||
| **价格** | 订阅与用量限制 | 免费且开源（AGPL-3.0）· 专有用途可选 [商业许可证](#license) |
|
||||
| **语音克隆** | ✅ 3 秒音频 | ✅ 3 秒音频，零样本 |
|
||||
| **声音设计** | ✅ 性别、年龄 | ✅ 性别、年龄、口音、音高、风格、方言 |
|
||||
| **有声书 / 故事** | ❌ | ✅ 完整有声书编辑器 + 多声音故事（EPUB/PDF 导入，.m4b 导出） |
|
||||
| **语言** | 取决于套餐和模型 | **646** |
|
||||
| **视频配音** | ✅ 仅云端 | ✅ 完全本地 |
|
||||
| **数据隐私** | 音频在远端处理 | 核心流程在本地运行；联网服务必须主动选择 |
|
||||
| **API 密钥** | 需要账号 | 本地流程不需要 |
|
||||
| **GPU 支持** | 不适用（云端） | CUDA · Apple Silicon · ROCm（Linux）· CPU |
|
||||
| **桌面应用** | ❌ | ✅ macOS · Windows · Linux |
|
||||
| **TTS 引擎** | 1 | **16** — [完整矩阵](#tts-engines) |
|
||||
| **ASR 引擎** | 1 | **11** — [完整阵容](#asr-engines) |
|
||||
| **MCP 服务器** | ❌ | ✅ 可从 Claude、Cursor 及任何 MCP 客户端使用 |
|
||||
| **自检** | ❌ | ✅ 诊断套件、错误日志、脱敏调试包 |
|
||||
| **可定制** | ❌ 闭源 | ✅ 随你 Fork、扩展、发布 |
|
||||
|
||||
专业级语音 AI，去掉订阅，也去掉云端。
|
||||
|
||||
<div align="center">
|
||||
<br/>
|
||||
<b>心动了？来和我们一起构建吧。</b><br/>
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Join_Discord-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="加入 Discord" /></a>
|
||||
<br/><br/>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ 系统要求
|
||||
|
||||
| | **最低配置** | **推荐配置** |
|
||||
|---|---|---|
|
||||
| **操作系统** | Windows 10、macOS 12+（Apple Silicon）、Ubuntu 24.04+（glibc 2.39+） | 任意现代 64 位操作系统 |
|
||||
| **内存** | 8 GB | 16 GB+ |
|
||||
| **显存（GPU）** | 4 GB（自动将 TTS 卸载到 CPU） | 8 GB+（NVIDIA RTX 3060+） |
|
||||
| **硬盘** | 10 GB 可用空间（模型 + 缓存） | 20 GB+ SSD |
|
||||
| **Python** | 3.10+（由 `uv` 管理） | 3.11–3.12 |
|
||||
| **GPU** | 可选——CPU 也能跑 | NVIDIA CUDA · Apple Silicon MPS · AMD ROCm（仅 Linux） |
|
||||
|
||||
> [!TIP]
|
||||
> 对于显存 **≤8 GB** 的 GPU，VoiceStudio 会在转录期间自动将 TTS 卸载到 CPU——无需配置。不需要专用 GPU；整条流水线都可以在 CPU 上运行（只是慢一些）。
|
||||
|
||||
> [!NOTE]
|
||||
> **AMD GPU：** ROCm 加速**仅限 Linux 且需手动开启**——在首次运行的设置界面选择 **“AMD GPU (ROCm)”**，或设置 `OMNIVOICE_TORCH_VARIANT=rocm`（[docs/install/linux.md](docs/install/linux.md#amd-gpu-rocm)）。在 **Docker/Podman** 中请改用专门的 ROCm 镜像：`ghcr.io/debpalash/omnivoice-studio:rocm`（[docs/install/docker.md](docs/install/docker.md#pull-and-run-amd-gpu--rocm)）。**在 Windows 上，AMD GPU（含 Ryzen AI 核显）只能以 CPU 运行**：PyTorch 没有 Windows 版 ROCm 轮子，因此 Windows 上的 GPU 加速仅限 NVIDIA/CUDA（[docs/install/windows.md](docs/install/windows.md#gpu-support)）。
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **macOS Intel（x86_64）不支持本地后端：** 应用 UI 可以安装，但 Python 后端无法运行，因为 PyTorch 已不再发布 Intel Mac 轮子（[#889](https://github.com/debpalash/VoiceStudio/issues/889)）。Intel Mac 用户仍可让 UI 指向另一台机器上的远程后端——参见 [docs/install/macos.md](docs/install/macos.md)。
|
||||
|
||||
<a id="hardware-recommendations"></a>
|
||||
|
||||
### 💡 按硬件推荐引擎配置
|
||||
|
||||
| 硬件配置 | 推荐 TTS 引擎 | 推荐 ASR 语音识别 | 优势 |
|
||||
|---|---|---|---|
|
||||
| **Apple Silicon (M1–M4)** | [MLX-Audio](docs/engines/mlx-audio.md) · [OmniVoice](docs/engines/omnivoice.md) (MPS) | [MLX Whisper](docs/engines/mlx-whisper.md) · [Parakeet MLX](docs/engines/parakeet-mlx.md) | 原生统一内存，macOS 上延迟最低、性能最强 |
|
||||
| **NVIDIA 显卡 (8 GB+ 显存)** | [OmniVoice](docs/engines/omnivoice.md) · [CosyVoice 3](docs/engines/cosyvoice.md) | [WhisperX](docs/engines/whisperx.md) | 极致零样本克隆品质、字级时间戳对齐与说话人分离 |
|
||||
| **低显存 / 仅 CPU 设备** | [PocketTTS](docs/engines/pockettts.md) · [Sherpa-ONNX](docs/engines/sherpa-onnx.md) · [KittenTTS](docs/engines/kittentts.md) | [Moonshine](docs/engines/moonshine.md) · [Faster-Whisper](docs/engines/faster-whisper.md) (`int8`) | 超低内存占用，针对 CPU 指令集深度优化 |
|
||||
|
||||
<a id="tts-engines"></a>
|
||||
|
||||
### 🗣️ TTS 引擎
|
||||
|
||||
**16 个引擎，一个选择器。** VoiceStudio（默认，支持 600+ 语言）始终可用；另有七个引擎可选装并自动检测（CosyVoice 3、GPT-SoVITS、VoxCPM2、MOSS-TTS-Nano、KittenTTS、MLX-Audio、Sherpa-ONNX），外加八个按需延迟安装的引擎（IndexTTS 2.5、OmniVoice GGUF、OmniVoice 子进程版、PocketTTS、Supertonic 3、MOSS-TTS-v1.5、dots.tts、Confucius4-TTS）。在 **设置 → TTS 引擎** 中切换；所选引擎将应用于所有语音合成场景。**每个引擎都有独立指南：[docs/engines](docs/engines/README.md)（英文）。**
|
||||
|
||||
<details>
|
||||
<summary><b>📊 完整矩阵</b>——16 个引擎 × 平台 × 克隆/指令 × 许可证</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| 引擎 | 语言 | 克隆 | 指令 | Linux | macOS ARM | Windows | 许可证 |
|
||||
|--------|:---------:|:-----:|:--------:|:-----:|:---------:|:-------:|:-------:|
|
||||
| **VoiceStudio**（默认，由 k2-fsa/OmniVoice 驱动） | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | 内置 |
|
||||
| **CosyVoice 3** | 9 + 18 种方言 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **GPT-SoVITS** | 5 | ✅ | — | ✅ CUDA/CPU | — | ✅ CUDA/CPU | MIT |
|
||||
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MOSS-TTS-Nano** | 20 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **KittenTTS** | 英语 | — | — | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
|
||||
| **MLX-Audio**（Kokoro、Qwen3-TTS、CSM、Dia 等） | 多语言 | 因模型而异 | 因模型而异 | ❌ | ✅ 原生 | ❌ | 因模型而异 |
|
||||
| **Sherpa-ONNX** | 20+ | — | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **IndexTTS 2.5** ⚡ | 中文 · 英语 · 日语 · 西班牙语 · 阿拉伯语 | ✅ | — | ✅ CUDA | — | ✅ CUDA | Bilibili 模型许可¹ |
|
||||
| **OmniVoice GGUF** ⚡ | 600+ | ✅ | ✅ | ✅ CPU | ✅ CPU | ✅ CPU | 内置 |
|
||||
| **Supertonic 3** ⚡ | 31 | — | — | ✅ CPU | ✅ CPU | ✅ CPU | OpenRAIL-M |
|
||||
| **MOSS-TTS-v1.5** ⚡（8B） | 31 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **dots.tts** ⚡（2B） | 24 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ❌ | Apache-2.0 |
|
||||
| **Confucius4-TTS** ⚡ | 14 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
|
||||
¹ 若月活跃用户超过 1 亿，或年收入超过人民币 10 亿元，使用 IndexTTS 2.5
|
||||
前必须另行取得 Bilibili 的书面许可。启用可选边车前，请审阅其
|
||||
[模型许可](https://huggingface.co/IndexTeam/IndexTTS-2.5/blob/main/LICENSE)。
|
||||
|
||||
> **CUDA** = GPU 加速 · **MPS** = Apple Silicon Metal · **CPU** = 随处可运行，大模型较慢 · KittenTTS 和 MOSS-TTS-Nano 可在 CPU 上实时运行 · MLX-Audio 仅限 Apple Silicon · ⚡ = 延迟注册（首次使用时安装）
|
||||
>
|
||||
> **克隆**能力的意义不止于单段生成：视频配音（以及任何固定了声音的批量任务）需要参考音频克隆来保持说话人身份，因此把不支持克隆的引擎（KittenTTS、Sherpa-ONNX、Supertonic 3）设为当前引擎时，这些任务会在开始前就给出可操作的失败提示，而不是静默回退到 VoiceStudio。
|
||||
>
|
||||
> **MOSS-TTS-v1.5**（8B，约 16 GB）、**dots.tts**（2B，约 9 GB）和 **Confucius4-TTS** 是重量级可选引擎，从本地克隆在各自独立的 venv 中运行。三者均不支持 Apple Silicon MPS（在 Mac 上以 CPU 运行）；dots.tts 没有 Windows 路径；Confucius4 建议使用 CUDA（CPU 可用，约为实时时长的 17 倍）。详情：[MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) · [dots.tts](docs/engines/dots-tts.md) · [Confucius4-TTS](docs/engines/confucius4-tts.md)。
|
||||
|
||||
</details>
|
||||
|
||||
<a id="asr-engines"></a>
|
||||
|
||||
### 🎧 ASR 引擎
|
||||
|
||||
**11 个引擎**——它们驱动听写、视频配音和字幕。**WhisperX** 是跨平台的默认引擎（约 100 种语言，词级时间对齐）；其余引擎均为可选装并自动检测。在 **设置 → 引擎** 中切换。十个完全在本地设备上运行；第十一个（OpenAI 兼容）是可选的远程客户端，可用于 Qwen3-ASR 或任何兼容的服务器。
|
||||
|
||||
<details>
|
||||
<summary><b>📊 完整阵容</b>——11 个引擎、各自的强项与计算类型说明</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| 引擎 | `OMNIVOICE_ASR_BACKEND` | 语言 | 最适合 |
|
||||
|--------|-------------------------|:---------:|----------|
|
||||
| **WhisperX**（默认） | `whisperx` | ~100 | 配音与字幕——通过 wav2vec2 强制对齐实现词级时间对齐 |
|
||||
| **Faster-Whisper** | `faster-whisper` | ~100 | Linux / macOS / Windows 上的快速转录（CTranslate2） |
|
||||
| **Faster-Whisper（隔离）** | `faster-whisper-isolated` | ~100 | 与 Faster-Whisper 相同，但在子进程中崩溃隔离——ASR 崩溃不会拖垮整个应用 |
|
||||
| **MLX Whisper** | `mlx-whisper` | ~100 | Apple Silicon 原生速度（Apple MLX / Metal） |
|
||||
| **PyTorch Whisper** | `pytorch-whisper` | ~100 | 经 🤗 Transformers 的 CUDA / CPU 兜底方案（无需 cuDNN 8） |
|
||||
| **Parakeet TDT** | `nemo-parakeet` | 英语 + 25 种欧洲语言 | 即使在 CPU 上也能以约 10 倍实时速度达到 SOTA 精度，自动语言检测（NVIDIA NeMo，CUDA/CPU） |
|
||||
| **Moonshine** | `moonshine` | 英语 | 边缘设备 / 低延迟，ONNX |
|
||||
| **FunASR** | `funasr` | 50+ | 多语言一体化——内置 VAD + 行内说话人分离（SenseVoice） |
|
||||
| **sherpa-onnx**（实时听写） | `sherpa-onnx-asr` | 25 种欧洲语言 + 90+ | 实时、快于实时的听写——小体积流式/离线 ONNX 模型（Parakeet TDT v3/v2、流式 Zipformer 与 Paraformer、Whisper Tiny），CPU 运行，macOS / Windows / Linux 表现完全一致。在 **设置 → 语音** 中按模型选择。 |
|
||||
| **OpenAI 兼容** ⚠️ 远程 | `openai-compat-asr` | 取决于服务器 | 当下通往 **Qwen3-ASR** 的路径（自托管服务器，无需等 transformers 支持）、任何 OpenAI 兼容的转录端点，或 OpenAI 官方 API——无需安装，在 **设置 → 引擎**（ASR 标签页）中配置并测试连接。音频会离开你的设备，发送到你指定的任何服务器；参见 [docs/engines/openai-compatible-asr.md](docs/engines/openai-compatible-asr.md)。 |
|
||||
|
||||
> Whisper 系列引擎覆盖约 100 种语言；**FunASR / SenseVoice** 额外提供一条多语言一体化路径，内置语音活动检测与行内说话人分离。**sherpa-onnx** 驱动实时听写的模型选择器——你边说，文字边出现。除可选的 OpenAI 兼容远程客户端外，所有引擎都在本地设备上运行——无需 API 密钥，无需云端。
|
||||
|
||||
> **GPU 不支持高效 float16？** 在较老的 NVIDIA GPU（Maxwell/Pascal、GTX 16xx）上，或在 CTranslate2/cuDNN 版本不匹配之后，CTranslate2 系 ASR 引擎（WhisperX、Faster-Whisper）无法运行 `float16`，VoiceStudio 会自动改用 `int8` 重试——无需配置。如果转录仍然失败，可用 `ASR_COMPUTE_TYPE` 环境变量固定计算类型（逃生舱口）：`ASR_COMPUTE_TYPE=int8`（CPU 用 `float32`）。将其设为 `int8` 并重启后端。
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ 架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Frontend (React) │
|
||||
│ DubTab · VoiceConsole · Stories · Audiobook · Gallery │
|
||||
│ Dictation · BatchQueue · Diagnostics · MCP Client │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Backend (FastAPI) │
|
||||
│ 100+ API endpoints · SSE+WSS streaming · SQLite │
|
||||
├──────────┬──────────┬──────────┬──────────┬────────────────┤
|
||||
│ WhisperX │ Demucs │VoiceStudio │ Pyannote │ Engine Routing │
|
||||
│ (+7 ASR │ Source │ (+10 │ Diariz- │ ↳ GPU preflight │
|
||||
│ engines) │ Sep. │ TTS) │ ation │ ↳ No silent CPU │
|
||||
└──────────┴──────────┴──────────┴──────────┴────────────────┘
|
||||
CUDA / MPS / ROCm / CPU (auto-detected + routed)
|
||||
```
|
||||
|
||||
<a id="openai-api"></a>
|
||||
|
||||
## 🔌 OpenAI 兼容 API
|
||||
|
||||
已经有会说 OpenAI 音频 API 的脚本、智能体或工具？把它指向 `http://localhost:3900/v1` 即可——不需要密钥，也不用改代码。后端为音频端点内置了即插即用的兼容接口，直接接到你当前启用的 TTS/ASR 引擎（没错，`voice` 参数接受你克隆的声音配置 ID）。
|
||||
|
||||
| 端点 | 作用 |
|
||||
| 需求 | 链接 |
|
||||
|---|---|
|
||||
| `POST /v1/audio/speech` | TTS——输入文本；输出 `mp3` / `wav` / `flac` / `opus` / `pcm`。`tts-1` / `tts-1-hd` 映射到你当前启用的引擎；也接受 OpenAI 的声音名称（`alloy` 等）。 |
|
||||
| `POST /v1/audio/transcriptions` | STT——输入音频文件；输出 `json`、`text`、`verbose_json`、`srt` 或 `vtt`。`whisper-1` 映射到你当前启用的 ASR 引擎。 |
|
||||
| `GET /v1/audio/voices` | VoiceStudio 扩展——列出所有声音配置和引擎，客户端可据此发现你的克隆声音。 |
|
||||
| 安装帮助 | [故障排查](docs/install/troubleshooting.md) · [模型下载](docs/downloading-models.md) |
|
||||
| 模型与音质 | [引擎指南](docs/engines/README.md) · [基准测试](docs/benchmarks.md) |
|
||||
| 集成 | [本地 API](docs/speech-platform.md) · [MCP](docs/mcp.md) · [示例](examples/README.md) |
|
||||
| 参与开发 | [贡献指南](.github/CONTRIBUTING.md) · [Electron](electron/README.md) · [更新日志](CHANGELOG.md) |
|
||||
|
||||
```sh
|
||||
curl http://localhost:3900/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "tts-1", "voice": "alloy", "input": "Generated on my own hardware.", "response_format": "wav"}' \
|
||||
--output speech.wav
|
||||
```
|
||||
安装智能体技能：`npx skills add debpalash/VoiceStudio`
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:3900/v1", api_key="none") # any string works — nothing checks it
|
||||
## 支持 VoiceStudio
|
||||
|
||||
result = client.audio.transcriptions.create(model="whisper-1", file=open("clip.wav", "rb"))
|
||||
print(result.text)
|
||||
```
|
||||
[Ko-fi](https://ko-fi.com/debpalash) · [PayPal](https://paypal.me/palashCoder) · [赞助项目](SPONSORS.md) · [商务合作](mailto:partner@voicestudio.sh)
|
||||
|
||||
想要完整的接口（100+ 端点）？完整的 REST API 参考已内嵌在应用中——**设置 → OpenAPI 参考**（由 Scalar 驱动），或点击页脚的 `{}` 按钮。
|
||||
**让语音应用开发者看到你的品牌。** 了解应用底部栏、集成目录、文档和 README 的付费展示合作。[申请合作](https://forms.gle/2PYCvd39hbwijzX37)或[发送邮件](mailto:partner@voicestudio.sh)。
|
||||
|
||||
### 📓 在 Google Colab 上运行
|
||||
## 许可与负责任使用
|
||||
|
||||
[](https://colab.research.google.com/github/debpalash/VoiceStudio/blob/main/notebooks/OmniVoice_Studio_Colab.ipynb)
|
||||
|
||||
没有本地 GPU？官方笔记本（[notebooks/OmniVoice_Studio_Colab.ipynb](notebooks/OmniVoice_Studio_Colab.ipynb)）可在免费的 Colab T4 上启动完整应用（包含 Web 界面）：在笔记本内直接构建前端，用 uv 安装后端（复用 Colab 预装的 CUDA PyTorch），并通过 Colab 内置端口代理打开界面。无需第三方隧道，也无需任何 API 密钥。随后还有一套覆盖全部主要功能的 API 导览，全部可在笔记本内直接播放：多语言 TTS、声音克隆与声音设计、已保存的声音档案、语音转写、AI 水印检测、OpenAI 兼容 API、多角色故事、带章节的 m4b 有声书，以及一个附带人声分离音轨的迷你视频配音。
|
||||
|
||||
### 🤝 智能体技能（Agent Skills）
|
||||
|
||||
用一条命令教会你的 AI 智能体（Claude Code、Cursor、Codex 等）使用 VoiceStudio：
|
||||
|
||||
```sh
|
||||
npx skills add debpalash/omnivoice-studio
|
||||
```
|
||||
|
||||
内含两个 [skills](https://skills.sh)：**`omnivoice`**——让任何智能体通过你的本地安装进行语音合成与转录（包括你克隆的声音），免费且离线；以及 **`oss-maintainer`**——本项目所遵循的维护者方法论，适合任何用智能体运营自己开源项目的人。
|
||||
|
||||
### 🔌 模型上下文协议（MCP 服务器）
|
||||
|
||||
VoiceStudio 在 `http://localhost:3900/mcp` 挂载了 MCP 服务，可供 Claude Desktop、Cursor 与自主智能体调用：
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicestudio": {
|
||||
"url": "http://localhost:3900/mcp"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
对于需要 stdio 管道传输的客户端，请使用内置的本地桥接脚本（`docs/mcp.json`）：
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"voicestudio": {
|
||||
"command": "python",
|
||||
"args": ["-m", "backend.mcp_shim"],
|
||||
"cwd": "/path/to/VoiceStudio"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
支持 `generate_speech`、`clone_voice`、`transcribe` 等工具与流式文件输出模式，详见 [docs/mcp.md](docs/mcp.md)。
|
||||
|
||||
---
|
||||
|
||||
## 🗺️ 路线图
|
||||
|
||||
### 🔜 即将推出
|
||||
|
||||
- 🎬 **唇形同步 v2** — 使用 wav2lip 进行视觉语音时间对齐
|
||||
- 🌐 **在线演示** — 无需安装即可体验 VoiceStudio
|
||||
- 🔌 **插件市场** — 社区贡献的 TTS 引擎与特效
|
||||
- 🎵 **实时变声器** — 通话中的麦克风实时变声
|
||||
|
||||
<details>
|
||||
<summary><b>✅ 已经发布的一切</b>——按类别列出的“成绩单”</summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| 分类 | 功能 |
|
||||
|----------|----------|
|
||||
| **长内容** | 有声书编辑器（文本/EPUB/PDF → 分章 .m4b）、Stories 多声音编辑器、两遍响度归一母带处理、渲染中断后的崩溃续渲、发音控制 + SSML-lite 韵律 |
|
||||
| **配音** | 完整流水线（转录→翻译→合成→封装）、场景感知分割、唇形同步评分、流式 TTS、逐说话人声音分配、Smart Fit 时长匹配 + 二次 QC、独立的配音主页 |
|
||||
| **声音** | 零样本克隆、声音设计、A/B 对比、声音预览控件、支持收藏/标签的声音库、便携声音角色包（`.ovsvoice`）、声音控制台工作区 |
|
||||
| **音频** | Demucs 人声分离、逐段增益、选择性音轨导出、分轨/SRT/VTT/MP3 导出、按句分块实现的无限长 TTS |
|
||||
| **多语言** | 多语言批量选择器、顺序 GPU 执行的批量配音队列 |
|
||||
| **说话人分离** | Pyannote 机器学习分离、自动说话人克隆提取、逐说话人声音分配 |
|
||||
| **ASR** | 9 个引擎（WhisperX、Faster-Whisper、隔离版 Faster-Whisper、MLX Whisper、PyTorch Whisper、Parakeet TDT、Moonshine、FunASR/SenseVoice、sherpa-onnx 实时听写）、崩溃隔离的子进程后端 |
|
||||
| **TTS** | 14 个引擎（VoiceStudio、CosyVoice 3、GPT-SoVITS、VoxCPM2、MOSS-TTS-Nano、KittenTTS、MLX-Audio、Sherpa-ONNX，+ 延迟安装：IndexTTS 2.5、OmniVoice GGUF、Supertonic 3、MOSS-TTS-v1.5、dots.tts、Confucius4-TTS）、带 GPU 预检的引擎路由 |
|
||||
| **基础设施** | Docker 部署、CUDA/MPS/ROCm 自动检测、cuDNN 8 兼容、显存感知模型卸载、引擎路由（绝不静默回退 CPU）、诊断套件与错误日志、受限网络镜像支持 |
|
||||
| **AI 溯源** | AudioSeal 不可见水印（类似 SynthID）、视频徽标叠加、水印检测 API |
|
||||
| **用户体验** | 撤销/重做、键盘快捷键、拖放、会话持久化、首次启动按屏幕推荐界面缩放，以及原生 WebKitGTK 缩放 |
|
||||
| **实时事件** | WebSocket 事件总线——数据变更时即时刷新侧边栏、指数退避重连 |
|
||||
| **状态管理** | Zustand 状态迁移——`uiSlice`、`pillSlice`、`dubSlice`、`generateSlice`、`prefsSlice`、`glossarySlice` |
|
||||
| **桌面** | 跨平台 Tauri 安装程序（macOS DMG——Apple Silicon；Intel 不支持本地后端，#889——Windows MSI、Linux deb/AppImage）、自动更新基础设施、单实例约束、关闭最小化到托盘、macOS Gatekeeper 修复 |
|
||||
| **听写** | 全局系统级热键（`⌘+⇧+Space`）、无边框浮动控件、WebSocket 流式 ASR、自动粘贴、可自定义热键、本地 LLM 转录润色 |
|
||||
| **批量流水线** | 完整批量 TTS：提取 → 转录 → 翻译 → 生成 → 混音 → 导出，带实时进度追踪 |
|
||||
| **MCP 服务器** | 让 VoiceStudio 成为 Claude、Cursor 及任何 MCP 客户端的本地 TTS/STT 提供方 |
|
||||
| **远程后端** | 让桌面 UI 指向远程后端 URL，支持 Bearer 认证（附 Tailscale 文档） |
|
||||
| **可靠性** | 启动开屏的卡死看门狗、逐引擎 GPU 兼容矩阵、引擎二进制不可执行时的可操作报错、setuptools 自动修复 |
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
<a id="sponsor--donate"></a>
|
||||
|
||||
## 💜 赞助 / 捐赠
|
||||
|
||||
VoiceStudio 由一位开发者使用 Claude Code 和 AI 智能体独立打造——而智能体账单是实打实的（过去三个月花了数千美元）。如果 VoiceStudio 为你创造了价值，帮忙分担一小部分账单，就能让开发保持全职推进。
|
||||
|
||||
<div align="center">
|
||||
|
||||
**本月智能体账单基金**
|
||||
|
||||
<img src="https://img.shields.io/badge/raised_%2410_of_%24200-5%25-EAB308?style=for-the-badge" alt="已筹 $10 / $200" />
|
||||
|
||||
<br/><br/>
|
||||
|
||||
<a href="https://ko-fi.com/debpalash"><img src="https://img.shields.io/badge/Ko--fi-Support_❤️-FF5E5B?style=for-the-badge&logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
|
||||
|
||||
<a href="https://paypal.me/palashCoder"><img src="https://img.shields.io/badge/PayPal-Donate-00457C?style=for-the-badge&logo=paypal&logoColor=white" alt="PayPal" /></a>
|
||||
|
||||
<br/>
|
||||
<sub>每一美元都直接用于支付智能体账单——让 VoiceStudio 的开发持续不断。</sub>
|
||||
|
||||
<br/><br/>
|
||||
|
||||
<sub><b>来自 VoiceStudio 作者的更多应用</b>——同样的本地优先理念：
|
||||
<a href="https://github.com/debpalash/Opal"><b>Opal</b> 💠</a>（播放一切——AI 时代的媒体播放器）·
|
||||
<a href="https://github.com/debpalash/memxt"><b>memxt</b> 🧠</a>（Claude Code 与编码智能体的本地记忆）。
|
||||
给它们点个 ⭐ 也是一种支持 → <a href="#more-from-the-maker">详见下文</a>。</sub>
|
||||
|
||||
</div>
|
||||
|
||||
<a id="sponsors"></a>
|
||||
|
||||
### 🌟 赞助商
|
||||
|
||||
VoiceStudio **免费**且采用 **AGPL-3.0** 许可——没有付费版，没有 SaaS 收入。赞助商让开发得以持续，作为回报，可以在这里、在应用内（顶级档位还包括项目官网）获得一个徽标位。这是一份感谢，绝不是付费墙。**[查看档位并成为赞助商 →](SPONSORS.md)**
|
||||
|
||||
<div align="center">
|
||||
|
||||
<!-- SPONSORS:START — logo slots are filled here as sponsors come aboard; see SPONSORS.md -->
|
||||
|
||||
**这里可以是你的徽标** — [成为赞助商](SPONSORS.md)
|
||||
|
||||
<!-- SPONSORS:END -->
|
||||
|
||||
</div>
|
||||
|
||||
<sub>💡 GitHub 也会在本仓库顶部显示一个 **Sponsor** 按钮，经由 <a href=".github/FUNDING.yml"><code>.github/FUNDING.yml</code></a> 指向相同的链接。</sub>
|
||||
|
||||
---
|
||||
|
||||
## 💬 社区
|
||||
|
||||
<div align="center">
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/💬_Discord-Join_Community-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="加入 Discord" /></a>
|
||||
<br/>
|
||||
<sub>设置类问题我们几小时内就会回复，而不是几天。</sub>
|
||||
</div>
|
||||
|
||||
<details>
|
||||
<summary><b>里面都在聊什么</b></summary>
|
||||
|
||||
<br/>
|
||||
|
||||
| 频道 | 那里发生什么 |
|
||||
|---------|--------------------|
|
||||
| `#announcements` | 发布消息与重大时刻——新版本最先在这里公布 |
|
||||
| `#releases` + `#changelog` | 每一个构建，以及里面究竟有什么 |
|
||||
| `#issues` | 以论坛帖子形式提交的 Bug 报告——直接分诊进 GitHub Issues |
|
||||
| `#ideas` | 功能请求，供讨论与投票 |
|
||||
| `#discuss-ideas` | 动手之前的设计讨论 |
|
||||
| `#general` | 安装帮助、GPU 疑难排查，以及晒你的配音成果 |
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
<a id="contributing"></a>
|
||||
|
||||
## 🤝 参与贡献
|
||||
|
||||
非常欢迎——Bug 修复、新的 TTS 引擎适配器、UI 改进、文档、翻译。统统欢迎。
|
||||
|
||||
- 📖 阅读 **[贡献指南](.github/CONTRIBUTING.md)** 了解环境搭建、代码风格和 PR 工作流
|
||||
- 🐛 浏览 [good first issues](https://github.com/debpalash/VoiceStudio/labels/good%20first%20issue)
|
||||
- 💬 加入我们的 [Discord](https://discord.gg/bzQavDfVV9) 讨论想法或寻求帮助
|
||||
|
||||
---
|
||||
|
||||
## ❓ 常见问题
|
||||
|
||||
<details>
|
||||
<summary><b>真的能和 ElevenLabs 一样好吗？</b></summary>
|
||||
<br/>
|
||||
诚实的回答：<b>取决于你要做什么。</b>
|
||||
|
||||
<b>VoiceStudio 真正有竞争力的地方：</b>从干净的参考音频进行语音克隆（最先进的开源扩散 TTS）、语言覆盖（646 种语言对他们的 32 种），以及所有结构性优势——没有按字符计费、没有用量上限、音频不离开你的设备、完整的流水线可定制性（14 个 TTS 引擎、10 个 ASR 引擎、翻译方案随你选）。
|
||||
|
||||
<b>ElevenLabs 仍然领先的地方：</b>开箱即用的稳定性与打磨程度，尤其是英语 TTS。他们的单一模型经过深度调优；我们的质量取决于你选择的引擎、你的硬件，以及（对克隆而言）参考音频——干燥、近麦的音频比嘈杂或有回声的音频克隆效果好得多。
|
||||
|
||||
<b>具体到配音：</b>配音是一条链——转录 → 翻译 → 克隆 → 合成——在<i>你的</i>素材上，它只取决于最薄弱的一环。如果部分输出语无伦次，先检查片段表里的<i>原文</i>：当转录本身就错了，换一个 ASR 引擎或使用更干净的源音频——修复点通常在这里，而不是声音。
|
||||
|
||||
拿你的真实素材试试——免费，下载一次即可。许多用户直接用它替换了 ElevenLabs；也有人两个都留着。这两种结果我们都乐见。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>能在 Apple Silicon（M1/M2/M3/M4）上运行吗？</b></summary>
|
||||
<br/>
|
||||
可以。MPS 加速会被自动检测。在 Apple 硬件上，MLX 优化的 Whisper 模型可提供更快的转录速度。<b>不支持 Intel Mac</b>：应用 UI 可以安装，但本地 Python 后端无法运行，因为 PyTorch 已不再发布 Intel Mac 轮子（<a href="https://github.com/debpalash/VoiceStudio/issues/889">#889</a>）——Intel Mac 只能配合远程后端使用。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>需要多少显存？</b></summary>
|
||||
<br/>
|
||||
<b>最低 4 GB。</b> 显存 ≤8 GB 时，TTS 模型会在转录期间自动卸载到 CPU。8 GB 以上时，所有组件同时在 GPU 上运行。完全没有 GPU？CPU 模式也能用——只是慢一些（TTS 约慢 3 倍）。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>可以用于商业用途吗？</b></summary>
|
||||
<br/>
|
||||
<b>可以——商业使用免费</b>，基于 <a href="https://www.gnu.org/licenses/agpl-3.0.html">AGPL-3.0</a>：运行它、出售用它生成的音频、为客户的视频配音、在团队中部署。只有一项义务：如果你<b>修改</b>了 VoiceStudio 并通过网络向他人提供该修改版本，你必须依据相同条款分享修改后的源代码。想把它嵌入闭源产品？可获取商业许可证——参见<a href="#license">许可证</a>。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>支持哪些语言？</b></summary>
|
||||
<br/>
|
||||
通过 VoiceStudio 模型的 TTS 支持 646 种语言。转录（WhisperX）支持 99 种语言。翻译覆盖范围取决于目标语言对。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>可以添加自己的 TTS 引擎吗？</b></summary>
|
||||
<br/>
|
||||
可以。在 <code>backend/services/tts_backend.py</code> 中继承 <code>TTSBackend</code>，并将其添加到 <code>_REGISTRY</code> 字典中——约 50 行代码。十四个内置引擎均以此方式实现；参见 <a href="#tts-engines">TTS 引擎</a>。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>VoiceStudio 会收集我的任何数据吗？</b></summary>
|
||||
<br/>
|
||||
<b>除非你明确同意，否则不会。</b>首次运行时应用会<i>询问</i>你——一个页面、两个同等分量的按钮，没有预先勾选。在你回答“是”之前，VoiceStudio 什么都不发送：没有分析、没有遥测、没有账号、没有“回传”。跳过提问就等于“否”。无论如何，你的文本、音频、声音和项目永远不会离开你的设备。
|
||||
|
||||
如果你选择同意（也可随时在 <b>设置 → 隐私 → “帮助改进 VoiceStudio”</b> 中开关），发送的只是匿名、不含内容的使用统计：生成信息（引擎、语言、生成耗时、字符<i>数量</i>、错误<i>类型</i>），以及应用生命周期——一次安装信号、版本更新（版本号之间）、崩溃（错误类别和<i>分桶后的</i>运行时长，绝不含日志）、错误<i>类型</i>（有上限、去重），以及卸载时的一次告别信号。绝不包含你的文本、音频、文件名或任何可识别信息——这由代码中的属性白名单强制保证（<code>backend/core/analytics.py</code>），而不只是一句承诺。源码构建根本没有分析数据的接收端，因此根本不会询问。你自己的统计数字在 <b>设置 → 用量</b> 中查看，本地计算，不发送到任何地方。
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>如何卸载它 / 删除它的所有数据？</b></summary>
|
||||
<br/>
|
||||
VoiceStudio 完全本地运行——卸载就是删除应用及其写入的文件夹（模型缓存、Python 环境、你的声音/项目、配置）。运行 <code>scripts/uninstall.sh</code>（macOS/Linux）或 <code>scripts\uninstall.ps1</code>（Windows）——它会先以干跑方式列出每个文件夹及其大小，加 <code>--yes</code> 才会真正删除。完整的各平台路径列表和应用移除步骤见 <a href="docs/install/uninstall.md"><b>docs/install/uninstall.md</b></a>。
|
||||
</details>
|
||||
|
||||
## 🛡️ 负责任使用与安全
|
||||
|
||||
VoiceStudio 在个人硬件上提供零样本语音克隆与语音创作能力。我们提倡负责任的技术使用：
|
||||
- **明确授权：** 严禁在未经说话人本人知情并明确授权的情况下克隆其声音。
|
||||
- **AI 溯源：** VoiceStudio 默认集成 [AudioSeal](https://github.com/facebookresearch/audioseal) 不可见神经音频水印，在完全不影响听感音质的前提下精准标记合成语音。
|
||||
- **本地隐私：** 默认本地工作流下，所有音频、声音档案、项目与转录文本始终保存在你的本地设备上；仅当你主动配置远程工作节点或第三方 ASR 端点时，相应数据才会传输到对应服务。
|
||||
|
||||
---
|
||||
|
||||
<a id="license"></a>
|
||||
|
||||
## 📜 许可证
|
||||
|
||||
VoiceStudio 是基于 [**GNU Affero 通用公共许可证 v3.0（AGPL-3.0）**](https://www.gnu.org/licenses/agpl-3.0.html) 的自由开源软件。
|
||||
|
||||
**可免费用于任何用途——包括商业和企业内部用途。** 运行它、出售用它生成的音频、为自己或客户的视频配音、在团队中推广——全部免费，无需许可证。作为一份**网络著佐权（copyleft）**许可证，AGPL 增加了一项义务：如果你**修改**了 VoiceStudio 并通过网络向他人提供该修改版本，你必须依据相同的 AGPL-3.0 条款向他们提供该修改版本的完整对应源代码。
|
||||
|
||||
希望将 VoiceStudio 嵌入**闭源或专有**产品或服务、又不受 AGPL-3.0 著佐权义务约束的组织，可获取**商业许可证**。**定价方案即将推出。** 咨询：**VoiceStudio@palash.dev**。
|
||||
|
||||
捆绑的 `omnivoice/` TTS 模型（作者 Han Zhu）在上游仍为 Apache-2.0 许可。完整且具约束力的条款请参见 [`LICENSE`](LICENSE)。
|
||||
|
||||
---
|
||||
|
||||
## 🙏 致谢
|
||||
|
||||
VoiceStudio 站在这些杰出开源工作的肩膀上：
|
||||
|
||||
| 项目 | 作用 |
|
||||
|---------|------|
|
||||
| [**VoiceStudio (k2-fsa)**](https://github.com/k2-fsa/OmniVoice) | 零样本扩散 TTS 引擎——核心语音合成模型 |
|
||||
| [**WhisperX**](https://github.com/m-bain/whisperX) | 词级别语音识别与时间对齐 |
|
||||
| [**Demucs (Meta)**](https://github.com/facebookresearch/demucs) | 音乐源分离，用于人声分离 |
|
||||
| [**Pyannote**](https://github.com/pyannote/pyannote-audio) | 说话人分离——谁说了什么 |
|
||||
| [**CTranslate2**](https://github.com/OpenNMT/CTranslate2) | CPU 和 GPU 上的优化 Transformer 推理 |
|
||||
| [**AudioSeal (Meta)**](https://github.com/facebookresearch/audioseal) | 用于 AI 溯源的不可见神经音频水印 |
|
||||
| [**Tauri**](https://tauri.app) | 原生桌面应用框架 |
|
||||
| [**Supertone / Supertonic 3**](https://huggingface.co/Supertone/supertonic-3) | ONNX TTS 引擎——31 种语言，CPU 高效 |
|
||||
| [**Sherpa-ONNX**](https://github.com/k2-fsa/sherpa-onnx) | 支持 WASM 的通用 TTS/ASR 运行时 |
|
||||
| [**GPT-SoVITS**](https://github.com/RVC-Boss/GPT-SoVITS) | 零样本 TTS 引擎——5 种语言，RTF 0.014 |
|
||||
|
||||
---
|
||||
|
||||
<a id="more-from-the-maker"></a>
|
||||
|
||||
## 🧰 来自同一作者的更多本地开源项目
|
||||
|
||||
喜欢这种本地优先的理念？它是一脉相承的——同一位作者，同一条准则：**你的数据只留在你的设备上。** 全部项目见 [palash.dev](https://palash.dev)。
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td align="center" width="50%" valign="top">
|
||||
<br/>
|
||||
<a href="https://github.com/debpalash/Opal"><img src="https://raw.githubusercontent.com/debpalash/Opal/main/assets/opal_logo.png" width="96" alt="Opal 徽标"/></a>
|
||||
<h3><a href="https://github.com/debpalash/Opal">Opal 💠</a></h3>
|
||||
<p><b>播放一切。</b>AI 时代的媒体播放器。</p>
|
||||
<p><sub>视频、动漫、漫画、种子、Jellyfin 和 Plex——一个播放器全部搞定，并内置本地 AI 记忆与上下文。使用 Zig 编写，支持 macOS 和 Windows。</sub></p>
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/Opal/stargazers"><img src="https://img.shields.io/github/stars/debpalash/Opal?style=flat-square&color=f59e0b" alt="Opal Star 数"/></a>
|
||||
<a href="https://palash.dev/opal"><img src="https://img.shields.io/badge/site-palash.dev%2Fopal-8b5cf6?style=flat-square" alt="Opal 官网"/></a>
|
||||
</p>
|
||||
</td>
|
||||
<td align="center" width="50%" valign="top">
|
||||
<br/>
|
||||
<a href="https://github.com/debpalash/memxt"><img src="https://raw.githubusercontent.com/debpalash/memxt/main/assets/logo-mark.svg" width="96" alt="memxt 徽标"/></a>
|
||||
<h3><a href="https://github.com/debpalash/memxt">memxt 🧠</a></h3>
|
||||
<p><b>经基准测试验证的最快开源 AI 记忆系统。</b></p>
|
||||
<p><sub>为 Claude Code 和编码智能体提供本地长期记忆——基于 SQLite + 嵌入向量的 MCP 服务器，100% 在你的设备上运行。你的智能体终于能记住昨天了。</sub></p>
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/memxt/stargazers"><img src="https://img.shields.io/github/stars/debpalash/memxt?style=flat-square&color=f59e0b" alt="memxt Star 数"/></a>
|
||||
<a href="https://github.com/debpalash/memxt#readme"><img src="https://img.shields.io/badge/docs-README-10b981?style=flat-square" alt="memxt 文档"/></a>
|
||||
</p>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
---
|
||||
|
||||
<div align="center">
|
||||
|
||||
<br/>
|
||||
|
||||
如果你读到了这里，你就是我们的同路人。<br/>
|
||||
**[⭐ 给这个仓库点个 Star](https://github.com/debpalash/VoiceStudio)**，让更多人能找到它。<br/>
|
||||
**[💬 加入 Discord](https://discord.gg/bzQavDfVV9)**，分享你的作品。<br/>
|
||||
**[❤️ 支持开发](https://ko-fi.com/debpalash)**——资助让 VoiceStudio 持续发布的 AI 智能体账单。
|
||||
|
||||
<br/>
|
||||
|
||||
<a href="https://star-history.com/#debpalash/VoiceStudio&Date">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=debpalash/VoiceStudio&type=Date&theme=dark" />
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=debpalash/VoiceStudio&type=Date" />
|
||||
<img alt="Star 历史" src="https://api.star-history.com/svg?repos=debpalash/VoiceStudio&type=Date&theme=dark" width="600" />
|
||||
</picture>
|
||||
</a>
|
||||
</div>
|
||||
应用采用 [AGPL-3.0](LICENSE) 许可。模型遵循各自的许可，商用前请确认其条款。克隆声音前须取得本人许可。详见[许可说明](LICENSE-NOTICE.md)。
|
||||
|
||||
@@ -420,8 +420,11 @@ def _omnivoice_sampling_kwargs(opts: ExpressiveOptions) -> dict:
|
||||
today exactly: num_step 32, guidance 2.0, and NO temperature/postprocess
|
||||
kwargs (the model keeps its own defaults). Emotion is never forwarded —
|
||||
the VoiceStudio config rejects unknown kwargs."""
|
||||
from services.performance_profiles import tts_defaults
|
||||
|
||||
defaults = tts_defaults()
|
||||
kw = {
|
||||
"num_step": opts.num_step if opts.num_step is not None else LONGFORM_NUM_STEP,
|
||||
"num_step": opts.num_step if opts.num_step is not None else defaults.get("num_step", LONGFORM_NUM_STEP),
|
||||
"guidance_scale": (
|
||||
opts.guidance_scale if opts.guidance_scale is not None else LONGFORM_GUIDANCE_SCALE
|
||||
),
|
||||
@@ -432,6 +435,8 @@ def _omnivoice_sampling_kwargs(opts: ExpressiveOptions) -> dict:
|
||||
kw["class_temperature"] = opts.class_temperature
|
||||
if opts.postprocess_output is not None:
|
||||
kw["postprocess_output"] = opts.postprocess_output
|
||||
elif "postprocess_output" in defaults:
|
||||
kw["postprocess_output"] = defaults["postprocess_output"]
|
||||
return kw
|
||||
|
||||
|
||||
|
||||
+463
-146
@@ -9,6 +9,8 @@ the SQLite `jobs` table for history, but the queue itself restarts empty
|
||||
on backend restart — intentional, since GPU jobs can't be safely resumed.
|
||||
"""
|
||||
import os
|
||||
import json
|
||||
import shutil
|
||||
import uuid
|
||||
import time
|
||||
import asyncio
|
||||
@@ -16,20 +18,42 @@ import logging
|
||||
from typing import Optional, List
|
||||
|
||||
from fastapi import APIRouter, File, UploadFile, HTTPException, Form
|
||||
from fastapi.responses import JSONResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from core.config import DATA_DIR
|
||||
from core import failure
|
||||
from core.logging_utils import log_safe
|
||||
from core.file_cleanup import FileCleanupError, unlink_if_present
|
||||
from services.dub_batching import (
|
||||
BATCH_WIDTH_ENV,
|
||||
batch_timeout_s as _batch_timeout_s,
|
||||
native_batch_width as _native_batch_width,
|
||||
)
|
||||
from services import gpu_gateway
|
||||
from services.segment_bundle import extract_segment_wavs, remove_segment_wavs
|
||||
from services.tts_backend import active_backend_id, resolve_generation_backend
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.batch")
|
||||
|
||||
# Compatibility values emitted by the established Tauri Batch picker. They
|
||||
# are taxonomy tokens, not arbitrary prose, and are resolved server-side so
|
||||
# native watch-folder uploads and both desktop clients use the same voice.
|
||||
_BATCH_PRESET_INSTRUCT = {
|
||||
"narrator": "male, middle-aged, low pitch, british accent",
|
||||
"excited_child": "child, high pitch",
|
||||
"anxious_whisper": "young adult, whisper",
|
||||
"surprised_woman": "female, young adult, high pitch",
|
||||
"elderly_story": "male, elderly, very low pitch",
|
||||
"sichuan": "female, young adult, moderate pitch, \u56db\u5ddd\u8bdd",
|
||||
}
|
||||
|
||||
# ── In-memory queue ─────────────────────────────────────────────────────
|
||||
|
||||
_queue: asyncio.Queue = None # Lazily initialised
|
||||
_worker_task: asyncio.Task = None # Background consumer
|
||||
_processing_job_ids: set[str] = set()
|
||||
_jobs: dict = {} # job_id → status dict
|
||||
|
||||
|
||||
@@ -40,11 +64,15 @@ class BatchJobStatus(BaseModel):
|
||||
langs: List[str]
|
||||
voice_id: Optional[str] = None
|
||||
preserve_bg: bool = True
|
||||
translation_provider: Optional[str] = None
|
||||
created_at: float
|
||||
started_at: Optional[float] = None
|
||||
finished_at: Optional[float] = None
|
||||
error: Optional[str] = None
|
||||
progress: Optional[dict] = None
|
||||
attempts: int = 1
|
||||
retry_ready: bool = True
|
||||
setup_required: Optional[dict] = None
|
||||
|
||||
|
||||
def _ensure_queue():
|
||||
@@ -66,6 +94,7 @@ async def _worker():
|
||||
|
||||
job["status"] = "running"
|
||||
job["started_at"] = time.time()
|
||||
_processing_job_ids.add(job_id)
|
||||
logger.info("Batch job %s starting: %s", job_id, job["filename"])
|
||||
|
||||
try:
|
||||
@@ -95,6 +124,9 @@ async def _worker():
|
||||
job["finished_at"] = time.time()
|
||||
logger.error("Batch job %s failed: %s", job_id, e, exc_info=True)
|
||||
finally:
|
||||
_processing_job_ids.discard(job_id)
|
||||
if job["status"] == "cancelled":
|
||||
job["retry_ready"] = True
|
||||
_queue.task_done()
|
||||
|
||||
|
||||
@@ -104,18 +136,62 @@ def _set_progress(job, stage, percent=0, **extra):
|
||||
|
||||
|
||||
#: Override for the native dub batch width. Set to 1 to disable batching.
|
||||
BATCH_WIDTH_ENV = "OMNIVOICE_DUB_BATCH_WIDTH"
|
||||
|
||||
#: Hard ceiling on the override — a batch this wide is already amortizing
|
||||
#: almost all of the per-call setup, and beyond it the failure mode is an OOM
|
||||
#: that costs more than the saving.
|
||||
_MAX_BATCH_WIDTH = 16
|
||||
|
||||
# Bound each allocation while persisting multipart uploads. Video inputs can
|
||||
# be many gigabytes; `await UploadFile.read()` with no size used to mirror the
|
||||
# entire file in process memory before writing it back out.
|
||||
_UPLOAD_CHUNK_BYTES = 1024 * 1024
|
||||
|
||||
_REMOTE_BATCH_OPERATION = "batch_segments"
|
||||
|
||||
|
||||
async def _resolve_batch_execution(voice: dict):
|
||||
"""Resolve Batch's TTS target without loading local weights remotely."""
|
||||
engine_id = active_backend_id()
|
||||
decision = gpu_gateway.decide("batch")
|
||||
if decision.remote:
|
||||
await gpu_gateway.preflight(
|
||||
engine_id,
|
||||
decision,
|
||||
operation=_REMOTE_BATCH_OPERATION,
|
||||
)
|
||||
return engine_id, decision, None
|
||||
backend = await resolve_generation_backend(
|
||||
require_cloning=voice["requires_cloning"],
|
||||
cloning_purpose="this batch job's pinned voice",
|
||||
)
|
||||
return engine_id, decision, backend
|
||||
|
||||
|
||||
def _decode_remote_batch(
|
||||
result: gpu_gateway.RemoteResult,
|
||||
batch_dir: str,
|
||||
expected: set[int],
|
||||
) -> tuple[dict[int, str], int]:
|
||||
"""Validate and unpack one worker result before accepting remote success."""
|
||||
import soundfile as sf
|
||||
|
||||
target = os.path.join(batch_dir, ".remote", result.task_id)
|
||||
paths = extract_segment_wavs(result.path or "", target)
|
||||
try:
|
||||
if set(paths) != expected:
|
||||
missing = sorted(expected - set(paths))
|
||||
extra = sorted(set(paths) - expected)
|
||||
raise ValueError(
|
||||
f"segment bundle mismatch (missing={missing}, extra={extra})"
|
||||
)
|
||||
rates = {int(sf.info(path).samplerate) for path in paths.values()}
|
||||
if len(rates) != 1 or next(iter(rates), 0) <= 0:
|
||||
raise ValueError("segment bundle has inconsistent sample rates")
|
||||
return paths, rates.pop()
|
||||
except BaseException:
|
||||
remove_segment_wavs(paths)
|
||||
raise
|
||||
|
||||
|
||||
async def _save_upload(upload: UploadFile, destination: str) -> None:
|
||||
try:
|
||||
@@ -130,64 +206,63 @@ async def _save_upload(upload: UploadFile, destination: str) -> None:
|
||||
raise
|
||||
|
||||
|
||||
def _native_batch_width(backend) -> int:
|
||||
"""How many segments to render in one native batch on THIS host.
|
||||
def _batch_voice(voice_id: str | None) -> dict:
|
||||
"""Resolve one queue-wide voice into concrete generation inputs.
|
||||
|
||||
A native batch widens the forward pass, so the width cannot be a constant.
|
||||
The default engine declares ``min_vram_gb = 6.0`` for a SINGLE job; an
|
||||
unconditional 8-wide batch would OOM the 4-8 GB CUDA cards and the MPS
|
||||
Macs where the per-segment path succeeds today — turning a throughput
|
||||
optimization into a regression on exactly the hardware that already
|
||||
struggles (#1616 is a 4 GB card reporting capacity failures). Default
|
||||
behaviour must not get riskier on a host, so the width is derived from
|
||||
measured headroom and falls back to 1 (no batching) when unknown.
|
||||
|
||||
CPU hosts get 1: batching there buys no kernel amortization and only
|
||||
multiplies peak RAM.
|
||||
Clone profiles contribute their reference; designed profiles contribute
|
||||
their healed instruction and seed. Legacy ``preset:`` selections become
|
||||
the same instruction used by Dubbing instead of falling through to the
|
||||
engine default.
|
||||
"""
|
||||
override = os.environ.get(BATCH_WIDTH_ENV, "").strip()
|
||||
if override:
|
||||
try:
|
||||
return max(1, min(_MAX_BATCH_WIDTH, int(override)))
|
||||
except (TypeError, ValueError):
|
||||
logger.warning(
|
||||
"%s=%r is not an integer — deriving the batch width from the host instead.",
|
||||
BATCH_WIDTH_ENV, override,
|
||||
)
|
||||
try:
|
||||
from core.device_caps import detect_host_caps
|
||||
caps = detect_host_caps()
|
||||
except Exception: # noqa: BLE001 — an unprobeable host takes the safe path
|
||||
return 1
|
||||
if caps.family == "cpu" or not caps.vram_gb:
|
||||
return 1
|
||||
headroom = caps.vram_gb - float(getattr(backend, "min_vram_gb", 0.0) or 0.0)
|
||||
if headroom < 2.0:
|
||||
return 1
|
||||
if headroom < 6.0:
|
||||
return 2
|
||||
if headroom < 12.0:
|
||||
return 4
|
||||
return 8
|
||||
resolved = {
|
||||
"ref_audio": None,
|
||||
"ref_text": None,
|
||||
"instruct": "",
|
||||
"seed": None,
|
||||
"requires_cloning": False,
|
||||
}
|
||||
if not voice_id:
|
||||
return resolved
|
||||
if voice_id.startswith("preset:"):
|
||||
preset_id = voice_id.removeprefix("preset:")
|
||||
instruct = _BATCH_PRESET_INSTRUCT.get(preset_id)
|
||||
if instruct is None:
|
||||
raise ValueError("That built-in voice preset no longer exists")
|
||||
from omnivoice.utils.voice_design import sanitize_instruct
|
||||
|
||||
resolved["instruct"] = sanitize_instruct(instruct)
|
||||
return resolved
|
||||
|
||||
def _batch_timeout_s(texts: list[str], backend) -> float:
|
||||
"""Execution budget for one native batch.
|
||||
from core.config import VOICES_DIR
|
||||
from core.db import db_conn
|
||||
|
||||
Not the sum of the per-item budgets: ``generate_timeout_s`` returns a
|
||||
floor (300s GPU / 600s CPU) plus per-length overage, so summing it across
|
||||
eight items yields a ~2400s budget — and a wedged batch would hold a
|
||||
GPU-pool worker for forty minutes before the reset this file depends on
|
||||
(#730). One floor covers wedge detection for the whole call; only the
|
||||
length-driven overage is genuinely additive.
|
||||
"""
|
||||
from services.model_manager import generate_timeout_s
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT * FROM voice_profiles WHERE id=?",
|
||||
(voice_id,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
raise ValueError("That saved voice no longer exists")
|
||||
|
||||
floor = generate_timeout_s("", engine=backend)
|
||||
overage = sum(
|
||||
max(0.0, generate_timeout_s(text, engine=backend) - floor) for text in texts
|
||||
)
|
||||
return floor + overage
|
||||
if row["kind"] == "design":
|
||||
from omnivoice.utils.voice_design import heal_design_instruct
|
||||
|
||||
resolved["instruct"] = heal_design_instruct(row["instruct"], row["vd_states"])
|
||||
resolved["seed"] = int(row["seed"]) if row["seed"] is not None else None
|
||||
return resolved
|
||||
|
||||
relative = row["locked_audio_path"] if row["is_locked"] else row["ref_audio_path"]
|
||||
if not relative:
|
||||
raise ValueError("That saved voice has no reference audio")
|
||||
ref_audio = os.path.join(VOICES_DIR, relative)
|
||||
if not os.path.isfile(ref_audio):
|
||||
raise ValueError("That saved voice's reference audio is missing")
|
||||
resolved.update({
|
||||
"ref_audio": ref_audio,
|
||||
"ref_text": row["ref_text"],
|
||||
"requires_cloning": True,
|
||||
})
|
||||
return resolved
|
||||
|
||||
|
||||
async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
@@ -280,19 +355,30 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
return
|
||||
|
||||
# ── Engine resolution (issue #312 class) ────────────────────────────
|
||||
# Batch used to hardcode VoiceStudio via get_model() regardless of the
|
||||
# engine selected in Model Catalogue. require_cloning only when a
|
||||
# specific voice is pinned (job["voice_id"]) — an unpinned job is fine on
|
||||
# any active engine. Resolved ONCE for the whole job (every language
|
||||
# Batch used to hardcode VoiceStudio regardless of the engine selected in
|
||||
# Model Catalogue. Clone profiles require a cloning-capable engine; presets
|
||||
# and designed voices use instruction mode. Resolve once for the whole job.
|
||||
# below shares the same active engine); an uncaught ValueError here
|
||||
# propagates to _worker()'s existing except-Exception handling, which
|
||||
# already records a structured job failure via core.failure.build_failure.
|
||||
from services.tts_backend import resolve_generation_backend
|
||||
backend = await resolve_generation_backend(
|
||||
require_cloning=bool(job.get("voice_id")),
|
||||
cloning_purpose="this batch job's pinned voice",
|
||||
)
|
||||
sr = backend.sample_rate
|
||||
voice = _batch_voice(job.get("voice_id"))
|
||||
engine_id, execution_target, backend = await _resolve_batch_execution(voice)
|
||||
sr = backend.sample_rate if backend is not None else 0
|
||||
from services.performance_profiles import tts_defaults
|
||||
_profile_defaults = tts_defaults(engine_id)
|
||||
_batch_num_step = _profile_defaults.get("num_step", 16)
|
||||
_batch_postprocess = _profile_defaults.get("postprocess_output", True)
|
||||
batch_run = gpu_gateway.JobRun("batch")
|
||||
|
||||
async def _prepare_local_batch() -> gpu_gateway.LocalCall:
|
||||
nonlocal backend, sr
|
||||
if backend is None:
|
||||
backend = await resolve_generation_backend(
|
||||
require_cloning=voice["requires_cloning"],
|
||||
cloning_purpose="this batch job's pinned voice",
|
||||
)
|
||||
sr = backend.sample_rate
|
||||
return gpu_gateway.LocalCall(fn=lambda: None, what="Batch TTS fallback")
|
||||
|
||||
# ── 3. Translate + Generate per language ───────────────────────────
|
||||
total_langs = len(langs)
|
||||
@@ -311,40 +397,65 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
|
||||
translated_segments = list(segments) # copy
|
||||
if target_lang != source_lang:
|
||||
try:
|
||||
def _translate_batch(segs, src, tgt):
|
||||
"""Translate segment texts via Google Translate."""
|
||||
from deep_translator import GoogleTranslator
|
||||
TRANSLATE_CODES = {
|
||||
"en": "en", "es": "es", "fr": "fr", "de": "de",
|
||||
"it": "it", "pt": "pt", "ru": "ru", "ja": "ja",
|
||||
"ko": "ko", "zh": "zh-CN", "ar": "ar", "hi": "hi",
|
||||
"tr": "tr", "pl": "pl", "nl": "nl", "sv": "sv",
|
||||
}
|
||||
src_code = TRANSLATE_CODES.get(src, src) or "auto"
|
||||
tgt_code = TRANSLATE_CODES.get(tgt, tgt)
|
||||
translator = GoogleTranslator(source=src_code, target=tgt_code)
|
||||
out = []
|
||||
for s in segs:
|
||||
s_copy = dict(s)
|
||||
text = s.get("text", "").strip()
|
||||
if text:
|
||||
try:
|
||||
s_copy["text"] = translator.translate(text) or text
|
||||
except Exception as e:
|
||||
logger.warning("Translate seg failed: %s", e)
|
||||
out.append(s_copy)
|
||||
return out
|
||||
# Use the same provider dispatch as interactive Dubbing. The old
|
||||
# batch-only implementation hardcoded Google and silently kept the
|
||||
# source text on failure, which could make an English track labelled
|
||||
# "es" while also sending text online despite an offline selection.
|
||||
from api.routers.dub_translate import dub_translate
|
||||
from schemas.requests import TranslateRequest
|
||||
|
||||
translated_segments = await loop.run_in_executor(
|
||||
_cpu_pool, _translate_batch,
|
||||
segments, source_lang, target_lang,
|
||||
from core import prefs
|
||||
|
||||
provider = job.get("translation_provider") or prefs.get("translation_backend", "argos")
|
||||
translation = await dub_translate(TranslateRequest(
|
||||
segments=[
|
||||
{
|
||||
"id": str(segment["id"]),
|
||||
"text": segment.get("text", ""),
|
||||
"start": segment.get("start"),
|
||||
"end": segment.get("end"),
|
||||
}
|
||||
for segment in segments
|
||||
],
|
||||
source_lang=source_lang,
|
||||
target_lang=target_lang,
|
||||
provider=provider,
|
||||
quality="fast",
|
||||
))
|
||||
if isinstance(translation, JSONResponse):
|
||||
try:
|
||||
payload = json.loads(translation.body)
|
||||
detail = payload.get("error") or payload.get("detail")
|
||||
if payload.get("code") == "argos_pack_missing":
|
||||
job["setup_required"] = {
|
||||
"kind": "argos_packs",
|
||||
"source_lang": source_lang,
|
||||
"target_langs": [
|
||||
pair["target_lang"]
|
||||
for pair in payload.get("pairs", [])
|
||||
if isinstance(pair, dict) and pair.get("target_lang")
|
||||
],
|
||||
}
|
||||
except Exception: # noqa: BLE001 — retain the stable fallback
|
||||
detail = None
|
||||
raise RuntimeError(
|
||||
detail or f"{provider} could not translate this batch"
|
||||
)
|
||||
except ImportError:
|
||||
logger.warning("deep_translator not installed, skipping translation for %s", target_lang)
|
||||
except Exception as e:
|
||||
logger.warning("Translation failed for %s: %s, using original", target_lang, e)
|
||||
translated_segments = segments
|
||||
rows = {
|
||||
str(row.get("id")): row
|
||||
for row in translation.get("translated", [])
|
||||
if isinstance(row, dict)
|
||||
}
|
||||
failed = [row for row in rows.values() if row.get("error")]
|
||||
if failed or len(rows) != len(segments):
|
||||
raise RuntimeError(
|
||||
f"{provider} translation failed for "
|
||||
f"{len(failed) or len(segments) - len(rows)} segment(s)"
|
||||
)
|
||||
translated_segments = [
|
||||
{**segment, "text": rows[str(segment["id"])]["text"]}
|
||||
for segment in segments
|
||||
]
|
||||
|
||||
if job["status"] == "cancelled":
|
||||
return
|
||||
@@ -362,6 +473,90 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
from services.audio_io import atomic_save_wav
|
||||
import torch
|
||||
|
||||
remote_segments: dict[int, str] = {}
|
||||
valid_rows = [
|
||||
(i, segment)
|
||||
for i, segment in enumerate(translated_segments)
|
||||
if segment.get("end", 0) - segment.get("start", 0) > 0.05
|
||||
and segment.get("text", "").strip()
|
||||
]
|
||||
if execution_target.remote and valid_rows:
|
||||
remote_rows = [
|
||||
{
|
||||
"index": i,
|
||||
"text": segment.get("text", "").strip(),
|
||||
"language": target_lang,
|
||||
"ref_text": voice["ref_text"],
|
||||
"instruct": voice["instruct"] or None,
|
||||
"duration": segment.get("end", 0) - segment.get("start", 0),
|
||||
"num_step": _batch_num_step,
|
||||
"postprocess_output": _batch_postprocess,
|
||||
"guidance_scale": 2.0,
|
||||
"speed": 1.0,
|
||||
"effect_preset": "batch",
|
||||
"seed": (
|
||||
voice["seed"] + i if voice["seed"] is not None else None
|
||||
),
|
||||
# The assembled track receives one watermark below. Marking
|
||||
# each line here would double-process remote output.
|
||||
"watermark": False,
|
||||
}
|
||||
for i, segment in valid_rows
|
||||
]
|
||||
expected = {row["index"] for row in remote_rows}
|
||||
|
||||
def _remote_state(state: dict) -> None:
|
||||
fraction = max(0.0, min(1.0, float(state.get("progress") or 0.0)))
|
||||
_set_progress(
|
||||
job,
|
||||
"generate",
|
||||
percent=int(((lang_idx + fraction) / total_langs) * 100),
|
||||
current_lang=target_lang,
|
||||
current_segment=min(len(remote_rows), round(fraction * len(remote_rows))),
|
||||
total_segments=len(remote_rows),
|
||||
execution_target=execution_target.label,
|
||||
execution_phase=state.get("phase"),
|
||||
)
|
||||
|
||||
route_task = asyncio.create_task(
|
||||
gpu_gateway.run(
|
||||
"batch",
|
||||
local=gpu_gateway.LocalCall(prepare=_prepare_local_batch),
|
||||
remote=gpu_gateway.RemoteCall(
|
||||
engine=engine_id,
|
||||
operation=_REMOTE_BATCH_OPERATION,
|
||||
params={
|
||||
"segments": remote_rows,
|
||||
"ref_audio": [voice["ref_audio"] for _ in remote_rows],
|
||||
"input_seconds": sum(
|
||||
float(row.get("duration") or 0.0) for row in remote_rows
|
||||
),
|
||||
},
|
||||
idempotency_key=f"batch:{job_id}:{target_lang}",
|
||||
decode=lambda result: _decode_remote_batch(
|
||||
result, batch_dir, expected
|
||||
),
|
||||
),
|
||||
decision=execution_target,
|
||||
job=batch_run,
|
||||
on_state=_remote_state,
|
||||
)
|
||||
)
|
||||
while not route_task.done():
|
||||
await asyncio.wait({route_task}, timeout=0.25)
|
||||
if job["status"] == "cancelled":
|
||||
route_task.cancel()
|
||||
try:
|
||||
await route_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
return
|
||||
routed = route_task.result()
|
||||
if routed is not None:
|
||||
remote_segments, sr = routed
|
||||
|
||||
# A remote-only empty transcript still needs a valid silent-track rate.
|
||||
sr = sr or 24_000
|
||||
total_samples = int(duration * sr)
|
||||
full_audio = torch.zeros(1, total_samples)
|
||||
total_segs = len(translated_segments)
|
||||
@@ -372,26 +567,15 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
# the established one-segment behavior below.
|
||||
from services.tts_backend import TTSBackend
|
||||
batched_audio: dict[int, torch.Tensor] = {}
|
||||
has_native_batch = type(backend).generate_batch is not TTSBackend.generate_batch
|
||||
has_native_batch = (
|
||||
backend is not None
|
||||
and type(backend).generate_batch is not TTSBackend.generate_batch
|
||||
)
|
||||
if has_native_batch:
|
||||
from services.text_normalization import normalize_for_tts
|
||||
|
||||
batch_ref_audio = None
|
||||
batch_ref_text = None
|
||||
if job.get("voice_id"):
|
||||
from core.db import db_conn
|
||||
from core.config import VOICES_DIR as _VD
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT * FROM voice_profiles WHERE id=?",
|
||||
(job["voice_id"],),
|
||||
).fetchone()
|
||||
if row:
|
||||
if row["is_locked"] and row["locked_audio_path"]:
|
||||
batch_ref_audio = os.path.join(_VD, row["locked_audio_path"])
|
||||
elif row["ref_audio_path"]:
|
||||
batch_ref_audio = os.path.join(_VD, row["ref_audio_path"])
|
||||
batch_ref_text = row["ref_text"]
|
||||
batch_ref_audio = voice["ref_audio"]
|
||||
batch_ref_text = voice["ref_text"]
|
||||
|
||||
batch_width = _native_batch_width(backend)
|
||||
|
||||
@@ -428,17 +612,20 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
]
|
||||
|
||||
def _render_native_batch():
|
||||
if voice["seed"] is not None:
|
||||
torch.manual_seed(voice["seed"])
|
||||
generated = backend.generate_batch(
|
||||
batch_texts,
|
||||
language=target_lang,
|
||||
ref_audio=batch_ref_audio,
|
||||
ref_text=batch_ref_text,
|
||||
instruct=voice["instruct"] or None,
|
||||
duration=batch_durations,
|
||||
num_step=16,
|
||||
num_step=_batch_num_step,
|
||||
guidance_scale=2.0,
|
||||
speed=1.0,
|
||||
denoise=True,
|
||||
postprocess_output=True,
|
||||
postprocess_output=_batch_postprocess,
|
||||
)
|
||||
if len(generated) != len(batch_indices):
|
||||
raise RuntimeError(
|
||||
@@ -473,6 +660,7 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
|
||||
for i, seg in enumerate(translated_segments):
|
||||
if job["status"] == "cancelled":
|
||||
remove_segment_wavs(remote_segments)
|
||||
return
|
||||
|
||||
_set_progress(
|
||||
@@ -499,32 +687,18 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
from services.text_normalization import normalize_for_tts
|
||||
text = normalize_for_tts(text, lang)
|
||||
|
||||
ref_audio = None
|
||||
ref_text = None
|
||||
|
||||
# Use voice_id if provided
|
||||
if job.get("voice_id"):
|
||||
from core.db import db_conn
|
||||
from core.config import VOICES_DIR as _VD
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT * FROM voice_profiles WHERE id=?",
|
||||
(job["voice_id"],),
|
||||
).fetchone()
|
||||
if row:
|
||||
if row["is_locked"] and row["locked_audio_path"]:
|
||||
ref_audio = os.path.join(_VD, row["locked_audio_path"])
|
||||
elif row["ref_audio_path"]:
|
||||
ref_audio = os.path.join(_VD, row["ref_audio_path"])
|
||||
ref_text = row.get("ref_text")
|
||||
|
||||
try:
|
||||
if backend is None:
|
||||
raise RuntimeError("the local TTS fallback was not prepared")
|
||||
if voice["seed"] is not None:
|
||||
torch.manual_seed(voice["seed"] + i)
|
||||
audio_out = backend.generate(
|
||||
text=text, language=lang,
|
||||
ref_audio=ref_audio, ref_text=ref_text,
|
||||
duration=dur, num_step=16,
|
||||
ref_audio=voice["ref_audio"], ref_text=voice["ref_text"],
|
||||
instruct=voice["instruct"] or None,
|
||||
duration=dur, num_step=_batch_num_step,
|
||||
guidance_scale=2.0, speed=1.0,
|
||||
denoise=True, postprocess_output=True,
|
||||
denoise=True, postprocess_output=_batch_postprocess,
|
||||
)
|
||||
if not getattr(backend, "applies_own_mastering", False):
|
||||
audio_out = apply_mastering(audio_out, sample_rate=sr)
|
||||
@@ -548,15 +722,43 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
# Budget is the shared length-scaled one (#1190): a long segment
|
||||
# on CPU-class hardware no longer dies on the flat 300s.
|
||||
from services.model_manager import generate_timeout_s
|
||||
if has_native_batch and i not in batched_audio:
|
||||
await _prefetch_batch(i)
|
||||
if i in batched_audio:
|
||||
audio_tensor = batched_audio.pop(i)
|
||||
remote_path = remote_segments.pop(i, None)
|
||||
if remote_path is not None:
|
||||
import soundfile as sf
|
||||
|
||||
try:
|
||||
audio_array, remote_sr = sf.read(
|
||||
remote_path,
|
||||
dtype="float32",
|
||||
always_2d=True,
|
||||
)
|
||||
if int(remote_sr) != sr:
|
||||
raise ValueError(
|
||||
f"remote segment sample rate changed from {sr} to {remote_sr}"
|
||||
)
|
||||
audio_tensor = torch.from_numpy(audio_array.T).mean(
|
||||
dim=0,
|
||||
keepdim=True,
|
||||
)
|
||||
finally:
|
||||
remove_segment_wavs({i: remote_path})
|
||||
else:
|
||||
audio_tensor = await run_on_gpu_pool_guarded(
|
||||
_gen, what="Batch generate",
|
||||
timeout=generate_timeout_s(seg_text, engine=backend),
|
||||
)
|
||||
if backend is None:
|
||||
await _prepare_local_batch()
|
||||
# This path means a validated remote bundle lost a row
|
||||
# after dispatch. Recover only that row; native batches
|
||||
# were not planned for this language.
|
||||
has_native_batch = False
|
||||
if has_native_batch and i not in batched_audio:
|
||||
await _prefetch_batch(i)
|
||||
if i in batched_audio:
|
||||
audio_tensor = batched_audio.pop(i)
|
||||
else:
|
||||
audio_tensor = await run_on_gpu_pool_guarded(
|
||||
_gen,
|
||||
what="Batch generate",
|
||||
timeout=generate_timeout_s(seg_text, engine=backend),
|
||||
)
|
||||
|
||||
# Fit to slot
|
||||
target_samples_seg = int(seg_duration * sr)
|
||||
@@ -604,6 +806,8 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
f"left silent: {e}"
|
||||
)
|
||||
|
||||
remove_segment_wavs(remote_segments)
|
||||
|
||||
# ── 3c. Save dubbed audio track ───────────────────────────────
|
||||
# Invisible provenance mark on the assembled track (#1169), tensor
|
||||
# stage, before the WAV write / aac mux — batch dubs used to ship
|
||||
@@ -670,6 +874,7 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
outputs[target_lang] = output_path
|
||||
|
||||
job["outputs"] = outputs
|
||||
job.pop("setup_required", None)
|
||||
_set_progress(job, "done", 100)
|
||||
|
||||
|
||||
@@ -681,6 +886,7 @@ async def enqueue_batch_job(
|
||||
langs: str = Form("es"), # comma-separated lang codes
|
||||
voice_id: Optional[str] = Form(None),
|
||||
preserve_bg: bool = Form(True),
|
||||
translation_provider: Optional[str] = Form(None),
|
||||
):
|
||||
"""Enqueue a video for batch dubbing.
|
||||
|
||||
@@ -694,6 +900,14 @@ async def enqueue_batch_job(
|
||||
if not lang_list:
|
||||
raise HTTPException(400, "At least one target language is required")
|
||||
|
||||
# Validate the snapshot before persisting a potentially large upload.
|
||||
# Resolve it again in the worker so deleting or editing a queued profile
|
||||
# cannot silently fall back to the engine's default voice.
|
||||
try:
|
||||
await asyncio.to_thread(_batch_voice, voice_id)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
|
||||
# TTS-only install: no ASR model on disk → typed 409 with a download CTA
|
||||
# now, instead of accepting the job and having the transcribe stage
|
||||
# silently auto-download multi-GB whisper weights (or fail) in the worker.
|
||||
@@ -702,6 +916,19 @@ async def enqueue_batch_job(
|
||||
if missing is not None:
|
||||
raise HTTPException(409, {**missing, "message": asr_model_missing_detail(missing)})
|
||||
|
||||
# Snapshot the selected translation engine when the user enqueues the job,
|
||||
# so a later Settings change cannot alter work already waiting in the queue.
|
||||
from core import prefs
|
||||
from services import translation_engines
|
||||
|
||||
provider = translation_provider or prefs.get("translation_backend", "argos")
|
||||
if not translation_engines.get_engine(provider):
|
||||
raise HTTPException(400, "Unknown translation engine")
|
||||
if not translation_engines.is_installed(provider):
|
||||
raise HTTPException(409, "Install the selected translation engine before adding this batch")
|
||||
if not translation_engines.is_ready(provider):
|
||||
raise HTTPException(409, "Configure the selected translation provider before adding this batch")
|
||||
|
||||
# Save the uploaded video
|
||||
batch_dir = os.path.join(DATA_DIR, "batch")
|
||||
os.makedirs(batch_dir, exist_ok=True)
|
||||
@@ -718,7 +945,9 @@ async def enqueue_batch_job(
|
||||
"langs": lang_list,
|
||||
"voice_id": voice_id,
|
||||
"preserve_bg": preserve_bg,
|
||||
"translation_provider": provider,
|
||||
"created_at": time.time(),
|
||||
"attempts": 1,
|
||||
"started_at": None,
|
||||
"finished_at": None,
|
||||
"error": None,
|
||||
@@ -741,6 +970,8 @@ def list_batch_jobs(status: Optional[str] = None, limit: int = 50):
|
||||
if status:
|
||||
if status == "active":
|
||||
jobs = [j for j in jobs if j["status"] in ("queued", "running")]
|
||||
elif status == "retryable":
|
||||
jobs = [j for j in jobs if j["status"] in ("failed", "cancelled")]
|
||||
else:
|
||||
jobs = [j for j in jobs if j["status"] == status]
|
||||
jobs.sort(key=lambda j: j["created_at"], reverse=True)
|
||||
@@ -764,14 +995,88 @@ def cancel_batch_job(job_id: str):
|
||||
raise HTTPException(404, "Job not found")
|
||||
if job["status"] in ("done", "failed", "cancelled"):
|
||||
return {"already": job["status"]}
|
||||
was_running = job["status"] == "running" or job_id in _processing_job_ids
|
||||
job["status"] = "cancelled"
|
||||
job["retry_ready"] = not was_running
|
||||
job["finished_at"] = time.time()
|
||||
return {"cancelled": True}
|
||||
|
||||
|
||||
@router.post("/batch/jobs/{job_id}/retry")
|
||||
async def retry_batch_job(job_id: str):
|
||||
"""Retry a terminal job using its original app-owned upload and settings."""
|
||||
job = _jobs.get(job_id)
|
||||
if not job:
|
||||
raise HTTPException(404, "Job not found")
|
||||
if job["status"] not in ("failed", "cancelled"):
|
||||
raise HTTPException(409, f"Job is {job['status']}, not retryable")
|
||||
if job_id in _processing_job_ids or not job.get("retry_ready", True):
|
||||
raise HTTPException(409, "The cancelled job is still stopping")
|
||||
if not os.path.isfile(job.get("video_path") or ""):
|
||||
raise HTTPException(409, "The original batch input is no longer available")
|
||||
|
||||
try:
|
||||
await asyncio.to_thread(_batch_voice, job.get("voice_id"))
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
|
||||
from services.asr_backend import asr_model_missing_detail, asr_model_missing_error
|
||||
missing = await asyncio.to_thread(asr_model_missing_error)
|
||||
if missing is not None:
|
||||
raise HTTPException(409, {**missing, "message": asr_model_missing_detail(missing)})
|
||||
|
||||
from services import translation_engines
|
||||
provider = job.get("translation_provider") or "argos"
|
||||
if not translation_engines.is_ready(provider):
|
||||
raise HTTPException(409, "Configure the selected translation provider before retrying")
|
||||
if provider == "argos" and job.get("source_lang"):
|
||||
status = await asyncio.to_thread(
|
||||
translation_engines.argos_pack_status,
|
||||
job["source_lang"],
|
||||
job["langs"],
|
||||
)
|
||||
if any(not pair["installed"] for pair in status["pairs"]):
|
||||
raise HTTPException(409, "Install the required Argos language packs before retrying")
|
||||
|
||||
batch_root = os.path.realpath(os.path.join(DATA_DIR, "batch"))
|
||||
output_dir = os.path.realpath(os.path.join(batch_root, job_id))
|
||||
if os.path.dirname(output_dir) != batch_root:
|
||||
raise HTTPException(status_code=400, detail="Invalid batch job path")
|
||||
try:
|
||||
if os.path.isdir(output_dir):
|
||||
await asyncio.to_thread(shutil.rmtree, output_dir)
|
||||
except OSError as exc:
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Could not reset the batch output files. Close any app using them and retry.",
|
||||
) from exc
|
||||
|
||||
for key in (
|
||||
"duration",
|
||||
"segments",
|
||||
"source_lang",
|
||||
"outputs",
|
||||
"warnings",
|
||||
"setup_required",
|
||||
"retry_ready",
|
||||
):
|
||||
job.pop(key, None)
|
||||
job.update({
|
||||
"status": "queued",
|
||||
"started_at": None,
|
||||
"finished_at": None,
|
||||
"error": None,
|
||||
"progress": None,
|
||||
"attempts": int(job.get("attempts", 1)) + 1,
|
||||
})
|
||||
_ensure_queue()
|
||||
await _queue.put(job_id)
|
||||
return {"job_id": job_id, "status": "queued", "queue_position": _queue.qsize()}
|
||||
|
||||
|
||||
@router.delete("/batch/jobs/{job_id}")
|
||||
def delete_batch_job(job_id: str):
|
||||
"""Delete a batch job record and its video file."""
|
||||
"""Delete a batch job record and every app-owned input/output file."""
|
||||
job = _jobs.get(job_id)
|
||||
if not job:
|
||||
raise HTTPException(404, "Job not found")
|
||||
@@ -783,6 +1088,18 @@ def delete_batch_job(job_id: str):
|
||||
status_code=500,
|
||||
detail="Could not delete the batch video file. Close any app using it and retry.",
|
||||
) from exc
|
||||
batch_root = os.path.realpath(os.path.join(DATA_DIR, "batch"))
|
||||
output_dir = os.path.realpath(os.path.join(batch_root, job_id))
|
||||
if os.path.dirname(output_dir) != batch_root:
|
||||
raise HTTPException(status_code=400, detail="Invalid batch job path")
|
||||
try:
|
||||
if os.path.isdir(output_dir):
|
||||
shutil.rmtree(output_dir)
|
||||
except OSError as exc:
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Could not delete the batch output files. Close any app using them and retry.",
|
||||
) from exc
|
||||
_jobs.pop(job_id, None)
|
||||
return {"deleted": True}
|
||||
|
||||
|
||||
@@ -60,7 +60,8 @@ async def transcribe_audio(
|
||||
language: Optional language hint (not currently used; auto-detected).
|
||||
model: Whisper model size (legacy; ignored in dual-mode architecture).
|
||||
mode: 'fast' (default) uses MLX Turbo for speed; 'accurate' uses
|
||||
WhisperX with forced alignment for word-level timing.
|
||||
the selected ASR engine with word-level timing. 'reference' uses
|
||||
the selected ASR engine without word-level timing.
|
||||
refine: Opt-in local-LLM cleanup of the final text (disfluencies,
|
||||
self-corrections, punctuation) — same pipeline the live
|
||||
dictation socket uses. Off by default so MCP/CLI callers don't
|
||||
@@ -91,7 +92,9 @@ async def transcribe_audio(
|
||||
tmp.write(content)
|
||||
tmp.close()
|
||||
|
||||
use_accurate = (mode or "").strip().lower() == "accurate"
|
||||
requested_mode = (mode or "").strip().lower()
|
||||
use_accurate = requested_mode == "accurate"
|
||||
use_active_asr = requested_mode in {"accurate", "reference"}
|
||||
|
||||
# TTS-only install: no ASR model on disk → typed 409 with a download
|
||||
# CTA, BEFORE any backend is constructed (the whisper backends
|
||||
@@ -99,7 +102,8 @@ async def transcribe_audio(
|
||||
from services.asr_backend import asr_model_missing_detail, asr_model_missing_error
|
||||
missing = await asyncio.to_thread(
|
||||
asr_model_missing_error,
|
||||
purpose="transcribe" if use_accurate else "dictation",
|
||||
purpose="transcribe" if use_active_asr else "dictation",
|
||||
require_installed=requested_mode == "reference",
|
||||
)
|
||||
if missing is not None:
|
||||
raise HTTPException(
|
||||
@@ -108,7 +112,7 @@ async def transcribe_audio(
|
||||
)
|
||||
|
||||
def _run():
|
||||
if use_accurate:
|
||||
if use_active_asr:
|
||||
# Accurate mode: full WhisperX with forced alignment —
|
||||
# for when the user explicitly wants word-level timing.
|
||||
# `load_*`, not `get_*`: the selector alone hands back an
|
||||
@@ -116,8 +120,8 @@ async def transcribe_audio(
|
||||
# chain is broken, which then 500s at `.transcribe()`. The
|
||||
# loader degrades to the next healthy engine (#1185).
|
||||
from services.asr_backend import load_active_asr_backend
|
||||
backend = load_active_asr_backend()
|
||||
result = backend.transcribe(tmp.name, word_timestamps=True)
|
||||
backend = load_active_asr_backend(require_installed=True) if requested_mode == "reference" else load_active_asr_backend()
|
||||
result = backend.transcribe(tmp.name, word_timestamps=use_accurate)
|
||||
else:
|
||||
# Fast mode (default): use the fastest available engine
|
||||
# (MLX Turbo on Apple Silicon). Skip word_timestamps for
|
||||
@@ -125,7 +129,8 @@ async def transcribe_audio(
|
||||
from services.asr_backend import get_capture_asr_backend
|
||||
backend = get_capture_asr_backend()
|
||||
result = backend.transcribe(tmp.name, word_timestamps=False)
|
||||
return result, backend.id
|
||||
sherpa_model_id = getattr(getattr(backend, "spec", None), "id", None)
|
||||
return result, backend.id, sherpa_model_id
|
||||
|
||||
from services.model_manager import _gpu_pool
|
||||
from services.asr_backend import (
|
||||
@@ -135,7 +140,7 @@ async def transcribe_audio(
|
||||
)
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
result, engine_id = await run_transcribe_guarded(
|
||||
result, engine_id, sherpa_model_id = await run_transcribe_guarded(
|
||||
_gpu_pool, _run, what="Dictation",
|
||||
)
|
||||
except ASRTimeoutError as e:
|
||||
@@ -151,6 +156,69 @@ async def transcribe_audio(
|
||||
status_code=409,
|
||||
detail={**e.payload, "message": asr_model_missing_detail(e.payload)},
|
||||
)
|
||||
|
||||
# Some sherpa-onnx NeMo-TDT builds load successfully but decode an
|
||||
# entire spoken clip to no tokens. Live dictation already recovers
|
||||
# from that failure; the shared file endpoint must do the same because
|
||||
# it also powers uploaded transcription and automatic profile text.
|
||||
# Retry only through an already-installed fallback, and demote the
|
||||
# silent model only when the second recognizer actually heard words.
|
||||
initial_text = str(result.get("text") or "").strip()
|
||||
if not initial_text and result.get("segments"):
|
||||
initial_text = " ".join(
|
||||
str(segment.get("text") or "")
|
||||
for segment in result["segments"]
|
||||
if isinstance(segment, dict)
|
||||
).strip()
|
||||
recovered_from = None
|
||||
if not use_active_asr and sherpa_model_id and not initial_text:
|
||||
fallback_missing = await asyncio.to_thread(
|
||||
asr_model_missing_error,
|
||||
purpose="dictation",
|
||||
skip_sherpa=True,
|
||||
require_installed=True,
|
||||
)
|
||||
if fallback_missing is None:
|
||||
def _run_fallback():
|
||||
from services.asr_backend import get_capture_asr_backend
|
||||
|
||||
fallback = get_capture_asr_backend(skip_sherpa=True)
|
||||
return (
|
||||
fallback.transcribe(tmp.name, word_timestamps=False),
|
||||
fallback.id,
|
||||
)
|
||||
|
||||
try:
|
||||
fallback_result, fallback_engine_id = await run_transcribe_guarded(
|
||||
_gpu_pool,
|
||||
_run_fallback,
|
||||
what="Dictation fallback",
|
||||
)
|
||||
fallback_text = str(fallback_result.get("text") or "").strip()
|
||||
if not fallback_text and fallback_result.get("segments"):
|
||||
fallback_text = " ".join(
|
||||
str(segment.get("text") or "")
|
||||
for segment in fallback_result["segments"]
|
||||
if isinstance(segment, dict)
|
||||
).strip()
|
||||
if fallback_text:
|
||||
from services.sherpa_dictation import demote_model
|
||||
|
||||
await asyncio.to_thread(demote_model, sherpa_model_id)
|
||||
result = fallback_result
|
||||
engine_id = fallback_engine_id
|
||||
recovered_from = sherpa_model_id
|
||||
logger.warning(
|
||||
"File transcription recovered from silent dictation model %s "
|
||||
"through installed engine %s",
|
||||
sherpa_model_id,
|
||||
fallback_engine_id,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Installed fallback failed after dictation model %s returned no text",
|
||||
sherpa_model_id,
|
||||
)
|
||||
elapsed = round(time.perf_counter() - t0, 2)
|
||||
|
||||
# Normalize result shape
|
||||
@@ -202,7 +270,7 @@ async def transcribe_audio(
|
||||
|
||||
logger.info(
|
||||
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s, refined=%s",
|
||||
engine_id, elapsed, duration, "accurate" if use_accurate else "fast",
|
||||
engine_id, elapsed, duration, requested_mode if use_active_asr else "fast",
|
||||
refined_text is not None,
|
||||
)
|
||||
|
||||
@@ -223,6 +291,8 @@ async def transcribe_audio(
|
||||
}
|
||||
if refined_text is not None:
|
||||
response["refined_text"] = refined_text
|
||||
if recovered_from is not None:
|
||||
response["model_silent"] = recovered_from
|
||||
return response
|
||||
finally:
|
||||
try:
|
||||
|
||||
@@ -21,7 +21,7 @@ import logging
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
from typing import Optional
|
||||
from typing import Literal, Optional
|
||||
|
||||
from api.dependencies import require_local
|
||||
from api.public_engine_metadata import public_unavailability
|
||||
@@ -87,13 +87,18 @@ def list_dictation_models():
|
||||
|
||||
|
||||
@router.get("/dictation/readiness", dependencies=[Depends(require_local)])
|
||||
def dictation_readiness(model_id: str | None = None) -> dict:
|
||||
"""Check capture's model selection without loading or downloading weights."""
|
||||
def dictation_readiness(
|
||||
model_id: str | None = None,
|
||||
purpose: Literal["dictation", "transcribe"] = "dictation",
|
||||
) -> dict:
|
||||
"""Check a selected ASR path without loading or downloading weights."""
|
||||
from services.asr_backend import asr_model_missing_error
|
||||
|
||||
missing = asr_model_missing_error(
|
||||
purpose="dictation",
|
||||
sherpa_model_id=model_id or _read_prefs()["model_id"],
|
||||
purpose=purpose,
|
||||
sherpa_model_id=(model_id or _read_prefs()["model_id"])
|
||||
if purpose == "dictation"
|
||||
else None,
|
||||
)
|
||||
return {"ready": missing is None, "missing": missing}
|
||||
|
||||
|
||||
+209
-17
@@ -355,6 +355,138 @@ async def dub_import_srt(job_id: str, file: UploadFile = File(...)):
|
||||
}
|
||||
|
||||
|
||||
def _select_downloaded_caption_track(
|
||||
tracks: dict[str, list[dict]], preferred: str | None,
|
||||
) -> str | None:
|
||||
"""Choose the closest original-language caption track deterministically."""
|
||||
available = [key for key, cues in tracks.items() if isinstance(cues, list) and cues]
|
||||
if not available:
|
||||
return None
|
||||
preferred_tag = (preferred or "").strip().lower().replace("_", "-")
|
||||
preferred_base = preferred_tag.split("-", 1)[0]
|
||||
|
||||
def rank(key: str) -> tuple[int, int, int, str]:
|
||||
tag = key.strip().lower().replace("_", "-")
|
||||
base = tag.split("-", 1)[0]
|
||||
if preferred_tag:
|
||||
language_rank = 0 if tag == preferred_tag else 1 if base == preferred_base else 2
|
||||
else:
|
||||
language_rank = 0
|
||||
return (
|
||||
language_rank,
|
||||
0 if tag.endswith("-orig") else 1,
|
||||
0 if "-" not in tag else 1,
|
||||
tag,
|
||||
)
|
||||
|
||||
return min(available, key=rank)
|
||||
|
||||
|
||||
def _prepare_downloaded_caption_segments(cues: list[dict], duration: float) -> list[dict]:
|
||||
"""Normalize downloaded VTT cues into safe, sequential Dub segments."""
|
||||
def cue_start(cue: dict) -> float:
|
||||
try:
|
||||
return float(cue.get("start") or 0.0)
|
||||
except (TypeError, ValueError):
|
||||
return 0.0
|
||||
|
||||
def remove_repeated_prefix(previous: str, current: str) -> str:
|
||||
previous_words = previous.split()
|
||||
current_words = current.split()
|
||||
folded_previous = [word.casefold() for word in previous_words]
|
||||
folded_current = [word.casefold() for word in current_words]
|
||||
for count in range(min(len(previous_words), len(current_words)), 0, -1):
|
||||
if folded_previous[-count:] == folded_current[:count]:
|
||||
return " ".join(current_words[count:])
|
||||
return current
|
||||
|
||||
prepared: list[dict] = []
|
||||
previous_end = 0.0
|
||||
ordered = sorted((cue for cue in cues if isinstance(cue, dict)), key=cue_start)
|
||||
for index, cue in enumerate(ordered):
|
||||
try:
|
||||
raw_start = max(0.0, float(cue.get("start") or 0.0))
|
||||
end = float(cue.get("end") or raw_start)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
text = " ".join(str(cue.get("text") or "").split())
|
||||
if duration > 0:
|
||||
if raw_start >= duration:
|
||||
continue
|
||||
end = min(end, duration)
|
||||
if prepared and raw_start < previous_end:
|
||||
text = remove_repeated_prefix(prepared[-1]["text"], text)
|
||||
if not text:
|
||||
prepared[-1]["end"] = round(max(previous_end, end), 3)
|
||||
previous_end = max(previous_end, end)
|
||||
continue
|
||||
# Caption hosts commonly emit slightly overlapping cues. Dubbing needs
|
||||
# a monotonic timeline, so trim the later cue rather than manufacture
|
||||
# overlapping speech slots.
|
||||
start = max(raw_start, previous_end)
|
||||
if not text or end <= start:
|
||||
continue
|
||||
prepared.append({
|
||||
"id": str(index),
|
||||
"start": round(start, 3),
|
||||
"end": round(end, 3),
|
||||
"text": text,
|
||||
"speaker_id": "Speaker 1",
|
||||
})
|
||||
previous_end = end
|
||||
|
||||
cleaned = clean_up_segments(prepared)
|
||||
return [
|
||||
{
|
||||
**segment,
|
||||
"id": index,
|
||||
"text_original": segment.get("text", ""),
|
||||
}
|
||||
for index, segment in enumerate(cleaned)
|
||||
]
|
||||
|
||||
|
||||
@router.post("/dub/use-downloaded-captions/{job_id}")
|
||||
def dub_use_downloaded_captions(job_id: str):
|
||||
"""Seed a prepared Dub job from its downloaded caption track."""
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
tracks = job.get("youtube_subs")
|
||||
if not isinstance(tracks, dict):
|
||||
raise HTTPException(status_code=404, detail="No downloaded captions are available")
|
||||
caption_lang = _select_downloaded_caption_track(
|
||||
tracks,
|
||||
job.get("source_lang_override") or job.get("source_lang"),
|
||||
)
|
||||
if caption_lang is None:
|
||||
raise HTTPException(status_code=404, detail="No downloaded captions are available")
|
||||
segments = _prepare_downloaded_caption_segments(
|
||||
tracks[caption_lang],
|
||||
float(job.get("duration") or 0.0),
|
||||
)
|
||||
if not segments:
|
||||
raise HTTPException(status_code=422, detail="Downloaded captions contain no usable cues")
|
||||
|
||||
source_lang = job.get("source_lang_override") or _detected_source_lang(caption_lang)
|
||||
job["segments"] = segments
|
||||
job["source_lang"] = source_lang
|
||||
job["full_transcript"] = " ".join(segment["text"] for segment in segments)
|
||||
# Caption files contain timing and text, but no trustworthy speaker or
|
||||
# reference-audio attribution. Never retain stale clone maps from a prior
|
||||
# transcript on the same job.
|
||||
job["segment_clones"] = {}
|
||||
job["speaker_clones"] = {}
|
||||
job.pop("cast_sources", None)
|
||||
_save_job(job_id, job)
|
||||
return {
|
||||
"segments": segments,
|
||||
"source_lang": source_lang,
|
||||
"caption_lang": caption_lang,
|
||||
"available": sorted(tracks.keys()),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/dub/cleanup-segments/{job_id}")
|
||||
def dub_cleanup_segments(job_id: str):
|
||||
"""Re-run merge/stitch passes on a job's existing segments to drop fragments."""
|
||||
@@ -375,8 +507,7 @@ def dub_abort(job_id: str):
|
||||
had_procs = bool(_active_procs.get(job_id))
|
||||
_kill_job_procs(job_id)
|
||||
try:
|
||||
if task_manager.cancel_task(job_id) is False:
|
||||
raise RuntimeError("task cancellation was declined")
|
||||
had_task = task_manager.cancel_task(job_id)
|
||||
except Exception as exc:
|
||||
logger.warning("Dub task cancellation failed")
|
||||
raise HTTPException(
|
||||
@@ -386,7 +517,13 @@ def dub_abort(job_id: str):
|
||||
job = _dub_jobs.get(job_id)
|
||||
if job is not None:
|
||||
job["aborted"] = True
|
||||
return {"aborted": True, "had_active_procs": had_procs}
|
||||
# Cancellation is idempotent: a missing active task means it already
|
||||
# stopped between the renderer aborting its stream and this request.
|
||||
return {
|
||||
"aborted": True,
|
||||
"had_active_procs": had_procs,
|
||||
"had_active_task": had_task,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/dub/history")
|
||||
@@ -615,8 +752,19 @@ async def dub_upload(
|
||||
os.makedirs(job_dir, exist_ok=True)
|
||||
|
||||
video_path = os.path.join(job_dir, f"original{ext}")
|
||||
with open(video_path, "wb") as f:
|
||||
f.write(await video.read())
|
||||
|
||||
def _stream_upload_to_disk() -> None:
|
||||
# UploadFile is already a spooled file. Copy it in bounded chunks on a
|
||||
# worker thread instead of materialising a multi-GB video in RAM and
|
||||
# blocking every API request while the event loop writes it.
|
||||
video.file.seek(0)
|
||||
with open(video_path, "wb") as output:
|
||||
shutil.copyfileobj(video.file, output, length=1024 * 1024)
|
||||
|
||||
try:
|
||||
await asyncio.to_thread(_stream_upload_to_disk)
|
||||
finally:
|
||||
await video.close()
|
||||
|
||||
filename = video.filename or f"video{ext}"
|
||||
task_id = f"prep_{job_id}"
|
||||
@@ -959,6 +1107,23 @@ async def dub_transcribe_stream(
|
||||
|
||||
job = _get_job(job_id)
|
||||
|
||||
# The durable job is written before the terminal SSE events below. If
|
||||
# the renderer, proxy, or backend connection drops in that narrow
|
||||
# window, reconnecting must replay the completed result instead of
|
||||
# running a second whole-file ASR pass. This is deliberately gated by
|
||||
# an explicit completion marker so partial work and imported subtitle
|
||||
# rows still take their established paths.
|
||||
if job and job.get("transcription_complete") and isinstance(job.get("segments"), list):
|
||||
yield _sse_event("final", {
|
||||
"segments": job["segments"],
|
||||
"source_lang": job.get("source_lang") or "en",
|
||||
"full_transcript": job.get("full_transcript") or "",
|
||||
"speaker_clones": job.get("cast_sources", {}),
|
||||
"cast_sources": job.get("cast_sources", {}),
|
||||
})
|
||||
yield _sse_event("done", {})
|
||||
return
|
||||
|
||||
preflight_error: Optional[str] = None
|
||||
# Extra machine-readable fields merged into the preflight `error` SSE event
|
||||
# (e.g. the typed asr_model_missing payload → download-CTA in the UI).
|
||||
@@ -1453,6 +1618,7 @@ async def dub_transcribe_stream(
|
||||
from services.model_manager import (
|
||||
DIARIZATION_ERR_LICENSE,
|
||||
DIARIZATION_ERR_NO_TOKEN,
|
||||
DIARIZATION_ERR_MISSING,
|
||||
)
|
||||
from core import error_docs_map
|
||||
|
||||
@@ -1545,7 +1711,23 @@ async def dub_transcribe_stream(
|
||||
from services import token_resolver
|
||||
resolved = token_resolver.resolve()
|
||||
|
||||
if err_sentinel == DIARIZATION_ERR_NO_TOKEN or not resolved:
|
||||
if err_sentinel == DIARIZATION_ERR_MISSING:
|
||||
from services.diarization_runtime import SORTFORMER, selected_backend
|
||||
native_selected = selected_backend() == SORTFORMER
|
||||
detail = (
|
||||
"Native Sortformer files are missing. Install audiocpp_cli beside "
|
||||
"the audio.cpp native bundle in Settings > Models > "
|
||||
"Diarisation, then retry transcription. "
|
||||
"Using silence gaps for now; rapid speaker turns may be merged."
|
||||
) if native_selected else (
|
||||
"Speaker diarization files are missing or incomplete. "
|
||||
"Install or repair pyannote in Settings > Models > Diarisation, "
|
||||
"then retry transcription. No models were downloaded during "
|
||||
"this job. Using silence gaps for now; rapid speaker turns "
|
||||
"may be merged."
|
||||
)
|
||||
error_class = "DIARIZATION_MODEL_MISSING"
|
||||
elif err_sentinel == DIARIZATION_ERR_NO_TOKEN:
|
||||
detail = (
|
||||
"Speaker diarization is disabled because no HuggingFace token "
|
||||
"was found in any source (Settings → API Keys, the HF_TOKEN "
|
||||
@@ -1558,12 +1740,12 @@ async def dub_transcribe_stream(
|
||||
)
|
||||
error_class = "HF_AUTH_FAILED"
|
||||
elif err_sentinel == DIARIZATION_ERR_LICENSE:
|
||||
who = resolved.username or "(whoami suppressed)"
|
||||
who = resolved.username if resolved else "(not signed in)"
|
||||
detail = (
|
||||
f"Speaker diarization model is gated — the "
|
||||
f"pyannote/speaker-diarization-3.1 license has not been "
|
||||
f"accepted on HuggingFace by this account "
|
||||
f"(source={resolved.source}, user={who}). Visit "
|
||||
f"(user={who}). Visit "
|
||||
f"huggingface.co/pyannote/speaker-diarization-3.1 AND "
|
||||
f"huggingface.co/pyannote/segmentation-3.0 while signed "
|
||||
f"in and click 'Agree and access repository' on both, "
|
||||
@@ -1575,17 +1757,13 @@ async def dub_transcribe_stream(
|
||||
else:
|
||||
# err_sentinel == DIARIZATION_ERR_LOAD (or unexpected None
|
||||
# with a resolved token — historical safety net).
|
||||
who = resolved.username or "(whoami suppressed)"
|
||||
detail = (
|
||||
f"Speaker diarization model failed to load even though an HF "
|
||||
f"token was found (source={resolved.source}, user={who}). "
|
||||
f"Most common causes: the pyannote/speaker-diarization-3.1 "
|
||||
f"license has not been accepted on HuggingFace, or there is "
|
||||
f"a pyannote/torch version mismatch. See backend logs for "
|
||||
f"The installed speaker diarization model failed to load. "
|
||||
f"See Settings > Logs > Backend for "
|
||||
f"the underlying error. Falling back to a silence-gap "
|
||||
f"heuristic; rapid speaker turns may be merged."
|
||||
)
|
||||
error_class = "PYANNOTE_LICENSE_REQUIRED"
|
||||
error_class = "DIARIZATION_LOAD_FAILED"
|
||||
warning = {
|
||||
"detail": detail + _hint_suffix(),
|
||||
"error_class": error_class,
|
||||
@@ -1606,7 +1784,13 @@ async def dub_transcribe_stream(
|
||||
# provided (#274). pyannote's apply() accepts num_speakers;
|
||||
# omit it entirely when None so we don't depend on the kwarg
|
||||
# existing in every pyannote build.
|
||||
if num_speakers:
|
||||
from services.diarization_native import NativeSortformer
|
||||
if isinstance(diar_pipe, NativeSortformer):
|
||||
diar = diar_pipe(
|
||||
asr_audio_target, num_speakers=num_speakers, job_id=job_id,
|
||||
cancel_check=lambda: bool(job.get("aborted")) or task_manager.is_cancelled(job_id),
|
||||
)
|
||||
elif num_speakers:
|
||||
logger.info("Diarizing with num_speakers=%d (user hint)", num_speakers)
|
||||
diar = diar_pipe(asr_audio_target, num_speakers=num_speakers)
|
||||
else:
|
||||
@@ -1632,7 +1816,7 @@ async def dub_transcribe_stream(
|
||||
len(asr_phrase_segments), separation,
|
||||
)
|
||||
return recovered_segments, None, "phrase_embeddings"
|
||||
return resplit, None, "pyannote"
|
||||
return resplit, None, "audiocpp-sortformer" if isinstance(diar_pipe, NativeSortformer) else "pyannote"
|
||||
except Exception as e:
|
||||
logger.exception("Diarization failed")
|
||||
# Inline ASR turns beat the silence-gap heuristic as a crash
|
||||
@@ -1681,6 +1865,9 @@ async def dub_transcribe_stream(
|
||||
final_segs, diar_warning, labels_source = done.pop().result()
|
||||
break
|
||||
yield _sse_event("ping", {})
|
||||
if job.get("aborted") or task_manager.is_cancelled(job_id):
|
||||
yield _sse_event("aborted", {})
|
||||
return
|
||||
if diar_warning:
|
||||
logger.warning("diarization fallback: %s", diar_warning.get("detail"))
|
||||
payload = {
|
||||
@@ -1695,6 +1882,8 @@ async def dub_transcribe_stream(
|
||||
payload["speaker_hint"] = diar_warning["speaker_hint"]
|
||||
yield _sse_event("warning", payload)
|
||||
|
||||
from services.segmentation import deduplicate_chunk_segments
|
||||
final_segs = deduplicate_chunk_segments(final_segs)
|
||||
job["segments"] = final_segs
|
||||
|
||||
# Auto-speaker-clone: sample each detected speaker's voice from the
|
||||
@@ -1845,6 +2034,7 @@ async def dub_transcribe_stream(
|
||||
detected_lang
|
||||
)
|
||||
job["full_transcript"] = " ".join(s.get("text", "") for s in final_segs)
|
||||
job["transcription_complete"] = True
|
||||
_save_job(job_id, job)
|
||||
|
||||
# Restore TTS model to GPU now that ASR is done. unload() blocks
|
||||
@@ -2106,6 +2296,8 @@ async def dub_transcribe(job_id: str, num_speakers: Optional[int] = None):
|
||||
raise
|
||||
if job.get("aborted"):
|
||||
raise HTTPException(status_code=499, detail="Transcription aborted")
|
||||
from services.segmentation import deduplicate_chunk_segments
|
||||
segments_result = deduplicate_chunk_segments(segments_result)
|
||||
job["segments"] = segments_result
|
||||
source_lang = job.get("source_lang")
|
||||
_save_job(job_id, job)
|
||||
|
||||
@@ -15,7 +15,7 @@ from core.http_headers import content_disposition
|
||||
from core.logging_utils import log_safe
|
||||
from core.path_security import UnsafePath, resolve_within
|
||||
from core.tasks import task_manager
|
||||
from fastapi import APIRouter, Header, HTTPException, Query, Response
|
||||
from fastapi import APIRouter, Header, HTTPException, Query, Request, Response
|
||||
from fastapi.responses import FileResponse, StreamingResponse
|
||||
from services.ffmpeg_utils import (
|
||||
bed_mix_filter,
|
||||
@@ -38,6 +38,31 @@ router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.api")
|
||||
|
||||
|
||||
async def _preserved_background(job: dict, job_id: str, lang: str, *, prepare: bool = True) -> str:
|
||||
"""All mixed preview/download paths share the same dialogue-only bed."""
|
||||
from services.dub_background import surgical_background
|
||||
|
||||
bed = _optional_dub_artifact(job.get("no_vocals_path"), job_id)
|
||||
source = _optional_dub_artifact(job.get("video_path"), job_id) or _optional_dub_artifact(job.get("audio_path"), job_id)
|
||||
if not bed or not source:
|
||||
raise HTTPException(status_code=409, detail={"code": "dub_background_unavailable", "message": "Original audio and background separation are required"})
|
||||
track = (job.get("dubbed_tracks") or {}).get(lang) or {}
|
||||
segments = track.get("source_segments") or job.get("segments") or []
|
||||
if not segments:
|
||||
raise HTTPException(status_code=409, detail={"code": "dub_background_unavailable", "message": "Dialogue timing is required"})
|
||||
if not prepare:
|
||||
return bed
|
||||
strategy = track.get("timing_strategy") or job.get("timing_strategy")
|
||||
plans = job.get("fit_plans" if strategy == "smart_fit" else "video_stretch_plans") or {}
|
||||
entry = (plans.get(lang) or {}) if strategy in {"smart_fit", "stretch_video"} else {}
|
||||
directory = os.path.join(_existing_job_dir_or_404(job_id), "exports")
|
||||
os.makedirs(directory, exist_ok=True)
|
||||
try:
|
||||
return await surgical_background(source, bed, directory, segments, entry.get("plan") or [], float(entry.get("orig_duration") or job.get("duration") or 0))
|
||||
except (ValueError, RuntimeError) as exc:
|
||||
raise HTTPException(status_code=409, detail={"code": "dub_background_unavailable", "message": str(exc)}) from exc
|
||||
|
||||
|
||||
def _unique_stamp() -> str:
|
||||
"""Return a short unique suffix like '20260415T142301-ab12cd34' for export files."""
|
||||
return f"{time.strftime('%Y%m%dT%H%M%S')}-{uuid.uuid4().hex[:8]}"
|
||||
@@ -596,7 +621,7 @@ def _build_audio_export_cmd(
|
||||
# Mix the dubbed voice over the original background bed (same weights
|
||||
# as the video mux path) so ambience/music is preserved.
|
||||
cmd += ["-i", bg_path, "-filter_complex",
|
||||
bed_mix_filter("1:a", "0:a"),
|
||||
bed_mix_filter("1:a", "0:a", bed_gain=1.0),
|
||||
"-map", "[aout]"]
|
||||
cmd += codec
|
||||
cmd.append(out_path)
|
||||
@@ -691,7 +716,7 @@ async def dub_download(
|
||||
else:
|
||||
output_name = f"dubbed_audio_{stamp}.m4a"
|
||||
out_path = os.path.join(exports_dir, output_name)
|
||||
bg = _optional_dub_artifact(job.get("no_vocals_path"), job_id) if preserve_bg else None
|
||||
bg = await _preserved_background(job, job_id, lang_code) if preserve_bg else None
|
||||
cmd = _build_audio_export_cmd(ffmpeg, track_info["path"], bg, out_path, fmt)
|
||||
try:
|
||||
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0)
|
||||
@@ -839,17 +864,16 @@ async def dub_download(
|
||||
retimed_idx = input_idx
|
||||
input_idx += 1
|
||||
|
||||
bg_audio = _optional_dub_artifact(job.get("no_vocals_path"), job_id) if preserve_bg else None
|
||||
bg_idx = None
|
||||
if bg_audio and filtered_tracks:
|
||||
cmd += ["-i", bg_audio]
|
||||
bg_idx = input_idx
|
||||
input_idx += 1
|
||||
|
||||
tracks_to_process = []
|
||||
for lang_code, track_info in filtered_tracks.items():
|
||||
if preserve_bg:
|
||||
bg_audio = await _preserved_background(job, job_id, lang_code)
|
||||
cmd += ["-i", bg_audio]
|
||||
bg_idx = input_idx
|
||||
input_idx += 1
|
||||
cmd += ["-i", track_info["path"]]
|
||||
tracks_to_process.append({"lang_code": lang_code, "idx": input_idx, "info": track_info})
|
||||
tracks_to_process.append({"lang_code": lang_code, "idx": input_idx, "bg_idx": bg_idx, "info": track_info})
|
||||
input_idx += 1
|
||||
|
||||
filter_parts: list[str] = []
|
||||
@@ -918,7 +942,7 @@ async def dub_download(
|
||||
for i, t in enumerate(tracks_to_process):
|
||||
tail = f",apad=whole_dur={apad_dur:.4f}" if apad_dur else ""
|
||||
filter_parts.append(bed_mix_filter(
|
||||
f"{bg_idx}:a", f"{t['idx']}:a", out=f"aout{i}", tail=tail, uniq=str(i),
|
||||
f"{t['bg_idx']}:a", f"{t['idx']}:a", out=f"aout{i}", tail=tail, uniq=str(i), bed_gain=1.0,
|
||||
))
|
||||
t["out_label"] = f"[aout{i}]"
|
||||
for t in tracks_to_process:
|
||||
@@ -1050,8 +1074,8 @@ _MEDIA_TYPES = {
|
||||
}
|
||||
|
||||
|
||||
@router.get("/dub/media/{job_id}")
|
||||
async def dub_get_media(job_id: str):
|
||||
@router.api_route("/dub/media/{job_id}", methods=["GET", "HEAD"])
|
||||
async def dub_get_media(job_id: str, request: Request):
|
||||
_job_dir_or_400(job_id)
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
@@ -1064,7 +1088,15 @@ async def dub_get_media(job_id: str):
|
||||
# silent black box. Default to video/mp4 because the ingest pipeline
|
||||
# remuxes URL downloads to mp4 (dub_pipeline.yt_download_sync).
|
||||
ext = os.path.splitext(video_path)[1].lower()
|
||||
return FileResponse(video_path, media_type=_MEDIA_TYPES.get(ext, "video/mp4"))
|
||||
media_type = _MEDIA_TYPES.get(ext, "video/mp4")
|
||||
headers = {
|
||||
"Cache-Control": "private, max-age=31536000, immutable",
|
||||
"Accept-Ranges": "bytes",
|
||||
}
|
||||
if request.method == "HEAD":
|
||||
headers["Content-Length"] = str(os.path.getsize(video_path))
|
||||
return Response(media_type=media_type, headers=headers)
|
||||
return FileResponse(video_path, media_type=media_type, headers=headers)
|
||||
|
||||
# One mux at a time per preview file. Without this, two overlapping requests
|
||||
# (e.g. the <video> element remounting right after a re-dub) both ran ffmpeg
|
||||
@@ -1081,8 +1113,9 @@ def _preview_lock(path: str) -> asyncio.Lock:
|
||||
return lock
|
||||
|
||||
|
||||
@router.get("/dub/preview-video/{job_id}")
|
||||
@router.api_route("/dub/preview-video/{job_id}", methods=["GET", "HEAD"])
|
||||
async def dub_preview_video(
|
||||
request: Request,
|
||||
job_id: str,
|
||||
lang: str = Query(..., description="Language code of the dubbed track to mux in"),
|
||||
preserve_bg: bool = Query(True),
|
||||
@@ -1110,7 +1143,7 @@ async def dub_preview_video(
|
||||
|
||||
video_path = _dub_artifact(job.get("video_path"), job_id, missing_detail="Source video missing")
|
||||
|
||||
bg_audio = _optional_dub_artifact(job.get("no_vocals_path"), job_id) if preserve_bg else None
|
||||
bg_audio = await _preserved_background(job, job_id, lang, prepare=request.method != "HEAD") if preserve_bg else None
|
||||
has_bg = bool(bg_audio)
|
||||
|
||||
# realpath-normalised + containment-checked inline BEFORE any filesystem
|
||||
@@ -1122,9 +1155,9 @@ async def dub_preview_video(
|
||||
if not exports_dir.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid job id")
|
||||
os.makedirs(exports_dir, exist_ok=True)
|
||||
bg_suffix = "bg" if (preserve_bg and has_bg) else "nobg"
|
||||
bg_suffix = "surgical_v2_" + Path(bg_audio).stem if (preserve_bg and has_bg) else "nobg"
|
||||
preview_path = os.path.realpath(
|
||||
os.path.join(exports_dir, f"preview_{lang}_{bg_suffix}.mp4")
|
||||
os.path.join(exports_dir, f"preview_v2_{lang}_{bg_suffix}.mp4")
|
||||
)
|
||||
if not preview_path.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid path")
|
||||
@@ -1138,6 +1171,18 @@ async def dub_preview_video(
|
||||
and os.path.getmtime(preview_path) >= track_mtime
|
||||
)
|
||||
|
||||
# Vidstack probes extensionless routes with HEAD before choosing a native
|
||||
# provider. Confirm that this preview is valid without starting an ffmpeg
|
||||
# mux; the following GET builds it lazily when needed.
|
||||
if request.method == "HEAD":
|
||||
headers = {
|
||||
"Cache-Control": "private, max-age=31536000, immutable",
|
||||
"Accept-Ranges": "bytes",
|
||||
}
|
||||
if _cache_ok():
|
||||
headers["Content-Length"] = str(os.path.getsize(preview_path))
|
||||
return Response(media_type="video/mp4", headers=headers)
|
||||
|
||||
async def _mux_preview():
|
||||
# Mux into a temp file and os.replace() into place so a concurrent
|
||||
# reader never sees a partially-written preview (#281: video stuck
|
||||
@@ -1249,7 +1294,7 @@ async def dub_preview_video(
|
||||
audio_map = f"{track_idx}:a:0"
|
||||
if bg_idx is not None:
|
||||
tail = f",apad=whole_dur={apad_dur:.4f}" if apad_dur else ""
|
||||
filter_parts.append(bed_mix_filter(f"{bg_idx}:a", f"{track_idx}:a", tail=tail))
|
||||
filter_parts.append(bed_mix_filter(f"{bg_idx}:a", f"{track_idx}:a", tail=tail, bed_gain=1.0))
|
||||
audio_map = "[aout]"
|
||||
elif apad_dur:
|
||||
filter_parts.append(f"[{track_idx}:a]apad=whole_dur={apad_dur:.4f}[aout]")
|
||||
@@ -1266,7 +1311,7 @@ async def dub_preview_video(
|
||||
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
|
||||
else:
|
||||
cmd += ["-c:v", "copy"]
|
||||
cmd += ["-c:a", "aac", "-b:a", "192k"]
|
||||
cmd += ["-c:a", "aac", "-b:a", "192k", "-movflags", "+faststart"]
|
||||
# `-shortest` would cut the retimed video at the (slightly different)
|
||||
# audio length and lose the trailing frame; only use it on the copy path.
|
||||
if not stretch_entry and retime_decision is None:
|
||||
@@ -1311,22 +1356,37 @@ async def dub_preview_video(
|
||||
if not _cache_ok():
|
||||
await _mux_preview()
|
||||
|
||||
# no-store: the URL is stable across re-dubs, so any HTTP-level caching
|
||||
# in the WebView would keep showing the previous dub after a re-generate
|
||||
# (#281: "edits don't change the result").
|
||||
# The renderer includes the segment-fingerprint revision in the URL, so a
|
||||
# regenerated track gets a fresh cache key. Keep each completed preview:
|
||||
# switching Original/Dub then reuses local ranges instead of re-reading a
|
||||
# multi-hundred-megabyte MP4 from the backend.
|
||||
return FileResponse(
|
||||
preview_path,
|
||||
media_type="video/mp4",
|
||||
headers={"Cache-Control": "no-store"},
|
||||
headers={"Cache-Control": "private, max-age=31536000, immutable", "Accept-Ranges": "bytes"},
|
||||
)
|
||||
|
||||
|
||||
def _compute_onsets_sync(src_path: str) -> list[float]:
|
||||
def _compute_timeline_sync(src_path: str) -> tuple[list[float], list[float]]:
|
||||
"""Blocking part of onset analysis — runs in a worker thread."""
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
from services.onset_align import detect_speech_onsets
|
||||
audio, sr = sf.read(src_path, dtype="float32")
|
||||
return detect_speech_onsets(audio, sr)
|
||||
onsets = detect_speech_onsets(audio, sr)
|
||||
mono = np.asarray(audio, dtype=np.float32)
|
||||
if mono.ndim > 1:
|
||||
mono = mono.mean(axis=1)
|
||||
mono = mono.reshape(-1)
|
||||
if mono.size == 0:
|
||||
return onsets, []
|
||||
bucket_count = min(2048, int(mono.size))
|
||||
bucket_width = max(1, (int(mono.size) + bucket_count - 1) // bucket_count)
|
||||
padded_size = bucket_count * bucket_width
|
||||
if padded_size != mono.size:
|
||||
mono = np.pad(mono, (0, padded_size - int(mono.size)))
|
||||
peaks = np.max(np.abs(mono.reshape(bucket_count, bucket_width)), axis=1)
|
||||
return onsets, [round(float(value), 5) for value in peaks]
|
||||
|
||||
|
||||
@router.get("/dub/onsets/{job_id}")
|
||||
@@ -1365,20 +1425,24 @@ async def dub_get_onsets(job_id: str):
|
||||
):
|
||||
with open(cache_path, "r", encoding="utf-8") as f:
|
||||
cached = json.load(f)
|
||||
if isinstance(cached, dict) and isinstance(cached.get("onsets"), list):
|
||||
if (
|
||||
isinstance(cached, dict)
|
||||
and isinstance(cached.get("onsets"), list)
|
||||
and isinstance(cached.get("peaks"), list)
|
||||
):
|
||||
return cached
|
||||
except (OSError, ValueError):
|
||||
pass # unreadable/corrupt cache → recompute below
|
||||
|
||||
try:
|
||||
onsets = await asyncio.to_thread(_compute_onsets_sync, src_path)
|
||||
onsets, peaks = await asyncio.to_thread(_compute_timeline_sync, src_path)
|
||||
except Exception as e:
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"Onset analysis failed: {str(e)[:200]}",
|
||||
)
|
||||
|
||||
payload = {"onsets": onsets, "source": source}
|
||||
payload = {"onsets": onsets, "peaks": peaks, "source": source}
|
||||
try:
|
||||
os.makedirs(os.path.dirname(cache_path), exist_ok=True)
|
||||
tmp_path = cache_path + ".tmp"
|
||||
@@ -1621,13 +1685,13 @@ async def dub_download_audio(
|
||||
exports_dir = os.path.join(job_dir, "exports")
|
||||
os.makedirs(exports_dir, exist_ok=True)
|
||||
|
||||
bg_audio = _optional_dub_artifact(job.get("no_vocals_path"), job_id) if preserve_bg else None
|
||||
bg_audio = await _preserved_background(job, job_id, lang_label) if preserve_bg else None
|
||||
if bg_audio:
|
||||
ffmpeg = find_ffmpeg()
|
||||
final_audio_path = os.path.join(exports_dir, f"mixed_dub_{stamp}.wav")
|
||||
cmd = [
|
||||
ffmpeg, "-i", bg_audio, "-i", wav_path,
|
||||
"-filter_complex", bed_mix_filter("0:a", "1:a"),
|
||||
"-filter_complex", bed_mix_filter("0:a", "1:a", bed_gain=1.0),
|
||||
"-map", "[aout]", "-c:a", "pcm_s16le", "-y", final_audio_path
|
||||
]
|
||||
try:
|
||||
@@ -1638,8 +1702,9 @@ async def dub_download_audio(
|
||||
raise Exception("ffmpeg mix produced no output file")
|
||||
wav_path = final_audio_path
|
||||
logger.info("Dub audio mix completed")
|
||||
except Exception:
|
||||
except Exception as exc:
|
||||
logger.exception("Failed to mix audio")
|
||||
raise HTTPException(status_code=500, detail={"code": "dub_background_unavailable", "message": "Could not preserve background audio"}) from exc
|
||||
|
||||
base_name = os.path.splitext(job.get('filename', 'audio'))[0]
|
||||
safe_name = ''.join(c for c in base_name if c.isalnum() or c in '-_ ').strip() or 'audio'
|
||||
@@ -1907,20 +1972,23 @@ async def dub_download_mp3(
|
||||
os.makedirs(exports_dir, exist_ok=True)
|
||||
|
||||
source_path = wav_path
|
||||
bg_audio = _optional_dub_artifact(job.get("no_vocals_path"), job_id) if preserve_bg else None
|
||||
bg_audio = await _preserved_background(job, job_id, lang_label) if preserve_bg else None
|
||||
if bg_audio:
|
||||
mixed_path = os.path.join(exports_dir, f"mixed_mp3_{stamp}.wav")
|
||||
cmd_mix = [
|
||||
ffmpeg, "-i", bg_audio, "-i", wav_path,
|
||||
"-filter_complex", bed_mix_filter("0:a", "1:a"),
|
||||
"-filter_complex", bed_mix_filter("0:a", "1:a", bed_gain=1.0),
|
||||
"-map", "[aout]", "-c:a", "pcm_s16le", "-y", mixed_path
|
||||
]
|
||||
try:
|
||||
rc, _, _ = await run_ffmpeg(cmd_mix, timeout=900.0)
|
||||
if rc == 0 and os.path.exists(mixed_path) and os.path.getsize(mixed_path) > 0:
|
||||
source_path = mixed_path
|
||||
except Exception:
|
||||
else:
|
||||
raise RuntimeError("Background mixing failed")
|
||||
except Exception as exc:
|
||||
logger.exception("Failed to mix audio for MP3")
|
||||
raise HTTPException(status_code=500, detail={"code": "dub_background_unavailable", "message": "Could not preserve background audio"}) from exc
|
||||
|
||||
mp3_path = os.path.join(exports_dir, f"dubbed_{stamp}.mp3")
|
||||
# Accept '128', '192k' etc. — normalize to ffmpeg's 'Nk' form and clamp
|
||||
|
||||
+409
-180
@@ -5,8 +5,6 @@ import struct
|
||||
import logging
|
||||
import time
|
||||
import asyncio
|
||||
import shutil
|
||||
import zipfile
|
||||
import torch
|
||||
import torchaudio
|
||||
from fastapi import APIRouter, HTTPException
|
||||
@@ -16,7 +14,8 @@ from core.config import DUB_DIR, VOICES_DIR, dub_seg_path
|
||||
from core.tasks import task_manager
|
||||
from schemas.requests import DubRequest
|
||||
from services.model_manager import _gpu_pool, run_on_gpu_pool_guarded
|
||||
from services.tts_backend import resolve_generation_backend, active_backend_id
|
||||
from services.tts_backend import TTSBackend, resolve_generation_backend, active_backend_id
|
||||
from services.dub_batching import batch_timeout_s, native_batch_width
|
||||
from services import gpu_gateway
|
||||
from services.audio_dsp import apply_mastering, normalize_audio, apply_effects_chain, get_effect_chain
|
||||
from services.audio_io import atomic_save_wav, _safe_torchaudio_save
|
||||
@@ -31,22 +30,29 @@ from services.ffmpeg_utils import (
|
||||
)
|
||||
from services.rvc import apply_rvc, is_enabled as rvc_is_enabled
|
||||
from services.incremental import segment_fingerprint, fit_fingerprint
|
||||
from services.fit_planner import UNDERRUN_TOLERANCE, FitParams, plan_fit
|
||||
from services.fit_planner import FitParams, plan_fit
|
||||
from services.watermark import mark_synthetic
|
||||
from services.speaker_clone import auto_profile_id
|
||||
from services.segment_bundle import extract_segment_wavs
|
||||
from api.routers.dub_core import _get_job, _save_job
|
||||
from omnivoice.utils.voice_design import heal_design_instruct
|
||||
|
||||
logger = logging.getLogger("omnivoice.dub")
|
||||
|
||||
# Maximum compression ratio we'll attempt with pitch-preserving stretch
|
||||
# before declaring "no way to fit cleanly" and falling back. atempo
|
||||
# remains intelligible up to ~1.5× then introduces audible WSOLA
|
||||
# artefacts; above ~1.8× speech becomes a fast garbled stream that no
|
||||
# DSP can rescue. The contributing-factor pipeline (CPS-aware slot-fit
|
||||
# in services/speech_rate.py, gap absorption below) keeps us under this
|
||||
# in practice — this is only a guard rail.
|
||||
MAX_STRETCH_RATIO = 1.8
|
||||
class _RemoteDubBackend:
|
||||
"""Sample-rate carrier while Dubbing runs without local TTS weights."""
|
||||
|
||||
sample_rate = 24_000
|
||||
|
||||
|
||||
async def _resolve_dub_execution():
|
||||
"""Resolve routing without loading local weights for a remote dub."""
|
||||
engine_id = active_backend_id()
|
||||
decision = gpu_gateway.decide("dub_segments")
|
||||
if decision.remote:
|
||||
await gpu_gateway.preflight(engine_id, decision, operation="dub_segments")
|
||||
return engine_id, decision, _RemoteDubBackend()
|
||||
return engine_id, decision, await resolve_generation_backend(require_cloning=True)
|
||||
|
||||
|
||||
def _prepare_oom_retry(error: Exception, *, execution_target: str) -> bool:
|
||||
@@ -362,6 +368,12 @@ def forget_missing_ref_warnings(job_id: str) -> None:
|
||||
_MISSING_REF_WARNED.pop(str(job_id), None)
|
||||
|
||||
|
||||
|
||||
def _ref_within_limit(info) -> bool:
|
||||
from services.speaker_clone import MAX_REF_DURATION_S
|
||||
return bool(info) and float(info.get("duration") or 0.0) <= MAX_REF_DURATION_S
|
||||
|
||||
|
||||
def resolve_consistent_ref(job: dict, speaker_key: str, memo: dict | None = None):
|
||||
"""ONE clone reference for every segment of `speaker_key`.
|
||||
|
||||
@@ -370,8 +382,9 @@ def resolve_consistent_ref(job: dict, speaker_key: str, memo: dict | None = None
|
||||
the per-line path uses as its fallback;
|
||||
2. no speaker clone (heuristic diarization skips extraction entirely —
|
||||
the key case): a deterministic pick among that speaker's per-segment
|
||||
clips: longest clip ≥3 s, tie-break lowest segment id. Clips all
|
||||
shorter than 3 s degrade to "longest overall", same tie-break.
|
||||
clips: longest usable clip ≥3 s within the shared reference limit,
|
||||
tie-break lowest segment id. Short clips fall back to longest usable.
|
||||
Oversized references must never strand every short line for a speaker.
|
||||
|
||||
Returns the clone info dict ({"ref_audio", "ref_text", ...}) or None.
|
||||
Pure function of the job dict; `memo` (keyed by speaker_key) just avoids
|
||||
@@ -380,7 +393,11 @@ def resolve_consistent_ref(job: dict, speaker_key: str, memo: dict | None = None
|
||||
if memo is not None and speaker_key in memo:
|
||||
return memo[speaker_key]
|
||||
|
||||
from services.speaker_clone import MAX_REF_DURATION_S
|
||||
|
||||
ref = _find_speaker_clone(job.get("speaker_clones") or {}, speaker_key)
|
||||
if ref and float(ref.get("duration") or 0.0) > MAX_REF_DURATION_S:
|
||||
ref = None
|
||||
if ref is None:
|
||||
seg_clones = job.get("segment_clones") or {}
|
||||
candidates = []
|
||||
@@ -392,7 +409,8 @@ def resolve_consistent_ref(job: dict, speaker_key: str, memo: dict | None = None
|
||||
continue
|
||||
sid = str(row.get("id", ""))
|
||||
info = seg_clones.get(sid)
|
||||
if info and info.get("ref_audio"):
|
||||
if (info and info.get("ref_audio")
|
||||
and float(info.get("duration") or 0.0) <= MAX_REF_DURATION_S):
|
||||
candidates.append((sid, info))
|
||||
if candidates:
|
||||
usable = [
|
||||
@@ -436,6 +454,9 @@ def _remote_voice(job: dict, profile_id: str | None, seg_id, voice_match: str,
|
||||
info = ((job.get("segment_clones") or {}).get(str(seg_id))
|
||||
or _find_speaker_clone(job.get("speaker_clones") or {}, key))
|
||||
single_use = str(seg_id) in (job.get("segment_clones") or {})
|
||||
if not _ref_within_limit(info):
|
||||
info = resolve_consistent_ref(job, key, memo)
|
||||
single_use = False
|
||||
if info:
|
||||
ref_audio, ref_text = info.get("ref_audio"), info.get("ref_text")
|
||||
elif profile_id:
|
||||
@@ -461,21 +482,12 @@ def _remote_voice(job: dict, profile_id: str | None, seg_id, voice_match: str,
|
||||
def _decode_remote_dub(result: gpu_gateway.RemoteResult) -> dict[int, str]:
|
||||
"""Extract the worker bundle into a task-scoped directory, path-safely."""
|
||||
target = os.path.join(DUB_DIR, ".remote", result.task_id)
|
||||
os.makedirs(target, exist_ok=True)
|
||||
paths: dict[int, str] = {}
|
||||
with zipfile.ZipFile(result.path) as archive:
|
||||
for member in archive.infolist():
|
||||
match = re.fullmatch(r"segments/(\d+)\.wav", member.filename)
|
||||
if not match:
|
||||
raise ValueError(f"unexpected dub artifact member: {member.filename}")
|
||||
index = int(match.group(1))
|
||||
destination = os.path.join(target, f"{index}.wav")
|
||||
partial = f"{destination}.part"
|
||||
with archive.open(member) as source, open(partial, "wb") as output:
|
||||
shutil.copyfileobj(source, output)
|
||||
os.replace(partial, destination)
|
||||
paths[index] = destination
|
||||
return paths
|
||||
try:
|
||||
return extract_segment_wavs(result.path or "", target)
|
||||
except ValueError as exc:
|
||||
# Preserve the established route-specific error wording consumed by
|
||||
# diagnostics and regression tests.
|
||||
raise ValueError(str(exc).replace("segment artifact", "dub artifact")) from exc
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
@@ -491,16 +503,25 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
)
|
||||
|
||||
# ── Engine resolution (issue #312 class) ────────────────────────────────
|
||||
# Dub used to hardcode VoiceStudio via get_model() regardless of the engine
|
||||
# selected in Model Catalogue — a SILENT fallback. Every real dub
|
||||
# segment's ref_audio resolves to either an auto:<speaker>/auto-seg:<id>
|
||||
# clone cut from the source video or a saved voice-profile row (see
|
||||
# `_gen` below), so require_cloning=True: an engine that can't clone
|
||||
# would either mis-clone per segment or fail deep into the job. Checked
|
||||
# ONCE here, before the streaming task starts, so a doomed job fails fast
|
||||
# with one clear message instead of N per-segment ones.
|
||||
# Every rendered segment clones either source speech or a saved profile, so
|
||||
# local execution still requires a cloning-capable engine. Remote execution
|
||||
# validates the selected worker here without loading duplicate local weights;
|
||||
# its local backend is prepared only if gateway fallback actually selects it.
|
||||
try:
|
||||
backend = await resolve_generation_backend(require_cloning=True)
|
||||
engine_id, decision, backend = await _resolve_dub_execution()
|
||||
except gpu_gateway.ModelNotDownloaded as e:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
"error": "model_not_downloaded",
|
||||
"message": str(e),
|
||||
"engine": e.engine,
|
||||
"repo_ids": e.repo_ids,
|
||||
"target": e.target,
|
||||
"target_label": e.target_label,
|
||||
"downloadable": e.downloadable,
|
||||
},
|
||||
) from e
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
@@ -516,6 +537,14 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
)
|
||||
raise HTTPException(status_code=503, detail=payload["detail"]) from e
|
||||
|
||||
# Resolve the global profile once for this job. Explicit Production
|
||||
# overrides remain authoritative, while ordinary Dubbing now follows the
|
||||
# same Fast/Balanced/Quality/Max contract as Clone and long-form work.
|
||||
from services.performance_profiles import tts_defaults
|
||||
_profile_defaults = tts_defaults(engine_id)
|
||||
_job_num_step = req.num_step if req.num_step is not None else _profile_defaults.get("num_step", 16)
|
||||
_job_postprocess = _profile_defaults.get("postprocess_output", True)
|
||||
|
||||
async def _stream(task_id):
|
||||
total = len(req.segments)
|
||||
all_segment_wavs = []
|
||||
@@ -677,8 +706,29 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
_wav_kind = (
|
||||
_kind_map.get(lang_code) if isinstance(_kind_map, dict) else job.get("seg_wav_kind")
|
||||
)
|
||||
if strategy != "strict_slot" and regen_only is not None and _wav_kind != "natural":
|
||||
if regen_only is not None and _wav_kind != "natural":
|
||||
regen_only = None
|
||||
# A partial rerun must repair absent or corrupt caches, including before
|
||||
# remote batching decides which lines need synthesis.
|
||||
if regen_only is not None:
|
||||
for index, segment in enumerate(req.segments):
|
||||
sid = seg_ids[index] if index < len(seg_ids) else f"seg_{index}"
|
||||
if sid in regen_only or not segment.text.strip():
|
||||
continue
|
||||
cache = _seg_lang_path(sid)
|
||||
if not os.path.exists(cache) and _legacy_seg_cache_ok(job, lang_code):
|
||||
for key in (sid, index):
|
||||
legacy = dub_seg_path(job_id, key)
|
||||
if os.path.exists(legacy):
|
||||
cache = legacy
|
||||
break
|
||||
try:
|
||||
info = torchaudio.info(cache)
|
||||
intact = info.num_frames > 0 and _cached_payload_intact(cache, info)
|
||||
except Exception:
|
||||
intact = False
|
||||
if not intact:
|
||||
regen_only.add(sid)
|
||||
# Manifest: stable segment id per current index. Per-segment WAVs are
|
||||
# named by stable id (dub_seg_path) so regen reuses the right audio after
|
||||
# reorder; index-keyed readers (preview/export) resolve via this manifest.
|
||||
@@ -701,11 +751,70 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
_t_start = time.perf_counter()
|
||||
_t_cache = 0.0
|
||||
_t_tts = 0.0
|
||||
_batched_audio: dict[int, torch.Tensor] = {}
|
||||
_profile_row_cache: dict[str, object | None] = {}
|
||||
_has_native_batch = (
|
||||
getattr(type(backend), "generate_batch", TTSBackend.generate_batch)
|
||||
is not TTSBackend.generate_batch
|
||||
)
|
||||
_native_batch_width = native_batch_width(backend) if _has_native_batch else 1
|
||||
|
||||
async def _prepare_local_dub():
|
||||
"""Load local TTS only when gateway fallback actually needs it."""
|
||||
nonlocal backend, _has_native_batch, _native_batch_width
|
||||
if isinstance(backend, _RemoteDubBackend):
|
||||
backend = await resolve_generation_backend(require_cloning=True)
|
||||
_has_native_batch = (
|
||||
getattr(type(backend), "generate_batch", TTSBackend.generate_batch)
|
||||
is not TTSBackend.generate_batch
|
||||
)
|
||||
_native_batch_width = (
|
||||
native_batch_width(backend) if _has_native_batch else 1
|
||||
)
|
||||
return gpu_gateway.LocalCall(fn=lambda: {})
|
||||
|
||||
def _segment_generation_args(index, segment) -> dict:
|
||||
"""Resolve the per-row controls shared by serial and native batches."""
|
||||
current_id = seg_ids[index] if index < len(seg_ids) else f"seg_{index}"
|
||||
duration = segment.end - segment.start
|
||||
profile_id = segment.profile_id or None
|
||||
speed = segment.speed if segment.speed is not None else req.speed
|
||||
language = segment.target_lang or req.language
|
||||
instruct = segment.instruct or req.instruct
|
||||
direction_text = getattr(segment, "direction", None)
|
||||
if direction_text and direction_text.strip():
|
||||
try:
|
||||
from services.director import parse as _parse_direction
|
||||
|
||||
direction = _parse_direction(direction_text)
|
||||
extra = direction.instruct_prompt()
|
||||
if extra:
|
||||
instruct = f"{instruct}, {extra}" if instruct else extra
|
||||
bias = direction.rate_bias()
|
||||
if (
|
||||
bias
|
||||
and abs(bias - 1.0) > 0.01
|
||||
and strategy == "strict_slot"
|
||||
):
|
||||
speed = (speed or 1.0) * bias
|
||||
except Exception as error:
|
||||
logger.debug("direction parse skipped for %s: %s", current_id, error)
|
||||
return {
|
||||
"seg_id": current_id,
|
||||
"text": segment.text,
|
||||
"language": language,
|
||||
"instruct": instruct,
|
||||
"duration": duration if strategy == "strict_slot" else None,
|
||||
"num_step": 8 if req.preview else _job_num_step,
|
||||
"guidance_scale": req.guidance_scale,
|
||||
"speed": speed,
|
||||
"profile_id": profile_id,
|
||||
"effect_preset": getattr(segment, "effect_preset", None) or "broadcast",
|
||||
}
|
||||
|
||||
# One coarse remote lease for every segment that actually needs fresh
|
||||
# synthesis. Assembly, fitting and the separately-pooled RVC pass stay
|
||||
# here; the worker returns a single verified bundle of segment WAVs.
|
||||
decision = gpu_gateway.decide("dub_segments")
|
||||
if decision.remote:
|
||||
remote_rows: list[dict] = []
|
||||
remote_refs: list[str | None] = []
|
||||
@@ -740,7 +849,8 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
"ref_text": ref_text, "ref_single_use": ref_single_use,
|
||||
"instruct": seg_instruct,
|
||||
"duration": (seg.end - seg.start) if strategy == "strict_slot" else None,
|
||||
"num_step": 8 if req.preview else req.num_step,
|
||||
"num_step": 8 if req.preview else _job_num_step,
|
||||
"postprocess_output": _job_postprocess,
|
||||
"guidance_scale": req.guidance_scale, "speed": seg_speed,
|
||||
"effect_preset": seg.effect_preset or "broadcast",
|
||||
"seed": seed,
|
||||
@@ -752,13 +862,13 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
if remote_rows:
|
||||
states: asyncio.Queue = asyncio.Queue()
|
||||
call = gpu_gateway.RemoteCall(
|
||||
engine=active_backend_id(), operation="dub_segments",
|
||||
engine=engine_id, operation="dub_segments",
|
||||
params={"segments": remote_rows, "ref_audio": remote_refs},
|
||||
decode=_decode_remote_dub,
|
||||
)
|
||||
dub_run = gpu_gateway.JobRun("dub_segments")
|
||||
run = asyncio.create_task(gpu_gateway.run(
|
||||
"dub_segments", local=gpu_gateway.LocalCall(fn=lambda: {}),
|
||||
"dub_segments", local=gpu_gateway.LocalCall(prepare=_prepare_local_dub),
|
||||
remote=call, decision=decision, job=dub_run,
|
||||
on_state=states.put_nowait,
|
||||
))
|
||||
@@ -777,11 +887,34 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
continue
|
||||
fraction = float(state.get("progress") or 0.0)
|
||||
yield f"data: {json.dumps({'type': 'progress', 'current': round(fraction * total, 2), 'total': total, 'text': state.get('stage') or state.get('phase')})}\n\n"
|
||||
remote_audio = await run
|
||||
try:
|
||||
remote_audio = await run
|
||||
except Exception as error:
|
||||
from core.public_errors import stream_generation_failure
|
||||
|
||||
detail = stream_generation_failure(error)["detail"]
|
||||
yield f"data: {json.dumps({'type': 'error', 'error': detail})}\n\n"
|
||||
return
|
||||
notice = dub_run.notice()
|
||||
if notice is not None:
|
||||
yield f"data: {json.dumps({'type': 'routing_notice', 'status': notice[0], 'reason': notice[1]})}\n\n"
|
||||
|
||||
if remote_audio and isinstance(backend, _RemoteDubBackend):
|
||||
first_remote = next(iter(remote_audio.values()))
|
||||
backend.sample_rate = int(torchaudio.info(first_remote).sample_rate)
|
||||
elif isinstance(backend, _RemoteDubBackend):
|
||||
# Fit-only / cache-only reruns synthesize nothing. Keep the cached
|
||||
# track's native rate when one exists instead of resampling it to
|
||||
# the carrier's conservative 24 kHz default.
|
||||
for cached_id in seg_ids:
|
||||
cached_path = _seg_lang_path(cached_id)
|
||||
if os.path.exists(cached_path):
|
||||
try:
|
||||
backend.sample_rate = int(torchaudio.info(cached_path).sample_rate)
|
||||
break
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
for i, seg in enumerate(req.segments):
|
||||
seg_id = seg_ids[i] if i < len(seg_ids) else f"seg_{i}"
|
||||
|
||||
@@ -848,7 +981,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
all_segment_wavs.append(
|
||||
(seg.start, seg.end, seg_wav_path, backend.sample_rate)
|
||||
)
|
||||
sync_scores.append(getattr(seg, 'sync_ratio', None) or 1.0)
|
||||
sync_scores.append(round(cached_info.num_frames / cached_info.sample_rate / max(seg_duration, 0.01), 3))
|
||||
_t_cache += time.perf_counter() - _t_cache_0
|
||||
continue
|
||||
|
||||
@@ -856,48 +989,30 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
if cached_sr != backend.sample_rate:
|
||||
import torchaudio.functional as AF
|
||||
cached_wav = AF.resample(cached_wav, cached_sr, backend.sample_rate)
|
||||
# strict_slot persists slot-sized buffers. Every other
|
||||
# strategy consumes natural-rate audio and lets the mix
|
||||
# loop fit it to the current timeline.
|
||||
if strategy == "strict_slot":
|
||||
target_samples = int(seg_duration * backend.sample_rate)
|
||||
current_samples = cached_wav.shape[-1]
|
||||
if target_samples > current_samples:
|
||||
cached_wav = torch.nn.functional.pad(cached_wav, (0, target_samples - current_samples))
|
||||
elif current_samples > target_samples:
|
||||
cached_wav = cached_wav[..., :target_samples]
|
||||
cached_ratio = round(cached_wav.shape[-1] / backend.sample_rate / max(seg_duration, 0.01), 3)
|
||||
all_segment_wavs.append(_store_mix_wav(seg.start, seg.end, cached_wav, backend.sample_rate, f"mix_{seg_id}"))
|
||||
try:
|
||||
del cached_wav
|
||||
except Exception:
|
||||
pass
|
||||
_release_audio_tensors()
|
||||
sync_scores.append(getattr(seg, 'sync_ratio', None) or 1.0)
|
||||
sync_scores.append(cached_ratio)
|
||||
_t_cache += time.perf_counter() - _t_cache_0
|
||||
continue
|
||||
except Exception as e:
|
||||
# Fall through to a silent placeholder if the cached WAV
|
||||
# is broken — cleaner than aborting the whole mix.
|
||||
yield f"data: {json.dumps({'type': 'warning', 'segment': i, 'message': f'cached seg lost, padding silence: {str(e)[:120]}'})}\n\n"
|
||||
sr = backend.sample_rate
|
||||
silence = torch.zeros(1, max(0, int(seg_duration * sr)))
|
||||
all_segment_wavs.append(_store_mix_wav(seg.start, seg.end, silence, sr, f"mix_{seg_id}"))
|
||||
try:
|
||||
del silence
|
||||
except Exception:
|
||||
pass
|
||||
_release_audio_tensors()
|
||||
sync_scores.append(1.0)
|
||||
continue
|
||||
except Exception:
|
||||
logger.exception("Dub cached segment could not be read: %s", seg_id)
|
||||
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'segment_id': seg_id, 'error_code': 'dub_speech_missing', 'error': 'Cached speech could not be read. Regenerate this segment before exporting.'})}\n\n"
|
||||
return
|
||||
|
||||
def _gen(text, lang, instruct_str, dur_s, nstep, cfg, spd, profile_id, effect_preset,
|
||||
*, execution_target="local"):
|
||||
*, execution_target="local", prepare_only=False, current_seg_id=None):
|
||||
# Normalize once at the segment's text→engine choke point
|
||||
# (covers the OOM-retry generate below too, which reuses this
|
||||
# closure's `text`). Pref-gated, idempotent, never raises.
|
||||
from services.text_normalization import normalize_for_tts
|
||||
text = normalize_for_tts(text, lang)
|
||||
|
||||
effective_seg_id = seg_id if current_seg_id is None else current_seg_id
|
||||
ref_audio = None
|
||||
ref_text = None
|
||||
used_seed = None
|
||||
@@ -928,7 +1043,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
# CROSS binding (sid != this segment) can only come from an
|
||||
# explicit request — honour its clip unchanged.
|
||||
_consistent_alt = None
|
||||
if voice_match == "consistent" and sid == str(seg_id):
|
||||
if voice_match == "consistent" and sid == str(effective_seg_id):
|
||||
_spk_key = _speaker_key_for_segment(job, sid)
|
||||
if _spk_key:
|
||||
_consistent_alt = resolve_consistent_ref(
|
||||
@@ -969,7 +1084,9 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
# editor's Voice dropdown can actually render ("From
|
||||
# Video → Speaker N"). `seg_id` is closed over from
|
||||
# the per-segment loop below.
|
||||
segment_speaker_key = _speaker_key_for_segment(job, seg_id)
|
||||
segment_speaker_key = _speaker_key_for_segment(
|
||||
job, effective_seg_id
|
||||
)
|
||||
# Legacy jobs may not persist diarized segment rows.
|
||||
# Preserve their established per-line preference; only
|
||||
# suppress it when current metadata proves the user
|
||||
@@ -978,11 +1095,11 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
segment_speaker_key is None or segment_speaker_key == key
|
||||
)
|
||||
seg_ref = (
|
||||
(job.get("segment_clones") or {}).get(str(seg_id))
|
||||
(job.get("segment_clones") or {}).get(str(effective_seg_id))
|
||||
if selected_is_segment_speaker
|
||||
else None
|
||||
)
|
||||
if seg_ref:
|
||||
if _ref_within_limit(seg_ref):
|
||||
ref_audio = seg_ref.get("ref_audio")
|
||||
ref_text = seg_ref.get("ref_text")
|
||||
ref_single_use = True
|
||||
@@ -990,7 +1107,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
auto = _find_speaker_clone(
|
||||
job.get("speaker_clones") or {}, key
|
||||
)
|
||||
if auto is None:
|
||||
if not _ref_within_limit(auto):
|
||||
# Short lines may have no line-specific clip.
|
||||
# Reuse this speaker's best source instead of
|
||||
# silently reverting to the engine default.
|
||||
@@ -1003,8 +1120,13 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
profile_id = None # prevent the voice_profiles lookup below
|
||||
|
||||
if profile_id:
|
||||
with db_conn() as conn:
|
||||
row = conn.execute("SELECT * FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
if profile_id not in _profile_row_cache:
|
||||
with db_conn() as conn:
|
||||
_profile_row_cache[profile_id] = conn.execute(
|
||||
"SELECT * FROM voice_profiles WHERE id=?",
|
||||
(profile_id,),
|
||||
).fetchone()
|
||||
row = _profile_row_cache[profile_id]
|
||||
if row:
|
||||
if row["is_locked"] and row["locked_audio_path"]:
|
||||
ref_audio = os.path.join(VOICES_DIR, row["locked_audio_path"])
|
||||
@@ -1024,15 +1146,33 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
_vd = None
|
||||
instruct_str = heal_design_instruct(row["instruct"], _vd)
|
||||
|
||||
if used_seed is not None:
|
||||
if used_seed is not None and not prepare_only:
|
||||
torch.manual_seed(used_seed)
|
||||
|
||||
# Last gate before the engine: every resolution branch above
|
||||
# produces a PATH, and none of them can know it still exists.
|
||||
ref_audio = warn_if_ref_missing(
|
||||
ref_audio, job_id=job_id, seg_id=seg_id, where="dub render",
|
||||
ref_audio, job_id=job_id, seg_id=effective_seg_id, where="dub render",
|
||||
)
|
||||
|
||||
if prepare_only:
|
||||
return {
|
||||
"text": text,
|
||||
"language": lang if lang != "Auto" else None,
|
||||
"ref_audio": ref_audio,
|
||||
"ref_text": ref_text,
|
||||
"cache_ref": not ref_single_use,
|
||||
"instruct": instruct_str if instruct_str else None,
|
||||
"duration": dur_s,
|
||||
"num_step": nstep,
|
||||
"guidance_scale": cfg,
|
||||
"speed": spd,
|
||||
"denoise": True,
|
||||
"postprocess_output": _job_postprocess,
|
||||
"effect_preset": effect_preset or "broadcast",
|
||||
"seed": used_seed,
|
||||
}
|
||||
|
||||
try:
|
||||
audio_out = backend.generate(
|
||||
text=text, language=lang if lang != "Auto" else None,
|
||||
@@ -1040,7 +1180,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
cache_ref=not ref_single_use,
|
||||
instruct=instruct_str if instruct_str else None,
|
||||
duration=dur_s, num_step=nstep, guidance_scale=cfg,
|
||||
speed=spd, denoise=True, postprocess_output=True,
|
||||
speed=spd, denoise=True, postprocess_output=_job_postprocess,
|
||||
)
|
||||
sr = backend.sample_rate
|
||||
|
||||
@@ -1081,7 +1221,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
cache_ref=not ref_single_use,
|
||||
instruct=instruct_str if instruct_str else None,
|
||||
duration=dur_s, num_step=retry_steps, guidance_scale=cfg,
|
||||
speed=spd, denoise=True, postprocess_output=True,
|
||||
speed=spd, denoise=True, postprocess_output=_job_postprocess,
|
||||
)
|
||||
sr = backend.sample_rate
|
||||
|
||||
@@ -1109,6 +1249,134 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
f"Underlying error: {retry_err}"
|
||||
) from retry_err
|
||||
|
||||
async def _prefetch_native_batch(first_index: int) -> None:
|
||||
"""Render one bounded batch and retain only its small output window."""
|
||||
if _native_batch_width < 2 or remote_audio:
|
||||
return
|
||||
batch: list[tuple[int, dict]] = []
|
||||
compatibility = None
|
||||
for candidate_index in range(first_index, len(req.segments)):
|
||||
candidate = req.segments[candidate_index]
|
||||
candidate_id = (
|
||||
seg_ids[candidate_index]
|
||||
if candidate_index < len(seg_ids)
|
||||
else f"seg_{candidate_index}"
|
||||
)
|
||||
if (
|
||||
candidate_index in _batched_audio
|
||||
or candidate.end - candidate.start <= 0.05
|
||||
or not candidate.text.strip()
|
||||
or (
|
||||
regen_only is not None
|
||||
and candidate_id not in regen_only
|
||||
)
|
||||
):
|
||||
continue
|
||||
args = _segment_generation_args(candidate_index, candidate)
|
||||
try:
|
||||
prepared = _gen(
|
||||
args["text"],
|
||||
args["language"],
|
||||
args["instruct"],
|
||||
args["duration"],
|
||||
args["num_step"],
|
||||
args["guidance_scale"],
|
||||
args["speed"],
|
||||
args["profile_id"],
|
||||
args["effect_preset"],
|
||||
prepare_only=True,
|
||||
current_seg_id=args["seg_id"],
|
||||
)
|
||||
except Exception:
|
||||
if candidate_index == first_index:
|
||||
raise
|
||||
break
|
||||
# Fixed-seed profiles deliberately keep their established
|
||||
# one-row deterministic RNG contract.
|
||||
if prepared["seed"] is not None:
|
||||
if candidate_index == first_index:
|
||||
return
|
||||
break
|
||||
candidate_compatibility = (
|
||||
prepared["cache_ref"],
|
||||
bool(prepared["ref_audio"]),
|
||||
prepared["num_step"],
|
||||
prepared["guidance_scale"],
|
||||
prepared["postprocess_output"],
|
||||
)
|
||||
if compatibility is None:
|
||||
compatibility = candidate_compatibility
|
||||
elif candidate_compatibility != compatibility:
|
||||
break
|
||||
batch.append((candidate_index, prepared))
|
||||
if len(batch) >= _native_batch_width:
|
||||
break
|
||||
if len(batch) < 2:
|
||||
return
|
||||
|
||||
def _render_batch() -> list[torch.Tensor]:
|
||||
prepared_rows = [prepared for _, prepared in batch]
|
||||
outputs = backend.generate_batch(
|
||||
[prepared["text"] for prepared in prepared_rows],
|
||||
language=[prepared["language"] for prepared in prepared_rows],
|
||||
ref_audio=[prepared["ref_audio"] for prepared in prepared_rows],
|
||||
ref_text=[prepared["ref_text"] for prepared in prepared_rows],
|
||||
cache_ref=prepared_rows[0]["cache_ref"],
|
||||
instruct=[prepared["instruct"] for prepared in prepared_rows],
|
||||
duration=[prepared["duration"] for prepared in prepared_rows],
|
||||
num_step=prepared_rows[0]["num_step"],
|
||||
guidance_scale=prepared_rows[0]["guidance_scale"],
|
||||
speed=[prepared["speed"] for prepared in prepared_rows],
|
||||
denoise=True,
|
||||
postprocess_output=prepared_rows[0]["postprocess_output"],
|
||||
)
|
||||
if len(outputs) != len(prepared_rows):
|
||||
raise RuntimeError(
|
||||
f"native batch returned {len(outputs)} outputs for "
|
||||
f"{len(prepared_rows)} segments"
|
||||
)
|
||||
rendered = []
|
||||
for output, prepared in zip(outputs, prepared_rows):
|
||||
preset = prepared["effect_preset"]
|
||||
if preset == "raw":
|
||||
rendered.append(output)
|
||||
continue
|
||||
mastered = output
|
||||
if not getattr(backend, "applies_own_mastering", False):
|
||||
mastered = apply_mastering(mastered, sample_rate=backend.sample_rate)
|
||||
effect_chain = get_effect_chain(preset)
|
||||
if effect_chain:
|
||||
mastered = apply_effects_chain(
|
||||
mastered,
|
||||
sample_rate=backend.sample_rate,
|
||||
chain=effect_chain,
|
||||
)
|
||||
rendered.append(normalize_audio(mastered, target_dBFS=-2.0))
|
||||
return rendered
|
||||
|
||||
try:
|
||||
outputs = await run_on_gpu_pool_guarded(
|
||||
_render_batch,
|
||||
what="Dub generate batch",
|
||||
timeout=batch_timeout_s(
|
||||
[prepared["text"] for _, prepared in batch], backend
|
||||
),
|
||||
)
|
||||
except TimeoutError:
|
||||
raise
|
||||
except Exception as error:
|
||||
_prepare_oom_retry(error, execution_target="local")
|
||||
logger.warning(
|
||||
"Native dub batch failed for segments %s-%s; falling back: %s",
|
||||
batch[0][0] + 1,
|
||||
batch[-1][0] + 1,
|
||||
error,
|
||||
)
|
||||
return
|
||||
_batched_audio.update(
|
||||
(index, output) for (index, _), output in zip(batch, outputs)
|
||||
)
|
||||
|
||||
seg_profile = seg.profile_id or None
|
||||
seg_speed = seg.speed if hasattr(seg, 'speed') and seg.speed is not None else req.speed
|
||||
seg_lang = seg.target_lang if getattr(seg, 'target_lang', None) else req.language
|
||||
@@ -1149,8 +1417,8 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
# quality for ~2× speed by dropping flow-matching steps.
|
||||
# Client sends `preview=true` when the user is iterating;
|
||||
# before final export the client should re-call without the
|
||||
# flag to restore num_step=req.num_step quality.
|
||||
_num_step = 8 if req.preview else req.num_step
|
||||
# flag to restore the explicit override or shared profile.
|
||||
_num_step = 8 if req.preview else _job_num_step
|
||||
_t_tts_0 = time.perf_counter()
|
||||
seg_effect_preset = getattr(seg, "effect_preset", None) or "broadcast"
|
||||
|
||||
@@ -1177,14 +1445,19 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
import torchaudio.functional as AF
|
||||
audio_tensor = AF.resample(audio_tensor, remote_sr, backend.sample_rate)
|
||||
else:
|
||||
audio_tensor = await run_on_gpu_pool_guarded(
|
||||
lambda: _gen(
|
||||
seg.text, seg_lang, seg_instruct, _dur_for_tts,
|
||||
_num_step, req.guidance_scale, seg_speed, seg_profile, seg_effect_preset,
|
||||
),
|
||||
what="Dub generate",
|
||||
timeout=generate_timeout_s(seg.text, engine=backend),
|
||||
)
|
||||
if i not in _batched_audio:
|
||||
await _prefetch_native_batch(i)
|
||||
if i in _batched_audio:
|
||||
audio_tensor = _batched_audio.pop(i)
|
||||
else:
|
||||
audio_tensor = await run_on_gpu_pool_guarded(
|
||||
lambda: _gen(
|
||||
seg.text, seg_lang, seg_instruct, _dur_for_tts,
|
||||
_num_step, req.guidance_scale, seg_speed, seg_profile, seg_effect_preset,
|
||||
),
|
||||
what="Dub generate",
|
||||
timeout=generate_timeout_s(seg.text, engine=backend),
|
||||
)
|
||||
_t_tts += time.perf_counter() - _t_tts_0
|
||||
|
||||
# Check abort immediately after GPU work completes
|
||||
@@ -1192,25 +1465,19 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
yield f"data: {json.dumps({'type': 'cancelled', 'segments_processed': i + 1})}\n\n"
|
||||
return
|
||||
|
||||
target_samples = int(seg_duration * backend.sample_rate)
|
||||
current_samples = audio_tensor.shape[-1]
|
||||
# Capture the real spoken duration before assembly fitting.
|
||||
# This is the evidence used by Agent timing and
|
||||
# keeps sync badges truthful for every timing strategy.
|
||||
natural_generated_dur = current_samples / backend.sample_rate
|
||||
|
||||
if strategy == "strict_slot":
|
||||
# Legacy: pad short audio + trim long audio so the mix
|
||||
# loop receives slot-sized buffers. The atempo squeeze
|
||||
# in the mix loop never fires here because we already
|
||||
# forced size = target_samples.
|
||||
if target_samples > current_samples:
|
||||
pad_amount = target_samples - current_samples
|
||||
audio_tensor = torch.nn.functional.pad(audio_tensor, (0, pad_amount))
|
||||
elif current_samples > target_samples:
|
||||
audio_tensor = audio_tensor[..., :target_samples]
|
||||
# concise / stretch_video / smart_fit: keep audio at its
|
||||
# natural length. The mix loop decides per-mode whether to
|
||||
# trim, slip, stretch the video, or split audio/video
|
||||
# retiming (smart_fit) to accommodate it.
|
||||
# Keep the complete waveform in every cache. Fitting happens
|
||||
# once during assembly; pre-trimming here destroyed words before
|
||||
# the pitch-preserving stretcher could see them.
|
||||
if current_samples == 0 or not torch.isfinite(audio_tensor).all() or not torch.any(audio_tensor.abs() > 1e-6):
|
||||
raise ValueError("The speech engine returned empty or silent audio")
|
||||
|
||||
generated_dur = audio_tensor.shape[-1] / backend.sample_rate
|
||||
generated_dur = natural_generated_dur
|
||||
sync_ratio = round(generated_dur / max(seg_duration, 0.01), 3)
|
||||
|
||||
sync_scores.append(sync_ratio)
|
||||
@@ -1218,7 +1485,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
# Duration-planner calibration sample: this text length spoke
|
||||
# for this long at natural rate. Keyed by stable seg id and
|
||||
# merged into the per-language job map after the loop.
|
||||
if strategy != "strict_slot" and seg.text.strip() and generated_dur > 0:
|
||||
if seg.text.strip() and generated_dur > 0:
|
||||
_natural_dur_records[str(seg_id)] = {
|
||||
"chars": len(seg.text.strip()),
|
||||
"dur": round(generated_dur, 4),
|
||||
@@ -1256,15 +1523,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
if rvc_sr == backend.sample_rate:
|
||||
audio_tensor = rvc_wav
|
||||
|
||||
if strategy == "strict_slot":
|
||||
target_samples = int(seg_duration * backend.sample_rate)
|
||||
current_samples = audio_tensor.shape[-1]
|
||||
if target_samples > current_samples:
|
||||
audio_tensor = torch.nn.functional.pad(
|
||||
audio_tensor, (0, target_samples - current_samples)
|
||||
)
|
||||
elif current_samples > target_samples:
|
||||
audio_tensor = audio_tensor[..., :target_samples]
|
||||
except Exception as e:
|
||||
yield f"data: {json.dumps({'type': 'warning', 'segment': i, 'message': f'RVC skipped: {str(e)[:120]}'})}\n\n"
|
||||
|
||||
@@ -1311,10 +1569,9 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
from core.public_errors import stream_generation_failure
|
||||
|
||||
error_detail = stream_generation_failure(e)["detail"]
|
||||
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'error': error_detail})}\n\n"
|
||||
sr = backend.sample_rate
|
||||
all_segment_wavs.append(_store_mix_wav(seg.start, seg.end, torch.zeros(1, max(0, int(seg_duration * sr))), sr, f"mix_{seg_id}"))
|
||||
sync_scores.append(1.0)
|
||||
logger.exception("Dub generation failed for segment %s", seg_id)
|
||||
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'segment_id': seg_id, 'error': error_detail})}\n\n"
|
||||
return
|
||||
|
||||
_t_loop_end = time.perf_counter()
|
||||
|
||||
@@ -1461,19 +1718,16 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
seg_gain = max(0.0, min(2.0, seg_gain))
|
||||
try:
|
||||
wav = _load_entry_wav((start, end, wav_path, sr), sr)
|
||||
except Exception as e:
|
||||
# A WAV header can be readable while its payload is
|
||||
# truncated. Direct cache reuse deliberately defers the
|
||||
# decode to assembly, so preserve the old recovery contract
|
||||
# here: warn and fill this slot with silence instead of
|
||||
# aborting the entire dub.
|
||||
warning = {
|
||||
"type": "warning",
|
||||
"segment": i,
|
||||
"message": f"cached seg lost, padding silence: {str(e)[:120]}",
|
||||
}
|
||||
yield f"data: {json.dumps(warning)}\n\n"
|
||||
wav = torch.zeros(1, max(0, int((end - start) * sr)))
|
||||
except Exception:
|
||||
logger.exception("Dub assembly could not read segment %d", i)
|
||||
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'error_code': 'dub_speech_missing', 'error': 'A speech segment could not be read. Regenerate it before exporting.'})}\n\n"
|
||||
return
|
||||
from services.audio_dsp import trim_speech_padding
|
||||
if seg_ref is not None and seg_ref.text.strip():
|
||||
if wav.numel() == 0 or not torch.isfinite(wav).all() or not torch.any(wav.abs() > 1e-6):
|
||||
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'error_code': 'dub_speech_missing', 'error': 'A speech segment is empty or silent. Regenerate it before exporting.'})}\n\n"
|
||||
return
|
||||
wav = trim_speech_padding(wav, sr)
|
||||
adjusted = wav * seg_gain
|
||||
if adjusted.ndim == 2 and adjusted.shape[0] > 1:
|
||||
adjusted = adjusted.mean(dim=0, keepdim=True)
|
||||
@@ -1521,12 +1775,12 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
align_corners=False,
|
||||
).squeeze(0)
|
||||
wl = adjusted.shape[-1]
|
||||
# Residual overflow → hard-trim to the segment's new video
|
||||
# slot (fade below keeps the cut pop-free).
|
||||
# Never publish a complete track with speech discarded by
|
||||
# the fit caps. The user can shorten text or relax the caps.
|
||||
new_slot_samples = int(max(0.0, sf.new_end - sf.new_start) * sr)
|
||||
if new_slot_samples > 0 and wl > new_slot_samples:
|
||||
adjusted = adjusted[..., :new_slot_samples]
|
||||
wl = adjusted.shape[-1]
|
||||
if new_slot_samples > 0 and wl > new_slot_samples + int(sr * 0.02):
|
||||
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'error_code': 'dub_timing_overflow', 'error': 'Speech exceeds the fitting limits. Shorten the translation or choose Strict Slot or Stretch Video before exporting.'})}\n\n"
|
||||
return
|
||||
# Truthful per-segment verdict for the UI badge.
|
||||
entry = {"status": sf.status}
|
||||
if abs(sf.audio_rate - 1.0) > 1e-6:
|
||||
@@ -1546,8 +1800,8 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
elif strategy == "concise":
|
||||
# Mode A: never compress. Allow the audio to extend into the
|
||||
# silent gap before the next seg (existing heuristic) plus
|
||||
# any extra `overflow_budget_s`. Beyond that, hard-trim with
|
||||
# a short fade so we never overlap the next speaker.
|
||||
# any extra `overflow_budget_s`. Beyond that, require a
|
||||
# timing/text adjustment instead of discarding speech.
|
||||
place_at = start
|
||||
effective_end = end
|
||||
if i + 1 < len(all_segment_wavs):
|
||||
@@ -1561,47 +1815,25 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
slot_samples_eff = int(max(0.0, (effective_end - start)) * sr)
|
||||
if slot_samples_eff > 0 and wl > slot_samples_eff:
|
||||
overflow_s = (wl - slot_samples_eff) / sr
|
||||
adjusted = adjusted[..., :slot_samples_eff]
|
||||
wl = adjusted.shape[-1]
|
||||
fit_status.append({
|
||||
"status": "overflows",
|
||||
"overflow_s": round(overflow_s, 3),
|
||||
})
|
||||
yield f"data: {json.dumps({'type': 'error', 'segment': i, 'segment_id': job['seg_order'][i], 'error_code': 'dub_timing_overflow', 'error': 'Speech exceeds its time slot. Shorten the translation or choose Strict Slot or Stretch Video before exporting.', 'overflow_s': round(overflow_s, 3)})}\n\n"
|
||||
return
|
||||
else:
|
||||
fit_status.append({"status": "fits"})
|
||||
|
||||
else:
|
||||
# strict_slot (legacy): preserve the previous atempo / trim /
|
||||
# off semantics so existing callers and back-compat tests
|
||||
# keep passing.
|
||||
# Strict Slot fits the complete speech to the original
|
||||
# slot. Explicit legacy trim/off choices remain available.
|
||||
place_at = start
|
||||
effective_end = end
|
||||
slowed_rate = None
|
||||
if i + 1 < len(all_segment_wavs):
|
||||
next_start = all_segment_wavs[i + 1][0]
|
||||
gap = next_start - end
|
||||
if gap > GAP_OVERFLOW_BUFFER_S:
|
||||
effective_end = end + min(
|
||||
gap - GAP_OVERFLOW_BUFFER_S, GAP_OVERFLOW_MAX_S,
|
||||
)
|
||||
slot_samples = int(max(0.0, (effective_end - start)) * sr)
|
||||
if slot_fit != "off" and slot_samples > 0 and wl > slot_samples:
|
||||
if slot_fit == "time_stretch":
|
||||
ratio = wl / slot_samples
|
||||
capped_ratio = min(ratio, MAX_STRETCH_RATIO)
|
||||
capped_target = int(wl / capped_ratio)
|
||||
try:
|
||||
adjusted = await _pitch_preserving_stretch(
|
||||
adjusted, capped_target, sr,
|
||||
adjusted, slot_samples, sr,
|
||||
)
|
||||
if adjusted.shape[-1] > slot_samples:
|
||||
adjusted = adjusted[..., :slot_samples]
|
||||
if ratio > MAX_STRETCH_RATIO:
|
||||
logger.info(
|
||||
"seg %d compression %.2f× exceeded cap; "
|
||||
"stretched to %.2f×, tail trimmed",
|
||||
i, ratio, capped_ratio,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"atempo stretch failed for seg %d (%.2f×), "
|
||||
@@ -1621,15 +1853,14 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
slot_fit == "time_stretch"
|
||||
and slot_samples > 0
|
||||
and wl > 0
|
||||
and wl < slot_samples * UNDERRUN_TOLERANCE
|
||||
and _underrun_min_rate() < 1.0 - 1e-6
|
||||
and wl < slot_samples
|
||||
):
|
||||
# Underrun fill (mirror of the compression above): the
|
||||
# dub finished early, leaving the on-screen mouth moving
|
||||
# over the thin under-speech bed residue — perceived as
|
||||
# dead air. Slow toward the slot, never below the floor.
|
||||
rate = max(wl / slot_samples, _underrun_min_rate())
|
||||
target = min(slot_samples, int(round(wl / rate)))
|
||||
rate = wl / slot_samples
|
||||
target = slot_samples
|
||||
try:
|
||||
adjusted = await _pitch_preserving_stretch(
|
||||
adjusted, target, sr,
|
||||
@@ -1730,6 +1961,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
"language_code": lang_code,
|
||||
"duration": round(track_dur, 4),
|
||||
"timing_strategy": strategy,
|
||||
"source_segments": [{"start": seg.start, "end": seg.end} for seg in req.segments],
|
||||
}
|
||||
|
||||
# Persist the timing strategy + (for Mode B) the per-segment stretch
|
||||
@@ -1773,12 +2005,9 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
"fit_fp": fit_fp,
|
||||
}
|
||||
job["dubbed_tracks"][lang_code]["fit_fp"] = fit_fp
|
||||
# Record what kind of per-segment WAVs are on disk so a later
|
||||
# smart_fit run knows whether partial regen / fit-only re-mix can
|
||||
# reuse them ("natural") or must regen once ("slotted"). Per-track
|
||||
# (P1.3) — each language renders under its own strategy; the flat
|
||||
# field stays in lock-step for older readers.
|
||||
_kind = "slotted" if strategy == "strict_slot" else "natural"
|
||||
# Every new cache preserves natural speech. Old slotted caches must
|
||||
# be regenerated once because their missing tails cannot be recovered.
|
||||
_kind = "natural"
|
||||
job.setdefault("seg_wav_kind_by_lang", {})[lang_code] = _kind
|
||||
job["seg_wav_kind"] = _kind
|
||||
_save_job(job_id, job)
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Optional
|
||||
from fastapi import APIRouter
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from schemas.requests import TranslateRequest
|
||||
from schemas.requests import AgentFitRequest, TranslateRequest
|
||||
from services.model_manager import _cpu_pool, _gpu_pool
|
||||
from services.hf_revisions import revision_for
|
||||
from services.translator import cinematic_available, cinematic_refine_many, _cinematic_budget
|
||||
@@ -19,10 +20,11 @@ _NLLB_REPO_ID = "facebook/nllb-200-distilled-600M"
|
||||
|
||||
|
||||
def _load_nllb_component(factory):
|
||||
"""Load a curated NLLB component from its reviewed immutable revision."""
|
||||
"""Load explicitly installed NLLB weights at their reviewed revision."""
|
||||
return factory.from_pretrained(
|
||||
_NLLB_REPO_ID,
|
||||
revision=revision_for(_NLLB_REPO_ID),
|
||||
local_files_only=True,
|
||||
)
|
||||
|
||||
TRANSLATE_CODES = {
|
||||
@@ -39,8 +41,33 @@ FLORES_CODES = {
|
||||
"hi": "hin_Deva", "tr": "tur_Latn", "pl": "pol_Latn", "nl": "nld_Latn",
|
||||
"sv": "swe_Latn", "th": "tha_Thai", "vi": "vie_Latn", "id": "ind_Latn",
|
||||
"uk": "ukr_Cyrl",
|
||||
"zh-TW": "zho_Hant", "zh-Hant": "zho_Hant", "cmn-Hant": "zho_Hant",
|
||||
"zh-Hans": "zho_Hans", "yue": "yue_Hant",
|
||||
"bn": "ben_Beng", "ta": "tam_Taml", "te": "tel_Telu", "ml": "mal_Mlym",
|
||||
"kn": "kan_Knda", "gu": "guj_Gujr", "mr": "mar_Deva", "ur": "urd_Arab",
|
||||
"fa": "pes_Arab", "he": "heb_Hebr", "el": "ell_Grek", "cs": "ces_Latn",
|
||||
"da": "dan_Latn", "fi": "fin_Latn", "nb": "nob_Latn", "nn": "nno_Latn",
|
||||
"ro": "ron_Latn", "hu": "hun_Latn", "bg": "bul_Cyrl", "sk": "slk_Latn",
|
||||
"sl": "slv_Latn", "hr": "hrv_Latn", "sr": "srp_Cyrl", "lt": "lit_Latn",
|
||||
"et": "est_Latn", "sw": "swh_Latn", "af": "afr_Latn", "ms": "zsm_Latn",
|
||||
}
|
||||
|
||||
|
||||
def _nllb_language(code: str) -> str | None:
|
||||
"""Resolve aliases or tokenizer-supported FLORES codes without loading weights."""
|
||||
from transformers.models.nllb.tokenization_nllb import FAIRSEQ_LANGUAGE_CODES
|
||||
|
||||
normalized = code.strip().replace("_", "-").lower()
|
||||
aliases = {key.lower(): value for key, value in FLORES_CODES.items()}
|
||||
if normalized in aliases:
|
||||
return aliases[normalized]
|
||||
exact = [value for value in FAIRSEQ_LANGUAGE_CODES if value.replace("_", "-").lower() == normalized]
|
||||
if exact:
|
||||
return exact[0]
|
||||
# Bare ISO-639-3 codes are safe only when the tokenizer has one script.
|
||||
matches = [value for value in FAIRSEQ_LANGUAGE_CODES if value.split("_")[0] == normalized]
|
||||
return matches[0] if len(matches) == 1 else None
|
||||
|
||||
# Human-readable language names for LLM prompts. Empirically a tiny / 7B
|
||||
# local LLM produces Devanagari Hindi reliably when told "translate into
|
||||
# Hindi" but drifts to German / English / phonetic-Latin when told
|
||||
@@ -163,9 +190,77 @@ def _looks_like_target(text: str, code: str, threshold: float = 0.5) -> bool:
|
||||
codepoints alone."""
|
||||
return _script_ratio(text, code) >= threshold
|
||||
|
||||
|
||||
def _translation_output_error(text: object) -> str | None:
|
||||
"""Reject provider error pages that arrive with HTTP 200.
|
||||
|
||||
Google's mobile endpoint occasionally returns its generic HTML error copy
|
||||
inside the element deep-translator treats as a successful translation.
|
||||
Passing that through would replace the user's transcript with the error
|
||||
page, so treat it like any other transient provider failure and retry.
|
||||
"""
|
||||
if not isinstance(text, str) or not text.strip():
|
||||
return "empty translation"
|
||||
normalized = " ".join(text.split()).casefold()
|
||||
error_markers = (
|
||||
"error 500 (server error)",
|
||||
"that's an error",
|
||||
"that’s an error",
|
||||
"there was an error. please try again later",
|
||||
"no translation was found using the current translator",
|
||||
)
|
||||
if "\ufffd" in text or any(marker in normalized for marker in error_markers):
|
||||
return "translation provider returned invalid output"
|
||||
return None
|
||||
|
||||
_nllb_model = None
|
||||
_nllb_tokenizer = None
|
||||
_nllb_device = None
|
||||
_NLLB_BATCH_SIZE_ENV = "OMNIVOICE_NLLB_BATCH_SIZE"
|
||||
_NLLB_MAX_BATCH_SIZE = 32
|
||||
|
||||
|
||||
def _nllb_batch_size() -> int:
|
||||
"""Bound NLLB forward-pass width; explicit overrides remain available."""
|
||||
configured = os.environ.get(_NLLB_BATCH_SIZE_ENV, "").strip()
|
||||
if configured:
|
||||
try:
|
||||
return max(1, min(_NLLB_MAX_BATCH_SIZE, int(configured)))
|
||||
except (TypeError, ValueError):
|
||||
logger.warning("%s=%r is not an integer; using the safe default", _NLLB_BATCH_SIZE_ENV, configured)
|
||||
# The 600M checkpoint leaves ample room on modern discrete GPUs. Scale the
|
||||
# forward-pass width there; CPU and unified-memory MPS keep the conservative
|
||||
# width because their failure recovery moves the whole model.
|
||||
if _nllb_device == "cuda":
|
||||
try:
|
||||
import torch
|
||||
|
||||
free_gib = int(torch.cuda.mem_get_info()[0]) / 1024**3
|
||||
if free_gib >= 16:
|
||||
return 24
|
||||
if free_gib >= 8:
|
||||
return 12
|
||||
except Exception:
|
||||
pass
|
||||
return 8
|
||||
return 4
|
||||
|
||||
|
||||
def _nllb_hypothesis_budget() -> int:
|
||||
"""Bound batch × beam hypotheses by currently available device memory."""
|
||||
if _nllb_device != "cuda":
|
||||
return 16
|
||||
try:
|
||||
import torch
|
||||
|
||||
free_gib = int(torch.cuda.mem_get_info()[0]) / 1024**3
|
||||
if free_gib >= 16:
|
||||
return 64
|
||||
if free_gib >= 8:
|
||||
return 32
|
||||
except Exception:
|
||||
pass
|
||||
return 16
|
||||
|
||||
|
||||
def _dialect_flags(req, applied: bool) -> dict:
|
||||
@@ -255,10 +350,11 @@ def _resolve_translation_context(req, client, model_name: str, timeout: float,
|
||||
|
||||
def _unload_nllb():
|
||||
"""Release NLLB VRAM so TTS model can reload."""
|
||||
global _nllb_model, _nllb_tokenizer
|
||||
global _nllb_device, _nllb_model, _nllb_tokenizer
|
||||
import gc
|
||||
_nllb_model = None
|
||||
_nllb_tokenizer = None
|
||||
_nllb_device = None
|
||||
gc.collect()
|
||||
try:
|
||||
import torch
|
||||
@@ -270,10 +366,33 @@ def _unload_nllb():
|
||||
pass
|
||||
|
||||
|
||||
def _should_unload_nllb() -> bool:
|
||||
"""Retain a warm local translator only when the accelerator has safe headroom."""
|
||||
override = os.environ.get("OMNIVOICE_UNLOAD_NLLB")
|
||||
if override is not None:
|
||||
return override.strip().lower() not in {"0", "false", "no", "off"}
|
||||
if _nllb_device != "cuda":
|
||||
return True
|
||||
try:
|
||||
import torch
|
||||
|
||||
free_bytes, total_bytes = torch.cuda.mem_get_info()
|
||||
return total_bytes < 16 * 1024**3 or free_bytes < 8 * 1024**3
|
||||
except Exception:
|
||||
return True
|
||||
|
||||
|
||||
@router.post("/dub/translate")
|
||||
async def dub_translate(req: TranslateRequest):
|
||||
try:
|
||||
provider = (req.provider if req.provider else os.environ.get("TRANSLATE_PROVIDER", "google")).lower()
|
||||
from services import translation_engines
|
||||
|
||||
if not translation_engines.get_engine(provider):
|
||||
return JSONResponse(
|
||||
status_code=400,
|
||||
content={"error": "Choose a supported translation engine."},
|
||||
)
|
||||
lang_code = TRANSLATE_CODES.get(req.target_lang, req.target_lang)
|
||||
api_key = os.environ.get("TRANSLATE_API_KEY", "")
|
||||
loop = asyncio.get_running_loop()
|
||||
@@ -281,8 +400,16 @@ async def dub_translate(req: TranslateRequest):
|
||||
|
||||
# Offline NLLB Transformer Translation
|
||||
if provider == "nllb":
|
||||
flores_tgt = FLORES_CODES.get(req.target_lang, "eng_Latn")
|
||||
flores_src = FLORES_CODES.get(src_lang, "eng_Latn")
|
||||
requested = [src_lang, req.target_lang, *(seg.target_lang for seg in req.segments if seg.target_lang)]
|
||||
resolved = {code: _nllb_language(code) for code in requested}
|
||||
unsupported = [code for code, language in resolved.items() if language is None]
|
||||
if unsupported:
|
||||
return JSONResponse(status_code=400, content={
|
||||
"error": "NLLB does not support the requested language.",
|
||||
"code": "unsupported_translation_language", "languages": unsupported,
|
||||
})
|
||||
flores_tgt = resolved[req.target_lang]
|
||||
flores_src = resolved[src_lang]
|
||||
|
||||
def _translate_nllb():
|
||||
global _nllb_model, _nllb_tokenizer, _nllb_device
|
||||
@@ -314,44 +441,112 @@ async def dub_translate(req: TranslateRequest):
|
||||
logger.exception("NLLB model load failed")
|
||||
return [{"id": seg.id, "text": seg.text, "error": f"Model load error: {str(e)}"} for seg in req.segments]
|
||||
|
||||
results = []
|
||||
for seg in req.segments:
|
||||
from services.performance_profiles import translation_decode_defaults
|
||||
|
||||
# Snapshot once so every segment and device fallback in this
|
||||
# job uses the same decoding effort even if preferences change.
|
||||
decode_options = translation_decode_defaults()
|
||||
def _generate_rows(rows, target_language):
|
||||
global _nllb_device
|
||||
|
||||
_nllb_tokenizer.src_lang = flores_src
|
||||
inputs = _nllb_tokenizer(
|
||||
[seg.text for _, seg in rows],
|
||||
return_tensors="pt",
|
||||
padding=True,
|
||||
)
|
||||
if _nllb_device and _nllb_device != "cpu":
|
||||
inputs = {key: value.to(_nllb_device) for key, value in inputs.items()}
|
||||
forced_bos_token_id = _nllb_tokenizer.convert_tokens_to_ids(target_language)
|
||||
try:
|
||||
if not seg.text or not seg.text.strip():
|
||||
results.append({"id": seg.id, "text": seg.text})
|
||||
continue
|
||||
tokens = _nllb_model.generate(
|
||||
**inputs,
|
||||
forced_bos_token_id=forced_bos_token_id,
|
||||
max_length=400,
|
||||
**decode_options,
|
||||
)
|
||||
except (RuntimeError, NotImplementedError) as error:
|
||||
if _nllb_device != "mps":
|
||||
raise
|
||||
logger.warning("MPS generate failed, retrying on CPU: %s", error)
|
||||
_nllb_model.to("cpu")
|
||||
_nllb_device = "cpu"
|
||||
inputs = {key: value.to("cpu") for key, value in inputs.items()}
|
||||
tokens = _nllb_model.generate(
|
||||
**inputs,
|
||||
forced_bos_token_id=forced_bos_token_id,
|
||||
max_length=400,
|
||||
**decode_options,
|
||||
)
|
||||
decoded = _nllb_tokenizer.batch_decode(tokens, skip_special_tokens=True)
|
||||
if len(decoded) != len(rows):
|
||||
raise RuntimeError(
|
||||
f"NLLB returned {len(decoded)} translations for {len(rows)} segments"
|
||||
)
|
||||
return decoded
|
||||
|
||||
tgt = FLORES_CODES.get(seg.target_lang, flores_tgt) if seg.target_lang else flores_tgt
|
||||
# A target-language BOS token is shared by a forward pass, so
|
||||
# group mixed-language rows first. Preserve request order in
|
||||
# the final response even though groups render independently.
|
||||
grouped: dict[str, list[tuple[int, object]]] = {}
|
||||
results_by_index: dict[int, dict] = {}
|
||||
for index, seg in enumerate(req.segments):
|
||||
if not seg.text or not seg.text.strip():
|
||||
results_by_index[index] = {"id": seg.id, "text": seg.text}
|
||||
continue
|
||||
target = resolved[seg.target_lang] if seg.target_lang else flores_tgt
|
||||
grouped.setdefault(target, []).append((index, seg))
|
||||
|
||||
_nllb_tokenizer.src_lang = flores_src
|
||||
inputs = _nllb_tokenizer(seg.text, return_tensors="pt")
|
||||
if _nllb_device and _nllb_device != "cpu":
|
||||
inputs = {k: v.to(_nllb_device) for k, v in inputs.items()}
|
||||
|
||||
forced_bos_token_id = _nllb_tokenizer.convert_tokens_to_ids(tgt)
|
||||
# Beam search multiplies decoder memory per row. Keep the
|
||||
# effective hypothesis count bounded while still widening the
|
||||
# Fast path aggressively.
|
||||
beam_count = max(1, int(decode_options.get("num_beams", 1)))
|
||||
width = min(
|
||||
_nllb_batch_size(),
|
||||
max(1, _nllb_hypothesis_budget() // beam_count),
|
||||
)
|
||||
for target, rows in grouped.items():
|
||||
for start in range(0, len(rows), width):
|
||||
batch = rows[start : start + width]
|
||||
try:
|
||||
translated_tokens = _nllb_model.generate(
|
||||
**inputs, forced_bos_token_id=forced_bos_token_id, max_length=400
|
||||
translated_texts = _generate_rows(batch, target)
|
||||
except Exception as batch_error:
|
||||
if len(batch) == 1:
|
||||
index, seg = batch[0]
|
||||
results_by_index[index] = {
|
||||
"id": seg.id,
|
||||
"text": seg.text,
|
||||
"error": str(batch_error),
|
||||
}
|
||||
continue
|
||||
# A single unusually long row must not sink its
|
||||
# neighbours. Clear a failed device allocation and
|
||||
# retain the established per-segment degradation.
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
logger.warning(
|
||||
"NLLB batch of %d failed; retrying rows individually: %s",
|
||||
len(batch),
|
||||
batch_error,
|
||||
)
|
||||
except (RuntimeError, NotImplementedError) as e:
|
||||
if _nllb_device == "mps":
|
||||
logger.warning("MPS generate failed, retrying on CPU: %s", e)
|
||||
_nllb_model.to("cpu")
|
||||
_nllb_device = "cpu"
|
||||
inputs = {k: v.to("cpu") for k, v in inputs.items()}
|
||||
translated_tokens = _nllb_model.generate(
|
||||
**inputs, forced_bos_token_id=forced_bos_token_id, max_length=400
|
||||
)
|
||||
else:
|
||||
raise
|
||||
translated_text = _nllb_tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
|
||||
results.append({"id": seg.id, "text": translated_text})
|
||||
except Exception as e:
|
||||
results.append({"id": seg.id, "text": seg.text, "error": str(e)})
|
||||
return results
|
||||
for index, seg in batch:
|
||||
try:
|
||||
translated_text = _generate_rows([(index, seg)], target)[0]
|
||||
results_by_index[index] = {"id": seg.id, "text": translated_text}
|
||||
except Exception as row_error:
|
||||
results_by_index[index] = {
|
||||
"id": seg.id,
|
||||
"text": seg.text,
|
||||
"error": str(row_error),
|
||||
}
|
||||
continue
|
||||
for (index, seg), translated_text in zip(batch, translated_texts):
|
||||
results_by_index[index] = {"id": seg.id, "text": translated_text}
|
||||
|
||||
return [results_by_index[index] for index in range(len(req.segments))]
|
||||
|
||||
translated = await loop.run_in_executor(_gpu_pool, _translate_nllb)
|
||||
if os.environ.get("OMNIVOICE_UNLOAD_NLLB", "1") == "1":
|
||||
if _should_unload_nllb():
|
||||
_unload_nllb()
|
||||
# Cinematic/Autofit refine + rate-ratio badges must run for NLLB too
|
||||
# (previously this returned before _maybe_cinematic, so a Cinematic
|
||||
@@ -472,6 +667,7 @@ async def dub_translate(req: TranslateRequest):
|
||||
f"You are a professional dubbing translator. "
|
||||
f"Translate the user's text from {src_name} into "
|
||||
f"{tgt_name}.{script_clause}{dia_clause} "
|
||||
f"{translation_style_brief(req)} "
|
||||
f"Reply ONLY with the translated {tgt_name} text, do not "
|
||||
f"add quotes, notes, headers, explanations, or commentary."
|
||||
)
|
||||
@@ -543,7 +739,7 @@ async def dub_translate(req: TranslateRequest):
|
||||
source_lang=src_lang,
|
||||
target_lang=tgt_code,
|
||||
target_name=LANG_NAMES.get(tgt_code, tgt_code),
|
||||
extra_clause=context_extra,
|
||||
extra_clause="\n".join(filter(None, [context_extra, translation_style_brief(req)])),
|
||||
)
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.warning("reflect pass skipped for %s: %s",
|
||||
@@ -594,18 +790,35 @@ async def dub_translate(req: TranslateRequest):
|
||||
f"switch the Engine dropdown to another provider."
|
||||
)
|
||||
return JSONResponse(status_code=400, content={"error": friendly})
|
||||
target_codes = list(dict.fromkeys(
|
||||
seg.target_lang if seg.target_lang else req.target_lang
|
||||
for seg in req.segments
|
||||
))
|
||||
try:
|
||||
pack_status = translation_engines.argos_pack_status(src_lang, target_codes)
|
||||
except (ImportError, ValueError) as exc:
|
||||
return JSONResponse(status_code=422, content={"error": str(exc)})
|
||||
missing_packs = [
|
||||
pair for pair in pack_status["pairs"] if not pair["installed"]
|
||||
]
|
||||
if missing_packs:
|
||||
pairs = ", ".join(
|
||||
f'{pair["source_lang"]} → {pair["target_lang"]}'
|
||||
for pair in missing_packs
|
||||
)
|
||||
return JSONResponse(
|
||||
status_code=409,
|
||||
content={
|
||||
"error": f"Install the Argos language pack for {pairs} before translating.",
|
||||
"code": "argos_pack_missing",
|
||||
"pairs": missing_packs,
|
||||
},
|
||||
)
|
||||
|
||||
def _translate_argos():
|
||||
cache_dir = os.environ.get("OMNIVOICE_CACHE_DIR")
|
||||
if cache_dir:
|
||||
argos_cache = os.path.join(cache_dir, "argos-translate")
|
||||
os.makedirs(argos_cache, exist_ok=True)
|
||||
os.environ.setdefault("ARGOS_PACKAGES_DIR", argos_cache)
|
||||
os.environ.setdefault("ARGOS_DATA_DIR", argos_cache)
|
||||
import argostranslate.package
|
||||
import argostranslate.translate
|
||||
|
||||
from_code = src_lang
|
||||
available_packages = argostranslate.package.get_installed_packages()
|
||||
from_code = pack_status["source_lang"]
|
||||
|
||||
results = []
|
||||
for seg in req.segments:
|
||||
@@ -614,19 +827,12 @@ async def dub_translate(req: TranslateRequest):
|
||||
results.append({"id": seg.id, "text": seg.text})
|
||||
continue
|
||||
to_code = seg.target_lang if seg.target_lang else req.target_lang
|
||||
installed_pkg = next(filter(lambda x: x.from_code == from_code and x.to_code == to_code, available_packages), None)
|
||||
|
||||
if installed_pkg is None:
|
||||
argostranslate.package.update_package_index()
|
||||
all_packages = argostranslate.package.get_available_packages()
|
||||
package_to_install = next(filter(lambda x: x.from_code == from_code and x.to_code == to_code, all_packages), None)
|
||||
if package_to_install:
|
||||
argostranslate.package.install_from_path(package_to_install.download())
|
||||
available_packages = argostranslate.package.get_installed_packages()
|
||||
else:
|
||||
raise Exception(f"No Argos package available for {from_code} -> {to_code}")
|
||||
|
||||
translated_text = argostranslate.translate.translate(seg.text, from_code, to_code)
|
||||
to_code = translation_engines.argos_lang_code(to_code)
|
||||
translated_text = (
|
||||
seg.text
|
||||
if from_code == to_code
|
||||
else argostranslate.translate.translate(seg.text, from_code, to_code)
|
||||
)
|
||||
results.append({"id": seg.id, "text": translated_text})
|
||||
except Exception as e:
|
||||
results.append({"id": seg.id, "text": seg.text, "error": str(e)})
|
||||
@@ -700,9 +906,10 @@ async def dub_translate(req: TranslateRequest):
|
||||
for attempt, src in enumerate([src_arg, src_arg, "auto"]):
|
||||
try:
|
||||
out = _build_translator(src, seg_lc).translate(seg.text)
|
||||
if out and out.strip():
|
||||
output_error = _translation_output_error(out)
|
||||
if output_error is None:
|
||||
return {"id": seg.id, "text": out}
|
||||
last_err = "empty translation"
|
||||
last_err = output_error
|
||||
except Exception as e:
|
||||
last_err = f"{type(e).__name__}: {e}"
|
||||
logger.warning(
|
||||
@@ -895,7 +1102,7 @@ async def _apply_fit_pass(rows, req, slots_by_id, source_by_id, quality, loop, d
|
||||
their current text and get ``rate_error='fit-budget'``. Only rows with a
|
||||
slot + text + no prior error participate.
|
||||
"""
|
||||
strict = (quality == "autofit")
|
||||
strict = quality in ("autofit", "agent")
|
||||
items = []
|
||||
for row in rows:
|
||||
seg_id = str(row["id"])
|
||||
@@ -958,7 +1165,7 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
|
||||
|
||||
# Fast (and anything unrecognised) returns the plain translation unchanged
|
||||
# (plus the pre-synthesis duration-plan badges — no LLM needed for those).
|
||||
if quality not in ("cinematic", "autofit"):
|
||||
if quality not in ("cinematic", "autofit", "agent"):
|
||||
await _finalize_duration_plan(translated, req, loop)
|
||||
return base
|
||||
|
||||
@@ -1029,7 +1236,7 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
|
||||
target_lang=req.target_lang,
|
||||
glossary=req.glossary,
|
||||
directions=directions,
|
||||
dialect_hint=dialect_hint,
|
||||
dialect_hint="\n".join(filter(None, [dialect_hint, translation_style_brief(req)])),
|
||||
executor=_cpu_pool,
|
||||
)
|
||||
refined_by_id = {r["id"]: r for r in refined}
|
||||
@@ -1076,3 +1283,70 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
|
||||
"quality_used": quality,
|
||||
**_dialect_flags(req, applied=bool(dialect_hint)),
|
||||
}
|
||||
|
||||
|
||||
def translation_style_brief(req) -> str:
|
||||
instructions = (getattr(req, "translation_instructions", None) or "").strip()
|
||||
return ("User translation style brief (tone and wording only; preserve meaning, timing and output format): "
|
||||
+ json.dumps(instructions, ensure_ascii=False)) if instructions else ""
|
||||
|
||||
|
||||
@router.post("/dub/agent-fit")
|
||||
async def dub_agent_fit(req: AgentFitRequest):
|
||||
"""Rewrite rendered lines from real duration evidence.
|
||||
|
||||
Synthesis stays in the normal Dubbing pipeline. The client renders each
|
||||
candidate, measures it, and may request one more bounded correction.
|
||||
"""
|
||||
from services import llm_skills
|
||||
from services.speech_rate import adjust_for_measured_slot_many
|
||||
|
||||
readiness = llm_skills.resolve_skill("slot_fitting")
|
||||
if not readiness.ready:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
"error": "llm_skill_unavailable",
|
||||
"skill": "slot_fitting",
|
||||
"reason": readiness.reason or "unavailable",
|
||||
},
|
||||
)
|
||||
|
||||
items = [
|
||||
(
|
||||
segment.id,
|
||||
segment.text,
|
||||
segment.slot_seconds,
|
||||
segment.measured_seconds,
|
||||
req.target_lang,
|
||||
segment.source_text,
|
||||
segment.context_before,
|
||||
segment.context_after,
|
||||
)
|
||||
for segment in req.segments
|
||||
]
|
||||
budget = _cinematic_budget()
|
||||
try:
|
||||
call = adjust_for_measured_slot_many(items, executor=_cpu_pool, translation_instructions=req.translation_instructions)
|
||||
rows = await asyncio.wait_for(call, timeout=budget) if budget and budget > 0 else await call
|
||||
except asyncio.TimeoutError:
|
||||
rows = {
|
||||
segment.id: {
|
||||
"text": segment.text,
|
||||
"changed": False,
|
||||
"measured_seconds": round(segment.measured_seconds, 3),
|
||||
"target_seconds": round(segment.slot_seconds, 3),
|
||||
"measured_ratio": round(
|
||||
segment.measured_seconds / max(segment.slot_seconds, 0.001), 3
|
||||
),
|
||||
"error": "fit-budget",
|
||||
}
|
||||
for segment in req.segments
|
||||
}
|
||||
return {
|
||||
"target_lang": req.target_lang,
|
||||
"segments": [
|
||||
{"id": segment.id, **rows[str(segment.id)]}
|
||||
for segment in req.segments
|
||||
],
|
||||
}
|
||||
|
||||
@@ -15,6 +15,7 @@ Environment variables (`OMNIVOICE_TTS_BACKEND`, `OMNIVOICE_ASR_BACKEND`,
|
||||
`OMNIVOICE_LLM_BACKEND`) still win over the UI choice so power-users can pin
|
||||
a backend without Settings silently undoing it.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
@@ -23,10 +24,11 @@ from time import perf_counter
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from huggingface_hub import utils as hf_utils
|
||||
from huggingface_hub.errors import HFValidationError
|
||||
from pydantic import BaseModel
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from api.dependencies import require_admin, require_admin_action, require_desktop
|
||||
from core import prefs
|
||||
from core.engine_licenses import LICENSE_GATED_ENGINES
|
||||
from services import tts_backend, asr_backend, llm_backend, translation_engines
|
||||
from services.audio_dsp import list_effect_presets
|
||||
from api.schemas import EffectPresetsResponse
|
||||
@@ -58,12 +60,49 @@ def _catalogue_active_id(family: str, module) -> str:
|
||||
|
||||
def _family_payload(family: str, module):
|
||||
"""Public inventory plus whether an environment pin owns this family."""
|
||||
active = _catalogue_active_id(family, module)
|
||||
model = None
|
||||
if family == "asr":
|
||||
model = asr_backend._offline_asr_repo(active)
|
||||
elif family == "llm" and active != "off":
|
||||
model = llm_backend.get_active_llm_backend().model_name
|
||||
elif family == "tts":
|
||||
if active in {"omnivoice", "omnivoice-subprocess"}:
|
||||
from services.model_manager import resolve_omnivoice_checkpoint
|
||||
model = resolve_omnivoice_checkpoint()
|
||||
elif active == "mlx-audio":
|
||||
from core import prefs
|
||||
cls = tts_backend.MLXAudioBackend
|
||||
key = prefs.resolve("mlx_audio_model_id", env="OMNIVOICE_MLX_AUDIO_MODEL", default=cls.DEFAULT_MODEL_KEY)
|
||||
model = cls.CURATED_MODELS.get(key, key)
|
||||
else:
|
||||
instance = getattr(tts_backend, "_active_instance", None)
|
||||
if instance is not None and getattr(tts_backend, "_active_instance_id", None) == active:
|
||||
model = instance.model_identity()
|
||||
backends = public_backends(module.list_backends())
|
||||
if family == "tts":
|
||||
from services import settings_store
|
||||
|
||||
for backend in backends:
|
||||
engine_id = backend.get("id")
|
||||
if engine_id in LICENSE_GATED_ENGINES:
|
||||
backend["license_required"] = True
|
||||
try:
|
||||
backend["license_accepted"] = settings_store.get_license_accepted(engine_id)
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"Could not read license acceptance for %s",
|
||||
engine_id,
|
||||
exc_info=True,
|
||||
)
|
||||
backend["license_accepted"] = False
|
||||
return {
|
||||
# MPS hides the explicit compatibility row, so legacy configs report
|
||||
# the visible canonical equivalent as active to picker consumers.
|
||||
"active": _catalogue_active_id(family, module),
|
||||
"active": active,
|
||||
"active_model": model,
|
||||
"env_override": bool(os.environ.get(f"OMNIVOICE_{family.upper()}_BACKEND")),
|
||||
"backends": public_backends(module.list_backends()),
|
||||
"backends": backends,
|
||||
}
|
||||
|
||||
def _is_hf_repo_id(value: str) -> bool:
|
||||
@@ -126,6 +165,90 @@ def list_effects_presets():
|
||||
return {"presets": list_effect_presets()}
|
||||
|
||||
|
||||
@router.get("/engines/diarisation")
|
||||
def diarisation_status():
|
||||
"""Describe the selected local diarisation runtime without loading weights."""
|
||||
from services.diarization_runtime import (
|
||||
PYANNOTE,
|
||||
SORTFORMER,
|
||||
selected_backend,
|
||||
sortformer_status,
|
||||
)
|
||||
|
||||
selected = selected_backend()
|
||||
native = selected == SORTFORMER
|
||||
options = []
|
||||
from api.routers.setup.models import KNOWN_MODELS, cache_is_complete, is_cached
|
||||
pyannote_repo = "pyannote/speaker-diarization-3.1"
|
||||
spec = next(model for model in KNOWN_MODELS if model["repo_id"] == pyannote_repo)
|
||||
pyannote_installed = is_cached(pyannote_repo) and cache_is_complete(spec)
|
||||
pyannote_reason = None if pyannote_installed else "Install the pyannote model bundle"
|
||||
options.append({
|
||||
"id": PYANNOTE,
|
||||
"label": "pyannote 3.1",
|
||||
"model": pyannote_repo,
|
||||
"installed": pyannote_installed,
|
||||
"reason": pyannote_reason,
|
||||
})
|
||||
|
||||
native_status = sortformer_status()
|
||||
native_installed = native_status["installed"]
|
||||
native_model = native_status["model"]
|
||||
native_reason = native_status["reason"]
|
||||
options.append({
|
||||
"id": SORTFORMER,
|
||||
"label": "Sortformer v1 (audio.cpp)",
|
||||
"model": native_model,
|
||||
"model_installed": native_status["model_installed"],
|
||||
"runtime_installed": native_status["runtime_installed"],
|
||||
"installed": native_installed,
|
||||
"reason": native_reason,
|
||||
})
|
||||
|
||||
if native:
|
||||
from services.diarization_native import is_running
|
||||
return {"active": SORTFORMER, "label": "Sortformer v1 (audio.cpp)",
|
||||
"model": native_model, "installed": native_installed, "loaded": False,
|
||||
"model_installed": native_status["model_installed"],
|
||||
"runtime_installed": native_status["runtime_installed"],
|
||||
"busy": is_running(), "reason": native_reason, "options": options}
|
||||
from services import model_manager
|
||||
return {"active": PYANNOTE, "label": "pyannote 3.1", "model": pyannote_repo,
|
||||
"installed": pyannote_installed,
|
||||
"loaded": model_manager._diar_pipeline is not None, "reason": pyannote_reason,
|
||||
"options": options}
|
||||
|
||||
|
||||
class DiarisationSelection(BaseModel):
|
||||
engine_id: str
|
||||
|
||||
|
||||
@router.post("/engines/diarisation/select", dependencies=[Depends(require_admin)])
|
||||
def select_diarisation_engine(request: DiarisationSelection):
|
||||
"""Persist an installed diarisation runtime; environment overrides still win."""
|
||||
from services.diarization_runtime import SORTFORMER, select_backend, selected_backend
|
||||
|
||||
status = diarisation_status()
|
||||
option = next(
|
||||
(item for item in status["options"] if item["id"] == request.engine_id),
|
||||
None,
|
||||
)
|
||||
if option is None:
|
||||
raise HTTPException(404, "Unknown diarisation engine")
|
||||
if not option["installed"]:
|
||||
raise HTTPException(409, option.get("reason") or "Install this diarisation engine first")
|
||||
select_backend(request.engine_id)
|
||||
if request.engine_id == SORTFORMER:
|
||||
# Native Sortformer is stateless. Release a previously loaded pyannote
|
||||
# pipeline so Engine Ready cannot hide stale accelerator memory.
|
||||
from services import model_manager
|
||||
model_manager.unload_diarization_pipeline()
|
||||
return {
|
||||
"active": selected_backend(),
|
||||
"env_override": bool(os.environ.get("OMNIVOICE_DIARIZATION_BACKEND")),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/engines/translation")
|
||||
def list_translation_engines():
|
||||
"""Translation engines with per-engine pip-package availability.
|
||||
@@ -136,6 +259,7 @@ def list_translation_engines():
|
||||
an engine whose Python dependency isn't importable yet.
|
||||
"""
|
||||
return {
|
||||
"active": prefs.get("translation_backend", "argos"),
|
||||
"engines": [
|
||||
{**entry, "availability_reason": public_unavailability(entry.get("availability_reason"))}
|
||||
for entry in translation_engines.list_engines()
|
||||
@@ -144,6 +268,69 @@ def list_translation_engines():
|
||||
}
|
||||
|
||||
|
||||
class TranslationSelection(BaseModel):
|
||||
engine_id: str
|
||||
|
||||
|
||||
class ArgosPackRequest(BaseModel):
|
||||
source_lang: str | None = None
|
||||
target_langs: list[str] = Field(min_length=1, max_length=32)
|
||||
job_id: str | None = None
|
||||
|
||||
|
||||
def _argos_pack_request(request: ArgosPackRequest) -> tuple[str, list[str]]:
|
||||
source = request.source_lang
|
||||
if not source and request.job_id:
|
||||
from api.routers.dub_core import _get_job
|
||||
|
||||
job = _get_job(request.job_id)
|
||||
source = job.get("source_lang") if job else None
|
||||
if not source:
|
||||
raise HTTPException(422, "Transcribe the source before installing its language pack")
|
||||
return source, request.target_langs
|
||||
|
||||
|
||||
@router.post(
|
||||
"/engines/translation/argos/packs/status",
|
||||
dependencies=[Depends(require_admin)],
|
||||
)
|
||||
def argos_pack_status(request: ArgosPackRequest):
|
||||
source, targets = _argos_pack_request(request)
|
||||
try:
|
||||
return translation_engines.argos_pack_status(source, targets)
|
||||
except (ImportError, ValueError) as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.post(
|
||||
"/engines/translation/argos/packs/install",
|
||||
dependencies=[Depends(require_admin)],
|
||||
)
|
||||
async def install_argos_packs(request: ArgosPackRequest):
|
||||
source, targets = _argos_pack_request(request)
|
||||
try:
|
||||
return await asyncio.to_thread(
|
||||
translation_engines.install_argos_packs,
|
||||
source,
|
||||
targets,
|
||||
)
|
||||
except (ImportError, ValueError) as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.post("/engines/translation/select", dependencies=[Depends(require_admin)])
|
||||
def select_translation_engine(request: TranslationSelection):
|
||||
entry = translation_engines.get_engine(request.engine_id)
|
||||
if not entry:
|
||||
raise HTTPException(404, "Unknown translation engine")
|
||||
if not translation_engines.is_installed(request.engine_id):
|
||||
raise HTTPException(409, "Install this translation engine before selecting it")
|
||||
if not translation_engines.is_ready(request.engine_id):
|
||||
raise HTTPException(409, "Configure this translation provider before selecting it")
|
||||
prefs.set_("translation_backend", request.engine_id)
|
||||
return {"active": request.engine_id}
|
||||
|
||||
|
||||
@router.post(
|
||||
"/engines/translation/{engine_id}/install",
|
||||
dependencies=[Depends(require_admin)],
|
||||
@@ -215,6 +402,32 @@ async def uninstall_translation_engine(engine_id: str):
|
||||
return {"status": "uninstalled", "engine": engine_id, "package": pkg, "log_tail": out[-800:]}
|
||||
|
||||
|
||||
# ── Checksummed native audio.cpp runtime install ───────────────────────────
|
||||
|
||||
|
||||
@router.get(
|
||||
"/engines/audiocpp/runtime/install/status",
|
||||
dependencies=[Depends(require_admin)],
|
||||
)
|
||||
def audiocpp_runtime_install_status():
|
||||
from services import audiocpp_runtime_install
|
||||
|
||||
return audiocpp_runtime_install.status()
|
||||
|
||||
|
||||
@router.post(
|
||||
"/engines/audiocpp/runtime/install",
|
||||
dependencies=[Depends(require_admin), Depends(require_desktop)],
|
||||
)
|
||||
def install_audiocpp_runtime():
|
||||
from services import audiocpp_runtime_install
|
||||
|
||||
try:
|
||||
return audiocpp_runtime_install.start_install()
|
||||
except RuntimeError as exc:
|
||||
raise HTTPException(status_code=409, detail=str(exc)) from exc
|
||||
|
||||
|
||||
# ── One-click sidecar-engine install (IndexTTS-2 & friends) ────────────────
|
||||
#
|
||||
# Sidecar engines (dedicated venv + source checkout + weights, isolated from
|
||||
@@ -693,6 +906,23 @@ def select_engine(req: SelectEngineRequest):
|
||||
"Hugging Face repo ID like 'owner/name'.",
|
||||
)
|
||||
prefs.set_("mlx_audio_model_id", req.model_id)
|
||||
if req.family == "asr" and req.model_id is not None:
|
||||
if req.backend_id not in {"faster-whisper", "faster-whisper-isolated"}:
|
||||
raise HTTPException(400, "This ASR engine does not accept a CTranslate2 model")
|
||||
from api.routers.setup.models import KNOWN_MODELS, is_cached
|
||||
|
||||
model = next((item for item in KNOWN_MODELS if item["repo_id"] == req.model_id), None)
|
||||
compatible = req.model_id.startswith("Systran/faster-") or req.model_id == (
|
||||
"deepdml/faster-whisper-large-v3-turbo-ct2"
|
||||
)
|
||||
if model is None or str(model.get("role", "")).lower() != "asr" or not compatible:
|
||||
raise HTTPException(400, "This model is not compatible with Faster-Whisper")
|
||||
if not is_cached(req.model_id):
|
||||
raise HTTPException(409, "Install this ASR model before selecting it")
|
||||
try:
|
||||
asr_backend.select_faster_whisper_model(req.model_id)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(409, str(exc)) from exc
|
||||
prefs.set_(pref_key, req.backend_id)
|
||||
return {
|
||||
"family": req.family,
|
||||
|
||||
@@ -1366,12 +1366,12 @@ async def generate_speech(
|
||||
ref_text: Optional[str] = Form(None),
|
||||
instruct: Optional[str] = Form(None),
|
||||
duration: Optional[float] = Form(None),
|
||||
num_step: int = Form(16),
|
||||
num_step: Optional[int] = Form(None),
|
||||
guidance_scale: float = Form(2.0),
|
||||
speed: float = Form(1.0),
|
||||
t_shift: Optional[float] = Form(None),
|
||||
denoise: bool = Form(True),
|
||||
postprocess_output: bool = Form(True),
|
||||
postprocess_output: Optional[bool] = Form(None),
|
||||
layer_penalty_factor: Optional[float] = Form(None),
|
||||
position_temperature: Optional[float] = Form(None),
|
||||
class_temperature: Optional[float] = Form(None),
|
||||
@@ -1417,6 +1417,13 @@ async def generate_speech(
|
||||
)
|
||||
|
||||
engine_id = engine or active_backend_id()
|
||||
from services.performance_profiles import tts_defaults
|
||||
|
||||
sampling_defaults = tts_defaults(engine_id)
|
||||
if num_step is None:
|
||||
num_step = sampling_defaults.get("num_step", 16)
|
||||
if postprocess_output is None:
|
||||
postprocess_output = sampling_defaults.get("postprocess_output", True)
|
||||
try:
|
||||
backend_cls = get_backend_class(engine_id)
|
||||
except ValueError:
|
||||
|
||||
@@ -0,0 +1,182 @@
|
||||
"""Keyless portrait search with bounded, normalized, safe thumbnails."""
|
||||
import asyncio
|
||||
import base64
|
||||
import html
|
||||
import re
|
||||
from html.parser import HTMLParser
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
|
||||
from core.profile_images import MAX_IMAGE_BYTES, normalize_portrait
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def trusted_thumbnail(url: str) -> bool:
|
||||
try:
|
||||
parsed = urlsplit(url)
|
||||
google = parsed.hostname in {
|
||||
f"encrypted-tbn{i}.gstatic.com" for i in range(4)
|
||||
}
|
||||
openverse = (
|
||||
parsed.hostname == "api.openverse.org"
|
||||
and re.fullmatch(r"/v1/images/[0-9a-f-]+/thumb/?", parsed.path) is not None
|
||||
)
|
||||
return (
|
||||
parsed.scheme == "https"
|
||||
and not parsed.username
|
||||
and not parsed.password
|
||||
and parsed.port in (None, 443)
|
||||
and (google or openverse)
|
||||
)
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def google_thumbnails(document: str) -> list[tuple[str, str]]:
|
||||
"""Extract result thumbnails only; never download third-party originals."""
|
||||
results = []
|
||||
seen = set()
|
||||
|
||||
def add(title, source):
|
||||
if not isinstance(source, str) or source in seen:
|
||||
return
|
||||
if not (trusted_thumbnail(source) or source.startswith("data:image/jpeg;base64,")):
|
||||
return
|
||||
seen.add(source)
|
||||
if len(results) < 20:
|
||||
results.append((title or "", source))
|
||||
|
||||
class Images(HTMLParser):
|
||||
def handle_starttag(self, tag, attrs):
|
||||
if tag == "img":
|
||||
values = dict(attrs)
|
||||
add(values.get("alt"), values.get("src") or values.get("data-src"))
|
||||
|
||||
Images().feed(document)
|
||||
# Google also assigns thumbnails from script strings after rendering.
|
||||
decoded = html.unescape(document)
|
||||
for escaped, literal in ((r"\u003d", "="), (r"\u0026", "&"), (r"\/", "/")):
|
||||
decoded = decoded.replace(escaped, literal)
|
||||
for match in re.finditer(r'https://encrypted-tbn[0-3]\.gstatic\.com/[^\s"\'<>\\]+|data:image/jpeg;base64,[A-Za-z0-9+/=]+', decoded):
|
||||
add("", match.group())
|
||||
return results
|
||||
|
||||
|
||||
async def openverse_thumbnails(client: httpx.AsyncClient, name: str) -> list[tuple[str, str]]:
|
||||
"""Public-domain/CC portrait fallback when Google returns its JS-only shell.
|
||||
|
||||
Openverse requires no user credential, excludes sensitive results by
|
||||
default, and can restrict results to licenses that allow modification and
|
||||
commercial use. We still fetch only its own thumbnail proxy.
|
||||
"""
|
||||
response = await client.get(
|
||||
"https://api.openverse.org/v1/images/",
|
||||
headers={
|
||||
"User-Agent": "VoiceStudio/0.5 (+https://github.com/debpalash/VoiceStudio)",
|
||||
"Accept": "application/json",
|
||||
},
|
||||
params={
|
||||
"q": name,
|
||||
"page_size": 20,
|
||||
"mature": "false",
|
||||
"extension": "jpg,png",
|
||||
"aspect_ratio": "square",
|
||||
"license_type": "commercial,modification",
|
||||
},
|
||||
)
|
||||
response.raise_for_status()
|
||||
if len(response.content) > 4 * 1024 * 1024:
|
||||
raise ValueError("Search response too large")
|
||||
payload = response.json()
|
||||
rows = payload.get("results") if isinstance(payload, dict) else None
|
||||
if not isinstance(rows, list):
|
||||
raise ValueError("Invalid search response")
|
||||
results = []
|
||||
seen = set()
|
||||
for row in rows:
|
||||
if not isinstance(row, dict):
|
||||
continue
|
||||
source = row.get("thumbnail")
|
||||
if not isinstance(source, str) or source in seen or not trusted_thumbnail(source):
|
||||
continue
|
||||
seen.add(source)
|
||||
title = str(row.get("title") or name)
|
||||
creator = str(row.get("creator") or "").strip()
|
||||
license_name = str(row.get("license") or "").upper()
|
||||
credit = " · ".join(value for value in (creator, license_name) if value)
|
||||
results.append((f"{title} — {credit}" if credit else title, source))
|
||||
return results
|
||||
|
||||
|
||||
@router.get("/profile-images/search")
|
||||
async def search_profile_images(name: str = Query(min_length=1, max_length=100)):
|
||||
if not name.strip():
|
||||
raise HTTPException(422, detail={"code": "image_search_failed"})
|
||||
async with httpx.AsyncClient(
|
||||
timeout=15,
|
||||
follow_redirects=False,
|
||||
headers={
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
||||
"AppleWebKit/537.36 Chrome/140.0.0.0 Safari/537.36",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
},
|
||||
) as client:
|
||||
results: list[tuple[str, str]] = []
|
||||
try:
|
||||
async with client.stream("GET", "https://www.google.com/search", params={
|
||||
"q": name.strip(), "udm": "2", "safe": "active", "tbs": "ift:jpg",
|
||||
}) as response:
|
||||
response.raise_for_status()
|
||||
document = bytearray()
|
||||
async for chunk in response.aiter_bytes():
|
||||
document.extend(chunk)
|
||||
if len(document) > 4 * 1024 * 1024:
|
||||
raise ValueError("Search page too large")
|
||||
page = document.decode("utf-8", errors="replace")
|
||||
results = google_thumbnails(page)
|
||||
except (httpx.HTTPError, ValueError, TypeError):
|
||||
# Search providers can change their anonymous HTML or reject a
|
||||
# non-browser request. The fallback below keeps this explicit,
|
||||
# user-triggered feature useful without requiring credentials.
|
||||
results = []
|
||||
|
||||
if not results:
|
||||
try:
|
||||
results = await openverse_thumbnails(client, name.strip())
|
||||
except (httpx.HTTPError, ValueError, TypeError):
|
||||
results = []
|
||||
if not results:
|
||||
raise HTTPException(502, detail={"code": "image_search_failed"})
|
||||
|
||||
async def thumbnail(title, url):
|
||||
try:
|
||||
if url.startswith("data:image/jpeg;base64,"):
|
||||
encoded = url.partition(",")[2]
|
||||
if len(encoded) > MAX_IMAGE_BYTES * 4 // 3 + 4:
|
||||
return None
|
||||
data = base64.b64decode(encoded, validate=True)
|
||||
else:
|
||||
if not trusted_thumbnail(url):
|
||||
return None
|
||||
async with client.stream("GET", url) as image:
|
||||
image.raise_for_status()
|
||||
data = bytearray()
|
||||
async for chunk in image.aiter_bytes():
|
||||
data.extend(chunk)
|
||||
if len(data) > MAX_IMAGE_BYTES:
|
||||
return None
|
||||
normalized = await asyncio.to_thread(normalize_portrait, bytes(data))
|
||||
return {"title": title[:200], "data": base64.b64encode(normalized).decode("ascii")}
|
||||
except (httpx.HTTPError, HTTPException, ValueError, TypeError):
|
||||
return None
|
||||
|
||||
images = []
|
||||
for start in range(0, len(results), 5):
|
||||
batch = await asyncio.gather(*(thumbnail(*result) for result in results[start:start + 5]))
|
||||
images.extend(image for image in batch if image)
|
||||
if len(images) >= 5:
|
||||
break
|
||||
return {"images": images[:5]}
|
||||
@@ -1,3 +1,5 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import uuid
|
||||
@@ -14,8 +16,21 @@ from core import event_bus
|
||||
from core.personalities import get_personalities
|
||||
from omnivoice.utils.voice_design import heal_design_instruct, sanitize_instruct
|
||||
from core.path_security import UnsafePath, resolve_within
|
||||
from core.profile_images import MAX_IMAGE_BYTES, normalize_portrait
|
||||
from starlette.datastructures import UploadFile as StarletteUploadFile
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.profiles")
|
||||
|
||||
|
||||
def _profile_record(row):
|
||||
result = dict(row)
|
||||
image_path = _voices_path(f"{result['id']}.portrait.jpg")
|
||||
result["image_url"] = (
|
||||
f"/profiles/{result['id']}/image?v={os.stat(image_path).st_mtime_ns}"
|
||||
if image_path and os.path.isfile(image_path) else None
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
class ProfileUpdate(BaseModel):
|
||||
@@ -35,7 +50,7 @@ def list_personalities():
|
||||
def list_profiles():
|
||||
with db_conn() as conn:
|
||||
rows = conn.execute("SELECT * FROM voice_profiles ORDER BY created_at DESC").fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
return [_profile_record(r) for r in rows]
|
||||
|
||||
_DESIGN_SEED = 42 # deterministic sample render, same as archetype previews
|
||||
|
||||
@@ -51,6 +66,7 @@ async def create_profile(
|
||||
personality: str = Form(""),
|
||||
kind: str = Form("clone"),
|
||||
vd_states: Optional[str] = Form(None),
|
||||
image: Optional[UploadFile] = File(None),
|
||||
):
|
||||
"""Create a voice profile (spec: docs/specs/voice-studio-unification.md §5).
|
||||
|
||||
@@ -60,6 +76,9 @@ async def create_profile(
|
||||
archetype materialization) and stores it as the profile's
|
||||
reference so the voice identity is stable across runs.
|
||||
"""
|
||||
name = name.strip()
|
||||
if not name:
|
||||
raise HTTPException(status_code=400, detail="A voice profile needs a name.")
|
||||
if kind not in ("clone", "design"):
|
||||
raise HTTPException(status_code=422, detail="kind must be 'clone' or 'design'")
|
||||
if kind == "clone" and ref_audio is None:
|
||||
@@ -106,13 +125,38 @@ async def create_profile(
|
||||
instruct = sanitize_instruct(instruct)
|
||||
|
||||
profile_id = str(uuid.uuid4())[:8]
|
||||
portrait = None
|
||||
if isinstance(image, StarletteUploadFile):
|
||||
portrait = normalize_portrait(await image.read(MAX_IMAGE_BYTES + 1))
|
||||
portrait_path = os.path.join(VOICES_DIR, f"{profile_id}.portrait.jpg")
|
||||
|
||||
if kind == "clone":
|
||||
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
|
||||
audio_filename = f"{profile_id}{ext}"
|
||||
audio_path = os.path.join(VOICES_DIR, audio_filename)
|
||||
# Storage can be removed after startup; recover before persisting uploads.
|
||||
os.makedirs(VOICES_DIR, exist_ok=True)
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(await ref_audio.read())
|
||||
# A matching transcript defines the boundary between the reference and
|
||||
# the requested line. Saving a blank transcript and waiting until the
|
||||
# first generation made that first take depend on the TTS model's
|
||||
# internal ASR fallback; short lines could then start with stray words
|
||||
# from the reference. Resolve it while the profile is being created so
|
||||
# every synthesis, including the first, uses stable conditioning. This
|
||||
# remains best-effort and local-only: transcribe_reference considers
|
||||
# only already-installed ASR/dictation models.
|
||||
if not ref_text.strip():
|
||||
try:
|
||||
from services.asr_backend import transcribe_reference
|
||||
|
||||
ref_text = (
|
||||
await asyncio.to_thread(transcribe_reference, audio_path) or ""
|
||||
).strip()
|
||||
except Exception as exc: # noqa: BLE001 — profile save remains usable
|
||||
logger.warning(
|
||||
"reference transcription during profile save failed: %s", exc
|
||||
)
|
||||
used_seed = seed
|
||||
else:
|
||||
# Saving a design profile is a pure persistence operation — it must not
|
||||
@@ -153,6 +197,10 @@ async def create_profile(
|
||||
used_seed = seed if seed is not None else _DESIGN_SEED
|
||||
|
||||
try:
|
||||
if portrait:
|
||||
os.makedirs(VOICES_DIR, exist_ok=True)
|
||||
with open(portrait_path, "wb") as out:
|
||||
out.write(portrait)
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, "
|
||||
@@ -162,12 +210,14 @@ async def create_profile(
|
||||
used_seed, personality, kind, vd_states, time.time())
|
||||
)
|
||||
except Exception:
|
||||
if os.path.exists(portrait_path):
|
||||
os.remove(portrait_path)
|
||||
# Clean up orphaned audio file if DB insert fails
|
||||
if os.path.exists(audio_path):
|
||||
os.remove(audio_path)
|
||||
raise
|
||||
event_bus.emit("profiles", {"action": "created", "id": profile_id})
|
||||
return {"id": profile_id, "name": name, "kind": kind}
|
||||
return get_profile(profile_id)
|
||||
|
||||
@router.get("/profiles/{profile_id}")
|
||||
def get_profile(profile_id: str):
|
||||
@@ -181,14 +231,47 @@ def get_profile(profile_id: str):
|
||||
status_code=404,
|
||||
detail="That voice profile doesn't exist. It may have been deleted from another tab.",
|
||||
)
|
||||
return dict(row)
|
||||
return _profile_record(row)
|
||||
|
||||
|
||||
@router.get("/profiles/{profile_id}/image")
|
||||
def get_profile_image(profile_id: str):
|
||||
get_profile(profile_id)
|
||||
path = _voices_path(f"{profile_id}.portrait.jpg")
|
||||
if not path or not os.path.isfile(path):
|
||||
raise HTTPException(404, "Profile image not found")
|
||||
return FileResponse(path, media_type="image/jpeg", headers={"Cache-Control": "no-cache"})
|
||||
|
||||
|
||||
@router.put("/profiles/{profile_id}/image")
|
||||
async def update_profile_image(profile_id: str, image: UploadFile = File(...)):
|
||||
get_profile(profile_id)
|
||||
path = _voices_path(f"{profile_id}.portrait.jpg")
|
||||
if path is None:
|
||||
raise HTTPException(404, "Profile not found")
|
||||
portrait = normalize_portrait(await image.read(MAX_IMAGE_BYTES + 1))
|
||||
os.makedirs(VOICES_DIR, exist_ok=True)
|
||||
with open(path, "wb") as out:
|
||||
out.write(portrait)
|
||||
event_bus.emit("profiles", {"action": "updated", "id": profile_id})
|
||||
return get_profile(profile_id)
|
||||
|
||||
|
||||
@router.put("/profiles/{profile_id}")
|
||||
def update_profile(profile_id: str, patch: ProfileUpdate):
|
||||
"""Partial update — only fields set on the payload are changed."""
|
||||
with db_conn() as conn:
|
||||
existing = conn.execute(
|
||||
"SELECT kind FROM voice_profiles WHERE id = ?", (profile_id,),
|
||||
).fetchone()
|
||||
if not existing:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail="That voice profile doesn't exist. It may have been deleted from another tab.",
|
||||
)
|
||||
fields = []
|
||||
params = []
|
||||
edited_instruct = None
|
||||
for col in ("name", "ref_text", "instruct", "language", "personality"):
|
||||
val = getattr(patch, col)
|
||||
if val is None:
|
||||
@@ -199,12 +282,22 @@ def update_profile(profile_id: str, patch: ProfileUpdate):
|
||||
# Never let an edit persist a validator-rejecting instruct (prose /
|
||||
# "[object Object]"); keep only whitelist tags (#550 #571 #594 #596).
|
||||
val = sanitize_instruct(val)
|
||||
edited_instruct = val
|
||||
fields.append(f"{col} = ?")
|
||||
params.append(val.strip() if col in ("name", "language") else val)
|
||||
if edited_instruct is not None and existing["kind"] == "design":
|
||||
# Keep the complete recipe synchronized with the editable instruct.
|
||||
# Otherwise clients restore a stale vd_states snapshot and a successful
|
||||
# style edit has no effect on the next generation.
|
||||
import json
|
||||
from core.describe_voice import instruct_to_vd_states
|
||||
|
||||
fields.append("vd_states = ?")
|
||||
params.append(json.dumps(instruct_to_vd_states(edited_instruct)))
|
||||
if not fields:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="PUT /profiles/{id} body contained no editable fields. Include at least one of: name, language, instruct, description.",
|
||||
detail="PUT /profiles/{id} body contained no editable fields. Include at least one of: name, language, ref_text, instruct, personality.",
|
||||
)
|
||||
params.append(profile_id)
|
||||
with db_conn() as conn:
|
||||
@@ -221,7 +314,7 @@ def update_profile(profile_id: str, patch: ProfileUpdate):
|
||||
"SELECT * FROM voice_profiles WHERE id = ?", (profile_id,),
|
||||
).fetchone()
|
||||
event_bus.emit("profiles", {"action": "updated", "id": profile_id})
|
||||
return dict(row)
|
||||
return _profile_record(row)
|
||||
|
||||
|
||||
@router.get("/profiles/{profile_id}/usage")
|
||||
@@ -252,8 +345,14 @@ def get_profile_usage(profile_id: str):
|
||||
state = json.loads(r["state_json"] or "{}")
|
||||
except Exception:
|
||||
continue
|
||||
segs = state.get("segments") or []
|
||||
n = sum(1 for s in segs if s.get("profile_id") == profile_id)
|
||||
if not isinstance(state, dict):
|
||||
continue
|
||||
# Current desktop snapshots use dubSegments. An explicit empty list
|
||||
# supersedes legacy segments retained in an older snapshot.
|
||||
segs = state.get("dubSegments", state.get("segments", []))
|
||||
if not isinstance(segs, list):
|
||||
continue
|
||||
n = sum(1 for s in segs if isinstance(s, dict) and s.get("profile_id") == profile_id)
|
||||
if n:
|
||||
project_hits.append({
|
||||
"project_id": r["id"],
|
||||
@@ -539,6 +638,9 @@ def delete_profile(profile_id: str):
|
||||
path = _voices_path(row[col])
|
||||
if path and os.path.exists(path):
|
||||
os.remove(path)
|
||||
portrait_path = _voices_path(f"{profile_id}.portrait.jpg")
|
||||
if portrait_path and os.path.isfile(portrait_path):
|
||||
os.remove(portrait_path)
|
||||
# Prevent FOREIGN KEY constraint failure
|
||||
conn.execute("UPDATE generation_history SET profile_id = NULL WHERE profile_id=?", (profile_id,))
|
||||
conn.execute("DELETE FROM voice_profiles WHERE id=?", (profile_id,))
|
||||
|
||||
@@ -20,6 +20,7 @@ from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from core.logging_utils import log_safe
|
||||
from core.engine_licenses import LICENSE_GATED_ENGINES
|
||||
from api.dependencies import require_admin, require_admin_action
|
||||
|
||||
logger = logging.getLogger("omnivoice.api.settings")
|
||||
@@ -96,9 +97,66 @@ def get_hf_token_state(fresh: bool = Query(False)):
|
||||
|
||||
_TORCH_COMPILE_KEY = "perf.torch_compile_disabled"
|
||||
|
||||
from services.performance_profiles import (
|
||||
_PERFORMANCE_PROFILE_KEY, _PERFORMANCE_TIERS, _PERFORMANCE_FAMILIES,
|
||||
activate_performance_tier,
|
||||
profile_state as _performance_profile_state,
|
||||
)
|
||||
|
||||
|
||||
class _PerformanceProfileBody(BaseModel):
|
||||
tier: str = Field(..., description="fast | balanced | quality | max")
|
||||
family: str | None = Field(None, description="Engine family, or null to set the global tier")
|
||||
|
||||
|
||||
|
||||
|
||||
@router.get("/performance-profile")
|
||||
def get_performance_profile():
|
||||
"""Return the global speed/quality preference and per-engine overrides."""
|
||||
return _performance_profile_state()
|
||||
|
||||
|
||||
@router.put("/performance-profile")
|
||||
def set_performance_profile(body: _PerformanceProfileBody):
|
||||
"""Persist a performance preference and apply installed Max-capacity picks."""
|
||||
from core import prefs
|
||||
|
||||
tier = body.tier.strip().lower()
|
||||
if tier not in _PERFORMANCE_TIERS:
|
||||
raise HTTPException(status_code=400, detail="Unknown performance tier")
|
||||
family = body.family.strip().lower() if body.family else None
|
||||
if family is not None and family not in _PERFORMANCE_FAMILIES:
|
||||
raise HTTPException(status_code=400, detail="Unknown engine family")
|
||||
state = _performance_profile_state()
|
||||
applicable = state["applicable_families"]
|
||||
if (family is not None and family not in applicable) or (family is None and not applicable):
|
||||
raise HTTPException(status_code=409, detail="The selected engines do not support this performance preset")
|
||||
from core import job_store
|
||||
from api.routers.batch import list_batch_jobs
|
||||
if job_store.list_jobs(status="active", limit=1) or list_batch_jobs(status="active", limit=1):
|
||||
raise HTTPException(status_code=409, detail="Wait for queued or running jobs to finish before changing performance presets")
|
||||
try:
|
||||
if family is None:
|
||||
# One atomic write clears family overrides together with the global
|
||||
# choice, so a crash cannot leave half of a global change persisted.
|
||||
prefs.update_mapping(_PERFORMANCE_PROFILE_KEY, {"global": tier}, replace=True)
|
||||
else:
|
||||
prefs.update_mapping(_PERFORMANCE_PROFILE_KEY, {family: tier})
|
||||
except Exception:
|
||||
logger.exception("set_performance_profile failed")
|
||||
raise HTTPException(status_code=500, detail="Failed to persist performance profile")
|
||||
activations = activate_performance_tier(tier, family)
|
||||
result = _performance_profile_state()
|
||||
if activations:
|
||||
result["runtime_activations"] = activations
|
||||
if tier == "max":
|
||||
result["capacity_activations"] = activations
|
||||
return result
|
||||
|
||||
|
||||
class _TorchCompileBody(BaseModel):
|
||||
enabled: bool = Field(..., description="True to set TORCH_COMPILE_DISABLE=1 on engine subprocesses")
|
||||
enabled: bool = Field(..., description="True to disable torch.compile (eager mode) for the engine")
|
||||
|
||||
|
||||
def _torch_compile_state() -> dict:
|
||||
@@ -112,15 +170,21 @@ def _torch_compile_state() -> dict:
|
||||
@router.get("/perf/torch-compile-disabled")
|
||||
def get_torch_compile_disabled():
|
||||
"""Return the current torch.compile-disabled toggle + the runtime platform.
|
||||
UI uses the platform to render the toggle disabled (with an explainer)
|
||||
on non-Windows hosts, since the OOM is Windows-specific (issue #65)."""
|
||||
|
||||
`platform` is still reported (clients may show it), but since #2135 the
|
||||
toggle is live on every host: it used to be rendered disabled off Windows
|
||||
on the assumption that only #65's Windows OOM needed it, which left the
|
||||
Linux/CUDA reporter of #2135 with no way to switch off the compile that
|
||||
was killing their backend.
|
||||
"""
|
||||
return _torch_compile_state()
|
||||
|
||||
|
||||
@router.put("/perf/torch-compile-disabled")
|
||||
def set_torch_compile_disabled(body: _TorchCompileBody):
|
||||
"""Persist the toggle. Honoured by `services.engine_env.build_engine_env()`
|
||||
which injects TORCH_COMPILE_DISABLE=1 on Windows when enabled."""
|
||||
(subprocess engines) and `services.engine_env.should_torch_compile()`
|
||||
(in-process), on every platform since #2135."""
|
||||
from services import settings_store
|
||||
|
||||
try:
|
||||
@@ -653,7 +717,7 @@ def set_llm_skill(skill_id: str, body: _LLMSkillBody):
|
||||
#: Engines that have an in-tree acceptance dialog. Adding a new engine
|
||||
#: here means adding a corresponding frontend dialog + a license URLs
|
||||
#: dict in its constants module. Until that, the API refuses the write.
|
||||
_LICENSE_ALLOWED_ENGINES: frozenset[str] = frozenset({"supertonic3", "pockettts"})
|
||||
_LICENSE_ALLOWED_ENGINES = LICENSE_GATED_ENGINES
|
||||
|
||||
|
||||
class _LicenseAcceptBody(BaseModel):
|
||||
|
||||
@@ -14,6 +14,7 @@ import logging
|
||||
import os
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
@@ -31,7 +32,7 @@ from utils import download_aggregator
|
||||
from .models import ( # noqa: F401
|
||||
KNOWN_MODELS,
|
||||
invalidate_cache,
|
||||
snapshot_has_weights,
|
||||
snapshot_is_complete,
|
||||
disk_space_error,
|
||||
_MIN_WEIGHT_BYTES,
|
||||
_WEIGHT_FLOORS,
|
||||
@@ -42,6 +43,9 @@ router = APIRouter()
|
||||
|
||||
# Cooldown: prevent rapid re-install after a failure. Maps repo_id → last_fail_time.
|
||||
_install_cooldowns: dict[str, float] = {}
|
||||
# Last classified failure per repo. The SSE stream carries the same detail live;
|
||||
# retaining it here keeps recovery useful after navigation or renderer reconnect.
|
||||
_install_failures: dict[str, dict] = {}
|
||||
_COOLDOWN_SECS = 60.0
|
||||
# Evict cooldown entries older than this so the dict can't grow unbounded across
|
||||
# a long-lived process (MM2-06). Anything past the cooldown window is dead state.
|
||||
@@ -54,6 +58,14 @@ def _sweep_cooldowns(now: float) -> None:
|
||||
stale = [k for k, t in _install_cooldowns.items() if (now - t) > _COOLDOWN_TTL_SECS]
|
||||
for k in stale:
|
||||
_install_cooldowns.pop(k, None)
|
||||
_install_failures.pop(k, None)
|
||||
stale_failures = [
|
||||
repo_id
|
||||
for repo_id, failure in _install_failures.items()
|
||||
if (now - float(failure.get("failed_at") or 0)) > _COOLDOWN_TTL_SECS
|
||||
]
|
||||
for repo_id in stale_failures:
|
||||
_install_failures.pop(repo_id, None)
|
||||
|
||||
|
||||
def clear_install_cooldowns() -> None:
|
||||
@@ -63,6 +75,7 @@ def clear_install_cooldowns() -> None:
|
||||
very next action is "retry the failed download on the new mirror", and a
|
||||
429 there would dead-end the wizard's switch-and-retry flow."""
|
||||
_install_cooldowns.clear()
|
||||
_install_failures.clear()
|
||||
|
||||
# Repo_ids the user asked to cancel (FDL-11). Checked between retry attempts.
|
||||
# Note: a single in-flight snapshot_download/Xet fetch is not interruptible
|
||||
@@ -180,6 +193,28 @@ def _repo_cancelled(repo_id: str) -> bool:
|
||||
return repo_id in _cancelled
|
||||
|
||||
|
||||
def _create_cache_pointer(blob_path: str, pointer: str) -> None:
|
||||
"""Keep the canonical blob while exposing it from the snapshot tree.
|
||||
|
||||
huggingface_hub's ``new_blob=True`` fallback moves the blob into the
|
||||
snapshot when Windows symlinks are unavailable. The next model load then
|
||||
sees a missing blob and downloads the same multi-gigabyte weight again.
|
||||
NTFS hardlinks preserve both cache paths without doubling disk usage; other
|
||||
filesystems fall back to Hugging Face's copy/symlink path.
|
||||
"""
|
||||
from huggingface_hub.file_download import _create_symlink
|
||||
|
||||
if os.name == "nt":
|
||||
try:
|
||||
os.link(blob_path, pointer)
|
||||
return
|
||||
except FileExistsError:
|
||||
return
|
||||
except OSError:
|
||||
pass
|
||||
_create_symlink(blob_path, pointer, new_blob=False)
|
||||
|
||||
|
||||
def _segmented_snapshot(repo_id: str, *, endpoint: "str | None", revision: str) -> str:
|
||||
"""Fetch every file of a repo via the segmented downloader into the HF
|
||||
cache, mirroring hf_hub_download's blob+snapshot+refs layout so the result
|
||||
@@ -190,7 +225,7 @@ def _segmented_snapshot(repo_id: str, *, endpoint: "str | None", revision: str)
|
||||
import asyncio as _asyncio
|
||||
from huggingface_hub import HfApi, constants as _C
|
||||
from huggingface_hub.file_download import (
|
||||
hf_hub_url, get_hf_file_metadata, repo_folder_name, _create_symlink,
|
||||
hf_hub_url, get_hf_file_metadata, repo_folder_name,
|
||||
)
|
||||
from services.segmented_download import segmented_download
|
||||
from services.token_resolver import resolve as _resolve_token
|
||||
@@ -229,7 +264,7 @@ def _segmented_snapshot(repo_id: str, *, endpoint: "str | None", revision: str)
|
||||
cancel_check=lambda: _repo_cancelled(repo_id),
|
||||
))
|
||||
if not os.path.lexists(pointer):
|
||||
_create_symlink(blob_path, pointer, new_blob=True)
|
||||
_create_cache_pointer(blob_path, pointer)
|
||||
|
||||
# refs/main → commit so scan_cache_dir maps the revision correctly.
|
||||
ref_path = os.path.join(refs_dir, "main")
|
||||
@@ -279,10 +314,17 @@ def _validate_snapshot_has_weights(repo_id: str, snapshot_path: str) -> None:
|
||||
retry loop and the UI's re-download path can deal with it, instead of at
|
||||
first synthesis with an opaque transformers error.
|
||||
|
||||
Delegates the weight check to ``models.snapshot_has_weights`` (single source of
|
||||
the floors); only the install-time error message lives here."""
|
||||
if snapshot_has_weights(snapshot_path):
|
||||
Delegates to ``models.snapshot_is_complete`` so configuration-only pipeline
|
||||
repositories use their declared required files instead of a weight floor."""
|
||||
model = next((m for m in KNOWN_MODELS if m["repo_id"] == repo_id), {"repo_id": repo_id})
|
||||
if snapshot_is_complete(model, snapshot_path):
|
||||
return
|
||||
if model.get("config_only"):
|
||||
required = ", ".join(model.get("config_required_files") or ())
|
||||
raise OSError(f"{repo_id}: download is incomplete; required configuration files: {required}")
|
||||
if model.get("required_files"):
|
||||
required = ", ".join(model["required_files"])
|
||||
raise OSError(f"{repo_id}: required model files are missing or incomplete: {required}")
|
||||
biggest = 0
|
||||
try:
|
||||
for root, _dirs, files in os.walk(snapshot_path, followlinks=True):
|
||||
@@ -346,6 +388,65 @@ class InstallModelRequest(BaseModel):
|
||||
target: str | None = None
|
||||
|
||||
|
||||
@router.get("/models/install/status")
|
||||
def model_install_status():
|
||||
"""Read local and remote jobs after navigation without starting downloads."""
|
||||
from services import gpu_gateway # noqa: PLC0415
|
||||
|
||||
now = time.time()
|
||||
_sweep_cooldowns(now)
|
||||
with _active_installs_lock:
|
||||
active = tuple(_active_installs)
|
||||
jobs = []
|
||||
for repo_id in active:
|
||||
aggregate = download_aggregator._get(repo_id)
|
||||
jobs.append(
|
||||
{
|
||||
"repo_id": repo_id,
|
||||
"target": "local",
|
||||
"state": "downloading",
|
||||
**(aggregate.snapshot() if aggregate else {}),
|
||||
}
|
||||
)
|
||||
detailed = set()
|
||||
for repo_id, failure in tuple(_install_failures.items()):
|
||||
failed_at = float(failure.get("failed_at") or 0)
|
||||
if repo_id in active or now - failed_at >= _COOLDOWN_SECS:
|
||||
continue
|
||||
detailed.add(repo_id)
|
||||
cooldown_at = _install_cooldowns.get(repo_id)
|
||||
retry_after = (
|
||||
max(0, int(_COOLDOWN_SECS - (now - cooldown_at) + 0.999))
|
||||
if cooldown_at is not None
|
||||
else 0
|
||||
)
|
||||
jobs.append(
|
||||
{
|
||||
"repo_id": repo_id,
|
||||
"target": "local",
|
||||
"state": "failed",
|
||||
"retry_after_seconds": retry_after,
|
||||
**failure,
|
||||
}
|
||||
)
|
||||
# Preserve status for callers/tests that seed the legacy cooldown map alone.
|
||||
jobs.extend(
|
||||
{
|
||||
"repo_id": repo_id,
|
||||
"target": "local",
|
||||
"state": "failed",
|
||||
"retry_after_seconds": max(
|
||||
0, int(_COOLDOWN_SECS - (now - failed_at) + 0.999)
|
||||
),
|
||||
}
|
||||
for repo_id, failed_at in tuple(_install_cooldowns.items())
|
||||
if repo_id not in active
|
||||
and repo_id not in detailed
|
||||
and now - failed_at < _COOLDOWN_SECS
|
||||
)
|
||||
jobs.extend(gpu_gateway.remote_download_jobs())
|
||||
return {"jobs": jobs}
|
||||
|
||||
|
||||
def _is_retryable_download_error(exc: BaseException) -> bool:
|
||||
"""Whether a failed download attempt is worth retrying.
|
||||
@@ -474,6 +575,9 @@ async def install_model(req: InstallModelRequest):
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
def _do():
|
||||
# Failure handling must work even when imports, token resolution or
|
||||
# revision lookup fail before the heartbeat thread is started.
|
||||
_resolving = threading.Event()
|
||||
token = hf_progress.current_repo_id.set(req.repo_id)
|
||||
target_token = hf_progress.current_target.set("local")
|
||||
hf_progress.emit({
|
||||
@@ -500,6 +604,10 @@ async def install_model(req: InstallModelRequest):
|
||||
"revision": revision_for(req.repo_id),
|
||||
"max_workers": _download_max_workers(),
|
||||
}
|
||||
from services.token_resolver import resolve as resolve_token
|
||||
resolved_token = resolve_token()
|
||||
if resolved_token:
|
||||
dl_kwargs["token"] = resolved_token.token
|
||||
if allow_patterns:
|
||||
dl_kwargs["allow_patterns"] = allow_patterns
|
||||
_tqdm_cls = hf_progress.tracked_tqdm_class()
|
||||
@@ -513,9 +621,7 @@ async def install_model(req: InstallModelRequest):
|
||||
|
||||
# Emit a 'resolving' heartbeat every 2s while snapshot_download
|
||||
# resolves repo metadata (before any tqdm bars appear).
|
||||
import threading
|
||||
import time as _t
|
||||
_resolving = threading.Event()
|
||||
|
||||
def _heartbeat():
|
||||
_step = 0
|
||||
@@ -549,8 +655,22 @@ async def install_model(req: InstallModelRequest):
|
||||
_preflight_kwargs["allow_patterns"] = allow_patterns
|
||||
if _endpoint:
|
||||
_preflight_kwargs["endpoint"] = _endpoint
|
||||
if resolved_token:
|
||||
_preflight_kwargs["token"] = resolved_token.token
|
||||
try:
|
||||
_plan = snapshot_download(**_preflight_kwargs) # nosec B615 -- immutable revision_for pin
|
||||
_plan = list(snapshot_download(**_preflight_kwargs)) # nosec B615 -- immutable revision_for pin
|
||||
for dependency in model_spec.get("dependencies") or ():
|
||||
if req.repo_id in _cancelled:
|
||||
raise _InstallCancelled()
|
||||
dependency_plan_kwargs = {
|
||||
**_preflight_kwargs,
|
||||
"repo_id": dependency["repo_id"],
|
||||
"revision": revision_for(dependency["repo_id"]),
|
||||
}
|
||||
dependency_plan_kwargs.pop("allow_patterns", None)
|
||||
if dependency.get("allow_patterns"):
|
||||
dependency_plan_kwargs["allow_patterns"] = dependency["allow_patterns"]
|
||||
_plan.extend(snapshot_download(**dependency_plan_kwargs)) # nosec B615 -- immutable revision_for pin
|
||||
_summary = compute_plan(_plan)
|
||||
# Disk-space guard (before a single byte flows): the preflight
|
||||
# gives an exact "to download" size, so reject an install that
|
||||
@@ -568,6 +688,11 @@ async def install_model(req: InstallModelRequest):
|
||||
"phase": "install_error",
|
||||
"error": _disk_err,
|
||||
})
|
||||
_install_failures[req.repo_id] = {
|
||||
"failed_at": time.time(),
|
||||
"error": _disk_err,
|
||||
"docs_topic": "DISK_SPACE_LOW",
|
||||
}
|
||||
# A disk-full is not a transient network failure — don't set
|
||||
# a cooldown (freeing space, not waiting, is the fix). The
|
||||
# outer finally still cleans up the aggregator + context.
|
||||
@@ -584,6 +709,8 @@ async def install_model(req: InstallModelRequest):
|
||||
"phase": "install_plan",
|
||||
**_summary,
|
||||
})
|
||||
except _InstallCancelled:
|
||||
raise
|
||||
except Exception as _pf_err:
|
||||
# No preflight (older/gated repo, mirror without dry-run, etc.):
|
||||
# fall back to today's fill-in-as-files-appear behaviour.
|
||||
@@ -651,6 +778,27 @@ async def install_model(req: InstallModelRequest):
|
||||
from huggingface_hub.constants import HF_HUB_CACHE
|
||||
from services.hf_revisions import remember_revision
|
||||
remember_revision(req.repo_id, dl_kwargs["revision"], HF_HUB_CACHE)
|
||||
# A pipeline config is not a runnable installation by itself.
|
||||
# Download its reviewed dependencies only inside this explicit
|
||||
# install action, retaining the parent cancellation/retry flow.
|
||||
for dependency in model_spec.get("dependencies") or ():
|
||||
if req.repo_id in _cancelled:
|
||||
raise _InstallCancelled()
|
||||
dependency_id = dependency["repo_id"]
|
||||
dependency_kwargs = {
|
||||
**dl_kwargs,
|
||||
"repo_id": dependency_id,
|
||||
"revision": revision_for(dependency_id),
|
||||
}
|
||||
dependency_kwargs.pop("allow_patterns", None)
|
||||
if dependency.get("allow_patterns"):
|
||||
dependency_kwargs["allow_patterns"] = dependency["allow_patterns"]
|
||||
dependency_path = snapshot_download(**dependency_kwargs) # nosec B615 -- immutable revision_for pin
|
||||
if not snapshot_is_complete(dependency, dependency_path):
|
||||
raise OSError(f"{dependency_id}: required model files are missing or incomplete")
|
||||
remember_revision(dependency_id, dependency_kwargs["revision"], HF_HUB_CACHE)
|
||||
if req.repo_id in _cancelled:
|
||||
raise _InstallCancelled()
|
||||
break
|
||||
except Exception as net_err:
|
||||
# #1224: a truncated body ("peer closed connection without
|
||||
@@ -714,12 +862,28 @@ async def install_model(req: InstallModelRequest):
|
||||
"phase": "install_done",
|
||||
})
|
||||
_install_cooldowns.pop(req.repo_id, None) # success clears any cooldown (MM2-06)
|
||||
_install_failures.pop(req.repo_id, None)
|
||||
invalidate_cache()
|
||||
# A saved performance pack owns the desired engine/model policy.
|
||||
# Reconcile after every successful local install so the new model
|
||||
# becomes usable without a restart or a second manual selection.
|
||||
try:
|
||||
from services.performance_profiles import reconcile_active_profile
|
||||
|
||||
activated = reconcile_active_profile()
|
||||
if activated:
|
||||
logger.info("model install activated performance profile: %s", activated)
|
||||
except Exception:
|
||||
# The model is fully installed even if optional preference
|
||||
# reconciliation fails; readiness refresh and manual selection
|
||||
# remain available instead of misreporting the download.
|
||||
logger.exception("performance profile reconciliation failed after model install")
|
||||
except _InstallCancelled:
|
||||
_resolving.set()
|
||||
logger.info("model install cancelled: %s", req.repo_id)
|
||||
# A cancel is user intent, not a failure — don't set a cooldown.
|
||||
_install_cooldowns.pop(req.repo_id, None)
|
||||
_install_failures.pop(req.repo_id, None)
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
@@ -730,7 +894,8 @@ async def install_model(req: InstallModelRequest):
|
||||
_resolving.set()
|
||||
logger.info("model install failed for %s: %s", req.repo_id, e)
|
||||
import time as _time_fail
|
||||
_install_cooldowns[req.repo_id] = _time_fail.time()
|
||||
_failed_at = _time_fail.time()
|
||||
_install_cooldowns[req.repo_id] = _failed_at
|
||||
# #874: when the install failed because the configured HF mirror is
|
||||
# unreachable, name the mirror + the setting instead of leaking the
|
||||
# raw connectivity error. #959: likewise for the SOCKS-proxy class
|
||||
@@ -739,15 +904,41 @@ async def install_model(req: InstallModelRequest):
|
||||
# class so the wizard can react structurally (HF_MIRROR_UNREACHABLE
|
||||
# raises the inline mirror picker) without string-matching.
|
||||
from core.failure import append_hint, classify
|
||||
_error = append_hint(str(e))
|
||||
_docs_topic = classify(str(e))
|
||||
# Gated catalogue entries own their recovery topic. Hugging Face
|
||||
# uses several exception wordings for the same access verdict, so
|
||||
# the UI must not depend on parsing an English 401/403 message.
|
||||
_catalogue_topic = str(model_spec.get("failure_topic") or "")
|
||||
if _catalogue_topic and _docs_topic in {
|
||||
"",
|
||||
"HF_AUTH_FAILED",
|
||||
"PYANNOTE_LICENSE_REQUIRED",
|
||||
}:
|
||||
_docs_topic = _catalogue_topic
|
||||
# Waiting cannot fix an access/token verdict. Let the user accept
|
||||
# the terms or update the token and retry immediately.
|
||||
if _docs_topic in {
|
||||
"HF_AUTH_FAILED",
|
||||
"PYANNOTE_LICENSE_REQUIRED",
|
||||
"POCKETTTS_GATED_WEIGHTS",
|
||||
}:
|
||||
_install_cooldowns.pop(req.repo_id, None)
|
||||
_install_failures[req.repo_id] = {
|
||||
"failed_at": _failed_at,
|
||||
"error": _error,
|
||||
"docs_topic": _docs_topic,
|
||||
}
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
"downloaded": 0, "total": 0, "pct": 0.0,
|
||||
"phase": "install_error",
|
||||
"error": append_hint(str(e)),
|
||||
"docs_topic": classify(str(e)),
|
||||
"error": _error,
|
||||
"docs_topic": _docs_topic,
|
||||
})
|
||||
finally:
|
||||
_resolving.set()
|
||||
_cancelled.discard(req.repo_id)
|
||||
download_aggregator.finish(req.repo_id, target=target or "local")
|
||||
hf_progress.current_repo_id.reset(token)
|
||||
@@ -762,6 +953,7 @@ async def install_model(req: InstallModelRequest):
|
||||
# Admission and task publication are one atomic generation boundary:
|
||||
# cancellation can never observe an admitted install without its task.
|
||||
_cancelled.discard(req.repo_id)
|
||||
_install_failures.pop(req.repo_id, None)
|
||||
try:
|
||||
task = loop.create_task(asyncio.to_thread(_do))
|
||||
_install_tasks.add(task)
|
||||
@@ -811,8 +1003,17 @@ async def cancel_install(req: InstallModelRequest):
|
||||
in hf_hub 1.7.2, so an already-streaming file finishes; the cancel takes
|
||||
effect at the next retry boundary. Clears the cooldown so the user can
|
||||
immediately restart."""
|
||||
target = (req.target or "local").strip() or "local"
|
||||
if target != "local":
|
||||
from services import gpu_gateway # noqa: PLC0415
|
||||
|
||||
try:
|
||||
return await gpu_gateway.cancel_download(req.repo_id, target=target)
|
||||
except gpu_gateway.GatewayError as exc:
|
||||
raise HTTPException(status_code=409, detail=str(exc)) from exc
|
||||
_cancelled.add(req.repo_id)
|
||||
_install_cooldowns.pop(req.repo_id, None)
|
||||
_install_failures.pop(req.repo_id, None)
|
||||
return {"cancelling": req.repo_id}
|
||||
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
|
||||
logger = logging.getLogger("omnivoice.setup.models")
|
||||
router = APIRouter()
|
||||
@@ -117,7 +117,7 @@ def _target_repo_inventory() -> tuple[str, set[str]] | None:
|
||||
for capability in live.record.capabilities or []:
|
||||
if capability.get("downloaded"):
|
||||
downloaded.update(str(repo) for repo in capability.get("repo_ids") or [])
|
||||
return live.id, downloaded
|
||||
return live.worker_id, downloaded
|
||||
|
||||
|
||||
def _current_platform_tags() -> list[str]:
|
||||
@@ -368,23 +368,44 @@ def _snapshot_dirs(repo_id: str) -> list[str]:
|
||||
return dirs
|
||||
|
||||
|
||||
def snapshot_is_complete(model: dict, snapshot_path: str) -> bool:
|
||||
"""Apply the same catalogue requirements during installation and listing."""
|
||||
config_only = bool(model.get("config_only"))
|
||||
required = tuple(str(name) for name in (
|
||||
model.get("config_required_files") if config_only else model.get("required_files")
|
||||
) or ())
|
||||
if config_only and not required:
|
||||
return False
|
||||
try:
|
||||
present = all(
|
||||
os.path.isfile(os.path.join(snapshot_path, name))
|
||||
and os.path.getsize(os.path.join(snapshot_path, name)) >= (
|
||||
1 if config_only else _WEIGHT_FLOORS.get(os.path.splitext(name)[1].lower(), 1)
|
||||
)
|
||||
for name in required
|
||||
)
|
||||
return present and (config_only or snapshot_has_weights(snapshot_path))
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def cache_is_complete(model: dict) -> bool:
|
||||
"""True when this model's on-disk cache is usable (not a truncated download).
|
||||
|
||||
Config-only repos (``config_only: true`` in models.yaml — e.g. pyannote's
|
||||
diarisation pipeline, whose real weights live in referenced sub-repos) carry no
|
||||
weight file of their own, so the weight check would false-positive them as
|
||||
incomplete (#622 caveat). They're exempt: cache presence alone means complete.
|
||||
A weight-bearing repo is complete only if at least one of its snapshots has
|
||||
weights; if no snapshot dir is found on disk we can't prove truncation, so we
|
||||
don't downgrade (the size-based caller already decided it's cached).
|
||||
Config-only repos carry no weight file of their own. Their catalogue entry
|
||||
declares the small files that make the pipeline usable, so a README left by a
|
||||
gated 403 is not mistaken for a completed install. A weight-bearing repo is
|
||||
complete only if at least one snapshot has weights; if no snapshot directory
|
||||
is found, the size-based caller's cached result is preserved.
|
||||
"""
|
||||
if model.get("config_only"):
|
||||
return True
|
||||
for dependency in model.get("dependencies") or ():
|
||||
snapshots = _snapshot_dirs(dependency["repo_id"])
|
||||
if not any(snapshot_is_complete(dependency, path) for path in snapshots):
|
||||
return False
|
||||
dirs = _snapshot_dirs(model["repo_id"])
|
||||
if not dirs:
|
||||
return True
|
||||
return any(snapshot_has_weights(d) for d in dirs)
|
||||
return any(snapshot_is_complete(model, snapshot) for snapshot in dirs)
|
||||
|
||||
|
||||
def _is_cached_on_disk(repo_id: str) -> bool:
|
||||
@@ -449,6 +470,17 @@ def _scan_cache_on_disk() -> dict[str, dict]:
|
||||
return out
|
||||
|
||||
|
||||
def _cache_dir_missing(exc: Exception) -> bool:
|
||||
"""Whether Hugging Face is reporting the normal empty-cache state.
|
||||
|
||||
``CacheNotFound`` is expected on a clean installation before the first
|
||||
download. Treating it like a damaged Windows cache makes every model probe
|
||||
perform a redundant filesystem fallback and fills the first-run log with
|
||||
warnings. Unexpected scan failures remain visible and recoverable below.
|
||||
"""
|
||||
return type(exc).__name__ == "CacheNotFound"
|
||||
|
||||
|
||||
def is_cached(repo_id: str) -> bool:
|
||||
"""Best-effort check: does HF have this repo in its cache on disk?"""
|
||||
try:
|
||||
@@ -459,6 +491,8 @@ def is_cached(repo_id: str) -> bool:
|
||||
return True
|
||||
return False
|
||||
except Exception as e:
|
||||
if _cache_dir_missing(e):
|
||||
return False
|
||||
# scan_cache_dir can raise on Windows (WinError 448 'untrusted mount
|
||||
# point'); fall back to a direct disk check so a cached model isn't
|
||||
# mistaken for missing and re-downloaded in a loop (#117/#118). Logged
|
||||
@@ -497,6 +531,62 @@ def invalidate_cache() -> None:
|
||||
|
||||
# ── Endpoints ──────────────────────────────────────────────────────────────
|
||||
|
||||
@router.get("/models/access/status")
|
||||
def model_access_status(repo_id: str = Query(...)):
|
||||
"""Check gated Hub access without downloading model files.
|
||||
|
||||
This route runs only after an explicit UI action. It never returns the
|
||||
token or a raw Hub exception; callers need only the per-repository verdict.
|
||||
"""
|
||||
model = _catalog.get(repo_id)
|
||||
if model is None:
|
||||
raise HTTPException(status_code=404, detail="Unknown model")
|
||||
if not model.get("gated"):
|
||||
return {
|
||||
"repo_id": repo_id,
|
||||
"token_present": False,
|
||||
"ready": True,
|
||||
"repositories": [],
|
||||
}
|
||||
|
||||
from services import token_resolver
|
||||
|
||||
resolved = token_resolver.resolve()
|
||||
repositories = [repo_id]
|
||||
prerequisite = str(model.get("prerequisite_repo_id") or "").strip()
|
||||
if prerequisite:
|
||||
repositories.append(prerequisite)
|
||||
if not resolved:
|
||||
return {
|
||||
"repo_id": repo_id,
|
||||
"token_present": False,
|
||||
"ready": False,
|
||||
"repositories": [
|
||||
{"repo_id": current, "access": "token_missing"}
|
||||
for current in repositories
|
||||
],
|
||||
}
|
||||
|
||||
from huggingface_hub import get_hf_file_metadata, hf_hub_url
|
||||
|
||||
results = []
|
||||
for current in repositories:
|
||||
try:
|
||||
url = hf_hub_url(current, filename=".gitattributes")
|
||||
get_hf_file_metadata(url, token=resolved.token)
|
||||
access = "granted"
|
||||
except Exception as exc: # Hub exception types vary across releases.
|
||||
status = getattr(getattr(exc, "response", None), "status_code", None)
|
||||
access = "required" if status in {401, 403, 404} else "unavailable"
|
||||
results.append({"repo_id": current, "access": access})
|
||||
return {
|
||||
"repo_id": repo_id,
|
||||
"token_present": True,
|
||||
"ready": all(item["access"] == "granted" for item in results),
|
||||
"repositories": results,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/models")
|
||||
def list_models():
|
||||
"""Catalogue every known model + its on-disk install state.
|
||||
@@ -532,10 +622,13 @@ def list_models():
|
||||
"nb_files": entry.nb_files,
|
||||
}
|
||||
except Exception as e:
|
||||
# WinError-448 fallback (#117/#118): use a direct disk scan so installed
|
||||
# models still show as installed instead of offering a re-download.
|
||||
logger.warning("scan_cache_dir failed (%s); using disk fallback", e)
|
||||
cached_by_repo = _scan_cache_on_disk()
|
||||
if _cache_dir_missing(e):
|
||||
cached_by_repo = {}
|
||||
else:
|
||||
# WinError-448 fallback (#117/#118): use a direct disk scan so installed
|
||||
# models still show as installed instead of offering a re-download.
|
||||
logger.warning("scan_cache_dir failed (%s); using disk fallback", e)
|
||||
cached_by_repo = _scan_cache_on_disk()
|
||||
|
||||
out = []
|
||||
host_tags = set(platform_tags)
|
||||
@@ -562,6 +655,7 @@ def list_models():
|
||||
"curated": _model_curated(m, host_tags),
|
||||
})
|
||||
response = {
|
||||
"target": target_key,
|
||||
"models": out,
|
||||
"total_installed_bytes": sum(m["size_on_disk_bytes"] for m in out),
|
||||
"hf_cache_dir": "" if remote_inventory is not None else hf_cache_dir(),
|
||||
@@ -623,21 +717,21 @@ def recommendations():
|
||||
rationale = (
|
||||
"NVIDIA preset: VoiceStudio (required) runs standalone. Optional ASR picks "
|
||||
"are CUDA-accelerated via CTranslate2 — Whisper large-v3 for dubbing "
|
||||
"(best word timestamps), Turbo for 5× faster transcription, Parakeet TDT "
|
||||
"v3 for live dictation. KittenTTS adds CPU-realtime English."
|
||||
"(best word timestamps) and Turbo for 5× faster transcription. Whisper "
|
||||
"Tiny provides broad-language local dictation. KittenTTS adds CPU-realtime English."
|
||||
)
|
||||
elif has_rocm:
|
||||
rationale = (
|
||||
"AMD/ROCm preset: VoiceStudio (required) runs standalone. CTranslate2 has "
|
||||
"no ROCm backend, so the PyTorch Whisper large-v3 build is the "
|
||||
"GPU-accelerated ASR route; faster-whisper works on CPU, and Parakeet "
|
||||
"TDT v3 handles live dictation."
|
||||
"GPU-accelerated ASR route; faster-whisper works on CPU, and Whisper Tiny "
|
||||
"provides broad-language local dictation."
|
||||
)
|
||||
else:
|
||||
rationale = (
|
||||
"CPU preset: VoiceStudio (required) runs standalone. Optional picks favour "
|
||||
"speed on CPU — Whisper large-v3 (int8) for accuracy, Turbo when speed "
|
||||
"matters, Parakeet TDT v3 (int8 ONNX) for live dictation, KittenTTS for "
|
||||
"matters, Whisper Tiny (ONNX) for live dictation, KittenTTS for "
|
||||
"instant English TTS."
|
||||
)
|
||||
|
||||
@@ -653,9 +747,12 @@ def recommendations():
|
||||
entry.repo_id for entry in info.repos if entry.size_on_disk > 0
|
||||
}
|
||||
except Exception as e:
|
||||
# WinError-448 fallback (#117/#118): recommend based on the disk scan.
|
||||
logger.debug("scan_cache_dir failed (%s); using disk fallback", e)
|
||||
cached_ids = set(_scan_cache_on_disk().keys())
|
||||
if _cache_dir_missing(e):
|
||||
cached_ids = set()
|
||||
else:
|
||||
# WinError-448 fallback (#117/#118): recommend based on the disk scan.
|
||||
logger.debug("scan_cache_dir failed (%s); using disk fallback", e)
|
||||
cached_ids = set(_scan_cache_on_disk().keys())
|
||||
|
||||
entries = []
|
||||
for meta in curated:
|
||||
@@ -679,6 +776,7 @@ def recommendations():
|
||||
all_installed = all(e["installed"] for e in entries)
|
||||
|
||||
return {
|
||||
"target": remote_inventory[0] if remote_inventory is not None else "local",
|
||||
"device": {
|
||||
"os": target_os,
|
||||
"arch": target_arch,
|
||||
|
||||
+170
-12
@@ -15,6 +15,8 @@ from api.dependencies import is_loopback, require_admin, require_admin_action
|
||||
from fastapi.responses import FileResponse, StreamingResponse
|
||||
import torch
|
||||
import shutil
|
||||
import subprocess
|
||||
import shlex
|
||||
|
||||
from core.config import OUTPUTS_DIR, DATA_DIR, CRASH_LOG_PATH, LOG_PATH, IDLE_TIMEOUT_SECONDS
|
||||
from core.version import APP_VERSION
|
||||
@@ -38,6 +40,10 @@ logger = logging.getLogger("omnivoice.api")
|
||||
# Cache device checks at module load — they don't change at runtime
|
||||
_is_mac = hasattr(torch.backends, "mps") and torch.backends.mps.is_available()
|
||||
_is_cuda = torch.cuda.is_available()
|
||||
try:
|
||||
_is_xpu = hasattr(torch, "xpu") and torch.xpu.is_available()
|
||||
except Exception:
|
||||
_is_xpu = False
|
||||
# Prime psutil's internal CPU counter so the first non-blocking call returns useful data
|
||||
psutil.cpu_percent(interval=None)
|
||||
|
||||
@@ -51,6 +57,14 @@ def _detect_cpu_model() -> str:
|
||||
for line in f:
|
||||
if line.lower().startswith("model name"):
|
||||
return line.split(":", 1)[1].strip()
|
||||
if sys.platform == "win32":
|
||||
import winreg
|
||||
|
||||
with winreg.OpenKey(
|
||||
winreg.HKEY_LOCAL_MACHINE,
|
||||
r"HARDWARE\DESCRIPTION\System\CentralProcessor\0",
|
||||
) as key:
|
||||
return str(winreg.QueryValueEx(key, "ProcessorNameString")[0]).strip()
|
||||
if sys.platform == "darwin":
|
||||
import subprocess
|
||||
return subprocess.check_output(
|
||||
@@ -61,6 +75,77 @@ def _detect_cpu_model() -> str:
|
||||
return platform.processor() or ""
|
||||
|
||||
|
||||
def _gpu_name_priority(name: str) -> tuple[int, int]:
|
||||
lowered = name.lower()
|
||||
if any(token in lowered for token in ("remote", "virtual", "basic display")):
|
||||
return (-1, len(name))
|
||||
if any(token in lowered for token in ("nvidia", "radeon", "amd", "intel arc")):
|
||||
return (2, len(name))
|
||||
return (1, len(name))
|
||||
|
||||
|
||||
def _detect_os_gpu_name() -> str:
|
||||
"""Best-effort display-adapter identity when the active torch build is CPU-only."""
|
||||
try:
|
||||
if sys.platform == "win32":
|
||||
executable = shutil.which("powershell.exe") or shutil.which("powershell")
|
||||
if not executable:
|
||||
return ""
|
||||
result = subprocess.run(
|
||||
[
|
||||
executable,
|
||||
"-NoProfile",
|
||||
"-NonInteractive",
|
||||
"-Command",
|
||||
"Get-CimInstance Win32_VideoController | Select-Object -ExpandProperty Name",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=3,
|
||||
check=False,
|
||||
creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0),
|
||||
)
|
||||
names = [line.strip() for line in result.stdout.splitlines() if line.strip()]
|
||||
return max(names, key=_gpu_name_priority, default="")
|
||||
if sys.platform.startswith("linux"):
|
||||
executable = shutil.which("lspci")
|
||||
if not executable:
|
||||
return ""
|
||||
result = subprocess.run(
|
||||
[executable, "-mm"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=2,
|
||||
check=False,
|
||||
)
|
||||
names = []
|
||||
for line in result.stdout.splitlines():
|
||||
parts = shlex.split(line)
|
||||
if len(parts) >= 4 and parts[1] in {"VGA compatible controller", "3D controller"}:
|
||||
names.append(" ".join(parts[2:4]))
|
||||
return max(names, key=_gpu_name_priority, default="")
|
||||
if sys.platform == "darwin":
|
||||
executable = shutil.which("system_profiler")
|
||||
if not executable:
|
||||
return ""
|
||||
result = subprocess.run(
|
||||
[executable, "SPDisplaysDataType"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=3,
|
||||
check=False,
|
||||
)
|
||||
names = [
|
||||
line.split(":", 1)[1].strip()
|
||||
for line in result.stdout.splitlines()
|
||||
if "Chipset Model:" in line
|
||||
]
|
||||
return max(names, key=_gpu_name_priority, default="")
|
||||
except (OSError, ValueError, subprocess.SubprocessError):
|
||||
return ""
|
||||
return ""
|
||||
|
||||
|
||||
def _detect_gpu() -> tuple[str, float]:
|
||||
"""(gpu_name, vram_total_gb) — static for the process lifetime.
|
||||
|
||||
@@ -71,11 +156,15 @@ def _detect_gpu() -> tuple[str, float]:
|
||||
if _is_cuda:
|
||||
props = torch.cuda.get_device_properties(0)
|
||||
return torch.cuda.get_device_name(0), round(props.total_memory / (1024 ** 3), 1)
|
||||
if _is_xpu:
|
||||
props = torch.xpu.get_device_properties(0)
|
||||
total_memory = float(getattr(props, "total_memory", 0.0))
|
||||
return torch.xpu.get_device_name(0), round(total_memory / (1024 ** 3), 1)
|
||||
if _is_mac:
|
||||
return "Apple Silicon (MPS)", 0.0
|
||||
except Exception:
|
||||
pass
|
||||
return "", 0.0
|
||||
return _detect_os_gpu_name(), 0.0
|
||||
|
||||
|
||||
# Static hardware facts, captured once — /system/info is hit on every
|
||||
@@ -93,6 +182,37 @@ def _disk_free_gb() -> float:
|
||||
return 0.0
|
||||
|
||||
|
||||
def _nvidia_live_stats() -> tuple[float, float, float] | None:
|
||||
"""Return GPU%, used VRAM GiB, total VRAM GiB without an optional Python dependency."""
|
||||
executable = shutil.which("nvidia-smi")
|
||||
if not executable:
|
||||
return None
|
||||
try:
|
||||
creationflags = subprocess.CREATE_NO_WINDOW if sys.platform == "win32" else 0
|
||||
result = subprocess.run(
|
||||
[
|
||||
executable,
|
||||
"--query-gpu=utilization.gpu,memory.used,memory.total",
|
||||
"--format=csv,noheader,nounits",
|
||||
"--id=0",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=1.5,
|
||||
check=False,
|
||||
creationflags=creationflags,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
return None
|
||||
values = [float(value.strip()) for value in result.stdout.splitlines()[0].split(",")]
|
||||
if len(values) != 3:
|
||||
return None
|
||||
utilization, used_mib, total_mib = values
|
||||
return utilization, used_mib / 1024, total_mib / 1024
|
||||
except (OSError, ValueError, IndexError, subprocess.SubprocessError):
|
||||
return None
|
||||
|
||||
|
||||
def _ui_port() -> int:
|
||||
"""The Vite UI dev-server port, single-sourced from OMNIVOICE_UI_PORT.
|
||||
|
||||
@@ -185,8 +305,8 @@ def loaded_models():
|
||||
@router.post("/model/unload/{model_id}")
|
||||
async def unload_model(model_id: str):
|
||||
"""Unload a specific model by id (MM2-04). Delegates to model_lifecycle;
|
||||
an unknown id maps to HTTP 400. ``tts`` | ``diarization`` |
|
||||
``sidecar:<id>`` | ``sidecars``."""
|
||||
an unknown id maps to HTTP 400. Supports every id returned by
|
||||
``GET /model/loaded`` plus the aggregate ``sidecars`` id."""
|
||||
from services import model_lifecycle
|
||||
try:
|
||||
return await model_lifecycle.unload(model_id)
|
||||
@@ -204,7 +324,18 @@ def system_info():
|
||||
try:
|
||||
_ffmpeg = find_ffmpeg()
|
||||
from services import model_manager as _mm
|
||||
from services import asr_backend as _asr_backend
|
||||
from core import prefs as _prefs_mod
|
||||
_asr_engine = _asr_backend.active_backend_id()
|
||||
_asr_model = (
|
||||
_asr_backend._offline_asr_repo(_asr_engine)
|
||||
or os.environ.get("ASR_MODEL")
|
||||
or _asr_engine
|
||||
)
|
||||
_translation_provider = (
|
||||
os.environ.get("TRANSLATE_PROVIDER")
|
||||
or _prefs_mod.get("translation_backend", "argos")
|
||||
)
|
||||
return {
|
||||
"app_version": APP_VERSION,
|
||||
"generate_timeout_s": _mm.GPU_JOB_TIMEOUT_S,
|
||||
@@ -224,8 +355,8 @@ def system_info():
|
||||
"crash_log_path": CRASH_LOG_PATH,
|
||||
"idle_timeout_seconds": IDLE_TIMEOUT_SECONDS,
|
||||
"model_checkpoint": resolve_omnivoice_checkpoint(), # #693: show the effective checkpoint, not a leaked raw value
|
||||
"asr_model": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
|
||||
"translate_provider": os.environ.get("TRANSLATE_PROVIDER", "google"),
|
||||
"asr_model": _asr_model,
|
||||
"translate_provider": _translation_provider,
|
||||
"has_hf_token": _has_hf_token(),
|
||||
"fast_download": _fast_download_status(),
|
||||
"device": get_best_device(),
|
||||
@@ -669,6 +800,7 @@ async def clear_tauri_logs():
|
||||
@router.get("/sysinfo", response_model=SysinfoResponse)
|
||||
def get_sys_info():
|
||||
vram = 0.0
|
||||
total_vram = 0.0
|
||||
gpu_active = False
|
||||
|
||||
try:
|
||||
@@ -681,18 +813,38 @@ def get_sys_info():
|
||||
vram = alloc() / (1024**3)
|
||||
elif _is_cuda:
|
||||
vram = torch.cuda.memory_allocated() / (1024**3)
|
||||
total_vram = torch.cuda.get_device_properties(torch.cuda.current_device()).total_memory / (1024**3)
|
||||
elif _is_xpu:
|
||||
vram = torch.xpu.memory_allocated() / (1024**3)
|
||||
total_vram = float(
|
||||
getattr(torch.xpu.get_device_properties(0), "total_memory", 0.0)
|
||||
) / (1024**3)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if vram > 0.01:
|
||||
gpu_active = True
|
||||
|
||||
gpu_utilization = None
|
||||
nvidia_stats = _nvidia_live_stats() if _is_cuda else None
|
||||
if nvidia_stats:
|
||||
gpu_utilization, vram, total_vram = nvidia_stats
|
||||
gpu_active = gpu_active or gpu_utilization > 0 or vram > 0.01
|
||||
|
||||
vm = psutil.virtual_memory()
|
||||
cpu_frequency = psutil.cpu_freq()
|
||||
return {
|
||||
"cpu": psutil.cpu_percent(interval=None),
|
||||
"cpu_model": _CPU_MODEL,
|
||||
"cpu_physical_cores": psutil.cpu_count(logical=False) or 0,
|
||||
"cpu_logical_cores": psutil.cpu_count(logical=True) or 0,
|
||||
"cpu_frequency_ghz": round((cpu_frequency.current if cpu_frequency else 0.0) / 1000, 2),
|
||||
"ram": vm.used / (1024**3),
|
||||
"total_ram": vm.total / (1024**3),
|
||||
"gpu_name": _GPU_NAME,
|
||||
"gpu_utilization": gpu_utilization,
|
||||
"vram": round(vram, 2),
|
||||
"total_vram": round(total_vram, 2),
|
||||
"gpu_active": gpu_active
|
||||
}
|
||||
|
||||
@@ -708,12 +860,18 @@ async def flush_memory(unload_model: bool = False):
|
||||
|
||||
freed_model = False
|
||||
if unload_model:
|
||||
import services.model_manager as mm
|
||||
async with mm._model_lock:
|
||||
# Also drops the clone-prompt side cache, which this path used to
|
||||
# leave resident — an "unload" that kept the encoded reference
|
||||
# tensors belonging to the model it just released (#1495).
|
||||
freed_model = mm.unload_shared_model()
|
||||
from services import model_lifecycle
|
||||
|
||||
# The user-facing action has always promised "Unload all". Route it
|
||||
# through the lifecycle facade so alternate TTS engines, dictation,
|
||||
# diarisation, translation and sidecars are released as well as the
|
||||
# shared OmniVoice model. Individual runtimes still decline while
|
||||
# leased by active work.
|
||||
released = await model_lifecycle.unload_all()
|
||||
freed_model = any(
|
||||
bool(result.get("success"))
|
||||
for result in released.get("results", {}).values()
|
||||
)
|
||||
|
||||
# Multi-pass GC to break reference cycles
|
||||
gc.collect(generation=2)
|
||||
@@ -883,7 +1041,7 @@ def system_notifications():
|
||||
from core import run_sentinel
|
||||
|
||||
rec = run_sentinel.newest_record()
|
||||
if rec is not None and not rec[1]:
|
||||
if rec is not None and not rec[1] and run_sentinel.warrants_user_notice(rec[0]):
|
||||
record = rec[0]
|
||||
last = record.get("last_activity") or {}
|
||||
doing = f" Last activity: {last.get('kind')}." if last.get("kind") else ""
|
||||
|
||||
@@ -217,7 +217,7 @@ async def convert_speech(
|
||||
# clone-less engine with the actionable switch-engine message (→ 400),
|
||||
# and a backend mid-shutdown raises ModelLoadInterruptedByShutdown out
|
||||
# of the model load → the global 503 [shutting_down] handler.
|
||||
from services.tts_backend import resolve_generation_backend
|
||||
from services.tts_backend import active_backend_id, resolve_generation_backend
|
||||
try:
|
||||
backend = await resolve_generation_backend(
|
||||
require_cloning=True, cloning_purpose="voice conversion",
|
||||
@@ -319,14 +319,16 @@ async def convert_speech(
|
||||
)
|
||||
|
||||
start_time = time.time()
|
||||
from services.performance_profiles import tts_defaults
|
||||
_profile_defaults = tts_defaults(active_backend_id())
|
||||
_render = functools.partial(
|
||||
_run_backend_inference,
|
||||
backend, text, language, cond["ref_audio_path"], cond["ref_text"],
|
||||
cond["instruct"],
|
||||
None, # duration — the model picks; match_duration owns pacing
|
||||
16, 2.0, # num_step / guidance_scale (the /generate defaults)
|
||||
_profile_defaults.get("num_step", 16), 2.0,
|
||||
1.0, # speed
|
||||
True, True, # denoise / postprocess_output
|
||||
True, _profile_defaults.get("postprocess_output", True),
|
||||
used_seed,
|
||||
)
|
||||
try:
|
||||
|
||||
@@ -116,6 +116,18 @@ def get_target(op: str = "") -> dict:
|
||||
return routing.status(op=op.strip() or None)
|
||||
|
||||
|
||||
@router.get("/runtime")
|
||||
async def get_runtime(engine: str = "", op: str = "tts") -> dict:
|
||||
"""Runtime/model facts for the machine that will execute this operation."""
|
||||
from services import gpu_gateway # noqa: PLC0415
|
||||
|
||||
return await gpu_gateway.status(
|
||||
engine=engine.strip() or None,
|
||||
op=op.strip() or "tts",
|
||||
control_plane=service.control_plane,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/target")
|
||||
def set_target(request: TargetRequest) -> dict:
|
||||
"""Choose where work runs. Exactly one target is active at a time."""
|
||||
|
||||
@@ -15,9 +15,16 @@ class SysinfoResponse(BaseModel):
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
cpu: float = Field(description="CPU usage percentage (0–100)")
|
||||
cpu_model: str = ""
|
||||
cpu_physical_cores: int = 0
|
||||
cpu_logical_cores: int = 0
|
||||
cpu_frequency_ghz: float = 0.0
|
||||
ram: float = Field(description="Used RAM in GiB")
|
||||
total_ram: float = Field(description="Total RAM in GiB")
|
||||
gpu_name: str = ""
|
||||
gpu_utilization: float | None = None
|
||||
vram: float = Field(0.0, description="Used VRAM in GiB")
|
||||
total_vram: float = Field(0.0, description="Total VRAM in GiB when reported by the runtime")
|
||||
gpu_active: bool = Field(False, description="Whether a GPU is actively used")
|
||||
|
||||
|
||||
@@ -82,7 +89,7 @@ class ModelStatusResponse(BaseModel):
|
||||
status: str = Field(description="idle | loading | ready")
|
||||
checkpoint: str | None = None
|
||||
loaded_at: str | None = None
|
||||
sub_stage: str | None = Field(None, description="Current loading sub-stage: importing | loading_weights | loading_asr | compiling | ready | error")
|
||||
sub_stage: str | None = Field(None, description="Current TTS loading sub-stage: importing | loading_weights | compiling | ready | error")
|
||||
detail: str | None = Field(None, description="Human-readable detail of current loading phase")
|
||||
error: str | None = Field(None, description="Error message if loading failed")
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
# repo_id (required) — HuggingFace repository ID
|
||||
# engines (required) — backend ids that load this repo; [] for pipeline weights no single engine owns (they list under "Other weights")
|
||||
# label (required) — Human-readable display name
|
||||
# role (required) — TTS | ASR | Diarisation
|
||||
# role (required) — TTS | ASR | Translation | Diarisation
|
||||
# size_gb (required) — Approximate download size in GiB
|
||||
# required (optional) — true if the app needs this model to function.
|
||||
# Only the TTS model is required: the app boots and
|
||||
@@ -46,13 +46,18 @@ models:
|
||||
curated_on: [all]
|
||||
|
||||
- repo_id: "audio-cpp/audio.cpp-gguf"
|
||||
label: "Breeze-TTS-2 Q8_0 for audio.cpp (English + Chinese, clone + design)"
|
||||
label: "audio.cpp native bundle (Breeze-TTS-2 + Sortformer diarisation)"
|
||||
role: TTS
|
||||
engines: [audiocpp]
|
||||
size_gb: 4.73
|
||||
families: [tts, diarisation]
|
||||
size_gb: 4.98
|
||||
required_files:
|
||||
- "Breeze-TTS-2-GGUF/breeze-tts-2-q8_0.gguf"
|
||||
- "Sortformer-Diar-4spk-v1-GGUF/sortformer-diar-4spk-v1-q8_0.gguf"
|
||||
allow_patterns:
|
||||
- "Breeze-TTS-2-GGUF/breeze-tts-2-q8_0.gguf"
|
||||
note: "Optional audio.cpp model. Research/non-commercial weights and self-hosted outputs; install only after reviewing the license."
|
||||
- "Sortformer-Diar-4spk-v1-GGUF/sortformer-diar-4spk-v1-q8_0.gguf"
|
||||
note: "Optional audio.cpp bundle for native voice cloning and up-to-four-speaker diarisation. Research/non-commercial weights and self-hosted outputs; install only after reviewing the licenses."
|
||||
|
||||
# ── ASR (optional — curated per platform) ─────────────────────────────
|
||||
# No ASR model is required to boot: TTS-only installs work. Dubbing,
|
||||
@@ -256,6 +261,15 @@ models:
|
||||
curated_on: [all]
|
||||
note: "Recommended cross-platform dictation default (auto-detect). CPU, int8 ONNX. Requires sherpa-onnx."
|
||||
|
||||
# ── Translation ──────────────────────────────────────────────────────
|
||||
|
||||
- repo_id: "facebook/nllb-200-distilled-600M"
|
||||
label: "NLLB-200 distilled 600M (local, 200 languages)"
|
||||
role: Translation
|
||||
engines: []
|
||||
size_gb: 2.4
|
||||
note: "Best fully-local translation quality. Install explicitly before selecting NLLB; translation never downloads these weights in the background."
|
||||
|
||||
# ── Diarisation ───────────────────────────────────────────────────────
|
||||
|
||||
- repo_id: "pyannote/speaker-diarization-3.1"
|
||||
@@ -264,7 +278,21 @@ models:
|
||||
engines: []
|
||||
size_gb: 0.8
|
||||
config_only: true # pipeline repo; real weights live in referenced sub-repos
|
||||
note: "Needs an HF_TOKEN with license accepted."
|
||||
config_required_files: ["config.yaml"]
|
||||
dependencies:
|
||||
- repo_id: "pyannote/segmentation-3.0"
|
||||
required_files: ["pytorch_model.bin"]
|
||||
allow_patterns: ["config.yaml", "pytorch_model.bin"]
|
||||
- repo_id: "pyannote/wespeaker-voxceleb-resnet34-LM"
|
||||
required_files: ["pytorch_model.bin"]
|
||||
allow_patterns: ["config.yaml", "pytorch_model.bin"]
|
||||
gated: true
|
||||
requires_hf_token: true
|
||||
access_url: "https://huggingface.co/pyannote/speaker-diarization-3.1"
|
||||
prerequisite_repo_id: "pyannote/segmentation-3.0"
|
||||
prerequisite_access_url: "https://huggingface.co/pyannote/segmentation-3.0"
|
||||
failure_topic: "PYANNOTE_LICENSE_REQUIRED"
|
||||
note: "Requires access to both pyannote repositories and an HF token."
|
||||
|
||||
# ── Optional TTS ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Native-crash diagnostics for the backend process (#2135).
|
||||
|
||||
A crash inside torch/CUDA — graph capture, a driver fault, an allocator abort —
|
||||
kills the interpreter below the level any ``except`` can reach. #2135's reporter
|
||||
saw exactly that: the backend "simply exited" mid-``/generate`` with no Python
|
||||
traceback, no HTTP response, and ``ConnectionRefused`` on the next ``/health``.
|
||||
There was nothing in the logs to diagnose because nothing in Python ever ran
|
||||
again.
|
||||
|
||||
``faulthandler`` installs handlers for the fatal signals (SIGSEGV, SIGABRT,
|
||||
SIGBUS, SIGFPE, SIGILL) that print every thread's Python stack to stderr on the
|
||||
way down. That is the difference between "the process vanished" and a named
|
||||
frame pointing at the engine call that killed it.
|
||||
|
||||
This is strictly a diagnostic: it does not prevent the crash, and it must never
|
||||
be the reason startup fails.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
_DISABLE_ENV = "OMNIVOICE_DISABLE_FAULTHANDLER"
|
||||
_TRUTHY = frozenset({"1", "true", "yes", "on"})
|
||||
|
||||
|
||||
def _disabled() -> bool:
|
||||
return os.environ.get(_DISABLE_ENV, "").strip().lower() in _TRUTHY
|
||||
|
||||
|
||||
def enable_fault_handler(stderr=None) -> bool:
|
||||
"""Arm fatal-signal tracebacks. Returns True when armed.
|
||||
|
||||
Call as early as possible — before torch is imported — so a crash during
|
||||
model load is covered too. Honours ``OMNIVOICE_DISABLE_FAULTHANDLER=1`` for
|
||||
hosts whose outer supervisor installs its own handlers.
|
||||
|
||||
Args:
|
||||
stderr: optional file object to write dumps to. Defaults to the real
|
||||
``sys.stderr`` (→ ``backend_err.log``). faulthandler keeps the
|
||||
underlying fd, so the object must stay open for the process
|
||||
lifetime.
|
||||
|
||||
Never raises: a frozen build with a detached stderr, or a platform without
|
||||
the signals, degrades to "no crash dump" rather than a failed boot.
|
||||
"""
|
||||
if _disabled():
|
||||
return False
|
||||
try:
|
||||
import faulthandler
|
||||
|
||||
# all_threads=True: the fatal frame is routinely on a GPU-pool or
|
||||
# compile worker, not whichever thread happens to take the signal.
|
||||
if stderr is not None:
|
||||
faulthandler.enable(file=stderr, all_threads=True)
|
||||
else:
|
||||
faulthandler.enable(all_threads=True)
|
||||
return True
|
||||
except Exception:
|
||||
return False
|
||||
@@ -73,7 +73,7 @@ def _origin_tuple(value: str | None) -> tuple[str, str, int | None] | None:
|
||||
):
|
||||
return None
|
||||
scheme = parsed.scheme.lower()
|
||||
if scheme not in {"http", "https", "tauri"}:
|
||||
if scheme not in {"http", "https", "tauri", "app"}:
|
||||
return None
|
||||
if port is None:
|
||||
if scheme == "http":
|
||||
@@ -83,6 +83,8 @@ def _origin_tuple(value: str | None) -> tuple[str, str, int | None] | None:
|
||||
return scheme, parsed.hostname.lower(), port
|
||||
|
||||
|
||||
DEFAULT_DESKTOP_ORIGINS = ("tauri://localhost", "http://tauri.localhost", "app://voicestudio")
|
||||
|
||||
def configured_allowed_origins() -> frozenset[tuple[str, str, int | None]]:
|
||||
raw_port = os.environ.get("OMNIVOICE_UI_PORT", "3901")
|
||||
try:
|
||||
@@ -92,7 +94,7 @@ def configured_allowed_origins() -> frozenset[tuple[str, str, int | None]]:
|
||||
values = os.environ.get(
|
||||
"OMNIVOICE_ALLOWED_ORIGINS",
|
||||
f"http://localhost:{ui_port},http://127.0.0.1:{ui_port},"
|
||||
"tauri://localhost,http://tauri.localhost",
|
||||
+ ",".join(DEFAULT_DESKTOP_ORIGINS),
|
||||
).split(",")
|
||||
return frozenset(
|
||||
origin
|
||||
|
||||
@@ -51,6 +51,23 @@ _DIALECTS = set(_VD._INSTRUCT_CATEGORIES[5]) # the 12 Chinese dialect tokens
|
||||
# the archetype ``attrs`` shape, so the response drops straight into vdStates.
|
||||
CATEGORY_ORDER = ("Gender", "Age", "Pitch", "Style", "EnglishAccent", "ChineseDialect")
|
||||
|
||||
|
||||
def instruct_to_vd_states(instruct: str | None) -> dict[str, str]:
|
||||
"""Project a saved validator-token instruct onto the complete UI recipe."""
|
||||
attrs = {category: "Auto" for category in CATEGORY_ORDER}
|
||||
sanitized = _VD.sanitize_instruct(instruct)
|
||||
if not sanitized:
|
||||
return attrs
|
||||
for token in sanitized.split(", "):
|
||||
category_index = _VD._instruct_category_index(token)
|
||||
if category_index < 0 or category_index >= len(CATEGORY_ORDER):
|
||||
continue
|
||||
# The first four frontend categories use the English canonical token;
|
||||
# dialects and accents already use their engine-native form.
|
||||
canonical = _VD._INSTRUCT_ZH_TO_EN.get(token, token)
|
||||
attrs[CATEGORY_ORDER[category_index]] = canonical
|
||||
return attrs
|
||||
|
||||
# ── Pinyin / romanized names → Chinese-dialect tokens (functional vocabulary) ─
|
||||
DIALECT_PINYIN = {
|
||||
"henan": "河南话",
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
"""Stable engine IDs whose first use requires local license acceptance."""
|
||||
|
||||
LICENSE_GATED_ENGINES: frozenset[str] = frozenset({"supertonic3", "pockettts"})
|
||||
@@ -4,7 +4,7 @@ Used by the React ErrorBoundary's "Open docs for this error" button (via the
|
||||
TypeScript mirror at `frontend/src/utils/errorDocsMap.ts`) and by the Phase 5
|
||||
bug-reporter for "this error has a docs page" links.
|
||||
|
||||
The 5-class taxonomy below is the contract — Phase 5 reporter consumes it,
|
||||
The error taxonomy below is the contract — Phase 5 reporter consumes it,
|
||||
the TS map mirrors it, and `test_error_docs_map.test_keys_match_taxonomy`
|
||||
locks the key set. To add a new class:
|
||||
|
||||
@@ -20,6 +20,8 @@ from core import links
|
||||
_BASE = links.PROJECT_REPO_BLOB_MAIN
|
||||
|
||||
ERROR_DOCS: dict[str, str] = {
|
||||
"DIARIZATION_LOAD_FAILED": f"{_BASE}/docs/features/diarization.md#troubleshooting",
|
||||
"DIARIZATION_MODEL_MISSING": f"{_BASE}/docs/features/diarization.md#local-installation-and-repair",
|
||||
"GATEKEEPER_QUARANTINE": f"{_BASE}/docs/install/macos.md#gatekeeper-quarantine",
|
||||
"APPIMAGE_WEBKIT_WHITESCREEN": f"{_BASE}/docs/install/linux.md#appimage-white-screen-on-fedora-44--ubuntu-2404",
|
||||
"PKG_RESOURCES_MISSING": f"{_BASE}/docs/install/troubleshooting.md#pkg_resources-missing",
|
||||
|
||||
+18
-1
@@ -85,6 +85,8 @@ _HINTS: dict[str, str] = {
|
||||
"GATEKEEPER_QUARANTINE": "Clear the macOS quarantine flag (xattr -cr the app), then reopen.",
|
||||
"APPIMAGE_WEBKIT_WHITESCREEN": "Launch with WEBKIT_DISABLE_DMABUF_RENDERER=1 set.",
|
||||
"HF_AUTH_FAILED": "Set a valid HF_TOKEN in Settings → Hugging Face and retry.",
|
||||
"DIARIZATION_MODEL_MISSING": "Install or repair the selected diarisation model in Settings > Models > Diarisation, then retry transcription.",
|
||||
"DIARIZATION_LOAD_FAILED": "Open Settings > Logs > Backend for the model load error, then retry transcription after correcting it.",
|
||||
"PYANNOTE_LICENSE_REQUIRED": "Accept the pyannote model licenses on Hugging Face, then retry.",
|
||||
"POCKETTTS_GATED_WEIGHTS": "PocketTTS weights are gated on HuggingFace. Accept the access agreement at huggingface.co/kyutai/pocket-tts, then set HF_TOKEN in Settings → Hugging Face and retry.",
|
||||
"COMPUTE_TYPE_UNSUPPORTED": "Your GPU doesn't support float16 — VoiceStudio retried on int8. If transcription still fails, set OMNIVOICE/ASR_COMPUTE_TYPE=int8 or use CPU.",
|
||||
@@ -404,7 +406,22 @@ def classify(reason: str) -> str:
|
||||
or "access conditions" in low
|
||||
) and ("pocket" in low or "kyutai" in low):
|
||||
return "POCKETTTS_GATED_WEIGHTS"
|
||||
if "pyannote" in low or ("gated" in low and "model" in low) or "accept the" in low:
|
||||
diarisation = any(marker in low for marker in (
|
||||
"pyannote", "diarization", "diarisation", "sortformer",
|
||||
))
|
||||
access_failure = any(marker in low for marker in (
|
||||
"gated", "unauthorized", "forbidden", "401", "403",
|
||||
"accept the", "license", "user conditions",
|
||||
))
|
||||
if diarisation and not access_failure:
|
||||
if any(marker in low for marker in (
|
||||
"files are missing", "files are missing or incomplete",
|
||||
"filenotfounderror", "localentrynotfounderror", "model is missing",
|
||||
)):
|
||||
return "DIARIZATION_MODEL_MISSING"
|
||||
if any(marker in low for marker in ("failed to load", "load failed", "runtime failed")):
|
||||
return "DIARIZATION_LOAD_FAILED"
|
||||
if (diarisation and access_failure) or ("gated" in low and "model" in low) or "accept the" in low:
|
||||
return "PYANNOTE_LICENSE_REQUIRED"
|
||||
# ASR robustness (#551 / #549): name the class so the no-segments toast is
|
||||
# actionable. Place before the generic returns so a compute-type/transformers
|
||||
|
||||
@@ -57,6 +57,48 @@ class HFTokenRedactor(logging.Filter):
|
||||
return True
|
||||
|
||||
|
||||
class RoutineHealthAccessFilter(logging.Filter):
|
||||
"""Drop only successful routine liveness access lines.
|
||||
|
||||
The desktop supervisor probes every two seconds. Startup/not-ready responses
|
||||
and every other request remain visible, while the steady-state 200 line no
|
||||
longer consumes the small rotating diagnostic log.
|
||||
"""
|
||||
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
try:
|
||||
args = record.args
|
||||
if not isinstance(args, tuple) or len(args) < 5:
|
||||
return True
|
||||
_client, method, path, _http_version, status = args[:5]
|
||||
return not (
|
||||
method == "GET"
|
||||
and str(path).partition("?")[0] == "/health"
|
||||
and int(status) == 200
|
||||
)
|
||||
except (TypeError, ValueError):
|
||||
return True
|
||||
|
||||
|
||||
class RoutineAsyncioTransportFilter(logging.Filter):
|
||||
"""Drop only expected socket-close noise from asyncio's transport layer."""
|
||||
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
try:
|
||||
message = record.getMessage()
|
||||
if record.levelno == logging.WARNING and "socket.send() raised exception" in message:
|
||||
return False
|
||||
exception = record.exc_info[1] if record.exc_info else None
|
||||
return not (
|
||||
message.startswith(
|
||||
"Exception in callback _ProactorBasePipeTransport._call_connection_lost"
|
||||
)
|
||||
and isinstance(exception, (BrokenPipeError, ConnectionResetError))
|
||||
)
|
||||
except Exception:
|
||||
return True
|
||||
|
||||
|
||||
def install_redaction_filter(root_logger: logging.Logger | None = None) -> None:
|
||||
"""Attach a single HFTokenRedactor to the root logger and to every
|
||||
existing handler. Idempotent — repeated calls do not stack up duplicate
|
||||
@@ -69,3 +111,17 @@ def install_redaction_filter(root_logger: logging.Logger | None = None) -> None:
|
||||
for handler in list(target.handlers):
|
||||
if not any(isinstance(f, HFTokenRedactor) for f in handler.filters):
|
||||
handler.addFilter(HFTokenRedactor())
|
||||
|
||||
|
||||
def install_access_log_filter(logger: logging.Logger | None = None) -> None:
|
||||
"""Install the routine-health filter on Uvicorn's access logger once."""
|
||||
target = logger or logging.getLogger("uvicorn.access")
|
||||
if not any(isinstance(item, RoutineHealthAccessFilter) for item in target.filters):
|
||||
target.addFilter(RoutineHealthAccessFilter())
|
||||
|
||||
|
||||
def install_asyncio_transport_filter(logger: logging.Logger | None = None) -> None:
|
||||
"""Install the expected transport-close filter on asyncio once."""
|
||||
target = logger or logging.getLogger("asyncio")
|
||||
if not any(isinstance(item, RoutineAsyncioTransportFilter) for item in target.filters):
|
||||
target.addFilter(RoutineAsyncioTransportFilter())
|
||||
|
||||
@@ -13,6 +13,23 @@ from typing import Any, BinaryIO, Callable, Optional
|
||||
WINDOWS_PIPE_POLL_INTERVAL_S = 0.25
|
||||
_FILE_TYPE_PIPE = 3 # winbase.h FILE_TYPE_PIPE
|
||||
|
||||
def _exit_after_parent_loss(code: int) -> None:
|
||||
"""Retire backend-only crash forensics before the desktop-owned exit.
|
||||
|
||||
Losing the containment pipe means the desktop process ended, including an
|
||||
Electron development reload. That is not a backend crash: the shell owns
|
||||
this child and the watchdog is deliberately terminating it. ``os._exit``
|
||||
skips FastAPI lifespan cleanup, so clear the run sentinel here first. A
|
||||
real backend abort/OOM never reaches this callback and remains detectable
|
||||
on the next start.
|
||||
"""
|
||||
try:
|
||||
from core import run_sentinel
|
||||
|
||||
run_sentinel.clear_sentinel()
|
||||
except Exception:
|
||||
pass
|
||||
os._exit(code)
|
||||
|
||||
def _watch_parent_pipe(reader: BinaryIO, exit_process: Callable[[int], None]) -> None:
|
||||
"""Block until the desktop-owned stdin pipe closes, then exit immediately."""
|
||||
@@ -91,12 +108,12 @@ def arm_desktop_parent_watchdog() -> bool:
|
||||
if reader is None:
|
||||
return False
|
||||
target: Callable[..., None] = _watch_parent_pipe
|
||||
args: tuple = (reader, os._exit)
|
||||
args: tuple = (reader, _exit_after_parent_loss)
|
||||
if os.name == "nt":
|
||||
handle = _windows_pipe_handle(reader)
|
||||
if handle is not None:
|
||||
target = _watch_parent_pipe_handle
|
||||
args = (handle, os._exit)
|
||||
args = (handle, _exit_after_parent_loss)
|
||||
# A non-pipe stdin (file, NUL) cannot have a read pending against a
|
||||
# pipe file object, so the blocking reader stays correct there.
|
||||
threading.Thread(
|
||||
|
||||
@@ -75,6 +75,17 @@ def set_(key: str, value: Any) -> None:
|
||||
_save(data)
|
||||
|
||||
|
||||
def update_mapping(key: str, changes: dict, *, replace: bool = False) -> None:
|
||||
"""Atomically update one preference object without losing concurrent edits."""
|
||||
with _MUTATE_LOCK:
|
||||
data = _load()
|
||||
current = data.get(key)
|
||||
value = dict(current) if isinstance(current, dict) and not replace else {}
|
||||
value.update(changes)
|
||||
data[key] = value
|
||||
_save(data)
|
||||
|
||||
|
||||
def delete(key: str) -> None:
|
||||
"""Remove *key* from prefs.json if present."""
|
||||
with _MUTATE_LOCK:
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
"""Small, metadata-free local profile portraits."""
|
||||
import io
|
||||
import warnings
|
||||
|
||||
from fastapi import HTTPException
|
||||
from PIL import Image, ImageOps, UnidentifiedImageError
|
||||
|
||||
MAX_IMAGE_BYTES = 5 * 1024 * 1024
|
||||
|
||||
|
||||
def normalize_portrait(data: bytes) -> bytes:
|
||||
if len(data) > MAX_IMAGE_BYTES:
|
||||
raise HTTPException(413, "Profile image exceeds 5 MB")
|
||||
try:
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("error", Image.DecompressionBombWarning)
|
||||
with Image.open(io.BytesIO(data)) as source:
|
||||
if source.format not in {"JPEG", "PNG", "WEBP"}:
|
||||
raise ValueError("unsupported image format")
|
||||
if source.width * source.height > 16_000_000:
|
||||
raise ValueError("image dimensions too large")
|
||||
portrait = ImageOps.fit(ImageOps.exif_transpose(source).convert("RGB"), (256, 256))
|
||||
output = io.BytesIO()
|
||||
portrait.save(output, format="JPEG", quality=88)
|
||||
return output.getvalue()
|
||||
except (UnidentifiedImageError, OSError, ValueError, Image.DecompressionBombWarning, Image.DecompressionBombError) as exc:
|
||||
raise HTTPException(422, "Use a valid JPEG, PNG or WebP image up to 16 megapixels") from exc
|
||||
@@ -70,6 +70,20 @@ LOG_TAIL_LINES = 40
|
||||
#: burst instead of one per request.
|
||||
ACTIVITY_THROTTLE_S = 2.0
|
||||
|
||||
# An idle desktop process can disappear with its owning shell during an OS
|
||||
# shutdown, package replacement, or a forced development relaunch. Keep that
|
||||
# forensic record, but do not nag the user unless there is evidence that work
|
||||
# was interrupted or the backend itself logged a fatal failure.
|
||||
_ACTIONABLE_LOG_MARKERS = (
|
||||
"traceback (most recent call last)",
|
||||
"critical",
|
||||
"fatal error",
|
||||
"out of memory",
|
||||
"memoryerror",
|
||||
"segmentation fault",
|
||||
"access violation",
|
||||
)
|
||||
|
||||
# In-memory run state. `owns` guards clear_sentinel()/touch_activity() so an
|
||||
# instance that skipped writing (another live instance holds the sentinel)
|
||||
# can never clobber or delete the other instance's sentinel.
|
||||
@@ -298,6 +312,24 @@ def _build_crash_record(sentinel: dict, now: float) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def warrants_user_notice(record: dict) -> bool:
|
||||
"""Whether an unclean record is actionable enough to interrupt the user.
|
||||
|
||||
The record remains available to diagnostics either way. A meaningful
|
||||
activity marker means a generation/transcription/task may have been lost;
|
||||
a strict fatal-log marker catches startup/native crashes that happened
|
||||
before an activity could be recorded. Idle shell-owned exits stay quiet.
|
||||
"""
|
||||
activity = record.get("last_activity")
|
||||
if isinstance(activity, dict) and str(activity.get("kind") or "").strip():
|
||||
return True
|
||||
tail = record.get("log_tail")
|
||||
if not isinstance(tail, list):
|
||||
return False
|
||||
joined = "\n".join(str(line).lower() for line in tail[-LOG_TAIL_LINES:])
|
||||
return any(marker in joined for marker in _ACTIONABLE_LOG_MARKERS)
|
||||
|
||||
|
||||
def _load_store() -> dict:
|
||||
store = _read_json(CRASH_RECORD_PATH) or {}
|
||||
records = store.get("records")
|
||||
|
||||
+32
-1
@@ -10,6 +10,27 @@ from core import run_sentinel
|
||||
logger = logging.getLogger("omnivoice.tasks")
|
||||
|
||||
|
||||
def _stream_failure(update):
|
||||
"""Recognize terminal SSE failures, including generators that do not raise."""
|
||||
if isinstance(update, bytes):
|
||||
update = update.decode("utf-8", errors="replace")
|
||||
if not isinstance(update, str):
|
||||
return None
|
||||
lines = update.splitlines()
|
||||
try:
|
||||
payload = json.loads("\n".join(line[5:].strip() for line in lines if line.startswith("data:")))
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
if not isinstance(payload, dict):
|
||||
return None
|
||||
if payload.get("type") != "error" and not any(line.strip() == "event: error" for line in lines):
|
||||
return None
|
||||
detail = payload.get("reason") or payload.get("error") or payload.get("detail")
|
||||
if isinstance(detail, dict):
|
||||
detail = detail.get("message") or detail.get("reason")
|
||||
return detail if isinstance(detail, str) and detail else "Task failed"
|
||||
|
||||
|
||||
class TaskManager:
|
||||
"""In-memory task dispatcher with SQLite-backed metadata.
|
||||
|
||||
@@ -125,9 +146,19 @@ class TaskManager:
|
||||
except Exception: logger.exception("job_store.mark_cancelled failed")
|
||||
break
|
||||
await self._push_event(task_id, update)
|
||||
stream_error = _stream_failure(update)
|
||||
if stream_error is not None:
|
||||
t["status"] = "failed"
|
||||
t["error"] = stream_error
|
||||
try:
|
||||
job_store.mark_failed(task_id, stream_error)
|
||||
except Exception:
|
||||
logger.exception("job_store.mark_failed failed")
|
||||
await res.aclose()
|
||||
break
|
||||
elif inspect.iscoroutine(res):
|
||||
await res
|
||||
if t["status"] != "cancelled":
|
||||
if t["status"] not in {"cancelled", "failed"}:
|
||||
t["status"] = "done"
|
||||
try: job_store.mark_done(task_id)
|
||||
except Exception: logger.exception("job_store.mark_done failed")
|
||||
|
||||
@@ -24,7 +24,7 @@ from pathlib import Path
|
||||
# tests/test_app_version.py::test_all_version_files_in_lockstep and bumped by
|
||||
# release.yml's version-bump job, so it stays equal to
|
||||
# pyproject/tauri.conf/Cargo/package.json.
|
||||
_FALLBACK_VERSION = "0.5.2"
|
||||
_FALLBACK_VERSION = "0.5.3"
|
||||
|
||||
|
||||
def _fallback_version() -> str:
|
||||
|
||||
@@ -124,9 +124,16 @@ def _get_model():
|
||||
return _model
|
||||
|
||||
|
||||
def _transcribe(audio_path, word_timestamps):
|
||||
def _transcribe(audio_path, word_timestamps, decode_options=None):
|
||||
options = decode_options or {}
|
||||
if not isinstance(options, dict) or any(
|
||||
key not in {"beam_size", "best_of"}
|
||||
or type(value) is not int or not 1 <= value <= 8
|
||||
for key, value in options.items()
|
||||
):
|
||||
raise ValueError("Invalid ASR decoding options")
|
||||
model = _get_model()
|
||||
segments, info = model.transcribe(audio_path, word_timestamps=word_timestamps)
|
||||
segments, info = model.transcribe(audio_path, word_timestamps=word_timestamps, **options)
|
||||
out = []
|
||||
for s in segments:
|
||||
seg = {"start": float(s.start), "end": float(s.end), "text": s.text}
|
||||
@@ -178,7 +185,7 @@ def main() -> int:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong"})
|
||||
elif op == "transcribe":
|
||||
result = _transcribe(msg.get("audio_path"), bool(msg.get("word_timestamps", True)))
|
||||
result = _transcribe(msg.get("audio_path"), bool(msg.get("word_timestamps", True)), msg.get("decode_options"))
|
||||
_send(stdout, {"op": "segments", "result": result})
|
||||
elif op == "shutdown":
|
||||
return 0
|
||||
|
||||
@@ -35,9 +35,9 @@ from pathlib import Path
|
||||
|
||||
logger = logging.getLogger("omnivoice.audiocpp.bootstrap")
|
||||
|
||||
#: Pinned audio.cpp release. BreezeTTS-2 support landed in 0.7.2 — older
|
||||
#: binaries have no ``breeze_tts`` family, so the floor is also the pin.
|
||||
VERSION = "v0.7.2"
|
||||
#: Pinned audio.cpp release. BreezeTTS-2 support landed in 0.7.2; 0.7.4 adds
|
||||
#: the current native fixes and Sortformer v2.1 streaming runtime.
|
||||
VERSION = "v0.7.4"
|
||||
|
||||
#: GitHub repo serving the prebuilt binaries.
|
||||
GH_REPO = "0xShug0/audio.cpp"
|
||||
@@ -45,7 +45,7 @@ GH_REPO = "0xShug0/audio.cpp"
|
||||
#: HuggingFace repo serving the GGUF model packages (not gated).
|
||||
HF_MODEL_REPO = "audio-cpp/audio.cpp-gguf"
|
||||
|
||||
# Immutable repository revision used for the v0.7.2 Breeze-TTS-2 package.
|
||||
# Immutable repository revision used for the Breeze-TTS-2 package.
|
||||
# Pinning prevents a later upstream file replacement from silently changing
|
||||
# the model exercised by this backend.
|
||||
HF_MODEL_REVISION = "dc6fecccc2b0c6bdda0a8b2f38fa61394fee0b9c"
|
||||
@@ -87,28 +87,34 @@ DEFAULT_PORT = 17860
|
||||
#: This package's owned binary dir (probe 3).
|
||||
_PKG_BIN_DIR: Path = Path(__file__).parent / "bin"
|
||||
|
||||
# Recommended (asset filename, sha256) per platform slug, from the v0.7.2
|
||||
# Recommended (asset filename, sha256) per platform slug, from the v0.7.4
|
||||
# release. Windows and Linux use the vendor-neutral Vulkan build, which also
|
||||
# exposes the native CPU backend. Upstream publishes the macOS builds under
|
||||
# the Metal package name. No linux-aarch64 prebuilt exists in v0.7.2.
|
||||
# the Metal package name. No linux-aarch64 prebuilt exists in v0.7.4.
|
||||
_ASSETS: dict[str, tuple[str, str]] = {
|
||||
"windows-x64": (
|
||||
"audio-v0.7.2-bin-windows-x64-vulkan.zip",
|
||||
"15b8232eae740e21e507d87f827a89966de9451b085a45932d9e214e032962c1",
|
||||
"audio-v0.7.4-bin-windows-x64-vulkan.zip",
|
||||
"057332f9e3fb37706a8ecb5075ac1797efcd85fdccd739f7b65761a5920f2828",
|
||||
),
|
||||
"linux-x64": (
|
||||
"audio-v0.7.2-bin-ubuntu-x64-vulkan.tar.gz",
|
||||
"fee1f978cee76453cf17f00196554bc2ee294645739538af0726a143b6a69a23",
|
||||
"audio-v0.7.4-bin-ubuntu-x64-vulkan.tar.gz",
|
||||
"e0ef3123a9f94e130ad463db0db5a69b65485ef8db1b46edead00c03a86fa787",
|
||||
),
|
||||
"darwin-arm64": (
|
||||
"audio-v0.7.2-bin-macos-arm64-metal.tar.gz",
|
||||
"c01e4f82971bedbe341697e63a9cebd5a5d1f72d5a9bcb51a3191f95ddab7a95",
|
||||
"audio-v0.7.4-bin-macos-arm64-metal.tar.gz",
|
||||
"639926715b1cb537f82aa31656aabbae5d9a85ac36568c402026968f3072e2b3",
|
||||
),
|
||||
"darwin-x64": (
|
||||
"audio-v0.7.2-bin-macos-x64-metal.tar.gz",
|
||||
"3862270f33439077225324169313f727064f727305b54d8ce920244d75ddcc24",
|
||||
"audio-v0.7.4-bin-macos-x64-metal.tar.gz",
|
||||
"bdb797d54dcf8416bd5ac0fac282ce5500dd08843f8f22e20e9fc378ebc24c1f",
|
||||
),
|
||||
}
|
||||
_ASSET_SIZES = {
|
||||
"windows-x64": 56_818_905,
|
||||
"linux-x64": 71_551_673,
|
||||
"darwin-arm64": 25_270_657,
|
||||
"darwin-x64": 26_718_959,
|
||||
}
|
||||
|
||||
#: Binary filename per platform.
|
||||
_BINARY_NAMES = {"windows-x64": "audiocpp_server.exe"}
|
||||
@@ -291,10 +297,23 @@ def _probe_paths() -> list[Path]:
|
||||
user_dir = os.environ.get(DIR_ENV, "").strip()
|
||||
if user_dir:
|
||||
out.append(Path(user_dir) / binary_name())
|
||||
out.append(managed_runtime_dir() / binary_name())
|
||||
out.append(_PKG_BIN_DIR / binary_name())
|
||||
return out
|
||||
|
||||
|
||||
def platform_slug() -> str:
|
||||
"""Stable release-platform key used by the managed runtime installer."""
|
||||
return _platform_slug()
|
||||
|
||||
|
||||
def managed_runtime_dir() -> Path:
|
||||
"""Update-surviving location for the checksummed app-managed runtime."""
|
||||
from core.config import DATA_DIR
|
||||
|
||||
return Path(DATA_DIR) / "engines" / "audio-cpp" / VERSION.lstrip("v") / _platform_slug()
|
||||
|
||||
|
||||
def is_installed() -> bool:
|
||||
"""Cheap precedence-aware check for a usable server binary."""
|
||||
try:
|
||||
@@ -576,6 +595,11 @@ def default_asset() -> tuple[str, str] | None:
|
||||
return _ASSETS.get(_platform_slug())
|
||||
|
||||
|
||||
def default_asset_size() -> int | None:
|
||||
"""Published byte size of this host's pinned release archive."""
|
||||
return _ASSET_SIZES.get(_platform_slug())
|
||||
|
||||
|
||||
def server_port() -> int:
|
||||
"""Loopback port for the managed server (env override or default)."""
|
||||
raw = os.environ.get(PORT_ENV, "").strip()
|
||||
@@ -725,9 +749,12 @@ __all__ = [
|
||||
"_materialize_gguf_cache_path",
|
||||
"binary_name",
|
||||
"default_asset",
|
||||
"default_asset_size",
|
||||
"invalidate",
|
||||
"is_installed",
|
||||
"managed_runtime_dir",
|
||||
"package_filename",
|
||||
"platform_slug",
|
||||
"parse_device_list",
|
||||
"probe_devices",
|
||||
"resolve_compute_selection",
|
||||
|
||||
@@ -34,6 +34,7 @@ import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import threading
|
||||
import traceback
|
||||
|
||||
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES (T-02-01).
|
||||
@@ -55,6 +56,8 @@ _GEN_KW_ALLOWLIST = (
|
||||
)
|
||||
|
||||
_model = None
|
||||
_SEND_LOCK = threading.Lock()
|
||||
_LOAD_HEARTBEAT_S = 5.0
|
||||
|
||||
|
||||
# ── wire protocol ─────────────────────────────────────────────────────────
|
||||
@@ -62,9 +65,12 @@ _model = None
|
||||
|
||||
def _send(stream, obj: dict) -> None:
|
||||
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
|
||||
stream.write(struct.pack("!I", len(body)))
|
||||
stream.write(body)
|
||||
stream.flush()
|
||||
# Progress callbacks and the cold-load heartbeat can write from different
|
||||
# threads. Keep each frame atomic or their header/body pairs can interleave.
|
||||
with _SEND_LOCK:
|
||||
stream.write(struct.pack("!I", len(body)))
|
||||
stream.write(body)
|
||||
stream.flush()
|
||||
|
||||
|
||||
def _recv(stream):
|
||||
@@ -133,11 +139,27 @@ def _load_model(stdout):
|
||||
# Forward real HF download/weight progress so the parent's recv loop keeps
|
||||
# its watchdog alive across a slow cold load (the parent consumes these
|
||||
# {"op": "progress"} frames and re-arms its deadline on each one).
|
||||
progress = {"percent": 0}
|
||||
|
||||
def _on_progress(ev):
|
||||
pct = ev.get("pct", 0.0)
|
||||
if pct:
|
||||
progress["percent"] = min(round(pct * 100), 99)
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model",
|
||||
"percent": min(round(pct * 100), 99)})
|
||||
"percent": progress["percent"]})
|
||||
|
||||
# Cached checkpoints produce no download callbacks. Loading and moving a
|
||||
# model onto MPS can still exceed the normal generation budget, so keep
|
||||
# both bounded parent watchdogs informed that the child remains alive.
|
||||
stop_heartbeat = threading.Event()
|
||||
|
||||
def _heartbeat():
|
||||
while not stop_heartbeat.wait(_LOAD_HEARTBEAT_S):
|
||||
_send(stdout, {
|
||||
"op": "progress",
|
||||
"stage": "loading_model",
|
||||
"percent": progress["percent"],
|
||||
})
|
||||
|
||||
torch = _lazy_torch()
|
||||
OmniVoice = _lazy_omnivoice()
|
||||
@@ -146,11 +168,19 @@ def _load_model(stdout):
|
||||
preload_asr = should_preload_tts_asr()
|
||||
|
||||
lid = register_listener(_on_progress)
|
||||
heartbeat = threading.Thread(
|
||||
target=_heartbeat,
|
||||
name="omnivoice-load-heartbeat",
|
||||
daemon=True,
|
||||
)
|
||||
heartbeat.start()
|
||||
try:
|
||||
_model = OmniVoice.from_pretrained(
|
||||
checkpoint, device_map=device, dtype=torch.float16, load_asr=preload_asr,
|
||||
)
|
||||
finally:
|
||||
stop_heartbeat.set()
|
||||
heartbeat.join()
|
||||
unregister_listener(lid)
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 100})
|
||||
return _model
|
||||
|
||||
+73
-40
@@ -9,6 +9,13 @@ _backend_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
if _backend_dir not in sys.path:
|
||||
sys.path.insert(0, _backend_dir)
|
||||
|
||||
# #2135: arm fatal-signal tracebacks before anything heavy is imported, so a
|
||||
# native crash inside torch/CUDA leaves a named frame in backend_err.log
|
||||
# instead of a silently vanished process. See core/crash_diagnostics.py.
|
||||
from core.crash_diagnostics import enable_fault_handler # noqa: E402
|
||||
|
||||
enable_fault_handler()
|
||||
|
||||
# PyInstaller re-executes this entry module when the frozen backend binary is
|
||||
# launched. Nested operation supervisors therefore dispatch here, before math,
|
||||
# logging, FastAPI, torch, or any application initialization. Source launches
|
||||
@@ -274,16 +281,15 @@ logging.basicConfig(
|
||||
# inherits the filter, so even handler-formatted output (file, stream,
|
||||
# JSON) strips real HF tokens. Cheap (regex on each record) and
|
||||
# idempotent — extra calls are no-ops.
|
||||
from core.logging_filter import install_redaction_filter # noqa: E402
|
||||
from core.logging_filter import ( # noqa: E402
|
||||
install_access_log_filter,
|
||||
install_asyncio_transport_filter,
|
||||
install_redaction_filter,
|
||||
)
|
||||
|
||||
install_redaction_filter()
|
||||
|
||||
class AsyncioExceptionFilter(logging.Filter):
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
if record.levelno == logging.WARNING and "socket.send() raised exception" in record.getMessage():
|
||||
return False
|
||||
return True
|
||||
|
||||
logging.getLogger("asyncio").addFilter(AsyncioExceptionFilter())
|
||||
install_access_log_filter()
|
||||
install_asyncio_transport_filter()
|
||||
|
||||
# Silence HF Hub unauthenticated warnings unless specifically requested.
|
||||
logging.getLogger("huggingface_hub.utils._http").setLevel(logging.ERROR)
|
||||
@@ -537,8 +543,8 @@ async def _cancel_and_await_tasks(*tasks, timeout: float = 3.0) -> None:
|
||||
# mutation, runs in an executor thread (deferred) or inline (eager).
|
||||
# Phase A finalize: router/mount registration — mutates the app, so it runs
|
||||
# ON the event loop (deferred) with no awaits inside, making it atomic with
|
||||
# respect to in-flight requests; the StartupGate keeps everything but
|
||||
# /health + /startup/progress out until ready regardless.
|
||||
# respect to in-flight requests; the StartupGate keeps work routes out until
|
||||
# ready while retaining readiness probes and deliberate desktop shutdown.
|
||||
# Phase B: the old lifespan startup body (DB, background services).
|
||||
|
||||
_phase_a_built = False
|
||||
@@ -664,6 +670,7 @@ def _phase_a_build_inner() -> None:
|
||||
from api.routers import (
|
||||
system,
|
||||
profiles,
|
||||
profile_images,
|
||||
exports,
|
||||
generation,
|
||||
dub_core,
|
||||
@@ -703,7 +710,7 @@ def _phase_a_build_inner() -> None:
|
||||
from api.routers import mcp_bindings as _mcp_bindings_router # noqa: E402
|
||||
from api.routers import workers as workers_router # noqa: E402
|
||||
_router_modules.extend([
|
||||
system, profiles, exports, generation, voice_convert, dub_core, dub_generate,
|
||||
system, profiles, profile_images, exports, generation, voice_convert, dub_core, dub_generate,
|
||||
dub_export, dub_translate, projects, glossary, engines, tools,
|
||||
stories, setup, gallery, archetypes, describe_voice, community,
|
||||
batch, watermark, events, capture, capture_ws, speech_platform, dictation,
|
||||
@@ -877,6 +884,18 @@ async def _phase_b(app: FastAPI) -> None:
|
||||
logger.exception("Startup job-sweep failed (non-fatal).")
|
||||
|
||||
_startup_progress.begin_step("services_start")
|
||||
# Reapply an explicitly saved speed/quality profile after the local model
|
||||
# inventory is available. Older builds saved the slider but could leave
|
||||
# ASR/Dictation pointing at missing models even when compatible weights
|
||||
# were already installed. This path is local-cache-only and download-free.
|
||||
try:
|
||||
from services.performance_profiles import reconcile_active_profile
|
||||
|
||||
recovered = reconcile_active_profile()
|
||||
if recovered:
|
||||
logger.info("Startup performance selections reconciled: %s", recovered)
|
||||
except Exception:
|
||||
logger.exception("Performance-profile reconciliation failed (non-fatal).")
|
||||
# Phase 1 Wave 3 — macOS Gatekeeper quarantine probe (#54). Informational
|
||||
# only; we never auto-run `xattr -cr`.
|
||||
try:
|
||||
@@ -918,12 +937,8 @@ async def _phase_b(app: FastAPI) -> None:
|
||||
"Capture ASR preload skipped: <4GB free RAM; "
|
||||
"dictation ASR will load on first use.")
|
||||
return
|
||||
loading_detail = None
|
||||
prev_loading_detail = None
|
||||
try:
|
||||
from services.model_manager import _gpu_pool, _loading_detail
|
||||
loading_detail = _loading_detail
|
||||
prev_loading_detail = dict(loading_detail)
|
||||
from services.model_manager import _gpu_pool
|
||||
loop = asyncio.get_running_loop()
|
||||
def _warm():
|
||||
from services.asr_backend import (
|
||||
@@ -937,20 +952,12 @@ async def _phase_b(app: FastAPI) -> None:
|
||||
"Capture ASR preload skipped: no ASR model installed; "
|
||||
"dictation will offer a download on first use.")
|
||||
return
|
||||
loading_detail["sub_stage"] = "loading_asr"
|
||||
loading_detail["detail"] = "Warming up ASR engine…"
|
||||
backend = get_capture_asr_backend()
|
||||
logger.info("Capture ASR backend selected: %s", backend.id)
|
||||
if hasattr(backend, 'warmup'):
|
||||
loading_detail["detail"] = f"Loading {backend.display_name}…"
|
||||
backend.warmup()
|
||||
loading_detail["sub_stage"] = "ready"
|
||||
loading_detail["detail"] = "ASR engine ready"
|
||||
await loop.run_in_executor(_gpu_pool, _warm)
|
||||
except Exception as e:
|
||||
if loading_detail is not None and loading_detail.get("sub_stage") == "loading_asr":
|
||||
loading_detail.clear()
|
||||
loading_detail.update(prev_loading_detail or {})
|
||||
logger.warning("Capture ASR preload skipped: %s", e)
|
||||
app.state.capture_preload_task = asyncio.create_task(_preload_capture_asr())
|
||||
else:
|
||||
@@ -1104,13 +1111,10 @@ async def lifespan(app: FastAPI):
|
||||
# correctness independent of how much of that tail runs, instead of
|
||||
# depending on the shell-side deadline being long enough to cover it.
|
||||
#
|
||||
# SCOPE, explicitly: this only helps platforms where lifespan teardown
|
||||
# actually BEGINS. On Windows it does not — tools.rs terminates the job
|
||||
# object with no graceful phase at all, so this line is never reached and
|
||||
# a deliberate quit is still misreported as a crash there. That needs the
|
||||
# shell to signal deliberate intent before the hard kill, which is a
|
||||
# separate Rust-side change and is tracked separately; nothing here
|
||||
# should be read as fixing Windows.
|
||||
# Desktop shells that must hard-kill a Windows process tree retire the
|
||||
# sentinel through /system/shutdown-intent before termination. This
|
||||
# remains the graceful-path fallback for every platform and direct server
|
||||
# runs.
|
||||
#
|
||||
# sentinel_cleared feeds the truthful "Shutdown: done."/degraded log at
|
||||
# the end of this function; nothing below re-clears the sentinel, so a
|
||||
@@ -1219,11 +1223,17 @@ async def lifespan(app: FastAPI):
|
||||
# Best-effort drain: a failure here must not abort the remaining
|
||||
# shutdown steps (model unload, MCP teardown) below.
|
||||
logger.warning("Watermark pool drain failed at shutdown", exc_info=True)
|
||||
# Unload the model and free GPU memory
|
||||
# Release every runtime that can retain model memory, then free allocator
|
||||
# caches. This includes alternate TTS engines, dictation and translation;
|
||||
# limiting shutdown to the shared OmniVoice model left those runtimes to
|
||||
# process-exit cleanup and made graceful restarts look like crashes.
|
||||
try:
|
||||
import services.model_manager as mm
|
||||
if mm.unload_shared_model():
|
||||
logger.info("Shutdown: model unloaded.")
|
||||
from services import model_lifecycle
|
||||
|
||||
released = await model_lifecycle.unload_all()
|
||||
if any(result.get("success") for result in released["results"].values()):
|
||||
logger.info("Shutdown: model runtimes unloaded.")
|
||||
# Still unconditional: there are allocator caches to hand back even when
|
||||
# no model was resident.
|
||||
mm.free_vram()
|
||||
@@ -1265,6 +1275,23 @@ app = FastAPI(
|
||||
)
|
||||
|
||||
|
||||
@app.post("/system/shutdown-intent", include_in_schema=False)
|
||||
def prepare_deliberate_shutdown_during_startup(request: Request):
|
||||
"""Retire crash forensics even while deferred startup is still gated.
|
||||
|
||||
Electron must hard-kill a Windows process tree after a bounded wait. The
|
||||
ordinary system router is registered only after native/ML imports finish,
|
||||
so a quit during those imports previously received the startup 503 and left
|
||||
a false crash sentinel behind. Keep this one tiny control route available
|
||||
from socket bind; its authorization remains identical to the system router.
|
||||
"""
|
||||
from api.dependencies import require_admin
|
||||
from core import run_sentinel
|
||||
|
||||
require_admin(request)
|
||||
return {"prepared": run_sentinel.clear_sentinel()}
|
||||
|
||||
|
||||
@app.get("/docs", include_in_schema=False)
|
||||
async def scalar_docs():
|
||||
"""Interactive API documentation powered by Scalar."""
|
||||
@@ -1454,13 +1481,17 @@ async def global_exception_handler(request: Request, exc: Exception):
|
||||
|
||||
_SHELL_PATHS = {"/", "/index.html", "/favicon.ico", "/health"}
|
||||
|
||||
# Paths that answer while the deferred startup is still running.
|
||||
_STARTUP_EXEMPT = {"/health", "/startup/progress"}
|
||||
# Paths that answer while deferred startup is still running. The shutdown
|
||||
# signal must exist before the ordinary system router so a bounded Windows
|
||||
# process-tree stop cannot leave a false crash sentinel.
|
||||
_STARTUP_EXEMPT = {"/health", "/startup/progress", "/system/shutdown-intent"}
|
||||
|
||||
|
||||
class StartupGateMiddleware:
|
||||
"""503 everything except /health + /startup/progress until the deferred
|
||||
startup completes. Two jobs: honest not-ready signaling (the [starting]
|
||||
"""503 work routes until deferred startup completes.
|
||||
|
||||
Readiness probes and deliberate desktop shutdown remain live. Two jobs:
|
||||
honest not-ready signaling (the [starting]
|
||||
marker keeps the UI from offering "Report" for it, same convention as
|
||||
[shutting_down]), and route-mutation safety — no request can reach the
|
||||
router while _phase_a_finalize is still adding routes, because the ready
|
||||
@@ -1693,10 +1724,12 @@ def _ui_port() -> int:
|
||||
return 3901
|
||||
|
||||
|
||||
from core.csrf import DEFAULT_DESKTOP_ORIGINS
|
||||
|
||||
_ui = _ui_port()
|
||||
_allowed = os.environ.get(
|
||||
"OMNIVOICE_ALLOWED_ORIGINS",
|
||||
f"http://localhost:{_ui},http://127.0.0.1:{_ui},tauri://localhost,http://tauri.localhost",
|
||||
f"http://localhost:{_ui},http://127.0.0.1:{_ui}," + ",".join(DEFAULT_DESKTOP_ORIGINS),
|
||||
).split(",")
|
||||
|
||||
# Registered FIRST → innermost: the startup gate holds every request except
|
||||
|
||||
+36
-12
@@ -1,4 +1,4 @@
|
||||
from pydantic import BaseModel, field_validator
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
from typing import List, Literal, Optional
|
||||
|
||||
from services.audio_dsp import EFFECT_PRESETS
|
||||
@@ -60,7 +60,9 @@ class DubRequest(BaseModel):
|
||||
language: str = "Auto"
|
||||
language_code: str = "und" # ISO 639-1 for ffmpeg metadata (e.g. "es", "fr", "de")
|
||||
instruct: str = ""
|
||||
num_step: int = 16
|
||||
# None means "use the shared performance profile". An explicit value is
|
||||
# still authoritative for Production overrides and existing API clients.
|
||||
num_step: Optional[int] = None
|
||||
guidance_scale: float = 2.0
|
||||
speed: float = 1.0
|
||||
# Phase 4.1 — partial regen. Parallel lists by index with `segments`.
|
||||
@@ -71,7 +73,8 @@ class DubRequest(BaseModel):
|
||||
regen_only: Optional[List[str]] = None
|
||||
# Fast-preview mode for interactive edits. When true, TTS runs at
|
||||
# num_step=8 (~2× faster, ~10-20% quality drop). Client is responsible
|
||||
# for re-rendering preview segs at full quality before final export.
|
||||
# for re-rendering preview segs with the explicit override or shared
|
||||
# performance profile before final export.
|
||||
preview: Optional[bool] = False
|
||||
# How to handle segs whose TTS audio is longer than its slot (the
|
||||
# "ghost lang" overlap bug otherwise). Options:
|
||||
@@ -86,9 +89,9 @@ class DubRequest(BaseModel):
|
||||
# (Bengali, Hindi, Arabic…). Three modes:
|
||||
# "concise" — never compress TTS audio. Trim text up-front via
|
||||
# speech_rate so it fits naturally; if it still
|
||||
# overflows, hard-trim at slot with a short fade and
|
||||
# surface fit_status="overflows" so the UI can prompt
|
||||
# the user to shorten the segment. DEFAULT.
|
||||
# overflows, fail without replacing the current track;
|
||||
# the user must shorten it or choose another fit mode.
|
||||
# DEFAULT.
|
||||
# "stretch_video" — never compress TTS audio. Re-lay the timeline so
|
||||
# each segment's video portion is stretched (via
|
||||
# ffmpeg setpts) to fit the natural-rate dub audio.
|
||||
@@ -97,13 +100,13 @@ class DubRequest(BaseModel):
|
||||
# mild pitch-preserving audio speed-up (≤1.2× alone,
|
||||
# ≤1.5× in hybrid) and a mild per-segment video
|
||||
# slow-down (≤2.0×), per services/fit_planner.py.
|
||||
# Residual overflow is trimmed and surfaced.
|
||||
# "strict_slot" — legacy: keep `slot_fit` semantics (atempo squeeze
|
||||
# when audio > slot). Kept for back-compat.
|
||||
# Residual overflow fails without discarding words.
|
||||
# "strict_slot" — pitch-preserving fit of the complete speech to
|
||||
# the original start/end; may sound faster or slower.
|
||||
timing_strategy: Optional[Literal["concise", "stretch_video", "strict_slot", "smart_fit"]] = "concise"
|
||||
|
||||
# Per-job slip budget for "concise" mode. Hard-trim only kicks in once
|
||||
# gap absorption + this much extra time has been consumed.
|
||||
# Per-job slip budget for "concise" mode. Overflow fails once gap
|
||||
# absorption + this much extra time has been consumed.
|
||||
overflow_budget_s: Optional[float] = 0.0
|
||||
|
||||
# Knob overrides for `smart_fit` (ignored by other strategies). Omitted
|
||||
@@ -150,7 +153,7 @@ class TranslateRequest(BaseModel):
|
||||
provider: Optional[str] = None
|
||||
source_lang: Optional[str] = None # ISO 639-1; overrides job detection
|
||||
job_id: Optional[str] = None # Dub job id, used to resolve detected source_lang
|
||||
quality: Optional[str] = "fast" # "fast" (one-shot) | "cinematic" (reflect→adapt) | "autofit" (cinematic + strict fit-to-slot)
|
||||
quality: Optional[str] = "fast" # fast | cinematic | autofit | agent (measured render/rewrite loop)
|
||||
glossary: Optional[List[dict]] = None # [{"source": "...", "target": "...", "note": "..."}]
|
||||
# Optional regional dialect (BCP-47, e.g. "es-AR", "pt-BR") — #280 item 2.
|
||||
# Applied by LLM-backed paths (provider="openai" or quality="cinematic"):
|
||||
@@ -158,6 +161,7 @@ class TranslateRequest(BaseModel):
|
||||
# voseo: "vos sos" instead of "tú eres"). Non-LLM providers (Argos, NLLB,
|
||||
# Google) can't honor it; the response then carries dialect_applied=false.
|
||||
dialect: Optional[str] = None
|
||||
translation_instructions: Optional[str] = Field(default=None, max_length=5000)
|
||||
# Two-stage LLM translation quality (provider="openai" only; MT engines
|
||||
# ignore both). None = default ON for the LLM engine.
|
||||
# auto_glossary — one up-front LLM pass over the full transcript extracts
|
||||
@@ -175,6 +179,26 @@ class TranslateRequest(BaseModel):
|
||||
# No LLM configured / LLM failure → silently no suggestion.
|
||||
condense: Optional[bool] = False
|
||||
|
||||
|
||||
class AgentFitSegment(BaseModel):
|
||||
"""One rendered translation and its measured timing evidence."""
|
||||
|
||||
id: str
|
||||
text: str
|
||||
source_text: Optional[str] = None
|
||||
context_before: Optional[str] = None
|
||||
context_after: Optional[str] = None
|
||||
slot_seconds: float
|
||||
measured_seconds: float
|
||||
|
||||
|
||||
class AgentFitRequest(BaseModel):
|
||||
"""Revise only rendered lines that missed their exact timeline slot."""
|
||||
|
||||
translation_instructions: Optional[str] = Field(default=None, max_length=5000)
|
||||
segments: List[AgentFitSegment]
|
||||
target_lang: str
|
||||
|
||||
class ParseSubtitleTextRequest(BaseModel):
|
||||
"""Raw pasted subtitle text (SRT/VTT-ish) to be parsed into timed cues.
|
||||
|
||||
|
||||
+213
-48
@@ -1005,13 +1005,11 @@ class FasterWhisperBackend(ASRBackend):
|
||||
# CTranslate2: CUDA or CPU (no upstream ROCm/HIP build — see WhisperX note).
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
def __init__(self):
|
||||
def __init__(self, model_name: str | None = None):
|
||||
# Defaulting to the CTranslate2-converted large-v3 repo. Matches
|
||||
# KNOWN_MODELS in api/routers/setup.py so the first-run wizard
|
||||
# downloads what the backend will actually load.
|
||||
self._model_name = os.environ.get(
|
||||
"ASR_MODEL_FASTER", "Systran/faster-whisper-large-v3"
|
||||
)
|
||||
self._model_name = model_name or faster_whisper_model_id()
|
||||
self._model = None # lazy — first transcribe() loads weights
|
||||
# Set by _ensure_model() to the device/compute_type that actually loaded
|
||||
# (after the #551 compute_type / #255 OOM→CPU fallback chain).
|
||||
@@ -1114,10 +1112,13 @@ class FasterWhisperBackend(ASRBackend):
|
||||
# faster-whisper returns a generator of Segment objects + an Info
|
||||
# struct. Materialise the generator so downstream consumers can
|
||||
# index / re-iterate.
|
||||
from services.performance_profiles import asr_decode_defaults
|
||||
|
||||
segments_iter, info = self._model.transcribe(
|
||||
audio_path,
|
||||
word_timestamps=word_timestamps,
|
||||
vad_filter=True, # built-in Silero VAD — cleaner segment starts
|
||||
**asr_decode_defaults(),
|
||||
)
|
||||
segments = list(segments_iter)
|
||||
# Normalise to the shape segment_transcript(...) expects: a dict with
|
||||
@@ -1864,6 +1865,8 @@ class SherpaDictationBackend(ASRBackend):
|
||||
f"{[s.id for s in _sd.list_specs()]}"
|
||||
)
|
||||
self._spec = spec
|
||||
from services.performance_profiles import requested_tier
|
||||
self.performance_tier = requested_tier("dictation")
|
||||
self._rec = None # lazy OfflineRecognizer / OnlineRecognizer
|
||||
# One backend is shared across live-dictation WS sessions (see
|
||||
# get_sherpa_dictation_backend), so guard the one-time recognizer build
|
||||
@@ -2823,7 +2826,7 @@ class ASRModelMissingError(RuntimeError):
|
||||
super().__init__(asr_model_missing_detail(payload))
|
||||
|
||||
|
||||
def load_active_asr_backend(*, asr_pipe=None) -> ASRBackend:
|
||||
def load_active_asr_backend(*, asr_pipe=None, require_installed: bool = False) -> ASRBackend:
|
||||
""":func:`get_active_asr_backend` + eager ``ensure_loaded()``, degrading
|
||||
past backends whose deep import chain is broken (#1185).
|
||||
|
||||
@@ -2852,11 +2855,11 @@ def load_active_asr_backend(*, asr_pipe=None) -> ASRBackend:
|
||||
while True:
|
||||
backend = get_active_asr_backend(asr_pipe=asr_pipe)
|
||||
bid = getattr(backend, "id", "?")
|
||||
if tried:
|
||||
if tried or require_installed:
|
||||
# Preflight the SPECIFIC candidate about to load — not the global
|
||||
# selection, which can disagree when an asr_pipe steers
|
||||
# get_active_asr_backend (Greptile review, #1198).
|
||||
missing = asr_model_missing_error(backend_id=bid)
|
||||
missing = asr_model_missing_error(backend_id=bid, require_installed=require_installed)
|
||||
if missing is not None:
|
||||
raise ASRModelMissingError(missing)
|
||||
try:
|
||||
@@ -2931,6 +2934,109 @@ def _ref_audio_fingerprint(audio_path: str) -> str | None:
|
||||
return None
|
||||
|
||||
|
||||
def _installed_reference_fallbacks(
|
||||
selected: list[ASRBackend],
|
||||
) -> list[ASRBackend]:
|
||||
"""Return the strongest compatible local fallbacks without changing prefs."""
|
||||
fallbacks: list[ASRBackend] = []
|
||||
selected_repos = {
|
||||
_fw_repo(str(getattr(item, "_model_name", "")))
|
||||
for item in selected
|
||||
if isinstance(item, FasterWhisperBackend)
|
||||
}
|
||||
try:
|
||||
from api.routers.setup.models import (
|
||||
KNOWN_MODELS,
|
||||
_model_supported,
|
||||
_snapshot_dirs,
|
||||
snapshot_is_complete,
|
||||
)
|
||||
|
||||
available, _reason = FasterWhisperBackend.is_available()
|
||||
if available:
|
||||
compatible = sorted(
|
||||
(
|
||||
model
|
||||
for model in KNOWN_MODELS
|
||||
if str(model.get("role", "")).lower() == "asr"
|
||||
and not model.get("dictation_id")
|
||||
and (
|
||||
str(model.get("repo_id", "")).startswith("Systran/faster-")
|
||||
or model.get("repo_id")
|
||||
== "deepdml/faster-whisper-large-v3-turbo-ct2"
|
||||
)
|
||||
and _model_supported(model)
|
||||
and model.get("repo_id") not in selected_repos
|
||||
),
|
||||
key=lambda model: float(model.get("size_gb") or 0),
|
||||
reverse=True,
|
||||
)
|
||||
for model in compatible:
|
||||
snapshots = [
|
||||
path
|
||||
for path in _snapshot_dirs(str(model["repo_id"]))
|
||||
if snapshot_is_complete(model, path)
|
||||
]
|
||||
if not snapshots:
|
||||
continue
|
||||
# A concrete complete snapshot cannot trigger a Hub download.
|
||||
snapshot = max(snapshots, key=lambda path: os.path.getmtime(path))
|
||||
backend = FasterWhisperBackend(model_name=snapshot)
|
||||
setattr(backend, "_reference_ephemeral", True)
|
||||
fallbacks.append(backend)
|
||||
break
|
||||
except Exception as exc: # noqa: BLE001 - optional local fallback
|
||||
logger.warning("reference ASR cache fallback unavailable (%s)", exc)
|
||||
|
||||
try:
|
||||
from services import sherpa_dictation
|
||||
|
||||
selected_sherpa = {
|
||||
item.spec.id
|
||||
for item in selected
|
||||
if isinstance(item, SherpaDictationBackend)
|
||||
}
|
||||
installed = sorted(
|
||||
(
|
||||
spec
|
||||
for spec in sherpa_dictation.list_specs()
|
||||
if spec.id not in selected_sherpa
|
||||
and sherpa_dictation.is_installed(spec)
|
||||
),
|
||||
key=lambda spec: float(spec.size_gb or 0),
|
||||
reverse=True,
|
||||
)
|
||||
if installed:
|
||||
fallbacks.append(get_sherpa_dictation_backend(installed[0].id))
|
||||
except Exception as exc: # noqa: BLE001 - optional local fallback
|
||||
logger.warning("reference dictation cache fallback unavailable (%s)", exc)
|
||||
return fallbacks
|
||||
|
||||
|
||||
def _transcribe_reference_candidates(
|
||||
candidates: list[ASRBackend], audio_path: str,
|
||||
) -> str:
|
||||
for backend in candidates:
|
||||
try:
|
||||
result = backend.transcribe(audio_path, word_timestamps=False) or {}
|
||||
candidate_text = result.get("text") or " ".join(
|
||||
(seg.get("text") or "").strip()
|
||||
for seg in result.get("segments", [])
|
||||
)
|
||||
candidate_text = (candidate_text or "").strip()
|
||||
if candidate_text:
|
||||
return candidate_text
|
||||
except Exception as exc: # noqa: BLE001 - try the next local engine
|
||||
logger.warning("transcribe_reference: %s failed (%s)", backend.id, exc)
|
||||
finally:
|
||||
if getattr(backend, "_reference_ephemeral", False):
|
||||
try:
|
||||
backend.unload()
|
||||
except Exception: # noqa: BLE001 - release is best-effort
|
||||
logger.warning("reference ASR fallback unload failed", exc_info=True)
|
||||
return ""
|
||||
|
||||
|
||||
def transcribe_reference(audio_path: str) -> str | None:
|
||||
"""Transcribe a voice-clone reference clip with the active ASR backend.
|
||||
|
||||
@@ -2952,42 +3058,55 @@ def transcribe_reference(audio_path: str) -> str | None:
|
||||
if cached is not None:
|
||||
_ref_transcript_cache.move_to_end(fingerprint)
|
||||
return cached
|
||||
# No ASR model installed (TTS-only install): skip quietly instead of
|
||||
# letting the backend auto-download multi-GB weights mid-/generate — this
|
||||
# path is best-effort by contract (the engine's built-in fallback applies).
|
||||
if asr_model_missing_error() is not None:
|
||||
logger.info("transcribe_reference: no ASR model installed — skipping "
|
||||
"reference auto-transcription (no silent download).")
|
||||
# Prefer the selected offline ASR engine. When its selected weights are not
|
||||
# installed, reuse the selected dictation engine if that model is already
|
||||
# local. Short clone references need plain transcription, which dictation
|
||||
# engines provide well. Falling straight through to the TTS model's bundled
|
||||
# fallback produced incomplete reference conditioning for longer clips and
|
||||
# introduced spurious words at the start of short generations. Neither
|
||||
# branch may download weights implicitly.
|
||||
candidates: list[ASRBackend] = []
|
||||
offline_missing = asr_model_missing_error()
|
||||
if offline_missing is None:
|
||||
try:
|
||||
# `load_*`, not `get_*`: a backend whose shallow probe passes but
|
||||
# whose deep import chain is broken must fall through cleanly.
|
||||
backend = load_active_asr_backend()
|
||||
if not isinstance(backend, PyTorchWhisperBackend):
|
||||
candidates.append(backend)
|
||||
except Exception as e: # noqa: BLE001 — reference ASR is best-effort
|
||||
logger.warning("transcribe_reference: offline ASR unavailable (%s)", e)
|
||||
|
||||
capture_missing = asr_model_missing_error(purpose="dictation")
|
||||
if capture_missing is None:
|
||||
try:
|
||||
capture = get_capture_asr_backend()
|
||||
if not isinstance(capture, PyTorchWhisperBackend) and not any(
|
||||
type(item) is type(capture) and item.id == capture.id
|
||||
for item in candidates
|
||||
):
|
||||
candidates.append(capture)
|
||||
except Exception as e: # noqa: BLE001 — reference ASR is best-effort
|
||||
logger.warning("transcribe_reference: dictation ASR unavailable (%s)", e)
|
||||
|
||||
text = _transcribe_reference_candidates(candidates, audio_path)
|
||||
fallbacks: list[ASRBackend] = []
|
||||
if not text:
|
||||
fallbacks = _installed_reference_fallbacks(candidates)
|
||||
text = _transcribe_reference_candidates(fallbacks, audio_path)
|
||||
|
||||
if not candidates and not fallbacks:
|
||||
logger.info(
|
||||
"transcribe_reference: no installed ASR model available — skipping "
|
||||
"reference auto-transcription (no silent download)."
|
||||
)
|
||||
return None
|
||||
try:
|
||||
# `load_*`, not `get_*`: a backend whose shallow probe passes but whose
|
||||
# deep import chain is broken would otherwise be handed back here and
|
||||
# fail at `.transcribe()` below, costing every clone-without-transcript
|
||||
# its reference text even with a healthy engine next in line (#1185).
|
||||
# This path is best-effort, so a genuinely exhausted chain still just
|
||||
# returns None and defers to the model's built-in fallback.
|
||||
backend = load_active_asr_backend()
|
||||
except Exception as e: # noqa: BLE001 — never let ASR break generation
|
||||
logger.warning("transcribe_reference: no ASR backend available (%s)", e)
|
||||
return None
|
||||
if isinstance(backend, PyTorchWhisperBackend):
|
||||
# The registry fell through to the model-attached pipeline; let the
|
||||
# model load it lazily rather than constructing a second copy here.
|
||||
return None
|
||||
try:
|
||||
result = backend.transcribe(audio_path, word_timestamps=False)
|
||||
except Exception as e: # noqa: BLE001 — degrade to the model fallback
|
||||
if not text:
|
||||
logger.warning(
|
||||
"transcribe_reference: %s failed (%s) — deferring to the model's "
|
||||
"built-in ASR fallback",
|
||||
backend.id, e,
|
||||
"transcribe_reference: installed ASR engines returned no transcript "
|
||||
"— deferring to the model's built-in ASR fallback"
|
||||
)
|
||||
return None
|
||||
result = result or {}
|
||||
text = result.get("text") or " ".join(
|
||||
(seg.get("text") or "").strip() for seg in result.get("segments", [])
|
||||
)
|
||||
text = (text or "").strip()
|
||||
if text and fingerprint is not None:
|
||||
with _ref_transcript_lock:
|
||||
_ref_transcript_cache[fingerprint] = text
|
||||
@@ -3089,10 +3208,14 @@ def get_sherpa_dictation_backend(model_id: str) -> "SherpaDictationBackend":
|
||||
:func:`get_capture_asr_backend`. Thread-safe: the recognizer is shared;
|
||||
each session creates its own decode stream (see capture_ws)."""
|
||||
global _capture_backend, _capture_backend_key
|
||||
from services.performance_profiles import requested_tier
|
||||
|
||||
performance_tier = requested_tier("dictation")
|
||||
_touch_capture() # any handout resets the idle clock
|
||||
with _capture_backend_lock:
|
||||
if (isinstance(_capture_backend, SherpaDictationBackend)
|
||||
and _capture_backend_key == model_id):
|
||||
and _capture_backend_key == model_id
|
||||
and _capture_backend.performance_tier == performance_tier):
|
||||
return _capture_backend
|
||||
backend = SherpaDictationBackend(model_id=model_id)
|
||||
_capture_backend = backend
|
||||
@@ -3270,8 +3393,11 @@ def get_capture_asr_backend(*, skip_sherpa: bool = False) -> ASRBackend:
|
||||
if sherpa_id:
|
||||
ok, _ = SherpaDictationBackend.is_available()
|
||||
if ok:
|
||||
from services.performance_profiles import requested_tier
|
||||
performance_tier = requested_tier("dictation")
|
||||
if not (isinstance(_capture_backend, SherpaDictationBackend)
|
||||
and _capture_backend_key == sherpa_id):
|
||||
and _capture_backend_key == sherpa_id
|
||||
and _capture_backend.performance_tier == performance_tier):
|
||||
try:
|
||||
_capture_backend = SherpaDictationBackend(model_id=sherpa_id)
|
||||
_capture_backend_key = sherpa_id
|
||||
@@ -3346,6 +3472,34 @@ ASR_MODEL_MISSING = "asr_model_missing"
|
||||
_PYTORCH_ASR_DEFAULT = "openai/whisper-large-v3-turbo"
|
||||
_FASTER_WHISPER_DEFAULT = "Systran/faster-whisper-large-v3"
|
||||
|
||||
|
||||
def faster_whisper_model_id() -> str:
|
||||
"""Resolve the UI-selected CTranslate2 model, with env pins authoritative."""
|
||||
from core import prefs
|
||||
|
||||
return str(
|
||||
prefs.resolve(
|
||||
"asr_model_faster",
|
||||
env="ASR_MODEL_FASTER",
|
||||
default=_FASTER_WHISPER_DEFAULT,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def select_faster_whisper_model(repo_id: str) -> None:
|
||||
"""Persist and apply a CTranslate2 model selection for this process."""
|
||||
from core import prefs
|
||||
|
||||
if prefs.is_env_shadowed("ASR_MODEL_FASTER"):
|
||||
raise ValueError("ASR_MODEL_FASTER is set outside VoiceStudio")
|
||||
prefs.set_("asr_model_faster", repo_id)
|
||||
# Sidecars inherit the process environment. Updating it here makes the
|
||||
# selection effective immediately as well as after the next app launch.
|
||||
os.environ["ASR_MODEL_FASTER"] = repo_id
|
||||
instance = _ISOLATED_INSTANCES.pop("faster-whisper-isolated", None)
|
||||
if instance is not None:
|
||||
instance.shutdown()
|
||||
|
||||
# faster-whisper / WhisperX short model aliases → the HF repo they download.
|
||||
# Covers our own defaults plus the documented size aliases; an unrecognized
|
||||
# alias returns None and the preflight stays out of the way (never blocks).
|
||||
@@ -3378,7 +3532,7 @@ def _offline_asr_repo(backend_id: str | None = None) -> str | None:
|
||||
if bid == "whisperx":
|
||||
return _fw_repo(os.environ.get("ASR_MODEL_WHISPERX", "large-v3"))
|
||||
if bid == "faster-whisper":
|
||||
return _fw_repo(os.environ.get("ASR_MODEL_FASTER", _FASTER_WHISPER_DEFAULT))
|
||||
return _fw_repo(faster_whisper_model_id())
|
||||
if bid == "faster-whisper-isolated":
|
||||
# Mirror the sidecar's own resolution (_asr_sidecar/main.py):
|
||||
# ASR_MODEL_FW is a sidecar-only override, otherwise the shared
|
||||
@@ -3386,7 +3540,7 @@ def _offline_asr_repo(backend_id: str | None = None) -> str | None:
|
||||
# download a different repo than the sidecar will load.
|
||||
return _fw_repo(
|
||||
os.environ.get("ASR_MODEL_FW")
|
||||
or os.environ.get("ASR_MODEL_FASTER")
|
||||
or faster_whisper_model_id()
|
||||
or _FASTER_WHISPER_DEFAULT
|
||||
)
|
||||
if bid == "mlx-whisper":
|
||||
@@ -3427,7 +3581,7 @@ def _capture_whisper_repo() -> str | None:
|
||||
# resolve but our alias table doesn't know) yields None here — FAIL
|
||||
# OPEN rather than coerce to the default repo and demand a download
|
||||
# of a model the user never picked.
|
||||
return _fw_repo(os.environ.get("ASR_MODEL_FASTER", _FASTER_WHISPER_DEFAULT))
|
||||
return _fw_repo(faster_whisper_model_id())
|
||||
return os.environ.get("OMNIVOICE_PYTORCH_ASR_MODEL", _PYTORCH_ASR_DEFAULT)
|
||||
|
||||
|
||||
@@ -3499,12 +3653,12 @@ def _recommended_asr_model(
|
||||
_INSTALLED_REPO_MEMO: set[str] = set()
|
||||
|
||||
|
||||
def _repo_installed(repo: str) -> bool:
|
||||
def _repo_installed(repo: str, *, refresh: bool = False) -> bool:
|
||||
"""``is_cached`` + ``cache_is_complete`` with a positive-only session memo.
|
||||
|
||||
Installed state comes from the same HF-cache helpers the model store uses,
|
||||
so the answer matches the Model Catalogue's install badges."""
|
||||
if repo in _INSTALLED_REPO_MEMO:
|
||||
so the answer matches the Model Catalogue → Models install badges."""
|
||||
if not refresh and repo in _INSTALLED_REPO_MEMO:
|
||||
return True
|
||||
from api.routers.setup.models import cache_is_complete, get_model_catalog, is_cached
|
||||
meta = get_model_catalog().get(repo) or {"repo_id": repo}
|
||||
@@ -3582,7 +3736,7 @@ def asr_model_missing_error(*, purpose: str = "transcribe",
|
||||
return None # explicit opt-in engine — can't (and shouldn't) preflight
|
||||
from api.routers.setup.models import get_model_catalog
|
||||
if require_installed:
|
||||
if _repo_installed(repo):
|
||||
if _repo_installed(repo, refresh=True):
|
||||
return None
|
||||
return {
|
||||
"error": ASR_MODEL_MISSING,
|
||||
@@ -3607,6 +3761,14 @@ def asr_model_missing_error(*, purpose: str = "transcribe",
|
||||
),
|
||||
}
|
||||
except Exception: # noqa: BLE001 — preflight is best-effort, never a blocker
|
||||
if require_installed:
|
||||
logger.warning("ASR install preflight failed; refusing implicit download", exc_info=True)
|
||||
return {
|
||||
"error": ASR_MODEL_MISSING,
|
||||
"missing_repo_id": "unverified-local-model",
|
||||
"reason": "verification_failed",
|
||||
"recommended": None,
|
||||
}
|
||||
logger.warning("ASR install preflight failed — proceeding without it",
|
||||
exc_info=True)
|
||||
return None
|
||||
@@ -3615,6 +3777,9 @@ def asr_model_missing_error(*, purpose: str = "transcribe",
|
||||
def asr_model_missing_detail(payload: dict) -> str:
|
||||
"""Human-readable (English) fallback message for the typed payload —
|
||||
what legacy clients / logs see; the frontend renders its own i18n copy."""
|
||||
if payload.get("reason") == "verification_failed":
|
||||
return ("Could not verify the local speech-to-text model. "
|
||||
"Check Settings > Logs > Backend, then retry. No model was downloaded.")
|
||||
rec = payload.get("recommended") or {}
|
||||
if rec.get("label"):
|
||||
return (
|
||||
|
||||
@@ -186,6 +186,26 @@ def trim_trailing_silence(
|
||||
return audio_tensor[..., :end]
|
||||
|
||||
|
||||
|
||||
def trim_speech_padding(audio_tensor: torch.Tensor, sample_rate: int) -> torch.Tensor:
|
||||
"""Remove generated edge silence before timing, retaining 50 ms of context.
|
||||
|
||||
Never compress silence into the spoken slot or delete internal pauses.
|
||||
Silent/invalid outputs remain intact for the generation integrity guard.
|
||||
"""
|
||||
if audio_tensor.numel() == 0 or sample_rate <= 0:
|
||||
return audio_tensor
|
||||
envelope = audio_tensor.abs()
|
||||
if envelope.ndim > 1:
|
||||
envelope = envelope.amax(dim=tuple(range(envelope.ndim - 1)))
|
||||
voiced = torch.nonzero(envelope > 10 ** (-50 / 20))
|
||||
if voiced.numel() == 0:
|
||||
return audio_tensor
|
||||
margin = int(sample_rate * 0.05)
|
||||
start = max(0, int(voiced[0].item()) - margin)
|
||||
end = min(audio_tensor.shape[-1], int(voiced[-1].item()) + 1 + margin)
|
||||
return audio_tensor[..., start:end]
|
||||
|
||||
def apply_effects_chain(audio_tensor, sample_rate: int, chain: list[dict]) -> torch.Tensor:
|
||||
"""Apply a chain of named effects to an audio tensor.
|
||||
|
||||
|
||||
@@ -66,6 +66,12 @@ logger = logging.getLogger("omnivoice.audio_io")
|
||||
PathOrBuf = Union[str, "os.PathLike[str]", BinaryIO, io.IOBase]
|
||||
|
||||
|
||||
def _ensure_audio_parent(path_or_buf: PathOrBuf) -> None:
|
||||
"""Recover app output folders removed after backend initialization."""
|
||||
if isinstance(path_or_buf, (str, os.PathLike)):
|
||||
os.makedirs(os.path.dirname(os.path.abspath(path_or_buf)), exist_ok=True)
|
||||
|
||||
|
||||
def _safe_torchaudio_save(
|
||||
path_or_buf: PathOrBuf,
|
||||
tensor: torch.Tensor,
|
||||
@@ -156,6 +162,7 @@ def _safe_torchaudio_save(
|
||||
|
||||
fmt = (format or "wav").lower()
|
||||
try:
|
||||
_ensure_audio_parent(path_or_buf)
|
||||
if fmt == "wav":
|
||||
torchaudio.save(
|
||||
path_or_buf,
|
||||
@@ -313,6 +320,7 @@ def _safe_soundfile_write(
|
||||
else:
|
||||
samples = np.ascontiguousarray(samples)
|
||||
|
||||
_ensure_audio_parent(path)
|
||||
sf.write(path, samples, sample_rate, subtype=subtype)
|
||||
|
||||
|
||||
@@ -334,7 +342,7 @@ def atomic_save_wav(
|
||||
publication AND audited tensor normalization.
|
||||
|
||||
Args:
|
||||
target_path: Final destination. Parent directory must already exist.
|
||||
target_path: Final destination. Missing parent directories are recreated.
|
||||
audio: ``(channels, samples)`` or ``(samples,)`` tensor.
|
||||
sample_rate: WAV sample rate in Hz.
|
||||
**kwargs: Forwarded to ``_safe_torchaudio_save`` (``format``,
|
||||
@@ -346,6 +354,7 @@ def atomic_save_wav(
|
||||
unlinked on failure so we do not leak ``.tmp`` files in
|
||||
``DUB_DIR``.
|
||||
"""
|
||||
_ensure_audio_parent(target_path)
|
||||
target_dir = os.path.dirname(target_path) or "."
|
||||
target_base = os.path.basename(target_path)
|
||||
# The temp file must end in ``.wav`` even though it is conceptually a
|
||||
|
||||
@@ -0,0 +1,197 @@
|
||||
"""Checksummed, user-triggered installer for the native audio.cpp runtime."""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import tarfile
|
||||
import tempfile
|
||||
import threading
|
||||
import urllib.request
|
||||
import zipfile
|
||||
|
||||
from engines.audiocpp import bootstrap
|
||||
|
||||
logger = logging.getLogger("omnivoice.audiocpp.install")
|
||||
_CHUNK = 256 * 1024
|
||||
_lock = threading.Lock()
|
||||
_job = {"state": "idle", "progress": 0.0, "error": None}
|
||||
|
||||
|
||||
def _snapshot() -> dict:
|
||||
with _lock:
|
||||
return dict(_job)
|
||||
|
||||
|
||||
def _update(**fields) -> None:
|
||||
with _lock:
|
||||
_job.update(fields)
|
||||
|
||||
|
||||
def _runtime_paths() -> tuple[Path | None, Path | None]:
|
||||
try:
|
||||
server = bootstrap.resolve_server_binary()
|
||||
except (OSError, RuntimeError):
|
||||
return None, None
|
||||
cli = server.with_name("audiocpp_cli.exe" if os.name == "nt" else "audiocpp_cli")
|
||||
if not cli.is_file() or (os.name != "nt" and not os.access(cli, os.X_OK)):
|
||||
return server, None
|
||||
return server, cli
|
||||
|
||||
|
||||
def status() -> dict:
|
||||
server, cli = _runtime_paths()
|
||||
managed = bootstrap.managed_runtime_dir()
|
||||
is_managed = bool(
|
||||
server
|
||||
and server.resolve() == (managed / bootstrap.binary_name()).resolve()
|
||||
)
|
||||
return {
|
||||
"supported": bootstrap.default_asset() is not None,
|
||||
"installed": server is not None and cli is not None,
|
||||
"managed": is_managed,
|
||||
"version": bootstrap.VERSION if server and cli and is_managed else None,
|
||||
"platform": bootstrap.platform_slug(),
|
||||
"job": _snapshot(),
|
||||
}
|
||||
|
||||
|
||||
def _download(url: str, destination: Path, digest: str, expected_size: int) -> None:
|
||||
if not url.startswith("https://github.com/"):
|
||||
raise ValueError("audio.cpp downloads require the pinned GitHub release")
|
||||
request = urllib.request.Request(url, headers={"User-Agent": "VoiceStudio"})
|
||||
hasher = hashlib.sha256()
|
||||
received = 0
|
||||
with urllib.request.urlopen(request, timeout=30) as response, destination.open("wb") as out:
|
||||
total = expected_size or int(response.headers.get("Content-Length") or 0)
|
||||
while chunk := response.read(_CHUNK):
|
||||
out.write(chunk)
|
||||
hasher.update(chunk)
|
||||
received += len(chunk)
|
||||
if total:
|
||||
_update(progress=min(received / total, 0.9))
|
||||
if received != expected_size:
|
||||
raise RuntimeError("The audio.cpp runtime download size did not match the release")
|
||||
if hasher.hexdigest() != digest:
|
||||
raise RuntimeError("The audio.cpp runtime checksum did not match the release")
|
||||
|
||||
|
||||
def _safe_destination(root: Path, name: str) -> Path:
|
||||
destination = (root / name.replace("\\", "/")).resolve()
|
||||
if destination != root and root not in destination.parents:
|
||||
raise RuntimeError("The audio.cpp archive contains an unsafe path")
|
||||
return destination
|
||||
|
||||
|
||||
def _extract(archive: Path, destination: Path) -> None:
|
||||
destination.mkdir(parents=True)
|
||||
root = destination.resolve()
|
||||
if archive.suffix.lower() == ".zip":
|
||||
with zipfile.ZipFile(archive) as bundle:
|
||||
for member in bundle.infolist():
|
||||
target = _safe_destination(root, member.filename)
|
||||
if member.is_dir():
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
continue
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
with bundle.open(member) as source, target.open("wb") as out:
|
||||
shutil.copyfileobj(source, out)
|
||||
return
|
||||
with tarfile.open(archive, "r:*") as bundle:
|
||||
for member in bundle.getmembers():
|
||||
target = _safe_destination(root, member.name)
|
||||
if member.isdir():
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
continue
|
||||
if not member.isfile():
|
||||
raise RuntimeError("The audio.cpp archive contains an unsupported link")
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
source = bundle.extractfile(member)
|
||||
if source is None:
|
||||
raise RuntimeError("The audio.cpp archive contains an unreadable file")
|
||||
with source, target.open("wb") as out:
|
||||
shutil.copyfileobj(source, out)
|
||||
target.chmod(member.mode & 0o700)
|
||||
|
||||
|
||||
def _install() -> None:
|
||||
asset = bootstrap.default_asset()
|
||||
expected_size = bootstrap.default_asset_size()
|
||||
if asset is None or expected_size is None:
|
||||
raise RuntimeError("No audio.cpp runtime is published for this platform")
|
||||
filename, digest = asset
|
||||
url = (
|
||||
f"https://github.com/{bootstrap.GH_REPO}/releases/download/"
|
||||
f"{bootstrap.VERSION}/{filename}"
|
||||
)
|
||||
target = bootstrap.managed_runtime_dir()
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
with tempfile.TemporaryDirectory(prefix="audiocpp-install-", dir=target.parent) as temp:
|
||||
temp_path = Path(temp)
|
||||
archive = temp_path / filename
|
||||
_download(url, archive, digest, expected_size)
|
||||
extracted = temp_path / "extracted"
|
||||
_extract(archive, extracted)
|
||||
candidates = sorted(
|
||||
extracted.rglob(bootstrap.binary_name()), key=lambda path: len(path.parts)
|
||||
)
|
||||
if not candidates:
|
||||
raise RuntimeError("The audio.cpp release does not contain its server")
|
||||
source_dir = candidates[0].parent
|
||||
cli_name = "audiocpp_cli.exe" if os.name == "nt" else "audiocpp_cli"
|
||||
if not (source_dir / cli_name).is_file():
|
||||
raise RuntimeError("The audio.cpp release does not contain its CLI")
|
||||
prepared = temp_path / "prepared"
|
||||
shutil.copytree(source_dir, prepared)
|
||||
for executable in (prepared / bootstrap.binary_name(), prepared / cli_name):
|
||||
executable.chmod(0o700)
|
||||
probe = subprocess.run( # nosec B603 -- checksummed fixed release binary
|
||||
[str(prepared / bootstrap.binary_name()), "--list-devices"],
|
||||
capture_output=True,
|
||||
timeout=20,
|
||||
check=False,
|
||||
)
|
||||
if probe.returncode != 0:
|
||||
raise RuntimeError("The downloaded audio.cpp runtime failed its device check")
|
||||
if target.exists():
|
||||
shutil.rmtree(target)
|
||||
os.replace(prepared, target)
|
||||
bootstrap.invalidate()
|
||||
|
||||
|
||||
def start_install(*, wait: bool = False) -> dict:
|
||||
current = status()
|
||||
if current["installed"]:
|
||||
return {"status": "already_installed", **current}
|
||||
if not current["supported"]:
|
||||
raise RuntimeError("No audio.cpp runtime is published for this platform")
|
||||
with _lock:
|
||||
running = _job["state"] == "running"
|
||||
if not running:
|
||||
_job.update(state="running", progress=0.0, error=None)
|
||||
if running:
|
||||
return {"status": "already_running", **status()}
|
||||
|
||||
def worker() -> None:
|
||||
try:
|
||||
_install()
|
||||
_update(state="done", progress=1.0, error=None)
|
||||
except Exception:
|
||||
logger.exception("audio.cpp runtime installation failed")
|
||||
_update(
|
||||
state="error",
|
||||
error="The audio.cpp runtime could not be installed. Check the backend log.",
|
||||
)
|
||||
|
||||
if wait:
|
||||
worker()
|
||||
else:
|
||||
threading.Thread(target=worker, name="audiocpp-install", daemon=True).start()
|
||||
return {"status": "started", **status()}
|
||||
|
||||
|
||||
def reset_job_for_tests() -> None:
|
||||
_update(state="idle", progress=0.0, error=None)
|
||||
@@ -0,0 +1,37 @@
|
||||
"""Resolve the installed pyannote bundle without network access at job time."""
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
from tempfile import TemporaryDirectory
|
||||
|
||||
|
||||
@contextmanager
|
||||
def local_pipeline_config():
|
||||
import yaml
|
||||
from huggingface_hub import hf_hub_download
|
||||
from huggingface_hub.constants import HF_HUB_CACHE
|
||||
from services.hf_revisions import installed_revision
|
||||
|
||||
def cached(repo: str, filename: str) -> str:
|
||||
return hf_hub_download(
|
||||
repo_id=repo,
|
||||
filename=filename,
|
||||
revision=installed_revision(repo, HF_HUB_CACHE),
|
||||
local_files_only=True,
|
||||
)
|
||||
|
||||
config_path = cached("pyannote/speaker-diarization-3.1", "config.yaml")
|
||||
config = yaml.safe_load(Path(config_path).read_text(encoding="utf-8"))
|
||||
params = config["pipeline"]["params"]
|
||||
# The reviewed pipeline references these two checkpoints. Local checkpoint
|
||||
# paths prevent pyannote's nested Model.from_pretrained calls fetching them.
|
||||
for key, repo in (
|
||||
("segmentation", "pyannote/segmentation-3.0"),
|
||||
("embedding", "pyannote/wespeaker-voxceleb-resnet34-LM"),
|
||||
):
|
||||
if params.get(key) != repo:
|
||||
raise ValueError(f"Unexpected diarisation {key} repository; repair the installed pipeline")
|
||||
params[key] = cached(repo, "pytorch_model.bin")
|
||||
with TemporaryDirectory(prefix="voicestudio-pyannote-") as directory:
|
||||
path = Path(directory) / "config.yaml"
|
||||
path.write_text(yaml.safe_dump(config), encoding="utf-8")
|
||||
yield str(path)
|
||||
@@ -0,0 +1,163 @@
|
||||
"""Explicit local audio.cpp Sortformer adapter for the shared diarisation flow."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
import threading
|
||||
from tempfile import TemporaryDirectory
|
||||
|
||||
logger = logging.getLogger("omnivoice.diarisation.native")
|
||||
_process_lock = threading.Lock()
|
||||
_processes: set = set()
|
||||
MAX_V1_AUDIO_SECONDS = 120.0
|
||||
SORTFORMER_FRAME_SAMPLES = 1280 # 80 ms at the required 16 kHz input rate.
|
||||
|
||||
|
||||
def _sortformer_command(binary: Path, model: Path, device, source: Path, output: Path):
|
||||
return [
|
||||
str(binary), "--task", "diar", "--family", "sortformer_diar",
|
||||
"--model", str(model), "--backend", device.backend,
|
||||
"--device", str(device.index), "--audio", str(source),
|
||||
"--turns-out", str(output),
|
||||
# Accelerator builds otherwise keep the default 20-second fixed graph
|
||||
# and reject ordinary clips. Grow remains bounded by the v1 limit below.
|
||||
"--session-option", "graph_capacity_mode=grow",
|
||||
]
|
||||
|
||||
|
||||
def _validated_turn(turn: dict, audio_frames: int) -> tuple[int, int, str]:
|
||||
start, end = turn.get("start_sample"), turn.get("end_sample")
|
||||
speaker = turn.get("speaker_id")
|
||||
if (
|
||||
type(start) is not int
|
||||
or type(end) is not int
|
||||
or start < 0
|
||||
or start >= end
|
||||
or end > audio_frames + SORTFORMER_FRAME_SAMPLES
|
||||
or not isinstance(speaker, str)
|
||||
or not speaker
|
||||
):
|
||||
raise ValueError("Invalid native speaker-turn boundaries")
|
||||
# The decoder works in 80 ms frames and can pad its final turn one frame
|
||||
# beyond a non-aligned WAV boundary. Keep the timeline inside the media.
|
||||
return start, min(end, audio_frames), speaker
|
||||
|
||||
|
||||
def is_running() -> bool:
|
||||
with _process_lock:
|
||||
return bool(_processes)
|
||||
|
||||
|
||||
class NativeSortformer:
|
||||
"""Stateless native invocation; the GGUF is never downloaded implicitly."""
|
||||
|
||||
def __init__(self):
|
||||
from engines.audiocpp.bootstrap import resolve_server_binary
|
||||
from services.diarization_runtime import sortformer_model_path
|
||||
|
||||
try:
|
||||
self.model = sortformer_model_path()
|
||||
except Exception as exc:
|
||||
raise FileNotFoundError(
|
||||
"Install the audio.cpp Sortformer model in Settings > Models > Diarisation"
|
||||
) from exc
|
||||
if not self.model.is_file() or self.model.suffix.lower() != ".gguf":
|
||||
raise FileNotFoundError("The configured Sortformer GGUF is missing")
|
||||
with self.model.open("rb") as model_file:
|
||||
if model_file.read(4) != b"GGUF":
|
||||
raise ValueError("The configured Sortformer model is not a GGUF file")
|
||||
server = resolve_server_binary()
|
||||
self.binary = server.with_name("audiocpp_cli.exe" if os.name == "nt" else "audiocpp_cli")
|
||||
if not self.binary.is_file():
|
||||
raise FileNotFoundError("The installed audio.cpp directory has no audiocpp_cli")
|
||||
|
||||
def __call__(self, audio_path, *, num_speakers=None, job_id=None, cancel_check=None):
|
||||
if num_speakers is not None:
|
||||
raise ValueError("Sortformer v1 detects up to four speakers but cannot enforce an exact speaker count")
|
||||
import soundfile as sf
|
||||
from pyannote.core import Annotation, Segment
|
||||
from core.contained_subprocess import spawn_owned
|
||||
from engines.audiocpp.bootstrap import resolve_compute_selection
|
||||
from services.proc_registry import register_proc, unregister_proc
|
||||
|
||||
def check_cancelled():
|
||||
if cancel_check is not None and cancel_check():
|
||||
raise RuntimeError("Native diarisation cancelled")
|
||||
|
||||
check_cancelled()
|
||||
|
||||
audio_info = sf.info(str(audio_path))
|
||||
if audio_info.duration > MAX_V1_AUDIO_SECONDS:
|
||||
raise ValueError(
|
||||
"Sortformer v1 supports recordings up to 120 seconds; "
|
||||
"select pyannote for longer recordings"
|
||||
)
|
||||
device = resolve_compute_selection().device
|
||||
with TemporaryDirectory(prefix="voicestudio-sortformer-") as directory:
|
||||
source = Path(audio_path).resolve()
|
||||
output = Path(directory) / "turns.json"
|
||||
command = _sortformer_command(
|
||||
self.binary, self.model, device, source, output
|
||||
)
|
||||
def run_owned(command, log_name):
|
||||
with (Path(directory) / log_name).open("wb") as log:
|
||||
check_cancelled()
|
||||
process = spawn_owned(command, stdout=log, stderr=subprocess.STDOUT)
|
||||
with _process_lock:
|
||||
_processes.add(process)
|
||||
try:
|
||||
if job_id is not None:
|
||||
register_proc(job_id, process)
|
||||
deadline = time.monotonic() + 600
|
||||
while True:
|
||||
check_cancelled()
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
raise subprocess.TimeoutExpired(command, 600)
|
||||
try:
|
||||
code = process.wait(timeout=min(0.25, remaining))
|
||||
break
|
||||
except subprocess.TimeoutExpired:
|
||||
continue
|
||||
except BaseException:
|
||||
process.kill()
|
||||
process.wait()
|
||||
raise
|
||||
finally:
|
||||
with _process_lock:
|
||||
_processes.discard(process)
|
||||
if job_id is not None:
|
||||
unregister_proc(job_id, process)
|
||||
check_cancelled()
|
||||
if code != 0:
|
||||
with (Path(directory) / log_name).open("rb") as diagnostic:
|
||||
diagnostic.seek(0, 2)
|
||||
diagnostic.seek(max(0, diagnostic.tell() - 8192))
|
||||
tail = diagnostic.read().decode("utf-8", errors="replace")
|
||||
logger.error("Sortformer exited with %s; native log tail:\n%s", code, tail)
|
||||
raise RuntimeError(f"Native Sortformer failed (exit {code})")
|
||||
if (audio_info.samplerate != 16000 or audio_info.channels != 1
|
||||
or audio_info.format != "WAV" or audio_info.subtype != "PCM_16"):
|
||||
from services.ffmpeg_utils import find_ffmpeg
|
||||
normalized = Path(directory) / "input.wav"
|
||||
run_owned([
|
||||
find_ffmpeg(), "-nostdin", "-hide_banner", "-loglevel", "error", "-y",
|
||||
"-i", str(source), "-vn", "-ac", "1", "-ar", "16000",
|
||||
"-c:a", "pcm_s16le", str(normalized),
|
||||
], "normalize.log")
|
||||
source = normalized
|
||||
audio_info = sf.info(str(source))
|
||||
command[command.index("--audio") + 1] = str(source)
|
||||
run_owned(command, "native.log")
|
||||
turns = json.loads(output.read_text(encoding="utf-8"))
|
||||
if not isinstance(turns, list):
|
||||
raise ValueError("Invalid native speaker-turn output")
|
||||
annotation = Annotation()
|
||||
for index, turn in enumerate(turns):
|
||||
start, end, speaker = _validated_turn(turn, audio_info.frames)
|
||||
annotation[Segment(start / 16000, end / 16000), index] = speaker
|
||||
return annotation
|
||||
@@ -0,0 +1,118 @@
|
||||
"""Persisted selection and installed-only resolution for diarisation runtimes."""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from core import prefs
|
||||
|
||||
PYANNOTE = "pyannote"
|
||||
SORTFORMER = "audiocpp-sortformer"
|
||||
SORTFORMER_REPO = "audio-cpp/audio.cpp-gguf"
|
||||
SORTFORMER_FILE = "Sortformer-Diar-4spk-v1-GGUF/sortformer-diar-4spk-v1-q8_0.gguf"
|
||||
|
||||
_SORTFORMER_MODEL_MISSING = "Install the Sortformer model bundle"
|
||||
_SORTFORMER_MODEL_BROKEN = "Repair the installed Sortformer model bundle"
|
||||
_SORTFORMER_RUNTIME_MISSING = (
|
||||
"The Sortformer model is installed. Install the audio.cpp runtime to use it"
|
||||
)
|
||||
_SORTFORMER_CLI_MISSING = (
|
||||
"The installed audio.cpp runtime does not include speaker diarisation"
|
||||
)
|
||||
|
||||
|
||||
def selected_backend() -> str:
|
||||
value = str(
|
||||
prefs.resolve(
|
||||
"diarization_backend",
|
||||
env="OMNIVOICE_DIARIZATION_BACKEND",
|
||||
default=PYANNOTE,
|
||||
)
|
||||
).strip()
|
||||
return value if value in {PYANNOTE, SORTFORMER} else PYANNOTE
|
||||
|
||||
|
||||
def sortformer_model_path() -> Path:
|
||||
configured = os.environ.get("OMNIVOICE_DIARIZATION_MODEL", "").strip()
|
||||
if configured:
|
||||
return Path(configured).expanduser().resolve()
|
||||
|
||||
# An installed-only lookup never reaches the network. Installation remains
|
||||
# an explicit Model Library action through the reviewed audio.cpp bundle.
|
||||
from huggingface_hub import hf_hub_download
|
||||
from services.hf_revisions import revision_for
|
||||
|
||||
return Path(
|
||||
hf_hub_download(
|
||||
repo_id=SORTFORMER_REPO,
|
||||
filename=SORTFORMER_FILE,
|
||||
revision=revision_for(SORTFORMER_REPO),
|
||||
local_files_only=True,
|
||||
)
|
||||
).resolve()
|
||||
|
||||
|
||||
def select_backend(backend: str) -> None:
|
||||
if backend not in {PYANNOTE, SORTFORMER}:
|
||||
raise ValueError("Unknown diarisation engine")
|
||||
prefs.set_("diarization_backend", backend)
|
||||
|
||||
|
||||
def sortformer_status() -> dict:
|
||||
"""Return path-free readiness for the model and its native executable."""
|
||||
status = {
|
||||
"model": SORTFORMER_REPO,
|
||||
"model_installed": False,
|
||||
"runtime_installed": False,
|
||||
"installed": False,
|
||||
"reason": _SORTFORMER_MODEL_MISSING,
|
||||
}
|
||||
try:
|
||||
model = sortformer_model_path()
|
||||
except Exception:
|
||||
return status
|
||||
try:
|
||||
if not model.is_file() or model.suffix.lower() != ".gguf":
|
||||
return status
|
||||
with model.open("rb") as model_file:
|
||||
if model_file.read(4) != b"GGUF":
|
||||
status["reason"] = _SORTFORMER_MODEL_BROKEN
|
||||
return status
|
||||
except OSError:
|
||||
status["reason"] = _SORTFORMER_MODEL_BROKEN
|
||||
return status
|
||||
|
||||
status["model_installed"] = True
|
||||
status["reason"] = _SORTFORMER_RUNTIME_MISSING
|
||||
try:
|
||||
from engines.audiocpp.bootstrap import resolve_server_binary
|
||||
|
||||
server = resolve_server_binary()
|
||||
except (OSError, RuntimeError):
|
||||
return status
|
||||
cli = server.with_name("audiocpp_cli.exe" if os.name == "nt" else "audiocpp_cli")
|
||||
if not cli.is_file() or (os.name != "nt" and not os.access(cli, os.X_OK)):
|
||||
status["reason"] = _SORTFORMER_CLI_MISSING
|
||||
return status
|
||||
status.update(runtime_installed=True, installed=True, reason=None)
|
||||
return status
|
||||
|
||||
|
||||
def installed_backends() -> set[str]:
|
||||
"""Return complete local runtimes without loading weights or downloading."""
|
||||
installed: set[str] = set()
|
||||
from api.routers.setup.models import KNOWN_MODELS, cache_is_complete, is_cached
|
||||
|
||||
repo_id = "pyannote/speaker-diarization-3.1"
|
||||
spec = next(model for model in KNOWN_MODELS if model["repo_id"] == repo_id)
|
||||
if is_cached(repo_id) and cache_is_complete(spec):
|
||||
installed.add(PYANNOTE)
|
||||
|
||||
try:
|
||||
native = sortformer_status()
|
||||
if not native["installed"]:
|
||||
return installed
|
||||
installed.add(SORTFORMER)
|
||||
except (OSError, RuntimeError, ValueError):
|
||||
pass
|
||||
return installed
|
||||
@@ -0,0 +1,130 @@
|
||||
"""Dialogue-only replacement beds: original outside speech, separated bed inside."""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
|
||||
from services.video_retime import expand_retime_chunks
|
||||
|
||||
RATE = 48000
|
||||
FADE_S = .01
|
||||
_locks: dict[str, asyncio.Lock] = {}
|
||||
|
||||
|
||||
def dialogue_intervals(segments: list[dict]) -> list[tuple[float, float]]:
|
||||
intervals = []
|
||||
for row in segments:
|
||||
a, b = float(row['start']), float(row['end'])
|
||||
if not math.isfinite(a) or not math.isfinite(b) or a < 0 or b <= a:
|
||||
raise ValueError('Invalid dialogue interval')
|
||||
intervals.append((a, b))
|
||||
merged: list[tuple[float, float]] = []
|
||||
for a, b in sorted(intervals):
|
||||
if merged and a <= merged[-1][1]:
|
||||
merged[-1] = (merged[-1][0], max(b, merged[-1][1]))
|
||||
else:
|
||||
merged.append((a, b))
|
||||
return merged
|
||||
|
||||
|
||||
def splice_background(original: str, separated: str, output: str, intervals: list[tuple[float, float]]) -> None:
|
||||
"""Stream in bounded memory; crossfades lie INSIDE dialogue intervals."""
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
|
||||
with sf.SoundFile(original) as src, sf.SoundFile(separated) as bed:
|
||||
if src.samplerate != bed.samplerate or src.channels != bed.channels:
|
||||
raise ValueError('Background inputs must have matching sample format')
|
||||
if bed.frames < src.frames - int(.1 * src.samplerate):
|
||||
raise ValueError('Separated background is incomplete')
|
||||
with sf.SoundFile(output, 'w', samplerate=src.samplerate, channels=src.channels, subtype='FLOAT') as out:
|
||||
offset = 0
|
||||
active = 0
|
||||
while True:
|
||||
wave = src.read(65536, dtype='float32', always_2d=True)
|
||||
if not len(wave):
|
||||
break
|
||||
background = bed.read(len(wave), dtype='float32', always_2d=True)
|
||||
if len(background) < len(wave):
|
||||
background = np.pad(background, ((0, len(wave)-len(background)), (0, 0)))
|
||||
times = np.arange(offset, offset + len(wave)) / src.samplerate
|
||||
mask = np.zeros(len(wave), dtype='float32')
|
||||
while active < len(intervals) and intervals[active][1] < times[0]:
|
||||
active += 1
|
||||
for a, b in intervals[active:]:
|
||||
if a > times[-1]:
|
||||
break
|
||||
fade = min(FADE_S, (b-a)/2)
|
||||
envelope = np.clip(np.minimum((times-a)/fade, (b-times)/fade), 0, 1)
|
||||
mask = np.maximum(mask, envelope)
|
||||
out.write(wave * (1-mask[:, None]) + background * mask[:, None])
|
||||
offset += len(wave)
|
||||
|
||||
|
||||
async def _checked(cmd: list[str]) -> None:
|
||||
rc, _, error = await run_ffmpeg(cmd, timeout=1800.0)
|
||||
if rc:
|
||||
raise RuntimeError('Could not preserve original background audio: ' + str(error)[-500:])
|
||||
|
||||
|
||||
async def surgical_background(source: str, separated: str, cache_dir: str, segments: list[dict], plan: list[dict], duration: float) -> str:
|
||||
for chunk in plan:
|
||||
ratio = float(chunk["stretch_ratio"])
|
||||
if not math.isfinite(ratio) or ratio <= 0:
|
||||
raise ValueError("Invalid background retiming ratio")
|
||||
intervals = dialogue_intervals(segments)
|
||||
if not intervals:
|
||||
raise ValueError('Dialogue timing is required to preserve original background audio')
|
||||
identity = [(p, os.stat(p).st_size, os.stat(p).st_mtime_ns) for p in (source, separated)]
|
||||
key = hashlib.sha256(json.dumps([1, identity, intervals, plan, duration], sort_keys=True).encode()).hexdigest()[:24]
|
||||
target = str(Path(cache_dir) / f'surgical_{key}.wav')
|
||||
async with _locks.setdefault(target, asyncio.Lock()):
|
||||
if os.path.isfile(target):
|
||||
return target
|
||||
ffmpeg = find_ffmpeg()
|
||||
with tempfile.TemporaryDirectory(prefix='.surgical-', dir=cache_dir) as tmp:
|
||||
original, bed, spliced = [str(Path(tmp)/name) for name in ('source.wav', 'bed.wav', 'spliced.wav')]
|
||||
for inp, out in ((source, original), (separated, bed)):
|
||||
await _checked([ffmpeg, '-y', '-i', inp, '-map', '0:a:0', '-vn', '-ar', str(RATE), '-ac', '2', '-c:a', 'pcm_f32le', out])
|
||||
await asyncio.to_thread(splice_background, original, bed, spliced, intervals)
|
||||
if plan and any(abs(float(p['stretch_ratio'])-1) > 1e-6 for p in plan):
|
||||
chunks = expand_retime_chunks(plan, duration)
|
||||
# Bound filter buffering for long projects; trim each batch's
|
||||
# input before splitting it among the chunk filters.
|
||||
batches = []
|
||||
for batch_index in range(0, len(chunks), 16):
|
||||
batch = chunks[batch_index:batch_index+16]
|
||||
origin = batch[0][0]
|
||||
filters = []
|
||||
for i, (a, b, ratio) in enumerate(batch):
|
||||
rate = 1 / ratio
|
||||
tempos = []
|
||||
while rate < .5:
|
||||
tempos.append('atempo=0.5')
|
||||
rate /= .5
|
||||
while rate > 2:
|
||||
tempos.append('atempo=2')
|
||||
rate /= 2
|
||||
tempos.append(f'atempo={rate:.9f}')
|
||||
length = (b-a)*ratio
|
||||
filters.append(f'[0:a]atrim=start={a-origin:.9f}:end={b-origin:.9f},asetpts=PTS-STARTPTS,' + ','.join(tempos) + f',apad,atrim=duration={length:.9f}[c{i}]')
|
||||
filters.append(''.join(f'[c{i}]' for i in range(len(batch))) + f'concat=n={len(batch)}:v=0:a=1[out]')
|
||||
script = Path(tmp)/'retime.txt'
|
||||
script.write_text(';'.join(filters))
|
||||
batch_name = f'batch{batch_index}.wav'
|
||||
output = str(Path(tmp)/batch_name)
|
||||
await _checked([ffmpeg, '-y', '-ss', str(origin), '-t', str(batch[-1][1]-origin), '-i', spliced, '-filter_complex_script', str(script), '-map', '[out]', '-c:a', 'pcm_f32le', output])
|
||||
batches.append(batch_name)
|
||||
listing = Path(tmp)/'concat.txt'
|
||||
listing.write_text(''.join(f"file '{name}'\n" for name in batches))
|
||||
retimed = str(Path(tmp)/'retimed.wav')
|
||||
await _checked([ffmpeg, '-y', '-f', 'concat', '-safe', '1', '-i', str(listing), '-c:a', 'copy', retimed])
|
||||
spliced = retimed
|
||||
os.replace(spliced, target)
|
||||
return target
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Shared native-TTS batching policy for interactive and queued dubbing."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
logger = logging.getLogger("omnivoice.dub_batching")
|
||||
|
||||
BATCH_WIDTH_ENV = "OMNIVOICE_DUB_BATCH_WIDTH"
|
||||
_MAX_BATCH_WIDTH = 16
|
||||
|
||||
|
||||
def native_batch_width(backend) -> int:
|
||||
"""Return a host-safe native batch width for ``backend``."""
|
||||
override = os.environ.get(BATCH_WIDTH_ENV, "").strip()
|
||||
if override:
|
||||
try:
|
||||
return max(1, min(_MAX_BATCH_WIDTH, int(override)))
|
||||
except (TypeError, ValueError):
|
||||
logger.warning(
|
||||
"%s=%r is not an integer; deriving the batch width from the host",
|
||||
BATCH_WIDTH_ENV,
|
||||
override,
|
||||
)
|
||||
try:
|
||||
from core.device_caps import detect_host_caps
|
||||
|
||||
caps = detect_host_caps()
|
||||
except Exception: # noqa: BLE001 - an unprobeable host takes the safe path
|
||||
return 1
|
||||
if caps.family == "cpu" or not caps.vram_gb:
|
||||
return 1
|
||||
headroom = caps.vram_gb - float(getattr(backend, "min_vram_gb", 0.0) or 0.0)
|
||||
if headroom < 2.0:
|
||||
return 1
|
||||
if headroom < 6.0:
|
||||
return 2
|
||||
if headroom < 12.0:
|
||||
return 4
|
||||
return 8
|
||||
|
||||
|
||||
def batch_timeout_s(texts: list[str], backend) -> float:
|
||||
"""Bound one native batch without multiplying the executor base timeout."""
|
||||
from services.model_manager import generate_timeout_s
|
||||
|
||||
floor = generate_timeout_s("", engine=backend)
|
||||
overage = sum(
|
||||
max(0.0, generate_timeout_s(text, engine=backend) - floor)
|
||||
for text in texts
|
||||
)
|
||||
return floor + overage
|
||||
|
||||
|
||||
__all__ = ["BATCH_WIDTH_ENV", "batch_timeout_s", "native_batch_width"]
|
||||
@@ -1084,9 +1084,7 @@ def yt_download_sync(
|
||||
if sub_langs:
|
||||
langs = list(sub_langs)
|
||||
else:
|
||||
orig = (info.get("language") or "").strip()
|
||||
manual = list((info.get("subtitles") or {}).keys())
|
||||
langs = sorted({*manual, *([orig] if orig else [])})
|
||||
langs = _default_caption_languages(info)
|
||||
if not langs:
|
||||
logger.info("No captions available on %s (skipping subtitle pass)", log_safe(url))
|
||||
else:
|
||||
@@ -1115,6 +1113,48 @@ def yt_download_sync(
|
||||
return video_path, title, sub_files
|
||||
|
||||
|
||||
def _default_caption_languages(info: dict) -> list[str]:
|
||||
"""Return original-language caption tracks without translated auto-captions.
|
||||
|
||||
Some extractors omit ``language`` even though yt-dlp exposes an original
|
||||
automatic-caption track such as ``en-orig``. Treat that explicit suffix as
|
||||
source metadata so caption-first ingest still works instead of needlessly
|
||||
loading ASR. Manual tracks remain eligible because they are authored source
|
||||
material and yt-dlp's ``skip=translated_subs`` guard still applies.
|
||||
"""
|
||||
original = str(info.get("language") or "").strip()
|
||||
manual = {
|
||||
str(language).strip()
|
||||
for language in (info.get("subtitles") or {})
|
||||
if str(language).strip()
|
||||
}
|
||||
automatic = {
|
||||
str(language).strip()
|
||||
for language in (info.get("automatic_captions") or {})
|
||||
if str(language).strip()
|
||||
}
|
||||
selected = set(manual)
|
||||
if original:
|
||||
primary = original.split("-", 1)[0]
|
||||
has_source_manual = any(
|
||||
language == original or language.split("-", 1)[0] == primary
|
||||
for language in manual
|
||||
)
|
||||
if not has_source_manual:
|
||||
for candidate in (
|
||||
f"{original}-orig",
|
||||
f"{primary}-orig",
|
||||
original,
|
||||
primary,
|
||||
):
|
||||
if candidate in automatic:
|
||||
selected.add(candidate)
|
||||
break
|
||||
else:
|
||||
selected.update(language for language in automatic if language.endswith("-orig"))
|
||||
return sorted(selected)
|
||||
|
||||
|
||||
def parse_vtt_segments(vtt_path: str) -> list[dict]:
|
||||
"""Very small WEBVTT parser → list of {start, end, text}.
|
||||
|
||||
|
||||
@@ -17,7 +17,6 @@ from __future__ import annotations
|
||||
import importlib.util
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger("omnivoice.engine_env")
|
||||
@@ -29,6 +28,53 @@ _TORCH_COMPILE_KEY = "perf.torch_compile_disabled"
|
||||
# (e.g. a brand-new architecture running through PTX forward-compat).
|
||||
_FORCE_COMPILE_ENV = "OMNIVOICE_FORCE_TORCH_COMPILE"
|
||||
|
||||
# #2135: the environment escape hatches that torch itself honours. `main.py`
|
||||
# sets TORCH_COMPILE_DISABLE/TORCHDYNAMO_DISABLE on win32, `build_engine_env`
|
||||
# injects TORCH_COMPILE_DISABLE into engine subprocesses, and
|
||||
# `docs/install/windows.md` tells users to export it — but the in-process gate
|
||||
# below never read them, so an operator who set the documented variable still
|
||||
# got a compiled model (and, on a cudagraph mode, a native crash they could not
|
||||
# turn off). Reading them here makes one knob mean one thing everywhere.
|
||||
_COMPILE_DISABLE_ENVS = (
|
||||
"TORCH_COMPILE_DISABLE",
|
||||
"TORCHDYNAMO_DISABLE",
|
||||
"TORCHINDUCTOR_DISABLE",
|
||||
)
|
||||
|
||||
_TRUTHY = frozenset({"1", "true", "yes", "on"})
|
||||
|
||||
|
||||
def _env_compile_disabled() -> Optional[str]:
|
||||
"""The name of the first set-and-truthy compile-disable env var, else None.
|
||||
|
||||
Mirrors torch's own reading of these variables so the app's decision and
|
||||
torch's behaviour cannot disagree — the state the reporter in #2135 hit,
|
||||
where the log said "torch.compile applied" while TORCH_COMPILE_DISABLE=1
|
||||
was exported.
|
||||
"""
|
||||
for name in _COMPILE_DISABLE_ENVS:
|
||||
if os.environ.get(name, "").strip().lower() in _TRUTHY:
|
||||
return name
|
||||
return None
|
||||
|
||||
|
||||
def _settings_db_path() -> str:
|
||||
"""The settings DB the compile toggle is actually read from (best-effort).
|
||||
|
||||
Logged alongside the toggle because #2135's reporter had three
|
||||
`omnivoice.db` files on the box and edited one the backend never opened;
|
||||
naming the path turns "the setting doesn't work" into a one-line diagnosis.
|
||||
"""
|
||||
try:
|
||||
from core.config import DB_PATH
|
||||
|
||||
from core.scrub import scrub_text
|
||||
|
||||
return scrub_text(str(DB_PATH))
|
||||
except Exception:
|
||||
return "<unknown>"
|
||||
|
||||
|
||||
# #278: set (with a reason) the first time torch.compile — or *running* the
|
||||
# compiled model — fails at runtime in this process. Once set, every later
|
||||
# load in the same session goes straight to eager instead of re-tripping the
|
||||
@@ -251,6 +297,15 @@ def should_torch_compile(device: str) -> bool:
|
||||
"""
|
||||
if device != "cuda":
|
||||
return False
|
||||
# #2135: honoured before every other gate — an explicit env opt-out is the
|
||||
# user's most direct statement of intent, and it must hold on every
|
||||
# platform (the reporter was on Linux, where this used to be ignored).
|
||||
disabled_by = _env_compile_disabled()
|
||||
if disabled_by is not None:
|
||||
logger.info(
|
||||
"torch.compile skipped: %s is set — using eager mode.", disabled_by,
|
||||
)
|
||||
return False
|
||||
if importlib.util.find_spec("triton") is None:
|
||||
logger.info("torch.compile skipped: Triton unavailable — using eager mode.")
|
||||
return False
|
||||
@@ -258,8 +313,18 @@ def should_torch_compile(device: str) -> bool:
|
||||
from services import settings_store
|
||||
|
||||
if settings_store.get_text(_TORCH_COMPILE_KEY, "0") == "1":
|
||||
logger.info("torch.compile skipped: disabled in Settings (Performance).")
|
||||
logger.info(
|
||||
"torch.compile skipped: disabled in Settings (Performance) [%s].",
|
||||
_settings_db_path(),
|
||||
)
|
||||
return False
|
||||
# #2135: say which DB answered "not disabled". Without this the only
|
||||
# observable outcome of a toggle that never reached the running
|
||||
# backend is a log line saying compile was applied anyway.
|
||||
logger.debug(
|
||||
"torch.compile: %s not set in %s — compile remains eligible.",
|
||||
_TORCH_COMPILE_KEY, _settings_db_path(),
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("should_torch_compile: settings read failed; proceeding")
|
||||
if _compile_runtime_failure is not None:
|
||||
@@ -328,19 +393,31 @@ def build_engine_env(
|
||||
except Exception:
|
||||
logger.exception("build_engine_env: token resolver failed (non-fatal)")
|
||||
|
||||
# INST-12: TORCH_COMPILE_DISABLE on Windows when the user opted in.
|
||||
# The flag is a Windows-only escape hatch — torch.compile OOMs the same
|
||||
# Triton kernel cache differently on macOS/Linux, so injecting on those
|
||||
# platforms would just slow the engine for no gain. (The in-process
|
||||
# should_torch_compile() gate handles the automatic Triton-absence case;
|
||||
# the subprocess var stays user-driven by design — see test_perf_settings.)
|
||||
if sys.platform.startswith("win"):
|
||||
try:
|
||||
from services import settings_store
|
||||
# INST-12 (#65), widened to every platform by #2135: TORCH_COMPILE_DISABLE
|
||||
# when the user opted in. This was win32-only on the theory that
|
||||
# torch.compile only misbehaves on Windows (no Triton wheel). #2135 is the
|
||||
# counter-example — a Linux/CUDA host where compile crashes the engine —
|
||||
# and a Settings toggle that silently does nothing on the user's platform
|
||||
# is worse than no toggle at all. Cost when enabled on Linux/macOS is a
|
||||
# slower engine, which is exactly what the user asked for by enabling it.
|
||||
try:
|
||||
from services import settings_store
|
||||
|
||||
if settings_store.get_text(_TORCH_COMPILE_KEY, "0") == "1":
|
||||
env["TORCH_COMPILE_DISABLE"] = "1"
|
||||
except Exception:
|
||||
logger.exception("build_engine_env: torch_compile_disabled read failed")
|
||||
if settings_store.get_text(_TORCH_COMPILE_KEY, "0") == "1":
|
||||
env["TORCH_COMPILE_DISABLE"] = "1"
|
||||
except Exception:
|
||||
logger.exception("build_engine_env: torch_compile_disabled read failed")
|
||||
|
||||
# #2135: an env opt-out on the parent must reach the child too. Without
|
||||
# this a user who exported TORCH_COMPILE_DISABLE=1 got an eager parent and
|
||||
# a compiled sidecar — the inconsistency that made the flag look ignored.
|
||||
disabled_by = _env_compile_disabled()
|
||||
if disabled_by is not None:
|
||||
if env.get("TORCH_COMPILE_DISABLE") != "1":
|
||||
logger.debug(
|
||||
"build_engine_env: %s is set — disabling torch.compile in the "
|
||||
"engine subprocess too.", disabled_by,
|
||||
)
|
||||
env["TORCH_COMPILE_DISABLE"] = "1"
|
||||
|
||||
return env
|
||||
|
||||
@@ -89,6 +89,7 @@ def bed_mix_filter(
|
||||
duration: str = "longest",
|
||||
tail: str = "",
|
||||
uniq: str = "",
|
||||
bed_gain: float = BED_GAIN,
|
||||
) -> str:
|
||||
"""One ffmpeg filter chain mixing `voice_in` over `bed_in` at original level.
|
||||
|
||||
@@ -110,22 +111,22 @@ def bed_mix_filter(
|
||||
# Gains applied per input, amix reduced to a plain sum: levels are
|
||||
# exact for the whole timeline, including after either stream ends.
|
||||
return (
|
||||
f"[{bed_in}]aresample={BED_MIX_SAMPLE_RATE},{stereo},volume={BED_GAIN:g}[{b}];"
|
||||
f"[{bed_in}]aresample={BED_MIX_SAMPLE_RATE},{stereo},volume={bed_gain:g}[{b}];"
|
||||
f"[{voice_in}]aresample={BED_MIX_SAMPLE_RATE},{stereo},volume={VOICE_GAIN:g}[{v}];"
|
||||
f"[{b}][{v}]amix=inputs=2:duration={duration}:dropout_transition=2:"
|
||||
f"normalize=0,alimiter=level=false:limit=0.98{tail}[{out}]"
|
||||
f"normalize=0,alimiter=level=false:limit=0.98:latency=1{tail}[{out}]"
|
||||
)
|
||||
# Legacy ffmpeg (<5, no `normalize`): cancel amix's normalization with a
|
||||
# compensating multiply. Exact while both streams run; if one ends early
|
||||
# the tail is over-boosted into the limiter until the graph ends — a known
|
||||
# quirk accepted only on old ffmpeg, where the alternative is no export.
|
||||
total = BED_GAIN + VOICE_GAIN
|
||||
total = bed_gain + VOICE_GAIN
|
||||
return (
|
||||
f"[{bed_in}]aresample={BED_MIX_SAMPLE_RATE},{stereo}[{b}];"
|
||||
f"[{voice_in}]aresample={BED_MIX_SAMPLE_RATE},{stereo}[{v}];"
|
||||
f"[{b}][{v}]amix=inputs=2:duration={duration}:dropout_transition=2:"
|
||||
f"weights={BED_GAIN:g} {VOICE_GAIN:g},volume={total:g},"
|
||||
f"alimiter=level=false:limit=0.98{tail}[{out}]"
|
||||
f"weights={bed_gain:g} {VOICE_GAIN:g},volume={total:g},"
|
||||
f"alimiter=level=false:limit=0.98:latency=1{tail}[{out}]"
|
||||
)
|
||||
|
||||
|
||||
|
||||
+214
-30
@@ -62,6 +62,8 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Callable, Optional
|
||||
|
||||
@@ -76,6 +78,87 @@ logger = logging.getLogger("omnivoice.gateway")
|
||||
# life of the process.
|
||||
_POLL_SECONDS = 0.5
|
||||
|
||||
# Remote model installs run as worker prewarms rather than scheduler tasks. Keep
|
||||
# their progress on the control plane so Models can reconnect without pretending
|
||||
# the download stopped as soon as POST /models/install returned.
|
||||
_REMOTE_DOWNLOAD_RETENTION_SECONDS = 60.0
|
||||
_remote_download_lock = threading.Lock()
|
||||
_remote_downloads: dict[tuple[str, str], dict[str, Any]] = {}
|
||||
|
||||
|
||||
def begin_remote_download(target: str, repo_id: str) -> None:
|
||||
with _remote_download_lock:
|
||||
_remote_downloads[(target, repo_id)] = {
|
||||
"repo_id": repo_id,
|
||||
"target": target,
|
||||
"state": "downloading",
|
||||
"phase": "starting",
|
||||
"updated_at": time.monotonic(),
|
||||
}
|
||||
|
||||
|
||||
def record_remote_download_progress(target: str, event: dict) -> None:
|
||||
"""Persist authenticated worker progress for reconnectable model rows."""
|
||||
repo_id = str(event.get("repo_id") or "").strip()
|
||||
if not repo_id:
|
||||
return
|
||||
phase = str(event.get("phase") or "progress")
|
||||
state = (
|
||||
"done"
|
||||
if phase == "install_done"
|
||||
else "failed"
|
||||
if phase == "install_error"
|
||||
else "install_cancelled"
|
||||
if phase in {"cancelled", "install_cancelled"}
|
||||
else "cancelling"
|
||||
if phase == "cancelling"
|
||||
else "downloading"
|
||||
)
|
||||
normalized = {
|
||||
"repo_id": repo_id,
|
||||
"target": target,
|
||||
"state": state,
|
||||
"phase": phase,
|
||||
"updated_at": time.monotonic(),
|
||||
}
|
||||
for source, destination in (
|
||||
("bytes_done", "bytes_done"),
|
||||
("downloaded", "bytes_done"),
|
||||
("total_bytes", "total_bytes"),
|
||||
("total", "total_bytes"),
|
||||
("rate", "rate"),
|
||||
("eta_seconds", "eta_seconds"),
|
||||
("files_done", "files_done"),
|
||||
("files_total", "files_total"),
|
||||
("error", "error"),
|
||||
("docs_topic", "docs_topic"),
|
||||
("failed_at", "failed_at"),
|
||||
("retry_after_seconds", "retry_after_seconds"),
|
||||
):
|
||||
value = event.get(source)
|
||||
if value is not None:
|
||||
normalized[destination] = value
|
||||
with _remote_download_lock:
|
||||
previous = _remote_downloads.get((target, repo_id), {})
|
||||
_remote_downloads[(target, repo_id)] = {**previous, **normalized}
|
||||
|
||||
|
||||
def remote_download_jobs() -> list[dict[str, Any]]:
|
||||
now = time.monotonic()
|
||||
with _remote_download_lock:
|
||||
expired = [
|
||||
key
|
||||
for key, job in _remote_downloads.items()
|
||||
if job.get("state") in {"done", "failed", "cancelled", "install_cancelled"}
|
||||
and now - float(job.get("updated_at") or 0) >= _REMOTE_DOWNLOAD_RETENTION_SECONDS
|
||||
]
|
||||
for key in expired:
|
||||
_remote_downloads.pop(key, None)
|
||||
return [
|
||||
{key: value for key, value in job.items() if key != "updated_at"}
|
||||
for job in _remote_downloads.values()
|
||||
]
|
||||
|
||||
# Consecutive remote failures a multi-unit job tolerates before it stops trying
|
||||
# the remote worker. One is a blip (a dropped stream, a worker restart); two in
|
||||
# a row is a machine that has gone away, and the remaining 160 chapters should
|
||||
@@ -352,7 +435,7 @@ async def prewarm(
|
||||
"""
|
||||
decision = decision or decide(op, control_plane=control_plane)
|
||||
if decision.remote:
|
||||
await preflight(engine, decision, control_plane=control_plane)
|
||||
await preflight(engine, decision, operation=op, control_plane=control_plane)
|
||||
plane = _plane(control_plane)
|
||||
if engine and plane is not None and getattr(plane, "servicer", None) is not None:
|
||||
await plane.servicer.prewarm(decision.worker_id, engine=engine)
|
||||
@@ -495,12 +578,22 @@ async def _run_remote(
|
||||
if scheduler is None:
|
||||
raise _NotDispatched("the control plane has no scheduler")
|
||||
|
||||
await preflight(call.engine, decision, call.model_id, control_plane=plane)
|
||||
await preflight(
|
||||
call.engine,
|
||||
decision,
|
||||
call.model_id,
|
||||
operation=call.operation,
|
||||
control_plane=plane,
|
||||
)
|
||||
|
||||
params = dict(call.params or {})
|
||||
deadline = call.deadline_seconds
|
||||
if deadline is None:
|
||||
deadline = _default_deadline(call.operation, params.get("text"))
|
||||
deadline = _default_deadline(
|
||||
call.operation,
|
||||
params.get("text"),
|
||||
input_seconds=float(params.get("input_seconds") or 0.0),
|
||||
)
|
||||
|
||||
try:
|
||||
submit = getattr(scheduler, "submit_async", None)
|
||||
@@ -547,6 +640,7 @@ async def preflight(
|
||||
decision: Decision,
|
||||
model_id: str = "",
|
||||
*,
|
||||
operation: str = "tts",
|
||||
control_plane=None,
|
||||
) -> None:
|
||||
"""Refuse a positively absent remote model before scheduler admission.
|
||||
@@ -558,7 +652,12 @@ async def preflight(
|
||||
"""
|
||||
if not engine:
|
||||
return
|
||||
target = await status(engine, decision=decision, control_plane=control_plane)
|
||||
target = await status(
|
||||
engine,
|
||||
decision=decision,
|
||||
op=operation,
|
||||
control_plane=control_plane,
|
||||
)
|
||||
for cap in target["models"]:
|
||||
if model_id and cap.get("model_id") not in (model_id, "", None):
|
||||
continue
|
||||
@@ -819,7 +918,7 @@ async def status(
|
||||
"remote": False,
|
||||
"label": decision.label,
|
||||
"reason": decision.reason,
|
||||
"models": _filtered(_local_capabilities(), engine),
|
||||
"models": _filtered(_local_capabilities(), engine, op),
|
||||
}
|
||||
|
||||
plane = _plane(control_plane)
|
||||
@@ -834,14 +933,27 @@ async def status(
|
||||
"remote": False,
|
||||
"label": decision.label,
|
||||
"reason": "the chosen worker is not connected",
|
||||
"models": _filtered(_local_capabilities(), engine),
|
||||
"models": _filtered(_local_capabilities(), engine, op),
|
||||
}
|
||||
models = []
|
||||
capacity = getattr(worker, "capacity", None)
|
||||
for advertised in worker.record.capabilities or []:
|
||||
model = dict(advertised)
|
||||
if capacity is not None:
|
||||
engine_id = str(model.get("engine") or "")
|
||||
model_id = str(model.get("model_id") or "")
|
||||
if engine_id and model_id:
|
||||
# Heartbeats are authoritative for residency. Capability
|
||||
# discovery is refreshed less often and can otherwise leave
|
||||
# Engine Ready green after a worker unloads a model.
|
||||
model["resident"] = capacity.is_resident(engine_id, model_id)
|
||||
models.append(model)
|
||||
return {
|
||||
"target": decision.worker_id,
|
||||
"remote": True,
|
||||
"label": decision.label,
|
||||
"reason": decision.reason,
|
||||
"models": _filtered(list(worker.record.capabilities or []), engine),
|
||||
"models": _filtered(models, engine, op),
|
||||
}
|
||||
|
||||
|
||||
@@ -851,15 +963,39 @@ def _local_capabilities() -> list[dict]:
|
||||
return capabilities.discover(include_unavailable=True)
|
||||
|
||||
|
||||
def _filtered(models: list[dict], engine: Optional[str]) -> list[dict]:
|
||||
if not engine:
|
||||
return models
|
||||
return [m for m in models if m.get("engine") == engine]
|
||||
def _filtered(models: list[dict], engine: Optional[str], operation: str = "tts") -> list[dict]:
|
||||
worker_operation = {
|
||||
"batch": "batch_segments",
|
||||
"dub": "dub_segments",
|
||||
"longform": "audiobook",
|
||||
}.get(operation, operation)
|
||||
return [
|
||||
model
|
||||
for model in models
|
||||
if (not engine or model.get("engine") == engine)
|
||||
and (
|
||||
not model.get("operations")
|
||||
or worker_operation in model.get("operations", ())
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
# ── Download: weights, onto the machine that needs them ────────────────────
|
||||
|
||||
|
||||
def _remote_model_capability(plane, worker_id: str, repo_id: str) -> Optional[dict]:
|
||||
pool = getattr(plane, "pool", None)
|
||||
worker = pool.get(worker_id) if pool is not None else None
|
||||
return next(
|
||||
(
|
||||
capability
|
||||
for capability in (worker.record.capabilities if worker is not None else [])
|
||||
if repo_id in (capability.get("repo_ids") or [])
|
||||
),
|
||||
None,
|
||||
)
|
||||
|
||||
|
||||
async def download(
|
||||
repo_id: str,
|
||||
*,
|
||||
@@ -869,23 +1005,17 @@ async def download(
|
||||
) -> dict:
|
||||
"""Fetch a catalog model onto the target machine.
|
||||
|
||||
Remote downloads are not implemented yet, and this refuses rather than
|
||||
falling back: downloading onto *this* machine when the user asked for the
|
||||
weights on the 4090 leaves the remote box exactly as unprepared, having
|
||||
reported success.
|
||||
Remote downloads are sent to the selected worker and never fall back to
|
||||
this machine: fetching weights onto the wrong host would report success
|
||||
while leaving the selected GPU exactly as unprepared.
|
||||
"""
|
||||
decision = decision or decide(op, control_plane=control_plane)
|
||||
if decision.remote:
|
||||
plane = _plane(control_plane)
|
||||
if plane is None or getattr(plane, "servicer", None) is None:
|
||||
raise RemoteUnsupported(f"{decision.label} is not connected.")
|
||||
live = plane.pool.get(decision.worker_id) if plane.pool is not None else None
|
||||
capability = next(
|
||||
(
|
||||
cap for cap in (live.record.capabilities if live is not None else [])
|
||||
if repo_id in (cap.get("repo_ids") or [])
|
||||
),
|
||||
None,
|
||||
capability = _remote_model_capability(
|
||||
plane, decision.worker_id, repo_id
|
||||
)
|
||||
if capability is None:
|
||||
raise GatewayError(f"Unknown model for {decision.label}: {repo_id!r}.")
|
||||
@@ -898,13 +1028,29 @@ async def download(
|
||||
f"{repo_id!r} must be installed directly on {decision.label}; "
|
||||
"remote sidecar installation is disabled."
|
||||
)
|
||||
sent = await plane.servicer.prewarm(
|
||||
decision.worker_id,
|
||||
engine=str(capability.get("engine") or ""),
|
||||
model_id=str(capability.get("model_id") or ""),
|
||||
download_if_missing=True,
|
||||
)
|
||||
begin_remote_download(decision.worker_id, repo_id)
|
||||
try:
|
||||
sent = await plane.servicer.prewarm(
|
||||
decision.worker_id,
|
||||
engine=str(capability.get("engine") or ""),
|
||||
model_id=str(capability.get("model_id") or ""),
|
||||
download_if_missing=True,
|
||||
)
|
||||
except Exception as exc:
|
||||
record_remote_download_progress(
|
||||
decision.worker_id,
|
||||
{"repo_id": repo_id, "phase": "install_error", "error": str(exc)},
|
||||
)
|
||||
raise
|
||||
if not sent:
|
||||
record_remote_download_progress(
|
||||
decision.worker_id,
|
||||
{
|
||||
"repo_id": repo_id,
|
||||
"phase": "install_error",
|
||||
"error": f"{decision.label} is not connected.",
|
||||
},
|
||||
)
|
||||
raise RemoteUnsupported(f"{decision.label} is not connected.")
|
||||
return {"status": "started", "repo_id": repo_id, "target": decision.worker_id}
|
||||
|
||||
@@ -921,6 +1067,33 @@ async def download(
|
||||
return await install_model(InstallModelRequest(repo_id=repo_id, target="local"))
|
||||
|
||||
|
||||
async def cancel_download(
|
||||
repo_id: str,
|
||||
*,
|
||||
target: str,
|
||||
control_plane=None,
|
||||
) -> dict:
|
||||
"""Cancel an explicit model install on its authenticated remote worker."""
|
||||
plane = _plane(control_plane)
|
||||
servicer = getattr(plane, "servicer", None) if plane is not None else None
|
||||
if servicer is None:
|
||||
raise RemoteUnsupported(f"{target} is not connected.")
|
||||
capability = _remote_model_capability(plane, target, repo_id)
|
||||
if capability is None:
|
||||
raise GatewayError(f"Unknown model for {target}: {repo_id!r}.")
|
||||
model_id = str(capability.get("model_id") or "")
|
||||
if not model_id:
|
||||
raise GatewayError(f"{repo_id!r} has no worker model identifier.")
|
||||
sent = await servicer.cancel_model_install(target, model_id=model_id)
|
||||
if not sent:
|
||||
raise RemoteUnsupported(f"{target} is not connected.")
|
||||
record_remote_download_progress(
|
||||
target,
|
||||
{"repo_id": repo_id, "phase": "cancelling"},
|
||||
)
|
||||
return {"cancelling": repo_id, "target": target}
|
||||
|
||||
|
||||
# ── Plumbing ───────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@@ -936,7 +1109,12 @@ def _plane(control_plane=None):
|
||||
return None
|
||||
|
||||
|
||||
def _default_deadline(operation: str, text: Optional[str]) -> float:
|
||||
def _default_deadline(
|
||||
operation: str,
|
||||
text: Optional[str],
|
||||
*,
|
||||
input_seconds: float = 0.0,
|
||||
) -> float:
|
||||
"""Worst-case wall time for one attempt, from the shared deadline policy.
|
||||
|
||||
Same budget the assignment itself carries, so the awaiting side cannot give
|
||||
@@ -944,7 +1122,13 @@ def _default_deadline(operation: str, text: Optional[str]) -> float:
|
||||
"""
|
||||
from worker import deadlines # noqa: PLC0415
|
||||
|
||||
return float(deadlines.for_task(operation, text=text).total_seconds)
|
||||
return float(
|
||||
deadlines.for_task(
|
||||
operation,
|
||||
text=text,
|
||||
input_seconds=max(0.0, float(input_seconds)),
|
||||
).total_seconds
|
||||
)
|
||||
|
||||
|
||||
def _model_load_timeout() -> float:
|
||||
|
||||
@@ -39,6 +39,8 @@ CURATED_REVISIONS: dict[str, str] = {
|
||||
"csukuangfj/sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23": "204ad334e2e683fd295359930cc16fc0432a23ac",
|
||||
"csukuangfj/sherpa-onnx-whisper-tiny": "65176e2deb88badc814a94058666cadccc29b61c",
|
||||
"pyannote/speaker-diarization-3.1": "84fd25912480287da0247647c3d2b4853cb3ee5d",
|
||||
"pyannote/segmentation-3.0": "e66f3d3b9eb0873085418a7b813d3b369bf160bb",
|
||||
"pyannote/wespeaker-voxceleb-resnet34-LM": "837717ddb9ff5507820346191109dc79c958d614",
|
||||
"OpenMOSS-Team/MOSS-TTS-Nano-100M": "44502f80dbf9743528fa921cc544d662c685ebec",
|
||||
"KittenML/kitten-tts-mini-0.8": "c02725660cea441db4c383af69f1f26f5cd00947",
|
||||
"openbmb/VoxCPM2": "bffb3df5a29440629464e5e839f4d214c8714c3d",
|
||||
|
||||
@@ -48,6 +48,27 @@ def _asr_device() -> str:
|
||||
return "cpu"
|
||||
|
||||
|
||||
def _backend_device(backend: object) -> str:
|
||||
"""Report the device this backend can actually use.
|
||||
|
||||
The host's best device is not evidence that a CPU-only runtime uses it.
|
||||
Prefer load-time facts, then constrain the fallback by the backend's
|
||||
declared compatibility contract.
|
||||
"""
|
||||
for attr in ("_device", "device", "execution_device"):
|
||||
value = getattr(backend, attr, None)
|
||||
if value is not None and not callable(value):
|
||||
text = str(value).strip()
|
||||
if text:
|
||||
return text
|
||||
compat = tuple(getattr(type(backend), "gpu_compat", ("cpu",)))
|
||||
preferred = get_best_device()
|
||||
family = preferred.split(":", 1)[0]
|
||||
if family in compat:
|
||||
return preferred
|
||||
return "cpu" if "cpu" in compat else (compat[0] if compat else "unknown")
|
||||
|
||||
|
||||
def _active_tts_id() -> Optional[str]:
|
||||
"""Configured TTS engine id, or None if it can't be resolved. Attribution
|
||||
is advisory — a prefs/import hiccup must never break /model/loaded."""
|
||||
@@ -169,6 +190,40 @@ def list_loaded() -> dict:
|
||||
logger.warning("Loaded-model inventory unavailable for in-process engines")
|
||||
degraded_sources.append("engines")
|
||||
|
||||
# The local generation path keeps its selected backend in a separate
|
||||
# active-instance slot. Non-OmniVoice models held there must be visible as
|
||||
# well; otherwise Model Settings can report an empty runtime while an
|
||||
# alternate TTS model still occupies memory.
|
||||
try:
|
||||
import services.tts_backend as tb
|
||||
from services.subprocess_backend import SubprocessBackend
|
||||
|
||||
inst = getattr(tb, "_active_instance", None)
|
||||
eid = getattr(tb, "_active_instance_id", None)
|
||||
if (
|
||||
inst is not None
|
||||
and eid
|
||||
and eid != "omnivoice"
|
||||
and not isinstance(inst, SubprocessBackend)
|
||||
and any(
|
||||
getattr(inst, attr, None) is not None
|
||||
for attr in getattr(inst, "_MODEL_ATTRS", ("_model", "_tts"))
|
||||
)
|
||||
):
|
||||
identity = getattr(inst, "model_identity", None)
|
||||
models.append({
|
||||
"id": f"active-engine:{eid}",
|
||||
"name": getattr(inst, "display_name", None) or f"{eid} (engine)",
|
||||
"checkpoint": identity() if callable(identity) else eid,
|
||||
"device": _backend_device(inst),
|
||||
"vram_mb": 0,
|
||||
"unloadable": True,
|
||||
**_tts_attribution(eid, active_tts),
|
||||
})
|
||||
except Exception:
|
||||
logger.warning("Loaded-model inventory unavailable for active TTS engine")
|
||||
degraded_sources.append("active-engine")
|
||||
|
||||
# 6. The warm capture/dictation ASR singleton — resident until idle-released
|
||||
# (#1101 class). Held separately from the co-loaded WhisperX ASR above.
|
||||
try:
|
||||
@@ -176,11 +231,12 @@ def list_loaded() -> dict:
|
||||
|
||||
cap = getattr(ab, "_capture_backend", None)
|
||||
if cap is not None:
|
||||
model_label = getattr(getattr(cap, "spec", None), "label", None)
|
||||
models.append({
|
||||
"id": "capture-asr",
|
||||
"name": f"{type(cap).__name__} (dictation)",
|
||||
"name": f"{model_label or getattr(cap, 'display_name', type(cap).__name__)} (dictation)",
|
||||
"checkpoint": getattr(ab, "_capture_backend_key", None) or type(cap).__name__,
|
||||
"device": get_best_device(),
|
||||
"device": _backend_device(cap),
|
||||
"vram_mb": 0,
|
||||
"unloadable": True,
|
||||
"note": "released after the idle timeout",
|
||||
@@ -189,6 +245,25 @@ def list_loaded() -> dict:
|
||||
logger.warning("Loaded-model inventory unavailable for dictation")
|
||||
degraded_sources.append("dictation")
|
||||
|
||||
# 7. Offline translation can remain resident when the user opts out of the
|
||||
# default post-job release. Keep it visible and manually unloadable.
|
||||
try:
|
||||
from api.routers import dub_translate as dt
|
||||
|
||||
if getattr(dt, "_nllb_model", None) is not None:
|
||||
models.append({
|
||||
"id": "translation:nllb",
|
||||
"name": "NLLB-200 Translation",
|
||||
"checkpoint": dt._NLLB_REPO_ID,
|
||||
"device": str(getattr(dt, "_nllb_device", None) or "cpu"),
|
||||
"vram_mb": 0,
|
||||
"unloadable": True,
|
||||
"note": "released after translation by default",
|
||||
})
|
||||
except Exception:
|
||||
logger.warning("Loaded-model inventory unavailable for translation")
|
||||
degraded_sources.append("translation")
|
||||
|
||||
# System memory snapshot — free/total RAM (and VRAM on a dedicated GPU) plus
|
||||
# a low-memory advisory, so the panel can show pressure instead of leaving
|
||||
# the 16 GB-Mac OOM class invisible until the backend dies.
|
||||
@@ -247,6 +322,14 @@ async def unload(model_id: str) -> dict:
|
||||
return {"unloaded": model_id, "success": True}
|
||||
return {"unloaded": model_id, "success": False, "reason": "in use by dictation"}
|
||||
|
||||
if model_id == "translation:nllb":
|
||||
from api.routers import dub_translate as dt
|
||||
|
||||
if getattr(dt, "_nllb_model", None) is None:
|
||||
return {"unloaded": model_id, "success": False, "reason": "not loaded"}
|
||||
dt._unload_nllb()
|
||||
return {"unloaded": model_id, "success": True}
|
||||
|
||||
# In-process engines (#1247). `list_loaded_models` has advertised these as
|
||||
# `engine:<id>` with `"unloadable": True` since they were made visible in
|
||||
# the panel — but this dispatcher never grew a branch for them, so pressing
|
||||
@@ -270,14 +353,34 @@ async def unload(model_id: str) -> dict:
|
||||
return {"unloaded": model_id, "success": True}
|
||||
return {"unloaded": model_id, "success": False, "reason": "not loaded"}
|
||||
|
||||
if model_id.startswith("active-engine:"):
|
||||
engine_id = model_id.split(":", 1)[1]
|
||||
import services.tts_backend as tb
|
||||
|
||||
if (
|
||||
getattr(tb, "_active_instance", None) is None
|
||||
or getattr(tb, "_active_instance_id", None) != engine_id
|
||||
):
|
||||
return {"unloaded": model_id, "success": False, "reason": "not loaded"}
|
||||
tb.reset_active_backend()
|
||||
return {"unloaded": model_id, "success": True}
|
||||
|
||||
raise ValueError(f"Unknown model id: {model_id}")
|
||||
|
||||
|
||||
async def unload_all() -> dict:
|
||||
"""Release every releasable model — in-process TTS + diarization + all
|
||||
sidecars. Convenience for app shutdown / a global flush."""
|
||||
"""Release shared/alternate TTS, diarisation, sidecars, dictation and translation."""
|
||||
results = {}
|
||||
for mid in ("tts", "diarization", "sidecars"):
|
||||
model_ids = ["tts", "diarization", "sidecars", "capture-asr", "translation:nllb"]
|
||||
try:
|
||||
model_ids.extend(
|
||||
entry["id"]
|
||||
for entry in list_loaded()["models"]
|
||||
if entry["id"].startswith(("engine:", "active-engine:"))
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("Could not enumerate optional engines during global unload")
|
||||
for mid in dict.fromkeys(model_ids):
|
||||
try:
|
||||
results[mid] = await unload(mid)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
|
||||
@@ -1372,7 +1372,7 @@ _last_used = time.time()
|
||||
# Updated by _load_model_sync() so get_model_status() can report
|
||||
# granular progress to the frontend pill.
|
||||
_loading_detail: dict = {
|
||||
"sub_stage": None, # importing | loading_weights | loading_asr | compiling | ready | error
|
||||
"sub_stage": None, # importing | loading_weights | compiling | ready | error
|
||||
"detail": "", # human-readable description
|
||||
"error": None, # error message string if failed
|
||||
"progress": None, # 0-100 percentage (None = indeterminate)
|
||||
@@ -1793,6 +1793,64 @@ _TORCH_COMPILE_MODE = "reduce-overhead"
|
||||
# would not.
|
||||
_CUDAGRAPH_COMPILE_MODES = frozenset({"reduce-overhead", "max-autotune"})
|
||||
|
||||
# ── #2135: CUDA-graph capture needs Ampere or newer ─────────────────────────
|
||||
# On a Turing T4 (sm_75) the cudagraph mode above took the whole backend
|
||||
# process down on the first generate — no Python traceback, no HTTP response,
|
||||
# just a dead PID (the native capture aborts below the interpreter, so neither
|
||||
# the #278 eager fallback nor any `except` can see it). The graph *capture* is
|
||||
# the risky part, not Inductor: dropping to the non-cudagraph "default" mode
|
||||
# keeps the compiled kernels (and most of the speedup) while removing the
|
||||
# crash surface. Ampere (sm_80) is the floor because that is where the app has
|
||||
# actual passing evidence; anything older takes the conservative path.
|
||||
_CUDAGRAPH_MIN_CAPABILITY = (8, 0)
|
||||
# Escape hatch in the other direction, for operators benchmarking on old GPUs.
|
||||
_FORCE_CUDAGRAPH_ENV = "OMNIVOICE_FORCE_CUDAGRAPH"
|
||||
|
||||
|
||||
def _resolve_compile_mode() -> str:
|
||||
"""The ``torch.compile`` mode to use on this GPU (#2135).
|
||||
|
||||
Returns the configured cudagraph mode on Ampere+, and the non-cudagraph
|
||||
``"default"`` on older architectures where graph capture has been observed
|
||||
to abort the process. Fails *safe* (→ "default") only when we positively
|
||||
identify a pre-Ampere device; any probe error keeps the configured mode so
|
||||
a weird torch build doesn't silently lose the optimization.
|
||||
"""
|
||||
if _TORCH_COMPILE_MODE not in _CUDAGRAPH_COMPILE_MODES:
|
||||
return _TORCH_COMPILE_MODE
|
||||
if os.environ.get(_FORCE_CUDAGRAPH_ENV, "").strip().lower() in {"1", "true", "yes", "on"}:
|
||||
logger.warning(
|
||||
"%s=1 — keeping torch.compile mode %r on a GPU where CUDA-graph "
|
||||
"capture is not known-good (#2135).",
|
||||
_FORCE_CUDAGRAPH_ENV, _TORCH_COMPILE_MODE,
|
||||
)
|
||||
return _TORCH_COMPILE_MODE
|
||||
try:
|
||||
import torch
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
return _TORCH_COMPILE_MODE
|
||||
capability = torch.cuda.get_device_capability(0)
|
||||
except Exception:
|
||||
logger.debug("compile-mode capability probe failed; keeping %r",
|
||||
_TORCH_COMPILE_MODE, exc_info=True)
|
||||
return _TORCH_COMPILE_MODE
|
||||
if tuple(capability) >= _CUDAGRAPH_MIN_CAPABILITY:
|
||||
return _TORCH_COMPILE_MODE
|
||||
try:
|
||||
device_name = torch.cuda.get_device_name(0)
|
||||
except Exception:
|
||||
device_name = "this GPU"
|
||||
logger.info(
|
||||
"torch.compile mode %r downgraded to 'default' on %s (sm_%d%d): CUDA-graph "
|
||||
"capture below sm_%d%d has been seen to abort the backend process (#2135). "
|
||||
"Compiled kernels are still used. Set %s=1 to override.",
|
||||
_TORCH_COMPILE_MODE, device_name, capability[0], capability[1],
|
||||
_CUDAGRAPH_MIN_CAPABILITY[0], _CUDAGRAPH_MIN_CAPABILITY[1],
|
||||
_FORCE_CUDAGRAPH_ENV,
|
||||
)
|
||||
return "default"
|
||||
|
||||
_compiled_inference_executor: "ThreadPoolExecutor | None" = None
|
||||
_compiled_inference_thread_ident: "int | None" = None
|
||||
|
||||
@@ -1851,6 +1909,15 @@ def _set_loading(sub_stage: str, detail: str = "", error: str | None = None, pro
|
||||
_loading_detail["detail"] = detail
|
||||
_loading_detail["error"] = error
|
||||
_loading_detail["progress"] = progress
|
||||
# Model state is a declared real-time event. Emit only at these explicit
|
||||
# lifecycle transitions; high-frequency Hugging Face byte progress updates
|
||||
# write the dict directly and remain covered by the active one-second poll.
|
||||
try:
|
||||
from core import event_bus
|
||||
|
||||
event_bus.emit("model_status", {"sub_stage": sub_stage})
|
||||
except Exception:
|
||||
logger.debug("Could not publish model status", exc_info=True)
|
||||
|
||||
|
||||
def _env_flag(name: str, default: bool = False) -> bool:
|
||||
@@ -2577,8 +2644,11 @@ def _load_model_sync():
|
||||
|
||||
if not flashinfer_applied and should_torch_compile(device):
|
||||
_set_loading("compiling", "Compiling model (torch.compile)…")
|
||||
# #2135: resolved per-GPU — pre-Ampere drops to the
|
||||
# non-cudagraph mode rather than risking a native abort.
|
||||
compile_mode = _resolve_compile_mode()
|
||||
try:
|
||||
_model.llm = torch.compile(_model.llm, mode=_TORCH_COMPILE_MODE)
|
||||
_model.llm = torch.compile(_model.llm, mode=compile_mode)
|
||||
except Exception as compile_exc:
|
||||
# #278: compile is an optimization, never a point of
|
||||
# failure — keep the eager model and remember the failure
|
||||
@@ -2595,7 +2665,7 @@ def _load_model_sync():
|
||||
# archs, #278). Wrap generate so that falls back to eager
|
||||
# instead of failing the generation.
|
||||
_install_compile_fallback(_model)
|
||||
if _TORCH_COMPILE_MODE in _CUDAGRAPH_COMPILE_MODES:
|
||||
if compile_mode in _CUDAGRAPH_COMPILE_MODES:
|
||||
# #315: reduce-overhead uses CUDA graphs, whose
|
||||
# captured state is thread-local. Pin all inference to
|
||||
# one dedicated thread so a later render dispatched to
|
||||
@@ -2606,12 +2676,30 @@ def _load_model_sync():
|
||||
logger.info(
|
||||
"torch.compile mode %r uses CUDA graphs — compiled-model "
|
||||
"inference pinned to a single dedicated thread (#315).",
|
||||
_TORCH_COMPILE_MODE,
|
||||
compile_mode,
|
||||
)
|
||||
logger.info("torch.compile applied.")
|
||||
logger.info("torch.compile applied (mode=%r).", compile_mode)
|
||||
except Exception as e:
|
||||
logger.info("torch.compile skipped: %s", e)
|
||||
|
||||
# Bind status identity to the object that actually finished loading.
|
||||
# Resolving preferences later can name a newly-selected checkpoint
|
||||
# while the previous one is still resident, and process-global load
|
||||
# metadata can be overwritten by a loader that completed after its
|
||||
# caller timed out. Instance metadata keeps /model/status honest.
|
||||
try:
|
||||
setattr(_model, "_voicestudio_checkpoint", checkpoint)
|
||||
setattr(
|
||||
_model,
|
||||
"_voicestudio_loaded_at",
|
||||
time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
)
|
||||
except Exception:
|
||||
# OmniVoice is an ordinary nn.Module and accepts attributes, but
|
||||
# a future slotted/proxied model must still be usable. Status falls
|
||||
# back to the effective configured checkpoint below.
|
||||
logger.debug("Could not attach resident model identity", exc_info=True)
|
||||
|
||||
_set_loading("ready", "Model ready", progress=100)
|
||||
logger.info("VoiceStudio model loaded successfully.")
|
||||
return _model
|
||||
@@ -3030,20 +3118,41 @@ def get_model_status():
|
||||
is_loading = False
|
||||
|
||||
status = "loading" if is_loading else ("ready" if is_loaded else "idle")
|
||||
checkpoint = None
|
||||
loaded_at = None
|
||||
if is_loaded:
|
||||
checkpoint = getattr(model, "_voicestudio_checkpoint", None)
|
||||
loaded_at = getattr(model, "_voicestudio_loaded_at", None)
|
||||
if not checkpoint:
|
||||
try:
|
||||
checkpoint = resolve_omnivoice_checkpoint()
|
||||
except Exception:
|
||||
# Status is a recovery surface. A broken preferences layer
|
||||
# must not turn a resident-model query into a 500.
|
||||
logger.debug("Could not resolve resident model identity", exc_info=True)
|
||||
|
||||
result = {
|
||||
"loaded": is_loaded,
|
||||
"loading": is_loading,
|
||||
"status": status,
|
||||
}
|
||||
# Attach sub-stage detail when loading or after an error
|
||||
if checkpoint is not None:
|
||||
result["checkpoint"] = checkpoint
|
||||
if loaded_at is not None:
|
||||
result["loaded_at"] = loaded_at
|
||||
# Attach sub-stage detail only while it describes the current resident/load
|
||||
# state, or when a failure must remain actionable. A completed model can be
|
||||
# unloaded while the last successful "ready" detail remains in memory.
|
||||
# Publishing that stale detail alongside status=idle/loaded=false gives
|
||||
# clients two contradictory readiness states.
|
||||
sub = _loading_detail.get("sub_stage")
|
||||
if sub:
|
||||
err = _loading_detail.get("error")
|
||||
if sub and (is_loading or is_loaded or err):
|
||||
result["sub_stage"] = sub
|
||||
result["detail"] = _loading_detail.get("detail", "")
|
||||
progress = _loading_detail.get("progress")
|
||||
if progress is not None:
|
||||
result["progress"] = progress
|
||||
err = _loading_detail.get("error")
|
||||
if err:
|
||||
result["error"] = err
|
||||
return result
|
||||
@@ -3430,6 +3539,7 @@ _diar_pipeline = None
|
||||
DIARIZATION_ERR_NO_TOKEN = "NO_TOKEN"
|
||||
DIARIZATION_ERR_LICENSE = "PYANNOTE_LICENSE_REQUIRED"
|
||||
DIARIZATION_ERR_LOAD = "LOAD_FAILED"
|
||||
DIARIZATION_ERR_MISSING = "MODEL_MISSING"
|
||||
|
||||
|
||||
def _classify_diarization_error(exc: BaseException) -> str:
|
||||
@@ -3446,13 +3556,14 @@ def _classify_diarization_error(exc: BaseException) -> str:
|
||||
"""
|
||||
name = type(exc).__name__.lower()
|
||||
msg = str(exc).lower()
|
||||
if "localentrynotfounderror" in name or isinstance(exc, FileNotFoundError):
|
||||
return DIARIZATION_ERR_MISSING
|
||||
if (
|
||||
"401" in msg
|
||||
or "403" in msg
|
||||
or "unauthorized" in msg
|
||||
or "gated" in msg
|
||||
or "accept" in msg and ("license" in msg or "terms" in msg or "user conditions" in msg)
|
||||
or "hfhubhttperror" in name
|
||||
or "gatedrepoerror" in name
|
||||
or "repositorynotfounderror" in name and "gated" in msg
|
||||
):
|
||||
@@ -3513,6 +3624,15 @@ def get_diarization_pipeline(return_error: bool = False):
|
||||
a docs deeplink — issue #78.
|
||||
"""
|
||||
global _diar_pipeline
|
||||
from services.diarization_runtime import SORTFORMER, selected_backend
|
||||
if selected_backend() == SORTFORMER:
|
||||
try:
|
||||
from services.diarization_native import NativeSortformer
|
||||
pipeline = NativeSortformer()
|
||||
return (pipeline, None) if return_error else pipeline
|
||||
except Exception as exc:
|
||||
logger.exception("Could not prepare native Sortformer")
|
||||
return (None, _classify_diarization_error(exc)) if return_error else None
|
||||
if _diar_pipeline is not None:
|
||||
return (_diar_pipeline, None) if return_error else _diar_pipeline
|
||||
|
||||
@@ -3521,9 +3641,9 @@ def get_diarization_pipeline(return_error: bool = False):
|
||||
# reads HF tokens, and that place is `token_resolver.resolve()`.
|
||||
from services import token_resolver
|
||||
resolved = token_resolver.resolve()
|
||||
if not resolved:
|
||||
return (None, DIARIZATION_ERR_NO_TOKEN) if return_error else None
|
||||
hf_token = resolved.token
|
||||
# Access is checked during explicit installation. An already-installed
|
||||
# local bundle remains usable after a token expires or is removed.
|
||||
hf_token = resolved.token if resolved else False
|
||||
try:
|
||||
torch = _lazy_torch()
|
||||
_ensure_pyannote_hf_token_compat() # #167: use_auth_token -> token
|
||||
@@ -3541,16 +3661,41 @@ def get_diarization_pipeline(return_error: bool = False):
|
||||
logger.debug("pyannote safe-globals allowlist skipped: %s", _glob_e)
|
||||
from pyannote.audio import Pipeline
|
||||
logger.info("Loading Pyannote Diarization Pipeline...")
|
||||
_diar_pipeline = Pipeline.from_pretrained("pyannote/speaker-diarization-3.1", use_auth_token=hf_token)
|
||||
from services.diarization_local import local_pipeline_config
|
||||
with local_pipeline_config() as config_path:
|
||||
pipeline = Pipeline.from_pretrained(config_path, use_auth_token=hf_token)
|
||||
if pipeline is None:
|
||||
raise RuntimeError("The installed diarisation pipeline could not be loaded")
|
||||
device = get_best_device()
|
||||
# Pyannote supports CUDA and CPU; route XPU/DirectML to CPU
|
||||
if device in ("cuda",):
|
||||
_diar_pipeline.to(torch.device(device))
|
||||
pipeline.to(torch.device(device))
|
||||
_diar_pipeline = pipeline
|
||||
logger.info("Pyannote Diarization Pipeline loaded on %s.", device)
|
||||
return (_diar_pipeline, None) if return_error else _diar_pipeline
|
||||
except Exception as e:
|
||||
err_class = _classify_diarization_error(e)
|
||||
# Without a token, a missing local bundle means the user must connect
|
||||
# Hugging Face before the explicit install can run. Once installed,
|
||||
# local_pipeline_config succeeds and diarisation remains fully local.
|
||||
if resolved is None and err_class == DIARIZATION_ERR_MISSING:
|
||||
err_class = DIARIZATION_ERR_NO_TOKEN
|
||||
logger.exception(
|
||||
"Failed to load Pyannote pipeline (class=%s)", err_class,
|
||||
)
|
||||
return (None, err_class) if return_error else None
|
||||
|
||||
|
||||
def unload_diarization_pipeline() -> bool:
|
||||
"""Release a resident pyannote pipeline after the runtime changes."""
|
||||
global _diar_pipeline
|
||||
pipeline = _diar_pipeline
|
||||
_diar_pipeline = None
|
||||
if pipeline is None:
|
||||
return False
|
||||
del pipeline
|
||||
try:
|
||||
free_vram()
|
||||
except Exception:
|
||||
logger.debug("Could not clear accelerator cache after diarisation unload", exc_info=True)
|
||||
return True
|
||||
|
||||
@@ -0,0 +1,463 @@
|
||||
"""Shared local performance preferences and runtime defaults. No model downloads."""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import os
|
||||
|
||||
_PERFORMANCE_PROFILE_KEY = "performance_profile"
|
||||
_PERFORMANCE_TIERS = ("fast", "balanced", "quality", "max")
|
||||
_PERFORMANCE_FAMILIES = (
|
||||
"tts",
|
||||
"asr",
|
||||
"dictation",
|
||||
"diarisation",
|
||||
"translation",
|
||||
"llm",
|
||||
)
|
||||
|
||||
# Advertise only implemented runtime controls, never speculative model switches.
|
||||
_PERFORMANCE_TARGETS = {
|
||||
"tts": {
|
||||
"fast": {"steps": 8, "postprocess": False},
|
||||
"balanced": {"steps": 16, "postprocess": True},
|
||||
"quality": {"steps": 32, "postprocess": True},
|
||||
"max": {"steps": 64, "postprocess": True, "model_policy": "largest-installed-compatible"},
|
||||
},
|
||||
"asr": {
|
||||
tier: {"beam_size": width, "best_of": width, "engine": "faster-whisper"}
|
||||
for tier, width in zip(_PERFORMANCE_TIERS, (1, 3, 5, 8))
|
||||
},
|
||||
"dictation": {
|
||||
"fast": {"decoding_method": "greedy_search", "max_active_paths": 1, "engine": "sherpa-onnx"},
|
||||
"balanced": {"decoding_method": "greedy_search", "max_active_paths": 4, "engine": "sherpa-onnx"},
|
||||
"quality": {"decoding_method": "modified_beam_search", "max_active_paths": 4, "engine": "sherpa-onnx"},
|
||||
"max": {"decoding_method": "modified_beam_search", "max_active_paths": 8, "engine": "sherpa-onnx"},
|
||||
},
|
||||
"diarisation": {
|
||||
"fast": {"engine": "audiocpp-sortformer"},
|
||||
"balanced": {"engine": "audiocpp-sortformer"},
|
||||
"quality": {"engine": "pyannote"},
|
||||
"max": {"engine": "pyannote"},
|
||||
},
|
||||
"translation": {
|
||||
"fast": {"num_beams": 1, "engine": "argos"},
|
||||
"balanced": {"num_beams": 3, "engine": "argos"},
|
||||
"quality": {"num_beams": 5, "engine": "nllb"},
|
||||
"max": {"num_beams": 8, "engine": "nllb"},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
_TIER_POSITION = {"fast": 0.0, "balanced": 0.5, "quality": 0.8, "max": 1.0}
|
||||
|
||||
|
||||
def _tier_choice(items: list, tier: str, *, size) -> object | None:
|
||||
"""Pick an installed model along the user's speed/quality continuum."""
|
||||
if not items:
|
||||
return None
|
||||
ordered = sorted(items, key=lambda item: (float(size(item) or 0), str(item)))
|
||||
position = _TIER_POSITION.get(tier, _TIER_POSITION["balanced"])
|
||||
index = math.floor(position * (len(ordered) - 1) + 0.5)
|
||||
return ordered[index]
|
||||
|
||||
|
||||
def _installed_ct2_models() -> list[dict]:
|
||||
"""Installed CTranslate2 Whisper models usable by the shared ASR runtime."""
|
||||
from api.routers.setup.models import (
|
||||
KNOWN_MODELS,
|
||||
_model_supported,
|
||||
cache_is_complete,
|
||||
is_cached,
|
||||
)
|
||||
|
||||
return [
|
||||
model
|
||||
for model in KNOWN_MODELS
|
||||
if str(model.get("role", "")).lower() == "asr"
|
||||
and not model.get("dictation_id")
|
||||
and (
|
||||
str(model.get("repo_id", "")).startswith("Systran/faster-")
|
||||
or model.get("repo_id") == "deepdml/faster-whisper-large-v3-turbo-ct2"
|
||||
)
|
||||
and _model_supported(model)
|
||||
and is_cached(model["repo_id"])
|
||||
and cache_is_complete(model)
|
||||
]
|
||||
|
||||
|
||||
def _faster_whisper_backend() -> str | None:
|
||||
from services import asr_backend
|
||||
|
||||
if asr_backend._probe_available(asr_backend.FasterWhisperBackend):
|
||||
return "faster-whisper"
|
||||
row = next(
|
||||
(
|
||||
item
|
||||
for item in asr_backend.list_backends()
|
||||
if item["id"] == "faster-whisper-isolated"
|
||||
),
|
||||
None,
|
||||
)
|
||||
return (
|
||||
"faster-whisper-isolated"
|
||||
if row and row.get("available") and row.get("routing_status") != "unavailable"
|
||||
else None
|
||||
)
|
||||
|
||||
|
||||
def _dictation_supports_locale(spec, language: str | None) -> bool:
|
||||
if not language or spec.id == "sherpa-whisper-tiny":
|
||||
return True
|
||||
if spec.id == "sherpa-parakeet-tdt-v3":
|
||||
from services.asr_backend import _PARAKEET_MLX_LANGS
|
||||
|
||||
return language in _PARAKEET_MLX_LANGS
|
||||
if spec.id in {"sherpa-parakeet-tdt-v2", "sherpa-zipformer-en-20m"}:
|
||||
return language == "en"
|
||||
if spec.id == "sherpa-zipformer-zh-14m":
|
||||
return language == "zh"
|
||||
if spec.id in {
|
||||
"sherpa-zipformer-bilingual-zh-en",
|
||||
"sherpa-paraformer-bilingual-zh-en",
|
||||
}:
|
||||
return language in {"en", "zh"}
|
||||
return True
|
||||
|
||||
|
||||
def _installed_dictation_models() -> list:
|
||||
from services import asr_backend, sherpa_dictation
|
||||
|
||||
language = asr_backend._locale_language()
|
||||
installed = [
|
||||
spec
|
||||
for spec in sherpa_dictation.list_specs()
|
||||
if sherpa_dictation.is_installed(spec)
|
||||
and not sherpa_dictation.is_demoted(spec.id)
|
||||
]
|
||||
compatible = [
|
||||
spec for spec in installed if _dictation_supports_locale(spec, language)
|
||||
]
|
||||
# A machine without a usable locale should still recover to an explicitly
|
||||
# installed model instead of claiming no speech model exists.
|
||||
return compatible or installed
|
||||
|
||||
|
||||
def _activate_asr_model(tier: str) -> dict | None:
|
||||
from core import prefs
|
||||
from services import asr_backend
|
||||
|
||||
if os.environ.get("OMNIVOICE_ASR_BACKEND") or prefs.is_env_shadowed(
|
||||
"ASR_MODEL_FASTER"
|
||||
):
|
||||
return None
|
||||
model = _tier_choice(
|
||||
_installed_ct2_models(), tier, size=lambda item: item.get("size_gb")
|
||||
)
|
||||
backend_id = _faster_whisper_backend()
|
||||
if model is None or backend_id is None:
|
||||
return None
|
||||
repo_id = str(model["repo_id"])
|
||||
if asr_backend.faster_whisper_model_id() != repo_id:
|
||||
asr_backend.select_faster_whisper_model(repo_id)
|
||||
if asr_backend.active_backend_id() != backend_id:
|
||||
prefs.set_("asr_backend", backend_id)
|
||||
return {"engine": backend_id, "model": repo_id}
|
||||
|
||||
|
||||
def _activate_dictation_model(tier: str) -> dict | None:
|
||||
from core import prefs
|
||||
from services import asr_backend, sherpa_dictation
|
||||
|
||||
if os.environ.get("OMNIVOICE_SHERPA_ASR_MODEL"):
|
||||
return None
|
||||
available, _ = sherpa_dictation.sherpa_available()
|
||||
if not available:
|
||||
return None
|
||||
model = _tier_choice(
|
||||
_installed_dictation_models(), tier, size=lambda item: item.size_gb
|
||||
)
|
||||
if model is None:
|
||||
return None
|
||||
if prefs.get("dictation.model_id") != model.id:
|
||||
prefs.set_("dictation.model_id", model.id)
|
||||
asr_backend._capture_backend = None
|
||||
asr_backend._capture_backend_key = None
|
||||
return {"engine": model.kind, "model": model.id}
|
||||
|
||||
|
||||
def _activate_translation_model(tier: str) -> dict | None:
|
||||
from core import prefs
|
||||
from services import translation_engines
|
||||
|
||||
current = str(prefs.get("translation_backend", "argos"))
|
||||
# Keep a usable explicitly chosen network provider. A stale provider whose
|
||||
# package/key disappeared must not strand Dubbing while an installed local
|
||||
# translator is ready.
|
||||
if current not in {"argos", "nllb"} and translation_engines.is_ready(current):
|
||||
return None
|
||||
target = str(_PERFORMANCE_TARGETS["translation"][tier]["engine"])
|
||||
if not translation_engines.is_ready(target):
|
||||
target = next(
|
||||
(
|
||||
candidate
|
||||
for candidate in ("argos", "nllb")
|
||||
if translation_engines.is_ready(candidate)
|
||||
),
|
||||
"",
|
||||
)
|
||||
if not target:
|
||||
return None
|
||||
if current != target:
|
||||
prefs.set_("translation_backend", target)
|
||||
return {
|
||||
"engine": target,
|
||||
"model": "facebook/nllb-200-distilled-600M" if target == "nllb" else target,
|
||||
}
|
||||
|
||||
|
||||
def _installed_selectable_families() -> set[str]:
|
||||
from services import sherpa_dictation, translation_engines
|
||||
|
||||
families: set[str] = set()
|
||||
if _installed_ct2_models() and _faster_whisper_backend():
|
||||
families.add("asr")
|
||||
sherpa_available, _ = sherpa_dictation.sherpa_available()
|
||||
if sherpa_available and _installed_dictation_models():
|
||||
families.add("dictation")
|
||||
if translation_engines.is_ready("nllb"):
|
||||
families.add("translation")
|
||||
return families
|
||||
|
||||
|
||||
def _activate_installed_models(tier: str, family: str | None) -> dict[str, dict]:
|
||||
requested = set(_PERFORMANCE_FAMILIES if family is None else (family,))
|
||||
activated: dict[str, dict] = {}
|
||||
selectors = {
|
||||
"asr": _activate_asr_model,
|
||||
"dictation": _activate_dictation_model,
|
||||
"translation": _activate_translation_model,
|
||||
}
|
||||
for name, select in selectors.items():
|
||||
if name in requested:
|
||||
result = select(tier)
|
||||
if result:
|
||||
activated[name] = result
|
||||
return activated
|
||||
|
||||
|
||||
|
||||
def profile_state() -> dict:
|
||||
from core import prefs
|
||||
from services import asr_backend, diarization_runtime
|
||||
from services.sherpa_dictation import get_spec as dictation_spec
|
||||
from services.tts_backend import active_backend_id as active_tts
|
||||
|
||||
selected_dictation = (
|
||||
dictation_spec(str(prefs.get("dictation.model_id", "")))
|
||||
if prefs.get("dictation.enabled", True)
|
||||
else None
|
||||
)
|
||||
diarisation_choices = diarization_runtime.installed_backends()
|
||||
tts_engine = active_tts()
|
||||
asr_engine = asr_backend.active_backend_id()
|
||||
translation_engine = str(prefs.get("translation_backend", "argos"))
|
||||
|
||||
active_engines = {
|
||||
"tts": tts_engine,
|
||||
"asr": asr_engine,
|
||||
"translation": translation_engine,
|
||||
"dictation": selected_dictation.kind if selected_dictation else "inactive",
|
||||
}
|
||||
supported_engines = {
|
||||
"tts": {"omnivoice", "omnivoice-isolated"},
|
||||
"asr": {"faster-whisper", "faster-whisper-isolated"},
|
||||
"translation": {"nllb"},
|
||||
"dictation": {"offline-transducer", "online-transducer"},
|
||||
}
|
||||
|
||||
stored = prefs.get(_PERFORMANCE_PROFILE_KEY, {})
|
||||
raw = stored if isinstance(stored, dict) else {}
|
||||
global_tier = str(raw.get("global", "balanced")).lower()
|
||||
if global_tier not in _PERFORMANCE_TIERS:
|
||||
global_tier = "balanced"
|
||||
overrides = {
|
||||
str(family): str(tier)
|
||||
for family, tier in (raw.items() if isinstance(raw, dict) else [])
|
||||
if family in _PERFORMANCE_FAMILIES and tier in _PERFORMANCE_TIERS
|
||||
}
|
||||
effective = {
|
||||
family: overrides.get(family, global_tier) for family in _PERFORMANCE_FAMILIES
|
||||
}
|
||||
applicable_families = [
|
||||
family
|
||||
for family, engines in supported_engines.items()
|
||||
if active_engines[family] in engines
|
||||
]
|
||||
for family in _installed_selectable_families():
|
||||
if family not in applicable_families:
|
||||
applicable_families.append(family)
|
||||
if len(diarisation_choices) > 1:
|
||||
applicable_families.append("diarisation")
|
||||
selections = {
|
||||
"tts": {
|
||||
"engine": tts_engine,
|
||||
# OmniVoice has one checkpoint family today; its performance tiers
|
||||
# tune sampling rather than silently changing voice capabilities.
|
||||
"model": "k2-fsa/OmniVoice"
|
||||
if tts_engine in {"omnivoice", "omnivoice-isolated", "omnivoice-subprocess"}
|
||||
else tts_engine,
|
||||
},
|
||||
"asr": {
|
||||
"engine": asr_engine,
|
||||
"model": asr_backend.faster_whisper_model_id()
|
||||
if asr_engine in {"faster-whisper", "faster-whisper-isolated"}
|
||||
else asr_engine,
|
||||
},
|
||||
"dictation": {
|
||||
"engine": selected_dictation.kind if selected_dictation else "inactive",
|
||||
"model": selected_dictation.id if selected_dictation else None,
|
||||
"label": selected_dictation.label if selected_dictation else None,
|
||||
},
|
||||
"diarisation": {
|
||||
"engine": diarization_runtime.selected_backend()
|
||||
if diarisation_choices
|
||||
else "inactive",
|
||||
"model": (
|
||||
diarization_runtime.SORTFORMER_REPO
|
||||
if diarization_runtime.selected_backend() == diarization_runtime.SORTFORMER
|
||||
else "pyannote/speaker-diarization-3.1"
|
||||
)
|
||||
if diarisation_choices
|
||||
else None,
|
||||
},
|
||||
"translation": {
|
||||
"engine": translation_engine,
|
||||
"model": "facebook/nllb-200-distilled-600M"
|
||||
if translation_engine == "nllb"
|
||||
else translation_engine,
|
||||
},
|
||||
"llm": {"engine": "inactive", "model": None},
|
||||
}
|
||||
return {
|
||||
"global": global_tier,
|
||||
"overrides": overrides,
|
||||
"effective": effective,
|
||||
"tiers": list(_PERFORMANCE_TIERS),
|
||||
"families": list(_PERFORMANCE_FAMILIES),
|
||||
"implemented_families": list(_PERFORMANCE_TARGETS),
|
||||
"applicable_families": applicable_families,
|
||||
"targets": {
|
||||
family: _PERFORMANCE_TARGETS[family][effective[family]]
|
||||
for family in _PERFORMANCE_TARGETS
|
||||
},
|
||||
"selections": selections,
|
||||
"downloads_started": False,
|
||||
}
|
||||
|
||||
|
||||
|
||||
def requested_tier(family: str) -> str | None:
|
||||
"""None preserves existing workflow defaults until a user picks a preset."""
|
||||
from core import prefs
|
||||
stored = prefs.get(_PERFORMANCE_PROFILE_KEY, {})
|
||||
if not isinstance(stored, dict):
|
||||
return None
|
||||
tier = stored.get(family, stored.get("global"))
|
||||
return tier if tier in _PERFORMANCE_TIERS else None
|
||||
|
||||
|
||||
def activate_maximum_capacity_models(family: str | None = None) -> dict:
|
||||
"""Select the strongest already-installed compatible local models.
|
||||
|
||||
This is intentionally download-free. Choosing Max is explicit permission to
|
||||
change model selections, but model installation remains its own reviewable
|
||||
action in the catalogue.
|
||||
"""
|
||||
return _activate_installed_models("max", family)
|
||||
|
||||
|
||||
def activate_performance_tier(tier: str, family: str | None = None) -> dict:
|
||||
"""Apply installed-only model/runtime selections implied by a preset."""
|
||||
requested = set(_PERFORMANCE_FAMILIES if family is None else (family,))
|
||||
activated = _activate_installed_models(tier, family)
|
||||
|
||||
if "diarisation" in requested and not os.environ.get(
|
||||
"OMNIVOICE_DIARIZATION_BACKEND"
|
||||
):
|
||||
from services import diarization_runtime
|
||||
|
||||
installed = diarization_runtime.installed_backends()
|
||||
if len(installed) > 1:
|
||||
engine = _PERFORMANCE_TARGETS["diarisation"][tier]["engine"]
|
||||
if engine in installed:
|
||||
diarization_runtime.select_backend(engine)
|
||||
if engine == diarization_runtime.SORTFORMER:
|
||||
from services import model_manager
|
||||
|
||||
model_manager.unload_diarization_pipeline()
|
||||
activated["diarisation"] = {"engine": engine}
|
||||
return activated
|
||||
|
||||
|
||||
def reconcile_active_profile() -> dict[str, dict]:
|
||||
"""Reapply a persisted profile after installs or an app restart.
|
||||
|
||||
Older builds persisted the slider but selected models only for Max. That
|
||||
left installed ASR/Dictation models stranded behind stale missing choices.
|
||||
Reconciliation is startup-only, installed-only, and never downloads.
|
||||
"""
|
||||
from core import prefs
|
||||
|
||||
# The UI presents Balanced as the selected initial value, so the runtime
|
||||
# must honor it even before the user changes the control for the first time.
|
||||
stored = prefs.get(_PERFORMANCE_PROFILE_KEY, {})
|
||||
if not isinstance(stored, dict):
|
||||
return {}
|
||||
global_tier = str(stored.get("global", "balanced")).lower()
|
||||
if global_tier not in _PERFORMANCE_TIERS:
|
||||
global_tier = "balanced"
|
||||
activated: dict[str, dict] = {}
|
||||
for family in _PERFORMANCE_TARGETS:
|
||||
tier = str(stored.get(family, global_tier)).lower()
|
||||
if tier not in _PERFORMANCE_TIERS:
|
||||
tier = global_tier
|
||||
activated.update(activate_performance_tier(tier, family))
|
||||
return activated
|
||||
|
||||
|
||||
def tts_defaults(engine: str = "omnivoice") -> dict:
|
||||
"""Only map sampling controls verified for the selected engine family."""
|
||||
tier = requested_tier("tts")
|
||||
if tier is None or engine not in {"omnivoice", "omnivoice-isolated"}:
|
||||
return {}
|
||||
target = _PERFORMANCE_TARGETS["tts"][tier]
|
||||
return {"num_step": target["steps"], "postprocess_output": target["postprocess"]}
|
||||
|
||||
|
||||
def asr_decode_defaults() -> dict:
|
||||
"""Bound Faster-Whisper's search effort without changing language coverage."""
|
||||
tier = requested_tier("asr")
|
||||
if tier is None:
|
||||
return {}
|
||||
target = _PERFORMANCE_TARGETS["asr"][tier]
|
||||
return {"beam_size": target["beam_size"], "best_of": target["best_of"]}
|
||||
|
||||
|
||||
def translation_decode_defaults() -> dict:
|
||||
"""Adjust local NLLB search effort without changing the chosen provider."""
|
||||
tier = requested_tier("translation")
|
||||
if tier is None:
|
||||
return {}
|
||||
return {"num_beams": _PERFORMANCE_TARGETS["translation"][tier]["num_beams"]}
|
||||
|
||||
|
||||
def dictation_decode_defaults() -> dict:
|
||||
"""Tune Sherpa transducer search without changing the selected language model."""
|
||||
tier = requested_tier("dictation")
|
||||
if tier is None:
|
||||
return {}
|
||||
target = _PERFORMANCE_TARGETS["dictation"][tier]
|
||||
return {
|
||||
"decoding_method": target["decoding_method"],
|
||||
"max_active_paths": target["max_active_paths"],
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
"""Safe extraction for remote multi-segment WAV results."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import zipfile
|
||||
|
||||
|
||||
_MEMBER = re.compile(r"segments/(\d+)\.wav")
|
||||
|
||||
|
||||
def extract_segment_wavs(artifact_path: str, target_dir: str) -> dict[int, str]:
|
||||
"""Extract an exact ``segments/<index>.wav`` bundle atomically.
|
||||
|
||||
The worker controls the ZIP member names, so accept only the protocol's
|
||||
flat numeric namespace. Streaming each member into a locally minted name
|
||||
also avoids ZipFile.extract() path traversal and symlink behaviour.
|
||||
"""
|
||||
if not artifact_path or not os.path.isfile(artifact_path):
|
||||
raise ValueError("the segment bundle is missing")
|
||||
|
||||
os.makedirs(target_dir, exist_ok=True)
|
||||
paths: dict[int, str] = {}
|
||||
partials: list[str] = []
|
||||
try:
|
||||
with zipfile.ZipFile(artifact_path) as archive:
|
||||
for member in archive.infolist():
|
||||
match = _MEMBER.fullmatch(member.filename)
|
||||
if not match:
|
||||
raise ValueError(
|
||||
f"unexpected segment artifact member: {member.filename}"
|
||||
)
|
||||
index = int(match.group(1))
|
||||
if index in paths:
|
||||
raise ValueError(f"duplicate segment artifact index: {index}")
|
||||
destination = os.path.join(target_dir, f"{index}.wav")
|
||||
partial = f"{destination}.part"
|
||||
partials.append(partial)
|
||||
with archive.open(member) as source, open(partial, "wb") as output:
|
||||
shutil.copyfileobj(source, output)
|
||||
os.replace(partial, destination)
|
||||
partials.remove(partial)
|
||||
paths[index] = destination
|
||||
if not paths:
|
||||
raise ValueError("the segment bundle is empty")
|
||||
return paths
|
||||
except BaseException:
|
||||
for path in (*partials, *paths.values()):
|
||||
try:
|
||||
os.unlink(path)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
try:
|
||||
os.rmdir(target_dir)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
def remove_segment_wavs(paths: dict[int, str]) -> None:
|
||||
"""Remove files minted by :func:`extract_segment_wavs`, then empty dirs."""
|
||||
directories = set()
|
||||
for path in paths.values():
|
||||
directories.add(os.path.dirname(path))
|
||||
try:
|
||||
os.unlink(path)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
for directory in sorted(directories, key=len, reverse=True):
|
||||
try:
|
||||
os.rmdir(directory)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
__all__ = ["extract_segment_wavs", "remove_segment_wavs"]
|
||||
@@ -491,6 +491,35 @@ def _apply_scene_cuts(segments: List[Segment], scene_cuts: Iterable[float]) -> L
|
||||
dur_total = remaining.duration
|
||||
if dur_total <= 0:
|
||||
break
|
||||
words = remaining.extra.get("words")
|
||||
if isinstance(words, list) and words:
|
||||
# A camera cut is not a word boundary. Character-proportional
|
||||
# splitting assigned words seconds away from their real speech.
|
||||
candidates = [
|
||||
i for i in range(1, len(words))
|
||||
if float(words[i - 1]["end"]) <= float(words[i]["start"])
|
||||
and abs(float(words[i]["start"]) - cut) <= 0.25
|
||||
]
|
||||
if not candidates:
|
||||
continue
|
||||
split = min(candidates, key=lambda i: abs(float(words[i]["start"]) - cut))
|
||||
left, right = words[:split], words[split:]
|
||||
left_text = _clean(" ".join(str(w.get("text", w.get("word", ""))) for w in left))
|
||||
right_text = _clean(" ".join(str(w.get("text", w.get("word", ""))) for w in right))
|
||||
left_end, right_start = float(left[-1]["end"]), float(right[0]["start"])
|
||||
if (len(left_text) < MIN_CHARS or len(right_text) < MIN_CHARS
|
||||
or left_end - remaining.start < MIN_DUR
|
||||
or remaining.end - right_start < MIN_DUR):
|
||||
continue
|
||||
out.append(Segment(
|
||||
start=remaining.start, end=left_end, text=left_text,
|
||||
speaker_id=remaining.speaker_id, extra={**remaining.extra, "words": left},
|
||||
))
|
||||
remaining = Segment(
|
||||
start=right_start, end=remaining.end, text=right_text,
|
||||
speaker_id=remaining.speaker_id, extra={**remaining.extra, "words": right},
|
||||
)
|
||||
continue
|
||||
ratio = (cut - remaining.start) / dur_total
|
||||
tentative_split = int(len(remaining.text) * ratio)
|
||||
pos = _best_boundary(remaining.text, tentative_split)
|
||||
@@ -805,3 +834,74 @@ def resplit_segments_by_turns(
|
||||
and t.get("end") is not None
|
||||
]
|
||||
return _resplit_core(segments, words, norm)
|
||||
|
||||
|
||||
def deduplicate_chunk_segments(segments: list[dict]) -> list[dict]:
|
||||
"""Remove repeated ASR context only when matching words share timestamps.
|
||||
|
||||
Preserve different speakers and genuine repeated speech at different times.
|
||||
Input order retains chunk provenance even when a later chunk starts earlier.
|
||||
"""
|
||||
import re
|
||||
|
||||
def token(word):
|
||||
return re.sub(r'[^\w]', '', str(word.get('text', word.get('word', ''))).casefold())
|
||||
|
||||
def timed_words(segment):
|
||||
words = segment.get('words') or []
|
||||
return words if words and all(isinstance(w, dict) and isinstance(w.get('start'), (int, float))
|
||||
and isinstance(w.get('end'), (int, float)) for w in words) else []
|
||||
|
||||
result = []
|
||||
for segment in segments:
|
||||
words = timed_words(segment)
|
||||
matches = []
|
||||
if words and segment.get("speaker_id"):
|
||||
prior_words = [w for previous in result
|
||||
if previous.get('speaker_id') == segment.get('speaker_id')
|
||||
for w in timed_words(previous)
|
||||
if w['end'] >= words[0]['start'] - .35 and w['start'] <= words[-1]['end'] + .35]
|
||||
for index, word in enumerate(words):
|
||||
if token(word) and any(token(word) == token(prior)
|
||||
and abs((word['start'] + word['end']) / 2 - (prior['start'] + prior['end']) / 2) <= .35
|
||||
for prior in prior_words):
|
||||
matches.append(index)
|
||||
# Require a substantial matching prefix; isolated common words cannot
|
||||
# authorize deleting speech. No timing-only truncation is performed.
|
||||
if len(matches) >= 3 and len(matches) / (matches[-1] + 1) >= .6:
|
||||
cutoff = max((w['end'] for w in prior_words), default=0)
|
||||
remaining = [w for w in words[matches[-1] + 1:] if w['start'] >= cutoff - .05]
|
||||
if not remaining:
|
||||
continue
|
||||
text = _clean(' '.join(str(w.get('text', w.get('word', ''))) for w in remaining))
|
||||
segment = {**segment, 'start': remaining[0]['start'], 'end': remaining[-1]['end'],
|
||||
'text': text, 'text_original': text, 'words': remaining}
|
||||
result.append(segment)
|
||||
def bounds(row):
|
||||
words = timed_words(row)
|
||||
if not words:
|
||||
return None
|
||||
if any(a['start'] > b['start'] for a, b in zip(words, words[1:])):
|
||||
# Older chunk stitching can attach an earlier word to a later
|
||||
# line. Never invert an interval or move it backwards over speech.
|
||||
inside = [w for w in words if row['start'] <= w['start'] <= w['end'] <= row['end']]
|
||||
if len(inside) < .6 * len(words):
|
||||
return None
|
||||
words = inside
|
||||
start, end = min(w['start'] for w in words), max(w['end'] for w in words)
|
||||
return (start, end) if end > start else None
|
||||
|
||||
# Repair stale camera-cut bounds only when timed words prove the two
|
||||
# spoken intervals are disjoint. Genuine overlapping speech stays intact.
|
||||
adjust = set()
|
||||
ordered = sorted(enumerate(result), key=lambda item: item[1]['start'])
|
||||
for position, (left_index, left) in enumerate(ordered):
|
||||
for right_index, right in ordered[position + 1:]:
|
||||
if right['start'] >= left['end']:
|
||||
break
|
||||
a, b = bounds(left), bounds(right)
|
||||
if a and b and (a[1] <= b[0] or b[1] <= a[0]):
|
||||
adjust.update((left_index, right_index))
|
||||
result = [{**row, 'start': bounds(row)[0], 'end': bounds(row)[1]}
|
||||
if index in adjust else row for index, row in enumerate(result)]
|
||||
return result
|
||||
|
||||
@@ -411,6 +411,8 @@ def build_offline_recognizer(spec: SherpaModelSpec, *, download: bool = True):
|
||||
return os.path.join(d, spec.files[role])
|
||||
|
||||
if spec.kind == "offline-transducer":
|
||||
from services.performance_profiles import dictation_decode_defaults
|
||||
|
||||
return sherpa_onnx.OfflineRecognizer.from_transducer(
|
||||
encoder=p("encoder"),
|
||||
decoder=p("decoder"),
|
||||
@@ -418,7 +420,7 @@ def build_offline_recognizer(spec: SherpaModelSpec, *, download: bool = True):
|
||||
tokens=p("tokens"),
|
||||
num_threads=_threads_for(spec),
|
||||
provider=_PROVIDER,
|
||||
decoding_method="greedy_search",
|
||||
**dictation_decode_defaults(),
|
||||
model_type=spec.model_type or "nemo_transducer",
|
||||
)
|
||||
if spec.kind == "offline-whisper":
|
||||
@@ -450,6 +452,8 @@ def build_online_recognizer(spec: SherpaModelSpec, *, download: bool = True):
|
||||
return os.path.join(d, spec.files[role])
|
||||
|
||||
if spec.kind == "online-transducer":
|
||||
from services.performance_profiles import dictation_decode_defaults
|
||||
|
||||
return sherpa_onnx.OnlineRecognizer.from_transducer(
|
||||
tokens=p("tokens"),
|
||||
encoder=p("encoder"),
|
||||
@@ -457,7 +461,7 @@ def build_online_recognizer(spec: SherpaModelSpec, *, download: bool = True):
|
||||
joiner=p("joiner"),
|
||||
num_threads=_threads_for(spec),
|
||||
provider=_PROVIDER,
|
||||
decoding_method="greedy_search",
|
||||
**dictation_decode_defaults(),
|
||||
enable_endpoint_detection=True,
|
||||
rule1_min_trailing_silence=rule1,
|
||||
rule2_min_trailing_silence=rule2,
|
||||
|
||||
@@ -71,6 +71,11 @@ TOL_HIGH = 1.08
|
||||
# Max LLM attempts per segment. Past this we just return the best we got.
|
||||
MAX_ATTEMPTS = 3
|
||||
|
||||
# Acceptance window for real TTS measurements. One rewrite is made between
|
||||
# renders; repeating guesses inside a call would discard the useful evidence.
|
||||
MEASURED_TOL_LOW = 0.9
|
||||
MEASURED_TOL_HIGH = 1.04
|
||||
|
||||
|
||||
def expected_duration(text: str, lang: str = "en") -> float:
|
||||
"""Rough CPS-based duration estimate. Returns seconds."""
|
||||
@@ -106,6 +111,121 @@ Reply with ONLY the new line. No quotes, no commentary."""
|
||||
_MIN_EXPANDABLE_RATIO = 0.15
|
||||
|
||||
|
||||
_MEASURED_PROMPT = """\
|
||||
You are a dialogue adaptation agent for precise dubbing. Rewrite the translated
|
||||
line so the SAME voice can speak it inside the exact target duration. The user
|
||||
provides the duration measured from a real render, so use the requested length
|
||||
change as a concrete constraint. Preserve meaning, tone, names, numbers,
|
||||
technical terms, and the target language. Shorten natural phrasing when long;
|
||||
gently expand only when short without inventing facts or dialogue.
|
||||
Reply with ONLY the revised line. No quotes or commentary."""
|
||||
|
||||
|
||||
def adjust_for_measured_slot(
|
||||
text: str,
|
||||
*,
|
||||
slot_seconds: float,
|
||||
measured_seconds: float,
|
||||
target_lang: str,
|
||||
source_text: Optional[str] = None,
|
||||
context_before: Optional[str] = None,
|
||||
context_after: Optional[str] = None,
|
||||
translation_instructions: Optional[str] = None,
|
||||
) -> dict:
|
||||
"""Make one evidence-based rewrite between real TTS measurements."""
|
||||
text = (text or "").strip()
|
||||
slot = max(0.0, float(slot_seconds or 0.0))
|
||||
measured = max(0.0, float(measured_seconds or 0.0))
|
||||
ratio = measured / slot if slot else 1.0
|
||||
base = {
|
||||
"text": text,
|
||||
"measured_seconds": round(measured, 3),
|
||||
"target_seconds": round(slot, 3),
|
||||
"measured_ratio": round(ratio, 3),
|
||||
"changed": False,
|
||||
}
|
||||
if not text or slot <= 0 or measured <= 0:
|
||||
return {**base, "error": "invalid-timing"}
|
||||
if MEASURED_TOL_LOW <= ratio <= MEASURED_TOL_HIGH:
|
||||
return {**base, "error": "already-fits"}
|
||||
# Leave honest silence for extremely short dialogue instead of inventing
|
||||
# speech merely to fill a long shot.
|
||||
if ratio < 0.45:
|
||||
return {**base, "error": "fit-skip-short"}
|
||||
|
||||
from services import llm_skills
|
||||
llm = llm_skills.skill_backend(_SKILL_ID, active=lambda: get_active_llm_backend())
|
||||
if isinstance(llm, OffBackend):
|
||||
return {**base, "error": "no-llm"}
|
||||
|
||||
desired = max(0.2, min(2.0, slot / measured))
|
||||
user_lines = [
|
||||
f"Target language: {target_lang}",
|
||||
f"Exact target duration: {slot:.2f}s",
|
||||
f"Measured duration of this line: {measured:.2f}s",
|
||||
f"Measured ratio: {ratio:.3f} (1.000 is exact)",
|
||||
f"Requested text-length factor: about {desired:.3f}x",
|
||||
f"Current translated line: {text}",
|
||||
]
|
||||
if source_text:
|
||||
user_lines.append(f"Source line (meaning authority): {source_text}")
|
||||
if context_before:
|
||||
user_lines.append(f"Previous source line (context only): {context_before}")
|
||||
if context_after:
|
||||
user_lines.append(f"Next source line (context only): {context_after}")
|
||||
try:
|
||||
reply = llm.chat(
|
||||
system=_MEASURED_PROMPT + ("\nUser translation style brief (preserve meaning and output format): " + translation_instructions if translation_instructions else ""),
|
||||
user="\n".join(user_lines),
|
||||
temperature=0.15,
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("measured slot-fit provider failed")
|
||||
return {**base, "error": "fit-provider-failed"}
|
||||
candidate = (reply or "").strip()
|
||||
if not candidate or candidate == text:
|
||||
return {**base, "error": "fit-unchanged"}
|
||||
ok, reason = refine_output_ok(text, candidate, target_lang)
|
||||
if not ok:
|
||||
logger.warning("measured slot-fit reply rejected (%s)", log_safe(reason))
|
||||
return {**base, "error": "fit-diverged"}
|
||||
return {**base, "text": candidate, "changed": True}
|
||||
|
||||
|
||||
async def adjust_for_measured_slot_many(
|
||||
items: Iterable[tuple], *, executor=None, concurrency: Optional[int] = None,
|
||||
translation_instructions: Optional[str] = None,
|
||||
) -> dict:
|
||||
"""Run one bounded measured rewrite per segment, keyed by segment id."""
|
||||
import asyncio
|
||||
import os
|
||||
|
||||
rows = list(items)
|
||||
if not rows:
|
||||
return {}
|
||||
loop = asyncio.get_running_loop()
|
||||
sem = asyncio.Semaphore(concurrency or int(os.environ.get("OMNIVOICE_LLM_CONCURRENCY", "6")))
|
||||
|
||||
async def _one(key, line, slot, measured, lang, source, before, after):
|
||||
async with sem:
|
||||
result = await loop.run_in_executor(
|
||||
executor,
|
||||
lambda: adjust_for_measured_slot(
|
||||
line,
|
||||
slot_seconds=slot,
|
||||
measured_seconds=measured,
|
||||
target_lang=lang,
|
||||
source_text=source,
|
||||
context_before=before,
|
||||
context_after=after,
|
||||
translation_instructions=translation_instructions,
|
||||
),
|
||||
)
|
||||
return key, result
|
||||
|
||||
return dict(await asyncio.gather(*(_one(*row) for row in rows)))
|
||||
|
||||
|
||||
def adjust_for_slot(
|
||||
text: str,
|
||||
*,
|
||||
|
||||
@@ -60,6 +60,11 @@ class SubprocessASRBackend(SubprocessBackend):
|
||||
def generate(self, text: str, **kw): # pragma: no cover - unused
|
||||
raise NotImplementedError("ASR sidecar does not synthesize speech")
|
||||
|
||||
def ensure_loaded(self) -> None:
|
||||
"""Prove the lazy ASR sidecar is ready for the shared loader."""
|
||||
with self._lock:
|
||||
self._spawn()
|
||||
|
||||
# ── ASR surface ────────────────────────────────────────────────────────
|
||||
@staticmethod
|
||||
def _device() -> str:
|
||||
@@ -109,13 +114,18 @@ class SubprocessASRBackend(SubprocessBackend):
|
||||
raise TimeoutError("timed out waiting for a free GPU worker")
|
||||
with self._lock:
|
||||
self._spawn()
|
||||
from services.performance_profiles import asr_decode_defaults
|
||||
self._send({
|
||||
"op": "transcribe",
|
||||
"audio_path": str(audio_path),
|
||||
"word_timestamps": bool(word_timestamps),
|
||||
"decode_options": asr_decode_defaults(),
|
||||
})
|
||||
reply = self._recv_with_timeout(ASR_RECV_TIMEOUT_S)
|
||||
if not reply:
|
||||
# EOF can arrive before Windows updates poll(); retire the
|
||||
# stale handle so an immediate retry respawns the sidecar.
|
||||
self.shutdown()
|
||||
# Pipe closed mid-transcription → the child crashed.
|
||||
raise RuntimeError(
|
||||
f"{self.id} ASR sidecar crashed mid-transcription "
|
||||
|
||||
@@ -747,6 +747,9 @@ class SubprocessBackend(TTSBackend):
|
||||
with self._lock:
|
||||
self._validate_generate_authorization()
|
||||
self._spawn()
|
||||
# getattr keeps lightweight protocol-loop test doubles valid;
|
||||
# real instances always initialise _proc in __init__.
|
||||
proc = getattr(self, "_proc", None)
|
||||
msg = {"op": "synthesize", "text": text}
|
||||
# Filter kwargs to JSON-safe primitives. Tensor / Path / etc.
|
||||
# don't survive json.dumps and are silently dropped — the
|
||||
@@ -754,8 +757,15 @@ class SubprocessBackend(TTSBackend):
|
||||
for k, v in kw.items():
|
||||
if _is_jsonable(v):
|
||||
msg[k] = v
|
||||
self._send(msg)
|
||||
reply = self._recv_with_timeout(self.recv_timeout_s)
|
||||
try:
|
||||
self._send(msg)
|
||||
reply = self._recv_with_timeout(self.recv_timeout_s)
|
||||
except (RuntimeError, OSError):
|
||||
# A broken or malformed protocol stream cannot be reused.
|
||||
# Reap it before releasing the request lock so an immediate
|
||||
# retry cannot race poll() and write to the same dead pipe.
|
||||
self._reap_unusable_process(proc)
|
||||
raise
|
||||
# A cold sidecar may emit non-terminal {"op": "progress"} frames
|
||||
# (during a model load, etc.) before the terminal audio frame.
|
||||
# Each recv re-arms the watchdog, so a long-but-active load
|
||||
@@ -779,12 +789,27 @@ class SubprocessBackend(TTSBackend):
|
||||
report_model_load_activity()
|
||||
except Exception:
|
||||
pass # the heartbeat is best-effort; never fail a synth over it
|
||||
reply = self._recv_with_timeout(self.recv_timeout_s)
|
||||
if not reply:
|
||||
raise RuntimeError(f"{self.id} sidecar closed pipe mid-generate")
|
||||
try:
|
||||
reply = self._recv_with_timeout(self.recv_timeout_s)
|
||||
except (RuntimeError, OSError):
|
||||
self._reap_unusable_process(proc)
|
||||
raise
|
||||
if not reply:
|
||||
self._reap_unusable_process(proc)
|
||||
raise RuntimeError(f"{self.id} sidecar closed pipe mid-generate")
|
||||
if reply.get("op") == "error":
|
||||
stage = str(reply.get("stage") or "unknown")
|
||||
message = str(reply.get("message") or "unknown sidecar error")
|
||||
traceback_text = str(reply.get("traceback") or "").strip()
|
||||
logger.error(
|
||||
"[%s] sidecar %s error: %s%s",
|
||||
self.id,
|
||||
stage,
|
||||
message,
|
||||
f"\n{traceback_text[:20_000]}" if traceback_text else "",
|
||||
)
|
||||
raise RuntimeError(
|
||||
f"{self.id} sidecar error: {reply.get('message')!r}"
|
||||
f"{self.id} sidecar {stage} error: {message}"
|
||||
)
|
||||
if reply.get("op") != "audio":
|
||||
raise RuntimeError(
|
||||
@@ -909,6 +934,29 @@ class SubprocessBackend(TTSBackend):
|
||||
item for item in self._timeout_quarantine if item is not proc
|
||||
]
|
||||
|
||||
def _reap_unusable_process(self, proc: Optional[subprocess.Popen]) -> None:
|
||||
"""Synchronously retire a sidecar whose protocol pipe is unusable."""
|
||||
if proc is None:
|
||||
return
|
||||
try:
|
||||
proc.wait(timeout=0.25)
|
||||
except Exception:
|
||||
try:
|
||||
proc.kill()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
proc.wait(timeout=2)
|
||||
except Exception:
|
||||
with self._timeout_quarantine_lock:
|
||||
if not any(item is proc for item in self._timeout_quarantine):
|
||||
self._timeout_quarantine.append(proc)
|
||||
return
|
||||
with self._timeout_quarantine_lock:
|
||||
self._timeout_quarantine = [
|
||||
item for item in self._timeout_quarantine if item is not proc
|
||||
]
|
||||
|
||||
def _retry_timeout_cleanup(self) -> bool:
|
||||
"""Retry bounded cleanup, retaining every owner that could still be live."""
|
||||
with self._timeout_quarantine_lock:
|
||||
|
||||
@@ -23,9 +23,14 @@ import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
|
||||
logger = logging.getLogger("omnivoice.translation_engines")
|
||||
|
||||
_NLLB_REPO_ID = "facebook/nllb-200-distilled-600M"
|
||||
_ARGOS_INSTALL_LOCK = threading.Lock()
|
||||
_ARGOS_LANG_ALIASES = {"cmn": "zh"}
|
||||
|
||||
|
||||
# Engine ID → registry entry. Keyed by the `provider` string sent from the
|
||||
# frontend (must match the values of `translateProvider` in the store).
|
||||
@@ -38,7 +43,7 @@ REGISTRY: dict[str, dict] = {
|
||||
"category": "offline",
|
||||
"needs_key": False,
|
||||
"builtin": True,
|
||||
"notes": "Pure-CPU offline translator. Downloads a ~50MB language pack on first use per pair.",
|
||||
"notes": "Pure-CPU offline translator. Install the required language pack explicitly for each pair.",
|
||||
},
|
||||
"nllb": {
|
||||
"id": "nllb",
|
||||
@@ -115,13 +120,25 @@ def is_frozen() -> bool:
|
||||
return bool(getattr(sys, "frozen", False) or os.environ.get("OMNIVOICE_FROZEN"))
|
||||
|
||||
|
||||
def _probe(entry: dict) -> tuple[bool, str]:
|
||||
def _probe(entry: dict) -> tuple[bool, str | None]:
|
||||
mod = entry.get("probe_module")
|
||||
if not mod:
|
||||
return True, "no module required"
|
||||
return True, None
|
||||
try:
|
||||
importlib.import_module(mod)
|
||||
return True, "ready"
|
||||
if entry.get("id") == "nllb":
|
||||
# Transformers being importable only proves the runtime exists.
|
||||
# The weights are a separate explicit model install; do not report
|
||||
# NLLB ready and let from_pretrained download 2.4 GB silently.
|
||||
from api.routers.setup.models import cache_is_complete, is_cached
|
||||
|
||||
model = {"repo_id": _NLLB_REPO_ID}
|
||||
if not is_cached(_NLLB_REPO_ID) or not cache_is_complete(model):
|
||||
return False, "NLLB model weights are not installed"
|
||||
# availability_reason is failure-only metadata. Returning a success
|
||||
# label here made every healthy provider look unavailable after the
|
||||
# public diagnostic scrubber intentionally replaced non-null details.
|
||||
return True, None
|
||||
except ImportError as e:
|
||||
return False, f"import {mod!r} failed: {e}"
|
||||
|
||||
@@ -164,23 +181,38 @@ def _llm_configured() -> tuple[bool, "str | None"]:
|
||||
return False, None
|
||||
|
||||
|
||||
def _configured(entry: dict) -> tuple[bool, str | None]:
|
||||
"""Whether an installed engine has the configuration needed to run."""
|
||||
engine_id = entry.get("id")
|
||||
if engine_id == "openai":
|
||||
return _llm_configured()
|
||||
if engine_id == "deepl":
|
||||
return bool(os.environ.get("DEEPL_API_KEY") or os.environ.get("TRANSLATE_API_KEY")), None
|
||||
if engine_id == "microsoft":
|
||||
return bool(os.environ.get("MICROSOFT_API_KEY") or os.environ.get("TRANSLATE_API_KEY")), None
|
||||
return True, None
|
||||
|
||||
|
||||
def list_engines() -> list[dict]:
|
||||
"""Return a UI-ready list with per-engine availability stamped in."""
|
||||
out = []
|
||||
for e in REGISTRY.values():
|
||||
installed, reason = _probe(e)
|
||||
configured, via = _configured(e)
|
||||
ready = installed and configured
|
||||
entry = {
|
||||
**e,
|
||||
"installed": installed,
|
||||
"availability_reason": reason,
|
||||
"configured": configured,
|
||||
"configured_via": via,
|
||||
"ready": ready,
|
||||
"availability_reason": reason or (
|
||||
None if configured else "Translation provider is not configured"
|
||||
),
|
||||
"install_command": install_command(e),
|
||||
}
|
||||
# LLM engines additionally need a provider/key — surface configured-ness
|
||||
# so the UI can distinguish "importable" from "actually ready to call".
|
||||
if e.get("category") == "llm":
|
||||
configured, via = _llm_configured()
|
||||
entry["configured"] = configured
|
||||
entry["configured_via"] = via
|
||||
out.append(entry)
|
||||
return out
|
||||
|
||||
@@ -256,6 +288,95 @@ def is_installed(engine_id: str) -> bool:
|
||||
return ok
|
||||
|
||||
|
||||
def is_ready(engine_id: str) -> bool:
|
||||
"""True only when both runtime/model and required configuration exist."""
|
||||
entry = REGISTRY.get(engine_id)
|
||||
if not entry:
|
||||
return False
|
||||
installed, _ = _probe(entry)
|
||||
configured, _ = _configured(entry)
|
||||
return installed and configured
|
||||
|
||||
|
||||
def argos_lang_code(value: str) -> str:
|
||||
"""Return the base language token used by Argos package metadata."""
|
||||
code = str(value or "").strip().lower().split("-", 1)[0]
|
||||
code = _ARGOS_LANG_ALIASES.get(code, code)
|
||||
if not re.fullmatch(r"[a-z]{2,3}", code):
|
||||
raise ValueError("Choose a valid source and target language")
|
||||
return code
|
||||
|
||||
|
||||
def _configure_argos_cache() -> None:
|
||||
cache_dir = os.environ.get("OMNIVOICE_CACHE_DIR")
|
||||
if not cache_dir:
|
||||
return
|
||||
argos_cache = os.path.join(cache_dir, "argos-translate")
|
||||
os.makedirs(argos_cache, exist_ok=True)
|
||||
os.environ.setdefault("ARGOS_PACKAGES_DIR", argos_cache)
|
||||
os.environ.setdefault("ARGOS_DATA_DIR", argos_cache)
|
||||
|
||||
|
||||
def argos_pack_status(source_lang: str, target_langs: list[str]) -> dict:
|
||||
"""Report installed Argos pairs without refreshing the remote index."""
|
||||
_configure_argos_cache()
|
||||
import argostranslate.package
|
||||
|
||||
source = argos_lang_code(source_lang)
|
||||
targets = list(dict.fromkeys(argos_lang_code(code) for code in target_langs))
|
||||
installed = {
|
||||
(package.from_code, package.to_code)
|
||||
for package in argostranslate.package.get_installed_packages()
|
||||
}
|
||||
return {
|
||||
"source_lang": source,
|
||||
"pairs": [
|
||||
{
|
||||
"source_lang": source,
|
||||
"target_lang": target,
|
||||
"installed": source == target or (source, target) in installed,
|
||||
}
|
||||
for target in targets
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def install_argos_packs(source_lang: str, target_langs: list[str]) -> dict:
|
||||
"""Explicitly download and install the requested Argos language pairs."""
|
||||
_configure_argos_cache()
|
||||
import argostranslate.package
|
||||
|
||||
source = argos_lang_code(source_lang)
|
||||
targets = list(dict.fromkeys(argos_lang_code(code) for code in target_langs))
|
||||
with _ARGOS_INSTALL_LOCK:
|
||||
status = argos_pack_status(source, targets)
|
||||
missing = {
|
||||
pair["target_lang"]
|
||||
for pair in status["pairs"]
|
||||
if not pair["installed"]
|
||||
}
|
||||
if missing:
|
||||
argostranslate.package.update_package_index()
|
||||
available = argostranslate.package.get_available_packages()
|
||||
for target in targets:
|
||||
if target not in missing:
|
||||
continue
|
||||
package = next(
|
||||
(
|
||||
item
|
||||
for item in available
|
||||
if item.from_code == source and item.to_code == target
|
||||
),
|
||||
None,
|
||||
)
|
||||
if package is None:
|
||||
raise ValueError(
|
||||
f"No Argos language pack is available for {source} → {target}"
|
||||
)
|
||||
argostranslate.package.install_from_path(package.download())
|
||||
return argos_pack_status(source, targets)
|
||||
|
||||
|
||||
def _in_virtualenv() -> bool:
|
||||
"""True if the current interpreter is inside a venv/virtualenv."""
|
||||
return getattr(sys, "base_prefix", sys.prefix) != sys.prefix or hasattr(sys, "real_prefix")
|
||||
|
||||
@@ -542,6 +542,26 @@ def _prompt_disk_load(key: tuple):
|
||||
return None
|
||||
|
||||
|
||||
def _prompt_cache_evict(key: tuple) -> None:
|
||||
"""Discard one prompt from both cache layers. Never raises.
|
||||
|
||||
Transcript-free prompts use a different identity from fully conditioned
|
||||
prompts. Once ASR resolves the transcript, the former must not remain as a
|
||||
viable stale fallback for the same reference clip.
|
||||
"""
|
||||
with _prompt_cache_lock:
|
||||
_prompt_cache.pop(key, None)
|
||||
cache_dir = _prompt_disk_dir()
|
||||
if cache_dir is None:
|
||||
return
|
||||
try:
|
||||
os.remove(_prompt_disk_path(cache_dir, key))
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
except OSError as exc:
|
||||
logger.debug("could not evict stale voice prompt: %s", exc)
|
||||
|
||||
|
||||
def _prompt_disk_save(key: tuple, prompt) -> None:
|
||||
"""Persist ``prompt`` under ``key`` and prune old entries. Never raises."""
|
||||
cache_dir = _prompt_disk_dir()
|
||||
@@ -605,6 +625,29 @@ def _get_clone_prompt(
|
||||
Every short segment falling back to its speaker ref then re-encodes it
|
||||
(~0.4 s each, measured). Scan-resistance, not a second cache policy.
|
||||
"""
|
||||
# Resolve transcript-free references through an already-installed ASR
|
||||
# before deriving the cache key. This protects every native OmniVoice
|
||||
# caller (generate, streaming, batch, dub, audiobook and OpenAI-compatible
|
||||
# speech), including routes that do not have a profile row on which to
|
||||
# persist the transcript. Incomplete reference conditioning can destabilize
|
||||
# the reference/target boundary and introduce words in the generated prefix.
|
||||
unresolved_key = None
|
||||
if ref_audio and not ref_text:
|
||||
try:
|
||||
unresolved_key = _clone_prompt_key(
|
||||
ref_audio, None, preprocess_prompt
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
from services.asr_backend import transcribe_reference
|
||||
|
||||
ref_text = transcribe_reference(ref_audio)
|
||||
except Exception as e: # noqa: BLE001 — model fallback remains available
|
||||
logger.warning("reference transcript resolution failed: %s", e)
|
||||
if ref_text and unresolved_key is not None:
|
||||
_prompt_cache_evict(unresolved_key)
|
||||
|
||||
try:
|
||||
key = _clone_prompt_key(ref_audio, ref_text, preprocess_prompt)
|
||||
except Exception:
|
||||
@@ -755,6 +798,27 @@ class OmniVoiceBackend(TTSBackend):
|
||||
# model_manager so memory isn't doubled.
|
||||
self._model = model
|
||||
|
||||
@property
|
||||
def execution_device(self) -> str | None:
|
||||
"""Actual device of the shared model, for live engine diagnostics."""
|
||||
if self._model is None:
|
||||
return None
|
||||
try:
|
||||
return str(next(self._model.parameters()).device)
|
||||
except Exception: # noqa: BLE001 - third-party model wrappers vary
|
||||
device = getattr(self._model, "device", None)
|
||||
return str(device) if device is not None else None
|
||||
|
||||
@property
|
||||
def dtype(self) -> str | None:
|
||||
"""Actual parameter precision of the shared model when resident."""
|
||||
if self._model is None:
|
||||
return None
|
||||
try:
|
||||
return str(next(self._model.parameters()).dtype)
|
||||
except Exception: # noqa: BLE001 - diagnostics must remain best effort
|
||||
return None
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
try:
|
||||
@@ -2371,7 +2435,7 @@ _INSTALL_HINTS: dict[str, str] = {
|
||||
"moss-tts-v15": "git clone OpenMOSS/MOSS-TTS + set OMNIVOICE_MOSS_TTS_V15_DIR (own venv, transformers==5.0; 8B, ~16 GB weights; CUDA/ROCm/XPU/NPU/CPU, no MPS; Apache-2.0)",
|
||||
"dots-tts": "git clone rednote-hilab/dots.tts + set OMNIVOICE_DOTS_TTS_DIR (own venv, transformers==4.57; 2B, ~9 GB weights; CUDA/CPU, Linux/macOS only — no Windows; Apache-2.0)",
|
||||
"confucius4-tts":"git clone netease-youdao/Confucius4-TTS + set OMNIVOICE_CONFUCIUS4_TTS_DIR (own Python 3.10 venv; 14-lang cross-lingual zero-shot clone; ~5 GB weights auto-download; CUDA/ROCm/XPU/NPU/CPU, no MPS; Apache-2.0)",
|
||||
"audiocpp": "download the matching audio.cpp v0.7.2 prebuilt + set OMNIVOICE_AUDIOCPP_BIN, then explicitly install Breeze-TTS-2 in the engine's Weights list in Model Catalogue (native CPU/Vulkan/CUDA/Metal GGUF server, no Python; en+zh clone+design; ~4.73 GiB; weights research/non-commercial only)",
|
||||
"audiocpp": "download the matching audio.cpp v0.7.4 prebuilt + set OMNIVOICE_AUDIOCPP_BIN, then explicitly install Breeze-TTS-2 in Model Catalogue → Models (native CPU/Vulkan/CUDA/Metal GGUF server, no Python; en+zh clone+design; ~4.73 GiB; weights research/non-commercial only)",
|
||||
}
|
||||
|
||||
|
||||
@@ -2598,6 +2662,18 @@ def list_backends(*, include_hidden: bool = False) -> list[dict]:
|
||||
loaded_instance = _active_instance
|
||||
if loaded_instance is None:
|
||||
loaded_instance = _ENGINE_INSTANCES.get(cls)
|
||||
if loaded_instance is None and bid == "omnivoice":
|
||||
# Startup preloads OmniVoice through model_manager directly, before
|
||||
# any generation route needs an adapter instance. Reflect that
|
||||
# shared resident model here instead of contradicting
|
||||
# /model/loaded with a stale `not_loaded` engine state.
|
||||
try:
|
||||
from services import model_manager
|
||||
|
||||
if model_manager.model is not None:
|
||||
loaded_instance = OmniVoiceBackend(model=model_manager.model)
|
||||
except Exception: # noqa: BLE001 - catalogue reads never fail on diagnostics
|
||||
pass
|
||||
out.append({
|
||||
"id": bid,
|
||||
"display_name": cls.display_name,
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
from services.dub_pipeline import _default_caption_languages
|
||||
|
||||
|
||||
def test_caption_languages_include_manual_and_declared_source_tracks():
|
||||
assert _default_caption_languages(
|
||||
{
|
||||
"language": "en",
|
||||
"subtitles": {"fr": [{}]},
|
||||
"automatic_captions": {"en": [{}], "en-orig": [{}], "es": [{}]},
|
||||
}
|
||||
) == ["en-orig", "fr"]
|
||||
|
||||
|
||||
def test_caption_languages_prefer_matching_manual_source_over_automatic_duplicate():
|
||||
assert _default_caption_languages(
|
||||
{
|
||||
"language": "en-US",
|
||||
"subtitles": {"en": [{}], "fr": [{}]},
|
||||
"automatic_captions": {"en-orig": [{}], "en-US": [{}]},
|
||||
}
|
||||
) == ["en", "fr"]
|
||||
|
||||
|
||||
def test_caption_languages_recover_original_auto_track_without_language_metadata():
|
||||
assert _default_caption_languages(
|
||||
{
|
||||
"automatic_captions": {
|
||||
"de": [{}],
|
||||
"ja-orig": [{}],
|
||||
"es": [{}],
|
||||
}
|
||||
}
|
||||
) == ["ja-orig"]
|
||||
|
||||
|
||||
def test_caption_languages_do_not_guess_from_translated_automatic_tracks():
|
||||
assert _default_caption_languages(
|
||||
{"automatic_captions": {"de": [{}], "es": [{}]}}
|
||||
) == []
|
||||
@@ -73,6 +73,10 @@ while True:
|
||||
sys.exit(0)
|
||||
elif op == "synthesize":
|
||||
t = m.get("text", "")
|
||||
if t == "ERROR":
|
||||
_send({"op": "error", "stage": "synthesize", "message": "bad model",
|
||||
"traceback": "Traceback: useful child frame"})
|
||||
continue
|
||||
if t == "CRASH":
|
||||
os._exit(137)
|
||||
if t == "HANG":
|
||||
@@ -534,6 +538,55 @@ def test_generate_does_not_deadlock_when_called_on_gpu_pool_worker(stub_sidecar,
|
||||
b.shutdown()
|
||||
|
||||
|
||||
def test_sidecar_traceback_is_preserved_in_backend_log(stub_sidecar, monkeypatch, caplog):
|
||||
_use_stub(monkeypatch, stub_sidecar)
|
||||
b = OmniVoiceSubprocessBackend()
|
||||
try:
|
||||
with caplog.at_level("ERROR"), pytest.raises(
|
||||
RuntimeError, match="sidecar synthesize error: bad model"
|
||||
):
|
||||
b.generate("ERROR")
|
||||
assert "Traceback: useful child frame" in caplog.text
|
||||
finally:
|
||||
b.shutdown()
|
||||
|
||||
|
||||
def test_cached_model_load_emits_periodic_heartbeats(monkeypatch):
|
||||
"""A slow cached MPS load has no HF progress events but is still alive."""
|
||||
from engines.omnivoice_subprocess import main as sidecar
|
||||
from services import model_manager
|
||||
from utils import hf_progress
|
||||
|
||||
frames = []
|
||||
|
||||
class FakeTorch:
|
||||
float16 = object()
|
||||
|
||||
class FakeOmniVoice:
|
||||
@classmethod
|
||||
def from_pretrained(cls, *_args, **_kwargs):
|
||||
time.sleep(0.06)
|
||||
return object()
|
||||
|
||||
monkeypatch.setattr(sidecar, "_model", None)
|
||||
monkeypatch.setattr(sidecar, "_LOAD_HEARTBEAT_S", 0.01)
|
||||
monkeypatch.setattr(sidecar, "_send", lambda _stream, frame: frames.append(frame))
|
||||
monkeypatch.setattr(model_manager, "_lazy_torch", lambda: FakeTorch())
|
||||
monkeypatch.setattr(model_manager, "_lazy_omnivoice", lambda: FakeOmniVoice)
|
||||
monkeypatch.setattr(model_manager, "resolve_omnivoice_checkpoint", lambda: "cached")
|
||||
monkeypatch.setattr(model_manager, "get_best_device", lambda: "mps")
|
||||
monkeypatch.setattr(model_manager, "should_preload_tts_asr", lambda: False)
|
||||
monkeypatch.setattr(hf_progress, "register_listener", lambda _listener: 1)
|
||||
monkeypatch.setattr(hf_progress, "unregister_listener", lambda _listener_id: None)
|
||||
|
||||
sidecar._load_model(object())
|
||||
|
||||
loading = [frame for frame in frames if frame.get("stage") == "loading_model"]
|
||||
assert loading[0]["percent"] == 0
|
||||
assert loading[-1]["percent"] == 100
|
||||
assert len(loading) >= 3, "cached model load went silent between 0% and 100%"
|
||||
|
||||
|
||||
def test_sidecar_forwards_native_controls_and_applies_seed(monkeypatch):
|
||||
import torch
|
||||
from engines.omnivoice_subprocess import main as sidecar
|
||||
|
||||
@@ -284,17 +284,40 @@ def test_touch_activity_without_ownership_never_writes(sentinel_env):
|
||||
assert not os.path.exists(run_sentinel.SENTINEL_PATH)
|
||||
|
||||
|
||||
def test_idle_shell_exit_is_retained_without_becoming_a_user_warning(sentinel_env):
|
||||
record = {
|
||||
"last_activity": None,
|
||||
"log_tail": [
|
||||
"INFO VoiceStudio model loaded successfully.",
|
||||
"INFO Preload complete - model ready.",
|
||||
],
|
||||
}
|
||||
assert run_sentinel.warrants_user_notice(record) is False
|
||||
|
||||
|
||||
def test_interrupted_work_or_fatal_startup_still_warrants_a_warning(sentinel_env):
|
||||
assert run_sentinel.warrants_user_notice(
|
||||
{"last_activity": {"kind": "generate"}, "log_tail": []}
|
||||
) is True
|
||||
assert run_sentinel.warrants_user_notice(
|
||||
{"last_activity": None, "log_tail": ["CRITICAL: native runtime failed"]}
|
||||
) is True
|
||||
|
||||
|
||||
# ── Record store semantics (mirrors crash.rs) ──────────────────────────────
|
||||
|
||||
|
||||
def _crash_once(kind="generate"):
|
||||
last_activity = None
|
||||
if kind is not None:
|
||||
last_activity = {"ts": time.time() - 5, "kind": kind, "detail": None}
|
||||
with open(run_sentinel.SENTINEL_PATH, "w", encoding="utf-8") as f:
|
||||
json.dump(
|
||||
{
|
||||
"pid": _dead_pid(),
|
||||
"started_at": time.time() - 60,
|
||||
"version": run_sentinel.APP_VERSION,
|
||||
"last_activity": {"ts": time.time() - 5, "kind": kind, "detail": None},
|
||||
"last_activity": last_activity,
|
||||
},
|
||||
f,
|
||||
)
|
||||
@@ -421,3 +444,15 @@ def test_notification_surfaces_unacked_crash_and_reack(client):
|
||||
fresh = [n for n in notes if n["id"].startswith("last-run-crash-")]
|
||||
assert len(fresh) == 1
|
||||
assert fresh[0]["id"] != crash_notes[0]["id"]
|
||||
|
||||
|
||||
def test_notification_keeps_idle_exit_forensics_without_repeated_warning(client):
|
||||
record = _crash_once(kind=None)
|
||||
assert record is not None
|
||||
|
||||
notes = client.get("/system/notifications").json()["notifications"]
|
||||
assert not [n for n in notes if n["id"].startswith("last-run-crash-")]
|
||||
|
||||
details = client.get("/system/last-run-crash").json()
|
||||
assert details["record"]["detected_at"] == record["detected_at"]
|
||||
assert details["acknowledged"] is False
|
||||
|
||||
@@ -21,6 +21,9 @@ divergent notion of what a worker can do.
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
from typing import Optional
|
||||
|
||||
from worker.capacity import derive_concurrency
|
||||
@@ -33,8 +36,66 @@ _CPU_ONLY = {"cpu"}
|
||||
|
||||
|
||||
def _free_memory_bytes(caps) -> int:
|
||||
vram_gb = float(getattr(caps, "vram_gb", 0) or 0)
|
||||
return int(vram_gb * 1024**3)
|
||||
return _accelerator_memory_bytes(caps)[0]
|
||||
|
||||
|
||||
def _accelerator_memory_bytes(caps) -> tuple[int, int]:
|
||||
"""Return live free/total accelerator memory, falling back to static VRAM."""
|
||||
fallback = int(float(getattr(caps, "vram_gb", 0) or 0) * 1024**3)
|
||||
family = getattr(caps, "family", "") or ""
|
||||
if family not in {"cuda", "rocm"}:
|
||||
return fallback, fallback
|
||||
try:
|
||||
import torch # noqa: PLC0415
|
||||
|
||||
free_bytes, total_bytes = torch.cuda.mem_get_info()
|
||||
return max(0, int(free_bytes)), max(0, int(total_bytes))
|
||||
except Exception:
|
||||
logger.debug("Live accelerator memory probe failed", exc_info=True)
|
||||
return fallback, fallback
|
||||
|
||||
|
||||
def _nvidia_driver_version() -> str:
|
||||
executable = shutil.which("nvidia-smi")
|
||||
if not executable and os.path.isfile("/usr/lib/wsl/lib/nvidia-smi"):
|
||||
executable = "/usr/lib/wsl/lib/nvidia-smi"
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[
|
||||
executable or "nvidia-smi",
|
||||
"--query-gpu=driver_version",
|
||||
"--format=csv,noheader,nounits",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=2,
|
||||
check=False,
|
||||
)
|
||||
except (FileNotFoundError, OSError, subprocess.TimeoutExpired):
|
||||
return ""
|
||||
if result.returncode != 0:
|
||||
return ""
|
||||
return next((line.strip() for line in result.stdout.splitlines() if line.strip()), "")
|
||||
|
||||
|
||||
def _accelerator_details(caps) -> tuple[str, str]:
|
||||
"""Return driver and architecture details without making registration brittle."""
|
||||
family = getattr(caps, "family", "") or ""
|
||||
driver = str(getattr(caps, "driver", "") or "")
|
||||
compute = ""
|
||||
try:
|
||||
import torch # noqa: PLC0415
|
||||
|
||||
if family == "cuda":
|
||||
major, minor = torch.cuda.get_device_capability(0)
|
||||
compute = f"{major}.{minor}"
|
||||
elif family == "rocm":
|
||||
compute = str(getattr(torch.cuda.get_device_properties(0), "gcnArchName", "") or "")
|
||||
except Exception:
|
||||
logger.debug("Accelerator architecture probe failed", exc_info=True)
|
||||
if family == "cuda" and not driver:
|
||||
driver = _nvidia_driver_version()
|
||||
return driver, compute
|
||||
|
||||
|
||||
def discover(*, include_unavailable: bool = False) -> list[dict]:
|
||||
@@ -228,17 +289,18 @@ def model_id_for(entry: dict) -> str:
|
||||
def _operations_for(entry: dict) -> list[str]:
|
||||
"""Which task kinds this engine can serve.
|
||||
|
||||
Cloning is the one genuine split — an engine that cannot clone must never
|
||||
be handed a clone task, and ``supports_cloning`` is ``None`` when the
|
||||
answer depends on the loaded model, which we treat as "no" rather than
|
||||
risk a task that fails at the last moment.
|
||||
Cloning is the one genuine split: an engine that cannot clone must never
|
||||
be handed a clone or Dubbing task. ``supports_cloning`` is ``None`` when
|
||||
the answer depends on the loaded model; that is treated as "no" instead of
|
||||
risking a task that fails at the last moment.
|
||||
"""
|
||||
|
||||
# Audiobook chapters use the same TTS engine, but are advertised as their
|
||||
# own schedulable operation so an older worker cannot accept a task whose
|
||||
# chapter assembler it does not implement.
|
||||
operations = ["audiobook", "dub_segments", "tts"]
|
||||
operations = ["audiobook", "batch_segments", "tts"]
|
||||
if entry.get("supports_cloning") is True:
|
||||
operations.append("clone")
|
||||
operations.extend(("clone", "dub_segments"))
|
||||
return operations
|
||||
|
||||
|
||||
@@ -276,14 +338,17 @@ def describe_gpus() -> list[dict]:
|
||||
if caps is None:
|
||||
return []
|
||||
family = getattr(caps, "family", "") or ""
|
||||
free_bytes, total_bytes = _accelerator_memory_bytes(caps)
|
||||
driver, compute = _accelerator_details(caps)
|
||||
return [
|
||||
{
|
||||
"vendor": _vendor_for(family),
|
||||
"model": getattr(caps, "device_name", "") or "",
|
||||
"backend": family,
|
||||
"memory_bytes": _free_memory_bytes(caps),
|
||||
"free_memory_bytes": _free_memory_bytes(caps),
|
||||
"driver_version": getattr(caps, "driver", "") or "",
|
||||
"memory_bytes": total_bytes,
|
||||
"free_memory_bytes": free_bytes,
|
||||
"driver_version": driver,
|
||||
"compute_capability": compute,
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
@@ -150,7 +150,11 @@ class WorkerCapacity:
|
||||
worker_id: str
|
||||
max_concurrent_tasks: int = 1
|
||||
active_tasks: int = 0
|
||||
free_memory_bytes: int = 0
|
||||
# ``None`` means the worker could not query VRAM. It is distinct from a
|
||||
# real zero-byte reading, which means the device is completely occupied.
|
||||
free_memory_bytes: Optional[int] = None
|
||||
cpu_percent: Optional[float] = None
|
||||
gpu_utilization_percent: Optional[float] = None
|
||||
backend: str = ""
|
||||
resident_models: set[str] = field(default_factory=set)
|
||||
slots: dict[str, ModelSlot] = field(default_factory=dict)
|
||||
@@ -287,6 +291,8 @@ class WorkerCapacity:
|
||||
available_slots: int,
|
||||
resident_models: Optional[set[str]] = None,
|
||||
free_memory_bytes: Optional[int] = None,
|
||||
cpu_percent: Optional[float] = None,
|
||||
gpu_utilization_percent: Optional[float] = None,
|
||||
now: Optional[float] = None,
|
||||
) -> None:
|
||||
"""Adopt a heartbeat snapshot. The worker is the source of truth for
|
||||
@@ -307,6 +313,10 @@ class WorkerCapacity:
|
||||
self.resident_models = set(resident_models)
|
||||
if free_memory_bytes is not None:
|
||||
self.free_memory_bytes = free_memory_bytes
|
||||
if cpu_percent is not None:
|
||||
self.cpu_percent = max(0.0, min(100.0, float(cpu_percent)))
|
||||
if gpu_utilization_percent is not None:
|
||||
self.gpu_utilization_percent = max(0.0, min(100.0, float(gpu_utilization_percent)))
|
||||
# Parks are released on a timer, and by the worker restarting — never
|
||||
# by the worker's own load report.
|
||||
#
|
||||
@@ -330,6 +340,9 @@ class WorkerCapacity:
|
||||
"zombie_tasks": self.zombie_tasks,
|
||||
"available_slots": self.available_slots,
|
||||
"resident_models": sorted(self.resident_models),
|
||||
"free_memory_bytes": self.free_memory_bytes,
|
||||
"cpu_percent": self.cpu_percent,
|
||||
"gpu_utilization_percent": self.gpu_utilization_percent,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -70,6 +70,7 @@ class Operation(str, enum.Enum):
|
||||
CLONE = "clone"
|
||||
ASR = "asr"
|
||||
DUB = "dub"
|
||||
BATCH_SEGMENTS = "batch_segments"
|
||||
AUDIOBOOK = "audiobook"
|
||||
|
||||
@classmethod
|
||||
@@ -90,6 +91,7 @@ _PROFILE: dict[Operation, tuple[float, int]] = {
|
||||
Operation.TTS: (1.0, 75),
|
||||
Operation.CLONE: (2.0, 75),
|
||||
Operation.DUB: (12.0, 90),
|
||||
Operation.BATCH_SEGMENTS: (12.0, 90),
|
||||
Operation.AUDIOBOOK: (24.0, 90),
|
||||
}
|
||||
|
||||
|
||||
@@ -188,7 +188,28 @@ def from_reason(reason: str, *, code: Optional[str] = None) -> WorkerError:
|
||||
|
||||
|
||||
def from_exception(exc: BaseException, *, code: Optional[str] = None) -> WorkerError:
|
||||
return from_reason(failure.describe_exception(exc), code=code)
|
||||
reason = failure.describe_exception(exc)
|
||||
if code is None and _is_invalid_generation_input(reason):
|
||||
# OmniVoice validates its closed voice-direction vocabulary inside
|
||||
# inference. The local route already turns these signatures into a
|
||||
# 400; a worker used to call the same deterministic ValueError
|
||||
# UNKNOWN/TRANSIENT and spend the task's retry budget on identical
|
||||
# renders. Keep the message (it contains the accepted vocabulary),
|
||||
# but stop the fleet after the first attempt.
|
||||
code = "INVALID_TASK_PARAMS"
|
||||
return from_reason(reason, code=code)
|
||||
|
||||
|
||||
def _is_invalid_generation_input(reason: str) -> bool:
|
||||
low = (reason or "").lower()
|
||||
return any(
|
||||
signature in low
|
||||
for signature in (
|
||||
"unsupported instruct items",
|
||||
"conflicting instruct items",
|
||||
"in a single instruct",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _hint_for(taxonomy_key: str) -> str:
|
||||
|
||||
+146
-12
@@ -135,6 +135,7 @@ class TaskExecutor:
|
||||
"tts": self._run_tts,
|
||||
"clone": self._run_tts,
|
||||
"audiobook": self._run_audiobook,
|
||||
"batch_segments": self._run_dub_segments,
|
||||
"dub_segments": self._run_dub_segments,
|
||||
}.get(operation)
|
||||
if handler is None:
|
||||
@@ -176,20 +177,95 @@ class TaskExecutor:
|
||||
)
|
||||
await report.loading(1.0, "model ready")
|
||||
rendered: list[tuple[int, bytes]] = []
|
||||
for index, row in enumerate(rows):
|
||||
row = dict(row)
|
||||
prepared_rows = []
|
||||
for index, source in enumerate(rows):
|
||||
row = dict(source)
|
||||
row["ref_audio"] = refs[index] if index < len(refs) else None
|
||||
audio = await self._bounded_thread(
|
||||
self._synthesize_dub_segment,
|
||||
backend,
|
||||
row,
|
||||
timeout=run_budget, code="EXECUTION_TIMEOUT", what=f"Dubbing segment {index + 1}",
|
||||
prepared_rows.append(row)
|
||||
|
||||
# Remote dubbing used to hold one coarse GPU lease but still call the
|
||||
# engine once per line. That bypassed the same native variable-length
|
||||
# batch path used by local dubbing, leaving large GPUs mostly idle.
|
||||
# Group only rows whose scalar generation contract is compatible;
|
||||
# language, reference, duration, speed and instruct remain per-row.
|
||||
try:
|
||||
from services.dub_batching import batch_timeout_s, native_batch_width
|
||||
from services.tts_backend import TTSBackend
|
||||
|
||||
has_native_batch = (
|
||||
getattr(type(backend), "generate_batch", TTSBackend.generate_batch)
|
||||
is not TTSBackend.generate_batch
|
||||
)
|
||||
payload, _meta = await self._thread_call(
|
||||
self._encode, audio, row, backend
|
||||
batch_width = native_batch_width(backend) if has_native_batch else 1
|
||||
except Exception: # noqa: BLE001 - capability probing takes the safe path
|
||||
batch_timeout_s = None
|
||||
batch_width = 1
|
||||
|
||||
index = 0
|
||||
while index < len(prepared_rows):
|
||||
row = prepared_rows[index]
|
||||
batch = [row]
|
||||
if batch_width > 1 and row.get("seed") is None:
|
||||
compatibility = self._dub_batch_compatibility(row)
|
||||
for candidate in prepared_rows[index + 1 : index + batch_width]:
|
||||
if (
|
||||
candidate.get("seed") is not None
|
||||
or self._dub_batch_compatibility(candidate) != compatibility
|
||||
):
|
||||
break
|
||||
batch.append(candidate)
|
||||
|
||||
audios = None
|
||||
if len(batch) > 1:
|
||||
try:
|
||||
timeout = (
|
||||
batch_timeout_s([str(item.get("text") or "") for item in batch], backend)
|
||||
if batch_timeout_s is not None
|
||||
else run_budget
|
||||
)
|
||||
audios = await self._bounded_thread(
|
||||
self._synthesize_dub_batch,
|
||||
backend,
|
||||
batch,
|
||||
timeout=min(run_budget, timeout),
|
||||
code="EXECUTION_TIMEOUT",
|
||||
what=f"Dubbing segments {index + 1}-{index + len(batch)}",
|
||||
)
|
||||
except TaskFailure:
|
||||
raise
|
||||
except Exception as exc: # native batching is an optimization
|
||||
logger.warning(
|
||||
"Native remote dub batch failed for segments %s-%s; falling back: %s",
|
||||
index + 1,
|
||||
index + len(batch),
|
||||
exc,
|
||||
)
|
||||
|
||||
if audios is None:
|
||||
batch = [row]
|
||||
audios = [
|
||||
await self._bounded_thread(
|
||||
self._synthesize_dub_segment,
|
||||
backend,
|
||||
row,
|
||||
timeout=run_budget,
|
||||
code="EXECUTION_TIMEOUT",
|
||||
what=f"Dubbing segment {index + 1}",
|
||||
)
|
||||
]
|
||||
|
||||
encoded = await asyncio.gather(
|
||||
*(self._thread_call(self._encode, audio, item, backend)
|
||||
for audio, item in zip(audios, batch))
|
||||
)
|
||||
rendered.append((int(row.get("index", index)), payload))
|
||||
await report.progress((index + 1) / len(rows), f"segment {index + 1} of {len(rows)}")
|
||||
for offset, (item, (payload, _meta)) in enumerate(zip(batch, encoded), 1):
|
||||
rendered.append((int(item.get("index", index + offset - 1)), payload))
|
||||
completed = index + offset
|
||||
await report.progress(
|
||||
completed / len(prepared_rows),
|
||||
f"segment {completed} of {len(prepared_rows)}",
|
||||
)
|
||||
index += len(batch)
|
||||
|
||||
bundle = io.BytesIO()
|
||||
with zipfile.ZipFile(bundle, "w", compression=zipfile.ZIP_STORED) as archive:
|
||||
@@ -221,7 +297,7 @@ class TaskExecutor:
|
||||
"num_step": int(row.get("num_step") or 16),
|
||||
"guidance_scale": float(row.get("guidance_scale") or 2.0),
|
||||
"speed": float(row.get("speed") or 1.0), "denoise": True,
|
||||
"postprocess_output": True,
|
||||
"postprocess_output": bool(row.get("postprocess_output", True)),
|
||||
}
|
||||
if (
|
||||
getattr(backend, "supports_native_omnivoice_controls", False)
|
||||
@@ -239,6 +315,64 @@ class TaskExecutor:
|
||||
audio = normalize_audio(audio, target_dBFS=-2.0)
|
||||
return audio
|
||||
|
||||
@staticmethod
|
||||
def _dub_batch_compatibility(row: dict) -> tuple:
|
||||
"""Scalar options that a native backend requires to match in a batch."""
|
||||
return (
|
||||
not bool(row.get("ref_single_use")),
|
||||
bool(row.get("ref_audio")),
|
||||
int(row.get("num_step") or 16),
|
||||
float(row.get("guidance_scale") or 2.0),
|
||||
bool(row.get("postprocess_output", True)),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _synthesize_dub_batch(backend, rows: list[dict]):
|
||||
"""Worker-side equivalent of local dubbing's native batch path."""
|
||||
from services.audio_dsp import (
|
||||
apply_effects_chain,
|
||||
apply_mastering,
|
||||
get_effect_chain,
|
||||
normalize_audio,
|
||||
)
|
||||
from services.text_normalization import normalize_for_tts
|
||||
|
||||
texts = [normalize_for_tts(row.get("text") or "", row.get("language")) for row in rows]
|
||||
outputs = backend.generate_batch(
|
||||
texts,
|
||||
language=[
|
||||
row.get("language") if row.get("language") != "Auto" else None for row in rows
|
||||
],
|
||||
ref_audio=[row.get("ref_audio") for row in rows],
|
||||
ref_text=[row.get("ref_text") for row in rows],
|
||||
cache_ref=not bool(rows[0].get("ref_single_use")),
|
||||
instruct=[row.get("instruct") or None for row in rows],
|
||||
duration=[row.get("duration") for row in rows],
|
||||
num_step=int(rows[0].get("num_step") or 16),
|
||||
guidance_scale=float(rows[0].get("guidance_scale") or 2.0),
|
||||
speed=[float(row.get("speed") or 1.0) for row in rows],
|
||||
denoise=True,
|
||||
postprocess_output=bool(rows[0].get("postprocess_output", True)),
|
||||
)
|
||||
if len(outputs) != len(rows):
|
||||
raise RuntimeError(
|
||||
f"native batch returned {len(outputs)} outputs for {len(rows)} segments"
|
||||
)
|
||||
rendered = []
|
||||
for output, row in zip(outputs, rows):
|
||||
preset = row.get("effect_preset") or "broadcast"
|
||||
if preset != "raw":
|
||||
if not getattr(backend, "applies_own_mastering", False):
|
||||
output = apply_mastering(output, sample_rate=backend.sample_rate)
|
||||
chain = get_effect_chain(preset)
|
||||
if chain:
|
||||
output = apply_effects_chain(
|
||||
output, sample_rate=backend.sample_rate, chain=chain
|
||||
)
|
||||
output = normalize_audio(output, target_dBFS=-2.0)
|
||||
rendered.append(output)
|
||||
return rendered
|
||||
|
||||
# ── Operations ────────────────────────────────────────────────────────
|
||||
|
||||
async def _run_tts(self, assignment, params: dict, report: "_Reporters") -> dict:
|
||||
|
||||
@@ -91,7 +91,17 @@ _TASK_TRANSITIONS: dict[TaskState, frozenset[TaskState]] = {
|
||||
}
|
||||
),
|
||||
TaskState.MODEL_LOADING: frozenset(
|
||||
{TaskState.RUNNING, TaskState.QUEUED, TaskState.CANCELLED, TaskState.TIMEOUT, TaskState.FAILED}
|
||||
{
|
||||
TaskState.RUNNING,
|
||||
# A worker reports ``started`` before loading. If the executor's
|
||||
# final phase report is model loading, a completed render can move
|
||||
# directly into bulk result delivery without another started frame.
|
||||
TaskState.RESULT_UPLOADING,
|
||||
TaskState.QUEUED,
|
||||
TaskState.CANCELLED,
|
||||
TaskState.TIMEOUT,
|
||||
TaskState.FAILED,
|
||||
}
|
||||
),
|
||||
TaskState.RUNNING: frozenset(
|
||||
{
|
||||
@@ -406,12 +416,15 @@ class Task:
|
||||
raise LifecycleError("stale session epoch")
|
||||
if attempt.state.terminal:
|
||||
raise LifecycleError(f"attempt {attempt_id} already terminal ({attempt.state.value})")
|
||||
if new is not attempt.state:
|
||||
attempt.phase_started_at = resolve(now)
|
||||
attempt.state = new
|
||||
implied = _ATTEMPT_TO_TASK.get(new)
|
||||
if implied is not None:
|
||||
self._set_state(implied, now=now)
|
||||
# Mutate the attempt only after the task transition succeeds. Otherwise
|
||||
# one rejected transition leaves the pair contradictory (for example
|
||||
# task=model_loading with attempt=uploading) and every retry is refused.
|
||||
if new is not attempt.state:
|
||||
attempt.phase_started_at = resolve(now)
|
||||
attempt.state = new
|
||||
return attempt
|
||||
|
||||
def accept(self, attempt_id: str, **kw) -> Attempt:
|
||||
|
||||
@@ -305,6 +305,8 @@ class WorkerPool:
|
||||
available_slots: int,
|
||||
resident_models: Optional[set[str]] = None,
|
||||
free_memory_bytes: Optional[int] = None,
|
||||
cpu_percent: Optional[float] = None,
|
||||
gpu_utilization_percent: Optional[float] = None,
|
||||
latency_ms: Optional[float] = None,
|
||||
now: Optional[float] = None,
|
||||
) -> Optional[ConnectedWorker]:
|
||||
@@ -319,6 +321,8 @@ class WorkerPool:
|
||||
available_slots=available_slots,
|
||||
resident_models=resident_models,
|
||||
free_memory_bytes=free_memory_bytes,
|
||||
cpu_percent=cpu_percent,
|
||||
gpu_utilization_percent=gpu_utilization_percent,
|
||||
)
|
||||
return worker
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -194,20 +194,22 @@ class RegisterResponse(_message.Message):
|
||||
def __init__(self, envelope: _Optional[_Union[Envelope, _Mapping]] = ..., worker_id: _Optional[str] = ..., session_token: _Optional[str] = ..., session_epoch: _Optional[int] = ..., protocol_version: _Optional[int] = ..., session_expires_at_unix: _Optional[int] = ..., heartbeat_interval_seconds: _Optional[int] = ..., authoritative_in_flight: _Optional[_Iterable[_Union[TaskRef, _Mapping]]] = ..., error: _Optional[_Union[Error, _Mapping]] = ...) -> None: ...
|
||||
|
||||
class Heartbeat(_message.Message):
|
||||
__slots__ = ("envelope", "active_tasks", "available_slots", "resident_models", "free_memory_bytes", "cpu_percent")
|
||||
__slots__ = ("envelope", "active_tasks", "available_slots", "resident_models", "free_memory_bytes", "cpu_percent", "gpu_utilization_percent")
|
||||
ENVELOPE_FIELD_NUMBER: _ClassVar[int]
|
||||
ACTIVE_TASKS_FIELD_NUMBER: _ClassVar[int]
|
||||
AVAILABLE_SLOTS_FIELD_NUMBER: _ClassVar[int]
|
||||
RESIDENT_MODELS_FIELD_NUMBER: _ClassVar[int]
|
||||
FREE_MEMORY_BYTES_FIELD_NUMBER: _ClassVar[int]
|
||||
CPU_PERCENT_FIELD_NUMBER: _ClassVar[int]
|
||||
GPU_UTILIZATION_PERCENT_FIELD_NUMBER: _ClassVar[int]
|
||||
envelope: Envelope
|
||||
active_tasks: int
|
||||
available_slots: int
|
||||
resident_models: _containers.RepeatedScalarFieldContainer[str]
|
||||
free_memory_bytes: int
|
||||
cpu_percent: float
|
||||
def __init__(self, envelope: _Optional[_Union[Envelope, _Mapping]] = ..., active_tasks: _Optional[int] = ..., available_slots: _Optional[int] = ..., resident_models: _Optional[_Iterable[str]] = ..., free_memory_bytes: _Optional[int] = ..., cpu_percent: _Optional[float] = ...) -> None: ...
|
||||
gpu_utilization_percent: float
|
||||
def __init__(self, envelope: _Optional[_Union[Envelope, _Mapping]] = ..., active_tasks: _Optional[int] = ..., available_slots: _Optional[int] = ..., resident_models: _Optional[_Iterable[str]] = ..., free_memory_bytes: _Optional[int] = ..., cpu_percent: _Optional[float] = ..., gpu_utilization_percent: _Optional[float] = ...) -> None: ...
|
||||
|
||||
class TaskAccepted(_message.Message):
|
||||
__slots__ = ("ref", "envelope")
|
||||
@@ -480,6 +482,14 @@ class PrewarmRequest(_message.Message):
|
||||
download_if_missing: bool
|
||||
def __init__(self, envelope: _Optional[_Union[Envelope, _Mapping]] = ..., engine: _Optional[str] = ..., model_id: _Optional[str] = ..., download_if_missing: _Optional[bool] = ...) -> None: ...
|
||||
|
||||
class ModelInstallCancelRequest(_message.Message):
|
||||
__slots__ = ("envelope", "model_id")
|
||||
ENVELOPE_FIELD_NUMBER: _ClassVar[int]
|
||||
MODEL_ID_FIELD_NUMBER: _ClassVar[int]
|
||||
envelope: Envelope
|
||||
model_id: str
|
||||
def __init__(self, envelope: _Optional[_Union[Envelope, _Mapping]] = ..., model_id: _Optional[str] = ...) -> None: ...
|
||||
|
||||
class Ping(_message.Message):
|
||||
__slots__ = ("envelope", "nonce")
|
||||
ENVELOPE_FIELD_NUMBER: _ClassVar[int]
|
||||
@@ -507,7 +517,7 @@ class Shutdown(_message.Message):
|
||||
def __init__(self, envelope: _Optional[_Union[Envelope, _Mapping]] = ..., reason: _Optional[str] = ...) -> None: ...
|
||||
|
||||
class ServerMessage(_message.Message):
|
||||
__slots__ = ("assignment", "cancel", "result_ack", "config", "ping", "drain", "shutdown", "prewarm", "registered")
|
||||
__slots__ = ("assignment", "cancel", "result_ack", "config", "ping", "drain", "shutdown", "prewarm", "registered", "model_install_cancel")
|
||||
ASSIGNMENT_FIELD_NUMBER: _ClassVar[int]
|
||||
CANCEL_FIELD_NUMBER: _ClassVar[int]
|
||||
RESULT_ACK_FIELD_NUMBER: _ClassVar[int]
|
||||
@@ -517,6 +527,7 @@ class ServerMessage(_message.Message):
|
||||
SHUTDOWN_FIELD_NUMBER: _ClassVar[int]
|
||||
PREWARM_FIELD_NUMBER: _ClassVar[int]
|
||||
REGISTERED_FIELD_NUMBER: _ClassVar[int]
|
||||
MODEL_INSTALL_CANCEL_FIELD_NUMBER: _ClassVar[int]
|
||||
assignment: TaskAssignment
|
||||
cancel: TaskCancel
|
||||
result_ack: ResultAckMessage
|
||||
@@ -526,7 +537,8 @@ class ServerMessage(_message.Message):
|
||||
shutdown: Shutdown
|
||||
prewarm: PrewarmRequest
|
||||
registered: RegisterResponse
|
||||
def __init__(self, assignment: _Optional[_Union[TaskAssignment, _Mapping]] = ..., cancel: _Optional[_Union[TaskCancel, _Mapping]] = ..., result_ack: _Optional[_Union[ResultAckMessage, _Mapping]] = ..., config: _Optional[_Union[ConfigUpdate, _Mapping]] = ..., ping: _Optional[_Union[Ping, _Mapping]] = ..., drain: _Optional[_Union[Drain, _Mapping]] = ..., shutdown: _Optional[_Union[Shutdown, _Mapping]] = ..., prewarm: _Optional[_Union[PrewarmRequest, _Mapping]] = ..., registered: _Optional[_Union[RegisterResponse, _Mapping]] = ...) -> None: ...
|
||||
model_install_cancel: ModelInstallCancelRequest
|
||||
def __init__(self, assignment: _Optional[_Union[TaskAssignment, _Mapping]] = ..., cancel: _Optional[_Union[TaskCancel, _Mapping]] = ..., result_ack: _Optional[_Union[ResultAckMessage, _Mapping]] = ..., config: _Optional[_Union[ConfigUpdate, _Mapping]] = ..., ping: _Optional[_Union[Ping, _Mapping]] = ..., drain: _Optional[_Union[Drain, _Mapping]] = ..., shutdown: _Optional[_Union[Shutdown, _Mapping]] = ..., prewarm: _Optional[_Union[PrewarmRequest, _Mapping]] = ..., registered: _Optional[_Union[RegisterResponse, _Mapping]] = ..., model_install_cancel: _Optional[_Union[ModelInstallCancelRequest, _Mapping]] = ...) -> None: ...
|
||||
|
||||
class ArtifactRef(_message.Message):
|
||||
__slots__ = ("artifact_id", "task_id", "attempt_id", "filename", "content_type", "size_bytes", "sha256", "session_token")
|
||||
|
||||
@@ -228,11 +228,9 @@ message Heartbeat {
|
||||
uint32 active_tasks = 2;
|
||||
uint32 available_slots = 3;
|
||||
repeated string resident_models = 4;
|
||||
uint64 free_memory_bytes = 5;
|
||||
double cpu_percent = 6;
|
||||
// GPU utilisation is deliberately absent: unobtainable on Apple without
|
||||
// sudo powermetrics and absent on CUDA without a new NVML dependency.
|
||||
// Slots + queue depth are the load signals (goal_v2.md A11).
|
||||
optional uint64 free_memory_bytes = 5;
|
||||
optional double cpu_percent = 6;
|
||||
optional double gpu_utilization_percent = 7;
|
||||
}
|
||||
|
||||
message TaskAccepted { TaskRef ref = 1; Envelope envelope = 2; }
|
||||
@@ -423,6 +421,14 @@ message PrewarmRequest {
|
||||
bool download_if_missing = 4;
|
||||
}
|
||||
|
||||
// Cancel the explicit catalogue install attached to a pre-warm. The opaque
|
||||
// model id is resolved against the worker's advertised capabilities; repository
|
||||
// paths and arbitrary install targets never cross this boundary.
|
||||
message ModelInstallCancelRequest {
|
||||
Envelope envelope = 1;
|
||||
string model_id = 2;
|
||||
}
|
||||
|
||||
message Ping { Envelope envelope = 1; uint64 nonce = 2; }
|
||||
|
||||
// Fleet operations: stop taking work, finish what you have, then reconnect.
|
||||
@@ -452,6 +458,7 @@ message ServerMessage {
|
||||
// never in a frame: a credential in the stream would be copied into every
|
||||
// protocol trace and every debug log that dumps one.
|
||||
RegisterResponse registered = 9;
|
||||
ModelInstallCancelRequest model_install_cancel = 10;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -42,8 +42,8 @@ producer for it — so ``decide(op=...)`` answers for the surface the user is
|
||||
looking at rather than for the machine, and the badge cannot read
|
||||
"gpu2 ● ready" on a tab whose work is 100% local.
|
||||
|
||||
Speech synthesis, chapter-at-a-time audiobook rendering, and coarse
|
||||
``dub_segments`` synthesis have remote producers. Dub assembly, ASR,
|
||||
Speech synthesis, chapter-at-a-time audiobook and Stories rendering, and
|
||||
coarse ``dub_segments`` synthesis have remote producers. Dub assembly, ASR,
|
||||
diarization, translation and RVC remain local. Dictation is
|
||||
intentionally local regardless of the selected target because its latency is
|
||||
the feature.
|
||||
@@ -63,24 +63,38 @@ LOCAL = "local"
|
||||
# Operations with a remote producer today. Ports land one at a time, and this
|
||||
# set is what keeps the picker honest about which ones have arrived.
|
||||
#
|
||||
# `dub` is the surface the user picks; `dub_segments` is the coarse worker op
|
||||
# it dispatches (dub_generate.py). Both belong here because the GPU work does
|
||||
# `dub` / `batch` are the surfaces the user picks; `dub_segments` and
|
||||
# `batch_segments` are their coarse worker ops. Both pairs belong here because the GPU work does
|
||||
# leave this machine — listing only the worker op would make the Dub tab read
|
||||
# "Local" while a remote card renders it.
|
||||
#
|
||||
# Dictation is deliberately absent and should stay that way: it runs ASR per
|
||||
# utterance inside a live WebSocket loop, where a round trip per utterance
|
||||
# would spend the one thing that route exists for.
|
||||
REMOTE_OPERATIONS = frozenset({"audiobook", "dub", "dub_segments", "tts"})
|
||||
REMOTE_OPERATIONS = frozenset(
|
||||
{
|
||||
"audiobook",
|
||||
"batch",
|
||||
"batch_segments",
|
||||
"clone",
|
||||
"dub",
|
||||
"dub_segments",
|
||||
"longform",
|
||||
"tts",
|
||||
}
|
||||
)
|
||||
|
||||
# Only for the sentence the user reads; an unknown op falls back to its id
|
||||
# rather than inventing a name for it.
|
||||
_OP_LABELS = {
|
||||
"tts": "speech synthesis",
|
||||
"clone": "voice cloning",
|
||||
"batch": "batch dubbing",
|
||||
"batch_segments": "batch dubbing",
|
||||
"dub": "dubbing",
|
||||
"dub_segments": "dubbing",
|
||||
"audiobook": "audiobook rendering",
|
||||
"longform": "story rendering",
|
||||
"dictation": "dictation",
|
||||
"asr": "transcription",
|
||||
}
|
||||
@@ -103,6 +117,13 @@ class Target:
|
||||
latency_ms: float = 0.0
|
||||
active_tasks: int = 0
|
||||
max_tasks: int = 0
|
||||
cpu_percent: Optional[float] = None
|
||||
free_memory_bytes: Optional[int] = None
|
||||
system_memory_bytes: int = 0
|
||||
cpu_count: int = 0
|
||||
gpu_name: str = ""
|
||||
gpu_memory_bytes: int = 0
|
||||
gpu_utilization_percent: Optional[float] = None
|
||||
|
||||
@property
|
||||
def is_local(self) -> bool:
|
||||
@@ -121,6 +142,13 @@ class Target:
|
||||
"latency_ms": round(self.latency_ms, 1),
|
||||
"active_tasks": self.active_tasks,
|
||||
"max_tasks": self.max_tasks,
|
||||
"cpu_percent": self.cpu_percent,
|
||||
"free_memory_bytes": self.free_memory_bytes,
|
||||
"system_memory_bytes": self.system_memory_bytes,
|
||||
"cpu_count": self.cpu_count,
|
||||
"gpu_name": self.gpu_name,
|
||||
"gpu_memory_bytes": self.gpu_memory_bytes,
|
||||
"gpu_utilization_percent": self.gpu_utilization_percent,
|
||||
}
|
||||
|
||||
|
||||
@@ -178,6 +206,8 @@ def list_targets(control_plane=None) -> list[Target]:
|
||||
pool = getattr(control_plane, "pool", None) if control_plane.running else None
|
||||
for record in enrolled:
|
||||
live = pool.get(record.id) if pool is not None else None
|
||||
host = record.host or {}
|
||||
gpu = (host.get("gpus") or [{}])[0]
|
||||
connected = live is not None and not live.stale()
|
||||
available, detail = _availability(record, live, pool)
|
||||
targets.append(
|
||||
@@ -193,6 +223,13 @@ def list_targets(control_plane=None) -> list[Target]:
|
||||
latency_ms=live.latency_ms if live else 0.0,
|
||||
active_tasks=live.capacity.active_tasks if live else 0,
|
||||
max_tasks=live.capacity.max_concurrent_tasks if live else 0,
|
||||
cpu_percent=live.capacity.cpu_percent if live else None,
|
||||
free_memory_bytes=live.capacity.free_memory_bytes if live else None,
|
||||
system_memory_bytes=int(host.get("system_memory_bytes") or 0),
|
||||
cpu_count=int(host.get("cpu_count") or 0),
|
||||
gpu_name=str(gpu.get("model") or ""),
|
||||
gpu_memory_bytes=int(gpu.get("memory_bytes") or 0),
|
||||
gpu_utilization_percent=live.capacity.gpu_utilization_percent if live else None,
|
||||
)
|
||||
)
|
||||
return targets
|
||||
|
||||
@@ -43,6 +43,8 @@ import platform
|
||||
import random
|
||||
import socket
|
||||
import sys
|
||||
import threading
|
||||
from concurrent.futures import Future
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Awaitable, Callable, Optional, Protocol
|
||||
|
||||
@@ -73,6 +75,33 @@ _FALLBACK_MODEL_LOAD_SECONDS = 1800.0
|
||||
# see _oversized_result_error for why that has to be a failure and not a retry.
|
||||
MAX_MESSAGE_BYTES = 8 * 1024 * 1024
|
||||
|
||||
|
||||
def _heartbeat_resources() -> tuple[Optional[float], Optional[int], Optional[float]]:
|
||||
"""Sample cheap host telemetry without making a heartbeat depend on CUDA."""
|
||||
cpu_percent = free_memory_bytes = gpu_utilization_percent = None
|
||||
try:
|
||||
import psutil
|
||||
|
||||
cpu_percent = float(psutil.cpu_percent(interval=None))
|
||||
except Exception:
|
||||
logger.debug("Could not sample worker CPU usage", exc_info=True)
|
||||
try:
|
||||
import torch
|
||||
|
||||
if torch.cuda.is_available():
|
||||
free_memory_bytes = int(torch.cuda.mem_get_info()[0])
|
||||
except Exception:
|
||||
logger.debug("Could not sample worker free VRAM", exc_info=True)
|
||||
try:
|
||||
import pynvml
|
||||
|
||||
pynvml.nvmlInit()
|
||||
handle = pynvml.nvmlDeviceGetHandleByIndex(0)
|
||||
gpu_utilization_percent = float(pynvml.nvmlDeviceGetUtilizationRates(handle).gpu)
|
||||
except Exception:
|
||||
logger.debug("Could not sample worker GPU usage", exc_info=True)
|
||||
return cpu_percent, free_memory_bytes, gpu_utilization_percent
|
||||
|
||||
# Room left for result_json, the ref, and protobuf framing when a payload does
|
||||
# ride inline. The inline decision is made on the payload alone, so without a
|
||||
# reserve a payload sized exactly at the frame cap would overflow it.
|
||||
@@ -274,12 +303,19 @@ def describe_host() -> dict:
|
||||
from core.version import APP_VERSION # noqa: PLC0415
|
||||
except Exception:
|
||||
APP_VERSION = ""
|
||||
try:
|
||||
import psutil # noqa: PLC0415
|
||||
|
||||
system_memory_bytes = int(psutil.virtual_memory().total)
|
||||
except Exception:
|
||||
system_memory_bytes = 0
|
||||
return {
|
||||
"hostname": socket.gethostname(),
|
||||
"os": {"darwin": "darwin", "win32": "windows"}.get(sys.platform, "linux"),
|
||||
"arch": platform.machine(),
|
||||
"worker_version": APP_VERSION,
|
||||
"cpu_count": os.cpu_count() or 0,
|
||||
"system_memory_bytes": system_memory_bytes,
|
||||
}
|
||||
|
||||
|
||||
@@ -332,6 +368,13 @@ class WorkerClient:
|
||||
self._running: dict[str, asyncio.Task] = {}
|
||||
self._keepalives: dict[str, asyncio.Task] = {}
|
||||
self._maintenance: set[asyncio.Task] = set()
|
||||
self._telemetry: tuple[Optional[float], Optional[int], Optional[float]] = (None, None, None)
|
||||
# A driver query can hang indefinitely. Keep that one query owned
|
||||
# rather than cancelling its awaiter and starting a fresh thread at
|
||||
# every heartbeat.
|
||||
self._telemetry_task: Optional[asyncio.Future] = None
|
||||
self._prewarms: dict[str, asyncio.Task] = {}
|
||||
self._prewarm_cancellations: dict[str, asyncio.Task] = {}
|
||||
self._epoch = 0
|
||||
self._session_token = ""
|
||||
# Negotiated by ConfigUpdate; None means "use the executor's own
|
||||
@@ -435,6 +478,8 @@ class WorkerClient:
|
||||
*draining, return_exceptions=True
|
||||
)
|
||||
self._maintenance.clear()
|
||||
self._prewarms.clear()
|
||||
self._prewarm_cancellations.clear()
|
||||
for key, task in running:
|
||||
if self._running.get(key) is task:
|
||||
self._running.pop(key, None)
|
||||
@@ -672,10 +717,59 @@ class WorkerClient:
|
||||
async def _heartbeat_loop(self, interval: float) -> None:
|
||||
while True:
|
||||
await asyncio.sleep(interval)
|
||||
await self._refresh_telemetry()
|
||||
await self._send(self.heartbeat_message())
|
||||
|
||||
async def _refresh_telemetry(self) -> None:
|
||||
"""Publish completed samples and retain one non-blocking probe.
|
||||
|
||||
CUDA/NVML calls may wedge in a driver. A timed ``to_thread`` await
|
||||
only cancels the awaiter, leaving that thread alive; retaining this
|
||||
task prevents later heartbeats from accumulating more blocked probes.
|
||||
"""
|
||||
if not self._accepting_assignments or self._stop.is_set():
|
||||
return
|
||||
task = self._telemetry_task
|
||||
if task is not None and task.done():
|
||||
try:
|
||||
sampled = task.result()
|
||||
except Exception:
|
||||
logger.debug("Could not sample worker telemetry", exc_info=True)
|
||||
else:
|
||||
# A partial failed sample must not erase an independent last
|
||||
# good value. Presence on the heartbeat remains honest until
|
||||
# that individual metric can next be measured.
|
||||
self._telemetry = tuple(
|
||||
current if value is None else value
|
||||
for current, value in zip(self._telemetry, sampled)
|
||||
)
|
||||
self._telemetry_task = None
|
||||
|
||||
if self._telemetry_task is None:
|
||||
# Read-only driver probes cannot be interrupted. Keep one across
|
||||
# reconnects, outside assignment drain and the shared executor
|
||||
# (whose shutdown would otherwise wait forever for a wedged driver).
|
||||
result = Future()
|
||||
self._telemetry_task = asyncio.wrap_future(result)
|
||||
|
||||
def sample() -> None:
|
||||
try:
|
||||
result.set_result(_heartbeat_resources())
|
||||
except Exception:
|
||||
logger.debug("Could not sample worker telemetry", exc_info=True)
|
||||
result.set_result((None, None, None))
|
||||
|
||||
threading.Thread(
|
||||
target=sample, name="worker-telemetry-probe", daemon=True,
|
||||
).start()
|
||||
|
||||
def heartbeat_message(self) -> pb.WorkerMessage:
|
||||
"""Build the worker's current liveness/capacity frame."""
|
||||
cpu_percent, free_memory_bytes, gpu_utilization_percent = self._telemetry
|
||||
telemetry = {}
|
||||
if cpu_percent is not None: telemetry["cpu_percent"] = cpu_percent
|
||||
if free_memory_bytes is not None: telemetry["free_memory_bytes"] = free_memory_bytes
|
||||
if gpu_utilization_percent is not None: telemetry["gpu_utilization_percent"] = gpu_utilization_percent
|
||||
return pb.WorkerMessage(
|
||||
heartbeat=pb.Heartbeat(
|
||||
active_tasks=len(self._running),
|
||||
@@ -683,6 +777,7 @@ class WorkerClient:
|
||||
0, self.config.max_concurrent_tasks - len(self._running)
|
||||
),
|
||||
resident_models=self._resident_models(),
|
||||
**telemetry,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -801,16 +896,105 @@ class WorkerClient:
|
||||
elif kind == "prewarm":
|
||||
if not self._accepting_assignments:
|
||||
return
|
||||
model_id = message.prewarm.model_id
|
||||
existing = self._prewarms.get(model_id)
|
||||
if model_id and existing is not None and not existing.done():
|
||||
return
|
||||
task = asyncio.create_task(
|
||||
self._on_prewarm(message.prewarm), name="worker-prewarm"
|
||||
)
|
||||
self._maintenance.add(task)
|
||||
if model_id:
|
||||
self._prewarms[model_id] = task
|
||||
task.add_done_callback(self._maintenance_finished)
|
||||
elif kind == "model_install_cancel":
|
||||
await self._cancel_model_install(message.model_install_cancel)
|
||||
|
||||
def _maintenance_finished(self, task: asyncio.Task) -> None:
|
||||
self._maintenance.discard(task)
|
||||
for tasks in (self._prewarms, self._prewarm_cancellations):
|
||||
for model_id, current in tuple(tasks.items()):
|
||||
if current is task:
|
||||
tasks.pop(model_id, None)
|
||||
self._maybe_finish_drain()
|
||||
|
||||
async def _cancel_model_install(
|
||||
self, request: pb.ModelInstallCancelRequest
|
||||
) -> None:
|
||||
"""Cancel one explicit catalogue install without blocking control I/O."""
|
||||
model_id = request.model_id.strip()
|
||||
capability = next(
|
||||
(
|
||||
cap
|
||||
for cap in (self.config.capabilities or [])
|
||||
if cap.get("model_id") == model_id
|
||||
),
|
||||
None,
|
||||
)
|
||||
repo_ids = list((capability or {}).get("repo_ids") or [])
|
||||
if len(repo_ids) != 1:
|
||||
logger.warning("Ignoring model cancellation for unknown model %s", model_id)
|
||||
return
|
||||
repo_id = repo_ids[0]
|
||||
task = self._prewarms.get(model_id)
|
||||
if task is None or task.done():
|
||||
await self._send_model_install_terminal(
|
||||
repo_id,
|
||||
"install_done"
|
||||
if bool((capability or {}).get("downloaded"))
|
||||
else "install_cancelled",
|
||||
)
|
||||
return
|
||||
existing = self._prewarm_cancellations.get(model_id)
|
||||
if existing is not None and not existing.done():
|
||||
return
|
||||
task.cancel()
|
||||
confirmation = asyncio.create_task(
|
||||
self._confirm_model_install_cancel(task, repo_id),
|
||||
name="worker-model-install-cancel",
|
||||
)
|
||||
self._maintenance.add(confirmation)
|
||||
self._prewarm_cancellations[model_id] = confirmation
|
||||
confirmation.add_done_callback(self._maintenance_finished)
|
||||
|
||||
async def _confirm_model_install_cancel(
|
||||
self, task: asyncio.Task, repo_id: str
|
||||
) -> None:
|
||||
await asyncio.gather(task, return_exceptions=True)
|
||||
capability = next(
|
||||
(
|
||||
cap
|
||||
for cap in (self.config.capabilities or [])
|
||||
if repo_id in (cap.get("repo_ids") or [])
|
||||
),
|
||||
None,
|
||||
)
|
||||
await self._send_model_install_terminal(
|
||||
repo_id,
|
||||
"install_done"
|
||||
if bool((capability or {}).get("downloaded"))
|
||||
else "install_cancelled",
|
||||
)
|
||||
|
||||
async def _send_model_install_terminal(self, repo_id: str, phase: str) -> None:
|
||||
event = {
|
||||
"repo_id": repo_id,
|
||||
"filename": repo_id,
|
||||
"downloaded": 0,
|
||||
"total": 0,
|
||||
"pct": 0.0,
|
||||
"phase": phase,
|
||||
}
|
||||
await self._send(
|
||||
pb.WorkerMessage(
|
||||
download_progress=pb.DownloadProgress(
|
||||
event_json=json.dumps(
|
||||
event, separators=(",", ":"), ensure_ascii=False
|
||||
)
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
def _maybe_finish_drain(self) -> None:
|
||||
if (
|
||||
self._draining
|
||||
@@ -906,6 +1090,21 @@ class WorkerClient:
|
||||
|
||||
async def _on_assignment(self, assignment: pb.TaskAssignment) -> None:
|
||||
key = self._key(assignment.ref)
|
||||
# Assignment delivery is at-least-once. A reconnect or a control-stream
|
||||
# retry may repeat the exact same attempt while it is still running or
|
||||
# waiting for its result acknowledgement. Treating that repeat as a
|
||||
# capacity rejection terminalizes the original attempt underneath its
|
||||
# result upload; starting it again spends the GPU twice. Reaffirm the
|
||||
# live claim, or redeliver the result we already hold.
|
||||
if key in self._running:
|
||||
await self._send(
|
||||
pb.WorkerMessage(accepted=pb.TaskAccepted(ref=assignment.ref))
|
||||
)
|
||||
return
|
||||
pending = self._pending.get(key)
|
||||
if pending is not None:
|
||||
await self._send(_result_message(pending), bulk=True)
|
||||
return
|
||||
if not self._accepting_assignments or self._stop.is_set():
|
||||
await self._send(
|
||||
pb.WorkerMessage(
|
||||
@@ -1164,7 +1363,8 @@ class WorkerClient:
|
||||
break
|
||||
resumed = int(ack.bytes_received)
|
||||
if ack.error.code and ack.error.code != "OFFSET_MISMATCH":
|
||||
raise RuntimeError(ack.error.message or "the control plane refused the upload")
|
||||
detail = ack.error.message or "the control plane refused the upload"
|
||||
raise RuntimeError(f"{ack.error.code}: {detail}")
|
||||
if resumed < 0 or resumed > len(payload) or resumed == offset:
|
||||
raise RuntimeError(ack.error.message or "the control plane could not resume the upload")
|
||||
offset = resumed
|
||||
|
||||
@@ -71,6 +71,7 @@ REQUIRED_FEATURES = frozenset({
|
||||
"task_progress_v1",
|
||||
"task_inputs_v1",
|
||||
"remote_model_download_v1",
|
||||
"remote_model_cancel_v1",
|
||||
# A generic backend.generate() call accepts the same wire shape but drops
|
||||
# profile conditioning controls. Require the canonical worker render path
|
||||
# so an older peer cannot successfully return a different voice.
|
||||
@@ -456,10 +457,22 @@ class _Upload:
|
||||
)
|
||||
await to_thread_and_drain_on_cancel(_write_all, self._handle, data)
|
||||
if self._discarded or self.session.revoked or self.attempt.state.terminal:
|
||||
logger.warning(
|
||||
"Refusing result upload for task %s attempt %s "
|
||||
"(attempt=%s, session_revoked=%s, discarded=%s, error=%s)",
|
||||
self.attempt.task_id,
|
||||
self.attempt.attempt_id,
|
||||
self.attempt.state.value,
|
||||
self.session.revoked,
|
||||
self._discarded,
|
||||
getattr(self.attempt.error, "code", None),
|
||||
)
|
||||
await self.discard_async()
|
||||
return _upload_refused(
|
||||
"ATTEMPT_NOT_LIVE",
|
||||
"This attempt stopped accepting a result during upload.",
|
||||
"This attempt stopped accepting a result during upload "
|
||||
f"(attempt={self.attempt.state.value}, "
|
||||
f"error={getattr(self.attempt.error, 'code', None)}).",
|
||||
error_class=pb.ERROR_CLASS_TRANSIENT,
|
||||
)
|
||||
self._digest.update(data)
|
||||
@@ -486,10 +499,22 @@ class _Upload:
|
||||
error_class=pb.ERROR_CLASS_TRANSIENT,
|
||||
)
|
||||
if self._discarded or self.session.revoked or self.attempt.state.terminal:
|
||||
logger.warning(
|
||||
"Refusing result commit for task %s attempt %s "
|
||||
"(attempt=%s, session_revoked=%s, discarded=%s, error=%s)",
|
||||
self.attempt.task_id,
|
||||
self.attempt.attempt_id,
|
||||
self.attempt.state.value,
|
||||
self.session.revoked,
|
||||
self._discarded,
|
||||
getattr(self.attempt.error, "code", None),
|
||||
)
|
||||
await self.discard_async()
|
||||
return _upload_refused(
|
||||
"ATTEMPT_NOT_LIVE",
|
||||
"This attempt is no longer accepting a result.",
|
||||
"This attempt is no longer accepting a result "
|
||||
f"(attempt={self.attempt.state.value}, "
|
||||
f"error={getattr(self.attempt.error, 'code', None)}).",
|
||||
error_class=pb.ERROR_CLASS_TRANSIENT,
|
||||
)
|
||||
try:
|
||||
@@ -509,6 +534,16 @@ class _Upload:
|
||||
# running in its thread. It must win before the commit callback spends
|
||||
# budget or this RPC licenses the worker to forget its only copy.
|
||||
if self._discarded or self.session.revoked or self.attempt.state.terminal:
|
||||
logger.warning(
|
||||
"Refusing result after durable write for task %s attempt %s "
|
||||
"(attempt=%s, session_revoked=%s, discarded=%s, error=%s)",
|
||||
self.attempt.task_id,
|
||||
self.attempt.attempt_id,
|
||||
self.attempt.state.value,
|
||||
self.session.revoked,
|
||||
self._discarded,
|
||||
getattr(self.attempt.error, "code", None),
|
||||
)
|
||||
try:
|
||||
os.remove(self.final)
|
||||
except OSError:
|
||||
@@ -517,7 +552,9 @@ class _Upload:
|
||||
self._on_finished(self)
|
||||
return _upload_refused(
|
||||
"ATTEMPT_NOT_LIVE",
|
||||
"This attempt stopped accepting a result during commit.",
|
||||
"This attempt stopped accepting a result during commit "
|
||||
f"(attempt={self.attempt.state.value}, "
|
||||
f"error={getattr(self.attempt.error, 'code', None)}).",
|
||||
error_class=pb.ERROR_CLASS_TRANSIENT,
|
||||
)
|
||||
self._on_finished(self)
|
||||
@@ -2031,7 +2068,9 @@ class WorkerServicer(pb_grpc.WorkerServiceServicer):
|
||||
active_tasks=active_tasks,
|
||||
available_slots=available_slots,
|
||||
resident_models=set(beat.resident_models),
|
||||
free_memory_bytes=beat.free_memory_bytes,
|
||||
free_memory_bytes=beat.free_memory_bytes if beat.HasField("free_memory_bytes") else None,
|
||||
cpu_percent=beat.cpu_percent if beat.HasField("cpu_percent") else None,
|
||||
gpu_utilization_percent=beat.gpu_utilization_percent if beat.HasField("gpu_utilization_percent") else None,
|
||||
)
|
||||
self._queue_heartbeat_touch(session)
|
||||
return
|
||||
@@ -2065,8 +2104,10 @@ class WorkerServicer(pb_grpc.WorkerServiceServicer):
|
||||
# The authenticated session, never the worker payload, is the
|
||||
# authoritative target identity.
|
||||
event["target"] = session.worker_id
|
||||
from services import gpu_gateway # noqa: PLC0415
|
||||
from utils import hf_progress # noqa: PLC0415
|
||||
|
||||
gpu_gateway.record_remote_download_progress(session.worker_id, event)
|
||||
hf_progress.emit(event)
|
||||
except (TypeError, ValueError, json.JSONDecodeError):
|
||||
logger.warning("Worker %s sent malformed download progress", session.worker_id)
|
||||
@@ -2723,6 +2764,17 @@ class WorkerServicer(pb_grpc.WorkerServiceServicer):
|
||||
)))
|
||||
return True
|
||||
|
||||
async def cancel_model_install(self, worker_id: str, *, model_id: str) -> bool:
|
||||
session = self._sessions.get(worker_id)
|
||||
if session is None:
|
||||
return False
|
||||
await session.send(
|
||||
pb.ServerMessage(
|
||||
model_install_cancel=pb.ModelInstallCancelRequest(model_id=model_id)
|
||||
)
|
||||
)
|
||||
return True
|
||||
|
||||
def revoke_worker_sessions(self, worker_id: str) -> int:
|
||||
"""Invalidate every transport generation for a durably revoked worker."""
|
||||
sessions = {
|
||||
@@ -3104,11 +3156,16 @@ class WorkerServicer(pb_grpc.WorkerServiceServicer):
|
||||
upload: Optional[_Upload] = None
|
||||
try:
|
||||
if not self._begin_uploading(attempt):
|
||||
task = self.scheduler.get(attempt.task_id)
|
||||
self._release_artifact_reservation(
|
||||
final, owner=reservation_owner
|
||||
)
|
||||
return None, _upload_refused(
|
||||
"ATTEMPT_NOT_LIVE", "This attempt is no longer accepting a result."
|
||||
"ATTEMPT_NOT_LIVE",
|
||||
"This attempt is no longer accepting a result "
|
||||
f"(task={getattr(getattr(task, 'state', None), 'value', 'missing')}, "
|
||||
f"attempt={attempt.state.value}, "
|
||||
f"error={getattr(attempt.error, 'code', None)}).",
|
||||
), session
|
||||
upload = _Upload(
|
||||
session=session,
|
||||
@@ -3167,6 +3224,15 @@ class WorkerServicer(pb_grpc.WorkerServiceServicer):
|
||||
"""
|
||||
task = self.scheduler.get(attempt.task_id)
|
||||
if task is None or task.state.terminal or attempt.state.terminal:
|
||||
logger.warning(
|
||||
"Refusing result upload admission for task %s attempt %s "
|
||||
"(task=%s, attempt=%s, error=%s)",
|
||||
attempt.task_id,
|
||||
attempt.attempt_id,
|
||||
getattr(getattr(task, "state", None), "value", "missing"),
|
||||
attempt.state.value,
|
||||
getattr(attempt.error, "code", None),
|
||||
)
|
||||
return False
|
||||
try:
|
||||
task.uploading(attempt.attempt_id, session_epoch=attempt.session_epoch)
|
||||
|
||||
+62
-26
@@ -71,26 +71,18 @@ git tag vX.Y.Z
|
||||
git push origin vX.Y.Z
|
||||
```
|
||||
|
||||
The `Desktop Release` workflow fires on tag push. It builds four targets in parallel on GitHub Actions runners:
|
||||
`electron-release.yml` builds Linux x64, Windows x64, macOS arm64 and macOS
|
||||
x64 installers with updater metadata and packaged startup checks. Ordinary tag
|
||||
pushes create drafts; a tag-scoped manual dispatch with `publish=true` publishes
|
||||
after all four targets pass. Signing checks apply by default.
|
||||
|
||||
| Target | Runner | Artifact |
|
||||
|---|---|---|
|
||||
| macOS Apple Silicon | macos-14 | `.dmg` + updater `.app.tar.gz` |
|
||||
| macOS Intel | macos-13 | `.dmg` + updater `.app.tar.gz` |
|
||||
| Windows x64 | windows-2022 | `.msi`, machine-wide and per-user, each with its updater `.sig` |
|
||||
| Linux x64 | ubuntu-22.04 | `.AppImage` + updater `.AppImage.sig` |
|
||||
|
||||
Each runner signs the updater payload with the stored `TAURI_SIGNING_PRIVATE_KEY`, merges into a single `latest.json`, and attaches everything to the draft release.
|
||||
|
||||
Workflow runtime: **~20-40 minutes** (PyInstaller + four platform builds). Follow progress at:
|
||||
`https://github.com/debpalash/VoiceStudio/actions`
|
||||
|
||||
The release stays a draft while the platforms build. Once every platform, the
|
||||
updater-manifest repair and the uninstall scripts are done, the
|
||||
`release-notes-checksums` job writes all four platforms' checksums into the
|
||||
notes and publishes it, with no manual step. A failed platform leaves the
|
||||
release a draft, so nothing half-built goes public. Existing clients detect
|
||||
the update on their next launch.
|
||||
For the one-time transition tag, set `TAURI_SUNSET_TAG`, dispatch `release.yml`
|
||||
on that tag with `draft=true`, and wait for its final Tauri installers and signed
|
||||
updater feeds. Then dispatch `electron-release.yml` on the same tag. Automatic
|
||||
Electron builds are skipped for this tag to avoid racing the Tauri draft.
|
||||
Keep the release draft until both builds and their checks have passed.
|
||||
See [Electron transition](#electron-transition-next-desktop-release) below for
|
||||
signing requirements and the explicit owner-only unsigned exception.
|
||||
|
||||
## 5b. Deployment channels — all must ship (hard rule, owner-set 2026-07-16)
|
||||
|
||||
@@ -100,8 +92,9 @@ bug to fix immediately, not backlog.
|
||||
|
||||
| Channel | Source | Produced by | How to verify |
|
||||
|---|---|---|---|
|
||||
| GitHub Release: installers + signed `latest.json` (**Stable** updater channel) | the `vX.Y.Z` tag | `release.yml` on tag push | Release page has dmg (arm+intel), msi (machine-wide and per-user), AppImage, `latest.json` and `latest-user.json`; body = the CHANGELOG section (not the auto-generated fallback), followed by per-platform checksums and a **Contributors** avatar strip (owner + every PR author for the tag — the `contributors-strip` job) |
|
||||
| **Preview** updater channel (rolling `preview` prerelease) | **`main` only** | `release.yml` nightly cron / manual dispatch | preview `latest.json` uses main's version when it is ahead; otherwise it advances the stable patch, then appends `-N` so it semver-sorts above stable |
|
||||
| GitHub Release: Electron installers and updater manifests | the `vX.Y.Z` tag | `electron-release.yml`, explicit publish dispatch | All four platforms, Electron manifests, SHA256SUMS.txt, versioned CHANGELOG notes; retained Tauri feeds point to the final Tauri tag |
|
||||
| Final Tauri installers and signed updater feeds | `TAURI_SUNSET_TAG` | `release.yml`, manual dispatch only | Both macOS architectures, Windows system/user installers, Linux AppImage, signed `latest.json` and `latest-user.json` |
|
||||
| Desktop preview channel | frozen during transition | no scheduled publishing | Existing preview assets remain available; new desktop previews are paused |
|
||||
| GHCR CUDA image: `:X.Y.Z`, `:X.Y`, `:stable` | the tag | `docker.yml` on tag push | `docker manifest inspect ghcr.io/debpalash/omnivoice-studio:X.Y.Z` |
|
||||
| GHCR ROCm image: `:X.Y.Z-rocm`, `:X.Y-rocm`, `:stable-rocm` | the tag | `docker.yml` on tag push | same, with `-rocm` suffix |
|
||||
| Docker Hub mirror of **all** the above tags | the tag | `docker.yml` (gated on `DOCKERHUB_*` secrets) | tag list at hub.docker.com/r/palashdeb/omnivoice-studio/tags |
|
||||
@@ -109,11 +102,8 @@ bug to fix immediately, not backlog.
|
||||
| Rolling Docker previews: `:latest`, `:main`, `:rocm` | **`main` only** | `docker.yml` on every main push | tag timestamps move with main |
|
||||
|
||||
**Preview/RC policy:** there are no RC tags (beta cadence — see CLAUDE.md).
|
||||
The preview channel *is* the release candidate, and it **always builds from
|
||||
`main`** — the preview-gate in `release.yml` refuses `publish_preview` from
|
||||
any other branch, and the rolling Docker tags track `main` by construction.
|
||||
To get users testing a fix: merge to `main`, then cut a preview. Never a
|
||||
side-branch build.
|
||||
Rolling Docker previews always build from `main`. Desktop preview publication
|
||||
is paused during the Electron transition; never publish a side-branch preview.
|
||||
|
||||
## 6. Expect-to-fail-first-time on Windows and Linux
|
||||
|
||||
@@ -161,3 +151,49 @@ before Tauri uploads them again. A macOS retry also replaces that architecture's
|
||||
versionless updater archive. Other versions, sibling platforms, and updater
|
||||
manifests remain intact. Inventory or deletion permission/network failures stop
|
||||
the job instead of hiding an upload collision.
|
||||
|
||||
## Electron transition (next desktop release)
|
||||
|
||||
Electron is the primary desktop distribution. electron-release.yml builds Linux
|
||||
x64, Windows x64, macOS arm64 and macOS x64, checks packaged startup and updater
|
||||
artifacts, then creates a draft. Publishing requires a tag-scoped manual dispatch
|
||||
with publish=true. Tag pushes never publish automatically. electron-build.yml
|
||||
remains the artifact-only rehearsal; run it before tagging.
|
||||
|
||||
Set TAURI_SUNSET_TAG to the final Tauri version tag. Run the manual release.yml
|
||||
on that tag first; it rejects other refs. Automatic Tauri builds and scheduled
|
||||
previews are retired. Keep the transition release draft until Electron on the
|
||||
same tag completes. Electron requires the final signed latest.json and
|
||||
latest-user.json assets; subsequent releases copy those feeds without changing
|
||||
their immutable sunset payload URLs. Retain the sunset release and its assets.
|
||||
|
||||
Write versioned CHANGELOG notes before release. Review all four platform builds,
|
||||
checksums, signing requirements and docs/electron-migration.md. Existing Electron
|
||||
artifact names and app IDs remain stable for updater compatibility. This pipeline
|
||||
ships stable releases; rolling preview publication is paused during transition.
|
||||
|
||||
Preparation is not proof of cross-platform packaging, signing, migration, or a
|
||||
real installed update hop. Record those results before release. Keep Tauri source
|
||||
and shared assets until remaining Electron resource references are relocated.
|
||||
No tag, version bump, or publishing is authorized by workflow preparation alone.
|
||||
|
||||
Electron signing uses ELECTRON_CSC_LINK and ELECTRON_CSC_KEY_PASSWORD secrets.
|
||||
Without them rehearsal/draft artifacts are unsigned or ad-hoc signed. Publishing
|
||||
checks macOS signing/notarization and Windows Authenticode signatures by default.
|
||||
The owner may explicitly choose the existing unsigned-release policy by dispatching
|
||||
with `allow_unsigned=true` (both dispatch and rerun actors must be the repository owner); the release notes then disclose OS trust warnings and
|
||||
unverified macOS automatic updates. Never select this exception without the owner's
|
||||
choice. Tauri's signing keys do not sign Electron packages.
|
||||
|
||||
For the transition tag, automatic Electron release jobs are skipped. Build the
|
||||
manual Tauri sunset draft first, then dispatch Electron on the same tag after
|
||||
its signed updater feeds exist. Later tags build Electron automatically.
|
||||
|
||||
|
||||
If a packaging-workflow fix is needed after tagging, keep the release tag
|
||||
immutable. Merge and validate the workflow fix on main, then dispatch
|
||||
`electron-release.yml` from main with `release_tag=vX.Y.Z`. Validation and every
|
||||
packaging/release job check out that exact tag; only the workflow comes from
|
||||
main. Empty signing secrets are omitted from the builder environment so drafts
|
||||
and explicitly accepted unsigned builds do not interpret the working directory
|
||||
as a certificate. Publication still requires `publish=true` and the same guards.
|
||||
|
||||
+3
-3
@@ -35,11 +35,11 @@ VoiceStudio/
|
||||
├── backend/ ⟵ FastAPI server
|
||||
│ ├── main.py the one entry point; its boot order is load-bearing —
|
||||
│ │ read the comments before reordering anything
|
||||
│ ├── api/routers/ 39 routers, auto-included; thin HTTP/WS surface
|
||||
│ ├── api/routers/ 40 routers, auto-included; thin HTTP/WS surface
|
||||
│ │ └── setup/ first-run wizard, model download
|
||||
│ ├── core/ config, db, job queue, event bus, auth/CSRF, path security,
|
||||
│ │ opt-in analytics, version, diagnostics
|
||||
│ ├── services/ 78 modules of business logic — TTS, dubbing pipeline,
|
||||
│ ├── services/ 86 modules of business logic — TTS, dubbing pipeline,
|
||||
│ │ audio DSP, GPU gateway, engine routing, model lifecycle
|
||||
│ ├── engines/ per-engine adapters: indextts, supertonic3, confucius4,
|
||||
│ │ dots_tts, moss_tts_v15, pockettts, audiocpp,
|
||||
@@ -103,7 +103,7 @@ VoiceStudio/
|
||||
│
|
||||
├── .agents/skills/ ⟵ canonical skill copies (vite, fastapi-python), pinned by
|
||||
│ skills-lock.json — followed by path, never symlinked
|
||||
├── skills/ ⟵ skills this repo publishes (omnivoice, oss-maintainer)
|
||||
├── skills/ ⟵ skills this repo publishes (voicestudio, voicestudio-maintainer)
|
||||
│
|
||||
├── infra/ ⟵ edge/deploy workers (not the Docker deploy path)
|
||||
│ └── install-redirect/ voicestudio.sh/install — UA-sniffing installer worker
|
||||
|
||||
+8
-1
@@ -11,7 +11,9 @@ spark identifies creation. The product voice is clear, calm, and direct.
|
||||
| README mark | `docs/logo.png` and `docs/logo-256.png` |
|
||||
| Browser icon | `frontend/public/favicon.svg` |
|
||||
| In-app mark | `frontend/src/components/brand/VoiceStudioMark.jsx` |
|
||||
| Desktop/platform icons | `frontend/src-tauri/icons/` |
|
||||
| Desktop/platform icons (Tauri and Electron) | `frontend/src-tauri/icons/` |
|
||||
| Electron sidebar and browser icon | `frontend/public/favicon.svg` via `electron/src/renderer/src/lib/brand.ts` |
|
||||
| Shared sidebar/launchpad artwork | `frontend/src/assets/signal-field.webp` |
|
||||
|
||||
Regenerate every desktop icon from the canonical vector after changing the
|
||||
mark:
|
||||
@@ -59,3 +61,8 @@ a separately tested migration exists:
|
||||
|
||||
Visible copy can explain those compatibility names, but must not silently rename
|
||||
them on disk or over the wire.
|
||||
|
||||
Electron uses the shared multi-resolution ICO for Windows window/taskbar and tray
|
||||
icons, the shared PNG for Linux and macOS runtime icons, and the ICNS for the macOS
|
||||
bundle. The tray icon restores the window; closing the app retains its existing
|
||||
quit behavior. Installed executable icons are applied when building the installer.
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
# Electron batch dubbing
|
||||
|
||||
Open Batch dubbing from the cloning sidebar or command search. Add video files,
|
||||
choose one or more target languages, optionally select a saved voice, and choose
|
||||
whether to preserve background audio. Add to Queue submits through the existing
|
||||
backend. Successfully submitted files leave the upload list; failed files remain.
|
||||
The backend enforces ASR readiness and owns translation, generation and mixing.
|
||||
The Electron setup sidebar accepts file picking or drag-and-drop, groups media,
|
||||
languages and voice/audio choices into stable cards, and exposes the same Add Videos
|
||||
action from the empty job view.
|
||||
|
||||
Active, Completed and Failed views poll backend jobs. Progress shows backend stages
|
||||
and percentages. Active jobs can be cancelled; finished records require an inline
|
||||
confirmation before deletion. Completed language outputs use the native save dialog.
|
||||
Reloading the renderer reads existing jobs and never re-enqueues them. Backend
|
||||
process restart recovery is not established by this behavior: the batch queue is
|
||||
in memory. Native watch folders detect settled files through a capability-scoped
|
||||
directory handle and stream multipart uploads directly to the selected local or
|
||||
HTTPS remote backend. Remote uploads receive the scoped session from Electron main;
|
||||
the renderer never handles it.
|
||||
|
||||
Verification: node electron/tests/batch-smoke.mjs uses mocked jobs to check file
|
||||
selection, enqueue, progress, renderer reload, cancellation, export availability
|
||||
and confirmed deletion. The enqueue unit regression covers partial failure and
|
||||
language/voice fields. The native helper smoke verifies confined streaming,
|
||||
authorization replacement, rename detection and revocation. No test yet establishes
|
||||
real model-backed batch completion or a separate remote-machine transfer.
|
||||
@@ -0,0 +1,50 @@
|
||||
# Electron network and credentials
|
||||
|
||||
Settings now exposes Network and Credentials in the shared settings shell. Sidebar search includes proxy, Hugging Face, DeepL and Microsoft labels.
|
||||
|
||||
Network reads the configured proxy from `/system/info`. Save and Clear update all six upper/lowercase HTTP, HTTPS and ALL proxy variables using the same helper as Tauri. Writes run sequentially; failures stop the sequence and do not show success. A partial failure can be retried or cleared. Saving does not modify the audio-tool executable setting; its link opens Audio tools.
|
||||
|
||||
Credentials shows the Hugging Face resolver sources (App, environment, CLI), masked values, active source and validation status. Ordinary reads use local state only. Test now explicitly requests fresh validation. Save clears the password input after success. Clearing asks inline and clears only the app token by default; CLI-file removal requires its separate opt-in switch. Environment credentials remain managed outside the app. Secret inputs never enter draft persistence or browser storage.
|
||||
|
||||
DeepL and Microsoft keys/base URLs use the existing persisted `/system/set-env` contract. Field definitions are shared with Tauri. Blank inputs cannot overwrite stored credentials; successful saves clear the input. Provider connectivity is not inferred from a successful settings write.
|
||||
|
||||
`electron/tests/connection-settings-smoke.mjs` checks proxy save/reload/clear, partial failure, local token reads, explicit validation, duplicate submission, app-only/CLI clearing, and provider-key submission against mocked routes. The live backend token-state read returned the expected source schema. No real proxy or credential values were changed for verification. Tauri Network and Translation regression tests pass after helper extraction.
|
||||
|
||||
Model settings now include Hugging Face mirror selection: automatic routing,
|
||||
backend-advertised presets and a custom URL. Settings reads use cached endpoint
|
||||
status; the network test runs only on explicit click. Writes use the existing
|
||||
backend and honor its restart-required response. Browser fixtures verify saves,
|
||||
reload, failed writes and Auto reset. The live read-only schema was verified;
|
||||
no real mirror preference was changed or probe triggered during verification.
|
||||
|
||||
Model downloads follow the compute target selected when the request starts. The
|
||||
control plane retains authenticated remote-worker progress so Models can recover
|
||||
it after navigation or renderer reconnect. Jobs are keyed by both repository and
|
||||
target, preventing a local download of the same repository from appearing on a
|
||||
remote model card. Cancel sends only the worker's advertised opaque model ID,
|
||||
keeps polling while the worker drains the install, and completes after the worker
|
||||
returns a terminal progress event.
|
||||
|
||||
Engine Ready resolves TTS against the selected execution target. For a remote
|
||||
device it shows that worker's advertised engine/model, install and download
|
||||
readiness, execution backend, and heartbeat-confirmed residency; ASR,
|
||||
translation, dictation and diarisation remain attributed to the local device
|
||||
because those operations are not remotely routed. Clone, Design, Stories,
|
||||
Audiobook, profile preview and comparison actions use the same
|
||||
operation-scoped readiness, including cloning-capability filtering for Clone,
|
||||
so a model available only on the selected worker is usable without a duplicate
|
||||
local installation. If the selected worker is
|
||||
offline or an operation is local-only, readiness follows the backend's real
|
||||
local fallback instead of blocking on the remote choice. Batch validates the
|
||||
selected target before files or watched-folder items enter the queue, sends one
|
||||
coarse segment bundle per target language to a capable worker, and keeps ASR,
|
||||
translation, timeline assembly and muxing local. Dubbing and Batch preflight the
|
||||
selected worker and load local TTS lazily only if dispatch falls back, so a
|
||||
remote-only installation does not require duplicate weights on the control
|
||||
device.
|
||||
|
||||
Sharing exposes the backend only after an inline confirmation. It shows the active PIN and LAN addresses, generates QR links locally, supports a configurable share port, and controls the backend's Tailscale serve integration. These actions remain explicit and do not run during settings reads.
|
||||
|
||||
Remote backend configuration is owned by Electron main. Connection tests validate a VoiceStudio health response and exchange an optional server master key once for a scoped, expiring session. The renderer clears the key immediately; only the URL is persisted. Main injects the session into production and development HTTP proxy traffic, native watch-folder uploads, and path-bound dictation WebSocket tickets. Switching back to the local backend is always available.
|
||||
|
||||
Remote-worker routing was exercised against an Ubuntu 26.04 WSL worker with an RTX 4090. A real profile-backed TTS request returned a WAV with `X-OmniVoice-Routing: remote`; stopping the worker changed the same selected target to an explicit local fallback, and a second request returned `X-OmniVoice-Routing: local_fallback`. Restarting the worker restored remote readiness without re-enrollment. Dubbing and Batch also completed real multi-segment worker tasks: the Batch proof returned two exact indexed, non-silent mono WAVs at 24 kHz in one committed bundle. The selected target was returned to Local after verification.
|
||||
@@ -0,0 +1,296 @@
|
||||
# Electron dubbing workspace
|
||||
|
||||
The idle workspace includes an original/dubbed demo comparison with compact
|
||||
player controls. Sync playheads aligns positions without starting both videos.
|
||||
Sample transcript edits are retained per language while the demo is mounted;
|
||||
they do not regenerate the prerecorded audio. Edit on the dubbed card imports
|
||||
that sample video into the normal upload/transcription and editing workflow.
|
||||
|
||||
Open Dub from the cloning sidebar or command search. Upload or drop audio/video, or explicitly submit a video URL;
|
||||
preparation completes before transcription starts. The editor shows source text,
|
||||
editable translated text, and per-segment voice/timing controls. Translation uses
|
||||
the selected Settings > Models > Translation provider. Choose a target language,
|
||||
translate, review the text, then generate. Completed tracks can be previewed and
|
||||
exported through the native save dialog.
|
||||
|
||||
Segment rows scan as compact source/translation pairs: speaker, voice, fit state,
|
||||
selection and timestamp stay visible, while row actions reveal on hover or keyboard
|
||||
focus. Inset hairline separators preserve the reading rhythm; source text and metadata
|
||||
stay dimmed until the row is active, while the translation remains the visual lead.
|
||||
Clicking a translation turns only that row into a growing editor. Advanced
|
||||
voice and timing controls remain folded behind the speaker header. When a workspace
|
||||
also opens local controls, constrained or scaled windows automatically use the main
|
||||
navigation rail so the transcript keeps the available width; expanding it remains an
|
||||
explicit temporary override.
|
||||
When several targets are selected, progress tabs above the transcript switch the
|
||||
active language in one click while preserving every target for Translate All and
|
||||
Generate. Persisted provider error pages are discarded and restored to the source
|
||||
dialogue with a retryable error state.
|
||||
|
||||
Long projects virtualize transcript rows, so only the visible editors are mounted.
|
||||
Timeline waveform peaks and onsets are computed once by the backend and cached as a
|
||||
small JSON payload; Chromium never decodes the full separated-vocals WAV to draw the
|
||||
timeline. Video previews are written atomically with MP4 fast-start metadata and are
|
||||
immutable per generated-track revision. The renderer warms the current target in the
|
||||
background, reuses one Vidstack player while switching tracks, preserves the playhead
|
||||
and playing state across Original/Dub changes, and uses byte ranges on subsequent
|
||||
playback. Extension-derived native MIME hints select Vidstack's native provider
|
||||
immediately for common MP4, WebM, Ogg, MOV and MKV sources; URL imports use the
|
||||
backend's normalized MP4 type even when their display name has no extension. The media endpoints
|
||||
also answer metadata-only `HEAD` requests, including after backend restart, without
|
||||
reading the source body or starting a preview mux. Local NLLB batches scale with available accelerator memory while
|
||||
bounding batch multiplied by beam count.
|
||||
|
||||
The backend remains responsible for separation, ASR, speaker cloning, translation,
|
||||
TTS, fitting, mixing and export. Electron reuses Tauri's speaker binding and
|
||||
segment generation helpers. A stream close without a terminal event is a failure,
|
||||
not success. Cancellation aborts the HTTP stream and requests backend task/job
|
||||
cancellation. Edits, target language, track metadata and task IDs persist locally across reloads.
|
||||
Interrupted preparation/generation offers Resume, which reads the existing task
|
||||
and replays its stream; generation is never resubmitted just because the UI reloaded.
|
||||
Interrupted transcription offers an explicit Retry against the existing prepared
|
||||
media, without uploading or preparing the source again. ASR restarts from the
|
||||
beginning because its backend stream is request-scoped, not a replayable task. Batch language runs and advanced QC controls remain in `electron/PARITY.md`.
|
||||
|
||||
Verification: `node electron/tests/dub-smoke.mjs` against the development renderer.
|
||||
The test mocks backend jobs and never uploads or generates user media. Optionally
|
||||
set `VOICESTUDIO_TEST_VIDEO` to a local MP4 fixture to verify native video transport.
|
||||
These checks establish UI wiring, not a completed real model-backed dubbing run.
|
||||
`node electron/tests/native-translation-agent-smoke.mjs <agent>` separately launches the packaged
|
||||
app with isolated data and verifies that the detected CLI returns complete, ordered translations
|
||||
for real time-budgeted segments without a backend or source checkout. Codex, Claude Code and
|
||||
OpenCode pass on the current Windows host; Pi remains gated on a host where it is installed.
|
||||
`uv run python scripts/smoke_dub_url_captions.py` separately verifies the live
|
||||
public downloader without loading speech models or touching app data. The current
|
||||
smoke downloaded a browser-safe MP4 and one original-language caption track, then
|
||||
parsed 165 usable cues.
|
||||
|
||||
A disconnected preparation or generation stream retains its existing task for Resume or Cancel. Editing and new jobs remain disabled until that task finishes or cancellation is confirmed; reconnecting never creates a replacement generation. A task already absent from the backend counts as cancelled. If the backend cannot confirm cancellation, Change file explicitly abandons the unreachable local recovery record so the workspace cannot become permanently blocked.
|
||||
|
||||
The setup sidebar groups the source, target language and translation engine, timing,
|
||||
production overrides and export choices into stable sections. Advanced controls stay
|
||||
collapsed until requested. Before media is loaded, the main workspace presents the
|
||||
three actual steps—upload and transcribe, translate, generate—and hides inactive
|
||||
generation actions.
|
||||
|
||||
Import .srt replaces the current segment text and timings after source preparation.
|
||||
The backend retains voice references only where their timing overlaps the new cues.
|
||||
Malformed, overlapping, or duration-clamped cue counts remain visible in the sidebar.
|
||||
Failed imports preserve the current edits. Generated track buttons clear on successful
|
||||
replacement to avoid presenting older audio as the new subtitles' output.
|
||||
URL import runs only after clicking Ingest; it uses the backend's existing yt-dlp
|
||||
pipeline. Explicit cookies.txt selection is available under URL sign-in options; optional caption downloads are available.
|
||||
|
||||
Translation quality uses the existing backend Fast, Autofit and Cinematic modes.
|
||||
The choice persists in the working draft and saved project (`translateQuality`),
|
||||
including legacy project imports. New media preserves the user's quality choice.
|
||||
If the backend reports that no LLM is configured, the UI selects Fast and shows
|
||||
an inline explanation with a link to LLM settings. It does not silently claim
|
||||
that Cinematic or Autofit completed. Browser fixtures cover this fallback and
|
||||
reload persistence; real LLM-backed quality passes remain unverified.
|
||||
|
||||
**Translate with Agent** detects the installed Codex, Claude Code, OpenCode and Pi CLIs. Dubbing
|
||||
presents Agent and the active Google, Argos, NLLB or API engine as separate translator choices;
|
||||
Fast, Autofit and Cinematic remain quality choices for engine translation. The selected agent receives the complete ordered
|
||||
dialogue, glossary, dialect and per-segment speech budget as untrusted data, returns a strict
|
||||
id-preserving translation, and never receives repository or filesystem write access. Google,
|
||||
Argos, NLLB and configured API translators remain available through the ordinary Translate All
|
||||
action. Every translate entry point honors the chosen translator. Agent translations use the same reviewable segment fields and, during generation, reuse the
|
||||
bounded measured-speech loop to rewrite only timing misses before rerendering. Timing-fit provenance
|
||||
is stored per target language, so every translated track keeps that behavior after language switches
|
||||
and draft reloads. Cancelling Dubbing also terminates the local agent process tree.
|
||||
|
||||
Export options expand inside the existing sidebar. Users can select included video
|
||||
tracks and the default track, background mixing, burned subtitles, dual layout and
|
||||
karaoke (disabled with dual layout). Audio supports WAV or MP3 with bitrate choice;
|
||||
SRT/VTT/ASS sidecars and per-language stem/segment ZIPs use the existing backend.
|
||||
Each download is explicit and targets the selected language. Export errors retain
|
||||
all choices for retry. Native save filters match the encoded file format.
|
||||
Browser fixtures verify MP3/SRT downloads, query options and failed-export retry;
|
||||
unit tests cover video/package parameters. Real rendered exports, batch presets
|
||||
and native save dialogs remain unverified or incomplete.
|
||||
|
||||
Timing options now match Tauri: Concise, Smart Fit, Stretch Video and Lip sync.
|
||||
Voice matching offers per-line references or one consistent reference per speaker.
|
||||
These choices persist in working drafts and projects and are sent to generation;
|
||||
existing defaults remain Strict slot and Per line. Interrupted generation retains
|
||||
its submitted timing strategy. Completed output retains that strategy separately
|
||||
from current controls, so changing the next render's settings does not change the
|
||||
export capability of existing audio. Stretch Video output disables incompatible
|
||||
subtitle burn-in and directs users to sidecar exports instead. Browser fixtures
|
||||
verify persisted settings, request values and the export guard; actual timing and
|
||||
voice consistency across a real multi-speaker render remain unverified.
|
||||
|
||||
ASR retry browser coverage includes stream interruption, reload with zero automatic
|
||||
requests and successful explicit retry. Cancel releases recovery only after the
|
||||
backend acknowledges it. A real mid-ASR disconnect remains unverified.
|
||||
|
||||
Spoken-language and speaker-count hints are available in a collapsed sidebar
|
||||
section. Automatic detection remains the default. Explicit language hints reach
|
||||
both local uploads and URL ingest; speaker counts (1?20) reach transcription and
|
||||
persist through retries, drafts and projects. Browser checks exercise selection,
|
||||
reload and retry; unit tests inspect both file and URL request bodies. These are
|
||||
hints to the existing backend, not a guarantee of diarization accuracy.
|
||||
|
||||
Cookie exports are selected explicitly for one import, limited to 1 MB and sent
|
||||
only over HTTPS or the local desktop transport. Selection clears on submission;
|
||||
contents never enter the persisted dubbing session. Tauri and Electron share size
|
||||
and transport validation constants/helpers. Browser fixtures verify oversized
|
||||
rejection, request contents and absence from local storage; the Tauri cookie tests
|
||||
remain green. No real authenticated website download was performed.
|
||||
|
||||
URL Advanced options include downloading available captions through the existing
|
||||
yt-dlp pipeline. When usable cues are returned, Electron chooses the closest
|
||||
source-language track, normalizes its timing and opens it directly in the editor.
|
||||
Missing or malformed tracks fall back to the normal ASR path without another user
|
||||
decision. The downloader skips automatic translations. Real caption downloads
|
||||
are verified by the isolated public-URL smoke above. Authenticated sites still
|
||||
require a user-owned cookies export for native acceptance.
|
||||
|
||||
Production overrides expose steps, guidance, speed and global voice direction,
|
||||
matching the existing Tauri generation request. Defaults remain 16 / 2 / 1 with
|
||||
no direction. Values persist in drafts and projects; Reset clears only these
|
||||
overrides. Browser checks verify reload followed by the exact generation values,
|
||||
including zero guidance. Engine-specific audible effects remain unverified.
|
||||
|
||||
Saved Smart Fit override values now survive project import and draft recovery and
|
||||
reach generation only when Smart Fit is selected. Other timing strategies omit
|
||||
those parameters. Unit tests cover round-trip preservation and request routing;
|
||||
there is no new tuning panel (Tauri exposes these as stored preferences).
|
||||
|
||||
Real export verification: `tests/test_smart_fit_export.py` passes all 43 tests
|
||||
with the installed app-managed FFmpeg/ffprobe explicitly supplied. Its seven
|
||||
integration cases render synthetic video/audio through retiming and the backend
|
||||
export endpoint, then probe output durations and fitted subtitle bounds. This
|
||||
proves those backend export paths with real codecs; it does not prove a complete
|
||||
Electron upload ? ASR ? multi-speaker synthesis run. The broader targeted export
|
||||
suite passed 73 tests before the codec paths were supplied (seven skipped then).
|
||||
|
||||
Export format, bitrate, track selection, background mixing and subtitle choices now
|
||||
persist in the working draft and saved project. Legacy Tauri export preferences
|
||||
populate the panel rather than merely being retained as unused data. Browser
|
||||
checks verify format/bitrate/background/dual-layout recovery after reload; project
|
||||
tests verify legacy disabled tracks and background preferences.
|
||||
|
||||
Each segment now exposes volume (0?2), optional speed and a direction note within
|
||||
its existing voice disclosure. Empty speed follows global speed. Segment gain
|
||||
reaches generation separately from shared TTS fingerprint inputs, matching Tauri;
|
||||
zero is preserved for muted segments. Browser checks cover edits, reload and
|
||||
request values; backend audible mixing is not established by these UI fixtures.
|
||||
|
||||
Segment rows support insertion, deletion, cursor-aware splitting and merging with
|
||||
either neighbor. Merge/split uses Tauri's shared attribution bookkeeping, so speaker,
|
||||
voice, direction, gain and target language survive moving words across a boundary.
|
||||
Undo and redo retain the latest 50 edit states and reset when a new source, subtitle
|
||||
file, translation or project becomes the editing baseline. The toolbar and
|
||||
Cmd/Ctrl shortcuts expose the same operations. Unit and browser regressions cover
|
||||
the full insert/delete/merge/split/undo/redo sequence and merge-to-split attribution.
|
||||
|
||||
The editor includes a proportional timing lane above the segment rows. Segment
|
||||
blocks select their matching row, drag to move, expose edge handles for resizing,
|
||||
and support keyboard nudging or deletion. Timing changes use Tauri's shared clamp
|
||||
and speed-recalculation helpers. Adjacent overlaps are highlighted in the lane with
|
||||
an explicit warning; the media duration reported by preparation defines its scale.
|
||||
Browser coverage verifies timeline rendering, keyboard timing edits and overlap QC.
|
||||
|
||||
Segment checkboxes now enable one bulk edit surface without changing the active
|
||||
timeline segment. A single operation applies a saved voice, per-segment target
|
||||
language reset/override, or deletion to the selection and records one undo step.
|
||||
Select all and Clear keep large transcripts manageable while generation is idle.
|
||||
|
||||
Dubbing uses a wider responsive secondary sidebar (up to 32rem) and a wider transcript workspace. Once media is prepared, the upload form becomes a compact source row; source, target and casting remain a visible three-step sequence. Automatic cast matching is the default, so its overrides start folded. Voice chips wrap instead of requiring horizontal scrolling. Segment rows show the original text only when it differs from the editable text, retaining source comparison after translation without duplicating every unmodified transcript.
|
||||
|
||||
Below 40rem of workspace width, Dubbing stacks its controls above the editor with controls limited to 40% of the available height. Both regions retain independent scrolling.
|
||||
|
||||
Global speed/quality changes preserve explicit Dubbing production steps, including settings restored from projects. Unset steps continue to follow the backend's current preset.
|
||||
|
||||
NLLB resolves explicit FLORES language/script codes, unambiguous ISO-639-3 codes and common short aliases, including Traditional Chinese. Unsupported or script-ambiguous source, target or per-segment languages are rejected before model loading instead of silently translating to English.
|
||||
|
||||
### Speech integrity and timing
|
||||
|
||||
A failed, empty or unreadable speech segment stops generation before a new track
|
||||
replaces the previous output. Partial regeneration repairs missing or corrupt
|
||||
segment caches, including missing clips outside the requested changed-line list;
|
||||
it does not substitute silence and report completion. Auto speaker references
|
||||
exclude oversized clips when selecting a shared fallback, so short lines reuse a
|
||||
usable reference from their own speaker.
|
||||
|
||||
New segment caches retain the full generated speech in every timing mode. Strict
|
||||
Slot removes edge silence and fits the complete clip to its original start/end
|
||||
with pitch-preserving speed adjustment. Very long or short translations can still
|
||||
sound unnaturally fast or slow; shorten or expand the translation for natural
|
||||
pacing. Legacy clipped caches require regeneration once, because fitting cannot
|
||||
recover discarded words. Explicit legacy Trim and Off options retain their
|
||||
respective clipping and overlap behavior.
|
||||
|
||||
Concise and Smart Fit stop on unresolved overflow rather than publishing cut-off
|
||||
words. Shorten the translation, choose Strict Slot, or allow Stretch Video before
|
||||
retrying. Camera-cut segmentation uses nearby timed word boundaries when available
|
||||
and skips cuts inside speech that cannot be assigned safely. This improves phrase
|
||||
timing; it does not promise phoneme-level lip sync or correct inaccurate source
|
||||
transcripts automatically.
|
||||
|
||||
### Preserve sound outside dialogue
|
||||
|
||||
Background-preserving previews and audio/video exports keep the original stereo
|
||||
sound outside dialogue intervals, including audience reactions, music and ambience.
|
||||
Inside those intervals, they mix dubbed speech over the separated background, with
|
||||
10 ms transitions contained within the dialogue boundaries. Each generated language
|
||||
stores its source intervals; older tracks use their saved project intervals.
|
||||
Retimed modes also retime this background to follow the video. Ordinary Strict Slot
|
||||
and Concise modes keep the original background timeline.
|
||||
|
||||
Original media and a complete separated background are required. A missing or failed
|
||||
background mix stops export rather than silently exporting speech alone. Explicit
|
||||
speech-only export remains available. The preserved bed is cached locally and rebuilt
|
||||
when source files, dialogue intervals or the language's retiming plan change.
|
||||
Separation can still affect sounds overlapping dialogue; exact isolation from a
|
||||
single mixed recording is not guaranteed. Correct dialogue boundaries matter.
|
||||
|
||||
### Custom translation style
|
||||
|
||||
Select **Translate with agent**, then fill in **Translation style prompt** beside
|
||||
the translator controls. Describe tone, audience, formality, humor, idiom handling
|
||||
and how freely the dialogue should be adapted. For example: “Conversational Bengali
|
||||
for a young adult audience. Preserve jokes, adapt idioms naturally, keep names and
|
||||
numbers unchanged, and avoid stiff literal phrasing.”
|
||||
|
||||
The optional brief is saved with the project and restored after reopening. It applies
|
||||
to every selected target language and subsequent agent timing rewrites, using either
|
||||
a local CLI agent or the configured LLM. Meaning, timing, glossary and structured
|
||||
output requirements remain in effect. Clear the field to restore the default style;
|
||||
changing the brief does not retranslate existing segments until you run translation.
|
||||
The field accepts up to 5,000 characters and is locked while work is running.
|
||||
|
||||
### Translation activity footer
|
||||
|
||||
Translation and timing rewrites use the same footer area as Repair Agent. It opens
|
||||
with live CLI stdout/stderr in **Logs**; **Translations** shows original text beside
|
||||
validated translated output. Collapse **Details** to keep the status, language,
|
||||
elapsed time and Cancel action visible. Output from each language stays available
|
||||
until dismissed, the app reloads, or translation starts in another project. Logs are bounded to the
|
||||
latest 250,000 characters per run and are not saved into project files.
|
||||
|
||||
The counter tracks validated returned segments, not estimated progress. A CLI may
|
||||
stream logs while withholding its translation JSON until completion; API-backed
|
||||
translation currently returns one response, so its count updates when that response
|
||||
arrives. No fabricated percentage is shown. Errors remain visible, with Retry for
|
||||
failed translation work in the same project. API retries use failed segments when
|
||||
available; an incomplete CLI response requires retrying that language. Cancel stops
|
||||
the active translation and prevents late results from applying.
|
||||
|
||||
### Long timelines and duplicate ASR context
|
||||
|
||||
The timeline draws segments at their actual duration. Use Zoom in/out and Fit all
|
||||
above it to inspect short lines in long recordings; the zoomed view scrolls
|
||||
horizontally. Tiny overview bars cannot be accidentally dragged or resized. Click
|
||||
an overlap warning to zoom to the first affected segment. Nested overlaps are
|
||||
included in detection; simultaneous speakers are not automatically shifted apart.
|
||||
|
||||
After transcription, repeated chunk context is removed only when at least three
|
||||
matching words form a substantial prefix at matching timestamps for the same
|
||||
speaker. New words beyond that context remain. Stale segment boundaries are aligned
|
||||
to their own word timestamps only when those prove that the speech is disjoint.
|
||||
Existing translations/renders do not become correct merely by editing source text:
|
||||
changed lines must be translated and regenerated. Preserve the prior project when
|
||||
repairing an older transcript.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Electron voice gallery
|
||||
|
||||
Open VoiceStudio Gallery from the cloning sidebar or command search. Browse the
|
||||
existing local archetype catalogue by category, search it, and load more results.
|
||||
Age, gender, pitch, accent, language and whisper filters reuse the Tauri taxonomy.
|
||||
Their labels reuse the existing translated voice-design vocabulary, so no internal
|
||||
translation keys appear in the filter sidebar.
|
||||
Star voices to keep local favorites; Favorites searches all matching catalogue
|
||||
pages, including voices beyond the currently loaded cards.
|
||||
Preview playback starts only on click and uses the shared Vidstack player.
|
||||
Use voice asks the existing backend to materialize a reusable design profile,
|
||||
then opens Voice Design with that profile's attributes and seed. The current
|
||||
script stays intact. Navigating away cancels the frontend request and prevents
|
||||
a late response from redirecting the user; the backend may still finish saving.
|
||||
|
||||
The catalogue and profile generation remain owned by the existing backend.
|
||||
API wire types are shared with Tauri. No new required network service is added.
|
||||
Community voices, local/search imports, inline trimming, and portable persona
|
||||
bundles use the same backend contracts and keep network actions explicit.
|
||||
|
||||
Verification: node electron/tests/gallery-smoke.mjs exercises categories, search,
|
||||
pagination, real native playback of synthetic audio, saved-profile handoff, and
|
||||
navigation during saving using mocked API responses. The running backend also
|
||||
returned 1,126 catalogue entries and seven categories during development. The
|
||||
live Community acceptance forces a temporary catalogue outage, retries in place,
|
||||
then favorites and previews a real item before handing its attributes to Designer.
|
||||
Separate real smokes cover upload, trim, persona import/export, and profile
|
||||
materialization without retaining disposable profiles. The packaged Ubuntu app
|
||||
also completed profile creation, native `.ovsvoice` Save As, bundle inspection
|
||||
and cleanup against a reused installed runtime without downloading anything.
|
||||
@@ -0,0 +1,5 @@
|
||||
# Electron Launchpad
|
||||
|
||||
The root route is VoiceStudio's Launchpad. The app logo returns to it from the full or compact sidebar without changing any workspace route. It provides direct entry to cloning, design, dubbing, Stories, Audiobook, Gallery, Transcriptions and Tools, plus the unified Projects library.
|
||||
|
||||
Saved voices and recent takes provide short continuation lists. Choosing a voice restores the clone profile before navigation; choosing a take opens that take in the existing cloning workspace. The Launchpad uses the shared brand artwork, theme tokens, profile/history queries and responsive card grid, with no new network dependency.
|
||||
@@ -0,0 +1,26 @@
|
||||
# Electron LLM provider settings
|
||||
|
||||
Settings > Models > LLM includes the existing provider catalogue. Translation
|
||||
settings links to it. Configure an endpoint/model and, where needed, an API key
|
||||
or account ID. Save preserves a stored key when the key input is blank. Keys
|
||||
are sent only to the existing backend credential storage, never localStorage.
|
||||
Environment-pinned fields and activation remain read-only with an explanation.
|
||||
|
||||
Save & use for translation explicitly activates the provider. Saving or testing
|
||||
alone does not imply activation. Test and Fetch models first save the current
|
||||
form, stop if saving fails, and then call the backend probe. They never run
|
||||
on page load. Provider calls may use the network only when explicitly requested;
|
||||
local endpoints remain supported. Failed probes use classified localized messages.
|
||||
|
||||
The browser smoke `node electron/tests/llm-providers-smoke.mjs` mocks credentials
|
||||
and provider responses. It verifies blank-key preservation, environment pinning,
|
||||
save-before-test, no probe after failed save, model choice and activation. A live
|
||||
catalogue read returned 17 provider descriptors without key material. Actual
|
||||
external credentials and remote-provider calls are not verified by those mocks.
|
||||
Per-skill routing is available beneath providers. Each backend capability can be
|
||||
disabled or assigned a configured provider, with an option to follow the active
|
||||
provider. Existing unavailable overrides stay visible. Readiness comes from the
|
||||
backend; it is not proof of a successful network probe. Non-LLM translation-provider
|
||||
credentials for DeepL and Microsoft are available under Settings > Credentials and are
|
||||
written through the backend environment-setting endpoint. The skills browser smoke
|
||||
verifies routing and disable behavior against mocked API responses.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user