Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
362dca9ae1 | ||
|
|
bbac66ef76 | ||
|
|
3d74611be8 | ||
|
|
ee08f56e1b |
@@ -1,32 +0,0 @@
|
||||
# RTK - Rust Token Killer (Google Antigravity)
|
||||
|
||||
**Usage**: Token-optimized CLI proxy for shell commands.
|
||||
|
||||
## Rule
|
||||
|
||||
Always prefix shell commands with `rtk` to minimize token consumption.
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
rtk git status
|
||||
rtk cargo test
|
||||
rtk ls src/
|
||||
rtk grep "pattern" src/
|
||||
rtk find "*.rs" .
|
||||
rtk docker ps
|
||||
rtk gh pr list
|
||||
```
|
||||
|
||||
## Meta Commands
|
||||
|
||||
```bash
|
||||
rtk gain # Show token savings
|
||||
rtk gain --history # Command history with savings
|
||||
rtk discover # Find missed RTK opportunities
|
||||
rtk proxy <cmd> # Run raw (no filtering, for debugging)
|
||||
```
|
||||
|
||||
## Why
|
||||
|
||||
RTK filters and compresses command output before it reaches the LLM context, saving 60-90% tokens on common operations. Always use `rtk <cmd>` instead of raw commands.
|
||||
+3
-119
@@ -7,26 +7,12 @@
|
||||
language: "en-US"
|
||||
early_access: false
|
||||
|
||||
# The review voice: a panel of senior domain experts, not a linter.
|
||||
tone_instructions: >-
|
||||
Review as a panel of principal engineers: ML inference, audio DSP, desktop
|
||||
systems, product polish. Cite exact lines, name the failure mode, give the
|
||||
concrete fix. No filler praise; raise nits only when they change a decision.
|
||||
|
||||
reviews:
|
||||
# "chill" keeps the bot from blocking merges — it comments, it does not gate.
|
||||
# Hard gating lives in CI (security.yml) and the constitution's human bar.
|
||||
profile: chill
|
||||
request_changes_workflow: false
|
||||
high_level_summary: true
|
||||
# Every walkthrough gets a visual: mermaid sequence diagrams for the
|
||||
# mechanics, plus (via the summary instructions) an ASCII before/after
|
||||
# sketch when the PR touches UI — so each PR is reviewable at a glance.
|
||||
sequence_diagrams: true
|
||||
high_level_summary_instructions: >-
|
||||
If the PR changes UI (JSX/TSX/CSS/Tauri windows), include a compact ASCII
|
||||
before/after sketch of the affected layout or component. If it changes
|
||||
behavior, include a short mermaid flowchart of the new mechanism.
|
||||
review_status: true
|
||||
poem: false
|
||||
|
||||
@@ -53,8 +39,7 @@ reviews:
|
||||
- "!**/*.onnx"
|
||||
- "!tests/fixtures/**"
|
||||
|
||||
# One expert lens per subsystem — encode what a passionate senior in each
|
||||
# domain would actually check, beyond what linters and CI already gate.
|
||||
# Encode the project's hard constraints so the bot reviews against them.
|
||||
path_instructions:
|
||||
- path: "**/*.{py,rs,js,jsx,ts,tsx}"
|
||||
instructions: >-
|
||||
@@ -63,119 +48,18 @@ reviews:
|
||||
model download, or an explicitly opt-in endpoint. Flag any code that
|
||||
persists or logs values matching *TOKEN*/*KEY*/*SECRET* or absolute user
|
||||
home paths (/Users/<name>/, C:\\Users\\<name>\\).
|
||||
- path: "backend/services/**/*.py"
|
||||
instructions: >-
|
||||
Review as an ML-inference/audio engineer. Check: thread-safety of model
|
||||
and cache state across the GPU worker pool; device/dtype assumptions
|
||||
that break on one of CUDA/MPS/ROCm/CPU; VRAM lifecycle (load/unload,
|
||||
leaks on the error path); sample-rate, channel-count and tensor-shape
|
||||
assumptions at engine boundaries; blocking calls inside async paths;
|
||||
model download/cache behavior when offline. Engine code must stay
|
||||
backward-compatible with already-installed on-disk model state.
|
||||
- path: "backend/**/*.py"
|
||||
instructions: >-
|
||||
Default features must behave identically on macOS, Windows and Linux.
|
||||
Platform-specific implementation is allowed, but a divergent user-visible
|
||||
default is a P0 bug — flag it and suggest an opt-in (Settings/env/flag).
|
||||
Any DB schema change must go through an alembic migration with an upgrade
|
||||
path; flag direct schema edits. The backend serves loopback HTTP: treat
|
||||
every query/path/form param as hostile (path traversal, log injection,
|
||||
CSRF from a browser tab), and never route user-chosen filesystem
|
||||
destinations through HTTP — that authorization belongs in the Tauri
|
||||
process.
|
||||
- path: "frontend/src/**/*.{js,jsx,ts,tsx}"
|
||||
instructions: >-
|
||||
Review as a product-minded senior frontend engineer. Check: stale state
|
||||
and races (async results landing after unmount or after newer requests);
|
||||
every user-visible failure has an actionable, non-technical error
|
||||
message; loading/disabled states during long operations. Every new
|
||||
user-facing string must be an i18n t('...') key present in ALL 21
|
||||
frontend/src/i18n/locales/*.json files — flag hardcoded UI strings and
|
||||
keys missing from any locale.
|
||||
- path: "frontend/src-tauri/**/*.rs"
|
||||
instructions: >-
|
||||
Review as a desktop-systems engineer. Check: every #[tauri::command] is
|
||||
callable from the webview — validate inputs and scope filesystem/process
|
||||
access accordingly; window and webview lifecycle on all three OSes;
|
||||
child-process spawn/exit-code/stderr handling; no unwrap/expect on
|
||||
user-controlled input; platform cfg blocks keep user-visible defaults
|
||||
identical across macOS/Windows/Linux.
|
||||
- path: "tests/**/*.py"
|
||||
instructions: >-
|
||||
Review as a test-infrastructure engineer. Check: the test would fail
|
||||
before the fix and pass after (no tautologies); no sleeps as
|
||||
synchronization; no module-level imports of app modules that go stale
|
||||
under sys.modules pollution (resolve at run time); TestClient instances
|
||||
are function-scoped and not lifespan-bound unless the test needs it;
|
||||
new functional CJK is allowlisted in tests/test_no_hardcoded_cjk.py with
|
||||
a justification.
|
||||
path; flag direct schema edits. Engine code must stay backward-compatible
|
||||
with already-installed on-disk model state (no forced reinstall).
|
||||
- path: ".github/workflows/**"
|
||||
instructions: >-
|
||||
Pin actions to a major version tag at minimum. Flag any workflow that
|
||||
grants write permissions it does not need.
|
||||
|
||||
# Non-gating pre-merge audits of the project's hard rules (warning mode —
|
||||
# the human owner is the gate, these make the checklist visible per-PR).
|
||||
pre_merge_checks:
|
||||
title:
|
||||
mode: warning
|
||||
requirements: >-
|
||||
Conventional-commit style with scope, e.g. "fix(dub): …", and the issue
|
||||
reference in title or body. Never propose a version bump.
|
||||
issue_assessment:
|
||||
mode: warning
|
||||
custom_checks:
|
||||
- name: "Cross-platform default parity"
|
||||
mode: warning
|
||||
instructions: >-
|
||||
If the PR changes any default-mode (out-of-the-box) behavior, verify
|
||||
it behaves identically on macOS, Windows and Linux, or is moved behind
|
||||
an explicit opt-in (Settings toggle, env var, or CLI flag). A
|
||||
platform-divergent default is a P0 per CLAUDE.md — fail this check
|
||||
and say which platform diverges.
|
||||
- name: "i18n completeness (21 locales)"
|
||||
mode: warning
|
||||
instructions: >-
|
||||
For every new or changed t('...') key in frontend code, verify the
|
||||
key exists in all 21 files under frontend/src/i18n/locales/. List any
|
||||
locale files missing the key. Also flag hardcoded user-facing strings
|
||||
that bypass i18n entirely.
|
||||
- name: "Local-first guarantee"
|
||||
mode: warning
|
||||
instructions: >-
|
||||
Verify the PR adds no required cloud calls, accounts, API keys or
|
||||
telemetry. Outbound traffic is only allowed to GitHub Issues (opt-in
|
||||
bug reporting) and HuggingFace model downloads. The app must remain
|
||||
fully functional offline and with reporting disabled.
|
||||
- name: "Backward compatibility"
|
||||
mode: warning
|
||||
instructions: >-
|
||||
Verify existing omnivoice_data/ (voices, projects, settings) and
|
||||
already-installed engine model state keep working without manual
|
||||
migration. Any DB schema change must ship an alembic migration with
|
||||
an upgrade path. Flag anything that would force users to reinstall an
|
||||
engine or re-download model weights.
|
||||
|
||||
finishing_touches:
|
||||
docstrings:
|
||||
enabled: true
|
||||
unit_tests:
|
||||
enabled: true
|
||||
|
||||
# Feed the bot the project constitution and docs, and let it accumulate
|
||||
# learnings from review conversations ("@coderabbitai always/never …").
|
||||
knowledge_base:
|
||||
code_guidelines:
|
||||
enabled: true
|
||||
filePatterns:
|
||||
- "CLAUDE.md"
|
||||
- "docs/**/*.md"
|
||||
learnings:
|
||||
scope: auto
|
||||
issues:
|
||||
scope: auto
|
||||
pull_requests:
|
||||
scope: auto
|
||||
|
||||
chat:
|
||||
auto_reply: true
|
||||
|
||||
+3
-4
@@ -1,7 +1,6 @@
|
||||
# These are supported funding model platforms
|
||||
# GitHub Sponsors isn't set up for this account — fund via Ko-fi or PayPal.
|
||||
|
||||
ko_fi: debpalash
|
||||
custom: ["https://paypal.me/palashCoder"]
|
||||
# github: [debpalash] # not available
|
||||
github: [debpalash]
|
||||
# ko_fi: omnivoice
|
||||
# open_collective: omnivoice-studio
|
||||
# custom: ["https://omnivoice.palash.dev/sponsor"]
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
name: 🐛 Bug Report
|
||||
about: Report a bug to help us improve OmniVoice Studio
|
||||
title: "[Bug] "
|
||||
labels: ["bug", "triage"]
|
||||
assignees: []
|
||||
---
|
||||
|
||||
## Describe the bug
|
||||
|
||||
A clear and concise description of what the bug is.
|
||||
|
||||
## To reproduce
|
||||
|
||||
Steps to reproduce the behavior:
|
||||
1. Go to '...'
|
||||
2. Click on '...'
|
||||
3. See error
|
||||
|
||||
## Expected behavior
|
||||
|
||||
What you expected to happen.
|
||||
|
||||
## Screenshots / Logs
|
||||
|
||||
If applicable, add screenshots or paste relevant logs from **Settings → Logs**.
|
||||
|
||||
> **Tip:** **Settings → About → "Save diagnostic bundle"** produces a zip
|
||||
> (self-check report, recent errors, scrubbed log tails) you can drag onto
|
||||
> this issue — it answers most environment questions below automatically.
|
||||
> Headless installs: `python backend/main.py --diagnose` prints the same
|
||||
> self-check (`--deep` also test-loads the active engine).
|
||||
|
||||
## Environment
|
||||
|
||||
- **OS:** [e.g. macOS 15.2, Windows 11, Ubuntu 24.04]
|
||||
- **Install method:** [Desktop app / Docker / From source]
|
||||
- **Version:** [e.g. v0.2.7 — check Settings → About]
|
||||
- **GPU:** [e.g. NVIDIA RTX 4090 / Apple M3 Pro / CPU only]
|
||||
- **RAM:** [e.g. 16 GB]
|
||||
- **Active TTS engine:** [e.g. omnivoice — check Settings → Engines]
|
||||
|
||||
## Additional context
|
||||
|
||||
Add any other context about the problem here.
|
||||
@@ -1,109 +0,0 @@
|
||||
name: 🐛 Bug report
|
||||
description: Something works incorrectly or crashes (not a first-run/install problem — use the install template for those).
|
||||
title: "[Bug] "
|
||||
labels: ["bug", "triage"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Thanks for helping improve OmniVoice Studio! 🎙️
|
||||
|
||||
**Fastest path to a fix:** **Settings → About → "Save diagnostic bundle"** makes a
|
||||
zip (self-check + recent errors + scrubbed log tails) — drag it onto this issue and
|
||||
most of the environment questions below are answered automatically.
|
||||
Headless: `python backend/main.py --diagnose` (add `--deep` to test-load the engine).
|
||||
- type: checkboxes
|
||||
id: preflight
|
||||
attributes:
|
||||
label: Before filing
|
||||
options:
|
||||
- label: I searched [existing issues](https://github.com/debpalash/OmniVoice-Studio/issues?q=is%3Aissue) and this isn't a duplicate.
|
||||
required: true
|
||||
- label: I'm on the latest release (or `main`) — older builds may already be fixed.
|
||||
required: false
|
||||
- type: textarea
|
||||
id: what-happened
|
||||
attributes:
|
||||
label: What happened?
|
||||
description: A clear description of the bug, including the exact error text / toast if any.
|
||||
placeholder: "Voice cloning failed with '…' after I clicked Generate."
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: repro
|
||||
attributes:
|
||||
label: Steps to reproduce
|
||||
value: |
|
||||
1.
|
||||
2.
|
||||
3.
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: expected
|
||||
attributes:
|
||||
label: What did you expect instead?
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: os
|
||||
attributes:
|
||||
label: Operating system
|
||||
options:
|
||||
- macOS (Apple Silicon)
|
||||
- macOS (Intel)
|
||||
- Windows (x64)
|
||||
- Linux (AppImage)
|
||||
- Linux (.deb)
|
||||
- Linux (other / from source)
|
||||
- Docker
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: install
|
||||
attributes:
|
||||
label: How did you install it?
|
||||
options:
|
||||
- Desktop app (installer / AppImage)
|
||||
- Docker image
|
||||
- From source (uv sync)
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: version
|
||||
attributes:
|
||||
label: Version
|
||||
description: Settings → About (e.g. v0.3.5), or the Docker tag / git SHA.
|
||||
placeholder: "v0.3.5"
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: device
|
||||
attributes:
|
||||
label: Compute device
|
||||
options:
|
||||
- NVIDIA GPU (CUDA)
|
||||
- AMD GPU (ROCm)
|
||||
- Apple Silicon (MPS)
|
||||
- Intel GPU (XPU)
|
||||
- CPU only
|
||||
- Not sure
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: engine
|
||||
attributes:
|
||||
label: Active TTS/ASR engine
|
||||
description: Settings → Engines (e.g. omnivoice, cosyvoice, indextts2, whisperx).
|
||||
placeholder: "omnivoice"
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Logs / diagnostic bundle
|
||||
description: Drag the diagnostic bundle here, or paste relevant lines from **Settings → Logs**. Secrets are scrubbed automatically.
|
||||
render: text
|
||||
- type: textarea
|
||||
id: extra
|
||||
attributes:
|
||||
label: Anything else?
|
||||
description: Screenshots, the input that triggered it, RAM/VRAM, etc.
|
||||
@@ -1,11 +0,0 @@
|
||||
blank_issues_enabled: false
|
||||
contact_links:
|
||||
- name: 💬 Discord — questions & quick help
|
||||
url: https://discord.gg/bzQavDfVV9
|
||||
about: Usage questions, setup help, and chat. Faster than an issue for "how do I…".
|
||||
- name: 🗣️ GitHub Discussions
|
||||
url: https://github.com/debpalash/OmniVoice-Studio/discussions
|
||||
about: Ideas, show-and-tell, and open-ended Q&A that isn't a bug or a specific feature ask.
|
||||
- name: 🔒 Security vulnerability
|
||||
url: https://github.com/debpalash/OmniVoice-Studio/security/policy
|
||||
about: Please report security issues privately — do NOT open a public issue.
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
name: ✨ Feature Request
|
||||
about: Suggest an idea for OmniVoice Studio
|
||||
title: "[Feature] "
|
||||
labels: ["enhancement"]
|
||||
assignees: []
|
||||
---
|
||||
|
||||
## Is your feature request related to a problem?
|
||||
|
||||
A clear description of what the problem is. Ex. "I'm always frustrated when..."
|
||||
|
||||
## Describe the solution you'd like
|
||||
|
||||
A clear description of what you want to happen.
|
||||
|
||||
## Describe alternatives you've considered
|
||||
|
||||
Any alternative solutions or features you've considered.
|
||||
|
||||
## Additional context
|
||||
|
||||
Add any other context, mockups, or screenshots about the feature request here.
|
||||
@@ -1,50 +0,0 @@
|
||||
name: ✨ Feature request
|
||||
description: Suggest an improvement or a new capability.
|
||||
title: "[Feature] "
|
||||
labels: ["enhancement"]
|
||||
body:
|
||||
- type: checkboxes
|
||||
id: preflight
|
||||
attributes:
|
||||
label: Before filing
|
||||
options:
|
||||
- label: I searched [existing issues](https://github.com/debpalash/OmniVoice-Studio/issues?q=is%3Aissue) and [discussions](https://github.com/debpalash/OmniVoice-Studio/discussions) for this idea.
|
||||
required: true
|
||||
- type: textarea
|
||||
id: problem
|
||||
attributes:
|
||||
label: What problem does this solve?
|
||||
description: The use case / friction this addresses ("When I … I can't …").
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: solution
|
||||
attributes:
|
||||
label: Proposed solution
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: alternatives
|
||||
attributes:
|
||||
label: Alternatives considered
|
||||
- type: dropdown
|
||||
id: area
|
||||
attributes:
|
||||
label: Area
|
||||
options:
|
||||
- Voice cloning
|
||||
- Voice design
|
||||
- Video dubbing
|
||||
- Real-time dictation
|
||||
- Audiobook / Stories (long-form)
|
||||
- TTS/ASR engines
|
||||
- Install / setup / packaging
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
> OmniVoice is **local-first** — features must work fully offline with no accounts,
|
||||
API keys, or cloud calls, and behave identically on macOS/Windows/Linux. Proposals
|
||||
that fit those constraints are easiest to land.
|
||||
@@ -1,86 +0,0 @@
|
||||
name: 🧩 Install / first-run problem
|
||||
description: The app won't install, set up, download models, or reach a first working output.
|
||||
title: "[Install] "
|
||||
labels: ["install", "triage"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
A first-run that *just works* is the whole point — sorry it didn't. Let's fix it.
|
||||
If the app launched far enough to open Settings, **Settings → About → "Save diagnostic
|
||||
bundle"** captures most of this; otherwise the fields below are enough.
|
||||
- type: dropdown
|
||||
id: stage
|
||||
attributes:
|
||||
label: Where did it fail?
|
||||
options:
|
||||
- App won't launch / blank or broken window
|
||||
- Python / uv environment bootstrap
|
||||
- Model download (HuggingFace)
|
||||
- Engine install (CosyVoice / IndexTTS / MLX / etc.)
|
||||
- First synthesis / dub never completes
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: error
|
||||
attributes:
|
||||
label: The error
|
||||
description: The exact message, traceback, or what you see on screen.
|
||||
render: text
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: os
|
||||
attributes:
|
||||
label: Operating system
|
||||
options:
|
||||
- macOS (Apple Silicon)
|
||||
- macOS (Intel)
|
||||
- Windows (x64)
|
||||
- Linux (AppImage)
|
||||
- Linux (.deb)
|
||||
- Linux (other / from source)
|
||||
- Docker
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: install
|
||||
attributes:
|
||||
label: How are you installing it?
|
||||
options:
|
||||
- Desktop app (installer / AppImage)
|
||||
- Docker image
|
||||
- From source (uv sync)
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: version
|
||||
attributes:
|
||||
label: Version
|
||||
placeholder: "v0.3.5 (or installer build / git SHA)"
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: network
|
||||
attributes:
|
||||
label: Network conditions (model/dependency downloads)
|
||||
description: Restricted networks are a known source of bootstrap failures (mirror fallback).
|
||||
options:
|
||||
- Normal / unrestricted
|
||||
- Behind a corporate proxy / firewall
|
||||
- Region with restricted access (e.g. China, Russia)
|
||||
- Offline / air-gapped
|
||||
- Not sure
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Logs / diagnostic bundle
|
||||
description: Drag a diagnostic bundle, or paste the install/bootstrap log. Headless self-check — `python backend/main.py --diagnose`.
|
||||
render: text
|
||||
- type: textarea
|
||||
id: tried
|
||||
attributes:
|
||||
label: What have you already tried?
|
||||
@@ -18,7 +18,7 @@
|
||||
- [ ] 📝 Documentation
|
||||
- [ ] 🧪 Tests
|
||||
- [ ] 🔧 CI / Build
|
||||
- [ ] 🚀 Release prep
|
||||
- [ ] 🚀 Release prep (RC or final)
|
||||
|
||||
## Testing
|
||||
|
||||
@@ -33,12 +33,17 @@
|
||||
- [ ] No local machine paths, logs, or personal env details in this PR
|
||||
- [ ] Version files are in sync (if version bump): `pyproject.toml`, `package.json`, `tauri.conf.json`, `Cargo.toml`
|
||||
- [ ] If this PR changes runtime behavior, the regression fixture at `tests/fixtures/omnivoice_data/` still loads green on the `smoke-matrix` CI job (macOS + Windows + Linux)
|
||||
- [ ] If this is part of a release, I've read the "Release cadence" section below and confirmed this PR targets the right RC
|
||||
|
||||
## Release cadence
|
||||
## Release cadence (read once per RC)
|
||||
|
||||
OmniVoice ships **continuous-to-main** — no release candidates, no soak windows.
|
||||
Every merged PR is immediately part of the rolling preview (`main`, Docker
|
||||
`:latest`, the desktop Preview channel). Versioned releases are tagged from
|
||||
`main` when it's ready; `main` then bumps to the next patch automatically.
|
||||
Users who want stability pin a release tag / Docker `:stable` / the desktop
|
||||
Stable channel.
|
||||
OmniVoice ships every minor on a **two-RC cadence**:
|
||||
- `vX.Y.0-rc1` — cut from `main` once all GATE-* requirements pass; clean-VM exercise on 4 OSes (per `REL-01`)
|
||||
- 48-hour soak (no new commits to release branch except fix-forward)
|
||||
- `vX.Y.0` — promotion if rc1 is clean
|
||||
|
||||
If your PR touches install / bootstrap / CI, it MUST land before rc1 cut, not between rc1 and the promotion. During a soak, any merge needs explicit OK from the release captain.
|
||||
|
||||
## Screenshots
|
||||
|
||||
<!-- If applicable, add screenshots or recordings. -->
|
||||
|
||||
@@ -92,10 +92,7 @@ jobs:
|
||||
|
||||
- name: Install frontend deps
|
||||
working-directory: frontend
|
||||
# --frozen-lockfile so a frontend/package.json change that forgets to
|
||||
# regenerate the root bun.lock fails HERE (fast) instead of only in the
|
||||
# Docker build (deploy/Dockerfile), which is what reddened main on #485.
|
||||
run: bun install --frozen-lockfile
|
||||
run: bun install
|
||||
|
||||
# checkJs is true in tsconfig for IDE feedback, but 947 pre-existing
|
||||
# JS errors remain. Override to false in CI so only .ts files block.
|
||||
@@ -105,19 +102,6 @@ jobs:
|
||||
working-directory: frontend
|
||||
run: bun run typecheck:ci
|
||||
|
||||
# oxlint gate — fast Rust linter, blocks on errors so lint debt can't
|
||||
# re-accumulate (warnings, incl. the react-compiler advisories in
|
||||
# `lint:hooks`, are non-blocking). See frontend/.oxlintrc.json.
|
||||
- name: Frontend lint (oxlint)
|
||||
working-directory: frontend
|
||||
run: bun run lint
|
||||
|
||||
# oxfmt format gate — JS/TS/JSX only (CSS/JSON/Tauri excluded; see
|
||||
# frontend/.oxfmtrc.json). `bun run format` fixes locally.
|
||||
- name: Frontend format check (oxfmt)
|
||||
working-directory: frontend
|
||||
run: bun run format:check
|
||||
|
||||
- name: Run Vitest (frontend)
|
||||
working-directory: frontend
|
||||
run: bunx vitest run
|
||||
@@ -190,10 +174,7 @@ jobs:
|
||||
|
||||
- name: Install frontend deps
|
||||
working-directory: frontend
|
||||
# --frozen-lockfile so a frontend/package.json change that forgets to
|
||||
# regenerate the root bun.lock fails HERE (fast) instead of only in the
|
||||
# Docker build (deploy/Dockerfile), which is what reddened main on #485.
|
||||
run: bun install --frozen-lockfile
|
||||
run: bun install
|
||||
|
||||
# tauri-build's setup hook reads tauri.conf.json's `frontendDist`
|
||||
# ("../dist"), which only exists after a frontend build. Without this,
|
||||
|
||||
@@ -5,21 +5,14 @@
|
||||
# - push to main branch → :main, :sha- (rolling "edge" build)
|
||||
# - workflow_dispatch → :sha- only (ad-hoc test build)
|
||||
#
|
||||
# Tag ↔ image mapping (versioning hard rule, owner-set 2026-06-11:
|
||||
# :latest IS the preview channel; stable users pin :stable or a version tag)
|
||||
# :latest — rolling preview: latest commit on main (always last release + 1 dev)
|
||||
# :main — alias of the same rolling main build (kept for back-compat)
|
||||
# :stable — most recent versioned release (set on every v* tag push)
|
||||
# :0.3.6 — exact version from the git tag
|
||||
# Tag ↔ image mapping
|
||||
# :latest — always the most recent versioned release (set on every v* tag push)
|
||||
# :0.3.0 — exact version from the git tag
|
||||
# :0.3 — major.minor floating tag (updated on every patch within the minor)
|
||||
# :main — latest commit on main; may be ahead of the last tagged release
|
||||
# :sha-xxxx — specific commit SHA; produced by workflow_dispatch
|
||||
#
|
||||
# Images land at: ghcr.io/debpalash/omnivoice-studio AND docker.io/palashdeb/omnivoice-studio
|
||||
# (Docker Hub push gated on the DOCKERHUB_USERNAME/DOCKERHUB_TOKEN secrets;
|
||||
# if unset the build still pushes to GHCR.)
|
||||
#
|
||||
# On main pushes the Docker Hub repository overview is also synced from
|
||||
# deploy/dockerhub-overview.md (source of truth for the hub.docker.com page).
|
||||
# Images land at: ghcr.io/debpalash/omnivoice-studio
|
||||
#
|
||||
# NOTE: the Docker image is the headless web-server build of OmniVoice (FastAPI
|
||||
# backend + pre-built React frontend served over HTTP). The Tauri desktop
|
||||
@@ -41,7 +34,6 @@ permissions:
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_NAME: ${{ github.repository }}
|
||||
DOCKERHUB_IMAGE: palashdeb/omnivoice-studio
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
|
||||
|
||||
jobs:
|
||||
@@ -64,43 +56,28 @@ jobs:
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
# Docker Hub login — only when the secret is present, so forks / runs
|
||||
# without the credential still publish to GHCR.
|
||||
- name: Check Docker Hub credentials
|
||||
id: dockerhub
|
||||
run: echo "enabled=${{ secrets.DOCKERHUB_TOKEN != '' }}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Log in to Docker Hub
|
||||
if: steps.dockerhub.outputs.enabled == 'true'
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
# Tag strategy (`:sha-<short>` is emitted on every trigger):
|
||||
# v0.3.6 tag push → :0.3.6, :0.3, :stable, :sha-
|
||||
# main branch push → :latest, :main, :sha-
|
||||
# v0.3.0 tag push → :0.3.0, :0.3, :latest, :sha-
|
||||
# main branch push → :main, :sha-
|
||||
# workflow_dispatch → :sha- only
|
||||
#
|
||||
# All mutable-tag rules stay gated on `github.event_name == 'push'` so a
|
||||
# manual workflow_dispatch can only ever produce a throwaway `:sha-` tag
|
||||
# (the stale-:latest fix from #249/#251). :stable excludes prerelease
|
||||
# tags (those contain a `-`) so a prerelease can't clobber it.
|
||||
# Fix for stale :latest (issues #249, #251):
|
||||
# The previous rule used `enable={{is_default_branch}}`, which evaluates
|
||||
# to false on tag pushes (detached HEAD) — so :latest was never updated
|
||||
# when a release tag was pushed. The version / :latest / :main rules are
|
||||
# gated on `github.event_name == 'push'` so a manual workflow_dispatch can
|
||||
# only ever produce a throwaway `:sha-` tag (never republish a mutable
|
||||
# tag), and :latest additionally excludes prerelease tags (those contain a
|
||||
# `-`, e.g. v1.0.0-rc.1) so a prerelease can't clobber :latest.
|
||||
- name: Extract metadata (tags, labels)
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
# Same tag set applied to both registries. The Docker Hub line is
|
||||
# blank when the secret is unset, so metadata-action emits GHCR-only
|
||||
# tags in that case.
|
||||
images: |
|
||||
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
${{ steps.dockerhub.outputs.enabled == 'true' && env.DOCKERHUB_IMAGE || '' }}
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
type=semver,pattern={{version}},enable=${{ github.event_name == 'push' }}
|
||||
type=semver,pattern={{major}}.{{minor}},enable=${{ github.event_name == 'push' }}
|
||||
type=raw,value=stable,enable=${{ github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-') }}
|
||||
type=raw,value=latest,enable=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
type=raw,value=latest,enable=${{ github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-') }}
|
||||
type=raw,value=main,enable=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
type=sha,prefix=sha-,format=short
|
||||
|
||||
@@ -114,25 +91,3 @@ jobs:
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
# Sync the Docker Hub repository overview from deploy/dockerhub-overview.md.
|
||||
# Only on main pushes (the overview tracks the rolling preview) and only
|
||||
# when Docker Hub creds are present, mirroring the push gating above.
|
||||
#
|
||||
# continue-on-error: the overview text is cosmetic, and the description
|
||||
# PATCH 403s unless DOCKERHUB_TOKEN carries description-edit scope (many
|
||||
# fine-grained Docker Hub tokens that can push still can't edit the
|
||||
# description). The image build+push is what matters — a creds-scope
|
||||
# mismatch on this cosmetic step must not fail the whole Docker run. To
|
||||
# actually sync the overview, use a token with read/write (incl.
|
||||
# description) scope, or the account password.
|
||||
- name: Update Docker Hub description
|
||||
if: steps.dockerhub.outputs.enabled == 'true' && github.event_name == 'push' && github.ref == 'refs/heads/main'
|
||||
continue-on-error: true
|
||||
uses: peter-evans/dockerhub-description@v4
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
repository: ${{ env.DOCKERHUB_IMAGE }}
|
||||
short-description: "Local ElevenLabs alternative: voice cloning, design & video dubbing in 646 languages. No API keys."
|
||||
readme-filepath: ./deploy/dockerhub-overview.md
|
||||
|
||||
@@ -1,97 +0,0 @@
|
||||
# Docs drift — daily inventory-vs-docs check with a single rolling issue.
|
||||
#
|
||||
# docs/features.yaml is the canonical inventory; scripts/check-docs-drift.py
|
||||
# diffs it against README.md, docs/, and the engine registries. On drift the
|
||||
# job updates (or creates) ONE issue labeled `docs-drift` in place — no issue
|
||||
# spam — and closes it automatically when the check is clean again.
|
||||
#
|
||||
# Companion to the PR-gating validate-install-docs.py step in ci.yml.
|
||||
# Spec: docs/competitive-analysis.md Spec 9a / parity program Wave 0.1.
|
||||
# Rolling-issue pattern adapted from Patter (MIT).
|
||||
|
||||
name: docs-drift
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Daily 03:30 UTC — after most merges, before EU morning triage.
|
||||
- cron: "30 3 * * *"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
drift:
|
||||
runs-on: ubuntu-22.04
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
- name: Install checker deps
|
||||
run: pip install "pyyaml>=6"
|
||||
|
||||
- name: Check inventory vs README/docs/registries
|
||||
id: drift
|
||||
continue-on-error: true
|
||||
run: python scripts/check-docs-drift.py --output drift-report.md
|
||||
|
||||
- name: Update rolling docs-drift issue
|
||||
uses: actions/github-script@v7
|
||||
env:
|
||||
DRIFT_OUTCOME: ${{ steps.drift.outcome }}
|
||||
with:
|
||||
script: |
|
||||
const fs = require('fs');
|
||||
const drifted = process.env.DRIFT_OUTCOME === 'failure';
|
||||
const { owner, repo } = context.repo;
|
||||
const label = 'docs-drift';
|
||||
|
||||
const open = await github.rest.issues.listForRepo({
|
||||
owner, repo, state: 'open', labels: label, per_page: 5,
|
||||
});
|
||||
|
||||
if (drifted) {
|
||||
let body = '';
|
||||
try {
|
||||
body = fs.readFileSync('drift-report.md', 'utf8');
|
||||
} catch {
|
||||
body = '# Docs drift report\n\nThe checker failed before writing a report — see the workflow run logs.';
|
||||
}
|
||||
body += `\n\n---\n_Last checked by [run ${context.runId}](https://github.com/${owner}/${repo}/actions/runs/${context.runId})._\n`;
|
||||
|
||||
if (open.data.length > 0) {
|
||||
await github.rest.issues.update({
|
||||
owner, repo, issue_number: open.data[0].number, body,
|
||||
});
|
||||
core.info(`Updated rolling issue #${open.data[0].number}`);
|
||||
} else {
|
||||
const created = await github.rest.issues.create({
|
||||
owner, repo,
|
||||
title: 'docs-drift: feature inventory vs docs mismatch',
|
||||
body,
|
||||
labels: [label, 'documentation'],
|
||||
});
|
||||
core.info(`Created rolling issue #${created.data.number}`);
|
||||
}
|
||||
} else {
|
||||
for (const issue of open.data) {
|
||||
await github.rest.issues.createComment({
|
||||
owner, repo, issue_number: issue.number,
|
||||
body: 'Drift resolved — nightly check is clean again. Closing automatically.',
|
||||
});
|
||||
await github.rest.issues.update({
|
||||
owner, repo, issue_number: issue.number, state: 'closed',
|
||||
});
|
||||
core.info(`Closed rolling issue #${issue.number}`);
|
||||
}
|
||||
}
|
||||
|
||||
- name: Surface drift as a failed run
|
||||
if: steps.drift.outcome == 'failure'
|
||||
run: |
|
||||
echo "Docs drift detected — see the rolling docs-drift issue."
|
||||
exit 1
|
||||
@@ -1,55 +0,0 @@
|
||||
# LLM-judge evals — semantic quality suites, NEVER a gate.
|
||||
#
|
||||
# Hard rule (parity program Wave 0.3 / competitive-analysis Spec 9b): LLM
|
||||
# judges never gate CI. This workflow is scheduled + manual only, the eval
|
||||
# step is continue-on-error, and the JSON report is the deliverable
|
||||
# (uploaded as an artifact). Deterministic probe judges in ci.yml remain
|
||||
# the only gates.
|
||||
#
|
||||
# On the hosted runner there is no local LLM endpoint, so the run usually
|
||||
# reports "skipped — no LLM backend configured"; the workflow exists so the
|
||||
# suites run anywhere a TRANSLATE_BASE_URL secret/endpoint is provided
|
||||
# (e.g. a self-hosted runner with Ollama).
|
||||
|
||||
name: evals
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Weekly, Sundays 04:00 UTC.
|
||||
- cron: "0 4 * * 0"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
evals:
|
||||
runs-on: ubuntu-22.04
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
- uses: astral-sh/setup-uv@v3
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
|
||||
- name: Install deps
|
||||
run: uv sync
|
||||
|
||||
- name: Run eval suites (non-gating)
|
||||
continue-on-error: true
|
||||
env:
|
||||
TRANSLATE_BASE_URL: ${{ secrets.EVALS_LLM_BASE_URL }}
|
||||
TRANSLATE_API_KEY: ${{ secrets.EVALS_LLM_API_KEY }}
|
||||
run: uv run python tests/evals/run_evals.py --output eval-report.json
|
||||
|
||||
- name: Upload report artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: eval-report
|
||||
path: eval-report.json
|
||||
if-no-files-found: warn
|
||||
+25
-203
@@ -4,16 +4,12 @@
|
||||
# - push of a tag matching `v*` (e.g. `v0.2.0`) → full STABLE release,
|
||||
# publishes artifacts + signed updater manifest (`latest.json`) to the
|
||||
# tag's GH Release. This is the default Stable updater channel.
|
||||
# - schedule (nightly, 07:00 UTC) → rolling `preview` PRERELEASE built from
|
||||
# `main` with its own signed `latest.json` at releases/download/preview/.
|
||||
# Feeds the opt-in Preview updater channel (Settings → About → Update
|
||||
# channel). The `preview-gate` job skips the matrix on nights when `main`
|
||||
# didn't move, so an idle day costs only a ~30s gate job — keeping Preview
|
||||
# ≤24h behind `main` at a predictable ~1-matrix/day cost. The stable
|
||||
# `latest` release is untouched.
|
||||
# - workflow_dispatch (publish_preview=true) → the same preview build on
|
||||
# demand from the selected branch (e.g. to preview a feature branch, or to
|
||||
# refresh immediately without waiting for the nightly).
|
||||
# - workflow_dispatch (publish_preview=true) → builds the selected branch and
|
||||
# publishes a rolling `preview` PRERELEASE with its own signed
|
||||
# `latest.json` at releases/download/preview/. This feeds the opt-in
|
||||
# Preview updater channel (Settings → About → Update channel). The stable
|
||||
# `latest` release is untouched. Run this manually whenever you want to cut
|
||||
# a preview from `main`.
|
||||
# - workflow_dispatch (publish_preview=false) → on-demand build (prior
|
||||
# behavior; draft release named after the branch).
|
||||
#
|
||||
@@ -32,10 +28,6 @@ name: Desktop Release
|
||||
on:
|
||||
push:
|
||||
tags: ['v*']
|
||||
schedule:
|
||||
# 07:00 UTC daily — rolling `preview` prerelease from `main`. The
|
||||
# preview-gate job no-ops the matrix when main hasn't moved in a day.
|
||||
- cron: '0 7 * * *'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
draft:
|
||||
@@ -129,41 +121,8 @@ jobs:
|
||||
working-directory: frontend
|
||||
run: node --experimental-strip-types --no-warnings --test ../tests/frontend/*.test.mjs
|
||||
|
||||
# Decide preview-vs-stable, and for nightly runs whether `main` actually
|
||||
# moved in the last day. Outputs gate the expensive matrix (`build`) and the
|
||||
# `preview-notes` job, so a no-commit night costs only this ~30s job.
|
||||
preview-gate:
|
||||
name: Preview gate
|
||||
runs-on: ubuntu-22.04
|
||||
outputs:
|
||||
is_preview: ${{ steps.decide.outputs.is_preview }}
|
||||
proceed: ${{ steps.decide.outputs.proceed }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 50
|
||||
- id: decide
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
event="${{ github.event_name }}"
|
||||
if [ "$event" = "schedule" ] || { [ "$event" = "workflow_dispatch" ] && [ "${{ inputs.publish_preview }}" = "true" ]; }; then
|
||||
echo "is_preview=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "is_preview=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
# Nightly: skip the matrix when main hasn't moved in the last day.
|
||||
if [ "$event" = "schedule" ] && [ -z "$(git log --since='25 hours ago' --oneline)" ]; then
|
||||
echo "No new commits on main in the last day — skipping nightly preview."
|
||||
echo "proceed=false" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "proceed=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
build:
|
||||
needs: [test, preview-gate]
|
||||
# Nightly runs with no new commits on main skip the 4-platform matrix.
|
||||
if: needs.preview-gate.outputs.proceed == 'true'
|
||||
needs: test
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -174,28 +133,9 @@ jobs:
|
||||
rust_target: aarch64-apple-darwin
|
||||
bundles: "app,dmg,updater"
|
||||
|
||||
# macOS Intel (#279): reinstated. The earlier "Rosetta 2 runs the
|
||||
# ARM build" rationale for dropping it was backwards — Rosetta only
|
||||
# translates x86_64→arm64, so Intel Macs (supported through macOS
|
||||
# Sequoia) simply cannot run the aarch64 bundle and had NO
|
||||
# installable artifact. Runner: `macos-15-intel`, GitHub's
|
||||
# designated migration target after macos-13 retired (Dec 2025);
|
||||
# it's a standard (public-repo-free) image supported through
|
||||
# August 2027 — the last x86_64 image Actions will offer. Building
|
||||
# natively (not cross-compiling from the arm64 leg) keeps the
|
||||
# per-TRIPLE uv/ffmpeg sidecar fetches, the DMG installer smoke,
|
||||
# and the ad-hoc signing verification (scripts/
|
||||
# verify-macos-signing.sh, PR #290) all exercising the real
|
||||
# x86_64 artifact on real Intel hardware. The macos-13 queue
|
||||
# backlog that motivated the original drop is contained by
|
||||
# fail-fast:false — a slow Intel leg can delay the release run but
|
||||
# can't fail the other targets.
|
||||
- os: macos-15-intel
|
||||
arch: x86_64-apple-darwin
|
||||
label: "macOS Intel"
|
||||
rust_target: x86_64-apple-darwin
|
||||
bundles: "app,dmg,updater"
|
||||
|
||||
# macOS Intel dropped: Apple shipped the last Intel Mac in 2023 and
|
||||
# Rosetta 2 runs the ARM build natively. macos-13 runner backlog
|
||||
# was also blocking every release tag for ~10 min.
|
||||
# Windows: force MSI bundling via --bundles. NSIS fails at makensis
|
||||
# because our PyInstaller payload approaches its ~2 GB stub limit.
|
||||
- os: windows-2022
|
||||
@@ -351,10 +291,7 @@ jobs:
|
||||
case "$TRIPLE" in
|
||||
aarch64-apple-darwin|x86_64-apple-darwin)
|
||||
# evermeet.cx ships each binary as a separate .zip containing
|
||||
# a single x86_64 Mach-O executable — natively correct on the
|
||||
# Intel leg, and runs via Rosetta 2 on the arm64 leg. Both
|
||||
# darwin TRIPLEs therefore bundle the same payload; only the
|
||||
# sidecar filename suffix differs.
|
||||
# a single x86_64 Mach-O executable (runs via Rosetta on arm64).
|
||||
for TOOL in ffmpeg ffprobe; do
|
||||
if [ "$TOOL" = "ffmpeg" ]; then
|
||||
URL="https://evermeet.cx/ffmpeg/getrelease/zip"
|
||||
@@ -461,21 +398,17 @@ jobs:
|
||||
# always reported the static 0.3.0 never looked "newer", so no update was
|
||||
# ever delivered). Ephemeral, CI-only — never committed. Tauri reads the
|
||||
# bundle + updater version from tauri.conf.json, so rewriting it here
|
||||
# stamps the artifacts + latest.json. Under the versioning hard rule
|
||||
# (owner-set 2026-06-11) main is always last-release + 1, so BASE-N is a
|
||||
# prerelease of the NEXT version and semver-sorts ABOVE the last stable
|
||||
# (0.3.6-N > 0.3.5) — preview users naturally upgrade past stable, and
|
||||
# the Windows MSI ProductVersion (which strips the prerelease → 0.3.6)
|
||||
# is also correctly above the last stable.
|
||||
# stamps the artifacts + latest.json. `0.3.0-preview.N` is a prerelease of
|
||||
# the current target, so previews converge to stable when 0.3.0 ships
|
||||
# (0.3.0 > 0.3.0-preview.N). NOTE: the Windows MSI ProductVersion strips
|
||||
# the prerelease (→ 0.3.0), a wrinkle to verify for win preview→preview
|
||||
# upgrades; mac/linux replace the bundle wholesale and are unaffected.
|
||||
- name: Stamp preview version
|
||||
if: needs.preview-gate.outputs.is_preview == 'true'
|
||||
if: github.event_name == 'workflow_dispatch' && inputs.publish_preview
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# package.json is the single source of truth; tauri.conf.json reads its
|
||||
# version from it ("version": "../package.json"), so stamping
|
||||
# package.json restamps the whole bundle.
|
||||
CONF=frontend/package.json
|
||||
CONF=frontend/src-tauri/tauri.conf.json
|
||||
BASE=$(jq -r .version "$CONF")
|
||||
# MSI/WiX requires the semver pre-release identifier to be numeric-only
|
||||
# (and <= 65535). "preview.N" hard-fails the Windows bundler, so the
|
||||
@@ -498,12 +431,6 @@ jobs:
|
||||
# only on the opt-in stable path, leaving them ABSENT (not "") on
|
||||
# preview/unsigned paths so Tauri's bundler skips cert import. A static
|
||||
# env: here would always set them to "" and break the mac build.
|
||||
# Unsigned paths still get a VALID ad-hoc seal from tauri.conf.json
|
||||
# (bundle.macOS.signingIdentity = "-"), so a downloaded build shows the
|
||||
# GUI-bypassable "unidentified developer" prompt (right-click → Open /
|
||||
# Settings → "Open Anyway") instead of the un-bypassable "damaged"
|
||||
# error. On the signed path APPLE_SIGNING_IDENTITY (env) overrides the
|
||||
# "-" default; once notarized, Gatekeeper accepts it with no prompt.
|
||||
# GH runners disable FUSE, so linuxdeploy's AppImage can't mount
|
||||
# itself at bundle time. This env tells linuxdeploy to extract-and-run
|
||||
# instead, which works without FUSE.
|
||||
@@ -515,11 +442,11 @@ jobs:
|
||||
# rolling `preview` prerelease for the updater's Preview channel.
|
||||
# Every other invocation — crucially the `v*` tag-push stable release
|
||||
# — evaluates these expressions to exactly their prior values.
|
||||
tagName: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'preview' || github.ref_name }}
|
||||
releaseName: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'OmniVoice Studio (Preview)' || format('OmniVoice Studio {0}', github.ref_name) }}
|
||||
tagName: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'preview' || github.ref_name }}
|
||||
releaseName: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'OmniVoice Studio (Preview)' || format('OmniVoice Studio {0}', github.ref_name) }}
|
||||
releaseBody: ${{ steps.changelog.outputs.body }}
|
||||
releaseDraft: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'false' || (inputs.draft || 'true') }}
|
||||
prerelease: ${{ needs.preview-gate.outputs.is_preview == 'true' }}
|
||||
releaseDraft: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'false' || (inputs.draft || 'true') }}
|
||||
prerelease: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) || false }}
|
||||
updaterJsonPreferNsis: false
|
||||
includeUpdaterJson: true
|
||||
|
||||
@@ -554,35 +481,6 @@ jobs:
|
||||
echo "OK — bundle has shell + uv + backend resources"
|
||||
hdiutil detach "$MOUNT" || true
|
||||
|
||||
# ── Signing / Gatekeeper / notarization verification ──────────────
|
||||
# Runs codesign --verify, spctl (Gatekeeper), nested-binary, and
|
||||
# stapler checks against the built .app (see docs/macos-signing-verification.md).
|
||||
# STRICT (--require-signed) only on the opt-in signed stable path — same
|
||||
# condition as "Configure Apple signing" above — so a failed or missing
|
||||
# signature/notarization FAILS the job and STOPS the release instead of
|
||||
# publishing an unsigned artifact. On every other (unsigned dev/preview)
|
||||
# path it runs report-only and never breaks the build.
|
||||
- name: Verify macOS signing
|
||||
if: runner.os == 'macOS'
|
||||
shell: bash
|
||||
env:
|
||||
STRICT: ${{ (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && vars.MACOS_SIGNING_ENABLED == 'true') && '1' || '0' }}
|
||||
APPLE_ID: ${{ secrets.APPLE_ID }}
|
||||
APPLE_PASSWORD: ${{ secrets.APPLE_PASSWORD }}
|
||||
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
|
||||
run: |
|
||||
set -uo pipefail
|
||||
APP=$(find "frontend/src-tauri/target/${{ matrix.rust_target }}/release/bundle/macos" -maxdepth 1 -name '*.app' | head -1)
|
||||
[ -n "$APP" ] || { echo "FAIL — no .app found to verify"; exit 1; }
|
||||
MODE=""
|
||||
if [ "$STRICT" = "1" ]; then
|
||||
MODE="--require-signed"
|
||||
echo "Signed stable release → STRICT verification (release stops on failure)."
|
||||
else
|
||||
echo "Unsigned dev/preview path → report-only verification."
|
||||
fi
|
||||
bash scripts/verify-macos-signing.sh "$APP" $MODE
|
||||
|
||||
- name: Installer smoke (Windows)
|
||||
if: runner.os == 'Windows'
|
||||
timeout-minutes: 5
|
||||
@@ -705,8 +603,8 @@ jobs:
|
||||
# preview-only — stable `v*` releases keep their CHANGELOG section + the
|
||||
# appended checksums.
|
||||
preview-notes:
|
||||
needs: [build, preview-gate]
|
||||
if: needs.preview-gate.outputs.is_preview == 'true'
|
||||
needs: build
|
||||
if: github.event_name == 'workflow_dispatch' && inputs.publish_preview
|
||||
runs-on: ubuntu-22.04
|
||||
permissions:
|
||||
contents: write
|
||||
@@ -737,81 +635,5 @@ jobs:
|
||||
echo ""
|
||||
echo "$CONTRIB"
|
||||
} > /tmp/preview-notes.md
|
||||
# --prerelease re-asserts the flag every run: a non-prerelease
|
||||
# `preview` release is eligible to become GitHub's "Latest", which is
|
||||
# the exact URL the Stable updater channel reads — so it must never
|
||||
# flip off.
|
||||
gh release edit preview --repo "$REPO" --prerelease --notes-file /tmp/preview-notes.md
|
||||
gh release edit preview --repo "$REPO" --notes-file /tmp/preview-notes.md
|
||||
echo "Applied auto-generated release notes + contributors to the preview release."
|
||||
|
||||
- name: Verify preview updater manifest (prerelease + platform parity)
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
REPO: ${{ github.repository }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# The preview release must stay a prerelease (or it can hijack the
|
||||
# Stable channel's releases/latest endpoint), and its updater manifest
|
||||
# must cover every platform stable does (else those users — e.g. Intel
|
||||
# Mac — silently get no preview updates).
|
||||
is_pre=$(gh release view preview --repo "$REPO" --json isPrerelease -q .isPrerelease)
|
||||
test "$is_pre" = "true" || { echo "::error::preview release is not a prerelease"; exit 1; }
|
||||
curl -fsSL "https://github.com/$REPO/releases/download/preview/latest.json" -o /tmp/preview-latest.json
|
||||
curl -fsSL "https://github.com/$REPO/releases/latest/download/latest.json" -o /tmp/stable-latest.json
|
||||
python3 - <<'PY'
|
||||
import json, re
|
||||
prev = json.load(open("/tmp/preview-latest.json"))
|
||||
stab = json.load(open("/tmp/stable-latest.json"))
|
||||
v = prev.get("version", "")
|
||||
assert re.fullmatch(r"\d+\.\d+\.\d+-\d+", v), f"preview version not X.Y.Z-N: {v!r}"
|
||||
pk, sk = set(prev.get("platforms", {})), set(stab.get("platforms", {}))
|
||||
missing = sk - pk
|
||||
assert not missing, f"preview manifest missing platforms vs stable: {sorted(missing)}"
|
||||
print(f"preview manifest OK: {v} platforms={sorted(pk)}")
|
||||
PY
|
||||
|
||||
# ── Post-release version bump (versioning hard rule, owner-set 2026-06-11) ──
|
||||
# main is always last-release + 1 patch. The moment a stable v* tag is
|
||||
# released, bump the three version sources on main to the next patch so every
|
||||
# PR and preview build identifies as the next version. Pushes directly to
|
||||
# main with the workflow token (a metadata-only commit; CI runs on PRs).
|
||||
version-bump:
|
||||
if: github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-')
|
||||
runs-on: ubuntu-22.04
|
||||
permissions:
|
||||
contents: write
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
ref: main
|
||||
fetch-depth: 0
|
||||
- name: Bump main to released version + 1 patch
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
RELEASED="${GITHUB_REF_NAME#v}"
|
||||
IFS=. read -r MAJ MIN PAT <<< "$RELEASED"
|
||||
NEXT="$MAJ.$MIN.$((PAT + 1))"
|
||||
# frontend/package.json is the SINGLE SOURCE OF TRUTH: vite injects
|
||||
# __APP_VERSION__ from it, and tauri.conf.json reads its bundle version
|
||||
# from it ("version": "../package.json"). Read CURRENT from it.
|
||||
CURRENT=$(jq -r .version frontend/package.json)
|
||||
if [ "$(printf '%s\n' "$NEXT" "$CURRENT" | sort -V | tail -1)" = "$CURRENT" ] && [ "$NEXT" != "$CURRENT" ]; then
|
||||
echo "main is already at $CURRENT (>= $NEXT) — nothing to bump"; exit 0
|
||||
fi
|
||||
# Bump the canonical (package.json), set absolutely so any prior drift
|
||||
# self-heals. tauri.conf.json needs no edit — it derives from this.
|
||||
tmp=$(mktemp)
|
||||
jq --arg v "$NEXT" '.version = $v' frontend/package.json > "$tmp"
|
||||
mv "$tmp" frontend/package.json
|
||||
# The remaining files are CI-guarded mirrors (cargo/uv require a
|
||||
# literal; the version.py literal is the frozen-backend last resort) —
|
||||
# bump them in lockstep with the canonical.
|
||||
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" frontend/src-tauri/Cargo.toml
|
||||
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" pyproject.toml
|
||||
sed -i "0,/_FALLBACK_VERSION = \"$CURRENT\"/s//_FALLBACK_VERSION = \"$NEXT\"/" backend/core/version.py
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
git add frontend/package.json frontend/src-tauri/Cargo.toml pyproject.toml backend/core/version.py
|
||||
git commit -m "chore(version): main -> $NEXT after $GITHUB_REF_NAME release"
|
||||
git push origin main
|
||||
|
||||
@@ -32,13 +32,9 @@ env:
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# PR branches: a new push cancels the superseded scan (no wasted runners).
|
||||
# main: every commit keeps its own group, so nothing is cancelled — a merge
|
||||
# train used to leave a permanent red ✗ ("cancelled") on every intermediate
|
||||
# commit in the history view even though nothing failed.
|
||||
concurrency:
|
||||
group: security-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || 'branch' }}
|
||||
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
|
||||
group: security-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
# ── Secret scanning (gating) ─────────────────────────────────────────────
|
||||
|
||||
@@ -1,17 +1,10 @@
|
||||
# SPIKE-02: Adopt `ModelsLab/omnivoice-singing` as singing variant of the existing engine
|
||||
|
||||
**Status:** ⚠️ **SUPERSEDED (2026-06-14)** by [`specs/006-dubbing-singing-mode/`](../../specs/006-dubbing-singing-mode/spec.md)
|
||||
**Date:** 2026-05-18 (superseded 2026-06-14)
|
||||
**Status:** Proposed (research-supported) — awaiting Phase 2 SubprocessBackend merge
|
||||
**Date:** 2026-05-18
|
||||
**Decision-makers:** [maintainer]
|
||||
**Related:** ROADMAP Phase 4; REQUIREMENTS SING-01..05; `.planning/phases/04-adaptive-specialty-engines-spike-first/04-RESEARCH.md`
|
||||
|
||||
> **Superseded:** This chose `ModelsLab/omnivoice-singing` for singing, but that
|
||||
> model has **no melody (F0/MIDI) conditioning** — it sings its own melody and
|
||||
> cannot follow the *source song* a dub must preserve. SoulX-Singer (arXiv
|
||||
> 2602.07803, published after this decision) provides F0/MIDI conditioning and is
|
||||
> selected in plan-06. This ADR stays valid only if reframed as an
|
||||
> expressive-TTS styling toggle, not melody-matched dubbing.
|
||||
|
||||
## Context
|
||||
|
||||
`ModelsLab/omnivoice-singing` (HuggingFace, 1,053 downloads/month, verified 2026-05-18) is a finetune of `k2-fsa/OmniVoice` — same Apache-2.0 license, same Qwen3-0.6B backbone, same Higgs Audio v2 codec at 24 kHz mono, same `omnivoice` PyPI library (0.1.5, 2026-04-28) already shipping in OmniVoice Studio v0.2.7. Trained on additional singing + emotion-tagged data and activated by a `[singing]` text control tag at generation time.
|
||||
|
||||
@@ -1,396 +0,0 @@
|
||||
---
|
||||
phase: 260613-fdl
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- pyproject.toml
|
||||
- backend/api/routers/setup/download.py
|
||||
- backend/api/routers/setup/models.py
|
||||
- backend/utils/hf_progress.py
|
||||
- backend/utils/download_aggregator.py # NEW
|
||||
- backend/services/segmented_download.py # NEW
|
||||
- backend/api/routers/system.py
|
||||
- frontend/src/pages/Settings.jsx
|
||||
- frontend/src/api/setup.ts
|
||||
- docs/downloading-models.md # NEW (docs-sync rule)
|
||||
- tests/backend/setup/test_download_preflight.py # NEW
|
||||
- tests/backend/services/test_segmented_download.py # NEW
|
||||
autonomous: true
|
||||
requirements:
|
||||
# ── Wave 0 — Spike / gate ───────────────────────────────────────────────────
|
||||
- FDL-00 # Classify all catalog repos Xet-backed vs legacy-LFS; the result sizes Wave 3
|
||||
# ── Wave 1 — Maximize + guarantee the Xet fast path (default, no new deps) ───
|
||||
- FDL-01 # Explicitly pin huggingface_hub>=1.7 + hf-xet in pyproject (today transitive/unpinned)
|
||||
- FDL-02 # Drive snapshot_download with explicit max_workers + tqdm_class + endpoint (not implicit monkeypatch)
|
||||
- FDL-03 # /system/info reports fast_download {xet_enabled, xet_version, high_performance}; logged at startup
|
||||
- FDL-04 # Opt-in HF_XET_HIGH_PERFORMANCE + HDD sequential-write toggles via prefs (env wins)
|
||||
# ── Wave 2 — Accurate downloaded/remaining + speed (the user-visible win) ────
|
||||
- FDL-05 # dry_run preflight -> emit install_plan {total_bytes, cached_bytes, to_download_bytes, n_files, n_cached}
|
||||
- FDL-06 # Backend aggregate tracker -> single 'aggregate' event {bytes_done, total_bytes, rate, eta, files_done/total}
|
||||
- FDL-07 # Frontend overall bar: speed + downloaded/remaining + ETA from aggregate; per-file detail collapsible; cached-skip shown
|
||||
# ── Wave 3 — Opt-in IDM-style accelerator for legacy-LFS repos ───────────────
|
||||
- FDL-08 # Custom httpx segmented downloader: parallel Range GETs, resume, auth-safe redirect, etag/sha verify, cancel (default OFF)
|
||||
- FDL-09 # Dispatch: accelerator ON + repo is LFS (not Xet) -> segmented path; else xet. Same aggregate progress + weight validation
|
||||
# ── Wave 4 — Opt-in mirror path + docs ───────────────────────────────────────
|
||||
- FDL-10 # Opt-in HF_ENDPOINT mirror setting (prefs); documented as classic-LFS fallback (no Xet); pairs with FDL-08
|
||||
- FDL-11 # Cancel-in-flight endpoint + cooldown interplay (composes with MM2-06 bounded cooldowns)
|
||||
- FDL-12 # docs/downloading-models.md (speed, fast-download status, HDD/high-perf toggles, mirror/restricted-network) + README pointer
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "Xet is the default download backend and is provably engaged: /system/info reports fast_download.xet_enabled=true with the hf_xet version, and a Xet-backed repo downloads via parallel chunk range-gets (not single-stream LFS)."
|
||||
- "Before any bytes flow, the UI shows an accurate denominator: total bytes to download, bytes already cached (skipped), and file count — sourced from snapshot_download(dry_run=True), not guessed from the first tqdm bar."
|
||||
- "During a download the UI shows ONE overall progress bar with instantaneous speed (sampled over a window, not a single file's rate), bytes downloaded / bytes remaining, and ETA — accurate even while Xet fetches many chunks/files in parallel."
|
||||
- "hf_transfer is NOT used or enabled anywhere (deprecated, breaks progress); the fast path is Xet only."
|
||||
- "The custom segmented downloader is OPT-IN (default off), only engages for non-Xet/legacy-LFS repos, never forwards the HF Authorization header to the redirected CDN host, verifies the downloaded file against its expected size/etag before marking complete, resumes a partial .part file, and can be cancelled mid-flight."
|
||||
- "Default download behavior is identical on macOS, Windows, Linux (Xet path, pure-Python). Every accelerator/mirror/high-perf knob is behind an explicit opt-in (Settings toggle or env var) per the cross-platform-parity strict rule — no bundled per-OS binary, no platform-divergent default."
|
||||
- "No new on-disk model-state format; existing HF cache layout and already-installed models are untouched; the segmented downloader writes into the same HF cache blob/snapshot structure (or hands off to it) so a model it fetches is indistinguishable from one snapshot_download fetched."
|
||||
- "uv run pytest tests/backend/setup/test_download_preflight.py tests/backend/services/test_segmented_download.py passes; existing download/install tests stay green."
|
||||
- "pyproject pins huggingface_hub>=1.7 and hf-xet explicitly; uv.lock resolves with single versions (uv tree shows no duplicate huggingface_hub)."
|
||||
artifacts:
|
||||
- path: "backend/utils/download_aggregator.py"
|
||||
provides: "Per-repo byte aggregator: sums bytes across parallel files/chunks, samples rate over a window, emits one 'aggregate' event"
|
||||
contains: "class DownloadAggregator AND def snapshot"
|
||||
- path: "backend/services/segmented_download.py"
|
||||
provides: "Opt-in multi-connection Range downloader for legacy-LFS repos (auth-safe, resume, verify, cancel)"
|
||||
contains: "async def segmented_download AND Range"
|
||||
- path: "backend/api/routers/setup/download.py"
|
||||
provides: "Driven snapshot_download (max_workers+tqdm_class+endpoint), dry_run preflight, dispatch to segmented path, cancel endpoint"
|
||||
contains: "dry_run AND tqdm_class"
|
||||
- path: "docs/downloading-models.md"
|
||||
provides: "User docs for download speed, fast-download status, HDD/high-perf toggles, mirror/restricted-network"
|
||||
contains: "Xet"
|
||||
key_links:
|
||||
- from: "install_model (download.py:122)"
|
||||
to: "snapshot_download(dry_run=True) preflight"
|
||||
via: "compute total/cached/remaining before the real download; emit 'install_plan'"
|
||||
pattern: "dry_run\\s*=\\s*True"
|
||||
- from: "snapshot_download / segmented_download byte updates"
|
||||
to: "DownloadAggregator -> single 'aggregate' SSE event"
|
||||
via: "tqdm_class forwards bytes into the aggregator; segmented path calls aggregator.add() directly"
|
||||
pattern: "aggregate"
|
||||
- from: "dispatch in install_model"
|
||||
to: "segmented_download vs snapshot_download"
|
||||
via: "prefs accelerator toggle AND repo-is-LFS classification (FDL-00 helper)"
|
||||
pattern: "segmented_download"
|
||||
- from: "system_info (system.py:245)"
|
||||
to: "fast_download status block"
|
||||
via: "probe hf_xet import + version + HF_XET_HIGH_PERFORMANCE"
|
||||
pattern: "fast_download"
|
||||
|
||||
---
|
||||
|
||||
<objective>
|
||||
Make model downloads as fast as possible AND show accurate speed / downloaded / remaining / ETA.
|
||||
|
||||
**Framing (validated by research — see 260613-fdl-RESEARCH below):** HuggingFace's **hf-xet** backend ALREADY implements the "IDM/uGet technique" — content-defined chunking, parallel byte-range fetches with adaptive concurrency, dedup, and automatic resume — and does it auth-safely. It ships by default in modern `huggingface_hub` and `hf_xet` is already installed here (huggingface_hub 1.7.2). HF closed the multi-connection-downloader feature request as "solved by Xet." So we do NOT build a custom segmented downloader as the default path; that would be redundant and would violate the cross-platform-parity rule.
|
||||
|
||||
What's actually missing:
|
||||
1. **We don't drive Xet well.** `install_model` calls `snapshot_download(**dl_kwargs)` with no `max_workers`, no `tqdm_class`, no `dry_run`, and no explicit dependency pin — progress rides on a global tqdm monkeypatch.
|
||||
2. **No pre-flight total**, so "downloaded/remaining" has no denominator until files appear, and aggregate speed is summed frontend-side from per-file events (inaccurate under parallel fetch).
|
||||
3. **Legacy non-Xet (LFS) repos get zero intra-file parallelism** — this is the one place a real IDM-style multi-connection fetch still helps, so we add it as an OPT-IN accelerator.
|
||||
|
||||
Five waves, in order (each independently shippable, continuous-to-main per v0.3.0 cadence):
|
||||
- **Wave 0 — Spike/gate (FDL-00):** classify every catalog repo Xet vs LFS. Sizes Wave 3's value; if ~all repos are Xet-backed, Wave 3 is low-priority polish.
|
||||
- **Wave 1 — Maximize + guarantee Xet (FDL-01..04):** pin deps, drive snapshot_download explicitly, surface fast-download status, opt-in high-perf/HDD knobs. No new deps, all platforms.
|
||||
- **Wave 2 — Accurate progress (FDL-05..07):** dry_run preflight + backend aggregate tracker + overall UI bar (speed/remaining/ETA). The biggest user-visible win.
|
||||
- **Wave 3 — Opt-in segmented accelerator (FDL-08..09):** custom httpx Range downloader for LFS repos. Default OFF, opt-in toggle.
|
||||
- **Wave 4 — Mirror path + docs (FDL-10..12):** opt-in HF_ENDPOINT, cancel endpoint, docs-sync.
|
||||
|
||||
Out of scope / explicitly rejected (call out, do NOT do):
|
||||
- **hf_transfer / HF_HUB_ENABLE_HF_TRANSFER** — deprecated, breaks progress callbacks. Never enable.
|
||||
- **Bundling aria2c** — per-OS GPLv2 binary + parity burden; the custom httpx path covers the same need without a binary.
|
||||
- **Making the segmented downloader the default** — redundant vs Xet, violates parity rule. Always opt-in.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/STATE.md
|
||||
@./CLAUDE.md
|
||||
@.planning/quick/260613-fdl-fast-model-downloads/260613-fdl-RESEARCH.md
|
||||
|
||||
# Files under edit (read before editing)
|
||||
@backend/api/routers/setup/download.py
|
||||
@backend/api/routers/setup/models.py
|
||||
@backend/utils/hf_progress.py
|
||||
@backend/api/routers/system.py
|
||||
@frontend/src/pages/Settings.jsx
|
||||
|
||||
# Reference only — patterns, do NOT modify
|
||||
@backend/core/prefs.py
|
||||
@frontend/src/api/setup.ts
|
||||
@frontend/src/api/hooks.ts
|
||||
|
||||
<interfaces>
|
||||
<!-- Verified during planning against the live env (huggingface_hub 1.7.2, hf_xet installed). -->
|
||||
|
||||
huggingface_hub 1.7.2 snapshot_download params (confirmed via inspect):
|
||||
repo_id, repo_type, revision, cache_dir, local_dir, library_name, library_version,
|
||||
user_agent, etag_timeout, force_download, token, local_files_only,
|
||||
allow_patterns, ignore_patterns, max_workers, tqdm_class, headers, endpoint, dry_run
|
||||
- dry_run=True -> returns per-file info incl. size + cached/not-cached (use for FDL-05 preflight).
|
||||
- tqdm_class=<cls> -> drives the AGGREGATE bar; Xet feeds bytes into it (this is the xet-aware progress hook).
|
||||
- max_workers -> parallel FILES (default 8); orthogonal to Xet intra-file chunk parallelism.
|
||||
- endpoint -> per-call HF endpoint override (FDL-10 mirror, instead of process-wide HF_ENDPOINT).
|
||||
|
||||
backend/utils/hf_progress.py (existing):
|
||||
- Monkeypatches huggingface_hub.utils.tqdm.tqdm -> TrackedTqdm (install() at startup).
|
||||
- register_listener/unregister_listener; emit(event); current_repo_id contextvar stamps events.
|
||||
- TrackedTqdm.update()/display() emit per-file {filename, downloaded, total, pct, rate, phase} throttled ~0.3s.
|
||||
- GAP: per-file only, no aggregate, no preflight total. Wave 2 adds the aggregator on top (keep TrackedTqdm; feed it).
|
||||
|
||||
backend/api/routers/setup/download.py (existing):
|
||||
- install_model (line 122): snapshot_download(**dl_kwargs) inside asyncio.to_thread; 5-retry backoff; heartbeat;
|
||||
_validate_snapshot_has_weights (line 55); _install_cooldowns (line 27, see MM2-06 for bounding).
|
||||
- SSE feed: GET /setup/download-stream (line 80) forwards hf_progress events.
|
||||
|
||||
backend/core/prefs.py:
|
||||
- resolve(key, *, env=None, default=None) (line 75) — env wins, then store, then default. Use for all new toggles.
|
||||
|
||||
Xet env knobs (research): HF_XET_HIGH_PERFORMANCE=1 (opt-in max throughput; needs RAM/bandwidth),
|
||||
HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1 (HDD), HF_XET_NUM_CONCURRENT_RANGE_GETS (default 16),
|
||||
HF_XET_DATA_PROGRESS_UPDATE_INTERVAL (200ms). hf_xet is 64-bit only.
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<!-- ════════════ WAVE 0 — SPIKE / GATE ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 0 (FDL-00): Classify catalog repos Xet vs LFS</name>
|
||||
<files>.planning/quick/260613-fdl-fast-model-downloads/260613-fdl-SPIKE.md</files>
|
||||
<action>
|
||||
For every repo in backend/config/models.yaml (25 entries), determine whether it's Xet-backed or legacy Git-LFS. Use huggingface_hub: `HfApi().repo_info(repo_id, files_metadata=True)` and inspect each LFS blob for xet info, OR call the model-info endpoint and check the `xetEnabled`/blob `xet` field. For gated/unavailable repos, record "unknown (gated/offline)".
|
||||
Write 260613-fdl-SPIKE.md: a table repo_id | role | backend (xet|lfs|unknown) | size, plus a one-line GO/LOW-PRIORITY verdict for Wave 3:
|
||||
- If the majority of *user-facing default* models (OmniVoice TTS, the default ASR) are Xet-backed -> Wave 3 is LOW priority (xet already fast); still build it for the LFS long tail.
|
||||
- If many defaults are still LFS -> Wave 3 is HIGH priority.
|
||||
This is read-only network classification — do not download anything (use repo_info, not snapshot_download).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>test -f .planning/quick/260613-fast-model-downloads/260613-fdl-SPIKE.md || test -f .planning/quick/260613-fdl-fast-model-downloads/260613-fdl-SPIKE.md && echo "spike written"</automated>
|
||||
</verify>
|
||||
<done>SPIKE.md lists every catalog repo with its storage backend and a GO/LOW-PRIORITY verdict for Wave 3.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 1 — MAXIMIZE + GUARANTEE THE XET FAST PATH ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1 (FDL-01): Pin huggingface_hub + hf-xet explicitly</name>
|
||||
<files>pyproject.toml</files>
|
||||
<action>
|
||||
Today huggingface_hub arrives transitively (1.7.2) and hf_xet is present but unpinned. Add explicit runtime pins so the fast path can never silently disappear on a resolve:
|
||||
- huggingface_hub>=1.7 (keep compatible with transformers>=5.3.0 already in deps)
|
||||
- hf-xet>=1.1 (the Xet backend; 64-bit only — fine for all OmniVoice targets)
|
||||
Do NOT add hf_transfer. Run `uv sync` then `uv tree huggingface_hub` to confirm a single resolved version (no duplicate). If a transitive constraint conflicts, prefer the higher version and note it in the SUMMARY.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "huggingface_hub\|hf-xet\|hf_xet\|hf-transfer\|hf_transfer" pyproject.toml</automated>
|
||||
<automated>uv run python -c "import huggingface_hub,hf_xet; print('hub',huggingface_hub.__version__,'xet ok')"</automated>
|
||||
</verify>
|
||||
<done>pyproject pins huggingface_hub>=1.7 and hf-xet; no hf_transfer; uv resolves cleanly with one huggingface_hub.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2 (FDL-02): Drive snapshot_download explicitly</name>
|
||||
<files>backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
In install_model's _do() (line ~148), build dl_kwargs with explicit, intentional args instead of the bare call:
|
||||
- tqdm_class=<the TrackedTqdm class> so progress is deterministic and xet-aware rather than relying solely on the global monkeypatch. Expose TrackedTqdm from hf_progress (add a getter, e.g. hf_progress.tracked_tqdm_class()).
|
||||
- max_workers: keep default 8 (don't crank — xet does intra-file parallelism; high max_workers multiplies buffer pressure). Make it prefs-overridable: prefs.resolve("download_max_workers", env="OMNIVOICE_DOWNLOAD_MAX_WORKERS", default=8).
|
||||
- endpoint=prefs.resolve("hf_endpoint", env="HF_ENDPOINT", default=None) — wires FDL-10 mirror without process-wide env.
|
||||
- Keep the existing 5-retry backoff, heartbeat, and _validate_snapshot_has_weights.
|
||||
Do not remove the global monkeypatch (other libs — transformers/mlx_whisper — still rely on it); this task just makes the install path drive its own tqdm_class explicitly.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "tqdm_class\|max_workers\|endpoint" backend/api/routers/setup/download.py</automated>
|
||||
<automated>uv run pytest tests/ -k "download or install" -q 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<done>install_model drives snapshot_download with explicit tqdm_class + max_workers + endpoint; retry/validate intact; tests green.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3 (FDL-03, FDL-04): fast_download status + opt-in xet knobs</name>
|
||||
<files>backend/api/routers/system.py, backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
- FDL-03: add a fast_download block to GET /system/info (system.py:245): {xet_enabled: bool, xet_version: str|None, high_performance: bool}. Probe by importing hf_xet (xet_enabled), reading its version, and reading the HF_XET_HIGH_PERFORMANCE env/pref. Must never throw (system_info is called on every Settings load). Log the same line once at startup ("fast download: Xet on (hf_xet X.Y), high_perf=...").
|
||||
- FDL-04: opt-in knobs via prefs, applied at process/download setup (env wins):
|
||||
high_performance = prefs.resolve("xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False)
|
||||
hdd_sequential = prefs.resolve("xet_hdd_sequential_write", env="HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY", default=False)
|
||||
When set, export the corresponding HF_XET_* env before the snapshot/segmented download runs. Both default OFF (high-perf can hurt low-RAM machines — surface that as a tooltip in Wave 2 UI).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>curl -s http://127.0.0.1:3900/system/info | python3 -c "import json,sys; print(json.load(sys.stdin).get('fast_download'))" 2>/dev/null || grep -n "fast_download" backend/api/routers/system.py</automated>
|
||||
</verify>
|
||||
<done>/system/info reports fast_download truthfully; high-perf + HDD knobs resolve via prefs with env precedence, default off; startup logs Xet status.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 2 — ACCURATE DOWNLOADED/REMAINING + SPEED ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 4 (FDL-05): dry_run preflight -> install_plan event</name>
|
||||
<files>backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
Before the real download in install_model, run snapshot_download(repo_id, dry_run=True, endpoint=...) on the worker thread. From the returned per-file info compute: total_bytes, cached_bytes (files already present), to_download_bytes, n_files, n_cached. Emit a new phase event:
|
||||
{repo_id, phase:"install_plan", total_bytes, cached_bytes, to_download_bytes, n_files, n_cached}
|
||||
This gives the UI an accurate denominator and a "M GB already cached, N GB to download" line BEFORE bytes flow. Wrap dry_run in try/except — if it fails (older/gated repo), emit install_plan with totals=None and fall back to today's behavior (denominator fills in as files appear). dry_run must respect the 'resolving' heartbeat (it can take a couple seconds).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "dry_run\|install_plan\|to_download_bytes" backend/api/routers/setup/download.py</automated>
|
||||
<automated>uv run pytest tests/backend/setup/test_download_preflight.py -q 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<done>An install emits install_plan with accurate total/cached/remaining before download; dry_run failure degrades gracefully to old behavior.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 5 (FDL-06): Backend aggregate progress tracker</name>
|
||||
<files>backend/utils/download_aggregator.py, backend/utils/hf_progress.py</files>
|
||||
<action>
|
||||
New backend/utils/download_aggregator.py: a per-repo DownloadAggregator that owns the TRUTH for overall progress, so the frontend stops summing potentially-misrouted per-file events.
|
||||
- Seeded by the install_plan totals (total_bytes, n_files).
|
||||
- add(filename, bytes_delta) / set_file(filename, downloaded, total): track bytes per file; bytes_done = sum.
|
||||
- Rate: sampled over a sliding window (e.g. last ~5-10s of (t, bytes_done) samples), not a single tqdm bar's rate. eta = remaining / rate.
|
||||
- snapshot() -> {repo_id, bytes_done, total_bytes, rate, eta_seconds, files_done, files_total, phase}.
|
||||
- Emits one throttled (~0.3-0.5s) phase:"aggregate" event via hf_progress.emit().
|
||||
Wire it: hf_progress's TrackedTqdm._emit_progress already has per-file (filename, downloaded, total) — also feed those into the active repo's aggregator (look up by current_repo_id). The segmented downloader (Wave 3) calls aggregator.add() directly. Keep the per-file events too (UI detail view) — aggregate is additive, not a replacement.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run python -c "from utils.download_aggregator import DownloadAggregator as A; a=A('r',total_bytes=100,files_total=2); a.set_file('f1',50,50); a.set_file('f2',25,50); s=a.snapshot(); print(s['bytes_done'], s['total_bytes'])"</automated>
|
||||
</verify>
|
||||
<done>DownloadAggregator sums bytes across parallel files, samples rate over a window, emits a single 'aggregate' event; fed by both tqdm and the segmented path.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 6 (FDL-07): Frontend overall progress bar</name>
|
||||
<files>frontend/src/pages/Settings.jsx, frontend/src/api/setup.ts</files>
|
||||
<action>
|
||||
- setup.ts: extend SetupProgressEvent phase union with "install_plan" | "aggregate" and their fields (total_bytes, cached_bytes, to_download_bytes, n_files, n_cached, bytes_done, rate, eta_seconds, files_done, files_total).
|
||||
- Settings.jsx ModelStoreTab: when an aggregate event arrives for a repo, render ONE overall progress row: a bar (bytes_done/total_bytes), instantaneous speed (format rate as MB/s), "X.X GB of Y.Y GB" downloaded/remaining, and ETA (mm:ss from eta_seconds). Seed the denominator from install_plan (show "M GB cached, N GB to download" before bytes flow). Keep the existing per-file rows as a collapsible "details" section instead of the primary display. Show a small "⚡ fast download" badge when /system/info fast_download.xet_enabled is true.
|
||||
- Prefer the backend aggregate's rate/eta over the frontend's own per-file ETA computation (Settings.jsx ~614-631) — replace that local ETA math with the aggregate fields; keep a fallback if no aggregate event has arrived yet.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd frontend && bun run typecheck 2>&1 | tail -15</automated>
|
||||
<automated>grep -n "aggregate\|install_plan\|eta_seconds\|fast download" frontend/src/pages/Settings.jsx frontend/src/api/setup.ts</automated>
|
||||
</verify>
|
||||
<done>UI shows one overall bar with live speed + downloaded/remaining + ETA from the aggregate event; per-file detail collapsible; fast-download badge; typecheck passes.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 3 — OPT-IN IDM-STYLE SEGMENTED ACCELERATOR (LFS REPOS) ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 7 (FDL-08): Custom httpx segmented downloader</name>
|
||||
<files>backend/services/segmented_download.py, tests/backend/services/test_segmented_download.py</files>
|
||||
<action>
|
||||
New backend/services/segmented_download.py — an OPT-IN multi-connection Range downloader for ONE file (the IDM/uGet technique) used only for legacy-LFS repos where Xet gives no intra-file parallelism. httpx is already a dep.
|
||||
Contract (async def segmented_download(url, dest, *, token, expected_size, expected_etag=None, num_connections=8, chunk_aggregator=None, cancel_event=None)):
|
||||
1. HEAD (or GET Range: bytes=0-0) the resolve URL to learn size + Accept-Ranges + the redirect target. If server doesn't honor Range (Accept-Ranges != bytes) -> fall back to a single streamed GET (still works, just not parallel).
|
||||
2. AUTH SAFETY (critical): send Authorization: Bearer <token> ONLY to the huggingface.co host. When the resolve URL 302-redirects to the CDN (cloudfront/etc.), do NOT forward Authorization to the CDN host — the presigned URL already carries auth. Follow redirects manually so you control header propagation per-host.
|
||||
3. Split expected_size into num_connections ranges; download each with Range: bytes=start-end concurrently (asyncio + httpx.AsyncClient). Write to dest+".part" at the right offsets (preallocate, or per-range temp files then concat).
|
||||
4. RESUME: if dest+".part" exists with a sidecar manifest of completed ranges, skip completed ranges.
|
||||
5. CANCEL: check cancel_event between chunks; on cancel, leave the .part for resume and raise CancelledError.
|
||||
6. VERIFY: after assembly, check size == expected_size and (if given) sha256/etag matches; only then atomically rename .part -> dest. On mismatch, raise (caller's retry/validate handles it).
|
||||
7. PROGRESS: call chunk_aggregator.add(filename, bytes_delta) as ranges complete bytes (feeds DownloadAggregator).
|
||||
Tests (use a local mock HTTP server / httpx MockTransport): honors Range + parallel assembly == single-GET bytes; falls back when Accept-Ranges absent; does NOT send Authorization to a different host on redirect; resumes from a partial .part; cancels and leaves resumable state; size/etag mismatch raises.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run pytest tests/backend/services/test_segmented_download.py -q 2>&1 | tail -20</automated>
|
||||
<automated>grep -n "Authorization\|Range\|cancel_event\|expected_size" backend/services/segmented_download.py</automated>
|
||||
</verify>
|
||||
<done>segmented_download fetches a file via parallel ranges, is auth-safe across the CDN redirect, resumes, cancels, and verifies size/etag before commit; all tests pass.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 8 (FDL-09): Dispatch — accelerator for LFS repos only</name>
|
||||
<files>backend/api/routers/setup/download.py, backend/api/routers/setup/models.py</files>
|
||||
<action>
|
||||
- models.py: add a small helper is_xet_backed(repo_id) -> bool|None (reuse FDL-00's classification approach; cache result). Used to decide the path.
|
||||
- download.py install_model dispatch:
|
||||
accelerator_on = prefs.resolve("segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=False)
|
||||
if accelerator_on and is_xet_backed(repo_id) is False:
|
||||
-> resolve each LFS file's URL via hf_hub_url + HfApi file metadata, download via segmented_download into the HF cache layout (or download to a temp dir then place via the cache API so the result is a normal cache entry), feeding the same DownloadAggregator. Run _validate_snapshot_has_weights at the end.
|
||||
else:
|
||||
-> existing snapshot_download path (xet).
|
||||
IMPORTANT: the segmented result MUST land in the same HF cache structure so /models install-state, delete, and is_cached() all keep working (truth: "indistinguishable from snapshot_download"). If matching the blob/snapshot symlink layout is too fiddly, the safe fallback is: segmented-download to a temp file, then hand the bytes to huggingface_hub so it finalizes the cache entry. Document the chosen approach in SUMMARY.
|
||||
Default OFF -> zero behavior change unless the user opts in.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "segmented_downloader\|is_xet_backed\|segmented_download" backend/api/routers/setup/download.py backend/api/routers/setup/models.py</automated>
|
||||
<automated>uv run pytest tests/ -k "download or install or model" -q 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<done>With the toggle ON, LFS repos download via the segmented path into the normal HF cache; Xet repos and the default (toggle OFF) use snapshot_download; install-state/delete unaffected.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 4 — MIRROR PATH + CANCEL + DOCS ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 9 (FDL-10, FDL-11): Mirror opt-in + cancel endpoint</name>
|
||||
<files>backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
- FDL-10: the endpoint= wiring from Task 2 already reads prefs hf_endpoint. Surface it as a setting and document (Task 10) that a mirror routes through the CLASSIC LFS path (no Xet) — so it pairs naturally with the FDL-08 segmented accelerator for speed on mirrors. No process-wide HF_ENDPOINT mutation; per-call endpoint only.
|
||||
- FDL-11: add POST /models/install/cancel {repo_id} that sets the repo's cancel_event (segmented path) and, for the snapshot path, best-effort marks the install cancelled (snapshot_download isn't trivially cancellable mid-file — at minimum stop retries and emit install_cancelled). Compose with MM2-06: on success OR cancel, clear the _install_cooldowns entry so a cancelled download isn't rate-limited. Emit phase:"install_cancelled".
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "install/cancel\|cancel_event\|install_cancelled\|hf_endpoint" backend/api/routers/setup/download.py</automated>
|
||||
</verify>
|
||||
<done>Per-call mirror endpoint wired (opt-in); cancel endpoint stops the segmented path and clears cooldown; emits install_cancelled.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 10 (FDL-12): Docs — downloading-models.md + README pointer</name>
|
||||
<files>docs/downloading-models.md, README.md</files>
|
||||
<action>
|
||||
Per the docs-sync hard rule, document the user-facing surface introduced here:
|
||||
- How fast downloads work (Xet on by default; what the ⚡ badge means; how to check via Settings/system info).
|
||||
- Advanced toggles: high-performance mode (warn: needs RAM/bandwidth, can hurt low-RAM machines), HDD sequential-write, max workers, segmented accelerator (opt-in, for legacy-LFS repos), and the mirror/restricted-network HF_ENDPOINT setting (note: mirror = classic LFS, no Xet; pair with the accelerator).
|
||||
- A short troubleshooting section (slow downloads, stuck at resolving, restricted networks/China).
|
||||
Add a one-line pointer from README.md to docs/downloading-models.md. Do NOT enable any opt-in by default in docs examples.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>test -f docs/downloading-models.md && grep -n "Xet\|HF_ENDPOINT\|high-performance\|segmented" docs/downloading-models.md | head</automated>
|
||||
<automated>grep -n "downloading-models" README.md</automated>
|
||||
</verify>
|
||||
<done>docs/downloading-models.md covers speed, status, all opt-in knobs, mirror/restricted-network, troubleshooting; README links it; no opt-in shown as default.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
Gate per wave; full set before the last PR:
|
||||
1. `uv run pytest tests/backend/setup/test_download_preflight.py tests/backend/services/test_segmented_download.py tests/ -k "download or install or model" -q` — green.
|
||||
2. Live smoke (backend running): an install emits install_plan (accurate total/cached/remaining) THEN aggregate events with rising bytes_done + a non-zero rate + decreasing ETA; on completion bytes_done == total_bytes.
|
||||
3. /system/info reports fast_download.xet_enabled=true with a version.
|
||||
4. Auth-safety unit test proves Authorization is NOT sent to a non-huggingface.co host on redirect.
|
||||
5. Default-off proof: with no opt-in set, an install uses snapshot_download (xet) — `OMNIVOICE_SEGMENTED_DOWNLOAD` unset means the segmented path is never taken.
|
||||
6. `uv tree huggingface_hub` shows one version; no hf_transfer anywhere (`grep -ri hf_transfer backend/` is empty).
|
||||
7. `cd frontend && bun run typecheck` passes.
|
||||
8. Cross-platform parity: the default path (Xet, pure-Python) is identical on all 3 OSes; every accelerator/mirror/high-perf knob is opt-in (Settings/env). No bundled binary added.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- Fast: Xet is pinned, engaged, and driven with explicit args; high-perf/HDD knobs available opt-in; legacy-LFS repos can use the opt-in segmented accelerator for real multi-connection speed.
|
||||
- Accurate: UI shows pre-flight total/cached/remaining, then one overall bar with live speed + downloaded/remaining + ETA sourced from a backend aggregate (not frontend guesswork).
|
||||
- Safe & compatible: no hf_transfer; segmented downloader is opt-in, auth-safe, resumable, verified, cancellable, and lands in the normal HF cache; default behavior identical on all 3 OSes; no new on-disk model state; existing installs untouched.
|
||||
- All listed tests + typecheck pass; docs updated in the same PR (docs-sync rule).
|
||||
</success_criteria>
|
||||
|
||||
<risks>
|
||||
- **Segmented downloader auth leak (FDL-08) — highest risk.** Forwarding the HF Authorization header to the CDN host on redirect would leak the token. Mitigation: manual redirect handling, per-host header allow-list (Authorization only to huggingface.co), and a dedicated unit test asserting no Authorization on the CDN hop. This is a must-have truth, not optional.
|
||||
- **Cache-layout mismatch (FDL-09).** If the segmented path writes files outside the HF cache blob/snapshot structure, /models install-state + delete + is_cached() break. Mitigation: prefer the temp-file-then-hand-to-huggingface_hub finalization approach over hand-rolling the symlink/blob layout; assert is_cached(repo_id) is true after a segmented install in a test.
|
||||
- **dry_run cost/availability (FDL-05).** dry_run adds a metadata round-trip and may not exist for gated/older repos. Mitigation: try/except -> totals=None fallback to current fill-in-as-you-go behavior; keep the resolving heartbeat so the UI isn't blank during preflight.
|
||||
- **Aggregate vs per-file double-count (FDL-06).** Feeding both tqdm per-file events and the aggregator risks the UI showing two competing numbers. Mitigation: aggregate is the single source of truth for the overall bar; per-file events only drive the collapsible detail view; the frontend's old per-file ETA math is removed (Task 6).
|
||||
- **High-performance mode hurting low-RAM machines (FDL-04).** HF_XET_HIGH_PERFORMANCE can need ~tens of GB RAM. Mitigation: default OFF, opt-in only, tooltip warning in the UI.
|
||||
- **Mirror + Xet confusion (FDL-10).** Users may expect Xet speed through a mirror; mirrors fall back to classic LFS. Mitigation: document explicitly; that's exactly why the segmented accelerator pairs with the mirror path.
|
||||
- **Scope: do not let the segmented path become default.** It's opt-in for LFS repos only. Xet stays the default; making it default would regress dedup + violate the parity rule.
|
||||
</risks>
|
||||
|
||||
<output>
|
||||
Write 260613-fdl-SPIKE.md (Task 0) and 260613-fdl-SUMMARY.md when done. SUMMARY must record: the Xet-vs-LFS catalog breakdown and how it changed Wave 3 priority; the cache-finalization approach chosen for the segmented path (and the is_cached-after-segmented test result); the exact new SSE event shapes (install_plan, aggregate); which opt-in prefs keys + env vars were added; and the auth-safety test output. Note any "use judgment" decision an executor made.
|
||||
|
||||
Docs-sync (CLAUDE.md hard rule): docs/downloading-models.md + README pointer ship in the SAME PR as the user-facing toggles (Task 10). If the Settings UI gains the new toggles, the docs describing them land together.
|
||||
</output>
|
||||
@@ -1,43 +0,0 @@
|
||||
# RESEARCH — Fast HuggingFace model downloads (2026)
|
||||
|
||||
**Date:** 2026-06-13 · **For:** 260613-fdl-PLAN.md
|
||||
|
||||
## Bottom line
|
||||
As of mid-2026 the fast path is **hf-xet, on by default** in modern `huggingface_hub`. Xet is itself a chunk-level, content-defined, massively-parallel downloader with adaptive concurrency — it **is** the "IDM/uGet-style segmented download," done for you and dedup-aware. `hf_transfer` is **deprecated**. Rolling your own segmented downloader or bridging to aria2 is **not worth it as a default**; the only thing we must build is (a) better driving + progress UI and (b) an **opt-in** segmented path for the legacy-LFS long tail (repos Xet doesn't back).
|
||||
|
||||
Installed in this repo: `huggingface_hub 1.7.2`, `hf_xet` present. `snapshot_download` here supports `max_workers`, `tqdm_class`, `endpoint`, `dry_run` (confirmed via inspect).
|
||||
|
||||
## 1. hf-xet — USE (default, no action needed beyond pinning)
|
||||
Content-defined chunks grouped into blocks ("xorbs") in a content-addressable store; download = send file SHA256 → get reconstruction metadata + presigned URLs → fetch needed xorb ranges **in parallel** → reassemble; already-present chunks skipped (dedup). Auto-used by `snapshot_download`/`hf_hub_download` for Xet-backed repos since huggingface_hub 0.32. 2–3× over Git-LFS, up to ~1 GB/s.
|
||||
Knobs (defaults already tuned): `HF_XET_NUM_CONCURRENT_RANGE_GETS` (16), adaptive concurrency ON (max 64), `HF_XET_DATA_MAX_CONCURRENT_FILE_DOWNLOADS` (8), chunk cache disabled by default (better for pure download), `HF_XET_HIGH_PERFORMANCE=1` (opt-in max throughput, needs RAM/bandwidth), `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1` (HDD). **64-bit only.**
|
||||
- https://huggingface.co/docs/huggingface_hub/en/guides/download
|
||||
- https://huggingface.co/docs/hub/en/xet/using-xet-storage
|
||||
- https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables
|
||||
|
||||
## 2. hf_transfer — AVOID (deprecated)
|
||||
`HF_HUB_ENABLE_HF_TRANSFER` flagged deprecated; Xet supersedes it. Historically **broke tqdm progress / had no callbacks** — directly conflicts with the accurate-progress goal. Successor for max throughput is `HF_XET_HIGH_PERFORMANCE=1`.
|
||||
- https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables
|
||||
- https://github.com/huggingface/hf_transfer/issues/63
|
||||
|
||||
## 3. huggingface_hub native concurrency — USE defaults
|
||||
`snapshot_download(max_workers=...)` = parallel FILES (default 8), orthogonal to Xet's intra-file chunk parallelism. For OmniVoice's 1–few-large-file models the win is mostly Xet's intra-file parallelism; don't crank max_workers (multiplies buffer pressure). Resume is automatic via cache + ETag (no `resume_download` flag to manage).
|
||||
- https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
|
||||
|
||||
## 4. Custom IDM-style Range downloader — AVOID as default, BUILD as opt-in for LFS
|
||||
`/resolve/<rev>/<file>` 302-redirects to CDN (Cloudfront) which honors Range + parallel byte-ranges. Catch: follow redirect, **do NOT forward Authorization to the CDN host** (presigned URL carries auth), verify ETag/sha256, auth on first hop only. Redundant vs Xet for Xet-backed repos (HF closed issue #3232 as "use Xet"), **but genuinely helps non-Xet/legacy-LFS repos** which get no intra-file parallelism. → our Wave 3 opt-in.
|
||||
- https://github.com/huggingface/huggingface_hub/issues/3232
|
||||
|
||||
## 5. aria2 — OPTIONAL, rejected for OmniVoice
|
||||
`aria2c -x16 -s16 -c --header="Authorization: Bearer <token>"` is 3–5× on plain LFS, but: no dedup (worse than Xet for Xet repos), per-OS GPLv2 binary to package (parity burden — would have to be opt-in anyway), stdout/RPC progress scraping. The custom httpx path covers the same need with no binary. → not bundled.
|
||||
- https://gist.github.com/padeoe/697678ab8e528b85a2a7bddafea1fa4f
|
||||
|
||||
## 6. Mirrors / HF_ENDPOINT — OPTIONAL, region-gated, breaks Xet
|
||||
`HF_ENDPOINT=https://hf-mirror.com` redirects Hub traffic (standard for China). **Xet CAS/presigned URLs point at HF infra → mirrors generally don't serve the Xet protocol → traffic falls back to classic LFS** (no dedup, no Xet parallelism). So mirror and Xet fast-path are mutually exclusive; the realistic China stack is mirror + LFS + (our opt-in) segmented accelerator. → our Wave 4 opt-in, per-call `endpoint=` not process-wide.
|
||||
|
||||
## 7. Progress / speed — USE `tqdm_class` (xet-aware) + `dry_run` preflight
|
||||
Unlike hf_transfer, **Xet reports progress through the same tqdm interface**; huggingface_hub aggregates per-file/thread bytes into a shared bar and feeds the `tqdm_class` you pass. So `snapshot_download(tqdm_class=...)` yields reliable aggregate bytes/total/rate/ETA even under parallel fetch. `snapshot_download(dry_run=True)` returns per-file sizes + cached flags → use for "will download X of Y, N GB" preflight. Speed sampling tunable via `HF_XET_DATA_PROGRESS_UPDATE_INTERVAL` (200ms).
|
||||
- https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
|
||||
- https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/_snapshot_download.py
|
||||
|
||||
## Recommended architecture (→ plan)
|
||||
Pin `huggingface_hub>=1.7` + `hf-xet`; let Xet be the default (it IS the IDM technique). Drive `snapshot_download(repo_id, tqdm_class=OmniVoiceProgress, max_workers=8, endpoint=<opt-in mirror>)`; `dry_run=True` first for total/remaining; aggregate bytes in a backend tracker → one overall bar (speed/remaining/ETA). Opt-in only: `HF_XET_HIGH_PERFORMANCE` (max speed), `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY` (HDD), a custom httpx **segmented downloader for legacy-LFS repos**, and an `HF_ENDPOINT` mirror (classic-LFS fallback). Never enable hf_transfer; never bundle aria2; never make the segmented path the default.
|
||||
@@ -1,28 +0,0 @@
|
||||
# SPIKE — FDL-00: Catalog Xet vs LFS classification
|
||||
|
||||
**Date:** 2026-06-13 · **Method:** HF API `GET /api/models/{repo}?expand[]=xetEnabled` (authoritative).
|
||||
|
||||
## Result: 25 / 25 catalog repos are Xet-backed
|
||||
|
||||
| backend | count |
|
||||
|---------|-------|
|
||||
| xet | 25 |
|
||||
| lfs | 0 |
|
||||
| unknown | 0 |
|
||||
|
||||
Every repo in `backend/config/models.yaml` — including both first-run defaults (`k2-fsa/OmniVoice` TTS, `Systran/faster-whisper-large-v3` ASR) — returns `xetEnabled: true`. Full list: all entries under TTS / ASR / Diarisation (k2-fsa, Systran×5, mlx-community×9, openai, nvidia×2, UsefulSensors×2, pyannote, OpenMOSS, KittenML, deepdml).
|
||||
|
||||
## Detection caveat (important for the executor)
|
||||
|
||||
The installed client is **huggingface_hub 1.7.2**, whose `repo_info(..., files_metadata=True)` siblings expose only `blob_id, lfs, rfilename, size` — **no `xet_file`, and no `xet_enabled` on the info object.** A first pass that inferred backend from siblings wrongly reported "0/25 xet, all LFS." Do **not** classify Xet status from `repo_info` siblings on this client version. The reliable signal is the Hub API `xetEnabled` expand field (used here) or `hf_xet` actually engaging at download time. Re-check after any `huggingface_hub` bump — newer versions surface `xet_enabled` directly.
|
||||
|
||||
## Verdict for Wave 3 (segmented accelerator): LOW priority
|
||||
|
||||
Because the entire current catalog is Xet-backed and `hf_xet` is installed, Xet already provides chunked parallel range-gets (the IDM/uGet behavior) for **every** model we ship. The custom segmented downloader (Wave 3) is therefore **not needed to speed up any current default model** — it remains valuable only for:
|
||||
- the **mirror / restricted-network path** (Wave 4: `HF_ENDPOINT` falls back to classic LFS, no Xet), and
|
||||
- any **future non-Xet repo** a user adds.
|
||||
|
||||
**Recommendation:** proceed with W1 (maximize/guarantee Xet) and W2 (accurate progress) as the real wins for today's catalog; keep W3 as opt-in, build it alongside W4's mirror path where it actually pays off. This matches the PLAN's original framing — confirmed, not changed.
|
||||
|
||||
## Consequence for W1/W2 framing
|
||||
W1 "guarantee the Xet fast path" is correctly the primary lever: these repos download via Xet **only if** the client engages it (hf_xet installed ✓ + huggingface_hub recent ✓). The W2 live smoke test should confirm Xet is actually used (fast parallel aggregate progress on a real install), since `xetEnabled=true` is a Hub-side capability, not proof the client took the Xet path.
|
||||
@@ -1,49 +0,0 @@
|
||||
# SUMMARY — FDL Waves 0–2 (fast model downloads)
|
||||
|
||||
**Date:** 2026-06-13 · **Scope shipped:** W0 (spike), W1 (maximize Xet), W2 (accurate progress). W3/W4 deferred.
|
||||
|
||||
## What landed
|
||||
|
||||
**W0 — spike (FDL-00).** Classified all 25 `models.yaml` repos via the HF API `xetEnabled` field → **25/25 Xet-backed** (incl. both first-run defaults). See `260613-fdl-SPIKE.md`. Verdict: Wave 3 (segmented accelerator) is **LOW priority** — Xet already gives parallel chunked transfer for every shipped model. Detection caveat recorded: `repo_info` siblings on hf_hub **1.7.2** expose no xet metadata; classify via the `xetEnabled` API field, not siblings.
|
||||
|
||||
**W1 — maximize + guarantee Xet (FDL-01..04).**
|
||||
- `pyproject.toml`: pinned `huggingface_hub>=1.7` + `hf-xet>=1.1` explicitly (was transitive/unpinned); no `hf_transfer`. Resolves to hf_hub 1.7.2 / hf-xet 1.4.2, single version.
|
||||
- `download.py`: `install_model` now drives `snapshot_download` with explicit `tqdm_class` (our progress-emitting subclass), `max_workers` (prefs `download_max_workers`, default 8), and `endpoint` (prefs `hf_endpoint` — W4 hook). `apply_xet_env()` applies opt-in `HF_XET_HIGH_PERFORMANCE` + `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY` (both default OFF, env wins).
|
||||
- `system.py`: `/system/info` now returns `fast_download {xet_enabled, xet_version, high_performance}`; logged once at startup.
|
||||
|
||||
**W2 — accurate downloaded/remaining + speed (FDL-05..07).**
|
||||
- Preflight `snapshot_download(dry_run=True)` → `compute_plan()` → `install_plan` SSE event with `total_bytes / cached_bytes / to_download_bytes / n_files / n_cached` **before bytes flow**. Degrades to totals=None on gated/older repos.
|
||||
- New `utils/download_aggregator.py`: one source of truth for overall progress. Fed by a byte-sink on the patched tqdm; distinguishes byte bars (unit 'B', keyed by bar id) from the "Fetching N files" count bar; emits one throttled `aggregate` event (bytes_done/total/windowed rate/eta/files).
|
||||
- Frontend `Settings.jsx` + `setup.ts`: overall bar driven by the aggregate; bar % = `max(byte%, file%)`; shows cached-skip + files-progress; `⚡ fast download` badge from `/system/info`. i18n keys added to `en.json`.
|
||||
|
||||
## Rebase reconciliation (main disabled Xet)
|
||||
Rebasing onto latest main surfaced that main now sets **`HF_HUB_DISABLE_XET=1`** (main.py) — a deliberate choice to force the classic LFS path because Xet's progress bypasses the tqdm hook (the exact limitation found here). Reconciled rather than fought:
|
||||
- `fast_download` status now reports the **runtime truth**: `xet_installed` + `xet_active` (active = installed AND not disabled) + `xet_enabled` alias. Default `xet_active=false`; the ⚡ badge only shows when Xet actually runs. Startup log: `downloads: Xet disabled → legacy LFS …`.
|
||||
- Docs rewritten: default backend is **legacy LFS for accurate progress**; Xet is opt-in via `HF_HUB_DISABLE_XET=0` (coarser progress). The hf-xet pin stays (harmless; ready for a future Xet progress hook).
|
||||
- Net: W2's progress is the value either way; W1's "maximize Xet" is dormant by main's design, not removed.
|
||||
|
||||
## Decisions / "use judgment" notes
|
||||
- **Xet progress limitation (verified by live smoke).** Under Xet + hf_hub 1.7.2 the per-file **byte** bars never advance `n` and never `close()` through our tqdm (Xet fetches chunks out-of-band). Only the **file-count bar** is live. So: mid-download the overall bar is **file-granular** (moves 0→N files), and `complete()` flushes `bytes_done` to the exact preflight total on success (verified: final `74420620/74420620`, files 4/4). True live byte-speed is only available on classic-LFS/mirror repos (W4). This is a real constraint, not a bug — documented here and worth surfacing in W4 docs.
|
||||
- Per-file detail kept inline (existing single-line summary, now aggregate-sourced) rather than a new collapsible panel — limited risk; can revisit.
|
||||
|
||||
## Drive-by fix
|
||||
- `download.py` imported no `os`, but `_validate_snapshot_has_weights` uses `os.walk` → latent `NameError` on every install. Added `import os`.
|
||||
|
||||
## Verification
|
||||
- `tests/backend/setup/test_download_preflight.py` — 10 pass (compute_plan splits, aggregator byte/count routing, close-credit, windowed rate/eta, registry feed + finish noop).
|
||||
- `pytest -k "download or install or model or engine or setup"` — 149 passed, 7 skipped, 0 failed.
|
||||
- `frontend typecheck:ci` — exit 0.
|
||||
- Live smoke (real install of `mlx-community/whisper-tiny-mlx`, then deleted): `install_plan` exact; aggregate files 0→1→4; final bytes==total; `/system/info` + startup log correct.
|
||||
|
||||
## W4 — mirror + cancel + docs (FDL-10..12, shipped)
|
||||
- **Mirror (FDL-10):** `snapshot_download(endpoint=…)` honours prefs `hf_endpoint` / env `HF_ENDPOINT` on both preflight and download — per-call, no process-wide mutation. Documented as the classic-LFS (non-Xet) path that restores continuous byte-speed.
|
||||
- **Cancel (FDL-11):** `POST /models/install/cancel {repo_id}` sets a cancel flag checked at each retry boundary → emits `install_cancelled`, clears the cooldown (cancel ≠ failure). Limitation: an in-flight single-file fetch isn't interruptible in hf_hub 1.7.2; cancel lands at the next retry boundary. Frontend treats `install_cancelled` as a terminator (clears row + refetch).
|
||||
- **Docs (FDL-12):** `docs/downloading-models.md` (Xet fast path, progress semantics incl. the byte-speed limitation, opt-in tuning knobs, mirror/restricted-network, cancel, troubleshooting) + README pointer. Docs-sync rule satisfied in-PR.
|
||||
|
||||
## W3 — opt-in segmented accelerator (FDL-08/09, shipped)
|
||||
Reprioritised from LOW to HIGH after the rebase: since main forces Xet off, the default path is single-stream legacy LFS, so a segmented downloader is the way to get **both** parallel speed and live byte progress.
|
||||
- `services/segmented_download.py`: async multi-connection Range downloader for one file — parallel byte-ranges, resume (`.part` + manifest), per-segment short-read truncation guard, optional sha256/etag verify, cancel, single-stream fallback when the server won't range. **Auth-safe**: the HF `Authorization` header goes only to `huggingface.co`/`hf.co`; never forwarded to a CDN host on redirect (unit-tested).
|
||||
- Dispatch (`download.py`): opt-in via prefs `segmented_downloader` / env `OMNIVOICE_SEGMENTED_DOWNLOAD` (default OFF). When on and Xet inactive, fetches each repo file into the HF cache mirroring `hf_hub_download` (blobs + snapshot symlinks + `refs/main`), feeding **real bytes** to the aggregator. Any failure falls back to `snapshot_download` — the accelerator can never break a correct install.
|
||||
- Verified live (accelerator ON): real mid-download byte progress (1.5 KB → 71 MB, rate ramping to **16.6 MB/s**), final `bytes_done == total`, `/models` shows `installed: True`, delete frees the right bytes.
|
||||
- Fixed a `complete()` double-count (was adding a full total on top of accumulated segmented bytes → 2×); now replaces byte bars so the sum is exactly total.
|
||||
- Tests: `tests/backend/services/test_segmented_download.py` (7 cases) covering parallel range reassembly, single-stream fallback, the auth header reaching only the HF host (never a CDN), size/truncation rejection, cancellation, and byte-callback totals — plus an aggregator double-count regression.
|
||||
@@ -1,363 +0,0 @@
|
||||
---
|
||||
phase: 260613-mm2
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- backend/services/tts_backend.py
|
||||
- backend/services/model_manager.py
|
||||
- backend/services/subprocess_backend.py
|
||||
- backend/services/model_lifecycle.py # NEW
|
||||
- backend/api/routers/system.py
|
||||
- backend/api/routers/setup/download.py
|
||||
- backend/api/routers/setup/models.py
|
||||
- tests/test_engines.py
|
||||
- tests/backend/services/test_model_lifecycle.py # NEW
|
||||
- tests/backend/services/test_subprocess_reaper.py
|
||||
autonomous: true
|
||||
requirements:
|
||||
# ── Tier 1 — Correctness (Wave 1) ──────────────────────────────────────────
|
||||
- MM2-01 # Registry reuses one active instance and calls unload() on engine switch
|
||||
- MM2-02 # Per-engine unload() overrides (in-process drop+free_vram; subprocess -> unload_sidecar)
|
||||
- MM2-03 # /model/loaded + /model/unload report ASR honestly; no unloadable:False-but-loaded lies
|
||||
# ── Tier 2 — Single lifecycle surface (Wave 2) ─────────────────────────────
|
||||
- MM2-04 # model_lifecycle facade owns list_loaded/unload/unload_all/free_vram across all 3 worlds
|
||||
- MM2-05 # Idle/timeout config unified through core.prefs.resolve (env still wins); no duplicated constants
|
||||
# ── Tier 3 — Robustness & observability (Wave 3) ───────────────────────────
|
||||
- MM2-06 # _install_cooldowns bounded (evict on success + TTL); no unbounded growth
|
||||
- MM2-07 # Snapshot weight validation is per-role, not one 5 MB magic number
|
||||
- MM2-08 # Subprocess sidecars self-report VRAM in pong; panel shows real MB, not 0
|
||||
- MM2-09 # scan_cache_dir() -> disk-walk fallback logs WHY it fell back (WinError #117/#118)
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "Switching the active TTS backend in Settings releases the outgoing engine's VRAM before the new one loads — verified by asserting the outgoing instance's unload() was called exactly once on switch."
|
||||
- "TTSBackend.unload() is overridden by OmniVoiceBackend (drops model ref + free_vram) and by every SubprocessBackend subclass (routes to unload_sidecar); all overrides are idempotent and safe before first generate()."
|
||||
- "/model/loaded never reports a model as loaded with a misleading unloadable flag: the ASR row's unloadable reflects whether it can actually be released independently of the TTS lifecycle."
|
||||
- "services.model_lifecycle is the single import surface for list_loaded()/unload(id)/unload_all()/free_vram(); system.py routers call it instead of re-enumerating models inline."
|
||||
- "Idle timeouts for the in-process model and subprocess sidecars resolve through core.prefs.resolve(... env=...) so an env var still wins and the Settings store can override; no module duplicates IDLE_TIMEOUT_SECONDS by hand."
|
||||
- "_install_cooldowns cannot grow without bound: entries are removed on successful install and stale entries are evicted by TTL."
|
||||
- "A live subprocess sidecar reports a non-zero vram_mb in /model/loaded when it actually holds GPU memory (pong carries the figure); CPU-only sidecars report 0 truthfully."
|
||||
- "When scan_cache_dir() raises and the code falls back to the on-disk walk, the reason is logged at WARNING with the exception type (the #117/#118 WinError-448 path is no longer silent)."
|
||||
- "uv run pytest tests/test_engines.py tests/backend/services/test_model_lifecycle.py tests/backend/services/test_subprocess_reaper.py tests/test_model_load_timeout.py passes."
|
||||
- "No on-disk model state changes; no new runtime dependency added; behavior degrades gracefully (not errors) on MPS/CPU where VRAM APIs are sparse."
|
||||
artifacts:
|
||||
- path: "backend/services/tts_backend.py"
|
||||
provides: "Active-instance reuse + unload-on-switch in get_active_tts_backend(); per-engine unload() overrides"
|
||||
contains: "_active_instance AND (def unload)"
|
||||
- path: "backend/services/model_lifecycle.py"
|
||||
provides: "Facade owning list_loaded/unload/unload_all/free_vram across in-process + subprocess models"
|
||||
contains: "def list_loaded AND def unload_all"
|
||||
- path: "backend/api/routers/system.py"
|
||||
provides: "Thin /model/loaded + /model/unload routers delegating to model_lifecycle"
|
||||
contains: "model_lifecycle"
|
||||
key_links:
|
||||
- from: "get_active_tts_backend() (tts_backend.py:1235)"
|
||||
to: "outgoing backend.unload()"
|
||||
via: "module-level _active_instance compared against newly-resolved active_backend_id()"
|
||||
pattern: "_active_instance"
|
||||
- from: "system.py /model/loaded + /model/unload (system.py:129, 210)"
|
||||
to: "model_lifecycle.list_loaded() / model_lifecycle.unload()"
|
||||
via: "import services.model_lifecycle"
|
||||
pattern: "model_lifecycle\\.(list_loaded|unload)"
|
||||
- from: "subprocess sidecar pong reply (subprocess_backend.py:435-438)"
|
||||
to: "list_live_sidecars() vram_mb field"
|
||||
via: "ping reply carries allocated VRAM measured inside the sidecar process"
|
||||
pattern: "vram_mb"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Clean up OmniVoice's model-management subsystem ("v2"). Today load / unload / list / free-VRAM each behave differently across three worlds — the in-process model (`model_manager.py`), the TTS backend registry (`tts_backend.py`), and subprocess sidecars (`subprocess_backend.py`) — with no single lifecycle owner. This produces one real user-facing bug (VRAM leak on engine switch), inaccurate VRAM/unloadable reporting, an unbounded cooldown dict, and a silent cache fallback.
|
||||
|
||||
This is **cleanup + correctness, not a rewrite.** The Wave 13 idle-reaper and the SubprocessBackend primitive are sound and stay. The `TTSBackend.unload()` contract already exists as a documented default no-op (`tts_backend.py:149`) explicitly deferred to "Phase 2"; this plan *is* that Phase-2 follow-through — wire the registry to call it, override it per engine, and unify the surrounding surface.
|
||||
|
||||
Three tiers, executed in order (each independently shippable, continuous-to-main per the v0.3.0 cadence):
|
||||
- **Wave 1 / Tier 1 — Correctness:** MM2-01..03. The VRAM leak on switch + honest unload reporting. Highest value; ship first.
|
||||
- **Wave 2 / Tier 2 — Single lifecycle surface:** MM2-04..05. Extract `model_lifecycle` facade + unify idle/timeout config.
|
||||
- **Wave 3 / Tier 3 — Robustness & observability:** MM2-06..09. Bounded cooldowns, per-role weight validation, sidecar VRAM self-report, cache-fallback logging.
|
||||
|
||||
Output: PRs on branches off `main` (one per wave is fine), each green on the listed pytest selection. No push until the orchestrator merges; tests added with each wave.
|
||||
|
||||
Out of scope (call out, do not touch): GPU-pool per-engine sizing (`model_manager.py:42`, `_GPU_VRAM_PER_JOB_GB`) and torch.compile tuning — those are performance, not cleanup, and carry regression risk against #278/#315.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/STATE.md
|
||||
@./CLAUDE.md
|
||||
|
||||
# Files under edit (read before editing)
|
||||
@backend/services/tts_backend.py
|
||||
@backend/services/model_manager.py
|
||||
@backend/services/subprocess_backend.py
|
||||
@backend/api/routers/system.py
|
||||
@backend/api/routers/setup/download.py
|
||||
@backend/api/routers/setup/models.py
|
||||
|
||||
# Reference only — establish patterns, do NOT modify
|
||||
@backend/core/prefs.py
|
||||
@tests/test_engines.py
|
||||
@tests/backend/services/test_subprocess_reaper.py
|
||||
|
||||
<interfaces>
|
||||
<!-- Verified during planning. Executor should use these directly. -->
|
||||
|
||||
backend/services/tts_backend.py
|
||||
- class TTSBackend(ABC) (line 59); unload() default no-op (line 149) — contract already documented:
|
||||
idempotent, synchronous, safe before first generate().
|
||||
- OmniVoiceBackend.__init__(self, model=None) (line 174); self._model reuses model_manager singleton.
|
||||
- _REGISTRY: dict[str, type[TTSBackend]] (line 1109, a _LazyRegistry).
|
||||
- active_backend_id() (line 1228) -> prefs.resolve("tts_backend", env="OMNIVOICE_TTS_BACKEND", default="omnivoice").
|
||||
- get_active_tts_backend(*, model=None) (line 1235) — builds a FRESH instance every call, no teardown. THE leak.
|
||||
|
||||
backend/core/prefs.py
|
||||
- resolve(key: str, *, env: Optional[str] = None, default: Any = None) -> Any (line 75) — env wins, then store, then default.
|
||||
|
||||
backend/services/model_manager.py
|
||||
- module global `model` (line 111); `_last_used`; free_vram() (line 678); idle_worker() (line 667).
|
||||
- IDLE_TIMEOUT_SECONDS imported from core.config (line 33); duplicated as _IDLE_TIMEOUT_SECONDS (line 114). Collapse.
|
||||
- offload_tts_for_asr() (line 701) / restore_tts_after_asr() — ad-hoc ASR<->TTS VRAM juggling; _diar_pipeline global.
|
||||
|
||||
backend/services/subprocess_backend.py
|
||||
- protocol op set (line 74); SIDECAR_IDLE_TIMEOUT_S = env-only float (line 107) — move to prefs.resolve.
|
||||
- list_live_sidecars() -> list[dict] (line 181); unload_sidecar(engine_id) (line 199); unload_all_sidecars() (line 205).
|
||||
- health/ping: _send({"op":"ping"}) then expect {"op":"pong"} (lines 435-438). Add vram_mb to the pong here AND
|
||||
in the sidecar entry-point that answers ping (search the sidecar worker for the "ping"->"pong" handler).
|
||||
|
||||
backend/api/routers/system.py
|
||||
- GET /model/loaded (line 129) — ~80 lines of inline enumeration of TTS/ASR/diar/sidecars. Replace body with
|
||||
model_lifecycle.list_loaded().
|
||||
- POST /model/unload/{model_id} (line 210) — handles "tts" | "diarization" | "sidecar:<id>" | "sidecars".
|
||||
Replace body with model_lifecycle.unload(model_id).
|
||||
|
||||
backend/api/routers/setup/download.py
|
||||
- _install_cooldowns dict (line 27) — unbounded. _validate_snapshot_has_weights (line 55) + _MIN_WEIGHT_BYTES 5 MB
|
||||
(line 45) — single magic number across roles.
|
||||
|
||||
backend/api/routers/setup/models.py
|
||||
- scan_cache_dir() with silent disk-walk fallback (~line 268-280) + _scan_cache_on_disk (line 177).
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<!-- ════════════ WAVE 1 / TIER 1 — CORRECTNESS ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1 (MM2-02): Per-engine unload() overrides</name>
|
||||
<files>backend/services/tts_backend.py</files>
|
||||
<action>
|
||||
The base-class `unload()` no-op already exists (tts_backend.py:149) with a documented contract. Override it where it matters. Do this BEFORE Task 2 — the registry switch (Task 2) calls these.
|
||||
|
||||
- OmniVoiceBackend (line 162): override `unload(self)`. Drop the local model ref (`self._model = None`) and, because OmniVoice shares the singleton owned by model_manager, also release that: `import services.model_manager as mm; mm.model = None; mm.free_vram()`. Idempotent — guard on `mm.model is not None` before free_vram(). Safe before first generate() (no-op when nothing loaded).
|
||||
- Every SubprocessBackend subclass: implement `unload(self)` on the SubprocessBackend base (subprocess_backend.py — the duck-typed `_is_subprocess_isolated` class) so all subclasses inherit it. It must call `unload_sidecar(self.id)` (force-shut this engine's sidecar; busy sidecars are skipped, never interrupted — existing semantics). Idempotent: unload_sidecar on a non-running engine returns 0, no raise.
|
||||
- In-process non-OmniVoice engines that hold their own model (e.g. KittenTTS/VoxCPM2 keep refs in __init__): override unload() to drop the ref + best-effort empty_cache via the existing free_vram() helper if they used GPU. Where an engine genuinely holds nothing resident, leave the base no-op (and note it in the SUMMARY so the future CI gate knows it's intentional, not missed).
|
||||
|
||||
Honor the contract comment verbatim: idempotent, synchronous, safe pre-load.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "def unload" backend/services/tts_backend.py backend/services/subprocess_backend.py</automated>
|
||||
<automated>uv run python -c "from services.tts_backend import OmniVoiceBackend; b=OmniVoiceBackend(); b.unload(); b.unload(); print('idempotent ok')"</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- OmniVoiceBackend.unload() drops both self._model and mm.model and calls free_vram(), guarded for idempotency.
|
||||
- SubprocessBackend.unload() routes to unload_sidecar(self.id); inherited by all subprocess engines.
|
||||
- Calling unload() twice, and before any generate(), never raises.
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2 (MM2-01): Registry reuses one active instance + unloads on switch</name>
|
||||
<files>backend/services/tts_backend.py</files>
|
||||
<action>
|
||||
Fix the leak at get_active_tts_backend() (line 1235). Today it builds a fresh instance every call with no teardown of the prior engine — switching engines (or repeated synth) leaks VRAM until GC. This is the root cause behind the #278 comment thread.
|
||||
|
||||
- Add a module-level cache: `_active_instance: TTSBackend | None = None` and `_active_instance_id: str | None = None`.
|
||||
- In get_active_tts_backend(): resolve `bid = active_backend_id()`. If `_active_instance is not None` and `_active_instance_id != bid`, call `_active_instance.unload()` (best-effort, wrap in try/except so a bad unload can't block the switch — log on failure) before discarding it.
|
||||
- Build the new instance, store it as `_active_instance` + `_active_instance_id = bid`, return it.
|
||||
- IMPORTANT subtlety: OmniVoiceBackend takes `model=`. When `model=` is passed (the caller already has a loaded model), do NOT cache that instance as the shared `_active_instance` blindly — it's a per-call view over the shared singleton. Keep current behavior for the `model=` path (return a fresh OmniVoiceBackend(model=model)) but still trigger unload() of a *different* outgoing engine first. Pick the simplest correct rule: the cache tracks the configured backend id; passing model= for the SAME id reuses, switching id always unloads the previous. Document the rule in a comment.
|
||||
- Add a module-level `reset_active_backend()` helper that unloads + clears the cache, for app shutdown and tests.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "_active_instance\|def reset_active_backend\|def get_active_tts_backend" backend/services/tts_backend.py</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- Switching backend id calls the outgoing instance's unload() exactly once before the new instance is built.
|
||||
- A bad/raising unload() is caught + logged, never blocks the switch.
|
||||
- reset_active_backend() exists and is idempotent.
|
||||
- The model= fast-path for OmniVoice still works (no double-load).
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3 (MM2-03): Honest /model/loaded + /model/unload for ASR</name>
|
||||
<files>backend/api/routers/system.py</files>
|
||||
<action>
|
||||
The ASR row (system.py:166-175) is reported as unloadable:False, vram_mb:0 even when loaded on GPU, and /model/unload doesn't expose the offload-to-CPU path. Make reporting truthful WITHOUT changing the ASR<->TTS lifecycle coupling (that coupling is intentional — offload_tts_for_asr/restore_tts_after_asr).
|
||||
|
||||
- ASR row: keep unloadable reflecting reality. If ASR truly cannot be released independently of TTS, keep unloadable:False but add a `note` field ("released with TTS") so the UI explains it rather than showing a dead button. Do not invent a separate ASR unload that breaks the WhisperX large-v3 offload path.
|
||||
- vram_mb: if ASR currently runs on CPU (device "cpu" in the row), 0 is correct — leave it but make the device value derive from where the pipe actually is, not a hardcoded "cpu".
|
||||
- This task is intentionally small; the bigger restructure is Task 4 (facade). Land MM2-03 as the honest-reporting fix, then Task 4 moves the enumeration into the facade.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run pytest tests/test_engines.py -q 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- No row reports loaded-but-with-a-misleading-unloadable flag; ASR carries an explanatory note when unloadable:False.
|
||||
- Device field reflects the actual device of the ASR pipe.
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 4 (MM2-01..03 tests): Wave 1 regression tests</name>
|
||||
<files>tests/test_engines.py</files>
|
||||
<action>
|
||||
Add tests proving the leak fix and the unload contract:
|
||||
- test_switching_backend_unloads_previous: monkeypatch two fake backends into _REGISTRY, set active to A (get_active_tts_backend), switch prefs to B, assert A.unload() was called exactly once before B is returned.
|
||||
- test_unload_is_idempotent_and_preload_safe: OmniVoiceBackend().unload() twice + before generate() never raises.
|
||||
- test_reset_active_backend_clears_cache: after reset_active_backend(), the next get_active_tts_backend() builds fresh.
|
||||
Reuse the existing fixture style in tests/test_engines.py (it already monkeypatches the registry / availability). Keep tests CPU-only (no real model load).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run pytest tests/test_engines.py -q 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<done>All three new tests pass; existing test_engines.py tests still green.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 2 / TIER 2 — SINGLE LIFECYCLE SURFACE ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 5 (MM2-04): Extract services/model_lifecycle.py facade</name>
|
||||
<files>backend/services/model_lifecycle.py</files>
|
||||
<action>
|
||||
Create backend/services/model_lifecycle.py as the single owner of cross-world model lifecycle. It composes the existing pieces — it does NOT reimplement loading.
|
||||
|
||||
Public surface:
|
||||
- list_loaded() -> list[dict]: returns the unified rows currently assembled inline in system.py:129-207 (TTS, ASR, diarization, subprocess sidecars). Move that logic here verbatim first, then improve (MM2-03 note field, MM2-08 sidecar vram once Task 8 lands).
|
||||
- unload(model_id: str) -> dict: the dispatch currently inline in system.py:210-242 ("tts" | "diarization" | "sidecar:<id>" | "sidecars"). Move here; keep async-lock semantics for the in-process model (mm._model_lock).
|
||||
- unload_all() -> dict: unload every releasable model (in-process TTS + diar + all sidecars). New convenience used by app shutdown.
|
||||
- free_vram(): thin re-export of model_manager.free_vram() so callers have one import.
|
||||
Keep the "never let sidecar enumeration break the panel" try/except guard.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run python -c "import services.model_lifecycle as ml; print([f for f in ('list_loaded','unload','unload_all','free_vram') if hasattr(ml,f)])"</automated>
|
||||
</verify>
|
||||
<done>model_lifecycle exposes list_loaded/unload/unload_all/free_vram; logic moved out of system.py (not duplicated).</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 6 (MM2-04): Thin system.py routers + facade tests</name>
|
||||
<files>backend/api/routers/system.py, tests/backend/services/test_model_lifecycle.py</files>
|
||||
<action>
|
||||
- Replace the bodies of GET /model/loaded (line 129) and POST /model/unload/{model_id} (line 210) with calls to model_lifecycle.list_loaded() / model_lifecycle.unload(model_id). Preserve the exact response shapes (frontend hooks.ts useModelStatus/useFlushMemory + the flush dropdown depend on {models, count} and {unloaded, success, ...}). The 400 on unknown model_id stays.
|
||||
- New tests/backend/services/test_model_lifecycle.py: list_loaded with nothing loaded returns {models:[], count:0}; unload("tts") when not loaded returns success:False reason:"not loaded"; unload("sidecars") with no sidecars returns count:0; unknown id raises/400 path. Mock model_manager + subprocess_backend so no real models load.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run pytest tests/backend/services/test_model_lifecycle.py -q 2>&1 | tail -20</automated>
|
||||
<automated>grep -n "model_lifecycle" backend/api/routers/system.py</automated>
|
||||
</verify>
|
||||
<done>system.py routers are thin delegations; response shapes unchanged; new facade tests pass.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 7 (MM2-05): Unify idle/timeout config through prefs.resolve</name>
|
||||
<files>backend/services/model_manager.py, backend/services/subprocess_backend.py</files>
|
||||
<action>
|
||||
- model_manager.py: remove the duplicated `_IDLE_TIMEOUT_SECONDS = IDLE_TIMEOUT_SECONDS` (line 114). Resolve at use-site in idle_worker() via prefs: `prefs.resolve("idle_timeout_seconds", env="OMNIVOICE_IDLE_TIMEOUT_S", default=IDLE_TIMEOUT_SECONDS)`. Keep core.config.IDLE_TIMEOUT_SECONDS as the default source.
|
||||
- subprocess_backend.py: replace the env-only `SIDECAR_IDLE_TIMEOUT_S` (line 107) read with prefs.resolve("sidecar_idle_timeout_seconds", env="OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S", default=300.0). Preserve "<=0 disables reaping" semantics and the existing reaper-start guard (line 222). Resolve lazily (function call), not at import, so a test/setting change takes effect — but keep a sensible cached default for the hot reaper loop.
|
||||
- Both must keep env precedence (env wins over store) — that's exactly what prefs.resolve already does.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "_IDLE_TIMEOUT_SECONDS\|prefs.resolve\|SIDECAR_IDLE_TIMEOUT" backend/services/model_manager.py backend/services/subprocess_backend.py</automated>
|
||||
<automated>uv run pytest tests/backend/services/test_subprocess_reaper.py -q 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- No hand-duplicated IDLE_TIMEOUT constant; both timeouts resolve via prefs with env precedence.
|
||||
- Reaper "<=0 disables" + busy-skip behavior unchanged; all 10+ reaper tests still pass.
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 3 / TIER 3 — ROBUSTNESS & OBSERVABILITY ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 8 (MM2-08): Subprocess sidecars self-report VRAM in pong</name>
|
||||
<files>backend/services/subprocess_backend.py</files>
|
||||
<action>
|
||||
Sidecar VRAM is reported as 0 (system.py:192-203 / list_live_sidecars) because the parent can't measure a child's GPU memory. Have the child measure itself.
|
||||
|
||||
- In the sidecar worker's ping handler (the code that answers {"op":"ping"} with {"op":"pong"} — find it in the sidecar entry-point module), include `vram_mb`: measure inside the child via torch.cuda.memory_allocated() (CUDA) or torch.mps.driver_allocated_memory() (MPS, guarded), else 0. Same degrade-gracefully pattern as system.py:147-156.
|
||||
- Parent: in the health-check ping/pong path (subprocess_backend.py:435-438), capture reply["vram_mb"] and stash it on the sidecar record so list_live_sidecars() (line 181) can surface it. Refresh opportunistically on each successful ping; default to last-known or 0 if never measured.
|
||||
- Keep the contract that enumeration never breaks the panel.
|
||||
This is CUDA/MPS-aware and degrades to 0 on CPU — honoring cross-platform parity (default behavior identical; the number is just more accurate where the API exists).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "vram_mb" backend/services/subprocess_backend.py</automated>
|
||||
<automated>uv run pytest tests/backend/services/test_subprocess_reaper.py -q 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<done>list_live_sidecars() exposes a vram_mb sourced from the child's own measurement; 0 only when truly CPU/unmeasured; reaper tests still green.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 9 (MM2-06, MM2-07): Bounded cooldowns + per-role weight validation</name>
|
||||
<files>backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
- MM2-06: _install_cooldowns (line 27) grows unbounded. On a successful install, delete the repo's cooldown entry. Add a TTL sweep: when reading/writing the dict, evict entries older than a fixed window (reuse the existing cooldown window constant; pick the larger of cooldown-window and e.g. 1h). Keep it simple — a dict + timestamps, swept on access. No new dep.
|
||||
- MM2-07: _validate_snapshot_has_weights (line 55) + _MIN_WEIGHT_BYTES 5 MB (line 45) is one magic number for all roles. Make the threshold per-role/per-extension: safetensors/bin/ckpt expect the existing floor; .onnx models (kittentts, supertonic, sherpa) can be legitimately smaller — set a lower, role-aware floor so a valid small ONNX model isn't flagged as truncated. Keep the #352 truncation-catch intent (catch a 0-byte / KB-sized partial), just stop false-positiving small-but-complete models.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "_install_cooldowns\|_MIN_WEIGHT_BYTES\|def _validate_snapshot_has_weights" backend/api/routers/setup/download.py</automated>
|
||||
<automated>uv run pytest tests/ -k "download or install or model" -q 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<done>Cooldown dict is bounded (evict-on-success + TTL sweep); weight validation floor varies by role/extension; #352 truncation still caught.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 10 (MM2-09): Log why scan_cache_dir() fell back to disk walk</name>
|
||||
<files>backend/api/routers/setup/models.py</files>
|
||||
<action>
|
||||
The scan_cache_dir() -> _scan_cache_on_disk() fallback (~line 268-280, helper at line 177) silently swallows the exception — this is the #117/#118 Windows WinError-448 path. Wrap the fallback so it logs at WARNING with the exception type and a one-line reason ("scan_cache_dir failed (%s); falling back to on-disk walk of %s") before walking. Do not change the fallback behavior itself — just stop it being invisible in logs. Keep it from ever raising out (the panel must still render).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "falling back\|logger.warning\|_scan_cache_on_disk\|scan_cache_dir" backend/api/routers/setup/models.py | head</automated>
|
||||
</verify>
|
||||
<done>The disk-walk fallback logs a WARNING naming the exception type; behavior otherwise unchanged; never raises out.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
Full-suite gate after each wave (run the relevant subset per wave, full set before the last PR):
|
||||
|
||||
1. `uv run pytest tests/test_engines.py tests/backend/services/test_model_lifecycle.py tests/backend/services/test_subprocess_reaper.py tests/test_model_load_timeout.py tests/test_model_manager_preload.py -q` — all green.
|
||||
2. `uv run pytest tests/ -k "download or install or model or engine" -q` — green (Tier 3 touch points).
|
||||
3. Response-shape guard: GET /model/loaded still returns {models, count}; POST /model/unload returns {unloaded, success, ...}; 400 on unknown id. (Covered by test_model_lifecycle.py.)
|
||||
4. No new runtime dependency: `git diff pyproject.toml uv.lock` is empty.
|
||||
5. Localization/CJK + redaction gates unaffected: `uv run pytest tests/test_no_hardcoded_cjk.py -q`.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- Tier 1: switching the active backend releases the previous engine's VRAM (unload() called once on switch); contract overridden for OmniVoice + all subprocess engines; ASR reporting is honest. (MM2-01..03)
|
||||
- Tier 2: services.model_lifecycle is the single lifecycle surface; system.py routers are thin delegations with unchanged response shapes; idle/timeout config flows through prefs.resolve with env precedence and no duplicated constants. (MM2-04..05)
|
||||
- Tier 3: cooldown dict bounded; weight validation is per-role; sidecars self-report real VRAM; cache-fallback logs its reason. (MM2-06..09)
|
||||
- All listed pytest selections pass; no on-disk model-state change; no new dep; cross-platform default behavior identical (VRAM numbers degrade gracefully on MPS/CPU).
|
||||
</success_criteria>
|
||||
|
||||
<risks>
|
||||
- **unload() correctness for the shared OmniVoice singleton (MM2-01/02):** OmniVoiceBackend shares model_manager's `model` global. unload() must release the shared singleton, but the idle_worker() + offload_tts_for_asr() paths also touch it. Risk: a switch during an in-flight ASR offload double-frees or races. Mitigation: take mm._model_lock around the shared release in unload(); guard on `mm.model is not None`; keep unload best-effort (try/except) so it can never wedge a switch. Add the idempotency test (Task 4).
|
||||
- **Response-shape drift (MM2-04):** Moving /model/loaded + /model/unload bodies into the facade risks changing the JSON the frontend depends on (hooks.ts, flush dropdown). Mitigation: move verbatim first, assert shapes in test_model_lifecycle.py, only then layer MM2-03/08 improvements.
|
||||
- **Sidecar protocol change (MM2-08):** Adding vram_mb to pong touches the parent/child wire format. Older sidecars (a long-running session mid-upgrade) won't send it. Mitigation: treat vram_mb as optional in the parent (`reply.get("vram_mb", <last-known or 0>)`); never require it; never break the existing pong==success check.
|
||||
- **prefs.resolve at import time (MM2-05):** Resolving timeouts at import freezes them; the reaper loop reads SIDECAR_IDLE_TIMEOUT_S. Mitigation: resolve lazily inside the reaper tick / idle_worker tick (cheap) so a settings change takes effect, while keeping the import-time default for the start-guard.
|
||||
- **Per-role weight floor (MM2-07):** Lowering the ONNX floor could let a genuinely-truncated ONNX through (#352 regression). Mitigation: keep a non-zero floor for every role (e.g. ONNX floor still >> a partial KB), key on extension, and keep the "largest file" heuristic — only the threshold becomes role-aware.
|
||||
- **Scope creep into perf:** GPU-pool sizing and torch.compile are explicitly out of scope. If an executor is tempted, stop — those regress #278/#315.
|
||||
</risks>
|
||||
|
||||
<output>
|
||||
Write `.planning/quick/260613-mm2-clean-model-management-v2/260613-mm2-SUMMARY.md` when done (per wave or once at the end), documenting: which engines got real unload() overrides vs intentional no-ops (for the future CI gate), the exact response shapes preserved on the two endpoints, the per-role weight-validation thresholds chosen, and the pytest output for the verification selection. Note any decision an executor made where the plan said "use judgment."
|
||||
|
||||
Docs-sync check (CLAUDE.md hard rule): this is internal lifecycle cleanup with no user-facing install/Docker/versioning change, so no README/docs edit is expected. If MM2-05 surfaces the new idle-timeout settings keys in the Settings UI, add them to the relevant settings doc in the same PR.
|
||||
</output>
|
||||
@@ -1,32 +0,0 @@
|
||||
# SUMMARY — model-management v2 cleanup (mm2)
|
||||
|
||||
**Date:** 2026-06-13 · **Scope:** all 3 tiers (MM2-01..09). Backend-only; no frontend, no on-disk model-state change, no new deps.
|
||||
|
||||
## Tier 1 — correctness
|
||||
- **MM2-01 (VRAM leak on engine switch):** `get_active_tts_backend()` now caches one instance per configured backend id and calls the outgoing engine's `unload()` before switching. Added `reset_active_backend()` for shutdown/tests. The `model=` OmniVoice fast-path still returns a fresh view over the shared singleton (no double-load) but a switch *away from* another engine still releases it. `tts_backend.py`.
|
||||
- **MM2-02 (per-engine unload()):** `OmniVoiceBackend.unload()` drops the local ref + the shared `model_manager.model` singleton + `free_vram()` (idempotent, preload-safe, best-effort — no async lock from the sync path). `SubprocessBackend.unload()` routes to `unload_sidecar(self.id)` (busy sidecars skipped) and is inherited by every subprocess engine.
|
||||
- **MM2-03 (honest ASR row):** `/model/loaded` ASR row now reports the pipe's actual device and carries a `note: "released with the TTS model"` so the disabled unload button is explained rather than silent.
|
||||
|
||||
## Tier 2 — single lifecycle surface
|
||||
- **MM2-04 (`services/model_lifecycle.py`):** new facade owns `list_loaded()` / `unload(id)` / `unload_all()` / `free_vram()` across in-process TTS+ASR, diarization, and sidecars. `system.py` `/model/loaded` + `/model/unload` are now thin delegations; **response shapes preserved exactly** (`{models,count}`, `{unloaded,success,...}`, 400 on unknown id) — frontend untouched.
|
||||
- **MM2-05 (unified idle config):** removed the duplicated `_IDLE_TIMEOUT_SECONDS`; the in-process idle timeout and the sidecar idle timeout both resolve per-tick via `prefs.resolve(... env=...)` (env wins, settings can tune without restart). New keys: `idle_timeout_seconds` (`OMNIVOICE_IDLE_TIMEOUT_S`), `sidecar_idle_timeout_seconds` (`OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S`). `<=0` still disables sidecar reaping.
|
||||
|
||||
## Tier 3 — robustness & observability
|
||||
- **MM2-06 (bounded cooldowns):** `_install_cooldowns` is swept (TTL 1h) on each install check and cleared on success — can no longer grow unbounded.
|
||||
- **MM2-07 (per-role weight floor):** `_validate_snapshot_has_weights` uses per-extension floors (tensor formats keep 5 MB; `.onnx` floor 64 KB) **OR** the original ≥5 MB catch — strictly more lenient, so a small-but-complete ONNX model is no longer false-flagged as truncated while a 0/KB partial is still rejected (#352 intact).
|
||||
- **MM2-08 (sidecar VRAM self-report):** the parent can't see a child's VRAM, so the GPU sidecar (`engines/indextts`) now reports `vram_mb` in its `pong` (CUDA/MPS-aware, 0 on CPU); the parent stashes the last-known figure and `list_live_sidecars()` surfaces it. CPU/absent sidecars honestly report 0.
|
||||
- **MM2-09 (cache-fallback logging):** the `is_cached` `scan_cache_dir → on-disk` fallback now logs at WARNING with the exception type (was DEBUG/invisible) — the #117/#118 WinError-448 path is triagable from logs.
|
||||
|
||||
## Out of scope (as planned, not done)
|
||||
GPU-pool per-engine sizing (`_GPU_VRAM_PER_JOB_GB`) and torch.compile tuning — perf, not cleanup; risk regressing #278/#315.
|
||||
|
||||
## Verification
|
||||
- New `tests/test_mm2_lifecycle.py` — 15 tests (reuse/switch-unload/reset/idempotent-unload, facade list/unload/unknown/sidecars shapes + honest ASR, env-wins idle config, cooldown sweep, per-role weight floor ×3).
|
||||
- Affected existing: `test_engines.py`, `test_subprocess_reaper.py`, `test_model_load_timeout.py`, `test_model_manager_preload.py` — green (no regressions).
|
||||
- **Full suite: 1379 passed, 0 failed.** Live: facade endpoints return preserved shapes; engine switch calls the previous engine's `unload()` exactly once.
|
||||
|
||||
## Test placement note
|
||||
MM2 tests live at top-level `tests/` (not `tests/backend/`) on purpose: adding files under `tests/backend/` reorders collection and can expose a pre-existing `sys.modules`-isolation leak in other backend fixtures (the issue debugged in the FDL PR). Top-level placement keeps `tests/backend/` order identical.
|
||||
|
||||
## Docs-sync
|
||||
The new idle-timeout settings keys are internal env/prefs knobs with no UI surface, so no README/docs change is required by the docs-sync rule. If a future Settings panel exposes them, document there.
|
||||
-873
@@ -6,879 +6,6 @@ The format is loosely based on [Keep a Changelog](https://keepachangelog.com/).
|
||||
Versions track the desktop app (`tauri.conf.json` + `frontend/src-tauri/Cargo.toml`).
|
||||
The bundled TTS model package (`pyproject.toml`) is versioned independently.
|
||||
|
||||
## [0.3.8] — 2026-07-01
|
||||
|
||||
A stability-focused release that makes first-run and Windows "just work," ships
|
||||
**live, faster-than-real-time local dictation** and a **user pronunciation
|
||||
dictionary**, and gives **Settings a full redesign**. It clears the wave of
|
||||
**"Can't reach the local backend"** reports at the source — the 8 GB-card OOM
|
||||
crash, the slow-load future-scheduling break, a Windows-only WhisperX load
|
||||
failure, an ASR engine that couldn't load CTranslate2 on newer Linux/WSL, and
|
||||
both transcription **and generation** stalls that *looked* like a dead backend
|
||||
(a wedged GPU job now resets the worker pool and returns an actionable timeout)
|
||||
are all fixed or now fail with a clear, actionable message. **macOS gets native file drag-and-drop back**
|
||||
(including macOS 26 Tahoe). Downloads are faster out of the box (parallel
|
||||
segmented transfer on by default) and the Hugging Face token that speeds them up
|
||||
is front-and-center on setup. Plus multi-voice story casting, faster long-form
|
||||
previews on Windows, and a friendlier, more honest batch of error messages
|
||||
across dub, generate, and design (a corrupt-binary failure no longer poses as
|
||||
"out of memory," a bad model id self-heals, and a stale dub job resets cleanly).
|
||||
|
||||
### Added
|
||||
|
||||
- **"Autofit" translation quality — the dub keeps the video's timing.** A new
|
||||
quality alongside Fast and Cinematic: the LLM rewrites each translated line so
|
||||
its target-language reading time fits *within* the segment's slot (a strict
|
||||
"never overrun" bound, per-language pronunciation-speed aware), so long
|
||||
translations no longer force the audio into a stressed >1.3× time-stretch.
|
||||
Cinematic still applies its reflect/adapt polish; Autofit adds the hard
|
||||
fit-to-slot pass on top. Needs an LLM (below); falls back to Fast with a clear
|
||||
notice if none is set. (#838)
|
||||
- **A new LLM Providers settings page — bring your own high-quality LLM.**
|
||||
Settings → System → **LLM Providers** configures the LLM that powers Cinematic
|
||||
and Autofit translation. One page for **16 providers** — OpenAI, OpenRouter,
|
||||
Groq, Cerebras, Google AI (Gemini), Mistral, Cohere, NVIDIA, GitHub Models,
|
||||
Cloudflare, Hugging Face, SambaNova, SiliconFlow, plus **local Ollama / LM
|
||||
Studio** (fully offline, no key) and a **Custom** OpenAI-compatible endpoint.
|
||||
Paste a key, pick a model, **Test** the connection in one click, and "use for
|
||||
translation" to make it active. Keys are stored **encrypted** (the same
|
||||
at-rest protection as the HF token) and never leave the machine unless you
|
||||
choose a cloud provider; env vars still override for power users. The dub
|
||||
translate menu now routes you straight here when you pick a high-quality
|
||||
style without an LLM, instead of dead-ending on a toast. (#838)
|
||||
- **A dedicated Network pane.** The HTTP/SOCKS proxy and FFmpeg-path controls
|
||||
(previously buried in General → Advanced) are promoted to their own category.
|
||||
- **Factory reset in Storage.** A confirm-dialog-guarded action that clears the
|
||||
locally-saved UI preferences and reloads — without touching your voices,
|
||||
projects, or generated audio on disk.
|
||||
- **Proactive, highlighted "Install" affordance for translation engines.** When
|
||||
you pick a Dub translation engine whose optional package isn't installed yet
|
||||
(e.g. Google / DeepL via `deep_translator`), the Engine selector now surfaces a
|
||||
bright accent **Install** button *before* you hit Translate — no more
|
||||
discovering the missing package only via a translate-time 400. On a from-source
|
||||
install it one-click installs into the backend's own interpreter; on a
|
||||
read-only **packaged build** it opens a popover with the exact `uv pip install …`
|
||||
command (copy-to-clipboard), a one-click **Switch to Argos (bundled, offline)**
|
||||
escape hatch, and a docs link. The install command is single-sourced in the
|
||||
backend registry, so the button and the 400 error can never disagree. New guide:
|
||||
`docs/dubbing/translation-engines.md`.
|
||||
|
||||
- **A user pronunciation dictionary that actually changes the audio.** Settings →
|
||||
General → Pronunciation lets you teach the engine how to say tricky words —
|
||||
each entry replaces a term with a respelling (`GIF` → `jiff`) right before
|
||||
synthesis, so it works on **every** engine, not just one. Scope an entry
|
||||
Global or to a single language (a German rule never fires on an English
|
||||
render), with longest-match-first, word-boundary-aware, case-insensitive
|
||||
substitution. For one-offs, write `[[word|respelling]]` inline in your text —
|
||||
it overrides the dictionary for that occurrence and never persists. A built-in
|
||||
Test field previews the substitution with no model call. Pure text transform,
|
||||
identical on macOS/Windows/Linux; plain text stays byte-identical, existing
|
||||
data upgrades cleanly via an additive migration. (Expressive-TTS Spec 01)
|
||||
|
||||
- **Live, faster-than-real-time dictation via a new sherpa-onnx ASR engine.**
|
||||
Pick one of seven small ONNX speech-to-text models (Parakeet TDT v3/v2,
|
||||
streaming Zipformer EN/ZH/bilingual, streaming Paraformer, multilingual
|
||||
Whisper Tiny) for dictation, and watch text appear *as you speak*. Streaming
|
||||
models emit partials frame-by-frame and commit a sentence on natural silence;
|
||||
offline models surface live partials too by re-decoding a growing buffer.
|
||||
Runs CPU-only and identically on macOS, Windows, and Linux — no GPU, no cloud,
|
||||
no extra setup beyond a ~75–180 MB one-time model download. Parakeet TDT v3 is
|
||||
the recommended default; existing Whisper/MLX/NeMo dictation engines are
|
||||
untouched and still the fallback.
|
||||
|
||||
- **New "Voice" settings panel for live dictation.** Settings → Capture now
|
||||
leads with a Voice card: an Enable Voice Dictation toggle (showing your real
|
||||
registered shortcut), a Toggle/Hold mode switch, and a Speech Model dropdown
|
||||
that lists all seven models with offline/streaming + recommended badges, size,
|
||||
one-line descriptions, the installed checkmark, and inline download/delete —
|
||||
reusing the model-store download progress. Picking an uninstalled model starts
|
||||
its download and switches to it once ready. **Toggle vs Hold** is wired for
|
||||
both the desktop global hotkey and the in-app Ctrl/Cmd+Shift+Space fallback, so
|
||||
the behaviour is identical on macOS, Windows, and Linux. While you speak, the
|
||||
dictation pill shows the transcript building **live**, and words type straight
|
||||
into the focused field *as you speak* — self-correcting with backspaces as the
|
||||
streaming recognizer refines, with clipboard-paste as an automatic fallback.
|
||||
|
||||
- **Tagged scripts auto-cast into a multi-voice podcast/audiobook.** Paste a
|
||||
`[Alice] … [Bob] …` script into Stories and hit Auto-cast: it now recognizes
|
||||
the `[Name]` tag format (alongside the existing `NAME:` screenplay and quoted
|
||||
prose), builds the cast, and assigns a voice per character automatically.
|
||||
Editing one line only re-synthesizes that line on export (the chapter cache
|
||||
is content-addressed), and inline markers like `[pause]` / `[voice:…]` are
|
||||
never mistaken for speakers. (#487)
|
||||
- **A dedicated Contact page.** Discord, email, GitHub issues, and the project
|
||||
website (palash.dev) as clean one-tap rows, reachable from the footer — so
|
||||
reaching the maker is never more than a click away.
|
||||
- **Live download speed, remaining size, and ETA on first-run setup.** The
|
||||
Models & Engines step now shows `38% · 5.2 MB/s · 1.2 GB left · ~3m` while a
|
||||
model downloads, instead of a bare "downloading…". (#657)
|
||||
- **Turn off auto-play of the preview after a render.** New Settings →
|
||||
Appearance toggle, "Auto-play preview" (on by default) — switch it off so a
|
||||
finished clip doesn't start playing on its own, ideal when batch-generating
|
||||
segments. (#666)
|
||||
- **App version in the status bar, one click from updates.** A `v<version>`
|
||||
badge sits by the network icon in the bottom bar; clicking it opens Settings →
|
||||
Updates, and it grows a pulsing dot the moment a new version is ready to
|
||||
install. (#671)
|
||||
|
||||
### Changed
|
||||
|
||||
- **Settings is now a sidebar-nav hub instead of an 11-tab strip.** The whole
|
||||
page was rebuilt from scratch as a grouped left-rail navigator (with a
|
||||
search/filter box) plus a scrollable content pane — the macOS System Settings /
|
||||
VS Code layout. Settings are organized into four groups and sixteen
|
||||
categories: **General** (Appearance · General), **Voice & Engines** (Engines ·
|
||||
Models · Dictation · Pronunciation · Translation), **System** (Performance &
|
||||
Device · Storage · Network · Sharing & Remote · Credentials), and **App**
|
||||
(Updates · Privacy & Reporting · Logs · About). Every existing control keeps
|
||||
its behavior and store/API bindings — this is a reorganization, not a rewrite.
|
||||
Typing in the search box filters the category list and jumps to the first
|
||||
match, and the rail collapses to a dropdown navigator below 760px so the full
|
||||
IA stays reachable on a narrow window. Categories whose changes need a backend
|
||||
restart (Models, Performance & Device, Sharing & Remote) carry a "restart
|
||||
required" badge.
|
||||
|
||||
- **The Settings pages got a full redesign — cleaner, denser, responsive.** A
|
||||
shared design system replaces the old patchwork: a left icon nav-rail,
|
||||
sentence-case section titles (no more debug-log uppercase), exactly one muted
|
||||
description per row, unified toggles/inputs, full-width content with proper
|
||||
padding, and horizontal font/theme pickers. Premium and compact instead of
|
||||
sparse and cluttered, and it adapts cleanly to window width. (#686, #690, #696)
|
||||
- **Adding a Hugging Face token on first-run is now a one-line input right by
|
||||
Continue.** Was a bulky card buried at the bottom of the model list; it's now a
|
||||
compact "paste a token, Save" bar pinned next to the "Waiting for required
|
||||
models…" button, so you can add it (for faster, authenticated downloads)
|
||||
without scrolling. (#687, #688)
|
||||
- **First-run setup is calmer and surfaces the best models for your machine.**
|
||||
Dimmed and tightened the setup descriptions (less wordy, more compact). The
|
||||
"Models & engines" step now shows the **platform-tuned** optional models up-front
|
||||
with a green "recommended" tag and their catalog note — e.g. MLX Whisper on
|
||||
Apple Silicon, CUDA-tuned variants on NVIDIA — instead of burying every optional
|
||||
model behind the fold (the universal long tail still folds).
|
||||
|
||||
- **Donations now go through Ko-fi or PayPal (GitHub Sponsors removed).** GitHub
|
||||
Sponsors isn't available, so the Support page no longer routes there: pick an
|
||||
amount (now $10 / $20 / $50) and then choose Ko-fi or PayPal — PayPal carries
|
||||
the amount straight into checkout. `.github/FUNDING.yml` and the README badges
|
||||
were updated to match.
|
||||
- **Simplified the Commercial License page.** Trimmed the six-tile benefit grid
|
||||
and FAQ down to the three things that actually drive the decision (you own the
|
||||
output, no per-minute cost, direct support) plus one clear "request a quote"
|
||||
contact — less wall-of-text, faster to act on.
|
||||
- **Model downloads are faster out of the box.** The built-in multi-connection
|
||||
(segmented) downloader — parallel byte-ranges with live speed/ETA — is now on
|
||||
by default, so the legacy-LFS path is no longer single-stream and slow. It
|
||||
falls back to the normal download on any error, so it can never compromise a
|
||||
correct install (`OMNIVOICE_SEGMENTED_DOWNLOAD=0` to disable). (#669)
|
||||
- **The Hugging Face token is now front-and-center on first-run.** Was a
|
||||
collapsed "advanced" fold almost nobody opened; it's now a prominent card right
|
||||
above Continue, framed around what it actually buys you — authenticated, faster,
|
||||
more reliable downloads (higher rate limits, fewer stalls) — with a one-click
|
||||
"get a free token" link. (#657, #669)
|
||||
### Fixed
|
||||
|
||||
- **Bug reports redact more secrets and every Windows username casing.** The
|
||||
opt-in bug-report scrubber now catches more credential shapes (JWT/Bearer,
|
||||
Google, Slack, AWS keys, and `?token=`/`?api_key=` URL secrets), redacts
|
||||
Windows home paths regardless of `Users`/`users` casing, and stops a superstring
|
||||
username (`/Users/john` vs `/Users/johnny`) from leaking a fragment. The
|
||||
prefilled-issue URL is now bounded by its *encoded* length so a large report
|
||||
can't silently truncate. Nothing new leaves the machine — this only makes the
|
||||
existing local-first, user-reviewed report stricter. (#856)
|
||||
|
||||
- **A hung TTS generate can no longer brick the backend ("Can't reach the local
|
||||
backend").** A GPU job that wedges on some Windows + CUDA setups occupies its
|
||||
worker forever — Python can't cancel the thread — so on the 1–2 worker pools we
|
||||
ship, one stuck job starved every other request and the next action surfaced as
|
||||
the misleading "Can't reach the local backend" even though the process was
|
||||
alive. ASR/dub/model-load already bounded and reset the pool on hang (#730); but
|
||||
**every generate path** — Studio synthesis, the streaming path, batch, the dub
|
||||
per-segment + preview render, archetype previews, and the OpenAI-compatible
|
||||
`/v1/audio/speech` API — was still an unguarded GPU dispatch, and the residual
|
||||
reports all failed on `generate:start (audio)`. Every one is now bounded by the
|
||||
same wall-clock guard (`OMNIVOICE_GENERATE_TIMEOUT_S`, default 300s) that
|
||||
abandons the wedged worker and rebuilds the pool, so capacity is restored
|
||||
automatically and you get an actionable timeout instead of a dead backend.
|
||||
Closes the whole class of GPU-job-hang reports (#851 — #850, #802, #755, #723,
|
||||
#721, and the 0.3.7 cohort, all tracked in #730).
|
||||
|
||||
- **An unsupported GPU now falls back to CPU instead of 500-ing every generate.**
|
||||
When the installed PyTorch build has no kernels for your GPU's compute
|
||||
capability — a too-old card (Pascal / GTX 10-series) or a too-new one
|
||||
(Blackwell RTX 50-series on pre-cu128 wheels) — CUDA failed at launch with the
|
||||
cryptic `CUDA error: no kernel image is available for execution`. The backend
|
||||
now detects that up front and runs on CPU (slower, but it works), and any raw
|
||||
occurrence is reported as "your GPU isn't supported — switch to CPU or install a
|
||||
matching PyTorch," not a Flush-the-memory dead end. Force the GPU anyway with
|
||||
`OMNIVOICE_FORCE_CUDA=1`. (#756)
|
||||
|
||||
- **The "TRANSLATION FAILED" banner now dismisses and clears itself.** The Dub
|
||||
translation-error banner used to be sticky — it survived a successful re-try and
|
||||
never went away. It now has a close (×), auto-clears on the next corrective
|
||||
action (re-translating, changing the engine, or installing the package), and
|
||||
self-clears after a short timeout — fixing the whole class of translate/pipeline
|
||||
banners that outlived the state that caused them.
|
||||
|
||||
- **Dubbing a video URL no longer fails with "ffmpeg is not installed."** yt-dlp
|
||||
downloads video and audio as separate streams and muxes them with ffmpeg, but
|
||||
it only looked on PATH — so on Windows (where OmniVoice's ffmpeg is a bundled
|
||||
sidecar / `imageio-ffmpeg` binary off PATH) the merge aborted before the dub
|
||||
could start. yt-dlp is now pointed at the same ffmpeg OmniVoice resolves. (#712)
|
||||
- **A synth that succeeded no longer 500s because of a history-logging hiccup.**
|
||||
If the local database somehow missed schema init, recording the clip to
|
||||
generation history failed with *"no such table: generation_history"* and
|
||||
surfaced as a 500 — even though the audio had already been generated and saved.
|
||||
The write now self-heals the schema and retries, and a history-logging failure
|
||||
never fails the generation: you get your audio regardless. (#710)
|
||||
- **Long-video dubs no longer spike RAM during assembly.** Dub generation used
|
||||
to hold every segment's audio in memory until the whole track was mixed, so a
|
||||
50-video batch or a single feature-length dub could exhaust RAM and crash. Each
|
||||
segment now streams to disk as it's rendered and the final track is assembled
|
||||
from those files via a 30s-chunk memmap writer, keeping memory flat regardless
|
||||
of video length. Per-segment download WAVs and the final track stay correctly
|
||||
watermarked (marked once at synthesis, no double-mark), and zero/negative-length
|
||||
segments no longer crash the run. (#639)
|
||||
- **A corrupt or wrong-architecture native component no longer masquerades as
|
||||
"out of memory."** A synth failure caused by a bad `.dll`/`.pyd`/`.exe` on
|
||||
Windows (`[WinError 193] %1 is not a valid Win32 application` — e.g. torch,
|
||||
ffmpeg, or an engine binary) was labelled *"ran out of memory — try Flush,"*
|
||||
sending users down the wrong path. It now says the component is corrupt or
|
||||
built for the wrong architecture and to reinstall/repair it. (#705)
|
||||
- **A "[Errno 32] Broken pipe" mid-generation no longer poses as "out of
|
||||
memory."** When the desktop app that launched the backend closes or relaunches,
|
||||
the backend's output pipe breaks and a synth can fail with `[Errno 32] Broken
|
||||
pipe`. That was labelled *"ran out of memory — try Flush,"* which never helps;
|
||||
it now tells you the backend lost its pipe and to restart the app. (#715)
|
||||
- **Settings content no longer sprawls or spills out of view.** The content
|
||||
column capped at 1280px, so on wide windows rows stretched edge-to-edge with a
|
||||
big empty gap between each label and its control ("too spread out"), and a few
|
||||
panels (API keys, the shared button rows, appearance scale) used rigid pixel
|
||||
widths that pushed controls past the card's padding on narrow content. Now the
|
||||
content sits at a readable measure (a single `--settings-measure` token), the
|
||||
shared button/badge rows wrap instead of overflowing, rigid widths can shrink,
|
||||
and rows decide whether to sit side-by-side or stack based on their **actual**
|
||||
width (a container query) — not the viewport, which the 168px nav rail skews.
|
||||
Everything stays inside its padding, edge to edge, on every width. (#696)
|
||||
- **File drag-and-drop works on macOS again.** The app's drop zones use HTML5
|
||||
file drops, but Tauri intercepts OS drag-and-drop by default (`dragDropEnabled`)
|
||||
and swallowed the files before the webview saw them — most visibly on macOS
|
||||
WKWebView, and fully broken on macOS 26 (Tahoe), where dropping a file did
|
||||
nothing. Disabled the interception so the webview handles native HTML5 drops
|
||||
on every platform. (#700)
|
||||
- **A misconfigured `OMNIVOICE_MODEL` no longer bricks model load with a 500.**
|
||||
A stale or leaked TTS *engine id* (e.g. `omnivoice`) reaching the model loader
|
||||
used to fail every launch with *"omnivoice is not a local folder and is not a
|
||||
valid model identifier."* It now self-heals — only a real HF repo id
|
||||
(`org/repo`) or an explicit local path is honored; anything else falls back to
|
||||
the default with a logged warning. Every consumer of the setting routes through
|
||||
the same resolver, so a bad value also can't silently disable model warm-up,
|
||||
mislabel the Settings checkpoint, or get baked into an exported persona bundle.
|
||||
(#693)
|
||||
- **ASR no longer crashes the dub/transcribe preflight when CTranslate2's native
|
||||
library can't load.** On hardened kernels / newer glibc (e.g. WSL2) the
|
||||
CTranslate2 `.so` is rejected with *"cannot enable executable stack"* — an
|
||||
OSError the WhisperX/faster-whisper checks didn't catch, so it took down the
|
||||
whole preflight. They now report the engine as unavailable and auto-detect
|
||||
falls back to PyTorch-Whisper instead of dead-ending. (#692)
|
||||
- **A wedged transcription can no longer take the whole backend offline ("Can't
|
||||
reach the local backend").** On some Windows + CUDA setups a whisperx/CTranslate2
|
||||
transcribe hangs hard and never returns. Because ASR shares a small (1–2 worker)
|
||||
GPU pool with TTS, one stuck worker starved every other request — so the next
|
||||
thing you did (often a TTS *generate*) failed with "can't reach backend" even
|
||||
though the process was alive. Two fixes: every transcribe path — whole-file
|
||||
(dub whole-file, batch, live dictation) **and** the chunked dub stream — is now
|
||||
wall-clock **bounded** like the dub QC / dictation / OpenAI paths already were;
|
||||
and on timeout the poisoned GPU worker is **abandoned and the pool rebuilt**, so
|
||||
capacity is restored without restarting the app. You still get an actionable
|
||||
message (Flush VRAM / pick a smaller ASR model) for the durable fix. (#730)
|
||||
- **The stale-dub-session recovery now also covers the first upload/ingest, not
|
||||
just retry/import.** A dubbing job that vanished server-side during the initial
|
||||
transcribe flow showed the scary *"Job not found … report a bug"* toast; it
|
||||
now resets gracefully and invites a fresh upload, like the other paths. (#695)
|
||||
- **In-app preview of finished audiobooks/stories now plays on Windows.**
|
||||
The preview decoded the entire render into one in-memory PCM buffer via Web
|
||||
Audio `decodeAudioData`, which fails on long-form `.m4b`/AAC under WebView2
|
||||
(`EncodingError: Unable to decode audio data`), and the blob-URL fallback can't
|
||||
play in a Tauri `<audio>` element — so nothing played. The fallback now uploads
|
||||
to the preview endpoint (ffmpeg-extracts a streamable WAV) and plays the HTTP
|
||||
URL, the same path video previews use. Short TTS previews are unchanged. (#653)
|
||||
|
||||
- **First-run setup splash no longer shows a raw `bootstrap.lines` key in English.**
|
||||
The log-line counter string was present in 4 locales but missing from the `en`
|
||||
reference, so English (and 16 other locales falling back to it) rendered the
|
||||
literal key instead of "{{count}} lines". Added it to `en`. Also removed 160
|
||||
dead `gallery.cat_*` keys (renamed to `archetypes.use_*` long ago) orphaned
|
||||
across 20 non-English locales, clearing the i18n orphan-key advisory.
|
||||
|
||||
- **Backend no longer hangs on startup (unreachable, no error) on Apple-Silicon Macs.**
|
||||
The MCP session manager could hang on its anyio task group during lifespan
|
||||
startup (observed on M1, #632); because that start was awaited before the server
|
||||
began serving, "Application startup complete" never fired and the whole backend
|
||||
was unreachable. The MCP start is now timeout-bounded (`OMNIVOICE_MCP_START_TIMEOUT_S`,
|
||||
default 30s) — a hang becomes a logged warning and the backend serves normally
|
||||
without MCP, instead of wedging. (#632)
|
||||
|
||||
- **Dubbing a URL no longer fails with `[Errno 22] Invalid argument` on Windows.**
|
||||
yt-dlp stamps the downloaded file's modified-time with the video's upload
|
||||
date; an out-of-range/invalid timestamp makes the `os.utime` call raise
|
||||
`[Errno 22]` and aborts the whole URL ingest. OmniVoice downloads to a throwaway
|
||||
file and never uses its mtime, so it now skips the stamp entirely
|
||||
(`updatetime=False`). (#642)
|
||||
|
||||
- **Dubbing a YouTube link that 403s now retries with a different player
|
||||
client.** Some videos serve their formats signature-protected to the default
|
||||
player client, so the media download fails with `HTTP Error 403: Forbidden`
|
||||
even though extraction worked — and a plain retry keeps 403ing. The URL
|
||||
download now escalates the YouTube player client (tv → android → web_safari)
|
||||
on a 403, which commonly bypasses it, before surfacing the actionable error.
|
||||
(#625)
|
||||
- **A synth glitch that produced unreadable audio is now caught instead of a
|
||||
misleading "out of memory".** A numerical glitch in the model (seen on Apple
|
||||
Silicon/MPS) could leave NaN/∞ samples, which wrote a WAV that then failed
|
||||
decoding with an opaque `ffmpeg returned error code: 183 / Invalid data` — and
|
||||
the generic error handler labelled it "ran out of memory". Non-finite samples
|
||||
are now sanitized to silence before any encode (so the WAV is always
|
||||
decodable), and a genuine decode failure is reported as "unreadable audio —
|
||||
Flush and regenerate", not OOM. (#629)
|
||||
- **A silent startup hang now leaves a diagnostic instead of nothing.** On some
|
||||
setups the backend could load all model weights and then hang forever before
|
||||
"Application startup complete" — no error, no crash, an unusable app (reported
|
||||
as a Mac M1 hang after `Loading weights: 527/527`, #632). A startup watchdog
|
||||
now dumps every thread's stack to the error log if startup stalls past a
|
||||
window (default 5 min, `OMNIVOICE_STARTUP_WATCHDOG_S` to tune, `0` to disable),
|
||||
so the deadlock is captured rather than invisible. It's disarmed the instant
|
||||
startup finishes, so a normal (even slow-first-download) boot never trips it.
|
||||
(#632)
|
||||
- **First-run demo voice is back.** The bundled demo clip
|
||||
(`backend/assets/samples/demo_voice.wav`) was a build artifact that never got
|
||||
committed, so it shipped absent — onboarding logged "Demo audio not found" and
|
||||
seeded nothing, leaving a brand-new install with an empty Launchpad and no
|
||||
`/demo_audio` route. The clip is now committed (it's already un-ignored and
|
||||
bundled via the Tauri `backend` resource), so first-run seeds the demo voice
|
||||
on every platform; onboarding still degrades gracefully (with a regenerate
|
||||
hint) if it's ever absent. (#621)
|
||||
- **Multi-speaker dubbing: two speakers' turns merged onto one line are now
|
||||
split apart.** Segmentation groups words into sentences *before* diarization
|
||||
runs, so a back-and-forth exchange could land in a single segment; the speaker
|
||||
pass then only *relabelled* that segment with its majority speaker, losing the
|
||||
turn boundary (the second half of #486; the per-speaker voice auto-assign was
|
||||
fixed earlier in #490). A new post-diarization pass re-splits any segment whose
|
||||
words span more than one speaker at the word-level boundary, assigning each
|
||||
piece its own speaker. Single-speaker segments pass through **byte-for-byte
|
||||
unchanged**, so single-speaker dubs and their timing never move, and a lone
|
||||
mis-attributed word (diarization noise) is smoothed rather than causing a
|
||||
spurious split. (#486)
|
||||
- **Designed voices saved with a bad style no longer render wrong or crash
|
||||
generation.** A designed voice could persist an `instruct` the engine
|
||||
validator rejects — either the literal `"[object Object]"` from an old build,
|
||||
or freeform prose typed into the style field — which made every generation or
|
||||
dub that used the voice fail with `Unsupported instruct items found in …`
|
||||
(surfacing to users as a 400/500 and, when it tore down mid-render, "Can't
|
||||
reach the local backend"). The previous fix only *blanked* `"[object Object]"`,
|
||||
which silently dropped the design — so an Indonesian **female** voice came out
|
||||
**male**. Now the stored instruct is sanitized down to valid tags at every
|
||||
seam (save, edit, and when a profile drives Generate or Dub), and when the
|
||||
stored value is unusable the tags are **rebuilt from the design's saved
|
||||
category picks (`vd_states`)** so the intended gender/age/pitch/accent survive.
|
||||
A migration (0007) heals existing poisoned profiles in place — no reinstall,
|
||||
no manual fix. (#550 #571 #594 #596)
|
||||
- **"Transcribe stream dropped … Likely ASR backend failed to load" now shows
|
||||
the *real* reason.** When transcription failed to load its ASR model (the
|
||||
reported case was WhisperX on Windows — typically a faster-whisper /
|
||||
CTranslate2-cuDNN mismatch, a missing model download, or the torch-2.6
|
||||
weights-only VAD regression), the UI dead-ended on a generic "stream dropped"
|
||||
message with no actionable cause. Two root causes: (1) WhisperX loads lazily
|
||||
*inside* transcription, so the load failure was buried in per-chunk errors and
|
||||
retried on every chunk; the transcribe pre-flight now eagerly loads the ASR
|
||||
model (new `ASRBackend.ensure_loaded()`), surfacing the genuine cause once, up
|
||||
front, as a structured error. (2) Pre-flight and audio-load errors closed the
|
||||
SSE stream with a bare `error` and no terminal `done`, so the browser's native
|
||||
EventSource connection-drop could race and win against the structured error —
|
||||
discarding the real cause and falling back to the generic message; every
|
||||
terminal error now emits `done`, and the frontend latches the structured cause
|
||||
so a connection drop can't overwrite it. Net: WhisperX load failures are
|
||||
diagnosable instead of a silent dead-end. Fail-before/pass-after regression
|
||||
test included. (#578)
|
||||
- **Dubbing: the PLAY button on the dubbed-video preview did nothing.** Same
|
||||
autoplay-policy trap that #510 fixed for the standalone audio player, but the
|
||||
dub editor's timeline player was missed. WaveSurfer builds its `AudioContext`
|
||||
at mount — before any user gesture — so on Windows WebView2 (and Linux
|
||||
Firefox/Chrome, Android Chrome) it stays `"suspended"`; `playPause()` then
|
||||
resolves with no sound and the preview just sits there. Every playback entry
|
||||
point in the dub timeline (the toolbar Play button and the per-segment "play
|
||||
this slot") now resumes the context via the shared `unlockAudio()` on the
|
||||
click before starting playback, and swallowed play() rejections are logged
|
||||
instead of hidden. A source-contract regression test pins the invariant so a
|
||||
future refactor can't quietly reintroduce a silent play path. macOS is
|
||||
unaffected (its context was never blocked). (#595)
|
||||
- **Voice design: the script text field couldn't be expanded.** The Script
|
||||
textarea was a `flex: 1` item inside a flex column, so flex-grow recomputed
|
||||
its height on every reflow and snapped the user's drag back — `resize:
|
||||
vertical` is silently ignored on a flex-grown item in Chromium/WebView2. The
|
||||
field now owns its own height (starts taller, and the corner grip grows it
|
||||
reliably on every platform). (#595)
|
||||
- **An interrupted model download now self-repairs instead of dead-ending.**
|
||||
When the OmniVoice TTS cache was missing weight shards (the usual aftermath of
|
||||
an interrupted first download), the next synthesize failed with a 500 and a
|
||||
"delete the model and install it again" instruction — a manual dead-end. The
|
||||
backend now detects the truncated-cache error on load, re-fetches just the
|
||||
missing files via `snapshot_download` (already-present blobs are skipped, so a
|
||||
near-complete cache repairs in seconds and a healthy cache is never touched),
|
||||
and retries the load automatically. Offline mode (`HF_HUB_OFFLINE`) is
|
||||
respected — repair never makes a network call the user opted out of — and if
|
||||
the re-fetch still can't fix it, the actionable delete-and-reinstall message
|
||||
is preserved as the fallback. (#581) The repair now also **retries** the
|
||||
re-fetch (3 attempts, resuming each time) so a single transient blip — the very
|
||||
thing that interrupts a download in the first place — doesn't bounce you back
|
||||
to a manual reinstall; tune with `OMNIVOICE_MODEL_REPAIR_RETRIES`. And if a
|
||||
resume-repair still won't load — the signature of a *corrupt* file that kept
|
||||
its size, which a resume trusts and never re-fetches — it now **force
|
||||
re-downloads** the model files once before giving up, so even a bit-rotted
|
||||
cache self-heals without a manual reinstall. (#739)
|
||||
- **Dubbing a YouTube URL no longer dies on a transient "Broken pipe."**
|
||||
Pasting a video link could fail outright with `download: Unable to download
|
||||
video: [Errno 32] Broken pipe` — a broken pipe raised while the write side of
|
||||
a pipe closes mid-stream (a killed ffmpeg merge child, a CDN reset during
|
||||
muxing). yt-dlp's own per-fragment retries don't cover that case, so a single
|
||||
transient blip aborted the whole ingest. The URL download now retries up to
|
||||
twice on broken-pipe / network-drop failures, wiping the partial download
|
||||
between attempts, and only surfaces the (already-actionable) "connection
|
||||
dropped — just retry" hint after the retries are exhausted. Unsupported links
|
||||
still fail fast with their own hint — no wasted retries. (#579, #598)
|
||||
- **`No module named 'omnivoice'` on installs whose venv lost its editable
|
||||
record.** An interrupted or offline `uv sync` (common during an in-place
|
||||
upgrade) could install all dependencies yet never lay the editable install of
|
||||
the project's own `omnivoice` package — or an antivirus quarantine could
|
||||
remove it. The venv still started uvicorn, so the bootstrap's health gate
|
||||
passed it through, and the app only failed at the first generate/dub with
|
||||
`No module named 'omnivoice'`. The bootstrap now also verifies `omnivoice` is
|
||||
importable (via a cheap `find_spec`, no torch load) and forces a repair
|
||||
`uv sync` that re-lays the editable install when it isn't; the backend also
|
||||
resolves `omnivoice` from its bundled source tree at runtime as a safety net.
|
||||
No reinstall needed — relaunch and it self-repairs. (#564)
|
||||
- **"cannot schedule new futures after shutdown" no longer breaks generate/dub
|
||||
after a slow first load.** When a model load timed out, the backend reset its
|
||||
GPU worker pool to recover — but several request handlers had captured the old
|
||||
pool object at import time and kept submitting to it, so every subsequent
|
||||
generate, dub, transcribe, or translate failed with `cannot schedule new
|
||||
futures after shutdown` (a 500, or "Can't reach the local backend" when it
|
||||
took the worker down). The GPU pool is now a single self-healing handle whose
|
||||
worker pool is rebuilt on demand, so a reset can never strand an in-flight or
|
||||
later request. No settings change; the recovery is automatic. (#589 #599)
|
||||
- **Transcription / dubbing works on Windows again.** WhisperX failed to load on
|
||||
Windows because speechbrain's guard that suppresses stray optional-integration
|
||||
imports used a POSIX-only path check, so a `k2_fsa` import error aborted the
|
||||
whole transcription. Fixed cross-platform — covers the entire class of optional
|
||||
integrations, not just k2. (#630 #611 #647)
|
||||
- **A slow transcription no longer looks like a dead backend.** Whole-file
|
||||
transcribe paths (dub QC, dictation, OpenAI-compat) ran unbounded, so a
|
||||
VRAM-starved `large-v3` could spin for minutes and hold a GPU worker — surfacing
|
||||
as "Can't reach the local backend". They're now time-bounded and return a clear,
|
||||
actionable 504 (free VRAM / pick a smaller ASR model / use CPU) instead of
|
||||
hanging. New troubleshooting section documents it. (#656)
|
||||
- **Windows preview playback fixed.** The audiobook/clone preview's streaming
|
||||
fallback fetched `localhost`, which on Windows resolves to IPv6 and missed the
|
||||
IPv4-only backend — so previews failed with "decode error" / "no supported
|
||||
sources". The preview API now targets `127.0.0.1` (matching the main client),
|
||||
and the expected decode→stream fallback is logged calmly instead of as a scary
|
||||
error. (#653 #659)
|
||||
- **A stale dub session resets cleanly instead of erroring.** Reopening the Dub
|
||||
tab after the backend restarted tried to resume a job that no longer existed and
|
||||
surfaced "Job not found" as a bug-report error. It now quietly clears the dead
|
||||
session and invites a fresh upload. (#660)
|
||||
- **A bad voice-style instruct is a clear 400, not a scary 500.** Typing free-form
|
||||
prose (or a non-English description) into the style/instruct field returned a
|
||||
500 telling you to Flush for memory you never ran out of; it now returns a clean
|
||||
400 that lists the valid style tags. The Voice Clone UI also drops unrecognized
|
||||
style text locally and generates anyway. (#664 #612)
|
||||
- **The ⊕ Insert token popover stays on screen.** On Voice Clone it could grow
|
||||
tall enough to clip off the top of the window; it's now a compact, scrollable
|
||||
box anchored above the button. (#672)
|
||||
- **First-run no longer hangs on Apple Silicon.** The MCP session-manager startup
|
||||
is now timeout-bounded so a slow/stuck mount can't wedge the whole backend boot
|
||||
on M1. (#632)
|
||||
|
||||
### CI
|
||||
|
||||
- **Feature-coverage test system.** A backend route-inventory test diffs all 213
|
||||
HTTP/WebSocket endpoints against a committed snapshot (plus a critical-endpoint
|
||||
guard and a route-count floor), and a frontend feature-coverage test asserts
|
||||
every app mode is wired to a page and every feature has its i18n namespace — so
|
||||
an endpoint or page silently disappearing now fails CI on every PR.
|
||||
- **`bun desktop` no longer kills its own dev backend.** The dev launcher runs the
|
||||
API and the Tauri app side-by-side, but the app's backend manager would "take
|
||||
ownership" of port 3900 and kill the API the moment it booted (before it was
|
||||
healthy), tearing the whole session down. The dev app now sets
|
||||
`TAURI_SKIP_BACKEND` so it attaches to the running API instead of fighting it —
|
||||
production launch is unaffected. (#745)
|
||||
|
||||
## [0.3.7] — 2026-06-20
|
||||
|
||||
A stabilization release that clears the wave of issues reported on the 0.3.6
|
||||
line — across voice design, dubbing, transcription, install, and the Linux/web
|
||||
UI — and lands two more opt-in cloning engines. The throughline is **non-English
|
||||
correctness and cross-platform playback**: cloned and designed voices now hold
|
||||
their language end-to-end, and audio plays inline in Linux/Android browsers,
|
||||
not just macOS. It also carries the v0.3.6 startup-crash fixes, so anyone still
|
||||
hitting "Can't reach the local backend" on v0.3.5/v0.3.6 only needs to update.
|
||||
|
||||
### Added
|
||||
|
||||
- **Two opt-in heavyweight TTS engines: MOSS-TTS-v1.5 (8B) and dots.tts (2B).**
|
||||
Both are zero-shot voice-cloning engines, each running in its own isolated
|
||||
subprocess venv (they pin a `transformers` version that conflicts with the
|
||||
parent's `>=5.3` — MOSS `==5.0`, dots.tts `==4.57`) via the same dedicated-venv
|
||||
pattern as IndexTTS-2, so they can't disturb the default install or its
|
||||
lockfile. Point `OMNIVOICE_MOSS_TTS_V15_DIR` / `OMNIVOICE_DOTS_TTS_DIR` at a
|
||||
local clone to enable. CUDA/CPU only — neither claims Apple-Silicon MPS, and
|
||||
dots.tts is gated off on Windows (upstream is Linux/macOS only). See
|
||||
[docs/engines/moss-tts-v15.md](docs/engines/moss-tts-v15.md) and
|
||||
[docs/engines/dots-tts.md](docs/engines/dots-tts.md). (#498)
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Non-English voices drifted to English / the wrong language.** Three
|
||||
independent root causes, all in the language path: (1) a voice profile's
|
||||
stored language was never read back into generation, so a German archetype
|
||||
that *previewed* in German *generated* in English (the preview passed the
|
||||
language; the user's Generate call didn't); (2) the audiobook/longform synth
|
||||
hardcoded `language=None`, letting the engine re-autodetect per chunk so a
|
||||
non-English clone could flip language mid-render on short/ambiguous lines; and
|
||||
(3) the duration estimator weighted Unicode combining marks at zero, so
|
||||
decomposed (NFD) diacritic text — common for Vietnamese — under-allocated
|
||||
frames and came out rushed. The profile/request language is now threaded
|
||||
through both the single-shot and longform paths (request wins, profile fills
|
||||
the gap), and text is NFC-normalized before duration estimation. Each fix has
|
||||
a fail-before/pass-after regression test. (#533, #505, #502)
|
||||
- **Audio playback on Linux Firefox/Chrome and Android Chrome.** Two separate
|
||||
root causes both masquerade as "the play button doesn't work" on non-macOS
|
||||
browsers — and both are invisible when developing on macOS, which is why they
|
||||
shipped. (1) The backend served `.wav` / `.flac` with Python's default
|
||||
`audio/x-wav` / `audio/x-flac` (vendor-experimental, never IANA-registered);
|
||||
macOS CoreAudio MIME-sniffs leniently and plays anyway, but Linux FFmpeg and
|
||||
Android ExoPlayer strictly honor the declared type and prompt to download.
|
||||
Fixed by registering the canonical `audio/wav` / `audio/flac` types before
|
||||
any `StaticFiles` mount. (2) WaveSurfer's `AudioContext` is constructed at
|
||||
component-mount time — i.e. before any user gesture — so on Linux FF/Chrome
|
||||
and Android Chrome it stays `suspended`, `decodeAudioData` hangs, the
|
||||
`ready` event never fires, and the play button never enables. macOS
|
||||
Safari/Chrome auto-resume on first interaction. Fixed by patching
|
||||
`window.AudioContext` to track every instance and resuming them on the first
|
||||
`pointerdown` / `keydown` / `touchstart`, plus resuming inline on the play
|
||||
click itself. The MIME fix has a backend regression test; the unlock path
|
||||
has a Vitest unit test covering idempotency, post-unlock contexts, and
|
||||
error isolation. (#510)
|
||||
- **Voice Studio "Save design as profile" poisoned the profile with
|
||||
"[object Object]" and then 400'd every generation** ("Unsupported instruct
|
||||
items found in [object Object]"). The save passed the instruct *builder
|
||||
object* to the form instead of its string. Fixed at the source + defended with
|
||||
a coercion helper; the engine now tolerates the sentinel, and a migration
|
||||
heals already-saved profiles. (#550, #545, #542, #537, #530, #525)
|
||||
- **Profile / persona / consent endpoints 500'd with `no such column:
|
||||
consent_audio_path`** (and the same class for `kind`/`vd_states`/…) after an
|
||||
in-place upgrade. The alembic migration existed but couldn't always apply
|
||||
(stamped at a removed revision, or alembic not importable) and the failure was
|
||||
swallowed. The runtime schema now self-heals — it ADDs any missing additive
|
||||
column from the canonical schema on startup. (#552, #547)
|
||||
- **Stories: the global reading-speed slider was ignored by preview and stem
|
||||
export.** The #415 global speed only flowed through the full longform export;
|
||||
per-segment preview and stem export still resolved a hardcoded `track.speed ||
|
||||
1.0`, so audio played at 1.0× even with the global set to e.g. 0.70×. A shared
|
||||
`effectiveSpeed(track, global)` helper (per-line override → global → engine
|
||||
default) now drives all three generation paths. (#508)
|
||||
- **Generate / Settings / Clone buttons were missing / unpressable on Linux.**
|
||||
The UI-scale fix round-trips correctly on Chromium, but older WebKitGTK treats
|
||||
`zoom` as a layout no-op, leaving a ~23% black band that pushed the bottom CTAs
|
||||
off-screen. The shell now probes the engine and fills the window when `zoom`
|
||||
doesn't lay out. (#523, #524)
|
||||
- **Settings tabs with little content rendered as a stunted box in a black
|
||||
void** (reported on Appearance). The page is now a flex column with a
|
||||
min-height floor — short tabs fill the panel, tall tabs grow and scroll
|
||||
exactly as before. The Appearance panel's previously hardcoded English
|
||||
strings ("UI scale", "Color theme", "Font") were also routed through i18n,
|
||||
per the localization rule. (#507)
|
||||
- **The engine "Install" button 500'd with "No virtual environment found."**
|
||||
`uv pip install` now targets the running interpreter (`--python
|
||||
sys.executable`) instead of relying on a venv it couldn't auto-discover.
|
||||
(#529, #527)
|
||||
- **Transcription failed with "no segments" on GPUs without efficient float16.**
|
||||
Both CTranslate2 ASR backends now fall back float16 → int8 instead of crashing
|
||||
at model load; a transcribe stream can no longer close without a terminal
|
||||
error event; and an incomplete `transformers` install reports an actionable
|
||||
message instead of "Could not import module 'AutoFeatureExtractor'".
|
||||
(#551, #549, #516)
|
||||
- **Audiobook import 500'd** with `'AudiobookPlan' object has no attribute
|
||||
'chapter_count'` for every format (.txt/.md/.epub/.pdf). (#543)
|
||||
- **Windows: generated audio auto-played in a separate, un-closeable black
|
||||
window.** Renders now play in-app through the shared playback manager. (#532)
|
||||
- **Cryptic video-download errors** now carry actionable hints: an unsupported
|
||||
link shape ("paste a direct video page, not a share/feed link") vs a transient
|
||||
network drop ("just retry — the partial download was cleaned up"). (#554, #536)
|
||||
- **A relocated, copied, or restored backend venv ("No module named
|
||||
'encodings'") now self-heals** (rebuilds once) instead of failing on every
|
||||
launch.
|
||||
- **The donate goal bar showed fabricated progress** ($137.50 / $200, 23
|
||||
sponsors). It now reflects the real figures ($10 / $200, 1 sponsor) in both the
|
||||
runtime JSON and the TypeScript fallback. (#513)
|
||||
- The **"Can't reach the local backend" startup-crash wave** (pkg_resources
|
||||
#248, `scalar_fastapi` #307, exit-106 broken venv) was fixed in v0.3.6 — this
|
||||
release carries those fixes, so updating from v0.3.5/older resolves them.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Version is now single-sourced from `frontend/package.json`.** Five
|
||||
hand-maintained literals drifting is exactly what shipped a 0.3.6 build that
|
||||
called itself 0.3.5. `package.json` is canonical (vite already injects it as
|
||||
`__APP_VERSION__`), `tauri.conf.json` reads its bundle version from it
|
||||
(`"version": "../package.json"`), and the remaining toolchain-required mirrors
|
||||
(Cargo.toml, pyproject.toml, the frozen-backend fallback) are CI-guarded to
|
||||
stay in lockstep. (#503)
|
||||
- **Updater: the Preview channel actually tracks `main` again.** It was stuck at
|
||||
`0.3.5-41` because its only build trigger was a manual dispatch; a nightly
|
||||
rebuild now enforces "preview = main" (no-opping on days `main` didn't move).
|
||||
Two latent hazards are closed: the `preview` release is re-asserted as a
|
||||
prerelease every run (a non-prerelease preview could hijack the Stable
|
||||
channel's "Latest"), and its manifest can no longer silently drop the
|
||||
Intel-Mac (darwin-x86_64) target. (#500)
|
||||
|
||||
### Internal
|
||||
|
||||
- **The frozen desktop backend reported `0.3.5` regardless of its real version.**
|
||||
In a synced env, `core.version.APP_VERSION` resolves from package metadata
|
||||
(correct, so CI stayed green), but the PyInstaller-frozen build has no
|
||||
`.dist-info`, hit `PackageNotFoundError`, and fell back to a hardcoded literal.
|
||||
The spec now bundles `omnivoice` metadata so the primary path works frozen too,
|
||||
and the resolution chain is metadata → pyproject → named fallback. This also
|
||||
fixes **About → Version rendering blank** in the web/Pinokio build (no Tauri,
|
||||
backend idle), which now falls back to the build-time version. (#501)
|
||||
|
||||
## [0.3.6] — 2026-06-16
|
||||
|
||||
A large release (168 commits since v0.3.5). The headline is the **Longform
|
||||
suite** — produce full audiobooks and multi-voice stories from text, EPUB, or
|
||||
PDF — alongside a real **engine-routing** layer that tells you up front when an
|
||||
engine will fall back to CPU instead of finding out mid-synth. Dubbing,
|
||||
first-run, and install reliability all get a pass too.
|
||||
|
||||
### Added
|
||||
|
||||
- **Longform: Stories + Audiobook editors.** Two new tabs turn long text into
|
||||
finished audio. **Audiobook** takes a script (or imports plain text / EPUB /
|
||||
PDF), auto-splits it into chapters, and renders a chaptered `.m4b` with
|
||||
metadata, cover art, and per-chapter preview/resume. **Stories** is a
|
||||
multi-voice editor — assign a different voice per line, preview, and export
|
||||
the whole thing through the same server-side renderer. Both share one render
|
||||
core (loudness, metadata, cover art) and one live SSE progress stream, and
|
||||
you can convert a project between Story and Audiobook in place.
|
||||
(#402, #403, #404, #408, #409, #411, #412, #413, #426, #435, #436, #447)
|
||||
- **Longform: PDF & EPUB ingest.** "Import" on the Audiobook tab accepts EPUB
|
||||
and PDF (not just plain text) and auto-chapters the result, so an existing
|
||||
ebook becomes an audiobook without manual copy-paste. (#412, #459)
|
||||
- **Longform: two-pass loudnorm mastering.** Audiobook/Story exports now run a
|
||||
measure-then-normalize loudnorm pass for accurate ACX/podcast loudness
|
||||
targets. A slow or broken measure pass degrades gracefully to single-pass
|
||||
rather than aborting the render. (#449, #455)
|
||||
- **Longform: crash-resume.** An interrupted render is resumable without
|
||||
re-submitting the original input — the compiled plan is persisted to the job
|
||||
dir and finished chapters are reused, so a crash mid-book doesn't cost you the
|
||||
whole render. (#470)
|
||||
- **Longform: pronunciation control + SSML-lite prosody.** A per-render
|
||||
pronunciation lexicon (word respelling) plus an in-app pronunciation editor
|
||||
and markup reference, and inline prosody markers — `[slow]` / `[fast]` /
|
||||
`[emphasis]` / `[spell]` — for fine-grained delivery. (#419, #421, #422)
|
||||
- **Stories: global reading-speed control.** A toolbar slider (0.5–2.0×) sets
|
||||
one speed for every line that doesn't have its own per-line override; the
|
||||
per-line slider still wins. Persisted as a UI preference. (#415, #416)
|
||||
- **Unified LongformProject store.** Audiobook metadata, scripts, and prefs
|
||||
persist in a single project store (with a `v4→v5` migration), and finished
|
||||
books/stories now show up alongside other work in **Projects**. (#417, #443,
|
||||
#444)
|
||||
- **Portable personas (`.ovsvoice`).** Export any voice as a self-contained,
|
||||
fully-local persona bundle — identity, optional reference clip, consent
|
||||
attestation, SPDX license, and a watermarked preview — and import it back into
|
||||
another OmniVoice install. A privacy toggle ships a **preview-only** bundle so
|
||||
no raw recording of your voice has to travel. Verified-own-voice status can't
|
||||
be forged by hand-editing a bundle (real recording + consent text + attestation
|
||||
required). Legacy `.omnivoice` files still import. See
|
||||
[docs/persona-format.md](docs/persona-format.md). (#29)
|
||||
- **Engine routing — no more silent CPU fallback.** A host device probe and
|
||||
routing resolver now decide where each engine actually runs, and the verdict
|
||||
is surfaced before you hit Synthesize: the **Settings → Engines** picker shows
|
||||
a per-engine compatibility matrix, and **preflight** / **diagnose** report the
|
||||
active engine's GPU verdict (accelerated / caveat / CPU-fallback /
|
||||
unavailable). At synth time every TTS entry point (`/generate`,
|
||||
`/v1/audio/speech`) enforces the same routing — an engine that can't use this
|
||||
host's GPU returns an explicit error or an `X-OmniVoice-Routing` header instead
|
||||
of silently dropping to CPU or dying mid-synth. (#21)
|
||||
- **Diagnostics suite.** New self-check tooling for when something's wrong: a
|
||||
`/system/diagnose` report (and matching backend `--diagnose`), a persistent
|
||||
**error journal** surfaced in Settings, and a scrubbed **diagnostic bundle**
|
||||
(home dirs stripped to `~/`, no tokens/keys) you can attach to a bug report.
|
||||
Paired with structured GitHub **Issue Forms** (bug / install / feature) for
|
||||
cleaner reports. (#433, #456)
|
||||
- **Dubbing: multi-speaker per-speaker voice assignment.** When diarization
|
||||
detects multiple speakers, each segment is now bound to its speaker's cloned
|
||||
voice automatically instead of landing on "Default" and needing manual fixes;
|
||||
per-segment reference clips are still preferred for quality where present. Also
|
||||
adds an optional speaker-count hint for diarization. (#275, #486, #490)
|
||||
- **Dubbing: Smart Fit timing + second-pass QC.** A Smart Fit timing strategy
|
||||
(planner, fingerprints, per-segment video retime + drift absorption + fitted
|
||||
subtitles) plus a second-pass ASR QC that flags lines whose dub drifts from the
|
||||
target timing — wired into the dub editor UI. Includes a timeline segment
|
||||
editor (drag, snap-to-onset, keyboard a11y), speech-onset alignment, regional
|
||||
dialect targeting, and per-segment clone references. (#280, #347, #350, #369,
|
||||
#370, #458)
|
||||
- **Dubbing: dedicated Dub home.** A projects/history landing for dubbing with
|
||||
project rename. (#435)
|
||||
- **Voice Console workspace.** Clone and Design are consolidated into one Voice
|
||||
workspace with right-side panels, a shared waveform player, an identity recipe
|
||||
line / Active-voice card, and a free-text "describe your voice" field that maps
|
||||
natural language to design parameters. (#317, #374, #376, #378, #395, #396,
|
||||
#397)
|
||||
- **Unified first-run setup.** Nothing installs until you confirm a plan: pick an
|
||||
install mode (installed / portable), a storage location, and (on restricted
|
||||
networks) custom PyPI/HF/python-build-standalone mirrors — with a
|
||||
minimum-free-space gate before anything downloads. Followed by a guided
|
||||
studio-console wizard with platform-aware hints, resume reassurance, and
|
||||
download ETAs. (#286, #295, #297, #298)
|
||||
- **Dictation: local-LLM refinement.** Opt-in local-LLM cleanup of final
|
||||
transcripts (collapsing Whisper hallucination loops), available on both live
|
||||
dictation and the REST `/transcribe` path; plus opt-in NLMS acoustic echo
|
||||
cancellation for dictating over playback. Configure a remote LLM endpoint
|
||||
(Ollama / vLLM / LM Studio) in Settings. (#356, #357, #363, #399, #400, #457)
|
||||
- **Unlimited-length TTS + streaming.** Sentence-boundary chunking with
|
||||
crossfade removes the per-generation length cap, and a new sentence-by-sentence
|
||||
`/ws/tts` streams audio as it's produced. An inline `[pause Nms]` marker
|
||||
inserts measured silence in generated speech. (#276, #357, #358)
|
||||
- **MCP server v1.** OmniVoice mounts an MCP server on `/mcp` (with a stdio shim
|
||||
and per-agent voice binding) so it can act as a local TTS/STT provider for
|
||||
agentic pipelines. (#368)
|
||||
- **Remote-backend access.** Point the desktop UI at a remote backend URL with a
|
||||
bearer key (Tailscale-documented), and an opt-in Hugging Face token field in
|
||||
the setup flow. (#303, #364)
|
||||
- **"Fund Claude Max" support experience.** The donate page gets a real goal bar
|
||||
with a "Join N supporters" social-proof line and suggested amounts, plus Pip
|
||||
the mascot and a non-blocking "postcard" toast that appears only *after* a
|
||||
success (a finished dub, a saved clone, a longform export) — never on errors,
|
||||
setup, or first run — with escalating cooldowns and a one-click "don't ask
|
||||
again". (#494)
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Transcription/dubbing failed when ffmpeg wasn't on `PATH`** (notably on
|
||||
Windows). WhisperX now decodes audio through OmniVoice's own validated ffmpeg
|
||||
binary instead of a bare `PATH` lookup, so ASR works without a system ffmpeg
|
||||
install. (#479)
|
||||
- **Translation defaulted the source language to English.** Dubbing/translation
|
||||
now guesses the source language from the text instead of assuming `en`,
|
||||
fixing wrong-direction translations. (#478)
|
||||
- **Cinematic / LLM dubbing features failed out of the box** because `openai`
|
||||
wasn't bundled. The client is now a runtime dependency, so those paths work on
|
||||
a fresh install. (#484)
|
||||
- **`pkg_resources missing` install dead-end (#248).** The auto-repair ran
|
||||
`uv pip install setuptools`, which `uv` treated as a no-op when setuptools
|
||||
*metadata* was present but its files had been removed (commonly by Windows
|
||||
Defender quarantine or a partial extract). Both repair sites now use
|
||||
`--reinstall` to force re-extraction, and the error/hint text suggests the
|
||||
working command plus an antivirus-exclusion note. (#248)
|
||||
- **A stuck backend trapped users on a buttonless splash (#474).** The bootstrap
|
||||
splash now has a per-stage stall watchdog: if a non-terminal stage sits past
|
||||
its budget (20 min for dep install, 120 s otherwise), it flips to the failed
|
||||
state with actionable hints, the live log, and Retry / Clean-&-Retry — instead
|
||||
of polling forever with no way out. (#474)
|
||||
- **Changing the model-download location in Settings had no effect (#480).** The
|
||||
desktop launcher injected a stale models dir that overrode the per-user value,
|
||||
so new downloads kept going to the old folder and "Effective location" stayed
|
||||
wrong. The per-user env file now wins, so the in-app Settings path is
|
||||
authoritative. (#480)
|
||||
- **Backend crashed on app upgrade with a stale venv (#307).** Dependencies are
|
||||
now synced on upgrade, and a structurally broken venv self-heals instead of
|
||||
exiting `106`. `scalar_fastapi` is now optional so its absence can't break
|
||||
startup. (#307, #314)
|
||||
- **`/generate` ignored the selected TTS engine (#312)** and GGUF speech-control
|
||||
parameters weren't forwarded — both now honored. (#306, #312)
|
||||
- **TTS generation failed on some GPUs.** `torch.compile` failures now fall back
|
||||
to eager execution so generation never hard-fails on unsupported GPUs, and
|
||||
cudagraph-compiled inference is pinned to one dedicated thread to avoid
|
||||
crashes. (#278, #315)
|
||||
- **Re-dub ignored transcript edits (#281).** Fingerprints are canonicalized, the
|
||||
preview cache is busted, and the mux is atomic, so editing the transcript and
|
||||
re-dubbing actually reflects your changes. Translated subtitles now burn in
|
||||
correctly and subtitle save no longer throws a JSON error. (#281, #309)
|
||||
- **macOS: app wouldn't open without using Terminal.** Builds are now ad-hoc
|
||||
signed (with signing/notarization verification), so the app launches normally.
|
||||
(#290)
|
||||
- **macOS dictation auto-paste stole focus**; it now writes the clipboard
|
||||
natively without grabbing focus, and microphone-permission handling adds OS
|
||||
usage descriptions, a WebView grant handler, and an actionable denied-state UI.
|
||||
(#287, #323)
|
||||
- **Clone-reference transcription was broken** (it used a removed transformers
|
||||
pipeline); it now routes through the ASR registry. A crash-isolated
|
||||
faster-whisper subprocess backend keeps an ASR crash from taking down the app.
|
||||
(#308, #393)
|
||||
- **Realtime status probe hit a gated route.** It now probes the auth-exempt
|
||||
`/health` instead of the gated `/model/status`, and the UI polls the backend
|
||||
over HTTP before opening the WebSocket to avoid startup `ECONNREFUSED`. (#439,
|
||||
#450)
|
||||
- **Non-executable or unreachable engine binaries showed cryptic errors** — these
|
||||
now produce actionable messages. (#437, #438, #454, #466)
|
||||
- **Design-profile save was coupled to a TTS render (#476)**, so saving a profile
|
||||
needlessly triggered synthesis; the two are now decoupled. (#476)
|
||||
- **UI scale / black bands.** The app shell now scales via `transform: scale` and
|
||||
always fills the viewport, fixing the WebKitGTK black-band issue on Linux and
|
||||
cramped/black layouts at narrow widths — a permanent fix across platforms.
|
||||
(#445, #452)
|
||||
- **Clone popover/CTA clipping and a non-resizable textarea** are fixed, the
|
||||
WaveformPlayer no longer pauses itself on play or ignores clicks, and several
|
||||
layout/history-display issues (phantom sidebar gap, title clamping, flicker)
|
||||
are cleaned up. (#379, #384, #398, #481)
|
||||
- **Windows: `desktop-prod` now runs from cmd/PowerShell** via a cross-platform
|
||||
launcher, `tqdm` is disabled on non-TTY to avoid an `OSError`, and ffmpeg
|
||||
validation guards against `WinError 193`. (#282, #305, #377)
|
||||
- **MLX import hardened** against PyInstaller dylib failures, with a proper
|
||||
platform gate so it's only loaded where it works. (#390)
|
||||
|
||||
### Changed
|
||||
|
||||
- **Restricted-network support.** A Hugging Face mirror (`HF_ENDPOINT`) setting,
|
||||
custom PyPI / HF / python-build-standalone mirrors in first-run setup, and
|
||||
region presets help installs complete behind restrictive networks. (#286, #391)
|
||||
- **Engine memory management.** Subprocess-engine sidecars now unload on demand
|
||||
and idle-reap to free VRAM. (#401, #406)
|
||||
- **Faster, more accurate model downloads** via a Xet fast path with accurate
|
||||
progress reporting, plus a model-management cleanup pass. (#424, #428)
|
||||
- **Voice profiles unified** under one model with a `kind` discriminator and
|
||||
stored design params, and consent-locked profiles (`verified_own_voice` +
|
||||
spoken-consent flow). (#354, #376)
|
||||
- **Updater** preview channel now offers the newest build across channels, and
|
||||
preview versions carry an MSI-legal numeric pre-release stamp. (#293, #326)
|
||||
- **Performance.** Voice-clone prompt embeddings are cached, and dub retime
|
||||
batches seek to their window instead of decoding from frame 0. (#387, #427)
|
||||
|
||||
### License
|
||||
|
||||
- **Relicensed from FSL-1.1-ALv2 to AGPL-3.0 (open-core).** The project is now
|
||||
under the GNU Affero General Public License v3, with a paid commercial license
|
||||
retained for proprietary/closed-source use without AGPL obligations. The
|
||||
bundled `omnivoice/` TTS model package stays Apache-2.0 upstream
|
||||
(AGPL-compatible). Manifests declare `AGPL-3.0-only`; the in-app Commercial
|
||||
License copy and README are updated, and the old "converts to Apache 2.0 after
|
||||
two years" FAQ is removed. In-app commercial-license strings are translated
|
||||
across all 20 locales. (#292)
|
||||
|
||||
### CI
|
||||
|
||||
- **macOS Intel (x86_64) build target reinstated** on `macos-15-intel`, so Intel
|
||||
Mac users get installers again. (#342)
|
||||
- **Docker Hub publishing.** Images now also publish to Docker Hub
|
||||
(`palashdeb/omnivoice-studio`), with the Docker Hub overview maintained in-repo
|
||||
and auto-synced from `main` (sync is non-fatal so it can't redden a build).
|
||||
(#375, #410, #414)
|
||||
- **Docs-drift guard.** A daily job compares the canonical feature inventory
|
||||
against README / docs / registries to catch stale docs. (#353)
|
||||
- **Security scans never cancel on `main`,** so merge trains no longer leave red
|
||||
✗ on intermediate commits. (#340)
|
||||
|
||||
## [0.3.5] — 2026-06-03
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
|
||||
**OmniVoice Studio**
|
||||
|
||||
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. It's an active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord. The latest stable release is **v0.3.5**; `main` rolls ahead at **v0.3.6** (latest release + 1 patch — see the Versioning rule below).
|
||||
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. Today it's a v0.2.7 active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord.
|
||||
|
||||
**Core Value:** **A first-run that actually works.** A user who downloads the installer (or clones the repo) should reach a working voice-cloning or dubbing output without hitting a wall — and when something does go wrong, the error or docs should tell them exactly what to do.
|
||||
|
||||
@@ -16,7 +16,7 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
|
||||
- **Default features must work on every platform (strict rule, 2026-05-20):** A feature that ships in default mode — out-of-the-box, no user customization, no opt-in toggle — must behave identically on macOS, Windows, and Linux. Platform-specific *implementation code* is allowed for OS APIs / shells / packaging, but the user-visible *default behavior* cannot diverge. Platform-only features (e.g., a macOS-only global shortcut, a Windows-only path picker) must go behind explicit user opt-in: Settings toggle, env var, or CLI flag. When a default doesn't work on a platform, that's a P0 bug — either fix it on the missing platform or move it behind opt-in. No third option.
|
||||
- **Backward-compatible project data**: Existing `omnivoice_data/` (user voices, projects, settings) must keep working without manual migration. Any DB schema change goes through alembic with a tested upgrade path.
|
||||
- **Local-first guarantee preserved**: Auto bug reporting (new addition) must be **opt-in**, must submit only to GitHub Issues (no third-party telemetry endpoint), and the app must remain fully functional with reporting disabled. No required cloud calls, accounts, or API keys.
|
||||
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** the v0.3.x line has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main; the owner tags a patch (`v0.3.Z`) from main whenever the current state is worth cutting. No `-rc` tags. No phased release. No `v0.4` deferrals while the v0.3.x line is open — every open issue and every open community PR gets absorbed into the v0.3.x line or explicitly declined. Users follow `main` for previews; users wanting stable stay on the latest tagged release (currently **v0.3.5**). ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
|
||||
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** v0.3.0 has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main. Tag `v0.3.0` once when the user calls "actually useful" — a qualitative bar, not a checklist. No `v0.3.0-rc1`. No phased release. No `v0.4` deferrals while v0.3.0 is open — every open issue and every open community PR gets absorbed into the v0.3.0 line or explicitly declined. Users follow `main` for previews; users wanting stable stay on `v0.2.7`. ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
|
||||
<!-- GSD:project-end -->
|
||||
|
||||
<!-- GSD:stack-start source:research/STACK.md -->
|
||||
@@ -190,22 +190,10 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
|
||||
<!-- GSD:conventions-start source:CONVENTIONS.md -->
|
||||
## Conventions
|
||||
|
||||
**Versioning (hard rule, owner-set 2026-06-11; single-source 2026-06-16):** main is always **latest release + 1 patch**. **`frontend/package.json` is the SINGLE SOURCE OF TRUTH for the app version** — vite injects `__APP_VERSION__` from it (first-run footer + every auto bug report), and `frontend/src-tauri/tauri.conf.json` reads its bundle version from it (`"version": "../package.json"`, so the MSI/dmg/updater version can't drift from the UI). Three toolchain-required **mirrors** are kept equal to it and bumped in lockstep — `frontend/src-tauri/Cargo.toml` + `pyproject.toml` (cargo/uv need a literal) and `backend/core/version.py`'s `_FALLBACK_VERSION` (the frozen-backend last resort; at runtime the backend reads its version from package metadata via `importlib.metadata`, which `backend.spec`'s `copy_metadata('omnivoice')` makes work in the frozen build too). Never hand-edit any mirror or re-hardcode a literal in `tauri.conf.json`. Guarded by `tests/test_app_version.py` (`test_all_version_files_in_lockstep` + `test_tauri_version_derives_from_package_json`). The moment `vX.Y.Z` is released, bump `package.json` (+ the mirrors) to `X.Y.(Z+1)`. Consequences:
|
||||
- Every PR and preview build identifies as the **next** version. Preview builds stamp `X.Y.(Z+1)-N` (run number), which semver-sorts **above** the last stable `X.Y.Z` — the updater ordering is natural, no comparator tricks needed.
|
||||
- Releasing = tag `vX.Y.(Z+1)` from main (version files already match), then immediately bump main to `X.Y.(Z+2)`. The post-release bump is automated by the `version-bump` job in release.yml; if it fails, do it manually in the same day.
|
||||
- Docker: `ghcr.io/debpalash/omnivoice-studio:latest` = **main** (rolling preview); `:X.Y.Z` + `:X.Y` + `:stable` = tagged releases. `:latest` is the preview channel by design — stable users pin `:stable` or a version tag.
|
||||
- Do not bump minor/major or invent RCs/codenames without the owner asking. No "defer to next version" labels — scope is absorbed or declined, never re-versioned.
|
||||
|
||||
**Docs-sync (hard rule, owner-set 2026-06-11):** any change that alters something these docs describe — README.md, CONTRIBUTING.md, SECURITY.md, SUPPORT.md, LICENSE, or `docs/**` (install flows, Docker tag semantics, platform support, versioning/release behavior, review process, supported versions) — must update those docs **in the same PR** as the change. If a doc impact is discovered after merge, the docs fix is the immediate next commit, not backlog. Stale docs are treated as bugs.
|
||||
|
||||
**Release notes / changelog (hard rule, owner-set 2026-06-16):** every tagged release gets a **high-quality, user-facing `## [X.Y.Z] — DATE` section in `CHANGELOG.md`** before (or in the same hour as) the tag — never the "Auto-generated release for vX.Y.Z…" fallback. `release.yml` extracts that section verbatim as the GitHub Release body (the `Extract CHANGELOG section for tag` step), so a missing/empty section ships a bare release. Quality bar = the existing house style: a one-paragraph headline, then `### Added` / `### Fixed` / `### Changed` / `### License` / `### CI` subsections; each entry is a **bold one-line lead** (what the user gets), 1–3 lines of plain-English why, and the `(#NNN)` issue/PR ref — grouped by theme, written for users, **not** raw commit dumps. This applies to **preview builds too**: preview release notes summarize what's new on `main` since the last stable, in the same style. Workflow: as features merge, keep `## [Unreleased]` current; at release time rename it to the version + date. If a release was already cut with the fallback body, the next action is to backfill `CHANGELOG.md` **and** `gh release edit <tag>` the live body — not backlog.
|
||||
**Versioning (hard rule):** Everything ships on `v0.3.0`. Never mention, suggest, or label anything with a version bump — no v0.4, no RCs, no "defer to next version", no future-version labels — unless the user explicitly asks to bump. Zero unprompted version chatter.
|
||||
|
||||
**Localization (hard rule):** No hardcoded non-English (CJK) **user-facing text** anywhere in the codebase except the translation layer (`frontend/src/i18n/`). All UI strings go through i18n (`t('...')` keys in `locales/*.json`); native language names live in `i18n/index.ts` (`LANGUAGES`). Functional CJK is allowed and tracked via the allowlist in `tests/test_no_hardcoded_cjk.py` — text-processing regexes, model/engine vocabulary & identifiers (e.g. CosyVoice speaker IDs), localized error matching, demo/eval data, and test fixtures. CI fails on any hardcoded CJK outside the allowlist; to add legitimate functional CJK, extend `_ALLOWED_FILES` there with a justification.
|
||||
|
||||
**Fix quality (hard rule, owner-set 2026-06-16):** Fix issues *properly* and future-maintenance-proof — don't stop at the symptom. Root-cause fully, fix the whole **class** of the bug (not just the one reported instance), add a fail-before/pass-after regression test, and harden against recurrence (e.g. if a lockfile drift only fails in Docker, also make CI catch it). Go the extra mile where it durably pays off. Be token-efficient about it — extra **effort**, not extra **verbosity**: no padding, no redundant re-checks, the smallest correct change that is also recurrence-proof. Don't be shy to spend the effort a proper fix needs; do be shy about wasting tokens.
|
||||
|
||||
**Keep main green (hard rule, owner-set 2026-06-16):** A merge must **never break `main`'s CI**. Before a change lands, verify the *full* CI matrix would pass — every workflow in `.github/workflows/` **and** `deploy/Dockerfile`, not only the checks you happened to run. Dependency / lockfile / config changes must be validated against **all** consumers. Specifically: `frontend/` is a bun **workspace monorepo** — the lockfile is the repo-root `bun.lock`, and `deploy/Dockerfile` runs `bun install --frozen-lockfile`, so any `frontend/package.json` change requires regenerating root `bun.lock` and confirming `bun install --frozen-lockfile` passes (plain `bun install` in `ci.yml` silently tolerates drift, so CI-green ≠ Docker-green). Likewise re-check CodeQL/Security on code changes and the Tauri `cargo` build on Rust/dep changes.
|
||||
|
||||
Other conventions not yet established. Will populate as patterns emerge during development.
|
||||
<!-- GSD:conventions-end -->
|
||||
|
||||
|
||||
+2
-100
@@ -42,22 +42,11 @@ This starts both services:
|
||||
### Desktop App (Tauri)
|
||||
|
||||
```bash
|
||||
bun run desktop # dev: hot-reload Tauri shell + backend
|
||||
bun run desktop-prod # production: builds, bundles the backend, then launches
|
||||
bun run desktop
|
||||
```
|
||||
|
||||
Both run `uv sync` first (so the Python backend env is set up) and start the
|
||||
backend automatically — you do **not** start it separately. Use the exact script
|
||||
names: there is no `desktop=prod` (note the **hyphen** in `desktop-prod`).
|
||||
`desktop-prod` is Windows-aware (auto-detects bash/git; see `scripts/desktop-prod.mjs`).
|
||||
|
||||
Requires [Rust](https://rustup.rs/) and platform-specific Tauri dependencies — see the [Tauri prerequisites](https://v2.tauri.app/start/prerequisites/).
|
||||
|
||||
If the app opens but stays on the **setup splash with no buttons**, the Python
|
||||
backend didn't finish starting — the splash surfaces the stall reason, a log
|
||||
panel, and a **Retry** button (and Settings → Logs → Backend has the full trace).
|
||||
The most common from-source cause is `uv` or Python not being on your PATH.
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
@@ -159,7 +148,7 @@ class MyEngineBackend(TTSBackend):
|
||||
|
||||
- **Components**: Functional components with hooks
|
||||
- **State**: Zustand stores in `src/stores/`, organized by slice
|
||||
- **CSS**: **Utilities-first + shadcn/ui, one stylesheet.** UI is built on the shadcn/ui primitives in `src/components/ui/` (wrapped by the `src/ui/` barrel, themed to the OmniVoice palette), composed with Tailwind v4 utility classes. **All styling now lives in a single file — `src/index.css`**: the `@theme` / `[data-theme]` token foundation plus the irreducible set utilities can't express (`@keyframes`, glassmorphism/`backdrop-filter`, pseudo-elements, `:has()`, unlayered cascade overrides, and styling hooks on library-generated DOM like virtualized rows / WaveSurfer). The per-component `.css` files were eliminated in the CSS→Tailwind/shadcn migration — **do not create new ones.** Reach for shadcn primitives + utilities; if a rule is genuinely irreducible, add it to `src/index.css` with a provenance comment. (The only other `.css` is the test-only visual harness. See `docs/shadcn-migration.md`.)
|
||||
- **CSS**: Vanilla CSS in component-level files — no Tailwind
|
||||
- **Naming**: `PascalCase` for components, `camelCase` for hooks and utils
|
||||
|
||||
### Rust (Tauri)
|
||||
@@ -169,30 +158,6 @@ class MyEngineBackend(TTSBackend):
|
||||
|
||||
---
|
||||
|
||||
## Frontend file structure & size limits
|
||||
|
||||
Frontend code stays modular so an edit loads one small file, not a 1900-line
|
||||
one. The rules:
|
||||
|
||||
- **Size caps:** **soft 300 lines**, **hard 500 lines** per `.jsx` file.
|
||||
Anything over 500 lines must be split. (The cap does **not** apply to
|
||||
`src/index.css` — it is the single, intentional styling foundation and the
|
||||
only app stylesheet; see the CSS rule above.)
|
||||
- **Pages are thin orchestrators.** A file in `frontend/src/pages/` is just
|
||||
layout + routing + state wiring that composes feature components — no inline
|
||||
sub-component over ~50 lines.
|
||||
- **One component per file.** Co-locate `Foo.jsx` + `Foo.test.jsx` together in a
|
||||
per-page feature folder under `frontend/src/components/` (e.g.
|
||||
`components/settings/`, `components/dub/`). Styling is **not** co-located —
|
||||
it's utilities + shadcn, with any irreducible rules in `src/index.css`.
|
||||
- **Shared bits go in a `primitives/` folder** inside the feature folder
|
||||
(`components/settings/primitives/` is the existing example).
|
||||
- **Enforced by ESLint `max-lines`** (`max: 500`) — **warn-only for now** so it
|
||||
never breaks CI, with the goal of upgrading to `error` once the backlog of
|
||||
oversized files clears.
|
||||
|
||||
---
|
||||
|
||||
## Commit Messages
|
||||
|
||||
Write clear, concise messages. The PR title becomes the squash-merge commit.
|
||||
@@ -227,69 +192,6 @@ cd frontend/src-tauri && cargo check
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## What code review looks like
|
||||
|
||||
Every PR is reviewed by two AI reviewers before a human looks at it:
|
||||
|
||||
- **CodeRabbit** posts a walkthrough (with a sequence diagram, and an ASCII
|
||||
before/after sketch for UI changes), inline findings, and warning-mode
|
||||
pre-merge checks against the project's hard rules.
|
||||
- **Greptile** reviews with the same project rubrics and learns from 👍/👎
|
||||
reactions on its comments — react to train it.
|
||||
|
||||
Both are advisory, not gating: CI and the maintainer's approval decide. Don't
|
||||
be surprised by detailed bot comments minutes after you open a PR — address
|
||||
what's right, push back (in a reply) on what's wrong.
|
||||
|
||||
**Commit & PR conventions:** conventional-commit style with a scope
|
||||
(`fix(dub): …`, `feat(setup): …`) and link the issue (`Closes #N` / `Refs #N`)
|
||||
in the title or body.
|
||||
|
||||
## Quality gates your PR must pass
|
||||
|
||||
- **Cross-platform parity (hard rule):** anything that ships in default mode
|
||||
must behave identically on macOS, Windows, and Linux. Platform-specific
|
||||
*implementation* is fine; platform-divergent *default behavior* is a P0.
|
||||
Platform-only features go behind an explicit opt-in (Settings toggle, env
|
||||
var, or CLI flag).
|
||||
- **i18n — all 21 locales (hard rule):** every user-facing string goes through
|
||||
`t('...')` and the key must exist in **all 21** files under
|
||||
`frontend/src/i18n/locales/`. Translate; don't copy English into non-English
|
||||
locales. CI fails on hardcoded CJK outside the allowlist in
|
||||
`tests/test_no_hardcoded_cjk.py` (extend `_ALLOWED_FILES` with a
|
||||
justification for legitimate functional CJK).
|
||||
- **DB schema changes** go through an alembic migration with a tested upgrade
|
||||
path — existing `omnivoice_data/` must keep working with no manual steps.
|
||||
- **Engine back-compat:** already-installed engines (model weights on disk)
|
||||
must not require reinstall or re-download.
|
||||
- **Local-first:** no new outbound calls except GitHub Issues (opt-in
|
||||
reporting) and HuggingFace model downloads. Never log or persist secrets or
|
||||
absolute home paths.
|
||||
- **Security posture:** the backend serves loopback HTTP — treat every
|
||||
query/path/form parameter as hostile. User-chosen filesystem destinations
|
||||
are authorized in the Tauri process (save dialog), never via HTTP params.
|
||||
|
||||
## Contribution licensing
|
||||
|
||||
OmniVoice Studio is **AGPL-3.0-only**, and the maintainer also offers a
|
||||
**commercial license** (see [LICENSE](LICENSE)). By submitting a contribution
|
||||
you agree that:
|
||||
|
||||
1. you have the right to submit it (your own work, or compatibly licensed);
|
||||
2. it is licensed to the project under **AGPL-3.0**; and
|
||||
3. you grant the project maintainer a perpetual, worldwide, non-exclusive
|
||||
right to also distribute your contribution under the project's commercial
|
||||
license terms.
|
||||
|
||||
This inbound grant is what keeps the dual-license model viable. If you can't
|
||||
agree to (3) for a particular contribution, say so in the PR and we'll discuss
|
||||
before merging. Adding a `Signed-off-by:` line (DCO) to your commits is
|
||||
appreciated but not required.
|
||||
|
||||
---
|
||||
|
||||
## Need Help?
|
||||
|
||||
- **Stuck on setup?** Ask in [Discord #help](https://discord.gg/bzQavDfVV9)
|
||||
|
||||
@@ -4,26 +4,33 @@
|
||||
<h3>The open-source ElevenLabs alternative.</h3>
|
||||
<p>Real-time dictation, zero-shot voice cloning, and cinematic video dubbing — all on your desktop.<br/>Open-source, no API keys, fully local. <b>646 languages.</b></p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/stargazers"><img src="https://img.shields.io/github/stars/debpalash/OmniVoice-Studio?style=flat-square&color=f59e0b" alt="Stars" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/OmniVoice-Studio?style=flat-square&color=10b981" alt="Release" /></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="License" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/issues"><img src="https://img.shields.io/github/issues/debpalash/OmniVoice-Studio?style=flat-square&color=ef4444" alt="Issues" /></a>
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Discord-Join_Community-5865F2?style=flat-square&logo=discord&logoColor=white" alt="Discord" /></a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="#quickstart">Quickstart</a> ·
|
||||
<a href="#features">Features</a> ·
|
||||
<a href="#why-ovs">Why OVS</a> ·
|
||||
<a href="#why-omnivoice-studio">Why OmniVoice Studio?</a> ·
|
||||
<a href="#tts-engines">TTS Engines</a> ·
|
||||
<a href="#asr-engines">ASR Engines</a> ·
|
||||
<a href="#sponsor--donate">Donate</a> ·
|
||||
<a href="#contributing">Contributing</a> ·
|
||||
<a href="https://discord.gg/bzQavDfVV9">Discord</a> ·
|
||||
<a href="README_CN.md"><strong>简体中文</strong></a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/stargazers"><img src="https://img.shields.io/github/stars/debpalash/OmniVoice-Studio?style=flat-square&color=f59e0b" alt="Stars" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/OmniVoice-Studio?style=flat-square&color=10b981" alt="Release" /></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="License" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/issues"><img src="https://img.shields.io/github/issues/debpalash/OmniVoice-Studio?style=flat-square&color=ef4444" alt="Issues" /></a>
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Discord-Join_Community-5865F2?style=flat-square&logo=discord&logoColor=white" alt="Discord" /></a>
|
||||
<a href="https://ko-fi.com/debpalash"><img src="https://img.shields.io/badge/Ko--fi-Support_Us-FF5E5B?style=flat-square&logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
|
||||
<a href="https://paypal.me/palashCoder"><img src="https://img.shields.io/badge/PayPal-Donate-00457C?style=flat-square&logo=paypal&logoColor=white" alt="PayPal" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_aarch64.dmg"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_x64_en-US.msi"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_amd64.AppImage"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_amd64.deb"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
|
||||
</p>
|
||||
<p>
|
||||
<sub><b>macOS:</b> first launch needs a one-time approval — right-click → <b>Open</b> (or System Settings → Privacy & Security → <b>"Open Anyway"</b> on macOS 15). No Terminal needed. <a href="docs/install/macos.md#gatekeeper-quarantine">Why?</a></sub>
|
||||
</p>
|
||||
</div>
|
||||
|
||||
@@ -50,28 +57,20 @@
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td align="center" width="25%">
|
||||
<td align="center" width="33%">
|
||||
<h3>🎙️ Voice Cloning</h3>
|
||||
<p>3-second clip → mirror any voice.<br/><b>646 languages</b>, zero-shot.</p>
|
||||
</td>
|
||||
<td align="center" width="25%">
|
||||
<td align="center" width="33%">
|
||||
<h3>🎨 Voice Design</h3>
|
||||
<p>Gender, age, accent, pitch, speed,<br/>emotion, dialect — <b>dial it in</b>.</p>
|
||||
</td>
|
||||
<td align="center" width="25%">
|
||||
<td align="center" width="33%">
|
||||
<h3>🎬 Video Dubbing</h3>
|
||||
<p>YouTube URL or file → transcribe →<br/>translate → re-voice → <b>MP4</b>.</p>
|
||||
</td>
|
||||
<td align="center" width="25%">
|
||||
<h3>📖 Audiobook Editor</h3>
|
||||
<p>Import text, EPUB, or PDF. Auto-chapter,<br/>loudnorm, metadata. Export <b>.m4b</b>.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
<h3>🎭 Stories</h3>
|
||||
<p>Multi-voice editor. Assign voices<br/>per-line, preview, <b>export full cast</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>⌨️ Dictation Widget</h3>
|
||||
<p><code>⌘+⇧+Space</code> from <b>any app</b>.<br/>Transcribes, auto-pastes, disappears.</p>
|
||||
@@ -98,10 +97,6 @@
|
||||
<h3>🛡️ AI Watermark</h3>
|
||||
<p>AudioSeal (Meta). <b>Invisible</b>,<br/>survives compression.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🔬 Diagnostics</h3>
|
||||
<p>Self-check, error journal,<br/>scrubbed <b>diagnostic bundle</b>.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
@@ -114,29 +109,7 @@
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🧩 Extensible</h3>
|
||||
<p>Subclass <code>TTSbackend</code>,<br/>add any engine in <b>~50 lines</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🧭 Engine Routing</h3>
|
||||
<p>Preflight GPU check per engine.<br/><b>No silent CPU fallback</b>.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
<h3>🎒 Portable Personas</h3>
|
||||
<p>Export voices as <code>.ovsvoice</code><br/>bundles — identity + <b>watermark</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>♾️ Unlimited TTS</h3>
|
||||
<p>Sentence-chunked generation.<br/><b>No length cap</b>. Streaming via WS.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🌐 Remote Backend</h3>
|
||||
<p>Point UI at a remote server.<br/>Tailscale-friendly. <b>Bearer auth</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🧠 Dictation + LLM</h3>
|
||||
<p>Local LLM cleanup of transcripts.<br/>Optional echo <b>cancellation</b>.</p>
|
||||
<p>Subclass <code>TTSBackend</code>,<br/>add any engine in <b>~50 lines</b>.</p>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
@@ -145,21 +118,12 @@
|
||||
|
||||
## Quickstart
|
||||
|
||||
<div align="center">
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
|
||||
<br/>
|
||||
<sub><b>macOS:</b> first launch needs a one-time approval — right-click → <b>Open</b> (or System Settings → Privacy & Security → <b>"Open Anyway"</b> on macOS 15). No Terminal needed. <a href="docs/install/macos.md#gatekeeper-quarantine">Why?</a></sub>
|
||||
</div>
|
||||
|
||||
Per-OS install guides — pick yours and follow it end-to-end:
|
||||
|
||||
- **macOS** — [docs/install/macos.md](docs/install/macos.md)
|
||||
- **Windows** — [docs/install/windows.md](docs/install/windows.md)
|
||||
- **Linux** — [docs/install/linux.md](docs/install/linux.md)
|
||||
- **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
|
||||
- **Docker** — [docs/install/docker.md](docs/install/docker.md)
|
||||
|
||||
Stuck? Run the built-in self-check first — **Settings → About → "Run
|
||||
self-check"** in the app, or `uv run python backend/main.py --diagnose` from
|
||||
@@ -172,9 +136,7 @@ bundle"** packages scrubbed logs + the self-check report for bug reports.
|
||||
For Hugging Face token setup, see
|
||||
[docs/setup/huggingface-token.md](docs/setup/huggingface-token.md). For
|
||||
diarization-specific gating, see
|
||||
[docs/features/diarization.md](docs/features/diarization.md). For download
|
||||
speed, the ⚡ fast-download (Xet) status, and restricted-network / mirror
|
||||
options, see [docs/downloading-models.md](docs/downloading-models.md).
|
||||
[docs/features/diarization.md](docs/features/diarization.md).
|
||||
|
||||
## Screenshots
|
||||
|
||||
@@ -226,7 +188,7 @@ options, see [docs/downloading-models.md](docs/downloading-models.md).
|
||||
|
||||
---
|
||||
|
||||
## Why OVS?
|
||||
## Why OmniVoice Studio?
|
||||
|
||||
ElevenLabs charges **$5–$330/mo** and processes your audio on their servers. OmniVoice Studio runs **on your hardware, with no usage limits.**
|
||||
|
||||
@@ -235,17 +197,12 @@ ElevenLabs charges **$5–$330/mo** and processes your audio on their servers. O
|
||||
| **Pricing** | $5–$330/mo, per-character billing | Free & open-source (AGPL-3.0) · [Commercial license](#license) for proprietary use |
|
||||
| **Voice Cloning** | ✅ 3s clip | ✅ 3s clip, zero-shot |
|
||||
| **Voice Design** | ✅ Gender, age | ✅ Gender, age, accent, pitch, style, dialect |
|
||||
| **Audiobook / Stories** | ❌ | ✅ Full audiobook editor + multi-voice stories (EPUB/PDF import, .m4b export) |
|
||||
| **Languages** | 32 | **646** |
|
||||
| **Video Dubbing** | ✅ Cloud-only | ✅ Fully local |
|
||||
| **Data Privacy** | Audio sent to cloud | **Nothing leaves your machine** |
|
||||
| **API Keys** | Required | Not needed |
|
||||
| **GPU Support** | N/A (cloud) | CUDA · Apple Silicon · ROCm · CPU |
|
||||
| **Desktop App** | ❌ | ✅ macOS · Windows · Linux |
|
||||
| **TTS Engines** | 1 | **11** (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, IndexTTS 2, OmniVoice GGUF, Supertonic 3) |
|
||||
| **ASR Engines** | 1 | **9** (WhisperX, Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet, Moonshine, FunASR, isolated Faster-Whisper, sherpa-onnx live dictation) |
|
||||
| **MCP Server** | ❌ | ✅ Use from Claude, Cursor, any MCP client |
|
||||
| **Self-check** | ❌ | ✅ Diagnostics suite, error journal, scrubbed debug bundles |
|
||||
| **Customizable** | ❌ Closed | ✅ Fork it, extend it, ship it |
|
||||
|
||||
OmniVoice Studio gives you professional-grade AI tools without the subscription or the cloud.
|
||||
@@ -281,21 +238,12 @@ OmniVoice ships a multi-engine TTS backend. The default engine (OmniVoice) is al
|
||||
|--------|:---------:|:-----:|:--------:|:-----:|:---------:|:-------:|:-------:|
|
||||
| **OmniVoice** (default) | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Built-in |
|
||||
| **CosyVoice 3** | 9 + 18 dialects | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **GPT-SoVITS** | 5 | ✅ | — | ✅ CUDA/CPU | — | ✅ CUDA/CPU | MIT |
|
||||
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MOSS-TTS-Nano** | 20 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **KittenTTS** | English | — | — | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
|
||||
| **MLX-Audio** (Kokoro, Qwen3-TTS, CSM, Dia, …) | Multi | Varies | Varies | ❌ | ✅ Native | ❌ | Varies |
|
||||
| **Sherpa-ONNX** | 20+ | — | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **IndexTTS 2** ⚡ | Multi | ✅ | — | ✅ CUDA | — | ✅ CUDA | Apache-2.0 |
|
||||
| **OmniVoice GGUF** ⚡ | 600+ | ✅ | ✅ | ✅ CPU | ✅ CPU | ✅ CPU | Built-in |
|
||||
| **Supertonic 3** ⚡ | 31 | — | — | ✅ CPU | ✅ CPU | ✅ CPU | OpenRAIL-M |
|
||||
| **MOSS-TTS-v1.5** ⚡ (8B) | 31 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **dots.tts** ⚡ (2B) | 24 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ❌ | Apache-2.0 |
|
||||
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MOSS-TTS-Nano** | 20 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **KittenTTS** | English | ❌ | ❌ | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
|
||||
|
||||
> **CUDA** = GPU-accelerated · **MPS** = Apple Silicon Metal · **CPU** = runs everywhere, slower for large models · KittenTTS and MOSS-TTS-Nano run realtime on CPU · MLX-Audio is Apple Silicon only · ⚡ = lazy-registered (installed on first use)
|
||||
>
|
||||
> **MOSS-TTS-v1.5** (8B, ~16 GB weights) and **dots.tts** (2B, ~9 GB weights) are heavyweight opt-in engines that run in their own isolated venv from a local clone — see [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) and [dots.tts](docs/engines/dots-tts.md). Neither claims Apple-Silicon **MPS** (upstream is CUDA/CPU only; on a Mac they run on CPU). dots.tts upstream is Linux/macOS only — no Windows path.
|
||||
> **CUDA** = GPU-accelerated · **MPS** = Apple Silicon Metal · **CPU** = runs everywhere, slower for large models · KittenTTS and MOSS-TTS-Nano run realtime on CPU · MLX-Audio is Apple Silicon only.
|
||||
|
||||
### ASR Engines
|
||||
|
||||
@@ -305,36 +253,31 @@ OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictatio
|
||||
|--------|-------------------------|:---------:|----------|
|
||||
| **WhisperX** (default) | `whisperx` | ~100 | Dubbing & subtitles — word-level timing via wav2vec2 forced alignment |
|
||||
| **Faster-Whisper** | `faster-whisper` | ~100 | Fast transcription on Linux / macOS / Windows (CTranslate2) |
|
||||
| **Faster-Whisper (isolated)** | `faster-whisper-isolated` | ~100 | Same as Faster-Whisper but crash-isolated in a subprocess — an ASR crash won't take down the app |
|
||||
| **MLX Whisper** | `mlx-whisper` | ~100 | Native Apple Silicon speed (Apple MLX / Metal) |
|
||||
| **PyTorch Whisper** | `pytorch-whisper` | ~100 | CUDA / CPU fallback via 🤗 Transformers (no cuDNN 8 needed) |
|
||||
| **PyTorch Whisper** | `pytorch-whisper` | ~100 | CUDA / CPU fallback via 🤗 Transformers |
|
||||
| **Parakeet TDT** | `nemo-parakeet` | English + 25 EU | SOTA English accuracy, auto language detection (NVIDIA NeMo, GPU only) |
|
||||
| **Moonshine** | `moonshine` | English | Edge / low-latency, ONNX |
|
||||
| **FunASR** | `funasr` | 50+ | All-in-one multilingual — built-in VAD + inline speaker diarization (SenseVoice) |
|
||||
| **sherpa-onnx** (live dictation) | `sherpa-onnx-asr` | 25 EU + 90+ | Live, faster-than-real-time dictation — small streaming/offline ONNX models (Parakeet TDT v3/v2, streaming Zipformer & Paraformer, Whisper Tiny), CPU, identical on macOS / Windows / Linux. Picked per-model in **Settings → Voice**. |
|
||||
|
||||
> Whisper-family engines cover ~100 languages; **FunASR / SenseVoice** adds an all-in-one multilingual path with built-in voice-activity detection and inline speaker diarization. **sherpa-onnx** powers the live dictation model picker — you talk and text appears as you speak. Every engine runs on-device — no API keys, no cloud.
|
||||
|
||||
> **GPU without efficient float16?** On older NVIDIA GPUs (Maxwell/Pascal, GTX 16xx) or after a CTranslate2/cuDNN mismatch, the CTranslate2 ASR engines (WhisperX, Faster-Whisper) can't run `float16` and OmniVoice automatically retries on `int8` — no config needed. If transcription still fails, pin the compute type with the `ASR_COMPUTE_TYPE` env var (escape hatch): `ASR_COMPUTE_TYPE=int8` (or `float32` for CPU). Set it to `int8` and restart the backend.
|
||||
> Whisper-family engines cover ~100 languages; **FunASR / SenseVoice** adds an all-in-one multilingual path with built-in voice-activity detection and inline speaker diarization. Every engine runs on-device — no API keys, no cloud.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Frontend (React) │
|
||||
│ DubTab · VoiceConsole · Stories · Audiobook · Gallery │
|
||||
│ Dictation · BatchQueue · Diagnostics · MCP Client │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Backend (FastAPI) │
|
||||
│ 100+ API endpoints · SSE+WSS streaming · SQLite │
|
||||
├──────────┬──────────┬──────────┬──────────┬────────────────┤
|
||||
│ WhisperX │ Demucs │OmniVoice │ Pyannote │ Engine Routing │
|
||||
│ (+7 ASR │ Source │ (+10 │ Diariz- │ ↳ GPU preflight │
|
||||
│ engines) │ Sep. │ TTS) │ ation │ ↳ No silent CPU │
|
||||
└──────────┴──────────┴──────────┴──────────┴────────────────┘
|
||||
CUDA / MPS / ROCm / CPU (auto-detected + routed)
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ Frontend (React) │
|
||||
│ DubTab · VoicePreview · BatchQueue · Gallery │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Backend (FastAPI) │
|
||||
│ 97 API endpoints · SSE streaming · SQLite │
|
||||
├──────────┬──────────┬──────────┬────────────────┤
|
||||
│ WhisperX │ Demucs │OmniVoice │ Pyannote │
|
||||
│ ASR │ Source │ TTS │ Diarization │
|
||||
│ │ Sep. │ │ │
|
||||
└──────────┴──────────┴──────────┴────────────────┘
|
||||
CUDA / MPS / ROCm / CPU (auto-detected)
|
||||
```
|
||||
|
||||
---
|
||||
@@ -345,55 +288,27 @@ OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictatio
|
||||
|
||||
| Category | Features |
|
||||
|----------|----------|
|
||||
| **Longform** | Audiobook editor (text/EPUB/PDF → chaptered .m4b), Stories multi-voice editor, two-pass loudnorm mastering, crash-resume for interrupted renders, pronunciation control + SSML-lite prosody |
|
||||
| **Dubbing** | Full pipeline (transcribe→translate→synthesize→mux), scene-aware splitting, lip-sync scoring, streaming TTS, per-speaker voice assignment, Smart Fit timing + second-pass QC, dedicated Dub home |
|
||||
| **Voice** | Zero-shot cloning, voice design, A/B comparison, voice preview widget, gallery with favorites/tags, portable persona bundles (`.ovsvoice`), voice console workspace |
|
||||
| **Audio** | Demucs vocal isolation, per-segment gain, selective track export, stem/SRT/VTT/MP3 export, unlimited-length TTS via sentence-chunked generation |
|
||||
| **Dubbing** | Full pipeline (transcribe→translate→synthesize→mux), scene-aware splitting, lip-sync scoring, streaming TTS |
|
||||
| **Voice** | Zero-shot cloning, voice design, A/B comparison, voice preview widget, gallery with favorites/tags |
|
||||
| **Audio** | Demucs vocal isolation, per-segment gain, selective track export, stem/SRT/VTT/MP3 export |
|
||||
| **Multi-Lang** | Multi-language batch picker, batch dubbing queue with sequential GPU execution |
|
||||
| **Diarization** | Pyannote ML diarization, auto speaker clone extraction, per-speaker voice assignment |
|
||||
| **ASR** | 9 engines (WhisperX, Faster-Whisper, isolated Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet TDT, Moonshine, FunASR/SenseVoice, sherpa-onnx live dictation), crash-isolated subprocess backend |
|
||||
| **TTS** | 11 engines (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, + lazy: IndexTTS 2, OmniVoice GGUF, Supertonic 3), engine routing with GPU preflight |
|
||||
| **Infra** | Docker deployment, CUDA/MPS/ROCm auto-detect, cuDNN 8 compat, VRAM-aware model offloading, engine routing (no silent CPU fallback), diagnostics suite & error journal, restricted-network mirror support |
|
||||
| **Infra** | Docker deployment, CUDA/MPS/ROCm auto-detect, cuDNN 8 compat, VRAM-aware model offloading |
|
||||
| **AI Provenance** | AudioSeal invisible watermarking (SynthID-like), video logo overlay, watermark detection API |
|
||||
| **UX** | Undo/redo, keyboard shortcuts, drag-and-drop, session persistence, glassmorphism design system, UI scale fix for Linux/WebKitGTK |
|
||||
| **UX** | Undo/redo, keyboard shortcuts, drag-and-drop, session persistence, glassmorphism design system |
|
||||
| **Real-time Events** | WebSocket event bus — instant sidebar refresh on data mutations, exponential backoff reconnect |
|
||||
| **State Management** | Zustand store migration — `uiSlice`, `pillSlice`, `dubSlice`, `generateSlice`, `prefsSlice`, `glossarySlice` |
|
||||
| **Desktop** | Cross-platform Tauri installers (macOS DMG/Intel, Windows MSI, Linux deb/AppImage), auto-update infrastructure, single-instance enforcement, close-to-tray, macOS Gatekeeper fix |
|
||||
| **Dictation** | Global system-wide hotkey (`⌘+⇧+Space`), frameless floating widget, streaming ASR via WebSocket, auto-paste, customizable hotkey, local-LLM transcript refinement |
|
||||
| **Desktop** | Cross-platform Tauri installers (macOS DMG, Windows MSI, Linux deb/AppImage), auto-update infrastructure |
|
||||
| **Windows Hardening** | Cross-platform log paths, Triton workaround, HF symlink bypass, 300s health check timeout |
|
||||
| **Dictation** | Global system-wide hotkey (`⌘+⇧+Space`), frameless floating widget, streaming ASR via WebSocket, auto-paste |
|
||||
| **Batch Pipeline** | Full batch TTS: extract → transcribe → translate → generate → mix → export, with live progress tracking |
|
||||
| **MCP Server** | OmniVoice as a local TTS/STT provider for Claude, Cursor, and any MCP client |
|
||||
| **Remote Backend** | Point the desktop UI at a remote backend URL with bearer auth (Tailscale-documented) |
|
||||
| **Reliability** | Stall watchdog on bootstrap splash, per-engine GPU compatibility matrix, actionable errors for non-executable engine binaries, setuptools auto-repair |
|
||||
|
||||
### 🔜 Up Next
|
||||
|
||||
- 🎬 **Lip-sync v2** — visual speech timing with wav2lip
|
||||
- 📖 **Audiobook Editor** — chapter-aware long-form narration
|
||||
- 🌐 **Hosted Demo** — try OmniVoice without installing anything
|
||||
- 🔌 **Plugin Marketplace** — community-contributed TTS engines and effects
|
||||
- 🎵 **Real-time Voice Changer** — live microphone transformation during calls
|
||||
|
||||
---
|
||||
|
||||
## Sponsor / Donate
|
||||
|
||||
OmniVoice Studio is built by one developer using Claude Code and AI agents — and the agent bills are real. Over the last three months I've spent thousands of dollars on Claude subscriptions to keep the features shipping, the bugs fixed, and your issues answered. If OmniVoice has created value for you, helping cover those bills means I can keep developing full-time.
|
||||
|
||||
<div align="center">
|
||||
|
||||
**This month's agent bill fund**
|
||||
|
||||
<img src="https://img.shields.io/badge/raised_%2410_of_%24200-5%25-EAB308?style=for-the-badge" alt="$10 / $200 raised" />
|
||||
|
||||
<br/><br/>
|
||||
|
||||
<a href="https://ko-fi.com/debpalash"><img src="https://img.shields.io/badge/Ko--fi-Support_❤️-FF5E5B?style=for-the-badge&logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
|
||||
|
||||
<a href="https://paypal.me/palashCoder"><img src="https://img.shields.io/badge/PayPal-Donate-00457C?style=for-the-badge&logo=paypal&logoColor=white" alt="PayPal" /></a>
|
||||
|
||||
<br/>
|
||||
<sub>Every dollar goes directly to agent bills — keeping OmniVoice development continuous.</sub>
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
@@ -462,7 +377,7 @@ Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are availab
|
||||
<details>
|
||||
<summary><b>Can I add my own TTS engine?</b></summary>
|
||||
<br/>
|
||||
Yes. OmniVoice uses a <b>built-in backend registry</b>. To add an engine in ~50 lines, subclass <code>TTSBackend</code> in <code>backend/services/tts_backend.py</code> and add it to the <code>_REGISTRY</code> dictionary. Eleven engines are built in: OmniVoice, CosyVoice 3, GPT-SoVITS, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, KittenTTS, Sherpa-ONNX, plus lazy-registered IndexTTS 2, OmniVoice GGUF, and Supertonic 3. See the <a href="#tts-engines">TTS Engines</a> section for details.
|
||||
Yes. OmniVoice uses a <b>built-in backend registry</b>. To add an engine in ~50 lines, subclass <code>TTSBackend</code> in <code>backend/services/tts_backend.py</code> and add it to the <code>_REGISTRY</code> dictionary at the bottom. Six engines are built in: OmniVoice, CosyVoice, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, and KittenTTS. See the <a href="#tts-engines">TTS Engines</a> section for details.
|
||||
</details>
|
||||
|
||||
---
|
||||
@@ -492,9 +407,6 @@ OmniVoice Studio is built on the shoulders of exceptional open-source work:
|
||||
| [**CTranslate2**](https://github.com/OpenNMT/CTranslate2) | Optimized Transformer inference on CPU and GPU |
|
||||
| [**AudioSeal (Meta)**](https://github.com/facebookresearch/audioseal) | Invisible neural audio watermarking for AI provenance |
|
||||
| [**Tauri**](https://tauri.app) | Native desktop app framework |
|
||||
| [**Supertone / Supertonic 3**](https://huggingface.co/Supertone/supertonic-3) | ONNX TTS engine — 31 languages, CPU-efficient |
|
||||
| [**Sherpa-ONNX**](https://github.com/k2-fsa/sherpa-onnx) | WASM-ready universal TTS/ASR runtime |
|
||||
| [**GPT-SoVITS**](https://github.com/RVC-Boss/GPT-SoVITS) | Zero-shot TTS engine — 5 languages, RTF 0.014 |
|
||||
|
||||
---
|
||||
|
||||
@@ -504,8 +416,7 @@ OmniVoice Studio is built on the shoulders of exceptional open-source work:
|
||||
|
||||
If you read this far, you're our kind of person.<br/>
|
||||
**[⭐ Star this repo](https://github.com/debpalash/OmniVoice-Studio)** so others can find it too.<br/>
|
||||
**[💬 Join the Discord](https://discord.gg/bzQavDfVV9)** to share what you build.<br/>
|
||||
**[❤️ Support development](https://ko-fi.com/debpalash)** — fund the AI agent bills that keep OmniVoice shipping.
|
||||
**[💬 Join the Discord](https://discord.gg/bzQavDfVV9)** to share what you build.
|
||||
|
||||
<br/>
|
||||
|
||||
|
||||
@@ -372,13 +372,9 @@ OmniVoice 配备多引擎 TTS 后端。默认引擎(OmniVoice)始终可用
|
||||
| **MLX-Audio**(Kokoro, Qwen3-TTS, CSM, Dia 等) | 多语言 | 因引擎而异 | 因引擎而异 | ❌ | ✅ 原生 | ❌ | 因引擎而异 |
|
||||
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MOSS-TTS-Nano** | 20 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MOSS-TTS-v1.5**(8B,可选装) | 31 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **dots.tts**(2B,可选装) | 24 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ❌ | Apache-2.0 |
|
||||
| **KittenTTS** | 英语 | ❌ | ❌ | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
|
||||
|
||||
> **CUDA** = GPU 加速 · **MPS** = Apple Silicon Metal · **CPU** = 随处可运行,大模型较慢 · KittenTTS 和 MOSS-TTS-Nano 可在 CPU 上实时运行 · MLX-Audio 仅限 Apple Silicon。
|
||||
>
|
||||
> **MOSS-TTS-v1.5**(8B,约 16 GB 权重)和 **dots.tts**(2B,约 9 GB 权重)是重量级可选引擎,从本地克隆在独立 venv 中运行——参见 [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) 和 [dots.tts](docs/engines/dots-tts.md)。两者均不支持 Apple Silicon **MPS**(上游仅支持 CUDA/CPU;在 Mac 上以 CPU 运行)。dots.tts 上游仅支持 Linux/macOS——无 Windows 路径。
|
||||
|
||||
---
|
||||
|
||||
|
||||
+4
-14
@@ -2,20 +2,10 @@
|
||||
|
||||
## Supported Versions
|
||||
|
||||
| Version | Supported |
|
||||
|---------|-----------|
|
||||
| 0.3.x (latest release + `main` previews) | ✅ Current — all fixes land here |
|
||||
| 0.2.7 | ⚠️ Legacy stable — security fixes only, upgrade recommended |
|
||||
| < 0.2.7 | ❌ No longer supported |
|
||||
|
||||
## Model supply chain
|
||||
|
||||
OmniVoice supports models from **public, verifiable sources only** (Hugging
|
||||
Face repos, official project releases). Privately sold or gated model files
|
||||
are not supported: an archive from a private source can carry anything
|
||||
(bundled executables, modified configs), and nobody else can verify or
|
||||
reproduce it. Treat any privately distributed model file as an untrusted
|
||||
download, and never run executables bundled with model archives.
|
||||
| Version | Supported |
|
||||
|---------|--------------------|
|
||||
| 0.2.x | ✅ Current release |
|
||||
| < 0.2 | ❌ No longer supported |
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
|
||||
-36
@@ -1,36 +0,0 @@
|
||||
# Support
|
||||
|
||||
## Where to get help
|
||||
|
||||
| Channel | Best for |
|
||||
|---|---|
|
||||
| [Discord](https://discord.gg/bzQavDfVV9) — `#help` | Setup problems, quick questions, sharing results |
|
||||
| [GitHub Issues](https://github.com/debpalash/OmniVoice-Studio/issues) | Bugs and feature requests — use the templates; attach the diagnostic bundle (Settings → About → "Save diagnostic bundle") |
|
||||
| [GitHub Discussions](https://github.com/debpalash/OmniVoice-Studio/discussions) | Design questions, ideas, show & tell |
|
||||
| Security issues | **Never a public issue** — see [SECURITY.md](SECURITY.md) for private reporting |
|
||||
|
||||
## Model sources we support
|
||||
|
||||
OmniVoice is built on the idea that everything it runs is **open and available
|
||||
to everyone**: free, public models with verifiable sources and licenses
|
||||
(Hugging Face repos, official project releases), so the whole community can
|
||||
use, test, and debug the same thing.
|
||||
|
||||
**We do not support privately sold, paywalled, or gated model files.** A model
|
||||
delivered privately can't be verified, reproduced, or shared — it doesn't fit
|
||||
the project's goals, and issues involving such models will be politely closed.
|
||||
As a general safety rule, never run executables bundled inside any model
|
||||
archive.
|
||||
|
||||
## Before filing a bug
|
||||
|
||||
1. Update to the latest release (or `main` if you follow previews) — fixes ship continuously.
|
||||
2. Run the in-app self-check: **Settings → About → Run self-check**.
|
||||
3. Search existing issues; add a 👍 + your details to an existing one rather than opening a duplicate.
|
||||
|
||||
## Response expectations
|
||||
|
||||
This is an open-source project maintained with the help of an automated triage
|
||||
bot: issues are typically triaged within hours and every report gets a human-
|
||||
approved response. Reproducible reports with a diagnostic bundle get fixed
|
||||
fastest.
|
||||
+1
-7
@@ -13,18 +13,12 @@
|
||||
# Run: uv run pyinstaller backend.spec --noconfirm --clean
|
||||
import platform
|
||||
import sys
|
||||
from PyInstaller.utils.hooks import collect_data_files, collect_all, collect_submodules, copy_metadata
|
||||
from PyInstaller.utils.hooks import collect_data_files, collect_all, collect_submodules
|
||||
|
||||
IS_MAC_ARM = sys.platform == "darwin" and platform.machine() == "arm64"
|
||||
|
||||
datas = []
|
||||
binaries = []
|
||||
|
||||
# Bundle the omnivoice package's .dist-info so importlib.metadata.version()
|
||||
# resolves inside the frozen build. Without it the backend can't read its own
|
||||
# version and falls back to the literal in backend/core/version.py — which is
|
||||
# how a 0.3.6 desktop build shipped reporting "0.3.5" in About + bug reports.
|
||||
datas += copy_metadata('omnivoice')
|
||||
hiddenimports = [
|
||||
# Web stack
|
||||
'uvicorn', 'uvicorn.logging', 'uvicorn.loops', 'uvicorn.loops.auto',
|
||||
|
||||
@@ -7,13 +7,9 @@ composed at the route or router level without surprises.
|
||||
Currently exposed:
|
||||
- `require_loopback`: 403 unless the request came from a loopback origin
|
||||
(bypassed in explicit server mode — see `_server_mode`).
|
||||
- `ws_remote_authorized`: whether a WebSocket handshake from a non-loopback
|
||||
client carries the remote API key (Wave 2.3) — used by WS endpoints that
|
||||
keep their own inline loopback guards.
|
||||
"""
|
||||
|
||||
import os
|
||||
import secrets
|
||||
|
||||
from fastapi import HTTPException, Request
|
||||
|
||||
@@ -75,31 +71,3 @@ def require_loopback(request: Request) -> None:
|
||||
if _server_mode():
|
||||
return
|
||||
raise HTTPException(status_code=403, detail="loopback origin required")
|
||||
|
||||
|
||||
def remote_api_key() -> str | None:
|
||||
"""The remote-backend bearer key (Wave 2.3), or None when remote mode is
|
||||
off. Read at call time so tests can monkeypatch the env."""
|
||||
return os.environ.get("OMNIVOICE_API_KEY") or None
|
||||
|
||||
|
||||
def ws_remote_authorized(websocket) -> bool:
|
||||
"""Whether a WebSocket handshake presents the remote API key.
|
||||
|
||||
Browser WebSockets cannot set an Authorization header, so the key may
|
||||
arrive as ``?api_key=`` or via the ``ov_key`` cookie that the bearer
|
||||
middleware sets on the first authenticated HTTP request. Returns False
|
||||
when remote mode is off — callers keep their loopback-only behavior.
|
||||
"""
|
||||
key = remote_api_key()
|
||||
if not key:
|
||||
return False
|
||||
auth = websocket.headers.get("authorization", "")
|
||||
supplied = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
|
||||
if not supplied:
|
||||
supplied = (
|
||||
websocket.query_params.get("api_key")
|
||||
or websocket.cookies.get("ov_key")
|
||||
or ""
|
||||
)
|
||||
return secrets.compare_digest(supplied, key)
|
||||
|
||||
@@ -18,6 +18,7 @@ Design notes
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
@@ -136,7 +137,7 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
|
||||
from api.routers.generation import ( # noqa: WPS433 — intentional lazy import
|
||||
get_model,
|
||||
_run_inference,
|
||||
run_on_gpu_pool_guarded,
|
||||
_gpu_pool,
|
||||
_safe_torchaudio_save,
|
||||
)
|
||||
|
||||
@@ -146,6 +147,8 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
|
||||
language = None
|
||||
text = (a.get("sample_script") or "").strip() or _FALLBACK_SCRIPT
|
||||
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
def _infer(seed: int):
|
||||
return _run_inference(
|
||||
model, # _model
|
||||
@@ -168,18 +171,14 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
|
||||
"broadcast", # effect_preset
|
||||
)
|
||||
|
||||
# Bounded + pool-reset on hang so a wedged preview render can't starve the
|
||||
# GPU pool and brick the backend (#730 class).
|
||||
audio_tensor = await run_on_gpu_pool_guarded(
|
||||
lambda: _infer(_PREVIEW_SEED), what="Archetype preview generate")
|
||||
audio_tensor = await loop.run_in_executor(_gpu_pool, _infer, _PREVIEW_SEED)
|
||||
if _is_unusable_audio(audio_tensor):
|
||||
# Blank OR a degenerate tonal buzz — retry once on a different seed to
|
||||
# step off the bad diffusion trajectory. Static message only: the
|
||||
# archetype id is request-derived (CodeQL log-injection); the seed is a
|
||||
# module constant, safe to log.
|
||||
logger.warning("Archetype rendered unusable at seed %d — retrying once", _PREVIEW_SEED)
|
||||
audio_tensor = await run_on_gpu_pool_guarded(
|
||||
lambda: _infer(_PREVIEW_SEED + 1), what="Archetype preview generate")
|
||||
audio_tensor = await loop.run_in_executor(_gpu_pool, _infer, _PREVIEW_SEED + 1)
|
||||
if _is_unusable_audio(audio_tensor):
|
||||
raise RuntimeError("the voice engine returned no audible audio for this archetype")
|
||||
|
||||
|
||||
@@ -1,759 +0,0 @@
|
||||
"""Audiobook creator endpoints (parity Wave 5).
|
||||
|
||||
``POST /audiobook/plan`` — pure preview: parse a chapter-delimited script
|
||||
(Markdown ``# H1`` chapters, inline ``[voice:NAME]`` / ``[pause …]``) into the
|
||||
chapter/span plan, no synthesis.
|
||||
|
||||
``POST /audiobook`` — the synth job: render each chapter through the active TTS
|
||||
backend (reusing ``services.audiobook.synthesize_chapter`` + ``chunked_tts``),
|
||||
then mux the chapter WAVs into a chapterized **m4b** (FFMETADATA1 chapters via
|
||||
``build_m4b_cmd``). Progress streams as Server-Sent Events, mirroring the dub
|
||||
pipeline. ffmpeg-gated — without ffmpeg the job reports an error event and
|
||||
stops (the m4b is the only output format).
|
||||
|
||||
``GET /audiobook/jobs`` + ``POST /audiobook/resume/{job_id}`` — durable
|
||||
crash-resume: an interrupted render persists its plan + params to a
|
||||
``resume.json`` manifest in the job work dir, so it can be resumed later (the
|
||||
content-addressed chapter cache makes finished chapters instant) even without
|
||||
the original script. The resume UI affordance remains a follow-up.
|
||||
|
||||
epub/pdf ingest, ACX mastering shipped; the resume UI surface remains a follow-up.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, File, HTTPException, UploadFile
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from services.audiobook import (
|
||||
parse_audiobook_script,
|
||||
synthesize_chapter,
|
||||
)
|
||||
from services.longform_render import (
|
||||
LOUDNESS_PRESETS,
|
||||
build_concat_list,
|
||||
build_ffmetadata,
|
||||
build_render_cmd,
|
||||
prune_cache_dir,
|
||||
)
|
||||
from services import longform_resume # pure (no torch) — durable resume manifest
|
||||
|
||||
logger = logging.getLogger("omnivoice.audiobook")
|
||||
router = APIRouter()
|
||||
|
||||
# A cover filename as produced by /audiobook/cover: 12 hex chars + image ext.
|
||||
# An exact-match allowlist is the strongest barrier (and the one CodeQL's
|
||||
# path-injection query recognizes) — anything else is rejected outright.
|
||||
_COVER_NAME_RE = re.compile(r"^[0-9a-f]{12}\.(?:jpg|jpeg|png)$")
|
||||
|
||||
|
||||
def _safe_cover_path(cover_path: str | None) -> str | None:
|
||||
"""Confine a user-supplied cover to the upload directory before it can flow
|
||||
into ffmpeg.
|
||||
|
||||
Covers only ever come from ``/audiobook/cover``, which writes them to
|
||||
``OUTPUTS_DIR/audiobook_covers`` with a generated name. We rebuild the path
|
||||
from the basename alone (``os.path.basename`` strips any directory component
|
||||
or ``..`` traversal) joined onto that fixed directory, so no caller-supplied
|
||||
path — absolute or relative — can escape it. Returns the path only if the
|
||||
file actually exists there, else None."""
|
||||
if not cover_path:
|
||||
return None
|
||||
from core.config import OUTPUTS_DIR
|
||||
name = os.path.basename(cover_path)
|
||||
if not _COVER_NAME_RE.match(name):
|
||||
return None # not a name the upload endpoint could have produced
|
||||
cover_dir = os.path.realpath(os.path.join(OUTPUTS_DIR, "audiobook_covers"))
|
||||
real = os.path.realpath(os.path.join(cover_dir, name))
|
||||
# Containment check on the resolved path itself — it must live inside the
|
||||
# covers dir. Belt-and-suspenders over the regex+basename above; the
|
||||
# commonpath form is the path-injection barrier static analysis recognizes.
|
||||
if os.path.commonpath([real, cover_dir]) != cover_dir:
|
||||
return None
|
||||
return real if os.path.isfile(real) else None
|
||||
|
||||
|
||||
class AudiobookPlanRequest(BaseModel):
|
||||
text: str
|
||||
default_voice: str | None = None
|
||||
|
||||
|
||||
@router.post("/audiobook/plan")
|
||||
def audiobook_plan(req: AudiobookPlanRequest) -> dict:
|
||||
"""Parse a script into a chapter/span plan (pure preview, no synthesis)."""
|
||||
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
|
||||
return plan.to_dict()
|
||||
|
||||
|
||||
#: Cover size cap mirrors longform_render's guard (8 MB — a book cover, not a
|
||||
#: payload). Kept in sync intentionally; the render builder re-validates too.
|
||||
_COVER_MAX_BYTES = 8 * 1024 * 1024
|
||||
#: Import upload cap — a generous ceiling for a .txt/.md/.epub manuscript that
|
||||
#: still stops a memory-exhaustion upload (the whole file is read into RAM).
|
||||
_IMPORT_MAX_BYTES = 64 * 1024 * 1024
|
||||
#: Upper bound on chapters in a single /longform/render plan — far above any real
|
||||
#: book, but stops a pathological request from allocating/holding the job forever.
|
||||
_MAX_CHAPTERS = 10_000
|
||||
|
||||
|
||||
@router.post("/audiobook/import")
|
||||
async def audiobook_import(file: UploadFile = File(...)) -> dict:
|
||||
"""Import a ``.txt``/``.md``/``.epub``/``.pdf`` into a chapter-delimited script.
|
||||
|
||||
EPUB is parsed in spine order (stdlib only, local); PDF text is extracted
|
||||
with pypdf (pure-Python) then chapterized; plain text gets ``# `` headings
|
||||
inserted ahead of obvious chapter-title lines. Returns the script text (for
|
||||
the editor) + the resulting chapter count."""
|
||||
from services.longform_import import (
|
||||
chapterize_plaintext,
|
||||
epub_to_chapter_script,
|
||||
pdf_to_chapter_script,
|
||||
)
|
||||
|
||||
name = (file.filename or "").lower()
|
||||
data = await file.read()
|
||||
if not data:
|
||||
raise HTTPException(status_code=400, detail="empty file")
|
||||
if len(data) > _IMPORT_MAX_BYTES:
|
||||
raise HTTPException(status_code=400, detail="file too large (max 64 MB)")
|
||||
if name.endswith(".epub"):
|
||||
try:
|
||||
script = epub_to_chapter_script(data)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=f"couldn't parse EPUB: {e}")
|
||||
elif name.endswith(".pdf"):
|
||||
try:
|
||||
script = pdf_to_chapter_script(data)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=f"couldn't parse PDF: {e}")
|
||||
else:
|
||||
script = chapterize_plaintext(data.decode("utf-8", "ignore"))
|
||||
if not script.strip():
|
||||
raise HTTPException(status_code=400, detail="no text found in the file")
|
||||
plan = parse_audiobook_script(script)
|
||||
return {"text": script, "chapters": plan.chapter_count}
|
||||
|
||||
|
||||
@router.post("/audiobook/cover")
|
||||
async def audiobook_cover(cover: UploadFile = File(...)) -> dict:
|
||||
"""Upload a cover image; returns a server-side ``path`` to pass back as
|
||||
``cover_path`` in the synth request. Validated here (jpg/png + size cap) and
|
||||
again at render time."""
|
||||
from core.config import OUTPUTS_DIR
|
||||
|
||||
ext = os.path.splitext(cover.filename or "")[1].lower()
|
||||
if ext not in (".jpg", ".jpeg", ".png"):
|
||||
raise HTTPException(status_code=400, detail="cover must be a .jpg or .png")
|
||||
data = await cover.read()
|
||||
if not data or len(data) > _COVER_MAX_BYTES:
|
||||
raise HTTPException(status_code=400, detail="cover must be between 1 byte and 8 MB")
|
||||
cover_dir = os.path.join(OUTPUTS_DIR, "audiobook_covers")
|
||||
os.makedirs(cover_dir, exist_ok=True)
|
||||
path = os.path.join(cover_dir, f"{uuid.uuid4().hex[:12]}{ext}")
|
||||
with open(path, "wb") as f:
|
||||
f.write(data)
|
||||
return {"path": path}
|
||||
|
||||
|
||||
class AudiobookRequest(BaseModel):
|
||||
text: str
|
||||
default_voice: str | None = None # voice profile id; None = engine default
|
||||
language: str | None = None # None/"Auto" → profile language, else autodetect (#505)
|
||||
bitrate: str = "128k"
|
||||
format: str = "m4b" # "m4b" | "mp3"
|
||||
loudness: str | None = None # None/"off" | "acx" | "podcast" (opt-in)
|
||||
cover_path: str | None = None # server-side path to a jpg/png cover
|
||||
# Global tags embedded in the output: {title, author, narrator, year,
|
||||
# genre, description}. Player-visible (Apple Books / Audible read these).
|
||||
metadata: dict | None = None
|
||||
# Optional pronunciation lexicon {word: respelling} applied before synthesis.
|
||||
lexicon: dict | None = None
|
||||
|
||||
|
||||
def _resolve_voice(profile_id: str | None) -> dict:
|
||||
"""Map a voice-profile id to (ref_audio, ref_text, instruct, seed).
|
||||
|
||||
Compact form of the resolver in generation.py — covers locked, design and
|
||||
clone profiles. Returns all-None for the engine default (no profile).
|
||||
"""
|
||||
out = {"ref_audio": None, "ref_text": None, "instruct": None, "seed": None}
|
||||
if not profile_id:
|
||||
return out
|
||||
from core.config import VOICES_DIR
|
||||
from core.db import db_conn
|
||||
|
||||
with db_conn() as conn:
|
||||
row = conn.execute("SELECT * FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
if not row:
|
||||
return out
|
||||
try:
|
||||
kind = row["kind"] or "clone"
|
||||
except (KeyError, IndexError):
|
||||
kind = "clone"
|
||||
if row["is_locked"] and row["locked_audio_path"]:
|
||||
out["ref_audio"] = os.path.join(VOICES_DIR, row["locked_audio_path"])
|
||||
out["ref_text"] = row["ref_text"]
|
||||
out["instruct"] = row["instruct"]
|
||||
elif kind == "design":
|
||||
out["ref_audio"] = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
|
||||
out["ref_text"] = row["ref_text"] if out["ref_audio"] else None
|
||||
out["instruct"] = row["instruct"]
|
||||
else:
|
||||
out["ref_audio"] = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
|
||||
out["ref_text"] = row["ref_text"]
|
||||
out["instruct"] = row["instruct"]
|
||||
try:
|
||||
if row["seed"] is not None:
|
||||
out["seed"] = row["seed"]
|
||||
except (KeyError, IndexError):
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
def _resolve_default_language(language: str | None, default_voice: str | None) -> str | None:
|
||||
"""Pick the language to thread into the longform synth callable.
|
||||
|
||||
Priority (mirrors the single-shot /generate path, #533): an explicit
|
||||
non-Auto request ``language`` wins; otherwise the selected profile's stored
|
||||
language drives it; otherwise ``None`` (genuine Auto — the engine
|
||||
autodetects, exactly as before). Hardcoding ``None`` here (#505 B2) let the
|
||||
engine re-autodetect per chunk, so a non-English clone flipped to the wrong
|
||||
language on short/ambiguous chapters.
|
||||
"""
|
||||
if language and language != "Auto":
|
||||
return language
|
||||
if default_voice:
|
||||
from core.db import db_conn
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT language FROM voice_profiles WHERE id=?", (default_voice,)
|
||||
).fetchone()
|
||||
if row:
|
||||
try:
|
||||
prof_lang = row["language"]
|
||||
except (KeyError, IndexError):
|
||||
prof_lang = None
|
||||
if prof_lang and prof_lang != "Auto":
|
||||
return prof_lang
|
||||
return None
|
||||
|
||||
|
||||
def _build_synth(default_voice: str | None, language: str | None = None) -> dict:
|
||||
"""Describe how to synthesize for the active TTS engine.
|
||||
|
||||
Returns a dict with ``mode``, ``resolve`` (voice-id → resolved refs, cached
|
||||
per id) and ``engine_id``. For OmniVoice it also carries the async
|
||||
``get_model``; other engines carry a ready ``synth`` + ``sample_rate``.
|
||||
:func:`_prepare_synth` turns this into a uniform ``(synth, sr, resolve,
|
||||
engine_id)`` once the (async) model is in hand.
|
||||
|
||||
``language`` (already resolved by :func:`_resolve_default_language`) is
|
||||
threaded into every chunk's ``generate`` so a non-English clone stays in its
|
||||
language instead of re-autodetecting per chunk (#505 B2). ``None`` keeps the
|
||||
engine's autodetect behavior unchanged.
|
||||
"""
|
||||
from services.tts_backend import OmniVoiceBackend, active_backend_id, get_backend_class
|
||||
|
||||
cache: dict = {}
|
||||
|
||||
def resolve(voice_id):
|
||||
key = voice_id or default_voice
|
||||
if key not in cache:
|
||||
cache[key] = _resolve_voice(key)
|
||||
return cache[key]
|
||||
|
||||
engine_id = active_backend_id()
|
||||
cls = get_backend_class(engine_id)
|
||||
if cls is OmniVoiceBackend:
|
||||
from services.model_manager import get_model
|
||||
return {"mode": "omnivoice", "resolve": resolve,
|
||||
"engine_id": engine_id, "get_model": get_model, "language": language}
|
||||
|
||||
backend = cls()
|
||||
|
||||
def synth(text, voice_id, speed=None):
|
||||
v = resolve(voice_id)
|
||||
return backend.generate(
|
||||
text, language=language, ref_audio=v["ref_audio"],
|
||||
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
|
||||
speed=float(speed) if speed else 1.0,
|
||||
)
|
||||
return {"mode": "generic", "resolve": resolve, "engine_id": engine_id,
|
||||
"synth": synth, "sample_rate": backend.sample_rate}
|
||||
|
||||
|
||||
async def _prepare_synth(default_voice: str | None, language: str | None = None):
|
||||
"""Resolve :func:`_build_synth` into ``(synth, sample_rate, resolve,
|
||||
engine_id)`` — awaiting the OmniVoice model load when needed. Shared by the
|
||||
full job and the per-chapter preview. ``language`` is threaded into every
|
||||
chunk so a non-English clone holds its language (#505 B2)."""
|
||||
info = _build_synth(default_voice, language=language)
|
||||
resolve, engine_id = info["resolve"], info["engine_id"]
|
||||
if info["mode"] == "omnivoice":
|
||||
lang = info["language"]
|
||||
model = await info["get_model"]()
|
||||
sr = getattr(model, "sampling_rate", 24000)
|
||||
|
||||
def synth(text, voice_id, speed=None):
|
||||
v = resolve(voice_id)
|
||||
return model.generate(
|
||||
text=text, language=lang, ref_audio=v["ref_audio"],
|
||||
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
|
||||
speed=float(speed) if speed else 1.0,
|
||||
)[0]
|
||||
return synth, sr, resolve, engine_id
|
||||
return info["synth"], info["sample_rate"], resolve, engine_id
|
||||
|
||||
|
||||
def _render_chapter_cached(chapter, synth, sr, engine_id, resolve, cache_dir, lexicon=None):
|
||||
"""Render one chapter, content-addressed so a re-run reuses it (resume).
|
||||
|
||||
Returns ``(wav_path, duration_s, was_cached)``. The WAV lives at
|
||||
``cache_dir/<key>.wav`` where ``key`` is :func:`chapter_cache_key` over the
|
||||
chapter's spans + sample rate + engine + each voice's resolved signature
|
||||
(+ the lexicon, so a lexicon edit re-renders), so an unchanged chapter is
|
||||
never re-synthesized. Runs in the GPU-pool executor.
|
||||
"""
|
||||
import json
|
||||
import wave
|
||||
|
||||
from services.audio_io import atomic_save_wav
|
||||
from services.longform_render import chapter_cache_key
|
||||
from services.pronunciation import normalize_lexicon
|
||||
|
||||
spans_tuples = [(s.voice_id, s.text, s.pause_ms_after, getattr(s, "speed", None))
|
||||
for s in chapter.spans]
|
||||
sig: dict = {}
|
||||
for s in chapter.spans:
|
||||
k = s.voice_id or ""
|
||||
if k not in sig:
|
||||
v = resolve(s.voice_id)
|
||||
sig[k] = f"{v.get('ref_audio')}|{v.get('ref_text')}|{v.get('instruct')}|{v.get('seed')}"
|
||||
if lexicon:
|
||||
# Fold the lexicon into the cache key so editing pronunciations
|
||||
# invalidates cached chapters (reserved key can't collide with a voice id).
|
||||
sig["\x00lexicon"] = json.dumps(normalize_lexicon(lexicon), sort_keys=True)
|
||||
key = chapter_cache_key(spans_tuples, sample_rate=sr, engine_id=engine_id, voice_sig=sig)
|
||||
wav_path = os.path.join(cache_dir, f"{key}.wav")
|
||||
|
||||
if os.path.exists(wav_path):
|
||||
try:
|
||||
with wave.open(wav_path, "rb") as w:
|
||||
dur = w.getnframes() / float(w.getframerate() or sr)
|
||||
return wav_path, dur, True
|
||||
except Exception:
|
||||
pass # corrupt cache entry — fall through and re-render
|
||||
|
||||
audio, dur = synthesize_chapter(chapter.spans, synth, sr, lexicon=lexicon)
|
||||
atomic_save_wav(wav_path, audio, sr)
|
||||
return wav_path, dur, False
|
||||
|
||||
|
||||
class AudiobookPreviewRequest(BaseModel):
|
||||
text: str
|
||||
chapter_index: int = 0
|
||||
default_voice: str | None = None
|
||||
language: str | None = None # None/"Auto" → profile language, else autodetect
|
||||
lexicon: dict | None = None
|
||||
|
||||
|
||||
@router.post("/audiobook/preview")
|
||||
async def audiobook_preview(req: AudiobookPreviewRequest) -> dict:
|
||||
"""Render a single chapter so the user can audition it before the full run.
|
||||
|
||||
Reuses the same content-addressed cache as the job, so a preview warms the
|
||||
cache (the later full render reuses it) and a re-preview is instant.
|
||||
"""
|
||||
from core.config import OUTPUTS_DIR
|
||||
from services.model_manager import _gpu_pool
|
||||
|
||||
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
|
||||
if not plan.chapters:
|
||||
raise HTTPException(status_code=400, detail="no chapters parsed from the script")
|
||||
n = len(plan.chapters)
|
||||
if not (0 <= req.chapter_index < n):
|
||||
raise HTTPException(status_code=400, detail=f"chapter_index out of range (0..{n - 1})")
|
||||
|
||||
chapter = plan.chapters[req.chapter_index]
|
||||
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache") # shared with _render_longform_sse
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
synth, sr, resolve, engine_id = await _prepare_synth(
|
||||
req.default_voice,
|
||||
language=_resolve_default_language(req.language, req.default_voice),
|
||||
)
|
||||
loop = asyncio.get_running_loop()
|
||||
wav_path, dur, was_cached = await loop.run_in_executor(
|
||||
_gpu_pool, _render_chapter_cached, chapter, synth, sr, engine_id, resolve, cache_dir,
|
||||
req.lexicon,
|
||||
)
|
||||
return {
|
||||
"output": os.path.relpath(wav_path, OUTPUTS_DIR), # served via /audio
|
||||
"duration_s": round(dur, 2),
|
||||
"cached": was_cached,
|
||||
"title": chapter.title,
|
||||
}
|
||||
|
||||
|
||||
async def _render_longform_sse(
|
||||
plan,
|
||||
*,
|
||||
default_voice: str | None,
|
||||
language: str | None = None,
|
||||
fmt: str = "m4b",
|
||||
bitrate: str = "128k",
|
||||
loudness: str | None = None,
|
||||
cover_path: str | None = None,
|
||||
metadata: dict | None = None,
|
||||
lexicon: dict | None = None,
|
||||
job_type: str = "audiobook",
|
||||
job_id: str | None = None,
|
||||
resume: bool = False,
|
||||
):
|
||||
"""Shared chapterized-render SSE generator for Audiobook *and* Stories.
|
||||
|
||||
Takes a ready ``plan`` (``.chapters`` → ``.title`` + ``.spans``) — Audiobook
|
||||
parses it from a script, Stories compiles it from cast/lines — and renders
|
||||
each chapter (content-addressed cache → resume), isolating per-chapter
|
||||
failures, then muxes the successful chapters into a tagged file. This is the
|
||||
convergence point: one renderer, two front doors.
|
||||
"""
|
||||
from core.config import OUTPUTS_DIR
|
||||
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
|
||||
from services.model_manager import _gpu_pool
|
||||
|
||||
# Resume reuses the original job_id (continuing the same job row + cached
|
||||
# chapters); a fresh render generates a new one. The id may arrive from the
|
||||
# /resume/{job_id} path param, so strip it to a safe token (no path
|
||||
# separators, no CR/LF) before it ever reaches a filesystem path or a log
|
||||
# line — CodeQL py/path-injection + py/log-injection. Empty after the strip
|
||||
# → a fresh id.
|
||||
job_id = re.sub(r"[^A-Za-z0-9_-]", "", job_id or "")[:64] or uuid.uuid4().hex[:16]
|
||||
try:
|
||||
from core import job_store
|
||||
if not resume:
|
||||
job_store.create(job_id, type=job_type)
|
||||
job_store.mark_running(job_id)
|
||||
except Exception:
|
||||
job_store = None # job history is best-effort; never block synthesis
|
||||
|
||||
# Persist a durable resume manifest (plan + params) so an interrupted render
|
||||
# can be resumed later even without the original script. Best-effort.
|
||||
try:
|
||||
title = (metadata or {}).get("title") or (plan.chapters[0].title if plan.chapters else "")
|
||||
longform_resume.write_manifest(longform_resume.build_manifest(
|
||||
job_id=job_id, job_type=job_type, title=title,
|
||||
plan_chapters=[
|
||||
{"title": c.title, "spans": [s.to_dict() for s in c.spans]}
|
||||
for c in plan.chapters
|
||||
],
|
||||
params={
|
||||
"default_voice": default_voice, "language": language,
|
||||
"fmt": fmt, "bitrate": bitrate,
|
||||
"loudness": loudness, "cover_path": cover_path,
|
||||
"metadata": metadata, "lexicon": lexicon,
|
||||
},
|
||||
))
|
||||
except Exception: # resume durability is an enhancement; never block the render
|
||||
logger.debug("[%s] resume manifest write skipped", job_id, exc_info=True)
|
||||
|
||||
def _emit(payload: dict) -> str:
|
||||
if job_store is not None:
|
||||
try:
|
||||
job_store.append_event(job_id, json.dumps(payload))
|
||||
except Exception:
|
||||
pass # best-effort job history; never block the stream
|
||||
return f"data: {json.dumps(payload)}\n\n"
|
||||
|
||||
if not plan.chapters:
|
||||
yield _emit({"type": "error", "error": "nothing to render (no chapters)"})
|
||||
return
|
||||
ffmpeg = find_ffmpeg()
|
||||
if not ffmpeg:
|
||||
yield _emit({"type": "error", "error": "ffmpeg not available; the output needs it"})
|
||||
return
|
||||
|
||||
# Confined work dir (job_id is already token-sanitized above; work_dir adds
|
||||
# the basename + realpath barrier so CodeQL sees a clean path).
|
||||
work = longform_resume.work_dir(job_type, job_id)
|
||||
if work is None:
|
||||
yield _emit({"type": "error", "error": "invalid job id"})
|
||||
return
|
||||
os.makedirs(work, exist_ok=True)
|
||||
# Chapter WAVs are content-addressed in a shared cache so a re-run (after a
|
||||
# failure or interruption) reuses what already rendered — only the
|
||||
# missing/changed chapters synthesize again (resume). Shared across both
|
||||
# front doors: an identical chapter renders once.
|
||||
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache")
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
prune_cache_dir(cache_dir) # bound disk before this job adds its chapters
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
try:
|
||||
synth, sr, resolve, engine_id = await _prepare_synth(
|
||||
default_voice, language=_resolve_default_language(language, default_voice)
|
||||
)
|
||||
|
||||
total = len(plan.chapters)
|
||||
chapter_files: list[str] = []
|
||||
chapters_meta: list[tuple[str, int]] = []
|
||||
cached_n = 0
|
||||
failed: list[int] = []
|
||||
yield _emit({"type": "started", "job_id": job_id, "chapters": total})
|
||||
|
||||
for i, chapter in enumerate(plan.chapters):
|
||||
try:
|
||||
wav_path, dur, was_cached = await loop.run_in_executor(
|
||||
_gpu_pool, _render_chapter_cached,
|
||||
chapter, synth, sr, engine_id, resolve, cache_dir, lexicon,
|
||||
)
|
||||
except Exception: # isolate a bad chapter — keep going
|
||||
logger.warning("[%s] chapter %d (%s) failed to render",
|
||||
job_id, i, chapter.title, exc_info=True)
|
||||
failed.append(i)
|
||||
yield _emit({"type": "chapter_error", "index": i, "total": total,
|
||||
"title": chapter.title, "error": "chapter failed to render"})
|
||||
continue
|
||||
chapter_files.append(wav_path)
|
||||
chapters_meta.append((chapter.title, int(round(dur * 1000))))
|
||||
cached_n += 1 if was_cached else 0
|
||||
yield _emit({"type": "chapter", "index": i, "total": total,
|
||||
"title": chapter.title, "duration_s": round(dur, 2),
|
||||
"cached": was_cached})
|
||||
|
||||
if not chapter_files:
|
||||
yield _emit({"type": "error", "error": "all chapters failed to render"})
|
||||
return
|
||||
|
||||
yield _emit({"type": "assembling"})
|
||||
meta_path = os.path.join(work, "chapters.ffmeta")
|
||||
with open(meta_path, "w", encoding="utf-8") as f:
|
||||
f.write(build_ffmetadata(chapters_meta, global_meta=metadata))
|
||||
concat_path = os.path.join(work, "concat.txt")
|
||||
with open(concat_path, "w", encoding="utf-8") as f:
|
||||
f.write(build_concat_list(chapter_files))
|
||||
ext = "mp3" if (fmt or "").lower() == "mp3" else "m4b"
|
||||
out_name = f"{job_type}_{job_id}.{ext}"
|
||||
out_path = os.path.join(OUTPUTS_DIR, out_name)
|
||||
|
||||
# Two-pass loudness master (#28): for a known preset, measure the
|
||||
# concatenated program first, then feed the measured values back into the
|
||||
# single mux encode. `measured is None` (skip OR any failure) → the mux
|
||||
# falls back to single-pass. Gated identically to the pure builders
|
||||
# (.lower(), no strip), so off/None/unknown/whitespace skip cleanly.
|
||||
measured = None
|
||||
norm = (loudness or "").lower()
|
||||
if norm in LOUDNESS_PRESETS:
|
||||
yield _emit({"type": "mastering", "preset": norm})
|
||||
from services.loudness import measure_loudness
|
||||
measured = await measure_loudness(ffmpeg, concat_path, norm, job_id=job_id)
|
||||
|
||||
await run_ffmpeg(
|
||||
build_render_cmd(
|
||||
ffmpeg, concat_path, meta_path, out_path,
|
||||
fmt=ext, bitrate=bitrate, cover_path=_safe_cover_path(cover_path),
|
||||
loudness=loudness, measured=measured,
|
||||
),
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if job_store is not None:
|
||||
try:
|
||||
job_store.mark_done(job_id)
|
||||
except Exception:
|
||||
pass # best-effort job history
|
||||
# The render finished — drop the resume manifest so this job is no longer
|
||||
# offered for resume.
|
||||
longform_resume.clear_manifest(job_type, job_id)
|
||||
total_s = sum(d for _, d in chapters_meta) / 1000.0
|
||||
done = {"type": "done", "output": out_name,
|
||||
"chapters": len(chapter_files), "duration_s": round(total_s, 2),
|
||||
"cached_chapters": cached_n, "failed_chapters": failed}
|
||||
# Loudness verdict only when a preset was requested — off/None paths keep
|
||||
# the exact legacy `done` shape (additive, old clients unaffected).
|
||||
if norm in LOUDNESS_PRESETS:
|
||||
p = LOUDNESS_PRESETS[norm]
|
||||
done["loudness"] = {
|
||||
"preset": norm, "target_i": p.i, "target_tp": p.tp,
|
||||
"two_pass": measured is not None,
|
||||
"measured_i": measured.input_i if measured else None,
|
||||
}
|
||||
yield _emit(done)
|
||||
except Exception as e: # surface, don't 500 the stream
|
||||
logger.exception("[%s] longform render failed", job_id)
|
||||
if job_store is not None:
|
||||
try:
|
||||
job_store.mark_failed(job_id, str(e))
|
||||
except Exception:
|
||||
pass # best-effort job history
|
||||
# Generic message only — don't leak the stack/exception text to the client.
|
||||
yield _emit({"type": "error", "error": "render failed (see backend log)"})
|
||||
|
||||
|
||||
@router.post("/audiobook")
|
||||
async def audiobook_synthesize(req: AudiobookRequest):
|
||||
"""Synthesize a chapterized audiobook from a script, streaming SSE progress."""
|
||||
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
|
||||
return StreamingResponse(
|
||||
_render_longform_sse(
|
||||
plan, default_voice=req.default_voice, language=req.language,
|
||||
fmt=req.format, bitrate=req.bitrate,
|
||||
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
|
||||
lexicon=req.lexicon, job_type="audiobook",
|
||||
),
|
||||
media_type="text/event-stream",
|
||||
)
|
||||
|
||||
|
||||
# ── Shared longform render: Stories (and any future front door) post a plan ──
|
||||
|
||||
class LongformSpan(BaseModel):
|
||||
voice_id: str | None = None
|
||||
text: str
|
||||
pause_ms_after: int = 0
|
||||
speed: float | None = None
|
||||
|
||||
|
||||
class LongformChapter(BaseModel):
|
||||
title: str = ""
|
||||
spans: list[LongformSpan] = []
|
||||
|
||||
|
||||
class LongformRenderRequest(BaseModel):
|
||||
chapters: list[LongformChapter] = []
|
||||
default_voice: str | None = None
|
||||
language: str | None = None # None/"Auto" → profile language, else autodetect (#505)
|
||||
bitrate: str = "128k"
|
||||
format: str = "m4b"
|
||||
loudness: str | None = None
|
||||
cover_path: str | None = None
|
||||
metadata: dict | None = None
|
||||
lexicon: dict | None = None
|
||||
|
||||
|
||||
@router.post("/longform/render")
|
||||
async def longform_render(req: LongformRenderRequest):
|
||||
"""Render a pre-built chapter/span plan (the Stories Editor's compiled
|
||||
cast+lines) through the shared chapterized renderer — same resume, loudness,
|
||||
cover, metadata, and output formats as the Audiobook job."""
|
||||
from services.audiobook import AudiobookPlan, Chapter, Span
|
||||
|
||||
if len(req.chapters) > _MAX_CHAPTERS:
|
||||
raise HTTPException(status_code=422, detail=f"too many chapters (max {_MAX_CHAPTERS})")
|
||||
|
||||
chapters = []
|
||||
for i, c in enumerate(req.chapters):
|
||||
# Keep a span if it has text to speak OR a pause to render (pause-only
|
||||
# spans carry inter-line silence with empty text).
|
||||
spans = [Span(voice_id=s.voice_id, text=(s.text or "").strip(),
|
||||
pause_ms_after=max(0, int(s.pause_ms_after)), speed=s.speed)
|
||||
for s in c.spans if ((s.text and s.text.strip()) or s.pause_ms_after > 0)]
|
||||
if spans:
|
||||
chapters.append(Chapter(title=c.title or f"Chapter {i + 1}", spans=spans))
|
||||
plan = AudiobookPlan(chapters=chapters)
|
||||
return StreamingResponse(
|
||||
_render_longform_sse(
|
||||
plan, default_voice=req.default_voice, language=req.language,
|
||||
fmt=req.format, bitrate=req.bitrate,
|
||||
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
|
||||
lexicon=req.lexicon, job_type="story",
|
||||
),
|
||||
media_type="text/event-stream",
|
||||
)
|
||||
|
||||
|
||||
# ── Durable resume: interrupted longform renders ────────────────────────────
|
||||
|
||||
|
||||
def _chapters_done(job_id: str) -> int:
|
||||
"""Count chapters that finished rendering, from the job's persisted events.
|
||||
Best-effort (0 if unavailable) — used only to show resume progress."""
|
||||
try:
|
||||
from core import job_store
|
||||
n = 0
|
||||
for ev in job_store.events_since(job_id, 0, limit=100_000):
|
||||
try:
|
||||
if json.loads(ev["payload"]).get("type") == "chapter":
|
||||
n += 1
|
||||
except (ValueError, KeyError, TypeError):
|
||||
continue
|
||||
return n
|
||||
except Exception:
|
||||
return 0
|
||||
|
||||
|
||||
@router.get("/audiobook/jobs")
|
||||
def list_resumable_jobs() -> dict:
|
||||
"""List interrupted longform renders that can be resumed — a work dir that
|
||||
still holds a resume manifest (a job left mid-render by a crash/quit). The
|
||||
ids come from scanning the filesystem, so the UI can offer one-click resume."""
|
||||
from core import job_store
|
||||
|
||||
out = []
|
||||
for e in longform_resume.scan_resumable():
|
||||
jid = e["job_id"]
|
||||
manifest = longform_resume.load_manifest_file(e["manifest_path"]) or {}
|
||||
job = job_store.get(jid) or {}
|
||||
out.append({
|
||||
"job_id": jid,
|
||||
"type": e["job_type"],
|
||||
"status": job.get("status", "interrupted"),
|
||||
"title": manifest.get("title", ""),
|
||||
"total_chapters": manifest.get("total_chapters", 0),
|
||||
"chapters_done": _chapters_done(jid),
|
||||
"created_at": job.get("created_at"),
|
||||
})
|
||||
return {"jobs": out}
|
||||
|
||||
|
||||
@router.post("/audiobook/resume/{job_id}")
|
||||
async def resume_longform(job_id: str):
|
||||
"""Resume an interrupted longform render from its persisted manifest. The
|
||||
already-rendered chapters are content-addressed in the shared cache, so they
|
||||
return instantly — only the unrendered chapters synthesize again. Streams the
|
||||
same SSE event shape as the original render, under the original job_id."""
|
||||
from services.audiobook import AudiobookPlan, Chapter, Span
|
||||
|
||||
# Find the requested job among the trusted filesystem scan (every path there
|
||||
# is os.listdir-sourced, never request input) and read its manifest via the
|
||||
# scan's own trusted path — the request job_id is used ONLY to *select* an
|
||||
# entry, never to build a path. No request-controlled value reaches a file
|
||||
# operation (CodeQL py/path-injection-safe).
|
||||
entry = next((e for e in longform_resume.scan_resumable()
|
||||
if e["job_id"] == job_id), None)
|
||||
if entry is None:
|
||||
raise HTTPException(status_code=404, detail="No resumable job for that id")
|
||||
manifest = longform_resume.load_manifest_file(entry["manifest_path"])
|
||||
if not manifest:
|
||||
raise HTTPException(status_code=404, detail="No resume manifest for that job")
|
||||
|
||||
chapters = [
|
||||
Chapter(title=c.get("title", ""),
|
||||
spans=[Span(**s) for s in c.get("spans", [])])
|
||||
for c in manifest["plan"]
|
||||
]
|
||||
plan = AudiobookPlan(chapters=chapters)
|
||||
p = manifest.get("params", {})
|
||||
# Retire the interrupted job's manifest (trusted scan path) so it stops
|
||||
# showing as resumable once we've kicked off the fresh-id resume.
|
||||
longform_resume.discard_manifest_file(entry["manifest_path"])
|
||||
# Resume under a FRESH job id (job_id=None → a server uuid in the renderer).
|
||||
# The chapter cache is content-addressed (keyed by chapter content, not the
|
||||
# job id), so the already-rendered chapters still hit instantly — only the
|
||||
# unrendered ones synthesize. Using a fresh id means the request's job_id
|
||||
# never names a work dir / output file (defence-in-depth path-injection).
|
||||
return StreamingResponse(
|
||||
_render_longform_sse(
|
||||
plan, default_voice=p.get("default_voice"), language=p.get("language"),
|
||||
fmt=p.get("fmt", "m4b"), bitrate=p.get("bitrate", "128k"),
|
||||
loudness=p.get("loudness"), cover_path=p.get("cover_path"),
|
||||
metadata=p.get("metadata"), lexicon=p.get("lexicon"),
|
||||
job_type=entry["job_type"],
|
||||
),
|
||||
media_type="text/event-stream",
|
||||
)
|
||||
@@ -142,7 +142,7 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
_set_progress(job, "transcribe", 0)
|
||||
|
||||
from services.asr_backend import get_active_asr_backend
|
||||
from services.model_manager import _gpu_pool, _cpu_pool, run_on_gpu_pool_guarded
|
||||
from services.model_manager import _gpu_pool, _cpu_pool
|
||||
from services.segmentation import (
|
||||
segment_transcript, assign_speakers_heuristic,
|
||||
)
|
||||
@@ -162,12 +162,7 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
pass
|
||||
return segments, detected_lang
|
||||
|
||||
# Bound the batch transcribe (#730) so a wedged whisperx/CTranslate2 call
|
||||
# can't hold its GPU-pool worker forever and starve the rest of the backend
|
||||
# ("can't reach backend"); run_transcribe_guarded also resets the pool on
|
||||
# timeout to restore capacity.
|
||||
from services.asr_backend import run_transcribe_guarded
|
||||
segments, source_lang = await run_transcribe_guarded(_gpu_pool, _transcribe, what="Batch")
|
||||
segments, source_lang = await loop.run_in_executor(_gpu_pool, _transcribe)
|
||||
source_lang = (source_lang or "en").split("_")[0][:2].lower()
|
||||
job["segments"] = segments
|
||||
job["source_lang"] = source_lang
|
||||
@@ -303,9 +298,6 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
denoise=True, postprocess_output=True,
|
||||
)
|
||||
audio_out = audios[0]
|
||||
# TODO(#312): this route runs the OmniVoice model directly (not the active
|
||||
# backend), so VoxCPM2 never reaches it. When these routes become
|
||||
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
|
||||
mastered = apply_mastering(
|
||||
audio_out,
|
||||
sample_rate=sr,
|
||||
@@ -316,9 +308,7 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
return torch.zeros(1, int(dur * sr))
|
||||
|
||||
try:
|
||||
# Bounded + pool-reset on hang so a wedged batch segment can't
|
||||
# starve the GPU pool and brick the backend (#730 class).
|
||||
audio_tensor = await run_on_gpu_pool_guarded(_gen, what="Batch generate")
|
||||
audio_tensor = await loop.run_in_executor(_gpu_pool, _gen)
|
||||
|
||||
# Fit to slot
|
||||
target_samples_seg = int(seg_duration * sr)
|
||||
|
||||
@@ -18,26 +18,19 @@ import os
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
from fastapi import APIRouter, File, Form, HTTPException, UploadFile
|
||||
from fastapi import APIRouter, File, Form, UploadFile
|
||||
from typing import Optional
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.capture")
|
||||
|
||||
|
||||
def _truthy(value: Optional[str]) -> bool:
|
||||
"""Parse a multipart form flag. Treats '1'/'true'/'yes'/'on'/'auto'
|
||||
(any case) as on; everything else — including None — as off."""
|
||||
return (value or "").strip().lower() in {"1", "true", "yes", "on", "auto"}
|
||||
|
||||
|
||||
@router.post("/transcribe")
|
||||
async def transcribe_audio(
|
||||
audio: UploadFile = File(...),
|
||||
language: Optional[str] = Form(None),
|
||||
model: Optional[str] = Form(None),
|
||||
mode: Optional[str] = Form(None),
|
||||
refine: Optional[str] = Form(None),
|
||||
):
|
||||
"""Transcribe an audio file to text.
|
||||
|
||||
@@ -47,19 +40,10 @@ async def transcribe_audio(
|
||||
model: Whisper model size (legacy; ignored in dual-mode architecture).
|
||||
mode: 'fast' (default) uses MLX Turbo for speed; 'accurate' uses
|
||||
WhisperX with forced alignment for word-level timing.
|
||||
refine: Opt-in local-LLM cleanup of the final text (disfluencies,
|
||||
self-corrections, punctuation) — same pipeline the live
|
||||
dictation socket uses. Off by default so MCP/CLI callers don't
|
||||
pay LLM latency unless they ask; honours the user's
|
||||
Settings → Dictation-refinement config and silently passes
|
||||
through when no LLM backend is configured. The raw ``text``
|
||||
is always returned; ``refined_text`` is added only when the
|
||||
LLM actually changed something.
|
||||
|
||||
Returns:
|
||||
{
|
||||
"text": "full transcription",
|
||||
"refined_text": "cleaned text", # only when refine=true changed it
|
||||
"segments": [ {"start": 0.0, "end": 1.5, "text": "..."}, ... ],
|
||||
"language": "en",
|
||||
"duration_s": 4.2,
|
||||
@@ -96,17 +80,9 @@ async def transcribe_audio(
|
||||
return result, backend.id
|
||||
|
||||
from services.model_manager import _gpu_pool
|
||||
from services.asr_backend import ASRTimeoutError, run_transcribe_guarded
|
||||
loop = asyncio.get_running_loop()
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
result, engine_id = await run_transcribe_guarded(
|
||||
_gpu_pool, _run, what="Dictation",
|
||||
)
|
||||
except ASRTimeoutError as e:
|
||||
# Backend is alive — ASR couldn't finish. 504 with guidance, not a
|
||||
# silent hang the UI reads as "can't reach the local backend".
|
||||
logger.warning("Capture transcription timed out: %s", e)
|
||||
raise HTTPException(status_code=504, detail=str(e))
|
||||
result, engine_id = await loop.run_in_executor(_gpu_pool, _run)
|
||||
elapsed = round(time.perf_counter() - t0, 2)
|
||||
|
||||
# Normalize result shape
|
||||
@@ -115,11 +91,6 @@ async def transcribe_audio(
|
||||
if not full_text and segments:
|
||||
full_text = " ".join(s.get("text", "") for s in segments).strip()
|
||||
|
||||
# Wave 1.1: strip Whisper hallucination loops from the final text.
|
||||
# Segments keep the raw recognition so their timings stay truthful.
|
||||
from services.refinement import collapse_repetitive_artifacts
|
||||
full_text = collapse_repetitive_artifacts(full_text)
|
||||
|
||||
# Calculate audio duration from segments if available
|
||||
duration = 0.0
|
||||
if segments:
|
||||
@@ -127,24 +98,12 @@ async def transcribe_audio(
|
||||
|
||||
detected_lang = result.get("language", language or "unknown")
|
||||
|
||||
# Opt-in Wave 2.1 refinement, mirroring the live-dictation socket
|
||||
# (capture_ws). Off-thread (it's a network call, not GPU); never
|
||||
# raises — maybe_refine swallows failures and a missing LLM into a
|
||||
# None pass-through, so the raw text always stands.
|
||||
refined_text = None
|
||||
if _truthy(refine) and full_text:
|
||||
from services.refinement import maybe_refine
|
||||
refined = await asyncio.to_thread(maybe_refine, full_text)
|
||||
if refined and refined != full_text:
|
||||
refined_text = refined
|
||||
|
||||
logger.info(
|
||||
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s, refined=%s",
|
||||
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s",
|
||||
engine_id, elapsed, duration, "accurate" if use_accurate else "fast",
|
||||
refined_text is not None,
|
||||
)
|
||||
|
||||
response = {
|
||||
return {
|
||||
"text": full_text,
|
||||
"segments": [
|
||||
{
|
||||
@@ -159,9 +118,6 @@ async def transcribe_audio(
|
||||
"transcription_time_s": elapsed,
|
||||
"engine": engine_id,
|
||||
}
|
||||
if refined_text is not None:
|
||||
response["refined_text"] = refined_text
|
||||
return response
|
||||
finally:
|
||||
try:
|
||||
os.unlink(tmp.name)
|
||||
|
||||
@@ -8,12 +8,6 @@ live dictation feedback.
|
||||
Protocol:
|
||||
→ Client sends binary audio frames (16-bit PCM or WebM/Opus blobs)
|
||||
← Server sends JSON messages:
|
||||
|
||||
Opt-in AEC mode (``?aec=1[&sr=16000]``, parity Action 8b): for dictating
|
||||
while the app plays audio. Frames must be raw int16 mono PCM, each tagged
|
||||
with a 1-byte prefix — 0x00 = microphone, 0x01 = playback reference. The
|
||||
server runs an NLMS echo canceller, cleaning the mic against the reference
|
||||
before transcription. Without the param the protocol is unchanged.
|
||||
{"type": "partial", "text": "Hello wor..."} — interim result
|
||||
{"type": "final", "text": "Hello world.", — committed result
|
||||
"segments": [...], "language": "en",
|
||||
@@ -31,7 +25,7 @@ import time
|
||||
|
||||
from fastapi import APIRouter, WebSocket, WebSocketDisconnect
|
||||
|
||||
from api.dependencies import _LOOPBACK_HOSTS, ws_remote_authorized
|
||||
from api.dependencies import _LOOPBACK_HOSTS
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.capture_ws")
|
||||
@@ -50,80 +44,6 @@ MIN_BUFFER_BYTES = 64000 # ~2s of 16-bit mono 16kHz — needs enough WebM frame
|
||||
# to transcribe whatever the user recorded, even short utterances.
|
||||
MIN_FINAL_BUFFER_BYTES = 4000 # ~125ms of 16-bit mono 16kHz
|
||||
|
||||
# ── Dictate-over-playback AEC (parity Action 8b, opt-in) ──────────────────
|
||||
# Activated by the ``?aec=1`` query param. When OFF (the default), the
|
||||
# protocol and behaviour are byte-for-byte unchanged. When ON, the client
|
||||
# streams raw int16 mono PCM frames tagged with a 1-byte type prefix so the
|
||||
# server can tell mic audio from the playback reference it must cancel:
|
||||
_AEC_NEAR = 0x00 # microphone frame (clean it, then buffer for ASR)
|
||||
_AEC_FAR = 0x01 # playback reference frame (feed the echo model only)
|
||||
|
||||
|
||||
def _demux_aec_frame(data: bytes) -> tuple[str, bytes]:
|
||||
"""Split a prefixed AEC binary frame into ``(kind, pcm)``.
|
||||
|
||||
``kind`` is ``"near"`` (mic) or ``"far"`` (playback reference). An empty
|
||||
or prefix-only frame yields an empty payload. Unknown prefixes are treated
|
||||
as ``"near"`` so a malformed tag degrades to plain dictation rather than
|
||||
dropping audio.
|
||||
"""
|
||||
if not data:
|
||||
return "near", b""
|
||||
kind = "far" if data[0] == _AEC_FAR else "near"
|
||||
return kind, data[1:]
|
||||
|
||||
|
||||
def _pcm16_to_wav(pcm: bytes, sample_rate: int) -> str | None:
|
||||
"""Write raw int16 mono PCM to a temp WAV via stdlib ``wave`` (no ffmpeg).
|
||||
|
||||
Used on the AEC path, where frames are already decoded PCM — the cleaned
|
||||
samples have no container, so the ffmpeg-sniffing ``_chunks_to_wav`` would
|
||||
misdetect them. Returns the temp path, or ``None`` for a too-short buffer.
|
||||
"""
|
||||
if not pcm or len(pcm) < 100:
|
||||
return None
|
||||
import wave
|
||||
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".wav")
|
||||
tmp.close()
|
||||
try:
|
||||
with wave.open(tmp.name, "wb") as wf:
|
||||
wf.setnchannels(1)
|
||||
wf.setsampwidth(2) # int16
|
||||
wf.setframerate(sample_rate)
|
||||
wf.writeframes(pcm)
|
||||
return tmp.name
|
||||
except Exception as e:
|
||||
logger.debug("PCM->WAV failed: %s", e)
|
||||
try:
|
||||
os.unlink(tmp.name)
|
||||
except OSError:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def _select_sherpa_spec(websocket: WebSocket):
|
||||
"""Resolve the sherpa dictation model for this WS session, or None.
|
||||
|
||||
A ``?model=<id>`` query param wins (the frontend can pin a model per
|
||||
session); otherwise the persisted ``dictation.model_id`` pref is used (only
|
||||
when dictation is enabled). Returns the :class:`SherpaModelSpec` or None
|
||||
(None → the legacy Whisper/WebM path runs unchanged).
|
||||
"""
|
||||
try:
|
||||
from services import sherpa_dictation as sd
|
||||
except Exception:
|
||||
return None
|
||||
requested = websocket.query_params.get("model")
|
||||
if requested:
|
||||
return sd.get_spec(requested) # explicit selection (may be None if bad)
|
||||
# Fall back to the persisted dictation pref.
|
||||
try:
|
||||
from services.asr_backend import dictation_model_id
|
||||
mid = dictation_model_id()
|
||||
except Exception:
|
||||
mid = None
|
||||
return sd.get_spec(mid) if mid else None
|
||||
|
||||
|
||||
@router.websocket("/ws/transcribe")
|
||||
async def ws_transcribe(websocket: WebSocket):
|
||||
@@ -133,51 +53,13 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
# WebSocket dependency injection differs across FastAPI versions, so we
|
||||
# inline the check before accept(). Without it, any local process could
|
||||
# stream the user's microphone over this endpoint.
|
||||
# Wave 2.3 (remote backend): a non-loopback client that presents the
|
||||
# OMNIVOICE_API_KEY bearer is the thin-client dictation case — the mic
|
||||
# lives on the user's machine, the GPU here — and is allowed through.
|
||||
host = websocket.client.host if websocket.client else None
|
||||
if host not in _LOOPBACK_HOSTS and not ws_remote_authorized(websocket):
|
||||
if host not in _LOOPBACK_HOSTS:
|
||||
await websocket.close(code=1008, reason="loopback origin required")
|
||||
return
|
||||
|
||||
await websocket.accept()
|
||||
|
||||
# Live-dictation engine selection. When a sherpa-onnx model is selected
|
||||
# (via ?model= or the dictation.model_id pref) AND sherpa is installed,
|
||||
# run the dedicated low-latency handler. Otherwise fall through to the
|
||||
# legacy Whisper/WebM path, byte-for-byte unchanged.
|
||||
spec = _select_sherpa_spec(websocket)
|
||||
if spec is not None:
|
||||
from services.asr_backend import SherpaDictationBackend
|
||||
ok, _reason = SherpaDictationBackend.is_available()
|
||||
if ok:
|
||||
if spec.streaming:
|
||||
await _run_sherpa_streaming(websocket, spec)
|
||||
else:
|
||||
await _run_sherpa_offline(websocket, spec)
|
||||
return
|
||||
# sherpa not installed → fall through to the legacy path so the user
|
||||
# still gets dictation (just not live partials).
|
||||
logger.info("sherpa dictation selected but unavailable — legacy path")
|
||||
|
||||
# Opt-in dictate-over-playback AEC (parity Action 8b). Default OFF →
|
||||
# identical legacy behaviour. When on, frames are 1-byte-tagged raw PCM
|
||||
# and the cleaned mic stream is muxed via stdlib wave (not ffmpeg).
|
||||
aec = None
|
||||
pcm_sr: int | None = None
|
||||
if websocket.query_params.get("aec") in ("1", "true", "on"):
|
||||
try:
|
||||
pcm_sr = int(websocket.query_params.get("sr", "16000"))
|
||||
from services.aec import NlmsEchoCanceller
|
||||
aec = NlmsEchoCanceller(sample_rate=pcm_sr)
|
||||
logger.info("AEC enabled for dictation session (sr=%d)", pcm_sr)
|
||||
except Exception as e:
|
||||
# Bad sr or import failure → fall back to plain dictation.
|
||||
logger.warning("AEC requested but disabled: %s", e)
|
||||
aec = None
|
||||
pcm_sr = None
|
||||
|
||||
audio_chunks: list[bytes] = []
|
||||
total_bytes = 0
|
||||
last_audio_time = time.monotonic()
|
||||
@@ -215,16 +97,6 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
# Empty binary frame also acts as EOF — connection stays open.
|
||||
running = False
|
||||
break
|
||||
if aec is not None:
|
||||
# Tagged PCM: route the playback reference into the echo
|
||||
# model and clean the mic before it reaches the buffer.
|
||||
kind, payload = _demux_aec_frame(data)
|
||||
if kind == "far":
|
||||
aec.push_far_end(payload)
|
||||
continue
|
||||
if not payload:
|
||||
continue
|
||||
data = aec.process_near_end(payload)
|
||||
audio_chunks.append(data)
|
||||
total_bytes += len(data)
|
||||
last_audio_time = time.monotonic()
|
||||
@@ -271,7 +143,7 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
|
||||
# Transcribe current buffer
|
||||
try:
|
||||
text = await _transcribe_buffer(audio_chunks[:], pcm_sr=pcm_sr)
|
||||
text = await _transcribe_buffer(audio_chunks[:])
|
||||
if text and text != partial_text:
|
||||
partial_text = text
|
||||
await _safe_send({
|
||||
@@ -301,16 +173,7 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
# Final transcription on complete buffer — skip if client already gone.
|
||||
if total_bytes > MIN_FINAL_BUFFER_BYTES:
|
||||
try:
|
||||
result = await _transcribe_buffer_full(audio_chunks, pcm_sr=pcm_sr)
|
||||
# Wave 2.1: optional local-LLM refinement of the final text.
|
||||
# Off-thread (network call, not GPU); pass-through on any
|
||||
# failure or when no LLM backend is configured. The raw text
|
||||
# always ships too — clients paste refined_text ?? text.
|
||||
if result.get("text"):
|
||||
from services.refinement import maybe_refine
|
||||
refined = await asyncio.to_thread(maybe_refine, result["text"])
|
||||
if refined and refined != result["text"]:
|
||||
result["refined_text"] = refined
|
||||
result = await _transcribe_buffer_full(audio_chunks)
|
||||
if not await _safe_send({"type": "final", **result}):
|
||||
logger.debug("Skipped final send — client already disconnected")
|
||||
except Exception as e:
|
||||
@@ -334,341 +197,24 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
pass
|
||||
|
||||
|
||||
# ── sherpa-onnx live dictation handlers ─────────────────────────────────────
|
||||
#
|
||||
# Both handlers read raw int16 mono PCM frames (reusing the AEC framing: an
|
||||
# opt-in 1-byte type prefix when ?aec=1, else bare PCM) at ?sr= (default 16000).
|
||||
# This is the low-latency transport — no WebM/ffmpeg in the hot path.
|
||||
|
||||
# How often the offline-kind handler re-decodes the growing buffer for a live
|
||||
# partial (streaming-kind decodes every frame, no cadence needed).
|
||||
SHERPA_OFFLINE_PARTIAL_S = float(os.environ.get("OMNIVOICE_SHERPA_OFFLINE_PARTIAL", "0.8"))
|
||||
|
||||
|
||||
def _pcm16_to_f32(pcm: bytes):
|
||||
"""int16 little-endian mono PCM bytes → float32 numpy in [-1, 1]."""
|
||||
import numpy as np
|
||||
if not pcm:
|
||||
return np.zeros(0, dtype=np.float32)
|
||||
# Guard against an odd trailing byte from a split frame.
|
||||
if len(pcm) % 2:
|
||||
pcm = pcm[:-1]
|
||||
return np.frombuffer(pcm, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
|
||||
|
||||
async def _sherpa_session(websocket: WebSocket):
|
||||
"""Shared WS receive setup for the sherpa handlers.
|
||||
|
||||
Returns ``(get_frame, state)`` where ``get_frame`` is an async callable
|
||||
that yields the next near-end (mic) PCM bytes, ``b""`` for a keepalive/ref
|
||||
frame, or ``None`` on EOF/disconnect. ``state`` carries sample rate, AEC,
|
||||
and the disconnect flag for the caller's finaliser.
|
||||
"""
|
||||
pcm_sr = 16000
|
||||
try:
|
||||
pcm_sr = int(websocket.query_params.get("sr", "16000"))
|
||||
except (TypeError, ValueError):
|
||||
pcm_sr = 16000
|
||||
aec = None
|
||||
if websocket.query_params.get("aec") in ("1", "true", "on"):
|
||||
try:
|
||||
from services.aec import NlmsEchoCanceller
|
||||
aec = NlmsEchoCanceller(sample_rate=pcm_sr)
|
||||
except Exception as e:
|
||||
logger.warning("AEC requested but disabled (sherpa): %s", e)
|
||||
aec = None
|
||||
return pcm_sr, aec
|
||||
|
||||
|
||||
async def _recv_pcm_frame(websocket: WebSocket, aec):
|
||||
"""Receive one frame; return (kind, pcm_bytes).
|
||||
|
||||
kind ∈ {"near","eof","skip"}. Demuxes AEC-tagged frames when ``aec`` is on
|
||||
and feeds the playback reference into the canceller. A text "EOF" or an
|
||||
empty/closed socket yields kind "eof".
|
||||
"""
|
||||
msg = await websocket.receive()
|
||||
mtype = msg.get("type")
|
||||
if mtype == "websocket.disconnect":
|
||||
return "eof", b""
|
||||
if mtype != "websocket.receive":
|
||||
return "skip", b""
|
||||
data = msg.get("bytes")
|
||||
if data is not None:
|
||||
if len(data) == 0:
|
||||
return "eof", b""
|
||||
if aec is not None:
|
||||
kind, payload = _demux_aec_frame(data)
|
||||
if kind == "far":
|
||||
aec.push_far_end(payload)
|
||||
return "skip", b""
|
||||
if not payload:
|
||||
return "skip", b""
|
||||
return "near", aec.process_near_end(payload)
|
||||
return "near", data
|
||||
if msg.get("text") == "EOF":
|
||||
return "eof", b""
|
||||
return "skip", b""
|
||||
|
||||
|
||||
async def _run_sherpa_streaming(websocket: WebSocket, spec):
|
||||
"""True streaming: feed the OnlineRecognizer frame-by-frame, emit `partial`
|
||||
every time the decoded text grows, and `final` on sherpa's endpoint (silence)
|
||||
detection and on EOF. <300ms perceived latency on CPU for the tiny models.
|
||||
"""
|
||||
import numpy as np
|
||||
from services.asr_backend import SherpaDictationBackend
|
||||
|
||||
pcm_sr, aec = await _sherpa_session(websocket)
|
||||
logger.info("sherpa streaming dictation: model=%s sr=%d aec=%s",
|
||||
spec.id, pcm_sr, bool(aec))
|
||||
|
||||
backend = SherpaDictationBackend(model_id=spec.id)
|
||||
# Build the recognizer off the event loop (download-on-first-use + ONNX
|
||||
# session init can take a moment); keep the socket responsive.
|
||||
try:
|
||||
await asyncio.to_thread(backend.ensure_loaded)
|
||||
except Exception as e:
|
||||
logger.error("sherpa streaming load failed: %s", e)
|
||||
try:
|
||||
await websocket.send_json({"type": "error", "detail": str(e)})
|
||||
await websocket.close()
|
||||
except Exception:
|
||||
pass
|
||||
return
|
||||
rec = backend._rec
|
||||
stream = rec.create_stream()
|
||||
|
||||
last_partial = ""
|
||||
committed: list[str] = [] # finalized utterances this session
|
||||
client_disconnected = False
|
||||
|
||||
async def _send(payload) -> bool:
|
||||
nonlocal client_disconnected
|
||||
if client_disconnected:
|
||||
return False
|
||||
try:
|
||||
await websocket.send_json(payload)
|
||||
return True
|
||||
except Exception:
|
||||
client_disconnected = True
|
||||
return False
|
||||
|
||||
def _decode_after_feed(pcm: bytes):
|
||||
"""Blocking: feed one PCM frame, decode, return (text, is_endpoint).
|
||||
Runs in a thread so the ONNX work never blocks the event loop."""
|
||||
samples = _pcm16_to_f32(pcm)
|
||||
if len(samples):
|
||||
stream.accept_waveform(pcm_sr, samples)
|
||||
while rec.is_ready(stream):
|
||||
rec.decode_stream(stream)
|
||||
endpoint = rec.is_endpoint(stream)
|
||||
text = (rec.get_result(stream) or "").strip()
|
||||
return text, endpoint
|
||||
|
||||
def _flush_final():
|
||||
"""Blocking: pad + drain the stream for the trailing utterance."""
|
||||
tail = np.zeros(int(0.5 * pcm_sr), dtype=np.float32)
|
||||
stream.accept_waveform(pcm_sr, tail)
|
||||
stream.input_finished()
|
||||
while rec.is_ready(stream):
|
||||
rec.decode_stream(stream)
|
||||
return (rec.get_result(stream) or "").strip()
|
||||
|
||||
try:
|
||||
while True:
|
||||
kind, pcm = await _recv_pcm_frame(websocket, aec)
|
||||
if kind == "eof":
|
||||
break
|
||||
if kind == "skip":
|
||||
continue
|
||||
text, endpoint = await asyncio.to_thread(_decode_after_feed, pcm)
|
||||
if endpoint:
|
||||
# Commit this utterance; reset for the next one.
|
||||
if text:
|
||||
committed.append(text)
|
||||
await _send({"type": "final", "text": text,
|
||||
"segments": [{"start": 0.0, "end": None, "text": text}],
|
||||
"language": "auto", "engine": backend.id})
|
||||
rec.reset(stream)
|
||||
last_partial = ""
|
||||
elif text and text != last_partial:
|
||||
last_partial = text
|
||||
await _send({"type": "partial", "text": text})
|
||||
except WebSocketDisconnect:
|
||||
client_disconnected = True
|
||||
except Exception as e:
|
||||
logger.warning("sherpa streaming loop ended: %s", e)
|
||||
client_disconnected = True
|
||||
|
||||
# Drain the trailing (un-endpointed) utterance on EOF.
|
||||
try:
|
||||
tail_text = await asyncio.to_thread(_flush_final)
|
||||
except Exception as e:
|
||||
logger.debug("sherpa streaming flush failed: %s", e)
|
||||
tail_text = ""
|
||||
if tail_text and tail_text != (committed[-1] if committed else None):
|
||||
committed.append(tail_text)
|
||||
|
||||
full = " ".join(t for t in committed if t).strip()
|
||||
segments = [{"start": 0.0, "end": None, "text": t} for t in committed if t]
|
||||
if not client_disconnected:
|
||||
if full:
|
||||
try:
|
||||
from services.refinement import maybe_refine
|
||||
refined = await asyncio.to_thread(maybe_refine, full)
|
||||
except Exception:
|
||||
refined = None
|
||||
payload = {"type": "final", "text": full, "segments": segments,
|
||||
"language": "auto", "engine": backend.id}
|
||||
if refined and refined != full:
|
||||
payload["refined_text"] = refined
|
||||
await _send(payload)
|
||||
else:
|
||||
await _send({"type": "final", "text": "", "segments": [],
|
||||
"language": "auto", "engine": backend.id})
|
||||
try:
|
||||
await websocket.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
async def _run_sherpa_offline(websocket: WebSocket, spec):
|
||||
"""Offline-kind sherpa model with live partials: buffer raw PCM and
|
||||
re-decode the growing buffer every ~800ms so the user still sees text
|
||||
appear while speaking; finalize on EOF/silence."""
|
||||
from services.asr_backend import SherpaDictationBackend
|
||||
|
||||
pcm_sr, aec = await _sherpa_session(websocket)
|
||||
logger.info("sherpa offline dictation: model=%s sr=%d aec=%s",
|
||||
spec.id, pcm_sr, bool(aec))
|
||||
|
||||
backend = SherpaDictationBackend(model_id=spec.id)
|
||||
try:
|
||||
await asyncio.to_thread(backend.ensure_loaded)
|
||||
except Exception as e:
|
||||
logger.error("sherpa offline load failed: %s", e)
|
||||
try:
|
||||
await websocket.send_json({"type": "error", "detail": str(e)})
|
||||
await websocket.close()
|
||||
except Exception:
|
||||
pass
|
||||
return
|
||||
|
||||
buf = bytearray()
|
||||
last_partial = ""
|
||||
running = True
|
||||
client_disconnected = False
|
||||
last_audio = time.monotonic()
|
||||
|
||||
async def _send(payload) -> bool:
|
||||
nonlocal client_disconnected
|
||||
if client_disconnected:
|
||||
return False
|
||||
try:
|
||||
await websocket.send_json(payload)
|
||||
return True
|
||||
except Exception:
|
||||
client_disconnected = True
|
||||
return False
|
||||
|
||||
def _decode_buffer() -> str:
|
||||
samples = _pcm16_to_f32(bytes(buf))
|
||||
if not len(samples):
|
||||
return ""
|
||||
return backend._decode_offline(samples, pcm_sr)
|
||||
|
||||
async def receive():
|
||||
nonlocal running, client_disconnected, last_audio
|
||||
try:
|
||||
while running:
|
||||
kind, pcm = await _recv_pcm_frame(websocket, aec)
|
||||
if kind == "eof":
|
||||
running = False
|
||||
break
|
||||
if kind == "skip":
|
||||
continue
|
||||
buf.extend(pcm)
|
||||
last_audio = time.monotonic()
|
||||
except WebSocketDisconnect:
|
||||
client_disconnected = True
|
||||
running = False
|
||||
except Exception as e:
|
||||
logger.debug("sherpa offline receive ended: %s", e)
|
||||
running = False
|
||||
|
||||
async def partials():
|
||||
nonlocal last_partial, running
|
||||
while running:
|
||||
await asyncio.sleep(SHERPA_OFFLINE_PARTIAL_S)
|
||||
if not running or len(buf) < 2000:
|
||||
continue
|
||||
try:
|
||||
text = await asyncio.to_thread(_decode_buffer)
|
||||
except Exception as e:
|
||||
logger.debug("sherpa offline partial failed: %s", e)
|
||||
continue
|
||||
if text and text != last_partial:
|
||||
last_partial = text
|
||||
await _send({"type": "partial", "text": text})
|
||||
|
||||
recv_task = asyncio.create_task(receive())
|
||||
part_task = asyncio.create_task(partials())
|
||||
await asyncio.wait([recv_task, part_task], return_when=asyncio.FIRST_COMPLETED)
|
||||
running = False
|
||||
for t in (recv_task, part_task):
|
||||
if not t.done():
|
||||
t.cancel()
|
||||
try:
|
||||
await t
|
||||
except (asyncio.CancelledError, Exception):
|
||||
pass
|
||||
|
||||
try:
|
||||
full = await asyncio.to_thread(_decode_buffer)
|
||||
except Exception as e:
|
||||
logger.error("sherpa offline final failed: %s", e)
|
||||
full = ""
|
||||
full = (full or "").strip()
|
||||
segments = [{"start": 0.0, "end": None, "text": full}] if full else []
|
||||
if not client_disconnected:
|
||||
payload = {"type": "final", "text": full, "segments": segments,
|
||||
"language": "auto", "engine": backend.id}
|
||||
if full:
|
||||
try:
|
||||
from services.refinement import maybe_refine
|
||||
refined = await asyncio.to_thread(maybe_refine, full)
|
||||
if refined and refined != full:
|
||||
payload["refined_text"] = refined
|
||||
except Exception:
|
||||
pass
|
||||
await _send(payload)
|
||||
try:
|
||||
await websocket.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None) -> str:
|
||||
async def _transcribe_buffer(chunks: list[bytes]) -> str:
|
||||
"""Quick partial transcription of the current audio buffer."""
|
||||
|
||||
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
|
||||
tmp = _chunks_to_wav(chunks)
|
||||
if tmp is None:
|
||||
return ""
|
||||
|
||||
try:
|
||||
from services.model_manager import _gpu_pool
|
||||
from services.asr_backend import get_capture_asr_backend, run_transcribe_guarded
|
||||
from services.asr_backend import get_capture_asr_backend
|
||||
|
||||
def _run():
|
||||
backend = get_capture_asr_backend()
|
||||
result = backend.transcribe(tmp, word_timestamps=False)
|
||||
return result.get("text", "")
|
||||
|
||||
# Bound dictation transcribes (#730): a wedged whisperx/CTranslate2 call
|
||||
# must not hold its GPU-pool worker forever and starve TTS / other ASR
|
||||
# into a "can't reach backend"; on timeout the pool is reset to recover.
|
||||
text = await run_transcribe_guarded(_gpu_pool, _run, what="Dictation")
|
||||
loop = asyncio.get_running_loop()
|
||||
text = await loop.run_in_executor(_gpu_pool, _run)
|
||||
return text.strip()
|
||||
finally:
|
||||
try:
|
||||
@@ -677,16 +223,16 @@ async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None)
|
||||
pass
|
||||
|
||||
|
||||
async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = None) -> dict:
|
||||
async def _transcribe_buffer_full(chunks: list[bytes]) -> dict:
|
||||
"""Full transcription with timing info for the final result."""
|
||||
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
|
||||
tmp = _chunks_to_wav(chunks)
|
||||
if tmp is None:
|
||||
return {"text": "", "segments": [], "language": "unknown",
|
||||
"duration_s": 0, "transcription_time_s": 0, "engine": "none"}
|
||||
|
||||
try:
|
||||
from services.model_manager import _gpu_pool
|
||||
from services.asr_backend import get_capture_asr_backend, run_transcribe_guarded
|
||||
from services.asr_backend import get_capture_asr_backend
|
||||
|
||||
def _run():
|
||||
backend = get_capture_asr_backend()
|
||||
@@ -699,12 +245,6 @@ async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = N
|
||||
if not full_text and segments:
|
||||
full_text = " ".join(s.get("text", "") for s in segments).strip()
|
||||
|
||||
# Wave 1.1: strip Whisper hallucination loops from the final
|
||||
# text (the string that gets auto-pasted). Segments keep the
|
||||
# raw recognition so their timings stay truthful.
|
||||
from services.refinement import collapse_repetitive_artifacts
|
||||
full_text = collapse_repetitive_artifacts(full_text)
|
||||
|
||||
duration = max((s.get("end", 0) for s in segments), default=0.0)
|
||||
|
||||
return {
|
||||
@@ -721,9 +261,8 @@ async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = N
|
||||
"engine": backend.id,
|
||||
}
|
||||
|
||||
# Bounded + pool-resetting on timeout (#730), same rationale as the
|
||||
# partial path above.
|
||||
return await run_transcribe_guarded(_gpu_pool, _run, what="Dictation")
|
||||
loop = asyncio.get_running_loop()
|
||||
return await loop.run_in_executor(_gpu_pool, _run)
|
||||
finally:
|
||||
try:
|
||||
os.unlink(tmp)
|
||||
|
||||
@@ -258,19 +258,13 @@ async def community_use(item_id: str, name: Optional[str] = Query(None)):
|
||||
raise HTTPException(status_code=503, detail=f"Couldn't add this voice right now. Error: {e}")
|
||||
|
||||
try:
|
||||
# A community "preset" is a synthetic designed voice (rendered from an
|
||||
# instruct string) → kind='design'; a "voice" carries a real reference
|
||||
# clip → kind='clone'. Setting kind makes the persona-gallery
|
||||
# synthetic-only gating work (§R3) instead of defaulting all imports to
|
||||
# 'clone'.
|
||||
kind = "design" if item["type"] == "preset" else "clone"
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO voice_profiles "
|
||||
"(id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at, kind) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
"(id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(profile_id, profile_name, audio_filename, ref_text, instruct,
|
||||
item.get("language", "Auto"), None, item["id"], time.time(), kind),
|
||||
item.get("language", "Auto"), None, item["id"], time.time()),
|
||||
)
|
||||
except Exception:
|
||||
with __import__("contextlib").suppress(OSError):
|
||||
|
||||
@@ -1,35 +0,0 @@
|
||||
"""Voice-design "describe your voice" API (issue #317).
|
||||
|
||||
Maps a free-text voice description onto the existing design parameter space
|
||||
via the deterministic keyword mapper in ``core.describe_voice``. Pure CPU +
|
||||
stdlib — no model, no network — so it imports and responds instantly in any
|
||||
environment, including test/CI without model weights.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from core.describe_voice import parse_description
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
class DescribeRequest(BaseModel):
|
||||
description: str = Field(default="", max_length=2000)
|
||||
|
||||
|
||||
@router.post("/design/describe")
|
||||
def describe_voice(req: DescribeRequest) -> dict:
|
||||
"""Parse a free-text description into design attrs + a validator-safe instruct.
|
||||
|
||||
Response shape::
|
||||
|
||||
{
|
||||
"attrs": {"Gender": "female", "Age": "elderly", ... or "Auto"},
|
||||
"instruct": "female, elderly, low pitch, british accent",
|
||||
"matched": [{"category": "Age", "token": "elderly", "phrase": "elderly"}, ...],
|
||||
"unmatched": ["slightly raspy"]
|
||||
}
|
||||
"""
|
||||
return parse_description(req.description)
|
||||
@@ -1,127 +0,0 @@
|
||||
"""
|
||||
Dictation router — sherpa-onnx live-dictation engine.
|
||||
|
||||
Exposes the seven sherpa-onnx dictation models and the dictation prefs the
|
||||
frontend dictation UI binds to.
|
||||
|
||||
GET /dictation/models → the 7 models + install state (frontend model list)
|
||||
GET /dictation/prefs → { enabled, mode, model_id }
|
||||
POST /dictation/prefs → persist any subset of those prefs
|
||||
|
||||
Install state reuses the same HF-cache check the model store uses, so a model
|
||||
shown "installed" here is the same snapshot the backend will load.
|
||||
|
||||
Prefs are stored in the shared ``prefs.json`` store under the ``dictation.*``
|
||||
namespace (``dictation.enabled``, ``dictation.mode``, ``dictation.model_id``),
|
||||
mirroring how the ASR/TTS engine picks persist.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
from typing import Optional
|
||||
|
||||
from api.dependencies import require_loopback
|
||||
from core import prefs
|
||||
from services import sherpa_dictation as sd
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.dictation")
|
||||
|
||||
# Pref keys (the binding contract — the frontend writes exactly these).
|
||||
PREF_ENABLED = "dictation.enabled"
|
||||
PREF_MODE = "dictation.mode"
|
||||
PREF_MODEL_ID = "dictation.model_id"
|
||||
|
||||
_DEFAULT_ENABLED = True
|
||||
_DEFAULT_MODE = "toggle"
|
||||
_VALID_MODES = ("toggle", "hold")
|
||||
|
||||
|
||||
def _read_prefs() -> dict:
|
||||
mid = prefs.get(PREF_MODEL_ID, sd.DEFAULT_MODEL_ID)
|
||||
if not sd.is_sherpa_model(mid):
|
||||
mid = sd.DEFAULT_MODEL_ID
|
||||
mode = prefs.get(PREF_MODE, _DEFAULT_MODE)
|
||||
if mode not in _VALID_MODES:
|
||||
mode = _DEFAULT_MODE
|
||||
return {
|
||||
"enabled": bool(prefs.get(PREF_ENABLED, _DEFAULT_ENABLED)),
|
||||
"mode": mode,
|
||||
"model_id": mid,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/dictation/models", dependencies=[Depends(require_loopback)])
|
||||
def list_dictation_models():
|
||||
"""The seven sherpa-onnx dictation models + install state.
|
||||
|
||||
Each entry: id, repo_id, label, tag ("offline"|"streaming"), recommended,
|
||||
size_gb, languages, kind, and install state (installed/installing). The
|
||||
``installed`` flag is computed from the same HF cache the model store reads,
|
||||
so it matches the model-store row state.
|
||||
"""
|
||||
available, reason = sd.sherpa_available()
|
||||
out = []
|
||||
for spec in sd.list_specs():
|
||||
out.append({
|
||||
"id": spec.id,
|
||||
"repo_id": spec.repo_id,
|
||||
"label": spec.label,
|
||||
"tag": spec.tag,
|
||||
"recommended": spec.recommended,
|
||||
"size_gb": spec.size_gb,
|
||||
"languages": spec.languages,
|
||||
"kind": spec.kind,
|
||||
"installed": sd.is_installed(spec),
|
||||
})
|
||||
return {
|
||||
"models": out,
|
||||
"engine_available": available,
|
||||
"engine_reason": None if available else reason,
|
||||
"default_model_id": sd.DEFAULT_MODEL_ID,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/dictation/prefs", dependencies=[Depends(require_loopback)])
|
||||
def get_dictation_prefs():
|
||||
return _read_prefs()
|
||||
|
||||
|
||||
class DictationPrefsUpdate(BaseModel):
|
||||
enabled: Optional[bool] = None
|
||||
mode: Optional[str] = None
|
||||
model_id: Optional[str] = None
|
||||
|
||||
|
||||
@router.post("/dictation/prefs", dependencies=[Depends(require_loopback)])
|
||||
def set_dictation_prefs(req: DictationPrefsUpdate):
|
||||
"""Persist any subset of the dictation prefs. Validates ``mode`` and
|
||||
``model_id`` so a bad value can't wedge the capture engine."""
|
||||
if req.mode is not None:
|
||||
if req.mode not in _VALID_MODES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"mode must be one of {_VALID_MODES}",
|
||||
)
|
||||
prefs.set_(PREF_MODE, req.mode)
|
||||
if req.model_id is not None:
|
||||
if not sd.is_sherpa_model(req.model_id):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"unknown dictation model_id {req.model_id!r}",
|
||||
)
|
||||
# Normalise to the canonical dictation id (accept repo_id too).
|
||||
prefs.set_(PREF_MODEL_ID, sd.get_spec(req.model_id).id)
|
||||
if req.enabled is not None:
|
||||
prefs.set_(PREF_ENABLED, bool(req.enabled))
|
||||
# Rebuild the cached capture singleton so the change takes effect at once.
|
||||
try:
|
||||
from services import asr_backend
|
||||
asr_backend._capture_backend = None
|
||||
asr_backend._capture_backend_key = None
|
||||
except Exception:
|
||||
pass
|
||||
return _read_prefs()
|
||||
+13
-167
@@ -23,36 +23,13 @@ from services.segmentation import (
|
||||
assign_speakers_from_diarization,
|
||||
assign_speakers_from_turns,
|
||||
assign_speakers_heuristic,
|
||||
resplit_segments_by_diarization,
|
||||
resplit_segments_by_turns,
|
||||
_words_from_whisper,
|
||||
clean_up_segments,
|
||||
)
|
||||
from services.onset_align import snap_segment_starts
|
||||
from services import dub_pipeline
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.api")
|
||||
|
||||
|
||||
def _reset_pool_on_wedge(pool) -> None:
|
||||
"""Abandon a GPU pool whose worker is wedged on a timed-out transcribe (#730).
|
||||
|
||||
Python can't kill the stuck thread, but dropping the poisoned pool means the
|
||||
next submit (the next chunk, or a concurrent TTS generate) gets a fresh
|
||||
worker instead of queueing behind the wedged one — the same recovery the
|
||||
whole-file paths get inside ``run_transcribe_guarded``. Best-effort and a
|
||||
no-op for a pool without ``reset`` (a plain executor), so it never raises on
|
||||
the failure path it's trying to recover from.
|
||||
"""
|
||||
_reset = getattr(pool, "reset", None)
|
||||
if callable(_reset):
|
||||
try:
|
||||
_reset()
|
||||
except Exception:
|
||||
logger.exception("GPU pool reset after transcribe timeout failed")
|
||||
|
||||
|
||||
# ── Legacy-name aliases to services/dub_pipeline.py ────────────────────────
|
||||
# Phase 2.4 moved the business logic into a service. Other routers
|
||||
# (dub_generate, dub_translate, dub_export) + internal call sites below still
|
||||
@@ -389,11 +366,7 @@ _prep_event_helper = dub_pipeline.prep_event # alias; we keep the module-local
|
||||
|
||||
|
||||
@router.get("/dub/transcribe-stream/{job_id}")
|
||||
async def dub_transcribe_stream(
|
||||
job_id: str,
|
||||
num_speakers: Optional[int] = None,
|
||||
per_segment_refs: bool = True,
|
||||
):
|
||||
async def dub_transcribe_stream(job_id: str, num_speakers: Optional[int] = None):
|
||||
"""Stream per-chunk segments via SSE, then emit diarized final pass.
|
||||
|
||||
Pre-flight checks (missing job, missing audio, ASR not loaded) are emitted
|
||||
@@ -447,22 +420,10 @@ async def dub_transcribe_stream(
|
||||
try:
|
||||
# The PyTorch-Whisper backend lazily builds its own pipeline
|
||||
# when no preloaded `_asr_pipe` is present (issue #255), so it
|
||||
# no longer needs OMNIVOICE_PRELOAD_TTS_ASR=1.
|
||||
# no longer needs OMNIVOICE_PRELOAD_TTS_ASR=1 — don't reject it
|
||||
# here; any load failure surfaces per-chunk with a real cause.
|
||||
_asr_backend = get_active_asr_backend(asr_pipe=getattr(_model, "_asr_pipe", None))
|
||||
# Eagerly load the model HERE so a real load failure (e.g.
|
||||
# WhisperX: missing weights, CTranslate2/cuDNN mismatch, the
|
||||
# torch-2.6 weights-only VAD regression) surfaces once, with
|
||||
# its actual cause, as a clean preflight `error` event —
|
||||
# instead of being buried in N cryptic per-chunk failures
|
||||
# and retried on every chunk (#578). Run in a thread so the
|
||||
# (blocking) load doesn't stall the event loop.
|
||||
_ensure_loaded = getattr(_asr_backend, "ensure_loaded", None)
|
||||
if callable(_ensure_loaded):
|
||||
await asyncio.get_running_loop().run_in_executor(
|
||||
_gpu_pool, _ensure_loaded
|
||||
)
|
||||
except Exception as e:
|
||||
logger.exception("transcribe preflight: ASR load failed (job=%s)", job_id)
|
||||
from core.failure import build_failure
|
||||
f = build_failure(e, stage="transcribe-preflight", include_diagnostic=False)
|
||||
preflight_error = "ASR backend initialization failed: " + f["reason"] + (
|
||||
@@ -470,16 +431,9 @@ async def dub_transcribe_stream(
|
||||
)
|
||||
scene_cuts = job.get("scene_cuts") or []
|
||||
|
||||
async def _gen_body():
|
||||
async def gen():
|
||||
if preflight_error:
|
||||
# Always follow a terminal `error` with `done` so the stream closes
|
||||
# via a named event, not a raw connection drop. A bare error+close
|
||||
# races the browser's native EventSource error (which carries no
|
||||
# `data`); if that native error wins, the client falls back to the
|
||||
# misleading generic "stream dropped … ASR backend failed" message
|
||||
# and the real cause (in `detail`) is lost (#578).
|
||||
yield _sse_event("error", {"detail": preflight_error, "retryable": True})
|
||||
yield _sse_event("done", {})
|
||||
yield _sse_event("error", {"detail": preflight_error})
|
||||
return
|
||||
import math
|
||||
import tempfile
|
||||
@@ -494,9 +448,7 @@ async def dub_transcribe_stream(
|
||||
try:
|
||||
audio_np, sr = await loop.run_in_executor(_cpu_pool, _load)
|
||||
except Exception as e:
|
||||
# Terminal error → always emit `done` (see preflight note, #578).
|
||||
yield _sse_event("error", {"detail": f"audio load failed: {e}", "retryable": True})
|
||||
yield _sse_event("done", {})
|
||||
yield _sse_event("error", {"detail": f"audio load failed: {e}"})
|
||||
return
|
||||
|
||||
total = float(len(audio_np)) / float(sr) if sr else 0.0
|
||||
@@ -513,9 +465,6 @@ async def dub_transcribe_stream(
|
||||
logger.warning("offload_tts_for_asr failed (continuing): %s", e)
|
||||
|
||||
all_segments: list[dict] = []
|
||||
# Words (global-timeline) retained so diarization can re-split a segment
|
||||
# that spans two speakers' turns at the word boundary (#486).
|
||||
all_words: list = []
|
||||
detected_lang = None
|
||||
next_seg_id = 0
|
||||
chunk_errors: list[str] = []
|
||||
@@ -587,11 +536,6 @@ async def dub_transcribe_stream(
|
||||
"Transcribe chunk %d/%d timed out after %.0fs (job=%s)",
|
||||
i + 1, chunks_n, TRANSCRIBE_CHUNK_TIMEOUT_S, job_id,
|
||||
)
|
||||
# #730: the wedged chunk thread keeps holding its GPU-pool worker.
|
||||
# Abandon the poisoned pool so the next chunk (and any TTS work)
|
||||
# gets a fresh worker instead of queueing behind the stuck one —
|
||||
# same recovery the whole-file paths get via run_transcribe_guarded.
|
||||
_reset_pool_on_wedge(_gpu_pool)
|
||||
part = {
|
||||
"chunks": [], "language": None,
|
||||
"error": f"Chunk {i+1} timed out after {TRANSCRIBE_CHUNK_TIMEOUT_S:.0f}s — "
|
||||
@@ -604,21 +548,6 @@ async def dub_transcribe_stream(
|
||||
detected_lang = part["language"]
|
||||
asr_speaker_turns.extend(part.get("speaker_turns") or [])
|
||||
chunk_segs = segment_transcript(part, duration=t1, scene_cuts=scene_cuts)
|
||||
# Same word source segment_transcript used (already global-timeline),
|
||||
# kept for the post-diarization speaker re-split (#486).
|
||||
try:
|
||||
all_words.extend(_words_from_whisper(part))
|
||||
except Exception:
|
||||
pass
|
||||
# #280: Whisper often stretches a segment's start back over
|
||||
# leading music/silence (classic case: speech begins at 0:03,
|
||||
# transcript says 0.0 → the dub plays 3 s early). Snap starts
|
||||
# forward to the actual speech onset. `audio_np` is the same
|
||||
# track ASR ran on — vocals.wav when Demucs succeeded.
|
||||
try:
|
||||
snap_segment_starts(chunk_segs, audio_np, sr)
|
||||
except Exception as e:
|
||||
logger.warning("onset alignment skipped for chunk %d: %s", i, e)
|
||||
chunk_segs = assign_speakers_heuristic(chunk_segs)
|
||||
for s in chunk_segs:
|
||||
s["id"] = f"s{next_seg_id:05x}"
|
||||
@@ -688,10 +617,7 @@ async def dub_transcribe_stream(
|
||||
# use its speaker turns directly and skip pyannote entirely (#182).
|
||||
if asr_speaker_turns:
|
||||
logger.info("Using inline ASR diarization (%d turns); skipping pyannote.", len(asr_speaker_turns))
|
||||
assigned = assign_speakers_from_turns(all_segments, asr_speaker_turns)
|
||||
# #486: split any segment that spans two speakers' turns at the
|
||||
# word boundary (single-speaker segments pass through unchanged).
|
||||
return resplit_segments_by_turns(assigned, all_words, asr_speaker_turns), None
|
||||
return assign_speakers_from_turns(all_segments, asr_speaker_turns), None
|
||||
|
||||
from services.model_manager import (
|
||||
DIARIZATION_ERR_LICENSE,
|
||||
@@ -768,10 +694,7 @@ async def dub_transcribe_stream(
|
||||
diar = diar_pipe(asr_audio_target, num_speakers=num_speakers)
|
||||
else:
|
||||
diar = diar_pipe(asr_audio_target)
|
||||
assigned = assign_speakers_from_diarization(all_segments, diar)
|
||||
# #486: split any segment that spans two speakers' turns at the
|
||||
# word boundary (single-speaker segments pass through unchanged).
|
||||
return resplit_segments_by_diarization(assigned, all_words, diar), None
|
||||
return assign_speakers_from_diarization(all_segments, diar), None
|
||||
except Exception as e:
|
||||
logger.error(f"Diarization failed: {e}")
|
||||
# Mid-run failure — classify against the same sentinels so a
|
||||
@@ -836,61 +759,16 @@ async def dub_transcribe_stream(
|
||||
clones = done.pop().result()
|
||||
break
|
||||
yield _sse_event("ping", {})
|
||||
# Wave 3.2: per-segment clone refs. Cut each long-enough segment's
|
||||
# own reference from the vocals so the dub of each line matches the
|
||||
# prosody of its source line. Short lines fall back to the
|
||||
# per-speaker clone below. Default on; the user can force
|
||||
# per-speaker by disabling it (job["per_segment_refs"]).
|
||||
seg_clones = {}
|
||||
job["per_segment_refs"] = per_segment_refs
|
||||
if per_segment_refs:
|
||||
try:
|
||||
from services.speaker_clone import extract_segment_refs
|
||||
seg_ids_for_clone = [s.get("id", i) for i, s in enumerate(final_segs)]
|
||||
seg_clones = await loop.run_in_executor(
|
||||
_cpu_pool, lambda: extract_segment_refs(
|
||||
vocals_for_clone, final_segs,
|
||||
os.path.dirname(vocals_for_clone),
|
||||
seg_ids=seg_ids_for_clone,
|
||||
),
|
||||
)
|
||||
if seg_clones:
|
||||
job["segment_clones"] = seg_clones
|
||||
except Exception as e:
|
||||
logger.warning("per-segment clone refs skipped: %s", e)
|
||||
|
||||
if clones or seg_clones:
|
||||
if clones:
|
||||
job["speaker_clones"] = clones
|
||||
# Default each segment's profile_id to its detected speaker's
|
||||
# auto-clone — but only if the user hasn't already assigned
|
||||
# something. (#486)
|
||||
#
|
||||
# We prefer the UI-visible `auto:{speaker}` id over the
|
||||
# per-segment `auto-seg:{id}` id even when a per-segment ref
|
||||
# exists, because the dub editor's Voice dropdown only renders
|
||||
# `auto:` options ("From Video → Speaker N"). An `auto-seg:`
|
||||
# value matches no <option>, so the row silently read
|
||||
# "Default" while the speaker was actually bound — exactly the
|
||||
# reported bug. The per-segment ref is NOT lost: dub_generate's
|
||||
# `auto:` branch transparently prefers this segment's own
|
||||
# per-segment ref (job["segment_clones"][seg_id]) when present,
|
||||
# so a row shown as "Speaker 1" still clones from its own line
|
||||
# when that line is long enough.
|
||||
if clones:
|
||||
job["speaker_clones"] = clones
|
||||
# Default each segment's profile_id to its speaker's auto-clone,
|
||||
# but only if the user hasn't already assigned something.
|
||||
for s in final_segs:
|
||||
if s.get("profile_id"):
|
||||
continue
|
||||
spk = s.get("speaker_id") or "Speaker 1"
|
||||
if spk in clones:
|
||||
s["profile_id"] = auto_profile_id(spk)
|
||||
continue
|
||||
# No per-speaker clone for this speaker (too little usable
|
||||
# audio overall) but this single line was long enough for
|
||||
# its own ref — fall back to the per-segment id. The editor
|
||||
# can't render it, but generation still clones correctly.
|
||||
sid = str(s.get("id", ""))
|
||||
if sid and sid in seg_clones:
|
||||
s["profile_id"] = f"auto-seg:{sid}"
|
||||
except Exception as e:
|
||||
logger.warning("speaker_clone extraction skipped: %s", e)
|
||||
|
||||
@@ -919,25 +797,6 @@ async def dub_transcribe_stream(
|
||||
})
|
||||
yield _sse_event("done", {})
|
||||
|
||||
async def gen():
|
||||
# Terminal-event guard (#516): the SSE stream must NEVER close without a
|
||||
# terminal event. Any unanticipated exception in the body (e.g. an ASR
|
||||
# load that escapes the per-chunk handler) previously dropped the
|
||||
# connection, which the frontend can only report as "stream dropped,
|
||||
# likely ASR failed" — hiding the real cause. Emit a structured `error`
|
||||
# (with the actionable hint from build_failure) then `done`, so the user
|
||||
# sees the real failure + a Retry instead of a silent disconnect.
|
||||
try:
|
||||
async for ev in _gen_body():
|
||||
yield ev
|
||||
except Exception as e: # noqa: BLE001 — last-resort stream finalizer
|
||||
logger.exception("transcribe stream crashed (job=%s)", job_id)
|
||||
from core.failure import build_failure
|
||||
f = build_failure(e, stage="transcribe", include_diagnostic=False)
|
||||
detail = f["reason"] + (f" — {f['hint']}" if f.get("hint") else "")
|
||||
yield _sse_event("error", {"detail": detail, "retryable": True})
|
||||
yield _sse_event("done", {})
|
||||
|
||||
return StreamingResponse(
|
||||
gen(),
|
||||
media_type="text/event-stream",
|
||||
@@ -1003,14 +862,6 @@ async def dub_transcribe(job_id: str):
|
||||
scene_cuts = job.get("scene_cuts") or []
|
||||
segments = segment_transcript(result, duration=job.get("duration", 0.0), scene_cuts=scene_cuts)
|
||||
|
||||
# #280: snap segment starts forward to the actual speech onset so the
|
||||
# dub doesn't begin seconds before the original speaker does.
|
||||
try:
|
||||
audio_for_onset, onset_sr = sf.read(asr_audio_target, dtype="float32")
|
||||
snap_segment_starts(segments, audio_for_onset, onset_sr)
|
||||
except Exception as e:
|
||||
logger.warning("onset alignment skipped: %s", e)
|
||||
|
||||
diar_pipe = get_diarization_pipeline()
|
||||
if diar_pipe:
|
||||
try:
|
||||
@@ -1041,12 +892,7 @@ async def dub_transcribe(job_id: str):
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
try:
|
||||
# Bound the whole-file transcribe (#730): a wedged whisperx/CTranslate2
|
||||
# call would otherwise hold its GPU-pool worker forever and starve
|
||||
# every other request into a "can't reach backend". run_transcribe_guarded
|
||||
# also resets the pool on timeout so capacity is restored.
|
||||
from services.asr_backend import run_transcribe_guarded
|
||||
segments_result = await run_transcribe_guarded(_gpu_pool, _transcribe, what="Dub")
|
||||
segments_result = await loop.run_in_executor(_gpu_pool, _transcribe)
|
||||
except asyncio.CancelledError:
|
||||
job["aborted"] = True
|
||||
raise
|
||||
|
||||
@@ -1,7 +1,5 @@
|
||||
import os
|
||||
import io
|
||||
import re
|
||||
import json
|
||||
import time
|
||||
import uuid
|
||||
import asyncio
|
||||
@@ -14,13 +12,6 @@ from core.config import DUB_DIR, dub_seg_path
|
||||
from core.tasks import task_manager
|
||||
from api.routers.dub_core import _get_job
|
||||
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
|
||||
from services.video_retime import (
|
||||
DRIFT_TOLERANCE_S,
|
||||
RetimeError,
|
||||
build_chunk_filter_graph,
|
||||
expand_retime_chunks,
|
||||
prepare_smart_fit_video,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.api")
|
||||
@@ -31,9 +22,6 @@ def _unique_stamp() -> str:
|
||||
return f"{time.strftime('%Y%m%dT%H%M%S')}-{uuid.uuid4().hex[:8]}"
|
||||
|
||||
|
||||
_SAFE_LANG = re.compile(r"^[A-Za-z0-9_-]{1,32}$")
|
||||
|
||||
|
||||
def _native_save(source: str, destination: str, display_name: str, media_type: str):
|
||||
"""Copy a generated export file to a user-chosen destination and return JSON."""
|
||||
import shutil
|
||||
@@ -170,23 +158,15 @@ async def dub_list_tracks(job_id: str):
|
||||
return {"tracks": job.get("dubbed_tracks", {})}
|
||||
|
||||
|
||||
def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool,
|
||||
fitted_segments: "list[dict] | None" = None) -> str | None:
|
||||
def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool) -> str | None:
|
||||
"""Build a temp SRT from job segments for use with ffmpeg's subtitles filter.
|
||||
|
||||
Returned path is already ffmpeg-filter-safe (plain ASCII basename under exports_dir).
|
||||
Returns None if there are no segments to render.
|
||||
|
||||
``fitted_segments`` (Smart Fit): {id, start, end} cue records on the
|
||||
fitted timeline — when provided, cue times come from there instead of
|
||||
the original ``job["segments"]`` timings, so burned subs track the
|
||||
retimed video / fitted audio rather than the source timeline.
|
||||
"""
|
||||
segments = job.get("segments", [])
|
||||
if not segments:
|
||||
return None
|
||||
if fitted_segments:
|
||||
segments = _apply_fitted_times(segments, fitted_segments)
|
||||
lines = []
|
||||
for i, seg in enumerate(segments):
|
||||
lines.append(str(i + 1))
|
||||
@@ -234,14 +214,41 @@ def _build_video_stretch_filter_graph(
|
||||
if not plan:
|
||||
return "", in_label or f"[{video_input_idx}:v]"
|
||||
|
||||
# Chunk expansion + graph emission live in services.video_retime now so
|
||||
# the Smart Fit batched pipeline shares the exact same boundary math.
|
||||
# With default options the emitted graph is byte-identical to the
|
||||
# original inline implementation.
|
||||
chunks = expand_retime_chunks(plan, orig_dur)
|
||||
chunks: list[tuple[float, float, float]] = [] # (a, b, ratio)
|
||||
cursor = 0.0
|
||||
for entry in plan:
|
||||
a = float(entry["orig_start"])
|
||||
b = float(entry["orig_end"])
|
||||
if a > cursor + 1e-3:
|
||||
chunks.append((cursor, a, 1.0)) # gap or pre-roll at native rate
|
||||
ratio = float(entry["stretch_ratio"])
|
||||
if b > a:
|
||||
chunks.append((a, b, ratio))
|
||||
cursor = max(cursor, b)
|
||||
if orig_dur > cursor + 1e-3:
|
||||
chunks.append((cursor, orig_dur, 1.0)) # tail at native rate
|
||||
chunks = [(a, b, r) for (a, b, r) in chunks if b > a]
|
||||
if not chunks:
|
||||
return "", in_label or f"[{video_input_idx}:v]"
|
||||
return build_chunk_filter_graph(chunks, in_label or f"[{video_input_idx}:v]")
|
||||
|
||||
src = in_label or f"[{video_input_idx}:v]"
|
||||
parts: list[str] = []
|
||||
labels: list[str] = []
|
||||
# `split` lets us tap the same source stream once per chunk without re-
|
||||
# decoding. setpts={ratio}*PTS slows down (ratio > 1) or speeds up
|
||||
# (ratio < 1) each chunk; PTS-STARTPTS first to normalise the timestamp
|
||||
# base after the trim.
|
||||
split_labels = [f"[vsplit{idx}]" for idx in range(len(chunks))]
|
||||
parts.append(f"{src}split={len(chunks)}{''.join(split_labels)}")
|
||||
for idx, ((a, b, ratio), split_lbl) in enumerate(zip(chunks, split_labels)):
|
||||
out_label = f"[vstr{idx}]"
|
||||
labels.append(out_label)
|
||||
parts.append(
|
||||
f"{split_lbl}trim=start={a:.4f}:end={b:.4f},"
|
||||
f"setpts=PTS-STARTPTS,setpts={ratio:.6f}*PTS{out_label}"
|
||||
)
|
||||
parts.append("".join(labels) + f"concat=n={len(chunks)}:v=1:a=0[vstretched]")
|
||||
return ";".join(parts), "[vstretched]"
|
||||
|
||||
|
||||
def _video_stretch_plan_for(job: dict, lang_code: str) -> dict | None:
|
||||
@@ -257,76 +264,6 @@ def _video_stretch_plan_for(job: dict, lang_code: str) -> dict | None:
|
||||
return entry
|
||||
|
||||
|
||||
def _video_retime_plan_for(job: dict, lang_code: str) -> "tuple[str, dict] | None":
|
||||
"""Resolve the video retime plan for ``lang_code`` across both keyspaces.
|
||||
|
||||
Returns ``(kind, entry)`` where kind is ``"stretch_video"`` (legacy
|
||||
Mode B plans — resolution byte-identical to ``_video_stretch_plan_for``)
|
||||
or ``"smart_fit"`` (Phase A ``job["fit_plans"]`` entries, gated on the
|
||||
track actually having been generated under smart_fit so a stale plan
|
||||
from an earlier run can't retime a track re-generated under another
|
||||
strategy). ``None`` when neither applies.
|
||||
"""
|
||||
legacy = _video_stretch_plan_for(job, lang_code)
|
||||
if legacy is not None:
|
||||
return "stretch_video", legacy
|
||||
entry = (job.get("fit_plans") or {}).get(lang_code)
|
||||
track = (job.get("dubbed_tracks") or {}).get(lang_code) or {}
|
||||
if entry and entry.get("plan") and track.get("timing_strategy") == "smart_fit":
|
||||
return "smart_fit", entry
|
||||
return None
|
||||
|
||||
|
||||
def _fitted_segments_for(job: dict, lang_code: "str | None") -> "list[dict] | None":
|
||||
"""Fitted-timeline subtitle cues ({id, start, end}) for a Smart Fit
|
||||
track, or None. Same staleness gate as ``_video_retime_plan_for``."""
|
||||
if not lang_code:
|
||||
return None
|
||||
entry = (job.get("fit_plans") or {}).get(lang_code)
|
||||
track = (job.get("dubbed_tracks") or {}).get(lang_code) or {}
|
||||
if not entry or track.get("timing_strategy") != "smart_fit":
|
||||
return None
|
||||
fitted = entry.get("fitted_segments")
|
||||
return fitted or None
|
||||
|
||||
|
||||
def _apply_fitted_times(segments: list[dict], fitted: list[dict]) -> list[dict]:
|
||||
"""Overlay fitted cue times onto subtitle segments (copies; non-destructive).
|
||||
|
||||
Matches by segment ``id``; when the fitted record carries no ids at all
|
||||
(defensive), falls back to positional pairing. Segments without a match
|
||||
keep their original timings.
|
||||
"""
|
||||
by_id = {str(f["id"]): f for f in fitted if f.get("id") is not None}
|
||||
out: list[dict] = []
|
||||
for i, seg in enumerate(segments):
|
||||
cue = None
|
||||
if seg.get("id") is not None:
|
||||
cue = by_id.get(str(seg["id"]))
|
||||
if cue is None and not by_id and i < len(fitted):
|
||||
cue = fitted[i]
|
||||
if cue is None:
|
||||
out.append(seg)
|
||||
continue
|
||||
patched = dict(seg)
|
||||
patched["start"] = float(cue["start"])
|
||||
patched["end"] = float(cue["end"])
|
||||
out.append(patched)
|
||||
return out
|
||||
|
||||
|
||||
def _burn_subs_allowed(retime_kind: "str | None") -> bool:
|
||||
"""Subtitle burn-in combined with video retime.
|
||||
|
||||
Allowed for ``smart_fit`` (fitted cue records exist, and the burn pass
|
||||
runs AFTER the retime graph so cues land on the retimed timeline) and
|
||||
for plain exports. Still rejected for legacy ``stretch_video``, which
|
||||
has no fitted-cue record — cues would burn at original timestamps onto
|
||||
a re-timed video and drift.
|
||||
"""
|
||||
return retime_kind != "stretch_video"
|
||||
|
||||
|
||||
#: Audio export formats → ffmpeg codec args. Unknown formats fall back to
|
||||
#: AAC/m4a so a bad request can never produce a broken command.
|
||||
_AUDIO_FORMAT_CODECS: dict[str, list[str]] = {
|
||||
@@ -375,11 +312,6 @@ async def dub_download(
|
||||
dual: bool = Query(False, description="When burn_subs=1, render translated on top of italicised original."),
|
||||
out_format: str = Query("m4a", description="Audio-only jobs (#119): output container — wav, m4a, mp3, or flac. Ignored for video jobs."),
|
||||
):
|
||||
# Strict allowlist on the path param BEFORE it reaches any filesystem
|
||||
# path or ffmpeg argv (export dir, retime work path, slice paths). Real
|
||||
# job ids are short uuid slices — alnum/hyphen/underscore only.
|
||||
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", job_id):
|
||||
raise HTTPException(status_code=400, detail="Invalid job id")
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
@@ -458,24 +390,18 @@ async def dub_download(
|
||||
)
|
||||
|
||||
# Determine whether this export should drive video through a per-segment
|
||||
# retime (legacy stretch_video Mode B, or Smart Fit). Retime is keyed off
|
||||
# the default_track's plan because the video can only physically follow
|
||||
# one timeline at a time. If multiple dub tracks are included, only the
|
||||
# default_track is visually in sync — other tracks share the same
|
||||
# (retimed) video. Single-track export is the supported common case.
|
||||
retime_kind: "str | None" = None
|
||||
retime_entry: "dict | None" = None
|
||||
if default_track and default_track != "original":
|
||||
_retime = _video_retime_plan_for(job, default_track)
|
||||
if _retime:
|
||||
retime_kind, retime_entry = _retime
|
||||
stretch_entry = retime_entry if retime_kind == "stretch_video" else None
|
||||
# Subtitle burn under legacy stretch_video would render cues at the
|
||||
# original timestamps onto a re-timed video — they'd drift (no fitted-cue
|
||||
# record exists for that mode). Skip the burn pass in that combo and log;
|
||||
# the user can still export the SRT/VTT separately. Smart Fit DOES carry
|
||||
# fitted cues, so burn+retime is allowed there (burn runs post-retime).
|
||||
if not _burn_subs_allowed(retime_kind) and burn_subs:
|
||||
# stretch graph (Mode B). Stretch is keyed off the default_track's plan
|
||||
# because the video can only physically follow one timeline at a time.
|
||||
# If multiple dub tracks are included and they were generated under
|
||||
# stretch_video, only the default_track is visually in sync — other
|
||||
# tracks share the same (stretched) video. Single-track export is the
|
||||
# supported common case.
|
||||
stretch_entry = _video_stretch_plan_for(job, default_track) if default_track and default_track != "original" else None
|
||||
# Subtitle burn under stretch_video would render cues at the original
|
||||
# timestamps onto a re-timed video — they'd drift. Skip the burn pass
|
||||
# in that combo and log; the user can still export the SRT/VTT
|
||||
# separately and the new-layout timing lives there.
|
||||
if stretch_entry and burn_subs:
|
||||
logger.warning(
|
||||
"stretch_video + burn_subs is not supported in one pass; "
|
||||
"skipping subtitle burn for job %s. Export the SRT/VTT separately.",
|
||||
@@ -483,71 +409,11 @@ async def dub_download(
|
||||
)
|
||||
burn_subs = False
|
||||
|
||||
# Smart Fit: cue times come from the fitted timeline — that's where the
|
||||
# dubbed audio actually sits, whether or not the video retime succeeds.
|
||||
fitted_segments = _fitted_segments_for(job, default_track) if default_track and default_track != "original" else None
|
||||
sub_path = _write_burn_srt(job, exports_dir, stamp, dual, fitted_segments=fitted_segments) if burn_subs else None
|
||||
|
||||
# ── Smart Fit video retime (two-tier) ─────────────────────────────────
|
||||
# Tier 1 (≤48 chunks): single filter_complex graph inlined into the mux
|
||||
# command below. Tier 2: batched slice renders joined by the concat
|
||||
# demuxer into an intermediate file, muxed as an extra input. Failures
|
||||
# fall back to an un-retimed export with a structured warning rather
|
||||
# than failing the whole download.
|
||||
retime_decision = None
|
||||
retime_warning: "dict | None" = None
|
||||
smart_track_dur = 0.0
|
||||
if retime_kind == "smart_fit" and retime_entry:
|
||||
smart_orig_dur = float(retime_entry.get("orig_duration") or job.get("duration") or 0.0)
|
||||
smart_track_dur = float(
|
||||
retime_entry.get("total_duration")
|
||||
or (filtered_tracks.get(default_track) or {}).get("duration")
|
||||
or 0.0
|
||||
)
|
||||
# A fresh export is a fresh user intent — clear any sticky abort flag
|
||||
# from a previous /dub/abort so it can't kill this run's first batch.
|
||||
job.pop("aborted", None)
|
||||
# realpath-normalised + containment-checked inline at the sink (the
|
||||
# file's established pattern — CodeQL does not track the guard
|
||||
# through a helper's return value).
|
||||
_base = os.path.realpath(DUB_DIR)
|
||||
retime_work_path = os.path.realpath(
|
||||
os.path.join(exports_dir, f"retimed_{stamp}.mp4")
|
||||
)
|
||||
if retime_work_path != _base and not retime_work_path.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid export path")
|
||||
try:
|
||||
retime_decision = await prepare_smart_fit_video(
|
||||
job_id=job_id,
|
||||
ffmpeg=ffmpeg,
|
||||
video_path=video_path,
|
||||
plan=retime_entry["plan"],
|
||||
orig_dur=smart_orig_dur,
|
||||
track_dur=smart_track_dur,
|
||||
work_path=retime_work_path,
|
||||
abort_check=lambda: bool(job.get("aborted")),
|
||||
)
|
||||
except Exception as e:
|
||||
if (isinstance(e, RetimeError) and e.stage == "aborted") or job.get("aborted"):
|
||||
raise HTTPException(status_code=409, detail="Export aborted")
|
||||
from core.failure import build_failure
|
||||
retime_warning = build_failure(e, stage="video-retime", include_diagnostic=False)
|
||||
job["last_export_warning"] = {"type": "video_retime_fallback", **retime_warning}
|
||||
logger.error(
|
||||
"Smart Fit video retime failed for job %s — exporting "
|
||||
"without per-segment retime: %s",
|
||||
job_id.replace("\n", " ").replace("\r", " "), e,
|
||||
)
|
||||
sub_path = _write_burn_srt(job, exports_dir, stamp, dual) if burn_subs else None
|
||||
|
||||
cmd = [ffmpeg, "-i", video_path]
|
||||
input_idx = 1
|
||||
|
||||
retimed_idx = None
|
||||
if retime_decision is not None and retime_decision.mode == "file":
|
||||
cmd += ["-i", retime_decision.file_path]
|
||||
retimed_idx = input_idx
|
||||
input_idx += 1
|
||||
|
||||
bg_audio = job.get("no_vocals_path") if preserve_bg else None
|
||||
bg_idx = None
|
||||
if bg_audio and os.path.exists(bg_audio) and filtered_tracks:
|
||||
@@ -563,37 +429,9 @@ async def dub_download(
|
||||
|
||||
filter_parts: list[str] = []
|
||||
video_map = "0:v:0"
|
||||
video_reencode = False
|
||||
if retime_decision is not None:
|
||||
if retime_decision.mode == "filter":
|
||||
filter_parts.append(retime_decision.graph)
|
||||
video_map = retime_decision.label
|
||||
video_reencode = True
|
||||
else:
|
||||
video_map = f"{retimed_idx}:v:0"
|
||||
# Residual drift after the batched render (fps rounding): video
|
||||
# shorter than the fitted track → freeze the last frame out to
|
||||
# the track length. Rare — the predicted tail pad inside the
|
||||
# render usually lands within tolerance.
|
||||
residual = smart_track_dur - retime_decision.video_dur
|
||||
if smart_track_dur and residual > DRIFT_TOLERANCE_S:
|
||||
filter_parts.append(
|
||||
f"[{retimed_idx}:v]tpad=stop_mode=clone:stop_duration={residual:.4f}[vtpad]"
|
||||
)
|
||||
video_map = "[vtpad]"
|
||||
video_reencode = True
|
||||
if sub_path:
|
||||
esc = _ffmpeg_filter_escape(sub_path)
|
||||
# Burn AFTER any retime so cues (already on the fitted timeline for
|
||||
# Smart Fit) land on the retimed video. Without retime this reduces
|
||||
# to the legacy `[0:v]subtitles=…[vsub]` graph.
|
||||
if video_map.startswith("["):
|
||||
sub_src = video_map
|
||||
elif retimed_idx is not None:
|
||||
sub_src = f"[{retimed_idx}:v]"
|
||||
else:
|
||||
sub_src = "[0:v]"
|
||||
filter_parts.append(f"{sub_src}subtitles='{esc}'[vsub]")
|
||||
filter_parts.append(f"[0:v]subtitles='{esc}'[vsub]")
|
||||
video_map = "[vsub]"
|
||||
if stretch_entry:
|
||||
orig_dur = float(stretch_entry.get("orig_duration") or job.get("duration") or 0.0)
|
||||
@@ -609,32 +447,10 @@ async def dub_download(
|
||||
if include_original:
|
||||
cmd += ["-map", "0:a:0"]
|
||||
|
||||
# Smart Fit drift absorption, audio side: when the retimed video runs
|
||||
# longer than the fitted track (its tail passes through at 1.0× beyond
|
||||
# the last cue, or encoder rounding), pad the dub-track chain with
|
||||
# silence out to the video length so players don't end audio early.
|
||||
apad_dur = 0.0
|
||||
if (
|
||||
retime_decision is not None
|
||||
and smart_track_dur
|
||||
and retime_decision.video_dur - smart_track_dur > DRIFT_TOLERANCE_S
|
||||
):
|
||||
apad_dur = retime_decision.video_dur
|
||||
|
||||
if bg_idx is not None:
|
||||
for i, t in enumerate(tracks_to_process):
|
||||
out_label = f"[aout{i}]"
|
||||
chain = f"[{bg_idx}:a][{t['idx']}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2"
|
||||
if apad_dur:
|
||||
chain += f",apad=whole_dur={apad_dur:.4f}"
|
||||
filter_parts.append(chain + out_label)
|
||||
t["out_label"] = out_label
|
||||
for t in tracks_to_process:
|
||||
cmd += ["-map", t["out_label"]]
|
||||
elif apad_dur:
|
||||
for i, t in enumerate(tracks_to_process):
|
||||
out_label = f"[aout{i}]"
|
||||
filter_parts.append(f"[{t['idx']}:a]apad=whole_dur={apad_dur:.4f}{out_label}")
|
||||
filter_parts.append(f"[{bg_idx}:a][{t['idx']}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2{out_label}")
|
||||
t["out_label"] = out_label
|
||||
for t in tracks_to_process:
|
||||
cmd += ["-map", t["out_label"]]
|
||||
@@ -645,11 +461,10 @@ async def dub_download(
|
||||
if filter_parts:
|
||||
cmd += ["-filter_complex", ";".join(filter_parts)]
|
||||
|
||||
# Burning subs or per-segment video retime both force a real video
|
||||
# re-encode; stream-copy is viable when nothing touches the video
|
||||
# filter chain — including the batched Smart Fit path, whose retimed
|
||||
# intermediate is already encoded with these exact settings.
|
||||
if sub_path or stretch_entry or video_reencode:
|
||||
# Burning subs or per-segment video stretch both force a real video
|
||||
# re-encode; stream-copy is only viable when nothing touches the video
|
||||
# filter chain.
|
||||
if sub_path or stretch_entry:
|
||||
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
|
||||
else:
|
||||
cmd += ["-c:v", "copy"]
|
||||
@@ -682,17 +497,17 @@ async def dub_download(
|
||||
break
|
||||
cmd += [f"-disposition:a:{target_idx}", "default"]
|
||||
|
||||
# When retiming (legacy stretch_video or Smart Fit) the video and audio
|
||||
# durations should match within sub-frame precision, but `-shortest`
|
||||
# can still cut off the trailing frame; let ffmpeg keep both streams.
|
||||
# Otherwise keep the legacy `-shortest` so a slightly-overrunning track
|
||||
# doesn't extend the mux past the video.
|
||||
if not stretch_entry and retime_decision is None:
|
||||
# In stretch_video mode the video and audio durations should match
|
||||
# within sub-frame precision, but `-shortest` can still cut off the
|
||||
# trailing frame; let ffmpeg keep both streams. Otherwise keep the
|
||||
# legacy `-shortest` so a slightly-overrunning track doesn't extend
|
||||
# the mux past the video.
|
||||
if not stretch_entry:
|
||||
cmd += ["-shortest"]
|
||||
cmd += [output_path, "-y"]
|
||||
|
||||
try:
|
||||
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0, job_id=job_id)
|
||||
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0)
|
||||
if rc != 0:
|
||||
raise Exception(stderr.decode(errors="replace") if stderr else "ffmpeg mux non-zero")
|
||||
except asyncio.TimeoutError:
|
||||
@@ -704,14 +519,6 @@ async def dub_download(
|
||||
status_code=500,
|
||||
detail=f"ffmpeg failed to combine video + dubbed audio: {e}. Verify ffmpeg is installed (`ffmpeg -version`), and check that every dubbed track file exists in the job folder.",
|
||||
)
|
||||
finally:
|
||||
# The batched retime intermediate is a full re-encoded video — never
|
||||
# leave it behind (success or failure; it's stamp-unique, no reuse).
|
||||
if retime_decision is not None and retime_decision.mode == "file":
|
||||
try:
|
||||
os.remove(retime_decision.file_path)
|
||||
except OSError as e:
|
||||
logger.debug("cleanup remove failed: %s", e)
|
||||
|
||||
if not os.path.exists(output_path) or os.path.getsize(output_path) == 0:
|
||||
raise HTTPException(status_code=500, detail="ffmpeg mux produced no output file")
|
||||
@@ -721,22 +528,12 @@ async def dub_download(
|
||||
safe_name = ''.join(c for c in base_name if c.isalnum() or c in '-_ ').strip() or 'output'
|
||||
dl_name = f"dubbed_{safe_name}_{stamp}.mp4"
|
||||
|
||||
# Structured warning surface for the Smart Fit fallback ladder: header is
|
||||
# a fixed ASCII token (FileResponse headers must be latin-1 safe); the
|
||||
# full build_failure payload is persisted on the job for the UI to read.
|
||||
extra_headers = {}
|
||||
if retime_warning is not None:
|
||||
extra_headers["X-Dub-Export-Warning"] = "video-retime-fallback"
|
||||
|
||||
if save_path:
|
||||
result = _native_save(output_path, save_path, dl_name, media_type="video/mp4")
|
||||
if retime_warning is not None:
|
||||
result["warning"] = {"type": "video_retime_fallback", **retime_warning}
|
||||
return result
|
||||
return _native_save(output_path, save_path, dl_name, media_type="video/mp4")
|
||||
|
||||
return FileResponse(
|
||||
output_path, media_type="video/mp4",
|
||||
headers={"Content-Disposition": f'attachment; filename="{dl_name}"', **extra_headers},
|
||||
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
|
||||
)
|
||||
|
||||
|
||||
@@ -771,21 +568,6 @@ async def dub_get_media(job_id: str):
|
||||
ext = os.path.splitext(video_path)[1].lower()
|
||||
return FileResponse(video_path, media_type=_MEDIA_TYPES.get(ext, "video/mp4"))
|
||||
|
||||
# One mux at a time per preview file. Without this, two overlapping requests
|
||||
# (e.g. the <video> element remounting right after a re-dub) both ran ffmpeg
|
||||
# against the same output path, and the mtime check below saw the half-written
|
||||
# file as a valid cache — serving a truncated MP4 that left the player stuck
|
||||
# loading forever (#281).
|
||||
_preview_mux_locks: dict[str, asyncio.Lock] = {}
|
||||
|
||||
|
||||
def _preview_lock(path: str) -> asyncio.Lock:
|
||||
lock = _preview_mux_locks.get(path)
|
||||
if lock is None:
|
||||
lock = _preview_mux_locks.setdefault(path, asyncio.Lock())
|
||||
return lock
|
||||
|
||||
|
||||
@router.get("/dub/preview-video/{job_id}")
|
||||
async def dub_preview_video(
|
||||
job_id: str,
|
||||
@@ -797,11 +579,6 @@ async def dub_preview_video(
|
||||
Caches per lang+preserve_bg combination under exports/preview_{lang}_{bg}.mp4.
|
||||
Cache is invalidated when the underlying dubbed track mtime is newer than the cache.
|
||||
"""
|
||||
# Strict allowlist on the path param BEFORE it reaches any filesystem
|
||||
# path or ffmpeg argv (exports dir, preview/retime work paths) — same
|
||||
# boundary check as dub_download.
|
||||
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", job_id):
|
||||
raise HTTPException(status_code=400, detail="Invalid job id")
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
@@ -822,92 +599,23 @@ async def dub_preview_video(
|
||||
bg_audio = job.get("no_vocals_path") if preserve_bg else None
|
||||
has_bg = bool(bg_audio and os.path.exists(bg_audio))
|
||||
|
||||
if not _SAFE_LANG.match(lang):
|
||||
raise HTTPException(status_code=400, detail="Invalid lang")
|
||||
# realpath-normalised + containment-checked inline BEFORE any filesystem
|
||||
# access so the guard dominates every sink (the file's established
|
||||
# pattern — see dub_preview_segment; CodeQL does not track the guard
|
||||
# through a helper's return value).
|
||||
_base = os.path.realpath(DUB_DIR)
|
||||
exports_dir = os.path.realpath(os.path.join(_base, job_id, "exports"))
|
||||
if not exports_dir.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid job id")
|
||||
exports_dir = os.path.join(DUB_DIR, job_id, "exports")
|
||||
os.makedirs(exports_dir, exist_ok=True)
|
||||
bg_suffix = "bg" if (preserve_bg and has_bg) else "nobg"
|
||||
preview_path = os.path.realpath(
|
||||
os.path.join(exports_dir, f"preview_{lang}_{bg_suffix}.mp4")
|
||||
)
|
||||
if not preview_path.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid path")
|
||||
preview_path = os.path.join(exports_dir, f"preview_{lang}_{bg_suffix}.mp4")
|
||||
|
||||
track_mtime = os.path.getmtime(track_path)
|
||||
cache_ok = (
|
||||
os.path.exists(preview_path)
|
||||
and os.path.getsize(preview_path) > 0
|
||||
and os.path.getmtime(preview_path) >= track_mtime
|
||||
)
|
||||
|
||||
def _cache_ok() -> bool:
|
||||
return (
|
||||
os.path.exists(preview_path)
|
||||
and os.path.getsize(preview_path) > 0
|
||||
and os.path.getmtime(preview_path) >= track_mtime
|
||||
)
|
||||
|
||||
async def _mux_preview():
|
||||
# Mux into a temp file and os.replace() into place so a concurrent
|
||||
# reader never sees a partially-written preview (#281: video stuck
|
||||
# loading forever after a re-dub).
|
||||
mux_path = preview_path + ".tmp.mp4"
|
||||
if not cache_ok:
|
||||
ffmpeg = find_ffmpeg()
|
||||
# Resolve the same retime plan the download path uses (legacy
|
||||
# stretch_video or Smart Fit) so the in-app preview matches export.
|
||||
retime = _video_retime_plan_for(job, lang)
|
||||
retime_kind, retime_entry = retime if retime else (None, None)
|
||||
stretch_entry = retime_entry if retime_kind == "stretch_video" else None
|
||||
|
||||
retime_decision = None
|
||||
smart_track_dur = 0.0
|
||||
if retime_kind == "smart_fit" and retime_entry:
|
||||
smart_orig_dur = float(retime_entry.get("orig_duration") or job.get("duration") or 0.0)
|
||||
smart_track_dur = float(
|
||||
retime_entry.get("total_duration") or track_info.get("duration") or 0.0
|
||||
)
|
||||
job.pop("aborted", None) # fresh user intent — clear sticky abort
|
||||
# realpath-normalised + containment-checked inline at the sink
|
||||
# (same pattern as preview_path above — _base is the realpath
|
||||
# of DUB_DIR from the top of this endpoint).
|
||||
retime_work_path = os.path.realpath(os.path.join(
|
||||
exports_dir, f"preview_retimed_{lang}_{bg_suffix}.tmp.mp4",
|
||||
))
|
||||
if retime_work_path != _base and not retime_work_path.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid export path")
|
||||
try:
|
||||
retime_decision = await prepare_smart_fit_video(
|
||||
job_id=job_id,
|
||||
ffmpeg=ffmpeg,
|
||||
video_path=video_path,
|
||||
plan=retime_entry["plan"],
|
||||
orig_dur=smart_orig_dur,
|
||||
track_dur=smart_track_dur,
|
||||
work_path=retime_work_path,
|
||||
abort_check=lambda: bool(job.get("aborted")),
|
||||
)
|
||||
except Exception as e:
|
||||
if (isinstance(e, RetimeError) and e.stage == "aborted") or job.get("aborted"):
|
||||
raise HTTPException(status_code=409, detail="Preview aborted")
|
||||
# Preview is best-effort: fall back to the un-retimed video
|
||||
# rather than a black player. The export path surfaces the
|
||||
# structured warning; here we just log.
|
||||
retime_decision = None
|
||||
logger.error(
|
||||
"Smart Fit preview retime failed for job %s — previewing "
|
||||
"without per-segment retime: %s",
|
||||
job_id.replace("\n", " ").replace("\r", " "), e,
|
||||
)
|
||||
|
||||
stretch_entry = _video_stretch_plan_for(job, lang)
|
||||
cmd = [ffmpeg, "-i", video_path]
|
||||
input_idx = 1
|
||||
retimed_idx = None
|
||||
if retime_decision is not None and retime_decision.mode == "file":
|
||||
cmd += ["-i", retime_decision.file_path]
|
||||
retimed_idx = input_idx
|
||||
input_idx += 1
|
||||
if preserve_bg and has_bg:
|
||||
cmd += ["-i", bg_audio]
|
||||
bg_idx = input_idx
|
||||
@@ -917,13 +625,12 @@ async def dub_preview_video(
|
||||
cmd += ["-i", track_path]
|
||||
track_idx = input_idx
|
||||
|
||||
# Build filter graph. Under a retime plan we splice the source video
|
||||
# into per-segment chunks, setpts each to match the dub audio
|
||||
# Build filter graph. In stretch_video mode we splice the source
|
||||
# video into per-segment chunks, setpts each to match the dub audio
|
||||
# layout, and concat them — so audio plays at natural rate and the
|
||||
# visuals follow. Otherwise we stream-copy video for speed.
|
||||
filter_parts: list[str] = []
|
||||
video_map = "0:v:0"
|
||||
video_reencode = False
|
||||
if stretch_entry:
|
||||
orig_dur = float(stretch_entry.get("orig_duration") or job.get("duration") or 0.0)
|
||||
graph, vlabel = _build_video_stretch_filter_graph(
|
||||
@@ -932,175 +639,49 @@ async def dub_preview_video(
|
||||
if graph:
|
||||
filter_parts.append(graph)
|
||||
video_map = vlabel
|
||||
elif retime_decision is not None:
|
||||
if retime_decision.mode == "filter":
|
||||
filter_parts.append(retime_decision.graph)
|
||||
video_map = retime_decision.label
|
||||
video_reencode = True
|
||||
else:
|
||||
video_map = f"{retimed_idx}:v:0"
|
||||
residual = smart_track_dur - retime_decision.video_dur
|
||||
if smart_track_dur and residual > DRIFT_TOLERANCE_S:
|
||||
filter_parts.append(
|
||||
f"[{retimed_idx}:v]tpad=stop_mode=clone:stop_duration={residual:.4f}[vtpad]"
|
||||
)
|
||||
video_map = "[vtpad]"
|
||||
video_reencode = True
|
||||
|
||||
# Smart Fit drift absorption (audio): silence-pad the dub chain out
|
||||
# to the retimed video length so the preview doesn't end audio early.
|
||||
apad_dur = 0.0
|
||||
if (
|
||||
retime_decision is not None
|
||||
and smart_track_dur
|
||||
and retime_decision.video_dur - smart_track_dur > DRIFT_TOLERANCE_S
|
||||
):
|
||||
apad_dur = retime_decision.video_dur
|
||||
|
||||
audio_map = f"{track_idx}:a:0"
|
||||
if bg_idx is not None:
|
||||
chain = f"[{bg_idx}:a][{track_idx}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2"
|
||||
if apad_dur:
|
||||
chain += f",apad=whole_dur={apad_dur:.4f}"
|
||||
filter_parts.append(chain + "[aout]")
|
||||
audio_map = "[aout]"
|
||||
elif apad_dur:
|
||||
filter_parts.append(f"[{track_idx}:a]apad=whole_dur={apad_dur:.4f}[aout]")
|
||||
audio_map = "[aout]"
|
||||
filter_parts.append(
|
||||
f"[{bg_idx}:a][{track_idx}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2[aout]"
|
||||
)
|
||||
|
||||
cmd += ["-map", video_map]
|
||||
cmd += ["-map", audio_map]
|
||||
if bg_idx is not None:
|
||||
cmd += ["-map", "[aout]"]
|
||||
else:
|
||||
cmd += ["-map", f"{track_idx}:a:0"]
|
||||
if filter_parts:
|
||||
cmd += ["-filter_complex", ";".join(filter_parts)]
|
||||
|
||||
# Retime path needs a real encode; stream-copy otherwise (the batched
|
||||
# Smart Fit intermediate is already encoded — copy unless tpad'ed).
|
||||
if stretch_entry or video_reencode:
|
||||
# Stretch path needs a real encode; stream-copy otherwise.
|
||||
if stretch_entry:
|
||||
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
|
||||
else:
|
||||
cmd += ["-c:v", "copy"]
|
||||
cmd += ["-c:a", "aac", "-b:a", "192k"]
|
||||
# `-shortest` would cut the retimed video at the (slightly different)
|
||||
# `-shortest` would cut the stretched video at the (slightly different)
|
||||
# audio length and lose the trailing frame; only use it on the copy path.
|
||||
if not stretch_entry and retime_decision is None:
|
||||
if not stretch_entry:
|
||||
cmd += ["-shortest"]
|
||||
cmd += [mux_path, "-y"]
|
||||
|
||||
def _discard_tmp():
|
||||
try:
|
||||
os.remove(mux_path)
|
||||
except OSError as e:
|
||||
logger.debug("cleanup remove failed: %s", e)
|
||||
cmd += [preview_path, "-y"]
|
||||
|
||||
try:
|
||||
rc, _, stderr = await run_ffmpeg(cmd, timeout=900.0, job_id=job_id)
|
||||
rc, _, stderr = await run_ffmpeg(cmd, timeout=900.0)
|
||||
if rc != 0:
|
||||
raise Exception(stderr.decode(errors="replace") if stderr else "ffmpeg mux non-zero")
|
||||
if not os.path.exists(mux_path) or os.path.getsize(mux_path) == 0:
|
||||
raise Exception("preview mux produced empty file")
|
||||
except asyncio.TimeoutError:
|
||||
_discard_tmp()
|
||||
raise HTTPException(status_code=504, detail="preview mux timed out")
|
||||
except HTTPException:
|
||||
_discard_tmp()
|
||||
raise
|
||||
except Exception as e:
|
||||
_discard_tmp()
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"ffmpeg failed to build the preview stream: {str(e)[:300]}. This usually means the source video can't be re-encoded on the fly — try downloading the MP4 instead.",
|
||||
)
|
||||
finally:
|
||||
if retime_decision is not None and retime_decision.mode == "file":
|
||||
try:
|
||||
os.remove(retime_decision.file_path)
|
||||
except OSError as e:
|
||||
# Best-effort scratch cleanup — never fail the export.
|
||||
logger.debug("retime intermediate cleanup failed: %s", e)
|
||||
|
||||
os.replace(mux_path, preview_path)
|
||||
if not os.path.exists(preview_path) or os.path.getsize(preview_path) == 0:
|
||||
raise HTTPException(status_code=500, detail="preview mux produced empty file")
|
||||
|
||||
async with _preview_lock(preview_path):
|
||||
if not _cache_ok():
|
||||
await _mux_preview()
|
||||
|
||||
# no-store: the URL is stable across re-dubs, so any HTTP-level caching
|
||||
# in the WebView would keep showing the previous dub after a re-generate
|
||||
# (#281: "edits don't change the result").
|
||||
return FileResponse(
|
||||
preview_path,
|
||||
media_type="video/mp4",
|
||||
headers={"Cache-Control": "no-store"},
|
||||
)
|
||||
|
||||
|
||||
def _compute_onsets_sync(src_path: str) -> list[float]:
|
||||
"""Blocking part of onset analysis — runs in a worker thread."""
|
||||
import soundfile as sf
|
||||
from services.onset_align import detect_speech_onsets
|
||||
audio, sr = sf.read(src_path, dtype="float32")
|
||||
return detect_speech_onsets(audio, sr)
|
||||
|
||||
|
||||
@router.get("/dub/onsets/{job_id}")
|
||||
async def dub_get_onsets(job_id: str):
|
||||
"""Speech-onset times for the timeline editor's snap-to-onset ticks (#280).
|
||||
|
||||
Prefers the Demucs-isolated vocals track (clean speech energy); falls
|
||||
back to the mixed audio. Computed once per job and cached as
|
||||
``onsets.json`` in the job directory; recomputed if the source audio is
|
||||
newer than the cache (e.g. re-ingest into the same job dir).
|
||||
"""
|
||||
import json
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
vocals = job.get("vocals_path")
|
||||
mix = job.get("audio_path")
|
||||
if vocals and os.path.exists(vocals):
|
||||
src_path, source = vocals, "vocals"
|
||||
elif mix and os.path.exists(mix):
|
||||
src_path, source = mix, "mix"
|
||||
else:
|
||||
raise HTTPException(status_code=404, detail="No audio track available for onset analysis")
|
||||
|
||||
# Containment inlined (not via _safe_job_path): CodeQL can't track the
|
||||
# sanitizer through a helper's return — the file's established idiom.
|
||||
base = os.path.realpath(DUB_DIR)
|
||||
cache_path = os.path.realpath(os.path.join(base, job_id, "onsets.json"))
|
||||
if not cache_path.startswith(base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid job id")
|
||||
try:
|
||||
if (
|
||||
os.path.exists(cache_path)
|
||||
and os.path.getmtime(cache_path) >= os.path.getmtime(src_path)
|
||||
):
|
||||
with open(cache_path, "r", encoding="utf-8") as f:
|
||||
cached = json.load(f)
|
||||
if isinstance(cached, dict) and isinstance(cached.get("onsets"), list):
|
||||
return cached
|
||||
except (OSError, ValueError):
|
||||
pass # unreadable/corrupt cache → recompute below
|
||||
|
||||
try:
|
||||
onsets = await asyncio.to_thread(_compute_onsets_sync, src_path)
|
||||
except Exception as e:
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"Onset analysis failed: {str(e)[:200]}",
|
||||
)
|
||||
|
||||
payload = {"onsets": onsets, "source": source}
|
||||
try:
|
||||
os.makedirs(os.path.dirname(cache_path), exist_ok=True)
|
||||
tmp_path = cache_path + ".tmp"
|
||||
with open(tmp_path, "w", encoding="utf-8") as f:
|
||||
json.dump(payload, f)
|
||||
os.replace(tmp_path, cache_path)
|
||||
except OSError as e:
|
||||
logger.warning("onsets cache write failed for %s: %s", job_id, e)
|
||||
return payload
|
||||
return FileResponse(preview_path, media_type="video/mp4")
|
||||
|
||||
|
||||
@router.get("/dub/thumb/{job_id}")
|
||||
@@ -1148,100 +729,6 @@ async def dub_preview_segment(job_id: str, segment_index: int):
|
||||
return FileResponse(seg_path, media_type="audio/wav")
|
||||
|
||||
|
||||
# ── Second-pass ASR QC (Wave 3.3 / Spec 5) ───────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/dub/qc/{job_id}")
|
||||
async def dub_qc_pass(job_id: str, lang: str = Query(None), drift_threshold: float = Query(0.5)):
|
||||
"""Re-recognize the dubbed audio and flag lines whose recognized text
|
||||
drifts from the target text. Opt-in, never fatal: the dub is untouched —
|
||||
this only annotates segments with a per-line drift score and a measured
|
||||
start/end, surfaced as "verify this line" markers feeding incremental
|
||||
re-dub. The generated text stays authoritative (design delta from
|
||||
pyvideotrans, which overwrites subtitles)."""
|
||||
from services import dub_qc
|
||||
from services.dub_pipeline import put_job, save_job
|
||||
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
tracks = job.get("dubbed_tracks", {})
|
||||
if lang and lang in tracks:
|
||||
wav_path = tracks[lang]["path"]
|
||||
elif tracks:
|
||||
wav_path = list(tracks.values())[0]["path"]
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail="No dubbed audio track generated yet")
|
||||
if not os.path.exists(wav_path):
|
||||
raise HTTPException(status_code=404, detail="Dubbed audio file not found")
|
||||
|
||||
segments = job.get("segments") or []
|
||||
if not segments:
|
||||
raise HTTPException(status_code=400, detail="Job has no segments")
|
||||
|
||||
def _recognize():
|
||||
from services.asr_backend import get_active_asr_backend
|
||||
backend = get_active_asr_backend()
|
||||
result = backend.transcribe(wav_path, word_timestamps=False)
|
||||
return result.get("segments", []), backend.id
|
||||
|
||||
try:
|
||||
from services.model_manager import _get_gpu_pool
|
||||
from services.asr_backend import ASRTimeoutError, run_transcribe_guarded
|
||||
recognized, engine_id = await run_transcribe_guarded(
|
||||
_get_gpu_pool(), _recognize, what="QC",
|
||||
)
|
||||
except ASRTimeoutError as e:
|
||||
# Backend is alive; ASR just couldn't finish in time. 504, not 500/connection.
|
||||
logger.warning("dub QC ASR pass timed out for %s: %s", job_id, e)
|
||||
raise HTTPException(status_code=504, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.exception("dub QC ASR pass failed for %s", job_id)
|
||||
raise HTTPException(status_code=500, detail=f"QC transcription failed: {e}")
|
||||
|
||||
seg_ids = job.get("seg_order") or [s.get("id", i) for i, s in enumerate(segments)]
|
||||
scored = dub_qc.score_dub(segments, recognized, drift_threshold=drift_threshold, seg_ids=seg_ids)
|
||||
|
||||
# Annotate each segment (non-destructive — content text untouched).
|
||||
by_id = {q.seg_id: q for q in scored}
|
||||
for i, s in enumerate(segments):
|
||||
sid = str(seg_ids[i]) if i < len(seg_ids) else str(s.get("id", i))
|
||||
q = by_id.get(sid)
|
||||
if q is None:
|
||||
continue
|
||||
s["qc_drift"] = q.drift
|
||||
s["qc_flagged"] = q.flagged
|
||||
s["qc_recognized"] = q.recognized_text
|
||||
if q.new_start is not None:
|
||||
s["qc_measured_start"] = q.new_start
|
||||
s["qc_measured_end"] = q.new_end
|
||||
put_job(job_id, job)
|
||||
save_job(job_id, job)
|
||||
|
||||
flagged = [q for q in scored if q.flagged]
|
||||
payload = json.dumps({"event": "qc_done", "engine": engine_id,
|
||||
"flagged": len(flagged), "total": len(scored)})
|
||||
try:
|
||||
from core import job_store
|
||||
job_store.append_event(job_id, f"data: {payload}\n\n")
|
||||
except Exception as e:
|
||||
# QC event fan-out is best-effort; the scores are already in the response.
|
||||
logger.debug("QC event append failed: %s", e)
|
||||
|
||||
return {
|
||||
"engine": engine_id,
|
||||
"total": len(scored),
|
||||
"flagged_count": len(flagged),
|
||||
"drift_threshold": drift_threshold,
|
||||
"segments": [
|
||||
{"seg_id": q.seg_id, "drift": q.drift, "flagged": q.flagged,
|
||||
"recognized_text": q.recognized_text,
|
||||
"measured_start": q.new_start, "measured_end": q.new_end}
|
||||
for q in scored
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/dub/download-audio/{job_id}")
|
||||
@router.get("/dub/download-audio/{job_id}/{filename}")
|
||||
async def dub_download_audio(job_id: str, lang: str = Query(None), preserve_bg: bool = Query(True), save_path: str = Query("")):
|
||||
@@ -1319,31 +806,9 @@ def _pick_subtitle_text(seg: dict, dual: bool) -> str:
|
||||
return f"{translated}\n<i>{original}</i>"
|
||||
|
||||
|
||||
# Subtitles deliberately have no ?save_path= variant: they're small text
|
||||
# bodies, so the Tauri side fetches them raw and writes the file itself via
|
||||
# the save_text_file command — the OS save dialog is the write authorization
|
||||
# (#309). The frontend's JSON-envelope save flow stays for binary exports.
|
||||
|
||||
|
||||
def _fitted_cue_times(job: dict, lang: str | None) -> list | None:
|
||||
"""Per-segment (start, end) on the fitted timeline when this job used
|
||||
stretch_video; None to use the original segment times. (Wave 3.1.)"""
|
||||
tracks = job.get("dubbed_tracks", {})
|
||||
lc = lang if (lang and lang in tracks) else (next(iter(tracks), None))
|
||||
entry = _video_stretch_plan_for(job, lc) if lc else None
|
||||
if not entry:
|
||||
return None
|
||||
from services.fitted_subtitles import fitted_cues
|
||||
return fitted_cues(job.get("segments", []), entry["plan"])
|
||||
|
||||
|
||||
@router.get("/dub/srt/{job_id}")
|
||||
@router.get("/dub/srt/{job_id}/{filename}")
|
||||
async def dub_export_srt(
|
||||
job_id: str,
|
||||
dual: bool = False,
|
||||
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
|
||||
):
|
||||
async def dub_export_srt(job_id: str, dual: bool = False):
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
@@ -1352,31 +817,22 @@ async def dub_export_srt(
|
||||
if not segments:
|
||||
raise HTTPException(status_code=400, detail="No transcript segments available")
|
||||
|
||||
# Subtitles must follow the audio the viewer hears, per timing strategy:
|
||||
# Smart Fit (phase B) overlays the fitted segment times directly;
|
||||
# stretch_video (Wave 3.1) regenerates cue times from the stretch plan.
|
||||
# Neither applies → original times.
|
||||
fitted = _fitted_segments_for(job, lang)
|
||||
if fitted:
|
||||
segments = _apply_fitted_times(segments, fitted)
|
||||
cues = None if fitted else _fitted_cue_times(job, lang)
|
||||
|
||||
srt_lines = []
|
||||
for i, seg in enumerate(segments):
|
||||
s, e = cues[i] if cues else (seg["start"], seg["end"])
|
||||
start_ts = _format_srt_time(seg["start"])
|
||||
end_ts = _format_srt_time(seg["end"])
|
||||
srt_lines.append(f"{i + 1}")
|
||||
srt_lines.append(f"{_format_srt_time(s)} --> {_format_srt_time(e)}")
|
||||
srt_lines.append(f"{start_ts} --> {end_ts}")
|
||||
srt_lines.append(_pick_subtitle_text(seg, dual))
|
||||
srt_lines.append("")
|
||||
|
||||
srt_content = "\n".join(srt_lines)
|
||||
base_name = os.path.splitext(job.get('filename', 'video'))[0]
|
||||
suffix = "_dual" if dual else ""
|
||||
dl_name = f"subtitles_{base_name}{suffix}.srt"
|
||||
return Response(
|
||||
content=srt_content,
|
||||
media_type="text/plain",
|
||||
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
|
||||
headers={"Content-Disposition": f'attachment; filename="subtitles_{base_name}{suffix}.srt"'},
|
||||
)
|
||||
|
||||
def _format_vtt_time(seconds):
|
||||
@@ -1388,11 +844,7 @@ def _format_vtt_time(seconds):
|
||||
|
||||
@router.get("/dub/vtt/{job_id}")
|
||||
@router.get("/dub/vtt/{job_id}/{filename}")
|
||||
async def dub_export_vtt(
|
||||
job_id: str,
|
||||
dual: bool = False,
|
||||
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
|
||||
):
|
||||
async def dub_export_vtt(job_id: str, dual: bool = False):
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
@@ -1401,28 +853,22 @@ async def dub_export_vtt(
|
||||
if not segments:
|
||||
raise HTTPException(status_code=400, detail="No transcript segments available")
|
||||
|
||||
# Same strategy-aware cue timing as /dub/srt (see comment there).
|
||||
fitted = _fitted_segments_for(job, lang)
|
||||
if fitted:
|
||||
segments = _apply_fitted_times(segments, fitted)
|
||||
cues = None if fitted else _fitted_cue_times(job, lang)
|
||||
|
||||
vtt_lines = ["WEBVTT", ""]
|
||||
for i, seg in enumerate(segments):
|
||||
s, e = cues[i] if cues else (seg["start"], seg["end"])
|
||||
start_ts = _format_vtt_time(seg["start"])
|
||||
end_ts = _format_vtt_time(seg["end"])
|
||||
vtt_lines.append(str(i + 1))
|
||||
vtt_lines.append(f"{_format_vtt_time(s)} --> {_format_vtt_time(e)}")
|
||||
vtt_lines.append(f"{start_ts} --> {end_ts}")
|
||||
vtt_lines.append(_pick_subtitle_text(seg, dual))
|
||||
vtt_lines.append("")
|
||||
|
||||
vtt_content = "\n".join(vtt_lines)
|
||||
base_name = os.path.splitext(job.get('filename', 'video'))[0]
|
||||
suffix = "_dual" if dual else ""
|
||||
dl_name = f"subtitles_{base_name}{suffix}.vtt"
|
||||
return Response(
|
||||
content=vtt_content,
|
||||
media_type="text/vtt",
|
||||
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
|
||||
headers={"Content-Disposition": f'attachment; filename="subtitles_{base_name}{suffix}.vtt"'},
|
||||
)
|
||||
|
||||
|
||||
|
||||
+237
-604
File diff suppressed because it is too large
Load Diff
@@ -2,7 +2,6 @@ import os
|
||||
import time
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Optional
|
||||
from fastapi import APIRouter
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
@@ -46,70 +45,6 @@ LANG_NAMES = {
|
||||
"id": "Indonesian", "uk": "Ukrainian",
|
||||
}
|
||||
|
||||
# Regional dialect hints (#280 item 2). Maps a BCP-47 dialect code to the
|
||||
# instruction injected into LLM translation prompts so the output uses that
|
||||
# region's vocabulary and grammar (the reporter's example: choosing Argentina
|
||||
# should yield "Vos sos muy listo", not the Peninsular "Tú eres muy listo").
|
||||
# Only LLM-backed paths can honor these — provider="openai" and the
|
||||
# quality="cinematic" refine pass. Keep entries short: they ride on every
|
||||
# per-segment prompt, so verbosity = wall time.
|
||||
DIALECT_HINTS = {
|
||||
# Spanish
|
||||
"es-ES": "European Spanish (Spain): use tú/vosotros forms and Peninsular vocabulary.",
|
||||
"es-MX": "Mexican Spanish: use tú/ustedes forms and Mexican vocabulary.",
|
||||
"es-AR": "Rioplatense Spanish (Argentina): use voseo — 'vos' with its verb forms (e.g. 'vos sos', 'tenés') and 'ustedes'; prefer Argentinian vocabulary.",
|
||||
"es-CO": "Colombian Spanish: use tú/usted as natural in Colombia and Colombian vocabulary.",
|
||||
"es-CL": "Chilean Spanish: use Chilean vocabulary and expressions.",
|
||||
# Portuguese
|
||||
"pt-BR": "Brazilian Portuguese: use 'você' forms, Brazilian vocabulary and spelling.",
|
||||
"pt-PT": "European Portuguese: use European vocabulary, spelling, and 'tu' where natural.",
|
||||
# English
|
||||
"en-US": "American English: use US spelling and vocabulary.",
|
||||
"en-GB": "British English: use UK spelling and vocabulary.",
|
||||
"en-AU": "Australian English: use Australian spelling and vocabulary.",
|
||||
"en-IN": "Indian English: use Indian English vocabulary and conventions.",
|
||||
# French
|
||||
"fr-FR": "Metropolitan French (France): use standard French vocabulary.",
|
||||
"fr-CA": "Canadian French (Québec): use Québécois vocabulary and expressions.",
|
||||
"fr-BE": "Belgian French: use Belgian vocabulary (e.g. septante, nonante).",
|
||||
# German
|
||||
"de-DE": "Standard German (Germany): use Federal German vocabulary.",
|
||||
"de-AT": "Austrian German: use Austrian vocabulary (e.g. Jänner, Erdapfel).",
|
||||
"de-CH": "Swiss Standard German: use Swiss vocabulary and 'ss' instead of 'ß'.",
|
||||
# Arabic
|
||||
"ar-EG": "Egyptian Arabic: use Egyptian colloquial vocabulary where natural for dubbing.",
|
||||
"ar-SA": "Gulf/Saudi Arabic flavor: prefer vocabulary natural to the Gulf region.",
|
||||
"ar-MA": "Moroccan Arabic (Darija) flavor: prefer vocabulary natural to Morocco.",
|
||||
# Dutch
|
||||
"nl-NL": "Netherlands Dutch: use vocabulary standard in the Netherlands.",
|
||||
"nl-BE": "Belgian Dutch (Flemish): use Flemish vocabulary and expressions.",
|
||||
}
|
||||
|
||||
|
||||
def dialect_clause(dialect: Optional[str]) -> str:
|
||||
"""Prompt fragment for a requested dialect, or '' when unset/unknown.
|
||||
|
||||
Unknown-but-plausible codes (e.g. "es-PE") still get a generic regional
|
||||
clause so users aren't limited to the curated list.
|
||||
"""
|
||||
if not dialect or not str(dialect).strip():
|
||||
return ""
|
||||
code = str(dialect).strip()
|
||||
hint = DIALECT_HINTS.get(code)
|
||||
if hint:
|
||||
return f" Target dialect — {hint}"
|
||||
# Generic fallback for any lang-REGION shaped code we don't curate.
|
||||
if "-" in code:
|
||||
lang, _, region = code.partition("-")
|
||||
lang_name = LANG_NAMES.get(lang, lang)
|
||||
if region:
|
||||
return (
|
||||
f" Use the vocabulary, grammar, and expressions of {lang_name} "
|
||||
f"as spoken in the region '{region}'."
|
||||
)
|
||||
return ""
|
||||
|
||||
|
||||
# Per-language script enforcement. Maps language code → required Unicode
|
||||
# block(s) the translation must contain. Used as a sanity gate after the
|
||||
# LLM responds: if the output contains <50% characters from the expected
|
||||
@@ -157,49 +92,15 @@ _nllb_tokenizer = None
|
||||
_nllb_device = None
|
||||
|
||||
|
||||
def _dialect_flags(req, applied: bool) -> dict:
|
||||
"""Response fields describing whether the requested dialect was honored.
|
||||
|
||||
Empty dict when no dialect was requested, so existing response shapes
|
||||
stay byte-identical for callers that never send one.
|
||||
"""
|
||||
if not getattr(req, "dialect", None):
|
||||
return {}
|
||||
return {"dialect": req.dialect, "dialect_applied": bool(applied)}
|
||||
|
||||
|
||||
def _guess_lang_from_text(segments) -> str | None:
|
||||
"""Best-effort source language from segment text, by script.
|
||||
|
||||
Used only as a last resort when neither the request nor the job carries a
|
||||
detected language. Without this, the bare "en" fallback below forces
|
||||
en -> en on non-English audio (e.g. Korean), which has no Argos package and
|
||||
fails every segment even though ASR detected the language correctly.
|
||||
"""
|
||||
text = " ".join((getattr(s, "text", "") or "") for s in (segments or [])[:8])
|
||||
has = lambda lo, hi: any(lo <= ord(c) <= hi for c in text)
|
||||
if has(0x3040, 0x30FF):
|
||||
return "ja" # Hiragana/Katakana — check before CJK (Japanese uses Kanji too)
|
||||
if has(0xAC00, 0xD7A3) or has(0x1100, 0x11FF):
|
||||
return "ko" # Hangul
|
||||
if has(0x4E00, 0x9FFF):
|
||||
return "zh" # CJK ideographs
|
||||
if has(0x0400, 0x04FF):
|
||||
return "ru" # Cyrillic
|
||||
if has(0x0600, 0x06FF):
|
||||
return "ar" # Arabic
|
||||
return None
|
||||
|
||||
|
||||
def _resolve_source_lang(req: TranslateRequest) -> str:
|
||||
"""Pick source language: explicit request > job.source_lang > text guess > 'en'."""
|
||||
"""Pick source language: explicit request > job.source_lang > 'en' fallback."""
|
||||
if getattr(req, "source_lang", None):
|
||||
return req.source_lang
|
||||
if getattr(req, "job_id", None):
|
||||
job = _get_job(req.job_id)
|
||||
if job and job.get("source_lang"):
|
||||
return job["source_lang"]
|
||||
return _guess_lang_from_text(getattr(req, "segments", None)) or "en"
|
||||
return "en"
|
||||
|
||||
|
||||
def _unload_nllb():
|
||||
@@ -302,8 +203,7 @@ async def dub_translate(req: TranslateRequest):
|
||||
translated = await loop.run_in_executor(_gpu_pool, _translate_nllb)
|
||||
if os.environ.get("OMNIVOICE_UNLOAD_NLLB", "1") == "1":
|
||||
_unload_nllb()
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
**_dialect_flags(req, applied=False)}
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
|
||||
|
||||
# OpenAI / Ollama Local LLM Translation
|
||||
if provider == "openai":
|
||||
@@ -336,16 +236,10 @@ async def dub_translate(req: TranslateRequest):
|
||||
f"only — do not use Latin/Roman letters, do not "
|
||||
f"transliterate, do not output any other language."
|
||||
)
|
||||
# #280 item 2 — regional dialect/vocabulary. Only applied when
|
||||
# the dialect belongs to the target language (a leftover
|
||||
# "es-AR" must not contaminate a French translation).
|
||||
dia_clause = ""
|
||||
if req.dialect and str(req.dialect).lower().startswith(str(tgt_code).lower()[:2]):
|
||||
dia_clause = dialect_clause(req.dialect)
|
||||
return (
|
||||
f"You are a professional dubbing translator. "
|
||||
f"Translate the user's text from {src_name} into "
|
||||
f"{tgt_name}.{script_clause}{dia_clause} "
|
||||
f"{tgt_name}.{script_clause} "
|
||||
f"Reply ONLY with the translated {tgt_name} text, do not "
|
||||
f"add quotes, notes, headers, explanations, or commentary."
|
||||
)
|
||||
@@ -405,24 +299,18 @@ async def dub_translate(req: TranslateRequest):
|
||||
tasks = [loop.run_in_executor(_cpu_pool, _translate_llm, seg) for seg in req.segments]
|
||||
translated = await asyncio.gather(*tasks)
|
||||
translated.sort(key=lambda x: str(x["id"]))
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
**_dialect_flags(req, applied=True)}
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
|
||||
|
||||
# Offline Argos Translate
|
||||
if provider == "argos" or provider == "libretranslate":
|
||||
try:
|
||||
import argostranslate # noqa: F401
|
||||
except ImportError:
|
||||
# Single-source the install command from the engine registry so
|
||||
# this 400 and the proactive Install button in the Engine
|
||||
# selector can never drift (see translation_engines.install_command).
|
||||
from services.translation_engines import install_command
|
||||
cmd = install_command("argos") or "uv pip install argostranslate"
|
||||
friendly = (
|
||||
f"The '{provider}' translation engine needs the optional "
|
||||
f"`argostranslate` Python package, which isn't installed in "
|
||||
f"this backend. Install it with `{cmd}` "
|
||||
f"and restart the server, or "
|
||||
f"this backend. Install it with `uv pip install argostranslate` "
|
||||
f"(or `pip install argostranslate`) and restart the server, or "
|
||||
f"switch the Engine dropdown to another provider."
|
||||
)
|
||||
return JSONResponse(status_code=400, content={"error": friendly})
|
||||
@@ -465,8 +353,7 @@ async def dub_translate(req: TranslateRequest):
|
||||
return results
|
||||
|
||||
translated = await loop.run_in_executor(_cpu_pool, _translate_argos)
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
**_dialect_flags(req, applied=False)}
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
|
||||
|
||||
# Legacy / API Deep_Translator logic.
|
||||
# Preflight the optional `deep_translator` dep once so we fail with a
|
||||
@@ -475,16 +362,11 @@ async def dub_translate(req: TranslateRequest):
|
||||
try:
|
||||
import deep_translator # noqa: F401
|
||||
except ImportError:
|
||||
# Same single-source install command as the Engine selector's Install
|
||||
# button (translation_engines.install_command) — google/deepl/
|
||||
# microsoft/mymemory all share the deep_translator package.
|
||||
from services.translation_engines import install_command
|
||||
cmd = install_command(provider) or "uv pip install deep_translator"
|
||||
friendly = (
|
||||
f"The '{provider}' translation engine needs the optional "
|
||||
f"`deep_translator` Python package, which isn't installed in "
|
||||
f"this backend. Install it with `{cmd}` "
|
||||
f"and restart the server, or "
|
||||
f"this backend. Install it with `uv pip install deep_translator` "
|
||||
f"(or `pip install deep_translator`) and restart the server, or "
|
||||
f"switch the Engine dropdown to Argos (local, bundled), NLLB "
|
||||
f"(local, heavier), or OpenAI (LLM)."
|
||||
)
|
||||
@@ -580,17 +462,13 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
except Exception as e:
|
||||
logger.debug("non-LLM rate_ratio prediction skipped: %s", e)
|
||||
|
||||
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
"quality_used": "fast", **_dialect_flags(req, applied=False)}
|
||||
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang, "quality_used": "fast"}
|
||||
|
||||
# Autofit is Cinematic + a strict "never exceed the slot" fit pass, so both
|
||||
# qualities take the LLM refine path below. Fast (and anything else) returns
|
||||
# the plain translation unchanged.
|
||||
if quality not in ("cinematic", "autofit"):
|
||||
if quality != "cinematic":
|
||||
return base
|
||||
|
||||
if not cinematic_available():
|
||||
logger.warning("%s requested but no LLM configured — returning Fast result.", quality)
|
||||
logger.warning("cinematic requested but no LLM configured — returning Fast result.")
|
||||
base["cinematic_skipped"] = "no-llm-configured"
|
||||
return base
|
||||
|
||||
@@ -614,19 +492,12 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
if not pairs:
|
||||
return base
|
||||
|
||||
# #280 item 2: thread the regional-dialect hint into the reflect/adapt
|
||||
# prompts. Guard against a stale dialect from another language.
|
||||
dialect_hint = ""
|
||||
if req.dialect and str(req.dialect).lower().startswith(str(req.target_lang).lower()[:2]):
|
||||
dialect_hint = dialect_clause(req.dialect)
|
||||
|
||||
refined = await cinematic_refine_many(
|
||||
pairs,
|
||||
source_lang=src_lang,
|
||||
target_lang=req.target_lang,
|
||||
glossary=req.glossary,
|
||||
directions=directions,
|
||||
dialect_hint=dialect_hint,
|
||||
executor=_cpu_pool,
|
||||
)
|
||||
refined_by_id = {r["id"]: r for r in refined}
|
||||
@@ -673,7 +544,6 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
slot_seconds=float(slot),
|
||||
target_lang=req.target_lang,
|
||||
source_text=source_by_id.get(seg_id),
|
||||
strict=(quality == "autofit"),
|
||||
)
|
||||
if fit.get("text"):
|
||||
out["text"] = fit["text"]
|
||||
@@ -689,6 +559,5 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
"translated": merged,
|
||||
"target_lang": req.target_lang,
|
||||
"source_lang": src_lang,
|
||||
"quality_used": quality,
|
||||
**_dialect_flags(req, applied=bool(dialect_hint)),
|
||||
"quality_used": "cinematic",
|
||||
}
|
||||
|
||||
@@ -266,26 +266,11 @@ class SelectEngineRequest(BaseModel):
|
||||
backend_id: str
|
||||
|
||||
|
||||
class SelectEngineResponse(BaseModel):
|
||||
family: str
|
||||
active: str
|
||||
env_override: bool
|
||||
# Routing verdict for the selected engine on THIS host (#21). Always present
|
||||
# so the UI can show a confirm/warning toast on a cpu_fallback pick without
|
||||
# branching on key presence; defaults match a legacy/degraded row.
|
||||
routing_status: str = "cpu_only"
|
||||
effective_device: str = "cpu"
|
||||
routing_reason: str | None = None
|
||||
|
||||
|
||||
@router.post("/engines/select", response_model=SelectEngineResponse)
|
||||
@router.post("/engines/select")
|
||||
def select_engine(req: SelectEngineRequest):
|
||||
"""Persist a family's engine pick to prefs.json. Refuses unknown backends,
|
||||
backends whose deps aren't installed, AND backends that cannot run on THIS
|
||||
host's hardware (routing_status == "unavailable") — so the UI can't silently
|
||||
brick a pipeline by picking an engine that needs a GPU this machine lacks.
|
||||
A `cpu_fallback` pick is allowed (it runs, just slower) — only a hard
|
||||
`unavailable` is blocked. LLM is never routing-gated (its status is "n/a")."""
|
||||
"""Persist a family's engine pick to prefs.json. Refuses unknown backends
|
||||
+ refuses backends whose deps aren't installed (so the UI can't silently
|
||||
brick a pipeline by picking an unavailable engine)."""
|
||||
family = _FAMILIES.get(req.family)
|
||||
if not family:
|
||||
raise HTTPException(400, f"Unknown family: {req.family}. Expected one of tts/asr/llm.")
|
||||
@@ -293,26 +278,12 @@ def select_engine(req: SelectEngineRequest):
|
||||
available = {b["id"]: b for b in module.list_backends()}
|
||||
if req.backend_id not in available:
|
||||
raise HTTPException(400, f"Unknown {req.family} backend: {req.backend_id!r}")
|
||||
entry = available[req.backend_id]
|
||||
if not entry["available"]:
|
||||
reason = entry.get("reason") or "unavailable"
|
||||
if not available[req.backend_id]["available"]:
|
||||
reason = available[req.backend_id].get("reason") or "unavailable"
|
||||
raise HTTPException(400, f"Backend {req.backend_id} not ready: {reason}")
|
||||
# Host-routing gate (no silent CPU fallback). `.get` is defensive so an
|
||||
# older/legacy payload without routing keys still selects cleanly.
|
||||
if entry.get("routing_status") == "unavailable":
|
||||
why = entry.get("routing_reason") or "requires a GPU this host doesn't have"
|
||||
raise HTTPException(
|
||||
400,
|
||||
f"Backend {req.backend_id} can't run on this machine: {why}. "
|
||||
f"Pick an engine with a CPU path, or one that supports this host's GPU.",
|
||||
)
|
||||
prefs.set_(pref_key, req.backend_id)
|
||||
return {
|
||||
"family": req.family,
|
||||
"active": module.active_backend_id(),
|
||||
"env_override": bool(__import__("os").environ.get(f"OMNIVOICE_{req.family.upper()}_BACKEND")),
|
||||
# Echo the routing verdict so the UI can warn on a cpu_fallback pick.
|
||||
"routing_status": entry.get("routing_status", "cpu_only"),
|
||||
"effective_device": entry.get("effective_device", "cpu"),
|
||||
"routing_reason": entry.get("routing_reason"),
|
||||
}
|
||||
|
||||
@@ -2,7 +2,6 @@ import os
|
||||
import io
|
||||
import uuid
|
||||
import time
|
||||
import random
|
||||
import asyncio
|
||||
import tempfile
|
||||
import contextlib
|
||||
@@ -12,36 +11,16 @@ from typing import Optional
|
||||
from fastapi import APIRouter, File, Form, UploadFile, HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
import sqlite3
|
||||
from core.db import db_conn, ensure_schema
|
||||
from core.db import db_conn
|
||||
from core.config import OUTPUTS_DIR, VOICES_DIR
|
||||
import functools
|
||||
from services.model_manager import (
|
||||
get_model, _gpu_pool, run_on_gpu_pool_guarded, GpuJobTimeoutError,
|
||||
)
|
||||
from services.model_manager import get_model, _gpu_pool
|
||||
from services.audio_io import _safe_torchaudio_save
|
||||
from core import event_bus
|
||||
from omnivoice.utils.voice_design import heal_design_instruct
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.generate")
|
||||
|
||||
|
||||
def _profile_instruct(row):
|
||||
"""Validator-safe instruct for a stored profile row.
|
||||
|
||||
Sanitizes the persisted instruct (dropping the ``"[object Object]"``
|
||||
sentinel / freeform prose that older builds saved) and, for a design row,
|
||||
rebuilds the tags from ``vd_states`` when the stored value is unusable — so
|
||||
a poisoned/legacy profile never 400-s generation (#550 #571 #594 #596).
|
||||
"""
|
||||
try:
|
||||
vd = row["vd_states"]
|
||||
except (KeyError, IndexError):
|
||||
vd = None
|
||||
return heal_design_instruct(row["instruct"], vd)
|
||||
|
||||
|
||||
def _render_with_pauses(gen_span, segments, sample_rate):
|
||||
"""Synthesize ``[(text, pause_ms), ...]`` spans and stitch silence between
|
||||
them (issue #276).
|
||||
@@ -80,172 +59,13 @@ def _render_with_pauses(gen_span, segments, sample_rate):
|
||||
parts.append(torch.zeros(*shape, dtype=ref.dtype, device=ref.device))
|
||||
return torch.cat(parts, dim=-1)
|
||||
|
||||
|
||||
def _sanitize_audio(audio_out):
|
||||
"""Replace non-finite samples (NaN / ±inf) with silence so a model glitch
|
||||
can't produce an unreadable WAV (#629). Returns the input unchanged when it's
|
||||
already finite or isn't a tensor. Never raises."""
|
||||
try:
|
||||
import torch
|
||||
if torch.is_tensor(audio_out) and not bool(torch.isfinite(audio_out).all()):
|
||||
logger.warning(
|
||||
"Generated audio contained non-finite samples (NaN/inf) — "
|
||||
"sanitizing to silence to keep the WAV decodable (#629)."
|
||||
)
|
||||
return torch.nan_to_num(audio_out, nan=0.0, posinf=0.0, neginf=0.0)
|
||||
except Exception:
|
||||
pass
|
||||
return audio_out
|
||||
|
||||
|
||||
def _apply_effect_chain(audio_out, sample_rate, effect_preset, *, skip_mastering=False):
|
||||
"""Shared post-DSP for /generate: preset validation → mastering →
|
||||
effect chain → loudness normalization.
|
||||
|
||||
``skip_mastering`` honors a backend's ``applies_own_mastering`` flag
|
||||
(issue #312): studio engines (e.g. VoxCPM2's native 48 kHz output)
|
||||
opt out of the broadcast Compressor + Reverb chain that's tuned for
|
||||
OmniVoice's 24 kHz clone output. Loudness normalization still runs —
|
||||
it's a benign peak scale. Mirrors ``_run_tts`` in openai_compat.py.
|
||||
"""
|
||||
from services.audio_dsp import (
|
||||
EFFECT_PRESETS, apply_mastering, normalize_audio,
|
||||
apply_effects_chain, get_effect_chain,
|
||||
)
|
||||
|
||||
# #629: a numerical glitch in the model (observed on MPS) can leave NaN/±inf
|
||||
# samples, which write an unreadable WAV that then fails decoding with an
|
||||
# opaque "ffmpeg returned error code: 183 / Invalid data" — surfaced to the
|
||||
# user as a misleading "ran out of memory". Replace non-finite samples with
|
||||
# silence here, before any DSP/encode touches the audio, so the output is
|
||||
# always a valid WAV. Covers the raw path too (it returns just below).
|
||||
audio_out = _sanitize_audio(audio_out)
|
||||
|
||||
preset = effect_preset or "broadcast"
|
||||
if preset not in EFFECT_PRESETS:
|
||||
raise ValueError(
|
||||
f"Unknown effect preset: {preset!r}. "
|
||||
f"Valid: {list(EFFECT_PRESETS.keys())}"
|
||||
)
|
||||
|
||||
if preset == "raw":
|
||||
# Raw: skip all DSP — return raw model output
|
||||
return audio_out
|
||||
|
||||
if not skip_mastering:
|
||||
audio_out = apply_mastering(audio_out, sample_rate=sample_rate)
|
||||
chain = get_effect_chain(preset)
|
||||
if chain:
|
||||
audio_out = apply_effects_chain(
|
||||
audio_out, sample_rate=sample_rate, chain=chain,
|
||||
)
|
||||
return normalize_audio(audio_out, target_dBFS=-2.0)
|
||||
|
||||
|
||||
def _oom_friendly_reraise(e):
|
||||
"""Best-effort cache flush + the user-facing OOM hint shared by both
|
||||
inference paths."""
|
||||
import gc
|
||||
import torch
|
||||
gc.collect()
|
||||
if torch.backends.mps.is_available():
|
||||
torch.mps.empty_cache()
|
||||
elif torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
# #278: don't mislabel a torch.compile/Triton/Inductor crash as an
|
||||
# out-of-memory condition. (model_manager's generate wrapper already
|
||||
# retries these eagerly; this only triggers if that retry also died.)
|
||||
from services.model_manager import _is_compile_runtime_failure
|
||||
if _is_compile_runtime_failure(e):
|
||||
raise RuntimeError(
|
||||
f"TTS engine hit a torch.compile/Triton error (not out of memory). "
|
||||
f"Disable torch.compile in Settings → Performance, use the Flush "
|
||||
f"button to reload the model, then regenerate. Underlying error: {e}"
|
||||
) from e
|
||||
# #437: a Permission-denied / exec failure (e.g. a bundled engine binary
|
||||
# that lost its +x bit) is NOT an OOM — don't send the user to the Flush
|
||||
# button; tell them what's actually wrong.
|
||||
es = str(e)
|
||||
if isinstance(e, PermissionError) or "Permission denied" in es or "Errno 13" in es:
|
||||
raise RuntimeError(
|
||||
f"A required engine binary couldn't be executed (permission denied). "
|
||||
f"This usually means a bundled binary lost its execute bit — reinstall, "
|
||||
f"or run `chmod +x` on the engine binary named in the error. "
|
||||
f"Underlying error: {e}"
|
||||
) from e
|
||||
# #629: a decode/ffmpeg failure on the rendered audio is NOT out of memory —
|
||||
# it's unreadable audio (usually a transient numerical glitch). Say so rather
|
||||
# than sending the user down the OOM path.
|
||||
if "ffmpeg returned error" in es or "Decoding failed" in es or "Invalid data found" in es:
|
||||
raise RuntimeError(
|
||||
f"The engine produced unreadable audio (a decode step failed) — this is "
|
||||
f"usually a transient glitch. Use the Flush button to reload the model, "
|
||||
f"then regenerate. Underlying error: {e}"
|
||||
) from e
|
||||
# #664: a bad voice-design instruct (free-form prose, mixed EN/ZH, or
|
||||
# conflicting tags) raises "Unsupported instruct items …" / "Cannot mix …
|
||||
# in a single instruct" / "Conflicting instruct items …" from omnivoice's
|
||||
# _resolve_instruct. That's a USER-INPUT validation error, not an OOM. Match
|
||||
# on the message signature (NOT the type — a lower layer can wrap the original
|
||||
# ValueError, which is why the route's `except ValueError` guard misses it)
|
||||
# and re-raise as a clean ValueError so the route returns a 400 with the
|
||||
# instruct guidance, instead of a 500 telling the user to Flush for memory
|
||||
# they never ran out of. (Complements the client-side guard in #658/#612.)
|
||||
_low = es.lower()
|
||||
if ("unsupported instruct items" in _low
|
||||
or "conflicting instruct items" in _low
|
||||
or "in a single instruct" in _low):
|
||||
raise ValueError(es) from e
|
||||
# #705: a corrupt or wrong-architecture native component (a .dll / .pyd / .exe
|
||||
# — torch, ffmpeg, or a bundled engine binary) fails to load/spawn on Windows
|
||||
# with "[WinError 193] %1 is not a valid Win32 application". That is NOT OOM,
|
||||
# and Flush won't help — reinstalling/repairing the component is the real fix.
|
||||
if "[winerror 193]" in _low or "is not a valid win32 application" in _low:
|
||||
raise RuntimeError(
|
||||
f"A native component (a DLL / .pyd / .exe — e.g. torch, ffmpeg, or an "
|
||||
f"engine binary) is corrupt or built for the wrong architecture "
|
||||
f"([WinError 193]). Reinstall or repair that component — the Flush "
|
||||
f"button won't help here. Underlying error: {e}"
|
||||
) from e
|
||||
# #715: a "[Errno 32] Broken pipe" (BrokenPipeError) surfacing from
|
||||
# generation is NOT out of memory — it means the backend's stdout/stderr
|
||||
# pipe to the desktop shell that launched it closed mid-render (an orphaned
|
||||
# backend whose parent shell exited or relaunched). main.py wraps
|
||||
# sys.stdout/stderr to swallow EPIPE, but a C-level write inside the native
|
||||
# engine/torch can still raise one past that guard. Flush won't help —
|
||||
# relaunching the app re-parents the backend to a live shell.
|
||||
# #756: the GPU's compute capability isn't in this PyTorch build's arch list,
|
||||
# so CUDA can't launch kernels ("no kernel image is available for execution").
|
||||
# NOT OOM. get_best_device() now falls back to CPU up front, but classify the
|
||||
# raw error too in case CUDA was forced (OMNIVOICE_FORCE_CUDA) or a sub-path
|
||||
# still ran on the GPU — point at the real fix, not the Flush button.
|
||||
if "no kernel image is available" in _low:
|
||||
raise RuntimeError(
|
||||
f"Your GPU isn't supported by the installed PyTorch build (CUDA can't "
|
||||
f"launch kernels for its compute capability). Switch the compute device "
|
||||
f"to CPU in Settings, or install a matching PyTorch (e.g. a cu128 build "
|
||||
f"for newer GPUs). The Flush button won't help. Underlying error: {e}"
|
||||
) from e
|
||||
if isinstance(e, BrokenPipeError) or "broken pipe" in _low or "errno 32" in _low:
|
||||
raise RuntimeError(
|
||||
f"The backend lost its output pipe mid-generation — the desktop app "
|
||||
f"that launched it closed or relaunched ([Errno 32] Broken pipe). "
|
||||
f"Restart the app and try again; the Flush button won't help here. "
|
||||
f"Underlying error: {e}"
|
||||
) from e
|
||||
raise RuntimeError(
|
||||
f"TTS engine stopped mid-generation. This usually means it ran out of memory. "
|
||||
f"Try the Flush button to reload the model, then regenerate. Underlying error: {e}"
|
||||
)
|
||||
|
||||
|
||||
def _run_inference(
|
||||
model, text, language, ref_audio_path, ref_text, instruct, duration,
|
||||
num_step, guidance_scale, speed, t_shift, denoise,
|
||||
postprocess_output, layer_penalty_factor, position_temperature,
|
||||
class_temperature, used_seed, effect_preset="broadcast",
|
||||
max_chunk_chars=None, crossfade_ms=None,
|
||||
):
|
||||
from services.audio_dsp import apply_mastering, normalize_audio, apply_effects_chain, get_effect_chain
|
||||
import torch
|
||||
try:
|
||||
if used_seed is not None:
|
||||
@@ -279,124 +99,53 @@ def _run_inference(
|
||||
)[0]
|
||||
audio_out = _render_with_pauses(_gen_span, segments, sr)
|
||||
else:
|
||||
# Wave 1.2: long text is split at sentence boundaries and the
|
||||
# per-chunk audio crossfaded — removes the length ceiling. Short
|
||||
# text takes the single-shot path below unchanged. [pause] inputs
|
||||
# keep the dedicated stitcher above (spans are already short).
|
||||
from services.chunked_tts import (
|
||||
DEFAULT_CROSSFADE_MS, DEFAULT_MAX_CHUNK_CHARS,
|
||||
concatenate_audio_chunks, split_text_into_chunks,
|
||||
audios = model.generate(
|
||||
text=text, language=language, ref_audio=ref_audio_path,
|
||||
ref_text=ref_text, instruct=instruct, duration=duration,
|
||||
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
|
||||
denoise=denoise, postprocess_output=postprocess_output,
|
||||
**kwargs
|
||||
)
|
||||
_max_chars = DEFAULT_MAX_CHUNK_CHARS if max_chunk_chars is None else max_chunk_chars
|
||||
_xfade_ms = DEFAULT_CROSSFADE_MS if crossfade_ms is None else crossfade_ms
|
||||
text_chunks = split_text_into_chunks(text, _max_chars)
|
||||
if len(text_chunks) > 1:
|
||||
parts = []
|
||||
for i, chunk_text in enumerate(text_chunks):
|
||||
# Vary the seed per chunk (deterministically) to avoid
|
||||
# correlated RNG artifacts across chunk boundaries.
|
||||
if used_seed is not None:
|
||||
torch.manual_seed(used_seed + i)
|
||||
parts.append(model.generate(
|
||||
text=chunk_text, language=language, ref_audio=ref_audio_path,
|
||||
ref_text=ref_text, instruct=instruct, duration=None,
|
||||
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
|
||||
denoise=denoise, postprocess_output=postprocess_output,
|
||||
**kwargs
|
||||
)[0])
|
||||
audio_out = concatenate_audio_chunks(parts, sr, _xfade_ms)
|
||||
else:
|
||||
audios = model.generate(
|
||||
text=text, language=language, ref_audio=ref_audio_path,
|
||||
ref_text=ref_text, instruct=instruct, duration=duration,
|
||||
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
|
||||
denoise=denoise, postprocess_output=postprocess_output,
|
||||
**kwargs
|
||||
)
|
||||
audio_out = audios[0]
|
||||
audio_out = audios[0]
|
||||
|
||||
# Apply DSP effect preset. The OmniVoice model never masters its own
|
||||
# output, so mastering always runs here (unchanged behavior).
|
||||
return _apply_effect_chain(audio_out, sr, effect_preset)
|
||||
# Apply DSP effect preset
|
||||
_effect_preset = effect_preset or "broadcast"
|
||||
|
||||
# Validate preset ID
|
||||
from services.audio_dsp import EFFECT_PRESETS
|
||||
if _effect_preset not in EFFECT_PRESETS:
|
||||
raise ValueError(
|
||||
f"Unknown effect preset: {_effect_preset!r}. "
|
||||
f"Valid: {list(EFFECT_PRESETS.keys())}"
|
||||
)
|
||||
|
||||
if _effect_preset == "raw":
|
||||
# Raw: skip all DSP — return raw model output
|
||||
return audio_out
|
||||
|
||||
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
|
||||
_chain = get_effect_chain(_effect_preset)
|
||||
if _chain:
|
||||
mastered_audio = apply_effects_chain(
|
||||
mastered_audio, sample_rate=sr, chain=_chain,
|
||||
)
|
||||
|
||||
return normalize_audio(mastered_audio, target_dBFS=-2.0)
|
||||
|
||||
except ValueError as e:
|
||||
# Don't wrap validation errors in OOM message
|
||||
raise e
|
||||
except Exception as e:
|
||||
_oom_friendly_reraise(e)
|
||||
|
||||
|
||||
def _run_backend_inference(
|
||||
backend, text, language, ref_audio_path, ref_text, instruct, duration,
|
||||
num_step, guidance_scale, speed, denoise, postprocess_output,
|
||||
used_seed, effect_preset="broadcast",
|
||||
max_chunk_chars=None, crossfade_ms=None,
|
||||
):
|
||||
"""Engine-aware twin of :func:`_run_inference` (issue #312).
|
||||
|
||||
Runs the request through a pluggable ``TTSBackend`` adapter instead of the
|
||||
OmniVoice model directly. The adapter protocol is narrower than the
|
||||
OmniVoice-native surface — engine-specific extras (``t_shift``,
|
||||
``layer_penalty_factor``, …) only exist on the native path, which is why
|
||||
OmniVoice itself still goes through ``_run_inference``.
|
||||
"""
|
||||
import torch
|
||||
try:
|
||||
if used_seed is not None:
|
||||
torch.manual_seed(used_seed)
|
||||
|
||||
if language and language.lower() == "auto":
|
||||
language = None
|
||||
|
||||
gen_kwargs = dict(
|
||||
language=language, ref_audio=ref_audio_path, ref_text=ref_text,
|
||||
instruct=instruct, num_step=num_step, guidance_scale=guidance_scale,
|
||||
speed=speed, denoise=denoise, postprocess_output=postprocess_output,
|
||||
import gc
|
||||
gc.collect()
|
||||
if torch.backends.mps.is_available():
|
||||
torch.mps.empty_cache()
|
||||
elif torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
raise RuntimeError(
|
||||
f"TTS engine stopped mid-generation. This usually means it ran out of memory. "
|
||||
f"Try the Flush button to reload the model, then regenerate. Underlying error: {e}"
|
||||
)
|
||||
sr = backend.sample_rate
|
||||
|
||||
# Inline [pause Nms] markers (issue #276) work for every engine — the
|
||||
# silence stitching is model-free.
|
||||
from omnivoice.utils.text import parse_pause_markers
|
||||
segments = parse_pause_markers(text)
|
||||
has_pause = len(segments) > 1 or (segments and segments[0][1] > 0)
|
||||
|
||||
if has_pause:
|
||||
def _gen_span(span_text):
|
||||
# Per-span duration is left to the engine; an explicit overall
|
||||
# `duration` can't be meaningfully split across spans.
|
||||
return backend.generate(span_text, duration=None, **gen_kwargs)
|
||||
audio_out = _render_with_pauses(_gen_span, segments, sr)
|
||||
else:
|
||||
# Wave 1.2: sentence-boundary chunking for long text (see
|
||||
# _run_inference for the rationale; behavior is identical here).
|
||||
from services.chunked_tts import (
|
||||
DEFAULT_CROSSFADE_MS, DEFAULT_MAX_CHUNK_CHARS,
|
||||
concatenate_audio_chunks, split_text_into_chunks,
|
||||
)
|
||||
_max_chars = DEFAULT_MAX_CHUNK_CHARS if max_chunk_chars is None else max_chunk_chars
|
||||
_xfade_ms = DEFAULT_CROSSFADE_MS if crossfade_ms is None else crossfade_ms
|
||||
text_chunks = split_text_into_chunks(text, _max_chars)
|
||||
if len(text_chunks) > 1:
|
||||
parts = []
|
||||
for i, chunk_text in enumerate(text_chunks):
|
||||
if used_seed is not None:
|
||||
torch.manual_seed(used_seed + i)
|
||||
parts.append(backend.generate(chunk_text, duration=None, **gen_kwargs))
|
||||
audio_out = concatenate_audio_chunks(parts, sr, _xfade_ms)
|
||||
else:
|
||||
audio_out = backend.generate(text, duration=duration, **gen_kwargs)
|
||||
|
||||
return _apply_effect_chain(
|
||||
audio_out, sr, effect_preset,
|
||||
skip_mastering=getattr(backend, "applies_own_mastering", False),
|
||||
)
|
||||
|
||||
except ValueError as e:
|
||||
# Don't wrap validation errors in OOM message
|
||||
raise e
|
||||
except Exception as e:
|
||||
_oom_friendly_reraise(e)
|
||||
|
||||
|
||||
@router.post("/generate")
|
||||
@@ -419,125 +168,30 @@ async def generate_speech(
|
||||
profile_id: Optional[str] = Form(None),
|
||||
seed: Optional[int] = Form(None),
|
||||
effect_preset: str = Form("broadcast"),
|
||||
engine: Optional[str] = Form(None),
|
||||
# Wave 1.2 — unlimited-length generation: long text is split at sentence
|
||||
# boundaries and crossfaded. 0 disables chunking (whole text to engine).
|
||||
max_chunk_chars: int = Form(800, ge=0),
|
||||
crossfade_ms: int = Form(50, ge=0, le=1000),
|
||||
# Expressive-TTS Spec 01: apply the user pronunciation dictionary + inline
|
||||
# [[…]] overrides to the text before synthesis. Default ON; the global
|
||||
# OMNIVOICE_PRONUNCIATION pref can disable it for power users. Omitting it
|
||||
# with an empty dictionary is byte-identical to legacy behavior.
|
||||
pronounce: bool = Form(True),
|
||||
):
|
||||
# #502: NFC-normalize the input text so decomposed (NFD) diacritics — common
|
||||
# in pasted Vietnamese and other Latin-with-marks text — are composed to the
|
||||
# single codepoints the tokenizer/model expect, instead of base-letter +
|
||||
# combining-mark sequences that render as distorted/garbled speech. NFC is a
|
||||
# no-op for already-composed text; mirrors the duration estimator
|
||||
# (utils/duration.py) so the estimate and the synthesis see the same text.
|
||||
import unicodedata
|
||||
text = unicodedata.normalize("NFC", text)
|
||||
|
||||
# ── Engine resolution (issue #312) ──────────────────────────────────────
|
||||
# The request runs on the engine selected in Settings (POST /engines/select,
|
||||
# env var OMNIVOICE_TTS_BACKEND wins), or an explicit per-request `engine`
|
||||
# override — same pattern as /ws/tts's `engine` field and /v1/audio/speech's
|
||||
# `model`. Omitting both keeps the historical default (OmniVoice), so
|
||||
# existing API consumers see no change.
|
||||
from services.tts_backend import (
|
||||
OmniVoiceBackend, _mask_hf_tokens, active_backend_id, get_backend_class,
|
||||
)
|
||||
|
||||
engine_id = engine or active_backend_id()
|
||||
try:
|
||||
backend_cls = get_backend_class(engine_id)
|
||||
except ValueError:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=(
|
||||
f"Unknown TTS engine: {engine_id!r}. "
|
||||
"See GET /engines/tts for the list of valid engine ids."
|
||||
),
|
||||
)
|
||||
|
||||
_model = None
|
||||
_backend = None
|
||||
if backend_cls is OmniVoiceBackend:
|
||||
# OmniVoice keeps its native path: it carries the full advanced
|
||||
# parameter surface (t_shift, layer/position/class controls) that the
|
||||
# generic adapter protocol doesn't. Byte-identical to the old behavior.
|
||||
_model = await get_model()
|
||||
else:
|
||||
try:
|
||||
ok, msg = backend_cls.is_available()
|
||||
except Exception as exc:
|
||||
ok, msg = False, f"{type(exc).__name__}: {exc}"
|
||||
if not ok:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"TTS engine '{engine_id}' is not available: {_mask_hf_tokens(msg)}",
|
||||
)
|
||||
# Reuse the per-process instance cache shared with the engine
|
||||
# health-check route so weights load once, not per request.
|
||||
from api.routers.engines import _get_engine_instance
|
||||
_backend = _get_engine_instance(backend_cls)
|
||||
|
||||
# ── Routing gate (#21 — no silent CPU fallback). Computed ONCE per request
|
||||
# (host caps are constant; the per-request engine= override bypasses the
|
||||
# /engines/select gate, so this is the only place it's enforced for synth).
|
||||
from core.device_caps import detect_host_caps
|
||||
from services.engine_routing import resolve_routing, routing_notice
|
||||
_routing = resolve_routing(getattr(backend_cls, "gpu_compat", ("cpu",)), detect_host_caps())
|
||||
if _routing["routing_status"] == "unavailable":
|
||||
# The engine needs an accelerator this host lacks and has no CPU path.
|
||||
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
|
||||
_routing_notice = routing_notice(_routing) # (status, reason) or None
|
||||
_model = await get_model()
|
||||
|
||||
ref_audio_path = None
|
||||
cleanup_ref = False
|
||||
used_seed = seed
|
||||
resolved_profile_id = None
|
||||
history_mode = None # profile.kind when a profile drives; else inferred at insert
|
||||
|
||||
if profile_id:
|
||||
with db_conn() as conn:
|
||||
row = conn.execute("SELECT * FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
if row:
|
||||
resolved_profile_id = profile_id
|
||||
# `kind` is authoritative (0005): 'design' profiles condition on
|
||||
# their deterministic rendered sample + instruct; 'clone' on the
|
||||
# user's reference. Lock always wins (it pins a specific take).
|
||||
# Rows from pre-0004 DBs mid-upgrade may lack the column → fall
|
||||
# back to the legacy is_locked/instruct inference.
|
||||
try:
|
||||
profile_kind = row["kind"] or "clone"
|
||||
except (KeyError, IndexError):
|
||||
profile_kind = "design" if (row["instruct"] and not row["is_locked"] and not row["ref_audio_path"]) else "clone"
|
||||
history_mode = profile_kind
|
||||
if row["is_locked"] and row["locked_audio_path"]:
|
||||
ref_audio_path = os.path.join(VOICES_DIR, row["locked_audio_path"])
|
||||
if not ref_text:
|
||||
ref_text = row["ref_text"]
|
||||
if not instruct:
|
||||
instruct = _profile_instruct(row)
|
||||
instruct = row["instruct"]
|
||||
if used_seed is None and row["seed"] is not None:
|
||||
used_seed = row["seed"]
|
||||
elif profile_kind == "design":
|
||||
# Rendered sample (if present) carries the voice identity;
|
||||
# instruct alone is the fallback for legacy archetype rows.
|
||||
ref_audio_path = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
|
||||
if ref_audio_path and not ref_text and row["ref_text"]:
|
||||
ref_text = row["ref_text"]
|
||||
elif row["instruct"] and not row["is_locked"]:
|
||||
if not instruct:
|
||||
instruct = _profile_instruct(row)
|
||||
if used_seed is None and row["seed"] is not None:
|
||||
used_seed = row["seed"]
|
||||
elif row["instruct"] and not row["is_locked"] and not row["ref_audio_path"]:
|
||||
# Legacy design-shaped row (pre-0004 archetype materialization
|
||||
# failure path): instruct-only conditioning.
|
||||
if not instruct:
|
||||
instruct = _profile_instruct(row)
|
||||
instruct = row["instruct"]
|
||||
if used_seed is None and row["seed"] is not None:
|
||||
used_seed = row["seed"]
|
||||
else:
|
||||
@@ -550,20 +204,6 @@ async def generate_speech(
|
||||
used_seed = row["seed"]
|
||||
if language == "Auto":
|
||||
language = None
|
||||
# #533: a profile's stored language must drive generation when the
|
||||
# request didn't pin one. Without this the German (etc.) archetype
|
||||
# generates with language=None and the model drifts to English —
|
||||
# even though the archetype PREVIEW renders correctly (archetypes.py
|
||||
# passes the language). An EXPLICIT non-Auto request language still
|
||||
# wins; we only fill the gap. `row` is a sqlite3.Row, so guard the
|
||||
# column lookup for pre-language DBs mid-upgrade.
|
||||
if language is None:
|
||||
try:
|
||||
prof_lang = row["language"]
|
||||
except (KeyError, IndexError):
|
||||
prof_lang = None
|
||||
if prof_lang and prof_lang != "Auto":
|
||||
language = prof_lang
|
||||
elif ref_audio is not None:
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".wav") as f:
|
||||
@@ -573,97 +213,16 @@ async def generate_speech(
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
# #308: a transcript-less reference is transcribed with the active ASR
|
||||
# backend (whisperx / faster-whisper / mlx-whisper) instead of the model's
|
||||
# built-in transformers pipeline, which cannot load whisper-large-v3-turbo
|
||||
# on transformers 5.3. On failure ref_text stays None and the model's
|
||||
# fallback behaves exactly as before.
|
||||
if ref_audio_path and not ref_text:
|
||||
from services.asr_backend import transcribe_reference
|
||||
# Same #730 hang risk as any whisperx transcribe — bound + reset the pool
|
||||
# so a wedged reference transcribe can't brick the backend. This path is
|
||||
# best-effort (transcribe_reference returns None on failure → the model's
|
||||
# built-in ASR fallback), so a timeout degrades to None rather than
|
||||
# failing the whole generate.
|
||||
try:
|
||||
ref_text = await run_on_gpu_pool_guarded(
|
||||
functools.partial(transcribe_reference, ref_audio_path),
|
||||
what="Reference transcribe",
|
||||
)
|
||||
except GpuJobTimeoutError as e:
|
||||
logger.warning("reference transcribe hung (%s); using model ASR fallback", e)
|
||||
ref_text = None
|
||||
|
||||
# #526: materialize a concrete seed when none was supplied (and no profile
|
||||
# pinned one) so the take is reproducible and we can hand it back via the
|
||||
# X-Seed header for the "keep this seed" control. An explicit request seed
|
||||
# or a profile's stored seed still wins — used_seed is only filled when it
|
||||
# is still None here, never overwritten.
|
||||
if used_seed is None:
|
||||
used_seed = random.randint(0, 2**31 - 1)
|
||||
|
||||
# Expressive-TTS Spec 01: apply the user pronunciation dictionary + inline
|
||||
# [[…]] one-off overrides to the text, here — AFTER `language` is fully
|
||||
# resolved (a profile may fill it above) so per-language entries match the
|
||||
# real render language, and BEFORE the text reaches either inference path
|
||||
# (native OmniVoice or a pluggable backend) and the chunk splitter. This is
|
||||
# the single point user text → normalized text → model, so the transform
|
||||
# covers generate for every engine. Pure text substitution → identical on
|
||||
# mac/Win/Linux. A disabled pref or empty dictionary is a pass-through, so
|
||||
# plain text stays byte-identical (#G5 backward-compat).
|
||||
from core import prefs as _prefs
|
||||
_pron_env = os.environ.get("OMNIVOICE_PRONUNCIATION")
|
||||
if _pron_env is not None:
|
||||
# Env wins (power-user override); "0"/"false"/"no"/"off" disable it.
|
||||
_pron_enabled = _pron_env.strip().lower() not in ("0", "false", "no", "off", "")
|
||||
else:
|
||||
_pron_enabled = bool(_prefs.get("pronunciation_enabled", True))
|
||||
if pronounce and _pron_enabled:
|
||||
from services.pronunciation import apply_pronunciation, load_entries_from_db
|
||||
try:
|
||||
_pron_rows = load_entries_from_db()
|
||||
except Exception: # noqa: BLE001 — table missing / DB locked → no-op
|
||||
_pron_rows = []
|
||||
text = apply_pronunciation(text, _pron_rows, language)
|
||||
else:
|
||||
# Even with the dictionary off, inline [[…]] overrides are an explicit,
|
||||
# in-text authoring choice → always honored (and never left as literal
|
||||
# double-bracket text the model would mispronounce).
|
||||
from services.pronunciation import apply_inline_overrides
|
||||
text = apply_inline_overrides(text)
|
||||
|
||||
start_time = time.time()
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
if _backend is not None:
|
||||
# Bounded + pool-reset on hang so a wedged generate can't starve the
|
||||
# GPU pool and brick the backend ("can't reach backend", #730 class).
|
||||
audio_tensor = await run_on_gpu_pool_guarded(
|
||||
functools.partial(
|
||||
_run_backend_inference,
|
||||
_backend, text, language, ref_audio_path, ref_text, instruct,
|
||||
duration, num_step, guidance_scale, speed, denoise,
|
||||
postprocess_output, used_seed, effect_preset,
|
||||
max_chunk_chars, crossfade_ms,
|
||||
),
|
||||
what="TTS generate",
|
||||
)
|
||||
# Read after generation: engines with lazy model loading report
|
||||
# their real rate only once weights are up.
|
||||
sample_rate = _backend.sample_rate
|
||||
else:
|
||||
audio_tensor = await run_on_gpu_pool_guarded(
|
||||
functools.partial(
|
||||
_run_inference,
|
||||
_model, text, language, ref_audio_path, ref_text, instruct, duration,
|
||||
num_step, guidance_scale, speed, t_shift, denoise,
|
||||
postprocess_output, layer_penalty_factor, position_temperature,
|
||||
class_temperature, used_seed, effect_preset,
|
||||
max_chunk_chars, crossfade_ms,
|
||||
),
|
||||
what="TTS generate",
|
||||
)
|
||||
sample_rate = _model.sampling_rate
|
||||
audio_tensor = await loop.run_in_executor(
|
||||
_gpu_pool, _run_inference,
|
||||
_model, text, language, ref_audio_path, ref_text, instruct, duration,
|
||||
num_step, guidance_scale, speed, t_shift, denoise,
|
||||
postprocess_output, layer_penalty_factor, position_temperature,
|
||||
class_temperature, used_seed, effect_preset,
|
||||
)
|
||||
# Invisible AudioSeal provenance watermark on the final audio. Embedding
|
||||
# was previously only wired into the dub pipeline (dub_generate.py), so
|
||||
# plain TTS came out unmarked despite the setting being on. embed_watermark
|
||||
@@ -672,44 +231,28 @@ async def generate_speech(
|
||||
# generation.
|
||||
from services.watermark import embed_watermark
|
||||
audio_tensor = await loop.run_in_executor(
|
||||
_gpu_pool, embed_watermark, audio_tensor, sample_rate
|
||||
_gpu_pool, embed_watermark, audio_tensor, _model.sampling_rate
|
||||
)
|
||||
gen_time = round(time.time() - start_time, 2)
|
||||
|
||||
audio_id = str(uuid.uuid4())[:8]
|
||||
audio_filename = f"{audio_id}.wav"
|
||||
audio_path = os.path.join(OUTPUTS_DIR, audio_filename)
|
||||
_safe_torchaudio_save(audio_path, audio_tensor, sample_rate)
|
||||
_safe_torchaudio_save(audio_path, audio_tensor, _model.sampling_rate)
|
||||
|
||||
audio_dur = round(audio_tensor.shape[-1] / sample_rate, 2)
|
||||
audio_dur = round(audio_tensor.shape[-1] / _model.sampling_rate, 2)
|
||||
|
||||
# #710: the clip is already generated and saved above. A history-write
|
||||
# failure — e.g. "no such table: generation_history" on a DB that missed
|
||||
# schema init — must NOT 500 the user's generation. Self-heal the schema
|
||||
# once and retry; if it still fails, log and return the audio anyway.
|
||||
def _write_history():
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO generation_history (id, text, mode, language, instruct, profile_id, audio_path, duration_seconds, generation_time, seed, created_at) VALUES (?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(audio_id, text[:200], history_mode or ("clone" if ref_audio_path else "design"),
|
||||
language or "Auto", instruct or "", resolved_profile_id,
|
||||
audio_filename, audio_dur, gen_time, used_seed, time.time())
|
||||
)
|
||||
try:
|
||||
_write_history()
|
||||
except sqlite3.OperationalError as e:
|
||||
logger.warning("generation history write failed (%s); healing schema + retrying", e)
|
||||
try:
|
||||
ensure_schema()
|
||||
_write_history()
|
||||
except Exception as e2:
|
||||
logger.warning("history write still failed after schema heal; returning audio anyway: %s", e2)
|
||||
except Exception as e:
|
||||
logger.warning("generation history write failed; returning audio anyway: %s", e)
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO generation_history (id, text, mode, language, instruct, profile_id, audio_path, duration_seconds, generation_time, seed, created_at) VALUES (?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(audio_id, text[:200], "clone" if ref_audio_path else "design",
|
||||
language or "Auto", instruct or "", resolved_profile_id,
|
||||
audio_filename, audio_dur, gen_time, used_seed, time.time())
|
||||
)
|
||||
event_bus.emit("generation_history", {"action": "created", "id": audio_id})
|
||||
|
||||
buffer = io.BytesIO()
|
||||
_safe_torchaudio_save(buffer, audio_tensor, sample_rate, format="wav")
|
||||
_safe_torchaudio_save(buffer, audio_tensor, _model.sampling_rate, format="wav")
|
||||
buffer.seek(0)
|
||||
wav_bytes = buffer.read()
|
||||
|
||||
@@ -718,35 +261,20 @@ async def generate_speech(
|
||||
for i in range(0, len(wav_bytes), chunk_size):
|
||||
yield wav_bytes[i:i + chunk_size]
|
||||
|
||||
_resp_headers = {
|
||||
"X-Audio-Id": audio_id,
|
||||
"X-Gen-Time": str(gen_time),
|
||||
"X-Audio-Path": audio_filename,
|
||||
"X-Seed": str(used_seed) if used_seed is not None else "",
|
||||
"X-Audio-Duration": str(audio_dur),
|
||||
"Content-Length": str(len(wav_bytes)),
|
||||
}
|
||||
# Routing notice (#21): cpu_fallback or accelerated-with-caveat only;
|
||||
# the WAV body is binary so the header channel is the carrier.
|
||||
if _routing_notice:
|
||||
from services.engine_routing import header_safe_reason
|
||||
_resp_headers["X-OmniVoice-Routing"] = _routing_notice[0]
|
||||
_hr = header_safe_reason(_routing_notice[1])
|
||||
if _hr:
|
||||
_resp_headers["X-OmniVoice-Routing-Reason"] = _hr
|
||||
return StreamingResponse(
|
||||
_stream_wav(),
|
||||
media_type="audio/wav",
|
||||
headers=_resp_headers,
|
||||
headers={
|
||||
"X-Audio-Id": audio_id,
|
||||
"X-Gen-Time": str(gen_time),
|
||||
"X-Audio-Path": audio_filename,
|
||||
"X-Seed": str(used_seed) if used_seed is not None else "",
|
||||
"X-Audio-Duration": str(audio_dur),
|
||||
"Content-Length": str(len(wav_bytes)),
|
||||
}
|
||||
)
|
||||
except HTTPException:
|
||||
raise
|
||||
except GpuJobTimeoutError as e:
|
||||
# A wedged GPU generate — the pool was already reset to restore capacity
|
||||
# (#730 class). Report the actionable timeout instead of the misleading
|
||||
# "can't reach backend" the frontend shows when the pool starves.
|
||||
logger.error("Generate timed out: %s", e)
|
||||
raise HTTPException(status_code=503, detail=str(e)) from e
|
||||
except ValueError as e:
|
||||
logger.error("Validation failed: %s", e)
|
||||
raise HTTPException(status_code=400, detail=str(e)) from e
|
||||
@@ -780,29 +308,9 @@ def _safe_output_path(name):
|
||||
|
||||
@router.get("/history")
|
||||
def list_history():
|
||||
"""Newest 50 generations whose audio still exists on disk.
|
||||
|
||||
Rows whose WAV was deleted out-of-band (cleared outputs dir, manual
|
||||
cleanup) used to come back anyway and render dead players that 404 on
|
||||
every fetch; prune them here so the UI never sees them again."""
|
||||
with db_conn() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM generation_history ORDER BY created_at DESC LIMIT 50"
|
||||
).fetchall()
|
||||
alive, stale_ids = [], []
|
||||
for r in rows:
|
||||
p = _safe_output_path(r["audio_path"]) if r["audio_path"] else None
|
||||
if r["audio_path"] and (not p or not os.path.exists(p)):
|
||||
stale_ids.append(r["id"])
|
||||
else:
|
||||
alive.append(dict(r))
|
||||
if stale_ids:
|
||||
conn.executemany(
|
||||
"DELETE FROM generation_history WHERE id=?",
|
||||
[(i,) for i in stale_ids],
|
||||
)
|
||||
logger.info("pruned %d stale history rows (audio file gone)", len(stale_ids))
|
||||
return alive
|
||||
rows = conn.execute("SELECT * FROM generation_history ORDER BY created_at DESC LIMIT 50").fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
@router.delete("/history")
|
||||
def clear_history():
|
||||
|
||||
@@ -1,158 +0,0 @@
|
||||
"""Longform Job Library (PR 7).
|
||||
|
||||
``GET /longform/jobs`` — list finished Audiobook + Story renders so the user can
|
||||
re-download them from the Projects view. The render itself (the m4b/mp3) already
|
||||
landed in ``OUTPUTS_DIR`` and is served at ``/audio/<output>``; here we just
|
||||
recover, from each finished job's persisted SSE tail, the output filename plus
|
||||
the chapter count and duration the ``done`` event carried.
|
||||
|
||||
Pure recovery, no synthesis. Defensive by construction: a job whose ``done``
|
||||
event is missing or unparseable is skipped, never surfaced and never a 500.
|
||||
|
||||
The work lives in :func:`build_longform_library`, a pure function over the
|
||||
job-store callables, so it's unit-testable without importing ``main`` (and the
|
||||
torch graph behind it).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Callable, Optional
|
||||
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
logger = logging.getLogger("omnivoice.longform_jobs")
|
||||
router = APIRouter()
|
||||
|
||||
#: Job types this library surfaces. Both flow through the shared longform
|
||||
#: renderer (``_render_longform_sse``) and emit the same ``done`` event shape.
|
||||
_LONGFORM_TYPES = ("audiobook", "story")
|
||||
|
||||
|
||||
def _done_payload_from_events(events: list[dict]) -> Optional[dict]:
|
||||
"""Recover the final ``{"type": "done", ...}`` payload from a job's SSE tail.
|
||||
|
||||
Each row's ``payload`` is the JSON the renderer stored via
|
||||
``job_store.append_event(job_id, json.dumps(payload))``. We scan newest-first
|
||||
and return the first parseable ``done`` event. Anything malformed is skipped
|
||||
— this never raises.
|
||||
"""
|
||||
for ev in reversed(events):
|
||||
raw = ev.get("payload") if isinstance(ev, dict) else None
|
||||
if not raw or not isinstance(raw, str):
|
||||
continue
|
||||
try:
|
||||
obj = json.loads(raw)
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
if isinstance(obj, dict) and obj.get("type") == "done":
|
||||
return obj
|
||||
return None
|
||||
|
||||
|
||||
def _coerce_int(value, default: int = 0) -> int:
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def _coerce_float(value, default: float = 0.0) -> float:
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def build_longform_library(
|
||||
list_jobs: Callable[..., list[dict]],
|
||||
events_since: Callable[..., list[dict]],
|
||||
*,
|
||||
limit: int = 50,
|
||||
) -> list[dict]:
|
||||
"""Build the newest-first list of finished longform renders.
|
||||
|
||||
Pure over the two job-store callables so tests can pass them directly:
|
||||
|
||||
* ``list_jobs(status="done", limit=...)`` → all done jobs, newest-first.
|
||||
* ``events_since(job_id)`` → that job's persisted SSE events.
|
||||
|
||||
Returns ``[{job_id, type, title?, output, duration_s, chapters,
|
||||
created_at}]``. Jobs that aren't a longform type, or whose ``done`` event /
|
||||
output filename can't be recovered, are silently skipped — the library only
|
||||
ever lists things the user can actually re-download.
|
||||
"""
|
||||
limit = max(1, min(_coerce_int(limit, 50), 500))
|
||||
try:
|
||||
# Over-fetch: non-longform done jobs (dub, etc.) get filtered out below,
|
||||
# so ask for more rows than the caller's limit to still fill the page.
|
||||
rows = list_jobs(status="done", limit=limit * 4)
|
||||
except Exception:
|
||||
logger.warning("longform library: list_jobs failed", exc_info=True)
|
||||
return []
|
||||
|
||||
out: list[dict] = []
|
||||
for row in rows or []:
|
||||
if len(out) >= limit:
|
||||
break
|
||||
try:
|
||||
job_type = row.get("type")
|
||||
job_id = row.get("id")
|
||||
if job_type not in _LONGFORM_TYPES or not job_id:
|
||||
continue
|
||||
try:
|
||||
events = events_since(job_id)
|
||||
except Exception:
|
||||
logger.warning("longform library: events_since failed for %s",
|
||||
job_id, exc_info=True)
|
||||
continue
|
||||
done = _done_payload_from_events(events or [])
|
||||
if not done:
|
||||
continue
|
||||
output = done.get("output")
|
||||
if not output or not isinstance(output, str):
|
||||
continue # nothing to re-download → not worth listing
|
||||
|
||||
item = {
|
||||
"job_id": job_id,
|
||||
"type": job_type,
|
||||
"output": output,
|
||||
"duration_s": round(_coerce_float(done.get("duration_s")), 2),
|
||||
"chapters": _coerce_int(done.get("chapters")),
|
||||
"created_at": row.get("created_at"),
|
||||
}
|
||||
# Title is optional — prefer the done event, fall back to job meta.
|
||||
title = done.get("title")
|
||||
if not title:
|
||||
meta_raw = row.get("meta_json")
|
||||
if isinstance(meta_raw, str) and meta_raw:
|
||||
try:
|
||||
meta = json.loads(meta_raw)
|
||||
if isinstance(meta, dict):
|
||||
title = meta.get("title")
|
||||
except (ValueError, TypeError):
|
||||
title = None
|
||||
if title:
|
||||
item["title"] = title
|
||||
out.append(item)
|
||||
except Exception:
|
||||
# Per-row isolation: one bad row never sinks the whole list.
|
||||
logger.warning("longform library: skipping unparseable job row",
|
||||
exc_info=True)
|
||||
continue
|
||||
return out
|
||||
|
||||
|
||||
@router.get("/longform/jobs")
|
||||
def longform_jobs(limit: int = Query(50, ge=1, le=500)) -> dict:
|
||||
"""Finished Audiobook + Story renders, newest-first, ready to re-download.
|
||||
|
||||
Each item's ``output`` is served at ``/audio/<output>``. Never 500s — on any
|
||||
backend hiccup it returns an empty list rather than an error.
|
||||
"""
|
||||
from core import job_store
|
||||
|
||||
jobs = build_longform_library(
|
||||
job_store.list_jobs, job_store.events_since, limit=limit,
|
||||
)
|
||||
return {"jobs": jobs}
|
||||
@@ -58,31 +58,6 @@ MAX_BUNDLE_BYTES = 100 * 1024 * 1024
|
||||
# ── Export ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _bundle_metadata(profile: dict, **extra) -> dict:
|
||||
"""Common .omnivoice metadata for export + publish.
|
||||
|
||||
Captures ``kind`` and ``vd_states`` so a *designed* persona survives the
|
||||
bundle round-trip as a design (not silently demoted to a clone) — required
|
||||
for the synthetic-only gating of the persona gallery (§R3). Old bundles
|
||||
without these keys import as ``kind='clone'`` (backward-compatible).
|
||||
"""
|
||||
meta = {
|
||||
"bundle_version": BUNDLE_VERSION,
|
||||
"profile_name": profile.get("name", "Unnamed"),
|
||||
"ref_text": profile.get("ref_text", ""),
|
||||
"instruct": profile.get("instruct", ""),
|
||||
"language": profile.get("language", "Auto"),
|
||||
"personality": profile.get("personality", ""),
|
||||
"seed": profile.get("seed"),
|
||||
"kind": profile.get("kind") or "clone",
|
||||
"vd_states": profile.get("vd_states"),
|
||||
"is_locked": bool(profile.get("is_locked")),
|
||||
"omnivoice_version": APP_VERSION,
|
||||
}
|
||||
meta.update(extra)
|
||||
return meta
|
||||
|
||||
|
||||
@router.post("/export/{profile_id}")
|
||||
def export_profile(profile_id: str):
|
||||
"""Export a voice profile as a downloadable .omnivoice bundle (ZIP)."""
|
||||
@@ -100,9 +75,19 @@ def export_profile(profile_id: str):
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
# Metadata
|
||||
metadata = _bundle_metadata(
|
||||
profile, created_at=profile.get("created_at"), exported_at=time.time(),
|
||||
)
|
||||
metadata = {
|
||||
"bundle_version": BUNDLE_VERSION,
|
||||
"profile_name": profile.get("name", "Unnamed"),
|
||||
"ref_text": profile.get("ref_text", ""),
|
||||
"instruct": profile.get("instruct", ""),
|
||||
"language": profile.get("language", "Auto"),
|
||||
"personality": profile.get("personality", ""),
|
||||
"seed": profile.get("seed"),
|
||||
"is_locked": bool(profile.get("is_locked")),
|
||||
"created_at": profile.get("created_at"),
|
||||
"exported_at": time.time(),
|
||||
"omnivoice_version": APP_VERSION,
|
||||
}
|
||||
zf.writestr("metadata.json", json.dumps(metadata, indent=2))
|
||||
|
||||
# Reference audio
|
||||
@@ -206,9 +191,8 @@ async def import_profile(
|
||||
conn.execute(
|
||||
"""INSERT INTO voice_profiles
|
||||
(id, name, ref_audio_path, ref_text, instruct, language,
|
||||
seed, personality, is_locked, locked_audio_path, created_at,
|
||||
kind, vd_states)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
seed, personality, is_locked, locked_audio_path, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(
|
||||
profile_id,
|
||||
metadata.get("profile_name", "Imported Voice"),
|
||||
@@ -221,10 +205,6 @@ async def import_profile(
|
||||
1 if is_locked else 0,
|
||||
locked_audio_filename or "",
|
||||
time.time(),
|
||||
# Preserve the design/clone distinction across the round-trip;
|
||||
# old bundles without these keys import as a clone.
|
||||
metadata.get("kind") or "clone",
|
||||
metadata.get("vd_states"),
|
||||
),
|
||||
)
|
||||
|
||||
@@ -273,11 +253,19 @@ def publish_to_marketplace(
|
||||
|
||||
# Build the bundle
|
||||
with zipfile.ZipFile(str(bundle_path), "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
metadata = _bundle_metadata(
|
||||
profile,
|
||||
tags=[t.strip() for t in tags.split(",") if t.strip()],
|
||||
published_at=time.time(),
|
||||
)
|
||||
metadata = {
|
||||
"bundle_version": BUNDLE_VERSION,
|
||||
"profile_name": profile.get("name", "Unnamed"),
|
||||
"ref_text": profile.get("ref_text", ""),
|
||||
"instruct": profile.get("instruct", ""),
|
||||
"language": profile.get("language", "Auto"),
|
||||
"personality": profile.get("personality", ""),
|
||||
"seed": profile.get("seed"),
|
||||
"is_locked": bool(profile.get("is_locked")),
|
||||
"tags": [t.strip() for t in tags.split(",") if t.strip()],
|
||||
"published_at": time.time(),
|
||||
"omnivoice_version": APP_VERSION,
|
||||
}
|
||||
zf.writestr("metadata.json", json.dumps(metadata, indent=2))
|
||||
|
||||
ref_path = profile.get("ref_audio_path")
|
||||
|
||||
@@ -1,53 +0,0 @@
|
||||
"""REST CRUD for per-agent MCP voice bindings (Wave 2.2 / Spec 2).
|
||||
|
||||
Loopback-gated — the Settings UI manages bindings here. The MCP tools
|
||||
themselves resolve voices via ``services.mcp_bindings.resolve_voice``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from api.dependencies import require_loopback
|
||||
from services import mcp_bindings
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/api/mcp",
|
||||
tags=["mcp"],
|
||||
dependencies=[Depends(require_loopback)],
|
||||
)
|
||||
|
||||
|
||||
class _BindingBody(BaseModel):
|
||||
client_id: str = Field(..., min_length=1, max_length=128)
|
||||
label: str | None = None
|
||||
profile_id: str | None = None
|
||||
default_engine: str | None = None
|
||||
|
||||
|
||||
@router.get("/bindings")
|
||||
def list_bindings():
|
||||
"""All per-agent voice bindings, most-recently-seen first."""
|
||||
return mcp_bindings.list_bindings()
|
||||
|
||||
|
||||
@router.put("/bindings")
|
||||
def upsert_binding(body: _BindingBody):
|
||||
"""Create or update the binding for an MCP client id."""
|
||||
try:
|
||||
return mcp_bindings.upsert_binding(
|
||||
body.client_id,
|
||||
label=body.label,
|
||||
profile_id=body.profile_id,
|
||||
default_engine=body.default_engine,
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
|
||||
@router.delete("/bindings/{client_id}")
|
||||
def delete_binding(client_id: str):
|
||||
if not mcp_bindings.delete_binding(client_id):
|
||||
raise HTTPException(status_code=404, detail="No binding for that client id")
|
||||
return {"deleted": client_id}
|
||||
@@ -22,6 +22,7 @@ from __future__ import annotations
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
import asyncio
|
||||
import tempfile
|
||||
from typing import Literal, Optional
|
||||
|
||||
@@ -29,7 +30,7 @@ from fastapi import APIRouter, File, Form, HTTPException, UploadFile
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from services.model_manager import _gpu_pool, run_on_gpu_pool_guarded
|
||||
from services.model_manager import _gpu_pool
|
||||
|
||||
logger = logging.getLogger("omnivoice.openai_compat")
|
||||
|
||||
@@ -81,33 +82,6 @@ class SpeechRequest(BaseModel):
|
||||
"E.g. 'young female, warm tone, slight British accent'.",
|
||||
)
|
||||
instruct: Optional[str] = Field(default=None, description="Style instruction for the TTS engine.")
|
||||
duration: Optional[float] = Field(
|
||||
default=None,
|
||||
gt=0,
|
||||
description="OmniVoice extension: target output duration in seconds.",
|
||||
)
|
||||
seed: Optional[int] = Field(
|
||||
default=None,
|
||||
description="OmniVoice extension: deterministic sampling seed.",
|
||||
)
|
||||
denoise: bool = Field(
|
||||
default=True,
|
||||
description="OmniVoice extension: prepend denoise control when supported.",
|
||||
)
|
||||
preprocess_prompt: bool = Field(
|
||||
default=True,
|
||||
description="OmniVoice extension: trim/preprocess reference prompt when supported.",
|
||||
)
|
||||
chunk_duration: Optional[float] = Field(
|
||||
default=None,
|
||||
ge=0,
|
||||
description="OmniVoice GGUF extension: long-form internal chunk duration.",
|
||||
)
|
||||
chunk_threshold: Optional[float] = Field(
|
||||
default=None,
|
||||
ge=0,
|
||||
description="OmniVoice GGUF extension: long-form internal chunk threshold.",
|
||||
)
|
||||
|
||||
|
||||
class TranscriptionResponse(BaseModel):
|
||||
@@ -235,14 +209,7 @@ def _run_tts(backend, text: str, kw: dict):
|
||||
from services.audio_dsp import apply_mastering, normalize_audio
|
||||
wav = backend.generate(text, **kw)
|
||||
sr = backend.sample_rate
|
||||
# Engines that already emit mastered, studio-grade audio (e.g. VoxCPM2's
|
||||
# native 48 kHz) opt out of apply_mastering via `applies_own_mastering`.
|
||||
# That chain's Compressor + 8% Reverb is tuned for OmniVoice's 24 kHz clone
|
||||
# output; applied to a studio engine it adds an audible level pump and a
|
||||
# reverb tail that degrade the very output we want clean. Loudness
|
||||
# normalisation still runs — it's a benign peak scale, not dynamics.
|
||||
if not getattr(backend, "applies_own_mastering", False):
|
||||
wav = apply_mastering(wav, sample_rate=sr)
|
||||
wav = apply_mastering(wav, sample_rate=sr)
|
||||
wav = normalize_audio(wav, target_dBFS=-2.0)
|
||||
return wav, sr
|
||||
|
||||
@@ -252,28 +219,10 @@ async def create_speech(req: SpeechRequest):
|
||||
"""Generate audio from text. Compatible with OpenAI's POST /v1/audio/speech."""
|
||||
backend = _resolve_engine(req.model)
|
||||
|
||||
# Routing gate (#21 — no silent CPU fallback), identical to REST /generate.
|
||||
from core.device_caps import detect_host_caps
|
||||
from services.engine_routing import resolve_routing, routing_notice
|
||||
_routing = resolve_routing(getattr(backend, "gpu_compat", ("cpu",)), detect_host_caps())
|
||||
if _routing["routing_status"] == "unavailable":
|
||||
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
|
||||
_routing_notice = routing_notice(_routing) # (status, reason) or None
|
||||
|
||||
# Build kwargs for the backend's generate() method
|
||||
kw: dict = {
|
||||
"speed": req.speed,
|
||||
"denoise": req.denoise,
|
||||
"preprocess_prompt": req.preprocess_prompt,
|
||||
}
|
||||
if req.duration is not None:
|
||||
kw["duration"] = req.duration
|
||||
if req.seed is not None:
|
||||
kw["seed"] = req.seed
|
||||
if req.chunk_duration is not None:
|
||||
kw["chunk_duration"] = req.chunk_duration
|
||||
if req.chunk_threshold is not None:
|
||||
kw["chunk_threshold"] = req.chunk_threshold
|
||||
if req.language:
|
||||
kw["language"] = req.language
|
||||
if req.instruct:
|
||||
@@ -302,8 +251,6 @@ async def create_speech(req: SpeechRequest):
|
||||
kw["ref_text"] = row["ref_text"]
|
||||
if row["instruct"] and not req.instruct:
|
||||
kw["instruct"] = row["instruct"]
|
||||
if req.seed is None and row["seed"] is not None:
|
||||
kw["seed"] = row["seed"]
|
||||
else:
|
||||
# Not a profile ID — forward as engine preset name
|
||||
kw["voice"] = voice
|
||||
@@ -312,30 +259,21 @@ async def create_speech(req: SpeechRequest):
|
||||
kw["voice"] = voice
|
||||
|
||||
try:
|
||||
# Bounded + pool-reset on hang so a wedged TTS request can't starve the
|
||||
# GPU pool and brick the backend (#730 class).
|
||||
wav, sr = await run_on_gpu_pool_guarded(
|
||||
lambda: _run_tts(backend, req.input, kw), what="OpenAI TTS generate")
|
||||
loop = asyncio.get_running_loop()
|
||||
wav, sr = await loop.run_in_executor(_gpu_pool, _run_tts, backend, req.input, kw)
|
||||
except Exception as e:
|
||||
logger.exception("OpenAI TTS failed: %s", e)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
audio_bytes, mime_type, ext = _encode_audio(wav, sr, req.response_format)
|
||||
|
||||
_headers = {
|
||||
"Content-Length": str(len(audio_bytes)),
|
||||
"Content-Disposition": f'inline; filename="speech.{ext}"',
|
||||
}
|
||||
if _routing_notice:
|
||||
from services.engine_routing import header_safe_reason
|
||||
_headers["X-OmniVoice-Routing"] = _routing_notice[0]
|
||||
_hr = header_safe_reason(_routing_notice[1])
|
||||
if _hr:
|
||||
_headers["X-OmniVoice-Routing-Reason"] = _hr
|
||||
return StreamingResponse(
|
||||
io.BytesIO(audio_bytes),
|
||||
media_type=mime_type,
|
||||
headers=_headers,
|
||||
headers={
|
||||
"Content-Length": str(len(audio_bytes)),
|
||||
"Content-Disposition": f'inline; filename="speech.{ext}"',
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@@ -385,15 +323,12 @@ async def create_transcription(
|
||||
try:
|
||||
backend = get_active_asr_backend()
|
||||
|
||||
# Run transcription in the thread pool to avoid blocking the event loop,
|
||||
# bounded so a stuck/starved ASR returns a 504 with guidance instead of
|
||||
# hanging the request forever (see run_transcribe_guarded).
|
||||
from services.asr_backend import run_transcribe_guarded
|
||||
# Run transcription in the thread pool to avoid blocking the event loop
|
||||
loop = asyncio.get_running_loop()
|
||||
word_ts = response_format == "verbose_json"
|
||||
result = await run_transcribe_guarded(
|
||||
result = await loop.run_in_executor(
|
||||
_gpu_pool,
|
||||
lambda: backend.transcribe(tmp_path, word_timestamps=word_ts),
|
||||
what="OpenAI",
|
||||
)
|
||||
|
||||
# Extract the full text from segments
|
||||
@@ -460,12 +395,6 @@ async def create_transcription(
|
||||
# Default: json
|
||||
return TranscriptionResponse(text=full_text)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except TimeoutError as e:
|
||||
# ASRTimeoutError (subclass): backend alive, ASR too heavy for compute.
|
||||
logger.warning("OpenAI transcription timed out: %s", e)
|
||||
raise HTTPException(status_code=504, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.exception("OpenAI transcription failed: %s", e)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
@@ -1,332 +0,0 @@
|
||||
"""HTTP layer for the `.ovsvoice` portable persona format (#29 / parity §R3 G1).
|
||||
|
||||
Thin router over `services.persona_bundle`:
|
||||
|
||||
POST /personas/export/{profile_id} → stream a downloadable .ovsvoice
|
||||
POST /personas/import → create a profile from a bundle
|
||||
POST /personas/inspect → read a bundle's manifest, no writes
|
||||
|
||||
Mirrors the legacy `.omnivoice` endpoints (`marketplace.py`) and reuses the
|
||||
same path-confinement (`_voices_path`) + consent floor. `.ovsvoice` is additive;
|
||||
`.omnivoice` import stays a compatible legacy reader.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import functools
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, File, HTTPException, Query, UploadFile
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from core import event_bus
|
||||
from core.config import VOICES_DIR # noqa: F401 — re-exported for tests/monkeypatch
|
||||
from core.db import db_conn
|
||||
from core.version import APP_VERSION
|
||||
from services import persona_bundle as pb
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.personas")
|
||||
|
||||
|
||||
def _safe_name(name: str, profile_id: str) -> str:
|
||||
"""Sanitised download filename stem (marketplace idiom); empty → persona_<id>."""
|
||||
cleaned = "".join(
|
||||
c if c.isalnum() or c in "-_ " else "" for c in (name or "")
|
||||
).strip().replace(" ", "_")[:40]
|
||||
return cleaned or f"persona_{profile_id}"
|
||||
|
||||
|
||||
# ── Export ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/personas/export/{profile_id}")
|
||||
async def export_persona(
|
||||
profile_id: str,
|
||||
license_spdx: str = Query(pb.DEFAULT_LICENSE),
|
||||
tags: str = Query(""),
|
||||
include_reference: bool = Query(True),
|
||||
):
|
||||
"""Build + stream a `.ovsvoice` bundle for a profile."""
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT * FROM voice_profiles WHERE id = ?", (profile_id,)
|
||||
).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Voice profile not found")
|
||||
profile = dict(row)
|
||||
|
||||
tag_list = [t.strip() for t in tags.split(",") if t.strip()]
|
||||
# #693: if OMNIVOICE_MODEL is set, record the *resolved* checkpoint in the
|
||||
# exported bundle so a leaked engine id (e.g. "omnivoice") can't be baked in;
|
||||
# keep "" when unset (the bundle's "engine unspecified" marker).
|
||||
from services.model_manager import resolve_omnivoice_checkpoint
|
||||
engine_id = resolve_omnivoice_checkpoint() if os.environ.get("OMNIVOICE_MODEL", "").strip() else ""
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
content = await loop.run_in_executor(
|
||||
None,
|
||||
functools.partial(
|
||||
pb.build_persona_bundle,
|
||||
profile,
|
||||
license_spdx=license_spdx,
|
||||
tags=tag_list,
|
||||
include_reference=include_reference,
|
||||
engine_id=engine_id,
|
||||
omnivoice_version=APP_VERSION,
|
||||
),
|
||||
)
|
||||
except pb.NoPreviewSource:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="This profile has no readable reference or locked audio to "
|
||||
"build a preview from — re-create or re-import it.",
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("persona export failed for %s", profile_id)
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="Could not build the persona bundle — see Settings → Logs.",
|
||||
)
|
||||
|
||||
filename = f"{_safe_name(profile.get('name'), profile_id)}.ovsvoice"
|
||||
from io import BytesIO
|
||||
return StreamingResponse(
|
||||
BytesIO(content),
|
||||
media_type="application/zip",
|
||||
headers={
|
||||
"Content-Disposition": f'attachment; filename="{filename}"',
|
||||
"Content-Length": str(len(content)),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# ── Import ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _voices_dest(filename: str) -> str:
|
||||
"""Resolve an output filename inside VOICES_DIR; 400 on escape (belt+braces —
|
||||
the name is always server-generated `{profile_id}…`)."""
|
||||
from api.routers.profiles import _voices_path
|
||||
path = _voices_path(filename)
|
||||
if path is None:
|
||||
raise HTTPException(status_code=400, detail="Invalid profile id")
|
||||
return path
|
||||
|
||||
|
||||
def _consent_verified(parsed: pb.ParsedPersona, consent_path: str | None) -> bool:
|
||||
"""B12-B16: trust verified-own-voice ONLY with a real recording (≥ floor) AND
|
||||
non-empty consent_text AND a consent.json present. The manifest flag alone
|
||||
can't forge it."""
|
||||
if not parsed.consent or not consent_path:
|
||||
return False
|
||||
if os.path.getsize(consent_path) < pb._MIN_CONSENT_AUDIO_BYTES:
|
||||
return False
|
||||
return bool((parsed.consent.get("consent_text") or "").strip())
|
||||
|
||||
|
||||
@router.post("/personas/import")
|
||||
async def import_persona(file: UploadFile = File(...)):
|
||||
"""Create a new voice profile from a `.ovsvoice` (or legacy `.omnivoice`) bundle."""
|
||||
name = (file.filename or "").lower()
|
||||
if not name.endswith(".ovsvoice") and not name.endswith(".omnivoice"):
|
||||
raise HTTPException(status_code=400, detail="File must be a .ovsvoice or .omnivoice bundle")
|
||||
|
||||
content = await file.read()
|
||||
try:
|
||||
parsed = pb.parse_persona_bundle(content)
|
||||
except pb.BundleError as e:
|
||||
raise HTTPException(status_code=e.status, detail=e.detail)
|
||||
|
||||
persona = parsed.manifest.get("persona") or {}
|
||||
written: list[str] = []
|
||||
|
||||
def _gen_id() -> str:
|
||||
return str(uuid.uuid4())[:8]
|
||||
|
||||
profile_id = _gen_id()
|
||||
try:
|
||||
# ── Audio members → server-named files (never the member name). ──
|
||||
ref_filename = None
|
||||
locked_filename = None
|
||||
if "ref_audio" in parsed.members:
|
||||
ref_filename = f"{profile_id}{parsed.member_ext('ref_audio')}"
|
||||
dest = _voices_dest(ref_filename)
|
||||
parsed.extract_member("ref_audio", dest); written.append(dest)
|
||||
if "locked_audio" in parsed.members:
|
||||
locked_filename = f"{profile_id}_locked{parsed.member_ext('locked_audio')}"
|
||||
dest = _voices_dest(locked_filename)
|
||||
parsed.extract_member("locked_audio", dest); written.append(dest)
|
||||
# Preview-only bundle (A12/B8): use the preview as the usable ref clip.
|
||||
if ref_filename is None and locked_filename is None and "preview" in parsed.members:
|
||||
ref_filename = f"{profile_id}{parsed.member_ext('preview')}"
|
||||
dest = _voices_dest(ref_filename)
|
||||
parsed.extract_member("preview", dest); written.append(dest)
|
||||
if ref_filename is None and locked_filename is None:
|
||||
raise HTTPException(status_code=400, detail="bundle has no usable audio")
|
||||
|
||||
# ── Consent recording (optional) ──
|
||||
consent_filename = None
|
||||
consent_path = None
|
||||
if "consent_audio" in parsed.members:
|
||||
consent_filename = f"{profile_id}_consent{parsed.member_ext('consent_audio')}"
|
||||
consent_path = _voices_dest(consent_filename)
|
||||
parsed.extract_member("consent_audio", consent_path); written.append(consent_path)
|
||||
|
||||
verified = _consent_verified(parsed, consent_path)
|
||||
consent_text = ((parsed.consent or {}).get("consent_text") or "").strip()
|
||||
recorded_at = None
|
||||
if verified:
|
||||
try:
|
||||
recorded_at = float(parsed.consent.get("recorded_at"))
|
||||
except (TypeError, ValueError):
|
||||
recorded_at = time.time()
|
||||
|
||||
is_locked = bool(persona.get("is_locked") and locked_filename)
|
||||
ref_for_db = ref_filename or locked_filename # at least one is set
|
||||
|
||||
def _insert(pid: str):
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"""INSERT INTO voice_profiles
|
||||
(id, name, ref_audio_path, ref_text, instruct, language,
|
||||
seed, personality, is_locked, locked_audio_path, created_at,
|
||||
kind, vd_states,
|
||||
verified_own_voice, consent_text, consent_audio_path, consent_recorded_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(
|
||||
pid,
|
||||
persona.get("name") or "Imported Voice",
|
||||
ref_for_db,
|
||||
persona.get("ref_text", ""),
|
||||
persona.get("instruct", ""),
|
||||
persona.get("language", "Auto"),
|
||||
persona.get("seed"),
|
||||
persona.get("personality", ""),
|
||||
1 if is_locked else 0,
|
||||
locked_filename or "",
|
||||
time.time(),
|
||||
persona.get("kind") or "clone",
|
||||
persona.get("vd_states"),
|
||||
1 if verified else 0,
|
||||
# Keep the attestation text so the user can re-attest locally,
|
||||
# even when imported unverified.
|
||||
consent_text,
|
||||
consent_filename if verified else "",
|
||||
recorded_at if verified else None,
|
||||
),
|
||||
)
|
||||
|
||||
import sqlite3
|
||||
try:
|
||||
_insert(profile_id)
|
||||
except sqlite3.IntegrityError:
|
||||
profile_id = _gen_id() # one retry on id collision (B20)
|
||||
# rename the on-disk files to the new id so they still match the row
|
||||
written = _rename_for_new_id(written, profile_id)
|
||||
ref_for_db = _retarget(ref_for_db, profile_id)
|
||||
locked_filename = _retarget(locked_filename, profile_id)
|
||||
consent_filename = _retarget(consent_filename, profile_id)
|
||||
_insert(profile_id)
|
||||
|
||||
except HTTPException:
|
||||
_cleanup(written)
|
||||
raise
|
||||
except Exception:
|
||||
_cleanup(written)
|
||||
logger.exception("persona import failed")
|
||||
raise HTTPException(status_code=500, detail="Import failed; no files were kept.")
|
||||
|
||||
event_bus.emit("profiles", {"action": "created", "id": profile_id})
|
||||
logger.info("Imported persona %r as %s (verified=%s)", persona.get("name"), profile_id, verified)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"profile_id": profile_id,
|
||||
"name": persona.get("name") or "Imported Voice",
|
||||
"kind": persona.get("kind") or "clone",
|
||||
"verified_own_voice": verified,
|
||||
"preview_only": parsed.preview_only,
|
||||
"license_spdx": parsed.license_spdx,
|
||||
"watermarked_preview": parsed.watermarked_preview,
|
||||
"source_bundle": file.filename,
|
||||
"schema_version_ahead": parsed.schema_version_ahead,
|
||||
}
|
||||
|
||||
|
||||
def _cleanup(paths: list[str]) -> None:
|
||||
for p in paths:
|
||||
try:
|
||||
if p and os.path.exists(p):
|
||||
os.remove(p)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _rename_for_new_id(written: list[str], new_id: str) -> list[str]:
|
||||
"""After an id-collision retry, rename each written file to carry the new id
|
||||
(filenames are `{old_id}…`; swap the leading 8-char stem)."""
|
||||
out = []
|
||||
for p in written:
|
||||
d, base = os.path.split(p)
|
||||
# base looks like {id}{ext} | {id}_locked{ext} | {id}_consent{ext}
|
||||
new_base = new_id + base[8:]
|
||||
new_path = os.path.join(d, new_base)
|
||||
try:
|
||||
os.replace(p, new_path)
|
||||
out.append(new_path)
|
||||
except OSError:
|
||||
out.append(p)
|
||||
return out
|
||||
|
||||
|
||||
def _retarget(filename: str | None, new_id: str) -> str | None:
|
||||
return new_id + filename[8:] if filename else filename
|
||||
|
||||
|
||||
# ── Inspect (no-write preview) ──────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/personas/inspect")
|
||||
async def inspect_persona(file: UploadFile = File(...)):
|
||||
"""Read a bundle's manifest + consent summary WITHOUT writing any file or row."""
|
||||
name = (file.filename or "").lower()
|
||||
if not name.endswith(".ovsvoice") and not name.endswith(".omnivoice"):
|
||||
raise HTTPException(status_code=400, detail="File must be a .ovsvoice or .omnivoice bundle")
|
||||
content = await file.read()
|
||||
try:
|
||||
parsed = pb.parse_persona_bundle(content)
|
||||
except pb.BundleError as e:
|
||||
raise HTTPException(status_code=e.status, detail=e.detail)
|
||||
|
||||
persona = parsed.manifest.get("persona") or {}
|
||||
consent_summary = None
|
||||
if parsed.consent:
|
||||
has_recording = "consent_audio" in parsed.members
|
||||
consent_summary = {
|
||||
"verified_claimed": bool(parsed.consent.get("verified_own_voice")),
|
||||
"method": parsed.consent.get("method", ""),
|
||||
"has_recording": has_recording,
|
||||
# would_verify mirrors import's gate, minus the byte-floor check
|
||||
# (inspect never extracts to measure size — advisory only).
|
||||
"would_verify": has_recording and bool((parsed.consent.get("consent_text") or "").strip()),
|
||||
}
|
||||
|
||||
return {
|
||||
"format": "omnivoice-legacy" if parsed.is_legacy else pb.OVSVOICE_FORMAT,
|
||||
"schema_version": parsed.manifest.get("schema_version", pb.OVSVOICE_SCHEMA_VERSION),
|
||||
"name": persona.get("name") or "Imported Voice",
|
||||
"kind": persona.get("kind") or "clone",
|
||||
"language": persona.get("language", "Auto"),
|
||||
"personality": persona.get("personality", ""),
|
||||
"is_locked": bool(persona.get("is_locked")),
|
||||
"license_spdx": parsed.license_spdx,
|
||||
"tags": parsed.manifest.get("tags") or [],
|
||||
"preview_only": parsed.preview_only,
|
||||
"watermarked_preview": parsed.watermarked_preview,
|
||||
"consent": consent_summary,
|
||||
"schema_version_ahead": parsed.schema_version_ahead,
|
||||
}
|
||||
+18
-288
@@ -1,5 +1,4 @@
|
||||
import os
|
||||
import re
|
||||
import uuid
|
||||
import time
|
||||
import shutil
|
||||
@@ -12,7 +11,6 @@ from core.db import db_conn
|
||||
from core.config import VOICES_DIR, OUTPUTS_DIR
|
||||
from core import event_bus
|
||||
from core.personalities import get_personalities
|
||||
from omnivoice.utils.voice_design import heal_design_instruct, sanitize_instruct
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -36,110 +34,29 @@ def list_profiles():
|
||||
rows = conn.execute("SELECT * FROM voice_profiles ORDER BY created_at DESC").fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
_DESIGN_SEED = 42 # deterministic sample render, same as archetype previews
|
||||
|
||||
|
||||
@router.post("/profiles")
|
||||
async def create_profile(
|
||||
name: str = Form(...),
|
||||
ref_audio: Optional[UploadFile] = File(None),
|
||||
ref_audio: UploadFile = File(...),
|
||||
ref_text: str = Form(""),
|
||||
instruct: str = Form(""),
|
||||
language: str = Form("Auto"),
|
||||
seed: Optional[int] = Form(None),
|
||||
personality: str = Form(""),
|
||||
kind: str = Form("clone"),
|
||||
vd_states: Optional[str] = Form(None),
|
||||
):
|
||||
"""Create a voice profile (spec: docs/specs/voice-studio-unification.md §5).
|
||||
|
||||
kind='clone' — requires `ref_audio` (the user's reference recording).
|
||||
kind='design' — requires `vd_states` (JSON of category picks); the server
|
||||
renders a deterministic sample WAV (seed 42, same path as
|
||||
archetype materialization) and stores it as the profile's
|
||||
reference so the voice identity is stable across runs.
|
||||
"""
|
||||
if kind not in ("clone", "design"):
|
||||
raise HTTPException(status_code=422, detail="kind must be 'clone' or 'design'")
|
||||
if kind == "clone" and ref_audio is None:
|
||||
raise HTTPException(status_code=422, detail="clone profiles require ref_audio")
|
||||
if kind == "design":
|
||||
if not (vd_states or "").strip():
|
||||
raise HTTPException(status_code=422, detail="design profiles require vd_states")
|
||||
import json as _json
|
||||
try:
|
||||
parsed = _json.loads(vd_states)
|
||||
if not isinstance(parsed, dict):
|
||||
raise ValueError("not an object")
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=422, detail="vd_states must be a JSON object")
|
||||
# An all-Auto design (every category left on "Auto") yields an empty
|
||||
# instruct — that's still a valid, saveable voice: synthesis falls back
|
||||
# to neutral instruct-only conditioning (see generation.py design path).
|
||||
# Don't gate save on a non-empty instruct.
|
||||
#
|
||||
# Defence-in-depth against the "[object Object]" / freeform-prose poison
|
||||
# (#550 #571 #594 #596): never persist an instruct the engine validator
|
||||
# would reject. Sanitize the submitted instruct and, if it's unusable,
|
||||
# rebuild the tags from vd_states — so the row is always generation-safe
|
||||
# regardless of which frontend build saved it.
|
||||
instruct = heal_design_instruct(instruct, parsed)
|
||||
|
||||
profile_id = str(uuid.uuid4())[:8]
|
||||
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
|
||||
audio_filename = f"{profile_id}{ext}"
|
||||
audio_path = os.path.join(VOICES_DIR, audio_filename)
|
||||
|
||||
if kind == "clone":
|
||||
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
|
||||
audio_filename = f"{profile_id}{ext}"
|
||||
audio_path = os.path.join(VOICES_DIR, audio_filename)
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(await ref_audio.read())
|
||||
used_seed = seed
|
||||
else:
|
||||
# Saving a design profile is a pure persistence operation — it must not
|
||||
# depend on a loaded TTS model (issue #476: on a fresh model-less Docker
|
||||
# image the render forced a full model load + inference that 503'd, so
|
||||
# the save failed). We try the deterministic identity sample opportunist-
|
||||
# ically through the one shared TTS path (archetypes' renderer, never a
|
||||
# second inference code path); if the engine isn't ready it's rendered
|
||||
# lazily on first preview/use. The row carries vd_states + instruct, so
|
||||
# the voice is fully usable without the sample (synthesis falls back to
|
||||
# instruct-only conditioning — see generation.py's design path).
|
||||
from pathlib import Path
|
||||
from api.routers.archetypes import _render_archetype_wav
|
||||
audio_filename = f"{profile_id}.wav"
|
||||
audio_path = os.path.join(VOICES_DIR, audio_filename)
|
||||
try:
|
||||
await _render_archetype_wav(
|
||||
{
|
||||
"language": language,
|
||||
"sample_script": ref_text, # optional custom sample line
|
||||
"instruct": instruct,
|
||||
},
|
||||
Path(audio_path),
|
||||
)
|
||||
except Exception:
|
||||
# Engine unavailable / OOM / inference failure — defer the sample.
|
||||
# Store the row with no ref_audio_path; the identity sample is
|
||||
# rendered on first preview or use. Never let this block the save.
|
||||
import logging
|
||||
logging.getLogger("omnivoice.profiles").info(
|
||||
"Design profile %s saved with sample pending — "
|
||||
"voice engine not ready; will render on first use", profile_id,
|
||||
)
|
||||
if os.path.exists(audio_path): # partial/blank render: don't keep it
|
||||
with __import__("contextlib").suppress(OSError):
|
||||
os.remove(audio_path)
|
||||
audio_filename = None
|
||||
used_seed = seed if seed is not None else _DESIGN_SEED
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(await ref_audio.read())
|
||||
|
||||
try:
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, "
|
||||
"language, seed, personality, kind, vd_states, created_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(profile_id, name, audio_filename, ref_text, instruct, language,
|
||||
used_seed, personality, kind, vd_states, time.time())
|
||||
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(profile_id, name, audio_filename, ref_text, instruct, language, seed, personality, time.time())
|
||||
)
|
||||
except Exception:
|
||||
# Clean up orphaned audio file if DB insert fails
|
||||
@@ -147,7 +64,7 @@ async def create_profile(
|
||||
os.remove(audio_path)
|
||||
raise
|
||||
event_bus.emit("profiles", {"action": "created", "id": profile_id})
|
||||
return {"id": profile_id, "name": name, "kind": kind}
|
||||
return {"id": profile_id, "name": name}
|
||||
|
||||
@router.get("/profiles/{profile_id}")
|
||||
def get_profile(profile_id: str):
|
||||
@@ -175,10 +92,6 @@ def update_profile(profile_id: str, patch: ProfileUpdate):
|
||||
continue
|
||||
if col == "name" and not val.strip():
|
||||
raise HTTPException(status_code=400, detail="A voice profile needs a name.")
|
||||
if col == "instruct":
|
||||
# Never let an edit persist a validator-rejecting instruct (prose /
|
||||
# "[object Object]"); keep only whitelist tags (#550 #571 #594 #596).
|
||||
val = sanitize_instruct(val)
|
||||
fields.append(f"{col} = ?")
|
||||
params.append(val.strip() if col in ("name", "language") else val)
|
||||
if not fields:
|
||||
@@ -250,96 +163,20 @@ def get_profile_usage(profile_id: str):
|
||||
}
|
||||
|
||||
|
||||
# profile_id is a request path param and the audio filename derives from it, so
|
||||
# constrain it to the generated-id charset (no separators / `..` possible) before
|
||||
# any path use, and read only a *direct child* of VOICES_DIR — os.path.basename()
|
||||
# strips any directory component (a path-injection / CWE-22 barrier).
|
||||
_PROFILE_ID_RE = re.compile(r"[A-Za-z0-9_-]{1,64}")
|
||||
|
||||
|
||||
@router.get("/profiles/{profile_id}/audio")
|
||||
async def get_profile_audio(profile_id: str):
|
||||
if not _PROFILE_ID_RE.fullmatch(profile_id or ""):
|
||||
return Response("Profile not found", status_code=404)
|
||||
def get_profile_audio(profile_id: str):
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT ref_audio_path, locked_audio_path, kind, instruct, language, ref_text "
|
||||
"FROM voice_profiles WHERE id=?",
|
||||
(profile_id,),
|
||||
).fetchone()
|
||||
row = conn.execute("SELECT ref_audio_path, locked_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
if not row:
|
||||
return Response("Profile not found", status_code=404)
|
||||
audio_file = row["locked_audio_path"] or row["ref_audio_path"]
|
||||
if not audio_file:
|
||||
# A design profile saved before the engine was ready (issue #476) has no
|
||||
# identity sample yet. Render it lazily now — the deterministic seed-42
|
||||
# sample is reproducible, so a deferred render matches a save-time one.
|
||||
rendered = await _materialize_design_sample(profile_id, row)
|
||||
if rendered is None:
|
||||
return Response("No audio available", status_code=404)
|
||||
audio_file = rendered
|
||||
# CWE-22: resolve the DB-stored filename strictly inside VOICES_DIR via the
|
||||
# shared guard — _voices_path() applies the os.path.basename() barrier plus
|
||||
# symlink-resolved containment (same path the consent endpoint trusts).
|
||||
audio_path = _voices_path(str(audio_file))
|
||||
if audio_path is None or not os.path.exists(audio_path):
|
||||
return Response("No audio available", status_code=404)
|
||||
audio_path = os.path.join(VOICES_DIR, audio_file)
|
||||
if not os.path.exists(audio_path):
|
||||
return Response("Audio file missing", status_code=404)
|
||||
return FileResponse(audio_path, media_type="audio/wav")
|
||||
|
||||
|
||||
async def _materialize_design_sample(profile_id: str, row) -> Optional[str]:
|
||||
"""Render a design profile's pending identity sample on first request.
|
||||
|
||||
Returns the stored filename on success, or None if this isn't a renderable
|
||||
design row. Raises HTTPException(503) with a precise "model not ready"
|
||||
message if the engine is genuinely unavailable — saving never depends on
|
||||
this, but a user who explicitly asks for the sample gets a clear signal.
|
||||
"""
|
||||
try:
|
||||
kind = row["kind"]
|
||||
except (KeyError, IndexError):
|
||||
kind = "clone"
|
||||
if kind != "design":
|
||||
return None
|
||||
|
||||
from pathlib import Path
|
||||
from api.routers.archetypes import _render_archetype_wav
|
||||
|
||||
audio_filename = f"{profile_id}.wav"
|
||||
# CWE-22: resolve under VOICES_DIR via the shared basename + containment
|
||||
# guard before rendering (rejects any escape).
|
||||
audio_path = _voices_path(audio_filename)
|
||||
if audio_path is None:
|
||||
raise HTTPException(status_code=400, detail="invalid profile identifier")
|
||||
try:
|
||||
await _render_archetype_wav(
|
||||
{
|
||||
"language": row["language"] or "Auto",
|
||||
"sample_script": row["ref_text"] or "",
|
||||
"instruct": row["instruct"] or "",
|
||||
},
|
||||
Path(audio_path),
|
||||
)
|
||||
except Exception as e:
|
||||
with __import__("contextlib").suppress(OSError):
|
||||
if os.path.exists(audio_path):
|
||||
os.remove(audio_path)
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail=(
|
||||
"The voice engine isn't ready yet, so this designed voice's "
|
||||
"preview sample can't be rendered. Finish setup / download a "
|
||||
f"model, then try again. ({e})"
|
||||
),
|
||||
)
|
||||
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"UPDATE voice_profiles SET ref_audio_path=? WHERE id=?",
|
||||
(audio_filename, profile_id),
|
||||
)
|
||||
return audio_filename
|
||||
|
||||
@router.post("/profiles/{profile_id}/lock")
|
||||
async def lock_profile(
|
||||
profile_id: str,
|
||||
@@ -397,122 +234,15 @@ async def unlock_profile(profile_id: str):
|
||||
event_bus.emit("profiles", {"action": "unlocked", "id": profile_id})
|
||||
return {"unlocked": True, "profile_id": profile_id}
|
||||
|
||||
# ── Consent lock (parity program Wave 0.2) ─────────────────────────────────
|
||||
#
|
||||
# A profile becomes "verified own voice" when its owner records themselves
|
||||
# reading a consent statement. The recording is provenance, not a voiceprint
|
||||
# check — agentic features and gallery sharing gate on the flag; plain local
|
||||
# synthesis never does. Spec: docs/competitive-analysis.md Action 22.
|
||||
|
||||
_MIN_CONSENT_AUDIO_BYTES = 1000 # same floor as the frontend recorder
|
||||
|
||||
# Upload filename extension whitelist — anything else falls back to .wav so a
|
||||
# crafted filename can never influence the on-disk path (py/path-injection).
|
||||
_CONSENT_EXT_RE = re.compile(r"^\.[A-Za-z0-9]{1,8}$")
|
||||
|
||||
|
||||
def _voices_path(filename: str) -> Optional[str]:
|
||||
"""Resolve a DB-stored audio filename strictly inside VOICES_DIR.
|
||||
|
||||
Rejects anything that isn't a bare filename or that escapes the voices
|
||||
directory after symlink resolution. Returns None instead of raising so
|
||||
cleanup paths can simply skip bad values.
|
||||
"""
|
||||
if not filename or os.path.basename(filename) != filename:
|
||||
return None
|
||||
root = os.path.realpath(VOICES_DIR)
|
||||
path = os.path.realpath(os.path.join(root, filename))
|
||||
if not path.startswith(root + os.sep):
|
||||
return None
|
||||
return path
|
||||
|
||||
|
||||
@router.post("/profiles/{profile_id}/consent")
|
||||
async def record_consent(
|
||||
profile_id: str,
|
||||
consent_audio: UploadFile = File(...),
|
||||
consent_text: str = Form(...),
|
||||
):
|
||||
if not consent_text.strip():
|
||||
raise HTTPException(status_code=422, detail="consent_text must not be empty")
|
||||
data = await consent_audio.read()
|
||||
if len(data) < _MIN_CONSENT_AUDIO_BYTES:
|
||||
raise HTTPException(status_code=422, detail="consent recording is too short")
|
||||
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT id, consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)
|
||||
).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Profile not found")
|
||||
|
||||
ext = os.path.splitext(consent_audio.filename or "")[1]
|
||||
if not _CONSENT_EXT_RE.match(ext):
|
||||
ext = ".wav"
|
||||
audio_filename = f"{profile_id}_consent{ext}"
|
||||
audio_path = _voices_path(audio_filename)
|
||||
if audio_path is None: # profile_id is server-generated; this is belt+braces
|
||||
raise HTTPException(status_code=400, detail="Invalid profile id")
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(data)
|
||||
|
||||
# A re-record may change the extension; drop the superseded file.
|
||||
old = row["consent_audio_path"]
|
||||
if old and old != audio_filename:
|
||||
old_path = _voices_path(old)
|
||||
if old_path and os.path.exists(old_path):
|
||||
os.remove(old_path)
|
||||
|
||||
recorded_at = time.time()
|
||||
try:
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"UPDATE voice_profiles SET verified_own_voice=1, consent_text=?, "
|
||||
"consent_audio_path=?, consent_recorded_at=? WHERE id=?",
|
||||
(consent_text.strip(), audio_filename, recorded_at, profile_id),
|
||||
)
|
||||
except Exception:
|
||||
if os.path.exists(audio_path):
|
||||
os.remove(audio_path)
|
||||
raise
|
||||
event_bus.emit("profiles", {"action": "consent_recorded", "id": profile_id})
|
||||
return {
|
||||
"id": profile_id,
|
||||
"verified_own_voice": True,
|
||||
"consent_recorded_at": recorded_at,
|
||||
}
|
||||
|
||||
|
||||
@router.delete("/profiles/{profile_id}/consent")
|
||||
def revoke_consent(profile_id: str):
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)
|
||||
).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Profile not found")
|
||||
conn.execute(
|
||||
"UPDATE voice_profiles SET verified_own_voice=0, consent_text='', "
|
||||
"consent_audio_path='', consent_recorded_at=NULL WHERE id=?",
|
||||
(profile_id,),
|
||||
)
|
||||
if row["consent_audio_path"]:
|
||||
path = _voices_path(row["consent_audio_path"])
|
||||
if path and os.path.exists(path):
|
||||
os.remove(path)
|
||||
event_bus.emit("profiles", {"action": "consent_revoked", "id": profile_id})
|
||||
return {"id": profile_id, "verified_own_voice": False}
|
||||
|
||||
|
||||
@router.delete("/profiles/{profile_id}")
|
||||
def delete_profile(profile_id: str):
|
||||
with db_conn() as conn:
|
||||
row = conn.execute("SELECT ref_audio_path, locked_audio_path, consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
row = conn.execute("SELECT ref_audio_path, locked_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
if row:
|
||||
for col in ["ref_audio_path", "locked_audio_path", "consent_audio_path"]:
|
||||
for col in ["ref_audio_path", "locked_audio_path"]:
|
||||
if row[col]:
|
||||
path = _voices_path(row[col])
|
||||
if path and os.path.exists(path):
|
||||
path = os.path.join(VOICES_DIR, row[col])
|
||||
if os.path.exists(path):
|
||||
os.remove(path)
|
||||
# Prevent FOREIGN KEY constraint failure
|
||||
conn.execute("UPDATE generation_history SET profile_id = NULL WHERE profile_id=?", (profile_id,))
|
||||
|
||||
@@ -2,7 +2,6 @@ import uuid
|
||||
import time
|
||||
import json
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from core.db import db_conn
|
||||
from core import event_bus
|
||||
@@ -10,10 +9,6 @@ from schemas.requests import ProjectSaveRequest
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
class ProjectRenameRequest(BaseModel):
|
||||
name: str
|
||||
|
||||
@router.get("/projects")
|
||||
async def list_projects():
|
||||
with db_conn() as conn:
|
||||
@@ -64,26 +59,6 @@ async def update_project(project_id: str, req: ProjectSaveRequest):
|
||||
event_bus.emit("projects", {"action": "updated", "id": project_id})
|
||||
return {"id": project_id, "name": req.name, "updated_at": now}
|
||||
|
||||
@router.patch("/projects/{project_id}")
|
||||
async def rename_project(project_id: str, req: ProjectRenameRequest):
|
||||
"""Lightweight rename — updates only the project name (and updated_at),
|
||||
without re-serialising the whole state blob like PUT does."""
|
||||
name = req.name.strip()
|
||||
if not name:
|
||||
raise HTTPException(status_code=400, detail="Project name cannot be empty")
|
||||
now = time.time()
|
||||
with db_conn() as conn:
|
||||
row = conn.execute("SELECT id FROM studio_projects WHERE id=?", (project_id,)).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Project not found")
|
||||
conn.execute(
|
||||
"UPDATE studio_projects SET name=?, updated_at=? WHERE id=?",
|
||||
(name, now, project_id),
|
||||
)
|
||||
event_bus.emit("projects", {"action": "renamed", "id": project_id})
|
||||
return {"id": project_id, "name": name, "updated_at": now}
|
||||
|
||||
|
||||
@router.delete("/projects/{project_id}")
|
||||
async def delete_project(project_id: str):
|
||||
with db_conn() as conn:
|
||||
|
||||
@@ -1,306 +0,0 @@
|
||||
"""
|
||||
Pronunciation dictionary router — Expressive-TTS Spec 01 Phase 1.
|
||||
|
||||
CRUD for the DB-backed, per-language pronunciation dictionary the
|
||||
``PronunciationPanel`` (Settings → Pronunciation) edits, plus a model-free
|
||||
``/pronunciation/test`` dry-run. Entries are applied as pure text substitution
|
||||
before synthesis (see ``services/pronunciation.apply_pronunciation`` and the
|
||||
generate path), so a saved entry actually changes the audio on every engine.
|
||||
|
||||
Endpoints (loopback-only, like the dictation router):
|
||||
GET /pronunciation → list every entry
|
||||
POST /pronunciation → create one entry
|
||||
PUT /pronunciation/{entry_id} → update an entry (partial)
|
||||
DELETE /pronunciation/{entry_id} → remove an entry
|
||||
POST /pronunciation/test → dry-run substitution (no model)
|
||||
GET /pronunciation/export → all entries as JSON (round-trips import)
|
||||
POST /pronunciation/import → bulk add entries from JSON
|
||||
|
||||
Scope: ``language='*'`` is global (applies to every request); a 2-letter code
|
||||
(``'en'``, ``'de'``) applies only when the request language matches.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
import uuid
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from api.dependencies import require_loopback
|
||||
from core.db import db_conn
|
||||
from services.pronunciation import apply_pronunciation, entries_for_language
|
||||
|
||||
logger = logging.getLogger("omnivoice.pronunciation")
|
||||
router = APIRouter()
|
||||
|
||||
_VALID_TYPES = ("respelling", "ipa", "cmu")
|
||||
_ALL_LANG = "*"
|
||||
|
||||
# IPA: the input is validated as a non-empty string of Unicode letters / IPA
|
||||
# extension codepoints + the usual suprasegmental marks; we reject ASCII control
|
||||
# and the bracket/pipe chars that would collide with the inline grammar. This is
|
||||
# a charset gate (catches obvious garbage early), not a full IPA grammar.
|
||||
_IPA_BAD = re.compile(r"[\[\]\|\x00-\x1f]")
|
||||
# CMU / ARPABET: space-separated phoneme tokens (letters + an optional 0-2 stress
|
||||
# digit), e.g. "N AH0 V AE1 D AH0". Reject anything else.
|
||||
_CMU_TOKEN = re.compile(r"^[A-Za-z]{1,3}[0-2]?$")
|
||||
|
||||
|
||||
def _validate_type_replacement(etype: str, replacement: str) -> None:
|
||||
"""Raise 400 on a phoneme replacement that's obviously malformed.
|
||||
|
||||
Respelling rows accept any text. IPA rows must be a non-empty string free of
|
||||
bracket/pipe/control chars. CMU rows must be space-separated ARPABET tokens.
|
||||
Validating on save (not at synth) means a model never sees garbage phonemes
|
||||
(Spec 01 §R3 — never pass unvalidated phoneme strings to a model).
|
||||
"""
|
||||
if etype == "respelling":
|
||||
return
|
||||
rep = (replacement or "").strip()
|
||||
if not rep:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"A {etype.upper()} entry needs a phoneme string in 'replacement'.",
|
||||
)
|
||||
if etype == "ipa":
|
||||
if _IPA_BAD.search(rep):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="That IPA string contains brackets, a pipe, or control characters. "
|
||||
"Use plain IPA symbols, e.g. ˈnɛvʌdə.",
|
||||
)
|
||||
elif etype == "cmu":
|
||||
tokens = rep.split()
|
||||
if not tokens or any(not _CMU_TOKEN.match(tok) for tok in tokens):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="That doesn't look like CMU/ARPABET. Use space-separated tokens with "
|
||||
"optional stress digits, e.g. N AH0 V AE1 D AH0.",
|
||||
)
|
||||
|
||||
|
||||
def _norm_language(language: Optional[str]) -> str:
|
||||
"""Normalize a scope to '*' (global) or a lowercase 2-letter code."""
|
||||
if not language:
|
||||
return _ALL_LANG
|
||||
s = str(language).strip()
|
||||
if not s or s == _ALL_LANG or s.lower() == "auto":
|
||||
return _ALL_LANG
|
||||
return s.lower()[:2]
|
||||
|
||||
|
||||
def _row_to_dict(r) -> dict:
|
||||
d = dict(r)
|
||||
d["enabled"] = bool(d.get("enabled"))
|
||||
# ``scope`` is the UI-facing alias for ``language`` ('*' shows as Global).
|
||||
d["scope"] = d.get("language") or _ALL_LANG
|
||||
return d
|
||||
|
||||
|
||||
# ── Schemas ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class PronEntry(BaseModel):
|
||||
term: str
|
||||
replacement: str = ""
|
||||
type: str = "respelling"
|
||||
language: str = _ALL_LANG
|
||||
enabled: bool = True
|
||||
|
||||
|
||||
class PronEntryUpdate(BaseModel):
|
||||
term: Optional[str] = None
|
||||
replacement: Optional[str] = None
|
||||
type: Optional[str] = None
|
||||
language: Optional[str] = None
|
||||
enabled: Optional[bool] = None
|
||||
|
||||
|
||||
class PronTestRequest(BaseModel):
|
||||
text: str
|
||||
language: Optional[str] = None
|
||||
|
||||
|
||||
class PronImportRequest(BaseModel):
|
||||
entries: List[PronEntry]
|
||||
replace: bool = False # True → clear existing rows first
|
||||
|
||||
|
||||
# ── CRUD ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/pronunciation", dependencies=[Depends(require_loopback)])
|
||||
def list_entries():
|
||||
with db_conn() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT id, term, replacement, type, language, enabled, created_at "
|
||||
"FROM pronunciation_entries ORDER BY created_at ASC, id ASC"
|
||||
).fetchall()
|
||||
return [_row_to_dict(r) for r in rows]
|
||||
|
||||
|
||||
@router.post("/pronunciation", dependencies=[Depends(require_loopback)])
|
||||
def create_entry(entry: PronEntry):
|
||||
term = entry.term.strip()
|
||||
if not term:
|
||||
raise HTTPException(status_code=400, detail="A pronunciation entry needs a term.")
|
||||
etype = (entry.type or "respelling").strip().lower()
|
||||
if etype not in _VALID_TYPES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Unknown entry type {entry.type!r}. Use one of: {', '.join(_VALID_TYPES)}.",
|
||||
)
|
||||
_validate_type_replacement(etype, entry.replacement)
|
||||
eid = str(uuid.uuid4())[:12]
|
||||
now = time.time()
|
||||
lang = _norm_language(entry.language)
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO pronunciation_entries (id, term, replacement, type, language, enabled, created_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?)",
|
||||
(eid, term, entry.replacement, etype, lang, 1 if entry.enabled else 0, now),
|
||||
)
|
||||
row = conn.execute(
|
||||
"SELECT id, term, replacement, type, language, enabled, created_at "
|
||||
"FROM pronunciation_entries WHERE id = ?", (eid,)
|
||||
).fetchone()
|
||||
return _row_to_dict(row)
|
||||
|
||||
|
||||
@router.put("/pronunciation/{entry_id}", dependencies=[Depends(require_loopback)])
|
||||
def update_entry(entry_id: str, patch: PronEntryUpdate):
|
||||
with db_conn() as conn:
|
||||
existing = conn.execute(
|
||||
"SELECT id, term, replacement, type, language, enabled, created_at "
|
||||
"FROM pronunciation_entries WHERE id = ?", (entry_id,)
|
||||
).fetchone()
|
||||
if existing is None:
|
||||
raise HTTPException(status_code=404, detail="No such pronunciation entry.")
|
||||
|
||||
# Resolve the post-update type + replacement so phoneme validation runs
|
||||
# against the final state (e.g. switching type without changing text).
|
||||
new_type = (patch.type.strip().lower() if patch.type is not None else existing["type"]) or "respelling"
|
||||
if new_type not in _VALID_TYPES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Unknown entry type {patch.type!r}. Use one of: {', '.join(_VALID_TYPES)}.",
|
||||
)
|
||||
new_replacement = patch.replacement if patch.replacement is not None else existing["replacement"]
|
||||
_validate_type_replacement(new_type, new_replacement)
|
||||
|
||||
fields, params = [], []
|
||||
if patch.term is not None:
|
||||
term = patch.term.strip()
|
||||
if not term:
|
||||
raise HTTPException(status_code=400, detail="A pronunciation entry needs a term.")
|
||||
fields.append("term = ?"); params.append(term)
|
||||
if patch.replacement is not None:
|
||||
fields.append("replacement = ?"); params.append(patch.replacement)
|
||||
if patch.type is not None:
|
||||
fields.append("type = ?"); params.append(new_type)
|
||||
if patch.language is not None:
|
||||
fields.append("language = ?"); params.append(_norm_language(patch.language))
|
||||
if patch.enabled is not None:
|
||||
fields.append("enabled = ?"); params.append(1 if patch.enabled else 0)
|
||||
if not fields:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="PUT body was empty. Include at least one field to change, or DELETE the entry.",
|
||||
)
|
||||
params.append(entry_id)
|
||||
# nosec B608 - `fields` are fixed literal assignments ("term = ?", …) from
|
||||
# the allowlist above; every user value is a bound `?` parameter, never
|
||||
# interpolated. The f-string only joins constant column fragments.
|
||||
conn.execute(
|
||||
f"UPDATE pronunciation_entries SET {', '.join(fields)} WHERE id = ?", # nosec B608
|
||||
params,
|
||||
)
|
||||
row = conn.execute(
|
||||
"SELECT id, term, replacement, type, language, enabled, created_at "
|
||||
"FROM pronunciation_entries WHERE id = ?", (entry_id,)
|
||||
).fetchone()
|
||||
return _row_to_dict(row)
|
||||
|
||||
|
||||
@router.delete("/pronunciation/{entry_id}", dependencies=[Depends(require_loopback)])
|
||||
def delete_entry(entry_id: str):
|
||||
with db_conn() as conn:
|
||||
cur = conn.execute("DELETE FROM pronunciation_entries WHERE id = ?", (entry_id,))
|
||||
return {"deleted": cur.rowcount > 0}
|
||||
|
||||
|
||||
# ── Dry-run + import/export ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/pronunciation/test", dependencies=[Depends(require_loopback)])
|
||||
def test_substitution(req: PronTestRequest):
|
||||
"""Show the post-substitution text for ``req.text`` — no model call.
|
||||
|
||||
Applies the same dictionary + inline ``[[…]]`` resolution the synth path
|
||||
runs, so the user sees exactly what the engine will be handed.
|
||||
"""
|
||||
with db_conn() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT id, term, replacement, type, language, enabled, created_at "
|
||||
"FROM pronunciation_entries"
|
||||
).fetchall()
|
||||
substituted = apply_pronunciation(req.text, rows, req.language)
|
||||
applied = entries_for_language(rows, req.language)
|
||||
return {
|
||||
"input": req.text,
|
||||
"substituted": substituted,
|
||||
"changed": substituted != req.text,
|
||||
"applied_terms": sorted(applied.keys(), key=len, reverse=True),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/pronunciation/export", dependencies=[Depends(require_loopback)])
|
||||
def export_entries():
|
||||
"""Every entry as a JSON-serializable list (round-trips ``/import``)."""
|
||||
with db_conn() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT term, replacement, type, language, enabled "
|
||||
"FROM pronunciation_entries ORDER BY created_at ASC, id ASC"
|
||||
).fetchall()
|
||||
return {"entries": [
|
||||
{"term": r["term"], "replacement": r["replacement"], "type": r["type"],
|
||||
"language": r["language"], "enabled": bool(r["enabled"])}
|
||||
for r in rows
|
||||
]}
|
||||
|
||||
|
||||
@router.post("/pronunciation/import", dependencies=[Depends(require_loopback)])
|
||||
def import_entries(req: PronImportRequest):
|
||||
"""Bulk-add entries. ``replace=true`` clears the table first.
|
||||
|
||||
Each entry is validated like ``POST /pronunciation``; one bad row fails the
|
||||
whole import (400) so the table is never left half-applied.
|
||||
"""
|
||||
now = time.time()
|
||||
cleaned = []
|
||||
for e in req.entries:
|
||||
term = e.term.strip()
|
||||
if not term:
|
||||
continue # silently skip blank terms — they're a no-op anyway
|
||||
etype = (e.type or "respelling").strip().lower()
|
||||
if etype not in _VALID_TYPES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Entry {term!r}: unknown type {e.type!r}.",
|
||||
)
|
||||
_validate_type_replacement(etype, e.replacement)
|
||||
cleaned.append((str(uuid.uuid4())[:12], term, e.replacement, etype,
|
||||
_norm_language(e.language), 1 if e.enabled else 0, now))
|
||||
with db_conn() as conn:
|
||||
if req.replace:
|
||||
conn.execute("DELETE FROM pronunciation_entries")
|
||||
conn.executemany(
|
||||
"INSERT INTO pronunciation_entries (id, term, replacement, type, language, enabled, created_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?)",
|
||||
cleaned,
|
||||
)
|
||||
return {"imported": len(cleaned), "replaced": req.replace}
|
||||
@@ -123,208 +123,6 @@ def set_torch_compile_disabled(body: _TorchCompileBody):
|
||||
return _torch_compile_state()
|
||||
|
||||
|
||||
# ── Dictation refinement (parity program Wave 2.1 / Spec 3 phase 2) ───────
|
||||
|
||||
|
||||
class _RefinementBody(BaseModel):
|
||||
auto: bool | None = None
|
||||
smart_cleanup: bool | None = None
|
||||
self_correction: bool | None = None
|
||||
preserve_technical: bool | None = None
|
||||
|
||||
|
||||
def _refinement_state():
|
||||
from services.refinement import get_refinement_config
|
||||
from services.llm_backend import get_active_llm_backend
|
||||
|
||||
cfg = get_refinement_config()
|
||||
# The UI shows whether refinement can actually run (needs an LLM).
|
||||
cfg["llm_ready"] = get_active_llm_backend().id != "off"
|
||||
return cfg
|
||||
|
||||
|
||||
@router.get("/dictation-refinement")
|
||||
def get_dictation_refinement():
|
||||
"""Current refinement config + whether an LLM backend is configured."""
|
||||
return _refinement_state()
|
||||
|
||||
|
||||
@router.put("/dictation-refinement")
|
||||
def set_dictation_refinement(body: _RefinementBody):
|
||||
from services.refinement import set_refinement_config
|
||||
|
||||
try:
|
||||
set_refinement_config({k: v for k, v in body.model_dump().items() if v is not None})
|
||||
except Exception:
|
||||
logger.exception("set_dictation_refinement failed")
|
||||
raise HTTPException(status_code=500, detail="Failed to persist setting")
|
||||
return _refinement_state()
|
||||
|
||||
|
||||
# ── LLM endpoint (parity program Wave 2.4 / §R2 rung 4) ───────────────────
|
||||
# Focused configuration for the OpenAI-compatible LLM endpoint that powers
|
||||
# cinematic translate, glossary auto-extract, and dictation refinement.
|
||||
# Persistence rides the existing TRANSLATE_BASE_URL / TRANSLATE_API_KEY /
|
||||
# TRANSLATE_MODEL env vars (already in system.py PERSISTENT_KEYS, restored
|
||||
# at startup) so the resolution path in llm_backend/translator is unchanged.
|
||||
|
||||
|
||||
class _LLMEndpointBody(BaseModel):
|
||||
base_url: str | None = None
|
||||
model: str | None = None
|
||||
api_key: str | None = None # None = leave unchanged; "" = clear
|
||||
|
||||
|
||||
def _mask(secret: str | None) -> str | None:
|
||||
if not secret:
|
||||
return None
|
||||
return f"…{secret[-4:]}" if len(secret) > 4 else "set"
|
||||
|
||||
|
||||
def _llm_endpoint_state():
|
||||
from services.llm_backend import OpenAICompatBackend
|
||||
|
||||
ok, reason = OpenAICompatBackend.is_available()
|
||||
return {
|
||||
"base_url": os.environ.get("TRANSLATE_BASE_URL", ""),
|
||||
"model": os.environ.get("TRANSLATE_MODEL", ""),
|
||||
"api_key_masked": _mask(
|
||||
os.environ.get("TRANSLATE_API_KEY") or os.environ.get("OPENAI_API_KEY")
|
||||
),
|
||||
"available": ok,
|
||||
"reason": None if ok else reason,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/llm-endpoint")
|
||||
def get_llm_endpoint():
|
||||
"""Current OpenAI-compatible LLM endpoint config + live availability."""
|
||||
return _llm_endpoint_state()
|
||||
|
||||
|
||||
@router.put("/llm-endpoint")
|
||||
def set_llm_endpoint(body: _LLMEndpointBody):
|
||||
"""Persist base URL / model / API key for the OpenAI-compatible endpoint.
|
||||
|
||||
Reuses the env-var persistence path (prefs.json, restored at startup):
|
||||
base_url -> TRANSLATE_BASE_URL, model -> TRANSLATE_MODEL,
|
||||
api_key -> TRANSLATE_API_KEY. A None field is left unchanged; an empty
|
||||
string clears it. Ollama ignores the key; vLLM / LM Studio require it.
|
||||
"""
|
||||
from core.prefs import set_ as prefs_set, delete as prefs_delete
|
||||
|
||||
mapping = {
|
||||
"TRANSLATE_BASE_URL": body.base_url,
|
||||
"TRANSLATE_MODEL": body.model,
|
||||
"TRANSLATE_API_KEY": body.api_key,
|
||||
}
|
||||
for env_key, val in mapping.items():
|
||||
if val is None:
|
||||
continue # untouched
|
||||
val = val.strip()
|
||||
if val:
|
||||
os.environ[env_key] = val
|
||||
prefs_set(f"env.{env_key}", val)
|
||||
else:
|
||||
os.environ.pop(env_key, None)
|
||||
prefs_delete(f"env.{env_key}")
|
||||
# get_active_llm_backend() builds a fresh backend (and its OpenAI client
|
||||
# reads env at construction) on every call, so there's no singleton to
|
||||
# invalidate — the next translate/refine picks up the new values.
|
||||
return _llm_endpoint_state()
|
||||
|
||||
|
||||
# ── Multi-provider LLM registry (Settings → LLM Providers) ────────────────
|
||||
# Keys persist ENCRYPTED via settings_store.set_secret (never .env, never
|
||||
# returned). base_url/model/account overrides are non-secret. Loopback-gated
|
||||
# by the router dep, so LAN peers can't read masks or write keys.
|
||||
|
||||
class _LLMProviderBody(BaseModel):
|
||||
api_key: str | None = Field(None, description="API key; '' clears it, None leaves unchanged")
|
||||
base_url: str | None = None
|
||||
model: str | None = None
|
||||
account_id: str | None = Field(None, description="Cloudflare account id")
|
||||
make_active: bool = False
|
||||
|
||||
|
||||
class _LLMActiveBody(BaseModel):
|
||||
provider: str = Field(..., description="provider id to activate")
|
||||
|
||||
|
||||
@router.get("/llm-providers")
|
||||
def list_llm_providers():
|
||||
"""All providers with resolved base_url/model + whether a key is configured.
|
||||
|
||||
Never returns key material — only `has_key`/`key_from_env` booleans.
|
||||
"""
|
||||
from services import llm_providers
|
||||
return {
|
||||
"active": llm_providers.active_provider_id(),
|
||||
"providers": [llm_providers.describe(p) for p in llm_providers.all_providers()],
|
||||
}
|
||||
|
||||
|
||||
@router.put("/llm-providers/{provider_id}")
|
||||
def save_llm_provider(provider_id: str, body: _LLMProviderBody):
|
||||
"""Save a provider's key (encrypted) + optional base_url/model/account.
|
||||
|
||||
A None field is left unchanged; an empty api_key clears the stored key.
|
||||
"""
|
||||
from services import llm_providers
|
||||
if llm_providers.get_provider(provider_id) is None:
|
||||
raise HTTPException(status_code=404, detail=f"unknown provider {provider_id!r}")
|
||||
if body.api_key is not None:
|
||||
llm_providers.save_key(provider_id, body.api_key.strip())
|
||||
llm_providers.save_overrides(
|
||||
provider_id, base_url=body.base_url, model=body.model,
|
||||
account_id=body.account_id,
|
||||
)
|
||||
if body.make_active:
|
||||
llm_providers.set_active_provider(provider_id)
|
||||
return list_llm_providers()
|
||||
|
||||
|
||||
@router.post("/llm-providers/active")
|
||||
def set_active_llm_provider(body: _LLMActiveBody):
|
||||
from services import llm_providers
|
||||
if llm_providers.get_provider(body.provider) is None:
|
||||
raise HTTPException(status_code=404, detail=f"unknown provider {body.provider!r}")
|
||||
llm_providers.set_active_provider(body.provider)
|
||||
return list_llm_providers()
|
||||
|
||||
|
||||
@router.post("/llm-providers/{provider_id}/test")
|
||||
def test_llm_provider(provider_id: str):
|
||||
"""One cheap round-trip against a provider to prove the key/URL work.
|
||||
|
||||
Temporarily activates the provider for the probe by resolving its config
|
||||
directly (does not change the persisted active selection).
|
||||
"""
|
||||
from services import llm_providers
|
||||
p = llm_providers.get_provider(provider_id)
|
||||
if p is None:
|
||||
raise HTTPException(status_code=404, detail=f"unknown provider {provider_id!r}")
|
||||
base_url = llm_providers.resolve_base_url(p)
|
||||
api_key = llm_providers.resolve_api_key(p)
|
||||
if not base_url:
|
||||
return {"ok": False, "detail": "No Base URL set for this provider."}
|
||||
if not api_key:
|
||||
return {"ok": False, "detail": "No API key configured for this provider."}
|
||||
try:
|
||||
from openai import OpenAI
|
||||
client = OpenAI(api_key=api_key, base_url=base_url)
|
||||
res = client.chat.completions.create(
|
||||
model=llm_providers.resolve_model(p),
|
||||
messages=[{"role": "user", "content": "Reply with the single word: ok"}],
|
||||
timeout=20,
|
||||
)
|
||||
reply = (res.choices[0].message.content or "").strip()
|
||||
return {"ok": True, "model": llm_providers.resolve_model(p), "reply": reply[:80]}
|
||||
except Exception as e: # noqa: BLE001 — surface a clean, scrubbed error to the UI
|
||||
from core.scrub import scrub_text
|
||||
return {"ok": False, "detail": scrub_text(f"{type(e).__name__}: {e}")}
|
||||
|
||||
|
||||
# ── License acceptance (Phase 3 Plan 03-01 / TTS-05) ──────────────────────
|
||||
# Frontend ``SupertonicLicenseDialog`` flips the engine-license bit via this
|
||||
# endpoint. The handler is loopback-gated (router-level dep) and the
|
||||
@@ -486,60 +284,3 @@ def set_models_dir(body: _ModelsDirBody):
|
||||
|
||||
user_env.set_user_env(_MODELS_DIR_ENV, path)
|
||||
return {"configured": path, "effective": _effective_models_dir(), "restart_required": True}
|
||||
|
||||
|
||||
# ── HF mirror endpoint (parity program Wave 4.3 / §R4 c) ──────────────────
|
||||
# Restricted-network users (e.g. behind the Great Firewall) need to point
|
||||
# huggingface_hub at a mirror. HF reads HF_ENDPOINT at import time, so a
|
||||
# change takes effect on the next backend start — persisted to the durable
|
||||
# per-user env so it survives Tauri/Finder launches that don't inherit a
|
||||
# shell. Loopback-gated via the router dep.
|
||||
|
||||
_HF_ENDPOINT_ENV = "HF_ENDPOINT"
|
||||
|
||||
# A few well-known mirrors, surfaced as quick-picks in the UI. hf-mirror.com
|
||||
# is the community mirror most-used in China; the official endpoint clears it.
|
||||
_HF_MIRROR_PRESETS = [
|
||||
{"label": "Hugging Face (official)", "url": ""},
|
||||
{"label": "hf-mirror.com (community, China)", "url": "https://hf-mirror.com"},
|
||||
]
|
||||
|
||||
|
||||
class _HFMirrorBody(BaseModel):
|
||||
url: str = Field("", description="HF_ENDPOINT URL; empty string clears it (official endpoint)")
|
||||
|
||||
|
||||
@router.get("/hf-mirror")
|
||||
def get_hf_mirror():
|
||||
from core import user_env
|
||||
|
||||
configured = user_env.get_user_env(_HF_ENDPOINT_ENV) or ""
|
||||
return {
|
||||
# The value that will apply after restart (persisted), and what's
|
||||
# live in this process (env may differ until then).
|
||||
"configured": configured,
|
||||
"effective": os.environ.get(_HF_ENDPOINT_ENV, ""),
|
||||
"presets": _HF_MIRROR_PRESETS,
|
||||
}
|
||||
|
||||
|
||||
@router.put("/hf-mirror")
|
||||
def set_hf_mirror(body: _HFMirrorBody):
|
||||
from core import user_env
|
||||
|
||||
url = (body.url or "").strip().rstrip("/")
|
||||
if url and not url.startswith(("http://", "https://")):
|
||||
raise HTTPException(status_code=400, detail="Mirror URL must start with http(s)://")
|
||||
try:
|
||||
if url:
|
||||
user_env.set_user_env(_HF_ENDPOINT_ENV, url)
|
||||
os.environ[_HF_ENDPOINT_ENV] = url # best-effort for new downloads this session
|
||||
else:
|
||||
user_env.unset_user_env(_HF_ENDPOINT_ENV)
|
||||
os.environ.pop(_HF_ENDPOINT_ENV, None)
|
||||
except Exception:
|
||||
logger.exception("set_hf_mirror failed")
|
||||
raise HTTPException(status_code=500, detail="Failed to persist mirror setting")
|
||||
# HF endpoint is read at import time by huggingface_hub, so the override
|
||||
# is only guaranteed once the backend restarts.
|
||||
return {"configured": url, "restart_required": True, "presets": _HF_MIRROR_PRESETS}
|
||||
|
||||
@@ -11,27 +11,14 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from core import prefs
|
||||
from utils import hf_progress
|
||||
from utils import download_aggregator
|
||||
# Weight-floor scan (MM2-07 / #352) lives in ``models.py`` — the lowest module in
|
||||
# the setup import graph — so install-time validation here, the first-run
|
||||
# install-state detector (#622), and load-time repair share one set of floors and
|
||||
# can't drift apart. ``_MIN_WEIGHT_BYTES``/``_WEIGHT_FLOORS`` re-exported for tests.
|
||||
from .models import ( # noqa: F401
|
||||
KNOWN_MODELS,
|
||||
invalidate_cache,
|
||||
snapshot_has_weights,
|
||||
_MIN_WEIGHT_BYTES,
|
||||
_WEIGHT_FLOORS,
|
||||
)
|
||||
from .models import KNOWN_MODELS, invalidate_cache
|
||||
|
||||
logger = logging.getLogger("omnivoice.setup.download")
|
||||
router = APIRouter()
|
||||
@@ -39,181 +26,6 @@ router = APIRouter()
|
||||
# Cooldown: prevent rapid re-install after a failure. Maps repo_id → last_fail_time.
|
||||
_install_cooldowns: dict[str, float] = {}
|
||||
_COOLDOWN_SECS = 60.0
|
||||
# Evict cooldown entries older than this so the dict can't grow unbounded across
|
||||
# a long-lived process (MM2-06). Anything past the cooldown window is dead state.
|
||||
_COOLDOWN_TTL_SECS = 3600.0
|
||||
|
||||
|
||||
def _sweep_cooldowns(now: float) -> None:
|
||||
"""Drop cooldown entries older than the TTL (MM2-06). Keeps the dict bounded
|
||||
— without this it accumulated one entry per ever-failed repo forever."""
|
||||
stale = [k for k, t in _install_cooldowns.items() if (now - t) > _COOLDOWN_TTL_SECS]
|
||||
for k in stale:
|
||||
_install_cooldowns.pop(k, None)
|
||||
|
||||
# Repo_ids the user asked to cancel (FDL-11). Checked between retry attempts.
|
||||
# Note: a single in-flight snapshot_download/Xet fetch is not interruptible
|
||||
# mid-file in hf_hub 1.7.2 — cancel stops further retries, marks the row
|
||||
# cancelled, and clears the cooldown so a cancel isn't rate-limited.
|
||||
_cancelled: set[str] = set()
|
||||
|
||||
|
||||
def _download_max_workers() -> int:
|
||||
"""Parallel-FILES worker count for snapshot_download (FDL-02). Default 8 —
|
||||
don't crank it: Xet already parallelises *within* each file via concurrent
|
||||
byte-range gets, so a high count just multiplies buffer pressure. Override
|
||||
via prefs / OMNIVOICE_DOWNLOAD_MAX_WORKERS for power users."""
|
||||
raw = prefs.resolve("download_max_workers", env="OMNIVOICE_DOWNLOAD_MAX_WORKERS", default=8)
|
||||
try:
|
||||
return max(1, int(raw))
|
||||
except (TypeError, ValueError):
|
||||
return 8
|
||||
|
||||
|
||||
def _download_endpoint() -> "str | None":
|
||||
"""Optional HF endpoint override (FDL-10 mirror path, opt-in). Returned as a
|
||||
per-call ``endpoint=`` rather than a process-wide HF_ENDPOINT mutation. A
|
||||
mirror routes through the classic LFS path (no Xet) — documented in
|
||||
docs/downloading-models.md."""
|
||||
ep = prefs.resolve("hf_endpoint", env="HF_ENDPOINT", default=None)
|
||||
return ep or None
|
||||
|
||||
|
||||
def apply_xet_env() -> None:
|
||||
"""Apply opt-in Xet tuning knobs to the environment before a download
|
||||
(FDL-04). Both default OFF; env wins over the prefs store. high-performance
|
||||
can *hurt* low-RAM machines (needs lots of RAM/bandwidth); HDD-sequential
|
||||
avoids parallel-write thrash on spinning disks. Idempotent."""
|
||||
import os as _os
|
||||
high_perf = prefs.resolve("xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False)
|
||||
if _truthy(high_perf):
|
||||
_os.environ["HF_XET_HIGH_PERFORMANCE"] = "1"
|
||||
hdd_seq = prefs.resolve("xet_hdd_sequential_write", env="HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY", default=False)
|
||||
if _truthy(hdd_seq):
|
||||
_os.environ["HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY"] = "1"
|
||||
|
||||
|
||||
def _truthy(v) -> bool:
|
||||
if isinstance(v, bool):
|
||||
return v
|
||||
return str(v).strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
class _InstallCancelled(Exception):
|
||||
"""Raised inside the install worker when the user cancels (FDL-11)."""
|
||||
|
||||
|
||||
def compute_plan(plan_files) -> dict:
|
||||
"""Summarise a snapshot_download(dry_run=True) result into the install_plan
|
||||
payload (FDL-05): total bytes, bytes already cached (skipped), bytes that
|
||||
will actually download, and file counts. ``will_download`` defaults to
|
||||
``not is_cached`` for forward-compat with older DryRunFileInfo shapes."""
|
||||
total = sum(int(getattr(f, "file_size", 0) or 0) for f in plan_files)
|
||||
cached = sum(
|
||||
int(getattr(f, "file_size", 0) or 0)
|
||||
for f in plan_files if getattr(f, "is_cached", False)
|
||||
)
|
||||
will = [
|
||||
f for f in plan_files
|
||||
if getattr(f, "will_download", not getattr(f, "is_cached", False))
|
||||
]
|
||||
to_dl = sum(int(getattr(f, "file_size", 0) or 0) for f in will)
|
||||
n_files = len(plan_files)
|
||||
n_cached = sum(1 for f in plan_files if getattr(f, "is_cached", False))
|
||||
return {
|
||||
"total_bytes": total,
|
||||
"cached_bytes": cached,
|
||||
"to_download_bytes": to_dl,
|
||||
"n_files": n_files,
|
||||
"n_cached": n_cached,
|
||||
}
|
||||
|
||||
|
||||
def _segmented_enabled() -> bool:
|
||||
"""IDM-style multi-connection accelerator (FDL-09), default **ON**. The app
|
||||
forces the legacy-LFS path (HF_HUB_DISABLE_XET=1) for clear progress, but that
|
||||
path is single-stream and slow — this restores parallel byte-range speed AND
|
||||
real live progress, and falls back to snapshot_download on any error so it
|
||||
can never compromise a correct install. Default-on so first-run downloads are
|
||||
fast out of the box (pairs with an HF token for higher rate limits); set
|
||||
OMNIVOICE_SEGMENTED_DOWNLOAD=0 to force the single-stream path."""
|
||||
return _truthy(prefs.resolve(
|
||||
"segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=True,
|
||||
))
|
||||
|
||||
|
||||
def _xet_active() -> bool:
|
||||
"""True only when hf_xet is installed AND not disabled. The app sets
|
||||
HF_HUB_DISABLE_XET=1 by default, so this is normally False — which is when
|
||||
the segmented accelerator pays off."""
|
||||
import importlib.util
|
||||
if importlib.util.find_spec("hf_xet") is None:
|
||||
return False
|
||||
return os.environ.get("HF_HUB_DISABLE_XET", "").strip().lower() not in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _repo_cancelled(repo_id: str) -> bool:
|
||||
return repo_id in _cancelled
|
||||
|
||||
|
||||
def _segmented_snapshot(repo_id: str, *, endpoint: "str | None") -> str:
|
||||
"""Fetch every file of a repo via the segmented downloader into the HF
|
||||
cache, mirroring hf_hub_download's blob+snapshot+refs layout so the result
|
||||
is indistinguishable from snapshot_download (FDL-09) — keeping /models
|
||||
install-state, is_cached, and delete working. Feeds real bytes to the
|
||||
aggregator. Raises on any error; the caller falls back to snapshot_download.
|
||||
"""
|
||||
import asyncio as _asyncio
|
||||
from huggingface_hub import HfApi, constants as _C
|
||||
from huggingface_hub.file_download import (
|
||||
hf_hub_url, get_hf_file_metadata, repo_folder_name, _create_symlink,
|
||||
)
|
||||
from services.segmented_download import segmented_download
|
||||
from services.token_resolver import resolve as _resolve_token
|
||||
|
||||
token = _resolve_token()
|
||||
api = HfApi(endpoint=endpoint, token=token)
|
||||
info = api.repo_info(repo_id, repo_type="model")
|
||||
commit = info.sha
|
||||
files = [s.rfilename for s in (info.siblings or [])]
|
||||
if not commit or not files:
|
||||
raise RuntimeError("repo_info returned no commit/siblings")
|
||||
|
||||
repo_dir = os.path.join(_C.HF_HUB_CACHE, repo_folder_name(repo_id=repo_id, repo_type="model"))
|
||||
blobs_dir = os.path.join(repo_dir, "blobs")
|
||||
snap_dir = os.path.join(repo_dir, "snapshots", commit)
|
||||
refs_dir = os.path.join(repo_dir, "refs")
|
||||
for d in (blobs_dir, snap_dir, refs_dir):
|
||||
os.makedirs(d, exist_ok=True)
|
||||
|
||||
for rel in files:
|
||||
if _repo_cancelled(repo_id):
|
||||
raise _InstallCancelled()
|
||||
url = hf_hub_url(repo_id, rel, endpoint=endpoint, revision=commit)
|
||||
meta = get_hf_file_metadata(url, token=token)
|
||||
etag = (meta.etag or "").strip('"')
|
||||
if not etag:
|
||||
raise RuntimeError(f"no etag for {rel}")
|
||||
blob_path = os.path.join(blobs_dir, etag)
|
||||
pointer = os.path.join(snap_dir, rel)
|
||||
os.makedirs(os.path.dirname(pointer), exist_ok=True)
|
||||
if not os.path.exists(blob_path):
|
||||
_asyncio.run(segmented_download(
|
||||
meta.location or url, blob_path,
|
||||
token=token, expected_size=meta.size, expected_etag=etag,
|
||||
on_bytes=lambda d, k=rel: download_aggregator.add_bytes(repo_id, k, d),
|
||||
cancel_check=lambda: _repo_cancelled(repo_id),
|
||||
))
|
||||
if not os.path.lexists(pointer):
|
||||
_create_symlink(blob_path, pointer, new_blob=True)
|
||||
|
||||
# refs/main → commit so scan_cache_dir maps the revision correctly.
|
||||
try:
|
||||
with open(os.path.join(refs_dir, "main"), "w") as f:
|
||||
f.write(commit)
|
||||
except OSError:
|
||||
pass
|
||||
return snap_dir
|
||||
|
||||
|
||||
# ── SSE Download Stream ───────────────────────────────────────────────────
|
||||
@@ -230,41 +42,6 @@ def _safe_put(queue: asyncio.Queue, event) -> None:
|
||||
pass
|
||||
|
||||
|
||||
# Minimum size for "this snapshot actually contains model weights". An
|
||||
# interrupted snapshot_download can leave config/tokenizer files but no
|
||||
# weights; the install then looks complete and synthesis later fails with
|
||||
# "does not appear to have a file named pytorch_model.bin or
|
||||
# model.safetensors" (#352). 5 MB clears every weight format we ship
|
||||
# (safetensors/bin shards, onnx, pt, gguf) without false-positiving on
|
||||
# config-only aux repos.
|
||||
def _validate_snapshot_has_weights(repo_id: str, snapshot_path: str) -> None:
|
||||
"""Raise OSError when a finished snapshot has no plausible weight file —
|
||||
surfaces the truncated-download class (#352) at install time, where the
|
||||
retry loop and the UI's re-download path can deal with it, instead of at
|
||||
first synthesis with an opaque transformers error.
|
||||
|
||||
Delegates the weight check to ``models.snapshot_has_weights`` (single source of
|
||||
the floors); only the install-time error message lives here."""
|
||||
if snapshot_has_weights(snapshot_path):
|
||||
return
|
||||
biggest = 0
|
||||
try:
|
||||
for root, _dirs, files in os.walk(snapshot_path, followlinks=True):
|
||||
for f in files:
|
||||
try:
|
||||
biggest = max(biggest, os.path.getsize(os.path.join(root, f)))
|
||||
except OSError:
|
||||
continue
|
||||
except OSError:
|
||||
pass
|
||||
raise OSError(
|
||||
f"{repo_id}: download finished but no model weights were found in the "
|
||||
"snapshot (largest file "
|
||||
f"{biggest} bytes). The download was likely interrupted — delete the "
|
||||
"model in Settings → Models and install it again."
|
||||
)
|
||||
|
||||
|
||||
@router.get("/setup/download-stream")
|
||||
async def setup_download_stream():
|
||||
"""SSE: forward every HuggingFace download tqdm update as a JSON event."""
|
||||
@@ -321,7 +98,6 @@ async def install_model(req: InstallModelRequest):
|
||||
)
|
||||
# Cooldown guard — don't retry if the same model just failed.
|
||||
import time as _time_check
|
||||
_sweep_cooldowns(_time_check.time()) # bound the dict (MM2-06)
|
||||
last_fail = _install_cooldowns.get(req.repo_id)
|
||||
if last_fail and (_time_check.time() - last_fail) < _COOLDOWN_SECS:
|
||||
remaining = int(_COOLDOWN_SECS - (_time_check.time() - last_fail))
|
||||
@@ -336,7 +112,6 @@ async def install_model(req: InstallModelRequest):
|
||||
|
||||
def _do():
|
||||
token = hf_progress.current_repo_id.set(req.repo_id)
|
||||
_cancelled.discard(req.repo_id) # clear any stale cancel from a prior run
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
@@ -350,22 +125,7 @@ async def install_model(req: InstallModelRequest):
|
||||
LocalEntryNotFoundError,
|
||||
)
|
||||
logger.info("model install starting: %s", req.repo_id)
|
||||
# Apply opt-in Xet tuning knobs (high-perf / HDD) before downloading.
|
||||
apply_xet_env()
|
||||
# Drive snapshot_download explicitly (FDL-02): pass our progress-
|
||||
# emitting tqdm subclass so progress is deterministic + Xet-aware
|
||||
# (Xet feeds bytes into whatever tqdm_class is supplied), bound the
|
||||
# parallel-files worker count, and honour an optional mirror endpoint.
|
||||
dl_kwargs: dict = {
|
||||
"repo_id": req.repo_id,
|
||||
"max_workers": _download_max_workers(),
|
||||
}
|
||||
_tqdm_cls = hf_progress.tracked_tqdm_class()
|
||||
if _tqdm_cls is not None:
|
||||
dl_kwargs["tqdm_class"] = _tqdm_cls
|
||||
_endpoint = _download_endpoint()
|
||||
if _endpoint:
|
||||
dl_kwargs["endpoint"] = _endpoint
|
||||
dl_kwargs: dict = {"repo_id": req.repo_id}
|
||||
if sys.platform == "win32":
|
||||
dl_kwargs["local_dir_use_symlinks"] = False
|
||||
|
||||
@@ -393,71 +153,12 @@ async def install_model(req: InstallModelRequest):
|
||||
hb = threading.Thread(target=_heartbeat, daemon=True)
|
||||
hb.start()
|
||||
|
||||
# Pre-flight (FDL-05): a dry-run resolve gives the UI an accurate
|
||||
# denominator — total bytes, bytes already cached (skipped), and the
|
||||
# bytes that will actually download — BEFORE any byte flows. Seeds
|
||||
# the overall aggregator so its bar/ETA are correct from the first
|
||||
# event. Degrades gracefully (totals=None) on older/gated repos.
|
||||
_preflight_kwargs = {"repo_id": req.repo_id, "dry_run": True}
|
||||
if _endpoint:
|
||||
_preflight_kwargs["endpoint"] = _endpoint
|
||||
try:
|
||||
_plan = snapshot_download(**_preflight_kwargs)
|
||||
_summary = compute_plan(_plan)
|
||||
download_aggregator.start(
|
||||
req.repo_id,
|
||||
total_bytes=_summary["to_download_bytes"],
|
||||
files_total=max(0, _summary["n_files"] - _summary["n_cached"]),
|
||||
)
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
"phase": "install_plan",
|
||||
**_summary,
|
||||
})
|
||||
except Exception as _pf_err:
|
||||
# No preflight (older/gated repo, mirror without dry-run, etc.):
|
||||
# fall back to today's fill-in-as-files-appear behaviour.
|
||||
logger.info("model install %s: preflight unavailable (%s)", req.repo_id, _pf_err)
|
||||
download_aggregator.start(req.repo_id)
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
"phase": "install_plan",
|
||||
"total_bytes": None,
|
||||
"cached_bytes": None,
|
||||
"to_download_bytes": None,
|
||||
"n_files": None,
|
||||
"n_cached": None,
|
||||
})
|
||||
|
||||
_max_attempts = 5
|
||||
_attempt = 0
|
||||
while True:
|
||||
if req.repo_id in _cancelled:
|
||||
raise _InstallCancelled()
|
||||
_attempt += 1
|
||||
try:
|
||||
# Segmented accelerator (FDL-09, default ON): parallel
|
||||
# byte-range fetch with real live progress, for the
|
||||
# legacy-LFS path. Any failure falls through to
|
||||
# snapshot_download — the accelerator can never compromise a
|
||||
# correct install.
|
||||
_snapshot_path = None
|
||||
if _attempt == 1 and _segmented_enabled() and not _xet_active():
|
||||
try:
|
||||
_snapshot_path = _segmented_snapshot(req.repo_id, endpoint=_endpoint)
|
||||
except _InstallCancelled:
|
||||
raise
|
||||
except Exception as _seg_err:
|
||||
logger.info(
|
||||
"segmented download for %s failed (%s); falling back to snapshot_download",
|
||||
req.repo_id, _seg_err,
|
||||
)
|
||||
_snapshot_path = None
|
||||
if _snapshot_path is None:
|
||||
_snapshot_path = snapshot_download(**dl_kwargs)
|
||||
_validate_snapshot_has_weights(req.repo_id, _snapshot_path)
|
||||
snapshot_download(**dl_kwargs)
|
||||
break
|
||||
except (HfHubHTTPError, LocalEntryNotFoundError, OSError) as net_err:
|
||||
if _attempt >= _max_attempts:
|
||||
@@ -478,10 +179,6 @@ async def install_model(req: InstallModelRequest):
|
||||
_t.sleep(_backoff)
|
||||
# Stop heartbeat once download completes
|
||||
_resolving.set()
|
||||
# Flush the overall bar to 100% with the true byte total (FDL-06):
|
||||
# under Xet the per-file byte bars don't surface completion, so the
|
||||
# aggregator can sit below 100% even though every file landed.
|
||||
download_aggregator.complete(req.repo_id)
|
||||
logger.info("model install done: %s", req.repo_id)
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
@@ -489,19 +186,7 @@ async def install_model(req: InstallModelRequest):
|
||||
"downloaded": 0, "total": 0, "pct": 1.0,
|
||||
"phase": "install_done",
|
||||
})
|
||||
_install_cooldowns.pop(req.repo_id, None) # success clears any cooldown (MM2-06)
|
||||
invalidate_cache()
|
||||
except _InstallCancelled:
|
||||
_resolving.set()
|
||||
logger.info("model install cancelled: %s", req.repo_id)
|
||||
# A cancel is user intent, not a failure — don't set a cooldown.
|
||||
_install_cooldowns.pop(req.repo_id, None)
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
"downloaded": 0, "total": 0, "pct": 0.0,
|
||||
"phase": "install_cancelled",
|
||||
})
|
||||
except Exception as e:
|
||||
_resolving.set()
|
||||
logger.info("model install failed for %s: %s", req.repo_id, e)
|
||||
@@ -515,28 +200,12 @@ async def install_model(req: InstallModelRequest):
|
||||
"error": str(e),
|
||||
})
|
||||
finally:
|
||||
_cancelled.discard(req.repo_id)
|
||||
download_aggregator.finish(req.repo_id)
|
||||
hf_progress.current_repo_id.reset(token)
|
||||
|
||||
loop.create_task(asyncio.to_thread(_do))
|
||||
return {"status": "install_started", "repo_id": req.repo_id}
|
||||
|
||||
|
||||
@router.post("/models/install/cancel")
|
||||
async def cancel_install(req: InstallModelRequest):
|
||||
"""Request cancellation of an in-flight install (FDL-11).
|
||||
|
||||
Best-effort: stops further retry attempts and marks the row cancelled. A
|
||||
single in-flight snapshot_download/Xet fetch isn't interruptible mid-file
|
||||
in hf_hub 1.7.2, so an already-streaming file finishes; the cancel takes
|
||||
effect at the next retry boundary. Clears the cooldown so the user can
|
||||
immediately restart."""
|
||||
_cancelled.add(req.repo_id)
|
||||
_install_cooldowns.pop(req.repo_id, None)
|
||||
return {"cancelling": req.repo_id}
|
||||
|
||||
|
||||
# ── Delete ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@router.delete("/models/{repo_id:path}")
|
||||
|
||||
@@ -146,94 +146,6 @@ def _hub_cache_roots() -> list[str]:
|
||||
return roots
|
||||
|
||||
|
||||
# ── Weight-presence (truncated-cache) detection ─────────────────────────────
|
||||
# A cache that downloaded config/tokenizer files but not the weight shard still
|
||||
# occupies bytes on disk, so a size-only "installed" check (#352/#581/#606) reads
|
||||
# it as installed and the first-run wizard hides the re-download button, stranding
|
||||
# the user (#622). These helpers tell a *complete* snapshot from a truncated one by
|
||||
# checking for a plausible weight file — the same class `download.py` guards at
|
||||
# install time and `model_manager.py` repairs at load time. Shared here (the lowest
|
||||
# module in the setup import graph; `download.py` imports from this module) so the
|
||||
# floors live in exactly one place and can't drift between the three call sites.
|
||||
|
||||
_MIN_WEIGHT_BYTES = 5 * 1024 * 1024 # tensor formats: a real shard is ≥ a few MB
|
||||
|
||||
# Per-extension floors. ONNX graphs are legitimately small (a complete model can be
|
||||
# well under 5 MB), so they get a lower floor that still rejects a bytes-only partial.
|
||||
_WEIGHT_FLOORS = {
|
||||
".safetensors": _MIN_WEIGHT_BYTES,
|
||||
".bin": _MIN_WEIGHT_BYTES,
|
||||
".ckpt": _MIN_WEIGHT_BYTES,
|
||||
".pt": _MIN_WEIGHT_BYTES,
|
||||
".pth": _MIN_WEIGHT_BYTES,
|
||||
".gguf": _MIN_WEIGHT_BYTES,
|
||||
".onnx": 64 * 1024,
|
||||
}
|
||||
|
||||
|
||||
def snapshot_has_weights(snapshot_path: str) -> bool:
|
||||
"""True when a finished snapshot dir holds a plausible weight file.
|
||||
|
||||
A snapshot is complete if it contains a recognized weight file meeting its
|
||||
per-extension floor OR any file ≥ the global 5 MB floor (the lenient catch for
|
||||
non-standard weight names). Returns True when the path can't be inspected — an
|
||||
un-walkable dir must never be reported as truncated, only a confirmed weight-less
|
||||
one. `getsize` follows symlinks, so HF's snapshot→blob links resolve correctly;
|
||||
a broken link (missing blob) raises OSError and is skipped, i.e. counts as absent.
|
||||
"""
|
||||
try:
|
||||
for root, _dirs, files in os.walk(snapshot_path, followlinks=True):
|
||||
for f in files:
|
||||
try:
|
||||
size = os.path.getsize(os.path.join(root, f))
|
||||
except OSError:
|
||||
continue
|
||||
ext = os.path.splitext(f)[1].lower()
|
||||
floor = _WEIGHT_FLOORS.get(ext)
|
||||
if floor is not None and size >= floor:
|
||||
return True
|
||||
if size >= _MIN_WEIGHT_BYTES:
|
||||
return True
|
||||
except OSError:
|
||||
return True # can't inspect — don't mislabel as truncated
|
||||
return False
|
||||
|
||||
|
||||
def _snapshot_dirs(repo_id: str) -> list[str]:
|
||||
"""Existing snapshot revision dirs for a repo across the candidate cache roots."""
|
||||
name = _repo_dir_name(repo_id)
|
||||
dirs: list[str] = []
|
||||
for root in _hub_cache_roots():
|
||||
snaps = os.path.join(root, name, "snapshots")
|
||||
try:
|
||||
for rev in os.listdir(snaps):
|
||||
rev_dir = os.path.join(snaps, rev)
|
||||
if os.path.isdir(rev_dir):
|
||||
dirs.append(rev_dir)
|
||||
except OSError:
|
||||
continue
|
||||
return dirs
|
||||
|
||||
|
||||
def cache_is_complete(model: dict) -> bool:
|
||||
"""True when this model's on-disk cache is usable (not a truncated download).
|
||||
|
||||
Config-only repos (``config_only: true`` in models.yaml — e.g. pyannote's
|
||||
diarisation pipeline, whose real weights live in referenced sub-repos) carry no
|
||||
weight file of their own, so the weight check would false-positive them as
|
||||
incomplete (#622 caveat). They're exempt: cache presence alone means complete.
|
||||
A weight-bearing repo is complete only if at least one of its snapshots has
|
||||
weights; if no snapshot dir is found on disk we can't prove truncation, so we
|
||||
don't downgrade (the size-based caller already decided it's cached).
|
||||
"""
|
||||
if model.get("config_only"):
|
||||
return True
|
||||
dirs = _snapshot_dirs(model["repo_id"])
|
||||
if not dirs:
|
||||
return True
|
||||
return any(snapshot_has_weights(d) for d in dirs)
|
||||
|
||||
|
||||
def _is_cached_on_disk(repo_id: str) -> bool:
|
||||
"""Direct-filesystem fallback for is_cached when scan_cache_dir is unavailable.
|
||||
|
||||
@@ -308,12 +220,8 @@ def is_cached(repo_id: str) -> bool:
|
||||
except Exception as e:
|
||||
# scan_cache_dir can raise on Windows (WinError 448 'untrusted mount
|
||||
# point'); fall back to a direct disk check so a cached model isn't
|
||||
# mistaken for missing and re-downloaded in a loop (#117/#118). Logged
|
||||
# at WARNING with the exception type (MM2-09) so this fallback isn't
|
||||
# invisible when triaging a Windows cache report — it previously logged
|
||||
# at DEBUG and never showed at the default level.
|
||||
logger.warning("is_cached: scan_cache_dir failed (%s: %s); using on-disk fallback for %s",
|
||||
type(e).__name__, e, repo_id)
|
||||
# mistaken for missing and re-downloaded in a loop (#117/#118).
|
||||
logger.debug("scan_cache_dir failed (%s); using disk fallback", e)
|
||||
return _is_cached_on_disk(repo_id)
|
||||
|
||||
|
||||
@@ -374,15 +282,9 @@ def list_models():
|
||||
out = []
|
||||
for m in KNOWN_MODELS:
|
||||
cached = cached_by_repo.get(m["repo_id"])
|
||||
on_disk = cached is not None and cached["size_on_disk"] > 0
|
||||
# A size-positive cache can still be a truncated download (config landed,
|
||||
# weight shard didn't). Treat that as not-installed + incomplete so the
|
||||
# wizard re-offers the download instead of stranding the user (#622).
|
||||
incomplete = on_disk and not cache_is_complete(m)
|
||||
out.append({
|
||||
**m,
|
||||
"installed": on_disk and not incomplete,
|
||||
"incomplete": incomplete,
|
||||
"installed": cached is not None and cached["size_on_disk"] > 0,
|
||||
"size_on_disk_bytes": cached["size_on_disk"] if cached else 0,
|
||||
"nb_files": cached["nb_files"] if cached else 0,
|
||||
"supported": _model_supported(m),
|
||||
@@ -477,9 +379,6 @@ def recommendations():
|
||||
entries = []
|
||||
for rid in recommended_ids:
|
||||
meta = known_by_id.get(rid, {})
|
||||
# Mirror /models: a truncated cache (weights missing) is not installed, so
|
||||
# the wizard counts it toward the remaining download instead of "all set".
|
||||
installed = rid in cached_ids and cache_is_complete(meta or {"repo_id": rid})
|
||||
entries.append({
|
||||
"repo_id": rid,
|
||||
"label": meta.get("label", rid),
|
||||
@@ -487,7 +386,7 @@ def recommendations():
|
||||
"size_gb": meta.get("size_gb", 0),
|
||||
"required": bool(meta.get("required", False)),
|
||||
"note": meta.get("note"),
|
||||
"installed": installed,
|
||||
"installed": rid in cached_ids,
|
||||
})
|
||||
|
||||
to_download_gb = sum(e["size_gb"] for e in entries if not e["installed"])
|
||||
|
||||
@@ -373,46 +373,6 @@ def preflight():
|
||||
"status": gpu_status, "detail": gpu_detail, "fix": gpu_fix,
|
||||
})
|
||||
|
||||
# ── GPU routing for the ACTIVE TTS engine (#21 — no silent CPU fallback).
|
||||
# Distinct from the hardware "gpu" check above: this asks "will the engine
|
||||
# the user actually selected use that GPU on this host?" Built from the same
|
||||
# canonical probe + resolver the Engine Compatibility Matrix uses.
|
||||
try:
|
||||
from services.tts_backend import gpu_routing_verdict
|
||||
gpu_routing = gpu_routing_verdict()
|
||||
except Exception as exc: # never break preflight on a routing hiccup
|
||||
logger.warning("preflight gpu_routing failed: %s", exc)
|
||||
gpu_routing = None
|
||||
if gpu_routing:
|
||||
_rs = gpu_routing.get("routing_status")
|
||||
_eng = gpu_routing.get("engine") or "active engine"
|
||||
_dev = gpu_routing.get("effective_device") or "?"
|
||||
_why = gpu_routing.get("routing_reason")
|
||||
if _rs == "accelerated" and not _why:
|
||||
r_status, r_detail, r_fix = "pass", f"{_eng} → {_dev} (accelerated)", None
|
||||
elif _rs == "accelerated": # driver/arch caveat
|
||||
r_status, r_detail, r_fix = "warn", f"{_eng} → {_dev}: {_why}", (
|
||||
"GPU selected but may fail at kernel launch — update drivers / "
|
||||
"reinstall torch for this GPU architecture.")
|
||||
elif _rs == "cpu_fallback":
|
||||
r_status, r_detail, r_fix = "warn", (
|
||||
f"{_eng} runs on CPU here: {_why or 'no GPU path for this host'}"), (
|
||||
"Pick an engine that supports this host's GPU for a speedup, or "
|
||||
"continue on CPU (slower).")
|
||||
elif _rs == "cpu_only":
|
||||
r_status, r_detail, r_fix = "pass", f"{_eng} → cpu (no accelerator on this host)", None
|
||||
elif _rs == "unavailable":
|
||||
r_status, r_detail, r_fix = "fail", (
|
||||
f"{_eng} can't run on this host: {_why or 'needs a GPU this machine lacks'}"), (
|
||||
"Select an engine with a CPU path in Settings → Engines.")
|
||||
else: # "none" / unknown
|
||||
r_status, r_detail, r_fix = "warn", "No active TTS engine resolved for routing.", (
|
||||
"Pick an engine in Settings → Engines.")
|
||||
checks.append({
|
||||
"id": "gpu_routing", "label": "Active engine routing",
|
||||
"status": r_status, "detail": r_detail, "fix": r_fix,
|
||||
})
|
||||
|
||||
# ── Network
|
||||
net_ok = _probe_network()
|
||||
checks.append({
|
||||
@@ -440,13 +400,9 @@ def preflight():
|
||||
"gpu_available": gpu["available"],
|
||||
"gpu_driver": gpu["driver"],
|
||||
"gpu_device_name": gpu["device_name"],
|
||||
# Canonical probe (distinguishes ROCm from CUDA):
|
||||
"gpu_family": (gpu_routing or {}).get("host_family", "cpu"),
|
||||
"vram_gb": (gpu_routing or {}).get("vram_gb", 0.0),
|
||||
"ram_gb": round(ram, 1),
|
||||
"disk_free_gb": round(free, 1),
|
||||
},
|
||||
"gpu_routing": gpu_routing,
|
||||
}
|
||||
|
||||
|
||||
|
||||
+87
-122
@@ -18,7 +18,7 @@ import shutil
|
||||
|
||||
from core.config import OUTPUTS_DIR, DATA_DIR, CRASH_LOG_PATH, LOG_PATH, IDLE_TIMEOUT_SECONDS
|
||||
from core.version import APP_VERSION
|
||||
from services.model_manager import get_model_status, get_best_device, resolve_omnivoice_checkpoint
|
||||
from services.model_manager import get_model_status, get_best_device
|
||||
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
|
||||
|
||||
# Router-level loopback gate. Every route mounted on `router` (GET + POST,
|
||||
@@ -107,52 +107,6 @@ def _ui_port() -> int:
|
||||
return 3901
|
||||
|
||||
|
||||
def _fast_download_status() -> dict:
|
||||
"""Report the download-acceleration state for the Settings UI (FDL-03).
|
||||
|
||||
Reports the *runtime* truth, not just whether hf_xet is importable. The app
|
||||
currently sets ``HF_HUB_DISABLE_XET=1`` by default (main.py) — Xet's chunked
|
||||
transfer is fast but its progress bypasses our tqdm patch, so the legacy-LFS
|
||||
path is forced to keep accurate byte progress. So:
|
||||
|
||||
* ``xet_installed`` — hf_xet present
|
||||
* ``xet_active`` — installed AND not disabled via HF_HUB_DISABLE_XET
|
||||
* ``xet_enabled`` — alias of xet_active (what the UI badge keys off)
|
||||
|
||||
Must never throw: /system/info is called on every Settings load.
|
||||
"""
|
||||
installed = False
|
||||
version = None
|
||||
try:
|
||||
import hf_xet # noqa: F401
|
||||
installed = True
|
||||
try:
|
||||
from importlib.metadata import version as _ver
|
||||
version = _ver("hf-xet")
|
||||
except Exception:
|
||||
version = None
|
||||
except Exception:
|
||||
installed = False
|
||||
disabled = str(os.environ.get("HF_HUB_DISABLE_XET", "")).strip().lower() in {"1", "true", "yes", "on"}
|
||||
active = installed and not disabled
|
||||
try:
|
||||
from core import prefs
|
||||
high_perf = prefs.resolve(
|
||||
"xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False
|
||||
)
|
||||
high_perf = high_perf if isinstance(high_perf, bool) else \
|
||||
str(high_perf).strip().lower() in {"1", "true", "yes", "on"}
|
||||
except Exception:
|
||||
high_perf = False
|
||||
return {
|
||||
"xet_installed": installed,
|
||||
"xet_active": active,
|
||||
"xet_enabled": active, # UI badge: only true when Xet actually runs
|
||||
"xet_version": version,
|
||||
"high_performance": bool(high_perf),
|
||||
}
|
||||
|
||||
|
||||
def _has_hf_token() -> bool:
|
||||
# Phase 1 AUTH-01..06 cascade. Delegates to the 3-source resolver
|
||||
# (App → Env → HF-CLI) instead of reading env/HF-CLI directly. This
|
||||
@@ -174,23 +128,88 @@ def model_status():
|
||||
|
||||
@router.get("/model/loaded")
|
||||
def loaded_models():
|
||||
"""List all currently loaded models for the flush dropdown (MM2-04).
|
||||
Thin delegation to the model_lifecycle facade — shape unchanged:
|
||||
``{models, count}``."""
|
||||
from services import model_lifecycle
|
||||
return model_lifecycle.list_loaded()
|
||||
"""Return details about all currently loaded models for the flush dropdown.
|
||||
|
||||
Returns a list of models with name, type, device, and estimated VRAM usage.
|
||||
"""
|
||||
import services.model_manager as mm
|
||||
|
||||
models = []
|
||||
|
||||
# 1. TTS model (OmniVoice)
|
||||
if mm.model is not None:
|
||||
device = "unknown"
|
||||
vram_mb = 0
|
||||
try:
|
||||
device = str(next(mm.model.parameters()).device) if hasattr(mm.model, 'parameters') else get_best_device()
|
||||
except Exception:
|
||||
device = get_best_device()
|
||||
try:
|
||||
torch = mm._lazy_torch()
|
||||
if torch.cuda.is_available():
|
||||
vram_mb = torch.cuda.memory_allocated() / (1024 ** 2)
|
||||
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
driver = getattr(torch.mps, "driver_allocated_memory", None)
|
||||
if driver:
|
||||
vram_mb = driver() / (1024 ** 2)
|
||||
except Exception:
|
||||
pass
|
||||
models.append({
|
||||
"id": "tts",
|
||||
"name": "OmniVoice TTS",
|
||||
"checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
|
||||
"device": device,
|
||||
"vram_mb": round(vram_mb, 1),
|
||||
"unloadable": True,
|
||||
})
|
||||
|
||||
# 2. ASR model (WhisperX)
|
||||
if mm.model is not None and hasattr(mm.model, '_asr_pipe') and mm.model._asr_pipe is not None:
|
||||
models.append({
|
||||
"id": "asr",
|
||||
"name": "WhisperX ASR",
|
||||
"checkpoint": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
|
||||
"device": "cpu",
|
||||
"vram_mb": 0,
|
||||
"unloadable": False, # tied to TTS model lifecycle
|
||||
})
|
||||
|
||||
# 3. Diarization pipeline
|
||||
if mm._diar_pipeline is not None:
|
||||
models.append({
|
||||
"id": "diarization",
|
||||
"name": "Pyannote Diarization",
|
||||
"checkpoint": "pyannote/speaker-diarization-3.1",
|
||||
"device": get_best_device(),
|
||||
"vram_mb": 0,
|
||||
"unloadable": True,
|
||||
})
|
||||
|
||||
return {"models": models, "count": len(models)}
|
||||
|
||||
|
||||
@router.post("/model/unload/{model_id}")
|
||||
async def unload_model(model_id: str):
|
||||
"""Unload a specific model by id (MM2-04). Delegates to model_lifecycle;
|
||||
an unknown id maps to HTTP 400. ``tts`` | ``diarization`` |
|
||||
``sidecar:<id>`` | ``sidecars``."""
|
||||
from services import model_lifecycle
|
||||
try:
|
||||
return await model_lifecycle.unload(model_id)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
"""Unload a specific model by ID."""
|
||||
import services.model_manager as mm
|
||||
|
||||
if model_id == "tts":
|
||||
async with mm._model_lock:
|
||||
if mm.model is not None:
|
||||
mm.model = None
|
||||
mm.free_vram()
|
||||
return {"unloaded": "tts", "success": True}
|
||||
return {"unloaded": "tts", "success": False, "reason": "not loaded"}
|
||||
|
||||
elif model_id == "diarization":
|
||||
if mm._diar_pipeline is not None:
|
||||
mm._diar_pipeline = None
|
||||
mm.free_vram()
|
||||
return {"unloaded": "diarization", "success": True}
|
||||
return {"unloaded": "diarization", "success": False, "reason": "not loaded"}
|
||||
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail=f"Unknown model id: {model_id}")
|
||||
|
||||
|
||||
@router.get("/system/info", response_model=SystemInfoResponse)
|
||||
@@ -208,11 +227,10 @@ def system_info():
|
||||
"outputs_dir": OUTPUTS_DIR,
|
||||
"crash_log_path": CRASH_LOG_PATH,
|
||||
"idle_timeout_seconds": IDLE_TIMEOUT_SECONDS,
|
||||
"model_checkpoint": resolve_omnivoice_checkpoint(), # #693: show the effective checkpoint, not a leaked raw value
|
||||
"model_checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
|
||||
"asr_model": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
|
||||
"translate_provider": os.environ.get("TRANSLATE_PROVIDER", "google"),
|
||||
"has_hf_token": _has_hf_token(),
|
||||
"fast_download": _fast_download_status(),
|
||||
"device": get_best_device(),
|
||||
"python": sys.version.split()[0],
|
||||
"platform": sys.platform,
|
||||
@@ -304,10 +322,9 @@ def _tauri_log_candidates():
|
||||
os.path.join(home, "Library/Logs/OmniVoice/backend_err.log"),
|
||||
]
|
||||
if sys.platform.startswith("linux"):
|
||||
data_dir = os.environ.get("XDG_DATA_HOME") or os.path.join(home, ".local/share")
|
||||
state_dir = os.environ.get("XDG_STATE_HOME") or os.path.join(home, ".local/state")
|
||||
return [
|
||||
os.path.join(data_dir, bid, "logs", "tauri.log"),
|
||||
os.path.join(home, ".local/share", bid, "logs", "tauri.log"),
|
||||
os.path.join(home, ".config", bid, "logs", "tauri.log"),
|
||||
os.path.join(state_dir, "OmniVoice", "backend.log"),
|
||||
os.path.join(state_dir, "OmniVoice", "backend_err.log"),
|
||||
@@ -447,14 +464,6 @@ async def clear_system_logs():
|
||||
status_code=500,
|
||||
detail=f"Could not clear log at {p}: {e}. The file may be open in another process or read-only — close tailing tools and retry.",
|
||||
)
|
||||
if cleared_any:
|
||||
# The crash log just shrank to zero — drop any stored ack so a stale
|
||||
# byte count can't suppress the next 'crash-last-session' notice.
|
||||
for key in ("crash_log_acked", "crash_log_acked_size"):
|
||||
try:
|
||||
prefs_delete(key)
|
||||
except Exception:
|
||||
pass
|
||||
return {"cleared": cleared_any}
|
||||
|
||||
|
||||
@@ -556,21 +565,6 @@ async def flush_memory(unload_model: bool = False):
|
||||
|
||||
# ── Actionable notifications ──────────────────────────────────────────────
|
||||
|
||||
_GPU_ARCH_WARNING: "list[str | None]" = [] # [-1] = computed result
|
||||
|
||||
|
||||
def _gpu_arch_warning_cached() -> "str | None":
|
||||
"""check_device_compatibility() once per process (it lazy-imports torch —
|
||||
too heavy for the 30s notifications poll)."""
|
||||
if not _GPU_ARCH_WARNING:
|
||||
try:
|
||||
from services.model_manager import check_device_compatibility
|
||||
compatible, warning = check_device_compatibility()
|
||||
_GPU_ARCH_WARNING.append(None if compatible else warning)
|
||||
except Exception:
|
||||
_GPU_ARCH_WARNING.append(None)
|
||||
return _GPU_ARCH_WARNING[-1]
|
||||
|
||||
|
||||
@router.get("/system/notifications")
|
||||
def system_notifications():
|
||||
@@ -602,20 +596,6 @@ def system_notifications():
|
||||
},
|
||||
})
|
||||
|
||||
# 1b. GPU compute capability unsupported by this torch build (#284) —
|
||||
# the model "runs" but emits pure noise, the worst silent failure mode
|
||||
# (RTX 50-series Blackwell sm_120 on pre-cu128 wheels). The loader logs
|
||||
# this, but a log line never reached the affected users — surface it in
|
||||
# the panel. Checked once per process: it lazy-imports torch.
|
||||
gpu_warn = _gpu_arch_warning_cached()
|
||||
if gpu_warn:
|
||||
notes.append({
|
||||
"id": "gpu-arch-unsupported",
|
||||
"level": "error",
|
||||
"title": "GPU not supported by this PyTorch build",
|
||||
"message": gpu_warn + " Until then, output will be noise/garbage.",
|
||||
})
|
||||
|
||||
# 2. Missing ffmpeg
|
||||
ffmpeg_ok = False
|
||||
try:
|
||||
@@ -706,33 +686,18 @@ def _crashed_last_session() -> bool:
|
||||
if not os.path.exists(CRASH_LOG_PATH):
|
||||
return False
|
||||
size = os.path.getsize(CRASH_LOG_PATH)
|
||||
if size == 0:
|
||||
acked = int(prefs_get("crash_log_acked_size", 0) or 0)
|
||||
if size <= acked:
|
||||
return False
|
||||
mtime = os.path.getmtime(CRASH_LOG_PATH)
|
||||
# Composite ack (size + mtime): a bare byte count goes stale after the log
|
||||
# is truncated — the next crash log can stay smaller than the old acked
|
||||
# size forever, silently suppressing 'crash-last-session'. The ack only
|
||||
# holds while it still covers the file's current state.
|
||||
ack = prefs_get("crash_log_acked")
|
||||
if isinstance(ack, dict):
|
||||
if float(ack.get("mtime", 0) or 0) >= mtime and int(ack.get("size", 0) or 0) >= size:
|
||||
return False
|
||||
else:
|
||||
# Legacy size-only ack from older builds.
|
||||
if size <= int(prefs_get("crash_log_acked_size", 0) or 0):
|
||||
return False
|
||||
return mtime < _PROCESS_START_TS
|
||||
return os.path.getmtime(CRASH_LOG_PATH) < _PROCESS_START_TS
|
||||
|
||||
|
||||
@router.post("/system/crash/ack")
|
||||
async def ack_crash():
|
||||
"""Mark the current crash log as seen — dismisses the
|
||||
'crash-last-session' notification until the log changes again."""
|
||||
size = mtime = 0
|
||||
if os.path.exists(CRASH_LOG_PATH):
|
||||
size = os.path.getsize(CRASH_LOG_PATH)
|
||||
mtime = os.path.getmtime(CRASH_LOG_PATH)
|
||||
prefs_set("crash_log_acked", {"size": size, "mtime": mtime})
|
||||
'crash-last-session' notification until the log grows again."""
|
||||
size = os.path.getsize(CRASH_LOG_PATH) if os.path.exists(CRASH_LOG_PATH) else 0
|
||||
prefs_set("crash_log_acked_size", size)
|
||||
return {"acked_size": size}
|
||||
|
||||
|
||||
|
||||
@@ -98,30 +98,6 @@ async def ws_tts(websocket: WebSocket):
|
||||
model = await get_model()
|
||||
backend = get_active_tts_backend(model=model)
|
||||
|
||||
# ── Routing gate (#21 — no silent CPU fallback). WebSockets have
|
||||
# no response headers, so this uses frames: an error frame +
|
||||
# close on `unavailable`, a one-time `routing` frame on
|
||||
# cpu_fallback / accelerated-with-caveat (before any audio).
|
||||
from core.device_caps import detect_host_caps
|
||||
from services.engine_routing import resolve_routing, routing_notice
|
||||
from core.scrub import scrub_text
|
||||
_routing = resolve_routing(
|
||||
getattr(backend, "gpu_compat", ("cpu",)), detect_host_caps())
|
||||
if _routing["routing_status"] == "unavailable":
|
||||
await websocket.send_json({
|
||||
"type": "error",
|
||||
"detail": scrub_text(_routing["routing_reason"])
|
||||
or "engine cannot run on this host",
|
||||
})
|
||||
continue # don't stream; wait for the next request
|
||||
_notice = routing_notice(_routing)
|
||||
if _notice:
|
||||
await websocket.send_json({
|
||||
"type": "routing",
|
||||
"status": _notice[0],
|
||||
"reason": scrub_text(_notice[1]) if _notice[1] else None,
|
||||
})
|
||||
|
||||
# Build generation kwargs
|
||||
kw: dict = {"speed": data.get("speed", 1.0)}
|
||||
if data.get("language"):
|
||||
@@ -168,80 +144,50 @@ async def ws_tts(websocket: WebSocket):
|
||||
except Exception:
|
||||
kw["voice"] = voice
|
||||
|
||||
# Wave 1.4: split the request into sentences so the first
|
||||
# sentence's audio streams while later sentences are still
|
||||
# synthesizing — this is the time-to-first-audio win. The
|
||||
# chunker handles abbreviations/acronyms/decimals and CJK /
|
||||
# non-Latin terminators; single-sentence requests behave
|
||||
# exactly like the old single-shot path.
|
||||
from services.sentence_chunker import SentenceChunker
|
||||
_chunker = SentenceChunker(language=(data.get("language") or "en"))
|
||||
sentences = _chunker.push(text)
|
||||
sentences.extend(_chunker.flush())
|
||||
if not sentences:
|
||||
sentences = [text]
|
||||
|
||||
# Run generation in the GPU pool
|
||||
import functools
|
||||
from services.model_manager import run_on_gpu_pool_guarded
|
||||
from services.model_manager import _gpu_pool
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
def _generate(sentence_text):
|
||||
def _generate():
|
||||
from services.audio_dsp import apply_mastering, normalize_audio
|
||||
wav = backend.generate(sentence_text, **kw)
|
||||
wav = backend.generate(text, **kw)
|
||||
sr_actual = backend.sample_rate
|
||||
# Like _run_tts in openai_compat: studio engines (VoxCPM2)
|
||||
# opt out of the broadcast mastering chain. This is the
|
||||
# other route that runs the active backend, so it needs the
|
||||
# same guard. Loudness normalisation still runs.
|
||||
if not getattr(backend, "applies_own_mastering", False):
|
||||
wav = apply_mastering(wav, sample_rate=sr_actual)
|
||||
wav = apply_mastering(wav, sample_rate=sr_actual)
|
||||
wav = normalize_audio(wav, target_dBFS=-2.0)
|
||||
return wav, sr_actual
|
||||
|
||||
wav_tensor, sr = await loop.run_in_executor(_gpu_pool, _generate)
|
||||
|
||||
# Send metadata after generation so sample_rate is real
|
||||
await websocket.send_json({
|
||||
"type": "start",
|
||||
"sample_rate": sr,
|
||||
"channels": 1,
|
||||
"format": "pcm16",
|
||||
"engine": backend.id,
|
||||
})
|
||||
|
||||
# Stream PCM16 chunks over the WebSocket
|
||||
import torch
|
||||
total_samples = 0
|
||||
sr = backend.sample_rate
|
||||
started = False
|
||||
# Convert to 16-bit PCM
|
||||
pcm = (wav_tensor * 32767).clamp(-32768, 32767).to(torch.int16)
|
||||
if pcm.ndim == 2:
|
||||
pcm = pcm[0] # mono
|
||||
pcm_bytes = pcm.numpy().tobytes()
|
||||
|
||||
for sentence in sentences:
|
||||
# Bounded + pool-reset on hang so a wedged generate can't
|
||||
# starve the GPU pool and brick the backend (#730 class). On
|
||||
# timeout GpuJobTimeoutError propagates to the handler below,
|
||||
# which sends an actionable error frame.
|
||||
wav_tensor, sr = await run_on_gpu_pool_guarded(
|
||||
functools.partial(_generate, sentence),
|
||||
what="TTS generate",
|
||||
)
|
||||
total_samples = len(pcm)
|
||||
sent_samples = 0
|
||||
chunk_bytes = CHUNK_SAMPLES * 2 # 2 bytes per int16 sample
|
||||
|
||||
if not started:
|
||||
# Send metadata after the first generation so
|
||||
# sample_rate is real (lazy-loading engines report
|
||||
# their true rate only once weights are up).
|
||||
await websocket.send_json({
|
||||
"type": "start",
|
||||
"sample_rate": sr,
|
||||
"channels": 1,
|
||||
"format": "pcm16",
|
||||
"engine": backend.id,
|
||||
})
|
||||
started = True
|
||||
|
||||
# Convert to 16-bit PCM and stream
|
||||
pcm = (wav_tensor * 32767).clamp(-32768, 32767).to(torch.int16)
|
||||
if pcm.ndim == 2:
|
||||
pcm = pcm[0] # mono
|
||||
pcm_bytes = pcm.numpy().tobytes()
|
||||
|
||||
n_samples = len(pcm)
|
||||
sent_samples = 0
|
||||
while sent_samples < n_samples:
|
||||
end = min(sent_samples + CHUNK_SAMPLES, n_samples)
|
||||
chunk = pcm_bytes[sent_samples * 2: end * 2]
|
||||
await websocket.send_bytes(chunk)
|
||||
sent_samples = end
|
||||
# Yield to event loop between chunks for responsiveness
|
||||
await asyncio.sleep(0)
|
||||
total_samples += n_samples
|
||||
while sent_samples < total_samples:
|
||||
end = min(sent_samples + CHUNK_SAMPLES, total_samples)
|
||||
start_byte = sent_samples * 2
|
||||
end_byte = end * 2
|
||||
chunk = pcm_bytes[start_byte:end_byte]
|
||||
await websocket.send_bytes(chunk)
|
||||
sent_samples = end
|
||||
# Yield to event loop between chunks for responsiveness
|
||||
await asyncio.sleep(0)
|
||||
|
||||
gen_time = round(time.perf_counter() - t0, 3)
|
||||
duration = round(total_samples / sr, 3)
|
||||
|
||||
@@ -34,8 +34,6 @@ class SystemInfoResponse(BaseModel):
|
||||
asr_model: str = "unknown"
|
||||
translate_provider: str = "unknown"
|
||||
has_hf_token: bool = False
|
||||
# Xet fast-download backend state (FDL-03): {xet_enabled, xet_version, high_performance}
|
||||
fast_download: dict | None = None
|
||||
device: str = "cpu"
|
||||
python: str = ""
|
||||
platform: str = ""
|
||||
@@ -129,41 +127,16 @@ class DeviceInfo(BaseModel):
|
||||
gpu_available: bool = False
|
||||
gpu_driver: str | None = None
|
||||
gpu_device_name: str | None = None
|
||||
# From the canonical device probe (core.device_caps) — distinguishes ROCm
|
||||
# from CUDA, unlike the legacy nvidia-smi-based gpu_vendor/gpu_backend.
|
||||
gpu_family: str = "cpu"
|
||||
vram_gb: float = 0.0
|
||||
ram_gb: float = 0.0
|
||||
disk_free_gb: float = 0.0
|
||||
|
||||
|
||||
class GpuRouting(BaseModel):
|
||||
"""Routing verdict for the active TTS engine on THIS host (#21).
|
||||
|
||||
Distinct from the per-engine `routing_*` keys in `/engines`: this is the
|
||||
single verdict for the *currently-selected* engine, surfaced in preflight +
|
||||
diagnose so the user hears about a CPU fallback / unavailable GPU before a
|
||||
slow or failed synth — no silent CPU fallback.
|
||||
"""
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
engine: str | None = None # active TTS engine id
|
||||
effective_device: str | None = None # device it will actually use here
|
||||
routing_status: str | None = None # accelerated|cpu_fallback|cpu_only|unavailable|none
|
||||
routing_reason: str | None = None # scrubbed; null when none
|
||||
host_family: str = "cpu" # detect_host_caps().family
|
||||
vram_gb: float = 0.0
|
||||
|
||||
|
||||
class PreflightResponse(BaseModel):
|
||||
"""GET /setup/preflight"""
|
||||
ok: bool
|
||||
has_warnings: bool = False
|
||||
checks: list[PreflightCheck] = Field(default_factory=list)
|
||||
device: DeviceInfo
|
||||
# Explicit field (PreflightResponse has no extra="allow") so the verdict
|
||||
# survives serialization instead of being silently dropped.
|
||||
gpu_routing: GpuRouting | None = None
|
||||
|
||||
|
||||
class InstallModelRequest(BaseModel):
|
||||
|
||||
Binary file not shown.
@@ -14,10 +14,6 @@
|
||||
# required (optional) — true if the app needs this model to function
|
||||
# platforms (optional) — restrict to specific OS+arch tags (e.g. darwin-arm64, cuda)
|
||||
# note (optional) — shown in the UI as a tooltip/footnote
|
||||
# config_only (optional) — true for pipeline repos that ship no weight file of
|
||||
# their own (weights live in referenced sub-repos). Such
|
||||
# a cache is legitimately tiny, so the truncated-download
|
||||
# (weights-missing) detector must NOT flag it incomplete.
|
||||
# ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
models:
|
||||
@@ -120,83 +116,12 @@ models:
|
||||
size_gb: 0.05
|
||||
note: "Smallest/fastest Moonshine, sub-200ms latency. Lower accuracy than base. Requires moonshine-onnx."
|
||||
|
||||
# ── sherpa-onnx live dictation (ONNX, CPU, streaming + offline) ────────
|
||||
# Live faster-than-real-time dictation via the k2-fsa/sherpa-onnx runtime.
|
||||
# `engine: sherpa-onnx`, `dictation_id` (backend model id), and `tag`
|
||||
# (offline | streaming) are extra fields the model-store list passes through
|
||||
# so the dictation UI can filter/group these (role=ASR, engine=sherpa-onnx).
|
||||
# Requires `uv add sherpa-onnx` (CPU wheels, all platforms).
|
||||
|
||||
- repo_id: "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8"
|
||||
label: "Parakeet TDT v3 (sherpa-onnx — dictation, 25 EU langs)"
|
||||
role: ASR
|
||||
size_gb: 0.18
|
||||
engine: sherpa-onnx
|
||||
dictation_id: sherpa-parakeet-tdt-v3
|
||||
tag: offline
|
||||
note: "Recommended live-dictation default. CPU, int8 ONNX. Requires sherpa-onnx."
|
||||
|
||||
- repo_id: "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8"
|
||||
label: "Parakeet TDT v2 (sherpa-onnx — dictation, English)"
|
||||
role: ASR
|
||||
size_gb: 0.17
|
||||
engine: sherpa-onnx
|
||||
dictation_id: sherpa-parakeet-tdt-v2
|
||||
tag: offline
|
||||
note: "English live dictation. CPU, int8 ONNX. Requires sherpa-onnx."
|
||||
|
||||
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20"
|
||||
label: "Zipformer Bilingual (sherpa-onnx — streaming, zh+en)"
|
||||
role: ASR
|
||||
size_gb: 0.13
|
||||
engine: sherpa-onnx
|
||||
dictation_id: sherpa-zipformer-bilingual-zh-en
|
||||
tag: streaming
|
||||
note: "True streaming partials as you speak (zh+en). CPU. Requires sherpa-onnx."
|
||||
|
||||
- repo_id: "csukuangfj/sherpa-onnx-streaming-paraformer-bilingual-zh-en"
|
||||
label: "Paraformer Bilingual (sherpa-onnx — streaming, zh+en)"
|
||||
role: ASR
|
||||
size_gb: 0.115
|
||||
engine: sherpa-onnx
|
||||
dictation_id: sherpa-paraformer-bilingual-zh-en
|
||||
tag: streaming
|
||||
note: "True streaming partials (zh+en). CPU. Requires sherpa-onnx."
|
||||
|
||||
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-en-20M-2023-02-17"
|
||||
label: "Zipformer Streaming EN 20M (sherpa-onnx — streaming, English)"
|
||||
role: ASR
|
||||
size_gb: 0.128
|
||||
engine: sherpa-onnx
|
||||
dictation_id: sherpa-zipformer-en-20m
|
||||
tag: streaming
|
||||
note: "Tiny English streaming model, very low latency. CPU. Requires sherpa-onnx."
|
||||
|
||||
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23"
|
||||
label: "Zipformer Streaming ZH 14M (sherpa-onnx — streaming, Chinese)"
|
||||
role: ASR
|
||||
size_gb: 0.074
|
||||
engine: sherpa-onnx
|
||||
dictation_id: sherpa-zipformer-zh-14m
|
||||
tag: streaming
|
||||
note: "Tiny Chinese streaming model, very low latency. CPU. Requires sherpa-onnx."
|
||||
|
||||
- repo_id: "csukuangfj/sherpa-onnx-whisper-tiny"
|
||||
label: "Whisper Tiny (sherpa-onnx — dictation, 90+ langs)"
|
||||
role: ASR
|
||||
size_gb: 0.116
|
||||
engine: sherpa-onnx
|
||||
dictation_id: sherpa-whisper-tiny
|
||||
tag: offline
|
||||
note: "Multilingual offline dictation (auto-detect). CPU, int8 ONNX. Requires sherpa-onnx."
|
||||
|
||||
# ── Diarisation ───────────────────────────────────────────────────────
|
||||
|
||||
- repo_id: "pyannote/speaker-diarization-3.1"
|
||||
label: "pyannote speaker diarisation (multi-speaker videos)"
|
||||
role: Diarisation
|
||||
size_gb: 0.8
|
||||
config_only: true # pipeline repo; real weights live in referenced sub-repos
|
||||
note: "Needs an HF_TOKEN with license accepted."
|
||||
|
||||
# ── Optional TTS ──────────────────────────────────────────────────────
|
||||
|
||||
+6
-124
@@ -49,12 +49,6 @@ _BASE_SCHEMA = """
|
||||
personality TEXT DEFAULT '',
|
||||
description TEXT DEFAULT '',
|
||||
is_demo INTEGER DEFAULT 0,
|
||||
verified_own_voice INTEGER DEFAULT 0,
|
||||
consent_text TEXT DEFAULT '',
|
||||
consent_audio_path TEXT DEFAULT '',
|
||||
consent_recorded_at REAL DEFAULT NULL,
|
||||
kind TEXT DEFAULT 'clone',
|
||||
vd_states TEXT DEFAULT NULL,
|
||||
created_at REAL
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS generation_history (
|
||||
@@ -144,35 +138,6 @@ _BASE_SCHEMA = """
|
||||
value TEXT NOT NULL,
|
||||
updated_at REAL NOT NULL
|
||||
);
|
||||
|
||||
-- Wave 2.2: per-agent MCP voice bindings. An MCP client (Claude Code,
|
||||
-- Cursor, …) identified by the X-OmniVoice-Client-Id header it sends is
|
||||
-- bound to a default voice profile / engine. Fresh installs create it
|
||||
-- here; v0.3.x upgrades get it via alembic 0004.
|
||||
CREATE TABLE IF NOT EXISTS mcp_client_bindings (
|
||||
client_id TEXT PRIMARY KEY,
|
||||
label TEXT NOT NULL DEFAULT '',
|
||||
profile_id TEXT,
|
||||
default_engine TEXT,
|
||||
last_seen_at REAL,
|
||||
created_at REAL
|
||||
);
|
||||
|
||||
-- Expressive-TTS Spec 01 Phase 1: user pronunciation dictionary. A
|
||||
-- per-language word→respelling map applied as pure text substitution
|
||||
-- before synthesis (Settings → Pronunciation). Fresh installs create it
|
||||
-- here; existing DBs get it via alembic 0008_pronunciation_dictionary.
|
||||
-- Both paths converge on this identical schema (dual-path discipline).
|
||||
CREATE TABLE IF NOT EXISTS pronunciation_entries (
|
||||
id TEXT PRIMARY KEY,
|
||||
term TEXT NOT NULL,
|
||||
replacement TEXT NOT NULL DEFAULT '',
|
||||
type TEXT NOT NULL DEFAULT 'respelling',
|
||||
language TEXT NOT NULL DEFAULT '*',
|
||||
enabled INTEGER NOT NULL DEFAULT 1,
|
||||
created_at REAL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_pron_lang ON pronunciation_entries(language);
|
||||
"""
|
||||
|
||||
# Only tables/columns this module is allowed to ALTER. Prevents SQL injection via
|
||||
@@ -222,72 +187,6 @@ def _migrate(conn, current: int) -> int:
|
||||
return current
|
||||
|
||||
|
||||
def _reconcile_additive_columns(conn) -> None:
|
||||
"""Make the live schema converge to ``_BASE_SCHEMA`` by ADDing any column the
|
||||
canonical schema declares but an existing table is missing — the belt for
|
||||
when alembic can't run on an upgraded DB.
|
||||
|
||||
``CREATE TABLE IF NOT EXISTS`` (init_db) never adds columns to a table that
|
||||
already exists, the legacy ``_migrate`` only knows pre-0.3 columns, and
|
||||
``_run_alembic_upgrade`` swallows failures. So a DB whose ``alembic_version``
|
||||
is stamped at a removed revision (e.g. after running a preview build), or
|
||||
where alembic isn't importable in the bundled interpreter, would otherwise
|
||||
lose every alembic-era additive column forever — the ``no such column:
|
||||
consent_audio_path`` 500 (#552/#547), and the same class for
|
||||
``kind``/``vd_states``/``is_demo``/.... Additive only: never drops or retypes
|
||||
a column, so it is safe and backward-compatible with existing user data. The
|
||||
canonical names/types/defaults come solely from ``_BASE_SCHEMA`` (developer
|
||||
controlled), so the ALTER is injection-safe.
|
||||
"""
|
||||
canon = sqlite3.connect(":memory:")
|
||||
try:
|
||||
canon.executescript(_BASE_SCHEMA)
|
||||
_tables_sql = "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'"
|
||||
live_tables = {r[0] for r in conn.execute(_tables_sql)}
|
||||
for table in (r[0] for r in canon.execute(_tables_sql)):
|
||||
if table not in live_tables:
|
||||
continue # whole table missing → init_db's CREATE already made it
|
||||
have = {r[1] for r in conn.execute(f"PRAGMA table_info({table})")}
|
||||
# (cid, name, type, notnull, dflt_value, pk)
|
||||
for _cid, name, ctype, notnull, dflt, _pk in canon.execute(f"PRAGMA table_info({table})"):
|
||||
if name in have or not _IDENT_RE.match(name):
|
||||
continue
|
||||
ddl = f'ALTER TABLE "{table}" ADD COLUMN "{name}" {ctype or "TEXT"}'
|
||||
if dflt is not None:
|
||||
ddl += f" DEFAULT {dflt}"
|
||||
elif notnull:
|
||||
ddl += " DEFAULT ''" # SQLite requires a default to ADD a NOT NULL column
|
||||
try:
|
||||
conn.execute(ddl)
|
||||
logger.info("schema reconcile: added missing column %s.%s", table, name)
|
||||
except sqlite3.OperationalError as exc:
|
||||
if "duplicate column" not in str(exc).lower():
|
||||
logger.warning("schema reconcile ALTER %s.%s failed: %s", table, name, exc)
|
||||
conn.commit()
|
||||
finally:
|
||||
canon.close()
|
||||
|
||||
|
||||
def ensure_schema() -> None:
|
||||
"""Idempotently ensure the base tables + additive columns exist.
|
||||
|
||||
A runtime self-heal for a DB that somehow missed init — e.g. a write hitting
|
||||
``no such table: generation_history`` (#710) because ``init_db()``'s
|
||||
``executescript`` never took on that DB. Safe to call anytime: it's just
|
||||
``CREATE ... IF NOT EXISTS`` plus the additive-only column reconcile, so it
|
||||
never drops or retypes anything and is backward-compatible with user data.
|
||||
Cheaper than ``init_db()`` (skips the legacy ``_migrate`` + alembic), so a
|
||||
write path can call it on a schema error and retry without a 500.
|
||||
"""
|
||||
conn = get_db()
|
||||
try:
|
||||
conn.executescript(_BASE_SCHEMA)
|
||||
_reconcile_additive_columns(conn)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def init_db():
|
||||
conn = get_db()
|
||||
try:
|
||||
@@ -296,11 +195,6 @@ def init_db():
|
||||
new_version = _migrate(conn, version)
|
||||
if new_version != version:
|
||||
conn.execute(f"PRAGMA user_version = {new_version}")
|
||||
# Converge any alembic-era additive columns that CREATE TABLE IF NOT
|
||||
# EXISTS + the legacy _migrate don't add to a pre-existing table
|
||||
# (consent_audio_path, kind, ...). Runs regardless of whether alembic
|
||||
# below succeeds, so an unrunnable alembic can't leave a 500-ing schema.
|
||||
_reconcile_additive_columns(conn)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
@@ -314,12 +208,10 @@ def init_db():
|
||||
|
||||
def _run_alembic_upgrade() -> None:
|
||||
"""Best-effort `alembic upgrade head` on startup. Non-fatal: if alembic
|
||||
isn't reachable (e.g. a stripped-down install) or its version is stamped at
|
||||
a revision no longer in versions/ (e.g. after running a preview build), log
|
||||
a warning and move on. The schema is still kept correct by
|
||||
_reconcile_additive_columns (run in init_db above and again here on failure)
|
||||
— CREATE TABLE IF NOT EXISTS alone does NOT add columns to a pre-existing
|
||||
table, so the reconcile is what actually guarantees additive columns land."""
|
||||
isn't reachable (e.g. user running a stripped-down install or migrations
|
||||
were already applied out-of-band), log a warning and move on. The
|
||||
_BASE_SCHEMA CREATE TABLE IF NOT EXISTS above guarantees the runtime
|
||||
schema is correct regardless."""
|
||||
try:
|
||||
import os
|
||||
from alembic import command
|
||||
@@ -337,16 +229,6 @@ def _run_alembic_upgrade() -> None:
|
||||
cfg.set_main_option("sqlalchemy.url", f"sqlite:///{DB_PATH}")
|
||||
command.upgrade(cfg, "head")
|
||||
except Exception as exc:
|
||||
# Don't block startup on a migration tooling problem. Converge the schema
|
||||
# directly so a swallowed failure (alembic not importable, or
|
||||
# alembic_version stamped at a removed revision) still lands the additive
|
||||
# columns instead of 500-ing on `no such column` (#552/#547).
|
||||
# Don't block startup on a migration tooling problem. The runtime
|
||||
# schema is already correct via _BASE_SCHEMA.
|
||||
logger.warning("alembic upgrade head skipped: %s", exc)
|
||||
try:
|
||||
conn = get_db()
|
||||
try:
|
||||
_reconcile_additive_columns(conn)
|
||||
finally:
|
||||
conn.close()
|
||||
except Exception as exc2: # noqa: BLE001
|
||||
logger.warning("schema reconcile after alembic failure also failed: %s", exc2)
|
||||
|
||||
@@ -1,336 +0,0 @@
|
||||
"""Free-text voice-description → voice-design parameter mapper (issue #317).
|
||||
|
||||
Parity with the hosted omnivoice.app "Describe your voice" field, implemented
|
||||
fully locally: a deterministic keyword/phrase mapper that projects a natural-
|
||||
language description (e.g. ``"a warm elderly British storyteller, slightly
|
||||
raspy"``) onto the **existing** voice-design parameter space — the same six
|
||||
categories the Design tab's attribute picker drives (Gender / Age / Pitch /
|
||||
Style / EnglishAccent / ChineseDialect).
|
||||
|
||||
Design notes
|
||||
============
|
||||
* **No model, no network.** This is an ordered synonym-table matcher, not an
|
||||
LLM call — it runs identically on macOS/Windows/Linux with zero deps beyond
|
||||
the stdlib, preserving the local-first guarantee.
|
||||
* **Single source of truth.** Every canonical token this module can emit is
|
||||
validated at import time against the engine taxonomy in
|
||||
``omnivoice/utils/voice_design.py`` (loaded via ``core.archetypes``), so the
|
||||
mapper can never produce an instruct item the engine validator would reject
|
||||
(the issue-#89 / #115 crash modes). The Chinese translations of each token
|
||||
(e.g. ``男``/``中年``) are *derived* from that taxonomy, never hardcoded.
|
||||
* **Ordered rules, first match wins.** Within a category, rules are checked in
|
||||
a hand-ordered list so more specific phrases outrank generic ones
|
||||
("young child" → child, not young adult; "very deep" → very low pitch, not
|
||||
low pitch). Within one rule, the earliest occurrence in the text is reported
|
||||
as the matched phrase. Deterministic by construction.
|
||||
* **Graceful degradation.** Anything the taxonomy can't express (timbre words
|
||||
like "raspy", role words like "storyteller") is returned in ``unmatched`` so
|
||||
the UI can tell the user exactly which parts were ignored instead of failing
|
||||
silently (issue #317's validation-feedback note). A description with no
|
||||
matches at all yields all-``Auto`` attrs and an empty instruct.
|
||||
|
||||
Localization note (CLAUDE.md): the only hardcoded CJK here is
|
||||
``DIALECT_PINYIN`` — a functional pinyin → Chinese-dialect-token mapping
|
||||
(model vocabulary, like ``frontend/src/utils/constants.js``). Registered in
|
||||
``tests/test_no_hardcoded_cjk.py``'s allowlist with this justification.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
# Reuse the taxonomy already loaded (stdlib-only, by file path) by the
|
||||
# archetype engine — same single source of truth, one loader to maintain.
|
||||
from core.archetypes import _VD
|
||||
|
||||
_EN_TO_ZH = _VD._INSTRUCT_EN_TO_ZH # {"male": "男", ...}
|
||||
_ZH_RE = _VD._ZH_RE
|
||||
_VALID = _VD._INSTRUCT_ALL_VALID # every token the engine accepts
|
||||
_DIALECTS = set(_VD._INSTRUCT_CATEGORIES[5]) # the 12 Chinese dialect tokens
|
||||
|
||||
# Category names match the frontend's CATEGORIES keys (utils/constants.js) and
|
||||
# the archetype ``attrs`` shape, so the response drops straight into vdStates.
|
||||
CATEGORY_ORDER = ("Gender", "Age", "Pitch", "Style", "EnglishAccent", "ChineseDialect")
|
||||
|
||||
# ── Pinyin / romanized names → Chinese-dialect tokens (functional vocabulary) ─
|
||||
DIALECT_PINYIN = {
|
||||
"henan": "河南话",
|
||||
"shaanxi": "陕西话",
|
||||
"sichuan": "四川话",
|
||||
"szechuan": "四川话",
|
||||
"guizhou": "贵州话",
|
||||
"yunnan": "云南话",
|
||||
"guilin": "桂林话",
|
||||
"jinan": "济南话",
|
||||
"shijiazhuang": "石家庄话",
|
||||
"gansu": "甘肃话",
|
||||
"ningxia": "宁夏话",
|
||||
"qingdao": "青岛话",
|
||||
"dongbei": "东北话",
|
||||
"northeastern chinese": "东北话",
|
||||
}
|
||||
|
||||
# ── Synonym tables ────────────────────────────────────────────────────────────
|
||||
# Per category: ordered list of (canonical_token, [phrases]). First rule with
|
||||
# any hit wins the category, so specific phrases must precede generic ones.
|
||||
# Each canonical token's Chinese translation from the taxonomy is appended
|
||||
# automatically at compile time (so "中年" maps to "middle-aged", etc.).
|
||||
|
||||
_GENDER_RULES = [
|
||||
("female", [
|
||||
"female", "woman", "women", "lady", "ladies", "girl", "girls",
|
||||
"feminine", "gal", "grandma", "grandmother", "granny", "mother",
|
||||
"mom", "mum", "aunt", "auntie", "queen", "princess", "actress",
|
||||
"she", "her",
|
||||
]),
|
||||
("male", [
|
||||
"male", "man", "men", "guy", "guys", "boy", "boys", "masculine",
|
||||
"gentleman", "gentlemen", "dude", "grandpa", "grandfather", "father",
|
||||
"dad", "uncle", "king", "prince", "actor", "he", "him", "his",
|
||||
]),
|
||||
]
|
||||
|
||||
# Order is load-bearing: "child" precedes "young adult" so "young child" →
|
||||
# child; "middle-aged" precedes "elderly" so elderly's bare "aged" synonym
|
||||
# can't fire inside the hyphenated "middle-aged" (hyphen is a \b boundary);
|
||||
# "elderly" precedes "young adult" so grandparent words don't fall through.
|
||||
_AGE_RULES = [
|
||||
("child", [
|
||||
"child", "children", "kid", "kiddo", "toddler", "little boy",
|
||||
"little girl", "young boy", "young girl", "small child", "childlike",
|
||||
]),
|
||||
("teenager", ["teenager", "teen", "teenage", "adolescent"]),
|
||||
("middle-aged", [
|
||||
"middle-aged", "middle aged", "middle age", "midlife", "forties",
|
||||
"fifties", "sixties", "mature",
|
||||
]),
|
||||
("elderly", [
|
||||
"elderly", "old man", "old woman", "old lady", "older man",
|
||||
"older woman", "elder", "senior", "aged", "grandpa", "grandfather",
|
||||
"grandma", "grandmother", "granny", "retired", "seventies",
|
||||
"eighties", "nineties", "old",
|
||||
]),
|
||||
("young adult", [
|
||||
"young adult", "young woman", "young man", "young lady", "youthful",
|
||||
"twenties", "thirties", "college", "young",
|
||||
]),
|
||||
]
|
||||
|
||||
# "very …" rules precede their plain counterparts so "very deep" doesn't stop
|
||||
# at "deep". Bare "low"/"high" only count next to a voice word (pitch/voice/
|
||||
# tone/register) to avoid false hits like "high quality" or "low effort".
|
||||
_PITCH_RULES = [
|
||||
("very low pitch", [
|
||||
"very low pitch", "very low-pitched", "very low pitched",
|
||||
"very low voice", "very low tone", "very deep", "extremely deep",
|
||||
"extremely low", "ultra deep", "booming",
|
||||
]),
|
||||
("very high pitch", [
|
||||
"very high pitch", "very high-pitched", "very high pitched",
|
||||
"very high voice", "very high tone", "extremely high", "squeaky",
|
||||
"shrill", "falsetto", "chipmunk",
|
||||
]),
|
||||
("low pitch", [
|
||||
"low pitch", "low-pitched", "low pitched", "low voice", "low tone",
|
||||
"low register", "deep", "deeper", "bass", "baritone", "husky",
|
||||
]),
|
||||
("high pitch", [
|
||||
"high pitch", "high-pitched", "high pitched", "high voice",
|
||||
"high tone", "high register", "soprano",
|
||||
]),
|
||||
("moderate pitch", [
|
||||
"moderate pitch", "medium pitch", "medium-pitched", "medium pitched",
|
||||
"mid-range", "midrange", "average pitch", "moderate",
|
||||
]),
|
||||
]
|
||||
|
||||
_STYLE_RULES = [
|
||||
("whisper", [
|
||||
"whisper", "whispering", "whispered", "whispery", "hushed",
|
||||
"breathy", "soft-spoken", "soft spoken",
|
||||
]),
|
||||
]
|
||||
|
||||
# Bare "english" means the language, so only the explicit "english accent"
|
||||
# phrase maps to british. "chinese" maps to the chinese *accent* (English
|
||||
# speech with a Chinese accent); actual dialect words live in DIALECT_PINYIN.
|
||||
_ACCENT_RULES = [
|
||||
("american accent", [
|
||||
"american", "america", "usa", "us accent", "midwestern",
|
||||
"californian", "new york",
|
||||
]),
|
||||
("british accent", [
|
||||
"british", "britain", "english accent", "england", "uk accent",
|
||||
"london", "cockney", "posh", "received pronunciation",
|
||||
]),
|
||||
("australian accent", ["australian", "australia", "aussie"]),
|
||||
("canadian accent", ["canadian", "canada"]),
|
||||
("indian accent", ["indian", "india"]),
|
||||
("chinese accent", ["chinese accent", "chinese-accented", "chinese"]),
|
||||
("korean accent", ["korean", "korea"]),
|
||||
("japanese accent", ["japanese", "japan"]),
|
||||
("portuguese accent", ["portuguese", "portugal", "brazilian", "brazil"]),
|
||||
("russian accent", ["russian", "russia"]),
|
||||
]
|
||||
|
||||
_DIALECT_RULES = [
|
||||
(token, [pinyin for pinyin, tok in DIALECT_PINYIN.items() if tok == token])
|
||||
for token in sorted(_DIALECTS)
|
||||
]
|
||||
|
||||
_RULES = {
|
||||
"Gender": _GENDER_RULES,
|
||||
"Age": _AGE_RULES,
|
||||
"Pitch": _PITCH_RULES,
|
||||
"Style": _STYLE_RULES,
|
||||
"EnglishAccent": _ACCENT_RULES,
|
||||
"ChineseDialect": _DIALECT_RULES,
|
||||
}
|
||||
|
||||
# Import-time guard: every canonical token must be in the engine taxonomy, so
|
||||
# a taxonomy rename upstream fails loudly here instead of at synthesis time.
|
||||
for _cat_rules in _RULES.values():
|
||||
for _token, _ in _cat_rules:
|
||||
assert _token in _VALID, f"describe_voice token not in taxonomy: {_token!r}"
|
||||
for _tok in DIALECT_PINYIN.values():
|
||||
assert _tok in _DIALECTS, f"DIALECT_PINYIN value not a taxonomy dialect: {_tok!r}"
|
||||
|
||||
|
||||
# ── Pattern compilation ───────────────────────────────────────────────────────
|
||||
def _compile_phrase(phrase: str) -> re.Pattern:
|
||||
"""Compile a synonym phrase to a regex.
|
||||
|
||||
Latin phrases get word boundaries (so "male" never fires inside "female",
|
||||
"old" never inside "bold") and flexible separators (space or hyphen, so
|
||||
"middle aged" also matches "middle-aged"). CJK phrases match as plain
|
||||
substrings — word boundaries are meaningless without spaces.
|
||||
"""
|
||||
if _ZH_RE.search(phrase):
|
||||
return re.compile(re.escape(phrase))
|
||||
parts = [re.escape(p) for p in re.split(r"[ -]+", phrase) if p]
|
||||
return re.compile(r"\b" + r"[\s\-]+".join(parts) + r"\b")
|
||||
|
||||
|
||||
def _compiled_rules():
|
||||
out = {}
|
||||
for cat, rules in _RULES.items():
|
||||
compiled = []
|
||||
for token, phrases in rules:
|
||||
pats = list(phrases)
|
||||
# Derive the Chinese form of each canonical token from the
|
||||
# taxonomy (e.g. "middle-aged" → "中年") — never hardcoded here.
|
||||
zh = _EN_TO_ZH.get(token)
|
||||
if zh:
|
||||
pats.append(zh)
|
||||
if token not in pats:
|
||||
pats.append(token) # the canonical token always matches itself
|
||||
compiled.append((token, [_compile_phrase(p) for p in pats]))
|
||||
out[cat] = compiled
|
||||
return out
|
||||
|
||||
|
||||
_COMPILED = _compiled_rules()
|
||||
|
||||
# "<N> year(s) old / <N>-year-old / <N> yo" → an age bracket. Runs before the
|
||||
# keyword rules so the trailing "old" never misfires as elderly.
|
||||
_AGE_NUM = re.compile(
|
||||
r"\b(\d{1,3})(?:[\s\-]*(?:years?|yrs?|yr)[\s\-]*old|[\s\-]*(?:yo|y/o))\b"
|
||||
)
|
||||
|
||||
|
||||
def _age_token_for(years: int) -> str:
|
||||
if years <= 12:
|
||||
return "child"
|
||||
if years <= 19:
|
||||
return "teenager"
|
||||
if years <= 39:
|
||||
return "young adult"
|
||||
if years <= 64:
|
||||
return "middle-aged"
|
||||
return "elderly"
|
||||
|
||||
|
||||
def _normalize(description: str) -> str:
|
||||
text = (description or "").lower()
|
||||
text = text.replace("’", "'").replace("‘", "'")
|
||||
text = text.replace("“", '"').replace("”", '"')
|
||||
return re.sub(r"[ \t]+", " ", text)
|
||||
|
||||
|
||||
def _match_category(category: str, text: str):
|
||||
"""Return (token, match) for the first rule with a hit, else None.
|
||||
|
||||
Rule order decides the winning token; within the winning rule the earliest
|
||||
occurrence in the text is reported as the matched phrase.
|
||||
"""
|
||||
if category == "Age":
|
||||
m = _AGE_NUM.search(text)
|
||||
if m:
|
||||
return _age_token_for(int(m.group(1))), m
|
||||
for token, patterns in _COMPILED[category]:
|
||||
best = None
|
||||
for pat in patterns:
|
||||
m = pat.search(text)
|
||||
if m is not None and (best is None or m.start() < best.start()):
|
||||
best = m
|
||||
if best is not None:
|
||||
return token, best
|
||||
return None
|
||||
|
||||
|
||||
# Fragment splitter for the "unmatched" report: clause separators (incl. the
|
||||
# CJK comma/ideographic stop, which CJK descriptions use instead of ASCII).
|
||||
_FRAGMENT = re.compile(r"[^,;.!?()\n,。;!?、]+")
|
||||
_HAS_CONTENT = re.compile(r"[\w一-鿿]")
|
||||
|
||||
|
||||
def parse_description(description: str) -> dict:
|
||||
"""Map a free-text voice description onto the design parameter space.
|
||||
|
||||
Returns a dict with:
|
||||
* ``attrs`` — full category → token map (``"Auto"`` where nothing
|
||||
matched); same shape as the Design tab's ``vdStates``.
|
||||
* ``instruct`` — validator-safe instruct string built from the matched
|
||||
tokens, in canonical category order (may be ``""``).
|
||||
* ``matched`` — list of ``{category, token, phrase}`` for transparency.
|
||||
* ``unmatched`` — clause fragments that contributed no attribute, so the
|
||||
UI can show what was ignored instead of failing silently.
|
||||
"""
|
||||
text = _normalize(description)
|
||||
attrs = {cat: "Auto" for cat in CATEGORY_ORDER}
|
||||
matched = []
|
||||
spans = []
|
||||
|
||||
for category in CATEGORY_ORDER:
|
||||
hit = _match_category(category, text)
|
||||
if hit is None:
|
||||
continue
|
||||
token, m = hit
|
||||
attrs[category] = token
|
||||
matched.append({"category": category, "token": token, "phrase": m.group(0)})
|
||||
spans.append((m.start(), m.end()))
|
||||
|
||||
# Accents are English-only and dialects Chinese-only in the engine
|
||||
# taxonomy; a dialect voice speaks Chinese, so an accent token alongside
|
||||
# it is contradictory (the issue-#114 conflict class). Dialect wins.
|
||||
if attrs["ChineseDialect"] != "Auto" and attrs["EnglishAccent"] != "Auto":
|
||||
dropped = attrs["EnglishAccent"]
|
||||
attrs["EnglishAccent"] = "Auto"
|
||||
matched = [m for m in matched if not (m["category"] == "EnglishAccent" and m["token"] == dropped)]
|
||||
|
||||
instruct = ", ".join(attrs[c] for c in CATEGORY_ORDER if attrs[c] != "Auto")
|
||||
|
||||
unmatched = []
|
||||
for frag in _FRAGMENT.finditer(text):
|
||||
if not _HAS_CONTENT.search(frag.group(0)):
|
||||
continue
|
||||
lo, hi = frag.start(), frag.end()
|
||||
if any(s < hi and e > lo for s, e in spans):
|
||||
continue
|
||||
unmatched.append(frag.group(0).strip())
|
||||
|
||||
return {
|
||||
"attrs": attrs,
|
||||
"instruct": instruct,
|
||||
"matched": matched,
|
||||
"unmatched": unmatched,
|
||||
}
|
||||
@@ -1,279 +0,0 @@
|
||||
"""Canonical host compute-capability probe — the single source of truth for
|
||||
"what can this machine actually accelerate on."
|
||||
|
||||
Every routing decision (the engine compatibility matrix, ``/setup/preflight``,
|
||||
``/system/diagnose``, and the synth-time no-silent-fallback gating) reads from
|
||||
``detect_host_caps()`` so the probe and the model loader can never disagree.
|
||||
|
||||
Design contract (load-bearing):
|
||||
- **Never raises** to a caller. A broken torch / driver crash degrades to a
|
||||
cached CPU-only ``probe_ok=False`` result; every endpoint stays responsive
|
||||
(local-first: the app must work with no GPU and even with a broken torch).
|
||||
- **No network call** — driver/sysctl reads only, no tensor allocation, so it
|
||||
stays kernel-free on cold start.
|
||||
- **No new regex** on any driver/device string (CodeQL py/polynomial-redos):
|
||||
the only string parse is the ``int(driver.split(".")[0])`` shape reused
|
||||
from the wizard, and arch comparison is plain list membership.
|
||||
- Distinguishes **ROCm from CUDA** (unlike the gguf ``hardware_probe``):
|
||||
ROCm-on-HIP presents through ``torch.cuda`` but is reported ``family="rocm"``.
|
||||
|
||||
The ``get_best_device()`` loader (``services.model_manager``) delegates its
|
||||
*family* decision here while keeping its own DirectML branch and the ROCm
|
||||
``HSA_OVERRIDE_GFX_VERSION`` env side-effect — the probe **reads**, the loader
|
||||
**writes**. (The gguf ``hardware_probe.detect_capabilities()`` rebase onto this
|
||||
module is a deliberate follow-up: it has its own torch-mocked test suite and a
|
||||
VRAM-driven quant table that is unaffected by the family rename, so it is kept
|
||||
out of this backend-only slice.)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import platform as _platform
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal
|
||||
|
||||
DeviceFamily = Literal["cuda", "rocm", "mps", "xpu", "cpu"]
|
||||
|
||||
# Stable substring stamped onto notes that represent a real kernel-launch risk
|
||||
# (arch/driver mismatch) — as opposed to advisory notes (multi-GPU, VRAM query
|
||||
# failed, DirectML present). ``engine_routing`` keys the "accelerated, but…"
|
||||
# caveat off this marker so advisory notes never downgrade an accelerated badge.
|
||||
KERNEL_RISK_MARKER = "may fail at kernel launch"
|
||||
|
||||
# Substring marking a DirectML-present (Windows GPU) host. The probe reports
|
||||
# such hosts as ``family="cpu"`` (DirectML is not a torch device family); the
|
||||
# router reads this marker to explain the neutral badge instead of "no GPU".
|
||||
DIRECTML_MARKER = "DirectML device present"
|
||||
|
||||
# NOTE: the NVIDIA driver-version check (min R555 for the bundled CUDA runtime)
|
||||
# is intentionally NOT done here — it requires shelling to ``nvidia-smi``, which
|
||||
# would put a subprocess on the cold-start probe path. That check stays in
|
||||
# ``wizard._detect_gpu`` (preflight), which already runs it. The probe only
|
||||
# emits the torch-visible SM-arch caveat (cheap, metadata-only).
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HostCaps:
|
||||
"""Snapshot of the host's accelerator capability. Immutable + cached."""
|
||||
|
||||
family: DeviceFamily
|
||||
"""Best available accelerator family, else ``"cpu"``."""
|
||||
|
||||
available_families: tuple[DeviceFamily, ...]
|
||||
"""Everything usable; **always includes** ``"cpu"`` (invariant)."""
|
||||
|
||||
device_name: str = ""
|
||||
"""Device 0's name, e.g. ``"NVIDIA RTX 4090"`` / ``"Apple Silicon (MPS)"``."""
|
||||
|
||||
vram_gb: float = 0.0
|
||||
"""CUDA/ROCm total VRAM in GB; MPS = system RAM / 2; 0 for cpu/xpu."""
|
||||
|
||||
driver: str | None = None
|
||||
"""Raw ROCm HIP version string (``torch.version.hip``) or ``None``. The
|
||||
NVIDIA driver-version check is owned by ``wizard._detect_gpu`` (it already
|
||||
shells to ``nvidia-smi``); the probe stays subprocess-free."""
|
||||
|
||||
notes: tuple[str, ...] = ()
|
||||
"""Author-controlled English advisories (never user input). Empty on a
|
||||
clean accelerated host."""
|
||||
|
||||
probe_ok: bool = True
|
||||
"""``False`` only when torch could not be imported (degraded CPU-only)."""
|
||||
|
||||
|
||||
def _probe() -> HostCaps:
|
||||
"""Run the probe once. Enumerates every failure branch from the spec's
|
||||
degradation contract; never raises."""
|
||||
try:
|
||||
import torch
|
||||
except Exception:
|
||||
return HostCaps(
|
||||
family="cpu",
|
||||
available_families=("cpu",),
|
||||
notes=("torch not importable; treating host as CPU-only",),
|
||||
probe_ok=False,
|
||||
)
|
||||
|
||||
notes: list[str] = []
|
||||
# Probe EVERY accelerator independently into this list (don't short-circuit
|
||||
# after the first hit) so `available_families` is honest on hybrid hosts
|
||||
# (e.g. an NVIDIA GPU + an Intel iGPU exposed via IPEX). The preferred
|
||||
# `family` is chosen by priority at the end.
|
||||
detected: list[DeviceFamily] = []
|
||||
device_name = ""
|
||||
vram_gb = 0.0
|
||||
driver: str | None = None
|
||||
|
||||
# ── CUDA / ROCm (both present through torch.cuda) ────────────────────
|
||||
cuda_ok = False
|
||||
try:
|
||||
cuda_ok = bool(torch.cuda.is_available())
|
||||
except Exception as exc: # broken CUDA init (forked process / driver crash)
|
||||
notes.append(f"CUDA init raised: {type(exc).__name__}")
|
||||
|
||||
if cuda_ok:
|
||||
try:
|
||||
count = int(torch.cuda.device_count())
|
||||
except Exception:
|
||||
count = 0
|
||||
if count == 0:
|
||||
notes.append("CUDA reports available but device_count==0")
|
||||
else:
|
||||
is_rocm = getattr(torch.version, "hip", None) is not None
|
||||
detected.append("rocm" if is_rocm else "cuda")
|
||||
if is_rocm:
|
||||
driver = getattr(torch.version, "hip", None)
|
||||
if count > 1:
|
||||
notes.append(f"{count} GPUs detected; routing reflects device 0")
|
||||
try:
|
||||
device_name = torch.cuda.get_device_name(0)
|
||||
except Exception:
|
||||
device_name = ""
|
||||
try:
|
||||
_free, total = torch.cuda.mem_get_info()
|
||||
vram_gb = float(total) / (1024 ** 3)
|
||||
except Exception:
|
||||
notes.append("VRAM query failed")
|
||||
# SM-arch mismatch (mirrors model_manager.check_device_compatibility).
|
||||
try:
|
||||
major, minor = torch.cuda.get_device_capability(0)
|
||||
arch_list = getattr(torch.cuda, "_get_arch_list", lambda: [])()
|
||||
if arch_list:
|
||||
sm_tag = f"sm_{major}{minor}"
|
||||
compute_tag = f"compute_{major}{minor}"
|
||||
if sm_tag not in arch_list and compute_tag not in arch_list:
|
||||
notes.append(
|
||||
f"{device_name or 'GPU'} ({sm_tag}) not in this torch "
|
||||
f"build's archs ({', '.join(arch_list)}) — "
|
||||
f"{KERNEL_RISK_MARKER}"
|
||||
)
|
||||
except Exception:
|
||||
# Arch metadata unavailable on this torch build — skip the check
|
||||
# (treated as compatible, exactly as check_device_compatibility).
|
||||
pass
|
||||
|
||||
# ── Intel XPU via IPEX ───────────────────────────────────────────────
|
||||
try:
|
||||
import intel_extension_for_pytorch # noqa: F401
|
||||
if hasattr(torch, "xpu") and torch.xpu.is_available():
|
||||
detected.append("xpu")
|
||||
if not device_name:
|
||||
try:
|
||||
device_name = torch.xpu.get_device_name(0)
|
||||
except Exception:
|
||||
# XPU present but unnamed — family classification still holds.
|
||||
pass
|
||||
notes.append("XPU VRAM not queried (unreliable across IPEX versions)")
|
||||
except Exception:
|
||||
# IPEX absent or XPU probe failed — no XPU on this host.
|
||||
pass
|
||||
|
||||
# ── Apple Silicon MPS ────────────────────────────────────────────────
|
||||
try:
|
||||
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
detected.append("mps")
|
||||
if not device_name:
|
||||
device_name = "Apple Silicon (MPS)"
|
||||
if not vram_gb:
|
||||
try:
|
||||
import psutil
|
||||
vram_gb = float(psutil.virtual_memory().total) / (1024 ** 3) / 2
|
||||
except Exception:
|
||||
notes.append("psutil unavailable; MPS VRAM unknown")
|
||||
except Exception:
|
||||
# MPS probe raised on a non-Apple/old torch — treat as no MPS.
|
||||
pass
|
||||
|
||||
# ── DirectML — Windows GPU, NOT a torch device family ────────────────
|
||||
try:
|
||||
import torch_directml
|
||||
if torch_directml.device_count() > 0:
|
||||
notes.append(
|
||||
f"{DIRECTML_MARKER} (Windows GPU); torch-family probe treats "
|
||||
f"as non-accelerated"
|
||||
)
|
||||
except Exception:
|
||||
# torch_directml absent (the common case) — no DirectML on this host.
|
||||
pass
|
||||
|
||||
# Preferred family by priority; cpu when nothing accelerated was detected.
|
||||
family: DeviceFamily = "cpu"
|
||||
for pref in ("cuda", "rocm", "xpu", "mps"):
|
||||
if pref in detected:
|
||||
family = pref # type: ignore[assignment]
|
||||
break
|
||||
# available_families: every detected accelerator + cpu, deduped, cpu last.
|
||||
available: tuple[DeviceFamily, ...] = tuple(dict.fromkeys([*detected, "cpu"]))
|
||||
|
||||
return HostCaps(
|
||||
family=family,
|
||||
available_families=available,
|
||||
device_name=device_name,
|
||||
vram_gb=vram_gb,
|
||||
driver=driver,
|
||||
notes=tuple(notes),
|
||||
probe_ok=True,
|
||||
)
|
||||
|
||||
|
||||
@functools.lru_cache(maxsize=1)
|
||||
def detect_host_caps() -> HostCaps:
|
||||
"""Cached per-process host capabilities. Never raises, makes no network
|
||||
call, kernel-free on cold start. Host compute capability does not change at
|
||||
runtime in any supported desktop flow (no GPU hot-plug; switching the active
|
||||
engine does not re-probe — routing is recomputed from these same caps), so
|
||||
a single probe per process is correct. ``probe_ok=False`` is cached too."""
|
||||
return _probe()
|
||||
|
||||
|
||||
def refresh() -> HostCaps:
|
||||
"""Clear the cache and re-probe. **TEST-ONLY** — nothing in the running app
|
||||
calls this (host caps are immutable per process)."""
|
||||
detect_host_caps.cache_clear()
|
||||
return detect_host_caps()
|
||||
|
||||
|
||||
def mlx_supported() -> tuple[bool, str]:
|
||||
"""``(ok, reason)``. ``ok=True`` **only** on Apple Silicon
|
||||
(``sys.platform == "darwin"`` and ``platform.machine() == "arm64"``) with
|
||||
torch MPS available — the shared gate for MLX-Audio / MLX-Whisper (#390).
|
||||
|
||||
Gates on exact-string equality (no regex → no CodeQL surface). On any
|
||||
non-Apple host it returns ``False`` **before** any package import, so a
|
||||
stray ``mlx_*`` wheel on Linux/Windows never reports available.
|
||||
"""
|
||||
if sys.platform != "darwin" or _platform.machine() != "arm64":
|
||||
if sys.platform == "darwin":
|
||||
return (False, "MLX requires Apple Silicon; this Mac is Intel")
|
||||
return (
|
||||
False,
|
||||
f"MLX requires Apple Silicon; this host is "
|
||||
f"{sys.platform}/{_platform.machine()}",
|
||||
)
|
||||
try:
|
||||
import torch
|
||||
except Exception:
|
||||
return (False, "torch not importable; cannot confirm MPS")
|
||||
try:
|
||||
if torch.backends.mps.is_available():
|
||||
return (True, "")
|
||||
except Exception:
|
||||
# MPS query raised — fall through to the conservative unavailable path.
|
||||
pass
|
||||
return (
|
||||
False,
|
||||
"Apple Silicon detected but torch MPS unavailable; "
|
||||
"reinstall torch with MPS support",
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"DeviceFamily",
|
||||
"HostCaps",
|
||||
"detect_host_caps",
|
||||
"refresh",
|
||||
"mlx_supported",
|
||||
"KERNEL_RISK_MARKER",
|
||||
"DIRECTML_MARKER",
|
||||
]
|
||||
@@ -4,7 +4,7 @@ One pass over everything a working install needs: Python, compute device,
|
||||
ffmpeg, HF token, disk, data-dir permissions, RAM, TTS engines, and (when
|
||||
requested) network reachability of the HuggingFace hub. Surfaced two ways:
|
||||
|
||||
- ``GET /system/diagnose`` (Settings > About -> "Run self-check")
|
||||
- ``GET /system/diagnose`` (Settings > About → "Run self-check")
|
||||
- ``python main.py --diagnose`` for headless installs / issue triage
|
||||
|
||||
Every ``detail``/``hint`` string is passed through ``core.scrub`` before it
|
||||
@@ -194,49 +194,6 @@ def _check_engines() -> dict:
|
||||
return _check("engines", "TTS engines", OK, detail)
|
||||
|
||||
|
||||
def _check_gpu_routing() -> dict:
|
||||
"""Routing verdict for the active TTS engine on THIS host (#21).
|
||||
|
||||
Surfaces a CPU fallback / unavailable-GPU *before* a slow or failed synth —
|
||||
the no-silent-fallback contract. `cpu_only` on a no-GPU machine is the
|
||||
expected normal state and stays OK (never noise-warns)."""
|
||||
try:
|
||||
from services.tts_backend import gpu_routing_verdict
|
||||
v = gpu_routing_verdict()
|
||||
except Exception as e:
|
||||
return _check("gpu_routing", "GPU routing", WARN, f"could not resolve: {e}")
|
||||
|
||||
status = v.get("routing_status")
|
||||
engine = v.get("engine") or "active engine"
|
||||
dev = v.get("effective_device") or "?"
|
||||
reason = v.get("routing_reason")
|
||||
host = v.get("host_family", "cpu")
|
||||
|
||||
if status == "accelerated":
|
||||
if reason: # driver/arch caveat — accelerated but at risk
|
||||
return _check("gpu_routing", "GPU routing", WARN,
|
||||
f"{engine} -> {dev}: {reason}",
|
||||
"The GPU is selected but may fail at kernel launch — "
|
||||
"update drivers / reinstall torch for this GPU arch.")
|
||||
return _check("gpu_routing", "GPU routing", OK, f"{engine} -> {dev} (accelerated)")
|
||||
if status == "cpu_fallback":
|
||||
return _check("gpu_routing", "GPU routing", WARN,
|
||||
f"{engine} runs on CPU: {reason or 'no GPU path for this host'}",
|
||||
"Pick an engine that supports this host's GPU for a big speedup, "
|
||||
"or continue on CPU (slower).")
|
||||
if status == "cpu_only":
|
||||
return _check("gpu_routing", "GPU routing", OK,
|
||||
f"{engine} -> cpu (no accelerator on this host)")
|
||||
if status == "unavailable":
|
||||
return _check("gpu_routing", "GPU routing", FAIL,
|
||||
f"{engine} can't run on this host: {reason or f'needs a GPU; host is {host}'}",
|
||||
"Select an engine with a CPU path in Settings -> Engines.")
|
||||
# status == "none" / unknown — no active engine resolved.
|
||||
return _check("gpu_routing", "GPU routing", WARN,
|
||||
"No active TTS engine resolved for routing.",
|
||||
"Pick an engine in Settings -> Engines.")
|
||||
|
||||
|
||||
_DEEP_TIMEOUT_S = 180
|
||||
|
||||
|
||||
@@ -307,8 +264,6 @@ def _check_network() -> dict:
|
||||
# all model downloads need to get started. urllib honors HTTP(S)_PROXY.
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
if not _HUB_URL.startswith("https://"): # constant today; guard the sink anyway
|
||||
raise ValueError(f"hub URL must be https, got {_HUB_URL!r}")
|
||||
req = urllib.request.Request(_HUB_URL, method="HEAD")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=_HUB_TIMEOUT_S):
|
||||
@@ -341,7 +296,6 @@ def run_diagnostics(include_network: bool = True, deep: bool = False) -> dict:
|
||||
_check_data_dir(),
|
||||
_check_ram(),
|
||||
_check_engines(),
|
||||
_check_gpu_routing(),
|
||||
]
|
||||
if include_network:
|
||||
checks.append(_check_network())
|
||||
|
||||
@@ -111,8 +111,7 @@ def classify_exception(exc: BaseException, trace: str = "") -> str:
|
||||
def _fingerprint(error_class: str, exc: BaseException) -> str:
|
||||
import hashlib
|
||||
raw = f"{error_class}|{type(exc).__name__}|{scrub_text(str(exc))[:200]}"
|
||||
# Dedup key for the journal, not a security boundary.
|
||||
return hashlib.sha1(raw.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:16]
|
||||
return hashlib.sha1(raw.encode("utf-8", "replace")).hexdigest()[:16]
|
||||
|
||||
|
||||
def _persist_locked() -> None:
|
||||
|
||||
+1
-54
@@ -32,16 +32,11 @@ _REDACTED_VALUE = "***REDACTED***"
|
||||
# One-line "what to do" per docs-taxonomy key. Keys mirror error_docs_map's
|
||||
# taxonomy; the docs URL itself stays owned by error_docs_map.
|
||||
_HINTS: dict[str, str] = {
|
||||
"PKG_RESOURCES_MISSING": "Run `uv pip install --reinstall 'setuptools>=75,<80'` in the backend venv (a plain install is skipped when setuptools' metadata is present but its pkg_resources files were removed by antivirus). Restart after.",
|
||||
"PKG_RESOURCES_MISSING": "Install setuptools in the backend environment (provides pkg_resources).",
|
||||
"GATEKEEPER_QUARANTINE": "Clear the macOS quarantine flag (xattr -cr the app), then reopen.",
|
||||
"APPIMAGE_WEBKIT_WHITESCREEN": "Launch with WEBKIT_DISABLE_DMABUF_RENDERER=1 set.",
|
||||
"HF_AUTH_FAILED": "Set a valid HF_TOKEN in Settings → Hugging Face and retry.",
|
||||
"PYANNOTE_LICENSE_REQUIRED": "Accept the pyannote model licenses on Hugging Face, then retry.",
|
||||
"COMPUTE_TYPE_UNSUPPORTED": "Your GPU doesn't support float16 — OmniVoice retried on int8. If transcription still fails, set OMNIVOICE/ASR_COMPUTE_TYPE=int8 or use CPU.",
|
||||
"TRANSFORMERS_IMPORT": "Your transformers install is incomplete. Reinstall it (`uv pip install --reinstall transformers`) or switch ASR to faster-whisper (Settings → Models).",
|
||||
"UNSUPPORTED_VIDEO_URL": "This link isn't a directly downloadable video. Paste a direct video page (e.g. a youtube.com/watch?v=… or douyin.com/video/<id> link), not a share/profile/feed link — or download the file and drop it in directly.",
|
||||
"VIDEO_DOWNLOAD_NETWORK": "The connection to the video server dropped mid-download (often a transient CDN/network blip or a regional rate-limit). Just retry — OmniVoice already cleaned up the partial download. If it keeps failing, check your network/VPN.",
|
||||
"BROKEN_VENV": "The Python backend environment was moved or damaged. OmniVoice rebuilds it automatically on the next launch; if it keeps failing, use Clean & Retry on the setup screen.",
|
||||
}
|
||||
|
||||
|
||||
@@ -60,58 +55,10 @@ def classify(reason: str) -> str:
|
||||
return "APPIMAGE_WEBKIT_WHITESCREEN"
|
||||
if "pyannote" in low or ("gated" in low and "model" in low) or "accept the" in low:
|
||||
return "PYANNOTE_LICENSE_REQUIRED"
|
||||
# ASR robustness (#551 / #549): name the class so the no-segments toast is
|
||||
# actionable. Place before the generic returns so a compute-type/transformers
|
||||
# failure gets its hint rather than falling through to "".
|
||||
if "compute type" in low or "efficient float16" in low:
|
||||
return "COMPUTE_TYPE_UNSUPPORTED"
|
||||
if (
|
||||
"could not import module" in low
|
||||
or "autofeatureextractor" in low
|
||||
# A corrupted/incomplete transformers install: a model load lazily
|
||||
# resolves a module file that's MISSING from site-packages (an
|
||||
# interrupted `uv sync`, antivirus removal, or a partial update), e.g.
|
||||
# `[Errno 2] No such file or directory:
|
||||
# '.../site-packages/transformers/models/qwen3/modeling_qwen3.py'`.
|
||||
# That's a FileNotFoundError, not an ImportError, so the matches above
|
||||
# miss it and the user got a useless "try restarting". Substring-match
|
||||
# the package + the missing-file signal (separately, so it works on both
|
||||
# POSIX `/` and Windows `\` paths).
|
||||
or (
|
||||
("no such file" in low or "errno 2" in low)
|
||||
and "transformers" in low
|
||||
and "site-packages" in low
|
||||
)
|
||||
):
|
||||
return "TRANSFORMERS_IMPORT"
|
||||
if ("huggingface" in low or "hf_token" in low or "401" in low or "unauthorized" in low) and (
|
||||
"token" in low or "auth" in low or "401" in low or "unauthorized" in low
|
||||
):
|
||||
return "HF_AUTH_FAILED"
|
||||
# Video download (#554/#536): a non-downloadable URL shape vs a transient
|
||||
# network drop — both previously surfaced as a bare yt-dlp string with no
|
||||
# next step. UNSUPPORTED first (more specific) so "Unable to download video:
|
||||
# Broken pipe" still classifies as a network blip.
|
||||
if "unsupported url" in low or "no video formats" in low or "is not a valid url" in low:
|
||||
return "UNSUPPORTED_VIDEO_URL"
|
||||
if (
|
||||
"broken pipe" in low
|
||||
or "connection reset" in low
|
||||
or "unable to download video" in low
|
||||
or "remote end closed" in low
|
||||
or "timed out" in low
|
||||
):
|
||||
return "VIDEO_DOWNLOAD_NETWORK"
|
||||
# A relocated/corrupted venv whose interpreter can't bootstrap its stdlib —
|
||||
# the Rust self-heal rebuilds it; this names the class for the toast.
|
||||
if "no module named 'encodings'" in low:
|
||||
return "BROKEN_VENV"
|
||||
# #564: the interpreter starts fine but the backend can't import its OWN
|
||||
# `omnivoice` package (a venv missing the editable install). Same self-heal
|
||||
# class — Clean & Retry / the bootstrap repair rebuilds it. The trailing
|
||||
# quote keeps a legitimately-named `omnivoice_*` helper from matching.
|
||||
if "no module named 'omnivoice'" in low:
|
||||
return "BROKEN_VENV"
|
||||
return ""
|
||||
|
||||
|
||||
|
||||
@@ -1,77 +0,0 @@
|
||||
"""Resolve the project's own ``omnivoice`` package from source when the venv's
|
||||
editable install is missing (#564).
|
||||
|
||||
``omnivoice`` is normally an editable install in the backend venv. An interrupted
|
||||
or offline ``uv sync`` can install dependencies yet never lay the editable record
|
||||
(``_editable_impl_omnivoice.pth``), or an antivirus quarantine can remove it —
|
||||
leaving a venv that starts uvicorn but cannot ``import omnivoice``, so it boots
|
||||
fine and only fails at the first model call (``No module named 'omnivoice'``).
|
||||
|
||||
The desktop layout always copies ``omnivoice/`` next to ``backend/``, so we fall
|
||||
back to importing it from there. The bootstrap now also gates on omnivoice being
|
||||
importable (re-syncing to re-lay the editable install), but this keeps the
|
||||
backend resilient even when that repair hasn't run yet.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
|
||||
def find_omnivoice_source_root(candidates):
|
||||
"""Return the first candidate dir holding ``omnivoice/__init__.py``, else None."""
|
||||
for root in candidates:
|
||||
if root and os.path.isfile(os.path.join(root, "omnivoice", "__init__.py")):
|
||||
return root
|
||||
return None
|
||||
|
||||
|
||||
def _candidate_roots(backend_dir):
|
||||
"""Source roots to probe, most-specific first.
|
||||
|
||||
``OMNIVOICE_PROJECT_ROOT`` lets the launcher point at the staged project dir
|
||||
explicitly; otherwise the desktop layout puts ``omnivoice/`` beside
|
||||
``backend/`` (parent of ``backend_dir``).
|
||||
"""
|
||||
roots = []
|
||||
env = os.environ.get("OMNIVOICE_PROJECT_ROOT")
|
||||
if env:
|
||||
roots.append(env)
|
||||
roots.append(os.path.dirname(os.path.abspath(backend_dir)))
|
||||
return roots
|
||||
|
||||
|
||||
def _already_importable():
|
||||
import importlib.util
|
||||
try:
|
||||
return importlib.util.find_spec("omnivoice") is not None
|
||||
except (ImportError, ValueError):
|
||||
# A half-laid spec (e.g. a stale .pth pointing at a deleted dir) raises
|
||||
# rather than returning None — treat it as "not importable" so we fall
|
||||
# back to the on-disk source.
|
||||
return False
|
||||
|
||||
|
||||
def ensure_omnivoice_importable(backend_dir, logger=None):
|
||||
"""Make ``import omnivoice`` work, falling back to the sibling source tree.
|
||||
|
||||
No-op when the editable/site-packages install already resolves it. Otherwise
|
||||
appends the first source root containing ``omnivoice/`` to ``sys.path``
|
||||
(appended, never inserted, so a real install keeps precedence). Returns the
|
||||
root that was added, or ``None`` if none was needed or found.
|
||||
"""
|
||||
if _already_importable():
|
||||
return None
|
||||
root = find_omnivoice_source_root(_candidate_roots(backend_dir))
|
||||
if root and root not in sys.path:
|
||||
sys.path.append(root)
|
||||
if logger:
|
||||
logger.warning(
|
||||
"omnivoice not importable from the venv (missing/broken editable "
|
||||
"install) — resolving it from source at %s (#564)", root,
|
||||
)
|
||||
elif logger and root is None:
|
||||
logger.error(
|
||||
"omnivoice is not importable and no source tree was found next to "
|
||||
"%s — the install is incomplete; relaunch to let the bootstrap "
|
||||
"repair the venv (#564)", backend_dir,
|
||||
)
|
||||
return root
|
||||
@@ -61,15 +61,9 @@ def seed_sample_project():
|
||||
if count > 0:
|
||||
return # Not first run — skip
|
||||
|
||||
# The demo clip is committed at backend/assets/samples/demo_voice.wav and
|
||||
# bundled with the app (#621). If it's somehow absent (e.g. a partial
|
||||
# checkout), skip the seed gracefully rather than seeding a profile that
|
||||
# points at a missing file — run scripts/build_demos.sh to regenerate it.
|
||||
# Check if demo audio exists
|
||||
if not os.path.isfile(_DEMO_AUDIO):
|
||||
logger.warning(
|
||||
"Demo audio not found at %s — skipping onboarding seed "
|
||||
"(regenerate with scripts/build_demos.sh)", _DEMO_AUDIO,
|
||||
)
|
||||
logger.warning("Demo audio not found at %s — skipping onboarding seed", _DEMO_AUDIO)
|
||||
return
|
||||
|
||||
# Copy demo audio to voices directory
|
||||
|
||||
+6
-36
@@ -36,39 +36,15 @@ _TOKEN_PATTERNS = (
|
||||
re.compile(r"github_pat_[A-Za-z0-9_]{20,}"), # GitHub fine-grained PAT
|
||||
re.compile(r"gh[pousr]_[A-Za-z0-9]{30,}"), # GitHub classic tokens
|
||||
re.compile(r"sk-[A-Za-z0-9_\-]{20,}"), # OpenAI-style API keys
|
||||
# A backend error can carry a secret from *any* provider (the LLM-providers
|
||||
# feature ships a dozen), so match the common credential shapes too, not
|
||||
# just the four vendors above — a leaked key in a public issue is real harm.
|
||||
re.compile(r"eyJ[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{6,}"), # JWT (Bearer)
|
||||
re.compile(r"AIza[0-9A-Za-z_\-]{35}"), # Google API key
|
||||
re.compile(r"xox[baprs]-[A-Za-z0-9\-]{10,}"), # Slack token
|
||||
re.compile(r"AKIA[0-9A-Z]{16}"), # AWS access key id
|
||||
re.compile(r"(?i)bearer\s+[A-Za-z0-9._\-]{16,}"), # opaque bearer tokens
|
||||
)
|
||||
|
||||
# Secrets carried in a URL query string (`?token=…`, `&api_key=…`). Redact the
|
||||
# VALUE while keeping the param name + separator so the URL stays legible. Bare
|
||||
# `key=` is intentionally excluded — too common in non-secret text; shaped keys
|
||||
# are already caught above and named env vars by the sweep below.
|
||||
_URL_SECRET_RE = re.compile(
|
||||
r"((?:access[_-]?token|api[_-]?key|apikey|auth[_-]?token|token|secret|password|passwd|pwd)=)"
|
||||
r"([^&\s\"'#]{6,})",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# Home-directory shapes for all three supported platforms. Matched
|
||||
# pattern-wise (not just this machine's $HOME) so paths quoted from a
|
||||
# user's pasted log on another OS get cleaned too.
|
||||
# IGNORECASE because Windows is case-insensitive and tools routinely emit the
|
||||
# lowercase `c:\users\<name>` form, which the CLAUDE.md redaction spec still
|
||||
# requires to become `~`. `Users`/`users`, `Home`/`home` all match.
|
||||
_HOME_PATTERNS = (
|
||||
# Windows-with-forward-slashes must run BEFORE the bare macOS shape, or
|
||||
# `/Users/<name>` inside `C:/Users/<name>` gets eaten first, leaving `C:~`.
|
||||
re.compile(r"[A-Za-z]:/Users/[^/\s\"']+", re.IGNORECASE), # Windows, forward slashes
|
||||
re.compile(r"/Users/[^/\s\"']+", re.IGNORECASE), # macOS
|
||||
re.compile(r"/home/[^/\s\"']+", re.IGNORECASE), # Linux
|
||||
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+", re.IGNORECASE), # Windows, backslashes
|
||||
re.compile(r"/Users/[^/\s\"']+"), # macOS
|
||||
re.compile(r"/home/[^/\s\"']+"), # Linux
|
||||
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+"), # Windows
|
||||
)
|
||||
|
||||
# Values shorter than this are too entropy-poor to be real secrets and too
|
||||
@@ -106,25 +82,19 @@ def scrub_text(text: str | None) -> str:
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 2. Credential-shaped substrings + URL query secrets.
|
||||
# 2. Credential-shaped substrings.
|
||||
for pat in _TOKEN_PATTERNS:
|
||||
try:
|
||||
s = pat.sub(REDACTED, s)
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
s = _URL_SECRET_RE.sub(lambda m: m.group(1) + REDACTED, s)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 3. This process's real home dir (covers symlinked/nonstandard homes
|
||||
# the generic patterns miss), then the per-OS shapes. Boundary-aware so
|
||||
# a home of `/Users/john` doesn't rewrite `/Users/johnny` to `~ny`
|
||||
# (leaking the fragment + mangling the path).
|
||||
# the generic patterns miss), then the per-OS shapes.
|
||||
try:
|
||||
home = os.path.expanduser("~")
|
||||
if home and home not in ("/", "~"):
|
||||
s = re.sub(re.escape(home) + r"(?=[/\\\s\"']|$)", "~", s)
|
||||
s = s.replace(home, "~")
|
||||
except Exception:
|
||||
pass
|
||||
for pat in _HOME_PATTERNS:
|
||||
|
||||
@@ -82,28 +82,3 @@ def unset_user_env(key: str, path: Optional[str] = None) -> None:
|
||||
prefix = f"{key}="
|
||||
lines = [ln for ln in _read_lines(path) if not ln.startswith(prefix)]
|
||||
_write_lines(path, lines)
|
||||
|
||||
|
||||
def load_into_environ(path: Optional[str] = None) -> bool:
|
||||
"""Load the durable per-user env file into ``os.environ``, **overriding**
|
||||
any value a launcher already injected. Returns True if a file was loaded.
|
||||
|
||||
This file is the in-app Settings source of truth. The desktop launcher
|
||||
(Tauri) injects defaults like ``OMNIVOICE_CACHE_DIR`` (and ``HF_ENDPOINT``)
|
||||
from its *own* config into the backend's environment *before* startup, so
|
||||
loading this file with ``override=False`` meant a models directory the user
|
||||
changed in Settings was silently ignored on every launch — the effective
|
||||
location stayed on the old one no matter how many restarts (#480). Both keys
|
||||
this file can hold are the user's explicit Settings choice and should beat
|
||||
the launcher's default, so we override. Restores this file's documented
|
||||
"values written here take effect on the next backend launch" contract.
|
||||
"""
|
||||
path = path or os.environ.get("OMNIVOICE_ENV_FILE") or USER_ENV_PATH
|
||||
if not os.path.isfile(path):
|
||||
return False
|
||||
try:
|
||||
import dotenv
|
||||
except ImportError:
|
||||
return False
|
||||
dotenv.load_dotenv(path, override=True)
|
||||
return True
|
||||
|
||||
+4
-33
@@ -2,43 +2,14 @@
|
||||
|
||||
Read from the installed package metadata (driven by ``pyproject.toml``) so the
|
||||
FastAPI/API version and exported-bundle metadata never drift to a stale literal
|
||||
— the prior "0.4.0" / "0.2.7" bug, and the v0.3.6 desktop build that reported
|
||||
"0.3.5" because the *frozen* backend couldn't read its own metadata.
|
||||
|
||||
Resolution order:
|
||||
1. installed package metadata — correct in any ``uv sync``'d env and, thanks
|
||||
to ``copy_metadata('omnivoice')`` in ``backend.spec``, in the frozen build;
|
||||
2. ``pyproject.toml`` walked up from this file — correct for a raw source
|
||||
checkout that was never installed;
|
||||
3. ``_FALLBACK_VERSION`` — a last resort, kept in lockstep with the four
|
||||
version files by ``tests/test_app_version.py`` so it can never silently
|
||||
drift again.
|
||||
again (the prior "0.4.0" / "0.2.7" bug). Falls back to a literal only when
|
||||
running from a raw source checkout that was never ``uv sync``'d.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from importlib.metadata import PackageNotFoundError, version
|
||||
from pathlib import Path
|
||||
|
||||
# Last-resort literal. Guarded by
|
||||
# tests/test_app_version.py::test_all_version_files_in_lockstep and bumped by
|
||||
# release.yml's version-bump job, so it stays equal to
|
||||
# pyproject/tauri.conf/Cargo/package.json.
|
||||
_FALLBACK_VERSION = "0.3.8"
|
||||
|
||||
|
||||
def _fallback_version() -> str:
|
||||
"""Version for contexts where package metadata is unavailable."""
|
||||
for parent in Path(__file__).resolve().parents:
|
||||
pyproject = parent / "pyproject.toml"
|
||||
if pyproject.is_file():
|
||||
match = re.search(r'(?m)^version\s*=\s*"([^"]+)"', pyproject.read_text())
|
||||
if match:
|
||||
return match.group(1)
|
||||
return _FALLBACK_VERSION
|
||||
|
||||
|
||||
try:
|
||||
APP_VERSION = version("omnivoice")
|
||||
except PackageNotFoundError: # frozen build w/o metadata, or non-installed checkout
|
||||
APP_VERSION = _fallback_version()
|
||||
except PackageNotFoundError: # non-installed source checkout
|
||||
APP_VERSION = "0.3.5"
|
||||
|
||||
@@ -1,148 +0,0 @@
|
||||
"""Crash-isolated faster-whisper ASR sidecar (Wave 4.2 / Spec 7).
|
||||
|
||||
Runs faster-whisper in a child process so a CTranslate2 GPU-teardown segfault
|
||||
becomes a failed job, not a dead backend. Speaks the SubprocessBackend wire
|
||||
protocol (length-prefixed JSON over stdin/stdout):
|
||||
|
||||
on start → {"op":"ready","engine":"faster-whisper-isolated"}
|
||||
{"op":"ping"} → {"op":"pong"}
|
||||
{"op":"transcribe","audio_path":...,"word_timestamps":bool}
|
||||
→ {"op":"segments","result":{"segments":[...],"language":...}}
|
||||
{"op":"shutdown"} → exit 0
|
||||
error → {"op":"error","message":...}
|
||||
|
||||
Runs under the PARENT venv (faster-whisper is already a dependency) — only the
|
||||
process boundary is new. torch/CTranslate2 import lazily inside transcribe so
|
||||
the ready handshake fits the spawn timeout.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
MAX_FRAME_BYTES = 64 * 1024 * 1024
|
||||
_model = None
|
||||
|
||||
|
||||
def _send(stream, obj):
|
||||
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
|
||||
stream.write(struct.pack("!I", len(body)))
|
||||
stream.write(body)
|
||||
stream.flush()
|
||||
|
||||
|
||||
def _recv(stream):
|
||||
header = stream.read(4)
|
||||
if len(header) < 4:
|
||||
return None
|
||||
(n,) = struct.unpack("!I", header)
|
||||
if n > MAX_FRAME_BYTES:
|
||||
raise IOError(f"frame too large: {n}")
|
||||
body = bytearray()
|
||||
while len(body) < n:
|
||||
chunk = stream.read(n - len(body))
|
||||
if not chunk:
|
||||
raise IOError("short read")
|
||||
body.extend(chunk)
|
||||
return json.loads(bytes(body).decode("utf-8"))
|
||||
|
||||
|
||||
# NOTE: keep this compute_type fallback in lockstep with
|
||||
# services/asr_backend.py:_compute_type_candidates / _is_compute_type_error.
|
||||
# This sidecar runs in a child proc with a clean import path, so we duplicate a
|
||||
# tiny copy rather than cross-importing the heavy services package (#551).
|
||||
def _ct_candidates(device):
|
||||
override = os.environ.get("ASR_COMPUTE_TYPE")
|
||||
if override:
|
||||
return [override]
|
||||
return ["float16", "int8_float16", "int8"] if device == "cuda" else ["int8", "float32"]
|
||||
|
||||
|
||||
def _is_ct_error(msg):
|
||||
low = msg.lower()
|
||||
return "compute type" in low or "efficient float16" in low
|
||||
|
||||
|
||||
def _get_model():
|
||||
global _model
|
||||
if _model is None:
|
||||
from faster_whisper import WhisperModel
|
||||
name = os.environ.get("ASR_MODEL_FW", "large-v3")
|
||||
try:
|
||||
import torch
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
except Exception:
|
||||
device = "cpu"
|
||||
# Degrade fp16 → int8 rather than crash on GPUs without efficient fp16
|
||||
# (older Maxwell/Pascal, GTX 16xx, CTranslate2/cuDNN mismatch) (#551).
|
||||
last_err = None
|
||||
for compute in _ct_candidates(device):
|
||||
try:
|
||||
_model = WhisperModel(name, device=device, compute_type=compute)
|
||||
break
|
||||
except (ValueError, RuntimeError) as e:
|
||||
last_err = e
|
||||
if _is_ct_error(str(e)):
|
||||
continue
|
||||
raise
|
||||
else:
|
||||
raise last_err
|
||||
return _model
|
||||
|
||||
|
||||
def _transcribe(audio_path, word_timestamps):
|
||||
model = _get_model()
|
||||
segments, info = model.transcribe(audio_path, word_timestamps=word_timestamps)
|
||||
out = []
|
||||
for s in segments:
|
||||
seg = {"start": float(s.start), "end": float(s.end), "text": s.text}
|
||||
if word_timestamps and getattr(s, "words", None):
|
||||
seg["words"] = [
|
||||
{"word": w.word, "start": float(w.start), "end": float(w.end),
|
||||
"probability": float(getattr(w, "probability", 0.0))}
|
||||
for w in s.words
|
||||
]
|
||||
out.append(seg)
|
||||
return {
|
||||
"segments": out,
|
||||
"text": " ".join(s["text"].strip() for s in out).strip(),
|
||||
"language": getattr(info, "language", "unknown"),
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
stdin, stdout = sys.stdin.buffer, sys.stdout.buffer
|
||||
_send(stdout, {"op": "ready", "engine": "faster-whisper-isolated"})
|
||||
while True:
|
||||
try:
|
||||
msg = _recv(stdin)
|
||||
except Exception as exc:
|
||||
_send(stdout, {"op": "error", "stage": "recv", "message": f"{type(exc).__name__}: {exc}"})
|
||||
return 1
|
||||
if msg is None:
|
||||
return 0
|
||||
op = msg.get("op")
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong"})
|
||||
elif op == "transcribe":
|
||||
result = _transcribe(msg.get("audio_path"), bool(msg.get("word_timestamps", True)))
|
||||
_send(stdout, {"op": "segments", "result": result})
|
||||
elif op == "shutdown":
|
||||
return 0
|
||||
else:
|
||||
_send(stdout, {"op": "error", "stage": "dispatch", "message": f"unknown op: {op!r}"})
|
||||
except Exception as exc:
|
||||
_send(stdout, {
|
||||
"op": "error", "stage": "handler",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -101,12 +101,6 @@ def main() -> int:
|
||||
return 0
|
||||
|
||||
op = msg.get("op")
|
||||
# Wave 4.2: deterministic "crash mid-transcription" hook — exit BEFORE
|
||||
# sending any reply so the parent's blocking recv sees a dead pipe
|
||||
# (reply=None). The crash-after-one hook below replies first, so it
|
||||
# can't deterministically exercise the no-reply path.
|
||||
if op == "transcribe" and os.environ.get("OMNIVOICE_ECHO_CRASH_NO_REPLY") == "1":
|
||||
os._exit(1)
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong"})
|
||||
@@ -119,19 +113,6 @@ def main() -> int:
|
||||
"sample_rate": sr,
|
||||
"n_samples": n_samples,
|
||||
})
|
||||
elif op == "transcribe":
|
||||
# Wave 4.2: echo ASR op — a canned segments result so the
|
||||
# SubprocessASRBackend round-trip + respawn path is testable
|
||||
# without a real ASR engine.
|
||||
_send(stdout, {
|
||||
"op": "segments",
|
||||
"result": {
|
||||
"segments": [{"start": 0.0, "end": 1.0,
|
||||
"text": f"echo:{msg.get('audio_path', '')}"}],
|
||||
"text": f"echo:{msg.get('audio_path', '')}",
|
||||
"language": "en",
|
||||
},
|
||||
})
|
||||
elif op == "shutdown":
|
||||
return 0
|
||||
elif op == "probe_env" and test_mode:
|
||||
|
||||
@@ -1,187 +0,0 @@
|
||||
"""dots.tts sidecar package (issue #498).
|
||||
|
||||
dots.tts is rednote-hilab's 2B fully-continuous autoregressive TTS — widely
|
||||
cited as among the strongest open zero-shot voice-cloning models. 24
|
||||
languages, 48 kHz output, Apache-2.0 (code + checkpoints).
|
||||
|
||||
It runs in its own subprocess **and its own venv**, isolated from the
|
||||
OmniVoice parent, for the same ``transformers`` reason as IndexTTS and
|
||||
MOSS-TTS-v1.5: dots.tts pins ``transformers==4.57.0`` (verified against
|
||||
``constraints/recommended.txt``), while OmniVoice pins
|
||||
``transformers>=5.3.0``. The two cannot share one interpreter.
|
||||
|
||||
Cross-platform honesty (the strict default-parity rule): dots.tts's
|
||||
upstream package declares **Linux + macOS** classifiers only — **no
|
||||
Windows** — and its device code is **CUDA-or-CPU with no MPS branch**
|
||||
(verified in ``runtime.py``). So:
|
||||
|
||||
* It is **opt-in** (engine-picker selection + a user-provided clone),
|
||||
never a default — so it never becomes a broken default on any platform.
|
||||
* ``is_available()`` returns ``False`` with a clear reason on **Windows**
|
||||
rather than offering an engine that can't run there. Windows users are
|
||||
pointed at WSL2 / a Linux or macOS host.
|
||||
* ``gpu_compat = ("cuda", "cpu")`` — no MPS claim. On Apple Silicon the
|
||||
upstream package runs on CPU (slow but correct); the faster MLX path is
|
||||
a community port we deliberately don't auto-wire here.
|
||||
|
||||
Three public entry points: ``DotsTTSBackend`` (this module), ``main.py``
|
||||
(sidecar, runs under dots.tts's ``transformers==4.57`` venv — never imported
|
||||
by the parent), and ``bootstrap.py`` (venv probe + lazy bootstrap).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sys
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from services.subprocess_backend import SubprocessBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import torch # noqa: F401
|
||||
|
||||
logger = logging.getLogger("omnivoice.dots_tts")
|
||||
|
||||
|
||||
class DotsTTSBackend(SubprocessBackend):
|
||||
"""dots.tts (rednote-hilab) — 2B, 24 langs, zero-shot clone, CUDA/CPU.
|
||||
|
||||
Runs in a long-lived sidecar over length-prefixed JSON-over-stdio in a
|
||||
dedicated venv (``transformers==4.57.0``). First synthesize cold-loads
|
||||
the ~9 GB checkpoint (bf16 on CUDA); subsequent calls reuse the process.
|
||||
|
||||
Installation (OmniVoice prefers a user's existing ``${DIR}/.venv``)::
|
||||
|
||||
git clone https://github.com/rednote-hilab/dots.tts.git
|
||||
cd dots.tts
|
||||
uv venv && uv pip install -e . -c constraints/recommended.txt
|
||||
|
||||
Set ``OMNIVOICE_DOTS_TTS_DIR`` to the clone root. OmniVoice creates
|
||||
``backend/engines/dots_tts/.venv`` lazily on first launch if no venv
|
||||
exists yet; the user's existing ``${DIR}/.venv`` is preferred if present.
|
||||
|
||||
Best cloning quality uses the ``dots.tts-soar`` checkpoint (the default)
|
||||
and BOTH a reference clip and its exact transcript (continuation
|
||||
cloning). License: Apache-2.0.
|
||||
"""
|
||||
|
||||
id = "dots-tts"
|
||||
display_name = (
|
||||
"dots.tts (2B, 24 langs, zero-shot clone, CUDA/CPU, 48 kHz, Apache-2.0)"
|
||||
)
|
||||
supports_voice_design = False # requires ref audio for timbre cloning
|
||||
# dots.tts emits 48 kHz (verified via checkpoint vocoder.sample_rate).
|
||||
_DEFAULT_SAMPLE_RATE = 48000
|
||||
# CUDA + CPU only; no MPS branch in upstream runtime.py.
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
# ── availability ───────────────────────────────────────────────────────
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
# Cross-platform parity: dots.tts upstream is Linux/macOS-only (no
|
||||
# Windows classifier, no Windows install path). Refuse cleanly on
|
||||
# Windows instead of advertising an engine that can't run.
|
||||
if sys.platform == "win32":
|
||||
return False, (
|
||||
"dots.tts is not supported on Windows — upstream targets "
|
||||
"Linux and macOS only. Run OmniVoice under WSL2, or use a "
|
||||
"Linux/macOS host. See docs/engines/dots-tts.md."
|
||||
)
|
||||
|
||||
# Do NOT import dots_tts here: it pins transformers==4.57, which can't
|
||||
# coexist with the parent's transformers>=5.3 in one interpreter —
|
||||
# the reason for the subprocess isolation. Verify the venv on disk
|
||||
# only; a real health-check is gated on the user's "Test engine"
|
||||
# action in Settings.
|
||||
from engines.dots_tts.bootstrap import (
|
||||
DOTS_TTS_SIDECAR_SCRIPT,
|
||||
is_dots_tts_installed,
|
||||
)
|
||||
if not is_dots_tts_installed():
|
||||
return False, (
|
||||
"dots.tts venv not found. Set OMNIVOICE_DOTS_TTS_DIR to your "
|
||||
"dots.tts clone (the directory containing pyproject.toml) and "
|
||||
"restart OmniVoice. CUDA or CPU only (no MPS). See "
|
||||
"docs/engines/dots-tts.md for the full install walk-through."
|
||||
)
|
||||
if not DOTS_TTS_SIDECAR_SCRIPT.exists():
|
||||
return False, (
|
||||
"dots.tts sidecar script missing at "
|
||||
f"{DOTS_TTS_SIDECAR_SCRIPT} — reinstall OmniVoice."
|
||||
)
|
||||
return True, "ok (CUDA when present, else CPU)"
|
||||
|
||||
@classmethod
|
||||
def venv_python(cls):
|
||||
from engines.dots_tts.bootstrap import resolve_dots_tts_venv
|
||||
return resolve_dots_tts_venv()
|
||||
|
||||
@classmethod
|
||||
def sidecar_script(cls):
|
||||
from engines.dots_tts.bootstrap import DOTS_TTS_SIDECAR_SCRIPT
|
||||
return DOTS_TTS_SIDECAR_SCRIPT
|
||||
|
||||
# ── TTSBackend protocol ────────────────────────────────────────────────
|
||||
|
||||
@property
|
||||
def sample_rate(self) -> int:
|
||||
return self._DEFAULT_SAMPLE_RATE
|
||||
|
||||
@property
|
||||
def supported_languages(self) -> list[str]:
|
||||
# 24 languages with auto-detect; expose "multi" on the protocol
|
||||
# surface and translate the caller's language at synthesize time.
|
||||
return ["multi"]
|
||||
|
||||
# ── generate (parent-side arbitration) ─────────────────────────────────
|
||||
|
||||
def generate(self, text: str, **kw) -> "torch.Tensor":
|
||||
"""Synthesize one utterance through the dots.tts sidecar.
|
||||
|
||||
kwargs honored:
|
||||
* ``ref_audio`` — reference clip path → ``prompt_audio_path``
|
||||
(zero-shot cloning). Optional.
|
||||
* ``ref_text`` — the reference transcript → ``prompt_text``.
|
||||
Best cloning fidelity ("continuation"). Upstream
|
||||
REQUIRES ``prompt_audio_path`` when ``prompt_text``
|
||||
is set, so we drop a stray ref_text with no
|
||||
ref_audio rather than let the sidecar raise.
|
||||
* ``language`` — ISO code / name / None (auto-detect).
|
||||
* ``num_step`` — flow-matching steps → ``num_steps`` (default 10;
|
||||
use 4 for the ``dots.tts-mf`` checkpoint).
|
||||
* ``guidance_scale`` — CFG (default 1.2; >2 amplifies energy).
|
||||
|
||||
Returns a tensor of shape (1, n_samples) at :attr:`sample_rate`.
|
||||
"""
|
||||
forwarded: dict = {}
|
||||
|
||||
ref_audio = kw.get("ref_audio")
|
||||
if ref_audio:
|
||||
forwarded["ref_audio"] = ref_audio
|
||||
ref_text = kw.get("ref_text")
|
||||
if ref_text:
|
||||
# continuation cloning — only valid alongside ref_audio.
|
||||
forwarded["ref_text"] = ref_text
|
||||
elif kw.get("ref_text"):
|
||||
logger.info(
|
||||
"dots-tts: ref_text supplied without ref_audio; ignoring "
|
||||
"(upstream requires prompt_audio_path when prompt_text is set)."
|
||||
)
|
||||
|
||||
language = kw.get("language")
|
||||
if language:
|
||||
forwarded["language"] = str(language)
|
||||
|
||||
# OmniVoice's generic num_step default is 16; dots.tts's own default
|
||||
# is 10. Honor an explicit value, else use the dots-appropriate 10.
|
||||
num_step = kw.get("num_step")
|
||||
forwarded["num_steps"] = int(num_step) if num_step is not None else 10
|
||||
|
||||
# dots.tts's own CFG default is 1.2 (the generic 2.0 over-energises).
|
||||
guidance = kw.get("guidance_scale")
|
||||
forwarded["guidance_scale"] = float(guidance) if guidance is not None else 1.2
|
||||
|
||||
return super().generate(text, **forwarded)
|
||||
|
||||
|
||||
__all__ = ["DotsTTSBackend"]
|
||||
@@ -1,226 +0,0 @@
|
||||
"""dots.tts venv probe + lazy bootstrap (issue #498).
|
||||
|
||||
Resolves which Python interpreter runs the dots.tts sidecar. Mirrors
|
||||
``engines.indextts.bootstrap`` / ``engines.moss_tts_v15.bootstrap`` because
|
||||
dots.tts has the same shape of problem: a hard ``transformers==4.57.0`` pin
|
||||
that conflicts with the parent's ``transformers>=5.3`` — so it runs in its
|
||||
own venv.
|
||||
|
||||
Probe order (priority — existing power-user installs win, zero migration):
|
||||
|
||||
1. ``${OMNIVOICE_DOTS_TTS_DIR}/.venv/`` — the user's clone-level venv.
|
||||
2. ``backend/engines/dots_tts/.venv/`` — this package's own venv.
|
||||
3. Bootstrap: ``uv venv`` then ``uv pip install -e <clone> -c
|
||||
<clone>/constraints/recommended.txt`` (the upstream-pinned stack:
|
||||
torch==2.8.0, transformers==4.57.0, …).
|
||||
|
||||
Caching: memoised after first success. Tests reset via :func:`invalidate`.
|
||||
|
||||
Security: same posture as IndexTTS — bootstrap never touches HF_TOKEN; the
|
||||
sidecar's stderr is redacted by the parent's ``HFTokenRedactor``; the
|
||||
editable install comes from a user-controlled clone they already trust.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger("omnivoice.dots_tts.bootstrap")
|
||||
|
||||
#: Absolute path to the sidecar entrypoint.
|
||||
DOTS_TTS_SIDECAR_SCRIPT: Path = Path(__file__).parent / "main.py"
|
||||
|
||||
#: This package's owned venv (Probe 2).
|
||||
_ENGINES_VENV_DIR: Path = Path(__file__).parent / ".venv"
|
||||
|
||||
#: Env var pointing at the user's dots.tts clone root.
|
||||
_CLONE_DIR_ENV: str = "OMNIVOICE_DOTS_TTS_DIR"
|
||||
|
||||
#: Per-process resolution cache. Cleared by :func:`invalidate` for tests.
|
||||
_resolved_python: Optional[Path] = None
|
||||
|
||||
_IMPORT_PROBE_TIMEOUT_S = 15
|
||||
_UV_VENV_TIMEOUT_S = 120
|
||||
_UV_PIP_INSTALL_TIMEOUT_S = 1800
|
||||
|
||||
|
||||
# ── public API ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def invalidate() -> None:
|
||||
"""Clear the resolved-python cache. Tests call this between scenarios."""
|
||||
global _resolved_python
|
||||
_resolved_python = None
|
||||
|
||||
|
||||
def is_dots_tts_installed() -> bool:
|
||||
"""Cheap file-existence check for a usable dots.tts venv. Does NOT spawn
|
||||
the venv Python — that's saved for :func:`resolve_dots_tts_venv`."""
|
||||
for cand in _probe_paths():
|
||||
if cand.is_file():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def resolve_dots_tts_venv() -> Path:
|
||||
"""Resolve the sidecar's Python interpreter (probe order in the module
|
||||
docstring). Memoised. Raises :exc:`RuntimeError` if none can be located
|
||||
and the bootstrap path is unavailable."""
|
||||
global _resolved_python
|
||||
if _resolved_python is not None:
|
||||
return _resolved_python
|
||||
|
||||
clone_dir = os.environ.get(_CLONE_DIR_ENV)
|
||||
|
||||
# Probe 1 — user's clone-level venv.
|
||||
if clone_dir:
|
||||
cand = _venv_python_path(Path(clone_dir) / ".venv")
|
||||
if cand.is_file() and _venv_can_import_dots(cand):
|
||||
logger.info(
|
||||
"dots.tts venv resolved from %s: %s", _CLONE_DIR_ENV, cand,
|
||||
)
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
# Probe 2 — this package's own venv.
|
||||
cand = _venv_python_path(_ENGINES_VENV_DIR)
|
||||
if cand.is_file() and _venv_can_import_dots(cand):
|
||||
logger.info("dots.tts venv resolved from engines path: %s", cand)
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
# Probe 3 — bootstrap.
|
||||
if not clone_dir:
|
||||
raise RuntimeError(
|
||||
"dots.tts is not installed. Set the "
|
||||
f"{_CLONE_DIR_ENV} environment variable to your dots.tts clone "
|
||||
"(the directory that contains pyproject.toml and constraints/), "
|
||||
"then restart OmniVoice. See docs/engines/dots-tts.md for the "
|
||||
"full install walk-through."
|
||||
)
|
||||
|
||||
cand = _bootstrap_engines_venv(Path(clone_dir))
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
|
||||
# ── internals ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _venv_python_path(venv_dir: Path) -> Path:
|
||||
if sys.platform == "win32":
|
||||
return venv_dir / "Scripts" / "python.exe"
|
||||
return venv_dir / "bin" / "python"
|
||||
|
||||
|
||||
def _probe_paths() -> list[Path]:
|
||||
out: list[Path] = []
|
||||
clone_dir = os.environ.get(_CLONE_DIR_ENV)
|
||||
if clone_dir:
|
||||
out.append(_venv_python_path(Path(clone_dir) / ".venv"))
|
||||
out.append(_venv_python_path(_ENGINES_VENV_DIR))
|
||||
return out
|
||||
|
||||
|
||||
def _venv_can_import_dots(python_path: Path) -> bool:
|
||||
"""Spawn the candidate python and verify ``import dots_tts.runtime`` works.
|
||||
Bounded by ``_IMPORT_PROBE_TIMEOUT_S``. False on any failure."""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[str(python_path), "-c", "import dots_tts.runtime"],
|
||||
capture_output=True,
|
||||
timeout=_IMPORT_PROBE_TIMEOUT_S,
|
||||
)
|
||||
except (subprocess.TimeoutExpired, OSError) as exc:
|
||||
logger.debug("dots.tts import probe failed for %s: %s", python_path, exc)
|
||||
return False
|
||||
if proc.returncode != 0:
|
||||
logger.debug(
|
||||
"dots.tts import probe non-zero for %s: %s",
|
||||
python_path,
|
||||
proc.stderr.decode("utf-8", errors="replace")[:200],
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _locate_uv() -> Optional[str]:
|
||||
bundled = os.environ.get("OMNIVOICE_BUNDLED_UV")
|
||||
if bundled and Path(bundled).is_file():
|
||||
return bundled
|
||||
return shutil.which("uv")
|
||||
|
||||
|
||||
def _bootstrap_engines_venv(clone_dir: Path) -> Path:
|
||||
"""Create engines/dots_tts/.venv and editable-install the user's clone
|
||||
with the upstream constraints file."""
|
||||
uv = _locate_uv()
|
||||
if not uv:
|
||||
raise RuntimeError(
|
||||
"uv is required to bootstrap the dots.tts venv but was not found "
|
||||
"on PATH (and OMNIVOICE_BUNDLED_UV was not set). Install uv from "
|
||||
"https://docs.astral.sh/uv/ and re-launch OmniVoice, or set "
|
||||
"OMNIVOICE_BUNDLED_UV to the absolute path of a uv binary."
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Bootstrapping dots.tts venv at %s from %s (this can take several "
|
||||
"minutes on first launch)", _ENGINES_VENV_DIR, clone_dir,
|
||||
)
|
||||
|
||||
try:
|
||||
subprocess.run(
|
||||
[uv, "venv", str(_ENGINES_VENV_DIR)],
|
||||
check=True, timeout=_UV_VENV_TIMEOUT_S, capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
f"uv venv failed for dots.tts bootstrap at {_ENGINES_VENV_DIR}: "
|
||||
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
|
||||
) from exc
|
||||
|
||||
python_path = _venv_python_path(_ENGINES_VENV_DIR)
|
||||
install_cmd = [
|
||||
uv, "pip", "install",
|
||||
"--python", str(python_path),
|
||||
"-e", str(clone_dir),
|
||||
]
|
||||
# Apply the upstream pin set when it ships with the clone.
|
||||
constraints = clone_dir / "constraints" / "recommended.txt"
|
||||
if constraints.is_file():
|
||||
install_cmd += ["-c", str(constraints)]
|
||||
try:
|
||||
subprocess.run(
|
||||
install_cmd, check=True,
|
||||
timeout=_UV_PIP_INSTALL_TIMEOUT_S, capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
"uv pip install -e failed during dots.tts bootstrap "
|
||||
f"({clone_dir}): "
|
||||
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}. "
|
||||
"See docs/engines/dots-tts.md."
|
||||
) from exc
|
||||
|
||||
if not _venv_can_import_dots(python_path):
|
||||
raise RuntimeError(
|
||||
"dots.tts bootstrap completed but `import dots_tts.runtime` still "
|
||||
f"fails from {python_path}. Verify that {clone_dir} is a valid "
|
||||
"dots.tts clone. See docs/engines/dots-tts.md."
|
||||
)
|
||||
|
||||
logger.info("dots.tts venv bootstrap successful: %s", python_path)
|
||||
return python_path
|
||||
|
||||
|
||||
__all__ = [
|
||||
"DOTS_TTS_SIDECAR_SCRIPT",
|
||||
"invalidate",
|
||||
"is_dots_tts_installed",
|
||||
"resolve_dots_tts_venv",
|
||||
]
|
||||
@@ -1,255 +0,0 @@
|
||||
"""dots.tts sidecar entry point (issue #498).
|
||||
|
||||
Runs inside ``engines/dots_tts/.venv`` (or the user's existing
|
||||
``${OMNIVOICE_DOTS_TTS_DIR}/.venv``) with ``transformers==4.57.0``, isolated
|
||||
from the OmniVoice parent (``transformers>=5.3``). Same isolation rationale
|
||||
as the IndexTTS / MOSS-TTS-v1.5 sidecars.
|
||||
|
||||
Stdlib-only at import time; ``dots_tts`` + torch are imported lazily on the
|
||||
first synthesize op so the ``ready`` frame fits inside the parent's 30 s
|
||||
spawn handshake even on a cold filesystem.
|
||||
|
||||
Wire protocol — length-prefixed JSON over stdin/stdout, byte-identical to
|
||||
``backend/services/subprocess_backend.py``::
|
||||
|
||||
[ 4-byte big-endian uint32 length ][ N bytes UTF-8 JSON ]
|
||||
|
||||
Op flow:
|
||||
1. Sidecar -> parent: {"op": "ready", "engine": "dots-tts",
|
||||
"sample_rate": 48000}
|
||||
2. parent -> sidecar: {"op": "ping"} -> {"op": "pong", "vram_mb": N}
|
||||
3. parent -> sidecar: {"op": "synthesize", "text": "...",
|
||||
"ref_audio": "/path/ref.wav",
|
||||
"ref_text": "transcript", "language": "EN",
|
||||
"num_steps": 10, "guidance_scale": 1.2}
|
||||
-> {"op": "progress", ...} (cold load) then
|
||||
-> {"op": "audio", "audio_pcm_b64": "...", "sample_rate": 48000,
|
||||
"n_samples": N}
|
||||
4. parent -> sidecar: {"op": "shutdown"} -> exit 0
|
||||
|
||||
Restrictions: NO imports from OmniVoice parent code (different venv). NO
|
||||
logging of ``os.environ`` contents. Single-frame DoS cap matches the
|
||||
parent's ``MAX_FRAME_BYTES``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
|
||||
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
|
||||
MAX_FRAME_BYTES = 64 * 1024 * 1024
|
||||
|
||||
#: dots.tts emits 48 kHz (checkpoint vocoder.sample_rate). Advertised in the
|
||||
#: ready frame; the real value is re-read from each generate() result.
|
||||
DOTS_SAMPLE_RATE = 48000
|
||||
|
||||
#: Default checkpoint. ``-soar`` is the best-cloning variant; ``-mf`` is the
|
||||
#: fastest (use num_steps=4). Overridable for air-gapped / mirror installs.
|
||||
_DEFAULT_REPO = "rednote-hilab/dots.tts-soar"
|
||||
|
||||
|
||||
# ── wire protocol ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _send(stream, obj: dict) -> None:
|
||||
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
|
||||
stream.write(struct.pack("!I", len(body)))
|
||||
stream.write(body)
|
||||
stream.flush()
|
||||
|
||||
|
||||
def _recv(stream):
|
||||
header = stream.read(4)
|
||||
if len(header) < 4:
|
||||
return None # EOF
|
||||
(n,) = struct.unpack("!I", header)
|
||||
if n > MAX_FRAME_BYTES:
|
||||
raise IOError(f"frame too large: {n}")
|
||||
body = bytearray()
|
||||
while len(body) < n:
|
||||
chunk = stream.read(n - len(body))
|
||||
if not chunk:
|
||||
raise IOError("short read")
|
||||
body.extend(chunk)
|
||||
return json.loads(bytes(body).decode("utf-8"))
|
||||
|
||||
|
||||
def _measure_vram_mb() -> float:
|
||||
"""This sidecar's own GPU memory in MB (MM2-08). 0 on CPU. Never raises."""
|
||||
try:
|
||||
import torch
|
||||
if torch.cuda.is_available():
|
||||
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
|
||||
except Exception:
|
||||
pass
|
||||
return 0.0
|
||||
|
||||
|
||||
# ── model loading (lazy, on first synthesize) ─────────────────────────────
|
||||
|
||||
|
||||
# Module-level singleton — (runtime,). Device is auto-selected inside the
|
||||
# dots.tts runtime (cuda-or-cpu, no MPS); we don't pass a device.
|
||||
_runtime = None
|
||||
|
||||
|
||||
def _load_runtime(stdout):
|
||||
"""Cold-construct the dots.tts runtime.
|
||||
|
||||
``DotsTtsRuntime.from_pretrained`` auto-selects cuda-or-cpu internally
|
||||
(no MPS path). precision is bf16 on CUDA; on CPU we fall back to fp32
|
||||
(bf16 CPU kernels are spotty). Both overridable via env.
|
||||
"""
|
||||
global _runtime
|
||||
if _runtime is not None:
|
||||
return _runtime
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 0})
|
||||
|
||||
import torch
|
||||
from dots_tts.runtime import DotsTtsRuntime # type: ignore[import-not-found]
|
||||
|
||||
repo = os.environ.get("OMNIVOICE_DOTS_TTS_MODEL", _DEFAULT_REPO)
|
||||
default_precision = "bfloat16" if torch.cuda.is_available() else "float32"
|
||||
precision = os.environ.get("OMNIVOICE_DOTS_TTS_PRECISION", default_precision)
|
||||
optimize = os.environ.get("OMNIVOICE_DOTS_TTS_OPTIMIZE", "0") == "1"
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 50})
|
||||
|
||||
_runtime = DotsTtsRuntime.from_pretrained(
|
||||
repo,
|
||||
precision=precision,
|
||||
optimize=optimize,
|
||||
)
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 100})
|
||||
return _runtime
|
||||
|
||||
|
||||
def _tensor_to_pcm_b64(audio, sample_rate: int) -> tuple[str, int, int]:
|
||||
"""Convert a torch waveform tensor (1, N) in [-1, 1] to base64 int16 PCM."""
|
||||
import numpy as np
|
||||
|
||||
arr = audio.detach().to("cpu").float().numpy()
|
||||
arr = np.asarray(arr, dtype=np.float32).squeeze()
|
||||
if arr.ndim > 1:
|
||||
arr = arr.mean(axis=0) # defensive downmix to mono
|
||||
arr = np.clip(arr, -1.0, 1.0)
|
||||
pcm = (arr * 32767.0).astype(np.int16).tobytes()
|
||||
return base64.b64encode(pcm).decode("ascii"), int(sample_rate), int(arr.shape[0])
|
||||
|
||||
|
||||
def _normalize_language(raw):
|
||||
"""Map OmniVoice's language value to what dots.tts accepts, or None.
|
||||
|
||||
dots.tts accepts None/"auto_detect", ISO codes upper-cased ("EN"/"ZH"),
|
||||
or names ("english"). A 2-letter ISO code is upper-cased; anything else
|
||||
is passed through; empty / "auto" → None (auto-detect)."""
|
||||
if not raw or not isinstance(raw, str):
|
||||
return None
|
||||
s = raw.strip()
|
||||
if not s or s.lower() == "auto":
|
||||
return None
|
||||
if len(s) == 2 and s.isalpha():
|
||||
return s.upper()
|
||||
return s
|
||||
|
||||
|
||||
def _handle_synthesize(msg: dict, stdout) -> None:
|
||||
"""Dispatch one synthesize request. Emits the audio frame or raises."""
|
||||
text = msg.get("text")
|
||||
if not text or not isinstance(text, str):
|
||||
raise ValueError("synthesize: missing or non-string 'text'")
|
||||
|
||||
runtime = _load_runtime(stdout)
|
||||
|
||||
gen_kwargs: dict = {
|
||||
"text": text,
|
||||
"num_steps": int(msg.get("num_steps", 10)),
|
||||
"guidance_scale": float(msg.get("guidance_scale", 1.2)),
|
||||
}
|
||||
|
||||
ref_audio = msg.get("ref_audio")
|
||||
if ref_audio:
|
||||
gen_kwargs["prompt_audio_path"] = ref_audio
|
||||
ref_text = msg.get("ref_text")
|
||||
if ref_text:
|
||||
# continuation cloning — upstream requires prompt_audio_path when
|
||||
# prompt_text is set (the parent already enforces this).
|
||||
gen_kwargs["prompt_text"] = ref_text
|
||||
|
||||
language = _normalize_language(msg.get("language"))
|
||||
if language:
|
||||
gen_kwargs["language"] = language
|
||||
|
||||
result = runtime.generate(**gen_kwargs)
|
||||
audio = result["audio"]
|
||||
sample_rate = int(result.get("sample_rate", DOTS_SAMPLE_RATE))
|
||||
|
||||
pcm_b64, sr, n_samples = _tensor_to_pcm_b64(audio, sample_rate)
|
||||
_send(stdout, {
|
||||
"op": "audio",
|
||||
"audio_pcm_b64": pcm_b64,
|
||||
"sample_rate": sr,
|
||||
"n_samples": n_samples,
|
||||
})
|
||||
|
||||
|
||||
# ── main loop ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def main() -> int:
|
||||
stdin = sys.stdin.buffer
|
||||
stdout = sys.stdout.buffer
|
||||
|
||||
# Ready handshake fires BEFORE any heavy import.
|
||||
_send(stdout, {
|
||||
"op": "ready",
|
||||
"engine": "dots-tts",
|
||||
"sample_rate": DOTS_SAMPLE_RATE,
|
||||
})
|
||||
|
||||
while True:
|
||||
try:
|
||||
msg = _recv(stdin)
|
||||
except Exception as exc:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": "recv",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
return 1
|
||||
if msg is None:
|
||||
return 0
|
||||
|
||||
op = msg.get("op") if isinstance(msg, dict) else None
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
|
||||
elif op == "synthesize":
|
||||
_handle_synthesize(msg, stdout)
|
||||
elif op == "shutdown":
|
||||
return 0
|
||||
else:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": "dispatch",
|
||||
"message": f"unknown op: {op!r}",
|
||||
})
|
||||
except Exception as exc:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": op or "unknown",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -79,11 +79,6 @@ class IndexTTS2Backend(SubprocessBackend):
|
||||
display_name = "IndexTTS2 (emotion control, duration control, zero-shot)"
|
||||
supports_voice_design = False # requires ref audio for timbre
|
||||
_DEFAULT_SAMPLE_RATE = 24000
|
||||
# Explicit so IndexTTS2 stops advertising the inherited CPU-only default:
|
||||
# the sidecar runs the IndexTTS PyTorch model on CUDA when present, else
|
||||
# CPU. ROCm left unclaimed (the sidecar's own venv would need a ROCm torch);
|
||||
# a ROCm host honestly resolves to cpu_fallback.
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
|
||||
@@ -74,24 +74,6 @@ import traceback
|
||||
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
|
||||
MAX_FRAME_BYTES = 64 * 1024 * 1024
|
||||
|
||||
|
||||
def _measure_vram_mb() -> float:
|
||||
"""This sidecar's own GPU memory in MB, for the loaded-models panel
|
||||
(MM2-08). The parent can't see a child's VRAM, so we self-report it in the
|
||||
pong. Degrades to 0 on CPU / when torch isn't loaded yet — never raises."""
|
||||
try:
|
||||
import torch # already a dep inside the indextts venv
|
||||
if torch.cuda.is_available():
|
||||
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
|
||||
mps = getattr(torch.backends, "mps", None)
|
||||
if mps is not None and mps.is_available():
|
||||
drv = getattr(torch.mps, "driver_allocated_memory", None)
|
||||
if drv:
|
||||
return round(drv() / (1024 ** 2), 1)
|
||||
except Exception:
|
||||
pass
|
||||
return 0.0
|
||||
|
||||
# Sample rate IndexTTS-2 emits natively. Advertised in the ready frame so
|
||||
# the parent doesn't have to import IndexTTS just to learn the rate.
|
||||
INDEXTTS_SAMPLE_RATE = 24000
|
||||
@@ -300,7 +282,7 @@ def main() -> int:
|
||||
op = msg.get("op") if isinstance(msg, dict) else None
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
|
||||
_send(stdout, {"op": "pong"})
|
||||
elif op == "synthesize":
|
||||
_handle_synthesize(msg, stdout)
|
||||
elif op == "shutdown":
|
||||
|
||||
@@ -1,194 +0,0 @@
|
||||
"""MOSS-TTS-v1.5 sidecar package (issue #498).
|
||||
|
||||
MOSS-TTS-v1.5 is OpenMOSS's 8B flagship TTS — a Qwen3-8B language backbone
|
||||
plus a 1.6B audio codec, 31 languages, zero-shot voice cloning, token-level
|
||||
duration control and inline ``[pause Ns]`` markers. Apache-2.0.
|
||||
|
||||
It runs in its own subprocess **and its own venv**, isolated from the
|
||||
OmniVoice parent process, for the *same* reason IndexTTS does: a hard
|
||||
``transformers`` version conflict. MOSS-TTS-v1.5's ``torch-runtime`` extra
|
||||
pins ``transformers==5.0.0`` (verified against the upstream
|
||||
``pyproject.toml``), while OmniVoice pins ``transformers>=5.3.0``. The two
|
||||
cannot share one interpreter — so MOSS lives behind ``SubprocessBackend``
|
||||
with a dedicated venv, exactly like ``engines.indextts``.
|
||||
|
||||
Three public entry points live in this package:
|
||||
|
||||
* ``MossTTSV15Backend`` (this module) — the SubprocessBackend subclass
|
||||
that ``services.tts_backend._LAZY_REGISTRY`` resolves on first access.
|
||||
Defined HERE (not in ``services.tts_backend``) to break the import
|
||||
cycle: ``services.subprocess_backend`` imports ``TTSBackend`` from
|
||||
``services.tts_backend``, so the backend class must live downstream of
|
||||
that module finishing its import. Same indirection as IndexTTS /
|
||||
Supertonic-3.
|
||||
* ``main.py`` — the sidecar entrypoint (runs under MOSS's venv with
|
||||
``transformers==5.0.0``; never imported by the parent).
|
||||
* ``bootstrap.py`` — the venv-probe + lazy-bootstrap helper.
|
||||
|
||||
Do NOT import ``main.py`` from the parent process — it runs under a
|
||||
different venv (``transformers==5.0.0``) and importing it in-process would
|
||||
re-introduce the exact conflict this isolation exists to avoid.
|
||||
|
||||
Hardware honesty (cross-platform rule): MOSS-TTS-v1.5's upstream documents
|
||||
only CUDA and CPU. There is **no documented or tested MPS path** — the
|
||||
custom ``trust_remote_code`` modelling code and the separate audio
|
||||
tokenizer are unverified on Apple Silicon. We therefore advertise
|
||||
``gpu_compat = ("cuda", "cpu")`` and the sidecar selects ``cuda`` when
|
||||
present else ``cpu`` — it never silently routes to MPS where it might
|
||||
crash. On Apple Silicon the engine honestly resolves to CPU (slow but
|
||||
correct), and the engine is opt-in regardless, so it never becomes a
|
||||
broken default on any platform.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from services.subprocess_backend import SubprocessBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import torch # noqa: F401
|
||||
|
||||
logger = logging.getLogger("omnivoice.moss_tts_v15")
|
||||
|
||||
#: 1 second of audio ≈ 12.5 codec tokens (MOSS-TTS-v1.5 model card). Used to
|
||||
#: translate OmniVoice's ``duration`` (seconds) into the model's ``tokens``
|
||||
#: duration-control argument.
|
||||
TOKENS_PER_SECOND: float = 12.5
|
||||
|
||||
|
||||
class MossTTSV15Backend(SubprocessBackend):
|
||||
"""MOSS-TTS-v1.5 (OpenMOSS) — 8B, 31 langs, zero-shot clone, CUDA/CPU.
|
||||
|
||||
Runs in a long-lived sidecar over length-prefixed JSON-over-stdio in a
|
||||
dedicated venv (``transformers==5.0.0``). The first synthesize cold-loads
|
||||
~16 GB of bf16 weights (CUDA) / fp32 (CPU); subsequent calls reuse the
|
||||
process and the in-memory model.
|
||||
|
||||
Installation (transparent to power users who already cloned MOSS-TTS —
|
||||
OmniVoice prefers their existing ``${DIR}/.venv``)::
|
||||
|
||||
git clone https://github.com/OpenMOSS/MOSS-TTS.git
|
||||
cd MOSS-TTS
|
||||
# CUDA host:
|
||||
uv venv && uv pip install -e ".[torch-runtime]"
|
||||
# non-CUDA host (CPU): install plain torch/transformers instead of +cu128
|
||||
|
||||
Set ``OMNIVOICE_MOSS_TTS_V15_DIR`` to the clone root. OmniVoice creates
|
||||
``backend/engines/moss_tts_v15/.venv`` lazily on first launch if no venv
|
||||
exists yet (CUDA hosts only — the upstream ``torch-runtime`` extra is
|
||||
``+cu128``); the user's existing ``${DIR}/.venv`` is preferred if
|
||||
present, so no re-install is needed.
|
||||
|
||||
License: Apache-2.0 (code + weights) — no acceptance gate needed.
|
||||
"""
|
||||
|
||||
id = "moss-tts-v15"
|
||||
display_name = (
|
||||
"MOSS-TTS-v1.5 (8B, 31 langs, zero-shot clone, CUDA/CPU, Apache-2.0)"
|
||||
)
|
||||
supports_voice_design = False # requires ref audio for timbre cloning
|
||||
_DEFAULT_SAMPLE_RATE = 24000
|
||||
# Honest hardware surface: upstream documents CUDA + CPU only. MPS is
|
||||
# undocumented / untested, so we do NOT claim it (cross-platform rule).
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
# ── availability ───────────────────────────────────────────────────────
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
# IMPORTANT: do NOT attempt to import MOSS / its transformers==5.0.0
|
||||
# here. The parent pins transformers>=5.3 — co-importing the two in
|
||||
# one interpreter is exactly the conflict this subprocess isolation
|
||||
# exists to avoid. We only verify the venv exists on disk; a real
|
||||
# health-check (spawn + ping) is gated on the user's "Test engine"
|
||||
# action in Settings, same as IndexTTS.
|
||||
from engines.moss_tts_v15.bootstrap import (
|
||||
MOSS_TTS_V15_SIDECAR_SCRIPT,
|
||||
is_moss_tts_v15_installed,
|
||||
)
|
||||
if not is_moss_tts_v15_installed():
|
||||
return False, (
|
||||
"MOSS-TTS-v1.5 venv not found. Set OMNIVOICE_MOSS_TTS_V15_DIR "
|
||||
"to your MOSS-TTS clone (the directory containing pyproject.toml) "
|
||||
"and restart OmniVoice. CUDA or CPU only (no MPS). See "
|
||||
"docs/engines/moss-tts-v15.md for the full install walk-through."
|
||||
)
|
||||
if not MOSS_TTS_V15_SIDECAR_SCRIPT.exists():
|
||||
return False, (
|
||||
"MOSS-TTS-v1.5 sidecar script missing at "
|
||||
f"{MOSS_TTS_V15_SIDECAR_SCRIPT} — reinstall OmniVoice."
|
||||
)
|
||||
return True, "ok (CUDA when present, else CPU)"
|
||||
|
||||
@classmethod
|
||||
def venv_python(cls):
|
||||
from engines.moss_tts_v15.bootstrap import resolve_moss_tts_v15_venv
|
||||
return resolve_moss_tts_v15_venv()
|
||||
|
||||
@classmethod
|
||||
def sidecar_script(cls):
|
||||
from engines.moss_tts_v15.bootstrap import MOSS_TTS_V15_SIDECAR_SCRIPT
|
||||
return MOSS_TTS_V15_SIDECAR_SCRIPT
|
||||
|
||||
# ── TTSBackend protocol ────────────────────────────────────────────────
|
||||
|
||||
@property
|
||||
def sample_rate(self) -> int:
|
||||
return self._DEFAULT_SAMPLE_RATE
|
||||
|
||||
@property
|
||||
def supported_languages(self) -> list[str]:
|
||||
# 31 languages with multilingual handling; expose "multi" on the
|
||||
# protocol surface (same as OmniVoice / CosyVoice / Supertonic-3) and
|
||||
# translate the caller's language at synthesize time.
|
||||
return ["multi"]
|
||||
|
||||
# ── generate (parent-side arbitration) ─────────────────────────────────
|
||||
|
||||
def generate(self, text: str, **kw) -> "torch.Tensor":
|
||||
"""Synthesize one utterance through the MOSS-TTS-v1.5 sidecar.
|
||||
|
||||
kwargs honored:
|
||||
* ``ref_audio`` — path to a reference clip. When present, MOSS
|
||||
runs zero-shot voice cloning (``reference=``).
|
||||
Optional: without it the model uses its own
|
||||
default voice.
|
||||
* ``ref_text`` — accepted but unused in clone mode (MOSS's
|
||||
zero-shot path needs only the audio); kept in
|
||||
the signature so the common call-site doesn't
|
||||
need engine-specific knowledge.
|
||||
* ``language`` — ISO code or name; mapped to a MOSS language name
|
||||
in the sidecar, omitted (auto-detect) if unknown.
|
||||
* ``duration`` — target seconds → ``tokens`` (1 s ≈ 12.5 tokens).
|
||||
* ``max_new_tokens`` — generation cap (default 4096).
|
||||
|
||||
Returns a tensor of shape (1, n_samples) at :attr:`sample_rate`.
|
||||
"""
|
||||
forwarded: dict = {}
|
||||
|
||||
ref_audio = kw.get("ref_audio")
|
||||
if ref_audio:
|
||||
forwarded["ref_audio"] = ref_audio
|
||||
ref_text = kw.get("ref_text")
|
||||
if ref_text:
|
||||
forwarded["ref_text"] = ref_text
|
||||
|
||||
language = kw.get("language")
|
||||
if language:
|
||||
forwarded["language"] = str(language)
|
||||
|
||||
duration = kw.get("duration")
|
||||
if duration is not None:
|
||||
target_tokens = int(float(duration) * TOKENS_PER_SECOND)
|
||||
if target_tokens > 0:
|
||||
forwarded["tokens"] = target_tokens
|
||||
|
||||
max_new_tokens = kw.get("max_new_tokens")
|
||||
if max_new_tokens is not None:
|
||||
forwarded["max_new_tokens"] = int(max_new_tokens)
|
||||
|
||||
return super().generate(text, **forwarded)
|
||||
|
||||
|
||||
__all__ = ["MossTTSV15Backend", "TOKENS_PER_SECOND"]
|
||||
@@ -1,272 +0,0 @@
|
||||
"""MOSS-TTS-v1.5 venv probe + lazy bootstrap (issue #498).
|
||||
|
||||
The parent process needs to know *which Python interpreter* to spawn the
|
||||
MOSS-TTS-v1.5 sidecar under. This module owns that resolution. It mirrors
|
||||
``engines.indextts.bootstrap`` because MOSS has the same shape of problem:
|
||||
a hard ``transformers`` pin (``==5.0.0``) that conflicts with the parent's
|
||||
``transformers>=5.3`` — so MOSS runs in its own venv.
|
||||
|
||||
Probe order (priority — existing power-user installs win, zero migration):
|
||||
|
||||
1. ``${OMNIVOICE_MOSS_TTS_V15_DIR}/.venv/`` — the user's clone-level
|
||||
venv. Highest priority: a user who already cloned MOSS-TTS and ran
|
||||
``uv pip install -e ".[torch-runtime]"`` (per upstream docs) gets
|
||||
reused verbatim, no re-download of the ~16 GB model.
|
||||
2. ``backend/engines/moss_tts_v15/.venv/`` — this package's own venv,
|
||||
created by step 3 if needed.
|
||||
3. Bootstrap: ``uv venv`` then ``uv pip install -e
|
||||
"${DIR}[torch-runtime]"``. Requires ``OMNIVOICE_MOSS_TTS_V15_DIR``.
|
||||
The upstream ``torch-runtime`` extra is CUDA (``+cu128``), so the
|
||||
auto-bootstrap targets CUDA hosts; non-CUDA (CPU/Mac) users set up
|
||||
their own venv per docs/engines/moss-tts-v15.md (Probe 1).
|
||||
|
||||
Caching: resolution is memoised after the first successful call. Tests
|
||||
reset via :func:`invalidate`.
|
||||
|
||||
Security: bootstrap never touches HF_TOKEN; the sidecar's stderr is drained
|
||||
by SubprocessBackend through the parent root logger where Phase 1's
|
||||
``HFTokenRedactor`` strips token bytes. ``uv pip install -e`` installs from
|
||||
a user-controlled clone the user already trusts (same posture as IndexTTS).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger("omnivoice.moss_tts_v15.bootstrap")
|
||||
|
||||
#: Absolute path to the sidecar entrypoint. ``MossTTSV15Backend.sidecar_script``
|
||||
#: returns this; SubprocessBackend spawns it with the resolved venv python.
|
||||
MOSS_TTS_V15_SIDECAR_SCRIPT: Path = Path(__file__).parent / "main.py"
|
||||
|
||||
#: This package's owned venv (Probe 2). The MOSS-TTS clone, when bootstrapped,
|
||||
#: is installed into this venv via ``uv pip install -e``.
|
||||
_ENGINES_VENV_DIR: Path = Path(__file__).parent / ".venv"
|
||||
|
||||
#: Env var pointing at the user's MOSS-TTS clone root.
|
||||
_CLONE_DIR_ENV: str = "OMNIVOICE_MOSS_TTS_V15_DIR"
|
||||
|
||||
#: Per-process resolution cache. Cleared by :func:`invalidate` for tests.
|
||||
_resolved_python: Optional[Path] = None
|
||||
|
||||
# Timeouts — bounded so a wedged venv never hangs the parent. The bootstrap
|
||||
# install can take many minutes on a cold cache (MOSS pulls a CUDA torch
|
||||
# build + transformers + an audio codec stack).
|
||||
_IMPORT_PROBE_TIMEOUT_S = 15
|
||||
_UV_VENV_TIMEOUT_S = 120
|
||||
_UV_PIP_INSTALL_TIMEOUT_S = 1800
|
||||
|
||||
|
||||
# ── public API ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def invalidate() -> None:
|
||||
"""Clear the resolved-python cache. Tests call this between scenarios."""
|
||||
global _resolved_python
|
||||
_resolved_python = None
|
||||
|
||||
|
||||
def is_moss_tts_v15_installed() -> bool:
|
||||
"""Cheap file-existence check for a usable MOSS-TTS-v1.5 venv.
|
||||
|
||||
Returns True if either Probe 1 or Probe 2 has a Python executable on
|
||||
disk. Does NOT spawn the venv Python — that's saved for
|
||||
:func:`resolve_moss_tts_v15_venv`, which is only invoked on the first
|
||||
generate() / health_check(). This fires on every Settings render via
|
||||
``MossTTSV15Backend.is_available()``, so it stays cheap.
|
||||
"""
|
||||
for cand in _probe_paths():
|
||||
if cand.is_file():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def resolve_moss_tts_v15_venv() -> Path:
|
||||
"""Resolve the path to the Python interpreter that runs the sidecar.
|
||||
|
||||
Probe order described in the module docstring. Memoised. Raises
|
||||
:exc:`RuntimeError` if no working venv can be located AND the bootstrap
|
||||
path is unavailable.
|
||||
"""
|
||||
global _resolved_python
|
||||
if _resolved_python is not None:
|
||||
return _resolved_python
|
||||
|
||||
clone_dir = os.environ.get(_CLONE_DIR_ENV)
|
||||
|
||||
# Probe 1 — user's clone-level venv (highest priority for back-compat).
|
||||
if clone_dir:
|
||||
cand = _venv_python_path(Path(clone_dir) / ".venv")
|
||||
if cand.is_file() and _venv_can_import_moss(cand):
|
||||
logger.info(
|
||||
"MOSS-TTS-v1.5 venv resolved from %s: %s", _CLONE_DIR_ENV, cand,
|
||||
)
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
# Probe 2 — this package's own venv.
|
||||
cand = _venv_python_path(_ENGINES_VENV_DIR)
|
||||
if cand.is_file() and _venv_can_import_moss(cand):
|
||||
logger.info("MOSS-TTS-v1.5 venv resolved from engines path: %s", cand)
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
# Probe 3 — bootstrap.
|
||||
if not clone_dir:
|
||||
raise RuntimeError(
|
||||
"MOSS-TTS-v1.5 is not installed. Set the "
|
||||
f"{_CLONE_DIR_ENV} environment variable to your MOSS-TTS clone "
|
||||
"(the directory that contains pyproject.toml), then restart "
|
||||
"OmniVoice. See docs/engines/moss-tts-v15.md for the full "
|
||||
"install walk-through."
|
||||
)
|
||||
|
||||
cand = _bootstrap_engines_venv(Path(clone_dir))
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
|
||||
# ── internals ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _venv_python_path(venv_dir: Path) -> Path:
|
||||
"""Return the python executable path inside a venv directory.
|
||||
|
||||
Handles the Unix (``bin/python``) vs Windows (``Scripts/python.exe``)
|
||||
layout. No filesystem access — caller checks .is_file().
|
||||
"""
|
||||
if sys.platform == "win32":
|
||||
return venv_dir / "Scripts" / "python.exe"
|
||||
return venv_dir / "bin" / "python"
|
||||
|
||||
|
||||
def _probe_paths() -> list[Path]:
|
||||
"""Ordered list of candidate venv-python paths (no .is_file() check)."""
|
||||
out: list[Path] = []
|
||||
clone_dir = os.environ.get(_CLONE_DIR_ENV)
|
||||
if clone_dir:
|
||||
out.append(_venv_python_path(Path(clone_dir) / ".venv"))
|
||||
out.append(_venv_python_path(_ENGINES_VENV_DIR))
|
||||
return out
|
||||
|
||||
|
||||
def _venv_can_import_moss(python_path: Path) -> bool:
|
||||
"""Spawn the candidate python and verify the MOSS stack imports.
|
||||
|
||||
MOSS-TTS-v1.5 loads via ``transformers`` + ``trust_remote_code`` (no
|
||||
fixed top-level package to import), so the readiness signal is that the
|
||||
venv has a working ``transformers`` + ``torch`` — which only the
|
||||
``[torch-runtime]`` install provides. Bounded by
|
||||
``_IMPORT_PROBE_TIMEOUT_S`` so a wedged venv never hangs the parent.
|
||||
Returns False on any failure (non-zero exit, timeout, OSError).
|
||||
"""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[str(python_path), "-c", "import transformers, torch"],
|
||||
capture_output=True,
|
||||
timeout=_IMPORT_PROBE_TIMEOUT_S,
|
||||
)
|
||||
except (subprocess.TimeoutExpired, OSError) as exc:
|
||||
logger.debug("moss-tts-v15 import probe failed for %s: %s", python_path, exc)
|
||||
return False
|
||||
if proc.returncode != 0:
|
||||
logger.debug(
|
||||
"moss-tts-v15 import probe non-zero for %s: %s",
|
||||
python_path,
|
||||
proc.stderr.decode("utf-8", errors="replace")[:200],
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _locate_uv() -> Optional[str]:
|
||||
"""Find the uv binary — bundled first (Tauri-set env var), else PATH."""
|
||||
bundled = os.environ.get("OMNIVOICE_BUNDLED_UV")
|
||||
if bundled and Path(bundled).is_file():
|
||||
return bundled
|
||||
sys_uv = shutil.which("uv")
|
||||
if sys_uv:
|
||||
return sys_uv
|
||||
return None
|
||||
|
||||
|
||||
def _bootstrap_engines_venv(clone_dir: Path) -> Path:
|
||||
"""Create engines/moss_tts_v15/.venv and install the user's clone into it.
|
||||
|
||||
Runs ``uv venv <engines_venv>`` then ``uv pip install --python
|
||||
<engines_venv>/bin/python -e "<clone>[torch-runtime]"``. Verifies the
|
||||
result by re-probing the import — a successful uv invocation that still
|
||||
can't import the stack indicates a deeper environment problem (e.g. the
|
||||
``+cu128`` torch-runtime extra can't resolve on a non-CUDA host) and we
|
||||
raise with whatever stderr we captured plus a docs pointer.
|
||||
"""
|
||||
uv = _locate_uv()
|
||||
if not uv:
|
||||
raise RuntimeError(
|
||||
"uv is required to bootstrap the MOSS-TTS-v1.5 venv but was not "
|
||||
"found on PATH (and OMNIVOICE_BUNDLED_UV was not set). Install uv "
|
||||
"from https://docs.astral.sh/uv/ and re-launch OmniVoice, or set "
|
||||
"OMNIVOICE_BUNDLED_UV to the absolute path of a uv binary."
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Bootstrapping MOSS-TTS-v1.5 venv at %s from %s (this can take "
|
||||
"several minutes on first launch)", _ENGINES_VENV_DIR, clone_dir,
|
||||
)
|
||||
|
||||
try:
|
||||
subprocess.run(
|
||||
[uv, "venv", str(_ENGINES_VENV_DIR)],
|
||||
check=True,
|
||||
timeout=_UV_VENV_TIMEOUT_S,
|
||||
capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
f"uv venv failed for MOSS-TTS-v1.5 bootstrap at {_ENGINES_VENV_DIR}: "
|
||||
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
|
||||
) from exc
|
||||
|
||||
python_path = _venv_python_path(_ENGINES_VENV_DIR)
|
||||
try:
|
||||
subprocess.run(
|
||||
[
|
||||
uv, "pip", "install",
|
||||
"--python", str(python_path),
|
||||
"-e", f"{clone_dir}[torch-runtime]",
|
||||
],
|
||||
check=True,
|
||||
timeout=_UV_PIP_INSTALL_TIMEOUT_S,
|
||||
capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
"uv pip install -e failed during MOSS-TTS-v1.5 bootstrap "
|
||||
f"({clone_dir}). On a non-CUDA host the upstream '[torch-runtime]' "
|
||||
"extra (cu128) cannot resolve — set up the venv manually per "
|
||||
"docs/engines/moss-tts-v15.md. Error: "
|
||||
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
|
||||
) from exc
|
||||
|
||||
if not _venv_can_import_moss(python_path):
|
||||
raise RuntimeError(
|
||||
"MOSS-TTS-v1.5 bootstrap completed but the transformers/torch "
|
||||
f"import still fails from {python_path}. Verify that {clone_dir} "
|
||||
"is a valid MOSS-TTS clone. See docs/engines/moss-tts-v15.md."
|
||||
)
|
||||
|
||||
logger.info("MOSS-TTS-v1.5 venv bootstrap successful: %s", python_path)
|
||||
return python_path
|
||||
|
||||
|
||||
__all__ = [
|
||||
"MOSS_TTS_V15_SIDECAR_SCRIPT",
|
||||
"invalidate",
|
||||
"is_moss_tts_v15_installed",
|
||||
"resolve_moss_tts_v15_venv",
|
||||
]
|
||||
@@ -1,303 +0,0 @@
|
||||
"""MOSS-TTS-v1.5 sidecar entry point (issue #498).
|
||||
|
||||
Runs inside ``engines/moss_tts_v15/.venv`` (or the user's existing
|
||||
``${OMNIVOICE_MOSS_TTS_V15_DIR}/.venv``) with ``transformers==5.0.0``,
|
||||
isolated from the OmniVoice parent process which pins ``transformers>=5.3``.
|
||||
Same isolation rationale as the IndexTTS sidecar.
|
||||
|
||||
Stdlib-only at import time. The model + transformers + torch are imported
|
||||
lazily on the first synthesize op so the sidecar emits its ``ready`` frame
|
||||
inside the parent's 30 s spawn handshake even on a cold filesystem (an 8B
|
||||
model takes well over 30 s to cold-load).
|
||||
|
||||
Wire protocol — length-prefixed JSON over stdin/stdout, byte-identical to
|
||||
``backend/services/subprocess_backend.py``::
|
||||
|
||||
[ 4-byte big-endian uint32 length ][ N bytes UTF-8 JSON ]
|
||||
|
||||
Op flow:
|
||||
1. Sidecar -> parent: {"op": "ready", "engine": "moss-tts-v15",
|
||||
"sample_rate": 24000}
|
||||
2. parent -> sidecar: {"op": "ping"} -> {"op": "pong", "vram_mb": N}
|
||||
3. parent -> sidecar: {"op": "synthesize", "text": "...",
|
||||
"ref_audio": "/path/spk.wav", "language": "fr",
|
||||
"tokens": 325, "max_new_tokens": 4096}
|
||||
-> {"op": "progress", ...} (cold load only) then
|
||||
-> {"op": "audio", "audio_pcm_b64": "...", "sample_rate": 24000,
|
||||
"n_samples": N}
|
||||
4. parent -> sidecar: {"op": "shutdown"} -> exit 0
|
||||
|
||||
Restrictions: NO imports from OmniVoice parent code (different venv). NO
|
||||
logging of ``os.environ`` contents. Single-frame DoS cap matches the
|
||||
parent's ``MAX_FRAME_BYTES``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
|
||||
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
|
||||
MAX_FRAME_BYTES = 64 * 1024 * 1024
|
||||
|
||||
#: Native sample rate MOSS-TTS-v1.5 emits. Advertised in the ready frame so
|
||||
#: the parent doesn't have to import MOSS just to learn the rate. Confirmed
|
||||
#: via ``processor.model_config.sampling_rate`` (the real value is read from
|
||||
#: the loaded model at synthesize time; this is the handshake default).
|
||||
MOSS_SAMPLE_RATE = 24000
|
||||
|
||||
#: HF repo id for the weights, overridable for air-gapped / mirror installs.
|
||||
_DEFAULT_REPO = "OpenMOSS-Team/MOSS-TTS-v1.5"
|
||||
|
||||
#: ISO-639-1 → MOSS language name. MOSS's ``build_user_message`` takes a
|
||||
#: language *name* ("French"), not a code. Unknown codes are omitted so the
|
||||
#: model auto-detects. Covers the high-traffic subset of MOSS's 31 langs.
|
||||
_ISO_TO_NAME = {
|
||||
"en": "English", "zh": "Chinese", "ja": "Japanese", "ko": "Korean",
|
||||
"fr": "French", "de": "German", "es": "Spanish", "it": "Italian",
|
||||
"pt": "Portuguese", "ru": "Russian", "ar": "Arabic", "hi": "Hindi",
|
||||
"nl": "Dutch", "pl": "Polish", "tr": "Turkish", "vi": "Vietnamese",
|
||||
"th": "Thai", "id": "Indonesian", "cs": "Czech", "el": "Greek",
|
||||
"he": "Hebrew", "fa": "Persian", "uk": "Ukrainian", "sv": "Swedish",
|
||||
}
|
||||
|
||||
|
||||
# ── wire protocol ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _send(stream, obj: dict) -> None:
|
||||
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
|
||||
stream.write(struct.pack("!I", len(body)))
|
||||
stream.write(body)
|
||||
stream.flush()
|
||||
|
||||
|
||||
def _recv(stream):
|
||||
header = stream.read(4)
|
||||
if len(header) < 4:
|
||||
return None # EOF
|
||||
(n,) = struct.unpack("!I", header)
|
||||
if n > MAX_FRAME_BYTES:
|
||||
raise IOError(f"frame too large: {n}")
|
||||
body = bytearray()
|
||||
while len(body) < n:
|
||||
chunk = stream.read(n - len(body))
|
||||
if not chunk:
|
||||
raise IOError("short read")
|
||||
body.extend(chunk)
|
||||
return json.loads(bytes(body).decode("utf-8"))
|
||||
|
||||
|
||||
def _measure_vram_mb() -> float:
|
||||
"""This sidecar's own GPU memory in MB (MM2-08). The parent can't see a
|
||||
child's VRAM, so we self-report it in the pong. 0 on CPU. Never raises."""
|
||||
try:
|
||||
import torch
|
||||
if torch.cuda.is_available():
|
||||
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
|
||||
except Exception:
|
||||
pass
|
||||
return 0.0
|
||||
|
||||
|
||||
# ── model loading (lazy, on first synthesize) ─────────────────────────────
|
||||
|
||||
|
||||
# Module-level singleton — populated on the first synthesize op and reused.
|
||||
# Holds (processor, model, device, sample_rate).
|
||||
_state = None
|
||||
|
||||
|
||||
def _load_model(stdout):
|
||||
"""Cold-construct the MOSS-TTS-v1.5 processor + model.
|
||||
|
||||
Device selection is CUDA-or-CPU only — MOSS's upstream documents no MPS
|
||||
path and the custom ``trust_remote_code`` modelling code is untested on
|
||||
Apple Silicon, so we never route to MPS where it might crash. dtype is
|
||||
bf16 on CUDA, fp32 on CPU (bf16 CPU ops are spotty). Emits progress
|
||||
frames so the parent can surface the multi-GB cold-load latency.
|
||||
"""
|
||||
global _state
|
||||
if _state is not None:
|
||||
return _state
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 0})
|
||||
|
||||
import torch
|
||||
from transformers import AutoModel, AutoProcessor
|
||||
|
||||
repo = os.environ.get("OMNIVOICE_MOSS_TTS_V15_MODEL", _DEFAULT_REPO)
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
dtype = torch.bfloat16 if device == "cuda" else torch.float32
|
||||
# "sdpa" works on CUDA + CPU and needs no extra dep. flash_attention_2
|
||||
# (Ampere+ CUDA, optional flash-attn) is opt-in via env.
|
||||
attn = os.environ.get("OMNIVOICE_MOSS_TTS_V15_ATTN", "sdpa")
|
||||
|
||||
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
|
||||
# The audio tokenizer is a separate sub-module that must be moved to the
|
||||
# device independently (easy to miss — see upstream README).
|
||||
processor.audio_tokenizer = processor.audio_tokenizer.to(device)
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 50})
|
||||
|
||||
model = AutoModel.from_pretrained(
|
||||
repo,
|
||||
trust_remote_code=True,
|
||||
attn_implementation=attn,
|
||||
torch_dtype=dtype,
|
||||
).to(device)
|
||||
model.eval()
|
||||
|
||||
sample_rate = int(getattr(processor.model_config, "sampling_rate", MOSS_SAMPLE_RATE))
|
||||
_state = (processor, model, device, sample_rate)
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 100})
|
||||
return _state
|
||||
|
||||
|
||||
def _tensor_to_pcm_b64(audio, sample_rate: int) -> tuple[str, int, int]:
|
||||
"""Convert a torch waveform tensor to base64 int16 PCM.
|
||||
|
||||
MOSS returns a float tensor in [-1, 1] (1-D or (1, N)); we squeeze to
|
||||
mono, clip, scale to int16, and base64 so the wire frame stays JSON-safe.
|
||||
"""
|
||||
import numpy as np
|
||||
|
||||
arr = audio.detach().to("cpu").float().numpy()
|
||||
arr = np.asarray(arr, dtype=np.float32).squeeze()
|
||||
if arr.ndim > 1:
|
||||
arr = arr.mean(axis=0) # defensive downmix to mono
|
||||
arr = np.clip(arr, -1.0, 1.0)
|
||||
pcm = (arr * 32767.0).astype(np.int16).tobytes()
|
||||
return base64.b64encode(pcm).decode("ascii"), int(sample_rate), int(arr.shape[0])
|
||||
|
||||
|
||||
def _resolve_language(raw):
|
||||
"""Map OmniVoice's language value to a MOSS language name, or None.
|
||||
|
||||
Accepts an ISO-639-1 code or a full name. Unknown / empty / "auto"
|
||||
values return None so MOSS auto-detects."""
|
||||
if not raw or not isinstance(raw, str):
|
||||
return None
|
||||
s = raw.strip()
|
||||
if not s or s.lower() == "auto":
|
||||
return None
|
||||
if s.lower() in _ISO_TO_NAME:
|
||||
return _ISO_TO_NAME[s.lower()]
|
||||
# Already a language name (or an unknown code) — pass it through; MOSS
|
||||
# ignores a language it doesn't recognise.
|
||||
return s
|
||||
|
||||
|
||||
def _handle_synthesize(msg: dict, stdout) -> None:
|
||||
"""Dispatch one synthesize request. Emits the audio frame or raises."""
|
||||
import torch
|
||||
|
||||
text = msg.get("text")
|
||||
if not text or not isinstance(text, str):
|
||||
raise ValueError("synthesize: missing or non-string 'text'")
|
||||
|
||||
processor, model, device, sample_rate = _load_model(stdout)
|
||||
|
||||
user_kwargs: dict = {"text": text}
|
||||
|
||||
ref_audio = msg.get("ref_audio")
|
||||
if ref_audio:
|
||||
# Zero-shot voice cloning: the reference audio alone is enough in
|
||||
# MOSS's clone mode (ref_text is not consumed here). The processor's
|
||||
# audio tokenizer encodes the reference into the prompt.
|
||||
user_kwargs["reference"] = [ref_audio]
|
||||
|
||||
language = _resolve_language(msg.get("language"))
|
||||
if language:
|
||||
user_kwargs["language"] = language
|
||||
|
||||
tokens = msg.get("tokens")
|
||||
if tokens is not None:
|
||||
user_kwargs["tokens"] = int(tokens)
|
||||
|
||||
max_new_tokens = int(msg.get("max_new_tokens", 4096))
|
||||
|
||||
conversations = [[processor.build_user_message(**user_kwargs)]]
|
||||
|
||||
with torch.no_grad():
|
||||
batch = processor(conversations, mode="generation")
|
||||
outputs = model.generate(
|
||||
input_ids=batch["input_ids"].to(device),
|
||||
attention_mask=batch["attention_mask"].to(device),
|
||||
max_new_tokens=max_new_tokens,
|
||||
)
|
||||
|
||||
decoded = processor.decode(outputs)
|
||||
audio = decoded[0].audio_codes_list[0]
|
||||
|
||||
pcm_b64, sr, n_samples = _tensor_to_pcm_b64(audio, sample_rate)
|
||||
_send(stdout, {
|
||||
"op": "audio",
|
||||
"audio_pcm_b64": pcm_b64,
|
||||
"sample_rate": sr,
|
||||
"n_samples": n_samples,
|
||||
})
|
||||
|
||||
|
||||
# ── main loop ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def main() -> int:
|
||||
stdin = sys.stdin.buffer
|
||||
stdout = sys.stdout.buffer
|
||||
|
||||
# Ready handshake fires BEFORE any heavy import — nothing above this line
|
||||
# touches transformers/torch, so we make the 30 s spawn window even cold.
|
||||
_send(stdout, {
|
||||
"op": "ready",
|
||||
"engine": "moss-tts-v15",
|
||||
"sample_rate": MOSS_SAMPLE_RATE,
|
||||
})
|
||||
|
||||
while True:
|
||||
try:
|
||||
msg = _recv(stdin)
|
||||
except Exception as exc:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": "recv",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
return 1
|
||||
if msg is None:
|
||||
return 0
|
||||
|
||||
op = msg.get("op") if isinstance(msg, dict) else None
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
|
||||
elif op == "synthesize":
|
||||
_handle_synthesize(msg, stdout)
|
||||
elif op == "shutdown":
|
||||
return 0
|
||||
else:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": "dispatch",
|
||||
"message": f"unknown op: {op!r}",
|
||||
})
|
||||
except Exception as exc:
|
||||
# Per-op failure is recoverable — emit the error frame and stay
|
||||
# alive so the parent can retry without paying the respawn +
|
||||
# multi-GB model-load cost again.
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": op or "unknown",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -353,21 +353,6 @@ def _make_backend_class():
|
||||
f"This clears the quarantine on the .app and its "
|
||||
f"bundled binaries. See docs/install/macos.md."
|
||||
)
|
||||
# Execute bit (issue #437). A `git clone` / zip extract on POSIX
|
||||
# can drop +x, which only surfaces at spawn time as
|
||||
# "[Errno 13] Permission denied" — and the generic synth handler
|
||||
# then mislabels it as out-of-memory. Self-heal here, AFTER the
|
||||
# SHA check has confirmed this is the right file (so we never
|
||||
# chmod a foreign binary). No-op on Windows.
|
||||
if os.name == "posix" and not os.access(bin_path, os.X_OK):
|
||||
try:
|
||||
bin_path.chmod(bin_path.stat().st_mode | 0o111)
|
||||
except OSError:
|
||||
return False, (
|
||||
f"GGUF binary {bin_path.name} isn't executable and "
|
||||
f"couldn't be made so — run `chmod +x {bin_path}` "
|
||||
f"and retry."
|
||||
)
|
||||
return True, "ready"
|
||||
except Exception as exc:
|
||||
return False, f"{type(exc).__name__}: {exc}"
|
||||
@@ -508,12 +493,6 @@ def _make_backend_class():
|
||||
* ``ref_audio`` (str/Path) — speaker reference WAV for cloning.
|
||||
* ``ref_text`` (str) — transcript of ``ref_audio``.
|
||||
* ``language`` (str) — ISO code or omnivoice-tts lang label.
|
||||
* ``instruct`` (str) — style instruction.
|
||||
* ``duration`` (float) — target duration in seconds.
|
||||
* ``seed`` (int) — deterministic sampling seed.
|
||||
* ``denoise`` (bool) — omit denoise token when false.
|
||||
* ``preprocess_prompt`` (bool) — skip prompt preprocessing when false.
|
||||
* ``chunk_duration`` / ``chunk_threshold`` (float) — binary long-form controls.
|
||||
"""
|
||||
import soundfile as sf # local import keeps module import cheap
|
||||
import torch
|
||||
@@ -524,32 +503,15 @@ def _make_backend_class():
|
||||
fd, out_str = tempfile.mkstemp(prefix="omnivoice-gguf-", suffix=".wav")
|
||||
os.close(fd)
|
||||
out_path = Path(out_str)
|
||||
ref_text_path: Optional[Path] = None
|
||||
|
||||
try:
|
||||
ref_text = kw.get("ref_text")
|
||||
if kw.get("ref_audio") and ref_text:
|
||||
text_fd, text_str = tempfile.mkstemp(
|
||||
prefix="omnivoice-gguf-ref-", suffix=".txt"
|
||||
)
|
||||
os.close(text_fd)
|
||||
ref_text_path = Path(text_str)
|
||||
ref_text_path.write_text(str(ref_text), encoding="utf-8")
|
||||
|
||||
argv = self._build_argv(
|
||||
base=base_path,
|
||||
tokenizer=tok_path,
|
||||
out_path=out_path,
|
||||
ref_audio=kw.get("ref_audio"),
|
||||
ref_text=str(ref_text_path) if ref_text_path else None,
|
||||
ref_text=kw.get("ref_text"),
|
||||
language=kw.get("language"),
|
||||
instruct=kw.get("instruct"),
|
||||
duration=kw.get("duration"),
|
||||
seed=kw.get("seed"),
|
||||
denoise=kw.get("denoise", True),
|
||||
preprocess_prompt=kw.get("preprocess_prompt", True),
|
||||
chunk_duration=kw.get("chunk_duration"),
|
||||
chunk_threshold=kw.get("chunk_threshold"),
|
||||
)
|
||||
self._run_subprocess(argv, stdin_text=text)
|
||||
wav, sr = sf.read(str(out_path))
|
||||
@@ -558,11 +520,6 @@ def _make_backend_class():
|
||||
out_path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
if ref_text_path is not None:
|
||||
try:
|
||||
ref_text_path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
# soundfile returns (n,) for mono or (n, c) for multichannel.
|
||||
# OmniVoice/Higgs Audio v2 is mono → (n,). Wrap to (1, n).
|
||||
@@ -587,13 +544,6 @@ def _make_backend_class():
|
||||
ref_audio: Optional[str],
|
||||
ref_text: Optional[str],
|
||||
language: Optional[str],
|
||||
instruct: Optional[str] = None,
|
||||
duration: Optional[float] = None,
|
||||
seed: Optional[int] = None,
|
||||
denoise: bool = True,
|
||||
preprocess_prompt: bool = True,
|
||||
chunk_duration: Optional[float] = None,
|
||||
chunk_threshold: Optional[float] = None,
|
||||
) -> list[str]:
|
||||
"""Compose argv from typed Path objects only (T-04-02)."""
|
||||
argv: list[str] = [
|
||||
@@ -605,20 +555,6 @@ def _make_backend_class():
|
||||
lang = _iso_to_omnivoice_lang(language)
|
||||
if lang:
|
||||
argv += ["--lang", lang]
|
||||
if instruct:
|
||||
argv += ["--instruct", str(instruct)]
|
||||
if duration is not None:
|
||||
argv += ["--duration", str(float(duration))]
|
||||
if seed is not None:
|
||||
argv += ["--seed", str(int(seed))]
|
||||
if denoise is False:
|
||||
argv += ["--no-denoise"]
|
||||
if preprocess_prompt is False:
|
||||
argv += ["--no-preprocess-prompt"]
|
||||
if chunk_duration is not None:
|
||||
argv += ["--chunk-duration", str(float(chunk_duration))]
|
||||
if chunk_threshold is not None:
|
||||
argv += ["--chunk-threshold", str(float(chunk_threshold))]
|
||||
if ref_audio:
|
||||
# Two-stage validation (defense in depth):
|
||||
# (a) Reject anything outside the project's voices /
|
||||
@@ -652,8 +588,11 @@ def _make_backend_class():
|
||||
)
|
||||
argv += ["--ref-wav", str(ref_path)]
|
||||
if ref_text:
|
||||
# The C++ runtime expects a transcript file path.
|
||||
# generate() creates this file in the system temp dir.
|
||||
# ref_text is free-form text; pass via stdin would
|
||||
# collide with the synthesis prompt, so the only safe
|
||||
# channel is argv. The binary treats this as a quoted
|
||||
# string at the OS layer (Popen escapes argv per
|
||||
# platform); we don't pre-escape.
|
||||
argv += ["--ref-text", str(ref_text)]
|
||||
return argv
|
||||
|
||||
|
||||
+6
-285
@@ -9,16 +9,6 @@ _backend_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
if _backend_dir not in sys.path:
|
||||
sys.path.insert(0, _backend_dir)
|
||||
|
||||
# #564: also make the project's OWN `omnivoice` package importable from source
|
||||
# when the venv's editable install is missing/broken (interrupted/offline
|
||||
# `uv sync`, antivirus-quarantined `_editable_impl_omnivoice.pth`, …). Without
|
||||
# this the backend boots fine and only fails at the first model call with
|
||||
# `No module named 'omnivoice'`. The bootstrap now gates on omnivoice being
|
||||
# importable too (re-syncing to re-lay the editable install); this is the
|
||||
# runtime safety net. See core/omnivoice_path.py for the full rationale.
|
||||
from core.omnivoice_path import ensure_omnivoice_importable
|
||||
ensure_omnivoice_importable(_backend_dir)
|
||||
|
||||
# Triton is unavailable on Windows — disable torch.compile / dynamo / inductor
|
||||
# to prevent TritonMissing errors at inference time. Must be set before torch
|
||||
# is imported (it is lazily imported in services/model_manager.py). Uses
|
||||
@@ -53,13 +43,11 @@ try:
|
||||
_project_env = os.path.join(os.path.dirname(_backend_dir), ".env")
|
||||
if os.path.isfile(_project_env):
|
||||
dotenv.load_dotenv(_project_env, override=False)
|
||||
# Load the durable per-user config (the in-app Settings source of truth) so
|
||||
# env vars set once survive Tauri/Finder launches that don't inherit a shell
|
||||
# environment. This OVERRIDES launcher-injected defaults: the desktop app
|
||||
# injects a stale OMNIVOICE_CACHE_DIR from its own config before startup, so
|
||||
# without override a models dir changed in Settings was ignored forever (#480).
|
||||
from core.user_env import load_into_environ as _load_user_env
|
||||
_load_user_env()
|
||||
# Also load the durable per-user config so env vars set once survive
|
||||
# Tauri/Finder launches that don't inherit a shell environment.
|
||||
_user_env = os.path.expanduser("~/.config/omnivoice/env")
|
||||
if os.path.isfile(_user_env):
|
||||
dotenv.load_dotenv(_user_env, override=False)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
@@ -296,12 +284,7 @@ from fastapi.responses import JSONResponse, RedirectResponse, Response
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from starlette.datastructures import MutableHeaders
|
||||
# Docs-only dependency: a venv created before scalar-fastapi entered the
|
||||
# dependency set must still boot the backend (#307) — /docs degrades instead.
|
||||
try:
|
||||
from scalar_fastapi import get_scalar_api_reference
|
||||
except ImportError:
|
||||
get_scalar_api_reference = None
|
||||
from scalar_fastapi import get_scalar_api_reference
|
||||
import traceback
|
||||
|
||||
_crash_log_lock = threading.Lock()
|
||||
@@ -330,22 +313,16 @@ from api.routers import (
|
||||
setup,
|
||||
gallery,
|
||||
archetypes,
|
||||
describe_voice,
|
||||
community,
|
||||
batch,
|
||||
watermark,
|
||||
events,
|
||||
capture,
|
||||
capture_ws,
|
||||
dictation,
|
||||
openai_compat,
|
||||
tts_stream,
|
||||
marketplace,
|
||||
personas,
|
||||
sonitranslate,
|
||||
audiobook,
|
||||
longform_jobs,
|
||||
pronunciation, # Expressive-TTS Spec 01: user pronunciation dictionary
|
||||
settings as settings_router, # Phase 1 AUTH-03: HF token save/clear/state
|
||||
)
|
||||
from utils import hf_progress
|
||||
@@ -355,30 +332,6 @@ from utils import hf_progress
|
||||
# the patched class, not the original.
|
||||
hf_progress.install()
|
||||
|
||||
# Wire the overall download aggregator's byte sink onto the patched tqdm so
|
||||
# parallel per-file updates feed one accurate overall bar (FDL-06).
|
||||
try:
|
||||
from utils import download_aggregator
|
||||
download_aggregator.install()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Log the download-acceleration state once at startup (FDL-03) so a slow
|
||||
# download report can be triaged from the logs without reproducing. Note: the
|
||||
# app sets HF_HUB_DISABLE_XET=1 above by default (legacy LFS for byte progress),
|
||||
# so xet_active is normally False even though hf_xet is installed.
|
||||
try:
|
||||
from api.routers.system import _fast_download_status as _fd_status
|
||||
_fd = _fd_status()
|
||||
_xet_ver = f" {_fd['xet_version']}" if _fd.get("xet_version") else ""
|
||||
logging.getLogger("omnivoice.model").info(
|
||||
"downloads: Xet %s (hf_xet%s installed=%s), high_perf=%s",
|
||||
"ACTIVE" if _fd["xet_active"] else "disabled → legacy LFS",
|
||||
_xet_ver, _fd["xet_installed"], _fd["high_performance"],
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _env_flag(name: str, default: bool = False) -> bool:
|
||||
value = os.environ.get(name)
|
||||
@@ -387,90 +340,8 @@ def _env_flag(name: str, default: bool = False) -> bool:
|
||||
return value.strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _mcp_start_timeout_s() -> float:
|
||||
"""Seconds to wait for the MCP session manager to start before giving up
|
||||
and serving without it (#632). Overridable via OMNIVOICE_MCP_START_TIMEOUT_S."""
|
||||
raw = os.environ.get("OMNIVOICE_MCP_START_TIMEOUT_S", "")
|
||||
try:
|
||||
v = float(raw)
|
||||
if v > 0:
|
||||
return v
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
return 30.0
|
||||
|
||||
|
||||
async def _serve_mcp(session_manager, ready: "asyncio.Event", stop: "asyncio.Event") -> None:
|
||||
"""Own the MCP session manager's full enter→exit lifecycle in ONE task.
|
||||
|
||||
FastMCP's ``run()`` opens an anyio task group, and anyio requires the cancel
|
||||
scope to be exited in the *same task* that entered it. So we must NOT enter
|
||||
it via ``wait_for`` (which runs the enter in a throwaway sub-task) or on the
|
||||
lifespan task and exit it elsewhere — either raises "Attempted to exit cancel
|
||||
scope in a different task". This coroutine enters and exits the context
|
||||
itself: it signals ``ready`` once mounted, then idles until ``stop``.
|
||||
"""
|
||||
try:
|
||||
async with session_manager.run():
|
||||
ready.set()
|
||||
await stop.wait()
|
||||
except Exception as e:
|
||||
logger.warning("MCP session manager stopped: %s", e)
|
||||
finally:
|
||||
ready.set() # never leave startup blocked on the readiness wait
|
||||
|
||||
|
||||
async def _start_mcp_session_manager(session_manager, *, timeout: float):
|
||||
"""Start MCP off the startup critical path; wait up to ``timeout`` for it to
|
||||
signal ready. Returns ``(task, stop_event, mounted)``.
|
||||
|
||||
The MCP layer is best-effort and must never wedge backend startup. On some
|
||||
platforms (observed: Apple-Silicon M1, #632) ``run()`` can *hang* on its
|
||||
anyio task group; the old code awaited the enter before serving, so the hang
|
||||
meant "Application startup complete" never fired and the whole backend was
|
||||
unreachable with no error. Now the enter lives in its own task and we only
|
||||
*optionally* wait on a ready signal — a hang becomes a logged warning + a
|
||||
backend that serves normally without MCP.
|
||||
"""
|
||||
stop = asyncio.Event()
|
||||
if session_manager is None:
|
||||
return None, stop, False
|
||||
ready = asyncio.Event()
|
||||
task = asyncio.create_task(_serve_mcp(session_manager, ready, stop))
|
||||
try:
|
||||
await asyncio.wait_for(ready.wait(), timeout=timeout)
|
||||
mounted = not task.done() # ready is also set on failure → not mounted
|
||||
except asyncio.TimeoutError:
|
||||
logger.warning(
|
||||
"MCP session manager did not signal ready within %.0fs (#632); "
|
||||
"serving without waiting. Set OMNIVOICE_MCP_START_TIMEOUT_S to adjust.",
|
||||
timeout,
|
||||
)
|
||||
mounted = False
|
||||
return task, stop, mounted
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
# Startup watchdog (#632): a silent hang during startup (e.g. a model-load /
|
||||
# MCP deadlock on some platforms) means "Application startup complete" never
|
||||
# logs and the app sits forever with no error. If startup hasn't finished
|
||||
# within the window, dump every thread's stack to stderr (→ backend_err.log)
|
||||
# so the hang point is captured instead of invisible. Cancelled the instant
|
||||
# startup completes, so a normal (even slow-download) boot never trips it.
|
||||
# Tune with OMNIVOICE_STARTUP_WATCHDOG_S (seconds; 0 disables). Best-effort —
|
||||
# never let the diagnostic itself break startup.
|
||||
_watchdog_armed = False
|
||||
try:
|
||||
import faulthandler
|
||||
_wd = float(os.environ.get("OMNIVOICE_STARTUP_WATCHDOG_S", "300"))
|
||||
if _wd > 0 and hasattr(faulthandler, "dump_traceback_later"):
|
||||
faulthandler.dump_traceback_later(_wd, repeat=False, exit=False)
|
||||
_watchdog_armed = True
|
||||
logger.info("Startup watchdog armed: thread dump if startup exceeds %.0fs (#632).", _wd)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
init_db()
|
||||
# Network sharing is loopback-only by default; the PIN middleware stays
|
||||
# inert until enable() sets a PIN. Seed the (disabled) state so the
|
||||
@@ -551,39 +422,9 @@ async def lifespan(app: FastAPI):
|
||||
capture_preload_task = asyncio.create_task(_preload_capture_asr())
|
||||
else:
|
||||
logger.info("Capture ASR preload disabled; dictation ASR will load on first use.")
|
||||
|
||||
# ── MCP session manager (Wave 2.2) ────────────────────────────────────
|
||||
# FastMCP's Streamable-HTTP transport needs its session manager running for
|
||||
# the lifetime of the app. Run it in its OWN task that owns the full
|
||||
# enter→exit lifecycle (anyio task-affinity, see _serve_mcp) and only wait,
|
||||
# with a timeout, for it to signal ready — so a hang on its anyio group
|
||||
# (observed on M1, #632) can never wedge "Application startup complete".
|
||||
_sm = getattr(app.state, "mcp_session_manager", None)
|
||||
mcp_task, mcp_stop, mcp_mounted = await _start_mcp_session_manager(
|
||||
_sm, timeout=_mcp_start_timeout_s()
|
||||
)
|
||||
if mcp_mounted:
|
||||
logger.info("MCP server mounted at /mcp")
|
||||
# Startup finished — disarm the hang watchdog before serving (#632).
|
||||
if _watchdog_armed:
|
||||
try:
|
||||
import faulthandler
|
||||
faulthandler.cancel_dump_traceback_later()
|
||||
except Exception:
|
||||
pass
|
||||
yield
|
||||
# ── Graceful shutdown (SIGTERM from Tauri, Ctrl+C, etc.) ────────────
|
||||
logger.info("Shutdown: cleaning up…")
|
||||
# Stop MCP first — signal its task to exit its own anyio context (correct
|
||||
# task-affinity), then bound the wait so a wedged manager can't hang exit.
|
||||
mcp_stop.set()
|
||||
if mcp_task is not None:
|
||||
try:
|
||||
await asyncio.wait_for(mcp_task, timeout=5.0)
|
||||
except (asyncio.TimeoutError, asyncio.CancelledError):
|
||||
pass
|
||||
except Exception:
|
||||
pass
|
||||
idle_task.cancel()
|
||||
worker_task.cancel()
|
||||
# Wait for tasks to finish their current iteration
|
||||
@@ -630,14 +471,6 @@ app = FastAPI(
|
||||
@app.get("/docs", include_in_schema=False)
|
||||
async def scalar_docs():
|
||||
"""Interactive API documentation powered by Scalar."""
|
||||
if get_scalar_api_reference is None:
|
||||
return JSONResponse(
|
||||
status_code=503,
|
||||
content={
|
||||
"detail": "API docs unavailable: scalar-fastapi is not installed "
|
||||
"in the backend environment (#307)."
|
||||
},
|
||||
)
|
||||
return get_scalar_api_reference(
|
||||
openapi_url=app.openapi_url,
|
||||
title=app.title,
|
||||
@@ -746,68 +579,6 @@ class NetworkAccessMiddleware:
|
||||
return await self.app(scope, receive, send)
|
||||
|
||||
|
||||
class BearerKeyMiddleware:
|
||||
"""When OMNIVOICE_API_KEY is set, non-loopback clients must present it on
|
||||
every HTTP + WebSocket request: ``Authorization: Bearer <key>``,
|
||||
``?api_key=<key>`` (browser WebSockets cannot set headers), or the
|
||||
``ov_key`` cookie (set on the first successful HTTP auth). Loopback
|
||||
always bypasses — the desktop default is unchanged — and the SPA shell
|
||||
paths stay reachable so a remote UI can load and show what's wrong.
|
||||
|
||||
Inert when the env var is unset (the default). Pure ASGI for the same
|
||||
no-buffering reason as NetworkAccessMiddleware above. Plain-HTTP caveat
|
||||
is documented in docs/remote-gpu.md: the key is sniffable outside a
|
||||
WireGuard (Tailscale) or TLS (tailscale serve) transport.
|
||||
"""
|
||||
|
||||
def __init__(self, app):
|
||||
self.app = app
|
||||
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope["type"] not in ("http", "websocket"):
|
||||
return await self.app(scope, receive, send)
|
||||
key = os.environ.get("OMNIVOICE_API_KEY") or ""
|
||||
if not key:
|
||||
return await self.app(scope, receive, send)
|
||||
client = scope["client"][0] if scope.get("client") else None
|
||||
if client in _LOOPBACK_CLIENTS:
|
||||
return await self.app(scope, receive, send)
|
||||
path = scope.get("path", "")
|
||||
if scope["type"] == "http" and (
|
||||
path in _SHELL_PATHS or path.startswith("/assets/") or path.startswith("/favicon")
|
||||
):
|
||||
return await self.app(scope, receive, send)
|
||||
|
||||
from starlette.requests import HTTPConnection
|
||||
|
||||
conn = HTTPConnection(scope)
|
||||
auth = conn.headers.get("authorization", "")
|
||||
supplied = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
|
||||
if not supplied:
|
||||
supplied = conn.query_params.get("api_key") or conn.cookies.get("ov_key") or ""
|
||||
|
||||
if not secrets.compare_digest(supplied, key):
|
||||
if scope["type"] == "websocket":
|
||||
# Reject the handshake; 1008 = policy violation.
|
||||
await receive() # consume websocket.connect
|
||||
await send({"type": "websocket.close", "code": 1008})
|
||||
return
|
||||
resp = JSONResponse({"detail": "API key required"}, status_code=401)
|
||||
return await resp(scope, receive, send)
|
||||
|
||||
if scope["type"] == "http" and conn.cookies.get("ov_key") != key:
|
||||
async def send_with_cookie(message):
|
||||
if message["type"] == "http.response.start":
|
||||
headers = MutableHeaders(scope=message)
|
||||
headers.append(
|
||||
"set-cookie", f"ov_key={key}; Path=/; SameSite=Lax"
|
||||
)
|
||||
await send(message)
|
||||
|
||||
return await self.app(scope, receive, send_with_cookie)
|
||||
return await self.app(scope, receive, send)
|
||||
|
||||
|
||||
# UI dev-server port — single-sourced from OMNIVOICE_UI_PORT so a user who
|
||||
# moves the Vite dev server off 3901 still gets a matching CORS allow-list.
|
||||
def _ui_port() -> int:
|
||||
@@ -839,29 +610,6 @@ app.add_middleware(
|
||||
# applied even to the 401 PIN-required responses). Inert unless a PIN is set.
|
||||
app.add_middleware(NetworkAccessMiddleware)
|
||||
|
||||
# Remote-backend bearer gate (parity program Wave 2.3 / §R2). Inert unless
|
||||
# OMNIVOICE_API_KEY is set. Distinct from the PIN gate above: the PIN guards
|
||||
# casual LAN-share guests for one session; the API key is the durable
|
||||
# credential for running this backend remotely (Tailscale / Docker GPU box).
|
||||
# Covers WebSockets too — the PIN gate never did, because every WS endpoint
|
||||
# carried its own loopback guard; remote mode is exactly the case where a
|
||||
# keyed non-loopback client must reach them.
|
||||
app.add_middleware(BearerKeyMiddleware)
|
||||
|
||||
# Register canonical audio MIME types before any StaticFiles mount.
|
||||
# Python's `mimetypes.guess_type()` returns `audio/x-wav` for `.wav` and
|
||||
# `audio/x-flac` for `.flac` on most platforms — these are vendor-experimental
|
||||
# (x- prefix, never IANA-registered). macOS Chrome/Safari MIME-sniff leniently
|
||||
# via CoreAudio so playback works there, but Linux Chrome/Firefox (FFmpeg) and
|
||||
# Android Chrome (ExoPlayer) strictly honor the declared type and treat the
|
||||
# x- variants as download-only — manifesting as the play button silently
|
||||
# doing nothing in the browser app while working in the Tauri desktop shell.
|
||||
# `audio/wav` / `audio/flac` are the IANA-canonical types.
|
||||
# Ref: https://www.iana.org/assignments/media-types/media-types.xhtml#audio
|
||||
import mimetypes as _mimetypes
|
||||
_mimetypes.add_type("audio/wav", ".wav")
|
||||
_mimetypes.add_type("audio/flac", ".flac")
|
||||
|
||||
app.mount("/audio", StaticFiles(directory=OUTPUTS_DIR), name="audio")
|
||||
app.mount("/voice_audio", StaticFiles(directory=VOICES_DIR), name="voice_audio")
|
||||
|
||||
@@ -904,44 +652,17 @@ app.include_router(stories.router)
|
||||
app.include_router(setup.router)
|
||||
app.include_router(gallery.router)
|
||||
app.include_router(archetypes.router)
|
||||
app.include_router(describe_voice.router) # issue #317: free-text voice design
|
||||
app.include_router(community.router)
|
||||
app.include_router(batch.router)
|
||||
app.include_router(watermark.router)
|
||||
app.include_router(events.router)
|
||||
app.include_router(capture.router)
|
||||
app.include_router(capture_ws.router)
|
||||
app.include_router(dictation.router)
|
||||
app.include_router(openai_compat.router)
|
||||
app.include_router(tts_stream.router)
|
||||
app.include_router(marketplace.router)
|
||||
app.include_router(personas.router)
|
||||
app.include_router(sonitranslate.router)
|
||||
app.include_router(audiobook.router)
|
||||
app.include_router(longform_jobs.router)
|
||||
app.include_router(pronunciation.router) # Expressive-TTS Spec 01: pronunciation dictionary
|
||||
app.include_router(settings_router.router) # Phase 1 AUTH-03 endpoints
|
||||
from api.routers import mcp_bindings as _mcp_bindings_router # noqa: E402
|
||||
app.include_router(_mcp_bindings_router.router) # Wave 2.2 per-agent voice bindings
|
||||
|
||||
# ── Mount the MCP server (Wave 2.2) ───────────────────────────────────────
|
||||
# FastMCP's Streamable-HTTP app is sub-mounted at /mcp; its session manager is
|
||||
# stashed on app.state for the lifespan above to run. Opt-out via
|
||||
# OMNIVOICE_MCP_DISABLE=1; best-effort so a missing mcp package or a build
|
||||
# without it never breaks startup.
|
||||
if os.environ.get("OMNIVOICE_MCP_DISABLE", "").strip().lower() not in ("1", "true", "yes", "on"):
|
||||
try:
|
||||
from mcp_server import create_mcp_server
|
||||
|
||||
_mcp = create_mcp_server()
|
||||
_mcp_app = _mcp.streamable_http_app()
|
||||
app.state.mcp_session_manager = _mcp.session_manager
|
||||
app.mount("/mcp", _mcp_app)
|
||||
logging.getLogger("omnivoice.api").info("MCP app mounted at /mcp")
|
||||
except Exception as _mcp_err: # noqa: BLE001
|
||||
logging.getLogger("omnivoice.api").info(
|
||||
"MCP server not mounted (%s); /mcp disabled.", _mcp_err
|
||||
)
|
||||
|
||||
frontend_path = os.path.join(os.path.dirname(__file__), "..", "frontend", "dist")
|
||||
if os.path.exists(frontend_path):
|
||||
|
||||
+1
-64
@@ -53,14 +53,6 @@ def create_mcp_server():
|
||||
"voice design, and video dubbing in 646 languages."
|
||||
),
|
||||
)
|
||||
# Serve the Streamable-HTTP transport at the app root so mounting the whole
|
||||
# app at "/mcp" on the main FastAPI yields the endpoint at "/mcp". FastMCP's
|
||||
# default path is "/mcp", which would double-prefix to "/mcp/mcp" when
|
||||
# sub-mounted. Harmless for the standalone CLI run() path.
|
||||
try:
|
||||
mcp.settings.streamable_http_path = "/"
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# ── Helpers ─────────────────────────────────────────────────────────
|
||||
|
||||
@@ -83,21 +75,6 @@ def create_mcp_server():
|
||||
|
||||
# ── Tools ───────────────────────────────────────────────────────────
|
||||
|
||||
def _current_client_id() -> str | None:
|
||||
"""The X-OmniVoice-Client-Id of the calling MCP client, if any.
|
||||
|
||||
FastMCP exposes the HTTP request via its request context on the
|
||||
Streamable-HTTP transport; stdio clients (and any version where the
|
||||
accessor differs) simply resolve to None and fall back to the
|
||||
global default voice."""
|
||||
try:
|
||||
req = mcp.get_context().request_context.request
|
||||
if req is not None:
|
||||
return req.headers.get("x-omnivoice-client-id")
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
@mcp.tool()
|
||||
async def generate_speech(
|
||||
text: str,
|
||||
@@ -112,8 +89,7 @@ def create_mcp_server():
|
||||
Args:
|
||||
text: The text to synthesize into speech.
|
||||
language: Target language (ISO code or 'Auto'). 646 languages supported.
|
||||
profile_id: ID of a saved voice profile to clone. Omit to use this
|
||||
agent's bound voice (Settings → MCP), else the global default.
|
||||
profile_id: ID of a saved voice profile to clone. Omit for voice design mode.
|
||||
instruct: Style instruction (e.g. 'whisper', 'excited', 'narrator').
|
||||
speed: Speech speed multiplier (0.5–2.0, default 1.0).
|
||||
steps: Diffusion steps (8=fast/draft, 16=balanced, 32=quality).
|
||||
@@ -122,17 +98,6 @@ def create_mcp_server():
|
||||
JSON with audio_id, generation_time, audio_duration, and
|
||||
base64-encoded WAV data.
|
||||
"""
|
||||
# Per-agent voice binding (Wave 2.2): explicit arg wins; otherwise
|
||||
# resolve this client's bound profile, then the global default.
|
||||
client_id = _current_client_id()
|
||||
try:
|
||||
from services import mcp_bindings
|
||||
resolved = mcp_bindings.resolve_voice(client_id, profile_id)
|
||||
profile_id = resolved.get("profile_id")
|
||||
mcp_bindings.touch_last_seen(client_id) if client_id else None
|
||||
except Exception:
|
||||
pass # binding layer unavailable — use whatever was passed
|
||||
|
||||
form = {
|
||||
"text": text,
|
||||
"language": language,
|
||||
@@ -194,34 +159,6 @@ def create_mcp_server():
|
||||
'],"note":"Pass any ISO 639 code or set language=Auto for detection."}'
|
||||
)
|
||||
|
||||
@mcp.tool()
|
||||
async def transcribe(audio_base64: str, language: str | None = None) -> str:
|
||||
"""Transcribe spoken audio to text.
|
||||
|
||||
Args:
|
||||
audio_base64: Base64-encoded audio bytes (wav/mp3/webm/m4a).
|
||||
language: Optional language hint; omit for auto-detect.
|
||||
|
||||
Returns:
|
||||
JSON with the recognized text, language, and duration.
|
||||
"""
|
||||
try:
|
||||
raw = base64.b64decode(audio_base64, validate=True)
|
||||
except Exception:
|
||||
return '{"error":"audio_base64 is not valid base64"}'
|
||||
# 200 MB cap — same spirit as voicebox's transcribe gate. Keeps a
|
||||
# buggy/hostile agent from posting an unbounded blob.
|
||||
if len(raw) > 200 * 1024 * 1024:
|
||||
return '{"error":"audio exceeds 200 MB limit"}'
|
||||
data = {}
|
||||
if language:
|
||||
data["language"] = language
|
||||
r = await _api_post_form(
|
||||
"/transcribe", data=data,
|
||||
files={"audio": ("audio.wav", raw, "application/octet-stream")},
|
||||
)
|
||||
return str(r.json())
|
||||
|
||||
@mcp.tool()
|
||||
async def check_health() -> str:
|
||||
"""Check if the OmniVoice backend is running and what GPU device is active."""
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
"""omnivoice-mcp — stdio MCP shim for clients that only speak stdio."""
|
||||
@@ -1,176 +0,0 @@
|
||||
"""omnivoice-mcp — stdio ↔ Streamable-HTTP MCP proxy (Wave 2.2).
|
||||
|
||||
Adapted from voicebox (https://github.com/jamiepine/voicebox), MIT License,
|
||||
Copyright (c) voicebox contributors.
|
||||
|
||||
Some MCP clients only speak stdio. They spawn this binary; we pipe each
|
||||
JSON-RPC message to ``http://127.0.0.1:<port>/mcp/`` (the FastMCP app mounted
|
||||
on the running OmniVoice backend) and stream the server's response back.
|
||||
|
||||
Environment variables:
|
||||
OMNIVOICE_PORT backend port (default 3900).
|
||||
OMNIVOICE_HOST host (default 127.0.0.1).
|
||||
OMNIVOICE_CLIENT_ID forwarded as X-OmniVoice-Client-Id on every request
|
||||
(drives per-agent voice binding).
|
||||
|
||||
Stdout is JSON-RPC only. Diagnostics go to stderr.
|
||||
Exit 0 on clean EOF, 1 on transport error, 2 if the backend never answers.
|
||||
|
||||
Usage in an MCP client config (stdio):
|
||||
command: python
|
||||
args: ["-m", "backend.mcp_shim"]
|
||||
env: { OMNIVOICE_CLIENT_ID: "claude-code" }
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
CLIENT_ID_HEADER = "X-OmniVoice-Client-Id"
|
||||
SESSION_HEADER = "mcp-session-id"
|
||||
HEALTH_TIMEOUT_S = 30.0
|
||||
DEFAULT_PORT = 3900
|
||||
|
||||
|
||||
def _err(msg: str) -> None:
|
||||
print(f"omnivoice-mcp: {msg}", file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def _base_url() -> tuple[str, str]:
|
||||
host = os.environ.get("OMNIVOICE_HOST", "127.0.0.1")
|
||||
port = int(os.environ.get("OMNIVOICE_PORT", str(DEFAULT_PORT)))
|
||||
return f"http://{host}:{port}/mcp/", f"http://{host}:{port}/health"
|
||||
|
||||
|
||||
async def _wait_for_backend(client: httpx.AsyncClient, health_url: str) -> bool:
|
||||
loop = asyncio.get_running_loop()
|
||||
deadline = loop.time() + HEALTH_TIMEOUT_S
|
||||
while loop.time() < deadline:
|
||||
try:
|
||||
r = await client.get(health_url, timeout=2.0)
|
||||
if r.status_code == 200:
|
||||
return True
|
||||
except Exception:
|
||||
pass
|
||||
await asyncio.sleep(0.5)
|
||||
return False
|
||||
|
||||
|
||||
async def _read_stdin_line() -> str | None:
|
||||
loop = asyncio.get_running_loop()
|
||||
line = await loop.run_in_executor(None, sys.stdin.readline)
|
||||
return line or None
|
||||
|
||||
|
||||
def _write_stdout(obj: Any) -> None:
|
||||
sys.stdout.write(json.dumps(obj, separators=(",", ":")))
|
||||
sys.stdout.write("\n")
|
||||
sys.stdout.flush()
|
||||
|
||||
|
||||
async def _handle_request(
|
||||
client: httpx.AsyncClient,
|
||||
url: str,
|
||||
raw: str,
|
||||
headers: dict[str, str],
|
||||
session_id: list[str | None],
|
||||
) -> None:
|
||||
try:
|
||||
message = json.loads(raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
_err(f"invalid JSON on stdin: {exc}")
|
||||
return
|
||||
|
||||
req_headers = {
|
||||
"Content-Type": "application/json",
|
||||
"Accept": "application/json, text/event-stream",
|
||||
**headers,
|
||||
}
|
||||
if session_id[0]:
|
||||
req_headers[SESSION_HEADER] = session_id[0]
|
||||
|
||||
is_notification = isinstance(message, dict) and "id" not in message
|
||||
|
||||
async with client.stream("POST", url, headers=req_headers, content=raw.encode("utf-8")) as response:
|
||||
if session_id[0] is None:
|
||||
sid = response.headers.get(SESSION_HEADER)
|
||||
if sid:
|
||||
session_id[0] = sid
|
||||
|
||||
if response.status_code == 202:
|
||||
return # notification acknowledged
|
||||
if response.status_code >= 400:
|
||||
body = await response.aread()
|
||||
_err(f"server {response.status_code}: {body.decode('utf-8', errors='replace')[:400]}")
|
||||
if is_notification:
|
||||
return
|
||||
_write_stdout({
|
||||
"jsonrpc": "2.0",
|
||||
"id": message.get("id"),
|
||||
"error": {"code": -32000, "message": f"OmniVoice MCP proxy got HTTP {response.status_code}"},
|
||||
})
|
||||
return
|
||||
|
||||
ctype = response.headers.get("content-type", "")
|
||||
if "text/event-stream" in ctype:
|
||||
async for line in response.aiter_lines():
|
||||
if line.startswith("data:"):
|
||||
payload = line[5:].strip()
|
||||
if not payload:
|
||||
continue
|
||||
try:
|
||||
_write_stdout(json.loads(payload))
|
||||
except json.JSONDecodeError:
|
||||
_err(f"malformed SSE payload: {payload[:200]}")
|
||||
else:
|
||||
body = await response.aread()
|
||||
try:
|
||||
_write_stdout(json.loads(body))
|
||||
except json.JSONDecodeError:
|
||||
_err(f"non-JSON response ({ctype}): {body.decode('utf-8', errors='replace')[:200]}")
|
||||
|
||||
|
||||
async def _run() -> int:
|
||||
url, health_url = _base_url()
|
||||
forward_headers: dict[str, str] = {}
|
||||
client_id = os.environ.get("OMNIVOICE_CLIENT_ID")
|
||||
if client_id:
|
||||
forward_headers[CLIENT_ID_HEADER] = client_id
|
||||
|
||||
session_id: list[str | None] = [None]
|
||||
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(300.0)) as client:
|
||||
if not await _wait_for_backend(client, health_url):
|
||||
_err(f"timed out waiting for OmniVoice at {health_url} — is the app running?")
|
||||
return 2
|
||||
try:
|
||||
while True:
|
||||
line = await _read_stdin_line()
|
||||
if line is None:
|
||||
return 0
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
await _handle_request(client, url, line, forward_headers, session_id)
|
||||
except (KeyboardInterrupt, SystemExit):
|
||||
return 0
|
||||
except Exception as exc:
|
||||
_err(f"proxy failed: {exc!r}")
|
||||
return 1
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
return asyncio.run(_run())
|
||||
except KeyboardInterrupt:
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,58 +0,0 @@
|
||||
"""Parity program Wave 0.2: consent-locked voice profiles
|
||||
|
||||
Revision ID: 0003_voice_profile_consent
|
||||
Revises: 0002_voice_profile_demo_fields
|
||||
Create Date: 2026-06-12 00:00:00.000000
|
||||
|
||||
Adds four additive columns to ``voice_profiles`` backing the
|
||||
``verified_own_voice`` consent lock (docs/competitive-analysis.md Action 22 /
|
||||
parity program Wave 0.2). A profile becomes "verified" when its owner records
|
||||
a spoken consent statement; agentic features and gallery sharing will require
|
||||
the flag — plain local synthesis never does.
|
||||
|
||||
* ``verified_own_voice INTEGER DEFAULT 0`` — the consent lock itself.
|
||||
* ``consent_text TEXT DEFAULT ''`` — the statement that was read aloud.
|
||||
* ``consent_audio_path TEXT DEFAULT ''`` — filename of the recorded
|
||||
statement in VOICES_DIR (kept as provenance, deletable via revoke).
|
||||
* ``consent_recorded_at REAL DEFAULT NULL`` — UNIX timestamp.
|
||||
|
||||
Behavior mirrors 0002: ``_has_column`` PRAGMA guards make upgrade a no-op on
|
||||
fresh installs (where _BASE_SCHEMA already has the columns), satisfying the
|
||||
"Backward-compatible project data" constraint; downgrade drops the columns
|
||||
(SQLite >= 3.35).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = "0003_voice_profile_consent"
|
||||
down_revision: Union[str, None] = "0002_voice_profile_demo_fields"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
_COLUMNS = (
|
||||
("verified_own_voice", sa.Column("verified_own_voice", sa.Integer(), nullable=False, server_default="0")),
|
||||
("consent_text", sa.Column("consent_text", sa.Text(), nullable=False, server_default="")),
|
||||
("consent_audio_path", sa.Column("consent_audio_path", sa.Text(), nullable=False, server_default="")),
|
||||
("consent_recorded_at", sa.Column("consent_recorded_at", sa.Float(), nullable=True)),
|
||||
)
|
||||
|
||||
|
||||
def _has_column(table: str, column: str) -> bool:
|
||||
bind = op.get_bind()
|
||||
rows = bind.execute(sa.text(f"PRAGMA table_info({table})")).fetchall()
|
||||
return any(r[1] == column for r in rows)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
for name, column in _COLUMNS:
|
||||
if not _has_column("voice_profiles", name):
|
||||
op.add_column("voice_profiles", column)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
for name, _ in reversed(_COLUMNS):
|
||||
if _has_column("voice_profiles", name):
|
||||
op.drop_column("voice_profiles", name)
|
||||
@@ -1,60 +0,0 @@
|
||||
"""Parity program Wave 2.2: per-agent MCP voice bindings
|
||||
|
||||
Revision ID: 0004_mcp_client_bindings
|
||||
Revises: 0003_voice_profile_consent
|
||||
Create Date: 2026-06-12 00:00:00.000000
|
||||
|
||||
Adds the ``mcp_client_bindings`` table backing per-agent voice binding
|
||||
(docs/competitive-analysis.md Spec 2): each MCP client (identified by the
|
||||
``X-OmniVoice-Client-Id`` header it sends) can be bound to a default voice
|
||||
profile / engine, so "Claude Code speaks in Morgan, Cursor in Scarlett".
|
||||
|
||||
* ``client_id`` TEXT PRIMARY KEY — the agent's stable id.
|
||||
* ``label`` TEXT — human label shown in Settings.
|
||||
* ``profile_id`` TEXT — voice profile to speak in (nullable FK-by-convention).
|
||||
* ``default_engine`` TEXT — engine override (nullable).
|
||||
* ``last_seen_at`` REAL — updated when the client calls a tool.
|
||||
* ``created_at`` REAL.
|
||||
|
||||
Additive + idempotent (guarded by sqlite_master), matching 0002/0003, so
|
||||
re-running on a fresh-install DB where _BASE_SCHEMA already created it is a
|
||||
no-op (Backward-compatible project data constraint).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = "0004_mcp_client_bindings"
|
||||
down_revision: Union[str, None] = "0003_voice_profile_consent"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def _has_table(name: str) -> bool:
|
||||
bind = op.get_bind()
|
||||
row = bind.execute(
|
||||
sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name=:n"),
|
||||
{"n": name},
|
||||
).fetchone()
|
||||
return row is not None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if _has_table("mcp_client_bindings"):
|
||||
return
|
||||
op.create_table(
|
||||
"mcp_client_bindings",
|
||||
sa.Column("client_id", sa.Text(), primary_key=True),
|
||||
sa.Column("label", sa.Text(), nullable=False, server_default=""),
|
||||
sa.Column("profile_id", sa.Text(), nullable=True),
|
||||
sa.Column("default_engine", sa.Text(), nullable=True),
|
||||
sa.Column("last_seen_at", sa.Float(), nullable=True),
|
||||
sa.Column("created_at", sa.Float(), nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_table("mcp_client_bindings"):
|
||||
op.drop_table("mcp_client_bindings")
|
||||
@@ -1,65 +0,0 @@
|
||||
"""Voice Studio unification: profile `kind` discriminator + design params
|
||||
|
||||
Revision ID: 0005_unified_profiles
|
||||
Revises: 0004_mcp_client_bindings
|
||||
Create Date: 2026-06-13 00:00:00.000000
|
||||
|
||||
Adds two additive columns to ``voice_profiles`` so a *designed* voice
|
||||
(category sliders + instruct, no user reference audio) is a first-class
|
||||
profile rather than a transient UI state
|
||||
(docs/specs/voice-studio-unification.md §3):
|
||||
|
||||
* ``kind TEXT DEFAULT 'clone'`` — ``'clone'`` (user reference audio) or
|
||||
``'design'`` (rendered sample + stored design params). Replaces the
|
||||
brittle is_locked/instruct inference in /generate.
|
||||
* ``vd_states TEXT DEFAULT NULL`` — JSON of the design category picks
|
||||
(Gender/Age/Pitch/Style/accent/dialect) so selecting a design profile
|
||||
can restore the sliders for re-editing.
|
||||
|
||||
Backfill: every existing row becomes ``kind='clone'`` — all of them carry a
|
||||
real or rendered ``ref_audio_path`` today (archetype materialization
|
||||
included), so the default is semantically true and no audio is re-rendered.
|
||||
|
||||
Behavior mirrors 0002/0003: ``_has_column`` PRAGMA guards make upgrade a
|
||||
no-op on fresh installs (where _BASE_SCHEMA already has the columns),
|
||||
satisfying the "Backward-compatible project data" constraint; downgrade
|
||||
drops the columns (SQLite >= 3.35).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = "0005_unified_profiles"
|
||||
down_revision: Union[str, None] = "0004_mcp_client_bindings"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def _has_column(table: str, column: str) -> bool:
|
||||
bind = op.get_bind()
|
||||
rows = bind.execute(sa.text(f"PRAGMA table_info({table})")).fetchall()
|
||||
return any(r[1] == column for r in rows)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if not _has_column("voice_profiles", "kind"):
|
||||
op.add_column(
|
||||
"voice_profiles",
|
||||
sa.Column("kind", sa.Text(), nullable=False, server_default="clone"),
|
||||
)
|
||||
# server_default covers new rows; make existing rows explicit too.
|
||||
op.execute("UPDATE voice_profiles SET kind='clone' WHERE kind IS NULL OR kind=''")
|
||||
if not _has_column("voice_profiles", "vd_states"):
|
||||
op.add_column(
|
||||
"voice_profiles",
|
||||
sa.Column("vd_states", sa.Text(), nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_column("voice_profiles", "vd_states"):
|
||||
op.drop_column("voice_profiles", "vd_states")
|
||||
if _has_column("voice_profiles", "kind"):
|
||||
op.drop_column("voice_profiles", "kind")
|
||||
@@ -1,36 +0,0 @@
|
||||
"""Heal voice_profiles.instruct poisoned with the "[object Object]" sentinel.
|
||||
|
||||
Revision ID: 0006_strip_object_object_instruct
|
||||
Revises: 0005_unified_profiles
|
||||
Create Date: 2026-06-20 00:00:00.000000
|
||||
|
||||
A pre-fix Voice Studio build ("Save design as profile") passed the
|
||||
``buildDesignInstruct()`` *object* straight to FormData, which string-coerced it
|
||||
to the literal ``"[object Object]"`` and persisted that into
|
||||
``voice_profiles.instruct`` (#550 #545 #542 #537 #530 #525). On first
|
||||
preview/use that value fails the engine instruct validator with a 400. The
|
||||
frontend + backend fixes stop any NEW poisoned rows; this migration heals the
|
||||
ones already saved on the buggy build (the local-first backward-compat rule —
|
||||
existing project data must keep working without manual migration).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy import inspect
|
||||
|
||||
revision: str = "0006_strip_object_object_instruct"
|
||||
down_revision: Union[str, None] = "0005_unified_profiles"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
bind = op.get_bind()
|
||||
if "voice_profiles" in inspect(bind).get_table_names():
|
||||
# Idempotent: only touches rows whose instruct is literally the sentinel.
|
||||
op.execute("UPDATE voice_profiles SET instruct='' WHERE instruct='[object Object]'")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# Irreversible heal — the original garbage sentinel is not worth restoring.
|
||||
pass
|
||||
@@ -1,118 +0,0 @@
|
||||
"""Rebuild design-profile instructs poisoned with prose / "[object Object]".
|
||||
|
||||
Revision ID: 0007_rebuild_poisoned_design_instruct
|
||||
Revises: 0006_strip_object_object_instruct
|
||||
Create Date: 2026-06-22 00:00:00.000000
|
||||
|
||||
Migration 0006 *blanked* the literal ``"[object Object]"`` sentinel. That stops
|
||||
the 400 on use, but it also throws away the designed voice: a row that read
|
||||
``"[object Object]"`` (or freeform prose like "A gentle, quiet male voice…")
|
||||
becomes ``instruct=''`` and then renders with the engine's neutral default —
|
||||
which is why an Indonesian *female* designed voice came out *male* (#594), and
|
||||
why prose-poisoned designs still 400 (#571 #596).
|
||||
|
||||
This migration heals it properly: for every design profile it recomputes a
|
||||
validator-safe instruct, preferring any whitelist tags already in the stored
|
||||
value and otherwise rebuilding the tags from ``vd_states`` (the authoritative
|
||||
category→pick map the Voice Design picker persists). Non-design rows simply get
|
||||
their instruct sanitized (poison dropped). Idempotent — a healthy row is left
|
||||
byte-for-byte unchanged, so re-running is a no-op.
|
||||
|
||||
Self-contained by design: alembic migrations must not import evolving app code
|
||||
(``omnivoice`` would also drag in torch at startup), so the tag whitelist is a
|
||||
frozen snapshot of ``omnivoice.utils.voice_design._INSTRUCT_ALL_VALID``.
|
||||
``tests/test_migration_0007_instruct_rebuild.py`` asserts the snapshot stays in
|
||||
sync with the canonical set.
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy import inspect
|
||||
|
||||
revision: str = "0007_rebuild_poisoned_design_instruct"
|
||||
down_revision: Union[str, None] = "0006_strip_object_object_instruct"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
# Frozen snapshot of the design-instruct whitelist + mutually-exclusive
|
||||
# categories (omnivoice/utils/voice_design.py). Kept self-contained so the
|
||||
# migration's behaviour is pinned to the data it heals, not to future vocab
|
||||
# edits. Parity is guarded by the migration test.
|
||||
_CATEGORIES = [
|
||||
{"male", "女", "female", "男"},
|
||||
{"child", "teenager", "young adult", "middle-aged", "elderly",
|
||||
"儿童", "少年", "青年", "中年", "老年"},
|
||||
{"very low pitch", "low pitch", "moderate pitch", "high pitch", "very high pitch",
|
||||
"极低音调", "低音调", "中音调", "高音调", "极高音调"},
|
||||
{"whisper", "耳语"},
|
||||
{"american accent", "british accent", "australian accent", "chinese accent",
|
||||
"canadian accent", "indian accent", "korean accent", "portuguese accent",
|
||||
"russian accent", "japanese accent"},
|
||||
{"河南话", "陕西话", "四川话", "贵州话", "云南话", "桂林话",
|
||||
"济南话", "石家庄话", "甘肃话", "宁夏话", "青岛话", "东北话"},
|
||||
]
|
||||
_ALL_VALID = set().union(*_CATEGORIES)
|
||||
|
||||
|
||||
def _valid_from_items(items) -> str:
|
||||
"""One whitelist tag per category, first-seen order; everything else dropped."""
|
||||
seen = set()
|
||||
out = []
|
||||
for raw in items:
|
||||
tag = str(raw if raw is not None else "").strip().lower()
|
||||
if not tag or tag not in _ALL_VALID:
|
||||
continue
|
||||
ci = next((i for i, c in enumerate(_CATEGORIES) if tag in c), -1)
|
||||
if ci in seen:
|
||||
continue
|
||||
seen.add(ci)
|
||||
out.append(tag)
|
||||
return ", ".join(out)
|
||||
|
||||
|
||||
def _heal(instruct, vd_states, is_design) -> str:
|
||||
healed = _valid_from_items(re.split(r"\s*[,,]\s*", str(instruct or "").strip()))
|
||||
if healed or not is_design:
|
||||
return healed
|
||||
# Stored instruct was all-poison — recover the design from vd_states.
|
||||
if not vd_states:
|
||||
return ""
|
||||
try:
|
||||
vd = json.loads(vd_states)
|
||||
except (ValueError, TypeError):
|
||||
return ""
|
||||
return _valid_from_items(vd.values()) if isinstance(vd, dict) else ""
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
bind = op.get_bind()
|
||||
insp = inspect(bind)
|
||||
if "voice_profiles" not in insp.get_table_names():
|
||||
return
|
||||
cols = {c["name"] for c in insp.get_columns("voice_profiles")}
|
||||
has_kind = "kind" in cols
|
||||
has_vd = "vd_states" in cols
|
||||
|
||||
select = "SELECT id, instruct"
|
||||
select += ", kind" if has_kind else ""
|
||||
select += ", vd_states" if has_vd else ""
|
||||
select += " FROM voice_profiles"
|
||||
|
||||
for row in bind.exec_driver_sql(select).mappings().all():
|
||||
instruct = row["instruct"] or ""
|
||||
is_design = (row["kind"] == "design") if has_kind else bool(instruct)
|
||||
vd = row["vd_states"] if has_vd else None
|
||||
healed = _heal(instruct, vd, is_design)
|
||||
if healed != instruct:
|
||||
bind.exec_driver_sql(
|
||||
"UPDATE voice_profiles SET instruct = ? WHERE id = ?",
|
||||
(healed, row["id"]),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# Irreversible heal — the original poisoned value isn't worth restoring.
|
||||
pass
|
||||
@@ -1,67 +0,0 @@
|
||||
"""Expressive-TTS Spec 01 Phase 1: user pronunciation dictionary
|
||||
|
||||
Revision ID: 0008_pronunciation_dictionary
|
||||
Revises: 0007_rebuild_poisoned_design_instruct
|
||||
Create Date: 2026-06-25 00:00:00.000000
|
||||
|
||||
Adds the ``pronunciation_entries`` table backing the user-editable, per-language
|
||||
pronunciation dictionary (Settings → Pronunciation). Each row maps a ``term`` to
|
||||
a ``replacement`` the engine pronounces correctly, scoped global (``language='*'``)
|
||||
or to a 2-letter language. Applied as pure text substitution before synthesis, so
|
||||
every engine honors it.
|
||||
|
||||
* ``id`` TEXT PRIMARY KEY — stable row id.
|
||||
* ``term`` TEXT — the word/phrase to match (whole-word, case-insensitive).
|
||||
* ``replacement`` TEXT — the respelling (or, for phoneme rows, the markup).
|
||||
* ``type`` TEXT — 'respelling' | 'ipa' | 'cmu'.
|
||||
* ``language`` TEXT — '*' = global, else a language code (e.g. 'en', 'de').
|
||||
* ``enabled`` INTEGER — 1 = applied, 0 = parked.
|
||||
* ``created_at`` REAL.
|
||||
|
||||
Additive + idempotent (guarded by sqlite_master), matching 0002/0003/0004, so
|
||||
re-running on a fresh-install DB where ``_BASE_SCHEMA`` already created the table
|
||||
is a no-op (Backward-compatible project data constraint). The same table is
|
||||
mirrored into ``core/db.py::_BASE_SCHEMA`` so fresh installs and migrated DBs
|
||||
converge on an identical end-state (the dual-path discipline).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = "0008_pronunciation_dictionary"
|
||||
down_revision: Union[str, None] = "0007_rebuild_poisoned_design_instruct"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def _has_table(name: str) -> bool:
|
||||
bind = op.get_bind()
|
||||
row = bind.execute(
|
||||
sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name=:n"),
|
||||
{"n": name},
|
||||
).fetchone()
|
||||
return row is not None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if _has_table("pronunciation_entries"):
|
||||
return
|
||||
op.create_table(
|
||||
"pronunciation_entries",
|
||||
sa.Column("id", sa.Text(), primary_key=True),
|
||||
sa.Column("term", sa.Text(), nullable=False),
|
||||
sa.Column("replacement", sa.Text(), nullable=False, server_default=""),
|
||||
sa.Column("type", sa.Text(), nullable=False, server_default="respelling"),
|
||||
sa.Column("language", sa.Text(), nullable=False, server_default="*"),
|
||||
sa.Column("enabled", sa.Integer(), nullable=False, server_default="1"),
|
||||
sa.Column("created_at", sa.Float(), nullable=True),
|
||||
)
|
||||
op.create_index("idx_pron_lang", "pronunciation_entries", ["language"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_table("pronunciation_entries"):
|
||||
op.drop_index("idx_pron_lang", table_name="pronunciation_entries")
|
||||
op.drop_table("pronunciation_entries")
|
||||
@@ -25,11 +25,6 @@ class DubSegment(BaseModel):
|
||||
speed: Optional[float] = None
|
||||
gain: Optional[float] = None # Per-segment volume (0.0 - 2.0, default 1.0)
|
||||
target_lang: Optional[str] = None # Per-segment language override (ISO code)
|
||||
# Phase 4.2 free-form directorial note ("urgent, whispered…"). The client
|
||||
# has always sent this; without the field pydantic silently dropped it,
|
||||
# so directions never reached TTS and never entered the regen
|
||||
# fingerprint (#281).
|
||||
direction: Optional[str] = None
|
||||
effect_preset: str = "broadcast" # NEW: DSP preset id (default: broadcast)
|
||||
|
||||
@field_validator("effect_preset")
|
||||
@@ -42,19 +37,6 @@ class DubSegment(BaseModel):
|
||||
)
|
||||
return v
|
||||
|
||||
class FitOptions(BaseModel):
|
||||
"""Optional knob overrides for the `smart_fit` timing strategy.
|
||||
|
||||
All fields default to None — the server fills in the canonical
|
||||
defaults (services.fit_planner.FitParams) so old clients and sparse
|
||||
payloads behave identically to a fully-populated default payload.
|
||||
"""
|
||||
max_audio_only_rate: Optional[float] = None # default 1.2
|
||||
audio_rate_cap: Optional[float] = None # default 1.5
|
||||
video_slow_cap: Optional[float] = None # default 2.0
|
||||
gap_guard_s: Optional[float] = None # default 0.05
|
||||
allow_video_retime: Optional[bool] = None # default True
|
||||
|
||||
class DubRequest(BaseModel):
|
||||
segments: List[DubSegment]
|
||||
language: str = "Auto"
|
||||
@@ -93,35 +75,18 @@ class DubRequest(BaseModel):
|
||||
# each segment's video portion is stretched (via
|
||||
# ffmpeg setpts) to fit the natural-rate dub audio.
|
||||
# Audio plays at 1.0×; total video duration grows.
|
||||
# "smart_fit" — dub-length fitting v2: split the burden between a
|
||||
# mild pitch-preserving audio speed-up (≤1.2× alone,
|
||||
# ≤1.5× in hybrid) and a mild per-segment video
|
||||
# slow-down (≤2.0×), per services/fit_planner.py.
|
||||
# Residual overflow is trimmed and surfaced.
|
||||
# "strict_slot" — legacy: keep `slot_fit` semantics (atempo squeeze
|
||||
# when audio > slot). Kept for back-compat.
|
||||
timing_strategy: Optional[Literal["concise", "stretch_video", "strict_slot", "smart_fit"]] = "concise"
|
||||
timing_strategy: Optional[Literal["concise", "stretch_video", "strict_slot"]] = "concise"
|
||||
|
||||
# Per-job slip budget for "concise" mode. Hard-trim only kicks in once
|
||||
# gap absorption + this much extra time has been consumed.
|
||||
overflow_budget_s: Optional[float] = 0.0
|
||||
|
||||
# Knob overrides for `smart_fit` (ignored by other strategies). Omitted
|
||||
# fields default server-side to fit_planner.FitParams values.
|
||||
fit_options: Optional[FitOptions] = None
|
||||
|
||||
class TranslateSegment(BaseModel):
|
||||
id: str
|
||||
text: str
|
||||
target_lang: Optional[str] = None
|
||||
# Free-form delivery direction ("urgent, whispering") — feeds the
|
||||
# cinematic reflect/adapt prompts. The frontend has sent this since
|
||||
# Phase 4.2 but pydantic silently dropped it as an undeclared extra,
|
||||
# so the per-segment direction hint never reached the LLM.
|
||||
direction: Optional[str] = None
|
||||
# Available time slot (end - start, seconds) for rate-ratio prediction
|
||||
# and the cinematic slot-fit pass. Same silent-drop fix as `direction`.
|
||||
slot_seconds: Optional[float] = None
|
||||
|
||||
class TranslateRequest(BaseModel):
|
||||
segments: List[TranslateSegment]
|
||||
@@ -129,14 +94,8 @@ class TranslateRequest(BaseModel):
|
||||
provider: Optional[str] = None
|
||||
source_lang: Optional[str] = None # ISO 639-1; overrides job detection
|
||||
job_id: Optional[str] = None # Dub job id, used to resolve detected source_lang
|
||||
quality: Optional[str] = "fast" # "fast" (one-shot) | "cinematic" (reflect→adapt) | "autofit" (cinematic + strict fit-to-slot)
|
||||
quality: Optional[str] = "fast" # "fast" (one-shot) | "cinematic" (reflect → adapt)
|
||||
glossary: Optional[List[dict]] = None # [{"source": "...", "target": "...", "note": "..."}]
|
||||
# Optional regional dialect (BCP-47, e.g. "es-AR", "pt-BR") — #280 item 2.
|
||||
# Applied by LLM-backed paths (provider="openai" or quality="cinematic"):
|
||||
# the prompt asks for that region's vocabulary/grammar (e.g. Argentinian
|
||||
# voseo: "vos sos" instead of "tú eres"). Non-LLM providers (Argos, NLLB,
|
||||
# Google) can't honor it; the response then carries dialect_applied=false.
|
||||
dialect: Optional[str] = None
|
||||
|
||||
class DubIngestUrlRequest(BaseModel):
|
||||
url: str
|
||||
|
||||
@@ -1,281 +0,0 @@
|
||||
"""Acoustic echo cancellation for dictate-over-playback (parity Action 8b).
|
||||
|
||||
When the user dictates while OmniVoice is *playing* audio (a TTS preview, a
|
||||
dub render, a video), the loudspeaker signal leaks back into the microphone.
|
||||
The streaming ASR on ``/ws/transcribe`` then transcribes that bleed as if it
|
||||
were speech — the "it typed back what the app just said" symptom. A browser's
|
||||
``getUserMedia({echoCancellation:true})`` would help, but its quality and even
|
||||
its availability differ per platform/webview, which would make a *default*
|
||||
feature behave differently on macOS/Windows/Linux — against the project's
|
||||
cross-platform-parity rule. A server-side canceller behaves identically
|
||||
everywhere, so it is the local-first, platform-neutral choice.
|
||||
|
||||
This is an NLMS (normalised least-mean-squares) time-domain adaptive filter
|
||||
with a Geigel double-talk detector. It is a clean-room-grade *port* of
|
||||
Patter's ``getpatter/audio/aec.py`` (MIT) — see docs/competitive-analysis.md,
|
||||
Action 8. It is NOT production-grade DSP (WebRTC AEC3 / Speex AEC are); it is
|
||||
a dependency-free, good-enough canceller that removes the steady-state echo
|
||||
the ASR would otherwise hallucinate on.
|
||||
|
||||
Wiring (one instance per ``/ws/transcribe`` session — NOT thread-safe)::
|
||||
|
||||
aec = NlmsEchoCanceller(sample_rate=16000)
|
||||
# Far-end: every PCM chunk the client is about to play through speakers.
|
||||
aec.push_far_end(playback_pcm_bytes)
|
||||
# Near-end: the mic PCM, cleaned before it reaches the ASR buffer.
|
||||
cleaned = aec.process_near_end(mic_pcm_bytes)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Final
|
||||
|
||||
import numpy as np
|
||||
|
||||
logger = logging.getLogger("omnivoice.aec")
|
||||
|
||||
|
||||
_DEFAULT_FILTER_TAPS: Final[int] = 512
|
||||
"""Adaptive-filter length in samples. 512 taps @ 16 kHz = 32 ms, covering a
|
||||
typical near-field laptop/desktop echo path. Longer tails (large rooms) can
|
||||
pass ``filter_taps=1024``+ at proportionally more CPU per frame; 512 converges
|
||||
in ~0.5 s with the warm-up ramp and is the sweet spot for dictation."""
|
||||
|
||||
_DEFAULT_STEP_SIZE: Final[float] = 0.1
|
||||
"""Steady-state NLMS step size. Larger = faster channel tracking but less
|
||||
stable; 0.1 is the textbook value for narrowband voice."""
|
||||
|
||||
_DEFAULT_WARMUP_STEP_SIZE: Final[float] = 0.5
|
||||
"""Aggressive step used during the warm-up window so the filter reaches a
|
||||
usable echo estimate within ~0.5 s instead of several seconds. The Geigel
|
||||
double-talk detector still gates updates, so the bigger step does not learn
|
||||
the user's own voice as echo."""
|
||||
|
||||
_DEFAULT_WARMUP_SECONDS: Final[float] = 0.5
|
||||
"""Length of the warm-up window. After this many seconds of processed
|
||||
near-end audio the step decays from ``warmup_step_size`` to ``step_size``."""
|
||||
|
||||
_DEFAULT_LEAKAGE: Final[float] = 0.9999
|
||||
"""Per-iteration weight leakage (slightly < 1) so the filter slowly forgets
|
||||
stale taps when the echo path drifts (the user moves the mic)."""
|
||||
|
||||
_DOUBLE_TALK_RHO: Final[float] = 0.6
|
||||
"""Geigel double-talk threshold. When ``max(|near|) > rho * max(|far|)`` the
|
||||
near-end carries energy the far-end cannot explain (the user is talking) →
|
||||
freeze adaptation so the filter does not model the user's voice as echo."""
|
||||
|
||||
_FAR_END_BUFFER_SECONDS: Final[float] = 0.5
|
||||
"""How much past far-end (playback) audio to retain. The echo arrives at the
|
||||
mic tens of ms after playback; the filter needs that much look-back to align.
|
||||
500 ms is generous headroom."""
|
||||
|
||||
|
||||
class NlmsEchoCanceller:
|
||||
"""Time-domain NLMS adaptive filter with Geigel double-talk detection.
|
||||
|
||||
Operates on narrowband mono PCM at 16 kHz (the rate the dictation path
|
||||
resamples to) or 8 kHz. Not thread-safe — each ``/ws/transcribe`` session
|
||||
owns its own instance.
|
||||
"""
|
||||
|
||||
# Far-end staleness window (seconds): once the most recent far-end push
|
||||
# is older than this, ``process_near_end`` passes the mic through instead
|
||||
# of cancelling against a frozen reference (which would superimpose the
|
||||
# same stale ~50 ms waveform on every mic frame as an audible buzz).
|
||||
_FAR_STALE_S: float = 0.25
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
sample_rate: int = 16000,
|
||||
*,
|
||||
filter_taps: int = _DEFAULT_FILTER_TAPS,
|
||||
step_size: float = _DEFAULT_STEP_SIZE,
|
||||
warmup_step_size: float = _DEFAULT_WARMUP_STEP_SIZE,
|
||||
warmup_seconds: float = _DEFAULT_WARMUP_SECONDS,
|
||||
leakage: float = _DEFAULT_LEAKAGE,
|
||||
double_talk_rho: float = _DOUBLE_TALK_RHO,
|
||||
) -> None:
|
||||
if sample_rate not in (8000, 16000):
|
||||
raise ValueError(
|
||||
"NlmsEchoCanceller supports 8000 Hz or 16000 Hz only; "
|
||||
f"got {sample_rate}."
|
||||
)
|
||||
if filter_taps < 64:
|
||||
raise ValueError(
|
||||
f"filter_taps must be >= 64 to model a meaningful echo path; "
|
||||
f"got {filter_taps}."
|
||||
)
|
||||
if not 0 < step_size <= 1:
|
||||
raise ValueError(f"step_size must be in (0, 1]; got {step_size}.")
|
||||
if not 0 < warmup_step_size <= 1:
|
||||
raise ValueError(
|
||||
f"warmup_step_size must be in (0, 1]; got {warmup_step_size}."
|
||||
)
|
||||
if warmup_seconds < 0:
|
||||
raise ValueError(f"warmup_seconds must be >= 0; got {warmup_seconds}.")
|
||||
if not 0 < leakage <= 1:
|
||||
raise ValueError(f"leakage must be in (0, 1]; got {leakage}.")
|
||||
|
||||
self._sample_rate = sample_rate
|
||||
self._taps = filter_taps
|
||||
self._step = float(step_size)
|
||||
self._warmup_step = float(warmup_step_size)
|
||||
self._warmup_samples = int(warmup_seconds * sample_rate)
|
||||
self._leakage = float(leakage)
|
||||
self._rho = float(double_talk_rho)
|
||||
# Counts near-end samples processed so the step can taper from
|
||||
# warmup_step to step over the first warmup_samples. Counted from the
|
||||
# first process_near_end call so the window aligns with playback start.
|
||||
self._processed_samples: int = 0
|
||||
self._last_far_push_monotonic: float | None = None
|
||||
|
||||
# Filter coefficients (zeros — adapts to the channel within ~0.5–2 s).
|
||||
self._w = np.zeros(filter_taps, dtype=np.float32)
|
||||
|
||||
# Far-end ring buffer holding >= filter_taps samples of playback
|
||||
# history, with headroom so push/process can interleave freely.
|
||||
max_buf_samples = max(
|
||||
filter_taps * 2,
|
||||
int(sample_rate * _FAR_END_BUFFER_SECONDS),
|
||||
)
|
||||
self._far_buf = np.zeros(max_buf_samples, dtype=np.float32)
|
||||
self._far_write_idx = 0 # next write position (head)
|
||||
self._far_filled = 0 # samples written so far (capped at len(far_buf))
|
||||
|
||||
# Diagnostics only — never read in the hot path.
|
||||
self.frames_processed: int = 0
|
||||
self.double_talk_frames: int = 0
|
||||
|
||||
# ── Public API ──────────────────────────────────────────────────────────
|
||||
|
||||
def push_far_end(self, pcm_bytes: bytes) -> None:
|
||||
"""Append far-end (playback) audio to the reference ring buffer.
|
||||
|
||||
Accepts raw int16 little-endian mono PCM at the configured rate.
|
||||
"""
|
||||
if not pcm_bytes:
|
||||
return
|
||||
self._last_far_push_monotonic = time.monotonic()
|
||||
samples = np.frombuffer(pcm_bytes, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
n = samples.shape[0]
|
||||
buf_len = self._far_buf.shape[0]
|
||||
if n >= buf_len:
|
||||
# More than the buffer holds — keep only the newest buf_len.
|
||||
self._far_buf[:] = samples[-buf_len:]
|
||||
self._far_write_idx = 0
|
||||
self._far_filled = buf_len
|
||||
return
|
||||
end = self._far_write_idx + n
|
||||
if end <= buf_len:
|
||||
self._far_buf[self._far_write_idx:end] = samples
|
||||
else:
|
||||
head = buf_len - self._far_write_idx
|
||||
self._far_buf[self._far_write_idx:] = samples[:head]
|
||||
self._far_buf[: n - head] = samples[head:]
|
||||
self._far_write_idx = (self._far_write_idx + n) % buf_len
|
||||
self._far_filled = min(self._far_filled + n, buf_len)
|
||||
|
||||
def process_near_end(self, pcm_bytes: bytes) -> bytes:
|
||||
"""Subtract the estimated echo from the near-end (mic) signal.
|
||||
|
||||
Returns int16 little-endian mono PCM with the estimated echo removed.
|
||||
Passes the frame through unchanged when there is nothing worth
|
||||
cancelling: no playback has been primed, or the far-end reference is
|
||||
stale (the app went silent).
|
||||
"""
|
||||
if not pcm_bytes:
|
||||
return pcm_bytes
|
||||
|
||||
# Not enough far-end history to fill the filter window yet — passing
|
||||
# through avoids emitting garbage on the first frames.
|
||||
if self._far_filled < self._taps:
|
||||
return pcm_bytes
|
||||
|
||||
# Far-end reference is stale (app stopped playing): the ring only
|
||||
# advances on push_far_end, so the "most recent" window is frozen at
|
||||
# the tail of the last playback. Convolving against it would buzz.
|
||||
last_push = self._last_far_push_monotonic
|
||||
if last_push is None or (time.monotonic() - last_push) > self._FAR_STALE_S:
|
||||
return pcm_bytes
|
||||
|
||||
near = np.frombuffer(pcm_bytes, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
cleaned = self._block_nlms(near)
|
||||
out = np.clip(cleaned * 32768.0, -32768.0, 32767.0).astype(np.int16)
|
||||
self.frames_processed += 1
|
||||
return out.tobytes()
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Clear filter coefficients and far-end history (e.g. on a new turn)."""
|
||||
self._w.fill(0)
|
||||
self._far_buf.fill(0)
|
||||
self._far_write_idx = 0
|
||||
self._far_filled = 0
|
||||
self._processed_samples = 0
|
||||
self._last_far_push_monotonic = None
|
||||
self.frames_processed = 0
|
||||
self.double_talk_frames = 0
|
||||
|
||||
# ── Internals ───────────────────────────────────────────────────────────
|
||||
|
||||
def _far_window(self, length: int) -> np.ndarray:
|
||||
"""Most recent ``length`` far-end samples, oldest first / newest last."""
|
||||
buf_len = self._far_buf.shape[0]
|
||||
if length > self._far_filled:
|
||||
length = self._far_filled
|
||||
end = self._far_write_idx # newest sample is at (end - 1) mod buf_len
|
||||
if end >= length:
|
||||
return self._far_buf[end - length: end]
|
||||
head = self._far_buf[buf_len - (length - end):]
|
||||
tail = self._far_buf[:end]
|
||||
return np.concatenate((head, tail))
|
||||
|
||||
def _block_nlms(self, near: np.ndarray) -> np.ndarray:
|
||||
"""Sample-by-sample NLMS over one frame of near-end samples.
|
||||
|
||||
Classical NLMS depends on the weights adapted at the previous sample,
|
||||
so the inner loop is sequential. Each sample is O(taps); numpy keeps a
|
||||
320-sample / 512-tap frame well under a millisecond on commodity CPUs.
|
||||
"""
|
||||
taps = self._taps
|
||||
far_window = self._far_window(taps + near.shape[0] - 1)
|
||||
if far_window.shape[0] < taps + near.shape[0] - 1:
|
||||
# Still warming up — left-pad with zeros so indices line up.
|
||||
pad = np.zeros(
|
||||
taps + near.shape[0] - 1 - far_window.shape[0], dtype=np.float32
|
||||
)
|
||||
far_window = np.concatenate((pad, far_window))
|
||||
|
||||
# Geigel double-talk detector (frame-wise).
|
||||
far_max = float(np.max(np.abs(far_window))) if far_window.size else 0.0
|
||||
near_max = float(np.max(np.abs(near)))
|
||||
# Freeze adaptation when the far reference is effectively silent
|
||||
# (<= -60 dBFS): adapting against a fade-out tail with near-zero norm
|
||||
# blows the weights up against user speech when playback resumes.
|
||||
if far_max <= 1e-3:
|
||||
return near
|
||||
double_talk = near_max > self._rho * far_max
|
||||
if double_talk:
|
||||
self.double_talk_frames += 1
|
||||
|
||||
out = np.empty_like(near)
|
||||
w = self._w
|
||||
leakage = self._leakage
|
||||
# Constant step within the frame keeps the inner loop branch-free.
|
||||
if self._processed_samples < self._warmup_samples:
|
||||
step = self._warmup_step
|
||||
else:
|
||||
step = self._step
|
||||
for i in range(near.shape[0]):
|
||||
x = far_window[i: i + taps]
|
||||
y_est = float(np.dot(w, x))
|
||||
e = float(near[i] - y_est)
|
||||
out[i] = e
|
||||
if not double_talk:
|
||||
# NLMS update with leakage. +1e-6 guards divide-by-zero.
|
||||
norm = float(np.dot(x, x)) + 1e-6
|
||||
w *= leakage
|
||||
w += (step * e / norm) * x
|
||||
self._processed_samples += near.shape[0]
|
||||
return out
|
||||
+36
-757
@@ -23,7 +23,6 @@ faster-whisper because it's available on every platform we ship to).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
@@ -31,143 +30,6 @@ from abc import ABC, abstractmethod
|
||||
|
||||
logger = logging.getLogger("omnivoice.asr")
|
||||
|
||||
# A single ASR transcribe must never block a request indefinitely. The chunked
|
||||
# dub pipeline already bounds each chunk (OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S);
|
||||
# the *whole-file* paths (dub QC re-transcribe, dictation, OpenAI-compat) ran
|
||||
# unbounded, so a slow/stuck transcribe — e.g. large-v3 on a VRAM-starved GPU
|
||||
# where the resident TTS model contends for memory — hung the request *and* tied
|
||||
# up a GPU-pool worker, surfacing in the UI as the misleading "can't reach the
|
||||
# local backend" (TamKieu / Vietnam report). Bound them so a hang becomes a fast,
|
||||
# actionable error instead. Generous default (whole-file large-v3 on CPU is slow
|
||||
# but valid); override with the env var for very long single files.
|
||||
ASR_TRANSCRIBE_TIMEOUT_S = float(os.environ.get("OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S", "300.0"))
|
||||
|
||||
|
||||
class ASRTimeoutError(TimeoutError):
|
||||
"""Raised when a whole-file transcribe exceeds ASR_TRANSCRIBE_TIMEOUT_S.
|
||||
|
||||
Carries a user-actionable message: the backend is alive (this is not a
|
||||
connection failure) — the ASR model is too heavy for the available compute.
|
||||
"""
|
||||
|
||||
|
||||
async def run_transcribe_guarded(executor, fn, *, what: str = "ASR",
|
||||
timeout: float = ASR_TRANSCRIBE_TIMEOUT_S):
|
||||
"""Run a blocking transcribe ``fn`` in ``executor`` with a hard wall-clock
|
||||
bound. On timeout, raise :class:`ASRTimeoutError` with guidance instead of
|
||||
letting the request hang forever.
|
||||
|
||||
``run_in_executor`` cannot cancel the underlying thread, so a wedged
|
||||
transcribe (a CTranslate2 / whisperx / VAD hang seen on some Windows + CUDA
|
||||
setups, #730) keeps occupying its GPU-pool worker. With a 1–2 worker pool
|
||||
that starves every *other* request — including TTS generate — and the next
|
||||
thing the user does surfaces as "Can't reach the local backend" even though
|
||||
the process is alive. So on timeout we also ``reset()`` the pool when it
|
||||
supports it (``_ResilientGpuPool``): the wedged thread is abandoned and the
|
||||
next submit gets a fresh worker, restoring capacity without an app restart.
|
||||
The orphaned thread still holds its VRAM until the process exits, which is
|
||||
why the message still recommends a smaller ASR model / Flush as the durable
|
||||
fix. Executors without ``reset`` (a plain ThreadPoolExecutor in tests) just
|
||||
get the bound + actionable error.
|
||||
"""
|
||||
loop = asyncio.get_running_loop()
|
||||
fut = loop.run_in_executor(executor, fn)
|
||||
try:
|
||||
return await asyncio.wait_for(fut, timeout=timeout)
|
||||
except asyncio.TimeoutError:
|
||||
# Free the poisoned pool so a hung transcribe can't keep starving TTS /
|
||||
# other ASR work (the "can't reach backend" symptom, #730).
|
||||
_reset = getattr(executor, "reset", None)
|
||||
if callable(_reset):
|
||||
try:
|
||||
_reset()
|
||||
logger.warning(
|
||||
"%s transcription exceeded %.0fs — abandoned the GPU-pool "
|
||||
"worker to restore capacity (#730).", what, timeout,
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("GPU pool reset after ASR timeout failed")
|
||||
raise ASRTimeoutError(
|
||||
f"{what} transcription exceeded {timeout:.0f}s and was abandoned — "
|
||||
"the backend is running, but the ASR model is too heavy for the "
|
||||
"available compute. Most often the GPU is VRAM-starved: the resident "
|
||||
"TTS model and a large ASR model (large-v3) contend for memory. "
|
||||
"Capacity was restored automatically, but for a durable fix Flush the "
|
||||
"TTS model to free VRAM, pick a smaller ASR model in Settings → "
|
||||
"Models, or set ASR to CPU. (Raise OMNIVOICE_ASR_TRANSCRIBE_TIMEOUT_S "
|
||||
"for very long single files.)"
|
||||
)
|
||||
|
||||
|
||||
def _compute_type_candidates(device: str) -> list[str]:
|
||||
"""Per-device compute_type fallback chain. int8 is supported by every
|
||||
CTranslate2 CUDA+CPU build; float16/int8_float16 only on GPUs with efficient
|
||||
fp16 — so degrade rather than crash (#551). Honors an ASR_COMPUTE_TYPE env
|
||||
override (power users on exotic hardware can pin int8/float32)."""
|
||||
import os
|
||||
override = os.environ.get("ASR_COMPUTE_TYPE")
|
||||
if override:
|
||||
return [override]
|
||||
return ["float16", "int8_float16", "int8"] if device == "cuda" else ["int8", "float32"]
|
||||
|
||||
|
||||
def _is_compute_type_error(msg: str) -> bool:
|
||||
low = msg.lower()
|
||||
return "compute type" in low or "efficient float16" in low
|
||||
|
||||
|
||||
def _decode_audio_16k_mono(audio_path: str):
|
||||
"""Decode `audio_path` to a 16 kHz mono float32 waveform using OmniVoice's
|
||||
*validated* ffmpeg, instead of whisperx.load_audio's bare ``"ffmpeg"`` PATH
|
||||
lookup.
|
||||
|
||||
whisperx (and openai-whisper) shell out to a literal ``"ffmpeg"`` resolved
|
||||
against the OS PATH. On Windows that resolves to whatever the system finds
|
||||
first — a WindowsApps alias stub or a corrupt/wrong-arch download — which
|
||||
passes `which` but explodes at spawn with ``[WinError 193] %1 is not a valid
|
||||
Win32 application``. whisperx only catches `CalledProcessError`, so the
|
||||
spawn-time `OSError` escapes and the dub/batch path reports the opaque
|
||||
"Transcription produced no segments" (#479). ``find_ffmpeg()`` probes each
|
||||
candidate with ``-version`` and returns a runnable binary (the bundled
|
||||
imageio-ffmpeg / Tauri sidecar) — or None, so we can raise an actionable
|
||||
error. This also fixes the imageio case a PATH-prepend can't: its binary is
|
||||
named ``ffmpeg-<plat>-vN.exe``, not ``ffmpeg``, so bare lookup never finds
|
||||
it. Mirrors whisperx.audio.load_audio's command exactly (16 kHz, mono, s16le).
|
||||
"""
|
||||
import subprocess
|
||||
|
||||
import numpy as np
|
||||
|
||||
from services.ffmpeg_utils import find_ffmpeg
|
||||
|
||||
ffmpeg = find_ffmpeg()
|
||||
if not ffmpeg:
|
||||
raise RuntimeError(
|
||||
"Cannot transcribe: ffmpeg is missing or not runnable. Install "
|
||||
"ffmpeg (or let OmniVoice's bundled binary download), then retry. "
|
||||
"On Windows a '[WinError 193]' here means the ffmpeg binary is "
|
||||
"corrupt or the wrong architecture — reinstall it or clear the "
|
||||
"imageio-ffmpeg cache."
|
||||
)
|
||||
cmd = [
|
||||
ffmpeg, "-nostdin", "-threads", "0", "-i", audio_path,
|
||||
"-f", "s16le", "-ac", "1", "-acodec", "pcm_s16le", "-ar", "16000", "-",
|
||||
]
|
||||
try:
|
||||
out = subprocess.run(cmd, capture_output=True, check=True).stdout
|
||||
except OSError as e:
|
||||
# Belt-and-suspenders: find_ffmpeg() already -version-validated this
|
||||
# binary, so a WinError 193 here is unexpected — surface it clearly
|
||||
# rather than letting it become "no segments".
|
||||
raise RuntimeError(
|
||||
f"ffmpeg at {ffmpeg!r} could not be executed ({e}). Reinstall "
|
||||
"ffmpeg or clear the imageio-ffmpeg cache."
|
||||
) from e
|
||||
except subprocess.CalledProcessError as e:
|
||||
stderr = (e.stderr or b"").decode(errors="replace")[:500]
|
||||
raise RuntimeError(f"Failed to decode audio for transcription: {stderr}") from e
|
||||
return np.frombuffer(out, np.int16).flatten().astype(np.float32) / 32768.0
|
||||
|
||||
|
||||
# ── Protocol ────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -175,15 +37,6 @@ def _decode_audio_16k_mono(audio_path: str):
|
||||
class ASRBackend(ABC):
|
||||
id: str = "base"
|
||||
display_name: str = "Base ASR"
|
||||
# Accelerator families this backend can use, in preference order; always
|
||||
# includes a fallback. Subset of {cuda, rocm, mps, xpu, cpu}. Mirrors the
|
||||
# TTSBackend.gpu_compat contract so engine_routing.resolve_routing() can
|
||||
# surface the effective device per host (no silent CPU fallback). The
|
||||
# conservative default is CPU-only; subclasses declare what they really run
|
||||
# on. (ROCm is intentionally NOT claimed yet for any ASR engine — see the
|
||||
# per-engine notes; an unverified `rocm` claim would route ROCm hosts to a
|
||||
# broken GPU path, strictly worse than the honest `cpu_fallback`.)
|
||||
gpu_compat: tuple[str, ...] = ("cpu",)
|
||||
|
||||
@classmethod
|
||||
@abstractmethod
|
||||
@@ -197,22 +50,6 @@ class ASRBackend(ABC):
|
||||
that already speak the shape plug in with zero adapter work.
|
||||
"""
|
||||
|
||||
def ensure_loaded(self) -> None:
|
||||
"""Eagerly load the model weights, raising the real cause on failure.
|
||||
|
||||
Backends load lazily inside ``transcribe()`` by default, so a load
|
||||
failure (missing weights, CUDA/cuDNN mismatch, torch-2.6 weights-only
|
||||
VAD regression, import error) first surfaces buried in per-chunk
|
||||
errors — and is retried on *every* chunk. The transcribe preflight
|
||||
calls this so the genuine cause is surfaced once, up front, as a clean
|
||||
terminal error event instead of N cryptic per-chunk failures (#578).
|
||||
|
||||
Default is a no-op; backends that hold a heavy model override it to
|
||||
trigger their lazy loader. It MUST raise the underlying exception (not
|
||||
swallow it) so the caller can classify and surface it.
|
||||
"""
|
||||
pass
|
||||
|
||||
def unload(self) -> None:
|
||||
"""Release the model from memory."""
|
||||
pass
|
||||
@@ -221,82 +58,9 @@ class ASRBackend(ABC):
|
||||
# ── WhisperX (cross-platform default — forced-alignment word timing) ────────
|
||||
|
||||
|
||||
def _harden_speechbrain_lazy_imports() -> None:
|
||||
"""Make speechbrain 1.x's lazy-import guard fire on Windows too (#630/#611/#647).
|
||||
|
||||
speechbrain 1.x exposes optional integrations (``k2_fsa``, ``numba`` losses,
|
||||
``spacy``/``flair`` nlp) as ``LazyModule`` redirects living in ``sys.modules``.
|
||||
Stray introspection — PyTorch's op-registration machinery, pickling, a
|
||||
``dir()``/``hasattr`` walk — touches one of these during ``whisperx.load_model``
|
||||
(pyannote → speechbrain), which would *actually* import the optional package.
|
||||
speechbrain guards against that by suppressing the import when the triggering
|
||||
frame is the stdlib ``inspect`` module — but the check is
|
||||
``filename.endswith("/inspect.py")``, a hardcoded POSIX separator. On Windows
|
||||
the frame filename uses backslashes (``...\\Lib\\inspect.py``), so the guard
|
||||
misses, the redirect imports ``speechbrain.integrations.k2_fsa`` → ``import k2``
|
||||
→ k2 isn't installed → ``ImportError: Lazy import of LazyModule(...k2_fsa...)
|
||||
failed``. That bubbles out of WhisperX and aborts transcription with zero
|
||||
segments. WhisperX is the *default* ASR, so this is a Windows-only break of a
|
||||
cross-platform-default feature (P0 parity).
|
||||
|
||||
Fix the whole class — every optional-integration redirect, not just k2 — by
|
||||
re-implementing ``LazyModule.ensure_module`` with an ``os.sep``-agnostic
|
||||
basename check. Idempotent and a no-op on macOS/Linux (basename match is a
|
||||
strict superset of the old forward-slash check) and when speechbrain is
|
||||
absent. A genuine access from real user code with k2 missing still raises
|
||||
ImportError unchanged — only inspect-triggered spurious imports are
|
||||
suppressed, on every platform.
|
||||
"""
|
||||
try:
|
||||
from speechbrain.utils import importutils as _iu
|
||||
except Exception: # speechbrain not installed / import side-effect — nothing to harden
|
||||
return
|
||||
if getattr(_iu.LazyModule, "_omnivoice_xplat_guard", False):
|
||||
return
|
||||
import importlib as _importlib
|
||||
import inspect as _inspect
|
||||
import sys as _sys
|
||||
import warnings as _warnings
|
||||
|
||||
def ensure_module(self, stacklevel):
|
||||
importer_frame = None
|
||||
try:
|
||||
importer_frame = _inspect.getframeinfo(_sys._getframe(stacklevel + 1))
|
||||
except AttributeError:
|
||||
_warnings.warn(
|
||||
"Failed to inspect frame to check if we should ignore importing a "
|
||||
"module lazily (OmniVoice cross-platform guard)."
|
||||
)
|
||||
if importer_frame is not None:
|
||||
# Normalise BOTH separators explicitly (not os.path.basename, which is
|
||||
# host-dependent) so the guard is correct regardless of which os.path
|
||||
# flavour is active. Upstream's `.endswith("/inspect.py")` matched only
|
||||
# POSIX paths — that is the Windows-only bug (#630/#611/#647).
|
||||
base = importer_frame.filename.replace("\\", "/").rsplit("/", 1)[-1]
|
||||
if base == "inspect.py":
|
||||
raise AttributeError()
|
||||
if self.lazy_module is None:
|
||||
try:
|
||||
if self.package is None:
|
||||
self.lazy_module = _importlib.import_module(self.target)
|
||||
else:
|
||||
self.lazy_module = _importlib.import_module(f".{self.target}", self.package)
|
||||
except Exception as e: # noqa: BLE001 — match upstream: wrap as ImportError
|
||||
raise ImportError(f"Lazy import of {repr(self)} failed") from e
|
||||
return self.lazy_module
|
||||
|
||||
_iu.LazyModule.ensure_module = ensure_module
|
||||
_iu.LazyModule._omnivoice_xplat_guard = True
|
||||
logger.debug("speechbrain LazyModule guard hardened for cross-platform inspect.py check")
|
||||
|
||||
|
||||
class WhisperXBackend(ASRBackend):
|
||||
id = "whisperx"
|
||||
display_name = "WhisperX (faster-whisper + wav2vec2 forced alignment)"
|
||||
# CTranslate2 backend: CUDA fp16 or CPU int8 (see _pick_device). ROCm not
|
||||
# claimed — CTranslate2 has no upstream HIP build, so a ROCm host honestly
|
||||
# gets cpu_fallback rather than a false GPU promise.
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
def __init__(self):
|
||||
self._model_name = os.environ.get("ASR_MODEL_WHISPERX", "large-v3")
|
||||
@@ -323,28 +87,10 @@ class WhisperXBackend(ASRBackend):
|
||||
return True, "ready"
|
||||
except ImportError as e:
|
||||
return False, f"whisperx not installed: {e}"
|
||||
except Exception as e: # noqa: BLE001
|
||||
# The import can fail while loading a native dep — CTranslate2's .so
|
||||
# is rejected by hardened kernels / newer glibc with "cannot enable
|
||||
# executable stack" (#692), an OSError, not an ImportError. An
|
||||
# availability probe must REPORT 'unusable here', never raise, so
|
||||
# engine selection falls back instead of crashing the ASR preflight.
|
||||
return False, f"whisperx failed to load ({type(e).__name__}): {e}"
|
||||
|
||||
def ensure_loaded(self) -> None:
|
||||
# Surface a whisperx/CTranslate2/torch load failure at preflight (once,
|
||||
# with the real cause) instead of buried per-chunk and retried N times
|
||||
# (#578). Re-raises whatever `_ensure_asr` raises after its fp16→int8
|
||||
# and OOM→CPU fallbacks are exhausted.
|
||||
self._ensure_asr()
|
||||
|
||||
def _ensure_asr(self):
|
||||
if self._asr is not None:
|
||||
return
|
||||
# Patch speechbrain's lazy-import guard BEFORE whisperx pulls in pyannote
|
||||
# → speechbrain, or a stray k2_fsa redirect import aborts ASR on Windows
|
||||
# (#630/#611/#647). No-op on macOS/Linux and when speechbrain is absent.
|
||||
_harden_speechbrain_lazy_imports()
|
||||
import whisperx
|
||||
logger.info(
|
||||
"whisperx loading ASR %s on %s (%s)",
|
||||
@@ -377,40 +123,7 @@ class WhisperXBackend(ASRBackend):
|
||||
# vad_method="silero" is the default; keep it so short gaps
|
||||
# get cleaned up before transcription.
|
||||
)
|
||||
except (ValueError, RuntimeError) as e:
|
||||
# #551: GPUs without efficient fp16 (older Maxwell/Pascal, GTX 16xx)
|
||||
# or a CTranslate2/cuDNN binary mismatch raise a *ValueError*
|
||||
# ("Requested float16 compute type, but the target device or backend
|
||||
# do not support efficient float16 computation") at load — not an
|
||||
# OOM, not a RuntimeError. Retry on the SAME device with the next
|
||||
# compute_type candidate (cuda: int8_float16 → int8) before touching
|
||||
# the OOM→CPU path, so we degrade rather than crash every chunk.
|
||||
if _is_compute_type_error(str(e)):
|
||||
candidates = _compute_type_candidates(self._device)
|
||||
try:
|
||||
nxt = candidates[candidates.index(self._compute_type) + 1:]
|
||||
except ValueError:
|
||||
nxt = [c for c in candidates if c != self._compute_type]
|
||||
for ct in nxt:
|
||||
logger.warning(
|
||||
"whisperx %s unsupported on %s — retrying with %s. Detail: %s",
|
||||
self._compute_type, self._device, ct, e,
|
||||
)
|
||||
self._compute_type = ct
|
||||
try:
|
||||
self._asr = whisperx.load_model(
|
||||
self._model_name,
|
||||
device=self._device,
|
||||
compute_type=self._compute_type,
|
||||
)
|
||||
return
|
||||
except (ValueError, RuntimeError) as e2:
|
||||
if _is_compute_type_error(str(e2)):
|
||||
e = e2
|
||||
continue
|
||||
raise
|
||||
# Exhausted compute-type candidates on this device — re-raise.
|
||||
raise
|
||||
except RuntimeError as e:
|
||||
# CUDA OOM: a resident TTS model + the GPU worker pool can starve
|
||||
# VRAM on small (e.g. 8 GB laptop) GPUs, so loading large-v3 on
|
||||
# CUDA dies here — which previously surfaced as a bare 500 from
|
||||
@@ -600,13 +313,10 @@ class WhisperXBackend(ASRBackend):
|
||||
return None
|
||||
|
||||
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
|
||||
import whisperx # used for whisperx.align() below
|
||||
import whisperx
|
||||
self._ensure_asr()
|
||||
logger.info("whisperx transcribing %s (word_timestamps=%s)", audio_path, word_timestamps)
|
||||
# Decode via OmniVoice's validated ffmpeg, NOT whisperx.load_audio's bare
|
||||
# "ffmpeg" PATH lookup which yields [WinError 193] -> "no segments" on
|
||||
# Windows (#479). Same 16 kHz mono s16le array whisperx expects.
|
||||
audio = _decode_audio_16k_mono(audio_path)
|
||||
audio = whisperx.load_audio(audio_path)
|
||||
try:
|
||||
result = self._asr.transcribe(audio)
|
||||
except IndexError:
|
||||
@@ -672,8 +382,6 @@ class WhisperXBackend(ASRBackend):
|
||||
class FasterWhisperBackend(ASRBackend):
|
||||
id = "faster-whisper"
|
||||
display_name = "Faster-Whisper (CTranslate2 — Linux/Windows/macOS)"
|
||||
# CTranslate2: CUDA or CPU (no upstream ROCm/HIP build — see WhisperX note).
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
def __init__(self):
|
||||
# Defaulting to the CTranslate2-converted large-v3 repo. Matches
|
||||
@@ -683,10 +391,6 @@ class FasterWhisperBackend(ASRBackend):
|
||||
"ASR_MODEL_FASTER", "Systran/faster-whisper-large-v3"
|
||||
)
|
||||
self._model = None # lazy — first transcribe() loads weights
|
||||
# Set by _ensure_model() to the device/compute_type that actually loaded
|
||||
# (after the #551 compute_type / #255 OOM→CPU fallback chain).
|
||||
self._device: str | None = None
|
||||
self._compute_type: str | None = None
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
@@ -695,11 +399,6 @@ class FasterWhisperBackend(ASRBackend):
|
||||
return True, "ready"
|
||||
except ImportError as e:
|
||||
return False, f"faster-whisper not installed: {e}"
|
||||
except Exception as e: # noqa: BLE001
|
||||
# faster-whisper pulls in CTranslate2, whose .so is rejected by
|
||||
# hardened kernels / newer glibc ("cannot enable executable stack",
|
||||
# #692) — an OSError. Report unavailable so we fall back, not crash.
|
||||
return False, f"faster-whisper failed to load ({type(e).__name__}): {e}"
|
||||
|
||||
def _ensure_model(self):
|
||||
if self._model is not None:
|
||||
@@ -720,58 +419,9 @@ class FasterWhisperBackend(ASRBackend):
|
||||
"faster-whisper loading %s on %s (%s)",
|
||||
self._model_name, device, compute_type,
|
||||
)
|
||||
# Try the per-device compute_type chain (cuda: float16 → int8_float16 →
|
||||
# int8; cpu: int8 → float32). A GPU without efficient fp16 (older
|
||||
# Maxwell/Pascal, GTX 16xx, or a CTranslate2/cuDNN mismatch) raises a
|
||||
# *ValueError* at construction (#551) — degrade to the next candidate
|
||||
# instead of failing every chunk. A genuine CUDA OOM falls back to CPU
|
||||
# (slower, same model/accuracy), preserving the existing #255 behaviour.
|
||||
candidates = _compute_type_candidates(device)
|
||||
if compute_type in candidates:
|
||||
candidates = candidates[candidates.index(compute_type):]
|
||||
last_err: Exception | None = None
|
||||
while True:
|
||||
for ct in candidates:
|
||||
try:
|
||||
self._model = WhisperModel(
|
||||
self._model_name, device=device, compute_type=ct
|
||||
)
|
||||
self._device, self._compute_type = device, ct
|
||||
return
|
||||
except (ValueError, RuntimeError) as e:
|
||||
last_err = e
|
||||
if _is_compute_type_error(str(e)):
|
||||
logger.warning(
|
||||
"faster-whisper %s unsupported on %s — trying next "
|
||||
"compute_type. Detail: %s", ct, device, e,
|
||||
)
|
||||
continue
|
||||
if device == "cuda" and "out of memory" in str(e).lower():
|
||||
# Stop scanning GPU candidates; fall back to CPU below.
|
||||
break
|
||||
raise
|
||||
# Exhausted candidates for this device. If we were on CUDA and the
|
||||
# last failure was an OOM, retry on CPU with its candidates (#255).
|
||||
if device == "cuda" and last_err is not None and (
|
||||
"out of memory" in str(last_err).lower()
|
||||
):
|
||||
logger.warning(
|
||||
"faster-whisper CUDA OOM loading %s — retrying on CPU "
|
||||
"(slower). Free VRAM (Flush the TTS model) for GPU-speed "
|
||||
"ASR. Detail: %s", self._model_name, last_err,
|
||||
)
|
||||
try:
|
||||
import torch
|
||||
torch.cuda.empty_cache()
|
||||
except Exception: # noqa: BLE001 — cache clear is best-effort
|
||||
pass
|
||||
device = "cpu"
|
||||
candidates = _compute_type_candidates(device)
|
||||
compute_type = candidates[0]
|
||||
continue
|
||||
# All candidates exhausted (and no OOM→CPU retry available) — surface
|
||||
# the last error.
|
||||
raise last_err
|
||||
self._model = WhisperModel(
|
||||
self._model_name, device=device, compute_type=compute_type
|
||||
)
|
||||
|
||||
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
|
||||
self._ensure_model()
|
||||
@@ -847,7 +497,6 @@ _MLX_MODEL_TURBO = "mlx-community/whisper-large-v3-turbo"
|
||||
class MLXWhisperBackend(ASRBackend):
|
||||
id = "mlx-whisper"
|
||||
display_name = "MLX Whisper (Apple Silicon CoreML)"
|
||||
gpu_compat = ("mps", "cpu")
|
||||
|
||||
def __init__(self, model_name: str | None = None):
|
||||
self._model_name = model_name or os.environ.get(
|
||||
@@ -856,23 +505,14 @@ class MLXWhisperBackend(ASRBackend):
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
# #390: shared platform gate FIRST — one rule for MLX-Audio + MLX-Whisper.
|
||||
# Returns False on Linux/Windows/mac-Intel before any package import, so
|
||||
# a stray mlx-whisper wheel never reports available or advertises `mps`.
|
||||
from core.device_caps import mlx_supported
|
||||
ok, why = mlx_supported()
|
||||
if not ok:
|
||||
return False, why
|
||||
try:
|
||||
import torch
|
||||
if not (hasattr(torch.backends, "mps") and torch.backends.mps.is_available()):
|
||||
return False, "Apple Silicon (MPS) not available."
|
||||
import mlx_whisper # noqa: F401
|
||||
return True, "ready"
|
||||
# Catch OSError/RuntimeError too, not just ImportError: in a
|
||||
# PyInstaller bundle mlx's native dylib/metallib can fail to load
|
||||
# even when the package imports, raising OSError/RuntimeError. We must
|
||||
# report unavailable (so the picker falls back) rather than crash the
|
||||
# registry scan (Wave 4.4).
|
||||
except (ImportError, OSError, RuntimeError) as e:
|
||||
return False, f"mlx-whisper unavailable: {e}"
|
||||
except ImportError as e:
|
||||
return False, f"mlx-whisper not installed: {e}"
|
||||
|
||||
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
|
||||
import mlx_whisper
|
||||
@@ -921,9 +561,6 @@ class MLXWhisperBackend(ASRBackend):
|
||||
class PyTorchWhisperBackend(ASRBackend):
|
||||
id = "pytorch-whisper"
|
||||
display_name = "PyTorch Whisper (CUDA / CPU via transformers pipeline)"
|
||||
# Pure transformers pipeline → runs wherever torch does (CUDA, MPS, CPU).
|
||||
# ROCm-via-HIP would also work but is left unclaimed pending verification.
|
||||
gpu_compat = ("cuda", "mps", "cpu")
|
||||
|
||||
def __init__(self, asr_pipe=None):
|
||||
# Reuses the `_asr_pipe` attached to the TTS model when available.
|
||||
@@ -960,25 +597,12 @@ class PyTorchWhisperBackend(ASRBackend):
|
||||
"PyTorchWhisperBackend: loading standalone ASR pipeline %s on %s",
|
||||
model_name, device,
|
||||
)
|
||||
try:
|
||||
self._pipe = hf_pipeline(
|
||||
"automatic-speech-recognition",
|
||||
model=model_name,
|
||||
dtype=asr_dtype,
|
||||
device_map=device,
|
||||
)
|
||||
except Exception as e:
|
||||
# #549: an incomplete transformers install fails to build the ASR
|
||||
# pipeline (e.g. "Could not import module 'AutoFeatureExtractor'").
|
||||
# The raw error is opaque; re-raise with an actionable next step so
|
||||
# the toast tells the user how to recover instead of "no segments".
|
||||
raise RuntimeError(
|
||||
"transformers ASR pipeline failed to import (AutoFeatureExtractor) "
|
||||
"— your transformers install is incomplete; reinstall with "
|
||||
"`uv pip install --reinstall transformers`, or use faster-whisper "
|
||||
"(OmniVoice's default ASR) which avoids the transformers pipeline. "
|
||||
f"Underlying: {e}"
|
||||
) from e
|
||||
self._pipe = hf_pipeline(
|
||||
"automatic-speech-recognition",
|
||||
model=model_name,
|
||||
dtype=asr_dtype,
|
||||
device_map=device,
|
||||
)
|
||||
|
||||
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
|
||||
import soundfile as sf
|
||||
@@ -1009,11 +633,6 @@ class NeMoASRBackend(ASRBackend):
|
||||
Requires NVIDIA GPU.
|
||||
"""
|
||||
id = "nemo-parakeet"
|
||||
# CUDA-only: is_available() hard-fails without a GPU ("Parakeet TDT requires
|
||||
# NVIDIA GPU (CUDA)"), so declaring a CPU path would be a false claim. On a
|
||||
# CPU host this correctly resolves to routing_status="unavailable", matching
|
||||
# is_available()=False (the matrix suppresses the routing badge there).
|
||||
gpu_compat = ("cuda",)
|
||||
display_name = "Parakeet TDT (NVIDIA NeMo — English SOTA)"
|
||||
|
||||
def __init__(self):
|
||||
@@ -1122,7 +741,6 @@ class MoonshineASRBackend(ASRBackend):
|
||||
Great for live capture and CPU-only environments.
|
||||
"""
|
||||
id = "moonshine"
|
||||
gpu_compat = ("cpu",) # edge/CPU-optimized by design
|
||||
display_name = "Moonshine (edge-optimized, ONNX)"
|
||||
|
||||
def __init__(self):
|
||||
@@ -1200,148 +818,6 @@ class MoonshineASRBackend(ASRBackend):
|
||||
self._transcriber = None
|
||||
|
||||
|
||||
# ── sherpa-onnx live dictation (ONNX, CPU, streaming + offline) ─────────────
|
||||
|
||||
|
||||
def _load_audio_16k_mono_f32(audio_path: str):
|
||||
"""Decode any audio file to 16 kHz mono float32 in [-1, 1] for sherpa.
|
||||
|
||||
Prefers soundfile (WAV/FLAC — the dictation buffers are already WAV) and
|
||||
resamples to 16 kHz when needed; falls back to OmniVoice's validated ffmpeg
|
||||
for containers soundfile can't read (WebM/Opus). 16 kHz is sherpa's cheapest
|
||||
feed; it resamples internally too, but doing it here keeps the contract tight.
|
||||
"""
|
||||
import numpy as np
|
||||
try:
|
||||
import soundfile as sf
|
||||
data, sr = sf.read(audio_path, dtype="float32", always_2d=False)
|
||||
if getattr(data, "ndim", 1) > 1:
|
||||
data = data.mean(axis=1)
|
||||
data = np.ascontiguousarray(data, dtype=np.float32)
|
||||
if sr != 16000:
|
||||
# Lightweight linear resample — adequate for ASR features.
|
||||
n = int(round(len(data) * 16000 / sr))
|
||||
if n > 0:
|
||||
xp = np.linspace(0.0, 1.0, num=len(data), endpoint=False)
|
||||
x = np.linspace(0.0, 1.0, num=n, endpoint=False)
|
||||
data = np.interp(x, xp, data).astype(np.float32)
|
||||
sr = 16000
|
||||
return data, sr
|
||||
except Exception:
|
||||
# Container soundfile can't read (WebM/Opus) — use the validated ffmpeg
|
||||
# path, which already yields 16 kHz mono float32.
|
||||
return _decode_audio_16k_mono(audio_path), 16000
|
||||
|
||||
|
||||
class SherpaDictationBackend(ASRBackend):
|
||||
"""k2-fsa/sherpa-onnx ONNX dictation engine (CPU, live + offline).
|
||||
|
||||
One :class:`ASRBackend` instance is bound to one of the seven sherpa
|
||||
dictation models (see :mod:`services.sherpa_dictation`). For the offline
|
||||
``transcribe(path)`` contract it runs an ``OfflineRecognizer`` for offline
|
||||
models and a one-shot ``OnlineRecognizer`` decode for streaming models
|
||||
(so ``POST /transcribe`` works for every sherpa model). The *live* WS path
|
||||
drives the streaming recognizer incrementally — see ``capture_ws.py``.
|
||||
|
||||
CPU provider only (cross-platform default-parity rule); no CUDA dep.
|
||||
"""
|
||||
id = "sherpa-onnx-asr"
|
||||
display_name = "Sherpa-ONNX dictation (live, CPU — streaming + offline)"
|
||||
gpu_compat = ("cpu",)
|
||||
|
||||
def __init__(self, model_id: str | None = None):
|
||||
from services import sherpa_dictation as _sd
|
||||
mid = model_id or os.environ.get(
|
||||
"OMNIVOICE_SHERPA_ASR_MODEL", _sd.DEFAULT_MODEL_ID
|
||||
)
|
||||
spec = _sd.get_spec(mid)
|
||||
if spec is None:
|
||||
raise ValueError(
|
||||
f"Unknown sherpa dictation model {mid!r}. Known: "
|
||||
f"{[s.id for s in _sd.list_specs()]}"
|
||||
)
|
||||
self._spec = spec
|
||||
self._rec = None # lazy OfflineRecognizer / OnlineRecognizer
|
||||
|
||||
@property
|
||||
def spec(self):
|
||||
return self._spec
|
||||
|
||||
@property
|
||||
def streaming(self) -> bool:
|
||||
return self._spec.streaming
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
from services.sherpa_dictation import sherpa_available
|
||||
return sherpa_available()
|
||||
|
||||
def ensure_loaded(self) -> None:
|
||||
self._ensure_rec()
|
||||
|
||||
def _ensure_rec(self):
|
||||
if self._rec is not None:
|
||||
return
|
||||
from services import sherpa_dictation as _sd
|
||||
if self._spec.streaming:
|
||||
self._rec = _sd.build_online_recognizer(self._spec)
|
||||
else:
|
||||
self._rec = _sd.build_offline_recognizer(self._spec)
|
||||
|
||||
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
|
||||
self._ensure_rec()
|
||||
logger.info(
|
||||
"sherpa-onnx dictation transcribing %s (model=%s, kind=%s)",
|
||||
audio_path, self._spec.id, self._spec.kind,
|
||||
)
|
||||
samples, sr = _load_audio_16k_mono_f32(audio_path)
|
||||
if self._spec.streaming:
|
||||
text = self._decode_online_oneshot(samples, sr)
|
||||
else:
|
||||
text = self._decode_offline(samples, sr)
|
||||
return _sherpa_result(text, samples, sr)
|
||||
|
||||
def _decode_offline(self, samples, sr) -> str:
|
||||
s = self._rec.create_stream()
|
||||
s.accept_waveform(sr, samples)
|
||||
self._rec.decode_stream(s)
|
||||
return (s.result.text or "").strip()
|
||||
|
||||
def _decode_online_oneshot(self, samples, sr) -> str:
|
||||
"""One-shot decode of a whole buffer through the streaming recognizer
|
||||
(for the non-streaming ``transcribe()`` / partial re-decode path)."""
|
||||
import numpy as np
|
||||
s = self._rec.create_stream()
|
||||
s.accept_waveform(sr, samples)
|
||||
tail = np.zeros(int(0.5 * sr), dtype=np.float32)
|
||||
s.accept_waveform(sr, tail)
|
||||
s.input_finished()
|
||||
while self._rec.is_ready(s):
|
||||
self._rec.decode_stream(s)
|
||||
return (self._rec.get_result(s) or "").strip()
|
||||
|
||||
def unload(self) -> None:
|
||||
self._rec = None
|
||||
import gc
|
||||
gc.collect()
|
||||
|
||||
|
||||
def _sherpa_result(text: str, samples, sr) -> dict:
|
||||
"""Normalise a sherpa decode to OmniVoice's ``{chunks, segments, language,
|
||||
text}`` contract. sherpa gives plain text (no VAD split), so emit a single
|
||||
segment spanning the buffer — same shape Moonshine uses."""
|
||||
text = (text or "").strip()
|
||||
try:
|
||||
duration = round(len(samples) / float(sr), 3)
|
||||
except Exception:
|
||||
duration = None
|
||||
segments = []
|
||||
if text:
|
||||
segments.append({"text": text, "start": 0.0, "end": duration, "words": []})
|
||||
chunks = [{"text": s["text"], "timestamp": (s["start"], s["end"])} for s in segments]
|
||||
return {"chunks": chunks, "segments": segments, "language": "auto", "text": text}
|
||||
|
||||
|
||||
# ── Registry ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@@ -1409,7 +885,6 @@ class FunASRBackend(ASRBackend):
|
||||
#182); WhisperX remains the cross-platform default.
|
||||
"""
|
||||
id = "funasr"
|
||||
gpu_compat = ("cuda", "cpu") # FunASR: CUDA or CPU
|
||||
display_name = "FunASR (SenseVoice — 50+ languages, all-in-one)"
|
||||
|
||||
def __init__(self):
|
||||
@@ -1456,47 +931,7 @@ class FunASRBackend(ASRBackend):
|
||||
pass
|
||||
|
||||
|
||||
def _isolated_faster_whisper():
|
||||
"""Lazy import so the subprocess_asr → subprocess_backend chain isn't
|
||||
pulled in at registry definition time."""
|
||||
from services.subprocess_asr import IsolatedFasterWhisperBackend
|
||||
return IsolatedFasterWhisperBackend
|
||||
|
||||
|
||||
class _LazyASRRegistry(dict):
|
||||
"""Registry with one lazily-resolved entry (Wave 4.2). Mirrors the TTS
|
||||
registry's lazy pattern so listing/selecting the crash-isolated ASR
|
||||
backend doesn't import the subprocess stack unless it's used."""
|
||||
|
||||
_LAZY = {"faster-whisper-isolated": _isolated_faster_whisper}
|
||||
|
||||
def __contains__(self, key):
|
||||
return dict.__contains__(self, key) or key in self._LAZY
|
||||
|
||||
def __getitem__(self, key):
|
||||
if dict.__contains__(self, key):
|
||||
return dict.__getitem__(self, key)
|
||||
if key in self._LAZY:
|
||||
cls = self._LAZY[key]()
|
||||
self[key] = cls
|
||||
return cls
|
||||
raise KeyError(key)
|
||||
|
||||
def __iter__(self):
|
||||
seen = set()
|
||||
for k in dict.__iter__(self):
|
||||
seen.add(k)
|
||||
yield k
|
||||
for k in self._LAZY:
|
||||
if k not in seen:
|
||||
yield k
|
||||
|
||||
def items(self):
|
||||
for k in self:
|
||||
yield k, self[k]
|
||||
|
||||
|
||||
_REGISTRY: dict[str, type[ASRBackend]] = _LazyASRRegistry({
|
||||
_REGISTRY: dict[str, type[ASRBackend]] = {
|
||||
"whisperx": WhisperXBackend,
|
||||
"faster-whisper": FasterWhisperBackend,
|
||||
"mlx-whisper": MLXWhisperBackend,
|
||||
@@ -1504,92 +939,22 @@ _REGISTRY: dict[str, type[ASRBackend]] = _LazyASRRegistry({
|
||||
"nemo-parakeet": NeMoASRBackend,
|
||||
"moonshine": MoonshineASRBackend,
|
||||
"funasr": FunASRBackend,
|
||||
"sherpa-onnx-asr": SherpaDictationBackend,
|
||||
# "faster-whisper-isolated": resolved lazily (crash-isolated subprocess).
|
||||
})
|
||||
|
||||
|
||||
# Short install hints surfaced as tooltips on the Settings → Engines UI
|
||||
# (parity with tts_backend._INSTALL_HINTS).
|
||||
_INSTALL_HINTS: dict[str, str] = {
|
||||
"whisperx": "pip install whisperx (CTranslate2 + wav2vec2 alignment; CUDA or CPU)",
|
||||
"faster-whisper": "pip install faster-whisper (CTranslate2; cross-platform, CUDA or CPU)",
|
||||
"mlx-whisper": "pip install mlx-whisper (Apple Silicon only)",
|
||||
"pytorch-whisper": "Bundled with transformers — no extra install (CUDA/MPS/CPU)",
|
||||
"nemo-parakeet": "pip install nemo_toolkit[asr] (NVIDIA Parakeet; CUDA or CPU)",
|
||||
"moonshine": "pip install useful-moonshine (edge/CPU-optimized ASR)",
|
||||
"funasr": "pip install funasr (SenseVoiceSmall + FSMN-VAD; CUDA or CPU)",
|
||||
"sherpa-onnx-asr": "uv add sherpa-onnx (ONNX live dictation; CPU, cross-platform)",
|
||||
}
|
||||
|
||||
# Most-recent failure per backend, so a transient probe error survives between
|
||||
# Settings refreshes (parity with tts_backend._LAST_ERRORS).
|
||||
_LAST_ERRORS: dict[str, str] = {}
|
||||
|
||||
|
||||
def list_backends() -> list[dict]:
|
||||
"""Enumerate every ASR backend with the **same 11-key shape as TTS** so the
|
||||
Engine Compatibility Matrix renders all families uniformly.
|
||||
|
||||
Per-entry: id, display_name, available, reason (scrubbed), install_hint,
|
||||
last_error, isolation_mode, gpu_compat, effective_device, routing_status,
|
||||
routing_reason. A backend whose ``is_available()`` raises is reported
|
||||
``available: false`` (never a 500), exactly like TTS.
|
||||
"""
|
||||
from core.device_caps import detect_host_caps
|
||||
from core.scrub import scrub_text
|
||||
from services.engine_routing import routing_fields
|
||||
caps = detect_host_caps()
|
||||
|
||||
out: list[dict] = []
|
||||
out = []
|
||||
for bid, cls in _REGISTRY.items():
|
||||
try:
|
||||
ok, msg = cls.is_available()
|
||||
except Exception as exc:
|
||||
ok = False
|
||||
msg = f"{type(exc).__name__}: {exc}"
|
||||
logger.warning(
|
||||
"asr list_backends: %s.is_available() raised — degrading "
|
||||
"gracefully so the picker still renders: %s", bid, msg,
|
||||
)
|
||||
if ok:
|
||||
_LAST_ERRORS.pop(bid, None)
|
||||
else:
|
||||
_LAST_ERRORS[bid] = scrub_text(msg)
|
||||
isolation = "subprocess" if getattr(cls, "_is_subprocess_isolated", False) else "in-process"
|
||||
gpu_compat = getattr(cls, "gpu_compat", ("cpu",))
|
||||
ok, msg = cls.is_available()
|
||||
out.append({
|
||||
"id": bid,
|
||||
"display_name": cls.display_name,
|
||||
"available": ok,
|
||||
# ASR previously emitted `reason` UNMASKED — scrub it now (closes a
|
||||
# pre-existing token-leak gap, matching TTS's redaction guarantee).
|
||||
"reason": None if ok else scrub_text(msg),
|
||||
"install_hint": _INSTALL_HINTS.get(bid),
|
||||
"last_error": _LAST_ERRORS.get(bid),
|
||||
"isolation_mode": isolation,
|
||||
"gpu_compat": list(gpu_compat),
|
||||
**routing_fields(gpu_compat, caps),
|
||||
"reason": None if ok else msg,
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def _probe_available(cls) -> bool:
|
||||
"""``is_available()`` that never raises. A probe that explodes (e.g. a native
|
||||
lib that refuses to load — CTranslate2's exec-stack rejection, #692) means the
|
||||
engine is unusable on this host, so treat it as unavailable and fall through
|
||||
to the next candidate rather than crash engine selection."""
|
||||
try:
|
||||
ok, _ = cls.is_available()
|
||||
return bool(ok)
|
||||
except Exception: # noqa: BLE001
|
||||
logger.warning(
|
||||
"ASR auto-detect: %s.is_available() raised — treating as unavailable",
|
||||
cls.__name__, exc_info=True,
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
def _auto_detect() -> str:
|
||||
"""Pick the best available ASR engine for the current hardware.
|
||||
|
||||
@@ -1608,14 +973,17 @@ def _auto_detect() -> str:
|
||||
4. pytorch-whisper — last resort; requires the TTS model to be loaded
|
||||
so it can reuse `_asr_pipe`.
|
||||
"""
|
||||
if _probe_available(WhisperXBackend):
|
||||
ok, _ = WhisperXBackend.is_available()
|
||||
if ok:
|
||||
return "whisperx"
|
||||
if _probe_available(FasterWhisperBackend):
|
||||
ok, _ = FasterWhisperBackend.is_available()
|
||||
if ok:
|
||||
return "faster-whisper"
|
||||
try:
|
||||
import torch
|
||||
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
if _probe_available(MLXWhisperBackend):
|
||||
ok, _ = MLXWhisperBackend.is_available()
|
||||
if ok:
|
||||
return "mlx-whisper"
|
||||
except Exception:
|
||||
pass
|
||||
@@ -1648,130 +1016,41 @@ def get_active_asr_backend(*, asr_pipe=None) -> ASRBackend:
|
||||
return _REGISTRY[bid]()
|
||||
|
||||
|
||||
def transcribe_reference(audio_path: str) -> str | None:
|
||||
"""Transcribe a voice-clone reference clip with the active ASR backend.
|
||||
|
||||
Voice cloning without a user-supplied transcript used to fall through to
|
||||
``OmniVoice.load_asr_model()`` — a transformers ``pipeline()`` load of
|
||||
whisper-large-v3-turbo that fails outright on transformers 5.3 (#308),
|
||||
even when whisperx / faster-whisper / mlx-whisper are installed and
|
||||
working. Route the reference transcript through the registry instead, so
|
||||
the model-attached pipeline is only reached when it is genuinely the last
|
||||
resort. Returns ``None`` on any failure — callers pass ``ref_text=None``
|
||||
through and the model's built-in fallback still gets its chance.
|
||||
"""
|
||||
try:
|
||||
backend = get_active_asr_backend()
|
||||
except Exception as e: # noqa: BLE001 — never let ASR break generation
|
||||
logger.warning("transcribe_reference: no ASR backend available (%s)", e)
|
||||
return None
|
||||
if isinstance(backend, PyTorchWhisperBackend):
|
||||
# The registry fell through to the model-attached pipeline; let the
|
||||
# model load it lazily rather than constructing a second copy here.
|
||||
return None
|
||||
try:
|
||||
result = backend.transcribe(audio_path, word_timestamps=False)
|
||||
except Exception as e: # noqa: BLE001 — degrade to the model fallback
|
||||
logger.warning(
|
||||
"transcribe_reference: %s failed (%s) — deferring to the model's "
|
||||
"built-in ASR fallback",
|
||||
backend.id, e,
|
||||
)
|
||||
return None
|
||||
result = result or {}
|
||||
text = result.get("text") or " ".join(
|
||||
(seg.get("text") or "").strip() for seg in result.get("segments", [])
|
||||
)
|
||||
text = (text or "").strip()
|
||||
return text or None
|
||||
|
||||
|
||||
_capture_backend: ASRBackend | None = None
|
||||
# The sherpa model id the cached capture backend was built for, so a model
|
||||
# switch in Settings rebuilds the singleton instead of serving the old model.
|
||||
_capture_backend_key: str | None = None
|
||||
|
||||
|
||||
def dictation_model_id() -> str | None:
|
||||
"""The selected sherpa dictation model id, or None when dictation is off /
|
||||
no sherpa model is chosen. Env var wins (power-user pin), then prefs."""
|
||||
explicit = os.environ.get("OMNIVOICE_SHERPA_ASR_MODEL")
|
||||
if explicit:
|
||||
return explicit
|
||||
try:
|
||||
from core import prefs
|
||||
if not prefs.get("dictation.enabled", True):
|
||||
return None
|
||||
mid = prefs.get("dictation.model_id")
|
||||
except Exception:
|
||||
return None
|
||||
from services.sherpa_dictation import is_sherpa_model
|
||||
return mid if is_sherpa_model(mid) else None
|
||||
|
||||
|
||||
def get_capture_asr_backend() -> ASRBackend:
|
||||
"""Pick the fastest ASR engine for capture / dictation.
|
||||
|
||||
Selection order:
|
||||
Priority order (speed-first — word alignment is unnecessary for
|
||||
dictation, so we skip WhisperX's forced-alignment overhead):
|
||||
|
||||
0. sherpa-onnx dictation — when ``dictation.model_id`` names one of the
|
||||
seven sherpa models (live/CPU; the new live-dictation path).
|
||||
1. mlx-whisper Turbo — Apple Silicon, ~5× faster than large-v3
|
||||
2. mlx-whisper large — still native Metal, faster than CPU int8
|
||||
3. faster-whisper — cross-platform CTranslate2 fallback
|
||||
4. pytorch-whisper — last resort
|
||||
1. mlx-whisper Turbo — Apple Silicon, ~5× faster than large-v3
|
||||
2. mlx-whisper large — still native Metal, faster than CPU int8
|
||||
3. faster-whisper — cross-platform CTranslate2 fallback
|
||||
4. pytorch-whisper — last resort
|
||||
|
||||
The caller should also pass ``word_timestamps=False`` to the returned
|
||||
backend to skip per-word timing and shave another ~30% latency.
|
||||
|
||||
Returns a cached singleton so the model stays warm between calls; the
|
||||
singleton is rebuilt if the selected sherpa model changes.
|
||||
Returns a cached singleton so the model stays warm between calls.
|
||||
"""
|
||||
global _capture_backend, _capture_backend_key
|
||||
|
||||
# 0. Honor an explicit sherpa dictation model selection.
|
||||
sherpa_id = dictation_model_id()
|
||||
if sherpa_id:
|
||||
ok, _ = SherpaDictationBackend.is_available()
|
||||
if ok:
|
||||
if not (isinstance(_capture_backend, SherpaDictationBackend)
|
||||
and _capture_backend_key == sherpa_id):
|
||||
try:
|
||||
_capture_backend = SherpaDictationBackend(model_id=sherpa_id)
|
||||
_capture_backend_key = sherpa_id
|
||||
except Exception as e: # noqa: BLE001 — fall through to Whisper
|
||||
logger.warning(
|
||||
"sherpa dictation model %r unavailable (%s) — falling "
|
||||
"back to Whisper capture engine", sherpa_id, e,
|
||||
)
|
||||
_capture_backend = None
|
||||
_capture_backend_key = None
|
||||
if _capture_backend is not None:
|
||||
return _capture_backend
|
||||
else:
|
||||
logger.info(
|
||||
"dictation.model_id=%r selected but sherpa-onnx not installed — "
|
||||
"falling back to Whisper capture engine", sherpa_id,
|
||||
)
|
||||
|
||||
if _capture_backend is not None and _capture_backend_key is None:
|
||||
global _capture_backend
|
||||
if _capture_backend is not None:
|
||||
return _capture_backend
|
||||
|
||||
# Prefer MLX Turbo on Apple Silicon
|
||||
ok, _ = MLXWhisperBackend.is_available()
|
||||
if ok:
|
||||
_capture_backend = MLXWhisperBackend(model_name=_MLX_MODEL_TURBO)
|
||||
_capture_backend_key = None
|
||||
return _capture_backend
|
||||
|
||||
# Fall back to faster-whisper (CPU int8 on non-Apple)
|
||||
ok, _ = FasterWhisperBackend.is_available()
|
||||
if ok:
|
||||
_capture_backend = FasterWhisperBackend()
|
||||
_capture_backend_key = None
|
||||
return _capture_backend
|
||||
|
||||
# Last resort
|
||||
_capture_backend = PyTorchWhisperBackend()
|
||||
_capture_backend_key = None
|
||||
return _capture_backend
|
||||
|
||||
@@ -1,174 +0,0 @@
|
||||
"""Audiobook creator — chapterized long-form narration (parity Wave 5).
|
||||
|
||||
Turns a chapter-delimited script into a chapterized audiobook. This module is
|
||||
the engine-agnostic core:
|
||||
|
||||
* ``parse_audiobook_script`` — pure parser: Markdown ``# H1`` headings become
|
||||
chapters; inline ``[voice:NAME]`` switches the narrator; ``[pause …]`` is
|
||||
delegated to the existing :func:`omnivoice.utils.text.parse_pause_markers`
|
||||
so audiobooks and single-shot synthesis share one pause dialect.
|
||||
* ``synthesize_chapter`` — orchestration: renders a chapter's spans through an
|
||||
injected ``synth(text, voice_id) -> tensor`` callable (reusing the
|
||||
``chunked_tts`` splitter + crossfade), stitching the inter-span silences.
|
||||
Injecting the synth keeps this unit-testable with a stub backend (no torch
|
||||
model, no GPU).
|
||||
* ``build_chapter_ffmetadata`` / ``build_m4b_cmd`` — pure builders for the
|
||||
ffmpeg chapterized-m4b mux (FFMETADATA1 ``[CHAPTER]`` blocks + concat-demux
|
||||
argv). The actual ffmpeg run lives in the (impure) caller.
|
||||
|
||||
Scope (first cut): plain chapter-delimited text/Markdown input. epub/pdf
|
||||
ingestion, the streaming synth job + UI are deferred follow-ups.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Callable, Optional
|
||||
|
||||
@dataclass
|
||||
class Span:
|
||||
"""One contiguous run of text in a single voice, plus trailing silence.
|
||||
|
||||
``speed`` (when set) is the per-span rate passed to the engine — Stories'
|
||||
per-line speed slider rides through here so the shared server render honours
|
||||
it the way the old client export did.
|
||||
"""
|
||||
voice_id: Optional[str]
|
||||
text: str
|
||||
pause_ms_after: int = 0
|
||||
speed: Optional[float] = None
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {"voice_id": self.voice_id, "text": self.text,
|
||||
"pause_ms_after": self.pause_ms_after, "speed": self.speed}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Chapter:
|
||||
title: str
|
||||
spans: list[Span] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def char_count(self) -> int:
|
||||
return sum(len(s.text) for s in self.spans)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {"title": self.title, "char_count": self.char_count,
|
||||
"spans": [s.to_dict() for s in self.spans]}
|
||||
|
||||
|
||||
@dataclass
|
||||
class AudiobookPlan:
|
||||
chapters: list[Chapter] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def char_count(self) -> int:
|
||||
return sum(c.char_count for c in self.chapters)
|
||||
|
||||
@property
|
||||
def chapter_count(self) -> int:
|
||||
return len(self.chapters)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"chapters": [c.to_dict() for c in self.chapters],
|
||||
"chapter_count": self.chapter_count,
|
||||
"char_count": self.char_count,
|
||||
}
|
||||
|
||||
|
||||
def parse_audiobook_script(text: str, *, default_voice: Optional[str] = None) -> AudiobookPlan:
|
||||
"""Parse a chapter-delimited script into an :class:`AudiobookPlan`.
|
||||
|
||||
Thin wrapper over the canonical :func:`services.longform_parser.
|
||||
parse_script_to_spans` (the single grammar source of truth, #27); wraps its
|
||||
span dicts in the ``Span``/``Chapter``/``AudiobookPlan`` dataclasses so the
|
||||
four router call sites and ``.to_dict()`` shape are unchanged.
|
||||
"""
|
||||
from services.longform_parser import parse_script_to_spans
|
||||
|
||||
chapters = [
|
||||
Chapter(title=c["title"], spans=[Span(**s) for s in c["spans"]])
|
||||
for c in parse_script_to_spans(text, default_voice=default_voice)
|
||||
]
|
||||
return AudiobookPlan(chapters=chapters)
|
||||
|
||||
|
||||
def synthesize_chapter(
|
||||
spans: list[Span],
|
||||
synth: Callable[[str, Optional[str], Optional[float]], "object"],
|
||||
sample_rate: int,
|
||||
*,
|
||||
crossfade_ms: int = 50,
|
||||
lexicon: Optional[dict] = None,
|
||||
):
|
||||
"""Render a chapter's spans to one waveform via an injected ``synth``.
|
||||
|
||||
``synth(text, voice_id, speed)`` returns a 1-D float32 audio tensor for a
|
||||
span of text in the given voice (``speed`` may be ``None`` for the engine
|
||||
default). Long spans are split with the ``chunked_tts`` splitter and
|
||||
crossfaded; inter-span ``pause_ms_after`` becomes silence. ``lexicon`` (when
|
||||
given) respells each span's text before chunking so the engine pronounces
|
||||
tricky words correctly; a ``None``/empty lexicon is a no-op pass-through.
|
||||
|
||||
Returns ``(audio_tensor, duration_seconds)``. torch + chunked_tts are
|
||||
imported lazily so this module stays import-light for the pure parser path.
|
||||
"""
|
||||
import torch
|
||||
from services.chunked_tts import concatenate_audio_chunks, split_text_into_chunks
|
||||
from services.pronunciation import apply_lexicon
|
||||
|
||||
parts: list = []
|
||||
for span in spans:
|
||||
if span.text:
|
||||
chunks = split_text_into_chunks(apply_lexicon(span.text, lexicon))
|
||||
rendered = [synth(c, span.voice_id, span.speed) for c in chunks]
|
||||
rendered = [r for r in rendered if r is not None and getattr(r, "numel", lambda: 0)()]
|
||||
if len(rendered) == 1:
|
||||
parts.append(rendered[0])
|
||||
elif rendered:
|
||||
parts.append(concatenate_audio_chunks(rendered, sample_rate, crossfade_ms=crossfade_ms))
|
||||
if span.pause_ms_after > 0:
|
||||
n = int(sample_rate * span.pause_ms_after / 1000.0)
|
||||
if n > 0:
|
||||
parts.append(torch.zeros(n, dtype=torch.float32))
|
||||
|
||||
if not parts:
|
||||
return torch.zeros(0, dtype=torch.float32), 0.0
|
||||
# Hard-concat spans + silences (crossfading silence would bleed the gap).
|
||||
audio = parts[0] if len(parts) == 1 else concatenate_audio_chunks(parts, sample_rate, crossfade_ms=0)
|
||||
return audio, audio.shape[-1] / float(sample_rate)
|
||||
|
||||
|
||||
# ── ffmpeg / metadata builders ──────────────────────────────────────────────
|
||||
#
|
||||
# These now live in the shared ``longform_render`` core (Stories + Audiobook
|
||||
# converge on one mux). The thin wrappers below preserve the original
|
||||
# audiobook-only call sites/signatures; new callers should use
|
||||
# ``longform_render`` directly to reach global metadata, cover art, loudness,
|
||||
# and mp3 output.
|
||||
from services.longform_render import ( # noqa: E402
|
||||
build_concat_list,
|
||||
build_ffmetadata,
|
||||
build_render_cmd,
|
||||
)
|
||||
|
||||
|
||||
def build_chapter_ffmetadata(chapters: list[tuple[str, int]]) -> str:
|
||||
"""Backward-compatible alias: chapters-only FFMETADATA (no global tags)."""
|
||||
return build_ffmetadata(chapters)
|
||||
|
||||
|
||||
def build_m4b_cmd(
|
||||
ffmpeg: str,
|
||||
concat_list_path: str,
|
||||
metadata_path: str,
|
||||
out_path: str,
|
||||
*,
|
||||
bitrate: str = "128k",
|
||||
) -> list[str]:
|
||||
"""Backward-compatible alias: a chapterized faststart m4b, no cover/loudness."""
|
||||
return build_render_cmd(
|
||||
ffmpeg, concat_list_path, metadata_path, out_path,
|
||||
fmt="m4b", bitrate=bitrate,
|
||||
)
|
||||
@@ -151,9 +151,6 @@ async def generate_segments_batched(
|
||||
# Raw: skip all DSP — return raw model output
|
||||
return audio_out
|
||||
|
||||
# TODO(#312): this route runs the OmniVoice model directly (not the active
|
||||
# backend), so VoxCPM2 never reaches it. When these routes become
|
||||
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
|
||||
mastered = apply_mastering(audio_out, sample_rate=sr)
|
||||
effect_chain = get_effect_chain(seg_effect_preset)
|
||||
if effect_chain:
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user