Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
362dca9ae1 | ||
|
|
bbac66ef76 | ||
|
|
3d74611be8 | ||
|
|
ee08f56e1b |
@@ -1,32 +0,0 @@
|
||||
# RTK - Rust Token Killer (Google Antigravity)
|
||||
|
||||
**Usage**: Token-optimized CLI proxy for shell commands.
|
||||
|
||||
## Rule
|
||||
|
||||
Always prefix shell commands with `rtk` to minimize token consumption.
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
rtk git status
|
||||
rtk cargo test
|
||||
rtk ls src/
|
||||
rtk grep "pattern" src/
|
||||
rtk find "*.rs" .
|
||||
rtk docker ps
|
||||
rtk gh pr list
|
||||
```
|
||||
|
||||
## Meta Commands
|
||||
|
||||
```bash
|
||||
rtk gain # Show token savings
|
||||
rtk gain --history # Command history with savings
|
||||
rtk discover # Find missed RTK opportunities
|
||||
rtk proxy <cmd> # Run raw (no filtering, for debugging)
|
||||
```
|
||||
|
||||
## Why
|
||||
|
||||
RTK filters and compresses command output before it reaches the LLM context, saving 60-90% tokens on common operations. Always use `rtk <cmd>` instead of raw commands.
|
||||
+3
-119
@@ -7,26 +7,12 @@
|
||||
language: "en-US"
|
||||
early_access: false
|
||||
|
||||
# The review voice: a panel of senior domain experts, not a linter.
|
||||
tone_instructions: >-
|
||||
Review as a panel of principal engineers: ML inference, audio DSP, desktop
|
||||
systems, product polish. Cite exact lines, name the failure mode, give the
|
||||
concrete fix. No filler praise; raise nits only when they change a decision.
|
||||
|
||||
reviews:
|
||||
# "chill" keeps the bot from blocking merges — it comments, it does not gate.
|
||||
# Hard gating lives in CI (security.yml) and the constitution's human bar.
|
||||
profile: chill
|
||||
request_changes_workflow: false
|
||||
high_level_summary: true
|
||||
# Every walkthrough gets a visual: mermaid sequence diagrams for the
|
||||
# mechanics, plus (via the summary instructions) an ASCII before/after
|
||||
# sketch when the PR touches UI — so each PR is reviewable at a glance.
|
||||
sequence_diagrams: true
|
||||
high_level_summary_instructions: >-
|
||||
If the PR changes UI (JSX/TSX/CSS/Tauri windows), include a compact ASCII
|
||||
before/after sketch of the affected layout or component. If it changes
|
||||
behavior, include a short mermaid flowchart of the new mechanism.
|
||||
review_status: true
|
||||
poem: false
|
||||
|
||||
@@ -53,8 +39,7 @@ reviews:
|
||||
- "!**/*.onnx"
|
||||
- "!tests/fixtures/**"
|
||||
|
||||
# One expert lens per subsystem — encode what a passionate senior in each
|
||||
# domain would actually check, beyond what linters and CI already gate.
|
||||
# Encode the project's hard constraints so the bot reviews against them.
|
||||
path_instructions:
|
||||
- path: "**/*.{py,rs,js,jsx,ts,tsx}"
|
||||
instructions: >-
|
||||
@@ -63,119 +48,18 @@ reviews:
|
||||
model download, or an explicitly opt-in endpoint. Flag any code that
|
||||
persists or logs values matching *TOKEN*/*KEY*/*SECRET* or absolute user
|
||||
home paths (/Users/<name>/, C:\\Users\\<name>\\).
|
||||
- path: "backend/services/**/*.py"
|
||||
instructions: >-
|
||||
Review as an ML-inference/audio engineer. Check: thread-safety of model
|
||||
and cache state across the GPU worker pool; device/dtype assumptions
|
||||
that break on one of CUDA/MPS/ROCm/CPU; VRAM lifecycle (load/unload,
|
||||
leaks on the error path); sample-rate, channel-count and tensor-shape
|
||||
assumptions at engine boundaries; blocking calls inside async paths;
|
||||
model download/cache behavior when offline. Engine code must stay
|
||||
backward-compatible with already-installed on-disk model state.
|
||||
- path: "backend/**/*.py"
|
||||
instructions: >-
|
||||
Default features must behave identically on macOS, Windows and Linux.
|
||||
Platform-specific implementation is allowed, but a divergent user-visible
|
||||
default is a P0 bug — flag it and suggest an opt-in (Settings/env/flag).
|
||||
Any DB schema change must go through an alembic migration with an upgrade
|
||||
path; flag direct schema edits. The backend serves loopback HTTP: treat
|
||||
every query/path/form param as hostile (path traversal, log injection,
|
||||
CSRF from a browser tab), and never route user-chosen filesystem
|
||||
destinations through HTTP — that authorization belongs in the Tauri
|
||||
process.
|
||||
- path: "frontend/src/**/*.{js,jsx,ts,tsx}"
|
||||
instructions: >-
|
||||
Review as a product-minded senior frontend engineer. Check: stale state
|
||||
and races (async results landing after unmount or after newer requests);
|
||||
every user-visible failure has an actionable, non-technical error
|
||||
message; loading/disabled states during long operations. Every new
|
||||
user-facing string must be an i18n t('...') key present in ALL 21
|
||||
frontend/src/i18n/locales/*.json files — flag hardcoded UI strings and
|
||||
keys missing from any locale.
|
||||
- path: "frontend/src-tauri/**/*.rs"
|
||||
instructions: >-
|
||||
Review as a desktop-systems engineer. Check: every #[tauri::command] is
|
||||
callable from the webview — validate inputs and scope filesystem/process
|
||||
access accordingly; window and webview lifecycle on all three OSes;
|
||||
child-process spawn/exit-code/stderr handling; no unwrap/expect on
|
||||
user-controlled input; platform cfg blocks keep user-visible defaults
|
||||
identical across macOS/Windows/Linux.
|
||||
- path: "tests/**/*.py"
|
||||
instructions: >-
|
||||
Review as a test-infrastructure engineer. Check: the test would fail
|
||||
before the fix and pass after (no tautologies); no sleeps as
|
||||
synchronization; no module-level imports of app modules that go stale
|
||||
under sys.modules pollution (resolve at run time); TestClient instances
|
||||
are function-scoped and not lifespan-bound unless the test needs it;
|
||||
new functional CJK is allowlisted in tests/test_no_hardcoded_cjk.py with
|
||||
a justification.
|
||||
path; flag direct schema edits. Engine code must stay backward-compatible
|
||||
with already-installed on-disk model state (no forced reinstall).
|
||||
- path: ".github/workflows/**"
|
||||
instructions: >-
|
||||
Pin actions to a major version tag at minimum. Flag any workflow that
|
||||
grants write permissions it does not need.
|
||||
|
||||
# Non-gating pre-merge audits of the project's hard rules (warning mode —
|
||||
# the human owner is the gate, these make the checklist visible per-PR).
|
||||
pre_merge_checks:
|
||||
title:
|
||||
mode: warning
|
||||
requirements: >-
|
||||
Conventional-commit style with scope, e.g. "fix(dub): …", and the issue
|
||||
reference in title or body. Never propose a version bump.
|
||||
issue_assessment:
|
||||
mode: warning
|
||||
custom_checks:
|
||||
- name: "Cross-platform default parity"
|
||||
mode: warning
|
||||
instructions: >-
|
||||
If the PR changes any default-mode (out-of-the-box) behavior, verify
|
||||
it behaves identically on macOS, Windows and Linux, or is moved behind
|
||||
an explicit opt-in (Settings toggle, env var, or CLI flag). A
|
||||
platform-divergent default is a P0 per CLAUDE.md — fail this check
|
||||
and say which platform diverges.
|
||||
- name: "i18n completeness (21 locales)"
|
||||
mode: warning
|
||||
instructions: >-
|
||||
For every new or changed t('...') key in frontend code, verify the
|
||||
key exists in all 21 files under frontend/src/i18n/locales/. List any
|
||||
locale files missing the key. Also flag hardcoded user-facing strings
|
||||
that bypass i18n entirely.
|
||||
- name: "Local-first guarantee"
|
||||
mode: warning
|
||||
instructions: >-
|
||||
Verify the PR adds no required cloud calls, accounts, API keys or
|
||||
telemetry. Outbound traffic is only allowed to GitHub Issues (opt-in
|
||||
bug reporting) and HuggingFace model downloads. The app must remain
|
||||
fully functional offline and with reporting disabled.
|
||||
- name: "Backward compatibility"
|
||||
mode: warning
|
||||
instructions: >-
|
||||
Verify existing omnivoice_data/ (voices, projects, settings) and
|
||||
already-installed engine model state keep working without manual
|
||||
migration. Any DB schema change must ship an alembic migration with
|
||||
an upgrade path. Flag anything that would force users to reinstall an
|
||||
engine or re-download model weights.
|
||||
|
||||
finishing_touches:
|
||||
docstrings:
|
||||
enabled: true
|
||||
unit_tests:
|
||||
enabled: true
|
||||
|
||||
# Feed the bot the project constitution and docs, and let it accumulate
|
||||
# learnings from review conversations ("@coderabbitai always/never …").
|
||||
knowledge_base:
|
||||
code_guidelines:
|
||||
enabled: true
|
||||
filePatterns:
|
||||
- "CLAUDE.md"
|
||||
- "docs/**/*.md"
|
||||
learnings:
|
||||
scope: auto
|
||||
issues:
|
||||
scope: auto
|
||||
pull_requests:
|
||||
scope: auto
|
||||
|
||||
chat:
|
||||
auto_reply: true
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
name: 🐛 Bug Report
|
||||
about: Report a bug to help us improve OmniVoice Studio
|
||||
title: "[Bug] "
|
||||
labels: ["bug", "triage"]
|
||||
assignees: []
|
||||
---
|
||||
|
||||
## Describe the bug
|
||||
|
||||
A clear and concise description of what the bug is.
|
||||
|
||||
## To reproduce
|
||||
|
||||
Steps to reproduce the behavior:
|
||||
1. Go to '...'
|
||||
2. Click on '...'
|
||||
3. See error
|
||||
|
||||
## Expected behavior
|
||||
|
||||
What you expected to happen.
|
||||
|
||||
## Screenshots / Logs
|
||||
|
||||
If applicable, add screenshots or paste relevant logs from **Settings → Logs**.
|
||||
|
||||
> **Tip:** **Settings → About → "Save diagnostic bundle"** produces a zip
|
||||
> (self-check report, recent errors, scrubbed log tails) you can drag onto
|
||||
> this issue — it answers most environment questions below automatically.
|
||||
> Headless installs: `python backend/main.py --diagnose` prints the same
|
||||
> self-check (`--deep` also test-loads the active engine).
|
||||
|
||||
## Environment
|
||||
|
||||
- **OS:** [e.g. macOS 15.2, Windows 11, Ubuntu 24.04]
|
||||
- **Install method:** [Desktop app / Docker / From source]
|
||||
- **Version:** [e.g. v0.2.7 — check Settings → About]
|
||||
- **GPU:** [e.g. NVIDIA RTX 4090 / Apple M3 Pro / CPU only]
|
||||
- **RAM:** [e.g. 16 GB]
|
||||
- **Active TTS engine:** [e.g. omnivoice — check Settings → Engines]
|
||||
|
||||
## Additional context
|
||||
|
||||
Add any other context about the problem here.
|
||||
@@ -1,109 +0,0 @@
|
||||
name: 🐛 Bug report
|
||||
description: Something works incorrectly or crashes (not a first-run/install problem — use the install template for those).
|
||||
title: "[Bug] "
|
||||
labels: ["bug", "triage"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Thanks for helping improve OmniVoice Studio! 🎙️
|
||||
|
||||
**Fastest path to a fix:** **Settings → About → "Save diagnostic bundle"** makes a
|
||||
zip (self-check + recent errors + scrubbed log tails) — drag it onto this issue and
|
||||
most of the environment questions below are answered automatically.
|
||||
Headless: `python backend/main.py --diagnose` (add `--deep` to test-load the engine).
|
||||
- type: checkboxes
|
||||
id: preflight
|
||||
attributes:
|
||||
label: Before filing
|
||||
options:
|
||||
- label: I searched [existing issues](https://github.com/debpalash/OmniVoice-Studio/issues?q=is%3Aissue) and this isn't a duplicate.
|
||||
required: true
|
||||
- label: I'm on the latest release (or `main`) — older builds may already be fixed.
|
||||
required: false
|
||||
- type: textarea
|
||||
id: what-happened
|
||||
attributes:
|
||||
label: What happened?
|
||||
description: A clear description of the bug, including the exact error text / toast if any.
|
||||
placeholder: "Voice cloning failed with '…' after I clicked Generate."
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: repro
|
||||
attributes:
|
||||
label: Steps to reproduce
|
||||
value: |
|
||||
1.
|
||||
2.
|
||||
3.
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: expected
|
||||
attributes:
|
||||
label: What did you expect instead?
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: os
|
||||
attributes:
|
||||
label: Operating system
|
||||
options:
|
||||
- macOS (Apple Silicon)
|
||||
- macOS (Intel)
|
||||
- Windows (x64)
|
||||
- Linux (AppImage)
|
||||
- Linux (.deb)
|
||||
- Linux (other / from source)
|
||||
- Docker
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: install
|
||||
attributes:
|
||||
label: How did you install it?
|
||||
options:
|
||||
- Desktop app (installer / AppImage)
|
||||
- Docker image
|
||||
- From source (uv sync)
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: version
|
||||
attributes:
|
||||
label: Version
|
||||
description: Settings → About (e.g. v0.3.5), or the Docker tag / git SHA.
|
||||
placeholder: "v0.3.5"
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: device
|
||||
attributes:
|
||||
label: Compute device
|
||||
options:
|
||||
- NVIDIA GPU (CUDA)
|
||||
- AMD GPU (ROCm)
|
||||
- Apple Silicon (MPS)
|
||||
- Intel GPU (XPU)
|
||||
- CPU only
|
||||
- Not sure
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: engine
|
||||
attributes:
|
||||
label: Active TTS/ASR engine
|
||||
description: Settings → Engines (e.g. omnivoice, cosyvoice, indextts2, whisperx).
|
||||
placeholder: "omnivoice"
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Logs / diagnostic bundle
|
||||
description: Drag the diagnostic bundle here, or paste relevant lines from **Settings → Logs**. Secrets are scrubbed automatically.
|
||||
render: text
|
||||
- type: textarea
|
||||
id: extra
|
||||
attributes:
|
||||
label: Anything else?
|
||||
description: Screenshots, the input that triggered it, RAM/VRAM, etc.
|
||||
@@ -1,11 +0,0 @@
|
||||
blank_issues_enabled: false
|
||||
contact_links:
|
||||
- name: 💬 Discord — questions & quick help
|
||||
url: https://discord.gg/bzQavDfVV9
|
||||
about: Usage questions, setup help, and chat. Faster than an issue for "how do I…".
|
||||
- name: 🗣️ GitHub Discussions
|
||||
url: https://github.com/debpalash/OmniVoice-Studio/discussions
|
||||
about: Ideas, show-and-tell, and open-ended Q&A that isn't a bug or a specific feature ask.
|
||||
- name: 🔒 Security vulnerability
|
||||
url: https://github.com/debpalash/OmniVoice-Studio/security/policy
|
||||
about: Please report security issues privately — do NOT open a public issue.
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
name: ✨ Feature Request
|
||||
about: Suggest an idea for OmniVoice Studio
|
||||
title: "[Feature] "
|
||||
labels: ["enhancement"]
|
||||
assignees: []
|
||||
---
|
||||
|
||||
## Is your feature request related to a problem?
|
||||
|
||||
A clear description of what the problem is. Ex. "I'm always frustrated when..."
|
||||
|
||||
## Describe the solution you'd like
|
||||
|
||||
A clear description of what you want to happen.
|
||||
|
||||
## Describe alternatives you've considered
|
||||
|
||||
Any alternative solutions or features you've considered.
|
||||
|
||||
## Additional context
|
||||
|
||||
Add any other context, mockups, or screenshots about the feature request here.
|
||||
@@ -1,50 +0,0 @@
|
||||
name: ✨ Feature request
|
||||
description: Suggest an improvement or a new capability.
|
||||
title: "[Feature] "
|
||||
labels: ["enhancement"]
|
||||
body:
|
||||
- type: checkboxes
|
||||
id: preflight
|
||||
attributes:
|
||||
label: Before filing
|
||||
options:
|
||||
- label: I searched [existing issues](https://github.com/debpalash/OmniVoice-Studio/issues?q=is%3Aissue) and [discussions](https://github.com/debpalash/OmniVoice-Studio/discussions) for this idea.
|
||||
required: true
|
||||
- type: textarea
|
||||
id: problem
|
||||
attributes:
|
||||
label: What problem does this solve?
|
||||
description: The use case / friction this addresses ("When I … I can't …").
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: solution
|
||||
attributes:
|
||||
label: Proposed solution
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: alternatives
|
||||
attributes:
|
||||
label: Alternatives considered
|
||||
- type: dropdown
|
||||
id: area
|
||||
attributes:
|
||||
label: Area
|
||||
options:
|
||||
- Voice cloning
|
||||
- Voice design
|
||||
- Video dubbing
|
||||
- Real-time dictation
|
||||
- Audiobook / Stories (long-form)
|
||||
- TTS/ASR engines
|
||||
- Install / setup / packaging
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
> OmniVoice is **local-first** — features must work fully offline with no accounts,
|
||||
API keys, or cloud calls, and behave identically on macOS/Windows/Linux. Proposals
|
||||
that fit those constraints are easiest to land.
|
||||
@@ -1,86 +0,0 @@
|
||||
name: 🧩 Install / first-run problem
|
||||
description: The app won't install, set up, download models, or reach a first working output.
|
||||
title: "[Install] "
|
||||
labels: ["install", "triage"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
A first-run that *just works* is the whole point — sorry it didn't. Let's fix it.
|
||||
If the app launched far enough to open Settings, **Settings → About → "Save diagnostic
|
||||
bundle"** captures most of this; otherwise the fields below are enough.
|
||||
- type: dropdown
|
||||
id: stage
|
||||
attributes:
|
||||
label: Where did it fail?
|
||||
options:
|
||||
- App won't launch / blank or broken window
|
||||
- Python / uv environment bootstrap
|
||||
- Model download (HuggingFace)
|
||||
- Engine install (CosyVoice / IndexTTS / MLX / etc.)
|
||||
- First synthesis / dub never completes
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: error
|
||||
attributes:
|
||||
label: The error
|
||||
description: The exact message, traceback, or what you see on screen.
|
||||
render: text
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: os
|
||||
attributes:
|
||||
label: Operating system
|
||||
options:
|
||||
- macOS (Apple Silicon)
|
||||
- macOS (Intel)
|
||||
- Windows (x64)
|
||||
- Linux (AppImage)
|
||||
- Linux (.deb)
|
||||
- Linux (other / from source)
|
||||
- Docker
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: install
|
||||
attributes:
|
||||
label: How are you installing it?
|
||||
options:
|
||||
- Desktop app (installer / AppImage)
|
||||
- Docker image
|
||||
- From source (uv sync)
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: version
|
||||
attributes:
|
||||
label: Version
|
||||
placeholder: "v0.3.5 (or installer build / git SHA)"
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: network
|
||||
attributes:
|
||||
label: Network conditions (model/dependency downloads)
|
||||
description: Restricted networks are a known source of bootstrap failures (mirror fallback).
|
||||
options:
|
||||
- Normal / unrestricted
|
||||
- Behind a corporate proxy / firewall
|
||||
- Region with restricted access (e.g. China, Russia)
|
||||
- Offline / air-gapped
|
||||
- Not sure
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Logs / diagnostic bundle
|
||||
description: Drag a diagnostic bundle, or paste the install/bootstrap log. Headless self-check — `python backend/main.py --diagnose`.
|
||||
render: text
|
||||
- type: textarea
|
||||
id: tried
|
||||
attributes:
|
||||
label: What have you already tried?
|
||||
@@ -18,7 +18,7 @@
|
||||
- [ ] 📝 Documentation
|
||||
- [ ] 🧪 Tests
|
||||
- [ ] 🔧 CI / Build
|
||||
- [ ] 🚀 Release prep
|
||||
- [ ] 🚀 Release prep (RC or final)
|
||||
|
||||
## Testing
|
||||
|
||||
@@ -33,12 +33,17 @@
|
||||
- [ ] No local machine paths, logs, or personal env details in this PR
|
||||
- [ ] Version files are in sync (if version bump): `pyproject.toml`, `package.json`, `tauri.conf.json`, `Cargo.toml`
|
||||
- [ ] If this PR changes runtime behavior, the regression fixture at `tests/fixtures/omnivoice_data/` still loads green on the `smoke-matrix` CI job (macOS + Windows + Linux)
|
||||
- [ ] If this is part of a release, I've read the "Release cadence" section below and confirmed this PR targets the right RC
|
||||
|
||||
## Release cadence
|
||||
## Release cadence (read once per RC)
|
||||
|
||||
OmniVoice ships **continuous-to-main** — no release candidates, no soak windows.
|
||||
Every merged PR is immediately part of the rolling preview (`main`, Docker
|
||||
`:latest`, the desktop Preview channel). Versioned releases are tagged from
|
||||
`main` when it's ready; `main` then bumps to the next patch automatically.
|
||||
Users who want stability pin a release tag / Docker `:stable` / the desktop
|
||||
Stable channel.
|
||||
OmniVoice ships every minor on a **two-RC cadence**:
|
||||
- `vX.Y.0-rc1` — cut from `main` once all GATE-* requirements pass; clean-VM exercise on 4 OSes (per `REL-01`)
|
||||
- 48-hour soak (no new commits to release branch except fix-forward)
|
||||
- `vX.Y.0` — promotion if rc1 is clean
|
||||
|
||||
If your PR touches install / bootstrap / CI, it MUST land before rc1 cut, not between rc1 and the promotion. During a soak, any merge needs explicit OK from the release captain.
|
||||
|
||||
## Screenshots
|
||||
|
||||
<!-- If applicable, add screenshots or recordings. -->
|
||||
|
||||
@@ -92,10 +92,7 @@ jobs:
|
||||
|
||||
- name: Install frontend deps
|
||||
working-directory: frontend
|
||||
# --frozen-lockfile so a frontend/package.json change that forgets to
|
||||
# regenerate the root bun.lock fails HERE (fast) instead of only in the
|
||||
# Docker build (deploy/Dockerfile), which is what reddened main on #485.
|
||||
run: bun install --frozen-lockfile
|
||||
run: bun install
|
||||
|
||||
# checkJs is true in tsconfig for IDE feedback, but 947 pre-existing
|
||||
# JS errors remain. Override to false in CI so only .ts files block.
|
||||
@@ -177,10 +174,7 @@ jobs:
|
||||
|
||||
- name: Install frontend deps
|
||||
working-directory: frontend
|
||||
# --frozen-lockfile so a frontend/package.json change that forgets to
|
||||
# regenerate the root bun.lock fails HERE (fast) instead of only in the
|
||||
# Docker build (deploy/Dockerfile), which is what reddened main on #485.
|
||||
run: bun install --frozen-lockfile
|
||||
run: bun install
|
||||
|
||||
# tauri-build's setup hook reads tauri.conf.json's `frontendDist`
|
||||
# ("../dist"), which only exists after a frontend build. Without this,
|
||||
|
||||
@@ -5,21 +5,14 @@
|
||||
# - push to main branch → :main, :sha- (rolling "edge" build)
|
||||
# - workflow_dispatch → :sha- only (ad-hoc test build)
|
||||
#
|
||||
# Tag ↔ image mapping (versioning hard rule, owner-set 2026-06-11:
|
||||
# :latest IS the preview channel; stable users pin :stable or a version tag)
|
||||
# :latest — rolling preview: latest commit on main (always last release + 1 dev)
|
||||
# :main — alias of the same rolling main build (kept for back-compat)
|
||||
# :stable — most recent versioned release (set on every v* tag push)
|
||||
# :0.3.6 — exact version from the git tag
|
||||
# Tag ↔ image mapping
|
||||
# :latest — always the most recent versioned release (set on every v* tag push)
|
||||
# :0.3.0 — exact version from the git tag
|
||||
# :0.3 — major.minor floating tag (updated on every patch within the minor)
|
||||
# :main — latest commit on main; may be ahead of the last tagged release
|
||||
# :sha-xxxx — specific commit SHA; produced by workflow_dispatch
|
||||
#
|
||||
# Images land at: ghcr.io/debpalash/omnivoice-studio AND docker.io/palashdeb/omnivoice-studio
|
||||
# (Docker Hub push gated on the DOCKERHUB_USERNAME/DOCKERHUB_TOKEN secrets;
|
||||
# if unset the build still pushes to GHCR.)
|
||||
#
|
||||
# On main pushes the Docker Hub repository overview is also synced from
|
||||
# deploy/dockerhub-overview.md (source of truth for the hub.docker.com page).
|
||||
# Images land at: ghcr.io/debpalash/omnivoice-studio
|
||||
#
|
||||
# NOTE: the Docker image is the headless web-server build of OmniVoice (FastAPI
|
||||
# backend + pre-built React frontend served over HTTP). The Tauri desktop
|
||||
@@ -41,7 +34,6 @@ permissions:
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_NAME: ${{ github.repository }}
|
||||
DOCKERHUB_IMAGE: palashdeb/omnivoice-studio
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
|
||||
|
||||
jobs:
|
||||
@@ -64,43 +56,28 @@ jobs:
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
# Docker Hub login — only when the secret is present, so forks / runs
|
||||
# without the credential still publish to GHCR.
|
||||
- name: Check Docker Hub credentials
|
||||
id: dockerhub
|
||||
run: echo "enabled=${{ secrets.DOCKERHUB_TOKEN != '' }}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Log in to Docker Hub
|
||||
if: steps.dockerhub.outputs.enabled == 'true'
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
# Tag strategy (`:sha-<short>` is emitted on every trigger):
|
||||
# v0.3.6 tag push → :0.3.6, :0.3, :stable, :sha-
|
||||
# main branch push → :latest, :main, :sha-
|
||||
# v0.3.0 tag push → :0.3.0, :0.3, :latest, :sha-
|
||||
# main branch push → :main, :sha-
|
||||
# workflow_dispatch → :sha- only
|
||||
#
|
||||
# All mutable-tag rules stay gated on `github.event_name == 'push'` so a
|
||||
# manual workflow_dispatch can only ever produce a throwaway `:sha-` tag
|
||||
# (the stale-:latest fix from #249/#251). :stable excludes prerelease
|
||||
# tags (those contain a `-`) so a prerelease can't clobber it.
|
||||
# Fix for stale :latest (issues #249, #251):
|
||||
# The previous rule used `enable={{is_default_branch}}`, which evaluates
|
||||
# to false on tag pushes (detached HEAD) — so :latest was never updated
|
||||
# when a release tag was pushed. The version / :latest / :main rules are
|
||||
# gated on `github.event_name == 'push'` so a manual workflow_dispatch can
|
||||
# only ever produce a throwaway `:sha-` tag (never republish a mutable
|
||||
# tag), and :latest additionally excludes prerelease tags (those contain a
|
||||
# `-`, e.g. v1.0.0-rc.1) so a prerelease can't clobber :latest.
|
||||
- name: Extract metadata (tags, labels)
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
# Same tag set applied to both registries. The Docker Hub line is
|
||||
# blank when the secret is unset, so metadata-action emits GHCR-only
|
||||
# tags in that case.
|
||||
images: |
|
||||
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
${{ steps.dockerhub.outputs.enabled == 'true' && env.DOCKERHUB_IMAGE || '' }}
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
type=semver,pattern={{version}},enable=${{ github.event_name == 'push' }}
|
||||
type=semver,pattern={{major}}.{{minor}},enable=${{ github.event_name == 'push' }}
|
||||
type=raw,value=stable,enable=${{ github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-') }}
|
||||
type=raw,value=latest,enable=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
type=raw,value=latest,enable=${{ github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-') }}
|
||||
type=raw,value=main,enable=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
type=sha,prefix=sha-,format=short
|
||||
|
||||
@@ -114,25 +91,3 @@ jobs:
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
# Sync the Docker Hub repository overview from deploy/dockerhub-overview.md.
|
||||
# Only on main pushes (the overview tracks the rolling preview) and only
|
||||
# when Docker Hub creds are present, mirroring the push gating above.
|
||||
#
|
||||
# continue-on-error: the overview text is cosmetic, and the description
|
||||
# PATCH 403s unless DOCKERHUB_TOKEN carries description-edit scope (many
|
||||
# fine-grained Docker Hub tokens that can push still can't edit the
|
||||
# description). The image build+push is what matters — a creds-scope
|
||||
# mismatch on this cosmetic step must not fail the whole Docker run. To
|
||||
# actually sync the overview, use a token with read/write (incl.
|
||||
# description) scope, or the account password.
|
||||
- name: Update Docker Hub description
|
||||
if: steps.dockerhub.outputs.enabled == 'true' && github.event_name == 'push' && github.ref == 'refs/heads/main'
|
||||
continue-on-error: true
|
||||
uses: peter-evans/dockerhub-description@v4
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
repository: ${{ env.DOCKERHUB_IMAGE }}
|
||||
short-description: "Local ElevenLabs alternative: voice cloning, design & video dubbing in 646 languages. No API keys."
|
||||
readme-filepath: ./deploy/dockerhub-overview.md
|
||||
|
||||
@@ -1,97 +0,0 @@
|
||||
# Docs drift — daily inventory-vs-docs check with a single rolling issue.
|
||||
#
|
||||
# docs/features.yaml is the canonical inventory; scripts/check-docs-drift.py
|
||||
# diffs it against README.md, docs/, and the engine registries. On drift the
|
||||
# job updates (or creates) ONE issue labeled `docs-drift` in place — no issue
|
||||
# spam — and closes it automatically when the check is clean again.
|
||||
#
|
||||
# Companion to the PR-gating validate-install-docs.py step in ci.yml.
|
||||
# Spec: docs/competitive-analysis.md Spec 9a / parity program Wave 0.1.
|
||||
# Rolling-issue pattern adapted from Patter (MIT).
|
||||
|
||||
name: docs-drift
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Daily 03:30 UTC — after most merges, before EU morning triage.
|
||||
- cron: "30 3 * * *"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
drift:
|
||||
runs-on: ubuntu-22.04
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
- name: Install checker deps
|
||||
run: pip install "pyyaml>=6"
|
||||
|
||||
- name: Check inventory vs README/docs/registries
|
||||
id: drift
|
||||
continue-on-error: true
|
||||
run: python scripts/check-docs-drift.py --output drift-report.md
|
||||
|
||||
- name: Update rolling docs-drift issue
|
||||
uses: actions/github-script@v7
|
||||
env:
|
||||
DRIFT_OUTCOME: ${{ steps.drift.outcome }}
|
||||
with:
|
||||
script: |
|
||||
const fs = require('fs');
|
||||
const drifted = process.env.DRIFT_OUTCOME === 'failure';
|
||||
const { owner, repo } = context.repo;
|
||||
const label = 'docs-drift';
|
||||
|
||||
const open = await github.rest.issues.listForRepo({
|
||||
owner, repo, state: 'open', labels: label, per_page: 5,
|
||||
});
|
||||
|
||||
if (drifted) {
|
||||
let body = '';
|
||||
try {
|
||||
body = fs.readFileSync('drift-report.md', 'utf8');
|
||||
} catch {
|
||||
body = '# Docs drift report\n\nThe checker failed before writing a report — see the workflow run logs.';
|
||||
}
|
||||
body += `\n\n---\n_Last checked by [run ${context.runId}](https://github.com/${owner}/${repo}/actions/runs/${context.runId})._\n`;
|
||||
|
||||
if (open.data.length > 0) {
|
||||
await github.rest.issues.update({
|
||||
owner, repo, issue_number: open.data[0].number, body,
|
||||
});
|
||||
core.info(`Updated rolling issue #${open.data[0].number}`);
|
||||
} else {
|
||||
const created = await github.rest.issues.create({
|
||||
owner, repo,
|
||||
title: 'docs-drift: feature inventory vs docs mismatch',
|
||||
body,
|
||||
labels: [label, 'documentation'],
|
||||
});
|
||||
core.info(`Created rolling issue #${created.data.number}`);
|
||||
}
|
||||
} else {
|
||||
for (const issue of open.data) {
|
||||
await github.rest.issues.createComment({
|
||||
owner, repo, issue_number: issue.number,
|
||||
body: 'Drift resolved — nightly check is clean again. Closing automatically.',
|
||||
});
|
||||
await github.rest.issues.update({
|
||||
owner, repo, issue_number: issue.number, state: 'closed',
|
||||
});
|
||||
core.info(`Closed rolling issue #${issue.number}`);
|
||||
}
|
||||
}
|
||||
|
||||
- name: Surface drift as a failed run
|
||||
if: steps.drift.outcome == 'failure'
|
||||
run: |
|
||||
echo "Docs drift detected — see the rolling docs-drift issue."
|
||||
exit 1
|
||||
@@ -1,55 +0,0 @@
|
||||
# LLM-judge evals — semantic quality suites, NEVER a gate.
|
||||
#
|
||||
# Hard rule (parity program Wave 0.3 / competitive-analysis Spec 9b): LLM
|
||||
# judges never gate CI. This workflow is scheduled + manual only, the eval
|
||||
# step is continue-on-error, and the JSON report is the deliverable
|
||||
# (uploaded as an artifact). Deterministic probe judges in ci.yml remain
|
||||
# the only gates.
|
||||
#
|
||||
# On the hosted runner there is no local LLM endpoint, so the run usually
|
||||
# reports "skipped — no LLM backend configured"; the workflow exists so the
|
||||
# suites run anywhere a TRANSLATE_BASE_URL secret/endpoint is provided
|
||||
# (e.g. a self-hosted runner with Ollama).
|
||||
|
||||
name: evals
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Weekly, Sundays 04:00 UTC.
|
||||
- cron: "0 4 * * 0"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
evals:
|
||||
runs-on: ubuntu-22.04
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
- uses: astral-sh/setup-uv@v3
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
|
||||
- name: Install deps
|
||||
run: uv sync
|
||||
|
||||
- name: Run eval suites (non-gating)
|
||||
continue-on-error: true
|
||||
env:
|
||||
TRANSLATE_BASE_URL: ${{ secrets.EVALS_LLM_BASE_URL }}
|
||||
TRANSLATE_API_KEY: ${{ secrets.EVALS_LLM_API_KEY }}
|
||||
run: uv run python tests/evals/run_evals.py --output eval-report.json
|
||||
|
||||
- name: Upload report artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: eval-report
|
||||
path: eval-report.json
|
||||
if-no-files-found: warn
|
||||
+25
-203
@@ -4,16 +4,12 @@
|
||||
# - push of a tag matching `v*` (e.g. `v0.2.0`) → full STABLE release,
|
||||
# publishes artifacts + signed updater manifest (`latest.json`) to the
|
||||
# tag's GH Release. This is the default Stable updater channel.
|
||||
# - schedule (nightly, 07:00 UTC) → rolling `preview` PRERELEASE built from
|
||||
# `main` with its own signed `latest.json` at releases/download/preview/.
|
||||
# Feeds the opt-in Preview updater channel (Settings → About → Update
|
||||
# channel). The `preview-gate` job skips the matrix on nights when `main`
|
||||
# didn't move, so an idle day costs only a ~30s gate job — keeping Preview
|
||||
# ≤24h behind `main` at a predictable ~1-matrix/day cost. The stable
|
||||
# `latest` release is untouched.
|
||||
# - workflow_dispatch (publish_preview=true) → the same preview build on
|
||||
# demand from the selected branch (e.g. to preview a feature branch, or to
|
||||
# refresh immediately without waiting for the nightly).
|
||||
# - workflow_dispatch (publish_preview=true) → builds the selected branch and
|
||||
# publishes a rolling `preview` PRERELEASE with its own signed
|
||||
# `latest.json` at releases/download/preview/. This feeds the opt-in
|
||||
# Preview updater channel (Settings → About → Update channel). The stable
|
||||
# `latest` release is untouched. Run this manually whenever you want to cut
|
||||
# a preview from `main`.
|
||||
# - workflow_dispatch (publish_preview=false) → on-demand build (prior
|
||||
# behavior; draft release named after the branch).
|
||||
#
|
||||
@@ -32,10 +28,6 @@ name: Desktop Release
|
||||
on:
|
||||
push:
|
||||
tags: ['v*']
|
||||
schedule:
|
||||
# 07:00 UTC daily — rolling `preview` prerelease from `main`. The
|
||||
# preview-gate job no-ops the matrix when main hasn't moved in a day.
|
||||
- cron: '0 7 * * *'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
draft:
|
||||
@@ -129,41 +121,8 @@ jobs:
|
||||
working-directory: frontend
|
||||
run: node --experimental-strip-types --no-warnings --test ../tests/frontend/*.test.mjs
|
||||
|
||||
# Decide preview-vs-stable, and for nightly runs whether `main` actually
|
||||
# moved in the last day. Outputs gate the expensive matrix (`build`) and the
|
||||
# `preview-notes` job, so a no-commit night costs only this ~30s job.
|
||||
preview-gate:
|
||||
name: Preview gate
|
||||
runs-on: ubuntu-22.04
|
||||
outputs:
|
||||
is_preview: ${{ steps.decide.outputs.is_preview }}
|
||||
proceed: ${{ steps.decide.outputs.proceed }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 50
|
||||
- id: decide
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
event="${{ github.event_name }}"
|
||||
if [ "$event" = "schedule" ] || { [ "$event" = "workflow_dispatch" ] && [ "${{ inputs.publish_preview }}" = "true" ]; }; then
|
||||
echo "is_preview=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "is_preview=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
# Nightly: skip the matrix when main hasn't moved in the last day.
|
||||
if [ "$event" = "schedule" ] && [ -z "$(git log --since='25 hours ago' --oneline)" ]; then
|
||||
echo "No new commits on main in the last day — skipping nightly preview."
|
||||
echo "proceed=false" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "proceed=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
build:
|
||||
needs: [test, preview-gate]
|
||||
# Nightly runs with no new commits on main skip the 4-platform matrix.
|
||||
if: needs.preview-gate.outputs.proceed == 'true'
|
||||
needs: test
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -174,28 +133,9 @@ jobs:
|
||||
rust_target: aarch64-apple-darwin
|
||||
bundles: "app,dmg,updater"
|
||||
|
||||
# macOS Intel (#279): reinstated. The earlier "Rosetta 2 runs the
|
||||
# ARM build" rationale for dropping it was backwards — Rosetta only
|
||||
# translates x86_64→arm64, so Intel Macs (supported through macOS
|
||||
# Sequoia) simply cannot run the aarch64 bundle and had NO
|
||||
# installable artifact. Runner: `macos-15-intel`, GitHub's
|
||||
# designated migration target after macos-13 retired (Dec 2025);
|
||||
# it's a standard (public-repo-free) image supported through
|
||||
# August 2027 — the last x86_64 image Actions will offer. Building
|
||||
# natively (not cross-compiling from the arm64 leg) keeps the
|
||||
# per-TRIPLE uv/ffmpeg sidecar fetches, the DMG installer smoke,
|
||||
# and the ad-hoc signing verification (scripts/
|
||||
# verify-macos-signing.sh, PR #290) all exercising the real
|
||||
# x86_64 artifact on real Intel hardware. The macos-13 queue
|
||||
# backlog that motivated the original drop is contained by
|
||||
# fail-fast:false — a slow Intel leg can delay the release run but
|
||||
# can't fail the other targets.
|
||||
- os: macos-15-intel
|
||||
arch: x86_64-apple-darwin
|
||||
label: "macOS Intel"
|
||||
rust_target: x86_64-apple-darwin
|
||||
bundles: "app,dmg,updater"
|
||||
|
||||
# macOS Intel dropped: Apple shipped the last Intel Mac in 2023 and
|
||||
# Rosetta 2 runs the ARM build natively. macos-13 runner backlog
|
||||
# was also blocking every release tag for ~10 min.
|
||||
# Windows: force MSI bundling via --bundles. NSIS fails at makensis
|
||||
# because our PyInstaller payload approaches its ~2 GB stub limit.
|
||||
- os: windows-2022
|
||||
@@ -351,10 +291,7 @@ jobs:
|
||||
case "$TRIPLE" in
|
||||
aarch64-apple-darwin|x86_64-apple-darwin)
|
||||
# evermeet.cx ships each binary as a separate .zip containing
|
||||
# a single x86_64 Mach-O executable — natively correct on the
|
||||
# Intel leg, and runs via Rosetta 2 on the arm64 leg. Both
|
||||
# darwin TRIPLEs therefore bundle the same payload; only the
|
||||
# sidecar filename suffix differs.
|
||||
# a single x86_64 Mach-O executable (runs via Rosetta on arm64).
|
||||
for TOOL in ffmpeg ffprobe; do
|
||||
if [ "$TOOL" = "ffmpeg" ]; then
|
||||
URL="https://evermeet.cx/ffmpeg/getrelease/zip"
|
||||
@@ -461,21 +398,17 @@ jobs:
|
||||
# always reported the static 0.3.0 never looked "newer", so no update was
|
||||
# ever delivered). Ephemeral, CI-only — never committed. Tauri reads the
|
||||
# bundle + updater version from tauri.conf.json, so rewriting it here
|
||||
# stamps the artifacts + latest.json. Under the versioning hard rule
|
||||
# (owner-set 2026-06-11) main is always last-release + 1, so BASE-N is a
|
||||
# prerelease of the NEXT version and semver-sorts ABOVE the last stable
|
||||
# (0.3.6-N > 0.3.5) — preview users naturally upgrade past stable, and
|
||||
# the Windows MSI ProductVersion (which strips the prerelease → 0.3.6)
|
||||
# is also correctly above the last stable.
|
||||
# stamps the artifacts + latest.json. `0.3.0-preview.N` is a prerelease of
|
||||
# the current target, so previews converge to stable when 0.3.0 ships
|
||||
# (0.3.0 > 0.3.0-preview.N). NOTE: the Windows MSI ProductVersion strips
|
||||
# the prerelease (→ 0.3.0), a wrinkle to verify for win preview→preview
|
||||
# upgrades; mac/linux replace the bundle wholesale and are unaffected.
|
||||
- name: Stamp preview version
|
||||
if: needs.preview-gate.outputs.is_preview == 'true'
|
||||
if: github.event_name == 'workflow_dispatch' && inputs.publish_preview
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# package.json is the single source of truth; tauri.conf.json reads its
|
||||
# version from it ("version": "../package.json"), so stamping
|
||||
# package.json restamps the whole bundle.
|
||||
CONF=frontend/package.json
|
||||
CONF=frontend/src-tauri/tauri.conf.json
|
||||
BASE=$(jq -r .version "$CONF")
|
||||
# MSI/WiX requires the semver pre-release identifier to be numeric-only
|
||||
# (and <= 65535). "preview.N" hard-fails the Windows bundler, so the
|
||||
@@ -498,12 +431,6 @@ jobs:
|
||||
# only on the opt-in stable path, leaving them ABSENT (not "") on
|
||||
# preview/unsigned paths so Tauri's bundler skips cert import. A static
|
||||
# env: here would always set them to "" and break the mac build.
|
||||
# Unsigned paths still get a VALID ad-hoc seal from tauri.conf.json
|
||||
# (bundle.macOS.signingIdentity = "-"), so a downloaded build shows the
|
||||
# GUI-bypassable "unidentified developer" prompt (right-click → Open /
|
||||
# Settings → "Open Anyway") instead of the un-bypassable "damaged"
|
||||
# error. On the signed path APPLE_SIGNING_IDENTITY (env) overrides the
|
||||
# "-" default; once notarized, Gatekeeper accepts it with no prompt.
|
||||
# GH runners disable FUSE, so linuxdeploy's AppImage can't mount
|
||||
# itself at bundle time. This env tells linuxdeploy to extract-and-run
|
||||
# instead, which works without FUSE.
|
||||
@@ -515,11 +442,11 @@ jobs:
|
||||
# rolling `preview` prerelease for the updater's Preview channel.
|
||||
# Every other invocation — crucially the `v*` tag-push stable release
|
||||
# — evaluates these expressions to exactly their prior values.
|
||||
tagName: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'preview' || github.ref_name }}
|
||||
releaseName: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'OmniVoice Studio (Preview)' || format('OmniVoice Studio {0}', github.ref_name) }}
|
||||
tagName: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'preview' || github.ref_name }}
|
||||
releaseName: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'OmniVoice Studio (Preview)' || format('OmniVoice Studio {0}', github.ref_name) }}
|
||||
releaseBody: ${{ steps.changelog.outputs.body }}
|
||||
releaseDraft: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'false' || (inputs.draft || 'true') }}
|
||||
prerelease: ${{ needs.preview-gate.outputs.is_preview == 'true' }}
|
||||
releaseDraft: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'false' || (inputs.draft || 'true') }}
|
||||
prerelease: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) || false }}
|
||||
updaterJsonPreferNsis: false
|
||||
includeUpdaterJson: true
|
||||
|
||||
@@ -554,35 +481,6 @@ jobs:
|
||||
echo "OK — bundle has shell + uv + backend resources"
|
||||
hdiutil detach "$MOUNT" || true
|
||||
|
||||
# ── Signing / Gatekeeper / notarization verification ──────────────
|
||||
# Runs codesign --verify, spctl (Gatekeeper), nested-binary, and
|
||||
# stapler checks against the built .app (see docs/macos-signing-verification.md).
|
||||
# STRICT (--require-signed) only on the opt-in signed stable path — same
|
||||
# condition as "Configure Apple signing" above — so a failed or missing
|
||||
# signature/notarization FAILS the job and STOPS the release instead of
|
||||
# publishing an unsigned artifact. On every other (unsigned dev/preview)
|
||||
# path it runs report-only and never breaks the build.
|
||||
- name: Verify macOS signing
|
||||
if: runner.os == 'macOS'
|
||||
shell: bash
|
||||
env:
|
||||
STRICT: ${{ (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && vars.MACOS_SIGNING_ENABLED == 'true') && '1' || '0' }}
|
||||
APPLE_ID: ${{ secrets.APPLE_ID }}
|
||||
APPLE_PASSWORD: ${{ secrets.APPLE_PASSWORD }}
|
||||
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
|
||||
run: |
|
||||
set -uo pipefail
|
||||
APP=$(find "frontend/src-tauri/target/${{ matrix.rust_target }}/release/bundle/macos" -maxdepth 1 -name '*.app' | head -1)
|
||||
[ -n "$APP" ] || { echo "FAIL — no .app found to verify"; exit 1; }
|
||||
MODE=""
|
||||
if [ "$STRICT" = "1" ]; then
|
||||
MODE="--require-signed"
|
||||
echo "Signed stable release → STRICT verification (release stops on failure)."
|
||||
else
|
||||
echo "Unsigned dev/preview path → report-only verification."
|
||||
fi
|
||||
bash scripts/verify-macos-signing.sh "$APP" $MODE
|
||||
|
||||
- name: Installer smoke (Windows)
|
||||
if: runner.os == 'Windows'
|
||||
timeout-minutes: 5
|
||||
@@ -705,8 +603,8 @@ jobs:
|
||||
# preview-only — stable `v*` releases keep their CHANGELOG section + the
|
||||
# appended checksums.
|
||||
preview-notes:
|
||||
needs: [build, preview-gate]
|
||||
if: needs.preview-gate.outputs.is_preview == 'true'
|
||||
needs: build
|
||||
if: github.event_name == 'workflow_dispatch' && inputs.publish_preview
|
||||
runs-on: ubuntu-22.04
|
||||
permissions:
|
||||
contents: write
|
||||
@@ -737,81 +635,5 @@ jobs:
|
||||
echo ""
|
||||
echo "$CONTRIB"
|
||||
} > /tmp/preview-notes.md
|
||||
# --prerelease re-asserts the flag every run: a non-prerelease
|
||||
# `preview` release is eligible to become GitHub's "Latest", which is
|
||||
# the exact URL the Stable updater channel reads — so it must never
|
||||
# flip off.
|
||||
gh release edit preview --repo "$REPO" --prerelease --notes-file /tmp/preview-notes.md
|
||||
gh release edit preview --repo "$REPO" --notes-file /tmp/preview-notes.md
|
||||
echo "Applied auto-generated release notes + contributors to the preview release."
|
||||
|
||||
- name: Verify preview updater manifest (prerelease + platform parity)
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
REPO: ${{ github.repository }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# The preview release must stay a prerelease (or it can hijack the
|
||||
# Stable channel's releases/latest endpoint), and its updater manifest
|
||||
# must cover every platform stable does (else those users — e.g. Intel
|
||||
# Mac — silently get no preview updates).
|
||||
is_pre=$(gh release view preview --repo "$REPO" --json isPrerelease -q .isPrerelease)
|
||||
test "$is_pre" = "true" || { echo "::error::preview release is not a prerelease"; exit 1; }
|
||||
curl -fsSL "https://github.com/$REPO/releases/download/preview/latest.json" -o /tmp/preview-latest.json
|
||||
curl -fsSL "https://github.com/$REPO/releases/latest/download/latest.json" -o /tmp/stable-latest.json
|
||||
python3 - <<'PY'
|
||||
import json, re
|
||||
prev = json.load(open("/tmp/preview-latest.json"))
|
||||
stab = json.load(open("/tmp/stable-latest.json"))
|
||||
v = prev.get("version", "")
|
||||
assert re.fullmatch(r"\d+\.\d+\.\d+-\d+", v), f"preview version not X.Y.Z-N: {v!r}"
|
||||
pk, sk = set(prev.get("platforms", {})), set(stab.get("platforms", {}))
|
||||
missing = sk - pk
|
||||
assert not missing, f"preview manifest missing platforms vs stable: {sorted(missing)}"
|
||||
print(f"preview manifest OK: {v} platforms={sorted(pk)}")
|
||||
PY
|
||||
|
||||
# ── Post-release version bump (versioning hard rule, owner-set 2026-06-11) ──
|
||||
# main is always last-release + 1 patch. The moment a stable v* tag is
|
||||
# released, bump the three version sources on main to the next patch so every
|
||||
# PR and preview build identifies as the next version. Pushes directly to
|
||||
# main with the workflow token (a metadata-only commit; CI runs on PRs).
|
||||
version-bump:
|
||||
if: github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-')
|
||||
runs-on: ubuntu-22.04
|
||||
permissions:
|
||||
contents: write
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
ref: main
|
||||
fetch-depth: 0
|
||||
- name: Bump main to released version + 1 patch
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
RELEASED="${GITHUB_REF_NAME#v}"
|
||||
IFS=. read -r MAJ MIN PAT <<< "$RELEASED"
|
||||
NEXT="$MAJ.$MIN.$((PAT + 1))"
|
||||
# frontend/package.json is the SINGLE SOURCE OF TRUTH: vite injects
|
||||
# __APP_VERSION__ from it, and tauri.conf.json reads its bundle version
|
||||
# from it ("version": "../package.json"). Read CURRENT from it.
|
||||
CURRENT=$(jq -r .version frontend/package.json)
|
||||
if [ "$(printf '%s\n' "$NEXT" "$CURRENT" | sort -V | tail -1)" = "$CURRENT" ] && [ "$NEXT" != "$CURRENT" ]; then
|
||||
echo "main is already at $CURRENT (>= $NEXT) — nothing to bump"; exit 0
|
||||
fi
|
||||
# Bump the canonical (package.json), set absolutely so any prior drift
|
||||
# self-heals. tauri.conf.json needs no edit — it derives from this.
|
||||
tmp=$(mktemp)
|
||||
jq --arg v "$NEXT" '.version = $v' frontend/package.json > "$tmp"
|
||||
mv "$tmp" frontend/package.json
|
||||
# The remaining files are CI-guarded mirrors (cargo/uv require a
|
||||
# literal; the version.py literal is the frozen-backend last resort) —
|
||||
# bump them in lockstep with the canonical.
|
||||
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" frontend/src-tauri/Cargo.toml
|
||||
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" pyproject.toml
|
||||
sed -i "0,/_FALLBACK_VERSION = \"$CURRENT\"/s//_FALLBACK_VERSION = \"$NEXT\"/" backend/core/version.py
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
git add frontend/package.json frontend/src-tauri/Cargo.toml pyproject.toml backend/core/version.py
|
||||
git commit -m "chore(version): main -> $NEXT after $GITHUB_REF_NAME release"
|
||||
git push origin main
|
||||
|
||||
@@ -32,13 +32,9 @@ env:
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# PR branches: a new push cancels the superseded scan (no wasted runners).
|
||||
# main: every commit keeps its own group, so nothing is cancelled — a merge
|
||||
# train used to leave a permanent red ✗ ("cancelled") on every intermediate
|
||||
# commit in the history view even though nothing failed.
|
||||
concurrency:
|
||||
group: security-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || 'branch' }}
|
||||
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
|
||||
group: security-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
# ── Secret scanning (gating) ─────────────────────────────────────────────
|
||||
|
||||
@@ -1,17 +1,10 @@
|
||||
# SPIKE-02: Adopt `ModelsLab/omnivoice-singing` as singing variant of the existing engine
|
||||
|
||||
**Status:** ⚠️ **SUPERSEDED (2026-06-14)** by [`specs/006-dubbing-singing-mode/`](../../specs/006-dubbing-singing-mode/spec.md)
|
||||
**Date:** 2026-05-18 (superseded 2026-06-14)
|
||||
**Status:** Proposed (research-supported) — awaiting Phase 2 SubprocessBackend merge
|
||||
**Date:** 2026-05-18
|
||||
**Decision-makers:** [maintainer]
|
||||
**Related:** ROADMAP Phase 4; REQUIREMENTS SING-01..05; `.planning/phases/04-adaptive-specialty-engines-spike-first/04-RESEARCH.md`
|
||||
|
||||
> **Superseded:** This chose `ModelsLab/omnivoice-singing` for singing, but that
|
||||
> model has **no melody (F0/MIDI) conditioning** — it sings its own melody and
|
||||
> cannot follow the *source song* a dub must preserve. SoulX-Singer (arXiv
|
||||
> 2602.07803, published after this decision) provides F0/MIDI conditioning and is
|
||||
> selected in plan-06. This ADR stays valid only if reframed as an
|
||||
> expressive-TTS styling toggle, not melody-matched dubbing.
|
||||
|
||||
## Context
|
||||
|
||||
`ModelsLab/omnivoice-singing` (HuggingFace, 1,053 downloads/month, verified 2026-05-18) is a finetune of `k2-fsa/OmniVoice` — same Apache-2.0 license, same Qwen3-0.6B backbone, same Higgs Audio v2 codec at 24 kHz mono, same `omnivoice` PyPI library (0.1.5, 2026-04-28) already shipping in OmniVoice Studio v0.2.7. Trained on additional singing + emotion-tagged data and activated by a `[singing]` text control tag at generation time.
|
||||
|
||||
@@ -1,396 +0,0 @@
|
||||
---
|
||||
phase: 260613-fdl
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- pyproject.toml
|
||||
- backend/api/routers/setup/download.py
|
||||
- backend/api/routers/setup/models.py
|
||||
- backend/utils/hf_progress.py
|
||||
- backend/utils/download_aggregator.py # NEW
|
||||
- backend/services/segmented_download.py # NEW
|
||||
- backend/api/routers/system.py
|
||||
- frontend/src/pages/Settings.jsx
|
||||
- frontend/src/api/setup.ts
|
||||
- docs/downloading-models.md # NEW (docs-sync rule)
|
||||
- tests/backend/setup/test_download_preflight.py # NEW
|
||||
- tests/backend/services/test_segmented_download.py # NEW
|
||||
autonomous: true
|
||||
requirements:
|
||||
# ── Wave 0 — Spike / gate ───────────────────────────────────────────────────
|
||||
- FDL-00 # Classify all catalog repos Xet-backed vs legacy-LFS; the result sizes Wave 3
|
||||
# ── Wave 1 — Maximize + guarantee the Xet fast path (default, no new deps) ───
|
||||
- FDL-01 # Explicitly pin huggingface_hub>=1.7 + hf-xet in pyproject (today transitive/unpinned)
|
||||
- FDL-02 # Drive snapshot_download with explicit max_workers + tqdm_class + endpoint (not implicit monkeypatch)
|
||||
- FDL-03 # /system/info reports fast_download {xet_enabled, xet_version, high_performance}; logged at startup
|
||||
- FDL-04 # Opt-in HF_XET_HIGH_PERFORMANCE + HDD sequential-write toggles via prefs (env wins)
|
||||
# ── Wave 2 — Accurate downloaded/remaining + speed (the user-visible win) ────
|
||||
- FDL-05 # dry_run preflight -> emit install_plan {total_bytes, cached_bytes, to_download_bytes, n_files, n_cached}
|
||||
- FDL-06 # Backend aggregate tracker -> single 'aggregate' event {bytes_done, total_bytes, rate, eta, files_done/total}
|
||||
- FDL-07 # Frontend overall bar: speed + downloaded/remaining + ETA from aggregate; per-file detail collapsible; cached-skip shown
|
||||
# ── Wave 3 — Opt-in IDM-style accelerator for legacy-LFS repos ───────────────
|
||||
- FDL-08 # Custom httpx segmented downloader: parallel Range GETs, resume, auth-safe redirect, etag/sha verify, cancel (default OFF)
|
||||
- FDL-09 # Dispatch: accelerator ON + repo is LFS (not Xet) -> segmented path; else xet. Same aggregate progress + weight validation
|
||||
# ── Wave 4 — Opt-in mirror path + docs ───────────────────────────────────────
|
||||
- FDL-10 # Opt-in HF_ENDPOINT mirror setting (prefs); documented as classic-LFS fallback (no Xet); pairs with FDL-08
|
||||
- FDL-11 # Cancel-in-flight endpoint + cooldown interplay (composes with MM2-06 bounded cooldowns)
|
||||
- FDL-12 # docs/downloading-models.md (speed, fast-download status, HDD/high-perf toggles, mirror/restricted-network) + README pointer
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "Xet is the default download backend and is provably engaged: /system/info reports fast_download.xet_enabled=true with the hf_xet version, and a Xet-backed repo downloads via parallel chunk range-gets (not single-stream LFS)."
|
||||
- "Before any bytes flow, the UI shows an accurate denominator: total bytes to download, bytes already cached (skipped), and file count — sourced from snapshot_download(dry_run=True), not guessed from the first tqdm bar."
|
||||
- "During a download the UI shows ONE overall progress bar with instantaneous speed (sampled over a window, not a single file's rate), bytes downloaded / bytes remaining, and ETA — accurate even while Xet fetches many chunks/files in parallel."
|
||||
- "hf_transfer is NOT used or enabled anywhere (deprecated, breaks progress); the fast path is Xet only."
|
||||
- "The custom segmented downloader is OPT-IN (default off), only engages for non-Xet/legacy-LFS repos, never forwards the HF Authorization header to the redirected CDN host, verifies the downloaded file against its expected size/etag before marking complete, resumes a partial .part file, and can be cancelled mid-flight."
|
||||
- "Default download behavior is identical on macOS, Windows, Linux (Xet path, pure-Python). Every accelerator/mirror/high-perf knob is behind an explicit opt-in (Settings toggle or env var) per the cross-platform-parity strict rule — no bundled per-OS binary, no platform-divergent default."
|
||||
- "No new on-disk model-state format; existing HF cache layout and already-installed models are untouched; the segmented downloader writes into the same HF cache blob/snapshot structure (or hands off to it) so a model it fetches is indistinguishable from one snapshot_download fetched."
|
||||
- "uv run pytest tests/backend/setup/test_download_preflight.py tests/backend/services/test_segmented_download.py passes; existing download/install tests stay green."
|
||||
- "pyproject pins huggingface_hub>=1.7 and hf-xet explicitly; uv.lock resolves with single versions (uv tree shows no duplicate huggingface_hub)."
|
||||
artifacts:
|
||||
- path: "backend/utils/download_aggregator.py"
|
||||
provides: "Per-repo byte aggregator: sums bytes across parallel files/chunks, samples rate over a window, emits one 'aggregate' event"
|
||||
contains: "class DownloadAggregator AND def snapshot"
|
||||
- path: "backend/services/segmented_download.py"
|
||||
provides: "Opt-in multi-connection Range downloader for legacy-LFS repos (auth-safe, resume, verify, cancel)"
|
||||
contains: "async def segmented_download AND Range"
|
||||
- path: "backend/api/routers/setup/download.py"
|
||||
provides: "Driven snapshot_download (max_workers+tqdm_class+endpoint), dry_run preflight, dispatch to segmented path, cancel endpoint"
|
||||
contains: "dry_run AND tqdm_class"
|
||||
- path: "docs/downloading-models.md"
|
||||
provides: "User docs for download speed, fast-download status, HDD/high-perf toggles, mirror/restricted-network"
|
||||
contains: "Xet"
|
||||
key_links:
|
||||
- from: "install_model (download.py:122)"
|
||||
to: "snapshot_download(dry_run=True) preflight"
|
||||
via: "compute total/cached/remaining before the real download; emit 'install_plan'"
|
||||
pattern: "dry_run\\s*=\\s*True"
|
||||
- from: "snapshot_download / segmented_download byte updates"
|
||||
to: "DownloadAggregator -> single 'aggregate' SSE event"
|
||||
via: "tqdm_class forwards bytes into the aggregator; segmented path calls aggregator.add() directly"
|
||||
pattern: "aggregate"
|
||||
- from: "dispatch in install_model"
|
||||
to: "segmented_download vs snapshot_download"
|
||||
via: "prefs accelerator toggle AND repo-is-LFS classification (FDL-00 helper)"
|
||||
pattern: "segmented_download"
|
||||
- from: "system_info (system.py:245)"
|
||||
to: "fast_download status block"
|
||||
via: "probe hf_xet import + version + HF_XET_HIGH_PERFORMANCE"
|
||||
pattern: "fast_download"
|
||||
|
||||
---
|
||||
|
||||
<objective>
|
||||
Make model downloads as fast as possible AND show accurate speed / downloaded / remaining / ETA.
|
||||
|
||||
**Framing (validated by research — see 260613-fdl-RESEARCH below):** HuggingFace's **hf-xet** backend ALREADY implements the "IDM/uGet technique" — content-defined chunking, parallel byte-range fetches with adaptive concurrency, dedup, and automatic resume — and does it auth-safely. It ships by default in modern `huggingface_hub` and `hf_xet` is already installed here (huggingface_hub 1.7.2). HF closed the multi-connection-downloader feature request as "solved by Xet." So we do NOT build a custom segmented downloader as the default path; that would be redundant and would violate the cross-platform-parity rule.
|
||||
|
||||
What's actually missing:
|
||||
1. **We don't drive Xet well.** `install_model` calls `snapshot_download(**dl_kwargs)` with no `max_workers`, no `tqdm_class`, no `dry_run`, and no explicit dependency pin — progress rides on a global tqdm monkeypatch.
|
||||
2. **No pre-flight total**, so "downloaded/remaining" has no denominator until files appear, and aggregate speed is summed frontend-side from per-file events (inaccurate under parallel fetch).
|
||||
3. **Legacy non-Xet (LFS) repos get zero intra-file parallelism** — this is the one place a real IDM-style multi-connection fetch still helps, so we add it as an OPT-IN accelerator.
|
||||
|
||||
Five waves, in order (each independently shippable, continuous-to-main per v0.3.0 cadence):
|
||||
- **Wave 0 — Spike/gate (FDL-00):** classify every catalog repo Xet vs LFS. Sizes Wave 3's value; if ~all repos are Xet-backed, Wave 3 is low-priority polish.
|
||||
- **Wave 1 — Maximize + guarantee Xet (FDL-01..04):** pin deps, drive snapshot_download explicitly, surface fast-download status, opt-in high-perf/HDD knobs. No new deps, all platforms.
|
||||
- **Wave 2 — Accurate progress (FDL-05..07):** dry_run preflight + backend aggregate tracker + overall UI bar (speed/remaining/ETA). The biggest user-visible win.
|
||||
- **Wave 3 — Opt-in segmented accelerator (FDL-08..09):** custom httpx Range downloader for LFS repos. Default OFF, opt-in toggle.
|
||||
- **Wave 4 — Mirror path + docs (FDL-10..12):** opt-in HF_ENDPOINT, cancel endpoint, docs-sync.
|
||||
|
||||
Out of scope / explicitly rejected (call out, do NOT do):
|
||||
- **hf_transfer / HF_HUB_ENABLE_HF_TRANSFER** — deprecated, breaks progress callbacks. Never enable.
|
||||
- **Bundling aria2c** — per-OS GPLv2 binary + parity burden; the custom httpx path covers the same need without a binary.
|
||||
- **Making the segmented downloader the default** — redundant vs Xet, violates parity rule. Always opt-in.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/STATE.md
|
||||
@./CLAUDE.md
|
||||
@.planning/quick/260613-fdl-fast-model-downloads/260613-fdl-RESEARCH.md
|
||||
|
||||
# Files under edit (read before editing)
|
||||
@backend/api/routers/setup/download.py
|
||||
@backend/api/routers/setup/models.py
|
||||
@backend/utils/hf_progress.py
|
||||
@backend/api/routers/system.py
|
||||
@frontend/src/pages/Settings.jsx
|
||||
|
||||
# Reference only — patterns, do NOT modify
|
||||
@backend/core/prefs.py
|
||||
@frontend/src/api/setup.ts
|
||||
@frontend/src/api/hooks.ts
|
||||
|
||||
<interfaces>
|
||||
<!-- Verified during planning against the live env (huggingface_hub 1.7.2, hf_xet installed). -->
|
||||
|
||||
huggingface_hub 1.7.2 snapshot_download params (confirmed via inspect):
|
||||
repo_id, repo_type, revision, cache_dir, local_dir, library_name, library_version,
|
||||
user_agent, etag_timeout, force_download, token, local_files_only,
|
||||
allow_patterns, ignore_patterns, max_workers, tqdm_class, headers, endpoint, dry_run
|
||||
- dry_run=True -> returns per-file info incl. size + cached/not-cached (use for FDL-05 preflight).
|
||||
- tqdm_class=<cls> -> drives the AGGREGATE bar; Xet feeds bytes into it (this is the xet-aware progress hook).
|
||||
- max_workers -> parallel FILES (default 8); orthogonal to Xet intra-file chunk parallelism.
|
||||
- endpoint -> per-call HF endpoint override (FDL-10 mirror, instead of process-wide HF_ENDPOINT).
|
||||
|
||||
backend/utils/hf_progress.py (existing):
|
||||
- Monkeypatches huggingface_hub.utils.tqdm.tqdm -> TrackedTqdm (install() at startup).
|
||||
- register_listener/unregister_listener; emit(event); current_repo_id contextvar stamps events.
|
||||
- TrackedTqdm.update()/display() emit per-file {filename, downloaded, total, pct, rate, phase} throttled ~0.3s.
|
||||
- GAP: per-file only, no aggregate, no preflight total. Wave 2 adds the aggregator on top (keep TrackedTqdm; feed it).
|
||||
|
||||
backend/api/routers/setup/download.py (existing):
|
||||
- install_model (line 122): snapshot_download(**dl_kwargs) inside asyncio.to_thread; 5-retry backoff; heartbeat;
|
||||
_validate_snapshot_has_weights (line 55); _install_cooldowns (line 27, see MM2-06 for bounding).
|
||||
- SSE feed: GET /setup/download-stream (line 80) forwards hf_progress events.
|
||||
|
||||
backend/core/prefs.py:
|
||||
- resolve(key, *, env=None, default=None) (line 75) — env wins, then store, then default. Use for all new toggles.
|
||||
|
||||
Xet env knobs (research): HF_XET_HIGH_PERFORMANCE=1 (opt-in max throughput; needs RAM/bandwidth),
|
||||
HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1 (HDD), HF_XET_NUM_CONCURRENT_RANGE_GETS (default 16),
|
||||
HF_XET_DATA_PROGRESS_UPDATE_INTERVAL (200ms). hf_xet is 64-bit only.
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<!-- ════════════ WAVE 0 — SPIKE / GATE ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 0 (FDL-00): Classify catalog repos Xet vs LFS</name>
|
||||
<files>.planning/quick/260613-fdl-fast-model-downloads/260613-fdl-SPIKE.md</files>
|
||||
<action>
|
||||
For every repo in backend/config/models.yaml (25 entries), determine whether it's Xet-backed or legacy Git-LFS. Use huggingface_hub: `HfApi().repo_info(repo_id, files_metadata=True)` and inspect each LFS blob for xet info, OR call the model-info endpoint and check the `xetEnabled`/blob `xet` field. For gated/unavailable repos, record "unknown (gated/offline)".
|
||||
Write 260613-fdl-SPIKE.md: a table repo_id | role | backend (xet|lfs|unknown) | size, plus a one-line GO/LOW-PRIORITY verdict for Wave 3:
|
||||
- If the majority of *user-facing default* models (OmniVoice TTS, the default ASR) are Xet-backed -> Wave 3 is LOW priority (xet already fast); still build it for the LFS long tail.
|
||||
- If many defaults are still LFS -> Wave 3 is HIGH priority.
|
||||
This is read-only network classification — do not download anything (use repo_info, not snapshot_download).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>test -f .planning/quick/260613-fast-model-downloads/260613-fdl-SPIKE.md || test -f .planning/quick/260613-fdl-fast-model-downloads/260613-fdl-SPIKE.md && echo "spike written"</automated>
|
||||
</verify>
|
||||
<done>SPIKE.md lists every catalog repo with its storage backend and a GO/LOW-PRIORITY verdict for Wave 3.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 1 — MAXIMIZE + GUARANTEE THE XET FAST PATH ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1 (FDL-01): Pin huggingface_hub + hf-xet explicitly</name>
|
||||
<files>pyproject.toml</files>
|
||||
<action>
|
||||
Today huggingface_hub arrives transitively (1.7.2) and hf_xet is present but unpinned. Add explicit runtime pins so the fast path can never silently disappear on a resolve:
|
||||
- huggingface_hub>=1.7 (keep compatible with transformers>=5.3.0 already in deps)
|
||||
- hf-xet>=1.1 (the Xet backend; 64-bit only — fine for all OmniVoice targets)
|
||||
Do NOT add hf_transfer. Run `uv sync` then `uv tree huggingface_hub` to confirm a single resolved version (no duplicate). If a transitive constraint conflicts, prefer the higher version and note it in the SUMMARY.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "huggingface_hub\|hf-xet\|hf_xet\|hf-transfer\|hf_transfer" pyproject.toml</automated>
|
||||
<automated>uv run python -c "import huggingface_hub,hf_xet; print('hub',huggingface_hub.__version__,'xet ok')"</automated>
|
||||
</verify>
|
||||
<done>pyproject pins huggingface_hub>=1.7 and hf-xet; no hf_transfer; uv resolves cleanly with one huggingface_hub.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2 (FDL-02): Drive snapshot_download explicitly</name>
|
||||
<files>backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
In install_model's _do() (line ~148), build dl_kwargs with explicit, intentional args instead of the bare call:
|
||||
- tqdm_class=<the TrackedTqdm class> so progress is deterministic and xet-aware rather than relying solely on the global monkeypatch. Expose TrackedTqdm from hf_progress (add a getter, e.g. hf_progress.tracked_tqdm_class()).
|
||||
- max_workers: keep default 8 (don't crank — xet does intra-file parallelism; high max_workers multiplies buffer pressure). Make it prefs-overridable: prefs.resolve("download_max_workers", env="OMNIVOICE_DOWNLOAD_MAX_WORKERS", default=8).
|
||||
- endpoint=prefs.resolve("hf_endpoint", env="HF_ENDPOINT", default=None) — wires FDL-10 mirror without process-wide env.
|
||||
- Keep the existing 5-retry backoff, heartbeat, and _validate_snapshot_has_weights.
|
||||
Do not remove the global monkeypatch (other libs — transformers/mlx_whisper — still rely on it); this task just makes the install path drive its own tqdm_class explicitly.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "tqdm_class\|max_workers\|endpoint" backend/api/routers/setup/download.py</automated>
|
||||
<automated>uv run pytest tests/ -k "download or install" -q 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<done>install_model drives snapshot_download with explicit tqdm_class + max_workers + endpoint; retry/validate intact; tests green.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3 (FDL-03, FDL-04): fast_download status + opt-in xet knobs</name>
|
||||
<files>backend/api/routers/system.py, backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
- FDL-03: add a fast_download block to GET /system/info (system.py:245): {xet_enabled: bool, xet_version: str|None, high_performance: bool}. Probe by importing hf_xet (xet_enabled), reading its version, and reading the HF_XET_HIGH_PERFORMANCE env/pref. Must never throw (system_info is called on every Settings load). Log the same line once at startup ("fast download: Xet on (hf_xet X.Y), high_perf=...").
|
||||
- FDL-04: opt-in knobs via prefs, applied at process/download setup (env wins):
|
||||
high_performance = prefs.resolve("xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False)
|
||||
hdd_sequential = prefs.resolve("xet_hdd_sequential_write", env="HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY", default=False)
|
||||
When set, export the corresponding HF_XET_* env before the snapshot/segmented download runs. Both default OFF (high-perf can hurt low-RAM machines — surface that as a tooltip in Wave 2 UI).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>curl -s http://127.0.0.1:3900/system/info | python3 -c "import json,sys; print(json.load(sys.stdin).get('fast_download'))" 2>/dev/null || grep -n "fast_download" backend/api/routers/system.py</automated>
|
||||
</verify>
|
||||
<done>/system/info reports fast_download truthfully; high-perf + HDD knobs resolve via prefs with env precedence, default off; startup logs Xet status.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 2 — ACCURATE DOWNLOADED/REMAINING + SPEED ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 4 (FDL-05): dry_run preflight -> install_plan event</name>
|
||||
<files>backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
Before the real download in install_model, run snapshot_download(repo_id, dry_run=True, endpoint=...) on the worker thread. From the returned per-file info compute: total_bytes, cached_bytes (files already present), to_download_bytes, n_files, n_cached. Emit a new phase event:
|
||||
{repo_id, phase:"install_plan", total_bytes, cached_bytes, to_download_bytes, n_files, n_cached}
|
||||
This gives the UI an accurate denominator and a "M GB already cached, N GB to download" line BEFORE bytes flow. Wrap dry_run in try/except — if it fails (older/gated repo), emit install_plan with totals=None and fall back to today's behavior (denominator fills in as files appear). dry_run must respect the 'resolving' heartbeat (it can take a couple seconds).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "dry_run\|install_plan\|to_download_bytes" backend/api/routers/setup/download.py</automated>
|
||||
<automated>uv run pytest tests/backend/setup/test_download_preflight.py -q 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<done>An install emits install_plan with accurate total/cached/remaining before download; dry_run failure degrades gracefully to old behavior.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 5 (FDL-06): Backend aggregate progress tracker</name>
|
||||
<files>backend/utils/download_aggregator.py, backend/utils/hf_progress.py</files>
|
||||
<action>
|
||||
New backend/utils/download_aggregator.py: a per-repo DownloadAggregator that owns the TRUTH for overall progress, so the frontend stops summing potentially-misrouted per-file events.
|
||||
- Seeded by the install_plan totals (total_bytes, n_files).
|
||||
- add(filename, bytes_delta) / set_file(filename, downloaded, total): track bytes per file; bytes_done = sum.
|
||||
- Rate: sampled over a sliding window (e.g. last ~5-10s of (t, bytes_done) samples), not a single tqdm bar's rate. eta = remaining / rate.
|
||||
- snapshot() -> {repo_id, bytes_done, total_bytes, rate, eta_seconds, files_done, files_total, phase}.
|
||||
- Emits one throttled (~0.3-0.5s) phase:"aggregate" event via hf_progress.emit().
|
||||
Wire it: hf_progress's TrackedTqdm._emit_progress already has per-file (filename, downloaded, total) — also feed those into the active repo's aggregator (look up by current_repo_id). The segmented downloader (Wave 3) calls aggregator.add() directly. Keep the per-file events too (UI detail view) — aggregate is additive, not a replacement.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run python -c "from utils.download_aggregator import DownloadAggregator as A; a=A('r',total_bytes=100,files_total=2); a.set_file('f1',50,50); a.set_file('f2',25,50); s=a.snapshot(); print(s['bytes_done'], s['total_bytes'])"</automated>
|
||||
</verify>
|
||||
<done>DownloadAggregator sums bytes across parallel files, samples rate over a window, emits a single 'aggregate' event; fed by both tqdm and the segmented path.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 6 (FDL-07): Frontend overall progress bar</name>
|
||||
<files>frontend/src/pages/Settings.jsx, frontend/src/api/setup.ts</files>
|
||||
<action>
|
||||
- setup.ts: extend SetupProgressEvent phase union with "install_plan" | "aggregate" and their fields (total_bytes, cached_bytes, to_download_bytes, n_files, n_cached, bytes_done, rate, eta_seconds, files_done, files_total).
|
||||
- Settings.jsx ModelStoreTab: when an aggregate event arrives for a repo, render ONE overall progress row: a bar (bytes_done/total_bytes), instantaneous speed (format rate as MB/s), "X.X GB of Y.Y GB" downloaded/remaining, and ETA (mm:ss from eta_seconds). Seed the denominator from install_plan (show "M GB cached, N GB to download" before bytes flow). Keep the existing per-file rows as a collapsible "details" section instead of the primary display. Show a small "⚡ fast download" badge when /system/info fast_download.xet_enabled is true.
|
||||
- Prefer the backend aggregate's rate/eta over the frontend's own per-file ETA computation (Settings.jsx ~614-631) — replace that local ETA math with the aggregate fields; keep a fallback if no aggregate event has arrived yet.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd frontend && bun run typecheck 2>&1 | tail -15</automated>
|
||||
<automated>grep -n "aggregate\|install_plan\|eta_seconds\|fast download" frontend/src/pages/Settings.jsx frontend/src/api/setup.ts</automated>
|
||||
</verify>
|
||||
<done>UI shows one overall bar with live speed + downloaded/remaining + ETA from the aggregate event; per-file detail collapsible; fast-download badge; typecheck passes.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 3 — OPT-IN IDM-STYLE SEGMENTED ACCELERATOR (LFS REPOS) ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 7 (FDL-08): Custom httpx segmented downloader</name>
|
||||
<files>backend/services/segmented_download.py, tests/backend/services/test_segmented_download.py</files>
|
||||
<action>
|
||||
New backend/services/segmented_download.py — an OPT-IN multi-connection Range downloader for ONE file (the IDM/uGet technique) used only for legacy-LFS repos where Xet gives no intra-file parallelism. httpx is already a dep.
|
||||
Contract (async def segmented_download(url, dest, *, token, expected_size, expected_etag=None, num_connections=8, chunk_aggregator=None, cancel_event=None)):
|
||||
1. HEAD (or GET Range: bytes=0-0) the resolve URL to learn size + Accept-Ranges + the redirect target. If server doesn't honor Range (Accept-Ranges != bytes) -> fall back to a single streamed GET (still works, just not parallel).
|
||||
2. AUTH SAFETY (critical): send Authorization: Bearer <token> ONLY to the huggingface.co host. When the resolve URL 302-redirects to the CDN (cloudfront/etc.), do NOT forward Authorization to the CDN host — the presigned URL already carries auth. Follow redirects manually so you control header propagation per-host.
|
||||
3. Split expected_size into num_connections ranges; download each with Range: bytes=start-end concurrently (asyncio + httpx.AsyncClient). Write to dest+".part" at the right offsets (preallocate, or per-range temp files then concat).
|
||||
4. RESUME: if dest+".part" exists with a sidecar manifest of completed ranges, skip completed ranges.
|
||||
5. CANCEL: check cancel_event between chunks; on cancel, leave the .part for resume and raise CancelledError.
|
||||
6. VERIFY: after assembly, check size == expected_size and (if given) sha256/etag matches; only then atomically rename .part -> dest. On mismatch, raise (caller's retry/validate handles it).
|
||||
7. PROGRESS: call chunk_aggregator.add(filename, bytes_delta) as ranges complete bytes (feeds DownloadAggregator).
|
||||
Tests (use a local mock HTTP server / httpx MockTransport): honors Range + parallel assembly == single-GET bytes; falls back when Accept-Ranges absent; does NOT send Authorization to a different host on redirect; resumes from a partial .part; cancels and leaves resumable state; size/etag mismatch raises.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run pytest tests/backend/services/test_segmented_download.py -q 2>&1 | tail -20</automated>
|
||||
<automated>grep -n "Authorization\|Range\|cancel_event\|expected_size" backend/services/segmented_download.py</automated>
|
||||
</verify>
|
||||
<done>segmented_download fetches a file via parallel ranges, is auth-safe across the CDN redirect, resumes, cancels, and verifies size/etag before commit; all tests pass.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 8 (FDL-09): Dispatch — accelerator for LFS repos only</name>
|
||||
<files>backend/api/routers/setup/download.py, backend/api/routers/setup/models.py</files>
|
||||
<action>
|
||||
- models.py: add a small helper is_xet_backed(repo_id) -> bool|None (reuse FDL-00's classification approach; cache result). Used to decide the path.
|
||||
- download.py install_model dispatch:
|
||||
accelerator_on = prefs.resolve("segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=False)
|
||||
if accelerator_on and is_xet_backed(repo_id) is False:
|
||||
-> resolve each LFS file's URL via hf_hub_url + HfApi file metadata, download via segmented_download into the HF cache layout (or download to a temp dir then place via the cache API so the result is a normal cache entry), feeding the same DownloadAggregator. Run _validate_snapshot_has_weights at the end.
|
||||
else:
|
||||
-> existing snapshot_download path (xet).
|
||||
IMPORTANT: the segmented result MUST land in the same HF cache structure so /models install-state, delete, and is_cached() all keep working (truth: "indistinguishable from snapshot_download"). If matching the blob/snapshot symlink layout is too fiddly, the safe fallback is: segmented-download to a temp file, then hand the bytes to huggingface_hub so it finalizes the cache entry. Document the chosen approach in SUMMARY.
|
||||
Default OFF -> zero behavior change unless the user opts in.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "segmented_downloader\|is_xet_backed\|segmented_download" backend/api/routers/setup/download.py backend/api/routers/setup/models.py</automated>
|
||||
<automated>uv run pytest tests/ -k "download or install or model" -q 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<done>With the toggle ON, LFS repos download via the segmented path into the normal HF cache; Xet repos and the default (toggle OFF) use snapshot_download; install-state/delete unaffected.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 4 — MIRROR PATH + CANCEL + DOCS ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 9 (FDL-10, FDL-11): Mirror opt-in + cancel endpoint</name>
|
||||
<files>backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
- FDL-10: the endpoint= wiring from Task 2 already reads prefs hf_endpoint. Surface it as a setting and document (Task 10) that a mirror routes through the CLASSIC LFS path (no Xet) — so it pairs naturally with the FDL-08 segmented accelerator for speed on mirrors. No process-wide HF_ENDPOINT mutation; per-call endpoint only.
|
||||
- FDL-11: add POST /models/install/cancel {repo_id} that sets the repo's cancel_event (segmented path) and, for the snapshot path, best-effort marks the install cancelled (snapshot_download isn't trivially cancellable mid-file — at minimum stop retries and emit install_cancelled). Compose with MM2-06: on success OR cancel, clear the _install_cooldowns entry so a cancelled download isn't rate-limited. Emit phase:"install_cancelled".
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "install/cancel\|cancel_event\|install_cancelled\|hf_endpoint" backend/api/routers/setup/download.py</automated>
|
||||
</verify>
|
||||
<done>Per-call mirror endpoint wired (opt-in); cancel endpoint stops the segmented path and clears cooldown; emits install_cancelled.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 10 (FDL-12): Docs — downloading-models.md + README pointer</name>
|
||||
<files>docs/downloading-models.md, README.md</files>
|
||||
<action>
|
||||
Per the docs-sync hard rule, document the user-facing surface introduced here:
|
||||
- How fast downloads work (Xet on by default; what the ⚡ badge means; how to check via Settings/system info).
|
||||
- Advanced toggles: high-performance mode (warn: needs RAM/bandwidth, can hurt low-RAM machines), HDD sequential-write, max workers, segmented accelerator (opt-in, for legacy-LFS repos), and the mirror/restricted-network HF_ENDPOINT setting (note: mirror = classic LFS, no Xet; pair with the accelerator).
|
||||
- A short troubleshooting section (slow downloads, stuck at resolving, restricted networks/China).
|
||||
Add a one-line pointer from README.md to docs/downloading-models.md. Do NOT enable any opt-in by default in docs examples.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>test -f docs/downloading-models.md && grep -n "Xet\|HF_ENDPOINT\|high-performance\|segmented" docs/downloading-models.md | head</automated>
|
||||
<automated>grep -n "downloading-models" README.md</automated>
|
||||
</verify>
|
||||
<done>docs/downloading-models.md covers speed, status, all opt-in knobs, mirror/restricted-network, troubleshooting; README links it; no opt-in shown as default.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
Gate per wave; full set before the last PR:
|
||||
1. `uv run pytest tests/backend/setup/test_download_preflight.py tests/backend/services/test_segmented_download.py tests/ -k "download or install or model" -q` — green.
|
||||
2. Live smoke (backend running): an install emits install_plan (accurate total/cached/remaining) THEN aggregate events with rising bytes_done + a non-zero rate + decreasing ETA; on completion bytes_done == total_bytes.
|
||||
3. /system/info reports fast_download.xet_enabled=true with a version.
|
||||
4. Auth-safety unit test proves Authorization is NOT sent to a non-huggingface.co host on redirect.
|
||||
5. Default-off proof: with no opt-in set, an install uses snapshot_download (xet) — `OMNIVOICE_SEGMENTED_DOWNLOAD` unset means the segmented path is never taken.
|
||||
6. `uv tree huggingface_hub` shows one version; no hf_transfer anywhere (`grep -ri hf_transfer backend/` is empty).
|
||||
7. `cd frontend && bun run typecheck` passes.
|
||||
8. Cross-platform parity: the default path (Xet, pure-Python) is identical on all 3 OSes; every accelerator/mirror/high-perf knob is opt-in (Settings/env). No bundled binary added.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- Fast: Xet is pinned, engaged, and driven with explicit args; high-perf/HDD knobs available opt-in; legacy-LFS repos can use the opt-in segmented accelerator for real multi-connection speed.
|
||||
- Accurate: UI shows pre-flight total/cached/remaining, then one overall bar with live speed + downloaded/remaining + ETA sourced from a backend aggregate (not frontend guesswork).
|
||||
- Safe & compatible: no hf_transfer; segmented downloader is opt-in, auth-safe, resumable, verified, cancellable, and lands in the normal HF cache; default behavior identical on all 3 OSes; no new on-disk model state; existing installs untouched.
|
||||
- All listed tests + typecheck pass; docs updated in the same PR (docs-sync rule).
|
||||
</success_criteria>
|
||||
|
||||
<risks>
|
||||
- **Segmented downloader auth leak (FDL-08) — highest risk.** Forwarding the HF Authorization header to the CDN host on redirect would leak the token. Mitigation: manual redirect handling, per-host header allow-list (Authorization only to huggingface.co), and a dedicated unit test asserting no Authorization on the CDN hop. This is a must-have truth, not optional.
|
||||
- **Cache-layout mismatch (FDL-09).** If the segmented path writes files outside the HF cache blob/snapshot structure, /models install-state + delete + is_cached() break. Mitigation: prefer the temp-file-then-hand-to-huggingface_hub finalization approach over hand-rolling the symlink/blob layout; assert is_cached(repo_id) is true after a segmented install in a test.
|
||||
- **dry_run cost/availability (FDL-05).** dry_run adds a metadata round-trip and may not exist for gated/older repos. Mitigation: try/except -> totals=None fallback to current fill-in-as-you-go behavior; keep the resolving heartbeat so the UI isn't blank during preflight.
|
||||
- **Aggregate vs per-file double-count (FDL-06).** Feeding both tqdm per-file events and the aggregator risks the UI showing two competing numbers. Mitigation: aggregate is the single source of truth for the overall bar; per-file events only drive the collapsible detail view; the frontend's old per-file ETA math is removed (Task 6).
|
||||
- **High-performance mode hurting low-RAM machines (FDL-04).** HF_XET_HIGH_PERFORMANCE can need ~tens of GB RAM. Mitigation: default OFF, opt-in only, tooltip warning in the UI.
|
||||
- **Mirror + Xet confusion (FDL-10).** Users may expect Xet speed through a mirror; mirrors fall back to classic LFS. Mitigation: document explicitly; that's exactly why the segmented accelerator pairs with the mirror path.
|
||||
- **Scope: do not let the segmented path become default.** It's opt-in for LFS repos only. Xet stays the default; making it default would regress dedup + violate the parity rule.
|
||||
</risks>
|
||||
|
||||
<output>
|
||||
Write 260613-fdl-SPIKE.md (Task 0) and 260613-fdl-SUMMARY.md when done. SUMMARY must record: the Xet-vs-LFS catalog breakdown and how it changed Wave 3 priority; the cache-finalization approach chosen for the segmented path (and the is_cached-after-segmented test result); the exact new SSE event shapes (install_plan, aggregate); which opt-in prefs keys + env vars were added; and the auth-safety test output. Note any "use judgment" decision an executor made.
|
||||
|
||||
Docs-sync (CLAUDE.md hard rule): docs/downloading-models.md + README pointer ship in the SAME PR as the user-facing toggles (Task 10). If the Settings UI gains the new toggles, the docs describing them land together.
|
||||
</output>
|
||||
@@ -1,43 +0,0 @@
|
||||
# RESEARCH — Fast HuggingFace model downloads (2026)
|
||||
|
||||
**Date:** 2026-06-13 · **For:** 260613-fdl-PLAN.md
|
||||
|
||||
## Bottom line
|
||||
As of mid-2026 the fast path is **hf-xet, on by default** in modern `huggingface_hub`. Xet is itself a chunk-level, content-defined, massively-parallel downloader with adaptive concurrency — it **is** the "IDM/uGet-style segmented download," done for you and dedup-aware. `hf_transfer` is **deprecated**. Rolling your own segmented downloader or bridging to aria2 is **not worth it as a default**; the only thing we must build is (a) better driving + progress UI and (b) an **opt-in** segmented path for the legacy-LFS long tail (repos Xet doesn't back).
|
||||
|
||||
Installed in this repo: `huggingface_hub 1.7.2`, `hf_xet` present. `snapshot_download` here supports `max_workers`, `tqdm_class`, `endpoint`, `dry_run` (confirmed via inspect).
|
||||
|
||||
## 1. hf-xet — USE (default, no action needed beyond pinning)
|
||||
Content-defined chunks grouped into blocks ("xorbs") in a content-addressable store; download = send file SHA256 → get reconstruction metadata + presigned URLs → fetch needed xorb ranges **in parallel** → reassemble; already-present chunks skipped (dedup). Auto-used by `snapshot_download`/`hf_hub_download` for Xet-backed repos since huggingface_hub 0.32. 2–3× over Git-LFS, up to ~1 GB/s.
|
||||
Knobs (defaults already tuned): `HF_XET_NUM_CONCURRENT_RANGE_GETS` (16), adaptive concurrency ON (max 64), `HF_XET_DATA_MAX_CONCURRENT_FILE_DOWNLOADS` (8), chunk cache disabled by default (better for pure download), `HF_XET_HIGH_PERFORMANCE=1` (opt-in max throughput, needs RAM/bandwidth), `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1` (HDD). **64-bit only.**
|
||||
- https://huggingface.co/docs/huggingface_hub/en/guides/download
|
||||
- https://huggingface.co/docs/hub/en/xet/using-xet-storage
|
||||
- https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables
|
||||
|
||||
## 2. hf_transfer — AVOID (deprecated)
|
||||
`HF_HUB_ENABLE_HF_TRANSFER` flagged deprecated; Xet supersedes it. Historically **broke tqdm progress / had no callbacks** — directly conflicts with the accurate-progress goal. Successor for max throughput is `HF_XET_HIGH_PERFORMANCE=1`.
|
||||
- https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables
|
||||
- https://github.com/huggingface/hf_transfer/issues/63
|
||||
|
||||
## 3. huggingface_hub native concurrency — USE defaults
|
||||
`snapshot_download(max_workers=...)` = parallel FILES (default 8), orthogonal to Xet's intra-file chunk parallelism. For OmniVoice's 1–few-large-file models the win is mostly Xet's intra-file parallelism; don't crank max_workers (multiplies buffer pressure). Resume is automatic via cache + ETag (no `resume_download` flag to manage).
|
||||
- https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
|
||||
|
||||
## 4. Custom IDM-style Range downloader — AVOID as default, BUILD as opt-in for LFS
|
||||
`/resolve/<rev>/<file>` 302-redirects to CDN (Cloudfront) which honors Range + parallel byte-ranges. Catch: follow redirect, **do NOT forward Authorization to the CDN host** (presigned URL carries auth), verify ETag/sha256, auth on first hop only. Redundant vs Xet for Xet-backed repos (HF closed issue #3232 as "use Xet"), **but genuinely helps non-Xet/legacy-LFS repos** which get no intra-file parallelism. → our Wave 3 opt-in.
|
||||
- https://github.com/huggingface/huggingface_hub/issues/3232
|
||||
|
||||
## 5. aria2 — OPTIONAL, rejected for OmniVoice
|
||||
`aria2c -x16 -s16 -c --header="Authorization: Bearer <token>"` is 3–5× on plain LFS, but: no dedup (worse than Xet for Xet repos), per-OS GPLv2 binary to package (parity burden — would have to be opt-in anyway), stdout/RPC progress scraping. The custom httpx path covers the same need with no binary. → not bundled.
|
||||
- https://gist.github.com/padeoe/697678ab8e528b85a2a7bddafea1fa4f
|
||||
|
||||
## 6. Mirrors / HF_ENDPOINT — OPTIONAL, region-gated, breaks Xet
|
||||
`HF_ENDPOINT=https://hf-mirror.com` redirects Hub traffic (standard for China). **Xet CAS/presigned URLs point at HF infra → mirrors generally don't serve the Xet protocol → traffic falls back to classic LFS** (no dedup, no Xet parallelism). So mirror and Xet fast-path are mutually exclusive; the realistic China stack is mirror + LFS + (our opt-in) segmented accelerator. → our Wave 4 opt-in, per-call `endpoint=` not process-wide.
|
||||
|
||||
## 7. Progress / speed — USE `tqdm_class` (xet-aware) + `dry_run` preflight
|
||||
Unlike hf_transfer, **Xet reports progress through the same tqdm interface**; huggingface_hub aggregates per-file/thread bytes into a shared bar and feeds the `tqdm_class` you pass. So `snapshot_download(tqdm_class=...)` yields reliable aggregate bytes/total/rate/ETA even under parallel fetch. `snapshot_download(dry_run=True)` returns per-file sizes + cached flags → use for "will download X of Y, N GB" preflight. Speed sampling tunable via `HF_XET_DATA_PROGRESS_UPDATE_INTERVAL` (200ms).
|
||||
- https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
|
||||
- https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/_snapshot_download.py
|
||||
|
||||
## Recommended architecture (→ plan)
|
||||
Pin `huggingface_hub>=1.7` + `hf-xet`; let Xet be the default (it IS the IDM technique). Drive `snapshot_download(repo_id, tqdm_class=OmniVoiceProgress, max_workers=8, endpoint=<opt-in mirror>)`; `dry_run=True` first for total/remaining; aggregate bytes in a backend tracker → one overall bar (speed/remaining/ETA). Opt-in only: `HF_XET_HIGH_PERFORMANCE` (max speed), `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY` (HDD), a custom httpx **segmented downloader for legacy-LFS repos**, and an `HF_ENDPOINT` mirror (classic-LFS fallback). Never enable hf_transfer; never bundle aria2; never make the segmented path the default.
|
||||
@@ -1,28 +0,0 @@
|
||||
# SPIKE — FDL-00: Catalog Xet vs LFS classification
|
||||
|
||||
**Date:** 2026-06-13 · **Method:** HF API `GET /api/models/{repo}?expand[]=xetEnabled` (authoritative).
|
||||
|
||||
## Result: 25 / 25 catalog repos are Xet-backed
|
||||
|
||||
| backend | count |
|
||||
|---------|-------|
|
||||
| xet | 25 |
|
||||
| lfs | 0 |
|
||||
| unknown | 0 |
|
||||
|
||||
Every repo in `backend/config/models.yaml` — including both first-run defaults (`k2-fsa/OmniVoice` TTS, `Systran/faster-whisper-large-v3` ASR) — returns `xetEnabled: true`. Full list: all entries under TTS / ASR / Diarisation (k2-fsa, Systran×5, mlx-community×9, openai, nvidia×2, UsefulSensors×2, pyannote, OpenMOSS, KittenML, deepdml).
|
||||
|
||||
## Detection caveat (important for the executor)
|
||||
|
||||
The installed client is **huggingface_hub 1.7.2**, whose `repo_info(..., files_metadata=True)` siblings expose only `blob_id, lfs, rfilename, size` — **no `xet_file`, and no `xet_enabled` on the info object.** A first pass that inferred backend from siblings wrongly reported "0/25 xet, all LFS." Do **not** classify Xet status from `repo_info` siblings on this client version. The reliable signal is the Hub API `xetEnabled` expand field (used here) or `hf_xet` actually engaging at download time. Re-check after any `huggingface_hub` bump — newer versions surface `xet_enabled` directly.
|
||||
|
||||
## Verdict for Wave 3 (segmented accelerator): LOW priority
|
||||
|
||||
Because the entire current catalog is Xet-backed and `hf_xet` is installed, Xet already provides chunked parallel range-gets (the IDM/uGet behavior) for **every** model we ship. The custom segmented downloader (Wave 3) is therefore **not needed to speed up any current default model** — it remains valuable only for:
|
||||
- the **mirror / restricted-network path** (Wave 4: `HF_ENDPOINT` falls back to classic LFS, no Xet), and
|
||||
- any **future non-Xet repo** a user adds.
|
||||
|
||||
**Recommendation:** proceed with W1 (maximize/guarantee Xet) and W2 (accurate progress) as the real wins for today's catalog; keep W3 as opt-in, build it alongside W4's mirror path where it actually pays off. This matches the PLAN's original framing — confirmed, not changed.
|
||||
|
||||
## Consequence for W1/W2 framing
|
||||
W1 "guarantee the Xet fast path" is correctly the primary lever: these repos download via Xet **only if** the client engages it (hf_xet installed ✓ + huggingface_hub recent ✓). The W2 live smoke test should confirm Xet is actually used (fast parallel aggregate progress on a real install), since `xetEnabled=true` is a Hub-side capability, not proof the client took the Xet path.
|
||||
@@ -1,49 +0,0 @@
|
||||
# SUMMARY — FDL Waves 0–2 (fast model downloads)
|
||||
|
||||
**Date:** 2026-06-13 · **Scope shipped:** W0 (spike), W1 (maximize Xet), W2 (accurate progress). W3/W4 deferred.
|
||||
|
||||
## What landed
|
||||
|
||||
**W0 — spike (FDL-00).** Classified all 25 `models.yaml` repos via the HF API `xetEnabled` field → **25/25 Xet-backed** (incl. both first-run defaults). See `260613-fdl-SPIKE.md`. Verdict: Wave 3 (segmented accelerator) is **LOW priority** — Xet already gives parallel chunked transfer for every shipped model. Detection caveat recorded: `repo_info` siblings on hf_hub **1.7.2** expose no xet metadata; classify via the `xetEnabled` API field, not siblings.
|
||||
|
||||
**W1 — maximize + guarantee Xet (FDL-01..04).**
|
||||
- `pyproject.toml`: pinned `huggingface_hub>=1.7` + `hf-xet>=1.1` explicitly (was transitive/unpinned); no `hf_transfer`. Resolves to hf_hub 1.7.2 / hf-xet 1.4.2, single version.
|
||||
- `download.py`: `install_model` now drives `snapshot_download` with explicit `tqdm_class` (our progress-emitting subclass), `max_workers` (prefs `download_max_workers`, default 8), and `endpoint` (prefs `hf_endpoint` — W4 hook). `apply_xet_env()` applies opt-in `HF_XET_HIGH_PERFORMANCE` + `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY` (both default OFF, env wins).
|
||||
- `system.py`: `/system/info` now returns `fast_download {xet_enabled, xet_version, high_performance}`; logged once at startup.
|
||||
|
||||
**W2 — accurate downloaded/remaining + speed (FDL-05..07).**
|
||||
- Preflight `snapshot_download(dry_run=True)` → `compute_plan()` → `install_plan` SSE event with `total_bytes / cached_bytes / to_download_bytes / n_files / n_cached` **before bytes flow**. Degrades to totals=None on gated/older repos.
|
||||
- New `utils/download_aggregator.py`: one source of truth for overall progress. Fed by a byte-sink on the patched tqdm; distinguishes byte bars (unit 'B', keyed by bar id) from the "Fetching N files" count bar; emits one throttled `aggregate` event (bytes_done/total/windowed rate/eta/files).
|
||||
- Frontend `Settings.jsx` + `setup.ts`: overall bar driven by the aggregate; bar % = `max(byte%, file%)`; shows cached-skip + files-progress; `⚡ fast download` badge from `/system/info`. i18n keys added to `en.json`.
|
||||
|
||||
## Rebase reconciliation (main disabled Xet)
|
||||
Rebasing onto latest main surfaced that main now sets **`HF_HUB_DISABLE_XET=1`** (main.py) — a deliberate choice to force the classic LFS path because Xet's progress bypasses the tqdm hook (the exact limitation found here). Reconciled rather than fought:
|
||||
- `fast_download` status now reports the **runtime truth**: `xet_installed` + `xet_active` (active = installed AND not disabled) + `xet_enabled` alias. Default `xet_active=false`; the ⚡ badge only shows when Xet actually runs. Startup log: `downloads: Xet disabled → legacy LFS …`.
|
||||
- Docs rewritten: default backend is **legacy LFS for accurate progress**; Xet is opt-in via `HF_HUB_DISABLE_XET=0` (coarser progress). The hf-xet pin stays (harmless; ready for a future Xet progress hook).
|
||||
- Net: W2's progress is the value either way; W1's "maximize Xet" is dormant by main's design, not removed.
|
||||
|
||||
## Decisions / "use judgment" notes
|
||||
- **Xet progress limitation (verified by live smoke).** Under Xet + hf_hub 1.7.2 the per-file **byte** bars never advance `n` and never `close()` through our tqdm (Xet fetches chunks out-of-band). Only the **file-count bar** is live. So: mid-download the overall bar is **file-granular** (moves 0→N files), and `complete()` flushes `bytes_done` to the exact preflight total on success (verified: final `74420620/74420620`, files 4/4). True live byte-speed is only available on classic-LFS/mirror repos (W4). This is a real constraint, not a bug — documented here and worth surfacing in W4 docs.
|
||||
- Per-file detail kept inline (existing single-line summary, now aggregate-sourced) rather than a new collapsible panel — limited risk; can revisit.
|
||||
|
||||
## Drive-by fix
|
||||
- `download.py` imported no `os`, but `_validate_snapshot_has_weights` uses `os.walk` → latent `NameError` on every install. Added `import os`.
|
||||
|
||||
## Verification
|
||||
- `tests/backend/setup/test_download_preflight.py` — 10 pass (compute_plan splits, aggregator byte/count routing, close-credit, windowed rate/eta, registry feed + finish noop).
|
||||
- `pytest -k "download or install or model or engine or setup"` — 149 passed, 7 skipped, 0 failed.
|
||||
- `frontend typecheck:ci` — exit 0.
|
||||
- Live smoke (real install of `mlx-community/whisper-tiny-mlx`, then deleted): `install_plan` exact; aggregate files 0→1→4; final bytes==total; `/system/info` + startup log correct.
|
||||
|
||||
## W4 — mirror + cancel + docs (FDL-10..12, shipped)
|
||||
- **Mirror (FDL-10):** `snapshot_download(endpoint=…)` honours prefs `hf_endpoint` / env `HF_ENDPOINT` on both preflight and download — per-call, no process-wide mutation. Documented as the classic-LFS (non-Xet) path that restores continuous byte-speed.
|
||||
- **Cancel (FDL-11):** `POST /models/install/cancel {repo_id}` sets a cancel flag checked at each retry boundary → emits `install_cancelled`, clears the cooldown (cancel ≠ failure). Limitation: an in-flight single-file fetch isn't interruptible in hf_hub 1.7.2; cancel lands at the next retry boundary. Frontend treats `install_cancelled` as a terminator (clears row + refetch).
|
||||
- **Docs (FDL-12):** `docs/downloading-models.md` (Xet fast path, progress semantics incl. the byte-speed limitation, opt-in tuning knobs, mirror/restricted-network, cancel, troubleshooting) + README pointer. Docs-sync rule satisfied in-PR.
|
||||
|
||||
## W3 — opt-in segmented accelerator (FDL-08/09, shipped)
|
||||
Reprioritised from LOW to HIGH after the rebase: since main forces Xet off, the default path is single-stream legacy LFS, so a segmented downloader is the way to get **both** parallel speed and live byte progress.
|
||||
- `services/segmented_download.py`: async multi-connection Range downloader for one file — parallel byte-ranges, resume (`.part` + manifest), per-segment short-read truncation guard, optional sha256/etag verify, cancel, single-stream fallback when the server won't range. **Auth-safe**: the HF `Authorization` header goes only to `huggingface.co`/`hf.co`; never forwarded to a CDN host on redirect (unit-tested).
|
||||
- Dispatch (`download.py`): opt-in via prefs `segmented_downloader` / env `OMNIVOICE_SEGMENTED_DOWNLOAD` (default OFF). When on and Xet inactive, fetches each repo file into the HF cache mirroring `hf_hub_download` (blobs + snapshot symlinks + `refs/main`), feeding **real bytes** to the aggregator. Any failure falls back to `snapshot_download` — the accelerator can never break a correct install.
|
||||
- Verified live (accelerator ON): real mid-download byte progress (1.5 KB → 71 MB, rate ramping to **16.6 MB/s**), final `bytes_done == total`, `/models` shows `installed: True`, delete frees the right bytes.
|
||||
- Fixed a `complete()` double-count (was adding a full total on top of accumulated segmented bytes → 2×); now replaces byte bars so the sum is exactly total.
|
||||
- Tests: `tests/backend/services/test_segmented_download.py` (7 cases) covering parallel range reassembly, single-stream fallback, the auth header reaching only the HF host (never a CDN), size/truncation rejection, cancellation, and byte-callback totals — plus an aggregator double-count regression.
|
||||
@@ -1,363 +0,0 @@
|
||||
---
|
||||
phase: 260613-mm2
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- backend/services/tts_backend.py
|
||||
- backend/services/model_manager.py
|
||||
- backend/services/subprocess_backend.py
|
||||
- backend/services/model_lifecycle.py # NEW
|
||||
- backend/api/routers/system.py
|
||||
- backend/api/routers/setup/download.py
|
||||
- backend/api/routers/setup/models.py
|
||||
- tests/test_engines.py
|
||||
- tests/backend/services/test_model_lifecycle.py # NEW
|
||||
- tests/backend/services/test_subprocess_reaper.py
|
||||
autonomous: true
|
||||
requirements:
|
||||
# ── Tier 1 — Correctness (Wave 1) ──────────────────────────────────────────
|
||||
- MM2-01 # Registry reuses one active instance and calls unload() on engine switch
|
||||
- MM2-02 # Per-engine unload() overrides (in-process drop+free_vram; subprocess -> unload_sidecar)
|
||||
- MM2-03 # /model/loaded + /model/unload report ASR honestly; no unloadable:False-but-loaded lies
|
||||
# ── Tier 2 — Single lifecycle surface (Wave 2) ─────────────────────────────
|
||||
- MM2-04 # model_lifecycle facade owns list_loaded/unload/unload_all/free_vram across all 3 worlds
|
||||
- MM2-05 # Idle/timeout config unified through core.prefs.resolve (env still wins); no duplicated constants
|
||||
# ── Tier 3 — Robustness & observability (Wave 3) ───────────────────────────
|
||||
- MM2-06 # _install_cooldowns bounded (evict on success + TTL); no unbounded growth
|
||||
- MM2-07 # Snapshot weight validation is per-role, not one 5 MB magic number
|
||||
- MM2-08 # Subprocess sidecars self-report VRAM in pong; panel shows real MB, not 0
|
||||
- MM2-09 # scan_cache_dir() -> disk-walk fallback logs WHY it fell back (WinError #117/#118)
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "Switching the active TTS backend in Settings releases the outgoing engine's VRAM before the new one loads — verified by asserting the outgoing instance's unload() was called exactly once on switch."
|
||||
- "TTSBackend.unload() is overridden by OmniVoiceBackend (drops model ref + free_vram) and by every SubprocessBackend subclass (routes to unload_sidecar); all overrides are idempotent and safe before first generate()."
|
||||
- "/model/loaded never reports a model as loaded with a misleading unloadable flag: the ASR row's unloadable reflects whether it can actually be released independently of the TTS lifecycle."
|
||||
- "services.model_lifecycle is the single import surface for list_loaded()/unload(id)/unload_all()/free_vram(); system.py routers call it instead of re-enumerating models inline."
|
||||
- "Idle timeouts for the in-process model and subprocess sidecars resolve through core.prefs.resolve(... env=...) so an env var still wins and the Settings store can override; no module duplicates IDLE_TIMEOUT_SECONDS by hand."
|
||||
- "_install_cooldowns cannot grow without bound: entries are removed on successful install and stale entries are evicted by TTL."
|
||||
- "A live subprocess sidecar reports a non-zero vram_mb in /model/loaded when it actually holds GPU memory (pong carries the figure); CPU-only sidecars report 0 truthfully."
|
||||
- "When scan_cache_dir() raises and the code falls back to the on-disk walk, the reason is logged at WARNING with the exception type (the #117/#118 WinError-448 path is no longer silent)."
|
||||
- "uv run pytest tests/test_engines.py tests/backend/services/test_model_lifecycle.py tests/backend/services/test_subprocess_reaper.py tests/test_model_load_timeout.py passes."
|
||||
- "No on-disk model state changes; no new runtime dependency added; behavior degrades gracefully (not errors) on MPS/CPU where VRAM APIs are sparse."
|
||||
artifacts:
|
||||
- path: "backend/services/tts_backend.py"
|
||||
provides: "Active-instance reuse + unload-on-switch in get_active_tts_backend(); per-engine unload() overrides"
|
||||
contains: "_active_instance AND (def unload)"
|
||||
- path: "backend/services/model_lifecycle.py"
|
||||
provides: "Facade owning list_loaded/unload/unload_all/free_vram across in-process + subprocess models"
|
||||
contains: "def list_loaded AND def unload_all"
|
||||
- path: "backend/api/routers/system.py"
|
||||
provides: "Thin /model/loaded + /model/unload routers delegating to model_lifecycle"
|
||||
contains: "model_lifecycle"
|
||||
key_links:
|
||||
- from: "get_active_tts_backend() (tts_backend.py:1235)"
|
||||
to: "outgoing backend.unload()"
|
||||
via: "module-level _active_instance compared against newly-resolved active_backend_id()"
|
||||
pattern: "_active_instance"
|
||||
- from: "system.py /model/loaded + /model/unload (system.py:129, 210)"
|
||||
to: "model_lifecycle.list_loaded() / model_lifecycle.unload()"
|
||||
via: "import services.model_lifecycle"
|
||||
pattern: "model_lifecycle\\.(list_loaded|unload)"
|
||||
- from: "subprocess sidecar pong reply (subprocess_backend.py:435-438)"
|
||||
to: "list_live_sidecars() vram_mb field"
|
||||
via: "ping reply carries allocated VRAM measured inside the sidecar process"
|
||||
pattern: "vram_mb"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Clean up OmniVoice's model-management subsystem ("v2"). Today load / unload / list / free-VRAM each behave differently across three worlds — the in-process model (`model_manager.py`), the TTS backend registry (`tts_backend.py`), and subprocess sidecars (`subprocess_backend.py`) — with no single lifecycle owner. This produces one real user-facing bug (VRAM leak on engine switch), inaccurate VRAM/unloadable reporting, an unbounded cooldown dict, and a silent cache fallback.
|
||||
|
||||
This is **cleanup + correctness, not a rewrite.** The Wave 13 idle-reaper and the SubprocessBackend primitive are sound and stay. The `TTSBackend.unload()` contract already exists as a documented default no-op (`tts_backend.py:149`) explicitly deferred to "Phase 2"; this plan *is* that Phase-2 follow-through — wire the registry to call it, override it per engine, and unify the surrounding surface.
|
||||
|
||||
Three tiers, executed in order (each independently shippable, continuous-to-main per the v0.3.0 cadence):
|
||||
- **Wave 1 / Tier 1 — Correctness:** MM2-01..03. The VRAM leak on switch + honest unload reporting. Highest value; ship first.
|
||||
- **Wave 2 / Tier 2 — Single lifecycle surface:** MM2-04..05. Extract `model_lifecycle` facade + unify idle/timeout config.
|
||||
- **Wave 3 / Tier 3 — Robustness & observability:** MM2-06..09. Bounded cooldowns, per-role weight validation, sidecar VRAM self-report, cache-fallback logging.
|
||||
|
||||
Output: PRs on branches off `main` (one per wave is fine), each green on the listed pytest selection. No push until the orchestrator merges; tests added with each wave.
|
||||
|
||||
Out of scope (call out, do not touch): GPU-pool per-engine sizing (`model_manager.py:42`, `_GPU_VRAM_PER_JOB_GB`) and torch.compile tuning — those are performance, not cleanup, and carry regression risk against #278/#315.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/STATE.md
|
||||
@./CLAUDE.md
|
||||
|
||||
# Files under edit (read before editing)
|
||||
@backend/services/tts_backend.py
|
||||
@backend/services/model_manager.py
|
||||
@backend/services/subprocess_backend.py
|
||||
@backend/api/routers/system.py
|
||||
@backend/api/routers/setup/download.py
|
||||
@backend/api/routers/setup/models.py
|
||||
|
||||
# Reference only — establish patterns, do NOT modify
|
||||
@backend/core/prefs.py
|
||||
@tests/test_engines.py
|
||||
@tests/backend/services/test_subprocess_reaper.py
|
||||
|
||||
<interfaces>
|
||||
<!-- Verified during planning. Executor should use these directly. -->
|
||||
|
||||
backend/services/tts_backend.py
|
||||
- class TTSBackend(ABC) (line 59); unload() default no-op (line 149) — contract already documented:
|
||||
idempotent, synchronous, safe before first generate().
|
||||
- OmniVoiceBackend.__init__(self, model=None) (line 174); self._model reuses model_manager singleton.
|
||||
- _REGISTRY: dict[str, type[TTSBackend]] (line 1109, a _LazyRegistry).
|
||||
- active_backend_id() (line 1228) -> prefs.resolve("tts_backend", env="OMNIVOICE_TTS_BACKEND", default="omnivoice").
|
||||
- get_active_tts_backend(*, model=None) (line 1235) — builds a FRESH instance every call, no teardown. THE leak.
|
||||
|
||||
backend/core/prefs.py
|
||||
- resolve(key: str, *, env: Optional[str] = None, default: Any = None) -> Any (line 75) — env wins, then store, then default.
|
||||
|
||||
backend/services/model_manager.py
|
||||
- module global `model` (line 111); `_last_used`; free_vram() (line 678); idle_worker() (line 667).
|
||||
- IDLE_TIMEOUT_SECONDS imported from core.config (line 33); duplicated as _IDLE_TIMEOUT_SECONDS (line 114). Collapse.
|
||||
- offload_tts_for_asr() (line 701) / restore_tts_after_asr() — ad-hoc ASR<->TTS VRAM juggling; _diar_pipeline global.
|
||||
|
||||
backend/services/subprocess_backend.py
|
||||
- protocol op set (line 74); SIDECAR_IDLE_TIMEOUT_S = env-only float (line 107) — move to prefs.resolve.
|
||||
- list_live_sidecars() -> list[dict] (line 181); unload_sidecar(engine_id) (line 199); unload_all_sidecars() (line 205).
|
||||
- health/ping: _send({"op":"ping"}) then expect {"op":"pong"} (lines 435-438). Add vram_mb to the pong here AND
|
||||
in the sidecar entry-point that answers ping (search the sidecar worker for the "ping"->"pong" handler).
|
||||
|
||||
backend/api/routers/system.py
|
||||
- GET /model/loaded (line 129) — ~80 lines of inline enumeration of TTS/ASR/diar/sidecars. Replace body with
|
||||
model_lifecycle.list_loaded().
|
||||
- POST /model/unload/{model_id} (line 210) — handles "tts" | "diarization" | "sidecar:<id>" | "sidecars".
|
||||
Replace body with model_lifecycle.unload(model_id).
|
||||
|
||||
backend/api/routers/setup/download.py
|
||||
- _install_cooldowns dict (line 27) — unbounded. _validate_snapshot_has_weights (line 55) + _MIN_WEIGHT_BYTES 5 MB
|
||||
(line 45) — single magic number across roles.
|
||||
|
||||
backend/api/routers/setup/models.py
|
||||
- scan_cache_dir() with silent disk-walk fallback (~line 268-280) + _scan_cache_on_disk (line 177).
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<!-- ════════════ WAVE 1 / TIER 1 — CORRECTNESS ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1 (MM2-02): Per-engine unload() overrides</name>
|
||||
<files>backend/services/tts_backend.py</files>
|
||||
<action>
|
||||
The base-class `unload()` no-op already exists (tts_backend.py:149) with a documented contract. Override it where it matters. Do this BEFORE Task 2 — the registry switch (Task 2) calls these.
|
||||
|
||||
- OmniVoiceBackend (line 162): override `unload(self)`. Drop the local model ref (`self._model = None`) and, because OmniVoice shares the singleton owned by model_manager, also release that: `import services.model_manager as mm; mm.model = None; mm.free_vram()`. Idempotent — guard on `mm.model is not None` before free_vram(). Safe before first generate() (no-op when nothing loaded).
|
||||
- Every SubprocessBackend subclass: implement `unload(self)` on the SubprocessBackend base (subprocess_backend.py — the duck-typed `_is_subprocess_isolated` class) so all subclasses inherit it. It must call `unload_sidecar(self.id)` (force-shut this engine's sidecar; busy sidecars are skipped, never interrupted — existing semantics). Idempotent: unload_sidecar on a non-running engine returns 0, no raise.
|
||||
- In-process non-OmniVoice engines that hold their own model (e.g. KittenTTS/VoxCPM2 keep refs in __init__): override unload() to drop the ref + best-effort empty_cache via the existing free_vram() helper if they used GPU. Where an engine genuinely holds nothing resident, leave the base no-op (and note it in the SUMMARY so the future CI gate knows it's intentional, not missed).
|
||||
|
||||
Honor the contract comment verbatim: idempotent, synchronous, safe pre-load.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "def unload" backend/services/tts_backend.py backend/services/subprocess_backend.py</automated>
|
||||
<automated>uv run python -c "from services.tts_backend import OmniVoiceBackend; b=OmniVoiceBackend(); b.unload(); b.unload(); print('idempotent ok')"</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- OmniVoiceBackend.unload() drops both self._model and mm.model and calls free_vram(), guarded for idempotency.
|
||||
- SubprocessBackend.unload() routes to unload_sidecar(self.id); inherited by all subprocess engines.
|
||||
- Calling unload() twice, and before any generate(), never raises.
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2 (MM2-01): Registry reuses one active instance + unloads on switch</name>
|
||||
<files>backend/services/tts_backend.py</files>
|
||||
<action>
|
||||
Fix the leak at get_active_tts_backend() (line 1235). Today it builds a fresh instance every call with no teardown of the prior engine — switching engines (or repeated synth) leaks VRAM until GC. This is the root cause behind the #278 comment thread.
|
||||
|
||||
- Add a module-level cache: `_active_instance: TTSBackend | None = None` and `_active_instance_id: str | None = None`.
|
||||
- In get_active_tts_backend(): resolve `bid = active_backend_id()`. If `_active_instance is not None` and `_active_instance_id != bid`, call `_active_instance.unload()` (best-effort, wrap in try/except so a bad unload can't block the switch — log on failure) before discarding it.
|
||||
- Build the new instance, store it as `_active_instance` + `_active_instance_id = bid`, return it.
|
||||
- IMPORTANT subtlety: OmniVoiceBackend takes `model=`. When `model=` is passed (the caller already has a loaded model), do NOT cache that instance as the shared `_active_instance` blindly — it's a per-call view over the shared singleton. Keep current behavior for the `model=` path (return a fresh OmniVoiceBackend(model=model)) but still trigger unload() of a *different* outgoing engine first. Pick the simplest correct rule: the cache tracks the configured backend id; passing model= for the SAME id reuses, switching id always unloads the previous. Document the rule in a comment.
|
||||
- Add a module-level `reset_active_backend()` helper that unloads + clears the cache, for app shutdown and tests.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "_active_instance\|def reset_active_backend\|def get_active_tts_backend" backend/services/tts_backend.py</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- Switching backend id calls the outgoing instance's unload() exactly once before the new instance is built.
|
||||
- A bad/raising unload() is caught + logged, never blocks the switch.
|
||||
- reset_active_backend() exists and is idempotent.
|
||||
- The model= fast-path for OmniVoice still works (no double-load).
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3 (MM2-03): Honest /model/loaded + /model/unload for ASR</name>
|
||||
<files>backend/api/routers/system.py</files>
|
||||
<action>
|
||||
The ASR row (system.py:166-175) is reported as unloadable:False, vram_mb:0 even when loaded on GPU, and /model/unload doesn't expose the offload-to-CPU path. Make reporting truthful WITHOUT changing the ASR<->TTS lifecycle coupling (that coupling is intentional — offload_tts_for_asr/restore_tts_after_asr).
|
||||
|
||||
- ASR row: keep unloadable reflecting reality. If ASR truly cannot be released independently of TTS, keep unloadable:False but add a `note` field ("released with TTS") so the UI explains it rather than showing a dead button. Do not invent a separate ASR unload that breaks the WhisperX large-v3 offload path.
|
||||
- vram_mb: if ASR currently runs on CPU (device "cpu" in the row), 0 is correct — leave it but make the device value derive from where the pipe actually is, not a hardcoded "cpu".
|
||||
- This task is intentionally small; the bigger restructure is Task 4 (facade). Land MM2-03 as the honest-reporting fix, then Task 4 moves the enumeration into the facade.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run pytest tests/test_engines.py -q 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- No row reports loaded-but-with-a-misleading-unloadable flag; ASR carries an explanatory note when unloadable:False.
|
||||
- Device field reflects the actual device of the ASR pipe.
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 4 (MM2-01..03 tests): Wave 1 regression tests</name>
|
||||
<files>tests/test_engines.py</files>
|
||||
<action>
|
||||
Add tests proving the leak fix and the unload contract:
|
||||
- test_switching_backend_unloads_previous: monkeypatch two fake backends into _REGISTRY, set active to A (get_active_tts_backend), switch prefs to B, assert A.unload() was called exactly once before B is returned.
|
||||
- test_unload_is_idempotent_and_preload_safe: OmniVoiceBackend().unload() twice + before generate() never raises.
|
||||
- test_reset_active_backend_clears_cache: after reset_active_backend(), the next get_active_tts_backend() builds fresh.
|
||||
Reuse the existing fixture style in tests/test_engines.py (it already monkeypatches the registry / availability). Keep tests CPU-only (no real model load).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run pytest tests/test_engines.py -q 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<done>All three new tests pass; existing test_engines.py tests still green.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 2 / TIER 2 — SINGLE LIFECYCLE SURFACE ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 5 (MM2-04): Extract services/model_lifecycle.py facade</name>
|
||||
<files>backend/services/model_lifecycle.py</files>
|
||||
<action>
|
||||
Create backend/services/model_lifecycle.py as the single owner of cross-world model lifecycle. It composes the existing pieces — it does NOT reimplement loading.
|
||||
|
||||
Public surface:
|
||||
- list_loaded() -> list[dict]: returns the unified rows currently assembled inline in system.py:129-207 (TTS, ASR, diarization, subprocess sidecars). Move that logic here verbatim first, then improve (MM2-03 note field, MM2-08 sidecar vram once Task 8 lands).
|
||||
- unload(model_id: str) -> dict: the dispatch currently inline in system.py:210-242 ("tts" | "diarization" | "sidecar:<id>" | "sidecars"). Move here; keep async-lock semantics for the in-process model (mm._model_lock).
|
||||
- unload_all() -> dict: unload every releasable model (in-process TTS + diar + all sidecars). New convenience used by app shutdown.
|
||||
- free_vram(): thin re-export of model_manager.free_vram() so callers have one import.
|
||||
Keep the "never let sidecar enumeration break the panel" try/except guard.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run python -c "import services.model_lifecycle as ml; print([f for f in ('list_loaded','unload','unload_all','free_vram') if hasattr(ml,f)])"</automated>
|
||||
</verify>
|
||||
<done>model_lifecycle exposes list_loaded/unload/unload_all/free_vram; logic moved out of system.py (not duplicated).</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 6 (MM2-04): Thin system.py routers + facade tests</name>
|
||||
<files>backend/api/routers/system.py, tests/backend/services/test_model_lifecycle.py</files>
|
||||
<action>
|
||||
- Replace the bodies of GET /model/loaded (line 129) and POST /model/unload/{model_id} (line 210) with calls to model_lifecycle.list_loaded() / model_lifecycle.unload(model_id). Preserve the exact response shapes (frontend hooks.ts useModelStatus/useFlushMemory + the flush dropdown depend on {models, count} and {unloaded, success, ...}). The 400 on unknown model_id stays.
|
||||
- New tests/backend/services/test_model_lifecycle.py: list_loaded with nothing loaded returns {models:[], count:0}; unload("tts") when not loaded returns success:False reason:"not loaded"; unload("sidecars") with no sidecars returns count:0; unknown id raises/400 path. Mock model_manager + subprocess_backend so no real models load.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run pytest tests/backend/services/test_model_lifecycle.py -q 2>&1 | tail -20</automated>
|
||||
<automated>grep -n "model_lifecycle" backend/api/routers/system.py</automated>
|
||||
</verify>
|
||||
<done>system.py routers are thin delegations; response shapes unchanged; new facade tests pass.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 7 (MM2-05): Unify idle/timeout config through prefs.resolve</name>
|
||||
<files>backend/services/model_manager.py, backend/services/subprocess_backend.py</files>
|
||||
<action>
|
||||
- model_manager.py: remove the duplicated `_IDLE_TIMEOUT_SECONDS = IDLE_TIMEOUT_SECONDS` (line 114). Resolve at use-site in idle_worker() via prefs: `prefs.resolve("idle_timeout_seconds", env="OMNIVOICE_IDLE_TIMEOUT_S", default=IDLE_TIMEOUT_SECONDS)`. Keep core.config.IDLE_TIMEOUT_SECONDS as the default source.
|
||||
- subprocess_backend.py: replace the env-only `SIDECAR_IDLE_TIMEOUT_S` (line 107) read with prefs.resolve("sidecar_idle_timeout_seconds", env="OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S", default=300.0). Preserve "<=0 disables reaping" semantics and the existing reaper-start guard (line 222). Resolve lazily (function call), not at import, so a test/setting change takes effect — but keep a sensible cached default for the hot reaper loop.
|
||||
- Both must keep env precedence (env wins over store) — that's exactly what prefs.resolve already does.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "_IDLE_TIMEOUT_SECONDS\|prefs.resolve\|SIDECAR_IDLE_TIMEOUT" backend/services/model_manager.py backend/services/subprocess_backend.py</automated>
|
||||
<automated>uv run pytest tests/backend/services/test_subprocess_reaper.py -q 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- No hand-duplicated IDLE_TIMEOUT constant; both timeouts resolve via prefs with env precedence.
|
||||
- Reaper "<=0 disables" + busy-skip behavior unchanged; all 10+ reaper tests still pass.
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 3 / TIER 3 — ROBUSTNESS & OBSERVABILITY ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 8 (MM2-08): Subprocess sidecars self-report VRAM in pong</name>
|
||||
<files>backend/services/subprocess_backend.py</files>
|
||||
<action>
|
||||
Sidecar VRAM is reported as 0 (system.py:192-203 / list_live_sidecars) because the parent can't measure a child's GPU memory. Have the child measure itself.
|
||||
|
||||
- In the sidecar worker's ping handler (the code that answers {"op":"ping"} with {"op":"pong"} — find it in the sidecar entry-point module), include `vram_mb`: measure inside the child via torch.cuda.memory_allocated() (CUDA) or torch.mps.driver_allocated_memory() (MPS, guarded), else 0. Same degrade-gracefully pattern as system.py:147-156.
|
||||
- Parent: in the health-check ping/pong path (subprocess_backend.py:435-438), capture reply["vram_mb"] and stash it on the sidecar record so list_live_sidecars() (line 181) can surface it. Refresh opportunistically on each successful ping; default to last-known or 0 if never measured.
|
||||
- Keep the contract that enumeration never breaks the panel.
|
||||
This is CUDA/MPS-aware and degrades to 0 on CPU — honoring cross-platform parity (default behavior identical; the number is just more accurate where the API exists).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "vram_mb" backend/services/subprocess_backend.py</automated>
|
||||
<automated>uv run pytest tests/backend/services/test_subprocess_reaper.py -q 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<done>list_live_sidecars() exposes a vram_mb sourced from the child's own measurement; 0 only when truly CPU/unmeasured; reaper tests still green.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 9 (MM2-06, MM2-07): Bounded cooldowns + per-role weight validation</name>
|
||||
<files>backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
- MM2-06: _install_cooldowns (line 27) grows unbounded. On a successful install, delete the repo's cooldown entry. Add a TTL sweep: when reading/writing the dict, evict entries older than a fixed window (reuse the existing cooldown window constant; pick the larger of cooldown-window and e.g. 1h). Keep it simple — a dict + timestamps, swept on access. No new dep.
|
||||
- MM2-07: _validate_snapshot_has_weights (line 55) + _MIN_WEIGHT_BYTES 5 MB (line 45) is one magic number for all roles. Make the threshold per-role/per-extension: safetensors/bin/ckpt expect the existing floor; .onnx models (kittentts, supertonic, sherpa) can be legitimately smaller — set a lower, role-aware floor so a valid small ONNX model isn't flagged as truncated. Keep the #352 truncation-catch intent (catch a 0-byte / KB-sized partial), just stop false-positiving small-but-complete models.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "_install_cooldowns\|_MIN_WEIGHT_BYTES\|def _validate_snapshot_has_weights" backend/api/routers/setup/download.py</automated>
|
||||
<automated>uv run pytest tests/ -k "download or install or model" -q 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<done>Cooldown dict is bounded (evict-on-success + TTL sweep); weight validation floor varies by role/extension; #352 truncation still caught.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 10 (MM2-09): Log why scan_cache_dir() fell back to disk walk</name>
|
||||
<files>backend/api/routers/setup/models.py</files>
|
||||
<action>
|
||||
The scan_cache_dir() -> _scan_cache_on_disk() fallback (~line 268-280, helper at line 177) silently swallows the exception — this is the #117/#118 Windows WinError-448 path. Wrap the fallback so it logs at WARNING with the exception type and a one-line reason ("scan_cache_dir failed (%s); falling back to on-disk walk of %s") before walking. Do not change the fallback behavior itself — just stop it being invisible in logs. Keep it from ever raising out (the panel must still render).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "falling back\|logger.warning\|_scan_cache_on_disk\|scan_cache_dir" backend/api/routers/setup/models.py | head</automated>
|
||||
</verify>
|
||||
<done>The disk-walk fallback logs a WARNING naming the exception type; behavior otherwise unchanged; never raises out.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
Full-suite gate after each wave (run the relevant subset per wave, full set before the last PR):
|
||||
|
||||
1. `uv run pytest tests/test_engines.py tests/backend/services/test_model_lifecycle.py tests/backend/services/test_subprocess_reaper.py tests/test_model_load_timeout.py tests/test_model_manager_preload.py -q` — all green.
|
||||
2. `uv run pytest tests/ -k "download or install or model or engine" -q` — green (Tier 3 touch points).
|
||||
3. Response-shape guard: GET /model/loaded still returns {models, count}; POST /model/unload returns {unloaded, success, ...}; 400 on unknown id. (Covered by test_model_lifecycle.py.)
|
||||
4. No new runtime dependency: `git diff pyproject.toml uv.lock` is empty.
|
||||
5. Localization/CJK + redaction gates unaffected: `uv run pytest tests/test_no_hardcoded_cjk.py -q`.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- Tier 1: switching the active backend releases the previous engine's VRAM (unload() called once on switch); contract overridden for OmniVoice + all subprocess engines; ASR reporting is honest. (MM2-01..03)
|
||||
- Tier 2: services.model_lifecycle is the single lifecycle surface; system.py routers are thin delegations with unchanged response shapes; idle/timeout config flows through prefs.resolve with env precedence and no duplicated constants. (MM2-04..05)
|
||||
- Tier 3: cooldown dict bounded; weight validation is per-role; sidecars self-report real VRAM; cache-fallback logs its reason. (MM2-06..09)
|
||||
- All listed pytest selections pass; no on-disk model-state change; no new dep; cross-platform default behavior identical (VRAM numbers degrade gracefully on MPS/CPU).
|
||||
</success_criteria>
|
||||
|
||||
<risks>
|
||||
- **unload() correctness for the shared OmniVoice singleton (MM2-01/02):** OmniVoiceBackend shares model_manager's `model` global. unload() must release the shared singleton, but the idle_worker() + offload_tts_for_asr() paths also touch it. Risk: a switch during an in-flight ASR offload double-frees or races. Mitigation: take mm._model_lock around the shared release in unload(); guard on `mm.model is not None`; keep unload best-effort (try/except) so it can never wedge a switch. Add the idempotency test (Task 4).
|
||||
- **Response-shape drift (MM2-04):** Moving /model/loaded + /model/unload bodies into the facade risks changing the JSON the frontend depends on (hooks.ts, flush dropdown). Mitigation: move verbatim first, assert shapes in test_model_lifecycle.py, only then layer MM2-03/08 improvements.
|
||||
- **Sidecar protocol change (MM2-08):** Adding vram_mb to pong touches the parent/child wire format. Older sidecars (a long-running session mid-upgrade) won't send it. Mitigation: treat vram_mb as optional in the parent (`reply.get("vram_mb", <last-known or 0>)`); never require it; never break the existing pong==success check.
|
||||
- **prefs.resolve at import time (MM2-05):** Resolving timeouts at import freezes them; the reaper loop reads SIDECAR_IDLE_TIMEOUT_S. Mitigation: resolve lazily inside the reaper tick / idle_worker tick (cheap) so a settings change takes effect, while keeping the import-time default for the start-guard.
|
||||
- **Per-role weight floor (MM2-07):** Lowering the ONNX floor could let a genuinely-truncated ONNX through (#352 regression). Mitigation: keep a non-zero floor for every role (e.g. ONNX floor still >> a partial KB), key on extension, and keep the "largest file" heuristic — only the threshold becomes role-aware.
|
||||
- **Scope creep into perf:** GPU-pool sizing and torch.compile are explicitly out of scope. If an executor is tempted, stop — those regress #278/#315.
|
||||
</risks>
|
||||
|
||||
<output>
|
||||
Write `.planning/quick/260613-mm2-clean-model-management-v2/260613-mm2-SUMMARY.md` when done (per wave or once at the end), documenting: which engines got real unload() overrides vs intentional no-ops (for the future CI gate), the exact response shapes preserved on the two endpoints, the per-role weight-validation thresholds chosen, and the pytest output for the verification selection. Note any decision an executor made where the plan said "use judgment."
|
||||
|
||||
Docs-sync check (CLAUDE.md hard rule): this is internal lifecycle cleanup with no user-facing install/Docker/versioning change, so no README/docs edit is expected. If MM2-05 surfaces the new idle-timeout settings keys in the Settings UI, add them to the relevant settings doc in the same PR.
|
||||
</output>
|
||||
@@ -1,32 +0,0 @@
|
||||
# SUMMARY — model-management v2 cleanup (mm2)
|
||||
|
||||
**Date:** 2026-06-13 · **Scope:** all 3 tiers (MM2-01..09). Backend-only; no frontend, no on-disk model-state change, no new deps.
|
||||
|
||||
## Tier 1 — correctness
|
||||
- **MM2-01 (VRAM leak on engine switch):** `get_active_tts_backend()` now caches one instance per configured backend id and calls the outgoing engine's `unload()` before switching. Added `reset_active_backend()` for shutdown/tests. The `model=` OmniVoice fast-path still returns a fresh view over the shared singleton (no double-load) but a switch *away from* another engine still releases it. `tts_backend.py`.
|
||||
- **MM2-02 (per-engine unload()):** `OmniVoiceBackend.unload()` drops the local ref + the shared `model_manager.model` singleton + `free_vram()` (idempotent, preload-safe, best-effort — no async lock from the sync path). `SubprocessBackend.unload()` routes to `unload_sidecar(self.id)` (busy sidecars skipped) and is inherited by every subprocess engine.
|
||||
- **MM2-03 (honest ASR row):** `/model/loaded` ASR row now reports the pipe's actual device and carries a `note: "released with the TTS model"` so the disabled unload button is explained rather than silent.
|
||||
|
||||
## Tier 2 — single lifecycle surface
|
||||
- **MM2-04 (`services/model_lifecycle.py`):** new facade owns `list_loaded()` / `unload(id)` / `unload_all()` / `free_vram()` across in-process TTS+ASR, diarization, and sidecars. `system.py` `/model/loaded` + `/model/unload` are now thin delegations; **response shapes preserved exactly** (`{models,count}`, `{unloaded,success,...}`, 400 on unknown id) — frontend untouched.
|
||||
- **MM2-05 (unified idle config):** removed the duplicated `_IDLE_TIMEOUT_SECONDS`; the in-process idle timeout and the sidecar idle timeout both resolve per-tick via `prefs.resolve(... env=...)` (env wins, settings can tune without restart). New keys: `idle_timeout_seconds` (`OMNIVOICE_IDLE_TIMEOUT_S`), `sidecar_idle_timeout_seconds` (`OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S`). `<=0` still disables sidecar reaping.
|
||||
|
||||
## Tier 3 — robustness & observability
|
||||
- **MM2-06 (bounded cooldowns):** `_install_cooldowns` is swept (TTL 1h) on each install check and cleared on success — can no longer grow unbounded.
|
||||
- **MM2-07 (per-role weight floor):** `_validate_snapshot_has_weights` uses per-extension floors (tensor formats keep 5 MB; `.onnx` floor 64 KB) **OR** the original ≥5 MB catch — strictly more lenient, so a small-but-complete ONNX model is no longer false-flagged as truncated while a 0/KB partial is still rejected (#352 intact).
|
||||
- **MM2-08 (sidecar VRAM self-report):** the parent can't see a child's VRAM, so the GPU sidecar (`engines/indextts`) now reports `vram_mb` in its `pong` (CUDA/MPS-aware, 0 on CPU); the parent stashes the last-known figure and `list_live_sidecars()` surfaces it. CPU/absent sidecars honestly report 0.
|
||||
- **MM2-09 (cache-fallback logging):** the `is_cached` `scan_cache_dir → on-disk` fallback now logs at WARNING with the exception type (was DEBUG/invisible) — the #117/#118 WinError-448 path is triagable from logs.
|
||||
|
||||
## Out of scope (as planned, not done)
|
||||
GPU-pool per-engine sizing (`_GPU_VRAM_PER_JOB_GB`) and torch.compile tuning — perf, not cleanup; risk regressing #278/#315.
|
||||
|
||||
## Verification
|
||||
- New `tests/test_mm2_lifecycle.py` — 15 tests (reuse/switch-unload/reset/idempotent-unload, facade list/unload/unknown/sidecars shapes + honest ASR, env-wins idle config, cooldown sweep, per-role weight floor ×3).
|
||||
- Affected existing: `test_engines.py`, `test_subprocess_reaper.py`, `test_model_load_timeout.py`, `test_model_manager_preload.py` — green (no regressions).
|
||||
- **Full suite: 1379 passed, 0 failed.** Live: facade endpoints return preserved shapes; engine switch calls the previous engine's `unload()` exactly once.
|
||||
|
||||
## Test placement note
|
||||
MM2 tests live at top-level `tests/` (not `tests/backend/`) on purpose: adding files under `tests/backend/` reorders collection and can expose a pre-existing `sys.modules`-isolation leak in other backend fixtures (the issue debugged in the FDL PR). Top-level placement keeps `tests/backend/` order identical.
|
||||
|
||||
## Docs-sync
|
||||
The new idle-timeout settings keys are internal env/prefs knobs with no UI surface, so no README/docs change is required by the docs-sync rule. If a future Settings panel exposes them, document there.
|
||||
-321
@@ -6,327 +6,6 @@ The format is loosely based on [Keep a Changelog](https://keepachangelog.com/).
|
||||
Versions track the desktop app (`tauri.conf.json` + `frontend/src-tauri/Cargo.toml`).
|
||||
The bundled TTS model package (`pyproject.toml`) is versioned independently.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
_Nothing yet — `main` is at v0.3.7 + 1 patch. New work lands here._
|
||||
|
||||
## [0.3.7] — 2026-06-20
|
||||
|
||||
A stabilization release. It tags the startup-crash fixes already on `main` (so
|
||||
users hitting "Can't reach the local backend" on v0.3.5/v0.3.6 only need to
|
||||
update), and clears the wave of issues reported on the 0.3.6 line across voice
|
||||
design, dubbing, transcription, install, and the Linux UI.
|
||||
|
||||
### Added
|
||||
|
||||
- **Two opt-in heavyweight TTS engines: MOSS-TTS-v1.5 (8B) and dots.tts (2B).**
|
||||
Both are zero-shot voice-cloning engines added per [#498](https://github.com/debpalash/OmniVoice-Studio/issues/498),
|
||||
running in their own isolated subprocess venv (each pins a `transformers`
|
||||
version that conflicts with the parent's `>=5.3` — MOSS `==5.0`, dots.tts
|
||||
`==4.57`) via the same dedicated-venv pattern as IndexTTS-2. Point
|
||||
`OMNIVOICE_MOSS_TTS_V15_DIR` / `OMNIVOICE_DOTS_TTS_DIR` at a local clone to
|
||||
enable. CUDA/CPU only — neither claims Apple-Silicon MPS; dots.tts upstream
|
||||
is Linux/macOS only (gated off on Windows). No change to the default install
|
||||
or its lockfile. See [docs/engines/moss-tts-v15.md](docs/engines/moss-tts-v15.md)
|
||||
and [docs/engines/dots-tts.md](docs/engines/dots-tts.md). (#498)
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Audio playback on Linux Firefox/Chrome and Android Chrome.** Two separate
|
||||
root causes both masquerade as "the play button doesn't work" on non-macOS
|
||||
browsers — and both are invisible when developing on macOS, which is why they
|
||||
shipped. (1) The backend served `.wav` / `.flac` with Python's default
|
||||
`audio/x-wav` / `audio/x-flac` (vendor-experimental, never IANA-registered);
|
||||
macOS CoreAudio MIME-sniffs leniently and plays anyway, but Linux FFmpeg and
|
||||
Android ExoPlayer strictly honor the declared type and prompt to download.
|
||||
Fixed by registering the canonical `audio/wav` / `audio/flac` types before
|
||||
any `StaticFiles` mount. (2) WaveSurfer's `AudioContext` is constructed at
|
||||
component-mount time — i.e. before any user gesture — so on Linux FF/Chrome
|
||||
and Android Chrome it stays `suspended`, `decodeAudioData` hangs, the
|
||||
`ready` event never fires, and the play button never enables. macOS
|
||||
Safari/Chrome auto-resume on first interaction. Fixed by patching
|
||||
`window.AudioContext` to track every instance and resuming them on the first
|
||||
`pointerdown` / `keydown` / `touchstart`, plus resuming inline on the play
|
||||
click itself. The MIME fix has a backend regression test; the unlock path
|
||||
has a Vitest unit test covering idempotency, post-unlock contexts, and
|
||||
error isolation. (#510)
|
||||
- **Voice Studio "Save design as profile" poisoned the profile with
|
||||
"[object Object]" and then 400'd every generation** ("Unsupported instruct
|
||||
items found in [object Object]"). The save passed the instruct *builder
|
||||
object* to the form instead of its string. Fixed at the source + defended with
|
||||
a coercion helper; the engine now tolerates the sentinel, and a migration
|
||||
heals already-saved profiles. (#550, #545, #542, #537, #530, #525)
|
||||
- **Profile / persona / consent endpoints 500'd with `no such column:
|
||||
consent_audio_path`** (and the same class for `kind`/`vd_states`/…) after an
|
||||
in-place upgrade. The alembic migration existed but couldn't always apply
|
||||
(stamped at a removed revision, or alembic not importable) and the failure was
|
||||
swallowed. The runtime schema now self-heals — it ADDs any missing additive
|
||||
column from the canonical schema on startup. (#552, #547)
|
||||
- **Generate / Settings / Clone buttons were missing / unpressable on Linux.**
|
||||
The UI-scale fix round-trips correctly on Chromium, but older WebKitGTK treats
|
||||
`zoom` as a layout no-op, leaving a ~23% black band that pushed the bottom CTAs
|
||||
off-screen. The shell now probes the engine and fills the window when `zoom`
|
||||
doesn't lay out. (#523, #524)
|
||||
- **The engine "Install" button 500'd with "No virtual environment found."**
|
||||
`uv pip install` now targets the running interpreter (`--python
|
||||
sys.executable`) instead of relying on a venv it couldn't auto-discover.
|
||||
(#529, #527)
|
||||
- **Transcription failed with "no segments" on GPUs without efficient float16.**
|
||||
Both CTranslate2 ASR backends now fall back float16 → int8 instead of crashing
|
||||
at model load; a transcribe stream can no longer close without a terminal
|
||||
error event; and an incomplete `transformers` install reports an actionable
|
||||
message instead of "Could not import module 'AutoFeatureExtractor'".
|
||||
(#551, #549, #516)
|
||||
- **Audiobook import 500'd** with `'AudiobookPlan' object has no attribute
|
||||
'chapter_count'` for every format (.txt/.md/.epub/.pdf). (#543)
|
||||
- **Windows: generated audio auto-played in a separate, un-closeable black
|
||||
window.** Renders now play in-app through the shared playback manager. (#532)
|
||||
- **Cryptic video-download errors** now carry actionable hints: an unsupported
|
||||
link shape ("paste a direct video page, not a share/feed link") vs a transient
|
||||
network drop ("just retry — the partial download was cleaned up"). (#554, #536)
|
||||
- **About → Version rendered blank in the web/Pinokio build** (no Tauri, backend
|
||||
idle); it now falls back to the build-time version.
|
||||
- **A relocated, copied, or restored backend venv ("No module named
|
||||
'encodings'") now self-heals** (rebuilds once) instead of failing on every
|
||||
launch.
|
||||
- **Non-English voices drifted to English / the wrong language.** A voice
|
||||
profile's stored language wasn't propagated into generation (a German
|
||||
archetype previewed in German but generated in English), the audiobook/longform
|
||||
synth hardcoded `language=None` (a non-English clone could flip language
|
||||
mid-render), and the duration estimator under-allocated frames for decomposed
|
||||
(NFD) diacritic text. The profile/request language is now threaded through both
|
||||
the single-shot and longform paths, and text is NFC-normalized. (#533, #505, #502)
|
||||
- The **"Can't reach the local backend" startup-crash wave** (pkg_resources
|
||||
#248, `scalar_fastapi` #307, exit-106 broken venv) was fixed in v0.3.6 — this
|
||||
release carries those fixes, so updating from v0.3.5/older resolves them.
|
||||
|
||||
## [0.3.6] — 2026-06-16
|
||||
|
||||
A large release (168 commits since v0.3.5). The headline is the **Longform
|
||||
suite** — produce full audiobooks and multi-voice stories from text, EPUB, or
|
||||
PDF — alongside a real **engine-routing** layer that tells you up front when an
|
||||
engine will fall back to CPU instead of finding out mid-synth. Dubbing,
|
||||
first-run, and install reliability all get a pass too.
|
||||
|
||||
### Added
|
||||
|
||||
- **Longform: Stories + Audiobook editors.** Two new tabs turn long text into
|
||||
finished audio. **Audiobook** takes a script (or imports plain text / EPUB /
|
||||
PDF), auto-splits it into chapters, and renders a chaptered `.m4b` with
|
||||
metadata, cover art, and per-chapter preview/resume. **Stories** is a
|
||||
multi-voice editor — assign a different voice per line, preview, and export
|
||||
the whole thing through the same server-side renderer. Both share one render
|
||||
core (loudness, metadata, cover art) and one live SSE progress stream, and
|
||||
you can convert a project between Story and Audiobook in place.
|
||||
(#402, #403, #404, #408, #409, #411, #412, #413, #426, #435, #436, #447)
|
||||
- **Longform: PDF & EPUB ingest.** "Import" on the Audiobook tab accepts EPUB
|
||||
and PDF (not just plain text) and auto-chapters the result, so an existing
|
||||
ebook becomes an audiobook without manual copy-paste. (#412, #459)
|
||||
- **Longform: two-pass loudnorm mastering.** Audiobook/Story exports now run a
|
||||
measure-then-normalize loudnorm pass for accurate ACX/podcast loudness
|
||||
targets. A slow or broken measure pass degrades gracefully to single-pass
|
||||
rather than aborting the render. (#449, #455)
|
||||
- **Longform: crash-resume.** An interrupted render is resumable without
|
||||
re-submitting the original input — the compiled plan is persisted to the job
|
||||
dir and finished chapters are reused, so a crash mid-book doesn't cost you the
|
||||
whole render. (#470)
|
||||
- **Longform: pronunciation control + SSML-lite prosody.** A per-render
|
||||
pronunciation lexicon (word respelling) plus an in-app pronunciation editor
|
||||
and markup reference, and inline prosody markers — `[slow]` / `[fast]` /
|
||||
`[emphasis]` / `[spell]` — for fine-grained delivery. (#419, #421, #422)
|
||||
- **Stories: global reading-speed control.** A toolbar slider (0.5–2.0×) sets
|
||||
one speed for every line that doesn't have its own per-line override; the
|
||||
per-line slider still wins. Persisted as a UI preference. (#415, #416)
|
||||
- **Unified LongformProject store.** Audiobook metadata, scripts, and prefs
|
||||
persist in a single project store (with a `v4→v5` migration), and finished
|
||||
books/stories now show up alongside other work in **Projects**. (#417, #443,
|
||||
#444)
|
||||
- **Portable personas (`.ovsvoice`).** Export any voice as a self-contained,
|
||||
fully-local persona bundle — identity, optional reference clip, consent
|
||||
attestation, SPDX license, and a watermarked preview — and import it back into
|
||||
another OmniVoice install. A privacy toggle ships a **preview-only** bundle so
|
||||
no raw recording of your voice has to travel. Verified-own-voice status can't
|
||||
be forged by hand-editing a bundle (real recording + consent text + attestation
|
||||
required). Legacy `.omnivoice` files still import. See
|
||||
[docs/persona-format.md](docs/persona-format.md). (#29)
|
||||
- **Engine routing — no more silent CPU fallback.** A host device probe and
|
||||
routing resolver now decide where each engine actually runs, and the verdict
|
||||
is surfaced before you hit Synthesize: the **Settings → Engines** picker shows
|
||||
a per-engine compatibility matrix, and **preflight** / **diagnose** report the
|
||||
active engine's GPU verdict (accelerated / caveat / CPU-fallback /
|
||||
unavailable). At synth time every TTS entry point (`/generate`,
|
||||
`/v1/audio/speech`) enforces the same routing — an engine that can't use this
|
||||
host's GPU returns an explicit error or an `X-OmniVoice-Routing` header instead
|
||||
of silently dropping to CPU or dying mid-synth. (#21)
|
||||
- **Diagnostics suite.** New self-check tooling for when something's wrong: a
|
||||
`/system/diagnose` report (and matching backend `--diagnose`), a persistent
|
||||
**error journal** surfaced in Settings, and a scrubbed **diagnostic bundle**
|
||||
(home dirs stripped to `~/`, no tokens/keys) you can attach to a bug report.
|
||||
Paired with structured GitHub **Issue Forms** (bug / install / feature) for
|
||||
cleaner reports. (#433, #456)
|
||||
- **Dubbing: multi-speaker per-speaker voice assignment.** When diarization
|
||||
detects multiple speakers, each segment is now bound to its speaker's cloned
|
||||
voice automatically instead of landing on "Default" and needing manual fixes;
|
||||
per-segment reference clips are still preferred for quality where present. Also
|
||||
adds an optional speaker-count hint for diarization. (#275, #486, #490)
|
||||
- **Dubbing: Smart Fit timing + second-pass QC.** A Smart Fit timing strategy
|
||||
(planner, fingerprints, per-segment video retime + drift absorption + fitted
|
||||
subtitles) plus a second-pass ASR QC that flags lines whose dub drifts from the
|
||||
target timing — wired into the dub editor UI. Includes a timeline segment
|
||||
editor (drag, snap-to-onset, keyboard a11y), speech-onset alignment, regional
|
||||
dialect targeting, and per-segment clone references. (#280, #347, #350, #369,
|
||||
#370, #458)
|
||||
- **Dubbing: dedicated Dub home.** A projects/history landing for dubbing with
|
||||
project rename. (#435)
|
||||
- **Voice Console workspace.** Clone and Design are consolidated into one Voice
|
||||
workspace with right-side panels, a shared waveform player, an identity recipe
|
||||
line / Active-voice card, and a free-text "describe your voice" field that maps
|
||||
natural language to design parameters. (#317, #374, #376, #378, #395, #396,
|
||||
#397)
|
||||
- **Unified first-run setup.** Nothing installs until you confirm a plan: pick an
|
||||
install mode (installed / portable), a storage location, and (on restricted
|
||||
networks) custom PyPI/HF/python-build-standalone mirrors — with a
|
||||
minimum-free-space gate before anything downloads. Followed by a guided
|
||||
studio-console wizard with platform-aware hints, resume reassurance, and
|
||||
download ETAs. (#286, #295, #297, #298)
|
||||
- **Dictation: local-LLM refinement.** Opt-in local-LLM cleanup of final
|
||||
transcripts (collapsing Whisper hallucination loops), available on both live
|
||||
dictation and the REST `/transcribe` path; plus opt-in NLMS acoustic echo
|
||||
cancellation for dictating over playback. Configure a remote LLM endpoint
|
||||
(Ollama / vLLM / LM Studio) in Settings. (#356, #357, #363, #399, #400, #457)
|
||||
- **Unlimited-length TTS + streaming.** Sentence-boundary chunking with
|
||||
crossfade removes the per-generation length cap, and a new sentence-by-sentence
|
||||
`/ws/tts` streams audio as it's produced. An inline `[pause Nms]` marker
|
||||
inserts measured silence in generated speech. (#276, #357, #358)
|
||||
- **MCP server v1.** OmniVoice mounts an MCP server on `/mcp` (with a stdio shim
|
||||
and per-agent voice binding) so it can act as a local TTS/STT provider for
|
||||
agentic pipelines. (#368)
|
||||
- **Remote-backend access.** Point the desktop UI at a remote backend URL with a
|
||||
bearer key (Tailscale-documented), and an opt-in Hugging Face token field in
|
||||
the setup flow. (#303, #364)
|
||||
- **"Fund Claude Max" support experience.** The donate page gets a real goal bar
|
||||
with a "Join N supporters" social-proof line and suggested amounts, plus Pip
|
||||
the mascot and a non-blocking "postcard" toast that appears only *after* a
|
||||
success (a finished dub, a saved clone, a longform export) — never on errors,
|
||||
setup, or first run — with escalating cooldowns and a one-click "don't ask
|
||||
again". (#494)
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Transcription/dubbing failed when ffmpeg wasn't on `PATH`** (notably on
|
||||
Windows). WhisperX now decodes audio through OmniVoice's own validated ffmpeg
|
||||
binary instead of a bare `PATH` lookup, so ASR works without a system ffmpeg
|
||||
install. (#479)
|
||||
- **Translation defaulted the source language to English.** Dubbing/translation
|
||||
now guesses the source language from the text instead of assuming `en`,
|
||||
fixing wrong-direction translations. (#478)
|
||||
- **Cinematic / LLM dubbing features failed out of the box** because `openai`
|
||||
wasn't bundled. The client is now a runtime dependency, so those paths work on
|
||||
a fresh install. (#484)
|
||||
- **`pkg_resources missing` install dead-end (#248).** The auto-repair ran
|
||||
`uv pip install setuptools`, which `uv` treated as a no-op when setuptools
|
||||
*metadata* was present but its files had been removed (commonly by Windows
|
||||
Defender quarantine or a partial extract). Both repair sites now use
|
||||
`--reinstall` to force re-extraction, and the error/hint text suggests the
|
||||
working command plus an antivirus-exclusion note. (#248)
|
||||
- **A stuck backend trapped users on a buttonless splash (#474).** The bootstrap
|
||||
splash now has a per-stage stall watchdog: if a non-terminal stage sits past
|
||||
its budget (20 min for dep install, 120 s otherwise), it flips to the failed
|
||||
state with actionable hints, the live log, and Retry / Clean-&-Retry — instead
|
||||
of polling forever with no way out. (#474)
|
||||
- **Changing the model-download location in Settings had no effect (#480).** The
|
||||
desktop launcher injected a stale models dir that overrode the per-user value,
|
||||
so new downloads kept going to the old folder and "Effective location" stayed
|
||||
wrong. The per-user env file now wins, so the in-app Settings path is
|
||||
authoritative. (#480)
|
||||
- **Backend crashed on app upgrade with a stale venv (#307).** Dependencies are
|
||||
now synced on upgrade, and a structurally broken venv self-heals instead of
|
||||
exiting `106`. `scalar_fastapi` is now optional so its absence can't break
|
||||
startup. (#307, #314)
|
||||
- **`/generate` ignored the selected TTS engine (#312)** and GGUF speech-control
|
||||
parameters weren't forwarded — both now honored. (#306, #312)
|
||||
- **TTS generation failed on some GPUs.** `torch.compile` failures now fall back
|
||||
to eager execution so generation never hard-fails on unsupported GPUs, and
|
||||
cudagraph-compiled inference is pinned to one dedicated thread to avoid
|
||||
crashes. (#278, #315)
|
||||
- **Re-dub ignored transcript edits (#281).** Fingerprints are canonicalized, the
|
||||
preview cache is busted, and the mux is atomic, so editing the transcript and
|
||||
re-dubbing actually reflects your changes. Translated subtitles now burn in
|
||||
correctly and subtitle save no longer throws a JSON error. (#281, #309)
|
||||
- **macOS: app wouldn't open without using Terminal.** Builds are now ad-hoc
|
||||
signed (with signing/notarization verification), so the app launches normally.
|
||||
(#290)
|
||||
- **macOS dictation auto-paste stole focus**; it now writes the clipboard
|
||||
natively without grabbing focus, and microphone-permission handling adds OS
|
||||
usage descriptions, a WebView grant handler, and an actionable denied-state UI.
|
||||
(#287, #323)
|
||||
- **Clone-reference transcription was broken** (it used a removed transformers
|
||||
pipeline); it now routes through the ASR registry. A crash-isolated
|
||||
faster-whisper subprocess backend keeps an ASR crash from taking down the app.
|
||||
(#308, #393)
|
||||
- **Realtime status probe hit a gated route.** It now probes the auth-exempt
|
||||
`/health` instead of the gated `/model/status`, and the UI polls the backend
|
||||
over HTTP before opening the WebSocket to avoid startup `ECONNREFUSED`. (#439,
|
||||
#450)
|
||||
- **Non-executable or unreachable engine binaries showed cryptic errors** — these
|
||||
now produce actionable messages. (#437, #438, #454, #466)
|
||||
- **Design-profile save was coupled to a TTS render (#476)**, so saving a profile
|
||||
needlessly triggered synthesis; the two are now decoupled. (#476)
|
||||
- **UI scale / black bands.** The app shell now scales via `transform: scale` and
|
||||
always fills the viewport, fixing the WebKitGTK black-band issue on Linux and
|
||||
cramped/black layouts at narrow widths — a permanent fix across platforms.
|
||||
(#445, #452)
|
||||
- **Clone popover/CTA clipping and a non-resizable textarea** are fixed, the
|
||||
WaveformPlayer no longer pauses itself on play or ignores clicks, and several
|
||||
layout/history-display issues (phantom sidebar gap, title clamping, flicker)
|
||||
are cleaned up. (#379, #384, #398, #481)
|
||||
- **Windows: `desktop-prod` now runs from cmd/PowerShell** via a cross-platform
|
||||
launcher, `tqdm` is disabled on non-TTY to avoid an `OSError`, and ffmpeg
|
||||
validation guards against `WinError 193`. (#282, #305, #377)
|
||||
- **MLX import hardened** against PyInstaller dylib failures, with a proper
|
||||
platform gate so it's only loaded where it works. (#390)
|
||||
|
||||
### Changed
|
||||
|
||||
- **Restricted-network support.** A Hugging Face mirror (`HF_ENDPOINT`) setting,
|
||||
custom PyPI / HF / python-build-standalone mirrors in first-run setup, and
|
||||
region presets help installs complete behind restrictive networks. (#286, #391)
|
||||
- **Engine memory management.** Subprocess-engine sidecars now unload on demand
|
||||
and idle-reap to free VRAM. (#401, #406)
|
||||
- **Faster, more accurate model downloads** via a Xet fast path with accurate
|
||||
progress reporting, plus a model-management cleanup pass. (#424, #428)
|
||||
- **Voice profiles unified** under one model with a `kind` discriminator and
|
||||
stored design params, and consent-locked profiles (`verified_own_voice` +
|
||||
spoken-consent flow). (#354, #376)
|
||||
- **Updater** preview channel now offers the newest build across channels, and
|
||||
preview versions carry an MSI-legal numeric pre-release stamp. (#293, #326)
|
||||
- **Performance.** Voice-clone prompt embeddings are cached, and dub retime
|
||||
batches seek to their window instead of decoding from frame 0. (#387, #427)
|
||||
|
||||
### License
|
||||
|
||||
- **Relicensed from FSL-1.1-ALv2 to AGPL-3.0 (open-core).** The project is now
|
||||
under the GNU Affero General Public License v3, with a paid commercial license
|
||||
retained for proprietary/closed-source use without AGPL obligations. The
|
||||
bundled `omnivoice/` TTS model package stays Apache-2.0 upstream
|
||||
(AGPL-compatible). Manifests declare `AGPL-3.0-only`; the in-app Commercial
|
||||
License copy and README are updated, and the old "converts to Apache 2.0 after
|
||||
two years" FAQ is removed. In-app commercial-license strings are translated
|
||||
across all 20 locales. (#292)
|
||||
|
||||
### CI
|
||||
|
||||
- **macOS Intel (x86_64) build target reinstated** on `macos-15-intel`, so Intel
|
||||
Mac users get installers again. (#342)
|
||||
- **Docker Hub publishing.** Images now also publish to Docker Hub
|
||||
(`palashdeb/omnivoice-studio`), with the Docker Hub overview maintained in-repo
|
||||
and auto-synced from `main` (sync is non-fatal so it can't redden a build).
|
||||
(#375, #410, #414)
|
||||
- **Docs-drift guard.** A daily job compares the canonical feature inventory
|
||||
against README / docs / registries to catch stale docs. (#353)
|
||||
- **Security scans never cancel on `main`,** so merge trains no longer leave red
|
||||
✗ on intermediate commits. (#340)
|
||||
|
||||
## [0.3.5] — 2026-06-03
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
|
||||
**OmniVoice Studio**
|
||||
|
||||
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. It's an active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord. The latest stable release is **v0.3.5**; `main` rolls ahead at **v0.3.6** (latest release + 1 patch — see the Versioning rule below).
|
||||
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. Today it's a v0.2.7 active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord.
|
||||
|
||||
**Core Value:** **A first-run that actually works.** A user who downloads the installer (or clones the repo) should reach a working voice-cloning or dubbing output without hitting a wall — and when something does go wrong, the error or docs should tell them exactly what to do.
|
||||
|
||||
@@ -16,7 +16,7 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
|
||||
- **Default features must work on every platform (strict rule, 2026-05-20):** A feature that ships in default mode — out-of-the-box, no user customization, no opt-in toggle — must behave identically on macOS, Windows, and Linux. Platform-specific *implementation code* is allowed for OS APIs / shells / packaging, but the user-visible *default behavior* cannot diverge. Platform-only features (e.g., a macOS-only global shortcut, a Windows-only path picker) must go behind explicit user opt-in: Settings toggle, env var, or CLI flag. When a default doesn't work on a platform, that's a P0 bug — either fix it on the missing platform or move it behind opt-in. No third option.
|
||||
- **Backward-compatible project data**: Existing `omnivoice_data/` (user voices, projects, settings) must keep working without manual migration. Any DB schema change goes through alembic with a tested upgrade path.
|
||||
- **Local-first guarantee preserved**: Auto bug reporting (new addition) must be **opt-in**, must submit only to GitHub Issues (no third-party telemetry endpoint), and the app must remain fully functional with reporting disabled. No required cloud calls, accounts, or API keys.
|
||||
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** the v0.3.x line has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main; the owner tags a patch (`v0.3.Z`) from main whenever the current state is worth cutting. No `-rc` tags. No phased release. No `v0.4` deferrals while the v0.3.x line is open — every open issue and every open community PR gets absorbed into the v0.3.x line or explicitly declined. Users follow `main` for previews; users wanting stable stay on the latest tagged release (currently **v0.3.5**). ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
|
||||
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** v0.3.0 has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main. Tag `v0.3.0` once when the user calls "actually useful" — a qualitative bar, not a checklist. No `v0.3.0-rc1`. No phased release. No `v0.4` deferrals while v0.3.0 is open — every open issue and every open community PR gets absorbed into the v0.3.0 line or explicitly declined. Users follow `main` for previews; users wanting stable stay on `v0.2.7`. ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
|
||||
<!-- GSD:project-end -->
|
||||
|
||||
<!-- GSD:stack-start source:research/STACK.md -->
|
||||
@@ -190,22 +190,10 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
|
||||
<!-- GSD:conventions-start source:CONVENTIONS.md -->
|
||||
## Conventions
|
||||
|
||||
**Versioning (hard rule, owner-set 2026-06-11; single-source 2026-06-16):** main is always **latest release + 1 patch**. **`frontend/package.json` is the SINGLE SOURCE OF TRUTH for the app version** — vite injects `__APP_VERSION__` from it (first-run footer + every auto bug report), and `frontend/src-tauri/tauri.conf.json` reads its bundle version from it (`"version": "../package.json"`, so the MSI/dmg/updater version can't drift from the UI). Three toolchain-required **mirrors** are kept equal to it and bumped in lockstep — `frontend/src-tauri/Cargo.toml` + `pyproject.toml` (cargo/uv need a literal) and `backend/core/version.py`'s `_FALLBACK_VERSION` (the frozen-backend last resort; at runtime the backend reads its version from package metadata via `importlib.metadata`, which `backend.spec`'s `copy_metadata('omnivoice')` makes work in the frozen build too). Never hand-edit any mirror or re-hardcode a literal in `tauri.conf.json`. Guarded by `tests/test_app_version.py` (`test_all_version_files_in_lockstep` + `test_tauri_version_derives_from_package_json`). The moment `vX.Y.Z` is released, bump `package.json` (+ the mirrors) to `X.Y.(Z+1)`. Consequences:
|
||||
- Every PR and preview build identifies as the **next** version. Preview builds stamp `X.Y.(Z+1)-N` (run number), which semver-sorts **above** the last stable `X.Y.Z` — the updater ordering is natural, no comparator tricks needed.
|
||||
- Releasing = tag `vX.Y.(Z+1)` from main (version files already match), then immediately bump main to `X.Y.(Z+2)`. The post-release bump is automated by the `version-bump` job in release.yml; if it fails, do it manually in the same day.
|
||||
- Docker: `ghcr.io/debpalash/omnivoice-studio:latest` = **main** (rolling preview); `:X.Y.Z` + `:X.Y` + `:stable` = tagged releases. `:latest` is the preview channel by design — stable users pin `:stable` or a version tag.
|
||||
- Do not bump minor/major or invent RCs/codenames without the owner asking. No "defer to next version" labels — scope is absorbed or declined, never re-versioned.
|
||||
|
||||
**Docs-sync (hard rule, owner-set 2026-06-11):** any change that alters something these docs describe — README.md, CONTRIBUTING.md, SECURITY.md, SUPPORT.md, LICENSE, or `docs/**` (install flows, Docker tag semantics, platform support, versioning/release behavior, review process, supported versions) — must update those docs **in the same PR** as the change. If a doc impact is discovered after merge, the docs fix is the immediate next commit, not backlog. Stale docs are treated as bugs.
|
||||
|
||||
**Release notes / changelog (hard rule, owner-set 2026-06-16):** every tagged release gets a **high-quality, user-facing `## [X.Y.Z] — DATE` section in `CHANGELOG.md`** before (or in the same hour as) the tag — never the "Auto-generated release for vX.Y.Z…" fallback. `release.yml` extracts that section verbatim as the GitHub Release body (the `Extract CHANGELOG section for tag` step), so a missing/empty section ships a bare release. Quality bar = the existing house style: a one-paragraph headline, then `### Added` / `### Fixed` / `### Changed` / `### License` / `### CI` subsections; each entry is a **bold one-line lead** (what the user gets), 1–3 lines of plain-English why, and the `(#NNN)` issue/PR ref — grouped by theme, written for users, **not** raw commit dumps. This applies to **preview builds too**: preview release notes summarize what's new on `main` since the last stable, in the same style. Workflow: as features merge, keep `## [Unreleased]` current; at release time rename it to the version + date. If a release was already cut with the fallback body, the next action is to backfill `CHANGELOG.md` **and** `gh release edit <tag>` the live body — not backlog.
|
||||
**Versioning (hard rule):** Everything ships on `v0.3.0`. Never mention, suggest, or label anything with a version bump — no v0.4, no RCs, no "defer to next version", no future-version labels — unless the user explicitly asks to bump. Zero unprompted version chatter.
|
||||
|
||||
**Localization (hard rule):** No hardcoded non-English (CJK) **user-facing text** anywhere in the codebase except the translation layer (`frontend/src/i18n/`). All UI strings go through i18n (`t('...')` keys in `locales/*.json`); native language names live in `i18n/index.ts` (`LANGUAGES`). Functional CJK is allowed and tracked via the allowlist in `tests/test_no_hardcoded_cjk.py` — text-processing regexes, model/engine vocabulary & identifiers (e.g. CosyVoice speaker IDs), localized error matching, demo/eval data, and test fixtures. CI fails on any hardcoded CJK outside the allowlist; to add legitimate functional CJK, extend `_ALLOWED_FILES` there with a justification.
|
||||
|
||||
**Fix quality (hard rule, owner-set 2026-06-16):** Fix issues *properly* and future-maintenance-proof — don't stop at the symptom. Root-cause fully, fix the whole **class** of the bug (not just the one reported instance), add a fail-before/pass-after regression test, and harden against recurrence (e.g. if a lockfile drift only fails in Docker, also make CI catch it). Go the extra mile where it durably pays off. Be token-efficient about it — extra **effort**, not extra **verbosity**: no padding, no redundant re-checks, the smallest correct change that is also recurrence-proof. Don't be shy to spend the effort a proper fix needs; do be shy about wasting tokens.
|
||||
|
||||
**Keep main green (hard rule, owner-set 2026-06-16):** A merge must **never break `main`'s CI**. Before a change lands, verify the *full* CI matrix would pass — every workflow in `.github/workflows/` **and** `deploy/Dockerfile`, not only the checks you happened to run. Dependency / lockfile / config changes must be validated against **all** consumers. Specifically: `frontend/` is a bun **workspace monorepo** — the lockfile is the repo-root `bun.lock`, and `deploy/Dockerfile` runs `bun install --frozen-lockfile`, so any `frontend/package.json` change requires regenerating root `bun.lock` and confirming `bun install --frozen-lockfile` passes (plain `bun install` in `ci.yml` silently tolerates drift, so CI-green ≠ Docker-green). Likewise re-check CodeQL/Security on code changes and the Tauri `cargo` build on Rust/dep changes.
|
||||
|
||||
Other conventions not yet established. Will populate as patterns emerge during development.
|
||||
<!-- GSD:conventions-end -->
|
||||
|
||||
|
||||
+1
-75
@@ -42,22 +42,11 @@ This starts both services:
|
||||
### Desktop App (Tauri)
|
||||
|
||||
```bash
|
||||
bun run desktop # dev: hot-reload Tauri shell + backend
|
||||
bun run desktop-prod # production: builds, bundles the backend, then launches
|
||||
bun run desktop
|
||||
```
|
||||
|
||||
Both run `uv sync` first (so the Python backend env is set up) and start the
|
||||
backend automatically — you do **not** start it separately. Use the exact script
|
||||
names: there is no `desktop=prod` (note the **hyphen** in `desktop-prod`).
|
||||
`desktop-prod` is Windows-aware (auto-detects bash/git; see `scripts/desktop-prod.mjs`).
|
||||
|
||||
Requires [Rust](https://rustup.rs/) and platform-specific Tauri dependencies — see the [Tauri prerequisites](https://v2.tauri.app/start/prerequisites/).
|
||||
|
||||
If the app opens but stays on the **setup splash with no buttons**, the Python
|
||||
backend didn't finish starting — the splash surfaces the stall reason, a log
|
||||
panel, and a **Retry** button (and Settings → Logs → Backend has the full trace).
|
||||
The most common from-source cause is `uv` or Python not being on your PATH.
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
@@ -203,69 +192,6 @@ cd frontend/src-tauri && cargo check
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## What code review looks like
|
||||
|
||||
Every PR is reviewed by two AI reviewers before a human looks at it:
|
||||
|
||||
- **CodeRabbit** posts a walkthrough (with a sequence diagram, and an ASCII
|
||||
before/after sketch for UI changes), inline findings, and warning-mode
|
||||
pre-merge checks against the project's hard rules.
|
||||
- **Greptile** reviews with the same project rubrics and learns from 👍/👎
|
||||
reactions on its comments — react to train it.
|
||||
|
||||
Both are advisory, not gating: CI and the maintainer's approval decide. Don't
|
||||
be surprised by detailed bot comments minutes after you open a PR — address
|
||||
what's right, push back (in a reply) on what's wrong.
|
||||
|
||||
**Commit & PR conventions:** conventional-commit style with a scope
|
||||
(`fix(dub): …`, `feat(setup): …`) and link the issue (`Closes #N` / `Refs #N`)
|
||||
in the title or body.
|
||||
|
||||
## Quality gates your PR must pass
|
||||
|
||||
- **Cross-platform parity (hard rule):** anything that ships in default mode
|
||||
must behave identically on macOS, Windows, and Linux. Platform-specific
|
||||
*implementation* is fine; platform-divergent *default behavior* is a P0.
|
||||
Platform-only features go behind an explicit opt-in (Settings toggle, env
|
||||
var, or CLI flag).
|
||||
- **i18n — all 21 locales (hard rule):** every user-facing string goes through
|
||||
`t('...')` and the key must exist in **all 21** files under
|
||||
`frontend/src/i18n/locales/`. Translate; don't copy English into non-English
|
||||
locales. CI fails on hardcoded CJK outside the allowlist in
|
||||
`tests/test_no_hardcoded_cjk.py` (extend `_ALLOWED_FILES` with a
|
||||
justification for legitimate functional CJK).
|
||||
- **DB schema changes** go through an alembic migration with a tested upgrade
|
||||
path — existing `omnivoice_data/` must keep working with no manual steps.
|
||||
- **Engine back-compat:** already-installed engines (model weights on disk)
|
||||
must not require reinstall or re-download.
|
||||
- **Local-first:** no new outbound calls except GitHub Issues (opt-in
|
||||
reporting) and HuggingFace model downloads. Never log or persist secrets or
|
||||
absolute home paths.
|
||||
- **Security posture:** the backend serves loopback HTTP — treat every
|
||||
query/path/form parameter as hostile. User-chosen filesystem destinations
|
||||
are authorized in the Tauri process (save dialog), never via HTTP params.
|
||||
|
||||
## Contribution licensing
|
||||
|
||||
OmniVoice Studio is **AGPL-3.0-only**, and the maintainer also offers a
|
||||
**commercial license** (see [LICENSE](LICENSE)). By submitting a contribution
|
||||
you agree that:
|
||||
|
||||
1. you have the right to submit it (your own work, or compatibly licensed);
|
||||
2. it is licensed to the project under **AGPL-3.0**; and
|
||||
3. you grant the project maintainer a perpetual, worldwide, non-exclusive
|
||||
right to also distribute your contribution under the project's commercial
|
||||
license terms.
|
||||
|
||||
This inbound grant is what keeps the dual-license model viable. If you can't
|
||||
agree to (3) for a particular contribution, say so in the PR and we'll discuss
|
||||
before merging. Adding a `Signed-off-by:` line (DCO) to your commits is
|
||||
appreciated but not required.
|
||||
|
||||
---
|
||||
|
||||
## Need Help?
|
||||
|
||||
- **Stuck on setup?** Ask in [Discord #help](https://discord.gg/bzQavDfVV9)
|
||||
|
||||
@@ -4,26 +4,33 @@
|
||||
<h3>The open-source ElevenLabs alternative.</h3>
|
||||
<p>Real-time dictation, zero-shot voice cloning, and cinematic video dubbing — all on your desktop.<br/>Open-source, no API keys, fully local. <b>646 languages.</b></p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/stargazers"><img src="https://img.shields.io/github/stars/debpalash/OmniVoice-Studio?style=flat-square&color=f59e0b" alt="Stars" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/OmniVoice-Studio?style=flat-square&color=10b981" alt="Release" /></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="License" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/issues"><img src="https://img.shields.io/github/issues/debpalash/OmniVoice-Studio?style=flat-square&color=ef4444" alt="Issues" /></a>
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Discord-Join_Community-5865F2?style=flat-square&logo=discord&logoColor=white" alt="Discord" /></a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="#quickstart">Quickstart</a> ·
|
||||
<a href="#features">Features</a> ·
|
||||
<a href="#why-ovs">Why OVS</a> ·
|
||||
<a href="#why-omnivoice-studio">Why OmniVoice Studio?</a> ·
|
||||
<a href="#tts-engines">TTS Engines</a> ·
|
||||
<a href="#asr-engines">ASR Engines</a> ·
|
||||
<a href="#sponsor--donate">Donate</a> ·
|
||||
<a href="#contributing">Contributing</a> ·
|
||||
<a href="https://discord.gg/bzQavDfVV9">Discord</a> ·
|
||||
<a href="README_CN.md"><strong>简体中文</strong></a>
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/stargazers"><img src="https://img.shields.io/github/stars/debpalash/OmniVoice-Studio?style=flat-square&color=f59e0b" alt="Stars" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/OmniVoice-Studio?style=flat-square&color=10b981" alt="Release" /></a>
|
||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="License" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/issues"><img src="https://img.shields.io/github/issues/debpalash/OmniVoice-Studio?style=flat-square&color=ef4444" alt="Issues" /></a>
|
||||
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Discord-Join_Community-5865F2?style=flat-square&logo=discord&logoColor=white" alt="Discord" /></a>
|
||||
<a href="https://ko-fi.com/debpalash"><img src="https://img.shields.io/badge/Ko--fi-Support_Us-FF5E5B?style=flat-square&logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
|
||||
<a href="https://github.com/sponsors/debpalash"><img src="https://img.shields.io/badge/GitHub-Sponsor-ff69b4?style=flat-square&logo=github&logoColor=white" alt="GitHub Sponsors" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_aarch64.dmg"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_x64_en-US.msi"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_amd64.AppImage"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_amd64.deb"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
|
||||
</p>
|
||||
<p>
|
||||
<sub><b>macOS:</b> first launch needs a one-time approval — right-click → <b>Open</b> (or System Settings → Privacy & Security → <b>"Open Anyway"</b> on macOS 15). No Terminal needed. <a href="docs/install/macos.md#gatekeeper-quarantine">Why?</a></sub>
|
||||
</p>
|
||||
</div>
|
||||
|
||||
@@ -50,28 +57,20 @@
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td align="center" width="25%">
|
||||
<td align="center" width="33%">
|
||||
<h3>🎙️ Voice Cloning</h3>
|
||||
<p>3-second clip → mirror any voice.<br/><b>646 languages</b>, zero-shot.</p>
|
||||
</td>
|
||||
<td align="center" width="25%">
|
||||
<td align="center" width="33%">
|
||||
<h3>🎨 Voice Design</h3>
|
||||
<p>Gender, age, accent, pitch, speed,<br/>emotion, dialect — <b>dial it in</b>.</p>
|
||||
</td>
|
||||
<td align="center" width="25%">
|
||||
<td align="center" width="33%">
|
||||
<h3>🎬 Video Dubbing</h3>
|
||||
<p>YouTube URL or file → transcribe →<br/>translate → re-voice → <b>MP4</b>.</p>
|
||||
</td>
|
||||
<td align="center" width="25%">
|
||||
<h3>📖 Audiobook Editor</h3>
|
||||
<p>Import text, EPUB, or PDF. Auto-chapter,<br/>loudnorm, metadata. Export <b>.m4b</b>.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
<h3>🎭 Stories</h3>
|
||||
<p>Multi-voice editor. Assign voices<br/>per-line, preview, <b>export full cast</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>⌨️ Dictation Widget</h3>
|
||||
<p><code>⌘+⇧+Space</code> from <b>any app</b>.<br/>Transcribes, auto-pastes, disappears.</p>
|
||||
@@ -98,10 +97,6 @@
|
||||
<h3>🛡️ AI Watermark</h3>
|
||||
<p>AudioSeal (Meta). <b>Invisible</b>,<br/>survives compression.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🔬 Diagnostics</h3>
|
||||
<p>Self-check, error journal,<br/>scrubbed <b>diagnostic bundle</b>.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
@@ -114,29 +109,7 @@
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🧩 Extensible</h3>
|
||||
<p>Subclass <code>TTSbackend</code>,<br/>add any engine in <b>~50 lines</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🧭 Engine Routing</h3>
|
||||
<p>Preflight GPU check per engine.<br/><b>No silent CPU fallback</b>.</p>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center" valign="top">
|
||||
<h3>🎒 Portable Personas</h3>
|
||||
<p>Export voices as <code>.ovsvoice</code><br/>bundles — identity + <b>watermark</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>♾️ Unlimited TTS</h3>
|
||||
<p>Sentence-chunked generation.<br/><b>No length cap</b>. Streaming via WS.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🌐 Remote Backend</h3>
|
||||
<p>Point UI at a remote server.<br/>Tailscale-friendly. <b>Bearer auth</b>.</p>
|
||||
</td>
|
||||
<td align="center" valign="top">
|
||||
<h3>🧠 Dictation + LLM</h3>
|
||||
<p>Local LLM cleanup of transcripts.<br/>Optional echo <b>cancellation</b>.</p>
|
||||
<p>Subclass <code>TTSBackend</code>,<br/>add any engine in <b>~50 lines</b>.</p>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
@@ -145,21 +118,12 @@
|
||||
|
||||
## Quickstart
|
||||
|
||||
<div align="center">
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
|
||||
<br/>
|
||||
<sub><b>macOS:</b> first launch needs a one-time approval — right-click → <b>Open</b> (or System Settings → Privacy & Security → <b>"Open Anyway"</b> on macOS 15). No Terminal needed. <a href="docs/install/macos.md#gatekeeper-quarantine">Why?</a></sub>
|
||||
</div>
|
||||
|
||||
Per-OS install guides — pick yours and follow it end-to-end:
|
||||
|
||||
- **macOS** — [docs/install/macos.md](docs/install/macos.md)
|
||||
- **Windows** — [docs/install/windows.md](docs/install/windows.md)
|
||||
- **Linux** — [docs/install/linux.md](docs/install/linux.md)
|
||||
- **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
|
||||
- **Docker** — [docs/install/docker.md](docs/install/docker.md)
|
||||
|
||||
Stuck? Run the built-in self-check first — **Settings → About → "Run
|
||||
self-check"** in the app, or `uv run python backend/main.py --diagnose` from
|
||||
@@ -172,9 +136,7 @@ bundle"** packages scrubbed logs + the self-check report for bug reports.
|
||||
For Hugging Face token setup, see
|
||||
[docs/setup/huggingface-token.md](docs/setup/huggingface-token.md). For
|
||||
diarization-specific gating, see
|
||||
[docs/features/diarization.md](docs/features/diarization.md). For download
|
||||
speed, the ⚡ fast-download (Xet) status, and restricted-network / mirror
|
||||
options, see [docs/downloading-models.md](docs/downloading-models.md).
|
||||
[docs/features/diarization.md](docs/features/diarization.md).
|
||||
|
||||
## Screenshots
|
||||
|
||||
@@ -226,7 +188,7 @@ options, see [docs/downloading-models.md](docs/downloading-models.md).
|
||||
|
||||
---
|
||||
|
||||
## Why OVS?
|
||||
## Why OmniVoice Studio?
|
||||
|
||||
ElevenLabs charges **$5–$330/mo** and processes your audio on their servers. OmniVoice Studio runs **on your hardware, with no usage limits.**
|
||||
|
||||
@@ -235,17 +197,12 @@ ElevenLabs charges **$5–$330/mo** and processes your audio on their servers. O
|
||||
| **Pricing** | $5–$330/mo, per-character billing | Free & open-source (AGPL-3.0) · [Commercial license](#license) for proprietary use |
|
||||
| **Voice Cloning** | ✅ 3s clip | ✅ 3s clip, zero-shot |
|
||||
| **Voice Design** | ✅ Gender, age | ✅ Gender, age, accent, pitch, style, dialect |
|
||||
| **Audiobook / Stories** | ❌ | ✅ Full audiobook editor + multi-voice stories (EPUB/PDF import, .m4b export) |
|
||||
| **Languages** | 32 | **646** |
|
||||
| **Video Dubbing** | ✅ Cloud-only | ✅ Fully local |
|
||||
| **Data Privacy** | Audio sent to cloud | **Nothing leaves your machine** |
|
||||
| **API Keys** | Required | Not needed |
|
||||
| **GPU Support** | N/A (cloud) | CUDA · Apple Silicon · ROCm · CPU |
|
||||
| **Desktop App** | ❌ | ✅ macOS · Windows · Linux |
|
||||
| **TTS Engines** | 1 | **11** (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, IndexTTS 2, OmniVoice GGUF, Supertonic 3) |
|
||||
| **ASR Engines** | 1 | **8** (WhisperX, Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet, Moonshine, FunASR, isolated Faster-Whisper) |
|
||||
| **MCP Server** | ❌ | ✅ Use from Claude, Cursor, any MCP client |
|
||||
| **Self-check** | ❌ | ✅ Diagnostics suite, error journal, scrubbed debug bundles |
|
||||
| **Customizable** | ❌ Closed | ✅ Fork it, extend it, ship it |
|
||||
|
||||
OmniVoice Studio gives you professional-grade AI tools without the subscription or the cloud.
|
||||
@@ -281,21 +238,12 @@ OmniVoice ships a multi-engine TTS backend. The default engine (OmniVoice) is al
|
||||
|--------|:---------:|:-----:|:--------:|:-----:|:---------:|:-------:|:-------:|
|
||||
| **OmniVoice** (default) | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Built-in |
|
||||
| **CosyVoice 3** | 9 + 18 dialects | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **GPT-SoVITS** | 5 | ✅ | — | ✅ CUDA/CPU | — | ✅ CUDA/CPU | MIT |
|
||||
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MOSS-TTS-Nano** | 20 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **KittenTTS** | English | — | — | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
|
||||
| **MLX-Audio** (Kokoro, Qwen3-TTS, CSM, Dia, …) | Multi | Varies | Varies | ❌ | ✅ Native | ❌ | Varies |
|
||||
| **Sherpa-ONNX** | 20+ | — | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **IndexTTS 2** ⚡ | Multi | ✅ | — | ✅ CUDA | — | ✅ CUDA | Apache-2.0 |
|
||||
| **OmniVoice GGUF** ⚡ | 600+ | ✅ | ✅ | ✅ CPU | ✅ CPU | ✅ CPU | Built-in |
|
||||
| **Supertonic 3** ⚡ | 31 | — | — | ✅ CPU | ✅ CPU | ✅ CPU | OpenRAIL-M |
|
||||
| **MOSS-TTS-v1.5** ⚡ (8B) | 31 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **dots.tts** ⚡ (2B) | 24 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ❌ | Apache-2.0 |
|
||||
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MOSS-TTS-Nano** | 20 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **KittenTTS** | English | ❌ | ❌ | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
|
||||
|
||||
> **CUDA** = GPU-accelerated · **MPS** = Apple Silicon Metal · **CPU** = runs everywhere, slower for large models · KittenTTS and MOSS-TTS-Nano run realtime on CPU · MLX-Audio is Apple Silicon only · ⚡ = lazy-registered (installed on first use)
|
||||
>
|
||||
> **MOSS-TTS-v1.5** (8B, ~16 GB weights) and **dots.tts** (2B, ~9 GB weights) are heavyweight opt-in engines that run in their own isolated venv from a local clone — see [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) and [dots.tts](docs/engines/dots-tts.md). Neither claims Apple-Silicon **MPS** (upstream is CUDA/CPU only; on a Mac they run on CPU). dots.tts upstream is Linux/macOS only — no Windows path.
|
||||
> **CUDA** = GPU-accelerated · **MPS** = Apple Silicon Metal · **CPU** = runs everywhere, slower for large models · KittenTTS and MOSS-TTS-Nano run realtime on CPU · MLX-Audio is Apple Silicon only.
|
||||
|
||||
### ASR Engines
|
||||
|
||||
@@ -305,35 +253,31 @@ OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictatio
|
||||
|--------|-------------------------|:---------:|----------|
|
||||
| **WhisperX** (default) | `whisperx` | ~100 | Dubbing & subtitles — word-level timing via wav2vec2 forced alignment |
|
||||
| **Faster-Whisper** | `faster-whisper` | ~100 | Fast transcription on Linux / macOS / Windows (CTranslate2) |
|
||||
| **Faster-Whisper (isolated)** | `faster-whisper-isolated` | ~100 | Same as Faster-Whisper but crash-isolated in a subprocess — an ASR crash won't take down the app |
|
||||
| **MLX Whisper** | `mlx-whisper` | ~100 | Native Apple Silicon speed (Apple MLX / Metal) |
|
||||
| **PyTorch Whisper** | `pytorch-whisper` | ~100 | CUDA / CPU fallback via 🤗 Transformers (no cuDNN 8 needed) |
|
||||
| **PyTorch Whisper** | `pytorch-whisper` | ~100 | CUDA / CPU fallback via 🤗 Transformers |
|
||||
| **Parakeet TDT** | `nemo-parakeet` | English + 25 EU | SOTA English accuracy, auto language detection (NVIDIA NeMo, GPU only) |
|
||||
| **Moonshine** | `moonshine` | English | Edge / low-latency, ONNX |
|
||||
| **FunASR** | `funasr` | 50+ | All-in-one multilingual — built-in VAD + inline speaker diarization (SenseVoice) |
|
||||
|
||||
> Whisper-family engines cover ~100 languages; **FunASR / SenseVoice** adds an all-in-one multilingual path with built-in voice-activity detection and inline speaker diarization. Every engine runs on-device — no API keys, no cloud.
|
||||
|
||||
> **GPU without efficient float16?** On older NVIDIA GPUs (Maxwell/Pascal, GTX 16xx) or after a CTranslate2/cuDNN mismatch, the CTranslate2 ASR engines (WhisperX, Faster-Whisper) can't run `float16` and OmniVoice automatically retries on `int8` — no config needed. If transcription still fails, pin the compute type with the `ASR_COMPUTE_TYPE` env var (escape hatch): `ASR_COMPUTE_TYPE=int8` (or `float32` for CPU). Set it to `int8` and restart the backend.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Frontend (React) │
|
||||
│ DubTab · VoiceConsole · Stories · Audiobook · Gallery │
|
||||
│ Dictation · BatchQueue · Diagnostics · MCP Client │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ Backend (FastAPI) │
|
||||
│ 100+ API endpoints · SSE+WSS streaming · SQLite │
|
||||
├──────────┬──────────┬──────────┬──────────┬────────────────┤
|
||||
│ WhisperX │ Demucs │OmniVoice │ Pyannote │ Engine Routing │
|
||||
│ (+7 ASR │ Source │ (+10 │ Diariz- │ ↳ GPU preflight │
|
||||
│ engines) │ Sep. │ TTS) │ ation │ ↳ No silent CPU │
|
||||
└──────────┴──────────┴──────────┴──────────┴────────────────┘
|
||||
CUDA / MPS / ROCm / CPU (auto-detected + routed)
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ Frontend (React) │
|
||||
│ DubTab · VoicePreview · BatchQueue · Gallery │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Backend (FastAPI) │
|
||||
│ 97 API endpoints · SSE streaming · SQLite │
|
||||
├──────────┬──────────┬──────────┬────────────────┤
|
||||
│ WhisperX │ Demucs │OmniVoice │ Pyannote │
|
||||
│ ASR │ Source │ TTS │ Diarization │
|
||||
│ │ Sep. │ │ │
|
||||
└──────────┴──────────┴──────────┴────────────────┘
|
||||
CUDA / MPS / ROCm / CPU (auto-detected)
|
||||
```
|
||||
|
||||
---
|
||||
@@ -344,57 +288,27 @@ OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictatio
|
||||
|
||||
| Category | Features |
|
||||
|----------|----------|
|
||||
| **Longform** | Audiobook editor (text/EPUB/PDF → chaptered .m4b), Stories multi-voice editor, two-pass loudnorm mastering, crash-resume for interrupted renders, pronunciation control + SSML-lite prosody |
|
||||
| **Dubbing** | Full pipeline (transcribe→translate→synthesize→mux), scene-aware splitting, lip-sync scoring, streaming TTS, per-speaker voice assignment, Smart Fit timing + second-pass QC, dedicated Dub home |
|
||||
| **Voice** | Zero-shot cloning, voice design, A/B comparison, voice preview widget, gallery with favorites/tags, portable persona bundles (`.ovsvoice`), voice console workspace |
|
||||
| **Audio** | Demucs vocal isolation, per-segment gain, selective track export, stem/SRT/VTT/MP3 export, unlimited-length TTS via sentence-chunked generation |
|
||||
| **Dubbing** | Full pipeline (transcribe→translate→synthesize→mux), scene-aware splitting, lip-sync scoring, streaming TTS |
|
||||
| **Voice** | Zero-shot cloning, voice design, A/B comparison, voice preview widget, gallery with favorites/tags |
|
||||
| **Audio** | Demucs vocal isolation, per-segment gain, selective track export, stem/SRT/VTT/MP3 export |
|
||||
| **Multi-Lang** | Multi-language batch picker, batch dubbing queue with sequential GPU execution |
|
||||
| **Diarization** | Pyannote ML diarization, auto speaker clone extraction, per-speaker voice assignment |
|
||||
| **ASR** | 8 engines (WhisperX, Faster-Whisper, isolated Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet TDT, Moonshine, FunASR/SenseVoice), crash-isolated subprocess backend |
|
||||
| **TTS** | 11 engines (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, + lazy: IndexTTS 2, OmniVoice GGUF, Supertonic 3), engine routing with GPU preflight |
|
||||
| **Infra** | Docker deployment, CUDA/MPS/ROCm auto-detect, cuDNN 8 compat, VRAM-aware model offloading, engine routing (no silent CPU fallback), diagnostics suite & error journal, restricted-network mirror support |
|
||||
| **Infra** | Docker deployment, CUDA/MPS/ROCm auto-detect, cuDNN 8 compat, VRAM-aware model offloading |
|
||||
| **AI Provenance** | AudioSeal invisible watermarking (SynthID-like), video logo overlay, watermark detection API |
|
||||
| **UX** | Undo/redo, keyboard shortcuts, drag-and-drop, session persistence, glassmorphism design system, UI scale fix for Linux/WebKitGTK |
|
||||
| **UX** | Undo/redo, keyboard shortcuts, drag-and-drop, session persistence, glassmorphism design system |
|
||||
| **Real-time Events** | WebSocket event bus — instant sidebar refresh on data mutations, exponential backoff reconnect |
|
||||
| **State Management** | Zustand store migration — `uiSlice`, `pillSlice`, `dubSlice`, `generateSlice`, `prefsSlice`, `glossarySlice` |
|
||||
| **Desktop** | Cross-platform Tauri installers (macOS DMG/Intel, Windows MSI, Linux deb/AppImage), auto-update infrastructure, single-instance enforcement, close-to-tray, macOS Gatekeeper fix |
|
||||
| **Dictation** | Global system-wide hotkey (`⌘+⇧+Space`), frameless floating widget, streaming ASR via WebSocket, auto-paste, customizable hotkey, local-LLM transcript refinement |
|
||||
| **Desktop** | Cross-platform Tauri installers (macOS DMG, Windows MSI, Linux deb/AppImage), auto-update infrastructure |
|
||||
| **Windows Hardening** | Cross-platform log paths, Triton workaround, HF symlink bypass, 300s health check timeout |
|
||||
| **Dictation** | Global system-wide hotkey (`⌘+⇧+Space`), frameless floating widget, streaming ASR via WebSocket, auto-paste |
|
||||
| **Batch Pipeline** | Full batch TTS: extract → transcribe → translate → generate → mix → export, with live progress tracking |
|
||||
| **MCP Server** | OmniVoice as a local TTS/STT provider for Claude, Cursor, and any MCP client |
|
||||
| **Remote Backend** | Point the desktop UI at a remote backend URL with bearer auth (Tailscale-documented) |
|
||||
| **Reliability** | Stall watchdog on bootstrap splash, per-engine GPU compatibility matrix, actionable errors for non-executable engine binaries, setuptools auto-repair |
|
||||
|
||||
### 🔜 Up Next
|
||||
|
||||
- 🎬 **Lip-sync v2** — visual speech timing with wav2lip
|
||||
- 📖 **Audiobook Editor** — chapter-aware long-form narration
|
||||
- 🌐 **Hosted Demo** — try OmniVoice without installing anything
|
||||
- 🔌 **Plugin Marketplace** — community-contributed TTS engines and effects
|
||||
- 🎵 **Real-time Voice Changer** — live microphone transformation during calls
|
||||
|
||||
---
|
||||
|
||||
## Sponsor / Donate
|
||||
|
||||
OmniVoice Studio is built by one developer using Claude Code and AI agents — and the agent bills are real. Over the last three months I've spent thousands of dollars on Claude subscriptions to keep the features shipping, the bugs fixed, and your issues answered. If OmniVoice has created value for you, helping cover those bills means I can keep developing full-time.
|
||||
|
||||
<div align="center">
|
||||
|
||||
**This month's agent bill fund**
|
||||
|
||||
<img src="https://img.shields.io/badge/raised_%2410_of_%24200-5%25-EAB308?style=for-the-badge" alt="$10 / $200 raised" />
|
||||
|
||||
<br/><br/>
|
||||
|
||||
<a href="https://ko-fi.com/debpalash"><img src="https://img.shields.io/badge/Ko--fi-Support_❤️-FF5E5B?style=for-the-badge&logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
|
||||
|
||||
<a href="https://paypal.me/palashCoder"><img src="https://img.shields.io/badge/PayPal-Donate-00457C?style=for-the-badge&logo=paypal&logoColor=white" alt="PayPal" /></a>
|
||||
|
||||
<a href="https://github.com/sponsors/debpalash"><img src="https://img.shields.io/badge/GitHub-Sponsor-ff69b4?style=for-the-badge&logo=github&logoColor=white" alt="GitHub Sponsors" /></a>
|
||||
|
||||
<br/>
|
||||
<sub>Every dollar goes directly to agent bills — keeping OmniVoice development continuous.</sub>
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
@@ -463,7 +377,7 @@ Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are availab
|
||||
<details>
|
||||
<summary><b>Can I add my own TTS engine?</b></summary>
|
||||
<br/>
|
||||
Yes. OmniVoice uses a <b>built-in backend registry</b>. To add an engine in ~50 lines, subclass <code>TTSBackend</code> in <code>backend/services/tts_backend.py</code> and add it to the <code>_REGISTRY</code> dictionary. Eleven engines are built in: OmniVoice, CosyVoice 3, GPT-SoVITS, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, KittenTTS, Sherpa-ONNX, plus lazy-registered IndexTTS 2, OmniVoice GGUF, and Supertonic 3. See the <a href="#tts-engines">TTS Engines</a> section for details.
|
||||
Yes. OmniVoice uses a <b>built-in backend registry</b>. To add an engine in ~50 lines, subclass <code>TTSBackend</code> in <code>backend/services/tts_backend.py</code> and add it to the <code>_REGISTRY</code> dictionary at the bottom. Six engines are built in: OmniVoice, CosyVoice, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, and KittenTTS. See the <a href="#tts-engines">TTS Engines</a> section for details.
|
||||
</details>
|
||||
|
||||
---
|
||||
@@ -493,9 +407,6 @@ OmniVoice Studio is built on the shoulders of exceptional open-source work:
|
||||
| [**CTranslate2**](https://github.com/OpenNMT/CTranslate2) | Optimized Transformer inference on CPU and GPU |
|
||||
| [**AudioSeal (Meta)**](https://github.com/facebookresearch/audioseal) | Invisible neural audio watermarking for AI provenance |
|
||||
| [**Tauri**](https://tauri.app) | Native desktop app framework |
|
||||
| [**Supertone / Supertonic 3**](https://huggingface.co/Supertone/supertonic-3) | ONNX TTS engine — 31 languages, CPU-efficient |
|
||||
| [**Sherpa-ONNX**](https://github.com/k2-fsa/sherpa-onnx) | WASM-ready universal TTS/ASR runtime |
|
||||
| [**GPT-SoVITS**](https://github.com/RVC-Boss/GPT-SoVITS) | Zero-shot TTS engine — 5 languages, RTF 0.014 |
|
||||
|
||||
---
|
||||
|
||||
@@ -505,8 +416,7 @@ OmniVoice Studio is built on the shoulders of exceptional open-source work:
|
||||
|
||||
If you read this far, you're our kind of person.<br/>
|
||||
**[⭐ Star this repo](https://github.com/debpalash/OmniVoice-Studio)** so others can find it too.<br/>
|
||||
**[💬 Join the Discord](https://discord.gg/bzQavDfVV9)** to share what you build.<br/>
|
||||
**[❤️ Support development](https://ko-fi.com/debpalash)** — fund the AI agent bills that keep OmniVoice shipping.
|
||||
**[💬 Join the Discord](https://discord.gg/bzQavDfVV9)** to share what you build.
|
||||
|
||||
<br/>
|
||||
|
||||
|
||||
@@ -372,13 +372,9 @@ OmniVoice 配备多引擎 TTS 后端。默认引擎(OmniVoice)始终可用
|
||||
| **MLX-Audio**(Kokoro, Qwen3-TTS, CSM, Dia 等) | 多语言 | 因引擎而异 | 因引擎而异 | ❌ | ✅ 原生 | ❌ | 因引擎而异 |
|
||||
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MOSS-TTS-Nano** | 20 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **MOSS-TTS-v1.5**(8B,可选装) | 31 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
|
||||
| **dots.tts**(2B,可选装) | 24 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ❌ | Apache-2.0 |
|
||||
| **KittenTTS** | 英语 | ❌ | ❌ | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
|
||||
|
||||
> **CUDA** = GPU 加速 · **MPS** = Apple Silicon Metal · **CPU** = 随处可运行,大模型较慢 · KittenTTS 和 MOSS-TTS-Nano 可在 CPU 上实时运行 · MLX-Audio 仅限 Apple Silicon。
|
||||
>
|
||||
> **MOSS-TTS-v1.5**(8B,约 16 GB 权重)和 **dots.tts**(2B,约 9 GB 权重)是重量级可选引擎,从本地克隆在独立 venv 中运行——参见 [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) 和 [dots.tts](docs/engines/dots-tts.md)。两者均不支持 Apple Silicon **MPS**(上游仅支持 CUDA/CPU;在 Mac 上以 CPU 运行)。dots.tts 上游仅支持 Linux/macOS——无 Windows 路径。
|
||||
|
||||
---
|
||||
|
||||
|
||||
+4
-14
@@ -2,20 +2,10 @@
|
||||
|
||||
## Supported Versions
|
||||
|
||||
| Version | Supported |
|
||||
|---------|-----------|
|
||||
| 0.3.x (latest release + `main` previews) | ✅ Current — all fixes land here |
|
||||
| 0.2.7 | ⚠️ Legacy stable — security fixes only, upgrade recommended |
|
||||
| < 0.2.7 | ❌ No longer supported |
|
||||
|
||||
## Model supply chain
|
||||
|
||||
OmniVoice supports models from **public, verifiable sources only** (Hugging
|
||||
Face repos, official project releases). Privately sold or gated model files
|
||||
are not supported: an archive from a private source can carry anything
|
||||
(bundled executables, modified configs), and nobody else can verify or
|
||||
reproduce it. Treat any privately distributed model file as an untrusted
|
||||
download, and never run executables bundled with model archives.
|
||||
| Version | Supported |
|
||||
|---------|--------------------|
|
||||
| 0.2.x | ✅ Current release |
|
||||
| < 0.2 | ❌ No longer supported |
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
|
||||
-36
@@ -1,36 +0,0 @@
|
||||
# Support
|
||||
|
||||
## Where to get help
|
||||
|
||||
| Channel | Best for |
|
||||
|---|---|
|
||||
| [Discord](https://discord.gg/bzQavDfVV9) — `#help` | Setup problems, quick questions, sharing results |
|
||||
| [GitHub Issues](https://github.com/debpalash/OmniVoice-Studio/issues) | Bugs and feature requests — use the templates; attach the diagnostic bundle (Settings → About → "Save diagnostic bundle") |
|
||||
| [GitHub Discussions](https://github.com/debpalash/OmniVoice-Studio/discussions) | Design questions, ideas, show & tell |
|
||||
| Security issues | **Never a public issue** — see [SECURITY.md](SECURITY.md) for private reporting |
|
||||
|
||||
## Model sources we support
|
||||
|
||||
OmniVoice is built on the idea that everything it runs is **open and available
|
||||
to everyone**: free, public models with verifiable sources and licenses
|
||||
(Hugging Face repos, official project releases), so the whole community can
|
||||
use, test, and debug the same thing.
|
||||
|
||||
**We do not support privately sold, paywalled, or gated model files.** A model
|
||||
delivered privately can't be verified, reproduced, or shared — it doesn't fit
|
||||
the project's goals, and issues involving such models will be politely closed.
|
||||
As a general safety rule, never run executables bundled inside any model
|
||||
archive.
|
||||
|
||||
## Before filing a bug
|
||||
|
||||
1. Update to the latest release (or `main` if you follow previews) — fixes ship continuously.
|
||||
2. Run the in-app self-check: **Settings → About → Run self-check**.
|
||||
3. Search existing issues; add a 👍 + your details to an existing one rather than opening a duplicate.
|
||||
|
||||
## Response expectations
|
||||
|
||||
This is an open-source project maintained with the help of an automated triage
|
||||
bot: issues are typically triaged within hours and every report gets a human-
|
||||
approved response. Reproducible reports with a diagnostic bundle get fixed
|
||||
fastest.
|
||||
+1
-7
@@ -13,18 +13,12 @@
|
||||
# Run: uv run pyinstaller backend.spec --noconfirm --clean
|
||||
import platform
|
||||
import sys
|
||||
from PyInstaller.utils.hooks import collect_data_files, collect_all, collect_submodules, copy_metadata
|
||||
from PyInstaller.utils.hooks import collect_data_files, collect_all, collect_submodules
|
||||
|
||||
IS_MAC_ARM = sys.platform == "darwin" and platform.machine() == "arm64"
|
||||
|
||||
datas = []
|
||||
binaries = []
|
||||
|
||||
# Bundle the omnivoice package's .dist-info so importlib.metadata.version()
|
||||
# resolves inside the frozen build. Without it the backend can't read its own
|
||||
# version and falls back to the literal in backend/core/version.py — which is
|
||||
# how a 0.3.6 desktop build shipped reporting "0.3.5" in About + bug reports.
|
||||
datas += copy_metadata('omnivoice')
|
||||
hiddenimports = [
|
||||
# Web stack
|
||||
'uvicorn', 'uvicorn.logging', 'uvicorn.loops', 'uvicorn.loops.auto',
|
||||
|
||||
@@ -7,13 +7,9 @@ composed at the route or router level without surprises.
|
||||
Currently exposed:
|
||||
- `require_loopback`: 403 unless the request came from a loopback origin
|
||||
(bypassed in explicit server mode — see `_server_mode`).
|
||||
- `ws_remote_authorized`: whether a WebSocket handshake from a non-loopback
|
||||
client carries the remote API key (Wave 2.3) — used by WS endpoints that
|
||||
keep their own inline loopback guards.
|
||||
"""
|
||||
|
||||
import os
|
||||
import secrets
|
||||
|
||||
from fastapi import HTTPException, Request
|
||||
|
||||
@@ -75,31 +71,3 @@ def require_loopback(request: Request) -> None:
|
||||
if _server_mode():
|
||||
return
|
||||
raise HTTPException(status_code=403, detail="loopback origin required")
|
||||
|
||||
|
||||
def remote_api_key() -> str | None:
|
||||
"""The remote-backend bearer key (Wave 2.3), or None when remote mode is
|
||||
off. Read at call time so tests can monkeypatch the env."""
|
||||
return os.environ.get("OMNIVOICE_API_KEY") or None
|
||||
|
||||
|
||||
def ws_remote_authorized(websocket) -> bool:
|
||||
"""Whether a WebSocket handshake presents the remote API key.
|
||||
|
||||
Browser WebSockets cannot set an Authorization header, so the key may
|
||||
arrive as ``?api_key=`` or via the ``ov_key`` cookie that the bearer
|
||||
middleware sets on the first authenticated HTTP request. Returns False
|
||||
when remote mode is off — callers keep their loopback-only behavior.
|
||||
"""
|
||||
key = remote_api_key()
|
||||
if not key:
|
||||
return False
|
||||
auth = websocket.headers.get("authorization", "")
|
||||
supplied = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
|
||||
if not supplied:
|
||||
supplied = (
|
||||
websocket.query_params.get("api_key")
|
||||
or websocket.cookies.get("ov_key")
|
||||
or ""
|
||||
)
|
||||
return secrets.compare_digest(supplied, key)
|
||||
|
||||
@@ -1,759 +0,0 @@
|
||||
"""Audiobook creator endpoints (parity Wave 5).
|
||||
|
||||
``POST /audiobook/plan`` — pure preview: parse a chapter-delimited script
|
||||
(Markdown ``# H1`` chapters, inline ``[voice:NAME]`` / ``[pause …]``) into the
|
||||
chapter/span plan, no synthesis.
|
||||
|
||||
``POST /audiobook`` — the synth job: render each chapter through the active TTS
|
||||
backend (reusing ``services.audiobook.synthesize_chapter`` + ``chunked_tts``),
|
||||
then mux the chapter WAVs into a chapterized **m4b** (FFMETADATA1 chapters via
|
||||
``build_m4b_cmd``). Progress streams as Server-Sent Events, mirroring the dub
|
||||
pipeline. ffmpeg-gated — without ffmpeg the job reports an error event and
|
||||
stops (the m4b is the only output format).
|
||||
|
||||
``GET /audiobook/jobs`` + ``POST /audiobook/resume/{job_id}`` — durable
|
||||
crash-resume: an interrupted render persists its plan + params to a
|
||||
``resume.json`` manifest in the job work dir, so it can be resumed later (the
|
||||
content-addressed chapter cache makes finished chapters instant) even without
|
||||
the original script. The resume UI affordance remains a follow-up.
|
||||
|
||||
epub/pdf ingest, ACX mastering shipped; the resume UI surface remains a follow-up.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, File, HTTPException, UploadFile
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from services.audiobook import (
|
||||
parse_audiobook_script,
|
||||
synthesize_chapter,
|
||||
)
|
||||
from services.longform_render import (
|
||||
LOUDNESS_PRESETS,
|
||||
build_concat_list,
|
||||
build_ffmetadata,
|
||||
build_render_cmd,
|
||||
prune_cache_dir,
|
||||
)
|
||||
from services import longform_resume # pure (no torch) — durable resume manifest
|
||||
|
||||
logger = logging.getLogger("omnivoice.audiobook")
|
||||
router = APIRouter()
|
||||
|
||||
# A cover filename as produced by /audiobook/cover: 12 hex chars + image ext.
|
||||
# An exact-match allowlist is the strongest barrier (and the one CodeQL's
|
||||
# path-injection query recognizes) — anything else is rejected outright.
|
||||
_COVER_NAME_RE = re.compile(r"^[0-9a-f]{12}\.(?:jpg|jpeg|png)$")
|
||||
|
||||
|
||||
def _safe_cover_path(cover_path: str | None) -> str | None:
|
||||
"""Confine a user-supplied cover to the upload directory before it can flow
|
||||
into ffmpeg.
|
||||
|
||||
Covers only ever come from ``/audiobook/cover``, which writes them to
|
||||
``OUTPUTS_DIR/audiobook_covers`` with a generated name. We rebuild the path
|
||||
from the basename alone (``os.path.basename`` strips any directory component
|
||||
or ``..`` traversal) joined onto that fixed directory, so no caller-supplied
|
||||
path — absolute or relative — can escape it. Returns the path only if the
|
||||
file actually exists there, else None."""
|
||||
if not cover_path:
|
||||
return None
|
||||
from core.config import OUTPUTS_DIR
|
||||
name = os.path.basename(cover_path)
|
||||
if not _COVER_NAME_RE.match(name):
|
||||
return None # not a name the upload endpoint could have produced
|
||||
cover_dir = os.path.realpath(os.path.join(OUTPUTS_DIR, "audiobook_covers"))
|
||||
real = os.path.realpath(os.path.join(cover_dir, name))
|
||||
# Containment check on the resolved path itself — it must live inside the
|
||||
# covers dir. Belt-and-suspenders over the regex+basename above; the
|
||||
# commonpath form is the path-injection barrier static analysis recognizes.
|
||||
if os.path.commonpath([real, cover_dir]) != cover_dir:
|
||||
return None
|
||||
return real if os.path.isfile(real) else None
|
||||
|
||||
|
||||
class AudiobookPlanRequest(BaseModel):
|
||||
text: str
|
||||
default_voice: str | None = None
|
||||
|
||||
|
||||
@router.post("/audiobook/plan")
|
||||
def audiobook_plan(req: AudiobookPlanRequest) -> dict:
|
||||
"""Parse a script into a chapter/span plan (pure preview, no synthesis)."""
|
||||
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
|
||||
return plan.to_dict()
|
||||
|
||||
|
||||
#: Cover size cap mirrors longform_render's guard (8 MB — a book cover, not a
|
||||
#: payload). Kept in sync intentionally; the render builder re-validates too.
|
||||
_COVER_MAX_BYTES = 8 * 1024 * 1024
|
||||
#: Import upload cap — a generous ceiling for a .txt/.md/.epub manuscript that
|
||||
#: still stops a memory-exhaustion upload (the whole file is read into RAM).
|
||||
_IMPORT_MAX_BYTES = 64 * 1024 * 1024
|
||||
#: Upper bound on chapters in a single /longform/render plan — far above any real
|
||||
#: book, but stops a pathological request from allocating/holding the job forever.
|
||||
_MAX_CHAPTERS = 10_000
|
||||
|
||||
|
||||
@router.post("/audiobook/import")
|
||||
async def audiobook_import(file: UploadFile = File(...)) -> dict:
|
||||
"""Import a ``.txt``/``.md``/``.epub``/``.pdf`` into a chapter-delimited script.
|
||||
|
||||
EPUB is parsed in spine order (stdlib only, local); PDF text is extracted
|
||||
with pypdf (pure-Python) then chapterized; plain text gets ``# `` headings
|
||||
inserted ahead of obvious chapter-title lines. Returns the script text (for
|
||||
the editor) + the resulting chapter count."""
|
||||
from services.longform_import import (
|
||||
chapterize_plaintext,
|
||||
epub_to_chapter_script,
|
||||
pdf_to_chapter_script,
|
||||
)
|
||||
|
||||
name = (file.filename or "").lower()
|
||||
data = await file.read()
|
||||
if not data:
|
||||
raise HTTPException(status_code=400, detail="empty file")
|
||||
if len(data) > _IMPORT_MAX_BYTES:
|
||||
raise HTTPException(status_code=400, detail="file too large (max 64 MB)")
|
||||
if name.endswith(".epub"):
|
||||
try:
|
||||
script = epub_to_chapter_script(data)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=f"couldn't parse EPUB: {e}")
|
||||
elif name.endswith(".pdf"):
|
||||
try:
|
||||
script = pdf_to_chapter_script(data)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=f"couldn't parse PDF: {e}")
|
||||
else:
|
||||
script = chapterize_plaintext(data.decode("utf-8", "ignore"))
|
||||
if not script.strip():
|
||||
raise HTTPException(status_code=400, detail="no text found in the file")
|
||||
plan = parse_audiobook_script(script)
|
||||
return {"text": script, "chapters": plan.chapter_count}
|
||||
|
||||
|
||||
@router.post("/audiobook/cover")
|
||||
async def audiobook_cover(cover: UploadFile = File(...)) -> dict:
|
||||
"""Upload a cover image; returns a server-side ``path`` to pass back as
|
||||
``cover_path`` in the synth request. Validated here (jpg/png + size cap) and
|
||||
again at render time."""
|
||||
from core.config import OUTPUTS_DIR
|
||||
|
||||
ext = os.path.splitext(cover.filename or "")[1].lower()
|
||||
if ext not in (".jpg", ".jpeg", ".png"):
|
||||
raise HTTPException(status_code=400, detail="cover must be a .jpg or .png")
|
||||
data = await cover.read()
|
||||
if not data or len(data) > _COVER_MAX_BYTES:
|
||||
raise HTTPException(status_code=400, detail="cover must be between 1 byte and 8 MB")
|
||||
cover_dir = os.path.join(OUTPUTS_DIR, "audiobook_covers")
|
||||
os.makedirs(cover_dir, exist_ok=True)
|
||||
path = os.path.join(cover_dir, f"{uuid.uuid4().hex[:12]}{ext}")
|
||||
with open(path, "wb") as f:
|
||||
f.write(data)
|
||||
return {"path": path}
|
||||
|
||||
|
||||
class AudiobookRequest(BaseModel):
|
||||
text: str
|
||||
default_voice: str | None = None # voice profile id; None = engine default
|
||||
language: str | None = None # None/"Auto" → profile language, else autodetect (#505)
|
||||
bitrate: str = "128k"
|
||||
format: str = "m4b" # "m4b" | "mp3"
|
||||
loudness: str | None = None # None/"off" | "acx" | "podcast" (opt-in)
|
||||
cover_path: str | None = None # server-side path to a jpg/png cover
|
||||
# Global tags embedded in the output: {title, author, narrator, year,
|
||||
# genre, description}. Player-visible (Apple Books / Audible read these).
|
||||
metadata: dict | None = None
|
||||
# Optional pronunciation lexicon {word: respelling} applied before synthesis.
|
||||
lexicon: dict | None = None
|
||||
|
||||
|
||||
def _resolve_voice(profile_id: str | None) -> dict:
|
||||
"""Map a voice-profile id to (ref_audio, ref_text, instruct, seed).
|
||||
|
||||
Compact form of the resolver in generation.py — covers locked, design and
|
||||
clone profiles. Returns all-None for the engine default (no profile).
|
||||
"""
|
||||
out = {"ref_audio": None, "ref_text": None, "instruct": None, "seed": None}
|
||||
if not profile_id:
|
||||
return out
|
||||
from core.config import VOICES_DIR
|
||||
from core.db import db_conn
|
||||
|
||||
with db_conn() as conn:
|
||||
row = conn.execute("SELECT * FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
if not row:
|
||||
return out
|
||||
try:
|
||||
kind = row["kind"] or "clone"
|
||||
except (KeyError, IndexError):
|
||||
kind = "clone"
|
||||
if row["is_locked"] and row["locked_audio_path"]:
|
||||
out["ref_audio"] = os.path.join(VOICES_DIR, row["locked_audio_path"])
|
||||
out["ref_text"] = row["ref_text"]
|
||||
out["instruct"] = row["instruct"]
|
||||
elif kind == "design":
|
||||
out["ref_audio"] = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
|
||||
out["ref_text"] = row["ref_text"] if out["ref_audio"] else None
|
||||
out["instruct"] = row["instruct"]
|
||||
else:
|
||||
out["ref_audio"] = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
|
||||
out["ref_text"] = row["ref_text"]
|
||||
out["instruct"] = row["instruct"]
|
||||
try:
|
||||
if row["seed"] is not None:
|
||||
out["seed"] = row["seed"]
|
||||
except (KeyError, IndexError):
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
def _resolve_default_language(language: str | None, default_voice: str | None) -> str | None:
|
||||
"""Pick the language to thread into the longform synth callable.
|
||||
|
||||
Priority (mirrors the single-shot /generate path, #533): an explicit
|
||||
non-Auto request ``language`` wins; otherwise the selected profile's stored
|
||||
language drives it; otherwise ``None`` (genuine Auto — the engine
|
||||
autodetects, exactly as before). Hardcoding ``None`` here (#505 B2) let the
|
||||
engine re-autodetect per chunk, so a non-English clone flipped to the wrong
|
||||
language on short/ambiguous chapters.
|
||||
"""
|
||||
if language and language != "Auto":
|
||||
return language
|
||||
if default_voice:
|
||||
from core.db import db_conn
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT language FROM voice_profiles WHERE id=?", (default_voice,)
|
||||
).fetchone()
|
||||
if row:
|
||||
try:
|
||||
prof_lang = row["language"]
|
||||
except (KeyError, IndexError):
|
||||
prof_lang = None
|
||||
if prof_lang and prof_lang != "Auto":
|
||||
return prof_lang
|
||||
return None
|
||||
|
||||
|
||||
def _build_synth(default_voice: str | None, language: str | None = None) -> dict:
|
||||
"""Describe how to synthesize for the active TTS engine.
|
||||
|
||||
Returns a dict with ``mode``, ``resolve`` (voice-id → resolved refs, cached
|
||||
per id) and ``engine_id``. For OmniVoice it also carries the async
|
||||
``get_model``; other engines carry a ready ``synth`` + ``sample_rate``.
|
||||
:func:`_prepare_synth` turns this into a uniform ``(synth, sr, resolve,
|
||||
engine_id)`` once the (async) model is in hand.
|
||||
|
||||
``language`` (already resolved by :func:`_resolve_default_language`) is
|
||||
threaded into every chunk's ``generate`` so a non-English clone stays in its
|
||||
language instead of re-autodetecting per chunk (#505 B2). ``None`` keeps the
|
||||
engine's autodetect behavior unchanged.
|
||||
"""
|
||||
from services.tts_backend import OmniVoiceBackend, active_backend_id, get_backend_class
|
||||
|
||||
cache: dict = {}
|
||||
|
||||
def resolve(voice_id):
|
||||
key = voice_id or default_voice
|
||||
if key not in cache:
|
||||
cache[key] = _resolve_voice(key)
|
||||
return cache[key]
|
||||
|
||||
engine_id = active_backend_id()
|
||||
cls = get_backend_class(engine_id)
|
||||
if cls is OmniVoiceBackend:
|
||||
from services.model_manager import get_model
|
||||
return {"mode": "omnivoice", "resolve": resolve,
|
||||
"engine_id": engine_id, "get_model": get_model, "language": language}
|
||||
|
||||
backend = cls()
|
||||
|
||||
def synth(text, voice_id, speed=None):
|
||||
v = resolve(voice_id)
|
||||
return backend.generate(
|
||||
text, language=language, ref_audio=v["ref_audio"],
|
||||
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
|
||||
speed=float(speed) if speed else 1.0,
|
||||
)
|
||||
return {"mode": "generic", "resolve": resolve, "engine_id": engine_id,
|
||||
"synth": synth, "sample_rate": backend.sample_rate}
|
||||
|
||||
|
||||
async def _prepare_synth(default_voice: str | None, language: str | None = None):
|
||||
"""Resolve :func:`_build_synth` into ``(synth, sample_rate, resolve,
|
||||
engine_id)`` — awaiting the OmniVoice model load when needed. Shared by the
|
||||
full job and the per-chapter preview. ``language`` is threaded into every
|
||||
chunk so a non-English clone holds its language (#505 B2)."""
|
||||
info = _build_synth(default_voice, language=language)
|
||||
resolve, engine_id = info["resolve"], info["engine_id"]
|
||||
if info["mode"] == "omnivoice":
|
||||
lang = info["language"]
|
||||
model = await info["get_model"]()
|
||||
sr = getattr(model, "sampling_rate", 24000)
|
||||
|
||||
def synth(text, voice_id, speed=None):
|
||||
v = resolve(voice_id)
|
||||
return model.generate(
|
||||
text=text, language=lang, ref_audio=v["ref_audio"],
|
||||
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
|
||||
speed=float(speed) if speed else 1.0,
|
||||
)[0]
|
||||
return synth, sr, resolve, engine_id
|
||||
return info["synth"], info["sample_rate"], resolve, engine_id
|
||||
|
||||
|
||||
def _render_chapter_cached(chapter, synth, sr, engine_id, resolve, cache_dir, lexicon=None):
|
||||
"""Render one chapter, content-addressed so a re-run reuses it (resume).
|
||||
|
||||
Returns ``(wav_path, duration_s, was_cached)``. The WAV lives at
|
||||
``cache_dir/<key>.wav`` where ``key`` is :func:`chapter_cache_key` over the
|
||||
chapter's spans + sample rate + engine + each voice's resolved signature
|
||||
(+ the lexicon, so a lexicon edit re-renders), so an unchanged chapter is
|
||||
never re-synthesized. Runs in the GPU-pool executor.
|
||||
"""
|
||||
import json
|
||||
import wave
|
||||
|
||||
from services.audio_io import atomic_save_wav
|
||||
from services.longform_render import chapter_cache_key
|
||||
from services.pronunciation import normalize_lexicon
|
||||
|
||||
spans_tuples = [(s.voice_id, s.text, s.pause_ms_after, getattr(s, "speed", None))
|
||||
for s in chapter.spans]
|
||||
sig: dict = {}
|
||||
for s in chapter.spans:
|
||||
k = s.voice_id or ""
|
||||
if k not in sig:
|
||||
v = resolve(s.voice_id)
|
||||
sig[k] = f"{v.get('ref_audio')}|{v.get('ref_text')}|{v.get('instruct')}|{v.get('seed')}"
|
||||
if lexicon:
|
||||
# Fold the lexicon into the cache key so editing pronunciations
|
||||
# invalidates cached chapters (reserved key can't collide with a voice id).
|
||||
sig["\x00lexicon"] = json.dumps(normalize_lexicon(lexicon), sort_keys=True)
|
||||
key = chapter_cache_key(spans_tuples, sample_rate=sr, engine_id=engine_id, voice_sig=sig)
|
||||
wav_path = os.path.join(cache_dir, f"{key}.wav")
|
||||
|
||||
if os.path.exists(wav_path):
|
||||
try:
|
||||
with wave.open(wav_path, "rb") as w:
|
||||
dur = w.getnframes() / float(w.getframerate() or sr)
|
||||
return wav_path, dur, True
|
||||
except Exception:
|
||||
pass # corrupt cache entry — fall through and re-render
|
||||
|
||||
audio, dur = synthesize_chapter(chapter.spans, synth, sr, lexicon=lexicon)
|
||||
atomic_save_wav(wav_path, audio, sr)
|
||||
return wav_path, dur, False
|
||||
|
||||
|
||||
class AudiobookPreviewRequest(BaseModel):
|
||||
text: str
|
||||
chapter_index: int = 0
|
||||
default_voice: str | None = None
|
||||
language: str | None = None # None/"Auto" → profile language, else autodetect
|
||||
lexicon: dict | None = None
|
||||
|
||||
|
||||
@router.post("/audiobook/preview")
|
||||
async def audiobook_preview(req: AudiobookPreviewRequest) -> dict:
|
||||
"""Render a single chapter so the user can audition it before the full run.
|
||||
|
||||
Reuses the same content-addressed cache as the job, so a preview warms the
|
||||
cache (the later full render reuses it) and a re-preview is instant.
|
||||
"""
|
||||
from core.config import OUTPUTS_DIR
|
||||
from services.model_manager import _gpu_pool
|
||||
|
||||
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
|
||||
if not plan.chapters:
|
||||
raise HTTPException(status_code=400, detail="no chapters parsed from the script")
|
||||
n = len(plan.chapters)
|
||||
if not (0 <= req.chapter_index < n):
|
||||
raise HTTPException(status_code=400, detail=f"chapter_index out of range (0..{n - 1})")
|
||||
|
||||
chapter = plan.chapters[req.chapter_index]
|
||||
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache") # shared with _render_longform_sse
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
synth, sr, resolve, engine_id = await _prepare_synth(
|
||||
req.default_voice,
|
||||
language=_resolve_default_language(req.language, req.default_voice),
|
||||
)
|
||||
loop = asyncio.get_running_loop()
|
||||
wav_path, dur, was_cached = await loop.run_in_executor(
|
||||
_gpu_pool, _render_chapter_cached, chapter, synth, sr, engine_id, resolve, cache_dir,
|
||||
req.lexicon,
|
||||
)
|
||||
return {
|
||||
"output": os.path.relpath(wav_path, OUTPUTS_DIR), # served via /audio
|
||||
"duration_s": round(dur, 2),
|
||||
"cached": was_cached,
|
||||
"title": chapter.title,
|
||||
}
|
||||
|
||||
|
||||
async def _render_longform_sse(
|
||||
plan,
|
||||
*,
|
||||
default_voice: str | None,
|
||||
language: str | None = None,
|
||||
fmt: str = "m4b",
|
||||
bitrate: str = "128k",
|
||||
loudness: str | None = None,
|
||||
cover_path: str | None = None,
|
||||
metadata: dict | None = None,
|
||||
lexicon: dict | None = None,
|
||||
job_type: str = "audiobook",
|
||||
job_id: str | None = None,
|
||||
resume: bool = False,
|
||||
):
|
||||
"""Shared chapterized-render SSE generator for Audiobook *and* Stories.
|
||||
|
||||
Takes a ready ``plan`` (``.chapters`` → ``.title`` + ``.spans``) — Audiobook
|
||||
parses it from a script, Stories compiles it from cast/lines — and renders
|
||||
each chapter (content-addressed cache → resume), isolating per-chapter
|
||||
failures, then muxes the successful chapters into a tagged file. This is the
|
||||
convergence point: one renderer, two front doors.
|
||||
"""
|
||||
from core.config import OUTPUTS_DIR
|
||||
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
|
||||
from services.model_manager import _gpu_pool
|
||||
|
||||
# Resume reuses the original job_id (continuing the same job row + cached
|
||||
# chapters); a fresh render generates a new one. The id may arrive from the
|
||||
# /resume/{job_id} path param, so strip it to a safe token (no path
|
||||
# separators, no CR/LF) before it ever reaches a filesystem path or a log
|
||||
# line — CodeQL py/path-injection + py/log-injection. Empty after the strip
|
||||
# → a fresh id.
|
||||
job_id = re.sub(r"[^A-Za-z0-9_-]", "", job_id or "")[:64] or uuid.uuid4().hex[:16]
|
||||
try:
|
||||
from core import job_store
|
||||
if not resume:
|
||||
job_store.create(job_id, type=job_type)
|
||||
job_store.mark_running(job_id)
|
||||
except Exception:
|
||||
job_store = None # job history is best-effort; never block synthesis
|
||||
|
||||
# Persist a durable resume manifest (plan + params) so an interrupted render
|
||||
# can be resumed later even without the original script. Best-effort.
|
||||
try:
|
||||
title = (metadata or {}).get("title") or (plan.chapters[0].title if plan.chapters else "")
|
||||
longform_resume.write_manifest(longform_resume.build_manifest(
|
||||
job_id=job_id, job_type=job_type, title=title,
|
||||
plan_chapters=[
|
||||
{"title": c.title, "spans": [s.to_dict() for s in c.spans]}
|
||||
for c in plan.chapters
|
||||
],
|
||||
params={
|
||||
"default_voice": default_voice, "language": language,
|
||||
"fmt": fmt, "bitrate": bitrate,
|
||||
"loudness": loudness, "cover_path": cover_path,
|
||||
"metadata": metadata, "lexicon": lexicon,
|
||||
},
|
||||
))
|
||||
except Exception: # resume durability is an enhancement; never block the render
|
||||
logger.debug("[%s] resume manifest write skipped", job_id, exc_info=True)
|
||||
|
||||
def _emit(payload: dict) -> str:
|
||||
if job_store is not None:
|
||||
try:
|
||||
job_store.append_event(job_id, json.dumps(payload))
|
||||
except Exception:
|
||||
pass # best-effort job history; never block the stream
|
||||
return f"data: {json.dumps(payload)}\n\n"
|
||||
|
||||
if not plan.chapters:
|
||||
yield _emit({"type": "error", "error": "nothing to render (no chapters)"})
|
||||
return
|
||||
ffmpeg = find_ffmpeg()
|
||||
if not ffmpeg:
|
||||
yield _emit({"type": "error", "error": "ffmpeg not available; the output needs it"})
|
||||
return
|
||||
|
||||
# Confined work dir (job_id is already token-sanitized above; work_dir adds
|
||||
# the basename + realpath barrier so CodeQL sees a clean path).
|
||||
work = longform_resume.work_dir(job_type, job_id)
|
||||
if work is None:
|
||||
yield _emit({"type": "error", "error": "invalid job id"})
|
||||
return
|
||||
os.makedirs(work, exist_ok=True)
|
||||
# Chapter WAVs are content-addressed in a shared cache so a re-run (after a
|
||||
# failure or interruption) reuses what already rendered — only the
|
||||
# missing/changed chapters synthesize again (resume). Shared across both
|
||||
# front doors: an identical chapter renders once.
|
||||
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache")
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
prune_cache_dir(cache_dir) # bound disk before this job adds its chapters
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
try:
|
||||
synth, sr, resolve, engine_id = await _prepare_synth(
|
||||
default_voice, language=_resolve_default_language(language, default_voice)
|
||||
)
|
||||
|
||||
total = len(plan.chapters)
|
||||
chapter_files: list[str] = []
|
||||
chapters_meta: list[tuple[str, int]] = []
|
||||
cached_n = 0
|
||||
failed: list[int] = []
|
||||
yield _emit({"type": "started", "job_id": job_id, "chapters": total})
|
||||
|
||||
for i, chapter in enumerate(plan.chapters):
|
||||
try:
|
||||
wav_path, dur, was_cached = await loop.run_in_executor(
|
||||
_gpu_pool, _render_chapter_cached,
|
||||
chapter, synth, sr, engine_id, resolve, cache_dir, lexicon,
|
||||
)
|
||||
except Exception: # isolate a bad chapter — keep going
|
||||
logger.warning("[%s] chapter %d (%s) failed to render",
|
||||
job_id, i, chapter.title, exc_info=True)
|
||||
failed.append(i)
|
||||
yield _emit({"type": "chapter_error", "index": i, "total": total,
|
||||
"title": chapter.title, "error": "chapter failed to render"})
|
||||
continue
|
||||
chapter_files.append(wav_path)
|
||||
chapters_meta.append((chapter.title, int(round(dur * 1000))))
|
||||
cached_n += 1 if was_cached else 0
|
||||
yield _emit({"type": "chapter", "index": i, "total": total,
|
||||
"title": chapter.title, "duration_s": round(dur, 2),
|
||||
"cached": was_cached})
|
||||
|
||||
if not chapter_files:
|
||||
yield _emit({"type": "error", "error": "all chapters failed to render"})
|
||||
return
|
||||
|
||||
yield _emit({"type": "assembling"})
|
||||
meta_path = os.path.join(work, "chapters.ffmeta")
|
||||
with open(meta_path, "w", encoding="utf-8") as f:
|
||||
f.write(build_ffmetadata(chapters_meta, global_meta=metadata))
|
||||
concat_path = os.path.join(work, "concat.txt")
|
||||
with open(concat_path, "w", encoding="utf-8") as f:
|
||||
f.write(build_concat_list(chapter_files))
|
||||
ext = "mp3" if (fmt or "").lower() == "mp3" else "m4b"
|
||||
out_name = f"{job_type}_{job_id}.{ext}"
|
||||
out_path = os.path.join(OUTPUTS_DIR, out_name)
|
||||
|
||||
# Two-pass loudness master (#28): for a known preset, measure the
|
||||
# concatenated program first, then feed the measured values back into the
|
||||
# single mux encode. `measured is None` (skip OR any failure) → the mux
|
||||
# falls back to single-pass. Gated identically to the pure builders
|
||||
# (.lower(), no strip), so off/None/unknown/whitespace skip cleanly.
|
||||
measured = None
|
||||
norm = (loudness or "").lower()
|
||||
if norm in LOUDNESS_PRESETS:
|
||||
yield _emit({"type": "mastering", "preset": norm})
|
||||
from services.loudness import measure_loudness
|
||||
measured = await measure_loudness(ffmpeg, concat_path, norm, job_id=job_id)
|
||||
|
||||
await run_ffmpeg(
|
||||
build_render_cmd(
|
||||
ffmpeg, concat_path, meta_path, out_path,
|
||||
fmt=ext, bitrate=bitrate, cover_path=_safe_cover_path(cover_path),
|
||||
loudness=loudness, measured=measured,
|
||||
),
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if job_store is not None:
|
||||
try:
|
||||
job_store.mark_done(job_id)
|
||||
except Exception:
|
||||
pass # best-effort job history
|
||||
# The render finished — drop the resume manifest so this job is no longer
|
||||
# offered for resume.
|
||||
longform_resume.clear_manifest(job_type, job_id)
|
||||
total_s = sum(d for _, d in chapters_meta) / 1000.0
|
||||
done = {"type": "done", "output": out_name,
|
||||
"chapters": len(chapter_files), "duration_s": round(total_s, 2),
|
||||
"cached_chapters": cached_n, "failed_chapters": failed}
|
||||
# Loudness verdict only when a preset was requested — off/None paths keep
|
||||
# the exact legacy `done` shape (additive, old clients unaffected).
|
||||
if norm in LOUDNESS_PRESETS:
|
||||
p = LOUDNESS_PRESETS[norm]
|
||||
done["loudness"] = {
|
||||
"preset": norm, "target_i": p.i, "target_tp": p.tp,
|
||||
"two_pass": measured is not None,
|
||||
"measured_i": measured.input_i if measured else None,
|
||||
}
|
||||
yield _emit(done)
|
||||
except Exception as e: # surface, don't 500 the stream
|
||||
logger.exception("[%s] longform render failed", job_id)
|
||||
if job_store is not None:
|
||||
try:
|
||||
job_store.mark_failed(job_id, str(e))
|
||||
except Exception:
|
||||
pass # best-effort job history
|
||||
# Generic message only — don't leak the stack/exception text to the client.
|
||||
yield _emit({"type": "error", "error": "render failed (see backend log)"})
|
||||
|
||||
|
||||
@router.post("/audiobook")
|
||||
async def audiobook_synthesize(req: AudiobookRequest):
|
||||
"""Synthesize a chapterized audiobook from a script, streaming SSE progress."""
|
||||
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
|
||||
return StreamingResponse(
|
||||
_render_longform_sse(
|
||||
plan, default_voice=req.default_voice, language=req.language,
|
||||
fmt=req.format, bitrate=req.bitrate,
|
||||
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
|
||||
lexicon=req.lexicon, job_type="audiobook",
|
||||
),
|
||||
media_type="text/event-stream",
|
||||
)
|
||||
|
||||
|
||||
# ── Shared longform render: Stories (and any future front door) post a plan ──
|
||||
|
||||
class LongformSpan(BaseModel):
|
||||
voice_id: str | None = None
|
||||
text: str
|
||||
pause_ms_after: int = 0
|
||||
speed: float | None = None
|
||||
|
||||
|
||||
class LongformChapter(BaseModel):
|
||||
title: str = ""
|
||||
spans: list[LongformSpan] = []
|
||||
|
||||
|
||||
class LongformRenderRequest(BaseModel):
|
||||
chapters: list[LongformChapter] = []
|
||||
default_voice: str | None = None
|
||||
language: str | None = None # None/"Auto" → profile language, else autodetect (#505)
|
||||
bitrate: str = "128k"
|
||||
format: str = "m4b"
|
||||
loudness: str | None = None
|
||||
cover_path: str | None = None
|
||||
metadata: dict | None = None
|
||||
lexicon: dict | None = None
|
||||
|
||||
|
||||
@router.post("/longform/render")
|
||||
async def longform_render(req: LongformRenderRequest):
|
||||
"""Render a pre-built chapter/span plan (the Stories Editor's compiled
|
||||
cast+lines) through the shared chapterized renderer — same resume, loudness,
|
||||
cover, metadata, and output formats as the Audiobook job."""
|
||||
from services.audiobook import AudiobookPlan, Chapter, Span
|
||||
|
||||
if len(req.chapters) > _MAX_CHAPTERS:
|
||||
raise HTTPException(status_code=422, detail=f"too many chapters (max {_MAX_CHAPTERS})")
|
||||
|
||||
chapters = []
|
||||
for i, c in enumerate(req.chapters):
|
||||
# Keep a span if it has text to speak OR a pause to render (pause-only
|
||||
# spans carry inter-line silence with empty text).
|
||||
spans = [Span(voice_id=s.voice_id, text=(s.text or "").strip(),
|
||||
pause_ms_after=max(0, int(s.pause_ms_after)), speed=s.speed)
|
||||
for s in c.spans if ((s.text and s.text.strip()) or s.pause_ms_after > 0)]
|
||||
if spans:
|
||||
chapters.append(Chapter(title=c.title or f"Chapter {i + 1}", spans=spans))
|
||||
plan = AudiobookPlan(chapters=chapters)
|
||||
return StreamingResponse(
|
||||
_render_longform_sse(
|
||||
plan, default_voice=req.default_voice, language=req.language,
|
||||
fmt=req.format, bitrate=req.bitrate,
|
||||
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
|
||||
lexicon=req.lexicon, job_type="story",
|
||||
),
|
||||
media_type="text/event-stream",
|
||||
)
|
||||
|
||||
|
||||
# ── Durable resume: interrupted longform renders ────────────────────────────
|
||||
|
||||
|
||||
def _chapters_done(job_id: str) -> int:
|
||||
"""Count chapters that finished rendering, from the job's persisted events.
|
||||
Best-effort (0 if unavailable) — used only to show resume progress."""
|
||||
try:
|
||||
from core import job_store
|
||||
n = 0
|
||||
for ev in job_store.events_since(job_id, 0, limit=100_000):
|
||||
try:
|
||||
if json.loads(ev["payload"]).get("type") == "chapter":
|
||||
n += 1
|
||||
except (ValueError, KeyError, TypeError):
|
||||
continue
|
||||
return n
|
||||
except Exception:
|
||||
return 0
|
||||
|
||||
|
||||
@router.get("/audiobook/jobs")
|
||||
def list_resumable_jobs() -> dict:
|
||||
"""List interrupted longform renders that can be resumed — a work dir that
|
||||
still holds a resume manifest (a job left mid-render by a crash/quit). The
|
||||
ids come from scanning the filesystem, so the UI can offer one-click resume."""
|
||||
from core import job_store
|
||||
|
||||
out = []
|
||||
for e in longform_resume.scan_resumable():
|
||||
jid = e["job_id"]
|
||||
manifest = longform_resume.load_manifest_file(e["manifest_path"]) or {}
|
||||
job = job_store.get(jid) or {}
|
||||
out.append({
|
||||
"job_id": jid,
|
||||
"type": e["job_type"],
|
||||
"status": job.get("status", "interrupted"),
|
||||
"title": manifest.get("title", ""),
|
||||
"total_chapters": manifest.get("total_chapters", 0),
|
||||
"chapters_done": _chapters_done(jid),
|
||||
"created_at": job.get("created_at"),
|
||||
})
|
||||
return {"jobs": out}
|
||||
|
||||
|
||||
@router.post("/audiobook/resume/{job_id}")
|
||||
async def resume_longform(job_id: str):
|
||||
"""Resume an interrupted longform render from its persisted manifest. The
|
||||
already-rendered chapters are content-addressed in the shared cache, so they
|
||||
return instantly — only the unrendered chapters synthesize again. Streams the
|
||||
same SSE event shape as the original render, under the original job_id."""
|
||||
from services.audiobook import AudiobookPlan, Chapter, Span
|
||||
|
||||
# Find the requested job among the trusted filesystem scan (every path there
|
||||
# is os.listdir-sourced, never request input) and read its manifest via the
|
||||
# scan's own trusted path — the request job_id is used ONLY to *select* an
|
||||
# entry, never to build a path. No request-controlled value reaches a file
|
||||
# operation (CodeQL py/path-injection-safe).
|
||||
entry = next((e for e in longform_resume.scan_resumable()
|
||||
if e["job_id"] == job_id), None)
|
||||
if entry is None:
|
||||
raise HTTPException(status_code=404, detail="No resumable job for that id")
|
||||
manifest = longform_resume.load_manifest_file(entry["manifest_path"])
|
||||
if not manifest:
|
||||
raise HTTPException(status_code=404, detail="No resume manifest for that job")
|
||||
|
||||
chapters = [
|
||||
Chapter(title=c.get("title", ""),
|
||||
spans=[Span(**s) for s in c.get("spans", [])])
|
||||
for c in manifest["plan"]
|
||||
]
|
||||
plan = AudiobookPlan(chapters=chapters)
|
||||
p = manifest.get("params", {})
|
||||
# Retire the interrupted job's manifest (trusted scan path) so it stops
|
||||
# showing as resumable once we've kicked off the fresh-id resume.
|
||||
longform_resume.discard_manifest_file(entry["manifest_path"])
|
||||
# Resume under a FRESH job id (job_id=None → a server uuid in the renderer).
|
||||
# The chapter cache is content-addressed (keyed by chapter content, not the
|
||||
# job id), so the already-rendered chapters still hit instantly — only the
|
||||
# unrendered ones synthesize. Using a fresh id means the request's job_id
|
||||
# never names a work dir / output file (defence-in-depth path-injection).
|
||||
return StreamingResponse(
|
||||
_render_longform_sse(
|
||||
plan, default_voice=p.get("default_voice"), language=p.get("language"),
|
||||
fmt=p.get("fmt", "m4b"), bitrate=p.get("bitrate", "128k"),
|
||||
loudness=p.get("loudness"), cover_path=p.get("cover_path"),
|
||||
metadata=p.get("metadata"), lexicon=p.get("lexicon"),
|
||||
job_type=entry["job_type"],
|
||||
),
|
||||
media_type="text/event-stream",
|
||||
)
|
||||
@@ -298,9 +298,6 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
denoise=True, postprocess_output=True,
|
||||
)
|
||||
audio_out = audios[0]
|
||||
# TODO(#312): this route runs the OmniVoice model directly (not the active
|
||||
# backend), so VoxCPM2 never reaches it. When these routes become
|
||||
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
|
||||
mastered = apply_mastering(
|
||||
audio_out,
|
||||
sample_rate=sr,
|
||||
|
||||
@@ -25,19 +25,12 @@ router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.capture")
|
||||
|
||||
|
||||
def _truthy(value: Optional[str]) -> bool:
|
||||
"""Parse a multipart form flag. Treats '1'/'true'/'yes'/'on'/'auto'
|
||||
(any case) as on; everything else — including None — as off."""
|
||||
return (value or "").strip().lower() in {"1", "true", "yes", "on", "auto"}
|
||||
|
||||
|
||||
@router.post("/transcribe")
|
||||
async def transcribe_audio(
|
||||
audio: UploadFile = File(...),
|
||||
language: Optional[str] = Form(None),
|
||||
model: Optional[str] = Form(None),
|
||||
mode: Optional[str] = Form(None),
|
||||
refine: Optional[str] = Form(None),
|
||||
):
|
||||
"""Transcribe an audio file to text.
|
||||
|
||||
@@ -47,19 +40,10 @@ async def transcribe_audio(
|
||||
model: Whisper model size (legacy; ignored in dual-mode architecture).
|
||||
mode: 'fast' (default) uses MLX Turbo for speed; 'accurate' uses
|
||||
WhisperX with forced alignment for word-level timing.
|
||||
refine: Opt-in local-LLM cleanup of the final text (disfluencies,
|
||||
self-corrections, punctuation) — same pipeline the live
|
||||
dictation socket uses. Off by default so MCP/CLI callers don't
|
||||
pay LLM latency unless they ask; honours the user's
|
||||
Settings → Dictation-refinement config and silently passes
|
||||
through when no LLM backend is configured. The raw ``text``
|
||||
is always returned; ``refined_text`` is added only when the
|
||||
LLM actually changed something.
|
||||
|
||||
Returns:
|
||||
{
|
||||
"text": "full transcription",
|
||||
"refined_text": "cleaned text", # only when refine=true changed it
|
||||
"segments": [ {"start": 0.0, "end": 1.5, "text": "..."}, ... ],
|
||||
"language": "en",
|
||||
"duration_s": 4.2,
|
||||
@@ -107,11 +91,6 @@ async def transcribe_audio(
|
||||
if not full_text and segments:
|
||||
full_text = " ".join(s.get("text", "") for s in segments).strip()
|
||||
|
||||
# Wave 1.1: strip Whisper hallucination loops from the final text.
|
||||
# Segments keep the raw recognition so their timings stay truthful.
|
||||
from services.refinement import collapse_repetitive_artifacts
|
||||
full_text = collapse_repetitive_artifacts(full_text)
|
||||
|
||||
# Calculate audio duration from segments if available
|
||||
duration = 0.0
|
||||
if segments:
|
||||
@@ -119,24 +98,12 @@ async def transcribe_audio(
|
||||
|
||||
detected_lang = result.get("language", language or "unknown")
|
||||
|
||||
# Opt-in Wave 2.1 refinement, mirroring the live-dictation socket
|
||||
# (capture_ws). Off-thread (it's a network call, not GPU); never
|
||||
# raises — maybe_refine swallows failures and a missing LLM into a
|
||||
# None pass-through, so the raw text always stands.
|
||||
refined_text = None
|
||||
if _truthy(refine) and full_text:
|
||||
from services.refinement import maybe_refine
|
||||
refined = await asyncio.to_thread(maybe_refine, full_text)
|
||||
if refined and refined != full_text:
|
||||
refined_text = refined
|
||||
|
||||
logger.info(
|
||||
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s, refined=%s",
|
||||
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s",
|
||||
engine_id, elapsed, duration, "accurate" if use_accurate else "fast",
|
||||
refined_text is not None,
|
||||
)
|
||||
|
||||
response = {
|
||||
return {
|
||||
"text": full_text,
|
||||
"segments": [
|
||||
{
|
||||
@@ -151,9 +118,6 @@ async def transcribe_audio(
|
||||
"transcription_time_s": elapsed,
|
||||
"engine": engine_id,
|
||||
}
|
||||
if refined_text is not None:
|
||||
response["refined_text"] = refined_text
|
||||
return response
|
||||
finally:
|
||||
try:
|
||||
os.unlink(tmp.name)
|
||||
|
||||
@@ -8,12 +8,6 @@ live dictation feedback.
|
||||
Protocol:
|
||||
→ Client sends binary audio frames (16-bit PCM or WebM/Opus blobs)
|
||||
← Server sends JSON messages:
|
||||
|
||||
Opt-in AEC mode (``?aec=1[&sr=16000]``, parity Action 8b): for dictating
|
||||
while the app plays audio. Frames must be raw int16 mono PCM, each tagged
|
||||
with a 1-byte prefix — 0x00 = microphone, 0x01 = playback reference. The
|
||||
server runs an NLMS echo canceller, cleaning the mic against the reference
|
||||
before transcription. Without the param the protocol is unchanged.
|
||||
{"type": "partial", "text": "Hello wor..."} — interim result
|
||||
{"type": "final", "text": "Hello world.", — committed result
|
||||
"segments": [...], "language": "en",
|
||||
@@ -31,7 +25,7 @@ import time
|
||||
|
||||
from fastapi import APIRouter, WebSocket, WebSocketDisconnect
|
||||
|
||||
from api.dependencies import _LOOPBACK_HOSTS, ws_remote_authorized
|
||||
from api.dependencies import _LOOPBACK_HOSTS
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.capture_ws")
|
||||
@@ -50,56 +44,6 @@ MIN_BUFFER_BYTES = 64000 # ~2s of 16-bit mono 16kHz — needs enough WebM frame
|
||||
# to transcribe whatever the user recorded, even short utterances.
|
||||
MIN_FINAL_BUFFER_BYTES = 4000 # ~125ms of 16-bit mono 16kHz
|
||||
|
||||
# ── Dictate-over-playback AEC (parity Action 8b, opt-in) ──────────────────
|
||||
# Activated by the ``?aec=1`` query param. When OFF (the default), the
|
||||
# protocol and behaviour are byte-for-byte unchanged. When ON, the client
|
||||
# streams raw int16 mono PCM frames tagged with a 1-byte type prefix so the
|
||||
# server can tell mic audio from the playback reference it must cancel:
|
||||
_AEC_NEAR = 0x00 # microphone frame (clean it, then buffer for ASR)
|
||||
_AEC_FAR = 0x01 # playback reference frame (feed the echo model only)
|
||||
|
||||
|
||||
def _demux_aec_frame(data: bytes) -> tuple[str, bytes]:
|
||||
"""Split a prefixed AEC binary frame into ``(kind, pcm)``.
|
||||
|
||||
``kind`` is ``"near"`` (mic) or ``"far"`` (playback reference). An empty
|
||||
or prefix-only frame yields an empty payload. Unknown prefixes are treated
|
||||
as ``"near"`` so a malformed tag degrades to plain dictation rather than
|
||||
dropping audio.
|
||||
"""
|
||||
if not data:
|
||||
return "near", b""
|
||||
kind = "far" if data[0] == _AEC_FAR else "near"
|
||||
return kind, data[1:]
|
||||
|
||||
|
||||
def _pcm16_to_wav(pcm: bytes, sample_rate: int) -> str | None:
|
||||
"""Write raw int16 mono PCM to a temp WAV via stdlib ``wave`` (no ffmpeg).
|
||||
|
||||
Used on the AEC path, where frames are already decoded PCM — the cleaned
|
||||
samples have no container, so the ffmpeg-sniffing ``_chunks_to_wav`` would
|
||||
misdetect them. Returns the temp path, or ``None`` for a too-short buffer.
|
||||
"""
|
||||
if not pcm or len(pcm) < 100:
|
||||
return None
|
||||
import wave
|
||||
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".wav")
|
||||
tmp.close()
|
||||
try:
|
||||
with wave.open(tmp.name, "wb") as wf:
|
||||
wf.setnchannels(1)
|
||||
wf.setsampwidth(2) # int16
|
||||
wf.setframerate(sample_rate)
|
||||
wf.writeframes(pcm)
|
||||
return tmp.name
|
||||
except Exception as e:
|
||||
logger.debug("PCM->WAV failed: %s", e)
|
||||
try:
|
||||
os.unlink(tmp.name)
|
||||
except OSError:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
@router.websocket("/ws/transcribe")
|
||||
async def ws_transcribe(websocket: WebSocket):
|
||||
@@ -109,33 +53,13 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
# WebSocket dependency injection differs across FastAPI versions, so we
|
||||
# inline the check before accept(). Without it, any local process could
|
||||
# stream the user's microphone over this endpoint.
|
||||
# Wave 2.3 (remote backend): a non-loopback client that presents the
|
||||
# OMNIVOICE_API_KEY bearer is the thin-client dictation case — the mic
|
||||
# lives on the user's machine, the GPU here — and is allowed through.
|
||||
host = websocket.client.host if websocket.client else None
|
||||
if host not in _LOOPBACK_HOSTS and not ws_remote_authorized(websocket):
|
||||
if host not in _LOOPBACK_HOSTS:
|
||||
await websocket.close(code=1008, reason="loopback origin required")
|
||||
return
|
||||
|
||||
await websocket.accept()
|
||||
|
||||
# Opt-in dictate-over-playback AEC (parity Action 8b). Default OFF →
|
||||
# identical legacy behaviour. When on, frames are 1-byte-tagged raw PCM
|
||||
# and the cleaned mic stream is muxed via stdlib wave (not ffmpeg).
|
||||
aec = None
|
||||
pcm_sr: int | None = None
|
||||
if websocket.query_params.get("aec") in ("1", "true", "on"):
|
||||
try:
|
||||
pcm_sr = int(websocket.query_params.get("sr", "16000"))
|
||||
from services.aec import NlmsEchoCanceller
|
||||
aec = NlmsEchoCanceller(sample_rate=pcm_sr)
|
||||
logger.info("AEC enabled for dictation session (sr=%d)", pcm_sr)
|
||||
except Exception as e:
|
||||
# Bad sr or import failure → fall back to plain dictation.
|
||||
logger.warning("AEC requested but disabled: %s", e)
|
||||
aec = None
|
||||
pcm_sr = None
|
||||
|
||||
audio_chunks: list[bytes] = []
|
||||
total_bytes = 0
|
||||
last_audio_time = time.monotonic()
|
||||
@@ -173,16 +97,6 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
# Empty binary frame also acts as EOF — connection stays open.
|
||||
running = False
|
||||
break
|
||||
if aec is not None:
|
||||
# Tagged PCM: route the playback reference into the echo
|
||||
# model and clean the mic before it reaches the buffer.
|
||||
kind, payload = _demux_aec_frame(data)
|
||||
if kind == "far":
|
||||
aec.push_far_end(payload)
|
||||
continue
|
||||
if not payload:
|
||||
continue
|
||||
data = aec.process_near_end(payload)
|
||||
audio_chunks.append(data)
|
||||
total_bytes += len(data)
|
||||
last_audio_time = time.monotonic()
|
||||
@@ -229,7 +143,7 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
|
||||
# Transcribe current buffer
|
||||
try:
|
||||
text = await _transcribe_buffer(audio_chunks[:], pcm_sr=pcm_sr)
|
||||
text = await _transcribe_buffer(audio_chunks[:])
|
||||
if text and text != partial_text:
|
||||
partial_text = text
|
||||
await _safe_send({
|
||||
@@ -259,16 +173,7 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
# Final transcription on complete buffer — skip if client already gone.
|
||||
if total_bytes > MIN_FINAL_BUFFER_BYTES:
|
||||
try:
|
||||
result = await _transcribe_buffer_full(audio_chunks, pcm_sr=pcm_sr)
|
||||
# Wave 2.1: optional local-LLM refinement of the final text.
|
||||
# Off-thread (network call, not GPU); pass-through on any
|
||||
# failure or when no LLM backend is configured. The raw text
|
||||
# always ships too — clients paste refined_text ?? text.
|
||||
if result.get("text"):
|
||||
from services.refinement import maybe_refine
|
||||
refined = await asyncio.to_thread(maybe_refine, result["text"])
|
||||
if refined and refined != result["text"]:
|
||||
result["refined_text"] = refined
|
||||
result = await _transcribe_buffer_full(audio_chunks)
|
||||
if not await _safe_send({"type": "final", **result}):
|
||||
logger.debug("Skipped final send — client already disconnected")
|
||||
except Exception as e:
|
||||
@@ -292,10 +197,10 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
pass
|
||||
|
||||
|
||||
async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None) -> str:
|
||||
async def _transcribe_buffer(chunks: list[bytes]) -> str:
|
||||
"""Quick partial transcription of the current audio buffer."""
|
||||
|
||||
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
|
||||
tmp = _chunks_to_wav(chunks)
|
||||
if tmp is None:
|
||||
return ""
|
||||
|
||||
@@ -318,9 +223,9 @@ async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None)
|
||||
pass
|
||||
|
||||
|
||||
async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = None) -> dict:
|
||||
async def _transcribe_buffer_full(chunks: list[bytes]) -> dict:
|
||||
"""Full transcription with timing info for the final result."""
|
||||
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
|
||||
tmp = _chunks_to_wav(chunks)
|
||||
if tmp is None:
|
||||
return {"text": "", "segments": [], "language": "unknown",
|
||||
"duration_s": 0, "transcription_time_s": 0, "engine": "none"}
|
||||
@@ -340,12 +245,6 @@ async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = N
|
||||
if not full_text and segments:
|
||||
full_text = " ".join(s.get("text", "") for s in segments).strip()
|
||||
|
||||
# Wave 1.1: strip Whisper hallucination loops from the final
|
||||
# text (the string that gets auto-pasted). Segments keep the
|
||||
# raw recognition so their timings stay truthful.
|
||||
from services.refinement import collapse_repetitive_artifacts
|
||||
full_text = collapse_repetitive_artifacts(full_text)
|
||||
|
||||
duration = max((s.get("end", 0) for s in segments), default=0.0)
|
||||
|
||||
return {
|
||||
|
||||
@@ -258,19 +258,13 @@ async def community_use(item_id: str, name: Optional[str] = Query(None)):
|
||||
raise HTTPException(status_code=503, detail=f"Couldn't add this voice right now. Error: {e}")
|
||||
|
||||
try:
|
||||
# A community "preset" is a synthetic designed voice (rendered from an
|
||||
# instruct string) → kind='design'; a "voice" carries a real reference
|
||||
# clip → kind='clone'. Setting kind makes the persona-gallery
|
||||
# synthetic-only gating work (§R3) instead of defaulting all imports to
|
||||
# 'clone'.
|
||||
kind = "design" if item["type"] == "preset" else "clone"
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO voice_profiles "
|
||||
"(id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at, kind) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
"(id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(profile_id, profile_name, audio_filename, ref_text, instruct,
|
||||
item.get("language", "Auto"), None, item["id"], time.time(), kind),
|
||||
item.get("language", "Auto"), None, item["id"], time.time()),
|
||||
)
|
||||
except Exception:
|
||||
with __import__("contextlib").suppress(OSError):
|
||||
|
||||
@@ -1,35 +0,0 @@
|
||||
"""Voice-design "describe your voice" API (issue #317).
|
||||
|
||||
Maps a free-text voice description onto the existing design parameter space
|
||||
via the deterministic keyword mapper in ``core.describe_voice``. Pure CPU +
|
||||
stdlib — no model, no network — so it imports and responds instantly in any
|
||||
environment, including test/CI without model weights.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from core.describe_voice import parse_description
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
class DescribeRequest(BaseModel):
|
||||
description: str = Field(default="", max_length=2000)
|
||||
|
||||
|
||||
@router.post("/design/describe")
|
||||
def describe_voice(req: DescribeRequest) -> dict:
|
||||
"""Parse a free-text description into design attrs + a validator-safe instruct.
|
||||
|
||||
Response shape::
|
||||
|
||||
{
|
||||
"attrs": {"Gender": "female", "Age": "elderly", ... or "Auto"},
|
||||
"instruct": "female, elderly, low pitch, british accent",
|
||||
"matched": [{"category": "Age", "token": "elderly", "phrase": "elderly"}, ...],
|
||||
"unmatched": ["slightly raspy"]
|
||||
}
|
||||
"""
|
||||
return parse_description(req.description)
|
||||
@@ -25,7 +25,6 @@ from services.segmentation import (
|
||||
assign_speakers_heuristic,
|
||||
clean_up_segments,
|
||||
)
|
||||
from services.onset_align import snap_segment_starts
|
||||
from services import dub_pipeline
|
||||
|
||||
router = APIRouter()
|
||||
@@ -367,11 +366,7 @@ _prep_event_helper = dub_pipeline.prep_event # alias; we keep the module-local
|
||||
|
||||
|
||||
@router.get("/dub/transcribe-stream/{job_id}")
|
||||
async def dub_transcribe_stream(
|
||||
job_id: str,
|
||||
num_speakers: Optional[int] = None,
|
||||
per_segment_refs: bool = True,
|
||||
):
|
||||
async def dub_transcribe_stream(job_id: str, num_speakers: Optional[int] = None):
|
||||
"""Stream per-chunk segments via SSE, then emit diarized final pass.
|
||||
|
||||
Pre-flight checks (missing job, missing audio, ASR not loaded) are emitted
|
||||
@@ -436,7 +431,7 @@ async def dub_transcribe_stream(
|
||||
)
|
||||
scene_cuts = job.get("scene_cuts") or []
|
||||
|
||||
async def _gen_body():
|
||||
async def gen():
|
||||
if preflight_error:
|
||||
yield _sse_event("error", {"detail": preflight_error})
|
||||
return
|
||||
@@ -553,15 +548,6 @@ async def dub_transcribe_stream(
|
||||
detected_lang = part["language"]
|
||||
asr_speaker_turns.extend(part.get("speaker_turns") or [])
|
||||
chunk_segs = segment_transcript(part, duration=t1, scene_cuts=scene_cuts)
|
||||
# #280: Whisper often stretches a segment's start back over
|
||||
# leading music/silence (classic case: speech begins at 0:03,
|
||||
# transcript says 0.0 → the dub plays 3 s early). Snap starts
|
||||
# forward to the actual speech onset. `audio_np` is the same
|
||||
# track ASR ran on — vocals.wav when Demucs succeeded.
|
||||
try:
|
||||
snap_segment_starts(chunk_segs, audio_np, sr)
|
||||
except Exception as e:
|
||||
logger.warning("onset alignment skipped for chunk %d: %s", i, e)
|
||||
chunk_segs = assign_speakers_heuristic(chunk_segs)
|
||||
for s in chunk_segs:
|
||||
s["id"] = f"s{next_seg_id:05x}"
|
||||
@@ -773,61 +759,16 @@ async def dub_transcribe_stream(
|
||||
clones = done.pop().result()
|
||||
break
|
||||
yield _sse_event("ping", {})
|
||||
# Wave 3.2: per-segment clone refs. Cut each long-enough segment's
|
||||
# own reference from the vocals so the dub of each line matches the
|
||||
# prosody of its source line. Short lines fall back to the
|
||||
# per-speaker clone below. Default on; the user can force
|
||||
# per-speaker by disabling it (job["per_segment_refs"]).
|
||||
seg_clones = {}
|
||||
job["per_segment_refs"] = per_segment_refs
|
||||
if per_segment_refs:
|
||||
try:
|
||||
from services.speaker_clone import extract_segment_refs
|
||||
seg_ids_for_clone = [s.get("id", i) for i, s in enumerate(final_segs)]
|
||||
seg_clones = await loop.run_in_executor(
|
||||
_cpu_pool, lambda: extract_segment_refs(
|
||||
vocals_for_clone, final_segs,
|
||||
os.path.dirname(vocals_for_clone),
|
||||
seg_ids=seg_ids_for_clone,
|
||||
),
|
||||
)
|
||||
if seg_clones:
|
||||
job["segment_clones"] = seg_clones
|
||||
except Exception as e:
|
||||
logger.warning("per-segment clone refs skipped: %s", e)
|
||||
|
||||
if clones or seg_clones:
|
||||
if clones:
|
||||
job["speaker_clones"] = clones
|
||||
# Default each segment's profile_id to its detected speaker's
|
||||
# auto-clone — but only if the user hasn't already assigned
|
||||
# something. (#486)
|
||||
#
|
||||
# We prefer the UI-visible `auto:{speaker}` id over the
|
||||
# per-segment `auto-seg:{id}` id even when a per-segment ref
|
||||
# exists, because the dub editor's Voice dropdown only renders
|
||||
# `auto:` options ("From Video → Speaker N"). An `auto-seg:`
|
||||
# value matches no <option>, so the row silently read
|
||||
# "Default" while the speaker was actually bound — exactly the
|
||||
# reported bug. The per-segment ref is NOT lost: dub_generate's
|
||||
# `auto:` branch transparently prefers this segment's own
|
||||
# per-segment ref (job["segment_clones"][seg_id]) when present,
|
||||
# so a row shown as "Speaker 1" still clones from its own line
|
||||
# when that line is long enough.
|
||||
if clones:
|
||||
job["speaker_clones"] = clones
|
||||
# Default each segment's profile_id to its speaker's auto-clone,
|
||||
# but only if the user hasn't already assigned something.
|
||||
for s in final_segs:
|
||||
if s.get("profile_id"):
|
||||
continue
|
||||
spk = s.get("speaker_id") or "Speaker 1"
|
||||
if spk in clones:
|
||||
s["profile_id"] = auto_profile_id(spk)
|
||||
continue
|
||||
# No per-speaker clone for this speaker (too little usable
|
||||
# audio overall) but this single line was long enough for
|
||||
# its own ref — fall back to the per-segment id. The editor
|
||||
# can't render it, but generation still clones correctly.
|
||||
sid = str(s.get("id", ""))
|
||||
if sid and sid in seg_clones:
|
||||
s["profile_id"] = f"auto-seg:{sid}"
|
||||
except Exception as e:
|
||||
logger.warning("speaker_clone extraction skipped: %s", e)
|
||||
|
||||
@@ -856,25 +797,6 @@ async def dub_transcribe_stream(
|
||||
})
|
||||
yield _sse_event("done", {})
|
||||
|
||||
async def gen():
|
||||
# Terminal-event guard (#516): the SSE stream must NEVER close without a
|
||||
# terminal event. Any unanticipated exception in the body (e.g. an ASR
|
||||
# load that escapes the per-chunk handler) previously dropped the
|
||||
# connection, which the frontend can only report as "stream dropped,
|
||||
# likely ASR failed" — hiding the real cause. Emit a structured `error`
|
||||
# (with the actionable hint from build_failure) then `done`, so the user
|
||||
# sees the real failure + a Retry instead of a silent disconnect.
|
||||
try:
|
||||
async for ev in _gen_body():
|
||||
yield ev
|
||||
except Exception as e: # noqa: BLE001 — last-resort stream finalizer
|
||||
logger.exception("transcribe stream crashed (job=%s)", job_id)
|
||||
from core.failure import build_failure
|
||||
f = build_failure(e, stage="transcribe", include_diagnostic=False)
|
||||
detail = f["reason"] + (f" — {f['hint']}" if f.get("hint") else "")
|
||||
yield _sse_event("error", {"detail": detail, "retryable": True})
|
||||
yield _sse_event("done", {})
|
||||
|
||||
return StreamingResponse(
|
||||
gen(),
|
||||
media_type="text/event-stream",
|
||||
@@ -940,14 +862,6 @@ async def dub_transcribe(job_id: str):
|
||||
scene_cuts = job.get("scene_cuts") or []
|
||||
segments = segment_transcript(result, duration=job.get("duration", 0.0), scene_cuts=scene_cuts)
|
||||
|
||||
# #280: snap segment starts forward to the actual speech onset so the
|
||||
# dub doesn't begin seconds before the original speaker does.
|
||||
try:
|
||||
audio_for_onset, onset_sr = sf.read(asr_audio_target, dtype="float32")
|
||||
snap_segment_starts(segments, audio_for_onset, onset_sr)
|
||||
except Exception as e:
|
||||
logger.warning("onset alignment skipped: %s", e)
|
||||
|
||||
diar_pipe = get_diarization_pipeline()
|
||||
if diar_pipe:
|
||||
try:
|
||||
|
||||
@@ -1,7 +1,5 @@
|
||||
import os
|
||||
import io
|
||||
import re
|
||||
import json
|
||||
import time
|
||||
import uuid
|
||||
import asyncio
|
||||
@@ -14,13 +12,6 @@ from core.config import DUB_DIR, dub_seg_path
|
||||
from core.tasks import task_manager
|
||||
from api.routers.dub_core import _get_job
|
||||
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
|
||||
from services.video_retime import (
|
||||
DRIFT_TOLERANCE_S,
|
||||
RetimeError,
|
||||
build_chunk_filter_graph,
|
||||
expand_retime_chunks,
|
||||
prepare_smart_fit_video,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.api")
|
||||
@@ -31,9 +22,6 @@ def _unique_stamp() -> str:
|
||||
return f"{time.strftime('%Y%m%dT%H%M%S')}-{uuid.uuid4().hex[:8]}"
|
||||
|
||||
|
||||
_SAFE_LANG = re.compile(r"^[A-Za-z0-9_-]{1,32}$")
|
||||
|
||||
|
||||
def _native_save(source: str, destination: str, display_name: str, media_type: str):
|
||||
"""Copy a generated export file to a user-chosen destination and return JSON."""
|
||||
import shutil
|
||||
@@ -170,23 +158,15 @@ async def dub_list_tracks(job_id: str):
|
||||
return {"tracks": job.get("dubbed_tracks", {})}
|
||||
|
||||
|
||||
def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool,
|
||||
fitted_segments: "list[dict] | None" = None) -> str | None:
|
||||
def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool) -> str | None:
|
||||
"""Build a temp SRT from job segments for use with ffmpeg's subtitles filter.
|
||||
|
||||
Returned path is already ffmpeg-filter-safe (plain ASCII basename under exports_dir).
|
||||
Returns None if there are no segments to render.
|
||||
|
||||
``fitted_segments`` (Smart Fit): {id, start, end} cue records on the
|
||||
fitted timeline — when provided, cue times come from there instead of
|
||||
the original ``job["segments"]`` timings, so burned subs track the
|
||||
retimed video / fitted audio rather than the source timeline.
|
||||
"""
|
||||
segments = job.get("segments", [])
|
||||
if not segments:
|
||||
return None
|
||||
if fitted_segments:
|
||||
segments = _apply_fitted_times(segments, fitted_segments)
|
||||
lines = []
|
||||
for i, seg in enumerate(segments):
|
||||
lines.append(str(i + 1))
|
||||
@@ -234,14 +214,41 @@ def _build_video_stretch_filter_graph(
|
||||
if not plan:
|
||||
return "", in_label or f"[{video_input_idx}:v]"
|
||||
|
||||
# Chunk expansion + graph emission live in services.video_retime now so
|
||||
# the Smart Fit batched pipeline shares the exact same boundary math.
|
||||
# With default options the emitted graph is byte-identical to the
|
||||
# original inline implementation.
|
||||
chunks = expand_retime_chunks(plan, orig_dur)
|
||||
chunks: list[tuple[float, float, float]] = [] # (a, b, ratio)
|
||||
cursor = 0.0
|
||||
for entry in plan:
|
||||
a = float(entry["orig_start"])
|
||||
b = float(entry["orig_end"])
|
||||
if a > cursor + 1e-3:
|
||||
chunks.append((cursor, a, 1.0)) # gap or pre-roll at native rate
|
||||
ratio = float(entry["stretch_ratio"])
|
||||
if b > a:
|
||||
chunks.append((a, b, ratio))
|
||||
cursor = max(cursor, b)
|
||||
if orig_dur > cursor + 1e-3:
|
||||
chunks.append((cursor, orig_dur, 1.0)) # tail at native rate
|
||||
chunks = [(a, b, r) for (a, b, r) in chunks if b > a]
|
||||
if not chunks:
|
||||
return "", in_label or f"[{video_input_idx}:v]"
|
||||
return build_chunk_filter_graph(chunks, in_label or f"[{video_input_idx}:v]")
|
||||
|
||||
src = in_label or f"[{video_input_idx}:v]"
|
||||
parts: list[str] = []
|
||||
labels: list[str] = []
|
||||
# `split` lets us tap the same source stream once per chunk without re-
|
||||
# decoding. setpts={ratio}*PTS slows down (ratio > 1) or speeds up
|
||||
# (ratio < 1) each chunk; PTS-STARTPTS first to normalise the timestamp
|
||||
# base after the trim.
|
||||
split_labels = [f"[vsplit{idx}]" for idx in range(len(chunks))]
|
||||
parts.append(f"{src}split={len(chunks)}{''.join(split_labels)}")
|
||||
for idx, ((a, b, ratio), split_lbl) in enumerate(zip(chunks, split_labels)):
|
||||
out_label = f"[vstr{idx}]"
|
||||
labels.append(out_label)
|
||||
parts.append(
|
||||
f"{split_lbl}trim=start={a:.4f}:end={b:.4f},"
|
||||
f"setpts=PTS-STARTPTS,setpts={ratio:.6f}*PTS{out_label}"
|
||||
)
|
||||
parts.append("".join(labels) + f"concat=n={len(chunks)}:v=1:a=0[vstretched]")
|
||||
return ";".join(parts), "[vstretched]"
|
||||
|
||||
|
||||
def _video_stretch_plan_for(job: dict, lang_code: str) -> dict | None:
|
||||
@@ -257,76 +264,6 @@ def _video_stretch_plan_for(job: dict, lang_code: str) -> dict | None:
|
||||
return entry
|
||||
|
||||
|
||||
def _video_retime_plan_for(job: dict, lang_code: str) -> "tuple[str, dict] | None":
|
||||
"""Resolve the video retime plan for ``lang_code`` across both keyspaces.
|
||||
|
||||
Returns ``(kind, entry)`` where kind is ``"stretch_video"`` (legacy
|
||||
Mode B plans — resolution byte-identical to ``_video_stretch_plan_for``)
|
||||
or ``"smart_fit"`` (Phase A ``job["fit_plans"]`` entries, gated on the
|
||||
track actually having been generated under smart_fit so a stale plan
|
||||
from an earlier run can't retime a track re-generated under another
|
||||
strategy). ``None`` when neither applies.
|
||||
"""
|
||||
legacy = _video_stretch_plan_for(job, lang_code)
|
||||
if legacy is not None:
|
||||
return "stretch_video", legacy
|
||||
entry = (job.get("fit_plans") or {}).get(lang_code)
|
||||
track = (job.get("dubbed_tracks") or {}).get(lang_code) or {}
|
||||
if entry and entry.get("plan") and track.get("timing_strategy") == "smart_fit":
|
||||
return "smart_fit", entry
|
||||
return None
|
||||
|
||||
|
||||
def _fitted_segments_for(job: dict, lang_code: "str | None") -> "list[dict] | None":
|
||||
"""Fitted-timeline subtitle cues ({id, start, end}) for a Smart Fit
|
||||
track, or None. Same staleness gate as ``_video_retime_plan_for``."""
|
||||
if not lang_code:
|
||||
return None
|
||||
entry = (job.get("fit_plans") or {}).get(lang_code)
|
||||
track = (job.get("dubbed_tracks") or {}).get(lang_code) or {}
|
||||
if not entry or track.get("timing_strategy") != "smart_fit":
|
||||
return None
|
||||
fitted = entry.get("fitted_segments")
|
||||
return fitted or None
|
||||
|
||||
|
||||
def _apply_fitted_times(segments: list[dict], fitted: list[dict]) -> list[dict]:
|
||||
"""Overlay fitted cue times onto subtitle segments (copies; non-destructive).
|
||||
|
||||
Matches by segment ``id``; when the fitted record carries no ids at all
|
||||
(defensive), falls back to positional pairing. Segments without a match
|
||||
keep their original timings.
|
||||
"""
|
||||
by_id = {str(f["id"]): f for f in fitted if f.get("id") is not None}
|
||||
out: list[dict] = []
|
||||
for i, seg in enumerate(segments):
|
||||
cue = None
|
||||
if seg.get("id") is not None:
|
||||
cue = by_id.get(str(seg["id"]))
|
||||
if cue is None and not by_id and i < len(fitted):
|
||||
cue = fitted[i]
|
||||
if cue is None:
|
||||
out.append(seg)
|
||||
continue
|
||||
patched = dict(seg)
|
||||
patched["start"] = float(cue["start"])
|
||||
patched["end"] = float(cue["end"])
|
||||
out.append(patched)
|
||||
return out
|
||||
|
||||
|
||||
def _burn_subs_allowed(retime_kind: "str | None") -> bool:
|
||||
"""Subtitle burn-in combined with video retime.
|
||||
|
||||
Allowed for ``smart_fit`` (fitted cue records exist, and the burn pass
|
||||
runs AFTER the retime graph so cues land on the retimed timeline) and
|
||||
for plain exports. Still rejected for legacy ``stretch_video``, which
|
||||
has no fitted-cue record — cues would burn at original timestamps onto
|
||||
a re-timed video and drift.
|
||||
"""
|
||||
return retime_kind != "stretch_video"
|
||||
|
||||
|
||||
#: Audio export formats → ffmpeg codec args. Unknown formats fall back to
|
||||
#: AAC/m4a so a bad request can never produce a broken command.
|
||||
_AUDIO_FORMAT_CODECS: dict[str, list[str]] = {
|
||||
@@ -375,11 +312,6 @@ async def dub_download(
|
||||
dual: bool = Query(False, description="When burn_subs=1, render translated on top of italicised original."),
|
||||
out_format: str = Query("m4a", description="Audio-only jobs (#119): output container — wav, m4a, mp3, or flac. Ignored for video jobs."),
|
||||
):
|
||||
# Strict allowlist on the path param BEFORE it reaches any filesystem
|
||||
# path or ffmpeg argv (export dir, retime work path, slice paths). Real
|
||||
# job ids are short uuid slices — alnum/hyphen/underscore only.
|
||||
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", job_id):
|
||||
raise HTTPException(status_code=400, detail="Invalid job id")
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
@@ -458,24 +390,18 @@ async def dub_download(
|
||||
)
|
||||
|
||||
# Determine whether this export should drive video through a per-segment
|
||||
# retime (legacy stretch_video Mode B, or Smart Fit). Retime is keyed off
|
||||
# the default_track's plan because the video can only physically follow
|
||||
# one timeline at a time. If multiple dub tracks are included, only the
|
||||
# default_track is visually in sync — other tracks share the same
|
||||
# (retimed) video. Single-track export is the supported common case.
|
||||
retime_kind: "str | None" = None
|
||||
retime_entry: "dict | None" = None
|
||||
if default_track and default_track != "original":
|
||||
_retime = _video_retime_plan_for(job, default_track)
|
||||
if _retime:
|
||||
retime_kind, retime_entry = _retime
|
||||
stretch_entry = retime_entry if retime_kind == "stretch_video" else None
|
||||
# Subtitle burn under legacy stretch_video would render cues at the
|
||||
# original timestamps onto a re-timed video — they'd drift (no fitted-cue
|
||||
# record exists for that mode). Skip the burn pass in that combo and log;
|
||||
# the user can still export the SRT/VTT separately. Smart Fit DOES carry
|
||||
# fitted cues, so burn+retime is allowed there (burn runs post-retime).
|
||||
if not _burn_subs_allowed(retime_kind) and burn_subs:
|
||||
# stretch graph (Mode B). Stretch is keyed off the default_track's plan
|
||||
# because the video can only physically follow one timeline at a time.
|
||||
# If multiple dub tracks are included and they were generated under
|
||||
# stretch_video, only the default_track is visually in sync — other
|
||||
# tracks share the same (stretched) video. Single-track export is the
|
||||
# supported common case.
|
||||
stretch_entry = _video_stretch_plan_for(job, default_track) if default_track and default_track != "original" else None
|
||||
# Subtitle burn under stretch_video would render cues at the original
|
||||
# timestamps onto a re-timed video — they'd drift. Skip the burn pass
|
||||
# in that combo and log; the user can still export the SRT/VTT
|
||||
# separately and the new-layout timing lives there.
|
||||
if stretch_entry and burn_subs:
|
||||
logger.warning(
|
||||
"stretch_video + burn_subs is not supported in one pass; "
|
||||
"skipping subtitle burn for job %s. Export the SRT/VTT separately.",
|
||||
@@ -483,71 +409,11 @@ async def dub_download(
|
||||
)
|
||||
burn_subs = False
|
||||
|
||||
# Smart Fit: cue times come from the fitted timeline — that's where the
|
||||
# dubbed audio actually sits, whether or not the video retime succeeds.
|
||||
fitted_segments = _fitted_segments_for(job, default_track) if default_track and default_track != "original" else None
|
||||
sub_path = _write_burn_srt(job, exports_dir, stamp, dual, fitted_segments=fitted_segments) if burn_subs else None
|
||||
|
||||
# ── Smart Fit video retime (two-tier) ─────────────────────────────────
|
||||
# Tier 1 (≤48 chunks): single filter_complex graph inlined into the mux
|
||||
# command below. Tier 2: batched slice renders joined by the concat
|
||||
# demuxer into an intermediate file, muxed as an extra input. Failures
|
||||
# fall back to an un-retimed export with a structured warning rather
|
||||
# than failing the whole download.
|
||||
retime_decision = None
|
||||
retime_warning: "dict | None" = None
|
||||
smart_track_dur = 0.0
|
||||
if retime_kind == "smart_fit" and retime_entry:
|
||||
smart_orig_dur = float(retime_entry.get("orig_duration") or job.get("duration") or 0.0)
|
||||
smart_track_dur = float(
|
||||
retime_entry.get("total_duration")
|
||||
or (filtered_tracks.get(default_track) or {}).get("duration")
|
||||
or 0.0
|
||||
)
|
||||
# A fresh export is a fresh user intent — clear any sticky abort flag
|
||||
# from a previous /dub/abort so it can't kill this run's first batch.
|
||||
job.pop("aborted", None)
|
||||
# realpath-normalised + containment-checked inline at the sink (the
|
||||
# file's established pattern — CodeQL does not track the guard
|
||||
# through a helper's return value).
|
||||
_base = os.path.realpath(DUB_DIR)
|
||||
retime_work_path = os.path.realpath(
|
||||
os.path.join(exports_dir, f"retimed_{stamp}.mp4")
|
||||
)
|
||||
if retime_work_path != _base and not retime_work_path.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid export path")
|
||||
try:
|
||||
retime_decision = await prepare_smart_fit_video(
|
||||
job_id=job_id,
|
||||
ffmpeg=ffmpeg,
|
||||
video_path=video_path,
|
||||
plan=retime_entry["plan"],
|
||||
orig_dur=smart_orig_dur,
|
||||
track_dur=smart_track_dur,
|
||||
work_path=retime_work_path,
|
||||
abort_check=lambda: bool(job.get("aborted")),
|
||||
)
|
||||
except Exception as e:
|
||||
if (isinstance(e, RetimeError) and e.stage == "aborted") or job.get("aborted"):
|
||||
raise HTTPException(status_code=409, detail="Export aborted")
|
||||
from core.failure import build_failure
|
||||
retime_warning = build_failure(e, stage="video-retime", include_diagnostic=False)
|
||||
job["last_export_warning"] = {"type": "video_retime_fallback", **retime_warning}
|
||||
logger.error(
|
||||
"Smart Fit video retime failed for job %s — exporting "
|
||||
"without per-segment retime: %s",
|
||||
job_id.replace("\n", " ").replace("\r", " "), e,
|
||||
)
|
||||
sub_path = _write_burn_srt(job, exports_dir, stamp, dual) if burn_subs else None
|
||||
|
||||
cmd = [ffmpeg, "-i", video_path]
|
||||
input_idx = 1
|
||||
|
||||
retimed_idx = None
|
||||
if retime_decision is not None and retime_decision.mode == "file":
|
||||
cmd += ["-i", retime_decision.file_path]
|
||||
retimed_idx = input_idx
|
||||
input_idx += 1
|
||||
|
||||
bg_audio = job.get("no_vocals_path") if preserve_bg else None
|
||||
bg_idx = None
|
||||
if bg_audio and os.path.exists(bg_audio) and filtered_tracks:
|
||||
@@ -563,37 +429,9 @@ async def dub_download(
|
||||
|
||||
filter_parts: list[str] = []
|
||||
video_map = "0:v:0"
|
||||
video_reencode = False
|
||||
if retime_decision is not None:
|
||||
if retime_decision.mode == "filter":
|
||||
filter_parts.append(retime_decision.graph)
|
||||
video_map = retime_decision.label
|
||||
video_reencode = True
|
||||
else:
|
||||
video_map = f"{retimed_idx}:v:0"
|
||||
# Residual drift after the batched render (fps rounding): video
|
||||
# shorter than the fitted track → freeze the last frame out to
|
||||
# the track length. Rare — the predicted tail pad inside the
|
||||
# render usually lands within tolerance.
|
||||
residual = smart_track_dur - retime_decision.video_dur
|
||||
if smart_track_dur and residual > DRIFT_TOLERANCE_S:
|
||||
filter_parts.append(
|
||||
f"[{retimed_idx}:v]tpad=stop_mode=clone:stop_duration={residual:.4f}[vtpad]"
|
||||
)
|
||||
video_map = "[vtpad]"
|
||||
video_reencode = True
|
||||
if sub_path:
|
||||
esc = _ffmpeg_filter_escape(sub_path)
|
||||
# Burn AFTER any retime so cues (already on the fitted timeline for
|
||||
# Smart Fit) land on the retimed video. Without retime this reduces
|
||||
# to the legacy `[0:v]subtitles=…[vsub]` graph.
|
||||
if video_map.startswith("["):
|
||||
sub_src = video_map
|
||||
elif retimed_idx is not None:
|
||||
sub_src = f"[{retimed_idx}:v]"
|
||||
else:
|
||||
sub_src = "[0:v]"
|
||||
filter_parts.append(f"{sub_src}subtitles='{esc}'[vsub]")
|
||||
filter_parts.append(f"[0:v]subtitles='{esc}'[vsub]")
|
||||
video_map = "[vsub]"
|
||||
if stretch_entry:
|
||||
orig_dur = float(stretch_entry.get("orig_duration") or job.get("duration") or 0.0)
|
||||
@@ -609,32 +447,10 @@ async def dub_download(
|
||||
if include_original:
|
||||
cmd += ["-map", "0:a:0"]
|
||||
|
||||
# Smart Fit drift absorption, audio side: when the retimed video runs
|
||||
# longer than the fitted track (its tail passes through at 1.0× beyond
|
||||
# the last cue, or encoder rounding), pad the dub-track chain with
|
||||
# silence out to the video length so players don't end audio early.
|
||||
apad_dur = 0.0
|
||||
if (
|
||||
retime_decision is not None
|
||||
and smart_track_dur
|
||||
and retime_decision.video_dur - smart_track_dur > DRIFT_TOLERANCE_S
|
||||
):
|
||||
apad_dur = retime_decision.video_dur
|
||||
|
||||
if bg_idx is not None:
|
||||
for i, t in enumerate(tracks_to_process):
|
||||
out_label = f"[aout{i}]"
|
||||
chain = f"[{bg_idx}:a][{t['idx']}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2"
|
||||
if apad_dur:
|
||||
chain += f",apad=whole_dur={apad_dur:.4f}"
|
||||
filter_parts.append(chain + out_label)
|
||||
t["out_label"] = out_label
|
||||
for t in tracks_to_process:
|
||||
cmd += ["-map", t["out_label"]]
|
||||
elif apad_dur:
|
||||
for i, t in enumerate(tracks_to_process):
|
||||
out_label = f"[aout{i}]"
|
||||
filter_parts.append(f"[{t['idx']}:a]apad=whole_dur={apad_dur:.4f}{out_label}")
|
||||
filter_parts.append(f"[{bg_idx}:a][{t['idx']}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2{out_label}")
|
||||
t["out_label"] = out_label
|
||||
for t in tracks_to_process:
|
||||
cmd += ["-map", t["out_label"]]
|
||||
@@ -645,11 +461,10 @@ async def dub_download(
|
||||
if filter_parts:
|
||||
cmd += ["-filter_complex", ";".join(filter_parts)]
|
||||
|
||||
# Burning subs or per-segment video retime both force a real video
|
||||
# re-encode; stream-copy is viable when nothing touches the video
|
||||
# filter chain — including the batched Smart Fit path, whose retimed
|
||||
# intermediate is already encoded with these exact settings.
|
||||
if sub_path or stretch_entry or video_reencode:
|
||||
# Burning subs or per-segment video stretch both force a real video
|
||||
# re-encode; stream-copy is only viable when nothing touches the video
|
||||
# filter chain.
|
||||
if sub_path or stretch_entry:
|
||||
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
|
||||
else:
|
||||
cmd += ["-c:v", "copy"]
|
||||
@@ -682,17 +497,17 @@ async def dub_download(
|
||||
break
|
||||
cmd += [f"-disposition:a:{target_idx}", "default"]
|
||||
|
||||
# When retiming (legacy stretch_video or Smart Fit) the video and audio
|
||||
# durations should match within sub-frame precision, but `-shortest`
|
||||
# can still cut off the trailing frame; let ffmpeg keep both streams.
|
||||
# Otherwise keep the legacy `-shortest` so a slightly-overrunning track
|
||||
# doesn't extend the mux past the video.
|
||||
if not stretch_entry and retime_decision is None:
|
||||
# In stretch_video mode the video and audio durations should match
|
||||
# within sub-frame precision, but `-shortest` can still cut off the
|
||||
# trailing frame; let ffmpeg keep both streams. Otherwise keep the
|
||||
# legacy `-shortest` so a slightly-overrunning track doesn't extend
|
||||
# the mux past the video.
|
||||
if not stretch_entry:
|
||||
cmd += ["-shortest"]
|
||||
cmd += [output_path, "-y"]
|
||||
|
||||
try:
|
||||
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0, job_id=job_id)
|
||||
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0)
|
||||
if rc != 0:
|
||||
raise Exception(stderr.decode(errors="replace") if stderr else "ffmpeg mux non-zero")
|
||||
except asyncio.TimeoutError:
|
||||
@@ -704,14 +519,6 @@ async def dub_download(
|
||||
status_code=500,
|
||||
detail=f"ffmpeg failed to combine video + dubbed audio: {e}. Verify ffmpeg is installed (`ffmpeg -version`), and check that every dubbed track file exists in the job folder.",
|
||||
)
|
||||
finally:
|
||||
# The batched retime intermediate is a full re-encoded video — never
|
||||
# leave it behind (success or failure; it's stamp-unique, no reuse).
|
||||
if retime_decision is not None and retime_decision.mode == "file":
|
||||
try:
|
||||
os.remove(retime_decision.file_path)
|
||||
except OSError as e:
|
||||
logger.debug("cleanup remove failed: %s", e)
|
||||
|
||||
if not os.path.exists(output_path) or os.path.getsize(output_path) == 0:
|
||||
raise HTTPException(status_code=500, detail="ffmpeg mux produced no output file")
|
||||
@@ -721,22 +528,12 @@ async def dub_download(
|
||||
safe_name = ''.join(c for c in base_name if c.isalnum() or c in '-_ ').strip() or 'output'
|
||||
dl_name = f"dubbed_{safe_name}_{stamp}.mp4"
|
||||
|
||||
# Structured warning surface for the Smart Fit fallback ladder: header is
|
||||
# a fixed ASCII token (FileResponse headers must be latin-1 safe); the
|
||||
# full build_failure payload is persisted on the job for the UI to read.
|
||||
extra_headers = {}
|
||||
if retime_warning is not None:
|
||||
extra_headers["X-Dub-Export-Warning"] = "video-retime-fallback"
|
||||
|
||||
if save_path:
|
||||
result = _native_save(output_path, save_path, dl_name, media_type="video/mp4")
|
||||
if retime_warning is not None:
|
||||
result["warning"] = {"type": "video_retime_fallback", **retime_warning}
|
||||
return result
|
||||
return _native_save(output_path, save_path, dl_name, media_type="video/mp4")
|
||||
|
||||
return FileResponse(
|
||||
output_path, media_type="video/mp4",
|
||||
headers={"Content-Disposition": f'attachment; filename="{dl_name}"', **extra_headers},
|
||||
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
|
||||
)
|
||||
|
||||
|
||||
@@ -771,21 +568,6 @@ async def dub_get_media(job_id: str):
|
||||
ext = os.path.splitext(video_path)[1].lower()
|
||||
return FileResponse(video_path, media_type=_MEDIA_TYPES.get(ext, "video/mp4"))
|
||||
|
||||
# One mux at a time per preview file. Without this, two overlapping requests
|
||||
# (e.g. the <video> element remounting right after a re-dub) both ran ffmpeg
|
||||
# against the same output path, and the mtime check below saw the half-written
|
||||
# file as a valid cache — serving a truncated MP4 that left the player stuck
|
||||
# loading forever (#281).
|
||||
_preview_mux_locks: dict[str, asyncio.Lock] = {}
|
||||
|
||||
|
||||
def _preview_lock(path: str) -> asyncio.Lock:
|
||||
lock = _preview_mux_locks.get(path)
|
||||
if lock is None:
|
||||
lock = _preview_mux_locks.setdefault(path, asyncio.Lock())
|
||||
return lock
|
||||
|
||||
|
||||
@router.get("/dub/preview-video/{job_id}")
|
||||
async def dub_preview_video(
|
||||
job_id: str,
|
||||
@@ -797,11 +579,6 @@ async def dub_preview_video(
|
||||
Caches per lang+preserve_bg combination under exports/preview_{lang}_{bg}.mp4.
|
||||
Cache is invalidated when the underlying dubbed track mtime is newer than the cache.
|
||||
"""
|
||||
# Strict allowlist on the path param BEFORE it reaches any filesystem
|
||||
# path or ffmpeg argv (exports dir, preview/retime work paths) — same
|
||||
# boundary check as dub_download.
|
||||
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", job_id):
|
||||
raise HTTPException(status_code=400, detail="Invalid job id")
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
@@ -822,92 +599,23 @@ async def dub_preview_video(
|
||||
bg_audio = job.get("no_vocals_path") if preserve_bg else None
|
||||
has_bg = bool(bg_audio and os.path.exists(bg_audio))
|
||||
|
||||
if not _SAFE_LANG.match(lang):
|
||||
raise HTTPException(status_code=400, detail="Invalid lang")
|
||||
# realpath-normalised + containment-checked inline BEFORE any filesystem
|
||||
# access so the guard dominates every sink (the file's established
|
||||
# pattern — see dub_preview_segment; CodeQL does not track the guard
|
||||
# through a helper's return value).
|
||||
_base = os.path.realpath(DUB_DIR)
|
||||
exports_dir = os.path.realpath(os.path.join(_base, job_id, "exports"))
|
||||
if not exports_dir.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid job id")
|
||||
exports_dir = os.path.join(DUB_DIR, job_id, "exports")
|
||||
os.makedirs(exports_dir, exist_ok=True)
|
||||
bg_suffix = "bg" if (preserve_bg and has_bg) else "nobg"
|
||||
preview_path = os.path.realpath(
|
||||
os.path.join(exports_dir, f"preview_{lang}_{bg_suffix}.mp4")
|
||||
)
|
||||
if not preview_path.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid path")
|
||||
preview_path = os.path.join(exports_dir, f"preview_{lang}_{bg_suffix}.mp4")
|
||||
|
||||
track_mtime = os.path.getmtime(track_path)
|
||||
cache_ok = (
|
||||
os.path.exists(preview_path)
|
||||
and os.path.getsize(preview_path) > 0
|
||||
and os.path.getmtime(preview_path) >= track_mtime
|
||||
)
|
||||
|
||||
def _cache_ok() -> bool:
|
||||
return (
|
||||
os.path.exists(preview_path)
|
||||
and os.path.getsize(preview_path) > 0
|
||||
and os.path.getmtime(preview_path) >= track_mtime
|
||||
)
|
||||
|
||||
async def _mux_preview():
|
||||
# Mux into a temp file and os.replace() into place so a concurrent
|
||||
# reader never sees a partially-written preview (#281: video stuck
|
||||
# loading forever after a re-dub).
|
||||
mux_path = preview_path + ".tmp.mp4"
|
||||
if not cache_ok:
|
||||
ffmpeg = find_ffmpeg()
|
||||
# Resolve the same retime plan the download path uses (legacy
|
||||
# stretch_video or Smart Fit) so the in-app preview matches export.
|
||||
retime = _video_retime_plan_for(job, lang)
|
||||
retime_kind, retime_entry = retime if retime else (None, None)
|
||||
stretch_entry = retime_entry if retime_kind == "stretch_video" else None
|
||||
|
||||
retime_decision = None
|
||||
smart_track_dur = 0.0
|
||||
if retime_kind == "smart_fit" and retime_entry:
|
||||
smart_orig_dur = float(retime_entry.get("orig_duration") or job.get("duration") or 0.0)
|
||||
smart_track_dur = float(
|
||||
retime_entry.get("total_duration") or track_info.get("duration") or 0.0
|
||||
)
|
||||
job.pop("aborted", None) # fresh user intent — clear sticky abort
|
||||
# realpath-normalised + containment-checked inline at the sink
|
||||
# (same pattern as preview_path above — _base is the realpath
|
||||
# of DUB_DIR from the top of this endpoint).
|
||||
retime_work_path = os.path.realpath(os.path.join(
|
||||
exports_dir, f"preview_retimed_{lang}_{bg_suffix}.tmp.mp4",
|
||||
))
|
||||
if retime_work_path != _base and not retime_work_path.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid export path")
|
||||
try:
|
||||
retime_decision = await prepare_smart_fit_video(
|
||||
job_id=job_id,
|
||||
ffmpeg=ffmpeg,
|
||||
video_path=video_path,
|
||||
plan=retime_entry["plan"],
|
||||
orig_dur=smart_orig_dur,
|
||||
track_dur=smart_track_dur,
|
||||
work_path=retime_work_path,
|
||||
abort_check=lambda: bool(job.get("aborted")),
|
||||
)
|
||||
except Exception as e:
|
||||
if (isinstance(e, RetimeError) and e.stage == "aborted") or job.get("aborted"):
|
||||
raise HTTPException(status_code=409, detail="Preview aborted")
|
||||
# Preview is best-effort: fall back to the un-retimed video
|
||||
# rather than a black player. The export path surfaces the
|
||||
# structured warning; here we just log.
|
||||
retime_decision = None
|
||||
logger.error(
|
||||
"Smart Fit preview retime failed for job %s — previewing "
|
||||
"without per-segment retime: %s",
|
||||
job_id.replace("\n", " ").replace("\r", " "), e,
|
||||
)
|
||||
|
||||
stretch_entry = _video_stretch_plan_for(job, lang)
|
||||
cmd = [ffmpeg, "-i", video_path]
|
||||
input_idx = 1
|
||||
retimed_idx = None
|
||||
if retime_decision is not None and retime_decision.mode == "file":
|
||||
cmd += ["-i", retime_decision.file_path]
|
||||
retimed_idx = input_idx
|
||||
input_idx += 1
|
||||
if preserve_bg and has_bg:
|
||||
cmd += ["-i", bg_audio]
|
||||
bg_idx = input_idx
|
||||
@@ -917,13 +625,12 @@ async def dub_preview_video(
|
||||
cmd += ["-i", track_path]
|
||||
track_idx = input_idx
|
||||
|
||||
# Build filter graph. Under a retime plan we splice the source video
|
||||
# into per-segment chunks, setpts each to match the dub audio
|
||||
# Build filter graph. In stretch_video mode we splice the source
|
||||
# video into per-segment chunks, setpts each to match the dub audio
|
||||
# layout, and concat them — so audio plays at natural rate and the
|
||||
# visuals follow. Otherwise we stream-copy video for speed.
|
||||
filter_parts: list[str] = []
|
||||
video_map = "0:v:0"
|
||||
video_reencode = False
|
||||
if stretch_entry:
|
||||
orig_dur = float(stretch_entry.get("orig_duration") or job.get("duration") or 0.0)
|
||||
graph, vlabel = _build_video_stretch_filter_graph(
|
||||
@@ -932,175 +639,49 @@ async def dub_preview_video(
|
||||
if graph:
|
||||
filter_parts.append(graph)
|
||||
video_map = vlabel
|
||||
elif retime_decision is not None:
|
||||
if retime_decision.mode == "filter":
|
||||
filter_parts.append(retime_decision.graph)
|
||||
video_map = retime_decision.label
|
||||
video_reencode = True
|
||||
else:
|
||||
video_map = f"{retimed_idx}:v:0"
|
||||
residual = smart_track_dur - retime_decision.video_dur
|
||||
if smart_track_dur and residual > DRIFT_TOLERANCE_S:
|
||||
filter_parts.append(
|
||||
f"[{retimed_idx}:v]tpad=stop_mode=clone:stop_duration={residual:.4f}[vtpad]"
|
||||
)
|
||||
video_map = "[vtpad]"
|
||||
video_reencode = True
|
||||
|
||||
# Smart Fit drift absorption (audio): silence-pad the dub chain out
|
||||
# to the retimed video length so the preview doesn't end audio early.
|
||||
apad_dur = 0.0
|
||||
if (
|
||||
retime_decision is not None
|
||||
and smart_track_dur
|
||||
and retime_decision.video_dur - smart_track_dur > DRIFT_TOLERANCE_S
|
||||
):
|
||||
apad_dur = retime_decision.video_dur
|
||||
|
||||
audio_map = f"{track_idx}:a:0"
|
||||
if bg_idx is not None:
|
||||
chain = f"[{bg_idx}:a][{track_idx}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2"
|
||||
if apad_dur:
|
||||
chain += f",apad=whole_dur={apad_dur:.4f}"
|
||||
filter_parts.append(chain + "[aout]")
|
||||
audio_map = "[aout]"
|
||||
elif apad_dur:
|
||||
filter_parts.append(f"[{track_idx}:a]apad=whole_dur={apad_dur:.4f}[aout]")
|
||||
audio_map = "[aout]"
|
||||
filter_parts.append(
|
||||
f"[{bg_idx}:a][{track_idx}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2[aout]"
|
||||
)
|
||||
|
||||
cmd += ["-map", video_map]
|
||||
cmd += ["-map", audio_map]
|
||||
if bg_idx is not None:
|
||||
cmd += ["-map", "[aout]"]
|
||||
else:
|
||||
cmd += ["-map", f"{track_idx}:a:0"]
|
||||
if filter_parts:
|
||||
cmd += ["-filter_complex", ";".join(filter_parts)]
|
||||
|
||||
# Retime path needs a real encode; stream-copy otherwise (the batched
|
||||
# Smart Fit intermediate is already encoded — copy unless tpad'ed).
|
||||
if stretch_entry or video_reencode:
|
||||
# Stretch path needs a real encode; stream-copy otherwise.
|
||||
if stretch_entry:
|
||||
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
|
||||
else:
|
||||
cmd += ["-c:v", "copy"]
|
||||
cmd += ["-c:a", "aac", "-b:a", "192k"]
|
||||
# `-shortest` would cut the retimed video at the (slightly different)
|
||||
# `-shortest` would cut the stretched video at the (slightly different)
|
||||
# audio length and lose the trailing frame; only use it on the copy path.
|
||||
if not stretch_entry and retime_decision is None:
|
||||
if not stretch_entry:
|
||||
cmd += ["-shortest"]
|
||||
cmd += [mux_path, "-y"]
|
||||
|
||||
def _discard_tmp():
|
||||
try:
|
||||
os.remove(mux_path)
|
||||
except OSError as e:
|
||||
logger.debug("cleanup remove failed: %s", e)
|
||||
cmd += [preview_path, "-y"]
|
||||
|
||||
try:
|
||||
rc, _, stderr = await run_ffmpeg(cmd, timeout=900.0, job_id=job_id)
|
||||
rc, _, stderr = await run_ffmpeg(cmd, timeout=900.0)
|
||||
if rc != 0:
|
||||
raise Exception(stderr.decode(errors="replace") if stderr else "ffmpeg mux non-zero")
|
||||
if not os.path.exists(mux_path) or os.path.getsize(mux_path) == 0:
|
||||
raise Exception("preview mux produced empty file")
|
||||
except asyncio.TimeoutError:
|
||||
_discard_tmp()
|
||||
raise HTTPException(status_code=504, detail="preview mux timed out")
|
||||
except HTTPException:
|
||||
_discard_tmp()
|
||||
raise
|
||||
except Exception as e:
|
||||
_discard_tmp()
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"ffmpeg failed to build the preview stream: {str(e)[:300]}. This usually means the source video can't be re-encoded on the fly — try downloading the MP4 instead.",
|
||||
)
|
||||
finally:
|
||||
if retime_decision is not None and retime_decision.mode == "file":
|
||||
try:
|
||||
os.remove(retime_decision.file_path)
|
||||
except OSError as e:
|
||||
# Best-effort scratch cleanup — never fail the export.
|
||||
logger.debug("retime intermediate cleanup failed: %s", e)
|
||||
|
||||
os.replace(mux_path, preview_path)
|
||||
if not os.path.exists(preview_path) or os.path.getsize(preview_path) == 0:
|
||||
raise HTTPException(status_code=500, detail="preview mux produced empty file")
|
||||
|
||||
async with _preview_lock(preview_path):
|
||||
if not _cache_ok():
|
||||
await _mux_preview()
|
||||
|
||||
# no-store: the URL is stable across re-dubs, so any HTTP-level caching
|
||||
# in the WebView would keep showing the previous dub after a re-generate
|
||||
# (#281: "edits don't change the result").
|
||||
return FileResponse(
|
||||
preview_path,
|
||||
media_type="video/mp4",
|
||||
headers={"Cache-Control": "no-store"},
|
||||
)
|
||||
|
||||
|
||||
def _compute_onsets_sync(src_path: str) -> list[float]:
|
||||
"""Blocking part of onset analysis — runs in a worker thread."""
|
||||
import soundfile as sf
|
||||
from services.onset_align import detect_speech_onsets
|
||||
audio, sr = sf.read(src_path, dtype="float32")
|
||||
return detect_speech_onsets(audio, sr)
|
||||
|
||||
|
||||
@router.get("/dub/onsets/{job_id}")
|
||||
async def dub_get_onsets(job_id: str):
|
||||
"""Speech-onset times for the timeline editor's snap-to-onset ticks (#280).
|
||||
|
||||
Prefers the Demucs-isolated vocals track (clean speech energy); falls
|
||||
back to the mixed audio. Computed once per job and cached as
|
||||
``onsets.json`` in the job directory; recomputed if the source audio is
|
||||
newer than the cache (e.g. re-ingest into the same job dir).
|
||||
"""
|
||||
import json
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
vocals = job.get("vocals_path")
|
||||
mix = job.get("audio_path")
|
||||
if vocals and os.path.exists(vocals):
|
||||
src_path, source = vocals, "vocals"
|
||||
elif mix and os.path.exists(mix):
|
||||
src_path, source = mix, "mix"
|
||||
else:
|
||||
raise HTTPException(status_code=404, detail="No audio track available for onset analysis")
|
||||
|
||||
# Containment inlined (not via _safe_job_path): CodeQL can't track the
|
||||
# sanitizer through a helper's return — the file's established idiom.
|
||||
base = os.path.realpath(DUB_DIR)
|
||||
cache_path = os.path.realpath(os.path.join(base, job_id, "onsets.json"))
|
||||
if not cache_path.startswith(base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid job id")
|
||||
try:
|
||||
if (
|
||||
os.path.exists(cache_path)
|
||||
and os.path.getmtime(cache_path) >= os.path.getmtime(src_path)
|
||||
):
|
||||
with open(cache_path, "r", encoding="utf-8") as f:
|
||||
cached = json.load(f)
|
||||
if isinstance(cached, dict) and isinstance(cached.get("onsets"), list):
|
||||
return cached
|
||||
except (OSError, ValueError):
|
||||
pass # unreadable/corrupt cache → recompute below
|
||||
|
||||
try:
|
||||
onsets = await asyncio.to_thread(_compute_onsets_sync, src_path)
|
||||
except Exception as e:
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"Onset analysis failed: {str(e)[:200]}",
|
||||
)
|
||||
|
||||
payload = {"onsets": onsets, "source": source}
|
||||
try:
|
||||
os.makedirs(os.path.dirname(cache_path), exist_ok=True)
|
||||
tmp_path = cache_path + ".tmp"
|
||||
with open(tmp_path, "w", encoding="utf-8") as f:
|
||||
json.dump(payload, f)
|
||||
os.replace(tmp_path, cache_path)
|
||||
except OSError as e:
|
||||
logger.warning("onsets cache write failed for %s: %s", job_id, e)
|
||||
return payload
|
||||
return FileResponse(preview_path, media_type="video/mp4")
|
||||
|
||||
|
||||
@router.get("/dub/thumb/{job_id}")
|
||||
@@ -1148,94 +729,6 @@ async def dub_preview_segment(job_id: str, segment_index: int):
|
||||
return FileResponse(seg_path, media_type="audio/wav")
|
||||
|
||||
|
||||
# ── Second-pass ASR QC (Wave 3.3 / Spec 5) ───────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/dub/qc/{job_id}")
|
||||
async def dub_qc_pass(job_id: str, lang: str = Query(None), drift_threshold: float = Query(0.5)):
|
||||
"""Re-recognize the dubbed audio and flag lines whose recognized text
|
||||
drifts from the target text. Opt-in, never fatal: the dub is untouched —
|
||||
this only annotates segments with a per-line drift score and a measured
|
||||
start/end, surfaced as "verify this line" markers feeding incremental
|
||||
re-dub. The generated text stays authoritative (design delta from
|
||||
pyvideotrans, which overwrites subtitles)."""
|
||||
from services import dub_qc
|
||||
from services.dub_pipeline import put_job, save_job
|
||||
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
tracks = job.get("dubbed_tracks", {})
|
||||
if lang and lang in tracks:
|
||||
wav_path = tracks[lang]["path"]
|
||||
elif tracks:
|
||||
wav_path = list(tracks.values())[0]["path"]
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail="No dubbed audio track generated yet")
|
||||
if not os.path.exists(wav_path):
|
||||
raise HTTPException(status_code=404, detail="Dubbed audio file not found")
|
||||
|
||||
segments = job.get("segments") or []
|
||||
if not segments:
|
||||
raise HTTPException(status_code=400, detail="Job has no segments")
|
||||
|
||||
def _recognize():
|
||||
from services.asr_backend import get_active_asr_backend
|
||||
backend = get_active_asr_backend()
|
||||
result = backend.transcribe(wav_path, word_timestamps=False)
|
||||
return result.get("segments", []), backend.id
|
||||
|
||||
try:
|
||||
from services.model_manager import _get_gpu_pool
|
||||
loop = asyncio.get_running_loop()
|
||||
recognized, engine_id = await loop.run_in_executor(_get_gpu_pool(), _recognize)
|
||||
except Exception as e:
|
||||
logger.exception("dub QC ASR pass failed for %s", job_id)
|
||||
raise HTTPException(status_code=500, detail=f"QC transcription failed: {e}")
|
||||
|
||||
seg_ids = job.get("seg_order") or [s.get("id", i) for i, s in enumerate(segments)]
|
||||
scored = dub_qc.score_dub(segments, recognized, drift_threshold=drift_threshold, seg_ids=seg_ids)
|
||||
|
||||
# Annotate each segment (non-destructive — content text untouched).
|
||||
by_id = {q.seg_id: q for q in scored}
|
||||
for i, s in enumerate(segments):
|
||||
sid = str(seg_ids[i]) if i < len(seg_ids) else str(s.get("id", i))
|
||||
q = by_id.get(sid)
|
||||
if q is None:
|
||||
continue
|
||||
s["qc_drift"] = q.drift
|
||||
s["qc_flagged"] = q.flagged
|
||||
s["qc_recognized"] = q.recognized_text
|
||||
if q.new_start is not None:
|
||||
s["qc_measured_start"] = q.new_start
|
||||
s["qc_measured_end"] = q.new_end
|
||||
put_job(job_id, job)
|
||||
save_job(job_id, job)
|
||||
|
||||
flagged = [q for q in scored if q.flagged]
|
||||
payload = json.dumps({"event": "qc_done", "engine": engine_id,
|
||||
"flagged": len(flagged), "total": len(scored)})
|
||||
try:
|
||||
from core import job_store
|
||||
job_store.append_event(job_id, f"data: {payload}\n\n")
|
||||
except Exception as e:
|
||||
# QC event fan-out is best-effort; the scores are already in the response.
|
||||
logger.debug("QC event append failed: %s", e)
|
||||
|
||||
return {
|
||||
"engine": engine_id,
|
||||
"total": len(scored),
|
||||
"flagged_count": len(flagged),
|
||||
"drift_threshold": drift_threshold,
|
||||
"segments": [
|
||||
{"seg_id": q.seg_id, "drift": q.drift, "flagged": q.flagged,
|
||||
"recognized_text": q.recognized_text,
|
||||
"measured_start": q.new_start, "measured_end": q.new_end}
|
||||
for q in scored
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/dub/download-audio/{job_id}")
|
||||
@router.get("/dub/download-audio/{job_id}/{filename}")
|
||||
async def dub_download_audio(job_id: str, lang: str = Query(None), preserve_bg: bool = Query(True), save_path: str = Query("")):
|
||||
@@ -1313,31 +806,9 @@ def _pick_subtitle_text(seg: dict, dual: bool) -> str:
|
||||
return f"{translated}\n<i>{original}</i>"
|
||||
|
||||
|
||||
# Subtitles deliberately have no ?save_path= variant: they're small text
|
||||
# bodies, so the Tauri side fetches them raw and writes the file itself via
|
||||
# the save_text_file command — the OS save dialog is the write authorization
|
||||
# (#309). The frontend's JSON-envelope save flow stays for binary exports.
|
||||
|
||||
|
||||
def _fitted_cue_times(job: dict, lang: str | None) -> list | None:
|
||||
"""Per-segment (start, end) on the fitted timeline when this job used
|
||||
stretch_video; None to use the original segment times. (Wave 3.1.)"""
|
||||
tracks = job.get("dubbed_tracks", {})
|
||||
lc = lang if (lang and lang in tracks) else (next(iter(tracks), None))
|
||||
entry = _video_stretch_plan_for(job, lc) if lc else None
|
||||
if not entry:
|
||||
return None
|
||||
from services.fitted_subtitles import fitted_cues
|
||||
return fitted_cues(job.get("segments", []), entry["plan"])
|
||||
|
||||
|
||||
@router.get("/dub/srt/{job_id}")
|
||||
@router.get("/dub/srt/{job_id}/{filename}")
|
||||
async def dub_export_srt(
|
||||
job_id: str,
|
||||
dual: bool = False,
|
||||
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
|
||||
):
|
||||
async def dub_export_srt(job_id: str, dual: bool = False):
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
@@ -1346,31 +817,22 @@ async def dub_export_srt(
|
||||
if not segments:
|
||||
raise HTTPException(status_code=400, detail="No transcript segments available")
|
||||
|
||||
# Subtitles must follow the audio the viewer hears, per timing strategy:
|
||||
# Smart Fit (phase B) overlays the fitted segment times directly;
|
||||
# stretch_video (Wave 3.1) regenerates cue times from the stretch plan.
|
||||
# Neither applies → original times.
|
||||
fitted = _fitted_segments_for(job, lang)
|
||||
if fitted:
|
||||
segments = _apply_fitted_times(segments, fitted)
|
||||
cues = None if fitted else _fitted_cue_times(job, lang)
|
||||
|
||||
srt_lines = []
|
||||
for i, seg in enumerate(segments):
|
||||
s, e = cues[i] if cues else (seg["start"], seg["end"])
|
||||
start_ts = _format_srt_time(seg["start"])
|
||||
end_ts = _format_srt_time(seg["end"])
|
||||
srt_lines.append(f"{i + 1}")
|
||||
srt_lines.append(f"{_format_srt_time(s)} --> {_format_srt_time(e)}")
|
||||
srt_lines.append(f"{start_ts} --> {end_ts}")
|
||||
srt_lines.append(_pick_subtitle_text(seg, dual))
|
||||
srt_lines.append("")
|
||||
|
||||
srt_content = "\n".join(srt_lines)
|
||||
base_name = os.path.splitext(job.get('filename', 'video'))[0]
|
||||
suffix = "_dual" if dual else ""
|
||||
dl_name = f"subtitles_{base_name}{suffix}.srt"
|
||||
return Response(
|
||||
content=srt_content,
|
||||
media_type="text/plain",
|
||||
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
|
||||
headers={"Content-Disposition": f'attachment; filename="subtitles_{base_name}{suffix}.srt"'},
|
||||
)
|
||||
|
||||
def _format_vtt_time(seconds):
|
||||
@@ -1382,11 +844,7 @@ def _format_vtt_time(seconds):
|
||||
|
||||
@router.get("/dub/vtt/{job_id}")
|
||||
@router.get("/dub/vtt/{job_id}/{filename}")
|
||||
async def dub_export_vtt(
|
||||
job_id: str,
|
||||
dual: bool = False,
|
||||
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
|
||||
):
|
||||
async def dub_export_vtt(job_id: str, dual: bool = False):
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
@@ -1395,28 +853,22 @@ async def dub_export_vtt(
|
||||
if not segments:
|
||||
raise HTTPException(status_code=400, detail="No transcript segments available")
|
||||
|
||||
# Same strategy-aware cue timing as /dub/srt (see comment there).
|
||||
fitted = _fitted_segments_for(job, lang)
|
||||
if fitted:
|
||||
segments = _apply_fitted_times(segments, fitted)
|
||||
cues = None if fitted else _fitted_cue_times(job, lang)
|
||||
|
||||
vtt_lines = ["WEBVTT", ""]
|
||||
for i, seg in enumerate(segments):
|
||||
s, e = cues[i] if cues else (seg["start"], seg["end"])
|
||||
start_ts = _format_vtt_time(seg["start"])
|
||||
end_ts = _format_vtt_time(seg["end"])
|
||||
vtt_lines.append(str(i + 1))
|
||||
vtt_lines.append(f"{_format_vtt_time(s)} --> {_format_vtt_time(e)}")
|
||||
vtt_lines.append(f"{start_ts} --> {end_ts}")
|
||||
vtt_lines.append(_pick_subtitle_text(seg, dual))
|
||||
vtt_lines.append("")
|
||||
|
||||
vtt_content = "\n".join(vtt_lines)
|
||||
base_name = os.path.splitext(job.get('filename', 'video'))[0]
|
||||
suffix = "_dual" if dual else ""
|
||||
dl_name = f"subtitles_{base_name}{suffix}.vtt"
|
||||
return Response(
|
||||
content=vtt_content,
|
||||
media_type="text/vtt",
|
||||
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
|
||||
headers={"Content-Disposition": f'attachment; filename="subtitles_{base_name}{suffix}.vtt"'},
|
||||
)
|
||||
|
||||
|
||||
|
||||
+101
-239
@@ -3,6 +3,7 @@ import json
|
||||
import logging
|
||||
import time
|
||||
import asyncio
|
||||
import numpy as np
|
||||
import torch
|
||||
import torchaudio
|
||||
from fastapi import APIRouter, HTTPException
|
||||
@@ -14,18 +15,9 @@ from schemas.requests import DubRequest
|
||||
from services.model_manager import get_model, _gpu_pool
|
||||
from services.audio_dsp import apply_mastering, normalize_audio, apply_effects_chain, get_effect_chain
|
||||
from services.audio_io import atomic_save_wav, _safe_torchaudio_save
|
||||
from services.ffmpeg_utils import (
|
||||
find_ffmpeg,
|
||||
spawn_subprocess,
|
||||
# Moved to ffmpeg_utils so the Smart Fit export pipeline (Phase B) can
|
||||
# reuse them; re-imported here so `dub_generate._atempo_chain` /
|
||||
# `_pitch_preserving_stretch` keep working for existing importers.
|
||||
_atempo_chain,
|
||||
_pitch_preserving_stretch,
|
||||
)
|
||||
from services.ffmpeg_utils import find_ffmpeg, spawn_subprocess
|
||||
from services.rvc import apply_rvc, is_enabled as rvc_is_enabled
|
||||
from services.incremental import segment_fingerprint, fit_fingerprint
|
||||
from services.fit_planner import FitParams, plan_fit
|
||||
from services.incremental import segment_fingerprint
|
||||
from services.watermark import embed_watermark
|
||||
from api.routers.dub_core import _get_job, _save_job
|
||||
|
||||
@@ -47,44 +39,75 @@ GAP_OVERFLOW_MAX_S = 0.25
|
||||
GAP_OVERFLOW_BUFFER_S = 0.05
|
||||
|
||||
|
||||
def _sync_job_segments(job: dict, req: DubRequest) -> None:
|
||||
"""Persist the segments this dub was actually generated from back onto the job.
|
||||
def _atempo_chain(ratio: float) -> str:
|
||||
"""Build an `atempo=…,atempo=…` filter chain for arbitrary ratios.
|
||||
|
||||
The editor only sends the (translated / user-edited) segment text in the
|
||||
generate request; the job itself kept the original-language ASR transcript.
|
||||
SRT/VTT export and ffmpeg subtitle burn-in read `job["segments"]`, so they
|
||||
rendered the source language instead of the dub the user just heard (#309).
|
||||
|
||||
Merge strategy: rebuild `job["segments"]` from the request, carrying over
|
||||
per-segment metadata (speaker_id, id, …) from the existing job segment
|
||||
matched by stable id (fallback: index). `text_original` always keeps the
|
||||
source-language text so dual-subtitle layouts can still stack it under the
|
||||
translation.
|
||||
ffmpeg's atempo filter is limited to [0.5, 2.0] per stage. Chaining
|
||||
multiple stages multiplies the effective ratio while keeping each
|
||||
individual stage inside the well-behaved range. Pitch is preserved
|
||||
(WSOLA-style time-domain stretching). ratio > 1 speeds up, < 1
|
||||
slows down.
|
||||
"""
|
||||
if not req.segments:
|
||||
return
|
||||
existing = [s for s in (job.get("segments") or []) if isinstance(s, dict)]
|
||||
by_id = {str(s["id"]): s for s in existing if s.get("id") is not None}
|
||||
seg_ids = req.segment_ids or []
|
||||
merged: list[dict] = []
|
||||
for i, seg in enumerate(req.segments):
|
||||
seg_id = seg_ids[i] if i < len(seg_ids) else None
|
||||
prev = by_id.get(str(seg_id)) if seg_id is not None else None
|
||||
if prev is None and i < len(existing):
|
||||
prev = existing[i]
|
||||
row = dict(prev) if prev else {}
|
||||
if seg_id is not None:
|
||||
# The request id is authoritative — seg_order and the per-segment
|
||||
# WAV manifest are keyed by it.
|
||||
row["id"] = seg_id
|
||||
# Source-language text survives the overwrite so dual-subtitle export
|
||||
# keeps working; never let the translation clobber it.
|
||||
row["text_original"] = row.get("text_original") or row.get("text") or ""
|
||||
row["start"] = seg.start
|
||||
row["end"] = seg.end
|
||||
row["text"] = seg.text
|
||||
merged.append(row)
|
||||
job["segments"] = merged
|
||||
stages: list[str] = []
|
||||
remaining = ratio
|
||||
while remaining > 2.0:
|
||||
stages.append("atempo=2.0")
|
||||
remaining /= 2.0
|
||||
while remaining < 0.5:
|
||||
stages.append("atempo=0.5")
|
||||
remaining /= 0.5
|
||||
stages.append(f"atempo={remaining:.6f}")
|
||||
return ",".join(stages)
|
||||
|
||||
|
||||
async def _pitch_preserving_stretch(
|
||||
wav: torch.Tensor, target_samples: int, sr: int,
|
||||
) -> torch.Tensor:
|
||||
"""Time-stretch a (1, samples) tensor to `target_samples` while
|
||||
preserving pitch, by piping the audio through `ffmpeg atempo`.
|
||||
|
||||
Async so it never blocks the event loop: it's awaited from the `_stream`
|
||||
generator, and each ffmpeg call is ~50-100 ms — a synchronous
|
||||
``subprocess.run`` here froze health-checks / SSE / every concurrent
|
||||
request for the whole multi-segment job.
|
||||
|
||||
Returns a (1, target_samples) tensor on the same device as input.
|
||||
Raises RuntimeError when ffmpeg fails — callers should fall back to
|
||||
naive linear interpolation, accepting the pitch shift, to ensure the
|
||||
output isn't silent.
|
||||
"""
|
||||
wl = int(wav.shape[-1])
|
||||
if target_samples <= 0 or wl == target_samples:
|
||||
return wav
|
||||
ratio = wl / target_samples
|
||||
filter_str = _atempo_chain(ratio)
|
||||
|
||||
# Mono float32 via stdin → ffmpeg → stdout. One subprocess per segment,
|
||||
# run off the event loop so concurrent requests stay responsive.
|
||||
arr = wav.detach().cpu().to(torch.float32).numpy().reshape(-1).astype(np.float32, copy=False)
|
||||
proc = await spawn_subprocess(
|
||||
find_ffmpeg(), "-hide_banner", "-loglevel", "error", "-y",
|
||||
"-f", "f32le", "-ar", str(sr), "-ac", "1", "-i", "pipe:0",
|
||||
"-af", filter_str,
|
||||
"-f", "f32le", "-ar", str(sr), "-ac", "1", "pipe:1",
|
||||
stdin=asyncio.subprocess.PIPE,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, stderr = await proc.communicate(input=arr.tobytes())
|
||||
if proc.returncode != 0 or not stdout:
|
||||
raise RuntimeError(
|
||||
(stderr.decode(errors="replace") or "atempo failed")[:200]
|
||||
)
|
||||
out_arr = np.frombuffer(stdout, dtype=np.float32)
|
||||
# atempo rarely lands exactly on the integer sample count, so
|
||||
# pad/trim to the requested slot length.
|
||||
if len(out_arr) < target_samples:
|
||||
pad = np.zeros(target_samples - len(out_arr), dtype=np.float32)
|
||||
out_arr = np.concatenate([out_arr, pad])
|
||||
elif len(out_arr) > target_samples:
|
||||
out_arr = out_arr[:target_samples]
|
||||
return torch.from_numpy(out_arr.copy()).unsqueeze(0).to(wav.device)
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
@@ -111,15 +134,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
# `seg_i.wav` on disk and slot into the final mix unchanged.
|
||||
regen_only = set(req.regen_only or []) if req.regen_only is not None else None
|
||||
seg_ids = req.segment_ids or []
|
||||
strategy = (req.timing_strategy or "concise").lower()
|
||||
# Strategy-transition guard: smart_fit re-mixes the *natural-rate*
|
||||
# per-segment WAVs from disk. If the previous run used strict_slot,
|
||||
# the on-disk WAVs are slot-squeezed ("slotted") — reusing them would
|
||||
# double-compress. Force one full regen; afterwards seg_wav_kind is
|
||||
# "natural" and partial regen / fit-only re-mix (regen_only=[]) work.
|
||||
# Jobs predating this field have unknown kind → also regen once.
|
||||
if strategy == "smart_fit" and regen_only is not None and job.get("seg_wav_kind") != "natural":
|
||||
regen_only = None
|
||||
# Manifest: stable segment id per current index. Per-segment WAVs are
|
||||
# named by stable id (dub_seg_path) so regen reuses the right audio after
|
||||
# reorder; index-keyed readers (preview/export) resolve via this manifest.
|
||||
@@ -170,17 +184,13 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
if cached_sr != _model.sampling_rate:
|
||||
import torchaudio.functional as AF
|
||||
cached_wav = AF.resample(cached_wav, cached_sr, _model.sampling_rate)
|
||||
# Pad/trim to slot — except smart_fit, whose mix
|
||||
# loop needs the natural-rate length to compute the
|
||||
# audio/video split (the seg_wav_kind guard above
|
||||
# guarantees these cached WAVs are natural-rate).
|
||||
if strategy != "smart_fit":
|
||||
target_samples = int(seg_duration * _model.sampling_rate)
|
||||
current_samples = cached_wav.shape[-1]
|
||||
if target_samples > current_samples:
|
||||
cached_wav = torch.nn.functional.pad(cached_wav, (0, target_samples - current_samples))
|
||||
elif current_samples > target_samples:
|
||||
cached_wav = cached_wav[..., :target_samples]
|
||||
# Pad/trim to slot.
|
||||
target_samples = int(seg_duration * _model.sampling_rate)
|
||||
current_samples = cached_wav.shape[-1]
|
||||
if target_samples > current_samples:
|
||||
cached_wav = torch.nn.functional.pad(cached_wav, (0, target_samples - current_samples))
|
||||
elif current_samples > target_samples:
|
||||
cached_wav = cached_wav[..., :target_samples]
|
||||
all_segment_wavs.append((seg.start, seg.end, cached_wav, _model.sampling_rate))
|
||||
sync_scores.append(getattr(seg, 'sync_ratio', None) or 1.0)
|
||||
_t_cache += time.perf_counter() - _t_cache_0
|
||||
@@ -204,42 +214,18 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
# (see services/speaker_clone.py) live at job["speaker_clones"]
|
||||
# keyed by speaker_id. We use the `auto:` prefix so they can't
|
||||
# collide with persistent voice_profiles.id values.
|
||||
# Wave 3.2: a per-segment clone ref (cut from this line's own
|
||||
# source audio) takes precedence over the per-speaker clone.
|
||||
if profile_id and profile_id.startswith("auto-seg:"):
|
||||
sid = profile_id[len("auto-seg:"):]
|
||||
info = (job.get("segment_clones") or {}).get(sid)
|
||||
if info:
|
||||
ref_audio = info.get("ref_audio")
|
||||
ref_text = info.get("ref_text")
|
||||
profile_id = None # prevent the voice_profiles lookup below
|
||||
|
||||
elif profile_id and profile_id.startswith("auto:"):
|
||||
# #486: an `auto:{speaker}` binding still prefers THIS
|
||||
# segment's own per-segment ref when one exists (cut from
|
||||
# this line's source audio → matches its prosody), falling
|
||||
# back to the per-speaker clone otherwise. This keeps the
|
||||
# Wave 3.2 per-segment-ref quality win while letting every
|
||||
# segment carry the UI-visible `auto:` id the dub editor's
|
||||
# Voice dropdown can actually render ("From Video →
|
||||
# Speaker N"). `seg_id` is closed over from the per-segment
|
||||
# loop below.
|
||||
seg_ref = (job.get("segment_clones") or {}).get(str(seg_id))
|
||||
if seg_ref:
|
||||
ref_audio = seg_ref.get("ref_audio")
|
||||
ref_text = seg_ref.get("ref_text")
|
||||
else:
|
||||
key = profile_id[len("auto:"):]
|
||||
clones = job.get("speaker_clones") or {}
|
||||
# Match by the safe-name key first, fall back to speaker_id.
|
||||
auto = None
|
||||
for spk, info in clones.items():
|
||||
if spk.lower().replace(" ", "_") == key or spk == key:
|
||||
auto = info
|
||||
break
|
||||
if auto:
|
||||
ref_audio = auto.get("ref_audio")
|
||||
ref_text = auto.get("ref_text")
|
||||
if profile_id and profile_id.startswith("auto:"):
|
||||
key = profile_id[len("auto:"):]
|
||||
clones = job.get("speaker_clones") or {}
|
||||
# Match by the safe-name key first, fall back to speaker_id.
|
||||
auto = None
|
||||
for spk, info in clones.items():
|
||||
if spk.lower().replace(" ", "_") == key or spk == key:
|
||||
auto = info
|
||||
break
|
||||
if auto:
|
||||
ref_audio = auto.get("ref_audio")
|
||||
ref_text = auto.get("ref_text")
|
||||
profile_id = None # prevent the voice_profiles lookup below
|
||||
|
||||
if profile_id:
|
||||
@@ -279,9 +265,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
if seg_effect_preset == "raw":
|
||||
return audio_out
|
||||
|
||||
# TODO(#312): this route runs the OmniVoice model directly (not the active
|
||||
# backend), so VoxCPM2 never reaches it. When these routes become
|
||||
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
|
||||
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
|
||||
effect_chain = get_effect_chain(seg_effect_preset)
|
||||
if effect_chain:
|
||||
@@ -327,9 +310,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
if seg_effect_preset == "raw":
|
||||
return audio_out
|
||||
|
||||
# TODO(#312): this route runs the OmniVoice model directly (not the active
|
||||
# backend), so VoxCPM2 never reaches it. When these routes become
|
||||
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
|
||||
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
|
||||
effect_chain = get_effect_chain(seg_effect_preset)
|
||||
if effect_chain:
|
||||
@@ -393,13 +373,13 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
_t_tts_0 = time.perf_counter()
|
||||
seg_effect_preset = getattr(seg, "effect_preset", None) or "broadcast"
|
||||
|
||||
# In concise / stretch_video / smart_fit modes we pass
|
||||
# dur_s=None so the TTS model speaks at its natural rate for
|
||||
# this text length — the whole point of the new timing
|
||||
# strategies is to never squeeze the speech to fit at
|
||||
# synthesis time. strict_slot keeps the legacy behaviour
|
||||
# where dur_s is the slot hint.
|
||||
_dur_for_tts = seg_duration if strategy == "strict_slot" else None
|
||||
# In concise / stretch_video modes we pass dur_s=None so the
|
||||
# TTS model speaks at its natural rate for this text length —
|
||||
# the whole point of the new timing strategies is to never
|
||||
# squeeze the speech to fit. strict_slot keeps the legacy
|
||||
# behaviour where dur_s is the slot hint.
|
||||
_strategy = (req.timing_strategy or "concise").lower()
|
||||
_dur_for_tts = seg_duration if _strategy == "strict_slot" else None
|
||||
|
||||
audio_tensor = await loop.run_in_executor(
|
||||
_gpu_pool, _gen,
|
||||
@@ -416,7 +396,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
target_samples = int(seg_duration * _model.sampling_rate)
|
||||
current_samples = audio_tensor.shape[-1]
|
||||
|
||||
if strategy == "strict_slot":
|
||||
if _strategy == "strict_slot":
|
||||
# Legacy: pad short audio + trim long audio so the mix
|
||||
# loop receives slot-sized buffers. The atempo squeeze
|
||||
# in the mix loop never fires here because we already
|
||||
@@ -426,10 +406,9 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
audio_tensor = torch.nn.functional.pad(audio_tensor, (0, pad_amount))
|
||||
elif current_samples > target_samples:
|
||||
audio_tensor = audio_tensor[..., :target_samples]
|
||||
# concise / stretch_video / smart_fit: keep audio at its
|
||||
# natural length. The mix loop decides per-mode whether to
|
||||
# trim, slip, stretch the video, or split audio/video
|
||||
# retiming (smart_fit) to accommodate it.
|
||||
# concise / stretch_video: keep audio at its natural length.
|
||||
# The mix loop decides per-mode whether to trim, slip, or
|
||||
# stretch the video to accommodate it.
|
||||
|
||||
generated_dur = audio_tensor.shape[-1] / _model.sampling_rate
|
||||
sync_ratio = round(generated_dur / max(seg_duration, 0.01), 3)
|
||||
@@ -508,6 +487,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
_t_diskw = time.perf_counter() - _t_diskw_0
|
||||
|
||||
sr = _model.sampling_rate
|
||||
strategy = (req.timing_strategy or "concise").lower()
|
||||
slot_fit = (req.slot_fit or "time_stretch").lower()
|
||||
overflow_budget_s = max(0.0, float(req.overflow_budget_s or 0.0))
|
||||
|
||||
@@ -522,7 +502,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
# the matching per-segment setpts filter chain on the source video.
|
||||
new_layout: list[tuple[float, float]] = []
|
||||
video_stretch_plan: list[dict] = []
|
||||
fit_plan = None # smart_fit only — services.fit_planner.FitPlan
|
||||
orig_total_dur = float(job.get("duration") or 0.0)
|
||||
|
||||
if strategy == "stretch_video":
|
||||
@@ -556,43 +535,9 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
cursor += max(0.0, orig_total_dur - last_orig_end)
|
||||
new_total_dur = max(cursor, orig_total_dur)
|
||||
total_samples = int(new_total_dur * sr)
|
||||
elif strategy == "smart_fit":
|
||||
# Smart Fit: plan the audio-rate / video-ratio split per segment
|
||||
# from the natural-rate WAV lengths. Pure planning — the mix
|
||||
# loop below applies the audio side; the video side ships as
|
||||
# fit_plans[lang] for the (Phase B) export pipeline.
|
||||
_fo = req.fit_options
|
||||
_fit_defaults = FitParams()
|
||||
fit_params = FitParams(
|
||||
max_audio_only_rate=float(getattr(_fo, "max_audio_only_rate", None) or _fit_defaults.max_audio_only_rate),
|
||||
audio_rate_cap=float(getattr(_fo, "audio_rate_cap", None) or _fit_defaults.audio_rate_cap),
|
||||
video_slow_cap=float(getattr(_fo, "video_slow_cap", None) or _fit_defaults.video_slow_cap),
|
||||
gap_guard_s=float(_fo.gap_guard_s) if _fo is not None and _fo.gap_guard_s is not None else _fit_defaults.gap_guard_s,
|
||||
allow_video_retime=bool(_fo.allow_video_retime) if _fo is not None and _fo.allow_video_retime is not None else _fit_defaults.allow_video_retime,
|
||||
)
|
||||
_seg_order = job.get("seg_order") or []
|
||||
fit_plan = plan_fit(
|
||||
[
|
||||
{
|
||||
"id": _seg_order[i] if i < len(_seg_order) else f"seg_{i}",
|
||||
"start": s,
|
||||
"end": e,
|
||||
}
|
||||
for i, (s, e, _w, _) in enumerate(all_segment_wavs)
|
||||
],
|
||||
[w.shape[-1] / sr for (_s, _e, w, _) in all_segment_wavs],
|
||||
orig_total_dur,
|
||||
fit_params,
|
||||
)
|
||||
total_samples = int(fit_plan.total_duration * sr)
|
||||
else:
|
||||
total_samples = int(orig_total_dur * sr)
|
||||
|
||||
# smart_fit: cue times for the fitted timeline, computed from the
|
||||
# ACTUAL stretched/trimmed sample positions in the mix loop below —
|
||||
# not from the plan — so subtitles land exactly on the audio.
|
||||
fitted_cues: list[dict] = []
|
||||
|
||||
full_audio = torch.zeros(1, total_samples)
|
||||
|
||||
for i, (start, end, wav, _) in enumerate(all_segment_wavs):
|
||||
@@ -615,55 +560,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
"stretch_ratio": round(natural_dur / max(orig_dur, 1e-3), 3),
|
||||
})
|
||||
|
||||
elif strategy == "smart_fit":
|
||||
# Smart Fit: apply the planner's audio_rate via the same
|
||||
# pitch-preserving atempo pipe strict_slot uses, place the
|
||||
# result at the planned new_start, and hard-trim whatever
|
||||
# the caps couldn't absorb. The video side (video_ratio per
|
||||
# chunk) is persisted below for the export pipeline.
|
||||
sf = fit_plan.segments[i]
|
||||
place_at = sf.new_start
|
||||
if sf.audio_rate > 1.0 + 1e-6 and wl > 0:
|
||||
target = max(1, int(round(wl / sf.audio_rate)))
|
||||
try:
|
||||
adjusted = await _pitch_preserving_stretch(
|
||||
adjusted, target, sr,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"atempo stretch failed for seg %d (%.2f×), "
|
||||
"falling back to linear interp: %s",
|
||||
i, sf.audio_rate, e,
|
||||
)
|
||||
adjusted = torch.nn.functional.interpolate(
|
||||
adjusted.unsqueeze(0),
|
||||
size=target,
|
||||
mode='linear',
|
||||
align_corners=False,
|
||||
).squeeze(0)
|
||||
wl = adjusted.shape[-1]
|
||||
# Residual overflow → hard-trim to the segment's new video
|
||||
# slot (fade below keeps the cut pop-free).
|
||||
new_slot_samples = int(max(0.0, sf.new_end - sf.new_start) * sr)
|
||||
if new_slot_samples > 0 and wl > new_slot_samples:
|
||||
adjusted = adjusted[..., :new_slot_samples]
|
||||
wl = adjusted.shape[-1]
|
||||
# Truthful per-segment verdict for the UI badge.
|
||||
entry = {"status": sf.status}
|
||||
if sf.audio_rate > 1.0 + 1e-6:
|
||||
entry["audio_rate"] = round(sf.audio_rate, 3)
|
||||
if sf.video_ratio > 1.0 + 1e-6:
|
||||
entry["video_ratio"] = round(sf.video_ratio, 3)
|
||||
if sf.overflow_s > 0:
|
||||
entry["overflow_s"] = round(sf.overflow_s, 3)
|
||||
fit_status.append(entry)
|
||||
# Cue times from the ACTUAL stretched sample positions.
|
||||
fitted_cues.append({
|
||||
"id": sf.seg_id,
|
||||
"start": round(place_at, 4),
|
||||
"end": round(place_at + wl / sr, 4),
|
||||
})
|
||||
|
||||
elif strategy == "concise":
|
||||
# Mode A: never compress. Allow the audio to extend into the
|
||||
# silent gap before the next seg (existing heuristic) plus
|
||||
@@ -785,9 +681,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
job["language"] = req.language
|
||||
job["language_code"] = lang_code
|
||||
job["timing_strategy"] = strategy
|
||||
# Keep job segments in lock-step with what was just rendered so
|
||||
# subtitle export / burn-in use the translated text (#309).
|
||||
_sync_job_segments(job, req)
|
||||
if strategy == "stretch_video":
|
||||
stretch_plans = job.setdefault("video_stretch_plans", {})
|
||||
stretch_plans[lang_code] = {
|
||||
@@ -795,34 +688,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
"total_duration": round(track_dur, 4),
|
||||
"orig_duration": round(orig_total_dur, 4),
|
||||
}
|
||||
elif strategy == "smart_fit" and fit_plan is not None:
|
||||
# video_stretch_plans stays untouched — smart_fit persists its
|
||||
# own keyspace so a job can carry both without clobbering.
|
||||
_fit_params_payload = {
|
||||
"timing_strategy": strategy,
|
||||
"max_audio_only_rate": fit_params.max_audio_only_rate,
|
||||
"audio_rate_cap": fit_params.audio_rate_cap,
|
||||
"video_slow_cap": fit_params.video_slow_cap,
|
||||
"gap_guard_s": fit_params.gap_guard_s,
|
||||
"allow_video_retime": fit_params.allow_video_retime,
|
||||
}
|
||||
fit_fp = fit_fingerprint(_fit_params_payload)
|
||||
job.setdefault("fit_plans", {})[lang_code] = {
|
||||
# Same dict shape _build_video_stretch_filter_graph consumes.
|
||||
"plan": fit_plan.video_plan,
|
||||
# Cue times from actual stretched sample positions — for
|
||||
# subtitle export on the fitted timeline.
|
||||
"fitted_segments": fitted_cues,
|
||||
"total_duration": round(track_dur, 4),
|
||||
"orig_duration": round(orig_total_dur, 4),
|
||||
"params": _fit_params_payload,
|
||||
"fit_fp": fit_fp,
|
||||
}
|
||||
job["dubbed_tracks"][lang_code]["fit_fp"] = fit_fp
|
||||
# Record what kind of per-segment WAVs are on disk so a later
|
||||
# smart_fit run knows whether partial regen / fit-only re-mix can
|
||||
# reuse them ("natural") or must regen once ("slotted").
|
||||
job["seg_wav_kind"] = "slotted" if strategy == "strict_slot" else "natural"
|
||||
_save_job(job_id, job)
|
||||
|
||||
_t_total = time.perf_counter() - _t_start
|
||||
@@ -919,9 +784,6 @@ async def preview_segment(job_id: str, req: SegmentPreviewRequest):
|
||||
postprocess_output=True,
|
||||
)
|
||||
audio_out = audios[0]
|
||||
# TODO(#312): this route runs the OmniVoice model directly (not the active
|
||||
# backend), so VoxCPM2 never reaches it. When these routes become
|
||||
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
|
||||
mastered = apply_mastering(
|
||||
audio_out,
|
||||
sample_rate=getattr(_model, "sampling_rate", 24000),
|
||||
|
||||
@@ -2,7 +2,6 @@ import os
|
||||
import time
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Optional
|
||||
from fastapi import APIRouter
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
@@ -46,70 +45,6 @@ LANG_NAMES = {
|
||||
"id": "Indonesian", "uk": "Ukrainian",
|
||||
}
|
||||
|
||||
# Regional dialect hints (#280 item 2). Maps a BCP-47 dialect code to the
|
||||
# instruction injected into LLM translation prompts so the output uses that
|
||||
# region's vocabulary and grammar (the reporter's example: choosing Argentina
|
||||
# should yield "Vos sos muy listo", not the Peninsular "Tú eres muy listo").
|
||||
# Only LLM-backed paths can honor these — provider="openai" and the
|
||||
# quality="cinematic" refine pass. Keep entries short: they ride on every
|
||||
# per-segment prompt, so verbosity = wall time.
|
||||
DIALECT_HINTS = {
|
||||
# Spanish
|
||||
"es-ES": "European Spanish (Spain): use tú/vosotros forms and Peninsular vocabulary.",
|
||||
"es-MX": "Mexican Spanish: use tú/ustedes forms and Mexican vocabulary.",
|
||||
"es-AR": "Rioplatense Spanish (Argentina): use voseo — 'vos' with its verb forms (e.g. 'vos sos', 'tenés') and 'ustedes'; prefer Argentinian vocabulary.",
|
||||
"es-CO": "Colombian Spanish: use tú/usted as natural in Colombia and Colombian vocabulary.",
|
||||
"es-CL": "Chilean Spanish: use Chilean vocabulary and expressions.",
|
||||
# Portuguese
|
||||
"pt-BR": "Brazilian Portuguese: use 'você' forms, Brazilian vocabulary and spelling.",
|
||||
"pt-PT": "European Portuguese: use European vocabulary, spelling, and 'tu' where natural.",
|
||||
# English
|
||||
"en-US": "American English: use US spelling and vocabulary.",
|
||||
"en-GB": "British English: use UK spelling and vocabulary.",
|
||||
"en-AU": "Australian English: use Australian spelling and vocabulary.",
|
||||
"en-IN": "Indian English: use Indian English vocabulary and conventions.",
|
||||
# French
|
||||
"fr-FR": "Metropolitan French (France): use standard French vocabulary.",
|
||||
"fr-CA": "Canadian French (Québec): use Québécois vocabulary and expressions.",
|
||||
"fr-BE": "Belgian French: use Belgian vocabulary (e.g. septante, nonante).",
|
||||
# German
|
||||
"de-DE": "Standard German (Germany): use Federal German vocabulary.",
|
||||
"de-AT": "Austrian German: use Austrian vocabulary (e.g. Jänner, Erdapfel).",
|
||||
"de-CH": "Swiss Standard German: use Swiss vocabulary and 'ss' instead of 'ß'.",
|
||||
# Arabic
|
||||
"ar-EG": "Egyptian Arabic: use Egyptian colloquial vocabulary where natural for dubbing.",
|
||||
"ar-SA": "Gulf/Saudi Arabic flavor: prefer vocabulary natural to the Gulf region.",
|
||||
"ar-MA": "Moroccan Arabic (Darija) flavor: prefer vocabulary natural to Morocco.",
|
||||
# Dutch
|
||||
"nl-NL": "Netherlands Dutch: use vocabulary standard in the Netherlands.",
|
||||
"nl-BE": "Belgian Dutch (Flemish): use Flemish vocabulary and expressions.",
|
||||
}
|
||||
|
||||
|
||||
def dialect_clause(dialect: Optional[str]) -> str:
|
||||
"""Prompt fragment for a requested dialect, or '' when unset/unknown.
|
||||
|
||||
Unknown-but-plausible codes (e.g. "es-PE") still get a generic regional
|
||||
clause so users aren't limited to the curated list.
|
||||
"""
|
||||
if not dialect or not str(dialect).strip():
|
||||
return ""
|
||||
code = str(dialect).strip()
|
||||
hint = DIALECT_HINTS.get(code)
|
||||
if hint:
|
||||
return f" Target dialect — {hint}"
|
||||
# Generic fallback for any lang-REGION shaped code we don't curate.
|
||||
if "-" in code:
|
||||
lang, _, region = code.partition("-")
|
||||
lang_name = LANG_NAMES.get(lang, lang)
|
||||
if region:
|
||||
return (
|
||||
f" Use the vocabulary, grammar, and expressions of {lang_name} "
|
||||
f"as spoken in the region '{region}'."
|
||||
)
|
||||
return ""
|
||||
|
||||
|
||||
# Per-language script enforcement. Maps language code → required Unicode
|
||||
# block(s) the translation must contain. Used as a sanity gate after the
|
||||
# LLM responds: if the output contains <50% characters from the expected
|
||||
@@ -157,49 +92,15 @@ _nllb_tokenizer = None
|
||||
_nllb_device = None
|
||||
|
||||
|
||||
def _dialect_flags(req, applied: bool) -> dict:
|
||||
"""Response fields describing whether the requested dialect was honored.
|
||||
|
||||
Empty dict when no dialect was requested, so existing response shapes
|
||||
stay byte-identical for callers that never send one.
|
||||
"""
|
||||
if not getattr(req, "dialect", None):
|
||||
return {}
|
||||
return {"dialect": req.dialect, "dialect_applied": bool(applied)}
|
||||
|
||||
|
||||
def _guess_lang_from_text(segments) -> str | None:
|
||||
"""Best-effort source language from segment text, by script.
|
||||
|
||||
Used only as a last resort when neither the request nor the job carries a
|
||||
detected language. Without this, the bare "en" fallback below forces
|
||||
en -> en on non-English audio (e.g. Korean), which has no Argos package and
|
||||
fails every segment even though ASR detected the language correctly.
|
||||
"""
|
||||
text = " ".join((getattr(s, "text", "") or "") for s in (segments or [])[:8])
|
||||
has = lambda lo, hi: any(lo <= ord(c) <= hi for c in text)
|
||||
if has(0x3040, 0x30FF):
|
||||
return "ja" # Hiragana/Katakana — check before CJK (Japanese uses Kanji too)
|
||||
if has(0xAC00, 0xD7A3) or has(0x1100, 0x11FF):
|
||||
return "ko" # Hangul
|
||||
if has(0x4E00, 0x9FFF):
|
||||
return "zh" # CJK ideographs
|
||||
if has(0x0400, 0x04FF):
|
||||
return "ru" # Cyrillic
|
||||
if has(0x0600, 0x06FF):
|
||||
return "ar" # Arabic
|
||||
return None
|
||||
|
||||
|
||||
def _resolve_source_lang(req: TranslateRequest) -> str:
|
||||
"""Pick source language: explicit request > job.source_lang > text guess > 'en'."""
|
||||
"""Pick source language: explicit request > job.source_lang > 'en' fallback."""
|
||||
if getattr(req, "source_lang", None):
|
||||
return req.source_lang
|
||||
if getattr(req, "job_id", None):
|
||||
job = _get_job(req.job_id)
|
||||
if job and job.get("source_lang"):
|
||||
return job["source_lang"]
|
||||
return _guess_lang_from_text(getattr(req, "segments", None)) or "en"
|
||||
return "en"
|
||||
|
||||
|
||||
def _unload_nllb():
|
||||
@@ -302,8 +203,7 @@ async def dub_translate(req: TranslateRequest):
|
||||
translated = await loop.run_in_executor(_gpu_pool, _translate_nllb)
|
||||
if os.environ.get("OMNIVOICE_UNLOAD_NLLB", "1") == "1":
|
||||
_unload_nllb()
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
**_dialect_flags(req, applied=False)}
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
|
||||
|
||||
# OpenAI / Ollama Local LLM Translation
|
||||
if provider == "openai":
|
||||
@@ -336,16 +236,10 @@ async def dub_translate(req: TranslateRequest):
|
||||
f"only — do not use Latin/Roman letters, do not "
|
||||
f"transliterate, do not output any other language."
|
||||
)
|
||||
# #280 item 2 — regional dialect/vocabulary. Only applied when
|
||||
# the dialect belongs to the target language (a leftover
|
||||
# "es-AR" must not contaminate a French translation).
|
||||
dia_clause = ""
|
||||
if req.dialect and str(req.dialect).lower().startswith(str(tgt_code).lower()[:2]):
|
||||
dia_clause = dialect_clause(req.dialect)
|
||||
return (
|
||||
f"You are a professional dubbing translator. "
|
||||
f"Translate the user's text from {src_name} into "
|
||||
f"{tgt_name}.{script_clause}{dia_clause} "
|
||||
f"{tgt_name}.{script_clause} "
|
||||
f"Reply ONLY with the translated {tgt_name} text, do not "
|
||||
f"add quotes, notes, headers, explanations, or commentary."
|
||||
)
|
||||
@@ -405,8 +299,7 @@ async def dub_translate(req: TranslateRequest):
|
||||
tasks = [loop.run_in_executor(_cpu_pool, _translate_llm, seg) for seg in req.segments]
|
||||
translated = await asyncio.gather(*tasks)
|
||||
translated.sort(key=lambda x: str(x["id"]))
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
**_dialect_flags(req, applied=True)}
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
|
||||
|
||||
# Offline Argos Translate
|
||||
if provider == "argos" or provider == "libretranslate":
|
||||
@@ -460,8 +353,7 @@ async def dub_translate(req: TranslateRequest):
|
||||
return results
|
||||
|
||||
translated = await loop.run_in_executor(_cpu_pool, _translate_argos)
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
**_dialect_flags(req, applied=False)}
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
|
||||
|
||||
# Legacy / API Deep_Translator logic.
|
||||
# Preflight the optional `deep_translator` dep once so we fail with a
|
||||
@@ -570,8 +462,7 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
except Exception as e:
|
||||
logger.debug("non-LLM rate_ratio prediction skipped: %s", e)
|
||||
|
||||
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
"quality_used": "fast", **_dialect_flags(req, applied=False)}
|
||||
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang, "quality_used": "fast"}
|
||||
|
||||
if quality != "cinematic":
|
||||
return base
|
||||
@@ -601,19 +492,12 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
if not pairs:
|
||||
return base
|
||||
|
||||
# #280 item 2: thread the regional-dialect hint into the reflect/adapt
|
||||
# prompts. Guard against a stale dialect from another language.
|
||||
dialect_hint = ""
|
||||
if req.dialect and str(req.dialect).lower().startswith(str(req.target_lang).lower()[:2]):
|
||||
dialect_hint = dialect_clause(req.dialect)
|
||||
|
||||
refined = await cinematic_refine_many(
|
||||
pairs,
|
||||
source_lang=src_lang,
|
||||
target_lang=req.target_lang,
|
||||
glossary=req.glossary,
|
||||
directions=directions,
|
||||
dialect_hint=dialect_hint,
|
||||
executor=_cpu_pool,
|
||||
)
|
||||
refined_by_id = {r["id"]: r for r in refined}
|
||||
@@ -676,5 +560,4 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
"target_lang": req.target_lang,
|
||||
"source_lang": src_lang,
|
||||
"quality_used": "cinematic",
|
||||
**_dialect_flags(req, applied=bool(dialect_hint)),
|
||||
}
|
||||
|
||||
@@ -266,26 +266,11 @@ class SelectEngineRequest(BaseModel):
|
||||
backend_id: str
|
||||
|
||||
|
||||
class SelectEngineResponse(BaseModel):
|
||||
family: str
|
||||
active: str
|
||||
env_override: bool
|
||||
# Routing verdict for the selected engine on THIS host (#21). Always present
|
||||
# so the UI can show a confirm/warning toast on a cpu_fallback pick without
|
||||
# branching on key presence; defaults match a legacy/degraded row.
|
||||
routing_status: str = "cpu_only"
|
||||
effective_device: str = "cpu"
|
||||
routing_reason: str | None = None
|
||||
|
||||
|
||||
@router.post("/engines/select", response_model=SelectEngineResponse)
|
||||
@router.post("/engines/select")
|
||||
def select_engine(req: SelectEngineRequest):
|
||||
"""Persist a family's engine pick to prefs.json. Refuses unknown backends,
|
||||
backends whose deps aren't installed, AND backends that cannot run on THIS
|
||||
host's hardware (routing_status == "unavailable") — so the UI can't silently
|
||||
brick a pipeline by picking an engine that needs a GPU this machine lacks.
|
||||
A `cpu_fallback` pick is allowed (it runs, just slower) — only a hard
|
||||
`unavailable` is blocked. LLM is never routing-gated (its status is "n/a")."""
|
||||
"""Persist a family's engine pick to prefs.json. Refuses unknown backends
|
||||
+ refuses backends whose deps aren't installed (so the UI can't silently
|
||||
brick a pipeline by picking an unavailable engine)."""
|
||||
family = _FAMILIES.get(req.family)
|
||||
if not family:
|
||||
raise HTTPException(400, f"Unknown family: {req.family}. Expected one of tts/asr/llm.")
|
||||
@@ -293,26 +278,12 @@ def select_engine(req: SelectEngineRequest):
|
||||
available = {b["id"]: b for b in module.list_backends()}
|
||||
if req.backend_id not in available:
|
||||
raise HTTPException(400, f"Unknown {req.family} backend: {req.backend_id!r}")
|
||||
entry = available[req.backend_id]
|
||||
if not entry["available"]:
|
||||
reason = entry.get("reason") or "unavailable"
|
||||
if not available[req.backend_id]["available"]:
|
||||
reason = available[req.backend_id].get("reason") or "unavailable"
|
||||
raise HTTPException(400, f"Backend {req.backend_id} not ready: {reason}")
|
||||
# Host-routing gate (no silent CPU fallback). `.get` is defensive so an
|
||||
# older/legacy payload without routing keys still selects cleanly.
|
||||
if entry.get("routing_status") == "unavailable":
|
||||
why = entry.get("routing_reason") or "requires a GPU this host doesn't have"
|
||||
raise HTTPException(
|
||||
400,
|
||||
f"Backend {req.backend_id} can't run on this machine: {why}. "
|
||||
f"Pick an engine with a CPU path, or one that supports this host's GPU.",
|
||||
)
|
||||
prefs.set_(pref_key, req.backend_id)
|
||||
return {
|
||||
"family": req.family,
|
||||
"active": module.active_backend_id(),
|
||||
"env_override": bool(__import__("os").environ.get(f"OMNIVOICE_{req.family.upper()}_BACKEND")),
|
||||
# Echo the routing verdict so the UI can warn on a cpu_fallback pick.
|
||||
"routing_status": entry.get("routing_status", "cpu_only"),
|
||||
"effective_device": entry.get("effective_device", "cpu"),
|
||||
"routing_reason": entry.get("routing_reason"),
|
||||
}
|
||||
|
||||
@@ -59,87 +59,13 @@ def _render_with_pauses(gen_span, segments, sample_rate):
|
||||
parts.append(torch.zeros(*shape, dtype=ref.dtype, device=ref.device))
|
||||
return torch.cat(parts, dim=-1)
|
||||
|
||||
|
||||
def _apply_effect_chain(audio_out, sample_rate, effect_preset, *, skip_mastering=False):
|
||||
"""Shared post-DSP for /generate: preset validation → mastering →
|
||||
effect chain → loudness normalization.
|
||||
|
||||
``skip_mastering`` honors a backend's ``applies_own_mastering`` flag
|
||||
(issue #312): studio engines (e.g. VoxCPM2's native 48 kHz output)
|
||||
opt out of the broadcast Compressor + Reverb chain that's tuned for
|
||||
OmniVoice's 24 kHz clone output. Loudness normalization still runs —
|
||||
it's a benign peak scale. Mirrors ``_run_tts`` in openai_compat.py.
|
||||
"""
|
||||
from services.audio_dsp import (
|
||||
EFFECT_PRESETS, apply_mastering, normalize_audio,
|
||||
apply_effects_chain, get_effect_chain,
|
||||
)
|
||||
|
||||
preset = effect_preset or "broadcast"
|
||||
if preset not in EFFECT_PRESETS:
|
||||
raise ValueError(
|
||||
f"Unknown effect preset: {preset!r}. "
|
||||
f"Valid: {list(EFFECT_PRESETS.keys())}"
|
||||
)
|
||||
|
||||
if preset == "raw":
|
||||
# Raw: skip all DSP — return raw model output
|
||||
return audio_out
|
||||
|
||||
if not skip_mastering:
|
||||
audio_out = apply_mastering(audio_out, sample_rate=sample_rate)
|
||||
chain = get_effect_chain(preset)
|
||||
if chain:
|
||||
audio_out = apply_effects_chain(
|
||||
audio_out, sample_rate=sample_rate, chain=chain,
|
||||
)
|
||||
return normalize_audio(audio_out, target_dBFS=-2.0)
|
||||
|
||||
|
||||
def _oom_friendly_reraise(e):
|
||||
"""Best-effort cache flush + the user-facing OOM hint shared by both
|
||||
inference paths."""
|
||||
import gc
|
||||
import torch
|
||||
gc.collect()
|
||||
if torch.backends.mps.is_available():
|
||||
torch.mps.empty_cache()
|
||||
elif torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
# #278: don't mislabel a torch.compile/Triton/Inductor crash as an
|
||||
# out-of-memory condition. (model_manager's generate wrapper already
|
||||
# retries these eagerly; this only triggers if that retry also died.)
|
||||
from services.model_manager import _is_compile_runtime_failure
|
||||
if _is_compile_runtime_failure(e):
|
||||
raise RuntimeError(
|
||||
f"TTS engine hit a torch.compile/Triton error (not out of memory). "
|
||||
f"Disable torch.compile in Settings → Performance, use the Flush "
|
||||
f"button to reload the model, then regenerate. Underlying error: {e}"
|
||||
) from e
|
||||
# #437: a Permission-denied / exec failure (e.g. a bundled engine binary
|
||||
# that lost its +x bit) is NOT an OOM — don't send the user to the Flush
|
||||
# button; tell them what's actually wrong.
|
||||
es = str(e)
|
||||
if isinstance(e, PermissionError) or "Permission denied" in es or "Errno 13" in es:
|
||||
raise RuntimeError(
|
||||
f"A required engine binary couldn't be executed (permission denied). "
|
||||
f"This usually means a bundled binary lost its execute bit — reinstall, "
|
||||
f"or run `chmod +x` on the engine binary named in the error. "
|
||||
f"Underlying error: {e}"
|
||||
) from e
|
||||
raise RuntimeError(
|
||||
f"TTS engine stopped mid-generation. This usually means it ran out of memory. "
|
||||
f"Try the Flush button to reload the model, then regenerate. Underlying error: {e}"
|
||||
)
|
||||
|
||||
|
||||
def _run_inference(
|
||||
model, text, language, ref_audio_path, ref_text, instruct, duration,
|
||||
num_step, guidance_scale, speed, t_shift, denoise,
|
||||
postprocess_output, layer_penalty_factor, position_temperature,
|
||||
class_temperature, used_seed, effect_preset="broadcast",
|
||||
max_chunk_chars=None, crossfade_ms=None,
|
||||
):
|
||||
from services.audio_dsp import apply_mastering, normalize_audio, apply_effects_chain, get_effect_chain
|
||||
import torch
|
||||
try:
|
||||
if used_seed is not None:
|
||||
@@ -173,124 +99,53 @@ def _run_inference(
|
||||
)[0]
|
||||
audio_out = _render_with_pauses(_gen_span, segments, sr)
|
||||
else:
|
||||
# Wave 1.2: long text is split at sentence boundaries and the
|
||||
# per-chunk audio crossfaded — removes the length ceiling. Short
|
||||
# text takes the single-shot path below unchanged. [pause] inputs
|
||||
# keep the dedicated stitcher above (spans are already short).
|
||||
from services.chunked_tts import (
|
||||
DEFAULT_CROSSFADE_MS, DEFAULT_MAX_CHUNK_CHARS,
|
||||
concatenate_audio_chunks, split_text_into_chunks,
|
||||
audios = model.generate(
|
||||
text=text, language=language, ref_audio=ref_audio_path,
|
||||
ref_text=ref_text, instruct=instruct, duration=duration,
|
||||
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
|
||||
denoise=denoise, postprocess_output=postprocess_output,
|
||||
**kwargs
|
||||
)
|
||||
_max_chars = DEFAULT_MAX_CHUNK_CHARS if max_chunk_chars is None else max_chunk_chars
|
||||
_xfade_ms = DEFAULT_CROSSFADE_MS if crossfade_ms is None else crossfade_ms
|
||||
text_chunks = split_text_into_chunks(text, _max_chars)
|
||||
if len(text_chunks) > 1:
|
||||
parts = []
|
||||
for i, chunk_text in enumerate(text_chunks):
|
||||
# Vary the seed per chunk (deterministically) to avoid
|
||||
# correlated RNG artifacts across chunk boundaries.
|
||||
if used_seed is not None:
|
||||
torch.manual_seed(used_seed + i)
|
||||
parts.append(model.generate(
|
||||
text=chunk_text, language=language, ref_audio=ref_audio_path,
|
||||
ref_text=ref_text, instruct=instruct, duration=None,
|
||||
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
|
||||
denoise=denoise, postprocess_output=postprocess_output,
|
||||
**kwargs
|
||||
)[0])
|
||||
audio_out = concatenate_audio_chunks(parts, sr, _xfade_ms)
|
||||
else:
|
||||
audios = model.generate(
|
||||
text=text, language=language, ref_audio=ref_audio_path,
|
||||
ref_text=ref_text, instruct=instruct, duration=duration,
|
||||
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
|
||||
denoise=denoise, postprocess_output=postprocess_output,
|
||||
**kwargs
|
||||
)
|
||||
audio_out = audios[0]
|
||||
audio_out = audios[0]
|
||||
|
||||
# Apply DSP effect preset. The OmniVoice model never masters its own
|
||||
# output, so mastering always runs here (unchanged behavior).
|
||||
return _apply_effect_chain(audio_out, sr, effect_preset)
|
||||
# Apply DSP effect preset
|
||||
_effect_preset = effect_preset or "broadcast"
|
||||
|
||||
# Validate preset ID
|
||||
from services.audio_dsp import EFFECT_PRESETS
|
||||
if _effect_preset not in EFFECT_PRESETS:
|
||||
raise ValueError(
|
||||
f"Unknown effect preset: {_effect_preset!r}. "
|
||||
f"Valid: {list(EFFECT_PRESETS.keys())}"
|
||||
)
|
||||
|
||||
if _effect_preset == "raw":
|
||||
# Raw: skip all DSP — return raw model output
|
||||
return audio_out
|
||||
|
||||
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
|
||||
_chain = get_effect_chain(_effect_preset)
|
||||
if _chain:
|
||||
mastered_audio = apply_effects_chain(
|
||||
mastered_audio, sample_rate=sr, chain=_chain,
|
||||
)
|
||||
|
||||
return normalize_audio(mastered_audio, target_dBFS=-2.0)
|
||||
|
||||
except ValueError as e:
|
||||
# Don't wrap validation errors in OOM message
|
||||
raise e
|
||||
except Exception as e:
|
||||
_oom_friendly_reraise(e)
|
||||
|
||||
|
||||
def _run_backend_inference(
|
||||
backend, text, language, ref_audio_path, ref_text, instruct, duration,
|
||||
num_step, guidance_scale, speed, denoise, postprocess_output,
|
||||
used_seed, effect_preset="broadcast",
|
||||
max_chunk_chars=None, crossfade_ms=None,
|
||||
):
|
||||
"""Engine-aware twin of :func:`_run_inference` (issue #312).
|
||||
|
||||
Runs the request through a pluggable ``TTSBackend`` adapter instead of the
|
||||
OmniVoice model directly. The adapter protocol is narrower than the
|
||||
OmniVoice-native surface — engine-specific extras (``t_shift``,
|
||||
``layer_penalty_factor``, …) only exist on the native path, which is why
|
||||
OmniVoice itself still goes through ``_run_inference``.
|
||||
"""
|
||||
import torch
|
||||
try:
|
||||
if used_seed is not None:
|
||||
torch.manual_seed(used_seed)
|
||||
|
||||
if language and language.lower() == "auto":
|
||||
language = None
|
||||
|
||||
gen_kwargs = dict(
|
||||
language=language, ref_audio=ref_audio_path, ref_text=ref_text,
|
||||
instruct=instruct, num_step=num_step, guidance_scale=guidance_scale,
|
||||
speed=speed, denoise=denoise, postprocess_output=postprocess_output,
|
||||
import gc
|
||||
gc.collect()
|
||||
if torch.backends.mps.is_available():
|
||||
torch.mps.empty_cache()
|
||||
elif torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
raise RuntimeError(
|
||||
f"TTS engine stopped mid-generation. This usually means it ran out of memory. "
|
||||
f"Try the Flush button to reload the model, then regenerate. Underlying error: {e}"
|
||||
)
|
||||
sr = backend.sample_rate
|
||||
|
||||
# Inline [pause Nms] markers (issue #276) work for every engine — the
|
||||
# silence stitching is model-free.
|
||||
from omnivoice.utils.text import parse_pause_markers
|
||||
segments = parse_pause_markers(text)
|
||||
has_pause = len(segments) > 1 or (segments and segments[0][1] > 0)
|
||||
|
||||
if has_pause:
|
||||
def _gen_span(span_text):
|
||||
# Per-span duration is left to the engine; an explicit overall
|
||||
# `duration` can't be meaningfully split across spans.
|
||||
return backend.generate(span_text, duration=None, **gen_kwargs)
|
||||
audio_out = _render_with_pauses(_gen_span, segments, sr)
|
||||
else:
|
||||
# Wave 1.2: sentence-boundary chunking for long text (see
|
||||
# _run_inference for the rationale; behavior is identical here).
|
||||
from services.chunked_tts import (
|
||||
DEFAULT_CROSSFADE_MS, DEFAULT_MAX_CHUNK_CHARS,
|
||||
concatenate_audio_chunks, split_text_into_chunks,
|
||||
)
|
||||
_max_chars = DEFAULT_MAX_CHUNK_CHARS if max_chunk_chars is None else max_chunk_chars
|
||||
_xfade_ms = DEFAULT_CROSSFADE_MS if crossfade_ms is None else crossfade_ms
|
||||
text_chunks = split_text_into_chunks(text, _max_chars)
|
||||
if len(text_chunks) > 1:
|
||||
parts = []
|
||||
for i, chunk_text in enumerate(text_chunks):
|
||||
if used_seed is not None:
|
||||
torch.manual_seed(used_seed + i)
|
||||
parts.append(backend.generate(chunk_text, duration=None, **gen_kwargs))
|
||||
audio_out = concatenate_audio_chunks(parts, sr, _xfade_ms)
|
||||
else:
|
||||
audio_out = backend.generate(text, duration=duration, **gen_kwargs)
|
||||
|
||||
return _apply_effect_chain(
|
||||
audio_out, sr, effect_preset,
|
||||
skip_mastering=getattr(backend, "applies_own_mastering", False),
|
||||
)
|
||||
|
||||
except ValueError as e:
|
||||
# Don't wrap validation errors in OOM message
|
||||
raise e
|
||||
except Exception as e:
|
||||
_oom_friendly_reraise(e)
|
||||
|
||||
|
||||
@router.post("/generate")
|
||||
@@ -313,88 +168,19 @@ async def generate_speech(
|
||||
profile_id: Optional[str] = Form(None),
|
||||
seed: Optional[int] = Form(None),
|
||||
effect_preset: str = Form("broadcast"),
|
||||
engine: Optional[str] = Form(None),
|
||||
# Wave 1.2 — unlimited-length generation: long text is split at sentence
|
||||
# boundaries and crossfaded. 0 disables chunking (whole text to engine).
|
||||
max_chunk_chars: int = Form(800, ge=0),
|
||||
crossfade_ms: int = Form(50, ge=0, le=1000),
|
||||
):
|
||||
# ── Engine resolution (issue #312) ──────────────────────────────────────
|
||||
# The request runs on the engine selected in Settings (POST /engines/select,
|
||||
# env var OMNIVOICE_TTS_BACKEND wins), or an explicit per-request `engine`
|
||||
# override — same pattern as /ws/tts's `engine` field and /v1/audio/speech's
|
||||
# `model`. Omitting both keeps the historical default (OmniVoice), so
|
||||
# existing API consumers see no change.
|
||||
from services.tts_backend import (
|
||||
OmniVoiceBackend, _mask_hf_tokens, active_backend_id, get_backend_class,
|
||||
)
|
||||
|
||||
engine_id = engine or active_backend_id()
|
||||
try:
|
||||
backend_cls = get_backend_class(engine_id)
|
||||
except ValueError:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=(
|
||||
f"Unknown TTS engine: {engine_id!r}. "
|
||||
"See GET /engines/tts for the list of valid engine ids."
|
||||
),
|
||||
)
|
||||
|
||||
_model = None
|
||||
_backend = None
|
||||
if backend_cls is OmniVoiceBackend:
|
||||
# OmniVoice keeps its native path: it carries the full advanced
|
||||
# parameter surface (t_shift, layer/position/class controls) that the
|
||||
# generic adapter protocol doesn't. Byte-identical to the old behavior.
|
||||
_model = await get_model()
|
||||
else:
|
||||
try:
|
||||
ok, msg = backend_cls.is_available()
|
||||
except Exception as exc:
|
||||
ok, msg = False, f"{type(exc).__name__}: {exc}"
|
||||
if not ok:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"TTS engine '{engine_id}' is not available: {_mask_hf_tokens(msg)}",
|
||||
)
|
||||
# Reuse the per-process instance cache shared with the engine
|
||||
# health-check route so weights load once, not per request.
|
||||
from api.routers.engines import _get_engine_instance
|
||||
_backend = _get_engine_instance(backend_cls)
|
||||
|
||||
# ── Routing gate (#21 — no silent CPU fallback). Computed ONCE per request
|
||||
# (host caps are constant; the per-request engine= override bypasses the
|
||||
# /engines/select gate, so this is the only place it's enforced for synth).
|
||||
from core.device_caps import detect_host_caps
|
||||
from services.engine_routing import resolve_routing, routing_notice
|
||||
_routing = resolve_routing(getattr(backend_cls, "gpu_compat", ("cpu",)), detect_host_caps())
|
||||
if _routing["routing_status"] == "unavailable":
|
||||
# The engine needs an accelerator this host lacks and has no CPU path.
|
||||
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
|
||||
_routing_notice = routing_notice(_routing) # (status, reason) or None
|
||||
_model = await get_model()
|
||||
|
||||
ref_audio_path = None
|
||||
cleanup_ref = False
|
||||
used_seed = seed
|
||||
resolved_profile_id = None
|
||||
history_mode = None # profile.kind when a profile drives; else inferred at insert
|
||||
|
||||
if profile_id:
|
||||
with db_conn() as conn:
|
||||
row = conn.execute("SELECT * FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
if row:
|
||||
resolved_profile_id = profile_id
|
||||
# `kind` is authoritative (0005): 'design' profiles condition on
|
||||
# their deterministic rendered sample + instruct; 'clone' on the
|
||||
# user's reference. Lock always wins (it pins a specific take).
|
||||
# Rows from pre-0004 DBs mid-upgrade may lack the column → fall
|
||||
# back to the legacy is_locked/instruct inference.
|
||||
try:
|
||||
profile_kind = row["kind"] or "clone"
|
||||
except (KeyError, IndexError):
|
||||
profile_kind = "design" if (row["instruct"] and not row["is_locked"] and not row["ref_audio_path"]) else "clone"
|
||||
history_mode = profile_kind
|
||||
if row["is_locked"] and row["locked_audio_path"]:
|
||||
ref_audio_path = os.path.join(VOICES_DIR, row["locked_audio_path"])
|
||||
if not ref_text:
|
||||
@@ -403,19 +189,7 @@ async def generate_speech(
|
||||
instruct = row["instruct"]
|
||||
if used_seed is None and row["seed"] is not None:
|
||||
used_seed = row["seed"]
|
||||
elif profile_kind == "design":
|
||||
# Rendered sample (if present) carries the voice identity;
|
||||
# instruct alone is the fallback for legacy archetype rows.
|
||||
ref_audio_path = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
|
||||
if ref_audio_path and not ref_text and row["ref_text"]:
|
||||
ref_text = row["ref_text"]
|
||||
if not instruct:
|
||||
instruct = row["instruct"]
|
||||
if used_seed is None and row["seed"] is not None:
|
||||
used_seed = row["seed"]
|
||||
elif row["instruct"] and not row["is_locked"] and not row["ref_audio_path"]:
|
||||
# Legacy design-shaped row (pre-0004 archetype materialization
|
||||
# failure path): instruct-only conditioning.
|
||||
elif row["instruct"] and not row["is_locked"]:
|
||||
if not instruct:
|
||||
instruct = row["instruct"]
|
||||
if used_seed is None and row["seed"] is not None:
|
||||
@@ -430,20 +204,6 @@ async def generate_speech(
|
||||
used_seed = row["seed"]
|
||||
if language == "Auto":
|
||||
language = None
|
||||
# #533: a profile's stored language must drive generation when the
|
||||
# request didn't pin one. Without this the German (etc.) archetype
|
||||
# generates with language=None and the model drifts to English —
|
||||
# even though the archetype PREVIEW renders correctly (archetypes.py
|
||||
# passes the language). An EXPLICIT non-Auto request language still
|
||||
# wins; we only fill the gap. `row` is a sqlite3.Row, so guard the
|
||||
# column lookup for pre-language DBs mid-upgrade.
|
||||
if language is None:
|
||||
try:
|
||||
prof_lang = row["language"]
|
||||
except (KeyError, IndexError):
|
||||
prof_lang = None
|
||||
if prof_lang and prof_lang != "Auto":
|
||||
language = prof_lang
|
||||
elif ref_audio is not None:
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".wav") as f:
|
||||
@@ -453,41 +213,16 @@ async def generate_speech(
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
# #308: a transcript-less reference is transcribed with the active ASR
|
||||
# backend (whisperx / faster-whisper / mlx-whisper) instead of the model's
|
||||
# built-in transformers pipeline, which cannot load whisper-large-v3-turbo
|
||||
# on transformers 5.3. On failure ref_text stays None and the model's
|
||||
# fallback behaves exactly as before.
|
||||
if ref_audio_path and not ref_text:
|
||||
from services.asr_backend import transcribe_reference
|
||||
ref_text = await asyncio.get_running_loop().run_in_executor(
|
||||
_gpu_pool, transcribe_reference, ref_audio_path
|
||||
)
|
||||
|
||||
start_time = time.time()
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
if _backend is not None:
|
||||
audio_tensor = await loop.run_in_executor(
|
||||
_gpu_pool, _run_backend_inference,
|
||||
_backend, text, language, ref_audio_path, ref_text, instruct,
|
||||
duration, num_step, guidance_scale, speed, denoise,
|
||||
postprocess_output, used_seed, effect_preset,
|
||||
max_chunk_chars, crossfade_ms,
|
||||
)
|
||||
# Read after generation: engines with lazy model loading report
|
||||
# their real rate only once weights are up.
|
||||
sample_rate = _backend.sample_rate
|
||||
else:
|
||||
audio_tensor = await loop.run_in_executor(
|
||||
_gpu_pool, _run_inference,
|
||||
_model, text, language, ref_audio_path, ref_text, instruct, duration,
|
||||
num_step, guidance_scale, speed, t_shift, denoise,
|
||||
postprocess_output, layer_penalty_factor, position_temperature,
|
||||
class_temperature, used_seed, effect_preset,
|
||||
max_chunk_chars, crossfade_ms,
|
||||
)
|
||||
sample_rate = _model.sampling_rate
|
||||
audio_tensor = await loop.run_in_executor(
|
||||
_gpu_pool, _run_inference,
|
||||
_model, text, language, ref_audio_path, ref_text, instruct, duration,
|
||||
num_step, guidance_scale, speed, t_shift, denoise,
|
||||
postprocess_output, layer_penalty_factor, position_temperature,
|
||||
class_temperature, used_seed, effect_preset,
|
||||
)
|
||||
# Invisible AudioSeal provenance watermark on the final audio. Embedding
|
||||
# was previously only wired into the dub pipeline (dub_generate.py), so
|
||||
# plain TTS came out unmarked despite the setting being on. embed_watermark
|
||||
@@ -496,28 +231,28 @@ async def generate_speech(
|
||||
# generation.
|
||||
from services.watermark import embed_watermark
|
||||
audio_tensor = await loop.run_in_executor(
|
||||
_gpu_pool, embed_watermark, audio_tensor, sample_rate
|
||||
_gpu_pool, embed_watermark, audio_tensor, _model.sampling_rate
|
||||
)
|
||||
gen_time = round(time.time() - start_time, 2)
|
||||
|
||||
audio_id = str(uuid.uuid4())[:8]
|
||||
audio_filename = f"{audio_id}.wav"
|
||||
audio_path = os.path.join(OUTPUTS_DIR, audio_filename)
|
||||
_safe_torchaudio_save(audio_path, audio_tensor, sample_rate)
|
||||
_safe_torchaudio_save(audio_path, audio_tensor, _model.sampling_rate)
|
||||
|
||||
audio_dur = round(audio_tensor.shape[-1] / sample_rate, 2)
|
||||
audio_dur = round(audio_tensor.shape[-1] / _model.sampling_rate, 2)
|
||||
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO generation_history (id, text, mode, language, instruct, profile_id, audio_path, duration_seconds, generation_time, seed, created_at) VALUES (?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(audio_id, text[:200], history_mode or ("clone" if ref_audio_path else "design"),
|
||||
(audio_id, text[:200], "clone" if ref_audio_path else "design",
|
||||
language or "Auto", instruct or "", resolved_profile_id,
|
||||
audio_filename, audio_dur, gen_time, used_seed, time.time())
|
||||
)
|
||||
event_bus.emit("generation_history", {"action": "created", "id": audio_id})
|
||||
|
||||
buffer = io.BytesIO()
|
||||
_safe_torchaudio_save(buffer, audio_tensor, sample_rate, format="wav")
|
||||
_safe_torchaudio_save(buffer, audio_tensor, _model.sampling_rate, format="wav")
|
||||
buffer.seek(0)
|
||||
wav_bytes = buffer.read()
|
||||
|
||||
@@ -526,26 +261,17 @@ async def generate_speech(
|
||||
for i in range(0, len(wav_bytes), chunk_size):
|
||||
yield wav_bytes[i:i + chunk_size]
|
||||
|
||||
_resp_headers = {
|
||||
"X-Audio-Id": audio_id,
|
||||
"X-Gen-Time": str(gen_time),
|
||||
"X-Audio-Path": audio_filename,
|
||||
"X-Seed": str(used_seed) if used_seed is not None else "",
|
||||
"X-Audio-Duration": str(audio_dur),
|
||||
"Content-Length": str(len(wav_bytes)),
|
||||
}
|
||||
# Routing notice (#21): cpu_fallback or accelerated-with-caveat only;
|
||||
# the WAV body is binary so the header channel is the carrier.
|
||||
if _routing_notice:
|
||||
from services.engine_routing import header_safe_reason
|
||||
_resp_headers["X-OmniVoice-Routing"] = _routing_notice[0]
|
||||
_hr = header_safe_reason(_routing_notice[1])
|
||||
if _hr:
|
||||
_resp_headers["X-OmniVoice-Routing-Reason"] = _hr
|
||||
return StreamingResponse(
|
||||
_stream_wav(),
|
||||
media_type="audio/wav",
|
||||
headers=_resp_headers,
|
||||
headers={
|
||||
"X-Audio-Id": audio_id,
|
||||
"X-Gen-Time": str(gen_time),
|
||||
"X-Audio-Path": audio_filename,
|
||||
"X-Seed": str(used_seed) if used_seed is not None else "",
|
||||
"X-Audio-Duration": str(audio_dur),
|
||||
"Content-Length": str(len(wav_bytes)),
|
||||
}
|
||||
)
|
||||
except HTTPException:
|
||||
raise
|
||||
@@ -582,29 +308,9 @@ def _safe_output_path(name):
|
||||
|
||||
@router.get("/history")
|
||||
def list_history():
|
||||
"""Newest 50 generations whose audio still exists on disk.
|
||||
|
||||
Rows whose WAV was deleted out-of-band (cleared outputs dir, manual
|
||||
cleanup) used to come back anyway and render dead players that 404 on
|
||||
every fetch; prune them here so the UI never sees them again."""
|
||||
with db_conn() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM generation_history ORDER BY created_at DESC LIMIT 50"
|
||||
).fetchall()
|
||||
alive, stale_ids = [], []
|
||||
for r in rows:
|
||||
p = _safe_output_path(r["audio_path"]) if r["audio_path"] else None
|
||||
if r["audio_path"] and (not p or not os.path.exists(p)):
|
||||
stale_ids.append(r["id"])
|
||||
else:
|
||||
alive.append(dict(r))
|
||||
if stale_ids:
|
||||
conn.executemany(
|
||||
"DELETE FROM generation_history WHERE id=?",
|
||||
[(i,) for i in stale_ids],
|
||||
)
|
||||
logger.info("pruned %d stale history rows (audio file gone)", len(stale_ids))
|
||||
return alive
|
||||
rows = conn.execute("SELECT * FROM generation_history ORDER BY created_at DESC LIMIT 50").fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
@router.delete("/history")
|
||||
def clear_history():
|
||||
|
||||
@@ -1,158 +0,0 @@
|
||||
"""Longform Job Library (PR 7).
|
||||
|
||||
``GET /longform/jobs`` — list finished Audiobook + Story renders so the user can
|
||||
re-download them from the Projects view. The render itself (the m4b/mp3) already
|
||||
landed in ``OUTPUTS_DIR`` and is served at ``/audio/<output>``; here we just
|
||||
recover, from each finished job's persisted SSE tail, the output filename plus
|
||||
the chapter count and duration the ``done`` event carried.
|
||||
|
||||
Pure recovery, no synthesis. Defensive by construction: a job whose ``done``
|
||||
event is missing or unparseable is skipped, never surfaced and never a 500.
|
||||
|
||||
The work lives in :func:`build_longform_library`, a pure function over the
|
||||
job-store callables, so it's unit-testable without importing ``main`` (and the
|
||||
torch graph behind it).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Callable, Optional
|
||||
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
logger = logging.getLogger("omnivoice.longform_jobs")
|
||||
router = APIRouter()
|
||||
|
||||
#: Job types this library surfaces. Both flow through the shared longform
|
||||
#: renderer (``_render_longform_sse``) and emit the same ``done`` event shape.
|
||||
_LONGFORM_TYPES = ("audiobook", "story")
|
||||
|
||||
|
||||
def _done_payload_from_events(events: list[dict]) -> Optional[dict]:
|
||||
"""Recover the final ``{"type": "done", ...}`` payload from a job's SSE tail.
|
||||
|
||||
Each row's ``payload`` is the JSON the renderer stored via
|
||||
``job_store.append_event(job_id, json.dumps(payload))``. We scan newest-first
|
||||
and return the first parseable ``done`` event. Anything malformed is skipped
|
||||
— this never raises.
|
||||
"""
|
||||
for ev in reversed(events):
|
||||
raw = ev.get("payload") if isinstance(ev, dict) else None
|
||||
if not raw or not isinstance(raw, str):
|
||||
continue
|
||||
try:
|
||||
obj = json.loads(raw)
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
if isinstance(obj, dict) and obj.get("type") == "done":
|
||||
return obj
|
||||
return None
|
||||
|
||||
|
||||
def _coerce_int(value, default: int = 0) -> int:
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def _coerce_float(value, default: float = 0.0) -> float:
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def build_longform_library(
|
||||
list_jobs: Callable[..., list[dict]],
|
||||
events_since: Callable[..., list[dict]],
|
||||
*,
|
||||
limit: int = 50,
|
||||
) -> list[dict]:
|
||||
"""Build the newest-first list of finished longform renders.
|
||||
|
||||
Pure over the two job-store callables so tests can pass them directly:
|
||||
|
||||
* ``list_jobs(status="done", limit=...)`` → all done jobs, newest-first.
|
||||
* ``events_since(job_id)`` → that job's persisted SSE events.
|
||||
|
||||
Returns ``[{job_id, type, title?, output, duration_s, chapters,
|
||||
created_at}]``. Jobs that aren't a longform type, or whose ``done`` event /
|
||||
output filename can't be recovered, are silently skipped — the library only
|
||||
ever lists things the user can actually re-download.
|
||||
"""
|
||||
limit = max(1, min(_coerce_int(limit, 50), 500))
|
||||
try:
|
||||
# Over-fetch: non-longform done jobs (dub, etc.) get filtered out below,
|
||||
# so ask for more rows than the caller's limit to still fill the page.
|
||||
rows = list_jobs(status="done", limit=limit * 4)
|
||||
except Exception:
|
||||
logger.warning("longform library: list_jobs failed", exc_info=True)
|
||||
return []
|
||||
|
||||
out: list[dict] = []
|
||||
for row in rows or []:
|
||||
if len(out) >= limit:
|
||||
break
|
||||
try:
|
||||
job_type = row.get("type")
|
||||
job_id = row.get("id")
|
||||
if job_type not in _LONGFORM_TYPES or not job_id:
|
||||
continue
|
||||
try:
|
||||
events = events_since(job_id)
|
||||
except Exception:
|
||||
logger.warning("longform library: events_since failed for %s",
|
||||
job_id, exc_info=True)
|
||||
continue
|
||||
done = _done_payload_from_events(events or [])
|
||||
if not done:
|
||||
continue
|
||||
output = done.get("output")
|
||||
if not output or not isinstance(output, str):
|
||||
continue # nothing to re-download → not worth listing
|
||||
|
||||
item = {
|
||||
"job_id": job_id,
|
||||
"type": job_type,
|
||||
"output": output,
|
||||
"duration_s": round(_coerce_float(done.get("duration_s")), 2),
|
||||
"chapters": _coerce_int(done.get("chapters")),
|
||||
"created_at": row.get("created_at"),
|
||||
}
|
||||
# Title is optional — prefer the done event, fall back to job meta.
|
||||
title = done.get("title")
|
||||
if not title:
|
||||
meta_raw = row.get("meta_json")
|
||||
if isinstance(meta_raw, str) and meta_raw:
|
||||
try:
|
||||
meta = json.loads(meta_raw)
|
||||
if isinstance(meta, dict):
|
||||
title = meta.get("title")
|
||||
except (ValueError, TypeError):
|
||||
title = None
|
||||
if title:
|
||||
item["title"] = title
|
||||
out.append(item)
|
||||
except Exception:
|
||||
# Per-row isolation: one bad row never sinks the whole list.
|
||||
logger.warning("longform library: skipping unparseable job row",
|
||||
exc_info=True)
|
||||
continue
|
||||
return out
|
||||
|
||||
|
||||
@router.get("/longform/jobs")
|
||||
def longform_jobs(limit: int = Query(50, ge=1, le=500)) -> dict:
|
||||
"""Finished Audiobook + Story renders, newest-first, ready to re-download.
|
||||
|
||||
Each item's ``output`` is served at ``/audio/<output>``. Never 500s — on any
|
||||
backend hiccup it returns an empty list rather than an error.
|
||||
"""
|
||||
from core import job_store
|
||||
|
||||
jobs = build_longform_library(
|
||||
job_store.list_jobs, job_store.events_since, limit=limit,
|
||||
)
|
||||
return {"jobs": jobs}
|
||||
@@ -58,31 +58,6 @@ MAX_BUNDLE_BYTES = 100 * 1024 * 1024
|
||||
# ── Export ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _bundle_metadata(profile: dict, **extra) -> dict:
|
||||
"""Common .omnivoice metadata for export + publish.
|
||||
|
||||
Captures ``kind`` and ``vd_states`` so a *designed* persona survives the
|
||||
bundle round-trip as a design (not silently demoted to a clone) — required
|
||||
for the synthetic-only gating of the persona gallery (§R3). Old bundles
|
||||
without these keys import as ``kind='clone'`` (backward-compatible).
|
||||
"""
|
||||
meta = {
|
||||
"bundle_version": BUNDLE_VERSION,
|
||||
"profile_name": profile.get("name", "Unnamed"),
|
||||
"ref_text": profile.get("ref_text", ""),
|
||||
"instruct": profile.get("instruct", ""),
|
||||
"language": profile.get("language", "Auto"),
|
||||
"personality": profile.get("personality", ""),
|
||||
"seed": profile.get("seed"),
|
||||
"kind": profile.get("kind") or "clone",
|
||||
"vd_states": profile.get("vd_states"),
|
||||
"is_locked": bool(profile.get("is_locked")),
|
||||
"omnivoice_version": APP_VERSION,
|
||||
}
|
||||
meta.update(extra)
|
||||
return meta
|
||||
|
||||
|
||||
@router.post("/export/{profile_id}")
|
||||
def export_profile(profile_id: str):
|
||||
"""Export a voice profile as a downloadable .omnivoice bundle (ZIP)."""
|
||||
@@ -100,9 +75,19 @@ def export_profile(profile_id: str):
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
# Metadata
|
||||
metadata = _bundle_metadata(
|
||||
profile, created_at=profile.get("created_at"), exported_at=time.time(),
|
||||
)
|
||||
metadata = {
|
||||
"bundle_version": BUNDLE_VERSION,
|
||||
"profile_name": profile.get("name", "Unnamed"),
|
||||
"ref_text": profile.get("ref_text", ""),
|
||||
"instruct": profile.get("instruct", ""),
|
||||
"language": profile.get("language", "Auto"),
|
||||
"personality": profile.get("personality", ""),
|
||||
"seed": profile.get("seed"),
|
||||
"is_locked": bool(profile.get("is_locked")),
|
||||
"created_at": profile.get("created_at"),
|
||||
"exported_at": time.time(),
|
||||
"omnivoice_version": APP_VERSION,
|
||||
}
|
||||
zf.writestr("metadata.json", json.dumps(metadata, indent=2))
|
||||
|
||||
# Reference audio
|
||||
@@ -206,9 +191,8 @@ async def import_profile(
|
||||
conn.execute(
|
||||
"""INSERT INTO voice_profiles
|
||||
(id, name, ref_audio_path, ref_text, instruct, language,
|
||||
seed, personality, is_locked, locked_audio_path, created_at,
|
||||
kind, vd_states)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
seed, personality, is_locked, locked_audio_path, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(
|
||||
profile_id,
|
||||
metadata.get("profile_name", "Imported Voice"),
|
||||
@@ -221,10 +205,6 @@ async def import_profile(
|
||||
1 if is_locked else 0,
|
||||
locked_audio_filename or "",
|
||||
time.time(),
|
||||
# Preserve the design/clone distinction across the round-trip;
|
||||
# old bundles without these keys import as a clone.
|
||||
metadata.get("kind") or "clone",
|
||||
metadata.get("vd_states"),
|
||||
),
|
||||
)
|
||||
|
||||
@@ -273,11 +253,19 @@ def publish_to_marketplace(
|
||||
|
||||
# Build the bundle
|
||||
with zipfile.ZipFile(str(bundle_path), "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
metadata = _bundle_metadata(
|
||||
profile,
|
||||
tags=[t.strip() for t in tags.split(",") if t.strip()],
|
||||
published_at=time.time(),
|
||||
)
|
||||
metadata = {
|
||||
"bundle_version": BUNDLE_VERSION,
|
||||
"profile_name": profile.get("name", "Unnamed"),
|
||||
"ref_text": profile.get("ref_text", ""),
|
||||
"instruct": profile.get("instruct", ""),
|
||||
"language": profile.get("language", "Auto"),
|
||||
"personality": profile.get("personality", ""),
|
||||
"seed": profile.get("seed"),
|
||||
"is_locked": bool(profile.get("is_locked")),
|
||||
"tags": [t.strip() for t in tags.split(",") if t.strip()],
|
||||
"published_at": time.time(),
|
||||
"omnivoice_version": APP_VERSION,
|
||||
}
|
||||
zf.writestr("metadata.json", json.dumps(metadata, indent=2))
|
||||
|
||||
ref_path = profile.get("ref_audio_path")
|
||||
|
||||
@@ -1,53 +0,0 @@
|
||||
"""REST CRUD for per-agent MCP voice bindings (Wave 2.2 / Spec 2).
|
||||
|
||||
Loopback-gated — the Settings UI manages bindings here. The MCP tools
|
||||
themselves resolve voices via ``services.mcp_bindings.resolve_voice``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from api.dependencies import require_loopback
|
||||
from services import mcp_bindings
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/api/mcp",
|
||||
tags=["mcp"],
|
||||
dependencies=[Depends(require_loopback)],
|
||||
)
|
||||
|
||||
|
||||
class _BindingBody(BaseModel):
|
||||
client_id: str = Field(..., min_length=1, max_length=128)
|
||||
label: str | None = None
|
||||
profile_id: str | None = None
|
||||
default_engine: str | None = None
|
||||
|
||||
|
||||
@router.get("/bindings")
|
||||
def list_bindings():
|
||||
"""All per-agent voice bindings, most-recently-seen first."""
|
||||
return mcp_bindings.list_bindings()
|
||||
|
||||
|
||||
@router.put("/bindings")
|
||||
def upsert_binding(body: _BindingBody):
|
||||
"""Create or update the binding for an MCP client id."""
|
||||
try:
|
||||
return mcp_bindings.upsert_binding(
|
||||
body.client_id,
|
||||
label=body.label,
|
||||
profile_id=body.profile_id,
|
||||
default_engine=body.default_engine,
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
|
||||
@router.delete("/bindings/{client_id}")
|
||||
def delete_binding(client_id: str):
|
||||
if not mcp_bindings.delete_binding(client_id):
|
||||
raise HTTPException(status_code=404, detail="No binding for that client id")
|
||||
return {"deleted": client_id}
|
||||
@@ -82,33 +82,6 @@ class SpeechRequest(BaseModel):
|
||||
"E.g. 'young female, warm tone, slight British accent'.",
|
||||
)
|
||||
instruct: Optional[str] = Field(default=None, description="Style instruction for the TTS engine.")
|
||||
duration: Optional[float] = Field(
|
||||
default=None,
|
||||
gt=0,
|
||||
description="OmniVoice extension: target output duration in seconds.",
|
||||
)
|
||||
seed: Optional[int] = Field(
|
||||
default=None,
|
||||
description="OmniVoice extension: deterministic sampling seed.",
|
||||
)
|
||||
denoise: bool = Field(
|
||||
default=True,
|
||||
description="OmniVoice extension: prepend denoise control when supported.",
|
||||
)
|
||||
preprocess_prompt: bool = Field(
|
||||
default=True,
|
||||
description="OmniVoice extension: trim/preprocess reference prompt when supported.",
|
||||
)
|
||||
chunk_duration: Optional[float] = Field(
|
||||
default=None,
|
||||
ge=0,
|
||||
description="OmniVoice GGUF extension: long-form internal chunk duration.",
|
||||
)
|
||||
chunk_threshold: Optional[float] = Field(
|
||||
default=None,
|
||||
ge=0,
|
||||
description="OmniVoice GGUF extension: long-form internal chunk threshold.",
|
||||
)
|
||||
|
||||
|
||||
class TranscriptionResponse(BaseModel):
|
||||
@@ -236,14 +209,7 @@ def _run_tts(backend, text: str, kw: dict):
|
||||
from services.audio_dsp import apply_mastering, normalize_audio
|
||||
wav = backend.generate(text, **kw)
|
||||
sr = backend.sample_rate
|
||||
# Engines that already emit mastered, studio-grade audio (e.g. VoxCPM2's
|
||||
# native 48 kHz) opt out of apply_mastering via `applies_own_mastering`.
|
||||
# That chain's Compressor + 8% Reverb is tuned for OmniVoice's 24 kHz clone
|
||||
# output; applied to a studio engine it adds an audible level pump and a
|
||||
# reverb tail that degrade the very output we want clean. Loudness
|
||||
# normalisation still runs — it's a benign peak scale, not dynamics.
|
||||
if not getattr(backend, "applies_own_mastering", False):
|
||||
wav = apply_mastering(wav, sample_rate=sr)
|
||||
wav = apply_mastering(wav, sample_rate=sr)
|
||||
wav = normalize_audio(wav, target_dBFS=-2.0)
|
||||
return wav, sr
|
||||
|
||||
@@ -253,28 +219,10 @@ async def create_speech(req: SpeechRequest):
|
||||
"""Generate audio from text. Compatible with OpenAI's POST /v1/audio/speech."""
|
||||
backend = _resolve_engine(req.model)
|
||||
|
||||
# Routing gate (#21 — no silent CPU fallback), identical to REST /generate.
|
||||
from core.device_caps import detect_host_caps
|
||||
from services.engine_routing import resolve_routing, routing_notice
|
||||
_routing = resolve_routing(getattr(backend, "gpu_compat", ("cpu",)), detect_host_caps())
|
||||
if _routing["routing_status"] == "unavailable":
|
||||
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
|
||||
_routing_notice = routing_notice(_routing) # (status, reason) or None
|
||||
|
||||
# Build kwargs for the backend's generate() method
|
||||
kw: dict = {
|
||||
"speed": req.speed,
|
||||
"denoise": req.denoise,
|
||||
"preprocess_prompt": req.preprocess_prompt,
|
||||
}
|
||||
if req.duration is not None:
|
||||
kw["duration"] = req.duration
|
||||
if req.seed is not None:
|
||||
kw["seed"] = req.seed
|
||||
if req.chunk_duration is not None:
|
||||
kw["chunk_duration"] = req.chunk_duration
|
||||
if req.chunk_threshold is not None:
|
||||
kw["chunk_threshold"] = req.chunk_threshold
|
||||
if req.language:
|
||||
kw["language"] = req.language
|
||||
if req.instruct:
|
||||
@@ -303,8 +251,6 @@ async def create_speech(req: SpeechRequest):
|
||||
kw["ref_text"] = row["ref_text"]
|
||||
if row["instruct"] and not req.instruct:
|
||||
kw["instruct"] = row["instruct"]
|
||||
if req.seed is None and row["seed"] is not None:
|
||||
kw["seed"] = row["seed"]
|
||||
else:
|
||||
# Not a profile ID — forward as engine preset name
|
||||
kw["voice"] = voice
|
||||
@@ -321,20 +267,13 @@ async def create_speech(req: SpeechRequest):
|
||||
|
||||
audio_bytes, mime_type, ext = _encode_audio(wav, sr, req.response_format)
|
||||
|
||||
_headers = {
|
||||
"Content-Length": str(len(audio_bytes)),
|
||||
"Content-Disposition": f'inline; filename="speech.{ext}"',
|
||||
}
|
||||
if _routing_notice:
|
||||
from services.engine_routing import header_safe_reason
|
||||
_headers["X-OmniVoice-Routing"] = _routing_notice[0]
|
||||
_hr = header_safe_reason(_routing_notice[1])
|
||||
if _hr:
|
||||
_headers["X-OmniVoice-Routing-Reason"] = _hr
|
||||
return StreamingResponse(
|
||||
io.BytesIO(audio_bytes),
|
||||
media_type=mime_type,
|
||||
headers=_headers,
|
||||
headers={
|
||||
"Content-Length": str(len(audio_bytes)),
|
||||
"Content-Disposition": f'inline; filename="speech.{ext}"',
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -1,327 +0,0 @@
|
||||
"""HTTP layer for the `.ovsvoice` portable persona format (#29 / parity §R3 G1).
|
||||
|
||||
Thin router over `services.persona_bundle`:
|
||||
|
||||
POST /personas/export/{profile_id} → stream a downloadable .ovsvoice
|
||||
POST /personas/import → create a profile from a bundle
|
||||
POST /personas/inspect → read a bundle's manifest, no writes
|
||||
|
||||
Mirrors the legacy `.omnivoice` endpoints (`marketplace.py`) and reuses the
|
||||
same path-confinement (`_voices_path`) + consent floor. `.ovsvoice` is additive;
|
||||
`.omnivoice` import stays a compatible legacy reader.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import functools
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, File, HTTPException, Query, UploadFile
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from core import event_bus
|
||||
from core.config import VOICES_DIR # noqa: F401 — re-exported for tests/monkeypatch
|
||||
from core.db import db_conn
|
||||
from core.version import APP_VERSION
|
||||
from services import persona_bundle as pb
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.personas")
|
||||
|
||||
|
||||
def _safe_name(name: str, profile_id: str) -> str:
|
||||
"""Sanitised download filename stem (marketplace idiom); empty → persona_<id>."""
|
||||
cleaned = "".join(
|
||||
c if c.isalnum() or c in "-_ " else "" for c in (name or "")
|
||||
).strip().replace(" ", "_")[:40]
|
||||
return cleaned or f"persona_{profile_id}"
|
||||
|
||||
|
||||
# ── Export ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/personas/export/{profile_id}")
|
||||
async def export_persona(
|
||||
profile_id: str,
|
||||
license_spdx: str = Query(pb.DEFAULT_LICENSE),
|
||||
tags: str = Query(""),
|
||||
include_reference: bool = Query(True),
|
||||
):
|
||||
"""Build + stream a `.ovsvoice` bundle for a profile."""
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT * FROM voice_profiles WHERE id = ?", (profile_id,)
|
||||
).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Voice profile not found")
|
||||
profile = dict(row)
|
||||
|
||||
tag_list = [t.strip() for t in tags.split(",") if t.strip()]
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
content = await loop.run_in_executor(
|
||||
None,
|
||||
functools.partial(
|
||||
pb.build_persona_bundle,
|
||||
profile,
|
||||
license_spdx=license_spdx,
|
||||
tags=tag_list,
|
||||
include_reference=include_reference,
|
||||
engine_id=os.environ.get("OMNIVOICE_MODEL", ""),
|
||||
omnivoice_version=APP_VERSION,
|
||||
),
|
||||
)
|
||||
except pb.NoPreviewSource:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="This profile has no readable reference or locked audio to "
|
||||
"build a preview from — re-create or re-import it.",
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("persona export failed for %s", profile_id)
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="Could not build the persona bundle — see Settings → Logs.",
|
||||
)
|
||||
|
||||
filename = f"{_safe_name(profile.get('name'), profile_id)}.ovsvoice"
|
||||
from io import BytesIO
|
||||
return StreamingResponse(
|
||||
BytesIO(content),
|
||||
media_type="application/zip",
|
||||
headers={
|
||||
"Content-Disposition": f'attachment; filename="{filename}"',
|
||||
"Content-Length": str(len(content)),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# ── Import ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _voices_dest(filename: str) -> str:
|
||||
"""Resolve an output filename inside VOICES_DIR; 400 on escape (belt+braces —
|
||||
the name is always server-generated `{profile_id}…`)."""
|
||||
from api.routers.profiles import _voices_path
|
||||
path = _voices_path(filename)
|
||||
if path is None:
|
||||
raise HTTPException(status_code=400, detail="Invalid profile id")
|
||||
return path
|
||||
|
||||
|
||||
def _consent_verified(parsed: pb.ParsedPersona, consent_path: str | None) -> bool:
|
||||
"""B12-B16: trust verified-own-voice ONLY with a real recording (≥ floor) AND
|
||||
non-empty consent_text AND a consent.json present. The manifest flag alone
|
||||
can't forge it."""
|
||||
if not parsed.consent or not consent_path:
|
||||
return False
|
||||
if os.path.getsize(consent_path) < pb._MIN_CONSENT_AUDIO_BYTES:
|
||||
return False
|
||||
return bool((parsed.consent.get("consent_text") or "").strip())
|
||||
|
||||
|
||||
@router.post("/personas/import")
|
||||
async def import_persona(file: UploadFile = File(...)):
|
||||
"""Create a new voice profile from a `.ovsvoice` (or legacy `.omnivoice`) bundle."""
|
||||
name = (file.filename or "").lower()
|
||||
if not name.endswith(".ovsvoice") and not name.endswith(".omnivoice"):
|
||||
raise HTTPException(status_code=400, detail="File must be a .ovsvoice or .omnivoice bundle")
|
||||
|
||||
content = await file.read()
|
||||
try:
|
||||
parsed = pb.parse_persona_bundle(content)
|
||||
except pb.BundleError as e:
|
||||
raise HTTPException(status_code=e.status, detail=e.detail)
|
||||
|
||||
persona = parsed.manifest.get("persona") or {}
|
||||
written: list[str] = []
|
||||
|
||||
def _gen_id() -> str:
|
||||
return str(uuid.uuid4())[:8]
|
||||
|
||||
profile_id = _gen_id()
|
||||
try:
|
||||
# ── Audio members → server-named files (never the member name). ──
|
||||
ref_filename = None
|
||||
locked_filename = None
|
||||
if "ref_audio" in parsed.members:
|
||||
ref_filename = f"{profile_id}{parsed.member_ext('ref_audio')}"
|
||||
dest = _voices_dest(ref_filename)
|
||||
parsed.extract_member("ref_audio", dest); written.append(dest)
|
||||
if "locked_audio" in parsed.members:
|
||||
locked_filename = f"{profile_id}_locked{parsed.member_ext('locked_audio')}"
|
||||
dest = _voices_dest(locked_filename)
|
||||
parsed.extract_member("locked_audio", dest); written.append(dest)
|
||||
# Preview-only bundle (A12/B8): use the preview as the usable ref clip.
|
||||
if ref_filename is None and locked_filename is None and "preview" in parsed.members:
|
||||
ref_filename = f"{profile_id}{parsed.member_ext('preview')}"
|
||||
dest = _voices_dest(ref_filename)
|
||||
parsed.extract_member("preview", dest); written.append(dest)
|
||||
if ref_filename is None and locked_filename is None:
|
||||
raise HTTPException(status_code=400, detail="bundle has no usable audio")
|
||||
|
||||
# ── Consent recording (optional) ──
|
||||
consent_filename = None
|
||||
consent_path = None
|
||||
if "consent_audio" in parsed.members:
|
||||
consent_filename = f"{profile_id}_consent{parsed.member_ext('consent_audio')}"
|
||||
consent_path = _voices_dest(consent_filename)
|
||||
parsed.extract_member("consent_audio", consent_path); written.append(consent_path)
|
||||
|
||||
verified = _consent_verified(parsed, consent_path)
|
||||
consent_text = ((parsed.consent or {}).get("consent_text") or "").strip()
|
||||
recorded_at = None
|
||||
if verified:
|
||||
try:
|
||||
recorded_at = float(parsed.consent.get("recorded_at"))
|
||||
except (TypeError, ValueError):
|
||||
recorded_at = time.time()
|
||||
|
||||
is_locked = bool(persona.get("is_locked") and locked_filename)
|
||||
ref_for_db = ref_filename or locked_filename # at least one is set
|
||||
|
||||
def _insert(pid: str):
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"""INSERT INTO voice_profiles
|
||||
(id, name, ref_audio_path, ref_text, instruct, language,
|
||||
seed, personality, is_locked, locked_audio_path, created_at,
|
||||
kind, vd_states,
|
||||
verified_own_voice, consent_text, consent_audio_path, consent_recorded_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(
|
||||
pid,
|
||||
persona.get("name") or "Imported Voice",
|
||||
ref_for_db,
|
||||
persona.get("ref_text", ""),
|
||||
persona.get("instruct", ""),
|
||||
persona.get("language", "Auto"),
|
||||
persona.get("seed"),
|
||||
persona.get("personality", ""),
|
||||
1 if is_locked else 0,
|
||||
locked_filename or "",
|
||||
time.time(),
|
||||
persona.get("kind") or "clone",
|
||||
persona.get("vd_states"),
|
||||
1 if verified else 0,
|
||||
# Keep the attestation text so the user can re-attest locally,
|
||||
# even when imported unverified.
|
||||
consent_text,
|
||||
consent_filename if verified else "",
|
||||
recorded_at if verified else None,
|
||||
),
|
||||
)
|
||||
|
||||
import sqlite3
|
||||
try:
|
||||
_insert(profile_id)
|
||||
except sqlite3.IntegrityError:
|
||||
profile_id = _gen_id() # one retry on id collision (B20)
|
||||
# rename the on-disk files to the new id so they still match the row
|
||||
written = _rename_for_new_id(written, profile_id)
|
||||
ref_for_db = _retarget(ref_for_db, profile_id)
|
||||
locked_filename = _retarget(locked_filename, profile_id)
|
||||
consent_filename = _retarget(consent_filename, profile_id)
|
||||
_insert(profile_id)
|
||||
|
||||
except HTTPException:
|
||||
_cleanup(written)
|
||||
raise
|
||||
except Exception:
|
||||
_cleanup(written)
|
||||
logger.exception("persona import failed")
|
||||
raise HTTPException(status_code=500, detail="Import failed; no files were kept.")
|
||||
|
||||
event_bus.emit("profiles", {"action": "created", "id": profile_id})
|
||||
logger.info("Imported persona %r as %s (verified=%s)", persona.get("name"), profile_id, verified)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"profile_id": profile_id,
|
||||
"name": persona.get("name") or "Imported Voice",
|
||||
"kind": persona.get("kind") or "clone",
|
||||
"verified_own_voice": verified,
|
||||
"preview_only": parsed.preview_only,
|
||||
"license_spdx": parsed.license_spdx,
|
||||
"watermarked_preview": parsed.watermarked_preview,
|
||||
"source_bundle": file.filename,
|
||||
"schema_version_ahead": parsed.schema_version_ahead,
|
||||
}
|
||||
|
||||
|
||||
def _cleanup(paths: list[str]) -> None:
|
||||
for p in paths:
|
||||
try:
|
||||
if p and os.path.exists(p):
|
||||
os.remove(p)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _rename_for_new_id(written: list[str], new_id: str) -> list[str]:
|
||||
"""After an id-collision retry, rename each written file to carry the new id
|
||||
(filenames are `{old_id}…`; swap the leading 8-char stem)."""
|
||||
out = []
|
||||
for p in written:
|
||||
d, base = os.path.split(p)
|
||||
# base looks like {id}{ext} | {id}_locked{ext} | {id}_consent{ext}
|
||||
new_base = new_id + base[8:]
|
||||
new_path = os.path.join(d, new_base)
|
||||
try:
|
||||
os.replace(p, new_path)
|
||||
out.append(new_path)
|
||||
except OSError:
|
||||
out.append(p)
|
||||
return out
|
||||
|
||||
|
||||
def _retarget(filename: str | None, new_id: str) -> str | None:
|
||||
return new_id + filename[8:] if filename else filename
|
||||
|
||||
|
||||
# ── Inspect (no-write preview) ──────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/personas/inspect")
|
||||
async def inspect_persona(file: UploadFile = File(...)):
|
||||
"""Read a bundle's manifest + consent summary WITHOUT writing any file or row."""
|
||||
name = (file.filename or "").lower()
|
||||
if not name.endswith(".ovsvoice") and not name.endswith(".omnivoice"):
|
||||
raise HTTPException(status_code=400, detail="File must be a .ovsvoice or .omnivoice bundle")
|
||||
content = await file.read()
|
||||
try:
|
||||
parsed = pb.parse_persona_bundle(content)
|
||||
except pb.BundleError as e:
|
||||
raise HTTPException(status_code=e.status, detail=e.detail)
|
||||
|
||||
persona = parsed.manifest.get("persona") or {}
|
||||
consent_summary = None
|
||||
if parsed.consent:
|
||||
has_recording = "consent_audio" in parsed.members
|
||||
consent_summary = {
|
||||
"verified_claimed": bool(parsed.consent.get("verified_own_voice")),
|
||||
"method": parsed.consent.get("method", ""),
|
||||
"has_recording": has_recording,
|
||||
# would_verify mirrors import's gate, minus the byte-floor check
|
||||
# (inspect never extracts to measure size — advisory only).
|
||||
"would_verify": has_recording and bool((parsed.consent.get("consent_text") or "").strip()),
|
||||
}
|
||||
|
||||
return {
|
||||
"format": "omnivoice-legacy" if parsed.is_legacy else pb.OVSVOICE_FORMAT,
|
||||
"schema_version": parsed.manifest.get("schema_version", pb.OVSVOICE_SCHEMA_VERSION),
|
||||
"name": persona.get("name") or "Imported Voice",
|
||||
"kind": persona.get("kind") or "clone",
|
||||
"language": persona.get("language", "Auto"),
|
||||
"personality": persona.get("personality", ""),
|
||||
"is_locked": bool(persona.get("is_locked")),
|
||||
"license_spdx": parsed.license_spdx,
|
||||
"tags": parsed.manifest.get("tags") or [],
|
||||
"preview_only": parsed.preview_only,
|
||||
"watermarked_preview": parsed.watermarked_preview,
|
||||
"consent": consent_summary,
|
||||
"schema_version_ahead": parsed.schema_version_ahead,
|
||||
}
|
||||
+18
-276
@@ -1,5 +1,4 @@
|
||||
import os
|
||||
import re
|
||||
import uuid
|
||||
import time
|
||||
import shutil
|
||||
@@ -35,103 +34,29 @@ def list_profiles():
|
||||
rows = conn.execute("SELECT * FROM voice_profiles ORDER BY created_at DESC").fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
_DESIGN_SEED = 42 # deterministic sample render, same as archetype previews
|
||||
|
||||
|
||||
@router.post("/profiles")
|
||||
async def create_profile(
|
||||
name: str = Form(...),
|
||||
ref_audio: Optional[UploadFile] = File(None),
|
||||
ref_audio: UploadFile = File(...),
|
||||
ref_text: str = Form(""),
|
||||
instruct: str = Form(""),
|
||||
language: str = Form("Auto"),
|
||||
seed: Optional[int] = Form(None),
|
||||
personality: str = Form(""),
|
||||
kind: str = Form("clone"),
|
||||
vd_states: Optional[str] = Form(None),
|
||||
):
|
||||
"""Create a voice profile (spec: docs/specs/voice-studio-unification.md §5).
|
||||
|
||||
kind='clone' — requires `ref_audio` (the user's reference recording).
|
||||
kind='design' — requires `vd_states` (JSON of category picks); the server
|
||||
renders a deterministic sample WAV (seed 42, same path as
|
||||
archetype materialization) and stores it as the profile's
|
||||
reference so the voice identity is stable across runs.
|
||||
"""
|
||||
if kind not in ("clone", "design"):
|
||||
raise HTTPException(status_code=422, detail="kind must be 'clone' or 'design'")
|
||||
if kind == "clone" and ref_audio is None:
|
||||
raise HTTPException(status_code=422, detail="clone profiles require ref_audio")
|
||||
if kind == "design":
|
||||
if not (vd_states or "").strip():
|
||||
raise HTTPException(status_code=422, detail="design profiles require vd_states")
|
||||
import json as _json
|
||||
try:
|
||||
parsed = _json.loads(vd_states)
|
||||
if not isinstance(parsed, dict):
|
||||
raise ValueError("not an object")
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=422, detail="vd_states must be a JSON object")
|
||||
# An all-Auto design (every category left on "Auto") yields an empty
|
||||
# instruct — that's still a valid, saveable voice: synthesis falls back
|
||||
# to neutral instruct-only conditioning (see generation.py design path).
|
||||
# Don't gate save on a non-empty instruct.
|
||||
|
||||
profile_id = str(uuid.uuid4())[:8]
|
||||
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
|
||||
audio_filename = f"{profile_id}{ext}"
|
||||
audio_path = os.path.join(VOICES_DIR, audio_filename)
|
||||
|
||||
if kind == "clone":
|
||||
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
|
||||
audio_filename = f"{profile_id}{ext}"
|
||||
audio_path = os.path.join(VOICES_DIR, audio_filename)
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(await ref_audio.read())
|
||||
used_seed = seed
|
||||
else:
|
||||
# Saving a design profile is a pure persistence operation — it must not
|
||||
# depend on a loaded TTS model (issue #476: on a fresh model-less Docker
|
||||
# image the render forced a full model load + inference that 503'd, so
|
||||
# the save failed). We try the deterministic identity sample opportunist-
|
||||
# ically through the one shared TTS path (archetypes' renderer, never a
|
||||
# second inference code path); if the engine isn't ready it's rendered
|
||||
# lazily on first preview/use. The row carries vd_states + instruct, so
|
||||
# the voice is fully usable without the sample (synthesis falls back to
|
||||
# instruct-only conditioning — see generation.py's design path).
|
||||
from pathlib import Path
|
||||
from api.routers.archetypes import _render_archetype_wav
|
||||
audio_filename = f"{profile_id}.wav"
|
||||
audio_path = os.path.join(VOICES_DIR, audio_filename)
|
||||
try:
|
||||
await _render_archetype_wav(
|
||||
{
|
||||
"language": language,
|
||||
"sample_script": ref_text, # optional custom sample line
|
||||
"instruct": instruct,
|
||||
},
|
||||
Path(audio_path),
|
||||
)
|
||||
except Exception:
|
||||
# Engine unavailable / OOM / inference failure — defer the sample.
|
||||
# Store the row with no ref_audio_path; the identity sample is
|
||||
# rendered on first preview or use. Never let this block the save.
|
||||
import logging
|
||||
logging.getLogger("omnivoice.profiles").info(
|
||||
"Design profile %s saved with sample pending — "
|
||||
"voice engine not ready; will render on first use", profile_id,
|
||||
)
|
||||
if os.path.exists(audio_path): # partial/blank render: don't keep it
|
||||
with __import__("contextlib").suppress(OSError):
|
||||
os.remove(audio_path)
|
||||
audio_filename = None
|
||||
used_seed = seed if seed is not None else _DESIGN_SEED
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(await ref_audio.read())
|
||||
|
||||
try:
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, "
|
||||
"language, seed, personality, kind, vd_states, created_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(profile_id, name, audio_filename, ref_text, instruct, language,
|
||||
used_seed, personality, kind, vd_states, time.time())
|
||||
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(profile_id, name, audio_filename, ref_text, instruct, language, seed, personality, time.time())
|
||||
)
|
||||
except Exception:
|
||||
# Clean up orphaned audio file if DB insert fails
|
||||
@@ -139,7 +64,7 @@ async def create_profile(
|
||||
os.remove(audio_path)
|
||||
raise
|
||||
event_bus.emit("profiles", {"action": "created", "id": profile_id})
|
||||
return {"id": profile_id, "name": name, "kind": kind}
|
||||
return {"id": profile_id, "name": name}
|
||||
|
||||
@router.get("/profiles/{profile_id}")
|
||||
def get_profile(profile_id: str):
|
||||
@@ -238,96 +163,20 @@ def get_profile_usage(profile_id: str):
|
||||
}
|
||||
|
||||
|
||||
# profile_id is a request path param and the audio filename derives from it, so
|
||||
# constrain it to the generated-id charset (no separators / `..` possible) before
|
||||
# any path use, and read only a *direct child* of VOICES_DIR — os.path.basename()
|
||||
# strips any directory component (a path-injection / CWE-22 barrier).
|
||||
_PROFILE_ID_RE = re.compile(r"[A-Za-z0-9_-]{1,64}")
|
||||
|
||||
|
||||
@router.get("/profiles/{profile_id}/audio")
|
||||
async def get_profile_audio(profile_id: str):
|
||||
if not _PROFILE_ID_RE.fullmatch(profile_id or ""):
|
||||
return Response("Profile not found", status_code=404)
|
||||
def get_profile_audio(profile_id: str):
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT ref_audio_path, locked_audio_path, kind, instruct, language, ref_text "
|
||||
"FROM voice_profiles WHERE id=?",
|
||||
(profile_id,),
|
||||
).fetchone()
|
||||
row = conn.execute("SELECT ref_audio_path, locked_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
if not row:
|
||||
return Response("Profile not found", status_code=404)
|
||||
audio_file = row["locked_audio_path"] or row["ref_audio_path"]
|
||||
if not audio_file:
|
||||
# A design profile saved before the engine was ready (issue #476) has no
|
||||
# identity sample yet. Render it lazily now — the deterministic seed-42
|
||||
# sample is reproducible, so a deferred render matches a save-time one.
|
||||
rendered = await _materialize_design_sample(profile_id, row)
|
||||
if rendered is None:
|
||||
return Response("No audio available", status_code=404)
|
||||
audio_file = rendered
|
||||
# CWE-22: resolve the DB-stored filename strictly inside VOICES_DIR via the
|
||||
# shared guard — _voices_path() applies the os.path.basename() barrier plus
|
||||
# symlink-resolved containment (same path the consent endpoint trusts).
|
||||
audio_path = _voices_path(str(audio_file))
|
||||
if audio_path is None or not os.path.exists(audio_path):
|
||||
return Response("No audio available", status_code=404)
|
||||
audio_path = os.path.join(VOICES_DIR, audio_file)
|
||||
if not os.path.exists(audio_path):
|
||||
return Response("Audio file missing", status_code=404)
|
||||
return FileResponse(audio_path, media_type="audio/wav")
|
||||
|
||||
|
||||
async def _materialize_design_sample(profile_id: str, row) -> Optional[str]:
|
||||
"""Render a design profile's pending identity sample on first request.
|
||||
|
||||
Returns the stored filename on success, or None if this isn't a renderable
|
||||
design row. Raises HTTPException(503) with a precise "model not ready"
|
||||
message if the engine is genuinely unavailable — saving never depends on
|
||||
this, but a user who explicitly asks for the sample gets a clear signal.
|
||||
"""
|
||||
try:
|
||||
kind = row["kind"]
|
||||
except (KeyError, IndexError):
|
||||
kind = "clone"
|
||||
if kind != "design":
|
||||
return None
|
||||
|
||||
from pathlib import Path
|
||||
from api.routers.archetypes import _render_archetype_wav
|
||||
|
||||
audio_filename = f"{profile_id}.wav"
|
||||
# CWE-22: resolve under VOICES_DIR via the shared basename + containment
|
||||
# guard before rendering (rejects any escape).
|
||||
audio_path = _voices_path(audio_filename)
|
||||
if audio_path is None:
|
||||
raise HTTPException(status_code=400, detail="invalid profile identifier")
|
||||
try:
|
||||
await _render_archetype_wav(
|
||||
{
|
||||
"language": row["language"] or "Auto",
|
||||
"sample_script": row["ref_text"] or "",
|
||||
"instruct": row["instruct"] or "",
|
||||
},
|
||||
Path(audio_path),
|
||||
)
|
||||
except Exception as e:
|
||||
with __import__("contextlib").suppress(OSError):
|
||||
if os.path.exists(audio_path):
|
||||
os.remove(audio_path)
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail=(
|
||||
"The voice engine isn't ready yet, so this designed voice's "
|
||||
"preview sample can't be rendered. Finish setup / download a "
|
||||
f"model, then try again. ({e})"
|
||||
),
|
||||
)
|
||||
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"UPDATE voice_profiles SET ref_audio_path=? WHERE id=?",
|
||||
(audio_filename, profile_id),
|
||||
)
|
||||
return audio_filename
|
||||
|
||||
@router.post("/profiles/{profile_id}/lock")
|
||||
async def lock_profile(
|
||||
profile_id: str,
|
||||
@@ -385,122 +234,15 @@ async def unlock_profile(profile_id: str):
|
||||
event_bus.emit("profiles", {"action": "unlocked", "id": profile_id})
|
||||
return {"unlocked": True, "profile_id": profile_id}
|
||||
|
||||
# ── Consent lock (parity program Wave 0.2) ─────────────────────────────────
|
||||
#
|
||||
# A profile becomes "verified own voice" when its owner records themselves
|
||||
# reading a consent statement. The recording is provenance, not a voiceprint
|
||||
# check — agentic features and gallery sharing gate on the flag; plain local
|
||||
# synthesis never does. Spec: docs/competitive-analysis.md Action 22.
|
||||
|
||||
_MIN_CONSENT_AUDIO_BYTES = 1000 # same floor as the frontend recorder
|
||||
|
||||
# Upload filename extension whitelist — anything else falls back to .wav so a
|
||||
# crafted filename can never influence the on-disk path (py/path-injection).
|
||||
_CONSENT_EXT_RE = re.compile(r"^\.[A-Za-z0-9]{1,8}$")
|
||||
|
||||
|
||||
def _voices_path(filename: str) -> Optional[str]:
|
||||
"""Resolve a DB-stored audio filename strictly inside VOICES_DIR.
|
||||
|
||||
Rejects anything that isn't a bare filename or that escapes the voices
|
||||
directory after symlink resolution. Returns None instead of raising so
|
||||
cleanup paths can simply skip bad values.
|
||||
"""
|
||||
if not filename or os.path.basename(filename) != filename:
|
||||
return None
|
||||
root = os.path.realpath(VOICES_DIR)
|
||||
path = os.path.realpath(os.path.join(root, filename))
|
||||
if not path.startswith(root + os.sep):
|
||||
return None
|
||||
return path
|
||||
|
||||
|
||||
@router.post("/profiles/{profile_id}/consent")
|
||||
async def record_consent(
|
||||
profile_id: str,
|
||||
consent_audio: UploadFile = File(...),
|
||||
consent_text: str = Form(...),
|
||||
):
|
||||
if not consent_text.strip():
|
||||
raise HTTPException(status_code=422, detail="consent_text must not be empty")
|
||||
data = await consent_audio.read()
|
||||
if len(data) < _MIN_CONSENT_AUDIO_BYTES:
|
||||
raise HTTPException(status_code=422, detail="consent recording is too short")
|
||||
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT id, consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)
|
||||
).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Profile not found")
|
||||
|
||||
ext = os.path.splitext(consent_audio.filename or "")[1]
|
||||
if not _CONSENT_EXT_RE.match(ext):
|
||||
ext = ".wav"
|
||||
audio_filename = f"{profile_id}_consent{ext}"
|
||||
audio_path = _voices_path(audio_filename)
|
||||
if audio_path is None: # profile_id is server-generated; this is belt+braces
|
||||
raise HTTPException(status_code=400, detail="Invalid profile id")
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(data)
|
||||
|
||||
# A re-record may change the extension; drop the superseded file.
|
||||
old = row["consent_audio_path"]
|
||||
if old and old != audio_filename:
|
||||
old_path = _voices_path(old)
|
||||
if old_path and os.path.exists(old_path):
|
||||
os.remove(old_path)
|
||||
|
||||
recorded_at = time.time()
|
||||
try:
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"UPDATE voice_profiles SET verified_own_voice=1, consent_text=?, "
|
||||
"consent_audio_path=?, consent_recorded_at=? WHERE id=?",
|
||||
(consent_text.strip(), audio_filename, recorded_at, profile_id),
|
||||
)
|
||||
except Exception:
|
||||
if os.path.exists(audio_path):
|
||||
os.remove(audio_path)
|
||||
raise
|
||||
event_bus.emit("profiles", {"action": "consent_recorded", "id": profile_id})
|
||||
return {
|
||||
"id": profile_id,
|
||||
"verified_own_voice": True,
|
||||
"consent_recorded_at": recorded_at,
|
||||
}
|
||||
|
||||
|
||||
@router.delete("/profiles/{profile_id}/consent")
|
||||
def revoke_consent(profile_id: str):
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)
|
||||
).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Profile not found")
|
||||
conn.execute(
|
||||
"UPDATE voice_profiles SET verified_own_voice=0, consent_text='', "
|
||||
"consent_audio_path='', consent_recorded_at=NULL WHERE id=?",
|
||||
(profile_id,),
|
||||
)
|
||||
if row["consent_audio_path"]:
|
||||
path = _voices_path(row["consent_audio_path"])
|
||||
if path and os.path.exists(path):
|
||||
os.remove(path)
|
||||
event_bus.emit("profiles", {"action": "consent_revoked", "id": profile_id})
|
||||
return {"id": profile_id, "verified_own_voice": False}
|
||||
|
||||
|
||||
@router.delete("/profiles/{profile_id}")
|
||||
def delete_profile(profile_id: str):
|
||||
with db_conn() as conn:
|
||||
row = conn.execute("SELECT ref_audio_path, locked_audio_path, consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
row = conn.execute("SELECT ref_audio_path, locked_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
if row:
|
||||
for col in ["ref_audio_path", "locked_audio_path", "consent_audio_path"]:
|
||||
for col in ["ref_audio_path", "locked_audio_path"]:
|
||||
if row[col]:
|
||||
path = _voices_path(row[col])
|
||||
if path and os.path.exists(path):
|
||||
path = os.path.join(VOICES_DIR, row[col])
|
||||
if os.path.exists(path):
|
||||
os.remove(path)
|
||||
# Prevent FOREIGN KEY constraint failure
|
||||
conn.execute("UPDATE generation_history SET profile_id = NULL WHERE profile_id=?", (profile_id,))
|
||||
|
||||
@@ -2,7 +2,6 @@ import uuid
|
||||
import time
|
||||
import json
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from core.db import db_conn
|
||||
from core import event_bus
|
||||
@@ -10,10 +9,6 @@ from schemas.requests import ProjectSaveRequest
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
class ProjectRenameRequest(BaseModel):
|
||||
name: str
|
||||
|
||||
@router.get("/projects")
|
||||
async def list_projects():
|
||||
with db_conn() as conn:
|
||||
@@ -64,26 +59,6 @@ async def update_project(project_id: str, req: ProjectSaveRequest):
|
||||
event_bus.emit("projects", {"action": "updated", "id": project_id})
|
||||
return {"id": project_id, "name": req.name, "updated_at": now}
|
||||
|
||||
@router.patch("/projects/{project_id}")
|
||||
async def rename_project(project_id: str, req: ProjectRenameRequest):
|
||||
"""Lightweight rename — updates only the project name (and updated_at),
|
||||
without re-serialising the whole state blob like PUT does."""
|
||||
name = req.name.strip()
|
||||
if not name:
|
||||
raise HTTPException(status_code=400, detail="Project name cannot be empty")
|
||||
now = time.time()
|
||||
with db_conn() as conn:
|
||||
row = conn.execute("SELECT id FROM studio_projects WHERE id=?", (project_id,)).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Project not found")
|
||||
conn.execute(
|
||||
"UPDATE studio_projects SET name=?, updated_at=? WHERE id=?",
|
||||
(name, now, project_id),
|
||||
)
|
||||
event_bus.emit("projects", {"action": "renamed", "id": project_id})
|
||||
return {"id": project_id, "name": name, "updated_at": now}
|
||||
|
||||
|
||||
@router.delete("/projects/{project_id}")
|
||||
async def delete_project(project_id: str):
|
||||
with db_conn() as conn:
|
||||
|
||||
@@ -123,117 +123,6 @@ def set_torch_compile_disabled(body: _TorchCompileBody):
|
||||
return _torch_compile_state()
|
||||
|
||||
|
||||
# ── Dictation refinement (parity program Wave 2.1 / Spec 3 phase 2) ───────
|
||||
|
||||
|
||||
class _RefinementBody(BaseModel):
|
||||
auto: bool | None = None
|
||||
smart_cleanup: bool | None = None
|
||||
self_correction: bool | None = None
|
||||
preserve_technical: bool | None = None
|
||||
|
||||
|
||||
def _refinement_state():
|
||||
from services.refinement import get_refinement_config
|
||||
from services.llm_backend import get_active_llm_backend
|
||||
|
||||
cfg = get_refinement_config()
|
||||
# The UI shows whether refinement can actually run (needs an LLM).
|
||||
cfg["llm_ready"] = get_active_llm_backend().id != "off"
|
||||
return cfg
|
||||
|
||||
|
||||
@router.get("/dictation-refinement")
|
||||
def get_dictation_refinement():
|
||||
"""Current refinement config + whether an LLM backend is configured."""
|
||||
return _refinement_state()
|
||||
|
||||
|
||||
@router.put("/dictation-refinement")
|
||||
def set_dictation_refinement(body: _RefinementBody):
|
||||
from services.refinement import set_refinement_config
|
||||
|
||||
try:
|
||||
set_refinement_config({k: v for k, v in body.model_dump().items() if v is not None})
|
||||
except Exception:
|
||||
logger.exception("set_dictation_refinement failed")
|
||||
raise HTTPException(status_code=500, detail="Failed to persist setting")
|
||||
return _refinement_state()
|
||||
|
||||
|
||||
# ── LLM endpoint (parity program Wave 2.4 / §R2 rung 4) ───────────────────
|
||||
# Focused configuration for the OpenAI-compatible LLM endpoint that powers
|
||||
# cinematic translate, glossary auto-extract, and dictation refinement.
|
||||
# Persistence rides the existing TRANSLATE_BASE_URL / TRANSLATE_API_KEY /
|
||||
# TRANSLATE_MODEL env vars (already in system.py PERSISTENT_KEYS, restored
|
||||
# at startup) so the resolution path in llm_backend/translator is unchanged.
|
||||
|
||||
|
||||
class _LLMEndpointBody(BaseModel):
|
||||
base_url: str | None = None
|
||||
model: str | None = None
|
||||
api_key: str | None = None # None = leave unchanged; "" = clear
|
||||
|
||||
|
||||
def _mask(secret: str | None) -> str | None:
|
||||
if not secret:
|
||||
return None
|
||||
return f"…{secret[-4:]}" if len(secret) > 4 else "set"
|
||||
|
||||
|
||||
def _llm_endpoint_state():
|
||||
from services.llm_backend import OpenAICompatBackend
|
||||
|
||||
ok, reason = OpenAICompatBackend.is_available()
|
||||
return {
|
||||
"base_url": os.environ.get("TRANSLATE_BASE_URL", ""),
|
||||
"model": os.environ.get("TRANSLATE_MODEL", ""),
|
||||
"api_key_masked": _mask(
|
||||
os.environ.get("TRANSLATE_API_KEY") or os.environ.get("OPENAI_API_KEY")
|
||||
),
|
||||
"available": ok,
|
||||
"reason": None if ok else reason,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/llm-endpoint")
|
||||
def get_llm_endpoint():
|
||||
"""Current OpenAI-compatible LLM endpoint config + live availability."""
|
||||
return _llm_endpoint_state()
|
||||
|
||||
|
||||
@router.put("/llm-endpoint")
|
||||
def set_llm_endpoint(body: _LLMEndpointBody):
|
||||
"""Persist base URL / model / API key for the OpenAI-compatible endpoint.
|
||||
|
||||
Reuses the env-var persistence path (prefs.json, restored at startup):
|
||||
base_url -> TRANSLATE_BASE_URL, model -> TRANSLATE_MODEL,
|
||||
api_key -> TRANSLATE_API_KEY. A None field is left unchanged; an empty
|
||||
string clears it. Ollama ignores the key; vLLM / LM Studio require it.
|
||||
"""
|
||||
from core.prefs import set_ as prefs_set, delete as prefs_delete
|
||||
|
||||
mapping = {
|
||||
"TRANSLATE_BASE_URL": body.base_url,
|
||||
"TRANSLATE_MODEL": body.model,
|
||||
"TRANSLATE_API_KEY": body.api_key,
|
||||
}
|
||||
for env_key, val in mapping.items():
|
||||
if val is None:
|
||||
continue # untouched
|
||||
val = val.strip()
|
||||
if val:
|
||||
os.environ[env_key] = val
|
||||
prefs_set(f"env.{env_key}", val)
|
||||
else:
|
||||
os.environ.pop(env_key, None)
|
||||
prefs_delete(f"env.{env_key}")
|
||||
# get_active_llm_backend() builds a fresh backend (and its OpenAI client
|
||||
# reads env at construction) on every call, so there's no singleton to
|
||||
# invalidate — the next translate/refine picks up the new values.
|
||||
return _llm_endpoint_state()
|
||||
|
||||
|
||||
# ── License acceptance (Phase 3 Plan 03-01 / TTS-05) ──────────────────────
|
||||
# Frontend ``SupertonicLicenseDialog`` flips the engine-license bit via this
|
||||
# endpoint. The handler is loopback-gated (router-level dep) and the
|
||||
@@ -395,60 +284,3 @@ def set_models_dir(body: _ModelsDirBody):
|
||||
|
||||
user_env.set_user_env(_MODELS_DIR_ENV, path)
|
||||
return {"configured": path, "effective": _effective_models_dir(), "restart_required": True}
|
||||
|
||||
|
||||
# ── HF mirror endpoint (parity program Wave 4.3 / §R4 c) ──────────────────
|
||||
# Restricted-network users (e.g. behind the Great Firewall) need to point
|
||||
# huggingface_hub at a mirror. HF reads HF_ENDPOINT at import time, so a
|
||||
# change takes effect on the next backend start — persisted to the durable
|
||||
# per-user env so it survives Tauri/Finder launches that don't inherit a
|
||||
# shell. Loopback-gated via the router dep.
|
||||
|
||||
_HF_ENDPOINT_ENV = "HF_ENDPOINT"
|
||||
|
||||
# A few well-known mirrors, surfaced as quick-picks in the UI. hf-mirror.com
|
||||
# is the community mirror most-used in China; the official endpoint clears it.
|
||||
_HF_MIRROR_PRESETS = [
|
||||
{"label": "Hugging Face (official)", "url": ""},
|
||||
{"label": "hf-mirror.com (community, China)", "url": "https://hf-mirror.com"},
|
||||
]
|
||||
|
||||
|
||||
class _HFMirrorBody(BaseModel):
|
||||
url: str = Field("", description="HF_ENDPOINT URL; empty string clears it (official endpoint)")
|
||||
|
||||
|
||||
@router.get("/hf-mirror")
|
||||
def get_hf_mirror():
|
||||
from core import user_env
|
||||
|
||||
configured = user_env.get_user_env(_HF_ENDPOINT_ENV) or ""
|
||||
return {
|
||||
# The value that will apply after restart (persisted), and what's
|
||||
# live in this process (env may differ until then).
|
||||
"configured": configured,
|
||||
"effective": os.environ.get(_HF_ENDPOINT_ENV, ""),
|
||||
"presets": _HF_MIRROR_PRESETS,
|
||||
}
|
||||
|
||||
|
||||
@router.put("/hf-mirror")
|
||||
def set_hf_mirror(body: _HFMirrorBody):
|
||||
from core import user_env
|
||||
|
||||
url = (body.url or "").strip().rstrip("/")
|
||||
if url and not url.startswith(("http://", "https://")):
|
||||
raise HTTPException(status_code=400, detail="Mirror URL must start with http(s)://")
|
||||
try:
|
||||
if url:
|
||||
user_env.set_user_env(_HF_ENDPOINT_ENV, url)
|
||||
os.environ[_HF_ENDPOINT_ENV] = url # best-effort for new downloads this session
|
||||
else:
|
||||
user_env.unset_user_env(_HF_ENDPOINT_ENV)
|
||||
os.environ.pop(_HF_ENDPOINT_ENV, None)
|
||||
except Exception:
|
||||
logger.exception("set_hf_mirror failed")
|
||||
raise HTTPException(status_code=500, detail="Failed to persist mirror setting")
|
||||
# HF endpoint is read at import time by huggingface_hub, so the override
|
||||
# is only guaranteed once the backend restarts.
|
||||
return {"configured": url, "restart_required": True, "presets": _HF_MIRROR_PRESETS}
|
||||
|
||||
@@ -11,16 +11,13 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from core import prefs
|
||||
from utils import hf_progress
|
||||
from utils import download_aggregator
|
||||
from .models import KNOWN_MODELS, invalidate_cache
|
||||
|
||||
logger = logging.getLogger("omnivoice.setup.download")
|
||||
@@ -29,177 +26,6 @@ router = APIRouter()
|
||||
# Cooldown: prevent rapid re-install after a failure. Maps repo_id → last_fail_time.
|
||||
_install_cooldowns: dict[str, float] = {}
|
||||
_COOLDOWN_SECS = 60.0
|
||||
# Evict cooldown entries older than this so the dict can't grow unbounded across
|
||||
# a long-lived process (MM2-06). Anything past the cooldown window is dead state.
|
||||
_COOLDOWN_TTL_SECS = 3600.0
|
||||
|
||||
|
||||
def _sweep_cooldowns(now: float) -> None:
|
||||
"""Drop cooldown entries older than the TTL (MM2-06). Keeps the dict bounded
|
||||
— without this it accumulated one entry per ever-failed repo forever."""
|
||||
stale = [k for k, t in _install_cooldowns.items() if (now - t) > _COOLDOWN_TTL_SECS]
|
||||
for k in stale:
|
||||
_install_cooldowns.pop(k, None)
|
||||
|
||||
# Repo_ids the user asked to cancel (FDL-11). Checked between retry attempts.
|
||||
# Note: a single in-flight snapshot_download/Xet fetch is not interruptible
|
||||
# mid-file in hf_hub 1.7.2 — cancel stops further retries, marks the row
|
||||
# cancelled, and clears the cooldown so a cancel isn't rate-limited.
|
||||
_cancelled: set[str] = set()
|
||||
|
||||
|
||||
def _download_max_workers() -> int:
|
||||
"""Parallel-FILES worker count for snapshot_download (FDL-02). Default 8 —
|
||||
don't crank it: Xet already parallelises *within* each file via concurrent
|
||||
byte-range gets, so a high count just multiplies buffer pressure. Override
|
||||
via prefs / OMNIVOICE_DOWNLOAD_MAX_WORKERS for power users."""
|
||||
raw = prefs.resolve("download_max_workers", env="OMNIVOICE_DOWNLOAD_MAX_WORKERS", default=8)
|
||||
try:
|
||||
return max(1, int(raw))
|
||||
except (TypeError, ValueError):
|
||||
return 8
|
||||
|
||||
|
||||
def _download_endpoint() -> "str | None":
|
||||
"""Optional HF endpoint override (FDL-10 mirror path, opt-in). Returned as a
|
||||
per-call ``endpoint=`` rather than a process-wide HF_ENDPOINT mutation. A
|
||||
mirror routes through the classic LFS path (no Xet) — documented in
|
||||
docs/downloading-models.md."""
|
||||
ep = prefs.resolve("hf_endpoint", env="HF_ENDPOINT", default=None)
|
||||
return ep or None
|
||||
|
||||
|
||||
def apply_xet_env() -> None:
|
||||
"""Apply opt-in Xet tuning knobs to the environment before a download
|
||||
(FDL-04). Both default OFF; env wins over the prefs store. high-performance
|
||||
can *hurt* low-RAM machines (needs lots of RAM/bandwidth); HDD-sequential
|
||||
avoids parallel-write thrash on spinning disks. Idempotent."""
|
||||
import os as _os
|
||||
high_perf = prefs.resolve("xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False)
|
||||
if _truthy(high_perf):
|
||||
_os.environ["HF_XET_HIGH_PERFORMANCE"] = "1"
|
||||
hdd_seq = prefs.resolve("xet_hdd_sequential_write", env="HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY", default=False)
|
||||
if _truthy(hdd_seq):
|
||||
_os.environ["HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY"] = "1"
|
||||
|
||||
|
||||
def _truthy(v) -> bool:
|
||||
if isinstance(v, bool):
|
||||
return v
|
||||
return str(v).strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
class _InstallCancelled(Exception):
|
||||
"""Raised inside the install worker when the user cancels (FDL-11)."""
|
||||
|
||||
|
||||
def compute_plan(plan_files) -> dict:
|
||||
"""Summarise a snapshot_download(dry_run=True) result into the install_plan
|
||||
payload (FDL-05): total bytes, bytes already cached (skipped), bytes that
|
||||
will actually download, and file counts. ``will_download`` defaults to
|
||||
``not is_cached`` for forward-compat with older DryRunFileInfo shapes."""
|
||||
total = sum(int(getattr(f, "file_size", 0) or 0) for f in plan_files)
|
||||
cached = sum(
|
||||
int(getattr(f, "file_size", 0) or 0)
|
||||
for f in plan_files if getattr(f, "is_cached", False)
|
||||
)
|
||||
will = [
|
||||
f for f in plan_files
|
||||
if getattr(f, "will_download", not getattr(f, "is_cached", False))
|
||||
]
|
||||
to_dl = sum(int(getattr(f, "file_size", 0) or 0) for f in will)
|
||||
n_files = len(plan_files)
|
||||
n_cached = sum(1 for f in plan_files if getattr(f, "is_cached", False))
|
||||
return {
|
||||
"total_bytes": total,
|
||||
"cached_bytes": cached,
|
||||
"to_download_bytes": to_dl,
|
||||
"n_files": n_files,
|
||||
"n_cached": n_cached,
|
||||
}
|
||||
|
||||
|
||||
def _segmented_enabled() -> bool:
|
||||
"""Opt-in IDM-style accelerator (FDL-09), default OFF. Most useful when Xet
|
||||
is inactive (the app's default): the legacy-LFS path is single-stream, so
|
||||
this restores parallel speed AND gives real live byte progress."""
|
||||
return _truthy(prefs.resolve(
|
||||
"segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=False,
|
||||
))
|
||||
|
||||
|
||||
def _xet_active() -> bool:
|
||||
"""True only when hf_xet is installed AND not disabled. The app sets
|
||||
HF_HUB_DISABLE_XET=1 by default, so this is normally False — which is when
|
||||
the segmented accelerator pays off."""
|
||||
import importlib.util
|
||||
if importlib.util.find_spec("hf_xet") is None:
|
||||
return False
|
||||
return os.environ.get("HF_HUB_DISABLE_XET", "").strip().lower() not in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _repo_cancelled(repo_id: str) -> bool:
|
||||
return repo_id in _cancelled
|
||||
|
||||
|
||||
def _segmented_snapshot(repo_id: str, *, endpoint: "str | None") -> str:
|
||||
"""Fetch every file of a repo via the segmented downloader into the HF
|
||||
cache, mirroring hf_hub_download's blob+snapshot+refs layout so the result
|
||||
is indistinguishable from snapshot_download (FDL-09) — keeping /models
|
||||
install-state, is_cached, and delete working. Feeds real bytes to the
|
||||
aggregator. Raises on any error; the caller falls back to snapshot_download.
|
||||
"""
|
||||
import asyncio as _asyncio
|
||||
from huggingface_hub import HfApi, constants as _C
|
||||
from huggingface_hub.file_download import (
|
||||
hf_hub_url, get_hf_file_metadata, repo_folder_name, _create_symlink,
|
||||
)
|
||||
from services.segmented_download import segmented_download
|
||||
from services.token_resolver import resolve as _resolve_token
|
||||
|
||||
token = _resolve_token()
|
||||
api = HfApi(endpoint=endpoint, token=token)
|
||||
info = api.repo_info(repo_id, repo_type="model")
|
||||
commit = info.sha
|
||||
files = [s.rfilename for s in (info.siblings or [])]
|
||||
if not commit or not files:
|
||||
raise RuntimeError("repo_info returned no commit/siblings")
|
||||
|
||||
repo_dir = os.path.join(_C.HF_HUB_CACHE, repo_folder_name(repo_id=repo_id, repo_type="model"))
|
||||
blobs_dir = os.path.join(repo_dir, "blobs")
|
||||
snap_dir = os.path.join(repo_dir, "snapshots", commit)
|
||||
refs_dir = os.path.join(repo_dir, "refs")
|
||||
for d in (blobs_dir, snap_dir, refs_dir):
|
||||
os.makedirs(d, exist_ok=True)
|
||||
|
||||
for rel in files:
|
||||
if _repo_cancelled(repo_id):
|
||||
raise _InstallCancelled()
|
||||
url = hf_hub_url(repo_id, rel, endpoint=endpoint, revision=commit)
|
||||
meta = get_hf_file_metadata(url, token=token)
|
||||
etag = (meta.etag or "").strip('"')
|
||||
if not etag:
|
||||
raise RuntimeError(f"no etag for {rel}")
|
||||
blob_path = os.path.join(blobs_dir, etag)
|
||||
pointer = os.path.join(snap_dir, rel)
|
||||
os.makedirs(os.path.dirname(pointer), exist_ok=True)
|
||||
if not os.path.exists(blob_path):
|
||||
_asyncio.run(segmented_download(
|
||||
meta.location or url, blob_path,
|
||||
token=token, expected_size=meta.size, expected_etag=etag,
|
||||
on_bytes=lambda d, k=rel: download_aggregator.add_bytes(repo_id, k, d),
|
||||
cancel_check=lambda: _repo_cancelled(repo_id),
|
||||
))
|
||||
if not os.path.lexists(pointer):
|
||||
_create_symlink(blob_path, pointer, new_blob=True)
|
||||
|
||||
# refs/main → commit so scan_cache_dir maps the revision correctly.
|
||||
try:
|
||||
with open(os.path.join(refs_dir, "main"), "w") as f:
|
||||
f.write(commit)
|
||||
except OSError:
|
||||
pass
|
||||
return snap_dir
|
||||
|
||||
|
||||
# ── SSE Download Stream ───────────────────────────────────────────────────
|
||||
@@ -216,66 +42,6 @@ def _safe_put(queue: asyncio.Queue, event) -> None:
|
||||
pass
|
||||
|
||||
|
||||
# Minimum size for "this snapshot actually contains model weights". An
|
||||
# interrupted snapshot_download can leave config/tokenizer files but no
|
||||
# weights; the install then looks complete and synthesis later fails with
|
||||
# "does not appear to have a file named pytorch_model.bin or
|
||||
# model.safetensors" (#352). 5 MB clears every weight format we ship
|
||||
# (safetensors/bin shards, onnx, pt, gguf) without false-positiving on
|
||||
# config-only aux repos.
|
||||
_MIN_WEIGHT_BYTES = 5 * 1024 * 1024
|
||||
|
||||
# Per-role weight-file floors (MM2-07). A valid model has at least one
|
||||
# recognized weight file at or above its extension's floor. ONNX graphs are
|
||||
# legitimately small (a complete model can be well under 5 MB), so a single
|
||||
# 5 MB rule false-positives on them as "truncated" (#352 over-trigger); give
|
||||
# .onnx a lower floor while still rejecting a 0/KB partial. Tensor formats keep
|
||||
# the original 5 MB floor.
|
||||
_WEIGHT_FLOORS = {
|
||||
".safetensors": _MIN_WEIGHT_BYTES,
|
||||
".bin": _MIN_WEIGHT_BYTES,
|
||||
".ckpt": _MIN_WEIGHT_BYTES,
|
||||
".pt": _MIN_WEIGHT_BYTES,
|
||||
".pth": _MIN_WEIGHT_BYTES,
|
||||
".gguf": _MIN_WEIGHT_BYTES,
|
||||
".onnx": 64 * 1024, # a real ONNX graph is ≥ tens of KB; a truncated one is bytes
|
||||
}
|
||||
|
||||
|
||||
def _validate_snapshot_has_weights(repo_id: str, snapshot_path: str) -> None:
|
||||
"""Raise OSError when a finished snapshot has no plausible weight file —
|
||||
surfaces the truncated-download class (#352) at install time, where the
|
||||
retry loop and the UI's re-download path can deal with it, instead of at
|
||||
first synthesis with an opaque transformers error.
|
||||
|
||||
A snapshot is valid if it contains a recognized weight file meeting its
|
||||
per-extension floor (MM2-07) OR any file ≥ the global 5 MB floor (the
|
||||
original lenient catch — kept so this is never stricter than before)."""
|
||||
try:
|
||||
biggest = 0
|
||||
for root, _dirs, files in os.walk(snapshot_path, followlinks=True):
|
||||
for f in files:
|
||||
try:
|
||||
size = os.path.getsize(os.path.join(root, f))
|
||||
except OSError:
|
||||
continue
|
||||
biggest = max(biggest, size)
|
||||
ext = os.path.splitext(f)[1].lower()
|
||||
floor = _WEIGHT_FLOORS.get(ext)
|
||||
if floor is not None and size >= floor:
|
||||
return # a recognized weight file of plausible size
|
||||
if size >= _MIN_WEIGHT_BYTES:
|
||||
return # original lenient catch (non-standard weight names)
|
||||
except OSError:
|
||||
return # can't inspect — don't block the install on the checker itself
|
||||
raise OSError(
|
||||
f"{repo_id}: download finished but no model weights were found in the "
|
||||
"snapshot (largest file "
|
||||
f"{biggest} bytes). The download was likely interrupted — delete the "
|
||||
"model in Settings → Models and install it again."
|
||||
)
|
||||
|
||||
|
||||
@router.get("/setup/download-stream")
|
||||
async def setup_download_stream():
|
||||
"""SSE: forward every HuggingFace download tqdm update as a JSON event."""
|
||||
@@ -332,7 +98,6 @@ async def install_model(req: InstallModelRequest):
|
||||
)
|
||||
# Cooldown guard — don't retry if the same model just failed.
|
||||
import time as _time_check
|
||||
_sweep_cooldowns(_time_check.time()) # bound the dict (MM2-06)
|
||||
last_fail = _install_cooldowns.get(req.repo_id)
|
||||
if last_fail and (_time_check.time() - last_fail) < _COOLDOWN_SECS:
|
||||
remaining = int(_COOLDOWN_SECS - (_time_check.time() - last_fail))
|
||||
@@ -347,7 +112,6 @@ async def install_model(req: InstallModelRequest):
|
||||
|
||||
def _do():
|
||||
token = hf_progress.current_repo_id.set(req.repo_id)
|
||||
_cancelled.discard(req.repo_id) # clear any stale cancel from a prior run
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
@@ -361,22 +125,7 @@ async def install_model(req: InstallModelRequest):
|
||||
LocalEntryNotFoundError,
|
||||
)
|
||||
logger.info("model install starting: %s", req.repo_id)
|
||||
# Apply opt-in Xet tuning knobs (high-perf / HDD) before downloading.
|
||||
apply_xet_env()
|
||||
# Drive snapshot_download explicitly (FDL-02): pass our progress-
|
||||
# emitting tqdm subclass so progress is deterministic + Xet-aware
|
||||
# (Xet feeds bytes into whatever tqdm_class is supplied), bound the
|
||||
# parallel-files worker count, and honour an optional mirror endpoint.
|
||||
dl_kwargs: dict = {
|
||||
"repo_id": req.repo_id,
|
||||
"max_workers": _download_max_workers(),
|
||||
}
|
||||
_tqdm_cls = hf_progress.tracked_tqdm_class()
|
||||
if _tqdm_cls is not None:
|
||||
dl_kwargs["tqdm_class"] = _tqdm_cls
|
||||
_endpoint = _download_endpoint()
|
||||
if _endpoint:
|
||||
dl_kwargs["endpoint"] = _endpoint
|
||||
dl_kwargs: dict = {"repo_id": req.repo_id}
|
||||
if sys.platform == "win32":
|
||||
dl_kwargs["local_dir_use_symlinks"] = False
|
||||
|
||||
@@ -404,70 +153,12 @@ async def install_model(req: InstallModelRequest):
|
||||
hb = threading.Thread(target=_heartbeat, daemon=True)
|
||||
hb.start()
|
||||
|
||||
# Pre-flight (FDL-05): a dry-run resolve gives the UI an accurate
|
||||
# denominator — total bytes, bytes already cached (skipped), and the
|
||||
# bytes that will actually download — BEFORE any byte flows. Seeds
|
||||
# the overall aggregator so its bar/ETA are correct from the first
|
||||
# event. Degrades gracefully (totals=None) on older/gated repos.
|
||||
_preflight_kwargs = {"repo_id": req.repo_id, "dry_run": True}
|
||||
if _endpoint:
|
||||
_preflight_kwargs["endpoint"] = _endpoint
|
||||
try:
|
||||
_plan = snapshot_download(**_preflight_kwargs)
|
||||
_summary = compute_plan(_plan)
|
||||
download_aggregator.start(
|
||||
req.repo_id,
|
||||
total_bytes=_summary["to_download_bytes"],
|
||||
files_total=max(0, _summary["n_files"] - _summary["n_cached"]),
|
||||
)
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
"phase": "install_plan",
|
||||
**_summary,
|
||||
})
|
||||
except Exception as _pf_err:
|
||||
# No preflight (older/gated repo, mirror without dry-run, etc.):
|
||||
# fall back to today's fill-in-as-files-appear behaviour.
|
||||
logger.info("model install %s: preflight unavailable (%s)", req.repo_id, _pf_err)
|
||||
download_aggregator.start(req.repo_id)
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
"phase": "install_plan",
|
||||
"total_bytes": None,
|
||||
"cached_bytes": None,
|
||||
"to_download_bytes": None,
|
||||
"n_files": None,
|
||||
"n_cached": None,
|
||||
})
|
||||
|
||||
_max_attempts = 5
|
||||
_attempt = 0
|
||||
while True:
|
||||
if req.repo_id in _cancelled:
|
||||
raise _InstallCancelled()
|
||||
_attempt += 1
|
||||
try:
|
||||
# Opt-in segmented accelerator (FDL-09): parallel byte-range
|
||||
# fetch with real live progress, for the legacy-LFS path.
|
||||
# Any failure falls through to snapshot_download — the
|
||||
# accelerator can never compromise a correct install.
|
||||
_snapshot_path = None
|
||||
if _attempt == 1 and _segmented_enabled() and not _xet_active():
|
||||
try:
|
||||
_snapshot_path = _segmented_snapshot(req.repo_id, endpoint=_endpoint)
|
||||
except _InstallCancelled:
|
||||
raise
|
||||
except Exception as _seg_err:
|
||||
logger.info(
|
||||
"segmented download for %s failed (%s); falling back to snapshot_download",
|
||||
req.repo_id, _seg_err,
|
||||
)
|
||||
_snapshot_path = None
|
||||
if _snapshot_path is None:
|
||||
_snapshot_path = snapshot_download(**dl_kwargs)
|
||||
_validate_snapshot_has_weights(req.repo_id, _snapshot_path)
|
||||
snapshot_download(**dl_kwargs)
|
||||
break
|
||||
except (HfHubHTTPError, LocalEntryNotFoundError, OSError) as net_err:
|
||||
if _attempt >= _max_attempts:
|
||||
@@ -488,10 +179,6 @@ async def install_model(req: InstallModelRequest):
|
||||
_t.sleep(_backoff)
|
||||
# Stop heartbeat once download completes
|
||||
_resolving.set()
|
||||
# Flush the overall bar to 100% with the true byte total (FDL-06):
|
||||
# under Xet the per-file byte bars don't surface completion, so the
|
||||
# aggregator can sit below 100% even though every file landed.
|
||||
download_aggregator.complete(req.repo_id)
|
||||
logger.info("model install done: %s", req.repo_id)
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
@@ -499,19 +186,7 @@ async def install_model(req: InstallModelRequest):
|
||||
"downloaded": 0, "total": 0, "pct": 1.0,
|
||||
"phase": "install_done",
|
||||
})
|
||||
_install_cooldowns.pop(req.repo_id, None) # success clears any cooldown (MM2-06)
|
||||
invalidate_cache()
|
||||
except _InstallCancelled:
|
||||
_resolving.set()
|
||||
logger.info("model install cancelled: %s", req.repo_id)
|
||||
# A cancel is user intent, not a failure — don't set a cooldown.
|
||||
_install_cooldowns.pop(req.repo_id, None)
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
"downloaded": 0, "total": 0, "pct": 0.0,
|
||||
"phase": "install_cancelled",
|
||||
})
|
||||
except Exception as e:
|
||||
_resolving.set()
|
||||
logger.info("model install failed for %s: %s", req.repo_id, e)
|
||||
@@ -525,28 +200,12 @@ async def install_model(req: InstallModelRequest):
|
||||
"error": str(e),
|
||||
})
|
||||
finally:
|
||||
_cancelled.discard(req.repo_id)
|
||||
download_aggregator.finish(req.repo_id)
|
||||
hf_progress.current_repo_id.reset(token)
|
||||
|
||||
loop.create_task(asyncio.to_thread(_do))
|
||||
return {"status": "install_started", "repo_id": req.repo_id}
|
||||
|
||||
|
||||
@router.post("/models/install/cancel")
|
||||
async def cancel_install(req: InstallModelRequest):
|
||||
"""Request cancellation of an in-flight install (FDL-11).
|
||||
|
||||
Best-effort: stops further retry attempts and marks the row cancelled. A
|
||||
single in-flight snapshot_download/Xet fetch isn't interruptible mid-file
|
||||
in hf_hub 1.7.2, so an already-streaming file finishes; the cancel takes
|
||||
effect at the next retry boundary. Clears the cooldown so the user can
|
||||
immediately restart."""
|
||||
_cancelled.add(req.repo_id)
|
||||
_install_cooldowns.pop(req.repo_id, None)
|
||||
return {"cancelling": req.repo_id}
|
||||
|
||||
|
||||
# ── Delete ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@router.delete("/models/{repo_id:path}")
|
||||
|
||||
@@ -220,12 +220,8 @@ def is_cached(repo_id: str) -> bool:
|
||||
except Exception as e:
|
||||
# scan_cache_dir can raise on Windows (WinError 448 'untrusted mount
|
||||
# point'); fall back to a direct disk check so a cached model isn't
|
||||
# mistaken for missing and re-downloaded in a loop (#117/#118). Logged
|
||||
# at WARNING with the exception type (MM2-09) so this fallback isn't
|
||||
# invisible when triaging a Windows cache report — it previously logged
|
||||
# at DEBUG and never showed at the default level.
|
||||
logger.warning("is_cached: scan_cache_dir failed (%s: %s); using on-disk fallback for %s",
|
||||
type(e).__name__, e, repo_id)
|
||||
# mistaken for missing and re-downloaded in a loop (#117/#118).
|
||||
logger.debug("scan_cache_dir failed (%s); using disk fallback", e)
|
||||
return _is_cached_on_disk(repo_id)
|
||||
|
||||
|
||||
|
||||
@@ -373,46 +373,6 @@ def preflight():
|
||||
"status": gpu_status, "detail": gpu_detail, "fix": gpu_fix,
|
||||
})
|
||||
|
||||
# ── GPU routing for the ACTIVE TTS engine (#21 — no silent CPU fallback).
|
||||
# Distinct from the hardware "gpu" check above: this asks "will the engine
|
||||
# the user actually selected use that GPU on this host?" Built from the same
|
||||
# canonical probe + resolver the Engine Compatibility Matrix uses.
|
||||
try:
|
||||
from services.tts_backend import gpu_routing_verdict
|
||||
gpu_routing = gpu_routing_verdict()
|
||||
except Exception as exc: # never break preflight on a routing hiccup
|
||||
logger.warning("preflight gpu_routing failed: %s", exc)
|
||||
gpu_routing = None
|
||||
if gpu_routing:
|
||||
_rs = gpu_routing.get("routing_status")
|
||||
_eng = gpu_routing.get("engine") or "active engine"
|
||||
_dev = gpu_routing.get("effective_device") or "?"
|
||||
_why = gpu_routing.get("routing_reason")
|
||||
if _rs == "accelerated" and not _why:
|
||||
r_status, r_detail, r_fix = "pass", f"{_eng} → {_dev} (accelerated)", None
|
||||
elif _rs == "accelerated": # driver/arch caveat
|
||||
r_status, r_detail, r_fix = "warn", f"{_eng} → {_dev}: {_why}", (
|
||||
"GPU selected but may fail at kernel launch — update drivers / "
|
||||
"reinstall torch for this GPU architecture.")
|
||||
elif _rs == "cpu_fallback":
|
||||
r_status, r_detail, r_fix = "warn", (
|
||||
f"{_eng} runs on CPU here: {_why or 'no GPU path for this host'}"), (
|
||||
"Pick an engine that supports this host's GPU for a speedup, or "
|
||||
"continue on CPU (slower).")
|
||||
elif _rs == "cpu_only":
|
||||
r_status, r_detail, r_fix = "pass", f"{_eng} → cpu (no accelerator on this host)", None
|
||||
elif _rs == "unavailable":
|
||||
r_status, r_detail, r_fix = "fail", (
|
||||
f"{_eng} can't run on this host: {_why or 'needs a GPU this machine lacks'}"), (
|
||||
"Select an engine with a CPU path in Settings → Engines.")
|
||||
else: # "none" / unknown
|
||||
r_status, r_detail, r_fix = "warn", "No active TTS engine resolved for routing.", (
|
||||
"Pick an engine in Settings → Engines.")
|
||||
checks.append({
|
||||
"id": "gpu_routing", "label": "Active engine routing",
|
||||
"status": r_status, "detail": r_detail, "fix": r_fix,
|
||||
})
|
||||
|
||||
# ── Network
|
||||
net_ok = _probe_network()
|
||||
checks.append({
|
||||
@@ -440,13 +400,9 @@ def preflight():
|
||||
"gpu_available": gpu["available"],
|
||||
"gpu_driver": gpu["driver"],
|
||||
"gpu_device_name": gpu["device_name"],
|
||||
# Canonical probe (distinguishes ROCm from CUDA):
|
||||
"gpu_family": (gpu_routing or {}).get("host_family", "cpu"),
|
||||
"vram_gb": (gpu_routing or {}).get("vram_gb", 0.0),
|
||||
"ram_gb": round(ram, 1),
|
||||
"disk_free_gb": round(free, 1),
|
||||
},
|
||||
"gpu_routing": gpu_routing,
|
||||
}
|
||||
|
||||
|
||||
|
||||
+85
-120
@@ -107,52 +107,6 @@ def _ui_port() -> int:
|
||||
return 3901
|
||||
|
||||
|
||||
def _fast_download_status() -> dict:
|
||||
"""Report the download-acceleration state for the Settings UI (FDL-03).
|
||||
|
||||
Reports the *runtime* truth, not just whether hf_xet is importable. The app
|
||||
currently sets ``HF_HUB_DISABLE_XET=1`` by default (main.py) — Xet's chunked
|
||||
transfer is fast but its progress bypasses our tqdm patch, so the legacy-LFS
|
||||
path is forced to keep accurate byte progress. So:
|
||||
|
||||
* ``xet_installed`` — hf_xet present
|
||||
* ``xet_active`` — installed AND not disabled via HF_HUB_DISABLE_XET
|
||||
* ``xet_enabled`` — alias of xet_active (what the UI badge keys off)
|
||||
|
||||
Must never throw: /system/info is called on every Settings load.
|
||||
"""
|
||||
installed = False
|
||||
version = None
|
||||
try:
|
||||
import hf_xet # noqa: F401
|
||||
installed = True
|
||||
try:
|
||||
from importlib.metadata import version as _ver
|
||||
version = _ver("hf-xet")
|
||||
except Exception:
|
||||
version = None
|
||||
except Exception:
|
||||
installed = False
|
||||
disabled = str(os.environ.get("HF_HUB_DISABLE_XET", "")).strip().lower() in {"1", "true", "yes", "on"}
|
||||
active = installed and not disabled
|
||||
try:
|
||||
from core import prefs
|
||||
high_perf = prefs.resolve(
|
||||
"xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False
|
||||
)
|
||||
high_perf = high_perf if isinstance(high_perf, bool) else \
|
||||
str(high_perf).strip().lower() in {"1", "true", "yes", "on"}
|
||||
except Exception:
|
||||
high_perf = False
|
||||
return {
|
||||
"xet_installed": installed,
|
||||
"xet_active": active,
|
||||
"xet_enabled": active, # UI badge: only true when Xet actually runs
|
||||
"xet_version": version,
|
||||
"high_performance": bool(high_perf),
|
||||
}
|
||||
|
||||
|
||||
def _has_hf_token() -> bool:
|
||||
# Phase 1 AUTH-01..06 cascade. Delegates to the 3-source resolver
|
||||
# (App → Env → HF-CLI) instead of reading env/HF-CLI directly. This
|
||||
@@ -174,23 +128,88 @@ def model_status():
|
||||
|
||||
@router.get("/model/loaded")
|
||||
def loaded_models():
|
||||
"""List all currently loaded models for the flush dropdown (MM2-04).
|
||||
Thin delegation to the model_lifecycle facade — shape unchanged:
|
||||
``{models, count}``."""
|
||||
from services import model_lifecycle
|
||||
return model_lifecycle.list_loaded()
|
||||
"""Return details about all currently loaded models for the flush dropdown.
|
||||
|
||||
Returns a list of models with name, type, device, and estimated VRAM usage.
|
||||
"""
|
||||
import services.model_manager as mm
|
||||
|
||||
models = []
|
||||
|
||||
# 1. TTS model (OmniVoice)
|
||||
if mm.model is not None:
|
||||
device = "unknown"
|
||||
vram_mb = 0
|
||||
try:
|
||||
device = str(next(mm.model.parameters()).device) if hasattr(mm.model, 'parameters') else get_best_device()
|
||||
except Exception:
|
||||
device = get_best_device()
|
||||
try:
|
||||
torch = mm._lazy_torch()
|
||||
if torch.cuda.is_available():
|
||||
vram_mb = torch.cuda.memory_allocated() / (1024 ** 2)
|
||||
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
driver = getattr(torch.mps, "driver_allocated_memory", None)
|
||||
if driver:
|
||||
vram_mb = driver() / (1024 ** 2)
|
||||
except Exception:
|
||||
pass
|
||||
models.append({
|
||||
"id": "tts",
|
||||
"name": "OmniVoice TTS",
|
||||
"checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
|
||||
"device": device,
|
||||
"vram_mb": round(vram_mb, 1),
|
||||
"unloadable": True,
|
||||
})
|
||||
|
||||
# 2. ASR model (WhisperX)
|
||||
if mm.model is not None and hasattr(mm.model, '_asr_pipe') and mm.model._asr_pipe is not None:
|
||||
models.append({
|
||||
"id": "asr",
|
||||
"name": "WhisperX ASR",
|
||||
"checkpoint": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
|
||||
"device": "cpu",
|
||||
"vram_mb": 0,
|
||||
"unloadable": False, # tied to TTS model lifecycle
|
||||
})
|
||||
|
||||
# 3. Diarization pipeline
|
||||
if mm._diar_pipeline is not None:
|
||||
models.append({
|
||||
"id": "diarization",
|
||||
"name": "Pyannote Diarization",
|
||||
"checkpoint": "pyannote/speaker-diarization-3.1",
|
||||
"device": get_best_device(),
|
||||
"vram_mb": 0,
|
||||
"unloadable": True,
|
||||
})
|
||||
|
||||
return {"models": models, "count": len(models)}
|
||||
|
||||
|
||||
@router.post("/model/unload/{model_id}")
|
||||
async def unload_model(model_id: str):
|
||||
"""Unload a specific model by id (MM2-04). Delegates to model_lifecycle;
|
||||
an unknown id maps to HTTP 400. ``tts`` | ``diarization`` |
|
||||
``sidecar:<id>`` | ``sidecars``."""
|
||||
from services import model_lifecycle
|
||||
try:
|
||||
return await model_lifecycle.unload(model_id)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
"""Unload a specific model by ID."""
|
||||
import services.model_manager as mm
|
||||
|
||||
if model_id == "tts":
|
||||
async with mm._model_lock:
|
||||
if mm.model is not None:
|
||||
mm.model = None
|
||||
mm.free_vram()
|
||||
return {"unloaded": "tts", "success": True}
|
||||
return {"unloaded": "tts", "success": False, "reason": "not loaded"}
|
||||
|
||||
elif model_id == "diarization":
|
||||
if mm._diar_pipeline is not None:
|
||||
mm._diar_pipeline = None
|
||||
mm.free_vram()
|
||||
return {"unloaded": "diarization", "success": True}
|
||||
return {"unloaded": "diarization", "success": False, "reason": "not loaded"}
|
||||
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail=f"Unknown model id: {model_id}")
|
||||
|
||||
|
||||
@router.get("/system/info", response_model=SystemInfoResponse)
|
||||
@@ -212,7 +231,6 @@ def system_info():
|
||||
"asr_model": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
|
||||
"translate_provider": os.environ.get("TRANSLATE_PROVIDER", "google"),
|
||||
"has_hf_token": _has_hf_token(),
|
||||
"fast_download": _fast_download_status(),
|
||||
"device": get_best_device(),
|
||||
"python": sys.version.split()[0],
|
||||
"platform": sys.platform,
|
||||
@@ -304,10 +322,9 @@ def _tauri_log_candidates():
|
||||
os.path.join(home, "Library/Logs/OmniVoice/backend_err.log"),
|
||||
]
|
||||
if sys.platform.startswith("linux"):
|
||||
data_dir = os.environ.get("XDG_DATA_HOME") or os.path.join(home, ".local/share")
|
||||
state_dir = os.environ.get("XDG_STATE_HOME") or os.path.join(home, ".local/state")
|
||||
return [
|
||||
os.path.join(data_dir, bid, "logs", "tauri.log"),
|
||||
os.path.join(home, ".local/share", bid, "logs", "tauri.log"),
|
||||
os.path.join(home, ".config", bid, "logs", "tauri.log"),
|
||||
os.path.join(state_dir, "OmniVoice", "backend.log"),
|
||||
os.path.join(state_dir, "OmniVoice", "backend_err.log"),
|
||||
@@ -447,14 +464,6 @@ async def clear_system_logs():
|
||||
status_code=500,
|
||||
detail=f"Could not clear log at {p}: {e}. The file may be open in another process or read-only — close tailing tools and retry.",
|
||||
)
|
||||
if cleared_any:
|
||||
# The crash log just shrank to zero — drop any stored ack so a stale
|
||||
# byte count can't suppress the next 'crash-last-session' notice.
|
||||
for key in ("crash_log_acked", "crash_log_acked_size"):
|
||||
try:
|
||||
prefs_delete(key)
|
||||
except Exception:
|
||||
pass
|
||||
return {"cleared": cleared_any}
|
||||
|
||||
|
||||
@@ -556,21 +565,6 @@ async def flush_memory(unload_model: bool = False):
|
||||
|
||||
# ── Actionable notifications ──────────────────────────────────────────────
|
||||
|
||||
_GPU_ARCH_WARNING: "list[str | None]" = [] # [-1] = computed result
|
||||
|
||||
|
||||
def _gpu_arch_warning_cached() -> "str | None":
|
||||
"""check_device_compatibility() once per process (it lazy-imports torch —
|
||||
too heavy for the 30s notifications poll)."""
|
||||
if not _GPU_ARCH_WARNING:
|
||||
try:
|
||||
from services.model_manager import check_device_compatibility
|
||||
compatible, warning = check_device_compatibility()
|
||||
_GPU_ARCH_WARNING.append(None if compatible else warning)
|
||||
except Exception:
|
||||
_GPU_ARCH_WARNING.append(None)
|
||||
return _GPU_ARCH_WARNING[-1]
|
||||
|
||||
|
||||
@router.get("/system/notifications")
|
||||
def system_notifications():
|
||||
@@ -602,20 +596,6 @@ def system_notifications():
|
||||
},
|
||||
})
|
||||
|
||||
# 1b. GPU compute capability unsupported by this torch build (#284) —
|
||||
# the model "runs" but emits pure noise, the worst silent failure mode
|
||||
# (RTX 50-series Blackwell sm_120 on pre-cu128 wheels). The loader logs
|
||||
# this, but a log line never reached the affected users — surface it in
|
||||
# the panel. Checked once per process: it lazy-imports torch.
|
||||
gpu_warn = _gpu_arch_warning_cached()
|
||||
if gpu_warn:
|
||||
notes.append({
|
||||
"id": "gpu-arch-unsupported",
|
||||
"level": "error",
|
||||
"title": "GPU not supported by this PyTorch build",
|
||||
"message": gpu_warn + " Until then, output will be noise/garbage.",
|
||||
})
|
||||
|
||||
# 2. Missing ffmpeg
|
||||
ffmpeg_ok = False
|
||||
try:
|
||||
@@ -706,33 +686,18 @@ def _crashed_last_session() -> bool:
|
||||
if not os.path.exists(CRASH_LOG_PATH):
|
||||
return False
|
||||
size = os.path.getsize(CRASH_LOG_PATH)
|
||||
if size == 0:
|
||||
acked = int(prefs_get("crash_log_acked_size", 0) or 0)
|
||||
if size <= acked:
|
||||
return False
|
||||
mtime = os.path.getmtime(CRASH_LOG_PATH)
|
||||
# Composite ack (size + mtime): a bare byte count goes stale after the log
|
||||
# is truncated — the next crash log can stay smaller than the old acked
|
||||
# size forever, silently suppressing 'crash-last-session'. The ack only
|
||||
# holds while it still covers the file's current state.
|
||||
ack = prefs_get("crash_log_acked")
|
||||
if isinstance(ack, dict):
|
||||
if float(ack.get("mtime", 0) or 0) >= mtime and int(ack.get("size", 0) or 0) >= size:
|
||||
return False
|
||||
else:
|
||||
# Legacy size-only ack from older builds.
|
||||
if size <= int(prefs_get("crash_log_acked_size", 0) or 0):
|
||||
return False
|
||||
return mtime < _PROCESS_START_TS
|
||||
return os.path.getmtime(CRASH_LOG_PATH) < _PROCESS_START_TS
|
||||
|
||||
|
||||
@router.post("/system/crash/ack")
|
||||
async def ack_crash():
|
||||
"""Mark the current crash log as seen — dismisses the
|
||||
'crash-last-session' notification until the log changes again."""
|
||||
size = mtime = 0
|
||||
if os.path.exists(CRASH_LOG_PATH):
|
||||
size = os.path.getsize(CRASH_LOG_PATH)
|
||||
mtime = os.path.getmtime(CRASH_LOG_PATH)
|
||||
prefs_set("crash_log_acked", {"size": size, "mtime": mtime})
|
||||
'crash-last-session' notification until the log grows again."""
|
||||
size = os.path.getsize(CRASH_LOG_PATH) if os.path.exists(CRASH_LOG_PATH) else 0
|
||||
prefs_set("crash_log_acked_size", size)
|
||||
return {"acked_size": size}
|
||||
|
||||
|
||||
|
||||
@@ -98,30 +98,6 @@ async def ws_tts(websocket: WebSocket):
|
||||
model = await get_model()
|
||||
backend = get_active_tts_backend(model=model)
|
||||
|
||||
# ── Routing gate (#21 — no silent CPU fallback). WebSockets have
|
||||
# no response headers, so this uses frames: an error frame +
|
||||
# close on `unavailable`, a one-time `routing` frame on
|
||||
# cpu_fallback / accelerated-with-caveat (before any audio).
|
||||
from core.device_caps import detect_host_caps
|
||||
from services.engine_routing import resolve_routing, routing_notice
|
||||
from core.scrub import scrub_text
|
||||
_routing = resolve_routing(
|
||||
getattr(backend, "gpu_compat", ("cpu",)), detect_host_caps())
|
||||
if _routing["routing_status"] == "unavailable":
|
||||
await websocket.send_json({
|
||||
"type": "error",
|
||||
"detail": scrub_text(_routing["routing_reason"])
|
||||
or "engine cannot run on this host",
|
||||
})
|
||||
continue # don't stream; wait for the next request
|
||||
_notice = routing_notice(_routing)
|
||||
if _notice:
|
||||
await websocket.send_json({
|
||||
"type": "routing",
|
||||
"status": _notice[0],
|
||||
"reason": scrub_text(_notice[1]) if _notice[1] else None,
|
||||
})
|
||||
|
||||
# Build generation kwargs
|
||||
kw: dict = {"speed": data.get("speed", 1.0)}
|
||||
if data.get("language"):
|
||||
@@ -168,75 +144,50 @@ async def ws_tts(websocket: WebSocket):
|
||||
except Exception:
|
||||
kw["voice"] = voice
|
||||
|
||||
# Wave 1.4: split the request into sentences so the first
|
||||
# sentence's audio streams while later sentences are still
|
||||
# synthesizing — this is the time-to-first-audio win. The
|
||||
# chunker handles abbreviations/acronyms/decimals and CJK /
|
||||
# non-Latin terminators; single-sentence requests behave
|
||||
# exactly like the old single-shot path.
|
||||
from services.sentence_chunker import SentenceChunker
|
||||
_chunker = SentenceChunker(language=(data.get("language") or "en"))
|
||||
sentences = _chunker.push(text)
|
||||
sentences.extend(_chunker.flush())
|
||||
if not sentences:
|
||||
sentences = [text]
|
||||
|
||||
# Run generation in the GPU pool
|
||||
from services.model_manager import _gpu_pool
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
def _generate(sentence_text):
|
||||
def _generate():
|
||||
from services.audio_dsp import apply_mastering, normalize_audio
|
||||
wav = backend.generate(sentence_text, **kw)
|
||||
wav = backend.generate(text, **kw)
|
||||
sr_actual = backend.sample_rate
|
||||
# Like _run_tts in openai_compat: studio engines (VoxCPM2)
|
||||
# opt out of the broadcast mastering chain. This is the
|
||||
# other route that runs the active backend, so it needs the
|
||||
# same guard. Loudness normalisation still runs.
|
||||
if not getattr(backend, "applies_own_mastering", False):
|
||||
wav = apply_mastering(wav, sample_rate=sr_actual)
|
||||
wav = apply_mastering(wav, sample_rate=sr_actual)
|
||||
wav = normalize_audio(wav, target_dBFS=-2.0)
|
||||
return wav, sr_actual
|
||||
|
||||
wav_tensor, sr = await loop.run_in_executor(_gpu_pool, _generate)
|
||||
|
||||
# Send metadata after generation so sample_rate is real
|
||||
await websocket.send_json({
|
||||
"type": "start",
|
||||
"sample_rate": sr,
|
||||
"channels": 1,
|
||||
"format": "pcm16",
|
||||
"engine": backend.id,
|
||||
})
|
||||
|
||||
# Stream PCM16 chunks over the WebSocket
|
||||
import torch
|
||||
total_samples = 0
|
||||
sr = backend.sample_rate
|
||||
started = False
|
||||
# Convert to 16-bit PCM
|
||||
pcm = (wav_tensor * 32767).clamp(-32768, 32767).to(torch.int16)
|
||||
if pcm.ndim == 2:
|
||||
pcm = pcm[0] # mono
|
||||
pcm_bytes = pcm.numpy().tobytes()
|
||||
|
||||
for sentence in sentences:
|
||||
wav_tensor, sr = await loop.run_in_executor(
|
||||
_gpu_pool, _generate, sentence
|
||||
)
|
||||
total_samples = len(pcm)
|
||||
sent_samples = 0
|
||||
chunk_bytes = CHUNK_SAMPLES * 2 # 2 bytes per int16 sample
|
||||
|
||||
if not started:
|
||||
# Send metadata after the first generation so
|
||||
# sample_rate is real (lazy-loading engines report
|
||||
# their true rate only once weights are up).
|
||||
await websocket.send_json({
|
||||
"type": "start",
|
||||
"sample_rate": sr,
|
||||
"channels": 1,
|
||||
"format": "pcm16",
|
||||
"engine": backend.id,
|
||||
})
|
||||
started = True
|
||||
|
||||
# Convert to 16-bit PCM and stream
|
||||
pcm = (wav_tensor * 32767).clamp(-32768, 32767).to(torch.int16)
|
||||
if pcm.ndim == 2:
|
||||
pcm = pcm[0] # mono
|
||||
pcm_bytes = pcm.numpy().tobytes()
|
||||
|
||||
n_samples = len(pcm)
|
||||
sent_samples = 0
|
||||
while sent_samples < n_samples:
|
||||
end = min(sent_samples + CHUNK_SAMPLES, n_samples)
|
||||
chunk = pcm_bytes[sent_samples * 2: end * 2]
|
||||
await websocket.send_bytes(chunk)
|
||||
sent_samples = end
|
||||
# Yield to event loop between chunks for responsiveness
|
||||
await asyncio.sleep(0)
|
||||
total_samples += n_samples
|
||||
while sent_samples < total_samples:
|
||||
end = min(sent_samples + CHUNK_SAMPLES, total_samples)
|
||||
start_byte = sent_samples * 2
|
||||
end_byte = end * 2
|
||||
chunk = pcm_bytes[start_byte:end_byte]
|
||||
await websocket.send_bytes(chunk)
|
||||
sent_samples = end
|
||||
# Yield to event loop between chunks for responsiveness
|
||||
await asyncio.sleep(0)
|
||||
|
||||
gen_time = round(time.perf_counter() - t0, 3)
|
||||
duration = round(total_samples / sr, 3)
|
||||
|
||||
@@ -34,8 +34,6 @@ class SystemInfoResponse(BaseModel):
|
||||
asr_model: str = "unknown"
|
||||
translate_provider: str = "unknown"
|
||||
has_hf_token: bool = False
|
||||
# Xet fast-download backend state (FDL-03): {xet_enabled, xet_version, high_performance}
|
||||
fast_download: dict | None = None
|
||||
device: str = "cpu"
|
||||
python: str = ""
|
||||
platform: str = ""
|
||||
@@ -129,41 +127,16 @@ class DeviceInfo(BaseModel):
|
||||
gpu_available: bool = False
|
||||
gpu_driver: str | None = None
|
||||
gpu_device_name: str | None = None
|
||||
# From the canonical device probe (core.device_caps) — distinguishes ROCm
|
||||
# from CUDA, unlike the legacy nvidia-smi-based gpu_vendor/gpu_backend.
|
||||
gpu_family: str = "cpu"
|
||||
vram_gb: float = 0.0
|
||||
ram_gb: float = 0.0
|
||||
disk_free_gb: float = 0.0
|
||||
|
||||
|
||||
class GpuRouting(BaseModel):
|
||||
"""Routing verdict for the active TTS engine on THIS host (#21).
|
||||
|
||||
Distinct from the per-engine `routing_*` keys in `/engines`: this is the
|
||||
single verdict for the *currently-selected* engine, surfaced in preflight +
|
||||
diagnose so the user hears about a CPU fallback / unavailable GPU before a
|
||||
slow or failed synth — no silent CPU fallback.
|
||||
"""
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
engine: str | None = None # active TTS engine id
|
||||
effective_device: str | None = None # device it will actually use here
|
||||
routing_status: str | None = None # accelerated|cpu_fallback|cpu_only|unavailable|none
|
||||
routing_reason: str | None = None # scrubbed; null when none
|
||||
host_family: str = "cpu" # detect_host_caps().family
|
||||
vram_gb: float = 0.0
|
||||
|
||||
|
||||
class PreflightResponse(BaseModel):
|
||||
"""GET /setup/preflight"""
|
||||
ok: bool
|
||||
has_warnings: bool = False
|
||||
checks: list[PreflightCheck] = Field(default_factory=list)
|
||||
device: DeviceInfo
|
||||
# Explicit field (PreflightResponse has no extra="allow") so the verdict
|
||||
# survives serialization instead of being silently dropped.
|
||||
gpu_routing: GpuRouting | None = None
|
||||
|
||||
|
||||
class InstallModelRequest(BaseModel):
|
||||
|
||||
+6
-88
@@ -49,12 +49,6 @@ _BASE_SCHEMA = """
|
||||
personality TEXT DEFAULT '',
|
||||
description TEXT DEFAULT '',
|
||||
is_demo INTEGER DEFAULT 0,
|
||||
verified_own_voice INTEGER DEFAULT 0,
|
||||
consent_text TEXT DEFAULT '',
|
||||
consent_audio_path TEXT DEFAULT '',
|
||||
consent_recorded_at REAL DEFAULT NULL,
|
||||
kind TEXT DEFAULT 'clone',
|
||||
vd_states TEXT DEFAULT NULL,
|
||||
created_at REAL
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS generation_history (
|
||||
@@ -144,19 +138,6 @@ _BASE_SCHEMA = """
|
||||
value TEXT NOT NULL,
|
||||
updated_at REAL NOT NULL
|
||||
);
|
||||
|
||||
-- Wave 2.2: per-agent MCP voice bindings. An MCP client (Claude Code,
|
||||
-- Cursor, …) identified by the X-OmniVoice-Client-Id header it sends is
|
||||
-- bound to a default voice profile / engine. Fresh installs create it
|
||||
-- here; v0.3.x upgrades get it via alembic 0004.
|
||||
CREATE TABLE IF NOT EXISTS mcp_client_bindings (
|
||||
client_id TEXT PRIMARY KEY,
|
||||
label TEXT NOT NULL DEFAULT '',
|
||||
profile_id TEXT,
|
||||
default_engine TEXT,
|
||||
last_seen_at REAL,
|
||||
created_at REAL
|
||||
);
|
||||
"""
|
||||
|
||||
# Only tables/columns this module is allowed to ALTER. Prevents SQL injection via
|
||||
@@ -206,52 +187,6 @@ def _migrate(conn, current: int) -> int:
|
||||
return current
|
||||
|
||||
|
||||
def _reconcile_additive_columns(conn) -> None:
|
||||
"""Make the live schema converge to ``_BASE_SCHEMA`` by ADDing any column the
|
||||
canonical schema declares but an existing table is missing — the belt for
|
||||
when alembic can't run on an upgraded DB.
|
||||
|
||||
``CREATE TABLE IF NOT EXISTS`` (init_db) never adds columns to a table that
|
||||
already exists, the legacy ``_migrate`` only knows pre-0.3 columns, and
|
||||
``_run_alembic_upgrade`` swallows failures. So a DB whose ``alembic_version``
|
||||
is stamped at a removed revision (e.g. after running a preview build), or
|
||||
where alembic isn't importable in the bundled interpreter, would otherwise
|
||||
lose every alembic-era additive column forever — the ``no such column:
|
||||
consent_audio_path`` 500 (#552/#547), and the same class for
|
||||
``kind``/``vd_states``/``is_demo``/.... Additive only: never drops or retypes
|
||||
a column, so it is safe and backward-compatible with existing user data. The
|
||||
canonical names/types/defaults come solely from ``_BASE_SCHEMA`` (developer
|
||||
controlled), so the ALTER is injection-safe.
|
||||
"""
|
||||
canon = sqlite3.connect(":memory:")
|
||||
try:
|
||||
canon.executescript(_BASE_SCHEMA)
|
||||
_tables_sql = "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'"
|
||||
live_tables = {r[0] for r in conn.execute(_tables_sql)}
|
||||
for table in (r[0] for r in canon.execute(_tables_sql)):
|
||||
if table not in live_tables:
|
||||
continue # whole table missing → init_db's CREATE already made it
|
||||
have = {r[1] for r in conn.execute(f"PRAGMA table_info({table})")}
|
||||
# (cid, name, type, notnull, dflt_value, pk)
|
||||
for _cid, name, ctype, notnull, dflt, _pk in canon.execute(f"PRAGMA table_info({table})"):
|
||||
if name in have or not _IDENT_RE.match(name):
|
||||
continue
|
||||
ddl = f'ALTER TABLE "{table}" ADD COLUMN "{name}" {ctype or "TEXT"}'
|
||||
if dflt is not None:
|
||||
ddl += f" DEFAULT {dflt}"
|
||||
elif notnull:
|
||||
ddl += " DEFAULT ''" # SQLite requires a default to ADD a NOT NULL column
|
||||
try:
|
||||
conn.execute(ddl)
|
||||
logger.info("schema reconcile: added missing column %s.%s", table, name)
|
||||
except sqlite3.OperationalError as exc:
|
||||
if "duplicate column" not in str(exc).lower():
|
||||
logger.warning("schema reconcile ALTER %s.%s failed: %s", table, name, exc)
|
||||
conn.commit()
|
||||
finally:
|
||||
canon.close()
|
||||
|
||||
|
||||
def init_db():
|
||||
conn = get_db()
|
||||
try:
|
||||
@@ -260,11 +195,6 @@ def init_db():
|
||||
new_version = _migrate(conn, version)
|
||||
if new_version != version:
|
||||
conn.execute(f"PRAGMA user_version = {new_version}")
|
||||
# Converge any alembic-era additive columns that CREATE TABLE IF NOT
|
||||
# EXISTS + the legacy _migrate don't add to a pre-existing table
|
||||
# (consent_audio_path, kind, ...). Runs regardless of whether alembic
|
||||
# below succeeds, so an unrunnable alembic can't leave a 500-ing schema.
|
||||
_reconcile_additive_columns(conn)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
@@ -278,12 +208,10 @@ def init_db():
|
||||
|
||||
def _run_alembic_upgrade() -> None:
|
||||
"""Best-effort `alembic upgrade head` on startup. Non-fatal: if alembic
|
||||
isn't reachable (e.g. a stripped-down install) or its version is stamped at
|
||||
a revision no longer in versions/ (e.g. after running a preview build), log
|
||||
a warning and move on. The schema is still kept correct by
|
||||
_reconcile_additive_columns (run in init_db above and again here on failure)
|
||||
— CREATE TABLE IF NOT EXISTS alone does NOT add columns to a pre-existing
|
||||
table, so the reconcile is what actually guarantees additive columns land."""
|
||||
isn't reachable (e.g. user running a stripped-down install or migrations
|
||||
were already applied out-of-band), log a warning and move on. The
|
||||
_BASE_SCHEMA CREATE TABLE IF NOT EXISTS above guarantees the runtime
|
||||
schema is correct regardless."""
|
||||
try:
|
||||
import os
|
||||
from alembic import command
|
||||
@@ -301,16 +229,6 @@ def _run_alembic_upgrade() -> None:
|
||||
cfg.set_main_option("sqlalchemy.url", f"sqlite:///{DB_PATH}")
|
||||
command.upgrade(cfg, "head")
|
||||
except Exception as exc:
|
||||
# Don't block startup on a migration tooling problem. Converge the schema
|
||||
# directly so a swallowed failure (alembic not importable, or
|
||||
# alembic_version stamped at a removed revision) still lands the additive
|
||||
# columns instead of 500-ing on `no such column` (#552/#547).
|
||||
# Don't block startup on a migration tooling problem. The runtime
|
||||
# schema is already correct via _BASE_SCHEMA.
|
||||
logger.warning("alembic upgrade head skipped: %s", exc)
|
||||
try:
|
||||
conn = get_db()
|
||||
try:
|
||||
_reconcile_additive_columns(conn)
|
||||
finally:
|
||||
conn.close()
|
||||
except Exception as exc2: # noqa: BLE001
|
||||
logger.warning("schema reconcile after alembic failure also failed: %s", exc2)
|
||||
|
||||
@@ -1,336 +0,0 @@
|
||||
"""Free-text voice-description → voice-design parameter mapper (issue #317).
|
||||
|
||||
Parity with the hosted omnivoice.app "Describe your voice" field, implemented
|
||||
fully locally: a deterministic keyword/phrase mapper that projects a natural-
|
||||
language description (e.g. ``"a warm elderly British storyteller, slightly
|
||||
raspy"``) onto the **existing** voice-design parameter space — the same six
|
||||
categories the Design tab's attribute picker drives (Gender / Age / Pitch /
|
||||
Style / EnglishAccent / ChineseDialect).
|
||||
|
||||
Design notes
|
||||
============
|
||||
* **No model, no network.** This is an ordered synonym-table matcher, not an
|
||||
LLM call — it runs identically on macOS/Windows/Linux with zero deps beyond
|
||||
the stdlib, preserving the local-first guarantee.
|
||||
* **Single source of truth.** Every canonical token this module can emit is
|
||||
validated at import time against the engine taxonomy in
|
||||
``omnivoice/utils/voice_design.py`` (loaded via ``core.archetypes``), so the
|
||||
mapper can never produce an instruct item the engine validator would reject
|
||||
(the issue-#89 / #115 crash modes). The Chinese translations of each token
|
||||
(e.g. ``男``/``中年``) are *derived* from that taxonomy, never hardcoded.
|
||||
* **Ordered rules, first match wins.** Within a category, rules are checked in
|
||||
a hand-ordered list so more specific phrases outrank generic ones
|
||||
("young child" → child, not young adult; "very deep" → very low pitch, not
|
||||
low pitch). Within one rule, the earliest occurrence in the text is reported
|
||||
as the matched phrase. Deterministic by construction.
|
||||
* **Graceful degradation.** Anything the taxonomy can't express (timbre words
|
||||
like "raspy", role words like "storyteller") is returned in ``unmatched`` so
|
||||
the UI can tell the user exactly which parts were ignored instead of failing
|
||||
silently (issue #317's validation-feedback note). A description with no
|
||||
matches at all yields all-``Auto`` attrs and an empty instruct.
|
||||
|
||||
Localization note (CLAUDE.md): the only hardcoded CJK here is
|
||||
``DIALECT_PINYIN`` — a functional pinyin → Chinese-dialect-token mapping
|
||||
(model vocabulary, like ``frontend/src/utils/constants.js``). Registered in
|
||||
``tests/test_no_hardcoded_cjk.py``'s allowlist with this justification.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
# Reuse the taxonomy already loaded (stdlib-only, by file path) by the
|
||||
# archetype engine — same single source of truth, one loader to maintain.
|
||||
from core.archetypes import _VD
|
||||
|
||||
_EN_TO_ZH = _VD._INSTRUCT_EN_TO_ZH # {"male": "男", ...}
|
||||
_ZH_RE = _VD._ZH_RE
|
||||
_VALID = _VD._INSTRUCT_ALL_VALID # every token the engine accepts
|
||||
_DIALECTS = set(_VD._INSTRUCT_CATEGORIES[5]) # the 12 Chinese dialect tokens
|
||||
|
||||
# Category names match the frontend's CATEGORIES keys (utils/constants.js) and
|
||||
# the archetype ``attrs`` shape, so the response drops straight into vdStates.
|
||||
CATEGORY_ORDER = ("Gender", "Age", "Pitch", "Style", "EnglishAccent", "ChineseDialect")
|
||||
|
||||
# ── Pinyin / romanized names → Chinese-dialect tokens (functional vocabulary) ─
|
||||
DIALECT_PINYIN = {
|
||||
"henan": "河南话",
|
||||
"shaanxi": "陕西话",
|
||||
"sichuan": "四川话",
|
||||
"szechuan": "四川话",
|
||||
"guizhou": "贵州话",
|
||||
"yunnan": "云南话",
|
||||
"guilin": "桂林话",
|
||||
"jinan": "济南话",
|
||||
"shijiazhuang": "石家庄话",
|
||||
"gansu": "甘肃话",
|
||||
"ningxia": "宁夏话",
|
||||
"qingdao": "青岛话",
|
||||
"dongbei": "东北话",
|
||||
"northeastern chinese": "东北话",
|
||||
}
|
||||
|
||||
# ── Synonym tables ────────────────────────────────────────────────────────────
|
||||
# Per category: ordered list of (canonical_token, [phrases]). First rule with
|
||||
# any hit wins the category, so specific phrases must precede generic ones.
|
||||
# Each canonical token's Chinese translation from the taxonomy is appended
|
||||
# automatically at compile time (so "中年" maps to "middle-aged", etc.).
|
||||
|
||||
_GENDER_RULES = [
|
||||
("female", [
|
||||
"female", "woman", "women", "lady", "ladies", "girl", "girls",
|
||||
"feminine", "gal", "grandma", "grandmother", "granny", "mother",
|
||||
"mom", "mum", "aunt", "auntie", "queen", "princess", "actress",
|
||||
"she", "her",
|
||||
]),
|
||||
("male", [
|
||||
"male", "man", "men", "guy", "guys", "boy", "boys", "masculine",
|
||||
"gentleman", "gentlemen", "dude", "grandpa", "grandfather", "father",
|
||||
"dad", "uncle", "king", "prince", "actor", "he", "him", "his",
|
||||
]),
|
||||
]
|
||||
|
||||
# Order is load-bearing: "child" precedes "young adult" so "young child" →
|
||||
# child; "middle-aged" precedes "elderly" so elderly's bare "aged" synonym
|
||||
# can't fire inside the hyphenated "middle-aged" (hyphen is a \b boundary);
|
||||
# "elderly" precedes "young adult" so grandparent words don't fall through.
|
||||
_AGE_RULES = [
|
||||
("child", [
|
||||
"child", "children", "kid", "kiddo", "toddler", "little boy",
|
||||
"little girl", "young boy", "young girl", "small child", "childlike",
|
||||
]),
|
||||
("teenager", ["teenager", "teen", "teenage", "adolescent"]),
|
||||
("middle-aged", [
|
||||
"middle-aged", "middle aged", "middle age", "midlife", "forties",
|
||||
"fifties", "sixties", "mature",
|
||||
]),
|
||||
("elderly", [
|
||||
"elderly", "old man", "old woman", "old lady", "older man",
|
||||
"older woman", "elder", "senior", "aged", "grandpa", "grandfather",
|
||||
"grandma", "grandmother", "granny", "retired", "seventies",
|
||||
"eighties", "nineties", "old",
|
||||
]),
|
||||
("young adult", [
|
||||
"young adult", "young woman", "young man", "young lady", "youthful",
|
||||
"twenties", "thirties", "college", "young",
|
||||
]),
|
||||
]
|
||||
|
||||
# "very …" rules precede their plain counterparts so "very deep" doesn't stop
|
||||
# at "deep". Bare "low"/"high" only count next to a voice word (pitch/voice/
|
||||
# tone/register) to avoid false hits like "high quality" or "low effort".
|
||||
_PITCH_RULES = [
|
||||
("very low pitch", [
|
||||
"very low pitch", "very low-pitched", "very low pitched",
|
||||
"very low voice", "very low tone", "very deep", "extremely deep",
|
||||
"extremely low", "ultra deep", "booming",
|
||||
]),
|
||||
("very high pitch", [
|
||||
"very high pitch", "very high-pitched", "very high pitched",
|
||||
"very high voice", "very high tone", "extremely high", "squeaky",
|
||||
"shrill", "falsetto", "chipmunk",
|
||||
]),
|
||||
("low pitch", [
|
||||
"low pitch", "low-pitched", "low pitched", "low voice", "low tone",
|
||||
"low register", "deep", "deeper", "bass", "baritone", "husky",
|
||||
]),
|
||||
("high pitch", [
|
||||
"high pitch", "high-pitched", "high pitched", "high voice",
|
||||
"high tone", "high register", "soprano",
|
||||
]),
|
||||
("moderate pitch", [
|
||||
"moderate pitch", "medium pitch", "medium-pitched", "medium pitched",
|
||||
"mid-range", "midrange", "average pitch", "moderate",
|
||||
]),
|
||||
]
|
||||
|
||||
_STYLE_RULES = [
|
||||
("whisper", [
|
||||
"whisper", "whispering", "whispered", "whispery", "hushed",
|
||||
"breathy", "soft-spoken", "soft spoken",
|
||||
]),
|
||||
]
|
||||
|
||||
# Bare "english" means the language, so only the explicit "english accent"
|
||||
# phrase maps to british. "chinese" maps to the chinese *accent* (English
|
||||
# speech with a Chinese accent); actual dialect words live in DIALECT_PINYIN.
|
||||
_ACCENT_RULES = [
|
||||
("american accent", [
|
||||
"american", "america", "usa", "us accent", "midwestern",
|
||||
"californian", "new york",
|
||||
]),
|
||||
("british accent", [
|
||||
"british", "britain", "english accent", "england", "uk accent",
|
||||
"london", "cockney", "posh", "received pronunciation",
|
||||
]),
|
||||
("australian accent", ["australian", "australia", "aussie"]),
|
||||
("canadian accent", ["canadian", "canada"]),
|
||||
("indian accent", ["indian", "india"]),
|
||||
("chinese accent", ["chinese accent", "chinese-accented", "chinese"]),
|
||||
("korean accent", ["korean", "korea"]),
|
||||
("japanese accent", ["japanese", "japan"]),
|
||||
("portuguese accent", ["portuguese", "portugal", "brazilian", "brazil"]),
|
||||
("russian accent", ["russian", "russia"]),
|
||||
]
|
||||
|
||||
_DIALECT_RULES = [
|
||||
(token, [pinyin for pinyin, tok in DIALECT_PINYIN.items() if tok == token])
|
||||
for token in sorted(_DIALECTS)
|
||||
]
|
||||
|
||||
_RULES = {
|
||||
"Gender": _GENDER_RULES,
|
||||
"Age": _AGE_RULES,
|
||||
"Pitch": _PITCH_RULES,
|
||||
"Style": _STYLE_RULES,
|
||||
"EnglishAccent": _ACCENT_RULES,
|
||||
"ChineseDialect": _DIALECT_RULES,
|
||||
}
|
||||
|
||||
# Import-time guard: every canonical token must be in the engine taxonomy, so
|
||||
# a taxonomy rename upstream fails loudly here instead of at synthesis time.
|
||||
for _cat_rules in _RULES.values():
|
||||
for _token, _ in _cat_rules:
|
||||
assert _token in _VALID, f"describe_voice token not in taxonomy: {_token!r}"
|
||||
for _tok in DIALECT_PINYIN.values():
|
||||
assert _tok in _DIALECTS, f"DIALECT_PINYIN value not a taxonomy dialect: {_tok!r}"
|
||||
|
||||
|
||||
# ── Pattern compilation ───────────────────────────────────────────────────────
|
||||
def _compile_phrase(phrase: str) -> re.Pattern:
|
||||
"""Compile a synonym phrase to a regex.
|
||||
|
||||
Latin phrases get word boundaries (so "male" never fires inside "female",
|
||||
"old" never inside "bold") and flexible separators (space or hyphen, so
|
||||
"middle aged" also matches "middle-aged"). CJK phrases match as plain
|
||||
substrings — word boundaries are meaningless without spaces.
|
||||
"""
|
||||
if _ZH_RE.search(phrase):
|
||||
return re.compile(re.escape(phrase))
|
||||
parts = [re.escape(p) for p in re.split(r"[ -]+", phrase) if p]
|
||||
return re.compile(r"\b" + r"[\s\-]+".join(parts) + r"\b")
|
||||
|
||||
|
||||
def _compiled_rules():
|
||||
out = {}
|
||||
for cat, rules in _RULES.items():
|
||||
compiled = []
|
||||
for token, phrases in rules:
|
||||
pats = list(phrases)
|
||||
# Derive the Chinese form of each canonical token from the
|
||||
# taxonomy (e.g. "middle-aged" → "中年") — never hardcoded here.
|
||||
zh = _EN_TO_ZH.get(token)
|
||||
if zh:
|
||||
pats.append(zh)
|
||||
if token not in pats:
|
||||
pats.append(token) # the canonical token always matches itself
|
||||
compiled.append((token, [_compile_phrase(p) for p in pats]))
|
||||
out[cat] = compiled
|
||||
return out
|
||||
|
||||
|
||||
_COMPILED = _compiled_rules()
|
||||
|
||||
# "<N> year(s) old / <N>-year-old / <N> yo" → an age bracket. Runs before the
|
||||
# keyword rules so the trailing "old" never misfires as elderly.
|
||||
_AGE_NUM = re.compile(
|
||||
r"\b(\d{1,3})(?:[\s\-]*(?:years?|yrs?|yr)[\s\-]*old|[\s\-]*(?:yo|y/o))\b"
|
||||
)
|
||||
|
||||
|
||||
def _age_token_for(years: int) -> str:
|
||||
if years <= 12:
|
||||
return "child"
|
||||
if years <= 19:
|
||||
return "teenager"
|
||||
if years <= 39:
|
||||
return "young adult"
|
||||
if years <= 64:
|
||||
return "middle-aged"
|
||||
return "elderly"
|
||||
|
||||
|
||||
def _normalize(description: str) -> str:
|
||||
text = (description or "").lower()
|
||||
text = text.replace("’", "'").replace("‘", "'")
|
||||
text = text.replace("“", '"').replace("”", '"')
|
||||
return re.sub(r"[ \t]+", " ", text)
|
||||
|
||||
|
||||
def _match_category(category: str, text: str):
|
||||
"""Return (token, match) for the first rule with a hit, else None.
|
||||
|
||||
Rule order decides the winning token; within the winning rule the earliest
|
||||
occurrence in the text is reported as the matched phrase.
|
||||
"""
|
||||
if category == "Age":
|
||||
m = _AGE_NUM.search(text)
|
||||
if m:
|
||||
return _age_token_for(int(m.group(1))), m
|
||||
for token, patterns in _COMPILED[category]:
|
||||
best = None
|
||||
for pat in patterns:
|
||||
m = pat.search(text)
|
||||
if m is not None and (best is None or m.start() < best.start()):
|
||||
best = m
|
||||
if best is not None:
|
||||
return token, best
|
||||
return None
|
||||
|
||||
|
||||
# Fragment splitter for the "unmatched" report: clause separators (incl. the
|
||||
# CJK comma/ideographic stop, which CJK descriptions use instead of ASCII).
|
||||
_FRAGMENT = re.compile(r"[^,;.!?()\n,。;!?、]+")
|
||||
_HAS_CONTENT = re.compile(r"[\w一-鿿]")
|
||||
|
||||
|
||||
def parse_description(description: str) -> dict:
|
||||
"""Map a free-text voice description onto the design parameter space.
|
||||
|
||||
Returns a dict with:
|
||||
* ``attrs`` — full category → token map (``"Auto"`` where nothing
|
||||
matched); same shape as the Design tab's ``vdStates``.
|
||||
* ``instruct`` — validator-safe instruct string built from the matched
|
||||
tokens, in canonical category order (may be ``""``).
|
||||
* ``matched`` — list of ``{category, token, phrase}`` for transparency.
|
||||
* ``unmatched`` — clause fragments that contributed no attribute, so the
|
||||
UI can show what was ignored instead of failing silently.
|
||||
"""
|
||||
text = _normalize(description)
|
||||
attrs = {cat: "Auto" for cat in CATEGORY_ORDER}
|
||||
matched = []
|
||||
spans = []
|
||||
|
||||
for category in CATEGORY_ORDER:
|
||||
hit = _match_category(category, text)
|
||||
if hit is None:
|
||||
continue
|
||||
token, m = hit
|
||||
attrs[category] = token
|
||||
matched.append({"category": category, "token": token, "phrase": m.group(0)})
|
||||
spans.append((m.start(), m.end()))
|
||||
|
||||
# Accents are English-only and dialects Chinese-only in the engine
|
||||
# taxonomy; a dialect voice speaks Chinese, so an accent token alongside
|
||||
# it is contradictory (the issue-#114 conflict class). Dialect wins.
|
||||
if attrs["ChineseDialect"] != "Auto" and attrs["EnglishAccent"] != "Auto":
|
||||
dropped = attrs["EnglishAccent"]
|
||||
attrs["EnglishAccent"] = "Auto"
|
||||
matched = [m for m in matched if not (m["category"] == "EnglishAccent" and m["token"] == dropped)]
|
||||
|
||||
instruct = ", ".join(attrs[c] for c in CATEGORY_ORDER if attrs[c] != "Auto")
|
||||
|
||||
unmatched = []
|
||||
for frag in _FRAGMENT.finditer(text):
|
||||
if not _HAS_CONTENT.search(frag.group(0)):
|
||||
continue
|
||||
lo, hi = frag.start(), frag.end()
|
||||
if any(s < hi and e > lo for s, e in spans):
|
||||
continue
|
||||
unmatched.append(frag.group(0).strip())
|
||||
|
||||
return {
|
||||
"attrs": attrs,
|
||||
"instruct": instruct,
|
||||
"matched": matched,
|
||||
"unmatched": unmatched,
|
||||
}
|
||||
@@ -1,279 +0,0 @@
|
||||
"""Canonical host compute-capability probe — the single source of truth for
|
||||
"what can this machine actually accelerate on."
|
||||
|
||||
Every routing decision (the engine compatibility matrix, ``/setup/preflight``,
|
||||
``/system/diagnose``, and the synth-time no-silent-fallback gating) reads from
|
||||
``detect_host_caps()`` so the probe and the model loader can never disagree.
|
||||
|
||||
Design contract (load-bearing):
|
||||
- **Never raises** to a caller. A broken torch / driver crash degrades to a
|
||||
cached CPU-only ``probe_ok=False`` result; every endpoint stays responsive
|
||||
(local-first: the app must work with no GPU and even with a broken torch).
|
||||
- **No network call** — driver/sysctl reads only, no tensor allocation, so it
|
||||
stays kernel-free on cold start.
|
||||
- **No new regex** on any driver/device string (CodeQL py/polynomial-redos):
|
||||
the only string parse is the ``int(driver.split(".")[0])`` shape reused
|
||||
from the wizard, and arch comparison is plain list membership.
|
||||
- Distinguishes **ROCm from CUDA** (unlike the gguf ``hardware_probe``):
|
||||
ROCm-on-HIP presents through ``torch.cuda`` but is reported ``family="rocm"``.
|
||||
|
||||
The ``get_best_device()`` loader (``services.model_manager``) delegates its
|
||||
*family* decision here while keeping its own DirectML branch and the ROCm
|
||||
``HSA_OVERRIDE_GFX_VERSION`` env side-effect — the probe **reads**, the loader
|
||||
**writes**. (The gguf ``hardware_probe.detect_capabilities()`` rebase onto this
|
||||
module is a deliberate follow-up: it has its own torch-mocked test suite and a
|
||||
VRAM-driven quant table that is unaffected by the family rename, so it is kept
|
||||
out of this backend-only slice.)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import platform as _platform
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal
|
||||
|
||||
DeviceFamily = Literal["cuda", "rocm", "mps", "xpu", "cpu"]
|
||||
|
||||
# Stable substring stamped onto notes that represent a real kernel-launch risk
|
||||
# (arch/driver mismatch) — as opposed to advisory notes (multi-GPU, VRAM query
|
||||
# failed, DirectML present). ``engine_routing`` keys the "accelerated, but…"
|
||||
# caveat off this marker so advisory notes never downgrade an accelerated badge.
|
||||
KERNEL_RISK_MARKER = "may fail at kernel launch"
|
||||
|
||||
# Substring marking a DirectML-present (Windows GPU) host. The probe reports
|
||||
# such hosts as ``family="cpu"`` (DirectML is not a torch device family); the
|
||||
# router reads this marker to explain the neutral badge instead of "no GPU".
|
||||
DIRECTML_MARKER = "DirectML device present"
|
||||
|
||||
# NOTE: the NVIDIA driver-version check (min R555 for the bundled CUDA runtime)
|
||||
# is intentionally NOT done here — it requires shelling to ``nvidia-smi``, which
|
||||
# would put a subprocess on the cold-start probe path. That check stays in
|
||||
# ``wizard._detect_gpu`` (preflight), which already runs it. The probe only
|
||||
# emits the torch-visible SM-arch caveat (cheap, metadata-only).
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HostCaps:
|
||||
"""Snapshot of the host's accelerator capability. Immutable + cached."""
|
||||
|
||||
family: DeviceFamily
|
||||
"""Best available accelerator family, else ``"cpu"``."""
|
||||
|
||||
available_families: tuple[DeviceFamily, ...]
|
||||
"""Everything usable; **always includes** ``"cpu"`` (invariant)."""
|
||||
|
||||
device_name: str = ""
|
||||
"""Device 0's name, e.g. ``"NVIDIA RTX 4090"`` / ``"Apple Silicon (MPS)"``."""
|
||||
|
||||
vram_gb: float = 0.0
|
||||
"""CUDA/ROCm total VRAM in GB; MPS = system RAM / 2; 0 for cpu/xpu."""
|
||||
|
||||
driver: str | None = None
|
||||
"""Raw ROCm HIP version string (``torch.version.hip``) or ``None``. The
|
||||
NVIDIA driver-version check is owned by ``wizard._detect_gpu`` (it already
|
||||
shells to ``nvidia-smi``); the probe stays subprocess-free."""
|
||||
|
||||
notes: tuple[str, ...] = ()
|
||||
"""Author-controlled English advisories (never user input). Empty on a
|
||||
clean accelerated host."""
|
||||
|
||||
probe_ok: bool = True
|
||||
"""``False`` only when torch could not be imported (degraded CPU-only)."""
|
||||
|
||||
|
||||
def _probe() -> HostCaps:
|
||||
"""Run the probe once. Enumerates every failure branch from the spec's
|
||||
degradation contract; never raises."""
|
||||
try:
|
||||
import torch
|
||||
except Exception:
|
||||
return HostCaps(
|
||||
family="cpu",
|
||||
available_families=("cpu",),
|
||||
notes=("torch not importable; treating host as CPU-only",),
|
||||
probe_ok=False,
|
||||
)
|
||||
|
||||
notes: list[str] = []
|
||||
# Probe EVERY accelerator independently into this list (don't short-circuit
|
||||
# after the first hit) so `available_families` is honest on hybrid hosts
|
||||
# (e.g. an NVIDIA GPU + an Intel iGPU exposed via IPEX). The preferred
|
||||
# `family` is chosen by priority at the end.
|
||||
detected: list[DeviceFamily] = []
|
||||
device_name = ""
|
||||
vram_gb = 0.0
|
||||
driver: str | None = None
|
||||
|
||||
# ── CUDA / ROCm (both present through torch.cuda) ────────────────────
|
||||
cuda_ok = False
|
||||
try:
|
||||
cuda_ok = bool(torch.cuda.is_available())
|
||||
except Exception as exc: # broken CUDA init (forked process / driver crash)
|
||||
notes.append(f"CUDA init raised: {type(exc).__name__}")
|
||||
|
||||
if cuda_ok:
|
||||
try:
|
||||
count = int(torch.cuda.device_count())
|
||||
except Exception:
|
||||
count = 0
|
||||
if count == 0:
|
||||
notes.append("CUDA reports available but device_count==0")
|
||||
else:
|
||||
is_rocm = getattr(torch.version, "hip", None) is not None
|
||||
detected.append("rocm" if is_rocm else "cuda")
|
||||
if is_rocm:
|
||||
driver = getattr(torch.version, "hip", None)
|
||||
if count > 1:
|
||||
notes.append(f"{count} GPUs detected; routing reflects device 0")
|
||||
try:
|
||||
device_name = torch.cuda.get_device_name(0)
|
||||
except Exception:
|
||||
device_name = ""
|
||||
try:
|
||||
_free, total = torch.cuda.mem_get_info()
|
||||
vram_gb = float(total) / (1024 ** 3)
|
||||
except Exception:
|
||||
notes.append("VRAM query failed")
|
||||
# SM-arch mismatch (mirrors model_manager.check_device_compatibility).
|
||||
try:
|
||||
major, minor = torch.cuda.get_device_capability(0)
|
||||
arch_list = getattr(torch.cuda, "_get_arch_list", lambda: [])()
|
||||
if arch_list:
|
||||
sm_tag = f"sm_{major}{minor}"
|
||||
compute_tag = f"compute_{major}{minor}"
|
||||
if sm_tag not in arch_list and compute_tag not in arch_list:
|
||||
notes.append(
|
||||
f"{device_name or 'GPU'} ({sm_tag}) not in this torch "
|
||||
f"build's archs ({', '.join(arch_list)}) — "
|
||||
f"{KERNEL_RISK_MARKER}"
|
||||
)
|
||||
except Exception:
|
||||
# Arch metadata unavailable on this torch build — skip the check
|
||||
# (treated as compatible, exactly as check_device_compatibility).
|
||||
pass
|
||||
|
||||
# ── Intel XPU via IPEX ───────────────────────────────────────────────
|
||||
try:
|
||||
import intel_extension_for_pytorch # noqa: F401
|
||||
if hasattr(torch, "xpu") and torch.xpu.is_available():
|
||||
detected.append("xpu")
|
||||
if not device_name:
|
||||
try:
|
||||
device_name = torch.xpu.get_device_name(0)
|
||||
except Exception:
|
||||
# XPU present but unnamed — family classification still holds.
|
||||
pass
|
||||
notes.append("XPU VRAM not queried (unreliable across IPEX versions)")
|
||||
except Exception:
|
||||
# IPEX absent or XPU probe failed — no XPU on this host.
|
||||
pass
|
||||
|
||||
# ── Apple Silicon MPS ────────────────────────────────────────────────
|
||||
try:
|
||||
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
detected.append("mps")
|
||||
if not device_name:
|
||||
device_name = "Apple Silicon (MPS)"
|
||||
if not vram_gb:
|
||||
try:
|
||||
import psutil
|
||||
vram_gb = float(psutil.virtual_memory().total) / (1024 ** 3) / 2
|
||||
except Exception:
|
||||
notes.append("psutil unavailable; MPS VRAM unknown")
|
||||
except Exception:
|
||||
# MPS probe raised on a non-Apple/old torch — treat as no MPS.
|
||||
pass
|
||||
|
||||
# ── DirectML — Windows GPU, NOT a torch device family ────────────────
|
||||
try:
|
||||
import torch_directml
|
||||
if torch_directml.device_count() > 0:
|
||||
notes.append(
|
||||
f"{DIRECTML_MARKER} (Windows GPU); torch-family probe treats "
|
||||
f"as non-accelerated"
|
||||
)
|
||||
except Exception:
|
||||
# torch_directml absent (the common case) — no DirectML on this host.
|
||||
pass
|
||||
|
||||
# Preferred family by priority; cpu when nothing accelerated was detected.
|
||||
family: DeviceFamily = "cpu"
|
||||
for pref in ("cuda", "rocm", "xpu", "mps"):
|
||||
if pref in detected:
|
||||
family = pref # type: ignore[assignment]
|
||||
break
|
||||
# available_families: every detected accelerator + cpu, deduped, cpu last.
|
||||
available: tuple[DeviceFamily, ...] = tuple(dict.fromkeys([*detected, "cpu"]))
|
||||
|
||||
return HostCaps(
|
||||
family=family,
|
||||
available_families=available,
|
||||
device_name=device_name,
|
||||
vram_gb=vram_gb,
|
||||
driver=driver,
|
||||
notes=tuple(notes),
|
||||
probe_ok=True,
|
||||
)
|
||||
|
||||
|
||||
@functools.lru_cache(maxsize=1)
|
||||
def detect_host_caps() -> HostCaps:
|
||||
"""Cached per-process host capabilities. Never raises, makes no network
|
||||
call, kernel-free on cold start. Host compute capability does not change at
|
||||
runtime in any supported desktop flow (no GPU hot-plug; switching the active
|
||||
engine does not re-probe — routing is recomputed from these same caps), so
|
||||
a single probe per process is correct. ``probe_ok=False`` is cached too."""
|
||||
return _probe()
|
||||
|
||||
|
||||
def refresh() -> HostCaps:
|
||||
"""Clear the cache and re-probe. **TEST-ONLY** — nothing in the running app
|
||||
calls this (host caps are immutable per process)."""
|
||||
detect_host_caps.cache_clear()
|
||||
return detect_host_caps()
|
||||
|
||||
|
||||
def mlx_supported() -> tuple[bool, str]:
|
||||
"""``(ok, reason)``. ``ok=True`` **only** on Apple Silicon
|
||||
(``sys.platform == "darwin"`` and ``platform.machine() == "arm64"``) with
|
||||
torch MPS available — the shared gate for MLX-Audio / MLX-Whisper (#390).
|
||||
|
||||
Gates on exact-string equality (no regex → no CodeQL surface). On any
|
||||
non-Apple host it returns ``False`` **before** any package import, so a
|
||||
stray ``mlx_*`` wheel on Linux/Windows never reports available.
|
||||
"""
|
||||
if sys.platform != "darwin" or _platform.machine() != "arm64":
|
||||
if sys.platform == "darwin":
|
||||
return (False, "MLX requires Apple Silicon; this Mac is Intel")
|
||||
return (
|
||||
False,
|
||||
f"MLX requires Apple Silicon; this host is "
|
||||
f"{sys.platform}/{_platform.machine()}",
|
||||
)
|
||||
try:
|
||||
import torch
|
||||
except Exception:
|
||||
return (False, "torch not importable; cannot confirm MPS")
|
||||
try:
|
||||
if torch.backends.mps.is_available():
|
||||
return (True, "")
|
||||
except Exception:
|
||||
# MPS query raised — fall through to the conservative unavailable path.
|
||||
pass
|
||||
return (
|
||||
False,
|
||||
"Apple Silicon detected but torch MPS unavailable; "
|
||||
"reinstall torch with MPS support",
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"DeviceFamily",
|
||||
"HostCaps",
|
||||
"detect_host_caps",
|
||||
"refresh",
|
||||
"mlx_supported",
|
||||
"KERNEL_RISK_MARKER",
|
||||
"DIRECTML_MARKER",
|
||||
]
|
||||
@@ -4,7 +4,7 @@ One pass over everything a working install needs: Python, compute device,
|
||||
ffmpeg, HF token, disk, data-dir permissions, RAM, TTS engines, and (when
|
||||
requested) network reachability of the HuggingFace hub. Surfaced two ways:
|
||||
|
||||
- ``GET /system/diagnose`` (Settings > About -> "Run self-check")
|
||||
- ``GET /system/diagnose`` (Settings > About → "Run self-check")
|
||||
- ``python main.py --diagnose`` for headless installs / issue triage
|
||||
|
||||
Every ``detail``/``hint`` string is passed through ``core.scrub`` before it
|
||||
@@ -194,49 +194,6 @@ def _check_engines() -> dict:
|
||||
return _check("engines", "TTS engines", OK, detail)
|
||||
|
||||
|
||||
def _check_gpu_routing() -> dict:
|
||||
"""Routing verdict for the active TTS engine on THIS host (#21).
|
||||
|
||||
Surfaces a CPU fallback / unavailable-GPU *before* a slow or failed synth —
|
||||
the no-silent-fallback contract. `cpu_only` on a no-GPU machine is the
|
||||
expected normal state and stays OK (never noise-warns)."""
|
||||
try:
|
||||
from services.tts_backend import gpu_routing_verdict
|
||||
v = gpu_routing_verdict()
|
||||
except Exception as e:
|
||||
return _check("gpu_routing", "GPU routing", WARN, f"could not resolve: {e}")
|
||||
|
||||
status = v.get("routing_status")
|
||||
engine = v.get("engine") or "active engine"
|
||||
dev = v.get("effective_device") or "?"
|
||||
reason = v.get("routing_reason")
|
||||
host = v.get("host_family", "cpu")
|
||||
|
||||
if status == "accelerated":
|
||||
if reason: # driver/arch caveat — accelerated but at risk
|
||||
return _check("gpu_routing", "GPU routing", WARN,
|
||||
f"{engine} -> {dev}: {reason}",
|
||||
"The GPU is selected but may fail at kernel launch — "
|
||||
"update drivers / reinstall torch for this GPU arch.")
|
||||
return _check("gpu_routing", "GPU routing", OK, f"{engine} -> {dev} (accelerated)")
|
||||
if status == "cpu_fallback":
|
||||
return _check("gpu_routing", "GPU routing", WARN,
|
||||
f"{engine} runs on CPU: {reason or 'no GPU path for this host'}",
|
||||
"Pick an engine that supports this host's GPU for a big speedup, "
|
||||
"or continue on CPU (slower).")
|
||||
if status == "cpu_only":
|
||||
return _check("gpu_routing", "GPU routing", OK,
|
||||
f"{engine} -> cpu (no accelerator on this host)")
|
||||
if status == "unavailable":
|
||||
return _check("gpu_routing", "GPU routing", FAIL,
|
||||
f"{engine} can't run on this host: {reason or f'needs a GPU; host is {host}'}",
|
||||
"Select an engine with a CPU path in Settings -> Engines.")
|
||||
# status == "none" / unknown — no active engine resolved.
|
||||
return _check("gpu_routing", "GPU routing", WARN,
|
||||
"No active TTS engine resolved for routing.",
|
||||
"Pick an engine in Settings -> Engines.")
|
||||
|
||||
|
||||
_DEEP_TIMEOUT_S = 180
|
||||
|
||||
|
||||
@@ -307,8 +264,6 @@ def _check_network() -> dict:
|
||||
# all model downloads need to get started. urllib honors HTTP(S)_PROXY.
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
if not _HUB_URL.startswith("https://"): # constant today; guard the sink anyway
|
||||
raise ValueError(f"hub URL must be https, got {_HUB_URL!r}")
|
||||
req = urllib.request.Request(_HUB_URL, method="HEAD")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=_HUB_TIMEOUT_S):
|
||||
@@ -341,7 +296,6 @@ def run_diagnostics(include_network: bool = True, deep: bool = False) -> dict:
|
||||
_check_data_dir(),
|
||||
_check_ram(),
|
||||
_check_engines(),
|
||||
_check_gpu_routing(),
|
||||
]
|
||||
if include_network:
|
||||
checks.append(_check_network())
|
||||
|
||||
@@ -111,8 +111,7 @@ def classify_exception(exc: BaseException, trace: str = "") -> str:
|
||||
def _fingerprint(error_class: str, exc: BaseException) -> str:
|
||||
import hashlib
|
||||
raw = f"{error_class}|{type(exc).__name__}|{scrub_text(str(exc))[:200]}"
|
||||
# Dedup key for the journal, not a security boundary.
|
||||
return hashlib.sha1(raw.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:16]
|
||||
return hashlib.sha1(raw.encode("utf-8", "replace")).hexdigest()[:16]
|
||||
|
||||
|
||||
def _persist_locked() -> None:
|
||||
|
||||
+1
-31
@@ -32,16 +32,11 @@ _REDACTED_VALUE = "***REDACTED***"
|
||||
# One-line "what to do" per docs-taxonomy key. Keys mirror error_docs_map's
|
||||
# taxonomy; the docs URL itself stays owned by error_docs_map.
|
||||
_HINTS: dict[str, str] = {
|
||||
"PKG_RESOURCES_MISSING": "Run `uv pip install --reinstall 'setuptools>=75,<80'` in the backend venv (a plain install is skipped when setuptools' metadata is present but its pkg_resources files were removed by antivirus). Restart after.",
|
||||
"PKG_RESOURCES_MISSING": "Install setuptools in the backend environment (provides pkg_resources).",
|
||||
"GATEKEEPER_QUARANTINE": "Clear the macOS quarantine flag (xattr -cr the app), then reopen.",
|
||||
"APPIMAGE_WEBKIT_WHITESCREEN": "Launch with WEBKIT_DISABLE_DMABUF_RENDERER=1 set.",
|
||||
"HF_AUTH_FAILED": "Set a valid HF_TOKEN in Settings → Hugging Face and retry.",
|
||||
"PYANNOTE_LICENSE_REQUIRED": "Accept the pyannote model licenses on Hugging Face, then retry.",
|
||||
"COMPUTE_TYPE_UNSUPPORTED": "Your GPU doesn't support float16 — OmniVoice retried on int8. If transcription still fails, set OMNIVOICE/ASR_COMPUTE_TYPE=int8 or use CPU.",
|
||||
"TRANSFORMERS_IMPORT": "Your transformers install is incomplete. Reinstall it (`uv pip install --reinstall transformers`) or switch ASR to faster-whisper (Settings → Models).",
|
||||
"UNSUPPORTED_VIDEO_URL": "This link isn't a directly downloadable video. Paste a direct video page (e.g. a youtube.com/watch?v=… or douyin.com/video/<id> link), not a share/profile/feed link — or download the file and drop it in directly.",
|
||||
"VIDEO_DOWNLOAD_NETWORK": "The connection to the video server dropped mid-download (often a transient CDN/network blip or a regional rate-limit). Just retry — OmniVoice already cleaned up the partial download. If it keeps failing, check your network/VPN.",
|
||||
"BROKEN_VENV": "The Python backend environment was moved or damaged. OmniVoice rebuilds it automatically on the next launch; if it keeps failing, use Clean & Retry on the setup screen.",
|
||||
}
|
||||
|
||||
|
||||
@@ -60,35 +55,10 @@ def classify(reason: str) -> str:
|
||||
return "APPIMAGE_WEBKIT_WHITESCREEN"
|
||||
if "pyannote" in low or ("gated" in low and "model" in low) or "accept the" in low:
|
||||
return "PYANNOTE_LICENSE_REQUIRED"
|
||||
# ASR robustness (#551 / #549): name the class so the no-segments toast is
|
||||
# actionable. Place before the generic returns so a compute-type/transformers
|
||||
# failure gets its hint rather than falling through to "".
|
||||
if "compute type" in low or "efficient float16" in low:
|
||||
return "COMPUTE_TYPE_UNSUPPORTED"
|
||||
if "could not import module" in low or "autofeatureextractor" in low:
|
||||
return "TRANSFORMERS_IMPORT"
|
||||
if ("huggingface" in low or "hf_token" in low or "401" in low or "unauthorized" in low) and (
|
||||
"token" in low or "auth" in low or "401" in low or "unauthorized" in low
|
||||
):
|
||||
return "HF_AUTH_FAILED"
|
||||
# Video download (#554/#536): a non-downloadable URL shape vs a transient
|
||||
# network drop — both previously surfaced as a bare yt-dlp string with no
|
||||
# next step. UNSUPPORTED first (more specific) so "Unable to download video:
|
||||
# Broken pipe" still classifies as a network blip.
|
||||
if "unsupported url" in low or "no video formats" in low or "is not a valid url" in low:
|
||||
return "UNSUPPORTED_VIDEO_URL"
|
||||
if (
|
||||
"broken pipe" in low
|
||||
or "connection reset" in low
|
||||
or "unable to download video" in low
|
||||
or "remote end closed" in low
|
||||
or "timed out" in low
|
||||
):
|
||||
return "VIDEO_DOWNLOAD_NETWORK"
|
||||
# A relocated/corrupted venv whose interpreter can't bootstrap its stdlib —
|
||||
# the Rust self-heal rebuilds it; this names the class for the toast.
|
||||
if "no module named 'encodings'" in low:
|
||||
return "BROKEN_VENV"
|
||||
return ""
|
||||
|
||||
|
||||
|
||||
@@ -42,12 +42,9 @@ _TOKEN_PATTERNS = (
|
||||
# pattern-wise (not just this machine's $HOME) so paths quoted from a
|
||||
# user's pasted log on another OS get cleaned too.
|
||||
_HOME_PATTERNS = (
|
||||
# Windows-with-forward-slashes must run BEFORE the bare macOS shape, or
|
||||
# `/Users/<name>` inside `C:/Users/<name>` gets eaten first, leaving `C:~`.
|
||||
re.compile(r"[A-Za-z]:/Users/[^/\s\"']+"), # Windows, forward slashes (file URLs, normalized traces)
|
||||
re.compile(r"/Users/[^/\s\"']+"), # macOS
|
||||
re.compile(r"/home/[^/\s\"']+"), # Linux
|
||||
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+"), # Windows, backslashes
|
||||
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+"), # Windows
|
||||
)
|
||||
|
||||
# Values shorter than this are too entropy-poor to be real secrets and too
|
||||
|
||||
@@ -82,28 +82,3 @@ def unset_user_env(key: str, path: Optional[str] = None) -> None:
|
||||
prefix = f"{key}="
|
||||
lines = [ln for ln in _read_lines(path) if not ln.startswith(prefix)]
|
||||
_write_lines(path, lines)
|
||||
|
||||
|
||||
def load_into_environ(path: Optional[str] = None) -> bool:
|
||||
"""Load the durable per-user env file into ``os.environ``, **overriding**
|
||||
any value a launcher already injected. Returns True if a file was loaded.
|
||||
|
||||
This file is the in-app Settings source of truth. The desktop launcher
|
||||
(Tauri) injects defaults like ``OMNIVOICE_CACHE_DIR`` (and ``HF_ENDPOINT``)
|
||||
from its *own* config into the backend's environment *before* startup, so
|
||||
loading this file with ``override=False`` meant a models directory the user
|
||||
changed in Settings was silently ignored on every launch — the effective
|
||||
location stayed on the old one no matter how many restarts (#480). Both keys
|
||||
this file can hold are the user's explicit Settings choice and should beat
|
||||
the launcher's default, so we override. Restores this file's documented
|
||||
"values written here take effect on the next backend launch" contract.
|
||||
"""
|
||||
path = path or os.environ.get("OMNIVOICE_ENV_FILE") or USER_ENV_PATH
|
||||
if not os.path.isfile(path):
|
||||
return False
|
||||
try:
|
||||
import dotenv
|
||||
except ImportError:
|
||||
return False
|
||||
dotenv.load_dotenv(path, override=True)
|
||||
return True
|
||||
|
||||
+4
-33
@@ -2,43 +2,14 @@
|
||||
|
||||
Read from the installed package metadata (driven by ``pyproject.toml``) so the
|
||||
FastAPI/API version and exported-bundle metadata never drift to a stale literal
|
||||
— the prior "0.4.0" / "0.2.7" bug, and the v0.3.6 desktop build that reported
|
||||
"0.3.5" because the *frozen* backend couldn't read its own metadata.
|
||||
|
||||
Resolution order:
|
||||
1. installed package metadata — correct in any ``uv sync``'d env and, thanks
|
||||
to ``copy_metadata('omnivoice')`` in ``backend.spec``, in the frozen build;
|
||||
2. ``pyproject.toml`` walked up from this file — correct for a raw source
|
||||
checkout that was never installed;
|
||||
3. ``_FALLBACK_VERSION`` — a last resort, kept in lockstep with the four
|
||||
version files by ``tests/test_app_version.py`` so it can never silently
|
||||
drift again.
|
||||
again (the prior "0.4.0" / "0.2.7" bug). Falls back to a literal only when
|
||||
running from a raw source checkout that was never ``uv sync``'d.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from importlib.metadata import PackageNotFoundError, version
|
||||
from pathlib import Path
|
||||
|
||||
# Last-resort literal. Guarded by
|
||||
# tests/test_app_version.py::test_all_version_files_in_lockstep and bumped by
|
||||
# release.yml's version-bump job, so it stays equal to
|
||||
# pyproject/tauri.conf/Cargo/package.json.
|
||||
_FALLBACK_VERSION = "0.3.7"
|
||||
|
||||
|
||||
def _fallback_version() -> str:
|
||||
"""Version for contexts where package metadata is unavailable."""
|
||||
for parent in Path(__file__).resolve().parents:
|
||||
pyproject = parent / "pyproject.toml"
|
||||
if pyproject.is_file():
|
||||
match = re.search(r'(?m)^version\s*=\s*"([^"]+)"', pyproject.read_text())
|
||||
if match:
|
||||
return match.group(1)
|
||||
return _FALLBACK_VERSION
|
||||
|
||||
|
||||
try:
|
||||
APP_VERSION = version("omnivoice")
|
||||
except PackageNotFoundError: # frozen build w/o metadata, or non-installed checkout
|
||||
APP_VERSION = _fallback_version()
|
||||
except PackageNotFoundError: # non-installed source checkout
|
||||
APP_VERSION = "0.3.5"
|
||||
|
||||
@@ -1,148 +0,0 @@
|
||||
"""Crash-isolated faster-whisper ASR sidecar (Wave 4.2 / Spec 7).
|
||||
|
||||
Runs faster-whisper in a child process so a CTranslate2 GPU-teardown segfault
|
||||
becomes a failed job, not a dead backend. Speaks the SubprocessBackend wire
|
||||
protocol (length-prefixed JSON over stdin/stdout):
|
||||
|
||||
on start → {"op":"ready","engine":"faster-whisper-isolated"}
|
||||
{"op":"ping"} → {"op":"pong"}
|
||||
{"op":"transcribe","audio_path":...,"word_timestamps":bool}
|
||||
→ {"op":"segments","result":{"segments":[...],"language":...}}
|
||||
{"op":"shutdown"} → exit 0
|
||||
error → {"op":"error","message":...}
|
||||
|
||||
Runs under the PARENT venv (faster-whisper is already a dependency) — only the
|
||||
process boundary is new. torch/CTranslate2 import lazily inside transcribe so
|
||||
the ready handshake fits the spawn timeout.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
MAX_FRAME_BYTES = 64 * 1024 * 1024
|
||||
_model = None
|
||||
|
||||
|
||||
def _send(stream, obj):
|
||||
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
|
||||
stream.write(struct.pack("!I", len(body)))
|
||||
stream.write(body)
|
||||
stream.flush()
|
||||
|
||||
|
||||
def _recv(stream):
|
||||
header = stream.read(4)
|
||||
if len(header) < 4:
|
||||
return None
|
||||
(n,) = struct.unpack("!I", header)
|
||||
if n > MAX_FRAME_BYTES:
|
||||
raise IOError(f"frame too large: {n}")
|
||||
body = bytearray()
|
||||
while len(body) < n:
|
||||
chunk = stream.read(n - len(body))
|
||||
if not chunk:
|
||||
raise IOError("short read")
|
||||
body.extend(chunk)
|
||||
return json.loads(bytes(body).decode("utf-8"))
|
||||
|
||||
|
||||
# NOTE: keep this compute_type fallback in lockstep with
|
||||
# services/asr_backend.py:_compute_type_candidates / _is_compute_type_error.
|
||||
# This sidecar runs in a child proc with a clean import path, so we duplicate a
|
||||
# tiny copy rather than cross-importing the heavy services package (#551).
|
||||
def _ct_candidates(device):
|
||||
override = os.environ.get("ASR_COMPUTE_TYPE")
|
||||
if override:
|
||||
return [override]
|
||||
return ["float16", "int8_float16", "int8"] if device == "cuda" else ["int8", "float32"]
|
||||
|
||||
|
||||
def _is_ct_error(msg):
|
||||
low = msg.lower()
|
||||
return "compute type" in low or "efficient float16" in low
|
||||
|
||||
|
||||
def _get_model():
|
||||
global _model
|
||||
if _model is None:
|
||||
from faster_whisper import WhisperModel
|
||||
name = os.environ.get("ASR_MODEL_FW", "large-v3")
|
||||
try:
|
||||
import torch
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
except Exception:
|
||||
device = "cpu"
|
||||
# Degrade fp16 → int8 rather than crash on GPUs without efficient fp16
|
||||
# (older Maxwell/Pascal, GTX 16xx, CTranslate2/cuDNN mismatch) (#551).
|
||||
last_err = None
|
||||
for compute in _ct_candidates(device):
|
||||
try:
|
||||
_model = WhisperModel(name, device=device, compute_type=compute)
|
||||
break
|
||||
except (ValueError, RuntimeError) as e:
|
||||
last_err = e
|
||||
if _is_ct_error(str(e)):
|
||||
continue
|
||||
raise
|
||||
else:
|
||||
raise last_err
|
||||
return _model
|
||||
|
||||
|
||||
def _transcribe(audio_path, word_timestamps):
|
||||
model = _get_model()
|
||||
segments, info = model.transcribe(audio_path, word_timestamps=word_timestamps)
|
||||
out = []
|
||||
for s in segments:
|
||||
seg = {"start": float(s.start), "end": float(s.end), "text": s.text}
|
||||
if word_timestamps and getattr(s, "words", None):
|
||||
seg["words"] = [
|
||||
{"word": w.word, "start": float(w.start), "end": float(w.end),
|
||||
"probability": float(getattr(w, "probability", 0.0))}
|
||||
for w in s.words
|
||||
]
|
||||
out.append(seg)
|
||||
return {
|
||||
"segments": out,
|
||||
"text": " ".join(s["text"].strip() for s in out).strip(),
|
||||
"language": getattr(info, "language", "unknown"),
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
stdin, stdout = sys.stdin.buffer, sys.stdout.buffer
|
||||
_send(stdout, {"op": "ready", "engine": "faster-whisper-isolated"})
|
||||
while True:
|
||||
try:
|
||||
msg = _recv(stdin)
|
||||
except Exception as exc:
|
||||
_send(stdout, {"op": "error", "stage": "recv", "message": f"{type(exc).__name__}: {exc}"})
|
||||
return 1
|
||||
if msg is None:
|
||||
return 0
|
||||
op = msg.get("op")
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong"})
|
||||
elif op == "transcribe":
|
||||
result = _transcribe(msg.get("audio_path"), bool(msg.get("word_timestamps", True)))
|
||||
_send(stdout, {"op": "segments", "result": result})
|
||||
elif op == "shutdown":
|
||||
return 0
|
||||
else:
|
||||
_send(stdout, {"op": "error", "stage": "dispatch", "message": f"unknown op: {op!r}"})
|
||||
except Exception as exc:
|
||||
_send(stdout, {
|
||||
"op": "error", "stage": "handler",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -101,12 +101,6 @@ def main() -> int:
|
||||
return 0
|
||||
|
||||
op = msg.get("op")
|
||||
# Wave 4.2: deterministic "crash mid-transcription" hook — exit BEFORE
|
||||
# sending any reply so the parent's blocking recv sees a dead pipe
|
||||
# (reply=None). The crash-after-one hook below replies first, so it
|
||||
# can't deterministically exercise the no-reply path.
|
||||
if op == "transcribe" and os.environ.get("OMNIVOICE_ECHO_CRASH_NO_REPLY") == "1":
|
||||
os._exit(1)
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong"})
|
||||
@@ -119,19 +113,6 @@ def main() -> int:
|
||||
"sample_rate": sr,
|
||||
"n_samples": n_samples,
|
||||
})
|
||||
elif op == "transcribe":
|
||||
# Wave 4.2: echo ASR op — a canned segments result so the
|
||||
# SubprocessASRBackend round-trip + respawn path is testable
|
||||
# without a real ASR engine.
|
||||
_send(stdout, {
|
||||
"op": "segments",
|
||||
"result": {
|
||||
"segments": [{"start": 0.0, "end": 1.0,
|
||||
"text": f"echo:{msg.get('audio_path', '')}"}],
|
||||
"text": f"echo:{msg.get('audio_path', '')}",
|
||||
"language": "en",
|
||||
},
|
||||
})
|
||||
elif op == "shutdown":
|
||||
return 0
|
||||
elif op == "probe_env" and test_mode:
|
||||
|
||||
@@ -1,187 +0,0 @@
|
||||
"""dots.tts sidecar package (issue #498).
|
||||
|
||||
dots.tts is rednote-hilab's 2B fully-continuous autoregressive TTS — widely
|
||||
cited as among the strongest open zero-shot voice-cloning models. 24
|
||||
languages, 48 kHz output, Apache-2.0 (code + checkpoints).
|
||||
|
||||
It runs in its own subprocess **and its own venv**, isolated from the
|
||||
OmniVoice parent, for the same ``transformers`` reason as IndexTTS and
|
||||
MOSS-TTS-v1.5: dots.tts pins ``transformers==4.57.0`` (verified against
|
||||
``constraints/recommended.txt``), while OmniVoice pins
|
||||
``transformers>=5.3.0``. The two cannot share one interpreter.
|
||||
|
||||
Cross-platform honesty (the strict default-parity rule): dots.tts's
|
||||
upstream package declares **Linux + macOS** classifiers only — **no
|
||||
Windows** — and its device code is **CUDA-or-CPU with no MPS branch**
|
||||
(verified in ``runtime.py``). So:
|
||||
|
||||
* It is **opt-in** (engine-picker selection + a user-provided clone),
|
||||
never a default — so it never becomes a broken default on any platform.
|
||||
* ``is_available()`` returns ``False`` with a clear reason on **Windows**
|
||||
rather than offering an engine that can't run there. Windows users are
|
||||
pointed at WSL2 / a Linux or macOS host.
|
||||
* ``gpu_compat = ("cuda", "cpu")`` — no MPS claim. On Apple Silicon the
|
||||
upstream package runs on CPU (slow but correct); the faster MLX path is
|
||||
a community port we deliberately don't auto-wire here.
|
||||
|
||||
Three public entry points: ``DotsTTSBackend`` (this module), ``main.py``
|
||||
(sidecar, runs under dots.tts's ``transformers==4.57`` venv — never imported
|
||||
by the parent), and ``bootstrap.py`` (venv probe + lazy bootstrap).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sys
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from services.subprocess_backend import SubprocessBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import torch # noqa: F401
|
||||
|
||||
logger = logging.getLogger("omnivoice.dots_tts")
|
||||
|
||||
|
||||
class DotsTTSBackend(SubprocessBackend):
|
||||
"""dots.tts (rednote-hilab) — 2B, 24 langs, zero-shot clone, CUDA/CPU.
|
||||
|
||||
Runs in a long-lived sidecar over length-prefixed JSON-over-stdio in a
|
||||
dedicated venv (``transformers==4.57.0``). First synthesize cold-loads
|
||||
the ~9 GB checkpoint (bf16 on CUDA); subsequent calls reuse the process.
|
||||
|
||||
Installation (OmniVoice prefers a user's existing ``${DIR}/.venv``)::
|
||||
|
||||
git clone https://github.com/rednote-hilab/dots.tts.git
|
||||
cd dots.tts
|
||||
uv venv && uv pip install -e . -c constraints/recommended.txt
|
||||
|
||||
Set ``OMNIVOICE_DOTS_TTS_DIR`` to the clone root. OmniVoice creates
|
||||
``backend/engines/dots_tts/.venv`` lazily on first launch if no venv
|
||||
exists yet; the user's existing ``${DIR}/.venv`` is preferred if present.
|
||||
|
||||
Best cloning quality uses the ``dots.tts-soar`` checkpoint (the default)
|
||||
and BOTH a reference clip and its exact transcript (continuation
|
||||
cloning). License: Apache-2.0.
|
||||
"""
|
||||
|
||||
id = "dots-tts"
|
||||
display_name = (
|
||||
"dots.tts (2B, 24 langs, zero-shot clone, CUDA/CPU, 48 kHz, Apache-2.0)"
|
||||
)
|
||||
supports_voice_design = False # requires ref audio for timbre cloning
|
||||
# dots.tts emits 48 kHz (verified via checkpoint vocoder.sample_rate).
|
||||
_DEFAULT_SAMPLE_RATE = 48000
|
||||
# CUDA + CPU only; no MPS branch in upstream runtime.py.
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
# ── availability ───────────────────────────────────────────────────────
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
# Cross-platform parity: dots.tts upstream is Linux/macOS-only (no
|
||||
# Windows classifier, no Windows install path). Refuse cleanly on
|
||||
# Windows instead of advertising an engine that can't run.
|
||||
if sys.platform == "win32":
|
||||
return False, (
|
||||
"dots.tts is not supported on Windows — upstream targets "
|
||||
"Linux and macOS only. Run OmniVoice under WSL2, or use a "
|
||||
"Linux/macOS host. See docs/engines/dots-tts.md."
|
||||
)
|
||||
|
||||
# Do NOT import dots_tts here: it pins transformers==4.57, which can't
|
||||
# coexist with the parent's transformers>=5.3 in one interpreter —
|
||||
# the reason for the subprocess isolation. Verify the venv on disk
|
||||
# only; a real health-check is gated on the user's "Test engine"
|
||||
# action in Settings.
|
||||
from engines.dots_tts.bootstrap import (
|
||||
DOTS_TTS_SIDECAR_SCRIPT,
|
||||
is_dots_tts_installed,
|
||||
)
|
||||
if not is_dots_tts_installed():
|
||||
return False, (
|
||||
"dots.tts venv not found. Set OMNIVOICE_DOTS_TTS_DIR to your "
|
||||
"dots.tts clone (the directory containing pyproject.toml) and "
|
||||
"restart OmniVoice. CUDA or CPU only (no MPS). See "
|
||||
"docs/engines/dots-tts.md for the full install walk-through."
|
||||
)
|
||||
if not DOTS_TTS_SIDECAR_SCRIPT.exists():
|
||||
return False, (
|
||||
"dots.tts sidecar script missing at "
|
||||
f"{DOTS_TTS_SIDECAR_SCRIPT} — reinstall OmniVoice."
|
||||
)
|
||||
return True, "ok (CUDA when present, else CPU)"
|
||||
|
||||
@classmethod
|
||||
def venv_python(cls):
|
||||
from engines.dots_tts.bootstrap import resolve_dots_tts_venv
|
||||
return resolve_dots_tts_venv()
|
||||
|
||||
@classmethod
|
||||
def sidecar_script(cls):
|
||||
from engines.dots_tts.bootstrap import DOTS_TTS_SIDECAR_SCRIPT
|
||||
return DOTS_TTS_SIDECAR_SCRIPT
|
||||
|
||||
# ── TTSBackend protocol ────────────────────────────────────────────────
|
||||
|
||||
@property
|
||||
def sample_rate(self) -> int:
|
||||
return self._DEFAULT_SAMPLE_RATE
|
||||
|
||||
@property
|
||||
def supported_languages(self) -> list[str]:
|
||||
# 24 languages with auto-detect; expose "multi" on the protocol
|
||||
# surface and translate the caller's language at synthesize time.
|
||||
return ["multi"]
|
||||
|
||||
# ── generate (parent-side arbitration) ─────────────────────────────────
|
||||
|
||||
def generate(self, text: str, **kw) -> "torch.Tensor":
|
||||
"""Synthesize one utterance through the dots.tts sidecar.
|
||||
|
||||
kwargs honored:
|
||||
* ``ref_audio`` — reference clip path → ``prompt_audio_path``
|
||||
(zero-shot cloning). Optional.
|
||||
* ``ref_text`` — the reference transcript → ``prompt_text``.
|
||||
Best cloning fidelity ("continuation"). Upstream
|
||||
REQUIRES ``prompt_audio_path`` when ``prompt_text``
|
||||
is set, so we drop a stray ref_text with no
|
||||
ref_audio rather than let the sidecar raise.
|
||||
* ``language`` — ISO code / name / None (auto-detect).
|
||||
* ``num_step`` — flow-matching steps → ``num_steps`` (default 10;
|
||||
use 4 for the ``dots.tts-mf`` checkpoint).
|
||||
* ``guidance_scale`` — CFG (default 1.2; >2 amplifies energy).
|
||||
|
||||
Returns a tensor of shape (1, n_samples) at :attr:`sample_rate`.
|
||||
"""
|
||||
forwarded: dict = {}
|
||||
|
||||
ref_audio = kw.get("ref_audio")
|
||||
if ref_audio:
|
||||
forwarded["ref_audio"] = ref_audio
|
||||
ref_text = kw.get("ref_text")
|
||||
if ref_text:
|
||||
# continuation cloning — only valid alongside ref_audio.
|
||||
forwarded["ref_text"] = ref_text
|
||||
elif kw.get("ref_text"):
|
||||
logger.info(
|
||||
"dots-tts: ref_text supplied without ref_audio; ignoring "
|
||||
"(upstream requires prompt_audio_path when prompt_text is set)."
|
||||
)
|
||||
|
||||
language = kw.get("language")
|
||||
if language:
|
||||
forwarded["language"] = str(language)
|
||||
|
||||
# OmniVoice's generic num_step default is 16; dots.tts's own default
|
||||
# is 10. Honor an explicit value, else use the dots-appropriate 10.
|
||||
num_step = kw.get("num_step")
|
||||
forwarded["num_steps"] = int(num_step) if num_step is not None else 10
|
||||
|
||||
# dots.tts's own CFG default is 1.2 (the generic 2.0 over-energises).
|
||||
guidance = kw.get("guidance_scale")
|
||||
forwarded["guidance_scale"] = float(guidance) if guidance is not None else 1.2
|
||||
|
||||
return super().generate(text, **forwarded)
|
||||
|
||||
|
||||
__all__ = ["DotsTTSBackend"]
|
||||
@@ -1,226 +0,0 @@
|
||||
"""dots.tts venv probe + lazy bootstrap (issue #498).
|
||||
|
||||
Resolves which Python interpreter runs the dots.tts sidecar. Mirrors
|
||||
``engines.indextts.bootstrap`` / ``engines.moss_tts_v15.bootstrap`` because
|
||||
dots.tts has the same shape of problem: a hard ``transformers==4.57.0`` pin
|
||||
that conflicts with the parent's ``transformers>=5.3`` — so it runs in its
|
||||
own venv.
|
||||
|
||||
Probe order (priority — existing power-user installs win, zero migration):
|
||||
|
||||
1. ``${OMNIVOICE_DOTS_TTS_DIR}/.venv/`` — the user's clone-level venv.
|
||||
2. ``backend/engines/dots_tts/.venv/`` — this package's own venv.
|
||||
3. Bootstrap: ``uv venv`` then ``uv pip install -e <clone> -c
|
||||
<clone>/constraints/recommended.txt`` (the upstream-pinned stack:
|
||||
torch==2.8.0, transformers==4.57.0, …).
|
||||
|
||||
Caching: memoised after first success. Tests reset via :func:`invalidate`.
|
||||
|
||||
Security: same posture as IndexTTS — bootstrap never touches HF_TOKEN; the
|
||||
sidecar's stderr is redacted by the parent's ``HFTokenRedactor``; the
|
||||
editable install comes from a user-controlled clone they already trust.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger("omnivoice.dots_tts.bootstrap")
|
||||
|
||||
#: Absolute path to the sidecar entrypoint.
|
||||
DOTS_TTS_SIDECAR_SCRIPT: Path = Path(__file__).parent / "main.py"
|
||||
|
||||
#: This package's owned venv (Probe 2).
|
||||
_ENGINES_VENV_DIR: Path = Path(__file__).parent / ".venv"
|
||||
|
||||
#: Env var pointing at the user's dots.tts clone root.
|
||||
_CLONE_DIR_ENV: str = "OMNIVOICE_DOTS_TTS_DIR"
|
||||
|
||||
#: Per-process resolution cache. Cleared by :func:`invalidate` for tests.
|
||||
_resolved_python: Optional[Path] = None
|
||||
|
||||
_IMPORT_PROBE_TIMEOUT_S = 15
|
||||
_UV_VENV_TIMEOUT_S = 120
|
||||
_UV_PIP_INSTALL_TIMEOUT_S = 1800
|
||||
|
||||
|
||||
# ── public API ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def invalidate() -> None:
|
||||
"""Clear the resolved-python cache. Tests call this between scenarios."""
|
||||
global _resolved_python
|
||||
_resolved_python = None
|
||||
|
||||
|
||||
def is_dots_tts_installed() -> bool:
|
||||
"""Cheap file-existence check for a usable dots.tts venv. Does NOT spawn
|
||||
the venv Python — that's saved for :func:`resolve_dots_tts_venv`."""
|
||||
for cand in _probe_paths():
|
||||
if cand.is_file():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def resolve_dots_tts_venv() -> Path:
|
||||
"""Resolve the sidecar's Python interpreter (probe order in the module
|
||||
docstring). Memoised. Raises :exc:`RuntimeError` if none can be located
|
||||
and the bootstrap path is unavailable."""
|
||||
global _resolved_python
|
||||
if _resolved_python is not None:
|
||||
return _resolved_python
|
||||
|
||||
clone_dir = os.environ.get(_CLONE_DIR_ENV)
|
||||
|
||||
# Probe 1 — user's clone-level venv.
|
||||
if clone_dir:
|
||||
cand = _venv_python_path(Path(clone_dir) / ".venv")
|
||||
if cand.is_file() and _venv_can_import_dots(cand):
|
||||
logger.info(
|
||||
"dots.tts venv resolved from %s: %s", _CLONE_DIR_ENV, cand,
|
||||
)
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
# Probe 2 — this package's own venv.
|
||||
cand = _venv_python_path(_ENGINES_VENV_DIR)
|
||||
if cand.is_file() and _venv_can_import_dots(cand):
|
||||
logger.info("dots.tts venv resolved from engines path: %s", cand)
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
# Probe 3 — bootstrap.
|
||||
if not clone_dir:
|
||||
raise RuntimeError(
|
||||
"dots.tts is not installed. Set the "
|
||||
f"{_CLONE_DIR_ENV} environment variable to your dots.tts clone "
|
||||
"(the directory that contains pyproject.toml and constraints/), "
|
||||
"then restart OmniVoice. See docs/engines/dots-tts.md for the "
|
||||
"full install walk-through."
|
||||
)
|
||||
|
||||
cand = _bootstrap_engines_venv(Path(clone_dir))
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
|
||||
# ── internals ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _venv_python_path(venv_dir: Path) -> Path:
|
||||
if sys.platform == "win32":
|
||||
return venv_dir / "Scripts" / "python.exe"
|
||||
return venv_dir / "bin" / "python"
|
||||
|
||||
|
||||
def _probe_paths() -> list[Path]:
|
||||
out: list[Path] = []
|
||||
clone_dir = os.environ.get(_CLONE_DIR_ENV)
|
||||
if clone_dir:
|
||||
out.append(_venv_python_path(Path(clone_dir) / ".venv"))
|
||||
out.append(_venv_python_path(_ENGINES_VENV_DIR))
|
||||
return out
|
||||
|
||||
|
||||
def _venv_can_import_dots(python_path: Path) -> bool:
|
||||
"""Spawn the candidate python and verify ``import dots_tts.runtime`` works.
|
||||
Bounded by ``_IMPORT_PROBE_TIMEOUT_S``. False on any failure."""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[str(python_path), "-c", "import dots_tts.runtime"],
|
||||
capture_output=True,
|
||||
timeout=_IMPORT_PROBE_TIMEOUT_S,
|
||||
)
|
||||
except (subprocess.TimeoutExpired, OSError) as exc:
|
||||
logger.debug("dots.tts import probe failed for %s: %s", python_path, exc)
|
||||
return False
|
||||
if proc.returncode != 0:
|
||||
logger.debug(
|
||||
"dots.tts import probe non-zero for %s: %s",
|
||||
python_path,
|
||||
proc.stderr.decode("utf-8", errors="replace")[:200],
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _locate_uv() -> Optional[str]:
|
||||
bundled = os.environ.get("OMNIVOICE_BUNDLED_UV")
|
||||
if bundled and Path(bundled).is_file():
|
||||
return bundled
|
||||
return shutil.which("uv")
|
||||
|
||||
|
||||
def _bootstrap_engines_venv(clone_dir: Path) -> Path:
|
||||
"""Create engines/dots_tts/.venv and editable-install the user's clone
|
||||
with the upstream constraints file."""
|
||||
uv = _locate_uv()
|
||||
if not uv:
|
||||
raise RuntimeError(
|
||||
"uv is required to bootstrap the dots.tts venv but was not found "
|
||||
"on PATH (and OMNIVOICE_BUNDLED_UV was not set). Install uv from "
|
||||
"https://docs.astral.sh/uv/ and re-launch OmniVoice, or set "
|
||||
"OMNIVOICE_BUNDLED_UV to the absolute path of a uv binary."
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Bootstrapping dots.tts venv at %s from %s (this can take several "
|
||||
"minutes on first launch)", _ENGINES_VENV_DIR, clone_dir,
|
||||
)
|
||||
|
||||
try:
|
||||
subprocess.run(
|
||||
[uv, "venv", str(_ENGINES_VENV_DIR)],
|
||||
check=True, timeout=_UV_VENV_TIMEOUT_S, capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
f"uv venv failed for dots.tts bootstrap at {_ENGINES_VENV_DIR}: "
|
||||
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
|
||||
) from exc
|
||||
|
||||
python_path = _venv_python_path(_ENGINES_VENV_DIR)
|
||||
install_cmd = [
|
||||
uv, "pip", "install",
|
||||
"--python", str(python_path),
|
||||
"-e", str(clone_dir),
|
||||
]
|
||||
# Apply the upstream pin set when it ships with the clone.
|
||||
constraints = clone_dir / "constraints" / "recommended.txt"
|
||||
if constraints.is_file():
|
||||
install_cmd += ["-c", str(constraints)]
|
||||
try:
|
||||
subprocess.run(
|
||||
install_cmd, check=True,
|
||||
timeout=_UV_PIP_INSTALL_TIMEOUT_S, capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
"uv pip install -e failed during dots.tts bootstrap "
|
||||
f"({clone_dir}): "
|
||||
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}. "
|
||||
"See docs/engines/dots-tts.md."
|
||||
) from exc
|
||||
|
||||
if not _venv_can_import_dots(python_path):
|
||||
raise RuntimeError(
|
||||
"dots.tts bootstrap completed but `import dots_tts.runtime` still "
|
||||
f"fails from {python_path}. Verify that {clone_dir} is a valid "
|
||||
"dots.tts clone. See docs/engines/dots-tts.md."
|
||||
)
|
||||
|
||||
logger.info("dots.tts venv bootstrap successful: %s", python_path)
|
||||
return python_path
|
||||
|
||||
|
||||
__all__ = [
|
||||
"DOTS_TTS_SIDECAR_SCRIPT",
|
||||
"invalidate",
|
||||
"is_dots_tts_installed",
|
||||
"resolve_dots_tts_venv",
|
||||
]
|
||||
@@ -1,255 +0,0 @@
|
||||
"""dots.tts sidecar entry point (issue #498).
|
||||
|
||||
Runs inside ``engines/dots_tts/.venv`` (or the user's existing
|
||||
``${OMNIVOICE_DOTS_TTS_DIR}/.venv``) with ``transformers==4.57.0``, isolated
|
||||
from the OmniVoice parent (``transformers>=5.3``). Same isolation rationale
|
||||
as the IndexTTS / MOSS-TTS-v1.5 sidecars.
|
||||
|
||||
Stdlib-only at import time; ``dots_tts`` + torch are imported lazily on the
|
||||
first synthesize op so the ``ready`` frame fits inside the parent's 30 s
|
||||
spawn handshake even on a cold filesystem.
|
||||
|
||||
Wire protocol — length-prefixed JSON over stdin/stdout, byte-identical to
|
||||
``backend/services/subprocess_backend.py``::
|
||||
|
||||
[ 4-byte big-endian uint32 length ][ N bytes UTF-8 JSON ]
|
||||
|
||||
Op flow:
|
||||
1. Sidecar -> parent: {"op": "ready", "engine": "dots-tts",
|
||||
"sample_rate": 48000}
|
||||
2. parent -> sidecar: {"op": "ping"} -> {"op": "pong", "vram_mb": N}
|
||||
3. parent -> sidecar: {"op": "synthesize", "text": "...",
|
||||
"ref_audio": "/path/ref.wav",
|
||||
"ref_text": "transcript", "language": "EN",
|
||||
"num_steps": 10, "guidance_scale": 1.2}
|
||||
-> {"op": "progress", ...} (cold load) then
|
||||
-> {"op": "audio", "audio_pcm_b64": "...", "sample_rate": 48000,
|
||||
"n_samples": N}
|
||||
4. parent -> sidecar: {"op": "shutdown"} -> exit 0
|
||||
|
||||
Restrictions: NO imports from OmniVoice parent code (different venv). NO
|
||||
logging of ``os.environ`` contents. Single-frame DoS cap matches the
|
||||
parent's ``MAX_FRAME_BYTES``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
|
||||
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
|
||||
MAX_FRAME_BYTES = 64 * 1024 * 1024
|
||||
|
||||
#: dots.tts emits 48 kHz (checkpoint vocoder.sample_rate). Advertised in the
|
||||
#: ready frame; the real value is re-read from each generate() result.
|
||||
DOTS_SAMPLE_RATE = 48000
|
||||
|
||||
#: Default checkpoint. ``-soar`` is the best-cloning variant; ``-mf`` is the
|
||||
#: fastest (use num_steps=4). Overridable for air-gapped / mirror installs.
|
||||
_DEFAULT_REPO = "rednote-hilab/dots.tts-soar"
|
||||
|
||||
|
||||
# ── wire protocol ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _send(stream, obj: dict) -> None:
|
||||
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
|
||||
stream.write(struct.pack("!I", len(body)))
|
||||
stream.write(body)
|
||||
stream.flush()
|
||||
|
||||
|
||||
def _recv(stream):
|
||||
header = stream.read(4)
|
||||
if len(header) < 4:
|
||||
return None # EOF
|
||||
(n,) = struct.unpack("!I", header)
|
||||
if n > MAX_FRAME_BYTES:
|
||||
raise IOError(f"frame too large: {n}")
|
||||
body = bytearray()
|
||||
while len(body) < n:
|
||||
chunk = stream.read(n - len(body))
|
||||
if not chunk:
|
||||
raise IOError("short read")
|
||||
body.extend(chunk)
|
||||
return json.loads(bytes(body).decode("utf-8"))
|
||||
|
||||
|
||||
def _measure_vram_mb() -> float:
|
||||
"""This sidecar's own GPU memory in MB (MM2-08). 0 on CPU. Never raises."""
|
||||
try:
|
||||
import torch
|
||||
if torch.cuda.is_available():
|
||||
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
|
||||
except Exception:
|
||||
pass
|
||||
return 0.0
|
||||
|
||||
|
||||
# ── model loading (lazy, on first synthesize) ─────────────────────────────
|
||||
|
||||
|
||||
# Module-level singleton — (runtime,). Device is auto-selected inside the
|
||||
# dots.tts runtime (cuda-or-cpu, no MPS); we don't pass a device.
|
||||
_runtime = None
|
||||
|
||||
|
||||
def _load_runtime(stdout):
|
||||
"""Cold-construct the dots.tts runtime.
|
||||
|
||||
``DotsTtsRuntime.from_pretrained`` auto-selects cuda-or-cpu internally
|
||||
(no MPS path). precision is bf16 on CUDA; on CPU we fall back to fp32
|
||||
(bf16 CPU kernels are spotty). Both overridable via env.
|
||||
"""
|
||||
global _runtime
|
||||
if _runtime is not None:
|
||||
return _runtime
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 0})
|
||||
|
||||
import torch
|
||||
from dots_tts.runtime import DotsTtsRuntime # type: ignore[import-not-found]
|
||||
|
||||
repo = os.environ.get("OMNIVOICE_DOTS_TTS_MODEL", _DEFAULT_REPO)
|
||||
default_precision = "bfloat16" if torch.cuda.is_available() else "float32"
|
||||
precision = os.environ.get("OMNIVOICE_DOTS_TTS_PRECISION", default_precision)
|
||||
optimize = os.environ.get("OMNIVOICE_DOTS_TTS_OPTIMIZE", "0") == "1"
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 50})
|
||||
|
||||
_runtime = DotsTtsRuntime.from_pretrained(
|
||||
repo,
|
||||
precision=precision,
|
||||
optimize=optimize,
|
||||
)
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 100})
|
||||
return _runtime
|
||||
|
||||
|
||||
def _tensor_to_pcm_b64(audio, sample_rate: int) -> tuple[str, int, int]:
|
||||
"""Convert a torch waveform tensor (1, N) in [-1, 1] to base64 int16 PCM."""
|
||||
import numpy as np
|
||||
|
||||
arr = audio.detach().to("cpu").float().numpy()
|
||||
arr = np.asarray(arr, dtype=np.float32).squeeze()
|
||||
if arr.ndim > 1:
|
||||
arr = arr.mean(axis=0) # defensive downmix to mono
|
||||
arr = np.clip(arr, -1.0, 1.0)
|
||||
pcm = (arr * 32767.0).astype(np.int16).tobytes()
|
||||
return base64.b64encode(pcm).decode("ascii"), int(sample_rate), int(arr.shape[0])
|
||||
|
||||
|
||||
def _normalize_language(raw):
|
||||
"""Map OmniVoice's language value to what dots.tts accepts, or None.
|
||||
|
||||
dots.tts accepts None/"auto_detect", ISO codes upper-cased ("EN"/"ZH"),
|
||||
or names ("english"). A 2-letter ISO code is upper-cased; anything else
|
||||
is passed through; empty / "auto" → None (auto-detect)."""
|
||||
if not raw or not isinstance(raw, str):
|
||||
return None
|
||||
s = raw.strip()
|
||||
if not s or s.lower() == "auto":
|
||||
return None
|
||||
if len(s) == 2 and s.isalpha():
|
||||
return s.upper()
|
||||
return s
|
||||
|
||||
|
||||
def _handle_synthesize(msg: dict, stdout) -> None:
|
||||
"""Dispatch one synthesize request. Emits the audio frame or raises."""
|
||||
text = msg.get("text")
|
||||
if not text or not isinstance(text, str):
|
||||
raise ValueError("synthesize: missing or non-string 'text'")
|
||||
|
||||
runtime = _load_runtime(stdout)
|
||||
|
||||
gen_kwargs: dict = {
|
||||
"text": text,
|
||||
"num_steps": int(msg.get("num_steps", 10)),
|
||||
"guidance_scale": float(msg.get("guidance_scale", 1.2)),
|
||||
}
|
||||
|
||||
ref_audio = msg.get("ref_audio")
|
||||
if ref_audio:
|
||||
gen_kwargs["prompt_audio_path"] = ref_audio
|
||||
ref_text = msg.get("ref_text")
|
||||
if ref_text:
|
||||
# continuation cloning — upstream requires prompt_audio_path when
|
||||
# prompt_text is set (the parent already enforces this).
|
||||
gen_kwargs["prompt_text"] = ref_text
|
||||
|
||||
language = _normalize_language(msg.get("language"))
|
||||
if language:
|
||||
gen_kwargs["language"] = language
|
||||
|
||||
result = runtime.generate(**gen_kwargs)
|
||||
audio = result["audio"]
|
||||
sample_rate = int(result.get("sample_rate", DOTS_SAMPLE_RATE))
|
||||
|
||||
pcm_b64, sr, n_samples = _tensor_to_pcm_b64(audio, sample_rate)
|
||||
_send(stdout, {
|
||||
"op": "audio",
|
||||
"audio_pcm_b64": pcm_b64,
|
||||
"sample_rate": sr,
|
||||
"n_samples": n_samples,
|
||||
})
|
||||
|
||||
|
||||
# ── main loop ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def main() -> int:
|
||||
stdin = sys.stdin.buffer
|
||||
stdout = sys.stdout.buffer
|
||||
|
||||
# Ready handshake fires BEFORE any heavy import.
|
||||
_send(stdout, {
|
||||
"op": "ready",
|
||||
"engine": "dots-tts",
|
||||
"sample_rate": DOTS_SAMPLE_RATE,
|
||||
})
|
||||
|
||||
while True:
|
||||
try:
|
||||
msg = _recv(stdin)
|
||||
except Exception as exc:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": "recv",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
return 1
|
||||
if msg is None:
|
||||
return 0
|
||||
|
||||
op = msg.get("op") if isinstance(msg, dict) else None
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
|
||||
elif op == "synthesize":
|
||||
_handle_synthesize(msg, stdout)
|
||||
elif op == "shutdown":
|
||||
return 0
|
||||
else:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": "dispatch",
|
||||
"message": f"unknown op: {op!r}",
|
||||
})
|
||||
except Exception as exc:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": op or "unknown",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -79,11 +79,6 @@ class IndexTTS2Backend(SubprocessBackend):
|
||||
display_name = "IndexTTS2 (emotion control, duration control, zero-shot)"
|
||||
supports_voice_design = False # requires ref audio for timbre
|
||||
_DEFAULT_SAMPLE_RATE = 24000
|
||||
# Explicit so IndexTTS2 stops advertising the inherited CPU-only default:
|
||||
# the sidecar runs the IndexTTS PyTorch model on CUDA when present, else
|
||||
# CPU. ROCm left unclaimed (the sidecar's own venv would need a ROCm torch);
|
||||
# a ROCm host honestly resolves to cpu_fallback.
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
|
||||
@@ -74,24 +74,6 @@ import traceback
|
||||
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
|
||||
MAX_FRAME_BYTES = 64 * 1024 * 1024
|
||||
|
||||
|
||||
def _measure_vram_mb() -> float:
|
||||
"""This sidecar's own GPU memory in MB, for the loaded-models panel
|
||||
(MM2-08). The parent can't see a child's VRAM, so we self-report it in the
|
||||
pong. Degrades to 0 on CPU / when torch isn't loaded yet — never raises."""
|
||||
try:
|
||||
import torch # already a dep inside the indextts venv
|
||||
if torch.cuda.is_available():
|
||||
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
|
||||
mps = getattr(torch.backends, "mps", None)
|
||||
if mps is not None and mps.is_available():
|
||||
drv = getattr(torch.mps, "driver_allocated_memory", None)
|
||||
if drv:
|
||||
return round(drv() / (1024 ** 2), 1)
|
||||
except Exception:
|
||||
pass
|
||||
return 0.0
|
||||
|
||||
# Sample rate IndexTTS-2 emits natively. Advertised in the ready frame so
|
||||
# the parent doesn't have to import IndexTTS just to learn the rate.
|
||||
INDEXTTS_SAMPLE_RATE = 24000
|
||||
@@ -300,7 +282,7 @@ def main() -> int:
|
||||
op = msg.get("op") if isinstance(msg, dict) else None
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
|
||||
_send(stdout, {"op": "pong"})
|
||||
elif op == "synthesize":
|
||||
_handle_synthesize(msg, stdout)
|
||||
elif op == "shutdown":
|
||||
|
||||
@@ -1,194 +0,0 @@
|
||||
"""MOSS-TTS-v1.5 sidecar package (issue #498).
|
||||
|
||||
MOSS-TTS-v1.5 is OpenMOSS's 8B flagship TTS — a Qwen3-8B language backbone
|
||||
plus a 1.6B audio codec, 31 languages, zero-shot voice cloning, token-level
|
||||
duration control and inline ``[pause Ns]`` markers. Apache-2.0.
|
||||
|
||||
It runs in its own subprocess **and its own venv**, isolated from the
|
||||
OmniVoice parent process, for the *same* reason IndexTTS does: a hard
|
||||
``transformers`` version conflict. MOSS-TTS-v1.5's ``torch-runtime`` extra
|
||||
pins ``transformers==5.0.0`` (verified against the upstream
|
||||
``pyproject.toml``), while OmniVoice pins ``transformers>=5.3.0``. The two
|
||||
cannot share one interpreter — so MOSS lives behind ``SubprocessBackend``
|
||||
with a dedicated venv, exactly like ``engines.indextts``.
|
||||
|
||||
Three public entry points live in this package:
|
||||
|
||||
* ``MossTTSV15Backend`` (this module) — the SubprocessBackend subclass
|
||||
that ``services.tts_backend._LAZY_REGISTRY`` resolves on first access.
|
||||
Defined HERE (not in ``services.tts_backend``) to break the import
|
||||
cycle: ``services.subprocess_backend`` imports ``TTSBackend`` from
|
||||
``services.tts_backend``, so the backend class must live downstream of
|
||||
that module finishing its import. Same indirection as IndexTTS /
|
||||
Supertonic-3.
|
||||
* ``main.py`` — the sidecar entrypoint (runs under MOSS's venv with
|
||||
``transformers==5.0.0``; never imported by the parent).
|
||||
* ``bootstrap.py`` — the venv-probe + lazy-bootstrap helper.
|
||||
|
||||
Do NOT import ``main.py`` from the parent process — it runs under a
|
||||
different venv (``transformers==5.0.0``) and importing it in-process would
|
||||
re-introduce the exact conflict this isolation exists to avoid.
|
||||
|
||||
Hardware honesty (cross-platform rule): MOSS-TTS-v1.5's upstream documents
|
||||
only CUDA and CPU. There is **no documented or tested MPS path** — the
|
||||
custom ``trust_remote_code`` modelling code and the separate audio
|
||||
tokenizer are unverified on Apple Silicon. We therefore advertise
|
||||
``gpu_compat = ("cuda", "cpu")`` and the sidecar selects ``cuda`` when
|
||||
present else ``cpu`` — it never silently routes to MPS where it might
|
||||
crash. On Apple Silicon the engine honestly resolves to CPU (slow but
|
||||
correct), and the engine is opt-in regardless, so it never becomes a
|
||||
broken default on any platform.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from services.subprocess_backend import SubprocessBackend
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import torch # noqa: F401
|
||||
|
||||
logger = logging.getLogger("omnivoice.moss_tts_v15")
|
||||
|
||||
#: 1 second of audio ≈ 12.5 codec tokens (MOSS-TTS-v1.5 model card). Used to
|
||||
#: translate OmniVoice's ``duration`` (seconds) into the model's ``tokens``
|
||||
#: duration-control argument.
|
||||
TOKENS_PER_SECOND: float = 12.5
|
||||
|
||||
|
||||
class MossTTSV15Backend(SubprocessBackend):
|
||||
"""MOSS-TTS-v1.5 (OpenMOSS) — 8B, 31 langs, zero-shot clone, CUDA/CPU.
|
||||
|
||||
Runs in a long-lived sidecar over length-prefixed JSON-over-stdio in a
|
||||
dedicated venv (``transformers==5.0.0``). The first synthesize cold-loads
|
||||
~16 GB of bf16 weights (CUDA) / fp32 (CPU); subsequent calls reuse the
|
||||
process and the in-memory model.
|
||||
|
||||
Installation (transparent to power users who already cloned MOSS-TTS —
|
||||
OmniVoice prefers their existing ``${DIR}/.venv``)::
|
||||
|
||||
git clone https://github.com/OpenMOSS/MOSS-TTS.git
|
||||
cd MOSS-TTS
|
||||
# CUDA host:
|
||||
uv venv && uv pip install -e ".[torch-runtime]"
|
||||
# non-CUDA host (CPU): install plain torch/transformers instead of +cu128
|
||||
|
||||
Set ``OMNIVOICE_MOSS_TTS_V15_DIR`` to the clone root. OmniVoice creates
|
||||
``backend/engines/moss_tts_v15/.venv`` lazily on first launch if no venv
|
||||
exists yet (CUDA hosts only — the upstream ``torch-runtime`` extra is
|
||||
``+cu128``); the user's existing ``${DIR}/.venv`` is preferred if
|
||||
present, so no re-install is needed.
|
||||
|
||||
License: Apache-2.0 (code + weights) — no acceptance gate needed.
|
||||
"""
|
||||
|
||||
id = "moss-tts-v15"
|
||||
display_name = (
|
||||
"MOSS-TTS-v1.5 (8B, 31 langs, zero-shot clone, CUDA/CPU, Apache-2.0)"
|
||||
)
|
||||
supports_voice_design = False # requires ref audio for timbre cloning
|
||||
_DEFAULT_SAMPLE_RATE = 24000
|
||||
# Honest hardware surface: upstream documents CUDA + CPU only. MPS is
|
||||
# undocumented / untested, so we do NOT claim it (cross-platform rule).
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
# ── availability ───────────────────────────────────────────────────────
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
# IMPORTANT: do NOT attempt to import MOSS / its transformers==5.0.0
|
||||
# here. The parent pins transformers>=5.3 — co-importing the two in
|
||||
# one interpreter is exactly the conflict this subprocess isolation
|
||||
# exists to avoid. We only verify the venv exists on disk; a real
|
||||
# health-check (spawn + ping) is gated on the user's "Test engine"
|
||||
# action in Settings, same as IndexTTS.
|
||||
from engines.moss_tts_v15.bootstrap import (
|
||||
MOSS_TTS_V15_SIDECAR_SCRIPT,
|
||||
is_moss_tts_v15_installed,
|
||||
)
|
||||
if not is_moss_tts_v15_installed():
|
||||
return False, (
|
||||
"MOSS-TTS-v1.5 venv not found. Set OMNIVOICE_MOSS_TTS_V15_DIR "
|
||||
"to your MOSS-TTS clone (the directory containing pyproject.toml) "
|
||||
"and restart OmniVoice. CUDA or CPU only (no MPS). See "
|
||||
"docs/engines/moss-tts-v15.md for the full install walk-through."
|
||||
)
|
||||
if not MOSS_TTS_V15_SIDECAR_SCRIPT.exists():
|
||||
return False, (
|
||||
"MOSS-TTS-v1.5 sidecar script missing at "
|
||||
f"{MOSS_TTS_V15_SIDECAR_SCRIPT} — reinstall OmniVoice."
|
||||
)
|
||||
return True, "ok (CUDA when present, else CPU)"
|
||||
|
||||
@classmethod
|
||||
def venv_python(cls):
|
||||
from engines.moss_tts_v15.bootstrap import resolve_moss_tts_v15_venv
|
||||
return resolve_moss_tts_v15_venv()
|
||||
|
||||
@classmethod
|
||||
def sidecar_script(cls):
|
||||
from engines.moss_tts_v15.bootstrap import MOSS_TTS_V15_SIDECAR_SCRIPT
|
||||
return MOSS_TTS_V15_SIDECAR_SCRIPT
|
||||
|
||||
# ── TTSBackend protocol ────────────────────────────────────────────────
|
||||
|
||||
@property
|
||||
def sample_rate(self) -> int:
|
||||
return self._DEFAULT_SAMPLE_RATE
|
||||
|
||||
@property
|
||||
def supported_languages(self) -> list[str]:
|
||||
# 31 languages with multilingual handling; expose "multi" on the
|
||||
# protocol surface (same as OmniVoice / CosyVoice / Supertonic-3) and
|
||||
# translate the caller's language at synthesize time.
|
||||
return ["multi"]
|
||||
|
||||
# ── generate (parent-side arbitration) ─────────────────────────────────
|
||||
|
||||
def generate(self, text: str, **kw) -> "torch.Tensor":
|
||||
"""Synthesize one utterance through the MOSS-TTS-v1.5 sidecar.
|
||||
|
||||
kwargs honored:
|
||||
* ``ref_audio`` — path to a reference clip. When present, MOSS
|
||||
runs zero-shot voice cloning (``reference=``).
|
||||
Optional: without it the model uses its own
|
||||
default voice.
|
||||
* ``ref_text`` — accepted but unused in clone mode (MOSS's
|
||||
zero-shot path needs only the audio); kept in
|
||||
the signature so the common call-site doesn't
|
||||
need engine-specific knowledge.
|
||||
* ``language`` — ISO code or name; mapped to a MOSS language name
|
||||
in the sidecar, omitted (auto-detect) if unknown.
|
||||
* ``duration`` — target seconds → ``tokens`` (1 s ≈ 12.5 tokens).
|
||||
* ``max_new_tokens`` — generation cap (default 4096).
|
||||
|
||||
Returns a tensor of shape (1, n_samples) at :attr:`sample_rate`.
|
||||
"""
|
||||
forwarded: dict = {}
|
||||
|
||||
ref_audio = kw.get("ref_audio")
|
||||
if ref_audio:
|
||||
forwarded["ref_audio"] = ref_audio
|
||||
ref_text = kw.get("ref_text")
|
||||
if ref_text:
|
||||
forwarded["ref_text"] = ref_text
|
||||
|
||||
language = kw.get("language")
|
||||
if language:
|
||||
forwarded["language"] = str(language)
|
||||
|
||||
duration = kw.get("duration")
|
||||
if duration is not None:
|
||||
target_tokens = int(float(duration) * TOKENS_PER_SECOND)
|
||||
if target_tokens > 0:
|
||||
forwarded["tokens"] = target_tokens
|
||||
|
||||
max_new_tokens = kw.get("max_new_tokens")
|
||||
if max_new_tokens is not None:
|
||||
forwarded["max_new_tokens"] = int(max_new_tokens)
|
||||
|
||||
return super().generate(text, **forwarded)
|
||||
|
||||
|
||||
__all__ = ["MossTTSV15Backend", "TOKENS_PER_SECOND"]
|
||||
@@ -1,272 +0,0 @@
|
||||
"""MOSS-TTS-v1.5 venv probe + lazy bootstrap (issue #498).
|
||||
|
||||
The parent process needs to know *which Python interpreter* to spawn the
|
||||
MOSS-TTS-v1.5 sidecar under. This module owns that resolution. It mirrors
|
||||
``engines.indextts.bootstrap`` because MOSS has the same shape of problem:
|
||||
a hard ``transformers`` pin (``==5.0.0``) that conflicts with the parent's
|
||||
``transformers>=5.3`` — so MOSS runs in its own venv.
|
||||
|
||||
Probe order (priority — existing power-user installs win, zero migration):
|
||||
|
||||
1. ``${OMNIVOICE_MOSS_TTS_V15_DIR}/.venv/`` — the user's clone-level
|
||||
venv. Highest priority: a user who already cloned MOSS-TTS and ran
|
||||
``uv pip install -e ".[torch-runtime]"`` (per upstream docs) gets
|
||||
reused verbatim, no re-download of the ~16 GB model.
|
||||
2. ``backend/engines/moss_tts_v15/.venv/`` — this package's own venv,
|
||||
created by step 3 if needed.
|
||||
3. Bootstrap: ``uv venv`` then ``uv pip install -e
|
||||
"${DIR}[torch-runtime]"``. Requires ``OMNIVOICE_MOSS_TTS_V15_DIR``.
|
||||
The upstream ``torch-runtime`` extra is CUDA (``+cu128``), so the
|
||||
auto-bootstrap targets CUDA hosts; non-CUDA (CPU/Mac) users set up
|
||||
their own venv per docs/engines/moss-tts-v15.md (Probe 1).
|
||||
|
||||
Caching: resolution is memoised after the first successful call. Tests
|
||||
reset via :func:`invalidate`.
|
||||
|
||||
Security: bootstrap never touches HF_TOKEN; the sidecar's stderr is drained
|
||||
by SubprocessBackend through the parent root logger where Phase 1's
|
||||
``HFTokenRedactor`` strips token bytes. ``uv pip install -e`` installs from
|
||||
a user-controlled clone the user already trusts (same posture as IndexTTS).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger("omnivoice.moss_tts_v15.bootstrap")
|
||||
|
||||
#: Absolute path to the sidecar entrypoint. ``MossTTSV15Backend.sidecar_script``
|
||||
#: returns this; SubprocessBackend spawns it with the resolved venv python.
|
||||
MOSS_TTS_V15_SIDECAR_SCRIPT: Path = Path(__file__).parent / "main.py"
|
||||
|
||||
#: This package's owned venv (Probe 2). The MOSS-TTS clone, when bootstrapped,
|
||||
#: is installed into this venv via ``uv pip install -e``.
|
||||
_ENGINES_VENV_DIR: Path = Path(__file__).parent / ".venv"
|
||||
|
||||
#: Env var pointing at the user's MOSS-TTS clone root.
|
||||
_CLONE_DIR_ENV: str = "OMNIVOICE_MOSS_TTS_V15_DIR"
|
||||
|
||||
#: Per-process resolution cache. Cleared by :func:`invalidate` for tests.
|
||||
_resolved_python: Optional[Path] = None
|
||||
|
||||
# Timeouts — bounded so a wedged venv never hangs the parent. The bootstrap
|
||||
# install can take many minutes on a cold cache (MOSS pulls a CUDA torch
|
||||
# build + transformers + an audio codec stack).
|
||||
_IMPORT_PROBE_TIMEOUT_S = 15
|
||||
_UV_VENV_TIMEOUT_S = 120
|
||||
_UV_PIP_INSTALL_TIMEOUT_S = 1800
|
||||
|
||||
|
||||
# ── public API ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def invalidate() -> None:
|
||||
"""Clear the resolved-python cache. Tests call this between scenarios."""
|
||||
global _resolved_python
|
||||
_resolved_python = None
|
||||
|
||||
|
||||
def is_moss_tts_v15_installed() -> bool:
|
||||
"""Cheap file-existence check for a usable MOSS-TTS-v1.5 venv.
|
||||
|
||||
Returns True if either Probe 1 or Probe 2 has a Python executable on
|
||||
disk. Does NOT spawn the venv Python — that's saved for
|
||||
:func:`resolve_moss_tts_v15_venv`, which is only invoked on the first
|
||||
generate() / health_check(). This fires on every Settings render via
|
||||
``MossTTSV15Backend.is_available()``, so it stays cheap.
|
||||
"""
|
||||
for cand in _probe_paths():
|
||||
if cand.is_file():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def resolve_moss_tts_v15_venv() -> Path:
|
||||
"""Resolve the path to the Python interpreter that runs the sidecar.
|
||||
|
||||
Probe order described in the module docstring. Memoised. Raises
|
||||
:exc:`RuntimeError` if no working venv can be located AND the bootstrap
|
||||
path is unavailable.
|
||||
"""
|
||||
global _resolved_python
|
||||
if _resolved_python is not None:
|
||||
return _resolved_python
|
||||
|
||||
clone_dir = os.environ.get(_CLONE_DIR_ENV)
|
||||
|
||||
# Probe 1 — user's clone-level venv (highest priority for back-compat).
|
||||
if clone_dir:
|
||||
cand = _venv_python_path(Path(clone_dir) / ".venv")
|
||||
if cand.is_file() and _venv_can_import_moss(cand):
|
||||
logger.info(
|
||||
"MOSS-TTS-v1.5 venv resolved from %s: %s", _CLONE_DIR_ENV, cand,
|
||||
)
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
# Probe 2 — this package's own venv.
|
||||
cand = _venv_python_path(_ENGINES_VENV_DIR)
|
||||
if cand.is_file() and _venv_can_import_moss(cand):
|
||||
logger.info("MOSS-TTS-v1.5 venv resolved from engines path: %s", cand)
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
# Probe 3 — bootstrap.
|
||||
if not clone_dir:
|
||||
raise RuntimeError(
|
||||
"MOSS-TTS-v1.5 is not installed. Set the "
|
||||
f"{_CLONE_DIR_ENV} environment variable to your MOSS-TTS clone "
|
||||
"(the directory that contains pyproject.toml), then restart "
|
||||
"OmniVoice. See docs/engines/moss-tts-v15.md for the full "
|
||||
"install walk-through."
|
||||
)
|
||||
|
||||
cand = _bootstrap_engines_venv(Path(clone_dir))
|
||||
_resolved_python = cand
|
||||
return cand
|
||||
|
||||
|
||||
# ── internals ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _venv_python_path(venv_dir: Path) -> Path:
|
||||
"""Return the python executable path inside a venv directory.
|
||||
|
||||
Handles the Unix (``bin/python``) vs Windows (``Scripts/python.exe``)
|
||||
layout. No filesystem access — caller checks .is_file().
|
||||
"""
|
||||
if sys.platform == "win32":
|
||||
return venv_dir / "Scripts" / "python.exe"
|
||||
return venv_dir / "bin" / "python"
|
||||
|
||||
|
||||
def _probe_paths() -> list[Path]:
|
||||
"""Ordered list of candidate venv-python paths (no .is_file() check)."""
|
||||
out: list[Path] = []
|
||||
clone_dir = os.environ.get(_CLONE_DIR_ENV)
|
||||
if clone_dir:
|
||||
out.append(_venv_python_path(Path(clone_dir) / ".venv"))
|
||||
out.append(_venv_python_path(_ENGINES_VENV_DIR))
|
||||
return out
|
||||
|
||||
|
||||
def _venv_can_import_moss(python_path: Path) -> bool:
|
||||
"""Spawn the candidate python and verify the MOSS stack imports.
|
||||
|
||||
MOSS-TTS-v1.5 loads via ``transformers`` + ``trust_remote_code`` (no
|
||||
fixed top-level package to import), so the readiness signal is that the
|
||||
venv has a working ``transformers`` + ``torch`` — which only the
|
||||
``[torch-runtime]`` install provides. Bounded by
|
||||
``_IMPORT_PROBE_TIMEOUT_S`` so a wedged venv never hangs the parent.
|
||||
Returns False on any failure (non-zero exit, timeout, OSError).
|
||||
"""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[str(python_path), "-c", "import transformers, torch"],
|
||||
capture_output=True,
|
||||
timeout=_IMPORT_PROBE_TIMEOUT_S,
|
||||
)
|
||||
except (subprocess.TimeoutExpired, OSError) as exc:
|
||||
logger.debug("moss-tts-v15 import probe failed for %s: %s", python_path, exc)
|
||||
return False
|
||||
if proc.returncode != 0:
|
||||
logger.debug(
|
||||
"moss-tts-v15 import probe non-zero for %s: %s",
|
||||
python_path,
|
||||
proc.stderr.decode("utf-8", errors="replace")[:200],
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _locate_uv() -> Optional[str]:
|
||||
"""Find the uv binary — bundled first (Tauri-set env var), else PATH."""
|
||||
bundled = os.environ.get("OMNIVOICE_BUNDLED_UV")
|
||||
if bundled and Path(bundled).is_file():
|
||||
return bundled
|
||||
sys_uv = shutil.which("uv")
|
||||
if sys_uv:
|
||||
return sys_uv
|
||||
return None
|
||||
|
||||
|
||||
def _bootstrap_engines_venv(clone_dir: Path) -> Path:
|
||||
"""Create engines/moss_tts_v15/.venv and install the user's clone into it.
|
||||
|
||||
Runs ``uv venv <engines_venv>`` then ``uv pip install --python
|
||||
<engines_venv>/bin/python -e "<clone>[torch-runtime]"``. Verifies the
|
||||
result by re-probing the import — a successful uv invocation that still
|
||||
can't import the stack indicates a deeper environment problem (e.g. the
|
||||
``+cu128`` torch-runtime extra can't resolve on a non-CUDA host) and we
|
||||
raise with whatever stderr we captured plus a docs pointer.
|
||||
"""
|
||||
uv = _locate_uv()
|
||||
if not uv:
|
||||
raise RuntimeError(
|
||||
"uv is required to bootstrap the MOSS-TTS-v1.5 venv but was not "
|
||||
"found on PATH (and OMNIVOICE_BUNDLED_UV was not set). Install uv "
|
||||
"from https://docs.astral.sh/uv/ and re-launch OmniVoice, or set "
|
||||
"OMNIVOICE_BUNDLED_UV to the absolute path of a uv binary."
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Bootstrapping MOSS-TTS-v1.5 venv at %s from %s (this can take "
|
||||
"several minutes on first launch)", _ENGINES_VENV_DIR, clone_dir,
|
||||
)
|
||||
|
||||
try:
|
||||
subprocess.run(
|
||||
[uv, "venv", str(_ENGINES_VENV_DIR)],
|
||||
check=True,
|
||||
timeout=_UV_VENV_TIMEOUT_S,
|
||||
capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
f"uv venv failed for MOSS-TTS-v1.5 bootstrap at {_ENGINES_VENV_DIR}: "
|
||||
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
|
||||
) from exc
|
||||
|
||||
python_path = _venv_python_path(_ENGINES_VENV_DIR)
|
||||
try:
|
||||
subprocess.run(
|
||||
[
|
||||
uv, "pip", "install",
|
||||
"--python", str(python_path),
|
||||
"-e", f"{clone_dir}[torch-runtime]",
|
||||
],
|
||||
check=True,
|
||||
timeout=_UV_PIP_INSTALL_TIMEOUT_S,
|
||||
capture_output=True,
|
||||
)
|
||||
except subprocess.CalledProcessError as exc:
|
||||
raise RuntimeError(
|
||||
"uv pip install -e failed during MOSS-TTS-v1.5 bootstrap "
|
||||
f"({clone_dir}). On a non-CUDA host the upstream '[torch-runtime]' "
|
||||
"extra (cu128) cannot resolve — set up the venv manually per "
|
||||
"docs/engines/moss-tts-v15.md. Error: "
|
||||
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
|
||||
) from exc
|
||||
|
||||
if not _venv_can_import_moss(python_path):
|
||||
raise RuntimeError(
|
||||
"MOSS-TTS-v1.5 bootstrap completed but the transformers/torch "
|
||||
f"import still fails from {python_path}. Verify that {clone_dir} "
|
||||
"is a valid MOSS-TTS clone. See docs/engines/moss-tts-v15.md."
|
||||
)
|
||||
|
||||
logger.info("MOSS-TTS-v1.5 venv bootstrap successful: %s", python_path)
|
||||
return python_path
|
||||
|
||||
|
||||
__all__ = [
|
||||
"MOSS_TTS_V15_SIDECAR_SCRIPT",
|
||||
"invalidate",
|
||||
"is_moss_tts_v15_installed",
|
||||
"resolve_moss_tts_v15_venv",
|
||||
]
|
||||
@@ -1,303 +0,0 @@
|
||||
"""MOSS-TTS-v1.5 sidecar entry point (issue #498).
|
||||
|
||||
Runs inside ``engines/moss_tts_v15/.venv`` (or the user's existing
|
||||
``${OMNIVOICE_MOSS_TTS_V15_DIR}/.venv``) with ``transformers==5.0.0``,
|
||||
isolated from the OmniVoice parent process which pins ``transformers>=5.3``.
|
||||
Same isolation rationale as the IndexTTS sidecar.
|
||||
|
||||
Stdlib-only at import time. The model + transformers + torch are imported
|
||||
lazily on the first synthesize op so the sidecar emits its ``ready`` frame
|
||||
inside the parent's 30 s spawn handshake even on a cold filesystem (an 8B
|
||||
model takes well over 30 s to cold-load).
|
||||
|
||||
Wire protocol — length-prefixed JSON over stdin/stdout, byte-identical to
|
||||
``backend/services/subprocess_backend.py``::
|
||||
|
||||
[ 4-byte big-endian uint32 length ][ N bytes UTF-8 JSON ]
|
||||
|
||||
Op flow:
|
||||
1. Sidecar -> parent: {"op": "ready", "engine": "moss-tts-v15",
|
||||
"sample_rate": 24000}
|
||||
2. parent -> sidecar: {"op": "ping"} -> {"op": "pong", "vram_mb": N}
|
||||
3. parent -> sidecar: {"op": "synthesize", "text": "...",
|
||||
"ref_audio": "/path/spk.wav", "language": "fr",
|
||||
"tokens": 325, "max_new_tokens": 4096}
|
||||
-> {"op": "progress", ...} (cold load only) then
|
||||
-> {"op": "audio", "audio_pcm_b64": "...", "sample_rate": 24000,
|
||||
"n_samples": N}
|
||||
4. parent -> sidecar: {"op": "shutdown"} -> exit 0
|
||||
|
||||
Restrictions: NO imports from OmniVoice parent code (different venv). NO
|
||||
logging of ``os.environ`` contents. Single-frame DoS cap matches the
|
||||
parent's ``MAX_FRAME_BYTES``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
|
||||
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
|
||||
MAX_FRAME_BYTES = 64 * 1024 * 1024
|
||||
|
||||
#: Native sample rate MOSS-TTS-v1.5 emits. Advertised in the ready frame so
|
||||
#: the parent doesn't have to import MOSS just to learn the rate. Confirmed
|
||||
#: via ``processor.model_config.sampling_rate`` (the real value is read from
|
||||
#: the loaded model at synthesize time; this is the handshake default).
|
||||
MOSS_SAMPLE_RATE = 24000
|
||||
|
||||
#: HF repo id for the weights, overridable for air-gapped / mirror installs.
|
||||
_DEFAULT_REPO = "OpenMOSS-Team/MOSS-TTS-v1.5"
|
||||
|
||||
#: ISO-639-1 → MOSS language name. MOSS's ``build_user_message`` takes a
|
||||
#: language *name* ("French"), not a code. Unknown codes are omitted so the
|
||||
#: model auto-detects. Covers the high-traffic subset of MOSS's 31 langs.
|
||||
_ISO_TO_NAME = {
|
||||
"en": "English", "zh": "Chinese", "ja": "Japanese", "ko": "Korean",
|
||||
"fr": "French", "de": "German", "es": "Spanish", "it": "Italian",
|
||||
"pt": "Portuguese", "ru": "Russian", "ar": "Arabic", "hi": "Hindi",
|
||||
"nl": "Dutch", "pl": "Polish", "tr": "Turkish", "vi": "Vietnamese",
|
||||
"th": "Thai", "id": "Indonesian", "cs": "Czech", "el": "Greek",
|
||||
"he": "Hebrew", "fa": "Persian", "uk": "Ukrainian", "sv": "Swedish",
|
||||
}
|
||||
|
||||
|
||||
# ── wire protocol ─────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _send(stream, obj: dict) -> None:
|
||||
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
|
||||
stream.write(struct.pack("!I", len(body)))
|
||||
stream.write(body)
|
||||
stream.flush()
|
||||
|
||||
|
||||
def _recv(stream):
|
||||
header = stream.read(4)
|
||||
if len(header) < 4:
|
||||
return None # EOF
|
||||
(n,) = struct.unpack("!I", header)
|
||||
if n > MAX_FRAME_BYTES:
|
||||
raise IOError(f"frame too large: {n}")
|
||||
body = bytearray()
|
||||
while len(body) < n:
|
||||
chunk = stream.read(n - len(body))
|
||||
if not chunk:
|
||||
raise IOError("short read")
|
||||
body.extend(chunk)
|
||||
return json.loads(bytes(body).decode("utf-8"))
|
||||
|
||||
|
||||
def _measure_vram_mb() -> float:
|
||||
"""This sidecar's own GPU memory in MB (MM2-08). The parent can't see a
|
||||
child's VRAM, so we self-report it in the pong. 0 on CPU. Never raises."""
|
||||
try:
|
||||
import torch
|
||||
if torch.cuda.is_available():
|
||||
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
|
||||
except Exception:
|
||||
pass
|
||||
return 0.0
|
||||
|
||||
|
||||
# ── model loading (lazy, on first synthesize) ─────────────────────────────
|
||||
|
||||
|
||||
# Module-level singleton — populated on the first synthesize op and reused.
|
||||
# Holds (processor, model, device, sample_rate).
|
||||
_state = None
|
||||
|
||||
|
||||
def _load_model(stdout):
|
||||
"""Cold-construct the MOSS-TTS-v1.5 processor + model.
|
||||
|
||||
Device selection is CUDA-or-CPU only — MOSS's upstream documents no MPS
|
||||
path and the custom ``trust_remote_code`` modelling code is untested on
|
||||
Apple Silicon, so we never route to MPS where it might crash. dtype is
|
||||
bf16 on CUDA, fp32 on CPU (bf16 CPU ops are spotty). Emits progress
|
||||
frames so the parent can surface the multi-GB cold-load latency.
|
||||
"""
|
||||
global _state
|
||||
if _state is not None:
|
||||
return _state
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 0})
|
||||
|
||||
import torch
|
||||
from transformers import AutoModel, AutoProcessor
|
||||
|
||||
repo = os.environ.get("OMNIVOICE_MOSS_TTS_V15_MODEL", _DEFAULT_REPO)
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
dtype = torch.bfloat16 if device == "cuda" else torch.float32
|
||||
# "sdpa" works on CUDA + CPU and needs no extra dep. flash_attention_2
|
||||
# (Ampere+ CUDA, optional flash-attn) is opt-in via env.
|
||||
attn = os.environ.get("OMNIVOICE_MOSS_TTS_V15_ATTN", "sdpa")
|
||||
|
||||
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
|
||||
# The audio tokenizer is a separate sub-module that must be moved to the
|
||||
# device independently (easy to miss — see upstream README).
|
||||
processor.audio_tokenizer = processor.audio_tokenizer.to(device)
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 50})
|
||||
|
||||
model = AutoModel.from_pretrained(
|
||||
repo,
|
||||
trust_remote_code=True,
|
||||
attn_implementation=attn,
|
||||
torch_dtype=dtype,
|
||||
).to(device)
|
||||
model.eval()
|
||||
|
||||
sample_rate = int(getattr(processor.model_config, "sampling_rate", MOSS_SAMPLE_RATE))
|
||||
_state = (processor, model, device, sample_rate)
|
||||
|
||||
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 100})
|
||||
return _state
|
||||
|
||||
|
||||
def _tensor_to_pcm_b64(audio, sample_rate: int) -> tuple[str, int, int]:
|
||||
"""Convert a torch waveform tensor to base64 int16 PCM.
|
||||
|
||||
MOSS returns a float tensor in [-1, 1] (1-D or (1, N)); we squeeze to
|
||||
mono, clip, scale to int16, and base64 so the wire frame stays JSON-safe.
|
||||
"""
|
||||
import numpy as np
|
||||
|
||||
arr = audio.detach().to("cpu").float().numpy()
|
||||
arr = np.asarray(arr, dtype=np.float32).squeeze()
|
||||
if arr.ndim > 1:
|
||||
arr = arr.mean(axis=0) # defensive downmix to mono
|
||||
arr = np.clip(arr, -1.0, 1.0)
|
||||
pcm = (arr * 32767.0).astype(np.int16).tobytes()
|
||||
return base64.b64encode(pcm).decode("ascii"), int(sample_rate), int(arr.shape[0])
|
||||
|
||||
|
||||
def _resolve_language(raw):
|
||||
"""Map OmniVoice's language value to a MOSS language name, or None.
|
||||
|
||||
Accepts an ISO-639-1 code or a full name. Unknown / empty / "auto"
|
||||
values return None so MOSS auto-detects."""
|
||||
if not raw or not isinstance(raw, str):
|
||||
return None
|
||||
s = raw.strip()
|
||||
if not s or s.lower() == "auto":
|
||||
return None
|
||||
if s.lower() in _ISO_TO_NAME:
|
||||
return _ISO_TO_NAME[s.lower()]
|
||||
# Already a language name (or an unknown code) — pass it through; MOSS
|
||||
# ignores a language it doesn't recognise.
|
||||
return s
|
||||
|
||||
|
||||
def _handle_synthesize(msg: dict, stdout) -> None:
|
||||
"""Dispatch one synthesize request. Emits the audio frame or raises."""
|
||||
import torch
|
||||
|
||||
text = msg.get("text")
|
||||
if not text or not isinstance(text, str):
|
||||
raise ValueError("synthesize: missing or non-string 'text'")
|
||||
|
||||
processor, model, device, sample_rate = _load_model(stdout)
|
||||
|
||||
user_kwargs: dict = {"text": text}
|
||||
|
||||
ref_audio = msg.get("ref_audio")
|
||||
if ref_audio:
|
||||
# Zero-shot voice cloning: the reference audio alone is enough in
|
||||
# MOSS's clone mode (ref_text is not consumed here). The processor's
|
||||
# audio tokenizer encodes the reference into the prompt.
|
||||
user_kwargs["reference"] = [ref_audio]
|
||||
|
||||
language = _resolve_language(msg.get("language"))
|
||||
if language:
|
||||
user_kwargs["language"] = language
|
||||
|
||||
tokens = msg.get("tokens")
|
||||
if tokens is not None:
|
||||
user_kwargs["tokens"] = int(tokens)
|
||||
|
||||
max_new_tokens = int(msg.get("max_new_tokens", 4096))
|
||||
|
||||
conversations = [[processor.build_user_message(**user_kwargs)]]
|
||||
|
||||
with torch.no_grad():
|
||||
batch = processor(conversations, mode="generation")
|
||||
outputs = model.generate(
|
||||
input_ids=batch["input_ids"].to(device),
|
||||
attention_mask=batch["attention_mask"].to(device),
|
||||
max_new_tokens=max_new_tokens,
|
||||
)
|
||||
|
||||
decoded = processor.decode(outputs)
|
||||
audio = decoded[0].audio_codes_list[0]
|
||||
|
||||
pcm_b64, sr, n_samples = _tensor_to_pcm_b64(audio, sample_rate)
|
||||
_send(stdout, {
|
||||
"op": "audio",
|
||||
"audio_pcm_b64": pcm_b64,
|
||||
"sample_rate": sr,
|
||||
"n_samples": n_samples,
|
||||
})
|
||||
|
||||
|
||||
# ── main loop ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def main() -> int:
|
||||
stdin = sys.stdin.buffer
|
||||
stdout = sys.stdout.buffer
|
||||
|
||||
# Ready handshake fires BEFORE any heavy import — nothing above this line
|
||||
# touches transformers/torch, so we make the 30 s spawn window even cold.
|
||||
_send(stdout, {
|
||||
"op": "ready",
|
||||
"engine": "moss-tts-v15",
|
||||
"sample_rate": MOSS_SAMPLE_RATE,
|
||||
})
|
||||
|
||||
while True:
|
||||
try:
|
||||
msg = _recv(stdin)
|
||||
except Exception as exc:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": "recv",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
return 1
|
||||
if msg is None:
|
||||
return 0
|
||||
|
||||
op = msg.get("op") if isinstance(msg, dict) else None
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
|
||||
elif op == "synthesize":
|
||||
_handle_synthesize(msg, stdout)
|
||||
elif op == "shutdown":
|
||||
return 0
|
||||
else:
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": "dispatch",
|
||||
"message": f"unknown op: {op!r}",
|
||||
})
|
||||
except Exception as exc:
|
||||
# Per-op failure is recoverable — emit the error frame and stay
|
||||
# alive so the parent can retry without paying the respawn +
|
||||
# multi-GB model-load cost again.
|
||||
_send(stdout, {
|
||||
"op": "error",
|
||||
"stage": op or "unknown",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -353,21 +353,6 @@ def _make_backend_class():
|
||||
f"This clears the quarantine on the .app and its "
|
||||
f"bundled binaries. See docs/install/macos.md."
|
||||
)
|
||||
# Execute bit (issue #437). A `git clone` / zip extract on POSIX
|
||||
# can drop +x, which only surfaces at spawn time as
|
||||
# "[Errno 13] Permission denied" — and the generic synth handler
|
||||
# then mislabels it as out-of-memory. Self-heal here, AFTER the
|
||||
# SHA check has confirmed this is the right file (so we never
|
||||
# chmod a foreign binary). No-op on Windows.
|
||||
if os.name == "posix" and not os.access(bin_path, os.X_OK):
|
||||
try:
|
||||
bin_path.chmod(bin_path.stat().st_mode | 0o111)
|
||||
except OSError:
|
||||
return False, (
|
||||
f"GGUF binary {bin_path.name} isn't executable and "
|
||||
f"couldn't be made so — run `chmod +x {bin_path}` "
|
||||
f"and retry."
|
||||
)
|
||||
return True, "ready"
|
||||
except Exception as exc:
|
||||
return False, f"{type(exc).__name__}: {exc}"
|
||||
@@ -508,12 +493,6 @@ def _make_backend_class():
|
||||
* ``ref_audio`` (str/Path) — speaker reference WAV for cloning.
|
||||
* ``ref_text`` (str) — transcript of ``ref_audio``.
|
||||
* ``language`` (str) — ISO code or omnivoice-tts lang label.
|
||||
* ``instruct`` (str) — style instruction.
|
||||
* ``duration`` (float) — target duration in seconds.
|
||||
* ``seed`` (int) — deterministic sampling seed.
|
||||
* ``denoise`` (bool) — omit denoise token when false.
|
||||
* ``preprocess_prompt`` (bool) — skip prompt preprocessing when false.
|
||||
* ``chunk_duration`` / ``chunk_threshold`` (float) — binary long-form controls.
|
||||
"""
|
||||
import soundfile as sf # local import keeps module import cheap
|
||||
import torch
|
||||
@@ -524,32 +503,15 @@ def _make_backend_class():
|
||||
fd, out_str = tempfile.mkstemp(prefix="omnivoice-gguf-", suffix=".wav")
|
||||
os.close(fd)
|
||||
out_path = Path(out_str)
|
||||
ref_text_path: Optional[Path] = None
|
||||
|
||||
try:
|
||||
ref_text = kw.get("ref_text")
|
||||
if kw.get("ref_audio") and ref_text:
|
||||
text_fd, text_str = tempfile.mkstemp(
|
||||
prefix="omnivoice-gguf-ref-", suffix=".txt"
|
||||
)
|
||||
os.close(text_fd)
|
||||
ref_text_path = Path(text_str)
|
||||
ref_text_path.write_text(str(ref_text), encoding="utf-8")
|
||||
|
||||
argv = self._build_argv(
|
||||
base=base_path,
|
||||
tokenizer=tok_path,
|
||||
out_path=out_path,
|
||||
ref_audio=kw.get("ref_audio"),
|
||||
ref_text=str(ref_text_path) if ref_text_path else None,
|
||||
ref_text=kw.get("ref_text"),
|
||||
language=kw.get("language"),
|
||||
instruct=kw.get("instruct"),
|
||||
duration=kw.get("duration"),
|
||||
seed=kw.get("seed"),
|
||||
denoise=kw.get("denoise", True),
|
||||
preprocess_prompt=kw.get("preprocess_prompt", True),
|
||||
chunk_duration=kw.get("chunk_duration"),
|
||||
chunk_threshold=kw.get("chunk_threshold"),
|
||||
)
|
||||
self._run_subprocess(argv, stdin_text=text)
|
||||
wav, sr = sf.read(str(out_path))
|
||||
@@ -558,11 +520,6 @@ def _make_backend_class():
|
||||
out_path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
if ref_text_path is not None:
|
||||
try:
|
||||
ref_text_path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
# soundfile returns (n,) for mono or (n, c) for multichannel.
|
||||
# OmniVoice/Higgs Audio v2 is mono → (n,). Wrap to (1, n).
|
||||
@@ -587,13 +544,6 @@ def _make_backend_class():
|
||||
ref_audio: Optional[str],
|
||||
ref_text: Optional[str],
|
||||
language: Optional[str],
|
||||
instruct: Optional[str] = None,
|
||||
duration: Optional[float] = None,
|
||||
seed: Optional[int] = None,
|
||||
denoise: bool = True,
|
||||
preprocess_prompt: bool = True,
|
||||
chunk_duration: Optional[float] = None,
|
||||
chunk_threshold: Optional[float] = None,
|
||||
) -> list[str]:
|
||||
"""Compose argv from typed Path objects only (T-04-02)."""
|
||||
argv: list[str] = [
|
||||
@@ -605,20 +555,6 @@ def _make_backend_class():
|
||||
lang = _iso_to_omnivoice_lang(language)
|
||||
if lang:
|
||||
argv += ["--lang", lang]
|
||||
if instruct:
|
||||
argv += ["--instruct", str(instruct)]
|
||||
if duration is not None:
|
||||
argv += ["--duration", str(float(duration))]
|
||||
if seed is not None:
|
||||
argv += ["--seed", str(int(seed))]
|
||||
if denoise is False:
|
||||
argv += ["--no-denoise"]
|
||||
if preprocess_prompt is False:
|
||||
argv += ["--no-preprocess-prompt"]
|
||||
if chunk_duration is not None:
|
||||
argv += ["--chunk-duration", str(float(chunk_duration))]
|
||||
if chunk_threshold is not None:
|
||||
argv += ["--chunk-threshold", str(float(chunk_threshold))]
|
||||
if ref_audio:
|
||||
# Two-stage validation (defense in depth):
|
||||
# (a) Reject anything outside the project's voices /
|
||||
@@ -652,8 +588,11 @@ def _make_backend_class():
|
||||
)
|
||||
argv += ["--ref-wav", str(ref_path)]
|
||||
if ref_text:
|
||||
# The C++ runtime expects a transcript file path.
|
||||
# generate() creates this file in the system temp dir.
|
||||
# ref_text is free-form text; pass via stdin would
|
||||
# collide with the synthesis prompt, so the only safe
|
||||
# channel is argv. The binary treats this as a quoted
|
||||
# string at the OS layer (Popen escapes argv per
|
||||
# platform); we don't pre-escape.
|
||||
argv += ["--ref-text", str(ref_text)]
|
||||
return argv
|
||||
|
||||
|
||||
+7
-176
@@ -43,13 +43,11 @@ try:
|
||||
_project_env = os.path.join(os.path.dirname(_backend_dir), ".env")
|
||||
if os.path.isfile(_project_env):
|
||||
dotenv.load_dotenv(_project_env, override=False)
|
||||
# Load the durable per-user config (the in-app Settings source of truth) so
|
||||
# env vars set once survive Tauri/Finder launches that don't inherit a shell
|
||||
# environment. This OVERRIDES launcher-injected defaults: the desktop app
|
||||
# injects a stale OMNIVOICE_CACHE_DIR from its own config before startup, so
|
||||
# without override a models dir changed in Settings was ignored forever (#480).
|
||||
from core.user_env import load_into_environ as _load_user_env
|
||||
_load_user_env()
|
||||
# Also load the durable per-user config so env vars set once survive
|
||||
# Tauri/Finder launches that don't inherit a shell environment.
|
||||
_user_env = os.path.expanduser("~/.config/omnivoice/env")
|
||||
if os.path.isfile(_user_env):
|
||||
dotenv.load_dotenv(_user_env, override=False)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
@@ -286,12 +284,7 @@ from fastapi.responses import JSONResponse, RedirectResponse, Response
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from starlette.datastructures import MutableHeaders
|
||||
# Docs-only dependency: a venv created before scalar-fastapi entered the
|
||||
# dependency set must still boot the backend (#307) — /docs degrades instead.
|
||||
try:
|
||||
from scalar_fastapi import get_scalar_api_reference
|
||||
except ImportError:
|
||||
get_scalar_api_reference = None
|
||||
from scalar_fastapi import get_scalar_api_reference
|
||||
import traceback
|
||||
|
||||
_crash_log_lock = threading.Lock()
|
||||
@@ -320,7 +313,6 @@ from api.routers import (
|
||||
setup,
|
||||
gallery,
|
||||
archetypes,
|
||||
describe_voice,
|
||||
community,
|
||||
batch,
|
||||
watermark,
|
||||
@@ -330,10 +322,7 @@ from api.routers import (
|
||||
openai_compat,
|
||||
tts_stream,
|
||||
marketplace,
|
||||
personas,
|
||||
sonitranslate,
|
||||
audiobook,
|
||||
longform_jobs,
|
||||
settings as settings_router, # Phase 1 AUTH-03: HF token save/clear/state
|
||||
)
|
||||
from utils import hf_progress
|
||||
@@ -343,30 +332,6 @@ from utils import hf_progress
|
||||
# the patched class, not the original.
|
||||
hf_progress.install()
|
||||
|
||||
# Wire the overall download aggregator's byte sink onto the patched tqdm so
|
||||
# parallel per-file updates feed one accurate overall bar (FDL-06).
|
||||
try:
|
||||
from utils import download_aggregator
|
||||
download_aggregator.install()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Log the download-acceleration state once at startup (FDL-03) so a slow
|
||||
# download report can be triaged from the logs without reproducing. Note: the
|
||||
# app sets HF_HUB_DISABLE_XET=1 above by default (legacy LFS for byte progress),
|
||||
# so xet_active is normally False even though hf_xet is installed.
|
||||
try:
|
||||
from api.routers.system import _fast_download_status as _fd_status
|
||||
_fd = _fd_status()
|
||||
_xet_ver = f" {_fd['xet_version']}" if _fd.get("xet_version") else ""
|
||||
logging.getLogger("omnivoice.model").info(
|
||||
"downloads: Xet %s (hf_xet%s installed=%s), high_perf=%s",
|
||||
"ACTIVE" if _fd["xet_active"] else "disabled → legacy LFS",
|
||||
_xet_ver, _fd["xet_installed"], _fd["high_performance"],
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _env_flag(name: str, default: bool = False) -> bool:
|
||||
value = os.environ.get(name)
|
||||
@@ -457,23 +422,7 @@ async def lifespan(app: FastAPI):
|
||||
capture_preload_task = asyncio.create_task(_preload_capture_asr())
|
||||
else:
|
||||
logger.info("Capture ASR preload disabled; dictation ASR will load on first use.")
|
||||
|
||||
# ── MCP session manager (Wave 2.2) ────────────────────────────────────
|
||||
# FastMCP's Streamable-HTTP transport needs its session manager running
|
||||
# for the lifetime of the app. It's created lazily by streamable_http_app()
|
||||
# (called in mount_mcp below), so we stack its `run()` context into ours
|
||||
# via AsyncExitStack rather than replacing this lifespan. Best-effort: a
|
||||
# missing/broken MCP layer must never stop the rest of the backend.
|
||||
from contextlib import AsyncExitStack
|
||||
async with AsyncExitStack() as _mcp_stack:
|
||||
_sm = getattr(app.state, "mcp_session_manager", None)
|
||||
if _sm is not None:
|
||||
try:
|
||||
await _mcp_stack.enter_async_context(_sm.run())
|
||||
logger.info("MCP server mounted at /mcp")
|
||||
except Exception as e:
|
||||
logger.warning("MCP session manager failed to start: %s", e)
|
||||
yield
|
||||
yield
|
||||
# ── Graceful shutdown (SIGTERM from Tauri, Ctrl+C, etc.) ────────────
|
||||
logger.info("Shutdown: cleaning up…")
|
||||
idle_task.cancel()
|
||||
@@ -522,14 +471,6 @@ app = FastAPI(
|
||||
@app.get("/docs", include_in_schema=False)
|
||||
async def scalar_docs():
|
||||
"""Interactive API documentation powered by Scalar."""
|
||||
if get_scalar_api_reference is None:
|
||||
return JSONResponse(
|
||||
status_code=503,
|
||||
content={
|
||||
"detail": "API docs unavailable: scalar-fastapi is not installed "
|
||||
"in the backend environment (#307)."
|
||||
},
|
||||
)
|
||||
return get_scalar_api_reference(
|
||||
openapi_url=app.openapi_url,
|
||||
title=app.title,
|
||||
@@ -638,68 +579,6 @@ class NetworkAccessMiddleware:
|
||||
return await self.app(scope, receive, send)
|
||||
|
||||
|
||||
class BearerKeyMiddleware:
|
||||
"""When OMNIVOICE_API_KEY is set, non-loopback clients must present it on
|
||||
every HTTP + WebSocket request: ``Authorization: Bearer <key>``,
|
||||
``?api_key=<key>`` (browser WebSockets cannot set headers), or the
|
||||
``ov_key`` cookie (set on the first successful HTTP auth). Loopback
|
||||
always bypasses — the desktop default is unchanged — and the SPA shell
|
||||
paths stay reachable so a remote UI can load and show what's wrong.
|
||||
|
||||
Inert when the env var is unset (the default). Pure ASGI for the same
|
||||
no-buffering reason as NetworkAccessMiddleware above. Plain-HTTP caveat
|
||||
is documented in docs/remote-gpu.md: the key is sniffable outside a
|
||||
WireGuard (Tailscale) or TLS (tailscale serve) transport.
|
||||
"""
|
||||
|
||||
def __init__(self, app):
|
||||
self.app = app
|
||||
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope["type"] not in ("http", "websocket"):
|
||||
return await self.app(scope, receive, send)
|
||||
key = os.environ.get("OMNIVOICE_API_KEY") or ""
|
||||
if not key:
|
||||
return await self.app(scope, receive, send)
|
||||
client = scope["client"][0] if scope.get("client") else None
|
||||
if client in _LOOPBACK_CLIENTS:
|
||||
return await self.app(scope, receive, send)
|
||||
path = scope.get("path", "")
|
||||
if scope["type"] == "http" and (
|
||||
path in _SHELL_PATHS or path.startswith("/assets/") or path.startswith("/favicon")
|
||||
):
|
||||
return await self.app(scope, receive, send)
|
||||
|
||||
from starlette.requests import HTTPConnection
|
||||
|
||||
conn = HTTPConnection(scope)
|
||||
auth = conn.headers.get("authorization", "")
|
||||
supplied = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
|
||||
if not supplied:
|
||||
supplied = conn.query_params.get("api_key") or conn.cookies.get("ov_key") or ""
|
||||
|
||||
if not secrets.compare_digest(supplied, key):
|
||||
if scope["type"] == "websocket":
|
||||
# Reject the handshake; 1008 = policy violation.
|
||||
await receive() # consume websocket.connect
|
||||
await send({"type": "websocket.close", "code": 1008})
|
||||
return
|
||||
resp = JSONResponse({"detail": "API key required"}, status_code=401)
|
||||
return await resp(scope, receive, send)
|
||||
|
||||
if scope["type"] == "http" and conn.cookies.get("ov_key") != key:
|
||||
async def send_with_cookie(message):
|
||||
if message["type"] == "http.response.start":
|
||||
headers = MutableHeaders(scope=message)
|
||||
headers.append(
|
||||
"set-cookie", f"ov_key={key}; Path=/; SameSite=Lax"
|
||||
)
|
||||
await send(message)
|
||||
|
||||
return await self.app(scope, receive, send_with_cookie)
|
||||
return await self.app(scope, receive, send)
|
||||
|
||||
|
||||
# UI dev-server port — single-sourced from OMNIVOICE_UI_PORT so a user who
|
||||
# moves the Vite dev server off 3901 still gets a matching CORS allow-list.
|
||||
def _ui_port() -> int:
|
||||
@@ -731,29 +610,6 @@ app.add_middleware(
|
||||
# applied even to the 401 PIN-required responses). Inert unless a PIN is set.
|
||||
app.add_middleware(NetworkAccessMiddleware)
|
||||
|
||||
# Remote-backend bearer gate (parity program Wave 2.3 / §R2). Inert unless
|
||||
# OMNIVOICE_API_KEY is set. Distinct from the PIN gate above: the PIN guards
|
||||
# casual LAN-share guests for one session; the API key is the durable
|
||||
# credential for running this backend remotely (Tailscale / Docker GPU box).
|
||||
# Covers WebSockets too — the PIN gate never did, because every WS endpoint
|
||||
# carried its own loopback guard; remote mode is exactly the case where a
|
||||
# keyed non-loopback client must reach them.
|
||||
app.add_middleware(BearerKeyMiddleware)
|
||||
|
||||
# Register canonical audio MIME types before any StaticFiles mount.
|
||||
# Python's `mimetypes.guess_type()` returns `audio/x-wav` for `.wav` and
|
||||
# `audio/x-flac` for `.flac` on most platforms — these are vendor-experimental
|
||||
# (x- prefix, never IANA-registered). macOS Chrome/Safari MIME-sniff leniently
|
||||
# via CoreAudio so playback works there, but Linux Chrome/Firefox (FFmpeg) and
|
||||
# Android Chrome (ExoPlayer) strictly honor the declared type and treat the
|
||||
# x- variants as download-only — manifesting as the play button silently
|
||||
# doing nothing in the browser app while working in the Tauri desktop shell.
|
||||
# `audio/wav` / `audio/flac` are the IANA-canonical types.
|
||||
# Ref: https://www.iana.org/assignments/media-types/media-types.xhtml#audio
|
||||
import mimetypes as _mimetypes
|
||||
_mimetypes.add_type("audio/wav", ".wav")
|
||||
_mimetypes.add_type("audio/flac", ".flac")
|
||||
|
||||
app.mount("/audio", StaticFiles(directory=OUTPUTS_DIR), name="audio")
|
||||
app.mount("/voice_audio", StaticFiles(directory=VOICES_DIR), name="voice_audio")
|
||||
|
||||
@@ -796,7 +652,6 @@ app.include_router(stories.router)
|
||||
app.include_router(setup.router)
|
||||
app.include_router(gallery.router)
|
||||
app.include_router(archetypes.router)
|
||||
app.include_router(describe_voice.router) # issue #317: free-text voice design
|
||||
app.include_router(community.router)
|
||||
app.include_router(batch.router)
|
||||
app.include_router(watermark.router)
|
||||
@@ -806,32 +661,8 @@ app.include_router(capture_ws.router)
|
||||
app.include_router(openai_compat.router)
|
||||
app.include_router(tts_stream.router)
|
||||
app.include_router(marketplace.router)
|
||||
app.include_router(personas.router)
|
||||
app.include_router(sonitranslate.router)
|
||||
app.include_router(audiobook.router)
|
||||
app.include_router(longform_jobs.router)
|
||||
app.include_router(settings_router.router) # Phase 1 AUTH-03 endpoints
|
||||
from api.routers import mcp_bindings as _mcp_bindings_router # noqa: E402
|
||||
app.include_router(_mcp_bindings_router.router) # Wave 2.2 per-agent voice bindings
|
||||
|
||||
# ── Mount the MCP server (Wave 2.2) ───────────────────────────────────────
|
||||
# FastMCP's Streamable-HTTP app is sub-mounted at /mcp; its session manager is
|
||||
# stashed on app.state for the lifespan above to run. Opt-out via
|
||||
# OMNIVOICE_MCP_DISABLE=1; best-effort so a missing mcp package or a build
|
||||
# without it never breaks startup.
|
||||
if os.environ.get("OMNIVOICE_MCP_DISABLE", "").strip().lower() not in ("1", "true", "yes", "on"):
|
||||
try:
|
||||
from mcp_server import create_mcp_server
|
||||
|
||||
_mcp = create_mcp_server()
|
||||
_mcp_app = _mcp.streamable_http_app()
|
||||
app.state.mcp_session_manager = _mcp.session_manager
|
||||
app.mount("/mcp", _mcp_app)
|
||||
logging.getLogger("omnivoice.api").info("MCP app mounted at /mcp")
|
||||
except Exception as _mcp_err: # noqa: BLE001
|
||||
logging.getLogger("omnivoice.api").info(
|
||||
"MCP server not mounted (%s); /mcp disabled.", _mcp_err
|
||||
)
|
||||
|
||||
frontend_path = os.path.join(os.path.dirname(__file__), "..", "frontend", "dist")
|
||||
if os.path.exists(frontend_path):
|
||||
|
||||
+1
-64
@@ -53,14 +53,6 @@ def create_mcp_server():
|
||||
"voice design, and video dubbing in 646 languages."
|
||||
),
|
||||
)
|
||||
# Serve the Streamable-HTTP transport at the app root so mounting the whole
|
||||
# app at "/mcp" on the main FastAPI yields the endpoint at "/mcp". FastMCP's
|
||||
# default path is "/mcp", which would double-prefix to "/mcp/mcp" when
|
||||
# sub-mounted. Harmless for the standalone CLI run() path.
|
||||
try:
|
||||
mcp.settings.streamable_http_path = "/"
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# ── Helpers ─────────────────────────────────────────────────────────
|
||||
|
||||
@@ -83,21 +75,6 @@ def create_mcp_server():
|
||||
|
||||
# ── Tools ───────────────────────────────────────────────────────────
|
||||
|
||||
def _current_client_id() -> str | None:
|
||||
"""The X-OmniVoice-Client-Id of the calling MCP client, if any.
|
||||
|
||||
FastMCP exposes the HTTP request via its request context on the
|
||||
Streamable-HTTP transport; stdio clients (and any version where the
|
||||
accessor differs) simply resolve to None and fall back to the
|
||||
global default voice."""
|
||||
try:
|
||||
req = mcp.get_context().request_context.request
|
||||
if req is not None:
|
||||
return req.headers.get("x-omnivoice-client-id")
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
@mcp.tool()
|
||||
async def generate_speech(
|
||||
text: str,
|
||||
@@ -112,8 +89,7 @@ def create_mcp_server():
|
||||
Args:
|
||||
text: The text to synthesize into speech.
|
||||
language: Target language (ISO code or 'Auto'). 646 languages supported.
|
||||
profile_id: ID of a saved voice profile to clone. Omit to use this
|
||||
agent's bound voice (Settings → MCP), else the global default.
|
||||
profile_id: ID of a saved voice profile to clone. Omit for voice design mode.
|
||||
instruct: Style instruction (e.g. 'whisper', 'excited', 'narrator').
|
||||
speed: Speech speed multiplier (0.5–2.0, default 1.0).
|
||||
steps: Diffusion steps (8=fast/draft, 16=balanced, 32=quality).
|
||||
@@ -122,17 +98,6 @@ def create_mcp_server():
|
||||
JSON with audio_id, generation_time, audio_duration, and
|
||||
base64-encoded WAV data.
|
||||
"""
|
||||
# Per-agent voice binding (Wave 2.2): explicit arg wins; otherwise
|
||||
# resolve this client's bound profile, then the global default.
|
||||
client_id = _current_client_id()
|
||||
try:
|
||||
from services import mcp_bindings
|
||||
resolved = mcp_bindings.resolve_voice(client_id, profile_id)
|
||||
profile_id = resolved.get("profile_id")
|
||||
mcp_bindings.touch_last_seen(client_id) if client_id else None
|
||||
except Exception:
|
||||
pass # binding layer unavailable — use whatever was passed
|
||||
|
||||
form = {
|
||||
"text": text,
|
||||
"language": language,
|
||||
@@ -194,34 +159,6 @@ def create_mcp_server():
|
||||
'],"note":"Pass any ISO 639 code or set language=Auto for detection."}'
|
||||
)
|
||||
|
||||
@mcp.tool()
|
||||
async def transcribe(audio_base64: str, language: str | None = None) -> str:
|
||||
"""Transcribe spoken audio to text.
|
||||
|
||||
Args:
|
||||
audio_base64: Base64-encoded audio bytes (wav/mp3/webm/m4a).
|
||||
language: Optional language hint; omit for auto-detect.
|
||||
|
||||
Returns:
|
||||
JSON with the recognized text, language, and duration.
|
||||
"""
|
||||
try:
|
||||
raw = base64.b64decode(audio_base64, validate=True)
|
||||
except Exception:
|
||||
return '{"error":"audio_base64 is not valid base64"}'
|
||||
# 200 MB cap — same spirit as voicebox's transcribe gate. Keeps a
|
||||
# buggy/hostile agent from posting an unbounded blob.
|
||||
if len(raw) > 200 * 1024 * 1024:
|
||||
return '{"error":"audio exceeds 200 MB limit"}'
|
||||
data = {}
|
||||
if language:
|
||||
data["language"] = language
|
||||
r = await _api_post_form(
|
||||
"/transcribe", data=data,
|
||||
files={"audio": ("audio.wav", raw, "application/octet-stream")},
|
||||
)
|
||||
return str(r.json())
|
||||
|
||||
@mcp.tool()
|
||||
async def check_health() -> str:
|
||||
"""Check if the OmniVoice backend is running and what GPU device is active."""
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
"""omnivoice-mcp — stdio MCP shim for clients that only speak stdio."""
|
||||
@@ -1,176 +0,0 @@
|
||||
"""omnivoice-mcp — stdio ↔ Streamable-HTTP MCP proxy (Wave 2.2).
|
||||
|
||||
Adapted from voicebox (https://github.com/jamiepine/voicebox), MIT License,
|
||||
Copyright (c) voicebox contributors.
|
||||
|
||||
Some MCP clients only speak stdio. They spawn this binary; we pipe each
|
||||
JSON-RPC message to ``http://127.0.0.1:<port>/mcp/`` (the FastMCP app mounted
|
||||
on the running OmniVoice backend) and stream the server's response back.
|
||||
|
||||
Environment variables:
|
||||
OMNIVOICE_PORT backend port (default 3900).
|
||||
OMNIVOICE_HOST host (default 127.0.0.1).
|
||||
OMNIVOICE_CLIENT_ID forwarded as X-OmniVoice-Client-Id on every request
|
||||
(drives per-agent voice binding).
|
||||
|
||||
Stdout is JSON-RPC only. Diagnostics go to stderr.
|
||||
Exit 0 on clean EOF, 1 on transport error, 2 if the backend never answers.
|
||||
|
||||
Usage in an MCP client config (stdio):
|
||||
command: python
|
||||
args: ["-m", "backend.mcp_shim"]
|
||||
env: { OMNIVOICE_CLIENT_ID: "claude-code" }
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
CLIENT_ID_HEADER = "X-OmniVoice-Client-Id"
|
||||
SESSION_HEADER = "mcp-session-id"
|
||||
HEALTH_TIMEOUT_S = 30.0
|
||||
DEFAULT_PORT = 3900
|
||||
|
||||
|
||||
def _err(msg: str) -> None:
|
||||
print(f"omnivoice-mcp: {msg}", file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def _base_url() -> tuple[str, str]:
|
||||
host = os.environ.get("OMNIVOICE_HOST", "127.0.0.1")
|
||||
port = int(os.environ.get("OMNIVOICE_PORT", str(DEFAULT_PORT)))
|
||||
return f"http://{host}:{port}/mcp/", f"http://{host}:{port}/health"
|
||||
|
||||
|
||||
async def _wait_for_backend(client: httpx.AsyncClient, health_url: str) -> bool:
|
||||
loop = asyncio.get_running_loop()
|
||||
deadline = loop.time() + HEALTH_TIMEOUT_S
|
||||
while loop.time() < deadline:
|
||||
try:
|
||||
r = await client.get(health_url, timeout=2.0)
|
||||
if r.status_code == 200:
|
||||
return True
|
||||
except Exception:
|
||||
pass
|
||||
await asyncio.sleep(0.5)
|
||||
return False
|
||||
|
||||
|
||||
async def _read_stdin_line() -> str | None:
|
||||
loop = asyncio.get_running_loop()
|
||||
line = await loop.run_in_executor(None, sys.stdin.readline)
|
||||
return line or None
|
||||
|
||||
|
||||
def _write_stdout(obj: Any) -> None:
|
||||
sys.stdout.write(json.dumps(obj, separators=(",", ":")))
|
||||
sys.stdout.write("\n")
|
||||
sys.stdout.flush()
|
||||
|
||||
|
||||
async def _handle_request(
|
||||
client: httpx.AsyncClient,
|
||||
url: str,
|
||||
raw: str,
|
||||
headers: dict[str, str],
|
||||
session_id: list[str | None],
|
||||
) -> None:
|
||||
try:
|
||||
message = json.loads(raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
_err(f"invalid JSON on stdin: {exc}")
|
||||
return
|
||||
|
||||
req_headers = {
|
||||
"Content-Type": "application/json",
|
||||
"Accept": "application/json, text/event-stream",
|
||||
**headers,
|
||||
}
|
||||
if session_id[0]:
|
||||
req_headers[SESSION_HEADER] = session_id[0]
|
||||
|
||||
is_notification = isinstance(message, dict) and "id" not in message
|
||||
|
||||
async with client.stream("POST", url, headers=req_headers, content=raw.encode("utf-8")) as response:
|
||||
if session_id[0] is None:
|
||||
sid = response.headers.get(SESSION_HEADER)
|
||||
if sid:
|
||||
session_id[0] = sid
|
||||
|
||||
if response.status_code == 202:
|
||||
return # notification acknowledged
|
||||
if response.status_code >= 400:
|
||||
body = await response.aread()
|
||||
_err(f"server {response.status_code}: {body.decode('utf-8', errors='replace')[:400]}")
|
||||
if is_notification:
|
||||
return
|
||||
_write_stdout({
|
||||
"jsonrpc": "2.0",
|
||||
"id": message.get("id"),
|
||||
"error": {"code": -32000, "message": f"OmniVoice MCP proxy got HTTP {response.status_code}"},
|
||||
})
|
||||
return
|
||||
|
||||
ctype = response.headers.get("content-type", "")
|
||||
if "text/event-stream" in ctype:
|
||||
async for line in response.aiter_lines():
|
||||
if line.startswith("data:"):
|
||||
payload = line[5:].strip()
|
||||
if not payload:
|
||||
continue
|
||||
try:
|
||||
_write_stdout(json.loads(payload))
|
||||
except json.JSONDecodeError:
|
||||
_err(f"malformed SSE payload: {payload[:200]}")
|
||||
else:
|
||||
body = await response.aread()
|
||||
try:
|
||||
_write_stdout(json.loads(body))
|
||||
except json.JSONDecodeError:
|
||||
_err(f"non-JSON response ({ctype}): {body.decode('utf-8', errors='replace')[:200]}")
|
||||
|
||||
|
||||
async def _run() -> int:
|
||||
url, health_url = _base_url()
|
||||
forward_headers: dict[str, str] = {}
|
||||
client_id = os.environ.get("OMNIVOICE_CLIENT_ID")
|
||||
if client_id:
|
||||
forward_headers[CLIENT_ID_HEADER] = client_id
|
||||
|
||||
session_id: list[str | None] = [None]
|
||||
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(300.0)) as client:
|
||||
if not await _wait_for_backend(client, health_url):
|
||||
_err(f"timed out waiting for OmniVoice at {health_url} — is the app running?")
|
||||
return 2
|
||||
try:
|
||||
while True:
|
||||
line = await _read_stdin_line()
|
||||
if line is None:
|
||||
return 0
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
await _handle_request(client, url, line, forward_headers, session_id)
|
||||
except (KeyboardInterrupt, SystemExit):
|
||||
return 0
|
||||
except Exception as exc:
|
||||
_err(f"proxy failed: {exc!r}")
|
||||
return 1
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
return asyncio.run(_run())
|
||||
except KeyboardInterrupt:
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,58 +0,0 @@
|
||||
"""Parity program Wave 0.2: consent-locked voice profiles
|
||||
|
||||
Revision ID: 0003_voice_profile_consent
|
||||
Revises: 0002_voice_profile_demo_fields
|
||||
Create Date: 2026-06-12 00:00:00.000000
|
||||
|
||||
Adds four additive columns to ``voice_profiles`` backing the
|
||||
``verified_own_voice`` consent lock (docs/competitive-analysis.md Action 22 /
|
||||
parity program Wave 0.2). A profile becomes "verified" when its owner records
|
||||
a spoken consent statement; agentic features and gallery sharing will require
|
||||
the flag — plain local synthesis never does.
|
||||
|
||||
* ``verified_own_voice INTEGER DEFAULT 0`` — the consent lock itself.
|
||||
* ``consent_text TEXT DEFAULT ''`` — the statement that was read aloud.
|
||||
* ``consent_audio_path TEXT DEFAULT ''`` — filename of the recorded
|
||||
statement in VOICES_DIR (kept as provenance, deletable via revoke).
|
||||
* ``consent_recorded_at REAL DEFAULT NULL`` — UNIX timestamp.
|
||||
|
||||
Behavior mirrors 0002: ``_has_column`` PRAGMA guards make upgrade a no-op on
|
||||
fresh installs (where _BASE_SCHEMA already has the columns), satisfying the
|
||||
"Backward-compatible project data" constraint; downgrade drops the columns
|
||||
(SQLite >= 3.35).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = "0003_voice_profile_consent"
|
||||
down_revision: Union[str, None] = "0002_voice_profile_demo_fields"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
_COLUMNS = (
|
||||
("verified_own_voice", sa.Column("verified_own_voice", sa.Integer(), nullable=False, server_default="0")),
|
||||
("consent_text", sa.Column("consent_text", sa.Text(), nullable=False, server_default="")),
|
||||
("consent_audio_path", sa.Column("consent_audio_path", sa.Text(), nullable=False, server_default="")),
|
||||
("consent_recorded_at", sa.Column("consent_recorded_at", sa.Float(), nullable=True)),
|
||||
)
|
||||
|
||||
|
||||
def _has_column(table: str, column: str) -> bool:
|
||||
bind = op.get_bind()
|
||||
rows = bind.execute(sa.text(f"PRAGMA table_info({table})")).fetchall()
|
||||
return any(r[1] == column for r in rows)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
for name, column in _COLUMNS:
|
||||
if not _has_column("voice_profiles", name):
|
||||
op.add_column("voice_profiles", column)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
for name, _ in reversed(_COLUMNS):
|
||||
if _has_column("voice_profiles", name):
|
||||
op.drop_column("voice_profiles", name)
|
||||
@@ -1,60 +0,0 @@
|
||||
"""Parity program Wave 2.2: per-agent MCP voice bindings
|
||||
|
||||
Revision ID: 0004_mcp_client_bindings
|
||||
Revises: 0003_voice_profile_consent
|
||||
Create Date: 2026-06-12 00:00:00.000000
|
||||
|
||||
Adds the ``mcp_client_bindings`` table backing per-agent voice binding
|
||||
(docs/competitive-analysis.md Spec 2): each MCP client (identified by the
|
||||
``X-OmniVoice-Client-Id`` header it sends) can be bound to a default voice
|
||||
profile / engine, so "Claude Code speaks in Morgan, Cursor in Scarlett".
|
||||
|
||||
* ``client_id`` TEXT PRIMARY KEY — the agent's stable id.
|
||||
* ``label`` TEXT — human label shown in Settings.
|
||||
* ``profile_id`` TEXT — voice profile to speak in (nullable FK-by-convention).
|
||||
* ``default_engine`` TEXT — engine override (nullable).
|
||||
* ``last_seen_at`` REAL — updated when the client calls a tool.
|
||||
* ``created_at`` REAL.
|
||||
|
||||
Additive + idempotent (guarded by sqlite_master), matching 0002/0003, so
|
||||
re-running on a fresh-install DB where _BASE_SCHEMA already created it is a
|
||||
no-op (Backward-compatible project data constraint).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = "0004_mcp_client_bindings"
|
||||
down_revision: Union[str, None] = "0003_voice_profile_consent"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def _has_table(name: str) -> bool:
|
||||
bind = op.get_bind()
|
||||
row = bind.execute(
|
||||
sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name=:n"),
|
||||
{"n": name},
|
||||
).fetchone()
|
||||
return row is not None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if _has_table("mcp_client_bindings"):
|
||||
return
|
||||
op.create_table(
|
||||
"mcp_client_bindings",
|
||||
sa.Column("client_id", sa.Text(), primary_key=True),
|
||||
sa.Column("label", sa.Text(), nullable=False, server_default=""),
|
||||
sa.Column("profile_id", sa.Text(), nullable=True),
|
||||
sa.Column("default_engine", sa.Text(), nullable=True),
|
||||
sa.Column("last_seen_at", sa.Float(), nullable=True),
|
||||
sa.Column("created_at", sa.Float(), nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_table("mcp_client_bindings"):
|
||||
op.drop_table("mcp_client_bindings")
|
||||
@@ -1,65 +0,0 @@
|
||||
"""Voice Studio unification: profile `kind` discriminator + design params
|
||||
|
||||
Revision ID: 0005_unified_profiles
|
||||
Revises: 0004_mcp_client_bindings
|
||||
Create Date: 2026-06-13 00:00:00.000000
|
||||
|
||||
Adds two additive columns to ``voice_profiles`` so a *designed* voice
|
||||
(category sliders + instruct, no user reference audio) is a first-class
|
||||
profile rather than a transient UI state
|
||||
(docs/specs/voice-studio-unification.md §3):
|
||||
|
||||
* ``kind TEXT DEFAULT 'clone'`` — ``'clone'`` (user reference audio) or
|
||||
``'design'`` (rendered sample + stored design params). Replaces the
|
||||
brittle is_locked/instruct inference in /generate.
|
||||
* ``vd_states TEXT DEFAULT NULL`` — JSON of the design category picks
|
||||
(Gender/Age/Pitch/Style/accent/dialect) so selecting a design profile
|
||||
can restore the sliders for re-editing.
|
||||
|
||||
Backfill: every existing row becomes ``kind='clone'`` — all of them carry a
|
||||
real or rendered ``ref_audio_path`` today (archetype materialization
|
||||
included), so the default is semantically true and no audio is re-rendered.
|
||||
|
||||
Behavior mirrors 0002/0003: ``_has_column`` PRAGMA guards make upgrade a
|
||||
no-op on fresh installs (where _BASE_SCHEMA already has the columns),
|
||||
satisfying the "Backward-compatible project data" constraint; downgrade
|
||||
drops the columns (SQLite >= 3.35).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = "0005_unified_profiles"
|
||||
down_revision: Union[str, None] = "0004_mcp_client_bindings"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def _has_column(table: str, column: str) -> bool:
|
||||
bind = op.get_bind()
|
||||
rows = bind.execute(sa.text(f"PRAGMA table_info({table})")).fetchall()
|
||||
return any(r[1] == column for r in rows)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if not _has_column("voice_profiles", "kind"):
|
||||
op.add_column(
|
||||
"voice_profiles",
|
||||
sa.Column("kind", sa.Text(), nullable=False, server_default="clone"),
|
||||
)
|
||||
# server_default covers new rows; make existing rows explicit too.
|
||||
op.execute("UPDATE voice_profiles SET kind='clone' WHERE kind IS NULL OR kind=''")
|
||||
if not _has_column("voice_profiles", "vd_states"):
|
||||
op.add_column(
|
||||
"voice_profiles",
|
||||
sa.Column("vd_states", sa.Text(), nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_column("voice_profiles", "vd_states"):
|
||||
op.drop_column("voice_profiles", "vd_states")
|
||||
if _has_column("voice_profiles", "kind"):
|
||||
op.drop_column("voice_profiles", "kind")
|
||||
@@ -1,36 +0,0 @@
|
||||
"""Heal voice_profiles.instruct poisoned with the "[object Object]" sentinel.
|
||||
|
||||
Revision ID: 0006_strip_object_object_instruct
|
||||
Revises: 0005_unified_profiles
|
||||
Create Date: 2026-06-20 00:00:00.000000
|
||||
|
||||
A pre-fix Voice Studio build ("Save design as profile") passed the
|
||||
``buildDesignInstruct()`` *object* straight to FormData, which string-coerced it
|
||||
to the literal ``"[object Object]"`` and persisted that into
|
||||
``voice_profiles.instruct`` (#550 #545 #542 #537 #530 #525). On first
|
||||
preview/use that value fails the engine instruct validator with a 400. The
|
||||
frontend + backend fixes stop any NEW poisoned rows; this migration heals the
|
||||
ones already saved on the buggy build (the local-first backward-compat rule —
|
||||
existing project data must keep working without manual migration).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
from sqlalchemy import inspect
|
||||
|
||||
revision: str = "0006_strip_object_object_instruct"
|
||||
down_revision: Union[str, None] = "0005_unified_profiles"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
bind = op.get_bind()
|
||||
if "voice_profiles" in inspect(bind).get_table_names():
|
||||
# Idempotent: only touches rows whose instruct is literally the sentinel.
|
||||
op.execute("UPDATE voice_profiles SET instruct='' WHERE instruct='[object Object]'")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# Irreversible heal — the original garbage sentinel is not worth restoring.
|
||||
pass
|
||||
@@ -25,11 +25,6 @@ class DubSegment(BaseModel):
|
||||
speed: Optional[float] = None
|
||||
gain: Optional[float] = None # Per-segment volume (0.0 - 2.0, default 1.0)
|
||||
target_lang: Optional[str] = None # Per-segment language override (ISO code)
|
||||
# Phase 4.2 free-form directorial note ("urgent, whispered…"). The client
|
||||
# has always sent this; without the field pydantic silently dropped it,
|
||||
# so directions never reached TTS and never entered the regen
|
||||
# fingerprint (#281).
|
||||
direction: Optional[str] = None
|
||||
effect_preset: str = "broadcast" # NEW: DSP preset id (default: broadcast)
|
||||
|
||||
@field_validator("effect_preset")
|
||||
@@ -42,19 +37,6 @@ class DubSegment(BaseModel):
|
||||
)
|
||||
return v
|
||||
|
||||
class FitOptions(BaseModel):
|
||||
"""Optional knob overrides for the `smart_fit` timing strategy.
|
||||
|
||||
All fields default to None — the server fills in the canonical
|
||||
defaults (services.fit_planner.FitParams) so old clients and sparse
|
||||
payloads behave identically to a fully-populated default payload.
|
||||
"""
|
||||
max_audio_only_rate: Optional[float] = None # default 1.2
|
||||
audio_rate_cap: Optional[float] = None # default 1.5
|
||||
video_slow_cap: Optional[float] = None # default 2.0
|
||||
gap_guard_s: Optional[float] = None # default 0.05
|
||||
allow_video_retime: Optional[bool] = None # default True
|
||||
|
||||
class DubRequest(BaseModel):
|
||||
segments: List[DubSegment]
|
||||
language: str = "Auto"
|
||||
@@ -93,35 +75,18 @@ class DubRequest(BaseModel):
|
||||
# each segment's video portion is stretched (via
|
||||
# ffmpeg setpts) to fit the natural-rate dub audio.
|
||||
# Audio plays at 1.0×; total video duration grows.
|
||||
# "smart_fit" — dub-length fitting v2: split the burden between a
|
||||
# mild pitch-preserving audio speed-up (≤1.2× alone,
|
||||
# ≤1.5× in hybrid) and a mild per-segment video
|
||||
# slow-down (≤2.0×), per services/fit_planner.py.
|
||||
# Residual overflow is trimmed and surfaced.
|
||||
# "strict_slot" — legacy: keep `slot_fit` semantics (atempo squeeze
|
||||
# when audio > slot). Kept for back-compat.
|
||||
timing_strategy: Optional[Literal["concise", "stretch_video", "strict_slot", "smart_fit"]] = "concise"
|
||||
timing_strategy: Optional[Literal["concise", "stretch_video", "strict_slot"]] = "concise"
|
||||
|
||||
# Per-job slip budget for "concise" mode. Hard-trim only kicks in once
|
||||
# gap absorption + this much extra time has been consumed.
|
||||
overflow_budget_s: Optional[float] = 0.0
|
||||
|
||||
# Knob overrides for `smart_fit` (ignored by other strategies). Omitted
|
||||
# fields default server-side to fit_planner.FitParams values.
|
||||
fit_options: Optional[FitOptions] = None
|
||||
|
||||
class TranslateSegment(BaseModel):
|
||||
id: str
|
||||
text: str
|
||||
target_lang: Optional[str] = None
|
||||
# Free-form delivery direction ("urgent, whispering") — feeds the
|
||||
# cinematic reflect/adapt prompts. The frontend has sent this since
|
||||
# Phase 4.2 but pydantic silently dropped it as an undeclared extra,
|
||||
# so the per-segment direction hint never reached the LLM.
|
||||
direction: Optional[str] = None
|
||||
# Available time slot (end - start, seconds) for rate-ratio prediction
|
||||
# and the cinematic slot-fit pass. Same silent-drop fix as `direction`.
|
||||
slot_seconds: Optional[float] = None
|
||||
|
||||
class TranslateRequest(BaseModel):
|
||||
segments: List[TranslateSegment]
|
||||
@@ -131,12 +96,6 @@ class TranslateRequest(BaseModel):
|
||||
job_id: Optional[str] = None # Dub job id, used to resolve detected source_lang
|
||||
quality: Optional[str] = "fast" # "fast" (one-shot) | "cinematic" (reflect → adapt)
|
||||
glossary: Optional[List[dict]] = None # [{"source": "...", "target": "...", "note": "..."}]
|
||||
# Optional regional dialect (BCP-47, e.g. "es-AR", "pt-BR") — #280 item 2.
|
||||
# Applied by LLM-backed paths (provider="openai" or quality="cinematic"):
|
||||
# the prompt asks for that region's vocabulary/grammar (e.g. Argentinian
|
||||
# voseo: "vos sos" instead of "tú eres"). Non-LLM providers (Argos, NLLB,
|
||||
# Google) can't honor it; the response then carries dialect_applied=false.
|
||||
dialect: Optional[str] = None
|
||||
|
||||
class DubIngestUrlRequest(BaseModel):
|
||||
url: str
|
||||
|
||||
@@ -1,281 +0,0 @@
|
||||
"""Acoustic echo cancellation for dictate-over-playback (parity Action 8b).
|
||||
|
||||
When the user dictates while OmniVoice is *playing* audio (a TTS preview, a
|
||||
dub render, a video), the loudspeaker signal leaks back into the microphone.
|
||||
The streaming ASR on ``/ws/transcribe`` then transcribes that bleed as if it
|
||||
were speech — the "it typed back what the app just said" symptom. A browser's
|
||||
``getUserMedia({echoCancellation:true})`` would help, but its quality and even
|
||||
its availability differ per platform/webview, which would make a *default*
|
||||
feature behave differently on macOS/Windows/Linux — against the project's
|
||||
cross-platform-parity rule. A server-side canceller behaves identically
|
||||
everywhere, so it is the local-first, platform-neutral choice.
|
||||
|
||||
This is an NLMS (normalised least-mean-squares) time-domain adaptive filter
|
||||
with a Geigel double-talk detector. It is a clean-room-grade *port* of
|
||||
Patter's ``getpatter/audio/aec.py`` (MIT) — see docs/competitive-analysis.md,
|
||||
Action 8. It is NOT production-grade DSP (WebRTC AEC3 / Speex AEC are); it is
|
||||
a dependency-free, good-enough canceller that removes the steady-state echo
|
||||
the ASR would otherwise hallucinate on.
|
||||
|
||||
Wiring (one instance per ``/ws/transcribe`` session — NOT thread-safe)::
|
||||
|
||||
aec = NlmsEchoCanceller(sample_rate=16000)
|
||||
# Far-end: every PCM chunk the client is about to play through speakers.
|
||||
aec.push_far_end(playback_pcm_bytes)
|
||||
# Near-end: the mic PCM, cleaned before it reaches the ASR buffer.
|
||||
cleaned = aec.process_near_end(mic_pcm_bytes)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Final
|
||||
|
||||
import numpy as np
|
||||
|
||||
logger = logging.getLogger("omnivoice.aec")
|
||||
|
||||
|
||||
_DEFAULT_FILTER_TAPS: Final[int] = 512
|
||||
"""Adaptive-filter length in samples. 512 taps @ 16 kHz = 32 ms, covering a
|
||||
typical near-field laptop/desktop echo path. Longer tails (large rooms) can
|
||||
pass ``filter_taps=1024``+ at proportionally more CPU per frame; 512 converges
|
||||
in ~0.5 s with the warm-up ramp and is the sweet spot for dictation."""
|
||||
|
||||
_DEFAULT_STEP_SIZE: Final[float] = 0.1
|
||||
"""Steady-state NLMS step size. Larger = faster channel tracking but less
|
||||
stable; 0.1 is the textbook value for narrowband voice."""
|
||||
|
||||
_DEFAULT_WARMUP_STEP_SIZE: Final[float] = 0.5
|
||||
"""Aggressive step used during the warm-up window so the filter reaches a
|
||||
usable echo estimate within ~0.5 s instead of several seconds. The Geigel
|
||||
double-talk detector still gates updates, so the bigger step does not learn
|
||||
the user's own voice as echo."""
|
||||
|
||||
_DEFAULT_WARMUP_SECONDS: Final[float] = 0.5
|
||||
"""Length of the warm-up window. After this many seconds of processed
|
||||
near-end audio the step decays from ``warmup_step_size`` to ``step_size``."""
|
||||
|
||||
_DEFAULT_LEAKAGE: Final[float] = 0.9999
|
||||
"""Per-iteration weight leakage (slightly < 1) so the filter slowly forgets
|
||||
stale taps when the echo path drifts (the user moves the mic)."""
|
||||
|
||||
_DOUBLE_TALK_RHO: Final[float] = 0.6
|
||||
"""Geigel double-talk threshold. When ``max(|near|) > rho * max(|far|)`` the
|
||||
near-end carries energy the far-end cannot explain (the user is talking) →
|
||||
freeze adaptation so the filter does not model the user's voice as echo."""
|
||||
|
||||
_FAR_END_BUFFER_SECONDS: Final[float] = 0.5
|
||||
"""How much past far-end (playback) audio to retain. The echo arrives at the
|
||||
mic tens of ms after playback; the filter needs that much look-back to align.
|
||||
500 ms is generous headroom."""
|
||||
|
||||
|
||||
class NlmsEchoCanceller:
|
||||
"""Time-domain NLMS adaptive filter with Geigel double-talk detection.
|
||||
|
||||
Operates on narrowband mono PCM at 16 kHz (the rate the dictation path
|
||||
resamples to) or 8 kHz. Not thread-safe — each ``/ws/transcribe`` session
|
||||
owns its own instance.
|
||||
"""
|
||||
|
||||
# Far-end staleness window (seconds): once the most recent far-end push
|
||||
# is older than this, ``process_near_end`` passes the mic through instead
|
||||
# of cancelling against a frozen reference (which would superimpose the
|
||||
# same stale ~50 ms waveform on every mic frame as an audible buzz).
|
||||
_FAR_STALE_S: float = 0.25
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
sample_rate: int = 16000,
|
||||
*,
|
||||
filter_taps: int = _DEFAULT_FILTER_TAPS,
|
||||
step_size: float = _DEFAULT_STEP_SIZE,
|
||||
warmup_step_size: float = _DEFAULT_WARMUP_STEP_SIZE,
|
||||
warmup_seconds: float = _DEFAULT_WARMUP_SECONDS,
|
||||
leakage: float = _DEFAULT_LEAKAGE,
|
||||
double_talk_rho: float = _DOUBLE_TALK_RHO,
|
||||
) -> None:
|
||||
if sample_rate not in (8000, 16000):
|
||||
raise ValueError(
|
||||
"NlmsEchoCanceller supports 8000 Hz or 16000 Hz only; "
|
||||
f"got {sample_rate}."
|
||||
)
|
||||
if filter_taps < 64:
|
||||
raise ValueError(
|
||||
f"filter_taps must be >= 64 to model a meaningful echo path; "
|
||||
f"got {filter_taps}."
|
||||
)
|
||||
if not 0 < step_size <= 1:
|
||||
raise ValueError(f"step_size must be in (0, 1]; got {step_size}.")
|
||||
if not 0 < warmup_step_size <= 1:
|
||||
raise ValueError(
|
||||
f"warmup_step_size must be in (0, 1]; got {warmup_step_size}."
|
||||
)
|
||||
if warmup_seconds < 0:
|
||||
raise ValueError(f"warmup_seconds must be >= 0; got {warmup_seconds}.")
|
||||
if not 0 < leakage <= 1:
|
||||
raise ValueError(f"leakage must be in (0, 1]; got {leakage}.")
|
||||
|
||||
self._sample_rate = sample_rate
|
||||
self._taps = filter_taps
|
||||
self._step = float(step_size)
|
||||
self._warmup_step = float(warmup_step_size)
|
||||
self._warmup_samples = int(warmup_seconds * sample_rate)
|
||||
self._leakage = float(leakage)
|
||||
self._rho = float(double_talk_rho)
|
||||
# Counts near-end samples processed so the step can taper from
|
||||
# warmup_step to step over the first warmup_samples. Counted from the
|
||||
# first process_near_end call so the window aligns with playback start.
|
||||
self._processed_samples: int = 0
|
||||
self._last_far_push_monotonic: float | None = None
|
||||
|
||||
# Filter coefficients (zeros — adapts to the channel within ~0.5–2 s).
|
||||
self._w = np.zeros(filter_taps, dtype=np.float32)
|
||||
|
||||
# Far-end ring buffer holding >= filter_taps samples of playback
|
||||
# history, with headroom so push/process can interleave freely.
|
||||
max_buf_samples = max(
|
||||
filter_taps * 2,
|
||||
int(sample_rate * _FAR_END_BUFFER_SECONDS),
|
||||
)
|
||||
self._far_buf = np.zeros(max_buf_samples, dtype=np.float32)
|
||||
self._far_write_idx = 0 # next write position (head)
|
||||
self._far_filled = 0 # samples written so far (capped at len(far_buf))
|
||||
|
||||
# Diagnostics only — never read in the hot path.
|
||||
self.frames_processed: int = 0
|
||||
self.double_talk_frames: int = 0
|
||||
|
||||
# ── Public API ──────────────────────────────────────────────────────────
|
||||
|
||||
def push_far_end(self, pcm_bytes: bytes) -> None:
|
||||
"""Append far-end (playback) audio to the reference ring buffer.
|
||||
|
||||
Accepts raw int16 little-endian mono PCM at the configured rate.
|
||||
"""
|
||||
if not pcm_bytes:
|
||||
return
|
||||
self._last_far_push_monotonic = time.monotonic()
|
||||
samples = np.frombuffer(pcm_bytes, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
n = samples.shape[0]
|
||||
buf_len = self._far_buf.shape[0]
|
||||
if n >= buf_len:
|
||||
# More than the buffer holds — keep only the newest buf_len.
|
||||
self._far_buf[:] = samples[-buf_len:]
|
||||
self._far_write_idx = 0
|
||||
self._far_filled = buf_len
|
||||
return
|
||||
end = self._far_write_idx + n
|
||||
if end <= buf_len:
|
||||
self._far_buf[self._far_write_idx:end] = samples
|
||||
else:
|
||||
head = buf_len - self._far_write_idx
|
||||
self._far_buf[self._far_write_idx:] = samples[:head]
|
||||
self._far_buf[: n - head] = samples[head:]
|
||||
self._far_write_idx = (self._far_write_idx + n) % buf_len
|
||||
self._far_filled = min(self._far_filled + n, buf_len)
|
||||
|
||||
def process_near_end(self, pcm_bytes: bytes) -> bytes:
|
||||
"""Subtract the estimated echo from the near-end (mic) signal.
|
||||
|
||||
Returns int16 little-endian mono PCM with the estimated echo removed.
|
||||
Passes the frame through unchanged when there is nothing worth
|
||||
cancelling: no playback has been primed, or the far-end reference is
|
||||
stale (the app went silent).
|
||||
"""
|
||||
if not pcm_bytes:
|
||||
return pcm_bytes
|
||||
|
||||
# Not enough far-end history to fill the filter window yet — passing
|
||||
# through avoids emitting garbage on the first frames.
|
||||
if self._far_filled < self._taps:
|
||||
return pcm_bytes
|
||||
|
||||
# Far-end reference is stale (app stopped playing): the ring only
|
||||
# advances on push_far_end, so the "most recent" window is frozen at
|
||||
# the tail of the last playback. Convolving against it would buzz.
|
||||
last_push = self._last_far_push_monotonic
|
||||
if last_push is None or (time.monotonic() - last_push) > self._FAR_STALE_S:
|
||||
return pcm_bytes
|
||||
|
||||
near = np.frombuffer(pcm_bytes, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
cleaned = self._block_nlms(near)
|
||||
out = np.clip(cleaned * 32768.0, -32768.0, 32767.0).astype(np.int16)
|
||||
self.frames_processed += 1
|
||||
return out.tobytes()
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Clear filter coefficients and far-end history (e.g. on a new turn)."""
|
||||
self._w.fill(0)
|
||||
self._far_buf.fill(0)
|
||||
self._far_write_idx = 0
|
||||
self._far_filled = 0
|
||||
self._processed_samples = 0
|
||||
self._last_far_push_monotonic = None
|
||||
self.frames_processed = 0
|
||||
self.double_talk_frames = 0
|
||||
|
||||
# ── Internals ───────────────────────────────────────────────────────────
|
||||
|
||||
def _far_window(self, length: int) -> np.ndarray:
|
||||
"""Most recent ``length`` far-end samples, oldest first / newest last."""
|
||||
buf_len = self._far_buf.shape[0]
|
||||
if length > self._far_filled:
|
||||
length = self._far_filled
|
||||
end = self._far_write_idx # newest sample is at (end - 1) mod buf_len
|
||||
if end >= length:
|
||||
return self._far_buf[end - length: end]
|
||||
head = self._far_buf[buf_len - (length - end):]
|
||||
tail = self._far_buf[:end]
|
||||
return np.concatenate((head, tail))
|
||||
|
||||
def _block_nlms(self, near: np.ndarray) -> np.ndarray:
|
||||
"""Sample-by-sample NLMS over one frame of near-end samples.
|
||||
|
||||
Classical NLMS depends on the weights adapted at the previous sample,
|
||||
so the inner loop is sequential. Each sample is O(taps); numpy keeps a
|
||||
320-sample / 512-tap frame well under a millisecond on commodity CPUs.
|
||||
"""
|
||||
taps = self._taps
|
||||
far_window = self._far_window(taps + near.shape[0] - 1)
|
||||
if far_window.shape[0] < taps + near.shape[0] - 1:
|
||||
# Still warming up — left-pad with zeros so indices line up.
|
||||
pad = np.zeros(
|
||||
taps + near.shape[0] - 1 - far_window.shape[0], dtype=np.float32
|
||||
)
|
||||
far_window = np.concatenate((pad, far_window))
|
||||
|
||||
# Geigel double-talk detector (frame-wise).
|
||||
far_max = float(np.max(np.abs(far_window))) if far_window.size else 0.0
|
||||
near_max = float(np.max(np.abs(near)))
|
||||
# Freeze adaptation when the far reference is effectively silent
|
||||
# (<= -60 dBFS): adapting against a fade-out tail with near-zero norm
|
||||
# blows the weights up against user speech when playback resumes.
|
||||
if far_max <= 1e-3:
|
||||
return near
|
||||
double_talk = near_max > self._rho * far_max
|
||||
if double_talk:
|
||||
self.double_talk_frames += 1
|
||||
|
||||
out = np.empty_like(near)
|
||||
w = self._w
|
||||
leakage = self._leakage
|
||||
# Constant step within the frame keeps the inner loop branch-free.
|
||||
if self._processed_samples < self._warmup_samples:
|
||||
step = self._warmup_step
|
||||
else:
|
||||
step = self._step
|
||||
for i in range(near.shape[0]):
|
||||
x = far_window[i: i + taps]
|
||||
y_est = float(np.dot(w, x))
|
||||
e = float(near[i] - y_est)
|
||||
out[i] = e
|
||||
if not double_talk:
|
||||
# NLMS update with leakage. +1e-6 guards divide-by-zero.
|
||||
norm = float(np.dot(x, x)) + 1e-6
|
||||
w *= leakage
|
||||
w += (step * e / norm) * x
|
||||
self._processed_samples += near.shape[0]
|
||||
return out
|
||||
+21
-358
@@ -31,91 +31,12 @@ from abc import ABC, abstractmethod
|
||||
logger = logging.getLogger("omnivoice.asr")
|
||||
|
||||
|
||||
def _compute_type_candidates(device: str) -> list[str]:
|
||||
"""Per-device compute_type fallback chain. int8 is supported by every
|
||||
CTranslate2 CUDA+CPU build; float16/int8_float16 only on GPUs with efficient
|
||||
fp16 — so degrade rather than crash (#551). Honors an ASR_COMPUTE_TYPE env
|
||||
override (power users on exotic hardware can pin int8/float32)."""
|
||||
import os
|
||||
override = os.environ.get("ASR_COMPUTE_TYPE")
|
||||
if override:
|
||||
return [override]
|
||||
return ["float16", "int8_float16", "int8"] if device == "cuda" else ["int8", "float32"]
|
||||
|
||||
|
||||
def _is_compute_type_error(msg: str) -> bool:
|
||||
low = msg.lower()
|
||||
return "compute type" in low or "efficient float16" in low
|
||||
|
||||
|
||||
def _decode_audio_16k_mono(audio_path: str):
|
||||
"""Decode `audio_path` to a 16 kHz mono float32 waveform using OmniVoice's
|
||||
*validated* ffmpeg, instead of whisperx.load_audio's bare ``"ffmpeg"`` PATH
|
||||
lookup.
|
||||
|
||||
whisperx (and openai-whisper) shell out to a literal ``"ffmpeg"`` resolved
|
||||
against the OS PATH. On Windows that resolves to whatever the system finds
|
||||
first — a WindowsApps alias stub or a corrupt/wrong-arch download — which
|
||||
passes `which` but explodes at spawn with ``[WinError 193] %1 is not a valid
|
||||
Win32 application``. whisperx only catches `CalledProcessError`, so the
|
||||
spawn-time `OSError` escapes and the dub/batch path reports the opaque
|
||||
"Transcription produced no segments" (#479). ``find_ffmpeg()`` probes each
|
||||
candidate with ``-version`` and returns a runnable binary (the bundled
|
||||
imageio-ffmpeg / Tauri sidecar) — or None, so we can raise an actionable
|
||||
error. This also fixes the imageio case a PATH-prepend can't: its binary is
|
||||
named ``ffmpeg-<plat>-vN.exe``, not ``ffmpeg``, so bare lookup never finds
|
||||
it. Mirrors whisperx.audio.load_audio's command exactly (16 kHz, mono, s16le).
|
||||
"""
|
||||
import subprocess
|
||||
|
||||
import numpy as np
|
||||
|
||||
from services.ffmpeg_utils import find_ffmpeg
|
||||
|
||||
ffmpeg = find_ffmpeg()
|
||||
if not ffmpeg:
|
||||
raise RuntimeError(
|
||||
"Cannot transcribe: ffmpeg is missing or not runnable. Install "
|
||||
"ffmpeg (or let OmniVoice's bundled binary download), then retry. "
|
||||
"On Windows a '[WinError 193]' here means the ffmpeg binary is "
|
||||
"corrupt or the wrong architecture — reinstall it or clear the "
|
||||
"imageio-ffmpeg cache."
|
||||
)
|
||||
cmd = [
|
||||
ffmpeg, "-nostdin", "-threads", "0", "-i", audio_path,
|
||||
"-f", "s16le", "-ac", "1", "-acodec", "pcm_s16le", "-ar", "16000", "-",
|
||||
]
|
||||
try:
|
||||
out = subprocess.run(cmd, capture_output=True, check=True).stdout
|
||||
except OSError as e:
|
||||
# Belt-and-suspenders: find_ffmpeg() already -version-validated this
|
||||
# binary, so a WinError 193 here is unexpected — surface it clearly
|
||||
# rather than letting it become "no segments".
|
||||
raise RuntimeError(
|
||||
f"ffmpeg at {ffmpeg!r} could not be executed ({e}). Reinstall "
|
||||
"ffmpeg or clear the imageio-ffmpeg cache."
|
||||
) from e
|
||||
except subprocess.CalledProcessError as e:
|
||||
stderr = (e.stderr or b"").decode(errors="replace")[:500]
|
||||
raise RuntimeError(f"Failed to decode audio for transcription: {stderr}") from e
|
||||
return np.frombuffer(out, np.int16).flatten().astype(np.float32) / 32768.0
|
||||
|
||||
|
||||
# ── Protocol ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class ASRBackend(ABC):
|
||||
id: str = "base"
|
||||
display_name: str = "Base ASR"
|
||||
# Accelerator families this backend can use, in preference order; always
|
||||
# includes a fallback. Subset of {cuda, rocm, mps, xpu, cpu}. Mirrors the
|
||||
# TTSBackend.gpu_compat contract so engine_routing.resolve_routing() can
|
||||
# surface the effective device per host (no silent CPU fallback). The
|
||||
# conservative default is CPU-only; subclasses declare what they really run
|
||||
# on. (ROCm is intentionally NOT claimed yet for any ASR engine — see the
|
||||
# per-engine notes; an unverified `rocm` claim would route ROCm hosts to a
|
||||
# broken GPU path, strictly worse than the honest `cpu_fallback`.)
|
||||
gpu_compat: tuple[str, ...] = ("cpu",)
|
||||
|
||||
@classmethod
|
||||
@abstractmethod
|
||||
@@ -140,10 +61,6 @@ class ASRBackend(ABC):
|
||||
class WhisperXBackend(ASRBackend):
|
||||
id = "whisperx"
|
||||
display_name = "WhisperX (faster-whisper + wav2vec2 forced alignment)"
|
||||
# CTranslate2 backend: CUDA fp16 or CPU int8 (see _pick_device). ROCm not
|
||||
# claimed — CTranslate2 has no upstream HIP build, so a ROCm host honestly
|
||||
# gets cpu_fallback rather than a false GPU promise.
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
def __init__(self):
|
||||
self._model_name = os.environ.get("ASR_MODEL_WHISPERX", "large-v3")
|
||||
@@ -206,40 +123,7 @@ class WhisperXBackend(ASRBackend):
|
||||
# vad_method="silero" is the default; keep it so short gaps
|
||||
# get cleaned up before transcription.
|
||||
)
|
||||
except (ValueError, RuntimeError) as e:
|
||||
# #551: GPUs without efficient fp16 (older Maxwell/Pascal, GTX 16xx)
|
||||
# or a CTranslate2/cuDNN binary mismatch raise a *ValueError*
|
||||
# ("Requested float16 compute type, but the target device or backend
|
||||
# do not support efficient float16 computation") at load — not an
|
||||
# OOM, not a RuntimeError. Retry on the SAME device with the next
|
||||
# compute_type candidate (cuda: int8_float16 → int8) before touching
|
||||
# the OOM→CPU path, so we degrade rather than crash every chunk.
|
||||
if _is_compute_type_error(str(e)):
|
||||
candidates = _compute_type_candidates(self._device)
|
||||
try:
|
||||
nxt = candidates[candidates.index(self._compute_type) + 1:]
|
||||
except ValueError:
|
||||
nxt = [c for c in candidates if c != self._compute_type]
|
||||
for ct in nxt:
|
||||
logger.warning(
|
||||
"whisperx %s unsupported on %s — retrying with %s. Detail: %s",
|
||||
self._compute_type, self._device, ct, e,
|
||||
)
|
||||
self._compute_type = ct
|
||||
try:
|
||||
self._asr = whisperx.load_model(
|
||||
self._model_name,
|
||||
device=self._device,
|
||||
compute_type=self._compute_type,
|
||||
)
|
||||
return
|
||||
except (ValueError, RuntimeError) as e2:
|
||||
if _is_compute_type_error(str(e2)):
|
||||
e = e2
|
||||
continue
|
||||
raise
|
||||
# Exhausted compute-type candidates on this device — re-raise.
|
||||
raise
|
||||
except RuntimeError as e:
|
||||
# CUDA OOM: a resident TTS model + the GPU worker pool can starve
|
||||
# VRAM on small (e.g. 8 GB laptop) GPUs, so loading large-v3 on
|
||||
# CUDA dies here — which previously surfaced as a bare 500 from
|
||||
@@ -429,13 +313,10 @@ class WhisperXBackend(ASRBackend):
|
||||
return None
|
||||
|
||||
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
|
||||
import whisperx # used for whisperx.align() below
|
||||
import whisperx
|
||||
self._ensure_asr()
|
||||
logger.info("whisperx transcribing %s (word_timestamps=%s)", audio_path, word_timestamps)
|
||||
# Decode via OmniVoice's validated ffmpeg, NOT whisperx.load_audio's bare
|
||||
# "ffmpeg" PATH lookup which yields [WinError 193] -> "no segments" on
|
||||
# Windows (#479). Same 16 kHz mono s16le array whisperx expects.
|
||||
audio = _decode_audio_16k_mono(audio_path)
|
||||
audio = whisperx.load_audio(audio_path)
|
||||
try:
|
||||
result = self._asr.transcribe(audio)
|
||||
except IndexError:
|
||||
@@ -501,8 +382,6 @@ class WhisperXBackend(ASRBackend):
|
||||
class FasterWhisperBackend(ASRBackend):
|
||||
id = "faster-whisper"
|
||||
display_name = "Faster-Whisper (CTranslate2 — Linux/Windows/macOS)"
|
||||
# CTranslate2: CUDA or CPU (no upstream ROCm/HIP build — see WhisperX note).
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
def __init__(self):
|
||||
# Defaulting to the CTranslate2-converted large-v3 repo. Matches
|
||||
@@ -512,10 +391,6 @@ class FasterWhisperBackend(ASRBackend):
|
||||
"ASR_MODEL_FASTER", "Systran/faster-whisper-large-v3"
|
||||
)
|
||||
self._model = None # lazy — first transcribe() loads weights
|
||||
# Set by _ensure_model() to the device/compute_type that actually loaded
|
||||
# (after the #551 compute_type / #255 OOM→CPU fallback chain).
|
||||
self._device: str | None = None
|
||||
self._compute_type: str | None = None
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
@@ -544,58 +419,9 @@ class FasterWhisperBackend(ASRBackend):
|
||||
"faster-whisper loading %s on %s (%s)",
|
||||
self._model_name, device, compute_type,
|
||||
)
|
||||
# Try the per-device compute_type chain (cuda: float16 → int8_float16 →
|
||||
# int8; cpu: int8 → float32). A GPU without efficient fp16 (older
|
||||
# Maxwell/Pascal, GTX 16xx, or a CTranslate2/cuDNN mismatch) raises a
|
||||
# *ValueError* at construction (#551) — degrade to the next candidate
|
||||
# instead of failing every chunk. A genuine CUDA OOM falls back to CPU
|
||||
# (slower, same model/accuracy), preserving the existing #255 behaviour.
|
||||
candidates = _compute_type_candidates(device)
|
||||
if compute_type in candidates:
|
||||
candidates = candidates[candidates.index(compute_type):]
|
||||
last_err: Exception | None = None
|
||||
while True:
|
||||
for ct in candidates:
|
||||
try:
|
||||
self._model = WhisperModel(
|
||||
self._model_name, device=device, compute_type=ct
|
||||
)
|
||||
self._device, self._compute_type = device, ct
|
||||
return
|
||||
except (ValueError, RuntimeError) as e:
|
||||
last_err = e
|
||||
if _is_compute_type_error(str(e)):
|
||||
logger.warning(
|
||||
"faster-whisper %s unsupported on %s — trying next "
|
||||
"compute_type. Detail: %s", ct, device, e,
|
||||
)
|
||||
continue
|
||||
if device == "cuda" and "out of memory" in str(e).lower():
|
||||
# Stop scanning GPU candidates; fall back to CPU below.
|
||||
break
|
||||
raise
|
||||
# Exhausted candidates for this device. If we were on CUDA and the
|
||||
# last failure was an OOM, retry on CPU with its candidates (#255).
|
||||
if device == "cuda" and last_err is not None and (
|
||||
"out of memory" in str(last_err).lower()
|
||||
):
|
||||
logger.warning(
|
||||
"faster-whisper CUDA OOM loading %s — retrying on CPU "
|
||||
"(slower). Free VRAM (Flush the TTS model) for GPU-speed "
|
||||
"ASR. Detail: %s", self._model_name, last_err,
|
||||
)
|
||||
try:
|
||||
import torch
|
||||
torch.cuda.empty_cache()
|
||||
except Exception: # noqa: BLE001 — cache clear is best-effort
|
||||
pass
|
||||
device = "cpu"
|
||||
candidates = _compute_type_candidates(device)
|
||||
compute_type = candidates[0]
|
||||
continue
|
||||
# All candidates exhausted (and no OOM→CPU retry available) — surface
|
||||
# the last error.
|
||||
raise last_err
|
||||
self._model = WhisperModel(
|
||||
self._model_name, device=device, compute_type=compute_type
|
||||
)
|
||||
|
||||
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
|
||||
self._ensure_model()
|
||||
@@ -671,7 +497,6 @@ _MLX_MODEL_TURBO = "mlx-community/whisper-large-v3-turbo"
|
||||
class MLXWhisperBackend(ASRBackend):
|
||||
id = "mlx-whisper"
|
||||
display_name = "MLX Whisper (Apple Silicon CoreML)"
|
||||
gpu_compat = ("mps", "cpu")
|
||||
|
||||
def __init__(self, model_name: str | None = None):
|
||||
self._model_name = model_name or os.environ.get(
|
||||
@@ -680,23 +505,14 @@ class MLXWhisperBackend(ASRBackend):
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
# #390: shared platform gate FIRST — one rule for MLX-Audio + MLX-Whisper.
|
||||
# Returns False on Linux/Windows/mac-Intel before any package import, so
|
||||
# a stray mlx-whisper wheel never reports available or advertises `mps`.
|
||||
from core.device_caps import mlx_supported
|
||||
ok, why = mlx_supported()
|
||||
if not ok:
|
||||
return False, why
|
||||
try:
|
||||
import torch
|
||||
if not (hasattr(torch.backends, "mps") and torch.backends.mps.is_available()):
|
||||
return False, "Apple Silicon (MPS) not available."
|
||||
import mlx_whisper # noqa: F401
|
||||
return True, "ready"
|
||||
# Catch OSError/RuntimeError too, not just ImportError: in a
|
||||
# PyInstaller bundle mlx's native dylib/metallib can fail to load
|
||||
# even when the package imports, raising OSError/RuntimeError. We must
|
||||
# report unavailable (so the picker falls back) rather than crash the
|
||||
# registry scan (Wave 4.4).
|
||||
except (ImportError, OSError, RuntimeError) as e:
|
||||
return False, f"mlx-whisper unavailable: {e}"
|
||||
except ImportError as e:
|
||||
return False, f"mlx-whisper not installed: {e}"
|
||||
|
||||
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
|
||||
import mlx_whisper
|
||||
@@ -745,9 +561,6 @@ class MLXWhisperBackend(ASRBackend):
|
||||
class PyTorchWhisperBackend(ASRBackend):
|
||||
id = "pytorch-whisper"
|
||||
display_name = "PyTorch Whisper (CUDA / CPU via transformers pipeline)"
|
||||
# Pure transformers pipeline → runs wherever torch does (CUDA, MPS, CPU).
|
||||
# ROCm-via-HIP would also work but is left unclaimed pending verification.
|
||||
gpu_compat = ("cuda", "mps", "cpu")
|
||||
|
||||
def __init__(self, asr_pipe=None):
|
||||
# Reuses the `_asr_pipe` attached to the TTS model when available.
|
||||
@@ -784,25 +597,12 @@ class PyTorchWhisperBackend(ASRBackend):
|
||||
"PyTorchWhisperBackend: loading standalone ASR pipeline %s on %s",
|
||||
model_name, device,
|
||||
)
|
||||
try:
|
||||
self._pipe = hf_pipeline(
|
||||
"automatic-speech-recognition",
|
||||
model=model_name,
|
||||
dtype=asr_dtype,
|
||||
device_map=device,
|
||||
)
|
||||
except Exception as e:
|
||||
# #549: an incomplete transformers install fails to build the ASR
|
||||
# pipeline (e.g. "Could not import module 'AutoFeatureExtractor'").
|
||||
# The raw error is opaque; re-raise with an actionable next step so
|
||||
# the toast tells the user how to recover instead of "no segments".
|
||||
raise RuntimeError(
|
||||
"transformers ASR pipeline failed to import (AutoFeatureExtractor) "
|
||||
"— your transformers install is incomplete; reinstall with "
|
||||
"`uv pip install --reinstall transformers`, or use faster-whisper "
|
||||
"(OmniVoice's default ASR) which avoids the transformers pipeline. "
|
||||
f"Underlying: {e}"
|
||||
) from e
|
||||
self._pipe = hf_pipeline(
|
||||
"automatic-speech-recognition",
|
||||
model=model_name,
|
||||
dtype=asr_dtype,
|
||||
device_map=device,
|
||||
)
|
||||
|
||||
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
|
||||
import soundfile as sf
|
||||
@@ -833,11 +633,6 @@ class NeMoASRBackend(ASRBackend):
|
||||
Requires NVIDIA GPU.
|
||||
"""
|
||||
id = "nemo-parakeet"
|
||||
# CUDA-only: is_available() hard-fails without a GPU ("Parakeet TDT requires
|
||||
# NVIDIA GPU (CUDA)"), so declaring a CPU path would be a false claim. On a
|
||||
# CPU host this correctly resolves to routing_status="unavailable", matching
|
||||
# is_available()=False (the matrix suppresses the routing badge there).
|
||||
gpu_compat = ("cuda",)
|
||||
display_name = "Parakeet TDT (NVIDIA NeMo — English SOTA)"
|
||||
|
||||
def __init__(self):
|
||||
@@ -946,7 +741,6 @@ class MoonshineASRBackend(ASRBackend):
|
||||
Great for live capture and CPU-only environments.
|
||||
"""
|
||||
id = "moonshine"
|
||||
gpu_compat = ("cpu",) # edge/CPU-optimized by design
|
||||
display_name = "Moonshine (edge-optimized, ONNX)"
|
||||
|
||||
def __init__(self):
|
||||
@@ -1091,7 +885,6 @@ class FunASRBackend(ASRBackend):
|
||||
#182); WhisperX remains the cross-platform default.
|
||||
"""
|
||||
id = "funasr"
|
||||
gpu_compat = ("cuda", "cpu") # FunASR: CUDA or CPU
|
||||
display_name = "FunASR (SenseVoice — 50+ languages, all-in-one)"
|
||||
|
||||
def __init__(self):
|
||||
@@ -1138,47 +931,7 @@ class FunASRBackend(ASRBackend):
|
||||
pass
|
||||
|
||||
|
||||
def _isolated_faster_whisper():
|
||||
"""Lazy import so the subprocess_asr → subprocess_backend chain isn't
|
||||
pulled in at registry definition time."""
|
||||
from services.subprocess_asr import IsolatedFasterWhisperBackend
|
||||
return IsolatedFasterWhisperBackend
|
||||
|
||||
|
||||
class _LazyASRRegistry(dict):
|
||||
"""Registry with one lazily-resolved entry (Wave 4.2). Mirrors the TTS
|
||||
registry's lazy pattern so listing/selecting the crash-isolated ASR
|
||||
backend doesn't import the subprocess stack unless it's used."""
|
||||
|
||||
_LAZY = {"faster-whisper-isolated": _isolated_faster_whisper}
|
||||
|
||||
def __contains__(self, key):
|
||||
return dict.__contains__(self, key) or key in self._LAZY
|
||||
|
||||
def __getitem__(self, key):
|
||||
if dict.__contains__(self, key):
|
||||
return dict.__getitem__(self, key)
|
||||
if key in self._LAZY:
|
||||
cls = self._LAZY[key]()
|
||||
self[key] = cls
|
||||
return cls
|
||||
raise KeyError(key)
|
||||
|
||||
def __iter__(self):
|
||||
seen = set()
|
||||
for k in dict.__iter__(self):
|
||||
seen.add(k)
|
||||
yield k
|
||||
for k in self._LAZY:
|
||||
if k not in seen:
|
||||
yield k
|
||||
|
||||
def items(self):
|
||||
for k in self:
|
||||
yield k, self[k]
|
||||
|
||||
|
||||
_REGISTRY: dict[str, type[ASRBackend]] = _LazyASRRegistry({
|
||||
_REGISTRY: dict[str, type[ASRBackend]] = {
|
||||
"whisperx": WhisperXBackend,
|
||||
"faster-whisper": FasterWhisperBackend,
|
||||
"mlx-whisper": MLXWhisperBackend,
|
||||
@@ -1186,70 +939,18 @@ _REGISTRY: dict[str, type[ASRBackend]] = _LazyASRRegistry({
|
||||
"nemo-parakeet": NeMoASRBackend,
|
||||
"moonshine": MoonshineASRBackend,
|
||||
"funasr": FunASRBackend,
|
||||
# "faster-whisper-isolated": resolved lazily (crash-isolated subprocess).
|
||||
})
|
||||
|
||||
|
||||
# Short install hints surfaced as tooltips on the Settings → Engines UI
|
||||
# (parity with tts_backend._INSTALL_HINTS).
|
||||
_INSTALL_HINTS: dict[str, str] = {
|
||||
"whisperx": "pip install whisperx (CTranslate2 + wav2vec2 alignment; CUDA or CPU)",
|
||||
"faster-whisper": "pip install faster-whisper (CTranslate2; cross-platform, CUDA or CPU)",
|
||||
"mlx-whisper": "pip install mlx-whisper (Apple Silicon only)",
|
||||
"pytorch-whisper": "Bundled with transformers — no extra install (CUDA/MPS/CPU)",
|
||||
"nemo-parakeet": "pip install nemo_toolkit[asr] (NVIDIA Parakeet; CUDA or CPU)",
|
||||
"moonshine": "pip install useful-moonshine (edge/CPU-optimized ASR)",
|
||||
"funasr": "pip install funasr (SenseVoiceSmall + FSMN-VAD; CUDA or CPU)",
|
||||
}
|
||||
|
||||
# Most-recent failure per backend, so a transient probe error survives between
|
||||
# Settings refreshes (parity with tts_backend._LAST_ERRORS).
|
||||
_LAST_ERRORS: dict[str, str] = {}
|
||||
|
||||
|
||||
def list_backends() -> list[dict]:
|
||||
"""Enumerate every ASR backend with the **same 11-key shape as TTS** so the
|
||||
Engine Compatibility Matrix renders all families uniformly.
|
||||
|
||||
Per-entry: id, display_name, available, reason (scrubbed), install_hint,
|
||||
last_error, isolation_mode, gpu_compat, effective_device, routing_status,
|
||||
routing_reason. A backend whose ``is_available()`` raises is reported
|
||||
``available: false`` (never a 500), exactly like TTS.
|
||||
"""
|
||||
from core.device_caps import detect_host_caps
|
||||
from core.scrub import scrub_text
|
||||
from services.engine_routing import routing_fields
|
||||
caps = detect_host_caps()
|
||||
|
||||
out: list[dict] = []
|
||||
out = []
|
||||
for bid, cls in _REGISTRY.items():
|
||||
try:
|
||||
ok, msg = cls.is_available()
|
||||
except Exception as exc:
|
||||
ok = False
|
||||
msg = f"{type(exc).__name__}: {exc}"
|
||||
logger.warning(
|
||||
"asr list_backends: %s.is_available() raised — degrading "
|
||||
"gracefully so the picker still renders: %s", bid, msg,
|
||||
)
|
||||
if ok:
|
||||
_LAST_ERRORS.pop(bid, None)
|
||||
else:
|
||||
_LAST_ERRORS[bid] = scrub_text(msg)
|
||||
isolation = "subprocess" if getattr(cls, "_is_subprocess_isolated", False) else "in-process"
|
||||
gpu_compat = getattr(cls, "gpu_compat", ("cpu",))
|
||||
ok, msg = cls.is_available()
|
||||
out.append({
|
||||
"id": bid,
|
||||
"display_name": cls.display_name,
|
||||
"available": ok,
|
||||
# ASR previously emitted `reason` UNMASKED — scrub it now (closes a
|
||||
# pre-existing token-leak gap, matching TTS's redaction guarantee).
|
||||
"reason": None if ok else scrub_text(msg),
|
||||
"install_hint": _INSTALL_HINTS.get(bid),
|
||||
"last_error": _LAST_ERRORS.get(bid),
|
||||
"isolation_mode": isolation,
|
||||
"gpu_compat": list(gpu_compat),
|
||||
**routing_fields(gpu_compat, caps),
|
||||
"reason": None if ok else msg,
|
||||
})
|
||||
return out
|
||||
|
||||
@@ -1315,44 +1016,6 @@ def get_active_asr_backend(*, asr_pipe=None) -> ASRBackend:
|
||||
return _REGISTRY[bid]()
|
||||
|
||||
|
||||
def transcribe_reference(audio_path: str) -> str | None:
|
||||
"""Transcribe a voice-clone reference clip with the active ASR backend.
|
||||
|
||||
Voice cloning without a user-supplied transcript used to fall through to
|
||||
``OmniVoice.load_asr_model()`` — a transformers ``pipeline()`` load of
|
||||
whisper-large-v3-turbo that fails outright on transformers 5.3 (#308),
|
||||
even when whisperx / faster-whisper / mlx-whisper are installed and
|
||||
working. Route the reference transcript through the registry instead, so
|
||||
the model-attached pipeline is only reached when it is genuinely the last
|
||||
resort. Returns ``None`` on any failure — callers pass ``ref_text=None``
|
||||
through and the model's built-in fallback still gets its chance.
|
||||
"""
|
||||
try:
|
||||
backend = get_active_asr_backend()
|
||||
except Exception as e: # noqa: BLE001 — never let ASR break generation
|
||||
logger.warning("transcribe_reference: no ASR backend available (%s)", e)
|
||||
return None
|
||||
if isinstance(backend, PyTorchWhisperBackend):
|
||||
# The registry fell through to the model-attached pipeline; let the
|
||||
# model load it lazily rather than constructing a second copy here.
|
||||
return None
|
||||
try:
|
||||
result = backend.transcribe(audio_path, word_timestamps=False)
|
||||
except Exception as e: # noqa: BLE001 — degrade to the model fallback
|
||||
logger.warning(
|
||||
"transcribe_reference: %s failed (%s) — deferring to the model's "
|
||||
"built-in ASR fallback",
|
||||
backend.id, e,
|
||||
)
|
||||
return None
|
||||
result = result or {}
|
||||
text = result.get("text") or " ".join(
|
||||
(seg.get("text") or "").strip() for seg in result.get("segments", [])
|
||||
)
|
||||
text = (text or "").strip()
|
||||
return text or None
|
||||
|
||||
|
||||
_capture_backend: ASRBackend | None = None
|
||||
|
||||
|
||||
|
||||
@@ -1,174 +0,0 @@
|
||||
"""Audiobook creator — chapterized long-form narration (parity Wave 5).
|
||||
|
||||
Turns a chapter-delimited script into a chapterized audiobook. This module is
|
||||
the engine-agnostic core:
|
||||
|
||||
* ``parse_audiobook_script`` — pure parser: Markdown ``# H1`` headings become
|
||||
chapters; inline ``[voice:NAME]`` switches the narrator; ``[pause …]`` is
|
||||
delegated to the existing :func:`omnivoice.utils.text.parse_pause_markers`
|
||||
so audiobooks and single-shot synthesis share one pause dialect.
|
||||
* ``synthesize_chapter`` — orchestration: renders a chapter's spans through an
|
||||
injected ``synth(text, voice_id) -> tensor`` callable (reusing the
|
||||
``chunked_tts`` splitter + crossfade), stitching the inter-span silences.
|
||||
Injecting the synth keeps this unit-testable with a stub backend (no torch
|
||||
model, no GPU).
|
||||
* ``build_chapter_ffmetadata`` / ``build_m4b_cmd`` — pure builders for the
|
||||
ffmpeg chapterized-m4b mux (FFMETADATA1 ``[CHAPTER]`` blocks + concat-demux
|
||||
argv). The actual ffmpeg run lives in the (impure) caller.
|
||||
|
||||
Scope (first cut): plain chapter-delimited text/Markdown input. epub/pdf
|
||||
ingestion, the streaming synth job + UI are deferred follow-ups.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Callable, Optional
|
||||
|
||||
@dataclass
|
||||
class Span:
|
||||
"""One contiguous run of text in a single voice, plus trailing silence.
|
||||
|
||||
``speed`` (when set) is the per-span rate passed to the engine — Stories'
|
||||
per-line speed slider rides through here so the shared server render honours
|
||||
it the way the old client export did.
|
||||
"""
|
||||
voice_id: Optional[str]
|
||||
text: str
|
||||
pause_ms_after: int = 0
|
||||
speed: Optional[float] = None
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {"voice_id": self.voice_id, "text": self.text,
|
||||
"pause_ms_after": self.pause_ms_after, "speed": self.speed}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Chapter:
|
||||
title: str
|
||||
spans: list[Span] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def char_count(self) -> int:
|
||||
return sum(len(s.text) for s in self.spans)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {"title": self.title, "char_count": self.char_count,
|
||||
"spans": [s.to_dict() for s in self.spans]}
|
||||
|
||||
|
||||
@dataclass
|
||||
class AudiobookPlan:
|
||||
chapters: list[Chapter] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def char_count(self) -> int:
|
||||
return sum(c.char_count for c in self.chapters)
|
||||
|
||||
@property
|
||||
def chapter_count(self) -> int:
|
||||
return len(self.chapters)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"chapters": [c.to_dict() for c in self.chapters],
|
||||
"chapter_count": self.chapter_count,
|
||||
"char_count": self.char_count,
|
||||
}
|
||||
|
||||
|
||||
def parse_audiobook_script(text: str, *, default_voice: Optional[str] = None) -> AudiobookPlan:
|
||||
"""Parse a chapter-delimited script into an :class:`AudiobookPlan`.
|
||||
|
||||
Thin wrapper over the canonical :func:`services.longform_parser.
|
||||
parse_script_to_spans` (the single grammar source of truth, #27); wraps its
|
||||
span dicts in the ``Span``/``Chapter``/``AudiobookPlan`` dataclasses so the
|
||||
four router call sites and ``.to_dict()`` shape are unchanged.
|
||||
"""
|
||||
from services.longform_parser import parse_script_to_spans
|
||||
|
||||
chapters = [
|
||||
Chapter(title=c["title"], spans=[Span(**s) for s in c["spans"]])
|
||||
for c in parse_script_to_spans(text, default_voice=default_voice)
|
||||
]
|
||||
return AudiobookPlan(chapters=chapters)
|
||||
|
||||
|
||||
def synthesize_chapter(
|
||||
spans: list[Span],
|
||||
synth: Callable[[str, Optional[str], Optional[float]], "object"],
|
||||
sample_rate: int,
|
||||
*,
|
||||
crossfade_ms: int = 50,
|
||||
lexicon: Optional[dict] = None,
|
||||
):
|
||||
"""Render a chapter's spans to one waveform via an injected ``synth``.
|
||||
|
||||
``synth(text, voice_id, speed)`` returns a 1-D float32 audio tensor for a
|
||||
span of text in the given voice (``speed`` may be ``None`` for the engine
|
||||
default). Long spans are split with the ``chunked_tts`` splitter and
|
||||
crossfaded; inter-span ``pause_ms_after`` becomes silence. ``lexicon`` (when
|
||||
given) respells each span's text before chunking so the engine pronounces
|
||||
tricky words correctly; a ``None``/empty lexicon is a no-op pass-through.
|
||||
|
||||
Returns ``(audio_tensor, duration_seconds)``. torch + chunked_tts are
|
||||
imported lazily so this module stays import-light for the pure parser path.
|
||||
"""
|
||||
import torch
|
||||
from services.chunked_tts import concatenate_audio_chunks, split_text_into_chunks
|
||||
from services.pronunciation import apply_lexicon
|
||||
|
||||
parts: list = []
|
||||
for span in spans:
|
||||
if span.text:
|
||||
chunks = split_text_into_chunks(apply_lexicon(span.text, lexicon))
|
||||
rendered = [synth(c, span.voice_id, span.speed) for c in chunks]
|
||||
rendered = [r for r in rendered if r is not None and getattr(r, "numel", lambda: 0)()]
|
||||
if len(rendered) == 1:
|
||||
parts.append(rendered[0])
|
||||
elif rendered:
|
||||
parts.append(concatenate_audio_chunks(rendered, sample_rate, crossfade_ms=crossfade_ms))
|
||||
if span.pause_ms_after > 0:
|
||||
n = int(sample_rate * span.pause_ms_after / 1000.0)
|
||||
if n > 0:
|
||||
parts.append(torch.zeros(n, dtype=torch.float32))
|
||||
|
||||
if not parts:
|
||||
return torch.zeros(0, dtype=torch.float32), 0.0
|
||||
# Hard-concat spans + silences (crossfading silence would bleed the gap).
|
||||
audio = parts[0] if len(parts) == 1 else concatenate_audio_chunks(parts, sample_rate, crossfade_ms=0)
|
||||
return audio, audio.shape[-1] / float(sample_rate)
|
||||
|
||||
|
||||
# ── ffmpeg / metadata builders ──────────────────────────────────────────────
|
||||
#
|
||||
# These now live in the shared ``longform_render`` core (Stories + Audiobook
|
||||
# converge on one mux). The thin wrappers below preserve the original
|
||||
# audiobook-only call sites/signatures; new callers should use
|
||||
# ``longform_render`` directly to reach global metadata, cover art, loudness,
|
||||
# and mp3 output.
|
||||
from services.longform_render import ( # noqa: E402
|
||||
build_concat_list,
|
||||
build_ffmetadata,
|
||||
build_render_cmd,
|
||||
)
|
||||
|
||||
|
||||
def build_chapter_ffmetadata(chapters: list[tuple[str, int]]) -> str:
|
||||
"""Backward-compatible alias: chapters-only FFMETADATA (no global tags)."""
|
||||
return build_ffmetadata(chapters)
|
||||
|
||||
|
||||
def build_m4b_cmd(
|
||||
ffmpeg: str,
|
||||
concat_list_path: str,
|
||||
metadata_path: str,
|
||||
out_path: str,
|
||||
*,
|
||||
bitrate: str = "128k",
|
||||
) -> list[str]:
|
||||
"""Backward-compatible alias: a chapterized faststart m4b, no cover/loudness."""
|
||||
return build_render_cmd(
|
||||
ffmpeg, concat_list_path, metadata_path, out_path,
|
||||
fmt="m4b", bitrate=bitrate,
|
||||
)
|
||||
@@ -151,9 +151,6 @@ async def generate_segments_batched(
|
||||
# Raw: skip all DSP — return raw model output
|
||||
return audio_out
|
||||
|
||||
# TODO(#312): this route runs the OmniVoice model directly (not the active
|
||||
# backend), so VoxCPM2 never reaches it. When these routes become
|
||||
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
|
||||
mastered = apply_mastering(audio_out, sample_rate=sr)
|
||||
effect_chain = get_effect_chain(seg_effect_preset)
|
||||
if effect_chain:
|
||||
|
||||
@@ -1,174 +0,0 @@
|
||||
"""Chunked TTS generation utilities (Wave 1.2 — unlimited-length generation).
|
||||
|
||||
Adapted from voicebox (https://github.com/jamiepine/voicebox), MIT License,
|
||||
Copyright (c) voicebox contributors. The concatenation half is reworked for
|
||||
torch tensors (our inference helpers pass raw model output — possibly
|
||||
multi-channel — to the effect chain), and the sample rate comes from the
|
||||
engine's declared rate rather than the first chunk (fixes a latent upstream
|
||||
bug where a mid-run rate change was silently ignored).
|
||||
|
||||
Splits long text into sentence-boundary chunks and joins the per-chunk audio
|
||||
with a short crossfade. Pure functions — the generation loop itself lives in
|
||||
``api/routers/generation.py`` next to the existing ``[pause]`` span stitcher,
|
||||
so this module stays unit-testable without a model.
|
||||
|
||||
Short text (<= max_chunk_chars) never reaches this module's concat path; the
|
||||
callers keep their unchanged single-shot fast path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import List
|
||||
|
||||
logger = logging.getLogger("omnivoice.chunked_tts")
|
||||
|
||||
# Default chunk size in characters. 0 disables chunking entirely.
|
||||
DEFAULT_MAX_CHUNK_CHARS = 800
|
||||
|
||||
# Default crossfade between chunks. 0 = hard cut.
|
||||
DEFAULT_CROSSFADE_MS = 50
|
||||
|
||||
# Common abbreviations that should NOT be treated as sentence endings.
|
||||
# Lowercase for case-insensitive matching.
|
||||
_ABBREVIATIONS = frozenset({
|
||||
"mr", "mrs", "ms", "dr", "prof", "sr", "jr", "st", "ave", "blvd",
|
||||
"inc", "ltd", "corp", "dept", "est", "approx", "vs", "etc",
|
||||
"e.g", "i.e", "a.m", "p.m", "u.s", "u.s.a", "u.k",
|
||||
})
|
||||
|
||||
# Inline bracket tags (paralinguistic tags like [laugh]; our own
|
||||
# [pause 300ms] markers). The splitter must never cut inside one.
|
||||
_BRACKET_TAG_RE = re.compile(r"\[[^\]]*\]")
|
||||
|
||||
|
||||
def split_text_into_chunks(text: str, max_chars: int = DEFAULT_MAX_CHUNK_CHARS) -> List[str]:
|
||||
"""Split *text* at natural boundaries into chunks of at most *max_chars*.
|
||||
|
||||
Priority: sentence-end (``.!?`` not after an abbreviation/decimal and not
|
||||
inside brackets, plus fullwidth equivalents) -> clause boundary
|
||||
(``;:,`` / em dash) -> whitespace -> hard cut that avoids splitting a
|
||||
``[tag]``.
|
||||
"""
|
||||
text = text.strip()
|
||||
if not text:
|
||||
return []
|
||||
if max_chars <= 0 or len(text) <= max_chars:
|
||||
return [text]
|
||||
|
||||
chunks: List[str] = []
|
||||
remaining = text
|
||||
|
||||
while remaining:
|
||||
remaining = remaining.lstrip()
|
||||
if not remaining:
|
||||
break
|
||||
if len(remaining) <= max_chars:
|
||||
chunks.append(remaining)
|
||||
break
|
||||
|
||||
segment = remaining[:max_chars]
|
||||
|
||||
split_pos = _find_last_sentence_end(segment)
|
||||
if split_pos == -1:
|
||||
split_pos = _find_last_clause_boundary(segment)
|
||||
if split_pos == -1:
|
||||
split_pos = segment.rfind(" ")
|
||||
if split_pos == -1:
|
||||
split_pos = _safe_hard_cut(segment, max_chars)
|
||||
|
||||
chunk = remaining[: split_pos + 1].strip()
|
||||
if chunk:
|
||||
chunks.append(chunk)
|
||||
remaining = remaining[split_pos + 1:]
|
||||
|
||||
return chunks
|
||||
|
||||
|
||||
def _find_last_sentence_end(text: str) -> int:
|
||||
"""Index of the last sentence-ending punctuation, or -1.
|
||||
|
||||
Skips periods after common abbreviations and decimals, anything inside
|
||||
a bracket tag, and also recognizes fullwidth sentence punctuation
|
||||
(ideographic full stop / fullwidth ! and ?) for no-space scripts.
|
||||
"""
|
||||
best = -1
|
||||
for m in re.finditer(r"[.!?](?:\s|$)", text):
|
||||
pos = m.start()
|
||||
if text[pos] == ".":
|
||||
word_start = pos - 1
|
||||
while word_start >= 0 and text[word_start].isalpha():
|
||||
word_start -= 1
|
||||
word = text[word_start + 1: pos].lower()
|
||||
if word in _ABBREVIATIONS:
|
||||
continue
|
||||
if word_start >= 0 and text[word_start].isdigit():
|
||||
continue
|
||||
if _inside_bracket_tag(text, pos):
|
||||
continue
|
||||
best = pos
|
||||
# Fullwidth sentence enders (ideographic full stop, fullwidth !, ?)
|
||||
# written as escapes to keep the repo's no-literal-CJK gate clean.
|
||||
for m in re.finditer("[\u3002\uff01\uff1f]", text):
|
||||
if m.start() > best:
|
||||
best = m.start()
|
||||
return best
|
||||
|
||||
|
||||
def _find_last_clause_boundary(text: str) -> int:
|
||||
best = -1
|
||||
for m in re.finditer(r"[;:,—](?:\s|$)", text):
|
||||
if _inside_bracket_tag(text, m.start()):
|
||||
continue
|
||||
best = m.start()
|
||||
return best
|
||||
|
||||
|
||||
def _inside_bracket_tag(text: str, pos: int) -> bool:
|
||||
for m in _BRACKET_TAG_RE.finditer(text):
|
||||
if m.start() < pos < m.end():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _safe_hard_cut(segment: str, max_chars: int) -> int:
|
||||
cut = max_chars - 1
|
||||
for m in _BRACKET_TAG_RE.finditer(segment):
|
||||
if m.start() < cut < m.end():
|
||||
return m.start() - 1 if m.start() > 0 else cut
|
||||
return cut
|
||||
|
||||
|
||||
def concatenate_audio_chunks(chunks: list, sample_rate: int,
|
||||
crossfade_ms: int = DEFAULT_CROSSFADE_MS):
|
||||
"""Join per-chunk waveforms with a linear crossfade on the sample axis.
|
||||
|
||||
``chunks`` are torch tensors as returned by the engine (1-D, or N-D with
|
||||
samples on the last axis — matching what ``_render_with_pauses`` handles).
|
||||
Crossfade overlap is clamped to the shorter neighbor; ``crossfade_ms=0``
|
||||
is a hard concat.
|
||||
"""
|
||||
import torch
|
||||
|
||||
chunks = [c for c in chunks if c is not None and c.shape[-1] > 0]
|
||||
if not chunks:
|
||||
return torch.zeros(1, dtype=torch.float32)
|
||||
if len(chunks) == 1:
|
||||
return chunks[0]
|
||||
|
||||
crossfade_samples = int(sample_rate * crossfade_ms / 1000)
|
||||
result = chunks[0]
|
||||
|
||||
for chunk in chunks[1:]:
|
||||
chunk = chunk.to(device=result.device, dtype=result.dtype)
|
||||
overlap = min(crossfade_samples, result.shape[-1], chunk.shape[-1])
|
||||
if overlap > 0:
|
||||
fade_out = torch.linspace(1.0, 0.0, overlap, dtype=result.dtype, device=result.device)
|
||||
fade_in = torch.linspace(0.0, 1.0, overlap, dtype=result.dtype, device=result.device)
|
||||
blended = result[..., -overlap:] * fade_out + chunk[..., :overlap] * fade_in
|
||||
result = torch.cat([result[..., :-overlap], blended, chunk[..., overlap:]], dim=-1)
|
||||
else:
|
||||
result = torch.cat([result, chunk], dim=-1)
|
||||
|
||||
return result
|
||||
@@ -13,8 +13,7 @@ What's here
|
||||
* **Content-hash cache lookup** — `compute_file_hash`, `find_cached_job`.
|
||||
* **Safe path resolution** — `safe_job_dir`.
|
||||
* **Process lifecycle** — ffmpeg/demucs subprocess tracking + `kill_job_procs`
|
||||
so `POST /dub/abort/{id}` can tear down in-flight work (implemented in
|
||||
`services.proc_registry`, re-exported here for compatibility).
|
||||
so `POST /dub/abort/{id}` can tear down in-flight work.
|
||||
* **SSE helpers** — `sse_event`, `prep_event`.
|
||||
|
||||
What stays in the router
|
||||
@@ -45,17 +44,6 @@ from core.config import DUB_DIR
|
||||
from fastapi import HTTPException
|
||||
from services.ffmpeg_utils import find_ffmpeg, find_ffprobe, _get_semaphore, _spawn_with_retry
|
||||
from services.model_manager import get_best_device
|
||||
# Process lifecycle moved to its own leaf module so ffmpeg_utils can import
|
||||
# it at module top (no dub_pipeline ↔ ffmpeg_utils cycle). Re-exported here —
|
||||
# dub_core and tests still alias these names through this module.
|
||||
from services.proc_registry import ( # noqa: F401 — re-exports
|
||||
_active_procs,
|
||||
_active_procs_lock,
|
||||
has_active_procs,
|
||||
kill_job_procs,
|
||||
register_proc,
|
||||
unregister_proc,
|
||||
)
|
||||
from core.db import db_conn
|
||||
from core import event_bus
|
||||
from core import failure
|
||||
@@ -68,6 +56,8 @@ logger = logging.getLogger("omnivoice.dub_pipeline")
|
||||
|
||||
_dub_jobs: dict[str, dict] = {}
|
||||
_dub_jobs_lock = threading.Lock()
|
||||
_active_procs: dict[str, list] = {}
|
||||
_active_procs_lock = threading.Lock()
|
||||
|
||||
_DUB_DIR_REAL = os.path.realpath(DUB_DIR)
|
||||
_HASH_BUF_SIZE = 1 << 18 # 256 KB chunks for hashing
|
||||
@@ -148,8 +138,42 @@ def find_cached_job(content_hash: str, exclude_job_id: str) -> Optional[dict]:
|
||||
|
||||
|
||||
# ── Process lifecycle ───────────────────────────────────────────────────────
|
||||
# register_proc / unregister_proc / kill_job_procs / has_active_procs live in
|
||||
# services.proc_registry (imported + re-exported above).
|
||||
|
||||
|
||||
def register_proc(job_id: str, proc) -> None:
|
||||
"""Track an in-flight subprocess so /dub/abort can kill it."""
|
||||
with _active_procs_lock:
|
||||
_active_procs.setdefault(job_id, []).append(proc)
|
||||
|
||||
|
||||
def unregister_proc(job_id: str, proc) -> None:
|
||||
with _active_procs_lock:
|
||||
lst = _active_procs.get(job_id)
|
||||
if lst and proc in lst:
|
||||
lst.remove(proc)
|
||||
if lst is not None and not lst:
|
||||
_active_procs.pop(job_id, None)
|
||||
|
||||
|
||||
def kill_job_procs(job_id: str) -> None:
|
||||
"""Kill every subprocess still running under a given job id. Idempotent."""
|
||||
with _active_procs_lock:
|
||||
procs = list(_active_procs.get(job_id, []))
|
||||
for proc in procs:
|
||||
try:
|
||||
if proc.returncode is None:
|
||||
proc.kill()
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.warning("Failed to kill subprocess for %s: %s", job_id, e)
|
||||
with _active_procs_lock:
|
||||
_active_procs.pop(job_id, None)
|
||||
|
||||
|
||||
def has_active_procs(job_id: str) -> bool:
|
||||
with _active_procs_lock:
|
||||
return bool(_active_procs.get(job_id))
|
||||
|
||||
|
||||
# ── Job state (in-memory + SQLite fallback) ────────────────────────────────
|
||||
@@ -171,10 +195,7 @@ def get_job(job_id: str) -> Optional[dict]:
|
||||
_dub_jobs[job_id] = job
|
||||
return job
|
||||
except json.JSONDecodeError as e:
|
||||
# job_id arrives from request paths — strip newlines so a crafted
|
||||
# id can't forge extra log lines (py/log-injection).
|
||||
safe_id = str(job_id).replace("\r", "").replace("\n", "")
|
||||
logger.error("Failed to decode dub_history.job_data for %s: %s", safe_id, e)
|
||||
logger.error("Failed to decode dub_history.job_data for %s: %s", job_id, e)
|
||||
return None
|
||||
|
||||
|
||||
|
||||
@@ -1,118 +0,0 @@
|
||||
"""Second-pass ASR quality control for dubs (Wave 3.3 / Spec 5).
|
||||
|
||||
After a dub is generated, re-recognize the synthetic audio and compare what
|
||||
the ASR *heard* against what we asked the TTS to *say*. Where the two drift
|
||||
apart, the line is flagged for the user to verify — turning subtitle timing
|
||||
and pronunciation from "trusted math" into "measured truth", and doubling as
|
||||
an automatic dub-quality check.
|
||||
|
||||
Design delta from pyvideotrans (whose second pass lets recognized text
|
||||
*replace* the subtitles wholesale): we keep the GENERATED text authoritative
|
||||
for content and use the second pass for *measurement* — timing + a drift
|
||||
score that feeds the incremental re-dub loop, never silently overwriting the
|
||||
translation.
|
||||
|
||||
Pure functions here (no ASR, no I/O) so the scoring is unit-testable; the
|
||||
pipeline stage that runs the ASR pass lives in the dub router.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
def _tokens(text: str) -> list[str]:
|
||||
"""Lowercase word tokens, punctuation stripped — the unit drift is scored
|
||||
in. Script-agnostic: for no-space scripts each character is a token, which
|
||||
still gives a sensible edit-distance ratio."""
|
||||
text = (text or "").lower().strip()
|
||||
if not text:
|
||||
return []
|
||||
words = re.findall(r"\w+", text, flags=re.UNICODE)
|
||||
return words or list(text.replace(" ", ""))
|
||||
|
||||
|
||||
def _edit_distance(a: list[str], b: list[str]) -> int:
|
||||
"""Levenshtein distance between two token lists (iterative, O(len(a)*len(b))
|
||||
time, O(len(b)) space)."""
|
||||
if not a:
|
||||
return len(b)
|
||||
if not b:
|
||||
return len(a)
|
||||
prev = list(range(len(b) + 1))
|
||||
for i, ta in enumerate(a, 1):
|
||||
cur = [i]
|
||||
for j, tb in enumerate(b, 1):
|
||||
cost = 0 if ta == tb else 1
|
||||
cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + cost))
|
||||
prev = cur
|
||||
return prev[-1]
|
||||
|
||||
|
||||
def word_error_rate(reference: str, hypothesis: str) -> float:
|
||||
"""Normalized token edit distance in [0.0, 1.0+].
|
||||
|
||||
0.0 = the ASR heard exactly the target text. ~1.0 = entirely different.
|
||||
Can exceed 1.0 when the hypothesis is much longer than the reference
|
||||
(insertions); callers clamp/threshold as needed. An empty reference with a
|
||||
non-empty hypothesis scores 1.0 (everything is an insertion)."""
|
||||
ref = _tokens(reference)
|
||||
hyp = _tokens(hypothesis)
|
||||
if not ref and not hyp:
|
||||
return 0.0
|
||||
if not ref:
|
||||
return 1.0
|
||||
return _edit_distance(ref, hyp) / len(ref)
|
||||
|
||||
|
||||
@dataclass
|
||||
class SegmentQC:
|
||||
seg_id: str
|
||||
target_text: str
|
||||
recognized_text: str
|
||||
drift: float # word_error_rate(target, recognized)
|
||||
flagged: bool # drift >= threshold
|
||||
new_start: float | None # measured onset from the dubbed-audio recognition
|
||||
new_end: float | None
|
||||
|
||||
|
||||
def _overlap(a0: float, a1: float, b0: float, b1: float) -> float:
|
||||
return max(0.0, min(a1, b1) - max(a0, b0))
|
||||
|
||||
|
||||
def score_dub(
|
||||
dub_segments: list[dict],
|
||||
recognized: list[dict],
|
||||
*,
|
||||
drift_threshold: float = 0.5,
|
||||
seg_ids: list | None = None,
|
||||
) -> list[SegmentQC]:
|
||||
"""Match the second-pass recognition to the dub segments and score drift.
|
||||
|
||||
``dub_segments`` are the segments we generated (each {start, end, text});
|
||||
``recognized`` are the ASR result segments on the dubbed audio (each
|
||||
{start, end, text}). Each dub segment is matched to the recognized
|
||||
segment(s) it overlaps in time; their text is concatenated as the
|
||||
hypothesis and scored against the dub segment's ``text``. The recognized
|
||||
span's bounds become the measured start/end (subtitle-timing truth).
|
||||
"""
|
||||
results: list[SegmentQC] = []
|
||||
for i, seg in enumerate(dub_segments):
|
||||
sid = str(seg_ids[i]) if (seg_ids and i < len(seg_ids)) else str(seg.get("id", i))
|
||||
s0, s1 = float(seg.get("start", 0.0)), float(seg.get("end", 0.0))
|
||||
hits = [r for r in recognized if _overlap(s0, s1, float(r.get("start", 0.0)), float(r.get("end", 0.0))) > 0]
|
||||
hyp = " ".join((r.get("text") or "").strip() for r in hits).strip()
|
||||
drift = word_error_rate(seg.get("text", ""), hyp)
|
||||
new_start = min((float(r.get("start", 0.0)) for r in hits), default=None)
|
||||
new_end = max((float(r.get("end", 0.0)) for r in hits), default=None)
|
||||
results.append(SegmentQC(
|
||||
seg_id=sid,
|
||||
target_text=(seg.get("text") or "").strip(),
|
||||
recognized_text=hyp,
|
||||
drift=round(drift, 3),
|
||||
flagged=drift >= drift_threshold,
|
||||
new_start=new_start,
|
||||
new_end=new_end,
|
||||
))
|
||||
return results
|
||||
@@ -24,78 +24,6 @@ logger = logging.getLogger("omnivoice.engine_env")
|
||||
|
||||
_TORCH_COMPILE_KEY = "perf.torch_compile_disabled"
|
||||
|
||||
# #278: explicit opt-in override — set to 1/true to attempt torch.compile even
|
||||
# when the GPU's compute capability is not in this PyTorch build's arch list
|
||||
# (e.g. a brand-new architecture running through PTX forward-compat).
|
||||
_FORCE_COMPILE_ENV = "OMNIVOICE_FORCE_TORCH_COMPILE"
|
||||
|
||||
# #278: set (with a reason) the first time torch.compile — or *running* the
|
||||
# compiled model — fails at runtime in this process. Once set, every later
|
||||
# load in the same session goes straight to eager instead of re-tripping the
|
||||
# same Dynamo/Inductor/Triton failure.
|
||||
_compile_runtime_failure: Optional[str] = None
|
||||
|
||||
|
||||
def mark_compile_runtime_failure(reason: str) -> None:
|
||||
"""Record that torch.compile (or compiled execution) failed at runtime.
|
||||
|
||||
Called by ``services.model_manager`` when compilation raises, or when a
|
||||
generation through the compiled model dies inside the Dynamo / Inductor /
|
||||
Triton stack (#278). Disables compile for the rest of the process — eager
|
||||
mode from here on; the next app restart probes again.
|
||||
"""
|
||||
global _compile_runtime_failure
|
||||
_compile_runtime_failure = reason or "unknown torch.compile runtime failure"
|
||||
logger.warning(
|
||||
"torch.compile disabled for this session after a runtime failure: %s",
|
||||
_compile_runtime_failure,
|
||||
)
|
||||
|
||||
|
||||
def _force_compile_requested() -> bool:
|
||||
value = os.environ.get(_FORCE_COMPILE_ENV, "")
|
||||
return value.strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _cuda_arch_supported_for_compile() -> "tuple[bool, str]":
|
||||
"""Check the GPU's compute capability against this torch build's arch list.
|
||||
|
||||
New GPU architectures (e.g. Blackwell sm_120, issue #278) routinely break
|
||||
torch.compile/Triton before upstream support lands: the eager model runs
|
||||
via PTX forward-compat, but Inductor/Triton kernel compilation targets the
|
||||
new arch directly and fails mid-generation. If the device's ``sm_XY`` tag
|
||||
is absent from ``torch.cuda.get_arch_list()`` we treat compile as
|
||||
unsupported and use eager.
|
||||
|
||||
Returns ``(supported, reason)``. Fails open — any probe error returns
|
||||
``(True, "")`` so a weird torch build never silently loses the
|
||||
optimization (the runtime fallback in model_manager still protects
|
||||
generation).
|
||||
"""
|
||||
try:
|
||||
import torch
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
return True, ""
|
||||
major, minor = torch.cuda.get_device_capability(0)
|
||||
arch_list = list(getattr(torch.cuda, "get_arch_list", lambda: [])() or [])
|
||||
if not arch_list:
|
||||
return True, ""
|
||||
sm_tag = f"sm_{major}{minor}"
|
||||
if sm_tag in arch_list or f"compute_{major}{minor}" in arch_list:
|
||||
return True, ""
|
||||
try:
|
||||
device_name = torch.cuda.get_device_name(0)
|
||||
except Exception:
|
||||
device_name = "GPU"
|
||||
return False, (
|
||||
f"{device_name} (compute capability {major}.{minor} / {sm_tag}) is not "
|
||||
f"in this PyTorch build's supported arch list ({', '.join(arch_list)})"
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("CUDA arch probe for torch.compile failed; assuming supported", exc_info=True)
|
||||
return True, ""
|
||||
|
||||
|
||||
def should_torch_compile(device: str) -> bool:
|
||||
"""Decide whether to apply ``torch.compile`` to an in-process model.
|
||||
@@ -106,14 +34,9 @@ def should_torch_compile(device: str) -> bool:
|
||||
- device == "cuda" (compile only helps the CUDA path here),
|
||||
- Triton importable (``find_spec`` — the cross-platform gate that closes
|
||||
#65; no Windows wheel ⇒ skip ⇒ eager),
|
||||
- the user has NOT set the ``perf.torch_compile_disabled`` escape hatch,
|
||||
- compile has NOT already failed at runtime in this process (#278),
|
||||
- the GPU's compute capability is in this torch build's arch list (#278)
|
||||
— overridable via ``OMNIVOICE_FORCE_TORCH_COMPILE=1``.
|
||||
- the user has NOT set the ``perf.torch_compile_disabled`` escape hatch.
|
||||
|
||||
Returns False (→ eager mode) on any of those, logging the reason at INFO.
|
||||
torch.compile is an optimization, never a requirement — generation must
|
||||
always work without it.
|
||||
"""
|
||||
if device != "cuda":
|
||||
return False
|
||||
@@ -128,25 +51,6 @@ def should_torch_compile(device: str) -> bool:
|
||||
return False
|
||||
except Exception:
|
||||
logger.exception("should_torch_compile: settings read failed; proceeding")
|
||||
if _compile_runtime_failure is not None:
|
||||
logger.info(
|
||||
"torch.compile skipped: failed earlier this session (%s) — using eager mode.",
|
||||
_compile_runtime_failure,
|
||||
)
|
||||
return False
|
||||
supported, reason = _cuda_arch_supported_for_compile()
|
||||
if not supported:
|
||||
if _force_compile_requested():
|
||||
logger.warning(
|
||||
"torch.compile forced via %s=1 despite: %s", _FORCE_COMPILE_ENV, reason,
|
||||
)
|
||||
return True
|
||||
logger.info(
|
||||
"torch.compile skipped: %s — using eager mode. "
|
||||
"(Set %s=1 to attempt compile anyway.)",
|
||||
reason, _FORCE_COMPILE_ENV,
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
|
||||
@@ -1,156 +0,0 @@
|
||||
"""Pure, host-aware routing resolver — maps an engine's declared ``gpu_compat``
|
||||
against the cached host capabilities to "where will this engine *actually* run
|
||||
on this machine, and is that a problem the user should hear about?"
|
||||
|
||||
No model load, no probe (the caller passes the cached ``HostCaps``), no I/O.
|
||||
Deterministic and byte-identical for a given ``(gpu_compat, HostCaps)`` across
|
||||
macOS/Windows/Linux — that cross-OS determinism is the whole point of the
|
||||
no-silent-fallback contract.
|
||||
|
||||
Reason strings are author-controlled English (interpolating only family/device
|
||||
names) but are **still** scrubbed by the caller (``core.scrub.scrub_text``)
|
||||
before serialization, because an interpolated ``device_name`` or probe note can
|
||||
carry a home path.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal, TypedDict
|
||||
|
||||
from core.device_caps import (
|
||||
DIRECTML_MARKER,
|
||||
KERNEL_RISK_MARKER,
|
||||
HostCaps,
|
||||
)
|
||||
|
||||
RoutingStatus = Literal["accelerated", "cpu_fallback", "cpu_only", "unavailable", "n/a"]
|
||||
|
||||
|
||||
class RoutingResult(TypedDict):
|
||||
effective_device: str # a DeviceFamily value or "cpu"
|
||||
routing_status: RoutingStatus # resolve_routing never emits "n/a" (LLM-only)
|
||||
routing_reason: str | None # raw, pre-scrub
|
||||
|
||||
|
||||
def _caveat(caps: HostCaps) -> str | None:
|
||||
"""A kernel-risk caveat string for an otherwise-accelerated host, or None.
|
||||
Advisory notes (multi-GPU, VRAM-query-failed, DirectML) never qualify."""
|
||||
for note in caps.notes:
|
||||
if KERNEL_RISK_MARKER in note:
|
||||
return f"{caps.family.upper()} selected, but: {note}"
|
||||
return None
|
||||
|
||||
|
||||
def resolve_routing(gpu_compat: tuple[str, ...], caps: HostCaps) -> RoutingResult:
|
||||
"""Resolve the effective device + status for an engine on this host.
|
||||
|
||||
Rules are evaluated in order; the first match wins (see spec §2)."""
|
||||
targets = tuple(gpu_compat or ())
|
||||
fam = caps.family
|
||||
|
||||
# 1. Empty compat — reserved for LLM (which never calls this). Defensive.
|
||||
if not targets:
|
||||
return {
|
||||
"effective_device": "cpu",
|
||||
"routing_status": "cpu_only",
|
||||
"routing_reason": "engine declares no compute targets",
|
||||
}
|
||||
|
||||
# 2. Host accelerator is one the engine supports → accelerated.
|
||||
if fam != "cpu" and fam in targets:
|
||||
return {
|
||||
"effective_device": fam,
|
||||
"routing_status": "accelerated",
|
||||
"routing_reason": _caveat(caps),
|
||||
}
|
||||
|
||||
# 3. Host has an accelerator the engine lacks, but engine supports cpu
|
||||
# → the no-silent-fallback signal.
|
||||
if fam != "cpu" and "cpu" in targets:
|
||||
if fam == "rocm" and "cuda" in targets and "rocm" not in targets:
|
||||
reason = "declares CUDA only; ROCm not in its compat set"
|
||||
else:
|
||||
reason = f"engine has no {fam.upper()} path; running on CPU"
|
||||
return {
|
||||
"effective_device": "cpu",
|
||||
"routing_status": "cpu_fallback",
|
||||
"routing_reason": reason,
|
||||
}
|
||||
|
||||
# 4. Genuine CPU-only host (or DirectML, which the probe reports as cpu)
|
||||
# and engine supports cpu → benign; must not warn or block.
|
||||
if fam == "cpu" and "cpu" in targets:
|
||||
reason = None
|
||||
for note in caps.notes:
|
||||
if DIRECTML_MARKER in note:
|
||||
reason = (
|
||||
"DirectML GPU present; engine routes via torch CPU path "
|
||||
"(DirectML acceleration not wired into routing)"
|
||||
)
|
||||
break
|
||||
return {
|
||||
"effective_device": "cpu",
|
||||
"routing_status": "cpu_only",
|
||||
"routing_reason": reason,
|
||||
}
|
||||
|
||||
# 5. Engine needs an accelerator this host lacks and has no cpu path.
|
||||
first = targets[0]
|
||||
return {
|
||||
"effective_device": first,
|
||||
"routing_status": "unavailable",
|
||||
"routing_reason": f"requires {', '.join(targets)}; this host has {fam}",
|
||||
}
|
||||
|
||||
|
||||
def routing_notice(result: RoutingResult) -> tuple[str, str | None] | None:
|
||||
"""`(status, reason)` when a synth-time notice SHOULD be surfaced to the
|
||||
user, else `None`. Surfaced for `cpu_fallback` (always) and for
|
||||
`accelerated` ONLY when it carries a driver/arch caveat reason — everything
|
||||
else (`cpu_only`, clean `accelerated`, `n/a`) is benign and stays silent."""
|
||||
st = result["routing_status"]
|
||||
if st == "cpu_fallback" or (st == "accelerated" and result["routing_reason"]):
|
||||
return (st, result["routing_reason"])
|
||||
return None
|
||||
|
||||
|
||||
def header_safe_reason(reason: str | None) -> str | None:
|
||||
"""A routing reason made safe for an HTTP header value: scrubbed, then
|
||||
ASCII-sanitized (headers are latin-1; a non-ASCII device name would 500 the
|
||||
response otherwise), **control characters stripped** (a CR/LF could split
|
||||
the header / inject a new one), and length-capped at 256. Returns None for
|
||||
an empty reason. No regex — `.encode`/membership only (CodeQL-clean)."""
|
||||
if not reason:
|
||||
return None
|
||||
from core.scrub import scrub_text
|
||||
ascii_only = scrub_text(reason).encode("ascii", "ignore").decode("ascii")
|
||||
# Drop ASCII control chars (0x00-0x1F + DEL 0x7F) — incl. CR/LF, so the
|
||||
# value can never break out of its header line.
|
||||
cleaned = "".join(c for c in ascii_only if 0x20 <= ord(c) < 0x7F)
|
||||
return cleaned[:256] or None
|
||||
|
||||
|
||||
def routing_fields(gpu_compat: tuple[str, ...], caps: HostCaps) -> dict:
|
||||
"""The three serialization-ready routing keys for a ``list_backends`` entry.
|
||||
|
||||
Resolves routing and applies the redaction contract: ``routing_reason`` is
|
||||
scrubbed via ``core.scrub.scrub_text`` only when truthy, so a ``None`` reason
|
||||
serializes as JSON ``null`` (NOT ``""`` — ``scrub_text(None)`` would coerce
|
||||
to ``""``). Used by tts/asr ``list_backends`` so the scrub rule lives in one
|
||||
place. (LLM emits its own literal ``network``/``n/a``/``null`` fields and
|
||||
does NOT call this.)
|
||||
"""
|
||||
from core.scrub import scrub_text
|
||||
|
||||
r = resolve_routing(tuple(gpu_compat or ()), caps)
|
||||
reason = r["routing_reason"]
|
||||
return {
|
||||
"effective_device": r["effective_device"],
|
||||
"routing_status": r["routing_status"],
|
||||
"routing_reason": scrub_text(reason) if reason else None,
|
||||
}
|
||||
|
||||
|
||||
__all__ = [
|
||||
"RoutingStatus", "RoutingResult", "resolve_routing", "routing_fields",
|
||||
"routing_notice", "header_safe_reason",
|
||||
]
|
||||
@@ -5,10 +5,6 @@ import os
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
# Leaf module (stdlib-only) — safe to import at module top, unlike
|
||||
# services.dub_pipeline which imports this module and would cycle.
|
||||
from services.proc_registry import register_proc, unregister_proc
|
||||
|
||||
logger = logging.getLogger("omnivoice.api")
|
||||
|
||||
# Cap concurrent ffmpeg jobs so macOS posix_spawn can't hit EAGAIN under load.
|
||||
@@ -23,36 +19,6 @@ def _get_semaphore() -> asyncio.Semaphore:
|
||||
return _FFMPEG_SEMAPHORE
|
||||
|
||||
|
||||
# Candidate paths that exist but won't run (validated once per process).
|
||||
# Windows users hit this as `[WinError 193] %1 is not a valid Win32
|
||||
# application` (#360/#361/#362): a corrupt/wrong-arch imageio-ffmpeg
|
||||
# download or a WindowsApps alias stub passes `os.path.isfile` / `which`
|
||||
# but explodes at spawn. Probe each candidate with `-version` and fall
|
||||
# through to the next source instead of returning a time bomb.
|
||||
_BINARY_OK: dict[str, bool] = {}
|
||||
|
||||
|
||||
def _binary_runs(path: str) -> bool:
|
||||
cached = _BINARY_OK.get(path)
|
||||
if cached is not None:
|
||||
return cached
|
||||
try:
|
||||
subprocess.run(
|
||||
[path, "-version"],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||
timeout=10, check=False,
|
||||
)
|
||||
ok = True
|
||||
except (OSError, subprocess.TimeoutExpired, subprocess.SubprocessError) as e:
|
||||
logger.warning(
|
||||
"Rejecting non-runnable ffmpeg/ffprobe candidate %s: %s",
|
||||
os.path.basename(str(path)), e,
|
||||
)
|
||||
ok = False
|
||||
_BINARY_OK[path] = ok
|
||||
return ok
|
||||
|
||||
|
||||
def find_ffmpeg():
|
||||
"""Locate an ffmpeg binary.
|
||||
|
||||
@@ -67,15 +33,15 @@ def find_ffmpeg():
|
||||
env_path = os.environ.get("FFMPEG_PATH")
|
||||
if env_path:
|
||||
resolved = shutil.which(env_path)
|
||||
if resolved and _binary_runs(resolved):
|
||||
if resolved:
|
||||
return resolved
|
||||
# 2. imageio-ffmpeg bundled static binary
|
||||
try:
|
||||
import imageio_ffmpeg
|
||||
candidate = imageio_ffmpeg.get_ffmpeg_exe()
|
||||
if candidate and os.path.isfile(candidate) and _binary_runs(candidate):
|
||||
if candidate and os.path.isfile(candidate):
|
||||
return candidate
|
||||
logger.debug("imageio_ffmpeg binary not usable at %s", candidate)
|
||||
logger.debug("imageio_ffmpeg binary not found at %s", candidate)
|
||||
except Exception as e:
|
||||
logger.debug("imageio_ffmpeg unavailable: %s", e)
|
||||
# 3. Well-known system paths + PATH lookup
|
||||
@@ -88,10 +54,9 @@ def find_ffmpeg():
|
||||
"ffmpeg",
|
||||
]
|
||||
for path in common:
|
||||
resolved = shutil.which(path)
|
||||
if resolved and _binary_runs(resolved):
|
||||
return resolved
|
||||
logger.warning("ffmpeg not found (or not runnable) in env, imageio, or system PATH")
|
||||
if shutil.which(path):
|
||||
return path
|
||||
logger.warning("ffmpeg not found in env, imageio, or system PATH")
|
||||
return None
|
||||
|
||||
|
||||
@@ -115,14 +80,14 @@ def resolve_ffprobe() -> str | None:
|
||||
continue
|
||||
# The env var may carry either an absolute path to a file OR a bare
|
||||
# command name (legacy). Accept both shapes — file first.
|
||||
if os.path.isfile(path) and _binary_runs(path):
|
||||
if os.path.isfile(path):
|
||||
return path
|
||||
resolved = shutil.which(path)
|
||||
if resolved and _binary_runs(resolved):
|
||||
if resolved:
|
||||
return resolved
|
||||
|
||||
system_probe = shutil.which("ffprobe")
|
||||
if system_probe and _binary_runs(system_probe):
|
||||
if system_probe:
|
||||
return system_probe
|
||||
return None
|
||||
|
||||
@@ -241,168 +206,16 @@ async def _spawn_with_retry(cmd, **kwargs):
|
||||
raise last_err if last_err else RuntimeError("spawn failed")
|
||||
|
||||
|
||||
def _atempo_chain(ratio: float) -> str:
|
||||
"""Build an `atempo=…,atempo=…` filter chain for arbitrary ratios.
|
||||
|
||||
ffmpeg's atempo filter is limited to [0.5, 2.0] per stage. Chaining
|
||||
multiple stages multiplies the effective ratio while keeping each
|
||||
individual stage inside the well-behaved range. Pitch is preserved
|
||||
(WSOLA-style time-domain stretching). ratio > 1 speeds up, < 1
|
||||
slows down.
|
||||
"""
|
||||
stages: list[str] = []
|
||||
remaining = ratio
|
||||
while remaining > 2.0:
|
||||
stages.append("atempo=2.0")
|
||||
remaining /= 2.0
|
||||
while remaining < 0.5:
|
||||
stages.append("atempo=0.5")
|
||||
remaining /= 0.5
|
||||
stages.append(f"atempo={remaining:.6f}")
|
||||
return ",".join(stages)
|
||||
|
||||
|
||||
async def _pitch_preserving_stretch(wav, target_samples: int, sr: int):
|
||||
"""Time-stretch a (1, samples) tensor to `target_samples` while
|
||||
preserving pitch, by piping the audio through `ffmpeg atempo`.
|
||||
|
||||
Async so it never blocks the event loop: it's awaited from the dub
|
||||
generate `_stream` generator, and each ffmpeg call is ~50-100 ms — a
|
||||
synchronous ``subprocess.run`` here froze health-checks / SSE / every
|
||||
concurrent request for the whole multi-segment job.
|
||||
|
||||
Returns a (1, target_samples) tensor on the same device as input.
|
||||
Raises RuntimeError when ffmpeg fails — callers should fall back to
|
||||
naive linear interpolation, accepting the pitch shift, to ensure the
|
||||
output isn't silent.
|
||||
"""
|
||||
# Lazy imports keep this module importable in torch-free contexts
|
||||
# (setup scripts, smoke probes) — only the stretch path needs them.
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
wl = int(wav.shape[-1])
|
||||
if target_samples <= 0 or wl == target_samples:
|
||||
return wav
|
||||
ratio = wl / target_samples
|
||||
filter_str = _atempo_chain(ratio)
|
||||
|
||||
# Mono float32 via stdin → ffmpeg → stdout. One subprocess per segment,
|
||||
# run off the event loop so concurrent requests stay responsive.
|
||||
arr = wav.detach().cpu().to(torch.float32).numpy().reshape(-1).astype(np.float32, copy=False)
|
||||
proc = await spawn_subprocess(
|
||||
find_ffmpeg(), "-hide_banner", "-loglevel", "error", "-y",
|
||||
"-f", "f32le", "-ar", str(sr), "-ac", "1", "-i", "pipe:0",
|
||||
"-af", filter_str,
|
||||
"-f", "f32le", "-ar", str(sr), "-ac", "1", "pipe:1",
|
||||
stdin=asyncio.subprocess.PIPE,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, stderr = await proc.communicate(input=arr.tobytes())
|
||||
if proc.returncode != 0 or not stdout:
|
||||
raise RuntimeError(
|
||||
(stderr.decode(errors="replace") or "atempo failed")[:200]
|
||||
)
|
||||
out_arr = np.frombuffer(stdout, dtype=np.float32)
|
||||
# atempo rarely lands exactly on the integer sample count, so
|
||||
# pad/trim to the requested slot length.
|
||||
if len(out_arr) < target_samples:
|
||||
pad = np.zeros(target_samples - len(out_arr), dtype=np.float32)
|
||||
out_arr = np.concatenate([out_arr, pad])
|
||||
elif len(out_arr) > target_samples:
|
||||
out_arr = out_arr[:target_samples]
|
||||
return torch.from_numpy(out_arr.copy()).unsqueeze(0).to(wav.device)
|
||||
|
||||
|
||||
async def probe_duration(path: str) -> float | None:
|
||||
"""Return a media file's duration in seconds via ffprobe, or None.
|
||||
|
||||
Used by the Smart Fit pipeline to sanity-check source/track lengths
|
||||
without loading the media. Never raises — probing is best-effort.
|
||||
"""
|
||||
ffprobe = find_ffprobe()
|
||||
if not ffprobe or not os.path.isfile(path):
|
||||
return None
|
||||
try:
|
||||
proc = await spawn_subprocess(
|
||||
ffprobe, "-v", "error",
|
||||
"-show_entries", "format=duration",
|
||||
"-of", "default=noprint_wrappers=1:nokey=1",
|
||||
path,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, _ = await proc.communicate()
|
||||
if proc.returncode != 0:
|
||||
return None
|
||||
return float(stdout.decode().strip())
|
||||
except Exception as e:
|
||||
logger.debug("probe_duration failed for %s: %s", os.path.basename(str(path)), e)
|
||||
return None
|
||||
|
||||
|
||||
async def probe_frame_rates(path: str) -> "tuple[str, str] | None":
|
||||
"""Return (r_frame_rate, avg_frame_rate) strings for the first video
|
||||
stream (e.g. ``("30000/1001", "2997/100")``), or None on any failure.
|
||||
|
||||
A mismatch between the two is the practical VFR signature — used by the
|
||||
Smart Fit retime pipeline to decide whether to normalise with ``fps=``
|
||||
before trim/setpts. Never raises — probing is best-effort.
|
||||
"""
|
||||
ffprobe = find_ffprobe()
|
||||
if not ffprobe or not os.path.isfile(path):
|
||||
return None
|
||||
try:
|
||||
proc = await spawn_subprocess(
|
||||
ffprobe, "-v", "error",
|
||||
"-select_streams", "v:0",
|
||||
"-show_entries", "stream=r_frame_rate,avg_frame_rate",
|
||||
"-of", "csv=p=0",
|
||||
path,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, _ = await proc.communicate()
|
||||
if proc.returncode != 0:
|
||||
return None
|
||||
parts = stdout.decode().strip().split(",")
|
||||
if len(parts) < 2:
|
||||
return None
|
||||
return parts[0].strip(), parts[1].strip()
|
||||
except Exception as e:
|
||||
logger.debug("probe_frame_rates failed for %s: %s", os.path.basename(str(path)), e)
|
||||
return None
|
||||
|
||||
|
||||
async def run_ffmpeg(cmd, timeout: float = 1800.0, capture: bool = True,
|
||||
job_id: "str | None" = None):
|
||||
async def run_ffmpeg(cmd, timeout: float = 1800.0, capture: bool = True):
|
||||
"""Run an ffmpeg subprocess with concurrency cap, timeout, and proper cleanup.
|
||||
|
||||
Returns (returncode, stdout_bytes, stderr_bytes). Raises asyncio.TimeoutError
|
||||
on hard timeout (after killing + reaping the process).
|
||||
|
||||
``job_id`` (optional) registers the process with the dub pipeline's
|
||||
process tracker (``services.proc_registry``) so ``/dub/abort`` can kill
|
||||
long export encodes (used by the Smart Fit batched retime).
|
||||
|
||||
Path-injection note: every filesystem path placed in ``cmd`` by callers
|
||||
is realpath-normalised and containment-checked against its workspace
|
||||
root (e.g. DUB_DIR) at the call site before the argv is assembled —
|
||||
see api.routers.dub_export and services.video_retime.
|
||||
"""
|
||||
stdout = asyncio.subprocess.PIPE if capture else asyncio.subprocess.DEVNULL
|
||||
stderr = asyncio.subprocess.PIPE
|
||||
async with _get_semaphore():
|
||||
proc = await _spawn_with_retry(cmd, stdout=stdout, stderr=stderr)
|
||||
if job_id:
|
||||
try:
|
||||
register_proc(job_id, proc)
|
||||
except Exception as e:
|
||||
# Newline-strip the id inline — it can originate from a path
|
||||
# param, and the log stream must stay one-event-per-line.
|
||||
logger.debug("register_proc failed for %s: %s",
|
||||
job_id.replace("\n", " ").replace("\r", " "), e)
|
||||
try:
|
||||
try:
|
||||
out, err = await asyncio.wait_for(proc.communicate(), timeout=timeout)
|
||||
@@ -418,12 +231,6 @@ async def run_ffmpeg(cmd, timeout: float = 1800.0, capture: bool = True,
|
||||
raise
|
||||
return proc.returncode, out, err
|
||||
finally:
|
||||
if job_id:
|
||||
try:
|
||||
unregister_proc(job_id, proc)
|
||||
except Exception as e:
|
||||
logger.debug("unregister_proc failed for %s: %s",
|
||||
job_id.replace("\n", " ").replace("\r", " "), e)
|
||||
# Guarantee reaping — prevents zombie pileup under timeouts or errors.
|
||||
if proc.returncode is None:
|
||||
try:
|
||||
|
||||
@@ -1,211 +0,0 @@
|
||||
"""Smart Fit planner — dub-length fitting v2, Phase A.
|
||||
|
||||
Pure planning functions for the ``smart_fit`` timing strategy: given the
|
||||
original segment timeline and the *natural-rate* duration of each dubbed
|
||||
segment's TTS audio, decide per segment how to reconcile the two by
|
||||
splitting the burden between a mild pitch-preserving audio speed-up and a
|
||||
mild per-segment video slow-down.
|
||||
|
||||
Clean-room note: this is a reimplementation from a *published description*
|
||||
of the audio-speedup + video-slowdown fitting approach (see
|
||||
docs/competitive-analysis.md, "Dub-length fitting"). No GPL source was
|
||||
consulted.
|
||||
|
||||
Algorithm per segment (defaults in :class:`FitParams`):
|
||||
|
||||
1. **Slack absorption** — the usable slot extends past the segment's
|
||||
original end into the silent gap before the next segment, keeping a
|
||||
small ``gap_guard_s`` clear of the next onset (the last segment may run
|
||||
to the end of the video). ``need = natural_dur / slot``.
|
||||
2. ``need <= 1.0`` — fits as-is; nothing to do.
|
||||
3. ``1.0 < need <= max_audio_only_rate`` — audio-only speed-up at exactly
|
||||
``need`` (imperceptible up to ~1.2×).
|
||||
4. ``need > max_audio_only_rate`` — geometric 50/50 split:
|
||||
``audio_rate = min(sqrt(need), audio_rate_cap)`` and
|
||||
``video_ratio = min(need / audio_rate, video_slow_cap)``. Whatever the
|
||||
caps can't absorb becomes ``overflow_s`` (trimmed at mix time).
|
||||
5. ``allow_video_retime=False`` — audio-only mode: rate capped at the
|
||||
legacy ``MAX_AUDIO_RATE_HARD`` (1.8, matching dub_generate's
|
||||
MAX_STRETCH_RATIO guard rail), residual overflows.
|
||||
6. **Timeline cursor** — mirrors the existing ``stretch_video`` layout
|
||||
loop: pre-roll and inter-segment gaps pass through at 1.0×; each
|
||||
segment's video chunk ``[start, effective_end]`` occupies
|
||||
``slot * video_ratio`` on the new timeline.
|
||||
|
||||
This module is deliberately I/O-free and torch-free so it can be unit- and
|
||||
golden-tested without a model, ffmpeg, or an event loop.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
# Hard ceiling for audio-only compression when video retiming is disabled.
|
||||
# Matches dub_generate.MAX_STRETCH_RATIO — above ~1.8× speech becomes a
|
||||
# garbled stream no DSP can rescue.
|
||||
MAX_AUDIO_RATE_HARD = 1.8
|
||||
|
||||
_EPS = 1e-9
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FitParams:
|
||||
"""Tunable knobs for the Smart Fit planner.
|
||||
|
||||
All defaults are deliberately conservative: ≤1.2× audio-only is
|
||||
imperceptible to most listeners; 1.5× audio is the intelligibility
|
||||
cap; 2.0× video slow-down is the limit before motion looks syrupy.
|
||||
"""
|
||||
max_audio_only_rate: float = 1.2
|
||||
audio_rate_cap: float = 1.5
|
||||
video_slow_cap: float = 2.0
|
||||
gap_guard_s: float = 0.05
|
||||
allow_video_retime: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class SegmentFit:
|
||||
"""Planner verdict for one segment."""
|
||||
index: int
|
||||
seg_id: str
|
||||
audio_rate: float # ≥ 1.0 — pitch-preserving speed-up applied to TTS audio
|
||||
video_ratio: float # ≥ 1.0 — setpts slow-down applied to the video chunk
|
||||
new_start: float # placement on the fitted (possibly longer) timeline
|
||||
new_end: float # end of the video chunk on the fitted timeline
|
||||
orig_start: float
|
||||
orig_end: float
|
||||
effective_end: float # orig_end + absorbed slack (≤ next start − gap guard)
|
||||
status: str # "fits" | "audio_stretched" | "hybrid" | "overflow_trimmed"
|
||||
overflow_s: float # seconds of (stretched) audio that still don't fit
|
||||
|
||||
|
||||
@dataclass
|
||||
class FitPlan:
|
||||
"""Full plan for one dub track."""
|
||||
segments: list[SegmentFit] = field(default_factory=list)
|
||||
# EXACT dict shape consumed by dub_export._build_video_stretch_filter_graph.
|
||||
video_plan: list[dict] = field(default_factory=list)
|
||||
total_duration: float = 0.0
|
||||
orig_duration: float = 0.0
|
||||
params: FitParams = field(default_factory=FitParams)
|
||||
|
||||
@property
|
||||
def needs_video_retime(self) -> bool:
|
||||
return any(s.video_ratio > 1.0 + 1e-6 for s in self.segments)
|
||||
|
||||
|
||||
def _fit_one(need: float, params: FitParams) -> tuple[float, float, str]:
|
||||
"""Resolve one segment's need ratio into (audio_rate, video_ratio, status)."""
|
||||
if need <= 1.0 + _EPS:
|
||||
return 1.0, 1.0, "fits"
|
||||
if need <= params.max_audio_only_rate + _EPS:
|
||||
return need, 1.0, "audio_stretched"
|
||||
if not params.allow_video_retime:
|
||||
audio_rate = min(need, MAX_AUDIO_RATE_HARD)
|
||||
status = "audio_stretched" if audio_rate >= need - _EPS else "overflow_trimmed"
|
||||
return audio_rate, 1.0, status
|
||||
# Geometric 50/50 split: equal perceptual burden on audio and video.
|
||||
audio_rate = min(math.sqrt(need), params.audio_rate_cap)
|
||||
video_ratio = min(need / audio_rate, params.video_slow_cap)
|
||||
if audio_rate * video_ratio >= need - _EPS:
|
||||
return audio_rate, video_ratio, "hybrid"
|
||||
return audio_rate, video_ratio, "overflow_trimmed"
|
||||
|
||||
|
||||
def plan_fit(
|
||||
segments: list[dict],
|
||||
natural_durs_s: list[float],
|
||||
total_dur_s: float,
|
||||
params: FitParams | None = None,
|
||||
) -> FitPlan:
|
||||
"""Plan the Smart Fit layout for a dub track.
|
||||
|
||||
``segments``: original-timeline segments in chronological order, each a
|
||||
dict with ``id``, ``start``, ``end`` (seconds). ``natural_durs_s``: the
|
||||
natural-rate TTS audio duration for each segment (parallel list).
|
||||
``total_dur_s``: original video duration (0/unknown tolerated — the last
|
||||
segment then gets no tail slack).
|
||||
|
||||
Pure function: no I/O, no torch, deterministic.
|
||||
"""
|
||||
params = params or FitParams()
|
||||
n = len(segments)
|
||||
if len(natural_durs_s) != n:
|
||||
raise ValueError(
|
||||
f"segments ({n}) and natural_durs_s ({len(natural_durs_s)}) must be parallel"
|
||||
)
|
||||
|
||||
plan = FitPlan(params=params, orig_duration=round(float(total_dur_s), 4))
|
||||
if n == 0:
|
||||
plan.total_duration = round(max(0.0, float(total_dur_s)), 4)
|
||||
return plan
|
||||
|
||||
cursor = 0.0
|
||||
for i, seg in enumerate(segments):
|
||||
start = float(seg["start"])
|
||||
end = float(seg["end"])
|
||||
natural = max(0.0, float(natural_durs_s[i]))
|
||||
|
||||
# (a) Slack absorption. Extend-only: the slot never shrinks below
|
||||
# the original [start, end] even when segments are back-to-back.
|
||||
if i + 1 < n:
|
||||
next_start = float(segments[i + 1]["start"])
|
||||
effective_end = max(end, next_start - params.gap_guard_s)
|
||||
# Never bleed past the next segment's onset (overlapping or
|
||||
# near-touching source segments).
|
||||
effective_end = min(max(effective_end, start), max(next_start, end))
|
||||
else:
|
||||
effective_end = max(end, float(total_dur_s)) if total_dur_s > 0 else end
|
||||
slot = max(effective_end - start, 1e-3)
|
||||
|
||||
need = natural / slot if natural > 0 else 0.0
|
||||
audio_rate, video_ratio, status = _fit_one(need, params)
|
||||
|
||||
# (f) Timeline cursor — mirror the stretch_video layout loop:
|
||||
# pre-roll and gaps at 1.0×, the segment's video chunk
|
||||
# [start, effective_end] occupies slot × video_ratio.
|
||||
if i == 0:
|
||||
cursor = start # pre-roll preserved at native rate
|
||||
new_start = cursor
|
||||
new_end = new_start + slot * video_ratio
|
||||
cursor = new_end
|
||||
if i + 1 < n:
|
||||
# Unretimed sliver between this chunk and the next chunk's
|
||||
# start (the gap guard, or more if extend-only clamped).
|
||||
cursor += max(0.0, float(segments[i + 1]["start"]) - effective_end)
|
||||
|
||||
# Residual overflow after both knobs: stretched audio length vs the
|
||||
# segment's new video slot.
|
||||
stretched = natural / audio_rate if audio_rate > 0 else natural
|
||||
overflow_s = max(0.0, stretched - slot * video_ratio)
|
||||
if overflow_s <= 1e-6:
|
||||
overflow_s = 0.0
|
||||
elif status != "overflow_trimmed":
|
||||
status = "overflow_trimmed"
|
||||
|
||||
plan.segments.append(SegmentFit(
|
||||
index=i,
|
||||
seg_id=str(seg.get("id", f"seg_{i}")),
|
||||
audio_rate=round(audio_rate, 6),
|
||||
video_ratio=round(video_ratio, 6),
|
||||
new_start=round(new_start, 4),
|
||||
new_end=round(new_end, 4),
|
||||
orig_start=round(start, 4),
|
||||
orig_end=round(end, 4),
|
||||
effective_end=round(effective_end, 4),
|
||||
status=status,
|
||||
overflow_s=round(overflow_s, 4),
|
||||
))
|
||||
plan.video_plan.append({
|
||||
"orig_start": round(start, 4),
|
||||
"orig_end": round(effective_end, 4),
|
||||
"new_start": round(new_start, 4),
|
||||
"new_end": round(new_end, 4),
|
||||
"stretch_ratio": round(video_ratio, 4),
|
||||
})
|
||||
|
||||
# Tail (anything after the last segment's effective end) at 1.0×.
|
||||
last_eff = float(plan.segments[-1].effective_end)
|
||||
cursor += max(0.0, float(total_dur_s) - last_eff)
|
||||
plan.total_duration = round(max(cursor, float(total_dur_s)), 4)
|
||||
return plan
|
||||
@@ -1,65 +0,0 @@
|
||||
"""Map subtitle cues onto the Smart-Fit timeline (Wave 3.1 / Spec 1).
|
||||
|
||||
When a dub uses ``stretch_video`` mode, the video is re-timed per segment so
|
||||
the dubbed audio fits (see fit_planner + the export stretch filter). The
|
||||
dubbed audio therefore plays at *fitted* positions, not the original
|
||||
timestamps. A subtitle file exported with the original times would drift
|
||||
against the dubbed video — so we regenerate the cue timeline from the same
|
||||
plan the video stretch uses ("subtitles track actual dub placement", the
|
||||
last piece of Spec 1).
|
||||
|
||||
Pure functions — no I/O — so the remapping is unit-testable. The plan is the
|
||||
persisted ``video_stretch_plan`` list of
|
||||
``{orig_start, orig_end, new_start, new_end, stretch_ratio}`` chunks.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def map_time_to_fitted(t: float, plan: list[dict]) -> float:
|
||||
"""Map a time on the original timeline to its position on the fitted one.
|
||||
|
||||
Finds the plan chunk whose original span contains ``t`` and interpolates
|
||||
linearly into that chunk's fitted span (a chunk's stretch is uniform).
|
||||
Before the first chunk maps 1:1; after the last chunk the trailing offset
|
||||
is carried at 1:1 (the planner runs gaps/tail at rate 1.0). Empty plan ⇒
|
||||
identity.
|
||||
"""
|
||||
if not plan:
|
||||
return t
|
||||
for chunk in plan:
|
||||
o0 = float(chunk.get("orig_start", 0.0))
|
||||
o1 = float(chunk.get("orig_end", 0.0))
|
||||
n0 = float(chunk.get("new_start", o0))
|
||||
n1 = float(chunk.get("new_end", o1))
|
||||
if t < o0:
|
||||
# In a gap before this chunk — carry the offset at 1:1 from the
|
||||
# previous chunk's fitted end (or from 0 for the very first).
|
||||
return n0 - (o0 - t)
|
||||
if o0 <= t <= o1:
|
||||
span = o1 - o0
|
||||
if span <= 0:
|
||||
return n0
|
||||
return n0 + (t - o0) / span * (n1 - n0)
|
||||
# Past the last chunk: 1:1 tail from its fitted end.
|
||||
last = plan[-1]
|
||||
return float(last.get("new_end", 0.0)) + (t - float(last.get("orig_end", 0.0)))
|
||||
|
||||
|
||||
def fitted_cues(segments: list[dict], plan: list[dict]) -> list[tuple[float, float]]:
|
||||
"""Return ``[(start, end), ...]`` for each segment on the fitted timeline.
|
||||
|
||||
Monotonicity guard: a cue's end is never before its start, and successive
|
||||
starts never go backwards (rounding across chunk seams can't produce a
|
||||
non-monotone SRT).
|
||||
"""
|
||||
out: list[tuple[float, float]] = []
|
||||
prev_end = 0.0
|
||||
for seg in segments:
|
||||
s = map_time_to_fitted(float(seg.get("start", 0.0)), plan)
|
||||
e = map_time_to_fitted(float(seg.get("end", 0.0)), plan)
|
||||
s = max(s, prev_end if out else 0.0)
|
||||
e = max(e, s)
|
||||
out.append((s, e))
|
||||
prev_end = e
|
||||
return out
|
||||
@@ -22,32 +22,6 @@ import json
|
||||
|
||||
_GEN_INPUT_FIELDS = ("text", "target_lang", "profile_id", "instruct", "speed", "direction", "effect_preset")
|
||||
|
||||
# Pydantic fills `effect_preset` with this default when the client omits it,
|
||||
# while the client-side recompute (/tools/incremental) sends nothing. Both
|
||||
# representations must hash identically or every segment looks "stale" after
|
||||
# every generate and incremental re-dub degrades to a full re-dub (#281).
|
||||
_DEFAULT_EFFECT_PRESET = "broadcast"
|
||||
|
||||
|
||||
def _canon_value(field: str, value):
|
||||
"""Normalise one generation-input value so that the server-side view
|
||||
(pydantic-parsed `DubSegment`, defaults filled in) and the client-side
|
||||
view (raw segment dict, unset keys omitted) produce the same hash.
|
||||
|
||||
- missing / None / "" all mean "default" for string fields
|
||||
- `effect_preset` default is "broadcast" (pydantic fills it server-side)
|
||||
- numbers are coerced to float so `speed: 1` (JS) == `speed: 1.0` (pydantic)
|
||||
"""
|
||||
if field == "effect_preset":
|
||||
return value or _DEFAULT_EFFECT_PRESET
|
||||
if value is None or value == "":
|
||||
return ""
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
return value
|
||||
|
||||
|
||||
def segment_fingerprint(seg: dict) -> str:
|
||||
"""Deterministic hash of the inputs that actually affect TTS output.
|
||||
@@ -57,63 +31,11 @@ def segment_fingerprint(seg: dict) -> str:
|
||||
badge don't trigger regen, which is what we want.
|
||||
|
||||
Currently includes: text, target_lang, profile_id, instruct, speed,
|
||||
direction, effect_preset. Values are canonicalised (see `_canon_value`)
|
||||
so a fingerprint computed from the generate request (server defaults
|
||||
filled in) matches one recomputed later from the client's raw segment
|
||||
state — the root cause of #281's "1 edit re-dubs all N lines".
|
||||
direction, effect_preset.
|
||||
"""
|
||||
payload = {k: _canon_value(k, seg.get(k)) for k in _GEN_INPUT_FIELDS}
|
||||
payload = {k: (seg.get(k) if seg.get(k) is not None else "") for k in _GEN_INPUT_FIELDS}
|
||||
blob = json.dumps(payload, sort_keys=True, ensure_ascii=False)
|
||||
return hashlib.sha1(blob.encode("utf-8"), usedforsecurity=False).hexdigest()[:16]
|
||||
|
||||
|
||||
# ── Smart Fit (dub-length fitting v2) fingerprints ─────────────────────────
|
||||
#
|
||||
# Fitting parameters stay OUT of segment_fingerprint on purpose: changing a
|
||||
# fit knob (caps, gap guard, strategy) must trigger a RE-MIX of the already
|
||||
# rendered natural-rate WAVs (generate with regen_only=[]), never a re-TTS.
|
||||
# A separate per-track fingerprint tracks the fit configuration; a dubbed
|
||||
# track is stale iff its fit_fp differs OR any segment hash differs.
|
||||
|
||||
_FIT_PARAM_FIELDS = (
|
||||
"timing_strategy",
|
||||
"max_audio_only_rate",
|
||||
"audio_rate_cap",
|
||||
"video_slow_cap",
|
||||
"gap_guard_s",
|
||||
"allow_video_retime",
|
||||
)
|
||||
|
||||
# Server-side defaults (must mirror services.fit_planner.FitParams). Filled
|
||||
# in for omitted keys so a fingerprint computed from a fully-populated
|
||||
# server view matches one recomputed from a sparse client payload — the
|
||||
# same #281 regression class segment_fingerprint already guards against.
|
||||
_FIT_PARAM_DEFAULTS = {
|
||||
"timing_strategy": "smart_fit",
|
||||
"max_audio_only_rate": 1.2,
|
||||
"audio_rate_cap": 1.5,
|
||||
"video_slow_cap": 2.0,
|
||||
"gap_guard_s": 0.05,
|
||||
"allow_video_retime": True,
|
||||
}
|
||||
|
||||
|
||||
def fit_fingerprint(params: dict) -> str:
|
||||
"""Deterministic hash of the fit configuration for one dub track.
|
||||
|
||||
Canonicalised with the same `_canon_value` rules as segment
|
||||
fingerprints (int vs float, None/"" vs omitted), plus default-filling
|
||||
so `{}` and `{"audio_rate_cap": 1.5}` hash identically.
|
||||
"""
|
||||
params = params or {}
|
||||
payload = {}
|
||||
for k in _FIT_PARAM_FIELDS:
|
||||
v = params.get(k)
|
||||
if v is None or v == "":
|
||||
v = _FIT_PARAM_DEFAULTS[k]
|
||||
payload[k] = _canon_value(k, v)
|
||||
blob = json.dumps(payload, sort_keys=True, ensure_ascii=False)
|
||||
return hashlib.sha1(blob.encode("utf-8"), usedforsecurity=False).hexdigest()[:16]
|
||||
return hashlib.sha1(blob.encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
|
||||
def plan_incremental(
|
||||
|
||||
@@ -37,12 +37,6 @@ logger = logging.getLogger("omnivoice.llm")
|
||||
class LLMBackend(ABC):
|
||||
id: str = "base"
|
||||
display_name: str = "Base LLM"
|
||||
# LLM backends call out over the network (OpenAI/Ollama/LM Studio) or are a
|
||||
# no-op — none run a model on the user's GPU. So `gpu_compat` is empty and
|
||||
# list_backends() labels the family `effective_device:"network"` /
|
||||
# routing_status:"n/a" rather than asserting a false GPU claim. Routing is
|
||||
# never gated for LLM (see engines.select_engine + diagnose).
|
||||
gpu_compat: tuple[str, ...] = ()
|
||||
|
||||
@classmethod
|
||||
@abstractmethod
|
||||
@@ -112,22 +106,6 @@ class OpenAICompatBackend(LLMBackend):
|
||||
return self._client
|
||||
|
||||
def chat(self, *, system: str, user: str, timeout: Optional[float] = None) -> str:
|
||||
return self.chat_messages(
|
||||
messages=[
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
],
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
def chat_messages(self, *, messages: list[dict], timeout: Optional[float] = None) -> str:
|
||||
"""One-shot completion over a full message list.
|
||||
|
||||
Additive surface for callers that need structured few-shot turns
|
||||
(dictation refinement, Wave 2.1) — small local models pattern-match
|
||||
and echo inline examples, so examples must arrive as prior chat
|
||||
turns, not inside the system prompt.
|
||||
"""
|
||||
if timeout is None:
|
||||
try:
|
||||
timeout = float(os.environ.get("OMNIVOICE_LLM_TIMEOUT", "45"))
|
||||
@@ -136,7 +114,10 @@ class OpenAICompatBackend(LLMBackend):
|
||||
res = self._get_client().chat.completions.create(
|
||||
model=self.model_name,
|
||||
timeout=timeout,
|
||||
messages=messages,
|
||||
messages=[
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
],
|
||||
)
|
||||
return (res.choices[0].message.content or "").strip()
|
||||
|
||||
@@ -162,9 +143,6 @@ class OffBackend(LLMBackend):
|
||||
"to use features that need one (Cinematic translate, glossary auto-extract)."
|
||||
)
|
||||
|
||||
def chat_messages(self, **kw) -> str:
|
||||
return self.chat(**kw)
|
||||
|
||||
|
||||
_REGISTRY: dict[str, type[LLMBackend]] = {
|
||||
"openai-compat": OpenAICompatBackend,
|
||||
@@ -172,53 +150,15 @@ _REGISTRY: dict[str, type[LLMBackend]] = {
|
||||
}
|
||||
|
||||
|
||||
# Most-recent failure per backend (parity with tts/asr list_backends).
|
||||
_LAST_ERRORS: dict[str, str] = {}
|
||||
|
||||
_INSTALL_HINTS: dict[str, str] = {
|
||||
"openai-compat": "Set TRANSLATE_BASE_URL (+ TRANSLATE_API_KEY) — OpenAI, "
|
||||
"Ollama (http://localhost:11434/v1), or any compatible host.",
|
||||
}
|
||||
|
||||
|
||||
def list_backends() -> list[dict]:
|
||||
"""Same 11-key shape as tts/asr so the matrix renders families uniformly.
|
||||
|
||||
LLM is NOT a GPU family: every entry carries literal
|
||||
``effective_device:"network"`` / ``routing_status:"n/a"`` /
|
||||
``routing_reason:null`` (NOT via resolve_routing — that would be a false
|
||||
GPU claim). ``effective_device:"network"`` is a label, not a probe: nothing
|
||||
here touches the network (local-first).
|
||||
"""
|
||||
from core.scrub import scrub_text
|
||||
|
||||
out: list[dict] = []
|
||||
out = []
|
||||
for bid, cls in _REGISTRY.items():
|
||||
try:
|
||||
ok, msg = cls.is_available()
|
||||
except Exception as exc:
|
||||
ok = False
|
||||
msg = f"{type(exc).__name__}: {exc}"
|
||||
logger.warning(
|
||||
"llm list_backends: %s.is_available() raised — degrading "
|
||||
"gracefully so the picker still renders: %s", bid, msg,
|
||||
)
|
||||
if ok:
|
||||
_LAST_ERRORS.pop(bid, None)
|
||||
else:
|
||||
_LAST_ERRORS[bid] = scrub_text(msg)
|
||||
ok, msg = cls.is_available()
|
||||
out.append({
|
||||
"id": bid,
|
||||
"display_name": cls.display_name,
|
||||
"available": ok,
|
||||
"reason": None if ok else scrub_text(msg),
|
||||
"install_hint": _INSTALL_HINTS.get(bid),
|
||||
"last_error": _LAST_ERRORS.get(bid),
|
||||
"isolation_mode": "in-process",
|
||||
"gpu_compat": list(getattr(cls, "gpu_compat", ())),
|
||||
"effective_device": "network",
|
||||
"routing_status": "n/a",
|
||||
"routing_reason": None,
|
||||
"reason": None if ok else msg,
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user