Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
362dca9ae1 | ||
|
|
bbac66ef76 | ||
|
|
3d74611be8 | ||
|
|
ee08f56e1b |
@@ -1,32 +0,0 @@
|
||||
# RTK - Rust Token Killer (Google Antigravity)
|
||||
|
||||
**Usage**: Token-optimized CLI proxy for shell commands.
|
||||
|
||||
## Rule
|
||||
|
||||
Always prefix shell commands with `rtk` to minimize token consumption.
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
rtk git status
|
||||
rtk cargo test
|
||||
rtk ls src/
|
||||
rtk grep "pattern" src/
|
||||
rtk find "*.rs" .
|
||||
rtk docker ps
|
||||
rtk gh pr list
|
||||
```
|
||||
|
||||
## Meta Commands
|
||||
|
||||
```bash
|
||||
rtk gain # Show token savings
|
||||
rtk gain --history # Command history with savings
|
||||
rtk discover # Find missed RTK opportunities
|
||||
rtk proxy <cmd> # Run raw (no filtering, for debugging)
|
||||
```
|
||||
|
||||
## Why
|
||||
|
||||
RTK filters and compresses command output before it reaches the LLM context, saving 60-90% tokens on common operations. Always use `rtk <cmd>` instead of raw commands.
|
||||
+3
-119
@@ -7,26 +7,12 @@
|
||||
language: "en-US"
|
||||
early_access: false
|
||||
|
||||
# The review voice: a panel of senior domain experts, not a linter.
|
||||
tone_instructions: >-
|
||||
Review as a panel of principal engineers: ML inference, audio DSP, desktop
|
||||
systems, product polish. Cite exact lines, name the failure mode, give the
|
||||
concrete fix. No filler praise; raise nits only when they change a decision.
|
||||
|
||||
reviews:
|
||||
# "chill" keeps the bot from blocking merges — it comments, it does not gate.
|
||||
# Hard gating lives in CI (security.yml) and the constitution's human bar.
|
||||
profile: chill
|
||||
request_changes_workflow: false
|
||||
high_level_summary: true
|
||||
# Every walkthrough gets a visual: mermaid sequence diagrams for the
|
||||
# mechanics, plus (via the summary instructions) an ASCII before/after
|
||||
# sketch when the PR touches UI — so each PR is reviewable at a glance.
|
||||
sequence_diagrams: true
|
||||
high_level_summary_instructions: >-
|
||||
If the PR changes UI (JSX/TSX/CSS/Tauri windows), include a compact ASCII
|
||||
before/after sketch of the affected layout or component. If it changes
|
||||
behavior, include a short mermaid flowchart of the new mechanism.
|
||||
review_status: true
|
||||
poem: false
|
||||
|
||||
@@ -53,8 +39,7 @@ reviews:
|
||||
- "!**/*.onnx"
|
||||
- "!tests/fixtures/**"
|
||||
|
||||
# One expert lens per subsystem — encode what a passionate senior in each
|
||||
# domain would actually check, beyond what linters and CI already gate.
|
||||
# Encode the project's hard constraints so the bot reviews against them.
|
||||
path_instructions:
|
||||
- path: "**/*.{py,rs,js,jsx,ts,tsx}"
|
||||
instructions: >-
|
||||
@@ -63,119 +48,18 @@ reviews:
|
||||
model download, or an explicitly opt-in endpoint. Flag any code that
|
||||
persists or logs values matching *TOKEN*/*KEY*/*SECRET* or absolute user
|
||||
home paths (/Users/<name>/, C:\\Users\\<name>\\).
|
||||
- path: "backend/services/**/*.py"
|
||||
instructions: >-
|
||||
Review as an ML-inference/audio engineer. Check: thread-safety of model
|
||||
and cache state across the GPU worker pool; device/dtype assumptions
|
||||
that break on one of CUDA/MPS/ROCm/CPU; VRAM lifecycle (load/unload,
|
||||
leaks on the error path); sample-rate, channel-count and tensor-shape
|
||||
assumptions at engine boundaries; blocking calls inside async paths;
|
||||
model download/cache behavior when offline. Engine code must stay
|
||||
backward-compatible with already-installed on-disk model state.
|
||||
- path: "backend/**/*.py"
|
||||
instructions: >-
|
||||
Default features must behave identically on macOS, Windows and Linux.
|
||||
Platform-specific implementation is allowed, but a divergent user-visible
|
||||
default is a P0 bug — flag it and suggest an opt-in (Settings/env/flag).
|
||||
Any DB schema change must go through an alembic migration with an upgrade
|
||||
path; flag direct schema edits. The backend serves loopback HTTP: treat
|
||||
every query/path/form param as hostile (path traversal, log injection,
|
||||
CSRF from a browser tab), and never route user-chosen filesystem
|
||||
destinations through HTTP — that authorization belongs in the Tauri
|
||||
process.
|
||||
- path: "frontend/src/**/*.{js,jsx,ts,tsx}"
|
||||
instructions: >-
|
||||
Review as a product-minded senior frontend engineer. Check: stale state
|
||||
and races (async results landing after unmount or after newer requests);
|
||||
every user-visible failure has an actionable, non-technical error
|
||||
message; loading/disabled states during long operations. Every new
|
||||
user-facing string must be an i18n t('...') key present in ALL 21
|
||||
frontend/src/i18n/locales/*.json files — flag hardcoded UI strings and
|
||||
keys missing from any locale.
|
||||
- path: "frontend/src-tauri/**/*.rs"
|
||||
instructions: >-
|
||||
Review as a desktop-systems engineer. Check: every #[tauri::command] is
|
||||
callable from the webview — validate inputs and scope filesystem/process
|
||||
access accordingly; window and webview lifecycle on all three OSes;
|
||||
child-process spawn/exit-code/stderr handling; no unwrap/expect on
|
||||
user-controlled input; platform cfg blocks keep user-visible defaults
|
||||
identical across macOS/Windows/Linux.
|
||||
- path: "tests/**/*.py"
|
||||
instructions: >-
|
||||
Review as a test-infrastructure engineer. Check: the test would fail
|
||||
before the fix and pass after (no tautologies); no sleeps as
|
||||
synchronization; no module-level imports of app modules that go stale
|
||||
under sys.modules pollution (resolve at run time); TestClient instances
|
||||
are function-scoped and not lifespan-bound unless the test needs it;
|
||||
new functional CJK is allowlisted in tests/test_no_hardcoded_cjk.py with
|
||||
a justification.
|
||||
path; flag direct schema edits. Engine code must stay backward-compatible
|
||||
with already-installed on-disk model state (no forced reinstall).
|
||||
- path: ".github/workflows/**"
|
||||
instructions: >-
|
||||
Pin actions to a major version tag at minimum. Flag any workflow that
|
||||
grants write permissions it does not need.
|
||||
|
||||
# Non-gating pre-merge audits of the project's hard rules (warning mode —
|
||||
# the human owner is the gate, these make the checklist visible per-PR).
|
||||
pre_merge_checks:
|
||||
title:
|
||||
mode: warning
|
||||
requirements: >-
|
||||
Conventional-commit style with scope, e.g. "fix(dub): …", and the issue
|
||||
reference in title or body. Never propose a version bump.
|
||||
issue_assessment:
|
||||
mode: warning
|
||||
custom_checks:
|
||||
- name: "Cross-platform default parity"
|
||||
mode: warning
|
||||
instructions: >-
|
||||
If the PR changes any default-mode (out-of-the-box) behavior, verify
|
||||
it behaves identically on macOS, Windows and Linux, or is moved behind
|
||||
an explicit opt-in (Settings toggle, env var, or CLI flag). A
|
||||
platform-divergent default is a P0 per CLAUDE.md — fail this check
|
||||
and say which platform diverges.
|
||||
- name: "i18n completeness (21 locales)"
|
||||
mode: warning
|
||||
instructions: >-
|
||||
For every new or changed t('...') key in frontend code, verify the
|
||||
key exists in all 21 files under frontend/src/i18n/locales/. List any
|
||||
locale files missing the key. Also flag hardcoded user-facing strings
|
||||
that bypass i18n entirely.
|
||||
- name: "Local-first guarantee"
|
||||
mode: warning
|
||||
instructions: >-
|
||||
Verify the PR adds no required cloud calls, accounts, API keys or
|
||||
telemetry. Outbound traffic is only allowed to GitHub Issues (opt-in
|
||||
bug reporting) and HuggingFace model downloads. The app must remain
|
||||
fully functional offline and with reporting disabled.
|
||||
- name: "Backward compatibility"
|
||||
mode: warning
|
||||
instructions: >-
|
||||
Verify existing omnivoice_data/ (voices, projects, settings) and
|
||||
already-installed engine model state keep working without manual
|
||||
migration. Any DB schema change must ship an alembic migration with
|
||||
an upgrade path. Flag anything that would force users to reinstall an
|
||||
engine or re-download model weights.
|
||||
|
||||
finishing_touches:
|
||||
docstrings:
|
||||
enabled: true
|
||||
unit_tests:
|
||||
enabled: true
|
||||
|
||||
# Feed the bot the project constitution and docs, and let it accumulate
|
||||
# learnings from review conversations ("@coderabbitai always/never …").
|
||||
knowledge_base:
|
||||
code_guidelines:
|
||||
enabled: true
|
||||
filePatterns:
|
||||
- "CLAUDE.md"
|
||||
- "docs/**/*.md"
|
||||
learnings:
|
||||
scope: auto
|
||||
issues:
|
||||
scope: auto
|
||||
pull_requests:
|
||||
scope: auto
|
||||
|
||||
chat:
|
||||
auto_reply: true
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
name: 🐛 Bug Report
|
||||
about: Report a bug to help us improve OmniVoice Studio
|
||||
title: "[Bug] "
|
||||
labels: ["bug", "triage"]
|
||||
assignees: []
|
||||
---
|
||||
|
||||
## Describe the bug
|
||||
|
||||
A clear and concise description of what the bug is.
|
||||
|
||||
## To reproduce
|
||||
|
||||
Steps to reproduce the behavior:
|
||||
1. Go to '...'
|
||||
2. Click on '...'
|
||||
3. See error
|
||||
|
||||
## Expected behavior
|
||||
|
||||
What you expected to happen.
|
||||
|
||||
## Screenshots / Logs
|
||||
|
||||
If applicable, add screenshots or paste relevant logs from **Settings → Logs**.
|
||||
|
||||
> **Tip:** **Settings → About → "Save diagnostic bundle"** produces a zip
|
||||
> (self-check report, recent errors, scrubbed log tails) you can drag onto
|
||||
> this issue — it answers most environment questions below automatically.
|
||||
> Headless installs: `python backend/main.py --diagnose` prints the same
|
||||
> self-check (`--deep` also test-loads the active engine).
|
||||
|
||||
## Environment
|
||||
|
||||
- **OS:** [e.g. macOS 15.2, Windows 11, Ubuntu 24.04]
|
||||
- **Install method:** [Desktop app / Docker / From source]
|
||||
- **Version:** [e.g. v0.2.7 — check Settings → About]
|
||||
- **GPU:** [e.g. NVIDIA RTX 4090 / Apple M3 Pro / CPU only]
|
||||
- **RAM:** [e.g. 16 GB]
|
||||
- **Active TTS engine:** [e.g. omnivoice — check Settings → Engines]
|
||||
|
||||
## Additional context
|
||||
|
||||
Add any other context about the problem here.
|
||||
@@ -1,109 +0,0 @@
|
||||
name: 🐛 Bug report
|
||||
description: Something works incorrectly or crashes (not a first-run/install problem — use the install template for those).
|
||||
title: "[Bug] "
|
||||
labels: ["bug", "triage"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Thanks for helping improve OmniVoice Studio! 🎙️
|
||||
|
||||
**Fastest path to a fix:** **Settings → About → "Save diagnostic bundle"** makes a
|
||||
zip (self-check + recent errors + scrubbed log tails) — drag it onto this issue and
|
||||
most of the environment questions below are answered automatically.
|
||||
Headless: `python backend/main.py --diagnose` (add `--deep` to test-load the engine).
|
||||
- type: checkboxes
|
||||
id: preflight
|
||||
attributes:
|
||||
label: Before filing
|
||||
options:
|
||||
- label: I searched [existing issues](https://github.com/debpalash/OmniVoice-Studio/issues?q=is%3Aissue) and this isn't a duplicate.
|
||||
required: true
|
||||
- label: I'm on the latest release (or `main`) — older builds may already be fixed.
|
||||
required: false
|
||||
- type: textarea
|
||||
id: what-happened
|
||||
attributes:
|
||||
label: What happened?
|
||||
description: A clear description of the bug, including the exact error text / toast if any.
|
||||
placeholder: "Voice cloning failed with '…' after I clicked Generate."
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: repro
|
||||
attributes:
|
||||
label: Steps to reproduce
|
||||
value: |
|
||||
1.
|
||||
2.
|
||||
3.
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: expected
|
||||
attributes:
|
||||
label: What did you expect instead?
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: os
|
||||
attributes:
|
||||
label: Operating system
|
||||
options:
|
||||
- macOS (Apple Silicon)
|
||||
- macOS (Intel)
|
||||
- Windows (x64)
|
||||
- Linux (AppImage)
|
||||
- Linux (.deb)
|
||||
- Linux (other / from source)
|
||||
- Docker
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: install
|
||||
attributes:
|
||||
label: How did you install it?
|
||||
options:
|
||||
- Desktop app (installer / AppImage)
|
||||
- Docker image
|
||||
- From source (uv sync)
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: version
|
||||
attributes:
|
||||
label: Version
|
||||
description: Settings → About (e.g. v0.3.5), or the Docker tag / git SHA.
|
||||
placeholder: "v0.3.5"
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: device
|
||||
attributes:
|
||||
label: Compute device
|
||||
options:
|
||||
- NVIDIA GPU (CUDA)
|
||||
- AMD GPU (ROCm)
|
||||
- Apple Silicon (MPS)
|
||||
- Intel GPU (XPU)
|
||||
- CPU only
|
||||
- Not sure
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: engine
|
||||
attributes:
|
||||
label: Active TTS/ASR engine
|
||||
description: Settings → Engines (e.g. omnivoice, cosyvoice, indextts2, whisperx).
|
||||
placeholder: "omnivoice"
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Logs / diagnostic bundle
|
||||
description: Drag the diagnostic bundle here, or paste relevant lines from **Settings → Logs**. Secrets are scrubbed automatically.
|
||||
render: text
|
||||
- type: textarea
|
||||
id: extra
|
||||
attributes:
|
||||
label: Anything else?
|
||||
description: Screenshots, the input that triggered it, RAM/VRAM, etc.
|
||||
@@ -1,11 +0,0 @@
|
||||
blank_issues_enabled: false
|
||||
contact_links:
|
||||
- name: 💬 Discord — questions & quick help
|
||||
url: https://discord.gg/bzQavDfVV9
|
||||
about: Usage questions, setup help, and chat. Faster than an issue for "how do I…".
|
||||
- name: 🗣️ GitHub Discussions
|
||||
url: https://github.com/debpalash/OmniVoice-Studio/discussions
|
||||
about: Ideas, show-and-tell, and open-ended Q&A that isn't a bug or a specific feature ask.
|
||||
- name: 🔒 Security vulnerability
|
||||
url: https://github.com/debpalash/OmniVoice-Studio/security/policy
|
||||
about: Please report security issues privately — do NOT open a public issue.
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
name: ✨ Feature Request
|
||||
about: Suggest an idea for OmniVoice Studio
|
||||
title: "[Feature] "
|
||||
labels: ["enhancement"]
|
||||
assignees: []
|
||||
---
|
||||
|
||||
## Is your feature request related to a problem?
|
||||
|
||||
A clear description of what the problem is. Ex. "I'm always frustrated when..."
|
||||
|
||||
## Describe the solution you'd like
|
||||
|
||||
A clear description of what you want to happen.
|
||||
|
||||
## Describe alternatives you've considered
|
||||
|
||||
Any alternative solutions or features you've considered.
|
||||
|
||||
## Additional context
|
||||
|
||||
Add any other context, mockups, or screenshots about the feature request here.
|
||||
@@ -1,50 +0,0 @@
|
||||
name: ✨ Feature request
|
||||
description: Suggest an improvement or a new capability.
|
||||
title: "[Feature] "
|
||||
labels: ["enhancement"]
|
||||
body:
|
||||
- type: checkboxes
|
||||
id: preflight
|
||||
attributes:
|
||||
label: Before filing
|
||||
options:
|
||||
- label: I searched [existing issues](https://github.com/debpalash/OmniVoice-Studio/issues?q=is%3Aissue) and [discussions](https://github.com/debpalash/OmniVoice-Studio/discussions) for this idea.
|
||||
required: true
|
||||
- type: textarea
|
||||
id: problem
|
||||
attributes:
|
||||
label: What problem does this solve?
|
||||
description: The use case / friction this addresses ("When I … I can't …").
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: solution
|
||||
attributes:
|
||||
label: Proposed solution
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: alternatives
|
||||
attributes:
|
||||
label: Alternatives considered
|
||||
- type: dropdown
|
||||
id: area
|
||||
attributes:
|
||||
label: Area
|
||||
options:
|
||||
- Voice cloning
|
||||
- Voice design
|
||||
- Video dubbing
|
||||
- Real-time dictation
|
||||
- Audiobook / Stories (long-form)
|
||||
- TTS/ASR engines
|
||||
- Install / setup / packaging
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
> OmniVoice is **local-first** — features must work fully offline with no accounts,
|
||||
API keys, or cloud calls, and behave identically on macOS/Windows/Linux. Proposals
|
||||
that fit those constraints are easiest to land.
|
||||
@@ -1,86 +0,0 @@
|
||||
name: 🧩 Install / first-run problem
|
||||
description: The app won't install, set up, download models, or reach a first working output.
|
||||
title: "[Install] "
|
||||
labels: ["install", "triage"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
A first-run that *just works* is the whole point — sorry it didn't. Let's fix it.
|
||||
If the app launched far enough to open Settings, **Settings → About → "Save diagnostic
|
||||
bundle"** captures most of this; otherwise the fields below are enough.
|
||||
- type: dropdown
|
||||
id: stage
|
||||
attributes:
|
||||
label: Where did it fail?
|
||||
options:
|
||||
- App won't launch / blank or broken window
|
||||
- Python / uv environment bootstrap
|
||||
- Model download (HuggingFace)
|
||||
- Engine install (CosyVoice / IndexTTS / MLX / etc.)
|
||||
- First synthesis / dub never completes
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: error
|
||||
attributes:
|
||||
label: The error
|
||||
description: The exact message, traceback, or what you see on screen.
|
||||
render: text
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: os
|
||||
attributes:
|
||||
label: Operating system
|
||||
options:
|
||||
- macOS (Apple Silicon)
|
||||
- macOS (Intel)
|
||||
- Windows (x64)
|
||||
- Linux (AppImage)
|
||||
- Linux (.deb)
|
||||
- Linux (other / from source)
|
||||
- Docker
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: install
|
||||
attributes:
|
||||
label: How are you installing it?
|
||||
options:
|
||||
- Desktop app (installer / AppImage)
|
||||
- Docker image
|
||||
- From source (uv sync)
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: version
|
||||
attributes:
|
||||
label: Version
|
||||
placeholder: "v0.3.5 (or installer build / git SHA)"
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: network
|
||||
attributes:
|
||||
label: Network conditions (model/dependency downloads)
|
||||
description: Restricted networks are a known source of bootstrap failures (mirror fallback).
|
||||
options:
|
||||
- Normal / unrestricted
|
||||
- Behind a corporate proxy / firewall
|
||||
- Region with restricted access (e.g. China, Russia)
|
||||
- Offline / air-gapped
|
||||
- Not sure
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Logs / diagnostic bundle
|
||||
description: Drag a diagnostic bundle, or paste the install/bootstrap log. Headless self-check — `python backend/main.py --diagnose`.
|
||||
render: text
|
||||
- type: textarea
|
||||
id: tried
|
||||
attributes:
|
||||
label: What have you already tried?
|
||||
@@ -18,7 +18,7 @@
|
||||
- [ ] 📝 Documentation
|
||||
- [ ] 🧪 Tests
|
||||
- [ ] 🔧 CI / Build
|
||||
- [ ] 🚀 Release prep
|
||||
- [ ] 🚀 Release prep (RC or final)
|
||||
|
||||
## Testing
|
||||
|
||||
@@ -33,12 +33,17 @@
|
||||
- [ ] No local machine paths, logs, or personal env details in this PR
|
||||
- [ ] Version files are in sync (if version bump): `pyproject.toml`, `package.json`, `tauri.conf.json`, `Cargo.toml`
|
||||
- [ ] If this PR changes runtime behavior, the regression fixture at `tests/fixtures/omnivoice_data/` still loads green on the `smoke-matrix` CI job (macOS + Windows + Linux)
|
||||
- [ ] If this is part of a release, I've read the "Release cadence" section below and confirmed this PR targets the right RC
|
||||
|
||||
## Release cadence
|
||||
## Release cadence (read once per RC)
|
||||
|
||||
OmniVoice ships **continuous-to-main** — no release candidates, no soak windows.
|
||||
Every merged PR is immediately part of the rolling preview (`main`, Docker
|
||||
`:latest`, the desktop Preview channel). Versioned releases are tagged from
|
||||
`main` when it's ready; `main` then bumps to the next patch automatically.
|
||||
Users who want stability pin a release tag / Docker `:stable` / the desktop
|
||||
Stable channel.
|
||||
OmniVoice ships every minor on a **two-RC cadence**:
|
||||
- `vX.Y.0-rc1` — cut from `main` once all GATE-* requirements pass; clean-VM exercise on 4 OSes (per `REL-01`)
|
||||
- 48-hour soak (no new commits to release branch except fix-forward)
|
||||
- `vX.Y.0` — promotion if rc1 is clean
|
||||
|
||||
If your PR touches install / bootstrap / CI, it MUST land before rc1 cut, not between rc1 and the promotion. During a soak, any merge needs explicit OK from the release captain.
|
||||
|
||||
## Screenshots
|
||||
|
||||
<!-- If applicable, add screenshots or recordings. -->
|
||||
|
||||
@@ -92,10 +92,7 @@ jobs:
|
||||
|
||||
- name: Install frontend deps
|
||||
working-directory: frontend
|
||||
# --frozen-lockfile so a frontend/package.json change that forgets to
|
||||
# regenerate the root bun.lock fails HERE (fast) instead of only in the
|
||||
# Docker build (deploy/Dockerfile), which is what reddened main on #485.
|
||||
run: bun install --frozen-lockfile
|
||||
run: bun install
|
||||
|
||||
# checkJs is true in tsconfig for IDE feedback, but 947 pre-existing
|
||||
# JS errors remain. Override to false in CI so only .ts files block.
|
||||
@@ -177,10 +174,7 @@ jobs:
|
||||
|
||||
- name: Install frontend deps
|
||||
working-directory: frontend
|
||||
# --frozen-lockfile so a frontend/package.json change that forgets to
|
||||
# regenerate the root bun.lock fails HERE (fast) instead of only in the
|
||||
# Docker build (deploy/Dockerfile), which is what reddened main on #485.
|
||||
run: bun install --frozen-lockfile
|
||||
run: bun install
|
||||
|
||||
# tauri-build's setup hook reads tauri.conf.json's `frontendDist`
|
||||
# ("../dist"), which only exists after a frontend build. Without this,
|
||||
|
||||
@@ -5,21 +5,14 @@
|
||||
# - push to main branch → :main, :sha- (rolling "edge" build)
|
||||
# - workflow_dispatch → :sha- only (ad-hoc test build)
|
||||
#
|
||||
# Tag ↔ image mapping (versioning hard rule, owner-set 2026-06-11:
|
||||
# :latest IS the preview channel; stable users pin :stable or a version tag)
|
||||
# :latest — rolling preview: latest commit on main (always last release + 1 dev)
|
||||
# :main — alias of the same rolling main build (kept for back-compat)
|
||||
# :stable — most recent versioned release (set on every v* tag push)
|
||||
# :0.3.6 — exact version from the git tag
|
||||
# Tag ↔ image mapping
|
||||
# :latest — always the most recent versioned release (set on every v* tag push)
|
||||
# :0.3.0 — exact version from the git tag
|
||||
# :0.3 — major.minor floating tag (updated on every patch within the minor)
|
||||
# :main — latest commit on main; may be ahead of the last tagged release
|
||||
# :sha-xxxx — specific commit SHA; produced by workflow_dispatch
|
||||
#
|
||||
# Images land at: ghcr.io/debpalash/omnivoice-studio AND docker.io/palashdeb/omnivoice-studio
|
||||
# (Docker Hub push gated on the DOCKERHUB_USERNAME/DOCKERHUB_TOKEN secrets;
|
||||
# if unset the build still pushes to GHCR.)
|
||||
#
|
||||
# On main pushes the Docker Hub repository overview is also synced from
|
||||
# deploy/dockerhub-overview.md (source of truth for the hub.docker.com page).
|
||||
# Images land at: ghcr.io/debpalash/omnivoice-studio
|
||||
#
|
||||
# NOTE: the Docker image is the headless web-server build of OmniVoice (FastAPI
|
||||
# backend + pre-built React frontend served over HTTP). The Tauri desktop
|
||||
@@ -41,7 +34,6 @@ permissions:
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_NAME: ${{ github.repository }}
|
||||
DOCKERHUB_IMAGE: palashdeb/omnivoice-studio
|
||||
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
|
||||
|
||||
jobs:
|
||||
@@ -64,43 +56,28 @@ jobs:
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
# Docker Hub login — only when the secret is present, so forks / runs
|
||||
# without the credential still publish to GHCR.
|
||||
- name: Check Docker Hub credentials
|
||||
id: dockerhub
|
||||
run: echo "enabled=${{ secrets.DOCKERHUB_TOKEN != '' }}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Log in to Docker Hub
|
||||
if: steps.dockerhub.outputs.enabled == 'true'
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
# Tag strategy (`:sha-<short>` is emitted on every trigger):
|
||||
# v0.3.6 tag push → :0.3.6, :0.3, :stable, :sha-
|
||||
# main branch push → :latest, :main, :sha-
|
||||
# v0.3.0 tag push → :0.3.0, :0.3, :latest, :sha-
|
||||
# main branch push → :main, :sha-
|
||||
# workflow_dispatch → :sha- only
|
||||
#
|
||||
# All mutable-tag rules stay gated on `github.event_name == 'push'` so a
|
||||
# manual workflow_dispatch can only ever produce a throwaway `:sha-` tag
|
||||
# (the stale-:latest fix from #249/#251). :stable excludes prerelease
|
||||
# tags (those contain a `-`) so a prerelease can't clobber it.
|
||||
# Fix for stale :latest (issues #249, #251):
|
||||
# The previous rule used `enable={{is_default_branch}}`, which evaluates
|
||||
# to false on tag pushes (detached HEAD) — so :latest was never updated
|
||||
# when a release tag was pushed. The version / :latest / :main rules are
|
||||
# gated on `github.event_name == 'push'` so a manual workflow_dispatch can
|
||||
# only ever produce a throwaway `:sha-` tag (never republish a mutable
|
||||
# tag), and :latest additionally excludes prerelease tags (those contain a
|
||||
# `-`, e.g. v1.0.0-rc.1) so a prerelease can't clobber :latest.
|
||||
- name: Extract metadata (tags, labels)
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
# Same tag set applied to both registries. The Docker Hub line is
|
||||
# blank when the secret is unset, so metadata-action emits GHCR-only
|
||||
# tags in that case.
|
||||
images: |
|
||||
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
${{ steps.dockerhub.outputs.enabled == 'true' && env.DOCKERHUB_IMAGE || '' }}
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
|
||||
tags: |
|
||||
type=semver,pattern={{version}},enable=${{ github.event_name == 'push' }}
|
||||
type=semver,pattern={{major}}.{{minor}},enable=${{ github.event_name == 'push' }}
|
||||
type=raw,value=stable,enable=${{ github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-') }}
|
||||
type=raw,value=latest,enable=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
type=raw,value=latest,enable=${{ github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-') }}
|
||||
type=raw,value=main,enable=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
type=sha,prefix=sha-,format=short
|
||||
|
||||
@@ -114,25 +91,3 @@ jobs:
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
# Sync the Docker Hub repository overview from deploy/dockerhub-overview.md.
|
||||
# Only on main pushes (the overview tracks the rolling preview) and only
|
||||
# when Docker Hub creds are present, mirroring the push gating above.
|
||||
#
|
||||
# continue-on-error: the overview text is cosmetic, and the description
|
||||
# PATCH 403s unless DOCKERHUB_TOKEN carries description-edit scope (many
|
||||
# fine-grained Docker Hub tokens that can push still can't edit the
|
||||
# description). The image build+push is what matters — a creds-scope
|
||||
# mismatch on this cosmetic step must not fail the whole Docker run. To
|
||||
# actually sync the overview, use a token with read/write (incl.
|
||||
# description) scope, or the account password.
|
||||
- name: Update Docker Hub description
|
||||
if: steps.dockerhub.outputs.enabled == 'true' && github.event_name == 'push' && github.ref == 'refs/heads/main'
|
||||
continue-on-error: true
|
||||
uses: peter-evans/dockerhub-description@v4
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
repository: ${{ env.DOCKERHUB_IMAGE }}
|
||||
short-description: "Local ElevenLabs alternative: voice cloning, design & video dubbing in 646 languages. No API keys."
|
||||
readme-filepath: ./deploy/dockerhub-overview.md
|
||||
|
||||
@@ -1,97 +0,0 @@
|
||||
# Docs drift — daily inventory-vs-docs check with a single rolling issue.
|
||||
#
|
||||
# docs/features.yaml is the canonical inventory; scripts/check-docs-drift.py
|
||||
# diffs it against README.md, docs/, and the engine registries. On drift the
|
||||
# job updates (or creates) ONE issue labeled `docs-drift` in place — no issue
|
||||
# spam — and closes it automatically when the check is clean again.
|
||||
#
|
||||
# Companion to the PR-gating validate-install-docs.py step in ci.yml.
|
||||
# Spec: docs/competitive-analysis.md Spec 9a / parity program Wave 0.1.
|
||||
# Rolling-issue pattern adapted from Patter (MIT).
|
||||
|
||||
name: docs-drift
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Daily 03:30 UTC — after most merges, before EU morning triage.
|
||||
- cron: "30 3 * * *"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
drift:
|
||||
runs-on: ubuntu-22.04
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
- name: Install checker deps
|
||||
run: pip install "pyyaml>=6"
|
||||
|
||||
- name: Check inventory vs README/docs/registries
|
||||
id: drift
|
||||
continue-on-error: true
|
||||
run: python scripts/check-docs-drift.py --output drift-report.md
|
||||
|
||||
- name: Update rolling docs-drift issue
|
||||
uses: actions/github-script@v7
|
||||
env:
|
||||
DRIFT_OUTCOME: ${{ steps.drift.outcome }}
|
||||
with:
|
||||
script: |
|
||||
const fs = require('fs');
|
||||
const drifted = process.env.DRIFT_OUTCOME === 'failure';
|
||||
const { owner, repo } = context.repo;
|
||||
const label = 'docs-drift';
|
||||
|
||||
const open = await github.rest.issues.listForRepo({
|
||||
owner, repo, state: 'open', labels: label, per_page: 5,
|
||||
});
|
||||
|
||||
if (drifted) {
|
||||
let body = '';
|
||||
try {
|
||||
body = fs.readFileSync('drift-report.md', 'utf8');
|
||||
} catch {
|
||||
body = '# Docs drift report\n\nThe checker failed before writing a report — see the workflow run logs.';
|
||||
}
|
||||
body += `\n\n---\n_Last checked by [run ${context.runId}](https://github.com/${owner}/${repo}/actions/runs/${context.runId})._\n`;
|
||||
|
||||
if (open.data.length > 0) {
|
||||
await github.rest.issues.update({
|
||||
owner, repo, issue_number: open.data[0].number, body,
|
||||
});
|
||||
core.info(`Updated rolling issue #${open.data[0].number}`);
|
||||
} else {
|
||||
const created = await github.rest.issues.create({
|
||||
owner, repo,
|
||||
title: 'docs-drift: feature inventory vs docs mismatch',
|
||||
body,
|
||||
labels: [label, 'documentation'],
|
||||
});
|
||||
core.info(`Created rolling issue #${created.data.number}`);
|
||||
}
|
||||
} else {
|
||||
for (const issue of open.data) {
|
||||
await github.rest.issues.createComment({
|
||||
owner, repo, issue_number: issue.number,
|
||||
body: 'Drift resolved — nightly check is clean again. Closing automatically.',
|
||||
});
|
||||
await github.rest.issues.update({
|
||||
owner, repo, issue_number: issue.number, state: 'closed',
|
||||
});
|
||||
core.info(`Closed rolling issue #${issue.number}`);
|
||||
}
|
||||
}
|
||||
|
||||
- name: Surface drift as a failed run
|
||||
if: steps.drift.outcome == 'failure'
|
||||
run: |
|
||||
echo "Docs drift detected — see the rolling docs-drift issue."
|
||||
exit 1
|
||||
@@ -1,55 +0,0 @@
|
||||
# LLM-judge evals — semantic quality suites, NEVER a gate.
|
||||
#
|
||||
# Hard rule (parity program Wave 0.3 / competitive-analysis Spec 9b): LLM
|
||||
# judges never gate CI. This workflow is scheduled + manual only, the eval
|
||||
# step is continue-on-error, and the JSON report is the deliverable
|
||||
# (uploaded as an artifact). Deterministic probe judges in ci.yml remain
|
||||
# the only gates.
|
||||
#
|
||||
# On the hosted runner there is no local LLM endpoint, so the run usually
|
||||
# reports "skipped — no LLM backend configured"; the workflow exists so the
|
||||
# suites run anywhere a TRANSLATE_BASE_URL secret/endpoint is provided
|
||||
# (e.g. a self-hosted runner with Ollama).
|
||||
|
||||
name: evals
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Weekly, Sundays 04:00 UTC.
|
||||
- cron: "0 4 * * 0"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
evals:
|
||||
runs-on: ubuntu-22.04
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
- uses: astral-sh/setup-uv@v3
|
||||
with:
|
||||
enable-cache: true
|
||||
cache-dependency-glob: "uv.lock"
|
||||
|
||||
- name: Install deps
|
||||
run: uv sync
|
||||
|
||||
- name: Run eval suites (non-gating)
|
||||
continue-on-error: true
|
||||
env:
|
||||
TRANSLATE_BASE_URL: ${{ secrets.EVALS_LLM_BASE_URL }}
|
||||
TRANSLATE_API_KEY: ${{ secrets.EVALS_LLM_API_KEY }}
|
||||
run: uv run python tests/evals/run_evals.py --output eval-report.json
|
||||
|
||||
- name: Upload report artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: eval-report
|
||||
path: eval-report.json
|
||||
if-no-files-found: warn
|
||||
@@ -133,28 +133,9 @@ jobs:
|
||||
rust_target: aarch64-apple-darwin
|
||||
bundles: "app,dmg,updater"
|
||||
|
||||
# macOS Intel (#279): reinstated. The earlier "Rosetta 2 runs the
|
||||
# ARM build" rationale for dropping it was backwards — Rosetta only
|
||||
# translates x86_64→arm64, so Intel Macs (supported through macOS
|
||||
# Sequoia) simply cannot run the aarch64 bundle and had NO
|
||||
# installable artifact. Runner: `macos-15-intel`, GitHub's
|
||||
# designated migration target after macos-13 retired (Dec 2025);
|
||||
# it's a standard (public-repo-free) image supported through
|
||||
# August 2027 — the last x86_64 image Actions will offer. Building
|
||||
# natively (not cross-compiling from the arm64 leg) keeps the
|
||||
# per-TRIPLE uv/ffmpeg sidecar fetches, the DMG installer smoke,
|
||||
# and the ad-hoc signing verification (scripts/
|
||||
# verify-macos-signing.sh, PR #290) all exercising the real
|
||||
# x86_64 artifact on real Intel hardware. The macos-13 queue
|
||||
# backlog that motivated the original drop is contained by
|
||||
# fail-fast:false — a slow Intel leg can delay the release run but
|
||||
# can't fail the other targets.
|
||||
- os: macos-15-intel
|
||||
arch: x86_64-apple-darwin
|
||||
label: "macOS Intel"
|
||||
rust_target: x86_64-apple-darwin
|
||||
bundles: "app,dmg,updater"
|
||||
|
||||
# macOS Intel dropped: Apple shipped the last Intel Mac in 2023 and
|
||||
# Rosetta 2 runs the ARM build natively. macos-13 runner backlog
|
||||
# was also blocking every release tag for ~10 min.
|
||||
# Windows: force MSI bundling via --bundles. NSIS fails at makensis
|
||||
# because our PyInstaller payload approaches its ~2 GB stub limit.
|
||||
- os: windows-2022
|
||||
@@ -310,10 +291,7 @@ jobs:
|
||||
case "$TRIPLE" in
|
||||
aarch64-apple-darwin|x86_64-apple-darwin)
|
||||
# evermeet.cx ships each binary as a separate .zip containing
|
||||
# a single x86_64 Mach-O executable — natively correct on the
|
||||
# Intel leg, and runs via Rosetta 2 on the arm64 leg. Both
|
||||
# darwin TRIPLEs therefore bundle the same payload; only the
|
||||
# sidecar filename suffix differs.
|
||||
# a single x86_64 Mach-O executable (runs via Rosetta on arm64).
|
||||
for TOOL in ffmpeg ffprobe; do
|
||||
if [ "$TOOL" = "ffmpeg" ]; then
|
||||
URL="https://evermeet.cx/ffmpeg/getrelease/zip"
|
||||
@@ -420,12 +398,11 @@ jobs:
|
||||
# always reported the static 0.3.0 never looked "newer", so no update was
|
||||
# ever delivered). Ephemeral, CI-only — never committed. Tauri reads the
|
||||
# bundle + updater version from tauri.conf.json, so rewriting it here
|
||||
# stamps the artifacts + latest.json. Under the versioning hard rule
|
||||
# (owner-set 2026-06-11) main is always last-release + 1, so BASE-N is a
|
||||
# prerelease of the NEXT version and semver-sorts ABOVE the last stable
|
||||
# (0.3.6-N > 0.3.5) — preview users naturally upgrade past stable, and
|
||||
# the Windows MSI ProductVersion (which strips the prerelease → 0.3.6)
|
||||
# is also correctly above the last stable.
|
||||
# stamps the artifacts + latest.json. `0.3.0-preview.N` is a prerelease of
|
||||
# the current target, so previews converge to stable when 0.3.0 ships
|
||||
# (0.3.0 > 0.3.0-preview.N). NOTE: the Windows MSI ProductVersion strips
|
||||
# the prerelease (→ 0.3.0), a wrinkle to verify for win preview→preview
|
||||
# upgrades; mac/linux replace the bundle wholesale and are unaffected.
|
||||
- name: Stamp preview version
|
||||
if: github.event_name == 'workflow_dispatch' && inputs.publish_preview
|
||||
shell: bash
|
||||
@@ -454,12 +431,6 @@ jobs:
|
||||
# only on the opt-in stable path, leaving them ABSENT (not "") on
|
||||
# preview/unsigned paths so Tauri's bundler skips cert import. A static
|
||||
# env: here would always set them to "" and break the mac build.
|
||||
# Unsigned paths still get a VALID ad-hoc seal from tauri.conf.json
|
||||
# (bundle.macOS.signingIdentity = "-"), so a downloaded build shows the
|
||||
# GUI-bypassable "unidentified developer" prompt (right-click → Open /
|
||||
# Settings → "Open Anyway") instead of the un-bypassable "damaged"
|
||||
# error. On the signed path APPLE_SIGNING_IDENTITY (env) overrides the
|
||||
# "-" default; once notarized, Gatekeeper accepts it with no prompt.
|
||||
# GH runners disable FUSE, so linuxdeploy's AppImage can't mount
|
||||
# itself at bundle time. This env tells linuxdeploy to extract-and-run
|
||||
# instead, which works without FUSE.
|
||||
@@ -510,35 +481,6 @@ jobs:
|
||||
echo "OK — bundle has shell + uv + backend resources"
|
||||
hdiutil detach "$MOUNT" || true
|
||||
|
||||
# ── Signing / Gatekeeper / notarization verification ──────────────
|
||||
# Runs codesign --verify, spctl (Gatekeeper), nested-binary, and
|
||||
# stapler checks against the built .app (see docs/macos-signing-verification.md).
|
||||
# STRICT (--require-signed) only on the opt-in signed stable path — same
|
||||
# condition as "Configure Apple signing" above — so a failed or missing
|
||||
# signature/notarization FAILS the job and STOPS the release instead of
|
||||
# publishing an unsigned artifact. On every other (unsigned dev/preview)
|
||||
# path it runs report-only and never breaks the build.
|
||||
- name: Verify macOS signing
|
||||
if: runner.os == 'macOS'
|
||||
shell: bash
|
||||
env:
|
||||
STRICT: ${{ (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && vars.MACOS_SIGNING_ENABLED == 'true') && '1' || '0' }}
|
||||
APPLE_ID: ${{ secrets.APPLE_ID }}
|
||||
APPLE_PASSWORD: ${{ secrets.APPLE_PASSWORD }}
|
||||
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
|
||||
run: |
|
||||
set -uo pipefail
|
||||
APP=$(find "frontend/src-tauri/target/${{ matrix.rust_target }}/release/bundle/macos" -maxdepth 1 -name '*.app' | head -1)
|
||||
[ -n "$APP" ] || { echo "FAIL — no .app found to verify"; exit 1; }
|
||||
MODE=""
|
||||
if [ "$STRICT" = "1" ]; then
|
||||
MODE="--require-signed"
|
||||
echo "Signed stable release → STRICT verification (release stops on failure)."
|
||||
else
|
||||
echo "Unsigned dev/preview path → report-only verification."
|
||||
fi
|
||||
bash scripts/verify-macos-signing.sh "$APP" $MODE
|
||||
|
||||
- name: Installer smoke (Windows)
|
||||
if: runner.os == 'Windows'
|
||||
timeout-minutes: 5
|
||||
@@ -695,46 +637,3 @@ jobs:
|
||||
} > /tmp/preview-notes.md
|
||||
gh release edit preview --repo "$REPO" --notes-file /tmp/preview-notes.md
|
||||
echo "Applied auto-generated release notes + contributors to the preview release."
|
||||
|
||||
# ── Post-release version bump (versioning hard rule, owner-set 2026-06-11) ──
|
||||
# main is always last-release + 1 patch. The moment a stable v* tag is
|
||||
# released, bump the three version sources on main to the next patch so every
|
||||
# PR and preview build identifies as the next version. Pushes directly to
|
||||
# main with the workflow token (a metadata-only commit; CI runs on PRs).
|
||||
version-bump:
|
||||
if: github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-')
|
||||
runs-on: ubuntu-22.04
|
||||
permissions:
|
||||
contents: write
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
ref: main
|
||||
fetch-depth: 0
|
||||
- name: Bump main to released version + 1 patch
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
RELEASED="${GITHUB_REF_NAME#v}"
|
||||
IFS=. read -r MAJ MIN PAT <<< "$RELEASED"
|
||||
NEXT="$MAJ.$MIN.$((PAT + 1))"
|
||||
CURRENT=$(jq -r .version frontend/src-tauri/tauri.conf.json)
|
||||
if [ "$(printf '%s\n' "$NEXT" "$CURRENT" | sort -V | tail -1)" = "$CURRENT" ] && [ "$NEXT" != "$CURRENT" ]; then
|
||||
echo "main is already at $CURRENT (>= $NEXT) — nothing to bump"; exit 0
|
||||
fi
|
||||
tmp=$(mktemp)
|
||||
jq --arg v "$NEXT" '.version = $v' frontend/src-tauri/tauri.conf.json > "$tmp"
|
||||
mv "$tmp" frontend/src-tauri/tauri.conf.json
|
||||
# frontend/package.json drives __APP_VERSION__ (vite.config.js) — the
|
||||
# first-run footer + every auto bug report. Keep it in lockstep too,
|
||||
# set absolutely (jq) so any prior drift self-heals. (#248-sweep finding)
|
||||
tmp=$(mktemp)
|
||||
jq --arg v "$NEXT" '.version = $v' frontend/package.json > "$tmp"
|
||||
mv "$tmp" frontend/package.json
|
||||
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" frontend/src-tauri/Cargo.toml
|
||||
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" pyproject.toml
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
git add frontend/src-tauri/tauri.conf.json frontend/package.json frontend/src-tauri/Cargo.toml pyproject.toml
|
||||
git commit -m "chore(version): main -> $NEXT after $GITHUB_REF_NAME release"
|
||||
git push origin main
|
||||
|
||||
@@ -32,13 +32,9 @@ env:
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# PR branches: a new push cancels the superseded scan (no wasted runners).
|
||||
# main: every commit keeps its own group, so nothing is cancelled — a merge
|
||||
# train used to leave a permanent red ✗ ("cancelled") on every intermediate
|
||||
# commit in the history view even though nothing failed.
|
||||
concurrency:
|
||||
group: security-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || 'branch' }}
|
||||
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
|
||||
group: security-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
# ── Secret scanning (gating) ─────────────────────────────────────────────
|
||||
|
||||
@@ -1,17 +1,10 @@
|
||||
# SPIKE-02: Adopt `ModelsLab/omnivoice-singing` as singing variant of the existing engine
|
||||
|
||||
**Status:** ⚠️ **SUPERSEDED (2026-06-14)** by [`specs/006-dubbing-singing-mode/`](../../specs/006-dubbing-singing-mode/spec.md)
|
||||
**Date:** 2026-05-18 (superseded 2026-06-14)
|
||||
**Status:** Proposed (research-supported) — awaiting Phase 2 SubprocessBackend merge
|
||||
**Date:** 2026-05-18
|
||||
**Decision-makers:** [maintainer]
|
||||
**Related:** ROADMAP Phase 4; REQUIREMENTS SING-01..05; `.planning/phases/04-adaptive-specialty-engines-spike-first/04-RESEARCH.md`
|
||||
|
||||
> **Superseded:** This chose `ModelsLab/omnivoice-singing` for singing, but that
|
||||
> model has **no melody (F0/MIDI) conditioning** — it sings its own melody and
|
||||
> cannot follow the *source song* a dub must preserve. SoulX-Singer (arXiv
|
||||
> 2602.07803, published after this decision) provides F0/MIDI conditioning and is
|
||||
> selected in plan-06. This ADR stays valid only if reframed as an
|
||||
> expressive-TTS styling toggle, not melody-matched dubbing.
|
||||
|
||||
## Context
|
||||
|
||||
`ModelsLab/omnivoice-singing` (HuggingFace, 1,053 downloads/month, verified 2026-05-18) is a finetune of `k2-fsa/OmniVoice` — same Apache-2.0 license, same Qwen3-0.6B backbone, same Higgs Audio v2 codec at 24 kHz mono, same `omnivoice` PyPI library (0.1.5, 2026-04-28) already shipping in OmniVoice Studio v0.2.7. Trained on additional singing + emotion-tagged data and activated by a `[singing]` text control tag at generation time.
|
||||
|
||||
@@ -1,396 +0,0 @@
|
||||
---
|
||||
phase: 260613-fdl
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- pyproject.toml
|
||||
- backend/api/routers/setup/download.py
|
||||
- backend/api/routers/setup/models.py
|
||||
- backend/utils/hf_progress.py
|
||||
- backend/utils/download_aggregator.py # NEW
|
||||
- backend/services/segmented_download.py # NEW
|
||||
- backend/api/routers/system.py
|
||||
- frontend/src/pages/Settings.jsx
|
||||
- frontend/src/api/setup.ts
|
||||
- docs/downloading-models.md # NEW (docs-sync rule)
|
||||
- tests/backend/setup/test_download_preflight.py # NEW
|
||||
- tests/backend/services/test_segmented_download.py # NEW
|
||||
autonomous: true
|
||||
requirements:
|
||||
# ── Wave 0 — Spike / gate ───────────────────────────────────────────────────
|
||||
- FDL-00 # Classify all catalog repos Xet-backed vs legacy-LFS; the result sizes Wave 3
|
||||
# ── Wave 1 — Maximize + guarantee the Xet fast path (default, no new deps) ───
|
||||
- FDL-01 # Explicitly pin huggingface_hub>=1.7 + hf-xet in pyproject (today transitive/unpinned)
|
||||
- FDL-02 # Drive snapshot_download with explicit max_workers + tqdm_class + endpoint (not implicit monkeypatch)
|
||||
- FDL-03 # /system/info reports fast_download {xet_enabled, xet_version, high_performance}; logged at startup
|
||||
- FDL-04 # Opt-in HF_XET_HIGH_PERFORMANCE + HDD sequential-write toggles via prefs (env wins)
|
||||
# ── Wave 2 — Accurate downloaded/remaining + speed (the user-visible win) ────
|
||||
- FDL-05 # dry_run preflight -> emit install_plan {total_bytes, cached_bytes, to_download_bytes, n_files, n_cached}
|
||||
- FDL-06 # Backend aggregate tracker -> single 'aggregate' event {bytes_done, total_bytes, rate, eta, files_done/total}
|
||||
- FDL-07 # Frontend overall bar: speed + downloaded/remaining + ETA from aggregate; per-file detail collapsible; cached-skip shown
|
||||
# ── Wave 3 — Opt-in IDM-style accelerator for legacy-LFS repos ───────────────
|
||||
- FDL-08 # Custom httpx segmented downloader: parallel Range GETs, resume, auth-safe redirect, etag/sha verify, cancel (default OFF)
|
||||
- FDL-09 # Dispatch: accelerator ON + repo is LFS (not Xet) -> segmented path; else xet. Same aggregate progress + weight validation
|
||||
# ── Wave 4 — Opt-in mirror path + docs ───────────────────────────────────────
|
||||
- FDL-10 # Opt-in HF_ENDPOINT mirror setting (prefs); documented as classic-LFS fallback (no Xet); pairs with FDL-08
|
||||
- FDL-11 # Cancel-in-flight endpoint + cooldown interplay (composes with MM2-06 bounded cooldowns)
|
||||
- FDL-12 # docs/downloading-models.md (speed, fast-download status, HDD/high-perf toggles, mirror/restricted-network) + README pointer
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "Xet is the default download backend and is provably engaged: /system/info reports fast_download.xet_enabled=true with the hf_xet version, and a Xet-backed repo downloads via parallel chunk range-gets (not single-stream LFS)."
|
||||
- "Before any bytes flow, the UI shows an accurate denominator: total bytes to download, bytes already cached (skipped), and file count — sourced from snapshot_download(dry_run=True), not guessed from the first tqdm bar."
|
||||
- "During a download the UI shows ONE overall progress bar with instantaneous speed (sampled over a window, not a single file's rate), bytes downloaded / bytes remaining, and ETA — accurate even while Xet fetches many chunks/files in parallel."
|
||||
- "hf_transfer is NOT used or enabled anywhere (deprecated, breaks progress); the fast path is Xet only."
|
||||
- "The custom segmented downloader is OPT-IN (default off), only engages for non-Xet/legacy-LFS repos, never forwards the HF Authorization header to the redirected CDN host, verifies the downloaded file against its expected size/etag before marking complete, resumes a partial .part file, and can be cancelled mid-flight."
|
||||
- "Default download behavior is identical on macOS, Windows, Linux (Xet path, pure-Python). Every accelerator/mirror/high-perf knob is behind an explicit opt-in (Settings toggle or env var) per the cross-platform-parity strict rule — no bundled per-OS binary, no platform-divergent default."
|
||||
- "No new on-disk model-state format; existing HF cache layout and already-installed models are untouched; the segmented downloader writes into the same HF cache blob/snapshot structure (or hands off to it) so a model it fetches is indistinguishable from one snapshot_download fetched."
|
||||
- "uv run pytest tests/backend/setup/test_download_preflight.py tests/backend/services/test_segmented_download.py passes; existing download/install tests stay green."
|
||||
- "pyproject pins huggingface_hub>=1.7 and hf-xet explicitly; uv.lock resolves with single versions (uv tree shows no duplicate huggingface_hub)."
|
||||
artifacts:
|
||||
- path: "backend/utils/download_aggregator.py"
|
||||
provides: "Per-repo byte aggregator: sums bytes across parallel files/chunks, samples rate over a window, emits one 'aggregate' event"
|
||||
contains: "class DownloadAggregator AND def snapshot"
|
||||
- path: "backend/services/segmented_download.py"
|
||||
provides: "Opt-in multi-connection Range downloader for legacy-LFS repos (auth-safe, resume, verify, cancel)"
|
||||
contains: "async def segmented_download AND Range"
|
||||
- path: "backend/api/routers/setup/download.py"
|
||||
provides: "Driven snapshot_download (max_workers+tqdm_class+endpoint), dry_run preflight, dispatch to segmented path, cancel endpoint"
|
||||
contains: "dry_run AND tqdm_class"
|
||||
- path: "docs/downloading-models.md"
|
||||
provides: "User docs for download speed, fast-download status, HDD/high-perf toggles, mirror/restricted-network"
|
||||
contains: "Xet"
|
||||
key_links:
|
||||
- from: "install_model (download.py:122)"
|
||||
to: "snapshot_download(dry_run=True) preflight"
|
||||
via: "compute total/cached/remaining before the real download; emit 'install_plan'"
|
||||
pattern: "dry_run\\s*=\\s*True"
|
||||
- from: "snapshot_download / segmented_download byte updates"
|
||||
to: "DownloadAggregator -> single 'aggregate' SSE event"
|
||||
via: "tqdm_class forwards bytes into the aggregator; segmented path calls aggregator.add() directly"
|
||||
pattern: "aggregate"
|
||||
- from: "dispatch in install_model"
|
||||
to: "segmented_download vs snapshot_download"
|
||||
via: "prefs accelerator toggle AND repo-is-LFS classification (FDL-00 helper)"
|
||||
pattern: "segmented_download"
|
||||
- from: "system_info (system.py:245)"
|
||||
to: "fast_download status block"
|
||||
via: "probe hf_xet import + version + HF_XET_HIGH_PERFORMANCE"
|
||||
pattern: "fast_download"
|
||||
|
||||
---
|
||||
|
||||
<objective>
|
||||
Make model downloads as fast as possible AND show accurate speed / downloaded / remaining / ETA.
|
||||
|
||||
**Framing (validated by research — see 260613-fdl-RESEARCH below):** HuggingFace's **hf-xet** backend ALREADY implements the "IDM/uGet technique" — content-defined chunking, parallel byte-range fetches with adaptive concurrency, dedup, and automatic resume — and does it auth-safely. It ships by default in modern `huggingface_hub` and `hf_xet` is already installed here (huggingface_hub 1.7.2). HF closed the multi-connection-downloader feature request as "solved by Xet." So we do NOT build a custom segmented downloader as the default path; that would be redundant and would violate the cross-platform-parity rule.
|
||||
|
||||
What's actually missing:
|
||||
1. **We don't drive Xet well.** `install_model` calls `snapshot_download(**dl_kwargs)` with no `max_workers`, no `tqdm_class`, no `dry_run`, and no explicit dependency pin — progress rides on a global tqdm monkeypatch.
|
||||
2. **No pre-flight total**, so "downloaded/remaining" has no denominator until files appear, and aggregate speed is summed frontend-side from per-file events (inaccurate under parallel fetch).
|
||||
3. **Legacy non-Xet (LFS) repos get zero intra-file parallelism** — this is the one place a real IDM-style multi-connection fetch still helps, so we add it as an OPT-IN accelerator.
|
||||
|
||||
Five waves, in order (each independently shippable, continuous-to-main per v0.3.0 cadence):
|
||||
- **Wave 0 — Spike/gate (FDL-00):** classify every catalog repo Xet vs LFS. Sizes Wave 3's value; if ~all repos are Xet-backed, Wave 3 is low-priority polish.
|
||||
- **Wave 1 — Maximize + guarantee Xet (FDL-01..04):** pin deps, drive snapshot_download explicitly, surface fast-download status, opt-in high-perf/HDD knobs. No new deps, all platforms.
|
||||
- **Wave 2 — Accurate progress (FDL-05..07):** dry_run preflight + backend aggregate tracker + overall UI bar (speed/remaining/ETA). The biggest user-visible win.
|
||||
- **Wave 3 — Opt-in segmented accelerator (FDL-08..09):** custom httpx Range downloader for LFS repos. Default OFF, opt-in toggle.
|
||||
- **Wave 4 — Mirror path + docs (FDL-10..12):** opt-in HF_ENDPOINT, cancel endpoint, docs-sync.
|
||||
|
||||
Out of scope / explicitly rejected (call out, do NOT do):
|
||||
- **hf_transfer / HF_HUB_ENABLE_HF_TRANSFER** — deprecated, breaks progress callbacks. Never enable.
|
||||
- **Bundling aria2c** — per-OS GPLv2 binary + parity burden; the custom httpx path covers the same need without a binary.
|
||||
- **Making the segmented downloader the default** — redundant vs Xet, violates parity rule. Always opt-in.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/STATE.md
|
||||
@./CLAUDE.md
|
||||
@.planning/quick/260613-fdl-fast-model-downloads/260613-fdl-RESEARCH.md
|
||||
|
||||
# Files under edit (read before editing)
|
||||
@backend/api/routers/setup/download.py
|
||||
@backend/api/routers/setup/models.py
|
||||
@backend/utils/hf_progress.py
|
||||
@backend/api/routers/system.py
|
||||
@frontend/src/pages/Settings.jsx
|
||||
|
||||
# Reference only — patterns, do NOT modify
|
||||
@backend/core/prefs.py
|
||||
@frontend/src/api/setup.ts
|
||||
@frontend/src/api/hooks.ts
|
||||
|
||||
<interfaces>
|
||||
<!-- Verified during planning against the live env (huggingface_hub 1.7.2, hf_xet installed). -->
|
||||
|
||||
huggingface_hub 1.7.2 snapshot_download params (confirmed via inspect):
|
||||
repo_id, repo_type, revision, cache_dir, local_dir, library_name, library_version,
|
||||
user_agent, etag_timeout, force_download, token, local_files_only,
|
||||
allow_patterns, ignore_patterns, max_workers, tqdm_class, headers, endpoint, dry_run
|
||||
- dry_run=True -> returns per-file info incl. size + cached/not-cached (use for FDL-05 preflight).
|
||||
- tqdm_class=<cls> -> drives the AGGREGATE bar; Xet feeds bytes into it (this is the xet-aware progress hook).
|
||||
- max_workers -> parallel FILES (default 8); orthogonal to Xet intra-file chunk parallelism.
|
||||
- endpoint -> per-call HF endpoint override (FDL-10 mirror, instead of process-wide HF_ENDPOINT).
|
||||
|
||||
backend/utils/hf_progress.py (existing):
|
||||
- Monkeypatches huggingface_hub.utils.tqdm.tqdm -> TrackedTqdm (install() at startup).
|
||||
- register_listener/unregister_listener; emit(event); current_repo_id contextvar stamps events.
|
||||
- TrackedTqdm.update()/display() emit per-file {filename, downloaded, total, pct, rate, phase} throttled ~0.3s.
|
||||
- GAP: per-file only, no aggregate, no preflight total. Wave 2 adds the aggregator on top (keep TrackedTqdm; feed it).
|
||||
|
||||
backend/api/routers/setup/download.py (existing):
|
||||
- install_model (line 122): snapshot_download(**dl_kwargs) inside asyncio.to_thread; 5-retry backoff; heartbeat;
|
||||
_validate_snapshot_has_weights (line 55); _install_cooldowns (line 27, see MM2-06 for bounding).
|
||||
- SSE feed: GET /setup/download-stream (line 80) forwards hf_progress events.
|
||||
|
||||
backend/core/prefs.py:
|
||||
- resolve(key, *, env=None, default=None) (line 75) — env wins, then store, then default. Use for all new toggles.
|
||||
|
||||
Xet env knobs (research): HF_XET_HIGH_PERFORMANCE=1 (opt-in max throughput; needs RAM/bandwidth),
|
||||
HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1 (HDD), HF_XET_NUM_CONCURRENT_RANGE_GETS (default 16),
|
||||
HF_XET_DATA_PROGRESS_UPDATE_INTERVAL (200ms). hf_xet is 64-bit only.
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<!-- ════════════ WAVE 0 — SPIKE / GATE ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 0 (FDL-00): Classify catalog repos Xet vs LFS</name>
|
||||
<files>.planning/quick/260613-fdl-fast-model-downloads/260613-fdl-SPIKE.md</files>
|
||||
<action>
|
||||
For every repo in backend/config/models.yaml (25 entries), determine whether it's Xet-backed or legacy Git-LFS. Use huggingface_hub: `HfApi().repo_info(repo_id, files_metadata=True)` and inspect each LFS blob for xet info, OR call the model-info endpoint and check the `xetEnabled`/blob `xet` field. For gated/unavailable repos, record "unknown (gated/offline)".
|
||||
Write 260613-fdl-SPIKE.md: a table repo_id | role | backend (xet|lfs|unknown) | size, plus a one-line GO/LOW-PRIORITY verdict for Wave 3:
|
||||
- If the majority of *user-facing default* models (OmniVoice TTS, the default ASR) are Xet-backed -> Wave 3 is LOW priority (xet already fast); still build it for the LFS long tail.
|
||||
- If many defaults are still LFS -> Wave 3 is HIGH priority.
|
||||
This is read-only network classification — do not download anything (use repo_info, not snapshot_download).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>test -f .planning/quick/260613-fast-model-downloads/260613-fdl-SPIKE.md || test -f .planning/quick/260613-fdl-fast-model-downloads/260613-fdl-SPIKE.md && echo "spike written"</automated>
|
||||
</verify>
|
||||
<done>SPIKE.md lists every catalog repo with its storage backend and a GO/LOW-PRIORITY verdict for Wave 3.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 1 — MAXIMIZE + GUARANTEE THE XET FAST PATH ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1 (FDL-01): Pin huggingface_hub + hf-xet explicitly</name>
|
||||
<files>pyproject.toml</files>
|
||||
<action>
|
||||
Today huggingface_hub arrives transitively (1.7.2) and hf_xet is present but unpinned. Add explicit runtime pins so the fast path can never silently disappear on a resolve:
|
||||
- huggingface_hub>=1.7 (keep compatible with transformers>=5.3.0 already in deps)
|
||||
- hf-xet>=1.1 (the Xet backend; 64-bit only — fine for all OmniVoice targets)
|
||||
Do NOT add hf_transfer. Run `uv sync` then `uv tree huggingface_hub` to confirm a single resolved version (no duplicate). If a transitive constraint conflicts, prefer the higher version and note it in the SUMMARY.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "huggingface_hub\|hf-xet\|hf_xet\|hf-transfer\|hf_transfer" pyproject.toml</automated>
|
||||
<automated>uv run python -c "import huggingface_hub,hf_xet; print('hub',huggingface_hub.__version__,'xet ok')"</automated>
|
||||
</verify>
|
||||
<done>pyproject pins huggingface_hub>=1.7 and hf-xet; no hf_transfer; uv resolves cleanly with one huggingface_hub.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2 (FDL-02): Drive snapshot_download explicitly</name>
|
||||
<files>backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
In install_model's _do() (line ~148), build dl_kwargs with explicit, intentional args instead of the bare call:
|
||||
- tqdm_class=<the TrackedTqdm class> so progress is deterministic and xet-aware rather than relying solely on the global monkeypatch. Expose TrackedTqdm from hf_progress (add a getter, e.g. hf_progress.tracked_tqdm_class()).
|
||||
- max_workers: keep default 8 (don't crank — xet does intra-file parallelism; high max_workers multiplies buffer pressure). Make it prefs-overridable: prefs.resolve("download_max_workers", env="OMNIVOICE_DOWNLOAD_MAX_WORKERS", default=8).
|
||||
- endpoint=prefs.resolve("hf_endpoint", env="HF_ENDPOINT", default=None) — wires FDL-10 mirror without process-wide env.
|
||||
- Keep the existing 5-retry backoff, heartbeat, and _validate_snapshot_has_weights.
|
||||
Do not remove the global monkeypatch (other libs — transformers/mlx_whisper — still rely on it); this task just makes the install path drive its own tqdm_class explicitly.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "tqdm_class\|max_workers\|endpoint" backend/api/routers/setup/download.py</automated>
|
||||
<automated>uv run pytest tests/ -k "download or install" -q 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<done>install_model drives snapshot_download with explicit tqdm_class + max_workers + endpoint; retry/validate intact; tests green.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3 (FDL-03, FDL-04): fast_download status + opt-in xet knobs</name>
|
||||
<files>backend/api/routers/system.py, backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
- FDL-03: add a fast_download block to GET /system/info (system.py:245): {xet_enabled: bool, xet_version: str|None, high_performance: bool}. Probe by importing hf_xet (xet_enabled), reading its version, and reading the HF_XET_HIGH_PERFORMANCE env/pref. Must never throw (system_info is called on every Settings load). Log the same line once at startup ("fast download: Xet on (hf_xet X.Y), high_perf=...").
|
||||
- FDL-04: opt-in knobs via prefs, applied at process/download setup (env wins):
|
||||
high_performance = prefs.resolve("xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False)
|
||||
hdd_sequential = prefs.resolve("xet_hdd_sequential_write", env="HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY", default=False)
|
||||
When set, export the corresponding HF_XET_* env before the snapshot/segmented download runs. Both default OFF (high-perf can hurt low-RAM machines — surface that as a tooltip in Wave 2 UI).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>curl -s http://127.0.0.1:3900/system/info | python3 -c "import json,sys; print(json.load(sys.stdin).get('fast_download'))" 2>/dev/null || grep -n "fast_download" backend/api/routers/system.py</automated>
|
||||
</verify>
|
||||
<done>/system/info reports fast_download truthfully; high-perf + HDD knobs resolve via prefs with env precedence, default off; startup logs Xet status.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 2 — ACCURATE DOWNLOADED/REMAINING + SPEED ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 4 (FDL-05): dry_run preflight -> install_plan event</name>
|
||||
<files>backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
Before the real download in install_model, run snapshot_download(repo_id, dry_run=True, endpoint=...) on the worker thread. From the returned per-file info compute: total_bytes, cached_bytes (files already present), to_download_bytes, n_files, n_cached. Emit a new phase event:
|
||||
{repo_id, phase:"install_plan", total_bytes, cached_bytes, to_download_bytes, n_files, n_cached}
|
||||
This gives the UI an accurate denominator and a "M GB already cached, N GB to download" line BEFORE bytes flow. Wrap dry_run in try/except — if it fails (older/gated repo), emit install_plan with totals=None and fall back to today's behavior (denominator fills in as files appear). dry_run must respect the 'resolving' heartbeat (it can take a couple seconds).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "dry_run\|install_plan\|to_download_bytes" backend/api/routers/setup/download.py</automated>
|
||||
<automated>uv run pytest tests/backend/setup/test_download_preflight.py -q 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<done>An install emits install_plan with accurate total/cached/remaining before download; dry_run failure degrades gracefully to old behavior.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 5 (FDL-06): Backend aggregate progress tracker</name>
|
||||
<files>backend/utils/download_aggregator.py, backend/utils/hf_progress.py</files>
|
||||
<action>
|
||||
New backend/utils/download_aggregator.py: a per-repo DownloadAggregator that owns the TRUTH for overall progress, so the frontend stops summing potentially-misrouted per-file events.
|
||||
- Seeded by the install_plan totals (total_bytes, n_files).
|
||||
- add(filename, bytes_delta) / set_file(filename, downloaded, total): track bytes per file; bytes_done = sum.
|
||||
- Rate: sampled over a sliding window (e.g. last ~5-10s of (t, bytes_done) samples), not a single tqdm bar's rate. eta = remaining / rate.
|
||||
- snapshot() -> {repo_id, bytes_done, total_bytes, rate, eta_seconds, files_done, files_total, phase}.
|
||||
- Emits one throttled (~0.3-0.5s) phase:"aggregate" event via hf_progress.emit().
|
||||
Wire it: hf_progress's TrackedTqdm._emit_progress already has per-file (filename, downloaded, total) — also feed those into the active repo's aggregator (look up by current_repo_id). The segmented downloader (Wave 3) calls aggregator.add() directly. Keep the per-file events too (UI detail view) — aggregate is additive, not a replacement.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run python -c "from utils.download_aggregator import DownloadAggregator as A; a=A('r',total_bytes=100,files_total=2); a.set_file('f1',50,50); a.set_file('f2',25,50); s=a.snapshot(); print(s['bytes_done'], s['total_bytes'])"</automated>
|
||||
</verify>
|
||||
<done>DownloadAggregator sums bytes across parallel files, samples rate over a window, emits a single 'aggregate' event; fed by both tqdm and the segmented path.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 6 (FDL-07): Frontend overall progress bar</name>
|
||||
<files>frontend/src/pages/Settings.jsx, frontend/src/api/setup.ts</files>
|
||||
<action>
|
||||
- setup.ts: extend SetupProgressEvent phase union with "install_plan" | "aggregate" and their fields (total_bytes, cached_bytes, to_download_bytes, n_files, n_cached, bytes_done, rate, eta_seconds, files_done, files_total).
|
||||
- Settings.jsx ModelStoreTab: when an aggregate event arrives for a repo, render ONE overall progress row: a bar (bytes_done/total_bytes), instantaneous speed (format rate as MB/s), "X.X GB of Y.Y GB" downloaded/remaining, and ETA (mm:ss from eta_seconds). Seed the denominator from install_plan (show "M GB cached, N GB to download" before bytes flow). Keep the existing per-file rows as a collapsible "details" section instead of the primary display. Show a small "⚡ fast download" badge when /system/info fast_download.xet_enabled is true.
|
||||
- Prefer the backend aggregate's rate/eta over the frontend's own per-file ETA computation (Settings.jsx ~614-631) — replace that local ETA math with the aggregate fields; keep a fallback if no aggregate event has arrived yet.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd frontend && bun run typecheck 2>&1 | tail -15</automated>
|
||||
<automated>grep -n "aggregate\|install_plan\|eta_seconds\|fast download" frontend/src/pages/Settings.jsx frontend/src/api/setup.ts</automated>
|
||||
</verify>
|
||||
<done>UI shows one overall bar with live speed + downloaded/remaining + ETA from the aggregate event; per-file detail collapsible; fast-download badge; typecheck passes.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 3 — OPT-IN IDM-STYLE SEGMENTED ACCELERATOR (LFS REPOS) ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 7 (FDL-08): Custom httpx segmented downloader</name>
|
||||
<files>backend/services/segmented_download.py, tests/backend/services/test_segmented_download.py</files>
|
||||
<action>
|
||||
New backend/services/segmented_download.py — an OPT-IN multi-connection Range downloader for ONE file (the IDM/uGet technique) used only for legacy-LFS repos where Xet gives no intra-file parallelism. httpx is already a dep.
|
||||
Contract (async def segmented_download(url, dest, *, token, expected_size, expected_etag=None, num_connections=8, chunk_aggregator=None, cancel_event=None)):
|
||||
1. HEAD (or GET Range: bytes=0-0) the resolve URL to learn size + Accept-Ranges + the redirect target. If server doesn't honor Range (Accept-Ranges != bytes) -> fall back to a single streamed GET (still works, just not parallel).
|
||||
2. AUTH SAFETY (critical): send Authorization: Bearer <token> ONLY to the huggingface.co host. When the resolve URL 302-redirects to the CDN (cloudfront/etc.), do NOT forward Authorization to the CDN host — the presigned URL already carries auth. Follow redirects manually so you control header propagation per-host.
|
||||
3. Split expected_size into num_connections ranges; download each with Range: bytes=start-end concurrently (asyncio + httpx.AsyncClient). Write to dest+".part" at the right offsets (preallocate, or per-range temp files then concat).
|
||||
4. RESUME: if dest+".part" exists with a sidecar manifest of completed ranges, skip completed ranges.
|
||||
5. CANCEL: check cancel_event between chunks; on cancel, leave the .part for resume and raise CancelledError.
|
||||
6. VERIFY: after assembly, check size == expected_size and (if given) sha256/etag matches; only then atomically rename .part -> dest. On mismatch, raise (caller's retry/validate handles it).
|
||||
7. PROGRESS: call chunk_aggregator.add(filename, bytes_delta) as ranges complete bytes (feeds DownloadAggregator).
|
||||
Tests (use a local mock HTTP server / httpx MockTransport): honors Range + parallel assembly == single-GET bytes; falls back when Accept-Ranges absent; does NOT send Authorization to a different host on redirect; resumes from a partial .part; cancels and leaves resumable state; size/etag mismatch raises.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run pytest tests/backend/services/test_segmented_download.py -q 2>&1 | tail -20</automated>
|
||||
<automated>grep -n "Authorization\|Range\|cancel_event\|expected_size" backend/services/segmented_download.py</automated>
|
||||
</verify>
|
||||
<done>segmented_download fetches a file via parallel ranges, is auth-safe across the CDN redirect, resumes, cancels, and verifies size/etag before commit; all tests pass.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 8 (FDL-09): Dispatch — accelerator for LFS repos only</name>
|
||||
<files>backend/api/routers/setup/download.py, backend/api/routers/setup/models.py</files>
|
||||
<action>
|
||||
- models.py: add a small helper is_xet_backed(repo_id) -> bool|None (reuse FDL-00's classification approach; cache result). Used to decide the path.
|
||||
- download.py install_model dispatch:
|
||||
accelerator_on = prefs.resolve("segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=False)
|
||||
if accelerator_on and is_xet_backed(repo_id) is False:
|
||||
-> resolve each LFS file's URL via hf_hub_url + HfApi file metadata, download via segmented_download into the HF cache layout (or download to a temp dir then place via the cache API so the result is a normal cache entry), feeding the same DownloadAggregator. Run _validate_snapshot_has_weights at the end.
|
||||
else:
|
||||
-> existing snapshot_download path (xet).
|
||||
IMPORTANT: the segmented result MUST land in the same HF cache structure so /models install-state, delete, and is_cached() all keep working (truth: "indistinguishable from snapshot_download"). If matching the blob/snapshot symlink layout is too fiddly, the safe fallback is: segmented-download to a temp file, then hand the bytes to huggingface_hub so it finalizes the cache entry. Document the chosen approach in SUMMARY.
|
||||
Default OFF -> zero behavior change unless the user opts in.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "segmented_downloader\|is_xet_backed\|segmented_download" backend/api/routers/setup/download.py backend/api/routers/setup/models.py</automated>
|
||||
<automated>uv run pytest tests/ -k "download or install or model" -q 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<done>With the toggle ON, LFS repos download via the segmented path into the normal HF cache; Xet repos and the default (toggle OFF) use snapshot_download; install-state/delete unaffected.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 4 — MIRROR PATH + CANCEL + DOCS ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 9 (FDL-10, FDL-11): Mirror opt-in + cancel endpoint</name>
|
||||
<files>backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
- FDL-10: the endpoint= wiring from Task 2 already reads prefs hf_endpoint. Surface it as a setting and document (Task 10) that a mirror routes through the CLASSIC LFS path (no Xet) — so it pairs naturally with the FDL-08 segmented accelerator for speed on mirrors. No process-wide HF_ENDPOINT mutation; per-call endpoint only.
|
||||
- FDL-11: add POST /models/install/cancel {repo_id} that sets the repo's cancel_event (segmented path) and, for the snapshot path, best-effort marks the install cancelled (snapshot_download isn't trivially cancellable mid-file — at minimum stop retries and emit install_cancelled). Compose with MM2-06: on success OR cancel, clear the _install_cooldowns entry so a cancelled download isn't rate-limited. Emit phase:"install_cancelled".
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "install/cancel\|cancel_event\|install_cancelled\|hf_endpoint" backend/api/routers/setup/download.py</automated>
|
||||
</verify>
|
||||
<done>Per-call mirror endpoint wired (opt-in); cancel endpoint stops the segmented path and clears cooldown; emits install_cancelled.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 10 (FDL-12): Docs — downloading-models.md + README pointer</name>
|
||||
<files>docs/downloading-models.md, README.md</files>
|
||||
<action>
|
||||
Per the docs-sync hard rule, document the user-facing surface introduced here:
|
||||
- How fast downloads work (Xet on by default; what the ⚡ badge means; how to check via Settings/system info).
|
||||
- Advanced toggles: high-performance mode (warn: needs RAM/bandwidth, can hurt low-RAM machines), HDD sequential-write, max workers, segmented accelerator (opt-in, for legacy-LFS repos), and the mirror/restricted-network HF_ENDPOINT setting (note: mirror = classic LFS, no Xet; pair with the accelerator).
|
||||
- A short troubleshooting section (slow downloads, stuck at resolving, restricted networks/China).
|
||||
Add a one-line pointer from README.md to docs/downloading-models.md. Do NOT enable any opt-in by default in docs examples.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>test -f docs/downloading-models.md && grep -n "Xet\|HF_ENDPOINT\|high-performance\|segmented" docs/downloading-models.md | head</automated>
|
||||
<automated>grep -n "downloading-models" README.md</automated>
|
||||
</verify>
|
||||
<done>docs/downloading-models.md covers speed, status, all opt-in knobs, mirror/restricted-network, troubleshooting; README links it; no opt-in shown as default.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
Gate per wave; full set before the last PR:
|
||||
1. `uv run pytest tests/backend/setup/test_download_preflight.py tests/backend/services/test_segmented_download.py tests/ -k "download or install or model" -q` — green.
|
||||
2. Live smoke (backend running): an install emits install_plan (accurate total/cached/remaining) THEN aggregate events with rising bytes_done + a non-zero rate + decreasing ETA; on completion bytes_done == total_bytes.
|
||||
3. /system/info reports fast_download.xet_enabled=true with a version.
|
||||
4. Auth-safety unit test proves Authorization is NOT sent to a non-huggingface.co host on redirect.
|
||||
5. Default-off proof: with no opt-in set, an install uses snapshot_download (xet) — `OMNIVOICE_SEGMENTED_DOWNLOAD` unset means the segmented path is never taken.
|
||||
6. `uv tree huggingface_hub` shows one version; no hf_transfer anywhere (`grep -ri hf_transfer backend/` is empty).
|
||||
7. `cd frontend && bun run typecheck` passes.
|
||||
8. Cross-platform parity: the default path (Xet, pure-Python) is identical on all 3 OSes; every accelerator/mirror/high-perf knob is opt-in (Settings/env). No bundled binary added.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- Fast: Xet is pinned, engaged, and driven with explicit args; high-perf/HDD knobs available opt-in; legacy-LFS repos can use the opt-in segmented accelerator for real multi-connection speed.
|
||||
- Accurate: UI shows pre-flight total/cached/remaining, then one overall bar with live speed + downloaded/remaining + ETA sourced from a backend aggregate (not frontend guesswork).
|
||||
- Safe & compatible: no hf_transfer; segmented downloader is opt-in, auth-safe, resumable, verified, cancellable, and lands in the normal HF cache; default behavior identical on all 3 OSes; no new on-disk model state; existing installs untouched.
|
||||
- All listed tests + typecheck pass; docs updated in the same PR (docs-sync rule).
|
||||
</success_criteria>
|
||||
|
||||
<risks>
|
||||
- **Segmented downloader auth leak (FDL-08) — highest risk.** Forwarding the HF Authorization header to the CDN host on redirect would leak the token. Mitigation: manual redirect handling, per-host header allow-list (Authorization only to huggingface.co), and a dedicated unit test asserting no Authorization on the CDN hop. This is a must-have truth, not optional.
|
||||
- **Cache-layout mismatch (FDL-09).** If the segmented path writes files outside the HF cache blob/snapshot structure, /models install-state + delete + is_cached() break. Mitigation: prefer the temp-file-then-hand-to-huggingface_hub finalization approach over hand-rolling the symlink/blob layout; assert is_cached(repo_id) is true after a segmented install in a test.
|
||||
- **dry_run cost/availability (FDL-05).** dry_run adds a metadata round-trip and may not exist for gated/older repos. Mitigation: try/except -> totals=None fallback to current fill-in-as-you-go behavior; keep the resolving heartbeat so the UI isn't blank during preflight.
|
||||
- **Aggregate vs per-file double-count (FDL-06).** Feeding both tqdm per-file events and the aggregator risks the UI showing two competing numbers. Mitigation: aggregate is the single source of truth for the overall bar; per-file events only drive the collapsible detail view; the frontend's old per-file ETA math is removed (Task 6).
|
||||
- **High-performance mode hurting low-RAM machines (FDL-04).** HF_XET_HIGH_PERFORMANCE can need ~tens of GB RAM. Mitigation: default OFF, opt-in only, tooltip warning in the UI.
|
||||
- **Mirror + Xet confusion (FDL-10).** Users may expect Xet speed through a mirror; mirrors fall back to classic LFS. Mitigation: document explicitly; that's exactly why the segmented accelerator pairs with the mirror path.
|
||||
- **Scope: do not let the segmented path become default.** It's opt-in for LFS repos only. Xet stays the default; making it default would regress dedup + violate the parity rule.
|
||||
</risks>
|
||||
|
||||
<output>
|
||||
Write 260613-fdl-SPIKE.md (Task 0) and 260613-fdl-SUMMARY.md when done. SUMMARY must record: the Xet-vs-LFS catalog breakdown and how it changed Wave 3 priority; the cache-finalization approach chosen for the segmented path (and the is_cached-after-segmented test result); the exact new SSE event shapes (install_plan, aggregate); which opt-in prefs keys + env vars were added; and the auth-safety test output. Note any "use judgment" decision an executor made.
|
||||
|
||||
Docs-sync (CLAUDE.md hard rule): docs/downloading-models.md + README pointer ship in the SAME PR as the user-facing toggles (Task 10). If the Settings UI gains the new toggles, the docs describing them land together.
|
||||
</output>
|
||||
@@ -1,43 +0,0 @@
|
||||
# RESEARCH — Fast HuggingFace model downloads (2026)
|
||||
|
||||
**Date:** 2026-06-13 · **For:** 260613-fdl-PLAN.md
|
||||
|
||||
## Bottom line
|
||||
As of mid-2026 the fast path is **hf-xet, on by default** in modern `huggingface_hub`. Xet is itself a chunk-level, content-defined, massively-parallel downloader with adaptive concurrency — it **is** the "IDM/uGet-style segmented download," done for you and dedup-aware. `hf_transfer` is **deprecated**. Rolling your own segmented downloader or bridging to aria2 is **not worth it as a default**; the only thing we must build is (a) better driving + progress UI and (b) an **opt-in** segmented path for the legacy-LFS long tail (repos Xet doesn't back).
|
||||
|
||||
Installed in this repo: `huggingface_hub 1.7.2`, `hf_xet` present. `snapshot_download` here supports `max_workers`, `tqdm_class`, `endpoint`, `dry_run` (confirmed via inspect).
|
||||
|
||||
## 1. hf-xet — USE (default, no action needed beyond pinning)
|
||||
Content-defined chunks grouped into blocks ("xorbs") in a content-addressable store; download = send file SHA256 → get reconstruction metadata + presigned URLs → fetch needed xorb ranges **in parallel** → reassemble; already-present chunks skipped (dedup). Auto-used by `snapshot_download`/`hf_hub_download` for Xet-backed repos since huggingface_hub 0.32. 2–3× over Git-LFS, up to ~1 GB/s.
|
||||
Knobs (defaults already tuned): `HF_XET_NUM_CONCURRENT_RANGE_GETS` (16), adaptive concurrency ON (max 64), `HF_XET_DATA_MAX_CONCURRENT_FILE_DOWNLOADS` (8), chunk cache disabled by default (better for pure download), `HF_XET_HIGH_PERFORMANCE=1` (opt-in max throughput, needs RAM/bandwidth), `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1` (HDD). **64-bit only.**
|
||||
- https://huggingface.co/docs/huggingface_hub/en/guides/download
|
||||
- https://huggingface.co/docs/hub/en/xet/using-xet-storage
|
||||
- https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables
|
||||
|
||||
## 2. hf_transfer — AVOID (deprecated)
|
||||
`HF_HUB_ENABLE_HF_TRANSFER` flagged deprecated; Xet supersedes it. Historically **broke tqdm progress / had no callbacks** — directly conflicts with the accurate-progress goal. Successor for max throughput is `HF_XET_HIGH_PERFORMANCE=1`.
|
||||
- https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables
|
||||
- https://github.com/huggingface/hf_transfer/issues/63
|
||||
|
||||
## 3. huggingface_hub native concurrency — USE defaults
|
||||
`snapshot_download(max_workers=...)` = parallel FILES (default 8), orthogonal to Xet's intra-file chunk parallelism. For OmniVoice's 1–few-large-file models the win is mostly Xet's intra-file parallelism; don't crank max_workers (multiplies buffer pressure). Resume is automatic via cache + ETag (no `resume_download` flag to manage).
|
||||
- https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
|
||||
|
||||
## 4. Custom IDM-style Range downloader — AVOID as default, BUILD as opt-in for LFS
|
||||
`/resolve/<rev>/<file>` 302-redirects to CDN (Cloudfront) which honors Range + parallel byte-ranges. Catch: follow redirect, **do NOT forward Authorization to the CDN host** (presigned URL carries auth), verify ETag/sha256, auth on first hop only. Redundant vs Xet for Xet-backed repos (HF closed issue #3232 as "use Xet"), **but genuinely helps non-Xet/legacy-LFS repos** which get no intra-file parallelism. → our Wave 3 opt-in.
|
||||
- https://github.com/huggingface/huggingface_hub/issues/3232
|
||||
|
||||
## 5. aria2 — OPTIONAL, rejected for OmniVoice
|
||||
`aria2c -x16 -s16 -c --header="Authorization: Bearer <token>"` is 3–5× on plain LFS, but: no dedup (worse than Xet for Xet repos), per-OS GPLv2 binary to package (parity burden — would have to be opt-in anyway), stdout/RPC progress scraping. The custom httpx path covers the same need with no binary. → not bundled.
|
||||
- https://gist.github.com/padeoe/697678ab8e528b85a2a7bddafea1fa4f
|
||||
|
||||
## 6. Mirrors / HF_ENDPOINT — OPTIONAL, region-gated, breaks Xet
|
||||
`HF_ENDPOINT=https://hf-mirror.com` redirects Hub traffic (standard for China). **Xet CAS/presigned URLs point at HF infra → mirrors generally don't serve the Xet protocol → traffic falls back to classic LFS** (no dedup, no Xet parallelism). So mirror and Xet fast-path are mutually exclusive; the realistic China stack is mirror + LFS + (our opt-in) segmented accelerator. → our Wave 4 opt-in, per-call `endpoint=` not process-wide.
|
||||
|
||||
## 7. Progress / speed — USE `tqdm_class` (xet-aware) + `dry_run` preflight
|
||||
Unlike hf_transfer, **Xet reports progress through the same tqdm interface**; huggingface_hub aggregates per-file/thread bytes into a shared bar and feeds the `tqdm_class` you pass. So `snapshot_download(tqdm_class=...)` yields reliable aggregate bytes/total/rate/ETA even under parallel fetch. `snapshot_download(dry_run=True)` returns per-file sizes + cached flags → use for "will download X of Y, N GB" preflight. Speed sampling tunable via `HF_XET_DATA_PROGRESS_UPDATE_INTERVAL` (200ms).
|
||||
- https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
|
||||
- https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/_snapshot_download.py
|
||||
|
||||
## Recommended architecture (→ plan)
|
||||
Pin `huggingface_hub>=1.7` + `hf-xet`; let Xet be the default (it IS the IDM technique). Drive `snapshot_download(repo_id, tqdm_class=OmniVoiceProgress, max_workers=8, endpoint=<opt-in mirror>)`; `dry_run=True` first for total/remaining; aggregate bytes in a backend tracker → one overall bar (speed/remaining/ETA). Opt-in only: `HF_XET_HIGH_PERFORMANCE` (max speed), `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY` (HDD), a custom httpx **segmented downloader for legacy-LFS repos**, and an `HF_ENDPOINT` mirror (classic-LFS fallback). Never enable hf_transfer; never bundle aria2; never make the segmented path the default.
|
||||
@@ -1,28 +0,0 @@
|
||||
# SPIKE — FDL-00: Catalog Xet vs LFS classification
|
||||
|
||||
**Date:** 2026-06-13 · **Method:** HF API `GET /api/models/{repo}?expand[]=xetEnabled` (authoritative).
|
||||
|
||||
## Result: 25 / 25 catalog repos are Xet-backed
|
||||
|
||||
| backend | count |
|
||||
|---------|-------|
|
||||
| xet | 25 |
|
||||
| lfs | 0 |
|
||||
| unknown | 0 |
|
||||
|
||||
Every repo in `backend/config/models.yaml` — including both first-run defaults (`k2-fsa/OmniVoice` TTS, `Systran/faster-whisper-large-v3` ASR) — returns `xetEnabled: true`. Full list: all entries under TTS / ASR / Diarisation (k2-fsa, Systran×5, mlx-community×9, openai, nvidia×2, UsefulSensors×2, pyannote, OpenMOSS, KittenML, deepdml).
|
||||
|
||||
## Detection caveat (important for the executor)
|
||||
|
||||
The installed client is **huggingface_hub 1.7.2**, whose `repo_info(..., files_metadata=True)` siblings expose only `blob_id, lfs, rfilename, size` — **no `xet_file`, and no `xet_enabled` on the info object.** A first pass that inferred backend from siblings wrongly reported "0/25 xet, all LFS." Do **not** classify Xet status from `repo_info` siblings on this client version. The reliable signal is the Hub API `xetEnabled` expand field (used here) or `hf_xet` actually engaging at download time. Re-check after any `huggingface_hub` bump — newer versions surface `xet_enabled` directly.
|
||||
|
||||
## Verdict for Wave 3 (segmented accelerator): LOW priority
|
||||
|
||||
Because the entire current catalog is Xet-backed and `hf_xet` is installed, Xet already provides chunked parallel range-gets (the IDM/uGet behavior) for **every** model we ship. The custom segmented downloader (Wave 3) is therefore **not needed to speed up any current default model** — it remains valuable only for:
|
||||
- the **mirror / restricted-network path** (Wave 4: `HF_ENDPOINT` falls back to classic LFS, no Xet), and
|
||||
- any **future non-Xet repo** a user adds.
|
||||
|
||||
**Recommendation:** proceed with W1 (maximize/guarantee Xet) and W2 (accurate progress) as the real wins for today's catalog; keep W3 as opt-in, build it alongside W4's mirror path where it actually pays off. This matches the PLAN's original framing — confirmed, not changed.
|
||||
|
||||
## Consequence for W1/W2 framing
|
||||
W1 "guarantee the Xet fast path" is correctly the primary lever: these repos download via Xet **only if** the client engages it (hf_xet installed ✓ + huggingface_hub recent ✓). The W2 live smoke test should confirm Xet is actually used (fast parallel aggregate progress on a real install), since `xetEnabled=true` is a Hub-side capability, not proof the client took the Xet path.
|
||||
@@ -1,49 +0,0 @@
|
||||
# SUMMARY — FDL Waves 0–2 (fast model downloads)
|
||||
|
||||
**Date:** 2026-06-13 · **Scope shipped:** W0 (spike), W1 (maximize Xet), W2 (accurate progress). W3/W4 deferred.
|
||||
|
||||
## What landed
|
||||
|
||||
**W0 — spike (FDL-00).** Classified all 25 `models.yaml` repos via the HF API `xetEnabled` field → **25/25 Xet-backed** (incl. both first-run defaults). See `260613-fdl-SPIKE.md`. Verdict: Wave 3 (segmented accelerator) is **LOW priority** — Xet already gives parallel chunked transfer for every shipped model. Detection caveat recorded: `repo_info` siblings on hf_hub **1.7.2** expose no xet metadata; classify via the `xetEnabled` API field, not siblings.
|
||||
|
||||
**W1 — maximize + guarantee Xet (FDL-01..04).**
|
||||
- `pyproject.toml`: pinned `huggingface_hub>=1.7` + `hf-xet>=1.1` explicitly (was transitive/unpinned); no `hf_transfer`. Resolves to hf_hub 1.7.2 / hf-xet 1.4.2, single version.
|
||||
- `download.py`: `install_model` now drives `snapshot_download` with explicit `tqdm_class` (our progress-emitting subclass), `max_workers` (prefs `download_max_workers`, default 8), and `endpoint` (prefs `hf_endpoint` — W4 hook). `apply_xet_env()` applies opt-in `HF_XET_HIGH_PERFORMANCE` + `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY` (both default OFF, env wins).
|
||||
- `system.py`: `/system/info` now returns `fast_download {xet_enabled, xet_version, high_performance}`; logged once at startup.
|
||||
|
||||
**W2 — accurate downloaded/remaining + speed (FDL-05..07).**
|
||||
- Preflight `snapshot_download(dry_run=True)` → `compute_plan()` → `install_plan` SSE event with `total_bytes / cached_bytes / to_download_bytes / n_files / n_cached` **before bytes flow**. Degrades to totals=None on gated/older repos.
|
||||
- New `utils/download_aggregator.py`: one source of truth for overall progress. Fed by a byte-sink on the patched tqdm; distinguishes byte bars (unit 'B', keyed by bar id) from the "Fetching N files" count bar; emits one throttled `aggregate` event (bytes_done/total/windowed rate/eta/files).
|
||||
- Frontend `Settings.jsx` + `setup.ts`: overall bar driven by the aggregate; bar % = `max(byte%, file%)`; shows cached-skip + files-progress; `⚡ fast download` badge from `/system/info`. i18n keys added to `en.json`.
|
||||
|
||||
## Rebase reconciliation (main disabled Xet)
|
||||
Rebasing onto latest main surfaced that main now sets **`HF_HUB_DISABLE_XET=1`** (main.py) — a deliberate choice to force the classic LFS path because Xet's progress bypasses the tqdm hook (the exact limitation found here). Reconciled rather than fought:
|
||||
- `fast_download` status now reports the **runtime truth**: `xet_installed` + `xet_active` (active = installed AND not disabled) + `xet_enabled` alias. Default `xet_active=false`; the ⚡ badge only shows when Xet actually runs. Startup log: `downloads: Xet disabled → legacy LFS …`.
|
||||
- Docs rewritten: default backend is **legacy LFS for accurate progress**; Xet is opt-in via `HF_HUB_DISABLE_XET=0` (coarser progress). The hf-xet pin stays (harmless; ready for a future Xet progress hook).
|
||||
- Net: W2's progress is the value either way; W1's "maximize Xet" is dormant by main's design, not removed.
|
||||
|
||||
## Decisions / "use judgment" notes
|
||||
- **Xet progress limitation (verified by live smoke).** Under Xet + hf_hub 1.7.2 the per-file **byte** bars never advance `n` and never `close()` through our tqdm (Xet fetches chunks out-of-band). Only the **file-count bar** is live. So: mid-download the overall bar is **file-granular** (moves 0→N files), and `complete()` flushes `bytes_done` to the exact preflight total on success (verified: final `74420620/74420620`, files 4/4). True live byte-speed is only available on classic-LFS/mirror repos (W4). This is a real constraint, not a bug — documented here and worth surfacing in W4 docs.
|
||||
- Per-file detail kept inline (existing single-line summary, now aggregate-sourced) rather than a new collapsible panel — limited risk; can revisit.
|
||||
|
||||
## Drive-by fix
|
||||
- `download.py` imported no `os`, but `_validate_snapshot_has_weights` uses `os.walk` → latent `NameError` on every install. Added `import os`.
|
||||
|
||||
## Verification
|
||||
- `tests/backend/setup/test_download_preflight.py` — 10 pass (compute_plan splits, aggregator byte/count routing, close-credit, windowed rate/eta, registry feed + finish noop).
|
||||
- `pytest -k "download or install or model or engine or setup"` — 149 passed, 7 skipped, 0 failed.
|
||||
- `frontend typecheck:ci` — exit 0.
|
||||
- Live smoke (real install of `mlx-community/whisper-tiny-mlx`, then deleted): `install_plan` exact; aggregate files 0→1→4; final bytes==total; `/system/info` + startup log correct.
|
||||
|
||||
## W4 — mirror + cancel + docs (FDL-10..12, shipped)
|
||||
- **Mirror (FDL-10):** `snapshot_download(endpoint=…)` honours prefs `hf_endpoint` / env `HF_ENDPOINT` on both preflight and download — per-call, no process-wide mutation. Documented as the classic-LFS (non-Xet) path that restores continuous byte-speed.
|
||||
- **Cancel (FDL-11):** `POST /models/install/cancel {repo_id}` sets a cancel flag checked at each retry boundary → emits `install_cancelled`, clears the cooldown (cancel ≠ failure). Limitation: an in-flight single-file fetch isn't interruptible in hf_hub 1.7.2; cancel lands at the next retry boundary. Frontend treats `install_cancelled` as a terminator (clears row + refetch).
|
||||
- **Docs (FDL-12):** `docs/downloading-models.md` (Xet fast path, progress semantics incl. the byte-speed limitation, opt-in tuning knobs, mirror/restricted-network, cancel, troubleshooting) + README pointer. Docs-sync rule satisfied in-PR.
|
||||
|
||||
## W3 — opt-in segmented accelerator (FDL-08/09, shipped)
|
||||
Reprioritised from LOW to HIGH after the rebase: since main forces Xet off, the default path is single-stream legacy LFS, so a segmented downloader is the way to get **both** parallel speed and live byte progress.
|
||||
- `services/segmented_download.py`: async multi-connection Range downloader for one file — parallel byte-ranges, resume (`.part` + manifest), per-segment short-read truncation guard, optional sha256/etag verify, cancel, single-stream fallback when the server won't range. **Auth-safe**: the HF `Authorization` header goes only to `huggingface.co`/`hf.co`; never forwarded to a CDN host on redirect (unit-tested).
|
||||
- Dispatch (`download.py`): opt-in via prefs `segmented_downloader` / env `OMNIVOICE_SEGMENTED_DOWNLOAD` (default OFF). When on and Xet inactive, fetches each repo file into the HF cache mirroring `hf_hub_download` (blobs + snapshot symlinks + `refs/main`), feeding **real bytes** to the aggregator. Any failure falls back to `snapshot_download` — the accelerator can never break a correct install.
|
||||
- Verified live (accelerator ON): real mid-download byte progress (1.5 KB → 71 MB, rate ramping to **16.6 MB/s**), final `bytes_done == total`, `/models` shows `installed: True`, delete frees the right bytes.
|
||||
- Fixed a `complete()` double-count (was adding a full total on top of accumulated segmented bytes → 2×); now replaces byte bars so the sum is exactly total.
|
||||
- Tests: `tests/backend/services/test_segmented_download.py` (7 cases) covering parallel range reassembly, single-stream fallback, the auth header reaching only the HF host (never a CDN), size/truncation rejection, cancellation, and byte-callback totals — plus an aggregator double-count regression.
|
||||
@@ -1,363 +0,0 @@
|
||||
---
|
||||
phase: 260613-mm2
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- backend/services/tts_backend.py
|
||||
- backend/services/model_manager.py
|
||||
- backend/services/subprocess_backend.py
|
||||
- backend/services/model_lifecycle.py # NEW
|
||||
- backend/api/routers/system.py
|
||||
- backend/api/routers/setup/download.py
|
||||
- backend/api/routers/setup/models.py
|
||||
- tests/test_engines.py
|
||||
- tests/backend/services/test_model_lifecycle.py # NEW
|
||||
- tests/backend/services/test_subprocess_reaper.py
|
||||
autonomous: true
|
||||
requirements:
|
||||
# ── Tier 1 — Correctness (Wave 1) ──────────────────────────────────────────
|
||||
- MM2-01 # Registry reuses one active instance and calls unload() on engine switch
|
||||
- MM2-02 # Per-engine unload() overrides (in-process drop+free_vram; subprocess -> unload_sidecar)
|
||||
- MM2-03 # /model/loaded + /model/unload report ASR honestly; no unloadable:False-but-loaded lies
|
||||
# ── Tier 2 — Single lifecycle surface (Wave 2) ─────────────────────────────
|
||||
- MM2-04 # model_lifecycle facade owns list_loaded/unload/unload_all/free_vram across all 3 worlds
|
||||
- MM2-05 # Idle/timeout config unified through core.prefs.resolve (env still wins); no duplicated constants
|
||||
# ── Tier 3 — Robustness & observability (Wave 3) ───────────────────────────
|
||||
- MM2-06 # _install_cooldowns bounded (evict on success + TTL); no unbounded growth
|
||||
- MM2-07 # Snapshot weight validation is per-role, not one 5 MB magic number
|
||||
- MM2-08 # Subprocess sidecars self-report VRAM in pong; panel shows real MB, not 0
|
||||
- MM2-09 # scan_cache_dir() -> disk-walk fallback logs WHY it fell back (WinError #117/#118)
|
||||
|
||||
must_haves:
|
||||
truths:
|
||||
- "Switching the active TTS backend in Settings releases the outgoing engine's VRAM before the new one loads — verified by asserting the outgoing instance's unload() was called exactly once on switch."
|
||||
- "TTSBackend.unload() is overridden by OmniVoiceBackend (drops model ref + free_vram) and by every SubprocessBackend subclass (routes to unload_sidecar); all overrides are idempotent and safe before first generate()."
|
||||
- "/model/loaded never reports a model as loaded with a misleading unloadable flag: the ASR row's unloadable reflects whether it can actually be released independently of the TTS lifecycle."
|
||||
- "services.model_lifecycle is the single import surface for list_loaded()/unload(id)/unload_all()/free_vram(); system.py routers call it instead of re-enumerating models inline."
|
||||
- "Idle timeouts for the in-process model and subprocess sidecars resolve through core.prefs.resolve(... env=...) so an env var still wins and the Settings store can override; no module duplicates IDLE_TIMEOUT_SECONDS by hand."
|
||||
- "_install_cooldowns cannot grow without bound: entries are removed on successful install and stale entries are evicted by TTL."
|
||||
- "A live subprocess sidecar reports a non-zero vram_mb in /model/loaded when it actually holds GPU memory (pong carries the figure); CPU-only sidecars report 0 truthfully."
|
||||
- "When scan_cache_dir() raises and the code falls back to the on-disk walk, the reason is logged at WARNING with the exception type (the #117/#118 WinError-448 path is no longer silent)."
|
||||
- "uv run pytest tests/test_engines.py tests/backend/services/test_model_lifecycle.py tests/backend/services/test_subprocess_reaper.py tests/test_model_load_timeout.py passes."
|
||||
- "No on-disk model state changes; no new runtime dependency added; behavior degrades gracefully (not errors) on MPS/CPU where VRAM APIs are sparse."
|
||||
artifacts:
|
||||
- path: "backend/services/tts_backend.py"
|
||||
provides: "Active-instance reuse + unload-on-switch in get_active_tts_backend(); per-engine unload() overrides"
|
||||
contains: "_active_instance AND (def unload)"
|
||||
- path: "backend/services/model_lifecycle.py"
|
||||
provides: "Facade owning list_loaded/unload/unload_all/free_vram across in-process + subprocess models"
|
||||
contains: "def list_loaded AND def unload_all"
|
||||
- path: "backend/api/routers/system.py"
|
||||
provides: "Thin /model/loaded + /model/unload routers delegating to model_lifecycle"
|
||||
contains: "model_lifecycle"
|
||||
key_links:
|
||||
- from: "get_active_tts_backend() (tts_backend.py:1235)"
|
||||
to: "outgoing backend.unload()"
|
||||
via: "module-level _active_instance compared against newly-resolved active_backend_id()"
|
||||
pattern: "_active_instance"
|
||||
- from: "system.py /model/loaded + /model/unload (system.py:129, 210)"
|
||||
to: "model_lifecycle.list_loaded() / model_lifecycle.unload()"
|
||||
via: "import services.model_lifecycle"
|
||||
pattern: "model_lifecycle\\.(list_loaded|unload)"
|
||||
- from: "subprocess sidecar pong reply (subprocess_backend.py:435-438)"
|
||||
to: "list_live_sidecars() vram_mb field"
|
||||
via: "ping reply carries allocated VRAM measured inside the sidecar process"
|
||||
pattern: "vram_mb"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Clean up OmniVoice's model-management subsystem ("v2"). Today load / unload / list / free-VRAM each behave differently across three worlds — the in-process model (`model_manager.py`), the TTS backend registry (`tts_backend.py`), and subprocess sidecars (`subprocess_backend.py`) — with no single lifecycle owner. This produces one real user-facing bug (VRAM leak on engine switch), inaccurate VRAM/unloadable reporting, an unbounded cooldown dict, and a silent cache fallback.
|
||||
|
||||
This is **cleanup + correctness, not a rewrite.** The Wave 13 idle-reaper and the SubprocessBackend primitive are sound and stay. The `TTSBackend.unload()` contract already exists as a documented default no-op (`tts_backend.py:149`) explicitly deferred to "Phase 2"; this plan *is* that Phase-2 follow-through — wire the registry to call it, override it per engine, and unify the surrounding surface.
|
||||
|
||||
Three tiers, executed in order (each independently shippable, continuous-to-main per the v0.3.0 cadence):
|
||||
- **Wave 1 / Tier 1 — Correctness:** MM2-01..03. The VRAM leak on switch + honest unload reporting. Highest value; ship first.
|
||||
- **Wave 2 / Tier 2 — Single lifecycle surface:** MM2-04..05. Extract `model_lifecycle` facade + unify idle/timeout config.
|
||||
- **Wave 3 / Tier 3 — Robustness & observability:** MM2-06..09. Bounded cooldowns, per-role weight validation, sidecar VRAM self-report, cache-fallback logging.
|
||||
|
||||
Output: PRs on branches off `main` (one per wave is fine), each green on the listed pytest selection. No push until the orchestrator merges; tests added with each wave.
|
||||
|
||||
Out of scope (call out, do not touch): GPU-pool per-engine sizing (`model_manager.py:42`, `_GPU_VRAM_PER_JOB_GB`) and torch.compile tuning — those are performance, not cleanup, and carry regression risk against #278/#315.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
|
||||
@$HOME/.claude/get-shit-done/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/STATE.md
|
||||
@./CLAUDE.md
|
||||
|
||||
# Files under edit (read before editing)
|
||||
@backend/services/tts_backend.py
|
||||
@backend/services/model_manager.py
|
||||
@backend/services/subprocess_backend.py
|
||||
@backend/api/routers/system.py
|
||||
@backend/api/routers/setup/download.py
|
||||
@backend/api/routers/setup/models.py
|
||||
|
||||
# Reference only — establish patterns, do NOT modify
|
||||
@backend/core/prefs.py
|
||||
@tests/test_engines.py
|
||||
@tests/backend/services/test_subprocess_reaper.py
|
||||
|
||||
<interfaces>
|
||||
<!-- Verified during planning. Executor should use these directly. -->
|
||||
|
||||
backend/services/tts_backend.py
|
||||
- class TTSBackend(ABC) (line 59); unload() default no-op (line 149) — contract already documented:
|
||||
idempotent, synchronous, safe before first generate().
|
||||
- OmniVoiceBackend.__init__(self, model=None) (line 174); self._model reuses model_manager singleton.
|
||||
- _REGISTRY: dict[str, type[TTSBackend]] (line 1109, a _LazyRegistry).
|
||||
- active_backend_id() (line 1228) -> prefs.resolve("tts_backend", env="OMNIVOICE_TTS_BACKEND", default="omnivoice").
|
||||
- get_active_tts_backend(*, model=None) (line 1235) — builds a FRESH instance every call, no teardown. THE leak.
|
||||
|
||||
backend/core/prefs.py
|
||||
- resolve(key: str, *, env: Optional[str] = None, default: Any = None) -> Any (line 75) — env wins, then store, then default.
|
||||
|
||||
backend/services/model_manager.py
|
||||
- module global `model` (line 111); `_last_used`; free_vram() (line 678); idle_worker() (line 667).
|
||||
- IDLE_TIMEOUT_SECONDS imported from core.config (line 33); duplicated as _IDLE_TIMEOUT_SECONDS (line 114). Collapse.
|
||||
- offload_tts_for_asr() (line 701) / restore_tts_after_asr() — ad-hoc ASR<->TTS VRAM juggling; _diar_pipeline global.
|
||||
|
||||
backend/services/subprocess_backend.py
|
||||
- protocol op set (line 74); SIDECAR_IDLE_TIMEOUT_S = env-only float (line 107) — move to prefs.resolve.
|
||||
- list_live_sidecars() -> list[dict] (line 181); unload_sidecar(engine_id) (line 199); unload_all_sidecars() (line 205).
|
||||
- health/ping: _send({"op":"ping"}) then expect {"op":"pong"} (lines 435-438). Add vram_mb to the pong here AND
|
||||
in the sidecar entry-point that answers ping (search the sidecar worker for the "ping"->"pong" handler).
|
||||
|
||||
backend/api/routers/system.py
|
||||
- GET /model/loaded (line 129) — ~80 lines of inline enumeration of TTS/ASR/diar/sidecars. Replace body with
|
||||
model_lifecycle.list_loaded().
|
||||
- POST /model/unload/{model_id} (line 210) — handles "tts" | "diarization" | "sidecar:<id>" | "sidecars".
|
||||
Replace body with model_lifecycle.unload(model_id).
|
||||
|
||||
backend/api/routers/setup/download.py
|
||||
- _install_cooldowns dict (line 27) — unbounded. _validate_snapshot_has_weights (line 55) + _MIN_WEIGHT_BYTES 5 MB
|
||||
(line 45) — single magic number across roles.
|
||||
|
||||
backend/api/routers/setup/models.py
|
||||
- scan_cache_dir() with silent disk-walk fallback (~line 268-280) + _scan_cache_on_disk (line 177).
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<!-- ════════════ WAVE 1 / TIER 1 — CORRECTNESS ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1 (MM2-02): Per-engine unload() overrides</name>
|
||||
<files>backend/services/tts_backend.py</files>
|
||||
<action>
|
||||
The base-class `unload()` no-op already exists (tts_backend.py:149) with a documented contract. Override it where it matters. Do this BEFORE Task 2 — the registry switch (Task 2) calls these.
|
||||
|
||||
- OmniVoiceBackend (line 162): override `unload(self)`. Drop the local model ref (`self._model = None`) and, because OmniVoice shares the singleton owned by model_manager, also release that: `import services.model_manager as mm; mm.model = None; mm.free_vram()`. Idempotent — guard on `mm.model is not None` before free_vram(). Safe before first generate() (no-op when nothing loaded).
|
||||
- Every SubprocessBackend subclass: implement `unload(self)` on the SubprocessBackend base (subprocess_backend.py — the duck-typed `_is_subprocess_isolated` class) so all subclasses inherit it. It must call `unload_sidecar(self.id)` (force-shut this engine's sidecar; busy sidecars are skipped, never interrupted — existing semantics). Idempotent: unload_sidecar on a non-running engine returns 0, no raise.
|
||||
- In-process non-OmniVoice engines that hold their own model (e.g. KittenTTS/VoxCPM2 keep refs in __init__): override unload() to drop the ref + best-effort empty_cache via the existing free_vram() helper if they used GPU. Where an engine genuinely holds nothing resident, leave the base no-op (and note it in the SUMMARY so the future CI gate knows it's intentional, not missed).
|
||||
|
||||
Honor the contract comment verbatim: idempotent, synchronous, safe pre-load.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "def unload" backend/services/tts_backend.py backend/services/subprocess_backend.py</automated>
|
||||
<automated>uv run python -c "from services.tts_backend import OmniVoiceBackend; b=OmniVoiceBackend(); b.unload(); b.unload(); print('idempotent ok')"</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- OmniVoiceBackend.unload() drops both self._model and mm.model and calls free_vram(), guarded for idempotency.
|
||||
- SubprocessBackend.unload() routes to unload_sidecar(self.id); inherited by all subprocess engines.
|
||||
- Calling unload() twice, and before any generate(), never raises.
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2 (MM2-01): Registry reuses one active instance + unloads on switch</name>
|
||||
<files>backend/services/tts_backend.py</files>
|
||||
<action>
|
||||
Fix the leak at get_active_tts_backend() (line 1235). Today it builds a fresh instance every call with no teardown of the prior engine — switching engines (or repeated synth) leaks VRAM until GC. This is the root cause behind the #278 comment thread.
|
||||
|
||||
- Add a module-level cache: `_active_instance: TTSBackend | None = None` and `_active_instance_id: str | None = None`.
|
||||
- In get_active_tts_backend(): resolve `bid = active_backend_id()`. If `_active_instance is not None` and `_active_instance_id != bid`, call `_active_instance.unload()` (best-effort, wrap in try/except so a bad unload can't block the switch — log on failure) before discarding it.
|
||||
- Build the new instance, store it as `_active_instance` + `_active_instance_id = bid`, return it.
|
||||
- IMPORTANT subtlety: OmniVoiceBackend takes `model=`. When `model=` is passed (the caller already has a loaded model), do NOT cache that instance as the shared `_active_instance` blindly — it's a per-call view over the shared singleton. Keep current behavior for the `model=` path (return a fresh OmniVoiceBackend(model=model)) but still trigger unload() of a *different* outgoing engine first. Pick the simplest correct rule: the cache tracks the configured backend id; passing model= for the SAME id reuses, switching id always unloads the previous. Document the rule in a comment.
|
||||
- Add a module-level `reset_active_backend()` helper that unloads + clears the cache, for app shutdown and tests.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "_active_instance\|def reset_active_backend\|def get_active_tts_backend" backend/services/tts_backend.py</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- Switching backend id calls the outgoing instance's unload() exactly once before the new instance is built.
|
||||
- A bad/raising unload() is caught + logged, never blocks the switch.
|
||||
- reset_active_backend() exists and is idempotent.
|
||||
- The model= fast-path for OmniVoice still works (no double-load).
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3 (MM2-03): Honest /model/loaded + /model/unload for ASR</name>
|
||||
<files>backend/api/routers/system.py</files>
|
||||
<action>
|
||||
The ASR row (system.py:166-175) is reported as unloadable:False, vram_mb:0 even when loaded on GPU, and /model/unload doesn't expose the offload-to-CPU path. Make reporting truthful WITHOUT changing the ASR<->TTS lifecycle coupling (that coupling is intentional — offload_tts_for_asr/restore_tts_after_asr).
|
||||
|
||||
- ASR row: keep unloadable reflecting reality. If ASR truly cannot be released independently of TTS, keep unloadable:False but add a `note` field ("released with TTS") so the UI explains it rather than showing a dead button. Do not invent a separate ASR unload that breaks the WhisperX large-v3 offload path.
|
||||
- vram_mb: if ASR currently runs on CPU (device "cpu" in the row), 0 is correct — leave it but make the device value derive from where the pipe actually is, not a hardcoded "cpu".
|
||||
- This task is intentionally small; the bigger restructure is Task 4 (facade). Land MM2-03 as the honest-reporting fix, then Task 4 moves the enumeration into the facade.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run pytest tests/test_engines.py -q 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- No row reports loaded-but-with-a-misleading-unloadable flag; ASR carries an explanatory note when unloadable:False.
|
||||
- Device field reflects the actual device of the ASR pipe.
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 4 (MM2-01..03 tests): Wave 1 regression tests</name>
|
||||
<files>tests/test_engines.py</files>
|
||||
<action>
|
||||
Add tests proving the leak fix and the unload contract:
|
||||
- test_switching_backend_unloads_previous: monkeypatch two fake backends into _REGISTRY, set active to A (get_active_tts_backend), switch prefs to B, assert A.unload() was called exactly once before B is returned.
|
||||
- test_unload_is_idempotent_and_preload_safe: OmniVoiceBackend().unload() twice + before generate() never raises.
|
||||
- test_reset_active_backend_clears_cache: after reset_active_backend(), the next get_active_tts_backend() builds fresh.
|
||||
Reuse the existing fixture style in tests/test_engines.py (it already monkeypatches the registry / availability). Keep tests CPU-only (no real model load).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run pytest tests/test_engines.py -q 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<done>All three new tests pass; existing test_engines.py tests still green.</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 2 / TIER 2 — SINGLE LIFECYCLE SURFACE ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 5 (MM2-04): Extract services/model_lifecycle.py facade</name>
|
||||
<files>backend/services/model_lifecycle.py</files>
|
||||
<action>
|
||||
Create backend/services/model_lifecycle.py as the single owner of cross-world model lifecycle. It composes the existing pieces — it does NOT reimplement loading.
|
||||
|
||||
Public surface:
|
||||
- list_loaded() -> list[dict]: returns the unified rows currently assembled inline in system.py:129-207 (TTS, ASR, diarization, subprocess sidecars). Move that logic here verbatim first, then improve (MM2-03 note field, MM2-08 sidecar vram once Task 8 lands).
|
||||
- unload(model_id: str) -> dict: the dispatch currently inline in system.py:210-242 ("tts" | "diarization" | "sidecar:<id>" | "sidecars"). Move here; keep async-lock semantics for the in-process model (mm._model_lock).
|
||||
- unload_all() -> dict: unload every releasable model (in-process TTS + diar + all sidecars). New convenience used by app shutdown.
|
||||
- free_vram(): thin re-export of model_manager.free_vram() so callers have one import.
|
||||
Keep the "never let sidecar enumeration break the panel" try/except guard.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run python -c "import services.model_lifecycle as ml; print([f for f in ('list_loaded','unload','unload_all','free_vram') if hasattr(ml,f)])"</automated>
|
||||
</verify>
|
||||
<done>model_lifecycle exposes list_loaded/unload/unload_all/free_vram; logic moved out of system.py (not duplicated).</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 6 (MM2-04): Thin system.py routers + facade tests</name>
|
||||
<files>backend/api/routers/system.py, tests/backend/services/test_model_lifecycle.py</files>
|
||||
<action>
|
||||
- Replace the bodies of GET /model/loaded (line 129) and POST /model/unload/{model_id} (line 210) with calls to model_lifecycle.list_loaded() / model_lifecycle.unload(model_id). Preserve the exact response shapes (frontend hooks.ts useModelStatus/useFlushMemory + the flush dropdown depend on {models, count} and {unloaded, success, ...}). The 400 on unknown model_id stays.
|
||||
- New tests/backend/services/test_model_lifecycle.py: list_loaded with nothing loaded returns {models:[], count:0}; unload("tts") when not loaded returns success:False reason:"not loaded"; unload("sidecars") with no sidecars returns count:0; unknown id raises/400 path. Mock model_manager + subprocess_backend so no real models load.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>uv run pytest tests/backend/services/test_model_lifecycle.py -q 2>&1 | tail -20</automated>
|
||||
<automated>grep -n "model_lifecycle" backend/api/routers/system.py</automated>
|
||||
</verify>
|
||||
<done>system.py routers are thin delegations; response shapes unchanged; new facade tests pass.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 7 (MM2-05): Unify idle/timeout config through prefs.resolve</name>
|
||||
<files>backend/services/model_manager.py, backend/services/subprocess_backend.py</files>
|
||||
<action>
|
||||
- model_manager.py: remove the duplicated `_IDLE_TIMEOUT_SECONDS = IDLE_TIMEOUT_SECONDS` (line 114). Resolve at use-site in idle_worker() via prefs: `prefs.resolve("idle_timeout_seconds", env="OMNIVOICE_IDLE_TIMEOUT_S", default=IDLE_TIMEOUT_SECONDS)`. Keep core.config.IDLE_TIMEOUT_SECONDS as the default source.
|
||||
- subprocess_backend.py: replace the env-only `SIDECAR_IDLE_TIMEOUT_S` (line 107) read with prefs.resolve("sidecar_idle_timeout_seconds", env="OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S", default=300.0). Preserve "<=0 disables reaping" semantics and the existing reaper-start guard (line 222). Resolve lazily (function call), not at import, so a test/setting change takes effect — but keep a sensible cached default for the hot reaper loop.
|
||||
- Both must keep env precedence (env wins over store) — that's exactly what prefs.resolve already does.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "_IDLE_TIMEOUT_SECONDS\|prefs.resolve\|SIDECAR_IDLE_TIMEOUT" backend/services/model_manager.py backend/services/subprocess_backend.py</automated>
|
||||
<automated>uv run pytest tests/backend/services/test_subprocess_reaper.py -q 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<done>
|
||||
- No hand-duplicated IDLE_TIMEOUT constant; both timeouts resolve via prefs with env precedence.
|
||||
- Reaper "<=0 disables" + busy-skip behavior unchanged; all 10+ reaper tests still pass.
|
||||
</done>
|
||||
</task>
|
||||
|
||||
<!-- ════════════ WAVE 3 / TIER 3 — ROBUSTNESS & OBSERVABILITY ════════════ -->
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 8 (MM2-08): Subprocess sidecars self-report VRAM in pong</name>
|
||||
<files>backend/services/subprocess_backend.py</files>
|
||||
<action>
|
||||
Sidecar VRAM is reported as 0 (system.py:192-203 / list_live_sidecars) because the parent can't measure a child's GPU memory. Have the child measure itself.
|
||||
|
||||
- In the sidecar worker's ping handler (the code that answers {"op":"ping"} with {"op":"pong"} — find it in the sidecar entry-point module), include `vram_mb`: measure inside the child via torch.cuda.memory_allocated() (CUDA) or torch.mps.driver_allocated_memory() (MPS, guarded), else 0. Same degrade-gracefully pattern as system.py:147-156.
|
||||
- Parent: in the health-check ping/pong path (subprocess_backend.py:435-438), capture reply["vram_mb"] and stash it on the sidecar record so list_live_sidecars() (line 181) can surface it. Refresh opportunistically on each successful ping; default to last-known or 0 if never measured.
|
||||
- Keep the contract that enumeration never breaks the panel.
|
||||
This is CUDA/MPS-aware and degrades to 0 on CPU — honoring cross-platform parity (default behavior identical; the number is just more accurate where the API exists).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "vram_mb" backend/services/subprocess_backend.py</automated>
|
||||
<automated>uv run pytest tests/backend/services/test_subprocess_reaper.py -q 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<done>list_live_sidecars() exposes a vram_mb sourced from the child's own measurement; 0 only when truly CPU/unmeasured; reaper tests still green.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 9 (MM2-06, MM2-07): Bounded cooldowns + per-role weight validation</name>
|
||||
<files>backend/api/routers/setup/download.py</files>
|
||||
<action>
|
||||
- MM2-06: _install_cooldowns (line 27) grows unbounded. On a successful install, delete the repo's cooldown entry. Add a TTL sweep: when reading/writing the dict, evict entries older than a fixed window (reuse the existing cooldown window constant; pick the larger of cooldown-window and e.g. 1h). Keep it simple — a dict + timestamps, swept on access. No new dep.
|
||||
- MM2-07: _validate_snapshot_has_weights (line 55) + _MIN_WEIGHT_BYTES 5 MB (line 45) is one magic number for all roles. Make the threshold per-role/per-extension: safetensors/bin/ckpt expect the existing floor; .onnx models (kittentts, supertonic, sherpa) can be legitimately smaller — set a lower, role-aware floor so a valid small ONNX model isn't flagged as truncated. Keep the #352 truncation-catch intent (catch a 0-byte / KB-sized partial), just stop false-positiving small-but-complete models.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "_install_cooldowns\|_MIN_WEIGHT_BYTES\|def _validate_snapshot_has_weights" backend/api/routers/setup/download.py</automated>
|
||||
<automated>uv run pytest tests/ -k "download or install or model" -q 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<done>Cooldown dict is bounded (evict-on-success + TTL sweep); weight validation floor varies by role/extension; #352 truncation still caught.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 10 (MM2-09): Log why scan_cache_dir() fell back to disk walk</name>
|
||||
<files>backend/api/routers/setup/models.py</files>
|
||||
<action>
|
||||
The scan_cache_dir() -> _scan_cache_on_disk() fallback (~line 268-280, helper at line 177) silently swallows the exception — this is the #117/#118 Windows WinError-448 path. Wrap the fallback so it logs at WARNING with the exception type and a one-line reason ("scan_cache_dir failed (%s); falling back to on-disk walk of %s") before walking. Do not change the fallback behavior itself — just stop it being invisible in logs. Keep it from ever raising out (the panel must still render).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>grep -n "falling back\|logger.warning\|_scan_cache_on_disk\|scan_cache_dir" backend/api/routers/setup/models.py | head</automated>
|
||||
</verify>
|
||||
<done>The disk-walk fallback logs a WARNING naming the exception type; behavior otherwise unchanged; never raises out.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<verification>
|
||||
Full-suite gate after each wave (run the relevant subset per wave, full set before the last PR):
|
||||
|
||||
1. `uv run pytest tests/test_engines.py tests/backend/services/test_model_lifecycle.py tests/backend/services/test_subprocess_reaper.py tests/test_model_load_timeout.py tests/test_model_manager_preload.py -q` — all green.
|
||||
2. `uv run pytest tests/ -k "download or install or model or engine" -q` — green (Tier 3 touch points).
|
||||
3. Response-shape guard: GET /model/loaded still returns {models, count}; POST /model/unload returns {unloaded, success, ...}; 400 on unknown id. (Covered by test_model_lifecycle.py.)
|
||||
4. No new runtime dependency: `git diff pyproject.toml uv.lock` is empty.
|
||||
5. Localization/CJK + redaction gates unaffected: `uv run pytest tests/test_no_hardcoded_cjk.py -q`.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- Tier 1: switching the active backend releases the previous engine's VRAM (unload() called once on switch); contract overridden for OmniVoice + all subprocess engines; ASR reporting is honest. (MM2-01..03)
|
||||
- Tier 2: services.model_lifecycle is the single lifecycle surface; system.py routers are thin delegations with unchanged response shapes; idle/timeout config flows through prefs.resolve with env precedence and no duplicated constants. (MM2-04..05)
|
||||
- Tier 3: cooldown dict bounded; weight validation is per-role; sidecars self-report real VRAM; cache-fallback logs its reason. (MM2-06..09)
|
||||
- All listed pytest selections pass; no on-disk model-state change; no new dep; cross-platform default behavior identical (VRAM numbers degrade gracefully on MPS/CPU).
|
||||
</success_criteria>
|
||||
|
||||
<risks>
|
||||
- **unload() correctness for the shared OmniVoice singleton (MM2-01/02):** OmniVoiceBackend shares model_manager's `model` global. unload() must release the shared singleton, but the idle_worker() + offload_tts_for_asr() paths also touch it. Risk: a switch during an in-flight ASR offload double-frees or races. Mitigation: take mm._model_lock around the shared release in unload(); guard on `mm.model is not None`; keep unload best-effort (try/except) so it can never wedge a switch. Add the idempotency test (Task 4).
|
||||
- **Response-shape drift (MM2-04):** Moving /model/loaded + /model/unload bodies into the facade risks changing the JSON the frontend depends on (hooks.ts, flush dropdown). Mitigation: move verbatim first, assert shapes in test_model_lifecycle.py, only then layer MM2-03/08 improvements.
|
||||
- **Sidecar protocol change (MM2-08):** Adding vram_mb to pong touches the parent/child wire format. Older sidecars (a long-running session mid-upgrade) won't send it. Mitigation: treat vram_mb as optional in the parent (`reply.get("vram_mb", <last-known or 0>)`); never require it; never break the existing pong==success check.
|
||||
- **prefs.resolve at import time (MM2-05):** Resolving timeouts at import freezes them; the reaper loop reads SIDECAR_IDLE_TIMEOUT_S. Mitigation: resolve lazily inside the reaper tick / idle_worker tick (cheap) so a settings change takes effect, while keeping the import-time default for the start-guard.
|
||||
- **Per-role weight floor (MM2-07):** Lowering the ONNX floor could let a genuinely-truncated ONNX through (#352 regression). Mitigation: keep a non-zero floor for every role (e.g. ONNX floor still >> a partial KB), key on extension, and keep the "largest file" heuristic — only the threshold becomes role-aware.
|
||||
- **Scope creep into perf:** GPU-pool sizing and torch.compile are explicitly out of scope. If an executor is tempted, stop — those regress #278/#315.
|
||||
</risks>
|
||||
|
||||
<output>
|
||||
Write `.planning/quick/260613-mm2-clean-model-management-v2/260613-mm2-SUMMARY.md` when done (per wave or once at the end), documenting: which engines got real unload() overrides vs intentional no-ops (for the future CI gate), the exact response shapes preserved on the two endpoints, the per-role weight-validation thresholds chosen, and the pytest output for the verification selection. Note any decision an executor made where the plan said "use judgment."
|
||||
|
||||
Docs-sync check (CLAUDE.md hard rule): this is internal lifecycle cleanup with no user-facing install/Docker/versioning change, so no README/docs edit is expected. If MM2-05 surfaces the new idle-timeout settings keys in the Settings UI, add them to the relevant settings doc in the same PR.
|
||||
</output>
|
||||
@@ -1,32 +0,0 @@
|
||||
# SUMMARY — model-management v2 cleanup (mm2)
|
||||
|
||||
**Date:** 2026-06-13 · **Scope:** all 3 tiers (MM2-01..09). Backend-only; no frontend, no on-disk model-state change, no new deps.
|
||||
|
||||
## Tier 1 — correctness
|
||||
- **MM2-01 (VRAM leak on engine switch):** `get_active_tts_backend()` now caches one instance per configured backend id and calls the outgoing engine's `unload()` before switching. Added `reset_active_backend()` for shutdown/tests. The `model=` OmniVoice fast-path still returns a fresh view over the shared singleton (no double-load) but a switch *away from* another engine still releases it. `tts_backend.py`.
|
||||
- **MM2-02 (per-engine unload()):** `OmniVoiceBackend.unload()` drops the local ref + the shared `model_manager.model` singleton + `free_vram()` (idempotent, preload-safe, best-effort — no async lock from the sync path). `SubprocessBackend.unload()` routes to `unload_sidecar(self.id)` (busy sidecars skipped) and is inherited by every subprocess engine.
|
||||
- **MM2-03 (honest ASR row):** `/model/loaded` ASR row now reports the pipe's actual device and carries a `note: "released with the TTS model"` so the disabled unload button is explained rather than silent.
|
||||
|
||||
## Tier 2 — single lifecycle surface
|
||||
- **MM2-04 (`services/model_lifecycle.py`):** new facade owns `list_loaded()` / `unload(id)` / `unload_all()` / `free_vram()` across in-process TTS+ASR, diarization, and sidecars. `system.py` `/model/loaded` + `/model/unload` are now thin delegations; **response shapes preserved exactly** (`{models,count}`, `{unloaded,success,...}`, 400 on unknown id) — frontend untouched.
|
||||
- **MM2-05 (unified idle config):** removed the duplicated `_IDLE_TIMEOUT_SECONDS`; the in-process idle timeout and the sidecar idle timeout both resolve per-tick via `prefs.resolve(... env=...)` (env wins, settings can tune without restart). New keys: `idle_timeout_seconds` (`OMNIVOICE_IDLE_TIMEOUT_S`), `sidecar_idle_timeout_seconds` (`OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S`). `<=0` still disables sidecar reaping.
|
||||
|
||||
## Tier 3 — robustness & observability
|
||||
- **MM2-06 (bounded cooldowns):** `_install_cooldowns` is swept (TTL 1h) on each install check and cleared on success — can no longer grow unbounded.
|
||||
- **MM2-07 (per-role weight floor):** `_validate_snapshot_has_weights` uses per-extension floors (tensor formats keep 5 MB; `.onnx` floor 64 KB) **OR** the original ≥5 MB catch — strictly more lenient, so a small-but-complete ONNX model is no longer false-flagged as truncated while a 0/KB partial is still rejected (#352 intact).
|
||||
- **MM2-08 (sidecar VRAM self-report):** the parent can't see a child's VRAM, so the GPU sidecar (`engines/indextts`) now reports `vram_mb` in its `pong` (CUDA/MPS-aware, 0 on CPU); the parent stashes the last-known figure and `list_live_sidecars()` surfaces it. CPU/absent sidecars honestly report 0.
|
||||
- **MM2-09 (cache-fallback logging):** the `is_cached` `scan_cache_dir → on-disk` fallback now logs at WARNING with the exception type (was DEBUG/invisible) — the #117/#118 WinError-448 path is triagable from logs.
|
||||
|
||||
## Out of scope (as planned, not done)
|
||||
GPU-pool per-engine sizing (`_GPU_VRAM_PER_JOB_GB`) and torch.compile tuning — perf, not cleanup; risk regressing #278/#315.
|
||||
|
||||
## Verification
|
||||
- New `tests/test_mm2_lifecycle.py` — 15 tests (reuse/switch-unload/reset/idempotent-unload, facade list/unload/unknown/sidecars shapes + honest ASR, env-wins idle config, cooldown sweep, per-role weight floor ×3).
|
||||
- Affected existing: `test_engines.py`, `test_subprocess_reaper.py`, `test_model_load_timeout.py`, `test_model_manager_preload.py` — green (no regressions).
|
||||
- **Full suite: 1379 passed, 0 failed.** Live: facade endpoints return preserved shapes; engine switch calls the previous engine's `unload()` exactly once.
|
||||
|
||||
## Test placement note
|
||||
MM2 tests live at top-level `tests/` (not `tests/backend/`) on purpose: adding files under `tests/backend/` reorders collection and can expose a pre-existing `sys.modules`-isolation leak in other backend fixtures (the issue debugged in the FDL PR). Top-level placement keeps `tests/backend/` order identical.
|
||||
|
||||
## Docs-sync
|
||||
The new idle-timeout settings keys are internal env/prefs knobs with no UI surface, so no README/docs change is required by the docs-sync rule. If a future Settings panel exposes them, document there.
|
||||
@@ -6,18 +6,6 @@ The format is loosely based on [Keep a Changelog](https://keepachangelog.com/).
|
||||
Versions track the desktop app (`tauri.conf.json` + `frontend/src-tauri/Cargo.toml`).
|
||||
The bundled TTS model package (`pyproject.toml`) is versioned independently.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- **Portable personas (`.ovsvoice`).** Export any voice as a self-contained,
|
||||
fully-local persona bundle — identity, optional reference clip, consent
|
||||
attestation, SPDX license, and a watermarked preview — and import it back into
|
||||
another OmniVoice install. A privacy toggle ships a **preview-only** bundle so
|
||||
no raw recording of your voice has to travel. Verified-own-voice status can't
|
||||
be forged by hand-editing a bundle (real recording + consent text + attestation
|
||||
required). Legacy `.omnivoice` files still import. See
|
||||
[docs/persona-format.md](docs/persona-format.md). (#29)
|
||||
|
||||
## [0.3.5] — 2026-06-03
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
|
||||
**OmniVoice Studio**
|
||||
|
||||
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. It's an active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord. The latest stable release is **v0.3.5**; `main` rolls ahead at **v0.3.6** (latest release + 1 patch — see the Versioning rule below).
|
||||
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. Today it's a v0.2.7 active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord.
|
||||
|
||||
**Core Value:** **A first-run that actually works.** A user who downloads the installer (or clones the repo) should reach a working voice-cloning or dubbing output without hitting a wall — and when something does go wrong, the error or docs should tell them exactly what to do.
|
||||
|
||||
@@ -16,7 +16,7 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
|
||||
- **Default features must work on every platform (strict rule, 2026-05-20):** A feature that ships in default mode — out-of-the-box, no user customization, no opt-in toggle — must behave identically on macOS, Windows, and Linux. Platform-specific *implementation code* is allowed for OS APIs / shells / packaging, but the user-visible *default behavior* cannot diverge. Platform-only features (e.g., a macOS-only global shortcut, a Windows-only path picker) must go behind explicit user opt-in: Settings toggle, env var, or CLI flag. When a default doesn't work on a platform, that's a P0 bug — either fix it on the missing platform or move it behind opt-in. No third option.
|
||||
- **Backward-compatible project data**: Existing `omnivoice_data/` (user voices, projects, settings) must keep working without manual migration. Any DB schema change goes through alembic with a tested upgrade path.
|
||||
- **Local-first guarantee preserved**: Auto bug reporting (new addition) must be **opt-in**, must submit only to GitHub Issues (no third-party telemetry endpoint), and the app must remain fully functional with reporting disabled. No required cloud calls, accounts, or API keys.
|
||||
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** the v0.3.x line has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main; the owner tags a patch (`v0.3.Z`) from main whenever the current state is worth cutting. No `-rc` tags. No phased release. No `v0.4` deferrals while the v0.3.x line is open — every open issue and every open community PR gets absorbed into the v0.3.x line or explicitly declined. Users follow `main` for previews; users wanting stable stay on the latest tagged release (currently **v0.3.5**). ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
|
||||
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** v0.3.0 has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main. Tag `v0.3.0` once when the user calls "actually useful" — a qualitative bar, not a checklist. No `v0.3.0-rc1`. No phased release. No `v0.4` deferrals while v0.3.0 is open — every open issue and every open community PR gets absorbed into the v0.3.0 line or explicitly declined. Users follow `main` for previews; users wanting stable stay on `v0.2.7`. ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
|
||||
<!-- GSD:project-end -->
|
||||
|
||||
<!-- GSD:stack-start source:research/STACK.md -->
|
||||
@@ -190,20 +190,10 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
|
||||
<!-- GSD:conventions-start source:CONVENTIONS.md -->
|
||||
## Conventions
|
||||
|
||||
**Versioning (hard rule, owner-set 2026-06-11):** main is always **latest release + 1 patch**. The moment `vX.Y.Z` is released, main's version files (`frontend/src-tauri/tauri.conf.json`, `frontend/src-tauri/Cargo.toml`, `pyproject.toml`, **and `frontend/package.json`** — keep all **four** in lockstep; `package.json` drives the runtime `__APP_VERSION__` via vite, shown in the first-run footer + every auto bug report, so a drift ships a build that misreports its own version — guarded by `tests/test_app_version.py::test_all_version_files_in_lockstep`) bump to `X.Y.(Z+1)`. Consequences:
|
||||
- Every PR and preview build identifies as the **next** version. Preview builds stamp `X.Y.(Z+1)-N` (run number), which semver-sorts **above** the last stable `X.Y.Z` — the updater ordering is natural, no comparator tricks needed.
|
||||
- Releasing = tag `vX.Y.(Z+1)` from main (version files already match), then immediately bump main to `X.Y.(Z+2)`. The post-release bump is automated by the `version-bump` job in release.yml; if it fails, do it manually in the same day.
|
||||
- Docker: `ghcr.io/debpalash/omnivoice-studio:latest` = **main** (rolling preview); `:X.Y.Z` + `:X.Y` + `:stable` = tagged releases. `:latest` is the preview channel by design — stable users pin `:stable` or a version tag.
|
||||
- Do not bump minor/major or invent RCs/codenames without the owner asking. No "defer to next version" labels — scope is absorbed or declined, never re-versioned.
|
||||
|
||||
**Docs-sync (hard rule, owner-set 2026-06-11):** any change that alters something these docs describe — README.md, CONTRIBUTING.md, SECURITY.md, SUPPORT.md, LICENSE, or `docs/**` (install flows, Docker tag semantics, platform support, versioning/release behavior, review process, supported versions) — must update those docs **in the same PR** as the change. If a doc impact is discovered after merge, the docs fix is the immediate next commit, not backlog. Stale docs are treated as bugs.
|
||||
**Versioning (hard rule):** Everything ships on `v0.3.0`. Never mention, suggest, or label anything with a version bump — no v0.4, no RCs, no "defer to next version", no future-version labels — unless the user explicitly asks to bump. Zero unprompted version chatter.
|
||||
|
||||
**Localization (hard rule):** No hardcoded non-English (CJK) **user-facing text** anywhere in the codebase except the translation layer (`frontend/src/i18n/`). All UI strings go through i18n (`t('...')` keys in `locales/*.json`); native language names live in `i18n/index.ts` (`LANGUAGES`). Functional CJK is allowed and tracked via the allowlist in `tests/test_no_hardcoded_cjk.py` — text-processing regexes, model/engine vocabulary & identifiers (e.g. CosyVoice speaker IDs), localized error matching, demo/eval data, and test fixtures. CI fails on any hardcoded CJK outside the allowlist; to add legitimate functional CJK, extend `_ALLOWED_FILES` there with a justification.
|
||||
|
||||
**Fix quality (hard rule, owner-set 2026-06-16):** Fix issues *properly* and future-maintenance-proof — don't stop at the symptom. Root-cause fully, fix the whole **class** of the bug (not just the one reported instance), add a fail-before/pass-after regression test, and harden against recurrence (e.g. if a lockfile drift only fails in Docker, also make CI catch it). Go the extra mile where it durably pays off. Be token-efficient about it — extra **effort**, not extra **verbosity**: no padding, no redundant re-checks, the smallest correct change that is also recurrence-proof. Don't be shy to spend the effort a proper fix needs; do be shy about wasting tokens.
|
||||
|
||||
**Keep main green (hard rule, owner-set 2026-06-16):** A merge must **never break `main`'s CI**. Before a change lands, verify the *full* CI matrix would pass — every workflow in `.github/workflows/` **and** `deploy/Dockerfile`, not only the checks you happened to run. Dependency / lockfile / config changes must be validated against **all** consumers. Specifically: `frontend/` is a bun **workspace monorepo** — the lockfile is the repo-root `bun.lock`, and `deploy/Dockerfile` runs `bun install --frozen-lockfile`, so any `frontend/package.json` change requires regenerating root `bun.lock` and confirming `bun install --frozen-lockfile` passes (plain `bun install` in `ci.yml` silently tolerates drift, so CI-green ≠ Docker-green). Likewise re-check CodeQL/Security on code changes and the Tauri `cargo` build on Rust/dep changes.
|
||||
|
||||
Other conventions not yet established. Will populate as patterns emerge during development.
|
||||
<!-- GSD:conventions-end -->
|
||||
|
||||
|
||||
+1
-75
@@ -42,22 +42,11 @@ This starts both services:
|
||||
### Desktop App (Tauri)
|
||||
|
||||
```bash
|
||||
bun run desktop # dev: hot-reload Tauri shell + backend
|
||||
bun run desktop-prod # production: builds, bundles the backend, then launches
|
||||
bun run desktop
|
||||
```
|
||||
|
||||
Both run `uv sync` first (so the Python backend env is set up) and start the
|
||||
backend automatically — you do **not** start it separately. Use the exact script
|
||||
names: there is no `desktop=prod` (note the **hyphen** in `desktop-prod`).
|
||||
`desktop-prod` is Windows-aware (auto-detects bash/git; see `scripts/desktop-prod.mjs`).
|
||||
|
||||
Requires [Rust](https://rustup.rs/) and platform-specific Tauri dependencies — see the [Tauri prerequisites](https://v2.tauri.app/start/prerequisites/).
|
||||
|
||||
If the app opens but stays on the **setup splash with no buttons**, the Python
|
||||
backend didn't finish starting — the splash surfaces the stall reason, a log
|
||||
panel, and a **Retry** button (and Settings → Logs → Backend has the full trace).
|
||||
The most common from-source cause is `uv` or Python not being on your PATH.
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
@@ -203,69 +192,6 @@ cd frontend/src-tauri && cargo check
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## What code review looks like
|
||||
|
||||
Every PR is reviewed by two AI reviewers before a human looks at it:
|
||||
|
||||
- **CodeRabbit** posts a walkthrough (with a sequence diagram, and an ASCII
|
||||
before/after sketch for UI changes), inline findings, and warning-mode
|
||||
pre-merge checks against the project's hard rules.
|
||||
- **Greptile** reviews with the same project rubrics and learns from 👍/👎
|
||||
reactions on its comments — react to train it.
|
||||
|
||||
Both are advisory, not gating: CI and the maintainer's approval decide. Don't
|
||||
be surprised by detailed bot comments minutes after you open a PR — address
|
||||
what's right, push back (in a reply) on what's wrong.
|
||||
|
||||
**Commit & PR conventions:** conventional-commit style with a scope
|
||||
(`fix(dub): …`, `feat(setup): …`) and link the issue (`Closes #N` / `Refs #N`)
|
||||
in the title or body.
|
||||
|
||||
## Quality gates your PR must pass
|
||||
|
||||
- **Cross-platform parity (hard rule):** anything that ships in default mode
|
||||
must behave identically on macOS, Windows, and Linux. Platform-specific
|
||||
*implementation* is fine; platform-divergent *default behavior* is a P0.
|
||||
Platform-only features go behind an explicit opt-in (Settings toggle, env
|
||||
var, or CLI flag).
|
||||
- **i18n — all 21 locales (hard rule):** every user-facing string goes through
|
||||
`t('...')` and the key must exist in **all 21** files under
|
||||
`frontend/src/i18n/locales/`. Translate; don't copy English into non-English
|
||||
locales. CI fails on hardcoded CJK outside the allowlist in
|
||||
`tests/test_no_hardcoded_cjk.py` (extend `_ALLOWED_FILES` with a
|
||||
justification for legitimate functional CJK).
|
||||
- **DB schema changes** go through an alembic migration with a tested upgrade
|
||||
path — existing `omnivoice_data/` must keep working with no manual steps.
|
||||
- **Engine back-compat:** already-installed engines (model weights on disk)
|
||||
must not require reinstall or re-download.
|
||||
- **Local-first:** no new outbound calls except GitHub Issues (opt-in
|
||||
reporting) and HuggingFace model downloads. Never log or persist secrets or
|
||||
absolute home paths.
|
||||
- **Security posture:** the backend serves loopback HTTP — treat every
|
||||
query/path/form parameter as hostile. User-chosen filesystem destinations
|
||||
are authorized in the Tauri process (save dialog), never via HTTP params.
|
||||
|
||||
## Contribution licensing
|
||||
|
||||
OmniVoice Studio is **AGPL-3.0-only**, and the maintainer also offers a
|
||||
**commercial license** (see [LICENSE](LICENSE)). By submitting a contribution
|
||||
you agree that:
|
||||
|
||||
1. you have the right to submit it (your own work, or compatibly licensed);
|
||||
2. it is licensed to the project under **AGPL-3.0**; and
|
||||
3. you grant the project maintainer a perpetual, worldwide, non-exclusive
|
||||
right to also distribute your contribution under the project's commercial
|
||||
license terms.
|
||||
|
||||
This inbound grant is what keeps the dual-license model viable. If you can't
|
||||
agree to (3) for a particular contribution, say so in the PR and we'll discuss
|
||||
before merging. Adding a `Signed-off-by:` line (DCO) to your commits is
|
||||
appreciated but not required.
|
||||
|
||||
---
|
||||
|
||||
## Need Help?
|
||||
|
||||
- **Stuck on setup?** Ask in [Discord #help](https://discord.gg/bzQavDfVV9)
|
||||
|
||||
@@ -24,11 +24,10 @@
|
||||
</p>
|
||||
|
||||
<p>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
|
||||
<!-- Pre-built macOS bundle is Apple Silicon. Intel Macs: build from source (docs/install/macos.md); a pre-built Intel target is tracked in #279. -->
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_aarch64.dmg"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_x64_en-US.msi"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_amd64.AppImage"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
|
||||
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_amd64.deb"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
|
||||
</p>
|
||||
<p>
|
||||
<sub><b>macOS:</b> first launch needs a one-time approval — right-click → <b>Open</b> (or System Settings → Privacy & Security → <b>"Open Anyway"</b> on macOS 15). No Terminal needed. <a href="docs/install/macos.md#gatekeeper-quarantine">Why?</a></sub>
|
||||
@@ -124,7 +123,7 @@ Per-OS install guides — pick yours and follow it end-to-end:
|
||||
- **macOS** — [docs/install/macos.md](docs/install/macos.md)
|
||||
- **Windows** — [docs/install/windows.md](docs/install/windows.md)
|
||||
- **Linux** — [docs/install/linux.md](docs/install/linux.md)
|
||||
- **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
|
||||
- **Docker** — [docs/install/docker.md](docs/install/docker.md)
|
||||
|
||||
Stuck? Run the built-in self-check first — **Settings → About → "Run
|
||||
self-check"** in the app, or `uv run python backend/main.py --diagnose` from
|
||||
@@ -137,9 +136,7 @@ bundle"** packages scrubbed logs + the self-check report for bug reports.
|
||||
For Hugging Face token setup, see
|
||||
[docs/setup/huggingface-token.md](docs/setup/huggingface-token.md). For
|
||||
diarization-specific gating, see
|
||||
[docs/features/diarization.md](docs/features/diarization.md). For download
|
||||
speed, the ⚡ fast-download (Xet) status, and restricted-network / mirror
|
||||
options, see [docs/downloading-models.md](docs/downloading-models.md).
|
||||
[docs/features/diarization.md](docs/features/diarization.md).
|
||||
|
||||
## Screenshots
|
||||
|
||||
|
||||
+4
-14
@@ -2,20 +2,10 @@
|
||||
|
||||
## Supported Versions
|
||||
|
||||
| Version | Supported |
|
||||
|---------|-----------|
|
||||
| 0.3.x (latest release + `main` previews) | ✅ Current — all fixes land here |
|
||||
| 0.2.7 | ⚠️ Legacy stable — security fixes only, upgrade recommended |
|
||||
| < 0.2.7 | ❌ No longer supported |
|
||||
|
||||
## Model supply chain
|
||||
|
||||
OmniVoice supports models from **public, verifiable sources only** (Hugging
|
||||
Face repos, official project releases). Privately sold or gated model files
|
||||
are not supported: an archive from a private source can carry anything
|
||||
(bundled executables, modified configs), and nobody else can verify or
|
||||
reproduce it. Treat any privately distributed model file as an untrusted
|
||||
download, and never run executables bundled with model archives.
|
||||
| Version | Supported |
|
||||
|---------|--------------------|
|
||||
| 0.2.x | ✅ Current release |
|
||||
| < 0.2 | ❌ No longer supported |
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
|
||||
-36
@@ -1,36 +0,0 @@
|
||||
# Support
|
||||
|
||||
## Where to get help
|
||||
|
||||
| Channel | Best for |
|
||||
|---|---|
|
||||
| [Discord](https://discord.gg/bzQavDfVV9) — `#help` | Setup problems, quick questions, sharing results |
|
||||
| [GitHub Issues](https://github.com/debpalash/OmniVoice-Studio/issues) | Bugs and feature requests — use the templates; attach the diagnostic bundle (Settings → About → "Save diagnostic bundle") |
|
||||
| [GitHub Discussions](https://github.com/debpalash/OmniVoice-Studio/discussions) | Design questions, ideas, show & tell |
|
||||
| Security issues | **Never a public issue** — see [SECURITY.md](SECURITY.md) for private reporting |
|
||||
|
||||
## Model sources we support
|
||||
|
||||
OmniVoice is built on the idea that everything it runs is **open and available
|
||||
to everyone**: free, public models with verifiable sources and licenses
|
||||
(Hugging Face repos, official project releases), so the whole community can
|
||||
use, test, and debug the same thing.
|
||||
|
||||
**We do not support privately sold, paywalled, or gated model files.** A model
|
||||
delivered privately can't be verified, reproduced, or shared — it doesn't fit
|
||||
the project's goals, and issues involving such models will be politely closed.
|
||||
As a general safety rule, never run executables bundled inside any model
|
||||
archive.
|
||||
|
||||
## Before filing a bug
|
||||
|
||||
1. Update to the latest release (or `main` if you follow previews) — fixes ship continuously.
|
||||
2. Run the in-app self-check: **Settings → About → Run self-check**.
|
||||
3. Search existing issues; add a 👍 + your details to an existing one rather than opening a duplicate.
|
||||
|
||||
## Response expectations
|
||||
|
||||
This is an open-source project maintained with the help of an automated triage
|
||||
bot: issues are typically triaged within hours and every report gets a human-
|
||||
approved response. Reproducible reports with a diagnostic bundle get fixed
|
||||
fastest.
|
||||
@@ -7,13 +7,9 @@ composed at the route or router level without surprises.
|
||||
Currently exposed:
|
||||
- `require_loopback`: 403 unless the request came from a loopback origin
|
||||
(bypassed in explicit server mode — see `_server_mode`).
|
||||
- `ws_remote_authorized`: whether a WebSocket handshake from a non-loopback
|
||||
client carries the remote API key (Wave 2.3) — used by WS endpoints that
|
||||
keep their own inline loopback guards.
|
||||
"""
|
||||
|
||||
import os
|
||||
import secrets
|
||||
|
||||
from fastapi import HTTPException, Request
|
||||
|
||||
@@ -75,31 +71,3 @@ def require_loopback(request: Request) -> None:
|
||||
if _server_mode():
|
||||
return
|
||||
raise HTTPException(status_code=403, detail="loopback origin required")
|
||||
|
||||
|
||||
def remote_api_key() -> str | None:
|
||||
"""The remote-backend bearer key (Wave 2.3), or None when remote mode is
|
||||
off. Read at call time so tests can monkeypatch the env."""
|
||||
return os.environ.get("OMNIVOICE_API_KEY") or None
|
||||
|
||||
|
||||
def ws_remote_authorized(websocket) -> bool:
|
||||
"""Whether a WebSocket handshake presents the remote API key.
|
||||
|
||||
Browser WebSockets cannot set an Authorization header, so the key may
|
||||
arrive as ``?api_key=`` or via the ``ov_key`` cookie that the bearer
|
||||
middleware sets on the first authenticated HTTP request. Returns False
|
||||
when remote mode is off — callers keep their loopback-only behavior.
|
||||
"""
|
||||
key = remote_api_key()
|
||||
if not key:
|
||||
return False
|
||||
auth = websocket.headers.get("authorization", "")
|
||||
supplied = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
|
||||
if not supplied:
|
||||
supplied = (
|
||||
websocket.query_params.get("api_key")
|
||||
or websocket.cookies.get("ov_key")
|
||||
or ""
|
||||
)
|
||||
return secrets.compare_digest(supplied, key)
|
||||
|
||||
@@ -1,712 +0,0 @@
|
||||
"""Audiobook creator endpoints (parity Wave 5).
|
||||
|
||||
``POST /audiobook/plan`` — pure preview: parse a chapter-delimited script
|
||||
(Markdown ``# H1`` chapters, inline ``[voice:NAME]`` / ``[pause …]``) into the
|
||||
chapter/span plan, no synthesis.
|
||||
|
||||
``POST /audiobook`` — the synth job: render each chapter through the active TTS
|
||||
backend (reusing ``services.audiobook.synthesize_chapter`` + ``chunked_tts``),
|
||||
then mux the chapter WAVs into a chapterized **m4b** (FFMETADATA1 chapters via
|
||||
``build_m4b_cmd``). Progress streams as Server-Sent Events, mirroring the dub
|
||||
pipeline. ffmpeg-gated — without ffmpeg the job reports an error event and
|
||||
stops (the m4b is the only output format).
|
||||
|
||||
``GET /audiobook/jobs`` + ``POST /audiobook/resume/{job_id}`` — durable
|
||||
crash-resume: an interrupted render persists its plan + params to a
|
||||
``resume.json`` manifest in the job work dir, so it can be resumed later (the
|
||||
content-addressed chapter cache makes finished chapters instant) even without
|
||||
the original script. The resume UI affordance remains a follow-up.
|
||||
|
||||
epub/pdf ingest, ACX mastering shipped; the resume UI surface remains a follow-up.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, File, HTTPException, UploadFile
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from services.audiobook import (
|
||||
parse_audiobook_script,
|
||||
synthesize_chapter,
|
||||
)
|
||||
from services.longform_render import (
|
||||
LOUDNESS_PRESETS,
|
||||
build_concat_list,
|
||||
build_ffmetadata,
|
||||
build_render_cmd,
|
||||
prune_cache_dir,
|
||||
)
|
||||
from services import longform_resume # pure (no torch) — durable resume manifest
|
||||
|
||||
logger = logging.getLogger("omnivoice.audiobook")
|
||||
router = APIRouter()
|
||||
|
||||
# A cover filename as produced by /audiobook/cover: 12 hex chars + image ext.
|
||||
# An exact-match allowlist is the strongest barrier (and the one CodeQL's
|
||||
# path-injection query recognizes) — anything else is rejected outright.
|
||||
_COVER_NAME_RE = re.compile(r"^[0-9a-f]{12}\.(?:jpg|jpeg|png)$")
|
||||
|
||||
|
||||
def _safe_cover_path(cover_path: str | None) -> str | None:
|
||||
"""Confine a user-supplied cover to the upload directory before it can flow
|
||||
into ffmpeg.
|
||||
|
||||
Covers only ever come from ``/audiobook/cover``, which writes them to
|
||||
``OUTPUTS_DIR/audiobook_covers`` with a generated name. We rebuild the path
|
||||
from the basename alone (``os.path.basename`` strips any directory component
|
||||
or ``..`` traversal) joined onto that fixed directory, so no caller-supplied
|
||||
path — absolute or relative — can escape it. Returns the path only if the
|
||||
file actually exists there, else None."""
|
||||
if not cover_path:
|
||||
return None
|
||||
from core.config import OUTPUTS_DIR
|
||||
name = os.path.basename(cover_path)
|
||||
if not _COVER_NAME_RE.match(name):
|
||||
return None # not a name the upload endpoint could have produced
|
||||
cover_dir = os.path.realpath(os.path.join(OUTPUTS_DIR, "audiobook_covers"))
|
||||
real = os.path.realpath(os.path.join(cover_dir, name))
|
||||
# Containment check on the resolved path itself — it must live inside the
|
||||
# covers dir. Belt-and-suspenders over the regex+basename above; the
|
||||
# commonpath form is the path-injection barrier static analysis recognizes.
|
||||
if os.path.commonpath([real, cover_dir]) != cover_dir:
|
||||
return None
|
||||
return real if os.path.isfile(real) else None
|
||||
|
||||
|
||||
class AudiobookPlanRequest(BaseModel):
|
||||
text: str
|
||||
default_voice: str | None = None
|
||||
|
||||
|
||||
@router.post("/audiobook/plan")
|
||||
def audiobook_plan(req: AudiobookPlanRequest) -> dict:
|
||||
"""Parse a script into a chapter/span plan (pure preview, no synthesis)."""
|
||||
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
|
||||
return plan.to_dict()
|
||||
|
||||
|
||||
#: Cover size cap mirrors longform_render's guard (8 MB — a book cover, not a
|
||||
#: payload). Kept in sync intentionally; the render builder re-validates too.
|
||||
_COVER_MAX_BYTES = 8 * 1024 * 1024
|
||||
#: Import upload cap — a generous ceiling for a .txt/.md/.epub manuscript that
|
||||
#: still stops a memory-exhaustion upload (the whole file is read into RAM).
|
||||
_IMPORT_MAX_BYTES = 64 * 1024 * 1024
|
||||
#: Upper bound on chapters in a single /longform/render plan — far above any real
|
||||
#: book, but stops a pathological request from allocating/holding the job forever.
|
||||
_MAX_CHAPTERS = 10_000
|
||||
|
||||
|
||||
@router.post("/audiobook/import")
|
||||
async def audiobook_import(file: UploadFile = File(...)) -> dict:
|
||||
"""Import a ``.txt``/``.md``/``.epub``/``.pdf`` into a chapter-delimited script.
|
||||
|
||||
EPUB is parsed in spine order (stdlib only, local); PDF text is extracted
|
||||
with pypdf (pure-Python) then chapterized; plain text gets ``# `` headings
|
||||
inserted ahead of obvious chapter-title lines. Returns the script text (for
|
||||
the editor) + the resulting chapter count."""
|
||||
from services.longform_import import (
|
||||
chapterize_plaintext,
|
||||
epub_to_chapter_script,
|
||||
pdf_to_chapter_script,
|
||||
)
|
||||
|
||||
name = (file.filename or "").lower()
|
||||
data = await file.read()
|
||||
if not data:
|
||||
raise HTTPException(status_code=400, detail="empty file")
|
||||
if len(data) > _IMPORT_MAX_BYTES:
|
||||
raise HTTPException(status_code=400, detail="file too large (max 64 MB)")
|
||||
if name.endswith(".epub"):
|
||||
try:
|
||||
script = epub_to_chapter_script(data)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=f"couldn't parse EPUB: {e}")
|
||||
elif name.endswith(".pdf"):
|
||||
try:
|
||||
script = pdf_to_chapter_script(data)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=f"couldn't parse PDF: {e}")
|
||||
else:
|
||||
script = chapterize_plaintext(data.decode("utf-8", "ignore"))
|
||||
if not script.strip():
|
||||
raise HTTPException(status_code=400, detail="no text found in the file")
|
||||
plan = parse_audiobook_script(script)
|
||||
return {"text": script, "chapters": plan.chapter_count}
|
||||
|
||||
|
||||
@router.post("/audiobook/cover")
|
||||
async def audiobook_cover(cover: UploadFile = File(...)) -> dict:
|
||||
"""Upload a cover image; returns a server-side ``path`` to pass back as
|
||||
``cover_path`` in the synth request. Validated here (jpg/png + size cap) and
|
||||
again at render time."""
|
||||
from core.config import OUTPUTS_DIR
|
||||
|
||||
ext = os.path.splitext(cover.filename or "")[1].lower()
|
||||
if ext not in (".jpg", ".jpeg", ".png"):
|
||||
raise HTTPException(status_code=400, detail="cover must be a .jpg or .png")
|
||||
data = await cover.read()
|
||||
if not data or len(data) > _COVER_MAX_BYTES:
|
||||
raise HTTPException(status_code=400, detail="cover must be between 1 byte and 8 MB")
|
||||
cover_dir = os.path.join(OUTPUTS_DIR, "audiobook_covers")
|
||||
os.makedirs(cover_dir, exist_ok=True)
|
||||
path = os.path.join(cover_dir, f"{uuid.uuid4().hex[:12]}{ext}")
|
||||
with open(path, "wb") as f:
|
||||
f.write(data)
|
||||
return {"path": path}
|
||||
|
||||
|
||||
class AudiobookRequest(BaseModel):
|
||||
text: str
|
||||
default_voice: str | None = None # voice profile id; None = engine default
|
||||
bitrate: str = "128k"
|
||||
format: str = "m4b" # "m4b" | "mp3"
|
||||
loudness: str | None = None # None/"off" | "acx" | "podcast" (opt-in)
|
||||
cover_path: str | None = None # server-side path to a jpg/png cover
|
||||
# Global tags embedded in the output: {title, author, narrator, year,
|
||||
# genre, description}. Player-visible (Apple Books / Audible read these).
|
||||
metadata: dict | None = None
|
||||
# Optional pronunciation lexicon {word: respelling} applied before synthesis.
|
||||
lexicon: dict | None = None
|
||||
|
||||
|
||||
def _resolve_voice(profile_id: str | None) -> dict:
|
||||
"""Map a voice-profile id to (ref_audio, ref_text, instruct, seed).
|
||||
|
||||
Compact form of the resolver in generation.py — covers locked, design and
|
||||
clone profiles. Returns all-None for the engine default (no profile).
|
||||
"""
|
||||
out = {"ref_audio": None, "ref_text": None, "instruct": None, "seed": None}
|
||||
if not profile_id:
|
||||
return out
|
||||
from core.config import VOICES_DIR
|
||||
from core.db import db_conn
|
||||
|
||||
with db_conn() as conn:
|
||||
row = conn.execute("SELECT * FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
if not row:
|
||||
return out
|
||||
try:
|
||||
kind = row["kind"] or "clone"
|
||||
except (KeyError, IndexError):
|
||||
kind = "clone"
|
||||
if row["is_locked"] and row["locked_audio_path"]:
|
||||
out["ref_audio"] = os.path.join(VOICES_DIR, row["locked_audio_path"])
|
||||
out["ref_text"] = row["ref_text"]
|
||||
out["instruct"] = row["instruct"]
|
||||
elif kind == "design":
|
||||
out["ref_audio"] = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
|
||||
out["ref_text"] = row["ref_text"] if out["ref_audio"] else None
|
||||
out["instruct"] = row["instruct"]
|
||||
else:
|
||||
out["ref_audio"] = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
|
||||
out["ref_text"] = row["ref_text"]
|
||||
out["instruct"] = row["instruct"]
|
||||
try:
|
||||
if row["seed"] is not None:
|
||||
out["seed"] = row["seed"]
|
||||
except (KeyError, IndexError):
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
def _build_synth(default_voice: str | None) -> dict:
|
||||
"""Describe how to synthesize for the active TTS engine.
|
||||
|
||||
Returns a dict with ``mode``, ``resolve`` (voice-id → resolved refs, cached
|
||||
per id) and ``engine_id``. For OmniVoice it also carries the async
|
||||
``get_model``; other engines carry a ready ``synth`` + ``sample_rate``.
|
||||
:func:`_prepare_synth` turns this into a uniform ``(synth, sr, resolve,
|
||||
engine_id)`` once the (async) model is in hand.
|
||||
"""
|
||||
from services.tts_backend import OmniVoiceBackend, active_backend_id, get_backend_class
|
||||
|
||||
cache: dict = {}
|
||||
|
||||
def resolve(voice_id):
|
||||
key = voice_id or default_voice
|
||||
if key not in cache:
|
||||
cache[key] = _resolve_voice(key)
|
||||
return cache[key]
|
||||
|
||||
engine_id = active_backend_id()
|
||||
cls = get_backend_class(engine_id)
|
||||
if cls is OmniVoiceBackend:
|
||||
from services.model_manager import get_model
|
||||
return {"mode": "omnivoice", "resolve": resolve,
|
||||
"engine_id": engine_id, "get_model": get_model}
|
||||
|
||||
backend = cls()
|
||||
|
||||
def synth(text, voice_id, speed=None):
|
||||
v = resolve(voice_id)
|
||||
return backend.generate(
|
||||
text, language=None, ref_audio=v["ref_audio"],
|
||||
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
|
||||
speed=float(speed) if speed else 1.0,
|
||||
)
|
||||
return {"mode": "generic", "resolve": resolve, "engine_id": engine_id,
|
||||
"synth": synth, "sample_rate": backend.sample_rate}
|
||||
|
||||
|
||||
async def _prepare_synth(default_voice: str | None):
|
||||
"""Resolve :func:`_build_synth` into ``(synth, sample_rate, resolve,
|
||||
engine_id)`` — awaiting the OmniVoice model load when needed. Shared by the
|
||||
full job and the per-chapter preview."""
|
||||
info = _build_synth(default_voice)
|
||||
resolve, engine_id = info["resolve"], info["engine_id"]
|
||||
if info["mode"] == "omnivoice":
|
||||
model = await info["get_model"]()
|
||||
sr = getattr(model, "sampling_rate", 24000)
|
||||
|
||||
def synth(text, voice_id, speed=None):
|
||||
v = resolve(voice_id)
|
||||
return model.generate(
|
||||
text=text, language=None, ref_audio=v["ref_audio"],
|
||||
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
|
||||
speed=float(speed) if speed else 1.0,
|
||||
)[0]
|
||||
return synth, sr, resolve, engine_id
|
||||
return info["synth"], info["sample_rate"], resolve, engine_id
|
||||
|
||||
|
||||
def _render_chapter_cached(chapter, synth, sr, engine_id, resolve, cache_dir, lexicon=None):
|
||||
"""Render one chapter, content-addressed so a re-run reuses it (resume).
|
||||
|
||||
Returns ``(wav_path, duration_s, was_cached)``. The WAV lives at
|
||||
``cache_dir/<key>.wav`` where ``key`` is :func:`chapter_cache_key` over the
|
||||
chapter's spans + sample rate + engine + each voice's resolved signature
|
||||
(+ the lexicon, so a lexicon edit re-renders), so an unchanged chapter is
|
||||
never re-synthesized. Runs in the GPU-pool executor.
|
||||
"""
|
||||
import json
|
||||
import wave
|
||||
|
||||
from services.audio_io import atomic_save_wav
|
||||
from services.longform_render import chapter_cache_key
|
||||
from services.pronunciation import normalize_lexicon
|
||||
|
||||
spans_tuples = [(s.voice_id, s.text, s.pause_ms_after, getattr(s, "speed", None))
|
||||
for s in chapter.spans]
|
||||
sig: dict = {}
|
||||
for s in chapter.spans:
|
||||
k = s.voice_id or ""
|
||||
if k not in sig:
|
||||
v = resolve(s.voice_id)
|
||||
sig[k] = f"{v.get('ref_audio')}|{v.get('ref_text')}|{v.get('instruct')}|{v.get('seed')}"
|
||||
if lexicon:
|
||||
# Fold the lexicon into the cache key so editing pronunciations
|
||||
# invalidates cached chapters (reserved key can't collide with a voice id).
|
||||
sig["\x00lexicon"] = json.dumps(normalize_lexicon(lexicon), sort_keys=True)
|
||||
key = chapter_cache_key(spans_tuples, sample_rate=sr, engine_id=engine_id, voice_sig=sig)
|
||||
wav_path = os.path.join(cache_dir, f"{key}.wav")
|
||||
|
||||
if os.path.exists(wav_path):
|
||||
try:
|
||||
with wave.open(wav_path, "rb") as w:
|
||||
dur = w.getnframes() / float(w.getframerate() or sr)
|
||||
return wav_path, dur, True
|
||||
except Exception:
|
||||
pass # corrupt cache entry — fall through and re-render
|
||||
|
||||
audio, dur = synthesize_chapter(chapter.spans, synth, sr, lexicon=lexicon)
|
||||
atomic_save_wav(wav_path, audio, sr)
|
||||
return wav_path, dur, False
|
||||
|
||||
|
||||
class AudiobookPreviewRequest(BaseModel):
|
||||
text: str
|
||||
chapter_index: int = 0
|
||||
default_voice: str | None = None
|
||||
lexicon: dict | None = None
|
||||
|
||||
|
||||
@router.post("/audiobook/preview")
|
||||
async def audiobook_preview(req: AudiobookPreviewRequest) -> dict:
|
||||
"""Render a single chapter so the user can audition it before the full run.
|
||||
|
||||
Reuses the same content-addressed cache as the job, so a preview warms the
|
||||
cache (the later full render reuses it) and a re-preview is instant.
|
||||
"""
|
||||
from core.config import OUTPUTS_DIR
|
||||
from services.model_manager import _gpu_pool
|
||||
|
||||
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
|
||||
if not plan.chapters:
|
||||
raise HTTPException(status_code=400, detail="no chapters parsed from the script")
|
||||
n = len(plan.chapters)
|
||||
if not (0 <= req.chapter_index < n):
|
||||
raise HTTPException(status_code=400, detail=f"chapter_index out of range (0..{n - 1})")
|
||||
|
||||
chapter = plan.chapters[req.chapter_index]
|
||||
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache") # shared with _render_longform_sse
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
synth, sr, resolve, engine_id = await _prepare_synth(req.default_voice)
|
||||
loop = asyncio.get_running_loop()
|
||||
wav_path, dur, was_cached = await loop.run_in_executor(
|
||||
_gpu_pool, _render_chapter_cached, chapter, synth, sr, engine_id, resolve, cache_dir,
|
||||
req.lexicon,
|
||||
)
|
||||
return {
|
||||
"output": os.path.relpath(wav_path, OUTPUTS_DIR), # served via /audio
|
||||
"duration_s": round(dur, 2),
|
||||
"cached": was_cached,
|
||||
"title": chapter.title,
|
||||
}
|
||||
|
||||
|
||||
async def _render_longform_sse(
|
||||
plan,
|
||||
*,
|
||||
default_voice: str | None,
|
||||
fmt: str = "m4b",
|
||||
bitrate: str = "128k",
|
||||
loudness: str | None = None,
|
||||
cover_path: str | None = None,
|
||||
metadata: dict | None = None,
|
||||
lexicon: dict | None = None,
|
||||
job_type: str = "audiobook",
|
||||
job_id: str | None = None,
|
||||
resume: bool = False,
|
||||
):
|
||||
"""Shared chapterized-render SSE generator for Audiobook *and* Stories.
|
||||
|
||||
Takes a ready ``plan`` (``.chapters`` → ``.title`` + ``.spans``) — Audiobook
|
||||
parses it from a script, Stories compiles it from cast/lines — and renders
|
||||
each chapter (content-addressed cache → resume), isolating per-chapter
|
||||
failures, then muxes the successful chapters into a tagged file. This is the
|
||||
convergence point: one renderer, two front doors.
|
||||
"""
|
||||
from core.config import OUTPUTS_DIR
|
||||
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
|
||||
from services.model_manager import _gpu_pool
|
||||
|
||||
# Resume reuses the original job_id (continuing the same job row + cached
|
||||
# chapters); a fresh render generates a new one. The id may arrive from the
|
||||
# /resume/{job_id} path param, so strip it to a safe token (no path
|
||||
# separators, no CR/LF) before it ever reaches a filesystem path or a log
|
||||
# line — CodeQL py/path-injection + py/log-injection. Empty after the strip
|
||||
# → a fresh id.
|
||||
job_id = re.sub(r"[^A-Za-z0-9_-]", "", job_id or "")[:64] or uuid.uuid4().hex[:16]
|
||||
try:
|
||||
from core import job_store
|
||||
if not resume:
|
||||
job_store.create(job_id, type=job_type)
|
||||
job_store.mark_running(job_id)
|
||||
except Exception:
|
||||
job_store = None # job history is best-effort; never block synthesis
|
||||
|
||||
# Persist a durable resume manifest (plan + params) so an interrupted render
|
||||
# can be resumed later even without the original script. Best-effort.
|
||||
try:
|
||||
title = (metadata or {}).get("title") or (plan.chapters[0].title if plan.chapters else "")
|
||||
longform_resume.write_manifest(longform_resume.build_manifest(
|
||||
job_id=job_id, job_type=job_type, title=title,
|
||||
plan_chapters=[
|
||||
{"title": c.title, "spans": [s.to_dict() for s in c.spans]}
|
||||
for c in plan.chapters
|
||||
],
|
||||
params={
|
||||
"default_voice": default_voice, "fmt": fmt, "bitrate": bitrate,
|
||||
"loudness": loudness, "cover_path": cover_path,
|
||||
"metadata": metadata, "lexicon": lexicon,
|
||||
},
|
||||
))
|
||||
except Exception: # resume durability is an enhancement; never block the render
|
||||
logger.debug("[%s] resume manifest write skipped", job_id, exc_info=True)
|
||||
|
||||
def _emit(payload: dict) -> str:
|
||||
if job_store is not None:
|
||||
try:
|
||||
job_store.append_event(job_id, json.dumps(payload))
|
||||
except Exception:
|
||||
pass # best-effort job history; never block the stream
|
||||
return f"data: {json.dumps(payload)}\n\n"
|
||||
|
||||
if not plan.chapters:
|
||||
yield _emit({"type": "error", "error": "nothing to render (no chapters)"})
|
||||
return
|
||||
ffmpeg = find_ffmpeg()
|
||||
if not ffmpeg:
|
||||
yield _emit({"type": "error", "error": "ffmpeg not available; the output needs it"})
|
||||
return
|
||||
|
||||
# Confined work dir (job_id is already token-sanitized above; work_dir adds
|
||||
# the basename + realpath barrier so CodeQL sees a clean path).
|
||||
work = longform_resume.work_dir(job_type, job_id)
|
||||
if work is None:
|
||||
yield _emit({"type": "error", "error": "invalid job id"})
|
||||
return
|
||||
os.makedirs(work, exist_ok=True)
|
||||
# Chapter WAVs are content-addressed in a shared cache so a re-run (after a
|
||||
# failure or interruption) reuses what already rendered — only the
|
||||
# missing/changed chapters synthesize again (resume). Shared across both
|
||||
# front doors: an identical chapter renders once.
|
||||
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache")
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
prune_cache_dir(cache_dir) # bound disk before this job adds its chapters
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
try:
|
||||
synth, sr, resolve, engine_id = await _prepare_synth(default_voice)
|
||||
|
||||
total = len(plan.chapters)
|
||||
chapter_files: list[str] = []
|
||||
chapters_meta: list[tuple[str, int]] = []
|
||||
cached_n = 0
|
||||
failed: list[int] = []
|
||||
yield _emit({"type": "started", "job_id": job_id, "chapters": total})
|
||||
|
||||
for i, chapter in enumerate(plan.chapters):
|
||||
try:
|
||||
wav_path, dur, was_cached = await loop.run_in_executor(
|
||||
_gpu_pool, _render_chapter_cached,
|
||||
chapter, synth, sr, engine_id, resolve, cache_dir, lexicon,
|
||||
)
|
||||
except Exception: # isolate a bad chapter — keep going
|
||||
logger.warning("[%s] chapter %d (%s) failed to render",
|
||||
job_id, i, chapter.title, exc_info=True)
|
||||
failed.append(i)
|
||||
yield _emit({"type": "chapter_error", "index": i, "total": total,
|
||||
"title": chapter.title, "error": "chapter failed to render"})
|
||||
continue
|
||||
chapter_files.append(wav_path)
|
||||
chapters_meta.append((chapter.title, int(round(dur * 1000))))
|
||||
cached_n += 1 if was_cached else 0
|
||||
yield _emit({"type": "chapter", "index": i, "total": total,
|
||||
"title": chapter.title, "duration_s": round(dur, 2),
|
||||
"cached": was_cached})
|
||||
|
||||
if not chapter_files:
|
||||
yield _emit({"type": "error", "error": "all chapters failed to render"})
|
||||
return
|
||||
|
||||
yield _emit({"type": "assembling"})
|
||||
meta_path = os.path.join(work, "chapters.ffmeta")
|
||||
with open(meta_path, "w", encoding="utf-8") as f:
|
||||
f.write(build_ffmetadata(chapters_meta, global_meta=metadata))
|
||||
concat_path = os.path.join(work, "concat.txt")
|
||||
with open(concat_path, "w", encoding="utf-8") as f:
|
||||
f.write(build_concat_list(chapter_files))
|
||||
ext = "mp3" if (fmt or "").lower() == "mp3" else "m4b"
|
||||
out_name = f"{job_type}_{job_id}.{ext}"
|
||||
out_path = os.path.join(OUTPUTS_DIR, out_name)
|
||||
|
||||
# Two-pass loudness master (#28): for a known preset, measure the
|
||||
# concatenated program first, then feed the measured values back into the
|
||||
# single mux encode. `measured is None` (skip OR any failure) → the mux
|
||||
# falls back to single-pass. Gated identically to the pure builders
|
||||
# (.lower(), no strip), so off/None/unknown/whitespace skip cleanly.
|
||||
measured = None
|
||||
norm = (loudness or "").lower()
|
||||
if norm in LOUDNESS_PRESETS:
|
||||
yield _emit({"type": "mastering", "preset": norm})
|
||||
from services.loudness import measure_loudness
|
||||
measured = await measure_loudness(ffmpeg, concat_path, norm, job_id=job_id)
|
||||
|
||||
await run_ffmpeg(
|
||||
build_render_cmd(
|
||||
ffmpeg, concat_path, meta_path, out_path,
|
||||
fmt=ext, bitrate=bitrate, cover_path=_safe_cover_path(cover_path),
|
||||
loudness=loudness, measured=measured,
|
||||
),
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if job_store is not None:
|
||||
try:
|
||||
job_store.mark_done(job_id)
|
||||
except Exception:
|
||||
pass # best-effort job history
|
||||
# The render finished — drop the resume manifest so this job is no longer
|
||||
# offered for resume.
|
||||
longform_resume.clear_manifest(job_type, job_id)
|
||||
total_s = sum(d for _, d in chapters_meta) / 1000.0
|
||||
done = {"type": "done", "output": out_name,
|
||||
"chapters": len(chapter_files), "duration_s": round(total_s, 2),
|
||||
"cached_chapters": cached_n, "failed_chapters": failed}
|
||||
# Loudness verdict only when a preset was requested — off/None paths keep
|
||||
# the exact legacy `done` shape (additive, old clients unaffected).
|
||||
if norm in LOUDNESS_PRESETS:
|
||||
p = LOUDNESS_PRESETS[norm]
|
||||
done["loudness"] = {
|
||||
"preset": norm, "target_i": p.i, "target_tp": p.tp,
|
||||
"two_pass": measured is not None,
|
||||
"measured_i": measured.input_i if measured else None,
|
||||
}
|
||||
yield _emit(done)
|
||||
except Exception as e: # surface, don't 500 the stream
|
||||
logger.exception("[%s] longform render failed", job_id)
|
||||
if job_store is not None:
|
||||
try:
|
||||
job_store.mark_failed(job_id, str(e))
|
||||
except Exception:
|
||||
pass # best-effort job history
|
||||
# Generic message only — don't leak the stack/exception text to the client.
|
||||
yield _emit({"type": "error", "error": "render failed (see backend log)"})
|
||||
|
||||
|
||||
@router.post("/audiobook")
|
||||
async def audiobook_synthesize(req: AudiobookRequest):
|
||||
"""Synthesize a chapterized audiobook from a script, streaming SSE progress."""
|
||||
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
|
||||
return StreamingResponse(
|
||||
_render_longform_sse(
|
||||
plan, default_voice=req.default_voice, fmt=req.format, bitrate=req.bitrate,
|
||||
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
|
||||
lexicon=req.lexicon, job_type="audiobook",
|
||||
),
|
||||
media_type="text/event-stream",
|
||||
)
|
||||
|
||||
|
||||
# ── Shared longform render: Stories (and any future front door) post a plan ──
|
||||
|
||||
class LongformSpan(BaseModel):
|
||||
voice_id: str | None = None
|
||||
text: str
|
||||
pause_ms_after: int = 0
|
||||
speed: float | None = None
|
||||
|
||||
|
||||
class LongformChapter(BaseModel):
|
||||
title: str = ""
|
||||
spans: list[LongformSpan] = []
|
||||
|
||||
|
||||
class LongformRenderRequest(BaseModel):
|
||||
chapters: list[LongformChapter] = []
|
||||
default_voice: str | None = None
|
||||
bitrate: str = "128k"
|
||||
format: str = "m4b"
|
||||
loudness: str | None = None
|
||||
cover_path: str | None = None
|
||||
metadata: dict | None = None
|
||||
lexicon: dict | None = None
|
||||
|
||||
|
||||
@router.post("/longform/render")
|
||||
async def longform_render(req: LongformRenderRequest):
|
||||
"""Render a pre-built chapter/span plan (the Stories Editor's compiled
|
||||
cast+lines) through the shared chapterized renderer — same resume, loudness,
|
||||
cover, metadata, and output formats as the Audiobook job."""
|
||||
from services.audiobook import AudiobookPlan, Chapter, Span
|
||||
|
||||
if len(req.chapters) > _MAX_CHAPTERS:
|
||||
raise HTTPException(status_code=422, detail=f"too many chapters (max {_MAX_CHAPTERS})")
|
||||
|
||||
chapters = []
|
||||
for i, c in enumerate(req.chapters):
|
||||
# Keep a span if it has text to speak OR a pause to render (pause-only
|
||||
# spans carry inter-line silence with empty text).
|
||||
spans = [Span(voice_id=s.voice_id, text=(s.text or "").strip(),
|
||||
pause_ms_after=max(0, int(s.pause_ms_after)), speed=s.speed)
|
||||
for s in c.spans if ((s.text and s.text.strip()) or s.pause_ms_after > 0)]
|
||||
if spans:
|
||||
chapters.append(Chapter(title=c.title or f"Chapter {i + 1}", spans=spans))
|
||||
plan = AudiobookPlan(chapters=chapters)
|
||||
return StreamingResponse(
|
||||
_render_longform_sse(
|
||||
plan, default_voice=req.default_voice, fmt=req.format, bitrate=req.bitrate,
|
||||
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
|
||||
lexicon=req.lexicon, job_type="story",
|
||||
),
|
||||
media_type="text/event-stream",
|
||||
)
|
||||
|
||||
|
||||
# ── Durable resume: interrupted longform renders ────────────────────────────
|
||||
|
||||
|
||||
def _chapters_done(job_id: str) -> int:
|
||||
"""Count chapters that finished rendering, from the job's persisted events.
|
||||
Best-effort (0 if unavailable) — used only to show resume progress."""
|
||||
try:
|
||||
from core import job_store
|
||||
n = 0
|
||||
for ev in job_store.events_since(job_id, 0, limit=100_000):
|
||||
try:
|
||||
if json.loads(ev["payload"]).get("type") == "chapter":
|
||||
n += 1
|
||||
except (ValueError, KeyError, TypeError):
|
||||
continue
|
||||
return n
|
||||
except Exception:
|
||||
return 0
|
||||
|
||||
|
||||
@router.get("/audiobook/jobs")
|
||||
def list_resumable_jobs() -> dict:
|
||||
"""List interrupted longform renders that can be resumed — a work dir that
|
||||
still holds a resume manifest (a job left mid-render by a crash/quit). The
|
||||
ids come from scanning the filesystem, so the UI can offer one-click resume."""
|
||||
from core import job_store
|
||||
|
||||
out = []
|
||||
for e in longform_resume.scan_resumable():
|
||||
jid = e["job_id"]
|
||||
manifest = longform_resume.load_manifest_file(e["manifest_path"]) or {}
|
||||
job = job_store.get(jid) or {}
|
||||
out.append({
|
||||
"job_id": jid,
|
||||
"type": e["job_type"],
|
||||
"status": job.get("status", "interrupted"),
|
||||
"title": manifest.get("title", ""),
|
||||
"total_chapters": manifest.get("total_chapters", 0),
|
||||
"chapters_done": _chapters_done(jid),
|
||||
"created_at": job.get("created_at"),
|
||||
})
|
||||
return {"jobs": out}
|
||||
|
||||
|
||||
@router.post("/audiobook/resume/{job_id}")
|
||||
async def resume_longform(job_id: str):
|
||||
"""Resume an interrupted longform render from its persisted manifest. The
|
||||
already-rendered chapters are content-addressed in the shared cache, so they
|
||||
return instantly — only the unrendered chapters synthesize again. Streams the
|
||||
same SSE event shape as the original render, under the original job_id."""
|
||||
from services.audiobook import AudiobookPlan, Chapter, Span
|
||||
|
||||
# Find the requested job among the trusted filesystem scan (every path there
|
||||
# is os.listdir-sourced, never request input) and read its manifest via the
|
||||
# scan's own trusted path — the request job_id is used ONLY to *select* an
|
||||
# entry, never to build a path. No request-controlled value reaches a file
|
||||
# operation (CodeQL py/path-injection-safe).
|
||||
entry = next((e for e in longform_resume.scan_resumable()
|
||||
if e["job_id"] == job_id), None)
|
||||
if entry is None:
|
||||
raise HTTPException(status_code=404, detail="No resumable job for that id")
|
||||
manifest = longform_resume.load_manifest_file(entry["manifest_path"])
|
||||
if not manifest:
|
||||
raise HTTPException(status_code=404, detail="No resume manifest for that job")
|
||||
|
||||
chapters = [
|
||||
Chapter(title=c.get("title", ""),
|
||||
spans=[Span(**s) for s in c.get("spans", [])])
|
||||
for c in manifest["plan"]
|
||||
]
|
||||
plan = AudiobookPlan(chapters=chapters)
|
||||
p = manifest.get("params", {})
|
||||
# Retire the interrupted job's manifest (trusted scan path) so it stops
|
||||
# showing as resumable once we've kicked off the fresh-id resume.
|
||||
longform_resume.discard_manifest_file(entry["manifest_path"])
|
||||
# Resume under a FRESH job id (job_id=None → a server uuid in the renderer).
|
||||
# The chapter cache is content-addressed (keyed by chapter content, not the
|
||||
# job id), so the already-rendered chapters still hit instantly — only the
|
||||
# unrendered ones synthesize. Using a fresh id means the request's job_id
|
||||
# never names a work dir / output file (defence-in-depth path-injection).
|
||||
return StreamingResponse(
|
||||
_render_longform_sse(
|
||||
plan, default_voice=p.get("default_voice"),
|
||||
fmt=p.get("fmt", "m4b"), bitrate=p.get("bitrate", "128k"),
|
||||
loudness=p.get("loudness"), cover_path=p.get("cover_path"),
|
||||
metadata=p.get("metadata"), lexicon=p.get("lexicon"),
|
||||
job_type=entry["job_type"],
|
||||
),
|
||||
media_type="text/event-stream",
|
||||
)
|
||||
@@ -298,9 +298,6 @@ async def _run_batch_pipeline(job_id: str, job: dict):
|
||||
denoise=True, postprocess_output=True,
|
||||
)
|
||||
audio_out = audios[0]
|
||||
# TODO(#312): this route runs the OmniVoice model directly (not the active
|
||||
# backend), so VoxCPM2 never reaches it. When these routes become
|
||||
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
|
||||
mastered = apply_mastering(
|
||||
audio_out,
|
||||
sample_rate=sr,
|
||||
|
||||
@@ -25,19 +25,12 @@ router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.capture")
|
||||
|
||||
|
||||
def _truthy(value: Optional[str]) -> bool:
|
||||
"""Parse a multipart form flag. Treats '1'/'true'/'yes'/'on'/'auto'
|
||||
(any case) as on; everything else — including None — as off."""
|
||||
return (value or "").strip().lower() in {"1", "true", "yes", "on", "auto"}
|
||||
|
||||
|
||||
@router.post("/transcribe")
|
||||
async def transcribe_audio(
|
||||
audio: UploadFile = File(...),
|
||||
language: Optional[str] = Form(None),
|
||||
model: Optional[str] = Form(None),
|
||||
mode: Optional[str] = Form(None),
|
||||
refine: Optional[str] = Form(None),
|
||||
):
|
||||
"""Transcribe an audio file to text.
|
||||
|
||||
@@ -47,19 +40,10 @@ async def transcribe_audio(
|
||||
model: Whisper model size (legacy; ignored in dual-mode architecture).
|
||||
mode: 'fast' (default) uses MLX Turbo for speed; 'accurate' uses
|
||||
WhisperX with forced alignment for word-level timing.
|
||||
refine: Opt-in local-LLM cleanup of the final text (disfluencies,
|
||||
self-corrections, punctuation) — same pipeline the live
|
||||
dictation socket uses. Off by default so MCP/CLI callers don't
|
||||
pay LLM latency unless they ask; honours the user's
|
||||
Settings → Dictation-refinement config and silently passes
|
||||
through when no LLM backend is configured. The raw ``text``
|
||||
is always returned; ``refined_text`` is added only when the
|
||||
LLM actually changed something.
|
||||
|
||||
Returns:
|
||||
{
|
||||
"text": "full transcription",
|
||||
"refined_text": "cleaned text", # only when refine=true changed it
|
||||
"segments": [ {"start": 0.0, "end": 1.5, "text": "..."}, ... ],
|
||||
"language": "en",
|
||||
"duration_s": 4.2,
|
||||
@@ -107,11 +91,6 @@ async def transcribe_audio(
|
||||
if not full_text and segments:
|
||||
full_text = " ".join(s.get("text", "") for s in segments).strip()
|
||||
|
||||
# Wave 1.1: strip Whisper hallucination loops from the final text.
|
||||
# Segments keep the raw recognition so their timings stay truthful.
|
||||
from services.refinement import collapse_repetitive_artifacts
|
||||
full_text = collapse_repetitive_artifacts(full_text)
|
||||
|
||||
# Calculate audio duration from segments if available
|
||||
duration = 0.0
|
||||
if segments:
|
||||
@@ -119,24 +98,12 @@ async def transcribe_audio(
|
||||
|
||||
detected_lang = result.get("language", language or "unknown")
|
||||
|
||||
# Opt-in Wave 2.1 refinement, mirroring the live-dictation socket
|
||||
# (capture_ws). Off-thread (it's a network call, not GPU); never
|
||||
# raises — maybe_refine swallows failures and a missing LLM into a
|
||||
# None pass-through, so the raw text always stands.
|
||||
refined_text = None
|
||||
if _truthy(refine) and full_text:
|
||||
from services.refinement import maybe_refine
|
||||
refined = await asyncio.to_thread(maybe_refine, full_text)
|
||||
if refined and refined != full_text:
|
||||
refined_text = refined
|
||||
|
||||
logger.info(
|
||||
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s, refined=%s",
|
||||
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s",
|
||||
engine_id, elapsed, duration, "accurate" if use_accurate else "fast",
|
||||
refined_text is not None,
|
||||
)
|
||||
|
||||
response = {
|
||||
return {
|
||||
"text": full_text,
|
||||
"segments": [
|
||||
{
|
||||
@@ -151,9 +118,6 @@ async def transcribe_audio(
|
||||
"transcription_time_s": elapsed,
|
||||
"engine": engine_id,
|
||||
}
|
||||
if refined_text is not None:
|
||||
response["refined_text"] = refined_text
|
||||
return response
|
||||
finally:
|
||||
try:
|
||||
os.unlink(tmp.name)
|
||||
|
||||
@@ -8,12 +8,6 @@ live dictation feedback.
|
||||
Protocol:
|
||||
→ Client sends binary audio frames (16-bit PCM or WebM/Opus blobs)
|
||||
← Server sends JSON messages:
|
||||
|
||||
Opt-in AEC mode (``?aec=1[&sr=16000]``, parity Action 8b): for dictating
|
||||
while the app plays audio. Frames must be raw int16 mono PCM, each tagged
|
||||
with a 1-byte prefix — 0x00 = microphone, 0x01 = playback reference. The
|
||||
server runs an NLMS echo canceller, cleaning the mic against the reference
|
||||
before transcription. Without the param the protocol is unchanged.
|
||||
{"type": "partial", "text": "Hello wor..."} — interim result
|
||||
{"type": "final", "text": "Hello world.", — committed result
|
||||
"segments": [...], "language": "en",
|
||||
@@ -31,7 +25,7 @@ import time
|
||||
|
||||
from fastapi import APIRouter, WebSocket, WebSocketDisconnect
|
||||
|
||||
from api.dependencies import _LOOPBACK_HOSTS, ws_remote_authorized
|
||||
from api.dependencies import _LOOPBACK_HOSTS
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.capture_ws")
|
||||
@@ -50,56 +44,6 @@ MIN_BUFFER_BYTES = 64000 # ~2s of 16-bit mono 16kHz — needs enough WebM frame
|
||||
# to transcribe whatever the user recorded, even short utterances.
|
||||
MIN_FINAL_BUFFER_BYTES = 4000 # ~125ms of 16-bit mono 16kHz
|
||||
|
||||
# ── Dictate-over-playback AEC (parity Action 8b, opt-in) ──────────────────
|
||||
# Activated by the ``?aec=1`` query param. When OFF (the default), the
|
||||
# protocol and behaviour are byte-for-byte unchanged. When ON, the client
|
||||
# streams raw int16 mono PCM frames tagged with a 1-byte type prefix so the
|
||||
# server can tell mic audio from the playback reference it must cancel:
|
||||
_AEC_NEAR = 0x00 # microphone frame (clean it, then buffer for ASR)
|
||||
_AEC_FAR = 0x01 # playback reference frame (feed the echo model only)
|
||||
|
||||
|
||||
def _demux_aec_frame(data: bytes) -> tuple[str, bytes]:
|
||||
"""Split a prefixed AEC binary frame into ``(kind, pcm)``.
|
||||
|
||||
``kind`` is ``"near"`` (mic) or ``"far"`` (playback reference). An empty
|
||||
or prefix-only frame yields an empty payload. Unknown prefixes are treated
|
||||
as ``"near"`` so a malformed tag degrades to plain dictation rather than
|
||||
dropping audio.
|
||||
"""
|
||||
if not data:
|
||||
return "near", b""
|
||||
kind = "far" if data[0] == _AEC_FAR else "near"
|
||||
return kind, data[1:]
|
||||
|
||||
|
||||
def _pcm16_to_wav(pcm: bytes, sample_rate: int) -> str | None:
|
||||
"""Write raw int16 mono PCM to a temp WAV via stdlib ``wave`` (no ffmpeg).
|
||||
|
||||
Used on the AEC path, where frames are already decoded PCM — the cleaned
|
||||
samples have no container, so the ffmpeg-sniffing ``_chunks_to_wav`` would
|
||||
misdetect them. Returns the temp path, or ``None`` for a too-short buffer.
|
||||
"""
|
||||
if not pcm or len(pcm) < 100:
|
||||
return None
|
||||
import wave
|
||||
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".wav")
|
||||
tmp.close()
|
||||
try:
|
||||
with wave.open(tmp.name, "wb") as wf:
|
||||
wf.setnchannels(1)
|
||||
wf.setsampwidth(2) # int16
|
||||
wf.setframerate(sample_rate)
|
||||
wf.writeframes(pcm)
|
||||
return tmp.name
|
||||
except Exception as e:
|
||||
logger.debug("PCM->WAV failed: %s", e)
|
||||
try:
|
||||
os.unlink(tmp.name)
|
||||
except OSError:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
@router.websocket("/ws/transcribe")
|
||||
async def ws_transcribe(websocket: WebSocket):
|
||||
@@ -109,33 +53,13 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
# WebSocket dependency injection differs across FastAPI versions, so we
|
||||
# inline the check before accept(). Without it, any local process could
|
||||
# stream the user's microphone over this endpoint.
|
||||
# Wave 2.3 (remote backend): a non-loopback client that presents the
|
||||
# OMNIVOICE_API_KEY bearer is the thin-client dictation case — the mic
|
||||
# lives on the user's machine, the GPU here — and is allowed through.
|
||||
host = websocket.client.host if websocket.client else None
|
||||
if host not in _LOOPBACK_HOSTS and not ws_remote_authorized(websocket):
|
||||
if host not in _LOOPBACK_HOSTS:
|
||||
await websocket.close(code=1008, reason="loopback origin required")
|
||||
return
|
||||
|
||||
await websocket.accept()
|
||||
|
||||
# Opt-in dictate-over-playback AEC (parity Action 8b). Default OFF →
|
||||
# identical legacy behaviour. When on, frames are 1-byte-tagged raw PCM
|
||||
# and the cleaned mic stream is muxed via stdlib wave (not ffmpeg).
|
||||
aec = None
|
||||
pcm_sr: int | None = None
|
||||
if websocket.query_params.get("aec") in ("1", "true", "on"):
|
||||
try:
|
||||
pcm_sr = int(websocket.query_params.get("sr", "16000"))
|
||||
from services.aec import NlmsEchoCanceller
|
||||
aec = NlmsEchoCanceller(sample_rate=pcm_sr)
|
||||
logger.info("AEC enabled for dictation session (sr=%d)", pcm_sr)
|
||||
except Exception as e:
|
||||
# Bad sr or import failure → fall back to plain dictation.
|
||||
logger.warning("AEC requested but disabled: %s", e)
|
||||
aec = None
|
||||
pcm_sr = None
|
||||
|
||||
audio_chunks: list[bytes] = []
|
||||
total_bytes = 0
|
||||
last_audio_time = time.monotonic()
|
||||
@@ -173,16 +97,6 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
# Empty binary frame also acts as EOF — connection stays open.
|
||||
running = False
|
||||
break
|
||||
if aec is not None:
|
||||
# Tagged PCM: route the playback reference into the echo
|
||||
# model and clean the mic before it reaches the buffer.
|
||||
kind, payload = _demux_aec_frame(data)
|
||||
if kind == "far":
|
||||
aec.push_far_end(payload)
|
||||
continue
|
||||
if not payload:
|
||||
continue
|
||||
data = aec.process_near_end(payload)
|
||||
audio_chunks.append(data)
|
||||
total_bytes += len(data)
|
||||
last_audio_time = time.monotonic()
|
||||
@@ -229,7 +143,7 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
|
||||
# Transcribe current buffer
|
||||
try:
|
||||
text = await _transcribe_buffer(audio_chunks[:], pcm_sr=pcm_sr)
|
||||
text = await _transcribe_buffer(audio_chunks[:])
|
||||
if text and text != partial_text:
|
||||
partial_text = text
|
||||
await _safe_send({
|
||||
@@ -259,16 +173,7 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
# Final transcription on complete buffer — skip if client already gone.
|
||||
if total_bytes > MIN_FINAL_BUFFER_BYTES:
|
||||
try:
|
||||
result = await _transcribe_buffer_full(audio_chunks, pcm_sr=pcm_sr)
|
||||
# Wave 2.1: optional local-LLM refinement of the final text.
|
||||
# Off-thread (network call, not GPU); pass-through on any
|
||||
# failure or when no LLM backend is configured. The raw text
|
||||
# always ships too — clients paste refined_text ?? text.
|
||||
if result.get("text"):
|
||||
from services.refinement import maybe_refine
|
||||
refined = await asyncio.to_thread(maybe_refine, result["text"])
|
||||
if refined and refined != result["text"]:
|
||||
result["refined_text"] = refined
|
||||
result = await _transcribe_buffer_full(audio_chunks)
|
||||
if not await _safe_send({"type": "final", **result}):
|
||||
logger.debug("Skipped final send — client already disconnected")
|
||||
except Exception as e:
|
||||
@@ -292,10 +197,10 @@ async def ws_transcribe(websocket: WebSocket):
|
||||
pass
|
||||
|
||||
|
||||
async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None) -> str:
|
||||
async def _transcribe_buffer(chunks: list[bytes]) -> str:
|
||||
"""Quick partial transcription of the current audio buffer."""
|
||||
|
||||
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
|
||||
tmp = _chunks_to_wav(chunks)
|
||||
if tmp is None:
|
||||
return ""
|
||||
|
||||
@@ -318,9 +223,9 @@ async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None)
|
||||
pass
|
||||
|
||||
|
||||
async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = None) -> dict:
|
||||
async def _transcribe_buffer_full(chunks: list[bytes]) -> dict:
|
||||
"""Full transcription with timing info for the final result."""
|
||||
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
|
||||
tmp = _chunks_to_wav(chunks)
|
||||
if tmp is None:
|
||||
return {"text": "", "segments": [], "language": "unknown",
|
||||
"duration_s": 0, "transcription_time_s": 0, "engine": "none"}
|
||||
@@ -340,12 +245,6 @@ async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = N
|
||||
if not full_text and segments:
|
||||
full_text = " ".join(s.get("text", "") for s in segments).strip()
|
||||
|
||||
# Wave 1.1: strip Whisper hallucination loops from the final
|
||||
# text (the string that gets auto-pasted). Segments keep the
|
||||
# raw recognition so their timings stay truthful.
|
||||
from services.refinement import collapse_repetitive_artifacts
|
||||
full_text = collapse_repetitive_artifacts(full_text)
|
||||
|
||||
duration = max((s.get("end", 0) for s in segments), default=0.0)
|
||||
|
||||
return {
|
||||
|
||||
@@ -258,19 +258,13 @@ async def community_use(item_id: str, name: Optional[str] = Query(None)):
|
||||
raise HTTPException(status_code=503, detail=f"Couldn't add this voice right now. Error: {e}")
|
||||
|
||||
try:
|
||||
# A community "preset" is a synthetic designed voice (rendered from an
|
||||
# instruct string) → kind='design'; a "voice" carries a real reference
|
||||
# clip → kind='clone'. Setting kind makes the persona-gallery
|
||||
# synthetic-only gating work (§R3) instead of defaulting all imports to
|
||||
# 'clone'.
|
||||
kind = "design" if item["type"] == "preset" else "clone"
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO voice_profiles "
|
||||
"(id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at, kind) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
"(id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(profile_id, profile_name, audio_filename, ref_text, instruct,
|
||||
item.get("language", "Auto"), None, item["id"], time.time(), kind),
|
||||
item.get("language", "Auto"), None, item["id"], time.time()),
|
||||
)
|
||||
except Exception:
|
||||
with __import__("contextlib").suppress(OSError):
|
||||
|
||||
@@ -1,35 +0,0 @@
|
||||
"""Voice-design "describe your voice" API (issue #317).
|
||||
|
||||
Maps a free-text voice description onto the existing design parameter space
|
||||
via the deterministic keyword mapper in ``core.describe_voice``. Pure CPU +
|
||||
stdlib — no model, no network — so it imports and responds instantly in any
|
||||
environment, including test/CI without model weights.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from core.describe_voice import parse_description
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
class DescribeRequest(BaseModel):
|
||||
description: str = Field(default="", max_length=2000)
|
||||
|
||||
|
||||
@router.post("/design/describe")
|
||||
def describe_voice(req: DescribeRequest) -> dict:
|
||||
"""Parse a free-text description into design attrs + a validator-safe instruct.
|
||||
|
||||
Response shape::
|
||||
|
||||
{
|
||||
"attrs": {"Gender": "female", "Age": "elderly", ... or "Auto"},
|
||||
"instruct": "female, elderly, low pitch, british accent",
|
||||
"matched": [{"category": "Age", "token": "elderly", "phrase": "elderly"}, ...],
|
||||
"unmatched": ["slightly raspy"]
|
||||
}
|
||||
"""
|
||||
return parse_description(req.description)
|
||||
@@ -25,7 +25,6 @@ from services.segmentation import (
|
||||
assign_speakers_heuristic,
|
||||
clean_up_segments,
|
||||
)
|
||||
from services.onset_align import snap_segment_starts
|
||||
from services import dub_pipeline
|
||||
|
||||
router = APIRouter()
|
||||
@@ -367,11 +366,7 @@ _prep_event_helper = dub_pipeline.prep_event # alias; we keep the module-local
|
||||
|
||||
|
||||
@router.get("/dub/transcribe-stream/{job_id}")
|
||||
async def dub_transcribe_stream(
|
||||
job_id: str,
|
||||
num_speakers: Optional[int] = None,
|
||||
per_segment_refs: bool = True,
|
||||
):
|
||||
async def dub_transcribe_stream(job_id: str, num_speakers: Optional[int] = None):
|
||||
"""Stream per-chunk segments via SSE, then emit diarized final pass.
|
||||
|
||||
Pre-flight checks (missing job, missing audio, ASR not loaded) are emitted
|
||||
@@ -553,15 +548,6 @@ async def dub_transcribe_stream(
|
||||
detected_lang = part["language"]
|
||||
asr_speaker_turns.extend(part.get("speaker_turns") or [])
|
||||
chunk_segs = segment_transcript(part, duration=t1, scene_cuts=scene_cuts)
|
||||
# #280: Whisper often stretches a segment's start back over
|
||||
# leading music/silence (classic case: speech begins at 0:03,
|
||||
# transcript says 0.0 → the dub plays 3 s early). Snap starts
|
||||
# forward to the actual speech onset. `audio_np` is the same
|
||||
# track ASR ran on — vocals.wav when Demucs succeeded.
|
||||
try:
|
||||
snap_segment_starts(chunk_segs, audio_np, sr)
|
||||
except Exception as e:
|
||||
logger.warning("onset alignment skipped for chunk %d: %s", i, e)
|
||||
chunk_segs = assign_speakers_heuristic(chunk_segs)
|
||||
for s in chunk_segs:
|
||||
s["id"] = f"s{next_seg_id:05x}"
|
||||
@@ -773,61 +759,16 @@ async def dub_transcribe_stream(
|
||||
clones = done.pop().result()
|
||||
break
|
||||
yield _sse_event("ping", {})
|
||||
# Wave 3.2: per-segment clone refs. Cut each long-enough segment's
|
||||
# own reference from the vocals so the dub of each line matches the
|
||||
# prosody of its source line. Short lines fall back to the
|
||||
# per-speaker clone below. Default on; the user can force
|
||||
# per-speaker by disabling it (job["per_segment_refs"]).
|
||||
seg_clones = {}
|
||||
job["per_segment_refs"] = per_segment_refs
|
||||
if per_segment_refs:
|
||||
try:
|
||||
from services.speaker_clone import extract_segment_refs
|
||||
seg_ids_for_clone = [s.get("id", i) for i, s in enumerate(final_segs)]
|
||||
seg_clones = await loop.run_in_executor(
|
||||
_cpu_pool, lambda: extract_segment_refs(
|
||||
vocals_for_clone, final_segs,
|
||||
os.path.dirname(vocals_for_clone),
|
||||
seg_ids=seg_ids_for_clone,
|
||||
),
|
||||
)
|
||||
if seg_clones:
|
||||
job["segment_clones"] = seg_clones
|
||||
except Exception as e:
|
||||
logger.warning("per-segment clone refs skipped: %s", e)
|
||||
|
||||
if clones or seg_clones:
|
||||
if clones:
|
||||
job["speaker_clones"] = clones
|
||||
# Default each segment's profile_id to its detected speaker's
|
||||
# auto-clone — but only if the user hasn't already assigned
|
||||
# something. (#486)
|
||||
#
|
||||
# We prefer the UI-visible `auto:{speaker}` id over the
|
||||
# per-segment `auto-seg:{id}` id even when a per-segment ref
|
||||
# exists, because the dub editor's Voice dropdown only renders
|
||||
# `auto:` options ("From Video → Speaker N"). An `auto-seg:`
|
||||
# value matches no <option>, so the row silently read
|
||||
# "Default" while the speaker was actually bound — exactly the
|
||||
# reported bug. The per-segment ref is NOT lost: dub_generate's
|
||||
# `auto:` branch transparently prefers this segment's own
|
||||
# per-segment ref (job["segment_clones"][seg_id]) when present,
|
||||
# so a row shown as "Speaker 1" still clones from its own line
|
||||
# when that line is long enough.
|
||||
if clones:
|
||||
job["speaker_clones"] = clones
|
||||
# Default each segment's profile_id to its speaker's auto-clone,
|
||||
# but only if the user hasn't already assigned something.
|
||||
for s in final_segs:
|
||||
if s.get("profile_id"):
|
||||
continue
|
||||
spk = s.get("speaker_id") or "Speaker 1"
|
||||
if spk in clones:
|
||||
s["profile_id"] = auto_profile_id(spk)
|
||||
continue
|
||||
# No per-speaker clone for this speaker (too little usable
|
||||
# audio overall) but this single line was long enough for
|
||||
# its own ref — fall back to the per-segment id. The editor
|
||||
# can't render it, but generation still clones correctly.
|
||||
sid = str(s.get("id", ""))
|
||||
if sid and sid in seg_clones:
|
||||
s["profile_id"] = f"auto-seg:{sid}"
|
||||
except Exception as e:
|
||||
logger.warning("speaker_clone extraction skipped: %s", e)
|
||||
|
||||
@@ -921,14 +862,6 @@ async def dub_transcribe(job_id: str):
|
||||
scene_cuts = job.get("scene_cuts") or []
|
||||
segments = segment_transcript(result, duration=job.get("duration", 0.0), scene_cuts=scene_cuts)
|
||||
|
||||
# #280: snap segment starts forward to the actual speech onset so the
|
||||
# dub doesn't begin seconds before the original speaker does.
|
||||
try:
|
||||
audio_for_onset, onset_sr = sf.read(asr_audio_target, dtype="float32")
|
||||
snap_segment_starts(segments, audio_for_onset, onset_sr)
|
||||
except Exception as e:
|
||||
logger.warning("onset alignment skipped: %s", e)
|
||||
|
||||
diar_pipe = get_diarization_pipeline()
|
||||
if diar_pipe:
|
||||
try:
|
||||
|
||||
@@ -1,7 +1,5 @@
|
||||
import os
|
||||
import io
|
||||
import re
|
||||
import json
|
||||
import time
|
||||
import uuid
|
||||
import asyncio
|
||||
@@ -14,13 +12,6 @@ from core.config import DUB_DIR, dub_seg_path
|
||||
from core.tasks import task_manager
|
||||
from api.routers.dub_core import _get_job
|
||||
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
|
||||
from services.video_retime import (
|
||||
DRIFT_TOLERANCE_S,
|
||||
RetimeError,
|
||||
build_chunk_filter_graph,
|
||||
expand_retime_chunks,
|
||||
prepare_smart_fit_video,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.api")
|
||||
@@ -31,9 +22,6 @@ def _unique_stamp() -> str:
|
||||
return f"{time.strftime('%Y%m%dT%H%M%S')}-{uuid.uuid4().hex[:8]}"
|
||||
|
||||
|
||||
_SAFE_LANG = re.compile(r"^[A-Za-z0-9_-]{1,32}$")
|
||||
|
||||
|
||||
def _native_save(source: str, destination: str, display_name: str, media_type: str):
|
||||
"""Copy a generated export file to a user-chosen destination and return JSON."""
|
||||
import shutil
|
||||
@@ -170,23 +158,15 @@ async def dub_list_tracks(job_id: str):
|
||||
return {"tracks": job.get("dubbed_tracks", {})}
|
||||
|
||||
|
||||
def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool,
|
||||
fitted_segments: "list[dict] | None" = None) -> str | None:
|
||||
def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool) -> str | None:
|
||||
"""Build a temp SRT from job segments for use with ffmpeg's subtitles filter.
|
||||
|
||||
Returned path is already ffmpeg-filter-safe (plain ASCII basename under exports_dir).
|
||||
Returns None if there are no segments to render.
|
||||
|
||||
``fitted_segments`` (Smart Fit): {id, start, end} cue records on the
|
||||
fitted timeline — when provided, cue times come from there instead of
|
||||
the original ``job["segments"]`` timings, so burned subs track the
|
||||
retimed video / fitted audio rather than the source timeline.
|
||||
"""
|
||||
segments = job.get("segments", [])
|
||||
if not segments:
|
||||
return None
|
||||
if fitted_segments:
|
||||
segments = _apply_fitted_times(segments, fitted_segments)
|
||||
lines = []
|
||||
for i, seg in enumerate(segments):
|
||||
lines.append(str(i + 1))
|
||||
@@ -234,14 +214,41 @@ def _build_video_stretch_filter_graph(
|
||||
if not plan:
|
||||
return "", in_label or f"[{video_input_idx}:v]"
|
||||
|
||||
# Chunk expansion + graph emission live in services.video_retime now so
|
||||
# the Smart Fit batched pipeline shares the exact same boundary math.
|
||||
# With default options the emitted graph is byte-identical to the
|
||||
# original inline implementation.
|
||||
chunks = expand_retime_chunks(plan, orig_dur)
|
||||
chunks: list[tuple[float, float, float]] = [] # (a, b, ratio)
|
||||
cursor = 0.0
|
||||
for entry in plan:
|
||||
a = float(entry["orig_start"])
|
||||
b = float(entry["orig_end"])
|
||||
if a > cursor + 1e-3:
|
||||
chunks.append((cursor, a, 1.0)) # gap or pre-roll at native rate
|
||||
ratio = float(entry["stretch_ratio"])
|
||||
if b > a:
|
||||
chunks.append((a, b, ratio))
|
||||
cursor = max(cursor, b)
|
||||
if orig_dur > cursor + 1e-3:
|
||||
chunks.append((cursor, orig_dur, 1.0)) # tail at native rate
|
||||
chunks = [(a, b, r) for (a, b, r) in chunks if b > a]
|
||||
if not chunks:
|
||||
return "", in_label or f"[{video_input_idx}:v]"
|
||||
return build_chunk_filter_graph(chunks, in_label or f"[{video_input_idx}:v]")
|
||||
|
||||
src = in_label or f"[{video_input_idx}:v]"
|
||||
parts: list[str] = []
|
||||
labels: list[str] = []
|
||||
# `split` lets us tap the same source stream once per chunk without re-
|
||||
# decoding. setpts={ratio}*PTS slows down (ratio > 1) or speeds up
|
||||
# (ratio < 1) each chunk; PTS-STARTPTS first to normalise the timestamp
|
||||
# base after the trim.
|
||||
split_labels = [f"[vsplit{idx}]" for idx in range(len(chunks))]
|
||||
parts.append(f"{src}split={len(chunks)}{''.join(split_labels)}")
|
||||
for idx, ((a, b, ratio), split_lbl) in enumerate(zip(chunks, split_labels)):
|
||||
out_label = f"[vstr{idx}]"
|
||||
labels.append(out_label)
|
||||
parts.append(
|
||||
f"{split_lbl}trim=start={a:.4f}:end={b:.4f},"
|
||||
f"setpts=PTS-STARTPTS,setpts={ratio:.6f}*PTS{out_label}"
|
||||
)
|
||||
parts.append("".join(labels) + f"concat=n={len(chunks)}:v=1:a=0[vstretched]")
|
||||
return ";".join(parts), "[vstretched]"
|
||||
|
||||
|
||||
def _video_stretch_plan_for(job: dict, lang_code: str) -> dict | None:
|
||||
@@ -257,76 +264,6 @@ def _video_stretch_plan_for(job: dict, lang_code: str) -> dict | None:
|
||||
return entry
|
||||
|
||||
|
||||
def _video_retime_plan_for(job: dict, lang_code: str) -> "tuple[str, dict] | None":
|
||||
"""Resolve the video retime plan for ``lang_code`` across both keyspaces.
|
||||
|
||||
Returns ``(kind, entry)`` where kind is ``"stretch_video"`` (legacy
|
||||
Mode B plans — resolution byte-identical to ``_video_stretch_plan_for``)
|
||||
or ``"smart_fit"`` (Phase A ``job["fit_plans"]`` entries, gated on the
|
||||
track actually having been generated under smart_fit so a stale plan
|
||||
from an earlier run can't retime a track re-generated under another
|
||||
strategy). ``None`` when neither applies.
|
||||
"""
|
||||
legacy = _video_stretch_plan_for(job, lang_code)
|
||||
if legacy is not None:
|
||||
return "stretch_video", legacy
|
||||
entry = (job.get("fit_plans") or {}).get(lang_code)
|
||||
track = (job.get("dubbed_tracks") or {}).get(lang_code) or {}
|
||||
if entry and entry.get("plan") and track.get("timing_strategy") == "smart_fit":
|
||||
return "smart_fit", entry
|
||||
return None
|
||||
|
||||
|
||||
def _fitted_segments_for(job: dict, lang_code: "str | None") -> "list[dict] | None":
|
||||
"""Fitted-timeline subtitle cues ({id, start, end}) for a Smart Fit
|
||||
track, or None. Same staleness gate as ``_video_retime_plan_for``."""
|
||||
if not lang_code:
|
||||
return None
|
||||
entry = (job.get("fit_plans") or {}).get(lang_code)
|
||||
track = (job.get("dubbed_tracks") or {}).get(lang_code) or {}
|
||||
if not entry or track.get("timing_strategy") != "smart_fit":
|
||||
return None
|
||||
fitted = entry.get("fitted_segments")
|
||||
return fitted or None
|
||||
|
||||
|
||||
def _apply_fitted_times(segments: list[dict], fitted: list[dict]) -> list[dict]:
|
||||
"""Overlay fitted cue times onto subtitle segments (copies; non-destructive).
|
||||
|
||||
Matches by segment ``id``; when the fitted record carries no ids at all
|
||||
(defensive), falls back to positional pairing. Segments without a match
|
||||
keep their original timings.
|
||||
"""
|
||||
by_id = {str(f["id"]): f for f in fitted if f.get("id") is not None}
|
||||
out: list[dict] = []
|
||||
for i, seg in enumerate(segments):
|
||||
cue = None
|
||||
if seg.get("id") is not None:
|
||||
cue = by_id.get(str(seg["id"]))
|
||||
if cue is None and not by_id and i < len(fitted):
|
||||
cue = fitted[i]
|
||||
if cue is None:
|
||||
out.append(seg)
|
||||
continue
|
||||
patched = dict(seg)
|
||||
patched["start"] = float(cue["start"])
|
||||
patched["end"] = float(cue["end"])
|
||||
out.append(patched)
|
||||
return out
|
||||
|
||||
|
||||
def _burn_subs_allowed(retime_kind: "str | None") -> bool:
|
||||
"""Subtitle burn-in combined with video retime.
|
||||
|
||||
Allowed for ``smart_fit`` (fitted cue records exist, and the burn pass
|
||||
runs AFTER the retime graph so cues land on the retimed timeline) and
|
||||
for plain exports. Still rejected for legacy ``stretch_video``, which
|
||||
has no fitted-cue record — cues would burn at original timestamps onto
|
||||
a re-timed video and drift.
|
||||
"""
|
||||
return retime_kind != "stretch_video"
|
||||
|
||||
|
||||
#: Audio export formats → ffmpeg codec args. Unknown formats fall back to
|
||||
#: AAC/m4a so a bad request can never produce a broken command.
|
||||
_AUDIO_FORMAT_CODECS: dict[str, list[str]] = {
|
||||
@@ -375,11 +312,6 @@ async def dub_download(
|
||||
dual: bool = Query(False, description="When burn_subs=1, render translated on top of italicised original."),
|
||||
out_format: str = Query("m4a", description="Audio-only jobs (#119): output container — wav, m4a, mp3, or flac. Ignored for video jobs."),
|
||||
):
|
||||
# Strict allowlist on the path param BEFORE it reaches any filesystem
|
||||
# path or ffmpeg argv (export dir, retime work path, slice paths). Real
|
||||
# job ids are short uuid slices — alnum/hyphen/underscore only.
|
||||
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", job_id):
|
||||
raise HTTPException(status_code=400, detail="Invalid job id")
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
@@ -458,24 +390,18 @@ async def dub_download(
|
||||
)
|
||||
|
||||
# Determine whether this export should drive video through a per-segment
|
||||
# retime (legacy stretch_video Mode B, or Smart Fit). Retime is keyed off
|
||||
# the default_track's plan because the video can only physically follow
|
||||
# one timeline at a time. If multiple dub tracks are included, only the
|
||||
# default_track is visually in sync — other tracks share the same
|
||||
# (retimed) video. Single-track export is the supported common case.
|
||||
retime_kind: "str | None" = None
|
||||
retime_entry: "dict | None" = None
|
||||
if default_track and default_track != "original":
|
||||
_retime = _video_retime_plan_for(job, default_track)
|
||||
if _retime:
|
||||
retime_kind, retime_entry = _retime
|
||||
stretch_entry = retime_entry if retime_kind == "stretch_video" else None
|
||||
# Subtitle burn under legacy stretch_video would render cues at the
|
||||
# original timestamps onto a re-timed video — they'd drift (no fitted-cue
|
||||
# record exists for that mode). Skip the burn pass in that combo and log;
|
||||
# the user can still export the SRT/VTT separately. Smart Fit DOES carry
|
||||
# fitted cues, so burn+retime is allowed there (burn runs post-retime).
|
||||
if not _burn_subs_allowed(retime_kind) and burn_subs:
|
||||
# stretch graph (Mode B). Stretch is keyed off the default_track's plan
|
||||
# because the video can only physically follow one timeline at a time.
|
||||
# If multiple dub tracks are included and they were generated under
|
||||
# stretch_video, only the default_track is visually in sync — other
|
||||
# tracks share the same (stretched) video. Single-track export is the
|
||||
# supported common case.
|
||||
stretch_entry = _video_stretch_plan_for(job, default_track) if default_track and default_track != "original" else None
|
||||
# Subtitle burn under stretch_video would render cues at the original
|
||||
# timestamps onto a re-timed video — they'd drift. Skip the burn pass
|
||||
# in that combo and log; the user can still export the SRT/VTT
|
||||
# separately and the new-layout timing lives there.
|
||||
if stretch_entry and burn_subs:
|
||||
logger.warning(
|
||||
"stretch_video + burn_subs is not supported in one pass; "
|
||||
"skipping subtitle burn for job %s. Export the SRT/VTT separately.",
|
||||
@@ -483,71 +409,11 @@ async def dub_download(
|
||||
)
|
||||
burn_subs = False
|
||||
|
||||
# Smart Fit: cue times come from the fitted timeline — that's where the
|
||||
# dubbed audio actually sits, whether or not the video retime succeeds.
|
||||
fitted_segments = _fitted_segments_for(job, default_track) if default_track and default_track != "original" else None
|
||||
sub_path = _write_burn_srt(job, exports_dir, stamp, dual, fitted_segments=fitted_segments) if burn_subs else None
|
||||
|
||||
# ── Smart Fit video retime (two-tier) ─────────────────────────────────
|
||||
# Tier 1 (≤48 chunks): single filter_complex graph inlined into the mux
|
||||
# command below. Tier 2: batched slice renders joined by the concat
|
||||
# demuxer into an intermediate file, muxed as an extra input. Failures
|
||||
# fall back to an un-retimed export with a structured warning rather
|
||||
# than failing the whole download.
|
||||
retime_decision = None
|
||||
retime_warning: "dict | None" = None
|
||||
smart_track_dur = 0.0
|
||||
if retime_kind == "smart_fit" and retime_entry:
|
||||
smart_orig_dur = float(retime_entry.get("orig_duration") or job.get("duration") or 0.0)
|
||||
smart_track_dur = float(
|
||||
retime_entry.get("total_duration")
|
||||
or (filtered_tracks.get(default_track) or {}).get("duration")
|
||||
or 0.0
|
||||
)
|
||||
# A fresh export is a fresh user intent — clear any sticky abort flag
|
||||
# from a previous /dub/abort so it can't kill this run's first batch.
|
||||
job.pop("aborted", None)
|
||||
# realpath-normalised + containment-checked inline at the sink (the
|
||||
# file's established pattern — CodeQL does not track the guard
|
||||
# through a helper's return value).
|
||||
_base = os.path.realpath(DUB_DIR)
|
||||
retime_work_path = os.path.realpath(
|
||||
os.path.join(exports_dir, f"retimed_{stamp}.mp4")
|
||||
)
|
||||
if retime_work_path != _base and not retime_work_path.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid export path")
|
||||
try:
|
||||
retime_decision = await prepare_smart_fit_video(
|
||||
job_id=job_id,
|
||||
ffmpeg=ffmpeg,
|
||||
video_path=video_path,
|
||||
plan=retime_entry["plan"],
|
||||
orig_dur=smart_orig_dur,
|
||||
track_dur=smart_track_dur,
|
||||
work_path=retime_work_path,
|
||||
abort_check=lambda: bool(job.get("aborted")),
|
||||
)
|
||||
except Exception as e:
|
||||
if (isinstance(e, RetimeError) and e.stage == "aborted") or job.get("aborted"):
|
||||
raise HTTPException(status_code=409, detail="Export aborted")
|
||||
from core.failure import build_failure
|
||||
retime_warning = build_failure(e, stage="video-retime", include_diagnostic=False)
|
||||
job["last_export_warning"] = {"type": "video_retime_fallback", **retime_warning}
|
||||
logger.error(
|
||||
"Smart Fit video retime failed for job %s — exporting "
|
||||
"without per-segment retime: %s",
|
||||
job_id.replace("\n", " ").replace("\r", " "), e,
|
||||
)
|
||||
sub_path = _write_burn_srt(job, exports_dir, stamp, dual) if burn_subs else None
|
||||
|
||||
cmd = [ffmpeg, "-i", video_path]
|
||||
input_idx = 1
|
||||
|
||||
retimed_idx = None
|
||||
if retime_decision is not None and retime_decision.mode == "file":
|
||||
cmd += ["-i", retime_decision.file_path]
|
||||
retimed_idx = input_idx
|
||||
input_idx += 1
|
||||
|
||||
bg_audio = job.get("no_vocals_path") if preserve_bg else None
|
||||
bg_idx = None
|
||||
if bg_audio and os.path.exists(bg_audio) and filtered_tracks:
|
||||
@@ -563,37 +429,9 @@ async def dub_download(
|
||||
|
||||
filter_parts: list[str] = []
|
||||
video_map = "0:v:0"
|
||||
video_reencode = False
|
||||
if retime_decision is not None:
|
||||
if retime_decision.mode == "filter":
|
||||
filter_parts.append(retime_decision.graph)
|
||||
video_map = retime_decision.label
|
||||
video_reencode = True
|
||||
else:
|
||||
video_map = f"{retimed_idx}:v:0"
|
||||
# Residual drift after the batched render (fps rounding): video
|
||||
# shorter than the fitted track → freeze the last frame out to
|
||||
# the track length. Rare — the predicted tail pad inside the
|
||||
# render usually lands within tolerance.
|
||||
residual = smart_track_dur - retime_decision.video_dur
|
||||
if smart_track_dur and residual > DRIFT_TOLERANCE_S:
|
||||
filter_parts.append(
|
||||
f"[{retimed_idx}:v]tpad=stop_mode=clone:stop_duration={residual:.4f}[vtpad]"
|
||||
)
|
||||
video_map = "[vtpad]"
|
||||
video_reencode = True
|
||||
if sub_path:
|
||||
esc = _ffmpeg_filter_escape(sub_path)
|
||||
# Burn AFTER any retime so cues (already on the fitted timeline for
|
||||
# Smart Fit) land on the retimed video. Without retime this reduces
|
||||
# to the legacy `[0:v]subtitles=…[vsub]` graph.
|
||||
if video_map.startswith("["):
|
||||
sub_src = video_map
|
||||
elif retimed_idx is not None:
|
||||
sub_src = f"[{retimed_idx}:v]"
|
||||
else:
|
||||
sub_src = "[0:v]"
|
||||
filter_parts.append(f"{sub_src}subtitles='{esc}'[vsub]")
|
||||
filter_parts.append(f"[0:v]subtitles='{esc}'[vsub]")
|
||||
video_map = "[vsub]"
|
||||
if stretch_entry:
|
||||
orig_dur = float(stretch_entry.get("orig_duration") or job.get("duration") or 0.0)
|
||||
@@ -609,32 +447,10 @@ async def dub_download(
|
||||
if include_original:
|
||||
cmd += ["-map", "0:a:0"]
|
||||
|
||||
# Smart Fit drift absorption, audio side: when the retimed video runs
|
||||
# longer than the fitted track (its tail passes through at 1.0× beyond
|
||||
# the last cue, or encoder rounding), pad the dub-track chain with
|
||||
# silence out to the video length so players don't end audio early.
|
||||
apad_dur = 0.0
|
||||
if (
|
||||
retime_decision is not None
|
||||
and smart_track_dur
|
||||
and retime_decision.video_dur - smart_track_dur > DRIFT_TOLERANCE_S
|
||||
):
|
||||
apad_dur = retime_decision.video_dur
|
||||
|
||||
if bg_idx is not None:
|
||||
for i, t in enumerate(tracks_to_process):
|
||||
out_label = f"[aout{i}]"
|
||||
chain = f"[{bg_idx}:a][{t['idx']}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2"
|
||||
if apad_dur:
|
||||
chain += f",apad=whole_dur={apad_dur:.4f}"
|
||||
filter_parts.append(chain + out_label)
|
||||
t["out_label"] = out_label
|
||||
for t in tracks_to_process:
|
||||
cmd += ["-map", t["out_label"]]
|
||||
elif apad_dur:
|
||||
for i, t in enumerate(tracks_to_process):
|
||||
out_label = f"[aout{i}]"
|
||||
filter_parts.append(f"[{t['idx']}:a]apad=whole_dur={apad_dur:.4f}{out_label}")
|
||||
filter_parts.append(f"[{bg_idx}:a][{t['idx']}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2{out_label}")
|
||||
t["out_label"] = out_label
|
||||
for t in tracks_to_process:
|
||||
cmd += ["-map", t["out_label"]]
|
||||
@@ -645,11 +461,10 @@ async def dub_download(
|
||||
if filter_parts:
|
||||
cmd += ["-filter_complex", ";".join(filter_parts)]
|
||||
|
||||
# Burning subs or per-segment video retime both force a real video
|
||||
# re-encode; stream-copy is viable when nothing touches the video
|
||||
# filter chain — including the batched Smart Fit path, whose retimed
|
||||
# intermediate is already encoded with these exact settings.
|
||||
if sub_path or stretch_entry or video_reencode:
|
||||
# Burning subs or per-segment video stretch both force a real video
|
||||
# re-encode; stream-copy is only viable when nothing touches the video
|
||||
# filter chain.
|
||||
if sub_path or stretch_entry:
|
||||
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
|
||||
else:
|
||||
cmd += ["-c:v", "copy"]
|
||||
@@ -682,17 +497,17 @@ async def dub_download(
|
||||
break
|
||||
cmd += [f"-disposition:a:{target_idx}", "default"]
|
||||
|
||||
# When retiming (legacy stretch_video or Smart Fit) the video and audio
|
||||
# durations should match within sub-frame precision, but `-shortest`
|
||||
# can still cut off the trailing frame; let ffmpeg keep both streams.
|
||||
# Otherwise keep the legacy `-shortest` so a slightly-overrunning track
|
||||
# doesn't extend the mux past the video.
|
||||
if not stretch_entry and retime_decision is None:
|
||||
# In stretch_video mode the video and audio durations should match
|
||||
# within sub-frame precision, but `-shortest` can still cut off the
|
||||
# trailing frame; let ffmpeg keep both streams. Otherwise keep the
|
||||
# legacy `-shortest` so a slightly-overrunning track doesn't extend
|
||||
# the mux past the video.
|
||||
if not stretch_entry:
|
||||
cmd += ["-shortest"]
|
||||
cmd += [output_path, "-y"]
|
||||
|
||||
try:
|
||||
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0, job_id=job_id)
|
||||
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0)
|
||||
if rc != 0:
|
||||
raise Exception(stderr.decode(errors="replace") if stderr else "ffmpeg mux non-zero")
|
||||
except asyncio.TimeoutError:
|
||||
@@ -704,14 +519,6 @@ async def dub_download(
|
||||
status_code=500,
|
||||
detail=f"ffmpeg failed to combine video + dubbed audio: {e}. Verify ffmpeg is installed (`ffmpeg -version`), and check that every dubbed track file exists in the job folder.",
|
||||
)
|
||||
finally:
|
||||
# The batched retime intermediate is a full re-encoded video — never
|
||||
# leave it behind (success or failure; it's stamp-unique, no reuse).
|
||||
if retime_decision is not None and retime_decision.mode == "file":
|
||||
try:
|
||||
os.remove(retime_decision.file_path)
|
||||
except OSError as e:
|
||||
logger.debug("cleanup remove failed: %s", e)
|
||||
|
||||
if not os.path.exists(output_path) or os.path.getsize(output_path) == 0:
|
||||
raise HTTPException(status_code=500, detail="ffmpeg mux produced no output file")
|
||||
@@ -721,22 +528,12 @@ async def dub_download(
|
||||
safe_name = ''.join(c for c in base_name if c.isalnum() or c in '-_ ').strip() or 'output'
|
||||
dl_name = f"dubbed_{safe_name}_{stamp}.mp4"
|
||||
|
||||
# Structured warning surface for the Smart Fit fallback ladder: header is
|
||||
# a fixed ASCII token (FileResponse headers must be latin-1 safe); the
|
||||
# full build_failure payload is persisted on the job for the UI to read.
|
||||
extra_headers = {}
|
||||
if retime_warning is not None:
|
||||
extra_headers["X-Dub-Export-Warning"] = "video-retime-fallback"
|
||||
|
||||
if save_path:
|
||||
result = _native_save(output_path, save_path, dl_name, media_type="video/mp4")
|
||||
if retime_warning is not None:
|
||||
result["warning"] = {"type": "video_retime_fallback", **retime_warning}
|
||||
return result
|
||||
return _native_save(output_path, save_path, dl_name, media_type="video/mp4")
|
||||
|
||||
return FileResponse(
|
||||
output_path, media_type="video/mp4",
|
||||
headers={"Content-Disposition": f'attachment; filename="{dl_name}"', **extra_headers},
|
||||
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
|
||||
)
|
||||
|
||||
|
||||
@@ -771,21 +568,6 @@ async def dub_get_media(job_id: str):
|
||||
ext = os.path.splitext(video_path)[1].lower()
|
||||
return FileResponse(video_path, media_type=_MEDIA_TYPES.get(ext, "video/mp4"))
|
||||
|
||||
# One mux at a time per preview file. Without this, two overlapping requests
|
||||
# (e.g. the <video> element remounting right after a re-dub) both ran ffmpeg
|
||||
# against the same output path, and the mtime check below saw the half-written
|
||||
# file as a valid cache — serving a truncated MP4 that left the player stuck
|
||||
# loading forever (#281).
|
||||
_preview_mux_locks: dict[str, asyncio.Lock] = {}
|
||||
|
||||
|
||||
def _preview_lock(path: str) -> asyncio.Lock:
|
||||
lock = _preview_mux_locks.get(path)
|
||||
if lock is None:
|
||||
lock = _preview_mux_locks.setdefault(path, asyncio.Lock())
|
||||
return lock
|
||||
|
||||
|
||||
@router.get("/dub/preview-video/{job_id}")
|
||||
async def dub_preview_video(
|
||||
job_id: str,
|
||||
@@ -797,11 +579,6 @@ async def dub_preview_video(
|
||||
Caches per lang+preserve_bg combination under exports/preview_{lang}_{bg}.mp4.
|
||||
Cache is invalidated when the underlying dubbed track mtime is newer than the cache.
|
||||
"""
|
||||
# Strict allowlist on the path param BEFORE it reaches any filesystem
|
||||
# path or ffmpeg argv (exports dir, preview/retime work paths) — same
|
||||
# boundary check as dub_download.
|
||||
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", job_id):
|
||||
raise HTTPException(status_code=400, detail="Invalid job id")
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
@@ -822,92 +599,23 @@ async def dub_preview_video(
|
||||
bg_audio = job.get("no_vocals_path") if preserve_bg else None
|
||||
has_bg = bool(bg_audio and os.path.exists(bg_audio))
|
||||
|
||||
if not _SAFE_LANG.match(lang):
|
||||
raise HTTPException(status_code=400, detail="Invalid lang")
|
||||
# realpath-normalised + containment-checked inline BEFORE any filesystem
|
||||
# access so the guard dominates every sink (the file's established
|
||||
# pattern — see dub_preview_segment; CodeQL does not track the guard
|
||||
# through a helper's return value).
|
||||
_base = os.path.realpath(DUB_DIR)
|
||||
exports_dir = os.path.realpath(os.path.join(_base, job_id, "exports"))
|
||||
if not exports_dir.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid job id")
|
||||
exports_dir = os.path.join(DUB_DIR, job_id, "exports")
|
||||
os.makedirs(exports_dir, exist_ok=True)
|
||||
bg_suffix = "bg" if (preserve_bg and has_bg) else "nobg"
|
||||
preview_path = os.path.realpath(
|
||||
os.path.join(exports_dir, f"preview_{lang}_{bg_suffix}.mp4")
|
||||
)
|
||||
if not preview_path.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid path")
|
||||
preview_path = os.path.join(exports_dir, f"preview_{lang}_{bg_suffix}.mp4")
|
||||
|
||||
track_mtime = os.path.getmtime(track_path)
|
||||
cache_ok = (
|
||||
os.path.exists(preview_path)
|
||||
and os.path.getsize(preview_path) > 0
|
||||
and os.path.getmtime(preview_path) >= track_mtime
|
||||
)
|
||||
|
||||
def _cache_ok() -> bool:
|
||||
return (
|
||||
os.path.exists(preview_path)
|
||||
and os.path.getsize(preview_path) > 0
|
||||
and os.path.getmtime(preview_path) >= track_mtime
|
||||
)
|
||||
|
||||
async def _mux_preview():
|
||||
# Mux into a temp file and os.replace() into place so a concurrent
|
||||
# reader never sees a partially-written preview (#281: video stuck
|
||||
# loading forever after a re-dub).
|
||||
mux_path = preview_path + ".tmp.mp4"
|
||||
if not cache_ok:
|
||||
ffmpeg = find_ffmpeg()
|
||||
# Resolve the same retime plan the download path uses (legacy
|
||||
# stretch_video or Smart Fit) so the in-app preview matches export.
|
||||
retime = _video_retime_plan_for(job, lang)
|
||||
retime_kind, retime_entry = retime if retime else (None, None)
|
||||
stretch_entry = retime_entry if retime_kind == "stretch_video" else None
|
||||
|
||||
retime_decision = None
|
||||
smart_track_dur = 0.0
|
||||
if retime_kind == "smart_fit" and retime_entry:
|
||||
smart_orig_dur = float(retime_entry.get("orig_duration") or job.get("duration") or 0.0)
|
||||
smart_track_dur = float(
|
||||
retime_entry.get("total_duration") or track_info.get("duration") or 0.0
|
||||
)
|
||||
job.pop("aborted", None) # fresh user intent — clear sticky abort
|
||||
# realpath-normalised + containment-checked inline at the sink
|
||||
# (same pattern as preview_path above — _base is the realpath
|
||||
# of DUB_DIR from the top of this endpoint).
|
||||
retime_work_path = os.path.realpath(os.path.join(
|
||||
exports_dir, f"preview_retimed_{lang}_{bg_suffix}.tmp.mp4",
|
||||
))
|
||||
if retime_work_path != _base and not retime_work_path.startswith(_base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid export path")
|
||||
try:
|
||||
retime_decision = await prepare_smart_fit_video(
|
||||
job_id=job_id,
|
||||
ffmpeg=ffmpeg,
|
||||
video_path=video_path,
|
||||
plan=retime_entry["plan"],
|
||||
orig_dur=smart_orig_dur,
|
||||
track_dur=smart_track_dur,
|
||||
work_path=retime_work_path,
|
||||
abort_check=lambda: bool(job.get("aborted")),
|
||||
)
|
||||
except Exception as e:
|
||||
if (isinstance(e, RetimeError) and e.stage == "aborted") or job.get("aborted"):
|
||||
raise HTTPException(status_code=409, detail="Preview aborted")
|
||||
# Preview is best-effort: fall back to the un-retimed video
|
||||
# rather than a black player. The export path surfaces the
|
||||
# structured warning; here we just log.
|
||||
retime_decision = None
|
||||
logger.error(
|
||||
"Smart Fit preview retime failed for job %s — previewing "
|
||||
"without per-segment retime: %s",
|
||||
job_id.replace("\n", " ").replace("\r", " "), e,
|
||||
)
|
||||
|
||||
stretch_entry = _video_stretch_plan_for(job, lang)
|
||||
cmd = [ffmpeg, "-i", video_path]
|
||||
input_idx = 1
|
||||
retimed_idx = None
|
||||
if retime_decision is not None and retime_decision.mode == "file":
|
||||
cmd += ["-i", retime_decision.file_path]
|
||||
retimed_idx = input_idx
|
||||
input_idx += 1
|
||||
if preserve_bg and has_bg:
|
||||
cmd += ["-i", bg_audio]
|
||||
bg_idx = input_idx
|
||||
@@ -917,13 +625,12 @@ async def dub_preview_video(
|
||||
cmd += ["-i", track_path]
|
||||
track_idx = input_idx
|
||||
|
||||
# Build filter graph. Under a retime plan we splice the source video
|
||||
# into per-segment chunks, setpts each to match the dub audio
|
||||
# Build filter graph. In stretch_video mode we splice the source
|
||||
# video into per-segment chunks, setpts each to match the dub audio
|
||||
# layout, and concat them — so audio plays at natural rate and the
|
||||
# visuals follow. Otherwise we stream-copy video for speed.
|
||||
filter_parts: list[str] = []
|
||||
video_map = "0:v:0"
|
||||
video_reencode = False
|
||||
if stretch_entry:
|
||||
orig_dur = float(stretch_entry.get("orig_duration") or job.get("duration") or 0.0)
|
||||
graph, vlabel = _build_video_stretch_filter_graph(
|
||||
@@ -932,175 +639,49 @@ async def dub_preview_video(
|
||||
if graph:
|
||||
filter_parts.append(graph)
|
||||
video_map = vlabel
|
||||
elif retime_decision is not None:
|
||||
if retime_decision.mode == "filter":
|
||||
filter_parts.append(retime_decision.graph)
|
||||
video_map = retime_decision.label
|
||||
video_reencode = True
|
||||
else:
|
||||
video_map = f"{retimed_idx}:v:0"
|
||||
residual = smart_track_dur - retime_decision.video_dur
|
||||
if smart_track_dur and residual > DRIFT_TOLERANCE_S:
|
||||
filter_parts.append(
|
||||
f"[{retimed_idx}:v]tpad=stop_mode=clone:stop_duration={residual:.4f}[vtpad]"
|
||||
)
|
||||
video_map = "[vtpad]"
|
||||
video_reencode = True
|
||||
|
||||
# Smart Fit drift absorption (audio): silence-pad the dub chain out
|
||||
# to the retimed video length so the preview doesn't end audio early.
|
||||
apad_dur = 0.0
|
||||
if (
|
||||
retime_decision is not None
|
||||
and smart_track_dur
|
||||
and retime_decision.video_dur - smart_track_dur > DRIFT_TOLERANCE_S
|
||||
):
|
||||
apad_dur = retime_decision.video_dur
|
||||
|
||||
audio_map = f"{track_idx}:a:0"
|
||||
if bg_idx is not None:
|
||||
chain = f"[{bg_idx}:a][{track_idx}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2"
|
||||
if apad_dur:
|
||||
chain += f",apad=whole_dur={apad_dur:.4f}"
|
||||
filter_parts.append(chain + "[aout]")
|
||||
audio_map = "[aout]"
|
||||
elif apad_dur:
|
||||
filter_parts.append(f"[{track_idx}:a]apad=whole_dur={apad_dur:.4f}[aout]")
|
||||
audio_map = "[aout]"
|
||||
filter_parts.append(
|
||||
f"[{bg_idx}:a][{track_idx}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2[aout]"
|
||||
)
|
||||
|
||||
cmd += ["-map", video_map]
|
||||
cmd += ["-map", audio_map]
|
||||
if bg_idx is not None:
|
||||
cmd += ["-map", "[aout]"]
|
||||
else:
|
||||
cmd += ["-map", f"{track_idx}:a:0"]
|
||||
if filter_parts:
|
||||
cmd += ["-filter_complex", ";".join(filter_parts)]
|
||||
|
||||
# Retime path needs a real encode; stream-copy otherwise (the batched
|
||||
# Smart Fit intermediate is already encoded — copy unless tpad'ed).
|
||||
if stretch_entry or video_reencode:
|
||||
# Stretch path needs a real encode; stream-copy otherwise.
|
||||
if stretch_entry:
|
||||
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
|
||||
else:
|
||||
cmd += ["-c:v", "copy"]
|
||||
cmd += ["-c:a", "aac", "-b:a", "192k"]
|
||||
# `-shortest` would cut the retimed video at the (slightly different)
|
||||
# `-shortest` would cut the stretched video at the (slightly different)
|
||||
# audio length and lose the trailing frame; only use it on the copy path.
|
||||
if not stretch_entry and retime_decision is None:
|
||||
if not stretch_entry:
|
||||
cmd += ["-shortest"]
|
||||
cmd += [mux_path, "-y"]
|
||||
|
||||
def _discard_tmp():
|
||||
try:
|
||||
os.remove(mux_path)
|
||||
except OSError as e:
|
||||
logger.debug("cleanup remove failed: %s", e)
|
||||
cmd += [preview_path, "-y"]
|
||||
|
||||
try:
|
||||
rc, _, stderr = await run_ffmpeg(cmd, timeout=900.0, job_id=job_id)
|
||||
rc, _, stderr = await run_ffmpeg(cmd, timeout=900.0)
|
||||
if rc != 0:
|
||||
raise Exception(stderr.decode(errors="replace") if stderr else "ffmpeg mux non-zero")
|
||||
if not os.path.exists(mux_path) or os.path.getsize(mux_path) == 0:
|
||||
raise Exception("preview mux produced empty file")
|
||||
except asyncio.TimeoutError:
|
||||
_discard_tmp()
|
||||
raise HTTPException(status_code=504, detail="preview mux timed out")
|
||||
except HTTPException:
|
||||
_discard_tmp()
|
||||
raise
|
||||
except Exception as e:
|
||||
_discard_tmp()
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"ffmpeg failed to build the preview stream: {str(e)[:300]}. This usually means the source video can't be re-encoded on the fly — try downloading the MP4 instead.",
|
||||
)
|
||||
finally:
|
||||
if retime_decision is not None and retime_decision.mode == "file":
|
||||
try:
|
||||
os.remove(retime_decision.file_path)
|
||||
except OSError as e:
|
||||
# Best-effort scratch cleanup — never fail the export.
|
||||
logger.debug("retime intermediate cleanup failed: %s", e)
|
||||
|
||||
os.replace(mux_path, preview_path)
|
||||
if not os.path.exists(preview_path) or os.path.getsize(preview_path) == 0:
|
||||
raise HTTPException(status_code=500, detail="preview mux produced empty file")
|
||||
|
||||
async with _preview_lock(preview_path):
|
||||
if not _cache_ok():
|
||||
await _mux_preview()
|
||||
|
||||
# no-store: the URL is stable across re-dubs, so any HTTP-level caching
|
||||
# in the WebView would keep showing the previous dub after a re-generate
|
||||
# (#281: "edits don't change the result").
|
||||
return FileResponse(
|
||||
preview_path,
|
||||
media_type="video/mp4",
|
||||
headers={"Cache-Control": "no-store"},
|
||||
)
|
||||
|
||||
|
||||
def _compute_onsets_sync(src_path: str) -> list[float]:
|
||||
"""Blocking part of onset analysis — runs in a worker thread."""
|
||||
import soundfile as sf
|
||||
from services.onset_align import detect_speech_onsets
|
||||
audio, sr = sf.read(src_path, dtype="float32")
|
||||
return detect_speech_onsets(audio, sr)
|
||||
|
||||
|
||||
@router.get("/dub/onsets/{job_id}")
|
||||
async def dub_get_onsets(job_id: str):
|
||||
"""Speech-onset times for the timeline editor's snap-to-onset ticks (#280).
|
||||
|
||||
Prefers the Demucs-isolated vocals track (clean speech energy); falls
|
||||
back to the mixed audio. Computed once per job and cached as
|
||||
``onsets.json`` in the job directory; recomputed if the source audio is
|
||||
newer than the cache (e.g. re-ingest into the same job dir).
|
||||
"""
|
||||
import json
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
vocals = job.get("vocals_path")
|
||||
mix = job.get("audio_path")
|
||||
if vocals and os.path.exists(vocals):
|
||||
src_path, source = vocals, "vocals"
|
||||
elif mix and os.path.exists(mix):
|
||||
src_path, source = mix, "mix"
|
||||
else:
|
||||
raise HTTPException(status_code=404, detail="No audio track available for onset analysis")
|
||||
|
||||
# Containment inlined (not via _safe_job_path): CodeQL can't track the
|
||||
# sanitizer through a helper's return — the file's established idiom.
|
||||
base = os.path.realpath(DUB_DIR)
|
||||
cache_path = os.path.realpath(os.path.join(base, job_id, "onsets.json"))
|
||||
if not cache_path.startswith(base + os.sep):
|
||||
raise HTTPException(status_code=400, detail="Invalid job id")
|
||||
try:
|
||||
if (
|
||||
os.path.exists(cache_path)
|
||||
and os.path.getmtime(cache_path) >= os.path.getmtime(src_path)
|
||||
):
|
||||
with open(cache_path, "r", encoding="utf-8") as f:
|
||||
cached = json.load(f)
|
||||
if isinstance(cached, dict) and isinstance(cached.get("onsets"), list):
|
||||
return cached
|
||||
except (OSError, ValueError):
|
||||
pass # unreadable/corrupt cache → recompute below
|
||||
|
||||
try:
|
||||
onsets = await asyncio.to_thread(_compute_onsets_sync, src_path)
|
||||
except Exception as e:
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"Onset analysis failed: {str(e)[:200]}",
|
||||
)
|
||||
|
||||
payload = {"onsets": onsets, "source": source}
|
||||
try:
|
||||
os.makedirs(os.path.dirname(cache_path), exist_ok=True)
|
||||
tmp_path = cache_path + ".tmp"
|
||||
with open(tmp_path, "w", encoding="utf-8") as f:
|
||||
json.dump(payload, f)
|
||||
os.replace(tmp_path, cache_path)
|
||||
except OSError as e:
|
||||
logger.warning("onsets cache write failed for %s: %s", job_id, e)
|
||||
return payload
|
||||
return FileResponse(preview_path, media_type="video/mp4")
|
||||
|
||||
|
||||
@router.get("/dub/thumb/{job_id}")
|
||||
@@ -1148,94 +729,6 @@ async def dub_preview_segment(job_id: str, segment_index: int):
|
||||
return FileResponse(seg_path, media_type="audio/wav")
|
||||
|
||||
|
||||
# ── Second-pass ASR QC (Wave 3.3 / Spec 5) ───────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/dub/qc/{job_id}")
|
||||
async def dub_qc_pass(job_id: str, lang: str = Query(None), drift_threshold: float = Query(0.5)):
|
||||
"""Re-recognize the dubbed audio and flag lines whose recognized text
|
||||
drifts from the target text. Opt-in, never fatal: the dub is untouched —
|
||||
this only annotates segments with a per-line drift score and a measured
|
||||
start/end, surfaced as "verify this line" markers feeding incremental
|
||||
re-dub. The generated text stays authoritative (design delta from
|
||||
pyvideotrans, which overwrites subtitles)."""
|
||||
from services import dub_qc
|
||||
from services.dub_pipeline import put_job, save_job
|
||||
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
tracks = job.get("dubbed_tracks", {})
|
||||
if lang and lang in tracks:
|
||||
wav_path = tracks[lang]["path"]
|
||||
elif tracks:
|
||||
wav_path = list(tracks.values())[0]["path"]
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail="No dubbed audio track generated yet")
|
||||
if not os.path.exists(wav_path):
|
||||
raise HTTPException(status_code=404, detail="Dubbed audio file not found")
|
||||
|
||||
segments = job.get("segments") or []
|
||||
if not segments:
|
||||
raise HTTPException(status_code=400, detail="Job has no segments")
|
||||
|
||||
def _recognize():
|
||||
from services.asr_backend import get_active_asr_backend
|
||||
backend = get_active_asr_backend()
|
||||
result = backend.transcribe(wav_path, word_timestamps=False)
|
||||
return result.get("segments", []), backend.id
|
||||
|
||||
try:
|
||||
from services.model_manager import _get_gpu_pool
|
||||
loop = asyncio.get_running_loop()
|
||||
recognized, engine_id = await loop.run_in_executor(_get_gpu_pool(), _recognize)
|
||||
except Exception as e:
|
||||
logger.exception("dub QC ASR pass failed for %s", job_id)
|
||||
raise HTTPException(status_code=500, detail=f"QC transcription failed: {e}")
|
||||
|
||||
seg_ids = job.get("seg_order") or [s.get("id", i) for i, s in enumerate(segments)]
|
||||
scored = dub_qc.score_dub(segments, recognized, drift_threshold=drift_threshold, seg_ids=seg_ids)
|
||||
|
||||
# Annotate each segment (non-destructive — content text untouched).
|
||||
by_id = {q.seg_id: q for q in scored}
|
||||
for i, s in enumerate(segments):
|
||||
sid = str(seg_ids[i]) if i < len(seg_ids) else str(s.get("id", i))
|
||||
q = by_id.get(sid)
|
||||
if q is None:
|
||||
continue
|
||||
s["qc_drift"] = q.drift
|
||||
s["qc_flagged"] = q.flagged
|
||||
s["qc_recognized"] = q.recognized_text
|
||||
if q.new_start is not None:
|
||||
s["qc_measured_start"] = q.new_start
|
||||
s["qc_measured_end"] = q.new_end
|
||||
put_job(job_id, job)
|
||||
save_job(job_id, job)
|
||||
|
||||
flagged = [q for q in scored if q.flagged]
|
||||
payload = json.dumps({"event": "qc_done", "engine": engine_id,
|
||||
"flagged": len(flagged), "total": len(scored)})
|
||||
try:
|
||||
from core import job_store
|
||||
job_store.append_event(job_id, f"data: {payload}\n\n")
|
||||
except Exception as e:
|
||||
# QC event fan-out is best-effort; the scores are already in the response.
|
||||
logger.debug("QC event append failed: %s", e)
|
||||
|
||||
return {
|
||||
"engine": engine_id,
|
||||
"total": len(scored),
|
||||
"flagged_count": len(flagged),
|
||||
"drift_threshold": drift_threshold,
|
||||
"segments": [
|
||||
{"seg_id": q.seg_id, "drift": q.drift, "flagged": q.flagged,
|
||||
"recognized_text": q.recognized_text,
|
||||
"measured_start": q.new_start, "measured_end": q.new_end}
|
||||
for q in scored
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/dub/download-audio/{job_id}")
|
||||
@router.get("/dub/download-audio/{job_id}/{filename}")
|
||||
async def dub_download_audio(job_id: str, lang: str = Query(None), preserve_bg: bool = Query(True), save_path: str = Query("")):
|
||||
@@ -1313,31 +806,9 @@ def _pick_subtitle_text(seg: dict, dual: bool) -> str:
|
||||
return f"{translated}\n<i>{original}</i>"
|
||||
|
||||
|
||||
# Subtitles deliberately have no ?save_path= variant: they're small text
|
||||
# bodies, so the Tauri side fetches them raw and writes the file itself via
|
||||
# the save_text_file command — the OS save dialog is the write authorization
|
||||
# (#309). The frontend's JSON-envelope save flow stays for binary exports.
|
||||
|
||||
|
||||
def _fitted_cue_times(job: dict, lang: str | None) -> list | None:
|
||||
"""Per-segment (start, end) on the fitted timeline when this job used
|
||||
stretch_video; None to use the original segment times. (Wave 3.1.)"""
|
||||
tracks = job.get("dubbed_tracks", {})
|
||||
lc = lang if (lang and lang in tracks) else (next(iter(tracks), None))
|
||||
entry = _video_stretch_plan_for(job, lc) if lc else None
|
||||
if not entry:
|
||||
return None
|
||||
from services.fitted_subtitles import fitted_cues
|
||||
return fitted_cues(job.get("segments", []), entry["plan"])
|
||||
|
||||
|
||||
@router.get("/dub/srt/{job_id}")
|
||||
@router.get("/dub/srt/{job_id}/{filename}")
|
||||
async def dub_export_srt(
|
||||
job_id: str,
|
||||
dual: bool = False,
|
||||
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
|
||||
):
|
||||
async def dub_export_srt(job_id: str, dual: bool = False):
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
@@ -1346,31 +817,22 @@ async def dub_export_srt(
|
||||
if not segments:
|
||||
raise HTTPException(status_code=400, detail="No transcript segments available")
|
||||
|
||||
# Subtitles must follow the audio the viewer hears, per timing strategy:
|
||||
# Smart Fit (phase B) overlays the fitted segment times directly;
|
||||
# stretch_video (Wave 3.1) regenerates cue times from the stretch plan.
|
||||
# Neither applies → original times.
|
||||
fitted = _fitted_segments_for(job, lang)
|
||||
if fitted:
|
||||
segments = _apply_fitted_times(segments, fitted)
|
||||
cues = None if fitted else _fitted_cue_times(job, lang)
|
||||
|
||||
srt_lines = []
|
||||
for i, seg in enumerate(segments):
|
||||
s, e = cues[i] if cues else (seg["start"], seg["end"])
|
||||
start_ts = _format_srt_time(seg["start"])
|
||||
end_ts = _format_srt_time(seg["end"])
|
||||
srt_lines.append(f"{i + 1}")
|
||||
srt_lines.append(f"{_format_srt_time(s)} --> {_format_srt_time(e)}")
|
||||
srt_lines.append(f"{start_ts} --> {end_ts}")
|
||||
srt_lines.append(_pick_subtitle_text(seg, dual))
|
||||
srt_lines.append("")
|
||||
|
||||
srt_content = "\n".join(srt_lines)
|
||||
base_name = os.path.splitext(job.get('filename', 'video'))[0]
|
||||
suffix = "_dual" if dual else ""
|
||||
dl_name = f"subtitles_{base_name}{suffix}.srt"
|
||||
return Response(
|
||||
content=srt_content,
|
||||
media_type="text/plain",
|
||||
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
|
||||
headers={"Content-Disposition": f'attachment; filename="subtitles_{base_name}{suffix}.srt"'},
|
||||
)
|
||||
|
||||
def _format_vtt_time(seconds):
|
||||
@@ -1382,11 +844,7 @@ def _format_vtt_time(seconds):
|
||||
|
||||
@router.get("/dub/vtt/{job_id}")
|
||||
@router.get("/dub/vtt/{job_id}/{filename}")
|
||||
async def dub_export_vtt(
|
||||
job_id: str,
|
||||
dual: bool = False,
|
||||
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
|
||||
):
|
||||
async def dub_export_vtt(job_id: str, dual: bool = False):
|
||||
job = _get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
@@ -1395,28 +853,22 @@ async def dub_export_vtt(
|
||||
if not segments:
|
||||
raise HTTPException(status_code=400, detail="No transcript segments available")
|
||||
|
||||
# Same strategy-aware cue timing as /dub/srt (see comment there).
|
||||
fitted = _fitted_segments_for(job, lang)
|
||||
if fitted:
|
||||
segments = _apply_fitted_times(segments, fitted)
|
||||
cues = None if fitted else _fitted_cue_times(job, lang)
|
||||
|
||||
vtt_lines = ["WEBVTT", ""]
|
||||
for i, seg in enumerate(segments):
|
||||
s, e = cues[i] if cues else (seg["start"], seg["end"])
|
||||
start_ts = _format_vtt_time(seg["start"])
|
||||
end_ts = _format_vtt_time(seg["end"])
|
||||
vtt_lines.append(str(i + 1))
|
||||
vtt_lines.append(f"{_format_vtt_time(s)} --> {_format_vtt_time(e)}")
|
||||
vtt_lines.append(f"{start_ts} --> {end_ts}")
|
||||
vtt_lines.append(_pick_subtitle_text(seg, dual))
|
||||
vtt_lines.append("")
|
||||
|
||||
vtt_content = "\n".join(vtt_lines)
|
||||
base_name = os.path.splitext(job.get('filename', 'video'))[0]
|
||||
suffix = "_dual" if dual else ""
|
||||
dl_name = f"subtitles_{base_name}{suffix}.vtt"
|
||||
return Response(
|
||||
content=vtt_content,
|
||||
media_type="text/vtt",
|
||||
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
|
||||
headers={"Content-Disposition": f'attachment; filename="subtitles_{base_name}{suffix}.vtt"'},
|
||||
)
|
||||
|
||||
|
||||
|
||||
+101
-239
@@ -3,6 +3,7 @@ import json
|
||||
import logging
|
||||
import time
|
||||
import asyncio
|
||||
import numpy as np
|
||||
import torch
|
||||
import torchaudio
|
||||
from fastapi import APIRouter, HTTPException
|
||||
@@ -14,18 +15,9 @@ from schemas.requests import DubRequest
|
||||
from services.model_manager import get_model, _gpu_pool
|
||||
from services.audio_dsp import apply_mastering, normalize_audio, apply_effects_chain, get_effect_chain
|
||||
from services.audio_io import atomic_save_wav, _safe_torchaudio_save
|
||||
from services.ffmpeg_utils import (
|
||||
find_ffmpeg,
|
||||
spawn_subprocess,
|
||||
# Moved to ffmpeg_utils so the Smart Fit export pipeline (Phase B) can
|
||||
# reuse them; re-imported here so `dub_generate._atempo_chain` /
|
||||
# `_pitch_preserving_stretch` keep working for existing importers.
|
||||
_atempo_chain,
|
||||
_pitch_preserving_stretch,
|
||||
)
|
||||
from services.ffmpeg_utils import find_ffmpeg, spawn_subprocess
|
||||
from services.rvc import apply_rvc, is_enabled as rvc_is_enabled
|
||||
from services.incremental import segment_fingerprint, fit_fingerprint
|
||||
from services.fit_planner import FitParams, plan_fit
|
||||
from services.incremental import segment_fingerprint
|
||||
from services.watermark import embed_watermark
|
||||
from api.routers.dub_core import _get_job, _save_job
|
||||
|
||||
@@ -47,44 +39,75 @@ GAP_OVERFLOW_MAX_S = 0.25
|
||||
GAP_OVERFLOW_BUFFER_S = 0.05
|
||||
|
||||
|
||||
def _sync_job_segments(job: dict, req: DubRequest) -> None:
|
||||
"""Persist the segments this dub was actually generated from back onto the job.
|
||||
def _atempo_chain(ratio: float) -> str:
|
||||
"""Build an `atempo=…,atempo=…` filter chain for arbitrary ratios.
|
||||
|
||||
The editor only sends the (translated / user-edited) segment text in the
|
||||
generate request; the job itself kept the original-language ASR transcript.
|
||||
SRT/VTT export and ffmpeg subtitle burn-in read `job["segments"]`, so they
|
||||
rendered the source language instead of the dub the user just heard (#309).
|
||||
|
||||
Merge strategy: rebuild `job["segments"]` from the request, carrying over
|
||||
per-segment metadata (speaker_id, id, …) from the existing job segment
|
||||
matched by stable id (fallback: index). `text_original` always keeps the
|
||||
source-language text so dual-subtitle layouts can still stack it under the
|
||||
translation.
|
||||
ffmpeg's atempo filter is limited to [0.5, 2.0] per stage. Chaining
|
||||
multiple stages multiplies the effective ratio while keeping each
|
||||
individual stage inside the well-behaved range. Pitch is preserved
|
||||
(WSOLA-style time-domain stretching). ratio > 1 speeds up, < 1
|
||||
slows down.
|
||||
"""
|
||||
if not req.segments:
|
||||
return
|
||||
existing = [s for s in (job.get("segments") or []) if isinstance(s, dict)]
|
||||
by_id = {str(s["id"]): s for s in existing if s.get("id") is not None}
|
||||
seg_ids = req.segment_ids or []
|
||||
merged: list[dict] = []
|
||||
for i, seg in enumerate(req.segments):
|
||||
seg_id = seg_ids[i] if i < len(seg_ids) else None
|
||||
prev = by_id.get(str(seg_id)) if seg_id is not None else None
|
||||
if prev is None and i < len(existing):
|
||||
prev = existing[i]
|
||||
row = dict(prev) if prev else {}
|
||||
if seg_id is not None:
|
||||
# The request id is authoritative — seg_order and the per-segment
|
||||
# WAV manifest are keyed by it.
|
||||
row["id"] = seg_id
|
||||
# Source-language text survives the overwrite so dual-subtitle export
|
||||
# keeps working; never let the translation clobber it.
|
||||
row["text_original"] = row.get("text_original") or row.get("text") or ""
|
||||
row["start"] = seg.start
|
||||
row["end"] = seg.end
|
||||
row["text"] = seg.text
|
||||
merged.append(row)
|
||||
job["segments"] = merged
|
||||
stages: list[str] = []
|
||||
remaining = ratio
|
||||
while remaining > 2.0:
|
||||
stages.append("atempo=2.0")
|
||||
remaining /= 2.0
|
||||
while remaining < 0.5:
|
||||
stages.append("atempo=0.5")
|
||||
remaining /= 0.5
|
||||
stages.append(f"atempo={remaining:.6f}")
|
||||
return ",".join(stages)
|
||||
|
||||
|
||||
async def _pitch_preserving_stretch(
|
||||
wav: torch.Tensor, target_samples: int, sr: int,
|
||||
) -> torch.Tensor:
|
||||
"""Time-stretch a (1, samples) tensor to `target_samples` while
|
||||
preserving pitch, by piping the audio through `ffmpeg atempo`.
|
||||
|
||||
Async so it never blocks the event loop: it's awaited from the `_stream`
|
||||
generator, and each ffmpeg call is ~50-100 ms — a synchronous
|
||||
``subprocess.run`` here froze health-checks / SSE / every concurrent
|
||||
request for the whole multi-segment job.
|
||||
|
||||
Returns a (1, target_samples) tensor on the same device as input.
|
||||
Raises RuntimeError when ffmpeg fails — callers should fall back to
|
||||
naive linear interpolation, accepting the pitch shift, to ensure the
|
||||
output isn't silent.
|
||||
"""
|
||||
wl = int(wav.shape[-1])
|
||||
if target_samples <= 0 or wl == target_samples:
|
||||
return wav
|
||||
ratio = wl / target_samples
|
||||
filter_str = _atempo_chain(ratio)
|
||||
|
||||
# Mono float32 via stdin → ffmpeg → stdout. One subprocess per segment,
|
||||
# run off the event loop so concurrent requests stay responsive.
|
||||
arr = wav.detach().cpu().to(torch.float32).numpy().reshape(-1).astype(np.float32, copy=False)
|
||||
proc = await spawn_subprocess(
|
||||
find_ffmpeg(), "-hide_banner", "-loglevel", "error", "-y",
|
||||
"-f", "f32le", "-ar", str(sr), "-ac", "1", "-i", "pipe:0",
|
||||
"-af", filter_str,
|
||||
"-f", "f32le", "-ar", str(sr), "-ac", "1", "pipe:1",
|
||||
stdin=asyncio.subprocess.PIPE,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, stderr = await proc.communicate(input=arr.tobytes())
|
||||
if proc.returncode != 0 or not stdout:
|
||||
raise RuntimeError(
|
||||
(stderr.decode(errors="replace") or "atempo failed")[:200]
|
||||
)
|
||||
out_arr = np.frombuffer(stdout, dtype=np.float32)
|
||||
# atempo rarely lands exactly on the integer sample count, so
|
||||
# pad/trim to the requested slot length.
|
||||
if len(out_arr) < target_samples:
|
||||
pad = np.zeros(target_samples - len(out_arr), dtype=np.float32)
|
||||
out_arr = np.concatenate([out_arr, pad])
|
||||
elif len(out_arr) > target_samples:
|
||||
out_arr = out_arr[:target_samples]
|
||||
return torch.from_numpy(out_arr.copy()).unsqueeze(0).to(wav.device)
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
@@ -111,15 +134,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
# `seg_i.wav` on disk and slot into the final mix unchanged.
|
||||
regen_only = set(req.regen_only or []) if req.regen_only is not None else None
|
||||
seg_ids = req.segment_ids or []
|
||||
strategy = (req.timing_strategy or "concise").lower()
|
||||
# Strategy-transition guard: smart_fit re-mixes the *natural-rate*
|
||||
# per-segment WAVs from disk. If the previous run used strict_slot,
|
||||
# the on-disk WAVs are slot-squeezed ("slotted") — reusing them would
|
||||
# double-compress. Force one full regen; afterwards seg_wav_kind is
|
||||
# "natural" and partial regen / fit-only re-mix (regen_only=[]) work.
|
||||
# Jobs predating this field have unknown kind → also regen once.
|
||||
if strategy == "smart_fit" and regen_only is not None and job.get("seg_wav_kind") != "natural":
|
||||
regen_only = None
|
||||
# Manifest: stable segment id per current index. Per-segment WAVs are
|
||||
# named by stable id (dub_seg_path) so regen reuses the right audio after
|
||||
# reorder; index-keyed readers (preview/export) resolve via this manifest.
|
||||
@@ -170,17 +184,13 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
if cached_sr != _model.sampling_rate:
|
||||
import torchaudio.functional as AF
|
||||
cached_wav = AF.resample(cached_wav, cached_sr, _model.sampling_rate)
|
||||
# Pad/trim to slot — except smart_fit, whose mix
|
||||
# loop needs the natural-rate length to compute the
|
||||
# audio/video split (the seg_wav_kind guard above
|
||||
# guarantees these cached WAVs are natural-rate).
|
||||
if strategy != "smart_fit":
|
||||
target_samples = int(seg_duration * _model.sampling_rate)
|
||||
current_samples = cached_wav.shape[-1]
|
||||
if target_samples > current_samples:
|
||||
cached_wav = torch.nn.functional.pad(cached_wav, (0, target_samples - current_samples))
|
||||
elif current_samples > target_samples:
|
||||
cached_wav = cached_wav[..., :target_samples]
|
||||
# Pad/trim to slot.
|
||||
target_samples = int(seg_duration * _model.sampling_rate)
|
||||
current_samples = cached_wav.shape[-1]
|
||||
if target_samples > current_samples:
|
||||
cached_wav = torch.nn.functional.pad(cached_wav, (0, target_samples - current_samples))
|
||||
elif current_samples > target_samples:
|
||||
cached_wav = cached_wav[..., :target_samples]
|
||||
all_segment_wavs.append((seg.start, seg.end, cached_wav, _model.sampling_rate))
|
||||
sync_scores.append(getattr(seg, 'sync_ratio', None) or 1.0)
|
||||
_t_cache += time.perf_counter() - _t_cache_0
|
||||
@@ -204,42 +214,18 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
# (see services/speaker_clone.py) live at job["speaker_clones"]
|
||||
# keyed by speaker_id. We use the `auto:` prefix so they can't
|
||||
# collide with persistent voice_profiles.id values.
|
||||
# Wave 3.2: a per-segment clone ref (cut from this line's own
|
||||
# source audio) takes precedence over the per-speaker clone.
|
||||
if profile_id and profile_id.startswith("auto-seg:"):
|
||||
sid = profile_id[len("auto-seg:"):]
|
||||
info = (job.get("segment_clones") or {}).get(sid)
|
||||
if info:
|
||||
ref_audio = info.get("ref_audio")
|
||||
ref_text = info.get("ref_text")
|
||||
profile_id = None # prevent the voice_profiles lookup below
|
||||
|
||||
elif profile_id and profile_id.startswith("auto:"):
|
||||
# #486: an `auto:{speaker}` binding still prefers THIS
|
||||
# segment's own per-segment ref when one exists (cut from
|
||||
# this line's source audio → matches its prosody), falling
|
||||
# back to the per-speaker clone otherwise. This keeps the
|
||||
# Wave 3.2 per-segment-ref quality win while letting every
|
||||
# segment carry the UI-visible `auto:` id the dub editor's
|
||||
# Voice dropdown can actually render ("From Video →
|
||||
# Speaker N"). `seg_id` is closed over from the per-segment
|
||||
# loop below.
|
||||
seg_ref = (job.get("segment_clones") or {}).get(str(seg_id))
|
||||
if seg_ref:
|
||||
ref_audio = seg_ref.get("ref_audio")
|
||||
ref_text = seg_ref.get("ref_text")
|
||||
else:
|
||||
key = profile_id[len("auto:"):]
|
||||
clones = job.get("speaker_clones") or {}
|
||||
# Match by the safe-name key first, fall back to speaker_id.
|
||||
auto = None
|
||||
for spk, info in clones.items():
|
||||
if spk.lower().replace(" ", "_") == key or spk == key:
|
||||
auto = info
|
||||
break
|
||||
if auto:
|
||||
ref_audio = auto.get("ref_audio")
|
||||
ref_text = auto.get("ref_text")
|
||||
if profile_id and profile_id.startswith("auto:"):
|
||||
key = profile_id[len("auto:"):]
|
||||
clones = job.get("speaker_clones") or {}
|
||||
# Match by the safe-name key first, fall back to speaker_id.
|
||||
auto = None
|
||||
for spk, info in clones.items():
|
||||
if spk.lower().replace(" ", "_") == key or spk == key:
|
||||
auto = info
|
||||
break
|
||||
if auto:
|
||||
ref_audio = auto.get("ref_audio")
|
||||
ref_text = auto.get("ref_text")
|
||||
profile_id = None # prevent the voice_profiles lookup below
|
||||
|
||||
if profile_id:
|
||||
@@ -279,9 +265,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
if seg_effect_preset == "raw":
|
||||
return audio_out
|
||||
|
||||
# TODO(#312): this route runs the OmniVoice model directly (not the active
|
||||
# backend), so VoxCPM2 never reaches it. When these routes become
|
||||
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
|
||||
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
|
||||
effect_chain = get_effect_chain(seg_effect_preset)
|
||||
if effect_chain:
|
||||
@@ -327,9 +310,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
if seg_effect_preset == "raw":
|
||||
return audio_out
|
||||
|
||||
# TODO(#312): this route runs the OmniVoice model directly (not the active
|
||||
# backend), so VoxCPM2 never reaches it. When these routes become
|
||||
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
|
||||
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
|
||||
effect_chain = get_effect_chain(seg_effect_preset)
|
||||
if effect_chain:
|
||||
@@ -393,13 +373,13 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
_t_tts_0 = time.perf_counter()
|
||||
seg_effect_preset = getattr(seg, "effect_preset", None) or "broadcast"
|
||||
|
||||
# In concise / stretch_video / smart_fit modes we pass
|
||||
# dur_s=None so the TTS model speaks at its natural rate for
|
||||
# this text length — the whole point of the new timing
|
||||
# strategies is to never squeeze the speech to fit at
|
||||
# synthesis time. strict_slot keeps the legacy behaviour
|
||||
# where dur_s is the slot hint.
|
||||
_dur_for_tts = seg_duration if strategy == "strict_slot" else None
|
||||
# In concise / stretch_video modes we pass dur_s=None so the
|
||||
# TTS model speaks at its natural rate for this text length —
|
||||
# the whole point of the new timing strategies is to never
|
||||
# squeeze the speech to fit. strict_slot keeps the legacy
|
||||
# behaviour where dur_s is the slot hint.
|
||||
_strategy = (req.timing_strategy or "concise").lower()
|
||||
_dur_for_tts = seg_duration if _strategy == "strict_slot" else None
|
||||
|
||||
audio_tensor = await loop.run_in_executor(
|
||||
_gpu_pool, _gen,
|
||||
@@ -416,7 +396,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
target_samples = int(seg_duration * _model.sampling_rate)
|
||||
current_samples = audio_tensor.shape[-1]
|
||||
|
||||
if strategy == "strict_slot":
|
||||
if _strategy == "strict_slot":
|
||||
# Legacy: pad short audio + trim long audio so the mix
|
||||
# loop receives slot-sized buffers. The atempo squeeze
|
||||
# in the mix loop never fires here because we already
|
||||
@@ -426,10 +406,9 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
audio_tensor = torch.nn.functional.pad(audio_tensor, (0, pad_amount))
|
||||
elif current_samples > target_samples:
|
||||
audio_tensor = audio_tensor[..., :target_samples]
|
||||
# concise / stretch_video / smart_fit: keep audio at its
|
||||
# natural length. The mix loop decides per-mode whether to
|
||||
# trim, slip, stretch the video, or split audio/video
|
||||
# retiming (smart_fit) to accommodate it.
|
||||
# concise / stretch_video: keep audio at its natural length.
|
||||
# The mix loop decides per-mode whether to trim, slip, or
|
||||
# stretch the video to accommodate it.
|
||||
|
||||
generated_dur = audio_tensor.shape[-1] / _model.sampling_rate
|
||||
sync_ratio = round(generated_dur / max(seg_duration, 0.01), 3)
|
||||
@@ -508,6 +487,7 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
_t_diskw = time.perf_counter() - _t_diskw_0
|
||||
|
||||
sr = _model.sampling_rate
|
||||
strategy = (req.timing_strategy or "concise").lower()
|
||||
slot_fit = (req.slot_fit or "time_stretch").lower()
|
||||
overflow_budget_s = max(0.0, float(req.overflow_budget_s or 0.0))
|
||||
|
||||
@@ -522,7 +502,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
# the matching per-segment setpts filter chain on the source video.
|
||||
new_layout: list[tuple[float, float]] = []
|
||||
video_stretch_plan: list[dict] = []
|
||||
fit_plan = None # smart_fit only — services.fit_planner.FitPlan
|
||||
orig_total_dur = float(job.get("duration") or 0.0)
|
||||
|
||||
if strategy == "stretch_video":
|
||||
@@ -556,43 +535,9 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
cursor += max(0.0, orig_total_dur - last_orig_end)
|
||||
new_total_dur = max(cursor, orig_total_dur)
|
||||
total_samples = int(new_total_dur * sr)
|
||||
elif strategy == "smart_fit":
|
||||
# Smart Fit: plan the audio-rate / video-ratio split per segment
|
||||
# from the natural-rate WAV lengths. Pure planning — the mix
|
||||
# loop below applies the audio side; the video side ships as
|
||||
# fit_plans[lang] for the (Phase B) export pipeline.
|
||||
_fo = req.fit_options
|
||||
_fit_defaults = FitParams()
|
||||
fit_params = FitParams(
|
||||
max_audio_only_rate=float(getattr(_fo, "max_audio_only_rate", None) or _fit_defaults.max_audio_only_rate),
|
||||
audio_rate_cap=float(getattr(_fo, "audio_rate_cap", None) or _fit_defaults.audio_rate_cap),
|
||||
video_slow_cap=float(getattr(_fo, "video_slow_cap", None) or _fit_defaults.video_slow_cap),
|
||||
gap_guard_s=float(_fo.gap_guard_s) if _fo is not None and _fo.gap_guard_s is not None else _fit_defaults.gap_guard_s,
|
||||
allow_video_retime=bool(_fo.allow_video_retime) if _fo is not None and _fo.allow_video_retime is not None else _fit_defaults.allow_video_retime,
|
||||
)
|
||||
_seg_order = job.get("seg_order") or []
|
||||
fit_plan = plan_fit(
|
||||
[
|
||||
{
|
||||
"id": _seg_order[i] if i < len(_seg_order) else f"seg_{i}",
|
||||
"start": s,
|
||||
"end": e,
|
||||
}
|
||||
for i, (s, e, _w, _) in enumerate(all_segment_wavs)
|
||||
],
|
||||
[w.shape[-1] / sr for (_s, _e, w, _) in all_segment_wavs],
|
||||
orig_total_dur,
|
||||
fit_params,
|
||||
)
|
||||
total_samples = int(fit_plan.total_duration * sr)
|
||||
else:
|
||||
total_samples = int(orig_total_dur * sr)
|
||||
|
||||
# smart_fit: cue times for the fitted timeline, computed from the
|
||||
# ACTUAL stretched/trimmed sample positions in the mix loop below —
|
||||
# not from the plan — so subtitles land exactly on the audio.
|
||||
fitted_cues: list[dict] = []
|
||||
|
||||
full_audio = torch.zeros(1, total_samples)
|
||||
|
||||
for i, (start, end, wav, _) in enumerate(all_segment_wavs):
|
||||
@@ -615,55 +560,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
"stretch_ratio": round(natural_dur / max(orig_dur, 1e-3), 3),
|
||||
})
|
||||
|
||||
elif strategy == "smart_fit":
|
||||
# Smart Fit: apply the planner's audio_rate via the same
|
||||
# pitch-preserving atempo pipe strict_slot uses, place the
|
||||
# result at the planned new_start, and hard-trim whatever
|
||||
# the caps couldn't absorb. The video side (video_ratio per
|
||||
# chunk) is persisted below for the export pipeline.
|
||||
sf = fit_plan.segments[i]
|
||||
place_at = sf.new_start
|
||||
if sf.audio_rate > 1.0 + 1e-6 and wl > 0:
|
||||
target = max(1, int(round(wl / sf.audio_rate)))
|
||||
try:
|
||||
adjusted = await _pitch_preserving_stretch(
|
||||
adjusted, target, sr,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"atempo stretch failed for seg %d (%.2f×), "
|
||||
"falling back to linear interp: %s",
|
||||
i, sf.audio_rate, e,
|
||||
)
|
||||
adjusted = torch.nn.functional.interpolate(
|
||||
adjusted.unsqueeze(0),
|
||||
size=target,
|
||||
mode='linear',
|
||||
align_corners=False,
|
||||
).squeeze(0)
|
||||
wl = adjusted.shape[-1]
|
||||
# Residual overflow → hard-trim to the segment's new video
|
||||
# slot (fade below keeps the cut pop-free).
|
||||
new_slot_samples = int(max(0.0, sf.new_end - sf.new_start) * sr)
|
||||
if new_slot_samples > 0 and wl > new_slot_samples:
|
||||
adjusted = adjusted[..., :new_slot_samples]
|
||||
wl = adjusted.shape[-1]
|
||||
# Truthful per-segment verdict for the UI badge.
|
||||
entry = {"status": sf.status}
|
||||
if sf.audio_rate > 1.0 + 1e-6:
|
||||
entry["audio_rate"] = round(sf.audio_rate, 3)
|
||||
if sf.video_ratio > 1.0 + 1e-6:
|
||||
entry["video_ratio"] = round(sf.video_ratio, 3)
|
||||
if sf.overflow_s > 0:
|
||||
entry["overflow_s"] = round(sf.overflow_s, 3)
|
||||
fit_status.append(entry)
|
||||
# Cue times from the ACTUAL stretched sample positions.
|
||||
fitted_cues.append({
|
||||
"id": sf.seg_id,
|
||||
"start": round(place_at, 4),
|
||||
"end": round(place_at + wl / sr, 4),
|
||||
})
|
||||
|
||||
elif strategy == "concise":
|
||||
# Mode A: never compress. Allow the audio to extend into the
|
||||
# silent gap before the next seg (existing heuristic) plus
|
||||
@@ -785,9 +681,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
job["language"] = req.language
|
||||
job["language_code"] = lang_code
|
||||
job["timing_strategy"] = strategy
|
||||
# Keep job segments in lock-step with what was just rendered so
|
||||
# subtitle export / burn-in use the translated text (#309).
|
||||
_sync_job_segments(job, req)
|
||||
if strategy == "stretch_video":
|
||||
stretch_plans = job.setdefault("video_stretch_plans", {})
|
||||
stretch_plans[lang_code] = {
|
||||
@@ -795,34 +688,6 @@ async def dub_generate(job_id: str, req: DubRequest):
|
||||
"total_duration": round(track_dur, 4),
|
||||
"orig_duration": round(orig_total_dur, 4),
|
||||
}
|
||||
elif strategy == "smart_fit" and fit_plan is not None:
|
||||
# video_stretch_plans stays untouched — smart_fit persists its
|
||||
# own keyspace so a job can carry both without clobbering.
|
||||
_fit_params_payload = {
|
||||
"timing_strategy": strategy,
|
||||
"max_audio_only_rate": fit_params.max_audio_only_rate,
|
||||
"audio_rate_cap": fit_params.audio_rate_cap,
|
||||
"video_slow_cap": fit_params.video_slow_cap,
|
||||
"gap_guard_s": fit_params.gap_guard_s,
|
||||
"allow_video_retime": fit_params.allow_video_retime,
|
||||
}
|
||||
fit_fp = fit_fingerprint(_fit_params_payload)
|
||||
job.setdefault("fit_plans", {})[lang_code] = {
|
||||
# Same dict shape _build_video_stretch_filter_graph consumes.
|
||||
"plan": fit_plan.video_plan,
|
||||
# Cue times from actual stretched sample positions — for
|
||||
# subtitle export on the fitted timeline.
|
||||
"fitted_segments": fitted_cues,
|
||||
"total_duration": round(track_dur, 4),
|
||||
"orig_duration": round(orig_total_dur, 4),
|
||||
"params": _fit_params_payload,
|
||||
"fit_fp": fit_fp,
|
||||
}
|
||||
job["dubbed_tracks"][lang_code]["fit_fp"] = fit_fp
|
||||
# Record what kind of per-segment WAVs are on disk so a later
|
||||
# smart_fit run knows whether partial regen / fit-only re-mix can
|
||||
# reuse them ("natural") or must regen once ("slotted").
|
||||
job["seg_wav_kind"] = "slotted" if strategy == "strict_slot" else "natural"
|
||||
_save_job(job_id, job)
|
||||
|
||||
_t_total = time.perf_counter() - _t_start
|
||||
@@ -919,9 +784,6 @@ async def preview_segment(job_id: str, req: SegmentPreviewRequest):
|
||||
postprocess_output=True,
|
||||
)
|
||||
audio_out = audios[0]
|
||||
# TODO(#312): this route runs the OmniVoice model directly (not the active
|
||||
# backend), so VoxCPM2 never reaches it. When these routes become
|
||||
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
|
||||
mastered = apply_mastering(
|
||||
audio_out,
|
||||
sample_rate=getattr(_model, "sampling_rate", 24000),
|
||||
|
||||
@@ -2,7 +2,6 @@ import os
|
||||
import time
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Optional
|
||||
from fastapi import APIRouter
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
@@ -46,70 +45,6 @@ LANG_NAMES = {
|
||||
"id": "Indonesian", "uk": "Ukrainian",
|
||||
}
|
||||
|
||||
# Regional dialect hints (#280 item 2). Maps a BCP-47 dialect code to the
|
||||
# instruction injected into LLM translation prompts so the output uses that
|
||||
# region's vocabulary and grammar (the reporter's example: choosing Argentina
|
||||
# should yield "Vos sos muy listo", not the Peninsular "Tú eres muy listo").
|
||||
# Only LLM-backed paths can honor these — provider="openai" and the
|
||||
# quality="cinematic" refine pass. Keep entries short: they ride on every
|
||||
# per-segment prompt, so verbosity = wall time.
|
||||
DIALECT_HINTS = {
|
||||
# Spanish
|
||||
"es-ES": "European Spanish (Spain): use tú/vosotros forms and Peninsular vocabulary.",
|
||||
"es-MX": "Mexican Spanish: use tú/ustedes forms and Mexican vocabulary.",
|
||||
"es-AR": "Rioplatense Spanish (Argentina): use voseo — 'vos' with its verb forms (e.g. 'vos sos', 'tenés') and 'ustedes'; prefer Argentinian vocabulary.",
|
||||
"es-CO": "Colombian Spanish: use tú/usted as natural in Colombia and Colombian vocabulary.",
|
||||
"es-CL": "Chilean Spanish: use Chilean vocabulary and expressions.",
|
||||
# Portuguese
|
||||
"pt-BR": "Brazilian Portuguese: use 'você' forms, Brazilian vocabulary and spelling.",
|
||||
"pt-PT": "European Portuguese: use European vocabulary, spelling, and 'tu' where natural.",
|
||||
# English
|
||||
"en-US": "American English: use US spelling and vocabulary.",
|
||||
"en-GB": "British English: use UK spelling and vocabulary.",
|
||||
"en-AU": "Australian English: use Australian spelling and vocabulary.",
|
||||
"en-IN": "Indian English: use Indian English vocabulary and conventions.",
|
||||
# French
|
||||
"fr-FR": "Metropolitan French (France): use standard French vocabulary.",
|
||||
"fr-CA": "Canadian French (Québec): use Québécois vocabulary and expressions.",
|
||||
"fr-BE": "Belgian French: use Belgian vocabulary (e.g. septante, nonante).",
|
||||
# German
|
||||
"de-DE": "Standard German (Germany): use Federal German vocabulary.",
|
||||
"de-AT": "Austrian German: use Austrian vocabulary (e.g. Jänner, Erdapfel).",
|
||||
"de-CH": "Swiss Standard German: use Swiss vocabulary and 'ss' instead of 'ß'.",
|
||||
# Arabic
|
||||
"ar-EG": "Egyptian Arabic: use Egyptian colloquial vocabulary where natural for dubbing.",
|
||||
"ar-SA": "Gulf/Saudi Arabic flavor: prefer vocabulary natural to the Gulf region.",
|
||||
"ar-MA": "Moroccan Arabic (Darija) flavor: prefer vocabulary natural to Morocco.",
|
||||
# Dutch
|
||||
"nl-NL": "Netherlands Dutch: use vocabulary standard in the Netherlands.",
|
||||
"nl-BE": "Belgian Dutch (Flemish): use Flemish vocabulary and expressions.",
|
||||
}
|
||||
|
||||
|
||||
def dialect_clause(dialect: Optional[str]) -> str:
|
||||
"""Prompt fragment for a requested dialect, or '' when unset/unknown.
|
||||
|
||||
Unknown-but-plausible codes (e.g. "es-PE") still get a generic regional
|
||||
clause so users aren't limited to the curated list.
|
||||
"""
|
||||
if not dialect or not str(dialect).strip():
|
||||
return ""
|
||||
code = str(dialect).strip()
|
||||
hint = DIALECT_HINTS.get(code)
|
||||
if hint:
|
||||
return f" Target dialect — {hint}"
|
||||
# Generic fallback for any lang-REGION shaped code we don't curate.
|
||||
if "-" in code:
|
||||
lang, _, region = code.partition("-")
|
||||
lang_name = LANG_NAMES.get(lang, lang)
|
||||
if region:
|
||||
return (
|
||||
f" Use the vocabulary, grammar, and expressions of {lang_name} "
|
||||
f"as spoken in the region '{region}'."
|
||||
)
|
||||
return ""
|
||||
|
||||
|
||||
# Per-language script enforcement. Maps language code → required Unicode
|
||||
# block(s) the translation must contain. Used as a sanity gate after the
|
||||
# LLM responds: if the output contains <50% characters from the expected
|
||||
@@ -157,49 +92,15 @@ _nllb_tokenizer = None
|
||||
_nllb_device = None
|
||||
|
||||
|
||||
def _dialect_flags(req, applied: bool) -> dict:
|
||||
"""Response fields describing whether the requested dialect was honored.
|
||||
|
||||
Empty dict when no dialect was requested, so existing response shapes
|
||||
stay byte-identical for callers that never send one.
|
||||
"""
|
||||
if not getattr(req, "dialect", None):
|
||||
return {}
|
||||
return {"dialect": req.dialect, "dialect_applied": bool(applied)}
|
||||
|
||||
|
||||
def _guess_lang_from_text(segments) -> str | None:
|
||||
"""Best-effort source language from segment text, by script.
|
||||
|
||||
Used only as a last resort when neither the request nor the job carries a
|
||||
detected language. Without this, the bare "en" fallback below forces
|
||||
en -> en on non-English audio (e.g. Korean), which has no Argos package and
|
||||
fails every segment even though ASR detected the language correctly.
|
||||
"""
|
||||
text = " ".join((getattr(s, "text", "") or "") for s in (segments or [])[:8])
|
||||
has = lambda lo, hi: any(lo <= ord(c) <= hi for c in text)
|
||||
if has(0x3040, 0x30FF):
|
||||
return "ja" # Hiragana/Katakana — check before CJK (Japanese uses Kanji too)
|
||||
if has(0xAC00, 0xD7A3) or has(0x1100, 0x11FF):
|
||||
return "ko" # Hangul
|
||||
if has(0x4E00, 0x9FFF):
|
||||
return "zh" # CJK ideographs
|
||||
if has(0x0400, 0x04FF):
|
||||
return "ru" # Cyrillic
|
||||
if has(0x0600, 0x06FF):
|
||||
return "ar" # Arabic
|
||||
return None
|
||||
|
||||
|
||||
def _resolve_source_lang(req: TranslateRequest) -> str:
|
||||
"""Pick source language: explicit request > job.source_lang > text guess > 'en'."""
|
||||
"""Pick source language: explicit request > job.source_lang > 'en' fallback."""
|
||||
if getattr(req, "source_lang", None):
|
||||
return req.source_lang
|
||||
if getattr(req, "job_id", None):
|
||||
job = _get_job(req.job_id)
|
||||
if job and job.get("source_lang"):
|
||||
return job["source_lang"]
|
||||
return _guess_lang_from_text(getattr(req, "segments", None)) or "en"
|
||||
return "en"
|
||||
|
||||
|
||||
def _unload_nllb():
|
||||
@@ -302,8 +203,7 @@ async def dub_translate(req: TranslateRequest):
|
||||
translated = await loop.run_in_executor(_gpu_pool, _translate_nllb)
|
||||
if os.environ.get("OMNIVOICE_UNLOAD_NLLB", "1") == "1":
|
||||
_unload_nllb()
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
**_dialect_flags(req, applied=False)}
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
|
||||
|
||||
# OpenAI / Ollama Local LLM Translation
|
||||
if provider == "openai":
|
||||
@@ -336,16 +236,10 @@ async def dub_translate(req: TranslateRequest):
|
||||
f"only — do not use Latin/Roman letters, do not "
|
||||
f"transliterate, do not output any other language."
|
||||
)
|
||||
# #280 item 2 — regional dialect/vocabulary. Only applied when
|
||||
# the dialect belongs to the target language (a leftover
|
||||
# "es-AR" must not contaminate a French translation).
|
||||
dia_clause = ""
|
||||
if req.dialect and str(req.dialect).lower().startswith(str(tgt_code).lower()[:2]):
|
||||
dia_clause = dialect_clause(req.dialect)
|
||||
return (
|
||||
f"You are a professional dubbing translator. "
|
||||
f"Translate the user's text from {src_name} into "
|
||||
f"{tgt_name}.{script_clause}{dia_clause} "
|
||||
f"{tgt_name}.{script_clause} "
|
||||
f"Reply ONLY with the translated {tgt_name} text, do not "
|
||||
f"add quotes, notes, headers, explanations, or commentary."
|
||||
)
|
||||
@@ -405,8 +299,7 @@ async def dub_translate(req: TranslateRequest):
|
||||
tasks = [loop.run_in_executor(_cpu_pool, _translate_llm, seg) for seg in req.segments]
|
||||
translated = await asyncio.gather(*tasks)
|
||||
translated.sort(key=lambda x: str(x["id"]))
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
**_dialect_flags(req, applied=True)}
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
|
||||
|
||||
# Offline Argos Translate
|
||||
if provider == "argos" or provider == "libretranslate":
|
||||
@@ -460,8 +353,7 @@ async def dub_translate(req: TranslateRequest):
|
||||
return results
|
||||
|
||||
translated = await loop.run_in_executor(_cpu_pool, _translate_argos)
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
**_dialect_flags(req, applied=False)}
|
||||
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
|
||||
|
||||
# Legacy / API Deep_Translator logic.
|
||||
# Preflight the optional `deep_translator` dep once so we fail with a
|
||||
@@ -570,8 +462,7 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
except Exception as e:
|
||||
logger.debug("non-LLM rate_ratio prediction skipped: %s", e)
|
||||
|
||||
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
|
||||
"quality_used": "fast", **_dialect_flags(req, applied=False)}
|
||||
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang, "quality_used": "fast"}
|
||||
|
||||
if quality != "cinematic":
|
||||
return base
|
||||
@@ -601,19 +492,12 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
if not pairs:
|
||||
return base
|
||||
|
||||
# #280 item 2: thread the regional-dialect hint into the reflect/adapt
|
||||
# prompts. Guard against a stale dialect from another language.
|
||||
dialect_hint = ""
|
||||
if req.dialect and str(req.dialect).lower().startswith(str(req.target_lang).lower()[:2]):
|
||||
dialect_hint = dialect_clause(req.dialect)
|
||||
|
||||
refined = await cinematic_refine_many(
|
||||
pairs,
|
||||
source_lang=src_lang,
|
||||
target_lang=req.target_lang,
|
||||
glossary=req.glossary,
|
||||
directions=directions,
|
||||
dialect_hint=dialect_hint,
|
||||
executor=_cpu_pool,
|
||||
)
|
||||
refined_by_id = {r["id"]: r for r in refined}
|
||||
@@ -676,5 +560,4 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
|
||||
"target_lang": req.target_lang,
|
||||
"source_lang": src_lang,
|
||||
"quality_used": "cinematic",
|
||||
**_dialect_flags(req, applied=bool(dialect_hint)),
|
||||
}
|
||||
|
||||
@@ -266,26 +266,11 @@ class SelectEngineRequest(BaseModel):
|
||||
backend_id: str
|
||||
|
||||
|
||||
class SelectEngineResponse(BaseModel):
|
||||
family: str
|
||||
active: str
|
||||
env_override: bool
|
||||
# Routing verdict for the selected engine on THIS host (#21). Always present
|
||||
# so the UI can show a confirm/warning toast on a cpu_fallback pick without
|
||||
# branching on key presence; defaults match a legacy/degraded row.
|
||||
routing_status: str = "cpu_only"
|
||||
effective_device: str = "cpu"
|
||||
routing_reason: str | None = None
|
||||
|
||||
|
||||
@router.post("/engines/select", response_model=SelectEngineResponse)
|
||||
@router.post("/engines/select")
|
||||
def select_engine(req: SelectEngineRequest):
|
||||
"""Persist a family's engine pick to prefs.json. Refuses unknown backends,
|
||||
backends whose deps aren't installed, AND backends that cannot run on THIS
|
||||
host's hardware (routing_status == "unavailable") — so the UI can't silently
|
||||
brick a pipeline by picking an engine that needs a GPU this machine lacks.
|
||||
A `cpu_fallback` pick is allowed (it runs, just slower) — only a hard
|
||||
`unavailable` is blocked. LLM is never routing-gated (its status is "n/a")."""
|
||||
"""Persist a family's engine pick to prefs.json. Refuses unknown backends
|
||||
+ refuses backends whose deps aren't installed (so the UI can't silently
|
||||
brick a pipeline by picking an unavailable engine)."""
|
||||
family = _FAMILIES.get(req.family)
|
||||
if not family:
|
||||
raise HTTPException(400, f"Unknown family: {req.family}. Expected one of tts/asr/llm.")
|
||||
@@ -293,26 +278,12 @@ def select_engine(req: SelectEngineRequest):
|
||||
available = {b["id"]: b for b in module.list_backends()}
|
||||
if req.backend_id not in available:
|
||||
raise HTTPException(400, f"Unknown {req.family} backend: {req.backend_id!r}")
|
||||
entry = available[req.backend_id]
|
||||
if not entry["available"]:
|
||||
reason = entry.get("reason") or "unavailable"
|
||||
if not available[req.backend_id]["available"]:
|
||||
reason = available[req.backend_id].get("reason") or "unavailable"
|
||||
raise HTTPException(400, f"Backend {req.backend_id} not ready: {reason}")
|
||||
# Host-routing gate (no silent CPU fallback). `.get` is defensive so an
|
||||
# older/legacy payload without routing keys still selects cleanly.
|
||||
if entry.get("routing_status") == "unavailable":
|
||||
why = entry.get("routing_reason") or "requires a GPU this host doesn't have"
|
||||
raise HTTPException(
|
||||
400,
|
||||
f"Backend {req.backend_id} can't run on this machine: {why}. "
|
||||
f"Pick an engine with a CPU path, or one that supports this host's GPU.",
|
||||
)
|
||||
prefs.set_(pref_key, req.backend_id)
|
||||
return {
|
||||
"family": req.family,
|
||||
"active": module.active_backend_id(),
|
||||
"env_override": bool(__import__("os").environ.get(f"OMNIVOICE_{req.family.upper()}_BACKEND")),
|
||||
# Echo the routing verdict so the UI can warn on a cpu_fallback pick.
|
||||
"routing_status": entry.get("routing_status", "cpu_only"),
|
||||
"effective_device": entry.get("effective_device", "cpu"),
|
||||
"routing_reason": entry.get("routing_reason"),
|
||||
}
|
||||
|
||||
@@ -59,87 +59,13 @@ def _render_with_pauses(gen_span, segments, sample_rate):
|
||||
parts.append(torch.zeros(*shape, dtype=ref.dtype, device=ref.device))
|
||||
return torch.cat(parts, dim=-1)
|
||||
|
||||
|
||||
def _apply_effect_chain(audio_out, sample_rate, effect_preset, *, skip_mastering=False):
|
||||
"""Shared post-DSP for /generate: preset validation → mastering →
|
||||
effect chain → loudness normalization.
|
||||
|
||||
``skip_mastering`` honors a backend's ``applies_own_mastering`` flag
|
||||
(issue #312): studio engines (e.g. VoxCPM2's native 48 kHz output)
|
||||
opt out of the broadcast Compressor + Reverb chain that's tuned for
|
||||
OmniVoice's 24 kHz clone output. Loudness normalization still runs —
|
||||
it's a benign peak scale. Mirrors ``_run_tts`` in openai_compat.py.
|
||||
"""
|
||||
from services.audio_dsp import (
|
||||
EFFECT_PRESETS, apply_mastering, normalize_audio,
|
||||
apply_effects_chain, get_effect_chain,
|
||||
)
|
||||
|
||||
preset = effect_preset or "broadcast"
|
||||
if preset not in EFFECT_PRESETS:
|
||||
raise ValueError(
|
||||
f"Unknown effect preset: {preset!r}. "
|
||||
f"Valid: {list(EFFECT_PRESETS.keys())}"
|
||||
)
|
||||
|
||||
if preset == "raw":
|
||||
# Raw: skip all DSP — return raw model output
|
||||
return audio_out
|
||||
|
||||
if not skip_mastering:
|
||||
audio_out = apply_mastering(audio_out, sample_rate=sample_rate)
|
||||
chain = get_effect_chain(preset)
|
||||
if chain:
|
||||
audio_out = apply_effects_chain(
|
||||
audio_out, sample_rate=sample_rate, chain=chain,
|
||||
)
|
||||
return normalize_audio(audio_out, target_dBFS=-2.0)
|
||||
|
||||
|
||||
def _oom_friendly_reraise(e):
|
||||
"""Best-effort cache flush + the user-facing OOM hint shared by both
|
||||
inference paths."""
|
||||
import gc
|
||||
import torch
|
||||
gc.collect()
|
||||
if torch.backends.mps.is_available():
|
||||
torch.mps.empty_cache()
|
||||
elif torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
# #278: don't mislabel a torch.compile/Triton/Inductor crash as an
|
||||
# out-of-memory condition. (model_manager's generate wrapper already
|
||||
# retries these eagerly; this only triggers if that retry also died.)
|
||||
from services.model_manager import _is_compile_runtime_failure
|
||||
if _is_compile_runtime_failure(e):
|
||||
raise RuntimeError(
|
||||
f"TTS engine hit a torch.compile/Triton error (not out of memory). "
|
||||
f"Disable torch.compile in Settings → Performance, use the Flush "
|
||||
f"button to reload the model, then regenerate. Underlying error: {e}"
|
||||
) from e
|
||||
# #437: a Permission-denied / exec failure (e.g. a bundled engine binary
|
||||
# that lost its +x bit) is NOT an OOM — don't send the user to the Flush
|
||||
# button; tell them what's actually wrong.
|
||||
es = str(e)
|
||||
if isinstance(e, PermissionError) or "Permission denied" in es or "Errno 13" in es:
|
||||
raise RuntimeError(
|
||||
f"A required engine binary couldn't be executed (permission denied). "
|
||||
f"This usually means a bundled binary lost its execute bit — reinstall, "
|
||||
f"or run `chmod +x` on the engine binary named in the error. "
|
||||
f"Underlying error: {e}"
|
||||
) from e
|
||||
raise RuntimeError(
|
||||
f"TTS engine stopped mid-generation. This usually means it ran out of memory. "
|
||||
f"Try the Flush button to reload the model, then regenerate. Underlying error: {e}"
|
||||
)
|
||||
|
||||
|
||||
def _run_inference(
|
||||
model, text, language, ref_audio_path, ref_text, instruct, duration,
|
||||
num_step, guidance_scale, speed, t_shift, denoise,
|
||||
postprocess_output, layer_penalty_factor, position_temperature,
|
||||
class_temperature, used_seed, effect_preset="broadcast",
|
||||
max_chunk_chars=None, crossfade_ms=None,
|
||||
):
|
||||
from services.audio_dsp import apply_mastering, normalize_audio, apply_effects_chain, get_effect_chain
|
||||
import torch
|
||||
try:
|
||||
if used_seed is not None:
|
||||
@@ -173,124 +99,53 @@ def _run_inference(
|
||||
)[0]
|
||||
audio_out = _render_with_pauses(_gen_span, segments, sr)
|
||||
else:
|
||||
# Wave 1.2: long text is split at sentence boundaries and the
|
||||
# per-chunk audio crossfaded — removes the length ceiling. Short
|
||||
# text takes the single-shot path below unchanged. [pause] inputs
|
||||
# keep the dedicated stitcher above (spans are already short).
|
||||
from services.chunked_tts import (
|
||||
DEFAULT_CROSSFADE_MS, DEFAULT_MAX_CHUNK_CHARS,
|
||||
concatenate_audio_chunks, split_text_into_chunks,
|
||||
audios = model.generate(
|
||||
text=text, language=language, ref_audio=ref_audio_path,
|
||||
ref_text=ref_text, instruct=instruct, duration=duration,
|
||||
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
|
||||
denoise=denoise, postprocess_output=postprocess_output,
|
||||
**kwargs
|
||||
)
|
||||
_max_chars = DEFAULT_MAX_CHUNK_CHARS if max_chunk_chars is None else max_chunk_chars
|
||||
_xfade_ms = DEFAULT_CROSSFADE_MS if crossfade_ms is None else crossfade_ms
|
||||
text_chunks = split_text_into_chunks(text, _max_chars)
|
||||
if len(text_chunks) > 1:
|
||||
parts = []
|
||||
for i, chunk_text in enumerate(text_chunks):
|
||||
# Vary the seed per chunk (deterministically) to avoid
|
||||
# correlated RNG artifacts across chunk boundaries.
|
||||
if used_seed is not None:
|
||||
torch.manual_seed(used_seed + i)
|
||||
parts.append(model.generate(
|
||||
text=chunk_text, language=language, ref_audio=ref_audio_path,
|
||||
ref_text=ref_text, instruct=instruct, duration=None,
|
||||
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
|
||||
denoise=denoise, postprocess_output=postprocess_output,
|
||||
**kwargs
|
||||
)[0])
|
||||
audio_out = concatenate_audio_chunks(parts, sr, _xfade_ms)
|
||||
else:
|
||||
audios = model.generate(
|
||||
text=text, language=language, ref_audio=ref_audio_path,
|
||||
ref_text=ref_text, instruct=instruct, duration=duration,
|
||||
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
|
||||
denoise=denoise, postprocess_output=postprocess_output,
|
||||
**kwargs
|
||||
)
|
||||
audio_out = audios[0]
|
||||
audio_out = audios[0]
|
||||
|
||||
# Apply DSP effect preset. The OmniVoice model never masters its own
|
||||
# output, so mastering always runs here (unchanged behavior).
|
||||
return _apply_effect_chain(audio_out, sr, effect_preset)
|
||||
# Apply DSP effect preset
|
||||
_effect_preset = effect_preset or "broadcast"
|
||||
|
||||
# Validate preset ID
|
||||
from services.audio_dsp import EFFECT_PRESETS
|
||||
if _effect_preset not in EFFECT_PRESETS:
|
||||
raise ValueError(
|
||||
f"Unknown effect preset: {_effect_preset!r}. "
|
||||
f"Valid: {list(EFFECT_PRESETS.keys())}"
|
||||
)
|
||||
|
||||
if _effect_preset == "raw":
|
||||
# Raw: skip all DSP — return raw model output
|
||||
return audio_out
|
||||
|
||||
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
|
||||
_chain = get_effect_chain(_effect_preset)
|
||||
if _chain:
|
||||
mastered_audio = apply_effects_chain(
|
||||
mastered_audio, sample_rate=sr, chain=_chain,
|
||||
)
|
||||
|
||||
return normalize_audio(mastered_audio, target_dBFS=-2.0)
|
||||
|
||||
except ValueError as e:
|
||||
# Don't wrap validation errors in OOM message
|
||||
raise e
|
||||
except Exception as e:
|
||||
_oom_friendly_reraise(e)
|
||||
|
||||
|
||||
def _run_backend_inference(
|
||||
backend, text, language, ref_audio_path, ref_text, instruct, duration,
|
||||
num_step, guidance_scale, speed, denoise, postprocess_output,
|
||||
used_seed, effect_preset="broadcast",
|
||||
max_chunk_chars=None, crossfade_ms=None,
|
||||
):
|
||||
"""Engine-aware twin of :func:`_run_inference` (issue #312).
|
||||
|
||||
Runs the request through a pluggable ``TTSBackend`` adapter instead of the
|
||||
OmniVoice model directly. The adapter protocol is narrower than the
|
||||
OmniVoice-native surface — engine-specific extras (``t_shift``,
|
||||
``layer_penalty_factor``, …) only exist on the native path, which is why
|
||||
OmniVoice itself still goes through ``_run_inference``.
|
||||
"""
|
||||
import torch
|
||||
try:
|
||||
if used_seed is not None:
|
||||
torch.manual_seed(used_seed)
|
||||
|
||||
if language and language.lower() == "auto":
|
||||
language = None
|
||||
|
||||
gen_kwargs = dict(
|
||||
language=language, ref_audio=ref_audio_path, ref_text=ref_text,
|
||||
instruct=instruct, num_step=num_step, guidance_scale=guidance_scale,
|
||||
speed=speed, denoise=denoise, postprocess_output=postprocess_output,
|
||||
import gc
|
||||
gc.collect()
|
||||
if torch.backends.mps.is_available():
|
||||
torch.mps.empty_cache()
|
||||
elif torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
raise RuntimeError(
|
||||
f"TTS engine stopped mid-generation. This usually means it ran out of memory. "
|
||||
f"Try the Flush button to reload the model, then regenerate. Underlying error: {e}"
|
||||
)
|
||||
sr = backend.sample_rate
|
||||
|
||||
# Inline [pause Nms] markers (issue #276) work for every engine — the
|
||||
# silence stitching is model-free.
|
||||
from omnivoice.utils.text import parse_pause_markers
|
||||
segments = parse_pause_markers(text)
|
||||
has_pause = len(segments) > 1 or (segments and segments[0][1] > 0)
|
||||
|
||||
if has_pause:
|
||||
def _gen_span(span_text):
|
||||
# Per-span duration is left to the engine; an explicit overall
|
||||
# `duration` can't be meaningfully split across spans.
|
||||
return backend.generate(span_text, duration=None, **gen_kwargs)
|
||||
audio_out = _render_with_pauses(_gen_span, segments, sr)
|
||||
else:
|
||||
# Wave 1.2: sentence-boundary chunking for long text (see
|
||||
# _run_inference for the rationale; behavior is identical here).
|
||||
from services.chunked_tts import (
|
||||
DEFAULT_CROSSFADE_MS, DEFAULT_MAX_CHUNK_CHARS,
|
||||
concatenate_audio_chunks, split_text_into_chunks,
|
||||
)
|
||||
_max_chars = DEFAULT_MAX_CHUNK_CHARS if max_chunk_chars is None else max_chunk_chars
|
||||
_xfade_ms = DEFAULT_CROSSFADE_MS if crossfade_ms is None else crossfade_ms
|
||||
text_chunks = split_text_into_chunks(text, _max_chars)
|
||||
if len(text_chunks) > 1:
|
||||
parts = []
|
||||
for i, chunk_text in enumerate(text_chunks):
|
||||
if used_seed is not None:
|
||||
torch.manual_seed(used_seed + i)
|
||||
parts.append(backend.generate(chunk_text, duration=None, **gen_kwargs))
|
||||
audio_out = concatenate_audio_chunks(parts, sr, _xfade_ms)
|
||||
else:
|
||||
audio_out = backend.generate(text, duration=duration, **gen_kwargs)
|
||||
|
||||
return _apply_effect_chain(
|
||||
audio_out, sr, effect_preset,
|
||||
skip_mastering=getattr(backend, "applies_own_mastering", False),
|
||||
)
|
||||
|
||||
except ValueError as e:
|
||||
# Don't wrap validation errors in OOM message
|
||||
raise e
|
||||
except Exception as e:
|
||||
_oom_friendly_reraise(e)
|
||||
|
||||
|
||||
@router.post("/generate")
|
||||
@@ -313,88 +168,19 @@ async def generate_speech(
|
||||
profile_id: Optional[str] = Form(None),
|
||||
seed: Optional[int] = Form(None),
|
||||
effect_preset: str = Form("broadcast"),
|
||||
engine: Optional[str] = Form(None),
|
||||
# Wave 1.2 — unlimited-length generation: long text is split at sentence
|
||||
# boundaries and crossfaded. 0 disables chunking (whole text to engine).
|
||||
max_chunk_chars: int = Form(800, ge=0),
|
||||
crossfade_ms: int = Form(50, ge=0, le=1000),
|
||||
):
|
||||
# ── Engine resolution (issue #312) ──────────────────────────────────────
|
||||
# The request runs on the engine selected in Settings (POST /engines/select,
|
||||
# env var OMNIVOICE_TTS_BACKEND wins), or an explicit per-request `engine`
|
||||
# override — same pattern as /ws/tts's `engine` field and /v1/audio/speech's
|
||||
# `model`. Omitting both keeps the historical default (OmniVoice), so
|
||||
# existing API consumers see no change.
|
||||
from services.tts_backend import (
|
||||
OmniVoiceBackend, _mask_hf_tokens, active_backend_id, get_backend_class,
|
||||
)
|
||||
|
||||
engine_id = engine or active_backend_id()
|
||||
try:
|
||||
backend_cls = get_backend_class(engine_id)
|
||||
except ValueError:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=(
|
||||
f"Unknown TTS engine: {engine_id!r}. "
|
||||
"See GET /engines/tts for the list of valid engine ids."
|
||||
),
|
||||
)
|
||||
|
||||
_model = None
|
||||
_backend = None
|
||||
if backend_cls is OmniVoiceBackend:
|
||||
# OmniVoice keeps its native path: it carries the full advanced
|
||||
# parameter surface (t_shift, layer/position/class controls) that the
|
||||
# generic adapter protocol doesn't. Byte-identical to the old behavior.
|
||||
_model = await get_model()
|
||||
else:
|
||||
try:
|
||||
ok, msg = backend_cls.is_available()
|
||||
except Exception as exc:
|
||||
ok, msg = False, f"{type(exc).__name__}: {exc}"
|
||||
if not ok:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"TTS engine '{engine_id}' is not available: {_mask_hf_tokens(msg)}",
|
||||
)
|
||||
# Reuse the per-process instance cache shared with the engine
|
||||
# health-check route so weights load once, not per request.
|
||||
from api.routers.engines import _get_engine_instance
|
||||
_backend = _get_engine_instance(backend_cls)
|
||||
|
||||
# ── Routing gate (#21 — no silent CPU fallback). Computed ONCE per request
|
||||
# (host caps are constant; the per-request engine= override bypasses the
|
||||
# /engines/select gate, so this is the only place it's enforced for synth).
|
||||
from core.device_caps import detect_host_caps
|
||||
from services.engine_routing import resolve_routing, routing_notice
|
||||
_routing = resolve_routing(getattr(backend_cls, "gpu_compat", ("cpu",)), detect_host_caps())
|
||||
if _routing["routing_status"] == "unavailable":
|
||||
# The engine needs an accelerator this host lacks and has no CPU path.
|
||||
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
|
||||
_routing_notice = routing_notice(_routing) # (status, reason) or None
|
||||
_model = await get_model()
|
||||
|
||||
ref_audio_path = None
|
||||
cleanup_ref = False
|
||||
used_seed = seed
|
||||
resolved_profile_id = None
|
||||
history_mode = None # profile.kind when a profile drives; else inferred at insert
|
||||
|
||||
if profile_id:
|
||||
with db_conn() as conn:
|
||||
row = conn.execute("SELECT * FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
if row:
|
||||
resolved_profile_id = profile_id
|
||||
# `kind` is authoritative (0005): 'design' profiles condition on
|
||||
# their deterministic rendered sample + instruct; 'clone' on the
|
||||
# user's reference. Lock always wins (it pins a specific take).
|
||||
# Rows from pre-0004 DBs mid-upgrade may lack the column → fall
|
||||
# back to the legacy is_locked/instruct inference.
|
||||
try:
|
||||
profile_kind = row["kind"] or "clone"
|
||||
except (KeyError, IndexError):
|
||||
profile_kind = "design" if (row["instruct"] and not row["is_locked"] and not row["ref_audio_path"]) else "clone"
|
||||
history_mode = profile_kind
|
||||
if row["is_locked"] and row["locked_audio_path"]:
|
||||
ref_audio_path = os.path.join(VOICES_DIR, row["locked_audio_path"])
|
||||
if not ref_text:
|
||||
@@ -403,19 +189,7 @@ async def generate_speech(
|
||||
instruct = row["instruct"]
|
||||
if used_seed is None and row["seed"] is not None:
|
||||
used_seed = row["seed"]
|
||||
elif profile_kind == "design":
|
||||
# Rendered sample (if present) carries the voice identity;
|
||||
# instruct alone is the fallback for legacy archetype rows.
|
||||
ref_audio_path = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
|
||||
if ref_audio_path and not ref_text and row["ref_text"]:
|
||||
ref_text = row["ref_text"]
|
||||
if not instruct:
|
||||
instruct = row["instruct"]
|
||||
if used_seed is None and row["seed"] is not None:
|
||||
used_seed = row["seed"]
|
||||
elif row["instruct"] and not row["is_locked"] and not row["ref_audio_path"]:
|
||||
# Legacy design-shaped row (pre-0004 archetype materialization
|
||||
# failure path): instruct-only conditioning.
|
||||
elif row["instruct"] and not row["is_locked"]:
|
||||
if not instruct:
|
||||
instruct = row["instruct"]
|
||||
if used_seed is None and row["seed"] is not None:
|
||||
@@ -439,41 +213,16 @@ async def generate_speech(
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
# #308: a transcript-less reference is transcribed with the active ASR
|
||||
# backend (whisperx / faster-whisper / mlx-whisper) instead of the model's
|
||||
# built-in transformers pipeline, which cannot load whisper-large-v3-turbo
|
||||
# on transformers 5.3. On failure ref_text stays None and the model's
|
||||
# fallback behaves exactly as before.
|
||||
if ref_audio_path and not ref_text:
|
||||
from services.asr_backend import transcribe_reference
|
||||
ref_text = await asyncio.get_running_loop().run_in_executor(
|
||||
_gpu_pool, transcribe_reference, ref_audio_path
|
||||
)
|
||||
|
||||
start_time = time.time()
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
if _backend is not None:
|
||||
audio_tensor = await loop.run_in_executor(
|
||||
_gpu_pool, _run_backend_inference,
|
||||
_backend, text, language, ref_audio_path, ref_text, instruct,
|
||||
duration, num_step, guidance_scale, speed, denoise,
|
||||
postprocess_output, used_seed, effect_preset,
|
||||
max_chunk_chars, crossfade_ms,
|
||||
)
|
||||
# Read after generation: engines with lazy model loading report
|
||||
# their real rate only once weights are up.
|
||||
sample_rate = _backend.sample_rate
|
||||
else:
|
||||
audio_tensor = await loop.run_in_executor(
|
||||
_gpu_pool, _run_inference,
|
||||
_model, text, language, ref_audio_path, ref_text, instruct, duration,
|
||||
num_step, guidance_scale, speed, t_shift, denoise,
|
||||
postprocess_output, layer_penalty_factor, position_temperature,
|
||||
class_temperature, used_seed, effect_preset,
|
||||
max_chunk_chars, crossfade_ms,
|
||||
)
|
||||
sample_rate = _model.sampling_rate
|
||||
audio_tensor = await loop.run_in_executor(
|
||||
_gpu_pool, _run_inference,
|
||||
_model, text, language, ref_audio_path, ref_text, instruct, duration,
|
||||
num_step, guidance_scale, speed, t_shift, denoise,
|
||||
postprocess_output, layer_penalty_factor, position_temperature,
|
||||
class_temperature, used_seed, effect_preset,
|
||||
)
|
||||
# Invisible AudioSeal provenance watermark on the final audio. Embedding
|
||||
# was previously only wired into the dub pipeline (dub_generate.py), so
|
||||
# plain TTS came out unmarked despite the setting being on. embed_watermark
|
||||
@@ -482,28 +231,28 @@ async def generate_speech(
|
||||
# generation.
|
||||
from services.watermark import embed_watermark
|
||||
audio_tensor = await loop.run_in_executor(
|
||||
_gpu_pool, embed_watermark, audio_tensor, sample_rate
|
||||
_gpu_pool, embed_watermark, audio_tensor, _model.sampling_rate
|
||||
)
|
||||
gen_time = round(time.time() - start_time, 2)
|
||||
|
||||
audio_id = str(uuid.uuid4())[:8]
|
||||
audio_filename = f"{audio_id}.wav"
|
||||
audio_path = os.path.join(OUTPUTS_DIR, audio_filename)
|
||||
_safe_torchaudio_save(audio_path, audio_tensor, sample_rate)
|
||||
_safe_torchaudio_save(audio_path, audio_tensor, _model.sampling_rate)
|
||||
|
||||
audio_dur = round(audio_tensor.shape[-1] / sample_rate, 2)
|
||||
audio_dur = round(audio_tensor.shape[-1] / _model.sampling_rate, 2)
|
||||
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO generation_history (id, text, mode, language, instruct, profile_id, audio_path, duration_seconds, generation_time, seed, created_at) VALUES (?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(audio_id, text[:200], history_mode or ("clone" if ref_audio_path else "design"),
|
||||
(audio_id, text[:200], "clone" if ref_audio_path else "design",
|
||||
language or "Auto", instruct or "", resolved_profile_id,
|
||||
audio_filename, audio_dur, gen_time, used_seed, time.time())
|
||||
)
|
||||
event_bus.emit("generation_history", {"action": "created", "id": audio_id})
|
||||
|
||||
buffer = io.BytesIO()
|
||||
_safe_torchaudio_save(buffer, audio_tensor, sample_rate, format="wav")
|
||||
_safe_torchaudio_save(buffer, audio_tensor, _model.sampling_rate, format="wav")
|
||||
buffer.seek(0)
|
||||
wav_bytes = buffer.read()
|
||||
|
||||
@@ -512,26 +261,17 @@ async def generate_speech(
|
||||
for i in range(0, len(wav_bytes), chunk_size):
|
||||
yield wav_bytes[i:i + chunk_size]
|
||||
|
||||
_resp_headers = {
|
||||
"X-Audio-Id": audio_id,
|
||||
"X-Gen-Time": str(gen_time),
|
||||
"X-Audio-Path": audio_filename,
|
||||
"X-Seed": str(used_seed) if used_seed is not None else "",
|
||||
"X-Audio-Duration": str(audio_dur),
|
||||
"Content-Length": str(len(wav_bytes)),
|
||||
}
|
||||
# Routing notice (#21): cpu_fallback or accelerated-with-caveat only;
|
||||
# the WAV body is binary so the header channel is the carrier.
|
||||
if _routing_notice:
|
||||
from services.engine_routing import header_safe_reason
|
||||
_resp_headers["X-OmniVoice-Routing"] = _routing_notice[0]
|
||||
_hr = header_safe_reason(_routing_notice[1])
|
||||
if _hr:
|
||||
_resp_headers["X-OmniVoice-Routing-Reason"] = _hr
|
||||
return StreamingResponse(
|
||||
_stream_wav(),
|
||||
media_type="audio/wav",
|
||||
headers=_resp_headers,
|
||||
headers={
|
||||
"X-Audio-Id": audio_id,
|
||||
"X-Gen-Time": str(gen_time),
|
||||
"X-Audio-Path": audio_filename,
|
||||
"X-Seed": str(used_seed) if used_seed is not None else "",
|
||||
"X-Audio-Duration": str(audio_dur),
|
||||
"Content-Length": str(len(wav_bytes)),
|
||||
}
|
||||
)
|
||||
except HTTPException:
|
||||
raise
|
||||
@@ -568,29 +308,9 @@ def _safe_output_path(name):
|
||||
|
||||
@router.get("/history")
|
||||
def list_history():
|
||||
"""Newest 50 generations whose audio still exists on disk.
|
||||
|
||||
Rows whose WAV was deleted out-of-band (cleared outputs dir, manual
|
||||
cleanup) used to come back anyway and render dead players that 404 on
|
||||
every fetch; prune them here so the UI never sees them again."""
|
||||
with db_conn() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM generation_history ORDER BY created_at DESC LIMIT 50"
|
||||
).fetchall()
|
||||
alive, stale_ids = [], []
|
||||
for r in rows:
|
||||
p = _safe_output_path(r["audio_path"]) if r["audio_path"] else None
|
||||
if r["audio_path"] and (not p or not os.path.exists(p)):
|
||||
stale_ids.append(r["id"])
|
||||
else:
|
||||
alive.append(dict(r))
|
||||
if stale_ids:
|
||||
conn.executemany(
|
||||
"DELETE FROM generation_history WHERE id=?",
|
||||
[(i,) for i in stale_ids],
|
||||
)
|
||||
logger.info("pruned %d stale history rows (audio file gone)", len(stale_ids))
|
||||
return alive
|
||||
rows = conn.execute("SELECT * FROM generation_history ORDER BY created_at DESC LIMIT 50").fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
@router.delete("/history")
|
||||
def clear_history():
|
||||
|
||||
@@ -1,158 +0,0 @@
|
||||
"""Longform Job Library (PR 7).
|
||||
|
||||
``GET /longform/jobs`` — list finished Audiobook + Story renders so the user can
|
||||
re-download them from the Projects view. The render itself (the m4b/mp3) already
|
||||
landed in ``OUTPUTS_DIR`` and is served at ``/audio/<output>``; here we just
|
||||
recover, from each finished job's persisted SSE tail, the output filename plus
|
||||
the chapter count and duration the ``done`` event carried.
|
||||
|
||||
Pure recovery, no synthesis. Defensive by construction: a job whose ``done``
|
||||
event is missing or unparseable is skipped, never surfaced and never a 500.
|
||||
|
||||
The work lives in :func:`build_longform_library`, a pure function over the
|
||||
job-store callables, so it's unit-testable without importing ``main`` (and the
|
||||
torch graph behind it).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Callable, Optional
|
||||
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
logger = logging.getLogger("omnivoice.longform_jobs")
|
||||
router = APIRouter()
|
||||
|
||||
#: Job types this library surfaces. Both flow through the shared longform
|
||||
#: renderer (``_render_longform_sse``) and emit the same ``done`` event shape.
|
||||
_LONGFORM_TYPES = ("audiobook", "story")
|
||||
|
||||
|
||||
def _done_payload_from_events(events: list[dict]) -> Optional[dict]:
|
||||
"""Recover the final ``{"type": "done", ...}`` payload from a job's SSE tail.
|
||||
|
||||
Each row's ``payload`` is the JSON the renderer stored via
|
||||
``job_store.append_event(job_id, json.dumps(payload))``. We scan newest-first
|
||||
and return the first parseable ``done`` event. Anything malformed is skipped
|
||||
— this never raises.
|
||||
"""
|
||||
for ev in reversed(events):
|
||||
raw = ev.get("payload") if isinstance(ev, dict) else None
|
||||
if not raw or not isinstance(raw, str):
|
||||
continue
|
||||
try:
|
||||
obj = json.loads(raw)
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
if isinstance(obj, dict) and obj.get("type") == "done":
|
||||
return obj
|
||||
return None
|
||||
|
||||
|
||||
def _coerce_int(value, default: int = 0) -> int:
|
||||
try:
|
||||
return int(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def _coerce_float(value, default: float = 0.0) -> float:
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def build_longform_library(
|
||||
list_jobs: Callable[..., list[dict]],
|
||||
events_since: Callable[..., list[dict]],
|
||||
*,
|
||||
limit: int = 50,
|
||||
) -> list[dict]:
|
||||
"""Build the newest-first list of finished longform renders.
|
||||
|
||||
Pure over the two job-store callables so tests can pass them directly:
|
||||
|
||||
* ``list_jobs(status="done", limit=...)`` → all done jobs, newest-first.
|
||||
* ``events_since(job_id)`` → that job's persisted SSE events.
|
||||
|
||||
Returns ``[{job_id, type, title?, output, duration_s, chapters,
|
||||
created_at}]``. Jobs that aren't a longform type, or whose ``done`` event /
|
||||
output filename can't be recovered, are silently skipped — the library only
|
||||
ever lists things the user can actually re-download.
|
||||
"""
|
||||
limit = max(1, min(_coerce_int(limit, 50), 500))
|
||||
try:
|
||||
# Over-fetch: non-longform done jobs (dub, etc.) get filtered out below,
|
||||
# so ask for more rows than the caller's limit to still fill the page.
|
||||
rows = list_jobs(status="done", limit=limit * 4)
|
||||
except Exception:
|
||||
logger.warning("longform library: list_jobs failed", exc_info=True)
|
||||
return []
|
||||
|
||||
out: list[dict] = []
|
||||
for row in rows or []:
|
||||
if len(out) >= limit:
|
||||
break
|
||||
try:
|
||||
job_type = row.get("type")
|
||||
job_id = row.get("id")
|
||||
if job_type not in _LONGFORM_TYPES or not job_id:
|
||||
continue
|
||||
try:
|
||||
events = events_since(job_id)
|
||||
except Exception:
|
||||
logger.warning("longform library: events_since failed for %s",
|
||||
job_id, exc_info=True)
|
||||
continue
|
||||
done = _done_payload_from_events(events or [])
|
||||
if not done:
|
||||
continue
|
||||
output = done.get("output")
|
||||
if not output or not isinstance(output, str):
|
||||
continue # nothing to re-download → not worth listing
|
||||
|
||||
item = {
|
||||
"job_id": job_id,
|
||||
"type": job_type,
|
||||
"output": output,
|
||||
"duration_s": round(_coerce_float(done.get("duration_s")), 2),
|
||||
"chapters": _coerce_int(done.get("chapters")),
|
||||
"created_at": row.get("created_at"),
|
||||
}
|
||||
# Title is optional — prefer the done event, fall back to job meta.
|
||||
title = done.get("title")
|
||||
if not title:
|
||||
meta_raw = row.get("meta_json")
|
||||
if isinstance(meta_raw, str) and meta_raw:
|
||||
try:
|
||||
meta = json.loads(meta_raw)
|
||||
if isinstance(meta, dict):
|
||||
title = meta.get("title")
|
||||
except (ValueError, TypeError):
|
||||
title = None
|
||||
if title:
|
||||
item["title"] = title
|
||||
out.append(item)
|
||||
except Exception:
|
||||
# Per-row isolation: one bad row never sinks the whole list.
|
||||
logger.warning("longform library: skipping unparseable job row",
|
||||
exc_info=True)
|
||||
continue
|
||||
return out
|
||||
|
||||
|
||||
@router.get("/longform/jobs")
|
||||
def longform_jobs(limit: int = Query(50, ge=1, le=500)) -> dict:
|
||||
"""Finished Audiobook + Story renders, newest-first, ready to re-download.
|
||||
|
||||
Each item's ``output`` is served at ``/audio/<output>``. Never 500s — on any
|
||||
backend hiccup it returns an empty list rather than an error.
|
||||
"""
|
||||
from core import job_store
|
||||
|
||||
jobs = build_longform_library(
|
||||
job_store.list_jobs, job_store.events_since, limit=limit,
|
||||
)
|
||||
return {"jobs": jobs}
|
||||
@@ -58,31 +58,6 @@ MAX_BUNDLE_BYTES = 100 * 1024 * 1024
|
||||
# ── Export ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _bundle_metadata(profile: dict, **extra) -> dict:
|
||||
"""Common .omnivoice metadata for export + publish.
|
||||
|
||||
Captures ``kind`` and ``vd_states`` so a *designed* persona survives the
|
||||
bundle round-trip as a design (not silently demoted to a clone) — required
|
||||
for the synthetic-only gating of the persona gallery (§R3). Old bundles
|
||||
without these keys import as ``kind='clone'`` (backward-compatible).
|
||||
"""
|
||||
meta = {
|
||||
"bundle_version": BUNDLE_VERSION,
|
||||
"profile_name": profile.get("name", "Unnamed"),
|
||||
"ref_text": profile.get("ref_text", ""),
|
||||
"instruct": profile.get("instruct", ""),
|
||||
"language": profile.get("language", "Auto"),
|
||||
"personality": profile.get("personality", ""),
|
||||
"seed": profile.get("seed"),
|
||||
"kind": profile.get("kind") or "clone",
|
||||
"vd_states": profile.get("vd_states"),
|
||||
"is_locked": bool(profile.get("is_locked")),
|
||||
"omnivoice_version": APP_VERSION,
|
||||
}
|
||||
meta.update(extra)
|
||||
return meta
|
||||
|
||||
|
||||
@router.post("/export/{profile_id}")
|
||||
def export_profile(profile_id: str):
|
||||
"""Export a voice profile as a downloadable .omnivoice bundle (ZIP)."""
|
||||
@@ -100,9 +75,19 @@ def export_profile(profile_id: str):
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
# Metadata
|
||||
metadata = _bundle_metadata(
|
||||
profile, created_at=profile.get("created_at"), exported_at=time.time(),
|
||||
)
|
||||
metadata = {
|
||||
"bundle_version": BUNDLE_VERSION,
|
||||
"profile_name": profile.get("name", "Unnamed"),
|
||||
"ref_text": profile.get("ref_text", ""),
|
||||
"instruct": profile.get("instruct", ""),
|
||||
"language": profile.get("language", "Auto"),
|
||||
"personality": profile.get("personality", ""),
|
||||
"seed": profile.get("seed"),
|
||||
"is_locked": bool(profile.get("is_locked")),
|
||||
"created_at": profile.get("created_at"),
|
||||
"exported_at": time.time(),
|
||||
"omnivoice_version": APP_VERSION,
|
||||
}
|
||||
zf.writestr("metadata.json", json.dumps(metadata, indent=2))
|
||||
|
||||
# Reference audio
|
||||
@@ -206,9 +191,8 @@ async def import_profile(
|
||||
conn.execute(
|
||||
"""INSERT INTO voice_profiles
|
||||
(id, name, ref_audio_path, ref_text, instruct, language,
|
||||
seed, personality, is_locked, locked_audio_path, created_at,
|
||||
kind, vd_states)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
seed, personality, is_locked, locked_audio_path, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(
|
||||
profile_id,
|
||||
metadata.get("profile_name", "Imported Voice"),
|
||||
@@ -221,10 +205,6 @@ async def import_profile(
|
||||
1 if is_locked else 0,
|
||||
locked_audio_filename or "",
|
||||
time.time(),
|
||||
# Preserve the design/clone distinction across the round-trip;
|
||||
# old bundles without these keys import as a clone.
|
||||
metadata.get("kind") or "clone",
|
||||
metadata.get("vd_states"),
|
||||
),
|
||||
)
|
||||
|
||||
@@ -273,11 +253,19 @@ def publish_to_marketplace(
|
||||
|
||||
# Build the bundle
|
||||
with zipfile.ZipFile(str(bundle_path), "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
metadata = _bundle_metadata(
|
||||
profile,
|
||||
tags=[t.strip() for t in tags.split(",") if t.strip()],
|
||||
published_at=time.time(),
|
||||
)
|
||||
metadata = {
|
||||
"bundle_version": BUNDLE_VERSION,
|
||||
"profile_name": profile.get("name", "Unnamed"),
|
||||
"ref_text": profile.get("ref_text", ""),
|
||||
"instruct": profile.get("instruct", ""),
|
||||
"language": profile.get("language", "Auto"),
|
||||
"personality": profile.get("personality", ""),
|
||||
"seed": profile.get("seed"),
|
||||
"is_locked": bool(profile.get("is_locked")),
|
||||
"tags": [t.strip() for t in tags.split(",") if t.strip()],
|
||||
"published_at": time.time(),
|
||||
"omnivoice_version": APP_VERSION,
|
||||
}
|
||||
zf.writestr("metadata.json", json.dumps(metadata, indent=2))
|
||||
|
||||
ref_path = profile.get("ref_audio_path")
|
||||
|
||||
@@ -1,53 +0,0 @@
|
||||
"""REST CRUD for per-agent MCP voice bindings (Wave 2.2 / Spec 2).
|
||||
|
||||
Loopback-gated — the Settings UI manages bindings here. The MCP tools
|
||||
themselves resolve voices via ``services.mcp_bindings.resolve_voice``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from api.dependencies import require_loopback
|
||||
from services import mcp_bindings
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/api/mcp",
|
||||
tags=["mcp"],
|
||||
dependencies=[Depends(require_loopback)],
|
||||
)
|
||||
|
||||
|
||||
class _BindingBody(BaseModel):
|
||||
client_id: str = Field(..., min_length=1, max_length=128)
|
||||
label: str | None = None
|
||||
profile_id: str | None = None
|
||||
default_engine: str | None = None
|
||||
|
||||
|
||||
@router.get("/bindings")
|
||||
def list_bindings():
|
||||
"""All per-agent voice bindings, most-recently-seen first."""
|
||||
return mcp_bindings.list_bindings()
|
||||
|
||||
|
||||
@router.put("/bindings")
|
||||
def upsert_binding(body: _BindingBody):
|
||||
"""Create or update the binding for an MCP client id."""
|
||||
try:
|
||||
return mcp_bindings.upsert_binding(
|
||||
body.client_id,
|
||||
label=body.label,
|
||||
profile_id=body.profile_id,
|
||||
default_engine=body.default_engine,
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
|
||||
@router.delete("/bindings/{client_id}")
|
||||
def delete_binding(client_id: str):
|
||||
if not mcp_bindings.delete_binding(client_id):
|
||||
raise HTTPException(status_code=404, detail="No binding for that client id")
|
||||
return {"deleted": client_id}
|
||||
@@ -82,33 +82,6 @@ class SpeechRequest(BaseModel):
|
||||
"E.g. 'young female, warm tone, slight British accent'.",
|
||||
)
|
||||
instruct: Optional[str] = Field(default=None, description="Style instruction for the TTS engine.")
|
||||
duration: Optional[float] = Field(
|
||||
default=None,
|
||||
gt=0,
|
||||
description="OmniVoice extension: target output duration in seconds.",
|
||||
)
|
||||
seed: Optional[int] = Field(
|
||||
default=None,
|
||||
description="OmniVoice extension: deterministic sampling seed.",
|
||||
)
|
||||
denoise: bool = Field(
|
||||
default=True,
|
||||
description="OmniVoice extension: prepend denoise control when supported.",
|
||||
)
|
||||
preprocess_prompt: bool = Field(
|
||||
default=True,
|
||||
description="OmniVoice extension: trim/preprocess reference prompt when supported.",
|
||||
)
|
||||
chunk_duration: Optional[float] = Field(
|
||||
default=None,
|
||||
ge=0,
|
||||
description="OmniVoice GGUF extension: long-form internal chunk duration.",
|
||||
)
|
||||
chunk_threshold: Optional[float] = Field(
|
||||
default=None,
|
||||
ge=0,
|
||||
description="OmniVoice GGUF extension: long-form internal chunk threshold.",
|
||||
)
|
||||
|
||||
|
||||
class TranscriptionResponse(BaseModel):
|
||||
@@ -236,14 +209,7 @@ def _run_tts(backend, text: str, kw: dict):
|
||||
from services.audio_dsp import apply_mastering, normalize_audio
|
||||
wav = backend.generate(text, **kw)
|
||||
sr = backend.sample_rate
|
||||
# Engines that already emit mastered, studio-grade audio (e.g. VoxCPM2's
|
||||
# native 48 kHz) opt out of apply_mastering via `applies_own_mastering`.
|
||||
# That chain's Compressor + 8% Reverb is tuned for OmniVoice's 24 kHz clone
|
||||
# output; applied to a studio engine it adds an audible level pump and a
|
||||
# reverb tail that degrade the very output we want clean. Loudness
|
||||
# normalisation still runs — it's a benign peak scale, not dynamics.
|
||||
if not getattr(backend, "applies_own_mastering", False):
|
||||
wav = apply_mastering(wav, sample_rate=sr)
|
||||
wav = apply_mastering(wav, sample_rate=sr)
|
||||
wav = normalize_audio(wav, target_dBFS=-2.0)
|
||||
return wav, sr
|
||||
|
||||
@@ -253,28 +219,10 @@ async def create_speech(req: SpeechRequest):
|
||||
"""Generate audio from text. Compatible with OpenAI's POST /v1/audio/speech."""
|
||||
backend = _resolve_engine(req.model)
|
||||
|
||||
# Routing gate (#21 — no silent CPU fallback), identical to REST /generate.
|
||||
from core.device_caps import detect_host_caps
|
||||
from services.engine_routing import resolve_routing, routing_notice
|
||||
_routing = resolve_routing(getattr(backend, "gpu_compat", ("cpu",)), detect_host_caps())
|
||||
if _routing["routing_status"] == "unavailable":
|
||||
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
|
||||
_routing_notice = routing_notice(_routing) # (status, reason) or None
|
||||
|
||||
# Build kwargs for the backend's generate() method
|
||||
kw: dict = {
|
||||
"speed": req.speed,
|
||||
"denoise": req.denoise,
|
||||
"preprocess_prompt": req.preprocess_prompt,
|
||||
}
|
||||
if req.duration is not None:
|
||||
kw["duration"] = req.duration
|
||||
if req.seed is not None:
|
||||
kw["seed"] = req.seed
|
||||
if req.chunk_duration is not None:
|
||||
kw["chunk_duration"] = req.chunk_duration
|
||||
if req.chunk_threshold is not None:
|
||||
kw["chunk_threshold"] = req.chunk_threshold
|
||||
if req.language:
|
||||
kw["language"] = req.language
|
||||
if req.instruct:
|
||||
@@ -303,8 +251,6 @@ async def create_speech(req: SpeechRequest):
|
||||
kw["ref_text"] = row["ref_text"]
|
||||
if row["instruct"] and not req.instruct:
|
||||
kw["instruct"] = row["instruct"]
|
||||
if req.seed is None and row["seed"] is not None:
|
||||
kw["seed"] = row["seed"]
|
||||
else:
|
||||
# Not a profile ID — forward as engine preset name
|
||||
kw["voice"] = voice
|
||||
@@ -321,20 +267,13 @@ async def create_speech(req: SpeechRequest):
|
||||
|
||||
audio_bytes, mime_type, ext = _encode_audio(wav, sr, req.response_format)
|
||||
|
||||
_headers = {
|
||||
"Content-Length": str(len(audio_bytes)),
|
||||
"Content-Disposition": f'inline; filename="speech.{ext}"',
|
||||
}
|
||||
if _routing_notice:
|
||||
from services.engine_routing import header_safe_reason
|
||||
_headers["X-OmniVoice-Routing"] = _routing_notice[0]
|
||||
_hr = header_safe_reason(_routing_notice[1])
|
||||
if _hr:
|
||||
_headers["X-OmniVoice-Routing-Reason"] = _hr
|
||||
return StreamingResponse(
|
||||
io.BytesIO(audio_bytes),
|
||||
media_type=mime_type,
|
||||
headers=_headers,
|
||||
headers={
|
||||
"Content-Length": str(len(audio_bytes)),
|
||||
"Content-Disposition": f'inline; filename="speech.{ext}"',
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -1,327 +0,0 @@
|
||||
"""HTTP layer for the `.ovsvoice` portable persona format (#29 / parity §R3 G1).
|
||||
|
||||
Thin router over `services.persona_bundle`:
|
||||
|
||||
POST /personas/export/{profile_id} → stream a downloadable .ovsvoice
|
||||
POST /personas/import → create a profile from a bundle
|
||||
POST /personas/inspect → read a bundle's manifest, no writes
|
||||
|
||||
Mirrors the legacy `.omnivoice` endpoints (`marketplace.py`) and reuses the
|
||||
same path-confinement (`_voices_path`) + consent floor. `.ovsvoice` is additive;
|
||||
`.omnivoice` import stays a compatible legacy reader.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import functools
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from fastapi import APIRouter, File, HTTPException, Query, UploadFile
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from core import event_bus
|
||||
from core.config import VOICES_DIR # noqa: F401 — re-exported for tests/monkeypatch
|
||||
from core.db import db_conn
|
||||
from core.version import APP_VERSION
|
||||
from services import persona_bundle as pb
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("omnivoice.personas")
|
||||
|
||||
|
||||
def _safe_name(name: str, profile_id: str) -> str:
|
||||
"""Sanitised download filename stem (marketplace idiom); empty → persona_<id>."""
|
||||
cleaned = "".join(
|
||||
c if c.isalnum() or c in "-_ " else "" for c in (name or "")
|
||||
).strip().replace(" ", "_")[:40]
|
||||
return cleaned or f"persona_{profile_id}"
|
||||
|
||||
|
||||
# ── Export ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/personas/export/{profile_id}")
|
||||
async def export_persona(
|
||||
profile_id: str,
|
||||
license_spdx: str = Query(pb.DEFAULT_LICENSE),
|
||||
tags: str = Query(""),
|
||||
include_reference: bool = Query(True),
|
||||
):
|
||||
"""Build + stream a `.ovsvoice` bundle for a profile."""
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT * FROM voice_profiles WHERE id = ?", (profile_id,)
|
||||
).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Voice profile not found")
|
||||
profile = dict(row)
|
||||
|
||||
tag_list = [t.strip() for t in tags.split(",") if t.strip()]
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
content = await loop.run_in_executor(
|
||||
None,
|
||||
functools.partial(
|
||||
pb.build_persona_bundle,
|
||||
profile,
|
||||
license_spdx=license_spdx,
|
||||
tags=tag_list,
|
||||
include_reference=include_reference,
|
||||
engine_id=os.environ.get("OMNIVOICE_MODEL", ""),
|
||||
omnivoice_version=APP_VERSION,
|
||||
),
|
||||
)
|
||||
except pb.NoPreviewSource:
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="This profile has no readable reference or locked audio to "
|
||||
"build a preview from — re-create or re-import it.",
|
||||
)
|
||||
except Exception:
|
||||
logger.exception("persona export failed for %s", profile_id)
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail="Could not build the persona bundle — see Settings → Logs.",
|
||||
)
|
||||
|
||||
filename = f"{_safe_name(profile.get('name'), profile_id)}.ovsvoice"
|
||||
from io import BytesIO
|
||||
return StreamingResponse(
|
||||
BytesIO(content),
|
||||
media_type="application/zip",
|
||||
headers={
|
||||
"Content-Disposition": f'attachment; filename="{filename}"',
|
||||
"Content-Length": str(len(content)),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# ── Import ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _voices_dest(filename: str) -> str:
|
||||
"""Resolve an output filename inside VOICES_DIR; 400 on escape (belt+braces —
|
||||
the name is always server-generated `{profile_id}…`)."""
|
||||
from api.routers.profiles import _voices_path
|
||||
path = _voices_path(filename)
|
||||
if path is None:
|
||||
raise HTTPException(status_code=400, detail="Invalid profile id")
|
||||
return path
|
||||
|
||||
|
||||
def _consent_verified(parsed: pb.ParsedPersona, consent_path: str | None) -> bool:
|
||||
"""B12-B16: trust verified-own-voice ONLY with a real recording (≥ floor) AND
|
||||
non-empty consent_text AND a consent.json present. The manifest flag alone
|
||||
can't forge it."""
|
||||
if not parsed.consent or not consent_path:
|
||||
return False
|
||||
if os.path.getsize(consent_path) < pb._MIN_CONSENT_AUDIO_BYTES:
|
||||
return False
|
||||
return bool((parsed.consent.get("consent_text") or "").strip())
|
||||
|
||||
|
||||
@router.post("/personas/import")
|
||||
async def import_persona(file: UploadFile = File(...)):
|
||||
"""Create a new voice profile from a `.ovsvoice` (or legacy `.omnivoice`) bundle."""
|
||||
name = (file.filename or "").lower()
|
||||
if not name.endswith(".ovsvoice") and not name.endswith(".omnivoice"):
|
||||
raise HTTPException(status_code=400, detail="File must be a .ovsvoice or .omnivoice bundle")
|
||||
|
||||
content = await file.read()
|
||||
try:
|
||||
parsed = pb.parse_persona_bundle(content)
|
||||
except pb.BundleError as e:
|
||||
raise HTTPException(status_code=e.status, detail=e.detail)
|
||||
|
||||
persona = parsed.manifest.get("persona") or {}
|
||||
written: list[str] = []
|
||||
|
||||
def _gen_id() -> str:
|
||||
return str(uuid.uuid4())[:8]
|
||||
|
||||
profile_id = _gen_id()
|
||||
try:
|
||||
# ── Audio members → server-named files (never the member name). ──
|
||||
ref_filename = None
|
||||
locked_filename = None
|
||||
if "ref_audio" in parsed.members:
|
||||
ref_filename = f"{profile_id}{parsed.member_ext('ref_audio')}"
|
||||
dest = _voices_dest(ref_filename)
|
||||
parsed.extract_member("ref_audio", dest); written.append(dest)
|
||||
if "locked_audio" in parsed.members:
|
||||
locked_filename = f"{profile_id}_locked{parsed.member_ext('locked_audio')}"
|
||||
dest = _voices_dest(locked_filename)
|
||||
parsed.extract_member("locked_audio", dest); written.append(dest)
|
||||
# Preview-only bundle (A12/B8): use the preview as the usable ref clip.
|
||||
if ref_filename is None and locked_filename is None and "preview" in parsed.members:
|
||||
ref_filename = f"{profile_id}{parsed.member_ext('preview')}"
|
||||
dest = _voices_dest(ref_filename)
|
||||
parsed.extract_member("preview", dest); written.append(dest)
|
||||
if ref_filename is None and locked_filename is None:
|
||||
raise HTTPException(status_code=400, detail="bundle has no usable audio")
|
||||
|
||||
# ── Consent recording (optional) ──
|
||||
consent_filename = None
|
||||
consent_path = None
|
||||
if "consent_audio" in parsed.members:
|
||||
consent_filename = f"{profile_id}_consent{parsed.member_ext('consent_audio')}"
|
||||
consent_path = _voices_dest(consent_filename)
|
||||
parsed.extract_member("consent_audio", consent_path); written.append(consent_path)
|
||||
|
||||
verified = _consent_verified(parsed, consent_path)
|
||||
consent_text = ((parsed.consent or {}).get("consent_text") or "").strip()
|
||||
recorded_at = None
|
||||
if verified:
|
||||
try:
|
||||
recorded_at = float(parsed.consent.get("recorded_at"))
|
||||
except (TypeError, ValueError):
|
||||
recorded_at = time.time()
|
||||
|
||||
is_locked = bool(persona.get("is_locked") and locked_filename)
|
||||
ref_for_db = ref_filename or locked_filename # at least one is set
|
||||
|
||||
def _insert(pid: str):
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"""INSERT INTO voice_profiles
|
||||
(id, name, ref_audio_path, ref_text, instruct, language,
|
||||
seed, personality, is_locked, locked_audio_path, created_at,
|
||||
kind, vd_states,
|
||||
verified_own_voice, consent_text, consent_audio_path, consent_recorded_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(
|
||||
pid,
|
||||
persona.get("name") or "Imported Voice",
|
||||
ref_for_db,
|
||||
persona.get("ref_text", ""),
|
||||
persona.get("instruct", ""),
|
||||
persona.get("language", "Auto"),
|
||||
persona.get("seed"),
|
||||
persona.get("personality", ""),
|
||||
1 if is_locked else 0,
|
||||
locked_filename or "",
|
||||
time.time(),
|
||||
persona.get("kind") or "clone",
|
||||
persona.get("vd_states"),
|
||||
1 if verified else 0,
|
||||
# Keep the attestation text so the user can re-attest locally,
|
||||
# even when imported unverified.
|
||||
consent_text,
|
||||
consent_filename if verified else "",
|
||||
recorded_at if verified else None,
|
||||
),
|
||||
)
|
||||
|
||||
import sqlite3
|
||||
try:
|
||||
_insert(profile_id)
|
||||
except sqlite3.IntegrityError:
|
||||
profile_id = _gen_id() # one retry on id collision (B20)
|
||||
# rename the on-disk files to the new id so they still match the row
|
||||
written = _rename_for_new_id(written, profile_id)
|
||||
ref_for_db = _retarget(ref_for_db, profile_id)
|
||||
locked_filename = _retarget(locked_filename, profile_id)
|
||||
consent_filename = _retarget(consent_filename, profile_id)
|
||||
_insert(profile_id)
|
||||
|
||||
except HTTPException:
|
||||
_cleanup(written)
|
||||
raise
|
||||
except Exception:
|
||||
_cleanup(written)
|
||||
logger.exception("persona import failed")
|
||||
raise HTTPException(status_code=500, detail="Import failed; no files were kept.")
|
||||
|
||||
event_bus.emit("profiles", {"action": "created", "id": profile_id})
|
||||
logger.info("Imported persona %r as %s (verified=%s)", persona.get("name"), profile_id, verified)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"profile_id": profile_id,
|
||||
"name": persona.get("name") or "Imported Voice",
|
||||
"kind": persona.get("kind") or "clone",
|
||||
"verified_own_voice": verified,
|
||||
"preview_only": parsed.preview_only,
|
||||
"license_spdx": parsed.license_spdx,
|
||||
"watermarked_preview": parsed.watermarked_preview,
|
||||
"source_bundle": file.filename,
|
||||
"schema_version_ahead": parsed.schema_version_ahead,
|
||||
}
|
||||
|
||||
|
||||
def _cleanup(paths: list[str]) -> None:
|
||||
for p in paths:
|
||||
try:
|
||||
if p and os.path.exists(p):
|
||||
os.remove(p)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _rename_for_new_id(written: list[str], new_id: str) -> list[str]:
|
||||
"""After an id-collision retry, rename each written file to carry the new id
|
||||
(filenames are `{old_id}…`; swap the leading 8-char stem)."""
|
||||
out = []
|
||||
for p in written:
|
||||
d, base = os.path.split(p)
|
||||
# base looks like {id}{ext} | {id}_locked{ext} | {id}_consent{ext}
|
||||
new_base = new_id + base[8:]
|
||||
new_path = os.path.join(d, new_base)
|
||||
try:
|
||||
os.replace(p, new_path)
|
||||
out.append(new_path)
|
||||
except OSError:
|
||||
out.append(p)
|
||||
return out
|
||||
|
||||
|
||||
def _retarget(filename: str | None, new_id: str) -> str | None:
|
||||
return new_id + filename[8:] if filename else filename
|
||||
|
||||
|
||||
# ── Inspect (no-write preview) ──────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/personas/inspect")
|
||||
async def inspect_persona(file: UploadFile = File(...)):
|
||||
"""Read a bundle's manifest + consent summary WITHOUT writing any file or row."""
|
||||
name = (file.filename or "").lower()
|
||||
if not name.endswith(".ovsvoice") and not name.endswith(".omnivoice"):
|
||||
raise HTTPException(status_code=400, detail="File must be a .ovsvoice or .omnivoice bundle")
|
||||
content = await file.read()
|
||||
try:
|
||||
parsed = pb.parse_persona_bundle(content)
|
||||
except pb.BundleError as e:
|
||||
raise HTTPException(status_code=e.status, detail=e.detail)
|
||||
|
||||
persona = parsed.manifest.get("persona") or {}
|
||||
consent_summary = None
|
||||
if parsed.consent:
|
||||
has_recording = "consent_audio" in parsed.members
|
||||
consent_summary = {
|
||||
"verified_claimed": bool(parsed.consent.get("verified_own_voice")),
|
||||
"method": parsed.consent.get("method", ""),
|
||||
"has_recording": has_recording,
|
||||
# would_verify mirrors import's gate, minus the byte-floor check
|
||||
# (inspect never extracts to measure size — advisory only).
|
||||
"would_verify": has_recording and bool((parsed.consent.get("consent_text") or "").strip()),
|
||||
}
|
||||
|
||||
return {
|
||||
"format": "omnivoice-legacy" if parsed.is_legacy else pb.OVSVOICE_FORMAT,
|
||||
"schema_version": parsed.manifest.get("schema_version", pb.OVSVOICE_SCHEMA_VERSION),
|
||||
"name": persona.get("name") or "Imported Voice",
|
||||
"kind": persona.get("kind") or "clone",
|
||||
"language": persona.get("language", "Auto"),
|
||||
"personality": persona.get("personality", ""),
|
||||
"is_locked": bool(persona.get("is_locked")),
|
||||
"license_spdx": parsed.license_spdx,
|
||||
"tags": parsed.manifest.get("tags") or [],
|
||||
"preview_only": parsed.preview_only,
|
||||
"watermarked_preview": parsed.watermarked_preview,
|
||||
"consent": consent_summary,
|
||||
"schema_version_ahead": parsed.schema_version_ahead,
|
||||
}
|
||||
+18
-276
@@ -1,5 +1,4 @@
|
||||
import os
|
||||
import re
|
||||
import uuid
|
||||
import time
|
||||
import shutil
|
||||
@@ -35,103 +34,29 @@ def list_profiles():
|
||||
rows = conn.execute("SELECT * FROM voice_profiles ORDER BY created_at DESC").fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
_DESIGN_SEED = 42 # deterministic sample render, same as archetype previews
|
||||
|
||||
|
||||
@router.post("/profiles")
|
||||
async def create_profile(
|
||||
name: str = Form(...),
|
||||
ref_audio: Optional[UploadFile] = File(None),
|
||||
ref_audio: UploadFile = File(...),
|
||||
ref_text: str = Form(""),
|
||||
instruct: str = Form(""),
|
||||
language: str = Form("Auto"),
|
||||
seed: Optional[int] = Form(None),
|
||||
personality: str = Form(""),
|
||||
kind: str = Form("clone"),
|
||||
vd_states: Optional[str] = Form(None),
|
||||
):
|
||||
"""Create a voice profile (spec: docs/specs/voice-studio-unification.md §5).
|
||||
|
||||
kind='clone' — requires `ref_audio` (the user's reference recording).
|
||||
kind='design' — requires `vd_states` (JSON of category picks); the server
|
||||
renders a deterministic sample WAV (seed 42, same path as
|
||||
archetype materialization) and stores it as the profile's
|
||||
reference so the voice identity is stable across runs.
|
||||
"""
|
||||
if kind not in ("clone", "design"):
|
||||
raise HTTPException(status_code=422, detail="kind must be 'clone' or 'design'")
|
||||
if kind == "clone" and ref_audio is None:
|
||||
raise HTTPException(status_code=422, detail="clone profiles require ref_audio")
|
||||
if kind == "design":
|
||||
if not (vd_states or "").strip():
|
||||
raise HTTPException(status_code=422, detail="design profiles require vd_states")
|
||||
import json as _json
|
||||
try:
|
||||
parsed = _json.loads(vd_states)
|
||||
if not isinstance(parsed, dict):
|
||||
raise ValueError("not an object")
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=422, detail="vd_states must be a JSON object")
|
||||
# An all-Auto design (every category left on "Auto") yields an empty
|
||||
# instruct — that's still a valid, saveable voice: synthesis falls back
|
||||
# to neutral instruct-only conditioning (see generation.py design path).
|
||||
# Don't gate save on a non-empty instruct.
|
||||
|
||||
profile_id = str(uuid.uuid4())[:8]
|
||||
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
|
||||
audio_filename = f"{profile_id}{ext}"
|
||||
audio_path = os.path.join(VOICES_DIR, audio_filename)
|
||||
|
||||
if kind == "clone":
|
||||
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
|
||||
audio_filename = f"{profile_id}{ext}"
|
||||
audio_path = os.path.join(VOICES_DIR, audio_filename)
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(await ref_audio.read())
|
||||
used_seed = seed
|
||||
else:
|
||||
# Saving a design profile is a pure persistence operation — it must not
|
||||
# depend on a loaded TTS model (issue #476: on a fresh model-less Docker
|
||||
# image the render forced a full model load + inference that 503'd, so
|
||||
# the save failed). We try the deterministic identity sample opportunist-
|
||||
# ically through the one shared TTS path (archetypes' renderer, never a
|
||||
# second inference code path); if the engine isn't ready it's rendered
|
||||
# lazily on first preview/use. The row carries vd_states + instruct, so
|
||||
# the voice is fully usable without the sample (synthesis falls back to
|
||||
# instruct-only conditioning — see generation.py's design path).
|
||||
from pathlib import Path
|
||||
from api.routers.archetypes import _render_archetype_wav
|
||||
audio_filename = f"{profile_id}.wav"
|
||||
audio_path = os.path.join(VOICES_DIR, audio_filename)
|
||||
try:
|
||||
await _render_archetype_wav(
|
||||
{
|
||||
"language": language,
|
||||
"sample_script": ref_text, # optional custom sample line
|
||||
"instruct": instruct,
|
||||
},
|
||||
Path(audio_path),
|
||||
)
|
||||
except Exception:
|
||||
# Engine unavailable / OOM / inference failure — defer the sample.
|
||||
# Store the row with no ref_audio_path; the identity sample is
|
||||
# rendered on first preview or use. Never let this block the save.
|
||||
import logging
|
||||
logging.getLogger("omnivoice.profiles").info(
|
||||
"Design profile %s saved with sample pending — "
|
||||
"voice engine not ready; will render on first use", profile_id,
|
||||
)
|
||||
if os.path.exists(audio_path): # partial/blank render: don't keep it
|
||||
with __import__("contextlib").suppress(OSError):
|
||||
os.remove(audio_path)
|
||||
audio_filename = None
|
||||
used_seed = seed if seed is not None else _DESIGN_SEED
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(await ref_audio.read())
|
||||
|
||||
try:
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, "
|
||||
"language, seed, personality, kind, vd_states, created_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(profile_id, name, audio_filename, ref_text, instruct, language,
|
||||
used_seed, personality, kind, vd_states, time.time())
|
||||
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(profile_id, name, audio_filename, ref_text, instruct, language, seed, personality, time.time())
|
||||
)
|
||||
except Exception:
|
||||
# Clean up orphaned audio file if DB insert fails
|
||||
@@ -139,7 +64,7 @@ async def create_profile(
|
||||
os.remove(audio_path)
|
||||
raise
|
||||
event_bus.emit("profiles", {"action": "created", "id": profile_id})
|
||||
return {"id": profile_id, "name": name, "kind": kind}
|
||||
return {"id": profile_id, "name": name}
|
||||
|
||||
@router.get("/profiles/{profile_id}")
|
||||
def get_profile(profile_id: str):
|
||||
@@ -238,96 +163,20 @@ def get_profile_usage(profile_id: str):
|
||||
}
|
||||
|
||||
|
||||
# profile_id is a request path param and the audio filename derives from it, so
|
||||
# constrain it to the generated-id charset (no separators / `..` possible) before
|
||||
# any path use, and read only a *direct child* of VOICES_DIR — os.path.basename()
|
||||
# strips any directory component (a path-injection / CWE-22 barrier).
|
||||
_PROFILE_ID_RE = re.compile(r"[A-Za-z0-9_-]{1,64}")
|
||||
|
||||
|
||||
@router.get("/profiles/{profile_id}/audio")
|
||||
async def get_profile_audio(profile_id: str):
|
||||
if not _PROFILE_ID_RE.fullmatch(profile_id or ""):
|
||||
return Response("Profile not found", status_code=404)
|
||||
def get_profile_audio(profile_id: str):
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT ref_audio_path, locked_audio_path, kind, instruct, language, ref_text "
|
||||
"FROM voice_profiles WHERE id=?",
|
||||
(profile_id,),
|
||||
).fetchone()
|
||||
row = conn.execute("SELECT ref_audio_path, locked_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
if not row:
|
||||
return Response("Profile not found", status_code=404)
|
||||
audio_file = row["locked_audio_path"] or row["ref_audio_path"]
|
||||
if not audio_file:
|
||||
# A design profile saved before the engine was ready (issue #476) has no
|
||||
# identity sample yet. Render it lazily now — the deterministic seed-42
|
||||
# sample is reproducible, so a deferred render matches a save-time one.
|
||||
rendered = await _materialize_design_sample(profile_id, row)
|
||||
if rendered is None:
|
||||
return Response("No audio available", status_code=404)
|
||||
audio_file = rendered
|
||||
# CWE-22: resolve the DB-stored filename strictly inside VOICES_DIR via the
|
||||
# shared guard — _voices_path() applies the os.path.basename() barrier plus
|
||||
# symlink-resolved containment (same path the consent endpoint trusts).
|
||||
audio_path = _voices_path(str(audio_file))
|
||||
if audio_path is None or not os.path.exists(audio_path):
|
||||
return Response("No audio available", status_code=404)
|
||||
audio_path = os.path.join(VOICES_DIR, audio_file)
|
||||
if not os.path.exists(audio_path):
|
||||
return Response("Audio file missing", status_code=404)
|
||||
return FileResponse(audio_path, media_type="audio/wav")
|
||||
|
||||
|
||||
async def _materialize_design_sample(profile_id: str, row) -> Optional[str]:
|
||||
"""Render a design profile's pending identity sample on first request.
|
||||
|
||||
Returns the stored filename on success, or None if this isn't a renderable
|
||||
design row. Raises HTTPException(503) with a precise "model not ready"
|
||||
message if the engine is genuinely unavailable — saving never depends on
|
||||
this, but a user who explicitly asks for the sample gets a clear signal.
|
||||
"""
|
||||
try:
|
||||
kind = row["kind"]
|
||||
except (KeyError, IndexError):
|
||||
kind = "clone"
|
||||
if kind != "design":
|
||||
return None
|
||||
|
||||
from pathlib import Path
|
||||
from api.routers.archetypes import _render_archetype_wav
|
||||
|
||||
audio_filename = f"{profile_id}.wav"
|
||||
# CWE-22: resolve under VOICES_DIR via the shared basename + containment
|
||||
# guard before rendering (rejects any escape).
|
||||
audio_path = _voices_path(audio_filename)
|
||||
if audio_path is None:
|
||||
raise HTTPException(status_code=400, detail="invalid profile identifier")
|
||||
try:
|
||||
await _render_archetype_wav(
|
||||
{
|
||||
"language": row["language"] or "Auto",
|
||||
"sample_script": row["ref_text"] or "",
|
||||
"instruct": row["instruct"] or "",
|
||||
},
|
||||
Path(audio_path),
|
||||
)
|
||||
except Exception as e:
|
||||
with __import__("contextlib").suppress(OSError):
|
||||
if os.path.exists(audio_path):
|
||||
os.remove(audio_path)
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail=(
|
||||
"The voice engine isn't ready yet, so this designed voice's "
|
||||
"preview sample can't be rendered. Finish setup / download a "
|
||||
f"model, then try again. ({e})"
|
||||
),
|
||||
)
|
||||
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"UPDATE voice_profiles SET ref_audio_path=? WHERE id=?",
|
||||
(audio_filename, profile_id),
|
||||
)
|
||||
return audio_filename
|
||||
|
||||
@router.post("/profiles/{profile_id}/lock")
|
||||
async def lock_profile(
|
||||
profile_id: str,
|
||||
@@ -385,122 +234,15 @@ async def unlock_profile(profile_id: str):
|
||||
event_bus.emit("profiles", {"action": "unlocked", "id": profile_id})
|
||||
return {"unlocked": True, "profile_id": profile_id}
|
||||
|
||||
# ── Consent lock (parity program Wave 0.2) ─────────────────────────────────
|
||||
#
|
||||
# A profile becomes "verified own voice" when its owner records themselves
|
||||
# reading a consent statement. The recording is provenance, not a voiceprint
|
||||
# check — agentic features and gallery sharing gate on the flag; plain local
|
||||
# synthesis never does. Spec: docs/competitive-analysis.md Action 22.
|
||||
|
||||
_MIN_CONSENT_AUDIO_BYTES = 1000 # same floor as the frontend recorder
|
||||
|
||||
# Upload filename extension whitelist — anything else falls back to .wav so a
|
||||
# crafted filename can never influence the on-disk path (py/path-injection).
|
||||
_CONSENT_EXT_RE = re.compile(r"^\.[A-Za-z0-9]{1,8}$")
|
||||
|
||||
|
||||
def _voices_path(filename: str) -> Optional[str]:
|
||||
"""Resolve a DB-stored audio filename strictly inside VOICES_DIR.
|
||||
|
||||
Rejects anything that isn't a bare filename or that escapes the voices
|
||||
directory after symlink resolution. Returns None instead of raising so
|
||||
cleanup paths can simply skip bad values.
|
||||
"""
|
||||
if not filename or os.path.basename(filename) != filename:
|
||||
return None
|
||||
root = os.path.realpath(VOICES_DIR)
|
||||
path = os.path.realpath(os.path.join(root, filename))
|
||||
if not path.startswith(root + os.sep):
|
||||
return None
|
||||
return path
|
||||
|
||||
|
||||
@router.post("/profiles/{profile_id}/consent")
|
||||
async def record_consent(
|
||||
profile_id: str,
|
||||
consent_audio: UploadFile = File(...),
|
||||
consent_text: str = Form(...),
|
||||
):
|
||||
if not consent_text.strip():
|
||||
raise HTTPException(status_code=422, detail="consent_text must not be empty")
|
||||
data = await consent_audio.read()
|
||||
if len(data) < _MIN_CONSENT_AUDIO_BYTES:
|
||||
raise HTTPException(status_code=422, detail="consent recording is too short")
|
||||
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT id, consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)
|
||||
).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Profile not found")
|
||||
|
||||
ext = os.path.splitext(consent_audio.filename or "")[1]
|
||||
if not _CONSENT_EXT_RE.match(ext):
|
||||
ext = ".wav"
|
||||
audio_filename = f"{profile_id}_consent{ext}"
|
||||
audio_path = _voices_path(audio_filename)
|
||||
if audio_path is None: # profile_id is server-generated; this is belt+braces
|
||||
raise HTTPException(status_code=400, detail="Invalid profile id")
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(data)
|
||||
|
||||
# A re-record may change the extension; drop the superseded file.
|
||||
old = row["consent_audio_path"]
|
||||
if old and old != audio_filename:
|
||||
old_path = _voices_path(old)
|
||||
if old_path and os.path.exists(old_path):
|
||||
os.remove(old_path)
|
||||
|
||||
recorded_at = time.time()
|
||||
try:
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"UPDATE voice_profiles SET verified_own_voice=1, consent_text=?, "
|
||||
"consent_audio_path=?, consent_recorded_at=? WHERE id=?",
|
||||
(consent_text.strip(), audio_filename, recorded_at, profile_id),
|
||||
)
|
||||
except Exception:
|
||||
if os.path.exists(audio_path):
|
||||
os.remove(audio_path)
|
||||
raise
|
||||
event_bus.emit("profiles", {"action": "consent_recorded", "id": profile_id})
|
||||
return {
|
||||
"id": profile_id,
|
||||
"verified_own_voice": True,
|
||||
"consent_recorded_at": recorded_at,
|
||||
}
|
||||
|
||||
|
||||
@router.delete("/profiles/{profile_id}/consent")
|
||||
def revoke_consent(profile_id: str):
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)
|
||||
).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Profile not found")
|
||||
conn.execute(
|
||||
"UPDATE voice_profiles SET verified_own_voice=0, consent_text='', "
|
||||
"consent_audio_path='', consent_recorded_at=NULL WHERE id=?",
|
||||
(profile_id,),
|
||||
)
|
||||
if row["consent_audio_path"]:
|
||||
path = _voices_path(row["consent_audio_path"])
|
||||
if path and os.path.exists(path):
|
||||
os.remove(path)
|
||||
event_bus.emit("profiles", {"action": "consent_revoked", "id": profile_id})
|
||||
return {"id": profile_id, "verified_own_voice": False}
|
||||
|
||||
|
||||
@router.delete("/profiles/{profile_id}")
|
||||
def delete_profile(profile_id: str):
|
||||
with db_conn() as conn:
|
||||
row = conn.execute("SELECT ref_audio_path, locked_audio_path, consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
row = conn.execute("SELECT ref_audio_path, locked_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
|
||||
if row:
|
||||
for col in ["ref_audio_path", "locked_audio_path", "consent_audio_path"]:
|
||||
for col in ["ref_audio_path", "locked_audio_path"]:
|
||||
if row[col]:
|
||||
path = _voices_path(row[col])
|
||||
if path and os.path.exists(path):
|
||||
path = os.path.join(VOICES_DIR, row[col])
|
||||
if os.path.exists(path):
|
||||
os.remove(path)
|
||||
# Prevent FOREIGN KEY constraint failure
|
||||
conn.execute("UPDATE generation_history SET profile_id = NULL WHERE profile_id=?", (profile_id,))
|
||||
|
||||
@@ -2,7 +2,6 @@ import uuid
|
||||
import time
|
||||
import json
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from core.db import db_conn
|
||||
from core import event_bus
|
||||
@@ -10,10 +9,6 @@ from schemas.requests import ProjectSaveRequest
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
class ProjectRenameRequest(BaseModel):
|
||||
name: str
|
||||
|
||||
@router.get("/projects")
|
||||
async def list_projects():
|
||||
with db_conn() as conn:
|
||||
@@ -64,26 +59,6 @@ async def update_project(project_id: str, req: ProjectSaveRequest):
|
||||
event_bus.emit("projects", {"action": "updated", "id": project_id})
|
||||
return {"id": project_id, "name": req.name, "updated_at": now}
|
||||
|
||||
@router.patch("/projects/{project_id}")
|
||||
async def rename_project(project_id: str, req: ProjectRenameRequest):
|
||||
"""Lightweight rename — updates only the project name (and updated_at),
|
||||
without re-serialising the whole state blob like PUT does."""
|
||||
name = req.name.strip()
|
||||
if not name:
|
||||
raise HTTPException(status_code=400, detail="Project name cannot be empty")
|
||||
now = time.time()
|
||||
with db_conn() as conn:
|
||||
row = conn.execute("SELECT id FROM studio_projects WHERE id=?", (project_id,)).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(status_code=404, detail="Project not found")
|
||||
conn.execute(
|
||||
"UPDATE studio_projects SET name=?, updated_at=? WHERE id=?",
|
||||
(name, now, project_id),
|
||||
)
|
||||
event_bus.emit("projects", {"action": "renamed", "id": project_id})
|
||||
return {"id": project_id, "name": name, "updated_at": now}
|
||||
|
||||
|
||||
@router.delete("/projects/{project_id}")
|
||||
async def delete_project(project_id: str):
|
||||
with db_conn() as conn:
|
||||
|
||||
@@ -123,117 +123,6 @@ def set_torch_compile_disabled(body: _TorchCompileBody):
|
||||
return _torch_compile_state()
|
||||
|
||||
|
||||
# ── Dictation refinement (parity program Wave 2.1 / Spec 3 phase 2) ───────
|
||||
|
||||
|
||||
class _RefinementBody(BaseModel):
|
||||
auto: bool | None = None
|
||||
smart_cleanup: bool | None = None
|
||||
self_correction: bool | None = None
|
||||
preserve_technical: bool | None = None
|
||||
|
||||
|
||||
def _refinement_state():
|
||||
from services.refinement import get_refinement_config
|
||||
from services.llm_backend import get_active_llm_backend
|
||||
|
||||
cfg = get_refinement_config()
|
||||
# The UI shows whether refinement can actually run (needs an LLM).
|
||||
cfg["llm_ready"] = get_active_llm_backend().id != "off"
|
||||
return cfg
|
||||
|
||||
|
||||
@router.get("/dictation-refinement")
|
||||
def get_dictation_refinement():
|
||||
"""Current refinement config + whether an LLM backend is configured."""
|
||||
return _refinement_state()
|
||||
|
||||
|
||||
@router.put("/dictation-refinement")
|
||||
def set_dictation_refinement(body: _RefinementBody):
|
||||
from services.refinement import set_refinement_config
|
||||
|
||||
try:
|
||||
set_refinement_config({k: v for k, v in body.model_dump().items() if v is not None})
|
||||
except Exception:
|
||||
logger.exception("set_dictation_refinement failed")
|
||||
raise HTTPException(status_code=500, detail="Failed to persist setting")
|
||||
return _refinement_state()
|
||||
|
||||
|
||||
# ── LLM endpoint (parity program Wave 2.4 / §R2 rung 4) ───────────────────
|
||||
# Focused configuration for the OpenAI-compatible LLM endpoint that powers
|
||||
# cinematic translate, glossary auto-extract, and dictation refinement.
|
||||
# Persistence rides the existing TRANSLATE_BASE_URL / TRANSLATE_API_KEY /
|
||||
# TRANSLATE_MODEL env vars (already in system.py PERSISTENT_KEYS, restored
|
||||
# at startup) so the resolution path in llm_backend/translator is unchanged.
|
||||
|
||||
|
||||
class _LLMEndpointBody(BaseModel):
|
||||
base_url: str | None = None
|
||||
model: str | None = None
|
||||
api_key: str | None = None # None = leave unchanged; "" = clear
|
||||
|
||||
|
||||
def _mask(secret: str | None) -> str | None:
|
||||
if not secret:
|
||||
return None
|
||||
return f"…{secret[-4:]}" if len(secret) > 4 else "set"
|
||||
|
||||
|
||||
def _llm_endpoint_state():
|
||||
from services.llm_backend import OpenAICompatBackend
|
||||
|
||||
ok, reason = OpenAICompatBackend.is_available()
|
||||
return {
|
||||
"base_url": os.environ.get("TRANSLATE_BASE_URL", ""),
|
||||
"model": os.environ.get("TRANSLATE_MODEL", ""),
|
||||
"api_key_masked": _mask(
|
||||
os.environ.get("TRANSLATE_API_KEY") or os.environ.get("OPENAI_API_KEY")
|
||||
),
|
||||
"available": ok,
|
||||
"reason": None if ok else reason,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/llm-endpoint")
|
||||
def get_llm_endpoint():
|
||||
"""Current OpenAI-compatible LLM endpoint config + live availability."""
|
||||
return _llm_endpoint_state()
|
||||
|
||||
|
||||
@router.put("/llm-endpoint")
|
||||
def set_llm_endpoint(body: _LLMEndpointBody):
|
||||
"""Persist base URL / model / API key for the OpenAI-compatible endpoint.
|
||||
|
||||
Reuses the env-var persistence path (prefs.json, restored at startup):
|
||||
base_url -> TRANSLATE_BASE_URL, model -> TRANSLATE_MODEL,
|
||||
api_key -> TRANSLATE_API_KEY. A None field is left unchanged; an empty
|
||||
string clears it. Ollama ignores the key; vLLM / LM Studio require it.
|
||||
"""
|
||||
from core.prefs import set_ as prefs_set, delete as prefs_delete
|
||||
|
||||
mapping = {
|
||||
"TRANSLATE_BASE_URL": body.base_url,
|
||||
"TRANSLATE_MODEL": body.model,
|
||||
"TRANSLATE_API_KEY": body.api_key,
|
||||
}
|
||||
for env_key, val in mapping.items():
|
||||
if val is None:
|
||||
continue # untouched
|
||||
val = val.strip()
|
||||
if val:
|
||||
os.environ[env_key] = val
|
||||
prefs_set(f"env.{env_key}", val)
|
||||
else:
|
||||
os.environ.pop(env_key, None)
|
||||
prefs_delete(f"env.{env_key}")
|
||||
# get_active_llm_backend() builds a fresh backend (and its OpenAI client
|
||||
# reads env at construction) on every call, so there's no singleton to
|
||||
# invalidate — the next translate/refine picks up the new values.
|
||||
return _llm_endpoint_state()
|
||||
|
||||
|
||||
# ── License acceptance (Phase 3 Plan 03-01 / TTS-05) ──────────────────────
|
||||
# Frontend ``SupertonicLicenseDialog`` flips the engine-license bit via this
|
||||
# endpoint. The handler is loopback-gated (router-level dep) and the
|
||||
@@ -395,60 +284,3 @@ def set_models_dir(body: _ModelsDirBody):
|
||||
|
||||
user_env.set_user_env(_MODELS_DIR_ENV, path)
|
||||
return {"configured": path, "effective": _effective_models_dir(), "restart_required": True}
|
||||
|
||||
|
||||
# ── HF mirror endpoint (parity program Wave 4.3 / §R4 c) ──────────────────
|
||||
# Restricted-network users (e.g. behind the Great Firewall) need to point
|
||||
# huggingface_hub at a mirror. HF reads HF_ENDPOINT at import time, so a
|
||||
# change takes effect on the next backend start — persisted to the durable
|
||||
# per-user env so it survives Tauri/Finder launches that don't inherit a
|
||||
# shell. Loopback-gated via the router dep.
|
||||
|
||||
_HF_ENDPOINT_ENV = "HF_ENDPOINT"
|
||||
|
||||
# A few well-known mirrors, surfaced as quick-picks in the UI. hf-mirror.com
|
||||
# is the community mirror most-used in China; the official endpoint clears it.
|
||||
_HF_MIRROR_PRESETS = [
|
||||
{"label": "Hugging Face (official)", "url": ""},
|
||||
{"label": "hf-mirror.com (community, China)", "url": "https://hf-mirror.com"},
|
||||
]
|
||||
|
||||
|
||||
class _HFMirrorBody(BaseModel):
|
||||
url: str = Field("", description="HF_ENDPOINT URL; empty string clears it (official endpoint)")
|
||||
|
||||
|
||||
@router.get("/hf-mirror")
|
||||
def get_hf_mirror():
|
||||
from core import user_env
|
||||
|
||||
configured = user_env.get_user_env(_HF_ENDPOINT_ENV) or ""
|
||||
return {
|
||||
# The value that will apply after restart (persisted), and what's
|
||||
# live in this process (env may differ until then).
|
||||
"configured": configured,
|
||||
"effective": os.environ.get(_HF_ENDPOINT_ENV, ""),
|
||||
"presets": _HF_MIRROR_PRESETS,
|
||||
}
|
||||
|
||||
|
||||
@router.put("/hf-mirror")
|
||||
def set_hf_mirror(body: _HFMirrorBody):
|
||||
from core import user_env
|
||||
|
||||
url = (body.url or "").strip().rstrip("/")
|
||||
if url and not url.startswith(("http://", "https://")):
|
||||
raise HTTPException(status_code=400, detail="Mirror URL must start with http(s)://")
|
||||
try:
|
||||
if url:
|
||||
user_env.set_user_env(_HF_ENDPOINT_ENV, url)
|
||||
os.environ[_HF_ENDPOINT_ENV] = url # best-effort for new downloads this session
|
||||
else:
|
||||
user_env.unset_user_env(_HF_ENDPOINT_ENV)
|
||||
os.environ.pop(_HF_ENDPOINT_ENV, None)
|
||||
except Exception:
|
||||
logger.exception("set_hf_mirror failed")
|
||||
raise HTTPException(status_code=500, detail="Failed to persist mirror setting")
|
||||
# HF endpoint is read at import time by huggingface_hub, so the override
|
||||
# is only guaranteed once the backend restarts.
|
||||
return {"configured": url, "restart_required": True, "presets": _HF_MIRROR_PRESETS}
|
||||
|
||||
@@ -11,16 +11,13 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from core import prefs
|
||||
from utils import hf_progress
|
||||
from utils import download_aggregator
|
||||
from .models import KNOWN_MODELS, invalidate_cache
|
||||
|
||||
logger = logging.getLogger("omnivoice.setup.download")
|
||||
@@ -29,177 +26,6 @@ router = APIRouter()
|
||||
# Cooldown: prevent rapid re-install after a failure. Maps repo_id → last_fail_time.
|
||||
_install_cooldowns: dict[str, float] = {}
|
||||
_COOLDOWN_SECS = 60.0
|
||||
# Evict cooldown entries older than this so the dict can't grow unbounded across
|
||||
# a long-lived process (MM2-06). Anything past the cooldown window is dead state.
|
||||
_COOLDOWN_TTL_SECS = 3600.0
|
||||
|
||||
|
||||
def _sweep_cooldowns(now: float) -> None:
|
||||
"""Drop cooldown entries older than the TTL (MM2-06). Keeps the dict bounded
|
||||
— without this it accumulated one entry per ever-failed repo forever."""
|
||||
stale = [k for k, t in _install_cooldowns.items() if (now - t) > _COOLDOWN_TTL_SECS]
|
||||
for k in stale:
|
||||
_install_cooldowns.pop(k, None)
|
||||
|
||||
# Repo_ids the user asked to cancel (FDL-11). Checked between retry attempts.
|
||||
# Note: a single in-flight snapshot_download/Xet fetch is not interruptible
|
||||
# mid-file in hf_hub 1.7.2 — cancel stops further retries, marks the row
|
||||
# cancelled, and clears the cooldown so a cancel isn't rate-limited.
|
||||
_cancelled: set[str] = set()
|
||||
|
||||
|
||||
def _download_max_workers() -> int:
|
||||
"""Parallel-FILES worker count for snapshot_download (FDL-02). Default 8 —
|
||||
don't crank it: Xet already parallelises *within* each file via concurrent
|
||||
byte-range gets, so a high count just multiplies buffer pressure. Override
|
||||
via prefs / OMNIVOICE_DOWNLOAD_MAX_WORKERS for power users."""
|
||||
raw = prefs.resolve("download_max_workers", env="OMNIVOICE_DOWNLOAD_MAX_WORKERS", default=8)
|
||||
try:
|
||||
return max(1, int(raw))
|
||||
except (TypeError, ValueError):
|
||||
return 8
|
||||
|
||||
|
||||
def _download_endpoint() -> "str | None":
|
||||
"""Optional HF endpoint override (FDL-10 mirror path, opt-in). Returned as a
|
||||
per-call ``endpoint=`` rather than a process-wide HF_ENDPOINT mutation. A
|
||||
mirror routes through the classic LFS path (no Xet) — documented in
|
||||
docs/downloading-models.md."""
|
||||
ep = prefs.resolve("hf_endpoint", env="HF_ENDPOINT", default=None)
|
||||
return ep or None
|
||||
|
||||
|
||||
def apply_xet_env() -> None:
|
||||
"""Apply opt-in Xet tuning knobs to the environment before a download
|
||||
(FDL-04). Both default OFF; env wins over the prefs store. high-performance
|
||||
can *hurt* low-RAM machines (needs lots of RAM/bandwidth); HDD-sequential
|
||||
avoids parallel-write thrash on spinning disks. Idempotent."""
|
||||
import os as _os
|
||||
high_perf = prefs.resolve("xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False)
|
||||
if _truthy(high_perf):
|
||||
_os.environ["HF_XET_HIGH_PERFORMANCE"] = "1"
|
||||
hdd_seq = prefs.resolve("xet_hdd_sequential_write", env="HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY", default=False)
|
||||
if _truthy(hdd_seq):
|
||||
_os.environ["HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY"] = "1"
|
||||
|
||||
|
||||
def _truthy(v) -> bool:
|
||||
if isinstance(v, bool):
|
||||
return v
|
||||
return str(v).strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
class _InstallCancelled(Exception):
|
||||
"""Raised inside the install worker when the user cancels (FDL-11)."""
|
||||
|
||||
|
||||
def compute_plan(plan_files) -> dict:
|
||||
"""Summarise a snapshot_download(dry_run=True) result into the install_plan
|
||||
payload (FDL-05): total bytes, bytes already cached (skipped), bytes that
|
||||
will actually download, and file counts. ``will_download`` defaults to
|
||||
``not is_cached`` for forward-compat with older DryRunFileInfo shapes."""
|
||||
total = sum(int(getattr(f, "file_size", 0) or 0) for f in plan_files)
|
||||
cached = sum(
|
||||
int(getattr(f, "file_size", 0) or 0)
|
||||
for f in plan_files if getattr(f, "is_cached", False)
|
||||
)
|
||||
will = [
|
||||
f for f in plan_files
|
||||
if getattr(f, "will_download", not getattr(f, "is_cached", False))
|
||||
]
|
||||
to_dl = sum(int(getattr(f, "file_size", 0) or 0) for f in will)
|
||||
n_files = len(plan_files)
|
||||
n_cached = sum(1 for f in plan_files if getattr(f, "is_cached", False))
|
||||
return {
|
||||
"total_bytes": total,
|
||||
"cached_bytes": cached,
|
||||
"to_download_bytes": to_dl,
|
||||
"n_files": n_files,
|
||||
"n_cached": n_cached,
|
||||
}
|
||||
|
||||
|
||||
def _segmented_enabled() -> bool:
|
||||
"""Opt-in IDM-style accelerator (FDL-09), default OFF. Most useful when Xet
|
||||
is inactive (the app's default): the legacy-LFS path is single-stream, so
|
||||
this restores parallel speed AND gives real live byte progress."""
|
||||
return _truthy(prefs.resolve(
|
||||
"segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=False,
|
||||
))
|
||||
|
||||
|
||||
def _xet_active() -> bool:
|
||||
"""True only when hf_xet is installed AND not disabled. The app sets
|
||||
HF_HUB_DISABLE_XET=1 by default, so this is normally False — which is when
|
||||
the segmented accelerator pays off."""
|
||||
import importlib.util
|
||||
if importlib.util.find_spec("hf_xet") is None:
|
||||
return False
|
||||
return os.environ.get("HF_HUB_DISABLE_XET", "").strip().lower() not in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _repo_cancelled(repo_id: str) -> bool:
|
||||
return repo_id in _cancelled
|
||||
|
||||
|
||||
def _segmented_snapshot(repo_id: str, *, endpoint: "str | None") -> str:
|
||||
"""Fetch every file of a repo via the segmented downloader into the HF
|
||||
cache, mirroring hf_hub_download's blob+snapshot+refs layout so the result
|
||||
is indistinguishable from snapshot_download (FDL-09) — keeping /models
|
||||
install-state, is_cached, and delete working. Feeds real bytes to the
|
||||
aggregator. Raises on any error; the caller falls back to snapshot_download.
|
||||
"""
|
||||
import asyncio as _asyncio
|
||||
from huggingface_hub import HfApi, constants as _C
|
||||
from huggingface_hub.file_download import (
|
||||
hf_hub_url, get_hf_file_metadata, repo_folder_name, _create_symlink,
|
||||
)
|
||||
from services.segmented_download import segmented_download
|
||||
from services.token_resolver import resolve as _resolve_token
|
||||
|
||||
token = _resolve_token()
|
||||
api = HfApi(endpoint=endpoint, token=token)
|
||||
info = api.repo_info(repo_id, repo_type="model")
|
||||
commit = info.sha
|
||||
files = [s.rfilename for s in (info.siblings or [])]
|
||||
if not commit or not files:
|
||||
raise RuntimeError("repo_info returned no commit/siblings")
|
||||
|
||||
repo_dir = os.path.join(_C.HF_HUB_CACHE, repo_folder_name(repo_id=repo_id, repo_type="model"))
|
||||
blobs_dir = os.path.join(repo_dir, "blobs")
|
||||
snap_dir = os.path.join(repo_dir, "snapshots", commit)
|
||||
refs_dir = os.path.join(repo_dir, "refs")
|
||||
for d in (blobs_dir, snap_dir, refs_dir):
|
||||
os.makedirs(d, exist_ok=True)
|
||||
|
||||
for rel in files:
|
||||
if _repo_cancelled(repo_id):
|
||||
raise _InstallCancelled()
|
||||
url = hf_hub_url(repo_id, rel, endpoint=endpoint, revision=commit)
|
||||
meta = get_hf_file_metadata(url, token=token)
|
||||
etag = (meta.etag or "").strip('"')
|
||||
if not etag:
|
||||
raise RuntimeError(f"no etag for {rel}")
|
||||
blob_path = os.path.join(blobs_dir, etag)
|
||||
pointer = os.path.join(snap_dir, rel)
|
||||
os.makedirs(os.path.dirname(pointer), exist_ok=True)
|
||||
if not os.path.exists(blob_path):
|
||||
_asyncio.run(segmented_download(
|
||||
meta.location or url, blob_path,
|
||||
token=token, expected_size=meta.size, expected_etag=etag,
|
||||
on_bytes=lambda d, k=rel: download_aggregator.add_bytes(repo_id, k, d),
|
||||
cancel_check=lambda: _repo_cancelled(repo_id),
|
||||
))
|
||||
if not os.path.lexists(pointer):
|
||||
_create_symlink(blob_path, pointer, new_blob=True)
|
||||
|
||||
# refs/main → commit so scan_cache_dir maps the revision correctly.
|
||||
try:
|
||||
with open(os.path.join(refs_dir, "main"), "w") as f:
|
||||
f.write(commit)
|
||||
except OSError:
|
||||
pass
|
||||
return snap_dir
|
||||
|
||||
|
||||
# ── SSE Download Stream ───────────────────────────────────────────────────
|
||||
@@ -216,66 +42,6 @@ def _safe_put(queue: asyncio.Queue, event) -> None:
|
||||
pass
|
||||
|
||||
|
||||
# Minimum size for "this snapshot actually contains model weights". An
|
||||
# interrupted snapshot_download can leave config/tokenizer files but no
|
||||
# weights; the install then looks complete and synthesis later fails with
|
||||
# "does not appear to have a file named pytorch_model.bin or
|
||||
# model.safetensors" (#352). 5 MB clears every weight format we ship
|
||||
# (safetensors/bin shards, onnx, pt, gguf) without false-positiving on
|
||||
# config-only aux repos.
|
||||
_MIN_WEIGHT_BYTES = 5 * 1024 * 1024
|
||||
|
||||
# Per-role weight-file floors (MM2-07). A valid model has at least one
|
||||
# recognized weight file at or above its extension's floor. ONNX graphs are
|
||||
# legitimately small (a complete model can be well under 5 MB), so a single
|
||||
# 5 MB rule false-positives on them as "truncated" (#352 over-trigger); give
|
||||
# .onnx a lower floor while still rejecting a 0/KB partial. Tensor formats keep
|
||||
# the original 5 MB floor.
|
||||
_WEIGHT_FLOORS = {
|
||||
".safetensors": _MIN_WEIGHT_BYTES,
|
||||
".bin": _MIN_WEIGHT_BYTES,
|
||||
".ckpt": _MIN_WEIGHT_BYTES,
|
||||
".pt": _MIN_WEIGHT_BYTES,
|
||||
".pth": _MIN_WEIGHT_BYTES,
|
||||
".gguf": _MIN_WEIGHT_BYTES,
|
||||
".onnx": 64 * 1024, # a real ONNX graph is ≥ tens of KB; a truncated one is bytes
|
||||
}
|
||||
|
||||
|
||||
def _validate_snapshot_has_weights(repo_id: str, snapshot_path: str) -> None:
|
||||
"""Raise OSError when a finished snapshot has no plausible weight file —
|
||||
surfaces the truncated-download class (#352) at install time, where the
|
||||
retry loop and the UI's re-download path can deal with it, instead of at
|
||||
first synthesis with an opaque transformers error.
|
||||
|
||||
A snapshot is valid if it contains a recognized weight file meeting its
|
||||
per-extension floor (MM2-07) OR any file ≥ the global 5 MB floor (the
|
||||
original lenient catch — kept so this is never stricter than before)."""
|
||||
try:
|
||||
biggest = 0
|
||||
for root, _dirs, files in os.walk(snapshot_path, followlinks=True):
|
||||
for f in files:
|
||||
try:
|
||||
size = os.path.getsize(os.path.join(root, f))
|
||||
except OSError:
|
||||
continue
|
||||
biggest = max(biggest, size)
|
||||
ext = os.path.splitext(f)[1].lower()
|
||||
floor = _WEIGHT_FLOORS.get(ext)
|
||||
if floor is not None and size >= floor:
|
||||
return # a recognized weight file of plausible size
|
||||
if size >= _MIN_WEIGHT_BYTES:
|
||||
return # original lenient catch (non-standard weight names)
|
||||
except OSError:
|
||||
return # can't inspect — don't block the install on the checker itself
|
||||
raise OSError(
|
||||
f"{repo_id}: download finished but no model weights were found in the "
|
||||
"snapshot (largest file "
|
||||
f"{biggest} bytes). The download was likely interrupted — delete the "
|
||||
"model in Settings → Models and install it again."
|
||||
)
|
||||
|
||||
|
||||
@router.get("/setup/download-stream")
|
||||
async def setup_download_stream():
|
||||
"""SSE: forward every HuggingFace download tqdm update as a JSON event."""
|
||||
@@ -332,7 +98,6 @@ async def install_model(req: InstallModelRequest):
|
||||
)
|
||||
# Cooldown guard — don't retry if the same model just failed.
|
||||
import time as _time_check
|
||||
_sweep_cooldowns(_time_check.time()) # bound the dict (MM2-06)
|
||||
last_fail = _install_cooldowns.get(req.repo_id)
|
||||
if last_fail and (_time_check.time() - last_fail) < _COOLDOWN_SECS:
|
||||
remaining = int(_COOLDOWN_SECS - (_time_check.time() - last_fail))
|
||||
@@ -347,7 +112,6 @@ async def install_model(req: InstallModelRequest):
|
||||
|
||||
def _do():
|
||||
token = hf_progress.current_repo_id.set(req.repo_id)
|
||||
_cancelled.discard(req.repo_id) # clear any stale cancel from a prior run
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
@@ -361,22 +125,7 @@ async def install_model(req: InstallModelRequest):
|
||||
LocalEntryNotFoundError,
|
||||
)
|
||||
logger.info("model install starting: %s", req.repo_id)
|
||||
# Apply opt-in Xet tuning knobs (high-perf / HDD) before downloading.
|
||||
apply_xet_env()
|
||||
# Drive snapshot_download explicitly (FDL-02): pass our progress-
|
||||
# emitting tqdm subclass so progress is deterministic + Xet-aware
|
||||
# (Xet feeds bytes into whatever tqdm_class is supplied), bound the
|
||||
# parallel-files worker count, and honour an optional mirror endpoint.
|
||||
dl_kwargs: dict = {
|
||||
"repo_id": req.repo_id,
|
||||
"max_workers": _download_max_workers(),
|
||||
}
|
||||
_tqdm_cls = hf_progress.tracked_tqdm_class()
|
||||
if _tqdm_cls is not None:
|
||||
dl_kwargs["tqdm_class"] = _tqdm_cls
|
||||
_endpoint = _download_endpoint()
|
||||
if _endpoint:
|
||||
dl_kwargs["endpoint"] = _endpoint
|
||||
dl_kwargs: dict = {"repo_id": req.repo_id}
|
||||
if sys.platform == "win32":
|
||||
dl_kwargs["local_dir_use_symlinks"] = False
|
||||
|
||||
@@ -404,70 +153,12 @@ async def install_model(req: InstallModelRequest):
|
||||
hb = threading.Thread(target=_heartbeat, daemon=True)
|
||||
hb.start()
|
||||
|
||||
# Pre-flight (FDL-05): a dry-run resolve gives the UI an accurate
|
||||
# denominator — total bytes, bytes already cached (skipped), and the
|
||||
# bytes that will actually download — BEFORE any byte flows. Seeds
|
||||
# the overall aggregator so its bar/ETA are correct from the first
|
||||
# event. Degrades gracefully (totals=None) on older/gated repos.
|
||||
_preflight_kwargs = {"repo_id": req.repo_id, "dry_run": True}
|
||||
if _endpoint:
|
||||
_preflight_kwargs["endpoint"] = _endpoint
|
||||
try:
|
||||
_plan = snapshot_download(**_preflight_kwargs)
|
||||
_summary = compute_plan(_plan)
|
||||
download_aggregator.start(
|
||||
req.repo_id,
|
||||
total_bytes=_summary["to_download_bytes"],
|
||||
files_total=max(0, _summary["n_files"] - _summary["n_cached"]),
|
||||
)
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
"phase": "install_plan",
|
||||
**_summary,
|
||||
})
|
||||
except Exception as _pf_err:
|
||||
# No preflight (older/gated repo, mirror without dry-run, etc.):
|
||||
# fall back to today's fill-in-as-files-appear behaviour.
|
||||
logger.info("model install %s: preflight unavailable (%s)", req.repo_id, _pf_err)
|
||||
download_aggregator.start(req.repo_id)
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
"phase": "install_plan",
|
||||
"total_bytes": None,
|
||||
"cached_bytes": None,
|
||||
"to_download_bytes": None,
|
||||
"n_files": None,
|
||||
"n_cached": None,
|
||||
})
|
||||
|
||||
_max_attempts = 5
|
||||
_attempt = 0
|
||||
while True:
|
||||
if req.repo_id in _cancelled:
|
||||
raise _InstallCancelled()
|
||||
_attempt += 1
|
||||
try:
|
||||
# Opt-in segmented accelerator (FDL-09): parallel byte-range
|
||||
# fetch with real live progress, for the legacy-LFS path.
|
||||
# Any failure falls through to snapshot_download — the
|
||||
# accelerator can never compromise a correct install.
|
||||
_snapshot_path = None
|
||||
if _attempt == 1 and _segmented_enabled() and not _xet_active():
|
||||
try:
|
||||
_snapshot_path = _segmented_snapshot(req.repo_id, endpoint=_endpoint)
|
||||
except _InstallCancelled:
|
||||
raise
|
||||
except Exception as _seg_err:
|
||||
logger.info(
|
||||
"segmented download for %s failed (%s); falling back to snapshot_download",
|
||||
req.repo_id, _seg_err,
|
||||
)
|
||||
_snapshot_path = None
|
||||
if _snapshot_path is None:
|
||||
_snapshot_path = snapshot_download(**dl_kwargs)
|
||||
_validate_snapshot_has_weights(req.repo_id, _snapshot_path)
|
||||
snapshot_download(**dl_kwargs)
|
||||
break
|
||||
except (HfHubHTTPError, LocalEntryNotFoundError, OSError) as net_err:
|
||||
if _attempt >= _max_attempts:
|
||||
@@ -488,10 +179,6 @@ async def install_model(req: InstallModelRequest):
|
||||
_t.sleep(_backoff)
|
||||
# Stop heartbeat once download completes
|
||||
_resolving.set()
|
||||
# Flush the overall bar to 100% with the true byte total (FDL-06):
|
||||
# under Xet the per-file byte bars don't surface completion, so the
|
||||
# aggregator can sit below 100% even though every file landed.
|
||||
download_aggregator.complete(req.repo_id)
|
||||
logger.info("model install done: %s", req.repo_id)
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
@@ -499,19 +186,7 @@ async def install_model(req: InstallModelRequest):
|
||||
"downloaded": 0, "total": 0, "pct": 1.0,
|
||||
"phase": "install_done",
|
||||
})
|
||||
_install_cooldowns.pop(req.repo_id, None) # success clears any cooldown (MM2-06)
|
||||
invalidate_cache()
|
||||
except _InstallCancelled:
|
||||
_resolving.set()
|
||||
logger.info("model install cancelled: %s", req.repo_id)
|
||||
# A cancel is user intent, not a failure — don't set a cooldown.
|
||||
_install_cooldowns.pop(req.repo_id, None)
|
||||
hf_progress.emit({
|
||||
"repo_id": req.repo_id,
|
||||
"filename": req.repo_id,
|
||||
"downloaded": 0, "total": 0, "pct": 0.0,
|
||||
"phase": "install_cancelled",
|
||||
})
|
||||
except Exception as e:
|
||||
_resolving.set()
|
||||
logger.info("model install failed for %s: %s", req.repo_id, e)
|
||||
@@ -525,28 +200,12 @@ async def install_model(req: InstallModelRequest):
|
||||
"error": str(e),
|
||||
})
|
||||
finally:
|
||||
_cancelled.discard(req.repo_id)
|
||||
download_aggregator.finish(req.repo_id)
|
||||
hf_progress.current_repo_id.reset(token)
|
||||
|
||||
loop.create_task(asyncio.to_thread(_do))
|
||||
return {"status": "install_started", "repo_id": req.repo_id}
|
||||
|
||||
|
||||
@router.post("/models/install/cancel")
|
||||
async def cancel_install(req: InstallModelRequest):
|
||||
"""Request cancellation of an in-flight install (FDL-11).
|
||||
|
||||
Best-effort: stops further retry attempts and marks the row cancelled. A
|
||||
single in-flight snapshot_download/Xet fetch isn't interruptible mid-file
|
||||
in hf_hub 1.7.2, so an already-streaming file finishes; the cancel takes
|
||||
effect at the next retry boundary. Clears the cooldown so the user can
|
||||
immediately restart."""
|
||||
_cancelled.add(req.repo_id)
|
||||
_install_cooldowns.pop(req.repo_id, None)
|
||||
return {"cancelling": req.repo_id}
|
||||
|
||||
|
||||
# ── Delete ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@router.delete("/models/{repo_id:path}")
|
||||
|
||||
@@ -220,12 +220,8 @@ def is_cached(repo_id: str) -> bool:
|
||||
except Exception as e:
|
||||
# scan_cache_dir can raise on Windows (WinError 448 'untrusted mount
|
||||
# point'); fall back to a direct disk check so a cached model isn't
|
||||
# mistaken for missing and re-downloaded in a loop (#117/#118). Logged
|
||||
# at WARNING with the exception type (MM2-09) so this fallback isn't
|
||||
# invisible when triaging a Windows cache report — it previously logged
|
||||
# at DEBUG and never showed at the default level.
|
||||
logger.warning("is_cached: scan_cache_dir failed (%s: %s); using on-disk fallback for %s",
|
||||
type(e).__name__, e, repo_id)
|
||||
# mistaken for missing and re-downloaded in a loop (#117/#118).
|
||||
logger.debug("scan_cache_dir failed (%s); using disk fallback", e)
|
||||
return _is_cached_on_disk(repo_id)
|
||||
|
||||
|
||||
|
||||
@@ -373,46 +373,6 @@ def preflight():
|
||||
"status": gpu_status, "detail": gpu_detail, "fix": gpu_fix,
|
||||
})
|
||||
|
||||
# ── GPU routing for the ACTIVE TTS engine (#21 — no silent CPU fallback).
|
||||
# Distinct from the hardware "gpu" check above: this asks "will the engine
|
||||
# the user actually selected use that GPU on this host?" Built from the same
|
||||
# canonical probe + resolver the Engine Compatibility Matrix uses.
|
||||
try:
|
||||
from services.tts_backend import gpu_routing_verdict
|
||||
gpu_routing = gpu_routing_verdict()
|
||||
except Exception as exc: # never break preflight on a routing hiccup
|
||||
logger.warning("preflight gpu_routing failed: %s", exc)
|
||||
gpu_routing = None
|
||||
if gpu_routing:
|
||||
_rs = gpu_routing.get("routing_status")
|
||||
_eng = gpu_routing.get("engine") or "active engine"
|
||||
_dev = gpu_routing.get("effective_device") or "?"
|
||||
_why = gpu_routing.get("routing_reason")
|
||||
if _rs == "accelerated" and not _why:
|
||||
r_status, r_detail, r_fix = "pass", f"{_eng} → {_dev} (accelerated)", None
|
||||
elif _rs == "accelerated": # driver/arch caveat
|
||||
r_status, r_detail, r_fix = "warn", f"{_eng} → {_dev}: {_why}", (
|
||||
"GPU selected but may fail at kernel launch — update drivers / "
|
||||
"reinstall torch for this GPU architecture.")
|
||||
elif _rs == "cpu_fallback":
|
||||
r_status, r_detail, r_fix = "warn", (
|
||||
f"{_eng} runs on CPU here: {_why or 'no GPU path for this host'}"), (
|
||||
"Pick an engine that supports this host's GPU for a speedup, or "
|
||||
"continue on CPU (slower).")
|
||||
elif _rs == "cpu_only":
|
||||
r_status, r_detail, r_fix = "pass", f"{_eng} → cpu (no accelerator on this host)", None
|
||||
elif _rs == "unavailable":
|
||||
r_status, r_detail, r_fix = "fail", (
|
||||
f"{_eng} can't run on this host: {_why or 'needs a GPU this machine lacks'}"), (
|
||||
"Select an engine with a CPU path in Settings → Engines.")
|
||||
else: # "none" / unknown
|
||||
r_status, r_detail, r_fix = "warn", "No active TTS engine resolved for routing.", (
|
||||
"Pick an engine in Settings → Engines.")
|
||||
checks.append({
|
||||
"id": "gpu_routing", "label": "Active engine routing",
|
||||
"status": r_status, "detail": r_detail, "fix": r_fix,
|
||||
})
|
||||
|
||||
# ── Network
|
||||
net_ok = _probe_network()
|
||||
checks.append({
|
||||
@@ -440,13 +400,9 @@ def preflight():
|
||||
"gpu_available": gpu["available"],
|
||||
"gpu_driver": gpu["driver"],
|
||||
"gpu_device_name": gpu["device_name"],
|
||||
# Canonical probe (distinguishes ROCm from CUDA):
|
||||
"gpu_family": (gpu_routing or {}).get("host_family", "cpu"),
|
||||
"vram_gb": (gpu_routing or {}).get("vram_gb", 0.0),
|
||||
"ram_gb": round(ram, 1),
|
||||
"disk_free_gb": round(free, 1),
|
||||
},
|
||||
"gpu_routing": gpu_routing,
|
||||
}
|
||||
|
||||
|
||||
|
||||
+85
-120
@@ -107,52 +107,6 @@ def _ui_port() -> int:
|
||||
return 3901
|
||||
|
||||
|
||||
def _fast_download_status() -> dict:
|
||||
"""Report the download-acceleration state for the Settings UI (FDL-03).
|
||||
|
||||
Reports the *runtime* truth, not just whether hf_xet is importable. The app
|
||||
currently sets ``HF_HUB_DISABLE_XET=1`` by default (main.py) — Xet's chunked
|
||||
transfer is fast but its progress bypasses our tqdm patch, so the legacy-LFS
|
||||
path is forced to keep accurate byte progress. So:
|
||||
|
||||
* ``xet_installed`` — hf_xet present
|
||||
* ``xet_active`` — installed AND not disabled via HF_HUB_DISABLE_XET
|
||||
* ``xet_enabled`` — alias of xet_active (what the UI badge keys off)
|
||||
|
||||
Must never throw: /system/info is called on every Settings load.
|
||||
"""
|
||||
installed = False
|
||||
version = None
|
||||
try:
|
||||
import hf_xet # noqa: F401
|
||||
installed = True
|
||||
try:
|
||||
from importlib.metadata import version as _ver
|
||||
version = _ver("hf-xet")
|
||||
except Exception:
|
||||
version = None
|
||||
except Exception:
|
||||
installed = False
|
||||
disabled = str(os.environ.get("HF_HUB_DISABLE_XET", "")).strip().lower() in {"1", "true", "yes", "on"}
|
||||
active = installed and not disabled
|
||||
try:
|
||||
from core import prefs
|
||||
high_perf = prefs.resolve(
|
||||
"xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False
|
||||
)
|
||||
high_perf = high_perf if isinstance(high_perf, bool) else \
|
||||
str(high_perf).strip().lower() in {"1", "true", "yes", "on"}
|
||||
except Exception:
|
||||
high_perf = False
|
||||
return {
|
||||
"xet_installed": installed,
|
||||
"xet_active": active,
|
||||
"xet_enabled": active, # UI badge: only true when Xet actually runs
|
||||
"xet_version": version,
|
||||
"high_performance": bool(high_perf),
|
||||
}
|
||||
|
||||
|
||||
def _has_hf_token() -> bool:
|
||||
# Phase 1 AUTH-01..06 cascade. Delegates to the 3-source resolver
|
||||
# (App → Env → HF-CLI) instead of reading env/HF-CLI directly. This
|
||||
@@ -174,23 +128,88 @@ def model_status():
|
||||
|
||||
@router.get("/model/loaded")
|
||||
def loaded_models():
|
||||
"""List all currently loaded models for the flush dropdown (MM2-04).
|
||||
Thin delegation to the model_lifecycle facade — shape unchanged:
|
||||
``{models, count}``."""
|
||||
from services import model_lifecycle
|
||||
return model_lifecycle.list_loaded()
|
||||
"""Return details about all currently loaded models for the flush dropdown.
|
||||
|
||||
Returns a list of models with name, type, device, and estimated VRAM usage.
|
||||
"""
|
||||
import services.model_manager as mm
|
||||
|
||||
models = []
|
||||
|
||||
# 1. TTS model (OmniVoice)
|
||||
if mm.model is not None:
|
||||
device = "unknown"
|
||||
vram_mb = 0
|
||||
try:
|
||||
device = str(next(mm.model.parameters()).device) if hasattr(mm.model, 'parameters') else get_best_device()
|
||||
except Exception:
|
||||
device = get_best_device()
|
||||
try:
|
||||
torch = mm._lazy_torch()
|
||||
if torch.cuda.is_available():
|
||||
vram_mb = torch.cuda.memory_allocated() / (1024 ** 2)
|
||||
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
driver = getattr(torch.mps, "driver_allocated_memory", None)
|
||||
if driver:
|
||||
vram_mb = driver() / (1024 ** 2)
|
||||
except Exception:
|
||||
pass
|
||||
models.append({
|
||||
"id": "tts",
|
||||
"name": "OmniVoice TTS",
|
||||
"checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
|
||||
"device": device,
|
||||
"vram_mb": round(vram_mb, 1),
|
||||
"unloadable": True,
|
||||
})
|
||||
|
||||
# 2. ASR model (WhisperX)
|
||||
if mm.model is not None and hasattr(mm.model, '_asr_pipe') and mm.model._asr_pipe is not None:
|
||||
models.append({
|
||||
"id": "asr",
|
||||
"name": "WhisperX ASR",
|
||||
"checkpoint": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
|
||||
"device": "cpu",
|
||||
"vram_mb": 0,
|
||||
"unloadable": False, # tied to TTS model lifecycle
|
||||
})
|
||||
|
||||
# 3. Diarization pipeline
|
||||
if mm._diar_pipeline is not None:
|
||||
models.append({
|
||||
"id": "diarization",
|
||||
"name": "Pyannote Diarization",
|
||||
"checkpoint": "pyannote/speaker-diarization-3.1",
|
||||
"device": get_best_device(),
|
||||
"vram_mb": 0,
|
||||
"unloadable": True,
|
||||
})
|
||||
|
||||
return {"models": models, "count": len(models)}
|
||||
|
||||
|
||||
@router.post("/model/unload/{model_id}")
|
||||
async def unload_model(model_id: str):
|
||||
"""Unload a specific model by id (MM2-04). Delegates to model_lifecycle;
|
||||
an unknown id maps to HTTP 400. ``tts`` | ``diarization`` |
|
||||
``sidecar:<id>`` | ``sidecars``."""
|
||||
from services import model_lifecycle
|
||||
try:
|
||||
return await model_lifecycle.unload(model_id)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
"""Unload a specific model by ID."""
|
||||
import services.model_manager as mm
|
||||
|
||||
if model_id == "tts":
|
||||
async with mm._model_lock:
|
||||
if mm.model is not None:
|
||||
mm.model = None
|
||||
mm.free_vram()
|
||||
return {"unloaded": "tts", "success": True}
|
||||
return {"unloaded": "tts", "success": False, "reason": "not loaded"}
|
||||
|
||||
elif model_id == "diarization":
|
||||
if mm._diar_pipeline is not None:
|
||||
mm._diar_pipeline = None
|
||||
mm.free_vram()
|
||||
return {"unloaded": "diarization", "success": True}
|
||||
return {"unloaded": "diarization", "success": False, "reason": "not loaded"}
|
||||
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail=f"Unknown model id: {model_id}")
|
||||
|
||||
|
||||
@router.get("/system/info", response_model=SystemInfoResponse)
|
||||
@@ -212,7 +231,6 @@ def system_info():
|
||||
"asr_model": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
|
||||
"translate_provider": os.environ.get("TRANSLATE_PROVIDER", "google"),
|
||||
"has_hf_token": _has_hf_token(),
|
||||
"fast_download": _fast_download_status(),
|
||||
"device": get_best_device(),
|
||||
"python": sys.version.split()[0],
|
||||
"platform": sys.platform,
|
||||
@@ -304,10 +322,9 @@ def _tauri_log_candidates():
|
||||
os.path.join(home, "Library/Logs/OmniVoice/backend_err.log"),
|
||||
]
|
||||
if sys.platform.startswith("linux"):
|
||||
data_dir = os.environ.get("XDG_DATA_HOME") or os.path.join(home, ".local/share")
|
||||
state_dir = os.environ.get("XDG_STATE_HOME") or os.path.join(home, ".local/state")
|
||||
return [
|
||||
os.path.join(data_dir, bid, "logs", "tauri.log"),
|
||||
os.path.join(home, ".local/share", bid, "logs", "tauri.log"),
|
||||
os.path.join(home, ".config", bid, "logs", "tauri.log"),
|
||||
os.path.join(state_dir, "OmniVoice", "backend.log"),
|
||||
os.path.join(state_dir, "OmniVoice", "backend_err.log"),
|
||||
@@ -447,14 +464,6 @@ async def clear_system_logs():
|
||||
status_code=500,
|
||||
detail=f"Could not clear log at {p}: {e}. The file may be open in another process or read-only — close tailing tools and retry.",
|
||||
)
|
||||
if cleared_any:
|
||||
# The crash log just shrank to zero — drop any stored ack so a stale
|
||||
# byte count can't suppress the next 'crash-last-session' notice.
|
||||
for key in ("crash_log_acked", "crash_log_acked_size"):
|
||||
try:
|
||||
prefs_delete(key)
|
||||
except Exception:
|
||||
pass
|
||||
return {"cleared": cleared_any}
|
||||
|
||||
|
||||
@@ -556,21 +565,6 @@ async def flush_memory(unload_model: bool = False):
|
||||
|
||||
# ── Actionable notifications ──────────────────────────────────────────────
|
||||
|
||||
_GPU_ARCH_WARNING: "list[str | None]" = [] # [-1] = computed result
|
||||
|
||||
|
||||
def _gpu_arch_warning_cached() -> "str | None":
|
||||
"""check_device_compatibility() once per process (it lazy-imports torch —
|
||||
too heavy for the 30s notifications poll)."""
|
||||
if not _GPU_ARCH_WARNING:
|
||||
try:
|
||||
from services.model_manager import check_device_compatibility
|
||||
compatible, warning = check_device_compatibility()
|
||||
_GPU_ARCH_WARNING.append(None if compatible else warning)
|
||||
except Exception:
|
||||
_GPU_ARCH_WARNING.append(None)
|
||||
return _GPU_ARCH_WARNING[-1]
|
||||
|
||||
|
||||
@router.get("/system/notifications")
|
||||
def system_notifications():
|
||||
@@ -602,20 +596,6 @@ def system_notifications():
|
||||
},
|
||||
})
|
||||
|
||||
# 1b. GPU compute capability unsupported by this torch build (#284) —
|
||||
# the model "runs" but emits pure noise, the worst silent failure mode
|
||||
# (RTX 50-series Blackwell sm_120 on pre-cu128 wheels). The loader logs
|
||||
# this, but a log line never reached the affected users — surface it in
|
||||
# the panel. Checked once per process: it lazy-imports torch.
|
||||
gpu_warn = _gpu_arch_warning_cached()
|
||||
if gpu_warn:
|
||||
notes.append({
|
||||
"id": "gpu-arch-unsupported",
|
||||
"level": "error",
|
||||
"title": "GPU not supported by this PyTorch build",
|
||||
"message": gpu_warn + " Until then, output will be noise/garbage.",
|
||||
})
|
||||
|
||||
# 2. Missing ffmpeg
|
||||
ffmpeg_ok = False
|
||||
try:
|
||||
@@ -706,33 +686,18 @@ def _crashed_last_session() -> bool:
|
||||
if not os.path.exists(CRASH_LOG_PATH):
|
||||
return False
|
||||
size = os.path.getsize(CRASH_LOG_PATH)
|
||||
if size == 0:
|
||||
acked = int(prefs_get("crash_log_acked_size", 0) or 0)
|
||||
if size <= acked:
|
||||
return False
|
||||
mtime = os.path.getmtime(CRASH_LOG_PATH)
|
||||
# Composite ack (size + mtime): a bare byte count goes stale after the log
|
||||
# is truncated — the next crash log can stay smaller than the old acked
|
||||
# size forever, silently suppressing 'crash-last-session'. The ack only
|
||||
# holds while it still covers the file's current state.
|
||||
ack = prefs_get("crash_log_acked")
|
||||
if isinstance(ack, dict):
|
||||
if float(ack.get("mtime", 0) or 0) >= mtime and int(ack.get("size", 0) or 0) >= size:
|
||||
return False
|
||||
else:
|
||||
# Legacy size-only ack from older builds.
|
||||
if size <= int(prefs_get("crash_log_acked_size", 0) or 0):
|
||||
return False
|
||||
return mtime < _PROCESS_START_TS
|
||||
return os.path.getmtime(CRASH_LOG_PATH) < _PROCESS_START_TS
|
||||
|
||||
|
||||
@router.post("/system/crash/ack")
|
||||
async def ack_crash():
|
||||
"""Mark the current crash log as seen — dismisses the
|
||||
'crash-last-session' notification until the log changes again."""
|
||||
size = mtime = 0
|
||||
if os.path.exists(CRASH_LOG_PATH):
|
||||
size = os.path.getsize(CRASH_LOG_PATH)
|
||||
mtime = os.path.getmtime(CRASH_LOG_PATH)
|
||||
prefs_set("crash_log_acked", {"size": size, "mtime": mtime})
|
||||
'crash-last-session' notification until the log grows again."""
|
||||
size = os.path.getsize(CRASH_LOG_PATH) if os.path.exists(CRASH_LOG_PATH) else 0
|
||||
prefs_set("crash_log_acked_size", size)
|
||||
return {"acked_size": size}
|
||||
|
||||
|
||||
|
||||
@@ -98,30 +98,6 @@ async def ws_tts(websocket: WebSocket):
|
||||
model = await get_model()
|
||||
backend = get_active_tts_backend(model=model)
|
||||
|
||||
# ── Routing gate (#21 — no silent CPU fallback). WebSockets have
|
||||
# no response headers, so this uses frames: an error frame +
|
||||
# close on `unavailable`, a one-time `routing` frame on
|
||||
# cpu_fallback / accelerated-with-caveat (before any audio).
|
||||
from core.device_caps import detect_host_caps
|
||||
from services.engine_routing import resolve_routing, routing_notice
|
||||
from core.scrub import scrub_text
|
||||
_routing = resolve_routing(
|
||||
getattr(backend, "gpu_compat", ("cpu",)), detect_host_caps())
|
||||
if _routing["routing_status"] == "unavailable":
|
||||
await websocket.send_json({
|
||||
"type": "error",
|
||||
"detail": scrub_text(_routing["routing_reason"])
|
||||
or "engine cannot run on this host",
|
||||
})
|
||||
continue # don't stream; wait for the next request
|
||||
_notice = routing_notice(_routing)
|
||||
if _notice:
|
||||
await websocket.send_json({
|
||||
"type": "routing",
|
||||
"status": _notice[0],
|
||||
"reason": scrub_text(_notice[1]) if _notice[1] else None,
|
||||
})
|
||||
|
||||
# Build generation kwargs
|
||||
kw: dict = {"speed": data.get("speed", 1.0)}
|
||||
if data.get("language"):
|
||||
@@ -168,75 +144,50 @@ async def ws_tts(websocket: WebSocket):
|
||||
except Exception:
|
||||
kw["voice"] = voice
|
||||
|
||||
# Wave 1.4: split the request into sentences so the first
|
||||
# sentence's audio streams while later sentences are still
|
||||
# synthesizing — this is the time-to-first-audio win. The
|
||||
# chunker handles abbreviations/acronyms/decimals and CJK /
|
||||
# non-Latin terminators; single-sentence requests behave
|
||||
# exactly like the old single-shot path.
|
||||
from services.sentence_chunker import SentenceChunker
|
||||
_chunker = SentenceChunker(language=(data.get("language") or "en"))
|
||||
sentences = _chunker.push(text)
|
||||
sentences.extend(_chunker.flush())
|
||||
if not sentences:
|
||||
sentences = [text]
|
||||
|
||||
# Run generation in the GPU pool
|
||||
from services.model_manager import _gpu_pool
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
def _generate(sentence_text):
|
||||
def _generate():
|
||||
from services.audio_dsp import apply_mastering, normalize_audio
|
||||
wav = backend.generate(sentence_text, **kw)
|
||||
wav = backend.generate(text, **kw)
|
||||
sr_actual = backend.sample_rate
|
||||
# Like _run_tts in openai_compat: studio engines (VoxCPM2)
|
||||
# opt out of the broadcast mastering chain. This is the
|
||||
# other route that runs the active backend, so it needs the
|
||||
# same guard. Loudness normalisation still runs.
|
||||
if not getattr(backend, "applies_own_mastering", False):
|
||||
wav = apply_mastering(wav, sample_rate=sr_actual)
|
||||
wav = apply_mastering(wav, sample_rate=sr_actual)
|
||||
wav = normalize_audio(wav, target_dBFS=-2.0)
|
||||
return wav, sr_actual
|
||||
|
||||
wav_tensor, sr = await loop.run_in_executor(_gpu_pool, _generate)
|
||||
|
||||
# Send metadata after generation so sample_rate is real
|
||||
await websocket.send_json({
|
||||
"type": "start",
|
||||
"sample_rate": sr,
|
||||
"channels": 1,
|
||||
"format": "pcm16",
|
||||
"engine": backend.id,
|
||||
})
|
||||
|
||||
# Stream PCM16 chunks over the WebSocket
|
||||
import torch
|
||||
total_samples = 0
|
||||
sr = backend.sample_rate
|
||||
started = False
|
||||
# Convert to 16-bit PCM
|
||||
pcm = (wav_tensor * 32767).clamp(-32768, 32767).to(torch.int16)
|
||||
if pcm.ndim == 2:
|
||||
pcm = pcm[0] # mono
|
||||
pcm_bytes = pcm.numpy().tobytes()
|
||||
|
||||
for sentence in sentences:
|
||||
wav_tensor, sr = await loop.run_in_executor(
|
||||
_gpu_pool, _generate, sentence
|
||||
)
|
||||
total_samples = len(pcm)
|
||||
sent_samples = 0
|
||||
chunk_bytes = CHUNK_SAMPLES * 2 # 2 bytes per int16 sample
|
||||
|
||||
if not started:
|
||||
# Send metadata after the first generation so
|
||||
# sample_rate is real (lazy-loading engines report
|
||||
# their true rate only once weights are up).
|
||||
await websocket.send_json({
|
||||
"type": "start",
|
||||
"sample_rate": sr,
|
||||
"channels": 1,
|
||||
"format": "pcm16",
|
||||
"engine": backend.id,
|
||||
})
|
||||
started = True
|
||||
|
||||
# Convert to 16-bit PCM and stream
|
||||
pcm = (wav_tensor * 32767).clamp(-32768, 32767).to(torch.int16)
|
||||
if pcm.ndim == 2:
|
||||
pcm = pcm[0] # mono
|
||||
pcm_bytes = pcm.numpy().tobytes()
|
||||
|
||||
n_samples = len(pcm)
|
||||
sent_samples = 0
|
||||
while sent_samples < n_samples:
|
||||
end = min(sent_samples + CHUNK_SAMPLES, n_samples)
|
||||
chunk = pcm_bytes[sent_samples * 2: end * 2]
|
||||
await websocket.send_bytes(chunk)
|
||||
sent_samples = end
|
||||
# Yield to event loop between chunks for responsiveness
|
||||
await asyncio.sleep(0)
|
||||
total_samples += n_samples
|
||||
while sent_samples < total_samples:
|
||||
end = min(sent_samples + CHUNK_SAMPLES, total_samples)
|
||||
start_byte = sent_samples * 2
|
||||
end_byte = end * 2
|
||||
chunk = pcm_bytes[start_byte:end_byte]
|
||||
await websocket.send_bytes(chunk)
|
||||
sent_samples = end
|
||||
# Yield to event loop between chunks for responsiveness
|
||||
await asyncio.sleep(0)
|
||||
|
||||
gen_time = round(time.perf_counter() - t0, 3)
|
||||
duration = round(total_samples / sr, 3)
|
||||
|
||||
@@ -34,8 +34,6 @@ class SystemInfoResponse(BaseModel):
|
||||
asr_model: str = "unknown"
|
||||
translate_provider: str = "unknown"
|
||||
has_hf_token: bool = False
|
||||
# Xet fast-download backend state (FDL-03): {xet_enabled, xet_version, high_performance}
|
||||
fast_download: dict | None = None
|
||||
device: str = "cpu"
|
||||
python: str = ""
|
||||
platform: str = ""
|
||||
@@ -129,41 +127,16 @@ class DeviceInfo(BaseModel):
|
||||
gpu_available: bool = False
|
||||
gpu_driver: str | None = None
|
||||
gpu_device_name: str | None = None
|
||||
# From the canonical device probe (core.device_caps) — distinguishes ROCm
|
||||
# from CUDA, unlike the legacy nvidia-smi-based gpu_vendor/gpu_backend.
|
||||
gpu_family: str = "cpu"
|
||||
vram_gb: float = 0.0
|
||||
ram_gb: float = 0.0
|
||||
disk_free_gb: float = 0.0
|
||||
|
||||
|
||||
class GpuRouting(BaseModel):
|
||||
"""Routing verdict for the active TTS engine on THIS host (#21).
|
||||
|
||||
Distinct from the per-engine `routing_*` keys in `/engines`: this is the
|
||||
single verdict for the *currently-selected* engine, surfaced in preflight +
|
||||
diagnose so the user hears about a CPU fallback / unavailable GPU before a
|
||||
slow or failed synth — no silent CPU fallback.
|
||||
"""
|
||||
model_config = ConfigDict(extra="allow")
|
||||
|
||||
engine: str | None = None # active TTS engine id
|
||||
effective_device: str | None = None # device it will actually use here
|
||||
routing_status: str | None = None # accelerated|cpu_fallback|cpu_only|unavailable|none
|
||||
routing_reason: str | None = None # scrubbed; null when none
|
||||
host_family: str = "cpu" # detect_host_caps().family
|
||||
vram_gb: float = 0.0
|
||||
|
||||
|
||||
class PreflightResponse(BaseModel):
|
||||
"""GET /setup/preflight"""
|
||||
ok: bool
|
||||
has_warnings: bool = False
|
||||
checks: list[PreflightCheck] = Field(default_factory=list)
|
||||
device: DeviceInfo
|
||||
# Explicit field (PreflightResponse has no extra="allow") so the verdict
|
||||
# survives serialization instead of being silently dropped.
|
||||
gpu_routing: GpuRouting | None = None
|
||||
|
||||
|
||||
class InstallModelRequest(BaseModel):
|
||||
|
||||
@@ -49,12 +49,6 @@ _BASE_SCHEMA = """
|
||||
personality TEXT DEFAULT '',
|
||||
description TEXT DEFAULT '',
|
||||
is_demo INTEGER DEFAULT 0,
|
||||
verified_own_voice INTEGER DEFAULT 0,
|
||||
consent_text TEXT DEFAULT '',
|
||||
consent_audio_path TEXT DEFAULT '',
|
||||
consent_recorded_at REAL DEFAULT NULL,
|
||||
kind TEXT DEFAULT 'clone',
|
||||
vd_states TEXT DEFAULT NULL,
|
||||
created_at REAL
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS generation_history (
|
||||
@@ -144,19 +138,6 @@ _BASE_SCHEMA = """
|
||||
value TEXT NOT NULL,
|
||||
updated_at REAL NOT NULL
|
||||
);
|
||||
|
||||
-- Wave 2.2: per-agent MCP voice bindings. An MCP client (Claude Code,
|
||||
-- Cursor, …) identified by the X-OmniVoice-Client-Id header it sends is
|
||||
-- bound to a default voice profile / engine. Fresh installs create it
|
||||
-- here; v0.3.x upgrades get it via alembic 0004.
|
||||
CREATE TABLE IF NOT EXISTS mcp_client_bindings (
|
||||
client_id TEXT PRIMARY KEY,
|
||||
label TEXT NOT NULL DEFAULT '',
|
||||
profile_id TEXT,
|
||||
default_engine TEXT,
|
||||
last_seen_at REAL,
|
||||
created_at REAL
|
||||
);
|
||||
"""
|
||||
|
||||
# Only tables/columns this module is allowed to ALTER. Prevents SQL injection via
|
||||
|
||||
@@ -1,336 +0,0 @@
|
||||
"""Free-text voice-description → voice-design parameter mapper (issue #317).
|
||||
|
||||
Parity with the hosted omnivoice.app "Describe your voice" field, implemented
|
||||
fully locally: a deterministic keyword/phrase mapper that projects a natural-
|
||||
language description (e.g. ``"a warm elderly British storyteller, slightly
|
||||
raspy"``) onto the **existing** voice-design parameter space — the same six
|
||||
categories the Design tab's attribute picker drives (Gender / Age / Pitch /
|
||||
Style / EnglishAccent / ChineseDialect).
|
||||
|
||||
Design notes
|
||||
============
|
||||
* **No model, no network.** This is an ordered synonym-table matcher, not an
|
||||
LLM call — it runs identically on macOS/Windows/Linux with zero deps beyond
|
||||
the stdlib, preserving the local-first guarantee.
|
||||
* **Single source of truth.** Every canonical token this module can emit is
|
||||
validated at import time against the engine taxonomy in
|
||||
``omnivoice/utils/voice_design.py`` (loaded via ``core.archetypes``), so the
|
||||
mapper can never produce an instruct item the engine validator would reject
|
||||
(the issue-#89 / #115 crash modes). The Chinese translations of each token
|
||||
(e.g. ``男``/``中年``) are *derived* from that taxonomy, never hardcoded.
|
||||
* **Ordered rules, first match wins.** Within a category, rules are checked in
|
||||
a hand-ordered list so more specific phrases outrank generic ones
|
||||
("young child" → child, not young adult; "very deep" → very low pitch, not
|
||||
low pitch). Within one rule, the earliest occurrence in the text is reported
|
||||
as the matched phrase. Deterministic by construction.
|
||||
* **Graceful degradation.** Anything the taxonomy can't express (timbre words
|
||||
like "raspy", role words like "storyteller") is returned in ``unmatched`` so
|
||||
the UI can tell the user exactly which parts were ignored instead of failing
|
||||
silently (issue #317's validation-feedback note). A description with no
|
||||
matches at all yields all-``Auto`` attrs and an empty instruct.
|
||||
|
||||
Localization note (CLAUDE.md): the only hardcoded CJK here is
|
||||
``DIALECT_PINYIN`` — a functional pinyin → Chinese-dialect-token mapping
|
||||
(model vocabulary, like ``frontend/src/utils/constants.js``). Registered in
|
||||
``tests/test_no_hardcoded_cjk.py``'s allowlist with this justification.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
# Reuse the taxonomy already loaded (stdlib-only, by file path) by the
|
||||
# archetype engine — same single source of truth, one loader to maintain.
|
||||
from core.archetypes import _VD
|
||||
|
||||
_EN_TO_ZH = _VD._INSTRUCT_EN_TO_ZH # {"male": "男", ...}
|
||||
_ZH_RE = _VD._ZH_RE
|
||||
_VALID = _VD._INSTRUCT_ALL_VALID # every token the engine accepts
|
||||
_DIALECTS = set(_VD._INSTRUCT_CATEGORIES[5]) # the 12 Chinese dialect tokens
|
||||
|
||||
# Category names match the frontend's CATEGORIES keys (utils/constants.js) and
|
||||
# the archetype ``attrs`` shape, so the response drops straight into vdStates.
|
||||
CATEGORY_ORDER = ("Gender", "Age", "Pitch", "Style", "EnglishAccent", "ChineseDialect")
|
||||
|
||||
# ── Pinyin / romanized names → Chinese-dialect tokens (functional vocabulary) ─
|
||||
DIALECT_PINYIN = {
|
||||
"henan": "河南话",
|
||||
"shaanxi": "陕西话",
|
||||
"sichuan": "四川话",
|
||||
"szechuan": "四川话",
|
||||
"guizhou": "贵州话",
|
||||
"yunnan": "云南话",
|
||||
"guilin": "桂林话",
|
||||
"jinan": "济南话",
|
||||
"shijiazhuang": "石家庄话",
|
||||
"gansu": "甘肃话",
|
||||
"ningxia": "宁夏话",
|
||||
"qingdao": "青岛话",
|
||||
"dongbei": "东北话",
|
||||
"northeastern chinese": "东北话",
|
||||
}
|
||||
|
||||
# ── Synonym tables ────────────────────────────────────────────────────────────
|
||||
# Per category: ordered list of (canonical_token, [phrases]). First rule with
|
||||
# any hit wins the category, so specific phrases must precede generic ones.
|
||||
# Each canonical token's Chinese translation from the taxonomy is appended
|
||||
# automatically at compile time (so "中年" maps to "middle-aged", etc.).
|
||||
|
||||
_GENDER_RULES = [
|
||||
("female", [
|
||||
"female", "woman", "women", "lady", "ladies", "girl", "girls",
|
||||
"feminine", "gal", "grandma", "grandmother", "granny", "mother",
|
||||
"mom", "mum", "aunt", "auntie", "queen", "princess", "actress",
|
||||
"she", "her",
|
||||
]),
|
||||
("male", [
|
||||
"male", "man", "men", "guy", "guys", "boy", "boys", "masculine",
|
||||
"gentleman", "gentlemen", "dude", "grandpa", "grandfather", "father",
|
||||
"dad", "uncle", "king", "prince", "actor", "he", "him", "his",
|
||||
]),
|
||||
]
|
||||
|
||||
# Order is load-bearing: "child" precedes "young adult" so "young child" →
|
||||
# child; "middle-aged" precedes "elderly" so elderly's bare "aged" synonym
|
||||
# can't fire inside the hyphenated "middle-aged" (hyphen is a \b boundary);
|
||||
# "elderly" precedes "young adult" so grandparent words don't fall through.
|
||||
_AGE_RULES = [
|
||||
("child", [
|
||||
"child", "children", "kid", "kiddo", "toddler", "little boy",
|
||||
"little girl", "young boy", "young girl", "small child", "childlike",
|
||||
]),
|
||||
("teenager", ["teenager", "teen", "teenage", "adolescent"]),
|
||||
("middle-aged", [
|
||||
"middle-aged", "middle aged", "middle age", "midlife", "forties",
|
||||
"fifties", "sixties", "mature",
|
||||
]),
|
||||
("elderly", [
|
||||
"elderly", "old man", "old woman", "old lady", "older man",
|
||||
"older woman", "elder", "senior", "aged", "grandpa", "grandfather",
|
||||
"grandma", "grandmother", "granny", "retired", "seventies",
|
||||
"eighties", "nineties", "old",
|
||||
]),
|
||||
("young adult", [
|
||||
"young adult", "young woman", "young man", "young lady", "youthful",
|
||||
"twenties", "thirties", "college", "young",
|
||||
]),
|
||||
]
|
||||
|
||||
# "very …" rules precede their plain counterparts so "very deep" doesn't stop
|
||||
# at "deep". Bare "low"/"high" only count next to a voice word (pitch/voice/
|
||||
# tone/register) to avoid false hits like "high quality" or "low effort".
|
||||
_PITCH_RULES = [
|
||||
("very low pitch", [
|
||||
"very low pitch", "very low-pitched", "very low pitched",
|
||||
"very low voice", "very low tone", "very deep", "extremely deep",
|
||||
"extremely low", "ultra deep", "booming",
|
||||
]),
|
||||
("very high pitch", [
|
||||
"very high pitch", "very high-pitched", "very high pitched",
|
||||
"very high voice", "very high tone", "extremely high", "squeaky",
|
||||
"shrill", "falsetto", "chipmunk",
|
||||
]),
|
||||
("low pitch", [
|
||||
"low pitch", "low-pitched", "low pitched", "low voice", "low tone",
|
||||
"low register", "deep", "deeper", "bass", "baritone", "husky",
|
||||
]),
|
||||
("high pitch", [
|
||||
"high pitch", "high-pitched", "high pitched", "high voice",
|
||||
"high tone", "high register", "soprano",
|
||||
]),
|
||||
("moderate pitch", [
|
||||
"moderate pitch", "medium pitch", "medium-pitched", "medium pitched",
|
||||
"mid-range", "midrange", "average pitch", "moderate",
|
||||
]),
|
||||
]
|
||||
|
||||
_STYLE_RULES = [
|
||||
("whisper", [
|
||||
"whisper", "whispering", "whispered", "whispery", "hushed",
|
||||
"breathy", "soft-spoken", "soft spoken",
|
||||
]),
|
||||
]
|
||||
|
||||
# Bare "english" means the language, so only the explicit "english accent"
|
||||
# phrase maps to british. "chinese" maps to the chinese *accent* (English
|
||||
# speech with a Chinese accent); actual dialect words live in DIALECT_PINYIN.
|
||||
_ACCENT_RULES = [
|
||||
("american accent", [
|
||||
"american", "america", "usa", "us accent", "midwestern",
|
||||
"californian", "new york",
|
||||
]),
|
||||
("british accent", [
|
||||
"british", "britain", "english accent", "england", "uk accent",
|
||||
"london", "cockney", "posh", "received pronunciation",
|
||||
]),
|
||||
("australian accent", ["australian", "australia", "aussie"]),
|
||||
("canadian accent", ["canadian", "canada"]),
|
||||
("indian accent", ["indian", "india"]),
|
||||
("chinese accent", ["chinese accent", "chinese-accented", "chinese"]),
|
||||
("korean accent", ["korean", "korea"]),
|
||||
("japanese accent", ["japanese", "japan"]),
|
||||
("portuguese accent", ["portuguese", "portugal", "brazilian", "brazil"]),
|
||||
("russian accent", ["russian", "russia"]),
|
||||
]
|
||||
|
||||
_DIALECT_RULES = [
|
||||
(token, [pinyin for pinyin, tok in DIALECT_PINYIN.items() if tok == token])
|
||||
for token in sorted(_DIALECTS)
|
||||
]
|
||||
|
||||
_RULES = {
|
||||
"Gender": _GENDER_RULES,
|
||||
"Age": _AGE_RULES,
|
||||
"Pitch": _PITCH_RULES,
|
||||
"Style": _STYLE_RULES,
|
||||
"EnglishAccent": _ACCENT_RULES,
|
||||
"ChineseDialect": _DIALECT_RULES,
|
||||
}
|
||||
|
||||
# Import-time guard: every canonical token must be in the engine taxonomy, so
|
||||
# a taxonomy rename upstream fails loudly here instead of at synthesis time.
|
||||
for _cat_rules in _RULES.values():
|
||||
for _token, _ in _cat_rules:
|
||||
assert _token in _VALID, f"describe_voice token not in taxonomy: {_token!r}"
|
||||
for _tok in DIALECT_PINYIN.values():
|
||||
assert _tok in _DIALECTS, f"DIALECT_PINYIN value not a taxonomy dialect: {_tok!r}"
|
||||
|
||||
|
||||
# ── Pattern compilation ───────────────────────────────────────────────────────
|
||||
def _compile_phrase(phrase: str) -> re.Pattern:
|
||||
"""Compile a synonym phrase to a regex.
|
||||
|
||||
Latin phrases get word boundaries (so "male" never fires inside "female",
|
||||
"old" never inside "bold") and flexible separators (space or hyphen, so
|
||||
"middle aged" also matches "middle-aged"). CJK phrases match as plain
|
||||
substrings — word boundaries are meaningless without spaces.
|
||||
"""
|
||||
if _ZH_RE.search(phrase):
|
||||
return re.compile(re.escape(phrase))
|
||||
parts = [re.escape(p) for p in re.split(r"[ -]+", phrase) if p]
|
||||
return re.compile(r"\b" + r"[\s\-]+".join(parts) + r"\b")
|
||||
|
||||
|
||||
def _compiled_rules():
|
||||
out = {}
|
||||
for cat, rules in _RULES.items():
|
||||
compiled = []
|
||||
for token, phrases in rules:
|
||||
pats = list(phrases)
|
||||
# Derive the Chinese form of each canonical token from the
|
||||
# taxonomy (e.g. "middle-aged" → "中年") — never hardcoded here.
|
||||
zh = _EN_TO_ZH.get(token)
|
||||
if zh:
|
||||
pats.append(zh)
|
||||
if token not in pats:
|
||||
pats.append(token) # the canonical token always matches itself
|
||||
compiled.append((token, [_compile_phrase(p) for p in pats]))
|
||||
out[cat] = compiled
|
||||
return out
|
||||
|
||||
|
||||
_COMPILED = _compiled_rules()
|
||||
|
||||
# "<N> year(s) old / <N>-year-old / <N> yo" → an age bracket. Runs before the
|
||||
# keyword rules so the trailing "old" never misfires as elderly.
|
||||
_AGE_NUM = re.compile(
|
||||
r"\b(\d{1,3})(?:[\s\-]*(?:years?|yrs?|yr)[\s\-]*old|[\s\-]*(?:yo|y/o))\b"
|
||||
)
|
||||
|
||||
|
||||
def _age_token_for(years: int) -> str:
|
||||
if years <= 12:
|
||||
return "child"
|
||||
if years <= 19:
|
||||
return "teenager"
|
||||
if years <= 39:
|
||||
return "young adult"
|
||||
if years <= 64:
|
||||
return "middle-aged"
|
||||
return "elderly"
|
||||
|
||||
|
||||
def _normalize(description: str) -> str:
|
||||
text = (description or "").lower()
|
||||
text = text.replace("’", "'").replace("‘", "'")
|
||||
text = text.replace("“", '"').replace("”", '"')
|
||||
return re.sub(r"[ \t]+", " ", text)
|
||||
|
||||
|
||||
def _match_category(category: str, text: str):
|
||||
"""Return (token, match) for the first rule with a hit, else None.
|
||||
|
||||
Rule order decides the winning token; within the winning rule the earliest
|
||||
occurrence in the text is reported as the matched phrase.
|
||||
"""
|
||||
if category == "Age":
|
||||
m = _AGE_NUM.search(text)
|
||||
if m:
|
||||
return _age_token_for(int(m.group(1))), m
|
||||
for token, patterns in _COMPILED[category]:
|
||||
best = None
|
||||
for pat in patterns:
|
||||
m = pat.search(text)
|
||||
if m is not None and (best is None or m.start() < best.start()):
|
||||
best = m
|
||||
if best is not None:
|
||||
return token, best
|
||||
return None
|
||||
|
||||
|
||||
# Fragment splitter for the "unmatched" report: clause separators (incl. the
|
||||
# CJK comma/ideographic stop, which CJK descriptions use instead of ASCII).
|
||||
_FRAGMENT = re.compile(r"[^,;.!?()\n,。;!?、]+")
|
||||
_HAS_CONTENT = re.compile(r"[\w一-鿿]")
|
||||
|
||||
|
||||
def parse_description(description: str) -> dict:
|
||||
"""Map a free-text voice description onto the design parameter space.
|
||||
|
||||
Returns a dict with:
|
||||
* ``attrs`` — full category → token map (``"Auto"`` where nothing
|
||||
matched); same shape as the Design tab's ``vdStates``.
|
||||
* ``instruct`` — validator-safe instruct string built from the matched
|
||||
tokens, in canonical category order (may be ``""``).
|
||||
* ``matched`` — list of ``{category, token, phrase}`` for transparency.
|
||||
* ``unmatched`` — clause fragments that contributed no attribute, so the
|
||||
UI can show what was ignored instead of failing silently.
|
||||
"""
|
||||
text = _normalize(description)
|
||||
attrs = {cat: "Auto" for cat in CATEGORY_ORDER}
|
||||
matched = []
|
||||
spans = []
|
||||
|
||||
for category in CATEGORY_ORDER:
|
||||
hit = _match_category(category, text)
|
||||
if hit is None:
|
||||
continue
|
||||
token, m = hit
|
||||
attrs[category] = token
|
||||
matched.append({"category": category, "token": token, "phrase": m.group(0)})
|
||||
spans.append((m.start(), m.end()))
|
||||
|
||||
# Accents are English-only and dialects Chinese-only in the engine
|
||||
# taxonomy; a dialect voice speaks Chinese, so an accent token alongside
|
||||
# it is contradictory (the issue-#114 conflict class). Dialect wins.
|
||||
if attrs["ChineseDialect"] != "Auto" and attrs["EnglishAccent"] != "Auto":
|
||||
dropped = attrs["EnglishAccent"]
|
||||
attrs["EnglishAccent"] = "Auto"
|
||||
matched = [m for m in matched if not (m["category"] == "EnglishAccent" and m["token"] == dropped)]
|
||||
|
||||
instruct = ", ".join(attrs[c] for c in CATEGORY_ORDER if attrs[c] != "Auto")
|
||||
|
||||
unmatched = []
|
||||
for frag in _FRAGMENT.finditer(text):
|
||||
if not _HAS_CONTENT.search(frag.group(0)):
|
||||
continue
|
||||
lo, hi = frag.start(), frag.end()
|
||||
if any(s < hi and e > lo for s, e in spans):
|
||||
continue
|
||||
unmatched.append(frag.group(0).strip())
|
||||
|
||||
return {
|
||||
"attrs": attrs,
|
||||
"instruct": instruct,
|
||||
"matched": matched,
|
||||
"unmatched": unmatched,
|
||||
}
|
||||
@@ -1,279 +0,0 @@
|
||||
"""Canonical host compute-capability probe — the single source of truth for
|
||||
"what can this machine actually accelerate on."
|
||||
|
||||
Every routing decision (the engine compatibility matrix, ``/setup/preflight``,
|
||||
``/system/diagnose``, and the synth-time no-silent-fallback gating) reads from
|
||||
``detect_host_caps()`` so the probe and the model loader can never disagree.
|
||||
|
||||
Design contract (load-bearing):
|
||||
- **Never raises** to a caller. A broken torch / driver crash degrades to a
|
||||
cached CPU-only ``probe_ok=False`` result; every endpoint stays responsive
|
||||
(local-first: the app must work with no GPU and even with a broken torch).
|
||||
- **No network call** — driver/sysctl reads only, no tensor allocation, so it
|
||||
stays kernel-free on cold start.
|
||||
- **No new regex** on any driver/device string (CodeQL py/polynomial-redos):
|
||||
the only string parse is the ``int(driver.split(".")[0])`` shape reused
|
||||
from the wizard, and arch comparison is plain list membership.
|
||||
- Distinguishes **ROCm from CUDA** (unlike the gguf ``hardware_probe``):
|
||||
ROCm-on-HIP presents through ``torch.cuda`` but is reported ``family="rocm"``.
|
||||
|
||||
The ``get_best_device()`` loader (``services.model_manager``) delegates its
|
||||
*family* decision here while keeping its own DirectML branch and the ROCm
|
||||
``HSA_OVERRIDE_GFX_VERSION`` env side-effect — the probe **reads**, the loader
|
||||
**writes**. (The gguf ``hardware_probe.detect_capabilities()`` rebase onto this
|
||||
module is a deliberate follow-up: it has its own torch-mocked test suite and a
|
||||
VRAM-driven quant table that is unaffected by the family rename, so it is kept
|
||||
out of this backend-only slice.)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import platform as _platform
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal
|
||||
|
||||
DeviceFamily = Literal["cuda", "rocm", "mps", "xpu", "cpu"]
|
||||
|
||||
# Stable substring stamped onto notes that represent a real kernel-launch risk
|
||||
# (arch/driver mismatch) — as opposed to advisory notes (multi-GPU, VRAM query
|
||||
# failed, DirectML present). ``engine_routing`` keys the "accelerated, but…"
|
||||
# caveat off this marker so advisory notes never downgrade an accelerated badge.
|
||||
KERNEL_RISK_MARKER = "may fail at kernel launch"
|
||||
|
||||
# Substring marking a DirectML-present (Windows GPU) host. The probe reports
|
||||
# such hosts as ``family="cpu"`` (DirectML is not a torch device family); the
|
||||
# router reads this marker to explain the neutral badge instead of "no GPU".
|
||||
DIRECTML_MARKER = "DirectML device present"
|
||||
|
||||
# NOTE: the NVIDIA driver-version check (min R555 for the bundled CUDA runtime)
|
||||
# is intentionally NOT done here — it requires shelling to ``nvidia-smi``, which
|
||||
# would put a subprocess on the cold-start probe path. That check stays in
|
||||
# ``wizard._detect_gpu`` (preflight), which already runs it. The probe only
|
||||
# emits the torch-visible SM-arch caveat (cheap, metadata-only).
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HostCaps:
|
||||
"""Snapshot of the host's accelerator capability. Immutable + cached."""
|
||||
|
||||
family: DeviceFamily
|
||||
"""Best available accelerator family, else ``"cpu"``."""
|
||||
|
||||
available_families: tuple[DeviceFamily, ...]
|
||||
"""Everything usable; **always includes** ``"cpu"`` (invariant)."""
|
||||
|
||||
device_name: str = ""
|
||||
"""Device 0's name, e.g. ``"NVIDIA RTX 4090"`` / ``"Apple Silicon (MPS)"``."""
|
||||
|
||||
vram_gb: float = 0.0
|
||||
"""CUDA/ROCm total VRAM in GB; MPS = system RAM / 2; 0 for cpu/xpu."""
|
||||
|
||||
driver: str | None = None
|
||||
"""Raw ROCm HIP version string (``torch.version.hip``) or ``None``. The
|
||||
NVIDIA driver-version check is owned by ``wizard._detect_gpu`` (it already
|
||||
shells to ``nvidia-smi``); the probe stays subprocess-free."""
|
||||
|
||||
notes: tuple[str, ...] = ()
|
||||
"""Author-controlled English advisories (never user input). Empty on a
|
||||
clean accelerated host."""
|
||||
|
||||
probe_ok: bool = True
|
||||
"""``False`` only when torch could not be imported (degraded CPU-only)."""
|
||||
|
||||
|
||||
def _probe() -> HostCaps:
|
||||
"""Run the probe once. Enumerates every failure branch from the spec's
|
||||
degradation contract; never raises."""
|
||||
try:
|
||||
import torch
|
||||
except Exception:
|
||||
return HostCaps(
|
||||
family="cpu",
|
||||
available_families=("cpu",),
|
||||
notes=("torch not importable; treating host as CPU-only",),
|
||||
probe_ok=False,
|
||||
)
|
||||
|
||||
notes: list[str] = []
|
||||
# Probe EVERY accelerator independently into this list (don't short-circuit
|
||||
# after the first hit) so `available_families` is honest on hybrid hosts
|
||||
# (e.g. an NVIDIA GPU + an Intel iGPU exposed via IPEX). The preferred
|
||||
# `family` is chosen by priority at the end.
|
||||
detected: list[DeviceFamily] = []
|
||||
device_name = ""
|
||||
vram_gb = 0.0
|
||||
driver: str | None = None
|
||||
|
||||
# ── CUDA / ROCm (both present through torch.cuda) ────────────────────
|
||||
cuda_ok = False
|
||||
try:
|
||||
cuda_ok = bool(torch.cuda.is_available())
|
||||
except Exception as exc: # broken CUDA init (forked process / driver crash)
|
||||
notes.append(f"CUDA init raised: {type(exc).__name__}")
|
||||
|
||||
if cuda_ok:
|
||||
try:
|
||||
count = int(torch.cuda.device_count())
|
||||
except Exception:
|
||||
count = 0
|
||||
if count == 0:
|
||||
notes.append("CUDA reports available but device_count==0")
|
||||
else:
|
||||
is_rocm = getattr(torch.version, "hip", None) is not None
|
||||
detected.append("rocm" if is_rocm else "cuda")
|
||||
if is_rocm:
|
||||
driver = getattr(torch.version, "hip", None)
|
||||
if count > 1:
|
||||
notes.append(f"{count} GPUs detected; routing reflects device 0")
|
||||
try:
|
||||
device_name = torch.cuda.get_device_name(0)
|
||||
except Exception:
|
||||
device_name = ""
|
||||
try:
|
||||
_free, total = torch.cuda.mem_get_info()
|
||||
vram_gb = float(total) / (1024 ** 3)
|
||||
except Exception:
|
||||
notes.append("VRAM query failed")
|
||||
# SM-arch mismatch (mirrors model_manager.check_device_compatibility).
|
||||
try:
|
||||
major, minor = torch.cuda.get_device_capability(0)
|
||||
arch_list = getattr(torch.cuda, "_get_arch_list", lambda: [])()
|
||||
if arch_list:
|
||||
sm_tag = f"sm_{major}{minor}"
|
||||
compute_tag = f"compute_{major}{minor}"
|
||||
if sm_tag not in arch_list and compute_tag not in arch_list:
|
||||
notes.append(
|
||||
f"{device_name or 'GPU'} ({sm_tag}) not in this torch "
|
||||
f"build's archs ({', '.join(arch_list)}) — "
|
||||
f"{KERNEL_RISK_MARKER}"
|
||||
)
|
||||
except Exception:
|
||||
# Arch metadata unavailable on this torch build — skip the check
|
||||
# (treated as compatible, exactly as check_device_compatibility).
|
||||
pass
|
||||
|
||||
# ── Intel XPU via IPEX ───────────────────────────────────────────────
|
||||
try:
|
||||
import intel_extension_for_pytorch # noqa: F401
|
||||
if hasattr(torch, "xpu") and torch.xpu.is_available():
|
||||
detected.append("xpu")
|
||||
if not device_name:
|
||||
try:
|
||||
device_name = torch.xpu.get_device_name(0)
|
||||
except Exception:
|
||||
# XPU present but unnamed — family classification still holds.
|
||||
pass
|
||||
notes.append("XPU VRAM not queried (unreliable across IPEX versions)")
|
||||
except Exception:
|
||||
# IPEX absent or XPU probe failed — no XPU on this host.
|
||||
pass
|
||||
|
||||
# ── Apple Silicon MPS ────────────────────────────────────────────────
|
||||
try:
|
||||
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
detected.append("mps")
|
||||
if not device_name:
|
||||
device_name = "Apple Silicon (MPS)"
|
||||
if not vram_gb:
|
||||
try:
|
||||
import psutil
|
||||
vram_gb = float(psutil.virtual_memory().total) / (1024 ** 3) / 2
|
||||
except Exception:
|
||||
notes.append("psutil unavailable; MPS VRAM unknown")
|
||||
except Exception:
|
||||
# MPS probe raised on a non-Apple/old torch — treat as no MPS.
|
||||
pass
|
||||
|
||||
# ── DirectML — Windows GPU, NOT a torch device family ────────────────
|
||||
try:
|
||||
import torch_directml
|
||||
if torch_directml.device_count() > 0:
|
||||
notes.append(
|
||||
f"{DIRECTML_MARKER} (Windows GPU); torch-family probe treats "
|
||||
f"as non-accelerated"
|
||||
)
|
||||
except Exception:
|
||||
# torch_directml absent (the common case) — no DirectML on this host.
|
||||
pass
|
||||
|
||||
# Preferred family by priority; cpu when nothing accelerated was detected.
|
||||
family: DeviceFamily = "cpu"
|
||||
for pref in ("cuda", "rocm", "xpu", "mps"):
|
||||
if pref in detected:
|
||||
family = pref # type: ignore[assignment]
|
||||
break
|
||||
# available_families: every detected accelerator + cpu, deduped, cpu last.
|
||||
available: tuple[DeviceFamily, ...] = tuple(dict.fromkeys([*detected, "cpu"]))
|
||||
|
||||
return HostCaps(
|
||||
family=family,
|
||||
available_families=available,
|
||||
device_name=device_name,
|
||||
vram_gb=vram_gb,
|
||||
driver=driver,
|
||||
notes=tuple(notes),
|
||||
probe_ok=True,
|
||||
)
|
||||
|
||||
|
||||
@functools.lru_cache(maxsize=1)
|
||||
def detect_host_caps() -> HostCaps:
|
||||
"""Cached per-process host capabilities. Never raises, makes no network
|
||||
call, kernel-free on cold start. Host compute capability does not change at
|
||||
runtime in any supported desktop flow (no GPU hot-plug; switching the active
|
||||
engine does not re-probe — routing is recomputed from these same caps), so
|
||||
a single probe per process is correct. ``probe_ok=False`` is cached too."""
|
||||
return _probe()
|
||||
|
||||
|
||||
def refresh() -> HostCaps:
|
||||
"""Clear the cache and re-probe. **TEST-ONLY** — nothing in the running app
|
||||
calls this (host caps are immutable per process)."""
|
||||
detect_host_caps.cache_clear()
|
||||
return detect_host_caps()
|
||||
|
||||
|
||||
def mlx_supported() -> tuple[bool, str]:
|
||||
"""``(ok, reason)``. ``ok=True`` **only** on Apple Silicon
|
||||
(``sys.platform == "darwin"`` and ``platform.machine() == "arm64"``) with
|
||||
torch MPS available — the shared gate for MLX-Audio / MLX-Whisper (#390).
|
||||
|
||||
Gates on exact-string equality (no regex → no CodeQL surface). On any
|
||||
non-Apple host it returns ``False`` **before** any package import, so a
|
||||
stray ``mlx_*`` wheel on Linux/Windows never reports available.
|
||||
"""
|
||||
if sys.platform != "darwin" or _platform.machine() != "arm64":
|
||||
if sys.platform == "darwin":
|
||||
return (False, "MLX requires Apple Silicon; this Mac is Intel")
|
||||
return (
|
||||
False,
|
||||
f"MLX requires Apple Silicon; this host is "
|
||||
f"{sys.platform}/{_platform.machine()}",
|
||||
)
|
||||
try:
|
||||
import torch
|
||||
except Exception:
|
||||
return (False, "torch not importable; cannot confirm MPS")
|
||||
try:
|
||||
if torch.backends.mps.is_available():
|
||||
return (True, "")
|
||||
except Exception:
|
||||
# MPS query raised — fall through to the conservative unavailable path.
|
||||
pass
|
||||
return (
|
||||
False,
|
||||
"Apple Silicon detected but torch MPS unavailable; "
|
||||
"reinstall torch with MPS support",
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"DeviceFamily",
|
||||
"HostCaps",
|
||||
"detect_host_caps",
|
||||
"refresh",
|
||||
"mlx_supported",
|
||||
"KERNEL_RISK_MARKER",
|
||||
"DIRECTML_MARKER",
|
||||
]
|
||||
@@ -4,7 +4,7 @@ One pass over everything a working install needs: Python, compute device,
|
||||
ffmpeg, HF token, disk, data-dir permissions, RAM, TTS engines, and (when
|
||||
requested) network reachability of the HuggingFace hub. Surfaced two ways:
|
||||
|
||||
- ``GET /system/diagnose`` (Settings > About -> "Run self-check")
|
||||
- ``GET /system/diagnose`` (Settings > About → "Run self-check")
|
||||
- ``python main.py --diagnose`` for headless installs / issue triage
|
||||
|
||||
Every ``detail``/``hint`` string is passed through ``core.scrub`` before it
|
||||
@@ -194,49 +194,6 @@ def _check_engines() -> dict:
|
||||
return _check("engines", "TTS engines", OK, detail)
|
||||
|
||||
|
||||
def _check_gpu_routing() -> dict:
|
||||
"""Routing verdict for the active TTS engine on THIS host (#21).
|
||||
|
||||
Surfaces a CPU fallback / unavailable-GPU *before* a slow or failed synth —
|
||||
the no-silent-fallback contract. `cpu_only` on a no-GPU machine is the
|
||||
expected normal state and stays OK (never noise-warns)."""
|
||||
try:
|
||||
from services.tts_backend import gpu_routing_verdict
|
||||
v = gpu_routing_verdict()
|
||||
except Exception as e:
|
||||
return _check("gpu_routing", "GPU routing", WARN, f"could not resolve: {e}")
|
||||
|
||||
status = v.get("routing_status")
|
||||
engine = v.get("engine") or "active engine"
|
||||
dev = v.get("effective_device") or "?"
|
||||
reason = v.get("routing_reason")
|
||||
host = v.get("host_family", "cpu")
|
||||
|
||||
if status == "accelerated":
|
||||
if reason: # driver/arch caveat — accelerated but at risk
|
||||
return _check("gpu_routing", "GPU routing", WARN,
|
||||
f"{engine} -> {dev}: {reason}",
|
||||
"The GPU is selected but may fail at kernel launch — "
|
||||
"update drivers / reinstall torch for this GPU arch.")
|
||||
return _check("gpu_routing", "GPU routing", OK, f"{engine} -> {dev} (accelerated)")
|
||||
if status == "cpu_fallback":
|
||||
return _check("gpu_routing", "GPU routing", WARN,
|
||||
f"{engine} runs on CPU: {reason or 'no GPU path for this host'}",
|
||||
"Pick an engine that supports this host's GPU for a big speedup, "
|
||||
"or continue on CPU (slower).")
|
||||
if status == "cpu_only":
|
||||
return _check("gpu_routing", "GPU routing", OK,
|
||||
f"{engine} -> cpu (no accelerator on this host)")
|
||||
if status == "unavailable":
|
||||
return _check("gpu_routing", "GPU routing", FAIL,
|
||||
f"{engine} can't run on this host: {reason or f'needs a GPU; host is {host}'}",
|
||||
"Select an engine with a CPU path in Settings -> Engines.")
|
||||
# status == "none" / unknown — no active engine resolved.
|
||||
return _check("gpu_routing", "GPU routing", WARN,
|
||||
"No active TTS engine resolved for routing.",
|
||||
"Pick an engine in Settings -> Engines.")
|
||||
|
||||
|
||||
_DEEP_TIMEOUT_S = 180
|
||||
|
||||
|
||||
@@ -307,8 +264,6 @@ def _check_network() -> dict:
|
||||
# all model downloads need to get started. urllib honors HTTP(S)_PROXY.
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
if not _HUB_URL.startswith("https://"): # constant today; guard the sink anyway
|
||||
raise ValueError(f"hub URL must be https, got {_HUB_URL!r}")
|
||||
req = urllib.request.Request(_HUB_URL, method="HEAD")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=_HUB_TIMEOUT_S):
|
||||
@@ -341,7 +296,6 @@ def run_diagnostics(include_network: bool = True, deep: bool = False) -> dict:
|
||||
_check_data_dir(),
|
||||
_check_ram(),
|
||||
_check_engines(),
|
||||
_check_gpu_routing(),
|
||||
]
|
||||
if include_network:
|
||||
checks.append(_check_network())
|
||||
|
||||
@@ -111,8 +111,7 @@ def classify_exception(exc: BaseException, trace: str = "") -> str:
|
||||
def _fingerprint(error_class: str, exc: BaseException) -> str:
|
||||
import hashlib
|
||||
raw = f"{error_class}|{type(exc).__name__}|{scrub_text(str(exc))[:200]}"
|
||||
# Dedup key for the journal, not a security boundary.
|
||||
return hashlib.sha1(raw.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:16]
|
||||
return hashlib.sha1(raw.encode("utf-8", "replace")).hexdigest()[:16]
|
||||
|
||||
|
||||
def _persist_locked() -> None:
|
||||
|
||||
@@ -32,7 +32,7 @@ _REDACTED_VALUE = "***REDACTED***"
|
||||
# One-line "what to do" per docs-taxonomy key. Keys mirror error_docs_map's
|
||||
# taxonomy; the docs URL itself stays owned by error_docs_map.
|
||||
_HINTS: dict[str, str] = {
|
||||
"PKG_RESOURCES_MISSING": "Run `uv pip install --reinstall 'setuptools>=75,<80'` in the backend venv (a plain install is skipped when setuptools' metadata is present but its pkg_resources files were removed by antivirus). Restart after.",
|
||||
"PKG_RESOURCES_MISSING": "Install setuptools in the backend environment (provides pkg_resources).",
|
||||
"GATEKEEPER_QUARANTINE": "Clear the macOS quarantine flag (xattr -cr the app), then reopen.",
|
||||
"APPIMAGE_WEBKIT_WHITESCREEN": "Launch with WEBKIT_DISABLE_DMABUF_RENDERER=1 set.",
|
||||
"HF_AUTH_FAILED": "Set a valid HF_TOKEN in Settings → Hugging Face and retry.",
|
||||
|
||||
@@ -42,12 +42,9 @@ _TOKEN_PATTERNS = (
|
||||
# pattern-wise (not just this machine's $HOME) so paths quoted from a
|
||||
# user's pasted log on another OS get cleaned too.
|
||||
_HOME_PATTERNS = (
|
||||
# Windows-with-forward-slashes must run BEFORE the bare macOS shape, or
|
||||
# `/Users/<name>` inside `C:/Users/<name>` gets eaten first, leaving `C:~`.
|
||||
re.compile(r"[A-Za-z]:/Users/[^/\s\"']+"), # Windows, forward slashes (file URLs, normalized traces)
|
||||
re.compile(r"/Users/[^/\s\"']+"), # macOS
|
||||
re.compile(r"/home/[^/\s\"']+"), # Linux
|
||||
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+"), # Windows, backslashes
|
||||
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+"), # Windows
|
||||
)
|
||||
|
||||
# Values shorter than this are too entropy-poor to be real secrets and too
|
||||
|
||||
@@ -82,28 +82,3 @@ def unset_user_env(key: str, path: Optional[str] = None) -> None:
|
||||
prefix = f"{key}="
|
||||
lines = [ln for ln in _read_lines(path) if not ln.startswith(prefix)]
|
||||
_write_lines(path, lines)
|
||||
|
||||
|
||||
def load_into_environ(path: Optional[str] = None) -> bool:
|
||||
"""Load the durable per-user env file into ``os.environ``, **overriding**
|
||||
any value a launcher already injected. Returns True if a file was loaded.
|
||||
|
||||
This file is the in-app Settings source of truth. The desktop launcher
|
||||
(Tauri) injects defaults like ``OMNIVOICE_CACHE_DIR`` (and ``HF_ENDPOINT``)
|
||||
from its *own* config into the backend's environment *before* startup, so
|
||||
loading this file with ``override=False`` meant a models directory the user
|
||||
changed in Settings was silently ignored on every launch — the effective
|
||||
location stayed on the old one no matter how many restarts (#480). Both keys
|
||||
this file can hold are the user's explicit Settings choice and should beat
|
||||
the launcher's default, so we override. Restores this file's documented
|
||||
"values written here take effect on the next backend launch" contract.
|
||||
"""
|
||||
path = path or os.environ.get("OMNIVOICE_ENV_FILE") or USER_ENV_PATH
|
||||
if not os.path.isfile(path):
|
||||
return False
|
||||
try:
|
||||
import dotenv
|
||||
except ImportError:
|
||||
return False
|
||||
dotenv.load_dotenv(path, override=True)
|
||||
return True
|
||||
|
||||
@@ -1,120 +0,0 @@
|
||||
"""Crash-isolated faster-whisper ASR sidecar (Wave 4.2 / Spec 7).
|
||||
|
||||
Runs faster-whisper in a child process so a CTranslate2 GPU-teardown segfault
|
||||
becomes a failed job, not a dead backend. Speaks the SubprocessBackend wire
|
||||
protocol (length-prefixed JSON over stdin/stdout):
|
||||
|
||||
on start → {"op":"ready","engine":"faster-whisper-isolated"}
|
||||
{"op":"ping"} → {"op":"pong"}
|
||||
{"op":"transcribe","audio_path":...,"word_timestamps":bool}
|
||||
→ {"op":"segments","result":{"segments":[...],"language":...}}
|
||||
{"op":"shutdown"} → exit 0
|
||||
error → {"op":"error","message":...}
|
||||
|
||||
Runs under the PARENT venv (faster-whisper is already a dependency) — only the
|
||||
process boundary is new. torch/CTranslate2 import lazily inside transcribe so
|
||||
the ready handshake fits the spawn timeout.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
MAX_FRAME_BYTES = 64 * 1024 * 1024
|
||||
_model = None
|
||||
|
||||
|
||||
def _send(stream, obj):
|
||||
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
|
||||
stream.write(struct.pack("!I", len(body)))
|
||||
stream.write(body)
|
||||
stream.flush()
|
||||
|
||||
|
||||
def _recv(stream):
|
||||
header = stream.read(4)
|
||||
if len(header) < 4:
|
||||
return None
|
||||
(n,) = struct.unpack("!I", header)
|
||||
if n > MAX_FRAME_BYTES:
|
||||
raise IOError(f"frame too large: {n}")
|
||||
body = bytearray()
|
||||
while len(body) < n:
|
||||
chunk = stream.read(n - len(body))
|
||||
if not chunk:
|
||||
raise IOError("short read")
|
||||
body.extend(chunk)
|
||||
return json.loads(bytes(body).decode("utf-8"))
|
||||
|
||||
|
||||
def _get_model():
|
||||
global _model
|
||||
if _model is None:
|
||||
from faster_whisper import WhisperModel
|
||||
name = os.environ.get("ASR_MODEL_FW", "large-v3")
|
||||
try:
|
||||
import torch
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
except Exception:
|
||||
device = "cpu"
|
||||
compute = "float16" if device == "cuda" else "int8"
|
||||
_model = WhisperModel(name, device=device, compute_type=compute)
|
||||
return _model
|
||||
|
||||
|
||||
def _transcribe(audio_path, word_timestamps):
|
||||
model = _get_model()
|
||||
segments, info = model.transcribe(audio_path, word_timestamps=word_timestamps)
|
||||
out = []
|
||||
for s in segments:
|
||||
seg = {"start": float(s.start), "end": float(s.end), "text": s.text}
|
||||
if word_timestamps and getattr(s, "words", None):
|
||||
seg["words"] = [
|
||||
{"word": w.word, "start": float(w.start), "end": float(w.end),
|
||||
"probability": float(getattr(w, "probability", 0.0))}
|
||||
for w in s.words
|
||||
]
|
||||
out.append(seg)
|
||||
return {
|
||||
"segments": out,
|
||||
"text": " ".join(s["text"].strip() for s in out).strip(),
|
||||
"language": getattr(info, "language", "unknown"),
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
stdin, stdout = sys.stdin.buffer, sys.stdout.buffer
|
||||
_send(stdout, {"op": "ready", "engine": "faster-whisper-isolated"})
|
||||
while True:
|
||||
try:
|
||||
msg = _recv(stdin)
|
||||
except Exception as exc:
|
||||
_send(stdout, {"op": "error", "stage": "recv", "message": f"{type(exc).__name__}: {exc}"})
|
||||
return 1
|
||||
if msg is None:
|
||||
return 0
|
||||
op = msg.get("op")
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong"})
|
||||
elif op == "transcribe":
|
||||
result = _transcribe(msg.get("audio_path"), bool(msg.get("word_timestamps", True)))
|
||||
_send(stdout, {"op": "segments", "result": result})
|
||||
elif op == "shutdown":
|
||||
return 0
|
||||
else:
|
||||
_send(stdout, {"op": "error", "stage": "dispatch", "message": f"unknown op: {op!r}"})
|
||||
except Exception as exc:
|
||||
_send(stdout, {
|
||||
"op": "error", "stage": "handler",
|
||||
"message": f"{type(exc).__name__}: {exc}",
|
||||
"traceback": traceback.format_exc(),
|
||||
})
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -101,12 +101,6 @@ def main() -> int:
|
||||
return 0
|
||||
|
||||
op = msg.get("op")
|
||||
# Wave 4.2: deterministic "crash mid-transcription" hook — exit BEFORE
|
||||
# sending any reply so the parent's blocking recv sees a dead pipe
|
||||
# (reply=None). The crash-after-one hook below replies first, so it
|
||||
# can't deterministically exercise the no-reply path.
|
||||
if op == "transcribe" and os.environ.get("OMNIVOICE_ECHO_CRASH_NO_REPLY") == "1":
|
||||
os._exit(1)
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong"})
|
||||
@@ -119,19 +113,6 @@ def main() -> int:
|
||||
"sample_rate": sr,
|
||||
"n_samples": n_samples,
|
||||
})
|
||||
elif op == "transcribe":
|
||||
# Wave 4.2: echo ASR op — a canned segments result so the
|
||||
# SubprocessASRBackend round-trip + respawn path is testable
|
||||
# without a real ASR engine.
|
||||
_send(stdout, {
|
||||
"op": "segments",
|
||||
"result": {
|
||||
"segments": [{"start": 0.0, "end": 1.0,
|
||||
"text": f"echo:{msg.get('audio_path', '')}"}],
|
||||
"text": f"echo:{msg.get('audio_path', '')}",
|
||||
"language": "en",
|
||||
},
|
||||
})
|
||||
elif op == "shutdown":
|
||||
return 0
|
||||
elif op == "probe_env" and test_mode:
|
||||
|
||||
@@ -79,11 +79,6 @@ class IndexTTS2Backend(SubprocessBackend):
|
||||
display_name = "IndexTTS2 (emotion control, duration control, zero-shot)"
|
||||
supports_voice_design = False # requires ref audio for timbre
|
||||
_DEFAULT_SAMPLE_RATE = 24000
|
||||
# Explicit so IndexTTS2 stops advertising the inherited CPU-only default:
|
||||
# the sidecar runs the IndexTTS PyTorch model on CUDA when present, else
|
||||
# CPU. ROCm left unclaimed (the sidecar's own venv would need a ROCm torch);
|
||||
# a ROCm host honestly resolves to cpu_fallback.
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
|
||||
@@ -74,24 +74,6 @@ import traceback
|
||||
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
|
||||
MAX_FRAME_BYTES = 64 * 1024 * 1024
|
||||
|
||||
|
||||
def _measure_vram_mb() -> float:
|
||||
"""This sidecar's own GPU memory in MB, for the loaded-models panel
|
||||
(MM2-08). The parent can't see a child's VRAM, so we self-report it in the
|
||||
pong. Degrades to 0 on CPU / when torch isn't loaded yet — never raises."""
|
||||
try:
|
||||
import torch # already a dep inside the indextts venv
|
||||
if torch.cuda.is_available():
|
||||
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
|
||||
mps = getattr(torch.backends, "mps", None)
|
||||
if mps is not None and mps.is_available():
|
||||
drv = getattr(torch.mps, "driver_allocated_memory", None)
|
||||
if drv:
|
||||
return round(drv() / (1024 ** 2), 1)
|
||||
except Exception:
|
||||
pass
|
||||
return 0.0
|
||||
|
||||
# Sample rate IndexTTS-2 emits natively. Advertised in the ready frame so
|
||||
# the parent doesn't have to import IndexTTS just to learn the rate.
|
||||
INDEXTTS_SAMPLE_RATE = 24000
|
||||
@@ -300,7 +282,7 @@ def main() -> int:
|
||||
op = msg.get("op") if isinstance(msg, dict) else None
|
||||
try:
|
||||
if op == "ping":
|
||||
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
|
||||
_send(stdout, {"op": "pong"})
|
||||
elif op == "synthesize":
|
||||
_handle_synthesize(msg, stdout)
|
||||
elif op == "shutdown":
|
||||
|
||||
@@ -353,21 +353,6 @@ def _make_backend_class():
|
||||
f"This clears the quarantine on the .app and its "
|
||||
f"bundled binaries. See docs/install/macos.md."
|
||||
)
|
||||
# Execute bit (issue #437). A `git clone` / zip extract on POSIX
|
||||
# can drop +x, which only surfaces at spawn time as
|
||||
# "[Errno 13] Permission denied" — and the generic synth handler
|
||||
# then mislabels it as out-of-memory. Self-heal here, AFTER the
|
||||
# SHA check has confirmed this is the right file (so we never
|
||||
# chmod a foreign binary). No-op on Windows.
|
||||
if os.name == "posix" and not os.access(bin_path, os.X_OK):
|
||||
try:
|
||||
bin_path.chmod(bin_path.stat().st_mode | 0o111)
|
||||
except OSError:
|
||||
return False, (
|
||||
f"GGUF binary {bin_path.name} isn't executable and "
|
||||
f"couldn't be made so — run `chmod +x {bin_path}` "
|
||||
f"and retry."
|
||||
)
|
||||
return True, "ready"
|
||||
except Exception as exc:
|
||||
return False, f"{type(exc).__name__}: {exc}"
|
||||
@@ -508,12 +493,6 @@ def _make_backend_class():
|
||||
* ``ref_audio`` (str/Path) — speaker reference WAV for cloning.
|
||||
* ``ref_text`` (str) — transcript of ``ref_audio``.
|
||||
* ``language`` (str) — ISO code or omnivoice-tts lang label.
|
||||
* ``instruct`` (str) — style instruction.
|
||||
* ``duration`` (float) — target duration in seconds.
|
||||
* ``seed`` (int) — deterministic sampling seed.
|
||||
* ``denoise`` (bool) — omit denoise token when false.
|
||||
* ``preprocess_prompt`` (bool) — skip prompt preprocessing when false.
|
||||
* ``chunk_duration`` / ``chunk_threshold`` (float) — binary long-form controls.
|
||||
"""
|
||||
import soundfile as sf # local import keeps module import cheap
|
||||
import torch
|
||||
@@ -524,32 +503,15 @@ def _make_backend_class():
|
||||
fd, out_str = tempfile.mkstemp(prefix="omnivoice-gguf-", suffix=".wav")
|
||||
os.close(fd)
|
||||
out_path = Path(out_str)
|
||||
ref_text_path: Optional[Path] = None
|
||||
|
||||
try:
|
||||
ref_text = kw.get("ref_text")
|
||||
if kw.get("ref_audio") and ref_text:
|
||||
text_fd, text_str = tempfile.mkstemp(
|
||||
prefix="omnivoice-gguf-ref-", suffix=".txt"
|
||||
)
|
||||
os.close(text_fd)
|
||||
ref_text_path = Path(text_str)
|
||||
ref_text_path.write_text(str(ref_text), encoding="utf-8")
|
||||
|
||||
argv = self._build_argv(
|
||||
base=base_path,
|
||||
tokenizer=tok_path,
|
||||
out_path=out_path,
|
||||
ref_audio=kw.get("ref_audio"),
|
||||
ref_text=str(ref_text_path) if ref_text_path else None,
|
||||
ref_text=kw.get("ref_text"),
|
||||
language=kw.get("language"),
|
||||
instruct=kw.get("instruct"),
|
||||
duration=kw.get("duration"),
|
||||
seed=kw.get("seed"),
|
||||
denoise=kw.get("denoise", True),
|
||||
preprocess_prompt=kw.get("preprocess_prompt", True),
|
||||
chunk_duration=kw.get("chunk_duration"),
|
||||
chunk_threshold=kw.get("chunk_threshold"),
|
||||
)
|
||||
self._run_subprocess(argv, stdin_text=text)
|
||||
wav, sr = sf.read(str(out_path))
|
||||
@@ -558,11 +520,6 @@ def _make_backend_class():
|
||||
out_path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
if ref_text_path is not None:
|
||||
try:
|
||||
ref_text_path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
# soundfile returns (n,) for mono or (n, c) for multichannel.
|
||||
# OmniVoice/Higgs Audio v2 is mono → (n,). Wrap to (1, n).
|
||||
@@ -587,13 +544,6 @@ def _make_backend_class():
|
||||
ref_audio: Optional[str],
|
||||
ref_text: Optional[str],
|
||||
language: Optional[str],
|
||||
instruct: Optional[str] = None,
|
||||
duration: Optional[float] = None,
|
||||
seed: Optional[int] = None,
|
||||
denoise: bool = True,
|
||||
preprocess_prompt: bool = True,
|
||||
chunk_duration: Optional[float] = None,
|
||||
chunk_threshold: Optional[float] = None,
|
||||
) -> list[str]:
|
||||
"""Compose argv from typed Path objects only (T-04-02)."""
|
||||
argv: list[str] = [
|
||||
@@ -605,20 +555,6 @@ def _make_backend_class():
|
||||
lang = _iso_to_omnivoice_lang(language)
|
||||
if lang:
|
||||
argv += ["--lang", lang]
|
||||
if instruct:
|
||||
argv += ["--instruct", str(instruct)]
|
||||
if duration is not None:
|
||||
argv += ["--duration", str(float(duration))]
|
||||
if seed is not None:
|
||||
argv += ["--seed", str(int(seed))]
|
||||
if denoise is False:
|
||||
argv += ["--no-denoise"]
|
||||
if preprocess_prompt is False:
|
||||
argv += ["--no-preprocess-prompt"]
|
||||
if chunk_duration is not None:
|
||||
argv += ["--chunk-duration", str(float(chunk_duration))]
|
||||
if chunk_threshold is not None:
|
||||
argv += ["--chunk-threshold", str(float(chunk_threshold))]
|
||||
if ref_audio:
|
||||
# Two-stage validation (defense in depth):
|
||||
# (a) Reject anything outside the project's voices /
|
||||
@@ -652,8 +588,11 @@ def _make_backend_class():
|
||||
)
|
||||
argv += ["--ref-wav", str(ref_path)]
|
||||
if ref_text:
|
||||
# The C++ runtime expects a transcript file path.
|
||||
# generate() creates this file in the system temp dir.
|
||||
# ref_text is free-form text; pass via stdin would
|
||||
# collide with the synthesis prompt, so the only safe
|
||||
# channel is argv. The binary treats this as a quoted
|
||||
# string at the OS layer (Popen escapes argv per
|
||||
# platform); we don't pre-escape.
|
||||
argv += ["--ref-text", str(ref_text)]
|
||||
return argv
|
||||
|
||||
|
||||
+7
-162
@@ -43,13 +43,11 @@ try:
|
||||
_project_env = os.path.join(os.path.dirname(_backend_dir), ".env")
|
||||
if os.path.isfile(_project_env):
|
||||
dotenv.load_dotenv(_project_env, override=False)
|
||||
# Load the durable per-user config (the in-app Settings source of truth) so
|
||||
# env vars set once survive Tauri/Finder launches that don't inherit a shell
|
||||
# environment. This OVERRIDES launcher-injected defaults: the desktop app
|
||||
# injects a stale OMNIVOICE_CACHE_DIR from its own config before startup, so
|
||||
# without override a models dir changed in Settings was ignored forever (#480).
|
||||
from core.user_env import load_into_environ as _load_user_env
|
||||
_load_user_env()
|
||||
# Also load the durable per-user config so env vars set once survive
|
||||
# Tauri/Finder launches that don't inherit a shell environment.
|
||||
_user_env = os.path.expanduser("~/.config/omnivoice/env")
|
||||
if os.path.isfile(_user_env):
|
||||
dotenv.load_dotenv(_user_env, override=False)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
@@ -286,12 +284,7 @@ from fastapi.responses import JSONResponse, RedirectResponse, Response
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from starlette.datastructures import MutableHeaders
|
||||
# Docs-only dependency: a venv created before scalar-fastapi entered the
|
||||
# dependency set must still boot the backend (#307) — /docs degrades instead.
|
||||
try:
|
||||
from scalar_fastapi import get_scalar_api_reference
|
||||
except ImportError:
|
||||
get_scalar_api_reference = None
|
||||
from scalar_fastapi import get_scalar_api_reference
|
||||
import traceback
|
||||
|
||||
_crash_log_lock = threading.Lock()
|
||||
@@ -320,7 +313,6 @@ from api.routers import (
|
||||
setup,
|
||||
gallery,
|
||||
archetypes,
|
||||
describe_voice,
|
||||
community,
|
||||
batch,
|
||||
watermark,
|
||||
@@ -330,10 +322,7 @@ from api.routers import (
|
||||
openai_compat,
|
||||
tts_stream,
|
||||
marketplace,
|
||||
personas,
|
||||
sonitranslate,
|
||||
audiobook,
|
||||
longform_jobs,
|
||||
settings as settings_router, # Phase 1 AUTH-03: HF token save/clear/state
|
||||
)
|
||||
from utils import hf_progress
|
||||
@@ -343,30 +332,6 @@ from utils import hf_progress
|
||||
# the patched class, not the original.
|
||||
hf_progress.install()
|
||||
|
||||
# Wire the overall download aggregator's byte sink onto the patched tqdm so
|
||||
# parallel per-file updates feed one accurate overall bar (FDL-06).
|
||||
try:
|
||||
from utils import download_aggregator
|
||||
download_aggregator.install()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Log the download-acceleration state once at startup (FDL-03) so a slow
|
||||
# download report can be triaged from the logs without reproducing. Note: the
|
||||
# app sets HF_HUB_DISABLE_XET=1 above by default (legacy LFS for byte progress),
|
||||
# so xet_active is normally False even though hf_xet is installed.
|
||||
try:
|
||||
from api.routers.system import _fast_download_status as _fd_status
|
||||
_fd = _fd_status()
|
||||
_xet_ver = f" {_fd['xet_version']}" if _fd.get("xet_version") else ""
|
||||
logging.getLogger("omnivoice.model").info(
|
||||
"downloads: Xet %s (hf_xet%s installed=%s), high_perf=%s",
|
||||
"ACTIVE" if _fd["xet_active"] else "disabled → legacy LFS",
|
||||
_xet_ver, _fd["xet_installed"], _fd["high_performance"],
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _env_flag(name: str, default: bool = False) -> bool:
|
||||
value = os.environ.get(name)
|
||||
@@ -457,23 +422,7 @@ async def lifespan(app: FastAPI):
|
||||
capture_preload_task = asyncio.create_task(_preload_capture_asr())
|
||||
else:
|
||||
logger.info("Capture ASR preload disabled; dictation ASR will load on first use.")
|
||||
|
||||
# ── MCP session manager (Wave 2.2) ────────────────────────────────────
|
||||
# FastMCP's Streamable-HTTP transport needs its session manager running
|
||||
# for the lifetime of the app. It's created lazily by streamable_http_app()
|
||||
# (called in mount_mcp below), so we stack its `run()` context into ours
|
||||
# via AsyncExitStack rather than replacing this lifespan. Best-effort: a
|
||||
# missing/broken MCP layer must never stop the rest of the backend.
|
||||
from contextlib import AsyncExitStack
|
||||
async with AsyncExitStack() as _mcp_stack:
|
||||
_sm = getattr(app.state, "mcp_session_manager", None)
|
||||
if _sm is not None:
|
||||
try:
|
||||
await _mcp_stack.enter_async_context(_sm.run())
|
||||
logger.info("MCP server mounted at /mcp")
|
||||
except Exception as e:
|
||||
logger.warning("MCP session manager failed to start: %s", e)
|
||||
yield
|
||||
yield
|
||||
# ── Graceful shutdown (SIGTERM from Tauri, Ctrl+C, etc.) ────────────
|
||||
logger.info("Shutdown: cleaning up…")
|
||||
idle_task.cancel()
|
||||
@@ -522,14 +471,6 @@ app = FastAPI(
|
||||
@app.get("/docs", include_in_schema=False)
|
||||
async def scalar_docs():
|
||||
"""Interactive API documentation powered by Scalar."""
|
||||
if get_scalar_api_reference is None:
|
||||
return JSONResponse(
|
||||
status_code=503,
|
||||
content={
|
||||
"detail": "API docs unavailable: scalar-fastapi is not installed "
|
||||
"in the backend environment (#307)."
|
||||
},
|
||||
)
|
||||
return get_scalar_api_reference(
|
||||
openapi_url=app.openapi_url,
|
||||
title=app.title,
|
||||
@@ -638,68 +579,6 @@ class NetworkAccessMiddleware:
|
||||
return await self.app(scope, receive, send)
|
||||
|
||||
|
||||
class BearerKeyMiddleware:
|
||||
"""When OMNIVOICE_API_KEY is set, non-loopback clients must present it on
|
||||
every HTTP + WebSocket request: ``Authorization: Bearer <key>``,
|
||||
``?api_key=<key>`` (browser WebSockets cannot set headers), or the
|
||||
``ov_key`` cookie (set on the first successful HTTP auth). Loopback
|
||||
always bypasses — the desktop default is unchanged — and the SPA shell
|
||||
paths stay reachable so a remote UI can load and show what's wrong.
|
||||
|
||||
Inert when the env var is unset (the default). Pure ASGI for the same
|
||||
no-buffering reason as NetworkAccessMiddleware above. Plain-HTTP caveat
|
||||
is documented in docs/remote-gpu.md: the key is sniffable outside a
|
||||
WireGuard (Tailscale) or TLS (tailscale serve) transport.
|
||||
"""
|
||||
|
||||
def __init__(self, app):
|
||||
self.app = app
|
||||
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope["type"] not in ("http", "websocket"):
|
||||
return await self.app(scope, receive, send)
|
||||
key = os.environ.get("OMNIVOICE_API_KEY") or ""
|
||||
if not key:
|
||||
return await self.app(scope, receive, send)
|
||||
client = scope["client"][0] if scope.get("client") else None
|
||||
if client in _LOOPBACK_CLIENTS:
|
||||
return await self.app(scope, receive, send)
|
||||
path = scope.get("path", "")
|
||||
if scope["type"] == "http" and (
|
||||
path in _SHELL_PATHS or path.startswith("/assets/") or path.startswith("/favicon")
|
||||
):
|
||||
return await self.app(scope, receive, send)
|
||||
|
||||
from starlette.requests import HTTPConnection
|
||||
|
||||
conn = HTTPConnection(scope)
|
||||
auth = conn.headers.get("authorization", "")
|
||||
supplied = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
|
||||
if not supplied:
|
||||
supplied = conn.query_params.get("api_key") or conn.cookies.get("ov_key") or ""
|
||||
|
||||
if not secrets.compare_digest(supplied, key):
|
||||
if scope["type"] == "websocket":
|
||||
# Reject the handshake; 1008 = policy violation.
|
||||
await receive() # consume websocket.connect
|
||||
await send({"type": "websocket.close", "code": 1008})
|
||||
return
|
||||
resp = JSONResponse({"detail": "API key required"}, status_code=401)
|
||||
return await resp(scope, receive, send)
|
||||
|
||||
if scope["type"] == "http" and conn.cookies.get("ov_key") != key:
|
||||
async def send_with_cookie(message):
|
||||
if message["type"] == "http.response.start":
|
||||
headers = MutableHeaders(scope=message)
|
||||
headers.append(
|
||||
"set-cookie", f"ov_key={key}; Path=/; SameSite=Lax"
|
||||
)
|
||||
await send(message)
|
||||
|
||||
return await self.app(scope, receive, send_with_cookie)
|
||||
return await self.app(scope, receive, send)
|
||||
|
||||
|
||||
# UI dev-server port — single-sourced from OMNIVOICE_UI_PORT so a user who
|
||||
# moves the Vite dev server off 3901 still gets a matching CORS allow-list.
|
||||
def _ui_port() -> int:
|
||||
@@ -731,15 +610,6 @@ app.add_middleware(
|
||||
# applied even to the 401 PIN-required responses). Inert unless a PIN is set.
|
||||
app.add_middleware(NetworkAccessMiddleware)
|
||||
|
||||
# Remote-backend bearer gate (parity program Wave 2.3 / §R2). Inert unless
|
||||
# OMNIVOICE_API_KEY is set. Distinct from the PIN gate above: the PIN guards
|
||||
# casual LAN-share guests for one session; the API key is the durable
|
||||
# credential for running this backend remotely (Tailscale / Docker GPU box).
|
||||
# Covers WebSockets too — the PIN gate never did, because every WS endpoint
|
||||
# carried its own loopback guard; remote mode is exactly the case where a
|
||||
# keyed non-loopback client must reach them.
|
||||
app.add_middleware(BearerKeyMiddleware)
|
||||
|
||||
app.mount("/audio", StaticFiles(directory=OUTPUTS_DIR), name="audio")
|
||||
app.mount("/voice_audio", StaticFiles(directory=VOICES_DIR), name="voice_audio")
|
||||
|
||||
@@ -782,7 +652,6 @@ app.include_router(stories.router)
|
||||
app.include_router(setup.router)
|
||||
app.include_router(gallery.router)
|
||||
app.include_router(archetypes.router)
|
||||
app.include_router(describe_voice.router) # issue #317: free-text voice design
|
||||
app.include_router(community.router)
|
||||
app.include_router(batch.router)
|
||||
app.include_router(watermark.router)
|
||||
@@ -792,32 +661,8 @@ app.include_router(capture_ws.router)
|
||||
app.include_router(openai_compat.router)
|
||||
app.include_router(tts_stream.router)
|
||||
app.include_router(marketplace.router)
|
||||
app.include_router(personas.router)
|
||||
app.include_router(sonitranslate.router)
|
||||
app.include_router(audiobook.router)
|
||||
app.include_router(longform_jobs.router)
|
||||
app.include_router(settings_router.router) # Phase 1 AUTH-03 endpoints
|
||||
from api.routers import mcp_bindings as _mcp_bindings_router # noqa: E402
|
||||
app.include_router(_mcp_bindings_router.router) # Wave 2.2 per-agent voice bindings
|
||||
|
||||
# ── Mount the MCP server (Wave 2.2) ───────────────────────────────────────
|
||||
# FastMCP's Streamable-HTTP app is sub-mounted at /mcp; its session manager is
|
||||
# stashed on app.state for the lifespan above to run. Opt-out via
|
||||
# OMNIVOICE_MCP_DISABLE=1; best-effort so a missing mcp package or a build
|
||||
# without it never breaks startup.
|
||||
if os.environ.get("OMNIVOICE_MCP_DISABLE", "").strip().lower() not in ("1", "true", "yes", "on"):
|
||||
try:
|
||||
from mcp_server import create_mcp_server
|
||||
|
||||
_mcp = create_mcp_server()
|
||||
_mcp_app = _mcp.streamable_http_app()
|
||||
app.state.mcp_session_manager = _mcp.session_manager
|
||||
app.mount("/mcp", _mcp_app)
|
||||
logging.getLogger("omnivoice.api").info("MCP app mounted at /mcp")
|
||||
except Exception as _mcp_err: # noqa: BLE001
|
||||
logging.getLogger("omnivoice.api").info(
|
||||
"MCP server not mounted (%s); /mcp disabled.", _mcp_err
|
||||
)
|
||||
|
||||
frontend_path = os.path.join(os.path.dirname(__file__), "..", "frontend", "dist")
|
||||
if os.path.exists(frontend_path):
|
||||
|
||||
+1
-64
@@ -53,14 +53,6 @@ def create_mcp_server():
|
||||
"voice design, and video dubbing in 646 languages."
|
||||
),
|
||||
)
|
||||
# Serve the Streamable-HTTP transport at the app root so mounting the whole
|
||||
# app at "/mcp" on the main FastAPI yields the endpoint at "/mcp". FastMCP's
|
||||
# default path is "/mcp", which would double-prefix to "/mcp/mcp" when
|
||||
# sub-mounted. Harmless for the standalone CLI run() path.
|
||||
try:
|
||||
mcp.settings.streamable_http_path = "/"
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# ── Helpers ─────────────────────────────────────────────────────────
|
||||
|
||||
@@ -83,21 +75,6 @@ def create_mcp_server():
|
||||
|
||||
# ── Tools ───────────────────────────────────────────────────────────
|
||||
|
||||
def _current_client_id() -> str | None:
|
||||
"""The X-OmniVoice-Client-Id of the calling MCP client, if any.
|
||||
|
||||
FastMCP exposes the HTTP request via its request context on the
|
||||
Streamable-HTTP transport; stdio clients (and any version where the
|
||||
accessor differs) simply resolve to None and fall back to the
|
||||
global default voice."""
|
||||
try:
|
||||
req = mcp.get_context().request_context.request
|
||||
if req is not None:
|
||||
return req.headers.get("x-omnivoice-client-id")
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
@mcp.tool()
|
||||
async def generate_speech(
|
||||
text: str,
|
||||
@@ -112,8 +89,7 @@ def create_mcp_server():
|
||||
Args:
|
||||
text: The text to synthesize into speech.
|
||||
language: Target language (ISO code or 'Auto'). 646 languages supported.
|
||||
profile_id: ID of a saved voice profile to clone. Omit to use this
|
||||
agent's bound voice (Settings → MCP), else the global default.
|
||||
profile_id: ID of a saved voice profile to clone. Omit for voice design mode.
|
||||
instruct: Style instruction (e.g. 'whisper', 'excited', 'narrator').
|
||||
speed: Speech speed multiplier (0.5–2.0, default 1.0).
|
||||
steps: Diffusion steps (8=fast/draft, 16=balanced, 32=quality).
|
||||
@@ -122,17 +98,6 @@ def create_mcp_server():
|
||||
JSON with audio_id, generation_time, audio_duration, and
|
||||
base64-encoded WAV data.
|
||||
"""
|
||||
# Per-agent voice binding (Wave 2.2): explicit arg wins; otherwise
|
||||
# resolve this client's bound profile, then the global default.
|
||||
client_id = _current_client_id()
|
||||
try:
|
||||
from services import mcp_bindings
|
||||
resolved = mcp_bindings.resolve_voice(client_id, profile_id)
|
||||
profile_id = resolved.get("profile_id")
|
||||
mcp_bindings.touch_last_seen(client_id) if client_id else None
|
||||
except Exception:
|
||||
pass # binding layer unavailable — use whatever was passed
|
||||
|
||||
form = {
|
||||
"text": text,
|
||||
"language": language,
|
||||
@@ -194,34 +159,6 @@ def create_mcp_server():
|
||||
'],"note":"Pass any ISO 639 code or set language=Auto for detection."}'
|
||||
)
|
||||
|
||||
@mcp.tool()
|
||||
async def transcribe(audio_base64: str, language: str | None = None) -> str:
|
||||
"""Transcribe spoken audio to text.
|
||||
|
||||
Args:
|
||||
audio_base64: Base64-encoded audio bytes (wav/mp3/webm/m4a).
|
||||
language: Optional language hint; omit for auto-detect.
|
||||
|
||||
Returns:
|
||||
JSON with the recognized text, language, and duration.
|
||||
"""
|
||||
try:
|
||||
raw = base64.b64decode(audio_base64, validate=True)
|
||||
except Exception:
|
||||
return '{"error":"audio_base64 is not valid base64"}'
|
||||
# 200 MB cap — same spirit as voicebox's transcribe gate. Keeps a
|
||||
# buggy/hostile agent from posting an unbounded blob.
|
||||
if len(raw) > 200 * 1024 * 1024:
|
||||
return '{"error":"audio exceeds 200 MB limit"}'
|
||||
data = {}
|
||||
if language:
|
||||
data["language"] = language
|
||||
r = await _api_post_form(
|
||||
"/transcribe", data=data,
|
||||
files={"audio": ("audio.wav", raw, "application/octet-stream")},
|
||||
)
|
||||
return str(r.json())
|
||||
|
||||
@mcp.tool()
|
||||
async def check_health() -> str:
|
||||
"""Check if the OmniVoice backend is running and what GPU device is active."""
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
"""omnivoice-mcp — stdio MCP shim for clients that only speak stdio."""
|
||||
@@ -1,176 +0,0 @@
|
||||
"""omnivoice-mcp — stdio ↔ Streamable-HTTP MCP proxy (Wave 2.2).
|
||||
|
||||
Adapted from voicebox (https://github.com/jamiepine/voicebox), MIT License,
|
||||
Copyright (c) voicebox contributors.
|
||||
|
||||
Some MCP clients only speak stdio. They spawn this binary; we pipe each
|
||||
JSON-RPC message to ``http://127.0.0.1:<port>/mcp/`` (the FastMCP app mounted
|
||||
on the running OmniVoice backend) and stream the server's response back.
|
||||
|
||||
Environment variables:
|
||||
OMNIVOICE_PORT backend port (default 3900).
|
||||
OMNIVOICE_HOST host (default 127.0.0.1).
|
||||
OMNIVOICE_CLIENT_ID forwarded as X-OmniVoice-Client-Id on every request
|
||||
(drives per-agent voice binding).
|
||||
|
||||
Stdout is JSON-RPC only. Diagnostics go to stderr.
|
||||
Exit 0 on clean EOF, 1 on transport error, 2 if the backend never answers.
|
||||
|
||||
Usage in an MCP client config (stdio):
|
||||
command: python
|
||||
args: ["-m", "backend.mcp_shim"]
|
||||
env: { OMNIVOICE_CLIENT_ID: "claude-code" }
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
CLIENT_ID_HEADER = "X-OmniVoice-Client-Id"
|
||||
SESSION_HEADER = "mcp-session-id"
|
||||
HEALTH_TIMEOUT_S = 30.0
|
||||
DEFAULT_PORT = 3900
|
||||
|
||||
|
||||
def _err(msg: str) -> None:
|
||||
print(f"omnivoice-mcp: {msg}", file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def _base_url() -> tuple[str, str]:
|
||||
host = os.environ.get("OMNIVOICE_HOST", "127.0.0.1")
|
||||
port = int(os.environ.get("OMNIVOICE_PORT", str(DEFAULT_PORT)))
|
||||
return f"http://{host}:{port}/mcp/", f"http://{host}:{port}/health"
|
||||
|
||||
|
||||
async def _wait_for_backend(client: httpx.AsyncClient, health_url: str) -> bool:
|
||||
loop = asyncio.get_running_loop()
|
||||
deadline = loop.time() + HEALTH_TIMEOUT_S
|
||||
while loop.time() < deadline:
|
||||
try:
|
||||
r = await client.get(health_url, timeout=2.0)
|
||||
if r.status_code == 200:
|
||||
return True
|
||||
except Exception:
|
||||
pass
|
||||
await asyncio.sleep(0.5)
|
||||
return False
|
||||
|
||||
|
||||
async def _read_stdin_line() -> str | None:
|
||||
loop = asyncio.get_running_loop()
|
||||
line = await loop.run_in_executor(None, sys.stdin.readline)
|
||||
return line or None
|
||||
|
||||
|
||||
def _write_stdout(obj: Any) -> None:
|
||||
sys.stdout.write(json.dumps(obj, separators=(",", ":")))
|
||||
sys.stdout.write("\n")
|
||||
sys.stdout.flush()
|
||||
|
||||
|
||||
async def _handle_request(
|
||||
client: httpx.AsyncClient,
|
||||
url: str,
|
||||
raw: str,
|
||||
headers: dict[str, str],
|
||||
session_id: list[str | None],
|
||||
) -> None:
|
||||
try:
|
||||
message = json.loads(raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
_err(f"invalid JSON on stdin: {exc}")
|
||||
return
|
||||
|
||||
req_headers = {
|
||||
"Content-Type": "application/json",
|
||||
"Accept": "application/json, text/event-stream",
|
||||
**headers,
|
||||
}
|
||||
if session_id[0]:
|
||||
req_headers[SESSION_HEADER] = session_id[0]
|
||||
|
||||
is_notification = isinstance(message, dict) and "id" not in message
|
||||
|
||||
async with client.stream("POST", url, headers=req_headers, content=raw.encode("utf-8")) as response:
|
||||
if session_id[0] is None:
|
||||
sid = response.headers.get(SESSION_HEADER)
|
||||
if sid:
|
||||
session_id[0] = sid
|
||||
|
||||
if response.status_code == 202:
|
||||
return # notification acknowledged
|
||||
if response.status_code >= 400:
|
||||
body = await response.aread()
|
||||
_err(f"server {response.status_code}: {body.decode('utf-8', errors='replace')[:400]}")
|
||||
if is_notification:
|
||||
return
|
||||
_write_stdout({
|
||||
"jsonrpc": "2.0",
|
||||
"id": message.get("id"),
|
||||
"error": {"code": -32000, "message": f"OmniVoice MCP proxy got HTTP {response.status_code}"},
|
||||
})
|
||||
return
|
||||
|
||||
ctype = response.headers.get("content-type", "")
|
||||
if "text/event-stream" in ctype:
|
||||
async for line in response.aiter_lines():
|
||||
if line.startswith("data:"):
|
||||
payload = line[5:].strip()
|
||||
if not payload:
|
||||
continue
|
||||
try:
|
||||
_write_stdout(json.loads(payload))
|
||||
except json.JSONDecodeError:
|
||||
_err(f"malformed SSE payload: {payload[:200]}")
|
||||
else:
|
||||
body = await response.aread()
|
||||
try:
|
||||
_write_stdout(json.loads(body))
|
||||
except json.JSONDecodeError:
|
||||
_err(f"non-JSON response ({ctype}): {body.decode('utf-8', errors='replace')[:200]}")
|
||||
|
||||
|
||||
async def _run() -> int:
|
||||
url, health_url = _base_url()
|
||||
forward_headers: dict[str, str] = {}
|
||||
client_id = os.environ.get("OMNIVOICE_CLIENT_ID")
|
||||
if client_id:
|
||||
forward_headers[CLIENT_ID_HEADER] = client_id
|
||||
|
||||
session_id: list[str | None] = [None]
|
||||
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(300.0)) as client:
|
||||
if not await _wait_for_backend(client, health_url):
|
||||
_err(f"timed out waiting for OmniVoice at {health_url} — is the app running?")
|
||||
return 2
|
||||
try:
|
||||
while True:
|
||||
line = await _read_stdin_line()
|
||||
if line is None:
|
||||
return 0
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
await _handle_request(client, url, line, forward_headers, session_id)
|
||||
except (KeyboardInterrupt, SystemExit):
|
||||
return 0
|
||||
except Exception as exc:
|
||||
_err(f"proxy failed: {exc!r}")
|
||||
return 1
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
return asyncio.run(_run())
|
||||
except KeyboardInterrupt:
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,58 +0,0 @@
|
||||
"""Parity program Wave 0.2: consent-locked voice profiles
|
||||
|
||||
Revision ID: 0003_voice_profile_consent
|
||||
Revises: 0002_voice_profile_demo_fields
|
||||
Create Date: 2026-06-12 00:00:00.000000
|
||||
|
||||
Adds four additive columns to ``voice_profiles`` backing the
|
||||
``verified_own_voice`` consent lock (docs/competitive-analysis.md Action 22 /
|
||||
parity program Wave 0.2). A profile becomes "verified" when its owner records
|
||||
a spoken consent statement; agentic features and gallery sharing will require
|
||||
the flag — plain local synthesis never does.
|
||||
|
||||
* ``verified_own_voice INTEGER DEFAULT 0`` — the consent lock itself.
|
||||
* ``consent_text TEXT DEFAULT ''`` — the statement that was read aloud.
|
||||
* ``consent_audio_path TEXT DEFAULT ''`` — filename of the recorded
|
||||
statement in VOICES_DIR (kept as provenance, deletable via revoke).
|
||||
* ``consent_recorded_at REAL DEFAULT NULL`` — UNIX timestamp.
|
||||
|
||||
Behavior mirrors 0002: ``_has_column`` PRAGMA guards make upgrade a no-op on
|
||||
fresh installs (where _BASE_SCHEMA already has the columns), satisfying the
|
||||
"Backward-compatible project data" constraint; downgrade drops the columns
|
||||
(SQLite >= 3.35).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = "0003_voice_profile_consent"
|
||||
down_revision: Union[str, None] = "0002_voice_profile_demo_fields"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
_COLUMNS = (
|
||||
("verified_own_voice", sa.Column("verified_own_voice", sa.Integer(), nullable=False, server_default="0")),
|
||||
("consent_text", sa.Column("consent_text", sa.Text(), nullable=False, server_default="")),
|
||||
("consent_audio_path", sa.Column("consent_audio_path", sa.Text(), nullable=False, server_default="")),
|
||||
("consent_recorded_at", sa.Column("consent_recorded_at", sa.Float(), nullable=True)),
|
||||
)
|
||||
|
||||
|
||||
def _has_column(table: str, column: str) -> bool:
|
||||
bind = op.get_bind()
|
||||
rows = bind.execute(sa.text(f"PRAGMA table_info({table})")).fetchall()
|
||||
return any(r[1] == column for r in rows)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
for name, column in _COLUMNS:
|
||||
if not _has_column("voice_profiles", name):
|
||||
op.add_column("voice_profiles", column)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
for name, _ in reversed(_COLUMNS):
|
||||
if _has_column("voice_profiles", name):
|
||||
op.drop_column("voice_profiles", name)
|
||||
@@ -1,60 +0,0 @@
|
||||
"""Parity program Wave 2.2: per-agent MCP voice bindings
|
||||
|
||||
Revision ID: 0004_mcp_client_bindings
|
||||
Revises: 0003_voice_profile_consent
|
||||
Create Date: 2026-06-12 00:00:00.000000
|
||||
|
||||
Adds the ``mcp_client_bindings`` table backing per-agent voice binding
|
||||
(docs/competitive-analysis.md Spec 2): each MCP client (identified by the
|
||||
``X-OmniVoice-Client-Id`` header it sends) can be bound to a default voice
|
||||
profile / engine, so "Claude Code speaks in Morgan, Cursor in Scarlett".
|
||||
|
||||
* ``client_id`` TEXT PRIMARY KEY — the agent's stable id.
|
||||
* ``label`` TEXT — human label shown in Settings.
|
||||
* ``profile_id`` TEXT — voice profile to speak in (nullable FK-by-convention).
|
||||
* ``default_engine`` TEXT — engine override (nullable).
|
||||
* ``last_seen_at`` REAL — updated when the client calls a tool.
|
||||
* ``created_at`` REAL.
|
||||
|
||||
Additive + idempotent (guarded by sqlite_master), matching 0002/0003, so
|
||||
re-running on a fresh-install DB where _BASE_SCHEMA already created it is a
|
||||
no-op (Backward-compatible project data constraint).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = "0004_mcp_client_bindings"
|
||||
down_revision: Union[str, None] = "0003_voice_profile_consent"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def _has_table(name: str) -> bool:
|
||||
bind = op.get_bind()
|
||||
row = bind.execute(
|
||||
sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name=:n"),
|
||||
{"n": name},
|
||||
).fetchone()
|
||||
return row is not None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if _has_table("mcp_client_bindings"):
|
||||
return
|
||||
op.create_table(
|
||||
"mcp_client_bindings",
|
||||
sa.Column("client_id", sa.Text(), primary_key=True),
|
||||
sa.Column("label", sa.Text(), nullable=False, server_default=""),
|
||||
sa.Column("profile_id", sa.Text(), nullable=True),
|
||||
sa.Column("default_engine", sa.Text(), nullable=True),
|
||||
sa.Column("last_seen_at", sa.Float(), nullable=True),
|
||||
sa.Column("created_at", sa.Float(), nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_table("mcp_client_bindings"):
|
||||
op.drop_table("mcp_client_bindings")
|
||||
@@ -1,65 +0,0 @@
|
||||
"""Voice Studio unification: profile `kind` discriminator + design params
|
||||
|
||||
Revision ID: 0005_unified_profiles
|
||||
Revises: 0004_mcp_client_bindings
|
||||
Create Date: 2026-06-13 00:00:00.000000
|
||||
|
||||
Adds two additive columns to ``voice_profiles`` so a *designed* voice
|
||||
(category sliders + instruct, no user reference audio) is a first-class
|
||||
profile rather than a transient UI state
|
||||
(docs/specs/voice-studio-unification.md §3):
|
||||
|
||||
* ``kind TEXT DEFAULT 'clone'`` — ``'clone'`` (user reference audio) or
|
||||
``'design'`` (rendered sample + stored design params). Replaces the
|
||||
brittle is_locked/instruct inference in /generate.
|
||||
* ``vd_states TEXT DEFAULT NULL`` — JSON of the design category picks
|
||||
(Gender/Age/Pitch/Style/accent/dialect) so selecting a design profile
|
||||
can restore the sliders for re-editing.
|
||||
|
||||
Backfill: every existing row becomes ``kind='clone'`` — all of them carry a
|
||||
real or rendered ``ref_audio_path`` today (archetype materialization
|
||||
included), so the default is semantically true and no audio is re-rendered.
|
||||
|
||||
Behavior mirrors 0002/0003: ``_has_column`` PRAGMA guards make upgrade a
|
||||
no-op on fresh installs (where _BASE_SCHEMA already has the columns),
|
||||
satisfying the "Backward-compatible project data" constraint; downgrade
|
||||
drops the columns (SQLite >= 3.35).
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision: str = "0005_unified_profiles"
|
||||
down_revision: Union[str, None] = "0004_mcp_client_bindings"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def _has_column(table: str, column: str) -> bool:
|
||||
bind = op.get_bind()
|
||||
rows = bind.execute(sa.text(f"PRAGMA table_info({table})")).fetchall()
|
||||
return any(r[1] == column for r in rows)
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if not _has_column("voice_profiles", "kind"):
|
||||
op.add_column(
|
||||
"voice_profiles",
|
||||
sa.Column("kind", sa.Text(), nullable=False, server_default="clone"),
|
||||
)
|
||||
# server_default covers new rows; make existing rows explicit too.
|
||||
op.execute("UPDATE voice_profiles SET kind='clone' WHERE kind IS NULL OR kind=''")
|
||||
if not _has_column("voice_profiles", "vd_states"):
|
||||
op.add_column(
|
||||
"voice_profiles",
|
||||
sa.Column("vd_states", sa.Text(), nullable=True),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if _has_column("voice_profiles", "vd_states"):
|
||||
op.drop_column("voice_profiles", "vd_states")
|
||||
if _has_column("voice_profiles", "kind"):
|
||||
op.drop_column("voice_profiles", "kind")
|
||||
@@ -25,11 +25,6 @@ class DubSegment(BaseModel):
|
||||
speed: Optional[float] = None
|
||||
gain: Optional[float] = None # Per-segment volume (0.0 - 2.0, default 1.0)
|
||||
target_lang: Optional[str] = None # Per-segment language override (ISO code)
|
||||
# Phase 4.2 free-form directorial note ("urgent, whispered…"). The client
|
||||
# has always sent this; without the field pydantic silently dropped it,
|
||||
# so directions never reached TTS and never entered the regen
|
||||
# fingerprint (#281).
|
||||
direction: Optional[str] = None
|
||||
effect_preset: str = "broadcast" # NEW: DSP preset id (default: broadcast)
|
||||
|
||||
@field_validator("effect_preset")
|
||||
@@ -42,19 +37,6 @@ class DubSegment(BaseModel):
|
||||
)
|
||||
return v
|
||||
|
||||
class FitOptions(BaseModel):
|
||||
"""Optional knob overrides for the `smart_fit` timing strategy.
|
||||
|
||||
All fields default to None — the server fills in the canonical
|
||||
defaults (services.fit_planner.FitParams) so old clients and sparse
|
||||
payloads behave identically to a fully-populated default payload.
|
||||
"""
|
||||
max_audio_only_rate: Optional[float] = None # default 1.2
|
||||
audio_rate_cap: Optional[float] = None # default 1.5
|
||||
video_slow_cap: Optional[float] = None # default 2.0
|
||||
gap_guard_s: Optional[float] = None # default 0.05
|
||||
allow_video_retime: Optional[bool] = None # default True
|
||||
|
||||
class DubRequest(BaseModel):
|
||||
segments: List[DubSegment]
|
||||
language: str = "Auto"
|
||||
@@ -93,35 +75,18 @@ class DubRequest(BaseModel):
|
||||
# each segment's video portion is stretched (via
|
||||
# ffmpeg setpts) to fit the natural-rate dub audio.
|
||||
# Audio plays at 1.0×; total video duration grows.
|
||||
# "smart_fit" — dub-length fitting v2: split the burden between a
|
||||
# mild pitch-preserving audio speed-up (≤1.2× alone,
|
||||
# ≤1.5× in hybrid) and a mild per-segment video
|
||||
# slow-down (≤2.0×), per services/fit_planner.py.
|
||||
# Residual overflow is trimmed and surfaced.
|
||||
# "strict_slot" — legacy: keep `slot_fit` semantics (atempo squeeze
|
||||
# when audio > slot). Kept for back-compat.
|
||||
timing_strategy: Optional[Literal["concise", "stretch_video", "strict_slot", "smart_fit"]] = "concise"
|
||||
timing_strategy: Optional[Literal["concise", "stretch_video", "strict_slot"]] = "concise"
|
||||
|
||||
# Per-job slip budget for "concise" mode. Hard-trim only kicks in once
|
||||
# gap absorption + this much extra time has been consumed.
|
||||
overflow_budget_s: Optional[float] = 0.0
|
||||
|
||||
# Knob overrides for `smart_fit` (ignored by other strategies). Omitted
|
||||
# fields default server-side to fit_planner.FitParams values.
|
||||
fit_options: Optional[FitOptions] = None
|
||||
|
||||
class TranslateSegment(BaseModel):
|
||||
id: str
|
||||
text: str
|
||||
target_lang: Optional[str] = None
|
||||
# Free-form delivery direction ("urgent, whispering") — feeds the
|
||||
# cinematic reflect/adapt prompts. The frontend has sent this since
|
||||
# Phase 4.2 but pydantic silently dropped it as an undeclared extra,
|
||||
# so the per-segment direction hint never reached the LLM.
|
||||
direction: Optional[str] = None
|
||||
# Available time slot (end - start, seconds) for rate-ratio prediction
|
||||
# and the cinematic slot-fit pass. Same silent-drop fix as `direction`.
|
||||
slot_seconds: Optional[float] = None
|
||||
|
||||
class TranslateRequest(BaseModel):
|
||||
segments: List[TranslateSegment]
|
||||
@@ -131,12 +96,6 @@ class TranslateRequest(BaseModel):
|
||||
job_id: Optional[str] = None # Dub job id, used to resolve detected source_lang
|
||||
quality: Optional[str] = "fast" # "fast" (one-shot) | "cinematic" (reflect → adapt)
|
||||
glossary: Optional[List[dict]] = None # [{"source": "...", "target": "...", "note": "..."}]
|
||||
# Optional regional dialect (BCP-47, e.g. "es-AR", "pt-BR") — #280 item 2.
|
||||
# Applied by LLM-backed paths (provider="openai" or quality="cinematic"):
|
||||
# the prompt asks for that region's vocabulary/grammar (e.g. Argentinian
|
||||
# voseo: "vos sos" instead of "tú eres"). Non-LLM providers (Argos, NLLB,
|
||||
# Google) can't honor it; the response then carries dialect_applied=false.
|
||||
dialect: Optional[str] = None
|
||||
|
||||
class DubIngestUrlRequest(BaseModel):
|
||||
url: str
|
||||
|
||||
@@ -1,281 +0,0 @@
|
||||
"""Acoustic echo cancellation for dictate-over-playback (parity Action 8b).
|
||||
|
||||
When the user dictates while OmniVoice is *playing* audio (a TTS preview, a
|
||||
dub render, a video), the loudspeaker signal leaks back into the microphone.
|
||||
The streaming ASR on ``/ws/transcribe`` then transcribes that bleed as if it
|
||||
were speech — the "it typed back what the app just said" symptom. A browser's
|
||||
``getUserMedia({echoCancellation:true})`` would help, but its quality and even
|
||||
its availability differ per platform/webview, which would make a *default*
|
||||
feature behave differently on macOS/Windows/Linux — against the project's
|
||||
cross-platform-parity rule. A server-side canceller behaves identically
|
||||
everywhere, so it is the local-first, platform-neutral choice.
|
||||
|
||||
This is an NLMS (normalised least-mean-squares) time-domain adaptive filter
|
||||
with a Geigel double-talk detector. It is a clean-room-grade *port* of
|
||||
Patter's ``getpatter/audio/aec.py`` (MIT) — see docs/competitive-analysis.md,
|
||||
Action 8. It is NOT production-grade DSP (WebRTC AEC3 / Speex AEC are); it is
|
||||
a dependency-free, good-enough canceller that removes the steady-state echo
|
||||
the ASR would otherwise hallucinate on.
|
||||
|
||||
Wiring (one instance per ``/ws/transcribe`` session — NOT thread-safe)::
|
||||
|
||||
aec = NlmsEchoCanceller(sample_rate=16000)
|
||||
# Far-end: every PCM chunk the client is about to play through speakers.
|
||||
aec.push_far_end(playback_pcm_bytes)
|
||||
# Near-end: the mic PCM, cleaned before it reaches the ASR buffer.
|
||||
cleaned = aec.process_near_end(mic_pcm_bytes)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Final
|
||||
|
||||
import numpy as np
|
||||
|
||||
logger = logging.getLogger("omnivoice.aec")
|
||||
|
||||
|
||||
_DEFAULT_FILTER_TAPS: Final[int] = 512
|
||||
"""Adaptive-filter length in samples. 512 taps @ 16 kHz = 32 ms, covering a
|
||||
typical near-field laptop/desktop echo path. Longer tails (large rooms) can
|
||||
pass ``filter_taps=1024``+ at proportionally more CPU per frame; 512 converges
|
||||
in ~0.5 s with the warm-up ramp and is the sweet spot for dictation."""
|
||||
|
||||
_DEFAULT_STEP_SIZE: Final[float] = 0.1
|
||||
"""Steady-state NLMS step size. Larger = faster channel tracking but less
|
||||
stable; 0.1 is the textbook value for narrowband voice."""
|
||||
|
||||
_DEFAULT_WARMUP_STEP_SIZE: Final[float] = 0.5
|
||||
"""Aggressive step used during the warm-up window so the filter reaches a
|
||||
usable echo estimate within ~0.5 s instead of several seconds. The Geigel
|
||||
double-talk detector still gates updates, so the bigger step does not learn
|
||||
the user's own voice as echo."""
|
||||
|
||||
_DEFAULT_WARMUP_SECONDS: Final[float] = 0.5
|
||||
"""Length of the warm-up window. After this many seconds of processed
|
||||
near-end audio the step decays from ``warmup_step_size`` to ``step_size``."""
|
||||
|
||||
_DEFAULT_LEAKAGE: Final[float] = 0.9999
|
||||
"""Per-iteration weight leakage (slightly < 1) so the filter slowly forgets
|
||||
stale taps when the echo path drifts (the user moves the mic)."""
|
||||
|
||||
_DOUBLE_TALK_RHO: Final[float] = 0.6
|
||||
"""Geigel double-talk threshold. When ``max(|near|) > rho * max(|far|)`` the
|
||||
near-end carries energy the far-end cannot explain (the user is talking) →
|
||||
freeze adaptation so the filter does not model the user's voice as echo."""
|
||||
|
||||
_FAR_END_BUFFER_SECONDS: Final[float] = 0.5
|
||||
"""How much past far-end (playback) audio to retain. The echo arrives at the
|
||||
mic tens of ms after playback; the filter needs that much look-back to align.
|
||||
500 ms is generous headroom."""
|
||||
|
||||
|
||||
class NlmsEchoCanceller:
|
||||
"""Time-domain NLMS adaptive filter with Geigel double-talk detection.
|
||||
|
||||
Operates on narrowband mono PCM at 16 kHz (the rate the dictation path
|
||||
resamples to) or 8 kHz. Not thread-safe — each ``/ws/transcribe`` session
|
||||
owns its own instance.
|
||||
"""
|
||||
|
||||
# Far-end staleness window (seconds): once the most recent far-end push
|
||||
# is older than this, ``process_near_end`` passes the mic through instead
|
||||
# of cancelling against a frozen reference (which would superimpose the
|
||||
# same stale ~50 ms waveform on every mic frame as an audible buzz).
|
||||
_FAR_STALE_S: float = 0.25
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
sample_rate: int = 16000,
|
||||
*,
|
||||
filter_taps: int = _DEFAULT_FILTER_TAPS,
|
||||
step_size: float = _DEFAULT_STEP_SIZE,
|
||||
warmup_step_size: float = _DEFAULT_WARMUP_STEP_SIZE,
|
||||
warmup_seconds: float = _DEFAULT_WARMUP_SECONDS,
|
||||
leakage: float = _DEFAULT_LEAKAGE,
|
||||
double_talk_rho: float = _DOUBLE_TALK_RHO,
|
||||
) -> None:
|
||||
if sample_rate not in (8000, 16000):
|
||||
raise ValueError(
|
||||
"NlmsEchoCanceller supports 8000 Hz or 16000 Hz only; "
|
||||
f"got {sample_rate}."
|
||||
)
|
||||
if filter_taps < 64:
|
||||
raise ValueError(
|
||||
f"filter_taps must be >= 64 to model a meaningful echo path; "
|
||||
f"got {filter_taps}."
|
||||
)
|
||||
if not 0 < step_size <= 1:
|
||||
raise ValueError(f"step_size must be in (0, 1]; got {step_size}.")
|
||||
if not 0 < warmup_step_size <= 1:
|
||||
raise ValueError(
|
||||
f"warmup_step_size must be in (0, 1]; got {warmup_step_size}."
|
||||
)
|
||||
if warmup_seconds < 0:
|
||||
raise ValueError(f"warmup_seconds must be >= 0; got {warmup_seconds}.")
|
||||
if not 0 < leakage <= 1:
|
||||
raise ValueError(f"leakage must be in (0, 1]; got {leakage}.")
|
||||
|
||||
self._sample_rate = sample_rate
|
||||
self._taps = filter_taps
|
||||
self._step = float(step_size)
|
||||
self._warmup_step = float(warmup_step_size)
|
||||
self._warmup_samples = int(warmup_seconds * sample_rate)
|
||||
self._leakage = float(leakage)
|
||||
self._rho = float(double_talk_rho)
|
||||
# Counts near-end samples processed so the step can taper from
|
||||
# warmup_step to step over the first warmup_samples. Counted from the
|
||||
# first process_near_end call so the window aligns with playback start.
|
||||
self._processed_samples: int = 0
|
||||
self._last_far_push_monotonic: float | None = None
|
||||
|
||||
# Filter coefficients (zeros — adapts to the channel within ~0.5–2 s).
|
||||
self._w = np.zeros(filter_taps, dtype=np.float32)
|
||||
|
||||
# Far-end ring buffer holding >= filter_taps samples of playback
|
||||
# history, with headroom so push/process can interleave freely.
|
||||
max_buf_samples = max(
|
||||
filter_taps * 2,
|
||||
int(sample_rate * _FAR_END_BUFFER_SECONDS),
|
||||
)
|
||||
self._far_buf = np.zeros(max_buf_samples, dtype=np.float32)
|
||||
self._far_write_idx = 0 # next write position (head)
|
||||
self._far_filled = 0 # samples written so far (capped at len(far_buf))
|
||||
|
||||
# Diagnostics only — never read in the hot path.
|
||||
self.frames_processed: int = 0
|
||||
self.double_talk_frames: int = 0
|
||||
|
||||
# ── Public API ──────────────────────────────────────────────────────────
|
||||
|
||||
def push_far_end(self, pcm_bytes: bytes) -> None:
|
||||
"""Append far-end (playback) audio to the reference ring buffer.
|
||||
|
||||
Accepts raw int16 little-endian mono PCM at the configured rate.
|
||||
"""
|
||||
if not pcm_bytes:
|
||||
return
|
||||
self._last_far_push_monotonic = time.monotonic()
|
||||
samples = np.frombuffer(pcm_bytes, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
n = samples.shape[0]
|
||||
buf_len = self._far_buf.shape[0]
|
||||
if n >= buf_len:
|
||||
# More than the buffer holds — keep only the newest buf_len.
|
||||
self._far_buf[:] = samples[-buf_len:]
|
||||
self._far_write_idx = 0
|
||||
self._far_filled = buf_len
|
||||
return
|
||||
end = self._far_write_idx + n
|
||||
if end <= buf_len:
|
||||
self._far_buf[self._far_write_idx:end] = samples
|
||||
else:
|
||||
head = buf_len - self._far_write_idx
|
||||
self._far_buf[self._far_write_idx:] = samples[:head]
|
||||
self._far_buf[: n - head] = samples[head:]
|
||||
self._far_write_idx = (self._far_write_idx + n) % buf_len
|
||||
self._far_filled = min(self._far_filled + n, buf_len)
|
||||
|
||||
def process_near_end(self, pcm_bytes: bytes) -> bytes:
|
||||
"""Subtract the estimated echo from the near-end (mic) signal.
|
||||
|
||||
Returns int16 little-endian mono PCM with the estimated echo removed.
|
||||
Passes the frame through unchanged when there is nothing worth
|
||||
cancelling: no playback has been primed, or the far-end reference is
|
||||
stale (the app went silent).
|
||||
"""
|
||||
if not pcm_bytes:
|
||||
return pcm_bytes
|
||||
|
||||
# Not enough far-end history to fill the filter window yet — passing
|
||||
# through avoids emitting garbage on the first frames.
|
||||
if self._far_filled < self._taps:
|
||||
return pcm_bytes
|
||||
|
||||
# Far-end reference is stale (app stopped playing): the ring only
|
||||
# advances on push_far_end, so the "most recent" window is frozen at
|
||||
# the tail of the last playback. Convolving against it would buzz.
|
||||
last_push = self._last_far_push_monotonic
|
||||
if last_push is None or (time.monotonic() - last_push) > self._FAR_STALE_S:
|
||||
return pcm_bytes
|
||||
|
||||
near = np.frombuffer(pcm_bytes, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
cleaned = self._block_nlms(near)
|
||||
out = np.clip(cleaned * 32768.0, -32768.0, 32767.0).astype(np.int16)
|
||||
self.frames_processed += 1
|
||||
return out.tobytes()
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Clear filter coefficients and far-end history (e.g. on a new turn)."""
|
||||
self._w.fill(0)
|
||||
self._far_buf.fill(0)
|
||||
self._far_write_idx = 0
|
||||
self._far_filled = 0
|
||||
self._processed_samples = 0
|
||||
self._last_far_push_monotonic = None
|
||||
self.frames_processed = 0
|
||||
self.double_talk_frames = 0
|
||||
|
||||
# ── Internals ───────────────────────────────────────────────────────────
|
||||
|
||||
def _far_window(self, length: int) -> np.ndarray:
|
||||
"""Most recent ``length`` far-end samples, oldest first / newest last."""
|
||||
buf_len = self._far_buf.shape[0]
|
||||
if length > self._far_filled:
|
||||
length = self._far_filled
|
||||
end = self._far_write_idx # newest sample is at (end - 1) mod buf_len
|
||||
if end >= length:
|
||||
return self._far_buf[end - length: end]
|
||||
head = self._far_buf[buf_len - (length - end):]
|
||||
tail = self._far_buf[:end]
|
||||
return np.concatenate((head, tail))
|
||||
|
||||
def _block_nlms(self, near: np.ndarray) -> np.ndarray:
|
||||
"""Sample-by-sample NLMS over one frame of near-end samples.
|
||||
|
||||
Classical NLMS depends on the weights adapted at the previous sample,
|
||||
so the inner loop is sequential. Each sample is O(taps); numpy keeps a
|
||||
320-sample / 512-tap frame well under a millisecond on commodity CPUs.
|
||||
"""
|
||||
taps = self._taps
|
||||
far_window = self._far_window(taps + near.shape[0] - 1)
|
||||
if far_window.shape[0] < taps + near.shape[0] - 1:
|
||||
# Still warming up — left-pad with zeros so indices line up.
|
||||
pad = np.zeros(
|
||||
taps + near.shape[0] - 1 - far_window.shape[0], dtype=np.float32
|
||||
)
|
||||
far_window = np.concatenate((pad, far_window))
|
||||
|
||||
# Geigel double-talk detector (frame-wise).
|
||||
far_max = float(np.max(np.abs(far_window))) if far_window.size else 0.0
|
||||
near_max = float(np.max(np.abs(near)))
|
||||
# Freeze adaptation when the far reference is effectively silent
|
||||
# (<= -60 dBFS): adapting against a fade-out tail with near-zero norm
|
||||
# blows the weights up against user speech when playback resumes.
|
||||
if far_max <= 1e-3:
|
||||
return near
|
||||
double_talk = near_max > self._rho * far_max
|
||||
if double_talk:
|
||||
self.double_talk_frames += 1
|
||||
|
||||
out = np.empty_like(near)
|
||||
w = self._w
|
||||
leakage = self._leakage
|
||||
# Constant step within the frame keeps the inner loop branch-free.
|
||||
if self._processed_samples < self._warmup_samples:
|
||||
step = self._warmup_step
|
||||
else:
|
||||
step = self._step
|
||||
for i in range(near.shape[0]):
|
||||
x = far_window[i: i + taps]
|
||||
y_est = float(np.dot(w, x))
|
||||
e = float(near[i] - y_est)
|
||||
out[i] = e
|
||||
if not double_talk:
|
||||
# NLMS update with leakage. +1e-6 guards divide-by-zero.
|
||||
norm = float(np.dot(x, x)) + 1e-6
|
||||
w *= leakage
|
||||
w += (step * e / norm) * x
|
||||
self._processed_samples += near.shape[0]
|
||||
return out
|
||||
+11
-232
@@ -31,74 +31,12 @@ from abc import ABC, abstractmethod
|
||||
logger = logging.getLogger("omnivoice.asr")
|
||||
|
||||
|
||||
def _decode_audio_16k_mono(audio_path: str):
|
||||
"""Decode `audio_path` to a 16 kHz mono float32 waveform using OmniVoice's
|
||||
*validated* ffmpeg, instead of whisperx.load_audio's bare ``"ffmpeg"`` PATH
|
||||
lookup.
|
||||
|
||||
whisperx (and openai-whisper) shell out to a literal ``"ffmpeg"`` resolved
|
||||
against the OS PATH. On Windows that resolves to whatever the system finds
|
||||
first — a WindowsApps alias stub or a corrupt/wrong-arch download — which
|
||||
passes `which` but explodes at spawn with ``[WinError 193] %1 is not a valid
|
||||
Win32 application``. whisperx only catches `CalledProcessError`, so the
|
||||
spawn-time `OSError` escapes and the dub/batch path reports the opaque
|
||||
"Transcription produced no segments" (#479). ``find_ffmpeg()`` probes each
|
||||
candidate with ``-version`` and returns a runnable binary (the bundled
|
||||
imageio-ffmpeg / Tauri sidecar) — or None, so we can raise an actionable
|
||||
error. This also fixes the imageio case a PATH-prepend can't: its binary is
|
||||
named ``ffmpeg-<plat>-vN.exe``, not ``ffmpeg``, so bare lookup never finds
|
||||
it. Mirrors whisperx.audio.load_audio's command exactly (16 kHz, mono, s16le).
|
||||
"""
|
||||
import subprocess
|
||||
|
||||
import numpy as np
|
||||
|
||||
from services.ffmpeg_utils import find_ffmpeg
|
||||
|
||||
ffmpeg = find_ffmpeg()
|
||||
if not ffmpeg:
|
||||
raise RuntimeError(
|
||||
"Cannot transcribe: ffmpeg is missing or not runnable. Install "
|
||||
"ffmpeg (or let OmniVoice's bundled binary download), then retry. "
|
||||
"On Windows a '[WinError 193]' here means the ffmpeg binary is "
|
||||
"corrupt or the wrong architecture — reinstall it or clear the "
|
||||
"imageio-ffmpeg cache."
|
||||
)
|
||||
cmd = [
|
||||
ffmpeg, "-nostdin", "-threads", "0", "-i", audio_path,
|
||||
"-f", "s16le", "-ac", "1", "-acodec", "pcm_s16le", "-ar", "16000", "-",
|
||||
]
|
||||
try:
|
||||
out = subprocess.run(cmd, capture_output=True, check=True).stdout
|
||||
except OSError as e:
|
||||
# Belt-and-suspenders: find_ffmpeg() already -version-validated this
|
||||
# binary, so a WinError 193 here is unexpected — surface it clearly
|
||||
# rather than letting it become "no segments".
|
||||
raise RuntimeError(
|
||||
f"ffmpeg at {ffmpeg!r} could not be executed ({e}). Reinstall "
|
||||
"ffmpeg or clear the imageio-ffmpeg cache."
|
||||
) from e
|
||||
except subprocess.CalledProcessError as e:
|
||||
stderr = (e.stderr or b"").decode(errors="replace")[:500]
|
||||
raise RuntimeError(f"Failed to decode audio for transcription: {stderr}") from e
|
||||
return np.frombuffer(out, np.int16).flatten().astype(np.float32) / 32768.0
|
||||
|
||||
|
||||
# ── Protocol ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class ASRBackend(ABC):
|
||||
id: str = "base"
|
||||
display_name: str = "Base ASR"
|
||||
# Accelerator families this backend can use, in preference order; always
|
||||
# includes a fallback. Subset of {cuda, rocm, mps, xpu, cpu}. Mirrors the
|
||||
# TTSBackend.gpu_compat contract so engine_routing.resolve_routing() can
|
||||
# surface the effective device per host (no silent CPU fallback). The
|
||||
# conservative default is CPU-only; subclasses declare what they really run
|
||||
# on. (ROCm is intentionally NOT claimed yet for any ASR engine — see the
|
||||
# per-engine notes; an unverified `rocm` claim would route ROCm hosts to a
|
||||
# broken GPU path, strictly worse than the honest `cpu_fallback`.)
|
||||
gpu_compat: tuple[str, ...] = ("cpu",)
|
||||
|
||||
@classmethod
|
||||
@abstractmethod
|
||||
@@ -123,10 +61,6 @@ class ASRBackend(ABC):
|
||||
class WhisperXBackend(ASRBackend):
|
||||
id = "whisperx"
|
||||
display_name = "WhisperX (faster-whisper + wav2vec2 forced alignment)"
|
||||
# CTranslate2 backend: CUDA fp16 or CPU int8 (see _pick_device). ROCm not
|
||||
# claimed — CTranslate2 has no upstream HIP build, so a ROCm host honestly
|
||||
# gets cpu_fallback rather than a false GPU promise.
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
def __init__(self):
|
||||
self._model_name = os.environ.get("ASR_MODEL_WHISPERX", "large-v3")
|
||||
@@ -379,13 +313,10 @@ class WhisperXBackend(ASRBackend):
|
||||
return None
|
||||
|
||||
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
|
||||
import whisperx # used for whisperx.align() below
|
||||
import whisperx
|
||||
self._ensure_asr()
|
||||
logger.info("whisperx transcribing %s (word_timestamps=%s)", audio_path, word_timestamps)
|
||||
# Decode via OmniVoice's validated ffmpeg, NOT whisperx.load_audio's bare
|
||||
# "ffmpeg" PATH lookup which yields [WinError 193] -> "no segments" on
|
||||
# Windows (#479). Same 16 kHz mono s16le array whisperx expects.
|
||||
audio = _decode_audio_16k_mono(audio_path)
|
||||
audio = whisperx.load_audio(audio_path)
|
||||
try:
|
||||
result = self._asr.transcribe(audio)
|
||||
except IndexError:
|
||||
@@ -451,8 +382,6 @@ class WhisperXBackend(ASRBackend):
|
||||
class FasterWhisperBackend(ASRBackend):
|
||||
id = "faster-whisper"
|
||||
display_name = "Faster-Whisper (CTranslate2 — Linux/Windows/macOS)"
|
||||
# CTranslate2: CUDA or CPU (no upstream ROCm/HIP build — see WhisperX note).
|
||||
gpu_compat = ("cuda", "cpu")
|
||||
|
||||
def __init__(self):
|
||||
# Defaulting to the CTranslate2-converted large-v3 repo. Matches
|
||||
@@ -568,7 +497,6 @@ _MLX_MODEL_TURBO = "mlx-community/whisper-large-v3-turbo"
|
||||
class MLXWhisperBackend(ASRBackend):
|
||||
id = "mlx-whisper"
|
||||
display_name = "MLX Whisper (Apple Silicon CoreML)"
|
||||
gpu_compat = ("mps", "cpu")
|
||||
|
||||
def __init__(self, model_name: str | None = None):
|
||||
self._model_name = model_name or os.environ.get(
|
||||
@@ -577,23 +505,14 @@ class MLXWhisperBackend(ASRBackend):
|
||||
|
||||
@classmethod
|
||||
def is_available(cls) -> tuple[bool, str]:
|
||||
# #390: shared platform gate FIRST — one rule for MLX-Audio + MLX-Whisper.
|
||||
# Returns False on Linux/Windows/mac-Intel before any package import, so
|
||||
# a stray mlx-whisper wheel never reports available or advertises `mps`.
|
||||
from core.device_caps import mlx_supported
|
||||
ok, why = mlx_supported()
|
||||
if not ok:
|
||||
return False, why
|
||||
try:
|
||||
import torch
|
||||
if not (hasattr(torch.backends, "mps") and torch.backends.mps.is_available()):
|
||||
return False, "Apple Silicon (MPS) not available."
|
||||
import mlx_whisper # noqa: F401
|
||||
return True, "ready"
|
||||
# Catch OSError/RuntimeError too, not just ImportError: in a
|
||||
# PyInstaller bundle mlx's native dylib/metallib can fail to load
|
||||
# even when the package imports, raising OSError/RuntimeError. We must
|
||||
# report unavailable (so the picker falls back) rather than crash the
|
||||
# registry scan (Wave 4.4).
|
||||
except (ImportError, OSError, RuntimeError) as e:
|
||||
return False, f"mlx-whisper unavailable: {e}"
|
||||
except ImportError as e:
|
||||
return False, f"mlx-whisper not installed: {e}"
|
||||
|
||||
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
|
||||
import mlx_whisper
|
||||
@@ -642,9 +561,6 @@ class MLXWhisperBackend(ASRBackend):
|
||||
class PyTorchWhisperBackend(ASRBackend):
|
||||
id = "pytorch-whisper"
|
||||
display_name = "PyTorch Whisper (CUDA / CPU via transformers pipeline)"
|
||||
# Pure transformers pipeline → runs wherever torch does (CUDA, MPS, CPU).
|
||||
# ROCm-via-HIP would also work but is left unclaimed pending verification.
|
||||
gpu_compat = ("cuda", "mps", "cpu")
|
||||
|
||||
def __init__(self, asr_pipe=None):
|
||||
# Reuses the `_asr_pipe` attached to the TTS model when available.
|
||||
@@ -717,11 +633,6 @@ class NeMoASRBackend(ASRBackend):
|
||||
Requires NVIDIA GPU.
|
||||
"""
|
||||
id = "nemo-parakeet"
|
||||
# CUDA-only: is_available() hard-fails without a GPU ("Parakeet TDT requires
|
||||
# NVIDIA GPU (CUDA)"), so declaring a CPU path would be a false claim. On a
|
||||
# CPU host this correctly resolves to routing_status="unavailable", matching
|
||||
# is_available()=False (the matrix suppresses the routing badge there).
|
||||
gpu_compat = ("cuda",)
|
||||
display_name = "Parakeet TDT (NVIDIA NeMo — English SOTA)"
|
||||
|
||||
def __init__(self):
|
||||
@@ -830,7 +741,6 @@ class MoonshineASRBackend(ASRBackend):
|
||||
Great for live capture and CPU-only environments.
|
||||
"""
|
||||
id = "moonshine"
|
||||
gpu_compat = ("cpu",) # edge/CPU-optimized by design
|
||||
display_name = "Moonshine (edge-optimized, ONNX)"
|
||||
|
||||
def __init__(self):
|
||||
@@ -975,7 +885,6 @@ class FunASRBackend(ASRBackend):
|
||||
#182); WhisperX remains the cross-platform default.
|
||||
"""
|
||||
id = "funasr"
|
||||
gpu_compat = ("cuda", "cpu") # FunASR: CUDA or CPU
|
||||
display_name = "FunASR (SenseVoice — 50+ languages, all-in-one)"
|
||||
|
||||
def __init__(self):
|
||||
@@ -1022,47 +931,7 @@ class FunASRBackend(ASRBackend):
|
||||
pass
|
||||
|
||||
|
||||
def _isolated_faster_whisper():
|
||||
"""Lazy import so the subprocess_asr → subprocess_backend chain isn't
|
||||
pulled in at registry definition time."""
|
||||
from services.subprocess_asr import IsolatedFasterWhisperBackend
|
||||
return IsolatedFasterWhisperBackend
|
||||
|
||||
|
||||
class _LazyASRRegistry(dict):
|
||||
"""Registry with one lazily-resolved entry (Wave 4.2). Mirrors the TTS
|
||||
registry's lazy pattern so listing/selecting the crash-isolated ASR
|
||||
backend doesn't import the subprocess stack unless it's used."""
|
||||
|
||||
_LAZY = {"faster-whisper-isolated": _isolated_faster_whisper}
|
||||
|
||||
def __contains__(self, key):
|
||||
return dict.__contains__(self, key) or key in self._LAZY
|
||||
|
||||
def __getitem__(self, key):
|
||||
if dict.__contains__(self, key):
|
||||
return dict.__getitem__(self, key)
|
||||
if key in self._LAZY:
|
||||
cls = self._LAZY[key]()
|
||||
self[key] = cls
|
||||
return cls
|
||||
raise KeyError(key)
|
||||
|
||||
def __iter__(self):
|
||||
seen = set()
|
||||
for k in dict.__iter__(self):
|
||||
seen.add(k)
|
||||
yield k
|
||||
for k in self._LAZY:
|
||||
if k not in seen:
|
||||
yield k
|
||||
|
||||
def items(self):
|
||||
for k in self:
|
||||
yield k, self[k]
|
||||
|
||||
|
||||
_REGISTRY: dict[str, type[ASRBackend]] = _LazyASRRegistry({
|
||||
_REGISTRY: dict[str, type[ASRBackend]] = {
|
||||
"whisperx": WhisperXBackend,
|
||||
"faster-whisper": FasterWhisperBackend,
|
||||
"mlx-whisper": MLXWhisperBackend,
|
||||
@@ -1070,70 +939,18 @@ _REGISTRY: dict[str, type[ASRBackend]] = _LazyASRRegistry({
|
||||
"nemo-parakeet": NeMoASRBackend,
|
||||
"moonshine": MoonshineASRBackend,
|
||||
"funasr": FunASRBackend,
|
||||
# "faster-whisper-isolated": resolved lazily (crash-isolated subprocess).
|
||||
})
|
||||
|
||||
|
||||
# Short install hints surfaced as tooltips on the Settings → Engines UI
|
||||
# (parity with tts_backend._INSTALL_HINTS).
|
||||
_INSTALL_HINTS: dict[str, str] = {
|
||||
"whisperx": "pip install whisperx (CTranslate2 + wav2vec2 alignment; CUDA or CPU)",
|
||||
"faster-whisper": "pip install faster-whisper (CTranslate2; cross-platform, CUDA or CPU)",
|
||||
"mlx-whisper": "pip install mlx-whisper (Apple Silicon only)",
|
||||
"pytorch-whisper": "Bundled with transformers — no extra install (CUDA/MPS/CPU)",
|
||||
"nemo-parakeet": "pip install nemo_toolkit[asr] (NVIDIA Parakeet; CUDA or CPU)",
|
||||
"moonshine": "pip install useful-moonshine (edge/CPU-optimized ASR)",
|
||||
"funasr": "pip install funasr (SenseVoiceSmall + FSMN-VAD; CUDA or CPU)",
|
||||
}
|
||||
|
||||
# Most-recent failure per backend, so a transient probe error survives between
|
||||
# Settings refreshes (parity with tts_backend._LAST_ERRORS).
|
||||
_LAST_ERRORS: dict[str, str] = {}
|
||||
|
||||
|
||||
def list_backends() -> list[dict]:
|
||||
"""Enumerate every ASR backend with the **same 11-key shape as TTS** so the
|
||||
Engine Compatibility Matrix renders all families uniformly.
|
||||
|
||||
Per-entry: id, display_name, available, reason (scrubbed), install_hint,
|
||||
last_error, isolation_mode, gpu_compat, effective_device, routing_status,
|
||||
routing_reason. A backend whose ``is_available()`` raises is reported
|
||||
``available: false`` (never a 500), exactly like TTS.
|
||||
"""
|
||||
from core.device_caps import detect_host_caps
|
||||
from core.scrub import scrub_text
|
||||
from services.engine_routing import routing_fields
|
||||
caps = detect_host_caps()
|
||||
|
||||
out: list[dict] = []
|
||||
out = []
|
||||
for bid, cls in _REGISTRY.items():
|
||||
try:
|
||||
ok, msg = cls.is_available()
|
||||
except Exception as exc:
|
||||
ok = False
|
||||
msg = f"{type(exc).__name__}: {exc}"
|
||||
logger.warning(
|
||||
"asr list_backends: %s.is_available() raised — degrading "
|
||||
"gracefully so the picker still renders: %s", bid, msg,
|
||||
)
|
||||
if ok:
|
||||
_LAST_ERRORS.pop(bid, None)
|
||||
else:
|
||||
_LAST_ERRORS[bid] = scrub_text(msg)
|
||||
isolation = "subprocess" if getattr(cls, "_is_subprocess_isolated", False) else "in-process"
|
||||
gpu_compat = getattr(cls, "gpu_compat", ("cpu",))
|
||||
ok, msg = cls.is_available()
|
||||
out.append({
|
||||
"id": bid,
|
||||
"display_name": cls.display_name,
|
||||
"available": ok,
|
||||
# ASR previously emitted `reason` UNMASKED — scrub it now (closes a
|
||||
# pre-existing token-leak gap, matching TTS's redaction guarantee).
|
||||
"reason": None if ok else scrub_text(msg),
|
||||
"install_hint": _INSTALL_HINTS.get(bid),
|
||||
"last_error": _LAST_ERRORS.get(bid),
|
||||
"isolation_mode": isolation,
|
||||
"gpu_compat": list(gpu_compat),
|
||||
**routing_fields(gpu_compat, caps),
|
||||
"reason": None if ok else msg,
|
||||
})
|
||||
return out
|
||||
|
||||
@@ -1199,44 +1016,6 @@ def get_active_asr_backend(*, asr_pipe=None) -> ASRBackend:
|
||||
return _REGISTRY[bid]()
|
||||
|
||||
|
||||
def transcribe_reference(audio_path: str) -> str | None:
|
||||
"""Transcribe a voice-clone reference clip with the active ASR backend.
|
||||
|
||||
Voice cloning without a user-supplied transcript used to fall through to
|
||||
``OmniVoice.load_asr_model()`` — a transformers ``pipeline()`` load of
|
||||
whisper-large-v3-turbo that fails outright on transformers 5.3 (#308),
|
||||
even when whisperx / faster-whisper / mlx-whisper are installed and
|
||||
working. Route the reference transcript through the registry instead, so
|
||||
the model-attached pipeline is only reached when it is genuinely the last
|
||||
resort. Returns ``None`` on any failure — callers pass ``ref_text=None``
|
||||
through and the model's built-in fallback still gets its chance.
|
||||
"""
|
||||
try:
|
||||
backend = get_active_asr_backend()
|
||||
except Exception as e: # noqa: BLE001 — never let ASR break generation
|
||||
logger.warning("transcribe_reference: no ASR backend available (%s)", e)
|
||||
return None
|
||||
if isinstance(backend, PyTorchWhisperBackend):
|
||||
# The registry fell through to the model-attached pipeline; let the
|
||||
# model load it lazily rather than constructing a second copy here.
|
||||
return None
|
||||
try:
|
||||
result = backend.transcribe(audio_path, word_timestamps=False)
|
||||
except Exception as e: # noqa: BLE001 — degrade to the model fallback
|
||||
logger.warning(
|
||||
"transcribe_reference: %s failed (%s) — deferring to the model's "
|
||||
"built-in ASR fallback",
|
||||
backend.id, e,
|
||||
)
|
||||
return None
|
||||
result = result or {}
|
||||
text = result.get("text") or " ".join(
|
||||
(seg.get("text") or "").strip() for seg in result.get("segments", [])
|
||||
)
|
||||
text = (text or "").strip()
|
||||
return text or None
|
||||
|
||||
|
||||
_capture_backend: ASRBackend | None = None
|
||||
|
||||
|
||||
|
||||
@@ -1,170 +0,0 @@
|
||||
"""Audiobook creator — chapterized long-form narration (parity Wave 5).
|
||||
|
||||
Turns a chapter-delimited script into a chapterized audiobook. This module is
|
||||
the engine-agnostic core:
|
||||
|
||||
* ``parse_audiobook_script`` — pure parser: Markdown ``# H1`` headings become
|
||||
chapters; inline ``[voice:NAME]`` switches the narrator; ``[pause …]`` is
|
||||
delegated to the existing :func:`omnivoice.utils.text.parse_pause_markers`
|
||||
so audiobooks and single-shot synthesis share one pause dialect.
|
||||
* ``synthesize_chapter`` — orchestration: renders a chapter's spans through an
|
||||
injected ``synth(text, voice_id) -> tensor`` callable (reusing the
|
||||
``chunked_tts`` splitter + crossfade), stitching the inter-span silences.
|
||||
Injecting the synth keeps this unit-testable with a stub backend (no torch
|
||||
model, no GPU).
|
||||
* ``build_chapter_ffmetadata`` / ``build_m4b_cmd`` — pure builders for the
|
||||
ffmpeg chapterized-m4b mux (FFMETADATA1 ``[CHAPTER]`` blocks + concat-demux
|
||||
argv). The actual ffmpeg run lives in the (impure) caller.
|
||||
|
||||
Scope (first cut): plain chapter-delimited text/Markdown input. epub/pdf
|
||||
ingestion, the streaming synth job + UI are deferred follow-ups.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Callable, Optional
|
||||
|
||||
@dataclass
|
||||
class Span:
|
||||
"""One contiguous run of text in a single voice, plus trailing silence.
|
||||
|
||||
``speed`` (when set) is the per-span rate passed to the engine — Stories'
|
||||
per-line speed slider rides through here so the shared server render honours
|
||||
it the way the old client export did.
|
||||
"""
|
||||
voice_id: Optional[str]
|
||||
text: str
|
||||
pause_ms_after: int = 0
|
||||
speed: Optional[float] = None
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {"voice_id": self.voice_id, "text": self.text,
|
||||
"pause_ms_after": self.pause_ms_after, "speed": self.speed}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Chapter:
|
||||
title: str
|
||||
spans: list[Span] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def char_count(self) -> int:
|
||||
return sum(len(s.text) for s in self.spans)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {"title": self.title, "char_count": self.char_count,
|
||||
"spans": [s.to_dict() for s in self.spans]}
|
||||
|
||||
|
||||
@dataclass
|
||||
class AudiobookPlan:
|
||||
chapters: list[Chapter] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def char_count(self) -> int:
|
||||
return sum(c.char_count for c in self.chapters)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"chapters": [c.to_dict() for c in self.chapters],
|
||||
"chapter_count": len(self.chapters),
|
||||
"char_count": self.char_count,
|
||||
}
|
||||
|
||||
|
||||
def parse_audiobook_script(text: str, *, default_voice: Optional[str] = None) -> AudiobookPlan:
|
||||
"""Parse a chapter-delimited script into an :class:`AudiobookPlan`.
|
||||
|
||||
Thin wrapper over the canonical :func:`services.longform_parser.
|
||||
parse_script_to_spans` (the single grammar source of truth, #27); wraps its
|
||||
span dicts in the ``Span``/``Chapter``/``AudiobookPlan`` dataclasses so the
|
||||
four router call sites and ``.to_dict()`` shape are unchanged.
|
||||
"""
|
||||
from services.longform_parser import parse_script_to_spans
|
||||
|
||||
chapters = [
|
||||
Chapter(title=c["title"], spans=[Span(**s) for s in c["spans"]])
|
||||
for c in parse_script_to_spans(text, default_voice=default_voice)
|
||||
]
|
||||
return AudiobookPlan(chapters=chapters)
|
||||
|
||||
|
||||
def synthesize_chapter(
|
||||
spans: list[Span],
|
||||
synth: Callable[[str, Optional[str], Optional[float]], "object"],
|
||||
sample_rate: int,
|
||||
*,
|
||||
crossfade_ms: int = 50,
|
||||
lexicon: Optional[dict] = None,
|
||||
):
|
||||
"""Render a chapter's spans to one waveform via an injected ``synth``.
|
||||
|
||||
``synth(text, voice_id, speed)`` returns a 1-D float32 audio tensor for a
|
||||
span of text in the given voice (``speed`` may be ``None`` for the engine
|
||||
default). Long spans are split with the ``chunked_tts`` splitter and
|
||||
crossfaded; inter-span ``pause_ms_after`` becomes silence. ``lexicon`` (when
|
||||
given) respells each span's text before chunking so the engine pronounces
|
||||
tricky words correctly; a ``None``/empty lexicon is a no-op pass-through.
|
||||
|
||||
Returns ``(audio_tensor, duration_seconds)``. torch + chunked_tts are
|
||||
imported lazily so this module stays import-light for the pure parser path.
|
||||
"""
|
||||
import torch
|
||||
from services.chunked_tts import concatenate_audio_chunks, split_text_into_chunks
|
||||
from services.pronunciation import apply_lexicon
|
||||
|
||||
parts: list = []
|
||||
for span in spans:
|
||||
if span.text:
|
||||
chunks = split_text_into_chunks(apply_lexicon(span.text, lexicon))
|
||||
rendered = [synth(c, span.voice_id, span.speed) for c in chunks]
|
||||
rendered = [r for r in rendered if r is not None and getattr(r, "numel", lambda: 0)()]
|
||||
if len(rendered) == 1:
|
||||
parts.append(rendered[0])
|
||||
elif rendered:
|
||||
parts.append(concatenate_audio_chunks(rendered, sample_rate, crossfade_ms=crossfade_ms))
|
||||
if span.pause_ms_after > 0:
|
||||
n = int(sample_rate * span.pause_ms_after / 1000.0)
|
||||
if n > 0:
|
||||
parts.append(torch.zeros(n, dtype=torch.float32))
|
||||
|
||||
if not parts:
|
||||
return torch.zeros(0, dtype=torch.float32), 0.0
|
||||
# Hard-concat spans + silences (crossfading silence would bleed the gap).
|
||||
audio = parts[0] if len(parts) == 1 else concatenate_audio_chunks(parts, sample_rate, crossfade_ms=0)
|
||||
return audio, audio.shape[-1] / float(sample_rate)
|
||||
|
||||
|
||||
# ── ffmpeg / metadata builders ──────────────────────────────────────────────
|
||||
#
|
||||
# These now live in the shared ``longform_render`` core (Stories + Audiobook
|
||||
# converge on one mux). The thin wrappers below preserve the original
|
||||
# audiobook-only call sites/signatures; new callers should use
|
||||
# ``longform_render`` directly to reach global metadata, cover art, loudness,
|
||||
# and mp3 output.
|
||||
from services.longform_render import ( # noqa: E402
|
||||
build_concat_list,
|
||||
build_ffmetadata,
|
||||
build_render_cmd,
|
||||
)
|
||||
|
||||
|
||||
def build_chapter_ffmetadata(chapters: list[tuple[str, int]]) -> str:
|
||||
"""Backward-compatible alias: chapters-only FFMETADATA (no global tags)."""
|
||||
return build_ffmetadata(chapters)
|
||||
|
||||
|
||||
def build_m4b_cmd(
|
||||
ffmpeg: str,
|
||||
concat_list_path: str,
|
||||
metadata_path: str,
|
||||
out_path: str,
|
||||
*,
|
||||
bitrate: str = "128k",
|
||||
) -> list[str]:
|
||||
"""Backward-compatible alias: a chapterized faststart m4b, no cover/loudness."""
|
||||
return build_render_cmd(
|
||||
ffmpeg, concat_list_path, metadata_path, out_path,
|
||||
fmt="m4b", bitrate=bitrate,
|
||||
)
|
||||
@@ -151,9 +151,6 @@ async def generate_segments_batched(
|
||||
# Raw: skip all DSP — return raw model output
|
||||
return audio_out
|
||||
|
||||
# TODO(#312): this route runs the OmniVoice model directly (not the active
|
||||
# backend), so VoxCPM2 never reaches it. When these routes become
|
||||
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
|
||||
mastered = apply_mastering(audio_out, sample_rate=sr)
|
||||
effect_chain = get_effect_chain(seg_effect_preset)
|
||||
if effect_chain:
|
||||
|
||||
@@ -1,174 +0,0 @@
|
||||
"""Chunked TTS generation utilities (Wave 1.2 — unlimited-length generation).
|
||||
|
||||
Adapted from voicebox (https://github.com/jamiepine/voicebox), MIT License,
|
||||
Copyright (c) voicebox contributors. The concatenation half is reworked for
|
||||
torch tensors (our inference helpers pass raw model output — possibly
|
||||
multi-channel — to the effect chain), and the sample rate comes from the
|
||||
engine's declared rate rather than the first chunk (fixes a latent upstream
|
||||
bug where a mid-run rate change was silently ignored).
|
||||
|
||||
Splits long text into sentence-boundary chunks and joins the per-chunk audio
|
||||
with a short crossfade. Pure functions — the generation loop itself lives in
|
||||
``api/routers/generation.py`` next to the existing ``[pause]`` span stitcher,
|
||||
so this module stays unit-testable without a model.
|
||||
|
||||
Short text (<= max_chunk_chars) never reaches this module's concat path; the
|
||||
callers keep their unchanged single-shot fast path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import List
|
||||
|
||||
logger = logging.getLogger("omnivoice.chunked_tts")
|
||||
|
||||
# Default chunk size in characters. 0 disables chunking entirely.
|
||||
DEFAULT_MAX_CHUNK_CHARS = 800
|
||||
|
||||
# Default crossfade between chunks. 0 = hard cut.
|
||||
DEFAULT_CROSSFADE_MS = 50
|
||||
|
||||
# Common abbreviations that should NOT be treated as sentence endings.
|
||||
# Lowercase for case-insensitive matching.
|
||||
_ABBREVIATIONS = frozenset({
|
||||
"mr", "mrs", "ms", "dr", "prof", "sr", "jr", "st", "ave", "blvd",
|
||||
"inc", "ltd", "corp", "dept", "est", "approx", "vs", "etc",
|
||||
"e.g", "i.e", "a.m", "p.m", "u.s", "u.s.a", "u.k",
|
||||
})
|
||||
|
||||
# Inline bracket tags (paralinguistic tags like [laugh]; our own
|
||||
# [pause 300ms] markers). The splitter must never cut inside one.
|
||||
_BRACKET_TAG_RE = re.compile(r"\[[^\]]*\]")
|
||||
|
||||
|
||||
def split_text_into_chunks(text: str, max_chars: int = DEFAULT_MAX_CHUNK_CHARS) -> List[str]:
|
||||
"""Split *text* at natural boundaries into chunks of at most *max_chars*.
|
||||
|
||||
Priority: sentence-end (``.!?`` not after an abbreviation/decimal and not
|
||||
inside brackets, plus fullwidth equivalents) -> clause boundary
|
||||
(``;:,`` / em dash) -> whitespace -> hard cut that avoids splitting a
|
||||
``[tag]``.
|
||||
"""
|
||||
text = text.strip()
|
||||
if not text:
|
||||
return []
|
||||
if max_chars <= 0 or len(text) <= max_chars:
|
||||
return [text]
|
||||
|
||||
chunks: List[str] = []
|
||||
remaining = text
|
||||
|
||||
while remaining:
|
||||
remaining = remaining.lstrip()
|
||||
if not remaining:
|
||||
break
|
||||
if len(remaining) <= max_chars:
|
||||
chunks.append(remaining)
|
||||
break
|
||||
|
||||
segment = remaining[:max_chars]
|
||||
|
||||
split_pos = _find_last_sentence_end(segment)
|
||||
if split_pos == -1:
|
||||
split_pos = _find_last_clause_boundary(segment)
|
||||
if split_pos == -1:
|
||||
split_pos = segment.rfind(" ")
|
||||
if split_pos == -1:
|
||||
split_pos = _safe_hard_cut(segment, max_chars)
|
||||
|
||||
chunk = remaining[: split_pos + 1].strip()
|
||||
if chunk:
|
||||
chunks.append(chunk)
|
||||
remaining = remaining[split_pos + 1:]
|
||||
|
||||
return chunks
|
||||
|
||||
|
||||
def _find_last_sentence_end(text: str) -> int:
|
||||
"""Index of the last sentence-ending punctuation, or -1.
|
||||
|
||||
Skips periods after common abbreviations and decimals, anything inside
|
||||
a bracket tag, and also recognizes fullwidth sentence punctuation
|
||||
(ideographic full stop / fullwidth ! and ?) for no-space scripts.
|
||||
"""
|
||||
best = -1
|
||||
for m in re.finditer(r"[.!?](?:\s|$)", text):
|
||||
pos = m.start()
|
||||
if text[pos] == ".":
|
||||
word_start = pos - 1
|
||||
while word_start >= 0 and text[word_start].isalpha():
|
||||
word_start -= 1
|
||||
word = text[word_start + 1: pos].lower()
|
||||
if word in _ABBREVIATIONS:
|
||||
continue
|
||||
if word_start >= 0 and text[word_start].isdigit():
|
||||
continue
|
||||
if _inside_bracket_tag(text, pos):
|
||||
continue
|
||||
best = pos
|
||||
# Fullwidth sentence enders (ideographic full stop, fullwidth !, ?)
|
||||
# written as escapes to keep the repo's no-literal-CJK gate clean.
|
||||
for m in re.finditer("[\u3002\uff01\uff1f]", text):
|
||||
if m.start() > best:
|
||||
best = m.start()
|
||||
return best
|
||||
|
||||
|
||||
def _find_last_clause_boundary(text: str) -> int:
|
||||
best = -1
|
||||
for m in re.finditer(r"[;:,—](?:\s|$)", text):
|
||||
if _inside_bracket_tag(text, m.start()):
|
||||
continue
|
||||
best = m.start()
|
||||
return best
|
||||
|
||||
|
||||
def _inside_bracket_tag(text: str, pos: int) -> bool:
|
||||
for m in _BRACKET_TAG_RE.finditer(text):
|
||||
if m.start() < pos < m.end():
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _safe_hard_cut(segment: str, max_chars: int) -> int:
|
||||
cut = max_chars - 1
|
||||
for m in _BRACKET_TAG_RE.finditer(segment):
|
||||
if m.start() < cut < m.end():
|
||||
return m.start() - 1 if m.start() > 0 else cut
|
||||
return cut
|
||||
|
||||
|
||||
def concatenate_audio_chunks(chunks: list, sample_rate: int,
|
||||
crossfade_ms: int = DEFAULT_CROSSFADE_MS):
|
||||
"""Join per-chunk waveforms with a linear crossfade on the sample axis.
|
||||
|
||||
``chunks`` are torch tensors as returned by the engine (1-D, or N-D with
|
||||
samples on the last axis — matching what ``_render_with_pauses`` handles).
|
||||
Crossfade overlap is clamped to the shorter neighbor; ``crossfade_ms=0``
|
||||
is a hard concat.
|
||||
"""
|
||||
import torch
|
||||
|
||||
chunks = [c for c in chunks if c is not None and c.shape[-1] > 0]
|
||||
if not chunks:
|
||||
return torch.zeros(1, dtype=torch.float32)
|
||||
if len(chunks) == 1:
|
||||
return chunks[0]
|
||||
|
||||
crossfade_samples = int(sample_rate * crossfade_ms / 1000)
|
||||
result = chunks[0]
|
||||
|
||||
for chunk in chunks[1:]:
|
||||
chunk = chunk.to(device=result.device, dtype=result.dtype)
|
||||
overlap = min(crossfade_samples, result.shape[-1], chunk.shape[-1])
|
||||
if overlap > 0:
|
||||
fade_out = torch.linspace(1.0, 0.0, overlap, dtype=result.dtype, device=result.device)
|
||||
fade_in = torch.linspace(0.0, 1.0, overlap, dtype=result.dtype, device=result.device)
|
||||
blended = result[..., -overlap:] * fade_out + chunk[..., :overlap] * fade_in
|
||||
result = torch.cat([result[..., :-overlap], blended, chunk[..., overlap:]], dim=-1)
|
||||
else:
|
||||
result = torch.cat([result, chunk], dim=-1)
|
||||
|
||||
return result
|
||||
@@ -13,8 +13,7 @@ What's here
|
||||
* **Content-hash cache lookup** — `compute_file_hash`, `find_cached_job`.
|
||||
* **Safe path resolution** — `safe_job_dir`.
|
||||
* **Process lifecycle** — ffmpeg/demucs subprocess tracking + `kill_job_procs`
|
||||
so `POST /dub/abort/{id}` can tear down in-flight work (implemented in
|
||||
`services.proc_registry`, re-exported here for compatibility).
|
||||
so `POST /dub/abort/{id}` can tear down in-flight work.
|
||||
* **SSE helpers** — `sse_event`, `prep_event`.
|
||||
|
||||
What stays in the router
|
||||
@@ -45,17 +44,6 @@ from core.config import DUB_DIR
|
||||
from fastapi import HTTPException
|
||||
from services.ffmpeg_utils import find_ffmpeg, find_ffprobe, _get_semaphore, _spawn_with_retry
|
||||
from services.model_manager import get_best_device
|
||||
# Process lifecycle moved to its own leaf module so ffmpeg_utils can import
|
||||
# it at module top (no dub_pipeline ↔ ffmpeg_utils cycle). Re-exported here —
|
||||
# dub_core and tests still alias these names through this module.
|
||||
from services.proc_registry import ( # noqa: F401 — re-exports
|
||||
_active_procs,
|
||||
_active_procs_lock,
|
||||
has_active_procs,
|
||||
kill_job_procs,
|
||||
register_proc,
|
||||
unregister_proc,
|
||||
)
|
||||
from core.db import db_conn
|
||||
from core import event_bus
|
||||
from core import failure
|
||||
@@ -68,6 +56,8 @@ logger = logging.getLogger("omnivoice.dub_pipeline")
|
||||
|
||||
_dub_jobs: dict[str, dict] = {}
|
||||
_dub_jobs_lock = threading.Lock()
|
||||
_active_procs: dict[str, list] = {}
|
||||
_active_procs_lock = threading.Lock()
|
||||
|
||||
_DUB_DIR_REAL = os.path.realpath(DUB_DIR)
|
||||
_HASH_BUF_SIZE = 1 << 18 # 256 KB chunks for hashing
|
||||
@@ -148,8 +138,42 @@ def find_cached_job(content_hash: str, exclude_job_id: str) -> Optional[dict]:
|
||||
|
||||
|
||||
# ── Process lifecycle ───────────────────────────────────────────────────────
|
||||
# register_proc / unregister_proc / kill_job_procs / has_active_procs live in
|
||||
# services.proc_registry (imported + re-exported above).
|
||||
|
||||
|
||||
def register_proc(job_id: str, proc) -> None:
|
||||
"""Track an in-flight subprocess so /dub/abort can kill it."""
|
||||
with _active_procs_lock:
|
||||
_active_procs.setdefault(job_id, []).append(proc)
|
||||
|
||||
|
||||
def unregister_proc(job_id: str, proc) -> None:
|
||||
with _active_procs_lock:
|
||||
lst = _active_procs.get(job_id)
|
||||
if lst and proc in lst:
|
||||
lst.remove(proc)
|
||||
if lst is not None and not lst:
|
||||
_active_procs.pop(job_id, None)
|
||||
|
||||
|
||||
def kill_job_procs(job_id: str) -> None:
|
||||
"""Kill every subprocess still running under a given job id. Idempotent."""
|
||||
with _active_procs_lock:
|
||||
procs = list(_active_procs.get(job_id, []))
|
||||
for proc in procs:
|
||||
try:
|
||||
if proc.returncode is None:
|
||||
proc.kill()
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.warning("Failed to kill subprocess for %s: %s", job_id, e)
|
||||
with _active_procs_lock:
|
||||
_active_procs.pop(job_id, None)
|
||||
|
||||
|
||||
def has_active_procs(job_id: str) -> bool:
|
||||
with _active_procs_lock:
|
||||
return bool(_active_procs.get(job_id))
|
||||
|
||||
|
||||
# ── Job state (in-memory + SQLite fallback) ────────────────────────────────
|
||||
@@ -171,10 +195,7 @@ def get_job(job_id: str) -> Optional[dict]:
|
||||
_dub_jobs[job_id] = job
|
||||
return job
|
||||
except json.JSONDecodeError as e:
|
||||
# job_id arrives from request paths — strip newlines so a crafted
|
||||
# id can't forge extra log lines (py/log-injection).
|
||||
safe_id = str(job_id).replace("\r", "").replace("\n", "")
|
||||
logger.error("Failed to decode dub_history.job_data for %s: %s", safe_id, e)
|
||||
logger.error("Failed to decode dub_history.job_data for %s: %s", job_id, e)
|
||||
return None
|
||||
|
||||
|
||||
|
||||
@@ -1,118 +0,0 @@
|
||||
"""Second-pass ASR quality control for dubs (Wave 3.3 / Spec 5).
|
||||
|
||||
After a dub is generated, re-recognize the synthetic audio and compare what
|
||||
the ASR *heard* against what we asked the TTS to *say*. Where the two drift
|
||||
apart, the line is flagged for the user to verify — turning subtitle timing
|
||||
and pronunciation from "trusted math" into "measured truth", and doubling as
|
||||
an automatic dub-quality check.
|
||||
|
||||
Design delta from pyvideotrans (whose second pass lets recognized text
|
||||
*replace* the subtitles wholesale): we keep the GENERATED text authoritative
|
||||
for content and use the second pass for *measurement* — timing + a drift
|
||||
score that feeds the incremental re-dub loop, never silently overwriting the
|
||||
translation.
|
||||
|
||||
Pure functions here (no ASR, no I/O) so the scoring is unit-testable; the
|
||||
pipeline stage that runs the ASR pass lives in the dub router.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
def _tokens(text: str) -> list[str]:
|
||||
"""Lowercase word tokens, punctuation stripped — the unit drift is scored
|
||||
in. Script-agnostic: for no-space scripts each character is a token, which
|
||||
still gives a sensible edit-distance ratio."""
|
||||
text = (text or "").lower().strip()
|
||||
if not text:
|
||||
return []
|
||||
words = re.findall(r"\w+", text, flags=re.UNICODE)
|
||||
return words or list(text.replace(" ", ""))
|
||||
|
||||
|
||||
def _edit_distance(a: list[str], b: list[str]) -> int:
|
||||
"""Levenshtein distance between two token lists (iterative, O(len(a)*len(b))
|
||||
time, O(len(b)) space)."""
|
||||
if not a:
|
||||
return len(b)
|
||||
if not b:
|
||||
return len(a)
|
||||
prev = list(range(len(b) + 1))
|
||||
for i, ta in enumerate(a, 1):
|
||||
cur = [i]
|
||||
for j, tb in enumerate(b, 1):
|
||||
cost = 0 if ta == tb else 1
|
||||
cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + cost))
|
||||
prev = cur
|
||||
return prev[-1]
|
||||
|
||||
|
||||
def word_error_rate(reference: str, hypothesis: str) -> float:
|
||||
"""Normalized token edit distance in [0.0, 1.0+].
|
||||
|
||||
0.0 = the ASR heard exactly the target text. ~1.0 = entirely different.
|
||||
Can exceed 1.0 when the hypothesis is much longer than the reference
|
||||
(insertions); callers clamp/threshold as needed. An empty reference with a
|
||||
non-empty hypothesis scores 1.0 (everything is an insertion)."""
|
||||
ref = _tokens(reference)
|
||||
hyp = _tokens(hypothesis)
|
||||
if not ref and not hyp:
|
||||
return 0.0
|
||||
if not ref:
|
||||
return 1.0
|
||||
return _edit_distance(ref, hyp) / len(ref)
|
||||
|
||||
|
||||
@dataclass
|
||||
class SegmentQC:
|
||||
seg_id: str
|
||||
target_text: str
|
||||
recognized_text: str
|
||||
drift: float # word_error_rate(target, recognized)
|
||||
flagged: bool # drift >= threshold
|
||||
new_start: float | None # measured onset from the dubbed-audio recognition
|
||||
new_end: float | None
|
||||
|
||||
|
||||
def _overlap(a0: float, a1: float, b0: float, b1: float) -> float:
|
||||
return max(0.0, min(a1, b1) - max(a0, b0))
|
||||
|
||||
|
||||
def score_dub(
|
||||
dub_segments: list[dict],
|
||||
recognized: list[dict],
|
||||
*,
|
||||
drift_threshold: float = 0.5,
|
||||
seg_ids: list | None = None,
|
||||
) -> list[SegmentQC]:
|
||||
"""Match the second-pass recognition to the dub segments and score drift.
|
||||
|
||||
``dub_segments`` are the segments we generated (each {start, end, text});
|
||||
``recognized`` are the ASR result segments on the dubbed audio (each
|
||||
{start, end, text}). Each dub segment is matched to the recognized
|
||||
segment(s) it overlaps in time; their text is concatenated as the
|
||||
hypothesis and scored against the dub segment's ``text``. The recognized
|
||||
span's bounds become the measured start/end (subtitle-timing truth).
|
||||
"""
|
||||
results: list[SegmentQC] = []
|
||||
for i, seg in enumerate(dub_segments):
|
||||
sid = str(seg_ids[i]) if (seg_ids and i < len(seg_ids)) else str(seg.get("id", i))
|
||||
s0, s1 = float(seg.get("start", 0.0)), float(seg.get("end", 0.0))
|
||||
hits = [r for r in recognized if _overlap(s0, s1, float(r.get("start", 0.0)), float(r.get("end", 0.0))) > 0]
|
||||
hyp = " ".join((r.get("text") or "").strip() for r in hits).strip()
|
||||
drift = word_error_rate(seg.get("text", ""), hyp)
|
||||
new_start = min((float(r.get("start", 0.0)) for r in hits), default=None)
|
||||
new_end = max((float(r.get("end", 0.0)) for r in hits), default=None)
|
||||
results.append(SegmentQC(
|
||||
seg_id=sid,
|
||||
target_text=(seg.get("text") or "").strip(),
|
||||
recognized_text=hyp,
|
||||
drift=round(drift, 3),
|
||||
flagged=drift >= drift_threshold,
|
||||
new_start=new_start,
|
||||
new_end=new_end,
|
||||
))
|
||||
return results
|
||||
@@ -24,78 +24,6 @@ logger = logging.getLogger("omnivoice.engine_env")
|
||||
|
||||
_TORCH_COMPILE_KEY = "perf.torch_compile_disabled"
|
||||
|
||||
# #278: explicit opt-in override — set to 1/true to attempt torch.compile even
|
||||
# when the GPU's compute capability is not in this PyTorch build's arch list
|
||||
# (e.g. a brand-new architecture running through PTX forward-compat).
|
||||
_FORCE_COMPILE_ENV = "OMNIVOICE_FORCE_TORCH_COMPILE"
|
||||
|
||||
# #278: set (with a reason) the first time torch.compile — or *running* the
|
||||
# compiled model — fails at runtime in this process. Once set, every later
|
||||
# load in the same session goes straight to eager instead of re-tripping the
|
||||
# same Dynamo/Inductor/Triton failure.
|
||||
_compile_runtime_failure: Optional[str] = None
|
||||
|
||||
|
||||
def mark_compile_runtime_failure(reason: str) -> None:
|
||||
"""Record that torch.compile (or compiled execution) failed at runtime.
|
||||
|
||||
Called by ``services.model_manager`` when compilation raises, or when a
|
||||
generation through the compiled model dies inside the Dynamo / Inductor /
|
||||
Triton stack (#278). Disables compile for the rest of the process — eager
|
||||
mode from here on; the next app restart probes again.
|
||||
"""
|
||||
global _compile_runtime_failure
|
||||
_compile_runtime_failure = reason or "unknown torch.compile runtime failure"
|
||||
logger.warning(
|
||||
"torch.compile disabled for this session after a runtime failure: %s",
|
||||
_compile_runtime_failure,
|
||||
)
|
||||
|
||||
|
||||
def _force_compile_requested() -> bool:
|
||||
value = os.environ.get(_FORCE_COMPILE_ENV, "")
|
||||
return value.strip().lower() in {"1", "true", "yes", "on"}
|
||||
|
||||
|
||||
def _cuda_arch_supported_for_compile() -> "tuple[bool, str]":
|
||||
"""Check the GPU's compute capability against this torch build's arch list.
|
||||
|
||||
New GPU architectures (e.g. Blackwell sm_120, issue #278) routinely break
|
||||
torch.compile/Triton before upstream support lands: the eager model runs
|
||||
via PTX forward-compat, but Inductor/Triton kernel compilation targets the
|
||||
new arch directly and fails mid-generation. If the device's ``sm_XY`` tag
|
||||
is absent from ``torch.cuda.get_arch_list()`` we treat compile as
|
||||
unsupported and use eager.
|
||||
|
||||
Returns ``(supported, reason)``. Fails open — any probe error returns
|
||||
``(True, "")`` so a weird torch build never silently loses the
|
||||
optimization (the runtime fallback in model_manager still protects
|
||||
generation).
|
||||
"""
|
||||
try:
|
||||
import torch
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
return True, ""
|
||||
major, minor = torch.cuda.get_device_capability(0)
|
||||
arch_list = list(getattr(torch.cuda, "get_arch_list", lambda: [])() or [])
|
||||
if not arch_list:
|
||||
return True, ""
|
||||
sm_tag = f"sm_{major}{minor}"
|
||||
if sm_tag in arch_list or f"compute_{major}{minor}" in arch_list:
|
||||
return True, ""
|
||||
try:
|
||||
device_name = torch.cuda.get_device_name(0)
|
||||
except Exception:
|
||||
device_name = "GPU"
|
||||
return False, (
|
||||
f"{device_name} (compute capability {major}.{minor} / {sm_tag}) is not "
|
||||
f"in this PyTorch build's supported arch list ({', '.join(arch_list)})"
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("CUDA arch probe for torch.compile failed; assuming supported", exc_info=True)
|
||||
return True, ""
|
||||
|
||||
|
||||
def should_torch_compile(device: str) -> bool:
|
||||
"""Decide whether to apply ``torch.compile`` to an in-process model.
|
||||
@@ -106,14 +34,9 @@ def should_torch_compile(device: str) -> bool:
|
||||
- device == "cuda" (compile only helps the CUDA path here),
|
||||
- Triton importable (``find_spec`` — the cross-platform gate that closes
|
||||
#65; no Windows wheel ⇒ skip ⇒ eager),
|
||||
- the user has NOT set the ``perf.torch_compile_disabled`` escape hatch,
|
||||
- compile has NOT already failed at runtime in this process (#278),
|
||||
- the GPU's compute capability is in this torch build's arch list (#278)
|
||||
— overridable via ``OMNIVOICE_FORCE_TORCH_COMPILE=1``.
|
||||
- the user has NOT set the ``perf.torch_compile_disabled`` escape hatch.
|
||||
|
||||
Returns False (→ eager mode) on any of those, logging the reason at INFO.
|
||||
torch.compile is an optimization, never a requirement — generation must
|
||||
always work without it.
|
||||
"""
|
||||
if device != "cuda":
|
||||
return False
|
||||
@@ -128,25 +51,6 @@ def should_torch_compile(device: str) -> bool:
|
||||
return False
|
||||
except Exception:
|
||||
logger.exception("should_torch_compile: settings read failed; proceeding")
|
||||
if _compile_runtime_failure is not None:
|
||||
logger.info(
|
||||
"torch.compile skipped: failed earlier this session (%s) — using eager mode.",
|
||||
_compile_runtime_failure,
|
||||
)
|
||||
return False
|
||||
supported, reason = _cuda_arch_supported_for_compile()
|
||||
if not supported:
|
||||
if _force_compile_requested():
|
||||
logger.warning(
|
||||
"torch.compile forced via %s=1 despite: %s", _FORCE_COMPILE_ENV, reason,
|
||||
)
|
||||
return True
|
||||
logger.info(
|
||||
"torch.compile skipped: %s — using eager mode. "
|
||||
"(Set %s=1 to attempt compile anyway.)",
|
||||
reason, _FORCE_COMPILE_ENV,
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
|
||||
@@ -1,156 +0,0 @@
|
||||
"""Pure, host-aware routing resolver — maps an engine's declared ``gpu_compat``
|
||||
against the cached host capabilities to "where will this engine *actually* run
|
||||
on this machine, and is that a problem the user should hear about?"
|
||||
|
||||
No model load, no probe (the caller passes the cached ``HostCaps``), no I/O.
|
||||
Deterministic and byte-identical for a given ``(gpu_compat, HostCaps)`` across
|
||||
macOS/Windows/Linux — that cross-OS determinism is the whole point of the
|
||||
no-silent-fallback contract.
|
||||
|
||||
Reason strings are author-controlled English (interpolating only family/device
|
||||
names) but are **still** scrubbed by the caller (``core.scrub.scrub_text``)
|
||||
before serialization, because an interpolated ``device_name`` or probe note can
|
||||
carry a home path.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Literal, TypedDict
|
||||
|
||||
from core.device_caps import (
|
||||
DIRECTML_MARKER,
|
||||
KERNEL_RISK_MARKER,
|
||||
HostCaps,
|
||||
)
|
||||
|
||||
RoutingStatus = Literal["accelerated", "cpu_fallback", "cpu_only", "unavailable", "n/a"]
|
||||
|
||||
|
||||
class RoutingResult(TypedDict):
|
||||
effective_device: str # a DeviceFamily value or "cpu"
|
||||
routing_status: RoutingStatus # resolve_routing never emits "n/a" (LLM-only)
|
||||
routing_reason: str | None # raw, pre-scrub
|
||||
|
||||
|
||||
def _caveat(caps: HostCaps) -> str | None:
|
||||
"""A kernel-risk caveat string for an otherwise-accelerated host, or None.
|
||||
Advisory notes (multi-GPU, VRAM-query-failed, DirectML) never qualify."""
|
||||
for note in caps.notes:
|
||||
if KERNEL_RISK_MARKER in note:
|
||||
return f"{caps.family.upper()} selected, but: {note}"
|
||||
return None
|
||||
|
||||
|
||||
def resolve_routing(gpu_compat: tuple[str, ...], caps: HostCaps) -> RoutingResult:
|
||||
"""Resolve the effective device + status for an engine on this host.
|
||||
|
||||
Rules are evaluated in order; the first match wins (see spec §2)."""
|
||||
targets = tuple(gpu_compat or ())
|
||||
fam = caps.family
|
||||
|
||||
# 1. Empty compat — reserved for LLM (which never calls this). Defensive.
|
||||
if not targets:
|
||||
return {
|
||||
"effective_device": "cpu",
|
||||
"routing_status": "cpu_only",
|
||||
"routing_reason": "engine declares no compute targets",
|
||||
}
|
||||
|
||||
# 2. Host accelerator is one the engine supports → accelerated.
|
||||
if fam != "cpu" and fam in targets:
|
||||
return {
|
||||
"effective_device": fam,
|
||||
"routing_status": "accelerated",
|
||||
"routing_reason": _caveat(caps),
|
||||
}
|
||||
|
||||
# 3. Host has an accelerator the engine lacks, but engine supports cpu
|
||||
# → the no-silent-fallback signal.
|
||||
if fam != "cpu" and "cpu" in targets:
|
||||
if fam == "rocm" and "cuda" in targets and "rocm" not in targets:
|
||||
reason = "declares CUDA only; ROCm not in its compat set"
|
||||
else:
|
||||
reason = f"engine has no {fam.upper()} path; running on CPU"
|
||||
return {
|
||||
"effective_device": "cpu",
|
||||
"routing_status": "cpu_fallback",
|
||||
"routing_reason": reason,
|
||||
}
|
||||
|
||||
# 4. Genuine CPU-only host (or DirectML, which the probe reports as cpu)
|
||||
# and engine supports cpu → benign; must not warn or block.
|
||||
if fam == "cpu" and "cpu" in targets:
|
||||
reason = None
|
||||
for note in caps.notes:
|
||||
if DIRECTML_MARKER in note:
|
||||
reason = (
|
||||
"DirectML GPU present; engine routes via torch CPU path "
|
||||
"(DirectML acceleration not wired into routing)"
|
||||
)
|
||||
break
|
||||
return {
|
||||
"effective_device": "cpu",
|
||||
"routing_status": "cpu_only",
|
||||
"routing_reason": reason,
|
||||
}
|
||||
|
||||
# 5. Engine needs an accelerator this host lacks and has no cpu path.
|
||||
first = targets[0]
|
||||
return {
|
||||
"effective_device": first,
|
||||
"routing_status": "unavailable",
|
||||
"routing_reason": f"requires {', '.join(targets)}; this host has {fam}",
|
||||
}
|
||||
|
||||
|
||||
def routing_notice(result: RoutingResult) -> tuple[str, str | None] | None:
|
||||
"""`(status, reason)` when a synth-time notice SHOULD be surfaced to the
|
||||
user, else `None`. Surfaced for `cpu_fallback` (always) and for
|
||||
`accelerated` ONLY when it carries a driver/arch caveat reason — everything
|
||||
else (`cpu_only`, clean `accelerated`, `n/a`) is benign and stays silent."""
|
||||
st = result["routing_status"]
|
||||
if st == "cpu_fallback" or (st == "accelerated" and result["routing_reason"]):
|
||||
return (st, result["routing_reason"])
|
||||
return None
|
||||
|
||||
|
||||
def header_safe_reason(reason: str | None) -> str | None:
|
||||
"""A routing reason made safe for an HTTP header value: scrubbed, then
|
||||
ASCII-sanitized (headers are latin-1; a non-ASCII device name would 500 the
|
||||
response otherwise), **control characters stripped** (a CR/LF could split
|
||||
the header / inject a new one), and length-capped at 256. Returns None for
|
||||
an empty reason. No regex — `.encode`/membership only (CodeQL-clean)."""
|
||||
if not reason:
|
||||
return None
|
||||
from core.scrub import scrub_text
|
||||
ascii_only = scrub_text(reason).encode("ascii", "ignore").decode("ascii")
|
||||
# Drop ASCII control chars (0x00-0x1F + DEL 0x7F) — incl. CR/LF, so the
|
||||
# value can never break out of its header line.
|
||||
cleaned = "".join(c for c in ascii_only if 0x20 <= ord(c) < 0x7F)
|
||||
return cleaned[:256] or None
|
||||
|
||||
|
||||
def routing_fields(gpu_compat: tuple[str, ...], caps: HostCaps) -> dict:
|
||||
"""The three serialization-ready routing keys for a ``list_backends`` entry.
|
||||
|
||||
Resolves routing and applies the redaction contract: ``routing_reason`` is
|
||||
scrubbed via ``core.scrub.scrub_text`` only when truthy, so a ``None`` reason
|
||||
serializes as JSON ``null`` (NOT ``""`` — ``scrub_text(None)`` would coerce
|
||||
to ``""``). Used by tts/asr ``list_backends`` so the scrub rule lives in one
|
||||
place. (LLM emits its own literal ``network``/``n/a``/``null`` fields and
|
||||
does NOT call this.)
|
||||
"""
|
||||
from core.scrub import scrub_text
|
||||
|
||||
r = resolve_routing(tuple(gpu_compat or ()), caps)
|
||||
reason = r["routing_reason"]
|
||||
return {
|
||||
"effective_device": r["effective_device"],
|
||||
"routing_status": r["routing_status"],
|
||||
"routing_reason": scrub_text(reason) if reason else None,
|
||||
}
|
||||
|
||||
|
||||
__all__ = [
|
||||
"RoutingStatus", "RoutingResult", "resolve_routing", "routing_fields",
|
||||
"routing_notice", "header_safe_reason",
|
||||
]
|
||||
@@ -5,10 +5,6 @@ import os
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
# Leaf module (stdlib-only) — safe to import at module top, unlike
|
||||
# services.dub_pipeline which imports this module and would cycle.
|
||||
from services.proc_registry import register_proc, unregister_proc
|
||||
|
||||
logger = logging.getLogger("omnivoice.api")
|
||||
|
||||
# Cap concurrent ffmpeg jobs so macOS posix_spawn can't hit EAGAIN under load.
|
||||
@@ -23,36 +19,6 @@ def _get_semaphore() -> asyncio.Semaphore:
|
||||
return _FFMPEG_SEMAPHORE
|
||||
|
||||
|
||||
# Candidate paths that exist but won't run (validated once per process).
|
||||
# Windows users hit this as `[WinError 193] %1 is not a valid Win32
|
||||
# application` (#360/#361/#362): a corrupt/wrong-arch imageio-ffmpeg
|
||||
# download or a WindowsApps alias stub passes `os.path.isfile` / `which`
|
||||
# but explodes at spawn. Probe each candidate with `-version` and fall
|
||||
# through to the next source instead of returning a time bomb.
|
||||
_BINARY_OK: dict[str, bool] = {}
|
||||
|
||||
|
||||
def _binary_runs(path: str) -> bool:
|
||||
cached = _BINARY_OK.get(path)
|
||||
if cached is not None:
|
||||
return cached
|
||||
try:
|
||||
subprocess.run(
|
||||
[path, "-version"],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
|
||||
timeout=10, check=False,
|
||||
)
|
||||
ok = True
|
||||
except (OSError, subprocess.TimeoutExpired, subprocess.SubprocessError) as e:
|
||||
logger.warning(
|
||||
"Rejecting non-runnable ffmpeg/ffprobe candidate %s: %s",
|
||||
os.path.basename(str(path)), e,
|
||||
)
|
||||
ok = False
|
||||
_BINARY_OK[path] = ok
|
||||
return ok
|
||||
|
||||
|
||||
def find_ffmpeg():
|
||||
"""Locate an ffmpeg binary.
|
||||
|
||||
@@ -67,15 +33,15 @@ def find_ffmpeg():
|
||||
env_path = os.environ.get("FFMPEG_PATH")
|
||||
if env_path:
|
||||
resolved = shutil.which(env_path)
|
||||
if resolved and _binary_runs(resolved):
|
||||
if resolved:
|
||||
return resolved
|
||||
# 2. imageio-ffmpeg bundled static binary
|
||||
try:
|
||||
import imageio_ffmpeg
|
||||
candidate = imageio_ffmpeg.get_ffmpeg_exe()
|
||||
if candidate and os.path.isfile(candidate) and _binary_runs(candidate):
|
||||
if candidate and os.path.isfile(candidate):
|
||||
return candidate
|
||||
logger.debug("imageio_ffmpeg binary not usable at %s", candidate)
|
||||
logger.debug("imageio_ffmpeg binary not found at %s", candidate)
|
||||
except Exception as e:
|
||||
logger.debug("imageio_ffmpeg unavailable: %s", e)
|
||||
# 3. Well-known system paths + PATH lookup
|
||||
@@ -88,10 +54,9 @@ def find_ffmpeg():
|
||||
"ffmpeg",
|
||||
]
|
||||
for path in common:
|
||||
resolved = shutil.which(path)
|
||||
if resolved and _binary_runs(resolved):
|
||||
return resolved
|
||||
logger.warning("ffmpeg not found (or not runnable) in env, imageio, or system PATH")
|
||||
if shutil.which(path):
|
||||
return path
|
||||
logger.warning("ffmpeg not found in env, imageio, or system PATH")
|
||||
return None
|
||||
|
||||
|
||||
@@ -115,14 +80,14 @@ def resolve_ffprobe() -> str | None:
|
||||
continue
|
||||
# The env var may carry either an absolute path to a file OR a bare
|
||||
# command name (legacy). Accept both shapes — file first.
|
||||
if os.path.isfile(path) and _binary_runs(path):
|
||||
if os.path.isfile(path):
|
||||
return path
|
||||
resolved = shutil.which(path)
|
||||
if resolved and _binary_runs(resolved):
|
||||
if resolved:
|
||||
return resolved
|
||||
|
||||
system_probe = shutil.which("ffprobe")
|
||||
if system_probe and _binary_runs(system_probe):
|
||||
if system_probe:
|
||||
return system_probe
|
||||
return None
|
||||
|
||||
@@ -241,168 +206,16 @@ async def _spawn_with_retry(cmd, **kwargs):
|
||||
raise last_err if last_err else RuntimeError("spawn failed")
|
||||
|
||||
|
||||
def _atempo_chain(ratio: float) -> str:
|
||||
"""Build an `atempo=…,atempo=…` filter chain for arbitrary ratios.
|
||||
|
||||
ffmpeg's atempo filter is limited to [0.5, 2.0] per stage. Chaining
|
||||
multiple stages multiplies the effective ratio while keeping each
|
||||
individual stage inside the well-behaved range. Pitch is preserved
|
||||
(WSOLA-style time-domain stretching). ratio > 1 speeds up, < 1
|
||||
slows down.
|
||||
"""
|
||||
stages: list[str] = []
|
||||
remaining = ratio
|
||||
while remaining > 2.0:
|
||||
stages.append("atempo=2.0")
|
||||
remaining /= 2.0
|
||||
while remaining < 0.5:
|
||||
stages.append("atempo=0.5")
|
||||
remaining /= 0.5
|
||||
stages.append(f"atempo={remaining:.6f}")
|
||||
return ",".join(stages)
|
||||
|
||||
|
||||
async def _pitch_preserving_stretch(wav, target_samples: int, sr: int):
|
||||
"""Time-stretch a (1, samples) tensor to `target_samples` while
|
||||
preserving pitch, by piping the audio through `ffmpeg atempo`.
|
||||
|
||||
Async so it never blocks the event loop: it's awaited from the dub
|
||||
generate `_stream` generator, and each ffmpeg call is ~50-100 ms — a
|
||||
synchronous ``subprocess.run`` here froze health-checks / SSE / every
|
||||
concurrent request for the whole multi-segment job.
|
||||
|
||||
Returns a (1, target_samples) tensor on the same device as input.
|
||||
Raises RuntimeError when ffmpeg fails — callers should fall back to
|
||||
naive linear interpolation, accepting the pitch shift, to ensure the
|
||||
output isn't silent.
|
||||
"""
|
||||
# Lazy imports keep this module importable in torch-free contexts
|
||||
# (setup scripts, smoke probes) — only the stretch path needs them.
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
wl = int(wav.shape[-1])
|
||||
if target_samples <= 0 or wl == target_samples:
|
||||
return wav
|
||||
ratio = wl / target_samples
|
||||
filter_str = _atempo_chain(ratio)
|
||||
|
||||
# Mono float32 via stdin → ffmpeg → stdout. One subprocess per segment,
|
||||
# run off the event loop so concurrent requests stay responsive.
|
||||
arr = wav.detach().cpu().to(torch.float32).numpy().reshape(-1).astype(np.float32, copy=False)
|
||||
proc = await spawn_subprocess(
|
||||
find_ffmpeg(), "-hide_banner", "-loglevel", "error", "-y",
|
||||
"-f", "f32le", "-ar", str(sr), "-ac", "1", "-i", "pipe:0",
|
||||
"-af", filter_str,
|
||||
"-f", "f32le", "-ar", str(sr), "-ac", "1", "pipe:1",
|
||||
stdin=asyncio.subprocess.PIPE,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, stderr = await proc.communicate(input=arr.tobytes())
|
||||
if proc.returncode != 0 or not stdout:
|
||||
raise RuntimeError(
|
||||
(stderr.decode(errors="replace") or "atempo failed")[:200]
|
||||
)
|
||||
out_arr = np.frombuffer(stdout, dtype=np.float32)
|
||||
# atempo rarely lands exactly on the integer sample count, so
|
||||
# pad/trim to the requested slot length.
|
||||
if len(out_arr) < target_samples:
|
||||
pad = np.zeros(target_samples - len(out_arr), dtype=np.float32)
|
||||
out_arr = np.concatenate([out_arr, pad])
|
||||
elif len(out_arr) > target_samples:
|
||||
out_arr = out_arr[:target_samples]
|
||||
return torch.from_numpy(out_arr.copy()).unsqueeze(0).to(wav.device)
|
||||
|
||||
|
||||
async def probe_duration(path: str) -> float | None:
|
||||
"""Return a media file's duration in seconds via ffprobe, or None.
|
||||
|
||||
Used by the Smart Fit pipeline to sanity-check source/track lengths
|
||||
without loading the media. Never raises — probing is best-effort.
|
||||
"""
|
||||
ffprobe = find_ffprobe()
|
||||
if not ffprobe or not os.path.isfile(path):
|
||||
return None
|
||||
try:
|
||||
proc = await spawn_subprocess(
|
||||
ffprobe, "-v", "error",
|
||||
"-show_entries", "format=duration",
|
||||
"-of", "default=noprint_wrappers=1:nokey=1",
|
||||
path,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, _ = await proc.communicate()
|
||||
if proc.returncode != 0:
|
||||
return None
|
||||
return float(stdout.decode().strip())
|
||||
except Exception as e:
|
||||
logger.debug("probe_duration failed for %s: %s", os.path.basename(str(path)), e)
|
||||
return None
|
||||
|
||||
|
||||
async def probe_frame_rates(path: str) -> "tuple[str, str] | None":
|
||||
"""Return (r_frame_rate, avg_frame_rate) strings for the first video
|
||||
stream (e.g. ``("30000/1001", "2997/100")``), or None on any failure.
|
||||
|
||||
A mismatch between the two is the practical VFR signature — used by the
|
||||
Smart Fit retime pipeline to decide whether to normalise with ``fps=``
|
||||
before trim/setpts. Never raises — probing is best-effort.
|
||||
"""
|
||||
ffprobe = find_ffprobe()
|
||||
if not ffprobe or not os.path.isfile(path):
|
||||
return None
|
||||
try:
|
||||
proc = await spawn_subprocess(
|
||||
ffprobe, "-v", "error",
|
||||
"-select_streams", "v:0",
|
||||
"-show_entries", "stream=r_frame_rate,avg_frame_rate",
|
||||
"-of", "csv=p=0",
|
||||
path,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, _ = await proc.communicate()
|
||||
if proc.returncode != 0:
|
||||
return None
|
||||
parts = stdout.decode().strip().split(",")
|
||||
if len(parts) < 2:
|
||||
return None
|
||||
return parts[0].strip(), parts[1].strip()
|
||||
except Exception as e:
|
||||
logger.debug("probe_frame_rates failed for %s: %s", os.path.basename(str(path)), e)
|
||||
return None
|
||||
|
||||
|
||||
async def run_ffmpeg(cmd, timeout: float = 1800.0, capture: bool = True,
|
||||
job_id: "str | None" = None):
|
||||
async def run_ffmpeg(cmd, timeout: float = 1800.0, capture: bool = True):
|
||||
"""Run an ffmpeg subprocess with concurrency cap, timeout, and proper cleanup.
|
||||
|
||||
Returns (returncode, stdout_bytes, stderr_bytes). Raises asyncio.TimeoutError
|
||||
on hard timeout (after killing + reaping the process).
|
||||
|
||||
``job_id`` (optional) registers the process with the dub pipeline's
|
||||
process tracker (``services.proc_registry``) so ``/dub/abort`` can kill
|
||||
long export encodes (used by the Smart Fit batched retime).
|
||||
|
||||
Path-injection note: every filesystem path placed in ``cmd`` by callers
|
||||
is realpath-normalised and containment-checked against its workspace
|
||||
root (e.g. DUB_DIR) at the call site before the argv is assembled —
|
||||
see api.routers.dub_export and services.video_retime.
|
||||
"""
|
||||
stdout = asyncio.subprocess.PIPE if capture else asyncio.subprocess.DEVNULL
|
||||
stderr = asyncio.subprocess.PIPE
|
||||
async with _get_semaphore():
|
||||
proc = await _spawn_with_retry(cmd, stdout=stdout, stderr=stderr)
|
||||
if job_id:
|
||||
try:
|
||||
register_proc(job_id, proc)
|
||||
except Exception as e:
|
||||
# Newline-strip the id inline — it can originate from a path
|
||||
# param, and the log stream must stay one-event-per-line.
|
||||
logger.debug("register_proc failed for %s: %s",
|
||||
job_id.replace("\n", " ").replace("\r", " "), e)
|
||||
try:
|
||||
try:
|
||||
out, err = await asyncio.wait_for(proc.communicate(), timeout=timeout)
|
||||
@@ -418,12 +231,6 @@ async def run_ffmpeg(cmd, timeout: float = 1800.0, capture: bool = True,
|
||||
raise
|
||||
return proc.returncode, out, err
|
||||
finally:
|
||||
if job_id:
|
||||
try:
|
||||
unregister_proc(job_id, proc)
|
||||
except Exception as e:
|
||||
logger.debug("unregister_proc failed for %s: %s",
|
||||
job_id.replace("\n", " ").replace("\r", " "), e)
|
||||
# Guarantee reaping — prevents zombie pileup under timeouts or errors.
|
||||
if proc.returncode is None:
|
||||
try:
|
||||
|
||||
@@ -1,211 +0,0 @@
|
||||
"""Smart Fit planner — dub-length fitting v2, Phase A.
|
||||
|
||||
Pure planning functions for the ``smart_fit`` timing strategy: given the
|
||||
original segment timeline and the *natural-rate* duration of each dubbed
|
||||
segment's TTS audio, decide per segment how to reconcile the two by
|
||||
splitting the burden between a mild pitch-preserving audio speed-up and a
|
||||
mild per-segment video slow-down.
|
||||
|
||||
Clean-room note: this is a reimplementation from a *published description*
|
||||
of the audio-speedup + video-slowdown fitting approach (see
|
||||
docs/competitive-analysis.md, "Dub-length fitting"). No GPL source was
|
||||
consulted.
|
||||
|
||||
Algorithm per segment (defaults in :class:`FitParams`):
|
||||
|
||||
1. **Slack absorption** — the usable slot extends past the segment's
|
||||
original end into the silent gap before the next segment, keeping a
|
||||
small ``gap_guard_s`` clear of the next onset (the last segment may run
|
||||
to the end of the video). ``need = natural_dur / slot``.
|
||||
2. ``need <= 1.0`` — fits as-is; nothing to do.
|
||||
3. ``1.0 < need <= max_audio_only_rate`` — audio-only speed-up at exactly
|
||||
``need`` (imperceptible up to ~1.2×).
|
||||
4. ``need > max_audio_only_rate`` — geometric 50/50 split:
|
||||
``audio_rate = min(sqrt(need), audio_rate_cap)`` and
|
||||
``video_ratio = min(need / audio_rate, video_slow_cap)``. Whatever the
|
||||
caps can't absorb becomes ``overflow_s`` (trimmed at mix time).
|
||||
5. ``allow_video_retime=False`` — audio-only mode: rate capped at the
|
||||
legacy ``MAX_AUDIO_RATE_HARD`` (1.8, matching dub_generate's
|
||||
MAX_STRETCH_RATIO guard rail), residual overflows.
|
||||
6. **Timeline cursor** — mirrors the existing ``stretch_video`` layout
|
||||
loop: pre-roll and inter-segment gaps pass through at 1.0×; each
|
||||
segment's video chunk ``[start, effective_end]`` occupies
|
||||
``slot * video_ratio`` on the new timeline.
|
||||
|
||||
This module is deliberately I/O-free and torch-free so it can be unit- and
|
||||
golden-tested without a model, ffmpeg, or an event loop.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
# Hard ceiling for audio-only compression when video retiming is disabled.
|
||||
# Matches dub_generate.MAX_STRETCH_RATIO — above ~1.8× speech becomes a
|
||||
# garbled stream no DSP can rescue.
|
||||
MAX_AUDIO_RATE_HARD = 1.8
|
||||
|
||||
_EPS = 1e-9
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FitParams:
|
||||
"""Tunable knobs for the Smart Fit planner.
|
||||
|
||||
All defaults are deliberately conservative: ≤1.2× audio-only is
|
||||
imperceptible to most listeners; 1.5× audio is the intelligibility
|
||||
cap; 2.0× video slow-down is the limit before motion looks syrupy.
|
||||
"""
|
||||
max_audio_only_rate: float = 1.2
|
||||
audio_rate_cap: float = 1.5
|
||||
video_slow_cap: float = 2.0
|
||||
gap_guard_s: float = 0.05
|
||||
allow_video_retime: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class SegmentFit:
|
||||
"""Planner verdict for one segment."""
|
||||
index: int
|
||||
seg_id: str
|
||||
audio_rate: float # ≥ 1.0 — pitch-preserving speed-up applied to TTS audio
|
||||
video_ratio: float # ≥ 1.0 — setpts slow-down applied to the video chunk
|
||||
new_start: float # placement on the fitted (possibly longer) timeline
|
||||
new_end: float # end of the video chunk on the fitted timeline
|
||||
orig_start: float
|
||||
orig_end: float
|
||||
effective_end: float # orig_end + absorbed slack (≤ next start − gap guard)
|
||||
status: str # "fits" | "audio_stretched" | "hybrid" | "overflow_trimmed"
|
||||
overflow_s: float # seconds of (stretched) audio that still don't fit
|
||||
|
||||
|
||||
@dataclass
|
||||
class FitPlan:
|
||||
"""Full plan for one dub track."""
|
||||
segments: list[SegmentFit] = field(default_factory=list)
|
||||
# EXACT dict shape consumed by dub_export._build_video_stretch_filter_graph.
|
||||
video_plan: list[dict] = field(default_factory=list)
|
||||
total_duration: float = 0.0
|
||||
orig_duration: float = 0.0
|
||||
params: FitParams = field(default_factory=FitParams)
|
||||
|
||||
@property
|
||||
def needs_video_retime(self) -> bool:
|
||||
return any(s.video_ratio > 1.0 + 1e-6 for s in self.segments)
|
||||
|
||||
|
||||
def _fit_one(need: float, params: FitParams) -> tuple[float, float, str]:
|
||||
"""Resolve one segment's need ratio into (audio_rate, video_ratio, status)."""
|
||||
if need <= 1.0 + _EPS:
|
||||
return 1.0, 1.0, "fits"
|
||||
if need <= params.max_audio_only_rate + _EPS:
|
||||
return need, 1.0, "audio_stretched"
|
||||
if not params.allow_video_retime:
|
||||
audio_rate = min(need, MAX_AUDIO_RATE_HARD)
|
||||
status = "audio_stretched" if audio_rate >= need - _EPS else "overflow_trimmed"
|
||||
return audio_rate, 1.0, status
|
||||
# Geometric 50/50 split: equal perceptual burden on audio and video.
|
||||
audio_rate = min(math.sqrt(need), params.audio_rate_cap)
|
||||
video_ratio = min(need / audio_rate, params.video_slow_cap)
|
||||
if audio_rate * video_ratio >= need - _EPS:
|
||||
return audio_rate, video_ratio, "hybrid"
|
||||
return audio_rate, video_ratio, "overflow_trimmed"
|
||||
|
||||
|
||||
def plan_fit(
|
||||
segments: list[dict],
|
||||
natural_durs_s: list[float],
|
||||
total_dur_s: float,
|
||||
params: FitParams | None = None,
|
||||
) -> FitPlan:
|
||||
"""Plan the Smart Fit layout for a dub track.
|
||||
|
||||
``segments``: original-timeline segments in chronological order, each a
|
||||
dict with ``id``, ``start``, ``end`` (seconds). ``natural_durs_s``: the
|
||||
natural-rate TTS audio duration for each segment (parallel list).
|
||||
``total_dur_s``: original video duration (0/unknown tolerated — the last
|
||||
segment then gets no tail slack).
|
||||
|
||||
Pure function: no I/O, no torch, deterministic.
|
||||
"""
|
||||
params = params or FitParams()
|
||||
n = len(segments)
|
||||
if len(natural_durs_s) != n:
|
||||
raise ValueError(
|
||||
f"segments ({n}) and natural_durs_s ({len(natural_durs_s)}) must be parallel"
|
||||
)
|
||||
|
||||
plan = FitPlan(params=params, orig_duration=round(float(total_dur_s), 4))
|
||||
if n == 0:
|
||||
plan.total_duration = round(max(0.0, float(total_dur_s)), 4)
|
||||
return plan
|
||||
|
||||
cursor = 0.0
|
||||
for i, seg in enumerate(segments):
|
||||
start = float(seg["start"])
|
||||
end = float(seg["end"])
|
||||
natural = max(0.0, float(natural_durs_s[i]))
|
||||
|
||||
# (a) Slack absorption. Extend-only: the slot never shrinks below
|
||||
# the original [start, end] even when segments are back-to-back.
|
||||
if i + 1 < n:
|
||||
next_start = float(segments[i + 1]["start"])
|
||||
effective_end = max(end, next_start - params.gap_guard_s)
|
||||
# Never bleed past the next segment's onset (overlapping or
|
||||
# near-touching source segments).
|
||||
effective_end = min(max(effective_end, start), max(next_start, end))
|
||||
else:
|
||||
effective_end = max(end, float(total_dur_s)) if total_dur_s > 0 else end
|
||||
slot = max(effective_end - start, 1e-3)
|
||||
|
||||
need = natural / slot if natural > 0 else 0.0
|
||||
audio_rate, video_ratio, status = _fit_one(need, params)
|
||||
|
||||
# (f) Timeline cursor — mirror the stretch_video layout loop:
|
||||
# pre-roll and gaps at 1.0×, the segment's video chunk
|
||||
# [start, effective_end] occupies slot × video_ratio.
|
||||
if i == 0:
|
||||
cursor = start # pre-roll preserved at native rate
|
||||
new_start = cursor
|
||||
new_end = new_start + slot * video_ratio
|
||||
cursor = new_end
|
||||
if i + 1 < n:
|
||||
# Unretimed sliver between this chunk and the next chunk's
|
||||
# start (the gap guard, or more if extend-only clamped).
|
||||
cursor += max(0.0, float(segments[i + 1]["start"]) - effective_end)
|
||||
|
||||
# Residual overflow after both knobs: stretched audio length vs the
|
||||
# segment's new video slot.
|
||||
stretched = natural / audio_rate if audio_rate > 0 else natural
|
||||
overflow_s = max(0.0, stretched - slot * video_ratio)
|
||||
if overflow_s <= 1e-6:
|
||||
overflow_s = 0.0
|
||||
elif status != "overflow_trimmed":
|
||||
status = "overflow_trimmed"
|
||||
|
||||
plan.segments.append(SegmentFit(
|
||||
index=i,
|
||||
seg_id=str(seg.get("id", f"seg_{i}")),
|
||||
audio_rate=round(audio_rate, 6),
|
||||
video_ratio=round(video_ratio, 6),
|
||||
new_start=round(new_start, 4),
|
||||
new_end=round(new_end, 4),
|
||||
orig_start=round(start, 4),
|
||||
orig_end=round(end, 4),
|
||||
effective_end=round(effective_end, 4),
|
||||
status=status,
|
||||
overflow_s=round(overflow_s, 4),
|
||||
))
|
||||
plan.video_plan.append({
|
||||
"orig_start": round(start, 4),
|
||||
"orig_end": round(effective_end, 4),
|
||||
"new_start": round(new_start, 4),
|
||||
"new_end": round(new_end, 4),
|
||||
"stretch_ratio": round(video_ratio, 4),
|
||||
})
|
||||
|
||||
# Tail (anything after the last segment's effective end) at 1.0×.
|
||||
last_eff = float(plan.segments[-1].effective_end)
|
||||
cursor += max(0.0, float(total_dur_s) - last_eff)
|
||||
plan.total_duration = round(max(cursor, float(total_dur_s)), 4)
|
||||
return plan
|
||||
@@ -1,65 +0,0 @@
|
||||
"""Map subtitle cues onto the Smart-Fit timeline (Wave 3.1 / Spec 1).
|
||||
|
||||
When a dub uses ``stretch_video`` mode, the video is re-timed per segment so
|
||||
the dubbed audio fits (see fit_planner + the export stretch filter). The
|
||||
dubbed audio therefore plays at *fitted* positions, not the original
|
||||
timestamps. A subtitle file exported with the original times would drift
|
||||
against the dubbed video — so we regenerate the cue timeline from the same
|
||||
plan the video stretch uses ("subtitles track actual dub placement", the
|
||||
last piece of Spec 1).
|
||||
|
||||
Pure functions — no I/O — so the remapping is unit-testable. The plan is the
|
||||
persisted ``video_stretch_plan`` list of
|
||||
``{orig_start, orig_end, new_start, new_end, stretch_ratio}`` chunks.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def map_time_to_fitted(t: float, plan: list[dict]) -> float:
|
||||
"""Map a time on the original timeline to its position on the fitted one.
|
||||
|
||||
Finds the plan chunk whose original span contains ``t`` and interpolates
|
||||
linearly into that chunk's fitted span (a chunk's stretch is uniform).
|
||||
Before the first chunk maps 1:1; after the last chunk the trailing offset
|
||||
is carried at 1:1 (the planner runs gaps/tail at rate 1.0). Empty plan ⇒
|
||||
identity.
|
||||
"""
|
||||
if not plan:
|
||||
return t
|
||||
for chunk in plan:
|
||||
o0 = float(chunk.get("orig_start", 0.0))
|
||||
o1 = float(chunk.get("orig_end", 0.0))
|
||||
n0 = float(chunk.get("new_start", o0))
|
||||
n1 = float(chunk.get("new_end", o1))
|
||||
if t < o0:
|
||||
# In a gap before this chunk — carry the offset at 1:1 from the
|
||||
# previous chunk's fitted end (or from 0 for the very first).
|
||||
return n0 - (o0 - t)
|
||||
if o0 <= t <= o1:
|
||||
span = o1 - o0
|
||||
if span <= 0:
|
||||
return n0
|
||||
return n0 + (t - o0) / span * (n1 - n0)
|
||||
# Past the last chunk: 1:1 tail from its fitted end.
|
||||
last = plan[-1]
|
||||
return float(last.get("new_end", 0.0)) + (t - float(last.get("orig_end", 0.0)))
|
||||
|
||||
|
||||
def fitted_cues(segments: list[dict], plan: list[dict]) -> list[tuple[float, float]]:
|
||||
"""Return ``[(start, end), ...]`` for each segment on the fitted timeline.
|
||||
|
||||
Monotonicity guard: a cue's end is never before its start, and successive
|
||||
starts never go backwards (rounding across chunk seams can't produce a
|
||||
non-monotone SRT).
|
||||
"""
|
||||
out: list[tuple[float, float]] = []
|
||||
prev_end = 0.0
|
||||
for seg in segments:
|
||||
s = map_time_to_fitted(float(seg.get("start", 0.0)), plan)
|
||||
e = map_time_to_fitted(float(seg.get("end", 0.0)), plan)
|
||||
s = max(s, prev_end if out else 0.0)
|
||||
e = max(e, s)
|
||||
out.append((s, e))
|
||||
prev_end = e
|
||||
return out
|
||||
@@ -22,32 +22,6 @@ import json
|
||||
|
||||
_GEN_INPUT_FIELDS = ("text", "target_lang", "profile_id", "instruct", "speed", "direction", "effect_preset")
|
||||
|
||||
# Pydantic fills `effect_preset` with this default when the client omits it,
|
||||
# while the client-side recompute (/tools/incremental) sends nothing. Both
|
||||
# representations must hash identically or every segment looks "stale" after
|
||||
# every generate and incremental re-dub degrades to a full re-dub (#281).
|
||||
_DEFAULT_EFFECT_PRESET = "broadcast"
|
||||
|
||||
|
||||
def _canon_value(field: str, value):
|
||||
"""Normalise one generation-input value so that the server-side view
|
||||
(pydantic-parsed `DubSegment`, defaults filled in) and the client-side
|
||||
view (raw segment dict, unset keys omitted) produce the same hash.
|
||||
|
||||
- missing / None / "" all mean "default" for string fields
|
||||
- `effect_preset` default is "broadcast" (pydantic fills it server-side)
|
||||
- numbers are coerced to float so `speed: 1` (JS) == `speed: 1.0` (pydantic)
|
||||
"""
|
||||
if field == "effect_preset":
|
||||
return value or _DEFAULT_EFFECT_PRESET
|
||||
if value is None or value == "":
|
||||
return ""
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
if isinstance(value, (int, float)):
|
||||
return float(value)
|
||||
return value
|
||||
|
||||
|
||||
def segment_fingerprint(seg: dict) -> str:
|
||||
"""Deterministic hash of the inputs that actually affect TTS output.
|
||||
@@ -57,63 +31,11 @@ def segment_fingerprint(seg: dict) -> str:
|
||||
badge don't trigger regen, which is what we want.
|
||||
|
||||
Currently includes: text, target_lang, profile_id, instruct, speed,
|
||||
direction, effect_preset. Values are canonicalised (see `_canon_value`)
|
||||
so a fingerprint computed from the generate request (server defaults
|
||||
filled in) matches one recomputed later from the client's raw segment
|
||||
state — the root cause of #281's "1 edit re-dubs all N lines".
|
||||
direction, effect_preset.
|
||||
"""
|
||||
payload = {k: _canon_value(k, seg.get(k)) for k in _GEN_INPUT_FIELDS}
|
||||
payload = {k: (seg.get(k) if seg.get(k) is not None else "") for k in _GEN_INPUT_FIELDS}
|
||||
blob = json.dumps(payload, sort_keys=True, ensure_ascii=False)
|
||||
return hashlib.sha1(blob.encode("utf-8"), usedforsecurity=False).hexdigest()[:16]
|
||||
|
||||
|
||||
# ── Smart Fit (dub-length fitting v2) fingerprints ─────────────────────────
|
||||
#
|
||||
# Fitting parameters stay OUT of segment_fingerprint on purpose: changing a
|
||||
# fit knob (caps, gap guard, strategy) must trigger a RE-MIX of the already
|
||||
# rendered natural-rate WAVs (generate with regen_only=[]), never a re-TTS.
|
||||
# A separate per-track fingerprint tracks the fit configuration; a dubbed
|
||||
# track is stale iff its fit_fp differs OR any segment hash differs.
|
||||
|
||||
_FIT_PARAM_FIELDS = (
|
||||
"timing_strategy",
|
||||
"max_audio_only_rate",
|
||||
"audio_rate_cap",
|
||||
"video_slow_cap",
|
||||
"gap_guard_s",
|
||||
"allow_video_retime",
|
||||
)
|
||||
|
||||
# Server-side defaults (must mirror services.fit_planner.FitParams). Filled
|
||||
# in for omitted keys so a fingerprint computed from a fully-populated
|
||||
# server view matches one recomputed from a sparse client payload — the
|
||||
# same #281 regression class segment_fingerprint already guards against.
|
||||
_FIT_PARAM_DEFAULTS = {
|
||||
"timing_strategy": "smart_fit",
|
||||
"max_audio_only_rate": 1.2,
|
||||
"audio_rate_cap": 1.5,
|
||||
"video_slow_cap": 2.0,
|
||||
"gap_guard_s": 0.05,
|
||||
"allow_video_retime": True,
|
||||
}
|
||||
|
||||
|
||||
def fit_fingerprint(params: dict) -> str:
|
||||
"""Deterministic hash of the fit configuration for one dub track.
|
||||
|
||||
Canonicalised with the same `_canon_value` rules as segment
|
||||
fingerprints (int vs float, None/"" vs omitted), plus default-filling
|
||||
so `{}` and `{"audio_rate_cap": 1.5}` hash identically.
|
||||
"""
|
||||
params = params or {}
|
||||
payload = {}
|
||||
for k in _FIT_PARAM_FIELDS:
|
||||
v = params.get(k)
|
||||
if v is None or v == "":
|
||||
v = _FIT_PARAM_DEFAULTS[k]
|
||||
payload[k] = _canon_value(k, v)
|
||||
blob = json.dumps(payload, sort_keys=True, ensure_ascii=False)
|
||||
return hashlib.sha1(blob.encode("utf-8"), usedforsecurity=False).hexdigest()[:16]
|
||||
return hashlib.sha1(blob.encode("utf-8")).hexdigest()[:16]
|
||||
|
||||
|
||||
def plan_incremental(
|
||||
|
||||
@@ -37,12 +37,6 @@ logger = logging.getLogger("omnivoice.llm")
|
||||
class LLMBackend(ABC):
|
||||
id: str = "base"
|
||||
display_name: str = "Base LLM"
|
||||
# LLM backends call out over the network (OpenAI/Ollama/LM Studio) or are a
|
||||
# no-op — none run a model on the user's GPU. So `gpu_compat` is empty and
|
||||
# list_backends() labels the family `effective_device:"network"` /
|
||||
# routing_status:"n/a" rather than asserting a false GPU claim. Routing is
|
||||
# never gated for LLM (see engines.select_engine + diagnose).
|
||||
gpu_compat: tuple[str, ...] = ()
|
||||
|
||||
@classmethod
|
||||
@abstractmethod
|
||||
@@ -112,22 +106,6 @@ class OpenAICompatBackend(LLMBackend):
|
||||
return self._client
|
||||
|
||||
def chat(self, *, system: str, user: str, timeout: Optional[float] = None) -> str:
|
||||
return self.chat_messages(
|
||||
messages=[
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
],
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
def chat_messages(self, *, messages: list[dict], timeout: Optional[float] = None) -> str:
|
||||
"""One-shot completion over a full message list.
|
||||
|
||||
Additive surface for callers that need structured few-shot turns
|
||||
(dictation refinement, Wave 2.1) — small local models pattern-match
|
||||
and echo inline examples, so examples must arrive as prior chat
|
||||
turns, not inside the system prompt.
|
||||
"""
|
||||
if timeout is None:
|
||||
try:
|
||||
timeout = float(os.environ.get("OMNIVOICE_LLM_TIMEOUT", "45"))
|
||||
@@ -136,7 +114,10 @@ class OpenAICompatBackend(LLMBackend):
|
||||
res = self._get_client().chat.completions.create(
|
||||
model=self.model_name,
|
||||
timeout=timeout,
|
||||
messages=messages,
|
||||
messages=[
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
],
|
||||
)
|
||||
return (res.choices[0].message.content or "").strip()
|
||||
|
||||
@@ -162,9 +143,6 @@ class OffBackend(LLMBackend):
|
||||
"to use features that need one (Cinematic translate, glossary auto-extract)."
|
||||
)
|
||||
|
||||
def chat_messages(self, **kw) -> str:
|
||||
return self.chat(**kw)
|
||||
|
||||
|
||||
_REGISTRY: dict[str, type[LLMBackend]] = {
|
||||
"openai-compat": OpenAICompatBackend,
|
||||
@@ -172,53 +150,15 @@ _REGISTRY: dict[str, type[LLMBackend]] = {
|
||||
}
|
||||
|
||||
|
||||
# Most-recent failure per backend (parity with tts/asr list_backends).
|
||||
_LAST_ERRORS: dict[str, str] = {}
|
||||
|
||||
_INSTALL_HINTS: dict[str, str] = {
|
||||
"openai-compat": "Set TRANSLATE_BASE_URL (+ TRANSLATE_API_KEY) — OpenAI, "
|
||||
"Ollama (http://localhost:11434/v1), or any compatible host.",
|
||||
}
|
||||
|
||||
|
||||
def list_backends() -> list[dict]:
|
||||
"""Same 11-key shape as tts/asr so the matrix renders families uniformly.
|
||||
|
||||
LLM is NOT a GPU family: every entry carries literal
|
||||
``effective_device:"network"`` / ``routing_status:"n/a"`` /
|
||||
``routing_reason:null`` (NOT via resolve_routing — that would be a false
|
||||
GPU claim). ``effective_device:"network"`` is a label, not a probe: nothing
|
||||
here touches the network (local-first).
|
||||
"""
|
||||
from core.scrub import scrub_text
|
||||
|
||||
out: list[dict] = []
|
||||
out = []
|
||||
for bid, cls in _REGISTRY.items():
|
||||
try:
|
||||
ok, msg = cls.is_available()
|
||||
except Exception as exc:
|
||||
ok = False
|
||||
msg = f"{type(exc).__name__}: {exc}"
|
||||
logger.warning(
|
||||
"llm list_backends: %s.is_available() raised — degrading "
|
||||
"gracefully so the picker still renders: %s", bid, msg,
|
||||
)
|
||||
if ok:
|
||||
_LAST_ERRORS.pop(bid, None)
|
||||
else:
|
||||
_LAST_ERRORS[bid] = scrub_text(msg)
|
||||
ok, msg = cls.is_available()
|
||||
out.append({
|
||||
"id": bid,
|
||||
"display_name": cls.display_name,
|
||||
"available": ok,
|
||||
"reason": None if ok else scrub_text(msg),
|
||||
"install_hint": _INSTALL_HINTS.get(bid),
|
||||
"last_error": _LAST_ERRORS.get(bid),
|
||||
"isolation_mode": "in-process",
|
||||
"gpu_compat": list(getattr(cls, "gpu_compat", ())),
|
||||
"effective_device": "network",
|
||||
"routing_status": "n/a",
|
||||
"routing_reason": None,
|
||||
"reason": None if ok else msg,
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
@@ -1,251 +0,0 @@
|
||||
"""Import plain text / EPUB into the chapter-delimited script the audiobook
|
||||
parser understands.
|
||||
|
||||
Both helpers are pure (bytes/str in, script-str out) so they're unit-tested
|
||||
without a server. EPUB parsing is **stdlib only** (zipfile + ElementTree +
|
||||
html.parser) — no new dependency, no network, consistent with the local-first
|
||||
guarantee. The output is the same ``# Heading`` + body grammar
|
||||
:func:`services.audiobook.parse_audiobook_script` already consumes, so import is
|
||||
just a front door onto the existing pipeline.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import posixpath
|
||||
import re
|
||||
import zipfile
|
||||
from html.parser import HTMLParser
|
||||
from xml.etree import ElementTree as ET
|
||||
|
||||
# A line that *starts* with a chapter keyword and is short enough to be a title
|
||||
# (not a sentence that happens to begin with "Chapter"). Anchored, no ambiguous
|
||||
# quantifiers → ReDoS-safe and applied per-line (short input) anyway.
|
||||
_CH_RE = re.compile(r"^(?:chapter|part|book|prologue|epilogue|section)\b", re.IGNORECASE)
|
||||
# Already-present Markdown H1 — if the text has any, we leave it untouched.
|
||||
_H1_RE = re.compile(r"^[ \t]*#[ \t]+\S", re.MULTILINE)
|
||||
_CHAPTER_TITLE_MAX = 60
|
||||
# Zip-bomb / OOM guards for EPUB ingestion: per-entry and cumulative caps on
|
||||
# *uncompressed* bytes read from the archive.
|
||||
_EPUB_MAX_ENTRY_BYTES = 25 * 1024 * 1024
|
||||
_EPUB_MAX_TOTAL_BYTES = 300 * 1024 * 1024
|
||||
|
||||
|
||||
def chapterize_plaintext(text: str) -> str:
|
||||
"""Insert ``# `` headings ahead of obvious chapter-title lines.
|
||||
|
||||
No-op if the text already has Markdown H1 headings (the user has structured
|
||||
it). Otherwise short standalone lines beginning with a chapter keyword
|
||||
(``Chapter 3``, ``Prologue`` …) become headings; everything else is left
|
||||
verbatim. Text with no detectable breaks falls through as a single chapter.
|
||||
"""
|
||||
text = text or ""
|
||||
if _H1_RE.search(text):
|
||||
return text
|
||||
out = []
|
||||
for line in text.split("\n"):
|
||||
s = line.strip()
|
||||
if s and len(s) <= _CHAPTER_TITLE_MAX and _CH_RE.match(s):
|
||||
out.append(f"# {s}")
|
||||
else:
|
||||
out.append(line)
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
class _TextExtractor(HTMLParser):
|
||||
"""Collect visible text from XHTML, dropping script/style and collapsing
|
||||
whitespace. First <h1>/<h2>/<title> seen is kept as the chapter title."""
|
||||
|
||||
_SKIP = {"script", "style", "head"}
|
||||
_BREAK = {"p", "br", "div", "h1", "h2", "h3", "li", "tr"}
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__(convert_charrefs=True)
|
||||
self._parts: list[str] = []
|
||||
self._skip_depth = 0
|
||||
self._in_title = False
|
||||
self.title = ""
|
||||
|
||||
def handle_starttag(self, tag, attrs):
|
||||
if tag in self._SKIP:
|
||||
self._skip_depth += 1
|
||||
if tag in ("h1", "h2", "title") and not self.title:
|
||||
self._in_title = True
|
||||
if tag in self._BREAK:
|
||||
self._parts.append("\n")
|
||||
|
||||
def handle_endtag(self, tag):
|
||||
if tag in self._SKIP and self._skip_depth:
|
||||
self._skip_depth -= 1
|
||||
if tag in ("h1", "h2", "title"):
|
||||
self._in_title = False
|
||||
|
||||
def handle_data(self, data):
|
||||
if self._skip_depth:
|
||||
return
|
||||
if self._in_title:
|
||||
# The first heading becomes the chapter's `# Title` (metadata, not
|
||||
# narrated) — capture it but keep it out of the body. Later headings
|
||||
# (title already set) fall through and are narrated as subheadings.
|
||||
if not self.title:
|
||||
self.title = data.strip()
|
||||
return
|
||||
self._parts.append(data)
|
||||
|
||||
def text(self) -> str:
|
||||
raw = "".join(self._parts)
|
||||
# Collapse runs of blank lines / trailing spaces into tidy paragraphs.
|
||||
lines = [ln.strip() for ln in raw.split("\n")]
|
||||
out: list[str] = []
|
||||
for ln in lines:
|
||||
if ln or (out and out[-1]):
|
||||
out.append(ln)
|
||||
return "\n".join(out).strip()
|
||||
|
||||
|
||||
def _html_to_title_body(xhtml: str) -> tuple[str, str]:
|
||||
p = _TextExtractor()
|
||||
try:
|
||||
p.feed(xhtml)
|
||||
except Exception:
|
||||
pass
|
||||
return p.title, p.text()
|
||||
|
||||
|
||||
_OPF_NS = {"opf": "http://www.idpf.org/2007/opf", "c": "urn:oasis:names:tc:opendocument:xmlns:container"}
|
||||
|
||||
|
||||
def _opf_path(zf: zipfile.ZipFile) -> str:
|
||||
container = zf.read("META-INF/container.xml")
|
||||
# The EPUB is a local file the user chose to import (not a remote/untrusted
|
||||
# surface); stdlib ElementTree doesn't expand external entities by default.
|
||||
root = ET.fromstring(container) # nosec B314
|
||||
rootfile = root.find(".//c:rootfiles/c:rootfile", _OPF_NS)
|
||||
if rootfile is None or not rootfile.get("full-path"):
|
||||
raise ValueError("EPUB container.xml has no rootfile")
|
||||
return rootfile.get("full-path")
|
||||
|
||||
|
||||
def epub_to_chapter_script(
|
||||
data: bytes,
|
||||
*,
|
||||
max_entry_bytes: int = _EPUB_MAX_ENTRY_BYTES,
|
||||
max_total_bytes: int = _EPUB_MAX_TOTAL_BYTES,
|
||||
) -> str:
|
||||
"""Convert EPUB bytes into a ``# Chapter`` / body script in spine order.
|
||||
|
||||
Reads the OPF manifest + spine (the publisher's reading order), extracts
|
||||
each document's title + visible text, and emits one ``# Title`` block per
|
||||
document with renderable text. ``max_entry_bytes`` / ``max_total_bytes``
|
||||
bound the *uncompressed* bytes read (zip-bomb guard). Raises ``ValueError``
|
||||
on a malformed EPUB.
|
||||
"""
|
||||
try:
|
||||
zf = zipfile.ZipFile(io.BytesIO(data))
|
||||
except zipfile.BadZipFile as e:
|
||||
raise ValueError(f"not a valid EPUB (zip) file: {e}") from e
|
||||
|
||||
opf_path = _opf_path(zf)
|
||||
opf = ET.fromstring(zf.read(opf_path)) # nosec B314 — local user EPUB; see _opf_path
|
||||
base = posixpath.dirname(opf_path)
|
||||
|
||||
manifest: dict[str, str] = {}
|
||||
for item in opf.findall(".//opf:manifest/opf:item", _OPF_NS):
|
||||
iid, href = item.get("id"), item.get("href")
|
||||
if iid and href:
|
||||
manifest[iid] = href
|
||||
|
||||
blocks: list[str] = []
|
||||
names = set(zf.namelist())
|
||||
total = 0 # cumulative uncompressed bytes read — zip-bomb guard
|
||||
for ref in opf.findall(".//opf:spine/opf:itemref", _OPF_NS):
|
||||
href = manifest.get(ref.get("idref") or "")
|
||||
if not href:
|
||||
continue
|
||||
full = posixpath.normpath(posixpath.join(base, href)) if base else href
|
||||
if full not in names:
|
||||
continue
|
||||
# Bound decompression: skip an absurdly large entry, and stop once the
|
||||
# cumulative uncompressed size crosses the ceiling (defends against a
|
||||
# zip bomb / a maliciously huge chapter exhausting memory).
|
||||
try:
|
||||
info = zf.getinfo(full)
|
||||
except KeyError:
|
||||
continue
|
||||
if info.file_size > max_entry_bytes:
|
||||
continue
|
||||
if total + info.file_size > max_total_bytes:
|
||||
break
|
||||
try:
|
||||
raw = zf.read(full)
|
||||
except KeyError:
|
||||
continue
|
||||
total += len(raw)
|
||||
title, body = _html_to_title_body(raw.decode("utf-8", "ignore"))
|
||||
if not body.strip():
|
||||
continue # nav docs, empty pages
|
||||
title = title or f"Chapter {len(blocks) + 1}"
|
||||
blocks.append(f"# {title}\n\n{body}")
|
||||
|
||||
if not blocks:
|
||||
raise ValueError("no readable chapters found in the EPUB")
|
||||
return "\n\n".join(blocks)
|
||||
|
||||
|
||||
# Page-count ceiling for PDF ingestion — a defence against a pathological
|
||||
# document tying up the worker. 5000 pages comfortably covers any real book.
|
||||
_PDF_MAX_PAGES = 5000
|
||||
|
||||
|
||||
def pdf_to_chapter_script(data: bytes, *, max_pages: int = _PDF_MAX_PAGES) -> str:
|
||||
"""Convert PDF bytes into a ``# Chapter`` / body script.
|
||||
|
||||
Extracts the embedded text layer page-by-page (in page order), joins it,
|
||||
and runs it through :func:`chapterize_plaintext` so ``Chapter N`` /
|
||||
``Prologue`` lines become headings — same grammar EPUB and plaintext emit.
|
||||
Unlike EPUB this needs a real parser (``pypdf``, pure-Python, no native
|
||||
deps → identical on every platform).
|
||||
|
||||
Limitations surfaced as ``ValueError`` (the route maps these to a 400 with
|
||||
the message, so the user gets actionable feedback rather than a silent
|
||||
empty import):
|
||||
|
||||
* **Scanned / image-only PDFs** have no text layer — there's nothing to
|
||||
extract without OCR, so we raise rather than return an empty script.
|
||||
* **Password-protected PDFs** that don't open with an empty password can't
|
||||
be read.
|
||||
"""
|
||||
from pypdf import PdfReader
|
||||
from pypdf.errors import PdfReadError
|
||||
|
||||
try:
|
||||
reader = PdfReader(io.BytesIO(data))
|
||||
except (PdfReadError, OSError, ValueError) as e:
|
||||
raise ValueError(f"not a valid PDF file: {e}") from e
|
||||
|
||||
if reader.is_encrypted:
|
||||
# Many PDFs are encrypted with an empty user password (owner-locked but
|
||||
# freely readable). Try that; a real password we can't supply.
|
||||
try:
|
||||
if reader.decrypt("") == 0: # 0 == wrong password
|
||||
raise ValueError("PDF is password-protected")
|
||||
except (NotImplementedError, PdfReadError) as e:
|
||||
raise ValueError(f"can't read this encrypted PDF: {e}") from e
|
||||
|
||||
pages = reader.pages
|
||||
if len(pages) > max_pages:
|
||||
raise ValueError(f"PDF has too many pages (max {max_pages})")
|
||||
|
||||
parts: list[str] = []
|
||||
for page in pages:
|
||||
try:
|
||||
text = page.extract_text() or ""
|
||||
except Exception: # noqa: BLE001 — one bad page shouldn't kill the import
|
||||
continue
|
||||
if text.strip():
|
||||
parts.append(text)
|
||||
|
||||
if not parts:
|
||||
raise ValueError(
|
||||
"no extractable text — this looks like a scanned or image-only PDF")
|
||||
return chapterize_plaintext("\n\n".join(parts))
|
||||
@@ -1,143 +0,0 @@
|
||||
"""Canonical longform marker parser (#27) — the single source of grammar truth.
|
||||
|
||||
The longform marker dialect (``# heading``, ``[voice:NAME]``, ``[pause …]``,
|
||||
``[slow]/[fast]/[emphasis]/[spell]``) was parsed by three independent code
|
||||
paths that disagreed (client/server/regex-level). This module is the one
|
||||
canonical Python parser; ``frontend/src/utils/longformParser.js`` is its
|
||||
mechanically-mirrored JS twin, and ``tests/fixtures/longform_parser_cases.json``
|
||||
is the shared golden corpus asserted byte-for-byte against both.
|
||||
|
||||
Pure text→plan, import-light (no torch). Grammar precedence (outer→inner):
|
||||
|
||||
# chapter → [voice:] → [pause] → SSML-lite → [spell]
|
||||
|
||||
It reuses the existing pause dialect (``omnivoice.utils.text.parse_pause_markers``)
|
||||
and SSML-lite (``services.ssml_lite``) verbatim so those modules stay the single
|
||||
home of their sub-grammars.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
from omnivoice.utils.text import parse_pause_markers
|
||||
|
||||
# A Markdown H1 (``# Title``) starts a new chapter. Deeper headings (``##``…)
|
||||
# stay in the body as ordinary text. The title capture starts with ``\S`` (a
|
||||
# non-space) so the leading ``[ \t]+`` and the title's ``.*`` can't both match
|
||||
# the same whitespace run — that overlap is what makes ``[ \t]+(.+)``
|
||||
# polynomial-time on adversarial tabs (ReDoS). Moved verbatim from
|
||||
# audiobook.py (already CodeQL-cleared). Stripped in code.
|
||||
_HEADING_RE = re.compile(r"^[ \t]*#[ \t]+(\S.*)$", re.MULTILINE)
|
||||
# ``[voice:NAME]`` switches the active narrator. The content class excludes BOTH
|
||||
# brackets (``[^\]\[]``) so nested ``[voice:`` prefixes can't create overlapping
|
||||
# match attempts across ``finditer`` (the ReDoS source). A voice name never
|
||||
# contains a bracket; the value is stripped in code. Empty → default voice.
|
||||
_VOICE_RE = re.compile(r"\[voice:([^\]\[]*)\]")
|
||||
|
||||
|
||||
def _normalize(text: Optional[str]) -> str:
|
||||
"""Coerce None→'' and normalize CRLF/CR→LF so ``$`` (re.MULTILINE) and span
|
||||
text never carry a stray ``\\r`` on Windows-authored scripts — a
|
||||
cross-platform default-behaviour divergence the JS twin mirrors exactly."""
|
||||
if not text:
|
||||
return ""
|
||||
return text.replace("\r\n", "\n").replace("\r", "\n")
|
||||
|
||||
|
||||
def _parse_chapter_body(
|
||||
body: str,
|
||||
*,
|
||||
default_voice: Optional[str] = None,
|
||||
default_speed: Optional[float] = None,
|
||||
) -> list[dict]:
|
||||
"""Voice→pause→SSML layering for ONE chapter body (no chapter split).
|
||||
|
||||
Returns a list of span dicts ``{voice_id, text, pause_ms_after, speed}``.
|
||||
A ``#`` inside ``body`` is NOT treated as a heading here — that is the
|
||||
caller's (chapter-split) concern. The JS twin (``parseChapterBody``) is what
|
||||
``storyToSpans`` calls per spoken track."""
|
||||
spans: list[dict] = []
|
||||
cur_voice = default_voice
|
||||
runs: list[tuple[Optional[str], str]] = []
|
||||
last = 0
|
||||
for m in _VOICE_RE.finditer(body):
|
||||
if m.start() > last:
|
||||
runs.append((cur_voice, body[last:m.start()]))
|
||||
cur_voice = (m.group(1).strip() or default_voice)
|
||||
last = m.end()
|
||||
runs.append((cur_voice, body[last:]))
|
||||
|
||||
from services.ssml_lite import parse_ssml_lite, spell_out
|
||||
|
||||
for voice, run_text in runs:
|
||||
for span_text, pause_ms in parse_pause_markers(run_text):
|
||||
t = span_text.strip()
|
||||
if not t and pause_ms == 0:
|
||||
continue # pure whitespace between markers — nothing to render
|
||||
rendered: list[tuple[str, Optional[float]]] = []
|
||||
for seg in (parse_ssml_lite(t) if t else []):
|
||||
st = (spell_out(seg["text"]) if seg["spell"] else seg["text"]).strip()
|
||||
if st:
|
||||
# Inline SSML speed overrides the per-line default; a plain
|
||||
# segment inherits default_speed.
|
||||
sp = seg["speed"] if seg["speed"] is not None else default_speed
|
||||
rendered.append((st, sp))
|
||||
if not rendered:
|
||||
# Only-markers / empty text but a real pause → carry the silence.
|
||||
if pause_ms > 0:
|
||||
spans.append({"voice_id": voice, "text": "",
|
||||
"pause_ms_after": pause_ms, "speed": None})
|
||||
continue
|
||||
for j, (st, sp) in enumerate(rendered):
|
||||
spans.append({
|
||||
"voice_id": voice, "text": st,
|
||||
"pause_ms_after": pause_ms if j == len(rendered) - 1 else 0,
|
||||
"speed": sp,
|
||||
})
|
||||
return spans
|
||||
|
||||
|
||||
def parse_script_to_spans(
|
||||
text: Optional[str],
|
||||
*,
|
||||
default_voice: Optional[str] = None,
|
||||
default_speed: Optional[float] = None,
|
||||
) -> list[dict]:
|
||||
"""Parse a chapter-delimited script into ``[{"title", "spans": [...]}, …]``.
|
||||
|
||||
span dict == ``{"voice_id": str|None, "text": str, "pause_ms_after": int,
|
||||
"speed": float|None}`` (key order matches ``Span.to_dict()``).
|
||||
|
||||
Contract:
|
||||
* None / "" / whitespace-only input → ``[]``.
|
||||
* CRLF/CR normalized to LF at entry (cross-platform parity).
|
||||
* H1 (``# <non-space>…``) opens a chapter; ``##``…``######`` and ``# ``
|
||||
(no ``\\S`` title) are body.
|
||||
* Each chapter body resets the active voice to ``default_voice``.
|
||||
* A span is dropped iff its text is empty AND pause_ms_after == 0.
|
||||
* Chapters with no surviving spans are dropped; untitled bodies are
|
||||
numbered ``Chapter {kept_so_far + 1}`` (post-drop numbering).
|
||||
"""
|
||||
text = _normalize(text)
|
||||
matches = list(_HEADING_RE.finditer(text))
|
||||
if not matches:
|
||||
raw = [(None, text)]
|
||||
else:
|
||||
raw = []
|
||||
intro = text[:matches[0].start()]
|
||||
if intro.strip():
|
||||
raw.append((None, intro))
|
||||
for i, m in enumerate(matches):
|
||||
end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
|
||||
raw.append((m.group(1).strip(), text[m.end():end]))
|
||||
|
||||
chapters: list[dict] = []
|
||||
for title, body in raw:
|
||||
spans = _parse_chapter_body(body, default_voice=default_voice,
|
||||
default_speed=default_speed)
|
||||
if not spans:
|
||||
continue
|
||||
chapters.append({"title": title or f"Chapter {len(chapters) + 1}",
|
||||
"spans": spans})
|
||||
return chapters
|
||||
@@ -1,389 +0,0 @@
|
||||
"""Shared long-form render core (Stories + Audiobook convergence).
|
||||
|
||||
Both the Audiobook tab and the Stories Editor produce the *same* artifact: a
|
||||
chapter-marked audio file built from chapter WAVs. This module owns the pure,
|
||||
engine-agnostic ffmpeg/metadata builders for that mux so neither feature has to
|
||||
reimplement it:
|
||||
|
||||
* ``build_ffmetadata`` — FFMETADATA1 doc: an optional ``[global]`` tag block
|
||||
(title / author / narrator / year / genre / description) followed by one
|
||||
``[CHAPTER]`` per (title, duration_ms).
|
||||
* ``build_concat_list`` — ffmpeg concat-demuxer list of chapter WAVs.
|
||||
* ``build_loudnorm_filter`` — an ``-af loudnorm=…`` string for an ACX /
|
||||
podcast loudness preset (off by default — opt-in, so the default-behavior
|
||||
stays platform-identical).
|
||||
* ``validate_cover_image`` — guard a cover path (type + size) before it
|
||||
reaches ffmpeg.
|
||||
* ``build_render_cmd`` — pure argv for the mux: chapter WAVs + FFMETADATA
|
||||
(+ optional cover art, loudness filter), output as ``m4b`` or ``mp3``.
|
||||
* ``chapter_cache_key`` — deterministic content hash so a re-run reuses
|
||||
already-rendered chapters (resume) and re-renders only what changed.
|
||||
|
||||
Every function here is pure (string/argv in, string/argv out) so it's unit
|
||||
tested without ffmpeg, torch, or a GPU. The impure ffmpeg run lives in the
|
||||
caller (the audiobook router today; the stories job tomorrow).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Iterable, Optional
|
||||
|
||||
_BITRATE_RE = re.compile(r"^\d{2,3}k$")
|
||||
#: Default ceiling for the content-addressed chapter cache. Above this, the
|
||||
#: oldest cached chapter WAVs are evicted (LRU by mtime). Override via
|
||||
#: OMNIVOICE_LONGFORM_CACHE_MAX_GB.
|
||||
_CACHE_MAX_BYTES = int(float(os.environ.get("OMNIVOICE_LONGFORM_CACHE_MAX_GB", "2")) * 1024 ** 3)
|
||||
_COVER_EXTS = {".jpg", ".jpeg", ".png"}
|
||||
_COVER_MAX_BYTES = 8 * 1024 * 1024 # 8 MB — a book cover, not a payload
|
||||
|
||||
#: Our metadata field → FFMETADATA tag key. Order is stable for deterministic
|
||||
#: output (tested). ``author`` maps to ``artist`` and ``narrator`` to
|
||||
#: ``composer`` — the tags audiobook players (Apple Books, Audible) read for
|
||||
#: those roles.
|
||||
_GLOBAL_TAG_KEYS: list[tuple[str, str]] = [
|
||||
("title", "title"),
|
||||
("author", "artist"),
|
||||
("album", "album"),
|
||||
("narrator", "composer"),
|
||||
("year", "date"),
|
||||
("genre", "genre"),
|
||||
("description", "comment"),
|
||||
]
|
||||
|
||||
|
||||
def _escape_meta(value: str) -> str:
|
||||
"""Escape an FFMETADATA value (``=``, ``;``, ``#``, ``\\``, newline)."""
|
||||
return re.sub(r"([=;#\\\n])", r"\\\1", value or "")
|
||||
|
||||
|
||||
def prune_cache_dir(cache_dir: str, max_bytes: int = _CACHE_MAX_BYTES) -> tuple[int, int]:
|
||||
"""Evict the oldest files in ``cache_dir`` until the total size is within
|
||||
``max_bytes`` (LRU by mtime). The content-addressed chapter cache otherwise
|
||||
grows without bound — uncompressed WAVs accumulate across every render.
|
||||
|
||||
Best-effort: returns ``(remaining_bytes, removed_count)`` and never raises
|
||||
(a missing dir / unstattable file is just skipped). Call it *before* writing
|
||||
a job's chapters so the fresh ones are never the eviction target.
|
||||
"""
|
||||
try:
|
||||
names = os.listdir(cache_dir)
|
||||
except OSError:
|
||||
return (0, 0)
|
||||
entries: list[tuple[float, int, str]] = []
|
||||
total = 0
|
||||
for name in names:
|
||||
p = os.path.join(cache_dir, name)
|
||||
try:
|
||||
if not os.path.isfile(p):
|
||||
continue
|
||||
size = os.path.getsize(p)
|
||||
mtime = os.path.getmtime(p)
|
||||
except OSError:
|
||||
continue
|
||||
entries.append((mtime, size, p))
|
||||
total += size
|
||||
if total <= max_bytes:
|
||||
return (total, 0)
|
||||
entries.sort() # oldest first
|
||||
removed = 0
|
||||
for _mtime, size, p in entries:
|
||||
if total <= max_bytes:
|
||||
break
|
||||
try:
|
||||
os.remove(p)
|
||||
total -= size
|
||||
removed += 1
|
||||
except OSError:
|
||||
continue
|
||||
return (total, removed)
|
||||
|
||||
|
||||
# ── Chapter cache key (resume) ──────────────────────────────────────────────
|
||||
|
||||
def chapter_cache_key(
|
||||
spans: Iterable[tuple],
|
||||
*,
|
||||
sample_rate: int,
|
||||
engine_id: str,
|
||||
voice_sig: Optional[dict] = None,
|
||||
) -> str:
|
||||
"""Deterministic content hash for a chapter's rendered audio.
|
||||
|
||||
``spans`` is an ordered list of ``(voice_id, text, pause_ms_after[, speed])``
|
||||
(speed optional, defaults to None). Same inputs → same key → reuse the
|
||||
cached chapter WAV on a re-run (resume); any change (text, voice, order,
|
||||
pauses, speed, sample rate, engine, or a voice's resolved signature) → new
|
||||
key → re-render. ``voice_sig`` maps each voice id to a stable signature
|
||||
string (e.g. ``ref_audio|instruct|seed``) so editing the underlying profile
|
||||
also invalidates the cache.
|
||||
"""
|
||||
payload = {
|
||||
"sr": int(sample_rate),
|
||||
"engine": engine_id or "",
|
||||
"spans": [[s[0], s[1], int(s[2]), (s[3] if len(s) > 3 else None)] for s in spans],
|
||||
"voices": {k: voice_sig[k] for k in sorted(voice_sig)} if voice_sig else {},
|
||||
}
|
||||
raw = json.dumps(payload, sort_keys=True, ensure_ascii=False)
|
||||
# Content-addressing only — not a security digest. usedforsecurity=False
|
||||
# keeps bandit's B324 (weak-hash) check quiet.
|
||||
return hashlib.sha1(raw.encode("utf-8"), usedforsecurity=False).hexdigest()[:20]
|
||||
|
||||
|
||||
# ── Loudness normalization ──────────────────────────────────────────────────
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LoudnessPreset:
|
||||
"""A loudnorm target. ``i`` = integrated LUFS, ``tp`` = true-peak ceiling
|
||||
(dBTP), ``lra`` = loudness range."""
|
||||
key: str
|
||||
i: float
|
||||
tp: float
|
||||
lra: float
|
||||
|
||||
|
||||
#: ``acx`` targets Audible/ACX submission (≈ -19 LUFS integrated, ≤ -3 dBTP
|
||||
#: peak — inside ACX's -23…-18 dB RMS / -3 dB peak window). ``podcast`` targets
|
||||
#: the -16 LUFS streaming norm.
|
||||
LOUDNESS_PRESETS: dict[str, LoudnessPreset] = {
|
||||
"acx": LoudnessPreset("acx", -19.0, -3.0, 11.0),
|
||||
"podcast": LoudnessPreset("podcast", -16.0, -1.5, 11.0),
|
||||
}
|
||||
|
||||
|
||||
def build_loudnorm_filter(preset: Optional[str]) -> Optional[str]:
|
||||
"""Return an ``-af`` loudnorm filter string for ``preset``, or ``None`` for
|
||||
off / unknown (single-pass; two-pass measure→apply is a runner enhancement).
|
||||
"""
|
||||
if not preset:
|
||||
return None
|
||||
p = LOUDNESS_PRESETS.get(preset.lower())
|
||||
if p is None: # "off", "none", or anything unrecognized → no filter
|
||||
return None
|
||||
return f"loudnorm=I={p.i}:TP={p.tp}:LRA={p.lra}"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MeasuredLoudness:
|
||||
"""The five loudnorm measure-pass values (FFmpeg JSON keys), all finite
|
||||
floats. Fed back into the second (apply) pass as ``measured_*`` + ``offset``."""
|
||||
input_i: float
|
||||
input_tp: float
|
||||
input_lra: float
|
||||
input_thresh: float
|
||||
target_offset: float
|
||||
|
||||
|
||||
def build_loudnorm_measure_filter(preset: Optional[str]) -> Optional[str]:
|
||||
"""First-pass loudnorm filter (``print_format=json``) for ``preset``, or
|
||||
``None`` for off/unknown — mirrors :func:`build_loudnorm_filter`'s lookup
|
||||
(no whitespace stripping) so the same values count as 'no filter'."""
|
||||
if not preset:
|
||||
return None
|
||||
p = LOUDNESS_PRESETS.get(preset.lower())
|
||||
if p is None:
|
||||
return None
|
||||
return f"loudnorm=I={p.i}:TP={p.tp}:LRA={p.lra}:print_format=json"
|
||||
|
||||
|
||||
def parse_loudnorm_measure(stderr_text: Optional[str]) -> Optional[MeasuredLoudness]:
|
||||
"""Extract the loudnorm measure JSON from ffmpeg stderr → MeasuredLoudness,
|
||||
or ``None`` on ANY failure (caller falls back to single-pass). FFmpeg prints
|
||||
the JSON object amid other non-JSON lines (and possibly a config dump block),
|
||||
so we take the LAST balanced ``{...}`` via a linear brace-depth scan — no
|
||||
regex (CodeQL-safe), O(n), no backtracking — then json.loads + coerce/validate
|
||||
the five required keys to finite floats."""
|
||||
if not stderr_text:
|
||||
return None
|
||||
# Find the last balanced top-level {...} block via a single linear scan.
|
||||
start = -1
|
||||
depth = 0
|
||||
block = None
|
||||
for i, ch in enumerate(stderr_text):
|
||||
if ch == "{":
|
||||
if depth == 0:
|
||||
start = i
|
||||
depth += 1
|
||||
elif ch == "}":
|
||||
if depth > 0:
|
||||
depth -= 1
|
||||
if depth == 0 and start != -1:
|
||||
block = stderr_text[start:i + 1] # keep scanning → last wins
|
||||
if block is None:
|
||||
return None
|
||||
try:
|
||||
obj = json.loads(block)
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
return None
|
||||
if not isinstance(obj, dict):
|
||||
return None
|
||||
keys = ("input_i", "input_tp", "input_lra", "input_thresh", "target_offset")
|
||||
vals = {}
|
||||
for k in keys:
|
||||
if k not in obj:
|
||||
return None
|
||||
try:
|
||||
v = float(obj[k])
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if not math.isfinite(v): # rejects "-inf"/"inf"/"nan" (silent clip)
|
||||
return None
|
||||
vals[k] = v
|
||||
return MeasuredLoudness(**vals)
|
||||
|
||||
|
||||
def build_loudnorm_apply_filter(
|
||||
preset: Optional[str], measured: Optional["MeasuredLoudness"],
|
||||
) -> Optional[str]:
|
||||
"""Second-pass (apply) loudnorm filter feeding the measured values back in.
|
||||
``None`` for off/unknown preset OR when ``measured`` is None (so a caller
|
||||
that forgot to branch never emits ``measured_I=None``)."""
|
||||
if not preset or measured is None:
|
||||
return None
|
||||
p = LOUDNESS_PRESETS.get(preset.lower())
|
||||
if p is None:
|
||||
return None
|
||||
return (
|
||||
f"loudnorm=I={p.i}:TP={p.tp}:LRA={p.lra}"
|
||||
f":measured_I={measured.input_i}:measured_TP={measured.input_tp}"
|
||||
f":measured_LRA={measured.input_lra}:measured_thresh={measured.input_thresh}"
|
||||
f":offset={measured.target_offset}:linear=true:print_format=summary"
|
||||
)
|
||||
|
||||
|
||||
def build_loudnorm_measure_cmd(ffmpeg: str, concat_list_path: str, filt: str) -> list[str]:
|
||||
"""Pure argv for the measure pass: decode the concat list, run the
|
||||
print_format=json loudnorm filter, discard audio to the portable null muxer.
|
||||
Input segment is byte-identical to build_render_cmd so measured == muxed."""
|
||||
return [
|
||||
ffmpeg, "-y", "-hide_banner", "-loglevel", "info",
|
||||
"-f", "concat", "-safe", "0", "-i", str(concat_list_path),
|
||||
"-af", filt, "-f", "null", "-",
|
||||
]
|
||||
|
||||
|
||||
# ── FFMETADATA ──────────────────────────────────────────────────────────────
|
||||
|
||||
def build_ffmetadata(
|
||||
chapters: Iterable[tuple[str, int]],
|
||||
global_meta: Optional[dict] = None,
|
||||
) -> str:
|
||||
"""Build an FFMETADATA1 doc: optional global tags + one ``[CHAPTER]`` per
|
||||
``(title, duration_ms)``. START/END are cumulative millisecond offsets.
|
||||
"""
|
||||
lines = [";FFMETADATA1"]
|
||||
if global_meta:
|
||||
for field_key, meta_key in _GLOBAL_TAG_KEYS:
|
||||
val = global_meta.get(field_key)
|
||||
if val is not None and str(val).strip():
|
||||
lines.append(f"{meta_key}={_escape_meta(str(val).strip())}")
|
||||
start = 0
|
||||
for title, dur_ms in chapters:
|
||||
end = start + max(0, int(dur_ms))
|
||||
lines += [
|
||||
"[CHAPTER]",
|
||||
"TIMEBASE=1/1000",
|
||||
f"START={start}",
|
||||
f"END={end}",
|
||||
f"title={_escape_meta(title)}",
|
||||
]
|
||||
start = end
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def build_concat_list(wav_paths: Iterable[str]) -> str:
|
||||
"""Build an ffmpeg concat-demuxer list. Single quotes in paths are escaped
|
||||
the ffmpeg way (``'`` → ``'\\''``) so paths can't break the list or inject
|
||||
arguments."""
|
||||
lines = []
|
||||
for p in wav_paths:
|
||||
safe = str(p).replace("'", "'\\''")
|
||||
lines.append(f"file '{safe}'")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
# ── Cover art ───────────────────────────────────────────────────────────────
|
||||
|
||||
def validate_cover_image(path: Optional[str]) -> bool:
|
||||
"""True if ``path`` is a readable jpg/png within the size cap. Anything
|
||||
dubious (missing, wrong type, too big, unreadable) → False, and the caller
|
||||
simply omits the cover rather than failing the render."""
|
||||
if not path:
|
||||
return False
|
||||
try:
|
||||
p = Path(path)
|
||||
return (
|
||||
p.is_file()
|
||||
and p.suffix.lower() in _COVER_EXTS
|
||||
and 0 < p.stat().st_size <= _COVER_MAX_BYTES
|
||||
)
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
# ── Render command ──────────────────────────────────────────────────────────
|
||||
|
||||
def build_render_cmd(
|
||||
ffmpeg: str,
|
||||
concat_list_path: str,
|
||||
metadata_path: str,
|
||||
out_path: str,
|
||||
*,
|
||||
fmt: str = "m4b",
|
||||
bitrate: str = "128k",
|
||||
cover_path: Optional[str] = None,
|
||||
loudness: Optional[str] = None,
|
||||
measured: Optional[MeasuredLoudness] = None,
|
||||
) -> list[str]:
|
||||
"""Pure argv for muxing chapter WAVs + FFMETADATA into a tagged,
|
||||
chapter-marked audio file.
|
||||
|
||||
Inputs: 0 = concat-demuxer list of chapter WAVs, 1 = FFMETADATA (chapters +
|
||||
global tags), 2 = cover image (only when present + valid). ``fmt`` is
|
||||
``m4b`` (AAC in mp4, faststart) or ``mp3`` (libmp3lame). A loudness preset
|
||||
adds an ``-af loudnorm`` pass; an invalid/oversized cover is silently
|
||||
dropped (see :func:`validate_cover_image`).
|
||||
"""
|
||||
if not _BITRATE_RE.match(bitrate or ""):
|
||||
bitrate = "128k"
|
||||
is_mp3 = (fmt or "").lower() == "mp3"
|
||||
# Cover art is embedded for M4B only. The MP3 muxer rejects an
|
||||
# ``attached_pic`` video stream via ``-c:v copy`` (produces a corrupt file
|
||||
# across ffmpeg versions), and a reliable cross-version ID3 APIC path is
|
||||
# finicky — so for MP3 we skip the cover rather than ship a broken file.
|
||||
# M4B is the cover-bearing audiobook format anyway.
|
||||
embed_cover = validate_cover_image(cover_path) and not is_mp3
|
||||
|
||||
cmd = [
|
||||
ffmpeg, "-y", "-hide_banner", "-loglevel", "error",
|
||||
"-f", "concat", "-safe", "0", "-i", str(concat_list_path),
|
||||
"-i", str(metadata_path),
|
||||
]
|
||||
if embed_cover:
|
||||
cmd += ["-i", str(cover_path)]
|
||||
|
||||
cmd += ["-map", "0:a", "-map_metadata", "1"]
|
||||
if embed_cover:
|
||||
cmd += ["-map", "2:v", "-disposition:v", "attached_pic"]
|
||||
|
||||
# Two-pass apply when measured values are present; else single-pass. Both
|
||||
# return None for a non-preset loudness, so the `if filt:` guard below
|
||||
# gives an off-render no -af (byte-identical to today).
|
||||
filt = build_loudnorm_apply_filter(loudness, measured) if measured is not None else build_loudnorm_filter(loudness)
|
||||
if filt:
|
||||
cmd += ["-af", filt]
|
||||
|
||||
if is_mp3:
|
||||
cmd += ["-c:a", "libmp3lame", "-b:a", bitrate, "-f", "mp3", str(out_path)]
|
||||
else: # m4b — AAC in an mp4 container
|
||||
cmd += ["-c:a", "aac", "-b:a", bitrate]
|
||||
if embed_cover:
|
||||
cmd += ["-c:v", "copy"]
|
||||
cmd += ["-movflags", "+faststart", "-f", "mp4", str(out_path)]
|
||||
return cmd
|
||||
@@ -1,195 +0,0 @@
|
||||
"""Durable resume for longform (audiobook / story) renders.
|
||||
|
||||
Chapter WAVs are already content-addressed in a shared cache, so re-rendering an
|
||||
identical plan reuses what finished — the synthesis-level resume. The missing
|
||||
piece this module adds is **durability of the *plan itself***: a render that's
|
||||
interrupted (crash, app quit, power loss) leaves a `resume.json` manifest in the
|
||||
job's work dir holding the compiled plan + render params, so the job can be
|
||||
resumed later *without the user still having the original script* — which matters
|
||||
for Stories, whose plan is compiled from cast+lines and can't be retyped.
|
||||
|
||||
Pure file/JSON I/O (no torch, no model) so it's unit-testable. The router wires
|
||||
it into the SSE renderer (write on start, clear on done) and exposes
|
||||
``GET /audiobook/jobs`` (resumable) + ``POST /audiobook/resume/{job_id}``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
MANIFEST_VERSION = 1
|
||||
_MANIFEST_NAME = "resume.json"
|
||||
# Longform front doors that produce a resumable work dir (job_type → dir prefix).
|
||||
RESUMABLE_TYPES = ("audiobook", "story")
|
||||
# A job id is server-generated (uuid4 hex) — confine to a strict token so a
|
||||
# request-supplied id (the /audiobook/resume/{job_id} path param) can never
|
||||
# carry a path separator, `..`, NUL, or anything that escapes OUTPUTS_DIR
|
||||
# (CodeQL py/path-injection). Anchored, single bounded quantifier → ReDoS-safe.
|
||||
_SAFE_ID_RE = re.compile(r"^[A-Za-z0-9_-]{1,64}$")
|
||||
# Exact-match allowlist on the WHOLE work-dir name (the value actually joined
|
||||
# onto OUTPUTS_DIR). Validating the joined string itself — not just job_id — is
|
||||
# the barrier CodeQL's path-injection query recognizes (same shape as
|
||||
# audiobook._safe_cover_path's _COVER_NAME_RE). Mirrors RESUMABLE_TYPES.
|
||||
_SAFE_SEG_RE = re.compile(r"^(?:audiobook|story)_[A-Za-z0-9_-]{1,64}$")
|
||||
|
||||
|
||||
def work_dir(job_type: str, job_id: str) -> Optional[str]:
|
||||
"""The per-job work directory ``OUTPUTS_DIR/<job_type>_<job_id>``, resolved
|
||||
strictly inside OUTPUTS_DIR. Returns None for an unknown ``job_type``, an
|
||||
id that isn't a bare safe token, or any path that escapes OUTPUTS_DIR — so a
|
||||
crafted ``job_id`` can never reach a foreign path (py/path-injection-safe)."""
|
||||
if job_type not in RESUMABLE_TYPES or not _SAFE_ID_RE.match(job_id or ""):
|
||||
return None
|
||||
# os.path.basename strips any directory component (the sanitizer CodeQL's
|
||||
# path-injection query recognizes — same as audiobook._safe_cover_path), and
|
||||
# the exact-match allowlist on the result is a second barrier: the joined
|
||||
# value is provably a single bare dir name of the expected shape.
|
||||
seg = os.path.basename(f"{job_type}_{job_id}")
|
||||
if not _SAFE_SEG_RE.match(seg):
|
||||
return None
|
||||
from core.config import OUTPUTS_DIR
|
||||
root = os.path.realpath(OUTPUTS_DIR)
|
||||
path = os.path.realpath(os.path.join(root, seg))
|
||||
# commonpath containment — the form static analysis recognizes (belt over
|
||||
# the regex). Raises ValueError on mixed drives (Windows) → reject.
|
||||
try:
|
||||
if os.path.commonpath([path, root]) != root:
|
||||
return None
|
||||
except ValueError:
|
||||
return None
|
||||
return path
|
||||
|
||||
|
||||
def manifest_path(job_type: str, job_id: str) -> Optional[str]:
|
||||
d = work_dir(job_type, job_id)
|
||||
return os.path.join(d, _MANIFEST_NAME) if d else None
|
||||
|
||||
|
||||
def build_manifest(
|
||||
*,
|
||||
job_id: str,
|
||||
job_type: str,
|
||||
plan_chapters: list[dict],
|
||||
params: dict,
|
||||
title: str = "",
|
||||
) -> dict:
|
||||
"""Assemble the manifest dict. ``plan_chapters`` is the canonical span-plan
|
||||
(``[{title, spans:[{voice_id,text,pause_ms_after,speed}]}]``); ``params`` is
|
||||
the render kwargs (default_voice / fmt / bitrate / loudness / cover_path /
|
||||
metadata / lexicon). Pure — no I/O."""
|
||||
return {
|
||||
"version": MANIFEST_VERSION,
|
||||
"job_id": job_id,
|
||||
"job_type": job_type,
|
||||
"title": title or "",
|
||||
"total_chapters": len(plan_chapters),
|
||||
"params": params,
|
||||
"plan": plan_chapters,
|
||||
}
|
||||
|
||||
|
||||
def write_manifest(manifest: dict) -> Optional[str]:
|
||||
"""Persist the manifest to the job work dir. Best-effort — resume is an
|
||||
enhancement, never block the render — returns the path or None on failure /
|
||||
unsafe id."""
|
||||
path = manifest_path(manifest.get("job_type", ""), manifest.get("job_id", ""))
|
||||
if path is None:
|
||||
return None
|
||||
try:
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
tmp = path + ".tmp"
|
||||
with open(tmp, "w", encoding="utf-8") as f:
|
||||
json.dump(manifest, f, ensure_ascii=False)
|
||||
os.replace(tmp, path) # atomic — a half-written manifest never resumes
|
||||
return path
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def read_manifest(job_type: str, job_id: str) -> Optional[dict]:
|
||||
"""Load a job's resume manifest, or None if absent/unreadable/foreign-shape/
|
||||
unsafe id."""
|
||||
path = manifest_path(job_type, job_id)
|
||||
if path is None:
|
||||
return None
|
||||
try:
|
||||
with open(path, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
if not isinstance(data, dict) or data.get("version") != MANIFEST_VERSION:
|
||||
return None
|
||||
if not isinstance(data.get("plan"), list):
|
||||
return None
|
||||
return data
|
||||
|
||||
|
||||
def clear_manifest(job_type: str, job_id: str) -> None:
|
||||
"""Remove the manifest once a job completes (no resume needed). Best-effort."""
|
||||
path = manifest_path(job_type, job_id)
|
||||
if path is None:
|
||||
return
|
||||
try:
|
||||
os.remove(path)
|
||||
except OSError:
|
||||
return # best-effort; already gone or unwritable
|
||||
|
||||
|
||||
def has_manifest(job_type: str, job_id: str) -> bool:
|
||||
path = manifest_path(job_type, job_id)
|
||||
return bool(path) and os.path.isfile(path)
|
||||
|
||||
|
||||
def scan_resumable() -> list[dict]:
|
||||
"""Enumerate resumable jobs by scanning OUTPUTS_DIR for ``<type>_<id>`` work
|
||||
dirs that hold a ``resume.json``.
|
||||
|
||||
Returns ``[{"job_type", "job_id", "manifest_path"}, …]`` where **every path
|
||||
component is sourced from os.listdir** (the trusted filesystem), never from
|
||||
request input. The ``manifest_path`` is built here from the listed dir name,
|
||||
so callers can read it directly without re-deriving a path from a
|
||||
request-supplied id (CodeQL py/path-injection-safe — no tainted value ever
|
||||
reaches a file operation)."""
|
||||
from core.config import OUTPUTS_DIR
|
||||
out: list[dict] = []
|
||||
try:
|
||||
root = os.path.realpath(OUTPUTS_DIR)
|
||||
names = os.listdir(root)
|
||||
except OSError:
|
||||
return out
|
||||
for name in names:
|
||||
for jt in RESUMABLE_TYPES:
|
||||
prefix = f"{jt}_"
|
||||
mpath = os.path.join(root, name, _MANIFEST_NAME)
|
||||
if name.startswith(prefix) and os.path.isfile(mpath):
|
||||
out.append({"job_type": jt, "job_id": name[len(prefix):],
|
||||
"manifest_path": mpath})
|
||||
return out
|
||||
|
||||
|
||||
def discard_manifest_file(path: str) -> None:
|
||||
"""Remove a manifest by a path obtained via :func:`scan_resumable` (trusted,
|
||||
os.listdir-derived). Best-effort — used to retire an interrupted job once
|
||||
it's been resumed under a fresh id."""
|
||||
try:
|
||||
os.remove(path)
|
||||
except OSError:
|
||||
return
|
||||
|
||||
|
||||
def load_manifest_file(path: str) -> Optional[dict]:
|
||||
"""Read + validate a manifest from a path obtained via :func:`scan_resumable`
|
||||
(a trusted, os.listdir-derived path — NOT a request-derived one). Returns
|
||||
None on missing/unreadable/foreign-shape."""
|
||||
try:
|
||||
with open(path, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
if not isinstance(data, dict) or data.get("version") != MANIFEST_VERSION:
|
||||
return None
|
||||
if not isinstance(data.get("plan"), list):
|
||||
return None
|
||||
return data
|
||||
@@ -1,68 +0,0 @@
|
||||
"""Two-pass loudnorm measure orchestrator (#28).
|
||||
|
||||
The impure half of the two-pass ACX/podcast master: run ffmpeg's measure pass
|
||||
over the concatenated chapters and parse the printed loudnorm JSON. The pure
|
||||
builders/parser live in :mod:`services.longform_render`; this only drives ffmpeg.
|
||||
|
||||
Contract: **never raises.** Every failure (skip / non-zero rc / timeout / spawn
|
||||
error / empty or unparseable stderr / silent program) is caught, logged at
|
||||
WARNING, and converted to ``None`` so the caller falls back to single-pass. A
|
||||
slow or broken measure must degrade the master, never abort the render.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from services.longform_render import (
|
||||
MeasuredLoudness,
|
||||
build_loudnorm_measure_cmd,
|
||||
build_loudnorm_measure_filter,
|
||||
parse_loudnorm_measure,
|
||||
)
|
||||
|
||||
logger = logging.getLogger("omnivoice.loudness")
|
||||
|
||||
|
||||
async def measure_loudness(
|
||||
ffmpeg: str,
|
||||
concat_list_path: str,
|
||||
preset: str,
|
||||
*,
|
||||
job_id: str,
|
||||
) -> Optional[MeasuredLoudness]:
|
||||
"""Measure the concatenated program's loudness for ``preset`` (acx/podcast),
|
||||
or ``None`` for off/unknown or on ANY failure (→ single-pass fallback).
|
||||
|
||||
Only the ffmpeg rc and a short static message are logged — never the raw
|
||||
stderr (it can carry the concat path under OUTPUTS_DIR), keeping the log
|
||||
local-first / path-safe.
|
||||
"""
|
||||
filt = build_loudnorm_measure_filter(preset)
|
||||
if filt is None:
|
||||
return None # off / unknown — a normal skip, not an error (no log)
|
||||
cmd = build_loudnorm_measure_cmd(ffmpeg, concat_list_path, filt)
|
||||
from services.ffmpeg_utils import run_ffmpeg # lazy → patchable at source
|
||||
try:
|
||||
# asyncio.TimeoutError is a subclass of Exception (Py≥3.11) — caught
|
||||
# here so a slow measure degrades to single-pass instead of killing the
|
||||
# whole render via the caller's outer except.
|
||||
rc, _out, err = await run_ffmpeg(cmd, capture=True, job_id=job_id)
|
||||
except Exception as exc:
|
||||
logger.warning("loudness measure pass did not run (%s) — single-pass fallback",
|
||||
type(exc).__name__)
|
||||
return None
|
||||
if rc != 0:
|
||||
logger.warning("loudness measure pass exited rc=%s — single-pass fallback", rc)
|
||||
return None
|
||||
try:
|
||||
stderr_text = err.decode("utf-8", "replace") if isinstance(err, (bytes, bytearray)) else (err or "")
|
||||
except Exception:
|
||||
return None
|
||||
measured = parse_loudnorm_measure(stderr_text)
|
||||
if measured is None:
|
||||
logger.warning("loudness measure output unparseable — single-pass fallback")
|
||||
return measured
|
||||
|
||||
|
||||
__all__ = ["measure_loudness"]
|
||||
@@ -1,126 +0,0 @@
|
||||
"""Per-agent MCP voice bindings (Wave 2.2 / Spec 2).
|
||||
|
||||
An MCP client identifies itself with the ``X-OmniVoice-Client-Id`` header.
|
||||
Each client can be bound to a default voice profile + engine so different
|
||||
agents speak in different voices ("Claude Code in Morgan, Cursor in
|
||||
Scarlett"). Pure data layer over the ``mcp_client_bindings`` table — the
|
||||
FastMCP tools call :func:`resolve_voice`; the Settings UI calls the CRUD
|
||||
helpers via the REST router.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
from core.db import db_conn
|
||||
|
||||
|
||||
def list_bindings() -> list[dict]:
|
||||
with db_conn() as conn:
|
||||
# SQLite sorts NULL as smallest, so DESC naturally puts never-seen
|
||||
# bindings after recently-active ones.
|
||||
rows = conn.execute(
|
||||
"SELECT * FROM mcp_client_bindings ORDER BY last_seen_at DESC, created_at DESC"
|
||||
).fetchall()
|
||||
return [dict(r) for r in rows]
|
||||
|
||||
|
||||
def get_binding(client_id: str) -> Optional[dict]:
|
||||
with db_conn() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT * FROM mcp_client_bindings WHERE client_id=?", (client_id,)
|
||||
).fetchone()
|
||||
return dict(row) if row else None
|
||||
|
||||
|
||||
def upsert_binding(
|
||||
client_id: str,
|
||||
*,
|
||||
label: Optional[str] = None,
|
||||
profile_id: Optional[str] = None,
|
||||
default_engine: Optional[str] = None,
|
||||
) -> dict:
|
||||
"""Create or update a binding. Fields left as None on an existing row are
|
||||
preserved; on a new row they default to empty/null."""
|
||||
if not client_id or not client_id.strip():
|
||||
raise ValueError("client_id must be non-empty")
|
||||
cid = client_id.strip()
|
||||
existing = get_binding(cid)
|
||||
now = time.time()
|
||||
if existing:
|
||||
merged = {
|
||||
"label": existing["label"] if label is None else label,
|
||||
"profile_id": existing["profile_id"] if profile_id is None else (profile_id or None),
|
||||
"default_engine": existing["default_engine"] if default_engine is None else (default_engine or None),
|
||||
}
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"UPDATE mcp_client_bindings SET label=?, profile_id=?, default_engine=? WHERE client_id=?",
|
||||
(merged["label"], merged["profile_id"], merged["default_engine"], cid),
|
||||
)
|
||||
else:
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO mcp_client_bindings "
|
||||
"(client_id, label, profile_id, default_engine, last_seen_at, created_at) "
|
||||
"VALUES (?, ?, ?, ?, NULL, ?)",
|
||||
(cid, label or "", profile_id or None, default_engine or None, now),
|
||||
)
|
||||
return get_binding(cid)
|
||||
|
||||
|
||||
def delete_binding(client_id: str) -> bool:
|
||||
with db_conn() as conn:
|
||||
cur = conn.execute("DELETE FROM mcp_client_bindings WHERE client_id=?", (client_id,))
|
||||
return cur.rowcount > 0
|
||||
|
||||
|
||||
def touch_last_seen(client_id: str) -> None:
|
||||
"""Best-effort 'last heard from this agent' stamp. Never raises — it's
|
||||
telemetry for the Settings list, not load-bearing."""
|
||||
if not client_id:
|
||||
return
|
||||
try:
|
||||
with db_conn() as conn:
|
||||
conn.execute(
|
||||
"UPDATE mcp_client_bindings SET last_seen_at=? WHERE client_id=?",
|
||||
(time.time(), client_id),
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def _global_default_profile() -> Optional[str]:
|
||||
"""The fallback voice when a client has no binding. Reads the same
|
||||
pref the Settings 'default playback voice' would set; None if unset."""
|
||||
try:
|
||||
from core import prefs
|
||||
return prefs.get("mcp_default_profile_id") or None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def resolve_voice(client_id: Optional[str], explicit_profile_id: Optional[str]) -> dict:
|
||||
"""Resolve which voice an MCP speak call should use.
|
||||
|
||||
Precedence (Spec 2): explicit tool arg → the client's binding →
|
||||
the global default → nothing (caller decides / errors with a hint).
|
||||
|
||||
Returns ``{profile_id, default_engine, source}`` where ``source`` is one
|
||||
of ``explicit`` | ``binding`` | ``global`` | ``none`` for diagnostics.
|
||||
"""
|
||||
if explicit_profile_id:
|
||||
return {"profile_id": explicit_profile_id, "default_engine": None, "source": "explicit"}
|
||||
if client_id:
|
||||
binding = get_binding(client_id)
|
||||
if binding and binding.get("profile_id"):
|
||||
return {
|
||||
"profile_id": binding["profile_id"],
|
||||
"default_engine": binding.get("default_engine"),
|
||||
"source": "binding",
|
||||
}
|
||||
g = _global_default_profile()
|
||||
if g:
|
||||
return {"profile_id": g, "default_engine": None, "source": "global"}
|
||||
return {"profile_id": None, "default_engine": None, "source": "none"}
|
||||
@@ -1,158 +0,0 @@
|
||||
"""Single lifecycle surface for loaded models (MM2-04).
|
||||
|
||||
Before this, ``GET /model/loaded`` and ``POST /model/unload`` each hand-rolled
|
||||
enumeration/dispatch across three worlds — the in-process TTS+ASR model
|
||||
(``model_manager``), the diarization pipeline, and subprocess sidecars
|
||||
(``subprocess_backend``). This module owns that logic so the routers are thin
|
||||
delegations and there's one place to reason about model lifecycle.
|
||||
|
||||
Response shapes are preserved exactly — the frontend (hooks.ts model status +
|
||||
the flush dropdown) depends on ``{models, count}`` and
|
||||
``{unloaded, success, ...}``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
import services.model_manager as mm
|
||||
from services.model_manager import get_best_device
|
||||
|
||||
|
||||
def _tts_vram_mb() -> float:
|
||||
"""Best-effort allocated VRAM for the in-process model. Accurate on CUDA,
|
||||
sparse on MPS, 0 elsewhere — degrade gracefully, never raise."""
|
||||
try:
|
||||
torch = mm._lazy_torch()
|
||||
if torch.cuda.is_available():
|
||||
return torch.cuda.memory_allocated() / (1024 ** 2)
|
||||
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
driver = getattr(torch.mps, "driver_allocated_memory", None)
|
||||
if driver:
|
||||
return driver() / (1024 ** 2)
|
||||
except Exception:
|
||||
pass
|
||||
return 0.0
|
||||
|
||||
|
||||
def _asr_device() -> str:
|
||||
"""Where the ASR pipe actually lives, rather than a hardcoded 'cpu'."""
|
||||
pipe = getattr(mm.model, "_asr_pipe", None)
|
||||
for attr in ("device",):
|
||||
dev = getattr(pipe, attr, None)
|
||||
if dev is not None:
|
||||
return str(dev)
|
||||
return "cpu"
|
||||
|
||||
|
||||
def list_loaded() -> dict:
|
||||
"""Enumerate every currently-loaded model. Shape: ``{"models": [...],
|
||||
"count": n}`` with per-model id/name/checkpoint/device/vram_mb/unloadable
|
||||
(+ optional ``note``)."""
|
||||
models: list[dict] = []
|
||||
|
||||
# 1. In-process TTS model (OmniVoice)
|
||||
if mm.model is not None:
|
||||
try:
|
||||
device = str(next(mm.model.parameters()).device) if hasattr(mm.model, "parameters") else get_best_device()
|
||||
except Exception:
|
||||
device = get_best_device()
|
||||
models.append({
|
||||
"id": "tts",
|
||||
"name": "OmniVoice TTS",
|
||||
"checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
|
||||
"device": device,
|
||||
"vram_mb": round(_tts_vram_mb(), 1),
|
||||
"unloadable": True,
|
||||
})
|
||||
|
||||
# 2. ASR (WhisperX) — co-loaded with and released alongside the TTS model.
|
||||
# Honest reporting (MM2-03): the device is read from the pipe, and the
|
||||
# dead "unload" button is explained by a note rather than left silent.
|
||||
if mm.model is not None and getattr(mm.model, "_asr_pipe", None) is not None:
|
||||
models.append({
|
||||
"id": "asr",
|
||||
"name": "WhisperX ASR",
|
||||
"checkpoint": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
|
||||
"device": _asr_device(),
|
||||
"vram_mb": 0,
|
||||
"unloadable": False,
|
||||
"note": "released with the TTS model",
|
||||
})
|
||||
|
||||
# 3. Diarization pipeline
|
||||
if mm._diar_pipeline is not None:
|
||||
models.append({
|
||||
"id": "diarization",
|
||||
"name": "Pyannote Diarization",
|
||||
"checkpoint": "pyannote/speaker-diarization-3.1",
|
||||
"device": get_best_device(),
|
||||
"vram_mb": 0,
|
||||
"unloadable": True,
|
||||
})
|
||||
|
||||
# 4. Subprocess engine sidecars — each holds a process (and on GPU, VRAM)
|
||||
# until idle-reaped. VRAM is reported by the child itself when available
|
||||
# (MM2-08); 0 means CPU-only or not-yet-measured. Enumeration must never
|
||||
# break the panel.
|
||||
try:
|
||||
from services.subprocess_backend import list_live_sidecars
|
||||
for s in list_live_sidecars():
|
||||
models.append({
|
||||
"id": f"sidecar:{s['id']}",
|
||||
"name": f"{s['id']} (sidecar)",
|
||||
"checkpoint": s["id"],
|
||||
"device": get_best_device(),
|
||||
"vram_mb": round(float(s.get("vram_mb") or 0), 1),
|
||||
"unloadable": True,
|
||||
})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {"models": models, "count": len(models)}
|
||||
|
||||
|
||||
async def unload(model_id: str) -> dict:
|
||||
"""Unload one model by id. Preserves the original per-id response shapes.
|
||||
|
||||
``tts`` | ``diarization`` | ``sidecar:<id>`` | ``sidecars``. Raises
|
||||
ValueError for an unknown id (router maps to HTTP 400)."""
|
||||
if model_id == "sidecars" or model_id.startswith("sidecar:"):
|
||||
from services.subprocess_backend import unload_all_sidecars, unload_sidecar
|
||||
n = unload_all_sidecars() if model_id == "sidecars" else unload_sidecar(model_id.split(":", 1)[1])
|
||||
return {"unloaded": model_id, "success": n > 0, "count": n,
|
||||
**({} if n > 0 else {"reason": "not running or busy"})}
|
||||
|
||||
if model_id == "tts":
|
||||
async with mm._model_lock:
|
||||
if mm.model is not None:
|
||||
mm.model = None
|
||||
mm.free_vram()
|
||||
return {"unloaded": "tts", "success": True}
|
||||
return {"unloaded": "tts", "success": False, "reason": "not loaded"}
|
||||
|
||||
if model_id == "diarization":
|
||||
if mm._diar_pipeline is not None:
|
||||
mm._diar_pipeline = None
|
||||
mm.free_vram()
|
||||
return {"unloaded": "diarization", "success": True}
|
||||
return {"unloaded": "diarization", "success": False, "reason": "not loaded"}
|
||||
|
||||
raise ValueError(f"Unknown model id: {model_id}")
|
||||
|
||||
|
||||
async def unload_all() -> dict:
|
||||
"""Release every releasable model — in-process TTS + diarization + all
|
||||
sidecars. Convenience for app shutdown / a global flush."""
|
||||
results = {}
|
||||
for mid in ("tts", "diarization", "sidecars"):
|
||||
try:
|
||||
results[mid] = await unload(mid)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
results[mid] = {"unloaded": mid, "success": False, "reason": str(exc)}
|
||||
return {"unloaded_all": True, "results": results}
|
||||
|
||||
|
||||
def free_vram() -> None:
|
||||
"""One import surface for callers that just want to drop GPU caches."""
|
||||
mm.free_vram()
|
||||
@@ -2,7 +2,6 @@ import os
|
||||
import time
|
||||
import asyncio
|
||||
import logging
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
# ── Lazy imports ─────────────────────────────────────────────────────
|
||||
@@ -111,8 +110,7 @@ def __getattr__(name: str):
|
||||
model = None # type: ignore
|
||||
_model_lock = asyncio.Lock()
|
||||
_last_used = time.time()
|
||||
# Idle timeout is resolved per-tick in _resolve_idle_timeout() (MM2-05) from
|
||||
# prefs/env/core.config — no module-level duplicate of IDLE_TIMEOUT_SECONDS.
|
||||
_IDLE_TIMEOUT_SECONDS = IDLE_TIMEOUT_SECONDS
|
||||
|
||||
# ── Loading sub-stage tracker ────────────────────────────────────────
|
||||
# Updated by _load_model_sync() so get_model_status() can report
|
||||
@@ -197,23 +195,12 @@ def get_best_device():
|
||||
"""Detect the best available compute device.
|
||||
|
||||
Priority: CUDA/ROCm > Intel XPU > DirectML > MPS > CPU
|
||||
|
||||
The *family* decision delegates to ``core.device_caps.detect_host_caps()``
|
||||
(the single source of truth) so the probe and this loader can never
|
||||
disagree. This function keeps the side-effects the probe deliberately
|
||||
avoids: the ROCm ``HSA_OVERRIDE_GFX_VERSION`` env override and the
|
||||
DirectML device-string return (DirectML is not a torch device family, so
|
||||
the probe reports it as ``cpu`` — we still resolve the real device string
|
||||
here for Windows DirectML users). The string contract is unchanged:
|
||||
``"cuda"`` / ``"xpu"`` / a DirectML device string / ``"mps"`` / ``"cpu"``.
|
||||
"""
|
||||
from core.device_caps import detect_host_caps
|
||||
|
||||
torch = _lazy_torch()
|
||||
family = detect_host_caps().family
|
||||
|
||||
# ── NVIDIA CUDA or AMD ROCm (both present through torch.cuda) ─────
|
||||
if family in ("cuda", "rocm"):
|
||||
# ── NVIDIA CUDA or AMD ROCm ──────────────────────────────────────
|
||||
# ROCm-enabled PyTorch reports through torch.cuda, so this covers both.
|
||||
if torch.cuda.is_available():
|
||||
_configure_rocm_if_needed(torch)
|
||||
compatible, warning = check_device_compatibility()
|
||||
if not compatible:
|
||||
@@ -221,23 +208,15 @@ def get_best_device():
|
||||
return "cuda"
|
||||
|
||||
# ── Intel Arc / discrete GPU via IPEX ────────────────────────────
|
||||
if family == "xpu":
|
||||
try:
|
||||
try:
|
||||
import intel_extension_for_pytorch # noqa: F401
|
||||
if hasattr(torch, "xpu") and torch.xpu.is_available():
|
||||
logger.info("Using Intel XPU device: %s", torch.xpu.get_device_name(0))
|
||||
except Exception:
|
||||
logger.info("Using Intel XPU device")
|
||||
return "xpu"
|
||||
return "xpu"
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# ── Apple Silicon MPS ────────────────────────────────────────────
|
||||
# Checked BEFORE DirectML to mirror the probe's family-priority order
|
||||
# (cuda > rocm > xpu > mps; DirectML is not a torch family) so the loader
|
||||
# and detect_host_caps() never disagree on a host that somehow exposes both.
|
||||
if family == "mps":
|
||||
return "mps"
|
||||
|
||||
# ── DirectML — universal Windows GPU (probe reports this as "cpu") ─
|
||||
# Reached only when no torch family was detected (family == "cpu"), which is
|
||||
# exactly the DirectML case — the probe classifies DirectML hosts as cpu.
|
||||
# ── DirectML — universal Windows GPU (AMD, Intel, NVIDIA fallback)
|
||||
try:
|
||||
import torch_directml
|
||||
if torch_directml.device_count() > 0:
|
||||
@@ -246,184 +225,12 @@ def get_best_device():
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# ── Apple Silicon MPS ────────────────────────────────────────────
|
||||
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
|
||||
return "mps"
|
||||
|
||||
return "cpu"
|
||||
|
||||
_COMPILE_ERR_MODULE_PREFIXES = ("torch._dynamo", "torch._inductor", "torch.fx", "triton")
|
||||
_COMPILE_ERR_TB_MARKERS = ("/_dynamo/", "/_inductor/", "/triton/", "torch/fx/")
|
||||
_COMPILE_ERR_MSG_MARKERS = (
|
||||
"dynamo", "inductor", "triton", "cudagraph",
|
||||
"symbolically trace", "torch.compile", "fx graph",
|
||||
)
|
||||
|
||||
|
||||
def _is_compile_runtime_failure(exc: BaseException) -> bool:
|
||||
"""True when an exception originates in the torch.compile stack (Dynamo /
|
||||
Inductor / Triton / FX / CUDA-graph trees) rather than in the model itself.
|
||||
|
||||
#278: on GPU architectures Triton doesn't support yet (e.g. Blackwell
|
||||
sm_120), the compiled model dies mid-generation with errors like
|
||||
"Detected that you are using FX to symbolically trace a dynamo-optimized
|
||||
function" or an AssertionError out of torch/_inductor/cudagraph_trees.py.
|
||||
Walks the exception chain and checks (a) the exception type's module,
|
||||
(b) the message, (c) the traceback file paths — the cudagraph case is a
|
||||
bare AssertionError, so the traceback check is load-bearing.
|
||||
"""
|
||||
import traceback as _tb
|
||||
|
||||
seen: set[int] = set()
|
||||
cur: BaseException | None = exc
|
||||
while cur is not None and id(cur) not in seen:
|
||||
seen.add(id(cur))
|
||||
mod = type(cur).__module__ or ""
|
||||
if mod.startswith(_COMPILE_ERR_MODULE_PREFIXES):
|
||||
return True
|
||||
msg = str(cur).lower()
|
||||
if any(marker in msg for marker in _COMPILE_ERR_MSG_MARKERS):
|
||||
return True
|
||||
try:
|
||||
for frame in _tb.extract_tb(cur.__traceback__):
|
||||
filename = (frame.filename or "").replace("\\", "/")
|
||||
if any(marker in filename for marker in _COMPILE_ERR_TB_MARKERS):
|
||||
return True
|
||||
except Exception as traceback_scan_error:
|
||||
logging.debug(
|
||||
"Skipping traceback marker scan while classifying compile runtime failure: %s",
|
||||
traceback_scan_error,
|
||||
)
|
||||
# Follow the chain, honoring `raise ... from None` (the eager-retry
|
||||
# path suppresses the original compile error so a genuine eager
|
||||
# failure isn't misclassified as a compile failure).
|
||||
if cur.__cause__ is not None:
|
||||
cur = cur.__cause__
|
||||
elif not cur.__suppress_context__:
|
||||
cur = cur.__context__
|
||||
else:
|
||||
cur = None
|
||||
return False
|
||||
|
||||
|
||||
def _install_compile_fallback(_model) -> None:
|
||||
"""Wrap ``model.generate`` so a torch.compile failure at inference time
|
||||
falls back to the eager (uncompiled) model instead of failing the
|
||||
generation (#278).
|
||||
|
||||
All TTS paths (generate, archetype previews, dub, stream, batch) funnel
|
||||
through ``model.generate``, so this is the single choke point. On a
|
||||
compile-stack failure we: log a clear warning, restore the eager module
|
||||
(``OptimizedModule._orig_mod``), disable compile for the rest of the
|
||||
session via ``engine_env.mark_compile_runtime_failure``, reset dynamo
|
||||
state, and retry the call once eagerly. Non-compile errors (real OOM,
|
||||
validation, …) propagate unchanged — fully backward compatible for users
|
||||
whose torch.compile works.
|
||||
"""
|
||||
orig_generate = _model.generate
|
||||
|
||||
def _generate_with_compile_fallback(*args, **kwargs):
|
||||
try:
|
||||
return orig_generate(*args, **kwargs)
|
||||
except Exception as exc:
|
||||
compiled = getattr(_model, "llm", None)
|
||||
eager = getattr(compiled, "_orig_mod", None)
|
||||
if eager is None or not _is_compile_runtime_failure(exc):
|
||||
raise
|
||||
logger.warning(
|
||||
"torch.compile runtime failure during generation (%s: %s) — "
|
||||
"falling back to the eager model and disabling torch.compile "
|
||||
"for this session. Generation is being retried without it.",
|
||||
type(exc).__name__, exc,
|
||||
)
|
||||
from services import engine_env
|
||||
engine_env.mark_compile_runtime_failure(f"{type(exc).__name__}: {exc}")
|
||||
_model.llm = eager
|
||||
try:
|
||||
torch = _lazy_torch()
|
||||
torch._dynamo.reset()
|
||||
except Exception as reset_exc:
|
||||
logger.debug(
|
||||
"Non-fatal: failed to reset torch._dynamo state after compile failure (%s: %s). "
|
||||
"Continuing with eager fallback.",
|
||||
type(reset_exc).__name__,
|
||||
reset_exc,
|
||||
)
|
||||
try:
|
||||
return orig_generate(*args, **kwargs)
|
||||
except Exception as eager_exc:
|
||||
# `from None` so a genuine eager failure (e.g. a real OOM)
|
||||
# isn't chained to — and misclassified as — the compile error.
|
||||
raise eager_exc from None
|
||||
|
||||
_model.generate = _generate_with_compile_fallback
|
||||
|
||||
|
||||
# ── #315: thread affinity for cudagraph-compiled models ─────────────────────
|
||||
# `torch.compile(mode="reduce-overhead")` captures CUDA graphs, and captured
|
||||
# graph state is **thread-local** (torch/_inductor/cudagraph_trees keys its
|
||||
# tree manager off the capturing thread). The `_gpu_pool` runs up to
|
||||
# `_GPU_WORKER_CAP` threads, so render #1 captures the graph on worker A and a
|
||||
# later render dispatched to worker B replays against mismatched cudagraph
|
||||
# state — silently corrupting the audio (static / slowed playback, no
|
||||
# exception, so the #278 eager fallback never fires). Fix: every call into a
|
||||
# cudagraph-compiled model executes on ONE dedicated thread; uncompiled
|
||||
# models (CPU / MPS / Windows-no-Triton / compile-disabled) keep the full pool.
|
||||
|
||||
_TORCH_COMPILE_MODE = "reduce-overhead"
|
||||
# Compile modes that enable CUDA graphs under the hood — these need the
|
||||
# single-thread affinity below. "default" / "max-autotune-no-cudagraphs"
|
||||
# would not.
|
||||
_CUDAGRAPH_COMPILE_MODES = frozenset({"reduce-overhead", "max-autotune"})
|
||||
|
||||
_compiled_inference_executor: "ThreadPoolExecutor | None" = None
|
||||
_compiled_inference_thread_ident: "int | None" = None
|
||||
|
||||
|
||||
def _get_compiled_inference_executor() -> ThreadPoolExecutor:
|
||||
"""The single-thread executor that owns ALL inference on a compiled model.
|
||||
|
||||
Created lazily the first time a model is compiled with a cudagraph mode;
|
||||
reused across model reloads (idle unload → reload keeps the same thread,
|
||||
which is fine — a fresh compile simply captures its graphs there too).
|
||||
The worker is spun up eagerly so its thread ident is known for the
|
||||
re-entrancy guard in `_install_compile_thread_affinity`.
|
||||
"""
|
||||
global _compiled_inference_executor, _compiled_inference_thread_ident
|
||||
if _compiled_inference_executor is None:
|
||||
_compiled_inference_executor = ThreadPoolExecutor(
|
||||
max_workers=1, thread_name_prefix="compiled-infer",
|
||||
)
|
||||
_compiled_inference_thread_ident = _compiled_inference_executor.submit(
|
||||
threading.get_ident
|
||||
).result()
|
||||
return _compiled_inference_executor
|
||||
|
||||
|
||||
def _install_compile_thread_affinity(_model) -> None:
|
||||
"""Pin every ``model.generate`` call to the dedicated compile thread (#315).
|
||||
|
||||
Wraps ``model.generate`` (the single choke point all TTS paths funnel
|
||||
through — generate, archetype previews, dub, stream, batch) so the call
|
||||
body always runs on `_get_compiled_inference_executor()`'s one thread.
|
||||
That makes the thread that *captures* the CUDA graph on the first render
|
||||
and the thread that *replays* it on every later render the same thread,
|
||||
deterministically, regardless of which `_gpu_pool` worker dispatched it.
|
||||
|
||||
Installed AFTER `_install_compile_fallback`, so the call-time order is:
|
||||
caller thread → hop to the dedicated thread → eager-fallback wrapper →
|
||||
real generate (the #278 classification/retry also runs on the dedicated
|
||||
thread, with native tracebacks). The hop is a no-op when already on the
|
||||
dedicated thread — a 1-worker executor submitting to itself would
|
||||
deadlock, so the re-entrancy guard is load-bearing.
|
||||
"""
|
||||
executor = _get_compiled_inference_executor()
|
||||
inner_generate = _model.generate
|
||||
|
||||
def _generate_on_compile_thread(*args, **kwargs):
|
||||
if threading.get_ident() == _compiled_inference_thread_ident:
|
||||
return inner_generate(*args, **kwargs)
|
||||
return executor.submit(inner_generate, *args, **kwargs).result()
|
||||
|
||||
_model.generate = _generate_on_compile_thread
|
||||
|
||||
|
||||
def _set_loading(sub_stage: str, detail: str = "", error: str | None = None, progress: float | None = None):
|
||||
"""Update the loading detail dict atomically."""
|
||||
_loading_detail["sub_stage"] = sub_stage
|
||||
@@ -483,23 +290,9 @@ def _load_model_sync():
|
||||
logger.info("Preloading PyTorch Whisper with TTS model.")
|
||||
else:
|
||||
logger.info("Skipping PyTorch Whisper preload; ASR will load on demand.")
|
||||
try:
|
||||
_model = OmniVoice.from_pretrained(
|
||||
checkpoint, device_map=device, dtype=torch.float16, load_asr=preload_asr,
|
||||
)
|
||||
except OSError as e:
|
||||
# #352: a truncated HF cache surfaces here as "does not appear to
|
||||
# have a file named pytorch_model.bin or model.safetensors".
|
||||
# Translate to an actionable message instead of the raw
|
||||
# transformers error.
|
||||
if "does not appear to have a file named" in str(e):
|
||||
raise RuntimeError(
|
||||
f"The TTS model cache for {checkpoint} is incomplete "
|
||||
"(weights missing — usually an interrupted download). "
|
||||
"Open Settings → Models, delete the OmniVoice TTS model, "
|
||||
"and install it again."
|
||||
) from e
|
||||
raise
|
||||
_model = OmniVoice.from_pretrained(
|
||||
checkpoint, device_map=device, dtype=torch.float16, load_asr=preload_asr,
|
||||
)
|
||||
|
||||
try:
|
||||
# plan-02 (#65): gate on Triton availability (+ user setting), not
|
||||
@@ -510,38 +303,8 @@ def _load_model_sync():
|
||||
|
||||
if should_torch_compile(device):
|
||||
_set_loading("compiling", "Compiling model (torch.compile)…")
|
||||
try:
|
||||
_model.llm = torch.compile(_model.llm, mode=_TORCH_COMPILE_MODE)
|
||||
except Exception as compile_exc:
|
||||
# #278: compile is an optimization, never a point of
|
||||
# failure — keep the eager model and remember the failure
|
||||
# so later loads this session skip compile up front.
|
||||
from services.engine_env import mark_compile_runtime_failure
|
||||
mark_compile_runtime_failure(f"{type(compile_exc).__name__}: {compile_exc}")
|
||||
logger.warning(
|
||||
"torch.compile failed (%s) — continuing with the eager model.",
|
||||
compile_exc,
|
||||
)
|
||||
else:
|
||||
# Compilation is lazy: Dynamo/Inductor/Triton can still
|
||||
# blow up on the first *forward* (e.g. unsupported new GPU
|
||||
# archs, #278). Wrap generate so that falls back to eager
|
||||
# instead of failing the generation.
|
||||
_install_compile_fallback(_model)
|
||||
if _TORCH_COMPILE_MODE in _CUDAGRAPH_COMPILE_MODES:
|
||||
# #315: reduce-overhead uses CUDA graphs, whose
|
||||
# captured state is thread-local. Pin all inference to
|
||||
# one dedicated thread so a later render dispatched to
|
||||
# a different _gpu_pool worker can't replay a graph it
|
||||
# didn't capture (static / slowed audio from the 2nd
|
||||
# render onward).
|
||||
_install_compile_thread_affinity(_model)
|
||||
logger.info(
|
||||
"torch.compile mode %r uses CUDA graphs — compiled-model "
|
||||
"inference pinned to a single dedicated thread (#315).",
|
||||
_TORCH_COMPILE_MODE,
|
||||
)
|
||||
logger.info("torch.compile applied.")
|
||||
_model.llm = torch.compile(_model.llm, mode="reduce-overhead")
|
||||
logger.info("torch.compile applied.")
|
||||
except Exception as e:
|
||||
logger.info("torch.compile skipped: %s", e)
|
||||
|
||||
@@ -680,28 +443,13 @@ def get_model_status():
|
||||
result["error"] = err
|
||||
return result
|
||||
|
||||
def _resolve_idle_timeout() -> float:
|
||||
"""In-process model idle timeout in seconds (MM2-05): prefs store → env →
|
||||
core.config default, env winning. Resolved per-tick so a settings change
|
||||
takes effect without a restart."""
|
||||
try:
|
||||
from core import prefs
|
||||
return float(prefs.resolve(
|
||||
"idle_timeout_seconds",
|
||||
env="OMNIVOICE_IDLE_TIMEOUT_S",
|
||||
default=IDLE_TIMEOUT_SECONDS,
|
||||
))
|
||||
except (TypeError, ValueError, ImportError):
|
||||
return float(IDLE_TIMEOUT_SECONDS)
|
||||
|
||||
|
||||
async def idle_worker():
|
||||
global model
|
||||
torch = _lazy_torch()
|
||||
while True:
|
||||
await asyncio.sleep(30)
|
||||
async with _model_lock:
|
||||
if model is not None and time.time() - _last_used > _resolve_idle_timeout():
|
||||
if model is not None and time.time() - _last_used > _IDLE_TIMEOUT_SECONDS:
|
||||
logger.info("Idle timeout reached. Unloading OmniVoice model to free VRAM.")
|
||||
model = None
|
||||
free_vram()
|
||||
|
||||
@@ -1,187 +0,0 @@
|
||||
"""
|
||||
Speech-onset alignment for transcript segments (issue #280, item 1).
|
||||
|
||||
Whisper-family ASR models are prone to stretching a segment's *start* back
|
||||
over leading non-speech (intro music, room tone, silence). The classic
|
||||
symptom from the issue report: the speaker starts talking at 0:02–0:03,
|
||||
but the first transcript segment says ``start=0.0`` — so the dubbed line
|
||||
plays the moment the video begins and everything feels desynchronised.
|
||||
|
||||
``snap_segment_starts`` post-processes segments against the actual audio
|
||||
(ideally the Demucs-isolated vocals track, which the dub pipeline already
|
||||
produces): for each segment it scans the waveform inside ``[start, end]``
|
||||
for the first frame whose RMS rises above an adaptive threshold and moves
|
||||
``start`` forward to just before that onset.
|
||||
|
||||
Design constraints:
|
||||
|
||||
* **Forward-only.** A segment start is never moved earlier — that could
|
||||
collide with the previous speaker. We only trim leading non-speech.
|
||||
* **Conservative.** Shifts below ``min_shift_s`` are ignored (word-level
|
||||
timestamps are usually within ~100 ms already); a minimum segment
|
||||
duration is always preserved; segments whose window looks silent
|
||||
(no frame above the absolute floor) are left untouched.
|
||||
* **Pure NumPy.** No model, no platform-specific code — identical
|
||||
behaviour on macOS / Windows / Linux, trivially unit-testable.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Sequence
|
||||
|
||||
import numpy as np
|
||||
|
||||
logger = logging.getLogger("omnivoice.onset_align")
|
||||
|
||||
# Analysis frame for RMS energy. 20 ms is fine-grained enough to localise
|
||||
# a syllable onset while staying cheap (a 10-min track is ~30k frames).
|
||||
FRAME_S = 0.02
|
||||
# Keep this much audio before the detected onset so plosives/breaths that
|
||||
# sit just under the threshold aren't clipped off.
|
||||
PRE_ROLL_S = 0.05
|
||||
# Shifts smaller than this are noise — word-level ASR timestamps are
|
||||
# usually accurate to ~0.1 s, so don't churn segment data for less.
|
||||
MIN_SHIFT_S = 0.15
|
||||
# Never shrink a segment below this duration when shifting its start.
|
||||
MIN_SEG_DUR_S = 0.30
|
||||
# A frame must exceed `RELATIVE_THRESHOLD × peak RMS of the window` to
|
||||
# count as speech onset…
|
||||
RELATIVE_THRESHOLD = 0.10
|
||||
# …and the window's peak RMS must exceed this absolute floor, otherwise
|
||||
# the whole window is treated as silence and left alone (we'd only be
|
||||
# snapping to noise).
|
||||
ABS_RMS_FLOOR = 1e-3
|
||||
|
||||
|
||||
def _frame_rms(x: np.ndarray, frame_len: int) -> np.ndarray:
|
||||
"""RMS per non-overlapping frame; the ragged tail frame is dropped."""
|
||||
n = (len(x) // frame_len) * frame_len
|
||||
if n <= 0:
|
||||
return np.zeros(0, dtype=np.float32)
|
||||
frames = x[:n].reshape(-1, frame_len).astype(np.float64, copy=False)
|
||||
return np.sqrt((frames * frames).mean(axis=1)).astype(np.float32)
|
||||
|
||||
|
||||
def detect_speech_onset(
|
||||
audio: np.ndarray,
|
||||
sr: int,
|
||||
start_s: float,
|
||||
end_s: float,
|
||||
) -> float | None:
|
||||
"""Return the absolute time (s) of the first speech-like frame inside
|
||||
``[start_s, end_s]``, or ``None`` when the window is empty / silent.
|
||||
"""
|
||||
if sr <= 0 or end_s <= start_s:
|
||||
return None
|
||||
i0 = max(0, int(start_s * sr))
|
||||
i1 = min(len(audio), int(end_s * sr))
|
||||
if i1 <= i0:
|
||||
return None
|
||||
window = audio[i0:i1]
|
||||
frame_len = max(1, int(FRAME_S * sr))
|
||||
rms = _frame_rms(window, frame_len)
|
||||
if rms.size == 0:
|
||||
return None
|
||||
peak = float(rms.max())
|
||||
if peak < ABS_RMS_FLOOR:
|
||||
return None # whole window is effectively silent
|
||||
threshold = max(RELATIVE_THRESHOLD * peak, ABS_RMS_FLOOR)
|
||||
above = np.nonzero(rms >= threshold)[0]
|
||||
if above.size == 0:
|
||||
return None
|
||||
return start_s + float(above[0]) * (frame_len / sr)
|
||||
|
||||
|
||||
# Hysteresis for full-track onset listing: after a frame crosses the
|
||||
# threshold, the energy must stay *below* it for at least this long before
|
||||
# the next rise counts as a new onset. Stops syllable-internal dips from
|
||||
# spamming the timeline with ticks.
|
||||
MIN_ONSET_GAP_S = 0.15
|
||||
|
||||
|
||||
def detect_speech_onsets(audio: np.ndarray, sr: int) -> list[float]:
|
||||
"""Return the times (s) of every speech-like onset across the whole track.
|
||||
|
||||
Powers the timeline editor's snap-to-onset ticks (issue #280, item 3):
|
||||
frame RMS over the full track, single adaptive threshold
|
||||
``max(RELATIVE_THRESHOLD × peak, ABS_RMS_FLOOR)``, and hysteresis — a
|
||||
new onset registers only when the energy rises above the threshold
|
||||
after at least ``MIN_ONSET_GAP_S`` below it.
|
||||
|
||||
Pure NumPy, identical behaviour on every platform. Returns ``[]`` for
|
||||
empty/silent audio.
|
||||
"""
|
||||
if sr <= 0 or audio is None or len(audio) == 0:
|
||||
return []
|
||||
if audio.ndim > 1:
|
||||
audio = audio.mean(axis=1)
|
||||
frame_len = max(1, int(FRAME_S * sr))
|
||||
rms = _frame_rms(audio, frame_len)
|
||||
if rms.size == 0:
|
||||
return []
|
||||
peak = float(rms.max())
|
||||
if peak < ABS_RMS_FLOOR:
|
||||
return [] # whole track is effectively silent
|
||||
threshold = max(RELATIVE_THRESHOLD * peak, ABS_RMS_FLOOR)
|
||||
gap_frames = max(1, int(round(MIN_ONSET_GAP_S / FRAME_S)))
|
||||
frame_s = frame_len / sr
|
||||
|
||||
onsets: list[float] = []
|
||||
below_run = gap_frames # armed, so speech at t=0 still counts
|
||||
for i, v in enumerate(rms):
|
||||
if v >= threshold:
|
||||
if below_run >= gap_frames:
|
||||
onsets.append(round(i * frame_s, 3))
|
||||
below_run = 0
|
||||
else:
|
||||
below_run += 1
|
||||
return onsets
|
||||
|
||||
|
||||
def snap_segment_starts(
|
||||
segments: Sequence[dict],
|
||||
audio: np.ndarray,
|
||||
sr: int,
|
||||
*,
|
||||
min_shift_s: float = MIN_SHIFT_S,
|
||||
) -> int:
|
||||
"""Snap each segment's ``start`` forward to the actual speech onset.
|
||||
|
||||
Mutates the segment dicts in place (the shape the dub pipeline passes
|
||||
around). Returns the number of segments adjusted.
|
||||
|
||||
``audio`` should be mono float; the Demucs vocals track gives the best
|
||||
signal but the mixed track still beats nothing.
|
||||
"""
|
||||
if sr <= 0 or audio is None or len(audio) == 0:
|
||||
return 0
|
||||
if audio.ndim > 1:
|
||||
audio = audio.mean(axis=1)
|
||||
|
||||
adjusted = 0
|
||||
for seg in segments:
|
||||
try:
|
||||
start = float(seg.get("start", 0.0))
|
||||
end = float(seg.get("end", 0.0))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
if end - start < MIN_SEG_DUR_S + min_shift_s:
|
||||
continue # too short for a meaningful shift
|
||||
onset = detect_speech_onset(audio, sr, start, end)
|
||||
if onset is None:
|
||||
continue
|
||||
new_start = max(start, onset - PRE_ROLL_S)
|
||||
shift = new_start - start
|
||||
if shift < min_shift_s:
|
||||
continue
|
||||
# Preserve a minimum playable duration.
|
||||
new_start = min(new_start, end - MIN_SEG_DUR_S)
|
||||
if new_start - start < min_shift_s:
|
||||
continue
|
||||
seg["start"] = round(new_start, 3)
|
||||
adjusted += 1
|
||||
|
||||
if adjusted:
|
||||
logger.info("onset-align: snapped %d/%d segment start(s) to speech onset",
|
||||
adjusted, len(segments))
|
||||
return adjusted
|
||||
@@ -1,436 +0,0 @@
|
||||
"""`.ovsvoice` persona-bundle format (#29 / parity §R3 G1).
|
||||
|
||||
A portable ZIP that packages a voice profile's identity + an optional reference
|
||||
clip + a consent attestation + an SPDX license tag + a watermarked preview.
|
||||
|
||||
This module owns the **pure, model-free** core: the format constants, SPDX
|
||||
normalization, and the manifest/consent builders. The audio preview + ZIP
|
||||
pack/unpack (which lazily import torchaudio/watermark) layer on top of these.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
import zipfile
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
# ── Format constants ─────────────────────────────────────────────────────────
|
||||
OVSVOICE_FORMAT = "ovsvoice"
|
||||
OVSVOICE_SCHEMA_VERSION = 1
|
||||
MAX_BUNDLE_BYTES = 100 * 1024 * 1024 # 100 MB (mirrors marketplace cap)
|
||||
_MIN_CONSENT_AUDIO_BYTES = 1000 # the consent-recording floor
|
||||
DEFAULT_LICENSE = "LicenseRef-OmniVoice-Personal"
|
||||
PREVIEW_MAX_SECONDS = 8.0 # preview length cap (A6)
|
||||
PREVIEW_SAMPLE_RATE = 24_000 # preview rate; mono, 16-bit PCM (A8)
|
||||
|
||||
# Audio member prefixes the importer recognises. The ZIP member NAME is never
|
||||
# used to build an output path (zip-slip safe, B10) — only its extension, and
|
||||
# only after the linear allowlist below.
|
||||
_AUDIO_MEMBER_PREFIXES = ("ref_audio", "locked_audio", "consent_audio", "preview")
|
||||
# Reused verbatim from profiles.py:306 — single linear quantifier, no ReDoS.
|
||||
_MEMBER_EXT_RE = re.compile(r"^\.[A-Za-z0-9]{1,8}$")
|
||||
|
||||
# Membership allowlist for SPDX validation — a fixed-string set + the
|
||||
# ``LicenseRef-`` prefix. NO regex over the (user-supplied) SPDX string, so this
|
||||
# carries no CodeQL py/polynomial-redos surface.
|
||||
_SPDX_ALLOWLIST: frozenset[str] = frozenset({
|
||||
"CC0-1.0", "CC-BY-4.0", "CC-BY-SA-4.0", "CC-BY-NC-4.0", "CC-BY-NC-SA-4.0",
|
||||
"CC-BY-ND-4.0", "MIT", "Apache-2.0", "LicenseRef-OmniVoice-Personal",
|
||||
})
|
||||
|
||||
|
||||
class BundleError(Exception):
|
||||
"""A bundle build/parse failure carrying the HTTP status the router maps to."""
|
||||
|
||||
def __init__(self, status: int, detail: str):
|
||||
super().__init__(detail)
|
||||
self.status = status
|
||||
self.detail = detail
|
||||
|
||||
|
||||
class NoPreviewSource(Exception):
|
||||
"""No readable source clip exists to build a preview from (A2/A3/A4/A5/A12).
|
||||
The router maps this to HTTP 503."""
|
||||
|
||||
|
||||
def _safe_member_ext(member_name: str) -> str:
|
||||
"""The extension for a ZIP member, allowlisted to ``^\\.[A-Za-z0-9]{1,8}$``
|
||||
(else ``.wav``). Used ONLY to choose the output extension — never the path
|
||||
(B11). Linear regex, no ReDoS."""
|
||||
ext = os.path.splitext(member_name)[1]
|
||||
return ext if _MEMBER_EXT_RE.match(ext) else ".wav"
|
||||
|
||||
|
||||
def normalize_spdx(spdx: Optional[str]) -> str:
|
||||
"""Return a safe SPDX id: the value if it's allowlisted or a ``LicenseRef-``
|
||||
custom id, else :data:`DEFAULT_LICENSE`. Never raises, never 400s — a junk
|
||||
id (incl. shell-injection attempts) normalizes to the default. Membership /
|
||||
fixed-prefix only — no regex (CodeQL-clean)."""
|
||||
if not spdx or not isinstance(spdx, str):
|
||||
return DEFAULT_LICENSE
|
||||
s = spdx.strip()
|
||||
if s in _SPDX_ALLOWLIST or s.startswith("LicenseRef-"):
|
||||
return s
|
||||
return DEFAULT_LICENSE
|
||||
|
||||
|
||||
def build_manifest(
|
||||
profile: dict,
|
||||
*,
|
||||
license_spdx: str,
|
||||
tags: list[str],
|
||||
engine_id: str = "",
|
||||
custom_license_text: Optional[str] = None,
|
||||
preview: Optional[dict] = None,
|
||||
members: Optional[dict] = None,
|
||||
omnivoice_version: str = "",
|
||||
) -> dict:
|
||||
"""Build the ``manifest.json`` object for a profile row. Mirrors the legacy
|
||||
``_bundle_metadata`` persona fields, adds the format discriminator, license
|
||||
(normalized — never raises on a bad id, A19), tags, preview + members blocks.
|
||||
Pure: no I/O, no model."""
|
||||
return {
|
||||
"format": OVSVOICE_FORMAT,
|
||||
"schema_version": OVSVOICE_SCHEMA_VERSION,
|
||||
"omnivoice_version": omnivoice_version or "",
|
||||
"exported_at": time.time(),
|
||||
"persona": {
|
||||
"name": profile.get("name") or "",
|
||||
"kind": profile.get("kind") or "clone",
|
||||
"language": profile.get("language") or "Auto",
|
||||
"personality": profile.get("personality") or "",
|
||||
"instruct": profile.get("instruct") or "",
|
||||
"ref_text": profile.get("ref_text") or "",
|
||||
"seed": profile.get("seed"), # int or None (A16)
|
||||
"is_locked": bool(profile.get("is_locked")),
|
||||
"vd_states": profile.get("vd_states"), # JSON string or None (A15) — never re-parsed
|
||||
},
|
||||
"engine": {"id": engine_id or "", "design_params": None},
|
||||
"license": {"spdx": normalize_spdx(license_spdx), "custom_text": custom_license_text or None},
|
||||
"tags": list(tags or []),
|
||||
"preview": preview, # set by the audio step; None for legacy/no-preview
|
||||
"members": members or {"ref_audio": None, "locked_audio": None, "consent_audio": None},
|
||||
}
|
||||
|
||||
|
||||
def build_consent_json(profile: dict, *, has_recording: bool) -> Optional[dict]:
|
||||
"""The optional ``consent.json`` for a profile, or None when there's nothing
|
||||
to attest. A ``design`` persona attests as designed-synthetic by definition;
|
||||
a verified clone attests as a self-recorded statement. Import treats these
|
||||
fields as ADVISORY — real verification needs the actual consent_audio member
|
||||
(see the import rules), so this can't forge verified-own-voice."""
|
||||
kind = profile.get("kind") or "clone"
|
||||
consent_text = (profile.get("consent_text") or "").strip()
|
||||
verified = bool(profile.get("verified_own_voice"))
|
||||
if kind == "design":
|
||||
method = "designed-synthetic"
|
||||
verified = True
|
||||
elif verified or consent_text or has_recording:
|
||||
method = "self-recorded-statement"
|
||||
else:
|
||||
return None # nothing to attest
|
||||
recorded_at = profile.get("consent_recorded_at")
|
||||
try:
|
||||
recorded_at = float(recorded_at)
|
||||
except (TypeError, ValueError):
|
||||
recorded_at = time.time()
|
||||
return {
|
||||
"verified_own_voice": verified,
|
||||
"method": method,
|
||||
"consent_text": consent_text,
|
||||
"recorded_at": recorded_at,
|
||||
"has_recording": bool(has_recording),
|
||||
}
|
||||
|
||||
|
||||
def _legacy_metadata(profile: dict, omnivoice_version: str) -> dict:
|
||||
"""A ``metadata.json`` payload shaped like marketplace ``_bundle_metadata`` so
|
||||
an OLDER OmniVoice (which only reads metadata.json) can still import the ref
|
||||
audio from a ``.ovsvoice`` bundle."""
|
||||
return {
|
||||
"bundle_version": 1,
|
||||
"profile_name": profile.get("name") or "",
|
||||
"ref_text": profile.get("ref_text") or "",
|
||||
"instruct": profile.get("instruct") or "",
|
||||
"language": profile.get("language") or "Auto",
|
||||
"personality": profile.get("personality") or "",
|
||||
"seed": profile.get("seed"),
|
||||
"kind": profile.get("kind") or "clone",
|
||||
"vd_states": profile.get("vd_states"),
|
||||
"is_locked": bool(profile.get("is_locked")),
|
||||
"omnivoice_version": omnivoice_version or "",
|
||||
}
|
||||
|
||||
|
||||
def _resolve_voice_file(filename: Optional[str]) -> Optional[str]:
|
||||
"""Resolve a DB-stored audio filename strictly inside VOICES_DIR, returning
|
||||
an absolute path only if the file actually exists. None on missing/escape —
|
||||
mirrors profiles._voices_path (basename + realpath confinement, E1)."""
|
||||
if not filename or os.path.basename(filename) != filename:
|
||||
return None
|
||||
from core.config import VOICES_DIR
|
||||
root = os.path.realpath(VOICES_DIR)
|
||||
path = os.path.realpath(os.path.join(root, filename))
|
||||
if not path.startswith(root + os.sep):
|
||||
return None
|
||||
return path if os.path.isfile(path) else None
|
||||
|
||||
|
||||
def _generate_preview(profile: dict, embed_fn) -> tuple[bytes, bool, float]:
|
||||
"""Load the profile's source clip, downmix→mono, resample→24 kHz, trim ≤8 s,
|
||||
watermark (forced), and return ``(wav_bytes, watermarked, duration_s)``.
|
||||
|
||||
Source precedence is locked-over-ref (profiles.py:230). Raises
|
||||
:class:`NoPreviewSource` when neither clip is readable (A2-A5). All heavy
|
||||
imports (torch/torchaudio/watermark) are lazy so the module stays model-free
|
||||
at collection time (avoids the known local torch/Triton segfault)."""
|
||||
import torch # noqa: F401 (torchaudio needs it loaded)
|
||||
import torchaudio
|
||||
from services.audio_io import _safe_torchaudio_save
|
||||
from services.watermark import _check_available
|
||||
|
||||
candidates = [profile.get("locked_audio_path"), profile.get("ref_audio_path")]
|
||||
wav = None
|
||||
for name in candidates:
|
||||
path = _resolve_voice_file(name)
|
||||
if not path:
|
||||
continue
|
||||
try:
|
||||
waveform, sr = torchaudio.load(path)
|
||||
except Exception: # noqa: BLE001 — try the next candidate (A4)
|
||||
continue
|
||||
if waveform.numel() == 0: # empty/zero-length (A5)
|
||||
continue
|
||||
if waveform.shape[0] > 1: # downmix to mono (A7)
|
||||
waveform = waveform.mean(dim=0, keepdim=True)
|
||||
if sr != PREVIEW_SAMPLE_RATE: # resample (A8)
|
||||
waveform = torchaudio.functional.resample(waveform, sr, PREVIEW_SAMPLE_RATE)
|
||||
cap = int(PREVIEW_SAMPLE_RATE * PREVIEW_MAX_SECONDS)
|
||||
waveform = waveform[:, :cap] # trim, shorter used whole (A6)
|
||||
wav = waveform
|
||||
break
|
||||
|
||||
if wav is None or wav.numel() == 0:
|
||||
raise NoPreviewSource("no readable reference or locked audio for a preview")
|
||||
|
||||
# Forced watermark — bypasses the user pref but still no-ops without AudioSeal.
|
||||
fn = embed_fn or _default_embed
|
||||
wav = fn(wav, PREVIEW_SAMPLE_RATE)
|
||||
watermarked = bool(_check_available()) # best-effort honesty (A11)
|
||||
|
||||
duration_s = round(wav.shape[-1] / PREVIEW_SAMPLE_RATE, 3)
|
||||
buf = io.BytesIO()
|
||||
_safe_torchaudio_save(buf, wav, PREVIEW_SAMPLE_RATE, format="wav", bits_per_sample=16)
|
||||
return buf.getvalue(), watermarked, duration_s
|
||||
|
||||
|
||||
def _default_embed(wav, sample_rate):
|
||||
"""Default preview watermarker: services.watermark.embed_watermark(force=True)."""
|
||||
from services.watermark import embed_watermark
|
||||
return embed_watermark(wav, sample_rate, force=True)
|
||||
|
||||
|
||||
def build_persona_bundle(
|
||||
profile: dict,
|
||||
*,
|
||||
license_spdx: str = DEFAULT_LICENSE,
|
||||
tags: Optional[list[str]] = None,
|
||||
custom_license_text: Optional[str] = None,
|
||||
include_reference: bool = True,
|
||||
engine_id: str = "",
|
||||
omnivoice_version: str = "",
|
||||
embed_fn=None,
|
||||
) -> bytes:
|
||||
"""Assemble a ``.ovsvoice`` ZIP in memory and return its bytes.
|
||||
|
||||
Always writes a watermarked ``preview.wav`` + ``manifest.json`` +
|
||||
(legacy-shaped) ``metadata.json``. Writes ``consent.json`` when there's
|
||||
something to attest, the raw ``ref_audio``/``locked_audio`` members unless
|
||||
``include_reference=False`` (privacy / preview-only, A12), and
|
||||
``consent_audio`` when a recording exists. Raises :class:`NoPreviewSource`
|
||||
(router → 503) when no source clip is readable."""
|
||||
preview_bytes, watermarked, duration_s = _generate_preview(profile, embed_fn)
|
||||
|
||||
members: dict = {"ref_audio": None, "locked_audio": None, "consent_audio": None}
|
||||
buf = io.BytesIO()
|
||||
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
if include_reference:
|
||||
ref_path = _resolve_voice_file(profile.get("ref_audio_path"))
|
||||
if ref_path:
|
||||
name = f"ref_audio{os.path.splitext(ref_path)[1] or '.wav'}"
|
||||
zf.write(ref_path, name)
|
||||
members["ref_audio"] = name
|
||||
locked_path = _resolve_voice_file(profile.get("locked_audio_path"))
|
||||
if locked_path:
|
||||
name = f"locked_audio{os.path.splitext(locked_path)[1] or '.wav'}"
|
||||
zf.write(locked_path, name)
|
||||
members["locked_audio"] = name
|
||||
|
||||
# Consent recording travels only when it exists and clears the floor.
|
||||
consent_path = _resolve_voice_file(profile.get("consent_audio_path"))
|
||||
has_recording = False
|
||||
if consent_path and os.path.getsize(consent_path) >= _MIN_CONSENT_AUDIO_BYTES:
|
||||
name = f"consent_audio{os.path.splitext(consent_path)[1] or '.wav'}"
|
||||
zf.write(consent_path, name)
|
||||
members["consent_audio"] = name
|
||||
has_recording = True
|
||||
|
||||
zf.writestr("preview.wav", preview_bytes)
|
||||
|
||||
preview_block = {
|
||||
"file": "preview.wav", "watermarked": watermarked,
|
||||
"duration_s": duration_s, "sample_rate": PREVIEW_SAMPLE_RATE,
|
||||
}
|
||||
manifest = build_manifest(
|
||||
profile, license_spdx=license_spdx, tags=tags or [],
|
||||
engine_id=engine_id, custom_license_text=custom_license_text,
|
||||
preview=preview_block, members=members,
|
||||
omnivoice_version=omnivoice_version,
|
||||
)
|
||||
zf.writestr("manifest.json", json.dumps(manifest, ensure_ascii=False, indent=2))
|
||||
zf.writestr("metadata.json",
|
||||
json.dumps(_legacy_metadata(profile, omnivoice_version),
|
||||
ensure_ascii=False, indent=2))
|
||||
consent = build_consent_json(profile, has_recording=has_recording)
|
||||
if consent is not None:
|
||||
zf.writestr("consent.json", json.dumps(consent, ensure_ascii=False, indent=2))
|
||||
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
@dataclass
|
||||
class ParsedPersona:
|
||||
manifest: dict # parsed manifest.json OR synthesized from metadata.json
|
||||
consent: Optional[dict] # parsed consent.json, or None
|
||||
is_legacy: bool # only metadata.json was found (B6/B23)
|
||||
schema_version_ahead: bool # manifest.schema_version > OVSVOICE_SCHEMA_VERSION (B7)
|
||||
license_spdx: str # normalized (B21)
|
||||
preview_only: bool # only preview.wav, no ref/locked member (A12/B8)
|
||||
members: dict # {prefix: member_name} for audio members present
|
||||
watermarked_preview: bool # manifest.preview.watermarked (False for legacy)
|
||||
_zip: zipfile.ZipFile # open handle; router extracts via extract_member()
|
||||
|
||||
def member_ext(self, prefix: str) -> str:
|
||||
name = self.members.get(prefix)
|
||||
return _safe_member_ext(name) if name else ".wav"
|
||||
|
||||
def extract_member(self, prefix: str, dest_path: str) -> bool:
|
||||
"""Stream the audio member named by ``prefix`` to ``dest_path`` (a path
|
||||
the CALLER derived from a server-generated id — never from the member
|
||||
name). Returns False when the member is absent. Last-wins on dup (B9)."""
|
||||
name = self.members.get(prefix)
|
||||
if not name:
|
||||
return False
|
||||
import shutil
|
||||
with self._zip.open(name) as src, open(dest_path, "wb") as dst:
|
||||
shutil.copyfileobj(src, dst)
|
||||
return True
|
||||
|
||||
|
||||
def parse_persona_bundle(content: bytes) -> ParsedPersona:
|
||||
"""Validate the ZIP and read manifest/consent WITHOUT touching the DB or
|
||||
writing files. Raises :class:`BundleError` (400|413) for B1-B11. The caller
|
||||
must use the returned ``ParsedPersona`` while the process holds ``content``
|
||||
(the open ZIP reads from the in-memory bytes)."""
|
||||
if len(content) > MAX_BUNDLE_BYTES:
|
||||
raise BundleError(413, f"Bundle too large. Max is {MAX_BUNDLE_BYTES} bytes.")
|
||||
try:
|
||||
zf = zipfile.ZipFile(io.BytesIO(content))
|
||||
except zipfile.BadZipFile:
|
||||
raise BundleError(400, "not a valid ZIP bundle")
|
||||
|
||||
names = [n for n in zf.namelist() if not n.endswith("/")]
|
||||
|
||||
# Manifest selection: prefer manifest.json, fall back to legacy metadata.json.
|
||||
manifest: dict = {}
|
||||
is_legacy = False
|
||||
if "manifest.json" in names:
|
||||
try:
|
||||
manifest = json.loads(zf.read("manifest.json"))
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
raise BundleError(400, "manifest is not valid JSON")
|
||||
if not isinstance(manifest, dict):
|
||||
raise BundleError(400, "manifest is not valid JSON")
|
||||
# A bundle whose format is neither ovsvoice nor absent → still read
|
||||
# leniently (B6); we only branch on schema_version below.
|
||||
elif "metadata.json" in names:
|
||||
is_legacy = True
|
||||
try:
|
||||
legacy = json.loads(zf.read("metadata.json"))
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
raise BundleError(400, "manifest is not valid JSON")
|
||||
if not isinstance(legacy, dict):
|
||||
raise BundleError(400, "manifest is not valid JSON")
|
||||
manifest = {
|
||||
"format": "omnivoice-legacy",
|
||||
"schema_version": OVSVOICE_SCHEMA_VERSION,
|
||||
"persona": {
|
||||
"name": legacy.get("profile_name") or legacy.get("name") or "Imported Voice",
|
||||
"kind": legacy.get("kind") or "clone",
|
||||
"language": legacy.get("language") or "Auto",
|
||||
"personality": legacy.get("personality") or "",
|
||||
"instruct": legacy.get("instruct") or "",
|
||||
"ref_text": legacy.get("ref_text") or "",
|
||||
"seed": legacy.get("seed"),
|
||||
"is_locked": bool(legacy.get("is_locked")),
|
||||
"vd_states": legacy.get("vd_states"),
|
||||
},
|
||||
"license": {"spdx": DEFAULT_LICENSE, "custom_text": None},
|
||||
"tags": [],
|
||||
"preview": None,
|
||||
"members": {},
|
||||
}
|
||||
else:
|
||||
raise BundleError(400, "bundle is missing a manifest")
|
||||
|
||||
# Audio members by prefix (last-wins on duplicates, B9). The member NAME is
|
||||
# retained only to read bytes + pick an extension — never to build a path.
|
||||
members: dict = {}
|
||||
for name in names:
|
||||
for prefix in _AUDIO_MEMBER_PREFIXES:
|
||||
if os.path.basename(name).startswith(prefix):
|
||||
members[prefix] = name
|
||||
has_audio = any(p in members for p in ("ref_audio", "locked_audio", "preview"))
|
||||
if not has_audio:
|
||||
raise BundleError(400, "bundle has no audio member")
|
||||
|
||||
consent = None
|
||||
if "consent.json" in names:
|
||||
try:
|
||||
parsed = json.loads(zf.read("consent.json"))
|
||||
if isinstance(parsed, dict):
|
||||
consent = parsed
|
||||
except (ValueError, UnicodeDecodeError):
|
||||
consent = None # advisory only — a bad consent.json never 400s
|
||||
|
||||
schema_version = manifest.get("schema_version", OVSVOICE_SCHEMA_VERSION)
|
||||
try:
|
||||
ahead = int(schema_version) > OVSVOICE_SCHEMA_VERSION
|
||||
except (TypeError, ValueError):
|
||||
ahead = False
|
||||
|
||||
preview_block = manifest.get("preview") or {}
|
||||
watermarked_preview = bool(preview_block.get("watermarked")) if isinstance(preview_block, dict) else False
|
||||
license_spdx = normalize_spdx((manifest.get("license") or {}).get("spdx"))
|
||||
preview_only = ("preview" in members
|
||||
and "ref_audio" not in members and "locked_audio" not in members)
|
||||
|
||||
return ParsedPersona(
|
||||
manifest=manifest, consent=consent, is_legacy=is_legacy,
|
||||
schema_version_ahead=ahead, license_spdx=license_spdx,
|
||||
preview_only=preview_only, members=members,
|
||||
watermarked_preview=watermarked_preview, _zip=zf,
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"OVSVOICE_FORMAT", "OVSVOICE_SCHEMA_VERSION", "MAX_BUNDLE_BYTES",
|
||||
"PREVIEW_MAX_SECONDS", "PREVIEW_SAMPLE_RATE", "DEFAULT_LICENSE",
|
||||
"BundleError", "NoPreviewSource", "ParsedPersona",
|
||||
"normalize_spdx", "build_manifest", "build_consent_json",
|
||||
"build_persona_bundle", "parse_persona_bundle",
|
||||
]
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user