Compare commits

..
Author SHA1 Message Date
debpalashandClaude Opus 4.8 362dca9ae1 docs(diagnostics): self-diagnosis section in troubleshooting + README pointer
Settings > About self-check / --diagnose / --deep / diagnostic bundle are
now the documented first step before the per-error entries — and the
support team's first ask on every issue.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:43:03 +05:30
debpalashandClaude Opus 4.8 bbac66ef76 feat(diagnostics): breadcrumbs, deep self-check, report sweep, issue search
Final slice of the bug-tracking work.

- toastErrorWithReport adopted at the high-traffic failure sites: TTS
  generation, dub upload/ingest/transcribe, engine install, engines-matrix
  load, voice profile save/delete/test, batch enqueue/cancel/delete.
  Validation toasts and cancellations stay plain on purpose.
- utils/breadcrumbs.js: local-only ring of the last 20 action names
  (closed-set names only — never content or paths), embedded as a
  'Recent actions' section in the prefilled report. Instrumented: view
  changes, generate, dub pipeline, export, engine switch.
- deep self-check: /system/diagnose?deep=true and --diagnose --deep load
  the active engine and synthesize a short utterance (num_step=4) —
  catches 'installed but broken'. 180s time-box, skips during model load,
  scrubbed failure detail. Verified live: cold-loaded omnivoice and
  produced 2.2s of audio in 43.9s on CUDA.
- 'Search similar issues' action on the ErrorBoundary: scrubbed,
  noise-stripped GitHub issue search URL — dedupe before filing.
- bug_report.md template now points at the diagnostic bundle and the
  --diagnose CLI so manual reports arrive with the same evidence.

Tests: pytest 107 passed (4 new deep-check tests, CJK gate green);
vitest 218 passed (breadcrumbs + issue-search suites); vite build green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:43:03 +05:30
debpalashandClaude Opus 4.8 3d74611be8 feat(diagnostics): error journal, diagnostic bundle, crash notice, global handlers
Second slice of the bug-tracking work — still zero outbound paths beyond
the user's own browser/file manager.

- core/error_journal.py: deduped ring of recent unhandled backend errors
  (fingerprint counts, error_class triage: GPU_OOM, HF_AUTH_FAILED,
  PYANNOTE_LICENSE_REQUIRED, DISK_FULL, FFMPEG_MISSING, NETWORK_ERROR),
  scrubbed, JSONL-persisted so the error that killed the last run survives
  restart. Wired into the global exception handler; 500 bodies now carry
  error_class; GET /system/errors/recent.
- core/diagnostic_bundle.py + POST /system/diagnostic-bundle + Settings >
  About 'Save diagnostic bundle': zip of self-check report, error journal,
  scrubbed log tails — drag onto a GitHub issue; bypasses the ~8k
  prefill-URL ceiling.
- crash-on-next-launch: /system/notifications flags a crash logged before
  this session started (size vs acked-size in prefs, mtime vs process
  start); POST /system/crash/ack; LogsFooter acks on action click.
- utils/globalErrorHandlers.js: uncaught errors + unhandled rejections get
  a throttled, noise-filtered 'Report this bug' toast.
- sidecar log parity fix: _tauri_log_candidates() now lists the Rust
  sidecar's backend.log/backend_err.log on Linux (XDG state dir) and
  Windows (LOCALAPPDATA) — sidecar crashes were only visible on macOS.

Tests: +19 pytest (journal, bundle); suite at 102 passed. Vitest 124
passed; vite build green. Live-verified: journal recorded and classified
a real HF 401 from the test run (HF_AUTH_FAILED, paths scrubbed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:43:03 +05:30
debpalashandClaude Opus 4.8 ee08f56e1b feat(diagnostics): local self-check + scrubbed bug-report pipeline
Closes the gap between 'something broke' and 'a useful GitHub issue
exists' — entirely within the local-first constraint: the only outbound
path remains the user's own browser opening a prefilled issues/new URL.

Backend:
- core/scrub.py: privacy scrubber for anything leaving the machine —
  env-var secret values (*TOKEN*|*KEY*|*SECRET*|*PASSWORD*), credential
  shapes (hf_/ghp_/github_pat_/sk-), home dirs on all three OSes
- core/diagnose.py: 9-check self-check (device+GPU, ffmpeg, HF token,
  disk, data-dir writability, RAM, engine registry, hub reachability),
  pre-scrubbed, ASCII-safe output
- GET /system/diagnose + 'python main.py --diagnose' (exit 0/1)
- /system/info: hardware inventory (os_version, cpu_model, cpu_count,
  ram_total_gb, gpu_name, vram_total_gb, disk_free_gb), cached statics

Frontend:
- utils/bugReport.js: single source for the prefilled-URL builder —
  scrubText twin, hardware context capture, scrubbed error+stack embed,
  URL-length cap; ReportBugButton refactored onto it
- ErrorBoundary 'Report this bug' action with the error attached
- utils/errorToast.jsx toastErrorWithReport(); wired into export toasts
- Settings > About 'Run self-check' with per-check status badges

Tests: 27 pytest (scrub, diagnose) + 15 vitest (bugReport); existing
suites green; verified live (--diagnose, TestClient, vite build).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:43:03 +05:30
476 changed files with 3664 additions and 59799 deletions
-32
View File
@@ -1,32 +0,0 @@
# RTK - Rust Token Killer (Google Antigravity)
**Usage**: Token-optimized CLI proxy for shell commands.
## Rule
Always prefix shell commands with `rtk` to minimize token consumption.
Examples:
```bash
rtk git status
rtk cargo test
rtk ls src/
rtk grep "pattern" src/
rtk find "*.rs" .
rtk docker ps
rtk gh pr list
```
## Meta Commands
```bash
rtk gain # Show token savings
rtk gain --history # Command history with savings
rtk discover # Find missed RTK opportunities
rtk proxy <cmd> # Run raw (no filtering, for debugging)
```
## Why
RTK filters and compresses command output before it reaches the LLM context, saving 60-90% tokens on common operations. Always use `rtk <cmd>` instead of raw commands.
+3 -119
View File
@@ -7,26 +7,12 @@
language: "en-US"
early_access: false
# The review voice: a panel of senior domain experts, not a linter.
tone_instructions: >-
Review as a panel of principal engineers: ML inference, audio DSP, desktop
systems, product polish. Cite exact lines, name the failure mode, give the
concrete fix. No filler praise; raise nits only when they change a decision.
reviews:
# "chill" keeps the bot from blocking merges — it comments, it does not gate.
# Hard gating lives in CI (security.yml) and the constitution's human bar.
profile: chill
request_changes_workflow: false
high_level_summary: true
# Every walkthrough gets a visual: mermaid sequence diagrams for the
# mechanics, plus (via the summary instructions) an ASCII before/after
# sketch when the PR touches UI — so each PR is reviewable at a glance.
sequence_diagrams: true
high_level_summary_instructions: >-
If the PR changes UI (JSX/TSX/CSS/Tauri windows), include a compact ASCII
before/after sketch of the affected layout or component. If it changes
behavior, include a short mermaid flowchart of the new mechanism.
review_status: true
poem: false
@@ -53,8 +39,7 @@ reviews:
- "!**/*.onnx"
- "!tests/fixtures/**"
# One expert lens per subsystem — encode what a passionate senior in each
# domain would actually check, beyond what linters and CI already gate.
# Encode the project's hard constraints so the bot reviews against them.
path_instructions:
- path: "**/*.{py,rs,js,jsx,ts,tsx}"
instructions: >-
@@ -63,119 +48,18 @@ reviews:
model download, or an explicitly opt-in endpoint. Flag any code that
persists or logs values matching *TOKEN*/*KEY*/*SECRET* or absolute user
home paths (/Users/<name>/, C:\\Users\\<name>\\).
- path: "backend/services/**/*.py"
instructions: >-
Review as an ML-inference/audio engineer. Check: thread-safety of model
and cache state across the GPU worker pool; device/dtype assumptions
that break on one of CUDA/MPS/ROCm/CPU; VRAM lifecycle (load/unload,
leaks on the error path); sample-rate, channel-count and tensor-shape
assumptions at engine boundaries; blocking calls inside async paths;
model download/cache behavior when offline. Engine code must stay
backward-compatible with already-installed on-disk model state.
- path: "backend/**/*.py"
instructions: >-
Default features must behave identically on macOS, Windows and Linux.
Platform-specific implementation is allowed, but a divergent user-visible
default is a P0 bug — flag it and suggest an opt-in (Settings/env/flag).
Any DB schema change must go through an alembic migration with an upgrade
path; flag direct schema edits. The backend serves loopback HTTP: treat
every query/path/form param as hostile (path traversal, log injection,
CSRF from a browser tab), and never route user-chosen filesystem
destinations through HTTP — that authorization belongs in the Tauri
process.
- path: "frontend/src/**/*.{js,jsx,ts,tsx}"
instructions: >-
Review as a product-minded senior frontend engineer. Check: stale state
and races (async results landing after unmount or after newer requests);
every user-visible failure has an actionable, non-technical error
message; loading/disabled states during long operations. Every new
user-facing string must be an i18n t('...') key present in ALL 21
frontend/src/i18n/locales/*.json files — flag hardcoded UI strings and
keys missing from any locale.
- path: "frontend/src-tauri/**/*.rs"
instructions: >-
Review as a desktop-systems engineer. Check: every #[tauri::command] is
callable from the webview — validate inputs and scope filesystem/process
access accordingly; window and webview lifecycle on all three OSes;
child-process spawn/exit-code/stderr handling; no unwrap/expect on
user-controlled input; platform cfg blocks keep user-visible defaults
identical across macOS/Windows/Linux.
- path: "tests/**/*.py"
instructions: >-
Review as a test-infrastructure engineer. Check: the test would fail
before the fix and pass after (no tautologies); no sleeps as
synchronization; no module-level imports of app modules that go stale
under sys.modules pollution (resolve at run time); TestClient instances
are function-scoped and not lifespan-bound unless the test needs it;
new functional CJK is allowlisted in tests/test_no_hardcoded_cjk.py with
a justification.
path; flag direct schema edits. Engine code must stay backward-compatible
with already-installed on-disk model state (no forced reinstall).
- path: ".github/workflows/**"
instructions: >-
Pin actions to a major version tag at minimum. Flag any workflow that
grants write permissions it does not need.
# Non-gating pre-merge audits of the project's hard rules (warning mode —
# the human owner is the gate, these make the checklist visible per-PR).
pre_merge_checks:
title:
mode: warning
requirements: >-
Conventional-commit style with scope, e.g. "fix(dub): …", and the issue
reference in title or body. Never propose a version bump.
issue_assessment:
mode: warning
custom_checks:
- name: "Cross-platform default parity"
mode: warning
instructions: >-
If the PR changes any default-mode (out-of-the-box) behavior, verify
it behaves identically on macOS, Windows and Linux, or is moved behind
an explicit opt-in (Settings toggle, env var, or CLI flag). A
platform-divergent default is a P0 per CLAUDE.md — fail this check
and say which platform diverges.
- name: "i18n completeness (21 locales)"
mode: warning
instructions: >-
For every new or changed t('...') key in frontend code, verify the
key exists in all 21 files under frontend/src/i18n/locales/. List any
locale files missing the key. Also flag hardcoded user-facing strings
that bypass i18n entirely.
- name: "Local-first guarantee"
mode: warning
instructions: >-
Verify the PR adds no required cloud calls, accounts, API keys or
telemetry. Outbound traffic is only allowed to GitHub Issues (opt-in
bug reporting) and HuggingFace model downloads. The app must remain
fully functional offline and with reporting disabled.
- name: "Backward compatibility"
mode: warning
instructions: >-
Verify existing omnivoice_data/ (voices, projects, settings) and
already-installed engine model state keep working without manual
migration. Any DB schema change must ship an alembic migration with
an upgrade path. Flag anything that would force users to reinstall an
engine or re-download model weights.
finishing_touches:
docstrings:
enabled: true
unit_tests:
enabled: true
# Feed the bot the project constitution and docs, and let it accumulate
# learnings from review conversations ("@coderabbitai always/never …").
knowledge_base:
code_guidelines:
enabled: true
filePatterns:
- "CLAUDE.md"
- "docs/**/*.md"
learnings:
scope: auto
issues:
scope: auto
pull_requests:
scope: auto
chat:
auto_reply: true
+45
View File
@@ -0,0 +1,45 @@
---
name: 🐛 Bug Report
about: Report a bug to help us improve OmniVoice Studio
title: "[Bug] "
labels: ["bug", "triage"]
assignees: []
---
## Describe the bug
A clear and concise description of what the bug is.
## To reproduce
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '...'
3. See error
## Expected behavior
What you expected to happen.
## Screenshots / Logs
If applicable, add screenshots or paste relevant logs from **Settings → Logs**.
> **Tip:** **Settings → About → "Save diagnostic bundle"** produces a zip
> (self-check report, recent errors, scrubbed log tails) you can drag onto
> this issue — it answers most environment questions below automatically.
> Headless installs: `python backend/main.py --diagnose` prints the same
> self-check (`--deep` also test-loads the active engine).
## Environment
- **OS:** [e.g. macOS 15.2, Windows 11, Ubuntu 24.04]
- **Install method:** [Desktop app / Docker / From source]
- **Version:** [e.g. v0.2.7 — check Settings → About]
- **GPU:** [e.g. NVIDIA RTX 4090 / Apple M3 Pro / CPU only]
- **RAM:** [e.g. 16 GB]
- **Active TTS engine:** [e.g. omnivoice — check Settings → Engines]
## Additional context
Add any other context about the problem here.
-109
View File
@@ -1,109 +0,0 @@
name: 🐛 Bug report
description: Something works incorrectly or crashes (not a first-run/install problem — use the install template for those).
title: "[Bug] "
labels: ["bug", "triage"]
body:
- type: markdown
attributes:
value: |
Thanks for helping improve OmniVoice Studio! 🎙️
**Fastest path to a fix:** **Settings → About → "Save diagnostic bundle"** makes a
zip (self-check + recent errors + scrubbed log tails) — drag it onto this issue and
most of the environment questions below are answered automatically.
Headless: `python backend/main.py --diagnose` (add `--deep` to test-load the engine).
- type: checkboxes
id: preflight
attributes:
label: Before filing
options:
- label: I searched [existing issues](https://github.com/debpalash/OmniVoice-Studio/issues?q=is%3Aissue) and this isn't a duplicate.
required: true
- label: I'm on the latest release (or `main`) — older builds may already be fixed.
required: false
- type: textarea
id: what-happened
attributes:
label: What happened?
description: A clear description of the bug, including the exact error text / toast if any.
placeholder: "Voice cloning failed with '…' after I clicked Generate."
validations:
required: true
- type: textarea
id: repro
attributes:
label: Steps to reproduce
value: |
1.
2.
3.
validations:
required: true
- type: textarea
id: expected
attributes:
label: What did you expect instead?
validations:
required: true
- type: dropdown
id: os
attributes:
label: Operating system
options:
- macOS (Apple Silicon)
- macOS (Intel)
- Windows (x64)
- Linux (AppImage)
- Linux (.deb)
- Linux (other / from source)
- Docker
validations:
required: true
- type: dropdown
id: install
attributes:
label: How did you install it?
options:
- Desktop app (installer / AppImage)
- Docker image
- From source (uv sync)
validations:
required: true
- type: input
id: version
attributes:
label: Version
description: Settings → About (e.g. v0.3.5), or the Docker tag / git SHA.
placeholder: "v0.3.5"
validations:
required: true
- type: dropdown
id: device
attributes:
label: Compute device
options:
- NVIDIA GPU (CUDA)
- AMD GPU (ROCm)
- Apple Silicon (MPS)
- Intel GPU (XPU)
- CPU only
- Not sure
validations:
required: true
- type: input
id: engine
attributes:
label: Active TTS/ASR engine
description: Settings → Engines (e.g. omnivoice, cosyvoice, indextts2, whisperx).
placeholder: "omnivoice"
- type: textarea
id: logs
attributes:
label: Logs / diagnostic bundle
description: Drag the diagnostic bundle here, or paste relevant lines from **Settings → Logs**. Secrets are scrubbed automatically.
render: text
- type: textarea
id: extra
attributes:
label: Anything else?
description: Screenshots, the input that triggered it, RAM/VRAM, etc.
-11
View File
@@ -1,11 +0,0 @@
blank_issues_enabled: false
contact_links:
- name: 💬 Discord — questions & quick help
url: https://discord.gg/bzQavDfVV9
about: Usage questions, setup help, and chat. Faster than an issue for "how do I…".
- name: 🗣️ GitHub Discussions
url: https://github.com/debpalash/OmniVoice-Studio/discussions
about: Ideas, show-and-tell, and open-ended Q&A that isn't a bug or a specific feature ask.
- name: 🔒 Security vulnerability
url: https://github.com/debpalash/OmniVoice-Studio/security/policy
about: Please report security issues privately — do NOT open a public issue.
+23
View File
@@ -0,0 +1,23 @@
---
name: ✨ Feature Request
about: Suggest an idea for OmniVoice Studio
title: "[Feature] "
labels: ["enhancement"]
assignees: []
---
## Is your feature request related to a problem?
A clear description of what the problem is. Ex. "I'm always frustrated when..."
## Describe the solution you'd like
A clear description of what you want to happen.
## Describe alternatives you've considered
Any alternative solutions or features you've considered.
## Additional context
Add any other context, mockups, or screenshots about the feature request here.
@@ -1,50 +0,0 @@
name: ✨ Feature request
description: Suggest an improvement or a new capability.
title: "[Feature] "
labels: ["enhancement"]
body:
- type: checkboxes
id: preflight
attributes:
label: Before filing
options:
- label: I searched [existing issues](https://github.com/debpalash/OmniVoice-Studio/issues?q=is%3Aissue) and [discussions](https://github.com/debpalash/OmniVoice-Studio/discussions) for this idea.
required: true
- type: textarea
id: problem
attributes:
label: What problem does this solve?
description: The use case / friction this addresses ("When I … I can't …").
validations:
required: true
- type: textarea
id: solution
attributes:
label: Proposed solution
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives considered
- type: dropdown
id: area
attributes:
label: Area
options:
- Voice cloning
- Voice design
- Video dubbing
- Real-time dictation
- Audiobook / Stories (long-form)
- TTS/ASR engines
- Install / setup / packaging
- Other
validations:
required: true
- type: markdown
attributes:
value: |
> OmniVoice is **local-first** — features must work fully offline with no accounts,
API keys, or cloud calls, and behave identically on macOS/Windows/Linux. Proposals
that fit those constraints are easiest to land.
@@ -1,86 +0,0 @@
name: 🧩 Install / first-run problem
description: The app won't install, set up, download models, or reach a first working output.
title: "[Install] "
labels: ["install", "triage"]
body:
- type: markdown
attributes:
value: |
A first-run that *just works* is the whole point — sorry it didn't. Let's fix it.
If the app launched far enough to open Settings, **Settings → About → "Save diagnostic
bundle"** captures most of this; otherwise the fields below are enough.
- type: dropdown
id: stage
attributes:
label: Where did it fail?
options:
- App won't launch / blank or broken window
- Python / uv environment bootstrap
- Model download (HuggingFace)
- Engine install (CosyVoice / IndexTTS / MLX / etc.)
- First synthesis / dub never completes
- Other
validations:
required: true
- type: textarea
id: error
attributes:
label: The error
description: The exact message, traceback, or what you see on screen.
render: text
validations:
required: true
- type: dropdown
id: os
attributes:
label: Operating system
options:
- macOS (Apple Silicon)
- macOS (Intel)
- Windows (x64)
- Linux (AppImage)
- Linux (.deb)
- Linux (other / from source)
- Docker
validations:
required: true
- type: dropdown
id: install
attributes:
label: How are you installing it?
options:
- Desktop app (installer / AppImage)
- Docker image
- From source (uv sync)
validations:
required: true
- type: input
id: version
attributes:
label: Version
placeholder: "v0.3.5 (or installer build / git SHA)"
validations:
required: true
- type: dropdown
id: network
attributes:
label: Network conditions (model/dependency downloads)
description: Restricted networks are a known source of bootstrap failures (mirror fallback).
options:
- Normal / unrestricted
- Behind a corporate proxy / firewall
- Region with restricted access (e.g. China, Russia)
- Offline / air-gapped
- Not sure
validations:
required: true
- type: textarea
id: logs
attributes:
label: Logs / diagnostic bundle
description: Drag a diagnostic bundle, or paste the install/bootstrap log. Headless self-check — `python backend/main.py --diagnose`.
render: text
- type: textarea
id: tried
attributes:
label: What have you already tried?
+13 -8
View File
@@ -18,7 +18,7 @@
- [ ] 📝 Documentation
- [ ] 🧪 Tests
- [ ] 🔧 CI / Build
- [ ] 🚀 Release prep
- [ ] 🚀 Release prep (RC or final)
## Testing
@@ -33,12 +33,17 @@
- [ ] No local machine paths, logs, or personal env details in this PR
- [ ] Version files are in sync (if version bump): `pyproject.toml`, `package.json`, `tauri.conf.json`, `Cargo.toml`
- [ ] If this PR changes runtime behavior, the regression fixture at `tests/fixtures/omnivoice_data/` still loads green on the `smoke-matrix` CI job (macOS + Windows + Linux)
- [ ] If this is part of a release, I've read the "Release cadence" section below and confirmed this PR targets the right RC
## Release cadence
## Release cadence (read once per RC)
OmniVoice ships **continuous-to-main** — no release candidates, no soak windows.
Every merged PR is immediately part of the rolling preview (`main`, Docker
`:latest`, the desktop Preview channel). Versioned releases are tagged from
`main` when it's ready; `main` then bumps to the next patch automatically.
Users who want stability pin a release tag / Docker `:stable` / the desktop
Stable channel.
OmniVoice ships every minor on a **two-RC cadence**:
- `vX.Y.0-rc1` — cut from `main` once all GATE-* requirements pass; clean-VM exercise on 4 OSes (per `REL-01`)
- 48-hour soak (no new commits to release branch except fix-forward)
- `vX.Y.0` — promotion if rc1 is clean
If your PR touches install / bootstrap / CI, it MUST land before rc1 cut, not between rc1 and the promotion. During a soak, any merge needs explicit OK from the release captain.
## Screenshots
<!-- If applicable, add screenshots or recordings. -->
+2 -8
View File
@@ -92,10 +92,7 @@ jobs:
- name: Install frontend deps
working-directory: frontend
# --frozen-lockfile so a frontend/package.json change that forgets to
# regenerate the root bun.lock fails HERE (fast) instead of only in the
# Docker build (deploy/Dockerfile), which is what reddened main on #485.
run: bun install --frozen-lockfile
run: bun install
# checkJs is true in tsconfig for IDE feedback, but 947 pre-existing
# JS errors remain. Override to false in CI so only .ts files block.
@@ -177,10 +174,7 @@ jobs:
- name: Install frontend deps
working-directory: frontend
# --frozen-lockfile so a frontend/package.json change that forgets to
# regenerate the root bun.lock fails HERE (fast) instead of only in the
# Docker build (deploy/Dockerfile), which is what reddened main on #485.
run: bun install --frozen-lockfile
run: bun install
# tauri-build's setup hook reads tauri.conf.json's `frontendDist`
# ("../dist"), which only exists after a frontend build. Without this,
+17 -62
View File
@@ -5,21 +5,14 @@
# - push to main branch → :main, :sha- (rolling "edge" build)
# - workflow_dispatch → :sha- only (ad-hoc test build)
#
# Tag ↔ image mapping (versioning hard rule, owner-set 2026-06-11:
# :latest IS the preview channel; stable users pin :stable or a version tag)
# :latest — rolling preview: latest commit on main (always last release + 1 dev)
# :main — alias of the same rolling main build (kept for back-compat)
# :stable — most recent versioned release (set on every v* tag push)
# :0.3.6 — exact version from the git tag
# Tag ↔ image mapping
# :latest — always the most recent versioned release (set on every v* tag push)
# :0.3.0 — exact version from the git tag
# :0.3 — major.minor floating tag (updated on every patch within the minor)
# :main — latest commit on main; may be ahead of the last tagged release
# :sha-xxxx — specific commit SHA; produced by workflow_dispatch
#
# Images land at: ghcr.io/debpalash/omnivoice-studio AND docker.io/palashdeb/omnivoice-studio
# (Docker Hub push gated on the DOCKERHUB_USERNAME/DOCKERHUB_TOKEN secrets;
# if unset the build still pushes to GHCR.)
#
# On main pushes the Docker Hub repository overview is also synced from
# deploy/dockerhub-overview.md (source of truth for the hub.docker.com page).
# Images land at: ghcr.io/debpalash/omnivoice-studio
#
# NOTE: the Docker image is the headless web-server build of OmniVoice (FastAPI
# backend + pre-built React frontend served over HTTP). The Tauri desktop
@@ -41,7 +34,6 @@ permissions:
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
DOCKERHUB_IMAGE: palashdeb/omnivoice-studio
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
jobs:
@@ -64,43 +56,28 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
# Docker Hub login — only when the secret is present, so forks / runs
# without the credential still publish to GHCR.
- name: Check Docker Hub credentials
id: dockerhub
run: echo "enabled=${{ secrets.DOCKERHUB_TOKEN != '' }}" >> "$GITHUB_OUTPUT"
- name: Log in to Docker Hub
if: steps.dockerhub.outputs.enabled == 'true'
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
# Tag strategy (`:sha-<short>` is emitted on every trigger):
# v0.3.6 tag push → :0.3.6, :0.3, :stable, :sha-
# main branch push → :latest, :main, :sha-
# v0.3.0 tag push → :0.3.0, :0.3, :latest, :sha-
# main branch push → :main, :sha-
# workflow_dispatch → :sha- only
#
# All mutable-tag rules stay gated on `github.event_name == 'push'` so a
# manual workflow_dispatch can only ever produce a throwaway `:sha-` tag
# (the stale-:latest fix from #249/#251). :stable excludes prerelease
# tags (those contain a `-`) so a prerelease can't clobber it.
# Fix for stale :latest (issues #249, #251):
# The previous rule used `enable={{is_default_branch}}`, which evaluates
# to false on tag pushes (detached HEAD) — so :latest was never updated
# when a release tag was pushed. The version / :latest / :main rules are
# gated on `github.event_name == 'push'` so a manual workflow_dispatch can
# only ever produce a throwaway `:sha-` tag (never republish a mutable
# tag), and :latest additionally excludes prerelease tags (those contain a
# `-`, e.g. v1.0.0-rc.1) so a prerelease can't clobber :latest.
- name: Extract metadata (tags, labels)
id: meta
uses: docker/metadata-action@v5
with:
# Same tag set applied to both registries. The Docker Hub line is
# blank when the secret is unset, so metadata-action emits GHCR-only
# tags in that case.
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
${{ steps.dockerhub.outputs.enabled == 'true' && env.DOCKERHUB_IMAGE || '' }}
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=semver,pattern={{version}},enable=${{ github.event_name == 'push' }}
type=semver,pattern={{major}}.{{minor}},enable=${{ github.event_name == 'push' }}
type=raw,value=stable,enable=${{ github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-') }}
type=raw,value=latest,enable=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
type=raw,value=latest,enable=${{ github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-') }}
type=raw,value=main,enable=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
type=sha,prefix=sha-,format=short
@@ -114,25 +91,3 @@ jobs:
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
# Sync the Docker Hub repository overview from deploy/dockerhub-overview.md.
# Only on main pushes (the overview tracks the rolling preview) and only
# when Docker Hub creds are present, mirroring the push gating above.
#
# continue-on-error: the overview text is cosmetic, and the description
# PATCH 403s unless DOCKERHUB_TOKEN carries description-edit scope (many
# fine-grained Docker Hub tokens that can push still can't edit the
# description). The image build+push is what matters — a creds-scope
# mismatch on this cosmetic step must not fail the whole Docker run. To
# actually sync the overview, use a token with read/write (incl.
# description) scope, or the account password.
- name: Update Docker Hub description
if: steps.dockerhub.outputs.enabled == 'true' && github.event_name == 'push' && github.ref == 'refs/heads/main'
continue-on-error: true
uses: peter-evans/dockerhub-description@v4
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
repository: ${{ env.DOCKERHUB_IMAGE }}
short-description: "Local ElevenLabs alternative: voice cloning, design & video dubbing in 646 languages. No API keys."
readme-filepath: ./deploy/dockerhub-overview.md
-97
View File
@@ -1,97 +0,0 @@
# Docs drift — daily inventory-vs-docs check with a single rolling issue.
#
# docs/features.yaml is the canonical inventory; scripts/check-docs-drift.py
# diffs it against README.md, docs/, and the engine registries. On drift the
# job updates (or creates) ONE issue labeled `docs-drift` in place — no issue
# spam — and closes it automatically when the check is clean again.
#
# Companion to the PR-gating validate-install-docs.py step in ci.yml.
# Spec: docs/competitive-analysis.md Spec 9a / parity program Wave 0.1.
# Rolling-issue pattern adapted from Patter (MIT).
name: docs-drift
on:
schedule:
# Daily 03:30 UTC — after most merges, before EU morning triage.
- cron: "30 3 * * *"
workflow_dispatch:
permissions:
contents: read
issues: write
jobs:
drift:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install checker deps
run: pip install "pyyaml>=6"
- name: Check inventory vs README/docs/registries
id: drift
continue-on-error: true
run: python scripts/check-docs-drift.py --output drift-report.md
- name: Update rolling docs-drift issue
uses: actions/github-script@v7
env:
DRIFT_OUTCOME: ${{ steps.drift.outcome }}
with:
script: |
const fs = require('fs');
const drifted = process.env.DRIFT_OUTCOME === 'failure';
const { owner, repo } = context.repo;
const label = 'docs-drift';
const open = await github.rest.issues.listForRepo({
owner, repo, state: 'open', labels: label, per_page: 5,
});
if (drifted) {
let body = '';
try {
body = fs.readFileSync('drift-report.md', 'utf8');
} catch {
body = '# Docs drift report\n\nThe checker failed before writing a report — see the workflow run logs.';
}
body += `\n\n---\n_Last checked by [run ${context.runId}](https://github.com/${owner}/${repo}/actions/runs/${context.runId})._\n`;
if (open.data.length > 0) {
await github.rest.issues.update({
owner, repo, issue_number: open.data[0].number, body,
});
core.info(`Updated rolling issue #${open.data[0].number}`);
} else {
const created = await github.rest.issues.create({
owner, repo,
title: 'docs-drift: feature inventory vs docs mismatch',
body,
labels: [label, 'documentation'],
});
core.info(`Created rolling issue #${created.data.number}`);
}
} else {
for (const issue of open.data) {
await github.rest.issues.createComment({
owner, repo, issue_number: issue.number,
body: 'Drift resolved — nightly check is clean again. Closing automatically.',
});
await github.rest.issues.update({
owner, repo, issue_number: issue.number, state: 'closed',
});
core.info(`Closed rolling issue #${issue.number}`);
}
}
- name: Surface drift as a failed run
if: steps.drift.outcome == 'failure'
run: |
echo "Docs drift detected — see the rolling docs-drift issue."
exit 1
-55
View File
@@ -1,55 +0,0 @@
# LLM-judge evals — semantic quality suites, NEVER a gate.
#
# Hard rule (parity program Wave 0.3 / competitive-analysis Spec 9b): LLM
# judges never gate CI. This workflow is scheduled + manual only, the eval
# step is continue-on-error, and the JSON report is the deliverable
# (uploaded as an artifact). Deterministic probe judges in ci.yml remain
# the only gates.
#
# On the hosted runner there is no local LLM endpoint, so the run usually
# reports "skipped — no LLM backend configured"; the workflow exists so the
# suites run anywhere a TRANSLATE_BASE_URL secret/endpoint is provided
# (e.g. a self-hosted runner with Ollama).
name: evals
on:
schedule:
# Weekly, Sundays 04:00 UTC.
- cron: "0 4 * * 0"
workflow_dispatch:
permissions:
contents: read
jobs:
evals:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- uses: astral-sh/setup-uv@v3
with:
enable-cache: true
cache-dependency-glob: "uv.lock"
- name: Install deps
run: uv sync
- name: Run eval suites (non-gating)
continue-on-error: true
env:
TRANSLATE_BASE_URL: ${{ secrets.EVALS_LLM_BASE_URL }}
TRANSLATE_API_KEY: ${{ secrets.EVALS_LLM_API_KEY }}
run: uv run python tests/evals/run_evals.py --output eval-report.json
- name: Upload report artifact
uses: actions/upload-artifact@v4
with:
name: eval-report
path: eval-report.json
if-no-files-found: warn
+9 -110
View File
@@ -133,28 +133,9 @@ jobs:
rust_target: aarch64-apple-darwin
bundles: "app,dmg,updater"
# macOS Intel (#279): reinstated. The earlier "Rosetta 2 runs the
# ARM build" rationale for dropping it was backwards — Rosetta only
# translates x86_64→arm64, so Intel Macs (supported through macOS
# Sequoia) simply cannot run the aarch64 bundle and had NO
# installable artifact. Runner: `macos-15-intel`, GitHub's
# designated migration target after macos-13 retired (Dec 2025);
# it's a standard (public-repo-free) image supported through
# August 2027 — the last x86_64 image Actions will offer. Building
# natively (not cross-compiling from the arm64 leg) keeps the
# per-TRIPLE uv/ffmpeg sidecar fetches, the DMG installer smoke,
# and the ad-hoc signing verification (scripts/
# verify-macos-signing.sh, PR #290) all exercising the real
# x86_64 artifact on real Intel hardware. The macos-13 queue
# backlog that motivated the original drop is contained by
# fail-fast:false — a slow Intel leg can delay the release run but
# can't fail the other targets.
- os: macos-15-intel
arch: x86_64-apple-darwin
label: "macOS Intel"
rust_target: x86_64-apple-darwin
bundles: "app,dmg,updater"
# macOS Intel dropped: Apple shipped the last Intel Mac in 2023 and
# Rosetta 2 runs the ARM build natively. macos-13 runner backlog
# was also blocking every release tag for ~10 min.
# Windows: force MSI bundling via --bundles. NSIS fails at makensis
# because our PyInstaller payload approaches its ~2 GB stub limit.
- os: windows-2022
@@ -310,10 +291,7 @@ jobs:
case "$TRIPLE" in
aarch64-apple-darwin|x86_64-apple-darwin)
# evermeet.cx ships each binary as a separate .zip containing
# a single x86_64 Mach-O executable — natively correct on the
# Intel leg, and runs via Rosetta 2 on the arm64 leg. Both
# darwin TRIPLEs therefore bundle the same payload; only the
# sidecar filename suffix differs.
# a single x86_64 Mach-O executable (runs via Rosetta on arm64).
for TOOL in ffmpeg ffprobe; do
if [ "$TOOL" = "ffmpeg" ]; then
URL="https://evermeet.cx/ffmpeg/getrelease/zip"
@@ -420,12 +398,11 @@ jobs:
# always reported the static 0.3.0 never looked "newer", so no update was
# ever delivered). Ephemeral, CI-only — never committed. Tauri reads the
# bundle + updater version from tauri.conf.json, so rewriting it here
# stamps the artifacts + latest.json. Under the versioning hard rule
# (owner-set 2026-06-11) main is always last-release + 1, so BASE-N is a
# prerelease of the NEXT version and semver-sorts ABOVE the last stable
# (0.3.6-N > 0.3.5) — preview users naturally upgrade past stable, and
# the Windows MSI ProductVersion (which strips the prerelease → 0.3.6)
# is also correctly above the last stable.
# stamps the artifacts + latest.json. `0.3.0-preview.N` is a prerelease of
# the current target, so previews converge to stable when 0.3.0 ships
# (0.3.0 > 0.3.0-preview.N). NOTE: the Windows MSI ProductVersion strips
# the prerelease (→ 0.3.0), a wrinkle to verify for win preview→preview
# upgrades; mac/linux replace the bundle wholesale and are unaffected.
- name: Stamp preview version
if: github.event_name == 'workflow_dispatch' && inputs.publish_preview
shell: bash
@@ -454,12 +431,6 @@ jobs:
# only on the opt-in stable path, leaving them ABSENT (not "") on
# preview/unsigned paths so Tauri's bundler skips cert import. A static
# env: here would always set them to "" and break the mac build.
# Unsigned paths still get a VALID ad-hoc seal from tauri.conf.json
# (bundle.macOS.signingIdentity = "-"), so a downloaded build shows the
# GUI-bypassable "unidentified developer" prompt (right-click → Open /
# Settings → "Open Anyway") instead of the un-bypassable "damaged"
# error. On the signed path APPLE_SIGNING_IDENTITY (env) overrides the
# "-" default; once notarized, Gatekeeper accepts it with no prompt.
# GH runners disable FUSE, so linuxdeploy's AppImage can't mount
# itself at bundle time. This env tells linuxdeploy to extract-and-run
# instead, which works without FUSE.
@@ -510,35 +481,6 @@ jobs:
echo "OK — bundle has shell + uv + backend resources"
hdiutil detach "$MOUNT" || true
# ── Signing / Gatekeeper / notarization verification ──────────────
# Runs codesign --verify, spctl (Gatekeeper), nested-binary, and
# stapler checks against the built .app (see docs/macos-signing-verification.md).
# STRICT (--require-signed) only on the opt-in signed stable path — same
# condition as "Configure Apple signing" above — so a failed or missing
# signature/notarization FAILS the job and STOPS the release instead of
# publishing an unsigned artifact. On every other (unsigned dev/preview)
# path it runs report-only and never breaks the build.
- name: Verify macOS signing
if: runner.os == 'macOS'
shell: bash
env:
STRICT: ${{ (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && vars.MACOS_SIGNING_ENABLED == 'true') && '1' || '0' }}
APPLE_ID: ${{ secrets.APPLE_ID }}
APPLE_PASSWORD: ${{ secrets.APPLE_PASSWORD }}
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
run: |
set -uo pipefail
APP=$(find "frontend/src-tauri/target/${{ matrix.rust_target }}/release/bundle/macos" -maxdepth 1 -name '*.app' | head -1)
[ -n "$APP" ] || { echo "FAIL — no .app found to verify"; exit 1; }
MODE=""
if [ "$STRICT" = "1" ]; then
MODE="--require-signed"
echo "Signed stable release → STRICT verification (release stops on failure)."
else
echo "Unsigned dev/preview path → report-only verification."
fi
bash scripts/verify-macos-signing.sh "$APP" $MODE
- name: Installer smoke (Windows)
if: runner.os == 'Windows'
timeout-minutes: 5
@@ -695,46 +637,3 @@ jobs:
} > /tmp/preview-notes.md
gh release edit preview --repo "$REPO" --notes-file /tmp/preview-notes.md
echo "Applied auto-generated release notes + contributors to the preview release."
# ── Post-release version bump (versioning hard rule, owner-set 2026-06-11) ──
# main is always last-release + 1 patch. The moment a stable v* tag is
# released, bump the three version sources on main to the next patch so every
# PR and preview build identifies as the next version. Pushes directly to
# main with the workflow token (a metadata-only commit; CI runs on PRs).
version-bump:
if: github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-')
runs-on: ubuntu-22.04
permissions:
contents: write
steps:
- uses: actions/checkout@v4
with:
ref: main
fetch-depth: 0
- name: Bump main to released version + 1 patch
shell: bash
run: |
set -euo pipefail
RELEASED="${GITHUB_REF_NAME#v}"
IFS=. read -r MAJ MIN PAT <<< "$RELEASED"
NEXT="$MAJ.$MIN.$((PAT + 1))"
CURRENT=$(jq -r .version frontend/src-tauri/tauri.conf.json)
if [ "$(printf '%s\n' "$NEXT" "$CURRENT" | sort -V | tail -1)" = "$CURRENT" ] && [ "$NEXT" != "$CURRENT" ]; then
echo "main is already at $CURRENT (>= $NEXT) — nothing to bump"; exit 0
fi
tmp=$(mktemp)
jq --arg v "$NEXT" '.version = $v' frontend/src-tauri/tauri.conf.json > "$tmp"
mv "$tmp" frontend/src-tauri/tauri.conf.json
# frontend/package.json drives __APP_VERSION__ (vite.config.js) — the
# first-run footer + every auto bug report. Keep it in lockstep too,
# set absolutely (jq) so any prior drift self-heals. (#248-sweep finding)
tmp=$(mktemp)
jq --arg v "$NEXT" '.version = $v' frontend/package.json > "$tmp"
mv "$tmp" frontend/package.json
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" frontend/src-tauri/Cargo.toml
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" pyproject.toml
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add frontend/src-tauri/tauri.conf.json frontend/package.json frontend/src-tauri/Cargo.toml pyproject.toml
git commit -m "chore(version): main -> $NEXT after $GITHUB_REF_NAME release"
git push origin main
+2 -6
View File
@@ -32,13 +32,9 @@ env:
permissions:
contents: read
# PR branches: a new push cancels the superseded scan (no wasted runners).
# main: every commit keeps its own group, so nothing is cancelled — a merge
# train used to leave a permanent red ✗ ("cancelled") on every intermediate
# commit in the history view even though nothing failed.
concurrency:
group: security-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || 'branch' }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
group: security-${{ github.ref }}
cancel-in-progress: true
jobs:
# ── Secret scanning (gating) ─────────────────────────────────────────────
+2 -9
View File
@@ -1,17 +1,10 @@
# SPIKE-02: Adopt `ModelsLab/omnivoice-singing` as singing variant of the existing engine
**Status:** ⚠️ **SUPERSEDED (2026-06-14)** by [`specs/006-dubbing-singing-mode/`](../../specs/006-dubbing-singing-mode/spec.md)
**Date:** 2026-05-18 (superseded 2026-06-14)
**Status:** Proposed (research-supported) — awaiting Phase 2 SubprocessBackend merge
**Date:** 2026-05-18
**Decision-makers:** [maintainer]
**Related:** ROADMAP Phase 4; REQUIREMENTS SING-01..05; `.planning/phases/04-adaptive-specialty-engines-spike-first/04-RESEARCH.md`
> **Superseded:** This chose `ModelsLab/omnivoice-singing` for singing, but that
> model has **no melody (F0/MIDI) conditioning** — it sings its own melody and
> cannot follow the *source song* a dub must preserve. SoulX-Singer (arXiv
> 2602.07803, published after this decision) provides F0/MIDI conditioning and is
> selected in plan-06. This ADR stays valid only if reframed as an
> expressive-TTS styling toggle, not melody-matched dubbing.
## Context
`ModelsLab/omnivoice-singing` (HuggingFace, 1,053 downloads/month, verified 2026-05-18) is a finetune of `k2-fsa/OmniVoice` — same Apache-2.0 license, same Qwen3-0.6B backbone, same Higgs Audio v2 codec at 24 kHz mono, same `omnivoice` PyPI library (0.1.5, 2026-04-28) already shipping in OmniVoice Studio v0.2.7. Trained on additional singing + emotion-tagged data and activated by a `[singing]` text control tag at generation time.
@@ -1,396 +0,0 @@
---
phase: 260613-fdl
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- pyproject.toml
- backend/api/routers/setup/download.py
- backend/api/routers/setup/models.py
- backend/utils/hf_progress.py
- backend/utils/download_aggregator.py # NEW
- backend/services/segmented_download.py # NEW
- backend/api/routers/system.py
- frontend/src/pages/Settings.jsx
- frontend/src/api/setup.ts
- docs/downloading-models.md # NEW (docs-sync rule)
- tests/backend/setup/test_download_preflight.py # NEW
- tests/backend/services/test_segmented_download.py # NEW
autonomous: true
requirements:
# ── Wave 0 — Spike / gate ───────────────────────────────────────────────────
- FDL-00 # Classify all catalog repos Xet-backed vs legacy-LFS; the result sizes Wave 3
# ── Wave 1 — Maximize + guarantee the Xet fast path (default, no new deps) ───
- FDL-01 # Explicitly pin huggingface_hub>=1.7 + hf-xet in pyproject (today transitive/unpinned)
- FDL-02 # Drive snapshot_download with explicit max_workers + tqdm_class + endpoint (not implicit monkeypatch)
- FDL-03 # /system/info reports fast_download {xet_enabled, xet_version, high_performance}; logged at startup
- FDL-04 # Opt-in HF_XET_HIGH_PERFORMANCE + HDD sequential-write toggles via prefs (env wins)
# ── Wave 2 — Accurate downloaded/remaining + speed (the user-visible win) ────
- FDL-05 # dry_run preflight -> emit install_plan {total_bytes, cached_bytes, to_download_bytes, n_files, n_cached}
- FDL-06 # Backend aggregate tracker -> single 'aggregate' event {bytes_done, total_bytes, rate, eta, files_done/total}
- FDL-07 # Frontend overall bar: speed + downloaded/remaining + ETA from aggregate; per-file detail collapsible; cached-skip shown
# ── Wave 3 — Opt-in IDM-style accelerator for legacy-LFS repos ───────────────
- FDL-08 # Custom httpx segmented downloader: parallel Range GETs, resume, auth-safe redirect, etag/sha verify, cancel (default OFF)
- FDL-09 # Dispatch: accelerator ON + repo is LFS (not Xet) -> segmented path; else xet. Same aggregate progress + weight validation
# ── Wave 4 — Opt-in mirror path + docs ───────────────────────────────────────
- FDL-10 # Opt-in HF_ENDPOINT mirror setting (prefs); documented as classic-LFS fallback (no Xet); pairs with FDL-08
- FDL-11 # Cancel-in-flight endpoint + cooldown interplay (composes with MM2-06 bounded cooldowns)
- FDL-12 # docs/downloading-models.md (speed, fast-download status, HDD/high-perf toggles, mirror/restricted-network) + README pointer
must_haves:
truths:
- "Xet is the default download backend and is provably engaged: /system/info reports fast_download.xet_enabled=true with the hf_xet version, and a Xet-backed repo downloads via parallel chunk range-gets (not single-stream LFS)."
- "Before any bytes flow, the UI shows an accurate denominator: total bytes to download, bytes already cached (skipped), and file count — sourced from snapshot_download(dry_run=True), not guessed from the first tqdm bar."
- "During a download the UI shows ONE overall progress bar with instantaneous speed (sampled over a window, not a single file's rate), bytes downloaded / bytes remaining, and ETA — accurate even while Xet fetches many chunks/files in parallel."
- "hf_transfer is NOT used or enabled anywhere (deprecated, breaks progress); the fast path is Xet only."
- "The custom segmented downloader is OPT-IN (default off), only engages for non-Xet/legacy-LFS repos, never forwards the HF Authorization header to the redirected CDN host, verifies the downloaded file against its expected size/etag before marking complete, resumes a partial .part file, and can be cancelled mid-flight."
- "Default download behavior is identical on macOS, Windows, Linux (Xet path, pure-Python). Every accelerator/mirror/high-perf knob is behind an explicit opt-in (Settings toggle or env var) per the cross-platform-parity strict rule — no bundled per-OS binary, no platform-divergent default."
- "No new on-disk model-state format; existing HF cache layout and already-installed models are untouched; the segmented downloader writes into the same HF cache blob/snapshot structure (or hands off to it) so a model it fetches is indistinguishable from one snapshot_download fetched."
- "uv run pytest tests/backend/setup/test_download_preflight.py tests/backend/services/test_segmented_download.py passes; existing download/install tests stay green."
- "pyproject pins huggingface_hub>=1.7 and hf-xet explicitly; uv.lock resolves with single versions (uv tree shows no duplicate huggingface_hub)."
artifacts:
- path: "backend/utils/download_aggregator.py"
provides: "Per-repo byte aggregator: sums bytes across parallel files/chunks, samples rate over a window, emits one 'aggregate' event"
contains: "class DownloadAggregator AND def snapshot"
- path: "backend/services/segmented_download.py"
provides: "Opt-in multi-connection Range downloader for legacy-LFS repos (auth-safe, resume, verify, cancel)"
contains: "async def segmented_download AND Range"
- path: "backend/api/routers/setup/download.py"
provides: "Driven snapshot_download (max_workers+tqdm_class+endpoint), dry_run preflight, dispatch to segmented path, cancel endpoint"
contains: "dry_run AND tqdm_class"
- path: "docs/downloading-models.md"
provides: "User docs for download speed, fast-download status, HDD/high-perf toggles, mirror/restricted-network"
contains: "Xet"
key_links:
- from: "install_model (download.py:122)"
to: "snapshot_download(dry_run=True) preflight"
via: "compute total/cached/remaining before the real download; emit 'install_plan'"
pattern: "dry_run\\s*=\\s*True"
- from: "snapshot_download / segmented_download byte updates"
to: "DownloadAggregator -> single 'aggregate' SSE event"
via: "tqdm_class forwards bytes into the aggregator; segmented path calls aggregator.add() directly"
pattern: "aggregate"
- from: "dispatch in install_model"
to: "segmented_download vs snapshot_download"
via: "prefs accelerator toggle AND repo-is-LFS classification (FDL-00 helper)"
pattern: "segmented_download"
- from: "system_info (system.py:245)"
to: "fast_download status block"
via: "probe hf_xet import + version + HF_XET_HIGH_PERFORMANCE"
pattern: "fast_download"
---
<objective>
Make model downloads as fast as possible AND show accurate speed / downloaded / remaining / ETA.
**Framing (validated by research — see 260613-fdl-RESEARCH below):** HuggingFace's **hf-xet** backend ALREADY implements the "IDM/uGet technique" — content-defined chunking, parallel byte-range fetches with adaptive concurrency, dedup, and automatic resume — and does it auth-safely. It ships by default in modern `huggingface_hub` and `hf_xet` is already installed here (huggingface_hub 1.7.2). HF closed the multi-connection-downloader feature request as "solved by Xet." So we do NOT build a custom segmented downloader as the default path; that would be redundant and would violate the cross-platform-parity rule.
What's actually missing:
1. **We don't drive Xet well.** `install_model` calls `snapshot_download(**dl_kwargs)` with no `max_workers`, no `tqdm_class`, no `dry_run`, and no explicit dependency pin — progress rides on a global tqdm monkeypatch.
2. **No pre-flight total**, so "downloaded/remaining" has no denominator until files appear, and aggregate speed is summed frontend-side from per-file events (inaccurate under parallel fetch).
3. **Legacy non-Xet (LFS) repos get zero intra-file parallelism** — this is the one place a real IDM-style multi-connection fetch still helps, so we add it as an OPT-IN accelerator.
Five waves, in order (each independently shippable, continuous-to-main per v0.3.0 cadence):
- **Wave 0 — Spike/gate (FDL-00):** classify every catalog repo Xet vs LFS. Sizes Wave 3's value; if ~all repos are Xet-backed, Wave 3 is low-priority polish.
- **Wave 1 — Maximize + guarantee Xet (FDL-01..04):** pin deps, drive snapshot_download explicitly, surface fast-download status, opt-in high-perf/HDD knobs. No new deps, all platforms.
- **Wave 2 — Accurate progress (FDL-05..07):** dry_run preflight + backend aggregate tracker + overall UI bar (speed/remaining/ETA). The biggest user-visible win.
- **Wave 3 — Opt-in segmented accelerator (FDL-08..09):** custom httpx Range downloader for LFS repos. Default OFF, opt-in toggle.
- **Wave 4 — Mirror path + docs (FDL-10..12):** opt-in HF_ENDPOINT, cancel endpoint, docs-sync.
Out of scope / explicitly rejected (call out, do NOT do):
- **hf_transfer / HF_HUB_ENABLE_HF_TRANSFER** — deprecated, breaks progress callbacks. Never enable.
- **Bundling aria2c** — per-OS GPLv2 binary + parity burden; the custom httpx path covers the same need without a binary.
- **Making the segmented downloader the default** — redundant vs Xet, violates parity rule. Always opt-in.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/STATE.md
@./CLAUDE.md
@.planning/quick/260613-fdl-fast-model-downloads/260613-fdl-RESEARCH.md
# Files under edit (read before editing)
@backend/api/routers/setup/download.py
@backend/api/routers/setup/models.py
@backend/utils/hf_progress.py
@backend/api/routers/system.py
@frontend/src/pages/Settings.jsx
# Reference only — patterns, do NOT modify
@backend/core/prefs.py
@frontend/src/api/setup.ts
@frontend/src/api/hooks.ts
<interfaces>
<!-- Verified during planning against the live env (huggingface_hub 1.7.2, hf_xet installed). -->
huggingface_hub 1.7.2 snapshot_download params (confirmed via inspect):
repo_id, repo_type, revision, cache_dir, local_dir, library_name, library_version,
user_agent, etag_timeout, force_download, token, local_files_only,
allow_patterns, ignore_patterns, max_workers, tqdm_class, headers, endpoint, dry_run
- dry_run=True -> returns per-file info incl. size + cached/not-cached (use for FDL-05 preflight).
- tqdm_class=<cls> -> drives the AGGREGATE bar; Xet feeds bytes into it (this is the xet-aware progress hook).
- max_workers -> parallel FILES (default 8); orthogonal to Xet intra-file chunk parallelism.
- endpoint -> per-call HF endpoint override (FDL-10 mirror, instead of process-wide HF_ENDPOINT).
backend/utils/hf_progress.py (existing):
- Monkeypatches huggingface_hub.utils.tqdm.tqdm -> TrackedTqdm (install() at startup).
- register_listener/unregister_listener; emit(event); current_repo_id contextvar stamps events.
- TrackedTqdm.update()/display() emit per-file {filename, downloaded, total, pct, rate, phase} throttled ~0.3s.
- GAP: per-file only, no aggregate, no preflight total. Wave 2 adds the aggregator on top (keep TrackedTqdm; feed it).
backend/api/routers/setup/download.py (existing):
- install_model (line 122): snapshot_download(**dl_kwargs) inside asyncio.to_thread; 5-retry backoff; heartbeat;
_validate_snapshot_has_weights (line 55); _install_cooldowns (line 27, see MM2-06 for bounding).
- SSE feed: GET /setup/download-stream (line 80) forwards hf_progress events.
backend/core/prefs.py:
- resolve(key, *, env=None, default=None) (line 75) — env wins, then store, then default. Use for all new toggles.
Xet env knobs (research): HF_XET_HIGH_PERFORMANCE=1 (opt-in max throughput; needs RAM/bandwidth),
HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1 (HDD), HF_XET_NUM_CONCURRENT_RANGE_GETS (default 16),
HF_XET_DATA_PROGRESS_UPDATE_INTERVAL (200ms). hf_xet is 64-bit only.
</interfaces>
</context>
<tasks>
<!-- ════════════ WAVE 0 — SPIKE / GATE ════════════ -->
<task type="auto">
<name>Task 0 (FDL-00): Classify catalog repos Xet vs LFS</name>
<files>.planning/quick/260613-fdl-fast-model-downloads/260613-fdl-SPIKE.md</files>
<action>
For every repo in backend/config/models.yaml (25 entries), determine whether it's Xet-backed or legacy Git-LFS. Use huggingface_hub: `HfApi().repo_info(repo_id, files_metadata=True)` and inspect each LFS blob for xet info, OR call the model-info endpoint and check the `xetEnabled`/blob `xet` field. For gated/unavailable repos, record "unknown (gated/offline)".
Write 260613-fdl-SPIKE.md: a table repo_id | role | backend (xet|lfs|unknown) | size, plus a one-line GO/LOW-PRIORITY verdict for Wave 3:
- If the majority of *user-facing default* models (OmniVoice TTS, the default ASR) are Xet-backed -> Wave 3 is LOW priority (xet already fast); still build it for the LFS long tail.
- If many defaults are still LFS -> Wave 3 is HIGH priority.
This is read-only network classification — do not download anything (use repo_info, not snapshot_download).
</action>
<verify>
<automated>test -f .planning/quick/260613-fast-model-downloads/260613-fdl-SPIKE.md || test -f .planning/quick/260613-fdl-fast-model-downloads/260613-fdl-SPIKE.md && echo "spike written"</automated>
</verify>
<done>SPIKE.md lists every catalog repo with its storage backend and a GO/LOW-PRIORITY verdict for Wave 3.</done>
</task>
<!-- ════════════ WAVE 1 — MAXIMIZE + GUARANTEE THE XET FAST PATH ════════════ -->
<task type="auto">
<name>Task 1 (FDL-01): Pin huggingface_hub + hf-xet explicitly</name>
<files>pyproject.toml</files>
<action>
Today huggingface_hub arrives transitively (1.7.2) and hf_xet is present but unpinned. Add explicit runtime pins so the fast path can never silently disappear on a resolve:
- huggingface_hub>=1.7 (keep compatible with transformers>=5.3.0 already in deps)
- hf-xet>=1.1 (the Xet backend; 64-bit only — fine for all OmniVoice targets)
Do NOT add hf_transfer. Run `uv sync` then `uv tree huggingface_hub` to confirm a single resolved version (no duplicate). If a transitive constraint conflicts, prefer the higher version and note it in the SUMMARY.
</action>
<verify>
<automated>grep -n "huggingface_hub\|hf-xet\|hf_xet\|hf-transfer\|hf_transfer" pyproject.toml</automated>
<automated>uv run python -c "import huggingface_hub,hf_xet; print('hub',huggingface_hub.__version__,'xet ok')"</automated>
</verify>
<done>pyproject pins huggingface_hub>=1.7 and hf-xet; no hf_transfer; uv resolves cleanly with one huggingface_hub.</done>
</task>
<task type="auto">
<name>Task 2 (FDL-02): Drive snapshot_download explicitly</name>
<files>backend/api/routers/setup/download.py</files>
<action>
In install_model's _do() (line ~148), build dl_kwargs with explicit, intentional args instead of the bare call:
- tqdm_class=<the TrackedTqdm class> so progress is deterministic and xet-aware rather than relying solely on the global monkeypatch. Expose TrackedTqdm from hf_progress (add a getter, e.g. hf_progress.tracked_tqdm_class()).
- max_workers: keep default 8 (don't crank — xet does intra-file parallelism; high max_workers multiplies buffer pressure). Make it prefs-overridable: prefs.resolve("download_max_workers", env="OMNIVOICE_DOWNLOAD_MAX_WORKERS", default=8).
- endpoint=prefs.resolve("hf_endpoint", env="HF_ENDPOINT", default=None) — wires FDL-10 mirror without process-wide env.
- Keep the existing 5-retry backoff, heartbeat, and _validate_snapshot_has_weights.
Do not remove the global monkeypatch (other libs — transformers/mlx_whisper — still rely on it); this task just makes the install path drive its own tqdm_class explicitly.
</action>
<verify>
<automated>grep -n "tqdm_class\|max_workers\|endpoint" backend/api/routers/setup/download.py</automated>
<automated>uv run pytest tests/ -k "download or install" -q 2>&amp;1 | tail -15</automated>
</verify>
<done>install_model drives snapshot_download with explicit tqdm_class + max_workers + endpoint; retry/validate intact; tests green.</done>
</task>
<task type="auto">
<name>Task 3 (FDL-03, FDL-04): fast_download status + opt-in xet knobs</name>
<files>backend/api/routers/system.py, backend/api/routers/setup/download.py</files>
<action>
- FDL-03: add a fast_download block to GET /system/info (system.py:245): {xet_enabled: bool, xet_version: str|None, high_performance: bool}. Probe by importing hf_xet (xet_enabled), reading its version, and reading the HF_XET_HIGH_PERFORMANCE env/pref. Must never throw (system_info is called on every Settings load). Log the same line once at startup ("fast download: Xet on (hf_xet X.Y), high_perf=...").
- FDL-04: opt-in knobs via prefs, applied at process/download setup (env wins):
high_performance = prefs.resolve("xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False)
hdd_sequential = prefs.resolve("xet_hdd_sequential_write", env="HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY", default=False)
When set, export the corresponding HF_XET_* env before the snapshot/segmented download runs. Both default OFF (high-perf can hurt low-RAM machines — surface that as a tooltip in Wave 2 UI).
</action>
<verify>
<automated>curl -s http://127.0.0.1:3900/system/info | python3 -c "import json,sys; print(json.load(sys.stdin).get('fast_download'))" 2>/dev/null || grep -n "fast_download" backend/api/routers/system.py</automated>
</verify>
<done>/system/info reports fast_download truthfully; high-perf + HDD knobs resolve via prefs with env precedence, default off; startup logs Xet status.</done>
</task>
<!-- ════════════ WAVE 2 — ACCURATE DOWNLOADED/REMAINING + SPEED ════════════ -->
<task type="auto">
<name>Task 4 (FDL-05): dry_run preflight -> install_plan event</name>
<files>backend/api/routers/setup/download.py</files>
<action>
Before the real download in install_model, run snapshot_download(repo_id, dry_run=True, endpoint=...) on the worker thread. From the returned per-file info compute: total_bytes, cached_bytes (files already present), to_download_bytes, n_files, n_cached. Emit a new phase event:
{repo_id, phase:"install_plan", total_bytes, cached_bytes, to_download_bytes, n_files, n_cached}
This gives the UI an accurate denominator and a "M GB already cached, N GB to download" line BEFORE bytes flow. Wrap dry_run in try/except — if it fails (older/gated repo), emit install_plan with totals=None and fall back to today's behavior (denominator fills in as files appear). dry_run must respect the 'resolving' heartbeat (it can take a couple seconds).
</action>
<verify>
<automated>grep -n "dry_run\|install_plan\|to_download_bytes" backend/api/routers/setup/download.py</automated>
<automated>uv run pytest tests/backend/setup/test_download_preflight.py -q 2>&amp;1 | tail -15</automated>
</verify>
<done>An install emits install_plan with accurate total/cached/remaining before download; dry_run failure degrades gracefully to old behavior.</done>
</task>
<task type="auto">
<name>Task 5 (FDL-06): Backend aggregate progress tracker</name>
<files>backend/utils/download_aggregator.py, backend/utils/hf_progress.py</files>
<action>
New backend/utils/download_aggregator.py: a per-repo DownloadAggregator that owns the TRUTH for overall progress, so the frontend stops summing potentially-misrouted per-file events.
- Seeded by the install_plan totals (total_bytes, n_files).
- add(filename, bytes_delta) / set_file(filename, downloaded, total): track bytes per file; bytes_done = sum.
- Rate: sampled over a sliding window (e.g. last ~5-10s of (t, bytes_done) samples), not a single tqdm bar's rate. eta = remaining / rate.
- snapshot() -> {repo_id, bytes_done, total_bytes, rate, eta_seconds, files_done, files_total, phase}.
- Emits one throttled (~0.3-0.5s) phase:"aggregate" event via hf_progress.emit().
Wire it: hf_progress's TrackedTqdm._emit_progress already has per-file (filename, downloaded, total) — also feed those into the active repo's aggregator (look up by current_repo_id). The segmented downloader (Wave 3) calls aggregator.add() directly. Keep the per-file events too (UI detail view) — aggregate is additive, not a replacement.
</action>
<verify>
<automated>uv run python -c "from utils.download_aggregator import DownloadAggregator as A; a=A('r',total_bytes=100,files_total=2); a.set_file('f1',50,50); a.set_file('f2',25,50); s=a.snapshot(); print(s['bytes_done'], s['total_bytes'])"</automated>
</verify>
<done>DownloadAggregator sums bytes across parallel files, samples rate over a window, emits a single 'aggregate' event; fed by both tqdm and the segmented path.</done>
</task>
<task type="auto">
<name>Task 6 (FDL-07): Frontend overall progress bar</name>
<files>frontend/src/pages/Settings.jsx, frontend/src/api/setup.ts</files>
<action>
- setup.ts: extend SetupProgressEvent phase union with "install_plan" | "aggregate" and their fields (total_bytes, cached_bytes, to_download_bytes, n_files, n_cached, bytes_done, rate, eta_seconds, files_done, files_total).
- Settings.jsx ModelStoreTab: when an aggregate event arrives for a repo, render ONE overall progress row: a bar (bytes_done/total_bytes), instantaneous speed (format rate as MB/s), "X.X GB of Y.Y GB" downloaded/remaining, and ETA (mm:ss from eta_seconds). Seed the denominator from install_plan (show "M GB cached, N GB to download" before bytes flow). Keep the existing per-file rows as a collapsible "details" section instead of the primary display. Show a small "⚡ fast download" badge when /system/info fast_download.xet_enabled is true.
- Prefer the backend aggregate's rate/eta over the frontend's own per-file ETA computation (Settings.jsx ~614-631) — replace that local ETA math with the aggregate fields; keep a fallback if no aggregate event has arrived yet.
</action>
<verify>
<automated>cd frontend && bun run typecheck 2>&amp;1 | tail -15</automated>
<automated>grep -n "aggregate\|install_plan\|eta_seconds\|fast download" frontend/src/pages/Settings.jsx frontend/src/api/setup.ts</automated>
</verify>
<done>UI shows one overall bar with live speed + downloaded/remaining + ETA from the aggregate event; per-file detail collapsible; fast-download badge; typecheck passes.</done>
</task>
<!-- ════════════ WAVE 3 — OPT-IN IDM-STYLE SEGMENTED ACCELERATOR (LFS REPOS) ════════════ -->
<task type="auto">
<name>Task 7 (FDL-08): Custom httpx segmented downloader</name>
<files>backend/services/segmented_download.py, tests/backend/services/test_segmented_download.py</files>
<action>
New backend/services/segmented_download.py — an OPT-IN multi-connection Range downloader for ONE file (the IDM/uGet technique) used only for legacy-LFS repos where Xet gives no intra-file parallelism. httpx is already a dep.
Contract (async def segmented_download(url, dest, *, token, expected_size, expected_etag=None, num_connections=8, chunk_aggregator=None, cancel_event=None)):
1. HEAD (or GET Range: bytes=0-0) the resolve URL to learn size + Accept-Ranges + the redirect target. If server doesn't honor Range (Accept-Ranges != bytes) -> fall back to a single streamed GET (still works, just not parallel).
2. AUTH SAFETY (critical): send Authorization: Bearer <token> ONLY to the huggingface.co host. When the resolve URL 302-redirects to the CDN (cloudfront/etc.), do NOT forward Authorization to the CDN host — the presigned URL already carries auth. Follow redirects manually so you control header propagation per-host.
3. Split expected_size into num_connections ranges; download each with Range: bytes=start-end concurrently (asyncio + httpx.AsyncClient). Write to dest+".part" at the right offsets (preallocate, or per-range temp files then concat).
4. RESUME: if dest+".part" exists with a sidecar manifest of completed ranges, skip completed ranges.
5. CANCEL: check cancel_event between chunks; on cancel, leave the .part for resume and raise CancelledError.
6. VERIFY: after assembly, check size == expected_size and (if given) sha256/etag matches; only then atomically rename .part -> dest. On mismatch, raise (caller's retry/validate handles it).
7. PROGRESS: call chunk_aggregator.add(filename, bytes_delta) as ranges complete bytes (feeds DownloadAggregator).
Tests (use a local mock HTTP server / httpx MockTransport): honors Range + parallel assembly == single-GET bytes; falls back when Accept-Ranges absent; does NOT send Authorization to a different host on redirect; resumes from a partial .part; cancels and leaves resumable state; size/etag mismatch raises.
</action>
<verify>
<automated>uv run pytest tests/backend/services/test_segmented_download.py -q 2>&amp;1 | tail -20</automated>
<automated>grep -n "Authorization\|Range\|cancel_event\|expected_size" backend/services/segmented_download.py</automated>
</verify>
<done>segmented_download fetches a file via parallel ranges, is auth-safe across the CDN redirect, resumes, cancels, and verifies size/etag before commit; all tests pass.</done>
</task>
<task type="auto">
<name>Task 8 (FDL-09): Dispatch — accelerator for LFS repos only</name>
<files>backend/api/routers/setup/download.py, backend/api/routers/setup/models.py</files>
<action>
- models.py: add a small helper is_xet_backed(repo_id) -> bool|None (reuse FDL-00's classification approach; cache result). Used to decide the path.
- download.py install_model dispatch:
accelerator_on = prefs.resolve("segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=False)
if accelerator_on and is_xet_backed(repo_id) is False:
-> resolve each LFS file's URL via hf_hub_url + HfApi file metadata, download via segmented_download into the HF cache layout (or download to a temp dir then place via the cache API so the result is a normal cache entry), feeding the same DownloadAggregator. Run _validate_snapshot_has_weights at the end.
else:
-> existing snapshot_download path (xet).
IMPORTANT: the segmented result MUST land in the same HF cache structure so /models install-state, delete, and is_cached() all keep working (truth: "indistinguishable from snapshot_download"). If matching the blob/snapshot symlink layout is too fiddly, the safe fallback is: segmented-download to a temp file, then hand the bytes to huggingface_hub so it finalizes the cache entry. Document the chosen approach in SUMMARY.
Default OFF -> zero behavior change unless the user opts in.
</action>
<verify>
<automated>grep -n "segmented_downloader\|is_xet_backed\|segmented_download" backend/api/routers/setup/download.py backend/api/routers/setup/models.py</automated>
<automated>uv run pytest tests/ -k "download or install or model" -q 2>&amp;1 | tail -20</automated>
</verify>
<done>With the toggle ON, LFS repos download via the segmented path into the normal HF cache; Xet repos and the default (toggle OFF) use snapshot_download; install-state/delete unaffected.</done>
</task>
<!-- ════════════ WAVE 4 — MIRROR PATH + CANCEL + DOCS ════════════ -->
<task type="auto">
<name>Task 9 (FDL-10, FDL-11): Mirror opt-in + cancel endpoint</name>
<files>backend/api/routers/setup/download.py</files>
<action>
- FDL-10: the endpoint= wiring from Task 2 already reads prefs hf_endpoint. Surface it as a setting and document (Task 10) that a mirror routes through the CLASSIC LFS path (no Xet) — so it pairs naturally with the FDL-08 segmented accelerator for speed on mirrors. No process-wide HF_ENDPOINT mutation; per-call endpoint only.
- FDL-11: add POST /models/install/cancel {repo_id} that sets the repo's cancel_event (segmented path) and, for the snapshot path, best-effort marks the install cancelled (snapshot_download isn't trivially cancellable mid-file — at minimum stop retries and emit install_cancelled). Compose with MM2-06: on success OR cancel, clear the _install_cooldowns entry so a cancelled download isn't rate-limited. Emit phase:"install_cancelled".
</action>
<verify>
<automated>grep -n "install/cancel\|cancel_event\|install_cancelled\|hf_endpoint" backend/api/routers/setup/download.py</automated>
</verify>
<done>Per-call mirror endpoint wired (opt-in); cancel endpoint stops the segmented path and clears cooldown; emits install_cancelled.</done>
</task>
<task type="auto">
<name>Task 10 (FDL-12): Docs — downloading-models.md + README pointer</name>
<files>docs/downloading-models.md, README.md</files>
<action>
Per the docs-sync hard rule, document the user-facing surface introduced here:
- How fast downloads work (Xet on by default; what the ⚡ badge means; how to check via Settings/system info).
- Advanced toggles: high-performance mode (warn: needs RAM/bandwidth, can hurt low-RAM machines), HDD sequential-write, max workers, segmented accelerator (opt-in, for legacy-LFS repos), and the mirror/restricted-network HF_ENDPOINT setting (note: mirror = classic LFS, no Xet; pair with the accelerator).
- A short troubleshooting section (slow downloads, stuck at resolving, restricted networks/China).
Add a one-line pointer from README.md to docs/downloading-models.md. Do NOT enable any opt-in by default in docs examples.
</action>
<verify>
<automated>test -f docs/downloading-models.md && grep -n "Xet\|HF_ENDPOINT\|high-performance\|segmented" docs/downloading-models.md | head</automated>
<automated>grep -n "downloading-models" README.md</automated>
</verify>
<done>docs/downloading-models.md covers speed, status, all opt-in knobs, mirror/restricted-network, troubleshooting; README links it; no opt-in shown as default.</done>
</task>
</tasks>
<verification>
Gate per wave; full set before the last PR:
1. `uv run pytest tests/backend/setup/test_download_preflight.py tests/backend/services/test_segmented_download.py tests/ -k "download or install or model" -q` — green.
2. Live smoke (backend running): an install emits install_plan (accurate total/cached/remaining) THEN aggregate events with rising bytes_done + a non-zero rate + decreasing ETA; on completion bytes_done == total_bytes.
3. /system/info reports fast_download.xet_enabled=true with a version.
4. Auth-safety unit test proves Authorization is NOT sent to a non-huggingface.co host on redirect.
5. Default-off proof: with no opt-in set, an install uses snapshot_download (xet) — `OMNIVOICE_SEGMENTED_DOWNLOAD` unset means the segmented path is never taken.
6. `uv tree huggingface_hub` shows one version; no hf_transfer anywhere (`grep -ri hf_transfer backend/` is empty).
7. `cd frontend && bun run typecheck` passes.
8. Cross-platform parity: the default path (Xet, pure-Python) is identical on all 3 OSes; every accelerator/mirror/high-perf knob is opt-in (Settings/env). No bundled binary added.
</verification>
<success_criteria>
- Fast: Xet is pinned, engaged, and driven with explicit args; high-perf/HDD knobs available opt-in; legacy-LFS repos can use the opt-in segmented accelerator for real multi-connection speed.
- Accurate: UI shows pre-flight total/cached/remaining, then one overall bar with live speed + downloaded/remaining + ETA sourced from a backend aggregate (not frontend guesswork).
- Safe & compatible: no hf_transfer; segmented downloader is opt-in, auth-safe, resumable, verified, cancellable, and lands in the normal HF cache; default behavior identical on all 3 OSes; no new on-disk model state; existing installs untouched.
- All listed tests + typecheck pass; docs updated in the same PR (docs-sync rule).
</success_criteria>
<risks>
- **Segmented downloader auth leak (FDL-08) — highest risk.** Forwarding the HF Authorization header to the CDN host on redirect would leak the token. Mitigation: manual redirect handling, per-host header allow-list (Authorization only to huggingface.co), and a dedicated unit test asserting no Authorization on the CDN hop. This is a must-have truth, not optional.
- **Cache-layout mismatch (FDL-09).** If the segmented path writes files outside the HF cache blob/snapshot structure, /models install-state + delete + is_cached() break. Mitigation: prefer the temp-file-then-hand-to-huggingface_hub finalization approach over hand-rolling the symlink/blob layout; assert is_cached(repo_id) is true after a segmented install in a test.
- **dry_run cost/availability (FDL-05).** dry_run adds a metadata round-trip and may not exist for gated/older repos. Mitigation: try/except -> totals=None fallback to current fill-in-as-you-go behavior; keep the resolving heartbeat so the UI isn't blank during preflight.
- **Aggregate vs per-file double-count (FDL-06).** Feeding both tqdm per-file events and the aggregator risks the UI showing two competing numbers. Mitigation: aggregate is the single source of truth for the overall bar; per-file events only drive the collapsible detail view; the frontend's old per-file ETA math is removed (Task 6).
- **High-performance mode hurting low-RAM machines (FDL-04).** HF_XET_HIGH_PERFORMANCE can need ~tens of GB RAM. Mitigation: default OFF, opt-in only, tooltip warning in the UI.
- **Mirror + Xet confusion (FDL-10).** Users may expect Xet speed through a mirror; mirrors fall back to classic LFS. Mitigation: document explicitly; that's exactly why the segmented accelerator pairs with the mirror path.
- **Scope: do not let the segmented path become default.** It's opt-in for LFS repos only. Xet stays the default; making it default would regress dedup + violate the parity rule.
</risks>
<output>
Write 260613-fdl-SPIKE.md (Task 0) and 260613-fdl-SUMMARY.md when done. SUMMARY must record: the Xet-vs-LFS catalog breakdown and how it changed Wave 3 priority; the cache-finalization approach chosen for the segmented path (and the is_cached-after-segmented test result); the exact new SSE event shapes (install_plan, aggregate); which opt-in prefs keys + env vars were added; and the auth-safety test output. Note any "use judgment" decision an executor made.
Docs-sync (CLAUDE.md hard rule): docs/downloading-models.md + README pointer ship in the SAME PR as the user-facing toggles (Task 10). If the Settings UI gains the new toggles, the docs describing them land together.
</output>
@@ -1,43 +0,0 @@
# RESEARCH — Fast HuggingFace model downloads (2026)
**Date:** 2026-06-13 · **For:** 260613-fdl-PLAN.md
## Bottom line
As of mid-2026 the fast path is **hf-xet, on by default** in modern `huggingface_hub`. Xet is itself a chunk-level, content-defined, massively-parallel downloader with adaptive concurrency — it **is** the "IDM/uGet-style segmented download," done for you and dedup-aware. `hf_transfer` is **deprecated**. Rolling your own segmented downloader or bridging to aria2 is **not worth it as a default**; the only thing we must build is (a) better driving + progress UI and (b) an **opt-in** segmented path for the legacy-LFS long tail (repos Xet doesn't back).
Installed in this repo: `huggingface_hub 1.7.2`, `hf_xet` present. `snapshot_download` here supports `max_workers`, `tqdm_class`, `endpoint`, `dry_run` (confirmed via inspect).
## 1. hf-xet — USE (default, no action needed beyond pinning)
Content-defined chunks grouped into blocks ("xorbs") in a content-addressable store; download = send file SHA256 → get reconstruction metadata + presigned URLs → fetch needed xorb ranges **in parallel** → reassemble; already-present chunks skipped (dedup). Auto-used by `snapshot_download`/`hf_hub_download` for Xet-backed repos since huggingface_hub 0.32. 23× over Git-LFS, up to ~1 GB/s.
Knobs (defaults already tuned): `HF_XET_NUM_CONCURRENT_RANGE_GETS` (16), adaptive concurrency ON (max 64), `HF_XET_DATA_MAX_CONCURRENT_FILE_DOWNLOADS` (8), chunk cache disabled by default (better for pure download), `HF_XET_HIGH_PERFORMANCE=1` (opt-in max throughput, needs RAM/bandwidth), `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1` (HDD). **64-bit only.**
- https://huggingface.co/docs/huggingface_hub/en/guides/download
- https://huggingface.co/docs/hub/en/xet/using-xet-storage
- https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables
## 2. hf_transfer — AVOID (deprecated)
`HF_HUB_ENABLE_HF_TRANSFER` flagged deprecated; Xet supersedes it. Historically **broke tqdm progress / had no callbacks** — directly conflicts with the accurate-progress goal. Successor for max throughput is `HF_XET_HIGH_PERFORMANCE=1`.
- https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables
- https://github.com/huggingface/hf_transfer/issues/63
## 3. huggingface_hub native concurrency — USE defaults
`snapshot_download(max_workers=...)` = parallel FILES (default 8), orthogonal to Xet's intra-file chunk parallelism. For OmniVoice's 1few-large-file models the win is mostly Xet's intra-file parallelism; don't crank max_workers (multiplies buffer pressure). Resume is automatic via cache + ETag (no `resume_download` flag to manage).
- https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
## 4. Custom IDM-style Range downloader — AVOID as default, BUILD as opt-in for LFS
`/resolve/<rev>/<file>` 302-redirects to CDN (Cloudfront) which honors Range + parallel byte-ranges. Catch: follow redirect, **do NOT forward Authorization to the CDN host** (presigned URL carries auth), verify ETag/sha256, auth on first hop only. Redundant vs Xet for Xet-backed repos (HF closed issue #3232 as "use Xet"), **but genuinely helps non-Xet/legacy-LFS repos** which get no intra-file parallelism. → our Wave 3 opt-in.
- https://github.com/huggingface/huggingface_hub/issues/3232
## 5. aria2 — OPTIONAL, rejected for OmniVoice
`aria2c -x16 -s16 -c --header="Authorization: Bearer <token>"` is 35× on plain LFS, but: no dedup (worse than Xet for Xet repos), per-OS GPLv2 binary to package (parity burden — would have to be opt-in anyway), stdout/RPC progress scraping. The custom httpx path covers the same need with no binary. → not bundled.
- https://gist.github.com/padeoe/697678ab8e528b85a2a7bddafea1fa4f
## 6. Mirrors / HF_ENDPOINT — OPTIONAL, region-gated, breaks Xet
`HF_ENDPOINT=https://hf-mirror.com` redirects Hub traffic (standard for China). **Xet CAS/presigned URLs point at HF infra → mirrors generally don't serve the Xet protocol → traffic falls back to classic LFS** (no dedup, no Xet parallelism). So mirror and Xet fast-path are mutually exclusive; the realistic China stack is mirror + LFS + (our opt-in) segmented accelerator. → our Wave 4 opt-in, per-call `endpoint=` not process-wide.
## 7. Progress / speed — USE `tqdm_class` (xet-aware) + `dry_run` preflight
Unlike hf_transfer, **Xet reports progress through the same tqdm interface**; huggingface_hub aggregates per-file/thread bytes into a shared bar and feeds the `tqdm_class` you pass. So `snapshot_download(tqdm_class=...)` yields reliable aggregate bytes/total/rate/ETA even under parallel fetch. `snapshot_download(dry_run=True)` returns per-file sizes + cached flags → use for "will download X of Y, N GB" preflight. Speed sampling tunable via `HF_XET_DATA_PROGRESS_UPDATE_INTERVAL` (200ms).
- https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
- https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/_snapshot_download.py
## Recommended architecture (→ plan)
Pin `huggingface_hub>=1.7` + `hf-xet`; let Xet be the default (it IS the IDM technique). Drive `snapshot_download(repo_id, tqdm_class=OmniVoiceProgress, max_workers=8, endpoint=<opt-in mirror>)`; `dry_run=True` first for total/remaining; aggregate bytes in a backend tracker → one overall bar (speed/remaining/ETA). Opt-in only: `HF_XET_HIGH_PERFORMANCE` (max speed), `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY` (HDD), a custom httpx **segmented downloader for legacy-LFS repos**, and an `HF_ENDPOINT` mirror (classic-LFS fallback). Never enable hf_transfer; never bundle aria2; never make the segmented path the default.
@@ -1,28 +0,0 @@
# SPIKE — FDL-00: Catalog Xet vs LFS classification
**Date:** 2026-06-13 · **Method:** HF API `GET /api/models/{repo}?expand[]=xetEnabled` (authoritative).
## Result: 25 / 25 catalog repos are Xet-backed
| backend | count |
|---------|-------|
| xet | 25 |
| lfs | 0 |
| unknown | 0 |
Every repo in `backend/config/models.yaml` — including both first-run defaults (`k2-fsa/OmniVoice` TTS, `Systran/faster-whisper-large-v3` ASR) — returns `xetEnabled: true`. Full list: all entries under TTS / ASR / Diarisation (k2-fsa, Systran×5, mlx-community×9, openai, nvidia×2, UsefulSensors×2, pyannote, OpenMOSS, KittenML, deepdml).
## Detection caveat (important for the executor)
The installed client is **huggingface_hub 1.7.2**, whose `repo_info(..., files_metadata=True)` siblings expose only `blob_id, lfs, rfilename, size`**no `xet_file`, and no `xet_enabled` on the info object.** A first pass that inferred backend from siblings wrongly reported "0/25 xet, all LFS." Do **not** classify Xet status from `repo_info` siblings on this client version. The reliable signal is the Hub API `xetEnabled` expand field (used here) or `hf_xet` actually engaging at download time. Re-check after any `huggingface_hub` bump — newer versions surface `xet_enabled` directly.
## Verdict for Wave 3 (segmented accelerator): LOW priority
Because the entire current catalog is Xet-backed and `hf_xet` is installed, Xet already provides chunked parallel range-gets (the IDM/uGet behavior) for **every** model we ship. The custom segmented downloader (Wave 3) is therefore **not needed to speed up any current default model** — it remains valuable only for:
- the **mirror / restricted-network path** (Wave 4: `HF_ENDPOINT` falls back to classic LFS, no Xet), and
- any **future non-Xet repo** a user adds.
**Recommendation:** proceed with W1 (maximize/guarantee Xet) and W2 (accurate progress) as the real wins for today's catalog; keep W3 as opt-in, build it alongside W4's mirror path where it actually pays off. This matches the PLAN's original framing — confirmed, not changed.
## Consequence for W1/W2 framing
W1 "guarantee the Xet fast path" is correctly the primary lever: these repos download via Xet **only if** the client engages it (hf_xet installed ✓ + huggingface_hub recent ✓). The W2 live smoke test should confirm Xet is actually used (fast parallel aggregate progress on a real install), since `xetEnabled=true` is a Hub-side capability, not proof the client took the Xet path.
@@ -1,49 +0,0 @@
# SUMMARY — FDL Waves 02 (fast model downloads)
**Date:** 2026-06-13 · **Scope shipped:** W0 (spike), W1 (maximize Xet), W2 (accurate progress). W3/W4 deferred.
## What landed
**W0 — spike (FDL-00).** Classified all 25 `models.yaml` repos via the HF API `xetEnabled` field → **25/25 Xet-backed** (incl. both first-run defaults). See `260613-fdl-SPIKE.md`. Verdict: Wave 3 (segmented accelerator) is **LOW priority** — Xet already gives parallel chunked transfer for every shipped model. Detection caveat recorded: `repo_info` siblings on hf_hub **1.7.2** expose no xet metadata; classify via the `xetEnabled` API field, not siblings.
**W1 — maximize + guarantee Xet (FDL-01..04).**
- `pyproject.toml`: pinned `huggingface_hub>=1.7` + `hf-xet>=1.1` explicitly (was transitive/unpinned); no `hf_transfer`. Resolves to hf_hub 1.7.2 / hf-xet 1.4.2, single version.
- `download.py`: `install_model` now drives `snapshot_download` with explicit `tqdm_class` (our progress-emitting subclass), `max_workers` (prefs `download_max_workers`, default 8), and `endpoint` (prefs `hf_endpoint` — W4 hook). `apply_xet_env()` applies opt-in `HF_XET_HIGH_PERFORMANCE` + `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY` (both default OFF, env wins).
- `system.py`: `/system/info` now returns `fast_download {xet_enabled, xet_version, high_performance}`; logged once at startup.
**W2 — accurate downloaded/remaining + speed (FDL-05..07).**
- Preflight `snapshot_download(dry_run=True)``compute_plan()``install_plan` SSE event with `total_bytes / cached_bytes / to_download_bytes / n_files / n_cached` **before bytes flow**. Degrades to totals=None on gated/older repos.
- New `utils/download_aggregator.py`: one source of truth for overall progress. Fed by a byte-sink on the patched tqdm; distinguishes byte bars (unit 'B', keyed by bar id) from the "Fetching N files" count bar; emits one throttled `aggregate` event (bytes_done/total/windowed rate/eta/files).
- Frontend `Settings.jsx` + `setup.ts`: overall bar driven by the aggregate; bar % = `max(byte%, file%)`; shows cached-skip + files-progress; `⚡ fast download` badge from `/system/info`. i18n keys added to `en.json`.
## Rebase reconciliation (main disabled Xet)
Rebasing onto latest main surfaced that main now sets **`HF_HUB_DISABLE_XET=1`** (main.py) — a deliberate choice to force the classic LFS path because Xet's progress bypasses the tqdm hook (the exact limitation found here). Reconciled rather than fought:
- `fast_download` status now reports the **runtime truth**: `xet_installed` + `xet_active` (active = installed AND not disabled) + `xet_enabled` alias. Default `xet_active=false`; the ⚡ badge only shows when Xet actually runs. Startup log: `downloads: Xet disabled → legacy LFS …`.
- Docs rewritten: default backend is **legacy LFS for accurate progress**; Xet is opt-in via `HF_HUB_DISABLE_XET=0` (coarser progress). The hf-xet pin stays (harmless; ready for a future Xet progress hook).
- Net: W2's progress is the value either way; W1's "maximize Xet" is dormant by main's design, not removed.
## Decisions / "use judgment" notes
- **Xet progress limitation (verified by live smoke).** Under Xet + hf_hub 1.7.2 the per-file **byte** bars never advance `n` and never `close()` through our tqdm (Xet fetches chunks out-of-band). Only the **file-count bar** is live. So: mid-download the overall bar is **file-granular** (moves 0→N files), and `complete()` flushes `bytes_done` to the exact preflight total on success (verified: final `74420620/74420620`, files 4/4). True live byte-speed is only available on classic-LFS/mirror repos (W4). This is a real constraint, not a bug — documented here and worth surfacing in W4 docs.
- Per-file detail kept inline (existing single-line summary, now aggregate-sourced) rather than a new collapsible panel — limited risk; can revisit.
## Drive-by fix
- `download.py` imported no `os`, but `_validate_snapshot_has_weights` uses `os.walk` → latent `NameError` on every install. Added `import os`.
## Verification
- `tests/backend/setup/test_download_preflight.py` — 10 pass (compute_plan splits, aggregator byte/count routing, close-credit, windowed rate/eta, registry feed + finish noop).
- `pytest -k "download or install or model or engine or setup"` — 149 passed, 7 skipped, 0 failed.
- `frontend typecheck:ci` — exit 0.
- Live smoke (real install of `mlx-community/whisper-tiny-mlx`, then deleted): `install_plan` exact; aggregate files 0→1→4; final bytes==total; `/system/info` + startup log correct.
## W4 — mirror + cancel + docs (FDL-10..12, shipped)
- **Mirror (FDL-10):** `snapshot_download(endpoint=…)` honours prefs `hf_endpoint` / env `HF_ENDPOINT` on both preflight and download — per-call, no process-wide mutation. Documented as the classic-LFS (non-Xet) path that restores continuous byte-speed.
- **Cancel (FDL-11):** `POST /models/install/cancel {repo_id}` sets a cancel flag checked at each retry boundary → emits `install_cancelled`, clears the cooldown (cancel ≠ failure). Limitation: an in-flight single-file fetch isn't interruptible in hf_hub 1.7.2; cancel lands at the next retry boundary. Frontend treats `install_cancelled` as a terminator (clears row + refetch).
- **Docs (FDL-12):** `docs/downloading-models.md` (Xet fast path, progress semantics incl. the byte-speed limitation, opt-in tuning knobs, mirror/restricted-network, cancel, troubleshooting) + README pointer. Docs-sync rule satisfied in-PR.
## W3 — opt-in segmented accelerator (FDL-08/09, shipped)
Reprioritised from LOW to HIGH after the rebase: since main forces Xet off, the default path is single-stream legacy LFS, so a segmented downloader is the way to get **both** parallel speed and live byte progress.
- `services/segmented_download.py`: async multi-connection Range downloader for one file — parallel byte-ranges, resume (`.part` + manifest), per-segment short-read truncation guard, optional sha256/etag verify, cancel, single-stream fallback when the server won't range. **Auth-safe**: the HF `Authorization` header goes only to `huggingface.co`/`hf.co`; never forwarded to a CDN host on redirect (unit-tested).
- Dispatch (`download.py`): opt-in via prefs `segmented_downloader` / env `OMNIVOICE_SEGMENTED_DOWNLOAD` (default OFF). When on and Xet inactive, fetches each repo file into the HF cache mirroring `hf_hub_download` (blobs + snapshot symlinks + `refs/main`), feeding **real bytes** to the aggregator. Any failure falls back to `snapshot_download` — the accelerator can never break a correct install.
- Verified live (accelerator ON): real mid-download byte progress (1.5 KB → 71 MB, rate ramping to **16.6 MB/s**), final `bytes_done == total`, `/models` shows `installed: True`, delete frees the right bytes.
- Fixed a `complete()` double-count (was adding a full total on top of accumulated segmented bytes → 2×); now replaces byte bars so the sum is exactly total.
- Tests: `tests/backend/services/test_segmented_download.py` (7 cases) covering parallel range reassembly, single-stream fallback, the auth header reaching only the HF host (never a CDN), size/truncation rejection, cancellation, and byte-callback totals — plus an aggregator double-count regression.
@@ -1,363 +0,0 @@
---
phase: 260613-mm2
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- backend/services/tts_backend.py
- backend/services/model_manager.py
- backend/services/subprocess_backend.py
- backend/services/model_lifecycle.py # NEW
- backend/api/routers/system.py
- backend/api/routers/setup/download.py
- backend/api/routers/setup/models.py
- tests/test_engines.py
- tests/backend/services/test_model_lifecycle.py # NEW
- tests/backend/services/test_subprocess_reaper.py
autonomous: true
requirements:
# ── Tier 1 — Correctness (Wave 1) ──────────────────────────────────────────
- MM2-01 # Registry reuses one active instance and calls unload() on engine switch
- MM2-02 # Per-engine unload() overrides (in-process drop+free_vram; subprocess -> unload_sidecar)
- MM2-03 # /model/loaded + /model/unload report ASR honestly; no unloadable:False-but-loaded lies
# ── Tier 2 — Single lifecycle surface (Wave 2) ─────────────────────────────
- MM2-04 # model_lifecycle facade owns list_loaded/unload/unload_all/free_vram across all 3 worlds
- MM2-05 # Idle/timeout config unified through core.prefs.resolve (env still wins); no duplicated constants
# ── Tier 3 — Robustness & observability (Wave 3) ───────────────────────────
- MM2-06 # _install_cooldowns bounded (evict on success + TTL); no unbounded growth
- MM2-07 # Snapshot weight validation is per-role, not one 5 MB magic number
- MM2-08 # Subprocess sidecars self-report VRAM in pong; panel shows real MB, not 0
- MM2-09 # scan_cache_dir() -> disk-walk fallback logs WHY it fell back (WinError #117/#118)
must_haves:
truths:
- "Switching the active TTS backend in Settings releases the outgoing engine's VRAM before the new one loads — verified by asserting the outgoing instance's unload() was called exactly once on switch."
- "TTSBackend.unload() is overridden by OmniVoiceBackend (drops model ref + free_vram) and by every SubprocessBackend subclass (routes to unload_sidecar); all overrides are idempotent and safe before first generate()."
- "/model/loaded never reports a model as loaded with a misleading unloadable flag: the ASR row's unloadable reflects whether it can actually be released independently of the TTS lifecycle."
- "services.model_lifecycle is the single import surface for list_loaded()/unload(id)/unload_all()/free_vram(); system.py routers call it instead of re-enumerating models inline."
- "Idle timeouts for the in-process model and subprocess sidecars resolve through core.prefs.resolve(... env=...) so an env var still wins and the Settings store can override; no module duplicates IDLE_TIMEOUT_SECONDS by hand."
- "_install_cooldowns cannot grow without bound: entries are removed on successful install and stale entries are evicted by TTL."
- "A live subprocess sidecar reports a non-zero vram_mb in /model/loaded when it actually holds GPU memory (pong carries the figure); CPU-only sidecars report 0 truthfully."
- "When scan_cache_dir() raises and the code falls back to the on-disk walk, the reason is logged at WARNING with the exception type (the #117/#118 WinError-448 path is no longer silent)."
- "uv run pytest tests/test_engines.py tests/backend/services/test_model_lifecycle.py tests/backend/services/test_subprocess_reaper.py tests/test_model_load_timeout.py passes."
- "No on-disk model state changes; no new runtime dependency added; behavior degrades gracefully (not errors) on MPS/CPU where VRAM APIs are sparse."
artifacts:
- path: "backend/services/tts_backend.py"
provides: "Active-instance reuse + unload-on-switch in get_active_tts_backend(); per-engine unload() overrides"
contains: "_active_instance AND (def unload)"
- path: "backend/services/model_lifecycle.py"
provides: "Facade owning list_loaded/unload/unload_all/free_vram across in-process + subprocess models"
contains: "def list_loaded AND def unload_all"
- path: "backend/api/routers/system.py"
provides: "Thin /model/loaded + /model/unload routers delegating to model_lifecycle"
contains: "model_lifecycle"
key_links:
- from: "get_active_tts_backend() (tts_backend.py:1235)"
to: "outgoing backend.unload()"
via: "module-level _active_instance compared against newly-resolved active_backend_id()"
pattern: "_active_instance"
- from: "system.py /model/loaded + /model/unload (system.py:129, 210)"
to: "model_lifecycle.list_loaded() / model_lifecycle.unload()"
via: "import services.model_lifecycle"
pattern: "model_lifecycle\\.(list_loaded|unload)"
- from: "subprocess sidecar pong reply (subprocess_backend.py:435-438)"
to: "list_live_sidecars() vram_mb field"
via: "ping reply carries allocated VRAM measured inside the sidecar process"
pattern: "vram_mb"
---
<objective>
Clean up OmniVoice's model-management subsystem ("v2"). Today load / unload / list / free-VRAM each behave differently across three worlds — the in-process model (`model_manager.py`), the TTS backend registry (`tts_backend.py`), and subprocess sidecars (`subprocess_backend.py`) — with no single lifecycle owner. This produces one real user-facing bug (VRAM leak on engine switch), inaccurate VRAM/unloadable reporting, an unbounded cooldown dict, and a silent cache fallback.
This is **cleanup + correctness, not a rewrite.** The Wave 13 idle-reaper and the SubprocessBackend primitive are sound and stay. The `TTSBackend.unload()` contract already exists as a documented default no-op (`tts_backend.py:149`) explicitly deferred to "Phase 2"; this plan *is* that Phase-2 follow-through — wire the registry to call it, override it per engine, and unify the surrounding surface.
Three tiers, executed in order (each independently shippable, continuous-to-main per the v0.3.0 cadence):
- **Wave 1 / Tier 1 — Correctness:** MM2-01..03. The VRAM leak on switch + honest unload reporting. Highest value; ship first.
- **Wave 2 / Tier 2 — Single lifecycle surface:** MM2-04..05. Extract `model_lifecycle` facade + unify idle/timeout config.
- **Wave 3 / Tier 3 — Robustness & observability:** MM2-06..09. Bounded cooldowns, per-role weight validation, sidecar VRAM self-report, cache-fallback logging.
Output: PRs on branches off `main` (one per wave is fine), each green on the listed pytest selection. No push until the orchestrator merges; tests added with each wave.
Out of scope (call out, do not touch): GPU-pool per-engine sizing (`model_manager.py:42`, `_GPU_VRAM_PER_JOB_GB`) and torch.compile tuning — those are performance, not cleanup, and carry regression risk against #278/#315.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/STATE.md
@./CLAUDE.md
# Files under edit (read before editing)
@backend/services/tts_backend.py
@backend/services/model_manager.py
@backend/services/subprocess_backend.py
@backend/api/routers/system.py
@backend/api/routers/setup/download.py
@backend/api/routers/setup/models.py
# Reference only — establish patterns, do NOT modify
@backend/core/prefs.py
@tests/test_engines.py
@tests/backend/services/test_subprocess_reaper.py
<interfaces>
<!-- Verified during planning. Executor should use these directly. -->
backend/services/tts_backend.py
- class TTSBackend(ABC) (line 59); unload() default no-op (line 149) — contract already documented:
idempotent, synchronous, safe before first generate().
- OmniVoiceBackend.__init__(self, model=None) (line 174); self._model reuses model_manager singleton.
- _REGISTRY: dict[str, type[TTSBackend]] (line 1109, a _LazyRegistry).
- active_backend_id() (line 1228) -> prefs.resolve("tts_backend", env="OMNIVOICE_TTS_BACKEND", default="omnivoice").
- get_active_tts_backend(*, model=None) (line 1235) — builds a FRESH instance every call, no teardown. THE leak.
backend/core/prefs.py
- resolve(key: str, *, env: Optional[str] = None, default: Any = None) -> Any (line 75) — env wins, then store, then default.
backend/services/model_manager.py
- module global `model` (line 111); `_last_used`; free_vram() (line 678); idle_worker() (line 667).
- IDLE_TIMEOUT_SECONDS imported from core.config (line 33); duplicated as _IDLE_TIMEOUT_SECONDS (line 114). Collapse.
- offload_tts_for_asr() (line 701) / restore_tts_after_asr() — ad-hoc ASR<->TTS VRAM juggling; _diar_pipeline global.
backend/services/subprocess_backend.py
- protocol op set (line 74); SIDECAR_IDLE_TIMEOUT_S = env-only float (line 107) — move to prefs.resolve.
- list_live_sidecars() -> list[dict] (line 181); unload_sidecar(engine_id) (line 199); unload_all_sidecars() (line 205).
- health/ping: _send({"op":"ping"}) then expect {"op":"pong"} (lines 435-438). Add vram_mb to the pong here AND
in the sidecar entry-point that answers ping (search the sidecar worker for the "ping"->"pong" handler).
backend/api/routers/system.py
- GET /model/loaded (line 129) — ~80 lines of inline enumeration of TTS/ASR/diar/sidecars. Replace body with
model_lifecycle.list_loaded().
- POST /model/unload/{model_id} (line 210) — handles "tts" | "diarization" | "sidecar:<id>" | "sidecars".
Replace body with model_lifecycle.unload(model_id).
backend/api/routers/setup/download.py
- _install_cooldowns dict (line 27) — unbounded. _validate_snapshot_has_weights (line 55) + _MIN_WEIGHT_BYTES 5 MB
(line 45) — single magic number across roles.
backend/api/routers/setup/models.py
- scan_cache_dir() with silent disk-walk fallback (~line 268-280) + _scan_cache_on_disk (line 177).
</interfaces>
</context>
<tasks>
<!-- ════════════ WAVE 1 / TIER 1 — CORRECTNESS ════════════ -->
<task type="auto">
<name>Task 1 (MM2-02): Per-engine unload() overrides</name>
<files>backend/services/tts_backend.py</files>
<action>
The base-class `unload()` no-op already exists (tts_backend.py:149) with a documented contract. Override it where it matters. Do this BEFORE Task 2 — the registry switch (Task 2) calls these.
- OmniVoiceBackend (line 162): override `unload(self)`. Drop the local model ref (`self._model = None`) and, because OmniVoice shares the singleton owned by model_manager, also release that: `import services.model_manager as mm; mm.model = None; mm.free_vram()`. Idempotent — guard on `mm.model is not None` before free_vram(). Safe before first generate() (no-op when nothing loaded).
- Every SubprocessBackend subclass: implement `unload(self)` on the SubprocessBackend base (subprocess_backend.py — the duck-typed `_is_subprocess_isolated` class) so all subclasses inherit it. It must call `unload_sidecar(self.id)` (force-shut this engine's sidecar; busy sidecars are skipped, never interrupted — existing semantics). Idempotent: unload_sidecar on a non-running engine returns 0, no raise.
- In-process non-OmniVoice engines that hold their own model (e.g. KittenTTS/VoxCPM2 keep refs in __init__): override unload() to drop the ref + best-effort empty_cache via the existing free_vram() helper if they used GPU. Where an engine genuinely holds nothing resident, leave the base no-op (and note it in the SUMMARY so the future CI gate knows it's intentional, not missed).
Honor the contract comment verbatim: idempotent, synchronous, safe pre-load.
</action>
<verify>
<automated>grep -n "def unload" backend/services/tts_backend.py backend/services/subprocess_backend.py</automated>
<automated>uv run python -c "from services.tts_backend import OmniVoiceBackend; b=OmniVoiceBackend(); b.unload(); b.unload(); print('idempotent ok')"</automated>
</verify>
<done>
- OmniVoiceBackend.unload() drops both self._model and mm.model and calls free_vram(), guarded for idempotency.
- SubprocessBackend.unload() routes to unload_sidecar(self.id); inherited by all subprocess engines.
- Calling unload() twice, and before any generate(), never raises.
</done>
</task>
<task type="auto">
<name>Task 2 (MM2-01): Registry reuses one active instance + unloads on switch</name>
<files>backend/services/tts_backend.py</files>
<action>
Fix the leak at get_active_tts_backend() (line 1235). Today it builds a fresh instance every call with no teardown of the prior engine — switching engines (or repeated synth) leaks VRAM until GC. This is the root cause behind the #278 comment thread.
- Add a module-level cache: `_active_instance: TTSBackend | None = None` and `_active_instance_id: str | None = None`.
- In get_active_tts_backend(): resolve `bid = active_backend_id()`. If `_active_instance is not None` and `_active_instance_id != bid`, call `_active_instance.unload()` (best-effort, wrap in try/except so a bad unload can't block the switch — log on failure) before discarding it.
- Build the new instance, store it as `_active_instance` + `_active_instance_id = bid`, return it.
- IMPORTANT subtlety: OmniVoiceBackend takes `model=`. When `model=` is passed (the caller already has a loaded model), do NOT cache that instance as the shared `_active_instance` blindly — it's a per-call view over the shared singleton. Keep current behavior for the `model=` path (return a fresh OmniVoiceBackend(model=model)) but still trigger unload() of a *different* outgoing engine first. Pick the simplest correct rule: the cache tracks the configured backend id; passing model= for the SAME id reuses, switching id always unloads the previous. Document the rule in a comment.
- Add a module-level `reset_active_backend()` helper that unloads + clears the cache, for app shutdown and tests.
</action>
<verify>
<automated>grep -n "_active_instance\|def reset_active_backend\|def get_active_tts_backend" backend/services/tts_backend.py</automated>
</verify>
<done>
- Switching backend id calls the outgoing instance's unload() exactly once before the new instance is built.
- A bad/raising unload() is caught + logged, never blocks the switch.
- reset_active_backend() exists and is idempotent.
- The model= fast-path for OmniVoice still works (no double-load).
</done>
</task>
<task type="auto">
<name>Task 3 (MM2-03): Honest /model/loaded + /model/unload for ASR</name>
<files>backend/api/routers/system.py</files>
<action>
The ASR row (system.py:166-175) is reported as unloadable:False, vram_mb:0 even when loaded on GPU, and /model/unload doesn't expose the offload-to-CPU path. Make reporting truthful WITHOUT changing the ASR<->TTS lifecycle coupling (that coupling is intentional — offload_tts_for_asr/restore_tts_after_asr).
- ASR row: keep unloadable reflecting reality. If ASR truly cannot be released independently of TTS, keep unloadable:False but add a `note` field ("released with TTS") so the UI explains it rather than showing a dead button. Do not invent a separate ASR unload that breaks the WhisperX large-v3 offload path.
- vram_mb: if ASR currently runs on CPU (device "cpu" in the row), 0 is correct — leave it but make the device value derive from where the pipe actually is, not a hardcoded "cpu".
- This task is intentionally small; the bigger restructure is Task 4 (facade). Land MM2-03 as the honest-reporting fix, then Task 4 moves the enumeration into the facade.
</action>
<verify>
<automated>uv run pytest tests/test_engines.py -q 2>&amp;1 | tail -15</automated>
</verify>
<done>
- No row reports loaded-but-with-a-misleading-unloadable flag; ASR carries an explanatory note when unloadable:False.
- Device field reflects the actual device of the ASR pipe.
</done>
</task>
<task type="auto">
<name>Task 4 (MM2-01..03 tests): Wave 1 regression tests</name>
<files>tests/test_engines.py</files>
<action>
Add tests proving the leak fix and the unload contract:
- test_switching_backend_unloads_previous: monkeypatch two fake backends into _REGISTRY, set active to A (get_active_tts_backend), switch prefs to B, assert A.unload() was called exactly once before B is returned.
- test_unload_is_idempotent_and_preload_safe: OmniVoiceBackend().unload() twice + before generate() never raises.
- test_reset_active_backend_clears_cache: after reset_active_backend(), the next get_active_tts_backend() builds fresh.
Reuse the existing fixture style in tests/test_engines.py (it already monkeypatches the registry / availability). Keep tests CPU-only (no real model load).
</action>
<verify>
<automated>uv run pytest tests/test_engines.py -q 2>&amp;1 | tail -20</automated>
</verify>
<done>All three new tests pass; existing test_engines.py tests still green.</done>
</task>
<!-- ════════════ WAVE 2 / TIER 2 — SINGLE LIFECYCLE SURFACE ════════════ -->
<task type="auto">
<name>Task 5 (MM2-04): Extract services/model_lifecycle.py facade</name>
<files>backend/services/model_lifecycle.py</files>
<action>
Create backend/services/model_lifecycle.py as the single owner of cross-world model lifecycle. It composes the existing pieces — it does NOT reimplement loading.
Public surface:
- list_loaded() -> list[dict]: returns the unified rows currently assembled inline in system.py:129-207 (TTS, ASR, diarization, subprocess sidecars). Move that logic here verbatim first, then improve (MM2-03 note field, MM2-08 sidecar vram once Task 8 lands).
- unload(model_id: str) -> dict: the dispatch currently inline in system.py:210-242 ("tts" | "diarization" | "sidecar:<id>" | "sidecars"). Move here; keep async-lock semantics for the in-process model (mm._model_lock).
- unload_all() -> dict: unload every releasable model (in-process TTS + diar + all sidecars). New convenience used by app shutdown.
- free_vram(): thin re-export of model_manager.free_vram() so callers have one import.
Keep the "never let sidecar enumeration break the panel" try/except guard.
</action>
<verify>
<automated>uv run python -c "import services.model_lifecycle as ml; print([f for f in ('list_loaded','unload','unload_all','free_vram') if hasattr(ml,f)])"</automated>
</verify>
<done>model_lifecycle exposes list_loaded/unload/unload_all/free_vram; logic moved out of system.py (not duplicated).</done>
</task>
<task type="auto">
<name>Task 6 (MM2-04): Thin system.py routers + facade tests</name>
<files>backend/api/routers/system.py, tests/backend/services/test_model_lifecycle.py</files>
<action>
- Replace the bodies of GET /model/loaded (line 129) and POST /model/unload/{model_id} (line 210) with calls to model_lifecycle.list_loaded() / model_lifecycle.unload(model_id). Preserve the exact response shapes (frontend hooks.ts useModelStatus/useFlushMemory + the flush dropdown depend on {models, count} and {unloaded, success, ...}). The 400 on unknown model_id stays.
- New tests/backend/services/test_model_lifecycle.py: list_loaded with nothing loaded returns {models:[], count:0}; unload("tts") when not loaded returns success:False reason:"not loaded"; unload("sidecars") with no sidecars returns count:0; unknown id raises/400 path. Mock model_manager + subprocess_backend so no real models load.
</action>
<verify>
<automated>uv run pytest tests/backend/services/test_model_lifecycle.py -q 2>&amp;1 | tail -20</automated>
<automated>grep -n "model_lifecycle" backend/api/routers/system.py</automated>
</verify>
<done>system.py routers are thin delegations; response shapes unchanged; new facade tests pass.</done>
</task>
<task type="auto">
<name>Task 7 (MM2-05): Unify idle/timeout config through prefs.resolve</name>
<files>backend/services/model_manager.py, backend/services/subprocess_backend.py</files>
<action>
- model_manager.py: remove the duplicated `_IDLE_TIMEOUT_SECONDS = IDLE_TIMEOUT_SECONDS` (line 114). Resolve at use-site in idle_worker() via prefs: `prefs.resolve("idle_timeout_seconds", env="OMNIVOICE_IDLE_TIMEOUT_S", default=IDLE_TIMEOUT_SECONDS)`. Keep core.config.IDLE_TIMEOUT_SECONDS as the default source.
- subprocess_backend.py: replace the env-only `SIDECAR_IDLE_TIMEOUT_S` (line 107) read with prefs.resolve("sidecar_idle_timeout_seconds", env="OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S", default=300.0). Preserve "<=0 disables reaping" semantics and the existing reaper-start guard (line 222). Resolve lazily (function call), not at import, so a test/setting change takes effect — but keep a sensible cached default for the hot reaper loop.
- Both must keep env precedence (env wins over store) — that's exactly what prefs.resolve already does.
</action>
<verify>
<automated>grep -n "_IDLE_TIMEOUT_SECONDS\|prefs.resolve\|SIDECAR_IDLE_TIMEOUT" backend/services/model_manager.py backend/services/subprocess_backend.py</automated>
<automated>uv run pytest tests/backend/services/test_subprocess_reaper.py -q 2>&amp;1 | tail -20</automated>
</verify>
<done>
- No hand-duplicated IDLE_TIMEOUT constant; both timeouts resolve via prefs with env precedence.
- Reaper "<=0 disables" + busy-skip behavior unchanged; all 10+ reaper tests still pass.
</done>
</task>
<!-- ════════════ WAVE 3 / TIER 3 — ROBUSTNESS & OBSERVABILITY ════════════ -->
<task type="auto">
<name>Task 8 (MM2-08): Subprocess sidecars self-report VRAM in pong</name>
<files>backend/services/subprocess_backend.py</files>
<action>
Sidecar VRAM is reported as 0 (system.py:192-203 / list_live_sidecars) because the parent can't measure a child's GPU memory. Have the child measure itself.
- In the sidecar worker's ping handler (the code that answers {"op":"ping"} with {"op":"pong"} — find it in the sidecar entry-point module), include `vram_mb`: measure inside the child via torch.cuda.memory_allocated() (CUDA) or torch.mps.driver_allocated_memory() (MPS, guarded), else 0. Same degrade-gracefully pattern as system.py:147-156.
- Parent: in the health-check ping/pong path (subprocess_backend.py:435-438), capture reply["vram_mb"] and stash it on the sidecar record so list_live_sidecars() (line 181) can surface it. Refresh opportunistically on each successful ping; default to last-known or 0 if never measured.
- Keep the contract that enumeration never breaks the panel.
This is CUDA/MPS-aware and degrades to 0 on CPU — honoring cross-platform parity (default behavior identical; the number is just more accurate where the API exists).
</action>
<verify>
<automated>grep -n "vram_mb" backend/services/subprocess_backend.py</automated>
<automated>uv run pytest tests/backend/services/test_subprocess_reaper.py -q 2>&amp;1 | tail -15</automated>
</verify>
<done>list_live_sidecars() exposes a vram_mb sourced from the child's own measurement; 0 only when truly CPU/unmeasured; reaper tests still green.</done>
</task>
<task type="auto">
<name>Task 9 (MM2-06, MM2-07): Bounded cooldowns + per-role weight validation</name>
<files>backend/api/routers/setup/download.py</files>
<action>
- MM2-06: _install_cooldowns (line 27) grows unbounded. On a successful install, delete the repo's cooldown entry. Add a TTL sweep: when reading/writing the dict, evict entries older than a fixed window (reuse the existing cooldown window constant; pick the larger of cooldown-window and e.g. 1h). Keep it simple — a dict + timestamps, swept on access. No new dep.
- MM2-07: _validate_snapshot_has_weights (line 55) + _MIN_WEIGHT_BYTES 5 MB (line 45) is one magic number for all roles. Make the threshold per-role/per-extension: safetensors/bin/ckpt expect the existing floor; .onnx models (kittentts, supertonic, sherpa) can be legitimately smaller — set a lower, role-aware floor so a valid small ONNX model isn't flagged as truncated. Keep the #352 truncation-catch intent (catch a 0-byte / KB-sized partial), just stop false-positiving small-but-complete models.
</action>
<verify>
<automated>grep -n "_install_cooldowns\|_MIN_WEIGHT_BYTES\|def _validate_snapshot_has_weights" backend/api/routers/setup/download.py</automated>
<automated>uv run pytest tests/ -k "download or install or model" -q 2>&amp;1 | tail -20</automated>
</verify>
<done>Cooldown dict is bounded (evict-on-success + TTL sweep); weight validation floor varies by role/extension; #352 truncation still caught.</done>
</task>
<task type="auto">
<name>Task 10 (MM2-09): Log why scan_cache_dir() fell back to disk walk</name>
<files>backend/api/routers/setup/models.py</files>
<action>
The scan_cache_dir() -> _scan_cache_on_disk() fallback (~line 268-280, helper at line 177) silently swallows the exception — this is the #117/#118 Windows WinError-448 path. Wrap the fallback so it logs at WARNING with the exception type and a one-line reason ("scan_cache_dir failed (%s); falling back to on-disk walk of %s") before walking. Do not change the fallback behavior itself — just stop it being invisible in logs. Keep it from ever raising out (the panel must still render).
</action>
<verify>
<automated>grep -n "falling back\|logger.warning\|_scan_cache_on_disk\|scan_cache_dir" backend/api/routers/setup/models.py | head</automated>
</verify>
<done>The disk-walk fallback logs a WARNING naming the exception type; behavior otherwise unchanged; never raises out.</done>
</task>
</tasks>
<verification>
Full-suite gate after each wave (run the relevant subset per wave, full set before the last PR):
1. `uv run pytest tests/test_engines.py tests/backend/services/test_model_lifecycle.py tests/backend/services/test_subprocess_reaper.py tests/test_model_load_timeout.py tests/test_model_manager_preload.py -q` — all green.
2. `uv run pytest tests/ -k "download or install or model or engine" -q` — green (Tier 3 touch points).
3. Response-shape guard: GET /model/loaded still returns {models, count}; POST /model/unload returns {unloaded, success, ...}; 400 on unknown id. (Covered by test_model_lifecycle.py.)
4. No new runtime dependency: `git diff pyproject.toml uv.lock` is empty.
5. Localization/CJK + redaction gates unaffected: `uv run pytest tests/test_no_hardcoded_cjk.py -q`.
</verification>
<success_criteria>
- Tier 1: switching the active backend releases the previous engine's VRAM (unload() called once on switch); contract overridden for OmniVoice + all subprocess engines; ASR reporting is honest. (MM2-01..03)
- Tier 2: services.model_lifecycle is the single lifecycle surface; system.py routers are thin delegations with unchanged response shapes; idle/timeout config flows through prefs.resolve with env precedence and no duplicated constants. (MM2-04..05)
- Tier 3: cooldown dict bounded; weight validation is per-role; sidecars self-report real VRAM; cache-fallback logs its reason. (MM2-06..09)
- All listed pytest selections pass; no on-disk model-state change; no new dep; cross-platform default behavior identical (VRAM numbers degrade gracefully on MPS/CPU).
</success_criteria>
<risks>
- **unload() correctness for the shared OmniVoice singleton (MM2-01/02):** OmniVoiceBackend shares model_manager's `model` global. unload() must release the shared singleton, but the idle_worker() + offload_tts_for_asr() paths also touch it. Risk: a switch during an in-flight ASR offload double-frees or races. Mitigation: take mm._model_lock around the shared release in unload(); guard on `mm.model is not None`; keep unload best-effort (try/except) so it can never wedge a switch. Add the idempotency test (Task 4).
- **Response-shape drift (MM2-04):** Moving /model/loaded + /model/unload bodies into the facade risks changing the JSON the frontend depends on (hooks.ts, flush dropdown). Mitigation: move verbatim first, assert shapes in test_model_lifecycle.py, only then layer MM2-03/08 improvements.
- **Sidecar protocol change (MM2-08):** Adding vram_mb to pong touches the parent/child wire format. Older sidecars (a long-running session mid-upgrade) won't send it. Mitigation: treat vram_mb as optional in the parent (`reply.get("vram_mb", <last-known or 0>)`); never require it; never break the existing pong==success check.
- **prefs.resolve at import time (MM2-05):** Resolving timeouts at import freezes them; the reaper loop reads SIDECAR_IDLE_TIMEOUT_S. Mitigation: resolve lazily inside the reaper tick / idle_worker tick (cheap) so a settings change takes effect, while keeping the import-time default for the start-guard.
- **Per-role weight floor (MM2-07):** Lowering the ONNX floor could let a genuinely-truncated ONNX through (#352 regression). Mitigation: keep a non-zero floor for every role (e.g. ONNX floor still >> a partial KB), key on extension, and keep the "largest file" heuristic — only the threshold becomes role-aware.
- **Scope creep into perf:** GPU-pool sizing and torch.compile are explicitly out of scope. If an executor is tempted, stop — those regress #278/#315.
</risks>
<output>
Write `.planning/quick/260613-mm2-clean-model-management-v2/260613-mm2-SUMMARY.md` when done (per wave or once at the end), documenting: which engines got real unload() overrides vs intentional no-ops (for the future CI gate), the exact response shapes preserved on the two endpoints, the per-role weight-validation thresholds chosen, and the pytest output for the verification selection. Note any decision an executor made where the plan said "use judgment."
Docs-sync check (CLAUDE.md hard rule): this is internal lifecycle cleanup with no user-facing install/Docker/versioning change, so no README/docs edit is expected. If MM2-05 surfaces the new idle-timeout settings keys in the Settings UI, add them to the relevant settings doc in the same PR.
</output>
@@ -1,32 +0,0 @@
# SUMMARY — model-management v2 cleanup (mm2)
**Date:** 2026-06-13 · **Scope:** all 3 tiers (MM2-01..09). Backend-only; no frontend, no on-disk model-state change, no new deps.
## Tier 1 — correctness
- **MM2-01 (VRAM leak on engine switch):** `get_active_tts_backend()` now caches one instance per configured backend id and calls the outgoing engine's `unload()` before switching. Added `reset_active_backend()` for shutdown/tests. The `model=` OmniVoice fast-path still returns a fresh view over the shared singleton (no double-load) but a switch *away from* another engine still releases it. `tts_backend.py`.
- **MM2-02 (per-engine unload()):** `OmniVoiceBackend.unload()` drops the local ref + the shared `model_manager.model` singleton + `free_vram()` (idempotent, preload-safe, best-effort — no async lock from the sync path). `SubprocessBackend.unload()` routes to `unload_sidecar(self.id)` (busy sidecars skipped) and is inherited by every subprocess engine.
- **MM2-03 (honest ASR row):** `/model/loaded` ASR row now reports the pipe's actual device and carries a `note: "released with the TTS model"` so the disabled unload button is explained rather than silent.
## Tier 2 — single lifecycle surface
- **MM2-04 (`services/model_lifecycle.py`):** new facade owns `list_loaded()` / `unload(id)` / `unload_all()` / `free_vram()` across in-process TTS+ASR, diarization, and sidecars. `system.py` `/model/loaded` + `/model/unload` are now thin delegations; **response shapes preserved exactly** (`{models,count}`, `{unloaded,success,...}`, 400 on unknown id) — frontend untouched.
- **MM2-05 (unified idle config):** removed the duplicated `_IDLE_TIMEOUT_SECONDS`; the in-process idle timeout and the sidecar idle timeout both resolve per-tick via `prefs.resolve(... env=...)` (env wins, settings can tune without restart). New keys: `idle_timeout_seconds` (`OMNIVOICE_IDLE_TIMEOUT_S`), `sidecar_idle_timeout_seconds` (`OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S`). `<=0` still disables sidecar reaping.
## Tier 3 — robustness & observability
- **MM2-06 (bounded cooldowns):** `_install_cooldowns` is swept (TTL 1h) on each install check and cleared on success — can no longer grow unbounded.
- **MM2-07 (per-role weight floor):** `_validate_snapshot_has_weights` uses per-extension floors (tensor formats keep 5 MB; `.onnx` floor 64 KB) **OR** the original ≥5 MB catch — strictly more lenient, so a small-but-complete ONNX model is no longer false-flagged as truncated while a 0/KB partial is still rejected (#352 intact).
- **MM2-08 (sidecar VRAM self-report):** the parent can't see a child's VRAM, so the GPU sidecar (`engines/indextts`) now reports `vram_mb` in its `pong` (CUDA/MPS-aware, 0 on CPU); the parent stashes the last-known figure and `list_live_sidecars()` surfaces it. CPU/absent sidecars honestly report 0.
- **MM2-09 (cache-fallback logging):** the `is_cached` `scan_cache_dir → on-disk` fallback now logs at WARNING with the exception type (was DEBUG/invisible) — the #117/#118 WinError-448 path is triagable from logs.
## Out of scope (as planned, not done)
GPU-pool per-engine sizing (`_GPU_VRAM_PER_JOB_GB`) and torch.compile tuning — perf, not cleanup; risk regressing #278/#315.
## Verification
- New `tests/test_mm2_lifecycle.py` — 15 tests (reuse/switch-unload/reset/idempotent-unload, facade list/unload/unknown/sidecars shapes + honest ASR, env-wins idle config, cooldown sweep, per-role weight floor ×3).
- Affected existing: `test_engines.py`, `test_subprocess_reaper.py`, `test_model_load_timeout.py`, `test_model_manager_preload.py` — green (no regressions).
- **Full suite: 1379 passed, 0 failed.** Live: facade endpoints return preserved shapes; engine switch calls the previous engine's `unload()` exactly once.
## Test placement note
MM2 tests live at top-level `tests/` (not `tests/backend/`) on purpose: adding files under `tests/backend/` reorders collection and can expose a pre-existing `sys.modules`-isolation leak in other backend fixtures (the issue debugged in the FDL PR). Top-level placement keeps `tests/backend/` order identical.
## Docs-sync
The new idle-timeout settings keys are internal env/prefs knobs with no UI surface, so no README/docs change is required by the docs-sync rule. If a future Settings panel exposes them, document there.
-12
View File
@@ -6,18 +6,6 @@ The format is loosely based on [Keep a Changelog](https://keepachangelog.com/).
Versions track the desktop app (`tauri.conf.json` + `frontend/src-tauri/Cargo.toml`).
The bundled TTS model package (`pyproject.toml`) is versioned independently.
## [Unreleased]
### Added
- **Portable personas (`.ovsvoice`).** Export any voice as a self-contained,
fully-local persona bundle — identity, optional reference clip, consent
attestation, SPDX license, and a watermarked preview — and import it back into
another OmniVoice install. A privacy toggle ships a **preview-only** bundle so
no raw recording of your voice has to travel. Verified-own-voice status can't
be forged by hand-editing a bundle (real recording + consent text + attestation
required). Legacy `.omnivoice` files still import. See
[docs/persona-format.md](docs/persona-format.md). (#29)
## [0.3.5] — 2026-06-03
### Fixed
+3 -13
View File
@@ -3,7 +3,7 @@
**OmniVoice Studio**
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. It's an active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord. The latest stable release is **v0.3.5**; `main` rolls ahead at **v0.3.6** (latest release + 1 patch — see the Versioning rule below).
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. Today it's a v0.2.7 active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord.
**Core Value:** **A first-run that actually works.** A user who downloads the installer (or clones the repo) should reach a working voice-cloning or dubbing output without hitting a wall — and when something does go wrong, the error or docs should tell them exactly what to do.
@@ -16,7 +16,7 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
- **Default features must work on every platform (strict rule, 2026-05-20):** A feature that ships in default mode — out-of-the-box, no user customization, no opt-in toggle — must behave identically on macOS, Windows, and Linux. Platform-specific *implementation code* is allowed for OS APIs / shells / packaging, but the user-visible *default behavior* cannot diverge. Platform-only features (e.g., a macOS-only global shortcut, a Windows-only path picker) must go behind explicit user opt-in: Settings toggle, env var, or CLI flag. When a default doesn't work on a platform, that's a P0 bug — either fix it on the missing platform or move it behind opt-in. No third option.
- **Backward-compatible project data**: Existing `omnivoice_data/` (user voices, projects, settings) must keep working without manual migration. Any DB schema change goes through alembic with a tested upgrade path.
- **Local-first guarantee preserved**: Auto bug reporting (new addition) must be **opt-in**, must submit only to GitHub Issues (no third-party telemetry endpoint), and the app must remain fully functional with reporting disabled. No required cloud calls, accounts, or API keys.
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** the v0.3.x line has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main; the owner tags a patch (`v0.3.Z`) from main whenever the current state is worth cutting. No `-rc` tags. No phased release. No `v0.4` deferrals while the v0.3.x line is open — every open issue and every open community PR gets absorbed into the v0.3.x line or explicitly declined. Users follow `main` for previews; users wanting stable stay on the latest tagged release (currently **v0.3.5**). ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** v0.3.0 has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main. Tag `v0.3.0` once when the user calls "actually useful" — a qualitative bar, not a checklist. No `v0.3.0-rc1`. No phased release. No `v0.4` deferrals while v0.3.0 is open — every open issue and every open community PR gets absorbed into the v0.3.0 line or explicitly declined. Users follow `main` for previews; users wanting stable stay on `v0.2.7`. ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
<!-- GSD:project-end -->
<!-- GSD:stack-start source:research/STACK.md -->
@@ -190,20 +190,10 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
<!-- GSD:conventions-start source:CONVENTIONS.md -->
## Conventions
**Versioning (hard rule, owner-set 2026-06-11):** main is always **latest release + 1 patch**. The moment `vX.Y.Z` is released, main's version files (`frontend/src-tauri/tauri.conf.json`, `frontend/src-tauri/Cargo.toml`, `pyproject.toml`, **and `frontend/package.json`** — keep all **four** in lockstep; `package.json` drives the runtime `__APP_VERSION__` via vite, shown in the first-run footer + every auto bug report, so a drift ships a build that misreports its own version — guarded by `tests/test_app_version.py::test_all_version_files_in_lockstep`) bump to `X.Y.(Z+1)`. Consequences:
- Every PR and preview build identifies as the **next** version. Preview builds stamp `X.Y.(Z+1)-N` (run number), which semver-sorts **above** the last stable `X.Y.Z` — the updater ordering is natural, no comparator tricks needed.
- Releasing = tag `vX.Y.(Z+1)` from main (version files already match), then immediately bump main to `X.Y.(Z+2)`. The post-release bump is automated by the `version-bump` job in release.yml; if it fails, do it manually in the same day.
- Docker: `ghcr.io/debpalash/omnivoice-studio:latest` = **main** (rolling preview); `:X.Y.Z` + `:X.Y` + `:stable` = tagged releases. `:latest` is the preview channel by design — stable users pin `:stable` or a version tag.
- Do not bump minor/major or invent RCs/codenames without the owner asking. No "defer to next version" labels — scope is absorbed or declined, never re-versioned.
**Docs-sync (hard rule, owner-set 2026-06-11):** any change that alters something these docs describe — README.md, CONTRIBUTING.md, SECURITY.md, SUPPORT.md, LICENSE, or `docs/**` (install flows, Docker tag semantics, platform support, versioning/release behavior, review process, supported versions) — must update those docs **in the same PR** as the change. If a doc impact is discovered after merge, the docs fix is the immediate next commit, not backlog. Stale docs are treated as bugs.
**Versioning (hard rule):** Everything ships on `v0.3.0`. Never mention, suggest, or label anything with a version bump — no v0.4, no RCs, no "defer to next version", no future-version labels — unless the user explicitly asks to bump. Zero unprompted version chatter.
**Localization (hard rule):** No hardcoded non-English (CJK) **user-facing text** anywhere in the codebase except the translation layer (`frontend/src/i18n/`). All UI strings go through i18n (`t('...')` keys in `locales/*.json`); native language names live in `i18n/index.ts` (`LANGUAGES`). Functional CJK is allowed and tracked via the allowlist in `tests/test_no_hardcoded_cjk.py` — text-processing regexes, model/engine vocabulary & identifiers (e.g. CosyVoice speaker IDs), localized error matching, demo/eval data, and test fixtures. CI fails on any hardcoded CJK outside the allowlist; to add legitimate functional CJK, extend `_ALLOWED_FILES` there with a justification.
**Fix quality (hard rule, owner-set 2026-06-16):** Fix issues *properly* and future-maintenance-proof — don't stop at the symptom. Root-cause fully, fix the whole **class** of the bug (not just the one reported instance), add a fail-before/pass-after regression test, and harden against recurrence (e.g. if a lockfile drift only fails in Docker, also make CI catch it). Go the extra mile where it durably pays off. Be token-efficient about it — extra **effort**, not extra **verbosity**: no padding, no redundant re-checks, the smallest correct change that is also recurrence-proof. Don't be shy to spend the effort a proper fix needs; do be shy about wasting tokens.
**Keep main green (hard rule, owner-set 2026-06-16):** A merge must **never break `main`'s CI**. Before a change lands, verify the *full* CI matrix would pass — every workflow in `.github/workflows/` **and** `deploy/Dockerfile`, not only the checks you happened to run. Dependency / lockfile / config changes must be validated against **all** consumers. Specifically: `frontend/` is a bun **workspace monorepo** — the lockfile is the repo-root `bun.lock`, and `deploy/Dockerfile` runs `bun install --frozen-lockfile`, so any `frontend/package.json` change requires regenerating root `bun.lock` and confirming `bun install --frozen-lockfile` passes (plain `bun install` in `ci.yml` silently tolerates drift, so CI-green ≠ Docker-green). Likewise re-check CodeQL/Security on code changes and the Tauri `cargo` build on Rust/dep changes.
Other conventions not yet established. Will populate as patterns emerge during development.
<!-- GSD:conventions-end -->
+1 -75
View File
@@ -42,22 +42,11 @@ This starts both services:
### Desktop App (Tauri)
```bash
bun run desktop # dev: hot-reload Tauri shell + backend
bun run desktop-prod # production: builds, bundles the backend, then launches
bun run desktop
```
Both run `uv sync` first (so the Python backend env is set up) and start the
backend automatically — you do **not** start it separately. Use the exact script
names: there is no `desktop=prod` (note the **hyphen** in `desktop-prod`).
`desktop-prod` is Windows-aware (auto-detects bash/git; see `scripts/desktop-prod.mjs`).
Requires [Rust](https://rustup.rs/) and platform-specific Tauri dependencies — see the [Tauri prerequisites](https://v2.tauri.app/start/prerequisites/).
If the app opens but stays on the **setup splash with no buttons**, the Python
backend didn't finish starting — the splash surfaces the stall reason, a log
panel, and a **Retry** button (and Settings → Logs → Backend has the full trace).
The most common from-source cause is `uv` or Python not being on your PATH.
---
## Project Structure
@@ -203,69 +192,6 @@ cd frontend/src-tauri && cargo check
---
---
## What code review looks like
Every PR is reviewed by two AI reviewers before a human looks at it:
- **CodeRabbit** posts a walkthrough (with a sequence diagram, and an ASCII
before/after sketch for UI changes), inline findings, and warning-mode
pre-merge checks against the project's hard rules.
- **Greptile** reviews with the same project rubrics and learns from 👍/👎
reactions on its comments — react to train it.
Both are advisory, not gating: CI and the maintainer's approval decide. Don't
be surprised by detailed bot comments minutes after you open a PR — address
what's right, push back (in a reply) on what's wrong.
**Commit & PR conventions:** conventional-commit style with a scope
(`fix(dub): …`, `feat(setup): …`) and link the issue (`Closes #N` / `Refs #N`)
in the title or body.
## Quality gates your PR must pass
- **Cross-platform parity (hard rule):** anything that ships in default mode
must behave identically on macOS, Windows, and Linux. Platform-specific
*implementation* is fine; platform-divergent *default behavior* is a P0.
Platform-only features go behind an explicit opt-in (Settings toggle, env
var, or CLI flag).
- **i18n — all 21 locales (hard rule):** every user-facing string goes through
`t('...')` and the key must exist in **all 21** files under
`frontend/src/i18n/locales/`. Translate; don't copy English into non-English
locales. CI fails on hardcoded CJK outside the allowlist in
`tests/test_no_hardcoded_cjk.py` (extend `_ALLOWED_FILES` with a
justification for legitimate functional CJK).
- **DB schema changes** go through an alembic migration with a tested upgrade
path — existing `omnivoice_data/` must keep working with no manual steps.
- **Engine back-compat:** already-installed engines (model weights on disk)
must not require reinstall or re-download.
- **Local-first:** no new outbound calls except GitHub Issues (opt-in
reporting) and HuggingFace model downloads. Never log or persist secrets or
absolute home paths.
- **Security posture:** the backend serves loopback HTTP — treat every
query/path/form parameter as hostile. User-chosen filesystem destinations
are authorized in the Tauri process (save dialog), never via HTTP params.
## Contribution licensing
OmniVoice Studio is **AGPL-3.0-only**, and the maintainer also offers a
**commercial license** (see [LICENSE](LICENSE)). By submitting a contribution
you agree that:
1. you have the right to submit it (your own work, or compatibly licensed);
2. it is licensed to the project under **AGPL-3.0**; and
3. you grant the project maintainer a perpetual, worldwide, non-exclusive
right to also distribute your contribution under the project's commercial
license terms.
This inbound grant is what keeps the dual-license model viable. If you can't
agree to (3) for a particular contribution, say so in the PR and we'll discuss
before merging. Adding a `Signed-off-by:` line (DCO) to your commits is
appreciated but not required.
---
## Need Help?
- **Stuck on setup?** Ask in [Discord #help](https://discord.gg/bzQavDfVV9)
+6 -9
View File
@@ -24,11 +24,10 @@
</p>
<p>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
<!-- Pre-built macOS bundle is Apple Silicon. Intel Macs: build from source (docs/install/macos.md); a pre-built Intel target is tracked in #279. -->
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_aarch64.dmg"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_x64_en-US.msi"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_amd64.AppImage"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_amd64.deb"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
</p>
<p>
<sub><b>macOS:</b> first launch needs a one-time approval — right-click → <b>Open</b> (or System Settings → Privacy &amp; Security → <b>"Open Anyway"</b> on macOS 15). No Terminal needed. <a href="docs/install/macos.md#gatekeeper-quarantine">Why?</a></sub>
@@ -124,7 +123,7 @@ Per-OS install guides — pick yours and follow it end-to-end:
- **macOS** — [docs/install/macos.md](docs/install/macos.md)
- **Windows** — [docs/install/windows.md](docs/install/windows.md)
- **Linux** — [docs/install/linux.md](docs/install/linux.md)
- **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
- **Docker** — [docs/install/docker.md](docs/install/docker.md)
Stuck? Run the built-in self-check first — **Settings → About → "Run
self-check"** in the app, or `uv run python backend/main.py --diagnose` from
@@ -137,9 +136,7 @@ bundle"** packages scrubbed logs + the self-check report for bug reports.
For Hugging Face token setup, see
[docs/setup/huggingface-token.md](docs/setup/huggingface-token.md). For
diarization-specific gating, see
[docs/features/diarization.md](docs/features/diarization.md). For download
speed, the ⚡ fast-download (Xet) status, and restricted-network / mirror
options, see [docs/downloading-models.md](docs/downloading-models.md).
[docs/features/diarization.md](docs/features/diarization.md).
## Screenshots
+4 -14
View File
@@ -2,20 +2,10 @@
## Supported Versions
| Version | Supported |
|---------|-----------|
| 0.3.x (latest release + `main` previews) | ✅ Current — all fixes land here |
| 0.2.7 | ⚠️ Legacy stable — security fixes only, upgrade recommended |
| < 0.2.7 | ❌ No longer supported |
## Model supply chain
OmniVoice supports models from **public, verifiable sources only** (Hugging
Face repos, official project releases). Privately sold or gated model files
are not supported: an archive from a private source can carry anything
(bundled executables, modified configs), and nobody else can verify or
reproduce it. Treat any privately distributed model file as an untrusted
download, and never run executables bundled with model archives.
| Version | Supported |
|---------|--------------------|
| 0.2.x | ✅ Current release |
| < 0.2 | ❌ No longer supported |
## Reporting a Vulnerability
-36
View File
@@ -1,36 +0,0 @@
# Support
## Where to get help
| Channel | Best for |
|---|---|
| [Discord](https://discord.gg/bzQavDfVV9) — `#help` | Setup problems, quick questions, sharing results |
| [GitHub Issues](https://github.com/debpalash/OmniVoice-Studio/issues) | Bugs and feature requests — use the templates; attach the diagnostic bundle (Settings → About → "Save diagnostic bundle") |
| [GitHub Discussions](https://github.com/debpalash/OmniVoice-Studio/discussions) | Design questions, ideas, show & tell |
| Security issues | **Never a public issue** — see [SECURITY.md](SECURITY.md) for private reporting |
## Model sources we support
OmniVoice is built on the idea that everything it runs is **open and available
to everyone**: free, public models with verifiable sources and licenses
(Hugging Face repos, official project releases), so the whole community can
use, test, and debug the same thing.
**We do not support privately sold, paywalled, or gated model files.** A model
delivered privately can't be verified, reproduced, or shared — it doesn't fit
the project's goals, and issues involving such models will be politely closed.
As a general safety rule, never run executables bundled inside any model
archive.
## Before filing a bug
1. Update to the latest release (or `main` if you follow previews) — fixes ship continuously.
2. Run the in-app self-check: **Settings → About → Run self-check**.
3. Search existing issues; add a 👍 + your details to an existing one rather than opening a duplicate.
## Response expectations
This is an open-source project maintained with the help of an automated triage
bot: issues are typically triaged within hours and every report gets a human-
approved response. Reproducible reports with a diagnostic bundle get fixed
fastest.
-32
View File
@@ -7,13 +7,9 @@ composed at the route or router level without surprises.
Currently exposed:
- `require_loopback`: 403 unless the request came from a loopback origin
(bypassed in explicit server mode — see `_server_mode`).
- `ws_remote_authorized`: whether a WebSocket handshake from a non-loopback
client carries the remote API key (Wave 2.3) — used by WS endpoints that
keep their own inline loopback guards.
"""
import os
import secrets
from fastapi import HTTPException, Request
@@ -75,31 +71,3 @@ def require_loopback(request: Request) -> None:
if _server_mode():
return
raise HTTPException(status_code=403, detail="loopback origin required")
def remote_api_key() -> str | None:
"""The remote-backend bearer key (Wave 2.3), or None when remote mode is
off. Read at call time so tests can monkeypatch the env."""
return os.environ.get("OMNIVOICE_API_KEY") or None
def ws_remote_authorized(websocket) -> bool:
"""Whether a WebSocket handshake presents the remote API key.
Browser WebSockets cannot set an Authorization header, so the key may
arrive as ``?api_key=`` or via the ``ov_key`` cookie that the bearer
middleware sets on the first authenticated HTTP request. Returns False
when remote mode is off — callers keep their loopback-only behavior.
"""
key = remote_api_key()
if not key:
return False
auth = websocket.headers.get("authorization", "")
supplied = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
if not supplied:
supplied = (
websocket.query_params.get("api_key")
or websocket.cookies.get("ov_key")
or ""
)
return secrets.compare_digest(supplied, key)
-712
View File
@@ -1,712 +0,0 @@
"""Audiobook creator endpoints (parity Wave 5).
``POST /audiobook/plan`` — pure preview: parse a chapter-delimited script
(Markdown ``# H1`` chapters, inline ``[voice:NAME]`` / ``[pause …]``) into the
chapter/span plan, no synthesis.
``POST /audiobook`` — the synth job: render each chapter through the active TTS
backend (reusing ``services.audiobook.synthesize_chapter`` + ``chunked_tts``),
then mux the chapter WAVs into a chapterized **m4b** (FFMETADATA1 chapters via
``build_m4b_cmd``). Progress streams as Server-Sent Events, mirroring the dub
pipeline. ffmpeg-gated — without ffmpeg the job reports an error event and
stops (the m4b is the only output format).
``GET /audiobook/jobs`` + ``POST /audiobook/resume/{job_id}`` — durable
crash-resume: an interrupted render persists its plan + params to a
``resume.json`` manifest in the job work dir, so it can be resumed later (the
content-addressed chapter cache makes finished chapters instant) even without
the original script. The resume UI affordance remains a follow-up.
epub/pdf ingest, ACX mastering shipped; the resume UI surface remains a follow-up.
"""
import asyncio
import json
import logging
import os
import re
import uuid
from fastapi import APIRouter, File, HTTPException, UploadFile
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from services.audiobook import (
parse_audiobook_script,
synthesize_chapter,
)
from services.longform_render import (
LOUDNESS_PRESETS,
build_concat_list,
build_ffmetadata,
build_render_cmd,
prune_cache_dir,
)
from services import longform_resume # pure (no torch) — durable resume manifest
logger = logging.getLogger("omnivoice.audiobook")
router = APIRouter()
# A cover filename as produced by /audiobook/cover: 12 hex chars + image ext.
# An exact-match allowlist is the strongest barrier (and the one CodeQL's
# path-injection query recognizes) — anything else is rejected outright.
_COVER_NAME_RE = re.compile(r"^[0-9a-f]{12}\.(?:jpg|jpeg|png)$")
def _safe_cover_path(cover_path: str | None) -> str | None:
"""Confine a user-supplied cover to the upload directory before it can flow
into ffmpeg.
Covers only ever come from ``/audiobook/cover``, which writes them to
``OUTPUTS_DIR/audiobook_covers`` with a generated name. We rebuild the path
from the basename alone (``os.path.basename`` strips any directory component
or ``..`` traversal) joined onto that fixed directory, so no caller-supplied
path — absolute or relative — can escape it. Returns the path only if the
file actually exists there, else None."""
if not cover_path:
return None
from core.config import OUTPUTS_DIR
name = os.path.basename(cover_path)
if not _COVER_NAME_RE.match(name):
return None # not a name the upload endpoint could have produced
cover_dir = os.path.realpath(os.path.join(OUTPUTS_DIR, "audiobook_covers"))
real = os.path.realpath(os.path.join(cover_dir, name))
# Containment check on the resolved path itself — it must live inside the
# covers dir. Belt-and-suspenders over the regex+basename above; the
# commonpath form is the path-injection barrier static analysis recognizes.
if os.path.commonpath([real, cover_dir]) != cover_dir:
return None
return real if os.path.isfile(real) else None
class AudiobookPlanRequest(BaseModel):
text: str
default_voice: str | None = None
@router.post("/audiobook/plan")
def audiobook_plan(req: AudiobookPlanRequest) -> dict:
"""Parse a script into a chapter/span plan (pure preview, no synthesis)."""
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
return plan.to_dict()
#: Cover size cap mirrors longform_render's guard (8 MB — a book cover, not a
#: payload). Kept in sync intentionally; the render builder re-validates too.
_COVER_MAX_BYTES = 8 * 1024 * 1024
#: Import upload cap — a generous ceiling for a .txt/.md/.epub manuscript that
#: still stops a memory-exhaustion upload (the whole file is read into RAM).
_IMPORT_MAX_BYTES = 64 * 1024 * 1024
#: Upper bound on chapters in a single /longform/render plan — far above any real
#: book, but stops a pathological request from allocating/holding the job forever.
_MAX_CHAPTERS = 10_000
@router.post("/audiobook/import")
async def audiobook_import(file: UploadFile = File(...)) -> dict:
"""Import a ``.txt``/``.md``/``.epub``/``.pdf`` into a chapter-delimited script.
EPUB is parsed in spine order (stdlib only, local); PDF text is extracted
with pypdf (pure-Python) then chapterized; plain text gets ``# `` headings
inserted ahead of obvious chapter-title lines. Returns the script text (for
the editor) + the resulting chapter count."""
from services.longform_import import (
chapterize_plaintext,
epub_to_chapter_script,
pdf_to_chapter_script,
)
name = (file.filename or "").lower()
data = await file.read()
if not data:
raise HTTPException(status_code=400, detail="empty file")
if len(data) > _IMPORT_MAX_BYTES:
raise HTTPException(status_code=400, detail="file too large (max 64 MB)")
if name.endswith(".epub"):
try:
script = epub_to_chapter_script(data)
except ValueError as e:
raise HTTPException(status_code=400, detail=f"couldn't parse EPUB: {e}")
elif name.endswith(".pdf"):
try:
script = pdf_to_chapter_script(data)
except ValueError as e:
raise HTTPException(status_code=400, detail=f"couldn't parse PDF: {e}")
else:
script = chapterize_plaintext(data.decode("utf-8", "ignore"))
if not script.strip():
raise HTTPException(status_code=400, detail="no text found in the file")
plan = parse_audiobook_script(script)
return {"text": script, "chapters": plan.chapter_count}
@router.post("/audiobook/cover")
async def audiobook_cover(cover: UploadFile = File(...)) -> dict:
"""Upload a cover image; returns a server-side ``path`` to pass back as
``cover_path`` in the synth request. Validated here (jpg/png + size cap) and
again at render time."""
from core.config import OUTPUTS_DIR
ext = os.path.splitext(cover.filename or "")[1].lower()
if ext not in (".jpg", ".jpeg", ".png"):
raise HTTPException(status_code=400, detail="cover must be a .jpg or .png")
data = await cover.read()
if not data or len(data) > _COVER_MAX_BYTES:
raise HTTPException(status_code=400, detail="cover must be between 1 byte and 8 MB")
cover_dir = os.path.join(OUTPUTS_DIR, "audiobook_covers")
os.makedirs(cover_dir, exist_ok=True)
path = os.path.join(cover_dir, f"{uuid.uuid4().hex[:12]}{ext}")
with open(path, "wb") as f:
f.write(data)
return {"path": path}
class AudiobookRequest(BaseModel):
text: str
default_voice: str | None = None # voice profile id; None = engine default
bitrate: str = "128k"
format: str = "m4b" # "m4b" | "mp3"
loudness: str | None = None # None/"off" | "acx" | "podcast" (opt-in)
cover_path: str | None = None # server-side path to a jpg/png cover
# Global tags embedded in the output: {title, author, narrator, year,
# genre, description}. Player-visible (Apple Books / Audible read these).
metadata: dict | None = None
# Optional pronunciation lexicon {word: respelling} applied before synthesis.
lexicon: dict | None = None
def _resolve_voice(profile_id: str | None) -> dict:
"""Map a voice-profile id to (ref_audio, ref_text, instruct, seed).
Compact form of the resolver in generation.py — covers locked, design and
clone profiles. Returns all-None for the engine default (no profile).
"""
out = {"ref_audio": None, "ref_text": None, "instruct": None, "seed": None}
if not profile_id:
return out
from core.config import VOICES_DIR
from core.db import db_conn
with db_conn() as conn:
row = conn.execute("SELECT * FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
if not row:
return out
try:
kind = row["kind"] or "clone"
except (KeyError, IndexError):
kind = "clone"
if row["is_locked"] and row["locked_audio_path"]:
out["ref_audio"] = os.path.join(VOICES_DIR, row["locked_audio_path"])
out["ref_text"] = row["ref_text"]
out["instruct"] = row["instruct"]
elif kind == "design":
out["ref_audio"] = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
out["ref_text"] = row["ref_text"] if out["ref_audio"] else None
out["instruct"] = row["instruct"]
else:
out["ref_audio"] = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
out["ref_text"] = row["ref_text"]
out["instruct"] = row["instruct"]
try:
if row["seed"] is not None:
out["seed"] = row["seed"]
except (KeyError, IndexError):
pass
return out
def _build_synth(default_voice: str | None) -> dict:
"""Describe how to synthesize for the active TTS engine.
Returns a dict with ``mode``, ``resolve`` (voice-id → resolved refs, cached
per id) and ``engine_id``. For OmniVoice it also carries the async
``get_model``; other engines carry a ready ``synth`` + ``sample_rate``.
:func:`_prepare_synth` turns this into a uniform ``(synth, sr, resolve,
engine_id)`` once the (async) model is in hand.
"""
from services.tts_backend import OmniVoiceBackend, active_backend_id, get_backend_class
cache: dict = {}
def resolve(voice_id):
key = voice_id or default_voice
if key not in cache:
cache[key] = _resolve_voice(key)
return cache[key]
engine_id = active_backend_id()
cls = get_backend_class(engine_id)
if cls is OmniVoiceBackend:
from services.model_manager import get_model
return {"mode": "omnivoice", "resolve": resolve,
"engine_id": engine_id, "get_model": get_model}
backend = cls()
def synth(text, voice_id, speed=None):
v = resolve(voice_id)
return backend.generate(
text, language=None, ref_audio=v["ref_audio"],
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
speed=float(speed) if speed else 1.0,
)
return {"mode": "generic", "resolve": resolve, "engine_id": engine_id,
"synth": synth, "sample_rate": backend.sample_rate}
async def _prepare_synth(default_voice: str | None):
"""Resolve :func:`_build_synth` into ``(synth, sample_rate, resolve,
engine_id)`` — awaiting the OmniVoice model load when needed. Shared by the
full job and the per-chapter preview."""
info = _build_synth(default_voice)
resolve, engine_id = info["resolve"], info["engine_id"]
if info["mode"] == "omnivoice":
model = await info["get_model"]()
sr = getattr(model, "sampling_rate", 24000)
def synth(text, voice_id, speed=None):
v = resolve(voice_id)
return model.generate(
text=text, language=None, ref_audio=v["ref_audio"],
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
speed=float(speed) if speed else 1.0,
)[0]
return synth, sr, resolve, engine_id
return info["synth"], info["sample_rate"], resolve, engine_id
def _render_chapter_cached(chapter, synth, sr, engine_id, resolve, cache_dir, lexicon=None):
"""Render one chapter, content-addressed so a re-run reuses it (resume).
Returns ``(wav_path, duration_s, was_cached)``. The WAV lives at
``cache_dir/<key>.wav`` where ``key`` is :func:`chapter_cache_key` over the
chapter's spans + sample rate + engine + each voice's resolved signature
(+ the lexicon, so a lexicon edit re-renders), so an unchanged chapter is
never re-synthesized. Runs in the GPU-pool executor.
"""
import json
import wave
from services.audio_io import atomic_save_wav
from services.longform_render import chapter_cache_key
from services.pronunciation import normalize_lexicon
spans_tuples = [(s.voice_id, s.text, s.pause_ms_after, getattr(s, "speed", None))
for s in chapter.spans]
sig: dict = {}
for s in chapter.spans:
k = s.voice_id or ""
if k not in sig:
v = resolve(s.voice_id)
sig[k] = f"{v.get('ref_audio')}|{v.get('ref_text')}|{v.get('instruct')}|{v.get('seed')}"
if lexicon:
# Fold the lexicon into the cache key so editing pronunciations
# invalidates cached chapters (reserved key can't collide with a voice id).
sig["\x00lexicon"] = json.dumps(normalize_lexicon(lexicon), sort_keys=True)
key = chapter_cache_key(spans_tuples, sample_rate=sr, engine_id=engine_id, voice_sig=sig)
wav_path = os.path.join(cache_dir, f"{key}.wav")
if os.path.exists(wav_path):
try:
with wave.open(wav_path, "rb") as w:
dur = w.getnframes() / float(w.getframerate() or sr)
return wav_path, dur, True
except Exception:
pass # corrupt cache entry — fall through and re-render
audio, dur = synthesize_chapter(chapter.spans, synth, sr, lexicon=lexicon)
atomic_save_wav(wav_path, audio, sr)
return wav_path, dur, False
class AudiobookPreviewRequest(BaseModel):
text: str
chapter_index: int = 0
default_voice: str | None = None
lexicon: dict | None = None
@router.post("/audiobook/preview")
async def audiobook_preview(req: AudiobookPreviewRequest) -> dict:
"""Render a single chapter so the user can audition it before the full run.
Reuses the same content-addressed cache as the job, so a preview warms the
cache (the later full render reuses it) and a re-preview is instant.
"""
from core.config import OUTPUTS_DIR
from services.model_manager import _gpu_pool
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
if not plan.chapters:
raise HTTPException(status_code=400, detail="no chapters parsed from the script")
n = len(plan.chapters)
if not (0 <= req.chapter_index < n):
raise HTTPException(status_code=400, detail=f"chapter_index out of range (0..{n - 1})")
chapter = plan.chapters[req.chapter_index]
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache") # shared with _render_longform_sse
os.makedirs(cache_dir, exist_ok=True)
synth, sr, resolve, engine_id = await _prepare_synth(req.default_voice)
loop = asyncio.get_running_loop()
wav_path, dur, was_cached = await loop.run_in_executor(
_gpu_pool, _render_chapter_cached, chapter, synth, sr, engine_id, resolve, cache_dir,
req.lexicon,
)
return {
"output": os.path.relpath(wav_path, OUTPUTS_DIR), # served via /audio
"duration_s": round(dur, 2),
"cached": was_cached,
"title": chapter.title,
}
async def _render_longform_sse(
plan,
*,
default_voice: str | None,
fmt: str = "m4b",
bitrate: str = "128k",
loudness: str | None = None,
cover_path: str | None = None,
metadata: dict | None = None,
lexicon: dict | None = None,
job_type: str = "audiobook",
job_id: str | None = None,
resume: bool = False,
):
"""Shared chapterized-render SSE generator for Audiobook *and* Stories.
Takes a ready ``plan`` (``.chapters`` → ``.title`` + ``.spans``) — Audiobook
parses it from a script, Stories compiles it from cast/lines — and renders
each chapter (content-addressed cache → resume), isolating per-chapter
failures, then muxes the successful chapters into a tagged file. This is the
convergence point: one renderer, two front doors.
"""
from core.config import OUTPUTS_DIR
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
from services.model_manager import _gpu_pool
# Resume reuses the original job_id (continuing the same job row + cached
# chapters); a fresh render generates a new one. The id may arrive from the
# /resume/{job_id} path param, so strip it to a safe token (no path
# separators, no CR/LF) before it ever reaches a filesystem path or a log
# line — CodeQL py/path-injection + py/log-injection. Empty after the strip
# → a fresh id.
job_id = re.sub(r"[^A-Za-z0-9_-]", "", job_id or "")[:64] or uuid.uuid4().hex[:16]
try:
from core import job_store
if not resume:
job_store.create(job_id, type=job_type)
job_store.mark_running(job_id)
except Exception:
job_store = None # job history is best-effort; never block synthesis
# Persist a durable resume manifest (plan + params) so an interrupted render
# can be resumed later even without the original script. Best-effort.
try:
title = (metadata or {}).get("title") or (plan.chapters[0].title if plan.chapters else "")
longform_resume.write_manifest(longform_resume.build_manifest(
job_id=job_id, job_type=job_type, title=title,
plan_chapters=[
{"title": c.title, "spans": [s.to_dict() for s in c.spans]}
for c in plan.chapters
],
params={
"default_voice": default_voice, "fmt": fmt, "bitrate": bitrate,
"loudness": loudness, "cover_path": cover_path,
"metadata": metadata, "lexicon": lexicon,
},
))
except Exception: # resume durability is an enhancement; never block the render
logger.debug("[%s] resume manifest write skipped", job_id, exc_info=True)
def _emit(payload: dict) -> str:
if job_store is not None:
try:
job_store.append_event(job_id, json.dumps(payload))
except Exception:
pass # best-effort job history; never block the stream
return f"data: {json.dumps(payload)}\n\n"
if not plan.chapters:
yield _emit({"type": "error", "error": "nothing to render (no chapters)"})
return
ffmpeg = find_ffmpeg()
if not ffmpeg:
yield _emit({"type": "error", "error": "ffmpeg not available; the output needs it"})
return
# Confined work dir (job_id is already token-sanitized above; work_dir adds
# the basename + realpath barrier so CodeQL sees a clean path).
work = longform_resume.work_dir(job_type, job_id)
if work is None:
yield _emit({"type": "error", "error": "invalid job id"})
return
os.makedirs(work, exist_ok=True)
# Chapter WAVs are content-addressed in a shared cache so a re-run (after a
# failure or interruption) reuses what already rendered — only the
# missing/changed chapters synthesize again (resume). Shared across both
# front doors: an identical chapter renders once.
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache")
os.makedirs(cache_dir, exist_ok=True)
prune_cache_dir(cache_dir) # bound disk before this job adds its chapters
loop = asyncio.get_running_loop()
try:
synth, sr, resolve, engine_id = await _prepare_synth(default_voice)
total = len(plan.chapters)
chapter_files: list[str] = []
chapters_meta: list[tuple[str, int]] = []
cached_n = 0
failed: list[int] = []
yield _emit({"type": "started", "job_id": job_id, "chapters": total})
for i, chapter in enumerate(plan.chapters):
try:
wav_path, dur, was_cached = await loop.run_in_executor(
_gpu_pool, _render_chapter_cached,
chapter, synth, sr, engine_id, resolve, cache_dir, lexicon,
)
except Exception: # isolate a bad chapter — keep going
logger.warning("[%s] chapter %d (%s) failed to render",
job_id, i, chapter.title, exc_info=True)
failed.append(i)
yield _emit({"type": "chapter_error", "index": i, "total": total,
"title": chapter.title, "error": "chapter failed to render"})
continue
chapter_files.append(wav_path)
chapters_meta.append((chapter.title, int(round(dur * 1000))))
cached_n += 1 if was_cached else 0
yield _emit({"type": "chapter", "index": i, "total": total,
"title": chapter.title, "duration_s": round(dur, 2),
"cached": was_cached})
if not chapter_files:
yield _emit({"type": "error", "error": "all chapters failed to render"})
return
yield _emit({"type": "assembling"})
meta_path = os.path.join(work, "chapters.ffmeta")
with open(meta_path, "w", encoding="utf-8") as f:
f.write(build_ffmetadata(chapters_meta, global_meta=metadata))
concat_path = os.path.join(work, "concat.txt")
with open(concat_path, "w", encoding="utf-8") as f:
f.write(build_concat_list(chapter_files))
ext = "mp3" if (fmt or "").lower() == "mp3" else "m4b"
out_name = f"{job_type}_{job_id}.{ext}"
out_path = os.path.join(OUTPUTS_DIR, out_name)
# Two-pass loudness master (#28): for a known preset, measure the
# concatenated program first, then feed the measured values back into the
# single mux encode. `measured is None` (skip OR any failure) → the mux
# falls back to single-pass. Gated identically to the pure builders
# (.lower(), no strip), so off/None/unknown/whitespace skip cleanly.
measured = None
norm = (loudness or "").lower()
if norm in LOUDNESS_PRESETS:
yield _emit({"type": "mastering", "preset": norm})
from services.loudness import measure_loudness
measured = await measure_loudness(ffmpeg, concat_path, norm, job_id=job_id)
await run_ffmpeg(
build_render_cmd(
ffmpeg, concat_path, meta_path, out_path,
fmt=ext, bitrate=bitrate, cover_path=_safe_cover_path(cover_path),
loudness=loudness, measured=measured,
),
job_id=job_id,
)
if job_store is not None:
try:
job_store.mark_done(job_id)
except Exception:
pass # best-effort job history
# The render finished — drop the resume manifest so this job is no longer
# offered for resume.
longform_resume.clear_manifest(job_type, job_id)
total_s = sum(d for _, d in chapters_meta) / 1000.0
done = {"type": "done", "output": out_name,
"chapters": len(chapter_files), "duration_s": round(total_s, 2),
"cached_chapters": cached_n, "failed_chapters": failed}
# Loudness verdict only when a preset was requested — off/None paths keep
# the exact legacy `done` shape (additive, old clients unaffected).
if norm in LOUDNESS_PRESETS:
p = LOUDNESS_PRESETS[norm]
done["loudness"] = {
"preset": norm, "target_i": p.i, "target_tp": p.tp,
"two_pass": measured is not None,
"measured_i": measured.input_i if measured else None,
}
yield _emit(done)
except Exception as e: # surface, don't 500 the stream
logger.exception("[%s] longform render failed", job_id)
if job_store is not None:
try:
job_store.mark_failed(job_id, str(e))
except Exception:
pass # best-effort job history
# Generic message only — don't leak the stack/exception text to the client.
yield _emit({"type": "error", "error": "render failed (see backend log)"})
@router.post("/audiobook")
async def audiobook_synthesize(req: AudiobookRequest):
"""Synthesize a chapterized audiobook from a script, streaming SSE progress."""
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
return StreamingResponse(
_render_longform_sse(
plan, default_voice=req.default_voice, fmt=req.format, bitrate=req.bitrate,
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
lexicon=req.lexicon, job_type="audiobook",
),
media_type="text/event-stream",
)
# ── Shared longform render: Stories (and any future front door) post a plan ──
class LongformSpan(BaseModel):
voice_id: str | None = None
text: str
pause_ms_after: int = 0
speed: float | None = None
class LongformChapter(BaseModel):
title: str = ""
spans: list[LongformSpan] = []
class LongformRenderRequest(BaseModel):
chapters: list[LongformChapter] = []
default_voice: str | None = None
bitrate: str = "128k"
format: str = "m4b"
loudness: str | None = None
cover_path: str | None = None
metadata: dict | None = None
lexicon: dict | None = None
@router.post("/longform/render")
async def longform_render(req: LongformRenderRequest):
"""Render a pre-built chapter/span plan (the Stories Editor's compiled
cast+lines) through the shared chapterized renderer — same resume, loudness,
cover, metadata, and output formats as the Audiobook job."""
from services.audiobook import AudiobookPlan, Chapter, Span
if len(req.chapters) > _MAX_CHAPTERS:
raise HTTPException(status_code=422, detail=f"too many chapters (max {_MAX_CHAPTERS})")
chapters = []
for i, c in enumerate(req.chapters):
# Keep a span if it has text to speak OR a pause to render (pause-only
# spans carry inter-line silence with empty text).
spans = [Span(voice_id=s.voice_id, text=(s.text or "").strip(),
pause_ms_after=max(0, int(s.pause_ms_after)), speed=s.speed)
for s in c.spans if ((s.text and s.text.strip()) or s.pause_ms_after > 0)]
if spans:
chapters.append(Chapter(title=c.title or f"Chapter {i + 1}", spans=spans))
plan = AudiobookPlan(chapters=chapters)
return StreamingResponse(
_render_longform_sse(
plan, default_voice=req.default_voice, fmt=req.format, bitrate=req.bitrate,
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
lexicon=req.lexicon, job_type="story",
),
media_type="text/event-stream",
)
# ── Durable resume: interrupted longform renders ────────────────────────────
def _chapters_done(job_id: str) -> int:
"""Count chapters that finished rendering, from the job's persisted events.
Best-effort (0 if unavailable) — used only to show resume progress."""
try:
from core import job_store
n = 0
for ev in job_store.events_since(job_id, 0, limit=100_000):
try:
if json.loads(ev["payload"]).get("type") == "chapter":
n += 1
except (ValueError, KeyError, TypeError):
continue
return n
except Exception:
return 0
@router.get("/audiobook/jobs")
def list_resumable_jobs() -> dict:
"""List interrupted longform renders that can be resumed — a work dir that
still holds a resume manifest (a job left mid-render by a crash/quit). The
ids come from scanning the filesystem, so the UI can offer one-click resume."""
from core import job_store
out = []
for e in longform_resume.scan_resumable():
jid = e["job_id"]
manifest = longform_resume.load_manifest_file(e["manifest_path"]) or {}
job = job_store.get(jid) or {}
out.append({
"job_id": jid,
"type": e["job_type"],
"status": job.get("status", "interrupted"),
"title": manifest.get("title", ""),
"total_chapters": manifest.get("total_chapters", 0),
"chapters_done": _chapters_done(jid),
"created_at": job.get("created_at"),
})
return {"jobs": out}
@router.post("/audiobook/resume/{job_id}")
async def resume_longform(job_id: str):
"""Resume an interrupted longform render from its persisted manifest. The
already-rendered chapters are content-addressed in the shared cache, so they
return instantly — only the unrendered chapters synthesize again. Streams the
same SSE event shape as the original render, under the original job_id."""
from services.audiobook import AudiobookPlan, Chapter, Span
# Find the requested job among the trusted filesystem scan (every path there
# is os.listdir-sourced, never request input) and read its manifest via the
# scan's own trusted path — the request job_id is used ONLY to *select* an
# entry, never to build a path. No request-controlled value reaches a file
# operation (CodeQL py/path-injection-safe).
entry = next((e for e in longform_resume.scan_resumable()
if e["job_id"] == job_id), None)
if entry is None:
raise HTTPException(status_code=404, detail="No resumable job for that id")
manifest = longform_resume.load_manifest_file(entry["manifest_path"])
if not manifest:
raise HTTPException(status_code=404, detail="No resume manifest for that job")
chapters = [
Chapter(title=c.get("title", ""),
spans=[Span(**s) for s in c.get("spans", [])])
for c in manifest["plan"]
]
plan = AudiobookPlan(chapters=chapters)
p = manifest.get("params", {})
# Retire the interrupted job's manifest (trusted scan path) so it stops
# showing as resumable once we've kicked off the fresh-id resume.
longform_resume.discard_manifest_file(entry["manifest_path"])
# Resume under a FRESH job id (job_id=None → a server uuid in the renderer).
# The chapter cache is content-addressed (keyed by chapter content, not the
# job id), so the already-rendered chapters still hit instantly — only the
# unrendered ones synthesize. Using a fresh id means the request's job_id
# never names a work dir / output file (defence-in-depth path-injection).
return StreamingResponse(
_render_longform_sse(
plan, default_voice=p.get("default_voice"),
fmt=p.get("fmt", "m4b"), bitrate=p.get("bitrate", "128k"),
loudness=p.get("loudness"), cover_path=p.get("cover_path"),
metadata=p.get("metadata"), lexicon=p.get("lexicon"),
job_type=entry["job_type"],
),
media_type="text/event-stream",
)
-3
View File
@@ -298,9 +298,6 @@ async def _run_batch_pipeline(job_id: str, job: dict):
denoise=True, postprocess_output=True,
)
audio_out = audios[0]
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered = apply_mastering(
audio_out,
sample_rate=sr,
+2 -38
View File
@@ -25,19 +25,12 @@ router = APIRouter()
logger = logging.getLogger("omnivoice.capture")
def _truthy(value: Optional[str]) -> bool:
"""Parse a multipart form flag. Treats '1'/'true'/'yes'/'on'/'auto'
(any case) as on; everything else — including None — as off."""
return (value or "").strip().lower() in {"1", "true", "yes", "on", "auto"}
@router.post("/transcribe")
async def transcribe_audio(
audio: UploadFile = File(...),
language: Optional[str] = Form(None),
model: Optional[str] = Form(None),
mode: Optional[str] = Form(None),
refine: Optional[str] = Form(None),
):
"""Transcribe an audio file to text.
@@ -47,19 +40,10 @@ async def transcribe_audio(
model: Whisper model size (legacy; ignored in dual-mode architecture).
mode: 'fast' (default) uses MLX Turbo for speed; 'accurate' uses
WhisperX with forced alignment for word-level timing.
refine: Opt-in local-LLM cleanup of the final text (disfluencies,
self-corrections, punctuation) — same pipeline the live
dictation socket uses. Off by default so MCP/CLI callers don't
pay LLM latency unless they ask; honours the user's
Settings → Dictation-refinement config and silently passes
through when no LLM backend is configured. The raw ``text``
is always returned; ``refined_text`` is added only when the
LLM actually changed something.
Returns:
{
"text": "full transcription",
"refined_text": "cleaned text", # only when refine=true changed it
"segments": [ {"start": 0.0, "end": 1.5, "text": "..."}, ... ],
"language": "en",
"duration_s": 4.2,
@@ -107,11 +91,6 @@ async def transcribe_audio(
if not full_text and segments:
full_text = " ".join(s.get("text", "") for s in segments).strip()
# Wave 1.1: strip Whisper hallucination loops from the final text.
# Segments keep the raw recognition so their timings stay truthful.
from services.refinement import collapse_repetitive_artifacts
full_text = collapse_repetitive_artifacts(full_text)
# Calculate audio duration from segments if available
duration = 0.0
if segments:
@@ -119,24 +98,12 @@ async def transcribe_audio(
detected_lang = result.get("language", language or "unknown")
# Opt-in Wave 2.1 refinement, mirroring the live-dictation socket
# (capture_ws). Off-thread (it's a network call, not GPU); never
# raises — maybe_refine swallows failures and a missing LLM into a
# None pass-through, so the raw text always stands.
refined_text = None
if _truthy(refine) and full_text:
from services.refinement import maybe_refine
refined = await asyncio.to_thread(maybe_refine, full_text)
if refined and refined != full_text:
refined_text = refined
logger.info(
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s, refined=%s",
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s",
engine_id, elapsed, duration, "accurate" if use_accurate else "fast",
refined_text is not None,
)
response = {
return {
"text": full_text,
"segments": [
{
@@ -151,9 +118,6 @@ async def transcribe_audio(
"transcription_time_s": elapsed,
"engine": engine_id,
}
if refined_text is not None:
response["refined_text"] = refined_text
return response
finally:
try:
os.unlink(tmp.name)
+8 -109
View File
@@ -8,12 +8,6 @@ live dictation feedback.
Protocol:
→ Client sends binary audio frames (16-bit PCM or WebM/Opus blobs)
← Server sends JSON messages:
Opt-in AEC mode (``?aec=1[&sr=16000]``, parity Action 8b): for dictating
while the app plays audio. Frames must be raw int16 mono PCM, each tagged
with a 1-byte prefix — 0x00 = microphone, 0x01 = playback reference. The
server runs an NLMS echo canceller, cleaning the mic against the reference
before transcription. Without the param the protocol is unchanged.
{"type": "partial", "text": "Hello wor..."} — interim result
{"type": "final", "text": "Hello world.", — committed result
"segments": [...], "language": "en",
@@ -31,7 +25,7 @@ import time
from fastapi import APIRouter, WebSocket, WebSocketDisconnect
from api.dependencies import _LOOPBACK_HOSTS, ws_remote_authorized
from api.dependencies import _LOOPBACK_HOSTS
router = APIRouter()
logger = logging.getLogger("omnivoice.capture_ws")
@@ -50,56 +44,6 @@ MIN_BUFFER_BYTES = 64000 # ~2s of 16-bit mono 16kHz — needs enough WebM frame
# to transcribe whatever the user recorded, even short utterances.
MIN_FINAL_BUFFER_BYTES = 4000 # ~125ms of 16-bit mono 16kHz
# ── Dictate-over-playback AEC (parity Action 8b, opt-in) ──────────────────
# Activated by the ``?aec=1`` query param. When OFF (the default), the
# protocol and behaviour are byte-for-byte unchanged. When ON, the client
# streams raw int16 mono PCM frames tagged with a 1-byte type prefix so the
# server can tell mic audio from the playback reference it must cancel:
_AEC_NEAR = 0x00 # microphone frame (clean it, then buffer for ASR)
_AEC_FAR = 0x01 # playback reference frame (feed the echo model only)
def _demux_aec_frame(data: bytes) -> tuple[str, bytes]:
"""Split a prefixed AEC binary frame into ``(kind, pcm)``.
``kind`` is ``"near"`` (mic) or ``"far"`` (playback reference). An empty
or prefix-only frame yields an empty payload. Unknown prefixes are treated
as ``"near"`` so a malformed tag degrades to plain dictation rather than
dropping audio.
"""
if not data:
return "near", b""
kind = "far" if data[0] == _AEC_FAR else "near"
return kind, data[1:]
def _pcm16_to_wav(pcm: bytes, sample_rate: int) -> str | None:
"""Write raw int16 mono PCM to a temp WAV via stdlib ``wave`` (no ffmpeg).
Used on the AEC path, where frames are already decoded PCM — the cleaned
samples have no container, so the ffmpeg-sniffing ``_chunks_to_wav`` would
misdetect them. Returns the temp path, or ``None`` for a too-short buffer.
"""
if not pcm or len(pcm) < 100:
return None
import wave
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".wav")
tmp.close()
try:
with wave.open(tmp.name, "wb") as wf:
wf.setnchannels(1)
wf.setsampwidth(2) # int16
wf.setframerate(sample_rate)
wf.writeframes(pcm)
return tmp.name
except Exception as e:
logger.debug("PCM->WAV failed: %s", e)
try:
os.unlink(tmp.name)
except OSError:
pass
return None
@router.websocket("/ws/transcribe")
async def ws_transcribe(websocket: WebSocket):
@@ -109,33 +53,13 @@ async def ws_transcribe(websocket: WebSocket):
# WebSocket dependency injection differs across FastAPI versions, so we
# inline the check before accept(). Without it, any local process could
# stream the user's microphone over this endpoint.
# Wave 2.3 (remote backend): a non-loopback client that presents the
# OMNIVOICE_API_KEY bearer is the thin-client dictation case — the mic
# lives on the user's machine, the GPU here — and is allowed through.
host = websocket.client.host if websocket.client else None
if host not in _LOOPBACK_HOSTS and not ws_remote_authorized(websocket):
if host not in _LOOPBACK_HOSTS:
await websocket.close(code=1008, reason="loopback origin required")
return
await websocket.accept()
# Opt-in dictate-over-playback AEC (parity Action 8b). Default OFF →
# identical legacy behaviour. When on, frames are 1-byte-tagged raw PCM
# and the cleaned mic stream is muxed via stdlib wave (not ffmpeg).
aec = None
pcm_sr: int | None = None
if websocket.query_params.get("aec") in ("1", "true", "on"):
try:
pcm_sr = int(websocket.query_params.get("sr", "16000"))
from services.aec import NlmsEchoCanceller
aec = NlmsEchoCanceller(sample_rate=pcm_sr)
logger.info("AEC enabled for dictation session (sr=%d)", pcm_sr)
except Exception as e:
# Bad sr or import failure → fall back to plain dictation.
logger.warning("AEC requested but disabled: %s", e)
aec = None
pcm_sr = None
audio_chunks: list[bytes] = []
total_bytes = 0
last_audio_time = time.monotonic()
@@ -173,16 +97,6 @@ async def ws_transcribe(websocket: WebSocket):
# Empty binary frame also acts as EOF — connection stays open.
running = False
break
if aec is not None:
# Tagged PCM: route the playback reference into the echo
# model and clean the mic before it reaches the buffer.
kind, payload = _demux_aec_frame(data)
if kind == "far":
aec.push_far_end(payload)
continue
if not payload:
continue
data = aec.process_near_end(payload)
audio_chunks.append(data)
total_bytes += len(data)
last_audio_time = time.monotonic()
@@ -229,7 +143,7 @@ async def ws_transcribe(websocket: WebSocket):
# Transcribe current buffer
try:
text = await _transcribe_buffer(audio_chunks[:], pcm_sr=pcm_sr)
text = await _transcribe_buffer(audio_chunks[:])
if text and text != partial_text:
partial_text = text
await _safe_send({
@@ -259,16 +173,7 @@ async def ws_transcribe(websocket: WebSocket):
# Final transcription on complete buffer — skip if client already gone.
if total_bytes > MIN_FINAL_BUFFER_BYTES:
try:
result = await _transcribe_buffer_full(audio_chunks, pcm_sr=pcm_sr)
# Wave 2.1: optional local-LLM refinement of the final text.
# Off-thread (network call, not GPU); pass-through on any
# failure or when no LLM backend is configured. The raw text
# always ships too — clients paste refined_text ?? text.
if result.get("text"):
from services.refinement import maybe_refine
refined = await asyncio.to_thread(maybe_refine, result["text"])
if refined and refined != result["text"]:
result["refined_text"] = refined
result = await _transcribe_buffer_full(audio_chunks)
if not await _safe_send({"type": "final", **result}):
logger.debug("Skipped final send — client already disconnected")
except Exception as e:
@@ -292,10 +197,10 @@ async def ws_transcribe(websocket: WebSocket):
pass
async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None) -> str:
async def _transcribe_buffer(chunks: list[bytes]) -> str:
"""Quick partial transcription of the current audio buffer."""
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
tmp = _chunks_to_wav(chunks)
if tmp is None:
return ""
@@ -318,9 +223,9 @@ async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None)
pass
async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = None) -> dict:
async def _transcribe_buffer_full(chunks: list[bytes]) -> dict:
"""Full transcription with timing info for the final result."""
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
tmp = _chunks_to_wav(chunks)
if tmp is None:
return {"text": "", "segments": [], "language": "unknown",
"duration_s": 0, "transcription_time_s": 0, "engine": "none"}
@@ -340,12 +245,6 @@ async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = N
if not full_text and segments:
full_text = " ".join(s.get("text", "") for s in segments).strip()
# Wave 1.1: strip Whisper hallucination loops from the final
# text (the string that gets auto-pasted). Segments keep the
# raw recognition so their timings stay truthful.
from services.refinement import collapse_repetitive_artifacts
full_text = collapse_repetitive_artifacts(full_text)
duration = max((s.get("end", 0) for s in segments), default=0.0)
return {
+3 -9
View File
@@ -258,19 +258,13 @@ async def community_use(item_id: str, name: Optional[str] = Query(None)):
raise HTTPException(status_code=503, detail=f"Couldn't add this voice right now. Error: {e}")
try:
# A community "preset" is a synthetic designed voice (rendered from an
# instruct string) → kind='design'; a "voice" carries a real reference
# clip → kind='clone'. Setting kind makes the persona-gallery
# synthetic-only gating work (§R3) instead of defaulting all imports to
# 'clone'.
kind = "design" if item["type"] == "preset" else "clone"
with db_conn() as conn:
conn.execute(
"INSERT INTO voice_profiles "
"(id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at, kind) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
"(id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(profile_id, profile_name, audio_filename, ref_text, instruct,
item.get("language", "Auto"), None, item["id"], time.time(), kind),
item.get("language", "Auto"), None, item["id"], time.time()),
)
except Exception:
with __import__("contextlib").suppress(OSError):
-35
View File
@@ -1,35 +0,0 @@
"""Voice-design "describe your voice" API (issue #317).
Maps a free-text voice description onto the existing design parameter space
via the deterministic keyword mapper in ``core.describe_voice``. Pure CPU +
stdlib no model, no network so it imports and responds instantly in any
environment, including test/CI without model weights.
"""
from __future__ import annotations
from fastapi import APIRouter
from pydantic import BaseModel, Field
from core.describe_voice import parse_description
router = APIRouter()
class DescribeRequest(BaseModel):
description: str = Field(default="", max_length=2000)
@router.post("/design/describe")
def describe_voice(req: DescribeRequest) -> dict:
"""Parse a free-text description into design attrs + a validator-safe instruct.
Response shape::
{
"attrs": {"Gender": "female", "Age": "elderly", ... or "Auto"},
"instruct": "female, elderly, low pitch, british accent",
"matched": [{"category": "Age", "token": "elderly", "phrase": "elderly"}, ...],
"unmatched": ["slightly raspy"]
}
"""
return parse_description(req.description)
+5 -72
View File
@@ -25,7 +25,6 @@ from services.segmentation import (
assign_speakers_heuristic,
clean_up_segments,
)
from services.onset_align import snap_segment_starts
from services import dub_pipeline
router = APIRouter()
@@ -367,11 +366,7 @@ _prep_event_helper = dub_pipeline.prep_event # alias; we keep the module-local
@router.get("/dub/transcribe-stream/{job_id}")
async def dub_transcribe_stream(
job_id: str,
num_speakers: Optional[int] = None,
per_segment_refs: bool = True,
):
async def dub_transcribe_stream(job_id: str, num_speakers: Optional[int] = None):
"""Stream per-chunk segments via SSE, then emit diarized final pass.
Pre-flight checks (missing job, missing audio, ASR not loaded) are emitted
@@ -553,15 +548,6 @@ async def dub_transcribe_stream(
detected_lang = part["language"]
asr_speaker_turns.extend(part.get("speaker_turns") or [])
chunk_segs = segment_transcript(part, duration=t1, scene_cuts=scene_cuts)
# #280: Whisper often stretches a segment's start back over
# leading music/silence (classic case: speech begins at 0:03,
# transcript says 0.0 → the dub plays 3 s early). Snap starts
# forward to the actual speech onset. `audio_np` is the same
# track ASR ran on — vocals.wav when Demucs succeeded.
try:
snap_segment_starts(chunk_segs, audio_np, sr)
except Exception as e:
logger.warning("onset alignment skipped for chunk %d: %s", i, e)
chunk_segs = assign_speakers_heuristic(chunk_segs)
for s in chunk_segs:
s["id"] = f"s{next_seg_id:05x}"
@@ -773,61 +759,16 @@ async def dub_transcribe_stream(
clones = done.pop().result()
break
yield _sse_event("ping", {})
# Wave 3.2: per-segment clone refs. Cut each long-enough segment's
# own reference from the vocals so the dub of each line matches the
# prosody of its source line. Short lines fall back to the
# per-speaker clone below. Default on; the user can force
# per-speaker by disabling it (job["per_segment_refs"]).
seg_clones = {}
job["per_segment_refs"] = per_segment_refs
if per_segment_refs:
try:
from services.speaker_clone import extract_segment_refs
seg_ids_for_clone = [s.get("id", i) for i, s in enumerate(final_segs)]
seg_clones = await loop.run_in_executor(
_cpu_pool, lambda: extract_segment_refs(
vocals_for_clone, final_segs,
os.path.dirname(vocals_for_clone),
seg_ids=seg_ids_for_clone,
),
)
if seg_clones:
job["segment_clones"] = seg_clones
except Exception as e:
logger.warning("per-segment clone refs skipped: %s", e)
if clones or seg_clones:
if clones:
job["speaker_clones"] = clones
# Default each segment's profile_id to its detected speaker's
# auto-clone — but only if the user hasn't already assigned
# something. (#486)
#
# We prefer the UI-visible `auto:{speaker}` id over the
# per-segment `auto-seg:{id}` id even when a per-segment ref
# exists, because the dub editor's Voice dropdown only renders
# `auto:` options ("From Video → Speaker N"). An `auto-seg:`
# value matches no <option>, so the row silently read
# "Default" while the speaker was actually bound — exactly the
# reported bug. The per-segment ref is NOT lost: dub_generate's
# `auto:` branch transparently prefers this segment's own
# per-segment ref (job["segment_clones"][seg_id]) when present,
# so a row shown as "Speaker 1" still clones from its own line
# when that line is long enough.
if clones:
job["speaker_clones"] = clones
# Default each segment's profile_id to its speaker's auto-clone,
# but only if the user hasn't already assigned something.
for s in final_segs:
if s.get("profile_id"):
continue
spk = s.get("speaker_id") or "Speaker 1"
if spk in clones:
s["profile_id"] = auto_profile_id(spk)
continue
# No per-speaker clone for this speaker (too little usable
# audio overall) but this single line was long enough for
# its own ref — fall back to the per-segment id. The editor
# can't render it, but generation still clones correctly.
sid = str(s.get("id", ""))
if sid and sid in seg_clones:
s["profile_id"] = f"auto-seg:{sid}"
except Exception as e:
logger.warning("speaker_clone extraction skipped: %s", e)
@@ -921,14 +862,6 @@ async def dub_transcribe(job_id: str):
scene_cuts = job.get("scene_cuts") or []
segments = segment_transcript(result, duration=job.get("duration", 0.0), scene_cuts=scene_cuts)
# #280: snap segment starts forward to the actual speech onset so the
# dub doesn't begin seconds before the original speaker does.
try:
audio_for_onset, onset_sr = sf.read(asr_audio_target, dtype="float32")
snap_segment_starts(segments, audio_for_onset, onset_sr)
except Exception as e:
logger.warning("onset alignment skipped: %s", e)
diar_pipe = get_diarization_pipeline()
if diar_pipe:
try:
+99 -647
View File
@@ -1,7 +1,5 @@
import os
import io
import re
import json
import time
import uuid
import asyncio
@@ -14,13 +12,6 @@ from core.config import DUB_DIR, dub_seg_path
from core.tasks import task_manager
from api.routers.dub_core import _get_job
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
from services.video_retime import (
DRIFT_TOLERANCE_S,
RetimeError,
build_chunk_filter_graph,
expand_retime_chunks,
prepare_smart_fit_video,
)
router = APIRouter()
logger = logging.getLogger("omnivoice.api")
@@ -31,9 +22,6 @@ def _unique_stamp() -> str:
return f"{time.strftime('%Y%m%dT%H%M%S')}-{uuid.uuid4().hex[:8]}"
_SAFE_LANG = re.compile(r"^[A-Za-z0-9_-]{1,32}$")
def _native_save(source: str, destination: str, display_name: str, media_type: str):
"""Copy a generated export file to a user-chosen destination and return JSON."""
import shutil
@@ -170,23 +158,15 @@ async def dub_list_tracks(job_id: str):
return {"tracks": job.get("dubbed_tracks", {})}
def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool,
fitted_segments: "list[dict] | None" = None) -> str | None:
def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool) -> str | None:
"""Build a temp SRT from job segments for use with ffmpeg's subtitles filter.
Returned path is already ffmpeg-filter-safe (plain ASCII basename under exports_dir).
Returns None if there are no segments to render.
``fitted_segments`` (Smart Fit): {id, start, end} cue records on the
fitted timeline when provided, cue times come from there instead of
the original ``job["segments"]`` timings, so burned subs track the
retimed video / fitted audio rather than the source timeline.
"""
segments = job.get("segments", [])
if not segments:
return None
if fitted_segments:
segments = _apply_fitted_times(segments, fitted_segments)
lines = []
for i, seg in enumerate(segments):
lines.append(str(i + 1))
@@ -234,14 +214,41 @@ def _build_video_stretch_filter_graph(
if not plan:
return "", in_label or f"[{video_input_idx}:v]"
# Chunk expansion + graph emission live in services.video_retime now so
# the Smart Fit batched pipeline shares the exact same boundary math.
# With default options the emitted graph is byte-identical to the
# original inline implementation.
chunks = expand_retime_chunks(plan, orig_dur)
chunks: list[tuple[float, float, float]] = [] # (a, b, ratio)
cursor = 0.0
for entry in plan:
a = float(entry["orig_start"])
b = float(entry["orig_end"])
if a > cursor + 1e-3:
chunks.append((cursor, a, 1.0)) # gap or pre-roll at native rate
ratio = float(entry["stretch_ratio"])
if b > a:
chunks.append((a, b, ratio))
cursor = max(cursor, b)
if orig_dur > cursor + 1e-3:
chunks.append((cursor, orig_dur, 1.0)) # tail at native rate
chunks = [(a, b, r) for (a, b, r) in chunks if b > a]
if not chunks:
return "", in_label or f"[{video_input_idx}:v]"
return build_chunk_filter_graph(chunks, in_label or f"[{video_input_idx}:v]")
src = in_label or f"[{video_input_idx}:v]"
parts: list[str] = []
labels: list[str] = []
# `split` lets us tap the same source stream once per chunk without re-
# decoding. setpts={ratio}*PTS slows down (ratio > 1) or speeds up
# (ratio < 1) each chunk; PTS-STARTPTS first to normalise the timestamp
# base after the trim.
split_labels = [f"[vsplit{idx}]" for idx in range(len(chunks))]
parts.append(f"{src}split={len(chunks)}{''.join(split_labels)}")
for idx, ((a, b, ratio), split_lbl) in enumerate(zip(chunks, split_labels)):
out_label = f"[vstr{idx}]"
labels.append(out_label)
parts.append(
f"{split_lbl}trim=start={a:.4f}:end={b:.4f},"
f"setpts=PTS-STARTPTS,setpts={ratio:.6f}*PTS{out_label}"
)
parts.append("".join(labels) + f"concat=n={len(chunks)}:v=1:a=0[vstretched]")
return ";".join(parts), "[vstretched]"
def _video_stretch_plan_for(job: dict, lang_code: str) -> dict | None:
@@ -257,76 +264,6 @@ def _video_stretch_plan_for(job: dict, lang_code: str) -> dict | None:
return entry
def _video_retime_plan_for(job: dict, lang_code: str) -> "tuple[str, dict] | None":
"""Resolve the video retime plan for ``lang_code`` across both keyspaces.
Returns ``(kind, entry)`` where kind is ``"stretch_video"`` (legacy
Mode B plans resolution byte-identical to ``_video_stretch_plan_for``)
or ``"smart_fit"`` (Phase A ``job["fit_plans"]`` entries, gated on the
track actually having been generated under smart_fit so a stale plan
from an earlier run can't retime a track re-generated under another
strategy). ``None`` when neither applies.
"""
legacy = _video_stretch_plan_for(job, lang_code)
if legacy is not None:
return "stretch_video", legacy
entry = (job.get("fit_plans") or {}).get(lang_code)
track = (job.get("dubbed_tracks") or {}).get(lang_code) or {}
if entry and entry.get("plan") and track.get("timing_strategy") == "smart_fit":
return "smart_fit", entry
return None
def _fitted_segments_for(job: dict, lang_code: "str | None") -> "list[dict] | None":
"""Fitted-timeline subtitle cues ({id, start, end}) for a Smart Fit
track, or None. Same staleness gate as ``_video_retime_plan_for``."""
if not lang_code:
return None
entry = (job.get("fit_plans") or {}).get(lang_code)
track = (job.get("dubbed_tracks") or {}).get(lang_code) or {}
if not entry or track.get("timing_strategy") != "smart_fit":
return None
fitted = entry.get("fitted_segments")
return fitted or None
def _apply_fitted_times(segments: list[dict], fitted: list[dict]) -> list[dict]:
"""Overlay fitted cue times onto subtitle segments (copies; non-destructive).
Matches by segment ``id``; when the fitted record carries no ids at all
(defensive), falls back to positional pairing. Segments without a match
keep their original timings.
"""
by_id = {str(f["id"]): f for f in fitted if f.get("id") is not None}
out: list[dict] = []
for i, seg in enumerate(segments):
cue = None
if seg.get("id") is not None:
cue = by_id.get(str(seg["id"]))
if cue is None and not by_id and i < len(fitted):
cue = fitted[i]
if cue is None:
out.append(seg)
continue
patched = dict(seg)
patched["start"] = float(cue["start"])
patched["end"] = float(cue["end"])
out.append(patched)
return out
def _burn_subs_allowed(retime_kind: "str | None") -> bool:
"""Subtitle burn-in combined with video retime.
Allowed for ``smart_fit`` (fitted cue records exist, and the burn pass
runs AFTER the retime graph so cues land on the retimed timeline) and
for plain exports. Still rejected for legacy ``stretch_video``, which
has no fitted-cue record cues would burn at original timestamps onto
a re-timed video and drift.
"""
return retime_kind != "stretch_video"
#: Audio export formats → ffmpeg codec args. Unknown formats fall back to
#: AAC/m4a so a bad request can never produce a broken command.
_AUDIO_FORMAT_CODECS: dict[str, list[str]] = {
@@ -375,11 +312,6 @@ async def dub_download(
dual: bool = Query(False, description="When burn_subs=1, render translated on top of italicised original."),
out_format: str = Query("m4a", description="Audio-only jobs (#119): output container — wav, m4a, mp3, or flac. Ignored for video jobs."),
):
# Strict allowlist on the path param BEFORE it reaches any filesystem
# path or ffmpeg argv (export dir, retime work path, slice paths). Real
# job ids are short uuid slices — alnum/hyphen/underscore only.
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", job_id):
raise HTTPException(status_code=400, detail="Invalid job id")
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
@@ -458,24 +390,18 @@ async def dub_download(
)
# Determine whether this export should drive video through a per-segment
# retime (legacy stretch_video Mode B, or Smart Fit). Retime is keyed off
# the default_track's plan because the video can only physically follow
# one timeline at a time. If multiple dub tracks are included, only the
# default_track is visually in sync — other tracks share the same
# (retimed) video. Single-track export is the supported common case.
retime_kind: "str | None" = None
retime_entry: "dict | None" = None
if default_track and default_track != "original":
_retime = _video_retime_plan_for(job, default_track)
if _retime:
retime_kind, retime_entry = _retime
stretch_entry = retime_entry if retime_kind == "stretch_video" else None
# Subtitle burn under legacy stretch_video would render cues at the
# original timestamps onto a re-timed video — they'd drift (no fitted-cue
# record exists for that mode). Skip the burn pass in that combo and log;
# the user can still export the SRT/VTT separately. Smart Fit DOES carry
# fitted cues, so burn+retime is allowed there (burn runs post-retime).
if not _burn_subs_allowed(retime_kind) and burn_subs:
# stretch graph (Mode B). Stretch is keyed off the default_track's plan
# because the video can only physically follow one timeline at a time.
# If multiple dub tracks are included and they were generated under
# stretch_video, only the default_track is visually in sync — other
# tracks share the same (stretched) video. Single-track export is the
# supported common case.
stretch_entry = _video_stretch_plan_for(job, default_track) if default_track and default_track != "original" else None
# Subtitle burn under stretch_video would render cues at the original
# timestamps onto a re-timed video — they'd drift. Skip the burn pass
# in that combo and log; the user can still export the SRT/VTT
# separately and the new-layout timing lives there.
if stretch_entry and burn_subs:
logger.warning(
"stretch_video + burn_subs is not supported in one pass; "
"skipping subtitle burn for job %s. Export the SRT/VTT separately.",
@@ -483,71 +409,11 @@ async def dub_download(
)
burn_subs = False
# Smart Fit: cue times come from the fitted timeline — that's where the
# dubbed audio actually sits, whether or not the video retime succeeds.
fitted_segments = _fitted_segments_for(job, default_track) if default_track and default_track != "original" else None
sub_path = _write_burn_srt(job, exports_dir, stamp, dual, fitted_segments=fitted_segments) if burn_subs else None
# ── Smart Fit video retime (two-tier) ─────────────────────────────────
# Tier 1 (≤48 chunks): single filter_complex graph inlined into the mux
# command below. Tier 2: batched slice renders joined by the concat
# demuxer into an intermediate file, muxed as an extra input. Failures
# fall back to an un-retimed export with a structured warning rather
# than failing the whole download.
retime_decision = None
retime_warning: "dict | None" = None
smart_track_dur = 0.0
if retime_kind == "smart_fit" and retime_entry:
smart_orig_dur = float(retime_entry.get("orig_duration") or job.get("duration") or 0.0)
smart_track_dur = float(
retime_entry.get("total_duration")
or (filtered_tracks.get(default_track) or {}).get("duration")
or 0.0
)
# A fresh export is a fresh user intent — clear any sticky abort flag
# from a previous /dub/abort so it can't kill this run's first batch.
job.pop("aborted", None)
# realpath-normalised + containment-checked inline at the sink (the
# file's established pattern — CodeQL does not track the guard
# through a helper's return value).
_base = os.path.realpath(DUB_DIR)
retime_work_path = os.path.realpath(
os.path.join(exports_dir, f"retimed_{stamp}.mp4")
)
if retime_work_path != _base and not retime_work_path.startswith(_base + os.sep):
raise HTTPException(status_code=400, detail="Invalid export path")
try:
retime_decision = await prepare_smart_fit_video(
job_id=job_id,
ffmpeg=ffmpeg,
video_path=video_path,
plan=retime_entry["plan"],
orig_dur=smart_orig_dur,
track_dur=smart_track_dur,
work_path=retime_work_path,
abort_check=lambda: bool(job.get("aborted")),
)
except Exception as e:
if (isinstance(e, RetimeError) and e.stage == "aborted") or job.get("aborted"):
raise HTTPException(status_code=409, detail="Export aborted")
from core.failure import build_failure
retime_warning = build_failure(e, stage="video-retime", include_diagnostic=False)
job["last_export_warning"] = {"type": "video_retime_fallback", **retime_warning}
logger.error(
"Smart Fit video retime failed for job %s — exporting "
"without per-segment retime: %s",
job_id.replace("\n", " ").replace("\r", " "), e,
)
sub_path = _write_burn_srt(job, exports_dir, stamp, dual) if burn_subs else None
cmd = [ffmpeg, "-i", video_path]
input_idx = 1
retimed_idx = None
if retime_decision is not None and retime_decision.mode == "file":
cmd += ["-i", retime_decision.file_path]
retimed_idx = input_idx
input_idx += 1
bg_audio = job.get("no_vocals_path") if preserve_bg else None
bg_idx = None
if bg_audio and os.path.exists(bg_audio) and filtered_tracks:
@@ -563,37 +429,9 @@ async def dub_download(
filter_parts: list[str] = []
video_map = "0:v:0"
video_reencode = False
if retime_decision is not None:
if retime_decision.mode == "filter":
filter_parts.append(retime_decision.graph)
video_map = retime_decision.label
video_reencode = True
else:
video_map = f"{retimed_idx}:v:0"
# Residual drift after the batched render (fps rounding): video
# shorter than the fitted track → freeze the last frame out to
# the track length. Rare — the predicted tail pad inside the
# render usually lands within tolerance.
residual = smart_track_dur - retime_decision.video_dur
if smart_track_dur and residual > DRIFT_TOLERANCE_S:
filter_parts.append(
f"[{retimed_idx}:v]tpad=stop_mode=clone:stop_duration={residual:.4f}[vtpad]"
)
video_map = "[vtpad]"
video_reencode = True
if sub_path:
esc = _ffmpeg_filter_escape(sub_path)
# Burn AFTER any retime so cues (already on the fitted timeline for
# Smart Fit) land on the retimed video. Without retime this reduces
# to the legacy `[0:v]subtitles=…[vsub]` graph.
if video_map.startswith("["):
sub_src = video_map
elif retimed_idx is not None:
sub_src = f"[{retimed_idx}:v]"
else:
sub_src = "[0:v]"
filter_parts.append(f"{sub_src}subtitles='{esc}'[vsub]")
filter_parts.append(f"[0:v]subtitles='{esc}'[vsub]")
video_map = "[vsub]"
if stretch_entry:
orig_dur = float(stretch_entry.get("orig_duration") or job.get("duration") or 0.0)
@@ -609,32 +447,10 @@ async def dub_download(
if include_original:
cmd += ["-map", "0:a:0"]
# Smart Fit drift absorption, audio side: when the retimed video runs
# longer than the fitted track (its tail passes through at 1.0× beyond
# the last cue, or encoder rounding), pad the dub-track chain with
# silence out to the video length so players don't end audio early.
apad_dur = 0.0
if (
retime_decision is not None
and smart_track_dur
and retime_decision.video_dur - smart_track_dur > DRIFT_TOLERANCE_S
):
apad_dur = retime_decision.video_dur
if bg_idx is not None:
for i, t in enumerate(tracks_to_process):
out_label = f"[aout{i}]"
chain = f"[{bg_idx}:a][{t['idx']}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2"
if apad_dur:
chain += f",apad=whole_dur={apad_dur:.4f}"
filter_parts.append(chain + out_label)
t["out_label"] = out_label
for t in tracks_to_process:
cmd += ["-map", t["out_label"]]
elif apad_dur:
for i, t in enumerate(tracks_to_process):
out_label = f"[aout{i}]"
filter_parts.append(f"[{t['idx']}:a]apad=whole_dur={apad_dur:.4f}{out_label}")
filter_parts.append(f"[{bg_idx}:a][{t['idx']}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2{out_label}")
t["out_label"] = out_label
for t in tracks_to_process:
cmd += ["-map", t["out_label"]]
@@ -645,11 +461,10 @@ async def dub_download(
if filter_parts:
cmd += ["-filter_complex", ";".join(filter_parts)]
# Burning subs or per-segment video retime both force a real video
# re-encode; stream-copy is viable when nothing touches the video
# filter chain — including the batched Smart Fit path, whose retimed
# intermediate is already encoded with these exact settings.
if sub_path or stretch_entry or video_reencode:
# Burning subs or per-segment video stretch both force a real video
# re-encode; stream-copy is only viable when nothing touches the video
# filter chain.
if sub_path or stretch_entry:
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
else:
cmd += ["-c:v", "copy"]
@@ -682,17 +497,17 @@ async def dub_download(
break
cmd += [f"-disposition:a:{target_idx}", "default"]
# When retiming (legacy stretch_video or Smart Fit) the video and audio
# durations should match within sub-frame precision, but `-shortest`
# can still cut off the trailing frame; let ffmpeg keep both streams.
# Otherwise keep the legacy `-shortest` so a slightly-overrunning track
# doesn't extend the mux past the video.
if not stretch_entry and retime_decision is None:
# In stretch_video mode the video and audio durations should match
# within sub-frame precision, but `-shortest` can still cut off the
# trailing frame; let ffmpeg keep both streams. Otherwise keep the
# legacy `-shortest` so a slightly-overrunning track doesn't extend
# the mux past the video.
if not stretch_entry:
cmd += ["-shortest"]
cmd += [output_path, "-y"]
try:
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0, job_id=job_id)
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0)
if rc != 0:
raise Exception(stderr.decode(errors="replace") if stderr else "ffmpeg mux non-zero")
except asyncio.TimeoutError:
@@ -704,14 +519,6 @@ async def dub_download(
status_code=500,
detail=f"ffmpeg failed to combine video + dubbed audio: {e}. Verify ffmpeg is installed (`ffmpeg -version`), and check that every dubbed track file exists in the job folder.",
)
finally:
# The batched retime intermediate is a full re-encoded video — never
# leave it behind (success or failure; it's stamp-unique, no reuse).
if retime_decision is not None and retime_decision.mode == "file":
try:
os.remove(retime_decision.file_path)
except OSError as e:
logger.debug("cleanup remove failed: %s", e)
if not os.path.exists(output_path) or os.path.getsize(output_path) == 0:
raise HTTPException(status_code=500, detail="ffmpeg mux produced no output file")
@@ -721,22 +528,12 @@ async def dub_download(
safe_name = ''.join(c for c in base_name if c.isalnum() or c in '-_ ').strip() or 'output'
dl_name = f"dubbed_{safe_name}_{stamp}.mp4"
# Structured warning surface for the Smart Fit fallback ladder: header is
# a fixed ASCII token (FileResponse headers must be latin-1 safe); the
# full build_failure payload is persisted on the job for the UI to read.
extra_headers = {}
if retime_warning is not None:
extra_headers["X-Dub-Export-Warning"] = "video-retime-fallback"
if save_path:
result = _native_save(output_path, save_path, dl_name, media_type="video/mp4")
if retime_warning is not None:
result["warning"] = {"type": "video_retime_fallback", **retime_warning}
return result
return _native_save(output_path, save_path, dl_name, media_type="video/mp4")
return FileResponse(
output_path, media_type="video/mp4",
headers={"Content-Disposition": f'attachment; filename="{dl_name}"', **extra_headers},
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
)
@@ -771,21 +568,6 @@ async def dub_get_media(job_id: str):
ext = os.path.splitext(video_path)[1].lower()
return FileResponse(video_path, media_type=_MEDIA_TYPES.get(ext, "video/mp4"))
# One mux at a time per preview file. Without this, two overlapping requests
# (e.g. the <video> element remounting right after a re-dub) both ran ffmpeg
# against the same output path, and the mtime check below saw the half-written
# file as a valid cache — serving a truncated MP4 that left the player stuck
# loading forever (#281).
_preview_mux_locks: dict[str, asyncio.Lock] = {}
def _preview_lock(path: str) -> asyncio.Lock:
lock = _preview_mux_locks.get(path)
if lock is None:
lock = _preview_mux_locks.setdefault(path, asyncio.Lock())
return lock
@router.get("/dub/preview-video/{job_id}")
async def dub_preview_video(
job_id: str,
@@ -797,11 +579,6 @@ async def dub_preview_video(
Caches per lang+preserve_bg combination under exports/preview_{lang}_{bg}.mp4.
Cache is invalidated when the underlying dubbed track mtime is newer than the cache.
"""
# Strict allowlist on the path param BEFORE it reaches any filesystem
# path or ffmpeg argv (exports dir, preview/retime work paths) — same
# boundary check as dub_download.
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", job_id):
raise HTTPException(status_code=400, detail="Invalid job id")
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
@@ -822,92 +599,23 @@ async def dub_preview_video(
bg_audio = job.get("no_vocals_path") if preserve_bg else None
has_bg = bool(bg_audio and os.path.exists(bg_audio))
if not _SAFE_LANG.match(lang):
raise HTTPException(status_code=400, detail="Invalid lang")
# realpath-normalised + containment-checked inline BEFORE any filesystem
# access so the guard dominates every sink (the file's established
# pattern — see dub_preview_segment; CodeQL does not track the guard
# through a helper's return value).
_base = os.path.realpath(DUB_DIR)
exports_dir = os.path.realpath(os.path.join(_base, job_id, "exports"))
if not exports_dir.startswith(_base + os.sep):
raise HTTPException(status_code=400, detail="Invalid job id")
exports_dir = os.path.join(DUB_DIR, job_id, "exports")
os.makedirs(exports_dir, exist_ok=True)
bg_suffix = "bg" if (preserve_bg and has_bg) else "nobg"
preview_path = os.path.realpath(
os.path.join(exports_dir, f"preview_{lang}_{bg_suffix}.mp4")
)
if not preview_path.startswith(_base + os.sep):
raise HTTPException(status_code=400, detail="Invalid path")
preview_path = os.path.join(exports_dir, f"preview_{lang}_{bg_suffix}.mp4")
track_mtime = os.path.getmtime(track_path)
cache_ok = (
os.path.exists(preview_path)
and os.path.getsize(preview_path) > 0
and os.path.getmtime(preview_path) >= track_mtime
)
def _cache_ok() -> bool:
return (
os.path.exists(preview_path)
and os.path.getsize(preview_path) > 0
and os.path.getmtime(preview_path) >= track_mtime
)
async def _mux_preview():
# Mux into a temp file and os.replace() into place so a concurrent
# reader never sees a partially-written preview (#281: video stuck
# loading forever after a re-dub).
mux_path = preview_path + ".tmp.mp4"
if not cache_ok:
ffmpeg = find_ffmpeg()
# Resolve the same retime plan the download path uses (legacy
# stretch_video or Smart Fit) so the in-app preview matches export.
retime = _video_retime_plan_for(job, lang)
retime_kind, retime_entry = retime if retime else (None, None)
stretch_entry = retime_entry if retime_kind == "stretch_video" else None
retime_decision = None
smart_track_dur = 0.0
if retime_kind == "smart_fit" and retime_entry:
smart_orig_dur = float(retime_entry.get("orig_duration") or job.get("duration") or 0.0)
smart_track_dur = float(
retime_entry.get("total_duration") or track_info.get("duration") or 0.0
)
job.pop("aborted", None) # fresh user intent — clear sticky abort
# realpath-normalised + containment-checked inline at the sink
# (same pattern as preview_path above — _base is the realpath
# of DUB_DIR from the top of this endpoint).
retime_work_path = os.path.realpath(os.path.join(
exports_dir, f"preview_retimed_{lang}_{bg_suffix}.tmp.mp4",
))
if retime_work_path != _base and not retime_work_path.startswith(_base + os.sep):
raise HTTPException(status_code=400, detail="Invalid export path")
try:
retime_decision = await prepare_smart_fit_video(
job_id=job_id,
ffmpeg=ffmpeg,
video_path=video_path,
plan=retime_entry["plan"],
orig_dur=smart_orig_dur,
track_dur=smart_track_dur,
work_path=retime_work_path,
abort_check=lambda: bool(job.get("aborted")),
)
except Exception as e:
if (isinstance(e, RetimeError) and e.stage == "aborted") or job.get("aborted"):
raise HTTPException(status_code=409, detail="Preview aborted")
# Preview is best-effort: fall back to the un-retimed video
# rather than a black player. The export path surfaces the
# structured warning; here we just log.
retime_decision = None
logger.error(
"Smart Fit preview retime failed for job %s — previewing "
"without per-segment retime: %s",
job_id.replace("\n", " ").replace("\r", " "), e,
)
stretch_entry = _video_stretch_plan_for(job, lang)
cmd = [ffmpeg, "-i", video_path]
input_idx = 1
retimed_idx = None
if retime_decision is not None and retime_decision.mode == "file":
cmd += ["-i", retime_decision.file_path]
retimed_idx = input_idx
input_idx += 1
if preserve_bg and has_bg:
cmd += ["-i", bg_audio]
bg_idx = input_idx
@@ -917,13 +625,12 @@ async def dub_preview_video(
cmd += ["-i", track_path]
track_idx = input_idx
# Build filter graph. Under a retime plan we splice the source video
# into per-segment chunks, setpts each to match the dub audio
# Build filter graph. In stretch_video mode we splice the source
# video into per-segment chunks, setpts each to match the dub audio
# layout, and concat them — so audio plays at natural rate and the
# visuals follow. Otherwise we stream-copy video for speed.
filter_parts: list[str] = []
video_map = "0:v:0"
video_reencode = False
if stretch_entry:
orig_dur = float(stretch_entry.get("orig_duration") or job.get("duration") or 0.0)
graph, vlabel = _build_video_stretch_filter_graph(
@@ -932,175 +639,49 @@ async def dub_preview_video(
if graph:
filter_parts.append(graph)
video_map = vlabel
elif retime_decision is not None:
if retime_decision.mode == "filter":
filter_parts.append(retime_decision.graph)
video_map = retime_decision.label
video_reencode = True
else:
video_map = f"{retimed_idx}:v:0"
residual = smart_track_dur - retime_decision.video_dur
if smart_track_dur and residual > DRIFT_TOLERANCE_S:
filter_parts.append(
f"[{retimed_idx}:v]tpad=stop_mode=clone:stop_duration={residual:.4f}[vtpad]"
)
video_map = "[vtpad]"
video_reencode = True
# Smart Fit drift absorption (audio): silence-pad the dub chain out
# to the retimed video length so the preview doesn't end audio early.
apad_dur = 0.0
if (
retime_decision is not None
and smart_track_dur
and retime_decision.video_dur - smart_track_dur > DRIFT_TOLERANCE_S
):
apad_dur = retime_decision.video_dur
audio_map = f"{track_idx}:a:0"
if bg_idx is not None:
chain = f"[{bg_idx}:a][{track_idx}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2"
if apad_dur:
chain += f",apad=whole_dur={apad_dur:.4f}"
filter_parts.append(chain + "[aout]")
audio_map = "[aout]"
elif apad_dur:
filter_parts.append(f"[{track_idx}:a]apad=whole_dur={apad_dur:.4f}[aout]")
audio_map = "[aout]"
filter_parts.append(
f"[{bg_idx}:a][{track_idx}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2[aout]"
)
cmd += ["-map", video_map]
cmd += ["-map", audio_map]
if bg_idx is not None:
cmd += ["-map", "[aout]"]
else:
cmd += ["-map", f"{track_idx}:a:0"]
if filter_parts:
cmd += ["-filter_complex", ";".join(filter_parts)]
# Retime path needs a real encode; stream-copy otherwise (the batched
# Smart Fit intermediate is already encoded — copy unless tpad'ed).
if stretch_entry or video_reencode:
# Stretch path needs a real encode; stream-copy otherwise.
if stretch_entry:
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
else:
cmd += ["-c:v", "copy"]
cmd += ["-c:a", "aac", "-b:a", "192k"]
# `-shortest` would cut the retimed video at the (slightly different)
# `-shortest` would cut the stretched video at the (slightly different)
# audio length and lose the trailing frame; only use it on the copy path.
if not stretch_entry and retime_decision is None:
if not stretch_entry:
cmd += ["-shortest"]
cmd += [mux_path, "-y"]
def _discard_tmp():
try:
os.remove(mux_path)
except OSError as e:
logger.debug("cleanup remove failed: %s", e)
cmd += [preview_path, "-y"]
try:
rc, _, stderr = await run_ffmpeg(cmd, timeout=900.0, job_id=job_id)
rc, _, stderr = await run_ffmpeg(cmd, timeout=900.0)
if rc != 0:
raise Exception(stderr.decode(errors="replace") if stderr else "ffmpeg mux non-zero")
if not os.path.exists(mux_path) or os.path.getsize(mux_path) == 0:
raise Exception("preview mux produced empty file")
except asyncio.TimeoutError:
_discard_tmp()
raise HTTPException(status_code=504, detail="preview mux timed out")
except HTTPException:
_discard_tmp()
raise
except Exception as e:
_discard_tmp()
raise HTTPException(
status_code=500,
detail=f"ffmpeg failed to build the preview stream: {str(e)[:300]}. This usually means the source video can't be re-encoded on the fly — try downloading the MP4 instead.",
)
finally:
if retime_decision is not None and retime_decision.mode == "file":
try:
os.remove(retime_decision.file_path)
except OSError as e:
# Best-effort scratch cleanup — never fail the export.
logger.debug("retime intermediate cleanup failed: %s", e)
os.replace(mux_path, preview_path)
if not os.path.exists(preview_path) or os.path.getsize(preview_path) == 0:
raise HTTPException(status_code=500, detail="preview mux produced empty file")
async with _preview_lock(preview_path):
if not _cache_ok():
await _mux_preview()
# no-store: the URL is stable across re-dubs, so any HTTP-level caching
# in the WebView would keep showing the previous dub after a re-generate
# (#281: "edits don't change the result").
return FileResponse(
preview_path,
media_type="video/mp4",
headers={"Cache-Control": "no-store"},
)
def _compute_onsets_sync(src_path: str) -> list[float]:
"""Blocking part of onset analysis — runs in a worker thread."""
import soundfile as sf
from services.onset_align import detect_speech_onsets
audio, sr = sf.read(src_path, dtype="float32")
return detect_speech_onsets(audio, sr)
@router.get("/dub/onsets/{job_id}")
async def dub_get_onsets(job_id: str):
"""Speech-onset times for the timeline editor's snap-to-onset ticks (#280).
Prefers the Demucs-isolated vocals track (clean speech energy); falls
back to the mixed audio. Computed once per job and cached as
``onsets.json`` in the job directory; recomputed if the source audio is
newer than the cache (e.g. re-ingest into the same job dir).
"""
import json
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
vocals = job.get("vocals_path")
mix = job.get("audio_path")
if vocals and os.path.exists(vocals):
src_path, source = vocals, "vocals"
elif mix and os.path.exists(mix):
src_path, source = mix, "mix"
else:
raise HTTPException(status_code=404, detail="No audio track available for onset analysis")
# Containment inlined (not via _safe_job_path): CodeQL can't track the
# sanitizer through a helper's return — the file's established idiom.
base = os.path.realpath(DUB_DIR)
cache_path = os.path.realpath(os.path.join(base, job_id, "onsets.json"))
if not cache_path.startswith(base + os.sep):
raise HTTPException(status_code=400, detail="Invalid job id")
try:
if (
os.path.exists(cache_path)
and os.path.getmtime(cache_path) >= os.path.getmtime(src_path)
):
with open(cache_path, "r", encoding="utf-8") as f:
cached = json.load(f)
if isinstance(cached, dict) and isinstance(cached.get("onsets"), list):
return cached
except (OSError, ValueError):
pass # unreadable/corrupt cache → recompute below
try:
onsets = await asyncio.to_thread(_compute_onsets_sync, src_path)
except Exception as e:
raise HTTPException(
status_code=500,
detail=f"Onset analysis failed: {str(e)[:200]}",
)
payload = {"onsets": onsets, "source": source}
try:
os.makedirs(os.path.dirname(cache_path), exist_ok=True)
tmp_path = cache_path + ".tmp"
with open(tmp_path, "w", encoding="utf-8") as f:
json.dump(payload, f)
os.replace(tmp_path, cache_path)
except OSError as e:
logger.warning("onsets cache write failed for %s: %s", job_id, e)
return payload
return FileResponse(preview_path, media_type="video/mp4")
@router.get("/dub/thumb/{job_id}")
@@ -1148,94 +729,6 @@ async def dub_preview_segment(job_id: str, segment_index: int):
return FileResponse(seg_path, media_type="audio/wav")
# ── Second-pass ASR QC (Wave 3.3 / Spec 5) ───────────────────────────────────
@router.post("/dub/qc/{job_id}")
async def dub_qc_pass(job_id: str, lang: str = Query(None), drift_threshold: float = Query(0.5)):
"""Re-recognize the dubbed audio and flag lines whose recognized text
drifts from the target text. Opt-in, never fatal: the dub is untouched
this only annotates segments with a per-line drift score and a measured
start/end, surfaced as "verify this line" markers feeding incremental
re-dub. The generated text stays authoritative (design delta from
pyvideotrans, which overwrites subtitles)."""
from services import dub_qc
from services.dub_pipeline import put_job, save_job
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
tracks = job.get("dubbed_tracks", {})
if lang and lang in tracks:
wav_path = tracks[lang]["path"]
elif tracks:
wav_path = list(tracks.values())[0]["path"]
else:
raise HTTPException(status_code=400, detail="No dubbed audio track generated yet")
if not os.path.exists(wav_path):
raise HTTPException(status_code=404, detail="Dubbed audio file not found")
segments = job.get("segments") or []
if not segments:
raise HTTPException(status_code=400, detail="Job has no segments")
def _recognize():
from services.asr_backend import get_active_asr_backend
backend = get_active_asr_backend()
result = backend.transcribe(wav_path, word_timestamps=False)
return result.get("segments", []), backend.id
try:
from services.model_manager import _get_gpu_pool
loop = asyncio.get_running_loop()
recognized, engine_id = await loop.run_in_executor(_get_gpu_pool(), _recognize)
except Exception as e:
logger.exception("dub QC ASR pass failed for %s", job_id)
raise HTTPException(status_code=500, detail=f"QC transcription failed: {e}")
seg_ids = job.get("seg_order") or [s.get("id", i) for i, s in enumerate(segments)]
scored = dub_qc.score_dub(segments, recognized, drift_threshold=drift_threshold, seg_ids=seg_ids)
# Annotate each segment (non-destructive — content text untouched).
by_id = {q.seg_id: q for q in scored}
for i, s in enumerate(segments):
sid = str(seg_ids[i]) if i < len(seg_ids) else str(s.get("id", i))
q = by_id.get(sid)
if q is None:
continue
s["qc_drift"] = q.drift
s["qc_flagged"] = q.flagged
s["qc_recognized"] = q.recognized_text
if q.new_start is not None:
s["qc_measured_start"] = q.new_start
s["qc_measured_end"] = q.new_end
put_job(job_id, job)
save_job(job_id, job)
flagged = [q for q in scored if q.flagged]
payload = json.dumps({"event": "qc_done", "engine": engine_id,
"flagged": len(flagged), "total": len(scored)})
try:
from core import job_store
job_store.append_event(job_id, f"data: {payload}\n\n")
except Exception as e:
# QC event fan-out is best-effort; the scores are already in the response.
logger.debug("QC event append failed: %s", e)
return {
"engine": engine_id,
"total": len(scored),
"flagged_count": len(flagged),
"drift_threshold": drift_threshold,
"segments": [
{"seg_id": q.seg_id, "drift": q.drift, "flagged": q.flagged,
"recognized_text": q.recognized_text,
"measured_start": q.new_start, "measured_end": q.new_end}
for q in scored
],
}
@router.get("/dub/download-audio/{job_id}")
@router.get("/dub/download-audio/{job_id}/{filename}")
async def dub_download_audio(job_id: str, lang: str = Query(None), preserve_bg: bool = Query(True), save_path: str = Query("")):
@@ -1313,31 +806,9 @@ def _pick_subtitle_text(seg: dict, dual: bool) -> str:
return f"{translated}\n<i>{original}</i>"
# Subtitles deliberately have no ?save_path= variant: they're small text
# bodies, so the Tauri side fetches them raw and writes the file itself via
# the save_text_file command — the OS save dialog is the write authorization
# (#309). The frontend's JSON-envelope save flow stays for binary exports.
def _fitted_cue_times(job: dict, lang: str | None) -> list | None:
"""Per-segment (start, end) on the fitted timeline when this job used
stretch_video; None to use the original segment times. (Wave 3.1.)"""
tracks = job.get("dubbed_tracks", {})
lc = lang if (lang and lang in tracks) else (next(iter(tracks), None))
entry = _video_stretch_plan_for(job, lc) if lc else None
if not entry:
return None
from services.fitted_subtitles import fitted_cues
return fitted_cues(job.get("segments", []), entry["plan"])
@router.get("/dub/srt/{job_id}")
@router.get("/dub/srt/{job_id}/{filename}")
async def dub_export_srt(
job_id: str,
dual: bool = False,
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
):
async def dub_export_srt(job_id: str, dual: bool = False):
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
@@ -1346,31 +817,22 @@ async def dub_export_srt(
if not segments:
raise HTTPException(status_code=400, detail="No transcript segments available")
# Subtitles must follow the audio the viewer hears, per timing strategy:
# Smart Fit (phase B) overlays the fitted segment times directly;
# stretch_video (Wave 3.1) regenerates cue times from the stretch plan.
# Neither applies → original times.
fitted = _fitted_segments_for(job, lang)
if fitted:
segments = _apply_fitted_times(segments, fitted)
cues = None if fitted else _fitted_cue_times(job, lang)
srt_lines = []
for i, seg in enumerate(segments):
s, e = cues[i] if cues else (seg["start"], seg["end"])
start_ts = _format_srt_time(seg["start"])
end_ts = _format_srt_time(seg["end"])
srt_lines.append(f"{i + 1}")
srt_lines.append(f"{_format_srt_time(s)} --> {_format_srt_time(e)}")
srt_lines.append(f"{start_ts} --> {end_ts}")
srt_lines.append(_pick_subtitle_text(seg, dual))
srt_lines.append("")
srt_content = "\n".join(srt_lines)
base_name = os.path.splitext(job.get('filename', 'video'))[0]
suffix = "_dual" if dual else ""
dl_name = f"subtitles_{base_name}{suffix}.srt"
return Response(
content=srt_content,
media_type="text/plain",
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
headers={"Content-Disposition": f'attachment; filename="subtitles_{base_name}{suffix}.srt"'},
)
def _format_vtt_time(seconds):
@@ -1382,11 +844,7 @@ def _format_vtt_time(seconds):
@router.get("/dub/vtt/{job_id}")
@router.get("/dub/vtt/{job_id}/{filename}")
async def dub_export_vtt(
job_id: str,
dual: bool = False,
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
):
async def dub_export_vtt(job_id: str, dual: bool = False):
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
@@ -1395,28 +853,22 @@ async def dub_export_vtt(
if not segments:
raise HTTPException(status_code=400, detail="No transcript segments available")
# Same strategy-aware cue timing as /dub/srt (see comment there).
fitted = _fitted_segments_for(job, lang)
if fitted:
segments = _apply_fitted_times(segments, fitted)
cues = None if fitted else _fitted_cue_times(job, lang)
vtt_lines = ["WEBVTT", ""]
for i, seg in enumerate(segments):
s, e = cues[i] if cues else (seg["start"], seg["end"])
start_ts = _format_vtt_time(seg["start"])
end_ts = _format_vtt_time(seg["end"])
vtt_lines.append(str(i + 1))
vtt_lines.append(f"{_format_vtt_time(s)} --> {_format_vtt_time(e)}")
vtt_lines.append(f"{start_ts} --> {end_ts}")
vtt_lines.append(_pick_subtitle_text(seg, dual))
vtt_lines.append("")
vtt_content = "\n".join(vtt_lines)
base_name = os.path.splitext(job.get('filename', 'video'))[0]
suffix = "_dual" if dual else ""
dl_name = f"subtitles_{base_name}{suffix}.vtt"
return Response(
content=vtt_content,
media_type="text/vtt",
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
headers={"Content-Disposition": f'attachment; filename="subtitles_{base_name}{suffix}.vtt"'},
)
+101 -239
View File
@@ -3,6 +3,7 @@ import json
import logging
import time
import asyncio
import numpy as np
import torch
import torchaudio
from fastapi import APIRouter, HTTPException
@@ -14,18 +15,9 @@ from schemas.requests import DubRequest
from services.model_manager import get_model, _gpu_pool
from services.audio_dsp import apply_mastering, normalize_audio, apply_effects_chain, get_effect_chain
from services.audio_io import atomic_save_wav, _safe_torchaudio_save
from services.ffmpeg_utils import (
find_ffmpeg,
spawn_subprocess,
# Moved to ffmpeg_utils so the Smart Fit export pipeline (Phase B) can
# reuse them; re-imported here so `dub_generate._atempo_chain` /
# `_pitch_preserving_stretch` keep working for existing importers.
_atempo_chain,
_pitch_preserving_stretch,
)
from services.ffmpeg_utils import find_ffmpeg, spawn_subprocess
from services.rvc import apply_rvc, is_enabled as rvc_is_enabled
from services.incremental import segment_fingerprint, fit_fingerprint
from services.fit_planner import FitParams, plan_fit
from services.incremental import segment_fingerprint
from services.watermark import embed_watermark
from api.routers.dub_core import _get_job, _save_job
@@ -47,44 +39,75 @@ GAP_OVERFLOW_MAX_S = 0.25
GAP_OVERFLOW_BUFFER_S = 0.05
def _sync_job_segments(job: dict, req: DubRequest) -> None:
"""Persist the segments this dub was actually generated from back onto the job.
def _atempo_chain(ratio: float) -> str:
"""Build an `atempo=…,atempo=…` filter chain for arbitrary ratios.
The editor only sends the (translated / user-edited) segment text in the
generate request; the job itself kept the original-language ASR transcript.
SRT/VTT export and ffmpeg subtitle burn-in read `job["segments"]`, so they
rendered the source language instead of the dub the user just heard (#309).
Merge strategy: rebuild `job["segments"]` from the request, carrying over
per-segment metadata (speaker_id, id, ) from the existing job segment
matched by stable id (fallback: index). `text_original` always keeps the
source-language text so dual-subtitle layouts can still stack it under the
translation.
ffmpeg's atempo filter is limited to [0.5, 2.0] per stage. Chaining
multiple stages multiplies the effective ratio while keeping each
individual stage inside the well-behaved range. Pitch is preserved
(WSOLA-style time-domain stretching). ratio > 1 speeds up, < 1
slows down.
"""
if not req.segments:
return
existing = [s for s in (job.get("segments") or []) if isinstance(s, dict)]
by_id = {str(s["id"]): s for s in existing if s.get("id") is not None}
seg_ids = req.segment_ids or []
merged: list[dict] = []
for i, seg in enumerate(req.segments):
seg_id = seg_ids[i] if i < len(seg_ids) else None
prev = by_id.get(str(seg_id)) if seg_id is not None else None
if prev is None and i < len(existing):
prev = existing[i]
row = dict(prev) if prev else {}
if seg_id is not None:
# The request id is authoritative — seg_order and the per-segment
# WAV manifest are keyed by it.
row["id"] = seg_id
# Source-language text survives the overwrite so dual-subtitle export
# keeps working; never let the translation clobber it.
row["text_original"] = row.get("text_original") or row.get("text") or ""
row["start"] = seg.start
row["end"] = seg.end
row["text"] = seg.text
merged.append(row)
job["segments"] = merged
stages: list[str] = []
remaining = ratio
while remaining > 2.0:
stages.append("atempo=2.0")
remaining /= 2.0
while remaining < 0.5:
stages.append("atempo=0.5")
remaining /= 0.5
stages.append(f"atempo={remaining:.6f}")
return ",".join(stages)
async def _pitch_preserving_stretch(
wav: torch.Tensor, target_samples: int, sr: int,
) -> torch.Tensor:
"""Time-stretch a (1, samples) tensor to `target_samples` while
preserving pitch, by piping the audio through `ffmpeg atempo`.
Async so it never blocks the event loop: it's awaited from the `_stream`
generator, and each ffmpeg call is ~50-100 ms a synchronous
``subprocess.run`` here froze health-checks / SSE / every concurrent
request for the whole multi-segment job.
Returns a (1, target_samples) tensor on the same device as input.
Raises RuntimeError when ffmpeg fails callers should fall back to
naive linear interpolation, accepting the pitch shift, to ensure the
output isn't silent.
"""
wl = int(wav.shape[-1])
if target_samples <= 0 or wl == target_samples:
return wav
ratio = wl / target_samples
filter_str = _atempo_chain(ratio)
# Mono float32 via stdin → ffmpeg → stdout. One subprocess per segment,
# run off the event loop so concurrent requests stay responsive.
arr = wav.detach().cpu().to(torch.float32).numpy().reshape(-1).astype(np.float32, copy=False)
proc = await spawn_subprocess(
find_ffmpeg(), "-hide_banner", "-loglevel", "error", "-y",
"-f", "f32le", "-ar", str(sr), "-ac", "1", "-i", "pipe:0",
"-af", filter_str,
"-f", "f32le", "-ar", str(sr), "-ac", "1", "pipe:1",
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, stderr = await proc.communicate(input=arr.tobytes())
if proc.returncode != 0 or not stdout:
raise RuntimeError(
(stderr.decode(errors="replace") or "atempo failed")[:200]
)
out_arr = np.frombuffer(stdout, dtype=np.float32)
# atempo rarely lands exactly on the integer sample count, so
# pad/trim to the requested slot length.
if len(out_arr) < target_samples:
pad = np.zeros(target_samples - len(out_arr), dtype=np.float32)
out_arr = np.concatenate([out_arr, pad])
elif len(out_arr) > target_samples:
out_arr = out_arr[:target_samples]
return torch.from_numpy(out_arr.copy()).unsqueeze(0).to(wav.device)
router = APIRouter()
@@ -111,15 +134,6 @@ async def dub_generate(job_id: str, req: DubRequest):
# `seg_i.wav` on disk and slot into the final mix unchanged.
regen_only = set(req.regen_only or []) if req.regen_only is not None else None
seg_ids = req.segment_ids or []
strategy = (req.timing_strategy or "concise").lower()
# Strategy-transition guard: smart_fit re-mixes the *natural-rate*
# per-segment WAVs from disk. If the previous run used strict_slot,
# the on-disk WAVs are slot-squeezed ("slotted") — reusing them would
# double-compress. Force one full regen; afterwards seg_wav_kind is
# "natural" and partial regen / fit-only re-mix (regen_only=[]) work.
# Jobs predating this field have unknown kind → also regen once.
if strategy == "smart_fit" and regen_only is not None and job.get("seg_wav_kind") != "natural":
regen_only = None
# Manifest: stable segment id per current index. Per-segment WAVs are
# named by stable id (dub_seg_path) so regen reuses the right audio after
# reorder; index-keyed readers (preview/export) resolve via this manifest.
@@ -170,17 +184,13 @@ async def dub_generate(job_id: str, req: DubRequest):
if cached_sr != _model.sampling_rate:
import torchaudio.functional as AF
cached_wav = AF.resample(cached_wav, cached_sr, _model.sampling_rate)
# Pad/trim to slot — except smart_fit, whose mix
# loop needs the natural-rate length to compute the
# audio/video split (the seg_wav_kind guard above
# guarantees these cached WAVs are natural-rate).
if strategy != "smart_fit":
target_samples = int(seg_duration * _model.sampling_rate)
current_samples = cached_wav.shape[-1]
if target_samples > current_samples:
cached_wav = torch.nn.functional.pad(cached_wav, (0, target_samples - current_samples))
elif current_samples > target_samples:
cached_wav = cached_wav[..., :target_samples]
# Pad/trim to slot.
target_samples = int(seg_duration * _model.sampling_rate)
current_samples = cached_wav.shape[-1]
if target_samples > current_samples:
cached_wav = torch.nn.functional.pad(cached_wav, (0, target_samples - current_samples))
elif current_samples > target_samples:
cached_wav = cached_wav[..., :target_samples]
all_segment_wavs.append((seg.start, seg.end, cached_wav, _model.sampling_rate))
sync_scores.append(getattr(seg, 'sync_ratio', None) or 1.0)
_t_cache += time.perf_counter() - _t_cache_0
@@ -204,42 +214,18 @@ async def dub_generate(job_id: str, req: DubRequest):
# (see services/speaker_clone.py) live at job["speaker_clones"]
# keyed by speaker_id. We use the `auto:` prefix so they can't
# collide with persistent voice_profiles.id values.
# Wave 3.2: a per-segment clone ref (cut from this line's own
# source audio) takes precedence over the per-speaker clone.
if profile_id and profile_id.startswith("auto-seg:"):
sid = profile_id[len("auto-seg:"):]
info = (job.get("segment_clones") or {}).get(sid)
if info:
ref_audio = info.get("ref_audio")
ref_text = info.get("ref_text")
profile_id = None # prevent the voice_profiles lookup below
elif profile_id and profile_id.startswith("auto:"):
# #486: an `auto:{speaker}` binding still prefers THIS
# segment's own per-segment ref when one exists (cut from
# this line's source audio → matches its prosody), falling
# back to the per-speaker clone otherwise. This keeps the
# Wave 3.2 per-segment-ref quality win while letting every
# segment carry the UI-visible `auto:` id the dub editor's
# Voice dropdown can actually render ("From Video →
# Speaker N"). `seg_id` is closed over from the per-segment
# loop below.
seg_ref = (job.get("segment_clones") or {}).get(str(seg_id))
if seg_ref:
ref_audio = seg_ref.get("ref_audio")
ref_text = seg_ref.get("ref_text")
else:
key = profile_id[len("auto:"):]
clones = job.get("speaker_clones") or {}
# Match by the safe-name key first, fall back to speaker_id.
auto = None
for spk, info in clones.items():
if spk.lower().replace(" ", "_") == key or spk == key:
auto = info
break
if auto:
ref_audio = auto.get("ref_audio")
ref_text = auto.get("ref_text")
if profile_id and profile_id.startswith("auto:"):
key = profile_id[len("auto:"):]
clones = job.get("speaker_clones") or {}
# Match by the safe-name key first, fall back to speaker_id.
auto = None
for spk, info in clones.items():
if spk.lower().replace(" ", "_") == key or spk == key:
auto = info
break
if auto:
ref_audio = auto.get("ref_audio")
ref_text = auto.get("ref_text")
profile_id = None # prevent the voice_profiles lookup below
if profile_id:
@@ -279,9 +265,6 @@ async def dub_generate(job_id: str, req: DubRequest):
if seg_effect_preset == "raw":
return audio_out
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
effect_chain = get_effect_chain(seg_effect_preset)
if effect_chain:
@@ -327,9 +310,6 @@ async def dub_generate(job_id: str, req: DubRequest):
if seg_effect_preset == "raw":
return audio_out
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
effect_chain = get_effect_chain(seg_effect_preset)
if effect_chain:
@@ -393,13 +373,13 @@ async def dub_generate(job_id: str, req: DubRequest):
_t_tts_0 = time.perf_counter()
seg_effect_preset = getattr(seg, "effect_preset", None) or "broadcast"
# In concise / stretch_video / smart_fit modes we pass
# dur_s=None so the TTS model speaks at its natural rate for
# this text length — the whole point of the new timing
# strategies is to never squeeze the speech to fit at
# synthesis time. strict_slot keeps the legacy behaviour
# where dur_s is the slot hint.
_dur_for_tts = seg_duration if strategy == "strict_slot" else None
# In concise / stretch_video modes we pass dur_s=None so the
# TTS model speaks at its natural rate for this text length —
# the whole point of the new timing strategies is to never
# squeeze the speech to fit. strict_slot keeps the legacy
# behaviour where dur_s is the slot hint.
_strategy = (req.timing_strategy or "concise").lower()
_dur_for_tts = seg_duration if _strategy == "strict_slot" else None
audio_tensor = await loop.run_in_executor(
_gpu_pool, _gen,
@@ -416,7 +396,7 @@ async def dub_generate(job_id: str, req: DubRequest):
target_samples = int(seg_duration * _model.sampling_rate)
current_samples = audio_tensor.shape[-1]
if strategy == "strict_slot":
if _strategy == "strict_slot":
# Legacy: pad short audio + trim long audio so the mix
# loop receives slot-sized buffers. The atempo squeeze
# in the mix loop never fires here because we already
@@ -426,10 +406,9 @@ async def dub_generate(job_id: str, req: DubRequest):
audio_tensor = torch.nn.functional.pad(audio_tensor, (0, pad_amount))
elif current_samples > target_samples:
audio_tensor = audio_tensor[..., :target_samples]
# concise / stretch_video / smart_fit: keep audio at its
# natural length. The mix loop decides per-mode whether to
# trim, slip, stretch the video, or split audio/video
# retiming (smart_fit) to accommodate it.
# concise / stretch_video: keep audio at its natural length.
# The mix loop decides per-mode whether to trim, slip, or
# stretch the video to accommodate it.
generated_dur = audio_tensor.shape[-1] / _model.sampling_rate
sync_ratio = round(generated_dur / max(seg_duration, 0.01), 3)
@@ -508,6 +487,7 @@ async def dub_generate(job_id: str, req: DubRequest):
_t_diskw = time.perf_counter() - _t_diskw_0
sr = _model.sampling_rate
strategy = (req.timing_strategy or "concise").lower()
slot_fit = (req.slot_fit or "time_stretch").lower()
overflow_budget_s = max(0.0, float(req.overflow_budget_s or 0.0))
@@ -522,7 +502,6 @@ async def dub_generate(job_id: str, req: DubRequest):
# the matching per-segment setpts filter chain on the source video.
new_layout: list[tuple[float, float]] = []
video_stretch_plan: list[dict] = []
fit_plan = None # smart_fit only — services.fit_planner.FitPlan
orig_total_dur = float(job.get("duration") or 0.0)
if strategy == "stretch_video":
@@ -556,43 +535,9 @@ async def dub_generate(job_id: str, req: DubRequest):
cursor += max(0.0, orig_total_dur - last_orig_end)
new_total_dur = max(cursor, orig_total_dur)
total_samples = int(new_total_dur * sr)
elif strategy == "smart_fit":
# Smart Fit: plan the audio-rate / video-ratio split per segment
# from the natural-rate WAV lengths. Pure planning — the mix
# loop below applies the audio side; the video side ships as
# fit_plans[lang] for the (Phase B) export pipeline.
_fo = req.fit_options
_fit_defaults = FitParams()
fit_params = FitParams(
max_audio_only_rate=float(getattr(_fo, "max_audio_only_rate", None) or _fit_defaults.max_audio_only_rate),
audio_rate_cap=float(getattr(_fo, "audio_rate_cap", None) or _fit_defaults.audio_rate_cap),
video_slow_cap=float(getattr(_fo, "video_slow_cap", None) or _fit_defaults.video_slow_cap),
gap_guard_s=float(_fo.gap_guard_s) if _fo is not None and _fo.gap_guard_s is not None else _fit_defaults.gap_guard_s,
allow_video_retime=bool(_fo.allow_video_retime) if _fo is not None and _fo.allow_video_retime is not None else _fit_defaults.allow_video_retime,
)
_seg_order = job.get("seg_order") or []
fit_plan = plan_fit(
[
{
"id": _seg_order[i] if i < len(_seg_order) else f"seg_{i}",
"start": s,
"end": e,
}
for i, (s, e, _w, _) in enumerate(all_segment_wavs)
],
[w.shape[-1] / sr for (_s, _e, w, _) in all_segment_wavs],
orig_total_dur,
fit_params,
)
total_samples = int(fit_plan.total_duration * sr)
else:
total_samples = int(orig_total_dur * sr)
# smart_fit: cue times for the fitted timeline, computed from the
# ACTUAL stretched/trimmed sample positions in the mix loop below —
# not from the plan — so subtitles land exactly on the audio.
fitted_cues: list[dict] = []
full_audio = torch.zeros(1, total_samples)
for i, (start, end, wav, _) in enumerate(all_segment_wavs):
@@ -615,55 +560,6 @@ async def dub_generate(job_id: str, req: DubRequest):
"stretch_ratio": round(natural_dur / max(orig_dur, 1e-3), 3),
})
elif strategy == "smart_fit":
# Smart Fit: apply the planner's audio_rate via the same
# pitch-preserving atempo pipe strict_slot uses, place the
# result at the planned new_start, and hard-trim whatever
# the caps couldn't absorb. The video side (video_ratio per
# chunk) is persisted below for the export pipeline.
sf = fit_plan.segments[i]
place_at = sf.new_start
if sf.audio_rate > 1.0 + 1e-6 and wl > 0:
target = max(1, int(round(wl / sf.audio_rate)))
try:
adjusted = await _pitch_preserving_stretch(
adjusted, target, sr,
)
except Exception as e:
logger.warning(
"atempo stretch failed for seg %d (%.2f×), "
"falling back to linear interp: %s",
i, sf.audio_rate, e,
)
adjusted = torch.nn.functional.interpolate(
adjusted.unsqueeze(0),
size=target,
mode='linear',
align_corners=False,
).squeeze(0)
wl = adjusted.shape[-1]
# Residual overflow → hard-trim to the segment's new video
# slot (fade below keeps the cut pop-free).
new_slot_samples = int(max(0.0, sf.new_end - sf.new_start) * sr)
if new_slot_samples > 0 and wl > new_slot_samples:
adjusted = adjusted[..., :new_slot_samples]
wl = adjusted.shape[-1]
# Truthful per-segment verdict for the UI badge.
entry = {"status": sf.status}
if sf.audio_rate > 1.0 + 1e-6:
entry["audio_rate"] = round(sf.audio_rate, 3)
if sf.video_ratio > 1.0 + 1e-6:
entry["video_ratio"] = round(sf.video_ratio, 3)
if sf.overflow_s > 0:
entry["overflow_s"] = round(sf.overflow_s, 3)
fit_status.append(entry)
# Cue times from the ACTUAL stretched sample positions.
fitted_cues.append({
"id": sf.seg_id,
"start": round(place_at, 4),
"end": round(place_at + wl / sr, 4),
})
elif strategy == "concise":
# Mode A: never compress. Allow the audio to extend into the
# silent gap before the next seg (existing heuristic) plus
@@ -785,9 +681,6 @@ async def dub_generate(job_id: str, req: DubRequest):
job["language"] = req.language
job["language_code"] = lang_code
job["timing_strategy"] = strategy
# Keep job segments in lock-step with what was just rendered so
# subtitle export / burn-in use the translated text (#309).
_sync_job_segments(job, req)
if strategy == "stretch_video":
stretch_plans = job.setdefault("video_stretch_plans", {})
stretch_plans[lang_code] = {
@@ -795,34 +688,6 @@ async def dub_generate(job_id: str, req: DubRequest):
"total_duration": round(track_dur, 4),
"orig_duration": round(orig_total_dur, 4),
}
elif strategy == "smart_fit" and fit_plan is not None:
# video_stretch_plans stays untouched — smart_fit persists its
# own keyspace so a job can carry both without clobbering.
_fit_params_payload = {
"timing_strategy": strategy,
"max_audio_only_rate": fit_params.max_audio_only_rate,
"audio_rate_cap": fit_params.audio_rate_cap,
"video_slow_cap": fit_params.video_slow_cap,
"gap_guard_s": fit_params.gap_guard_s,
"allow_video_retime": fit_params.allow_video_retime,
}
fit_fp = fit_fingerprint(_fit_params_payload)
job.setdefault("fit_plans", {})[lang_code] = {
# Same dict shape _build_video_stretch_filter_graph consumes.
"plan": fit_plan.video_plan,
# Cue times from actual stretched sample positions — for
# subtitle export on the fitted timeline.
"fitted_segments": fitted_cues,
"total_duration": round(track_dur, 4),
"orig_duration": round(orig_total_dur, 4),
"params": _fit_params_payload,
"fit_fp": fit_fp,
}
job["dubbed_tracks"][lang_code]["fit_fp"] = fit_fp
# Record what kind of per-segment WAVs are on disk so a later
# smart_fit run knows whether partial regen / fit-only re-mix can
# reuse them ("natural") or must regen once ("slotted").
job["seg_wav_kind"] = "slotted" if strategy == "strict_slot" else "natural"
_save_job(job_id, job)
_t_total = time.perf_counter() - _t_start
@@ -919,9 +784,6 @@ async def preview_segment(job_id: str, req: SegmentPreviewRequest):
postprocess_output=True,
)
audio_out = audios[0]
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered = apply_mastering(
audio_out,
sample_rate=getattr(_model, "sampling_rate", 24000),
+7 -124
View File
@@ -2,7 +2,6 @@ import os
import time
import asyncio
import logging
from typing import Optional
from fastapi import APIRouter
from fastapi.responses import JSONResponse
@@ -46,70 +45,6 @@ LANG_NAMES = {
"id": "Indonesian", "uk": "Ukrainian",
}
# Regional dialect hints (#280 item 2). Maps a BCP-47 dialect code to the
# instruction injected into LLM translation prompts so the output uses that
# region's vocabulary and grammar (the reporter's example: choosing Argentina
# should yield "Vos sos muy listo", not the Peninsular "Tú eres muy listo").
# Only LLM-backed paths can honor these — provider="openai" and the
# quality="cinematic" refine pass. Keep entries short: they ride on every
# per-segment prompt, so verbosity = wall time.
DIALECT_HINTS = {
# Spanish
"es-ES": "European Spanish (Spain): use tú/vosotros forms and Peninsular vocabulary.",
"es-MX": "Mexican Spanish: use tú/ustedes forms and Mexican vocabulary.",
"es-AR": "Rioplatense Spanish (Argentina): use voseo — 'vos' with its verb forms (e.g. 'vos sos', 'tenés') and 'ustedes'; prefer Argentinian vocabulary.",
"es-CO": "Colombian Spanish: use tú/usted as natural in Colombia and Colombian vocabulary.",
"es-CL": "Chilean Spanish: use Chilean vocabulary and expressions.",
# Portuguese
"pt-BR": "Brazilian Portuguese: use 'você' forms, Brazilian vocabulary and spelling.",
"pt-PT": "European Portuguese: use European vocabulary, spelling, and 'tu' where natural.",
# English
"en-US": "American English: use US spelling and vocabulary.",
"en-GB": "British English: use UK spelling and vocabulary.",
"en-AU": "Australian English: use Australian spelling and vocabulary.",
"en-IN": "Indian English: use Indian English vocabulary and conventions.",
# French
"fr-FR": "Metropolitan French (France): use standard French vocabulary.",
"fr-CA": "Canadian French (Québec): use Québécois vocabulary and expressions.",
"fr-BE": "Belgian French: use Belgian vocabulary (e.g. septante, nonante).",
# German
"de-DE": "Standard German (Germany): use Federal German vocabulary.",
"de-AT": "Austrian German: use Austrian vocabulary (e.g. Jänner, Erdapfel).",
"de-CH": "Swiss Standard German: use Swiss vocabulary and 'ss' instead of 'ß'.",
# Arabic
"ar-EG": "Egyptian Arabic: use Egyptian colloquial vocabulary where natural for dubbing.",
"ar-SA": "Gulf/Saudi Arabic flavor: prefer vocabulary natural to the Gulf region.",
"ar-MA": "Moroccan Arabic (Darija) flavor: prefer vocabulary natural to Morocco.",
# Dutch
"nl-NL": "Netherlands Dutch: use vocabulary standard in the Netherlands.",
"nl-BE": "Belgian Dutch (Flemish): use Flemish vocabulary and expressions.",
}
def dialect_clause(dialect: Optional[str]) -> str:
"""Prompt fragment for a requested dialect, or '' when unset/unknown.
Unknown-but-plausible codes (e.g. "es-PE") still get a generic regional
clause so users aren't limited to the curated list.
"""
if not dialect or not str(dialect).strip():
return ""
code = str(dialect).strip()
hint = DIALECT_HINTS.get(code)
if hint:
return f" Target dialect — {hint}"
# Generic fallback for any lang-REGION shaped code we don't curate.
if "-" in code:
lang, _, region = code.partition("-")
lang_name = LANG_NAMES.get(lang, lang)
if region:
return (
f" Use the vocabulary, grammar, and expressions of {lang_name} "
f"as spoken in the region '{region}'."
)
return ""
# Per-language script enforcement. Maps language code → required Unicode
# block(s) the translation must contain. Used as a sanity gate after the
# LLM responds: if the output contains <50% characters from the expected
@@ -157,49 +92,15 @@ _nllb_tokenizer = None
_nllb_device = None
def _dialect_flags(req, applied: bool) -> dict:
"""Response fields describing whether the requested dialect was honored.
Empty dict when no dialect was requested, so existing response shapes
stay byte-identical for callers that never send one.
"""
if not getattr(req, "dialect", None):
return {}
return {"dialect": req.dialect, "dialect_applied": bool(applied)}
def _guess_lang_from_text(segments) -> str | None:
"""Best-effort source language from segment text, by script.
Used only as a last resort when neither the request nor the job carries a
detected language. Without this, the bare "en" fallback below forces
en -> en on non-English audio (e.g. Korean), which has no Argos package and
fails every segment even though ASR detected the language correctly.
"""
text = " ".join((getattr(s, "text", "") or "") for s in (segments or [])[:8])
has = lambda lo, hi: any(lo <= ord(c) <= hi for c in text)
if has(0x3040, 0x30FF):
return "ja" # Hiragana/Katakana — check before CJK (Japanese uses Kanji too)
if has(0xAC00, 0xD7A3) or has(0x1100, 0x11FF):
return "ko" # Hangul
if has(0x4E00, 0x9FFF):
return "zh" # CJK ideographs
if has(0x0400, 0x04FF):
return "ru" # Cyrillic
if has(0x0600, 0x06FF):
return "ar" # Arabic
return None
def _resolve_source_lang(req: TranslateRequest) -> str:
"""Pick source language: explicit request > job.source_lang > text guess > 'en'."""
"""Pick source language: explicit request > job.source_lang > 'en' fallback."""
if getattr(req, "source_lang", None):
return req.source_lang
if getattr(req, "job_id", None):
job = _get_job(req.job_id)
if job and job.get("source_lang"):
return job["source_lang"]
return _guess_lang_from_text(getattr(req, "segments", None)) or "en"
return "en"
def _unload_nllb():
@@ -302,8 +203,7 @@ async def dub_translate(req: TranslateRequest):
translated = await loop.run_in_executor(_gpu_pool, _translate_nllb)
if os.environ.get("OMNIVOICE_UNLOAD_NLLB", "1") == "1":
_unload_nllb()
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
**_dialect_flags(req, applied=False)}
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
# OpenAI / Ollama Local LLM Translation
if provider == "openai":
@@ -336,16 +236,10 @@ async def dub_translate(req: TranslateRequest):
f"only — do not use Latin/Roman letters, do not "
f"transliterate, do not output any other language."
)
# #280 item 2 — regional dialect/vocabulary. Only applied when
# the dialect belongs to the target language (a leftover
# "es-AR" must not contaminate a French translation).
dia_clause = ""
if req.dialect and str(req.dialect).lower().startswith(str(tgt_code).lower()[:2]):
dia_clause = dialect_clause(req.dialect)
return (
f"You are a professional dubbing translator. "
f"Translate the user's text from {src_name} into "
f"{tgt_name}.{script_clause}{dia_clause} "
f"{tgt_name}.{script_clause} "
f"Reply ONLY with the translated {tgt_name} text, do not "
f"add quotes, notes, headers, explanations, or commentary."
)
@@ -405,8 +299,7 @@ async def dub_translate(req: TranslateRequest):
tasks = [loop.run_in_executor(_cpu_pool, _translate_llm, seg) for seg in req.segments]
translated = await asyncio.gather(*tasks)
translated.sort(key=lambda x: str(x["id"]))
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
**_dialect_flags(req, applied=True)}
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
# Offline Argos Translate
if provider == "argos" or provider == "libretranslate":
@@ -460,8 +353,7 @@ async def dub_translate(req: TranslateRequest):
return results
translated = await loop.run_in_executor(_cpu_pool, _translate_argos)
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
**_dialect_flags(req, applied=False)}
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
# Legacy / API Deep_Translator logic.
# Preflight the optional `deep_translator` dep once so we fail with a
@@ -570,8 +462,7 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
except Exception as e:
logger.debug("non-LLM rate_ratio prediction skipped: %s", e)
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
"quality_used": "fast", **_dialect_flags(req, applied=False)}
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang, "quality_used": "fast"}
if quality != "cinematic":
return base
@@ -601,19 +492,12 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
if not pairs:
return base
# #280 item 2: thread the regional-dialect hint into the reflect/adapt
# prompts. Guard against a stale dialect from another language.
dialect_hint = ""
if req.dialect and str(req.dialect).lower().startswith(str(req.target_lang).lower()[:2]):
dialect_hint = dialect_clause(req.dialect)
refined = await cinematic_refine_many(
pairs,
source_lang=src_lang,
target_lang=req.target_lang,
glossary=req.glossary,
directions=directions,
dialect_hint=dialect_hint,
executor=_cpu_pool,
)
refined_by_id = {r["id"]: r for r in refined}
@@ -676,5 +560,4 @@ async def _maybe_cinematic(translated, req, src_lang, loop):
"target_lang": req.target_lang,
"source_lang": src_lang,
"quality_used": "cinematic",
**_dialect_flags(req, applied=bool(dialect_hint)),
}
+6 -35
View File
@@ -266,26 +266,11 @@ class SelectEngineRequest(BaseModel):
backend_id: str
class SelectEngineResponse(BaseModel):
family: str
active: str
env_override: bool
# Routing verdict for the selected engine on THIS host (#21). Always present
# so the UI can show a confirm/warning toast on a cpu_fallback pick without
# branching on key presence; defaults match a legacy/degraded row.
routing_status: str = "cpu_only"
effective_device: str = "cpu"
routing_reason: str | None = None
@router.post("/engines/select", response_model=SelectEngineResponse)
@router.post("/engines/select")
def select_engine(req: SelectEngineRequest):
"""Persist a family's engine pick to prefs.json. Refuses unknown backends,
backends whose deps aren't installed, AND backends that cannot run on THIS
host's hardware (routing_status == "unavailable") — so the UI can't silently
brick a pipeline by picking an engine that needs a GPU this machine lacks.
A `cpu_fallback` pick is allowed (it runs, just slower) only a hard
`unavailable` is blocked. LLM is never routing-gated (its status is "n/a")."""
"""Persist a family's engine pick to prefs.json. Refuses unknown backends
+ refuses backends whose deps aren't installed (so the UI can't silently
brick a pipeline by picking an unavailable engine)."""
family = _FAMILIES.get(req.family)
if not family:
raise HTTPException(400, f"Unknown family: {req.family}. Expected one of tts/asr/llm.")
@@ -293,26 +278,12 @@ def select_engine(req: SelectEngineRequest):
available = {b["id"]: b for b in module.list_backends()}
if req.backend_id not in available:
raise HTTPException(400, f"Unknown {req.family} backend: {req.backend_id!r}")
entry = available[req.backend_id]
if not entry["available"]:
reason = entry.get("reason") or "unavailable"
if not available[req.backend_id]["available"]:
reason = available[req.backend_id].get("reason") or "unavailable"
raise HTTPException(400, f"Backend {req.backend_id} not ready: {reason}")
# Host-routing gate (no silent CPU fallback). `.get` is defensive so an
# older/legacy payload without routing keys still selects cleanly.
if entry.get("routing_status") == "unavailable":
why = entry.get("routing_reason") or "requires a GPU this host doesn't have"
raise HTTPException(
400,
f"Backend {req.backend_id} can't run on this machine: {why}. "
f"Pick an engine with a CPU path, or one that supports this host's GPU.",
)
prefs.set_(pref_key, req.backend_id)
return {
"family": req.family,
"active": module.active_backend_id(),
"env_override": bool(__import__("os").environ.get(f"OMNIVOICE_{req.family.upper()}_BACKEND")),
# Echo the routing verdict so the UI can warn on a cpu_fallback pick.
"routing_status": entry.get("routing_status", "cpu_only"),
"effective_device": entry.get("effective_device", "cpu"),
"routing_reason": entry.get("routing_reason"),
}
+64 -344
View File
@@ -59,87 +59,13 @@ def _render_with_pauses(gen_span, segments, sample_rate):
parts.append(torch.zeros(*shape, dtype=ref.dtype, device=ref.device))
return torch.cat(parts, dim=-1)
def _apply_effect_chain(audio_out, sample_rate, effect_preset, *, skip_mastering=False):
"""Shared post-DSP for /generate: preset validation → mastering →
effect chain loudness normalization.
``skip_mastering`` honors a backend's ``applies_own_mastering`` flag
(issue #312): studio engines (e.g. VoxCPM2's native 48 kHz output)
opt out of the broadcast Compressor + Reverb chain that's tuned for
OmniVoice's 24 kHz clone output. Loudness normalization still runs —
it's a benign peak scale. Mirrors ``_run_tts`` in openai_compat.py.
"""
from services.audio_dsp import (
EFFECT_PRESETS, apply_mastering, normalize_audio,
apply_effects_chain, get_effect_chain,
)
preset = effect_preset or "broadcast"
if preset not in EFFECT_PRESETS:
raise ValueError(
f"Unknown effect preset: {preset!r}. "
f"Valid: {list(EFFECT_PRESETS.keys())}"
)
if preset == "raw":
# Raw: skip all DSP — return raw model output
return audio_out
if not skip_mastering:
audio_out = apply_mastering(audio_out, sample_rate=sample_rate)
chain = get_effect_chain(preset)
if chain:
audio_out = apply_effects_chain(
audio_out, sample_rate=sample_rate, chain=chain,
)
return normalize_audio(audio_out, target_dBFS=-2.0)
def _oom_friendly_reraise(e):
"""Best-effort cache flush + the user-facing OOM hint shared by both
inference paths."""
import gc
import torch
gc.collect()
if torch.backends.mps.is_available():
torch.mps.empty_cache()
elif torch.cuda.is_available():
torch.cuda.empty_cache()
# #278: don't mislabel a torch.compile/Triton/Inductor crash as an
# out-of-memory condition. (model_manager's generate wrapper already
# retries these eagerly; this only triggers if that retry also died.)
from services.model_manager import _is_compile_runtime_failure
if _is_compile_runtime_failure(e):
raise RuntimeError(
f"TTS engine hit a torch.compile/Triton error (not out of memory). "
f"Disable torch.compile in Settings → Performance, use the Flush "
f"button to reload the model, then regenerate. Underlying error: {e}"
) from e
# #437: a Permission-denied / exec failure (e.g. a bundled engine binary
# that lost its +x bit) is NOT an OOM — don't send the user to the Flush
# button; tell them what's actually wrong.
es = str(e)
if isinstance(e, PermissionError) or "Permission denied" in es or "Errno 13" in es:
raise RuntimeError(
f"A required engine binary couldn't be executed (permission denied). "
f"This usually means a bundled binary lost its execute bit — reinstall, "
f"or run `chmod +x` on the engine binary named in the error. "
f"Underlying error: {e}"
) from e
raise RuntimeError(
f"TTS engine stopped mid-generation. This usually means it ran out of memory. "
f"Try the Flush button to reload the model, then regenerate. Underlying error: {e}"
)
def _run_inference(
model, text, language, ref_audio_path, ref_text, instruct, duration,
num_step, guidance_scale, speed, t_shift, denoise,
postprocess_output, layer_penalty_factor, position_temperature,
class_temperature, used_seed, effect_preset="broadcast",
max_chunk_chars=None, crossfade_ms=None,
):
from services.audio_dsp import apply_mastering, normalize_audio, apply_effects_chain, get_effect_chain
import torch
try:
if used_seed is not None:
@@ -173,124 +99,53 @@ def _run_inference(
)[0]
audio_out = _render_with_pauses(_gen_span, segments, sr)
else:
# Wave 1.2: long text is split at sentence boundaries and the
# per-chunk audio crossfaded — removes the length ceiling. Short
# text takes the single-shot path below unchanged. [pause] inputs
# keep the dedicated stitcher above (spans are already short).
from services.chunked_tts import (
DEFAULT_CROSSFADE_MS, DEFAULT_MAX_CHUNK_CHARS,
concatenate_audio_chunks, split_text_into_chunks,
audios = model.generate(
text=text, language=language, ref_audio=ref_audio_path,
ref_text=ref_text, instruct=instruct, duration=duration,
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
denoise=denoise, postprocess_output=postprocess_output,
**kwargs
)
_max_chars = DEFAULT_MAX_CHUNK_CHARS if max_chunk_chars is None else max_chunk_chars
_xfade_ms = DEFAULT_CROSSFADE_MS if crossfade_ms is None else crossfade_ms
text_chunks = split_text_into_chunks(text, _max_chars)
if len(text_chunks) > 1:
parts = []
for i, chunk_text in enumerate(text_chunks):
# Vary the seed per chunk (deterministically) to avoid
# correlated RNG artifacts across chunk boundaries.
if used_seed is not None:
torch.manual_seed(used_seed + i)
parts.append(model.generate(
text=chunk_text, language=language, ref_audio=ref_audio_path,
ref_text=ref_text, instruct=instruct, duration=None,
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
denoise=denoise, postprocess_output=postprocess_output,
**kwargs
)[0])
audio_out = concatenate_audio_chunks(parts, sr, _xfade_ms)
else:
audios = model.generate(
text=text, language=language, ref_audio=ref_audio_path,
ref_text=ref_text, instruct=instruct, duration=duration,
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
denoise=denoise, postprocess_output=postprocess_output,
**kwargs
)
audio_out = audios[0]
audio_out = audios[0]
# Apply DSP effect preset. The OmniVoice model never masters its own
# output, so mastering always runs here (unchanged behavior).
return _apply_effect_chain(audio_out, sr, effect_preset)
# Apply DSP effect preset
_effect_preset = effect_preset or "broadcast"
# Validate preset ID
from services.audio_dsp import EFFECT_PRESETS
if _effect_preset not in EFFECT_PRESETS:
raise ValueError(
f"Unknown effect preset: {_effect_preset!r}. "
f"Valid: {list(EFFECT_PRESETS.keys())}"
)
if _effect_preset == "raw":
# Raw: skip all DSP — return raw model output
return audio_out
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
_chain = get_effect_chain(_effect_preset)
if _chain:
mastered_audio = apply_effects_chain(
mastered_audio, sample_rate=sr, chain=_chain,
)
return normalize_audio(mastered_audio, target_dBFS=-2.0)
except ValueError as e:
# Don't wrap validation errors in OOM message
raise e
except Exception as e:
_oom_friendly_reraise(e)
def _run_backend_inference(
backend, text, language, ref_audio_path, ref_text, instruct, duration,
num_step, guidance_scale, speed, denoise, postprocess_output,
used_seed, effect_preset="broadcast",
max_chunk_chars=None, crossfade_ms=None,
):
"""Engine-aware twin of :func:`_run_inference` (issue #312).
Runs the request through a pluggable ``TTSBackend`` adapter instead of the
OmniVoice model directly. The adapter protocol is narrower than the
OmniVoice-native surface engine-specific extras (``t_shift``,
``layer_penalty_factor``, ) only exist on the native path, which is why
OmniVoice itself still goes through ``_run_inference``.
"""
import torch
try:
if used_seed is not None:
torch.manual_seed(used_seed)
if language and language.lower() == "auto":
language = None
gen_kwargs = dict(
language=language, ref_audio=ref_audio_path, ref_text=ref_text,
instruct=instruct, num_step=num_step, guidance_scale=guidance_scale,
speed=speed, denoise=denoise, postprocess_output=postprocess_output,
import gc
gc.collect()
if torch.backends.mps.is_available():
torch.mps.empty_cache()
elif torch.cuda.is_available():
torch.cuda.empty_cache()
raise RuntimeError(
f"TTS engine stopped mid-generation. This usually means it ran out of memory. "
f"Try the Flush button to reload the model, then regenerate. Underlying error: {e}"
)
sr = backend.sample_rate
# Inline [pause Nms] markers (issue #276) work for every engine — the
# silence stitching is model-free.
from omnivoice.utils.text import parse_pause_markers
segments = parse_pause_markers(text)
has_pause = len(segments) > 1 or (segments and segments[0][1] > 0)
if has_pause:
def _gen_span(span_text):
# Per-span duration is left to the engine; an explicit overall
# `duration` can't be meaningfully split across spans.
return backend.generate(span_text, duration=None, **gen_kwargs)
audio_out = _render_with_pauses(_gen_span, segments, sr)
else:
# Wave 1.2: sentence-boundary chunking for long text (see
# _run_inference for the rationale; behavior is identical here).
from services.chunked_tts import (
DEFAULT_CROSSFADE_MS, DEFAULT_MAX_CHUNK_CHARS,
concatenate_audio_chunks, split_text_into_chunks,
)
_max_chars = DEFAULT_MAX_CHUNK_CHARS if max_chunk_chars is None else max_chunk_chars
_xfade_ms = DEFAULT_CROSSFADE_MS if crossfade_ms is None else crossfade_ms
text_chunks = split_text_into_chunks(text, _max_chars)
if len(text_chunks) > 1:
parts = []
for i, chunk_text in enumerate(text_chunks):
if used_seed is not None:
torch.manual_seed(used_seed + i)
parts.append(backend.generate(chunk_text, duration=None, **gen_kwargs))
audio_out = concatenate_audio_chunks(parts, sr, _xfade_ms)
else:
audio_out = backend.generate(text, duration=duration, **gen_kwargs)
return _apply_effect_chain(
audio_out, sr, effect_preset,
skip_mastering=getattr(backend, "applies_own_mastering", False),
)
except ValueError as e:
# Don't wrap validation errors in OOM message
raise e
except Exception as e:
_oom_friendly_reraise(e)
@router.post("/generate")
@@ -313,88 +168,19 @@ async def generate_speech(
profile_id: Optional[str] = Form(None),
seed: Optional[int] = Form(None),
effect_preset: str = Form("broadcast"),
engine: Optional[str] = Form(None),
# Wave 1.2 — unlimited-length generation: long text is split at sentence
# boundaries and crossfaded. 0 disables chunking (whole text to engine).
max_chunk_chars: int = Form(800, ge=0),
crossfade_ms: int = Form(50, ge=0, le=1000),
):
# ── Engine resolution (issue #312) ──────────────────────────────────────
# The request runs on the engine selected in Settings (POST /engines/select,
# env var OMNIVOICE_TTS_BACKEND wins), or an explicit per-request `engine`
# override — same pattern as /ws/tts's `engine` field and /v1/audio/speech's
# `model`. Omitting both keeps the historical default (OmniVoice), so
# existing API consumers see no change.
from services.tts_backend import (
OmniVoiceBackend, _mask_hf_tokens, active_backend_id, get_backend_class,
)
engine_id = engine or active_backend_id()
try:
backend_cls = get_backend_class(engine_id)
except ValueError:
raise HTTPException(
status_code=400,
detail=(
f"Unknown TTS engine: {engine_id!r}. "
"See GET /engines/tts for the list of valid engine ids."
),
)
_model = None
_backend = None
if backend_cls is OmniVoiceBackend:
# OmniVoice keeps its native path: it carries the full advanced
# parameter surface (t_shift, layer/position/class controls) that the
# generic adapter protocol doesn't. Byte-identical to the old behavior.
_model = await get_model()
else:
try:
ok, msg = backend_cls.is_available()
except Exception as exc:
ok, msg = False, f"{type(exc).__name__}: {exc}"
if not ok:
raise HTTPException(
status_code=400,
detail=f"TTS engine '{engine_id}' is not available: {_mask_hf_tokens(msg)}",
)
# Reuse the per-process instance cache shared with the engine
# health-check route so weights load once, not per request.
from api.routers.engines import _get_engine_instance
_backend = _get_engine_instance(backend_cls)
# ── Routing gate (#21 — no silent CPU fallback). Computed ONCE per request
# (host caps are constant; the per-request engine= override bypasses the
# /engines/select gate, so this is the only place it's enforced for synth).
from core.device_caps import detect_host_caps
from services.engine_routing import resolve_routing, routing_notice
_routing = resolve_routing(getattr(backend_cls, "gpu_compat", ("cpu",)), detect_host_caps())
if _routing["routing_status"] == "unavailable":
# The engine needs an accelerator this host lacks and has no CPU path.
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
_routing_notice = routing_notice(_routing) # (status, reason) or None
_model = await get_model()
ref_audio_path = None
cleanup_ref = False
used_seed = seed
resolved_profile_id = None
history_mode = None # profile.kind when a profile drives; else inferred at insert
if profile_id:
with db_conn() as conn:
row = conn.execute("SELECT * FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
if row:
resolved_profile_id = profile_id
# `kind` is authoritative (0005): 'design' profiles condition on
# their deterministic rendered sample + instruct; 'clone' on the
# user's reference. Lock always wins (it pins a specific take).
# Rows from pre-0004 DBs mid-upgrade may lack the column → fall
# back to the legacy is_locked/instruct inference.
try:
profile_kind = row["kind"] or "clone"
except (KeyError, IndexError):
profile_kind = "design" if (row["instruct"] and not row["is_locked"] and not row["ref_audio_path"]) else "clone"
history_mode = profile_kind
if row["is_locked"] and row["locked_audio_path"]:
ref_audio_path = os.path.join(VOICES_DIR, row["locked_audio_path"])
if not ref_text:
@@ -403,19 +189,7 @@ async def generate_speech(
instruct = row["instruct"]
if used_seed is None and row["seed"] is not None:
used_seed = row["seed"]
elif profile_kind == "design":
# Rendered sample (if present) carries the voice identity;
# instruct alone is the fallback for legacy archetype rows.
ref_audio_path = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
if ref_audio_path and not ref_text and row["ref_text"]:
ref_text = row["ref_text"]
if not instruct:
instruct = row["instruct"]
if used_seed is None and row["seed"] is not None:
used_seed = row["seed"]
elif row["instruct"] and not row["is_locked"] and not row["ref_audio_path"]:
# Legacy design-shaped row (pre-0004 archetype materialization
# failure path): instruct-only conditioning.
elif row["instruct"] and not row["is_locked"]:
if not instruct:
instruct = row["instruct"]
if used_seed is None and row["seed"] is not None:
@@ -439,41 +213,16 @@ async def generate_speech(
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
# #308: a transcript-less reference is transcribed with the active ASR
# backend (whisperx / faster-whisper / mlx-whisper) instead of the model's
# built-in transformers pipeline, which cannot load whisper-large-v3-turbo
# on transformers 5.3. On failure ref_text stays None and the model's
# fallback behaves exactly as before.
if ref_audio_path and not ref_text:
from services.asr_backend import transcribe_reference
ref_text = await asyncio.get_running_loop().run_in_executor(
_gpu_pool, transcribe_reference, ref_audio_path
)
start_time = time.time()
try:
loop = asyncio.get_running_loop()
if _backend is not None:
audio_tensor = await loop.run_in_executor(
_gpu_pool, _run_backend_inference,
_backend, text, language, ref_audio_path, ref_text, instruct,
duration, num_step, guidance_scale, speed, denoise,
postprocess_output, used_seed, effect_preset,
max_chunk_chars, crossfade_ms,
)
# Read after generation: engines with lazy model loading report
# their real rate only once weights are up.
sample_rate = _backend.sample_rate
else:
audio_tensor = await loop.run_in_executor(
_gpu_pool, _run_inference,
_model, text, language, ref_audio_path, ref_text, instruct, duration,
num_step, guidance_scale, speed, t_shift, denoise,
postprocess_output, layer_penalty_factor, position_temperature,
class_temperature, used_seed, effect_preset,
max_chunk_chars, crossfade_ms,
)
sample_rate = _model.sampling_rate
audio_tensor = await loop.run_in_executor(
_gpu_pool, _run_inference,
_model, text, language, ref_audio_path, ref_text, instruct, duration,
num_step, guidance_scale, speed, t_shift, denoise,
postprocess_output, layer_penalty_factor, position_temperature,
class_temperature, used_seed, effect_preset,
)
# Invisible AudioSeal provenance watermark on the final audio. Embedding
# was previously only wired into the dub pipeline (dub_generate.py), so
# plain TTS came out unmarked despite the setting being on. embed_watermark
@@ -482,28 +231,28 @@ async def generate_speech(
# generation.
from services.watermark import embed_watermark
audio_tensor = await loop.run_in_executor(
_gpu_pool, embed_watermark, audio_tensor, sample_rate
_gpu_pool, embed_watermark, audio_tensor, _model.sampling_rate
)
gen_time = round(time.time() - start_time, 2)
audio_id = str(uuid.uuid4())[:8]
audio_filename = f"{audio_id}.wav"
audio_path = os.path.join(OUTPUTS_DIR, audio_filename)
_safe_torchaudio_save(audio_path, audio_tensor, sample_rate)
_safe_torchaudio_save(audio_path, audio_tensor, _model.sampling_rate)
audio_dur = round(audio_tensor.shape[-1] / sample_rate, 2)
audio_dur = round(audio_tensor.shape[-1] / _model.sampling_rate, 2)
with db_conn() as conn:
conn.execute(
"INSERT INTO generation_history (id, text, mode, language, instruct, profile_id, audio_path, duration_seconds, generation_time, seed, created_at) VALUES (?,?,?,?,?,?,?,?,?,?,?)",
(audio_id, text[:200], history_mode or ("clone" if ref_audio_path else "design"),
(audio_id, text[:200], "clone" if ref_audio_path else "design",
language or "Auto", instruct or "", resolved_profile_id,
audio_filename, audio_dur, gen_time, used_seed, time.time())
)
event_bus.emit("generation_history", {"action": "created", "id": audio_id})
buffer = io.BytesIO()
_safe_torchaudio_save(buffer, audio_tensor, sample_rate, format="wav")
_safe_torchaudio_save(buffer, audio_tensor, _model.sampling_rate, format="wav")
buffer.seek(0)
wav_bytes = buffer.read()
@@ -512,26 +261,17 @@ async def generate_speech(
for i in range(0, len(wav_bytes), chunk_size):
yield wav_bytes[i:i + chunk_size]
_resp_headers = {
"X-Audio-Id": audio_id,
"X-Gen-Time": str(gen_time),
"X-Audio-Path": audio_filename,
"X-Seed": str(used_seed) if used_seed is not None else "",
"X-Audio-Duration": str(audio_dur),
"Content-Length": str(len(wav_bytes)),
}
# Routing notice (#21): cpu_fallback or accelerated-with-caveat only;
# the WAV body is binary so the header channel is the carrier.
if _routing_notice:
from services.engine_routing import header_safe_reason
_resp_headers["X-OmniVoice-Routing"] = _routing_notice[0]
_hr = header_safe_reason(_routing_notice[1])
if _hr:
_resp_headers["X-OmniVoice-Routing-Reason"] = _hr
return StreamingResponse(
_stream_wav(),
media_type="audio/wav",
headers=_resp_headers,
headers={
"X-Audio-Id": audio_id,
"X-Gen-Time": str(gen_time),
"X-Audio-Path": audio_filename,
"X-Seed": str(used_seed) if used_seed is not None else "",
"X-Audio-Duration": str(audio_dur),
"Content-Length": str(len(wav_bytes)),
}
)
except HTTPException:
raise
@@ -568,29 +308,9 @@ def _safe_output_path(name):
@router.get("/history")
def list_history():
"""Newest 50 generations whose audio still exists on disk.
Rows whose WAV was deleted out-of-band (cleared outputs dir, manual
cleanup) used to come back anyway and render dead players that 404 on
every fetch; prune them here so the UI never sees them again."""
with db_conn() as conn:
rows = conn.execute(
"SELECT * FROM generation_history ORDER BY created_at DESC LIMIT 50"
).fetchall()
alive, stale_ids = [], []
for r in rows:
p = _safe_output_path(r["audio_path"]) if r["audio_path"] else None
if r["audio_path"] and (not p or not os.path.exists(p)):
stale_ids.append(r["id"])
else:
alive.append(dict(r))
if stale_ids:
conn.executemany(
"DELETE FROM generation_history WHERE id=?",
[(i,) for i in stale_ids],
)
logger.info("pruned %d stale history rows (audio file gone)", len(stale_ids))
return alive
rows = conn.execute("SELECT * FROM generation_history ORDER BY created_at DESC LIMIT 50").fetchall()
return [dict(r) for r in rows]
@router.delete("/history")
def clear_history():
-158
View File
@@ -1,158 +0,0 @@
"""Longform Job Library (PR 7).
``GET /longform/jobs`` list finished Audiobook + Story renders so the user can
re-download them from the Projects view. The render itself (the m4b/mp3) already
landed in ``OUTPUTS_DIR`` and is served at ``/audio/<output>``; here we just
recover, from each finished job's persisted SSE tail, the output filename plus
the chapter count and duration the ``done`` event carried.
Pure recovery, no synthesis. Defensive by construction: a job whose ``done``
event is missing or unparseable is skipped, never surfaced and never a 500.
The work lives in :func:`build_longform_library`, a pure function over the
job-store callables, so it's unit-testable without importing ``main`` (and the
torch graph behind it).
"""
from __future__ import annotations
import json
import logging
from typing import Callable, Optional
from fastapi import APIRouter, Query
logger = logging.getLogger("omnivoice.longform_jobs")
router = APIRouter()
#: Job types this library surfaces. Both flow through the shared longform
#: renderer (``_render_longform_sse``) and emit the same ``done`` event shape.
_LONGFORM_TYPES = ("audiobook", "story")
def _done_payload_from_events(events: list[dict]) -> Optional[dict]:
"""Recover the final ``{"type": "done", ...}`` payload from a job's SSE tail.
Each row's ``payload`` is the JSON the renderer stored via
``job_store.append_event(job_id, json.dumps(payload))``. We scan newest-first
and return the first parseable ``done`` event. Anything malformed is skipped
this never raises.
"""
for ev in reversed(events):
raw = ev.get("payload") if isinstance(ev, dict) else None
if not raw or not isinstance(raw, str):
continue
try:
obj = json.loads(raw)
except (ValueError, TypeError):
continue
if isinstance(obj, dict) and obj.get("type") == "done":
return obj
return None
def _coerce_int(value, default: int = 0) -> int:
try:
return int(value)
except (TypeError, ValueError):
return default
def _coerce_float(value, default: float = 0.0) -> float:
try:
return float(value)
except (TypeError, ValueError):
return default
def build_longform_library(
list_jobs: Callable[..., list[dict]],
events_since: Callable[..., list[dict]],
*,
limit: int = 50,
) -> list[dict]:
"""Build the newest-first list of finished longform renders.
Pure over the two job-store callables so tests can pass them directly:
* ``list_jobs(status="done", limit=...)`` all done jobs, newest-first.
* ``events_since(job_id)`` that job's persisted SSE events.
Returns ``[{job_id, type, title?, output, duration_s, chapters,
created_at}]``. Jobs that aren't a longform type, or whose ``done`` event /
output filename can't be recovered, are silently skipped — the library only
ever lists things the user can actually re-download.
"""
limit = max(1, min(_coerce_int(limit, 50), 500))
try:
# Over-fetch: non-longform done jobs (dub, etc.) get filtered out below,
# so ask for more rows than the caller's limit to still fill the page.
rows = list_jobs(status="done", limit=limit * 4)
except Exception:
logger.warning("longform library: list_jobs failed", exc_info=True)
return []
out: list[dict] = []
for row in rows or []:
if len(out) >= limit:
break
try:
job_type = row.get("type")
job_id = row.get("id")
if job_type not in _LONGFORM_TYPES or not job_id:
continue
try:
events = events_since(job_id)
except Exception:
logger.warning("longform library: events_since failed for %s",
job_id, exc_info=True)
continue
done = _done_payload_from_events(events or [])
if not done:
continue
output = done.get("output")
if not output or not isinstance(output, str):
continue # nothing to re-download → not worth listing
item = {
"job_id": job_id,
"type": job_type,
"output": output,
"duration_s": round(_coerce_float(done.get("duration_s")), 2),
"chapters": _coerce_int(done.get("chapters")),
"created_at": row.get("created_at"),
}
# Title is optional — prefer the done event, fall back to job meta.
title = done.get("title")
if not title:
meta_raw = row.get("meta_json")
if isinstance(meta_raw, str) and meta_raw:
try:
meta = json.loads(meta_raw)
if isinstance(meta, dict):
title = meta.get("title")
except (ValueError, TypeError):
title = None
if title:
item["title"] = title
out.append(item)
except Exception:
# Per-row isolation: one bad row never sinks the whole list.
logger.warning("longform library: skipping unparseable job row",
exc_info=True)
continue
return out
@router.get("/longform/jobs")
def longform_jobs(limit: int = Query(50, ge=1, le=500)) -> dict:
"""Finished Audiobook + Story renders, newest-first, ready to re-download.
Each item's ``output`` is served at ``/audio/<output>``. Never 500s — on any
backend hiccup it returns an empty list rather than an error.
"""
from core import job_store
jobs = build_longform_library(
job_store.list_jobs, job_store.events_since, limit=limit,
)
return {"jobs": jobs}
+28 -40
View File
@@ -58,31 +58,6 @@ MAX_BUNDLE_BYTES = 100 * 1024 * 1024
# ── Export ──────────────────────────────────────────────────────────────────
def _bundle_metadata(profile: dict, **extra) -> dict:
"""Common .omnivoice metadata for export + publish.
Captures ``kind`` and ``vd_states`` so a *designed* persona survives the
bundle round-trip as a design (not silently demoted to a clone) required
for the synthetic-only gating of the persona gallery (§R3). Old bundles
without these keys import as ``kind='clone'`` (backward-compatible).
"""
meta = {
"bundle_version": BUNDLE_VERSION,
"profile_name": profile.get("name", "Unnamed"),
"ref_text": profile.get("ref_text", ""),
"instruct": profile.get("instruct", ""),
"language": profile.get("language", "Auto"),
"personality": profile.get("personality", ""),
"seed": profile.get("seed"),
"kind": profile.get("kind") or "clone",
"vd_states": profile.get("vd_states"),
"is_locked": bool(profile.get("is_locked")),
"omnivoice_version": APP_VERSION,
}
meta.update(extra)
return meta
@router.post("/export/{profile_id}")
def export_profile(profile_id: str):
"""Export a voice profile as a downloadable .omnivoice bundle (ZIP)."""
@@ -100,9 +75,19 @@ def export_profile(profile_id: str):
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
# Metadata
metadata = _bundle_metadata(
profile, created_at=profile.get("created_at"), exported_at=time.time(),
)
metadata = {
"bundle_version": BUNDLE_VERSION,
"profile_name": profile.get("name", "Unnamed"),
"ref_text": profile.get("ref_text", ""),
"instruct": profile.get("instruct", ""),
"language": profile.get("language", "Auto"),
"personality": profile.get("personality", ""),
"seed": profile.get("seed"),
"is_locked": bool(profile.get("is_locked")),
"created_at": profile.get("created_at"),
"exported_at": time.time(),
"omnivoice_version": APP_VERSION,
}
zf.writestr("metadata.json", json.dumps(metadata, indent=2))
# Reference audio
@@ -206,9 +191,8 @@ async def import_profile(
conn.execute(
"""INSERT INTO voice_profiles
(id, name, ref_audio_path, ref_text, instruct, language,
seed, personality, is_locked, locked_audio_path, created_at,
kind, vd_states)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
seed, personality, is_locked, locked_audio_path, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(
profile_id,
metadata.get("profile_name", "Imported Voice"),
@@ -221,10 +205,6 @@ async def import_profile(
1 if is_locked else 0,
locked_audio_filename or "",
time.time(),
# Preserve the design/clone distinction across the round-trip;
# old bundles without these keys import as a clone.
metadata.get("kind") or "clone",
metadata.get("vd_states"),
),
)
@@ -273,11 +253,19 @@ def publish_to_marketplace(
# Build the bundle
with zipfile.ZipFile(str(bundle_path), "w", zipfile.ZIP_DEFLATED) as zf:
metadata = _bundle_metadata(
profile,
tags=[t.strip() for t in tags.split(",") if t.strip()],
published_at=time.time(),
)
metadata = {
"bundle_version": BUNDLE_VERSION,
"profile_name": profile.get("name", "Unnamed"),
"ref_text": profile.get("ref_text", ""),
"instruct": profile.get("instruct", ""),
"language": profile.get("language", "Auto"),
"personality": profile.get("personality", ""),
"seed": profile.get("seed"),
"is_locked": bool(profile.get("is_locked")),
"tags": [t.strip() for t in tags.split(",") if t.strip()],
"published_at": time.time(),
"omnivoice_version": APP_VERSION,
}
zf.writestr("metadata.json", json.dumps(metadata, indent=2))
ref_path = profile.get("ref_audio_path")
-53
View File
@@ -1,53 +0,0 @@
"""REST CRUD for per-agent MCP voice bindings (Wave 2.2 / Spec 2).
Loopback-gated the Settings UI manages bindings here. The MCP tools
themselves resolve voices via ``services.mcp_bindings.resolve_voice``.
"""
from __future__ import annotations
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from api.dependencies import require_loopback
from services import mcp_bindings
router = APIRouter(
prefix="/api/mcp",
tags=["mcp"],
dependencies=[Depends(require_loopback)],
)
class _BindingBody(BaseModel):
client_id: str = Field(..., min_length=1, max_length=128)
label: str | None = None
profile_id: str | None = None
default_engine: str | None = None
@router.get("/bindings")
def list_bindings():
"""All per-agent voice bindings, most-recently-seen first."""
return mcp_bindings.list_bindings()
@router.put("/bindings")
def upsert_binding(body: _BindingBody):
"""Create or update the binding for an MCP client id."""
try:
return mcp_bindings.upsert_binding(
body.client_id,
label=body.label,
profile_id=body.profile_id,
default_engine=body.default_engine,
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@router.delete("/bindings/{client_id}")
def delete_binding(client_id: str):
if not mcp_bindings.delete_binding(client_id):
raise HTTPException(status_code=404, detail="No binding for that client id")
return {"deleted": client_id}
+5 -66
View File
@@ -82,33 +82,6 @@ class SpeechRequest(BaseModel):
"E.g. 'young female, warm tone, slight British accent'.",
)
instruct: Optional[str] = Field(default=None, description="Style instruction for the TTS engine.")
duration: Optional[float] = Field(
default=None,
gt=0,
description="OmniVoice extension: target output duration in seconds.",
)
seed: Optional[int] = Field(
default=None,
description="OmniVoice extension: deterministic sampling seed.",
)
denoise: bool = Field(
default=True,
description="OmniVoice extension: prepend denoise control when supported.",
)
preprocess_prompt: bool = Field(
default=True,
description="OmniVoice extension: trim/preprocess reference prompt when supported.",
)
chunk_duration: Optional[float] = Field(
default=None,
ge=0,
description="OmniVoice GGUF extension: long-form internal chunk duration.",
)
chunk_threshold: Optional[float] = Field(
default=None,
ge=0,
description="OmniVoice GGUF extension: long-form internal chunk threshold.",
)
class TranscriptionResponse(BaseModel):
@@ -236,14 +209,7 @@ def _run_tts(backend, text: str, kw: dict):
from services.audio_dsp import apply_mastering, normalize_audio
wav = backend.generate(text, **kw)
sr = backend.sample_rate
# Engines that already emit mastered, studio-grade audio (e.g. VoxCPM2's
# native 48 kHz) opt out of apply_mastering via `applies_own_mastering`.
# That chain's Compressor + 8% Reverb is tuned for OmniVoice's 24 kHz clone
# output; applied to a studio engine it adds an audible level pump and a
# reverb tail that degrade the very output we want clean. Loudness
# normalisation still runs — it's a benign peak scale, not dynamics.
if not getattr(backend, "applies_own_mastering", False):
wav = apply_mastering(wav, sample_rate=sr)
wav = apply_mastering(wav, sample_rate=sr)
wav = normalize_audio(wav, target_dBFS=-2.0)
return wav, sr
@@ -253,28 +219,10 @@ async def create_speech(req: SpeechRequest):
"""Generate audio from text. Compatible with OpenAI's POST /v1/audio/speech."""
backend = _resolve_engine(req.model)
# Routing gate (#21 — no silent CPU fallback), identical to REST /generate.
from core.device_caps import detect_host_caps
from services.engine_routing import resolve_routing, routing_notice
_routing = resolve_routing(getattr(backend, "gpu_compat", ("cpu",)), detect_host_caps())
if _routing["routing_status"] == "unavailable":
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
_routing_notice = routing_notice(_routing) # (status, reason) or None
# Build kwargs for the backend's generate() method
kw: dict = {
"speed": req.speed,
"denoise": req.denoise,
"preprocess_prompt": req.preprocess_prompt,
}
if req.duration is not None:
kw["duration"] = req.duration
if req.seed is not None:
kw["seed"] = req.seed
if req.chunk_duration is not None:
kw["chunk_duration"] = req.chunk_duration
if req.chunk_threshold is not None:
kw["chunk_threshold"] = req.chunk_threshold
if req.language:
kw["language"] = req.language
if req.instruct:
@@ -303,8 +251,6 @@ async def create_speech(req: SpeechRequest):
kw["ref_text"] = row["ref_text"]
if row["instruct"] and not req.instruct:
kw["instruct"] = row["instruct"]
if req.seed is None and row["seed"] is not None:
kw["seed"] = row["seed"]
else:
# Not a profile ID — forward as engine preset name
kw["voice"] = voice
@@ -321,20 +267,13 @@ async def create_speech(req: SpeechRequest):
audio_bytes, mime_type, ext = _encode_audio(wav, sr, req.response_format)
_headers = {
"Content-Length": str(len(audio_bytes)),
"Content-Disposition": f'inline; filename="speech.{ext}"',
}
if _routing_notice:
from services.engine_routing import header_safe_reason
_headers["X-OmniVoice-Routing"] = _routing_notice[0]
_hr = header_safe_reason(_routing_notice[1])
if _hr:
_headers["X-OmniVoice-Routing-Reason"] = _hr
return StreamingResponse(
io.BytesIO(audio_bytes),
media_type=mime_type,
headers=_headers,
headers={
"Content-Length": str(len(audio_bytes)),
"Content-Disposition": f'inline; filename="speech.{ext}"',
},
)
-327
View File
@@ -1,327 +0,0 @@
"""HTTP layer for the `.ovsvoice` portable persona format (#29 / parity §R3 G1).
Thin router over `services.persona_bundle`:
POST /personas/export/{profile_id} stream a downloadable .ovsvoice
POST /personas/import create a profile from a bundle
POST /personas/inspect read a bundle's manifest, no writes
Mirrors the legacy `.omnivoice` endpoints (`marketplace.py`) and reuses the
same path-confinement (`_voices_path`) + consent floor. `.ovsvoice` is additive;
`.omnivoice` import stays a compatible legacy reader.
"""
from __future__ import annotations
import asyncio
import functools
import logging
import os
import time
import uuid
from fastapi import APIRouter, File, HTTPException, Query, UploadFile
from fastapi.responses import StreamingResponse
from core import event_bus
from core.config import VOICES_DIR # noqa: F401 — re-exported for tests/monkeypatch
from core.db import db_conn
from core.version import APP_VERSION
from services import persona_bundle as pb
router = APIRouter()
logger = logging.getLogger("omnivoice.personas")
def _safe_name(name: str, profile_id: str) -> str:
"""Sanitised download filename stem (marketplace idiom); empty → persona_<id>."""
cleaned = "".join(
c if c.isalnum() or c in "-_ " else "" for c in (name or "")
).strip().replace(" ", "_")[:40]
return cleaned or f"persona_{profile_id}"
# ── Export ────────────────────────────────────────────────────────────────
@router.post("/personas/export/{profile_id}")
async def export_persona(
profile_id: str,
license_spdx: str = Query(pb.DEFAULT_LICENSE),
tags: str = Query(""),
include_reference: bool = Query(True),
):
"""Build + stream a `.ovsvoice` bundle for a profile."""
with db_conn() as conn:
row = conn.execute(
"SELECT * FROM voice_profiles WHERE id = ?", (profile_id,)
).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Voice profile not found")
profile = dict(row)
tag_list = [t.strip() for t in tags.split(",") if t.strip()]
try:
loop = asyncio.get_running_loop()
content = await loop.run_in_executor(
None,
functools.partial(
pb.build_persona_bundle,
profile,
license_spdx=license_spdx,
tags=tag_list,
include_reference=include_reference,
engine_id=os.environ.get("OMNIVOICE_MODEL", ""),
omnivoice_version=APP_VERSION,
),
)
except pb.NoPreviewSource:
raise HTTPException(
status_code=503,
detail="This profile has no readable reference or locked audio to "
"build a preview from — re-create or re-import it.",
)
except Exception:
logger.exception("persona export failed for %s", profile_id)
raise HTTPException(
status_code=503,
detail="Could not build the persona bundle — see Settings → Logs.",
)
filename = f"{_safe_name(profile.get('name'), profile_id)}.ovsvoice"
from io import BytesIO
return StreamingResponse(
BytesIO(content),
media_type="application/zip",
headers={
"Content-Disposition": f'attachment; filename="{filename}"',
"Content-Length": str(len(content)),
},
)
# ── Import ────────────────────────────────────────────────────────────────
def _voices_dest(filename: str) -> str:
"""Resolve an output filename inside VOICES_DIR; 400 on escape (belt+braces —
the name is always server-generated `{profile_id}`)."""
from api.routers.profiles import _voices_path
path = _voices_path(filename)
if path is None:
raise HTTPException(status_code=400, detail="Invalid profile id")
return path
def _consent_verified(parsed: pb.ParsedPersona, consent_path: str | None) -> bool:
"""B12-B16: trust verified-own-voice ONLY with a real recording (≥ floor) AND
non-empty consent_text AND a consent.json present. The manifest flag alone
can't forge it."""
if not parsed.consent or not consent_path:
return False
if os.path.getsize(consent_path) < pb._MIN_CONSENT_AUDIO_BYTES:
return False
return bool((parsed.consent.get("consent_text") or "").strip())
@router.post("/personas/import")
async def import_persona(file: UploadFile = File(...)):
"""Create a new voice profile from a `.ovsvoice` (or legacy `.omnivoice`) bundle."""
name = (file.filename or "").lower()
if not name.endswith(".ovsvoice") and not name.endswith(".omnivoice"):
raise HTTPException(status_code=400, detail="File must be a .ovsvoice or .omnivoice bundle")
content = await file.read()
try:
parsed = pb.parse_persona_bundle(content)
except pb.BundleError as e:
raise HTTPException(status_code=e.status, detail=e.detail)
persona = parsed.manifest.get("persona") or {}
written: list[str] = []
def _gen_id() -> str:
return str(uuid.uuid4())[:8]
profile_id = _gen_id()
try:
# ── Audio members → server-named files (never the member name). ──
ref_filename = None
locked_filename = None
if "ref_audio" in parsed.members:
ref_filename = f"{profile_id}{parsed.member_ext('ref_audio')}"
dest = _voices_dest(ref_filename)
parsed.extract_member("ref_audio", dest); written.append(dest)
if "locked_audio" in parsed.members:
locked_filename = f"{profile_id}_locked{parsed.member_ext('locked_audio')}"
dest = _voices_dest(locked_filename)
parsed.extract_member("locked_audio", dest); written.append(dest)
# Preview-only bundle (A12/B8): use the preview as the usable ref clip.
if ref_filename is None and locked_filename is None and "preview" in parsed.members:
ref_filename = f"{profile_id}{parsed.member_ext('preview')}"
dest = _voices_dest(ref_filename)
parsed.extract_member("preview", dest); written.append(dest)
if ref_filename is None and locked_filename is None:
raise HTTPException(status_code=400, detail="bundle has no usable audio")
# ── Consent recording (optional) ──
consent_filename = None
consent_path = None
if "consent_audio" in parsed.members:
consent_filename = f"{profile_id}_consent{parsed.member_ext('consent_audio')}"
consent_path = _voices_dest(consent_filename)
parsed.extract_member("consent_audio", consent_path); written.append(consent_path)
verified = _consent_verified(parsed, consent_path)
consent_text = ((parsed.consent or {}).get("consent_text") or "").strip()
recorded_at = None
if verified:
try:
recorded_at = float(parsed.consent.get("recorded_at"))
except (TypeError, ValueError):
recorded_at = time.time()
is_locked = bool(persona.get("is_locked") and locked_filename)
ref_for_db = ref_filename or locked_filename # at least one is set
def _insert(pid: str):
with db_conn() as conn:
conn.execute(
"""INSERT INTO voice_profiles
(id, name, ref_audio_path, ref_text, instruct, language,
seed, personality, is_locked, locked_audio_path, created_at,
kind, vd_states,
verified_own_voice, consent_text, consent_audio_path, consent_recorded_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(
pid,
persona.get("name") or "Imported Voice",
ref_for_db,
persona.get("ref_text", ""),
persona.get("instruct", ""),
persona.get("language", "Auto"),
persona.get("seed"),
persona.get("personality", ""),
1 if is_locked else 0,
locked_filename or "",
time.time(),
persona.get("kind") or "clone",
persona.get("vd_states"),
1 if verified else 0,
# Keep the attestation text so the user can re-attest locally,
# even when imported unverified.
consent_text,
consent_filename if verified else "",
recorded_at if verified else None,
),
)
import sqlite3
try:
_insert(profile_id)
except sqlite3.IntegrityError:
profile_id = _gen_id() # one retry on id collision (B20)
# rename the on-disk files to the new id so they still match the row
written = _rename_for_new_id(written, profile_id)
ref_for_db = _retarget(ref_for_db, profile_id)
locked_filename = _retarget(locked_filename, profile_id)
consent_filename = _retarget(consent_filename, profile_id)
_insert(profile_id)
except HTTPException:
_cleanup(written)
raise
except Exception:
_cleanup(written)
logger.exception("persona import failed")
raise HTTPException(status_code=500, detail="Import failed; no files were kept.")
event_bus.emit("profiles", {"action": "created", "id": profile_id})
logger.info("Imported persona %r as %s (verified=%s)", persona.get("name"), profile_id, verified)
return {
"success": True,
"profile_id": profile_id,
"name": persona.get("name") or "Imported Voice",
"kind": persona.get("kind") or "clone",
"verified_own_voice": verified,
"preview_only": parsed.preview_only,
"license_spdx": parsed.license_spdx,
"watermarked_preview": parsed.watermarked_preview,
"source_bundle": file.filename,
"schema_version_ahead": parsed.schema_version_ahead,
}
def _cleanup(paths: list[str]) -> None:
for p in paths:
try:
if p and os.path.exists(p):
os.remove(p)
except OSError:
pass
def _rename_for_new_id(written: list[str], new_id: str) -> list[str]:
"""After an id-collision retry, rename each written file to carry the new id
(filenames are `{old_id}`; swap the leading 8-char stem)."""
out = []
for p in written:
d, base = os.path.split(p)
# base looks like {id}{ext} | {id}_locked{ext} | {id}_consent{ext}
new_base = new_id + base[8:]
new_path = os.path.join(d, new_base)
try:
os.replace(p, new_path)
out.append(new_path)
except OSError:
out.append(p)
return out
def _retarget(filename: str | None, new_id: str) -> str | None:
return new_id + filename[8:] if filename else filename
# ── Inspect (no-write preview) ──────────────────────────────────────────────
@router.post("/personas/inspect")
async def inspect_persona(file: UploadFile = File(...)):
"""Read a bundle's manifest + consent summary WITHOUT writing any file or row."""
name = (file.filename or "").lower()
if not name.endswith(".ovsvoice") and not name.endswith(".omnivoice"):
raise HTTPException(status_code=400, detail="File must be a .ovsvoice or .omnivoice bundle")
content = await file.read()
try:
parsed = pb.parse_persona_bundle(content)
except pb.BundleError as e:
raise HTTPException(status_code=e.status, detail=e.detail)
persona = parsed.manifest.get("persona") or {}
consent_summary = None
if parsed.consent:
has_recording = "consent_audio" in parsed.members
consent_summary = {
"verified_claimed": bool(parsed.consent.get("verified_own_voice")),
"method": parsed.consent.get("method", ""),
"has_recording": has_recording,
# would_verify mirrors import's gate, minus the byte-floor check
# (inspect never extracts to measure size — advisory only).
"would_verify": has_recording and bool((parsed.consent.get("consent_text") or "").strip()),
}
return {
"format": "omnivoice-legacy" if parsed.is_legacy else pb.OVSVOICE_FORMAT,
"schema_version": parsed.manifest.get("schema_version", pb.OVSVOICE_SCHEMA_VERSION),
"name": persona.get("name") or "Imported Voice",
"kind": persona.get("kind") or "clone",
"language": persona.get("language", "Auto"),
"personality": persona.get("personality", ""),
"is_locked": bool(persona.get("is_locked")),
"license_spdx": parsed.license_spdx,
"tags": parsed.manifest.get("tags") or [],
"preview_only": parsed.preview_only,
"watermarked_preview": parsed.watermarked_preview,
"consent": consent_summary,
"schema_version_ahead": parsed.schema_version_ahead,
}
+18 -276
View File
@@ -1,5 +1,4 @@
import os
import re
import uuid
import time
import shutil
@@ -35,103 +34,29 @@ def list_profiles():
rows = conn.execute("SELECT * FROM voice_profiles ORDER BY created_at DESC").fetchall()
return [dict(r) for r in rows]
_DESIGN_SEED = 42 # deterministic sample render, same as archetype previews
@router.post("/profiles")
async def create_profile(
name: str = Form(...),
ref_audio: Optional[UploadFile] = File(None),
ref_audio: UploadFile = File(...),
ref_text: str = Form(""),
instruct: str = Form(""),
language: str = Form("Auto"),
seed: Optional[int] = Form(None),
personality: str = Form(""),
kind: str = Form("clone"),
vd_states: Optional[str] = Form(None),
):
"""Create a voice profile (spec: docs/specs/voice-studio-unification.md §5).
kind='clone' requires `ref_audio` (the user's reference recording).
kind='design' requires `vd_states` (JSON of category picks); the server
renders a deterministic sample WAV (seed 42, same path as
archetype materialization) and stores it as the profile's
reference so the voice identity is stable across runs.
"""
if kind not in ("clone", "design"):
raise HTTPException(status_code=422, detail="kind must be 'clone' or 'design'")
if kind == "clone" and ref_audio is None:
raise HTTPException(status_code=422, detail="clone profiles require ref_audio")
if kind == "design":
if not (vd_states or "").strip():
raise HTTPException(status_code=422, detail="design profiles require vd_states")
import json as _json
try:
parsed = _json.loads(vd_states)
if not isinstance(parsed, dict):
raise ValueError("not an object")
except ValueError:
raise HTTPException(status_code=422, detail="vd_states must be a JSON object")
# An all-Auto design (every category left on "Auto") yields an empty
# instruct — that's still a valid, saveable voice: synthesis falls back
# to neutral instruct-only conditioning (see generation.py design path).
# Don't gate save on a non-empty instruct.
profile_id = str(uuid.uuid4())[:8]
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
audio_filename = f"{profile_id}{ext}"
audio_path = os.path.join(VOICES_DIR, audio_filename)
if kind == "clone":
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
audio_filename = f"{profile_id}{ext}"
audio_path = os.path.join(VOICES_DIR, audio_filename)
with open(audio_path, "wb") as f:
f.write(await ref_audio.read())
used_seed = seed
else:
# Saving a design profile is a pure persistence operation — it must not
# depend on a loaded TTS model (issue #476: on a fresh model-less Docker
# image the render forced a full model load + inference that 503'd, so
# the save failed). We try the deterministic identity sample opportunist-
# ically through the one shared TTS path (archetypes' renderer, never a
# second inference code path); if the engine isn't ready it's rendered
# lazily on first preview/use. The row carries vd_states + instruct, so
# the voice is fully usable without the sample (synthesis falls back to
# instruct-only conditioning — see generation.py's design path).
from pathlib import Path
from api.routers.archetypes import _render_archetype_wav
audio_filename = f"{profile_id}.wav"
audio_path = os.path.join(VOICES_DIR, audio_filename)
try:
await _render_archetype_wav(
{
"language": language,
"sample_script": ref_text, # optional custom sample line
"instruct": instruct,
},
Path(audio_path),
)
except Exception:
# Engine unavailable / OOM / inference failure — defer the sample.
# Store the row with no ref_audio_path; the identity sample is
# rendered on first preview or use. Never let this block the save.
import logging
logging.getLogger("omnivoice.profiles").info(
"Design profile %s saved with sample pending — "
"voice engine not ready; will render on first use", profile_id,
)
if os.path.exists(audio_path): # partial/blank render: don't keep it
with __import__("contextlib").suppress(OSError):
os.remove(audio_path)
audio_filename = None
used_seed = seed if seed is not None else _DESIGN_SEED
with open(audio_path, "wb") as f:
f.write(await ref_audio.read())
try:
with db_conn() as conn:
conn.execute(
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, "
"language, seed, personality, kind, vd_states, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
(profile_id, name, audio_filename, ref_text, instruct, language,
used_seed, personality, kind, vd_states, time.time())
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(profile_id, name, audio_filename, ref_text, instruct, language, seed, personality, time.time())
)
except Exception:
# Clean up orphaned audio file if DB insert fails
@@ -139,7 +64,7 @@ async def create_profile(
os.remove(audio_path)
raise
event_bus.emit("profiles", {"action": "created", "id": profile_id})
return {"id": profile_id, "name": name, "kind": kind}
return {"id": profile_id, "name": name}
@router.get("/profiles/{profile_id}")
def get_profile(profile_id: str):
@@ -238,96 +163,20 @@ def get_profile_usage(profile_id: str):
}
# profile_id is a request path param and the audio filename derives from it, so
# constrain it to the generated-id charset (no separators / `..` possible) before
# any path use, and read only a *direct child* of VOICES_DIR — os.path.basename()
# strips any directory component (a path-injection / CWE-22 barrier).
_PROFILE_ID_RE = re.compile(r"[A-Za-z0-9_-]{1,64}")
@router.get("/profiles/{profile_id}/audio")
async def get_profile_audio(profile_id: str):
if not _PROFILE_ID_RE.fullmatch(profile_id or ""):
return Response("Profile not found", status_code=404)
def get_profile_audio(profile_id: str):
with db_conn() as conn:
row = conn.execute(
"SELECT ref_audio_path, locked_audio_path, kind, instruct, language, ref_text "
"FROM voice_profiles WHERE id=?",
(profile_id,),
).fetchone()
row = conn.execute("SELECT ref_audio_path, locked_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
if not row:
return Response("Profile not found", status_code=404)
audio_file = row["locked_audio_path"] or row["ref_audio_path"]
if not audio_file:
# A design profile saved before the engine was ready (issue #476) has no
# identity sample yet. Render it lazily now — the deterministic seed-42
# sample is reproducible, so a deferred render matches a save-time one.
rendered = await _materialize_design_sample(profile_id, row)
if rendered is None:
return Response("No audio available", status_code=404)
audio_file = rendered
# CWE-22: resolve the DB-stored filename strictly inside VOICES_DIR via the
# shared guard — _voices_path() applies the os.path.basename() barrier plus
# symlink-resolved containment (same path the consent endpoint trusts).
audio_path = _voices_path(str(audio_file))
if audio_path is None or not os.path.exists(audio_path):
return Response("No audio available", status_code=404)
audio_path = os.path.join(VOICES_DIR, audio_file)
if not os.path.exists(audio_path):
return Response("Audio file missing", status_code=404)
return FileResponse(audio_path, media_type="audio/wav")
async def _materialize_design_sample(profile_id: str, row) -> Optional[str]:
"""Render a design profile's pending identity sample on first request.
Returns the stored filename on success, or None if this isn't a renderable
design row. Raises HTTPException(503) with a precise "model not ready"
message if the engine is genuinely unavailable saving never depends on
this, but a user who explicitly asks for the sample gets a clear signal.
"""
try:
kind = row["kind"]
except (KeyError, IndexError):
kind = "clone"
if kind != "design":
return None
from pathlib import Path
from api.routers.archetypes import _render_archetype_wav
audio_filename = f"{profile_id}.wav"
# CWE-22: resolve under VOICES_DIR via the shared basename + containment
# guard before rendering (rejects any escape).
audio_path = _voices_path(audio_filename)
if audio_path is None:
raise HTTPException(status_code=400, detail="invalid profile identifier")
try:
await _render_archetype_wav(
{
"language": row["language"] or "Auto",
"sample_script": row["ref_text"] or "",
"instruct": row["instruct"] or "",
},
Path(audio_path),
)
except Exception as e:
with __import__("contextlib").suppress(OSError):
if os.path.exists(audio_path):
os.remove(audio_path)
raise HTTPException(
status_code=503,
detail=(
"The voice engine isn't ready yet, so this designed voice's "
"preview sample can't be rendered. Finish setup / download a "
f"model, then try again. ({e})"
),
)
with db_conn() as conn:
conn.execute(
"UPDATE voice_profiles SET ref_audio_path=? WHERE id=?",
(audio_filename, profile_id),
)
return audio_filename
@router.post("/profiles/{profile_id}/lock")
async def lock_profile(
profile_id: str,
@@ -385,122 +234,15 @@ async def unlock_profile(profile_id: str):
event_bus.emit("profiles", {"action": "unlocked", "id": profile_id})
return {"unlocked": True, "profile_id": profile_id}
# ── Consent lock (parity program Wave 0.2) ─────────────────────────────────
#
# A profile becomes "verified own voice" when its owner records themselves
# reading a consent statement. The recording is provenance, not a voiceprint
# check — agentic features and gallery sharing gate on the flag; plain local
# synthesis never does. Spec: docs/competitive-analysis.md Action 22.
_MIN_CONSENT_AUDIO_BYTES = 1000 # same floor as the frontend recorder
# Upload filename extension whitelist — anything else falls back to .wav so a
# crafted filename can never influence the on-disk path (py/path-injection).
_CONSENT_EXT_RE = re.compile(r"^\.[A-Za-z0-9]{1,8}$")
def _voices_path(filename: str) -> Optional[str]:
"""Resolve a DB-stored audio filename strictly inside VOICES_DIR.
Rejects anything that isn't a bare filename or that escapes the voices
directory after symlink resolution. Returns None instead of raising so
cleanup paths can simply skip bad values.
"""
if not filename or os.path.basename(filename) != filename:
return None
root = os.path.realpath(VOICES_DIR)
path = os.path.realpath(os.path.join(root, filename))
if not path.startswith(root + os.sep):
return None
return path
@router.post("/profiles/{profile_id}/consent")
async def record_consent(
profile_id: str,
consent_audio: UploadFile = File(...),
consent_text: str = Form(...),
):
if not consent_text.strip():
raise HTTPException(status_code=422, detail="consent_text must not be empty")
data = await consent_audio.read()
if len(data) < _MIN_CONSENT_AUDIO_BYTES:
raise HTTPException(status_code=422, detail="consent recording is too short")
with db_conn() as conn:
row = conn.execute(
"SELECT id, consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)
).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Profile not found")
ext = os.path.splitext(consent_audio.filename or "")[1]
if not _CONSENT_EXT_RE.match(ext):
ext = ".wav"
audio_filename = f"{profile_id}_consent{ext}"
audio_path = _voices_path(audio_filename)
if audio_path is None: # profile_id is server-generated; this is belt+braces
raise HTTPException(status_code=400, detail="Invalid profile id")
with open(audio_path, "wb") as f:
f.write(data)
# A re-record may change the extension; drop the superseded file.
old = row["consent_audio_path"]
if old and old != audio_filename:
old_path = _voices_path(old)
if old_path and os.path.exists(old_path):
os.remove(old_path)
recorded_at = time.time()
try:
with db_conn() as conn:
conn.execute(
"UPDATE voice_profiles SET verified_own_voice=1, consent_text=?, "
"consent_audio_path=?, consent_recorded_at=? WHERE id=?",
(consent_text.strip(), audio_filename, recorded_at, profile_id),
)
except Exception:
if os.path.exists(audio_path):
os.remove(audio_path)
raise
event_bus.emit("profiles", {"action": "consent_recorded", "id": profile_id})
return {
"id": profile_id,
"verified_own_voice": True,
"consent_recorded_at": recorded_at,
}
@router.delete("/profiles/{profile_id}/consent")
def revoke_consent(profile_id: str):
with db_conn() as conn:
row = conn.execute(
"SELECT consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)
).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Profile not found")
conn.execute(
"UPDATE voice_profiles SET verified_own_voice=0, consent_text='', "
"consent_audio_path='', consent_recorded_at=NULL WHERE id=?",
(profile_id,),
)
if row["consent_audio_path"]:
path = _voices_path(row["consent_audio_path"])
if path and os.path.exists(path):
os.remove(path)
event_bus.emit("profiles", {"action": "consent_revoked", "id": profile_id})
return {"id": profile_id, "verified_own_voice": False}
@router.delete("/profiles/{profile_id}")
def delete_profile(profile_id: str):
with db_conn() as conn:
row = conn.execute("SELECT ref_audio_path, locked_audio_path, consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
row = conn.execute("SELECT ref_audio_path, locked_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
if row:
for col in ["ref_audio_path", "locked_audio_path", "consent_audio_path"]:
for col in ["ref_audio_path", "locked_audio_path"]:
if row[col]:
path = _voices_path(row[col])
if path and os.path.exists(path):
path = os.path.join(VOICES_DIR, row[col])
if os.path.exists(path):
os.remove(path)
# Prevent FOREIGN KEY constraint failure
conn.execute("UPDATE generation_history SET profile_id = NULL WHERE profile_id=?", (profile_id,))
-25
View File
@@ -2,7 +2,6 @@ import uuid
import time
import json
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
from core.db import db_conn
from core import event_bus
@@ -10,10 +9,6 @@ from schemas.requests import ProjectSaveRequest
router = APIRouter()
class ProjectRenameRequest(BaseModel):
name: str
@router.get("/projects")
async def list_projects():
with db_conn() as conn:
@@ -64,26 +59,6 @@ async def update_project(project_id: str, req: ProjectSaveRequest):
event_bus.emit("projects", {"action": "updated", "id": project_id})
return {"id": project_id, "name": req.name, "updated_at": now}
@router.patch("/projects/{project_id}")
async def rename_project(project_id: str, req: ProjectRenameRequest):
"""Lightweight rename — updates only the project name (and updated_at),
without re-serialising the whole state blob like PUT does."""
name = req.name.strip()
if not name:
raise HTTPException(status_code=400, detail="Project name cannot be empty")
now = time.time()
with db_conn() as conn:
row = conn.execute("SELECT id FROM studio_projects WHERE id=?", (project_id,)).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Project not found")
conn.execute(
"UPDATE studio_projects SET name=?, updated_at=? WHERE id=?",
(name, now, project_id),
)
event_bus.emit("projects", {"action": "renamed", "id": project_id})
return {"id": project_id, "name": name, "updated_at": now}
@router.delete("/projects/{project_id}")
async def delete_project(project_id: str):
with db_conn() as conn:
-168
View File
@@ -123,117 +123,6 @@ def set_torch_compile_disabled(body: _TorchCompileBody):
return _torch_compile_state()
# ── Dictation refinement (parity program Wave 2.1 / Spec 3 phase 2) ───────
class _RefinementBody(BaseModel):
auto: bool | None = None
smart_cleanup: bool | None = None
self_correction: bool | None = None
preserve_technical: bool | None = None
def _refinement_state():
from services.refinement import get_refinement_config
from services.llm_backend import get_active_llm_backend
cfg = get_refinement_config()
# The UI shows whether refinement can actually run (needs an LLM).
cfg["llm_ready"] = get_active_llm_backend().id != "off"
return cfg
@router.get("/dictation-refinement")
def get_dictation_refinement():
"""Current refinement config + whether an LLM backend is configured."""
return _refinement_state()
@router.put("/dictation-refinement")
def set_dictation_refinement(body: _RefinementBody):
from services.refinement import set_refinement_config
try:
set_refinement_config({k: v for k, v in body.model_dump().items() if v is not None})
except Exception:
logger.exception("set_dictation_refinement failed")
raise HTTPException(status_code=500, detail="Failed to persist setting")
return _refinement_state()
# ── LLM endpoint (parity program Wave 2.4 / §R2 rung 4) ───────────────────
# Focused configuration for the OpenAI-compatible LLM endpoint that powers
# cinematic translate, glossary auto-extract, and dictation refinement.
# Persistence rides the existing TRANSLATE_BASE_URL / TRANSLATE_API_KEY /
# TRANSLATE_MODEL env vars (already in system.py PERSISTENT_KEYS, restored
# at startup) so the resolution path in llm_backend/translator is unchanged.
class _LLMEndpointBody(BaseModel):
base_url: str | None = None
model: str | None = None
api_key: str | None = None # None = leave unchanged; "" = clear
def _mask(secret: str | None) -> str | None:
if not secret:
return None
return f"{secret[-4:]}" if len(secret) > 4 else "set"
def _llm_endpoint_state():
from services.llm_backend import OpenAICompatBackend
ok, reason = OpenAICompatBackend.is_available()
return {
"base_url": os.environ.get("TRANSLATE_BASE_URL", ""),
"model": os.environ.get("TRANSLATE_MODEL", ""),
"api_key_masked": _mask(
os.environ.get("TRANSLATE_API_KEY") or os.environ.get("OPENAI_API_KEY")
),
"available": ok,
"reason": None if ok else reason,
}
@router.get("/llm-endpoint")
def get_llm_endpoint():
"""Current OpenAI-compatible LLM endpoint config + live availability."""
return _llm_endpoint_state()
@router.put("/llm-endpoint")
def set_llm_endpoint(body: _LLMEndpointBody):
"""Persist base URL / model / API key for the OpenAI-compatible endpoint.
Reuses the env-var persistence path (prefs.json, restored at startup):
base_url -> TRANSLATE_BASE_URL, model -> TRANSLATE_MODEL,
api_key -> TRANSLATE_API_KEY. A None field is left unchanged; an empty
string clears it. Ollama ignores the key; vLLM / LM Studio require it.
"""
from core.prefs import set_ as prefs_set, delete as prefs_delete
mapping = {
"TRANSLATE_BASE_URL": body.base_url,
"TRANSLATE_MODEL": body.model,
"TRANSLATE_API_KEY": body.api_key,
}
for env_key, val in mapping.items():
if val is None:
continue # untouched
val = val.strip()
if val:
os.environ[env_key] = val
prefs_set(f"env.{env_key}", val)
else:
os.environ.pop(env_key, None)
prefs_delete(f"env.{env_key}")
# get_active_llm_backend() builds a fresh backend (and its OpenAI client
# reads env at construction) on every call, so there's no singleton to
# invalidate — the next translate/refine picks up the new values.
return _llm_endpoint_state()
# ── License acceptance (Phase 3 Plan 03-01 / TTS-05) ──────────────────────
# Frontend ``SupertonicLicenseDialog`` flips the engine-license bit via this
# endpoint. The handler is loopback-gated (router-level dep) and the
@@ -395,60 +284,3 @@ def set_models_dir(body: _ModelsDirBody):
user_env.set_user_env(_MODELS_DIR_ENV, path)
return {"configured": path, "effective": _effective_models_dir(), "restart_required": True}
# ── HF mirror endpoint (parity program Wave 4.3 / §R4 c) ──────────────────
# Restricted-network users (e.g. behind the Great Firewall) need to point
# huggingface_hub at a mirror. HF reads HF_ENDPOINT at import time, so a
# change takes effect on the next backend start — persisted to the durable
# per-user env so it survives Tauri/Finder launches that don't inherit a
# shell. Loopback-gated via the router dep.
_HF_ENDPOINT_ENV = "HF_ENDPOINT"
# A few well-known mirrors, surfaced as quick-picks in the UI. hf-mirror.com
# is the community mirror most-used in China; the official endpoint clears it.
_HF_MIRROR_PRESETS = [
{"label": "Hugging Face (official)", "url": ""},
{"label": "hf-mirror.com (community, China)", "url": "https://hf-mirror.com"},
]
class _HFMirrorBody(BaseModel):
url: str = Field("", description="HF_ENDPOINT URL; empty string clears it (official endpoint)")
@router.get("/hf-mirror")
def get_hf_mirror():
from core import user_env
configured = user_env.get_user_env(_HF_ENDPOINT_ENV) or ""
return {
# The value that will apply after restart (persisted), and what's
# live in this process (env may differ until then).
"configured": configured,
"effective": os.environ.get(_HF_ENDPOINT_ENV, ""),
"presets": _HF_MIRROR_PRESETS,
}
@router.put("/hf-mirror")
def set_hf_mirror(body: _HFMirrorBody):
from core import user_env
url = (body.url or "").strip().rstrip("/")
if url and not url.startswith(("http://", "https://")):
raise HTTPException(status_code=400, detail="Mirror URL must start with http(s)://")
try:
if url:
user_env.set_user_env(_HF_ENDPOINT_ENV, url)
os.environ[_HF_ENDPOINT_ENV] = url # best-effort for new downloads this session
else:
user_env.unset_user_env(_HF_ENDPOINT_ENV)
os.environ.pop(_HF_ENDPOINT_ENV, None)
except Exception:
logger.exception("set_hf_mirror failed")
raise HTTPException(status_code=500, detail="Failed to persist mirror setting")
# HF endpoint is read at import time by huggingface_hub, so the override
# is only guaranteed once the backend restarts.
return {"configured": url, "restart_required": True, "presets": _HF_MIRROR_PRESETS}
+2 -343
View File
@@ -11,16 +11,13 @@ from __future__ import annotations
import asyncio
import json
import logging
import os
import sys
from fastapi import APIRouter, HTTPException
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from core import prefs
from utils import hf_progress
from utils import download_aggregator
from .models import KNOWN_MODELS, invalidate_cache
logger = logging.getLogger("omnivoice.setup.download")
@@ -29,177 +26,6 @@ router = APIRouter()
# Cooldown: prevent rapid re-install after a failure. Maps repo_id → last_fail_time.
_install_cooldowns: dict[str, float] = {}
_COOLDOWN_SECS = 60.0
# Evict cooldown entries older than this so the dict can't grow unbounded across
# a long-lived process (MM2-06). Anything past the cooldown window is dead state.
_COOLDOWN_TTL_SECS = 3600.0
def _sweep_cooldowns(now: float) -> None:
"""Drop cooldown entries older than the TTL (MM2-06). Keeps the dict bounded
without this it accumulated one entry per ever-failed repo forever."""
stale = [k for k, t in _install_cooldowns.items() if (now - t) > _COOLDOWN_TTL_SECS]
for k in stale:
_install_cooldowns.pop(k, None)
# Repo_ids the user asked to cancel (FDL-11). Checked between retry attempts.
# Note: a single in-flight snapshot_download/Xet fetch is not interruptible
# mid-file in hf_hub 1.7.2 — cancel stops further retries, marks the row
# cancelled, and clears the cooldown so a cancel isn't rate-limited.
_cancelled: set[str] = set()
def _download_max_workers() -> int:
"""Parallel-FILES worker count for snapshot_download (FDL-02). Default 8 —
don't crank it: Xet already parallelises *within* each file via concurrent
byte-range gets, so a high count just multiplies buffer pressure. Override
via prefs / OMNIVOICE_DOWNLOAD_MAX_WORKERS for power users."""
raw = prefs.resolve("download_max_workers", env="OMNIVOICE_DOWNLOAD_MAX_WORKERS", default=8)
try:
return max(1, int(raw))
except (TypeError, ValueError):
return 8
def _download_endpoint() -> "str | None":
"""Optional HF endpoint override (FDL-10 mirror path, opt-in). Returned as a
per-call ``endpoint=`` rather than a process-wide HF_ENDPOINT mutation. A
mirror routes through the classic LFS path (no Xet) documented in
docs/downloading-models.md."""
ep = prefs.resolve("hf_endpoint", env="HF_ENDPOINT", default=None)
return ep or None
def apply_xet_env() -> None:
"""Apply opt-in Xet tuning knobs to the environment before a download
(FDL-04). Both default OFF; env wins over the prefs store. high-performance
can *hurt* low-RAM machines (needs lots of RAM/bandwidth); HDD-sequential
avoids parallel-write thrash on spinning disks. Idempotent."""
import os as _os
high_perf = prefs.resolve("xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False)
if _truthy(high_perf):
_os.environ["HF_XET_HIGH_PERFORMANCE"] = "1"
hdd_seq = prefs.resolve("xet_hdd_sequential_write", env="HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY", default=False)
if _truthy(hdd_seq):
_os.environ["HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY"] = "1"
def _truthy(v) -> bool:
if isinstance(v, bool):
return v
return str(v).strip().lower() in {"1", "true", "yes", "on"}
class _InstallCancelled(Exception):
"""Raised inside the install worker when the user cancels (FDL-11)."""
def compute_plan(plan_files) -> dict:
"""Summarise a snapshot_download(dry_run=True) result into the install_plan
payload (FDL-05): total bytes, bytes already cached (skipped), bytes that
will actually download, and file counts. ``will_download`` defaults to
``not is_cached`` for forward-compat with older DryRunFileInfo shapes."""
total = sum(int(getattr(f, "file_size", 0) or 0) for f in plan_files)
cached = sum(
int(getattr(f, "file_size", 0) or 0)
for f in plan_files if getattr(f, "is_cached", False)
)
will = [
f for f in plan_files
if getattr(f, "will_download", not getattr(f, "is_cached", False))
]
to_dl = sum(int(getattr(f, "file_size", 0) or 0) for f in will)
n_files = len(plan_files)
n_cached = sum(1 for f in plan_files if getattr(f, "is_cached", False))
return {
"total_bytes": total,
"cached_bytes": cached,
"to_download_bytes": to_dl,
"n_files": n_files,
"n_cached": n_cached,
}
def _segmented_enabled() -> bool:
"""Opt-in IDM-style accelerator (FDL-09), default OFF. Most useful when Xet
is inactive (the app's default): the legacy-LFS path is single-stream, so
this restores parallel speed AND gives real live byte progress."""
return _truthy(prefs.resolve(
"segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=False,
))
def _xet_active() -> bool:
"""True only when hf_xet is installed AND not disabled. The app sets
HF_HUB_DISABLE_XET=1 by default, so this is normally False which is when
the segmented accelerator pays off."""
import importlib.util
if importlib.util.find_spec("hf_xet") is None:
return False
return os.environ.get("HF_HUB_DISABLE_XET", "").strip().lower() not in {"1", "true", "yes", "on"}
def _repo_cancelled(repo_id: str) -> bool:
return repo_id in _cancelled
def _segmented_snapshot(repo_id: str, *, endpoint: "str | None") -> str:
"""Fetch every file of a repo via the segmented downloader into the HF
cache, mirroring hf_hub_download's blob+snapshot+refs layout so the result
is indistinguishable from snapshot_download (FDL-09) keeping /models
install-state, is_cached, and delete working. Feeds real bytes to the
aggregator. Raises on any error; the caller falls back to snapshot_download.
"""
import asyncio as _asyncio
from huggingface_hub import HfApi, constants as _C
from huggingface_hub.file_download import (
hf_hub_url, get_hf_file_metadata, repo_folder_name, _create_symlink,
)
from services.segmented_download import segmented_download
from services.token_resolver import resolve as _resolve_token
token = _resolve_token()
api = HfApi(endpoint=endpoint, token=token)
info = api.repo_info(repo_id, repo_type="model")
commit = info.sha
files = [s.rfilename for s in (info.siblings or [])]
if not commit or not files:
raise RuntimeError("repo_info returned no commit/siblings")
repo_dir = os.path.join(_C.HF_HUB_CACHE, repo_folder_name(repo_id=repo_id, repo_type="model"))
blobs_dir = os.path.join(repo_dir, "blobs")
snap_dir = os.path.join(repo_dir, "snapshots", commit)
refs_dir = os.path.join(repo_dir, "refs")
for d in (blobs_dir, snap_dir, refs_dir):
os.makedirs(d, exist_ok=True)
for rel in files:
if _repo_cancelled(repo_id):
raise _InstallCancelled()
url = hf_hub_url(repo_id, rel, endpoint=endpoint, revision=commit)
meta = get_hf_file_metadata(url, token=token)
etag = (meta.etag or "").strip('"')
if not etag:
raise RuntimeError(f"no etag for {rel}")
blob_path = os.path.join(blobs_dir, etag)
pointer = os.path.join(snap_dir, rel)
os.makedirs(os.path.dirname(pointer), exist_ok=True)
if not os.path.exists(blob_path):
_asyncio.run(segmented_download(
meta.location or url, blob_path,
token=token, expected_size=meta.size, expected_etag=etag,
on_bytes=lambda d, k=rel: download_aggregator.add_bytes(repo_id, k, d),
cancel_check=lambda: _repo_cancelled(repo_id),
))
if not os.path.lexists(pointer):
_create_symlink(blob_path, pointer, new_blob=True)
# refs/main → commit so scan_cache_dir maps the revision correctly.
try:
with open(os.path.join(refs_dir, "main"), "w") as f:
f.write(commit)
except OSError:
pass
return snap_dir
# ── SSE Download Stream ───────────────────────────────────────────────────
@@ -216,66 +42,6 @@ def _safe_put(queue: asyncio.Queue, event) -> None:
pass
# Minimum size for "this snapshot actually contains model weights". An
# interrupted snapshot_download can leave config/tokenizer files but no
# weights; the install then looks complete and synthesis later fails with
# "does not appear to have a file named pytorch_model.bin or
# model.safetensors" (#352). 5 MB clears every weight format we ship
# (safetensors/bin shards, onnx, pt, gguf) without false-positiving on
# config-only aux repos.
_MIN_WEIGHT_BYTES = 5 * 1024 * 1024
# Per-role weight-file floors (MM2-07). A valid model has at least one
# recognized weight file at or above its extension's floor. ONNX graphs are
# legitimately small (a complete model can be well under 5 MB), so a single
# 5 MB rule false-positives on them as "truncated" (#352 over-trigger); give
# .onnx a lower floor while still rejecting a 0/KB partial. Tensor formats keep
# the original 5 MB floor.
_WEIGHT_FLOORS = {
".safetensors": _MIN_WEIGHT_BYTES,
".bin": _MIN_WEIGHT_BYTES,
".ckpt": _MIN_WEIGHT_BYTES,
".pt": _MIN_WEIGHT_BYTES,
".pth": _MIN_WEIGHT_BYTES,
".gguf": _MIN_WEIGHT_BYTES,
".onnx": 64 * 1024, # a real ONNX graph is ≥ tens of KB; a truncated one is bytes
}
def _validate_snapshot_has_weights(repo_id: str, snapshot_path: str) -> None:
"""Raise OSError when a finished snapshot has no plausible weight file —
surfaces the truncated-download class (#352) at install time, where the
retry loop and the UI's re-download path can deal with it, instead of at
first synthesis with an opaque transformers error.
A snapshot is valid if it contains a recognized weight file meeting its
per-extension floor (MM2-07) OR any file the global 5 MB floor (the
original lenient catch kept so this is never stricter than before)."""
try:
biggest = 0
for root, _dirs, files in os.walk(snapshot_path, followlinks=True):
for f in files:
try:
size = os.path.getsize(os.path.join(root, f))
except OSError:
continue
biggest = max(biggest, size)
ext = os.path.splitext(f)[1].lower()
floor = _WEIGHT_FLOORS.get(ext)
if floor is not None and size >= floor:
return # a recognized weight file of plausible size
if size >= _MIN_WEIGHT_BYTES:
return # original lenient catch (non-standard weight names)
except OSError:
return # can't inspect — don't block the install on the checker itself
raise OSError(
f"{repo_id}: download finished but no model weights were found in the "
"snapshot (largest file "
f"{biggest} bytes). The download was likely interrupted — delete the "
"model in Settings → Models and install it again."
)
@router.get("/setup/download-stream")
async def setup_download_stream():
"""SSE: forward every HuggingFace download tqdm update as a JSON event."""
@@ -332,7 +98,6 @@ async def install_model(req: InstallModelRequest):
)
# Cooldown guard — don't retry if the same model just failed.
import time as _time_check
_sweep_cooldowns(_time_check.time()) # bound the dict (MM2-06)
last_fail = _install_cooldowns.get(req.repo_id)
if last_fail and (_time_check.time() - last_fail) < _COOLDOWN_SECS:
remaining = int(_COOLDOWN_SECS - (_time_check.time() - last_fail))
@@ -347,7 +112,6 @@ async def install_model(req: InstallModelRequest):
def _do():
token = hf_progress.current_repo_id.set(req.repo_id)
_cancelled.discard(req.repo_id) # clear any stale cancel from a prior run
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
@@ -361,22 +125,7 @@ async def install_model(req: InstallModelRequest):
LocalEntryNotFoundError,
)
logger.info("model install starting: %s", req.repo_id)
# Apply opt-in Xet tuning knobs (high-perf / HDD) before downloading.
apply_xet_env()
# Drive snapshot_download explicitly (FDL-02): pass our progress-
# emitting tqdm subclass so progress is deterministic + Xet-aware
# (Xet feeds bytes into whatever tqdm_class is supplied), bound the
# parallel-files worker count, and honour an optional mirror endpoint.
dl_kwargs: dict = {
"repo_id": req.repo_id,
"max_workers": _download_max_workers(),
}
_tqdm_cls = hf_progress.tracked_tqdm_class()
if _tqdm_cls is not None:
dl_kwargs["tqdm_class"] = _tqdm_cls
_endpoint = _download_endpoint()
if _endpoint:
dl_kwargs["endpoint"] = _endpoint
dl_kwargs: dict = {"repo_id": req.repo_id}
if sys.platform == "win32":
dl_kwargs["local_dir_use_symlinks"] = False
@@ -404,70 +153,12 @@ async def install_model(req: InstallModelRequest):
hb = threading.Thread(target=_heartbeat, daemon=True)
hb.start()
# Pre-flight (FDL-05): a dry-run resolve gives the UI an accurate
# denominator — total bytes, bytes already cached (skipped), and the
# bytes that will actually download — BEFORE any byte flows. Seeds
# the overall aggregator so its bar/ETA are correct from the first
# event. Degrades gracefully (totals=None) on older/gated repos.
_preflight_kwargs = {"repo_id": req.repo_id, "dry_run": True}
if _endpoint:
_preflight_kwargs["endpoint"] = _endpoint
try:
_plan = snapshot_download(**_preflight_kwargs)
_summary = compute_plan(_plan)
download_aggregator.start(
req.repo_id,
total_bytes=_summary["to_download_bytes"],
files_total=max(0, _summary["n_files"] - _summary["n_cached"]),
)
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"phase": "install_plan",
**_summary,
})
except Exception as _pf_err:
# No preflight (older/gated repo, mirror without dry-run, etc.):
# fall back to today's fill-in-as-files-appear behaviour.
logger.info("model install %s: preflight unavailable (%s)", req.repo_id, _pf_err)
download_aggregator.start(req.repo_id)
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"phase": "install_plan",
"total_bytes": None,
"cached_bytes": None,
"to_download_bytes": None,
"n_files": None,
"n_cached": None,
})
_max_attempts = 5
_attempt = 0
while True:
if req.repo_id in _cancelled:
raise _InstallCancelled()
_attempt += 1
try:
# Opt-in segmented accelerator (FDL-09): parallel byte-range
# fetch with real live progress, for the legacy-LFS path.
# Any failure falls through to snapshot_download — the
# accelerator can never compromise a correct install.
_snapshot_path = None
if _attempt == 1 and _segmented_enabled() and not _xet_active():
try:
_snapshot_path = _segmented_snapshot(req.repo_id, endpoint=_endpoint)
except _InstallCancelled:
raise
except Exception as _seg_err:
logger.info(
"segmented download for %s failed (%s); falling back to snapshot_download",
req.repo_id, _seg_err,
)
_snapshot_path = None
if _snapshot_path is None:
_snapshot_path = snapshot_download(**dl_kwargs)
_validate_snapshot_has_weights(req.repo_id, _snapshot_path)
snapshot_download(**dl_kwargs)
break
except (HfHubHTTPError, LocalEntryNotFoundError, OSError) as net_err:
if _attempt >= _max_attempts:
@@ -488,10 +179,6 @@ async def install_model(req: InstallModelRequest):
_t.sleep(_backoff)
# Stop heartbeat once download completes
_resolving.set()
# Flush the overall bar to 100% with the true byte total (FDL-06):
# under Xet the per-file byte bars don't surface completion, so the
# aggregator can sit below 100% even though every file landed.
download_aggregator.complete(req.repo_id)
logger.info("model install done: %s", req.repo_id)
hf_progress.emit({
"repo_id": req.repo_id,
@@ -499,19 +186,7 @@ async def install_model(req: InstallModelRequest):
"downloaded": 0, "total": 0, "pct": 1.0,
"phase": "install_done",
})
_install_cooldowns.pop(req.repo_id, None) # success clears any cooldown (MM2-06)
invalidate_cache()
except _InstallCancelled:
_resolving.set()
logger.info("model install cancelled: %s", req.repo_id)
# A cancel is user intent, not a failure — don't set a cooldown.
_install_cooldowns.pop(req.repo_id, None)
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"downloaded": 0, "total": 0, "pct": 0.0,
"phase": "install_cancelled",
})
except Exception as e:
_resolving.set()
logger.info("model install failed for %s: %s", req.repo_id, e)
@@ -525,28 +200,12 @@ async def install_model(req: InstallModelRequest):
"error": str(e),
})
finally:
_cancelled.discard(req.repo_id)
download_aggregator.finish(req.repo_id)
hf_progress.current_repo_id.reset(token)
loop.create_task(asyncio.to_thread(_do))
return {"status": "install_started", "repo_id": req.repo_id}
@router.post("/models/install/cancel")
async def cancel_install(req: InstallModelRequest):
"""Request cancellation of an in-flight install (FDL-11).
Best-effort: stops further retry attempts and marks the row cancelled. A
single in-flight snapshot_download/Xet fetch isn't interruptible mid-file
in hf_hub 1.7.2, so an already-streaming file finishes; the cancel takes
effect at the next retry boundary. Clears the cooldown so the user can
immediately restart."""
_cancelled.add(req.repo_id)
_install_cooldowns.pop(req.repo_id, None)
return {"cancelling": req.repo_id}
# ── Delete ─────────────────────────────────────────────────────────────────
@router.delete("/models/{repo_id:path}")
+2 -6
View File
@@ -220,12 +220,8 @@ def is_cached(repo_id: str) -> bool:
except Exception as e:
# scan_cache_dir can raise on Windows (WinError 448 'untrusted mount
# point'); fall back to a direct disk check so a cached model isn't
# mistaken for missing and re-downloaded in a loop (#117/#118). Logged
# at WARNING with the exception type (MM2-09) so this fallback isn't
# invisible when triaging a Windows cache report — it previously logged
# at DEBUG and never showed at the default level.
logger.warning("is_cached: scan_cache_dir failed (%s: %s); using on-disk fallback for %s",
type(e).__name__, e, repo_id)
# mistaken for missing and re-downloaded in a loop (#117/#118).
logger.debug("scan_cache_dir failed (%s); using disk fallback", e)
return _is_cached_on_disk(repo_id)
-44
View File
@@ -373,46 +373,6 @@ def preflight():
"status": gpu_status, "detail": gpu_detail, "fix": gpu_fix,
})
# ── GPU routing for the ACTIVE TTS engine (#21 — no silent CPU fallback).
# Distinct from the hardware "gpu" check above: this asks "will the engine
# the user actually selected use that GPU on this host?" Built from the same
# canonical probe + resolver the Engine Compatibility Matrix uses.
try:
from services.tts_backend import gpu_routing_verdict
gpu_routing = gpu_routing_verdict()
except Exception as exc: # never break preflight on a routing hiccup
logger.warning("preflight gpu_routing failed: %s", exc)
gpu_routing = None
if gpu_routing:
_rs = gpu_routing.get("routing_status")
_eng = gpu_routing.get("engine") or "active engine"
_dev = gpu_routing.get("effective_device") or "?"
_why = gpu_routing.get("routing_reason")
if _rs == "accelerated" and not _why:
r_status, r_detail, r_fix = "pass", f"{_eng}{_dev} (accelerated)", None
elif _rs == "accelerated": # driver/arch caveat
r_status, r_detail, r_fix = "warn", f"{_eng}{_dev}: {_why}", (
"GPU selected but may fail at kernel launch — update drivers / "
"reinstall torch for this GPU architecture.")
elif _rs == "cpu_fallback":
r_status, r_detail, r_fix = "warn", (
f"{_eng} runs on CPU here: {_why or 'no GPU path for this host'}"), (
"Pick an engine that supports this host's GPU for a speedup, or "
"continue on CPU (slower).")
elif _rs == "cpu_only":
r_status, r_detail, r_fix = "pass", f"{_eng} → cpu (no accelerator on this host)", None
elif _rs == "unavailable":
r_status, r_detail, r_fix = "fail", (
f"{_eng} can't run on this host: {_why or 'needs a GPU this machine lacks'}"), (
"Select an engine with a CPU path in Settings → Engines.")
else: # "none" / unknown
r_status, r_detail, r_fix = "warn", "No active TTS engine resolved for routing.", (
"Pick an engine in Settings → Engines.")
checks.append({
"id": "gpu_routing", "label": "Active engine routing",
"status": r_status, "detail": r_detail, "fix": r_fix,
})
# ── Network
net_ok = _probe_network()
checks.append({
@@ -440,13 +400,9 @@ def preflight():
"gpu_available": gpu["available"],
"gpu_driver": gpu["driver"],
"gpu_device_name": gpu["device_name"],
# Canonical probe (distinguishes ROCm from CUDA):
"gpu_family": (gpu_routing or {}).get("host_family", "cpu"),
"vram_gb": (gpu_routing or {}).get("vram_gb", 0.0),
"ram_gb": round(ram, 1),
"disk_free_gb": round(free, 1),
},
"gpu_routing": gpu_routing,
}
+85 -120
View File
@@ -107,52 +107,6 @@ def _ui_port() -> int:
return 3901
def _fast_download_status() -> dict:
"""Report the download-acceleration state for the Settings UI (FDL-03).
Reports the *runtime* truth, not just whether hf_xet is importable. The app
currently sets ``HF_HUB_DISABLE_XET=1`` by default (main.py) Xet's chunked
transfer is fast but its progress bypasses our tqdm patch, so the legacy-LFS
path is forced to keep accurate byte progress. So:
* ``xet_installed`` hf_xet present
* ``xet_active`` installed AND not disabled via HF_HUB_DISABLE_XET
* ``xet_enabled`` alias of xet_active (what the UI badge keys off)
Must never throw: /system/info is called on every Settings load.
"""
installed = False
version = None
try:
import hf_xet # noqa: F401
installed = True
try:
from importlib.metadata import version as _ver
version = _ver("hf-xet")
except Exception:
version = None
except Exception:
installed = False
disabled = str(os.environ.get("HF_HUB_DISABLE_XET", "")).strip().lower() in {"1", "true", "yes", "on"}
active = installed and not disabled
try:
from core import prefs
high_perf = prefs.resolve(
"xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False
)
high_perf = high_perf if isinstance(high_perf, bool) else \
str(high_perf).strip().lower() in {"1", "true", "yes", "on"}
except Exception:
high_perf = False
return {
"xet_installed": installed,
"xet_active": active,
"xet_enabled": active, # UI badge: only true when Xet actually runs
"xet_version": version,
"high_performance": bool(high_perf),
}
def _has_hf_token() -> bool:
# Phase 1 AUTH-01..06 cascade. Delegates to the 3-source resolver
# (App → Env → HF-CLI) instead of reading env/HF-CLI directly. This
@@ -174,23 +128,88 @@ def model_status():
@router.get("/model/loaded")
def loaded_models():
"""List all currently loaded models for the flush dropdown (MM2-04).
Thin delegation to the model_lifecycle facade shape unchanged:
``{models, count}``."""
from services import model_lifecycle
return model_lifecycle.list_loaded()
"""Return details about all currently loaded models for the flush dropdown.
Returns a list of models with name, type, device, and estimated VRAM usage.
"""
import services.model_manager as mm
models = []
# 1. TTS model (OmniVoice)
if mm.model is not None:
device = "unknown"
vram_mb = 0
try:
device = str(next(mm.model.parameters()).device) if hasattr(mm.model, 'parameters') else get_best_device()
except Exception:
device = get_best_device()
try:
torch = mm._lazy_torch()
if torch.cuda.is_available():
vram_mb = torch.cuda.memory_allocated() / (1024 ** 2)
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
driver = getattr(torch.mps, "driver_allocated_memory", None)
if driver:
vram_mb = driver() / (1024 ** 2)
except Exception:
pass
models.append({
"id": "tts",
"name": "OmniVoice TTS",
"checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
"device": device,
"vram_mb": round(vram_mb, 1),
"unloadable": True,
})
# 2. ASR model (WhisperX)
if mm.model is not None and hasattr(mm.model, '_asr_pipe') and mm.model._asr_pipe is not None:
models.append({
"id": "asr",
"name": "WhisperX ASR",
"checkpoint": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
"device": "cpu",
"vram_mb": 0,
"unloadable": False, # tied to TTS model lifecycle
})
# 3. Diarization pipeline
if mm._diar_pipeline is not None:
models.append({
"id": "diarization",
"name": "Pyannote Diarization",
"checkpoint": "pyannote/speaker-diarization-3.1",
"device": get_best_device(),
"vram_mb": 0,
"unloadable": True,
})
return {"models": models, "count": len(models)}
@router.post("/model/unload/{model_id}")
async def unload_model(model_id: str):
"""Unload a specific model by id (MM2-04). Delegates to model_lifecycle;
an unknown id maps to HTTP 400. ``tts`` | ``diarization`` |
``sidecar:<id>`` | ``sidecars``."""
from services import model_lifecycle
try:
return await model_lifecycle.unload(model_id)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
"""Unload a specific model by ID."""
import services.model_manager as mm
if model_id == "tts":
async with mm._model_lock:
if mm.model is not None:
mm.model = None
mm.free_vram()
return {"unloaded": "tts", "success": True}
return {"unloaded": "tts", "success": False, "reason": "not loaded"}
elif model_id == "diarization":
if mm._diar_pipeline is not None:
mm._diar_pipeline = None
mm.free_vram()
return {"unloaded": "diarization", "success": True}
return {"unloaded": "diarization", "success": False, "reason": "not loaded"}
else:
raise HTTPException(status_code=400, detail=f"Unknown model id: {model_id}")
@router.get("/system/info", response_model=SystemInfoResponse)
@@ -212,7 +231,6 @@ def system_info():
"asr_model": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
"translate_provider": os.environ.get("TRANSLATE_PROVIDER", "google"),
"has_hf_token": _has_hf_token(),
"fast_download": _fast_download_status(),
"device": get_best_device(),
"python": sys.version.split()[0],
"platform": sys.platform,
@@ -304,10 +322,9 @@ def _tauri_log_candidates():
os.path.join(home, "Library/Logs/OmniVoice/backend_err.log"),
]
if sys.platform.startswith("linux"):
data_dir = os.environ.get("XDG_DATA_HOME") or os.path.join(home, ".local/share")
state_dir = os.environ.get("XDG_STATE_HOME") or os.path.join(home, ".local/state")
return [
os.path.join(data_dir, bid, "logs", "tauri.log"),
os.path.join(home, ".local/share", bid, "logs", "tauri.log"),
os.path.join(home, ".config", bid, "logs", "tauri.log"),
os.path.join(state_dir, "OmniVoice", "backend.log"),
os.path.join(state_dir, "OmniVoice", "backend_err.log"),
@@ -447,14 +464,6 @@ async def clear_system_logs():
status_code=500,
detail=f"Could not clear log at {p}: {e}. The file may be open in another process or read-only — close tailing tools and retry.",
)
if cleared_any:
# The crash log just shrank to zero — drop any stored ack so a stale
# byte count can't suppress the next 'crash-last-session' notice.
for key in ("crash_log_acked", "crash_log_acked_size"):
try:
prefs_delete(key)
except Exception:
pass
return {"cleared": cleared_any}
@@ -556,21 +565,6 @@ async def flush_memory(unload_model: bool = False):
# ── Actionable notifications ──────────────────────────────────────────────
_GPU_ARCH_WARNING: "list[str | None]" = [] # [-1] = computed result
def _gpu_arch_warning_cached() -> "str | None":
"""check_device_compatibility() once per process (it lazy-imports torch —
too heavy for the 30s notifications poll)."""
if not _GPU_ARCH_WARNING:
try:
from services.model_manager import check_device_compatibility
compatible, warning = check_device_compatibility()
_GPU_ARCH_WARNING.append(None if compatible else warning)
except Exception:
_GPU_ARCH_WARNING.append(None)
return _GPU_ARCH_WARNING[-1]
@router.get("/system/notifications")
def system_notifications():
@@ -602,20 +596,6 @@ def system_notifications():
},
})
# 1b. GPU compute capability unsupported by this torch build (#284) —
# the model "runs" but emits pure noise, the worst silent failure mode
# (RTX 50-series Blackwell sm_120 on pre-cu128 wheels). The loader logs
# this, but a log line never reached the affected users — surface it in
# the panel. Checked once per process: it lazy-imports torch.
gpu_warn = _gpu_arch_warning_cached()
if gpu_warn:
notes.append({
"id": "gpu-arch-unsupported",
"level": "error",
"title": "GPU not supported by this PyTorch build",
"message": gpu_warn + " Until then, output will be noise/garbage.",
})
# 2. Missing ffmpeg
ffmpeg_ok = False
try:
@@ -706,33 +686,18 @@ def _crashed_last_session() -> bool:
if not os.path.exists(CRASH_LOG_PATH):
return False
size = os.path.getsize(CRASH_LOG_PATH)
if size == 0:
acked = int(prefs_get("crash_log_acked_size", 0) or 0)
if size <= acked:
return False
mtime = os.path.getmtime(CRASH_LOG_PATH)
# Composite ack (size + mtime): a bare byte count goes stale after the log
# is truncated — the next crash log can stay smaller than the old acked
# size forever, silently suppressing 'crash-last-session'. The ack only
# holds while it still covers the file's current state.
ack = prefs_get("crash_log_acked")
if isinstance(ack, dict):
if float(ack.get("mtime", 0) or 0) >= mtime and int(ack.get("size", 0) or 0) >= size:
return False
else:
# Legacy size-only ack from older builds.
if size <= int(prefs_get("crash_log_acked_size", 0) or 0):
return False
return mtime < _PROCESS_START_TS
return os.path.getmtime(CRASH_LOG_PATH) < _PROCESS_START_TS
@router.post("/system/crash/ack")
async def ack_crash():
"""Mark the current crash log as seen — dismisses the
'crash-last-session' notification until the log changes again."""
size = mtime = 0
if os.path.exists(CRASH_LOG_PATH):
size = os.path.getsize(CRASH_LOG_PATH)
mtime = os.path.getmtime(CRASH_LOG_PATH)
prefs_set("crash_log_acked", {"size": size, "mtime": mtime})
'crash-last-session' notification until the log grows again."""
size = os.path.getsize(CRASH_LOG_PATH) if os.path.exists(CRASH_LOG_PATH) else 0
prefs_set("crash_log_acked_size", size)
return {"acked_size": size}
+32 -81
View File
@@ -98,30 +98,6 @@ async def ws_tts(websocket: WebSocket):
model = await get_model()
backend = get_active_tts_backend(model=model)
# ── Routing gate (#21 — no silent CPU fallback). WebSockets have
# no response headers, so this uses frames: an error frame +
# close on `unavailable`, a one-time `routing` frame on
# cpu_fallback / accelerated-with-caveat (before any audio).
from core.device_caps import detect_host_caps
from services.engine_routing import resolve_routing, routing_notice
from core.scrub import scrub_text
_routing = resolve_routing(
getattr(backend, "gpu_compat", ("cpu",)), detect_host_caps())
if _routing["routing_status"] == "unavailable":
await websocket.send_json({
"type": "error",
"detail": scrub_text(_routing["routing_reason"])
or "engine cannot run on this host",
})
continue # don't stream; wait for the next request
_notice = routing_notice(_routing)
if _notice:
await websocket.send_json({
"type": "routing",
"status": _notice[0],
"reason": scrub_text(_notice[1]) if _notice[1] else None,
})
# Build generation kwargs
kw: dict = {"speed": data.get("speed", 1.0)}
if data.get("language"):
@@ -168,75 +144,50 @@ async def ws_tts(websocket: WebSocket):
except Exception:
kw["voice"] = voice
# Wave 1.4: split the request into sentences so the first
# sentence's audio streams while later sentences are still
# synthesizing — this is the time-to-first-audio win. The
# chunker handles abbreviations/acronyms/decimals and CJK /
# non-Latin terminators; single-sentence requests behave
# exactly like the old single-shot path.
from services.sentence_chunker import SentenceChunker
_chunker = SentenceChunker(language=(data.get("language") or "en"))
sentences = _chunker.push(text)
sentences.extend(_chunker.flush())
if not sentences:
sentences = [text]
# Run generation in the GPU pool
from services.model_manager import _gpu_pool
loop = asyncio.get_running_loop()
def _generate(sentence_text):
def _generate():
from services.audio_dsp import apply_mastering, normalize_audio
wav = backend.generate(sentence_text, **kw)
wav = backend.generate(text, **kw)
sr_actual = backend.sample_rate
# Like _run_tts in openai_compat: studio engines (VoxCPM2)
# opt out of the broadcast mastering chain. This is the
# other route that runs the active backend, so it needs the
# same guard. Loudness normalisation still runs.
if not getattr(backend, "applies_own_mastering", False):
wav = apply_mastering(wav, sample_rate=sr_actual)
wav = apply_mastering(wav, sample_rate=sr_actual)
wav = normalize_audio(wav, target_dBFS=-2.0)
return wav, sr_actual
wav_tensor, sr = await loop.run_in_executor(_gpu_pool, _generate)
# Send metadata after generation so sample_rate is real
await websocket.send_json({
"type": "start",
"sample_rate": sr,
"channels": 1,
"format": "pcm16",
"engine": backend.id,
})
# Stream PCM16 chunks over the WebSocket
import torch
total_samples = 0
sr = backend.sample_rate
started = False
# Convert to 16-bit PCM
pcm = (wav_tensor * 32767).clamp(-32768, 32767).to(torch.int16)
if pcm.ndim == 2:
pcm = pcm[0] # mono
pcm_bytes = pcm.numpy().tobytes()
for sentence in sentences:
wav_tensor, sr = await loop.run_in_executor(
_gpu_pool, _generate, sentence
)
total_samples = len(pcm)
sent_samples = 0
chunk_bytes = CHUNK_SAMPLES * 2 # 2 bytes per int16 sample
if not started:
# Send metadata after the first generation so
# sample_rate is real (lazy-loading engines report
# their true rate only once weights are up).
await websocket.send_json({
"type": "start",
"sample_rate": sr,
"channels": 1,
"format": "pcm16",
"engine": backend.id,
})
started = True
# Convert to 16-bit PCM and stream
pcm = (wav_tensor * 32767).clamp(-32768, 32767).to(torch.int16)
if pcm.ndim == 2:
pcm = pcm[0] # mono
pcm_bytes = pcm.numpy().tobytes()
n_samples = len(pcm)
sent_samples = 0
while sent_samples < n_samples:
end = min(sent_samples + CHUNK_SAMPLES, n_samples)
chunk = pcm_bytes[sent_samples * 2: end * 2]
await websocket.send_bytes(chunk)
sent_samples = end
# Yield to event loop between chunks for responsiveness
await asyncio.sleep(0)
total_samples += n_samples
while sent_samples < total_samples:
end = min(sent_samples + CHUNK_SAMPLES, total_samples)
start_byte = sent_samples * 2
end_byte = end * 2
chunk = pcm_bytes[start_byte:end_byte]
await websocket.send_bytes(chunk)
sent_samples = end
# Yield to event loop between chunks for responsiveness
await asyncio.sleep(0)
gen_time = round(time.perf_counter() - t0, 3)
duration = round(total_samples / sr, 3)
-27
View File
@@ -34,8 +34,6 @@ class SystemInfoResponse(BaseModel):
asr_model: str = "unknown"
translate_provider: str = "unknown"
has_hf_token: bool = False
# Xet fast-download backend state (FDL-03): {xet_enabled, xet_version, high_performance}
fast_download: dict | None = None
device: str = "cpu"
python: str = ""
platform: str = ""
@@ -129,41 +127,16 @@ class DeviceInfo(BaseModel):
gpu_available: bool = False
gpu_driver: str | None = None
gpu_device_name: str | None = None
# From the canonical device probe (core.device_caps) — distinguishes ROCm
# from CUDA, unlike the legacy nvidia-smi-based gpu_vendor/gpu_backend.
gpu_family: str = "cpu"
vram_gb: float = 0.0
ram_gb: float = 0.0
disk_free_gb: float = 0.0
class GpuRouting(BaseModel):
"""Routing verdict for the active TTS engine on THIS host (#21).
Distinct from the per-engine `routing_*` keys in `/engines`: this is the
single verdict for the *currently-selected* engine, surfaced in preflight +
diagnose so the user hears about a CPU fallback / unavailable GPU before a
slow or failed synth no silent CPU fallback.
"""
model_config = ConfigDict(extra="allow")
engine: str | None = None # active TTS engine id
effective_device: str | None = None # device it will actually use here
routing_status: str | None = None # accelerated|cpu_fallback|cpu_only|unavailable|none
routing_reason: str | None = None # scrubbed; null when none
host_family: str = "cpu" # detect_host_caps().family
vram_gb: float = 0.0
class PreflightResponse(BaseModel):
"""GET /setup/preflight"""
ok: bool
has_warnings: bool = False
checks: list[PreflightCheck] = Field(default_factory=list)
device: DeviceInfo
# Explicit field (PreflightResponse has no extra="allow") so the verdict
# survives serialization instead of being silently dropped.
gpu_routing: GpuRouting | None = None
class InstallModelRequest(BaseModel):
-19
View File
@@ -49,12 +49,6 @@ _BASE_SCHEMA = """
personality TEXT DEFAULT '',
description TEXT DEFAULT '',
is_demo INTEGER DEFAULT 0,
verified_own_voice INTEGER DEFAULT 0,
consent_text TEXT DEFAULT '',
consent_audio_path TEXT DEFAULT '',
consent_recorded_at REAL DEFAULT NULL,
kind TEXT DEFAULT 'clone',
vd_states TEXT DEFAULT NULL,
created_at REAL
);
CREATE TABLE IF NOT EXISTS generation_history (
@@ -144,19 +138,6 @@ _BASE_SCHEMA = """
value TEXT NOT NULL,
updated_at REAL NOT NULL
);
-- Wave 2.2: per-agent MCP voice bindings. An MCP client (Claude Code,
-- Cursor, ) identified by the X-OmniVoice-Client-Id header it sends is
-- bound to a default voice profile / engine. Fresh installs create it
-- here; v0.3.x upgrades get it via alembic 0004.
CREATE TABLE IF NOT EXISTS mcp_client_bindings (
client_id TEXT PRIMARY KEY,
label TEXT NOT NULL DEFAULT '',
profile_id TEXT,
default_engine TEXT,
last_seen_at REAL,
created_at REAL
);
"""
# Only tables/columns this module is allowed to ALTER. Prevents SQL injection via
-336
View File
@@ -1,336 +0,0 @@
"""Free-text voice-description → voice-design parameter mapper (issue #317).
Parity with the hosted omnivoice.app "Describe your voice" field, implemented
fully locally: a deterministic keyword/phrase mapper that projects a natural-
language description (e.g. ``"a warm elderly British storyteller, slightly
raspy"``) onto the **existing** voice-design parameter space — the same six
categories the Design tab's attribute picker drives (Gender / Age / Pitch /
Style / EnglishAccent / ChineseDialect).
Design notes
============
* **No model, no network.** This is an ordered synonym-table matcher, not an
LLM call it runs identically on macOS/Windows/Linux with zero deps beyond
the stdlib, preserving the local-first guarantee.
* **Single source of truth.** Every canonical token this module can emit is
validated at import time against the engine taxonomy in
``omnivoice/utils/voice_design.py`` (loaded via ``core.archetypes``), so the
mapper can never produce an instruct item the engine validator would reject
(the issue-#89 / #115 crash modes). The Chinese translations of each token
(e.g. ````/``中年``) are *derived* from that taxonomy, never hardcoded.
* **Ordered rules, first match wins.** Within a category, rules are checked in
a hand-ordered list so more specific phrases outrank generic ones
("young child" child, not young adult; "very deep" very low pitch, not
low pitch). Within one rule, the earliest occurrence in the text is reported
as the matched phrase. Deterministic by construction.
* **Graceful degradation.** Anything the taxonomy can't express (timbre words
like "raspy", role words like "storyteller") is returned in ``unmatched`` so
the UI can tell the user exactly which parts were ignored instead of failing
silently (issue #317's validation-feedback note). A description with no
matches at all yields all-``Auto`` attrs and an empty instruct.
Localization note (CLAUDE.md): the only hardcoded CJK here is
``DIALECT_PINYIN`` a functional pinyin Chinese-dialect-token mapping
(model vocabulary, like ``frontend/src/utils/constants.js``). Registered in
``tests/test_no_hardcoded_cjk.py``'s allowlist with this justification.
"""
from __future__ import annotations
import re
# Reuse the taxonomy already loaded (stdlib-only, by file path) by the
# archetype engine — same single source of truth, one loader to maintain.
from core.archetypes import _VD
_EN_TO_ZH = _VD._INSTRUCT_EN_TO_ZH # {"male": "男", ...}
_ZH_RE = _VD._ZH_RE
_VALID = _VD._INSTRUCT_ALL_VALID # every token the engine accepts
_DIALECTS = set(_VD._INSTRUCT_CATEGORIES[5]) # the 12 Chinese dialect tokens
# Category names match the frontend's CATEGORIES keys (utils/constants.js) and
# the archetype ``attrs`` shape, so the response drops straight into vdStates.
CATEGORY_ORDER = ("Gender", "Age", "Pitch", "Style", "EnglishAccent", "ChineseDialect")
# ── Pinyin / romanized names → Chinese-dialect tokens (functional vocabulary) ─
DIALECT_PINYIN = {
"henan": "河南话",
"shaanxi": "陕西话",
"sichuan": "四川话",
"szechuan": "四川话",
"guizhou": "贵州话",
"yunnan": "云南话",
"guilin": "桂林话",
"jinan": "济南话",
"shijiazhuang": "石家庄话",
"gansu": "甘肃话",
"ningxia": "宁夏话",
"qingdao": "青岛话",
"dongbei": "东北话",
"northeastern chinese": "东北话",
}
# ── Synonym tables ────────────────────────────────────────────────────────────
# Per category: ordered list of (canonical_token, [phrases]). First rule with
# any hit wins the category, so specific phrases must precede generic ones.
# Each canonical token's Chinese translation from the taxonomy is appended
# automatically at compile time (so "中年" maps to "middle-aged", etc.).
_GENDER_RULES = [
("female", [
"female", "woman", "women", "lady", "ladies", "girl", "girls",
"feminine", "gal", "grandma", "grandmother", "granny", "mother",
"mom", "mum", "aunt", "auntie", "queen", "princess", "actress",
"she", "her",
]),
("male", [
"male", "man", "men", "guy", "guys", "boy", "boys", "masculine",
"gentleman", "gentlemen", "dude", "grandpa", "grandfather", "father",
"dad", "uncle", "king", "prince", "actor", "he", "him", "his",
]),
]
# Order is load-bearing: "child" precedes "young adult" so "young child" →
# child; "middle-aged" precedes "elderly" so elderly's bare "aged" synonym
# can't fire inside the hyphenated "middle-aged" (hyphen is a \b boundary);
# "elderly" precedes "young adult" so grandparent words don't fall through.
_AGE_RULES = [
("child", [
"child", "children", "kid", "kiddo", "toddler", "little boy",
"little girl", "young boy", "young girl", "small child", "childlike",
]),
("teenager", ["teenager", "teen", "teenage", "adolescent"]),
("middle-aged", [
"middle-aged", "middle aged", "middle age", "midlife", "forties",
"fifties", "sixties", "mature",
]),
("elderly", [
"elderly", "old man", "old woman", "old lady", "older man",
"older woman", "elder", "senior", "aged", "grandpa", "grandfather",
"grandma", "grandmother", "granny", "retired", "seventies",
"eighties", "nineties", "old",
]),
("young adult", [
"young adult", "young woman", "young man", "young lady", "youthful",
"twenties", "thirties", "college", "young",
]),
]
# "very …" rules precede their plain counterparts so "very deep" doesn't stop
# at "deep". Bare "low"/"high" only count next to a voice word (pitch/voice/
# tone/register) to avoid false hits like "high quality" or "low effort".
_PITCH_RULES = [
("very low pitch", [
"very low pitch", "very low-pitched", "very low pitched",
"very low voice", "very low tone", "very deep", "extremely deep",
"extremely low", "ultra deep", "booming",
]),
("very high pitch", [
"very high pitch", "very high-pitched", "very high pitched",
"very high voice", "very high tone", "extremely high", "squeaky",
"shrill", "falsetto", "chipmunk",
]),
("low pitch", [
"low pitch", "low-pitched", "low pitched", "low voice", "low tone",
"low register", "deep", "deeper", "bass", "baritone", "husky",
]),
("high pitch", [
"high pitch", "high-pitched", "high pitched", "high voice",
"high tone", "high register", "soprano",
]),
("moderate pitch", [
"moderate pitch", "medium pitch", "medium-pitched", "medium pitched",
"mid-range", "midrange", "average pitch", "moderate",
]),
]
_STYLE_RULES = [
("whisper", [
"whisper", "whispering", "whispered", "whispery", "hushed",
"breathy", "soft-spoken", "soft spoken",
]),
]
# Bare "english" means the language, so only the explicit "english accent"
# phrase maps to british. "chinese" maps to the chinese *accent* (English
# speech with a Chinese accent); actual dialect words live in DIALECT_PINYIN.
_ACCENT_RULES = [
("american accent", [
"american", "america", "usa", "us accent", "midwestern",
"californian", "new york",
]),
("british accent", [
"british", "britain", "english accent", "england", "uk accent",
"london", "cockney", "posh", "received pronunciation",
]),
("australian accent", ["australian", "australia", "aussie"]),
("canadian accent", ["canadian", "canada"]),
("indian accent", ["indian", "india"]),
("chinese accent", ["chinese accent", "chinese-accented", "chinese"]),
("korean accent", ["korean", "korea"]),
("japanese accent", ["japanese", "japan"]),
("portuguese accent", ["portuguese", "portugal", "brazilian", "brazil"]),
("russian accent", ["russian", "russia"]),
]
_DIALECT_RULES = [
(token, [pinyin for pinyin, tok in DIALECT_PINYIN.items() if tok == token])
for token in sorted(_DIALECTS)
]
_RULES = {
"Gender": _GENDER_RULES,
"Age": _AGE_RULES,
"Pitch": _PITCH_RULES,
"Style": _STYLE_RULES,
"EnglishAccent": _ACCENT_RULES,
"ChineseDialect": _DIALECT_RULES,
}
# Import-time guard: every canonical token must be in the engine taxonomy, so
# a taxonomy rename upstream fails loudly here instead of at synthesis time.
for _cat_rules in _RULES.values():
for _token, _ in _cat_rules:
assert _token in _VALID, f"describe_voice token not in taxonomy: {_token!r}"
for _tok in DIALECT_PINYIN.values():
assert _tok in _DIALECTS, f"DIALECT_PINYIN value not a taxonomy dialect: {_tok!r}"
# ── Pattern compilation ───────────────────────────────────────────────────────
def _compile_phrase(phrase: str) -> re.Pattern:
"""Compile a synonym phrase to a regex.
Latin phrases get word boundaries (so "male" never fires inside "female",
"old" never inside "bold") and flexible separators (space or hyphen, so
"middle aged" also matches "middle-aged"). CJK phrases match as plain
substrings word boundaries are meaningless without spaces.
"""
if _ZH_RE.search(phrase):
return re.compile(re.escape(phrase))
parts = [re.escape(p) for p in re.split(r"[ -]+", phrase) if p]
return re.compile(r"\b" + r"[\s\-]+".join(parts) + r"\b")
def _compiled_rules():
out = {}
for cat, rules in _RULES.items():
compiled = []
for token, phrases in rules:
pats = list(phrases)
# Derive the Chinese form of each canonical token from the
# taxonomy (e.g. "middle-aged" → "中年") — never hardcoded here.
zh = _EN_TO_ZH.get(token)
if zh:
pats.append(zh)
if token not in pats:
pats.append(token) # the canonical token always matches itself
compiled.append((token, [_compile_phrase(p) for p in pats]))
out[cat] = compiled
return out
_COMPILED = _compiled_rules()
# "<N> year(s) old / <N>-year-old / <N> yo" → an age bracket. Runs before the
# keyword rules so the trailing "old" never misfires as elderly.
_AGE_NUM = re.compile(
r"\b(\d{1,3})(?:[\s\-]*(?:years?|yrs?|yr)[\s\-]*old|[\s\-]*(?:yo|y/o))\b"
)
def _age_token_for(years: int) -> str:
if years <= 12:
return "child"
if years <= 19:
return "teenager"
if years <= 39:
return "young adult"
if years <= 64:
return "middle-aged"
return "elderly"
def _normalize(description: str) -> str:
text = (description or "").lower()
text = text.replace("", "'").replace("", "'")
text = text.replace("", '"').replace("", '"')
return re.sub(r"[ \t]+", " ", text)
def _match_category(category: str, text: str):
"""Return (token, match) for the first rule with a hit, else None.
Rule order decides the winning token; within the winning rule the earliest
occurrence in the text is reported as the matched phrase.
"""
if category == "Age":
m = _AGE_NUM.search(text)
if m:
return _age_token_for(int(m.group(1))), m
for token, patterns in _COMPILED[category]:
best = None
for pat in patterns:
m = pat.search(text)
if m is not None and (best is None or m.start() < best.start()):
best = m
if best is not None:
return token, best
return None
# Fragment splitter for the "unmatched" report: clause separators (incl. the
# CJK comma/ideographic stop, which CJK descriptions use instead of ASCII).
_FRAGMENT = re.compile(r"[^,;.!?()\n,。;!?、]+")
_HAS_CONTENT = re.compile(r"[\w一-鿿]")
def parse_description(description: str) -> dict:
"""Map a free-text voice description onto the design parameter space.
Returns a dict with:
* ``attrs`` full category token map (``"Auto"`` where nothing
matched); same shape as the Design tab's ``vdStates``.
* ``instruct`` validator-safe instruct string built from the matched
tokens, in canonical category order (may be ``""``).
* ``matched`` list of ``{category, token, phrase}`` for transparency.
* ``unmatched`` clause fragments that contributed no attribute, so the
UI can show what was ignored instead of failing silently.
"""
text = _normalize(description)
attrs = {cat: "Auto" for cat in CATEGORY_ORDER}
matched = []
spans = []
for category in CATEGORY_ORDER:
hit = _match_category(category, text)
if hit is None:
continue
token, m = hit
attrs[category] = token
matched.append({"category": category, "token": token, "phrase": m.group(0)})
spans.append((m.start(), m.end()))
# Accents are English-only and dialects Chinese-only in the engine
# taxonomy; a dialect voice speaks Chinese, so an accent token alongside
# it is contradictory (the issue-#114 conflict class). Dialect wins.
if attrs["ChineseDialect"] != "Auto" and attrs["EnglishAccent"] != "Auto":
dropped = attrs["EnglishAccent"]
attrs["EnglishAccent"] = "Auto"
matched = [m for m in matched if not (m["category"] == "EnglishAccent" and m["token"] == dropped)]
instruct = ", ".join(attrs[c] for c in CATEGORY_ORDER if attrs[c] != "Auto")
unmatched = []
for frag in _FRAGMENT.finditer(text):
if not _HAS_CONTENT.search(frag.group(0)):
continue
lo, hi = frag.start(), frag.end()
if any(s < hi and e > lo for s, e in spans):
continue
unmatched.append(frag.group(0).strip())
return {
"attrs": attrs,
"instruct": instruct,
"matched": matched,
"unmatched": unmatched,
}
-279
View File
@@ -1,279 +0,0 @@
"""Canonical host compute-capability probe — the single source of truth for
"what can this machine actually accelerate on."
Every routing decision (the engine compatibility matrix, ``/setup/preflight``,
``/system/diagnose``, and the synth-time no-silent-fallback gating) reads from
``detect_host_caps()`` so the probe and the model loader can never disagree.
Design contract (load-bearing):
- **Never raises** to a caller. A broken torch / driver crash degrades to a
cached CPU-only ``probe_ok=False`` result; every endpoint stays responsive
(local-first: the app must work with no GPU and even with a broken torch).
- **No network call** driver/sysctl reads only, no tensor allocation, so it
stays kernel-free on cold start.
- **No new regex** on any driver/device string (CodeQL py/polynomial-redos):
the only string parse is the ``int(driver.split(".")[0])`` shape reused
from the wizard, and arch comparison is plain list membership.
- Distinguishes **ROCm from CUDA** (unlike the gguf ``hardware_probe``):
ROCm-on-HIP presents through ``torch.cuda`` but is reported ``family="rocm"``.
The ``get_best_device()`` loader (``services.model_manager``) delegates its
*family* decision here while keeping its own DirectML branch and the ROCm
``HSA_OVERRIDE_GFX_VERSION`` env side-effect the probe **reads**, the loader
**writes**. (The gguf ``hardware_probe.detect_capabilities()`` rebase onto this
module is a deliberate follow-up: it has its own torch-mocked test suite and a
VRAM-driven quant table that is unaffected by the family rename, so it is kept
out of this backend-only slice.)
"""
from __future__ import annotations
import functools
import platform as _platform
import sys
from dataclasses import dataclass
from typing import Literal
DeviceFamily = Literal["cuda", "rocm", "mps", "xpu", "cpu"]
# Stable substring stamped onto notes that represent a real kernel-launch risk
# (arch/driver mismatch) — as opposed to advisory notes (multi-GPU, VRAM query
# failed, DirectML present). ``engine_routing`` keys the "accelerated, but…"
# caveat off this marker so advisory notes never downgrade an accelerated badge.
KERNEL_RISK_MARKER = "may fail at kernel launch"
# Substring marking a DirectML-present (Windows GPU) host. The probe reports
# such hosts as ``family="cpu"`` (DirectML is not a torch device family); the
# router reads this marker to explain the neutral badge instead of "no GPU".
DIRECTML_MARKER = "DirectML device present"
# NOTE: the NVIDIA driver-version check (min R555 for the bundled CUDA runtime)
# is intentionally NOT done here — it requires shelling to ``nvidia-smi``, which
# would put a subprocess on the cold-start probe path. That check stays in
# ``wizard._detect_gpu`` (preflight), which already runs it. The probe only
# emits the torch-visible SM-arch caveat (cheap, metadata-only).
@dataclass(frozen=True)
class HostCaps:
"""Snapshot of the host's accelerator capability. Immutable + cached."""
family: DeviceFamily
"""Best available accelerator family, else ``"cpu"``."""
available_families: tuple[DeviceFamily, ...]
"""Everything usable; **always includes** ``"cpu"`` (invariant)."""
device_name: str = ""
"""Device 0's name, e.g. ``"NVIDIA RTX 4090"`` / ``"Apple Silicon (MPS)"``."""
vram_gb: float = 0.0
"""CUDA/ROCm total VRAM in GB; MPS = system RAM / 2; 0 for cpu/xpu."""
driver: str | None = None
"""Raw ROCm HIP version string (``torch.version.hip``) or ``None``. The
NVIDIA driver-version check is owned by ``wizard._detect_gpu`` (it already
shells to ``nvidia-smi``); the probe stays subprocess-free."""
notes: tuple[str, ...] = ()
"""Author-controlled English advisories (never user input). Empty on a
clean accelerated host."""
probe_ok: bool = True
"""``False`` only when torch could not be imported (degraded CPU-only)."""
def _probe() -> HostCaps:
"""Run the probe once. Enumerates every failure branch from the spec's
degradation contract; never raises."""
try:
import torch
except Exception:
return HostCaps(
family="cpu",
available_families=("cpu",),
notes=("torch not importable; treating host as CPU-only",),
probe_ok=False,
)
notes: list[str] = []
# Probe EVERY accelerator independently into this list (don't short-circuit
# after the first hit) so `available_families` is honest on hybrid hosts
# (e.g. an NVIDIA GPU + an Intel iGPU exposed via IPEX). The preferred
# `family` is chosen by priority at the end.
detected: list[DeviceFamily] = []
device_name = ""
vram_gb = 0.0
driver: str | None = None
# ── CUDA / ROCm (both present through torch.cuda) ────────────────────
cuda_ok = False
try:
cuda_ok = bool(torch.cuda.is_available())
except Exception as exc: # broken CUDA init (forked process / driver crash)
notes.append(f"CUDA init raised: {type(exc).__name__}")
if cuda_ok:
try:
count = int(torch.cuda.device_count())
except Exception:
count = 0
if count == 0:
notes.append("CUDA reports available but device_count==0")
else:
is_rocm = getattr(torch.version, "hip", None) is not None
detected.append("rocm" if is_rocm else "cuda")
if is_rocm:
driver = getattr(torch.version, "hip", None)
if count > 1:
notes.append(f"{count} GPUs detected; routing reflects device 0")
try:
device_name = torch.cuda.get_device_name(0)
except Exception:
device_name = ""
try:
_free, total = torch.cuda.mem_get_info()
vram_gb = float(total) / (1024 ** 3)
except Exception:
notes.append("VRAM query failed")
# SM-arch mismatch (mirrors model_manager.check_device_compatibility).
try:
major, minor = torch.cuda.get_device_capability(0)
arch_list = getattr(torch.cuda, "_get_arch_list", lambda: [])()
if arch_list:
sm_tag = f"sm_{major}{minor}"
compute_tag = f"compute_{major}{minor}"
if sm_tag not in arch_list and compute_tag not in arch_list:
notes.append(
f"{device_name or 'GPU'} ({sm_tag}) not in this torch "
f"build's archs ({', '.join(arch_list)}) — "
f"{KERNEL_RISK_MARKER}"
)
except Exception:
# Arch metadata unavailable on this torch build — skip the check
# (treated as compatible, exactly as check_device_compatibility).
pass
# ── Intel XPU via IPEX ───────────────────────────────────────────────
try:
import intel_extension_for_pytorch # noqa: F401
if hasattr(torch, "xpu") and torch.xpu.is_available():
detected.append("xpu")
if not device_name:
try:
device_name = torch.xpu.get_device_name(0)
except Exception:
# XPU present but unnamed — family classification still holds.
pass
notes.append("XPU VRAM not queried (unreliable across IPEX versions)")
except Exception:
# IPEX absent or XPU probe failed — no XPU on this host.
pass
# ── Apple Silicon MPS ────────────────────────────────────────────────
try:
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
detected.append("mps")
if not device_name:
device_name = "Apple Silicon (MPS)"
if not vram_gb:
try:
import psutil
vram_gb = float(psutil.virtual_memory().total) / (1024 ** 3) / 2
except Exception:
notes.append("psutil unavailable; MPS VRAM unknown")
except Exception:
# MPS probe raised on a non-Apple/old torch — treat as no MPS.
pass
# ── DirectML — Windows GPU, NOT a torch device family ────────────────
try:
import torch_directml
if torch_directml.device_count() > 0:
notes.append(
f"{DIRECTML_MARKER} (Windows GPU); torch-family probe treats "
f"as non-accelerated"
)
except Exception:
# torch_directml absent (the common case) — no DirectML on this host.
pass
# Preferred family by priority; cpu when nothing accelerated was detected.
family: DeviceFamily = "cpu"
for pref in ("cuda", "rocm", "xpu", "mps"):
if pref in detected:
family = pref # type: ignore[assignment]
break
# available_families: every detected accelerator + cpu, deduped, cpu last.
available: tuple[DeviceFamily, ...] = tuple(dict.fromkeys([*detected, "cpu"]))
return HostCaps(
family=family,
available_families=available,
device_name=device_name,
vram_gb=vram_gb,
driver=driver,
notes=tuple(notes),
probe_ok=True,
)
@functools.lru_cache(maxsize=1)
def detect_host_caps() -> HostCaps:
"""Cached per-process host capabilities. Never raises, makes no network
call, kernel-free on cold start. Host compute capability does not change at
runtime in any supported desktop flow (no GPU hot-plug; switching the active
engine does not re-probe routing is recomputed from these same caps), so
a single probe per process is correct. ``probe_ok=False`` is cached too."""
return _probe()
def refresh() -> HostCaps:
"""Clear the cache and re-probe. **TEST-ONLY** — nothing in the running app
calls this (host caps are immutable per process)."""
detect_host_caps.cache_clear()
return detect_host_caps()
def mlx_supported() -> tuple[bool, str]:
"""``(ok, reason)``. ``ok=True`` **only** on Apple Silicon
(``sys.platform == "darwin"`` and ``platform.machine() == "arm64"``) with
torch MPS available the shared gate for MLX-Audio / MLX-Whisper (#390).
Gates on exact-string equality (no regex no CodeQL surface). On any
non-Apple host it returns ``False`` **before** any package import, so a
stray ``mlx_*`` wheel on Linux/Windows never reports available.
"""
if sys.platform != "darwin" or _platform.machine() != "arm64":
if sys.platform == "darwin":
return (False, "MLX requires Apple Silicon; this Mac is Intel")
return (
False,
f"MLX requires Apple Silicon; this host is "
f"{sys.platform}/{_platform.machine()}",
)
try:
import torch
except Exception:
return (False, "torch not importable; cannot confirm MPS")
try:
if torch.backends.mps.is_available():
return (True, "")
except Exception:
# MPS query raised — fall through to the conservative unavailable path.
pass
return (
False,
"Apple Silicon detected but torch MPS unavailable; "
"reinstall torch with MPS support",
)
__all__ = [
"DeviceFamily",
"HostCaps",
"detect_host_caps",
"refresh",
"mlx_supported",
"KERNEL_RISK_MARKER",
"DIRECTML_MARKER",
]
+1 -47
View File
@@ -4,7 +4,7 @@ One pass over everything a working install needs: Python, compute device,
ffmpeg, HF token, disk, data-dir permissions, RAM, TTS engines, and (when
requested) network reachability of the HuggingFace hub. Surfaced two ways:
- ``GET /system/diagnose`` (Settings > About -> "Run self-check")
- ``GET /system/diagnose`` (Settings > About "Run self-check")
- ``python main.py --diagnose`` for headless installs / issue triage
Every ``detail``/``hint`` string is passed through ``core.scrub`` before it
@@ -194,49 +194,6 @@ def _check_engines() -> dict:
return _check("engines", "TTS engines", OK, detail)
def _check_gpu_routing() -> dict:
"""Routing verdict for the active TTS engine on THIS host (#21).
Surfaces a CPU fallback / unavailable-GPU *before* a slow or failed synth
the no-silent-fallback contract. `cpu_only` on a no-GPU machine is the
expected normal state and stays OK (never noise-warns)."""
try:
from services.tts_backend import gpu_routing_verdict
v = gpu_routing_verdict()
except Exception as e:
return _check("gpu_routing", "GPU routing", WARN, f"could not resolve: {e}")
status = v.get("routing_status")
engine = v.get("engine") or "active engine"
dev = v.get("effective_device") or "?"
reason = v.get("routing_reason")
host = v.get("host_family", "cpu")
if status == "accelerated":
if reason: # driver/arch caveat — accelerated but at risk
return _check("gpu_routing", "GPU routing", WARN,
f"{engine} -> {dev}: {reason}",
"The GPU is selected but may fail at kernel launch — "
"update drivers / reinstall torch for this GPU arch.")
return _check("gpu_routing", "GPU routing", OK, f"{engine} -> {dev} (accelerated)")
if status == "cpu_fallback":
return _check("gpu_routing", "GPU routing", WARN,
f"{engine} runs on CPU: {reason or 'no GPU path for this host'}",
"Pick an engine that supports this host's GPU for a big speedup, "
"or continue on CPU (slower).")
if status == "cpu_only":
return _check("gpu_routing", "GPU routing", OK,
f"{engine} -> cpu (no accelerator on this host)")
if status == "unavailable":
return _check("gpu_routing", "GPU routing", FAIL,
f"{engine} can't run on this host: {reason or f'needs a GPU; host is {host}'}",
"Select an engine with a CPU path in Settings -> Engines.")
# status == "none" / unknown — no active engine resolved.
return _check("gpu_routing", "GPU routing", WARN,
"No active TTS engine resolved for routing.",
"Pick an engine in Settings -> Engines.")
_DEEP_TIMEOUT_S = 180
@@ -307,8 +264,6 @@ def _check_network() -> dict:
# all model downloads need to get started. urllib honors HTTP(S)_PROXY.
import urllib.request
import urllib.error
if not _HUB_URL.startswith("https://"): # constant today; guard the sink anyway
raise ValueError(f"hub URL must be https, got {_HUB_URL!r}")
req = urllib.request.Request(_HUB_URL, method="HEAD")
try:
with urllib.request.urlopen(req, timeout=_HUB_TIMEOUT_S):
@@ -341,7 +296,6 @@ def run_diagnostics(include_network: bool = True, deep: bool = False) -> dict:
_check_data_dir(),
_check_ram(),
_check_engines(),
_check_gpu_routing(),
]
if include_network:
checks.append(_check_network())
+1 -2
View File
@@ -111,8 +111,7 @@ def classify_exception(exc: BaseException, trace: str = "") -> str:
def _fingerprint(error_class: str, exc: BaseException) -> str:
import hashlib
raw = f"{error_class}|{type(exc).__name__}|{scrub_text(str(exc))[:200]}"
# Dedup key for the journal, not a security boundary.
return hashlib.sha1(raw.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:16]
return hashlib.sha1(raw.encode("utf-8", "replace")).hexdigest()[:16]
def _persist_locked() -> None:
+1 -1
View File
@@ -32,7 +32,7 @@ _REDACTED_VALUE = "***REDACTED***"
# One-line "what to do" per docs-taxonomy key. Keys mirror error_docs_map's
# taxonomy; the docs URL itself stays owned by error_docs_map.
_HINTS: dict[str, str] = {
"PKG_RESOURCES_MISSING": "Run `uv pip install --reinstall 'setuptools>=75,<80'` in the backend venv (a plain install is skipped when setuptools' metadata is present but its pkg_resources files were removed by antivirus). Restart after.",
"PKG_RESOURCES_MISSING": "Install setuptools in the backend environment (provides pkg_resources).",
"GATEKEEPER_QUARANTINE": "Clear the macOS quarantine flag (xattr -cr the app), then reopen.",
"APPIMAGE_WEBKIT_WHITESCREEN": "Launch with WEBKIT_DISABLE_DMABUF_RENDERER=1 set.",
"HF_AUTH_FAILED": "Set a valid HF_TOKEN in Settings → Hugging Face and retry.",
+1 -4
View File
@@ -42,12 +42,9 @@ _TOKEN_PATTERNS = (
# pattern-wise (not just this machine's $HOME) so paths quoted from a
# user's pasted log on another OS get cleaned too.
_HOME_PATTERNS = (
# Windows-with-forward-slashes must run BEFORE the bare macOS shape, or
# `/Users/<name>` inside `C:/Users/<name>` gets eaten first, leaving `C:~`.
re.compile(r"[A-Za-z]:/Users/[^/\s\"']+"), # Windows, forward slashes (file URLs, normalized traces)
re.compile(r"/Users/[^/\s\"']+"), # macOS
re.compile(r"/home/[^/\s\"']+"), # Linux
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+"), # Windows, backslashes
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+"), # Windows
)
# Values shorter than this are too entropy-poor to be real secrets and too
-25
View File
@@ -82,28 +82,3 @@ def unset_user_env(key: str, path: Optional[str] = None) -> None:
prefix = f"{key}="
lines = [ln for ln in _read_lines(path) if not ln.startswith(prefix)]
_write_lines(path, lines)
def load_into_environ(path: Optional[str] = None) -> bool:
"""Load the durable per-user env file into ``os.environ``, **overriding**
any value a launcher already injected. Returns True if a file was loaded.
This file is the in-app Settings source of truth. The desktop launcher
(Tauri) injects defaults like ``OMNIVOICE_CACHE_DIR`` (and ``HF_ENDPOINT``)
from its *own* config into the backend's environment *before* startup, so
loading this file with ``override=False`` meant a models directory the user
changed in Settings was silently ignored on every launch the effective
location stayed on the old one no matter how many restarts (#480). Both keys
this file can hold are the user's explicit Settings choice and should beat
the launcher's default, so we override. Restores this file's documented
"values written here take effect on the next backend launch" contract.
"""
path = path or os.environ.get("OMNIVOICE_ENV_FILE") or USER_ENV_PATH
if not os.path.isfile(path):
return False
try:
import dotenv
except ImportError:
return False
dotenv.load_dotenv(path, override=True)
return True
-120
View File
@@ -1,120 +0,0 @@
"""Crash-isolated faster-whisper ASR sidecar (Wave 4.2 / Spec 7).
Runs faster-whisper in a child process so a CTranslate2 GPU-teardown segfault
becomes a failed job, not a dead backend. Speaks the SubprocessBackend wire
protocol (length-prefixed JSON over stdin/stdout):
on start {"op":"ready","engine":"faster-whisper-isolated"}
{"op":"ping"} {"op":"pong"}
{"op":"transcribe","audio_path":...,"word_timestamps":bool}
{"op":"segments","result":{"segments":[...],"language":...}}
{"op":"shutdown"} exit 0
error {"op":"error","message":...}
Runs under the PARENT venv (faster-whisper is already a dependency) only the
process boundary is new. torch/CTranslate2 import lazily inside transcribe so
the ready handshake fits the spawn timeout.
"""
from __future__ import annotations
import json
import os
import struct
import sys
import traceback
MAX_FRAME_BYTES = 64 * 1024 * 1024
_model = None
def _send(stream, obj):
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
stream.write(struct.pack("!I", len(body)))
stream.write(body)
stream.flush()
def _recv(stream):
header = stream.read(4)
if len(header) < 4:
return None
(n,) = struct.unpack("!I", header)
if n > MAX_FRAME_BYTES:
raise IOError(f"frame too large: {n}")
body = bytearray()
while len(body) < n:
chunk = stream.read(n - len(body))
if not chunk:
raise IOError("short read")
body.extend(chunk)
return json.loads(bytes(body).decode("utf-8"))
def _get_model():
global _model
if _model is None:
from faster_whisper import WhisperModel
name = os.environ.get("ASR_MODEL_FW", "large-v3")
try:
import torch
device = "cuda" if torch.cuda.is_available() else "cpu"
except Exception:
device = "cpu"
compute = "float16" if device == "cuda" else "int8"
_model = WhisperModel(name, device=device, compute_type=compute)
return _model
def _transcribe(audio_path, word_timestamps):
model = _get_model()
segments, info = model.transcribe(audio_path, word_timestamps=word_timestamps)
out = []
for s in segments:
seg = {"start": float(s.start), "end": float(s.end), "text": s.text}
if word_timestamps and getattr(s, "words", None):
seg["words"] = [
{"word": w.word, "start": float(w.start), "end": float(w.end),
"probability": float(getattr(w, "probability", 0.0))}
for w in s.words
]
out.append(seg)
return {
"segments": out,
"text": " ".join(s["text"].strip() for s in out).strip(),
"language": getattr(info, "language", "unknown"),
}
def main() -> int:
stdin, stdout = sys.stdin.buffer, sys.stdout.buffer
_send(stdout, {"op": "ready", "engine": "faster-whisper-isolated"})
while True:
try:
msg = _recv(stdin)
except Exception as exc:
_send(stdout, {"op": "error", "stage": "recv", "message": f"{type(exc).__name__}: {exc}"})
return 1
if msg is None:
return 0
op = msg.get("op")
try:
if op == "ping":
_send(stdout, {"op": "pong"})
elif op == "transcribe":
result = _transcribe(msg.get("audio_path"), bool(msg.get("word_timestamps", True)))
_send(stdout, {"op": "segments", "result": result})
elif op == "shutdown":
return 0
else:
_send(stdout, {"op": "error", "stage": "dispatch", "message": f"unknown op: {op!r}"})
except Exception as exc:
_send(stdout, {
"op": "error", "stage": "handler",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
return 0
if __name__ == "__main__":
sys.exit(main())
-19
View File
@@ -101,12 +101,6 @@ def main() -> int:
return 0
op = msg.get("op")
# Wave 4.2: deterministic "crash mid-transcription" hook — exit BEFORE
# sending any reply so the parent's blocking recv sees a dead pipe
# (reply=None). The crash-after-one hook below replies first, so it
# can't deterministically exercise the no-reply path.
if op == "transcribe" and os.environ.get("OMNIVOICE_ECHO_CRASH_NO_REPLY") == "1":
os._exit(1)
try:
if op == "ping":
_send(stdout, {"op": "pong"})
@@ -119,19 +113,6 @@ def main() -> int:
"sample_rate": sr,
"n_samples": n_samples,
})
elif op == "transcribe":
# Wave 4.2: echo ASR op — a canned segments result so the
# SubprocessASRBackend round-trip + respawn path is testable
# without a real ASR engine.
_send(stdout, {
"op": "segments",
"result": {
"segments": [{"start": 0.0, "end": 1.0,
"text": f"echo:{msg.get('audio_path', '')}"}],
"text": f"echo:{msg.get('audio_path', '')}",
"language": "en",
},
})
elif op == "shutdown":
return 0
elif op == "probe_env" and test_mode:
-5
View File
@@ -79,11 +79,6 @@ class IndexTTS2Backend(SubprocessBackend):
display_name = "IndexTTS2 (emotion control, duration control, zero-shot)"
supports_voice_design = False # requires ref audio for timbre
_DEFAULT_SAMPLE_RATE = 24000
# Explicit so IndexTTS2 stops advertising the inherited CPU-only default:
# the sidecar runs the IndexTTS PyTorch model on CUDA when present, else
# CPU. ROCm left unclaimed (the sidecar's own venv would need a ROCm torch);
# a ROCm host honestly resolves to cpu_fallback.
gpu_compat = ("cuda", "cpu")
@classmethod
def is_available(cls) -> tuple[bool, str]:
+1 -19
View File
@@ -74,24 +74,6 @@ import traceback
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
MAX_FRAME_BYTES = 64 * 1024 * 1024
def _measure_vram_mb() -> float:
"""This sidecar's own GPU memory in MB, for the loaded-models panel
(MM2-08). The parent can't see a child's VRAM, so we self-report it in the
pong. Degrades to 0 on CPU / when torch isn't loaded yet — never raises."""
try:
import torch # already a dep inside the indextts venv
if torch.cuda.is_available():
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
mps = getattr(torch.backends, "mps", None)
if mps is not None and mps.is_available():
drv = getattr(torch.mps, "driver_allocated_memory", None)
if drv:
return round(drv() / (1024 ** 2), 1)
except Exception:
pass
return 0.0
# Sample rate IndexTTS-2 emits natively. Advertised in the ready frame so
# the parent doesn't have to import IndexTTS just to learn the rate.
INDEXTTS_SAMPLE_RATE = 24000
@@ -300,7 +282,7 @@ def main() -> int:
op = msg.get("op") if isinstance(msg, dict) else None
try:
if op == "ping":
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
_send(stdout, {"op": "pong"})
elif op == "synthesize":
_handle_synthesize(msg, stdout)
elif op == "shutdown":
+6 -67
View File
@@ -353,21 +353,6 @@ def _make_backend_class():
f"This clears the quarantine on the .app and its "
f"bundled binaries. See docs/install/macos.md."
)
# Execute bit (issue #437). A `git clone` / zip extract on POSIX
# can drop +x, which only surfaces at spawn time as
# "[Errno 13] Permission denied" — and the generic synth handler
# then mislabels it as out-of-memory. Self-heal here, AFTER the
# SHA check has confirmed this is the right file (so we never
# chmod a foreign binary). No-op on Windows.
if os.name == "posix" and not os.access(bin_path, os.X_OK):
try:
bin_path.chmod(bin_path.stat().st_mode | 0o111)
except OSError:
return False, (
f"GGUF binary {bin_path.name} isn't executable and "
f"couldn't be made so — run `chmod +x {bin_path}` "
f"and retry."
)
return True, "ready"
except Exception as exc:
return False, f"{type(exc).__name__}: {exc}"
@@ -508,12 +493,6 @@ def _make_backend_class():
* ``ref_audio`` (str/Path) speaker reference WAV for cloning.
* ``ref_text`` (str) transcript of ``ref_audio``.
* ``language`` (str) ISO code or omnivoice-tts lang label.
* ``instruct`` (str) style instruction.
* ``duration`` (float) target duration in seconds.
* ``seed`` (int) deterministic sampling seed.
* ``denoise`` (bool) omit denoise token when false.
* ``preprocess_prompt`` (bool) skip prompt preprocessing when false.
* ``chunk_duration`` / ``chunk_threshold`` (float) binary long-form controls.
"""
import soundfile as sf # local import keeps module import cheap
import torch
@@ -524,32 +503,15 @@ def _make_backend_class():
fd, out_str = tempfile.mkstemp(prefix="omnivoice-gguf-", suffix=".wav")
os.close(fd)
out_path = Path(out_str)
ref_text_path: Optional[Path] = None
try:
ref_text = kw.get("ref_text")
if kw.get("ref_audio") and ref_text:
text_fd, text_str = tempfile.mkstemp(
prefix="omnivoice-gguf-ref-", suffix=".txt"
)
os.close(text_fd)
ref_text_path = Path(text_str)
ref_text_path.write_text(str(ref_text), encoding="utf-8")
argv = self._build_argv(
base=base_path,
tokenizer=tok_path,
out_path=out_path,
ref_audio=kw.get("ref_audio"),
ref_text=str(ref_text_path) if ref_text_path else None,
ref_text=kw.get("ref_text"),
language=kw.get("language"),
instruct=kw.get("instruct"),
duration=kw.get("duration"),
seed=kw.get("seed"),
denoise=kw.get("denoise", True),
preprocess_prompt=kw.get("preprocess_prompt", True),
chunk_duration=kw.get("chunk_duration"),
chunk_threshold=kw.get("chunk_threshold"),
)
self._run_subprocess(argv, stdin_text=text)
wav, sr = sf.read(str(out_path))
@@ -558,11 +520,6 @@ def _make_backend_class():
out_path.unlink()
except OSError:
pass
if ref_text_path is not None:
try:
ref_text_path.unlink()
except OSError:
pass
# soundfile returns (n,) for mono or (n, c) for multichannel.
# OmniVoice/Higgs Audio v2 is mono → (n,). Wrap to (1, n).
@@ -587,13 +544,6 @@ def _make_backend_class():
ref_audio: Optional[str],
ref_text: Optional[str],
language: Optional[str],
instruct: Optional[str] = None,
duration: Optional[float] = None,
seed: Optional[int] = None,
denoise: bool = True,
preprocess_prompt: bool = True,
chunk_duration: Optional[float] = None,
chunk_threshold: Optional[float] = None,
) -> list[str]:
"""Compose argv from typed Path objects only (T-04-02)."""
argv: list[str] = [
@@ -605,20 +555,6 @@ def _make_backend_class():
lang = _iso_to_omnivoice_lang(language)
if lang:
argv += ["--lang", lang]
if instruct:
argv += ["--instruct", str(instruct)]
if duration is not None:
argv += ["--duration", str(float(duration))]
if seed is not None:
argv += ["--seed", str(int(seed))]
if denoise is False:
argv += ["--no-denoise"]
if preprocess_prompt is False:
argv += ["--no-preprocess-prompt"]
if chunk_duration is not None:
argv += ["--chunk-duration", str(float(chunk_duration))]
if chunk_threshold is not None:
argv += ["--chunk-threshold", str(float(chunk_threshold))]
if ref_audio:
# Two-stage validation (defense in depth):
# (a) Reject anything outside the project's voices /
@@ -652,8 +588,11 @@ def _make_backend_class():
)
argv += ["--ref-wav", str(ref_path)]
if ref_text:
# The C++ runtime expects a transcript file path.
# generate() creates this file in the system temp dir.
# ref_text is free-form text; pass via stdin would
# collide with the synthesis prompt, so the only safe
# channel is argv. The binary treats this as a quoted
# string at the OS layer (Popen escapes argv per
# platform); we don't pre-escape.
argv += ["--ref-text", str(ref_text)]
return argv
+7 -162
View File
@@ -43,13 +43,11 @@ try:
_project_env = os.path.join(os.path.dirname(_backend_dir), ".env")
if os.path.isfile(_project_env):
dotenv.load_dotenv(_project_env, override=False)
# Load the durable per-user config (the in-app Settings source of truth) so
# env vars set once survive Tauri/Finder launches that don't inherit a shell
# environment. This OVERRIDES launcher-injected defaults: the desktop app
# injects a stale OMNIVOICE_CACHE_DIR from its own config before startup, so
# without override a models dir changed in Settings was ignored forever (#480).
from core.user_env import load_into_environ as _load_user_env
_load_user_env()
# Also load the durable per-user config so env vars set once survive
# Tauri/Finder launches that don't inherit a shell environment.
_user_env = os.path.expanduser("~/.config/omnivoice/env")
if os.path.isfile(_user_env):
dotenv.load_dotenv(_user_env, override=False)
except ImportError:
pass
@@ -286,12 +284,7 @@ from fastapi.responses import JSONResponse, RedirectResponse, Response
from fastapi.staticfiles import StaticFiles
from fastapi.middleware.cors import CORSMiddleware
from starlette.datastructures import MutableHeaders
# Docs-only dependency: a venv created before scalar-fastapi entered the
# dependency set must still boot the backend (#307) — /docs degrades instead.
try:
from scalar_fastapi import get_scalar_api_reference
except ImportError:
get_scalar_api_reference = None
from scalar_fastapi import get_scalar_api_reference
import traceback
_crash_log_lock = threading.Lock()
@@ -320,7 +313,6 @@ from api.routers import (
setup,
gallery,
archetypes,
describe_voice,
community,
batch,
watermark,
@@ -330,10 +322,7 @@ from api.routers import (
openai_compat,
tts_stream,
marketplace,
personas,
sonitranslate,
audiobook,
longform_jobs,
settings as settings_router, # Phase 1 AUTH-03: HF token save/clear/state
)
from utils import hf_progress
@@ -343,30 +332,6 @@ from utils import hf_progress
# the patched class, not the original.
hf_progress.install()
# Wire the overall download aggregator's byte sink onto the patched tqdm so
# parallel per-file updates feed one accurate overall bar (FDL-06).
try:
from utils import download_aggregator
download_aggregator.install()
except Exception:
pass
# Log the download-acceleration state once at startup (FDL-03) so a slow
# download report can be triaged from the logs without reproducing. Note: the
# app sets HF_HUB_DISABLE_XET=1 above by default (legacy LFS for byte progress),
# so xet_active is normally False even though hf_xet is installed.
try:
from api.routers.system import _fast_download_status as _fd_status
_fd = _fd_status()
_xet_ver = f" {_fd['xet_version']}" if _fd.get("xet_version") else ""
logging.getLogger("omnivoice.model").info(
"downloads: Xet %s (hf_xet%s installed=%s), high_perf=%s",
"ACTIVE" if _fd["xet_active"] else "disabled → legacy LFS",
_xet_ver, _fd["xet_installed"], _fd["high_performance"],
)
except Exception:
pass
def _env_flag(name: str, default: bool = False) -> bool:
value = os.environ.get(name)
@@ -457,23 +422,7 @@ async def lifespan(app: FastAPI):
capture_preload_task = asyncio.create_task(_preload_capture_asr())
else:
logger.info("Capture ASR preload disabled; dictation ASR will load on first use.")
# ── MCP session manager (Wave 2.2) ────────────────────────────────────
# FastMCP's Streamable-HTTP transport needs its session manager running
# for the lifetime of the app. It's created lazily by streamable_http_app()
# (called in mount_mcp below), so we stack its `run()` context into ours
# via AsyncExitStack rather than replacing this lifespan. Best-effort: a
# missing/broken MCP layer must never stop the rest of the backend.
from contextlib import AsyncExitStack
async with AsyncExitStack() as _mcp_stack:
_sm = getattr(app.state, "mcp_session_manager", None)
if _sm is not None:
try:
await _mcp_stack.enter_async_context(_sm.run())
logger.info("MCP server mounted at /mcp")
except Exception as e:
logger.warning("MCP session manager failed to start: %s", e)
yield
yield
# ── Graceful shutdown (SIGTERM from Tauri, Ctrl+C, etc.) ────────────
logger.info("Shutdown: cleaning up…")
idle_task.cancel()
@@ -522,14 +471,6 @@ app = FastAPI(
@app.get("/docs", include_in_schema=False)
async def scalar_docs():
"""Interactive API documentation powered by Scalar."""
if get_scalar_api_reference is None:
return JSONResponse(
status_code=503,
content={
"detail": "API docs unavailable: scalar-fastapi is not installed "
"in the backend environment (#307)."
},
)
return get_scalar_api_reference(
openapi_url=app.openapi_url,
title=app.title,
@@ -638,68 +579,6 @@ class NetworkAccessMiddleware:
return await self.app(scope, receive, send)
class BearerKeyMiddleware:
"""When OMNIVOICE_API_KEY is set, non-loopback clients must present it on
every HTTP + WebSocket request: ``Authorization: Bearer <key>``,
``?api_key=<key>`` (browser WebSockets cannot set headers), or the
``ov_key`` cookie (set on the first successful HTTP auth). Loopback
always bypasses the desktop default is unchanged and the SPA shell
paths stay reachable so a remote UI can load and show what's wrong.
Inert when the env var is unset (the default). Pure ASGI for the same
no-buffering reason as NetworkAccessMiddleware above. Plain-HTTP caveat
is documented in docs/remote-gpu.md: the key is sniffable outside a
WireGuard (Tailscale) or TLS (tailscale serve) transport.
"""
def __init__(self, app):
self.app = app
async def __call__(self, scope, receive, send):
if scope["type"] not in ("http", "websocket"):
return await self.app(scope, receive, send)
key = os.environ.get("OMNIVOICE_API_KEY") or ""
if not key:
return await self.app(scope, receive, send)
client = scope["client"][0] if scope.get("client") else None
if client in _LOOPBACK_CLIENTS:
return await self.app(scope, receive, send)
path = scope.get("path", "")
if scope["type"] == "http" and (
path in _SHELL_PATHS or path.startswith("/assets/") or path.startswith("/favicon")
):
return await self.app(scope, receive, send)
from starlette.requests import HTTPConnection
conn = HTTPConnection(scope)
auth = conn.headers.get("authorization", "")
supplied = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
if not supplied:
supplied = conn.query_params.get("api_key") or conn.cookies.get("ov_key") or ""
if not secrets.compare_digest(supplied, key):
if scope["type"] == "websocket":
# Reject the handshake; 1008 = policy violation.
await receive() # consume websocket.connect
await send({"type": "websocket.close", "code": 1008})
return
resp = JSONResponse({"detail": "API key required"}, status_code=401)
return await resp(scope, receive, send)
if scope["type"] == "http" and conn.cookies.get("ov_key") != key:
async def send_with_cookie(message):
if message["type"] == "http.response.start":
headers = MutableHeaders(scope=message)
headers.append(
"set-cookie", f"ov_key={key}; Path=/; SameSite=Lax"
)
await send(message)
return await self.app(scope, receive, send_with_cookie)
return await self.app(scope, receive, send)
# UI dev-server port — single-sourced from OMNIVOICE_UI_PORT so a user who
# moves the Vite dev server off 3901 still gets a matching CORS allow-list.
def _ui_port() -> int:
@@ -731,15 +610,6 @@ app.add_middleware(
# applied even to the 401 PIN-required responses). Inert unless a PIN is set.
app.add_middleware(NetworkAccessMiddleware)
# Remote-backend bearer gate (parity program Wave 2.3 / §R2). Inert unless
# OMNIVOICE_API_KEY is set. Distinct from the PIN gate above: the PIN guards
# casual LAN-share guests for one session; the API key is the durable
# credential for running this backend remotely (Tailscale / Docker GPU box).
# Covers WebSockets too — the PIN gate never did, because every WS endpoint
# carried its own loopback guard; remote mode is exactly the case where a
# keyed non-loopback client must reach them.
app.add_middleware(BearerKeyMiddleware)
app.mount("/audio", StaticFiles(directory=OUTPUTS_DIR), name="audio")
app.mount("/voice_audio", StaticFiles(directory=VOICES_DIR), name="voice_audio")
@@ -782,7 +652,6 @@ app.include_router(stories.router)
app.include_router(setup.router)
app.include_router(gallery.router)
app.include_router(archetypes.router)
app.include_router(describe_voice.router) # issue #317: free-text voice design
app.include_router(community.router)
app.include_router(batch.router)
app.include_router(watermark.router)
@@ -792,32 +661,8 @@ app.include_router(capture_ws.router)
app.include_router(openai_compat.router)
app.include_router(tts_stream.router)
app.include_router(marketplace.router)
app.include_router(personas.router)
app.include_router(sonitranslate.router)
app.include_router(audiobook.router)
app.include_router(longform_jobs.router)
app.include_router(settings_router.router) # Phase 1 AUTH-03 endpoints
from api.routers import mcp_bindings as _mcp_bindings_router # noqa: E402
app.include_router(_mcp_bindings_router.router) # Wave 2.2 per-agent voice bindings
# ── Mount the MCP server (Wave 2.2) ───────────────────────────────────────
# FastMCP's Streamable-HTTP app is sub-mounted at /mcp; its session manager is
# stashed on app.state for the lifespan above to run. Opt-out via
# OMNIVOICE_MCP_DISABLE=1; best-effort so a missing mcp package or a build
# without it never breaks startup.
if os.environ.get("OMNIVOICE_MCP_DISABLE", "").strip().lower() not in ("1", "true", "yes", "on"):
try:
from mcp_server import create_mcp_server
_mcp = create_mcp_server()
_mcp_app = _mcp.streamable_http_app()
app.state.mcp_session_manager = _mcp.session_manager
app.mount("/mcp", _mcp_app)
logging.getLogger("omnivoice.api").info("MCP app mounted at /mcp")
except Exception as _mcp_err: # noqa: BLE001
logging.getLogger("omnivoice.api").info(
"MCP server not mounted (%s); /mcp disabled.", _mcp_err
)
frontend_path = os.path.join(os.path.dirname(__file__), "..", "frontend", "dist")
if os.path.exists(frontend_path):
+1 -64
View File
@@ -53,14 +53,6 @@ def create_mcp_server():
"voice design, and video dubbing in 646 languages."
),
)
# Serve the Streamable-HTTP transport at the app root so mounting the whole
# app at "/mcp" on the main FastAPI yields the endpoint at "/mcp". FastMCP's
# default path is "/mcp", which would double-prefix to "/mcp/mcp" when
# sub-mounted. Harmless for the standalone CLI run() path.
try:
mcp.settings.streamable_http_path = "/"
except Exception:
pass
# ── Helpers ─────────────────────────────────────────────────────────
@@ -83,21 +75,6 @@ def create_mcp_server():
# ── Tools ───────────────────────────────────────────────────────────
def _current_client_id() -> str | None:
"""The X-OmniVoice-Client-Id of the calling MCP client, if any.
FastMCP exposes the HTTP request via its request context on the
Streamable-HTTP transport; stdio clients (and any version where the
accessor differs) simply resolve to None and fall back to the
global default voice."""
try:
req = mcp.get_context().request_context.request
if req is not None:
return req.headers.get("x-omnivoice-client-id")
except Exception:
pass
return None
@mcp.tool()
async def generate_speech(
text: str,
@@ -112,8 +89,7 @@ def create_mcp_server():
Args:
text: The text to synthesize into speech.
language: Target language (ISO code or 'Auto'). 646 languages supported.
profile_id: ID of a saved voice profile to clone. Omit to use this
agent's bound voice (Settings → MCP), else the global default.
profile_id: ID of a saved voice profile to clone. Omit for voice design mode.
instruct: Style instruction (e.g. 'whisper', 'excited', 'narrator').
speed: Speech speed multiplier (0.52.0, default 1.0).
steps: Diffusion steps (8=fast/draft, 16=balanced, 32=quality).
@@ -122,17 +98,6 @@ def create_mcp_server():
JSON with audio_id, generation_time, audio_duration, and
base64-encoded WAV data.
"""
# Per-agent voice binding (Wave 2.2): explicit arg wins; otherwise
# resolve this client's bound profile, then the global default.
client_id = _current_client_id()
try:
from services import mcp_bindings
resolved = mcp_bindings.resolve_voice(client_id, profile_id)
profile_id = resolved.get("profile_id")
mcp_bindings.touch_last_seen(client_id) if client_id else None
except Exception:
pass # binding layer unavailable — use whatever was passed
form = {
"text": text,
"language": language,
@@ -194,34 +159,6 @@ def create_mcp_server():
'],"note":"Pass any ISO 639 code or set language=Auto for detection."}'
)
@mcp.tool()
async def transcribe(audio_base64: str, language: str | None = None) -> str:
"""Transcribe spoken audio to text.
Args:
audio_base64: Base64-encoded audio bytes (wav/mp3/webm/m4a).
language: Optional language hint; omit for auto-detect.
Returns:
JSON with the recognized text, language, and duration.
"""
try:
raw = base64.b64decode(audio_base64, validate=True)
except Exception:
return '{"error":"audio_base64 is not valid base64"}'
# 200 MB cap — same spirit as voicebox's transcribe gate. Keeps a
# buggy/hostile agent from posting an unbounded blob.
if len(raw) > 200 * 1024 * 1024:
return '{"error":"audio exceeds 200 MB limit"}'
data = {}
if language:
data["language"] = language
r = await _api_post_form(
"/transcribe", data=data,
files={"audio": ("audio.wav", raw, "application/octet-stream")},
)
return str(r.json())
@mcp.tool()
async def check_health() -> str:
"""Check if the OmniVoice backend is running and what GPU device is active."""
-1
View File
@@ -1 +0,0 @@
"""omnivoice-mcp — stdio MCP shim for clients that only speak stdio."""
-176
View File
@@ -1,176 +0,0 @@
"""omnivoice-mcp — stdio ↔ Streamable-HTTP MCP proxy (Wave 2.2).
Adapted from voicebox (https://github.com/jamiepine/voicebox), MIT License,
Copyright (c) voicebox contributors.
Some MCP clients only speak stdio. They spawn this binary; we pipe each
JSON-RPC message to ``http://127.0.0.1:<port>/mcp/`` (the FastMCP app mounted
on the running OmniVoice backend) and stream the server's response back.
Environment variables:
OMNIVOICE_PORT backend port (default 3900).
OMNIVOICE_HOST host (default 127.0.0.1).
OMNIVOICE_CLIENT_ID forwarded as X-OmniVoice-Client-Id on every request
(drives per-agent voice binding).
Stdout is JSON-RPC only. Diagnostics go to stderr.
Exit 0 on clean EOF, 1 on transport error, 2 if the backend never answers.
Usage in an MCP client config (stdio):
command: python
args: ["-m", "backend.mcp_shim"]
env: { OMNIVOICE_CLIENT_ID: "claude-code" }
"""
from __future__ import annotations
import asyncio
import json
import os
import sys
from typing import Any
import httpx
CLIENT_ID_HEADER = "X-OmniVoice-Client-Id"
SESSION_HEADER = "mcp-session-id"
HEALTH_TIMEOUT_S = 30.0
DEFAULT_PORT = 3900
def _err(msg: str) -> None:
print(f"omnivoice-mcp: {msg}", file=sys.stderr, flush=True)
def _base_url() -> tuple[str, str]:
host = os.environ.get("OMNIVOICE_HOST", "127.0.0.1")
port = int(os.environ.get("OMNIVOICE_PORT", str(DEFAULT_PORT)))
return f"http://{host}:{port}/mcp/", f"http://{host}:{port}/health"
async def _wait_for_backend(client: httpx.AsyncClient, health_url: str) -> bool:
loop = asyncio.get_running_loop()
deadline = loop.time() + HEALTH_TIMEOUT_S
while loop.time() < deadline:
try:
r = await client.get(health_url, timeout=2.0)
if r.status_code == 200:
return True
except Exception:
pass
await asyncio.sleep(0.5)
return False
async def _read_stdin_line() -> str | None:
loop = asyncio.get_running_loop()
line = await loop.run_in_executor(None, sys.stdin.readline)
return line or None
def _write_stdout(obj: Any) -> None:
sys.stdout.write(json.dumps(obj, separators=(",", ":")))
sys.stdout.write("\n")
sys.stdout.flush()
async def _handle_request(
client: httpx.AsyncClient,
url: str,
raw: str,
headers: dict[str, str],
session_id: list[str | None],
) -> None:
try:
message = json.loads(raw)
except json.JSONDecodeError as exc:
_err(f"invalid JSON on stdin: {exc}")
return
req_headers = {
"Content-Type": "application/json",
"Accept": "application/json, text/event-stream",
**headers,
}
if session_id[0]:
req_headers[SESSION_HEADER] = session_id[0]
is_notification = isinstance(message, dict) and "id" not in message
async with client.stream("POST", url, headers=req_headers, content=raw.encode("utf-8")) as response:
if session_id[0] is None:
sid = response.headers.get(SESSION_HEADER)
if sid:
session_id[0] = sid
if response.status_code == 202:
return # notification acknowledged
if response.status_code >= 400:
body = await response.aread()
_err(f"server {response.status_code}: {body.decode('utf-8', errors='replace')[:400]}")
if is_notification:
return
_write_stdout({
"jsonrpc": "2.0",
"id": message.get("id"),
"error": {"code": -32000, "message": f"OmniVoice MCP proxy got HTTP {response.status_code}"},
})
return
ctype = response.headers.get("content-type", "")
if "text/event-stream" in ctype:
async for line in response.aiter_lines():
if line.startswith("data:"):
payload = line[5:].strip()
if not payload:
continue
try:
_write_stdout(json.loads(payload))
except json.JSONDecodeError:
_err(f"malformed SSE payload: {payload[:200]}")
else:
body = await response.aread()
try:
_write_stdout(json.loads(body))
except json.JSONDecodeError:
_err(f"non-JSON response ({ctype}): {body.decode('utf-8', errors='replace')[:200]}")
async def _run() -> int:
url, health_url = _base_url()
forward_headers: dict[str, str] = {}
client_id = os.environ.get("OMNIVOICE_CLIENT_ID")
if client_id:
forward_headers[CLIENT_ID_HEADER] = client_id
session_id: list[str | None] = [None]
async with httpx.AsyncClient(timeout=httpx.Timeout(300.0)) as client:
if not await _wait_for_backend(client, health_url):
_err(f"timed out waiting for OmniVoice at {health_url} — is the app running?")
return 2
try:
while True:
line = await _read_stdin_line()
if line is None:
return 0
line = line.strip()
if not line:
continue
await _handle_request(client, url, line, forward_headers, session_id)
except (KeyboardInterrupt, SystemExit):
return 0
except Exception as exc:
_err(f"proxy failed: {exc!r}")
return 1
def main() -> int:
try:
return asyncio.run(_run())
except KeyboardInterrupt:
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -1,58 +0,0 @@
"""Parity program Wave 0.2: consent-locked voice profiles
Revision ID: 0003_voice_profile_consent
Revises: 0002_voice_profile_demo_fields
Create Date: 2026-06-12 00:00:00.000000
Adds four additive columns to ``voice_profiles`` backing the
``verified_own_voice`` consent lock (docs/competitive-analysis.md Action 22 /
parity program Wave 0.2). A profile becomes "verified" when its owner records
a spoken consent statement; agentic features and gallery sharing will require
the flag plain local synthesis never does.
* ``verified_own_voice INTEGER DEFAULT 0`` the consent lock itself.
* ``consent_text TEXT DEFAULT ''`` the statement that was read aloud.
* ``consent_audio_path TEXT DEFAULT ''`` filename of the recorded
statement in VOICES_DIR (kept as provenance, deletable via revoke).
* ``consent_recorded_at REAL DEFAULT NULL`` UNIX timestamp.
Behavior mirrors 0002: ``_has_column`` PRAGMA guards make upgrade a no-op on
fresh installs (where _BASE_SCHEMA already has the columns), satisfying the
"Backward-compatible project data" constraint; downgrade drops the columns
(SQLite >= 3.35).
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0003_voice_profile_consent"
down_revision: Union[str, None] = "0002_voice_profile_demo_fields"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_COLUMNS = (
("verified_own_voice", sa.Column("verified_own_voice", sa.Integer(), nullable=False, server_default="0")),
("consent_text", sa.Column("consent_text", sa.Text(), nullable=False, server_default="")),
("consent_audio_path", sa.Column("consent_audio_path", sa.Text(), nullable=False, server_default="")),
("consent_recorded_at", sa.Column("consent_recorded_at", sa.Float(), nullable=True)),
)
def _has_column(table: str, column: str) -> bool:
bind = op.get_bind()
rows = bind.execute(sa.text(f"PRAGMA table_info({table})")).fetchall()
return any(r[1] == column for r in rows)
def upgrade() -> None:
for name, column in _COLUMNS:
if not _has_column("voice_profiles", name):
op.add_column("voice_profiles", column)
def downgrade() -> None:
for name, _ in reversed(_COLUMNS):
if _has_column("voice_profiles", name):
op.drop_column("voice_profiles", name)
@@ -1,60 +0,0 @@
"""Parity program Wave 2.2: per-agent MCP voice bindings
Revision ID: 0004_mcp_client_bindings
Revises: 0003_voice_profile_consent
Create Date: 2026-06-12 00:00:00.000000
Adds the ``mcp_client_bindings`` table backing per-agent voice binding
(docs/competitive-analysis.md Spec 2): each MCP client (identified by the
``X-OmniVoice-Client-Id`` header it sends) can be bound to a default voice
profile / engine, so "Claude Code speaks in Morgan, Cursor in Scarlett".
* ``client_id`` TEXT PRIMARY KEY the agent's stable id.
* ``label`` TEXT human label shown in Settings.
* ``profile_id`` TEXT voice profile to speak in (nullable FK-by-convention).
* ``default_engine`` TEXT engine override (nullable).
* ``last_seen_at`` REAL updated when the client calls a tool.
* ``created_at`` REAL.
Additive + idempotent (guarded by sqlite_master), matching 0002/0003, so
re-running on a fresh-install DB where _BASE_SCHEMA already created it is a
no-op (Backward-compatible project data constraint).
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0004_mcp_client_bindings"
down_revision: Union[str, None] = "0003_voice_profile_consent"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _has_table(name: str) -> bool:
bind = op.get_bind()
row = bind.execute(
sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name=:n"),
{"n": name},
).fetchone()
return row is not None
def upgrade() -> None:
if _has_table("mcp_client_bindings"):
return
op.create_table(
"mcp_client_bindings",
sa.Column("client_id", sa.Text(), primary_key=True),
sa.Column("label", sa.Text(), nullable=False, server_default=""),
sa.Column("profile_id", sa.Text(), nullable=True),
sa.Column("default_engine", sa.Text(), nullable=True),
sa.Column("last_seen_at", sa.Float(), nullable=True),
sa.Column("created_at", sa.Float(), nullable=True),
)
def downgrade() -> None:
if _has_table("mcp_client_bindings"):
op.drop_table("mcp_client_bindings")
@@ -1,65 +0,0 @@
"""Voice Studio unification: profile `kind` discriminator + design params
Revision ID: 0005_unified_profiles
Revises: 0004_mcp_client_bindings
Create Date: 2026-06-13 00:00:00.000000
Adds two additive columns to ``voice_profiles`` so a *designed* voice
(category sliders + instruct, no user reference audio) is a first-class
profile rather than a transient UI state
(docs/specs/voice-studio-unification.md §3):
* ``kind TEXT DEFAULT 'clone'`` ``'clone'`` (user reference audio) or
``'design'`` (rendered sample + stored design params). Replaces the
brittle is_locked/instruct inference in /generate.
* ``vd_states TEXT DEFAULT NULL`` JSON of the design category picks
(Gender/Age/Pitch/Style/accent/dialect) so selecting a design profile
can restore the sliders for re-editing.
Backfill: every existing row becomes ``kind='clone'`` all of them carry a
real or rendered ``ref_audio_path`` today (archetype materialization
included), so the default is semantically true and no audio is re-rendered.
Behavior mirrors 0002/0003: ``_has_column`` PRAGMA guards make upgrade a
no-op on fresh installs (where _BASE_SCHEMA already has the columns),
satisfying the "Backward-compatible project data" constraint; downgrade
drops the columns (SQLite >= 3.35).
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0005_unified_profiles"
down_revision: Union[str, None] = "0004_mcp_client_bindings"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _has_column(table: str, column: str) -> bool:
bind = op.get_bind()
rows = bind.execute(sa.text(f"PRAGMA table_info({table})")).fetchall()
return any(r[1] == column for r in rows)
def upgrade() -> None:
if not _has_column("voice_profiles", "kind"):
op.add_column(
"voice_profiles",
sa.Column("kind", sa.Text(), nullable=False, server_default="clone"),
)
# server_default covers new rows; make existing rows explicit too.
op.execute("UPDATE voice_profiles SET kind='clone' WHERE kind IS NULL OR kind=''")
if not _has_column("voice_profiles", "vd_states"):
op.add_column(
"voice_profiles",
sa.Column("vd_states", sa.Text(), nullable=True),
)
def downgrade() -> None:
if _has_column("voice_profiles", "vd_states"):
op.drop_column("voice_profiles", "vd_states")
if _has_column("voice_profiles", "kind"):
op.drop_column("voice_profiles", "kind")
+1 -42
View File
@@ -25,11 +25,6 @@ class DubSegment(BaseModel):
speed: Optional[float] = None
gain: Optional[float] = None # Per-segment volume (0.0 - 2.0, default 1.0)
target_lang: Optional[str] = None # Per-segment language override (ISO code)
# Phase 4.2 free-form directorial note ("urgent, whispered…"). The client
# has always sent this; without the field pydantic silently dropped it,
# so directions never reached TTS and never entered the regen
# fingerprint (#281).
direction: Optional[str] = None
effect_preset: str = "broadcast" # NEW: DSP preset id (default: broadcast)
@field_validator("effect_preset")
@@ -42,19 +37,6 @@ class DubSegment(BaseModel):
)
return v
class FitOptions(BaseModel):
"""Optional knob overrides for the `smart_fit` timing strategy.
All fields default to None the server fills in the canonical
defaults (services.fit_planner.FitParams) so old clients and sparse
payloads behave identically to a fully-populated default payload.
"""
max_audio_only_rate: Optional[float] = None # default 1.2
audio_rate_cap: Optional[float] = None # default 1.5
video_slow_cap: Optional[float] = None # default 2.0
gap_guard_s: Optional[float] = None # default 0.05
allow_video_retime: Optional[bool] = None # default True
class DubRequest(BaseModel):
segments: List[DubSegment]
language: str = "Auto"
@@ -93,35 +75,18 @@ class DubRequest(BaseModel):
# each segment's video portion is stretched (via
# ffmpeg setpts) to fit the natural-rate dub audio.
# Audio plays at 1.0×; total video duration grows.
# "smart_fit" — dub-length fitting v2: split the burden between a
# mild pitch-preserving audio speed-up (≤1.2× alone,
# ≤1.5× in hybrid) and a mild per-segment video
# slow-down (≤2.0×), per services/fit_planner.py.
# Residual overflow is trimmed and surfaced.
# "strict_slot" — legacy: keep `slot_fit` semantics (atempo squeeze
# when audio > slot). Kept for back-compat.
timing_strategy: Optional[Literal["concise", "stretch_video", "strict_slot", "smart_fit"]] = "concise"
timing_strategy: Optional[Literal["concise", "stretch_video", "strict_slot"]] = "concise"
# Per-job slip budget for "concise" mode. Hard-trim only kicks in once
# gap absorption + this much extra time has been consumed.
overflow_budget_s: Optional[float] = 0.0
# Knob overrides for `smart_fit` (ignored by other strategies). Omitted
# fields default server-side to fit_planner.FitParams values.
fit_options: Optional[FitOptions] = None
class TranslateSegment(BaseModel):
id: str
text: str
target_lang: Optional[str] = None
# Free-form delivery direction ("urgent, whispering") — feeds the
# cinematic reflect/adapt prompts. The frontend has sent this since
# Phase 4.2 but pydantic silently dropped it as an undeclared extra,
# so the per-segment direction hint never reached the LLM.
direction: Optional[str] = None
# Available time slot (end - start, seconds) for rate-ratio prediction
# and the cinematic slot-fit pass. Same silent-drop fix as `direction`.
slot_seconds: Optional[float] = None
class TranslateRequest(BaseModel):
segments: List[TranslateSegment]
@@ -131,12 +96,6 @@ class TranslateRequest(BaseModel):
job_id: Optional[str] = None # Dub job id, used to resolve detected source_lang
quality: Optional[str] = "fast" # "fast" (one-shot) | "cinematic" (reflect → adapt)
glossary: Optional[List[dict]] = None # [{"source": "...", "target": "...", "note": "..."}]
# Optional regional dialect (BCP-47, e.g. "es-AR", "pt-BR") — #280 item 2.
# Applied by LLM-backed paths (provider="openai" or quality="cinematic"):
# the prompt asks for that region's vocabulary/grammar (e.g. Argentinian
# voseo: "vos sos" instead of "tú eres"). Non-LLM providers (Argos, NLLB,
# Google) can't honor it; the response then carries dialect_applied=false.
dialect: Optional[str] = None
class DubIngestUrlRequest(BaseModel):
url: str
-281
View File
@@ -1,281 +0,0 @@
"""Acoustic echo cancellation for dictate-over-playback (parity Action 8b).
When the user dictates while OmniVoice is *playing* audio (a TTS preview, a
dub render, a video), the loudspeaker signal leaks back into the microphone.
The streaming ASR on ``/ws/transcribe`` then transcribes that bleed as if it
were speech the "it typed back what the app just said" symptom. A browser's
``getUserMedia({echoCancellation:true})`` would help, but its quality and even
its availability differ per platform/webview, which would make a *default*
feature behave differently on macOS/Windows/Linux against the project's
cross-platform-parity rule. A server-side canceller behaves identically
everywhere, so it is the local-first, platform-neutral choice.
This is an NLMS (normalised least-mean-squares) time-domain adaptive filter
with a Geigel double-talk detector. It is a clean-room-grade *port* of
Patter's ``getpatter/audio/aec.py`` (MIT) — see docs/competitive-analysis.md,
Action 8. It is NOT production-grade DSP (WebRTC AEC3 / Speex AEC are); it is
a dependency-free, good-enough canceller that removes the steady-state echo
the ASR would otherwise hallucinate on.
Wiring (one instance per ``/ws/transcribe`` session NOT thread-safe)::
aec = NlmsEchoCanceller(sample_rate=16000)
# Far-end: every PCM chunk the client is about to play through speakers.
aec.push_far_end(playback_pcm_bytes)
# Near-end: the mic PCM, cleaned before it reaches the ASR buffer.
cleaned = aec.process_near_end(mic_pcm_bytes)
"""
from __future__ import annotations
import logging
import time
from typing import Final
import numpy as np
logger = logging.getLogger("omnivoice.aec")
_DEFAULT_FILTER_TAPS: Final[int] = 512
"""Adaptive-filter length in samples. 512 taps @ 16 kHz = 32 ms, covering a
typical near-field laptop/desktop echo path. Longer tails (large rooms) can
pass ``filter_taps=1024``+ at proportionally more CPU per frame; 512 converges
in ~0.5 s with the warm-up ramp and is the sweet spot for dictation."""
_DEFAULT_STEP_SIZE: Final[float] = 0.1
"""Steady-state NLMS step size. Larger = faster channel tracking but less
stable; 0.1 is the textbook value for narrowband voice."""
_DEFAULT_WARMUP_STEP_SIZE: Final[float] = 0.5
"""Aggressive step used during the warm-up window so the filter reaches a
usable echo estimate within ~0.5 s instead of several seconds. The Geigel
double-talk detector still gates updates, so the bigger step does not learn
the user's own voice as echo."""
_DEFAULT_WARMUP_SECONDS: Final[float] = 0.5
"""Length of the warm-up window. After this many seconds of processed
near-end audio the step decays from ``warmup_step_size`` to ``step_size``."""
_DEFAULT_LEAKAGE: Final[float] = 0.9999
"""Per-iteration weight leakage (slightly < 1) so the filter slowly forgets
stale taps when the echo path drifts (the user moves the mic)."""
_DOUBLE_TALK_RHO: Final[float] = 0.6
"""Geigel double-talk threshold. When ``max(|near|) > rho * max(|far|)`` the
near-end carries energy the far-end cannot explain (the user is talking)
freeze adaptation so the filter does not model the user's voice as echo."""
_FAR_END_BUFFER_SECONDS: Final[float] = 0.5
"""How much past far-end (playback) audio to retain. The echo arrives at the
mic tens of ms after playback; the filter needs that much look-back to align.
500 ms is generous headroom."""
class NlmsEchoCanceller:
"""Time-domain NLMS adaptive filter with Geigel double-talk detection.
Operates on narrowband mono PCM at 16 kHz (the rate the dictation path
resamples to) or 8 kHz. Not thread-safe each ``/ws/transcribe`` session
owns its own instance.
"""
# Far-end staleness window (seconds): once the most recent far-end push
# is older than this, ``process_near_end`` passes the mic through instead
# of cancelling against a frozen reference (which would superimpose the
# same stale ~50 ms waveform on every mic frame as an audible buzz).
_FAR_STALE_S: float = 0.25
def __init__(
self,
sample_rate: int = 16000,
*,
filter_taps: int = _DEFAULT_FILTER_TAPS,
step_size: float = _DEFAULT_STEP_SIZE,
warmup_step_size: float = _DEFAULT_WARMUP_STEP_SIZE,
warmup_seconds: float = _DEFAULT_WARMUP_SECONDS,
leakage: float = _DEFAULT_LEAKAGE,
double_talk_rho: float = _DOUBLE_TALK_RHO,
) -> None:
if sample_rate not in (8000, 16000):
raise ValueError(
"NlmsEchoCanceller supports 8000 Hz or 16000 Hz only; "
f"got {sample_rate}."
)
if filter_taps < 64:
raise ValueError(
f"filter_taps must be >= 64 to model a meaningful echo path; "
f"got {filter_taps}."
)
if not 0 < step_size <= 1:
raise ValueError(f"step_size must be in (0, 1]; got {step_size}.")
if not 0 < warmup_step_size <= 1:
raise ValueError(
f"warmup_step_size must be in (0, 1]; got {warmup_step_size}."
)
if warmup_seconds < 0:
raise ValueError(f"warmup_seconds must be >= 0; got {warmup_seconds}.")
if not 0 < leakage <= 1:
raise ValueError(f"leakage must be in (0, 1]; got {leakage}.")
self._sample_rate = sample_rate
self._taps = filter_taps
self._step = float(step_size)
self._warmup_step = float(warmup_step_size)
self._warmup_samples = int(warmup_seconds * sample_rate)
self._leakage = float(leakage)
self._rho = float(double_talk_rho)
# Counts near-end samples processed so the step can taper from
# warmup_step to step over the first warmup_samples. Counted from the
# first process_near_end call so the window aligns with playback start.
self._processed_samples: int = 0
self._last_far_push_monotonic: float | None = None
# Filter coefficients (zeros — adapts to the channel within ~0.52 s).
self._w = np.zeros(filter_taps, dtype=np.float32)
# Far-end ring buffer holding >= filter_taps samples of playback
# history, with headroom so push/process can interleave freely.
max_buf_samples = max(
filter_taps * 2,
int(sample_rate * _FAR_END_BUFFER_SECONDS),
)
self._far_buf = np.zeros(max_buf_samples, dtype=np.float32)
self._far_write_idx = 0 # next write position (head)
self._far_filled = 0 # samples written so far (capped at len(far_buf))
# Diagnostics only — never read in the hot path.
self.frames_processed: int = 0
self.double_talk_frames: int = 0
# ── Public API ──────────────────────────────────────────────────────────
def push_far_end(self, pcm_bytes: bytes) -> None:
"""Append far-end (playback) audio to the reference ring buffer.
Accepts raw int16 little-endian mono PCM at the configured rate.
"""
if not pcm_bytes:
return
self._last_far_push_monotonic = time.monotonic()
samples = np.frombuffer(pcm_bytes, dtype=np.int16).astype(np.float32) / 32768.0
n = samples.shape[0]
buf_len = self._far_buf.shape[0]
if n >= buf_len:
# More than the buffer holds — keep only the newest buf_len.
self._far_buf[:] = samples[-buf_len:]
self._far_write_idx = 0
self._far_filled = buf_len
return
end = self._far_write_idx + n
if end <= buf_len:
self._far_buf[self._far_write_idx:end] = samples
else:
head = buf_len - self._far_write_idx
self._far_buf[self._far_write_idx:] = samples[:head]
self._far_buf[: n - head] = samples[head:]
self._far_write_idx = (self._far_write_idx + n) % buf_len
self._far_filled = min(self._far_filled + n, buf_len)
def process_near_end(self, pcm_bytes: bytes) -> bytes:
"""Subtract the estimated echo from the near-end (mic) signal.
Returns int16 little-endian mono PCM with the estimated echo removed.
Passes the frame through unchanged when there is nothing worth
cancelling: no playback has been primed, or the far-end reference is
stale (the app went silent).
"""
if not pcm_bytes:
return pcm_bytes
# Not enough far-end history to fill the filter window yet — passing
# through avoids emitting garbage on the first frames.
if self._far_filled < self._taps:
return pcm_bytes
# Far-end reference is stale (app stopped playing): the ring only
# advances on push_far_end, so the "most recent" window is frozen at
# the tail of the last playback. Convolving against it would buzz.
last_push = self._last_far_push_monotonic
if last_push is None or (time.monotonic() - last_push) > self._FAR_STALE_S:
return pcm_bytes
near = np.frombuffer(pcm_bytes, dtype=np.int16).astype(np.float32) / 32768.0
cleaned = self._block_nlms(near)
out = np.clip(cleaned * 32768.0, -32768.0, 32767.0).astype(np.int16)
self.frames_processed += 1
return out.tobytes()
def reset(self) -> None:
"""Clear filter coefficients and far-end history (e.g. on a new turn)."""
self._w.fill(0)
self._far_buf.fill(0)
self._far_write_idx = 0
self._far_filled = 0
self._processed_samples = 0
self._last_far_push_monotonic = None
self.frames_processed = 0
self.double_talk_frames = 0
# ── Internals ───────────────────────────────────────────────────────────
def _far_window(self, length: int) -> np.ndarray:
"""Most recent ``length`` far-end samples, oldest first / newest last."""
buf_len = self._far_buf.shape[0]
if length > self._far_filled:
length = self._far_filled
end = self._far_write_idx # newest sample is at (end - 1) mod buf_len
if end >= length:
return self._far_buf[end - length: end]
head = self._far_buf[buf_len - (length - end):]
tail = self._far_buf[:end]
return np.concatenate((head, tail))
def _block_nlms(self, near: np.ndarray) -> np.ndarray:
"""Sample-by-sample NLMS over one frame of near-end samples.
Classical NLMS depends on the weights adapted at the previous sample,
so the inner loop is sequential. Each sample is O(taps); numpy keeps a
320-sample / 512-tap frame well under a millisecond on commodity CPUs.
"""
taps = self._taps
far_window = self._far_window(taps + near.shape[0] - 1)
if far_window.shape[0] < taps + near.shape[0] - 1:
# Still warming up — left-pad with zeros so indices line up.
pad = np.zeros(
taps + near.shape[0] - 1 - far_window.shape[0], dtype=np.float32
)
far_window = np.concatenate((pad, far_window))
# Geigel double-talk detector (frame-wise).
far_max = float(np.max(np.abs(far_window))) if far_window.size else 0.0
near_max = float(np.max(np.abs(near)))
# Freeze adaptation when the far reference is effectively silent
# (<= -60 dBFS): adapting against a fade-out tail with near-zero norm
# blows the weights up against user speech when playback resumes.
if far_max <= 1e-3:
return near
double_talk = near_max > self._rho * far_max
if double_talk:
self.double_talk_frames += 1
out = np.empty_like(near)
w = self._w
leakage = self._leakage
# Constant step within the frame keeps the inner loop branch-free.
if self._processed_samples < self._warmup_samples:
step = self._warmup_step
else:
step = self._step
for i in range(near.shape[0]):
x = far_window[i: i + taps]
y_est = float(np.dot(w, x))
e = float(near[i] - y_est)
out[i] = e
if not double_talk:
# NLMS update with leakage. +1e-6 guards divide-by-zero.
norm = float(np.dot(x, x)) + 1e-6
w *= leakage
w += (step * e / norm) * x
self._processed_samples += near.shape[0]
return out
+11 -232
View File
@@ -31,74 +31,12 @@ from abc import ABC, abstractmethod
logger = logging.getLogger("omnivoice.asr")
def _decode_audio_16k_mono(audio_path: str):
"""Decode `audio_path` to a 16 kHz mono float32 waveform using OmniVoice's
*validated* ffmpeg, instead of whisperx.load_audio's bare ``"ffmpeg"`` PATH
lookup.
whisperx (and openai-whisper) shell out to a literal ``"ffmpeg"`` resolved
against the OS PATH. On Windows that resolves to whatever the system finds
first a WindowsApps alias stub or a corrupt/wrong-arch download which
passes `which` but explodes at spawn with ``[WinError 193] %1 is not a valid
Win32 application``. whisperx only catches `CalledProcessError`, so the
spawn-time `OSError` escapes and the dub/batch path reports the opaque
"Transcription produced no segments" (#479). ``find_ffmpeg()`` probes each
candidate with ``-version`` and returns a runnable binary (the bundled
imageio-ffmpeg / Tauri sidecar) or None, so we can raise an actionable
error. This also fixes the imageio case a PATH-prepend can't: its binary is
named ``ffmpeg-<plat>-vN.exe``, not ``ffmpeg``, so bare lookup never finds
it. Mirrors whisperx.audio.load_audio's command exactly (16 kHz, mono, s16le).
"""
import subprocess
import numpy as np
from services.ffmpeg_utils import find_ffmpeg
ffmpeg = find_ffmpeg()
if not ffmpeg:
raise RuntimeError(
"Cannot transcribe: ffmpeg is missing or not runnable. Install "
"ffmpeg (or let OmniVoice's bundled binary download), then retry. "
"On Windows a '[WinError 193]' here means the ffmpeg binary is "
"corrupt or the wrong architecture — reinstall it or clear the "
"imageio-ffmpeg cache."
)
cmd = [
ffmpeg, "-nostdin", "-threads", "0", "-i", audio_path,
"-f", "s16le", "-ac", "1", "-acodec", "pcm_s16le", "-ar", "16000", "-",
]
try:
out = subprocess.run(cmd, capture_output=True, check=True).stdout
except OSError as e:
# Belt-and-suspenders: find_ffmpeg() already -version-validated this
# binary, so a WinError 193 here is unexpected — surface it clearly
# rather than letting it become "no segments".
raise RuntimeError(
f"ffmpeg at {ffmpeg!r} could not be executed ({e}). Reinstall "
"ffmpeg or clear the imageio-ffmpeg cache."
) from e
except subprocess.CalledProcessError as e:
stderr = (e.stderr or b"").decode(errors="replace")[:500]
raise RuntimeError(f"Failed to decode audio for transcription: {stderr}") from e
return np.frombuffer(out, np.int16).flatten().astype(np.float32) / 32768.0
# ── Protocol ────────────────────────────────────────────────────────────────
class ASRBackend(ABC):
id: str = "base"
display_name: str = "Base ASR"
# Accelerator families this backend can use, in preference order; always
# includes a fallback. Subset of {cuda, rocm, mps, xpu, cpu}. Mirrors the
# TTSBackend.gpu_compat contract so engine_routing.resolve_routing() can
# surface the effective device per host (no silent CPU fallback). The
# conservative default is CPU-only; subclasses declare what they really run
# on. (ROCm is intentionally NOT claimed yet for any ASR engine — see the
# per-engine notes; an unverified `rocm` claim would route ROCm hosts to a
# broken GPU path, strictly worse than the honest `cpu_fallback`.)
gpu_compat: tuple[str, ...] = ("cpu",)
@classmethod
@abstractmethod
@@ -123,10 +61,6 @@ class ASRBackend(ABC):
class WhisperXBackend(ASRBackend):
id = "whisperx"
display_name = "WhisperX (faster-whisper + wav2vec2 forced alignment)"
# CTranslate2 backend: CUDA fp16 or CPU int8 (see _pick_device). ROCm not
# claimed — CTranslate2 has no upstream HIP build, so a ROCm host honestly
# gets cpu_fallback rather than a false GPU promise.
gpu_compat = ("cuda", "cpu")
def __init__(self):
self._model_name = os.environ.get("ASR_MODEL_WHISPERX", "large-v3")
@@ -379,13 +313,10 @@ class WhisperXBackend(ASRBackend):
return None
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
import whisperx # used for whisperx.align() below
import whisperx
self._ensure_asr()
logger.info("whisperx transcribing %s (word_timestamps=%s)", audio_path, word_timestamps)
# Decode via OmniVoice's validated ffmpeg, NOT whisperx.load_audio's bare
# "ffmpeg" PATH lookup which yields [WinError 193] -> "no segments" on
# Windows (#479). Same 16 kHz mono s16le array whisperx expects.
audio = _decode_audio_16k_mono(audio_path)
audio = whisperx.load_audio(audio_path)
try:
result = self._asr.transcribe(audio)
except IndexError:
@@ -451,8 +382,6 @@ class WhisperXBackend(ASRBackend):
class FasterWhisperBackend(ASRBackend):
id = "faster-whisper"
display_name = "Faster-Whisper (CTranslate2 — Linux/Windows/macOS)"
# CTranslate2: CUDA or CPU (no upstream ROCm/HIP build — see WhisperX note).
gpu_compat = ("cuda", "cpu")
def __init__(self):
# Defaulting to the CTranslate2-converted large-v3 repo. Matches
@@ -568,7 +497,6 @@ _MLX_MODEL_TURBO = "mlx-community/whisper-large-v3-turbo"
class MLXWhisperBackend(ASRBackend):
id = "mlx-whisper"
display_name = "MLX Whisper (Apple Silicon CoreML)"
gpu_compat = ("mps", "cpu")
def __init__(self, model_name: str | None = None):
self._model_name = model_name or os.environ.get(
@@ -577,23 +505,14 @@ class MLXWhisperBackend(ASRBackend):
@classmethod
def is_available(cls) -> tuple[bool, str]:
# #390: shared platform gate FIRST — one rule for MLX-Audio + MLX-Whisper.
# Returns False on Linux/Windows/mac-Intel before any package import, so
# a stray mlx-whisper wheel never reports available or advertises `mps`.
from core.device_caps import mlx_supported
ok, why = mlx_supported()
if not ok:
return False, why
try:
import torch
if not (hasattr(torch.backends, "mps") and torch.backends.mps.is_available()):
return False, "Apple Silicon (MPS) not available."
import mlx_whisper # noqa: F401
return True, "ready"
# Catch OSError/RuntimeError too, not just ImportError: in a
# PyInstaller bundle mlx's native dylib/metallib can fail to load
# even when the package imports, raising OSError/RuntimeError. We must
# report unavailable (so the picker falls back) rather than crash the
# registry scan (Wave 4.4).
except (ImportError, OSError, RuntimeError) as e:
return False, f"mlx-whisper unavailable: {e}"
except ImportError as e:
return False, f"mlx-whisper not installed: {e}"
def transcribe(self, audio_path: str, *, word_timestamps: bool = True) -> dict:
import mlx_whisper
@@ -642,9 +561,6 @@ class MLXWhisperBackend(ASRBackend):
class PyTorchWhisperBackend(ASRBackend):
id = "pytorch-whisper"
display_name = "PyTorch Whisper (CUDA / CPU via transformers pipeline)"
# Pure transformers pipeline → runs wherever torch does (CUDA, MPS, CPU).
# ROCm-via-HIP would also work but is left unclaimed pending verification.
gpu_compat = ("cuda", "mps", "cpu")
def __init__(self, asr_pipe=None):
# Reuses the `_asr_pipe` attached to the TTS model when available.
@@ -717,11 +633,6 @@ class NeMoASRBackend(ASRBackend):
Requires NVIDIA GPU.
"""
id = "nemo-parakeet"
# CUDA-only: is_available() hard-fails without a GPU ("Parakeet TDT requires
# NVIDIA GPU (CUDA)"), so declaring a CPU path would be a false claim. On a
# CPU host this correctly resolves to routing_status="unavailable", matching
# is_available()=False (the matrix suppresses the routing badge there).
gpu_compat = ("cuda",)
display_name = "Parakeet TDT (NVIDIA NeMo — English SOTA)"
def __init__(self):
@@ -830,7 +741,6 @@ class MoonshineASRBackend(ASRBackend):
Great for live capture and CPU-only environments.
"""
id = "moonshine"
gpu_compat = ("cpu",) # edge/CPU-optimized by design
display_name = "Moonshine (edge-optimized, ONNX)"
def __init__(self):
@@ -975,7 +885,6 @@ class FunASRBackend(ASRBackend):
#182); WhisperX remains the cross-platform default.
"""
id = "funasr"
gpu_compat = ("cuda", "cpu") # FunASR: CUDA or CPU
display_name = "FunASR (SenseVoice — 50+ languages, all-in-one)"
def __init__(self):
@@ -1022,47 +931,7 @@ class FunASRBackend(ASRBackend):
pass
def _isolated_faster_whisper():
"""Lazy import so the subprocess_asr → subprocess_backend chain isn't
pulled in at registry definition time."""
from services.subprocess_asr import IsolatedFasterWhisperBackend
return IsolatedFasterWhisperBackend
class _LazyASRRegistry(dict):
"""Registry with one lazily-resolved entry (Wave 4.2). Mirrors the TTS
registry's lazy pattern so listing/selecting the crash-isolated ASR
backend doesn't import the subprocess stack unless it's used."""
_LAZY = {"faster-whisper-isolated": _isolated_faster_whisper}
def __contains__(self, key):
return dict.__contains__(self, key) or key in self._LAZY
def __getitem__(self, key):
if dict.__contains__(self, key):
return dict.__getitem__(self, key)
if key in self._LAZY:
cls = self._LAZY[key]()
self[key] = cls
return cls
raise KeyError(key)
def __iter__(self):
seen = set()
for k in dict.__iter__(self):
seen.add(k)
yield k
for k in self._LAZY:
if k not in seen:
yield k
def items(self):
for k in self:
yield k, self[k]
_REGISTRY: dict[str, type[ASRBackend]] = _LazyASRRegistry({
_REGISTRY: dict[str, type[ASRBackend]] = {
"whisperx": WhisperXBackend,
"faster-whisper": FasterWhisperBackend,
"mlx-whisper": MLXWhisperBackend,
@@ -1070,70 +939,18 @@ _REGISTRY: dict[str, type[ASRBackend]] = _LazyASRRegistry({
"nemo-parakeet": NeMoASRBackend,
"moonshine": MoonshineASRBackend,
"funasr": FunASRBackend,
# "faster-whisper-isolated": resolved lazily (crash-isolated subprocess).
})
# Short install hints surfaced as tooltips on the Settings → Engines UI
# (parity with tts_backend._INSTALL_HINTS).
_INSTALL_HINTS: dict[str, str] = {
"whisperx": "pip install whisperx (CTranslate2 + wav2vec2 alignment; CUDA or CPU)",
"faster-whisper": "pip install faster-whisper (CTranslate2; cross-platform, CUDA or CPU)",
"mlx-whisper": "pip install mlx-whisper (Apple Silicon only)",
"pytorch-whisper": "Bundled with transformers — no extra install (CUDA/MPS/CPU)",
"nemo-parakeet": "pip install nemo_toolkit[asr] (NVIDIA Parakeet; CUDA or CPU)",
"moonshine": "pip install useful-moonshine (edge/CPU-optimized ASR)",
"funasr": "pip install funasr (SenseVoiceSmall + FSMN-VAD; CUDA or CPU)",
}
# Most-recent failure per backend, so a transient probe error survives between
# Settings refreshes (parity with tts_backend._LAST_ERRORS).
_LAST_ERRORS: dict[str, str] = {}
def list_backends() -> list[dict]:
"""Enumerate every ASR backend with the **same 11-key shape as TTS** so the
Engine Compatibility Matrix renders all families uniformly.
Per-entry: id, display_name, available, reason (scrubbed), install_hint,
last_error, isolation_mode, gpu_compat, effective_device, routing_status,
routing_reason. A backend whose ``is_available()`` raises is reported
``available: false`` (never a 500), exactly like TTS.
"""
from core.device_caps import detect_host_caps
from core.scrub import scrub_text
from services.engine_routing import routing_fields
caps = detect_host_caps()
out: list[dict] = []
out = []
for bid, cls in _REGISTRY.items():
try:
ok, msg = cls.is_available()
except Exception as exc:
ok = False
msg = f"{type(exc).__name__}: {exc}"
logger.warning(
"asr list_backends: %s.is_available() raised — degrading "
"gracefully so the picker still renders: %s", bid, msg,
)
if ok:
_LAST_ERRORS.pop(bid, None)
else:
_LAST_ERRORS[bid] = scrub_text(msg)
isolation = "subprocess" if getattr(cls, "_is_subprocess_isolated", False) else "in-process"
gpu_compat = getattr(cls, "gpu_compat", ("cpu",))
ok, msg = cls.is_available()
out.append({
"id": bid,
"display_name": cls.display_name,
"available": ok,
# ASR previously emitted `reason` UNMASKED — scrub it now (closes a
# pre-existing token-leak gap, matching TTS's redaction guarantee).
"reason": None if ok else scrub_text(msg),
"install_hint": _INSTALL_HINTS.get(bid),
"last_error": _LAST_ERRORS.get(bid),
"isolation_mode": isolation,
"gpu_compat": list(gpu_compat),
**routing_fields(gpu_compat, caps),
"reason": None if ok else msg,
})
return out
@@ -1199,44 +1016,6 @@ def get_active_asr_backend(*, asr_pipe=None) -> ASRBackend:
return _REGISTRY[bid]()
def transcribe_reference(audio_path: str) -> str | None:
"""Transcribe a voice-clone reference clip with the active ASR backend.
Voice cloning without a user-supplied transcript used to fall through to
``OmniVoice.load_asr_model()`` a transformers ``pipeline()`` load of
whisper-large-v3-turbo that fails outright on transformers 5.3 (#308),
even when whisperx / faster-whisper / mlx-whisper are installed and
working. Route the reference transcript through the registry instead, so
the model-attached pipeline is only reached when it is genuinely the last
resort. Returns ``None`` on any failure callers pass ``ref_text=None``
through and the model's built-in fallback still gets its chance.
"""
try:
backend = get_active_asr_backend()
except Exception as e: # noqa: BLE001 — never let ASR break generation
logger.warning("transcribe_reference: no ASR backend available (%s)", e)
return None
if isinstance(backend, PyTorchWhisperBackend):
# The registry fell through to the model-attached pipeline; let the
# model load it lazily rather than constructing a second copy here.
return None
try:
result = backend.transcribe(audio_path, word_timestamps=False)
except Exception as e: # noqa: BLE001 — degrade to the model fallback
logger.warning(
"transcribe_reference: %s failed (%s) — deferring to the model's "
"built-in ASR fallback",
backend.id, e,
)
return None
result = result or {}
text = result.get("text") or " ".join(
(seg.get("text") or "").strip() for seg in result.get("segments", [])
)
text = (text or "").strip()
return text or None
_capture_backend: ASRBackend | None = None
-170
View File
@@ -1,170 +0,0 @@
"""Audiobook creator — chapterized long-form narration (parity Wave 5).
Turns a chapter-delimited script into a chapterized audiobook. This module is
the engine-agnostic core:
* ``parse_audiobook_script`` pure parser: Markdown ``# H1`` headings become
chapters; inline ``[voice:NAME]`` switches the narrator; ``[pause ]`` is
delegated to the existing :func:`omnivoice.utils.text.parse_pause_markers`
so audiobooks and single-shot synthesis share one pause dialect.
* ``synthesize_chapter`` orchestration: renders a chapter's spans through an
injected ``synth(text, voice_id) -> tensor`` callable (reusing the
``chunked_tts`` splitter + crossfade), stitching the inter-span silences.
Injecting the synth keeps this unit-testable with a stub backend (no torch
model, no GPU).
* ``build_chapter_ffmetadata`` / ``build_m4b_cmd`` pure builders for the
ffmpeg chapterized-m4b mux (FFMETADATA1 ``[CHAPTER]`` blocks + concat-demux
argv). The actual ffmpeg run lives in the (impure) caller.
Scope (first cut): plain chapter-delimited text/Markdown input. epub/pdf
ingestion, the streaming synth job + UI are deferred follow-ups.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Callable, Optional
@dataclass
class Span:
"""One contiguous run of text in a single voice, plus trailing silence.
``speed`` (when set) is the per-span rate passed to the engine Stories'
per-line speed slider rides through here so the shared server render honours
it the way the old client export did.
"""
voice_id: Optional[str]
text: str
pause_ms_after: int = 0
speed: Optional[float] = None
def to_dict(self) -> dict:
return {"voice_id": self.voice_id, "text": self.text,
"pause_ms_after": self.pause_ms_after, "speed": self.speed}
@dataclass
class Chapter:
title: str
spans: list[Span] = field(default_factory=list)
@property
def char_count(self) -> int:
return sum(len(s.text) for s in self.spans)
def to_dict(self) -> dict:
return {"title": self.title, "char_count": self.char_count,
"spans": [s.to_dict() for s in self.spans]}
@dataclass
class AudiobookPlan:
chapters: list[Chapter] = field(default_factory=list)
@property
def char_count(self) -> int:
return sum(c.char_count for c in self.chapters)
def to_dict(self) -> dict:
return {
"chapters": [c.to_dict() for c in self.chapters],
"chapter_count": len(self.chapters),
"char_count": self.char_count,
}
def parse_audiobook_script(text: str, *, default_voice: Optional[str] = None) -> AudiobookPlan:
"""Parse a chapter-delimited script into an :class:`AudiobookPlan`.
Thin wrapper over the canonical :func:`services.longform_parser.
parse_script_to_spans` (the single grammar source of truth, #27); wraps its
span dicts in the ``Span``/``Chapter``/``AudiobookPlan`` dataclasses so the
four router call sites and ``.to_dict()`` shape are unchanged.
"""
from services.longform_parser import parse_script_to_spans
chapters = [
Chapter(title=c["title"], spans=[Span(**s) for s in c["spans"]])
for c in parse_script_to_spans(text, default_voice=default_voice)
]
return AudiobookPlan(chapters=chapters)
def synthesize_chapter(
spans: list[Span],
synth: Callable[[str, Optional[str], Optional[float]], "object"],
sample_rate: int,
*,
crossfade_ms: int = 50,
lexicon: Optional[dict] = None,
):
"""Render a chapter's spans to one waveform via an injected ``synth``.
``synth(text, voice_id, speed)`` returns a 1-D float32 audio tensor for a
span of text in the given voice (``speed`` may be ``None`` for the engine
default). Long spans are split with the ``chunked_tts`` splitter and
crossfaded; inter-span ``pause_ms_after`` becomes silence. ``lexicon`` (when
given) respells each span's text before chunking so the engine pronounces
tricky words correctly; a ``None``/empty lexicon is a no-op pass-through.
Returns ``(audio_tensor, duration_seconds)``. torch + chunked_tts are
imported lazily so this module stays import-light for the pure parser path.
"""
import torch
from services.chunked_tts import concatenate_audio_chunks, split_text_into_chunks
from services.pronunciation import apply_lexicon
parts: list = []
for span in spans:
if span.text:
chunks = split_text_into_chunks(apply_lexicon(span.text, lexicon))
rendered = [synth(c, span.voice_id, span.speed) for c in chunks]
rendered = [r for r in rendered if r is not None and getattr(r, "numel", lambda: 0)()]
if len(rendered) == 1:
parts.append(rendered[0])
elif rendered:
parts.append(concatenate_audio_chunks(rendered, sample_rate, crossfade_ms=crossfade_ms))
if span.pause_ms_after > 0:
n = int(sample_rate * span.pause_ms_after / 1000.0)
if n > 0:
parts.append(torch.zeros(n, dtype=torch.float32))
if not parts:
return torch.zeros(0, dtype=torch.float32), 0.0
# Hard-concat spans + silences (crossfading silence would bleed the gap).
audio = parts[0] if len(parts) == 1 else concatenate_audio_chunks(parts, sample_rate, crossfade_ms=0)
return audio, audio.shape[-1] / float(sample_rate)
# ── ffmpeg / metadata builders ──────────────────────────────────────────────
#
# These now live in the shared ``longform_render`` core (Stories + Audiobook
# converge on one mux). The thin wrappers below preserve the original
# audiobook-only call sites/signatures; new callers should use
# ``longform_render`` directly to reach global metadata, cover art, loudness,
# and mp3 output.
from services.longform_render import ( # noqa: E402
build_concat_list,
build_ffmetadata,
build_render_cmd,
)
def build_chapter_ffmetadata(chapters: list[tuple[str, int]]) -> str:
"""Backward-compatible alias: chapters-only FFMETADATA (no global tags)."""
return build_ffmetadata(chapters)
def build_m4b_cmd(
ffmpeg: str,
concat_list_path: str,
metadata_path: str,
out_path: str,
*,
bitrate: str = "128k",
) -> list[str]:
"""Backward-compatible alias: a chapterized faststart m4b, no cover/loudness."""
return build_render_cmd(
ffmpeg, concat_list_path, metadata_path, out_path,
fmt="m4b", bitrate=bitrate,
)
-3
View File
@@ -151,9 +151,6 @@ async def generate_segments_batched(
# Raw: skip all DSP — return raw model output
return audio_out
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered = apply_mastering(audio_out, sample_rate=sr)
effect_chain = get_effect_chain(seg_effect_preset)
if effect_chain:
-174
View File
@@ -1,174 +0,0 @@
"""Chunked TTS generation utilities (Wave 1.2 — unlimited-length generation).
Adapted from voicebox (https://github.com/jamiepine/voicebox), MIT License,
Copyright (c) voicebox contributors. The concatenation half is reworked for
torch tensors (our inference helpers pass raw model output possibly
multi-channel to the effect chain), and the sample rate comes from the
engine's declared rate rather than the first chunk (fixes a latent upstream
bug where a mid-run rate change was silently ignored).
Splits long text into sentence-boundary chunks and joins the per-chunk audio
with a short crossfade. Pure functions the generation loop itself lives in
``api/routers/generation.py`` next to the existing ``[pause]`` span stitcher,
so this module stays unit-testable without a model.
Short text (<= max_chunk_chars) never reaches this module's concat path; the
callers keep their unchanged single-shot fast path.
"""
from __future__ import annotations
import logging
import re
from typing import List
logger = logging.getLogger("omnivoice.chunked_tts")
# Default chunk size in characters. 0 disables chunking entirely.
DEFAULT_MAX_CHUNK_CHARS = 800
# Default crossfade between chunks. 0 = hard cut.
DEFAULT_CROSSFADE_MS = 50
# Common abbreviations that should NOT be treated as sentence endings.
# Lowercase for case-insensitive matching.
_ABBREVIATIONS = frozenset({
"mr", "mrs", "ms", "dr", "prof", "sr", "jr", "st", "ave", "blvd",
"inc", "ltd", "corp", "dept", "est", "approx", "vs", "etc",
"e.g", "i.e", "a.m", "p.m", "u.s", "u.s.a", "u.k",
})
# Inline bracket tags (paralinguistic tags like [laugh]; our own
# [pause 300ms] markers). The splitter must never cut inside one.
_BRACKET_TAG_RE = re.compile(r"\[[^\]]*\]")
def split_text_into_chunks(text: str, max_chars: int = DEFAULT_MAX_CHUNK_CHARS) -> List[str]:
"""Split *text* at natural boundaries into chunks of at most *max_chars*.
Priority: sentence-end (``.!?`` not after an abbreviation/decimal and not
inside brackets, plus fullwidth equivalents) -> clause boundary
(``;:,`` / em dash) -> whitespace -> hard cut that avoids splitting a
``[tag]``.
"""
text = text.strip()
if not text:
return []
if max_chars <= 0 or len(text) <= max_chars:
return [text]
chunks: List[str] = []
remaining = text
while remaining:
remaining = remaining.lstrip()
if not remaining:
break
if len(remaining) <= max_chars:
chunks.append(remaining)
break
segment = remaining[:max_chars]
split_pos = _find_last_sentence_end(segment)
if split_pos == -1:
split_pos = _find_last_clause_boundary(segment)
if split_pos == -1:
split_pos = segment.rfind(" ")
if split_pos == -1:
split_pos = _safe_hard_cut(segment, max_chars)
chunk = remaining[: split_pos + 1].strip()
if chunk:
chunks.append(chunk)
remaining = remaining[split_pos + 1:]
return chunks
def _find_last_sentence_end(text: str) -> int:
"""Index of the last sentence-ending punctuation, or -1.
Skips periods after common abbreviations and decimals, anything inside
a bracket tag, and also recognizes fullwidth sentence punctuation
(ideographic full stop / fullwidth ! and ?) for no-space scripts.
"""
best = -1
for m in re.finditer(r"[.!?](?:\s|$)", text):
pos = m.start()
if text[pos] == ".":
word_start = pos - 1
while word_start >= 0 and text[word_start].isalpha():
word_start -= 1
word = text[word_start + 1: pos].lower()
if word in _ABBREVIATIONS:
continue
if word_start >= 0 and text[word_start].isdigit():
continue
if _inside_bracket_tag(text, pos):
continue
best = pos
# Fullwidth sentence enders (ideographic full stop, fullwidth !, ?)
# written as escapes to keep the repo's no-literal-CJK gate clean.
for m in re.finditer("[\u3002\uff01\uff1f]", text):
if m.start() > best:
best = m.start()
return best
def _find_last_clause_boundary(text: str) -> int:
best = -1
for m in re.finditer(r"[;:,—](?:\s|$)", text):
if _inside_bracket_tag(text, m.start()):
continue
best = m.start()
return best
def _inside_bracket_tag(text: str, pos: int) -> bool:
for m in _BRACKET_TAG_RE.finditer(text):
if m.start() < pos < m.end():
return True
return False
def _safe_hard_cut(segment: str, max_chars: int) -> int:
cut = max_chars - 1
for m in _BRACKET_TAG_RE.finditer(segment):
if m.start() < cut < m.end():
return m.start() - 1 if m.start() > 0 else cut
return cut
def concatenate_audio_chunks(chunks: list, sample_rate: int,
crossfade_ms: int = DEFAULT_CROSSFADE_MS):
"""Join per-chunk waveforms with a linear crossfade on the sample axis.
``chunks`` are torch tensors as returned by the engine (1-D, or N-D with
samples on the last axis matching what ``_render_with_pauses`` handles).
Crossfade overlap is clamped to the shorter neighbor; ``crossfade_ms=0``
is a hard concat.
"""
import torch
chunks = [c for c in chunks if c is not None and c.shape[-1] > 0]
if not chunks:
return torch.zeros(1, dtype=torch.float32)
if len(chunks) == 1:
return chunks[0]
crossfade_samples = int(sample_rate * crossfade_ms / 1000)
result = chunks[0]
for chunk in chunks[1:]:
chunk = chunk.to(device=result.device, dtype=result.dtype)
overlap = min(crossfade_samples, result.shape[-1], chunk.shape[-1])
if overlap > 0:
fade_out = torch.linspace(1.0, 0.0, overlap, dtype=result.dtype, device=result.device)
fade_in = torch.linspace(0.0, 1.0, overlap, dtype=result.dtype, device=result.device)
blended = result[..., -overlap:] * fade_out + chunk[..., :overlap] * fade_in
result = torch.cat([result[..., :-overlap], blended, chunk[..., overlap:]], dim=-1)
else:
result = torch.cat([result, chunk], dim=-1)
return result
+40 -19
View File
@@ -13,8 +13,7 @@ What's here
* **Content-hash cache lookup** `compute_file_hash`, `find_cached_job`.
* **Safe path resolution** `safe_job_dir`.
* **Process lifecycle** ffmpeg/demucs subprocess tracking + `kill_job_procs`
so `POST /dub/abort/{id}` can tear down in-flight work (implemented in
`services.proc_registry`, re-exported here for compatibility).
so `POST /dub/abort/{id}` can tear down in-flight work.
* **SSE helpers** `sse_event`, `prep_event`.
What stays in the router
@@ -45,17 +44,6 @@ from core.config import DUB_DIR
from fastapi import HTTPException
from services.ffmpeg_utils import find_ffmpeg, find_ffprobe, _get_semaphore, _spawn_with_retry
from services.model_manager import get_best_device
# Process lifecycle moved to its own leaf module so ffmpeg_utils can import
# it at module top (no dub_pipeline ↔ ffmpeg_utils cycle). Re-exported here —
# dub_core and tests still alias these names through this module.
from services.proc_registry import ( # noqa: F401 — re-exports
_active_procs,
_active_procs_lock,
has_active_procs,
kill_job_procs,
register_proc,
unregister_proc,
)
from core.db import db_conn
from core import event_bus
from core import failure
@@ -68,6 +56,8 @@ logger = logging.getLogger("omnivoice.dub_pipeline")
_dub_jobs: dict[str, dict] = {}
_dub_jobs_lock = threading.Lock()
_active_procs: dict[str, list] = {}
_active_procs_lock = threading.Lock()
_DUB_DIR_REAL = os.path.realpath(DUB_DIR)
_HASH_BUF_SIZE = 1 << 18 # 256 KB chunks for hashing
@@ -148,8 +138,42 @@ def find_cached_job(content_hash: str, exclude_job_id: str) -> Optional[dict]:
# ── Process lifecycle ───────────────────────────────────────────────────────
# register_proc / unregister_proc / kill_job_procs / has_active_procs live in
# services.proc_registry (imported + re-exported above).
def register_proc(job_id: str, proc) -> None:
"""Track an in-flight subprocess so /dub/abort can kill it."""
with _active_procs_lock:
_active_procs.setdefault(job_id, []).append(proc)
def unregister_proc(job_id: str, proc) -> None:
with _active_procs_lock:
lst = _active_procs.get(job_id)
if lst and proc in lst:
lst.remove(proc)
if lst is not None and not lst:
_active_procs.pop(job_id, None)
def kill_job_procs(job_id: str) -> None:
"""Kill every subprocess still running under a given job id. Idempotent."""
with _active_procs_lock:
procs = list(_active_procs.get(job_id, []))
for proc in procs:
try:
if proc.returncode is None:
proc.kill()
except ProcessLookupError:
pass
except Exception as e:
logger.warning("Failed to kill subprocess for %s: %s", job_id, e)
with _active_procs_lock:
_active_procs.pop(job_id, None)
def has_active_procs(job_id: str) -> bool:
with _active_procs_lock:
return bool(_active_procs.get(job_id))
# ── Job state (in-memory + SQLite fallback) ────────────────────────────────
@@ -171,10 +195,7 @@ def get_job(job_id: str) -> Optional[dict]:
_dub_jobs[job_id] = job
return job
except json.JSONDecodeError as e:
# job_id arrives from request paths — strip newlines so a crafted
# id can't forge extra log lines (py/log-injection).
safe_id = str(job_id).replace("\r", "").replace("\n", "")
logger.error("Failed to decode dub_history.job_data for %s: %s", safe_id, e)
logger.error("Failed to decode dub_history.job_data for %s: %s", job_id, e)
return None
-118
View File
@@ -1,118 +0,0 @@
"""Second-pass ASR quality control for dubs (Wave 3.3 / Spec 5).
After a dub is generated, re-recognize the synthetic audio and compare what
the ASR *heard* against what we asked the TTS to *say*. Where the two drift
apart, the line is flagged for the user to verify turning subtitle timing
and pronunciation from "trusted math" into "measured truth", and doubling as
an automatic dub-quality check.
Design delta from pyvideotrans (whose second pass lets recognized text
*replace* the subtitles wholesale): we keep the GENERATED text authoritative
for content and use the second pass for *measurement* timing + a drift
score that feeds the incremental re-dub loop, never silently overwriting the
translation.
Pure functions here (no ASR, no I/O) so the scoring is unit-testable; the
pipeline stage that runs the ASR pass lives in the dub router.
"""
from __future__ import annotations
import re
from dataclasses import dataclass
def _tokens(text: str) -> list[str]:
"""Lowercase word tokens, punctuation stripped — the unit drift is scored
in. Script-agnostic: for no-space scripts each character is a token, which
still gives a sensible edit-distance ratio."""
text = (text or "").lower().strip()
if not text:
return []
words = re.findall(r"\w+", text, flags=re.UNICODE)
return words or list(text.replace(" ", ""))
def _edit_distance(a: list[str], b: list[str]) -> int:
"""Levenshtein distance between two token lists (iterative, O(len(a)*len(b))
time, O(len(b)) space)."""
if not a:
return len(b)
if not b:
return len(a)
prev = list(range(len(b) + 1))
for i, ta in enumerate(a, 1):
cur = [i]
for j, tb in enumerate(b, 1):
cost = 0 if ta == tb else 1
cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + cost))
prev = cur
return prev[-1]
def word_error_rate(reference: str, hypothesis: str) -> float:
"""Normalized token edit distance in [0.0, 1.0+].
0.0 = the ASR heard exactly the target text. ~1.0 = entirely different.
Can exceed 1.0 when the hypothesis is much longer than the reference
(insertions); callers clamp/threshold as needed. An empty reference with a
non-empty hypothesis scores 1.0 (everything is an insertion)."""
ref = _tokens(reference)
hyp = _tokens(hypothesis)
if not ref and not hyp:
return 0.0
if not ref:
return 1.0
return _edit_distance(ref, hyp) / len(ref)
@dataclass
class SegmentQC:
seg_id: str
target_text: str
recognized_text: str
drift: float # word_error_rate(target, recognized)
flagged: bool # drift >= threshold
new_start: float | None # measured onset from the dubbed-audio recognition
new_end: float | None
def _overlap(a0: float, a1: float, b0: float, b1: float) -> float:
return max(0.0, min(a1, b1) - max(a0, b0))
def score_dub(
dub_segments: list[dict],
recognized: list[dict],
*,
drift_threshold: float = 0.5,
seg_ids: list | None = None,
) -> list[SegmentQC]:
"""Match the second-pass recognition to the dub segments and score drift.
``dub_segments`` are the segments we generated (each {start, end, text});
``recognized`` are the ASR result segments on the dubbed audio (each
{start, end, text}). Each dub segment is matched to the recognized
segment(s) it overlaps in time; their text is concatenated as the
hypothesis and scored against the dub segment's ``text``. The recognized
span's bounds become the measured start/end (subtitle-timing truth).
"""
results: list[SegmentQC] = []
for i, seg in enumerate(dub_segments):
sid = str(seg_ids[i]) if (seg_ids and i < len(seg_ids)) else str(seg.get("id", i))
s0, s1 = float(seg.get("start", 0.0)), float(seg.get("end", 0.0))
hits = [r for r in recognized if _overlap(s0, s1, float(r.get("start", 0.0)), float(r.get("end", 0.0))) > 0]
hyp = " ".join((r.get("text") or "").strip() for r in hits).strip()
drift = word_error_rate(seg.get("text", ""), hyp)
new_start = min((float(r.get("start", 0.0)) for r in hits), default=None)
new_end = max((float(r.get("end", 0.0)) for r in hits), default=None)
results.append(SegmentQC(
seg_id=sid,
target_text=(seg.get("text") or "").strip(),
recognized_text=hyp,
drift=round(drift, 3),
flagged=drift >= drift_threshold,
new_start=new_start,
new_end=new_end,
))
return results
+1 -97
View File
@@ -24,78 +24,6 @@ logger = logging.getLogger("omnivoice.engine_env")
_TORCH_COMPILE_KEY = "perf.torch_compile_disabled"
# #278: explicit opt-in override — set to 1/true to attempt torch.compile even
# when the GPU's compute capability is not in this PyTorch build's arch list
# (e.g. a brand-new architecture running through PTX forward-compat).
_FORCE_COMPILE_ENV = "OMNIVOICE_FORCE_TORCH_COMPILE"
# #278: set (with a reason) the first time torch.compile — or *running* the
# compiled model — fails at runtime in this process. Once set, every later
# load in the same session goes straight to eager instead of re-tripping the
# same Dynamo/Inductor/Triton failure.
_compile_runtime_failure: Optional[str] = None
def mark_compile_runtime_failure(reason: str) -> None:
"""Record that torch.compile (or compiled execution) failed at runtime.
Called by ``services.model_manager`` when compilation raises, or when a
generation through the compiled model dies inside the Dynamo / Inductor /
Triton stack (#278). Disables compile for the rest of the process — eager
mode from here on; the next app restart probes again.
"""
global _compile_runtime_failure
_compile_runtime_failure = reason or "unknown torch.compile runtime failure"
logger.warning(
"torch.compile disabled for this session after a runtime failure: %s",
_compile_runtime_failure,
)
def _force_compile_requested() -> bool:
value = os.environ.get(_FORCE_COMPILE_ENV, "")
return value.strip().lower() in {"1", "true", "yes", "on"}
def _cuda_arch_supported_for_compile() -> "tuple[bool, str]":
"""Check the GPU's compute capability against this torch build's arch list.
New GPU architectures (e.g. Blackwell sm_120, issue #278) routinely break
torch.compile/Triton before upstream support lands: the eager model runs
via PTX forward-compat, but Inductor/Triton kernel compilation targets the
new arch directly and fails mid-generation. If the device's ``sm_XY`` tag
is absent from ``torch.cuda.get_arch_list()`` we treat compile as
unsupported and use eager.
Returns ``(supported, reason)``. Fails open any probe error returns
``(True, "")`` so a weird torch build never silently loses the
optimization (the runtime fallback in model_manager still protects
generation).
"""
try:
import torch
if not torch.cuda.is_available():
return True, ""
major, minor = torch.cuda.get_device_capability(0)
arch_list = list(getattr(torch.cuda, "get_arch_list", lambda: [])() or [])
if not arch_list:
return True, ""
sm_tag = f"sm_{major}{minor}"
if sm_tag in arch_list or f"compute_{major}{minor}" in arch_list:
return True, ""
try:
device_name = torch.cuda.get_device_name(0)
except Exception:
device_name = "GPU"
return False, (
f"{device_name} (compute capability {major}.{minor} / {sm_tag}) is not "
f"in this PyTorch build's supported arch list ({', '.join(arch_list)})"
)
except Exception:
logger.debug("CUDA arch probe for torch.compile failed; assuming supported", exc_info=True)
return True, ""
def should_torch_compile(device: str) -> bool:
"""Decide whether to apply ``torch.compile`` to an in-process model.
@@ -106,14 +34,9 @@ def should_torch_compile(device: str) -> bool:
- device == "cuda" (compile only helps the CUDA path here),
- Triton importable (``find_spec`` the cross-platform gate that closes
#65; no Windows wheel ⇒ skip ⇒ eager),
- the user has NOT set the ``perf.torch_compile_disabled`` escape hatch,
- compile has NOT already failed at runtime in this process (#278),
- the GPU's compute capability is in this torch build's arch list (#278)
overridable via ``OMNIVOICE_FORCE_TORCH_COMPILE=1``.
- the user has NOT set the ``perf.torch_compile_disabled`` escape hatch.
Returns False ( eager mode) on any of those, logging the reason at INFO.
torch.compile is an optimization, never a requirement generation must
always work without it.
"""
if device != "cuda":
return False
@@ -128,25 +51,6 @@ def should_torch_compile(device: str) -> bool:
return False
except Exception:
logger.exception("should_torch_compile: settings read failed; proceeding")
if _compile_runtime_failure is not None:
logger.info(
"torch.compile skipped: failed earlier this session (%s) — using eager mode.",
_compile_runtime_failure,
)
return False
supported, reason = _cuda_arch_supported_for_compile()
if not supported:
if _force_compile_requested():
logger.warning(
"torch.compile forced via %s=1 despite: %s", _FORCE_COMPILE_ENV, reason,
)
return True
logger.info(
"torch.compile skipped: %s — using eager mode. "
"(Set %s=1 to attempt compile anyway.)",
reason, _FORCE_COMPILE_ENV,
)
return False
return True
-156
View File
@@ -1,156 +0,0 @@
"""Pure, host-aware routing resolver — maps an engine's declared ``gpu_compat``
against the cached host capabilities to "where will this engine *actually* run
on this machine, and is that a problem the user should hear about?"
No model load, no probe (the caller passes the cached ``HostCaps``), no I/O.
Deterministic and byte-identical for a given ``(gpu_compat, HostCaps)`` across
macOS/Windows/Linux that cross-OS determinism is the whole point of the
no-silent-fallback contract.
Reason strings are author-controlled English (interpolating only family/device
names) but are **still** scrubbed by the caller (``core.scrub.scrub_text``)
before serialization, because an interpolated ``device_name`` or probe note can
carry a home path.
"""
from __future__ import annotations
from typing import Literal, TypedDict
from core.device_caps import (
DIRECTML_MARKER,
KERNEL_RISK_MARKER,
HostCaps,
)
RoutingStatus = Literal["accelerated", "cpu_fallback", "cpu_only", "unavailable", "n/a"]
class RoutingResult(TypedDict):
effective_device: str # a DeviceFamily value or "cpu"
routing_status: RoutingStatus # resolve_routing never emits "n/a" (LLM-only)
routing_reason: str | None # raw, pre-scrub
def _caveat(caps: HostCaps) -> str | None:
"""A kernel-risk caveat string for an otherwise-accelerated host, or None.
Advisory notes (multi-GPU, VRAM-query-failed, DirectML) never qualify."""
for note in caps.notes:
if KERNEL_RISK_MARKER in note:
return f"{caps.family.upper()} selected, but: {note}"
return None
def resolve_routing(gpu_compat: tuple[str, ...], caps: HostCaps) -> RoutingResult:
"""Resolve the effective device + status for an engine on this host.
Rules are evaluated in order; the first match wins (see spec §2)."""
targets = tuple(gpu_compat or ())
fam = caps.family
# 1. Empty compat — reserved for LLM (which never calls this). Defensive.
if not targets:
return {
"effective_device": "cpu",
"routing_status": "cpu_only",
"routing_reason": "engine declares no compute targets",
}
# 2. Host accelerator is one the engine supports → accelerated.
if fam != "cpu" and fam in targets:
return {
"effective_device": fam,
"routing_status": "accelerated",
"routing_reason": _caveat(caps),
}
# 3. Host has an accelerator the engine lacks, but engine supports cpu
# → the no-silent-fallback signal.
if fam != "cpu" and "cpu" in targets:
if fam == "rocm" and "cuda" in targets and "rocm" not in targets:
reason = "declares CUDA only; ROCm not in its compat set"
else:
reason = f"engine has no {fam.upper()} path; running on CPU"
return {
"effective_device": "cpu",
"routing_status": "cpu_fallback",
"routing_reason": reason,
}
# 4. Genuine CPU-only host (or DirectML, which the probe reports as cpu)
# and engine supports cpu → benign; must not warn or block.
if fam == "cpu" and "cpu" in targets:
reason = None
for note in caps.notes:
if DIRECTML_MARKER in note:
reason = (
"DirectML GPU present; engine routes via torch CPU path "
"(DirectML acceleration not wired into routing)"
)
break
return {
"effective_device": "cpu",
"routing_status": "cpu_only",
"routing_reason": reason,
}
# 5. Engine needs an accelerator this host lacks and has no cpu path.
first = targets[0]
return {
"effective_device": first,
"routing_status": "unavailable",
"routing_reason": f"requires {', '.join(targets)}; this host has {fam}",
}
def routing_notice(result: RoutingResult) -> tuple[str, str | None] | None:
"""`(status, reason)` when a synth-time notice SHOULD be surfaced to the
user, else `None`. Surfaced for `cpu_fallback` (always) and for
`accelerated` ONLY when it carries a driver/arch caveat reason everything
else (`cpu_only`, clean `accelerated`, `n/a`) is benign and stays silent."""
st = result["routing_status"]
if st == "cpu_fallback" or (st == "accelerated" and result["routing_reason"]):
return (st, result["routing_reason"])
return None
def header_safe_reason(reason: str | None) -> str | None:
"""A routing reason made safe for an HTTP header value: scrubbed, then
ASCII-sanitized (headers are latin-1; a non-ASCII device name would 500 the
response otherwise), **control characters stripped** (a CR/LF could split
the header / inject a new one), and length-capped at 256. Returns None for
an empty reason. No regex `.encode`/membership only (CodeQL-clean)."""
if not reason:
return None
from core.scrub import scrub_text
ascii_only = scrub_text(reason).encode("ascii", "ignore").decode("ascii")
# Drop ASCII control chars (0x00-0x1F + DEL 0x7F) — incl. CR/LF, so the
# value can never break out of its header line.
cleaned = "".join(c for c in ascii_only if 0x20 <= ord(c) < 0x7F)
return cleaned[:256] or None
def routing_fields(gpu_compat: tuple[str, ...], caps: HostCaps) -> dict:
"""The three serialization-ready routing keys for a ``list_backends`` entry.
Resolves routing and applies the redaction contract: ``routing_reason`` is
scrubbed via ``core.scrub.scrub_text`` only when truthy, so a ``None`` reason
serializes as JSON ``null`` (NOT ``""`` ``scrub_text(None)`` would coerce
to ``""``). Used by tts/asr ``list_backends`` so the scrub rule lives in one
place. (LLM emits its own literal ``network``/``n/a``/``null`` fields and
does NOT call this.)
"""
from core.scrub import scrub_text
r = resolve_routing(tuple(gpu_compat or ()), caps)
reason = r["routing_reason"]
return {
"effective_device": r["effective_device"],
"routing_status": r["routing_status"],
"routing_reason": scrub_text(reason) if reason else None,
}
__all__ = [
"RoutingStatus", "RoutingResult", "resolve_routing", "routing_fields",
"routing_notice", "header_safe_reason",
]
+10 -203
View File
@@ -5,10 +5,6 @@ import os
import shutil
import subprocess
# Leaf module (stdlib-only) — safe to import at module top, unlike
# services.dub_pipeline which imports this module and would cycle.
from services.proc_registry import register_proc, unregister_proc
logger = logging.getLogger("omnivoice.api")
# Cap concurrent ffmpeg jobs so macOS posix_spawn can't hit EAGAIN under load.
@@ -23,36 +19,6 @@ def _get_semaphore() -> asyncio.Semaphore:
return _FFMPEG_SEMAPHORE
# Candidate paths that exist but won't run (validated once per process).
# Windows users hit this as `[WinError 193] %1 is not a valid Win32
# application` (#360/#361/#362): a corrupt/wrong-arch imageio-ffmpeg
# download or a WindowsApps alias stub passes `os.path.isfile` / `which`
# but explodes at spawn. Probe each candidate with `-version` and fall
# through to the next source instead of returning a time bomb.
_BINARY_OK: dict[str, bool] = {}
def _binary_runs(path: str) -> bool:
cached = _BINARY_OK.get(path)
if cached is not None:
return cached
try:
subprocess.run(
[path, "-version"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
timeout=10, check=False,
)
ok = True
except (OSError, subprocess.TimeoutExpired, subprocess.SubprocessError) as e:
logger.warning(
"Rejecting non-runnable ffmpeg/ffprobe candidate %s: %s",
os.path.basename(str(path)), e,
)
ok = False
_BINARY_OK[path] = ok
return ok
def find_ffmpeg():
"""Locate an ffmpeg binary.
@@ -67,15 +33,15 @@ def find_ffmpeg():
env_path = os.environ.get("FFMPEG_PATH")
if env_path:
resolved = shutil.which(env_path)
if resolved and _binary_runs(resolved):
if resolved:
return resolved
# 2. imageio-ffmpeg bundled static binary
try:
import imageio_ffmpeg
candidate = imageio_ffmpeg.get_ffmpeg_exe()
if candidate and os.path.isfile(candidate) and _binary_runs(candidate):
if candidate and os.path.isfile(candidate):
return candidate
logger.debug("imageio_ffmpeg binary not usable at %s", candidate)
logger.debug("imageio_ffmpeg binary not found at %s", candidate)
except Exception as e:
logger.debug("imageio_ffmpeg unavailable: %s", e)
# 3. Well-known system paths + PATH lookup
@@ -88,10 +54,9 @@ def find_ffmpeg():
"ffmpeg",
]
for path in common:
resolved = shutil.which(path)
if resolved and _binary_runs(resolved):
return resolved
logger.warning("ffmpeg not found (or not runnable) in env, imageio, or system PATH")
if shutil.which(path):
return path
logger.warning("ffmpeg not found in env, imageio, or system PATH")
return None
@@ -115,14 +80,14 @@ def resolve_ffprobe() -> str | None:
continue
# The env var may carry either an absolute path to a file OR a bare
# command name (legacy). Accept both shapes — file first.
if os.path.isfile(path) and _binary_runs(path):
if os.path.isfile(path):
return path
resolved = shutil.which(path)
if resolved and _binary_runs(resolved):
if resolved:
return resolved
system_probe = shutil.which("ffprobe")
if system_probe and _binary_runs(system_probe):
if system_probe:
return system_probe
return None
@@ -241,168 +206,16 @@ async def _spawn_with_retry(cmd, **kwargs):
raise last_err if last_err else RuntimeError("spawn failed")
def _atempo_chain(ratio: float) -> str:
"""Build an `atempo=…,atempo=…` filter chain for arbitrary ratios.
ffmpeg's atempo filter is limited to [0.5, 2.0] per stage. Chaining
multiple stages multiplies the effective ratio while keeping each
individual stage inside the well-behaved range. Pitch is preserved
(WSOLA-style time-domain stretching). ratio > 1 speeds up, < 1
slows down.
"""
stages: list[str] = []
remaining = ratio
while remaining > 2.0:
stages.append("atempo=2.0")
remaining /= 2.0
while remaining < 0.5:
stages.append("atempo=0.5")
remaining /= 0.5
stages.append(f"atempo={remaining:.6f}")
return ",".join(stages)
async def _pitch_preserving_stretch(wav, target_samples: int, sr: int):
"""Time-stretch a (1, samples) tensor to `target_samples` while
preserving pitch, by piping the audio through `ffmpeg atempo`.
Async so it never blocks the event loop: it's awaited from the dub
generate `_stream` generator, and each ffmpeg call is ~50-100 ms a
synchronous ``subprocess.run`` here froze health-checks / SSE / every
concurrent request for the whole multi-segment job.
Returns a (1, target_samples) tensor on the same device as input.
Raises RuntimeError when ffmpeg fails callers should fall back to
naive linear interpolation, accepting the pitch shift, to ensure the
output isn't silent.
"""
# Lazy imports keep this module importable in torch-free contexts
# (setup scripts, smoke probes) — only the stretch path needs them.
import numpy as np
import torch
wl = int(wav.shape[-1])
if target_samples <= 0 or wl == target_samples:
return wav
ratio = wl / target_samples
filter_str = _atempo_chain(ratio)
# Mono float32 via stdin → ffmpeg → stdout. One subprocess per segment,
# run off the event loop so concurrent requests stay responsive.
arr = wav.detach().cpu().to(torch.float32).numpy().reshape(-1).astype(np.float32, copy=False)
proc = await spawn_subprocess(
find_ffmpeg(), "-hide_banner", "-loglevel", "error", "-y",
"-f", "f32le", "-ar", str(sr), "-ac", "1", "-i", "pipe:0",
"-af", filter_str,
"-f", "f32le", "-ar", str(sr), "-ac", "1", "pipe:1",
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, stderr = await proc.communicate(input=arr.tobytes())
if proc.returncode != 0 or not stdout:
raise RuntimeError(
(stderr.decode(errors="replace") or "atempo failed")[:200]
)
out_arr = np.frombuffer(stdout, dtype=np.float32)
# atempo rarely lands exactly on the integer sample count, so
# pad/trim to the requested slot length.
if len(out_arr) < target_samples:
pad = np.zeros(target_samples - len(out_arr), dtype=np.float32)
out_arr = np.concatenate([out_arr, pad])
elif len(out_arr) > target_samples:
out_arr = out_arr[:target_samples]
return torch.from_numpy(out_arr.copy()).unsqueeze(0).to(wav.device)
async def probe_duration(path: str) -> float | None:
"""Return a media file's duration in seconds via ffprobe, or None.
Used by the Smart Fit pipeline to sanity-check source/track lengths
without loading the media. Never raises probing is best-effort.
"""
ffprobe = find_ffprobe()
if not ffprobe or not os.path.isfile(path):
return None
try:
proc = await spawn_subprocess(
ffprobe, "-v", "error",
"-show_entries", "format=duration",
"-of", "default=noprint_wrappers=1:nokey=1",
path,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, _ = await proc.communicate()
if proc.returncode != 0:
return None
return float(stdout.decode().strip())
except Exception as e:
logger.debug("probe_duration failed for %s: %s", os.path.basename(str(path)), e)
return None
async def probe_frame_rates(path: str) -> "tuple[str, str] | None":
"""Return (r_frame_rate, avg_frame_rate) strings for the first video
stream (e.g. ``("30000/1001", "2997/100")``), or None on any failure.
A mismatch between the two is the practical VFR signature used by the
Smart Fit retime pipeline to decide whether to normalise with ``fps=``
before trim/setpts. Never raises probing is best-effort.
"""
ffprobe = find_ffprobe()
if not ffprobe or not os.path.isfile(path):
return None
try:
proc = await spawn_subprocess(
ffprobe, "-v", "error",
"-select_streams", "v:0",
"-show_entries", "stream=r_frame_rate,avg_frame_rate",
"-of", "csv=p=0",
path,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, _ = await proc.communicate()
if proc.returncode != 0:
return None
parts = stdout.decode().strip().split(",")
if len(parts) < 2:
return None
return parts[0].strip(), parts[1].strip()
except Exception as e:
logger.debug("probe_frame_rates failed for %s: %s", os.path.basename(str(path)), e)
return None
async def run_ffmpeg(cmd, timeout: float = 1800.0, capture: bool = True,
job_id: "str | None" = None):
async def run_ffmpeg(cmd, timeout: float = 1800.0, capture: bool = True):
"""Run an ffmpeg subprocess with concurrency cap, timeout, and proper cleanup.
Returns (returncode, stdout_bytes, stderr_bytes). Raises asyncio.TimeoutError
on hard timeout (after killing + reaping the process).
``job_id`` (optional) registers the process with the dub pipeline's
process tracker (``services.proc_registry``) so ``/dub/abort`` can kill
long export encodes (used by the Smart Fit batched retime).
Path-injection note: every filesystem path placed in ``cmd`` by callers
is realpath-normalised and containment-checked against its workspace
root (e.g. DUB_DIR) at the call site before the argv is assembled
see api.routers.dub_export and services.video_retime.
"""
stdout = asyncio.subprocess.PIPE if capture else asyncio.subprocess.DEVNULL
stderr = asyncio.subprocess.PIPE
async with _get_semaphore():
proc = await _spawn_with_retry(cmd, stdout=stdout, stderr=stderr)
if job_id:
try:
register_proc(job_id, proc)
except Exception as e:
# Newline-strip the id inline — it can originate from a path
# param, and the log stream must stay one-event-per-line.
logger.debug("register_proc failed for %s: %s",
job_id.replace("\n", " ").replace("\r", " "), e)
try:
try:
out, err = await asyncio.wait_for(proc.communicate(), timeout=timeout)
@@ -418,12 +231,6 @@ async def run_ffmpeg(cmd, timeout: float = 1800.0, capture: bool = True,
raise
return proc.returncode, out, err
finally:
if job_id:
try:
unregister_proc(job_id, proc)
except Exception as e:
logger.debug("unregister_proc failed for %s: %s",
job_id.replace("\n", " ").replace("\r", " "), e)
# Guarantee reaping — prevents zombie pileup under timeouts or errors.
if proc.returncode is None:
try:
-211
View File
@@ -1,211 +0,0 @@
"""Smart Fit planner — dub-length fitting v2, Phase A.
Pure planning functions for the ``smart_fit`` timing strategy: given the
original segment timeline and the *natural-rate* duration of each dubbed
segment's TTS audio, decide per segment how to reconcile the two by
splitting the burden between a mild pitch-preserving audio speed-up and a
mild per-segment video slow-down.
Clean-room note: this is a reimplementation from a *published description*
of the audio-speedup + video-slowdown fitting approach (see
docs/competitive-analysis.md, "Dub-length fitting"). No GPL source was
consulted.
Algorithm per segment (defaults in :class:`FitParams`):
1. **Slack absorption** the usable slot extends past the segment's
original end into the silent gap before the next segment, keeping a
small ``gap_guard_s`` clear of the next onset (the last segment may run
to the end of the video). ``need = natural_dur / slot``.
2. ``need <= 1.0`` fits as-is; nothing to do.
3. ``1.0 < need <= max_audio_only_rate`` audio-only speed-up at exactly
``need`` (imperceptible up to ~1.2×).
4. ``need > max_audio_only_rate`` geometric 50/50 split:
``audio_rate = min(sqrt(need), audio_rate_cap)`` and
``video_ratio = min(need / audio_rate, video_slow_cap)``. Whatever the
caps can't absorb becomes ``overflow_s`` (trimmed at mix time).
5. ``allow_video_retime=False`` audio-only mode: rate capped at the
legacy ``MAX_AUDIO_RATE_HARD`` (1.8, matching dub_generate's
MAX_STRETCH_RATIO guard rail), residual overflows.
6. **Timeline cursor** mirrors the existing ``stretch_video`` layout
loop: pre-roll and inter-segment gaps pass through at 1.0×; each
segment's video chunk ``[start, effective_end]`` occupies
``slot * video_ratio`` on the new timeline.
This module is deliberately I/O-free and torch-free so it can be unit- and
golden-tested without a model, ffmpeg, or an event loop.
"""
from __future__ import annotations
import math
from dataclasses import dataclass, field
# Hard ceiling for audio-only compression when video retiming is disabled.
# Matches dub_generate.MAX_STRETCH_RATIO — above ~1.8× speech becomes a
# garbled stream no DSP can rescue.
MAX_AUDIO_RATE_HARD = 1.8
_EPS = 1e-9
@dataclass(frozen=True)
class FitParams:
"""Tunable knobs for the Smart Fit planner.
All defaults are deliberately conservative: 1.2× audio-only is
imperceptible to most listeners; 1.5× audio is the intelligibility
cap; 2.0× video slow-down is the limit before motion looks syrupy.
"""
max_audio_only_rate: float = 1.2
audio_rate_cap: float = 1.5
video_slow_cap: float = 2.0
gap_guard_s: float = 0.05
allow_video_retime: bool = True
@dataclass
class SegmentFit:
"""Planner verdict for one segment."""
index: int
seg_id: str
audio_rate: float # ≥ 1.0 — pitch-preserving speed-up applied to TTS audio
video_ratio: float # ≥ 1.0 — setpts slow-down applied to the video chunk
new_start: float # placement on the fitted (possibly longer) timeline
new_end: float # end of the video chunk on the fitted timeline
orig_start: float
orig_end: float
effective_end: float # orig_end + absorbed slack (≤ next start gap guard)
status: str # "fits" | "audio_stretched" | "hybrid" | "overflow_trimmed"
overflow_s: float # seconds of (stretched) audio that still don't fit
@dataclass
class FitPlan:
"""Full plan for one dub track."""
segments: list[SegmentFit] = field(default_factory=list)
# EXACT dict shape consumed by dub_export._build_video_stretch_filter_graph.
video_plan: list[dict] = field(default_factory=list)
total_duration: float = 0.0
orig_duration: float = 0.0
params: FitParams = field(default_factory=FitParams)
@property
def needs_video_retime(self) -> bool:
return any(s.video_ratio > 1.0 + 1e-6 for s in self.segments)
def _fit_one(need: float, params: FitParams) -> tuple[float, float, str]:
"""Resolve one segment's need ratio into (audio_rate, video_ratio, status)."""
if need <= 1.0 + _EPS:
return 1.0, 1.0, "fits"
if need <= params.max_audio_only_rate + _EPS:
return need, 1.0, "audio_stretched"
if not params.allow_video_retime:
audio_rate = min(need, MAX_AUDIO_RATE_HARD)
status = "audio_stretched" if audio_rate >= need - _EPS else "overflow_trimmed"
return audio_rate, 1.0, status
# Geometric 50/50 split: equal perceptual burden on audio and video.
audio_rate = min(math.sqrt(need), params.audio_rate_cap)
video_ratio = min(need / audio_rate, params.video_slow_cap)
if audio_rate * video_ratio >= need - _EPS:
return audio_rate, video_ratio, "hybrid"
return audio_rate, video_ratio, "overflow_trimmed"
def plan_fit(
segments: list[dict],
natural_durs_s: list[float],
total_dur_s: float,
params: FitParams | None = None,
) -> FitPlan:
"""Plan the Smart Fit layout for a dub track.
``segments``: original-timeline segments in chronological order, each a
dict with ``id``, ``start``, ``end`` (seconds). ``natural_durs_s``: the
natural-rate TTS audio duration for each segment (parallel list).
``total_dur_s``: original video duration (0/unknown tolerated the last
segment then gets no tail slack).
Pure function: no I/O, no torch, deterministic.
"""
params = params or FitParams()
n = len(segments)
if len(natural_durs_s) != n:
raise ValueError(
f"segments ({n}) and natural_durs_s ({len(natural_durs_s)}) must be parallel"
)
plan = FitPlan(params=params, orig_duration=round(float(total_dur_s), 4))
if n == 0:
plan.total_duration = round(max(0.0, float(total_dur_s)), 4)
return plan
cursor = 0.0
for i, seg in enumerate(segments):
start = float(seg["start"])
end = float(seg["end"])
natural = max(0.0, float(natural_durs_s[i]))
# (a) Slack absorption. Extend-only: the slot never shrinks below
# the original [start, end] even when segments are back-to-back.
if i + 1 < n:
next_start = float(segments[i + 1]["start"])
effective_end = max(end, next_start - params.gap_guard_s)
# Never bleed past the next segment's onset (overlapping or
# near-touching source segments).
effective_end = min(max(effective_end, start), max(next_start, end))
else:
effective_end = max(end, float(total_dur_s)) if total_dur_s > 0 else end
slot = max(effective_end - start, 1e-3)
need = natural / slot if natural > 0 else 0.0
audio_rate, video_ratio, status = _fit_one(need, params)
# (f) Timeline cursor — mirror the stretch_video layout loop:
# pre-roll and gaps at 1.0×, the segment's video chunk
# [start, effective_end] occupies slot × video_ratio.
if i == 0:
cursor = start # pre-roll preserved at native rate
new_start = cursor
new_end = new_start + slot * video_ratio
cursor = new_end
if i + 1 < n:
# Unretimed sliver between this chunk and the next chunk's
# start (the gap guard, or more if extend-only clamped).
cursor += max(0.0, float(segments[i + 1]["start"]) - effective_end)
# Residual overflow after both knobs: stretched audio length vs the
# segment's new video slot.
stretched = natural / audio_rate if audio_rate > 0 else natural
overflow_s = max(0.0, stretched - slot * video_ratio)
if overflow_s <= 1e-6:
overflow_s = 0.0
elif status != "overflow_trimmed":
status = "overflow_trimmed"
plan.segments.append(SegmentFit(
index=i,
seg_id=str(seg.get("id", f"seg_{i}")),
audio_rate=round(audio_rate, 6),
video_ratio=round(video_ratio, 6),
new_start=round(new_start, 4),
new_end=round(new_end, 4),
orig_start=round(start, 4),
orig_end=round(end, 4),
effective_end=round(effective_end, 4),
status=status,
overflow_s=round(overflow_s, 4),
))
plan.video_plan.append({
"orig_start": round(start, 4),
"orig_end": round(effective_end, 4),
"new_start": round(new_start, 4),
"new_end": round(new_end, 4),
"stretch_ratio": round(video_ratio, 4),
})
# Tail (anything after the last segment's effective end) at 1.0×.
last_eff = float(plan.segments[-1].effective_end)
cursor += max(0.0, float(total_dur_s) - last_eff)
plan.total_duration = round(max(cursor, float(total_dur_s)), 4)
return plan
-65
View File
@@ -1,65 +0,0 @@
"""Map subtitle cues onto the Smart-Fit timeline (Wave 3.1 / Spec 1).
When a dub uses ``stretch_video`` mode, the video is re-timed per segment so
the dubbed audio fits (see fit_planner + the export stretch filter). The
dubbed audio therefore plays at *fitted* positions, not the original
timestamps. A subtitle file exported with the original times would drift
against the dubbed video so we regenerate the cue timeline from the same
plan the video stretch uses ("subtitles track actual dub placement", the
last piece of Spec 1).
Pure functions no I/O so the remapping is unit-testable. The plan is the
persisted ``video_stretch_plan`` list of
``{orig_start, orig_end, new_start, new_end, stretch_ratio}`` chunks.
"""
from __future__ import annotations
def map_time_to_fitted(t: float, plan: list[dict]) -> float:
"""Map a time on the original timeline to its position on the fitted one.
Finds the plan chunk whose original span contains ``t`` and interpolates
linearly into that chunk's fitted span (a chunk's stretch is uniform).
Before the first chunk maps 1:1; after the last chunk the trailing offset
is carried at 1:1 (the planner runs gaps/tail at rate 1.0). Empty plan
identity.
"""
if not plan:
return t
for chunk in plan:
o0 = float(chunk.get("orig_start", 0.0))
o1 = float(chunk.get("orig_end", 0.0))
n0 = float(chunk.get("new_start", o0))
n1 = float(chunk.get("new_end", o1))
if t < o0:
# In a gap before this chunk — carry the offset at 1:1 from the
# previous chunk's fitted end (or from 0 for the very first).
return n0 - (o0 - t)
if o0 <= t <= o1:
span = o1 - o0
if span <= 0:
return n0
return n0 + (t - o0) / span * (n1 - n0)
# Past the last chunk: 1:1 tail from its fitted end.
last = plan[-1]
return float(last.get("new_end", 0.0)) + (t - float(last.get("orig_end", 0.0)))
def fitted_cues(segments: list[dict], plan: list[dict]) -> list[tuple[float, float]]:
"""Return ``[(start, end), ...]`` for each segment on the fitted timeline.
Monotonicity guard: a cue's end is never before its start, and successive
starts never go backwards (rounding across chunk seams can't produce a
non-monotone SRT).
"""
out: list[tuple[float, float]] = []
prev_end = 0.0
for seg in segments:
s = map_time_to_fitted(float(seg.get("start", 0.0)), plan)
e = map_time_to_fitted(float(seg.get("end", 0.0)), plan)
s = max(s, prev_end if out else 0.0)
e = max(e, s)
out.append((s, e))
prev_end = e
return out
+3 -81
View File
@@ -22,32 +22,6 @@ import json
_GEN_INPUT_FIELDS = ("text", "target_lang", "profile_id", "instruct", "speed", "direction", "effect_preset")
# Pydantic fills `effect_preset` with this default when the client omits it,
# while the client-side recompute (/tools/incremental) sends nothing. Both
# representations must hash identically or every segment looks "stale" after
# every generate and incremental re-dub degrades to a full re-dub (#281).
_DEFAULT_EFFECT_PRESET = "broadcast"
def _canon_value(field: str, value):
"""Normalise one generation-input value so that the server-side view
(pydantic-parsed `DubSegment`, defaults filled in) and the client-side
view (raw segment dict, unset keys omitted) produce the same hash.
- missing / None / "" all mean "default" for string fields
- `effect_preset` default is "broadcast" (pydantic fills it server-side)
- numbers are coerced to float so `speed: 1` (JS) == `speed: 1.0` (pydantic)
"""
if field == "effect_preset":
return value or _DEFAULT_EFFECT_PRESET
if value is None or value == "":
return ""
if isinstance(value, bool):
return value
if isinstance(value, (int, float)):
return float(value)
return value
def segment_fingerprint(seg: dict) -> str:
"""Deterministic hash of the inputs that actually affect TTS output.
@@ -57,63 +31,11 @@ def segment_fingerprint(seg: dict) -> str:
badge don't trigger regen, which is what we want.
Currently includes: text, target_lang, profile_id, instruct, speed,
direction, effect_preset. Values are canonicalised (see `_canon_value`)
so a fingerprint computed from the generate request (server defaults
filled in) matches one recomputed later from the client's raw segment
state the root cause of #281's "1 edit re-dubs all N lines".
direction, effect_preset.
"""
payload = {k: _canon_value(k, seg.get(k)) for k in _GEN_INPUT_FIELDS}
payload = {k: (seg.get(k) if seg.get(k) is not None else "") for k in _GEN_INPUT_FIELDS}
blob = json.dumps(payload, sort_keys=True, ensure_ascii=False)
return hashlib.sha1(blob.encode("utf-8"), usedforsecurity=False).hexdigest()[:16]
# ── Smart Fit (dub-length fitting v2) fingerprints ─────────────────────────
#
# Fitting parameters stay OUT of segment_fingerprint on purpose: changing a
# fit knob (caps, gap guard, strategy) must trigger a RE-MIX of the already
# rendered natural-rate WAVs (generate with regen_only=[]), never a re-TTS.
# A separate per-track fingerprint tracks the fit configuration; a dubbed
# track is stale iff its fit_fp differs OR any segment hash differs.
_FIT_PARAM_FIELDS = (
"timing_strategy",
"max_audio_only_rate",
"audio_rate_cap",
"video_slow_cap",
"gap_guard_s",
"allow_video_retime",
)
# Server-side defaults (must mirror services.fit_planner.FitParams). Filled
# in for omitted keys so a fingerprint computed from a fully-populated
# server view matches one recomputed from a sparse client payload — the
# same #281 regression class segment_fingerprint already guards against.
_FIT_PARAM_DEFAULTS = {
"timing_strategy": "smart_fit",
"max_audio_only_rate": 1.2,
"audio_rate_cap": 1.5,
"video_slow_cap": 2.0,
"gap_guard_s": 0.05,
"allow_video_retime": True,
}
def fit_fingerprint(params: dict) -> str:
"""Deterministic hash of the fit configuration for one dub track.
Canonicalised with the same `_canon_value` rules as segment
fingerprints (int vs float, None/"" vs omitted), plus default-filling
so `{}` and `{"audio_rate_cap": 1.5}` hash identically.
"""
params = params or {}
payload = {}
for k in _FIT_PARAM_FIELDS:
v = params.get(k)
if v is None or v == "":
v = _FIT_PARAM_DEFAULTS[k]
payload[k] = _canon_value(k, v)
blob = json.dumps(payload, sort_keys=True, ensure_ascii=False)
return hashlib.sha1(blob.encode("utf-8"), usedforsecurity=False).hexdigest()[:16]
return hashlib.sha1(blob.encode("utf-8")).hexdigest()[:16]
def plan_incremental(
+7 -67
View File
@@ -37,12 +37,6 @@ logger = logging.getLogger("omnivoice.llm")
class LLMBackend(ABC):
id: str = "base"
display_name: str = "Base LLM"
# LLM backends call out over the network (OpenAI/Ollama/LM Studio) or are a
# no-op — none run a model on the user's GPU. So `gpu_compat` is empty and
# list_backends() labels the family `effective_device:"network"` /
# routing_status:"n/a" rather than asserting a false GPU claim. Routing is
# never gated for LLM (see engines.select_engine + diagnose).
gpu_compat: tuple[str, ...] = ()
@classmethod
@abstractmethod
@@ -112,22 +106,6 @@ class OpenAICompatBackend(LLMBackend):
return self._client
def chat(self, *, system: str, user: str, timeout: Optional[float] = None) -> str:
return self.chat_messages(
messages=[
{"role": "system", "content": system},
{"role": "user", "content": user},
],
timeout=timeout,
)
def chat_messages(self, *, messages: list[dict], timeout: Optional[float] = None) -> str:
"""One-shot completion over a full message list.
Additive surface for callers that need structured few-shot turns
(dictation refinement, Wave 2.1) small local models pattern-match
and echo inline examples, so examples must arrive as prior chat
turns, not inside the system prompt.
"""
if timeout is None:
try:
timeout = float(os.environ.get("OMNIVOICE_LLM_TIMEOUT", "45"))
@@ -136,7 +114,10 @@ class OpenAICompatBackend(LLMBackend):
res = self._get_client().chat.completions.create(
model=self.model_name,
timeout=timeout,
messages=messages,
messages=[
{"role": "system", "content": system},
{"role": "user", "content": user},
],
)
return (res.choices[0].message.content or "").strip()
@@ -162,9 +143,6 @@ class OffBackend(LLMBackend):
"to use features that need one (Cinematic translate, glossary auto-extract)."
)
def chat_messages(self, **kw) -> str:
return self.chat(**kw)
_REGISTRY: dict[str, type[LLMBackend]] = {
"openai-compat": OpenAICompatBackend,
@@ -172,53 +150,15 @@ _REGISTRY: dict[str, type[LLMBackend]] = {
}
# Most-recent failure per backend (parity with tts/asr list_backends).
_LAST_ERRORS: dict[str, str] = {}
_INSTALL_HINTS: dict[str, str] = {
"openai-compat": "Set TRANSLATE_BASE_URL (+ TRANSLATE_API_KEY) — OpenAI, "
"Ollama (http://localhost:11434/v1), or any compatible host.",
}
def list_backends() -> list[dict]:
"""Same 11-key shape as tts/asr so the matrix renders families uniformly.
LLM is NOT a GPU family: every entry carries literal
``effective_device:"network"`` / ``routing_status:"n/a"`` /
``routing_reason:null`` (NOT via resolve_routing that would be a false
GPU claim). ``effective_device:"network"`` is a label, not a probe: nothing
here touches the network (local-first).
"""
from core.scrub import scrub_text
out: list[dict] = []
out = []
for bid, cls in _REGISTRY.items():
try:
ok, msg = cls.is_available()
except Exception as exc:
ok = False
msg = f"{type(exc).__name__}: {exc}"
logger.warning(
"llm list_backends: %s.is_available() raised — degrading "
"gracefully so the picker still renders: %s", bid, msg,
)
if ok:
_LAST_ERRORS.pop(bid, None)
else:
_LAST_ERRORS[bid] = scrub_text(msg)
ok, msg = cls.is_available()
out.append({
"id": bid,
"display_name": cls.display_name,
"available": ok,
"reason": None if ok else scrub_text(msg),
"install_hint": _INSTALL_HINTS.get(bid),
"last_error": _LAST_ERRORS.get(bid),
"isolation_mode": "in-process",
"gpu_compat": list(getattr(cls, "gpu_compat", ())),
"effective_device": "network",
"routing_status": "n/a",
"routing_reason": None,
"reason": None if ok else msg,
})
return out
-251
View File
@@ -1,251 +0,0 @@
"""Import plain text / EPUB into the chapter-delimited script the audiobook
parser understands.
Both helpers are pure (bytes/str in, script-str out) so they're unit-tested
without a server. EPUB parsing is **stdlib only** (zipfile + ElementTree +
html.parser) no new dependency, no network, consistent with the local-first
guarantee. The output is the same ``# Heading`` + body grammar
:func:`services.audiobook.parse_audiobook_script` already consumes, so import is
just a front door onto the existing pipeline.
"""
from __future__ import annotations
import io
import posixpath
import re
import zipfile
from html.parser import HTMLParser
from xml.etree import ElementTree as ET
# A line that *starts* with a chapter keyword and is short enough to be a title
# (not a sentence that happens to begin with "Chapter"). Anchored, no ambiguous
# quantifiers → ReDoS-safe and applied per-line (short input) anyway.
_CH_RE = re.compile(r"^(?:chapter|part|book|prologue|epilogue|section)\b", re.IGNORECASE)
# Already-present Markdown H1 — if the text has any, we leave it untouched.
_H1_RE = re.compile(r"^[ \t]*#[ \t]+\S", re.MULTILINE)
_CHAPTER_TITLE_MAX = 60
# Zip-bomb / OOM guards for EPUB ingestion: per-entry and cumulative caps on
# *uncompressed* bytes read from the archive.
_EPUB_MAX_ENTRY_BYTES = 25 * 1024 * 1024
_EPUB_MAX_TOTAL_BYTES = 300 * 1024 * 1024
def chapterize_plaintext(text: str) -> str:
"""Insert ``# `` headings ahead of obvious chapter-title lines.
No-op if the text already has Markdown H1 headings (the user has structured
it). Otherwise short standalone lines beginning with a chapter keyword
(``Chapter 3``, ``Prologue`` ) become headings; everything else is left
verbatim. Text with no detectable breaks falls through as a single chapter.
"""
text = text or ""
if _H1_RE.search(text):
return text
out = []
for line in text.split("\n"):
s = line.strip()
if s and len(s) <= _CHAPTER_TITLE_MAX and _CH_RE.match(s):
out.append(f"# {s}")
else:
out.append(line)
return "\n".join(out)
class _TextExtractor(HTMLParser):
"""Collect visible text from XHTML, dropping script/style and collapsing
whitespace. First <h1>/<h2>/<title> seen is kept as the chapter title."""
_SKIP = {"script", "style", "head"}
_BREAK = {"p", "br", "div", "h1", "h2", "h3", "li", "tr"}
def __init__(self) -> None:
super().__init__(convert_charrefs=True)
self._parts: list[str] = []
self._skip_depth = 0
self._in_title = False
self.title = ""
def handle_starttag(self, tag, attrs):
if tag in self._SKIP:
self._skip_depth += 1
if tag in ("h1", "h2", "title") and not self.title:
self._in_title = True
if tag in self._BREAK:
self._parts.append("\n")
def handle_endtag(self, tag):
if tag in self._SKIP and self._skip_depth:
self._skip_depth -= 1
if tag in ("h1", "h2", "title"):
self._in_title = False
def handle_data(self, data):
if self._skip_depth:
return
if self._in_title:
# The first heading becomes the chapter's `# Title` (metadata, not
# narrated) — capture it but keep it out of the body. Later headings
# (title already set) fall through and are narrated as subheadings.
if not self.title:
self.title = data.strip()
return
self._parts.append(data)
def text(self) -> str:
raw = "".join(self._parts)
# Collapse runs of blank lines / trailing spaces into tidy paragraphs.
lines = [ln.strip() for ln in raw.split("\n")]
out: list[str] = []
for ln in lines:
if ln or (out and out[-1]):
out.append(ln)
return "\n".join(out).strip()
def _html_to_title_body(xhtml: str) -> tuple[str, str]:
p = _TextExtractor()
try:
p.feed(xhtml)
except Exception:
pass
return p.title, p.text()
_OPF_NS = {"opf": "http://www.idpf.org/2007/opf", "c": "urn:oasis:names:tc:opendocument:xmlns:container"}
def _opf_path(zf: zipfile.ZipFile) -> str:
container = zf.read("META-INF/container.xml")
# The EPUB is a local file the user chose to import (not a remote/untrusted
# surface); stdlib ElementTree doesn't expand external entities by default.
root = ET.fromstring(container) # nosec B314
rootfile = root.find(".//c:rootfiles/c:rootfile", _OPF_NS)
if rootfile is None or not rootfile.get("full-path"):
raise ValueError("EPUB container.xml has no rootfile")
return rootfile.get("full-path")
def epub_to_chapter_script(
data: bytes,
*,
max_entry_bytes: int = _EPUB_MAX_ENTRY_BYTES,
max_total_bytes: int = _EPUB_MAX_TOTAL_BYTES,
) -> str:
"""Convert EPUB bytes into a ``# Chapter`` / body script in spine order.
Reads the OPF manifest + spine (the publisher's reading order), extracts
each document's title + visible text, and emits one ``# Title`` block per
document with renderable text. ``max_entry_bytes`` / ``max_total_bytes``
bound the *uncompressed* bytes read (zip-bomb guard). Raises ``ValueError``
on a malformed EPUB.
"""
try:
zf = zipfile.ZipFile(io.BytesIO(data))
except zipfile.BadZipFile as e:
raise ValueError(f"not a valid EPUB (zip) file: {e}") from e
opf_path = _opf_path(zf)
opf = ET.fromstring(zf.read(opf_path)) # nosec B314 — local user EPUB; see _opf_path
base = posixpath.dirname(opf_path)
manifest: dict[str, str] = {}
for item in opf.findall(".//opf:manifest/opf:item", _OPF_NS):
iid, href = item.get("id"), item.get("href")
if iid and href:
manifest[iid] = href
blocks: list[str] = []
names = set(zf.namelist())
total = 0 # cumulative uncompressed bytes read — zip-bomb guard
for ref in opf.findall(".//opf:spine/opf:itemref", _OPF_NS):
href = manifest.get(ref.get("idref") or "")
if not href:
continue
full = posixpath.normpath(posixpath.join(base, href)) if base else href
if full not in names:
continue
# Bound decompression: skip an absurdly large entry, and stop once the
# cumulative uncompressed size crosses the ceiling (defends against a
# zip bomb / a maliciously huge chapter exhausting memory).
try:
info = zf.getinfo(full)
except KeyError:
continue
if info.file_size > max_entry_bytes:
continue
if total + info.file_size > max_total_bytes:
break
try:
raw = zf.read(full)
except KeyError:
continue
total += len(raw)
title, body = _html_to_title_body(raw.decode("utf-8", "ignore"))
if not body.strip():
continue # nav docs, empty pages
title = title or f"Chapter {len(blocks) + 1}"
blocks.append(f"# {title}\n\n{body}")
if not blocks:
raise ValueError("no readable chapters found in the EPUB")
return "\n\n".join(blocks)
# Page-count ceiling for PDF ingestion — a defence against a pathological
# document tying up the worker. 5000 pages comfortably covers any real book.
_PDF_MAX_PAGES = 5000
def pdf_to_chapter_script(data: bytes, *, max_pages: int = _PDF_MAX_PAGES) -> str:
"""Convert PDF bytes into a ``# Chapter`` / body script.
Extracts the embedded text layer page-by-page (in page order), joins it,
and runs it through :func:`chapterize_plaintext` so ``Chapter N`` /
``Prologue`` lines become headings same grammar EPUB and plaintext emit.
Unlike EPUB this needs a real parser (``pypdf``, pure-Python, no native
deps identical on every platform).
Limitations surfaced as ``ValueError`` (the route maps these to a 400 with
the message, so the user gets actionable feedback rather than a silent
empty import):
* **Scanned / image-only PDFs** have no text layer there's nothing to
extract without OCR, so we raise rather than return an empty script.
* **Password-protected PDFs** that don't open with an empty password can't
be read.
"""
from pypdf import PdfReader
from pypdf.errors import PdfReadError
try:
reader = PdfReader(io.BytesIO(data))
except (PdfReadError, OSError, ValueError) as e:
raise ValueError(f"not a valid PDF file: {e}") from e
if reader.is_encrypted:
# Many PDFs are encrypted with an empty user password (owner-locked but
# freely readable). Try that; a real password we can't supply.
try:
if reader.decrypt("") == 0: # 0 == wrong password
raise ValueError("PDF is password-protected")
except (NotImplementedError, PdfReadError) as e:
raise ValueError(f"can't read this encrypted PDF: {e}") from e
pages = reader.pages
if len(pages) > max_pages:
raise ValueError(f"PDF has too many pages (max {max_pages})")
parts: list[str] = []
for page in pages:
try:
text = page.extract_text() or ""
except Exception: # noqa: BLE001 — one bad page shouldn't kill the import
continue
if text.strip():
parts.append(text)
if not parts:
raise ValueError(
"no extractable text — this looks like a scanned or image-only PDF")
return chapterize_plaintext("\n\n".join(parts))
-143
View File
@@ -1,143 +0,0 @@
"""Canonical longform marker parser (#27) — the single source of grammar truth.
The longform marker dialect (``# heading``, ``[voice:NAME]``, ``[pause …]``,
``[slow]/[fast]/[emphasis]/[spell]``) was parsed by three independent code
paths that disagreed (client/server/regex-level). This module is the one
canonical Python parser; ``frontend/src/utils/longformParser.js`` is its
mechanically-mirrored JS twin, and ``tests/fixtures/longform_parser_cases.json``
is the shared golden corpus asserted byte-for-byte against both.
Pure textplan, import-light (no torch). Grammar precedence (outerinner):
# chapter → [voice:] → [pause] → SSML-lite → [spell]
It reuses the existing pause dialect (``omnivoice.utils.text.parse_pause_markers``)
and SSML-lite (``services.ssml_lite``) verbatim so those modules stay the single
home of their sub-grammars.
"""
from __future__ import annotations
import re
from typing import Optional
from omnivoice.utils.text import parse_pause_markers
# A Markdown H1 (``# Title``) starts a new chapter. Deeper headings (``##``…)
# stay in the body as ordinary text. The title capture starts with ``\S`` (a
# non-space) so the leading ``[ \t]+`` and the title's ``.*`` can't both match
# the same whitespace run — that overlap is what makes ``[ \t]+(.+)``
# polynomial-time on adversarial tabs (ReDoS). Moved verbatim from
# audiobook.py (already CodeQL-cleared). Stripped in code.
_HEADING_RE = re.compile(r"^[ \t]*#[ \t]+(\S.*)$", re.MULTILINE)
# ``[voice:NAME]`` switches the active narrator. The content class excludes BOTH
# brackets (``[^\]\[]``) so nested ``[voice:`` prefixes can't create overlapping
# match attempts across ``finditer`` (the ReDoS source). A voice name never
# contains a bracket; the value is stripped in code. Empty → default voice.
_VOICE_RE = re.compile(r"\[voice:([^\]\[]*)\]")
def _normalize(text: Optional[str]) -> str:
"""Coerce None→'' and normalize CRLF/CR→LF so ``$`` (re.MULTILINE) and span
text never carry a stray ``\\r`` on Windows-authored scripts a
cross-platform default-behaviour divergence the JS twin mirrors exactly."""
if not text:
return ""
return text.replace("\r\n", "\n").replace("\r", "\n")
def _parse_chapter_body(
body: str,
*,
default_voice: Optional[str] = None,
default_speed: Optional[float] = None,
) -> list[dict]:
"""Voice→pause→SSML layering for ONE chapter body (no chapter split).
Returns a list of span dicts ``{voice_id, text, pause_ms_after, speed}``.
A ``#`` inside ``body`` is NOT treated as a heading here — that is the
caller's (chapter-split) concern. The JS twin (``parseChapterBody``) is what
``storyToSpans`` calls per spoken track."""
spans: list[dict] = []
cur_voice = default_voice
runs: list[tuple[Optional[str], str]] = []
last = 0
for m in _VOICE_RE.finditer(body):
if m.start() > last:
runs.append((cur_voice, body[last:m.start()]))
cur_voice = (m.group(1).strip() or default_voice)
last = m.end()
runs.append((cur_voice, body[last:]))
from services.ssml_lite import parse_ssml_lite, spell_out
for voice, run_text in runs:
for span_text, pause_ms in parse_pause_markers(run_text):
t = span_text.strip()
if not t and pause_ms == 0:
continue # pure whitespace between markers — nothing to render
rendered: list[tuple[str, Optional[float]]] = []
for seg in (parse_ssml_lite(t) if t else []):
st = (spell_out(seg["text"]) if seg["spell"] else seg["text"]).strip()
if st:
# Inline SSML speed overrides the per-line default; a plain
# segment inherits default_speed.
sp = seg["speed"] if seg["speed"] is not None else default_speed
rendered.append((st, sp))
if not rendered:
# Only-markers / empty text but a real pause → carry the silence.
if pause_ms > 0:
spans.append({"voice_id": voice, "text": "",
"pause_ms_after": pause_ms, "speed": None})
continue
for j, (st, sp) in enumerate(rendered):
spans.append({
"voice_id": voice, "text": st,
"pause_ms_after": pause_ms if j == len(rendered) - 1 else 0,
"speed": sp,
})
return spans
def parse_script_to_spans(
text: Optional[str],
*,
default_voice: Optional[str] = None,
default_speed: Optional[float] = None,
) -> list[dict]:
"""Parse a chapter-delimited script into ``[{"title", "spans": [...]}, …]``.
span dict == ``{"voice_id": str|None, "text": str, "pause_ms_after": int,
"speed": float|None}`` (key order matches ``Span.to_dict()``).
Contract:
* None / "" / whitespace-only input ``[]``.
* CRLF/CR normalized to LF at entry (cross-platform parity).
* H1 (``# <non-space>…``) opens a chapter; ``##``…``######`` and ``# ``
(no ``\\S`` title) are body.
* Each chapter body resets the active voice to ``default_voice``.
* A span is dropped iff its text is empty AND pause_ms_after == 0.
* Chapters with no surviving spans are dropped; untitled bodies are
numbered ``Chapter {kept_so_far + 1}`` (post-drop numbering).
"""
text = _normalize(text)
matches = list(_HEADING_RE.finditer(text))
if not matches:
raw = [(None, text)]
else:
raw = []
intro = text[:matches[0].start()]
if intro.strip():
raw.append((None, intro))
for i, m in enumerate(matches):
end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
raw.append((m.group(1).strip(), text[m.end():end]))
chapters: list[dict] = []
for title, body in raw:
spans = _parse_chapter_body(body, default_voice=default_voice,
default_speed=default_speed)
if not spans:
continue
chapters.append({"title": title or f"Chapter {len(chapters) + 1}",
"spans": spans})
return chapters
-389
View File
@@ -1,389 +0,0 @@
"""Shared long-form render core (Stories + Audiobook convergence).
Both the Audiobook tab and the Stories Editor produce the *same* artifact: a
chapter-marked audio file built from chapter WAVs. This module owns the pure,
engine-agnostic ffmpeg/metadata builders for that mux so neither feature has to
reimplement it:
* ``build_ffmetadata`` FFMETADATA1 doc: an optional ``[global]`` tag block
(title / author / narrator / year / genre / description) followed by one
``[CHAPTER]`` per (title, duration_ms).
* ``build_concat_list`` ffmpeg concat-demuxer list of chapter WAVs.
* ``build_loudnorm_filter`` an ``-af loudnorm=`` string for an ACX /
podcast loudness preset (off by default opt-in, so the default-behavior
stays platform-identical).
* ``validate_cover_image`` guard a cover path (type + size) before it
reaches ffmpeg.
* ``build_render_cmd`` pure argv for the mux: chapter WAVs + FFMETADATA
(+ optional cover art, loudness filter), output as ``m4b`` or ``mp3``.
* ``chapter_cache_key`` deterministic content hash so a re-run reuses
already-rendered chapters (resume) and re-renders only what changed.
Every function here is pure (string/argv in, string/argv out) so it's unit
tested without ffmpeg, torch, or a GPU. The impure ffmpeg run lives in the
caller (the audiobook router today; the stories job tomorrow).
"""
from __future__ import annotations
import hashlib
import json
import math
import os
import re
from dataclasses import dataclass
from pathlib import Path
from typing import Iterable, Optional
_BITRATE_RE = re.compile(r"^\d{2,3}k$")
#: Default ceiling for the content-addressed chapter cache. Above this, the
#: oldest cached chapter WAVs are evicted (LRU by mtime). Override via
#: OMNIVOICE_LONGFORM_CACHE_MAX_GB.
_CACHE_MAX_BYTES = int(float(os.environ.get("OMNIVOICE_LONGFORM_CACHE_MAX_GB", "2")) * 1024 ** 3)
_COVER_EXTS = {".jpg", ".jpeg", ".png"}
_COVER_MAX_BYTES = 8 * 1024 * 1024 # 8 MB — a book cover, not a payload
#: Our metadata field → FFMETADATA tag key. Order is stable for deterministic
#: output (tested). ``author`` maps to ``artist`` and ``narrator`` to
#: ``composer`` — the tags audiobook players (Apple Books, Audible) read for
#: those roles.
_GLOBAL_TAG_KEYS: list[tuple[str, str]] = [
("title", "title"),
("author", "artist"),
("album", "album"),
("narrator", "composer"),
("year", "date"),
("genre", "genre"),
("description", "comment"),
]
def _escape_meta(value: str) -> str:
"""Escape an FFMETADATA value (``=``, ``;``, ``#``, ``\\``, newline)."""
return re.sub(r"([=;#\\\n])", r"\\\1", value or "")
def prune_cache_dir(cache_dir: str, max_bytes: int = _CACHE_MAX_BYTES) -> tuple[int, int]:
"""Evict the oldest files in ``cache_dir`` until the total size is within
``max_bytes`` (LRU by mtime). The content-addressed chapter cache otherwise
grows without bound uncompressed WAVs accumulate across every render.
Best-effort: returns ``(remaining_bytes, removed_count)`` and never raises
(a missing dir / unstattable file is just skipped). Call it *before* writing
a job's chapters so the fresh ones are never the eviction target.
"""
try:
names = os.listdir(cache_dir)
except OSError:
return (0, 0)
entries: list[tuple[float, int, str]] = []
total = 0
for name in names:
p = os.path.join(cache_dir, name)
try:
if not os.path.isfile(p):
continue
size = os.path.getsize(p)
mtime = os.path.getmtime(p)
except OSError:
continue
entries.append((mtime, size, p))
total += size
if total <= max_bytes:
return (total, 0)
entries.sort() # oldest first
removed = 0
for _mtime, size, p in entries:
if total <= max_bytes:
break
try:
os.remove(p)
total -= size
removed += 1
except OSError:
continue
return (total, removed)
# ── Chapter cache key (resume) ──────────────────────────────────────────────
def chapter_cache_key(
spans: Iterable[tuple],
*,
sample_rate: int,
engine_id: str,
voice_sig: Optional[dict] = None,
) -> str:
"""Deterministic content hash for a chapter's rendered audio.
``spans`` is an ordered list of ``(voice_id, text, pause_ms_after[, speed])``
(speed optional, defaults to None). Same inputs same key reuse the
cached chapter WAV on a re-run (resume); any change (text, voice, order,
pauses, speed, sample rate, engine, or a voice's resolved signature) → new
key re-render. ``voice_sig`` maps each voice id to a stable signature
string (e.g. ``ref_audio|instruct|seed``) so editing the underlying profile
also invalidates the cache.
"""
payload = {
"sr": int(sample_rate),
"engine": engine_id or "",
"spans": [[s[0], s[1], int(s[2]), (s[3] if len(s) > 3 else None)] for s in spans],
"voices": {k: voice_sig[k] for k in sorted(voice_sig)} if voice_sig else {},
}
raw = json.dumps(payload, sort_keys=True, ensure_ascii=False)
# Content-addressing only — not a security digest. usedforsecurity=False
# keeps bandit's B324 (weak-hash) check quiet.
return hashlib.sha1(raw.encode("utf-8"), usedforsecurity=False).hexdigest()[:20]
# ── Loudness normalization ──────────────────────────────────────────────────
@dataclass(frozen=True)
class LoudnessPreset:
"""A loudnorm target. ``i`` = integrated LUFS, ``tp`` = true-peak ceiling
(dBTP), ``lra`` = loudness range."""
key: str
i: float
tp: float
lra: float
#: ``acx`` targets Audible/ACX submission (≈ -19 LUFS integrated, ≤ -3 dBTP
#: peak — inside ACX's -23…-18 dB RMS / -3 dB peak window). ``podcast`` targets
#: the -16 LUFS streaming norm.
LOUDNESS_PRESETS: dict[str, LoudnessPreset] = {
"acx": LoudnessPreset("acx", -19.0, -3.0, 11.0),
"podcast": LoudnessPreset("podcast", -16.0, -1.5, 11.0),
}
def build_loudnorm_filter(preset: Optional[str]) -> Optional[str]:
"""Return an ``-af`` loudnorm filter string for ``preset``, or ``None`` for
off / unknown (single-pass; two-pass measureapply is a runner enhancement).
"""
if not preset:
return None
p = LOUDNESS_PRESETS.get(preset.lower())
if p is None: # "off", "none", or anything unrecognized → no filter
return None
return f"loudnorm=I={p.i}:TP={p.tp}:LRA={p.lra}"
@dataclass(frozen=True)
class MeasuredLoudness:
"""The five loudnorm measure-pass values (FFmpeg JSON keys), all finite
floats. Fed back into the second (apply) pass as ``measured_*`` + ``offset``."""
input_i: float
input_tp: float
input_lra: float
input_thresh: float
target_offset: float
def build_loudnorm_measure_filter(preset: Optional[str]) -> Optional[str]:
"""First-pass loudnorm filter (``print_format=json``) for ``preset``, or
``None`` for off/unknown mirrors :func:`build_loudnorm_filter`'s lookup
(no whitespace stripping) so the same values count as 'no filter'."""
if not preset:
return None
p = LOUDNESS_PRESETS.get(preset.lower())
if p is None:
return None
return f"loudnorm=I={p.i}:TP={p.tp}:LRA={p.lra}:print_format=json"
def parse_loudnorm_measure(stderr_text: Optional[str]) -> Optional[MeasuredLoudness]:
"""Extract the loudnorm measure JSON from ffmpeg stderr → MeasuredLoudness,
or ``None`` on ANY failure (caller falls back to single-pass). FFmpeg prints
the JSON object amid other non-JSON lines (and possibly a config dump block),
so we take the LAST balanced ``{...}`` via a linear brace-depth scan no
regex (CodeQL-safe), O(n), no backtracking then json.loads + coerce/validate
the five required keys to finite floats."""
if not stderr_text:
return None
# Find the last balanced top-level {...} block via a single linear scan.
start = -1
depth = 0
block = None
for i, ch in enumerate(stderr_text):
if ch == "{":
if depth == 0:
start = i
depth += 1
elif ch == "}":
if depth > 0:
depth -= 1
if depth == 0 and start != -1:
block = stderr_text[start:i + 1] # keep scanning → last wins
if block is None:
return None
try:
obj = json.loads(block)
except (json.JSONDecodeError, ValueError):
return None
if not isinstance(obj, dict):
return None
keys = ("input_i", "input_tp", "input_lra", "input_thresh", "target_offset")
vals = {}
for k in keys:
if k not in obj:
return None
try:
v = float(obj[k])
except (TypeError, ValueError):
return None
if not math.isfinite(v): # rejects "-inf"/"inf"/"nan" (silent clip)
return None
vals[k] = v
return MeasuredLoudness(**vals)
def build_loudnorm_apply_filter(
preset: Optional[str], measured: Optional["MeasuredLoudness"],
) -> Optional[str]:
"""Second-pass (apply) loudnorm filter feeding the measured values back in.
``None`` for off/unknown preset OR when ``measured`` is None (so a caller
that forgot to branch never emits ``measured_I=None``)."""
if not preset or measured is None:
return None
p = LOUDNESS_PRESETS.get(preset.lower())
if p is None:
return None
return (
f"loudnorm=I={p.i}:TP={p.tp}:LRA={p.lra}"
f":measured_I={measured.input_i}:measured_TP={measured.input_tp}"
f":measured_LRA={measured.input_lra}:measured_thresh={measured.input_thresh}"
f":offset={measured.target_offset}:linear=true:print_format=summary"
)
def build_loudnorm_measure_cmd(ffmpeg: str, concat_list_path: str, filt: str) -> list[str]:
"""Pure argv for the measure pass: decode the concat list, run the
print_format=json loudnorm filter, discard audio to the portable null muxer.
Input segment is byte-identical to build_render_cmd so measured == muxed."""
return [
ffmpeg, "-y", "-hide_banner", "-loglevel", "info",
"-f", "concat", "-safe", "0", "-i", str(concat_list_path),
"-af", filt, "-f", "null", "-",
]
# ── FFMETADATA ──────────────────────────────────────────────────────────────
def build_ffmetadata(
chapters: Iterable[tuple[str, int]],
global_meta: Optional[dict] = None,
) -> str:
"""Build an FFMETADATA1 doc: optional global tags + one ``[CHAPTER]`` per
``(title, duration_ms)``. START/END are cumulative millisecond offsets.
"""
lines = [";FFMETADATA1"]
if global_meta:
for field_key, meta_key in _GLOBAL_TAG_KEYS:
val = global_meta.get(field_key)
if val is not None and str(val).strip():
lines.append(f"{meta_key}={_escape_meta(str(val).strip())}")
start = 0
for title, dur_ms in chapters:
end = start + max(0, int(dur_ms))
lines += [
"[CHAPTER]",
"TIMEBASE=1/1000",
f"START={start}",
f"END={end}",
f"title={_escape_meta(title)}",
]
start = end
return "\n".join(lines) + "\n"
def build_concat_list(wav_paths: Iterable[str]) -> str:
"""Build an ffmpeg concat-demuxer list. Single quotes in paths are escaped
the ffmpeg way (``'`` → ``'\\''``) so paths can't break the list or inject
arguments."""
lines = []
for p in wav_paths:
safe = str(p).replace("'", "'\\''")
lines.append(f"file '{safe}'")
return "\n".join(lines) + "\n"
# ── Cover art ───────────────────────────────────────────────────────────────
def validate_cover_image(path: Optional[str]) -> bool:
"""True if ``path`` is a readable jpg/png within the size cap. Anything
dubious (missing, wrong type, too big, unreadable) False, and the caller
simply omits the cover rather than failing the render."""
if not path:
return False
try:
p = Path(path)
return (
p.is_file()
and p.suffix.lower() in _COVER_EXTS
and 0 < p.stat().st_size <= _COVER_MAX_BYTES
)
except OSError:
return False
# ── Render command ──────────────────────────────────────────────────────────
def build_render_cmd(
ffmpeg: str,
concat_list_path: str,
metadata_path: str,
out_path: str,
*,
fmt: str = "m4b",
bitrate: str = "128k",
cover_path: Optional[str] = None,
loudness: Optional[str] = None,
measured: Optional[MeasuredLoudness] = None,
) -> list[str]:
"""Pure argv for muxing chapter WAVs + FFMETADATA into a tagged,
chapter-marked audio file.
Inputs: 0 = concat-demuxer list of chapter WAVs, 1 = FFMETADATA (chapters +
global tags), 2 = cover image (only when present + valid). ``fmt`` is
``m4b`` (AAC in mp4, faststart) or ``mp3`` (libmp3lame). A loudness preset
adds an ``-af loudnorm`` pass; an invalid/oversized cover is silently
dropped (see :func:`validate_cover_image`).
"""
if not _BITRATE_RE.match(bitrate or ""):
bitrate = "128k"
is_mp3 = (fmt or "").lower() == "mp3"
# Cover art is embedded for M4B only. The MP3 muxer rejects an
# ``attached_pic`` video stream via ``-c:v copy`` (produces a corrupt file
# across ffmpeg versions), and a reliable cross-version ID3 APIC path is
# finicky — so for MP3 we skip the cover rather than ship a broken file.
# M4B is the cover-bearing audiobook format anyway.
embed_cover = validate_cover_image(cover_path) and not is_mp3
cmd = [
ffmpeg, "-y", "-hide_banner", "-loglevel", "error",
"-f", "concat", "-safe", "0", "-i", str(concat_list_path),
"-i", str(metadata_path),
]
if embed_cover:
cmd += ["-i", str(cover_path)]
cmd += ["-map", "0:a", "-map_metadata", "1"]
if embed_cover:
cmd += ["-map", "2:v", "-disposition:v", "attached_pic"]
# Two-pass apply when measured values are present; else single-pass. Both
# return None for a non-preset loudness, so the `if filt:` guard below
# gives an off-render no -af (byte-identical to today).
filt = build_loudnorm_apply_filter(loudness, measured) if measured is not None else build_loudnorm_filter(loudness)
if filt:
cmd += ["-af", filt]
if is_mp3:
cmd += ["-c:a", "libmp3lame", "-b:a", bitrate, "-f", "mp3", str(out_path)]
else: # m4b — AAC in an mp4 container
cmd += ["-c:a", "aac", "-b:a", bitrate]
if embed_cover:
cmd += ["-c:v", "copy"]
cmd += ["-movflags", "+faststart", "-f", "mp4", str(out_path)]
return cmd
-195
View File
@@ -1,195 +0,0 @@
"""Durable resume for longform (audiobook / story) renders.
Chapter WAVs are already content-addressed in a shared cache, so re-rendering an
identical plan reuses what finished the synthesis-level resume. The missing
piece this module adds is **durability of the *plan itself***: a render that's
interrupted (crash, app quit, power loss) leaves a `resume.json` manifest in the
job's work dir holding the compiled plan + render params, so the job can be
resumed later *without the user still having the original script* which matters
for Stories, whose plan is compiled from cast+lines and can't be retyped.
Pure file/JSON I/O (no torch, no model) so it's unit-testable. The router wires
it into the SSE renderer (write on start, clear on done) and exposes
``GET /audiobook/jobs`` (resumable) + ``POST /audiobook/resume/{job_id}``.
"""
from __future__ import annotations
import json
import os
import re
from typing import Optional
MANIFEST_VERSION = 1
_MANIFEST_NAME = "resume.json"
# Longform front doors that produce a resumable work dir (job_type → dir prefix).
RESUMABLE_TYPES = ("audiobook", "story")
# A job id is server-generated (uuid4 hex) — confine to a strict token so a
# request-supplied id (the /audiobook/resume/{job_id} path param) can never
# carry a path separator, `..`, NUL, or anything that escapes OUTPUTS_DIR
# (CodeQL py/path-injection). Anchored, single bounded quantifier → ReDoS-safe.
_SAFE_ID_RE = re.compile(r"^[A-Za-z0-9_-]{1,64}$")
# Exact-match allowlist on the WHOLE work-dir name (the value actually joined
# onto OUTPUTS_DIR). Validating the joined string itself — not just job_id — is
# the barrier CodeQL's path-injection query recognizes (same shape as
# audiobook._safe_cover_path's _COVER_NAME_RE). Mirrors RESUMABLE_TYPES.
_SAFE_SEG_RE = re.compile(r"^(?:audiobook|story)_[A-Za-z0-9_-]{1,64}$")
def work_dir(job_type: str, job_id: str) -> Optional[str]:
"""The per-job work directory ``OUTPUTS_DIR/<job_type>_<job_id>``, resolved
strictly inside OUTPUTS_DIR. Returns None for an unknown ``job_type``, an
id that isn't a bare safe token, or any path that escapes OUTPUTS_DIR — so a
crafted ``job_id`` can never reach a foreign path (py/path-injection-safe)."""
if job_type not in RESUMABLE_TYPES or not _SAFE_ID_RE.match(job_id or ""):
return None
# os.path.basename strips any directory component (the sanitizer CodeQL's
# path-injection query recognizes — same as audiobook._safe_cover_path), and
# the exact-match allowlist on the result is a second barrier: the joined
# value is provably a single bare dir name of the expected shape.
seg = os.path.basename(f"{job_type}_{job_id}")
if not _SAFE_SEG_RE.match(seg):
return None
from core.config import OUTPUTS_DIR
root = os.path.realpath(OUTPUTS_DIR)
path = os.path.realpath(os.path.join(root, seg))
# commonpath containment — the form static analysis recognizes (belt over
# the regex). Raises ValueError on mixed drives (Windows) → reject.
try:
if os.path.commonpath([path, root]) != root:
return None
except ValueError:
return None
return path
def manifest_path(job_type: str, job_id: str) -> Optional[str]:
d = work_dir(job_type, job_id)
return os.path.join(d, _MANIFEST_NAME) if d else None
def build_manifest(
*,
job_id: str,
job_type: str,
plan_chapters: list[dict],
params: dict,
title: str = "",
) -> dict:
"""Assemble the manifest dict. ``plan_chapters`` is the canonical span-plan
(``[{title, spans:[{voice_id,text,pause_ms_after,speed}]}]``); ``params`` is
the render kwargs (default_voice / fmt / bitrate / loudness / cover_path /
metadata / lexicon). Pure no I/O."""
return {
"version": MANIFEST_VERSION,
"job_id": job_id,
"job_type": job_type,
"title": title or "",
"total_chapters": len(plan_chapters),
"params": params,
"plan": plan_chapters,
}
def write_manifest(manifest: dict) -> Optional[str]:
"""Persist the manifest to the job work dir. Best-effort — resume is an
enhancement, never block the render returns the path or None on failure /
unsafe id."""
path = manifest_path(manifest.get("job_type", ""), manifest.get("job_id", ""))
if path is None:
return None
try:
os.makedirs(os.path.dirname(path), exist_ok=True)
tmp = path + ".tmp"
with open(tmp, "w", encoding="utf-8") as f:
json.dump(manifest, f, ensure_ascii=False)
os.replace(tmp, path) # atomic — a half-written manifest never resumes
return path
except OSError:
return None
def read_manifest(job_type: str, job_id: str) -> Optional[dict]:
"""Load a job's resume manifest, or None if absent/unreadable/foreign-shape/
unsafe id."""
path = manifest_path(job_type, job_id)
if path is None:
return None
try:
with open(path, encoding="utf-8") as f:
data = json.load(f)
except (OSError, ValueError):
return None
if not isinstance(data, dict) or data.get("version") != MANIFEST_VERSION:
return None
if not isinstance(data.get("plan"), list):
return None
return data
def clear_manifest(job_type: str, job_id: str) -> None:
"""Remove the manifest once a job completes (no resume needed). Best-effort."""
path = manifest_path(job_type, job_id)
if path is None:
return
try:
os.remove(path)
except OSError:
return # best-effort; already gone or unwritable
def has_manifest(job_type: str, job_id: str) -> bool:
path = manifest_path(job_type, job_id)
return bool(path) and os.path.isfile(path)
def scan_resumable() -> list[dict]:
"""Enumerate resumable jobs by scanning OUTPUTS_DIR for ``<type>_<id>`` work
dirs that hold a ``resume.json``.
Returns ``[{"job_type", "job_id", "manifest_path"}, ]`` where **every path
component is sourced from os.listdir** (the trusted filesystem), never from
request input. The ``manifest_path`` is built here from the listed dir name,
so callers can read it directly without re-deriving a path from a
request-supplied id (CodeQL py/path-injection-safe no tainted value ever
reaches a file operation)."""
from core.config import OUTPUTS_DIR
out: list[dict] = []
try:
root = os.path.realpath(OUTPUTS_DIR)
names = os.listdir(root)
except OSError:
return out
for name in names:
for jt in RESUMABLE_TYPES:
prefix = f"{jt}_"
mpath = os.path.join(root, name, _MANIFEST_NAME)
if name.startswith(prefix) and os.path.isfile(mpath):
out.append({"job_type": jt, "job_id": name[len(prefix):],
"manifest_path": mpath})
return out
def discard_manifest_file(path: str) -> None:
"""Remove a manifest by a path obtained via :func:`scan_resumable` (trusted,
os.listdir-derived). Best-effort used to retire an interrupted job once
it's been resumed under a fresh id."""
try:
os.remove(path)
except OSError:
return
def load_manifest_file(path: str) -> Optional[dict]:
"""Read + validate a manifest from a path obtained via :func:`scan_resumable`
(a trusted, os.listdir-derived path NOT a request-derived one). Returns
None on missing/unreadable/foreign-shape."""
try:
with open(path, encoding="utf-8") as f:
data = json.load(f)
except (OSError, ValueError):
return None
if not isinstance(data, dict) or data.get("version") != MANIFEST_VERSION:
return None
if not isinstance(data.get("plan"), list):
return None
return data
-68
View File
@@ -1,68 +0,0 @@
"""Two-pass loudnorm measure orchestrator (#28).
The impure half of the two-pass ACX/podcast master: run ffmpeg's measure pass
over the concatenated chapters and parse the printed loudnorm JSON. The pure
builders/parser live in :mod:`services.longform_render`; this only drives ffmpeg.
Contract: **never raises.** Every failure (skip / non-zero rc / timeout / spawn
error / empty or unparseable stderr / silent program) is caught, logged at
WARNING, and converted to ``None`` so the caller falls back to single-pass. A
slow or broken measure must degrade the master, never abort the render.
"""
from __future__ import annotations
import logging
from typing import Optional
from services.longform_render import (
MeasuredLoudness,
build_loudnorm_measure_cmd,
build_loudnorm_measure_filter,
parse_loudnorm_measure,
)
logger = logging.getLogger("omnivoice.loudness")
async def measure_loudness(
ffmpeg: str,
concat_list_path: str,
preset: str,
*,
job_id: str,
) -> Optional[MeasuredLoudness]:
"""Measure the concatenated program's loudness for ``preset`` (acx/podcast),
or ``None`` for off/unknown or on ANY failure ( single-pass fallback).
Only the ffmpeg rc and a short static message are logged never the raw
stderr (it can carry the concat path under OUTPUTS_DIR), keeping the log
local-first / path-safe.
"""
filt = build_loudnorm_measure_filter(preset)
if filt is None:
return None # off / unknown — a normal skip, not an error (no log)
cmd = build_loudnorm_measure_cmd(ffmpeg, concat_list_path, filt)
from services.ffmpeg_utils import run_ffmpeg # lazy → patchable at source
try:
# asyncio.TimeoutError is a subclass of Exception (Py≥3.11) — caught
# here so a slow measure degrades to single-pass instead of killing the
# whole render via the caller's outer except.
rc, _out, err = await run_ffmpeg(cmd, capture=True, job_id=job_id)
except Exception as exc:
logger.warning("loudness measure pass did not run (%s) — single-pass fallback",
type(exc).__name__)
return None
if rc != 0:
logger.warning("loudness measure pass exited rc=%s — single-pass fallback", rc)
return None
try:
stderr_text = err.decode("utf-8", "replace") if isinstance(err, (bytes, bytearray)) else (err or "")
except Exception:
return None
measured = parse_loudnorm_measure(stderr_text)
if measured is None:
logger.warning("loudness measure output unparseable — single-pass fallback")
return measured
__all__ = ["measure_loudness"]
-126
View File
@@ -1,126 +0,0 @@
"""Per-agent MCP voice bindings (Wave 2.2 / Spec 2).
An MCP client identifies itself with the ``X-OmniVoice-Client-Id`` header.
Each client can be bound to a default voice profile + engine so different
agents speak in different voices ("Claude Code in Morgan, Cursor in
Scarlett"). Pure data layer over the ``mcp_client_bindings`` table — the
FastMCP tools call :func:`resolve_voice`; the Settings UI calls the CRUD
helpers via the REST router.
"""
from __future__ import annotations
import time
from typing import Optional
from core.db import db_conn
def list_bindings() -> list[dict]:
with db_conn() as conn:
# SQLite sorts NULL as smallest, so DESC naturally puts never-seen
# bindings after recently-active ones.
rows = conn.execute(
"SELECT * FROM mcp_client_bindings ORDER BY last_seen_at DESC, created_at DESC"
).fetchall()
return [dict(r) for r in rows]
def get_binding(client_id: str) -> Optional[dict]:
with db_conn() as conn:
row = conn.execute(
"SELECT * FROM mcp_client_bindings WHERE client_id=?", (client_id,)
).fetchone()
return dict(row) if row else None
def upsert_binding(
client_id: str,
*,
label: Optional[str] = None,
profile_id: Optional[str] = None,
default_engine: Optional[str] = None,
) -> dict:
"""Create or update a binding. Fields left as None on an existing row are
preserved; on a new row they default to empty/null."""
if not client_id or not client_id.strip():
raise ValueError("client_id must be non-empty")
cid = client_id.strip()
existing = get_binding(cid)
now = time.time()
if existing:
merged = {
"label": existing["label"] if label is None else label,
"profile_id": existing["profile_id"] if profile_id is None else (profile_id or None),
"default_engine": existing["default_engine"] if default_engine is None else (default_engine or None),
}
with db_conn() as conn:
conn.execute(
"UPDATE mcp_client_bindings SET label=?, profile_id=?, default_engine=? WHERE client_id=?",
(merged["label"], merged["profile_id"], merged["default_engine"], cid),
)
else:
with db_conn() as conn:
conn.execute(
"INSERT INTO mcp_client_bindings "
"(client_id, label, profile_id, default_engine, last_seen_at, created_at) "
"VALUES (?, ?, ?, ?, NULL, ?)",
(cid, label or "", profile_id or None, default_engine or None, now),
)
return get_binding(cid)
def delete_binding(client_id: str) -> bool:
with db_conn() as conn:
cur = conn.execute("DELETE FROM mcp_client_bindings WHERE client_id=?", (client_id,))
return cur.rowcount > 0
def touch_last_seen(client_id: str) -> None:
"""Best-effort 'last heard from this agent' stamp. Never raises — it's
telemetry for the Settings list, not load-bearing."""
if not client_id:
return
try:
with db_conn() as conn:
conn.execute(
"UPDATE mcp_client_bindings SET last_seen_at=? WHERE client_id=?",
(time.time(), client_id),
)
except Exception:
pass
def _global_default_profile() -> Optional[str]:
"""The fallback voice when a client has no binding. Reads the same
pref the Settings 'default playback voice' would set; None if unset."""
try:
from core import prefs
return prefs.get("mcp_default_profile_id") or None
except Exception:
return None
def resolve_voice(client_id: Optional[str], explicit_profile_id: Optional[str]) -> dict:
"""Resolve which voice an MCP speak call should use.
Precedence (Spec 2): explicit tool arg the client's binding →
the global default nothing (caller decides / errors with a hint).
Returns ``{profile_id, default_engine, source}`` where ``source`` is one
of ``explicit`` | ``binding`` | ``global`` | ``none`` for diagnostics.
"""
if explicit_profile_id:
return {"profile_id": explicit_profile_id, "default_engine": None, "source": "explicit"}
if client_id:
binding = get_binding(client_id)
if binding and binding.get("profile_id"):
return {
"profile_id": binding["profile_id"],
"default_engine": binding.get("default_engine"),
"source": "binding",
}
g = _global_default_profile()
if g:
return {"profile_id": g, "default_engine": None, "source": "global"}
return {"profile_id": None, "default_engine": None, "source": "none"}
-158
View File
@@ -1,158 +0,0 @@
"""Single lifecycle surface for loaded models (MM2-04).
Before this, ``GET /model/loaded`` and ``POST /model/unload`` each hand-rolled
enumeration/dispatch across three worlds the in-process TTS+ASR model
(``model_manager``), the diarization pipeline, and subprocess sidecars
(``subprocess_backend``). This module owns that logic so the routers are thin
delegations and there's one place to reason about model lifecycle.
Response shapes are preserved exactly the frontend (hooks.ts model status +
the flush dropdown) depends on ``{models, count}`` and
``{unloaded, success, ...}``.
"""
from __future__ import annotations
import os
from typing import Optional
import services.model_manager as mm
from services.model_manager import get_best_device
def _tts_vram_mb() -> float:
"""Best-effort allocated VRAM for the in-process model. Accurate on CUDA,
sparse on MPS, 0 elsewhere degrade gracefully, never raise."""
try:
torch = mm._lazy_torch()
if torch.cuda.is_available():
return torch.cuda.memory_allocated() / (1024 ** 2)
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
driver = getattr(torch.mps, "driver_allocated_memory", None)
if driver:
return driver() / (1024 ** 2)
except Exception:
pass
return 0.0
def _asr_device() -> str:
"""Where the ASR pipe actually lives, rather than a hardcoded 'cpu'."""
pipe = getattr(mm.model, "_asr_pipe", None)
for attr in ("device",):
dev = getattr(pipe, attr, None)
if dev is not None:
return str(dev)
return "cpu"
def list_loaded() -> dict:
"""Enumerate every currently-loaded model. Shape: ``{"models": [...],
"count": n}`` with per-model id/name/checkpoint/device/vram_mb/unloadable
(+ optional ``note``)."""
models: list[dict] = []
# 1. In-process TTS model (OmniVoice)
if mm.model is not None:
try:
device = str(next(mm.model.parameters()).device) if hasattr(mm.model, "parameters") else get_best_device()
except Exception:
device = get_best_device()
models.append({
"id": "tts",
"name": "OmniVoice TTS",
"checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
"device": device,
"vram_mb": round(_tts_vram_mb(), 1),
"unloadable": True,
})
# 2. ASR (WhisperX) — co-loaded with and released alongside the TTS model.
# Honest reporting (MM2-03): the device is read from the pipe, and the
# dead "unload" button is explained by a note rather than left silent.
if mm.model is not None and getattr(mm.model, "_asr_pipe", None) is not None:
models.append({
"id": "asr",
"name": "WhisperX ASR",
"checkpoint": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
"device": _asr_device(),
"vram_mb": 0,
"unloadable": False,
"note": "released with the TTS model",
})
# 3. Diarization pipeline
if mm._diar_pipeline is not None:
models.append({
"id": "diarization",
"name": "Pyannote Diarization",
"checkpoint": "pyannote/speaker-diarization-3.1",
"device": get_best_device(),
"vram_mb": 0,
"unloadable": True,
})
# 4. Subprocess engine sidecars — each holds a process (and on GPU, VRAM)
# until idle-reaped. VRAM is reported by the child itself when available
# (MM2-08); 0 means CPU-only or not-yet-measured. Enumeration must never
# break the panel.
try:
from services.subprocess_backend import list_live_sidecars
for s in list_live_sidecars():
models.append({
"id": f"sidecar:{s['id']}",
"name": f"{s['id']} (sidecar)",
"checkpoint": s["id"],
"device": get_best_device(),
"vram_mb": round(float(s.get("vram_mb") or 0), 1),
"unloadable": True,
})
except Exception:
pass
return {"models": models, "count": len(models)}
async def unload(model_id: str) -> dict:
"""Unload one model by id. Preserves the original per-id response shapes.
``tts`` | ``diarization`` | ``sidecar:<id>`` | ``sidecars``. Raises
ValueError for an unknown id (router maps to HTTP 400)."""
if model_id == "sidecars" or model_id.startswith("sidecar:"):
from services.subprocess_backend import unload_all_sidecars, unload_sidecar
n = unload_all_sidecars() if model_id == "sidecars" else unload_sidecar(model_id.split(":", 1)[1])
return {"unloaded": model_id, "success": n > 0, "count": n,
**({} if n > 0 else {"reason": "not running or busy"})}
if model_id == "tts":
async with mm._model_lock:
if mm.model is not None:
mm.model = None
mm.free_vram()
return {"unloaded": "tts", "success": True}
return {"unloaded": "tts", "success": False, "reason": "not loaded"}
if model_id == "diarization":
if mm._diar_pipeline is not None:
mm._diar_pipeline = None
mm.free_vram()
return {"unloaded": "diarization", "success": True}
return {"unloaded": "diarization", "success": False, "reason": "not loaded"}
raise ValueError(f"Unknown model id: {model_id}")
async def unload_all() -> dict:
"""Release every releasable model — in-process TTS + diarization + all
sidecars. Convenience for app shutdown / a global flush."""
results = {}
for mid in ("tts", "diarization", "sidecars"):
try:
results[mid] = await unload(mid)
except Exception as exc: # noqa: BLE001
results[mid] = {"unloaded": mid, "success": False, "reason": str(exc)}
return {"unloaded_all": True, "results": results}
def free_vram() -> None:
"""One import surface for callers that just want to drop GPU caches."""
mm.free_vram()
+21 -273
View File
@@ -2,7 +2,6 @@ import os
import time
import asyncio
import logging
import threading
from concurrent.futures import ThreadPoolExecutor
# ── Lazy imports ─────────────────────────────────────────────────────
@@ -111,8 +110,7 @@ def __getattr__(name: str):
model = None # type: ignore
_model_lock = asyncio.Lock()
_last_used = time.time()
# Idle timeout is resolved per-tick in _resolve_idle_timeout() (MM2-05) from
# prefs/env/core.config — no module-level duplicate of IDLE_TIMEOUT_SECONDS.
_IDLE_TIMEOUT_SECONDS = IDLE_TIMEOUT_SECONDS
# ── Loading sub-stage tracker ────────────────────────────────────────
# Updated by _load_model_sync() so get_model_status() can report
@@ -197,23 +195,12 @@ def get_best_device():
"""Detect the best available compute device.
Priority: CUDA/ROCm > Intel XPU > DirectML > MPS > CPU
The *family* decision delegates to ``core.device_caps.detect_host_caps()``
(the single source of truth) so the probe and this loader can never
disagree. This function keeps the side-effects the probe deliberately
avoids: the ROCm ``HSA_OVERRIDE_GFX_VERSION`` env override and the
DirectML device-string return (DirectML is not a torch device family, so
the probe reports it as ``cpu`` we still resolve the real device string
here for Windows DirectML users). The string contract is unchanged:
``"cuda"`` / ``"xpu"`` / a DirectML device string / ``"mps"`` / ``"cpu"``.
"""
from core.device_caps import detect_host_caps
torch = _lazy_torch()
family = detect_host_caps().family
# ── NVIDIA CUDA or AMD ROCm (both present through torch.cuda) ─────
if family in ("cuda", "rocm"):
# ── NVIDIA CUDA or AMD ROCm ──────────────────────────────────────
# ROCm-enabled PyTorch reports through torch.cuda, so this covers both.
if torch.cuda.is_available():
_configure_rocm_if_needed(torch)
compatible, warning = check_device_compatibility()
if not compatible:
@@ -221,23 +208,15 @@ def get_best_device():
return "cuda"
# ── Intel Arc / discrete GPU via IPEX ────────────────────────────
if family == "xpu":
try:
try:
import intel_extension_for_pytorch # noqa: F401
if hasattr(torch, "xpu") and torch.xpu.is_available():
logger.info("Using Intel XPU device: %s", torch.xpu.get_device_name(0))
except Exception:
logger.info("Using Intel XPU device")
return "xpu"
return "xpu"
except ImportError:
pass
# ── Apple Silicon MPS ────────────────────────────────────────────
# Checked BEFORE DirectML to mirror the probe's family-priority order
# (cuda > rocm > xpu > mps; DirectML is not a torch family) so the loader
# and detect_host_caps() never disagree on a host that somehow exposes both.
if family == "mps":
return "mps"
# ── DirectML — universal Windows GPU (probe reports this as "cpu") ─
# Reached only when no torch family was detected (family == "cpu"), which is
# exactly the DirectML case — the probe classifies DirectML hosts as cpu.
# ── DirectML — universal Windows GPU (AMD, Intel, NVIDIA fallback)
try:
import torch_directml
if torch_directml.device_count() > 0:
@@ -246,184 +225,12 @@ def get_best_device():
except ImportError:
pass
# ── Apple Silicon MPS ────────────────────────────────────────────
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
return "mps"
return "cpu"
_COMPILE_ERR_MODULE_PREFIXES = ("torch._dynamo", "torch._inductor", "torch.fx", "triton")
_COMPILE_ERR_TB_MARKERS = ("/_dynamo/", "/_inductor/", "/triton/", "torch/fx/")
_COMPILE_ERR_MSG_MARKERS = (
"dynamo", "inductor", "triton", "cudagraph",
"symbolically trace", "torch.compile", "fx graph",
)
def _is_compile_runtime_failure(exc: BaseException) -> bool:
"""True when an exception originates in the torch.compile stack (Dynamo /
Inductor / Triton / FX / CUDA-graph trees) rather than in the model itself.
#278: on GPU architectures Triton doesn't support yet (e.g. Blackwell
sm_120), the compiled model dies mid-generation with errors like
"Detected that you are using FX to symbolically trace a dynamo-optimized
function" or an AssertionError out of torch/_inductor/cudagraph_trees.py.
Walks the exception chain and checks (a) the exception type's module,
(b) the message, (c) the traceback file paths the cudagraph case is a
bare AssertionError, so the traceback check is load-bearing.
"""
import traceback as _tb
seen: set[int] = set()
cur: BaseException | None = exc
while cur is not None and id(cur) not in seen:
seen.add(id(cur))
mod = type(cur).__module__ or ""
if mod.startswith(_COMPILE_ERR_MODULE_PREFIXES):
return True
msg = str(cur).lower()
if any(marker in msg for marker in _COMPILE_ERR_MSG_MARKERS):
return True
try:
for frame in _tb.extract_tb(cur.__traceback__):
filename = (frame.filename or "").replace("\\", "/")
if any(marker in filename for marker in _COMPILE_ERR_TB_MARKERS):
return True
except Exception as traceback_scan_error:
logging.debug(
"Skipping traceback marker scan while classifying compile runtime failure: %s",
traceback_scan_error,
)
# Follow the chain, honoring `raise ... from None` (the eager-retry
# path suppresses the original compile error so a genuine eager
# failure isn't misclassified as a compile failure).
if cur.__cause__ is not None:
cur = cur.__cause__
elif not cur.__suppress_context__:
cur = cur.__context__
else:
cur = None
return False
def _install_compile_fallback(_model) -> None:
"""Wrap ``model.generate`` so a torch.compile failure at inference time
falls back to the eager (uncompiled) model instead of failing the
generation (#278).
All TTS paths (generate, archetype previews, dub, stream, batch) funnel
through ``model.generate``, so this is the single choke point. On a
compile-stack failure we: log a clear warning, restore the eager module
(``OptimizedModule._orig_mod``), disable compile for the rest of the
session via ``engine_env.mark_compile_runtime_failure``, reset dynamo
state, and retry the call once eagerly. Non-compile errors (real OOM,
validation, ) propagate unchanged fully backward compatible for users
whose torch.compile works.
"""
orig_generate = _model.generate
def _generate_with_compile_fallback(*args, **kwargs):
try:
return orig_generate(*args, **kwargs)
except Exception as exc:
compiled = getattr(_model, "llm", None)
eager = getattr(compiled, "_orig_mod", None)
if eager is None or not _is_compile_runtime_failure(exc):
raise
logger.warning(
"torch.compile runtime failure during generation (%s: %s) — "
"falling back to the eager model and disabling torch.compile "
"for this session. Generation is being retried without it.",
type(exc).__name__, exc,
)
from services import engine_env
engine_env.mark_compile_runtime_failure(f"{type(exc).__name__}: {exc}")
_model.llm = eager
try:
torch = _lazy_torch()
torch._dynamo.reset()
except Exception as reset_exc:
logger.debug(
"Non-fatal: failed to reset torch._dynamo state after compile failure (%s: %s). "
"Continuing with eager fallback.",
type(reset_exc).__name__,
reset_exc,
)
try:
return orig_generate(*args, **kwargs)
except Exception as eager_exc:
# `from None` so a genuine eager failure (e.g. a real OOM)
# isn't chained to — and misclassified as — the compile error.
raise eager_exc from None
_model.generate = _generate_with_compile_fallback
# ── #315: thread affinity for cudagraph-compiled models ─────────────────────
# `torch.compile(mode="reduce-overhead")` captures CUDA graphs, and captured
# graph state is **thread-local** (torch/_inductor/cudagraph_trees keys its
# tree manager off the capturing thread). The `_gpu_pool` runs up to
# `_GPU_WORKER_CAP` threads, so render #1 captures the graph on worker A and a
# later render dispatched to worker B replays against mismatched cudagraph
# state — silently corrupting the audio (static / slowed playback, no
# exception, so the #278 eager fallback never fires). Fix: every call into a
# cudagraph-compiled model executes on ONE dedicated thread; uncompiled
# models (CPU / MPS / Windows-no-Triton / compile-disabled) keep the full pool.
_TORCH_COMPILE_MODE = "reduce-overhead"
# Compile modes that enable CUDA graphs under the hood — these need the
# single-thread affinity below. "default" / "max-autotune-no-cudagraphs"
# would not.
_CUDAGRAPH_COMPILE_MODES = frozenset({"reduce-overhead", "max-autotune"})
_compiled_inference_executor: "ThreadPoolExecutor | None" = None
_compiled_inference_thread_ident: "int | None" = None
def _get_compiled_inference_executor() -> ThreadPoolExecutor:
"""The single-thread executor that owns ALL inference on a compiled model.
Created lazily the first time a model is compiled with a cudagraph mode;
reused across model reloads (idle unload reload keeps the same thread,
which is fine a fresh compile simply captures its graphs there too).
The worker is spun up eagerly so its thread ident is known for the
re-entrancy guard in `_install_compile_thread_affinity`.
"""
global _compiled_inference_executor, _compiled_inference_thread_ident
if _compiled_inference_executor is None:
_compiled_inference_executor = ThreadPoolExecutor(
max_workers=1, thread_name_prefix="compiled-infer",
)
_compiled_inference_thread_ident = _compiled_inference_executor.submit(
threading.get_ident
).result()
return _compiled_inference_executor
def _install_compile_thread_affinity(_model) -> None:
"""Pin every ``model.generate`` call to the dedicated compile thread (#315).
Wraps ``model.generate`` (the single choke point all TTS paths funnel
through generate, archetype previews, dub, stream, batch) so the call
body always runs on `_get_compiled_inference_executor()`'s one thread.
That makes the thread that *captures* the CUDA graph on the first render
and the thread that *replays* it on every later render the same thread,
deterministically, regardless of which `_gpu_pool` worker dispatched it.
Installed AFTER `_install_compile_fallback`, so the call-time order is:
caller thread hop to the dedicated thread eager-fallback wrapper
real generate (the #278 classification/retry also runs on the dedicated
thread, with native tracebacks). The hop is a no-op when already on the
dedicated thread a 1-worker executor submitting to itself would
deadlock, so the re-entrancy guard is load-bearing.
"""
executor = _get_compiled_inference_executor()
inner_generate = _model.generate
def _generate_on_compile_thread(*args, **kwargs):
if threading.get_ident() == _compiled_inference_thread_ident:
return inner_generate(*args, **kwargs)
return executor.submit(inner_generate, *args, **kwargs).result()
_model.generate = _generate_on_compile_thread
def _set_loading(sub_stage: str, detail: str = "", error: str | None = None, progress: float | None = None):
"""Update the loading detail dict atomically."""
_loading_detail["sub_stage"] = sub_stage
@@ -483,23 +290,9 @@ def _load_model_sync():
logger.info("Preloading PyTorch Whisper with TTS model.")
else:
logger.info("Skipping PyTorch Whisper preload; ASR will load on demand.")
try:
_model = OmniVoice.from_pretrained(
checkpoint, device_map=device, dtype=torch.float16, load_asr=preload_asr,
)
except OSError as e:
# #352: a truncated HF cache surfaces here as "does not appear to
# have a file named pytorch_model.bin or model.safetensors".
# Translate to an actionable message instead of the raw
# transformers error.
if "does not appear to have a file named" in str(e):
raise RuntimeError(
f"The TTS model cache for {checkpoint} is incomplete "
"(weights missing — usually an interrupted download). "
"Open Settings → Models, delete the OmniVoice TTS model, "
"and install it again."
) from e
raise
_model = OmniVoice.from_pretrained(
checkpoint, device_map=device, dtype=torch.float16, load_asr=preload_asr,
)
try:
# plan-02 (#65): gate on Triton availability (+ user setting), not
@@ -510,38 +303,8 @@ def _load_model_sync():
if should_torch_compile(device):
_set_loading("compiling", "Compiling model (torch.compile)…")
try:
_model.llm = torch.compile(_model.llm, mode=_TORCH_COMPILE_MODE)
except Exception as compile_exc:
# #278: compile is an optimization, never a point of
# failure — keep the eager model and remember the failure
# so later loads this session skip compile up front.
from services.engine_env import mark_compile_runtime_failure
mark_compile_runtime_failure(f"{type(compile_exc).__name__}: {compile_exc}")
logger.warning(
"torch.compile failed (%s) — continuing with the eager model.",
compile_exc,
)
else:
# Compilation is lazy: Dynamo/Inductor/Triton can still
# blow up on the first *forward* (e.g. unsupported new GPU
# archs, #278). Wrap generate so that falls back to eager
# instead of failing the generation.
_install_compile_fallback(_model)
if _TORCH_COMPILE_MODE in _CUDAGRAPH_COMPILE_MODES:
# #315: reduce-overhead uses CUDA graphs, whose
# captured state is thread-local. Pin all inference to
# one dedicated thread so a later render dispatched to
# a different _gpu_pool worker can't replay a graph it
# didn't capture (static / slowed audio from the 2nd
# render onward).
_install_compile_thread_affinity(_model)
logger.info(
"torch.compile mode %r uses CUDA graphs — compiled-model "
"inference pinned to a single dedicated thread (#315).",
_TORCH_COMPILE_MODE,
)
logger.info("torch.compile applied.")
_model.llm = torch.compile(_model.llm, mode="reduce-overhead")
logger.info("torch.compile applied.")
except Exception as e:
logger.info("torch.compile skipped: %s", e)
@@ -680,28 +443,13 @@ def get_model_status():
result["error"] = err
return result
def _resolve_idle_timeout() -> float:
"""In-process model idle timeout in seconds (MM2-05): prefs store → env →
core.config default, env winning. Resolved per-tick so a settings change
takes effect without a restart."""
try:
from core import prefs
return float(prefs.resolve(
"idle_timeout_seconds",
env="OMNIVOICE_IDLE_TIMEOUT_S",
default=IDLE_TIMEOUT_SECONDS,
))
except (TypeError, ValueError, ImportError):
return float(IDLE_TIMEOUT_SECONDS)
async def idle_worker():
global model
torch = _lazy_torch()
while True:
await asyncio.sleep(30)
async with _model_lock:
if model is not None and time.time() - _last_used > _resolve_idle_timeout():
if model is not None and time.time() - _last_used > _IDLE_TIMEOUT_SECONDS:
logger.info("Idle timeout reached. Unloading OmniVoice model to free VRAM.")
model = None
free_vram()
-187
View File
@@ -1,187 +0,0 @@
"""
Speech-onset alignment for transcript segments (issue #280, item 1).
Whisper-family ASR models are prone to stretching a segment's *start* back
over leading non-speech (intro music, room tone, silence). The classic
symptom from the issue report: the speaker starts talking at 0:020:03,
but the first transcript segment says ``start=0.0`` so the dubbed line
plays the moment the video begins and everything feels desynchronised.
``snap_segment_starts`` post-processes segments against the actual audio
(ideally the Demucs-isolated vocals track, which the dub pipeline already
produces): for each segment it scans the waveform inside ``[start, end]``
for the first frame whose RMS rises above an adaptive threshold and moves
``start`` forward to just before that onset.
Design constraints:
* **Forward-only.** A segment start is never moved earlier that could
collide with the previous speaker. We only trim leading non-speech.
* **Conservative.** Shifts below ``min_shift_s`` are ignored (word-level
timestamps are usually within ~100 ms already); a minimum segment
duration is always preserved; segments whose window looks silent
(no frame above the absolute floor) are left untouched.
* **Pure NumPy.** No model, no platform-specific code identical
behaviour on macOS / Windows / Linux, trivially unit-testable.
"""
from __future__ import annotations
import logging
from typing import Sequence
import numpy as np
logger = logging.getLogger("omnivoice.onset_align")
# Analysis frame for RMS energy. 20 ms is fine-grained enough to localise
# a syllable onset while staying cheap (a 10-min track is ~30k frames).
FRAME_S = 0.02
# Keep this much audio before the detected onset so plosives/breaths that
# sit just under the threshold aren't clipped off.
PRE_ROLL_S = 0.05
# Shifts smaller than this are noise — word-level ASR timestamps are
# usually accurate to ~0.1 s, so don't churn segment data for less.
MIN_SHIFT_S = 0.15
# Never shrink a segment below this duration when shifting its start.
MIN_SEG_DUR_S = 0.30
# A frame must exceed `RELATIVE_THRESHOLD × peak RMS of the window` to
# count as speech onset…
RELATIVE_THRESHOLD = 0.10
# …and the window's peak RMS must exceed this absolute floor, otherwise
# the whole window is treated as silence and left alone (we'd only be
# snapping to noise).
ABS_RMS_FLOOR = 1e-3
def _frame_rms(x: np.ndarray, frame_len: int) -> np.ndarray:
"""RMS per non-overlapping frame; the ragged tail frame is dropped."""
n = (len(x) // frame_len) * frame_len
if n <= 0:
return np.zeros(0, dtype=np.float32)
frames = x[:n].reshape(-1, frame_len).astype(np.float64, copy=False)
return np.sqrt((frames * frames).mean(axis=1)).astype(np.float32)
def detect_speech_onset(
audio: np.ndarray,
sr: int,
start_s: float,
end_s: float,
) -> float | None:
"""Return the absolute time (s) of the first speech-like frame inside
``[start_s, end_s]``, or ``None`` when the window is empty / silent.
"""
if sr <= 0 or end_s <= start_s:
return None
i0 = max(0, int(start_s * sr))
i1 = min(len(audio), int(end_s * sr))
if i1 <= i0:
return None
window = audio[i0:i1]
frame_len = max(1, int(FRAME_S * sr))
rms = _frame_rms(window, frame_len)
if rms.size == 0:
return None
peak = float(rms.max())
if peak < ABS_RMS_FLOOR:
return None # whole window is effectively silent
threshold = max(RELATIVE_THRESHOLD * peak, ABS_RMS_FLOOR)
above = np.nonzero(rms >= threshold)[0]
if above.size == 0:
return None
return start_s + float(above[0]) * (frame_len / sr)
# Hysteresis for full-track onset listing: after a frame crosses the
# threshold, the energy must stay *below* it for at least this long before
# the next rise counts as a new onset. Stops syllable-internal dips from
# spamming the timeline with ticks.
MIN_ONSET_GAP_S = 0.15
def detect_speech_onsets(audio: np.ndarray, sr: int) -> list[float]:
"""Return the times (s) of every speech-like onset across the whole track.
Powers the timeline editor's snap-to-onset ticks (issue #280, item 3):
frame RMS over the full track, single adaptive threshold
``max(RELATIVE_THRESHOLD × peak, ABS_RMS_FLOOR)``, and hysteresis a
new onset registers only when the energy rises above the threshold
after at least ``MIN_ONSET_GAP_S`` below it.
Pure NumPy, identical behaviour on every platform. Returns ``[]`` for
empty/silent audio.
"""
if sr <= 0 or audio is None or len(audio) == 0:
return []
if audio.ndim > 1:
audio = audio.mean(axis=1)
frame_len = max(1, int(FRAME_S * sr))
rms = _frame_rms(audio, frame_len)
if rms.size == 0:
return []
peak = float(rms.max())
if peak < ABS_RMS_FLOOR:
return [] # whole track is effectively silent
threshold = max(RELATIVE_THRESHOLD * peak, ABS_RMS_FLOOR)
gap_frames = max(1, int(round(MIN_ONSET_GAP_S / FRAME_S)))
frame_s = frame_len / sr
onsets: list[float] = []
below_run = gap_frames # armed, so speech at t=0 still counts
for i, v in enumerate(rms):
if v >= threshold:
if below_run >= gap_frames:
onsets.append(round(i * frame_s, 3))
below_run = 0
else:
below_run += 1
return onsets
def snap_segment_starts(
segments: Sequence[dict],
audio: np.ndarray,
sr: int,
*,
min_shift_s: float = MIN_SHIFT_S,
) -> int:
"""Snap each segment's ``start`` forward to the actual speech onset.
Mutates the segment dicts in place (the shape the dub pipeline passes
around). Returns the number of segments adjusted.
``audio`` should be mono float; the Demucs vocals track gives the best
signal but the mixed track still beats nothing.
"""
if sr <= 0 or audio is None or len(audio) == 0:
return 0
if audio.ndim > 1:
audio = audio.mean(axis=1)
adjusted = 0
for seg in segments:
try:
start = float(seg.get("start", 0.0))
end = float(seg.get("end", 0.0))
except (TypeError, ValueError):
continue
if end - start < MIN_SEG_DUR_S + min_shift_s:
continue # too short for a meaningful shift
onset = detect_speech_onset(audio, sr, start, end)
if onset is None:
continue
new_start = max(start, onset - PRE_ROLL_S)
shift = new_start - start
if shift < min_shift_s:
continue
# Preserve a minimum playable duration.
new_start = min(new_start, end - MIN_SEG_DUR_S)
if new_start - start < min_shift_s:
continue
seg["start"] = round(new_start, 3)
adjusted += 1
if adjusted:
logger.info("onset-align: snapped %d/%d segment start(s) to speech onset",
adjusted, len(segments))
return adjusted
-436
View File
@@ -1,436 +0,0 @@
"""`.ovsvoice` persona-bundle format (#29 / parity §R3 G1).
A portable ZIP that packages a voice profile's identity + an optional reference
clip + a consent attestation + an SPDX license tag + a watermarked preview.
This module owns the **pure, model-free** core: the format constants, SPDX
normalization, and the manifest/consent builders. The audio preview + ZIP
pack/unpack (which lazily import torchaudio/watermark) layer on top of these.
"""
from __future__ import annotations
import io
import json
import os
import re
import time
import zipfile
from dataclasses import dataclass
from typing import Optional
# ── Format constants ─────────────────────────────────────────────────────────
OVSVOICE_FORMAT = "ovsvoice"
OVSVOICE_SCHEMA_VERSION = 1
MAX_BUNDLE_BYTES = 100 * 1024 * 1024 # 100 MB (mirrors marketplace cap)
_MIN_CONSENT_AUDIO_BYTES = 1000 # the consent-recording floor
DEFAULT_LICENSE = "LicenseRef-OmniVoice-Personal"
PREVIEW_MAX_SECONDS = 8.0 # preview length cap (A6)
PREVIEW_SAMPLE_RATE = 24_000 # preview rate; mono, 16-bit PCM (A8)
# Audio member prefixes the importer recognises. The ZIP member NAME is never
# used to build an output path (zip-slip safe, B10) — only its extension, and
# only after the linear allowlist below.
_AUDIO_MEMBER_PREFIXES = ("ref_audio", "locked_audio", "consent_audio", "preview")
# Reused verbatim from profiles.py:306 — single linear quantifier, no ReDoS.
_MEMBER_EXT_RE = re.compile(r"^\.[A-Za-z0-9]{1,8}$")
# Membership allowlist for SPDX validation — a fixed-string set + the
# ``LicenseRef-`` prefix. NO regex over the (user-supplied) SPDX string, so this
# carries no CodeQL py/polynomial-redos surface.
_SPDX_ALLOWLIST: frozenset[str] = frozenset({
"CC0-1.0", "CC-BY-4.0", "CC-BY-SA-4.0", "CC-BY-NC-4.0", "CC-BY-NC-SA-4.0",
"CC-BY-ND-4.0", "MIT", "Apache-2.0", "LicenseRef-OmniVoice-Personal",
})
class BundleError(Exception):
"""A bundle build/parse failure carrying the HTTP status the router maps to."""
def __init__(self, status: int, detail: str):
super().__init__(detail)
self.status = status
self.detail = detail
class NoPreviewSource(Exception):
"""No readable source clip exists to build a preview from (A2/A3/A4/A5/A12).
The router maps this to HTTP 503."""
def _safe_member_ext(member_name: str) -> str:
"""The extension for a ZIP member, allowlisted to ``^\\.[A-Za-z0-9]{1,8}$``
(else ``.wav``). Used ONLY to choose the output extension never the path
(B11). Linear regex, no ReDoS."""
ext = os.path.splitext(member_name)[1]
return ext if _MEMBER_EXT_RE.match(ext) else ".wav"
def normalize_spdx(spdx: Optional[str]) -> str:
"""Return a safe SPDX id: the value if it's allowlisted or a ``LicenseRef-``
custom id, else :data:`DEFAULT_LICENSE`. Never raises, never 400s a junk
id (incl. shell-injection attempts) normalizes to the default. Membership /
fixed-prefix only no regex (CodeQL-clean)."""
if not spdx or not isinstance(spdx, str):
return DEFAULT_LICENSE
s = spdx.strip()
if s in _SPDX_ALLOWLIST or s.startswith("LicenseRef-"):
return s
return DEFAULT_LICENSE
def build_manifest(
profile: dict,
*,
license_spdx: str,
tags: list[str],
engine_id: str = "",
custom_license_text: Optional[str] = None,
preview: Optional[dict] = None,
members: Optional[dict] = None,
omnivoice_version: str = "",
) -> dict:
"""Build the ``manifest.json`` object for a profile row. Mirrors the legacy
``_bundle_metadata`` persona fields, adds the format discriminator, license
(normalized never raises on a bad id, A19), tags, preview + members blocks.
Pure: no I/O, no model."""
return {
"format": OVSVOICE_FORMAT,
"schema_version": OVSVOICE_SCHEMA_VERSION,
"omnivoice_version": omnivoice_version or "",
"exported_at": time.time(),
"persona": {
"name": profile.get("name") or "",
"kind": profile.get("kind") or "clone",
"language": profile.get("language") or "Auto",
"personality": profile.get("personality") or "",
"instruct": profile.get("instruct") or "",
"ref_text": profile.get("ref_text") or "",
"seed": profile.get("seed"), # int or None (A16)
"is_locked": bool(profile.get("is_locked")),
"vd_states": profile.get("vd_states"), # JSON string or None (A15) — never re-parsed
},
"engine": {"id": engine_id or "", "design_params": None},
"license": {"spdx": normalize_spdx(license_spdx), "custom_text": custom_license_text or None},
"tags": list(tags or []),
"preview": preview, # set by the audio step; None for legacy/no-preview
"members": members or {"ref_audio": None, "locked_audio": None, "consent_audio": None},
}
def build_consent_json(profile: dict, *, has_recording: bool) -> Optional[dict]:
"""The optional ``consent.json`` for a profile, or None when there's nothing
to attest. A ``design`` persona attests as designed-synthetic by definition;
a verified clone attests as a self-recorded statement. Import treats these
fields as ADVISORY real verification needs the actual consent_audio member
(see the import rules), so this can't forge verified-own-voice."""
kind = profile.get("kind") or "clone"
consent_text = (profile.get("consent_text") or "").strip()
verified = bool(profile.get("verified_own_voice"))
if kind == "design":
method = "designed-synthetic"
verified = True
elif verified or consent_text or has_recording:
method = "self-recorded-statement"
else:
return None # nothing to attest
recorded_at = profile.get("consent_recorded_at")
try:
recorded_at = float(recorded_at)
except (TypeError, ValueError):
recorded_at = time.time()
return {
"verified_own_voice": verified,
"method": method,
"consent_text": consent_text,
"recorded_at": recorded_at,
"has_recording": bool(has_recording),
}
def _legacy_metadata(profile: dict, omnivoice_version: str) -> dict:
"""A ``metadata.json`` payload shaped like marketplace ``_bundle_metadata`` so
an OLDER OmniVoice (which only reads metadata.json) can still import the ref
audio from a ``.ovsvoice`` bundle."""
return {
"bundle_version": 1,
"profile_name": profile.get("name") or "",
"ref_text": profile.get("ref_text") or "",
"instruct": profile.get("instruct") or "",
"language": profile.get("language") or "Auto",
"personality": profile.get("personality") or "",
"seed": profile.get("seed"),
"kind": profile.get("kind") or "clone",
"vd_states": profile.get("vd_states"),
"is_locked": bool(profile.get("is_locked")),
"omnivoice_version": omnivoice_version or "",
}
def _resolve_voice_file(filename: Optional[str]) -> Optional[str]:
"""Resolve a DB-stored audio filename strictly inside VOICES_DIR, returning
an absolute path only if the file actually exists. None on missing/escape
mirrors profiles._voices_path (basename + realpath confinement, E1)."""
if not filename or os.path.basename(filename) != filename:
return None
from core.config import VOICES_DIR
root = os.path.realpath(VOICES_DIR)
path = os.path.realpath(os.path.join(root, filename))
if not path.startswith(root + os.sep):
return None
return path if os.path.isfile(path) else None
def _generate_preview(profile: dict, embed_fn) -> tuple[bytes, bool, float]:
"""Load the profile's source clip, downmix→mono, resample→24 kHz, trim ≤8 s,
watermark (forced), and return ``(wav_bytes, watermarked, duration_s)``.
Source precedence is locked-over-ref (profiles.py:230). Raises
:class:`NoPreviewSource` when neither clip is readable (A2-A5). All heavy
imports (torch/torchaudio/watermark) are lazy so the module stays model-free
at collection time (avoids the known local torch/Triton segfault)."""
import torch # noqa: F401 (torchaudio needs it loaded)
import torchaudio
from services.audio_io import _safe_torchaudio_save
from services.watermark import _check_available
candidates = [profile.get("locked_audio_path"), profile.get("ref_audio_path")]
wav = None
for name in candidates:
path = _resolve_voice_file(name)
if not path:
continue
try:
waveform, sr = torchaudio.load(path)
except Exception: # noqa: BLE001 — try the next candidate (A4)
continue
if waveform.numel() == 0: # empty/zero-length (A5)
continue
if waveform.shape[0] > 1: # downmix to mono (A7)
waveform = waveform.mean(dim=0, keepdim=True)
if sr != PREVIEW_SAMPLE_RATE: # resample (A8)
waveform = torchaudio.functional.resample(waveform, sr, PREVIEW_SAMPLE_RATE)
cap = int(PREVIEW_SAMPLE_RATE * PREVIEW_MAX_SECONDS)
waveform = waveform[:, :cap] # trim, shorter used whole (A6)
wav = waveform
break
if wav is None or wav.numel() == 0:
raise NoPreviewSource("no readable reference or locked audio for a preview")
# Forced watermark — bypasses the user pref but still no-ops without AudioSeal.
fn = embed_fn or _default_embed
wav = fn(wav, PREVIEW_SAMPLE_RATE)
watermarked = bool(_check_available()) # best-effort honesty (A11)
duration_s = round(wav.shape[-1] / PREVIEW_SAMPLE_RATE, 3)
buf = io.BytesIO()
_safe_torchaudio_save(buf, wav, PREVIEW_SAMPLE_RATE, format="wav", bits_per_sample=16)
return buf.getvalue(), watermarked, duration_s
def _default_embed(wav, sample_rate):
"""Default preview watermarker: services.watermark.embed_watermark(force=True)."""
from services.watermark import embed_watermark
return embed_watermark(wav, sample_rate, force=True)
def build_persona_bundle(
profile: dict,
*,
license_spdx: str = DEFAULT_LICENSE,
tags: Optional[list[str]] = None,
custom_license_text: Optional[str] = None,
include_reference: bool = True,
engine_id: str = "",
omnivoice_version: str = "",
embed_fn=None,
) -> bytes:
"""Assemble a ``.ovsvoice`` ZIP in memory and return its bytes.
Always writes a watermarked ``preview.wav`` + ``manifest.json`` +
(legacy-shaped) ``metadata.json``. Writes ``consent.json`` when there's
something to attest, the raw ``ref_audio``/``locked_audio`` members unless
``include_reference=False`` (privacy / preview-only, A12), and
``consent_audio`` when a recording exists. Raises :class:`NoPreviewSource`
(router 503) when no source clip is readable."""
preview_bytes, watermarked, duration_s = _generate_preview(profile, embed_fn)
members: dict = {"ref_audio": None, "locked_audio": None, "consent_audio": None}
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
if include_reference:
ref_path = _resolve_voice_file(profile.get("ref_audio_path"))
if ref_path:
name = f"ref_audio{os.path.splitext(ref_path)[1] or '.wav'}"
zf.write(ref_path, name)
members["ref_audio"] = name
locked_path = _resolve_voice_file(profile.get("locked_audio_path"))
if locked_path:
name = f"locked_audio{os.path.splitext(locked_path)[1] or '.wav'}"
zf.write(locked_path, name)
members["locked_audio"] = name
# Consent recording travels only when it exists and clears the floor.
consent_path = _resolve_voice_file(profile.get("consent_audio_path"))
has_recording = False
if consent_path and os.path.getsize(consent_path) >= _MIN_CONSENT_AUDIO_BYTES:
name = f"consent_audio{os.path.splitext(consent_path)[1] or '.wav'}"
zf.write(consent_path, name)
members["consent_audio"] = name
has_recording = True
zf.writestr("preview.wav", preview_bytes)
preview_block = {
"file": "preview.wav", "watermarked": watermarked,
"duration_s": duration_s, "sample_rate": PREVIEW_SAMPLE_RATE,
}
manifest = build_manifest(
profile, license_spdx=license_spdx, tags=tags or [],
engine_id=engine_id, custom_license_text=custom_license_text,
preview=preview_block, members=members,
omnivoice_version=omnivoice_version,
)
zf.writestr("manifest.json", json.dumps(manifest, ensure_ascii=False, indent=2))
zf.writestr("metadata.json",
json.dumps(_legacy_metadata(profile, omnivoice_version),
ensure_ascii=False, indent=2))
consent = build_consent_json(profile, has_recording=has_recording)
if consent is not None:
zf.writestr("consent.json", json.dumps(consent, ensure_ascii=False, indent=2))
return buf.getvalue()
@dataclass
class ParsedPersona:
manifest: dict # parsed manifest.json OR synthesized from metadata.json
consent: Optional[dict] # parsed consent.json, or None
is_legacy: bool # only metadata.json was found (B6/B23)
schema_version_ahead: bool # manifest.schema_version > OVSVOICE_SCHEMA_VERSION (B7)
license_spdx: str # normalized (B21)
preview_only: bool # only preview.wav, no ref/locked member (A12/B8)
members: dict # {prefix: member_name} for audio members present
watermarked_preview: bool # manifest.preview.watermarked (False for legacy)
_zip: zipfile.ZipFile # open handle; router extracts via extract_member()
def member_ext(self, prefix: str) -> str:
name = self.members.get(prefix)
return _safe_member_ext(name) if name else ".wav"
def extract_member(self, prefix: str, dest_path: str) -> bool:
"""Stream the audio member named by ``prefix`` to ``dest_path`` (a path
the CALLER derived from a server-generated id never from the member
name). Returns False when the member is absent. Last-wins on dup (B9)."""
name = self.members.get(prefix)
if not name:
return False
import shutil
with self._zip.open(name) as src, open(dest_path, "wb") as dst:
shutil.copyfileobj(src, dst)
return True
def parse_persona_bundle(content: bytes) -> ParsedPersona:
"""Validate the ZIP and read manifest/consent WITHOUT touching the DB or
writing files. Raises :class:`BundleError` (400|413) for B1-B11. The caller
must use the returned ``ParsedPersona`` while the process holds ``content``
(the open ZIP reads from the in-memory bytes)."""
if len(content) > MAX_BUNDLE_BYTES:
raise BundleError(413, f"Bundle too large. Max is {MAX_BUNDLE_BYTES} bytes.")
try:
zf = zipfile.ZipFile(io.BytesIO(content))
except zipfile.BadZipFile:
raise BundleError(400, "not a valid ZIP bundle")
names = [n for n in zf.namelist() if not n.endswith("/")]
# Manifest selection: prefer manifest.json, fall back to legacy metadata.json.
manifest: dict = {}
is_legacy = False
if "manifest.json" in names:
try:
manifest = json.loads(zf.read("manifest.json"))
except (ValueError, UnicodeDecodeError):
raise BundleError(400, "manifest is not valid JSON")
if not isinstance(manifest, dict):
raise BundleError(400, "manifest is not valid JSON")
# A bundle whose format is neither ovsvoice nor absent → still read
# leniently (B6); we only branch on schema_version below.
elif "metadata.json" in names:
is_legacy = True
try:
legacy = json.loads(zf.read("metadata.json"))
except (ValueError, UnicodeDecodeError):
raise BundleError(400, "manifest is not valid JSON")
if not isinstance(legacy, dict):
raise BundleError(400, "manifest is not valid JSON")
manifest = {
"format": "omnivoice-legacy",
"schema_version": OVSVOICE_SCHEMA_VERSION,
"persona": {
"name": legacy.get("profile_name") or legacy.get("name") or "Imported Voice",
"kind": legacy.get("kind") or "clone",
"language": legacy.get("language") or "Auto",
"personality": legacy.get("personality") or "",
"instruct": legacy.get("instruct") or "",
"ref_text": legacy.get("ref_text") or "",
"seed": legacy.get("seed"),
"is_locked": bool(legacy.get("is_locked")),
"vd_states": legacy.get("vd_states"),
},
"license": {"spdx": DEFAULT_LICENSE, "custom_text": None},
"tags": [],
"preview": None,
"members": {},
}
else:
raise BundleError(400, "bundle is missing a manifest")
# Audio members by prefix (last-wins on duplicates, B9). The member NAME is
# retained only to read bytes + pick an extension — never to build a path.
members: dict = {}
for name in names:
for prefix in _AUDIO_MEMBER_PREFIXES:
if os.path.basename(name).startswith(prefix):
members[prefix] = name
has_audio = any(p in members for p in ("ref_audio", "locked_audio", "preview"))
if not has_audio:
raise BundleError(400, "bundle has no audio member")
consent = None
if "consent.json" in names:
try:
parsed = json.loads(zf.read("consent.json"))
if isinstance(parsed, dict):
consent = parsed
except (ValueError, UnicodeDecodeError):
consent = None # advisory only — a bad consent.json never 400s
schema_version = manifest.get("schema_version", OVSVOICE_SCHEMA_VERSION)
try:
ahead = int(schema_version) > OVSVOICE_SCHEMA_VERSION
except (TypeError, ValueError):
ahead = False
preview_block = manifest.get("preview") or {}
watermarked_preview = bool(preview_block.get("watermarked")) if isinstance(preview_block, dict) else False
license_spdx = normalize_spdx((manifest.get("license") or {}).get("spdx"))
preview_only = ("preview" in members
and "ref_audio" not in members and "locked_audio" not in members)
return ParsedPersona(
manifest=manifest, consent=consent, is_legacy=is_legacy,
schema_version_ahead=ahead, license_spdx=license_spdx,
preview_only=preview_only, members=members,
watermarked_preview=watermarked_preview, _zip=zf,
)
__all__ = [
"OVSVOICE_FORMAT", "OVSVOICE_SCHEMA_VERSION", "MAX_BUNDLE_BYTES",
"PREVIEW_MAX_SECONDS", "PREVIEW_SAMPLE_RATE", "DEFAULT_LICENSE",
"BundleError", "NoPreviewSource", "ParsedPersona",
"normalize_spdx", "build_manifest", "build_consent_json",
"build_persona_bundle", "parse_persona_bundle",
]

Some files were not shown because too many files have changed in this diff Show More