Compare commits

..
Author SHA1 Message Date
debpalashandClaude Opus 4.8 362dca9ae1 docs(diagnostics): self-diagnosis section in troubleshooting + README pointer
Settings > About self-check / --diagnose / --deep / diagnostic bundle are
now the documented first step before the per-error entries — and the
support team's first ask on every issue.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:43:03 +05:30
debpalashandClaude Opus 4.8 bbac66ef76 feat(diagnostics): breadcrumbs, deep self-check, report sweep, issue search
Final slice of the bug-tracking work.

- toastErrorWithReport adopted at the high-traffic failure sites: TTS
  generation, dub upload/ingest/transcribe, engine install, engines-matrix
  load, voice profile save/delete/test, batch enqueue/cancel/delete.
  Validation toasts and cancellations stay plain on purpose.
- utils/breadcrumbs.js: local-only ring of the last 20 action names
  (closed-set names only — never content or paths), embedded as a
  'Recent actions' section in the prefilled report. Instrumented: view
  changes, generate, dub pipeline, export, engine switch.
- deep self-check: /system/diagnose?deep=true and --diagnose --deep load
  the active engine and synthesize a short utterance (num_step=4) —
  catches 'installed but broken'. 180s time-box, skips during model load,
  scrubbed failure detail. Verified live: cold-loaded omnivoice and
  produced 2.2s of audio in 43.9s on CUDA.
- 'Search similar issues' action on the ErrorBoundary: scrubbed,
  noise-stripped GitHub issue search URL — dedupe before filing.
- bug_report.md template now points at the diagnostic bundle and the
  --diagnose CLI so manual reports arrive with the same evidence.

Tests: pytest 107 passed (4 new deep-check tests, CJK gate green);
vitest 218 passed (breadcrumbs + issue-search suites); vite build green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:43:03 +05:30
debpalashandClaude Opus 4.8 3d74611be8 feat(diagnostics): error journal, diagnostic bundle, crash notice, global handlers
Second slice of the bug-tracking work — still zero outbound paths beyond
the user's own browser/file manager.

- core/error_journal.py: deduped ring of recent unhandled backend errors
  (fingerprint counts, error_class triage: GPU_OOM, HF_AUTH_FAILED,
  PYANNOTE_LICENSE_REQUIRED, DISK_FULL, FFMPEG_MISSING, NETWORK_ERROR),
  scrubbed, JSONL-persisted so the error that killed the last run survives
  restart. Wired into the global exception handler; 500 bodies now carry
  error_class; GET /system/errors/recent.
- core/diagnostic_bundle.py + POST /system/diagnostic-bundle + Settings >
  About 'Save diagnostic bundle': zip of self-check report, error journal,
  scrubbed log tails — drag onto a GitHub issue; bypasses the ~8k
  prefill-URL ceiling.
- crash-on-next-launch: /system/notifications flags a crash logged before
  this session started (size vs acked-size in prefs, mtime vs process
  start); POST /system/crash/ack; LogsFooter acks on action click.
- utils/globalErrorHandlers.js: uncaught errors + unhandled rejections get
  a throttled, noise-filtered 'Report this bug' toast.
- sidecar log parity fix: _tauri_log_candidates() now lists the Rust
  sidecar's backend.log/backend_err.log on Linux (XDG state dir) and
  Windows (LOCALAPPDATA) — sidecar crashes were only visible on macOS.

Tests: +19 pytest (journal, bundle); suite at 102 passed. Vitest 124
passed; vite build green. Live-verified: journal recorded and classified
a real HF 401 from the test run (HF_AUTH_FAILED, paths scrubbed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:43:03 +05:30
debpalashandClaude Opus 4.8 ee08f56e1b feat(diagnostics): local self-check + scrubbed bug-report pipeline
Closes the gap between 'something broke' and 'a useful GitHub issue
exists' — entirely within the local-first constraint: the only outbound
path remains the user's own browser opening a prefilled issues/new URL.

Backend:
- core/scrub.py: privacy scrubber for anything leaving the machine —
  env-var secret values (*TOKEN*|*KEY*|*SECRET*|*PASSWORD*), credential
  shapes (hf_/ghp_/github_pat_/sk-), home dirs on all three OSes
- core/diagnose.py: 9-check self-check (device+GPU, ffmpeg, HF token,
  disk, data-dir writability, RAM, engine registry, hub reachability),
  pre-scrubbed, ASCII-safe output
- GET /system/diagnose + 'python main.py --diagnose' (exit 0/1)
- /system/info: hardware inventory (os_version, cpu_model, cpu_count,
  ram_total_gb, gpu_name, vram_total_gb, disk_free_gb), cached statics

Frontend:
- utils/bugReport.js: single source for the prefilled-URL builder —
  scrubText twin, hardware context capture, scrubbed error+stack embed,
  URL-length cap; ReportBugButton refactored onto it
- ErrorBoundary 'Report this bug' action with the error attached
- utils/errorToast.jsx toastErrorWithReport(); wired into export toasts
- Settings > About 'Run self-check' with per-check status badges

Tests: 27 pytest (scrub, diagnose) + 15 vitest (bugReport); existing
suites green; verified live (--diagnose, TestClient, vite build).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:43:03 +05:30
916 changed files with 27437 additions and 129043 deletions
-32
View File
@@ -1,32 +0,0 @@
# RTK - Rust Token Killer (Google Antigravity)
**Usage**: Token-optimized CLI proxy for shell commands.
## Rule
Always prefix shell commands with `rtk` to minimize token consumption.
Examples:
```bash
rtk git status
rtk cargo test
rtk ls src/
rtk grep "pattern" src/
rtk find "*.rs" .
rtk docker ps
rtk gh pr list
```
## Meta Commands
```bash
rtk gain # Show token savings
rtk gain --history # Command history with savings
rtk discover # Find missed RTK opportunities
rtk proxy <cmd> # Run raw (no filtering, for debugging)
```
## Why
RTK filters and compresses command output before it reaches the LLM context, saving 60-90% tokens on common operations. Always use `rtk <cmd>` instead of raw commands.
+3 -119
View File
@@ -7,26 +7,12 @@
language: "en-US"
early_access: false
# The review voice: a panel of senior domain experts, not a linter.
tone_instructions: >-
Review as a panel of principal engineers: ML inference, audio DSP, desktop
systems, product polish. Cite exact lines, name the failure mode, give the
concrete fix. No filler praise; raise nits only when they change a decision.
reviews:
# "chill" keeps the bot from blocking merges — it comments, it does not gate.
# Hard gating lives in CI (security.yml) and the constitution's human bar.
profile: chill
request_changes_workflow: false
high_level_summary: true
# Every walkthrough gets a visual: mermaid sequence diagrams for the
# mechanics, plus (via the summary instructions) an ASCII before/after
# sketch when the PR touches UI — so each PR is reviewable at a glance.
sequence_diagrams: true
high_level_summary_instructions: >-
If the PR changes UI (JSX/TSX/CSS/Tauri windows), include a compact ASCII
before/after sketch of the affected layout or component. If it changes
behavior, include a short mermaid flowchart of the new mechanism.
review_status: true
poem: false
@@ -53,8 +39,7 @@ reviews:
- "!**/*.onnx"
- "!tests/fixtures/**"
# One expert lens per subsystem — encode what a passionate senior in each
# domain would actually check, beyond what linters and CI already gate.
# Encode the project's hard constraints so the bot reviews against them.
path_instructions:
- path: "**/*.{py,rs,js,jsx,ts,tsx}"
instructions: >-
@@ -63,119 +48,18 @@ reviews:
model download, or an explicitly opt-in endpoint. Flag any code that
persists or logs values matching *TOKEN*/*KEY*/*SECRET* or absolute user
home paths (/Users/<name>/, C:\\Users\\<name>\\).
- path: "backend/services/**/*.py"
instructions: >-
Review as an ML-inference/audio engineer. Check: thread-safety of model
and cache state across the GPU worker pool; device/dtype assumptions
that break on one of CUDA/MPS/ROCm/CPU; VRAM lifecycle (load/unload,
leaks on the error path); sample-rate, channel-count and tensor-shape
assumptions at engine boundaries; blocking calls inside async paths;
model download/cache behavior when offline. Engine code must stay
backward-compatible with already-installed on-disk model state.
- path: "backend/**/*.py"
instructions: >-
Default features must behave identically on macOS, Windows and Linux.
Platform-specific implementation is allowed, but a divergent user-visible
default is a P0 bug — flag it and suggest an opt-in (Settings/env/flag).
Any DB schema change must go through an alembic migration with an upgrade
path; flag direct schema edits. The backend serves loopback HTTP: treat
every query/path/form param as hostile (path traversal, log injection,
CSRF from a browser tab), and never route user-chosen filesystem
destinations through HTTP — that authorization belongs in the Tauri
process.
- path: "frontend/src/**/*.{js,jsx,ts,tsx}"
instructions: >-
Review as a product-minded senior frontend engineer. Check: stale state
and races (async results landing after unmount or after newer requests);
every user-visible failure has an actionable, non-technical error
message; loading/disabled states during long operations. Every new
user-facing string must be an i18n t('...') key present in ALL 21
frontend/src/i18n/locales/*.json files — flag hardcoded UI strings and
keys missing from any locale.
- path: "frontend/src-tauri/**/*.rs"
instructions: >-
Review as a desktop-systems engineer. Check: every #[tauri::command] is
callable from the webview — validate inputs and scope filesystem/process
access accordingly; window and webview lifecycle on all three OSes;
child-process spawn/exit-code/stderr handling; no unwrap/expect on
user-controlled input; platform cfg blocks keep user-visible defaults
identical across macOS/Windows/Linux.
- path: "tests/**/*.py"
instructions: >-
Review as a test-infrastructure engineer. Check: the test would fail
before the fix and pass after (no tautologies); no sleeps as
synchronization; no module-level imports of app modules that go stale
under sys.modules pollution (resolve at run time); TestClient instances
are function-scoped and not lifespan-bound unless the test needs it;
new functional CJK is allowlisted in tests/test_no_hardcoded_cjk.py with
a justification.
path; flag direct schema edits. Engine code must stay backward-compatible
with already-installed on-disk model state (no forced reinstall).
- path: ".github/workflows/**"
instructions: >-
Pin actions to a major version tag at minimum. Flag any workflow that
grants write permissions it does not need.
# Non-gating pre-merge audits of the project's hard rules (warning mode —
# the human owner is the gate, these make the checklist visible per-PR).
pre_merge_checks:
title:
mode: warning
requirements: >-
Conventional-commit style with scope, e.g. "fix(dub): …", and the issue
reference in title or body. Never propose a version bump.
issue_assessment:
mode: warning
custom_checks:
- name: "Cross-platform default parity"
mode: warning
instructions: >-
If the PR changes any default-mode (out-of-the-box) behavior, verify
it behaves identically on macOS, Windows and Linux, or is moved behind
an explicit opt-in (Settings toggle, env var, or CLI flag). A
platform-divergent default is a P0 per CLAUDE.md — fail this check
and say which platform diverges.
- name: "i18n completeness (21 locales)"
mode: warning
instructions: >-
For every new or changed t('...') key in frontend code, verify the
key exists in all 21 files under frontend/src/i18n/locales/. List any
locale files missing the key. Also flag hardcoded user-facing strings
that bypass i18n entirely.
- name: "Local-first guarantee"
mode: warning
instructions: >-
Verify the PR adds no required cloud calls, accounts, API keys or
telemetry. Outbound traffic is only allowed to GitHub Issues (opt-in
bug reporting) and HuggingFace model downloads. The app must remain
fully functional offline and with reporting disabled.
- name: "Backward compatibility"
mode: warning
instructions: >-
Verify existing omnivoice_data/ (voices, projects, settings) and
already-installed engine model state keep working without manual
migration. Any DB schema change must ship an alembic migration with
an upgrade path. Flag anything that would force users to reinstall an
engine or re-download model weights.
finishing_touches:
docstrings:
enabled: true
unit_tests:
enabled: true
# Feed the bot the project constitution and docs, and let it accumulate
# learnings from review conversations ("@coderabbitai always/never …").
knowledge_base:
code_guidelines:
enabled: true
filePatterns:
- "CLAUDE.md"
- "docs/**/*.md"
learnings:
scope: auto
issues:
scope: auto
pull_requests:
scope: auto
chat:
auto_reply: true
+3 -6
View File
@@ -1,9 +1,6 @@
# These are supported funding model platforms
# GitHub Sponsors isn't set up for this account — fund via Ko-fi or PayPal.
ko_fi: debpalash
custom:
- "https://paypal.me/palashCoder"
- "https://github.com/debpalash/OmniVoice-Studio/blob/main/SPONSORS.md"
# github: [debpalash] # not available
github: [debpalash]
# ko_fi: omnivoice
# open_collective: omnivoice-studio
# custom: ["https://omnivoice.palash.dev/sponsor"]
+45
View File
@@ -0,0 +1,45 @@
---
name: 🐛 Bug Report
about: Report a bug to help us improve OmniVoice Studio
title: "[Bug] "
labels: ["bug", "triage"]
assignees: []
---
## Describe the bug
A clear and concise description of what the bug is.
## To reproduce
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '...'
3. See error
## Expected behavior
What you expected to happen.
## Screenshots / Logs
If applicable, add screenshots or paste relevant logs from **Settings → Logs**.
> **Tip:** **Settings → About → "Save diagnostic bundle"** produces a zip
> (self-check report, recent errors, scrubbed log tails) you can drag onto
> this issue — it answers most environment questions below automatically.
> Headless installs: `python backend/main.py --diagnose` prints the same
> self-check (`--deep` also test-loads the active engine).
## Environment
- **OS:** [e.g. macOS 15.2, Windows 11, Ubuntu 24.04]
- **Install method:** [Desktop app / Docker / From source]
- **Version:** [e.g. v0.2.7 — check Settings → About]
- **GPU:** [e.g. NVIDIA RTX 4090 / Apple M3 Pro / CPU only]
- **RAM:** [e.g. 16 GB]
- **Active TTS engine:** [e.g. omnivoice — check Settings → Engines]
## Additional context
Add any other context about the problem here.
-109
View File
@@ -1,109 +0,0 @@
name: 🐛 Bug report
description: Something works incorrectly or crashes (not a first-run/install problem — use the install template for those).
title: "[Bug] "
labels: ["bug", "triage"]
body:
- type: markdown
attributes:
value: |
Thanks for helping improve OmniVoice Studio! 🎙️
**Fastest path to a fix:** **Settings → About → "Save diagnostic bundle"** makes a
zip (self-check + recent errors + scrubbed log tails) — drag it onto this issue and
most of the environment questions below are answered automatically.
Headless: `python backend/main.py --diagnose` (add `--deep` to test-load the engine).
- type: checkboxes
id: preflight
attributes:
label: Before filing
options:
- label: I searched [existing issues](https://github.com/debpalash/OmniVoice-Studio/issues?q=is%3Aissue) and this isn't a duplicate.
required: true
- label: I'm on the latest release (or `main`) — older builds may already be fixed.
required: false
- type: textarea
id: what-happened
attributes:
label: What happened?
description: A clear description of the bug, including the exact error text / toast if any.
placeholder: "Voice cloning failed with '…' after I clicked Generate."
validations:
required: true
- type: textarea
id: repro
attributes:
label: Steps to reproduce
value: |
1.
2.
3.
validations:
required: true
- type: textarea
id: expected
attributes:
label: What did you expect instead?
validations:
required: true
- type: dropdown
id: os
attributes:
label: Operating system
options:
- macOS (Apple Silicon)
- macOS (Intel)
- Windows (x64)
- Linux (AppImage)
- Linux (.deb)
- Linux (other / from source)
- Docker
validations:
required: true
- type: dropdown
id: install
attributes:
label: How did you install it?
options:
- Desktop app (installer / AppImage)
- Docker image
- From source (uv sync)
validations:
required: true
- type: input
id: version
attributes:
label: Version
description: Settings → About (e.g. v0.3.5), or the Docker tag / git SHA.
placeholder: "v0.3.5"
validations:
required: true
- type: dropdown
id: device
attributes:
label: Compute device
options:
- NVIDIA GPU (CUDA)
- AMD GPU (ROCm)
- Apple Silicon (MPS)
- Intel GPU (XPU)
- CPU only
- Not sure
validations:
required: true
- type: input
id: engine
attributes:
label: Active TTS/ASR engine
description: Settings → Engines (e.g. omnivoice, cosyvoice, indextts2, whisperx).
placeholder: "omnivoice"
- type: textarea
id: logs
attributes:
label: Logs / diagnostic bundle
description: Drag the diagnostic bundle here, or paste relevant lines from **Settings → Logs**. Secrets are scrubbed automatically.
render: text
- type: textarea
id: extra
attributes:
label: Anything else?
description: Screenshots, the input that triggered it, RAM/VRAM, etc.
-11
View File
@@ -1,11 +0,0 @@
blank_issues_enabled: false
contact_links:
- name: 💬 Discord — questions & quick help
url: https://discord.gg/bzQavDfVV9
about: Usage questions, setup help, and chat. Faster than an issue for "how do I…".
- name: 🗣️ GitHub Discussions
url: https://github.com/debpalash/OmniVoice-Studio/discussions
about: Ideas, show-and-tell, and open-ended Q&A that isn't a bug or a specific feature ask.
- name: 🔒 Security vulnerability
url: https://github.com/debpalash/OmniVoice-Studio/security/policy
about: Please report security issues privately — do NOT open a public issue.
+23
View File
@@ -0,0 +1,23 @@
---
name: ✨ Feature Request
about: Suggest an idea for OmniVoice Studio
title: "[Feature] "
labels: ["enhancement"]
assignees: []
---
## Is your feature request related to a problem?
A clear description of what the problem is. Ex. "I'm always frustrated when..."
## Describe the solution you'd like
A clear description of what you want to happen.
## Describe alternatives you've considered
Any alternative solutions or features you've considered.
## Additional context
Add any other context, mockups, or screenshots about the feature request here.
@@ -1,50 +0,0 @@
name: ✨ Feature request
description: Suggest an improvement or a new capability.
title: "[Feature] "
labels: ["enhancement"]
body:
- type: checkboxes
id: preflight
attributes:
label: Before filing
options:
- label: I searched [existing issues](https://github.com/debpalash/OmniVoice-Studio/issues?q=is%3Aissue) and [discussions](https://github.com/debpalash/OmniVoice-Studio/discussions) for this idea.
required: true
- type: textarea
id: problem
attributes:
label: What problem does this solve?
description: The use case / friction this addresses ("When I … I can't …").
validations:
required: true
- type: textarea
id: solution
attributes:
label: Proposed solution
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives considered
- type: dropdown
id: area
attributes:
label: Area
options:
- Voice cloning
- Voice design
- Video dubbing
- Real-time dictation
- Audiobook / Stories (long-form)
- TTS/ASR engines
- Install / setup / packaging
- Other
validations:
required: true
- type: markdown
attributes:
value: |
> OmniVoice is **local-first** — features must work fully offline with no accounts,
API keys, or cloud calls, and behave identically on macOS/Windows/Linux. Proposals
that fit those constraints are easiest to land.
@@ -1,86 +0,0 @@
name: 🧩 Install / first-run problem
description: The app won't install, set up, download models, or reach a first working output.
title: "[Install] "
labels: ["install", "triage"]
body:
- type: markdown
attributes:
value: |
A first-run that *just works* is the whole point — sorry it didn't. Let's fix it.
If the app launched far enough to open Settings, **Settings → About → "Save diagnostic
bundle"** captures most of this; otherwise the fields below are enough.
- type: dropdown
id: stage
attributes:
label: Where did it fail?
options:
- App won't launch / blank or broken window
- Python / uv environment bootstrap
- Model download (HuggingFace)
- Engine install (CosyVoice / IndexTTS / MLX / etc.)
- First synthesis / dub never completes
- Other
validations:
required: true
- type: textarea
id: error
attributes:
label: The error
description: The exact message, traceback, or what you see on screen.
render: text
validations:
required: true
- type: dropdown
id: os
attributes:
label: Operating system
options:
- macOS (Apple Silicon)
- macOS (Intel)
- Windows (x64)
- Linux (AppImage)
- Linux (.deb)
- Linux (other / from source)
- Docker
validations:
required: true
- type: dropdown
id: install
attributes:
label: How are you installing it?
options:
- Desktop app (installer / AppImage)
- Docker image
- From source (uv sync)
validations:
required: true
- type: input
id: version
attributes:
label: Version
placeholder: "v0.3.5 (or installer build / git SHA)"
validations:
required: true
- type: dropdown
id: network
attributes:
label: Network conditions (model/dependency downloads)
description: Restricted networks are a known source of bootstrap failures (mirror fallback).
options:
- Normal / unrestricted
- Behind a corporate proxy / firewall
- Region with restricted access (e.g. China, Russia)
- Offline / air-gapped
- Not sure
validations:
required: true
- type: textarea
id: logs
attributes:
label: Logs / diagnostic bundle
description: Drag a diagnostic bundle, or paste the install/bootstrap log. Headless self-check — `python backend/main.py --diagnose`.
render: text
- type: textarea
id: tried
attributes:
label: What have you already tried?
-78
View File
@@ -1,78 +0,0 @@
name: 🤝 Sponsorship inquiry
description: Support OmniVoice and (optionally) claim a logo slot. Not for bugs or feature requests.
title: "Sponsorship inquiry: "
labels: ["sponsor"]
body:
- type: markdown
attributes:
value: |
Thanks for considering sponsoring **OmniVoice Studio** 💛
OmniVoice is free, local-first, and AGPL-3.0 — sponsorship keeps development going.
See **[SPONSORS.md](https://github.com/debpalash/OmniVoice-Studio/blob/main/SPONSORS.md)** for tiers, placements, and logo guidelines.
Prefer to just donate? [Ko-fi](https://ko-fi.com/debpalash) (recurring) or [PayPal](https://paypal.me/palashCoder) (one-time) — you don't need this form for that.
- type: input
id: name
attributes:
label: Name or organization
description: How you'd like to be credited (person or company).
validations:
required: true
- type: input
id: website
attributes:
label: Website / link
description: The URL your name or logo should link to (homepage, product page, profile…).
placeholder: https://example.com
- type: input
id: logo
attributes:
label: Logo URL (optional)
description: Link to your logo (SVG preferred, else 2× PNG, transparent background). You can also attach it in the description below.
placeholder: https://example.com/logo.svg
- type: dropdown
id: tier
attributes:
label: Tier you're interested in
description: See SPONSORS.md for what each tier includes. Not sure? Pick "Not sure yet".
options:
- Backer
- Bronze
- Silver
- Gold
- Not sure yet — let's talk
- Custom / annual arrangement
validations:
required: true
- type: dropdown
id: method
attributes:
label: How you'd like to support
options:
- Ko-fi (recurring)
- Ko-fi (one-time)
- PayPal (one-time)
- Not sure yet — let's discuss
validations:
required: true
- type: input
id: contact
attributes:
label: How should we reach you?
description: Email or another contact. (GitHub will also notify you on this issue.)
validations:
required: true
- type: textarea
id: notes
attributes:
label: Anything else?
description: Questions, constraints, timeline, or context. Attach your logo here if you didn't link it above.
- type: checkboxes
id: ack
attributes:
label: Acknowledgements
options:
- label: I understand sponsorship is a thank-you, not a paywall — OmniVoice stays fully free and AGPL-3.0, and sponsors don't get gated features.
required: true
- label: If I provide a logo, I have the right to use it and grant OmniVoice permission to display it in the README, the app, and the project website.
required: false
+13 -8
View File
@@ -18,7 +18,7 @@
- [ ] 📝 Documentation
- [ ] 🧪 Tests
- [ ] 🔧 CI / Build
- [ ] 🚀 Release prep
- [ ] 🚀 Release prep (RC or final)
## Testing
@@ -33,12 +33,17 @@
- [ ] No local machine paths, logs, or personal env details in this PR
- [ ] Version files are in sync (if version bump): `pyproject.toml`, `package.json`, `tauri.conf.json`, `Cargo.toml`
- [ ] If this PR changes runtime behavior, the regression fixture at `tests/fixtures/omnivoice_data/` still loads green on the `smoke-matrix` CI job (macOS + Windows + Linux)
- [ ] If this is part of a release, I've read the "Release cadence" section below and confirmed this PR targets the right RC
## Release cadence
## Release cadence (read once per RC)
OmniVoice ships **continuous-to-main** — no release candidates, no soak windows.
Every merged PR is immediately part of the rolling preview (`main`, Docker
`:latest`, the desktop Preview channel). Versioned releases are tagged from
`main` when it's ready; `main` then bumps to the next patch automatically.
Users who want stability pin a release tag / Docker `:stable` / the desktop
Stable channel.
OmniVoice ships every minor on a **two-RC cadence**:
- `vX.Y.0-rc1` — cut from `main` once all GATE-* requirements pass; clean-VM exercise on 4 OSes (per `REL-01`)
- 48-hour soak (no new commits to release branch except fix-forward)
- `vX.Y.0` — promotion if rc1 is clean
If your PR touches install / bootstrap / CI, it MUST land before rc1 cut, not between rc1 and the promotion. During a soak, any merge needs explicit OK from the release captain.
## Screenshots
<!-- If applicable, add screenshots or recordings. -->
+2 -21
View File
@@ -92,10 +92,7 @@ jobs:
- name: Install frontend deps
working-directory: frontend
# --frozen-lockfile so a frontend/package.json change that forgets to
# regenerate the root bun.lock fails HERE (fast) instead of only in the
# Docker build (deploy/Dockerfile), which is what reddened main on #485.
run: bun install --frozen-lockfile
run: bun install
# checkJs is true in tsconfig for IDE feedback, but 947 pre-existing
# JS errors remain. Override to false in CI so only .ts files block.
@@ -105,19 +102,6 @@ jobs:
working-directory: frontend
run: bun run typecheck:ci
# oxlint gate — fast Rust linter, blocks on errors so lint debt can't
# re-accumulate (warnings, incl. the react-compiler advisories in
# `lint:hooks`, are non-blocking). See frontend/.oxlintrc.json.
- name: Frontend lint (oxlint)
working-directory: frontend
run: bun run lint
# oxfmt format gate — JS/TS/JSX only (CSS/JSON/Tauri excluded; see
# frontend/.oxfmtrc.json). `bun run format` fixes locally.
- name: Frontend format check (oxfmt)
working-directory: frontend
run: bun run format:check
- name: Run Vitest (frontend)
working-directory: frontend
run: bunx vitest run
@@ -190,10 +174,7 @@ jobs:
- name: Install frontend deps
working-directory: frontend
# --frozen-lockfile so a frontend/package.json change that forgets to
# regenerate the root bun.lock fails HERE (fast) instead of only in the
# Docker build (deploy/Dockerfile), which is what reddened main on #485.
run: bun install --frozen-lockfile
run: bun install
# tauri-build's setup hook reads tauri.conf.json's `frontendDist`
# ("../dist"), which only exists after a frontend build. Without this,
+17 -62
View File
@@ -5,21 +5,14 @@
# - push to main branch → :main, :sha- (rolling "edge" build)
# - workflow_dispatch → :sha- only (ad-hoc test build)
#
# Tag ↔ image mapping (versioning hard rule, owner-set 2026-06-11:
# :latest IS the preview channel; stable users pin :stable or a version tag)
# :latest — rolling preview: latest commit on main (always last release + 1 dev)
# :main — alias of the same rolling main build (kept for back-compat)
# :stable — most recent versioned release (set on every v* tag push)
# :0.3.6 — exact version from the git tag
# Tag ↔ image mapping
# :latest — always the most recent versioned release (set on every v* tag push)
# :0.3.0 — exact version from the git tag
# :0.3 — major.minor floating tag (updated on every patch within the minor)
# :main — latest commit on main; may be ahead of the last tagged release
# :sha-xxxx — specific commit SHA; produced by workflow_dispatch
#
# Images land at: ghcr.io/debpalash/omnivoice-studio AND docker.io/palashdeb/omnivoice-studio
# (Docker Hub push gated on the DOCKERHUB_USERNAME/DOCKERHUB_TOKEN secrets;
# if unset the build still pushes to GHCR.)
#
# On main pushes the Docker Hub repository overview is also synced from
# deploy/dockerhub-overview.md (source of truth for the hub.docker.com page).
# Images land at: ghcr.io/debpalash/omnivoice-studio
#
# NOTE: the Docker image is the headless web-server build of OmniVoice (FastAPI
# backend + pre-built React frontend served over HTTP). The Tauri desktop
@@ -41,7 +34,6 @@ permissions:
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
DOCKERHUB_IMAGE: palashdeb/omnivoice-studio
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
jobs:
@@ -64,43 +56,28 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
# Docker Hub login — only when the secret is present, so forks / runs
# without the credential still publish to GHCR.
- name: Check Docker Hub credentials
id: dockerhub
run: echo "enabled=${{ secrets.DOCKERHUB_TOKEN != '' }}" >> "$GITHUB_OUTPUT"
- name: Log in to Docker Hub
if: steps.dockerhub.outputs.enabled == 'true'
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
# Tag strategy (`:sha-<short>` is emitted on every trigger):
# v0.3.6 tag push → :0.3.6, :0.3, :stable, :sha-
# main branch push → :latest, :main, :sha-
# v0.3.0 tag push → :0.3.0, :0.3, :latest, :sha-
# main branch push → :main, :sha-
# workflow_dispatch → :sha- only
#
# All mutable-tag rules stay gated on `github.event_name == 'push'` so a
# manual workflow_dispatch can only ever produce a throwaway `:sha-` tag
# (the stale-:latest fix from #249/#251). :stable excludes prerelease
# tags (those contain a `-`) so a prerelease can't clobber it.
# Fix for stale :latest (issues #249, #251):
# The previous rule used `enable={{is_default_branch}}`, which evaluates
# to false on tag pushes (detached HEAD) — so :latest was never updated
# when a release tag was pushed. The version / :latest / :main rules are
# gated on `github.event_name == 'push'` so a manual workflow_dispatch can
# only ever produce a throwaway `:sha-` tag (never republish a mutable
# tag), and :latest additionally excludes prerelease tags (those contain a
# `-`, e.g. v1.0.0-rc.1) so a prerelease can't clobber :latest.
- name: Extract metadata (tags, labels)
id: meta
uses: docker/metadata-action@v5
with:
# Same tag set applied to both registries. The Docker Hub line is
# blank when the secret is unset, so metadata-action emits GHCR-only
# tags in that case.
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
${{ steps.dockerhub.outputs.enabled == 'true' && env.DOCKERHUB_IMAGE || '' }}
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=semver,pattern={{version}},enable=${{ github.event_name == 'push' }}
type=semver,pattern={{major}}.{{minor}},enable=${{ github.event_name == 'push' }}
type=raw,value=stable,enable=${{ github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-') }}
type=raw,value=latest,enable=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
type=raw,value=latest,enable=${{ github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-') }}
type=raw,value=main,enable=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
type=sha,prefix=sha-,format=short
@@ -114,25 +91,3 @@ jobs:
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
# Sync the Docker Hub repository overview from deploy/dockerhub-overview.md.
# Only on main pushes (the overview tracks the rolling preview) and only
# when Docker Hub creds are present, mirroring the push gating above.
#
# continue-on-error: the overview text is cosmetic, and the description
# PATCH 403s unless DOCKERHUB_TOKEN carries description-edit scope (many
# fine-grained Docker Hub tokens that can push still can't edit the
# description). The image build+push is what matters — a creds-scope
# mismatch on this cosmetic step must not fail the whole Docker run. To
# actually sync the overview, use a token with read/write (incl.
# description) scope, or the account password.
- name: Update Docker Hub description
if: steps.dockerhub.outputs.enabled == 'true' && github.event_name == 'push' && github.ref == 'refs/heads/main'
continue-on-error: true
uses: peter-evans/dockerhub-description@v4
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
repository: ${{ env.DOCKERHUB_IMAGE }}
short-description: "Local ElevenLabs alternative: voice cloning, design & video dubbing in 646 languages. No API keys."
readme-filepath: ./deploy/dockerhub-overview.md
-97
View File
@@ -1,97 +0,0 @@
# Docs drift — daily inventory-vs-docs check with a single rolling issue.
#
# docs/features.yaml is the canonical inventory; scripts/check-docs-drift.py
# diffs it against README.md, docs/, and the engine registries. On drift the
# job updates (or creates) ONE issue labeled `docs-drift` in place — no issue
# spam — and closes it automatically when the check is clean again.
#
# Companion to the PR-gating validate-install-docs.py step in ci.yml.
# Spec: docs/competitive-analysis.md Spec 9a / parity program Wave 0.1.
# Rolling-issue pattern adapted from Patter (MIT).
name: docs-drift
on:
schedule:
# Daily 03:30 UTC — after most merges, before EU morning triage.
- cron: "30 3 * * *"
workflow_dispatch:
permissions:
contents: read
issues: write
jobs:
drift:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install checker deps
run: pip install "pyyaml>=6"
- name: Check inventory vs README/docs/registries
id: drift
continue-on-error: true
run: python scripts/check-docs-drift.py --output drift-report.md
- name: Update rolling docs-drift issue
uses: actions/github-script@v7
env:
DRIFT_OUTCOME: ${{ steps.drift.outcome }}
with:
script: |
const fs = require('fs');
const drifted = process.env.DRIFT_OUTCOME === 'failure';
const { owner, repo } = context.repo;
const label = 'docs-drift';
const open = await github.rest.issues.listForRepo({
owner, repo, state: 'open', labels: label, per_page: 5,
});
if (drifted) {
let body = '';
try {
body = fs.readFileSync('drift-report.md', 'utf8');
} catch {
body = '# Docs drift report\n\nThe checker failed before writing a report — see the workflow run logs.';
}
body += `\n\n---\n_Last checked by [run ${context.runId}](https://github.com/${owner}/${repo}/actions/runs/${context.runId})._\n`;
if (open.data.length > 0) {
await github.rest.issues.update({
owner, repo, issue_number: open.data[0].number, body,
});
core.info(`Updated rolling issue #${open.data[0].number}`);
} else {
const created = await github.rest.issues.create({
owner, repo,
title: 'docs-drift: feature inventory vs docs mismatch',
body,
labels: [label, 'documentation'],
});
core.info(`Created rolling issue #${created.data.number}`);
}
} else {
for (const issue of open.data) {
await github.rest.issues.createComment({
owner, repo, issue_number: issue.number,
body: 'Drift resolved — nightly check is clean again. Closing automatically.',
});
await github.rest.issues.update({
owner, repo, issue_number: issue.number, state: 'closed',
});
core.info(`Closed rolling issue #${issue.number}`);
}
}
- name: Surface drift as a failed run
if: steps.drift.outcome == 'failure'
run: |
echo "Docs drift detected — see the rolling docs-drift issue."
exit 1
-55
View File
@@ -1,55 +0,0 @@
# LLM-judge evals — semantic quality suites, NEVER a gate.
#
# Hard rule (parity program Wave 0.3 / competitive-analysis Spec 9b): LLM
# judges never gate CI. This workflow is scheduled + manual only, the eval
# step is continue-on-error, and the JSON report is the deliverable
# (uploaded as an artifact). Deterministic probe judges in ci.yml remain
# the only gates.
#
# On the hosted runner there is no local LLM endpoint, so the run usually
# reports "skipped — no LLM backend configured"; the workflow exists so the
# suites run anywhere a TRANSLATE_BASE_URL secret/endpoint is provided
# (e.g. a self-hosted runner with Ollama).
name: evals
on:
schedule:
# Weekly, Sundays 04:00 UTC.
- cron: "0 4 * * 0"
workflow_dispatch:
permissions:
contents: read
jobs:
evals:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- uses: astral-sh/setup-uv@v3
with:
enable-cache: true
cache-dependency-glob: "uv.lock"
- name: Install deps
run: uv sync
- name: Run eval suites (non-gating)
continue-on-error: true
env:
TRANSLATE_BASE_URL: ${{ secrets.EVALS_LLM_BASE_URL }}
TRANSLATE_API_KEY: ${{ secrets.EVALS_LLM_API_KEY }}
run: uv run python tests/evals/run_evals.py --output eval-report.json
- name: Upload report artifact
uses: actions/upload-artifact@v4
with:
name: eval-report
path: eval-report.json
if-no-files-found: warn
+25 -218
View File
@@ -4,16 +4,12 @@
# - push of a tag matching `v*` (e.g. `v0.2.0`) → full STABLE release,
# publishes artifacts + signed updater manifest (`latest.json`) to the
# tag's GH Release. This is the default Stable updater channel.
# - schedule (nightly, 07:00 UTC) → rolling `preview` PRERELEASE built from
# `main` with its own signed `latest.json` at releases/download/preview/.
# Feeds the opt-in Preview updater channel (Settings → About → Update
# channel). The `preview-gate` job skips the matrix on nights when `main`
# didn't move, so an idle day costs only a ~30s gate job — keeping Preview
# ≤24h behind `main` at a predictable ~1-matrix/day cost. The stable
# `latest` release is untouched.
# - workflow_dispatch (publish_preview=true) → the same preview build on
# demand from the selected branch (e.g. to preview a feature branch, or to
# refresh immediately without waiting for the nightly).
# - workflow_dispatch (publish_preview=true) → builds the selected branch and
# publishes a rolling `preview` PRERELEASE with its own signed
# `latest.json` at releases/download/preview/. This feeds the opt-in
# Preview updater channel (Settings → About → Update channel). The stable
# `latest` release is untouched. Run this manually whenever you want to cut
# a preview from `main`.
# - workflow_dispatch (publish_preview=false) → on-demand build (prior
# behavior; draft release named after the branch).
#
@@ -32,10 +28,6 @@ name: Desktop Release
on:
push:
tags: ['v*']
schedule:
# 07:00 UTC daily — rolling `preview` prerelease from `main`. The
# preview-gate job no-ops the matrix when main hasn't moved in a day.
- cron: '0 7 * * *'
workflow_dispatch:
inputs:
draft:
@@ -129,41 +121,8 @@ jobs:
working-directory: frontend
run: node --experimental-strip-types --no-warnings --test ../tests/frontend/*.test.mjs
# Decide preview-vs-stable, and for nightly runs whether `main` actually
# moved in the last day. Outputs gate the expensive matrix (`build`) and the
# `preview-notes` job, so a no-commit night costs only this ~30s job.
preview-gate:
name: Preview gate
runs-on: ubuntu-22.04
outputs:
is_preview: ${{ steps.decide.outputs.is_preview }}
proceed: ${{ steps.decide.outputs.proceed }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 50
- id: decide
shell: bash
run: |
set -euo pipefail
event="${{ github.event_name }}"
if [ "$event" = "schedule" ] || { [ "$event" = "workflow_dispatch" ] && [ "${{ inputs.publish_preview }}" = "true" ]; }; then
echo "is_preview=true" >> "$GITHUB_OUTPUT"
else
echo "is_preview=false" >> "$GITHUB_OUTPUT"
fi
# Nightly: skip the matrix when main hasn't moved in the last day.
if [ "$event" = "schedule" ] && [ -z "$(git log --since='25 hours ago' --oneline)" ]; then
echo "No new commits on main in the last day — skipping nightly preview."
echo "proceed=false" >> "$GITHUB_OUTPUT"
else
echo "proceed=true" >> "$GITHUB_OUTPUT"
fi
build:
needs: [test, preview-gate]
# Nightly runs with no new commits on main skip the 4-platform matrix.
if: needs.preview-gate.outputs.proceed == 'true'
needs: test
strategy:
fail-fast: false
matrix:
@@ -174,36 +133,9 @@ jobs:
rust_target: aarch64-apple-darwin
bundles: "app,dmg,updater"
# macOS Intel (#279): reinstated. The earlier "Rosetta 2 runs the
# ARM build" rationale for dropping it was backwards — Rosetta only
# translates x86_64→arm64, so Intel Macs (supported through macOS
# Sequoia) simply cannot run the aarch64 bundle and had NO
# installable artifact. Runner: `macos-15-intel`, GitHub's
# designated migration target after macos-13 retired (Dec 2025);
# it's a standard (public-repo-free) image supported through
# August 2027 — the last x86_64 image Actions will offer. Building
# natively (not cross-compiling from the arm64 leg) keeps the
# per-TRIPLE uv/ffmpeg sidecar fetches, the DMG installer smoke,
# and the ad-hoc signing verification (scripts/
# verify-macos-signing.sh, PR #290) all exercising the real
# x86_64 artifact on real Intel hardware. The macos-13 queue
# backlog that motivated the original drop is contained by
# fail-fast:false — a slow Intel leg can delay the release run but
# can't fail the other targets.
#
# #889 (2026-07): Intel macOS is now UNSUPPORTED for the local
# backend — torch ≥2.3 ships no macOS x86_64 wheels, so the venv
# bootstrap can never succeed on Intel. The shipped x64 artifact is
# effectively UI-only (usable with a remote backend); the app now
# pre-fails first-run bootstrap with an honest message on Intel.
# Whether to keep shipping this x64 leg (UI-only) or drop it is an
# OWNER CALL — deliberately not changed in the #889 PR.
- os: macos-15-intel
arch: x86_64-apple-darwin
label: "macOS Intel"
rust_target: x86_64-apple-darwin
bundles: "app,dmg,updater"
# macOS Intel dropped: Apple shipped the last Intel Mac in 2023 and
# Rosetta 2 runs the ARM build natively. macos-13 runner backlog
# was also blocking every release tag for ~10 min.
# Windows: force MSI bundling via --bundles. NSIS fails at makensis
# because our PyInstaller payload approaches its ~2 GB stub limit.
- os: windows-2022
@@ -359,10 +291,7 @@ jobs:
case "$TRIPLE" in
aarch64-apple-darwin|x86_64-apple-darwin)
# evermeet.cx ships each binary as a separate .zip containing
# a single x86_64 Mach-O executable — natively correct on the
# Intel leg, and runs via Rosetta 2 on the arm64 leg. Both
# darwin TRIPLEs therefore bundle the same payload; only the
# sidecar filename suffix differs.
# a single x86_64 Mach-O executable (runs via Rosetta on arm64).
for TOOL in ffmpeg ffprobe; do
if [ "$TOOL" = "ffmpeg" ]; then
URL="https://evermeet.cx/ffmpeg/getrelease/zip"
@@ -469,21 +398,17 @@ jobs:
# always reported the static 0.3.0 never looked "newer", so no update was
# ever delivered). Ephemeral, CI-only — never committed. Tauri reads the
# bundle + updater version from tauri.conf.json, so rewriting it here
# stamps the artifacts + latest.json. Under the versioning hard rule
# (owner-set 2026-06-11) main is always last-release + 1, so BASE-N is a
# prerelease of the NEXT version and semver-sorts ABOVE the last stable
# (0.3.6-N > 0.3.5) — preview users naturally upgrade past stable, and
# the Windows MSI ProductVersion (which strips the prerelease → 0.3.6)
# is also correctly above the last stable.
# stamps the artifacts + latest.json. `0.3.0-preview.N` is a prerelease of
# the current target, so previews converge to stable when 0.3.0 ships
# (0.3.0 > 0.3.0-preview.N). NOTE: the Windows MSI ProductVersion strips
# the prerelease (→ 0.3.0), a wrinkle to verify for win preview→preview
# upgrades; mac/linux replace the bundle wholesale and are unaffected.
- name: Stamp preview version
if: needs.preview-gate.outputs.is_preview == 'true'
if: github.event_name == 'workflow_dispatch' && inputs.publish_preview
shell: bash
run: |
set -euo pipefail
# package.json is the single source of truth; tauri.conf.json reads its
# version from it ("version": "../package.json"), so stamping
# package.json restamps the whole bundle.
CONF=frontend/package.json
CONF=frontend/src-tauri/tauri.conf.json
BASE=$(jq -r .version "$CONF")
# MSI/WiX requires the semver pre-release identifier to be numeric-only
# (and <= 65535). "preview.N" hard-fails the Windows bundler, so the
@@ -506,12 +431,6 @@ jobs:
# only on the opt-in stable path, leaving them ABSENT (not "") on
# preview/unsigned paths so Tauri's bundler skips cert import. A static
# env: here would always set them to "" and break the mac build.
# Unsigned paths still get a VALID ad-hoc seal from tauri.conf.json
# (bundle.macOS.signingIdentity = "-"), so a downloaded build shows the
# GUI-bypassable "unidentified developer" prompt (right-click → Open /
# Settings → "Open Anyway") instead of the un-bypassable "damaged"
# error. On the signed path APPLE_SIGNING_IDENTITY (env) overrides the
# "-" default; once notarized, Gatekeeper accepts it with no prompt.
# GH runners disable FUSE, so linuxdeploy's AppImage can't mount
# itself at bundle time. This env tells linuxdeploy to extract-and-run
# instead, which works without FUSE.
@@ -523,13 +442,11 @@ jobs:
# rolling `preview` prerelease for the updater's Preview channel.
# Every other invocation — crucially the `v*` tag-push stable release
# — evaluates these expressions to exactly their prior values.
tagName: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'preview' || github.ref_name }}
# Version-first so the tag is readable in GitHub's truncated
# release-list sidebar (which clips the title mid-string).
releaseName: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'Preview — OmniVoice Studio' || format('{0} — OmniVoice Studio', github.ref_name) }}
tagName: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'preview' || github.ref_name }}
releaseName: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'OmniVoice Studio (Preview)' || format('OmniVoice Studio {0}', github.ref_name) }}
releaseBody: ${{ steps.changelog.outputs.body }}
releaseDraft: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'false' || (inputs.draft || 'true') }}
prerelease: ${{ needs.preview-gate.outputs.is_preview == 'true' }}
releaseDraft: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'false' || (inputs.draft || 'true') }}
prerelease: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) || false }}
updaterJsonPreferNsis: false
includeUpdaterJson: true
@@ -564,35 +481,6 @@ jobs:
echo "OK — bundle has shell + uv + backend resources"
hdiutil detach "$MOUNT" || true
# ── Signing / Gatekeeper / notarization verification ──────────────
# Runs codesign --verify, spctl (Gatekeeper), nested-binary, and
# stapler checks against the built .app (see docs/macos-signing-verification.md).
# STRICT (--require-signed) only on the opt-in signed stable path — same
# condition as "Configure Apple signing" above — so a failed or missing
# signature/notarization FAILS the job and STOPS the release instead of
# publishing an unsigned artifact. On every other (unsigned dev/preview)
# path it runs report-only and never breaks the build.
- name: Verify macOS signing
if: runner.os == 'macOS'
shell: bash
env:
STRICT: ${{ (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && vars.MACOS_SIGNING_ENABLED == 'true') && '1' || '0' }}
APPLE_ID: ${{ secrets.APPLE_ID }}
APPLE_PASSWORD: ${{ secrets.APPLE_PASSWORD }}
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
run: |
set -uo pipefail
APP=$(find "frontend/src-tauri/target/${{ matrix.rust_target }}/release/bundle/macos" -maxdepth 1 -name '*.app' | head -1)
[ -n "$APP" ] || { echo "FAIL — no .app found to verify"; exit 1; }
MODE=""
if [ "$STRICT" = "1" ]; then
MODE="--require-signed"
echo "Signed stable release → STRICT verification (release stops on failure)."
else
echo "Unsigned dev/preview path → report-only verification."
fi
bash scripts/verify-macos-signing.sh "$APP" $MODE
- name: Installer smoke (Windows)
if: runner.os == 'Windows'
timeout-minutes: 5
@@ -715,8 +603,8 @@ jobs:
# preview-only — stable `v*` releases keep their CHANGELOG section + the
# appended checksums.
preview-notes:
needs: [build, preview-gate]
if: needs.preview-gate.outputs.is_preview == 'true'
needs: build
if: github.event_name == 'workflow_dispatch' && inputs.publish_preview
runs-on: ubuntu-22.04
permissions:
contents: write
@@ -747,86 +635,5 @@ jobs:
echo ""
echo "$CONTRIB"
} > /tmp/preview-notes.md
# --prerelease re-asserts the flag every run: a non-prerelease
# `preview` release is eligible to become GitHub's "Latest", which is
# the exact URL the Stable updater channel reads — so it must never
# flip off.
gh release edit preview --repo "$REPO" --prerelease --notes-file /tmp/preview-notes.md
gh release edit preview --repo "$REPO" --notes-file /tmp/preview-notes.md
echo "Applied auto-generated release notes + contributors to the preview release."
- name: Verify preview updater manifest (prerelease + platform parity)
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REPO: ${{ github.repository }}
run: |
set -euo pipefail
# The preview release must stay a prerelease (or it can hijack the
# Stable channel's releases/latest endpoint), and its updater manifest
# must cover every platform stable does (else those users — e.g. Intel
# Mac — silently get no preview updates).
is_pre=$(gh release view preview --repo "$REPO" --json isPrerelease -q .isPrerelease)
test "$is_pre" = "true" || { echo "::error::preview release is not a prerelease"; exit 1; }
curl -fsSL "https://github.com/$REPO/releases/download/preview/latest.json" -o /tmp/preview-latest.json
curl -fsSL "https://github.com/$REPO/releases/latest/download/latest.json" -o /tmp/stable-latest.json
python3 - <<'PY'
import json, re
prev = json.load(open("/tmp/preview-latest.json"))
stab = json.load(open("/tmp/stable-latest.json"))
v = prev.get("version", "")
assert re.fullmatch(r"\d+\.\d+\.\d+-\d+", v), f"preview version not X.Y.Z-N: {v!r}"
pk, sk = set(prev.get("platforms", {})), set(stab.get("platforms", {}))
missing = sk - pk
assert not missing, f"preview manifest missing platforms vs stable: {sorted(missing)}"
print(f"preview manifest OK: {v} platforms={sorted(pk)}")
PY
# ── Post-release version bump (OWNER-GATED as of 2026-07-01) ──────────────
# Previously auto-ran after every stable v* tag to keep main = release + 1.
# The owner now controls bumps manually ("keep 0.3.8; I say when to bump"), so
# this job is OPT-IN: it runs ONLY when the repo variable AUTO_VERSION_BUMP is
# set to 'true' (Settings → Secrets and variables → Actions → Variables).
# Unset/anything-else → main stays at whatever it is after release. Re-enable
# by setting the variable; disable again by unsetting it.
version-bump:
if: >-
github.event_name == 'push' && github.ref_type == 'tag'
&& !contains(github.ref, '-')
&& vars.AUTO_VERSION_BUMP == 'true'
runs-on: ubuntu-22.04
permissions:
contents: write
steps:
- uses: actions/checkout@v4
with:
ref: main
fetch-depth: 0
- name: Bump main to released version + 1 patch
shell: bash
run: |
set -euo pipefail
RELEASED="${GITHUB_REF_NAME#v}"
IFS=. read -r MAJ MIN PAT <<< "$RELEASED"
NEXT="$MAJ.$MIN.$((PAT + 1))"
# frontend/package.json is the SINGLE SOURCE OF TRUTH: vite injects
# __APP_VERSION__ from it, and tauri.conf.json reads its bundle version
# from it ("version": "../package.json"). Read CURRENT from it.
CURRENT=$(jq -r .version frontend/package.json)
if [ "$(printf '%s\n' "$NEXT" "$CURRENT" | sort -V | tail -1)" = "$CURRENT" ] && [ "$NEXT" != "$CURRENT" ]; then
echo "main is already at $CURRENT (>= $NEXT) — nothing to bump"; exit 0
fi
# Bump the canonical (package.json), set absolutely so any prior drift
# self-heals. tauri.conf.json needs no edit — it derives from this.
tmp=$(mktemp)
jq --arg v "$NEXT" '.version = $v' frontend/package.json > "$tmp"
mv "$tmp" frontend/package.json
# The remaining files are CI-guarded mirrors (cargo/uv require a
# literal; the version.py literal is the frozen-backend last resort) —
# bump them in lockstep with the canonical.
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" frontend/src-tauri/Cargo.toml
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" pyproject.toml
sed -i "0,/_FALLBACK_VERSION = \"$CURRENT\"/s//_FALLBACK_VERSION = \"$NEXT\"/" backend/core/version.py
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add frontend/package.json frontend/src-tauri/Cargo.toml pyproject.toml backend/core/version.py
git commit -m "chore(version): main -> $NEXT after $GITHUB_REF_NAME release"
git push origin main
+2 -6
View File
@@ -32,13 +32,9 @@ env:
permissions:
contents: read
# PR branches: a new push cancels the superseded scan (no wasted runners).
# main: every commit keeps its own group, so nothing is cancelled — a merge
# train used to leave a permanent red ✗ ("cancelled") on every intermediate
# commit in the history view even though nothing failed.
concurrency:
group: security-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || 'branch' }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
group: security-${{ github.ref }}
cancel-in-progress: true
jobs:
# ── Secret scanning (gating) ─────────────────────────────────────────────
+2 -9
View File
@@ -1,17 +1,10 @@
# SPIKE-02: Adopt `ModelsLab/omnivoice-singing` as singing variant of the existing engine
**Status:** ⚠️ **SUPERSEDED (2026-06-14)** by [`specs/006-dubbing-singing-mode/`](../../specs/006-dubbing-singing-mode/spec.md)
**Date:** 2026-05-18 (superseded 2026-06-14)
**Status:** Proposed (research-supported) — awaiting Phase 2 SubprocessBackend merge
**Date:** 2026-05-18
**Decision-makers:** [maintainer]
**Related:** ROADMAP Phase 4; REQUIREMENTS SING-01..05; `.planning/phases/04-adaptive-specialty-engines-spike-first/04-RESEARCH.md`
> **Superseded:** This chose `ModelsLab/omnivoice-singing` for singing, but that
> model has **no melody (F0/MIDI) conditioning** — it sings its own melody and
> cannot follow the *source song* a dub must preserve. SoulX-Singer (arXiv
> 2602.07803, published after this decision) provides F0/MIDI conditioning and is
> selected in plan-06. This ADR stays valid only if reframed as an
> expressive-TTS styling toggle, not melody-matched dubbing.
## Context
`ModelsLab/omnivoice-singing` (HuggingFace, 1,053 downloads/month, verified 2026-05-18) is a finetune of `k2-fsa/OmniVoice` — same Apache-2.0 license, same Qwen3-0.6B backbone, same Higgs Audio v2 codec at 24 kHz mono, same `omnivoice` PyPI library (0.1.5, 2026-04-28) already shipping in OmniVoice Studio v0.2.7. Trained on additional singing + emotion-tagged data and activated by a `[singing]` text control tag at generation time.
@@ -1,396 +0,0 @@
---
phase: 260613-fdl
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- pyproject.toml
- backend/api/routers/setup/download.py
- backend/api/routers/setup/models.py
- backend/utils/hf_progress.py
- backend/utils/download_aggregator.py # NEW
- backend/services/segmented_download.py # NEW
- backend/api/routers/system.py
- frontend/src/pages/Settings.jsx
- frontend/src/api/setup.ts
- docs/downloading-models.md # NEW (docs-sync rule)
- tests/backend/setup/test_download_preflight.py # NEW
- tests/backend/services/test_segmented_download.py # NEW
autonomous: true
requirements:
# ── Wave 0 — Spike / gate ───────────────────────────────────────────────────
- FDL-00 # Classify all catalog repos Xet-backed vs legacy-LFS; the result sizes Wave 3
# ── Wave 1 — Maximize + guarantee the Xet fast path (default, no new deps) ───
- FDL-01 # Explicitly pin huggingface_hub>=1.7 + hf-xet in pyproject (today transitive/unpinned)
- FDL-02 # Drive snapshot_download with explicit max_workers + tqdm_class + endpoint (not implicit monkeypatch)
- FDL-03 # /system/info reports fast_download {xet_enabled, xet_version, high_performance}; logged at startup
- FDL-04 # Opt-in HF_XET_HIGH_PERFORMANCE + HDD sequential-write toggles via prefs (env wins)
# ── Wave 2 — Accurate downloaded/remaining + speed (the user-visible win) ────
- FDL-05 # dry_run preflight -> emit install_plan {total_bytes, cached_bytes, to_download_bytes, n_files, n_cached}
- FDL-06 # Backend aggregate tracker -> single 'aggregate' event {bytes_done, total_bytes, rate, eta, files_done/total}
- FDL-07 # Frontend overall bar: speed + downloaded/remaining + ETA from aggregate; per-file detail collapsible; cached-skip shown
# ── Wave 3 — Opt-in IDM-style accelerator for legacy-LFS repos ───────────────
- FDL-08 # Custom httpx segmented downloader: parallel Range GETs, resume, auth-safe redirect, etag/sha verify, cancel (default OFF)
- FDL-09 # Dispatch: accelerator ON + repo is LFS (not Xet) -> segmented path; else xet. Same aggregate progress + weight validation
# ── Wave 4 — Opt-in mirror path + docs ───────────────────────────────────────
- FDL-10 # Opt-in HF_ENDPOINT mirror setting (prefs); documented as classic-LFS fallback (no Xet); pairs with FDL-08
- FDL-11 # Cancel-in-flight endpoint + cooldown interplay (composes with MM2-06 bounded cooldowns)
- FDL-12 # docs/downloading-models.md (speed, fast-download status, HDD/high-perf toggles, mirror/restricted-network) + README pointer
must_haves:
truths:
- "Xet is the default download backend and is provably engaged: /system/info reports fast_download.xet_enabled=true with the hf_xet version, and a Xet-backed repo downloads via parallel chunk range-gets (not single-stream LFS)."
- "Before any bytes flow, the UI shows an accurate denominator: total bytes to download, bytes already cached (skipped), and file count — sourced from snapshot_download(dry_run=True), not guessed from the first tqdm bar."
- "During a download the UI shows ONE overall progress bar with instantaneous speed (sampled over a window, not a single file's rate), bytes downloaded / bytes remaining, and ETA — accurate even while Xet fetches many chunks/files in parallel."
- "hf_transfer is NOT used or enabled anywhere (deprecated, breaks progress); the fast path is Xet only."
- "The custom segmented downloader is OPT-IN (default off), only engages for non-Xet/legacy-LFS repos, never forwards the HF Authorization header to the redirected CDN host, verifies the downloaded file against its expected size/etag before marking complete, resumes a partial .part file, and can be cancelled mid-flight."
- "Default download behavior is identical on macOS, Windows, Linux (Xet path, pure-Python). Every accelerator/mirror/high-perf knob is behind an explicit opt-in (Settings toggle or env var) per the cross-platform-parity strict rule — no bundled per-OS binary, no platform-divergent default."
- "No new on-disk model-state format; existing HF cache layout and already-installed models are untouched; the segmented downloader writes into the same HF cache blob/snapshot structure (or hands off to it) so a model it fetches is indistinguishable from one snapshot_download fetched."
- "uv run pytest tests/backend/setup/test_download_preflight.py tests/backend/services/test_segmented_download.py passes; existing download/install tests stay green."
- "pyproject pins huggingface_hub>=1.7 and hf-xet explicitly; uv.lock resolves with single versions (uv tree shows no duplicate huggingface_hub)."
artifacts:
- path: "backend/utils/download_aggregator.py"
provides: "Per-repo byte aggregator: sums bytes across parallel files/chunks, samples rate over a window, emits one 'aggregate' event"
contains: "class DownloadAggregator AND def snapshot"
- path: "backend/services/segmented_download.py"
provides: "Opt-in multi-connection Range downloader for legacy-LFS repos (auth-safe, resume, verify, cancel)"
contains: "async def segmented_download AND Range"
- path: "backend/api/routers/setup/download.py"
provides: "Driven snapshot_download (max_workers+tqdm_class+endpoint), dry_run preflight, dispatch to segmented path, cancel endpoint"
contains: "dry_run AND tqdm_class"
- path: "docs/downloading-models.md"
provides: "User docs for download speed, fast-download status, HDD/high-perf toggles, mirror/restricted-network"
contains: "Xet"
key_links:
- from: "install_model (download.py:122)"
to: "snapshot_download(dry_run=True) preflight"
via: "compute total/cached/remaining before the real download; emit 'install_plan'"
pattern: "dry_run\\s*=\\s*True"
- from: "snapshot_download / segmented_download byte updates"
to: "DownloadAggregator -> single 'aggregate' SSE event"
via: "tqdm_class forwards bytes into the aggregator; segmented path calls aggregator.add() directly"
pattern: "aggregate"
- from: "dispatch in install_model"
to: "segmented_download vs snapshot_download"
via: "prefs accelerator toggle AND repo-is-LFS classification (FDL-00 helper)"
pattern: "segmented_download"
- from: "system_info (system.py:245)"
to: "fast_download status block"
via: "probe hf_xet import + version + HF_XET_HIGH_PERFORMANCE"
pattern: "fast_download"
---
<objective>
Make model downloads as fast as possible AND show accurate speed / downloaded / remaining / ETA.
**Framing (validated by research — see 260613-fdl-RESEARCH below):** HuggingFace's **hf-xet** backend ALREADY implements the "IDM/uGet technique" — content-defined chunking, parallel byte-range fetches with adaptive concurrency, dedup, and automatic resume — and does it auth-safely. It ships by default in modern `huggingface_hub` and `hf_xet` is already installed here (huggingface_hub 1.7.2). HF closed the multi-connection-downloader feature request as "solved by Xet." So we do NOT build a custom segmented downloader as the default path; that would be redundant and would violate the cross-platform-parity rule.
What's actually missing:
1. **We don't drive Xet well.** `install_model` calls `snapshot_download(**dl_kwargs)` with no `max_workers`, no `tqdm_class`, no `dry_run`, and no explicit dependency pin — progress rides on a global tqdm monkeypatch.
2. **No pre-flight total**, so "downloaded/remaining" has no denominator until files appear, and aggregate speed is summed frontend-side from per-file events (inaccurate under parallel fetch).
3. **Legacy non-Xet (LFS) repos get zero intra-file parallelism** — this is the one place a real IDM-style multi-connection fetch still helps, so we add it as an OPT-IN accelerator.
Five waves, in order (each independently shippable, continuous-to-main per v0.3.0 cadence):
- **Wave 0 — Spike/gate (FDL-00):** classify every catalog repo Xet vs LFS. Sizes Wave 3's value; if ~all repos are Xet-backed, Wave 3 is low-priority polish.
- **Wave 1 — Maximize + guarantee Xet (FDL-01..04):** pin deps, drive snapshot_download explicitly, surface fast-download status, opt-in high-perf/HDD knobs. No new deps, all platforms.
- **Wave 2 — Accurate progress (FDL-05..07):** dry_run preflight + backend aggregate tracker + overall UI bar (speed/remaining/ETA). The biggest user-visible win.
- **Wave 3 — Opt-in segmented accelerator (FDL-08..09):** custom httpx Range downloader for LFS repos. Default OFF, opt-in toggle.
- **Wave 4 — Mirror path + docs (FDL-10..12):** opt-in HF_ENDPOINT, cancel endpoint, docs-sync.
Out of scope / explicitly rejected (call out, do NOT do):
- **hf_transfer / HF_HUB_ENABLE_HF_TRANSFER** — deprecated, breaks progress callbacks. Never enable.
- **Bundling aria2c** — per-OS GPLv2 binary + parity burden; the custom httpx path covers the same need without a binary.
- **Making the segmented downloader the default** — redundant vs Xet, violates parity rule. Always opt-in.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/STATE.md
@./CLAUDE.md
@.planning/quick/260613-fdl-fast-model-downloads/260613-fdl-RESEARCH.md
# Files under edit (read before editing)
@backend/api/routers/setup/download.py
@backend/api/routers/setup/models.py
@backend/utils/hf_progress.py
@backend/api/routers/system.py
@frontend/src/pages/Settings.jsx
# Reference only — patterns, do NOT modify
@backend/core/prefs.py
@frontend/src/api/setup.ts
@frontend/src/api/hooks.ts
<interfaces>
<!-- Verified during planning against the live env (huggingface_hub 1.7.2, hf_xet installed). -->
huggingface_hub 1.7.2 snapshot_download params (confirmed via inspect):
repo_id, repo_type, revision, cache_dir, local_dir, library_name, library_version,
user_agent, etag_timeout, force_download, token, local_files_only,
allow_patterns, ignore_patterns, max_workers, tqdm_class, headers, endpoint, dry_run
- dry_run=True -> returns per-file info incl. size + cached/not-cached (use for FDL-05 preflight).
- tqdm_class=<cls> -> drives the AGGREGATE bar; Xet feeds bytes into it (this is the xet-aware progress hook).
- max_workers -> parallel FILES (default 8); orthogonal to Xet intra-file chunk parallelism.
- endpoint -> per-call HF endpoint override (FDL-10 mirror, instead of process-wide HF_ENDPOINT).
backend/utils/hf_progress.py (existing):
- Monkeypatches huggingface_hub.utils.tqdm.tqdm -> TrackedTqdm (install() at startup).
- register_listener/unregister_listener; emit(event); current_repo_id contextvar stamps events.
- TrackedTqdm.update()/display() emit per-file {filename, downloaded, total, pct, rate, phase} throttled ~0.3s.
- GAP: per-file only, no aggregate, no preflight total. Wave 2 adds the aggregator on top (keep TrackedTqdm; feed it).
backend/api/routers/setup/download.py (existing):
- install_model (line 122): snapshot_download(**dl_kwargs) inside asyncio.to_thread; 5-retry backoff; heartbeat;
_validate_snapshot_has_weights (line 55); _install_cooldowns (line 27, see MM2-06 for bounding).
- SSE feed: GET /setup/download-stream (line 80) forwards hf_progress events.
backend/core/prefs.py:
- resolve(key, *, env=None, default=None) (line 75) — env wins, then store, then default. Use for all new toggles.
Xet env knobs (research): HF_XET_HIGH_PERFORMANCE=1 (opt-in max throughput; needs RAM/bandwidth),
HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1 (HDD), HF_XET_NUM_CONCURRENT_RANGE_GETS (default 16),
HF_XET_DATA_PROGRESS_UPDATE_INTERVAL (200ms). hf_xet is 64-bit only.
</interfaces>
</context>
<tasks>
<!-- ════════════ WAVE 0 — SPIKE / GATE ════════════ -->
<task type="auto">
<name>Task 0 (FDL-00): Classify catalog repos Xet vs LFS</name>
<files>.planning/quick/260613-fdl-fast-model-downloads/260613-fdl-SPIKE.md</files>
<action>
For every repo in backend/config/models.yaml (25 entries), determine whether it's Xet-backed or legacy Git-LFS. Use huggingface_hub: `HfApi().repo_info(repo_id, files_metadata=True)` and inspect each LFS blob for xet info, OR call the model-info endpoint and check the `xetEnabled`/blob `xet` field. For gated/unavailable repos, record "unknown (gated/offline)".
Write 260613-fdl-SPIKE.md: a table repo_id | role | backend (xet|lfs|unknown) | size, plus a one-line GO/LOW-PRIORITY verdict for Wave 3:
- If the majority of *user-facing default* models (OmniVoice TTS, the default ASR) are Xet-backed -> Wave 3 is LOW priority (xet already fast); still build it for the LFS long tail.
- If many defaults are still LFS -> Wave 3 is HIGH priority.
This is read-only network classification — do not download anything (use repo_info, not snapshot_download).
</action>
<verify>
<automated>test -f .planning/quick/260613-fast-model-downloads/260613-fdl-SPIKE.md || test -f .planning/quick/260613-fdl-fast-model-downloads/260613-fdl-SPIKE.md && echo "spike written"</automated>
</verify>
<done>SPIKE.md lists every catalog repo with its storage backend and a GO/LOW-PRIORITY verdict for Wave 3.</done>
</task>
<!-- ════════════ WAVE 1 — MAXIMIZE + GUARANTEE THE XET FAST PATH ════════════ -->
<task type="auto">
<name>Task 1 (FDL-01): Pin huggingface_hub + hf-xet explicitly</name>
<files>pyproject.toml</files>
<action>
Today huggingface_hub arrives transitively (1.7.2) and hf_xet is present but unpinned. Add explicit runtime pins so the fast path can never silently disappear on a resolve:
- huggingface_hub>=1.7 (keep compatible with transformers>=5.3.0 already in deps)
- hf-xet>=1.1 (the Xet backend; 64-bit only — fine for all OmniVoice targets)
Do NOT add hf_transfer. Run `uv sync` then `uv tree huggingface_hub` to confirm a single resolved version (no duplicate). If a transitive constraint conflicts, prefer the higher version and note it in the SUMMARY.
</action>
<verify>
<automated>grep -n "huggingface_hub\|hf-xet\|hf_xet\|hf-transfer\|hf_transfer" pyproject.toml</automated>
<automated>uv run python -c "import huggingface_hub,hf_xet; print('hub',huggingface_hub.__version__,'xet ok')"</automated>
</verify>
<done>pyproject pins huggingface_hub>=1.7 and hf-xet; no hf_transfer; uv resolves cleanly with one huggingface_hub.</done>
</task>
<task type="auto">
<name>Task 2 (FDL-02): Drive snapshot_download explicitly</name>
<files>backend/api/routers/setup/download.py</files>
<action>
In install_model's _do() (line ~148), build dl_kwargs with explicit, intentional args instead of the bare call:
- tqdm_class=<the TrackedTqdm class> so progress is deterministic and xet-aware rather than relying solely on the global monkeypatch. Expose TrackedTqdm from hf_progress (add a getter, e.g. hf_progress.tracked_tqdm_class()).
- max_workers: keep default 8 (don't crank — xet does intra-file parallelism; high max_workers multiplies buffer pressure). Make it prefs-overridable: prefs.resolve("download_max_workers", env="OMNIVOICE_DOWNLOAD_MAX_WORKERS", default=8).
- endpoint=prefs.resolve("hf_endpoint", env="HF_ENDPOINT", default=None) — wires FDL-10 mirror without process-wide env.
- Keep the existing 5-retry backoff, heartbeat, and _validate_snapshot_has_weights.
Do not remove the global monkeypatch (other libs — transformers/mlx_whisper — still rely on it); this task just makes the install path drive its own tqdm_class explicitly.
</action>
<verify>
<automated>grep -n "tqdm_class\|max_workers\|endpoint" backend/api/routers/setup/download.py</automated>
<automated>uv run pytest tests/ -k "download or install" -q 2>&amp;1 | tail -15</automated>
</verify>
<done>install_model drives snapshot_download with explicit tqdm_class + max_workers + endpoint; retry/validate intact; tests green.</done>
</task>
<task type="auto">
<name>Task 3 (FDL-03, FDL-04): fast_download status + opt-in xet knobs</name>
<files>backend/api/routers/system.py, backend/api/routers/setup/download.py</files>
<action>
- FDL-03: add a fast_download block to GET /system/info (system.py:245): {xet_enabled: bool, xet_version: str|None, high_performance: bool}. Probe by importing hf_xet (xet_enabled), reading its version, and reading the HF_XET_HIGH_PERFORMANCE env/pref. Must never throw (system_info is called on every Settings load). Log the same line once at startup ("fast download: Xet on (hf_xet X.Y), high_perf=...").
- FDL-04: opt-in knobs via prefs, applied at process/download setup (env wins):
high_performance = prefs.resolve("xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False)
hdd_sequential = prefs.resolve("xet_hdd_sequential_write", env="HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY", default=False)
When set, export the corresponding HF_XET_* env before the snapshot/segmented download runs. Both default OFF (high-perf can hurt low-RAM machines — surface that as a tooltip in Wave 2 UI).
</action>
<verify>
<automated>curl -s http://127.0.0.1:3900/system/info | python3 -c "import json,sys; print(json.load(sys.stdin).get('fast_download'))" 2>/dev/null || grep -n "fast_download" backend/api/routers/system.py</automated>
</verify>
<done>/system/info reports fast_download truthfully; high-perf + HDD knobs resolve via prefs with env precedence, default off; startup logs Xet status.</done>
</task>
<!-- ════════════ WAVE 2 — ACCURATE DOWNLOADED/REMAINING + SPEED ════════════ -->
<task type="auto">
<name>Task 4 (FDL-05): dry_run preflight -> install_plan event</name>
<files>backend/api/routers/setup/download.py</files>
<action>
Before the real download in install_model, run snapshot_download(repo_id, dry_run=True, endpoint=...) on the worker thread. From the returned per-file info compute: total_bytes, cached_bytes (files already present), to_download_bytes, n_files, n_cached. Emit a new phase event:
{repo_id, phase:"install_plan", total_bytes, cached_bytes, to_download_bytes, n_files, n_cached}
This gives the UI an accurate denominator and a "M GB already cached, N GB to download" line BEFORE bytes flow. Wrap dry_run in try/except — if it fails (older/gated repo), emit install_plan with totals=None and fall back to today's behavior (denominator fills in as files appear). dry_run must respect the 'resolving' heartbeat (it can take a couple seconds).
</action>
<verify>
<automated>grep -n "dry_run\|install_plan\|to_download_bytes" backend/api/routers/setup/download.py</automated>
<automated>uv run pytest tests/backend/setup/test_download_preflight.py -q 2>&amp;1 | tail -15</automated>
</verify>
<done>An install emits install_plan with accurate total/cached/remaining before download; dry_run failure degrades gracefully to old behavior.</done>
</task>
<task type="auto">
<name>Task 5 (FDL-06): Backend aggregate progress tracker</name>
<files>backend/utils/download_aggregator.py, backend/utils/hf_progress.py</files>
<action>
New backend/utils/download_aggregator.py: a per-repo DownloadAggregator that owns the TRUTH for overall progress, so the frontend stops summing potentially-misrouted per-file events.
- Seeded by the install_plan totals (total_bytes, n_files).
- add(filename, bytes_delta) / set_file(filename, downloaded, total): track bytes per file; bytes_done = sum.
- Rate: sampled over a sliding window (e.g. last ~5-10s of (t, bytes_done) samples), not a single tqdm bar's rate. eta = remaining / rate.
- snapshot() -> {repo_id, bytes_done, total_bytes, rate, eta_seconds, files_done, files_total, phase}.
- Emits one throttled (~0.3-0.5s) phase:"aggregate" event via hf_progress.emit().
Wire it: hf_progress's TrackedTqdm._emit_progress already has per-file (filename, downloaded, total) — also feed those into the active repo's aggregator (look up by current_repo_id). The segmented downloader (Wave 3) calls aggregator.add() directly. Keep the per-file events too (UI detail view) — aggregate is additive, not a replacement.
</action>
<verify>
<automated>uv run python -c "from utils.download_aggregator import DownloadAggregator as A; a=A('r',total_bytes=100,files_total=2); a.set_file('f1',50,50); a.set_file('f2',25,50); s=a.snapshot(); print(s['bytes_done'], s['total_bytes'])"</automated>
</verify>
<done>DownloadAggregator sums bytes across parallel files, samples rate over a window, emits a single 'aggregate' event; fed by both tqdm and the segmented path.</done>
</task>
<task type="auto">
<name>Task 6 (FDL-07): Frontend overall progress bar</name>
<files>frontend/src/pages/Settings.jsx, frontend/src/api/setup.ts</files>
<action>
- setup.ts: extend SetupProgressEvent phase union with "install_plan" | "aggregate" and their fields (total_bytes, cached_bytes, to_download_bytes, n_files, n_cached, bytes_done, rate, eta_seconds, files_done, files_total).
- Settings.jsx ModelStoreTab: when an aggregate event arrives for a repo, render ONE overall progress row: a bar (bytes_done/total_bytes), instantaneous speed (format rate as MB/s), "X.X GB of Y.Y GB" downloaded/remaining, and ETA (mm:ss from eta_seconds). Seed the denominator from install_plan (show "M GB cached, N GB to download" before bytes flow). Keep the existing per-file rows as a collapsible "details" section instead of the primary display. Show a small "⚡ fast download" badge when /system/info fast_download.xet_enabled is true.
- Prefer the backend aggregate's rate/eta over the frontend's own per-file ETA computation (Settings.jsx ~614-631) — replace that local ETA math with the aggregate fields; keep a fallback if no aggregate event has arrived yet.
</action>
<verify>
<automated>cd frontend && bun run typecheck 2>&amp;1 | tail -15</automated>
<automated>grep -n "aggregate\|install_plan\|eta_seconds\|fast download" frontend/src/pages/Settings.jsx frontend/src/api/setup.ts</automated>
</verify>
<done>UI shows one overall bar with live speed + downloaded/remaining + ETA from the aggregate event; per-file detail collapsible; fast-download badge; typecheck passes.</done>
</task>
<!-- ════════════ WAVE 3 — OPT-IN IDM-STYLE SEGMENTED ACCELERATOR (LFS REPOS) ════════════ -->
<task type="auto">
<name>Task 7 (FDL-08): Custom httpx segmented downloader</name>
<files>backend/services/segmented_download.py, tests/backend/services/test_segmented_download.py</files>
<action>
New backend/services/segmented_download.py — an OPT-IN multi-connection Range downloader for ONE file (the IDM/uGet technique) used only for legacy-LFS repos where Xet gives no intra-file parallelism. httpx is already a dep.
Contract (async def segmented_download(url, dest, *, token, expected_size, expected_etag=None, num_connections=8, chunk_aggregator=None, cancel_event=None)):
1. HEAD (or GET Range: bytes=0-0) the resolve URL to learn size + Accept-Ranges + the redirect target. If server doesn't honor Range (Accept-Ranges != bytes) -> fall back to a single streamed GET (still works, just not parallel).
2. AUTH SAFETY (critical): send Authorization: Bearer <token> ONLY to the huggingface.co host. When the resolve URL 302-redirects to the CDN (cloudfront/etc.), do NOT forward Authorization to the CDN host — the presigned URL already carries auth. Follow redirects manually so you control header propagation per-host.
3. Split expected_size into num_connections ranges; download each with Range: bytes=start-end concurrently (asyncio + httpx.AsyncClient). Write to dest+".part" at the right offsets (preallocate, or per-range temp files then concat).
4. RESUME: if dest+".part" exists with a sidecar manifest of completed ranges, skip completed ranges.
5. CANCEL: check cancel_event between chunks; on cancel, leave the .part for resume and raise CancelledError.
6. VERIFY: after assembly, check size == expected_size and (if given) sha256/etag matches; only then atomically rename .part -> dest. On mismatch, raise (caller's retry/validate handles it).
7. PROGRESS: call chunk_aggregator.add(filename, bytes_delta) as ranges complete bytes (feeds DownloadAggregator).
Tests (use a local mock HTTP server / httpx MockTransport): honors Range + parallel assembly == single-GET bytes; falls back when Accept-Ranges absent; does NOT send Authorization to a different host on redirect; resumes from a partial .part; cancels and leaves resumable state; size/etag mismatch raises.
</action>
<verify>
<automated>uv run pytest tests/backend/services/test_segmented_download.py -q 2>&amp;1 | tail -20</automated>
<automated>grep -n "Authorization\|Range\|cancel_event\|expected_size" backend/services/segmented_download.py</automated>
</verify>
<done>segmented_download fetches a file via parallel ranges, is auth-safe across the CDN redirect, resumes, cancels, and verifies size/etag before commit; all tests pass.</done>
</task>
<task type="auto">
<name>Task 8 (FDL-09): Dispatch — accelerator for LFS repos only</name>
<files>backend/api/routers/setup/download.py, backend/api/routers/setup/models.py</files>
<action>
- models.py: add a small helper is_xet_backed(repo_id) -> bool|None (reuse FDL-00's classification approach; cache result). Used to decide the path.
- download.py install_model dispatch:
accelerator_on = prefs.resolve("segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=False)
if accelerator_on and is_xet_backed(repo_id) is False:
-> resolve each LFS file's URL via hf_hub_url + HfApi file metadata, download via segmented_download into the HF cache layout (or download to a temp dir then place via the cache API so the result is a normal cache entry), feeding the same DownloadAggregator. Run _validate_snapshot_has_weights at the end.
else:
-> existing snapshot_download path (xet).
IMPORTANT: the segmented result MUST land in the same HF cache structure so /models install-state, delete, and is_cached() all keep working (truth: "indistinguishable from snapshot_download"). If matching the blob/snapshot symlink layout is too fiddly, the safe fallback is: segmented-download to a temp file, then hand the bytes to huggingface_hub so it finalizes the cache entry. Document the chosen approach in SUMMARY.
Default OFF -> zero behavior change unless the user opts in.
</action>
<verify>
<automated>grep -n "segmented_downloader\|is_xet_backed\|segmented_download" backend/api/routers/setup/download.py backend/api/routers/setup/models.py</automated>
<automated>uv run pytest tests/ -k "download or install or model" -q 2>&amp;1 | tail -20</automated>
</verify>
<done>With the toggle ON, LFS repos download via the segmented path into the normal HF cache; Xet repos and the default (toggle OFF) use snapshot_download; install-state/delete unaffected.</done>
</task>
<!-- ════════════ WAVE 4 — MIRROR PATH + CANCEL + DOCS ════════════ -->
<task type="auto">
<name>Task 9 (FDL-10, FDL-11): Mirror opt-in + cancel endpoint</name>
<files>backend/api/routers/setup/download.py</files>
<action>
- FDL-10: the endpoint= wiring from Task 2 already reads prefs hf_endpoint. Surface it as a setting and document (Task 10) that a mirror routes through the CLASSIC LFS path (no Xet) — so it pairs naturally with the FDL-08 segmented accelerator for speed on mirrors. No process-wide HF_ENDPOINT mutation; per-call endpoint only.
- FDL-11: add POST /models/install/cancel {repo_id} that sets the repo's cancel_event (segmented path) and, for the snapshot path, best-effort marks the install cancelled (snapshot_download isn't trivially cancellable mid-file — at minimum stop retries and emit install_cancelled). Compose with MM2-06: on success OR cancel, clear the _install_cooldowns entry so a cancelled download isn't rate-limited. Emit phase:"install_cancelled".
</action>
<verify>
<automated>grep -n "install/cancel\|cancel_event\|install_cancelled\|hf_endpoint" backend/api/routers/setup/download.py</automated>
</verify>
<done>Per-call mirror endpoint wired (opt-in); cancel endpoint stops the segmented path and clears cooldown; emits install_cancelled.</done>
</task>
<task type="auto">
<name>Task 10 (FDL-12): Docs — downloading-models.md + README pointer</name>
<files>docs/downloading-models.md, README.md</files>
<action>
Per the docs-sync hard rule, document the user-facing surface introduced here:
- How fast downloads work (Xet on by default; what the ⚡ badge means; how to check via Settings/system info).
- Advanced toggles: high-performance mode (warn: needs RAM/bandwidth, can hurt low-RAM machines), HDD sequential-write, max workers, segmented accelerator (opt-in, for legacy-LFS repos), and the mirror/restricted-network HF_ENDPOINT setting (note: mirror = classic LFS, no Xet; pair with the accelerator).
- A short troubleshooting section (slow downloads, stuck at resolving, restricted networks/China).
Add a one-line pointer from README.md to docs/downloading-models.md. Do NOT enable any opt-in by default in docs examples.
</action>
<verify>
<automated>test -f docs/downloading-models.md && grep -n "Xet\|HF_ENDPOINT\|high-performance\|segmented" docs/downloading-models.md | head</automated>
<automated>grep -n "downloading-models" README.md</automated>
</verify>
<done>docs/downloading-models.md covers speed, status, all opt-in knobs, mirror/restricted-network, troubleshooting; README links it; no opt-in shown as default.</done>
</task>
</tasks>
<verification>
Gate per wave; full set before the last PR:
1. `uv run pytest tests/backend/setup/test_download_preflight.py tests/backend/services/test_segmented_download.py tests/ -k "download or install or model" -q` — green.
2. Live smoke (backend running): an install emits install_plan (accurate total/cached/remaining) THEN aggregate events with rising bytes_done + a non-zero rate + decreasing ETA; on completion bytes_done == total_bytes.
3. /system/info reports fast_download.xet_enabled=true with a version.
4. Auth-safety unit test proves Authorization is NOT sent to a non-huggingface.co host on redirect.
5. Default-off proof: with no opt-in set, an install uses snapshot_download (xet) — `OMNIVOICE_SEGMENTED_DOWNLOAD` unset means the segmented path is never taken.
6. `uv tree huggingface_hub` shows one version; no hf_transfer anywhere (`grep -ri hf_transfer backend/` is empty).
7. `cd frontend && bun run typecheck` passes.
8. Cross-platform parity: the default path (Xet, pure-Python) is identical on all 3 OSes; every accelerator/mirror/high-perf knob is opt-in (Settings/env). No bundled binary added.
</verification>
<success_criteria>
- Fast: Xet is pinned, engaged, and driven with explicit args; high-perf/HDD knobs available opt-in; legacy-LFS repos can use the opt-in segmented accelerator for real multi-connection speed.
- Accurate: UI shows pre-flight total/cached/remaining, then one overall bar with live speed + downloaded/remaining + ETA sourced from a backend aggregate (not frontend guesswork).
- Safe & compatible: no hf_transfer; segmented downloader is opt-in, auth-safe, resumable, verified, cancellable, and lands in the normal HF cache; default behavior identical on all 3 OSes; no new on-disk model state; existing installs untouched.
- All listed tests + typecheck pass; docs updated in the same PR (docs-sync rule).
</success_criteria>
<risks>
- **Segmented downloader auth leak (FDL-08) — highest risk.** Forwarding the HF Authorization header to the CDN host on redirect would leak the token. Mitigation: manual redirect handling, per-host header allow-list (Authorization only to huggingface.co), and a dedicated unit test asserting no Authorization on the CDN hop. This is a must-have truth, not optional.
- **Cache-layout mismatch (FDL-09).** If the segmented path writes files outside the HF cache blob/snapshot structure, /models install-state + delete + is_cached() break. Mitigation: prefer the temp-file-then-hand-to-huggingface_hub finalization approach over hand-rolling the symlink/blob layout; assert is_cached(repo_id) is true after a segmented install in a test.
- **dry_run cost/availability (FDL-05).** dry_run adds a metadata round-trip and may not exist for gated/older repos. Mitigation: try/except -> totals=None fallback to current fill-in-as-you-go behavior; keep the resolving heartbeat so the UI isn't blank during preflight.
- **Aggregate vs per-file double-count (FDL-06).** Feeding both tqdm per-file events and the aggregator risks the UI showing two competing numbers. Mitigation: aggregate is the single source of truth for the overall bar; per-file events only drive the collapsible detail view; the frontend's old per-file ETA math is removed (Task 6).
- **High-performance mode hurting low-RAM machines (FDL-04).** HF_XET_HIGH_PERFORMANCE can need ~tens of GB RAM. Mitigation: default OFF, opt-in only, tooltip warning in the UI.
- **Mirror + Xet confusion (FDL-10).** Users may expect Xet speed through a mirror; mirrors fall back to classic LFS. Mitigation: document explicitly; that's exactly why the segmented accelerator pairs with the mirror path.
- **Scope: do not let the segmented path become default.** It's opt-in for LFS repos only. Xet stays the default; making it default would regress dedup + violate the parity rule.
</risks>
<output>
Write 260613-fdl-SPIKE.md (Task 0) and 260613-fdl-SUMMARY.md when done. SUMMARY must record: the Xet-vs-LFS catalog breakdown and how it changed Wave 3 priority; the cache-finalization approach chosen for the segmented path (and the is_cached-after-segmented test result); the exact new SSE event shapes (install_plan, aggregate); which opt-in prefs keys + env vars were added; and the auth-safety test output. Note any "use judgment" decision an executor made.
Docs-sync (CLAUDE.md hard rule): docs/downloading-models.md + README pointer ship in the SAME PR as the user-facing toggles (Task 10). If the Settings UI gains the new toggles, the docs describing them land together.
</output>
@@ -1,43 +0,0 @@
# RESEARCH — Fast HuggingFace model downloads (2026)
**Date:** 2026-06-13 · **For:** 260613-fdl-PLAN.md
## Bottom line
As of mid-2026 the fast path is **hf-xet, on by default** in modern `huggingface_hub`. Xet is itself a chunk-level, content-defined, massively-parallel downloader with adaptive concurrency — it **is** the "IDM/uGet-style segmented download," done for you and dedup-aware. `hf_transfer` is **deprecated**. Rolling your own segmented downloader or bridging to aria2 is **not worth it as a default**; the only thing we must build is (a) better driving + progress UI and (b) an **opt-in** segmented path for the legacy-LFS long tail (repos Xet doesn't back).
Installed in this repo: `huggingface_hub 1.7.2`, `hf_xet` present. `snapshot_download` here supports `max_workers`, `tqdm_class`, `endpoint`, `dry_run` (confirmed via inspect).
## 1. hf-xet — USE (default, no action needed beyond pinning)
Content-defined chunks grouped into blocks ("xorbs") in a content-addressable store; download = send file SHA256 → get reconstruction metadata + presigned URLs → fetch needed xorb ranges **in parallel** → reassemble; already-present chunks skipped (dedup). Auto-used by `snapshot_download`/`hf_hub_download` for Xet-backed repos since huggingface_hub 0.32. 23× over Git-LFS, up to ~1 GB/s.
Knobs (defaults already tuned): `HF_XET_NUM_CONCURRENT_RANGE_GETS` (16), adaptive concurrency ON (max 64), `HF_XET_DATA_MAX_CONCURRENT_FILE_DOWNLOADS` (8), chunk cache disabled by default (better for pure download), `HF_XET_HIGH_PERFORMANCE=1` (opt-in max throughput, needs RAM/bandwidth), `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1` (HDD). **64-bit only.**
- https://huggingface.co/docs/huggingface_hub/en/guides/download
- https://huggingface.co/docs/hub/en/xet/using-xet-storage
- https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables
## 2. hf_transfer — AVOID (deprecated)
`HF_HUB_ENABLE_HF_TRANSFER` flagged deprecated; Xet supersedes it. Historically **broke tqdm progress / had no callbacks** — directly conflicts with the accurate-progress goal. Successor for max throughput is `HF_XET_HIGH_PERFORMANCE=1`.
- https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables
- https://github.com/huggingface/hf_transfer/issues/63
## 3. huggingface_hub native concurrency — USE defaults
`snapshot_download(max_workers=...)` = parallel FILES (default 8), orthogonal to Xet's intra-file chunk parallelism. For OmniVoice's 1few-large-file models the win is mostly Xet's intra-file parallelism; don't crank max_workers (multiplies buffer pressure). Resume is automatic via cache + ETag (no `resume_download` flag to manage).
- https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
## 4. Custom IDM-style Range downloader — AVOID as default, BUILD as opt-in for LFS
`/resolve/<rev>/<file>` 302-redirects to CDN (Cloudfront) which honors Range + parallel byte-ranges. Catch: follow redirect, **do NOT forward Authorization to the CDN host** (presigned URL carries auth), verify ETag/sha256, auth on first hop only. Redundant vs Xet for Xet-backed repos (HF closed issue #3232 as "use Xet"), **but genuinely helps non-Xet/legacy-LFS repos** which get no intra-file parallelism. → our Wave 3 opt-in.
- https://github.com/huggingface/huggingface_hub/issues/3232
## 5. aria2 — OPTIONAL, rejected for OmniVoice
`aria2c -x16 -s16 -c --header="Authorization: Bearer <token>"` is 35× on plain LFS, but: no dedup (worse than Xet for Xet repos), per-OS GPLv2 binary to package (parity burden — would have to be opt-in anyway), stdout/RPC progress scraping. The custom httpx path covers the same need with no binary. → not bundled.
- https://gist.github.com/padeoe/697678ab8e528b85a2a7bddafea1fa4f
## 6. Mirrors / HF_ENDPOINT — OPTIONAL, region-gated, breaks Xet
`HF_ENDPOINT=https://hf-mirror.com` redirects Hub traffic (standard for China). **Xet CAS/presigned URLs point at HF infra → mirrors generally don't serve the Xet protocol → traffic falls back to classic LFS** (no dedup, no Xet parallelism). So mirror and Xet fast-path are mutually exclusive; the realistic China stack is mirror + LFS + (our opt-in) segmented accelerator. → our Wave 4 opt-in, per-call `endpoint=` not process-wide.
## 7. Progress / speed — USE `tqdm_class` (xet-aware) + `dry_run` preflight
Unlike hf_transfer, **Xet reports progress through the same tqdm interface**; huggingface_hub aggregates per-file/thread bytes into a shared bar and feeds the `tqdm_class` you pass. So `snapshot_download(tqdm_class=...)` yields reliable aggregate bytes/total/rate/ETA even under parallel fetch. `snapshot_download(dry_run=True)` returns per-file sizes + cached flags → use for "will download X of Y, N GB" preflight. Speed sampling tunable via `HF_XET_DATA_PROGRESS_UPDATE_INTERVAL` (200ms).
- https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
- https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/_snapshot_download.py
## Recommended architecture (→ plan)
Pin `huggingface_hub>=1.7` + `hf-xet`; let Xet be the default (it IS the IDM technique). Drive `snapshot_download(repo_id, tqdm_class=OmniVoiceProgress, max_workers=8, endpoint=<opt-in mirror>)`; `dry_run=True` first for total/remaining; aggregate bytes in a backend tracker → one overall bar (speed/remaining/ETA). Opt-in only: `HF_XET_HIGH_PERFORMANCE` (max speed), `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY` (HDD), a custom httpx **segmented downloader for legacy-LFS repos**, and an `HF_ENDPOINT` mirror (classic-LFS fallback). Never enable hf_transfer; never bundle aria2; never make the segmented path the default.
@@ -1,28 +0,0 @@
# SPIKE — FDL-00: Catalog Xet vs LFS classification
**Date:** 2026-06-13 · **Method:** HF API `GET /api/models/{repo}?expand[]=xetEnabled` (authoritative).
## Result: 25 / 25 catalog repos are Xet-backed
| backend | count |
|---------|-------|
| xet | 25 |
| lfs | 0 |
| unknown | 0 |
Every repo in `backend/config/models.yaml` — including both first-run defaults (`k2-fsa/OmniVoice` TTS, `Systran/faster-whisper-large-v3` ASR) — returns `xetEnabled: true`. Full list: all entries under TTS / ASR / Diarisation (k2-fsa, Systran×5, mlx-community×9, openai, nvidia×2, UsefulSensors×2, pyannote, OpenMOSS, KittenML, deepdml).
## Detection caveat (important for the executor)
The installed client is **huggingface_hub 1.7.2**, whose `repo_info(..., files_metadata=True)` siblings expose only `blob_id, lfs, rfilename, size`**no `xet_file`, and no `xet_enabled` on the info object.** A first pass that inferred backend from siblings wrongly reported "0/25 xet, all LFS." Do **not** classify Xet status from `repo_info` siblings on this client version. The reliable signal is the Hub API `xetEnabled` expand field (used here) or `hf_xet` actually engaging at download time. Re-check after any `huggingface_hub` bump — newer versions surface `xet_enabled` directly.
## Verdict for Wave 3 (segmented accelerator): LOW priority
Because the entire current catalog is Xet-backed and `hf_xet` is installed, Xet already provides chunked parallel range-gets (the IDM/uGet behavior) for **every** model we ship. The custom segmented downloader (Wave 3) is therefore **not needed to speed up any current default model** — it remains valuable only for:
- the **mirror / restricted-network path** (Wave 4: `HF_ENDPOINT` falls back to classic LFS, no Xet), and
- any **future non-Xet repo** a user adds.
**Recommendation:** proceed with W1 (maximize/guarantee Xet) and W2 (accurate progress) as the real wins for today's catalog; keep W3 as opt-in, build it alongside W4's mirror path where it actually pays off. This matches the PLAN's original framing — confirmed, not changed.
## Consequence for W1/W2 framing
W1 "guarantee the Xet fast path" is correctly the primary lever: these repos download via Xet **only if** the client engages it (hf_xet installed ✓ + huggingface_hub recent ✓). The W2 live smoke test should confirm Xet is actually used (fast parallel aggregate progress on a real install), since `xetEnabled=true` is a Hub-side capability, not proof the client took the Xet path.
@@ -1,49 +0,0 @@
# SUMMARY — FDL Waves 02 (fast model downloads)
**Date:** 2026-06-13 · **Scope shipped:** W0 (spike), W1 (maximize Xet), W2 (accurate progress). W3/W4 deferred.
## What landed
**W0 — spike (FDL-00).** Classified all 25 `models.yaml` repos via the HF API `xetEnabled` field → **25/25 Xet-backed** (incl. both first-run defaults). See `260613-fdl-SPIKE.md`. Verdict: Wave 3 (segmented accelerator) is **LOW priority** — Xet already gives parallel chunked transfer for every shipped model. Detection caveat recorded: `repo_info` siblings on hf_hub **1.7.2** expose no xet metadata; classify via the `xetEnabled` API field, not siblings.
**W1 — maximize + guarantee Xet (FDL-01..04).**
- `pyproject.toml`: pinned `huggingface_hub>=1.7` + `hf-xet>=1.1` explicitly (was transitive/unpinned); no `hf_transfer`. Resolves to hf_hub 1.7.2 / hf-xet 1.4.2, single version.
- `download.py`: `install_model` now drives `snapshot_download` with explicit `tqdm_class` (our progress-emitting subclass), `max_workers` (prefs `download_max_workers`, default 8), and `endpoint` (prefs `hf_endpoint` — W4 hook). `apply_xet_env()` applies opt-in `HF_XET_HIGH_PERFORMANCE` + `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY` (both default OFF, env wins).
- `system.py`: `/system/info` now returns `fast_download {xet_enabled, xet_version, high_performance}`; logged once at startup.
**W2 — accurate downloaded/remaining + speed (FDL-05..07).**
- Preflight `snapshot_download(dry_run=True)``compute_plan()``install_plan` SSE event with `total_bytes / cached_bytes / to_download_bytes / n_files / n_cached` **before bytes flow**. Degrades to totals=None on gated/older repos.
- New `utils/download_aggregator.py`: one source of truth for overall progress. Fed by a byte-sink on the patched tqdm; distinguishes byte bars (unit 'B', keyed by bar id) from the "Fetching N files" count bar; emits one throttled `aggregate` event (bytes_done/total/windowed rate/eta/files).
- Frontend `Settings.jsx` + `setup.ts`: overall bar driven by the aggregate; bar % = `max(byte%, file%)`; shows cached-skip + files-progress; `⚡ fast download` badge from `/system/info`. i18n keys added to `en.json`.
## Rebase reconciliation (main disabled Xet)
Rebasing onto latest main surfaced that main now sets **`HF_HUB_DISABLE_XET=1`** (main.py) — a deliberate choice to force the classic LFS path because Xet's progress bypasses the tqdm hook (the exact limitation found here). Reconciled rather than fought:
- `fast_download` status now reports the **runtime truth**: `xet_installed` + `xet_active` (active = installed AND not disabled) + `xet_enabled` alias. Default `xet_active=false`; the ⚡ badge only shows when Xet actually runs. Startup log: `downloads: Xet disabled → legacy LFS …`.
- Docs rewritten: default backend is **legacy LFS for accurate progress**; Xet is opt-in via `HF_HUB_DISABLE_XET=0` (coarser progress). The hf-xet pin stays (harmless; ready for a future Xet progress hook).
- Net: W2's progress is the value either way; W1's "maximize Xet" is dormant by main's design, not removed.
## Decisions / "use judgment" notes
- **Xet progress limitation (verified by live smoke).** Under Xet + hf_hub 1.7.2 the per-file **byte** bars never advance `n` and never `close()` through our tqdm (Xet fetches chunks out-of-band). Only the **file-count bar** is live. So: mid-download the overall bar is **file-granular** (moves 0→N files), and `complete()` flushes `bytes_done` to the exact preflight total on success (verified: final `74420620/74420620`, files 4/4). True live byte-speed is only available on classic-LFS/mirror repos (W4). This is a real constraint, not a bug — documented here and worth surfacing in W4 docs.
- Per-file detail kept inline (existing single-line summary, now aggregate-sourced) rather than a new collapsible panel — limited risk; can revisit.
## Drive-by fix
- `download.py` imported no `os`, but `_validate_snapshot_has_weights` uses `os.walk` → latent `NameError` on every install. Added `import os`.
## Verification
- `tests/backend/setup/test_download_preflight.py` — 10 pass (compute_plan splits, aggregator byte/count routing, close-credit, windowed rate/eta, registry feed + finish noop).
- `pytest -k "download or install or model or engine or setup"` — 149 passed, 7 skipped, 0 failed.
- `frontend typecheck:ci` — exit 0.
- Live smoke (real install of `mlx-community/whisper-tiny-mlx`, then deleted): `install_plan` exact; aggregate files 0→1→4; final bytes==total; `/system/info` + startup log correct.
## W4 — mirror + cancel + docs (FDL-10..12, shipped)
- **Mirror (FDL-10):** `snapshot_download(endpoint=…)` honours prefs `hf_endpoint` / env `HF_ENDPOINT` on both preflight and download — per-call, no process-wide mutation. Documented as the classic-LFS (non-Xet) path that restores continuous byte-speed.
- **Cancel (FDL-11):** `POST /models/install/cancel {repo_id}` sets a cancel flag checked at each retry boundary → emits `install_cancelled`, clears the cooldown (cancel ≠ failure). Limitation: an in-flight single-file fetch isn't interruptible in hf_hub 1.7.2; cancel lands at the next retry boundary. Frontend treats `install_cancelled` as a terminator (clears row + refetch).
- **Docs (FDL-12):** `docs/downloading-models.md` (Xet fast path, progress semantics incl. the byte-speed limitation, opt-in tuning knobs, mirror/restricted-network, cancel, troubleshooting) + README pointer. Docs-sync rule satisfied in-PR.
## W3 — opt-in segmented accelerator (FDL-08/09, shipped)
Reprioritised from LOW to HIGH after the rebase: since main forces Xet off, the default path is single-stream legacy LFS, so a segmented downloader is the way to get **both** parallel speed and live byte progress.
- `services/segmented_download.py`: async multi-connection Range downloader for one file — parallel byte-ranges, resume (`.part` + manifest), per-segment short-read truncation guard, optional sha256/etag verify, cancel, single-stream fallback when the server won't range. **Auth-safe**: the HF `Authorization` header goes only to `huggingface.co`/`hf.co`; never forwarded to a CDN host on redirect (unit-tested).
- Dispatch (`download.py`): opt-in via prefs `segmented_downloader` / env `OMNIVOICE_SEGMENTED_DOWNLOAD` (default OFF). When on and Xet inactive, fetches each repo file into the HF cache mirroring `hf_hub_download` (blobs + snapshot symlinks + `refs/main`), feeding **real bytes** to the aggregator. Any failure falls back to `snapshot_download` — the accelerator can never break a correct install.
- Verified live (accelerator ON): real mid-download byte progress (1.5 KB → 71 MB, rate ramping to **16.6 MB/s**), final `bytes_done == total`, `/models` shows `installed: True`, delete frees the right bytes.
- Fixed a `complete()` double-count (was adding a full total on top of accumulated segmented bytes → 2×); now replaces byte bars so the sum is exactly total.
- Tests: `tests/backend/services/test_segmented_download.py` (7 cases) covering parallel range reassembly, single-stream fallback, the auth header reaching only the HF host (never a CDN), size/truncation rejection, cancellation, and byte-callback totals — plus an aggregator double-count regression.
@@ -1,363 +0,0 @@
---
phase: 260613-mm2
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- backend/services/tts_backend.py
- backend/services/model_manager.py
- backend/services/subprocess_backend.py
- backend/services/model_lifecycle.py # NEW
- backend/api/routers/system.py
- backend/api/routers/setup/download.py
- backend/api/routers/setup/models.py
- tests/test_engines.py
- tests/backend/services/test_model_lifecycle.py # NEW
- tests/backend/services/test_subprocess_reaper.py
autonomous: true
requirements:
# ── Tier 1 — Correctness (Wave 1) ──────────────────────────────────────────
- MM2-01 # Registry reuses one active instance and calls unload() on engine switch
- MM2-02 # Per-engine unload() overrides (in-process drop+free_vram; subprocess -> unload_sidecar)
- MM2-03 # /model/loaded + /model/unload report ASR honestly; no unloadable:False-but-loaded lies
# ── Tier 2 — Single lifecycle surface (Wave 2) ─────────────────────────────
- MM2-04 # model_lifecycle facade owns list_loaded/unload/unload_all/free_vram across all 3 worlds
- MM2-05 # Idle/timeout config unified through core.prefs.resolve (env still wins); no duplicated constants
# ── Tier 3 — Robustness & observability (Wave 3) ───────────────────────────
- MM2-06 # _install_cooldowns bounded (evict on success + TTL); no unbounded growth
- MM2-07 # Snapshot weight validation is per-role, not one 5 MB magic number
- MM2-08 # Subprocess sidecars self-report VRAM in pong; panel shows real MB, not 0
- MM2-09 # scan_cache_dir() -> disk-walk fallback logs WHY it fell back (WinError #117/#118)
must_haves:
truths:
- "Switching the active TTS backend in Settings releases the outgoing engine's VRAM before the new one loads — verified by asserting the outgoing instance's unload() was called exactly once on switch."
- "TTSBackend.unload() is overridden by OmniVoiceBackend (drops model ref + free_vram) and by every SubprocessBackend subclass (routes to unload_sidecar); all overrides are idempotent and safe before first generate()."
- "/model/loaded never reports a model as loaded with a misleading unloadable flag: the ASR row's unloadable reflects whether it can actually be released independently of the TTS lifecycle."
- "services.model_lifecycle is the single import surface for list_loaded()/unload(id)/unload_all()/free_vram(); system.py routers call it instead of re-enumerating models inline."
- "Idle timeouts for the in-process model and subprocess sidecars resolve through core.prefs.resolve(... env=...) so an env var still wins and the Settings store can override; no module duplicates IDLE_TIMEOUT_SECONDS by hand."
- "_install_cooldowns cannot grow without bound: entries are removed on successful install and stale entries are evicted by TTL."
- "A live subprocess sidecar reports a non-zero vram_mb in /model/loaded when it actually holds GPU memory (pong carries the figure); CPU-only sidecars report 0 truthfully."
- "When scan_cache_dir() raises and the code falls back to the on-disk walk, the reason is logged at WARNING with the exception type (the #117/#118 WinError-448 path is no longer silent)."
- "uv run pytest tests/test_engines.py tests/backend/services/test_model_lifecycle.py tests/backend/services/test_subprocess_reaper.py tests/test_model_load_timeout.py passes."
- "No on-disk model state changes; no new runtime dependency added; behavior degrades gracefully (not errors) on MPS/CPU where VRAM APIs are sparse."
artifacts:
- path: "backend/services/tts_backend.py"
provides: "Active-instance reuse + unload-on-switch in get_active_tts_backend(); per-engine unload() overrides"
contains: "_active_instance AND (def unload)"
- path: "backend/services/model_lifecycle.py"
provides: "Facade owning list_loaded/unload/unload_all/free_vram across in-process + subprocess models"
contains: "def list_loaded AND def unload_all"
- path: "backend/api/routers/system.py"
provides: "Thin /model/loaded + /model/unload routers delegating to model_lifecycle"
contains: "model_lifecycle"
key_links:
- from: "get_active_tts_backend() (tts_backend.py:1235)"
to: "outgoing backend.unload()"
via: "module-level _active_instance compared against newly-resolved active_backend_id()"
pattern: "_active_instance"
- from: "system.py /model/loaded + /model/unload (system.py:129, 210)"
to: "model_lifecycle.list_loaded() / model_lifecycle.unload()"
via: "import services.model_lifecycle"
pattern: "model_lifecycle\\.(list_loaded|unload)"
- from: "subprocess sidecar pong reply (subprocess_backend.py:435-438)"
to: "list_live_sidecars() vram_mb field"
via: "ping reply carries allocated VRAM measured inside the sidecar process"
pattern: "vram_mb"
---
<objective>
Clean up OmniVoice's model-management subsystem ("v2"). Today load / unload / list / free-VRAM each behave differently across three worlds — the in-process model (`model_manager.py`), the TTS backend registry (`tts_backend.py`), and subprocess sidecars (`subprocess_backend.py`) — with no single lifecycle owner. This produces one real user-facing bug (VRAM leak on engine switch), inaccurate VRAM/unloadable reporting, an unbounded cooldown dict, and a silent cache fallback.
This is **cleanup + correctness, not a rewrite.** The Wave 13 idle-reaper and the SubprocessBackend primitive are sound and stay. The `TTSBackend.unload()` contract already exists as a documented default no-op (`tts_backend.py:149`) explicitly deferred to "Phase 2"; this plan *is* that Phase-2 follow-through — wire the registry to call it, override it per engine, and unify the surrounding surface.
Three tiers, executed in order (each independently shippable, continuous-to-main per the v0.3.0 cadence):
- **Wave 1 / Tier 1 — Correctness:** MM2-01..03. The VRAM leak on switch + honest unload reporting. Highest value; ship first.
- **Wave 2 / Tier 2 — Single lifecycle surface:** MM2-04..05. Extract `model_lifecycle` facade + unify idle/timeout config.
- **Wave 3 / Tier 3 — Robustness & observability:** MM2-06..09. Bounded cooldowns, per-role weight validation, sidecar VRAM self-report, cache-fallback logging.
Output: PRs on branches off `main` (one per wave is fine), each green on the listed pytest selection. No push until the orchestrator merges; tests added with each wave.
Out of scope (call out, do not touch): GPU-pool per-engine sizing (`model_manager.py:42`, `_GPU_VRAM_PER_JOB_GB`) and torch.compile tuning — those are performance, not cleanup, and carry regression risk against #278/#315.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/STATE.md
@./CLAUDE.md
# Files under edit (read before editing)
@backend/services/tts_backend.py
@backend/services/model_manager.py
@backend/services/subprocess_backend.py
@backend/api/routers/system.py
@backend/api/routers/setup/download.py
@backend/api/routers/setup/models.py
# Reference only — establish patterns, do NOT modify
@backend/core/prefs.py
@tests/test_engines.py
@tests/backend/services/test_subprocess_reaper.py
<interfaces>
<!-- Verified during planning. Executor should use these directly. -->
backend/services/tts_backend.py
- class TTSBackend(ABC) (line 59); unload() default no-op (line 149) — contract already documented:
idempotent, synchronous, safe before first generate().
- OmniVoiceBackend.__init__(self, model=None) (line 174); self._model reuses model_manager singleton.
- _REGISTRY: dict[str, type[TTSBackend]] (line 1109, a _LazyRegistry).
- active_backend_id() (line 1228) -> prefs.resolve("tts_backend", env="OMNIVOICE_TTS_BACKEND", default="omnivoice").
- get_active_tts_backend(*, model=None) (line 1235) — builds a FRESH instance every call, no teardown. THE leak.
backend/core/prefs.py
- resolve(key: str, *, env: Optional[str] = None, default: Any = None) -> Any (line 75) — env wins, then store, then default.
backend/services/model_manager.py
- module global `model` (line 111); `_last_used`; free_vram() (line 678); idle_worker() (line 667).
- IDLE_TIMEOUT_SECONDS imported from core.config (line 33); duplicated as _IDLE_TIMEOUT_SECONDS (line 114). Collapse.
- offload_tts_for_asr() (line 701) / restore_tts_after_asr() — ad-hoc ASR<->TTS VRAM juggling; _diar_pipeline global.
backend/services/subprocess_backend.py
- protocol op set (line 74); SIDECAR_IDLE_TIMEOUT_S = env-only float (line 107) — move to prefs.resolve.
- list_live_sidecars() -> list[dict] (line 181); unload_sidecar(engine_id) (line 199); unload_all_sidecars() (line 205).
- health/ping: _send({"op":"ping"}) then expect {"op":"pong"} (lines 435-438). Add vram_mb to the pong here AND
in the sidecar entry-point that answers ping (search the sidecar worker for the "ping"->"pong" handler).
backend/api/routers/system.py
- GET /model/loaded (line 129) — ~80 lines of inline enumeration of TTS/ASR/diar/sidecars. Replace body with
model_lifecycle.list_loaded().
- POST /model/unload/{model_id} (line 210) — handles "tts" | "diarization" | "sidecar:<id>" | "sidecars".
Replace body with model_lifecycle.unload(model_id).
backend/api/routers/setup/download.py
- _install_cooldowns dict (line 27) — unbounded. _validate_snapshot_has_weights (line 55) + _MIN_WEIGHT_BYTES 5 MB
(line 45) — single magic number across roles.
backend/api/routers/setup/models.py
- scan_cache_dir() with silent disk-walk fallback (~line 268-280) + _scan_cache_on_disk (line 177).
</interfaces>
</context>
<tasks>
<!-- ════════════ WAVE 1 / TIER 1 — CORRECTNESS ════════════ -->
<task type="auto">
<name>Task 1 (MM2-02): Per-engine unload() overrides</name>
<files>backend/services/tts_backend.py</files>
<action>
The base-class `unload()` no-op already exists (tts_backend.py:149) with a documented contract. Override it where it matters. Do this BEFORE Task 2 — the registry switch (Task 2) calls these.
- OmniVoiceBackend (line 162): override `unload(self)`. Drop the local model ref (`self._model = None`) and, because OmniVoice shares the singleton owned by model_manager, also release that: `import services.model_manager as mm; mm.model = None; mm.free_vram()`. Idempotent — guard on `mm.model is not None` before free_vram(). Safe before first generate() (no-op when nothing loaded).
- Every SubprocessBackend subclass: implement `unload(self)` on the SubprocessBackend base (subprocess_backend.py — the duck-typed `_is_subprocess_isolated` class) so all subclasses inherit it. It must call `unload_sidecar(self.id)` (force-shut this engine's sidecar; busy sidecars are skipped, never interrupted — existing semantics). Idempotent: unload_sidecar on a non-running engine returns 0, no raise.
- In-process non-OmniVoice engines that hold their own model (e.g. KittenTTS/VoxCPM2 keep refs in __init__): override unload() to drop the ref + best-effort empty_cache via the existing free_vram() helper if they used GPU. Where an engine genuinely holds nothing resident, leave the base no-op (and note it in the SUMMARY so the future CI gate knows it's intentional, not missed).
Honor the contract comment verbatim: idempotent, synchronous, safe pre-load.
</action>
<verify>
<automated>grep -n "def unload" backend/services/tts_backend.py backend/services/subprocess_backend.py</automated>
<automated>uv run python -c "from services.tts_backend import OmniVoiceBackend; b=OmniVoiceBackend(); b.unload(); b.unload(); print('idempotent ok')"</automated>
</verify>
<done>
- OmniVoiceBackend.unload() drops both self._model and mm.model and calls free_vram(), guarded for idempotency.
- SubprocessBackend.unload() routes to unload_sidecar(self.id); inherited by all subprocess engines.
- Calling unload() twice, and before any generate(), never raises.
</done>
</task>
<task type="auto">
<name>Task 2 (MM2-01): Registry reuses one active instance + unloads on switch</name>
<files>backend/services/tts_backend.py</files>
<action>
Fix the leak at get_active_tts_backend() (line 1235). Today it builds a fresh instance every call with no teardown of the prior engine — switching engines (or repeated synth) leaks VRAM until GC. This is the root cause behind the #278 comment thread.
- Add a module-level cache: `_active_instance: TTSBackend | None = None` and `_active_instance_id: str | None = None`.
- In get_active_tts_backend(): resolve `bid = active_backend_id()`. If `_active_instance is not None` and `_active_instance_id != bid`, call `_active_instance.unload()` (best-effort, wrap in try/except so a bad unload can't block the switch — log on failure) before discarding it.
- Build the new instance, store it as `_active_instance` + `_active_instance_id = bid`, return it.
- IMPORTANT subtlety: OmniVoiceBackend takes `model=`. When `model=` is passed (the caller already has a loaded model), do NOT cache that instance as the shared `_active_instance` blindly — it's a per-call view over the shared singleton. Keep current behavior for the `model=` path (return a fresh OmniVoiceBackend(model=model)) but still trigger unload() of a *different* outgoing engine first. Pick the simplest correct rule: the cache tracks the configured backend id; passing model= for the SAME id reuses, switching id always unloads the previous. Document the rule in a comment.
- Add a module-level `reset_active_backend()` helper that unloads + clears the cache, for app shutdown and tests.
</action>
<verify>
<automated>grep -n "_active_instance\|def reset_active_backend\|def get_active_tts_backend" backend/services/tts_backend.py</automated>
</verify>
<done>
- Switching backend id calls the outgoing instance's unload() exactly once before the new instance is built.
- A bad/raising unload() is caught + logged, never blocks the switch.
- reset_active_backend() exists and is idempotent.
- The model= fast-path for OmniVoice still works (no double-load).
</done>
</task>
<task type="auto">
<name>Task 3 (MM2-03): Honest /model/loaded + /model/unload for ASR</name>
<files>backend/api/routers/system.py</files>
<action>
The ASR row (system.py:166-175) is reported as unloadable:False, vram_mb:0 even when loaded on GPU, and /model/unload doesn't expose the offload-to-CPU path. Make reporting truthful WITHOUT changing the ASR<->TTS lifecycle coupling (that coupling is intentional — offload_tts_for_asr/restore_tts_after_asr).
- ASR row: keep unloadable reflecting reality. If ASR truly cannot be released independently of TTS, keep unloadable:False but add a `note` field ("released with TTS") so the UI explains it rather than showing a dead button. Do not invent a separate ASR unload that breaks the WhisperX large-v3 offload path.
- vram_mb: if ASR currently runs on CPU (device "cpu" in the row), 0 is correct — leave it but make the device value derive from where the pipe actually is, not a hardcoded "cpu".
- This task is intentionally small; the bigger restructure is Task 4 (facade). Land MM2-03 as the honest-reporting fix, then Task 4 moves the enumeration into the facade.
</action>
<verify>
<automated>uv run pytest tests/test_engines.py -q 2>&amp;1 | tail -15</automated>
</verify>
<done>
- No row reports loaded-but-with-a-misleading-unloadable flag; ASR carries an explanatory note when unloadable:False.
- Device field reflects the actual device of the ASR pipe.
</done>
</task>
<task type="auto">
<name>Task 4 (MM2-01..03 tests): Wave 1 regression tests</name>
<files>tests/test_engines.py</files>
<action>
Add tests proving the leak fix and the unload contract:
- test_switching_backend_unloads_previous: monkeypatch two fake backends into _REGISTRY, set active to A (get_active_tts_backend), switch prefs to B, assert A.unload() was called exactly once before B is returned.
- test_unload_is_idempotent_and_preload_safe: OmniVoiceBackend().unload() twice + before generate() never raises.
- test_reset_active_backend_clears_cache: after reset_active_backend(), the next get_active_tts_backend() builds fresh.
Reuse the existing fixture style in tests/test_engines.py (it already monkeypatches the registry / availability). Keep tests CPU-only (no real model load).
</action>
<verify>
<automated>uv run pytest tests/test_engines.py -q 2>&amp;1 | tail -20</automated>
</verify>
<done>All three new tests pass; existing test_engines.py tests still green.</done>
</task>
<!-- ════════════ WAVE 2 / TIER 2 — SINGLE LIFECYCLE SURFACE ════════════ -->
<task type="auto">
<name>Task 5 (MM2-04): Extract services/model_lifecycle.py facade</name>
<files>backend/services/model_lifecycle.py</files>
<action>
Create backend/services/model_lifecycle.py as the single owner of cross-world model lifecycle. It composes the existing pieces — it does NOT reimplement loading.
Public surface:
- list_loaded() -> list[dict]: returns the unified rows currently assembled inline in system.py:129-207 (TTS, ASR, diarization, subprocess sidecars). Move that logic here verbatim first, then improve (MM2-03 note field, MM2-08 sidecar vram once Task 8 lands).
- unload(model_id: str) -> dict: the dispatch currently inline in system.py:210-242 ("tts" | "diarization" | "sidecar:<id>" | "sidecars"). Move here; keep async-lock semantics for the in-process model (mm._model_lock).
- unload_all() -> dict: unload every releasable model (in-process TTS + diar + all sidecars). New convenience used by app shutdown.
- free_vram(): thin re-export of model_manager.free_vram() so callers have one import.
Keep the "never let sidecar enumeration break the panel" try/except guard.
</action>
<verify>
<automated>uv run python -c "import services.model_lifecycle as ml; print([f for f in ('list_loaded','unload','unload_all','free_vram') if hasattr(ml,f)])"</automated>
</verify>
<done>model_lifecycle exposes list_loaded/unload/unload_all/free_vram; logic moved out of system.py (not duplicated).</done>
</task>
<task type="auto">
<name>Task 6 (MM2-04): Thin system.py routers + facade tests</name>
<files>backend/api/routers/system.py, tests/backend/services/test_model_lifecycle.py</files>
<action>
- Replace the bodies of GET /model/loaded (line 129) and POST /model/unload/{model_id} (line 210) with calls to model_lifecycle.list_loaded() / model_lifecycle.unload(model_id). Preserve the exact response shapes (frontend hooks.ts useModelStatus/useFlushMemory + the flush dropdown depend on {models, count} and {unloaded, success, ...}). The 400 on unknown model_id stays.
- New tests/backend/services/test_model_lifecycle.py: list_loaded with nothing loaded returns {models:[], count:0}; unload("tts") when not loaded returns success:False reason:"not loaded"; unload("sidecars") with no sidecars returns count:0; unknown id raises/400 path. Mock model_manager + subprocess_backend so no real models load.
</action>
<verify>
<automated>uv run pytest tests/backend/services/test_model_lifecycle.py -q 2>&amp;1 | tail -20</automated>
<automated>grep -n "model_lifecycle" backend/api/routers/system.py</automated>
</verify>
<done>system.py routers are thin delegations; response shapes unchanged; new facade tests pass.</done>
</task>
<task type="auto">
<name>Task 7 (MM2-05): Unify idle/timeout config through prefs.resolve</name>
<files>backend/services/model_manager.py, backend/services/subprocess_backend.py</files>
<action>
- model_manager.py: remove the duplicated `_IDLE_TIMEOUT_SECONDS = IDLE_TIMEOUT_SECONDS` (line 114). Resolve at use-site in idle_worker() via prefs: `prefs.resolve("idle_timeout_seconds", env="OMNIVOICE_IDLE_TIMEOUT_S", default=IDLE_TIMEOUT_SECONDS)`. Keep core.config.IDLE_TIMEOUT_SECONDS as the default source.
- subprocess_backend.py: replace the env-only `SIDECAR_IDLE_TIMEOUT_S` (line 107) read with prefs.resolve("sidecar_idle_timeout_seconds", env="OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S", default=300.0). Preserve "<=0 disables reaping" semantics and the existing reaper-start guard (line 222). Resolve lazily (function call), not at import, so a test/setting change takes effect — but keep a sensible cached default for the hot reaper loop.
- Both must keep env precedence (env wins over store) — that's exactly what prefs.resolve already does.
</action>
<verify>
<automated>grep -n "_IDLE_TIMEOUT_SECONDS\|prefs.resolve\|SIDECAR_IDLE_TIMEOUT" backend/services/model_manager.py backend/services/subprocess_backend.py</automated>
<automated>uv run pytest tests/backend/services/test_subprocess_reaper.py -q 2>&amp;1 | tail -20</automated>
</verify>
<done>
- No hand-duplicated IDLE_TIMEOUT constant; both timeouts resolve via prefs with env precedence.
- Reaper "<=0 disables" + busy-skip behavior unchanged; all 10+ reaper tests still pass.
</done>
</task>
<!-- ════════════ WAVE 3 / TIER 3 — ROBUSTNESS & OBSERVABILITY ════════════ -->
<task type="auto">
<name>Task 8 (MM2-08): Subprocess sidecars self-report VRAM in pong</name>
<files>backend/services/subprocess_backend.py</files>
<action>
Sidecar VRAM is reported as 0 (system.py:192-203 / list_live_sidecars) because the parent can't measure a child's GPU memory. Have the child measure itself.
- In the sidecar worker's ping handler (the code that answers {"op":"ping"} with {"op":"pong"} — find it in the sidecar entry-point module), include `vram_mb`: measure inside the child via torch.cuda.memory_allocated() (CUDA) or torch.mps.driver_allocated_memory() (MPS, guarded), else 0. Same degrade-gracefully pattern as system.py:147-156.
- Parent: in the health-check ping/pong path (subprocess_backend.py:435-438), capture reply["vram_mb"] and stash it on the sidecar record so list_live_sidecars() (line 181) can surface it. Refresh opportunistically on each successful ping; default to last-known or 0 if never measured.
- Keep the contract that enumeration never breaks the panel.
This is CUDA/MPS-aware and degrades to 0 on CPU — honoring cross-platform parity (default behavior identical; the number is just more accurate where the API exists).
</action>
<verify>
<automated>grep -n "vram_mb" backend/services/subprocess_backend.py</automated>
<automated>uv run pytest tests/backend/services/test_subprocess_reaper.py -q 2>&amp;1 | tail -15</automated>
</verify>
<done>list_live_sidecars() exposes a vram_mb sourced from the child's own measurement; 0 only when truly CPU/unmeasured; reaper tests still green.</done>
</task>
<task type="auto">
<name>Task 9 (MM2-06, MM2-07): Bounded cooldowns + per-role weight validation</name>
<files>backend/api/routers/setup/download.py</files>
<action>
- MM2-06: _install_cooldowns (line 27) grows unbounded. On a successful install, delete the repo's cooldown entry. Add a TTL sweep: when reading/writing the dict, evict entries older than a fixed window (reuse the existing cooldown window constant; pick the larger of cooldown-window and e.g. 1h). Keep it simple — a dict + timestamps, swept on access. No new dep.
- MM2-07: _validate_snapshot_has_weights (line 55) + _MIN_WEIGHT_BYTES 5 MB (line 45) is one magic number for all roles. Make the threshold per-role/per-extension: safetensors/bin/ckpt expect the existing floor; .onnx models (kittentts, supertonic, sherpa) can be legitimately smaller — set a lower, role-aware floor so a valid small ONNX model isn't flagged as truncated. Keep the #352 truncation-catch intent (catch a 0-byte / KB-sized partial), just stop false-positiving small-but-complete models.
</action>
<verify>
<automated>grep -n "_install_cooldowns\|_MIN_WEIGHT_BYTES\|def _validate_snapshot_has_weights" backend/api/routers/setup/download.py</automated>
<automated>uv run pytest tests/ -k "download or install or model" -q 2>&amp;1 | tail -20</automated>
</verify>
<done>Cooldown dict is bounded (evict-on-success + TTL sweep); weight validation floor varies by role/extension; #352 truncation still caught.</done>
</task>
<task type="auto">
<name>Task 10 (MM2-09): Log why scan_cache_dir() fell back to disk walk</name>
<files>backend/api/routers/setup/models.py</files>
<action>
The scan_cache_dir() -> _scan_cache_on_disk() fallback (~line 268-280, helper at line 177) silently swallows the exception — this is the #117/#118 Windows WinError-448 path. Wrap the fallback so it logs at WARNING with the exception type and a one-line reason ("scan_cache_dir failed (%s); falling back to on-disk walk of %s") before walking. Do not change the fallback behavior itself — just stop it being invisible in logs. Keep it from ever raising out (the panel must still render).
</action>
<verify>
<automated>grep -n "falling back\|logger.warning\|_scan_cache_on_disk\|scan_cache_dir" backend/api/routers/setup/models.py | head</automated>
</verify>
<done>The disk-walk fallback logs a WARNING naming the exception type; behavior otherwise unchanged; never raises out.</done>
</task>
</tasks>
<verification>
Full-suite gate after each wave (run the relevant subset per wave, full set before the last PR):
1. `uv run pytest tests/test_engines.py tests/backend/services/test_model_lifecycle.py tests/backend/services/test_subprocess_reaper.py tests/test_model_load_timeout.py tests/test_model_manager_preload.py -q` — all green.
2. `uv run pytest tests/ -k "download or install or model or engine" -q` — green (Tier 3 touch points).
3. Response-shape guard: GET /model/loaded still returns {models, count}; POST /model/unload returns {unloaded, success, ...}; 400 on unknown id. (Covered by test_model_lifecycle.py.)
4. No new runtime dependency: `git diff pyproject.toml uv.lock` is empty.
5. Localization/CJK + redaction gates unaffected: `uv run pytest tests/test_no_hardcoded_cjk.py -q`.
</verification>
<success_criteria>
- Tier 1: switching the active backend releases the previous engine's VRAM (unload() called once on switch); contract overridden for OmniVoice + all subprocess engines; ASR reporting is honest. (MM2-01..03)
- Tier 2: services.model_lifecycle is the single lifecycle surface; system.py routers are thin delegations with unchanged response shapes; idle/timeout config flows through prefs.resolve with env precedence and no duplicated constants. (MM2-04..05)
- Tier 3: cooldown dict bounded; weight validation is per-role; sidecars self-report real VRAM; cache-fallback logs its reason. (MM2-06..09)
- All listed pytest selections pass; no on-disk model-state change; no new dep; cross-platform default behavior identical (VRAM numbers degrade gracefully on MPS/CPU).
</success_criteria>
<risks>
- **unload() correctness for the shared OmniVoice singleton (MM2-01/02):** OmniVoiceBackend shares model_manager's `model` global. unload() must release the shared singleton, but the idle_worker() + offload_tts_for_asr() paths also touch it. Risk: a switch during an in-flight ASR offload double-frees or races. Mitigation: take mm._model_lock around the shared release in unload(); guard on `mm.model is not None`; keep unload best-effort (try/except) so it can never wedge a switch. Add the idempotency test (Task 4).
- **Response-shape drift (MM2-04):** Moving /model/loaded + /model/unload bodies into the facade risks changing the JSON the frontend depends on (hooks.ts, flush dropdown). Mitigation: move verbatim first, assert shapes in test_model_lifecycle.py, only then layer MM2-03/08 improvements.
- **Sidecar protocol change (MM2-08):** Adding vram_mb to pong touches the parent/child wire format. Older sidecars (a long-running session mid-upgrade) won't send it. Mitigation: treat vram_mb as optional in the parent (`reply.get("vram_mb", <last-known or 0>)`); never require it; never break the existing pong==success check.
- **prefs.resolve at import time (MM2-05):** Resolving timeouts at import freezes them; the reaper loop reads SIDECAR_IDLE_TIMEOUT_S. Mitigation: resolve lazily inside the reaper tick / idle_worker tick (cheap) so a settings change takes effect, while keeping the import-time default for the start-guard.
- **Per-role weight floor (MM2-07):** Lowering the ONNX floor could let a genuinely-truncated ONNX through (#352 regression). Mitigation: keep a non-zero floor for every role (e.g. ONNX floor still >> a partial KB), key on extension, and keep the "largest file" heuristic — only the threshold becomes role-aware.
- **Scope creep into perf:** GPU-pool sizing and torch.compile are explicitly out of scope. If an executor is tempted, stop — those regress #278/#315.
</risks>
<output>
Write `.planning/quick/260613-mm2-clean-model-management-v2/260613-mm2-SUMMARY.md` when done (per wave or once at the end), documenting: which engines got real unload() overrides vs intentional no-ops (for the future CI gate), the exact response shapes preserved on the two endpoints, the per-role weight-validation thresholds chosen, and the pytest output for the verification selection. Note any decision an executor made where the plan said "use judgment."
Docs-sync check (CLAUDE.md hard rule): this is internal lifecycle cleanup with no user-facing install/Docker/versioning change, so no README/docs edit is expected. If MM2-05 surfaces the new idle-timeout settings keys in the Settings UI, add them to the relevant settings doc in the same PR.
</output>
@@ -1,32 +0,0 @@
# SUMMARY — model-management v2 cleanup (mm2)
**Date:** 2026-06-13 · **Scope:** all 3 tiers (MM2-01..09). Backend-only; no frontend, no on-disk model-state change, no new deps.
## Tier 1 — correctness
- **MM2-01 (VRAM leak on engine switch):** `get_active_tts_backend()` now caches one instance per configured backend id and calls the outgoing engine's `unload()` before switching. Added `reset_active_backend()` for shutdown/tests. The `model=` OmniVoice fast-path still returns a fresh view over the shared singleton (no double-load) but a switch *away from* another engine still releases it. `tts_backend.py`.
- **MM2-02 (per-engine unload()):** `OmniVoiceBackend.unload()` drops the local ref + the shared `model_manager.model` singleton + `free_vram()` (idempotent, preload-safe, best-effort — no async lock from the sync path). `SubprocessBackend.unload()` routes to `unload_sidecar(self.id)` (busy sidecars skipped) and is inherited by every subprocess engine.
- **MM2-03 (honest ASR row):** `/model/loaded` ASR row now reports the pipe's actual device and carries a `note: "released with the TTS model"` so the disabled unload button is explained rather than silent.
## Tier 2 — single lifecycle surface
- **MM2-04 (`services/model_lifecycle.py`):** new facade owns `list_loaded()` / `unload(id)` / `unload_all()` / `free_vram()` across in-process TTS+ASR, diarization, and sidecars. `system.py` `/model/loaded` + `/model/unload` are now thin delegations; **response shapes preserved exactly** (`{models,count}`, `{unloaded,success,...}`, 400 on unknown id) — frontend untouched.
- **MM2-05 (unified idle config):** removed the duplicated `_IDLE_TIMEOUT_SECONDS`; the in-process idle timeout and the sidecar idle timeout both resolve per-tick via `prefs.resolve(... env=...)` (env wins, settings can tune without restart). New keys: `idle_timeout_seconds` (`OMNIVOICE_IDLE_TIMEOUT_S`), `sidecar_idle_timeout_seconds` (`OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S`). `<=0` still disables sidecar reaping.
## Tier 3 — robustness & observability
- **MM2-06 (bounded cooldowns):** `_install_cooldowns` is swept (TTL 1h) on each install check and cleared on success — can no longer grow unbounded.
- **MM2-07 (per-role weight floor):** `_validate_snapshot_has_weights` uses per-extension floors (tensor formats keep 5 MB; `.onnx` floor 64 KB) **OR** the original ≥5 MB catch — strictly more lenient, so a small-but-complete ONNX model is no longer false-flagged as truncated while a 0/KB partial is still rejected (#352 intact).
- **MM2-08 (sidecar VRAM self-report):** the parent can't see a child's VRAM, so the GPU sidecar (`engines/indextts`) now reports `vram_mb` in its `pong` (CUDA/MPS-aware, 0 on CPU); the parent stashes the last-known figure and `list_live_sidecars()` surfaces it. CPU/absent sidecars honestly report 0.
- **MM2-09 (cache-fallback logging):** the `is_cached` `scan_cache_dir → on-disk` fallback now logs at WARNING with the exception type (was DEBUG/invisible) — the #117/#118 WinError-448 path is triagable from logs.
## Out of scope (as planned, not done)
GPU-pool per-engine sizing (`_GPU_VRAM_PER_JOB_GB`) and torch.compile tuning — perf, not cleanup; risk regressing #278/#315.
## Verification
- New `tests/test_mm2_lifecycle.py` — 15 tests (reuse/switch-unload/reset/idempotent-unload, facade list/unload/unknown/sidecars shapes + honest ASR, env-wins idle config, cooldown sweep, per-role weight floor ×3).
- Affected existing: `test_engines.py`, `test_subprocess_reaper.py`, `test_model_load_timeout.py`, `test_model_manager_preload.py` — green (no regressions).
- **Full suite: 1379 passed, 0 failed.** Live: facade endpoints return preserved shapes; engine switch calls the previous engine's `unload()` exactly once.
## Test placement note
MM2 tests live at top-level `tests/` (not `tests/backend/`) on purpose: adding files under `tests/backend/` reorders collection and can expose a pre-existing `sys.modules`-isolation leak in other backend fixtures (the issue debugged in the FDL PR). Top-level placement keeps `tests/backend/` order identical.
## Docs-sync
The new idle-timeout settings keys are internal env/prefs knobs with no UI surface, so no README/docs change is required by the docs-sync rule. If a future Settings panel exposes them, document there.
-933
View File
@@ -6,939 +6,6 @@ The format is loosely based on [Keep a Changelog](https://keepachangelog.com/).
Versions track the desktop app (`tauri.conf.json` + `frontend/src-tauri/Cargo.toml`).
The bundled TTS model package (`pyproject.toml`) is versioned independently.
## [0.3.9] — 2026-07-04
The dictation release — and a deep reliability pass driven by live-testing the entire app. **Dictation is rebuilt end-to-end**: instant feedback with a live waveform, words that commit about half a second after you stop speaking, clean punctuation, and text insertion that never lies about success. **LLM providers get one-click connection testing** with real diagnostics and model discovery, in all 21 languages. The app now **always opens maximized**, bottom buttons **can't hide under the footer** at small window sizes, and a wave of "out of memory / can't reach the backend / stuck at preparing" reports were traced to their real causes and fixed — including the silent VRAM crash on 8 GB cards, dead-IPC startup hangs after a Windows BSOD, and misleading error labels. Intel-Mac support status is now stated honestly, Confucius4-TTS is validated end-to-end, and Parakeet — roughly 20× faster than the default transcriber on CPU — is unlocked for every machine.
### Added
- **Sponsor OmniVoice.** A new `SPONSORS.md` (tiers, logo guidelines, how to sponsor), a README Sponsors section, and an in-app Sponsors area (Support page + a footer link) let people back the project — with a one-click "Become a sponsor" that opens a structured GitHub issue form, no account or token needed. Sponsorship is a thank-you, not a paywall: OmniVoice stays free and AGPL-3.0. (#923, #924)
- **OpenAPI reference in Settings.** A new Settings → OpenAPI page embeds an interactive Scalar reference for OmniVoice's local backend API, with a one-click footer button. Fully local — Scalar is bundled, not loaded from a CDN, and phones home to nothing. (#928)
- **Engine Self-test.** The Engines matrix gains a "Self-test" button for in-process TTS engines that runs a tiny real synthesis and reports duration + sample rate — proving an engine actually makes audio, not just imports — plus a copy-paste `export OMNIVOICE_*_DIR=…` setup line for opt-in engines right in the "Why unavailable?" panel. (#930)
- **One canonical HuggingFace-token store + incomplete-download visibility.** The Model Store token field now saves to and is cleared from the same encrypted store as Settings → Credentials (no more two-stores split), and a truncated model cache shows an "incomplete · N MB" state with one-click Repair and Delete instead of masquerading as "not installed". (#927)
- **Launchpad, reimagined as a deck of cards.** The seven feature cards now fan out with animated waveform faces in each card's accent color; hover or keyboard-focus any card and it comes forward while the rest tuck underneath, and the layout stays usable down to the minimum window size. (#904)
- **See exactly what OmniVoice keeps on disk — and get warned before space runs out.** Settings → Storage shows real usage for the model cache (with your largest models), app data, engine environments and temp files, plus a free-space gauge and low-disk / near-full-volume warnings with one-click paths to open folders or reclaim space. (#906)
- **A "What's new" changelog reader in Settings → Updates.** The available update's real release notes now render in-app, alongside an offline changelog viewer and a one-time "what's new" note after each update. (#909)
- **Route each AI feature to its own LLM — or switch it off.** A new Settings → LLM Skills panel lists every LLM-powered capability (Cinematic/Autofit translation, slot fitting, glossary auto-extract, direction parsing, dictation cleanup) with a per-skill toggle and provider picker, so sensitive work can stay on a local model while heavier jobs use a remote one. Disabled skills fall back to the exact non-LLM behavior. (#912)
- **A small thank-you moment, done right.** After a successful export, dub, audiobook, or batch run, OmniVoice may — rarely — show a friendly, dismissible note by the footer heart about supporting development: never more than once a session, at most every 7 days, never for brand-new users, with a permanent "don't ask again". The logs bar also gained an icon and the footer icons now share one size. (#898)
- **Dictation, rebuilt.** The dictation pill now shows a live waveform the moment the mic opens, streams words as you speak with real download/loading progress on first use, and finishes what you say in about half a second of silence instead of two-and-a-half. Transcripts come out properly capitalized and punctuated. Text insertion is now honest and safe: your clipboard is preserved and restored, failures show what to do (including a one-click jump to macOS Accessibility settings when permission is missing) instead of a false "Pasted", and Esc cancels cleanly at any point. The dictation model also pre-warms in the background after launch, so the first press of the hotkey no longer sits on a cold model load.
- **LLM Providers: one-click connection testing with real diagnostics.** The Test button in Settings → LLM Providers now measures round-trip latency and turns failures into plain-language guidance — bad key (401/403), wrong model or URL (404), rate-limited (429), or unreachable server — instead of a raw exception dump. A new "Fetch models" button lists every model your key can access so you pick from real names instead of guessing. The whole panel is now translated into all 21 languages, provider error messages never echo your API key, and the settings API gained full test coverage.
### Changed
- **A "Get in touch" page that actually guides you.** The Contact page is now clearly-labelled cards (report a bug, request a feature, get community help, support the project, report a security issue) with a sentence each on when to use them, instead of a flat link list. (#925)
- **Release titles are version-first.** GitHub's release-list sidebar truncates the title, so "OmniVoice Studio v0.3.8" hid the version; releases are now named "vX.Y.Z — OmniVoice Studio" so the version is always visible. (#922)
- **Launchpad feature cards now fill the window.** The seven cards (Voice Clone, Voice Design, Video Dubbing, Stories, Audiobook, Voice Gallery, Transcripts) span the full content width on a maximized display instead of a fixed ~780px fan, and reflow responsively (7→3→1 columns) down to the 900×600 minimum — driven by the shell's own width, keeping the animated card faces, hover/keyboard-focus raise, and reduced-motion fallback. (#915)
- **LLM Providers settings, de-confused.** The old inline "LLM endpoint" box in Translation is gone — LLM Providers is now the one place that owns it. Fields pinned by an environment variable are shown disabled with an explainer instead of silently reverting, the make-active button explains when a provider is env-pinned, and the Cloudflare Account ID is remembered and editable. (#907)
- **Intel Macs: honestly unsupported for the local backend.** PyTorch no longer ships Intel-Mac builds, so the backend cannot run there; instead of a cryptic dependency error, Intel users now get a clear explanation up front (with the remote-backend option), and the README/docs say so plainly. (#889, #891)
- **The app now always opens maximized (not fullscreen).** Window size and position are no longer carried over from the previous session — one manual resize used to make every later launch reopen at that smaller size, overriding the intended maximized default. Same behavior on macOS (zoomed window, not a fullscreen Space), Windows, and Linux.
### Fixed
- **Sherpa-ONNX "model not set" now reads as a setup problem, not out-of-memory.** Selecting the sherpa-onnx engine without `OMNIVOICE_SHERPA_MODEL` configured used to fail with a misleading "ran out of memory — press Flush" 500; it now names the exact variable, points at Settings → Engines, and the engine is marked unavailable-with-a-reason in the picker (with a copy-paste setup line) instead of selectable-but-broken. Generalized so any env-gated engine surfaces actionable setup guidance. (#919)
- **Cinematic & Autofit now actually run on every translation engine.** Picking Cinematic or Autofit on the default Argos engine (or NLLB) used to silently fall back to Fast with a success toast; it now runs the full LLM refine + fit pass, the Autofit fit pass is bounded by the same wall-clock budget as Cinematic, and provider errors are scrubbed of keys/user-ids. (#910)
- **Dictation no longer freezes on a slow or dead LLM.** Transcript refinement is now hard-bounded (default 4s): a placeholder key or unreachable endpoint falls back to clean unrefined text instead of stalling the paste ~51 seconds. The dictation model is genuinely pre-warmed and reused across sessions, REST transcription is polished like live dictation, and Settings flags a configured-but-failing LLM. (#911)
- **Model installs fail loudly, not silently.** Failed downloads keep their mirror-aware reason on the row with Retry/Dismiss instead of vanishing after a moment; installs check free disk space up front before overrunning it; in-progress installs get a Cancel button; and the HF-mirror setting only asks for a restart when it actually changed. (#908)
- **Engines settings, sharper and honest.** The Supertonic license "Accept" button works again (it was inert since it shipped), the engine matrix refreshes the instant you pick an engine, picking a GPU engine that lands on CPU now warns you with the reason, CPU-only engines stop being mislabelled "CPU fallback", and an in-process "Test engine" pass reads as a dependency check instead of a fake "0 ms" latency. (#905)
- **Updates can no longer cost you data.** Before any database migration runs on first launch of a new version, the database is snapshotted next to itself (newest three kept), and a failed migration stops with the backup path named instead of silently running on a half-upgraded database; the environment self-heal now verifies it's actually broken before rebuilding. (#909)
- **CUDA transcription now works on packaged NVIDIA installs — the cuDNN 8 compat libraries install automatically at launch.** The install step only existed in the dev-loop `scripts/setup.py`, which isn't bundled into the packaged app, so real installs never got the libs and WhisperX / faster-whisper failed with `Could not locate cudnn_ops_infer64_8.dll`. The Rust bootstrap now side-loads them on CUDA machines; CPU/AMD/ROCm boxes skip the download and cache the result so their launches stay instant. (#827, #869)
- **`scripts/setup.py` no longer fails with `No module named pip` when installing the cuDNN 8 libs in the dev loop.** `uv venv` doesn't seed pip into the venv, so `python -m pip install` always broke; the script now uses `uv pip install --python` instead. (#869)
- **Generation timeouts now give device-honest advice.** A CPU-only machine is no longer told the GPU is "VRAM-starved" or to "set the engine to CPU" — CPU hosts get compute-bound guidance (shorter text, the CPU-tuned GGUF/Supertonic-3 engines, the OMNIVOICE_GENERATE_TIMEOUT_S knob) while GPU hosts keep the VRAM-contention explanation. (#896)
- **Model-download failures now name the mirror that failed.** When a Hugging Face mirror is configured and unreachable, every affected surface (generate, dub, Model Store installs) names the mirror and points at the exact setting instead of leaking a raw network error; auto-repair failures now say *why* the repair failed. (#874, #890)
- **No more infinite "preparing" after an unclean shutdown.** If Windows corrupts the WebView cache (e.g. after a BSOD), the splash detects the dead IPC channel, proceeds via a direct backend health check, and — if truly stuck — offers a one-click "Repair and restart". (#879, #892)
- **"Out of memory" is no longer the default excuse.** A failed model download mid-generation was mislabeled as OOM with useless "flush VRAM" advice; network failures are now classified honestly, only real OOM signatures get the OOM treatment, and first-use engine downloads retry once with a fresh connection. (#880, #893)
- **Hung transcriptions recover the same way everywhere.** Chunked dub transcription now shares the same guarded-timeout + GPU-pool reset as the rest of the app, and repeated timeouts recommend the crash-isolated ASR engine — now properly selectable in Settings. (#730, #895)
- **A raw `[Errno 22]` transcribe error now tells you what to fix.** When the OS rejects the temporary WAV write during dub transcription (a missing, read-only, or full temp directory, or antivirus interference), the stream used to dead-end as *"Transcription produced no segments. [Errno 22] Invalid argument"* with no next step; it now classifies the EINVAL and appends an actionable temp-dir/disk/AV hint — the same treatment the ffmpeg and compute-type failure classes already get. (#763)
- **Buttons can no longer hide under the logs footer on small windows.** The bottom status/logs bar was a fixed overlay that pages had to compensate for with padding — any view that missed it (voice-card grids in Gallery and Community, bottom action rows) clipped under the bar at small window sizes, a class previously patched one page at a time (#476, #504). The footer is now a real row of the app shell, so content physically ends at its top edge at every window size, collapsed or expanded — guarded by a new layout test plus a 900×600 Playwright check at the app's minimum window size.
- **Confucius4-TTS is now validated end-to-end — and actually loads.** The opt-in engine's first live run (Apple Silicon, CPU) caught three scaffold-era faults: the sidecar could never import `confuciustts` (upstream ships no packaging, so the documented `pip install -e` fails — the sidecar and bootstrap probe now put the clone on `sys.path`, like upstream's own example), the assumed 24 kHz sample rate was wrong (confirmed **22 050 Hz**, now regression-tested), and the docs demanded an Amphion/MaskGCT install that doesn't exist (all weights auto-download from HuggingFace). CPU is ~17× realtime, so CUDA stays the recommended path; `gpu_compat` now advertises `("cuda", "cpu")`. (#590)
- **Parakeet TDT transcription now works without an NVIDIA GPU.** The `nemo-parakeet` ASR engine (parakeet-tdt-0.6b-v3, 25 languages, word timestamps) was hard-gated behind CUDA — but a live measurement on an Apple Silicon M2 shows it transcribing at ~10× realtime *on CPU*, roughly 20× faster than the default whisper-large-v3 on the same machine at equal accuracy. The false GPU gate is removed, so Mac and CPU-only users can now pick the dramatically faster engine in Settings → Engines.
- **8 GB GPUs: voice-clone/dub transcription no longer kills the backend.** On cards where the TTS model already held most of the VRAM (e.g. RTX 4060 Ti 8 GB), loading whisper `large-v3` in float16 for a reference-clip or dub transcription died as a *native* CUDA out-of-memory abort — the whole backend process vanished with no error logged, and the app showed "Can't reach the local OmniVoice backend." A new VRAM preflight re-checks free GPU memory right before the ASR load and steps down float16 → int8 → CPU instead of attempting a load that can't fit (opt-out: `OMNIVOICE_ASR_VRAM_PREFLIGHT=0`). (#723)
### CI
- **A migration can no longer silence the app's logs.** Alembic's startup config was disabling every existing logger process-wide (a latent bug the new pre-migration backup logging exposed); fixed, and the migration-safety tests are now immune to full-suite ordering. (#909, #917)
- **Deterministically green tests + real install proof.** Tests can no longer read the developer's real `.env` or app data (the order-dependent flake class, #878, #894), and a new cross-platform install-test workflow builds all four installers and proves a real first run — model download plus verified synthesis — on macOS, Windows, and Linux runners.
## [0.3.8] — 2026-07-01
A stability-focused release that makes first-run and Windows "just work," ships
**live, faster-than-real-time local dictation** and a **user pronunciation
dictionary**, and gives **Settings a full redesign**. It clears the wave of
**"Can't reach the local backend"** reports at the source — the 8 GB-card OOM
crash, the slow-load future-scheduling break, a Windows-only WhisperX load
failure, an ASR engine that couldn't load CTranslate2 on newer Linux/WSL, and
both transcription **and generation** stalls that *looked* like a dead backend
(a wedged GPU job now resets the worker pool and returns an actionable timeout)
are all fixed or now fail with a clear, actionable message. **macOS gets native file drag-and-drop back**
(including macOS 26 Tahoe). Downloads are faster out of the box (parallel
segmented transfer on by default) and the Hugging Face token that speeds them up
is front-and-center on setup. Plus multi-voice story casting, faster long-form
previews on Windows, and a friendlier, more honest batch of error messages
across dub, generate, and design (a corrupt-binary failure no longer poses as
"out of memory," a bad model id self-heals, and a stale dub job resets cleanly).
### Added
- **"Autofit" translation quality — the dub keeps the video's timing.** A new
quality alongside Fast and Cinematic: the LLM rewrites each translated line so
its target-language reading time fits *within* the segment's slot (a strict
"never overrun" bound, per-language pronunciation-speed aware), so long
translations no longer force the audio into a stressed >1.3× time-stretch.
Cinematic still applies its reflect/adapt polish; Autofit adds the hard
fit-to-slot pass on top. Needs an LLM (below); falls back to Fast with a clear
notice if none is set. (#838)
- **A new LLM Providers settings page — bring your own high-quality LLM.**
Settings → System → **LLM Providers** configures the LLM that powers Cinematic
and Autofit translation. One page for **16 providers** — OpenAI, OpenRouter,
Groq, Cerebras, Google AI (Gemini), Mistral, Cohere, NVIDIA, GitHub Models,
Cloudflare, Hugging Face, SambaNova, SiliconFlow, plus **local Ollama / LM
Studio** (fully offline, no key) and a **Custom** OpenAI-compatible endpoint.
Paste a key, pick a model, **Test** the connection in one click, and "use for
translation" to make it active. Keys are stored **encrypted** (the same
at-rest protection as the HF token) and never leave the machine unless you
choose a cloud provider; env vars still override for power users. The dub
translate menu now routes you straight here when you pick a high-quality
style without an LLM, instead of dead-ending on a toast. (#838)
- **A dedicated Network pane.** The HTTP/SOCKS proxy and FFmpeg-path controls
(previously buried in General → Advanced) are promoted to their own category.
- **Factory reset in Storage.** A confirm-dialog-guarded action that clears the
locally-saved UI preferences and reloads — without touching your voices,
projects, or generated audio on disk.
- **Proactive, highlighted "Install" affordance for translation engines.** When
you pick a Dub translation engine whose optional package isn't installed yet
(e.g. Google / DeepL via `deep_translator`), the Engine selector now surfaces a
bright accent **Install** button *before* you hit Translate — no more
discovering the missing package only via a translate-time 400. On a from-source
install it one-click installs into the backend's own interpreter; on a
read-only **packaged build** it opens a popover with the exact `uv pip install …`
command (copy-to-clipboard), a one-click **Switch to Argos (bundled, offline)**
escape hatch, and a docs link. The install command is single-sourced in the
backend registry, so the button and the 400 error can never disagree. New guide:
`docs/dubbing/translation-engines.md`.
- **A user pronunciation dictionary that actually changes the audio.** Settings →
General → Pronunciation lets you teach the engine how to say tricky words —
each entry replaces a term with a respelling (`GIF``jiff`) right before
synthesis, so it works on **every** engine, not just one. Scope an entry
Global or to a single language (a German rule never fires on an English
render), with longest-match-first, word-boundary-aware, case-insensitive
substitution. For one-offs, write `[[word|respelling]]` inline in your text —
it overrides the dictionary for that occurrence and never persists. A built-in
Test field previews the substitution with no model call. Pure text transform,
identical on macOS/Windows/Linux; plain text stays byte-identical, existing
data upgrades cleanly via an additive migration. (Expressive-TTS Spec 01)
- **Live, faster-than-real-time dictation via a new sherpa-onnx ASR engine.**
Pick one of seven small ONNX speech-to-text models (Parakeet TDT v3/v2,
streaming Zipformer EN/ZH/bilingual, streaming Paraformer, multilingual
Whisper Tiny) for dictation, and watch text appear *as you speak*. Streaming
models emit partials frame-by-frame and commit a sentence on natural silence;
offline models surface live partials too by re-decoding a growing buffer.
Runs CPU-only and identically on macOS, Windows, and Linux — no GPU, no cloud,
no extra setup beyond a ~75180 MB one-time model download. Parakeet TDT v3 is
the recommended default; existing Whisper/MLX/NeMo dictation engines are
untouched and still the fallback.
- **New "Voice" settings panel for live dictation.** Settings → Capture now
leads with a Voice card: an Enable Voice Dictation toggle (showing your real
registered shortcut), a Toggle/Hold mode switch, and a Speech Model dropdown
that lists all seven models with offline/streaming + recommended badges, size,
one-line descriptions, the installed checkmark, and inline download/delete —
reusing the model-store download progress. Picking an uninstalled model starts
its download and switches to it once ready. **Toggle vs Hold** is wired for
both the desktop global hotkey and the in-app Ctrl/Cmd+Shift+Space fallback, so
the behaviour is identical on macOS, Windows, and Linux. While you speak, the
dictation pill shows the transcript building **live**, and words type straight
into the focused field *as you speak* — self-correcting with backspaces as the
streaming recognizer refines, with clipboard-paste as an automatic fallback.
- **Tagged scripts auto-cast into a multi-voice podcast/audiobook.** Paste a
`[Alice] … [Bob] …` script into Stories and hit Auto-cast: it now recognizes
the `[Name]` tag format (alongside the existing `NAME:` screenplay and quoted
prose), builds the cast, and assigns a voice per character automatically.
Editing one line only re-synthesizes that line on export (the chapter cache
is content-addressed), and inline markers like `[pause]` / `[voice:…]` are
never mistaken for speakers. (#487)
- **A dedicated Contact page.** Discord, email, GitHub issues, and the project
website (palash.dev) as clean one-tap rows, reachable from the footer — so
reaching the maker is never more than a click away.
- **Live download speed, remaining size, and ETA on first-run setup.** The
Models & Engines step now shows `38% · 5.2 MB/s · 1.2 GB left · ~3m` while a
model downloads, instead of a bare "downloading…". (#657)
- **Turn off auto-play of the preview after a render.** New Settings →
Appearance toggle, "Auto-play preview" (on by default) — switch it off so a
finished clip doesn't start playing on its own, ideal when batch-generating
segments. (#666)
- **App version in the status bar, one click from updates.** A `v<version>`
badge sits by the network icon in the bottom bar; clicking it opens Settings →
Updates, and it grows a pulsing dot the moment a new version is ready to
install. (#671)
### Changed
- **Settings is now a sidebar-nav hub instead of an 11-tab strip.** The whole
page was rebuilt from scratch as a grouped left-rail navigator (with a
search/filter box) plus a scrollable content pane — the macOS System Settings /
VS Code layout. Settings are organized into four groups and sixteen
categories: **General** (Appearance · General), **Voice & Engines** (Engines ·
Models · Dictation · Pronunciation · Translation), **System** (Performance &
Device · Storage · Network · Sharing & Remote · Credentials), and **App**
(Updates · Privacy & Reporting · Logs · About). Every existing control keeps
its behavior and store/API bindings — this is a reorganization, not a rewrite.
Typing in the search box filters the category list and jumps to the first
match, and the rail collapses to a dropdown navigator below 760px so the full
IA stays reachable on a narrow window. Categories whose changes need a backend
restart (Models, Performance & Device, Sharing & Remote) carry a "restart
required" badge.
- **The Settings pages got a full redesign — cleaner, denser, responsive.** A
shared design system replaces the old patchwork: a left icon nav-rail,
sentence-case section titles (no more debug-log uppercase), exactly one muted
description per row, unified toggles/inputs, full-width content with proper
padding, and horizontal font/theme pickers. Premium and compact instead of
sparse and cluttered, and it adapts cleanly to window width. (#686, #690, #696)
- **Adding a Hugging Face token on first-run is now a one-line input right by
Continue.** Was a bulky card buried at the bottom of the model list; it's now a
compact "paste a token, Save" bar pinned next to the "Waiting for required
models…" button, so you can add it (for faster, authenticated downloads)
without scrolling. (#687, #688)
- **First-run setup is calmer and surfaces the best models for your machine.**
Dimmed and tightened the setup descriptions (less wordy, more compact). The
"Models & engines" step now shows the **platform-tuned** optional models up-front
with a green "recommended" tag and their catalog note — e.g. MLX Whisper on
Apple Silicon, CUDA-tuned variants on NVIDIA — instead of burying every optional
model behind the fold (the universal long tail still folds).
- **Donations now go through Ko-fi or PayPal (GitHub Sponsors removed).** GitHub
Sponsors isn't available, so the Support page no longer routes there: pick an
amount (now $10 / $20 / $50) and then choose Ko-fi or PayPal — PayPal carries
the amount straight into checkout. `.github/FUNDING.yml` and the README badges
were updated to match.
- **Simplified the Commercial License page.** Trimmed the six-tile benefit grid
and FAQ down to the three things that actually drive the decision (you own the
output, no per-minute cost, direct support) plus one clear "request a quote"
contact — less wall-of-text, faster to act on.
- **Model downloads are faster out of the box.** The built-in multi-connection
(segmented) downloader — parallel byte-ranges with live speed/ETA — is now on
by default, so the legacy-LFS path is no longer single-stream and slow. It
falls back to the normal download on any error, so it can never compromise a
correct install (`OMNIVOICE_SEGMENTED_DOWNLOAD=0` to disable). (#669)
- **The Hugging Face token is now front-and-center on first-run.** Was a
collapsed "advanced" fold almost nobody opened; it's now a prominent card right
above Continue, framed around what it actually buys you — authenticated, faster,
more reliable downloads (higher rate limits, fewer stalls) — with a one-click
"get a free token" link. (#657, #669)
### Fixed
- **Bug reports redact more secrets and every Windows username casing.** The
opt-in bug-report scrubber now catches more credential shapes (JWT/Bearer,
Google, Slack, AWS keys, and `?token=`/`?api_key=` URL secrets), redacts
Windows home paths regardless of `Users`/`users` casing, and stops a superstring
username (`/Users/john` vs `/Users/johnny`) from leaking a fragment. The
prefilled-issue URL is now bounded by its *encoded* length so a large report
can't silently truncate. Nothing new leaves the machine — this only makes the
existing local-first, user-reviewed report stricter. (#856)
- **A hung TTS generate can no longer brick the backend ("Can't reach the local
backend").** A GPU job that wedges on some Windows + CUDA setups occupies its
worker forever — Python can't cancel the thread — so on the 12 worker pools we
ship, one stuck job starved every other request and the next action surfaced as
the misleading "Can't reach the local backend" even though the process was
alive. ASR/dub/model-load already bounded and reset the pool on hang (#730); but
**every generate path** — Studio synthesis, the streaming path, batch, the dub
per-segment + preview render, archetype previews, and the OpenAI-compatible
`/v1/audio/speech` API — was still an unguarded GPU dispatch, and the residual
reports all failed on `generate:start (audio)`. Every one is now bounded by the
same wall-clock guard (`OMNIVOICE_GENERATE_TIMEOUT_S`, default 300s) that
abandons the wedged worker and rebuilds the pool, so capacity is restored
automatically and you get an actionable timeout instead of a dead backend.
Closes the whole class of GPU-job-hang reports (#851#850, #802, #755, #723,
#721, and the 0.3.7 cohort, all tracked in #730).
- **An unsupported GPU now falls back to CPU instead of 500-ing every generate.**
When the installed PyTorch build has no kernels for your GPU's compute
capability — a too-old card (Pascal / GTX 10-series) or a too-new one
(Blackwell RTX 50-series on pre-cu128 wheels) — CUDA failed at launch with the
cryptic `CUDA error: no kernel image is available for execution`. The backend
now detects that up front and runs on CPU (slower, but it works), and any raw
occurrence is reported as "your GPU isn't supported — switch to CPU or install a
matching PyTorch," not a Flush-the-memory dead end. Force the GPU anyway with
`OMNIVOICE_FORCE_CUDA=1`. (#756)
- **The "TRANSLATION FAILED" banner now dismisses and clears itself.** The Dub
translation-error banner used to be sticky — it survived a successful re-try and
never went away. It now has a close (×), auto-clears on the next corrective
action (re-translating, changing the engine, or installing the package), and
self-clears after a short timeout — fixing the whole class of translate/pipeline
banners that outlived the state that caused them.
- **Dubbing a video URL no longer fails with "ffmpeg is not installed."** yt-dlp
downloads video and audio as separate streams and muxes them with ffmpeg, but
it only looked on PATH — so on Windows (where OmniVoice's ffmpeg is a bundled
sidecar / `imageio-ffmpeg` binary off PATH) the merge aborted before the dub
could start. yt-dlp is now pointed at the same ffmpeg OmniVoice resolves. (#712)
- **A synth that succeeded no longer 500s because of a history-logging hiccup.**
If the local database somehow missed schema init, recording the clip to
generation history failed with *"no such table: generation_history"* and
surfaced as a 500 — even though the audio had already been generated and saved.
The write now self-heals the schema and retries, and a history-logging failure
never fails the generation: you get your audio regardless. (#710)
- **Long-video dubs no longer spike RAM during assembly.** Dub generation used
to hold every segment's audio in memory until the whole track was mixed, so a
50-video batch or a single feature-length dub could exhaust RAM and crash. Each
segment now streams to disk as it's rendered and the final track is assembled
from those files via a 30s-chunk memmap writer, keeping memory flat regardless
of video length. Per-segment download WAVs and the final track stay correctly
watermarked (marked once at synthesis, no double-mark), and zero/negative-length
segments no longer crash the run. (#639)
- **A corrupt or wrong-architecture native component no longer masquerades as
"out of memory."** A synth failure caused by a bad `.dll`/`.pyd`/`.exe` on
Windows (`[WinError 193] %1 is not a valid Win32 application` — e.g. torch,
ffmpeg, or an engine binary) was labelled *"ran out of memory — try Flush,"*
sending users down the wrong path. It now says the component is corrupt or
built for the wrong architecture and to reinstall/repair it. (#705)
- **A "[Errno 32] Broken pipe" mid-generation no longer poses as "out of
memory."** When the desktop app that launched the backend closes or relaunches,
the backend's output pipe breaks and a synth can fail with `[Errno 32] Broken
pipe`. That was labelled *"ran out of memory — try Flush,"* which never helps;
it now tells you the backend lost its pipe and to restart the app. (#715)
- **Settings content no longer sprawls or spills out of view.** The content
column capped at 1280px, so on wide windows rows stretched edge-to-edge with a
big empty gap between each label and its control ("too spread out"), and a few
panels (API keys, the shared button rows, appearance scale) used rigid pixel
widths that pushed controls past the card's padding on narrow content. Now the
content sits at a readable measure (a single `--settings-measure` token), the
shared button/badge rows wrap instead of overflowing, rigid widths can shrink,
and rows decide whether to sit side-by-side or stack based on their **actual**
width (a container query) — not the viewport, which the 168px nav rail skews.
Everything stays inside its padding, edge to edge, on every width. (#696)
- **File drag-and-drop works on macOS again.** The app's drop zones use HTML5
file drops, but Tauri intercepts OS drag-and-drop by default (`dragDropEnabled`)
and swallowed the files before the webview saw them — most visibly on macOS
WKWebView, and fully broken on macOS 26 (Tahoe), where dropping a file did
nothing. Disabled the interception so the webview handles native HTML5 drops
on every platform. (#700)
- **A misconfigured `OMNIVOICE_MODEL` no longer bricks model load with a 500.**
A stale or leaked TTS *engine id* (e.g. `omnivoice`) reaching the model loader
used to fail every launch with *"omnivoice is not a local folder and is not a
valid model identifier."* It now self-heals — only a real HF repo id
(`org/repo`) or an explicit local path is honored; anything else falls back to
the default with a logged warning. Every consumer of the setting routes through
the same resolver, so a bad value also can't silently disable model warm-up,
mislabel the Settings checkpoint, or get baked into an exported persona bundle.
(#693)
- **ASR no longer crashes the dub/transcribe preflight when CTranslate2's native
library can't load.** On hardened kernels / newer glibc (e.g. WSL2) the
CTranslate2 `.so` is rejected with *"cannot enable executable stack"* — an
OSError the WhisperX/faster-whisper checks didn't catch, so it took down the
whole preflight. They now report the engine as unavailable and auto-detect
falls back to PyTorch-Whisper instead of dead-ending. (#692)
- **A wedged transcription can no longer take the whole backend offline ("Can't
reach the local backend").** On some Windows + CUDA setups a whisperx/CTranslate2
transcribe hangs hard and never returns. Because ASR shares a small (12 worker)
GPU pool with TTS, one stuck worker starved every other request — so the next
thing you did (often a TTS *generate*) failed with "can't reach backend" even
though the process was alive. Two fixes: every transcribe path — whole-file
(dub whole-file, batch, live dictation) **and** the chunked dub stream — is now
wall-clock **bounded** like the dub QC / dictation / OpenAI paths already were;
and on timeout the poisoned GPU worker is **abandoned and the pool rebuilt**, so
capacity is restored without restarting the app. You still get an actionable
message (Flush VRAM / pick a smaller ASR model) for the durable fix. (#730)
- **The stale-dub-session recovery now also covers the first upload/ingest, not
just retry/import.** A dubbing job that vanished server-side during the initial
transcribe flow showed the scary *"Job not found … report a bug"* toast; it
now resets gracefully and invites a fresh upload, like the other paths. (#695)
- **In-app preview of finished audiobooks/stories now plays on Windows.**
The preview decoded the entire render into one in-memory PCM buffer via Web
Audio `decodeAudioData`, which fails on long-form `.m4b`/AAC under WebView2
(`EncodingError: Unable to decode audio data`), and the blob-URL fallback can't
play in a Tauri `<audio>` element — so nothing played. The fallback now uploads
to the preview endpoint (ffmpeg-extracts a streamable WAV) and plays the HTTP
URL, the same path video previews use. Short TTS previews are unchanged. (#653)
- **First-run setup splash no longer shows a raw `bootstrap.lines` key in English.**
The log-line counter string was present in 4 locales but missing from the `en`
reference, so English (and 16 other locales falling back to it) rendered the
literal key instead of "{{count}} lines". Added it to `en`. Also removed 160
dead `gallery.cat_*` keys (renamed to `archetypes.use_*` long ago) orphaned
across 20 non-English locales, clearing the i18n orphan-key advisory.
- **Backend no longer hangs on startup (unreachable, no error) on Apple-Silicon Macs.**
The MCP session manager could hang on its anyio task group during lifespan
startup (observed on M1, #632); because that start was awaited before the server
began serving, "Application startup complete" never fired and the whole backend
was unreachable. The MCP start is now timeout-bounded (`OMNIVOICE_MCP_START_TIMEOUT_S`,
default 30s) — a hang becomes a logged warning and the backend serves normally
without MCP, instead of wedging. (#632)
- **Dubbing a URL no longer fails with `[Errno 22] Invalid argument` on Windows.**
yt-dlp stamps the downloaded file's modified-time with the video's upload
date; an out-of-range/invalid timestamp makes the `os.utime` call raise
`[Errno 22]` and aborts the whole URL ingest. OmniVoice downloads to a throwaway
file and never uses its mtime, so it now skips the stamp entirely
(`updatetime=False`). (#642)
- **Dubbing a YouTube link that 403s now retries with a different player
client.** Some videos serve their formats signature-protected to the default
player client, so the media download fails with `HTTP Error 403: Forbidden`
even though extraction worked — and a plain retry keeps 403ing. The URL
download now escalates the YouTube player client (tv → android → web_safari)
on a 403, which commonly bypasses it, before surfacing the actionable error.
(#625)
- **A synth glitch that produced unreadable audio is now caught instead of a
misleading "out of memory".** A numerical glitch in the model (seen on Apple
Silicon/MPS) could leave NaN/∞ samples, which wrote a WAV that then failed
decoding with an opaque `ffmpeg returned error code: 183 / Invalid data` — and
the generic error handler labelled it "ran out of memory". Non-finite samples
are now sanitized to silence before any encode (so the WAV is always
decodable), and a genuine decode failure is reported as "unreadable audio —
Flush and regenerate", not OOM. (#629)
- **A silent startup hang now leaves a diagnostic instead of nothing.** On some
setups the backend could load all model weights and then hang forever before
"Application startup complete" — no error, no crash, an unusable app (reported
as a Mac M1 hang after `Loading weights: 527/527`, #632). A startup watchdog
now dumps every thread's stack to the error log if startup stalls past a
window (default 5 min, `OMNIVOICE_STARTUP_WATCHDOG_S` to tune, `0` to disable),
so the deadlock is captured rather than invisible. It's disarmed the instant
startup finishes, so a normal (even slow-first-download) boot never trips it.
(#632)
- **First-run demo voice is back.** The bundled demo clip
(`backend/assets/samples/demo_voice.wav`) was a build artifact that never got
committed, so it shipped absent — onboarding logged "Demo audio not found" and
seeded nothing, leaving a brand-new install with an empty Launchpad and no
`/demo_audio` route. The clip is now committed (it's already un-ignored and
bundled via the Tauri `backend` resource), so first-run seeds the demo voice
on every platform; onboarding still degrades gracefully (with a regenerate
hint) if it's ever absent. (#621)
- **Multi-speaker dubbing: two speakers' turns merged onto one line are now
split apart.** Segmentation groups words into sentences *before* diarization
runs, so a back-and-forth exchange could land in a single segment; the speaker
pass then only *relabelled* that segment with its majority speaker, losing the
turn boundary (the second half of #486; the per-speaker voice auto-assign was
fixed earlier in #490). A new post-diarization pass re-splits any segment whose
words span more than one speaker at the word-level boundary, assigning each
piece its own speaker. Single-speaker segments pass through **byte-for-byte
unchanged**, so single-speaker dubs and their timing never move, and a lone
mis-attributed word (diarization noise) is smoothed rather than causing a
spurious split. (#486)
- **Designed voices saved with a bad style no longer render wrong or crash
generation.** A designed voice could persist an `instruct` the engine
validator rejects — either the literal `"[object Object]"` from an old build,
or freeform prose typed into the style field — which made every generation or
dub that used the voice fail with `Unsupported instruct items found in …`
(surfacing to users as a 400/500 and, when it tore down mid-render, "Can't
reach the local backend"). The previous fix only *blanked* `"[object Object]"`,
which silently dropped the design — so an Indonesian **female** voice came out
**male**. Now the stored instruct is sanitized down to valid tags at every
seam (save, edit, and when a profile drives Generate or Dub), and when the
stored value is unusable the tags are **rebuilt from the design's saved
category picks (`vd_states`)** so the intended gender/age/pitch/accent survive.
A migration (0007) heals existing poisoned profiles in place — no reinstall,
no manual fix. (#550 #571 #594 #596)
- **"Transcribe stream dropped … Likely ASR backend failed to load" now shows
the *real* reason.** When transcription failed to load its ASR model (the
reported case was WhisperX on Windows — typically a faster-whisper /
CTranslate2-cuDNN mismatch, a missing model download, or the torch-2.6
weights-only VAD regression), the UI dead-ended on a generic "stream dropped"
message with no actionable cause. Two root causes: (1) WhisperX loads lazily
*inside* transcription, so the load failure was buried in per-chunk errors and
retried on every chunk; the transcribe pre-flight now eagerly loads the ASR
model (new `ASRBackend.ensure_loaded()`), surfacing the genuine cause once, up
front, as a structured error. (2) Pre-flight and audio-load errors closed the
SSE stream with a bare `error` and no terminal `done`, so the browser's native
EventSource connection-drop could race and win against the structured error —
discarding the real cause and falling back to the generic message; every
terminal error now emits `done`, and the frontend latches the structured cause
so a connection drop can't overwrite it. Net: WhisperX load failures are
diagnosable instead of a silent dead-end. Fail-before/pass-after regression
test included. (#578)
- **Dubbing: the PLAY button on the dubbed-video preview did nothing.** Same
autoplay-policy trap that #510 fixed for the standalone audio player, but the
dub editor's timeline player was missed. WaveSurfer builds its `AudioContext`
at mount — before any user gesture — so on Windows WebView2 (and Linux
Firefox/Chrome, Android Chrome) it stays `"suspended"`; `playPause()` then
resolves with no sound and the preview just sits there. Every playback entry
point in the dub timeline (the toolbar Play button and the per-segment "play
this slot") now resumes the context via the shared `unlockAudio()` on the
click before starting playback, and swallowed play() rejections are logged
instead of hidden. A source-contract regression test pins the invariant so a
future refactor can't quietly reintroduce a silent play path. macOS is
unaffected (its context was never blocked). (#595)
- **Voice design: the script text field couldn't be expanded.** The Script
textarea was a `flex: 1` item inside a flex column, so flex-grow recomputed
its height on every reflow and snapped the user's drag back — `resize:
vertical` is silently ignored on a flex-grown item in Chromium/WebView2. The
field now owns its own height (starts taller, and the corner grip grows it
reliably on every platform). (#595)
- **An interrupted model download now self-repairs instead of dead-ending.**
When the OmniVoice TTS cache was missing weight shards (the usual aftermath of
an interrupted first download), the next synthesize failed with a 500 and a
"delete the model and install it again" instruction — a manual dead-end. The
backend now detects the truncated-cache error on load, re-fetches just the
missing files via `snapshot_download` (already-present blobs are skipped, so a
near-complete cache repairs in seconds and a healthy cache is never touched),
and retries the load automatically. Offline mode (`HF_HUB_OFFLINE`) is
respected — repair never makes a network call the user opted out of — and if
the re-fetch still can't fix it, the actionable delete-and-reinstall message
is preserved as the fallback. (#581) The repair now also **retries** the
re-fetch (3 attempts, resuming each time) so a single transient blip — the very
thing that interrupts a download in the first place — doesn't bounce you back
to a manual reinstall; tune with `OMNIVOICE_MODEL_REPAIR_RETRIES`. And if a
resume-repair still won't load — the signature of a *corrupt* file that kept
its size, which a resume trusts and never re-fetches — it now **force
re-downloads** the model files once before giving up, so even a bit-rotted
cache self-heals without a manual reinstall. (#739)
- **Dubbing a YouTube URL no longer dies on a transient "Broken pipe."**
Pasting a video link could fail outright with `download: Unable to download
video: [Errno 32] Broken pipe` — a broken pipe raised while the write side of
a pipe closes mid-stream (a killed ffmpeg merge child, a CDN reset during
muxing). yt-dlp's own per-fragment retries don't cover that case, so a single
transient blip aborted the whole ingest. The URL download now retries up to
twice on broken-pipe / network-drop failures, wiping the partial download
between attempts, and only surfaces the (already-actionable) "connection
dropped — just retry" hint after the retries are exhausted. Unsupported links
still fail fast with their own hint — no wasted retries. (#579, #598)
- **`No module named 'omnivoice'` on installs whose venv lost its editable
record.** An interrupted or offline `uv sync` (common during an in-place
upgrade) could install all dependencies yet never lay the editable install of
the project's own `omnivoice` package — or an antivirus quarantine could
remove it. The venv still started uvicorn, so the bootstrap's health gate
passed it through, and the app only failed at the first generate/dub with
`No module named 'omnivoice'`. The bootstrap now also verifies `omnivoice` is
importable (via a cheap `find_spec`, no torch load) and forces a repair
`uv sync` that re-lays the editable install when it isn't; the backend also
resolves `omnivoice` from its bundled source tree at runtime as a safety net.
No reinstall needed — relaunch and it self-repairs. (#564)
- **"cannot schedule new futures after shutdown" no longer breaks generate/dub
after a slow first load.** When a model load timed out, the backend reset its
GPU worker pool to recover — but several request handlers had captured the old
pool object at import time and kept submitting to it, so every subsequent
generate, dub, transcribe, or translate failed with `cannot schedule new
futures after shutdown` (a 500, or "Can't reach the local backend" when it
took the worker down). The GPU pool is now a single self-healing handle whose
worker pool is rebuilt on demand, so a reset can never strand an in-flight or
later request. No settings change; the recovery is automatic. (#589 #599)
- **Transcription / dubbing works on Windows again.** WhisperX failed to load on
Windows because speechbrain's guard that suppresses stray optional-integration
imports used a POSIX-only path check, so a `k2_fsa` import error aborted the
whole transcription. Fixed cross-platform — covers the entire class of optional
integrations, not just k2. (#630 #611 #647)
- **A slow transcription no longer looks like a dead backend.** Whole-file
transcribe paths (dub QC, dictation, OpenAI-compat) ran unbounded, so a
VRAM-starved `large-v3` could spin for minutes and hold a GPU worker — surfacing
as "Can't reach the local backend". They're now time-bounded and return a clear,
actionable 504 (free VRAM / pick a smaller ASR model / use CPU) instead of
hanging. New troubleshooting section documents it. (#656)
- **Windows preview playback fixed.** The audiobook/clone preview's streaming
fallback fetched `localhost`, which on Windows resolves to IPv6 and missed the
IPv4-only backend — so previews failed with "decode error" / "no supported
sources". The preview API now targets `127.0.0.1` (matching the main client),
and the expected decode→stream fallback is logged calmly instead of as a scary
error. (#653 #659)
- **A stale dub session resets cleanly instead of erroring.** Reopening the Dub
tab after the backend restarted tried to resume a job that no longer existed and
surfaced "Job not found" as a bug-report error. It now quietly clears the dead
session and invites a fresh upload. (#660)
- **A bad voice-style instruct is a clear 400, not a scary 500.** Typing free-form
prose (or a non-English description) into the style/instruct field returned a
500 telling you to Flush for memory you never ran out of; it now returns a clean
400 that lists the valid style tags. The Voice Clone UI also drops unrecognized
style text locally and generates anyway. (#664 #612)
- **The ⊕ Insert token popover stays on screen.** On Voice Clone it could grow
tall enough to clip off the top of the window; it's now a compact, scrollable
box anchored above the button. (#672)
- **First-run no longer hangs on Apple Silicon.** The MCP session-manager startup
is now timeout-bounded so a slow/stuck mount can't wedge the whole backend boot
on M1. (#632)
### CI
- **Feature-coverage test system.** A backend route-inventory test diffs all 213
HTTP/WebSocket endpoints against a committed snapshot (plus a critical-endpoint
guard and a route-count floor), and a frontend feature-coverage test asserts
every app mode is wired to a page and every feature has its i18n namespace — so
an endpoint or page silently disappearing now fails CI on every PR.
- **`bun desktop` no longer kills its own dev backend.** The dev launcher runs the
API and the Tauri app side-by-side, but the app's backend manager would "take
ownership" of port 3900 and kill the API the moment it booted (before it was
healthy), tearing the whole session down. The dev app now sets
`TAURI_SKIP_BACKEND` so it attaches to the running API instead of fighting it —
production launch is unaffected. (#745)
## [0.3.7] — 2026-06-20
A stabilization release that clears the wave of issues reported on the 0.3.6
line — across voice design, dubbing, transcription, install, and the Linux/web
UI — and lands two more opt-in cloning engines. The throughline is **non-English
correctness and cross-platform playback**: cloned and designed voices now hold
their language end-to-end, and audio plays inline in Linux/Android browsers,
not just macOS. It also carries the v0.3.6 startup-crash fixes, so anyone still
hitting "Can't reach the local backend" on v0.3.5/v0.3.6 only needs to update.
### Added
- **Two opt-in heavyweight TTS engines: MOSS-TTS-v1.5 (8B) and dots.tts (2B).**
Both are zero-shot voice-cloning engines, each running in its own isolated
subprocess venv (they pin a `transformers` version that conflicts with the
parent's `>=5.3` — MOSS `==5.0`, dots.tts `==4.57`) via the same dedicated-venv
pattern as IndexTTS-2, so they can't disturb the default install or its
lockfile. Point `OMNIVOICE_MOSS_TTS_V15_DIR` / `OMNIVOICE_DOTS_TTS_DIR` at a
local clone to enable. CUDA/CPU only — neither claims Apple-Silicon MPS, and
dots.tts is gated off on Windows (upstream is Linux/macOS only). See
[docs/engines/moss-tts-v15.md](docs/engines/moss-tts-v15.md) and
[docs/engines/dots-tts.md](docs/engines/dots-tts.md). (#498)
### Fixed
- **Non-English voices drifted to English / the wrong language.** Three
independent root causes, all in the language path: (1) a voice profile's
stored language was never read back into generation, so a German archetype
that *previewed* in German *generated* in English (the preview passed the
language; the user's Generate call didn't); (2) the audiobook/longform synth
hardcoded `language=None`, letting the engine re-autodetect per chunk so a
non-English clone could flip language mid-render on short/ambiguous lines; and
(3) the duration estimator weighted Unicode combining marks at zero, so
decomposed (NFD) diacritic text — common for Vietnamese — under-allocated
frames and came out rushed. The profile/request language is now threaded
through both the single-shot and longform paths (request wins, profile fills
the gap), and text is NFC-normalized before duration estimation. Each fix has
a fail-before/pass-after regression test. (#533, #505, #502)
- **Audio playback on Linux Firefox/Chrome and Android Chrome.** Two separate
root causes both masquerade as "the play button doesn't work" on non-macOS
browsers — and both are invisible when developing on macOS, which is why they
shipped. (1) The backend served `.wav` / `.flac` with Python's default
`audio/x-wav` / `audio/x-flac` (vendor-experimental, never IANA-registered);
macOS CoreAudio MIME-sniffs leniently and plays anyway, but Linux FFmpeg and
Android ExoPlayer strictly honor the declared type and prompt to download.
Fixed by registering the canonical `audio/wav` / `audio/flac` types before
any `StaticFiles` mount. (2) WaveSurfer's `AudioContext` is constructed at
component-mount time — i.e. before any user gesture — so on Linux FF/Chrome
and Android Chrome it stays `suspended`, `decodeAudioData` hangs, the
`ready` event never fires, and the play button never enables. macOS
Safari/Chrome auto-resume on first interaction. Fixed by patching
`window.AudioContext` to track every instance and resuming them on the first
`pointerdown` / `keydown` / `touchstart`, plus resuming inline on the play
click itself. The MIME fix has a backend regression test; the unlock path
has a Vitest unit test covering idempotency, post-unlock contexts, and
error isolation. (#510)
- **Voice Studio "Save design as profile" poisoned the profile with
"[object Object]" and then 400'd every generation** ("Unsupported instruct
items found in [object Object]"). The save passed the instruct *builder
object* to the form instead of its string. Fixed at the source + defended with
a coercion helper; the engine now tolerates the sentinel, and a migration
heals already-saved profiles. (#550, #545, #542, #537, #530, #525)
- **Profile / persona / consent endpoints 500'd with `no such column:
consent_audio_path`** (and the same class for `kind`/`vd_states`/…) after an
in-place upgrade. The alembic migration existed but couldn't always apply
(stamped at a removed revision, or alembic not importable) and the failure was
swallowed. The runtime schema now self-heals — it ADDs any missing additive
column from the canonical schema on startup. (#552, #547)
- **Stories: the global reading-speed slider was ignored by preview and stem
export.** The #415 global speed only flowed through the full longform export;
per-segment preview and stem export still resolved a hardcoded `track.speed ||
1.0`, so audio played at 1.0× even with the global set to e.g. 0.70×. A shared
`effectiveSpeed(track, global)` helper (per-line override → global → engine
default) now drives all three generation paths. (#508)
- **Generate / Settings / Clone buttons were missing / unpressable on Linux.**
The UI-scale fix round-trips correctly on Chromium, but older WebKitGTK treats
`zoom` as a layout no-op, leaving a ~23% black band that pushed the bottom CTAs
off-screen. The shell now probes the engine and fills the window when `zoom`
doesn't lay out. (#523, #524)
- **Settings tabs with little content rendered as a stunted box in a black
void** (reported on Appearance). The page is now a flex column with a
min-height floor — short tabs fill the panel, tall tabs grow and scroll
exactly as before. The Appearance panel's previously hardcoded English
strings ("UI scale", "Color theme", "Font") were also routed through i18n,
per the localization rule. (#507)
- **The engine "Install" button 500'd with "No virtual environment found."**
`uv pip install` now targets the running interpreter (`--python
sys.executable`) instead of relying on a venv it couldn't auto-discover.
(#529, #527)
- **Transcription failed with "no segments" on GPUs without efficient float16.**
Both CTranslate2 ASR backends now fall back float16 → int8 instead of crashing
at model load; a transcribe stream can no longer close without a terminal
error event; and an incomplete `transformers` install reports an actionable
message instead of "Could not import module 'AutoFeatureExtractor'".
(#551, #549, #516)
- **Audiobook import 500'd** with `'AudiobookPlan' object has no attribute
'chapter_count'` for every format (.txt/.md/.epub/.pdf). (#543)
- **Windows: generated audio auto-played in a separate, un-closeable black
window.** Renders now play in-app through the shared playback manager. (#532)
- **Cryptic video-download errors** now carry actionable hints: an unsupported
link shape ("paste a direct video page, not a share/feed link") vs a transient
network drop ("just retry — the partial download was cleaned up"). (#554, #536)
- **A relocated, copied, or restored backend venv ("No module named
'encodings'") now self-heals** (rebuilds once) instead of failing on every
launch.
- **The donate goal bar showed fabricated progress** ($137.50 / $200, 23
sponsors). It now reflects the real figures ($10 / $200, 1 sponsor) in both the
runtime JSON and the TypeScript fallback. (#513)
- The **"Can't reach the local backend" startup-crash wave** (pkg_resources
#248, `scalar_fastapi` #307, exit-106 broken venv) was fixed in v0.3.6 — this
release carries those fixes, so updating from v0.3.5/older resolves them.
### Changed
- **Version is now single-sourced from `frontend/package.json`.** Five
hand-maintained literals drifting is exactly what shipped a 0.3.6 build that
called itself 0.3.5. `package.json` is canonical (vite already injects it as
`__APP_VERSION__`), `tauri.conf.json` reads its bundle version from it
(`"version": "../package.json"`), and the remaining toolchain-required mirrors
(Cargo.toml, pyproject.toml, the frozen-backend fallback) are CI-guarded to
stay in lockstep. (#503)
- **Updater: the Preview channel actually tracks `main` again.** It was stuck at
`0.3.5-41` because its only build trigger was a manual dispatch; a nightly
rebuild now enforces "preview = main" (no-opping on days `main` didn't move).
Two latent hazards are closed: the `preview` release is re-asserted as a
prerelease every run (a non-prerelease preview could hijack the Stable
channel's "Latest"), and its manifest can no longer silently drop the
Intel-Mac (darwin-x86_64) target. (#500)
### Internal
- **The frozen desktop backend reported `0.3.5` regardless of its real version.**
In a synced env, `core.version.APP_VERSION` resolves from package metadata
(correct, so CI stayed green), but the PyInstaller-frozen build has no
`.dist-info`, hit `PackageNotFoundError`, and fell back to a hardcoded literal.
The spec now bundles `omnivoice` metadata so the primary path works frozen too,
and the resolution chain is metadata → pyproject → named fallback. This also
fixes **About → Version rendering blank** in the web/Pinokio build (no Tauri,
backend idle), which now falls back to the build-time version. (#501)
## [0.3.6] — 2026-06-16
A large release (168 commits since v0.3.5). The headline is the **Longform
suite** — produce full audiobooks and multi-voice stories from text, EPUB, or
PDF — alongside a real **engine-routing** layer that tells you up front when an
engine will fall back to CPU instead of finding out mid-synth. Dubbing,
first-run, and install reliability all get a pass too.
### Added
- **Longform: Stories + Audiobook editors.** Two new tabs turn long text into
finished audio. **Audiobook** takes a script (or imports plain text / EPUB /
PDF), auto-splits it into chapters, and renders a chaptered `.m4b` with
metadata, cover art, and per-chapter preview/resume. **Stories** is a
multi-voice editor — assign a different voice per line, preview, and export
the whole thing through the same server-side renderer. Both share one render
core (loudness, metadata, cover art) and one live SSE progress stream, and
you can convert a project between Story and Audiobook in place.
(#402, #403, #404, #408, #409, #411, #412, #413, #426, #435, #436, #447)
- **Longform: PDF & EPUB ingest.** "Import" on the Audiobook tab accepts EPUB
and PDF (not just plain text) and auto-chapters the result, so an existing
ebook becomes an audiobook without manual copy-paste. (#412, #459)
- **Longform: two-pass loudnorm mastering.** Audiobook/Story exports now run a
measure-then-normalize loudnorm pass for accurate ACX/podcast loudness
targets. A slow or broken measure pass degrades gracefully to single-pass
rather than aborting the render. (#449, #455)
- **Longform: crash-resume.** An interrupted render is resumable without
re-submitting the original input — the compiled plan is persisted to the job
dir and finished chapters are reused, so a crash mid-book doesn't cost you the
whole render. (#470)
- **Longform: pronunciation control + SSML-lite prosody.** A per-render
pronunciation lexicon (word respelling) plus an in-app pronunciation editor
and markup reference, and inline prosody markers — `[slow]` / `[fast]` /
`[emphasis]` / `[spell]` — for fine-grained delivery. (#419, #421, #422)
- **Stories: global reading-speed control.** A toolbar slider (0.52.0×) sets
one speed for every line that doesn't have its own per-line override; the
per-line slider still wins. Persisted as a UI preference. (#415, #416)
- **Unified LongformProject store.** Audiobook metadata, scripts, and prefs
persist in a single project store (with a `v4→v5` migration), and finished
books/stories now show up alongside other work in **Projects**. (#417, #443,
#444)
- **Portable personas (`.ovsvoice`).** Export any voice as a self-contained,
fully-local persona bundle — identity, optional reference clip, consent
attestation, SPDX license, and a watermarked preview — and import it back into
another OmniVoice install. A privacy toggle ships a **preview-only** bundle so
no raw recording of your voice has to travel. Verified-own-voice status can't
be forged by hand-editing a bundle (real recording + consent text + attestation
required). Legacy `.omnivoice` files still import. See
[docs/persona-format.md](docs/persona-format.md). (#29)
- **Engine routing — no more silent CPU fallback.** A host device probe and
routing resolver now decide where each engine actually runs, and the verdict
is surfaced before you hit Synthesize: the **Settings → Engines** picker shows
a per-engine compatibility matrix, and **preflight** / **diagnose** report the
active engine's GPU verdict (accelerated / caveat / CPU-fallback /
unavailable). At synth time every TTS entry point (`/generate`,
`/v1/audio/speech`) enforces the same routing — an engine that can't use this
host's GPU returns an explicit error or an `X-OmniVoice-Routing` header instead
of silently dropping to CPU or dying mid-synth. (#21)
- **Diagnostics suite.** New self-check tooling for when something's wrong: a
`/system/diagnose` report (and matching backend `--diagnose`), a persistent
**error journal** surfaced in Settings, and a scrubbed **diagnostic bundle**
(home dirs stripped to `~/`, no tokens/keys) you can attach to a bug report.
Paired with structured GitHub **Issue Forms** (bug / install / feature) for
cleaner reports. (#433, #456)
- **Dubbing: multi-speaker per-speaker voice assignment.** When diarization
detects multiple speakers, each segment is now bound to its speaker's cloned
voice automatically instead of landing on "Default" and needing manual fixes;
per-segment reference clips are still preferred for quality where present. Also
adds an optional speaker-count hint for diarization. (#275, #486, #490)
- **Dubbing: Smart Fit timing + second-pass QC.** A Smart Fit timing strategy
(planner, fingerprints, per-segment video retime + drift absorption + fitted
subtitles) plus a second-pass ASR QC that flags lines whose dub drifts from the
target timing — wired into the dub editor UI. Includes a timeline segment
editor (drag, snap-to-onset, keyboard a11y), speech-onset alignment, regional
dialect targeting, and per-segment clone references. (#280, #347, #350, #369,
#370, #458)
- **Dubbing: dedicated Dub home.** A projects/history landing for dubbing with
project rename. (#435)
- **Voice Console workspace.** Clone and Design are consolidated into one Voice
workspace with right-side panels, a shared waveform player, an identity recipe
line / Active-voice card, and a free-text "describe your voice" field that maps
natural language to design parameters. (#317, #374, #376, #378, #395, #396,
#397)
- **Unified first-run setup.** Nothing installs until you confirm a plan: pick an
install mode (installed / portable), a storage location, and (on restricted
networks) custom PyPI/HF/python-build-standalone mirrors — with a
minimum-free-space gate before anything downloads. Followed by a guided
studio-console wizard with platform-aware hints, resume reassurance, and
download ETAs. (#286, #295, #297, #298)
- **Dictation: local-LLM refinement.** Opt-in local-LLM cleanup of final
transcripts (collapsing Whisper hallucination loops), available on both live
dictation and the REST `/transcribe` path; plus opt-in NLMS acoustic echo
cancellation for dictating over playback. Configure a remote LLM endpoint
(Ollama / vLLM / LM Studio) in Settings. (#356, #357, #363, #399, #400, #457)
- **Unlimited-length TTS + streaming.** Sentence-boundary chunking with
crossfade removes the per-generation length cap, and a new sentence-by-sentence
`/ws/tts` streams audio as it's produced. An inline `[pause Nms]` marker
inserts measured silence in generated speech. (#276, #357, #358)
- **MCP server v1.** OmniVoice mounts an MCP server on `/mcp` (with a stdio shim
and per-agent voice binding) so it can act as a local TTS/STT provider for
agentic pipelines. (#368)
- **Remote-backend access.** Point the desktop UI at a remote backend URL with a
bearer key (Tailscale-documented), and an opt-in Hugging Face token field in
the setup flow. (#303, #364)
- **"Fund Claude Max" support experience.** The donate page gets a real goal bar
with a "Join N supporters" social-proof line and suggested amounts, plus Pip
the mascot and a non-blocking "postcard" toast that appears only *after* a
success (a finished dub, a saved clone, a longform export) — never on errors,
setup, or first run — with escalating cooldowns and a one-click "don't ask
again". (#494)
### Fixed
- **Transcription/dubbing failed when ffmpeg wasn't on `PATH`** (notably on
Windows). WhisperX now decodes audio through OmniVoice's own validated ffmpeg
binary instead of a bare `PATH` lookup, so ASR works without a system ffmpeg
install. (#479)
- **Translation defaulted the source language to English.** Dubbing/translation
now guesses the source language from the text instead of assuming `en`,
fixing wrong-direction translations. (#478)
- **Cinematic / LLM dubbing features failed out of the box** because `openai`
wasn't bundled. The client is now a runtime dependency, so those paths work on
a fresh install. (#484)
- **`pkg_resources missing` install dead-end (#248).** The auto-repair ran
`uv pip install setuptools`, which `uv` treated as a no-op when setuptools
*metadata* was present but its files had been removed (commonly by Windows
Defender quarantine or a partial extract). Both repair sites now use
`--reinstall` to force re-extraction, and the error/hint text suggests the
working command plus an antivirus-exclusion note. (#248)
- **A stuck backend trapped users on a buttonless splash (#474).** The bootstrap
splash now has a per-stage stall watchdog: if a non-terminal stage sits past
its budget (20 min for dep install, 120 s otherwise), it flips to the failed
state with actionable hints, the live log, and Retry / Clean-&-Retry — instead
of polling forever with no way out. (#474)
- **Changing the model-download location in Settings had no effect (#480).** The
desktop launcher injected a stale models dir that overrode the per-user value,
so new downloads kept going to the old folder and "Effective location" stayed
wrong. The per-user env file now wins, so the in-app Settings path is
authoritative. (#480)
- **Backend crashed on app upgrade with a stale venv (#307).** Dependencies are
now synced on upgrade, and a structurally broken venv self-heals instead of
exiting `106`. `scalar_fastapi` is now optional so its absence can't break
startup. (#307, #314)
- **`/generate` ignored the selected TTS engine (#312)** and GGUF speech-control
parameters weren't forwarded — both now honored. (#306, #312)
- **TTS generation failed on some GPUs.** `torch.compile` failures now fall back
to eager execution so generation never hard-fails on unsupported GPUs, and
cudagraph-compiled inference is pinned to one dedicated thread to avoid
crashes. (#278, #315)
- **Re-dub ignored transcript edits (#281).** Fingerprints are canonicalized, the
preview cache is busted, and the mux is atomic, so editing the transcript and
re-dubbing actually reflects your changes. Translated subtitles now burn in
correctly and subtitle save no longer throws a JSON error. (#281, #309)
- **macOS: app wouldn't open without using Terminal.** Builds are now ad-hoc
signed (with signing/notarization verification), so the app launches normally.
(#290)
- **macOS dictation auto-paste stole focus**; it now writes the clipboard
natively without grabbing focus, and microphone-permission handling adds OS
usage descriptions, a WebView grant handler, and an actionable denied-state UI.
(#287, #323)
- **Clone-reference transcription was broken** (it used a removed transformers
pipeline); it now routes through the ASR registry. A crash-isolated
faster-whisper subprocess backend keeps an ASR crash from taking down the app.
(#308, #393)
- **Realtime status probe hit a gated route.** It now probes the auth-exempt
`/health` instead of the gated `/model/status`, and the UI polls the backend
over HTTP before opening the WebSocket to avoid startup `ECONNREFUSED`. (#439,
#450)
- **Non-executable or unreachable engine binaries showed cryptic errors** — these
now produce actionable messages. (#437, #438, #454, #466)
- **Design-profile save was coupled to a TTS render (#476)**, so saving a profile
needlessly triggered synthesis; the two are now decoupled. (#476)
- **UI scale / black bands.** The app shell now scales via `transform: scale` and
always fills the viewport, fixing the WebKitGTK black-band issue on Linux and
cramped/black layouts at narrow widths — a permanent fix across platforms.
(#445, #452)
- **Clone popover/CTA clipping and a non-resizable textarea** are fixed, the
WaveformPlayer no longer pauses itself on play or ignores clicks, and several
layout/history-display issues (phantom sidebar gap, title clamping, flicker)
are cleaned up. (#379, #384, #398, #481)
- **Windows: `desktop-prod` now runs from cmd/PowerShell** via a cross-platform
launcher, `tqdm` is disabled on non-TTY to avoid an `OSError`, and ffmpeg
validation guards against `WinError 193`. (#282, #305, #377)
- **MLX import hardened** against PyInstaller dylib failures, with a proper
platform gate so it's only loaded where it works. (#390)
### Changed
- **Restricted-network support.** A Hugging Face mirror (`HF_ENDPOINT`) setting,
custom PyPI / HF / python-build-standalone mirrors in first-run setup, and
region presets help installs complete behind restrictive networks. (#286, #391)
- **Engine memory management.** Subprocess-engine sidecars now unload on demand
and idle-reap to free VRAM. (#401, #406)
- **Faster, more accurate model downloads** via a Xet fast path with accurate
progress reporting, plus a model-management cleanup pass. (#424, #428)
- **Voice profiles unified** under one model with a `kind` discriminator and
stored design params, and consent-locked profiles (`verified_own_voice` +
spoken-consent flow). (#354, #376)
- **Updater** preview channel now offers the newest build across channels, and
preview versions carry an MSI-legal numeric pre-release stamp. (#293, #326)
- **Performance.** Voice-clone prompt embeddings are cached, and dub retime
batches seek to their window instead of decoding from frame 0. (#387, #427)
### License
- **Relicensed from FSL-1.1-ALv2 to AGPL-3.0 (open-core).** The project is now
under the GNU Affero General Public License v3, with a paid commercial license
retained for proprietary/closed-source use without AGPL obligations. The
bundled `omnivoice/` TTS model package stays Apache-2.0 upstream
(AGPL-compatible). Manifests declare `AGPL-3.0-only`; the in-app Commercial
License copy and README are updated, and the old "converts to Apache 2.0 after
two years" FAQ is removed. In-app commercial-license strings are translated
across all 20 locales. (#292)
### CI
- **macOS Intel (x86_64) build target reinstated** on `macos-15-intel`, so Intel
Mac users get installers again. (#342)
- **Docker Hub publishing.** Images now also publish to Docker Hub
(`palashdeb/omnivoice-studio`), with the Docker Hub overview maintained in-repo
and auto-synced from `main` (sync is non-fatal so it can't redden a build).
(#375, #410, #414)
- **Docs-drift guard.** A daily job compares the canonical feature inventory
against README / docs / registries to catch stale docs. (#353)
- **Security scans never cancel on `main`,** so merge trains no longer leave red
✗ on intermediate commits. (#340)
## [0.3.5] — 2026-06-03
### Fixed
+3 -15
View File
@@ -3,7 +3,7 @@
**OmniVoice Studio**
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. It's an active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord. The latest stable release is **v0.3.5**; `main` rolls ahead at **v0.3.6** (latest release + 1 patch — see the Versioning rule below).
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. Today it's a v0.2.7 active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord.
**Core Value:** **A first-run that actually works.** A user who downloads the installer (or clones the repo) should reach a working voice-cloning or dubbing output without hitting a wall — and when something does go wrong, the error or docs should tell them exactly what to do.
@@ -16,7 +16,7 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
- **Default features must work on every platform (strict rule, 2026-05-20):** A feature that ships in default mode — out-of-the-box, no user customization, no opt-in toggle — must behave identically on macOS, Windows, and Linux. Platform-specific *implementation code* is allowed for OS APIs / shells / packaging, but the user-visible *default behavior* cannot diverge. Platform-only features (e.g., a macOS-only global shortcut, a Windows-only path picker) must go behind explicit user opt-in: Settings toggle, env var, or CLI flag. When a default doesn't work on a platform, that's a P0 bug — either fix it on the missing platform or move it behind opt-in. No third option.
- **Backward-compatible project data**: Existing `omnivoice_data/` (user voices, projects, settings) must keep working without manual migration. Any DB schema change goes through alembic with a tested upgrade path.
- **Local-first guarantee preserved**: Auto bug reporting (new addition) must be **opt-in**, must submit only to GitHub Issues (no third-party telemetry endpoint), and the app must remain fully functional with reporting disabled. No required cloud calls, accounts, or API keys.
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** the v0.3.x line has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main; the owner tags a patch (`v0.3.Z`) from main whenever the current state is worth cutting. No `-rc` tags. No phased release. No `v0.4` deferrals while the v0.3.x line is open — every open issue and every open community PR gets absorbed into the v0.3.x line or explicitly declined. Users follow `main` for previews; users wanting stable stay on the latest tagged release (currently **v0.3.5**). ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** v0.3.0 has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main. Tag `v0.3.0` once when the user calls "actually useful" — a qualitative bar, not a checklist. No `v0.3.0-rc1`. No phased release. No `v0.4` deferrals while v0.3.0 is open — every open issue and every open community PR gets absorbed into the v0.3.0 line or explicitly declined. Users follow `main` for previews; users wanting stable stay on `v0.2.7`. ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
<!-- GSD:project-end -->
<!-- GSD:stack-start source:research/STACK.md -->
@@ -190,22 +190,10 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
<!-- GSD:conventions-start source:CONVENTIONS.md -->
## Conventions
**Versioning (hard rule, owner-set 2026-06-11; single-source 2026-06-16):** main is always **latest release + 1 patch**. **`frontend/package.json` is the SINGLE SOURCE OF TRUTH for the app version** — vite injects `__APP_VERSION__` from it (first-run footer + every auto bug report), and `frontend/src-tauri/tauri.conf.json` reads its bundle version from it (`"version": "../package.json"`, so the MSI/dmg/updater version can't drift from the UI). Three toolchain-required **mirrors** are kept equal to it and bumped in lockstep — `frontend/src-tauri/Cargo.toml` + `pyproject.toml` (cargo/uv need a literal) and `backend/core/version.py`'s `_FALLBACK_VERSION` (the frozen-backend last resort; at runtime the backend reads its version from package metadata via `importlib.metadata`, which `backend.spec`'s `copy_metadata('omnivoice')` makes work in the frozen build too). Never hand-edit any mirror or re-hardcode a literal in `tauri.conf.json`. Guarded by `tests/test_app_version.py` (`test_all_version_files_in_lockstep` + `test_tauri_version_derives_from_package_json`). The moment `vX.Y.Z` is released, bump `package.json` (+ the mirrors) to `X.Y.(Z+1)`. Consequences:
- Every PR and preview build identifies as the **next** version. Preview builds stamp `X.Y.(Z+1)-N` (run number), which semver-sorts **above** the last stable `X.Y.Z` — the updater ordering is natural, no comparator tricks needed.
- Releasing = tag `vX.Y.(Z+1)` from main (version files already match), then immediately bump main to `X.Y.(Z+2)`. **Owner override (2026-07-01): the post-release bump is now MANUAL — the `version-bump` job in release.yml is opt-in behind the `AUTO_VERSION_BUMP` repo variable (default off), so `main` stays at the released version until the owner explicitly asks to bump.** (Historically the bump auto-ran; re-enable that by setting `AUTO_VERSION_BUMP=true`.) When pinned, `main` == the released tag; preview-build ordering and "release + 1" only resume once a bump is requested.
- Docker: `ghcr.io/debpalash/omnivoice-studio:latest` = **main** (rolling preview); `:X.Y.Z` + `:X.Y` + `:stable` = tagged releases. `:latest` is the preview channel by design — stable users pin `:stable` or a version tag.
- Do not bump minor/major or invent RCs/codenames without the owner asking. No "defer to next version" labels — scope is absorbed or declined, never re-versioned.
**Docs-sync (hard rule, owner-set 2026-06-11):** any change that alters something these docs describe — README.md, CONTRIBUTING.md, SECURITY.md, SUPPORT.md, LICENSE, or `docs/**` (install flows, Docker tag semantics, platform support, versioning/release behavior, review process, supported versions) — must update those docs **in the same PR** as the change. If a doc impact is discovered after merge, the docs fix is the immediate next commit, not backlog. Stale docs are treated as bugs.
**Release notes / changelog (hard rule, owner-set 2026-06-16):** every tagged release gets a **high-quality, user-facing `## [X.Y.Z] — DATE` section in `CHANGELOG.md`** before (or in the same hour as) the tag — never the "Auto-generated release for vX.Y.Z…" fallback. `release.yml` extracts that section verbatim as the GitHub Release body (the `Extract CHANGELOG section for tag` step), so a missing/empty section ships a bare release. Quality bar = the existing house style: a one-paragraph headline, then `### Added` / `### Fixed` / `### Changed` / `### License` / `### CI` subsections; each entry is a **bold one-line lead** (what the user gets), 13 lines of plain-English why, and the `(#NNN)` issue/PR ref — grouped by theme, written for users, **not** raw commit dumps. This applies to **preview builds too**: preview release notes summarize what's new on `main` since the last stable, in the same style. Workflow: as features merge, keep `## [Unreleased]` current; at release time rename it to the version + date. If a release was already cut with the fallback body, the next action is to backfill `CHANGELOG.md` **and** `gh release edit <tag>` the live body — not backlog.
**Versioning (hard rule):** Everything ships on `v0.3.0`. Never mention, suggest, or label anything with a version bump — no v0.4, no RCs, no "defer to next version", no future-version labels — unless the user explicitly asks to bump. Zero unprompted version chatter.
**Localization (hard rule):** No hardcoded non-English (CJK) **user-facing text** anywhere in the codebase except the translation layer (`frontend/src/i18n/`). All UI strings go through i18n (`t('...')` keys in `locales/*.json`); native language names live in `i18n/index.ts` (`LANGUAGES`). Functional CJK is allowed and tracked via the allowlist in `tests/test_no_hardcoded_cjk.py` — text-processing regexes, model/engine vocabulary & identifiers (e.g. CosyVoice speaker IDs), localized error matching, demo/eval data, and test fixtures. CI fails on any hardcoded CJK outside the allowlist; to add legitimate functional CJK, extend `_ALLOWED_FILES` there with a justification.
**Fix quality (hard rule, owner-set 2026-06-16):** Fix issues *properly* and future-maintenance-proof — don't stop at the symptom. Root-cause fully, fix the whole **class** of the bug (not just the one reported instance), add a fail-before/pass-after regression test, and harden against recurrence (e.g. if a lockfile drift only fails in Docker, also make CI catch it). Go the extra mile where it durably pays off. Be token-efficient about it — extra **effort**, not extra **verbosity**: no padding, no redundant re-checks, the smallest correct change that is also recurrence-proof. Don't be shy to spend the effort a proper fix needs; do be shy about wasting tokens.
**Keep main green (hard rule, owner-set 2026-06-16):** A merge must **never break `main`'s CI**. Before a change lands, verify the *full* CI matrix would pass — every workflow in `.github/workflows/` **and** `deploy/Dockerfile`, not only the checks you happened to run. Dependency / lockfile / config changes must be validated against **all** consumers. Specifically: `frontend/` is a bun **workspace monorepo** — the lockfile is the repo-root `bun.lock`, and `deploy/Dockerfile` runs `bun install --frozen-lockfile`, so any `frontend/package.json` change requires regenerating root `bun.lock` and confirming `bun install --frozen-lockfile` passes (plain `bun install` in `ci.yml` silently tolerates drift, so CI-green ≠ Docker-green). Likewise re-check CodeQL/Security on code changes and the Tauri `cargo` build on Rust/dep changes.
Other conventions not yet established. Will populate as patterns emerge during development.
<!-- GSD:conventions-end -->
+2 -100
View File
@@ -42,22 +42,11 @@ This starts both services:
### Desktop App (Tauri)
```bash
bun run desktop # dev: hot-reload Tauri shell + backend
bun run desktop-prod # production: builds, bundles the backend, then launches
bun run desktop
```
Both run `uv sync` first (so the Python backend env is set up) and start the
backend automatically — you do **not** start it separately. Use the exact script
names: there is no `desktop=prod` (note the **hyphen** in `desktop-prod`).
`desktop-prod` is Windows-aware (auto-detects bash/git; see `scripts/desktop-prod.mjs`).
Requires [Rust](https://rustup.rs/) and platform-specific Tauri dependencies — see the [Tauri prerequisites](https://v2.tauri.app/start/prerequisites/).
If the app opens but stays on the **setup splash with no buttons**, the Python
backend didn't finish starting — the splash surfaces the stall reason, a log
panel, and a **Retry** button (and Settings → Logs → Backend has the full trace).
The most common from-source cause is `uv` or Python not being on your PATH.
---
## Project Structure
@@ -159,7 +148,7 @@ class MyEngineBackend(TTSBackend):
- **Components**: Functional components with hooks
- **State**: Zustand stores in `src/stores/`, organized by slice
- **CSS**: **Utilities-first + shadcn/ui, one stylesheet.** UI is built on the shadcn/ui primitives in `src/components/ui/` (wrapped by the `src/ui/` barrel, themed to the OmniVoice palette), composed with Tailwind v4 utility classes. **All styling now lives in a single file — `src/index.css`**: the `@theme` / `[data-theme]` token foundation plus the irreducible set utilities can't express (`@keyframes`, glassmorphism/`backdrop-filter`, pseudo-elements, `:has()`, unlayered cascade overrides, and styling hooks on library-generated DOM like virtualized rows / WaveSurfer). The per-component `.css` files were eliminated in the CSS→Tailwind/shadcn migration — **do not create new ones.** Reach for shadcn primitives + utilities; if a rule is genuinely irreducible, add it to `src/index.css` with a provenance comment. (The only other `.css` is the test-only visual harness. See `docs/shadcn-migration.md`.)
- **CSS**: Vanilla CSS in component-level files — no Tailwind
- **Naming**: `PascalCase` for components, `camelCase` for hooks and utils
### Rust (Tauri)
@@ -169,30 +158,6 @@ class MyEngineBackend(TTSBackend):
---
## Frontend file structure & size limits
Frontend code stays modular so an edit loads one small file, not a 1900-line
one. The rules:
- **Size caps:** **soft 300 lines**, **hard 500 lines** per `.jsx` file.
Anything over 500 lines must be split. (The cap does **not** apply to
`src/index.css` — it is the single, intentional styling foundation and the
only app stylesheet; see the CSS rule above.)
- **Pages are thin orchestrators.** A file in `frontend/src/pages/` is just
layout + routing + state wiring that composes feature components — no inline
sub-component over ~50 lines.
- **One component per file.** Co-locate `Foo.jsx` + `Foo.test.jsx` together in a
per-page feature folder under `frontend/src/components/` (e.g.
`components/settings/`, `components/dub/`). Styling is **not** co-located —
it's utilities + shadcn, with any irreducible rules in `src/index.css`.
- **Shared bits go in a `primitives/` folder** inside the feature folder
(`components/settings/primitives/` is the existing example).
- **Enforced by ESLint `max-lines`** (`max: 500`) — **warn-only for now** so it
never breaks CI, with the goal of upgrading to `error` once the backlog of
oversized files clears.
---
## Commit Messages
Write clear, concise messages. The PR title becomes the squash-merge commit.
@@ -227,69 +192,6 @@ cd frontend/src-tauri && cargo check
---
---
## What code review looks like
Every PR is reviewed by two AI reviewers before a human looks at it:
- **CodeRabbit** posts a walkthrough (with a sequence diagram, and an ASCII
before/after sketch for UI changes), inline findings, and warning-mode
pre-merge checks against the project's hard rules.
- **Greptile** reviews with the same project rubrics and learns from 👍/👎
reactions on its comments — react to train it.
Both are advisory, not gating: CI and the maintainer's approval decide. Don't
be surprised by detailed bot comments minutes after you open a PR — address
what's right, push back (in a reply) on what's wrong.
**Commit & PR conventions:** conventional-commit style with a scope
(`fix(dub): …`, `feat(setup): …`) and link the issue (`Closes #N` / `Refs #N`)
in the title or body.
## Quality gates your PR must pass
- **Cross-platform parity (hard rule):** anything that ships in default mode
must behave identically on macOS, Windows, and Linux. Platform-specific
*implementation* is fine; platform-divergent *default behavior* is a P0.
Platform-only features go behind an explicit opt-in (Settings toggle, env
var, or CLI flag).
- **i18n — all 21 locales (hard rule):** every user-facing string goes through
`t('...')` and the key must exist in **all 21** files under
`frontend/src/i18n/locales/`. Translate; don't copy English into non-English
locales. CI fails on hardcoded CJK outside the allowlist in
`tests/test_no_hardcoded_cjk.py` (extend `_ALLOWED_FILES` with a
justification for legitimate functional CJK).
- **DB schema changes** go through an alembic migration with a tested upgrade
path — existing `omnivoice_data/` must keep working with no manual steps.
- **Engine back-compat:** already-installed engines (model weights on disk)
must not require reinstall or re-download.
- **Local-first:** no new outbound calls except GitHub Issues (opt-in
reporting) and HuggingFace model downloads. Never log or persist secrets or
absolute home paths.
- **Security posture:** the backend serves loopback HTTP — treat every
query/path/form parameter as hostile. User-chosen filesystem destinations
are authorized in the Tauri process (save dialog), never via HTTP params.
## Contribution licensing
OmniVoice Studio is **AGPL-3.0-only**, and the maintainer also offers a
**commercial license** (see [LICENSE](LICENSE)). By submitting a contribution
you agree that:
1. you have the right to submit it (your own work, or compatibly licensed);
2. it is licensed to the project under **AGPL-3.0**; and
3. you grant the project maintainer a perpetual, worldwide, non-exclusive
right to also distribute your contribution under the project's commercial
license terms.
This inbound grant is what keeps the dual-license model viable. If you can't
agree to (3) for a particular contribution, say so in the PR and we'll discuss
before merging. Adding a `Signed-off-by:` line (DCO) to your commits is
appreciated but not required.
---
## Need Help?
- **Stuck on setup?** Ask in [Discord #help](https://discord.gg/bzQavDfVV9)
+55 -167
View File
@@ -4,27 +4,33 @@
<h3>The open-source ElevenLabs alternative.</h3>
<p>Real-time dictation, zero-shot voice cloning, and cinematic video dubbing — all on your desktop.<br/>Open-source, no API keys, fully local. <b>646 languages.</b></p>
<p>
<a href="https://github.com/debpalash/OmniVoice-Studio/stargazers"><img src="https://img.shields.io/github/stars/debpalash/OmniVoice-Studio?style=flat-square&color=f59e0b" alt="Stars" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/OmniVoice-Studio?style=flat-square&color=10b981" alt="Release" /></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="License" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/issues"><img src="https://img.shields.io/github/issues/debpalash/OmniVoice-Studio?style=flat-square&color=ef4444" alt="Issues" /></a>
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Discord-Join_Community-5865F2?style=flat-square&logo=discord&logoColor=white" alt="Discord" /></a>
</p>
<p>
<a href="#quickstart">Quickstart</a> ·
<a href="#features">Features</a> ·
<a href="#why-ovs">Why OVS</a> ·
<a href="#why-omnivoice-studio">Why OmniVoice Studio?</a> ·
<a href="#tts-engines">TTS Engines</a> ·
<a href="#asr-engines">ASR Engines</a> ·
<a href="#sponsors">Sponsors</a> ·
<a href="#sponsor--donate">Donate</a> ·
<a href="#contributing">Contributing</a> ·
<a href="https://discord.gg/bzQavDfVV9">Discord</a> ·
<a href="README_CN.md"><strong>简体中文</strong></a>
</p>
<p>
<a href="https://github.com/debpalash/OmniVoice-Studio/stargazers"><img src="https://img.shields.io/github/stars/debpalash/OmniVoice-Studio?style=flat-square&color=f59e0b" alt="Stars" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/OmniVoice-Studio?style=flat-square&color=10b981" alt="Release" /></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="License" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/issues"><img src="https://img.shields.io/github/issues/debpalash/OmniVoice-Studio?style=flat-square&color=ef4444" alt="Issues" /></a>
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Discord-Join_Community-5865F2?style=flat-square&logo=discord&logoColor=white" alt="Discord" /></a>
<a href="https://ko-fi.com/debpalash"><img src="https://img.shields.io/badge/Ko--fi-Support_Us-FF5E5B?style=flat-square&logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
<a href="https://paypal.me/palashCoder"><img src="https://img.shields.io/badge/PayPal-Donate-00457C?style=flat-square&logo=paypal&logoColor=white" alt="PayPal" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_aarch64.dmg"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_x64_en-US.msi"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_amd64.AppImage"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_amd64.deb"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
</p>
<p>
<sub><b>macOS:</b> first launch needs a one-time approval — right-click → <b>Open</b> (or System Settings → Privacy &amp; Security → <b>"Open Anyway"</b> on macOS 15). No Terminal needed. <a href="docs/install/macos.md#gatekeeper-quarantine">Why?</a></sub>
</p>
</div>
@@ -51,28 +57,20 @@
<table>
<tr>
<td align="center" width="25%">
<td align="center" width="33%">
<h3>🎙️ Voice Cloning</h3>
<p>3-second clip → mirror any voice.<br/><b>646 languages</b>, zero-shot.</p>
</td>
<td align="center" width="25%">
<td align="center" width="33%">
<h3>🎨 Voice Design</h3>
<p>Gender, age, accent, pitch, speed,<br/>emotion, dialect — <b>dial it in</b>.</p>
</td>
<td align="center" width="25%">
<td align="center" width="33%">
<h3>🎬 Video Dubbing</h3>
<p>YouTube URL or file → transcribe →<br/>translate → re-voice → <b>MP4</b>.</p>
</td>
<td align="center" width="25%">
<h3>📖 Audiobook Editor</h3>
<p>Import text, EPUB, or PDF. Auto-chapter,<br/>loudnorm, metadata. Export <b>.m4b</b>.</p>
</td>
</tr>
<tr>
<td align="center" valign="top">
<h3>🎭 Stories</h3>
<p>Multi-voice editor. Assign voices<br/>per-line, preview, <b>export full cast</b>.</p>
</td>
<td align="center" valign="top">
<h3>⌨️ Dictation Widget</h3>
<p><code>⌘+⇧+Space</code> from <b>any app</b>.<br/>Transcribes, auto-pastes, disappears.</p>
@@ -99,10 +97,6 @@
<h3>🛡️ AI Watermark</h3>
<p>AudioSeal (Meta). <b>Invisible</b>,<br/>survives compression.</p>
</td>
<td align="center" valign="top">
<h3>🔬 Diagnostics</h3>
<p>Self-check, error journal,<br/>scrubbed <b>diagnostic bundle</b>.</p>
</td>
</tr>
<tr>
<td align="center" valign="top">
@@ -115,29 +109,7 @@
</td>
<td align="center" valign="top">
<h3>🧩 Extensible</h3>
<p>Subclass <code>TTSbackend</code>,<br/>add any engine in <b>~50 lines</b>.</p>
</td>
<td align="center" valign="top">
<h3>🧭 Engine Routing</h3>
<p>Preflight GPU check per engine.<br/><b>No silent CPU fallback</b>.</p>
</td>
</tr>
<tr>
<td align="center" valign="top">
<h3>🎒 Portable Personas</h3>
<p>Export voices as <code>.ovsvoice</code><br/>bundles — identity + <b>watermark</b>.</p>
</td>
<td align="center" valign="top">
<h3>♾️ Unlimited TTS</h3>
<p>Sentence-chunked generation.<br/><b>No length cap</b>. Streaming via WS.</p>
</td>
<td align="center" valign="top">
<h3>🌐 Remote Backend</h3>
<p>Point UI at a remote server.<br/>Tailscale-friendly. <b>Bearer auth</b>.</p>
</td>
<td align="center" valign="top">
<h3>🧠 Dictation + LLM</h3>
<p>Local LLM cleanup of transcripts.<br/>Optional echo <b>cancellation</b>.</p>
<p>Subclass <code>TTSBackend</code>,<br/>add any engine in <b>~50 lines</b>.</p>
</td>
</tr>
</table>
@@ -146,23 +118,12 @@
## Quickstart
<div align="center">
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
<br/>
<sub><b>macOS:</b> first launch needs a one-time approval — right-click → <b>Open</b> (or System Settings → Privacy &amp; Security → <b>"Open Anyway"</b> on macOS 15). No Terminal needed. <a href="docs/install/macos.md#gatekeeper-quarantine">Why?</a></sub>
<br/>
<sub><b>Intel Macs are not supported for the local backend:</b> the app UI installs, but the Python backend cannot run because PyTorch no longer ships Intel-Mac (x86_64) wheels (<a href="https://github.com/debpalash/OmniVoice-Studio/issues/889">#889</a>) — see <a href="docs/install/macos.md">docs/install/macos.md</a>.</sub>
</div>
Per-OS install guides — pick yours and follow it end-to-end:
- **macOS** — [docs/install/macos.md](docs/install/macos.md)
- **Windows** — [docs/install/windows.md](docs/install/windows.md)
- **Linux** — [docs/install/linux.md](docs/install/linux.md)
- **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
- **Docker** — [docs/install/docker.md](docs/install/docker.md)
Stuck? Run the built-in self-check first — **Settings → About → "Run
self-check"** in the app, or `uv run python backend/main.py --diagnose` from
@@ -175,9 +136,7 @@ bundle"** packages scrubbed logs + the self-check report for bug reports.
For Hugging Face token setup, see
[docs/setup/huggingface-token.md](docs/setup/huggingface-token.md). For
diarization-specific gating, see
[docs/features/diarization.md](docs/features/diarization.md). For download
speed, the ⚡ fast-download (Xet) status, and restricted-network / mirror
options, see [docs/downloading-models.md](docs/downloading-models.md).
[docs/features/diarization.md](docs/features/diarization.md).
## Screenshots
@@ -229,7 +188,7 @@ options, see [docs/downloading-models.md](docs/downloading-models.md).
---
## Why OVS?
## Why OmniVoice Studio?
ElevenLabs charges **$5$330/mo** and processes your audio on their servers. OmniVoice Studio runs **on your hardware, with no usage limits.**
@@ -238,17 +197,12 @@ ElevenLabs charges **$5$330/mo** and processes your audio on their servers. O
| **Pricing** | $5$330/mo, per-character billing | Free & open-source (AGPL-3.0) · [Commercial license](#license) for proprietary use |
| **Voice Cloning** | ✅ 3s clip | ✅ 3s clip, zero-shot |
| **Voice Design** | ✅ Gender, age | ✅ Gender, age, accent, pitch, style, dialect |
| **Audiobook / Stories** | ❌ | ✅ Full audiobook editor + multi-voice stories (EPUB/PDF import, .m4b export) |
| **Languages** | 32 | **646** |
| **Video Dubbing** | ✅ Cloud-only | ✅ Fully local |
| **Data Privacy** | Audio sent to cloud | **Nothing leaves your machine** |
| **API Keys** | Required | Not needed |
| **GPU Support** | N/A (cloud) | CUDA · Apple Silicon · ROCm · CPU |
| **Desktop App** | ❌ | ✅ macOS · Windows · Linux |
| **TTS Engines** | 1 | **11** (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, IndexTTS 2, OmniVoice GGUF, Supertonic 3) |
| **ASR Engines** | 1 | **9** (WhisperX, Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet, Moonshine, FunASR, isolated Faster-Whisper, sherpa-onnx live dictation) |
| **MCP Server** | ❌ | ✅ Use from Claude, Cursor, any MCP client |
| **Self-check** | ❌ | ✅ Diagnostics suite, error journal, scrubbed debug bundles |
| **Customizable** | ❌ Closed | ✅ Fork it, extend it, ship it |
OmniVoice Studio gives you professional-grade AI tools without the subscription or the cloud.
@@ -266,7 +220,7 @@ OmniVoice Studio gives you professional-grade AI tools without the subscription
| | **Minimum** | **Recommended** |
|---|---|---|
| **OS** | Windows 10, macOS 12+ (Apple Silicon), Ubuntu 20.04+ | Any modern 64-bit OS |
| **OS** | Windows 10, macOS 12+, Ubuntu 20.04+ | Any modern 64-bit OS |
| **RAM** | 8 GB | 16 GB+ |
| **VRAM (GPU)** | 4 GB (auto-offloads TTS to CPU) | 8 GB+ (NVIDIA RTX 3060+) |
| **Disk** | 10 GB free (models + cache) | 20 GB+ SSD |
@@ -276,9 +230,6 @@ OmniVoice Studio gives you professional-grade AI tools without the subscription
> [!TIP]
> On GPUs with **≤8 GB VRAM**, OmniVoice automatically offloads TTS to CPU during transcription — no config needed. A dedicated GPU is not required; the entire pipeline runs on CPU (just slower).
> [!IMPORTANT]
> **macOS Intel (x86_64) is unsupported for the local backend:** the app UI installs, but the Python backend cannot run because PyTorch no longer ships Intel-Mac wheels ([#889](https://github.com/debpalash/OmniVoice-Studio/issues/889)). Intel-Mac users can still point the UI at a remote backend on another machine — see [docs/install/macos.md](docs/install/macos.md).
### TTS Engines
OmniVoice ships a multi-engine TTS backend. The default engine (OmniVoice) is always available; additional engines are opt-in and auto-detected. Switch engines in **Settings → TTS Engine** or via the `OMNIVOICE_TTS_BACKEND` env var.
@@ -287,22 +238,12 @@ OmniVoice ships a multi-engine TTS backend. The default engine (OmniVoice) is al
|--------|:---------:|:-----:|:--------:|:-----:|:---------:|:-------:|:-------:|
| **OmniVoice** (default) | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Built-in |
| **CosyVoice 3** | 9 + 18 dialects | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
| **GPT-SoVITS** | 5 | ✅ | — | ✅ CUDA/CPU | — | ✅ CUDA/CPU | MIT |
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
| **MOSS-TTS-Nano** | 20 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **KittenTTS** | English | — | — | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
| **MLX-Audio** (Kokoro, Qwen3-TTS, CSM, Dia, …) | Multi | Varies | Varies | ❌ | ✅ Native | ❌ | Varies |
| **Sherpa-ONNX** | 20+ | | | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **IndexTTS 2** ⚡ | Multi | ✅ | | ✅ CUDA | — | ✅ CUDA | Apache-2.0 |
| **OmniVoice GGUF** ⚡ | 600+ | | | ✅ CPU | ✅ CPU | ✅ CPU | Built-in |
| **Supertonic 3** ⚡ | 31 | — | — | ✅ CPU | ✅ CPU | ✅ CPU | OpenRAIL-M |
| **MOSS-TTS-v1.5** ⚡ (8B) | 31 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **dots.tts** ⚡ (2B) | 24 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ❌ | Apache-2.0 |
| **Confucius4-TTS** ⚡ | 14 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **VoxCPM2** | 30 | | | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
| **MOSS-TTS-Nano** | 20 | ✅ | | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **KittenTTS** | English | | | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
> **CUDA** = GPU-accelerated · **MPS** = Apple Silicon Metal · **CPU** = runs everywhere, slower for large models · KittenTTS and MOSS-TTS-Nano run realtime on CPU · MLX-Audio is Apple Silicon only · ⚡ = lazy-registered (installed on first use)
>
> **MOSS-TTS-v1.5** (8B, ~16 GB weights) and **dots.tts** (2B, ~9 GB weights) are heavyweight opt-in engines that run in their own isolated venv from a local clone — see [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) and [dots.tts](docs/engines/dots-tts.md). Neither claims Apple-Silicon **MPS** (upstream is CUDA/CPU only; on a Mac they run on CPU). dots.tts upstream is Linux/macOS only — no Windows path. **Confucius4-TTS** (14-language cross-lingual zero-shot cloning) is similar — its own Python 3.10 venv from a clone; CUDA recommended, CPU validated end-to-end (slow, ~17× realtime; no MPS — tested slower than CPU); see [Confucius4-TTS](docs/engines/confucius4-tts.md).
> **CUDA** = GPU-accelerated · **MPS** = Apple Silicon Metal · **CPU** = runs everywhere, slower for large models · KittenTTS and MOSS-TTS-Nano run realtime on CPU · MLX-Audio is Apple Silicon only.
### ASR Engines
@@ -312,36 +253,31 @@ OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictatio
|--------|-------------------------|:---------:|----------|
| **WhisperX** (default) | `whisperx` | ~100 | Dubbing & subtitles — word-level timing via wav2vec2 forced alignment |
| **Faster-Whisper** | `faster-whisper` | ~100 | Fast transcription on Linux / macOS / Windows (CTranslate2) |
| **Faster-Whisper (isolated)** | `faster-whisper-isolated` | ~100 | Same as Faster-Whisper but crash-isolated in a subprocess — an ASR crash won't take down the app |
| **MLX Whisper** | `mlx-whisper` | ~100 | Native Apple Silicon speed (Apple MLX / Metal) |
| **PyTorch Whisper** | `pytorch-whisper` | ~100 | CUDA / CPU fallback via 🤗 Transformers (no cuDNN 8 needed) |
| **Parakeet TDT** | `nemo-parakeet` | English + 25 EU | SOTA accuracy at ~10× realtime even on CPU, auto language detection (NVIDIA NeMo, CUDA/CPU) |
| **PyTorch Whisper** | `pytorch-whisper` | ~100 | CUDA / CPU fallback via 🤗 Transformers |
| **Parakeet TDT** | `nemo-parakeet` | English + 25 EU | SOTA English accuracy, auto language detection (NVIDIA NeMo, GPU only) |
| **Moonshine** | `moonshine` | English | Edge / low-latency, ONNX |
| **FunASR** | `funasr` | 50+ | All-in-one multilingual — built-in VAD + inline speaker diarization (SenseVoice) |
| **sherpa-onnx** (live dictation) | `sherpa-onnx-asr` | 25 EU + 90+ | Live, faster-than-real-time dictation — small streaming/offline ONNX models (Parakeet TDT v3/v2, streaming Zipformer & Paraformer, Whisper Tiny), CPU, identical on macOS / Windows / Linux. Picked per-model in **Settings → Voice**. |
> Whisper-family engines cover ~100 languages; **FunASR / SenseVoice** adds an all-in-one multilingual path with built-in voice-activity detection and inline speaker diarization. **sherpa-onnx** powers the live dictation model picker — you talk and text appears as you speak. Every engine runs on-device — no API keys, no cloud.
> **GPU without efficient float16?** On older NVIDIA GPUs (Maxwell/Pascal, GTX 16xx) or after a CTranslate2/cuDNN mismatch, the CTranslate2 ASR engines (WhisperX, Faster-Whisper) can't run `float16` and OmniVoice automatically retries on `int8` — no config needed. If transcription still fails, pin the compute type with the `ASR_COMPUTE_TYPE` env var (escape hatch): `ASR_COMPUTE_TYPE=int8` (or `float32` for CPU). Set it to `int8` and restart the backend.
> Whisper-family engines cover ~100 languages; **FunASR / SenseVoice** adds an all-in-one multilingual path with built-in voice-activity detection and inline speaker diarization. Every engine runs on-device — no API keys, no cloud.
---
## Architecture
```
┌─────────────────────────────────────────────────────────────
Frontend (React)
│ DubTab · VoiceConsole · Stories · Audiobook · Gallery
│ Dictation · BatchQueue · Diagnostics · MCP Client │
├─────────────────────────────────────────────────────────────┤
Backend (FastAPI)
│ 100+ API endpoints · SSE+WSS streaming · SQLite │
├──────────┬──────────┬──────────┬──────────┬────────────────┤
WhisperX │ Demucs │OmniVoice │ Pyannote │ Engine Routing
(+7 ASR │ Source │ (+10 │ Diariz- │ ↳ GPU preflight
│ engines) │ Sep. │ TTS) │ ation │ ↳ No silent CPU │
└──────────┴──────────┴──────────┴──────────┴────────────────┘
CUDA / MPS / ROCm / CPU (auto-detected + routed)
┌─────────────────────────────────────────────────┐
│ Frontend (React) │
│ DubTab · VoicePreview · BatchQueue · Gallery │
├─────────────────────────────────────────────────┤
│ Backend (FastAPI) │
97 API endpoints · SSE streaming · SQLite
├──────────┬──────────┬──────────┬────────────────┤
│ WhisperX │ Demucs │OmniVoice │ Pyannote │
ASR │ Source TTS │ Diarization
│ Sep. │ │
└──────────┴──────────┴──────────┴────────────────┘
CUDA / MPS / ROCm / CPU (auto-detected)
```
---
@@ -352,71 +288,27 @@ OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictatio
| Category | Features |
|----------|----------|
| **Longform** | Audiobook editor (text/EPUB/PDF → chaptered .m4b), Stories multi-voice editor, two-pass loudnorm mastering, crash-resume for interrupted renders, pronunciation control + SSML-lite prosody |
| **Dubbing** | Full pipeline (transcribe→translate→synthesize→mux), scene-aware splitting, lip-sync scoring, streaming TTS, per-speaker voice assignment, Smart Fit timing + second-pass QC, dedicated Dub home |
| **Voice** | Zero-shot cloning, voice design, A/B comparison, voice preview widget, gallery with favorites/tags, portable persona bundles (`.ovsvoice`), voice console workspace |
| **Audio** | Demucs vocal isolation, per-segment gain, selective track export, stem/SRT/VTT/MP3 export, unlimited-length TTS via sentence-chunked generation |
| **Dubbing** | Full pipeline (transcribe→translate→synthesize→mux), scene-aware splitting, lip-sync scoring, streaming TTS |
| **Voice** | Zero-shot cloning, voice design, A/B comparison, voice preview widget, gallery with favorites/tags |
| **Audio** | Demucs vocal isolation, per-segment gain, selective track export, stem/SRT/VTT/MP3 export |
| **Multi-Lang** | Multi-language batch picker, batch dubbing queue with sequential GPU execution |
| **Diarization** | Pyannote ML diarization, auto speaker clone extraction, per-speaker voice assignment |
| **ASR** | 9 engines (WhisperX, Faster-Whisper, isolated Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet TDT, Moonshine, FunASR/SenseVoice, sherpa-onnx live dictation), crash-isolated subprocess backend |
| **TTS** | 11 engines (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, + lazy: IndexTTS 2, OmniVoice GGUF, Supertonic 3), engine routing with GPU preflight |
| **Infra** | Docker deployment, CUDA/MPS/ROCm auto-detect, cuDNN 8 compat, VRAM-aware model offloading, engine routing (no silent CPU fallback), diagnostics suite & error journal, restricted-network mirror support |
| **Infra** | Docker deployment, CUDA/MPS/ROCm auto-detect, cuDNN 8 compat, VRAM-aware model offloading |
| **AI Provenance** | AudioSeal invisible watermarking (SynthID-like), video logo overlay, watermark detection API |
| **UX** | Undo/redo, keyboard shortcuts, drag-and-drop, session persistence, glassmorphism design system, UI scale fix for Linux/WebKitGTK |
| **UX** | Undo/redo, keyboard shortcuts, drag-and-drop, session persistence, glassmorphism design system |
| **Real-time Events** | WebSocket event bus — instant sidebar refresh on data mutations, exponential backoff reconnect |
| **State Management** | Zustand store migration — `uiSlice`, `pillSlice`, `dubSlice`, `generateSlice`, `prefsSlice`, `glossarySlice` |
| **Desktop** | Cross-platform Tauri installers (macOS DMG — Apple Silicon; Intel unsupported for the local backend, #889 — Windows MSI, Linux deb/AppImage), auto-update infrastructure, single-instance enforcement, close-to-tray, macOS Gatekeeper fix |
| **Dictation** | Global system-wide hotkey (`⌘+⇧+Space`), frameless floating widget, streaming ASR via WebSocket, auto-paste, customizable hotkey, local-LLM transcript refinement |
| **Desktop** | Cross-platform Tauri installers (macOS DMG, Windows MSI, Linux deb/AppImage), auto-update infrastructure |
| **Windows Hardening** | Cross-platform log paths, Triton workaround, HF symlink bypass, 300s health check timeout |
| **Dictation** | Global system-wide hotkey (`⌘+⇧+Space`), frameless floating widget, streaming ASR via WebSocket, auto-paste |
| **Batch Pipeline** | Full batch TTS: extract → transcribe → translate → generate → mix → export, with live progress tracking |
| **MCP Server** | OmniVoice as a local TTS/STT provider for Claude, Cursor, and any MCP client |
| **Remote Backend** | Point the desktop UI at a remote backend URL with bearer auth (Tailscale-documented) |
| **Reliability** | Stall watchdog on bootstrap splash, per-engine GPU compatibility matrix, actionable errors for non-executable engine binaries, setuptools auto-repair |
### 🔜 Up Next
- 🎬 **Lip-sync v2** — visual speech timing with wav2lip
- 📖 **Audiobook Editor** — chapter-aware long-form narration
- 🌐 **Hosted Demo** — try OmniVoice without installing anything
- 🔌 **Plugin Marketplace** — community-contributed TTS engines and effects
- 🎵 **Real-time Voice Changer** — live microphone transformation during calls
---
## Sponsor / Donate
OmniVoice Studio is built by one developer using Claude Code and AI agents — and the agent bills are real. Over the last three months I've spent thousands of dollars on Claude subscriptions to keep the features shipping, the bugs fixed, and your issues answered. If OmniVoice has created value for you, helping cover those bills means I can keep developing full-time.
<div align="center">
**This month's agent bill fund**
<img src="https://img.shields.io/badge/raised_%2410_of_%24200-5%25-EAB308?style=for-the-badge" alt="$10 / $200 raised" />
<br/><br/>
<a href="https://ko-fi.com/debpalash"><img src="https://img.shields.io/badge/Ko--fi-Support_❤️-FF5E5B?style=for-the-badge&logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
&nbsp;&nbsp;
<a href="https://paypal.me/palashCoder"><img src="https://img.shields.io/badge/PayPal-Donate-00457C?style=for-the-badge&logo=paypal&logoColor=white" alt="PayPal" /></a>
<br/>
<sub>Every dollar goes directly to agent bills — keeping OmniVoice development continuous.</sub>
</div>
### Sponsors
OmniVoice is **free** and **AGPL-3.0** — no paid tier, no SaaS revenue. Sponsors keep development going, and in return get a logo slot here, in the app, and (for top tiers) on the project website. It's a thank-you, never a paywall. **[See tiers & become a sponsor →](SPONSORS.md)**
<div align="center">
<!-- SPONSORS:START — logo slots are filled here as sponsors come aboard; see SPONSORS.md -->
**Your logo here** — [become a sponsor](SPONSORS.md)
<!-- SPONSORS:END -->
</div>
<sub>💡 GitHub also shows a **Sponsor** button at the top of this repo, wired to the same links via <a href=".github/FUNDING.yml"><code>.github/FUNDING.yml</code></a>.</sub>
---
@@ -461,7 +353,7 @@ For voice cloning and dubbing, yes — OmniVoice uses a state-of-the-art diffusi
<details>
<summary><b>Does it work on Apple Silicon (M1/M2/M3/M4)?</b></summary>
<br/>
Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are available for faster transcription on Apple hardware. <b>Intel Macs are not supported</b>: the app UI installs, but the local Python backend cannot run because PyTorch no longer ships Intel-Mac wheels (<a href="https://github.com/debpalash/OmniVoice-Studio/issues/889">#889</a>) — an Intel Mac can only be used with a remote backend.
Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are available for faster transcription on Apple hardware.
</details>
<details>
@@ -485,7 +377,7 @@ Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are availab
<details>
<summary><b>Can I add my own TTS engine?</b></summary>
<br/>
Yes. OmniVoice uses a <b>built-in backend registry</b>. To add an engine in ~50 lines, subclass <code>TTSBackend</code> in <code>backend/services/tts_backend.py</code> and add it to the <code>_REGISTRY</code> dictionary. Eleven engines are built in: OmniVoice, CosyVoice 3, GPT-SoVITS, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, KittenTTS, Sherpa-ONNX, plus lazy-registered IndexTTS 2, OmniVoice GGUF, and Supertonic 3. See the <a href="#tts-engines">TTS Engines</a> section for details.
Yes. OmniVoice uses a <b>built-in backend registry</b>. To add an engine in ~50 lines, subclass <code>TTSBackend</code> in <code>backend/services/tts_backend.py</code> and add it to the <code>_REGISTRY</code> dictionary at the bottom. Six engines are built in: OmniVoice, CosyVoice, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, and KittenTTS. See the <a href="#tts-engines">TTS Engines</a> section for details.
</details>
---
@@ -515,9 +407,6 @@ OmniVoice Studio is built on the shoulders of exceptional open-source work:
| [**CTranslate2**](https://github.com/OpenNMT/CTranslate2) | Optimized Transformer inference on CPU and GPU |
| [**AudioSeal (Meta)**](https://github.com/facebookresearch/audioseal) | Invisible neural audio watermarking for AI provenance |
| [**Tauri**](https://tauri.app) | Native desktop app framework |
| [**Supertone / Supertonic 3**](https://huggingface.co/Supertone/supertonic-3) | ONNX TTS engine — 31 languages, CPU-efficient |
| [**Sherpa-ONNX**](https://github.com/k2-fsa/sherpa-onnx) | WASM-ready universal TTS/ASR runtime |
| [**GPT-SoVITS**](https://github.com/RVC-Boss/GPT-SoVITS) | Zero-shot TTS engine — 5 languages, RTF 0.014 |
---
@@ -527,8 +416,7 @@ OmniVoice Studio is built on the shoulders of exceptional open-source work:
If you read this far, you're our kind of person.<br/>
**[⭐ Star this repo](https://github.com/debpalash/OmniVoice-Studio)** so others can find it too.<br/>
**[💬 Join the Discord](https://discord.gg/bzQavDfVV9)** to share what you build.<br/>
**[❤️ Support development](https://ko-fi.com/debpalash)** — fund the AI agent bills that keep OmniVoice shipping.
**[💬 Join the Discord](https://discord.gg/bzQavDfVV9)** to share what you build.
<br/>
-4
View File
@@ -372,13 +372,9 @@ OmniVoice 配备多引擎 TTS 后端。默认引擎(OmniVoice)始终可用
| **MLX-Audio**Kokoro, Qwen3-TTS, CSM, Dia 等) | 多语言 | 因引擎而异 | 因引擎而异 | ❌ | ✅ 原生 | ❌ | 因引擎而异 |
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
| **MOSS-TTS-Nano** | 20 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **MOSS-TTS-v1.5**(8B,可选装) | 31 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **dots.tts**(2B,可选装) | 24 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ❌ | Apache-2.0 |
| **KittenTTS** | 英语 | ❌ | ❌ | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
> **CUDA** = GPU 加速 · **MPS** = Apple Silicon Metal · **CPU** = 随处可运行,大模型较慢 · KittenTTS 和 MOSS-TTS-Nano 可在 CPU 上实时运行 · MLX-Audio 仅限 Apple Silicon。
>
> **MOSS-TTS-v1.5**8B,约 16 GB 权重)和 **dots.tts**(2B,约 9 GB 权重)是重量级可选引擎,从本地克隆在独立 venv 中运行——参见 [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) 和 [dots.tts](docs/engines/dots-tts.md)。两者均不支持 Apple Silicon **MPS**(上游仅支持 CUDA/CPU;在 Mac 上以 CPU 运行)。dots.tts 上游仅支持 Linux/macOS——无 Windows 路径。
---
+4 -14
View File
@@ -2,20 +2,10 @@
## Supported Versions
| Version | Supported |
|---------|-----------|
| 0.3.x (latest release + `main` previews) | ✅ Current — all fixes land here |
| 0.2.7 | ⚠️ Legacy stable — security fixes only, upgrade recommended |
| < 0.2.7 | ❌ No longer supported |
## Model supply chain
OmniVoice supports models from **public, verifiable sources only** (Hugging
Face repos, official project releases). Privately sold or gated model files
are not supported: an archive from a private source can carry anything
(bundled executables, modified configs), and nobody else can verify or
reproduce it. Treat any privately distributed model file as an untrusted
download, and never run executables bundled with model archives.
| Version | Supported |
|---------|--------------------|
| 0.2.x | ✅ Current release |
| < 0.2 | ❌ No longer supported |
## Reporting a Vulnerability
-119
View File
@@ -1,119 +0,0 @@
<div align="center">
<img src="docs/logo.png" alt="OmniVoice Logo" width="96" />
<h1>Sponsor OmniVoice Studio</h1>
<p><b>Keep the open-source ElevenLabs alternative free, local, and shipping.</b></p>
</div>
---
## Why sponsor?
OmniVoice Studio is built by one developer, in the open, using Claude Code and AI agents — and the agent bills are real. Over the last few months I've spent thousands of dollars on Claude subscriptions to keep features shipping, bugs fixed, and your issues answered.
OmniVoice is **free**, **fully local**, and **AGPL-3.0**. There's no paid tier, no accounts, no cloud, and no SaaS revenue — nothing runs on a server we bill you for, because nothing runs on a server at all. That's the whole point, and it's also why there's no recurring revenue to fund development. Sponsorship is what makes continued full-time work possible.
If OmniVoice has created value for you or your company, sponsoring means the next release keeps coming — and you get a thank-you (and, at most tiers, a logo slot) in return.
### Where your money goes
Every dollar goes to the cost of building OmniVoice — chiefly the **AI agent bills that keep it shipping** (Claude subscriptions and API usage), plus the occasional signing certificate, test hardware, and model-hosting costs. It is not a salary top-up; it's what keeps the lights on for continuous development.
---
## Sponsorship tiers
Tiers are about **visibility and gratitude** — what you get is placement, not gated features (see [Not a paywall](#not-a-paywall)). Higher tiers include everything in the tiers below them.
| Tier | Suggested monthly | What you get |
|------|-------------------|--------------|
| **🥉 Backer** | _set by owner_ <!-- OWNER: set amounts --> | Your name or handle listed in the **Backers** section of this file, with a link of your choice. |
| **🟫 Bronze** | _set by owner_ <!-- OWNER: set amounts --> | Everything above, **plus** a small logo in `SPONSORS.md` **and** in the README [Sponsors section](README.md#sponsors). |
| **🥈 Silver** | _set by owner_ <!-- OWNER: set amounts --> | Everything above, **plus** your logo in the **README** and in the app's **in-app Sponsors page footer** (as that page ships). |
| **🥇 Gold** | _set by owner_ <!-- OWNER: set amounts --> | Everything above, **plus** a **prominent logo slot** and link on the project **website / landing page**. |
> **Amounts are set by the maintainer** — look for the `<!-- OWNER: set amounts -->` markers in this file's source. If you don't see a price that fits, say so in your inquiry; custom and annual arrangements are welcome.
Placements marked "as that page ships" (the in-app Sponsors page and the project website) are on the near-term roadmap. Until they exist, Silver/Gold logos live in `SPONSORS.md` and the README, and are added to the app and site the moment those land — no re-application needed.
---
## How to become a sponsor
**1. Open a sponsorship inquiry (recommended).** This opens a short GitHub form (name/org, logo, tier, contact) so we can get you set up:
> **[→ Open a sponsorship inquiry](https://github.com/debpalash/OmniVoice-Studio/issues/new?template=sponsor.yml)**
**2. Or start recurring support directly:**
- **Ko-fi (recurring or one-time):** [ko-fi.com/debpalash](https://ko-fi.com/debpalash)
- **PayPal (one-time):** [paypal.me/palashCoder](https://paypal.me/palashCoder)
If you sponsor via Ko-fi/PayPal and want a logo slot, still open an inquiry (or drop a note there) so we know who to credit and where to link.
**3. Prefer to talk first?** Reach out directly:
- Email: <!-- OWNER: add your sponsor contact email here if you want one public -->
- Or ask in the `#dev` / `#announcements` channels on [Discord](https://discord.gg/bzQavDfVV9).
---
## Logo & asset guidelines
To make your logo look sharp everywhere (README on GitHub, the in-app page, the website), please send:
- **Format:** **SVG preferred** (scales cleanly); otherwise **PNG at 2× resolution**.
- **Background:** **transparent** — no baked-in white/black box.
- **Contrast:** send a variant that stays legible on **both light and dark** backgrounds, or one light-mode and one dark-mode file (GitHub and the app both render in either theme).
- **Dimensions:** legible at **~40px tall**; keep the wordmark within roughly **480px wide**. Landscape/wordmark shapes work best in the README row.
- **File size:** keep SVGs under ~50 KB and PNGs under ~100 KB.
- **Link target:** the destination URL you want the logo to point to (usually your homepage).
**How your logo gets added:**
- **Easiest:** attach the asset and link in your [sponsorship inquiry](https://github.com/debpalash/OmniVoice-Studio/issues/new?template=sponsor.yml) — the maintainer places it.
- **Or open a PR:** add your asset under `docs/sponsors/` and an entry to the tables in this file. Silver/Gold logos are also wired into the app's in-app Sponsors page (via the `sponsors.js` manifest) and the project website as those surfaces ship.
By sponsoring you confirm you have the right to use the submitted logo and grant OmniVoice permission to display it in the contexts above. We won't alter your logo beyond scaling, and we'll remove it promptly on request.
---
## Current sponsors
OmniVoice doesn't have any sponsors yet — **you could be the first.** These slots fill in as sponsors come aboard.
### 🥇 Gold
_Be the first Gold sponsor — [claim this slot](#how-to-become-a-sponsor)._
### 🥈 Silver
_Open — [become a Silver sponsor](#how-to-become-a-sponsor)._
### 🟫 Bronze
_Open — [become a Bronze sponsor](#how-to-become-a-sponsor)._
### 🥉 Backers
_Open — [become a Backer](#how-to-become-a-sponsor)._
<!-- When a sponsor joins, add them to the matching section above:
- Logo tiers (Bronze+): <a href="https://sponsor.example"><img src="docs/sponsors/name.svg" alt="Name" height="48" /></a>
- Backers: - [Name / handle](https://link) -->
---
## Not a paywall
Sponsorship is a **thank-you, never a paywall.**
Every feature of OmniVoice Studio is and will remain **free** and **open-source under [AGPL-3.0](LICENSE)**. Sponsors do **not** get private builds, gated features, license exceptions, or anything that degrades the experience for people who don't (or can't) pay. What sponsors get is **visibility and our gratitude** — and the knowledge that they're directly funding the next release.
OmniVoice stays local-first and fully functional with zero dollars spent. Sponsoring just helps it keep getting better, faster.
---
<div align="center">
<sub>Thank you for keeping local-first voice AI alive and free. ❤️</sub><br/>
<sub>Questions? <a href="https://github.com/debpalash/OmniVoice-Studio/issues/new?template=sponsor.yml">Open an inquiry</a> · <a href="https://discord.gg/bzQavDfVV9">Discord</a></sub>
</div>
-36
View File
@@ -1,36 +0,0 @@
# Support
## Where to get help
| Channel | Best for |
|---|---|
| [Discord](https://discord.gg/bzQavDfVV9) — `#help` | Setup problems, quick questions, sharing results |
| [GitHub Issues](https://github.com/debpalash/OmniVoice-Studio/issues) | Bugs and feature requests — use the templates; attach the diagnostic bundle (Settings → About → "Save diagnostic bundle") |
| [GitHub Discussions](https://github.com/debpalash/OmniVoice-Studio/discussions) | Design questions, ideas, show & tell |
| Security issues | **Never a public issue** — see [SECURITY.md](SECURITY.md) for private reporting |
## Model sources we support
OmniVoice is built on the idea that everything it runs is **open and available
to everyone**: free, public models with verifiable sources and licenses
(Hugging Face repos, official project releases), so the whole community can
use, test, and debug the same thing.
**We do not support privately sold, paywalled, or gated model files.** A model
delivered privately can't be verified, reproduced, or shared — it doesn't fit
the project's goals, and issues involving such models will be politely closed.
As a general safety rule, never run executables bundled inside any model
archive.
## Before filing a bug
1. Update to the latest release (or `main` if you follow previews) — fixes ship continuously.
2. Run the in-app self-check: **Settings → About → Run self-check**.
3. Search existing issues; add a 👍 + your details to an existing one rather than opening a duplicate.
## Response expectations
This is an open-source project maintained with the help of an automated triage
bot: issues are typically triaged within hours and every report gets a human-
approved response. Reproducible reports with a diagnostic bundle get fixed
fastest.
+1 -7
View File
@@ -13,18 +13,12 @@
# Run: uv run pyinstaller backend.spec --noconfirm --clean
import platform
import sys
from PyInstaller.utils.hooks import collect_data_files, collect_all, collect_submodules, copy_metadata
from PyInstaller.utils.hooks import collect_data_files, collect_all, collect_submodules
IS_MAC_ARM = sys.platform == "darwin" and platform.machine() == "arm64"
datas = []
binaries = []
# Bundle the omnivoice package's .dist-info so importlib.metadata.version()
# resolves inside the frozen build. Without it the backend can't read its own
# version and falls back to the literal in backend/core/version.py — which is
# how a 0.3.6 desktop build shipped reporting "0.3.5" in About + bug reports.
datas += copy_metadata('omnivoice')
hiddenimports = [
# Web stack
'uvicorn', 'uvicorn.logging', 'uvicorn.loops', 'uvicorn.loops.auto',
-32
View File
@@ -7,13 +7,9 @@ composed at the route or router level without surprises.
Currently exposed:
- `require_loopback`: 403 unless the request came from a loopback origin
(bypassed in explicit server mode see `_server_mode`).
- `ws_remote_authorized`: whether a WebSocket handshake from a non-loopback
client carries the remote API key (Wave 2.3) used by WS endpoints that
keep their own inline loopback guards.
"""
import os
import secrets
from fastapi import HTTPException, Request
@@ -75,31 +71,3 @@ def require_loopback(request: Request) -> None:
if _server_mode():
return
raise HTTPException(status_code=403, detail="loopback origin required")
def remote_api_key() -> str | None:
"""The remote-backend bearer key (Wave 2.3), or None when remote mode is
off. Read at call time so tests can monkeypatch the env."""
return os.environ.get("OMNIVOICE_API_KEY") or None
def ws_remote_authorized(websocket) -> bool:
"""Whether a WebSocket handshake presents the remote API key.
Browser WebSockets cannot set an Authorization header, so the key may
arrive as ``?api_key=`` or via the ``ov_key`` cookie that the bearer
middleware sets on the first authenticated HTTP request. Returns False
when remote mode is off callers keep their loopback-only behavior.
"""
key = remote_api_key()
if not key:
return False
auth = websocket.headers.get("authorization", "")
supplied = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
if not supplied:
supplied = (
websocket.query_params.get("api_key")
or websocket.cookies.get("ov_key")
or ""
)
return secrets.compare_digest(supplied, key)
+6 -7
View File
@@ -18,6 +18,7 @@ Design notes
"""
from __future__ import annotations
import asyncio
import hashlib
import logging
import os
@@ -136,7 +137,7 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
from api.routers.generation import ( # noqa: WPS433 — intentional lazy import
get_model,
_run_inference,
run_on_gpu_pool_guarded,
_gpu_pool,
_safe_torchaudio_save,
)
@@ -146,6 +147,8 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
language = None
text = (a.get("sample_script") or "").strip() or _FALLBACK_SCRIPT
loop = asyncio.get_running_loop()
def _infer(seed: int):
return _run_inference(
model, # _model
@@ -168,18 +171,14 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
"broadcast", # effect_preset
)
# Bounded + pool-reset on hang so a wedged preview render can't starve the
# GPU pool and brick the backend (#730 class).
audio_tensor = await run_on_gpu_pool_guarded(
lambda: _infer(_PREVIEW_SEED), what="Archetype preview generate")
audio_tensor = await loop.run_in_executor(_gpu_pool, _infer, _PREVIEW_SEED)
if _is_unusable_audio(audio_tensor):
# Blank OR a degenerate tonal buzz — retry once on a different seed to
# step off the bad diffusion trajectory. Static message only: the
# archetype id is request-derived (CodeQL log-injection); the seed is a
# module constant, safe to log.
logger.warning("Archetype rendered unusable at seed %d — retrying once", _PREVIEW_SEED)
audio_tensor = await run_on_gpu_pool_guarded(
lambda: _infer(_PREVIEW_SEED + 1), what="Archetype preview generate")
audio_tensor = await loop.run_in_executor(_gpu_pool, _infer, _PREVIEW_SEED + 1)
if _is_unusable_audio(audio_tensor):
raise RuntimeError("the voice engine returned no audible audio for this archetype")
-759
View File
@@ -1,759 +0,0 @@
"""Audiobook creator endpoints (parity Wave 5).
``POST /audiobook/plan`` pure preview: parse a chapter-delimited script
(Markdown ``# H1`` chapters, inline ``[voice:NAME]`` / ``[pause …]``) into the
chapter/span plan, no synthesis.
``POST /audiobook`` the synth job: render each chapter through the active TTS
backend (reusing ``services.audiobook.synthesize_chapter`` + ``chunked_tts``),
then mux the chapter WAVs into a chapterized **m4b** (FFMETADATA1 chapters via
``build_m4b_cmd``). Progress streams as Server-Sent Events, mirroring the dub
pipeline. ffmpeg-gated without ffmpeg the job reports an error event and
stops (the m4b is the only output format).
``GET /audiobook/jobs`` + ``POST /audiobook/resume/{job_id}`` durable
crash-resume: an interrupted render persists its plan + params to a
``resume.json`` manifest in the job work dir, so it can be resumed later (the
content-addressed chapter cache makes finished chapters instant) even without
the original script. The resume UI affordance remains a follow-up.
epub/pdf ingest, ACX mastering shipped; the resume UI surface remains a follow-up.
"""
import asyncio
import json
import logging
import os
import re
import uuid
from fastapi import APIRouter, File, HTTPException, UploadFile
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from services.audiobook import (
parse_audiobook_script,
synthesize_chapter,
)
from services.longform_render import (
LOUDNESS_PRESETS,
build_concat_list,
build_ffmetadata,
build_render_cmd,
prune_cache_dir,
)
from services import longform_resume # pure (no torch) — durable resume manifest
logger = logging.getLogger("omnivoice.audiobook")
router = APIRouter()
# A cover filename as produced by /audiobook/cover: 12 hex chars + image ext.
# An exact-match allowlist is the strongest barrier (and the one CodeQL's
# path-injection query recognizes) — anything else is rejected outright.
_COVER_NAME_RE = re.compile(r"^[0-9a-f]{12}\.(?:jpg|jpeg|png)$")
def _safe_cover_path(cover_path: str | None) -> str | None:
"""Confine a user-supplied cover to the upload directory before it can flow
into ffmpeg.
Covers only ever come from ``/audiobook/cover``, which writes them to
``OUTPUTS_DIR/audiobook_covers`` with a generated name. We rebuild the path
from the basename alone (``os.path.basename`` strips any directory component
or ``..`` traversal) joined onto that fixed directory, so no caller-supplied
path absolute or relative can escape it. Returns the path only if the
file actually exists there, else None."""
if not cover_path:
return None
from core.config import OUTPUTS_DIR
name = os.path.basename(cover_path)
if not _COVER_NAME_RE.match(name):
return None # not a name the upload endpoint could have produced
cover_dir = os.path.realpath(os.path.join(OUTPUTS_DIR, "audiobook_covers"))
real = os.path.realpath(os.path.join(cover_dir, name))
# Containment check on the resolved path itself — it must live inside the
# covers dir. Belt-and-suspenders over the regex+basename above; the
# commonpath form is the path-injection barrier static analysis recognizes.
if os.path.commonpath([real, cover_dir]) != cover_dir:
return None
return real if os.path.isfile(real) else None
class AudiobookPlanRequest(BaseModel):
text: str
default_voice: str | None = None
@router.post("/audiobook/plan")
def audiobook_plan(req: AudiobookPlanRequest) -> dict:
"""Parse a script into a chapter/span plan (pure preview, no synthesis)."""
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
return plan.to_dict()
#: Cover size cap mirrors longform_render's guard (8 MB — a book cover, not a
#: payload). Kept in sync intentionally; the render builder re-validates too.
_COVER_MAX_BYTES = 8 * 1024 * 1024
#: Import upload cap — a generous ceiling for a .txt/.md/.epub manuscript that
#: still stops a memory-exhaustion upload (the whole file is read into RAM).
_IMPORT_MAX_BYTES = 64 * 1024 * 1024
#: Upper bound on chapters in a single /longform/render plan — far above any real
#: book, but stops a pathological request from allocating/holding the job forever.
_MAX_CHAPTERS = 10_000
@router.post("/audiobook/import")
async def audiobook_import(file: UploadFile = File(...)) -> dict:
"""Import a ``.txt``/``.md``/``.epub``/``.pdf`` into a chapter-delimited script.
EPUB is parsed in spine order (stdlib only, local); PDF text is extracted
with pypdf (pure-Python) then chapterized; plain text gets ``# `` headings
inserted ahead of obvious chapter-title lines. Returns the script text (for
the editor) + the resulting chapter count."""
from services.longform_import import (
chapterize_plaintext,
epub_to_chapter_script,
pdf_to_chapter_script,
)
name = (file.filename or "").lower()
data = await file.read()
if not data:
raise HTTPException(status_code=400, detail="empty file")
if len(data) > _IMPORT_MAX_BYTES:
raise HTTPException(status_code=400, detail="file too large (max 64 MB)")
if name.endswith(".epub"):
try:
script = epub_to_chapter_script(data)
except ValueError as e:
raise HTTPException(status_code=400, detail=f"couldn't parse EPUB: {e}")
elif name.endswith(".pdf"):
try:
script = pdf_to_chapter_script(data)
except ValueError as e:
raise HTTPException(status_code=400, detail=f"couldn't parse PDF: {e}")
else:
script = chapterize_plaintext(data.decode("utf-8", "ignore"))
if not script.strip():
raise HTTPException(status_code=400, detail="no text found in the file")
plan = parse_audiobook_script(script)
return {"text": script, "chapters": plan.chapter_count}
@router.post("/audiobook/cover")
async def audiobook_cover(cover: UploadFile = File(...)) -> dict:
"""Upload a cover image; returns a server-side ``path`` to pass back as
``cover_path`` in the synth request. Validated here (jpg/png + size cap) and
again at render time."""
from core.config import OUTPUTS_DIR
ext = os.path.splitext(cover.filename or "")[1].lower()
if ext not in (".jpg", ".jpeg", ".png"):
raise HTTPException(status_code=400, detail="cover must be a .jpg or .png")
data = await cover.read()
if not data or len(data) > _COVER_MAX_BYTES:
raise HTTPException(status_code=400, detail="cover must be between 1 byte and 8 MB")
cover_dir = os.path.join(OUTPUTS_DIR, "audiobook_covers")
os.makedirs(cover_dir, exist_ok=True)
path = os.path.join(cover_dir, f"{uuid.uuid4().hex[:12]}{ext}")
with open(path, "wb") as f:
f.write(data)
return {"path": path}
class AudiobookRequest(BaseModel):
text: str
default_voice: str | None = None # voice profile id; None = engine default
language: str | None = None # None/"Auto" → profile language, else autodetect (#505)
bitrate: str = "128k"
format: str = "m4b" # "m4b" | "mp3"
loudness: str | None = None # None/"off" | "acx" | "podcast" (opt-in)
cover_path: str | None = None # server-side path to a jpg/png cover
# Global tags embedded in the output: {title, author, narrator, year,
# genre, description}. Player-visible (Apple Books / Audible read these).
metadata: dict | None = None
# Optional pronunciation lexicon {word: respelling} applied before synthesis.
lexicon: dict | None = None
def _resolve_voice(profile_id: str | None) -> dict:
"""Map a voice-profile id to (ref_audio, ref_text, instruct, seed).
Compact form of the resolver in generation.py covers locked, design and
clone profiles. Returns all-None for the engine default (no profile).
"""
out = {"ref_audio": None, "ref_text": None, "instruct": None, "seed": None}
if not profile_id:
return out
from core.config import VOICES_DIR
from core.db import db_conn
with db_conn() as conn:
row = conn.execute("SELECT * FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
if not row:
return out
try:
kind = row["kind"] or "clone"
except (KeyError, IndexError):
kind = "clone"
if row["is_locked"] and row["locked_audio_path"]:
out["ref_audio"] = os.path.join(VOICES_DIR, row["locked_audio_path"])
out["ref_text"] = row["ref_text"]
out["instruct"] = row["instruct"]
elif kind == "design":
out["ref_audio"] = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
out["ref_text"] = row["ref_text"] if out["ref_audio"] else None
out["instruct"] = row["instruct"]
else:
out["ref_audio"] = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
out["ref_text"] = row["ref_text"]
out["instruct"] = row["instruct"]
try:
if row["seed"] is not None:
out["seed"] = row["seed"]
except (KeyError, IndexError):
pass
return out
def _resolve_default_language(language: str | None, default_voice: str | None) -> str | None:
"""Pick the language to thread into the longform synth callable.
Priority (mirrors the single-shot /generate path, #533): an explicit
non-Auto request ``language`` wins; otherwise the selected profile's stored
language drives it; otherwise ``None`` (genuine Auto the engine
autodetects, exactly as before). Hardcoding ``None`` here (#505 B2) let the
engine re-autodetect per chunk, so a non-English clone flipped to the wrong
language on short/ambiguous chapters.
"""
if language and language != "Auto":
return language
if default_voice:
from core.db import db_conn
with db_conn() as conn:
row = conn.execute(
"SELECT language FROM voice_profiles WHERE id=?", (default_voice,)
).fetchone()
if row:
try:
prof_lang = row["language"]
except (KeyError, IndexError):
prof_lang = None
if prof_lang and prof_lang != "Auto":
return prof_lang
return None
def _build_synth(default_voice: str | None, language: str | None = None) -> dict:
"""Describe how to synthesize for the active TTS engine.
Returns a dict with ``mode``, ``resolve`` (voice-id resolved refs, cached
per id) and ``engine_id``. For OmniVoice it also carries the async
``get_model``; other engines carry a ready ``synth`` + ``sample_rate``.
:func:`_prepare_synth` turns this into a uniform ``(synth, sr, resolve,
engine_id)`` once the (async) model is in hand.
``language`` (already resolved by :func:`_resolve_default_language`) is
threaded into every chunk's ``generate`` so a non-English clone stays in its
language instead of re-autodetecting per chunk (#505 B2). ``None`` keeps the
engine's autodetect behavior unchanged.
"""
from services.tts_backend import OmniVoiceBackend, active_backend_id, get_backend_class
cache: dict = {}
def resolve(voice_id):
key = voice_id or default_voice
if key not in cache:
cache[key] = _resolve_voice(key)
return cache[key]
engine_id = active_backend_id()
cls = get_backend_class(engine_id)
if cls is OmniVoiceBackend:
from services.model_manager import get_model
return {"mode": "omnivoice", "resolve": resolve,
"engine_id": engine_id, "get_model": get_model, "language": language}
backend = cls()
def synth(text, voice_id, speed=None):
v = resolve(voice_id)
return backend.generate(
text, language=language, ref_audio=v["ref_audio"],
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
speed=float(speed) if speed else 1.0,
)
return {"mode": "generic", "resolve": resolve, "engine_id": engine_id,
"synth": synth, "sample_rate": backend.sample_rate}
async def _prepare_synth(default_voice: str | None, language: str | None = None):
"""Resolve :func:`_build_synth` into ``(synth, sample_rate, resolve,
engine_id)`` awaiting the OmniVoice model load when needed. Shared by the
full job and the per-chapter preview. ``language`` is threaded into every
chunk so a non-English clone holds its language (#505 B2)."""
info = _build_synth(default_voice, language=language)
resolve, engine_id = info["resolve"], info["engine_id"]
if info["mode"] == "omnivoice":
lang = info["language"]
model = await info["get_model"]()
sr = getattr(model, "sampling_rate", 24000)
def synth(text, voice_id, speed=None):
v = resolve(voice_id)
return model.generate(
text=text, language=lang, ref_audio=v["ref_audio"],
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
speed=float(speed) if speed else 1.0,
)[0]
return synth, sr, resolve, engine_id
return info["synth"], info["sample_rate"], resolve, engine_id
def _render_chapter_cached(chapter, synth, sr, engine_id, resolve, cache_dir, lexicon=None):
"""Render one chapter, content-addressed so a re-run reuses it (resume).
Returns ``(wav_path, duration_s, was_cached)``. The WAV lives at
``cache_dir/<key>.wav`` where ``key`` is :func:`chapter_cache_key` over the
chapter's spans + sample rate + engine + each voice's resolved signature
(+ the lexicon, so a lexicon edit re-renders), so an unchanged chapter is
never re-synthesized. Runs in the GPU-pool executor.
"""
import json
import wave
from services.audio_io import atomic_save_wav
from services.longform_render import chapter_cache_key
from services.pronunciation import normalize_lexicon
spans_tuples = [(s.voice_id, s.text, s.pause_ms_after, getattr(s, "speed", None))
for s in chapter.spans]
sig: dict = {}
for s in chapter.spans:
k = s.voice_id or ""
if k not in sig:
v = resolve(s.voice_id)
sig[k] = f"{v.get('ref_audio')}|{v.get('ref_text')}|{v.get('instruct')}|{v.get('seed')}"
if lexicon:
# Fold the lexicon into the cache key so editing pronunciations
# invalidates cached chapters (reserved key can't collide with a voice id).
sig["\x00lexicon"] = json.dumps(normalize_lexicon(lexicon), sort_keys=True)
key = chapter_cache_key(spans_tuples, sample_rate=sr, engine_id=engine_id, voice_sig=sig)
wav_path = os.path.join(cache_dir, f"{key}.wav")
if os.path.exists(wav_path):
try:
with wave.open(wav_path, "rb") as w:
dur = w.getnframes() / float(w.getframerate() or sr)
return wav_path, dur, True
except Exception:
pass # corrupt cache entry — fall through and re-render
audio, dur = synthesize_chapter(chapter.spans, synth, sr, lexicon=lexicon)
atomic_save_wav(wav_path, audio, sr)
return wav_path, dur, False
class AudiobookPreviewRequest(BaseModel):
text: str
chapter_index: int = 0
default_voice: str | None = None
language: str | None = None # None/"Auto" → profile language, else autodetect
lexicon: dict | None = None
@router.post("/audiobook/preview")
async def audiobook_preview(req: AudiobookPreviewRequest) -> dict:
"""Render a single chapter so the user can audition it before the full run.
Reuses the same content-addressed cache as the job, so a preview warms the
cache (the later full render reuses it) and a re-preview is instant.
"""
from core.config import OUTPUTS_DIR
from services.model_manager import _gpu_pool
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
if not plan.chapters:
raise HTTPException(status_code=400, detail="no chapters parsed from the script")
n = len(plan.chapters)
if not (0 <= req.chapter_index < n):
raise HTTPException(status_code=400, detail=f"chapter_index out of range (0..{n - 1})")
chapter = plan.chapters[req.chapter_index]
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache") # shared with _render_longform_sse
os.makedirs(cache_dir, exist_ok=True)
synth, sr, resolve, engine_id = await _prepare_synth(
req.default_voice,
language=_resolve_default_language(req.language, req.default_voice),
)
loop = asyncio.get_running_loop()
wav_path, dur, was_cached = await loop.run_in_executor(
_gpu_pool, _render_chapter_cached, chapter, synth, sr, engine_id, resolve, cache_dir,
req.lexicon,
)
return {
"output": os.path.relpath(wav_path, OUTPUTS_DIR), # served via /audio
"duration_s": round(dur, 2),
"cached": was_cached,
"title": chapter.title,
}
async def _render_longform_sse(
plan,
*,
default_voice: str | None,
language: str | None = None,
fmt: str = "m4b",
bitrate: str = "128k",
loudness: str | None = None,
cover_path: str | None = None,
metadata: dict | None = None,
lexicon: dict | None = None,
job_type: str = "audiobook",
job_id: str | None = None,
resume: bool = False,
):
"""Shared chapterized-render SSE generator for Audiobook *and* Stories.
Takes a ready ``plan`` (``.chapters`` ``.title`` + ``.spans``) Audiobook
parses it from a script, Stories compiles it from cast/lines and renders
each chapter (content-addressed cache resume), isolating per-chapter
failures, then muxes the successful chapters into a tagged file. This is the
convergence point: one renderer, two front doors.
"""
from core.config import OUTPUTS_DIR
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
from services.model_manager import _gpu_pool
# Resume reuses the original job_id (continuing the same job row + cached
# chapters); a fresh render generates a new one. The id may arrive from the
# /resume/{job_id} path param, so strip it to a safe token (no path
# separators, no CR/LF) before it ever reaches a filesystem path or a log
# line — CodeQL py/path-injection + py/log-injection. Empty after the strip
# → a fresh id.
job_id = re.sub(r"[^A-Za-z0-9_-]", "", job_id or "")[:64] or uuid.uuid4().hex[:16]
try:
from core import job_store
if not resume:
job_store.create(job_id, type=job_type)
job_store.mark_running(job_id)
except Exception:
job_store = None # job history is best-effort; never block synthesis
# Persist a durable resume manifest (plan + params) so an interrupted render
# can be resumed later even without the original script. Best-effort.
try:
title = (metadata or {}).get("title") or (plan.chapters[0].title if plan.chapters else "")
longform_resume.write_manifest(longform_resume.build_manifest(
job_id=job_id, job_type=job_type, title=title,
plan_chapters=[
{"title": c.title, "spans": [s.to_dict() for s in c.spans]}
for c in plan.chapters
],
params={
"default_voice": default_voice, "language": language,
"fmt": fmt, "bitrate": bitrate,
"loudness": loudness, "cover_path": cover_path,
"metadata": metadata, "lexicon": lexicon,
},
))
except Exception: # resume durability is an enhancement; never block the render
logger.debug("[%s] resume manifest write skipped", job_id, exc_info=True)
def _emit(payload: dict) -> str:
if job_store is not None:
try:
job_store.append_event(job_id, json.dumps(payload))
except Exception:
pass # best-effort job history; never block the stream
return f"data: {json.dumps(payload)}\n\n"
if not plan.chapters:
yield _emit({"type": "error", "error": "nothing to render (no chapters)"})
return
ffmpeg = find_ffmpeg()
if not ffmpeg:
yield _emit({"type": "error", "error": "ffmpeg not available; the output needs it"})
return
# Confined work dir (job_id is already token-sanitized above; work_dir adds
# the basename + realpath barrier so CodeQL sees a clean path).
work = longform_resume.work_dir(job_type, job_id)
if work is None:
yield _emit({"type": "error", "error": "invalid job id"})
return
os.makedirs(work, exist_ok=True)
# Chapter WAVs are content-addressed in a shared cache so a re-run (after a
# failure or interruption) reuses what already rendered — only the
# missing/changed chapters synthesize again (resume). Shared across both
# front doors: an identical chapter renders once.
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache")
os.makedirs(cache_dir, exist_ok=True)
prune_cache_dir(cache_dir) # bound disk before this job adds its chapters
loop = asyncio.get_running_loop()
try:
synth, sr, resolve, engine_id = await _prepare_synth(
default_voice, language=_resolve_default_language(language, default_voice)
)
total = len(plan.chapters)
chapter_files: list[str] = []
chapters_meta: list[tuple[str, int]] = []
cached_n = 0
failed: list[int] = []
yield _emit({"type": "started", "job_id": job_id, "chapters": total})
for i, chapter in enumerate(plan.chapters):
try:
wav_path, dur, was_cached = await loop.run_in_executor(
_gpu_pool, _render_chapter_cached,
chapter, synth, sr, engine_id, resolve, cache_dir, lexicon,
)
except Exception: # isolate a bad chapter — keep going
logger.warning("[%s] chapter %d (%s) failed to render",
job_id, i, chapter.title, exc_info=True)
failed.append(i)
yield _emit({"type": "chapter_error", "index": i, "total": total,
"title": chapter.title, "error": "chapter failed to render"})
continue
chapter_files.append(wav_path)
chapters_meta.append((chapter.title, int(round(dur * 1000))))
cached_n += 1 if was_cached else 0
yield _emit({"type": "chapter", "index": i, "total": total,
"title": chapter.title, "duration_s": round(dur, 2),
"cached": was_cached})
if not chapter_files:
yield _emit({"type": "error", "error": "all chapters failed to render"})
return
yield _emit({"type": "assembling"})
meta_path = os.path.join(work, "chapters.ffmeta")
with open(meta_path, "w", encoding="utf-8") as f:
f.write(build_ffmetadata(chapters_meta, global_meta=metadata))
concat_path = os.path.join(work, "concat.txt")
with open(concat_path, "w", encoding="utf-8") as f:
f.write(build_concat_list(chapter_files))
ext = "mp3" if (fmt or "").lower() == "mp3" else "m4b"
out_name = f"{job_type}_{job_id}.{ext}"
out_path = os.path.join(OUTPUTS_DIR, out_name)
# Two-pass loudness master (#28): for a known preset, measure the
# concatenated program first, then feed the measured values back into the
# single mux encode. `measured is None` (skip OR any failure) → the mux
# falls back to single-pass. Gated identically to the pure builders
# (.lower(), no strip), so off/None/unknown/whitespace skip cleanly.
measured = None
norm = (loudness or "").lower()
if norm in LOUDNESS_PRESETS:
yield _emit({"type": "mastering", "preset": norm})
from services.loudness import measure_loudness
measured = await measure_loudness(ffmpeg, concat_path, norm, job_id=job_id)
await run_ffmpeg(
build_render_cmd(
ffmpeg, concat_path, meta_path, out_path,
fmt=ext, bitrate=bitrate, cover_path=_safe_cover_path(cover_path),
loudness=loudness, measured=measured,
),
job_id=job_id,
)
if job_store is not None:
try:
job_store.mark_done(job_id)
except Exception:
pass # best-effort job history
# The render finished — drop the resume manifest so this job is no longer
# offered for resume.
longform_resume.clear_manifest(job_type, job_id)
total_s = sum(d for _, d in chapters_meta) / 1000.0
done = {"type": "done", "output": out_name,
"chapters": len(chapter_files), "duration_s": round(total_s, 2),
"cached_chapters": cached_n, "failed_chapters": failed}
# Loudness verdict only when a preset was requested — off/None paths keep
# the exact legacy `done` shape (additive, old clients unaffected).
if norm in LOUDNESS_PRESETS:
p = LOUDNESS_PRESETS[norm]
done["loudness"] = {
"preset": norm, "target_i": p.i, "target_tp": p.tp,
"two_pass": measured is not None,
"measured_i": measured.input_i if measured else None,
}
yield _emit(done)
except Exception as e: # surface, don't 500 the stream
logger.exception("[%s] longform render failed", job_id)
if job_store is not None:
try:
job_store.mark_failed(job_id, str(e))
except Exception:
pass # best-effort job history
# Generic message only — don't leak the stack/exception text to the client.
yield _emit({"type": "error", "error": "render failed (see backend log)"})
@router.post("/audiobook")
async def audiobook_synthesize(req: AudiobookRequest):
"""Synthesize a chapterized audiobook from a script, streaming SSE progress."""
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
return StreamingResponse(
_render_longform_sse(
plan, default_voice=req.default_voice, language=req.language,
fmt=req.format, bitrate=req.bitrate,
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
lexicon=req.lexicon, job_type="audiobook",
),
media_type="text/event-stream",
)
# ── Shared longform render: Stories (and any future front door) post a plan ──
class LongformSpan(BaseModel):
voice_id: str | None = None
text: str
pause_ms_after: int = 0
speed: float | None = None
class LongformChapter(BaseModel):
title: str = ""
spans: list[LongformSpan] = []
class LongformRenderRequest(BaseModel):
chapters: list[LongformChapter] = []
default_voice: str | None = None
language: str | None = None # None/"Auto" → profile language, else autodetect (#505)
bitrate: str = "128k"
format: str = "m4b"
loudness: str | None = None
cover_path: str | None = None
metadata: dict | None = None
lexicon: dict | None = None
@router.post("/longform/render")
async def longform_render(req: LongformRenderRequest):
"""Render a pre-built chapter/span plan (the Stories Editor's compiled
cast+lines) through the shared chapterized renderer same resume, loudness,
cover, metadata, and output formats as the Audiobook job."""
from services.audiobook import AudiobookPlan, Chapter, Span
if len(req.chapters) > _MAX_CHAPTERS:
raise HTTPException(status_code=422, detail=f"too many chapters (max {_MAX_CHAPTERS})")
chapters = []
for i, c in enumerate(req.chapters):
# Keep a span if it has text to speak OR a pause to render (pause-only
# spans carry inter-line silence with empty text).
spans = [Span(voice_id=s.voice_id, text=(s.text or "").strip(),
pause_ms_after=max(0, int(s.pause_ms_after)), speed=s.speed)
for s in c.spans if ((s.text and s.text.strip()) or s.pause_ms_after > 0)]
if spans:
chapters.append(Chapter(title=c.title or f"Chapter {i + 1}", spans=spans))
plan = AudiobookPlan(chapters=chapters)
return StreamingResponse(
_render_longform_sse(
plan, default_voice=req.default_voice, language=req.language,
fmt=req.format, bitrate=req.bitrate,
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
lexicon=req.lexicon, job_type="story",
),
media_type="text/event-stream",
)
# ── Durable resume: interrupted longform renders ────────────────────────────
def _chapters_done(job_id: str) -> int:
"""Count chapters that finished rendering, from the job's persisted events.
Best-effort (0 if unavailable) used only to show resume progress."""
try:
from core import job_store
n = 0
for ev in job_store.events_since(job_id, 0, limit=100_000):
try:
if json.loads(ev["payload"]).get("type") == "chapter":
n += 1
except (ValueError, KeyError, TypeError):
continue
return n
except Exception:
return 0
@router.get("/audiobook/jobs")
def list_resumable_jobs() -> dict:
"""List interrupted longform renders that can be resumed — a work dir that
still holds a resume manifest (a job left mid-render by a crash/quit). The
ids come from scanning the filesystem, so the UI can offer one-click resume."""
from core import job_store
out = []
for e in longform_resume.scan_resumable():
jid = e["job_id"]
manifest = longform_resume.load_manifest_file(e["manifest_path"]) or {}
job = job_store.get(jid) or {}
out.append({
"job_id": jid,
"type": e["job_type"],
"status": job.get("status", "interrupted"),
"title": manifest.get("title", ""),
"total_chapters": manifest.get("total_chapters", 0),
"chapters_done": _chapters_done(jid),
"created_at": job.get("created_at"),
})
return {"jobs": out}
@router.post("/audiobook/resume/{job_id}")
async def resume_longform(job_id: str):
"""Resume an interrupted longform render from its persisted manifest. The
already-rendered chapters are content-addressed in the shared cache, so they
return instantly only the unrendered chapters synthesize again. Streams the
same SSE event shape as the original render, under the original job_id."""
from services.audiobook import AudiobookPlan, Chapter, Span
# Find the requested job among the trusted filesystem scan (every path there
# is os.listdir-sourced, never request input) and read its manifest via the
# scan's own trusted path — the request job_id is used ONLY to *select* an
# entry, never to build a path. No request-controlled value reaches a file
# operation (CodeQL py/path-injection-safe).
entry = next((e for e in longform_resume.scan_resumable()
if e["job_id"] == job_id), None)
if entry is None:
raise HTTPException(status_code=404, detail="No resumable job for that id")
manifest = longform_resume.load_manifest_file(entry["manifest_path"])
if not manifest:
raise HTTPException(status_code=404, detail="No resume manifest for that job")
chapters = [
Chapter(title=c.get("title", ""),
spans=[Span(**s) for s in c.get("spans", [])])
for c in manifest["plan"]
]
plan = AudiobookPlan(chapters=chapters)
p = manifest.get("params", {})
# Retire the interrupted job's manifest (trusted scan path) so it stops
# showing as resumable once we've kicked off the fresh-id resume.
longform_resume.discard_manifest_file(entry["manifest_path"])
# Resume under a FRESH job id (job_id=None → a server uuid in the renderer).
# The chapter cache is content-addressed (keyed by chapter content, not the
# job id), so the already-rendered chapters still hit instantly — only the
# unrendered ones synthesize. Using a fresh id means the request's job_id
# never names a work dir / output file (defence-in-depth path-injection).
return StreamingResponse(
_render_longform_sse(
plan, default_voice=p.get("default_voice"), language=p.get("language"),
fmt=p.get("fmt", "m4b"), bitrate=p.get("bitrate", "128k"),
loudness=p.get("loudness"), cover_path=p.get("cover_path"),
metadata=p.get("metadata"), lexicon=p.get("lexicon"),
job_type=entry["job_type"],
),
media_type="text/event-stream",
)
+3 -13
View File
@@ -142,7 +142,7 @@ async def _run_batch_pipeline(job_id: str, job: dict):
_set_progress(job, "transcribe", 0)
from services.asr_backend import get_active_asr_backend
from services.model_manager import _gpu_pool, _cpu_pool, run_on_gpu_pool_guarded
from services.model_manager import _gpu_pool, _cpu_pool
from services.segmentation import (
segment_transcript, assign_speakers_heuristic,
)
@@ -162,12 +162,7 @@ async def _run_batch_pipeline(job_id: str, job: dict):
pass
return segments, detected_lang
# Bound the batch transcribe (#730) so a wedged whisperx/CTranslate2 call
# can't hold its GPU-pool worker forever and starve the rest of the backend
# ("can't reach backend"); run_transcribe_guarded also resets the pool on
# timeout to restore capacity.
from services.asr_backend import run_transcribe_guarded
segments, source_lang = await run_transcribe_guarded(_gpu_pool, _transcribe, what="Batch")
segments, source_lang = await loop.run_in_executor(_gpu_pool, _transcribe)
source_lang = (source_lang or "en").split("_")[0][:2].lower()
job["segments"] = segments
job["source_lang"] = source_lang
@@ -303,9 +298,6 @@ async def _run_batch_pipeline(job_id: str, job: dict):
denoise=True, postprocess_output=True,
)
audio_out = audios[0]
# TODO(#312): this route runs the OmniVoice model directly (not the active
# backend), so VoxCPM2 never reaches it. When these routes become
# engine-aware, guard with `if not getattr(backend, "applies_own_mastering", False)`.
mastered = apply_mastering(
audio_out,
sample_rate=sr,
@@ -316,9 +308,7 @@ async def _run_batch_pipeline(job_id: str, job: dict):
return torch.zeros(1, int(dur * sr))
try:
# Bounded + pool-reset on hang so a wedged batch segment can't
# starve the GPU pool and brick the backend (#730 class).
audio_tensor = await run_on_gpu_pool_guarded(_gen, what="Batch generate")
audio_tensor = await loop.run_in_executor(_gpu_pool, _gen)
# Fit to slot
target_samples_seg = int(seg_duration * sr)
+5 -62
View File
@@ -18,26 +18,19 @@ import os
import tempfile
import time
from fastapi import APIRouter, File, Form, HTTPException, UploadFile
from fastapi import APIRouter, File, Form, UploadFile
from typing import Optional
router = APIRouter()
logger = logging.getLogger("omnivoice.capture")
def _truthy(value: Optional[str]) -> bool:
"""Parse a multipart form flag. Treats '1'/'true'/'yes'/'on'/'auto'
(any case) as on; everything else including None as off."""
return (value or "").strip().lower() in {"1", "true", "yes", "on", "auto"}
@router.post("/transcribe")
async def transcribe_audio(
audio: UploadFile = File(...),
language: Optional[str] = Form(None),
model: Optional[str] = Form(None),
mode: Optional[str] = Form(None),
refine: Optional[str] = Form(None),
):
"""Transcribe an audio file to text.
@@ -47,19 +40,10 @@ async def transcribe_audio(
model: Whisper model size (legacy; ignored in dual-mode architecture).
mode: 'fast' (default) uses MLX Turbo for speed; 'accurate' uses
WhisperX with forced alignment for word-level timing.
refine: Opt-in local-LLM cleanup of the final text (disfluencies,
self-corrections, punctuation) same pipeline the live
dictation socket uses. Off by default so MCP/CLI callers don't
pay LLM latency unless they ask; honours the user's
Settings Dictation-refinement config and silently passes
through when no LLM backend is configured. The raw ``text``
is always returned; ``refined_text`` is added only when the
LLM actually changed something.
Returns:
{
"text": "full transcription",
"refined_text": "cleaned text", # only when refine=true changed it
"segments": [ {"start": 0.0, "end": 1.5, "text": "..."}, ... ],
"language": "en",
"duration_s": 4.2,
@@ -96,17 +80,9 @@ async def transcribe_audio(
return result, backend.id
from services.model_manager import _gpu_pool
from services.asr_backend import ASRTimeoutError, run_transcribe_guarded
loop = asyncio.get_running_loop()
t0 = time.perf_counter()
try:
result, engine_id = await run_transcribe_guarded(
_gpu_pool, _run, what="Dictation",
)
except ASRTimeoutError as e:
# Backend is alive — ASR couldn't finish. 504 with guidance, not a
# silent hang the UI reads as "can't reach the local backend".
logger.warning("Capture transcription timed out: %s", e)
raise HTTPException(status_code=504, detail=str(e))
result, engine_id = await loop.run_in_executor(_gpu_pool, _run)
elapsed = round(time.perf_counter() - t0, 2)
# Normalize result shape
@@ -115,20 +91,6 @@ async def transcribe_audio(
if not full_text and segments:
full_text = " ".join(s.get("text", "") for s in segments).strip()
# Wave 1.1: strip Whisper hallucination loops from the final text.
# Segments keep the raw recognition so their timings stay truthful.
from services.refinement import collapse_repetitive_artifacts
full_text = collapse_repetitive_artifacts(full_text)
# Cross-transport parity: deterministically polish the final text
# (leading capital + terminal punctuation) exactly like the live
# dictation socket (capture_ws) does, so the widget's POST fallback and
# MCP/CLI callers get the same typed-looking result the WS returns —
# not the raw "...test" the REST path used to leak. Segments stay raw
# (their timings/verbatim recognition are the contract).
from services.text_polish import polish_text
full_text = polish_text(full_text)
# Calculate audio duration from segments if available
duration = 0.0
if segments:
@@ -136,28 +98,12 @@ async def transcribe_audio(
detected_lang = result.get("language", language or "unknown")
# Opt-in Wave 2.1 refinement, mirroring the live-dictation socket
# (capture_ws). Off-thread (it's a network call, not GPU); never
# raises — maybe_refine swallows failures and a missing LLM into a
# None pass-through, so the raw text always stands.
refined_text = None
if _truthy(refine) and full_text:
from services.refinement import maybe_refine
refined = await asyncio.to_thread(maybe_refine, full_text)
if refined:
# Polish the refined text too, so both surfaced strings read as
# typed text (mirrors the raw-vs-refined contract of the WS).
refined = polish_text(refined)
if refined != full_text:
refined_text = refined
logger.info(
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s, refined=%s",
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s",
engine_id, elapsed, duration, "accurate" if use_accurate else "fast",
refined_text is not None,
)
response = {
return {
"text": full_text,
"segments": [
{
@@ -172,9 +118,6 @@ async def transcribe_audio(
"transcription_time_s": elapsed,
"engine": engine_id,
}
if refined_text is not None:
response["refined_text"] = refined_text
return response
finally:
try:
os.unlink(tmp.name)
+16 -587
View File
@@ -8,26 +8,12 @@ live dictation feedback.
Protocol:
Client sends binary audio frames (16-bit PCM or WebM/Opus blobs)
Server sends JSON messages:
Opt-in AEC mode (``?aec=1[&sr=16000]``, parity Action 8b): for dictating
while the app plays audio. Frames must be raw int16 mono PCM, each tagged
with a 1-byte prefix 0x00 = microphone, 0x01 = playback reference. The
server runs an NLMS echo canceller, cleaning the mic against the reference
before transcription. Without the param the protocol is unchanged.
{"type": "partial", "text": "Hello wor..."} interim result
{"type": "final", "text": "Hello world.", committed result
"segments": [...], "language": "en",
"duration_s": 4.2, "transcription_time_s": 0.8,
"engine": "mlx-whisper"}
{"type": "status", "stage": "downloading"|"loading"|"ready"}
model cold-start
{"type": "error", "message": "...", "kind": "...",
"detail": "..."} error ("detail"
kept for legacy)
Every ``final`` text is normalised by services.text_polish (leading
capital for Latin scripts, terminal punctuation, single-spaced) so the
pasted result reads like typed text. Partials are raw.
{"type": "error", "detail": "..."} error
"""
from __future__ import annotations
@@ -39,8 +25,7 @@ import time
from fastapi import APIRouter, WebSocket, WebSocketDisconnect
from api.dependencies import _LOOPBACK_HOSTS, ws_remote_authorized
from services.text_polish import polish_text
from api.dependencies import _LOOPBACK_HOSTS
router = APIRouter()
logger = logging.getLogger("omnivoice.capture_ws")
@@ -59,80 +44,6 @@ MIN_BUFFER_BYTES = 64000 # ~2s of 16-bit mono 16kHz — needs enough WebM frame
# to transcribe whatever the user recorded, even short utterances.
MIN_FINAL_BUFFER_BYTES = 4000 # ~125ms of 16-bit mono 16kHz
# ── Dictate-over-playback AEC (parity Action 8b, opt-in) ──────────────────
# Activated by the ``?aec=1`` query param. When OFF (the default), the
# protocol and behaviour are byte-for-byte unchanged. When ON, the client
# streams raw int16 mono PCM frames tagged with a 1-byte type prefix so the
# server can tell mic audio from the playback reference it must cancel:
_AEC_NEAR = 0x00 # microphone frame (clean it, then buffer for ASR)
_AEC_FAR = 0x01 # playback reference frame (feed the echo model only)
def _demux_aec_frame(data: bytes) -> tuple[str, bytes]:
"""Split a prefixed AEC binary frame into ``(kind, pcm)``.
``kind`` is ``"near"`` (mic) or ``"far"`` (playback reference). An empty
or prefix-only frame yields an empty payload. Unknown prefixes are treated
as ``"near"`` so a malformed tag degrades to plain dictation rather than
dropping audio.
"""
if not data:
return "near", b""
kind = "far" if data[0] == _AEC_FAR else "near"
return kind, data[1:]
def _pcm16_to_wav(pcm: bytes, sample_rate: int) -> str | None:
"""Write raw int16 mono PCM to a temp WAV via stdlib ``wave`` (no ffmpeg).
Used on the AEC path, where frames are already decoded PCM the cleaned
samples have no container, so the ffmpeg-sniffing ``_chunks_to_wav`` would
misdetect them. Returns the temp path, or ``None`` for a too-short buffer.
"""
if not pcm or len(pcm) < 100:
return None
import wave
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".wav")
tmp.close()
try:
with wave.open(tmp.name, "wb") as wf:
wf.setnchannels(1)
wf.setsampwidth(2) # int16
wf.setframerate(sample_rate)
wf.writeframes(pcm)
return tmp.name
except Exception as e:
logger.debug("PCM->WAV failed: %s", e)
try:
os.unlink(tmp.name)
except OSError:
pass
return None
def _select_sherpa_spec(websocket: WebSocket):
"""Resolve the sherpa dictation model for this WS session, or None.
A ``?model=<id>`` query param wins (the frontend can pin a model per
session); otherwise the persisted ``dictation.model_id`` pref is used (only
when dictation is enabled). Returns the :class:`SherpaModelSpec` or None
(None the legacy Whisper/WebM path runs unchanged).
"""
try:
from services import sherpa_dictation as sd
except Exception:
return None
requested = websocket.query_params.get("model")
if requested:
return sd.get_spec(requested) # explicit selection (may be None if bad)
# Fall back to the persisted dictation pref.
try:
from services.asr_backend import dictation_model_id
mid = dictation_model_id()
except Exception:
mid = None
return sd.get_spec(mid) if mid else None
@router.websocket("/ws/transcribe")
async def ws_transcribe(websocket: WebSocket):
@@ -142,51 +53,13 @@ async def ws_transcribe(websocket: WebSocket):
# WebSocket dependency injection differs across FastAPI versions, so we
# inline the check before accept(). Without it, any local process could
# stream the user's microphone over this endpoint.
# Wave 2.3 (remote backend): a non-loopback client that presents the
# OMNIVOICE_API_KEY bearer is the thin-client dictation case — the mic
# lives on the user's machine, the GPU here — and is allowed through.
host = websocket.client.host if websocket.client else None
if host not in _LOOPBACK_HOSTS and not ws_remote_authorized(websocket):
if host not in _LOOPBACK_HOSTS:
await websocket.close(code=1008, reason="loopback origin required")
return
await websocket.accept()
# Live-dictation engine selection. When a sherpa-onnx model is selected
# (via ?model= or the dictation.model_id pref) AND sherpa is installed,
# run the dedicated low-latency handler. Otherwise fall through to the
# legacy Whisper/WebM path, byte-for-byte unchanged.
spec = _select_sherpa_spec(websocket)
if spec is not None:
from services.asr_backend import SherpaDictationBackend
ok, _reason = SherpaDictationBackend.is_available()
if ok:
if spec.streaming:
await _run_sherpa_streaming(websocket, spec)
else:
await _run_sherpa_offline(websocket, spec)
return
# sherpa not installed → fall through to the legacy path so the user
# still gets dictation (just not live partials).
logger.info("sherpa dictation selected but unavailable — legacy path")
# Opt-in dictate-over-playback AEC (parity Action 8b). Default OFF →
# identical legacy behaviour. When on, frames are 1-byte-tagged raw PCM
# and the cleaned mic stream is muxed via stdlib wave (not ffmpeg).
aec = None
pcm_sr: int | None = None
if websocket.query_params.get("aec") in ("1", "true", "on"):
try:
pcm_sr = int(websocket.query_params.get("sr", "16000"))
from services.aec import NlmsEchoCanceller
aec = NlmsEchoCanceller(sample_rate=pcm_sr)
logger.info("AEC enabled for dictation session (sr=%d)", pcm_sr)
except Exception as e:
# Bad sr or import failure → fall back to plain dictation.
logger.warning("AEC requested but disabled: %s", e)
aec = None
pcm_sr = None
audio_chunks: list[bytes] = []
total_bytes = 0
last_audio_time = time.monotonic()
@@ -224,16 +97,6 @@ async def ws_transcribe(websocket: WebSocket):
# Empty binary frame also acts as EOF — connection stays open.
running = False
break
if aec is not None:
# Tagged PCM: route the playback reference into the echo
# model and clean the mic before it reaches the buffer.
kind, payload = _demux_aec_frame(data)
if kind == "far":
aec.push_far_end(payload)
continue
if not payload:
continue
data = aec.process_near_end(payload)
audio_chunks.append(data)
total_bytes += len(data)
last_audio_time = time.monotonic()
@@ -280,7 +143,7 @@ async def ws_transcribe(websocket: WebSocket):
# Transcribe current buffer
try:
text = await _transcribe_buffer(audio_chunks[:], pcm_sr=pcm_sr)
text = await _transcribe_buffer(audio_chunks[:])
if text and text != partial_text:
partial_text = text
await _safe_send({
@@ -310,30 +173,12 @@ async def ws_transcribe(websocket: WebSocket):
# Final transcription on complete buffer — skip if client already gone.
if total_bytes > MIN_FINAL_BUFFER_BYTES:
try:
result = await _transcribe_buffer_full(audio_chunks, pcm_sr=pcm_sr)
# Dictation v2: deterministic polish so the pasted final reads
# like typed text (leading capital, terminal punctuation).
result["text"] = polish_text(result.get("text", ""))
# Wave 2.1: optional local-LLM refinement of the final text.
# HARD-BOUNDED (maybe_refine_async, ~4s OMNIVOICE_REFINE_TIMEOUT_S):
# a slow/dead LLM can never delay this `final` beyond the budget —
# it falls back to the unrefined (but polished) text. Best-effort:
# never let refinement turn a good final into an error. The raw
# text always ships too — clients paste refined_text ?? text.
if result.get("text"):
try:
from services.refinement import maybe_refine_async
refined = await maybe_refine_async(result["text"])
if refined and refined != result["text"]:
result["refined_text"] = refined
except Exception as e: # noqa: BLE001
logger.debug("Dictation refinement skipped: %s", e)
result = await _transcribe_buffer_full(audio_chunks)
if not await _safe_send({"type": "final", **result}):
logger.debug("Skipped final send — client already disconnected")
except Exception as e:
logger.error("Final transcription failed: %s", e)
await _safe_send({"type": "error", "message": str(e),
"kind": "transcribe", "detail": str(e)})
await _safe_send({"type": "error", "detail": str(e)})
else:
await _safe_send({
"type": "final",
@@ -352,433 +197,24 @@ async def ws_transcribe(websocket: WebSocket):
pass
# ── sherpa-onnx live dictation handlers ─────────────────────────────────────
#
# Both handlers read raw int16 mono PCM frames (reusing the AEC framing: an
# opt-in 1-byte type prefix when ?aec=1, else bare PCM) at ?sr= (default 16000).
# This is the low-latency transport — no WebM/ffmpeg in the hot path.
# How often the offline-kind handler re-decodes the live window for a partial
# (streaming-kind decodes every frame, no cadence needed).
SHERPA_OFFLINE_PARTIAL_S = float(os.environ.get("OMNIVOICE_SHERPA_OFFLINE_PARTIAL", "0.8"))
# Utterance gate for the offline-kind handler: once the trailing this-many
# seconds of the live buffer fall below the RMS floor, the utterance is
# COMMITTED — decoded, flushed as a `final`, and dropped from the buffer. Each
# decode is thereby bounded by one utterance instead of the whole session
# (the old full-buffer re-decode was O(n²)), and a sentence commits ~0.6s
# after the user stops speaking instead of only at EOF.
SHERPA_OFFLINE_SILENCE_S = float(os.environ.get("OMNIVOICE_SHERPA_OFFLINE_SILENCE", "0.6"))
SHERPA_OFFLINE_RMS_FLOOR = float(os.environ.get("OMNIVOICE_SHERPA_OFFLINE_RMS", "0.01"))
def _pcm16_to_f32(pcm: bytes):
"""int16 little-endian mono PCM bytes → float32 numpy in [-1, 1]."""
import numpy as np
if not pcm:
return np.zeros(0, dtype=np.float32)
# Guard against an odd trailing byte from a split frame.
if len(pcm) % 2:
pcm = pcm[:-1]
return np.frombuffer(pcm, dtype=np.int16).astype(np.float32) / 32768.0
async def _sherpa_session(websocket: WebSocket):
"""Shared WS receive setup for the sherpa handlers.
Returns ``(get_frame, state)`` where ``get_frame`` is an async callable
that yields the next near-end (mic) PCM bytes, ``b""`` for a keepalive/ref
frame, or ``None`` on EOF/disconnect. ``state`` carries sample rate, AEC,
and the disconnect flag for the caller's finaliser.
"""
pcm_sr = 16000
try:
pcm_sr = int(websocket.query_params.get("sr", "16000"))
except (TypeError, ValueError):
pcm_sr = 16000
aec = None
if websocket.query_params.get("aec") in ("1", "true", "on"):
try:
from services.aec import NlmsEchoCanceller
aec = NlmsEchoCanceller(sample_rate=pcm_sr)
except Exception as e:
logger.warning("AEC requested but disabled (sherpa): %s", e)
aec = None
return pcm_sr, aec
async def _recv_pcm_frame(websocket: WebSocket, aec):
"""Receive one frame; return (kind, pcm_bytes).
kind {"near","eof","skip"}. Demuxes AEC-tagged frames when ``aec`` is on
and feeds the playback reference into the canceller. A text "EOF" or an
empty/closed socket yields kind "eof".
"""
msg = await websocket.receive()
mtype = msg.get("type")
if mtype == "websocket.disconnect":
return "eof", b""
if mtype != "websocket.receive":
return "skip", b""
data = msg.get("bytes")
if data is not None:
if len(data) == 0:
return "eof", b""
if aec is not None:
kind, payload = _demux_aec_frame(data)
if kind == "far":
aec.push_far_end(payload)
return "skip", b""
if not payload:
return "skip", b""
return "near", aec.process_near_end(payload)
return "near", data
if msg.get("text") == "EOF":
return "eof", b""
return "skip", b""
async def _sherpa_load_with_status(websocket: WebSocket, backend, spec) -> bool:
"""Build the recognizer off the event loop, narrating cold-start progress.
Sends ``{"type":"status","stage":"downloading"|"loading"}`` before the
load ("downloading" when the pinned assets aren't in the HF cache yet;
stage-only HF's per-file progress isn't worth a callback plumb-through)
and ``{"type":"status","stage":"ready"}`` after, so the widget can show
*why* the first dictation takes a moment. Returns False when the load
failed (the error frame is sent and the socket closed here).
"""
try:
from services import sherpa_dictation as _sd
stage = "loading" if _sd.is_installed(spec) else "downloading"
except Exception:
stage = "loading"
try:
await websocket.send_json({"type": "status", "stage": stage})
except Exception:
pass
try:
await asyncio.to_thread(backend.ensure_loaded)
except Exception as e:
logger.error("sherpa dictation load failed (%s): %s", spec.id, e)
try:
await websocket.send_json({"type": "error", "message": str(e),
"kind": "load", "detail": str(e)})
await websocket.close()
except Exception:
pass
return False
try:
await websocket.send_json({"type": "status", "stage": "ready"})
except Exception:
pass
return True
async def _run_sherpa_streaming(websocket: WebSocket, spec):
"""True streaming: feed the OnlineRecognizer frame-by-frame, emit `partial`
every time the decoded text grows, and `final` on sherpa's endpoint (silence)
detection and on EOF. <300ms perceived latency on CPU for the tiny models.
"""
import numpy as np
from services.asr_backend import get_sherpa_dictation_backend
pcm_sr, aec = await _sherpa_session(websocket)
logger.info("sherpa streaming dictation: model=%s sr=%d aec=%s",
spec.id, pcm_sr, bool(aec))
# Reuse the shared, per-model warm backend (#888): the recognizer is built
# once and shared across sessions instead of rebuilt (1.32.5s) per connect,
# so the first dictation is instant when the preload warmed it. Each session
# still gets its own decode stream below.
backend = get_sherpa_dictation_backend(spec.id)
# Build the recognizer off the event loop if it isn't warm yet
# (download-on-first-use + ONNX session init can take a moment); status
# frames keep the widget honest.
if not await _sherpa_load_with_status(websocket, backend, spec):
return
rec = backend._rec
stream = rec.create_stream()
last_partial = ""
committed: list[str] = [] # finalized utterances this session
client_disconnected = False
async def _send(payload) -> bool:
nonlocal client_disconnected
if client_disconnected:
return False
try:
await websocket.send_json(payload)
return True
except Exception:
client_disconnected = True
return False
def _decode_after_feed(pcm: bytes):
"""Blocking: feed one PCM frame, decode, return (text, is_endpoint).
Runs in a thread so the ONNX work never blocks the event loop."""
samples = _pcm16_to_f32(pcm)
if len(samples):
stream.accept_waveform(pcm_sr, samples)
while rec.is_ready(stream):
rec.decode_stream(stream)
endpoint = rec.is_endpoint(stream)
text = (rec.get_result(stream) or "").strip()
return text, endpoint
def _flush_final():
"""Blocking: pad + drain the stream for the trailing utterance."""
tail = np.zeros(int(0.5 * pcm_sr), dtype=np.float32)
stream.accept_waveform(pcm_sr, tail)
stream.input_finished()
while rec.is_ready(stream):
rec.decode_stream(stream)
return (rec.get_result(stream) or "").strip()
try:
while True:
kind, pcm = await _recv_pcm_frame(websocket, aec)
if kind == "eof":
break
if kind == "skip":
continue
text, endpoint = await asyncio.to_thread(_decode_after_feed, pcm)
if endpoint:
# Commit this utterance (polished — it gets pasted); reset
# for the next one.
text = polish_text(text)
if text:
committed.append(text)
await _send({"type": "final", "text": text,
"segments": [{"start": 0.0, "end": None, "text": text}],
"language": "auto", "engine": backend.id})
rec.reset(stream)
last_partial = ""
elif text and text != last_partial:
last_partial = text
await _send({"type": "partial", "text": text})
except WebSocketDisconnect:
client_disconnected = True
except Exception as e:
logger.warning("sherpa streaming loop ended: %s", e)
client_disconnected = True
# Drain the trailing (un-endpointed) utterance on EOF.
try:
tail_text = await asyncio.to_thread(_flush_final)
except Exception as e:
logger.debug("sherpa streaming flush failed: %s", e)
tail_text = ""
tail_text = polish_text(tail_text)
if tail_text and tail_text != (committed[-1] if committed else None):
committed.append(tail_text)
# Pieces are already polished; the join is too (polish is idempotent).
full = " ".join(t for t in committed if t).strip()
segments = [{"start": 0.0, "end": None, "text": t} for t in committed if t]
if not client_disconnected:
if full:
# Hard-bounded refinement (~4s): never delays this summary `final`
# beyond OMNIVOICE_REFINE_TIMEOUT_S even with a dead LLM endpoint.
try:
from services.refinement import maybe_refine_async
refined = await maybe_refine_async(full)
except Exception:
refined = None
payload = {"type": "final", "text": full, "segments": segments,
"language": "auto", "engine": backend.id}
if refined and refined != full:
payload["refined_text"] = refined
await _send(payload)
else:
await _send({"type": "final", "text": "", "segments": [],
"language": "auto", "engine": backend.id})
try:
await websocket.close()
except Exception:
pass
async def _run_sherpa_offline(websocket: WebSocket, spec):
"""Offline-kind sherpa model with live partials, utterance-windowed.
Raw PCM accumulates in a *live* buffer holding only the current
(uncommitted) utterance. Every ~800ms the live window is re-decoded for a
``partial``; when the trailing ~0.6s of it fall below the RMS floor the
utterance is committed decoded once more, flushed as a ``final``, and
its samples dropped so per-partial cost is bounded by one utterance
(not the whole session) and sentences commit as the user pauses instead
of only at EOF."""
from services.asr_backend import get_sherpa_dictation_backend
pcm_sr, aec = await _sherpa_session(websocket)
logger.info("sherpa offline dictation: model=%s sr=%d aec=%s",
spec.id, pcm_sr, bool(aec))
# Shared, per-model warm backend (#888) — built once, reused per session.
backend = get_sherpa_dictation_backend(spec.id)
if not await _sherpa_load_with_status(websocket, backend, spec):
return
buf = bytearray() # live (uncommitted) PCM only
committed: list[str] = [] # polished utterances already flushed
last_partial = ""
running = True
client_disconnected = False
last_audio = time.monotonic()
# Trailing-silence gate window, in bytes of int16 mono PCM.
sil_bytes = max(2, int(SHERPA_OFFLINE_SILENCE_S * pcm_sr) * 2)
async def _send(payload) -> bool:
nonlocal client_disconnected
if client_disconnected:
return False
try:
await websocket.send_json(payload)
return True
except Exception:
client_disconnected = True
return False
def _rms(pcm: bytes) -> float:
samples = _pcm16_to_f32(pcm)
if not len(samples):
return 0.0
return float((samples * samples).mean() ** 0.5)
def _decode_window(pcm: bytes) -> str:
samples = _pcm16_to_f32(pcm)
if not len(samples):
return ""
return backend._decode_offline(samples, pcm_sr)
async def receive():
nonlocal running, client_disconnected, last_audio
try:
while running:
kind, pcm = await _recv_pcm_frame(websocket, aec)
if kind == "eof":
running = False
break
if kind == "skip":
continue
buf.extend(pcm)
last_audio = time.monotonic()
except WebSocketDisconnect:
client_disconnected = True
running = False
except Exception as e:
logger.debug("sherpa offline receive ended: %s", e)
running = False
async def _commit(snapshot: bytes):
"""Finalize one utterance: decode it off-thread, flush a polished
`final`, drop its samples from the live buffer. `receive()` may
append while we decode only the snapshot's prefix is dropped."""
nonlocal last_partial
try:
text = await asyncio.to_thread(_decode_window, snapshot)
except Exception as e:
logger.debug("sherpa offline commit decode failed: %s", e)
return
del buf[:len(snapshot)]
last_partial = ""
text = polish_text(text)
if text:
committed.append(text)
await _send({"type": "final", "text": text,
"segments": [{"start": 0.0, "end": None, "text": text}],
"language": "auto", "engine": backend.id})
async def partials():
nonlocal last_partial, running
while running:
await asyncio.sleep(SHERPA_OFFLINE_PARTIAL_S)
if not running or len(buf) < 2000:
continue
snapshot = bytes(buf)
if len(snapshot) > sil_bytes and \
_rms(snapshot[-sil_bytes:]) < SHERPA_OFFLINE_RMS_FLOOR:
if _rms(snapshot[:-sil_bytes]) >= SHERPA_OFFLINE_RMS_FLOOR:
await _commit(snapshot)
else:
# Pure silence — drop it (keep the gate window for
# continuity) so a long pause can't grow the buffer.
del buf[:len(snapshot) - sil_bytes]
continue
try:
text = await asyncio.to_thread(_decode_window, snapshot)
except Exception as e:
logger.debug("sherpa offline partial failed: %s", e)
continue
if text and text != last_partial:
last_partial = text
await _send({"type": "partial", "text": text})
recv_task = asyncio.create_task(receive())
part_task = asyncio.create_task(partials())
await asyncio.wait([recv_task, part_task], return_when=asyncio.FIRST_COMPLETED)
running = False
for t in (recv_task, part_task):
if not t.done():
t.cancel()
try:
await t
except (asyncio.CancelledError, Exception):
pass
# Drain the trailing (un-committed) utterance on EOF.
try:
tail = await asyncio.to_thread(_decode_window, bytes(buf))
except Exception as e:
logger.error("sherpa offline final failed: %s", e)
tail = ""
tail = polish_text(tail)
if tail:
committed.append(tail)
# Pieces are already polished; the join is too (polish is idempotent).
full = " ".join(committed).strip()
segments = [{"start": 0.0, "end": None, "text": t} for t in committed]
if not client_disconnected:
payload = {"type": "final", "text": full, "segments": segments,
"language": "auto", "engine": backend.id}
if full:
# Hard-bounded refinement (~4s) — never delays the `final`.
try:
from services.refinement import maybe_refine_async
refined = await maybe_refine_async(full)
if refined and refined != full:
payload["refined_text"] = refined
except Exception:
pass
await _send(payload)
try:
await websocket.close()
except Exception:
pass
async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None) -> str:
async def _transcribe_buffer(chunks: list[bytes]) -> str:
"""Quick partial transcription of the current audio buffer."""
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
tmp = _chunks_to_wav(chunks)
if tmp is None:
return ""
try:
from services.model_manager import _gpu_pool
from services.asr_backend import get_capture_asr_backend, run_transcribe_guarded
from services.asr_backend import get_capture_asr_backend
def _run():
backend = get_capture_asr_backend()
result = backend.transcribe(tmp, word_timestamps=False)
return result.get("text", "")
# Bound dictation transcribes (#730): a wedged whisperx/CTranslate2 call
# must not hold its GPU-pool worker forever and starve TTS / other ASR
# into a "can't reach backend"; on timeout the pool is reset to recover.
text = await run_transcribe_guarded(_gpu_pool, _run, what="Dictation")
loop = asyncio.get_running_loop()
text = await loop.run_in_executor(_gpu_pool, _run)
return text.strip()
finally:
try:
@@ -787,16 +223,16 @@ async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None)
pass
async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = None) -> dict:
async def _transcribe_buffer_full(chunks: list[bytes]) -> dict:
"""Full transcription with timing info for the final result."""
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
tmp = _chunks_to_wav(chunks)
if tmp is None:
return {"text": "", "segments": [], "language": "unknown",
"duration_s": 0, "transcription_time_s": 0, "engine": "none"}
try:
from services.model_manager import _gpu_pool
from services.asr_backend import get_capture_asr_backend, run_transcribe_guarded
from services.asr_backend import get_capture_asr_backend
def _run():
backend = get_capture_asr_backend()
@@ -809,12 +245,6 @@ async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = N
if not full_text and segments:
full_text = " ".join(s.get("text", "") for s in segments).strip()
# Wave 1.1: strip Whisper hallucination loops from the final
# text (the string that gets auto-pasted). Segments keep the
# raw recognition so their timings stay truthful.
from services.refinement import collapse_repetitive_artifacts
full_text = collapse_repetitive_artifacts(full_text)
duration = max((s.get("end", 0) for s in segments), default=0.0)
return {
@@ -831,9 +261,8 @@ async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = N
"engine": backend.id,
}
# Bounded + pool-resetting on timeout (#730), same rationale as the
# partial path above.
return await run_transcribe_guarded(_gpu_pool, _run, what="Dictation")
loop = asyncio.get_running_loop()
return await loop.run_in_executor(_gpu_pool, _run)
finally:
try:
os.unlink(tmp)
+3 -9
View File
@@ -258,19 +258,13 @@ async def community_use(item_id: str, name: Optional[str] = Query(None)):
raise HTTPException(status_code=503, detail=f"Couldn't add this voice right now. Error: {e}")
try:
# A community "preset" is a synthetic designed voice (rendered from an
# instruct string) → kind='design'; a "voice" carries a real reference
# clip → kind='clone'. Setting kind makes the persona-gallery
# synthetic-only gating work (§R3) instead of defaulting all imports to
# 'clone'.
kind = "design" if item["type"] == "preset" else "clone"
with db_conn() as conn:
conn.execute(
"INSERT INTO voice_profiles "
"(id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at, kind) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
"(id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(profile_id, profile_name, audio_filename, ref_text, instruct,
item.get("language", "Auto"), None, item["id"], time.time(), kind),
item.get("language", "Auto"), None, item["id"], time.time()),
)
except Exception:
with __import__("contextlib").suppress(OSError):
-35
View File
@@ -1,35 +0,0 @@
"""Voice-design "describe your voice" API (issue #317).
Maps a free-text voice description onto the existing design parameter space
via the deterministic keyword mapper in ``core.describe_voice``. Pure CPU +
stdlib no model, no network so it imports and responds instantly in any
environment, including test/CI without model weights.
"""
from __future__ import annotations
from fastapi import APIRouter
from pydantic import BaseModel, Field
from core.describe_voice import parse_description
router = APIRouter()
class DescribeRequest(BaseModel):
description: str = Field(default="", max_length=2000)
@router.post("/design/describe")
def describe_voice(req: DescribeRequest) -> dict:
"""Parse a free-text description into design attrs + a validator-safe instruct.
Response shape::
{
"attrs": {"Gender": "female", "Age": "elderly", ... or "Auto"},
"instruct": "female, elderly, low pitch, british accent",
"matched": [{"category": "Age", "token": "elderly", "phrase": "elderly"}, ...],
"unmatched": ["slightly raspy"]
}
"""
return parse_description(req.description)
-127
View File
@@ -1,127 +0,0 @@
"""
Dictation router sherpa-onnx live-dictation engine.
Exposes the seven sherpa-onnx dictation models and the dictation prefs the
frontend dictation UI binds to.
GET /dictation/models the 7 models + install state (frontend model list)
GET /dictation/prefs { enabled, mode, model_id }
POST /dictation/prefs persist any subset of those prefs
Install state reuses the same HF-cache check the model store uses, so a model
shown "installed" here is the same snapshot the backend will load.
Prefs are stored in the shared ``prefs.json`` store under the ``dictation.*``
namespace (``dictation.enabled``, ``dictation.mode``, ``dictation.model_id``),
mirroring how the ASR/TTS engine picks persist.
"""
from __future__ import annotations
import logging
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from typing import Optional
from api.dependencies import require_loopback
from core import prefs
from services import sherpa_dictation as sd
router = APIRouter()
logger = logging.getLogger("omnivoice.dictation")
# Pref keys (the binding contract — the frontend writes exactly these).
PREF_ENABLED = "dictation.enabled"
PREF_MODE = "dictation.mode"
PREF_MODEL_ID = "dictation.model_id"
_DEFAULT_ENABLED = True
_DEFAULT_MODE = "toggle"
_VALID_MODES = ("toggle", "hold")
def _read_prefs() -> dict:
mid = prefs.get(PREF_MODEL_ID, sd.DEFAULT_MODEL_ID)
if not sd.is_sherpa_model(mid):
mid = sd.DEFAULT_MODEL_ID
mode = prefs.get(PREF_MODE, _DEFAULT_MODE)
if mode not in _VALID_MODES:
mode = _DEFAULT_MODE
return {
"enabled": bool(prefs.get(PREF_ENABLED, _DEFAULT_ENABLED)),
"mode": mode,
"model_id": mid,
}
@router.get("/dictation/models", dependencies=[Depends(require_loopback)])
def list_dictation_models():
"""The seven sherpa-onnx dictation models + install state.
Each entry: id, repo_id, label, tag ("offline"|"streaming"), recommended,
size_gb, languages, kind, and install state (installed/installing). The
``installed`` flag is computed from the same HF cache the model store reads,
so it matches the model-store row state.
"""
available, reason = sd.sherpa_available()
out = []
for spec in sd.list_specs():
out.append({
"id": spec.id,
"repo_id": spec.repo_id,
"label": spec.label,
"tag": spec.tag,
"recommended": spec.recommended,
"size_gb": spec.size_gb,
"languages": spec.languages,
"kind": spec.kind,
"installed": sd.is_installed(spec),
})
return {
"models": out,
"engine_available": available,
"engine_reason": None if available else reason,
"default_model_id": sd.DEFAULT_MODEL_ID,
}
@router.get("/dictation/prefs", dependencies=[Depends(require_loopback)])
def get_dictation_prefs():
return _read_prefs()
class DictationPrefsUpdate(BaseModel):
enabled: Optional[bool] = None
mode: Optional[str] = None
model_id: Optional[str] = None
@router.post("/dictation/prefs", dependencies=[Depends(require_loopback)])
def set_dictation_prefs(req: DictationPrefsUpdate):
"""Persist any subset of the dictation prefs. Validates ``mode`` and
``model_id`` so a bad value can't wedge the capture engine."""
if req.mode is not None:
if req.mode not in _VALID_MODES:
raise HTTPException(
status_code=400,
detail=f"mode must be one of {_VALID_MODES}",
)
prefs.set_(PREF_MODE, req.mode)
if req.model_id is not None:
if not sd.is_sherpa_model(req.model_id):
raise HTTPException(
status_code=400,
detail=f"unknown dictation model_id {req.model_id!r}",
)
# Normalise to the canonical dictation id (accept repo_id too).
prefs.set_(PREF_MODEL_ID, sd.get_spec(req.model_id).id)
if req.enabled is not None:
prefs.set_(PREF_ENABLED, bool(req.enabled))
# Rebuild the cached capture singleton so the change takes effect at once.
try:
from services import asr_backend
asr_backend._capture_backend = None
asr_backend._capture_backend_key = None
except Exception:
pass
return _read_prefs()
+34 -198
View File
@@ -16,7 +16,6 @@ from core.tasks import task_manager
from core import event_bus
from schemas.requests import DubIngestUrlRequest
from services.model_manager import get_model, _gpu_pool, _cpu_pool, get_diarization_pipeline, offload_tts_for_asr, restore_tts_after_asr
from services.asr_backend import ASRTimeoutError, reset_pool_after_wedge, run_transcribe_guarded
from services.audio_io import _safe_soundfile_write
from services.ffmpeg_utils import find_ffmpeg
from services.segmentation import (
@@ -24,18 +23,13 @@ from services.segmentation import (
assign_speakers_from_diarization,
assign_speakers_from_turns,
assign_speakers_heuristic,
resplit_segments_by_diarization,
resplit_segments_by_turns,
_words_from_whisper,
clean_up_segments,
)
from services.onset_align import snap_segment_starts
from services import dub_pipeline
router = APIRouter()
logger = logging.getLogger("omnivoice.api")
# ── Legacy-name aliases to services/dub_pipeline.py ────────────────────────
# Phase 2.4 moved the business logic into a service. Other routers
# (dub_generate, dub_translate, dub_export) + internal call sites below still
@@ -365,11 +359,6 @@ async def dub_ingest_url(req: DubIngestUrlRequest):
TRANSCRIBE_CHUNK_S = float(os.environ.get("OMNIVOICE_TRANSCRIBE_CHUNK_S", "30.0"))
TRANSCRIBE_CHUNK_TIMEOUT_S = float(os.environ.get("OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S", "120.0"))
#: How many times to attempt each transcribe chunk before giving up on it. A
#: transient wedge (esp. the first chunk, where whisperx cold-loads its model)
#: shouldn't silently drop that whole window — retry once on a fresh pool so the
#: transcript doesn't come back "missing the beginning".
_CHUNK_TRANSCRIBE_ATTEMPTS = max(1, int(os.environ.get("OMNIVOICE_TRANSCRIBE_CHUNK_ATTEMPTS", "2")))
_sse_event = dub_pipeline.sse_event
@@ -377,11 +366,7 @@ _prep_event_helper = dub_pipeline.prep_event # alias; we keep the module-local
@router.get("/dub/transcribe-stream/{job_id}")
async def dub_transcribe_stream(
job_id: str,
num_speakers: Optional[int] = None,
per_segment_refs: bool = True,
):
async def dub_transcribe_stream(job_id: str, num_speakers: Optional[int] = None):
"""Stream per-chunk segments via SSE, then emit diarized final pass.
Pre-flight checks (missing job, missing audio, ASR not loaded) are emitted
@@ -435,22 +420,10 @@ async def dub_transcribe_stream(
try:
# The PyTorch-Whisper backend lazily builds its own pipeline
# when no preloaded `_asr_pipe` is present (issue #255), so it
# no longer needs OMNIVOICE_PRELOAD_TTS_ASR=1.
# no longer needs OMNIVOICE_PRELOAD_TTS_ASR=1 — don't reject it
# here; any load failure surfaces per-chunk with a real cause.
_asr_backend = get_active_asr_backend(asr_pipe=getattr(_model, "_asr_pipe", None))
# Eagerly load the model HERE so a real load failure (e.g.
# WhisperX: missing weights, CTranslate2/cuDNN mismatch, the
# torch-2.6 weights-only VAD regression) surfaces once, with
# its actual cause, as a clean preflight `error` event —
# instead of being buried in N cryptic per-chunk failures
# and retried on every chunk (#578). Run in a thread so the
# (blocking) load doesn't stall the event loop.
_ensure_loaded = getattr(_asr_backend, "ensure_loaded", None)
if callable(_ensure_loaded):
await asyncio.get_running_loop().run_in_executor(
_gpu_pool, _ensure_loaded
)
except Exception as e:
logger.exception("transcribe preflight: ASR load failed (job=%s)", job_id)
from core.failure import build_failure
f = build_failure(e, stage="transcribe-preflight", include_diagnostic=False)
preflight_error = "ASR backend initialization failed: " + f["reason"] + (
@@ -458,16 +431,9 @@ async def dub_transcribe_stream(
)
scene_cuts = job.get("scene_cuts") or []
async def _gen_body():
async def gen():
if preflight_error:
# Always follow a terminal `error` with `done` so the stream closes
# via a named event, not a raw connection drop. A bare error+close
# races the browser's native EventSource error (which carries no
# `data`); if that native error wins, the client falls back to the
# misleading generic "stream dropped … ASR backend failed" message
# and the real cause (in `detail`) is lost (#578).
yield _sse_event("error", {"detail": preflight_error, "retryable": True})
yield _sse_event("done", {})
yield _sse_event("error", {"detail": preflight_error})
return
import math
import tempfile
@@ -482,9 +448,7 @@ async def dub_transcribe_stream(
try:
audio_np, sr = await loop.run_in_executor(_cpu_pool, _load)
except Exception as e:
# Terminal error → always emit `done` (see preflight note, #578).
yield _sse_event("error", {"detail": f"audio load failed: {e}", "retryable": True})
yield _sse_event("done", {})
yield _sse_event("error", {"detail": f"audio load failed: {e}"})
return
total = float(len(audio_np)) / float(sr) if sr else 0.0
@@ -501,9 +465,6 @@ async def dub_transcribe_stream(
logger.warning("offload_tts_for_asr failed (continuing): %s", e)
all_segments: list[dict] = []
# Words (global-timeline) retained so diarization can re-split a segment
# that spans two speakers' turns at the word boundary (#486).
all_words: list = []
detected_lang = None
next_seg_id = 0
chunk_errors: list[str] = []
@@ -555,59 +516,31 @@ async def dub_transcribe_stream(
logger.exception("chunk transcribe failed (backend=%s)", _asr_backend.id)
return {"chunks": [], "language": None, "error": str(e)}
# Retry a failed/timed-out chunk once on a fresh pool before giving
# up. Otherwise a transient wedge on the FIRST chunk (whisperx often
# cold-loads its model there, the #730 hang) drops that whole window
# and the transcript is "missing the beginning, only middle+end".
# The retry reuses the same audio window, so a recovered chunk fills
# the hole instead of leaving silent gaps.
part = None
for _attempt in range(1, _CHUNK_TRANSCRIBE_ATTEMPTS + 1):
# A wedged chunk gets the SAME guarded-timeout + pool-reset
# semantics as the whole-file paths (#730/#851):
# run_transcribe_guarded bounds the call, abandons the poisoned
# pool so the retry (and any concurrent TTS work) gets a fresh
# worker, and raises the actionable ASRTimeoutError. Run it as
# a task and poll so we can keep yielding pings — the
# EventSource connection drops without them.
pool_reset_by_guard = False
task = asyncio.ensure_future(run_transcribe_guarded(
_gpu_pool, _transcribe_chunk,
what=f"Dub chunk {i + 1}/{chunks_n}",
timeout=TRANSCRIBE_CHUNK_TIMEOUT_S,
timeout_env="OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S",
))
try:
# wait_for in a loop to yield pings so the EventSource connection doesn't drop
fut = loop.run_in_executor(_gpu_pool, _transcribe_chunk)
waited = 0.0
part = None
while True:
done, _pending = await asyncio.wait({task}, timeout=5.0)
done, pending = await asyncio.wait([fut], timeout=5.0)
if done:
part = done.pop().result()
break
yield _sse_event("ping", {})
try:
part = task.result()
except ASRTimeoutError as e:
# The guard already reset the pool; keep the actionable
# message (it names the durable fixes, and — after repeated
# timeouts — the crash-isolated engine escape hatch).
pool_reset_by_guard = True
logger.error(
"Transcribe chunk %d/%d timed out after %.0fs (attempt %d/%d, job=%s)",
i + 1, chunks_n, TRANSCRIBE_CHUNK_TIMEOUT_S, _attempt,
_CHUNK_TRANSCRIBE_ATTEMPTS, job_id,
)
part = {"chunks": [], "language": None, "error": str(e)}
# Success → keep it. Failure/timeout → retry once on a fresh
# worker (the internal _transcribe_chunk except returns an
# error-part; the timeout path already reset the pool).
if part is not None and not part.get("error"):
break
if _attempt < _CHUNK_TRANSCRIBE_ATTEMPTS:
logger.warning(
"Retrying transcribe chunk %d/%d after failure/timeout (next attempt %d/%d, job=%s)",
i + 1, chunks_n, _attempt + 1, _CHUNK_TRANSCRIBE_ATTEMPTS, job_id,
)
if not pool_reset_by_guard:
reset_pool_after_wedge(
_gpu_pool, what=f"Dub chunk {i + 1}/{chunks_n}")
waited += 5.0
if waited >= TRANSCRIBE_CHUNK_TIMEOUT_S:
# Re-raise TimeoutError if we exceed the overall limit
raise asyncio.TimeoutError()
except asyncio.TimeoutError:
logger.error(
"Transcribe chunk %d/%d timed out after %.0fs (job=%s)",
i + 1, chunks_n, TRANSCRIBE_CHUNK_TIMEOUT_S, job_id,
)
part = {
"chunks": [], "language": None,
"error": f"Chunk {i+1} timed out after {TRANSCRIBE_CHUNK_TIMEOUT_S:.0f}s — "
f"ASR backend may be stuck. Try restarting the server.",
}
if part.get("error"):
chunk_errors.append(part["error"])
logger.warning("Chunk %d/%d error: %s", i + 1, chunks_n, part["error"])
@@ -615,21 +548,6 @@ async def dub_transcribe_stream(
detected_lang = part["language"]
asr_speaker_turns.extend(part.get("speaker_turns") or [])
chunk_segs = segment_transcript(part, duration=t1, scene_cuts=scene_cuts)
# Same word source segment_transcript used (already global-timeline),
# kept for the post-diarization speaker re-split (#486).
try:
all_words.extend(_words_from_whisper(part))
except Exception:
pass
# #280: Whisper often stretches a segment's start back over
# leading music/silence (classic case: speech begins at 0:03,
# transcript says 0.0 → the dub plays 3 s early). Snap starts
# forward to the actual speech onset. `audio_np` is the same
# track ASR ran on — vocals.wav when Demucs succeeded.
try:
snap_segment_starts(chunk_segs, audio_np, sr)
except Exception as e:
logger.warning("onset alignment skipped for chunk %d: %s", i, e)
chunk_segs = assign_speakers_heuristic(chunk_segs)
for s in chunk_segs:
s["id"] = f"s{next_seg_id:05x}"
@@ -699,10 +617,7 @@ async def dub_transcribe_stream(
# use its speaker turns directly and skip pyannote entirely (#182).
if asr_speaker_turns:
logger.info("Using inline ASR diarization (%d turns); skipping pyannote.", len(asr_speaker_turns))
assigned = assign_speakers_from_turns(all_segments, asr_speaker_turns)
# #486: split any segment that spans two speakers' turns at the
# word boundary (single-speaker segments pass through unchanged).
return resplit_segments_by_turns(assigned, all_words, asr_speaker_turns), None
return assign_speakers_from_turns(all_segments, asr_speaker_turns), None
from services.model_manager import (
DIARIZATION_ERR_LICENSE,
@@ -779,10 +694,7 @@ async def dub_transcribe_stream(
diar = diar_pipe(asr_audio_target, num_speakers=num_speakers)
else:
diar = diar_pipe(asr_audio_target)
assigned = assign_speakers_from_diarization(all_segments, diar)
# #486: split any segment that spans two speakers' turns at the
# word boundary (single-speaker segments pass through unchanged).
return resplit_segments_by_diarization(assigned, all_words, diar), None
return assign_speakers_from_diarization(all_segments, diar), None
except Exception as e:
logger.error(f"Diarization failed: {e}")
# Mid-run failure — classify against the same sentinels so a
@@ -847,61 +759,16 @@ async def dub_transcribe_stream(
clones = done.pop().result()
break
yield _sse_event("ping", {})
# Wave 3.2: per-segment clone refs. Cut each long-enough segment's
# own reference from the vocals so the dub of each line matches the
# prosody of its source line. Short lines fall back to the
# per-speaker clone below. Default on; the user can force
# per-speaker by disabling it (job["per_segment_refs"]).
seg_clones = {}
job["per_segment_refs"] = per_segment_refs
if per_segment_refs:
try:
from services.speaker_clone import extract_segment_refs
seg_ids_for_clone = [s.get("id", i) for i, s in enumerate(final_segs)]
seg_clones = await loop.run_in_executor(
_cpu_pool, lambda: extract_segment_refs(
vocals_for_clone, final_segs,
os.path.dirname(vocals_for_clone),
seg_ids=seg_ids_for_clone,
),
)
if seg_clones:
job["segment_clones"] = seg_clones
except Exception as e:
logger.warning("per-segment clone refs skipped: %s", e)
if clones or seg_clones:
if clones:
job["speaker_clones"] = clones
# Default each segment's profile_id to its detected speaker's
# auto-clone — but only if the user hasn't already assigned
# something. (#486)
#
# We prefer the UI-visible `auto:{speaker}` id over the
# per-segment `auto-seg:{id}` id even when a per-segment ref
# exists, because the dub editor's Voice dropdown only renders
# `auto:` options ("From Video → Speaker N"). An `auto-seg:`
# value matches no <option>, so the row silently read
# "Default" while the speaker was actually bound — exactly the
# reported bug. The per-segment ref is NOT lost: dub_generate's
# `auto:` branch transparently prefers this segment's own
# per-segment ref (job["segment_clones"][seg_id]) when present,
# so a row shown as "Speaker 1" still clones from its own line
# when that line is long enough.
if clones:
job["speaker_clones"] = clones
# Default each segment's profile_id to its speaker's auto-clone,
# but only if the user hasn't already assigned something.
for s in final_segs:
if s.get("profile_id"):
continue
spk = s.get("speaker_id") or "Speaker 1"
if spk in clones:
s["profile_id"] = auto_profile_id(spk)
continue
# No per-speaker clone for this speaker (too little usable
# audio overall) but this single line was long enough for
# its own ref — fall back to the per-segment id. The editor
# can't render it, but generation still clones correctly.
sid = str(s.get("id", ""))
if sid and sid in seg_clones:
s["profile_id"] = f"auto-seg:{sid}"
except Exception as e:
logger.warning("speaker_clone extraction skipped: %s", e)
@@ -930,25 +797,6 @@ async def dub_transcribe_stream(
})
yield _sse_event("done", {})
async def gen():
# Terminal-event guard (#516): the SSE stream must NEVER close without a
# terminal event. Any unanticipated exception in the body (e.g. an ASR
# load that escapes the per-chunk handler) previously dropped the
# connection, which the frontend can only report as "stream dropped,
# likely ASR failed" — hiding the real cause. Emit a structured `error`
# (with the actionable hint from build_failure) then `done`, so the user
# sees the real failure + a Retry instead of a silent disconnect.
try:
async for ev in _gen_body():
yield ev
except Exception as e: # noqa: BLE001 — last-resort stream finalizer
logger.exception("transcribe stream crashed (job=%s)", job_id)
from core.failure import build_failure
f = build_failure(e, stage="transcribe", include_diagnostic=False)
detail = f["reason"] + (f"{f['hint']}" if f.get("hint") else "")
yield _sse_event("error", {"detail": detail, "retryable": True})
yield _sse_event("done", {})
return StreamingResponse(
gen(),
media_type="text/event-stream",
@@ -1014,14 +862,6 @@ async def dub_transcribe(job_id: str):
scene_cuts = job.get("scene_cuts") or []
segments = segment_transcript(result, duration=job.get("duration", 0.0), scene_cuts=scene_cuts)
# #280: snap segment starts forward to the actual speech onset so the
# dub doesn't begin seconds before the original speaker does.
try:
audio_for_onset, onset_sr = sf.read(asr_audio_target, dtype="float32")
snap_segment_starts(segments, audio_for_onset, onset_sr)
except Exception as e:
logger.warning("onset alignment skipped: %s", e)
diar_pipe = get_diarization_pipeline()
if diar_pipe:
try:
@@ -1052,11 +892,7 @@ async def dub_transcribe(job_id: str):
try:
loop = asyncio.get_running_loop()
try:
# Bound the whole-file transcribe (#730): a wedged whisperx/CTranslate2
# call would otherwise hold its GPU-pool worker forever and starve
# every other request into a "can't reach backend". run_transcribe_guarded
# also resets the pool on timeout so capacity is restored.
segments_result = await run_transcribe_guarded(_gpu_pool, _transcribe, what="Dub")
segments_result = await loop.run_in_executor(_gpu_pool, _transcribe)
except asyncio.CancelledError:
job["aborted"] = True
raise
+99 -653
View File
@@ -1,7 +1,5 @@
import os
import io
import re
import json
import time
import uuid
import asyncio
@@ -14,13 +12,6 @@ from core.config import DUB_DIR, dub_seg_path
from core.tasks import task_manager
from api.routers.dub_core import _get_job
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
from services.video_retime import (
DRIFT_TOLERANCE_S,
RetimeError,
build_chunk_filter_graph,
expand_retime_chunks,
prepare_smart_fit_video,
)
router = APIRouter()
logger = logging.getLogger("omnivoice.api")
@@ -31,9 +22,6 @@ def _unique_stamp() -> str:
return f"{time.strftime('%Y%m%dT%H%M%S')}-{uuid.uuid4().hex[:8]}"
_SAFE_LANG = re.compile(r"^[A-Za-z0-9_-]{1,32}$")
def _native_save(source: str, destination: str, display_name: str, media_type: str):
"""Copy a generated export file to a user-chosen destination and return JSON."""
import shutil
@@ -170,23 +158,15 @@ async def dub_list_tracks(job_id: str):
return {"tracks": job.get("dubbed_tracks", {})}
def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool,
fitted_segments: "list[dict] | None" = None) -> str | None:
def _write_burn_srt(job: dict, exports_dir: str, stamp: str, dual: bool) -> str | None:
"""Build a temp SRT from job segments for use with ffmpeg's subtitles filter.
Returned path is already ffmpeg-filter-safe (plain ASCII basename under exports_dir).
Returns None if there are no segments to render.
``fitted_segments`` (Smart Fit): {id, start, end} cue records on the
fitted timeline when provided, cue times come from there instead of
the original ``job["segments"]`` timings, so burned subs track the
retimed video / fitted audio rather than the source timeline.
"""
segments = job.get("segments", [])
if not segments:
return None
if fitted_segments:
segments = _apply_fitted_times(segments, fitted_segments)
lines = []
for i, seg in enumerate(segments):
lines.append(str(i + 1))
@@ -234,14 +214,41 @@ def _build_video_stretch_filter_graph(
if not plan:
return "", in_label or f"[{video_input_idx}:v]"
# Chunk expansion + graph emission live in services.video_retime now so
# the Smart Fit batched pipeline shares the exact same boundary math.
# With default options the emitted graph is byte-identical to the
# original inline implementation.
chunks = expand_retime_chunks(plan, orig_dur)
chunks: list[tuple[float, float, float]] = [] # (a, b, ratio)
cursor = 0.0
for entry in plan:
a = float(entry["orig_start"])
b = float(entry["orig_end"])
if a > cursor + 1e-3:
chunks.append((cursor, a, 1.0)) # gap or pre-roll at native rate
ratio = float(entry["stretch_ratio"])
if b > a:
chunks.append((a, b, ratio))
cursor = max(cursor, b)
if orig_dur > cursor + 1e-3:
chunks.append((cursor, orig_dur, 1.0)) # tail at native rate
chunks = [(a, b, r) for (a, b, r) in chunks if b > a]
if not chunks:
return "", in_label or f"[{video_input_idx}:v]"
return build_chunk_filter_graph(chunks, in_label or f"[{video_input_idx}:v]")
src = in_label or f"[{video_input_idx}:v]"
parts: list[str] = []
labels: list[str] = []
# `split` lets us tap the same source stream once per chunk without re-
# decoding. setpts={ratio}*PTS slows down (ratio > 1) or speeds up
# (ratio < 1) each chunk; PTS-STARTPTS first to normalise the timestamp
# base after the trim.
split_labels = [f"[vsplit{idx}]" for idx in range(len(chunks))]
parts.append(f"{src}split={len(chunks)}{''.join(split_labels)}")
for idx, ((a, b, ratio), split_lbl) in enumerate(zip(chunks, split_labels)):
out_label = f"[vstr{idx}]"
labels.append(out_label)
parts.append(
f"{split_lbl}trim=start={a:.4f}:end={b:.4f},"
f"setpts=PTS-STARTPTS,setpts={ratio:.6f}*PTS{out_label}"
)
parts.append("".join(labels) + f"concat=n={len(chunks)}:v=1:a=0[vstretched]")
return ";".join(parts), "[vstretched]"
def _video_stretch_plan_for(job: dict, lang_code: str) -> dict | None:
@@ -257,76 +264,6 @@ def _video_stretch_plan_for(job: dict, lang_code: str) -> dict | None:
return entry
def _video_retime_plan_for(job: dict, lang_code: str) -> "tuple[str, dict] | None":
"""Resolve the video retime plan for ``lang_code`` across both keyspaces.
Returns ``(kind, entry)`` where kind is ``"stretch_video"`` (legacy
Mode B plans resolution byte-identical to ``_video_stretch_plan_for``)
or ``"smart_fit"`` (Phase A ``job["fit_plans"]`` entries, gated on the
track actually having been generated under smart_fit so a stale plan
from an earlier run can't retime a track re-generated under another
strategy). ``None`` when neither applies.
"""
legacy = _video_stretch_plan_for(job, lang_code)
if legacy is not None:
return "stretch_video", legacy
entry = (job.get("fit_plans") or {}).get(lang_code)
track = (job.get("dubbed_tracks") or {}).get(lang_code) or {}
if entry and entry.get("plan") and track.get("timing_strategy") == "smart_fit":
return "smart_fit", entry
return None
def _fitted_segments_for(job: dict, lang_code: "str | None") -> "list[dict] | None":
"""Fitted-timeline subtitle cues ({id, start, end}) for a Smart Fit
track, or None. Same staleness gate as ``_video_retime_plan_for``."""
if not lang_code:
return None
entry = (job.get("fit_plans") or {}).get(lang_code)
track = (job.get("dubbed_tracks") or {}).get(lang_code) or {}
if not entry or track.get("timing_strategy") != "smart_fit":
return None
fitted = entry.get("fitted_segments")
return fitted or None
def _apply_fitted_times(segments: list[dict], fitted: list[dict]) -> list[dict]:
"""Overlay fitted cue times onto subtitle segments (copies; non-destructive).
Matches by segment ``id``; when the fitted record carries no ids at all
(defensive), falls back to positional pairing. Segments without a match
keep their original timings.
"""
by_id = {str(f["id"]): f for f in fitted if f.get("id") is not None}
out: list[dict] = []
for i, seg in enumerate(segments):
cue = None
if seg.get("id") is not None:
cue = by_id.get(str(seg["id"]))
if cue is None and not by_id and i < len(fitted):
cue = fitted[i]
if cue is None:
out.append(seg)
continue
patched = dict(seg)
patched["start"] = float(cue["start"])
patched["end"] = float(cue["end"])
out.append(patched)
return out
def _burn_subs_allowed(retime_kind: "str | None") -> bool:
"""Subtitle burn-in combined with video retime.
Allowed for ``smart_fit`` (fitted cue records exist, and the burn pass
runs AFTER the retime graph so cues land on the retimed timeline) and
for plain exports. Still rejected for legacy ``stretch_video``, which
has no fitted-cue record cues would burn at original timestamps onto
a re-timed video and drift.
"""
return retime_kind != "stretch_video"
#: Audio export formats → ffmpeg codec args. Unknown formats fall back to
#: AAC/m4a so a bad request can never produce a broken command.
_AUDIO_FORMAT_CODECS: dict[str, list[str]] = {
@@ -375,11 +312,6 @@ async def dub_download(
dual: bool = Query(False, description="When burn_subs=1, render translated on top of italicised original."),
out_format: str = Query("m4a", description="Audio-only jobs (#119): output container — wav, m4a, mp3, or flac. Ignored for video jobs."),
):
# Strict allowlist on the path param BEFORE it reaches any filesystem
# path or ffmpeg argv (export dir, retime work path, slice paths). Real
# job ids are short uuid slices — alnum/hyphen/underscore only.
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", job_id):
raise HTTPException(status_code=400, detail="Invalid job id")
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
@@ -458,24 +390,18 @@ async def dub_download(
)
# Determine whether this export should drive video through a per-segment
# retime (legacy stretch_video Mode B, or Smart Fit). Retime is keyed off
# the default_track's plan because the video can only physically follow
# one timeline at a time. If multiple dub tracks are included, only the
# default_track is visually in sync — other tracks share the same
# (retimed) video. Single-track export is the supported common case.
retime_kind: "str | None" = None
retime_entry: "dict | None" = None
if default_track and default_track != "original":
_retime = _video_retime_plan_for(job, default_track)
if _retime:
retime_kind, retime_entry = _retime
stretch_entry = retime_entry if retime_kind == "stretch_video" else None
# Subtitle burn under legacy stretch_video would render cues at the
# original timestamps onto a re-timed video — they'd drift (no fitted-cue
# record exists for that mode). Skip the burn pass in that combo and log;
# the user can still export the SRT/VTT separately. Smart Fit DOES carry
# fitted cues, so burn+retime is allowed there (burn runs post-retime).
if not _burn_subs_allowed(retime_kind) and burn_subs:
# stretch graph (Mode B). Stretch is keyed off the default_track's plan
# because the video can only physically follow one timeline at a time.
# If multiple dub tracks are included and they were generated under
# stretch_video, only the default_track is visually in sync — other
# tracks share the same (stretched) video. Single-track export is the
# supported common case.
stretch_entry = _video_stretch_plan_for(job, default_track) if default_track and default_track != "original" else None
# Subtitle burn under stretch_video would render cues at the original
# timestamps onto a re-timed video — they'd drift. Skip the burn pass
# in that combo and log; the user can still export the SRT/VTT
# separately and the new-layout timing lives there.
if stretch_entry and burn_subs:
logger.warning(
"stretch_video + burn_subs is not supported in one pass; "
"skipping subtitle burn for job %s. Export the SRT/VTT separately.",
@@ -483,71 +409,11 @@ async def dub_download(
)
burn_subs = False
# Smart Fit: cue times come from the fitted timeline — that's where the
# dubbed audio actually sits, whether or not the video retime succeeds.
fitted_segments = _fitted_segments_for(job, default_track) if default_track and default_track != "original" else None
sub_path = _write_burn_srt(job, exports_dir, stamp, dual, fitted_segments=fitted_segments) if burn_subs else None
# ── Smart Fit video retime (two-tier) ─────────────────────────────────
# Tier 1 (≤48 chunks): single filter_complex graph inlined into the mux
# command below. Tier 2: batched slice renders joined by the concat
# demuxer into an intermediate file, muxed as an extra input. Failures
# fall back to an un-retimed export with a structured warning rather
# than failing the whole download.
retime_decision = None
retime_warning: "dict | None" = None
smart_track_dur = 0.0
if retime_kind == "smart_fit" and retime_entry:
smart_orig_dur = float(retime_entry.get("orig_duration") or job.get("duration") or 0.0)
smart_track_dur = float(
retime_entry.get("total_duration")
or (filtered_tracks.get(default_track) or {}).get("duration")
or 0.0
)
# A fresh export is a fresh user intent — clear any sticky abort flag
# from a previous /dub/abort so it can't kill this run's first batch.
job.pop("aborted", None)
# realpath-normalised + containment-checked inline at the sink (the
# file's established pattern — CodeQL does not track the guard
# through a helper's return value).
_base = os.path.realpath(DUB_DIR)
retime_work_path = os.path.realpath(
os.path.join(exports_dir, f"retimed_{stamp}.mp4")
)
if retime_work_path != _base and not retime_work_path.startswith(_base + os.sep):
raise HTTPException(status_code=400, detail="Invalid export path")
try:
retime_decision = await prepare_smart_fit_video(
job_id=job_id,
ffmpeg=ffmpeg,
video_path=video_path,
plan=retime_entry["plan"],
orig_dur=smart_orig_dur,
track_dur=smart_track_dur,
work_path=retime_work_path,
abort_check=lambda: bool(job.get("aborted")),
)
except Exception as e:
if (isinstance(e, RetimeError) and e.stage == "aborted") or job.get("aborted"):
raise HTTPException(status_code=409, detail="Export aborted")
from core.failure import build_failure
retime_warning = build_failure(e, stage="video-retime", include_diagnostic=False)
job["last_export_warning"] = {"type": "video_retime_fallback", **retime_warning}
logger.error(
"Smart Fit video retime failed for job %s — exporting "
"without per-segment retime: %s",
job_id.replace("\n", " ").replace("\r", " "), e,
)
sub_path = _write_burn_srt(job, exports_dir, stamp, dual) if burn_subs else None
cmd = [ffmpeg, "-i", video_path]
input_idx = 1
retimed_idx = None
if retime_decision is not None and retime_decision.mode == "file":
cmd += ["-i", retime_decision.file_path]
retimed_idx = input_idx
input_idx += 1
bg_audio = job.get("no_vocals_path") if preserve_bg else None
bg_idx = None
if bg_audio and os.path.exists(bg_audio) and filtered_tracks:
@@ -563,37 +429,9 @@ async def dub_download(
filter_parts: list[str] = []
video_map = "0:v:0"
video_reencode = False
if retime_decision is not None:
if retime_decision.mode == "filter":
filter_parts.append(retime_decision.graph)
video_map = retime_decision.label
video_reencode = True
else:
video_map = f"{retimed_idx}:v:0"
# Residual drift after the batched render (fps rounding): video
# shorter than the fitted track → freeze the last frame out to
# the track length. Rare — the predicted tail pad inside the
# render usually lands within tolerance.
residual = smart_track_dur - retime_decision.video_dur
if smart_track_dur and residual > DRIFT_TOLERANCE_S:
filter_parts.append(
f"[{retimed_idx}:v]tpad=stop_mode=clone:stop_duration={residual:.4f}[vtpad]"
)
video_map = "[vtpad]"
video_reencode = True
if sub_path:
esc = _ffmpeg_filter_escape(sub_path)
# Burn AFTER any retime so cues (already on the fitted timeline for
# Smart Fit) land on the retimed video. Without retime this reduces
# to the legacy `[0:v]subtitles=…[vsub]` graph.
if video_map.startswith("["):
sub_src = video_map
elif retimed_idx is not None:
sub_src = f"[{retimed_idx}:v]"
else:
sub_src = "[0:v]"
filter_parts.append(f"{sub_src}subtitles='{esc}'[vsub]")
filter_parts.append(f"[0:v]subtitles='{esc}'[vsub]")
video_map = "[vsub]"
if stretch_entry:
orig_dur = float(stretch_entry.get("orig_duration") or job.get("duration") or 0.0)
@@ -609,32 +447,10 @@ async def dub_download(
if include_original:
cmd += ["-map", "0:a:0"]
# Smart Fit drift absorption, audio side: when the retimed video runs
# longer than the fitted track (its tail passes through at 1.0× beyond
# the last cue, or encoder rounding), pad the dub-track chain with
# silence out to the video length so players don't end audio early.
apad_dur = 0.0
if (
retime_decision is not None
and smart_track_dur
and retime_decision.video_dur - smart_track_dur > DRIFT_TOLERANCE_S
):
apad_dur = retime_decision.video_dur
if bg_idx is not None:
for i, t in enumerate(tracks_to_process):
out_label = f"[aout{i}]"
chain = f"[{bg_idx}:a][{t['idx']}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2"
if apad_dur:
chain += f",apad=whole_dur={apad_dur:.4f}"
filter_parts.append(chain + out_label)
t["out_label"] = out_label
for t in tracks_to_process:
cmd += ["-map", t["out_label"]]
elif apad_dur:
for i, t in enumerate(tracks_to_process):
out_label = f"[aout{i}]"
filter_parts.append(f"[{t['idx']}:a]apad=whole_dur={apad_dur:.4f}{out_label}")
filter_parts.append(f"[{bg_idx}:a][{t['idx']}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2{out_label}")
t["out_label"] = out_label
for t in tracks_to_process:
cmd += ["-map", t["out_label"]]
@@ -645,11 +461,10 @@ async def dub_download(
if filter_parts:
cmd += ["-filter_complex", ";".join(filter_parts)]
# Burning subs or per-segment video retime both force a real video
# re-encode; stream-copy is viable when nothing touches the video
# filter chain — including the batched Smart Fit path, whose retimed
# intermediate is already encoded with these exact settings.
if sub_path or stretch_entry or video_reencode:
# Burning subs or per-segment video stretch both force a real video
# re-encode; stream-copy is only viable when nothing touches the video
# filter chain.
if sub_path or stretch_entry:
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
else:
cmd += ["-c:v", "copy"]
@@ -682,17 +497,17 @@ async def dub_download(
break
cmd += [f"-disposition:a:{target_idx}", "default"]
# When retiming (legacy stretch_video or Smart Fit) the video and audio
# durations should match within sub-frame precision, but `-shortest`
# can still cut off the trailing frame; let ffmpeg keep both streams.
# Otherwise keep the legacy `-shortest` so a slightly-overrunning track
# doesn't extend the mux past the video.
if not stretch_entry and retime_decision is None:
# In stretch_video mode the video and audio durations should match
# within sub-frame precision, but `-shortest` can still cut off the
# trailing frame; let ffmpeg keep both streams. Otherwise keep the
# legacy `-shortest` so a slightly-overrunning track doesn't extend
# the mux past the video.
if not stretch_entry:
cmd += ["-shortest"]
cmd += [output_path, "-y"]
try:
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0, job_id=job_id)
rc, _, stderr = await run_ffmpeg(cmd, timeout=1800.0)
if rc != 0:
raise Exception(stderr.decode(errors="replace") if stderr else "ffmpeg mux non-zero")
except asyncio.TimeoutError:
@@ -704,14 +519,6 @@ async def dub_download(
status_code=500,
detail=f"ffmpeg failed to combine video + dubbed audio: {e}. Verify ffmpeg is installed (`ffmpeg -version`), and check that every dubbed track file exists in the job folder.",
)
finally:
# The batched retime intermediate is a full re-encoded video — never
# leave it behind (success or failure; it's stamp-unique, no reuse).
if retime_decision is not None and retime_decision.mode == "file":
try:
os.remove(retime_decision.file_path)
except OSError as e:
logger.debug("cleanup remove failed: %s", e)
if not os.path.exists(output_path) or os.path.getsize(output_path) == 0:
raise HTTPException(status_code=500, detail="ffmpeg mux produced no output file")
@@ -721,22 +528,12 @@ async def dub_download(
safe_name = ''.join(c for c in base_name if c.isalnum() or c in '-_ ').strip() or 'output'
dl_name = f"dubbed_{safe_name}_{stamp}.mp4"
# Structured warning surface for the Smart Fit fallback ladder: header is
# a fixed ASCII token (FileResponse headers must be latin-1 safe); the
# full build_failure payload is persisted on the job for the UI to read.
extra_headers = {}
if retime_warning is not None:
extra_headers["X-Dub-Export-Warning"] = "video-retime-fallback"
if save_path:
result = _native_save(output_path, save_path, dl_name, media_type="video/mp4")
if retime_warning is not None:
result["warning"] = {"type": "video_retime_fallback", **retime_warning}
return result
return _native_save(output_path, save_path, dl_name, media_type="video/mp4")
return FileResponse(
output_path, media_type="video/mp4",
headers={"Content-Disposition": f'attachment; filename="{dl_name}"', **extra_headers},
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
)
@@ -771,21 +568,6 @@ async def dub_get_media(job_id: str):
ext = os.path.splitext(video_path)[1].lower()
return FileResponse(video_path, media_type=_MEDIA_TYPES.get(ext, "video/mp4"))
# One mux at a time per preview file. Without this, two overlapping requests
# (e.g. the <video> element remounting right after a re-dub) both ran ffmpeg
# against the same output path, and the mtime check below saw the half-written
# file as a valid cache — serving a truncated MP4 that left the player stuck
# loading forever (#281).
_preview_mux_locks: dict[str, asyncio.Lock] = {}
def _preview_lock(path: str) -> asyncio.Lock:
lock = _preview_mux_locks.get(path)
if lock is None:
lock = _preview_mux_locks.setdefault(path, asyncio.Lock())
return lock
@router.get("/dub/preview-video/{job_id}")
async def dub_preview_video(
job_id: str,
@@ -797,11 +579,6 @@ async def dub_preview_video(
Caches per lang+preserve_bg combination under exports/preview_{lang}_{bg}.mp4.
Cache is invalidated when the underlying dubbed track mtime is newer than the cache.
"""
# Strict allowlist on the path param BEFORE it reaches any filesystem
# path or ffmpeg argv (exports dir, preview/retime work paths) — same
# boundary check as dub_download.
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", job_id):
raise HTTPException(status_code=400, detail="Invalid job id")
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
@@ -822,92 +599,23 @@ async def dub_preview_video(
bg_audio = job.get("no_vocals_path") if preserve_bg else None
has_bg = bool(bg_audio and os.path.exists(bg_audio))
if not _SAFE_LANG.match(lang):
raise HTTPException(status_code=400, detail="Invalid lang")
# realpath-normalised + containment-checked inline BEFORE any filesystem
# access so the guard dominates every sink (the file's established
# pattern — see dub_preview_segment; CodeQL does not track the guard
# through a helper's return value).
_base = os.path.realpath(DUB_DIR)
exports_dir = os.path.realpath(os.path.join(_base, job_id, "exports"))
if not exports_dir.startswith(_base + os.sep):
raise HTTPException(status_code=400, detail="Invalid job id")
exports_dir = os.path.join(DUB_DIR, job_id, "exports")
os.makedirs(exports_dir, exist_ok=True)
bg_suffix = "bg" if (preserve_bg and has_bg) else "nobg"
preview_path = os.path.realpath(
os.path.join(exports_dir, f"preview_{lang}_{bg_suffix}.mp4")
)
if not preview_path.startswith(_base + os.sep):
raise HTTPException(status_code=400, detail="Invalid path")
preview_path = os.path.join(exports_dir, f"preview_{lang}_{bg_suffix}.mp4")
track_mtime = os.path.getmtime(track_path)
cache_ok = (
os.path.exists(preview_path)
and os.path.getsize(preview_path) > 0
and os.path.getmtime(preview_path) >= track_mtime
)
def _cache_ok() -> bool:
return (
os.path.exists(preview_path)
and os.path.getsize(preview_path) > 0
and os.path.getmtime(preview_path) >= track_mtime
)
async def _mux_preview():
# Mux into a temp file and os.replace() into place so a concurrent
# reader never sees a partially-written preview (#281: video stuck
# loading forever after a re-dub).
mux_path = preview_path + ".tmp.mp4"
if not cache_ok:
ffmpeg = find_ffmpeg()
# Resolve the same retime plan the download path uses (legacy
# stretch_video or Smart Fit) so the in-app preview matches export.
retime = _video_retime_plan_for(job, lang)
retime_kind, retime_entry = retime if retime else (None, None)
stretch_entry = retime_entry if retime_kind == "stretch_video" else None
retime_decision = None
smart_track_dur = 0.0
if retime_kind == "smart_fit" and retime_entry:
smart_orig_dur = float(retime_entry.get("orig_duration") or job.get("duration") or 0.0)
smart_track_dur = float(
retime_entry.get("total_duration") or track_info.get("duration") or 0.0
)
job.pop("aborted", None) # fresh user intent — clear sticky abort
# realpath-normalised + containment-checked inline at the sink
# (same pattern as preview_path above — _base is the realpath
# of DUB_DIR from the top of this endpoint).
retime_work_path = os.path.realpath(os.path.join(
exports_dir, f"preview_retimed_{lang}_{bg_suffix}.tmp.mp4",
))
if retime_work_path != _base and not retime_work_path.startswith(_base + os.sep):
raise HTTPException(status_code=400, detail="Invalid export path")
try:
retime_decision = await prepare_smart_fit_video(
job_id=job_id,
ffmpeg=ffmpeg,
video_path=video_path,
plan=retime_entry["plan"],
orig_dur=smart_orig_dur,
track_dur=smart_track_dur,
work_path=retime_work_path,
abort_check=lambda: bool(job.get("aborted")),
)
except Exception as e:
if (isinstance(e, RetimeError) and e.stage == "aborted") or job.get("aborted"):
raise HTTPException(status_code=409, detail="Preview aborted")
# Preview is best-effort: fall back to the un-retimed video
# rather than a black player. The export path surfaces the
# structured warning; here we just log.
retime_decision = None
logger.error(
"Smart Fit preview retime failed for job %s — previewing "
"without per-segment retime: %s",
job_id.replace("\n", " ").replace("\r", " "), e,
)
stretch_entry = _video_stretch_plan_for(job, lang)
cmd = [ffmpeg, "-i", video_path]
input_idx = 1
retimed_idx = None
if retime_decision is not None and retime_decision.mode == "file":
cmd += ["-i", retime_decision.file_path]
retimed_idx = input_idx
input_idx += 1
if preserve_bg and has_bg:
cmd += ["-i", bg_audio]
bg_idx = input_idx
@@ -917,13 +625,12 @@ async def dub_preview_video(
cmd += ["-i", track_path]
track_idx = input_idx
# Build filter graph. Under a retime plan we splice the source video
# into per-segment chunks, setpts each to match the dub audio
# Build filter graph. In stretch_video mode we splice the source
# video into per-segment chunks, setpts each to match the dub audio
# layout, and concat them — so audio plays at natural rate and the
# visuals follow. Otherwise we stream-copy video for speed.
filter_parts: list[str] = []
video_map = "0:v:0"
video_reencode = False
if stretch_entry:
orig_dur = float(stretch_entry.get("orig_duration") or job.get("duration") or 0.0)
graph, vlabel = _build_video_stretch_filter_graph(
@@ -932,175 +639,49 @@ async def dub_preview_video(
if graph:
filter_parts.append(graph)
video_map = vlabel
elif retime_decision is not None:
if retime_decision.mode == "filter":
filter_parts.append(retime_decision.graph)
video_map = retime_decision.label
video_reencode = True
else:
video_map = f"{retimed_idx}:v:0"
residual = smart_track_dur - retime_decision.video_dur
if smart_track_dur and residual > DRIFT_TOLERANCE_S:
filter_parts.append(
f"[{retimed_idx}:v]tpad=stop_mode=clone:stop_duration={residual:.4f}[vtpad]"
)
video_map = "[vtpad]"
video_reencode = True
# Smart Fit drift absorption (audio): silence-pad the dub chain out
# to the retimed video length so the preview doesn't end audio early.
apad_dur = 0.0
if (
retime_decision is not None
and smart_track_dur
and retime_decision.video_dur - smart_track_dur > DRIFT_TOLERANCE_S
):
apad_dur = retime_decision.video_dur
audio_map = f"{track_idx}:a:0"
if bg_idx is not None:
chain = f"[{bg_idx}:a][{track_idx}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2"
if apad_dur:
chain += f",apad=whole_dur={apad_dur:.4f}"
filter_parts.append(chain + "[aout]")
audio_map = "[aout]"
elif apad_dur:
filter_parts.append(f"[{track_idx}:a]apad=whole_dur={apad_dur:.4f}[aout]")
audio_map = "[aout]"
filter_parts.append(
f"[{bg_idx}:a][{track_idx}:a]amix=inputs=2:duration=longest:dropout_transition=2:weights=0.8 1.2[aout]"
)
cmd += ["-map", video_map]
cmd += ["-map", audio_map]
if bg_idx is not None:
cmd += ["-map", "[aout]"]
else:
cmd += ["-map", f"{track_idx}:a:0"]
if filter_parts:
cmd += ["-filter_complex", ";".join(filter_parts)]
# Retime path needs a real encode; stream-copy otherwise (the batched
# Smart Fit intermediate is already encoded — copy unless tpad'ed).
if stretch_entry or video_reencode:
# Stretch path needs a real encode; stream-copy otherwise.
if stretch_entry:
cmd += ["-c:v", "libx264", "-preset", "medium", "-crf", "20", "-pix_fmt", "yuv420p"]
else:
cmd += ["-c:v", "copy"]
cmd += ["-c:a", "aac", "-b:a", "192k"]
# `-shortest` would cut the retimed video at the (slightly different)
# `-shortest` would cut the stretched video at the (slightly different)
# audio length and lose the trailing frame; only use it on the copy path.
if not stretch_entry and retime_decision is None:
if not stretch_entry:
cmd += ["-shortest"]
cmd += [mux_path, "-y"]
def _discard_tmp():
try:
os.remove(mux_path)
except OSError as e:
logger.debug("cleanup remove failed: %s", e)
cmd += [preview_path, "-y"]
try:
rc, _, stderr = await run_ffmpeg(cmd, timeout=900.0, job_id=job_id)
rc, _, stderr = await run_ffmpeg(cmd, timeout=900.0)
if rc != 0:
raise Exception(stderr.decode(errors="replace") if stderr else "ffmpeg mux non-zero")
if not os.path.exists(mux_path) or os.path.getsize(mux_path) == 0:
raise Exception("preview mux produced empty file")
except asyncio.TimeoutError:
_discard_tmp()
raise HTTPException(status_code=504, detail="preview mux timed out")
except HTTPException:
_discard_tmp()
raise
except Exception as e:
_discard_tmp()
raise HTTPException(
status_code=500,
detail=f"ffmpeg failed to build the preview stream: {str(e)[:300]}. This usually means the source video can't be re-encoded on the fly — try downloading the MP4 instead.",
)
finally:
if retime_decision is not None and retime_decision.mode == "file":
try:
os.remove(retime_decision.file_path)
except OSError as e:
# Best-effort scratch cleanup — never fail the export.
logger.debug("retime intermediate cleanup failed: %s", e)
os.replace(mux_path, preview_path)
if not os.path.exists(preview_path) or os.path.getsize(preview_path) == 0:
raise HTTPException(status_code=500, detail="preview mux produced empty file")
async with _preview_lock(preview_path):
if not _cache_ok():
await _mux_preview()
# no-store: the URL is stable across re-dubs, so any HTTP-level caching
# in the WebView would keep showing the previous dub after a re-generate
# (#281: "edits don't change the result").
return FileResponse(
preview_path,
media_type="video/mp4",
headers={"Cache-Control": "no-store"},
)
def _compute_onsets_sync(src_path: str) -> list[float]:
"""Blocking part of onset analysis — runs in a worker thread."""
import soundfile as sf
from services.onset_align import detect_speech_onsets
audio, sr = sf.read(src_path, dtype="float32")
return detect_speech_onsets(audio, sr)
@router.get("/dub/onsets/{job_id}")
async def dub_get_onsets(job_id: str):
"""Speech-onset times for the timeline editor's snap-to-onset ticks (#280).
Prefers the Demucs-isolated vocals track (clean speech energy); falls
back to the mixed audio. Computed once per job and cached as
``onsets.json`` in the job directory; recomputed if the source audio is
newer than the cache (e.g. re-ingest into the same job dir).
"""
import json
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
vocals = job.get("vocals_path")
mix = job.get("audio_path")
if vocals and os.path.exists(vocals):
src_path, source = vocals, "vocals"
elif mix and os.path.exists(mix):
src_path, source = mix, "mix"
else:
raise HTTPException(status_code=404, detail="No audio track available for onset analysis")
# Containment inlined (not via _safe_job_path): CodeQL can't track the
# sanitizer through a helper's return — the file's established idiom.
base = os.path.realpath(DUB_DIR)
cache_path = os.path.realpath(os.path.join(base, job_id, "onsets.json"))
if not cache_path.startswith(base + os.sep):
raise HTTPException(status_code=400, detail="Invalid job id")
try:
if (
os.path.exists(cache_path)
and os.path.getmtime(cache_path) >= os.path.getmtime(src_path)
):
with open(cache_path, "r", encoding="utf-8") as f:
cached = json.load(f)
if isinstance(cached, dict) and isinstance(cached.get("onsets"), list):
return cached
except (OSError, ValueError):
pass # unreadable/corrupt cache → recompute below
try:
onsets = await asyncio.to_thread(_compute_onsets_sync, src_path)
except Exception as e:
raise HTTPException(
status_code=500,
detail=f"Onset analysis failed: {str(e)[:200]}",
)
payload = {"onsets": onsets, "source": source}
try:
os.makedirs(os.path.dirname(cache_path), exist_ok=True)
tmp_path = cache_path + ".tmp"
with open(tmp_path, "w", encoding="utf-8") as f:
json.dump(payload, f)
os.replace(tmp_path, cache_path)
except OSError as e:
logger.warning("onsets cache write failed for %s: %s", job_id, e)
return payload
return FileResponse(preview_path, media_type="video/mp4")
@router.get("/dub/thumb/{job_id}")
@@ -1148,100 +729,6 @@ async def dub_preview_segment(job_id: str, segment_index: int):
return FileResponse(seg_path, media_type="audio/wav")
# ── Second-pass ASR QC (Wave 3.3 / Spec 5) ───────────────────────────────────
@router.post("/dub/qc/{job_id}")
async def dub_qc_pass(job_id: str, lang: str = Query(None), drift_threshold: float = Query(0.5)):
"""Re-recognize the dubbed audio and flag lines whose recognized text
drifts from the target text. Opt-in, never fatal: the dub is untouched
this only annotates segments with a per-line drift score and a measured
start/end, surfaced as "verify this line" markers feeding incremental
re-dub. The generated text stays authoritative (design delta from
pyvideotrans, which overwrites subtitles)."""
from services import dub_qc
from services.dub_pipeline import put_job, save_job
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
tracks = job.get("dubbed_tracks", {})
if lang and lang in tracks:
wav_path = tracks[lang]["path"]
elif tracks:
wav_path = list(tracks.values())[0]["path"]
else:
raise HTTPException(status_code=400, detail="No dubbed audio track generated yet")
if not os.path.exists(wav_path):
raise HTTPException(status_code=404, detail="Dubbed audio file not found")
segments = job.get("segments") or []
if not segments:
raise HTTPException(status_code=400, detail="Job has no segments")
def _recognize():
from services.asr_backend import get_active_asr_backend
backend = get_active_asr_backend()
result = backend.transcribe(wav_path, word_timestamps=False)
return result.get("segments", []), backend.id
try:
from services.model_manager import _get_gpu_pool
from services.asr_backend import ASRTimeoutError, run_transcribe_guarded
recognized, engine_id = await run_transcribe_guarded(
_get_gpu_pool(), _recognize, what="QC",
)
except ASRTimeoutError as e:
# Backend is alive; ASR just couldn't finish in time. 504, not 500/connection.
logger.warning("dub QC ASR pass timed out for %s: %s", job_id, e)
raise HTTPException(status_code=504, detail=str(e))
except Exception as e:
logger.exception("dub QC ASR pass failed for %s", job_id)
raise HTTPException(status_code=500, detail=f"QC transcription failed: {e}")
seg_ids = job.get("seg_order") or [s.get("id", i) for i, s in enumerate(segments)]
scored = dub_qc.score_dub(segments, recognized, drift_threshold=drift_threshold, seg_ids=seg_ids)
# Annotate each segment (non-destructive — content text untouched).
by_id = {q.seg_id: q for q in scored}
for i, s in enumerate(segments):
sid = str(seg_ids[i]) if i < len(seg_ids) else str(s.get("id", i))
q = by_id.get(sid)
if q is None:
continue
s["qc_drift"] = q.drift
s["qc_flagged"] = q.flagged
s["qc_recognized"] = q.recognized_text
if q.new_start is not None:
s["qc_measured_start"] = q.new_start
s["qc_measured_end"] = q.new_end
put_job(job_id, job)
save_job(job_id, job)
flagged = [q for q in scored if q.flagged]
payload = json.dumps({"event": "qc_done", "engine": engine_id,
"flagged": len(flagged), "total": len(scored)})
try:
from core import job_store
job_store.append_event(job_id, f"data: {payload}\n\n")
except Exception as e:
# QC event fan-out is best-effort; the scores are already in the response.
logger.debug("QC event append failed: %s", e)
return {
"engine": engine_id,
"total": len(scored),
"flagged_count": len(flagged),
"drift_threshold": drift_threshold,
"segments": [
{"seg_id": q.seg_id, "drift": q.drift, "flagged": q.flagged,
"recognized_text": q.recognized_text,
"measured_start": q.new_start, "measured_end": q.new_end}
for q in scored
],
}
@router.get("/dub/download-audio/{job_id}")
@router.get("/dub/download-audio/{job_id}/{filename}")
async def dub_download_audio(job_id: str, lang: str = Query(None), preserve_bg: bool = Query(True), save_path: str = Query("")):
@@ -1319,31 +806,9 @@ def _pick_subtitle_text(seg: dict, dual: bool) -> str:
return f"{translated}\n<i>{original}</i>"
# Subtitles deliberately have no ?save_path= variant: they're small text
# bodies, so the Tauri side fetches them raw and writes the file itself via
# the save_text_file command — the OS save dialog is the write authorization
# (#309). The frontend's JSON-envelope save flow stays for binary exports.
def _fitted_cue_times(job: dict, lang: str | None) -> list | None:
"""Per-segment (start, end) on the fitted timeline when this job used
stretch_video; None to use the original segment times. (Wave 3.1.)"""
tracks = job.get("dubbed_tracks", {})
lc = lang if (lang and lang in tracks) else (next(iter(tracks), None))
entry = _video_stretch_plan_for(job, lc) if lc else None
if not entry:
return None
from services.fitted_subtitles import fitted_cues
return fitted_cues(job.get("segments", []), entry["plan"])
@router.get("/dub/srt/{job_id}")
@router.get("/dub/srt/{job_id}/{filename}")
async def dub_export_srt(
job_id: str,
dual: bool = False,
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
):
async def dub_export_srt(job_id: str, dual: bool = False):
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
@@ -1352,31 +817,22 @@ async def dub_export_srt(
if not segments:
raise HTTPException(status_code=400, detail="No transcript segments available")
# Subtitles must follow the audio the viewer hears, per timing strategy:
# Smart Fit (phase B) overlays the fitted segment times directly;
# stretch_video (Wave 3.1) regenerates cue times from the stretch plan.
# Neither applies → original times.
fitted = _fitted_segments_for(job, lang)
if fitted:
segments = _apply_fitted_times(segments, fitted)
cues = None if fitted else _fitted_cue_times(job, lang)
srt_lines = []
for i, seg in enumerate(segments):
s, e = cues[i] if cues else (seg["start"], seg["end"])
start_ts = _format_srt_time(seg["start"])
end_ts = _format_srt_time(seg["end"])
srt_lines.append(f"{i + 1}")
srt_lines.append(f"{_format_srt_time(s)} --> {_format_srt_time(e)}")
srt_lines.append(f"{start_ts} --> {end_ts}")
srt_lines.append(_pick_subtitle_text(seg, dual))
srt_lines.append("")
srt_content = "\n".join(srt_lines)
base_name = os.path.splitext(job.get('filename', 'video'))[0]
suffix = "_dual" if dual else ""
dl_name = f"subtitles_{base_name}{suffix}.srt"
return Response(
content=srt_content,
media_type="text/plain",
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
headers={"Content-Disposition": f'attachment; filename="subtitles_{base_name}{suffix}.srt"'},
)
def _format_vtt_time(seconds):
@@ -1388,11 +844,7 @@ def _format_vtt_time(seconds):
@router.get("/dub/vtt/{job_id}")
@router.get("/dub/vtt/{job_id}/{filename}")
async def dub_export_vtt(
job_id: str,
dual: bool = False,
lang: str = Query(None, description="Track language code. When that track was generated under Smart Fit or stretch_video, cue times come from the fitted timeline."),
):
async def dub_export_vtt(job_id: str, dual: bool = False):
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
@@ -1401,28 +853,22 @@ async def dub_export_vtt(
if not segments:
raise HTTPException(status_code=400, detail="No transcript segments available")
# Same strategy-aware cue timing as /dub/srt (see comment there).
fitted = _fitted_segments_for(job, lang)
if fitted:
segments = _apply_fitted_times(segments, fitted)
cues = None if fitted else _fitted_cue_times(job, lang)
vtt_lines = ["WEBVTT", ""]
for i, seg in enumerate(segments):
s, e = cues[i] if cues else (seg["start"], seg["end"])
start_ts = _format_vtt_time(seg["start"])
end_ts = _format_vtt_time(seg["end"])
vtt_lines.append(str(i + 1))
vtt_lines.append(f"{_format_vtt_time(s)} --> {_format_vtt_time(e)}")
vtt_lines.append(f"{start_ts} --> {end_ts}")
vtt_lines.append(_pick_subtitle_text(seg, dual))
vtt_lines.append("")
vtt_content = "\n".join(vtt_lines)
base_name = os.path.splitext(job.get('filename', 'video'))[0]
suffix = "_dual" if dual else ""
dl_name = f"subtitles_{base_name}{suffix}.vtt"
return Response(
content=vtt_content,
media_type="text/vtt",
headers={"Content-Disposition": f'attachment; filename="{dl_name}"'},
headers={"Content-Disposition": f'attachment; filename="subtitles_{base_name}{suffix}.vtt"'},
)
File diff suppressed because it is too large Load Diff
+69 -275
View File
@@ -2,13 +2,12 @@ import os
import time
import asyncio
import logging
from typing import Optional
from fastapi import APIRouter
from fastapi.responses import JSONResponse
from schemas.requests import TranslateRequest
from services.model_manager import _cpu_pool, _gpu_pool
from services.translator import cinematic_available, cinematic_refine_many, _cinematic_budget
from services.translator import cinematic_available, cinematic_refine_many
from api.routers.dub_core import _get_job
router = APIRouter()
@@ -46,70 +45,6 @@ LANG_NAMES = {
"id": "Indonesian", "uk": "Ukrainian",
}
# Regional dialect hints (#280 item 2). Maps a BCP-47 dialect code to the
# instruction injected into LLM translation prompts so the output uses that
# region's vocabulary and grammar (the reporter's example: choosing Argentina
# should yield "Vos sos muy listo", not the Peninsular "Tú eres muy listo").
# Only LLM-backed paths can honor these — provider="openai" and the
# quality="cinematic" refine pass. Keep entries short: they ride on every
# per-segment prompt, so verbosity = wall time.
DIALECT_HINTS = {
# Spanish
"es-ES": "European Spanish (Spain): use tú/vosotros forms and Peninsular vocabulary.",
"es-MX": "Mexican Spanish: use tú/ustedes forms and Mexican vocabulary.",
"es-AR": "Rioplatense Spanish (Argentina): use voseo — 'vos' with its verb forms (e.g. 'vos sos', 'tenés') and 'ustedes'; prefer Argentinian vocabulary.",
"es-CO": "Colombian Spanish: use tú/usted as natural in Colombia and Colombian vocabulary.",
"es-CL": "Chilean Spanish: use Chilean vocabulary and expressions.",
# Portuguese
"pt-BR": "Brazilian Portuguese: use 'você' forms, Brazilian vocabulary and spelling.",
"pt-PT": "European Portuguese: use European vocabulary, spelling, and 'tu' where natural.",
# English
"en-US": "American English: use US spelling and vocabulary.",
"en-GB": "British English: use UK spelling and vocabulary.",
"en-AU": "Australian English: use Australian spelling and vocabulary.",
"en-IN": "Indian English: use Indian English vocabulary and conventions.",
# French
"fr-FR": "Metropolitan French (France): use standard French vocabulary.",
"fr-CA": "Canadian French (Québec): use Québécois vocabulary and expressions.",
"fr-BE": "Belgian French: use Belgian vocabulary (e.g. septante, nonante).",
# German
"de-DE": "Standard German (Germany): use Federal German vocabulary.",
"de-AT": "Austrian German: use Austrian vocabulary (e.g. Jänner, Erdapfel).",
"de-CH": "Swiss Standard German: use Swiss vocabulary and 'ss' instead of 'ß'.",
# Arabic
"ar-EG": "Egyptian Arabic: use Egyptian colloquial vocabulary where natural for dubbing.",
"ar-SA": "Gulf/Saudi Arabic flavor: prefer vocabulary natural to the Gulf region.",
"ar-MA": "Moroccan Arabic (Darija) flavor: prefer vocabulary natural to Morocco.",
# Dutch
"nl-NL": "Netherlands Dutch: use vocabulary standard in the Netherlands.",
"nl-BE": "Belgian Dutch (Flemish): use Flemish vocabulary and expressions.",
}
def dialect_clause(dialect: Optional[str]) -> str:
"""Prompt fragment for a requested dialect, or '' when unset/unknown.
Unknown-but-plausible codes (e.g. "es-PE") still get a generic regional
clause so users aren't limited to the curated list.
"""
if not dialect or not str(dialect).strip():
return ""
code = str(dialect).strip()
hint = DIALECT_HINTS.get(code)
if hint:
return f" Target dialect — {hint}"
# Generic fallback for any lang-REGION shaped code we don't curate.
if "-" in code:
lang, _, region = code.partition("-")
lang_name = LANG_NAMES.get(lang, lang)
if region:
return (
f" Use the vocabulary, grammar, and expressions of {lang_name} "
f"as spoken in the region '{region}'."
)
return ""
# Per-language script enforcement. Maps language code → required Unicode
# block(s) the translation must contain. Used as a sanity gate after the
# LLM responds: if the output contains <50% characters from the expected
@@ -157,49 +92,15 @@ _nllb_tokenizer = None
_nllb_device = None
def _dialect_flags(req, applied: bool) -> dict:
"""Response fields describing whether the requested dialect was honored.
Empty dict when no dialect was requested, so existing response shapes
stay byte-identical for callers that never send one.
"""
if not getattr(req, "dialect", None):
return {}
return {"dialect": req.dialect, "dialect_applied": bool(applied)}
def _guess_lang_from_text(segments) -> str | None:
"""Best-effort source language from segment text, by script.
Used only as a last resort when neither the request nor the job carries a
detected language. Without this, the bare "en" fallback below forces
en -> en on non-English audio (e.g. Korean), which has no Argos package and
fails every segment even though ASR detected the language correctly.
"""
text = " ".join((getattr(s, "text", "") or "") for s in (segments or [])[:8])
has = lambda lo, hi: any(lo <= ord(c) <= hi for c in text)
if has(0x3040, 0x30FF):
return "ja" # Hiragana/Katakana — check before CJK (Japanese uses Kanji too)
if has(0xAC00, 0xD7A3) or has(0x1100, 0x11FF):
return "ko" # Hangul
if has(0x4E00, 0x9FFF):
return "zh" # CJK ideographs
if has(0x0400, 0x04FF):
return "ru" # Cyrillic
if has(0x0600, 0x06FF):
return "ar" # Arabic
return None
def _resolve_source_lang(req: TranslateRequest) -> str:
"""Pick source language: explicit request > job.source_lang > text guess > 'en'."""
"""Pick source language: explicit request > job.source_lang > 'en' fallback."""
if getattr(req, "source_lang", None):
return req.source_lang
if getattr(req, "job_id", None):
job = _get_job(req.job_id)
if job and job.get("source_lang"):
return job["source_lang"]
return _guess_lang_from_text(getattr(req, "segments", None)) or "en"
return "en"
def _unload_nllb():
@@ -302,20 +203,14 @@ async def dub_translate(req: TranslateRequest):
translated = await loop.run_in_executor(_gpu_pool, _translate_nllb)
if os.environ.get("OMNIVOICE_UNLOAD_NLLB", "1") == "1":
_unload_nllb()
# Cinematic/Autofit refine + rate-ratio badges must run for NLLB too
# (previously this returned before _maybe_cinematic, so a Cinematic
# pick on NLLB silently produced plain Fast output). Unloading NLLB
# first is fine — the refine LLM is a separate network provider.
return await _maybe_cinematic(translated, req, src_lang, loop)
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
# OpenAI / Ollama Local LLM Translation
if provider == "openai":
base_url = os.environ.get("TRANSLATE_BASE_URL")
model_name = os.environ.get("TRANSLATE_MODEL", "gpt-3.5-turbo")
from openai import OpenAI
# max_retries=0: a 429 + long Retry-After must not let one segment's
# SDK call sleep+retry and blow the overall translate wall time.
client = OpenAI(base_url=base_url, api_key=api_key or "local", max_retries=0)
client = OpenAI(base_url=base_url, api_key=api_key or "local")
def _build_prompt(src_code: str, tgt_code: str) -> str:
"""Build a system prompt that resists hallucinations on small
@@ -341,16 +236,10 @@ async def dub_translate(req: TranslateRequest):
f"only — do not use Latin/Roman letters, do not "
f"transliterate, do not output any other language."
)
# #280 item 2 — regional dialect/vocabulary. Only applied when
# the dialect belongs to the target language (a leftover
# "es-AR" must not contaminate a French translation).
dia_clause = ""
if req.dialect and str(req.dialect).lower().startswith(str(tgt_code).lower()[:2]):
dia_clause = dialect_clause(req.dialect)
return (
f"You are a professional dubbing translator. "
f"Translate the user's text from {src_name} into "
f"{tgt_name}.{script_clause}{dia_clause} "
f"{tgt_name}.{script_clause} "
f"Reply ONLY with the translated {tgt_name} text, do not "
f"add quotes, notes, headers, explanations, or commentary."
)
@@ -404,38 +293,24 @@ async def dub_translate(req: TranslateRequest):
seg.id, attempt + 1, e,
)
# Both attempts failed — keep source text + flag error so the
# frontend can surface "fallback to literal" warning. Scrub the
# provider error: some OpenAI-compatible providers echo the key
# or a user_id in the body, which must not reach the UI verbatim.
from core.scrub import scrub_provider_error
return {"id": seg.id, "text": seg.text,
"error": scrub_provider_error(last_err, api_key) or "llm-failed"}
# frontend can surface "fallback to literal" warning.
return {"id": seg.id, "text": seg.text, "error": last_err or "llm-failed"}
tasks = [loop.run_in_executor(_cpu_pool, _translate_llm, seg) for seg in req.segments]
translated = await asyncio.gather(*tasks)
translated.sort(key=lambda x: str(x["id"]))
# provider="openai" is already an LLM translation — _maybe_cinematic
# skips the reflect/adapt re-refine (already_llm) but still stamps
# rate-ratio badges and runs the bounded Autofit fit pass. Before
# this it returned here, so Cinematic/Autofit on the LLM engine did
# nothing.
return await _maybe_cinematic(translated, req, src_lang, loop, already_llm=True)
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
# Offline Argos Translate
if provider == "argos" or provider == "libretranslate":
try:
import argostranslate # noqa: F401
except ImportError:
# Single-source the install command from the engine registry so
# this 400 and the proactive Install button in the Engine
# selector can never drift (see translation_engines.install_command).
from services.translation_engines import install_command
cmd = install_command("argos") or "uv pip install argostranslate"
friendly = (
f"The '{provider}' translation engine needs the optional "
f"`argostranslate` Python package, which isn't installed in "
f"this backend. Install it with `{cmd}` "
f"and restart the server, or "
f"this backend. Install it with `uv pip install argostranslate` "
f"(or `pip install argostranslate`) and restart the server, or "
f"switch the Engine dropdown to another provider."
)
return JSONResponse(status_code=400, content={"error": friendly})
@@ -478,11 +353,7 @@ async def dub_translate(req: TranslateRequest):
return results
translated = await loop.run_in_executor(_cpu_pool, _translate_argos)
# Argos is the DEFAULT engine — routing it through _maybe_cinematic is
# the headline fix: a user who picks Cinematic/Autofit on Argos now
# gets the LLM refine + fit pass (and rate-ratio badges in Fast mode)
# instead of silent plain-Fast output.
return await _maybe_cinematic(translated, req, src_lang, loop)
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
# Legacy / API Deep_Translator logic.
# Preflight the optional `deep_translator` dep once so we fail with a
@@ -491,16 +362,11 @@ async def dub_translate(req: TranslateRequest):
try:
import deep_translator # noqa: F401
except ImportError:
# Same single-source install command as the Engine selector's Install
# button (translation_engines.install_command) — google/deepl/
# microsoft/mymemory all share the deep_translator package.
from services.translation_engines import install_command
cmd = install_command(provider) or "uv pip install deep_translator"
friendly = (
f"The '{provider}' translation engine needs the optional "
f"`deep_translator` Python package, which isn't installed in "
f"this backend. Install it with `{cmd}` "
f"and restart the server, or "
f"this backend. Install it with `uv pip install deep_translator` "
f"(or `pip install deep_translator`) and restart the server, or "
f"switch the Engine dropdown to Argos (local, bundled), NLLB "
f"(local, heavier), or OpenAI (LLM)."
)
@@ -556,11 +422,7 @@ async def dub_translate(req: TranslateRequest):
)
time.sleep(0.25 * (attempt + 1))
logger.error("translate %s -> %s gave up (provider=%s): %s", src_arg, seg_lc, provider, last_err)
# Scrub before it reaches the UI — DeepL/Microsoft errors can echo
# the API key (same class as the OpenAI user_id leak).
from core.scrub import scrub_provider_error
return {"id": seg.id, "text": seg.text,
"error": scrub_provider_error(last_err, _deepl_key or _msft_key or api_key) or "unknown"}
return {"id": seg.id, "text": seg.text, "error": last_err or "unknown"}
tasks = [loop.run_in_executor(_cpu_pool, _translate_single, seg) for seg in req.segments]
translated = await asyncio.gather(*tasks)
@@ -574,19 +436,24 @@ async def dub_translate(req: TranslateRequest):
return JSONResponse(status_code=500, content={"error": str(e)})
def _stamp_predicted_rate_ratio(translated, req) -> None:
"""Stamp a predicted ``rate_ratio`` on every row that has a known slot.
No LLM needed just the per-language CPS table from ``services/speech_rate``.
The UI's ``seg-rate-badge`` reads it (Fast mode included) to show which
segments will compress hard at generation time, so users can edit text or
pick a heavier quality. Mutates ``translated`` in place; never raises.
async def _maybe_cinematic(translated, req, src_lang, loop):
"""If quality=cinematic and a usable LLM is configured, run REFLECT+ADAPT.
Otherwise return Fast-mode shape unchanged.
"""
quality = (getattr(req, "quality", None) or "fast").lower()
# Stamp the predicted rate_ratio on every translated row that has a
# known slot. Works for Fast mode too — no LLM needed; just the CPS
# table from services/speech_rate. The UI's `seg-rate-badge` reads
# this value and shows users which segments will compress hard at
# generation time, so they can edit text or pick Cinematic quality.
try:
from services.speech_rate import rate_ratio as _predict_rate_ratio
slots = {str(s.id): getattr(s, "slot_seconds", None) for s in req.segments}
for row in translated:
slot = slots.get(str(row["id"]))
seg_ref = next(
(s for s in req.segments if str(s.id) == str(row["id"])),
None,
)
slot = getattr(seg_ref, "slot_seconds", None) if seg_ref else None
text = (row.get("text") or "").strip()
if slot and text and not row.get("error"):
row["rate_ratio"] = round(
@@ -595,119 +462,18 @@ def _stamp_predicted_rate_ratio(translated, req) -> None:
except Exception as e:
logger.debug("non-LLM rate_ratio prediction skipped: %s", e)
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang, "quality_used": "fast"}
async def _apply_fit_pass(rows, req, slots_by_id, source_by_id, quality, loop, deadline) -> None:
"""Run the Autofit slot-fit pass over ``rows`` concurrently, in place.
Bounded by ``deadline`` (shared with the cinematic refine) so a slow /
rate-limited LLM can't spin the fit pass per-segment unbounded — the old
behavior, which ran one blocking ``adjust_for_slot`` per segment in the
merge loop, outside any budget. Segments still running at the deadline keep
their current text and get ``rate_error='fit-budget'``. Only rows with a
slot + text + no prior error participate.
"""
strict = (quality == "autofit")
items = []
for row in rows:
seg_id = str(row["id"])
slot = slots_by_id.get(seg_id)
text = row.get("text") or ""
if slot and text and not row.get("error"):
items.append((seg_id, text, float(slot), req.target_lang,
source_by_id.get(seg_id), strict))
if not items:
return
try:
from services.speech_rate import adjust_for_slot_many
fits = await adjust_for_slot_many(
items, executor=_cpu_pool, deadline=deadline, loop=loop,
)
except Exception as e:
logger.warning("rate-fit pass skipped: %s", e)
return
for row in rows:
f = fits.get(str(row["id"]))
if not f:
continue
if f.get("text"):
row["text"] = f["text"]
if f.get("rate_ratio") is not None:
row["rate_ratio"] = f["rate_ratio"]
if f.get("error"):
row["rate_error"] = f["error"]
async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False):
"""Post-process a literal translation into Cinematic/Autofit output.
Runs for EVERY provider now (Argos/NLLB/Google//OpenAI). The three
LLM-independent branches (nllb/argos) and the openai branch used to return
*before* reaching this, so a Cinematic/Autofit pick on them including the
DEFAULT Argos engine silently produced plain Fast output with a success
toast. Fast mode still returns the plain translation (plus rate-ratio badges).
``already_llm`` (provider="openai"): the translation was itself produced by
an LLM, so the REFLECT+ADAPT *re*-refine is skipped, but the bounded Autofit
fit pass + rate-ratio stamping still run, and the dialect the translate
prompt already baked in is reported as applied.
"""
quality = (getattr(req, "quality", None) or "fast").lower()
_stamp_predicted_rate_ratio(translated, req)
# #280 item 2 — regional dialect hint, guarded against a stale dialect from
# another language. For already_llm the initial translate prompt already
# applied it, so it's reported applied in the Fast-shape base too.
dialect_hint = ""
_dialect = getattr(req, "dialect", None)
if _dialect and str(_dialect).lower().startswith(str(req.target_lang).lower()[:2]):
dialect_hint = dialect_clause(_dialect)
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
"quality_used": "fast",
**_dialect_flags(req, applied=(already_llm and bool(dialect_hint)))}
# Fast (and anything unrecognised) returns the plain translation unchanged.
if quality not in ("cinematic", "autofit"):
if quality != "cinematic":
return base
source_by_id: dict[str, str] = {str(s.id): s.text for s in req.segments}
slots_by_id = {
str(s.id): getattr(s, "slot_seconds", None)
for s in req.segments
if getattr(s, "slot_seconds", None)
}
# One wall-clock deadline shared by the whole LLM phase (refine + fit), so a
# slow/rate-limited provider can't run either pass unbounded. <=0 disables.
budget = _cinematic_budget()
deadline = (loop.time() + budget) if budget and budget > 0 else None
# provider="openai": already an LLM translation → skip REFLECT+ADAPT, keep
# the rate-ratio badges, still run the bounded fit pass.
if already_llm:
merged = []
for row in translated:
out = {"id": row["id"],
"text": row.get("text", "") or "",
"literal": row.get("text", "") or ""}
if row.get("error"):
out["error"] = row["error"]
if "rate_ratio" in row:
out["rate_ratio"] = row["rate_ratio"]
merged.append(out)
await _apply_fit_pass(merged, req, slots_by_id, source_by_id, quality, loop, deadline)
return {"translated": merged, "target_lang": req.target_lang,
"source_lang": src_lang, "quality_used": quality,
**_dialect_flags(req, applied=bool(dialect_hint))}
# Non-LLM provider → the reflect/adapt refine needs a separately-configured
# LLM (Settings → LLM Providers). Without one, degrade to Fast with a flag.
if not cinematic_available():
logger.warning("%s requested but no LLM configured — returning Fast result.", quality)
logger.warning("cinematic requested but no LLM configured — returning Fast result.")
base["cinematic_skipped"] = "no-llm-configured"
return base
# Build a map from id → original segment (to fetch source text + direction).
source_by_id: dict[str, str] = {str(s.id): s.text for s in req.segments}
directions: dict[str, str] = {
str(s.id): s.direction
for s in req.segments
@@ -715,7 +481,7 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
}
pairs = []
passthrough_index = {}
for row in translated:
for i, row in enumerate(translated):
seg_id = str(row["id"])
literal = row.get("text", "") or ""
if row.get("error") or not literal.strip():
@@ -732,11 +498,20 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
target_lang=req.target_lang,
glossary=req.glossary,
directions=directions,
dialect_hint=dialect_hint,
executor=_cpu_pool,
)
refined_by_id = {r["id"]: r for r in refined}
# Phase 4.4 — speech-rate fit pass. Segment boundaries aren't in the
# translate request (by design — translator is boundary-agnostic), so we
# only run it when the caller supplied `slot_seconds` on each segment.
# The frontend populates this for Cinematic calls from the edit view.
slots_by_id = {
str(s.id): getattr(s, "slot_seconds", None)
for s in req.segments
if getattr(s, "slot_seconds", None)
}
merged = []
for row in translated:
seg_id = str(row["id"])
@@ -755,15 +530,34 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
}
if r.get("error"):
out["error"] = r["error"]
merged.append(out)
# Phase 4.4 speech-rate fit pass — now concurrent + bounded (see helper).
await _apply_fit_pass(merged, req, slots_by_id, source_by_id, quality, loop, deadline)
# Optional slot-fit pass — only when the caller asked for cinematic
# *and* provided a slot. Runs best-effort; no-LLM or mid-loop failure
# just leaves the cinematic text untouched.
slot = slots_by_id.get(seg_id)
if slot and out["text"]:
try:
from services.speech_rate import adjust_for_slot
fit = await asyncio.to_thread(
adjust_for_slot,
out["text"],
slot_seconds=float(slot),
target_lang=req.target_lang,
source_text=source_by_id.get(seg_id),
)
if fit.get("text"):
out["text"] = fit["text"]
out["rate_ratio"] = fit.get("rate_ratio")
if fit.get("error"):
out["rate_error"] = fit["error"]
except Exception as e:
logger.warning("rate-fit skipped for %s: %s", seg_id, e)
merged.append(out)
return {
"translated": merged,
"target_lang": req.target_lang,
"source_lang": src_lang,
"quality_used": quality,
**_dialect_flags(req, applied=bool(dialect_hint)),
"quality_used": "cinematic",
}
+6 -200
View File
@@ -15,8 +15,6 @@ Environment variables (`OMNIVOICE_TTS_BACKEND`, `OMNIVOICE_ASR_BACKEND`,
`OMNIVOICE_LLM_BACKEND`) still win over the UI choice so power-users can pin
a backend without Settings silently undoing it.
"""
import os
import threading
from time import perf_counter
from fastapi import APIRouter, Depends, HTTPException
@@ -263,194 +261,16 @@ def engine_health(engine_id: str):
}
# ── Real-synthesis self-test (in-process TTS engines) ──────────────────────
#
# ``/health`` above is a liveness/import probe — for an in-process backend it
# only calls ``is_available()`` and the UI labels the result "deps OK". This
# route goes one step further: for an AVAILABLE, IN-PROCESS TTS engine it runs
# a *tiny real synthesis* from a fixed short phrase and reports duration +
# sample-rate + sample count, proving the engine actually emits audio rather
# than merely importing. The Compat Matrix's "Self-test" button calls it.
#
# Guardrails (kept identical across macOS/Windows/Linux per the default-feature
# rule — the phrase, timeout and gating don't branch on OS):
# * TTS family + available + in-process only. Subprocess engines keep their
# spawn-and-ping ``health_check`` (a real synth there is a sidecar
# cold-start — out of scope for a click-to-test affordance).
# * Bounded wall-clock timeout (``OMNIVOICE_SELFTEST_TIMEOUT_S``, default 90s):
# a runaway synth returns ``ok=False`` / ``timed_out=True`` instead of
# hanging the Settings panel. The orphaned worker is best-effort daemon.
# * A process-wide lock serialises self-tests so a click-storm can't stack
# concurrent model loads.
# * Only ever on user click (POST) — never on Settings load. Loopback-gated.
# Deliberately short + ASCII so the synth stays CPU-cheap and the phrase never
# trips the no-hardcoded-CJK guard.
_SELFTEST_PHRASE = "OmniVoice engine self test."
_SELFTEST_LOCK = threading.Lock()
def _selftest_timeout_s() -> float:
try:
return max(1.0, float(os.environ.get("OMNIVOICE_SELFTEST_TIMEOUT_S", "90")))
except (TypeError, ValueError):
return 90.0
def _sample_count(audio) -> int:
"""Total sample count of an engine's ``generate()`` return, tolerant of
torch.Tensor / numpy.ndarray / list shapes. 0 when it can't be measured."""
try:
shape = getattr(audio, "shape", None)
if shape is not None and len(shape) > 0:
return int(shape[-1])
return int(len(audio))
except Exception:
return 0
def _run_synth_bounded(backend, timeout_s: float) -> dict | None:
"""Run one tiny synthesis in a daemon thread, bounded by ``timeout_s``.
Returns ``{"audio": .., "duration_ms": ..}`` on success, ``{"error": exc}``
on a synth exception, or ``None`` when the timeout elapsed (worker left
running best-effort Python threads can't be force-killed)."""
box: dict = {}
def _worker():
t0 = perf_counter()
try:
audio = backend.generate(_SELFTEST_PHRASE, language="en", num_step=8)
box["audio"] = audio
except Exception as exc: # noqa: BLE001 — surfaced to the caller as ok=False
box["error"] = exc
finally:
box["duration_ms"] = (perf_counter() - t0) * 1000.0
th = threading.Thread(target=_worker, name="engine-selftest", daemon=True)
th.start()
th.join(timeout_s)
if th.is_alive():
return None
return box
class SelfTestResponse(BaseModel):
id: str
ok: bool
message: str
duration_ms: float
sample_rate: int | None = None
num_samples: int | None = None
audio_seconds: float | None = None
timed_out: bool = False
@router.post(
"/engines/{engine_id}/selftest",
response_model=SelfTestResponse,
dependencies=[Depends(require_loopback)],
)
def engine_selftest(engine_id: str):
"""Run a bounded, real synthesis on an available in-process TTS engine.
404 for an unknown TTS id; 400 when the engine is subprocess-isolated or
not currently available (a real synth on either is meaningless). Never
raises through to a 500 on a synth failure the exception is captured into
``ok=False`` / ``message`` so the panel renders a per-row failure."""
if engine_id not in tts_backend._REGISTRY:
raise HTTPException(
status_code=404,
detail=f"unknown TTS engine id: {engine_id!r}",
)
cls = tts_backend._REGISTRY[engine_id]
if getattr(cls, "_is_subprocess_isolated", False):
raise HTTPException(
status_code=400,
detail=(
f"{engine_id} is subprocess-isolated — self-test runs real "
"synthesis for in-process engines only. Use Test engine "
"(spawn-and-ping) for subprocess engines."
),
)
try:
ok, msg = cls.is_available()
except Exception as exc: # noqa: BLE001
ok, msg = False, f"{type(exc).__name__}: {exc}"
if not ok:
raise HTTPException(
status_code=400,
detail=(
f"{engine_id} is not available: {tts_backend._mask_hf_tokens(msg)}. "
"Install/enable the engine, then self-test."
),
)
timeout_s = _selftest_timeout_s()
# Serialise so a click-storm can't stack concurrent model loads.
with _SELFTEST_LOCK:
backend = _get_engine_instance(cls)
res = _run_synth_bounded(backend, timeout_s)
if res is None:
return SelfTestResponse(
id=engine_id,
ok=False,
message=f"timed out after {timeout_s:.0f}s (model still loading?)",
duration_ms=timeout_s * 1000.0,
timed_out=True,
)
if "error" in res:
exc = res["error"]
return SelfTestResponse(
id=engine_id,
ok=False,
message=tts_backend._mask_hf_tokens(f"{type(exc).__name__}: {exc}"),
duration_ms=res.get("duration_ms", 0.0),
)
n = _sample_count(res.get("audio"))
try:
sr = int(getattr(backend, "sample_rate", 0) or 0) or None
except Exception:
sr = None
secs = round(n / sr, 3) if (sr and n) else None
return SelfTestResponse(
id=engine_id,
ok=n > 0,
message="synthesized" if n > 0 else "engine returned no audio",
duration_ms=res["duration_ms"],
sample_rate=sr,
num_samples=n or None,
audio_seconds=secs,
)
class SelectEngineRequest(BaseModel):
family: str # "tts" | "asr" | "llm"
backend_id: str
class SelectEngineResponse(BaseModel):
family: str
active: str
env_override: bool
# Routing verdict for the selected engine on THIS host (#21). Always present
# so the UI can show a confirm/warning toast on a cpu_fallback pick without
# branching on key presence; defaults match a legacy/degraded row.
routing_status: str = "cpu_only"
effective_device: str = "cpu"
routing_reason: str | None = None
@router.post("/engines/select", response_model=SelectEngineResponse)
@router.post("/engines/select")
def select_engine(req: SelectEngineRequest):
"""Persist a family's engine pick to prefs.json. Refuses unknown backends,
backends whose deps aren't installed, AND backends that cannot run on THIS
host's hardware (routing_status == "unavailable") — so the UI can't silently
brick a pipeline by picking an engine that needs a GPU this machine lacks.
A `cpu_fallback` pick is allowed (it runs, just slower) only a hard
`unavailable` is blocked. LLM is never routing-gated (its status is "n/a")."""
"""Persist a family's engine pick to prefs.json. Refuses unknown backends
+ refuses backends whose deps aren't installed (so the UI can't silently
brick a pipeline by picking an unavailable engine)."""
family = _FAMILIES.get(req.family)
if not family:
raise HTTPException(400, f"Unknown family: {req.family}. Expected one of tts/asr/llm.")
@@ -458,26 +278,12 @@ def select_engine(req: SelectEngineRequest):
available = {b["id"]: b for b in module.list_backends()}
if req.backend_id not in available:
raise HTTPException(400, f"Unknown {req.family} backend: {req.backend_id!r}")
entry = available[req.backend_id]
if not entry["available"]:
reason = entry.get("reason") or "unavailable"
if not available[req.backend_id]["available"]:
reason = available[req.backend_id].get("reason") or "unavailable"
raise HTTPException(400, f"Backend {req.backend_id} not ready: {reason}")
# Host-routing gate (no silent CPU fallback). `.get` is defensive so an
# older/legacy payload without routing keys still selects cleanly.
if entry.get("routing_status") == "unavailable":
why = entry.get("routing_reason") or "requires a GPU this host doesn't have"
raise HTTPException(
400,
f"Backend {req.backend_id} can't run on this machine: {why}. "
f"Pick an engine with a CPU path, or one that supports this host's GPU.",
)
prefs.set_(pref_key, req.backend_id)
return {
"family": req.family,
"active": module.active_backend_id(),
"env_override": bool(__import__("os").environ.get(f"OMNIVOICE_{req.family.upper()}_BACKEND")),
# Echo the routing verdict so the UI can warn on a cpu_fallback pick.
"routing_status": entry.get("routing_status", "cpu_only"),
"effective_device": entry.get("effective_device", "cpu"),
"routing_reason": entry.get("routing_reason"),
}
+74 -743
View File
@@ -1,9 +1,7 @@
import os
import io
import re
import uuid
import time
import random
import asyncio
import tempfile
import contextlib
@@ -13,36 +11,16 @@ from typing import Optional
from fastapi import APIRouter, File, Form, UploadFile, HTTPException
from fastapi.responses import StreamingResponse
import sqlite3
from core.db import db_conn, ensure_schema
from core.db import db_conn
from core.config import OUTPUTS_DIR, VOICES_DIR
import functools
from services.model_manager import (
get_model, _gpu_pool, run_on_gpu_pool_guarded, GpuJobTimeoutError,
)
from services.model_manager import get_model, _gpu_pool
from services.audio_io import _safe_torchaudio_save
from core import event_bus
from omnivoice.utils.voice_design import heal_design_instruct
router = APIRouter()
logger = logging.getLogger("omnivoice.generate")
def _profile_instruct(row):
"""Validator-safe instruct for a stored profile row.
Sanitizes the persisted instruct (dropping the ``"[object Object]"``
sentinel / freeform prose that older builds saved) and, for a design row,
rebuilds the tags from ``vd_states`` when the stored value is unusable so
a poisoned/legacy profile never 400-s generation (#550 #571 #594 #596).
"""
try:
vd = row["vd_states"]
except (KeyError, IndexError):
vd = None
return heal_design_instruct(row["instruct"], vd)
def _render_with_pauses(gen_span, segments, sample_rate):
"""Synthesize ``[(text, pause_ms), ...]`` spans and stitch silence between
them (issue #276).
@@ -81,348 +59,13 @@ def _render_with_pauses(gen_span, segments, sample_rate):
parts.append(torch.zeros(*shape, dtype=ref.dtype, device=ref.device))
return torch.cat(parts, dim=-1)
def _sanitize_audio(audio_out):
"""Replace non-finite samples (NaN / ±inf) with silence so a model glitch
can't produce an unreadable WAV (#629). Returns the input unchanged when it's
already finite or isn't a tensor. Never raises."""
try:
import torch
if torch.is_tensor(audio_out) and not bool(torch.isfinite(audio_out).all()):
logger.warning(
"Generated audio contained non-finite samples (NaN/inf) — "
"sanitizing to silence to keep the WAV decodable (#629)."
)
return torch.nan_to_num(audio_out, nan=0.0, posinf=0.0, neginf=0.0)
except Exception:
pass
return audio_out
def _apply_effect_chain(audio_out, sample_rate, effect_preset, *, skip_mastering=False):
"""Shared post-DSP for /generate: preset validation → mastering →
effect chain loudness normalization.
``skip_mastering`` honors a backend's ``applies_own_mastering`` flag
(issue #312): studio engines (e.g. VoxCPM2's native 48 kHz output)
opt out of the broadcast Compressor + Reverb chain that's tuned for
OmniVoice's 24 kHz clone output. Loudness normalization still runs —
it's a benign peak scale. Mirrors ``_run_tts`` in openai_compat.py.
"""
from services.audio_dsp import (
EFFECT_PRESETS, apply_mastering, normalize_audio,
apply_effects_chain, get_effect_chain,
)
# #629: a numerical glitch in the model (observed on MPS) can leave NaN/±inf
# samples, which write an unreadable WAV that then fails decoding with an
# opaque "ffmpeg returned error code: 183 / Invalid data" — surfaced to the
# user as a misleading "ran out of memory". Replace non-finite samples with
# silence here, before any DSP/encode touches the audio, so the output is
# always a valid WAV. Covers the raw path too (it returns just below).
audio_out = _sanitize_audio(audio_out)
preset = effect_preset or "broadcast"
if preset not in EFFECT_PRESETS:
raise ValueError(
f"Unknown effect preset: {preset!r}. "
f"Valid: {list(EFFECT_PRESETS.keys())}"
)
if preset == "raw":
# Raw: skip all DSP — return raw model output
return audio_out
if not skip_mastering:
audio_out = apply_mastering(audio_out, sample_rate=sample_rate)
chain = get_effect_chain(preset)
if chain:
audio_out = apply_effects_chain(
audio_out, sample_rate=sample_rate, chain=chain,
)
return normalize_audio(audio_out, target_dBFS=-2.0)
def _exception_chain(e):
"""Yield ``e`` plus every ``__cause__``/``__context__`` beneath it
(cycle-safe). Engines and hub libraries routinely wrap the original
transport/allocator error, so classification must look at the whole
chain, not just the outermost message."""
seen = set()
stack = [e]
while stack:
exc = stack.pop()
if exc is None or id(exc) in seen:
continue
seen.add(id(exc))
yield exc
stack.append(exc.__cause__)
stack.append(exc.__context__)
# #880: transport-level exception type names from httpx (huggingface_hub ≥1.x
# downloads over it) and requests/urllib3 (older engine deps). Any of these
# anywhere in the exception chain means the network — not memory — killed the
# generation.
_NETWORK_EXC_NAMES = frozenset({
# httpx
"ConnectError", "ConnectTimeout", "ReadTimeout", "ReadError",
"WriteError", "WriteTimeout", "PoolTimeout", "NetworkError",
"TransportError", "RemoteProtocolError", "ProxyError", "CloseError",
# requests / urllib3
"ConnectionError", "ChunkedEncodingError", "MaxRetryError",
"NewConnectionError", "ProtocolError",
# stdlib socket-level drops mid-download
"ConnectionResetError", "ConnectionAbortedError", "ConnectionRefusedError",
# huggingface_hub: failed first-use download with nothing in the disk cache
"LocalEntryNotFoundError",
})
# Same class, but the transport error was stringified into a wrapper message
# (so the type name is gone). All lowercase; matched against .lower().
_NETWORK_MSG_SIGNATURES = (
"client has been closed", # httpx closed-client lifecycle error (#880)
"cannot send a request", # httpx: same error, message head
"connection error", # requests / huggingface_hub wording
"connection reset", # ECONNRESET mid-download
"read timed out", # requests/urllib3 timeout wording
"max retries exceeded", # urllib3 retry exhaustion
"temporary failure in name resolution", # DNS down (glibc)
"name or service not known", # DNS down (glibc)
"getaddrinfo failed", # DNS down (Windows)
)
def _is_network_failure(e) -> bool:
"""True iff the failure (anywhere in its chain) is an HTTP-client
lifecycle / network-transport error e.g. a first-use model download
from the HF Hub dying mid-generation (#880)."""
for exc in _exception_chain(e):
if type(exc).__name__ in _NETWORK_EXC_NAMES:
return True
low = str(exc).lower()
if any(sig in low for sig in _NETWORK_MSG_SIGNATURES):
return True
return False
# Signatures of an *actual* out-of-memory condition. All lowercase.
_OOM_MSG_SIGNATURES = (
"out of memory", # CUDA / MPS / generic torch wording
"not enough memory", # torch CPU DefaultCPUAllocator
"cannot allocate memory", # OS-level ENOMEM
"std::bad_alloc", # C++ allocator failure
"cublas_status_alloc_failed", # cuBLAS workspace allocation
"cuda_error_out_of_memory", # raw CUDA driver error name
"paging file is too small", # Windows [WinError 1455] mapping DLLs
)
def _is_oom_failure(e) -> bool:
"""True iff the failure (anywhere in its chain) actually looks like an
out-of-memory condition the only case where the Flush hint is honest."""
for exc in _exception_chain(e):
if isinstance(exc, MemoryError):
return True
# torch.cuda.OutOfMemoryError subclasses RuntimeError; match by name
# so this needs no torch import (and covers other frameworks' twins).
if type(exc).__name__ == "OutOfMemoryError":
return True
low = str(exc).lower()
if any(sig in low for sig in _OOM_MSG_SIGNATURES):
return True
return False
# #919: an engine that requires a model path / env var which isn't set (or is
# set to a directory missing its model files) fails with a *configuration*
# error, not a runtime one. The reporting user selected sherpa-onnx and hit
# "OMNIVOICE_SHERPA_MODEL not set. Point it to a sherpa-onnx TTS model
# directory …" — a pure setup problem — yet the OOM catch-all told them (on a
# 63 GB-RAM box) to press Flush for memory they never ran out of. Classify the
# whole CLASS of "engine not configured / required env var not set" errors so
# any current or future opt-in engine (sherpa/Confucius4/dots/MOSS …) surfaces
# actionable setup guidance instead of the memory hint. All lowercase; matched
# over the whole exception chain (engines wrap the original error).
_CONFIG_MSG_SIGNATURES = (
"not set. point it to", # sherpa: OMNIVOICE_SHERPA_MODEL not set
"no model.onnx found in", # sherpa: dir set but the model file is missing
"not configured", # generic "engine not configured" wording
"venv not found. set", # confucius4/dots/MOSS dedicated-venv opt-ins
"unavailable: omnivoice_", # is_available() reason wrapped by _ensure_loaded
)
# An OMNIVOICE_* engine env var named alongside "not set" / "point it to" /
# "set omnivoice_…" is the strongest config-missing signal and generalizes to
# any engine gated on such a var (issue #919 class).
_CONFIG_ENV_RE = re.compile(r"omnivoice_[a-z0-9_]+")
def _is_config_failure(e) -> bool:
"""True iff the failure is a *configuration* problem — a required engine
model path / env var that isn't set (or points nowhere) — rather than a
runtime fault. The remedy is to set the value, never to Flush VRAM."""
for exc in _exception_chain(e):
low = str(exc).lower()
if any(sig in low for sig in _CONFIG_MSG_SIGNATURES):
return True
if _CONFIG_ENV_RE.search(low) and (
"not set" in low or "point it to" in low or "set omnivoice_" in low
):
return True
return False
def _oom_friendly_reraise(e):
"""Best-effort cache flush + the user-facing OOM hint shared by both
inference paths."""
import gc
import torch
gc.collect()
if torch.backends.mps.is_available():
torch.mps.empty_cache()
elif torch.cuda.is_available():
torch.cuda.empty_cache()
# #278: don't mislabel a torch.compile/Triton/Inductor crash as an
# out-of-memory condition. (model_manager's generate wrapper already
# retries these eagerly; this only triggers if that retry also died.)
from services.model_manager import _is_compile_runtime_failure
if _is_compile_runtime_failure(e):
raise RuntimeError(
f"TTS engine hit a torch.compile/Triton error (not out of memory). "
f"Disable torch.compile in Settings → Performance, use the Flush "
f"button to reload the model, then regenerate. Underlying error: {e}"
) from e
# #437: a Permission-denied / exec failure (e.g. a bundled engine binary
# that lost its +x bit) is NOT an OOM — don't send the user to the Flush
# button; tell them what's actually wrong.
es = str(e)
if isinstance(e, PermissionError) or "Permission denied" in es or "Errno 13" in es:
raise RuntimeError(
f"A required engine binary couldn't be executed (permission denied). "
f"This usually means a bundled binary lost its execute bit — reinstall, "
f"or run `chmod +x` on the engine binary named in the error. "
f"Underlying error: {e}"
) from e
# #629: a decode/ffmpeg failure on the rendered audio is NOT out of memory —
# it's unreadable audio (usually a transient numerical glitch). Say so rather
# than sending the user down the OOM path.
if "ffmpeg returned error" in es or "Decoding failed" in es or "Invalid data found" in es:
raise RuntimeError(
f"The engine produced unreadable audio (a decode step failed) — this is "
f"usually a transient glitch. Use the Flush button to reload the model, "
f"then regenerate. Underlying error: {e}"
) from e
# #664: a bad voice-design instruct (free-form prose, mixed EN/ZH, or
# conflicting tags) raises "Unsupported instruct items …" / "Cannot mix …
# in a single instruct" / "Conflicting instruct items …" from omnivoice's
# _resolve_instruct. That's a USER-INPUT validation error, not an OOM. Match
# on the message signature (NOT the type — a lower layer can wrap the original
# ValueError, which is why the route's `except ValueError` guard misses it)
# and re-raise as a clean ValueError so the route returns a 400 with the
# instruct guidance, instead of a 500 telling the user to Flush for memory
# they never ran out of. (Complements the client-side guard in #658/#612.)
_low = es.lower()
if ("unsupported instruct items" in _low
or "conflicting instruct items" in _low
or "in a single instruct" in _low):
raise ValueError(es) from e
# #705: a corrupt or wrong-architecture native component (a .dll / .pyd / .exe
# — torch, ffmpeg, or a bundled engine binary) fails to load/spawn on Windows
# with "[WinError 193] %1 is not a valid Win32 application". That is NOT OOM,
# and Flush won't help — reinstalling/repairing the component is the real fix.
if "[winerror 193]" in _low or "is not a valid win32 application" in _low:
raise RuntimeError(
f"A native component (a DLL / .pyd / .exe — e.g. torch, ffmpeg, or an "
f"engine binary) is corrupt or built for the wrong architecture "
f"([WinError 193]). Reinstall or repair that component — the Flush "
f"button won't help here. Underlying error: {e}"
) from e
# #715: a "[Errno 32] Broken pipe" (BrokenPipeError) surfacing from
# generation is NOT out of memory — it means the backend's stdout/stderr
# pipe to the desktop shell that launched it closed mid-render (an orphaned
# backend whose parent shell exited or relaunched). main.py wraps
# sys.stdout/stderr to swallow EPIPE, but a C-level write inside the native
# engine/torch can still raise one past that guard. Flush won't help —
# relaunching the app re-parents the backend to a live shell.
# #756: the GPU's compute capability isn't in this PyTorch build's arch list,
# so CUDA can't launch kernels ("no kernel image is available for execution").
# NOT OOM. get_best_device() now falls back to CPU up front, but classify the
# raw error too in case CUDA was forced (OMNIVOICE_FORCE_CUDA) or a sub-path
# still ran on the GPU — point at the real fix, not the Flush button.
if "no kernel image is available" in _low:
raise RuntimeError(
f"Your GPU isn't supported by the installed PyTorch build (CUDA can't "
f"launch kernels for its compute capability). Switch the compute device "
f"to CPU in Settings, or install a matching PyTorch (e.g. a cu128 build "
f"for newer GPUs). The Flush button won't help. Underlying error: {e}"
) from e
if isinstance(e, BrokenPipeError) or "broken pipe" in _low or "errno 32" in _low:
raise RuntimeError(
f"The backend lost its output pipe mid-generation — the desktop app "
f"that launched it closed or relaunched ([Errno 32] Broken pipe). "
f"Restart the app and try again; the Flush button won't help here. "
f"Underlying error: {e}"
) from e
# #880: an httpx/requests transport failure surfacing from generation —
# most commonly a first-use model download from the HF Hub dying with
# httpx's "Cannot send a request, as the client has been closed" (the
# shared client got closed mid-lifecycle), a connect/read timeout, or a
# dropped connection — is NOT out of memory. The model never finished
# loading, so Flush is the wrong remedy; retrying is. Matched over the
# whole exception chain (type names + stringified signatures) because
# engines wrap the original transport error.
if _is_network_failure(e):
raise RuntimeError(
f"A model download or network call failed mid-generation (usually "
f"the engine fetching its model files on first use). This is a "
f"network problem, not a memory problem — flushing VRAM won't "
f"help. Retry the generation; if it keeps failing, check your "
f"internet connection and any HF_ENDPOINT/mirror setting. "
f"Underlying error: {e}"
) from e
# #919: a required engine model path / env var that isn't set is a pure
# CONFIGURATION problem, not a runtime one. sherpa-onnx's
# "OMNIVOICE_SHERPA_MODEL not set. Point it to …" used to fall through to
# the OOM catch-all, telling a user with 63 GB of RAM to press Flush. Point
# at the real fix — set the variable — and never mention memory or Flush.
# The underlying error already names the exact variable + what to point it
# at (and Settings → Engines shows a copy-paste setup line), so keep it
# front-and-center. Checked before the OOM branch so a config error can
# never be mislabeled as memory.
if _is_config_failure(e):
raise RuntimeError(
f"This TTS engine isn't set up yet — it needs a model path or "
f"environment variable that isn't configured, so nothing was "
f"generated. Set it as the underlying error describes (it names the "
f"exact variable and what to point it at), then restart OmniVoice — "
f"or pick a ready engine in Settings → Engines. This is a setup "
f"problem, not a memory one. Underlying error: {e}"
) from e
# #880 (the class bug): the OOM hint used to be the catch-all fallback,
# so ANY unrecognized error told the user to press Flush for memory they
# never ran out of. Only claim OOM when something in the chain actually
# looks like one; everything else surfaces as what it is — unrecognized —
# with the real error front and center.
if _is_oom_failure(e):
raise RuntimeError(
f"TTS engine stopped mid-generation. This usually means it ran out of memory. "
f"Try the Flush button to reload the model, then regenerate. Underlying error: {e}"
) from e
raise RuntimeError(
f"TTS engine stopped mid-generation with an error OmniVoice doesn't "
f"recognize. Retry once; if it keeps failing, please report it with "
f"the full trace. Underlying error: {e}"
) from e
def _run_inference(
model, text, language, ref_audio_path, ref_text, instruct, duration,
num_step, guidance_scale, speed, t_shift, denoise,
postprocess_output, layer_penalty_factor, position_temperature,
class_temperature, used_seed, effect_preset="broadcast",
max_chunk_chars=None, crossfade_ms=None,
):
from services.audio_dsp import apply_mastering, normalize_audio, apply_effects_chain, get_effect_chain
import torch
try:
if used_seed is not None:
@@ -456,124 +99,53 @@ def _run_inference(
)[0]
audio_out = _render_with_pauses(_gen_span, segments, sr)
else:
# Wave 1.2: long text is split at sentence boundaries and the
# per-chunk audio crossfaded — removes the length ceiling. Short
# text takes the single-shot path below unchanged. [pause] inputs
# keep the dedicated stitcher above (spans are already short).
from services.chunked_tts import (
DEFAULT_CROSSFADE_MS, DEFAULT_MAX_CHUNK_CHARS,
concatenate_audio_chunks, split_text_into_chunks,
audios = model.generate(
text=text, language=language, ref_audio=ref_audio_path,
ref_text=ref_text, instruct=instruct, duration=duration,
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
denoise=denoise, postprocess_output=postprocess_output,
**kwargs
)
_max_chars = DEFAULT_MAX_CHUNK_CHARS if max_chunk_chars is None else max_chunk_chars
_xfade_ms = DEFAULT_CROSSFADE_MS if crossfade_ms is None else crossfade_ms
text_chunks = split_text_into_chunks(text, _max_chars)
if len(text_chunks) > 1:
parts = []
for i, chunk_text in enumerate(text_chunks):
# Vary the seed per chunk (deterministically) to avoid
# correlated RNG artifacts across chunk boundaries.
if used_seed is not None:
torch.manual_seed(used_seed + i)
parts.append(model.generate(
text=chunk_text, language=language, ref_audio=ref_audio_path,
ref_text=ref_text, instruct=instruct, duration=None,
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
denoise=denoise, postprocess_output=postprocess_output,
**kwargs
)[0])
audio_out = concatenate_audio_chunks(parts, sr, _xfade_ms)
else:
audios = model.generate(
text=text, language=language, ref_audio=ref_audio_path,
ref_text=ref_text, instruct=instruct, duration=duration,
num_step=num_step, guidance_scale=guidance_scale, speed=speed,
denoise=denoise, postprocess_output=postprocess_output,
**kwargs
)
audio_out = audios[0]
audio_out = audios[0]
# Apply DSP effect preset. The OmniVoice model never masters its own
# output, so mastering always runs here (unchanged behavior).
return _apply_effect_chain(audio_out, sr, effect_preset)
# Apply DSP effect preset
_effect_preset = effect_preset or "broadcast"
# Validate preset ID
from services.audio_dsp import EFFECT_PRESETS
if _effect_preset not in EFFECT_PRESETS:
raise ValueError(
f"Unknown effect preset: {_effect_preset!r}. "
f"Valid: {list(EFFECT_PRESETS.keys())}"
)
if _effect_preset == "raw":
# Raw: skip all DSP — return raw model output
return audio_out
mastered_audio = apply_mastering(audio_out, sample_rate=sr)
_chain = get_effect_chain(_effect_preset)
if _chain:
mastered_audio = apply_effects_chain(
mastered_audio, sample_rate=sr, chain=_chain,
)
return normalize_audio(mastered_audio, target_dBFS=-2.0)
except ValueError as e:
# Don't wrap validation errors in OOM message
raise e
except Exception as e:
_oom_friendly_reraise(e)
def _run_backend_inference(
backend, text, language, ref_audio_path, ref_text, instruct, duration,
num_step, guidance_scale, speed, denoise, postprocess_output,
used_seed, effect_preset="broadcast",
max_chunk_chars=None, crossfade_ms=None,
):
"""Engine-aware twin of :func:`_run_inference` (issue #312).
Runs the request through a pluggable ``TTSBackend`` adapter instead of the
OmniVoice model directly. The adapter protocol is narrower than the
OmniVoice-native surface engine-specific extras (``t_shift``,
``layer_penalty_factor``, ) only exist on the native path, which is why
OmniVoice itself still goes through ``_run_inference``.
"""
import torch
try:
if used_seed is not None:
torch.manual_seed(used_seed)
if language and language.lower() == "auto":
language = None
gen_kwargs = dict(
language=language, ref_audio=ref_audio_path, ref_text=ref_text,
instruct=instruct, num_step=num_step, guidance_scale=guidance_scale,
speed=speed, denoise=denoise, postprocess_output=postprocess_output,
import gc
gc.collect()
if torch.backends.mps.is_available():
torch.mps.empty_cache()
elif torch.cuda.is_available():
torch.cuda.empty_cache()
raise RuntimeError(
f"TTS engine stopped mid-generation. This usually means it ran out of memory. "
f"Try the Flush button to reload the model, then regenerate. Underlying error: {e}"
)
sr = backend.sample_rate
# Inline [pause Nms] markers (issue #276) work for every engine — the
# silence stitching is model-free.
from omnivoice.utils.text import parse_pause_markers
segments = parse_pause_markers(text)
has_pause = len(segments) > 1 or (segments and segments[0][1] > 0)
if has_pause:
def _gen_span(span_text):
# Per-span duration is left to the engine; an explicit overall
# `duration` can't be meaningfully split across spans.
return backend.generate(span_text, duration=None, **gen_kwargs)
audio_out = _render_with_pauses(_gen_span, segments, sr)
else:
# Wave 1.2: sentence-boundary chunking for long text (see
# _run_inference for the rationale; behavior is identical here).
from services.chunked_tts import (
DEFAULT_CROSSFADE_MS, DEFAULT_MAX_CHUNK_CHARS,
concatenate_audio_chunks, split_text_into_chunks,
)
_max_chars = DEFAULT_MAX_CHUNK_CHARS if max_chunk_chars is None else max_chunk_chars
_xfade_ms = DEFAULT_CROSSFADE_MS if crossfade_ms is None else crossfade_ms
text_chunks = split_text_into_chunks(text, _max_chars)
if len(text_chunks) > 1:
parts = []
for i, chunk_text in enumerate(text_chunks):
if used_seed is not None:
torch.manual_seed(used_seed + i)
parts.append(backend.generate(chunk_text, duration=None, **gen_kwargs))
audio_out = concatenate_audio_chunks(parts, sr, _xfade_ms)
else:
audio_out = backend.generate(text, duration=duration, **gen_kwargs)
return _apply_effect_chain(
audio_out, sr, effect_preset,
skip_mastering=getattr(backend, "applies_own_mastering", False),
)
except ValueError as e:
# Don't wrap validation errors in OOM message
raise e
except Exception as e:
_oom_friendly_reraise(e)
@router.post("/generate")
@@ -596,125 +168,30 @@ async def generate_speech(
profile_id: Optional[str] = Form(None),
seed: Optional[int] = Form(None),
effect_preset: str = Form("broadcast"),
engine: Optional[str] = Form(None),
# Wave 1.2 — unlimited-length generation: long text is split at sentence
# boundaries and crossfaded. 0 disables chunking (whole text to engine).
max_chunk_chars: int = Form(800, ge=0),
crossfade_ms: int = Form(50, ge=0, le=1000),
# Expressive-TTS Spec 01: apply the user pronunciation dictionary + inline
# [[…]] overrides to the text before synthesis. Default ON; the global
# OMNIVOICE_PRONUNCIATION pref can disable it for power users. Omitting it
# with an empty dictionary is byte-identical to legacy behavior.
pronounce: bool = Form(True),
):
# #502: NFC-normalize the input text so decomposed (NFD) diacritics — common
# in pasted Vietnamese and other Latin-with-marks text — are composed to the
# single codepoints the tokenizer/model expect, instead of base-letter +
# combining-mark sequences that render as distorted/garbled speech. NFC is a
# no-op for already-composed text; mirrors the duration estimator
# (utils/duration.py) so the estimate and the synthesis see the same text.
import unicodedata
text = unicodedata.normalize("NFC", text)
# ── Engine resolution (issue #312) ──────────────────────────────────────
# The request runs on the engine selected in Settings (POST /engines/select,
# env var OMNIVOICE_TTS_BACKEND wins), or an explicit per-request `engine`
# override — same pattern as /ws/tts's `engine` field and /v1/audio/speech's
# `model`. Omitting both keeps the historical default (OmniVoice), so
# existing API consumers see no change.
from services.tts_backend import (
OmniVoiceBackend, _mask_hf_tokens, active_backend_id, get_backend_class,
)
engine_id = engine or active_backend_id()
try:
backend_cls = get_backend_class(engine_id)
except ValueError:
raise HTTPException(
status_code=400,
detail=(
f"Unknown TTS engine: {engine_id!r}. "
"See GET /engines/tts for the list of valid engine ids."
),
)
_model = None
_backend = None
if backend_cls is OmniVoiceBackend:
# OmniVoice keeps its native path: it carries the full advanced
# parameter surface (t_shift, layer/position/class controls) that the
# generic adapter protocol doesn't. Byte-identical to the old behavior.
_model = await get_model()
else:
try:
ok, msg = backend_cls.is_available()
except Exception as exc:
ok, msg = False, f"{type(exc).__name__}: {exc}"
if not ok:
raise HTTPException(
status_code=400,
detail=f"TTS engine '{engine_id}' is not available: {_mask_hf_tokens(msg)}",
)
# Reuse the per-process instance cache shared with the engine
# health-check route so weights load once, not per request.
from api.routers.engines import _get_engine_instance
_backend = _get_engine_instance(backend_cls)
# ── Routing gate (#21 — no silent CPU fallback). Computed ONCE per request
# (host caps are constant; the per-request engine= override bypasses the
# /engines/select gate, so this is the only place it's enforced for synth).
from core.device_caps import detect_host_caps
from services.engine_routing import resolve_routing, routing_notice
_routing = resolve_routing(getattr(backend_cls, "gpu_compat", ("cpu",)), detect_host_caps())
if _routing["routing_status"] == "unavailable":
# The engine needs an accelerator this host lacks and has no CPU path.
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
_routing_notice = routing_notice(_routing) # (status, reason) or None
_model = await get_model()
ref_audio_path = None
cleanup_ref = False
used_seed = seed
resolved_profile_id = None
history_mode = None # profile.kind when a profile drives; else inferred at insert
if profile_id:
with db_conn() as conn:
row = conn.execute("SELECT * FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
if row:
resolved_profile_id = profile_id
# `kind` is authoritative (0005): 'design' profiles condition on
# their deterministic rendered sample + instruct; 'clone' on the
# user's reference. Lock always wins (it pins a specific take).
# Rows from pre-0004 DBs mid-upgrade may lack the column → fall
# back to the legacy is_locked/instruct inference.
try:
profile_kind = row["kind"] or "clone"
except (KeyError, IndexError):
profile_kind = "design" if (row["instruct"] and not row["is_locked"] and not row["ref_audio_path"]) else "clone"
history_mode = profile_kind
if row["is_locked"] and row["locked_audio_path"]:
ref_audio_path = os.path.join(VOICES_DIR, row["locked_audio_path"])
if not ref_text:
ref_text = row["ref_text"]
if not instruct:
instruct = _profile_instruct(row)
instruct = row["instruct"]
if used_seed is None and row["seed"] is not None:
used_seed = row["seed"]
elif profile_kind == "design":
# Rendered sample (if present) carries the voice identity;
# instruct alone is the fallback for legacy archetype rows.
ref_audio_path = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
if ref_audio_path and not ref_text and row["ref_text"]:
ref_text = row["ref_text"]
elif row["instruct"] and not row["is_locked"]:
if not instruct:
instruct = _profile_instruct(row)
if used_seed is None and row["seed"] is not None:
used_seed = row["seed"]
elif row["instruct"] and not row["is_locked"] and not row["ref_audio_path"]:
# Legacy design-shaped row (pre-0004 archetype materialization
# failure path): instruct-only conditioning.
if not instruct:
instruct = _profile_instruct(row)
instruct = row["instruct"]
if used_seed is None and row["seed"] is not None:
used_seed = row["seed"]
else:
@@ -727,20 +204,6 @@ async def generate_speech(
used_seed = row["seed"]
if language == "Auto":
language = None
# #533: a profile's stored language must drive generation when the
# request didn't pin one. Without this the German (etc.) archetype
# generates with language=None and the model drifts to English —
# even though the archetype PREVIEW renders correctly (archetypes.py
# passes the language). An EXPLICIT non-Auto request language still
# wins; we only fill the gap. `row` is a sqlite3.Row, so guard the
# column lookup for pre-language DBs mid-upgrade.
if language is None:
try:
prof_lang = row["language"]
except (KeyError, IndexError):
prof_lang = None
if prof_lang and prof_lang != "Auto":
language = prof_lang
elif ref_audio is not None:
try:
with tempfile.NamedTemporaryFile(delete=False, suffix=".wav") as f:
@@ -750,97 +213,16 @@ async def generate_speech(
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
# #308: a transcript-less reference is transcribed with the active ASR
# backend (whisperx / faster-whisper / mlx-whisper) instead of the model's
# built-in transformers pipeline, which cannot load whisper-large-v3-turbo
# on transformers 5.3. On failure ref_text stays None and the model's
# fallback behaves exactly as before.
if ref_audio_path and not ref_text:
from services.asr_backend import transcribe_reference
# Same #730 hang risk as any whisperx transcribe — bound + reset the pool
# so a wedged reference transcribe can't brick the backend. This path is
# best-effort (transcribe_reference returns None on failure → the model's
# built-in ASR fallback), so a timeout degrades to None rather than
# failing the whole generate.
try:
ref_text = await run_on_gpu_pool_guarded(
functools.partial(transcribe_reference, ref_audio_path),
what="Reference transcribe",
)
except GpuJobTimeoutError as e:
logger.warning("reference transcribe hung (%s); using model ASR fallback", e)
ref_text = None
# #526: materialize a concrete seed when none was supplied (and no profile
# pinned one) so the take is reproducible and we can hand it back via the
# X-Seed header for the "keep this seed" control. An explicit request seed
# or a profile's stored seed still wins — used_seed is only filled when it
# is still None here, never overwritten.
if used_seed is None:
used_seed = random.randint(0, 2**31 - 1)
# Expressive-TTS Spec 01: apply the user pronunciation dictionary + inline
# [[…]] one-off overrides to the text, here — AFTER `language` is fully
# resolved (a profile may fill it above) so per-language entries match the
# real render language, and BEFORE the text reaches either inference path
# (native OmniVoice or a pluggable backend) and the chunk splitter. This is
# the single point user text → normalized text → model, so the transform
# covers generate for every engine. Pure text substitution → identical on
# mac/Win/Linux. A disabled pref or empty dictionary is a pass-through, so
# plain text stays byte-identical (#G5 backward-compat).
from core import prefs as _prefs
_pron_env = os.environ.get("OMNIVOICE_PRONUNCIATION")
if _pron_env is not None:
# Env wins (power-user override); "0"/"false"/"no"/"off" disable it.
_pron_enabled = _pron_env.strip().lower() not in ("0", "false", "no", "off", "")
else:
_pron_enabled = bool(_prefs.get("pronunciation_enabled", True))
if pronounce and _pron_enabled:
from services.pronunciation import apply_pronunciation, load_entries_from_db
try:
_pron_rows = load_entries_from_db()
except Exception: # noqa: BLE001 — table missing / DB locked → no-op
_pron_rows = []
text = apply_pronunciation(text, _pron_rows, language)
else:
# Even with the dictionary off, inline [[…]] overrides are an explicit,
# in-text authoring choice → always honored (and never left as literal
# double-bracket text the model would mispronounce).
from services.pronunciation import apply_inline_overrides
text = apply_inline_overrides(text)
start_time = time.time()
try:
loop = asyncio.get_running_loop()
if _backend is not None:
# Bounded + pool-reset on hang so a wedged generate can't starve the
# GPU pool and brick the backend ("can't reach backend", #730 class).
audio_tensor = await run_on_gpu_pool_guarded(
functools.partial(
_run_backend_inference,
_backend, text, language, ref_audio_path, ref_text, instruct,
duration, num_step, guidance_scale, speed, denoise,
postprocess_output, used_seed, effect_preset,
max_chunk_chars, crossfade_ms,
),
what="TTS generate",
)
# Read after generation: engines with lazy model loading report
# their real rate only once weights are up.
sample_rate = _backend.sample_rate
else:
audio_tensor = await run_on_gpu_pool_guarded(
functools.partial(
_run_inference,
_model, text, language, ref_audio_path, ref_text, instruct, duration,
num_step, guidance_scale, speed, t_shift, denoise,
postprocess_output, layer_penalty_factor, position_temperature,
class_temperature, used_seed, effect_preset,
max_chunk_chars, crossfade_ms,
),
what="TTS generate",
)
sample_rate = _model.sampling_rate
audio_tensor = await loop.run_in_executor(
_gpu_pool, _run_inference,
_model, text, language, ref_audio_path, ref_text, instruct, duration,
num_step, guidance_scale, speed, t_shift, denoise,
postprocess_output, layer_penalty_factor, position_temperature,
class_temperature, used_seed, effect_preset,
)
# Invisible AudioSeal provenance watermark on the final audio. Embedding
# was previously only wired into the dub pipeline (dub_generate.py), so
# plain TTS came out unmarked despite the setting being on. embed_watermark
@@ -849,44 +231,28 @@ async def generate_speech(
# generation.
from services.watermark import embed_watermark
audio_tensor = await loop.run_in_executor(
_gpu_pool, embed_watermark, audio_tensor, sample_rate
_gpu_pool, embed_watermark, audio_tensor, _model.sampling_rate
)
gen_time = round(time.time() - start_time, 2)
audio_id = str(uuid.uuid4())[:8]
audio_filename = f"{audio_id}.wav"
audio_path = os.path.join(OUTPUTS_DIR, audio_filename)
_safe_torchaudio_save(audio_path, audio_tensor, sample_rate)
_safe_torchaudio_save(audio_path, audio_tensor, _model.sampling_rate)
audio_dur = round(audio_tensor.shape[-1] / sample_rate, 2)
audio_dur = round(audio_tensor.shape[-1] / _model.sampling_rate, 2)
# #710: the clip is already generated and saved above. A history-write
# failure — e.g. "no such table: generation_history" on a DB that missed
# schema init — must NOT 500 the user's generation. Self-heal the schema
# once and retry; if it still fails, log and return the audio anyway.
def _write_history():
with db_conn() as conn:
conn.execute(
"INSERT INTO generation_history (id, text, mode, language, instruct, profile_id, audio_path, duration_seconds, generation_time, seed, created_at) VALUES (?,?,?,?,?,?,?,?,?,?,?)",
(audio_id, text[:200], history_mode or ("clone" if ref_audio_path else "design"),
language or "Auto", instruct or "", resolved_profile_id,
audio_filename, audio_dur, gen_time, used_seed, time.time())
)
try:
_write_history()
except sqlite3.OperationalError as e:
logger.warning("generation history write failed (%s); healing schema + retrying", e)
try:
ensure_schema()
_write_history()
except Exception as e2:
logger.warning("history write still failed after schema heal; returning audio anyway: %s", e2)
except Exception as e:
logger.warning("generation history write failed; returning audio anyway: %s", e)
with db_conn() as conn:
conn.execute(
"INSERT INTO generation_history (id, text, mode, language, instruct, profile_id, audio_path, duration_seconds, generation_time, seed, created_at) VALUES (?,?,?,?,?,?,?,?,?,?,?)",
(audio_id, text[:200], "clone" if ref_audio_path else "design",
language or "Auto", instruct or "", resolved_profile_id,
audio_filename, audio_dur, gen_time, used_seed, time.time())
)
event_bus.emit("generation_history", {"action": "created", "id": audio_id})
buffer = io.BytesIO()
_safe_torchaudio_save(buffer, audio_tensor, sample_rate, format="wav")
_safe_torchaudio_save(buffer, audio_tensor, _model.sampling_rate, format="wav")
buffer.seek(0)
wav_bytes = buffer.read()
@@ -895,35 +261,20 @@ async def generate_speech(
for i in range(0, len(wav_bytes), chunk_size):
yield wav_bytes[i:i + chunk_size]
_resp_headers = {
"X-Audio-Id": audio_id,
"X-Gen-Time": str(gen_time),
"X-Audio-Path": audio_filename,
"X-Seed": str(used_seed) if used_seed is not None else "",
"X-Audio-Duration": str(audio_dur),
"Content-Length": str(len(wav_bytes)),
}
# Routing notice (#21): cpu_fallback or accelerated-with-caveat only;
# the WAV body is binary so the header channel is the carrier.
if _routing_notice:
from services.engine_routing import header_safe_reason
_resp_headers["X-OmniVoice-Routing"] = _routing_notice[0]
_hr = header_safe_reason(_routing_notice[1])
if _hr:
_resp_headers["X-OmniVoice-Routing-Reason"] = _hr
return StreamingResponse(
_stream_wav(),
media_type="audio/wav",
headers=_resp_headers,
headers={
"X-Audio-Id": audio_id,
"X-Gen-Time": str(gen_time),
"X-Audio-Path": audio_filename,
"X-Seed": str(used_seed) if used_seed is not None else "",
"X-Audio-Duration": str(audio_dur),
"Content-Length": str(len(wav_bytes)),
}
)
except HTTPException:
raise
except GpuJobTimeoutError as e:
# A wedged GPU generate — the pool was already reset to restore capacity
# (#730 class). Report the actionable timeout instead of the misleading
# "can't reach backend" the frontend shows when the pool starves.
logger.error("Generate timed out: %s", e)
raise HTTPException(status_code=503, detail=str(e)) from e
except ValueError as e:
logger.error("Validation failed: %s", e)
raise HTTPException(status_code=400, detail=str(e)) from e
@@ -957,29 +308,9 @@ def _safe_output_path(name):
@router.get("/history")
def list_history():
"""Newest 50 generations whose audio still exists on disk.
Rows whose WAV was deleted out-of-band (cleared outputs dir, manual
cleanup) used to come back anyway and render dead players that 404 on
every fetch; prune them here so the UI never sees them again."""
with db_conn() as conn:
rows = conn.execute(
"SELECT * FROM generation_history ORDER BY created_at DESC LIMIT 50"
).fetchall()
alive, stale_ids = [], []
for r in rows:
p = _safe_output_path(r["audio_path"]) if r["audio_path"] else None
if r["audio_path"] and (not p or not os.path.exists(p)):
stale_ids.append(r["id"])
else:
alive.append(dict(r))
if stale_ids:
conn.executemany(
"DELETE FROM generation_history WHERE id=?",
[(i,) for i in stale_ids],
)
logger.info("pruned %d stale history rows (audio file gone)", len(stale_ids))
return alive
rows = conn.execute("SELECT * FROM generation_history ORDER BY created_at DESC LIMIT 50").fetchall()
return [dict(r) for r in rows]
@router.delete("/history")
def clear_history():
+9 -24
View File
@@ -189,21 +189,15 @@ def auto_extract(project_id: str, req: AutoExtractRequest):
Writes them as `auto=1` rows. Existing terms with the same (source,target)
are NOT duplicated. Returns the full current glossary after the pass.
"""
# Resolved through the LLM Skills registry so auto-extract can be toggled
# or routed to its own provider (Settings → LLM Skills) independently of
# the translation pipeline. None == disabled or no provider configured.
from services import llm_skills
from services.translator import _llm_client, _llm_model, _llm_timeout # reuse same client
handle = llm_skills.resolve_skill_client("glossary_extract")
if handle is None:
client = _llm_client()
if client is None:
raise HTTPException(
status_code=503,
detail=(
"Auto-extract needs an LLM. Set one up in Settings → LLM Providers "
"(pick a provider, add its key, choose a model, Test) — or use local "
"Ollama / LM Studio for a fully offline setup — and make sure the "
"Glossary auto-extract skill is enabled in Settings → LLM Skills, "
"then try again."
"Auto-extract needs an LLM. Set TRANSLATE_BASE_URL + TRANSLATE_API_KEY "
"(Ollama works locally: base_url=http://localhost:11434/v1) and try again."
),
)
@@ -226,9 +220,9 @@ def auto_extract(project_id: str, req: AutoExtractRequest):
)
try:
res = handle.client.chat.completions.create(
model=handle.model,
timeout=handle.timeout,
res = client.chat.completions.create(
model=_llm_model(),
timeout=_llm_timeout(),
messages=[
{"role": "system", "content": system},
{"role": "user", "content": user},
@@ -237,18 +231,9 @@ def auto_extract(project_id: str, req: AutoExtractRequest):
body = (res.choices[0].message.content or "").strip()
except Exception as e:
logger.warning("auto-extract LLM call failed: %s", e)
# Scrub the provider error — some OpenAI-compatible providers echo the
# API key or a user_id in the body, which must not reach the UI verbatim.
from core.scrub import scrub_provider_error
from services import llm_providers
_p = llm_providers.active_provider()
_key = llm_providers.resolve_api_key(_p) if _p else None
raise HTTPException(
status_code=502,
detail=(
"LLM didn't respond. Check Settings → Logs → Backend for the trace. "
f"Error: {scrub_provider_error(e, _key)}"
),
detail=f"LLM didn't respond. Check Settings → Logs → Backend for the trace. Error: {e}",
)
# Parse: SOURCE || TARGET || note (lines are allowed to be sloppy — we're forgiving).
-158
View File
@@ -1,158 +0,0 @@
"""Longform Job Library (PR 7).
``GET /longform/jobs`` list finished Audiobook + Story renders so the user can
re-download them from the Projects view. The render itself (the m4b/mp3) already
landed in ``OUTPUTS_DIR`` and is served at ``/audio/<output>``; here we just
recover, from each finished job's persisted SSE tail, the output filename plus
the chapter count and duration the ``done`` event carried.
Pure recovery, no synthesis. Defensive by construction: a job whose ``done``
event is missing or unparseable is skipped, never surfaced and never a 500.
The work lives in :func:`build_longform_library`, a pure function over the
job-store callables, so it's unit-testable without importing ``main`` (and the
torch graph behind it).
"""
from __future__ import annotations
import json
import logging
from typing import Callable, Optional
from fastapi import APIRouter, Query
logger = logging.getLogger("omnivoice.longform_jobs")
router = APIRouter()
#: Job types this library surfaces. Both flow through the shared longform
#: renderer (``_render_longform_sse``) and emit the same ``done`` event shape.
_LONGFORM_TYPES = ("audiobook", "story")
def _done_payload_from_events(events: list[dict]) -> Optional[dict]:
"""Recover the final ``{"type": "done", ...}`` payload from a job's SSE tail.
Each row's ``payload`` is the JSON the renderer stored via
``job_store.append_event(job_id, json.dumps(payload))``. We scan newest-first
and return the first parseable ``done`` event. Anything malformed is skipped
this never raises.
"""
for ev in reversed(events):
raw = ev.get("payload") if isinstance(ev, dict) else None
if not raw or not isinstance(raw, str):
continue
try:
obj = json.loads(raw)
except (ValueError, TypeError):
continue
if isinstance(obj, dict) and obj.get("type") == "done":
return obj
return None
def _coerce_int(value, default: int = 0) -> int:
try:
return int(value)
except (TypeError, ValueError):
return default
def _coerce_float(value, default: float = 0.0) -> float:
try:
return float(value)
except (TypeError, ValueError):
return default
def build_longform_library(
list_jobs: Callable[..., list[dict]],
events_since: Callable[..., list[dict]],
*,
limit: int = 50,
) -> list[dict]:
"""Build the newest-first list of finished longform renders.
Pure over the two job-store callables so tests can pass them directly:
* ``list_jobs(status="done", limit=...)`` all done jobs, newest-first.
* ``events_since(job_id)`` that job's persisted SSE events.
Returns ``[{job_id, type, title?, output, duration_s, chapters,
created_at}]``. Jobs that aren't a longform type, or whose ``done`` event /
output filename can't be recovered, are silently skipped — the library only
ever lists things the user can actually re-download.
"""
limit = max(1, min(_coerce_int(limit, 50), 500))
try:
# Over-fetch: non-longform done jobs (dub, etc.) get filtered out below,
# so ask for more rows than the caller's limit to still fill the page.
rows = list_jobs(status="done", limit=limit * 4)
except Exception:
logger.warning("longform library: list_jobs failed", exc_info=True)
return []
out: list[dict] = []
for row in rows or []:
if len(out) >= limit:
break
try:
job_type = row.get("type")
job_id = row.get("id")
if job_type not in _LONGFORM_TYPES or not job_id:
continue
try:
events = events_since(job_id)
except Exception:
logger.warning("longform library: events_since failed for %s",
job_id, exc_info=True)
continue
done = _done_payload_from_events(events or [])
if not done:
continue
output = done.get("output")
if not output or not isinstance(output, str):
continue # nothing to re-download → not worth listing
item = {
"job_id": job_id,
"type": job_type,
"output": output,
"duration_s": round(_coerce_float(done.get("duration_s")), 2),
"chapters": _coerce_int(done.get("chapters")),
"created_at": row.get("created_at"),
}
# Title is optional — prefer the done event, fall back to job meta.
title = done.get("title")
if not title:
meta_raw = row.get("meta_json")
if isinstance(meta_raw, str) and meta_raw:
try:
meta = json.loads(meta_raw)
if isinstance(meta, dict):
title = meta.get("title")
except (ValueError, TypeError):
title = None
if title:
item["title"] = title
out.append(item)
except Exception:
# Per-row isolation: one bad row never sinks the whole list.
logger.warning("longform library: skipping unparseable job row",
exc_info=True)
continue
return out
@router.get("/longform/jobs")
def longform_jobs(limit: int = Query(50, ge=1, le=500)) -> dict:
"""Finished Audiobook + Story renders, newest-first, ready to re-download.
Each item's ``output`` is served at ``/audio/<output>``. Never 500s — on any
backend hiccup it returns an empty list rather than an error.
"""
from core import job_store
jobs = build_longform_library(
job_store.list_jobs, job_store.events_since, limit=limit,
)
return {"jobs": jobs}
+28 -40
View File
@@ -58,31 +58,6 @@ MAX_BUNDLE_BYTES = 100 * 1024 * 1024
# ── Export ──────────────────────────────────────────────────────────────────
def _bundle_metadata(profile: dict, **extra) -> dict:
"""Common .omnivoice metadata for export + publish.
Captures ``kind`` and ``vd_states`` so a *designed* persona survives the
bundle round-trip as a design (not silently demoted to a clone) required
for the synthetic-only gating of the persona gallery (§R3). Old bundles
without these keys import as ``kind='clone'`` (backward-compatible).
"""
meta = {
"bundle_version": BUNDLE_VERSION,
"profile_name": profile.get("name", "Unnamed"),
"ref_text": profile.get("ref_text", ""),
"instruct": profile.get("instruct", ""),
"language": profile.get("language", "Auto"),
"personality": profile.get("personality", ""),
"seed": profile.get("seed"),
"kind": profile.get("kind") or "clone",
"vd_states": profile.get("vd_states"),
"is_locked": bool(profile.get("is_locked")),
"omnivoice_version": APP_VERSION,
}
meta.update(extra)
return meta
@router.post("/export/{profile_id}")
def export_profile(profile_id: str):
"""Export a voice profile as a downloadable .omnivoice bundle (ZIP)."""
@@ -100,9 +75,19 @@ def export_profile(profile_id: str):
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
# Metadata
metadata = _bundle_metadata(
profile, created_at=profile.get("created_at"), exported_at=time.time(),
)
metadata = {
"bundle_version": BUNDLE_VERSION,
"profile_name": profile.get("name", "Unnamed"),
"ref_text": profile.get("ref_text", ""),
"instruct": profile.get("instruct", ""),
"language": profile.get("language", "Auto"),
"personality": profile.get("personality", ""),
"seed": profile.get("seed"),
"is_locked": bool(profile.get("is_locked")),
"created_at": profile.get("created_at"),
"exported_at": time.time(),
"omnivoice_version": APP_VERSION,
}
zf.writestr("metadata.json", json.dumps(metadata, indent=2))
# Reference audio
@@ -206,9 +191,8 @@ async def import_profile(
conn.execute(
"""INSERT INTO voice_profiles
(id, name, ref_audio_path, ref_text, instruct, language,
seed, personality, is_locked, locked_audio_path, created_at,
kind, vd_states)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
seed, personality, is_locked, locked_audio_path, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(
profile_id,
metadata.get("profile_name", "Imported Voice"),
@@ -221,10 +205,6 @@ async def import_profile(
1 if is_locked else 0,
locked_audio_filename or "",
time.time(),
# Preserve the design/clone distinction across the round-trip;
# old bundles without these keys import as a clone.
metadata.get("kind") or "clone",
metadata.get("vd_states"),
),
)
@@ -273,11 +253,19 @@ def publish_to_marketplace(
# Build the bundle
with zipfile.ZipFile(str(bundle_path), "w", zipfile.ZIP_DEFLATED) as zf:
metadata = _bundle_metadata(
profile,
tags=[t.strip() for t in tags.split(",") if t.strip()],
published_at=time.time(),
)
metadata = {
"bundle_version": BUNDLE_VERSION,
"profile_name": profile.get("name", "Unnamed"),
"ref_text": profile.get("ref_text", ""),
"instruct": profile.get("instruct", ""),
"language": profile.get("language", "Auto"),
"personality": profile.get("personality", ""),
"seed": profile.get("seed"),
"is_locked": bool(profile.get("is_locked")),
"tags": [t.strip() for t in tags.split(",") if t.strip()],
"published_at": time.time(),
"omnivoice_version": APP_VERSION,
}
zf.writestr("metadata.json", json.dumps(metadata, indent=2))
ref_path = profile.get("ref_audio_path")
-53
View File
@@ -1,53 +0,0 @@
"""REST CRUD for per-agent MCP voice bindings (Wave 2.2 / Spec 2).
Loopback-gated the Settings UI manages bindings here. The MCP tools
themselves resolve voices via ``services.mcp_bindings.resolve_voice``.
"""
from __future__ import annotations
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from api.dependencies import require_loopback
from services import mcp_bindings
router = APIRouter(
prefix="/api/mcp",
tags=["mcp"],
dependencies=[Depends(require_loopback)],
)
class _BindingBody(BaseModel):
client_id: str = Field(..., min_length=1, max_length=128)
label: str | None = None
profile_id: str | None = None
default_engine: str | None = None
@router.get("/bindings")
def list_bindings():
"""All per-agent voice bindings, most-recently-seen first."""
return mcp_bindings.list_bindings()
@router.put("/bindings")
def upsert_binding(body: _BindingBody):
"""Create or update the binding for an MCP client id."""
try:
return mcp_bindings.upsert_binding(
body.client_id,
label=body.label,
profile_id=body.profile_id,
default_engine=body.default_engine,
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@router.delete("/bindings/{client_id}")
def delete_binding(client_id: str):
if not mcp_bindings.delete_binding(client_id):
raise HTTPException(status_code=404, detail="No binding for that client id")
return {"deleted": client_id}
+12 -83
View File
@@ -22,6 +22,7 @@ from __future__ import annotations
import io
import logging
import os
import asyncio
import tempfile
from typing import Literal, Optional
@@ -29,7 +30,7 @@ from fastapi import APIRouter, File, Form, HTTPException, UploadFile
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
from services.model_manager import _gpu_pool, run_on_gpu_pool_guarded
from services.model_manager import _gpu_pool
logger = logging.getLogger("omnivoice.openai_compat")
@@ -81,33 +82,6 @@ class SpeechRequest(BaseModel):
"E.g. 'young female, warm tone, slight British accent'.",
)
instruct: Optional[str] = Field(default=None, description="Style instruction for the TTS engine.")
duration: Optional[float] = Field(
default=None,
gt=0,
description="OmniVoice extension: target output duration in seconds.",
)
seed: Optional[int] = Field(
default=None,
description="OmniVoice extension: deterministic sampling seed.",
)
denoise: bool = Field(
default=True,
description="OmniVoice extension: prepend denoise control when supported.",
)
preprocess_prompt: bool = Field(
default=True,
description="OmniVoice extension: trim/preprocess reference prompt when supported.",
)
chunk_duration: Optional[float] = Field(
default=None,
ge=0,
description="OmniVoice GGUF extension: long-form internal chunk duration.",
)
chunk_threshold: Optional[float] = Field(
default=None,
ge=0,
description="OmniVoice GGUF extension: long-form internal chunk threshold.",
)
class TranscriptionResponse(BaseModel):
@@ -235,14 +209,7 @@ def _run_tts(backend, text: str, kw: dict):
from services.audio_dsp import apply_mastering, normalize_audio
wav = backend.generate(text, **kw)
sr = backend.sample_rate
# Engines that already emit mastered, studio-grade audio (e.g. VoxCPM2's
# native 48 kHz) opt out of apply_mastering via `applies_own_mastering`.
# That chain's Compressor + 8% Reverb is tuned for OmniVoice's 24 kHz clone
# output; applied to a studio engine it adds an audible level pump and a
# reverb tail that degrade the very output we want clean. Loudness
# normalisation still runs — it's a benign peak scale, not dynamics.
if not getattr(backend, "applies_own_mastering", False):
wav = apply_mastering(wav, sample_rate=sr)
wav = apply_mastering(wav, sample_rate=sr)
wav = normalize_audio(wav, target_dBFS=-2.0)
return wav, sr
@@ -252,28 +219,10 @@ async def create_speech(req: SpeechRequest):
"""Generate audio from text. Compatible with OpenAI's POST /v1/audio/speech."""
backend = _resolve_engine(req.model)
# Routing gate (#21 — no silent CPU fallback), identical to REST /generate.
from core.device_caps import detect_host_caps
from services.engine_routing import resolve_routing, routing_notice
_routing = resolve_routing(getattr(backend, "gpu_compat", ("cpu",)), detect_host_caps())
if _routing["routing_status"] == "unavailable":
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
_routing_notice = routing_notice(_routing) # (status, reason) or None
# Build kwargs for the backend's generate() method
kw: dict = {
"speed": req.speed,
"denoise": req.denoise,
"preprocess_prompt": req.preprocess_prompt,
}
if req.duration is not None:
kw["duration"] = req.duration
if req.seed is not None:
kw["seed"] = req.seed
if req.chunk_duration is not None:
kw["chunk_duration"] = req.chunk_duration
if req.chunk_threshold is not None:
kw["chunk_threshold"] = req.chunk_threshold
if req.language:
kw["language"] = req.language
if req.instruct:
@@ -302,8 +251,6 @@ async def create_speech(req: SpeechRequest):
kw["ref_text"] = row["ref_text"]
if row["instruct"] and not req.instruct:
kw["instruct"] = row["instruct"]
if req.seed is None and row["seed"] is not None:
kw["seed"] = row["seed"]
else:
# Not a profile ID — forward as engine preset name
kw["voice"] = voice
@@ -312,30 +259,21 @@ async def create_speech(req: SpeechRequest):
kw["voice"] = voice
try:
# Bounded + pool-reset on hang so a wedged TTS request can't starve the
# GPU pool and brick the backend (#730 class).
wav, sr = await run_on_gpu_pool_guarded(
lambda: _run_tts(backend, req.input, kw), what="OpenAI TTS generate")
loop = asyncio.get_running_loop()
wav, sr = await loop.run_in_executor(_gpu_pool, _run_tts, backend, req.input, kw)
except Exception as e:
logger.exception("OpenAI TTS failed: %s", e)
raise HTTPException(status_code=500, detail=str(e))
audio_bytes, mime_type, ext = _encode_audio(wav, sr, req.response_format)
_headers = {
"Content-Length": str(len(audio_bytes)),
"Content-Disposition": f'inline; filename="speech.{ext}"',
}
if _routing_notice:
from services.engine_routing import header_safe_reason
_headers["X-OmniVoice-Routing"] = _routing_notice[0]
_hr = header_safe_reason(_routing_notice[1])
if _hr:
_headers["X-OmniVoice-Routing-Reason"] = _hr
return StreamingResponse(
io.BytesIO(audio_bytes),
media_type=mime_type,
headers=_headers,
headers={
"Content-Length": str(len(audio_bytes)),
"Content-Disposition": f'inline; filename="speech.{ext}"',
},
)
@@ -385,15 +323,12 @@ async def create_transcription(
try:
backend = get_active_asr_backend()
# Run transcription in the thread pool to avoid blocking the event loop,
# bounded so a stuck/starved ASR returns a 504 with guidance instead of
# hanging the request forever (see run_transcribe_guarded).
from services.asr_backend import run_transcribe_guarded
# Run transcription in the thread pool to avoid blocking the event loop
loop = asyncio.get_running_loop()
word_ts = response_format == "verbose_json"
result = await run_transcribe_guarded(
result = await loop.run_in_executor(
_gpu_pool,
lambda: backend.transcribe(tmp_path, word_timestamps=word_ts),
what="OpenAI",
)
# Extract the full text from segments
@@ -460,12 +395,6 @@ async def create_transcription(
# Default: json
return TranscriptionResponse(text=full_text)
except HTTPException:
raise
except TimeoutError as e:
# ASRTimeoutError (subclass): backend alive, ASR too heavy for compute.
logger.warning("OpenAI transcription timed out: %s", e)
raise HTTPException(status_code=504, detail=str(e))
except Exception as e:
logger.exception("OpenAI transcription failed: %s", e)
raise HTTPException(status_code=500, detail=str(e))
-332
View File
@@ -1,332 +0,0 @@
"""HTTP layer for the `.ovsvoice` portable persona format (#29 / parity §R3 G1).
Thin router over `services.persona_bundle`:
POST /personas/export/{profile_id} stream a downloadable .ovsvoice
POST /personas/import create a profile from a bundle
POST /personas/inspect read a bundle's manifest, no writes
Mirrors the legacy `.omnivoice` endpoints (`marketplace.py`) and reuses the
same path-confinement (`_voices_path`) + consent floor. `.ovsvoice` is additive;
`.omnivoice` import stays a compatible legacy reader.
"""
from __future__ import annotations
import asyncio
import functools
import logging
import os
import time
import uuid
from fastapi import APIRouter, File, HTTPException, Query, UploadFile
from fastapi.responses import StreamingResponse
from core import event_bus
from core.config import VOICES_DIR # noqa: F401 — re-exported for tests/monkeypatch
from core.db import db_conn
from core.version import APP_VERSION
from services import persona_bundle as pb
router = APIRouter()
logger = logging.getLogger("omnivoice.personas")
def _safe_name(name: str, profile_id: str) -> str:
"""Sanitised download filename stem (marketplace idiom); empty → persona_<id>."""
cleaned = "".join(
c if c.isalnum() or c in "-_ " else "" for c in (name or "")
).strip().replace(" ", "_")[:40]
return cleaned or f"persona_{profile_id}"
# ── Export ────────────────────────────────────────────────────────────────
@router.post("/personas/export/{profile_id}")
async def export_persona(
profile_id: str,
license_spdx: str = Query(pb.DEFAULT_LICENSE),
tags: str = Query(""),
include_reference: bool = Query(True),
):
"""Build + stream a `.ovsvoice` bundle for a profile."""
with db_conn() as conn:
row = conn.execute(
"SELECT * FROM voice_profiles WHERE id = ?", (profile_id,)
).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Voice profile not found")
profile = dict(row)
tag_list = [t.strip() for t in tags.split(",") if t.strip()]
# #693: if OMNIVOICE_MODEL is set, record the *resolved* checkpoint in the
# exported bundle so a leaked engine id (e.g. "omnivoice") can't be baked in;
# keep "" when unset (the bundle's "engine unspecified" marker).
from services.model_manager import resolve_omnivoice_checkpoint
engine_id = resolve_omnivoice_checkpoint() if os.environ.get("OMNIVOICE_MODEL", "").strip() else ""
try:
loop = asyncio.get_running_loop()
content = await loop.run_in_executor(
None,
functools.partial(
pb.build_persona_bundle,
profile,
license_spdx=license_spdx,
tags=tag_list,
include_reference=include_reference,
engine_id=engine_id,
omnivoice_version=APP_VERSION,
),
)
except pb.NoPreviewSource:
raise HTTPException(
status_code=503,
detail="This profile has no readable reference or locked audio to "
"build a preview from — re-create or re-import it.",
)
except Exception:
logger.exception("persona export failed for %s", profile_id)
raise HTTPException(
status_code=503,
detail="Could not build the persona bundle — see Settings → Logs.",
)
filename = f"{_safe_name(profile.get('name'), profile_id)}.ovsvoice"
from io import BytesIO
return StreamingResponse(
BytesIO(content),
media_type="application/zip",
headers={
"Content-Disposition": f'attachment; filename="{filename}"',
"Content-Length": str(len(content)),
},
)
# ── Import ────────────────────────────────────────────────────────────────
def _voices_dest(filename: str) -> str:
"""Resolve an output filename inside VOICES_DIR; 400 on escape (belt+braces —
the name is always server-generated `{profile_id}`)."""
from api.routers.profiles import _voices_path
path = _voices_path(filename)
if path is None:
raise HTTPException(status_code=400, detail="Invalid profile id")
return path
def _consent_verified(parsed: pb.ParsedPersona, consent_path: str | None) -> bool:
"""B12-B16: trust verified-own-voice ONLY with a real recording (≥ floor) AND
non-empty consent_text AND a consent.json present. The manifest flag alone
can't forge it."""
if not parsed.consent or not consent_path:
return False
if os.path.getsize(consent_path) < pb._MIN_CONSENT_AUDIO_BYTES:
return False
return bool((parsed.consent.get("consent_text") or "").strip())
@router.post("/personas/import")
async def import_persona(file: UploadFile = File(...)):
"""Create a new voice profile from a `.ovsvoice` (or legacy `.omnivoice`) bundle."""
name = (file.filename or "").lower()
if not name.endswith(".ovsvoice") and not name.endswith(".omnivoice"):
raise HTTPException(status_code=400, detail="File must be a .ovsvoice or .omnivoice bundle")
content = await file.read()
try:
parsed = pb.parse_persona_bundle(content)
except pb.BundleError as e:
raise HTTPException(status_code=e.status, detail=e.detail)
persona = parsed.manifest.get("persona") or {}
written: list[str] = []
def _gen_id() -> str:
return str(uuid.uuid4())[:8]
profile_id = _gen_id()
try:
# ── Audio members → server-named files (never the member name). ──
ref_filename = None
locked_filename = None
if "ref_audio" in parsed.members:
ref_filename = f"{profile_id}{parsed.member_ext('ref_audio')}"
dest = _voices_dest(ref_filename)
parsed.extract_member("ref_audio", dest); written.append(dest)
if "locked_audio" in parsed.members:
locked_filename = f"{profile_id}_locked{parsed.member_ext('locked_audio')}"
dest = _voices_dest(locked_filename)
parsed.extract_member("locked_audio", dest); written.append(dest)
# Preview-only bundle (A12/B8): use the preview as the usable ref clip.
if ref_filename is None and locked_filename is None and "preview" in parsed.members:
ref_filename = f"{profile_id}{parsed.member_ext('preview')}"
dest = _voices_dest(ref_filename)
parsed.extract_member("preview", dest); written.append(dest)
if ref_filename is None and locked_filename is None:
raise HTTPException(status_code=400, detail="bundle has no usable audio")
# ── Consent recording (optional) ──
consent_filename = None
consent_path = None
if "consent_audio" in parsed.members:
consent_filename = f"{profile_id}_consent{parsed.member_ext('consent_audio')}"
consent_path = _voices_dest(consent_filename)
parsed.extract_member("consent_audio", consent_path); written.append(consent_path)
verified = _consent_verified(parsed, consent_path)
consent_text = ((parsed.consent or {}).get("consent_text") or "").strip()
recorded_at = None
if verified:
try:
recorded_at = float(parsed.consent.get("recorded_at"))
except (TypeError, ValueError):
recorded_at = time.time()
is_locked = bool(persona.get("is_locked") and locked_filename)
ref_for_db = ref_filename or locked_filename # at least one is set
def _insert(pid: str):
with db_conn() as conn:
conn.execute(
"""INSERT INTO voice_profiles
(id, name, ref_audio_path, ref_text, instruct, language,
seed, personality, is_locked, locked_audio_path, created_at,
kind, vd_states,
verified_own_voice, consent_text, consent_audio_path, consent_recorded_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(
pid,
persona.get("name") or "Imported Voice",
ref_for_db,
persona.get("ref_text", ""),
persona.get("instruct", ""),
persona.get("language", "Auto"),
persona.get("seed"),
persona.get("personality", ""),
1 if is_locked else 0,
locked_filename or "",
time.time(),
persona.get("kind") or "clone",
persona.get("vd_states"),
1 if verified else 0,
# Keep the attestation text so the user can re-attest locally,
# even when imported unverified.
consent_text,
consent_filename if verified else "",
recorded_at if verified else None,
),
)
import sqlite3
try:
_insert(profile_id)
except sqlite3.IntegrityError:
profile_id = _gen_id() # one retry on id collision (B20)
# rename the on-disk files to the new id so they still match the row
written = _rename_for_new_id(written, profile_id)
ref_for_db = _retarget(ref_for_db, profile_id)
locked_filename = _retarget(locked_filename, profile_id)
consent_filename = _retarget(consent_filename, profile_id)
_insert(profile_id)
except HTTPException:
_cleanup(written)
raise
except Exception:
_cleanup(written)
logger.exception("persona import failed")
raise HTTPException(status_code=500, detail="Import failed; no files were kept.")
event_bus.emit("profiles", {"action": "created", "id": profile_id})
logger.info("Imported persona %r as %s (verified=%s)", persona.get("name"), profile_id, verified)
return {
"success": True,
"profile_id": profile_id,
"name": persona.get("name") or "Imported Voice",
"kind": persona.get("kind") or "clone",
"verified_own_voice": verified,
"preview_only": parsed.preview_only,
"license_spdx": parsed.license_spdx,
"watermarked_preview": parsed.watermarked_preview,
"source_bundle": file.filename,
"schema_version_ahead": parsed.schema_version_ahead,
}
def _cleanup(paths: list[str]) -> None:
for p in paths:
try:
if p and os.path.exists(p):
os.remove(p)
except OSError:
pass
def _rename_for_new_id(written: list[str], new_id: str) -> list[str]:
"""After an id-collision retry, rename each written file to carry the new id
(filenames are `{old_id}`; swap the leading 8-char stem)."""
out = []
for p in written:
d, base = os.path.split(p)
# base looks like {id}{ext} | {id}_locked{ext} | {id}_consent{ext}
new_base = new_id + base[8:]
new_path = os.path.join(d, new_base)
try:
os.replace(p, new_path)
out.append(new_path)
except OSError:
out.append(p)
return out
def _retarget(filename: str | None, new_id: str) -> str | None:
return new_id + filename[8:] if filename else filename
# ── Inspect (no-write preview) ──────────────────────────────────────────────
@router.post("/personas/inspect")
async def inspect_persona(file: UploadFile = File(...)):
"""Read a bundle's manifest + consent summary WITHOUT writing any file or row."""
name = (file.filename or "").lower()
if not name.endswith(".ovsvoice") and not name.endswith(".omnivoice"):
raise HTTPException(status_code=400, detail="File must be a .ovsvoice or .omnivoice bundle")
content = await file.read()
try:
parsed = pb.parse_persona_bundle(content)
except pb.BundleError as e:
raise HTTPException(status_code=e.status, detail=e.detail)
persona = parsed.manifest.get("persona") or {}
consent_summary = None
if parsed.consent:
has_recording = "consent_audio" in parsed.members
consent_summary = {
"verified_claimed": bool(parsed.consent.get("verified_own_voice")),
"method": parsed.consent.get("method", ""),
"has_recording": has_recording,
# would_verify mirrors import's gate, minus the byte-floor check
# (inspect never extracts to measure size — advisory only).
"would_verify": has_recording and bool((parsed.consent.get("consent_text") or "").strip()),
}
return {
"format": "omnivoice-legacy" if parsed.is_legacy else pb.OVSVOICE_FORMAT,
"schema_version": parsed.manifest.get("schema_version", pb.OVSVOICE_SCHEMA_VERSION),
"name": persona.get("name") or "Imported Voice",
"kind": persona.get("kind") or "clone",
"language": persona.get("language", "Auto"),
"personality": persona.get("personality", ""),
"is_locked": bool(persona.get("is_locked")),
"license_spdx": parsed.license_spdx,
"tags": parsed.manifest.get("tags") or [],
"preview_only": parsed.preview_only,
"watermarked_preview": parsed.watermarked_preview,
"consent": consent_summary,
"schema_version_ahead": parsed.schema_version_ahead,
}
+18 -288
View File
@@ -1,5 +1,4 @@
import os
import re
import uuid
import time
import shutil
@@ -12,7 +11,6 @@ from core.db import db_conn
from core.config import VOICES_DIR, OUTPUTS_DIR
from core import event_bus
from core.personalities import get_personalities
from omnivoice.utils.voice_design import heal_design_instruct, sanitize_instruct
router = APIRouter()
@@ -36,110 +34,29 @@ def list_profiles():
rows = conn.execute("SELECT * FROM voice_profiles ORDER BY created_at DESC").fetchall()
return [dict(r) for r in rows]
_DESIGN_SEED = 42 # deterministic sample render, same as archetype previews
@router.post("/profiles")
async def create_profile(
name: str = Form(...),
ref_audio: Optional[UploadFile] = File(None),
ref_audio: UploadFile = File(...),
ref_text: str = Form(""),
instruct: str = Form(""),
language: str = Form("Auto"),
seed: Optional[int] = Form(None),
personality: str = Form(""),
kind: str = Form("clone"),
vd_states: Optional[str] = Form(None),
):
"""Create a voice profile (spec: docs/specs/voice-studio-unification.md §5).
kind='clone' requires `ref_audio` (the user's reference recording).
kind='design' requires `vd_states` (JSON of category picks); the server
renders a deterministic sample WAV (seed 42, same path as
archetype materialization) and stores it as the profile's
reference so the voice identity is stable across runs.
"""
if kind not in ("clone", "design"):
raise HTTPException(status_code=422, detail="kind must be 'clone' or 'design'")
if kind == "clone" and ref_audio is None:
raise HTTPException(status_code=422, detail="clone profiles require ref_audio")
if kind == "design":
if not (vd_states or "").strip():
raise HTTPException(status_code=422, detail="design profiles require vd_states")
import json as _json
try:
parsed = _json.loads(vd_states)
if not isinstance(parsed, dict):
raise ValueError("not an object")
except ValueError:
raise HTTPException(status_code=422, detail="vd_states must be a JSON object")
# An all-Auto design (every category left on "Auto") yields an empty
# instruct — that's still a valid, saveable voice: synthesis falls back
# to neutral instruct-only conditioning (see generation.py design path).
# Don't gate save on a non-empty instruct.
#
# Defence-in-depth against the "[object Object]" / freeform-prose poison
# (#550 #571 #594 #596): never persist an instruct the engine validator
# would reject. Sanitize the submitted instruct and, if it's unusable,
# rebuild the tags from vd_states — so the row is always generation-safe
# regardless of which frontend build saved it.
instruct = heal_design_instruct(instruct, parsed)
profile_id = str(uuid.uuid4())[:8]
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
audio_filename = f"{profile_id}{ext}"
audio_path = os.path.join(VOICES_DIR, audio_filename)
if kind == "clone":
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
audio_filename = f"{profile_id}{ext}"
audio_path = os.path.join(VOICES_DIR, audio_filename)
with open(audio_path, "wb") as f:
f.write(await ref_audio.read())
used_seed = seed
else:
# Saving a design profile is a pure persistence operation — it must not
# depend on a loaded TTS model (issue #476: on a fresh model-less Docker
# image the render forced a full model load + inference that 503'd, so
# the save failed). We try the deterministic identity sample opportunist-
# ically through the one shared TTS path (archetypes' renderer, never a
# second inference code path); if the engine isn't ready it's rendered
# lazily on first preview/use. The row carries vd_states + instruct, so
# the voice is fully usable without the sample (synthesis falls back to
# instruct-only conditioning — see generation.py's design path).
from pathlib import Path
from api.routers.archetypes import _render_archetype_wav
audio_filename = f"{profile_id}.wav"
audio_path = os.path.join(VOICES_DIR, audio_filename)
try:
await _render_archetype_wav(
{
"language": language,
"sample_script": ref_text, # optional custom sample line
"instruct": instruct,
},
Path(audio_path),
)
except Exception:
# Engine unavailable / OOM / inference failure — defer the sample.
# Store the row with no ref_audio_path; the identity sample is
# rendered on first preview or use. Never let this block the save.
import logging
logging.getLogger("omnivoice.profiles").info(
"Design profile %s saved with sample pending — "
"voice engine not ready; will render on first use", profile_id,
)
if os.path.exists(audio_path): # partial/blank render: don't keep it
with __import__("contextlib").suppress(OSError):
os.remove(audio_path)
audio_filename = None
used_seed = seed if seed is not None else _DESIGN_SEED
with open(audio_path, "wb") as f:
f.write(await ref_audio.read())
try:
with db_conn() as conn:
conn.execute(
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, "
"language, seed, personality, kind, vd_states, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
(profile_id, name, audio_filename, ref_text, instruct, language,
used_seed, personality, kind, vd_states, time.time())
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(profile_id, name, audio_filename, ref_text, instruct, language, seed, personality, time.time())
)
except Exception:
# Clean up orphaned audio file if DB insert fails
@@ -147,7 +64,7 @@ async def create_profile(
os.remove(audio_path)
raise
event_bus.emit("profiles", {"action": "created", "id": profile_id})
return {"id": profile_id, "name": name, "kind": kind}
return {"id": profile_id, "name": name}
@router.get("/profiles/{profile_id}")
def get_profile(profile_id: str):
@@ -175,10 +92,6 @@ def update_profile(profile_id: str, patch: ProfileUpdate):
continue
if col == "name" and not val.strip():
raise HTTPException(status_code=400, detail="A voice profile needs a name.")
if col == "instruct":
# Never let an edit persist a validator-rejecting instruct (prose /
# "[object Object]"); keep only whitelist tags (#550 #571 #594 #596).
val = sanitize_instruct(val)
fields.append(f"{col} = ?")
params.append(val.strip() if col in ("name", "language") else val)
if not fields:
@@ -250,96 +163,20 @@ def get_profile_usage(profile_id: str):
}
# profile_id is a request path param and the audio filename derives from it, so
# constrain it to the generated-id charset (no separators / `..` possible) before
# any path use, and read only a *direct child* of VOICES_DIR — os.path.basename()
# strips any directory component (a path-injection / CWE-22 barrier).
_PROFILE_ID_RE = re.compile(r"[A-Za-z0-9_-]{1,64}")
@router.get("/profiles/{profile_id}/audio")
async def get_profile_audio(profile_id: str):
if not _PROFILE_ID_RE.fullmatch(profile_id or ""):
return Response("Profile not found", status_code=404)
def get_profile_audio(profile_id: str):
with db_conn() as conn:
row = conn.execute(
"SELECT ref_audio_path, locked_audio_path, kind, instruct, language, ref_text "
"FROM voice_profiles WHERE id=?",
(profile_id,),
).fetchone()
row = conn.execute("SELECT ref_audio_path, locked_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
if not row:
return Response("Profile not found", status_code=404)
audio_file = row["locked_audio_path"] or row["ref_audio_path"]
if not audio_file:
# A design profile saved before the engine was ready (issue #476) has no
# identity sample yet. Render it lazily now — the deterministic seed-42
# sample is reproducible, so a deferred render matches a save-time one.
rendered = await _materialize_design_sample(profile_id, row)
if rendered is None:
return Response("No audio available", status_code=404)
audio_file = rendered
# CWE-22: resolve the DB-stored filename strictly inside VOICES_DIR via the
# shared guard — _voices_path() applies the os.path.basename() barrier plus
# symlink-resolved containment (same path the consent endpoint trusts).
audio_path = _voices_path(str(audio_file))
if audio_path is None or not os.path.exists(audio_path):
return Response("No audio available", status_code=404)
audio_path = os.path.join(VOICES_DIR, audio_file)
if not os.path.exists(audio_path):
return Response("Audio file missing", status_code=404)
return FileResponse(audio_path, media_type="audio/wav")
async def _materialize_design_sample(profile_id: str, row) -> Optional[str]:
"""Render a design profile's pending identity sample on first request.
Returns the stored filename on success, or None if this isn't a renderable
design row. Raises HTTPException(503) with a precise "model not ready"
message if the engine is genuinely unavailable saving never depends on
this, but a user who explicitly asks for the sample gets a clear signal.
"""
try:
kind = row["kind"]
except (KeyError, IndexError):
kind = "clone"
if kind != "design":
return None
from pathlib import Path
from api.routers.archetypes import _render_archetype_wav
audio_filename = f"{profile_id}.wav"
# CWE-22: resolve under VOICES_DIR via the shared basename + containment
# guard before rendering (rejects any escape).
audio_path = _voices_path(audio_filename)
if audio_path is None:
raise HTTPException(status_code=400, detail="invalid profile identifier")
try:
await _render_archetype_wav(
{
"language": row["language"] or "Auto",
"sample_script": row["ref_text"] or "",
"instruct": row["instruct"] or "",
},
Path(audio_path),
)
except Exception as e:
with __import__("contextlib").suppress(OSError):
if os.path.exists(audio_path):
os.remove(audio_path)
raise HTTPException(
status_code=503,
detail=(
"The voice engine isn't ready yet, so this designed voice's "
"preview sample can't be rendered. Finish setup / download a "
f"model, then try again. ({e})"
),
)
with db_conn() as conn:
conn.execute(
"UPDATE voice_profiles SET ref_audio_path=? WHERE id=?",
(audio_filename, profile_id),
)
return audio_filename
@router.post("/profiles/{profile_id}/lock")
async def lock_profile(
profile_id: str,
@@ -397,122 +234,15 @@ async def unlock_profile(profile_id: str):
event_bus.emit("profiles", {"action": "unlocked", "id": profile_id})
return {"unlocked": True, "profile_id": profile_id}
# ── Consent lock (parity program Wave 0.2) ─────────────────────────────────
#
# A profile becomes "verified own voice" when its owner records themselves
# reading a consent statement. The recording is provenance, not a voiceprint
# check — agentic features and gallery sharing gate on the flag; plain local
# synthesis never does. Spec: docs/competitive-analysis.md Action 22.
_MIN_CONSENT_AUDIO_BYTES = 1000 # same floor as the frontend recorder
# Upload filename extension whitelist — anything else falls back to .wav so a
# crafted filename can never influence the on-disk path (py/path-injection).
_CONSENT_EXT_RE = re.compile(r"^\.[A-Za-z0-9]{1,8}$")
def _voices_path(filename: str) -> Optional[str]:
"""Resolve a DB-stored audio filename strictly inside VOICES_DIR.
Rejects anything that isn't a bare filename or that escapes the voices
directory after symlink resolution. Returns None instead of raising so
cleanup paths can simply skip bad values.
"""
if not filename or os.path.basename(filename) != filename:
return None
root = os.path.realpath(VOICES_DIR)
path = os.path.realpath(os.path.join(root, filename))
if not path.startswith(root + os.sep):
return None
return path
@router.post("/profiles/{profile_id}/consent")
async def record_consent(
profile_id: str,
consent_audio: UploadFile = File(...),
consent_text: str = Form(...),
):
if not consent_text.strip():
raise HTTPException(status_code=422, detail="consent_text must not be empty")
data = await consent_audio.read()
if len(data) < _MIN_CONSENT_AUDIO_BYTES:
raise HTTPException(status_code=422, detail="consent recording is too short")
with db_conn() as conn:
row = conn.execute(
"SELECT id, consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)
).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Profile not found")
ext = os.path.splitext(consent_audio.filename or "")[1]
if not _CONSENT_EXT_RE.match(ext):
ext = ".wav"
audio_filename = f"{profile_id}_consent{ext}"
audio_path = _voices_path(audio_filename)
if audio_path is None: # profile_id is server-generated; this is belt+braces
raise HTTPException(status_code=400, detail="Invalid profile id")
with open(audio_path, "wb") as f:
f.write(data)
# A re-record may change the extension; drop the superseded file.
old = row["consent_audio_path"]
if old and old != audio_filename:
old_path = _voices_path(old)
if old_path and os.path.exists(old_path):
os.remove(old_path)
recorded_at = time.time()
try:
with db_conn() as conn:
conn.execute(
"UPDATE voice_profiles SET verified_own_voice=1, consent_text=?, "
"consent_audio_path=?, consent_recorded_at=? WHERE id=?",
(consent_text.strip(), audio_filename, recorded_at, profile_id),
)
except Exception:
if os.path.exists(audio_path):
os.remove(audio_path)
raise
event_bus.emit("profiles", {"action": "consent_recorded", "id": profile_id})
return {
"id": profile_id,
"verified_own_voice": True,
"consent_recorded_at": recorded_at,
}
@router.delete("/profiles/{profile_id}/consent")
def revoke_consent(profile_id: str):
with db_conn() as conn:
row = conn.execute(
"SELECT consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)
).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Profile not found")
conn.execute(
"UPDATE voice_profiles SET verified_own_voice=0, consent_text='', "
"consent_audio_path='', consent_recorded_at=NULL WHERE id=?",
(profile_id,),
)
if row["consent_audio_path"]:
path = _voices_path(row["consent_audio_path"])
if path and os.path.exists(path):
os.remove(path)
event_bus.emit("profiles", {"action": "consent_revoked", "id": profile_id})
return {"id": profile_id, "verified_own_voice": False}
@router.delete("/profiles/{profile_id}")
def delete_profile(profile_id: str):
with db_conn() as conn:
row = conn.execute("SELECT ref_audio_path, locked_audio_path, consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
row = conn.execute("SELECT ref_audio_path, locked_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
if row:
for col in ["ref_audio_path", "locked_audio_path", "consent_audio_path"]:
for col in ["ref_audio_path", "locked_audio_path"]:
if row[col]:
path = _voices_path(row[col])
if path and os.path.exists(path):
path = os.path.join(VOICES_DIR, row[col])
if os.path.exists(path):
os.remove(path)
# Prevent FOREIGN KEY constraint failure
conn.execute("UPDATE generation_history SET profile_id = NULL WHERE profile_id=?", (profile_id,))
-25
View File
@@ -2,7 +2,6 @@ import uuid
import time
import json
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
from core.db import db_conn
from core import event_bus
@@ -10,10 +9,6 @@ from schemas.requests import ProjectSaveRequest
router = APIRouter()
class ProjectRenameRequest(BaseModel):
name: str
@router.get("/projects")
async def list_projects():
with db_conn() as conn:
@@ -64,26 +59,6 @@ async def update_project(project_id: str, req: ProjectSaveRequest):
event_bus.emit("projects", {"action": "updated", "id": project_id})
return {"id": project_id, "name": req.name, "updated_at": now}
@router.patch("/projects/{project_id}")
async def rename_project(project_id: str, req: ProjectRenameRequest):
"""Lightweight rename — updates only the project name (and updated_at),
without re-serialising the whole state blob like PUT does."""
name = req.name.strip()
if not name:
raise HTTPException(status_code=400, detail="Project name cannot be empty")
now = time.time()
with db_conn() as conn:
row = conn.execute("SELECT id FROM studio_projects WHERE id=?", (project_id,)).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Project not found")
conn.execute(
"UPDATE studio_projects SET name=?, updated_at=? WHERE id=?",
(name, now, project_id),
)
event_bus.emit("projects", {"action": "renamed", "id": project_id})
return {"id": project_id, "name": name, "updated_at": now}
@router.delete("/projects/{project_id}")
async def delete_project(project_id: str):
with db_conn() as conn:
-306
View File
@@ -1,306 +0,0 @@
"""
Pronunciation dictionary router Expressive-TTS Spec 01 Phase 1.
CRUD for the DB-backed, per-language pronunciation dictionary the
``PronunciationPanel`` (Settings Pronunciation) edits, plus a model-free
``/pronunciation/test`` dry-run. Entries are applied as pure text substitution
before synthesis (see ``services/pronunciation.apply_pronunciation`` and the
generate path), so a saved entry actually changes the audio on every engine.
Endpoints (loopback-only, like the dictation router):
GET /pronunciation list every entry
POST /pronunciation create one entry
PUT /pronunciation/{entry_id} update an entry (partial)
DELETE /pronunciation/{entry_id} remove an entry
POST /pronunciation/test dry-run substitution (no model)
GET /pronunciation/export all entries as JSON (round-trips import)
POST /pronunciation/import bulk add entries from JSON
Scope: ``language='*'`` is global (applies to every request); a 2-letter code
(``'en'``, ``'de'``) applies only when the request language matches.
"""
from __future__ import annotations
import logging
import re
import time
import uuid
from typing import List, Optional
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from api.dependencies import require_loopback
from core.db import db_conn
from services.pronunciation import apply_pronunciation, entries_for_language
logger = logging.getLogger("omnivoice.pronunciation")
router = APIRouter()
_VALID_TYPES = ("respelling", "ipa", "cmu")
_ALL_LANG = "*"
# IPA: the input is validated as a non-empty string of Unicode letters / IPA
# extension codepoints + the usual suprasegmental marks; we reject ASCII control
# and the bracket/pipe chars that would collide with the inline grammar. This is
# a charset gate (catches obvious garbage early), not a full IPA grammar.
_IPA_BAD = re.compile(r"[\[\]\|\x00-\x1f]")
# CMU / ARPABET: space-separated phoneme tokens (letters + an optional 0-2 stress
# digit), e.g. "N AH0 V AE1 D AH0". Reject anything else.
_CMU_TOKEN = re.compile(r"^[A-Za-z]{1,3}[0-2]?$")
def _validate_type_replacement(etype: str, replacement: str) -> None:
"""Raise 400 on a phoneme replacement that's obviously malformed.
Respelling rows accept any text. IPA rows must be a non-empty string free of
bracket/pipe/control chars. CMU rows must be space-separated ARPABET tokens.
Validating on save (not at synth) means a model never sees garbage phonemes
(Spec 01 §R3 never pass unvalidated phoneme strings to a model).
"""
if etype == "respelling":
return
rep = (replacement or "").strip()
if not rep:
raise HTTPException(
status_code=400,
detail=f"A {etype.upper()} entry needs a phoneme string in 'replacement'.",
)
if etype == "ipa":
if _IPA_BAD.search(rep):
raise HTTPException(
status_code=400,
detail="That IPA string contains brackets, a pipe, or control characters. "
"Use plain IPA symbols, e.g. ˈnɛvʌdə.",
)
elif etype == "cmu":
tokens = rep.split()
if not tokens or any(not _CMU_TOKEN.match(tok) for tok in tokens):
raise HTTPException(
status_code=400,
detail="That doesn't look like CMU/ARPABET. Use space-separated tokens with "
"optional stress digits, e.g. N AH0 V AE1 D AH0.",
)
def _norm_language(language: Optional[str]) -> str:
"""Normalize a scope to '*' (global) or a lowercase 2-letter code."""
if not language:
return _ALL_LANG
s = str(language).strip()
if not s or s == _ALL_LANG or s.lower() == "auto":
return _ALL_LANG
return s.lower()[:2]
def _row_to_dict(r) -> dict:
d = dict(r)
d["enabled"] = bool(d.get("enabled"))
# ``scope`` is the UI-facing alias for ``language`` ('*' shows as Global).
d["scope"] = d.get("language") or _ALL_LANG
return d
# ── Schemas ──────────────────────────────────────────────────────────────────
class PronEntry(BaseModel):
term: str
replacement: str = ""
type: str = "respelling"
language: str = _ALL_LANG
enabled: bool = True
class PronEntryUpdate(BaseModel):
term: Optional[str] = None
replacement: Optional[str] = None
type: Optional[str] = None
language: Optional[str] = None
enabled: Optional[bool] = None
class PronTestRequest(BaseModel):
text: str
language: Optional[str] = None
class PronImportRequest(BaseModel):
entries: List[PronEntry]
replace: bool = False # True → clear existing rows first
# ── CRUD ─────────────────────────────────────────────────────────────────────
@router.get("/pronunciation", dependencies=[Depends(require_loopback)])
def list_entries():
with db_conn() as conn:
rows = conn.execute(
"SELECT id, term, replacement, type, language, enabled, created_at "
"FROM pronunciation_entries ORDER BY created_at ASC, id ASC"
).fetchall()
return [_row_to_dict(r) for r in rows]
@router.post("/pronunciation", dependencies=[Depends(require_loopback)])
def create_entry(entry: PronEntry):
term = entry.term.strip()
if not term:
raise HTTPException(status_code=400, detail="A pronunciation entry needs a term.")
etype = (entry.type or "respelling").strip().lower()
if etype not in _VALID_TYPES:
raise HTTPException(
status_code=400,
detail=f"Unknown entry type {entry.type!r}. Use one of: {', '.join(_VALID_TYPES)}.",
)
_validate_type_replacement(etype, entry.replacement)
eid = str(uuid.uuid4())[:12]
now = time.time()
lang = _norm_language(entry.language)
with db_conn() as conn:
conn.execute(
"INSERT INTO pronunciation_entries (id, term, replacement, type, language, enabled, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?)",
(eid, term, entry.replacement, etype, lang, 1 if entry.enabled else 0, now),
)
row = conn.execute(
"SELECT id, term, replacement, type, language, enabled, created_at "
"FROM pronunciation_entries WHERE id = ?", (eid,)
).fetchone()
return _row_to_dict(row)
@router.put("/pronunciation/{entry_id}", dependencies=[Depends(require_loopback)])
def update_entry(entry_id: str, patch: PronEntryUpdate):
with db_conn() as conn:
existing = conn.execute(
"SELECT id, term, replacement, type, language, enabled, created_at "
"FROM pronunciation_entries WHERE id = ?", (entry_id,)
).fetchone()
if existing is None:
raise HTTPException(status_code=404, detail="No such pronunciation entry.")
# Resolve the post-update type + replacement so phoneme validation runs
# against the final state (e.g. switching type without changing text).
new_type = (patch.type.strip().lower() if patch.type is not None else existing["type"]) or "respelling"
if new_type not in _VALID_TYPES:
raise HTTPException(
status_code=400,
detail=f"Unknown entry type {patch.type!r}. Use one of: {', '.join(_VALID_TYPES)}.",
)
new_replacement = patch.replacement if patch.replacement is not None else existing["replacement"]
_validate_type_replacement(new_type, new_replacement)
fields, params = [], []
if patch.term is not None:
term = patch.term.strip()
if not term:
raise HTTPException(status_code=400, detail="A pronunciation entry needs a term.")
fields.append("term = ?"); params.append(term)
if patch.replacement is not None:
fields.append("replacement = ?"); params.append(patch.replacement)
if patch.type is not None:
fields.append("type = ?"); params.append(new_type)
if patch.language is not None:
fields.append("language = ?"); params.append(_norm_language(patch.language))
if patch.enabled is not None:
fields.append("enabled = ?"); params.append(1 if patch.enabled else 0)
if not fields:
raise HTTPException(
status_code=400,
detail="PUT body was empty. Include at least one field to change, or DELETE the entry.",
)
params.append(entry_id)
# nosec B608 - `fields` are fixed literal assignments ("term = ?", …) from
# the allowlist above; every user value is a bound `?` parameter, never
# interpolated. The f-string only joins constant column fragments.
conn.execute(
f"UPDATE pronunciation_entries SET {', '.join(fields)} WHERE id = ?", # nosec B608
params,
)
row = conn.execute(
"SELECT id, term, replacement, type, language, enabled, created_at "
"FROM pronunciation_entries WHERE id = ?", (entry_id,)
).fetchone()
return _row_to_dict(row)
@router.delete("/pronunciation/{entry_id}", dependencies=[Depends(require_loopback)])
def delete_entry(entry_id: str):
with db_conn() as conn:
cur = conn.execute("DELETE FROM pronunciation_entries WHERE id = ?", (entry_id,))
return {"deleted": cur.rowcount > 0}
# ── Dry-run + import/export ───────────────────────────────────────────────────
@router.post("/pronunciation/test", dependencies=[Depends(require_loopback)])
def test_substitution(req: PronTestRequest):
"""Show the post-substitution text for ``req.text`` — no model call.
Applies the same dictionary + inline ``[[]]`` resolution the synth path
runs, so the user sees exactly what the engine will be handed.
"""
with db_conn() as conn:
rows = conn.execute(
"SELECT id, term, replacement, type, language, enabled, created_at "
"FROM pronunciation_entries"
).fetchall()
substituted = apply_pronunciation(req.text, rows, req.language)
applied = entries_for_language(rows, req.language)
return {
"input": req.text,
"substituted": substituted,
"changed": substituted != req.text,
"applied_terms": sorted(applied.keys(), key=len, reverse=True),
}
@router.get("/pronunciation/export", dependencies=[Depends(require_loopback)])
def export_entries():
"""Every entry as a JSON-serializable list (round-trips ``/import``)."""
with db_conn() as conn:
rows = conn.execute(
"SELECT term, replacement, type, language, enabled "
"FROM pronunciation_entries ORDER BY created_at ASC, id ASC"
).fetchall()
return {"entries": [
{"term": r["term"], "replacement": r["replacement"], "type": r["type"],
"language": r["language"], "enabled": bool(r["enabled"])}
for r in rows
]}
@router.post("/pronunciation/import", dependencies=[Depends(require_loopback)])
def import_entries(req: PronImportRequest):
"""Bulk-add entries. ``replace=true`` clears the table first.
Each entry is validated like ``POST /pronunciation``; one bad row fails the
whole import (400) so the table is never left half-applied.
"""
now = time.time()
cleaned = []
for e in req.entries:
term = e.term.strip()
if not term:
continue # silently skip blank terms — they're a no-op anyway
etype = (e.type or "respelling").strip().lower()
if etype not in _VALID_TYPES:
raise HTTPException(
status_code=400,
detail=f"Entry {term!r}: unknown type {e.type!r}.",
)
_validate_type_replacement(etype, e.replacement)
cleaned.append((str(uuid.uuid4())[:12], term, e.replacement, etype,
_norm_language(e.language), 1 if e.enabled else 0, now))
with db_conn() as conn:
if req.replace:
conn.execute("DELETE FROM pronunciation_entries")
conn.executemany(
"INSERT INTO pronunciation_entries (id, term, replacement, type, language, enabled, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?)",
cleaned,
)
return {"imported": len(cleaned), "replaced": req.replace}
-497
View File
@@ -12,7 +12,6 @@ The state endpoint duplicates `/system/hf-token/state` (which lives on
"""
from __future__ import annotations
import asyncio
import logging
import os
from dataclasses import asdict
@@ -124,358 +123,6 @@ def set_torch_compile_disabled(body: _TorchCompileBody):
return _torch_compile_state()
# ── Dictation refinement (parity program Wave 2.1 / Spec 3 phase 2) ───────
class _RefinementBody(BaseModel):
auto: bool | None = None
smart_cleanup: bool | None = None
self_correction: bool | None = None
preserve_technical: bool | None = None
def _refinement_state():
from services.refinement import (
_skill_llm,
get_last_refine_status,
get_refinement_config,
)
cfg = get_refinement_config()
# `llm_ready` only means "an endpoint is CONFIGURED" — a placeholder/dead
# endpoint still reads ready. It's resolved through the LLM Skills registry
# so a disabled dictation_refinement skill / per-skill provider override
# reads the same here as on the actual refine path. The honesty layer is
# `last_refine_status`: {ok, reason, at} from the most recent final, so the
# panel can flag a configured-but-failing LLM (the real safety is the hard
# refine timeout, which keeps a dead endpoint from ever stalling the final).
cfg["llm_ready"] = _skill_llm().id != "off"
cfg["last_refine_status"] = get_last_refine_status()
return cfg
@router.get("/dictation-refinement")
def get_dictation_refinement():
"""Current refinement config + whether an LLM backend is configured."""
return _refinement_state()
@router.put("/dictation-refinement")
def set_dictation_refinement(body: _RefinementBody):
from services.refinement import set_refinement_config
try:
set_refinement_config({k: v for k, v in body.model_dump().items() if v is not None})
except Exception:
logger.exception("set_dictation_refinement failed")
raise HTTPException(status_code=500, detail="Failed to persist setting")
return _refinement_state()
# ── LLM endpoint (parity program Wave 2.4 / §R2 rung 4) ───────────────────
# Focused configuration for the OpenAI-compatible LLM endpoint that powers
# cinematic translate, glossary auto-extract, and dictation refinement.
# Persistence rides the existing TRANSLATE_BASE_URL / TRANSLATE_API_KEY /
# TRANSLATE_MODEL env vars (already in system.py PERSISTENT_KEYS, restored
# at startup) so the resolution path in llm_backend/translator is unchanged.
class _LLMEndpointBody(BaseModel):
base_url: str | None = None
model: str | None = None
api_key: str | None = None # None = leave unchanged; "" = clear
def _mask(secret: str | None) -> str | None:
if not secret:
return None
return f"{secret[-4:]}" if len(secret) > 4 else "set"
def _llm_endpoint_state():
from services.llm_backend import OpenAICompatBackend
ok, reason = OpenAICompatBackend.is_available()
return {
"base_url": os.environ.get("TRANSLATE_BASE_URL", ""),
"model": os.environ.get("TRANSLATE_MODEL", ""),
"api_key_masked": _mask(
os.environ.get("TRANSLATE_API_KEY") or os.environ.get("OPENAI_API_KEY")
),
"available": ok,
"reason": None if ok else reason,
}
@router.get("/llm-endpoint")
def get_llm_endpoint():
"""Current OpenAI-compatible LLM endpoint config + live availability."""
return _llm_endpoint_state()
@router.put("/llm-endpoint")
def set_llm_endpoint(body: _LLMEndpointBody):
"""Persist base URL / model / API key for the OpenAI-compatible endpoint.
Reuses the env-var persistence path (prefs.json, restored at startup):
base_url -> TRANSLATE_BASE_URL, model -> TRANSLATE_MODEL,
api_key -> TRANSLATE_API_KEY. A None field is left unchanged; an empty
string clears it. Ollama ignores the key; vLLM / LM Studio require it.
"""
from core.prefs import set_ as prefs_set, delete as prefs_delete
mapping = {
"TRANSLATE_BASE_URL": body.base_url,
"TRANSLATE_MODEL": body.model,
"TRANSLATE_API_KEY": body.api_key,
}
for env_key, val in mapping.items():
if val is None:
continue # untouched
val = val.strip()
if val:
os.environ[env_key] = val
prefs_set(f"env.{env_key}", val)
else:
os.environ.pop(env_key, None)
prefs_delete(f"env.{env_key}")
# get_active_llm_backend() builds a fresh backend (and its OpenAI client
# reads env at construction) on every call, so there's no singleton to
# invalidate — the next translate/refine picks up the new values.
return _llm_endpoint_state()
# ── Multi-provider LLM registry (Settings → LLM Providers) ────────────────
# Keys persist ENCRYPTED via settings_store.set_secret (never .env, never
# returned). base_url/model/account overrides are non-secret. Loopback-gated
# by the router dep, so LAN peers can't read masks or write keys.
class _LLMProviderBody(BaseModel):
api_key: str | None = Field(None, description="API key; '' clears it, None leaves unchanged")
base_url: str | None = None
model: str | None = None
account_id: str | None = Field(None, description="Cloudflare account id")
make_active: bool = False
class _LLMActiveBody(BaseModel):
provider: str = Field(..., description="provider id to activate")
@router.get("/llm-providers")
def list_llm_providers():
"""All providers with resolved base_url/model + whether a key is configured.
Never returns key material only `has_key`/`key_from_env` booleans.
"""
from services import llm_providers
return {
"active": llm_providers.active_provider_id(),
"providers": [llm_providers.describe(p) for p in llm_providers.all_providers()],
}
@router.put("/llm-providers/{provider_id}")
def save_llm_provider(provider_id: str, body: _LLMProviderBody):
"""Save a provider's key (encrypted) + optional base_url/model/account.
A None field is left unchanged; an empty api_key clears the stored key.
"""
from services import llm_providers
if llm_providers.get_provider(provider_id) is None:
raise HTTPException(status_code=404, detail=f"unknown provider {provider_id!r}")
if body.api_key is not None:
llm_providers.save_key(provider_id, body.api_key.strip())
llm_providers.save_overrides(
provider_id, base_url=body.base_url, model=body.model,
account_id=body.account_id,
)
if body.make_active:
llm_providers.set_active_provider(provider_id)
return list_llm_providers()
@router.post("/llm-providers/active")
def set_active_llm_provider(body: _LLMActiveBody):
from services import llm_providers
if llm_providers.get_provider(body.provider) is None:
raise HTTPException(status_code=404, detail=f"unknown provider {body.provider!r}")
llm_providers.set_active_provider(body.provider)
return list_llm_providers()
def _scrub_llm_detail(e: Exception, api_key: str | None) -> str:
"""Scrubbed, UI-safe failure text. scrub_text() covers env secrets and
home paths but a STORE-persisted key isn't in the env, and some
providers echo the key in error bodies, so redact the exact resolved key
explicitly before the generic pass."""
from core.scrub import scrub_text
detail = f"{type(e).__name__}: {e}"
if api_key and api_key != "local" and len(api_key) >= 8:
detail = detail.replace(api_key, "•••")
return scrub_text(detail)
def _classify_llm_error(e: Exception) -> str:
"""Map a provider-call failure to an actionable kind the UI can localize.
Kinds: auth (bad/missing key), not_found (model or endpoint path),
rate_limit, network (DNS/conn/timeout), error (everything else).
Status codes win when the OpenAI SDK provides one; exception-family
names catch the non-HTTP failures (DNS, refused, TLS, timeout).
"""
status = getattr(e, "status_code", None)
if status in (401, 403):
return "auth"
if status == 404:
return "not_found"
if status == 429:
return "rate_limit"
name = type(e).__name__
if name in ("APIConnectionError", "APITimeoutError", "ConnectError",
"ConnectTimeout", "TimeoutError"):
return "network"
if name == "AuthenticationError":
return "auth"
if name == "NotFoundError":
return "not_found"
if name == "RateLimitError":
return "rate_limit"
return "error"
@router.post("/llm-providers/{provider_id}/test")
def test_llm_provider(provider_id: str):
"""One cheap round-trip against a provider to prove the key/URL work.
Temporarily activates the provider for the probe by resolving its config
directly (does not change the persisted active selection). Returns
latency_ms plus, on failure, a classified ``kind`` (config / auth /
not_found / rate_limit / network / error) so the UI shows an actionable,
localizable message instead of a raw exception string.
"""
import time as _time
from services import llm_providers
p = llm_providers.get_provider(provider_id)
if p is None:
raise HTTPException(status_code=404, detail=f"unknown provider {provider_id!r}")
base_url = llm_providers.resolve_base_url(p)
api_key = llm_providers.resolve_api_key(p)
if not base_url:
return {"ok": False, "kind": "config", "detail": "No Base URL set for this provider."}
if not api_key:
return {"ok": False, "kind": "config", "detail": "No API key configured for this provider."}
t0 = _time.monotonic()
try:
from openai import OpenAI
# max_retries=0: this is an interactive probe with a live spinner — the
# SDK's default 2 automatic retries turn a 429/timeout into a ~34s hang.
# Surface the first failure immediately instead.
client = OpenAI(api_key=api_key, base_url=base_url, max_retries=0)
res = client.chat.completions.create(
model=llm_providers.resolve_model(p),
messages=[{"role": "user", "content": "Reply with the single word: ok"}],
timeout=20,
)
reply = (res.choices[0].message.content or "").strip()
return {
"ok": True,
"model": llm_providers.resolve_model(p),
"reply": reply[:80],
"latency_ms": int((_time.monotonic() - t0) * 1000),
}
except Exception as e: # noqa: BLE001 — surface a clean, scrubbed error to the UI
return {
"ok": False,
"kind": _classify_llm_error(e),
"detail": _scrub_llm_detail(e, api_key),
"latency_ms": int((_time.monotonic() - t0) * 1000),
}
@router.get("/llm-providers/{provider_id}/models")
def list_llm_provider_models(provider_id: str):
"""List model ids the provider's key can access (OpenAI-compat /models).
Powers the model-picker datalist in Settings LLM Providers so users
don't have to guess model names. Read-only; failures return the same
classified shape as /test; capped so a huge catalog can't bloat the UI.
"""
from services import llm_providers
p = llm_providers.get_provider(provider_id)
if p is None:
raise HTTPException(status_code=404, detail=f"unknown provider {provider_id!r}")
base_url = llm_providers.resolve_base_url(p)
api_key = llm_providers.resolve_api_key(p)
if not base_url or not api_key:
return {"ok": False, "kind": "config", "models": []}
try:
from openai import OpenAI
# max_retries=0: interactive probe — fail fast, don't burn ~34s on the
# SDK's default retry ladder when the key/URL is wrong (matches /test).
client = OpenAI(api_key=api_key, base_url=base_url, max_retries=0)
ids = sorted(m.id for m in client.models.list(timeout=10))
# Cap so a huge catalog can't bloat the datalist; flag the cap so the UI
# can say "first 200 shown" rather than implying it's the full list.
return {"ok": True, "models": ids[:200], "truncated": len(ids) > 200}
except Exception as e: # noqa: BLE001
return {
"ok": False,
"kind": _classify_llm_error(e),
"detail": _scrub_llm_detail(e, api_key),
"models": [],
}
# ── LLM Skills (Settings → LLM Skills) ─────────────────────────────────────
# Per-feature enable/route control for every LLM consumption point. Each
# skill can be toggled off (degrades exactly like "no LLM configured") or
# routed to a specific provider (local Ollama/LM Studio vs a remote key)
# instead of the one global active provider. Loopback-gated (router dep).
class _LLMSkillBody(BaseModel):
enabled: bool | None = Field(None, description="None leaves the toggle unchanged")
provider_override: str | None = Field(
None,
description="provider id to route this skill to; '' or null clears "
"it (skill follows the active provider). Omit to leave "
"unchanged.",
)
@router.get("/llm-skills")
def list_llm_skills():
"""Every LLM skill with its toggle, routing, and resolved ready status."""
from services import llm_skills
return {"skills": [llm_skills.describe(s.id) for s in llm_skills.all_skills()]}
@router.put("/llm-skills/{skill_id}")
def set_llm_skill(skill_id: str, body: _LLMSkillBody):
"""Toggle a skill and/or set its provider routing.
Field semantics match the providers PUT: an omitted field is left
unchanged; ``provider_override: ""``/``null`` clears the override.
404 for an unknown skill or an unknown provider id.
"""
from services import llm_skills
if llm_skills.get_skill(skill_id) is None:
raise HTTPException(status_code=404, detail=f"unknown LLM skill {skill_id!r}")
kwargs = {}
if body.enabled is not None:
kwargs["enabled"] = body.enabled
if "provider_override" in body.model_fields_set:
kwargs["provider_override"] = body.provider_override
try:
if kwargs:
llm_skills.configure_skill(skill_id, **kwargs)
except ValueError as e:
raise HTTPException(status_code=404, detail=str(e))
return list_llm_skills()
# ── License acceptance (Phase 3 Plan 03-01 / TTS-05) ──────────────────────
# Frontend ``SupertonicLicenseDialog`` flips the engine-license bit via this
# endpoint. The handler is loopback-gated (router-level dep) and the
@@ -637,147 +284,3 @@ def set_models_dir(body: _ModelsDirBody):
user_env.set_user_env(_MODELS_DIR_ENV, path)
return {"configured": path, "effective": _effective_models_dir(), "restart_required": True}
# ── Storage report (Settings → Storage) ────────────────────────────────────
# Per-volume disk totals + du-style sizes for everything the app owns (HF
# model cache, app data subtotals, engine venvs, temp files) with server-side
# warnings. Heavy directory walks run in a worker thread with per-category
# deadlines and a 5-minute in-process cache (services.storage_report), so the
# endpoint stays cheap on repeat Settings visits. Loopback-gated via the
# router-level dep like every sibling.
@router.get("/storage")
async def get_storage_report(refresh: bool = Query(False)):
"""Disk + per-category storage usage for the Settings → Storage panel.
`refresh=1` bypasses the 5-minute cache and rescans. `min_free_gb`
reuses the setup wizard's constant so both surfaces warn at the same
threshold.
"""
from api.routers.setup.wizard import MIN_FREE_GB
from core.config import DATA_DIR
from services import storage_report
try:
return await asyncio.to_thread(
storage_report.get_report,
data_dir=DATA_DIR,
hf_cache_dir=_effective_models_dir(),
app_venv=storage_report.default_app_venv(),
min_free_gb=MIN_FREE_GB,
refresh=refresh,
)
except Exception:
logger.exception("storage report failed")
raise HTTPException(status_code=500, detail="Failed to compute storage report")
# ── HF mirror endpoint (parity program Wave 4.3 / §R4 c) ──────────────────
# Restricted-network users (e.g. behind the Great Firewall) need to point
# huggingface_hub at a mirror. HF reads HF_ENDPOINT at import time, so a
# change takes effect on the next backend start — persisted to the durable
# per-user env so it survives Tauri/Finder launches that don't inherit a
# shell. Loopback-gated via the router dep.
_HF_ENDPOINT_ENV = "HF_ENDPOINT"
# A few well-known mirrors, surfaced as quick-picks in the UI. hf-mirror.com
# is the community mirror most-used in China; the official endpoint clears it.
_HF_MIRROR_PRESETS = [
{"label": "Hugging Face (official)", "url": ""},
{"label": "hf-mirror.com (community, China)", "url": "https://hf-mirror.com"},
]
class _HFMirrorBody(BaseModel):
url: str = Field("", description="HF_ENDPOINT URL; empty string clears it (official endpoint)")
@router.get("/hf-mirror")
def get_hf_mirror():
from core import user_env
configured = user_env.get_user_env(_HF_ENDPOINT_ENV) or ""
return {
# The value that will apply after restart (persisted), and what's
# live in this process (env may differ until then).
"configured": configured,
"effective": os.environ.get(_HF_ENDPOINT_ENV, ""),
"presets": _HF_MIRROR_PRESETS,
}
@router.put("/hf-mirror")
def set_hf_mirror(body: _HFMirrorBody):
from core import user_env
url = (body.url or "").strip().rstrip("/")
if url and not url.startswith(("http://", "https://")):
raise HTTPException(status_code=400, detail="Mirror URL must start with http(s)://")
# Compare against the currently-persisted value (normalised the same way) so
# a no-op save doesn't nag the user to restart. Only a real change to the
# persisted endpoint can require a restart.
previous = (user_env.get_user_env(_HF_ENDPOINT_ENV) or "").strip().rstrip("/")
changed = url != previous
try:
if url:
user_env.set_user_env(_HF_ENDPOINT_ENV, url)
os.environ[_HF_ENDPOINT_ENV] = url # best-effort for new downloads this session
else:
user_env.unset_user_env(_HF_ENDPOINT_ENV)
os.environ.pop(_HF_ENDPOINT_ENV, None)
except Exception:
logger.exception("set_hf_mirror failed")
raise HTTPException(status_code=500, detail="Failed to persist mirror setting")
# Model Store downloads pick up the new mirror immediately — the download
# path resolves the endpoint per-call and we updated os.environ above. Only
# transformers-side model *loads* (which read HF_ENDPOINT at import time)
# need a restart, so restart_required is True ONLY when the value actually
# changed — a no-op re-save never asks for a restart.
return {"configured": url, "restart_required": changed, "presets": _HF_MIRROR_PRESETS}
# ── Updates panel: shipped changelog + pre-migration DB backup state ────────
# (feat/safe-updates). Both are read-only, local-first surfaces for
# Settings → Updates: the "What's new" viewer reads the CHANGELOG.md that
# ships with the app, and the backup line shows the newest pre-migration
# snapshot written by core.db_backup before `alembic upgrade head` runs.
@router.get("/changelog")
def get_changelog(limit_versions: int = Query(5, ge=1, le=50)):
"""Structured release notes from the shipped CHANGELOG.md (newest first).
Bullets are raw markdown-lite (bold leads, `code`, (#NNN) refs) — the
frontend renders them safely without HTML. `available: false` when this
install has no changelog (never an error: the viewer just hides)."""
from core import changelog
path = changelog.changelog_path()
if not path:
return {"available": False, "releases": []}
try:
with open(path, encoding="utf-8") as fh:
releases = changelog.parse_changelog(fh.read(), limit_versions)
except Exception:
logger.exception("changelog parse failed")
return {"available": False, "releases": []}
return {"available": bool(releases), "releases": releases}
@router.get("/db-backup")
def get_db_backup_state():
"""Newest pre-migration database backup (or none yet). Feeds the
"your data is backed up before every update" line in Settings Updates."""
from core import db_backup
from core.config import DB_PATH
latest = db_backup.latest_backup(DB_PATH)
return {
"available": latest is not None,
"latest": latest,
"count": len(db_backup.list_backups(DB_PATH)),
"keep": db_backup.KEEP_BACKUPS,
}
+4 -360
View File
@@ -11,28 +11,14 @@ from __future__ import annotations
import asyncio
import json
import logging
import os
import sys
from fastapi import APIRouter, HTTPException
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from core import prefs
from utils import hf_progress
from utils import download_aggregator
# Weight-floor scan (MM2-07 / #352) lives in ``models.py`` — the lowest module in
# the setup import graph — so install-time validation here, the first-run
# install-state detector (#622), and load-time repair share one set of floors and
# can't drift apart. ``_MIN_WEIGHT_BYTES``/``_WEIGHT_FLOORS`` re-exported for tests.
from .models import ( # noqa: F401
KNOWN_MODELS,
invalidate_cache,
snapshot_has_weights,
disk_space_error,
_MIN_WEIGHT_BYTES,
_WEIGHT_FLOORS,
)
from .models import KNOWN_MODELS, invalidate_cache
logger = logging.getLogger("omnivoice.setup.download")
router = APIRouter()
@@ -40,181 +26,6 @@ router = APIRouter()
# Cooldown: prevent rapid re-install after a failure. Maps repo_id → last_fail_time.
_install_cooldowns: dict[str, float] = {}
_COOLDOWN_SECS = 60.0
# Evict cooldown entries older than this so the dict can't grow unbounded across
# a long-lived process (MM2-06). Anything past the cooldown window is dead state.
_COOLDOWN_TTL_SECS = 3600.0
def _sweep_cooldowns(now: float) -> None:
"""Drop cooldown entries older than the TTL (MM2-06). Keeps the dict bounded
without this it accumulated one entry per ever-failed repo forever."""
stale = [k for k, t in _install_cooldowns.items() if (now - t) > _COOLDOWN_TTL_SECS]
for k in stale:
_install_cooldowns.pop(k, None)
# Repo_ids the user asked to cancel (FDL-11). Checked between retry attempts.
# Note: a single in-flight snapshot_download/Xet fetch is not interruptible
# mid-file in hf_hub 1.7.2 — cancel stops further retries, marks the row
# cancelled, and clears the cooldown so a cancel isn't rate-limited.
_cancelled: set[str] = set()
def _download_max_workers() -> int:
"""Parallel-FILES worker count for snapshot_download (FDL-02). Default 8 —
don't crank it: Xet already parallelises *within* each file via concurrent
byte-range gets, so a high count just multiplies buffer pressure. Override
via prefs / OMNIVOICE_DOWNLOAD_MAX_WORKERS for power users."""
raw = prefs.resolve("download_max_workers", env="OMNIVOICE_DOWNLOAD_MAX_WORKERS", default=8)
try:
return max(1, int(raw))
except (TypeError, ValueError):
return 8
def _download_endpoint() -> "str | None":
"""Optional HF endpoint override (FDL-10 mirror path, opt-in). Returned as a
per-call ``endpoint=`` rather than a process-wide HF_ENDPOINT mutation. A
mirror routes through the classic LFS path (no Xet) documented in
docs/downloading-models.md."""
ep = prefs.resolve("hf_endpoint", env="HF_ENDPOINT", default=None)
return ep or None
def apply_xet_env() -> None:
"""Apply opt-in Xet tuning knobs to the environment before a download
(FDL-04). Both default OFF; env wins over the prefs store. high-performance
can *hurt* low-RAM machines (needs lots of RAM/bandwidth); HDD-sequential
avoids parallel-write thrash on spinning disks. Idempotent."""
import os as _os
high_perf = prefs.resolve("xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False)
if _truthy(high_perf):
_os.environ["HF_XET_HIGH_PERFORMANCE"] = "1"
hdd_seq = prefs.resolve("xet_hdd_sequential_write", env="HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY", default=False)
if _truthy(hdd_seq):
_os.environ["HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY"] = "1"
def _truthy(v) -> bool:
if isinstance(v, bool):
return v
return str(v).strip().lower() in {"1", "true", "yes", "on"}
class _InstallCancelled(Exception):
"""Raised inside the install worker when the user cancels (FDL-11)."""
def compute_plan(plan_files) -> dict:
"""Summarise a snapshot_download(dry_run=True) result into the install_plan
payload (FDL-05): total bytes, bytes already cached (skipped), bytes that
will actually download, and file counts. ``will_download`` defaults to
``not is_cached`` for forward-compat with older DryRunFileInfo shapes."""
total = sum(int(getattr(f, "file_size", 0) or 0) for f in plan_files)
cached = sum(
int(getattr(f, "file_size", 0) or 0)
for f in plan_files if getattr(f, "is_cached", False)
)
will = [
f for f in plan_files
if getattr(f, "will_download", not getattr(f, "is_cached", False))
]
to_dl = sum(int(getattr(f, "file_size", 0) or 0) for f in will)
n_files = len(plan_files)
n_cached = sum(1 for f in plan_files if getattr(f, "is_cached", False))
return {
"total_bytes": total,
"cached_bytes": cached,
"to_download_bytes": to_dl,
"n_files": n_files,
"n_cached": n_cached,
}
def _segmented_enabled() -> bool:
"""IDM-style multi-connection accelerator (FDL-09), default **ON**. The app
forces the legacy-LFS path (HF_HUB_DISABLE_XET=1) for clear progress, but that
path is single-stream and slow this restores parallel byte-range speed AND
real live progress, and falls back to snapshot_download on any error so it
can never compromise a correct install. Default-on so first-run downloads are
fast out of the box (pairs with an HF token for higher rate limits); set
OMNIVOICE_SEGMENTED_DOWNLOAD=0 to force the single-stream path."""
return _truthy(prefs.resolve(
"segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=True,
))
def _xet_active() -> bool:
"""True only when hf_xet is installed AND not disabled. The app sets
HF_HUB_DISABLE_XET=1 by default, so this is normally False which is when
the segmented accelerator pays off."""
import importlib.util
if importlib.util.find_spec("hf_xet") is None:
return False
return os.environ.get("HF_HUB_DISABLE_XET", "").strip().lower() not in {"1", "true", "yes", "on"}
def _repo_cancelled(repo_id: str) -> bool:
return repo_id in _cancelled
def _segmented_snapshot(repo_id: str, *, endpoint: "str | None") -> str:
"""Fetch every file of a repo via the segmented downloader into the HF
cache, mirroring hf_hub_download's blob+snapshot+refs layout so the result
is indistinguishable from snapshot_download (FDL-09) keeping /models
install-state, is_cached, and delete working. Feeds real bytes to the
aggregator. Raises on any error; the caller falls back to snapshot_download.
"""
import asyncio as _asyncio
from huggingface_hub import HfApi, constants as _C
from huggingface_hub.file_download import (
hf_hub_url, get_hf_file_metadata, repo_folder_name, _create_symlink,
)
from services.segmented_download import segmented_download
from services.token_resolver import resolve as _resolve_token
token = _resolve_token()
api = HfApi(endpoint=endpoint, token=token)
info = api.repo_info(repo_id, repo_type="model")
commit = info.sha
files = [s.rfilename for s in (info.siblings or [])]
if not commit or not files:
raise RuntimeError("repo_info returned no commit/siblings")
repo_dir = os.path.join(_C.HF_HUB_CACHE, repo_folder_name(repo_id=repo_id, repo_type="model"))
blobs_dir = os.path.join(repo_dir, "blobs")
snap_dir = os.path.join(repo_dir, "snapshots", commit)
refs_dir = os.path.join(repo_dir, "refs")
for d in (blobs_dir, snap_dir, refs_dir):
os.makedirs(d, exist_ok=True)
for rel in files:
if _repo_cancelled(repo_id):
raise _InstallCancelled()
url = hf_hub_url(repo_id, rel, endpoint=endpoint, revision=commit)
meta = get_hf_file_metadata(url, token=token)
etag = (meta.etag or "").strip('"')
if not etag:
raise RuntimeError(f"no etag for {rel}")
blob_path = os.path.join(blobs_dir, etag)
pointer = os.path.join(snap_dir, rel)
os.makedirs(os.path.dirname(pointer), exist_ok=True)
if not os.path.exists(blob_path):
_asyncio.run(segmented_download(
meta.location or url, blob_path,
token=token, expected_size=meta.size, expected_etag=etag,
on_bytes=lambda d, k=rel: download_aggregator.add_bytes(repo_id, k, d),
cancel_check=lambda: _repo_cancelled(repo_id),
))
if not os.path.lexists(pointer):
_create_symlink(blob_path, pointer, new_blob=True)
# refs/main → commit so scan_cache_dir maps the revision correctly.
try:
with open(os.path.join(refs_dir, "main"), "w") as f:
f.write(commit)
except OSError:
pass
return snap_dir
# ── SSE Download Stream ───────────────────────────────────────────────────
@@ -231,41 +42,6 @@ def _safe_put(queue: asyncio.Queue, event) -> None:
pass
# Minimum size for "this snapshot actually contains model weights". An
# interrupted snapshot_download can leave config/tokenizer files but no
# weights; the install then looks complete and synthesis later fails with
# "does not appear to have a file named pytorch_model.bin or
# model.safetensors" (#352). 5 MB clears every weight format we ship
# (safetensors/bin shards, onnx, pt, gguf) without false-positiving on
# config-only aux repos.
def _validate_snapshot_has_weights(repo_id: str, snapshot_path: str) -> None:
"""Raise OSError when a finished snapshot has no plausible weight file —
surfaces the truncated-download class (#352) at install time, where the
retry loop and the UI's re-download path can deal with it, instead of at
first synthesis with an opaque transformers error.
Delegates the weight check to ``models.snapshot_has_weights`` (single source of
the floors); only the install-time error message lives here."""
if snapshot_has_weights(snapshot_path):
return
biggest = 0
try:
for root, _dirs, files in os.walk(snapshot_path, followlinks=True):
for f in files:
try:
biggest = max(biggest, os.path.getsize(os.path.join(root, f)))
except OSError:
continue
except OSError:
pass
raise OSError(
f"{repo_id}: download finished but no model weights were found in the "
"snapshot (largest file "
f"{biggest} bytes). The download was likely interrupted — delete the "
"model in Settings → Models and install it again."
)
@router.get("/setup/download-stream")
async def setup_download_stream():
"""SSE: forward every HuggingFace download tqdm update as a JSON event."""
@@ -322,7 +98,6 @@ async def install_model(req: InstallModelRequest):
)
# Cooldown guard — don't retry if the same model just failed.
import time as _time_check
_sweep_cooldowns(_time_check.time()) # bound the dict (MM2-06)
last_fail = _install_cooldowns.get(req.repo_id)
if last_fail and (_time_check.time() - last_fail) < _COOLDOWN_SECS:
remaining = int(_COOLDOWN_SECS - (_time_check.time() - last_fail))
@@ -337,7 +112,6 @@ async def install_model(req: InstallModelRequest):
def _do():
token = hf_progress.current_repo_id.set(req.repo_id)
_cancelled.discard(req.repo_id) # clear any stale cancel from a prior run
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
@@ -351,22 +125,7 @@ async def install_model(req: InstallModelRequest):
LocalEntryNotFoundError,
)
logger.info("model install starting: %s", req.repo_id)
# Apply opt-in Xet tuning knobs (high-perf / HDD) before downloading.
apply_xet_env()
# Drive snapshot_download explicitly (FDL-02): pass our progress-
# emitting tqdm subclass so progress is deterministic + Xet-aware
# (Xet feeds bytes into whatever tqdm_class is supplied), bound the
# parallel-files worker count, and honour an optional mirror endpoint.
dl_kwargs: dict = {
"repo_id": req.repo_id,
"max_workers": _download_max_workers(),
}
_tqdm_cls = hf_progress.tracked_tqdm_class()
if _tqdm_cls is not None:
dl_kwargs["tqdm_class"] = _tqdm_cls
_endpoint = _download_endpoint()
if _endpoint:
dl_kwargs["endpoint"] = _endpoint
dl_kwargs: dict = {"repo_id": req.repo_id}
if sys.platform == "win32":
dl_kwargs["local_dir_use_symlinks"] = False
@@ -394,91 +153,12 @@ async def install_model(req: InstallModelRequest):
hb = threading.Thread(target=_heartbeat, daemon=True)
hb.start()
# Pre-flight (FDL-05): a dry-run resolve gives the UI an accurate
# denominator — total bytes, bytes already cached (skipped), and the
# bytes that will actually download — BEFORE any byte flows. Seeds
# the overall aggregator so its bar/ETA are correct from the first
# event. Degrades gracefully (totals=None) on older/gated repos.
_preflight_kwargs = {"repo_id": req.repo_id, "dry_run": True}
if _endpoint:
_preflight_kwargs["endpoint"] = _endpoint
try:
_plan = snapshot_download(**_preflight_kwargs)
_summary = compute_plan(_plan)
# Disk-space guard (before a single byte flows): the preflight
# gives an exact "to download" size, so reject an install that
# would overrun the cache volume — with the numbers named —
# instead of failing mid-download with a cryptic OSError. No-op
# when it fits or the size is unknown. Same on every platform.
_disk_err = disk_space_error(_summary["to_download_bytes"])
if _disk_err:
logger.info("model install %s: rejected — %s", req.repo_id, _disk_err)
_resolving.set() # stop the heartbeat thread before we bail
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"downloaded": 0, "total": 0, "pct": 0.0,
"phase": "install_error",
"error": _disk_err,
})
# A disk-full is not a transient network failure — don't set
# a cooldown (freeing space, not waiting, is the fix). The
# outer finally still cleans up the aggregator + context.
return
download_aggregator.start(
req.repo_id,
total_bytes=_summary["to_download_bytes"],
files_total=max(0, _summary["n_files"] - _summary["n_cached"]),
)
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"phase": "install_plan",
**_summary,
})
except Exception as _pf_err:
# No preflight (older/gated repo, mirror without dry-run, etc.):
# fall back to today's fill-in-as-files-appear behaviour.
logger.info("model install %s: preflight unavailable (%s)", req.repo_id, _pf_err)
download_aggregator.start(req.repo_id)
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"phase": "install_plan",
"total_bytes": None,
"cached_bytes": None,
"to_download_bytes": None,
"n_files": None,
"n_cached": None,
})
_max_attempts = 5
_attempt = 0
while True:
if req.repo_id in _cancelled:
raise _InstallCancelled()
_attempt += 1
try:
# Segmented accelerator (FDL-09, default ON): parallel
# byte-range fetch with real live progress, for the
# legacy-LFS path. Any failure falls through to
# snapshot_download — the accelerator can never compromise a
# correct install.
_snapshot_path = None
if _attempt == 1 and _segmented_enabled() and not _xet_active():
try:
_snapshot_path = _segmented_snapshot(req.repo_id, endpoint=_endpoint)
except _InstallCancelled:
raise
except Exception as _seg_err:
logger.info(
"segmented download for %s failed (%s); falling back to snapshot_download",
req.repo_id, _seg_err,
)
_snapshot_path = None
if _snapshot_path is None:
_snapshot_path = snapshot_download(**dl_kwargs)
_validate_snapshot_has_weights(req.repo_id, _snapshot_path)
snapshot_download(**dl_kwargs)
break
except (HfHubHTTPError, LocalEntryNotFoundError, OSError) as net_err:
if _attempt >= _max_attempts:
@@ -499,10 +179,6 @@ async def install_model(req: InstallModelRequest):
_t.sleep(_backoff)
# Stop heartbeat once download completes
_resolving.set()
# Flush the overall bar to 100% with the true byte total (FDL-06):
# under Xet the per-file byte bars don't surface completion, so the
# aggregator can sit below 100% even though every file landed.
download_aggregator.complete(req.repo_id)
logger.info("model install done: %s", req.repo_id)
hf_progress.emit({
"repo_id": req.repo_id,
@@ -510,58 +186,26 @@ async def install_model(req: InstallModelRequest):
"downloaded": 0, "total": 0, "pct": 1.0,
"phase": "install_done",
})
_install_cooldowns.pop(req.repo_id, None) # success clears any cooldown (MM2-06)
invalidate_cache()
except _InstallCancelled:
_resolving.set()
logger.info("model install cancelled: %s", req.repo_id)
# A cancel is user intent, not a failure — don't set a cooldown.
_install_cooldowns.pop(req.repo_id, None)
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"downloaded": 0, "total": 0, "pct": 0.0,
"phase": "install_cancelled",
})
except Exception as e:
_resolving.set()
logger.info("model install failed for %s: %s", req.repo_id, e)
import time as _time_fail
_install_cooldowns[req.repo_id] = _time_fail.time()
# #874: when the install failed because the configured HF mirror is
# unreachable, name the mirror + the setting instead of leaking the
# raw connectivity error. No-op for every other failure.
from core.failure import append_hf_mirror_hint
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"downloaded": 0, "total": 0, "pct": 0.0,
"phase": "install_error",
"error": append_hf_mirror_hint(str(e)),
"error": str(e),
})
finally:
_cancelled.discard(req.repo_id)
download_aggregator.finish(req.repo_id)
hf_progress.current_repo_id.reset(token)
loop.create_task(asyncio.to_thread(_do))
return {"status": "install_started", "repo_id": req.repo_id}
@router.post("/models/install/cancel")
async def cancel_install(req: InstallModelRequest):
"""Request cancellation of an in-flight install (FDL-11).
Best-effort: stops further retry attempts and marks the row cancelled. A
single in-flight snapshot_download/Xet fetch isn't interruptible mid-file
in hf_hub 1.7.2, so an already-streaming file finishes; the cancel takes
effect at the next retry boundary. Clears the cooldown so the user can
immediately restart."""
_cancelled.add(req.repo_id)
_install_cooldowns.pop(req.repo_id, None)
return {"cancelling": req.repo_id}
# ── Delete ─────────────────────────────────────────────────────────────────
@router.delete("/models/{repo_id:path}")
+4 -169
View File
@@ -123,66 +123,6 @@ def hf_cache_dir() -> str:
)
# ── Disk-space guard (shared, single-sourced) ──────────────────────────────
# MIN_FREE_GB is the headroom we insist on keeping free on the model-cache
# volume — the wizard's absolute pre-install floor AND the extra buffer the
# per-install check demands on top of the download itself, so an "Install all"
# can't fill the disk to the brim (setup/download.py). Lives here — the lowest
# module in the setup import graph — so the wizard, the /models header, and the
# install endpoint can't drift apart (mirrors the weight-floor single-sourcing).
_GIB = 1024 ** 3
MIN_FREE_GB = 10
def disk_free_bytes(path: "str | None" = None) -> int:
"""Free bytes on the volume backing *path* (defaults to the HF cache).
Walks up to the nearest existing ancestor so a not-yet-created cache dir
still probes the correct mount point. ``shutil.disk_usage`` is cross-platform
(macOS/Windows/Linux) so this behaves identically everywhere. Never raises.
"""
import shutil
try:
p = Path(path or hf_cache_dir()).resolve()
while not p.exists():
parent = p.parent
if parent == p: # reached the volume root
break
p = parent
return int(shutil.disk_usage(str(p)).free)
except Exception:
return 0
def disk_space_error(to_download_bytes: "int | None", *, cache_dir: "str | None" = None) -> "str | None":
"""Actionable message when *to_download_bytes* (+ MIN_FREE_GB headroom) won't
fit on the cache volume; ``None`` when it fits, the size is unknown, or the
volume can't be probed (never block on missing information).
Names the three numbers a user needs to act needs X, headroom Y, have Z
so "Install all" can't silently overrun the disk (issue: no pre-install disk
check). Platform-agnostic; applied identically on macOS/Windows/Linux.
"""
if not to_download_bytes or to_download_bytes <= 0:
return None # unknown plan (older/gated repo, mirror without dry-run) → don't block
cache = cache_dir or hf_cache_dir()
free = disk_free_bytes(cache)
if free <= 0:
return None # couldn't probe the volume → don't block on missing info
required = int(to_download_bytes) + MIN_FREE_GB * _GIB
if free >= required:
return None
def _gb(n: int) -> str:
return f"{n / _GIB:.1f} GB"
return (
f"Not enough disk space to install: this download needs {_gb(int(to_download_bytes))} "
f"plus {MIN_FREE_GB} GB free headroom ({_gb(required)} total), but only {_gb(free)} "
f"is free at {cache}. Free up space (or move the model cache to a bigger volume) and retry."
)
def _repo_dir_name(repo_id: str) -> str:
"""HF cache dir name for a repo: 'k2-fsa/OmniVoice''models--k2-fsa--OmniVoice'."""
return "models--" + repo_id.replace("/", "--")
@@ -206,94 +146,6 @@ def _hub_cache_roots() -> list[str]:
return roots
# ── Weight-presence (truncated-cache) detection ─────────────────────────────
# A cache that downloaded config/tokenizer files but not the weight shard still
# occupies bytes on disk, so a size-only "installed" check (#352/#581/#606) reads
# it as installed and the first-run wizard hides the re-download button, stranding
# the user (#622). These helpers tell a *complete* snapshot from a truncated one by
# checking for a plausible weight file — the same class `download.py` guards at
# install time and `model_manager.py` repairs at load time. Shared here (the lowest
# module in the setup import graph; `download.py` imports from this module) so the
# floors live in exactly one place and can't drift between the three call sites.
_MIN_WEIGHT_BYTES = 5 * 1024 * 1024 # tensor formats: a real shard is ≥ a few MB
# Per-extension floors. ONNX graphs are legitimately small (a complete model can be
# well under 5 MB), so they get a lower floor that still rejects a bytes-only partial.
_WEIGHT_FLOORS = {
".safetensors": _MIN_WEIGHT_BYTES,
".bin": _MIN_WEIGHT_BYTES,
".ckpt": _MIN_WEIGHT_BYTES,
".pt": _MIN_WEIGHT_BYTES,
".pth": _MIN_WEIGHT_BYTES,
".gguf": _MIN_WEIGHT_BYTES,
".onnx": 64 * 1024,
}
def snapshot_has_weights(snapshot_path: str) -> bool:
"""True when a finished snapshot dir holds a plausible weight file.
A snapshot is complete if it contains a recognized weight file meeting its
per-extension floor OR any file the global 5 MB floor (the lenient catch for
non-standard weight names). Returns True when the path can't be inspected — an
un-walkable dir must never be reported as truncated, only a confirmed weight-less
one. `getsize` follows symlinks, so HF's snapshot→blob links resolve correctly;
a broken link (missing blob) raises OSError and is skipped, i.e. counts as absent.
"""
try:
for root, _dirs, files in os.walk(snapshot_path, followlinks=True):
for f in files:
try:
size = os.path.getsize(os.path.join(root, f))
except OSError:
continue
ext = os.path.splitext(f)[1].lower()
floor = _WEIGHT_FLOORS.get(ext)
if floor is not None and size >= floor:
return True
if size >= _MIN_WEIGHT_BYTES:
return True
except OSError:
return True # can't inspect — don't mislabel as truncated
return False
def _snapshot_dirs(repo_id: str) -> list[str]:
"""Existing snapshot revision dirs for a repo across the candidate cache roots."""
name = _repo_dir_name(repo_id)
dirs: list[str] = []
for root in _hub_cache_roots():
snaps = os.path.join(root, name, "snapshots")
try:
for rev in os.listdir(snaps):
rev_dir = os.path.join(snaps, rev)
if os.path.isdir(rev_dir):
dirs.append(rev_dir)
except OSError:
continue
return dirs
def cache_is_complete(model: dict) -> bool:
"""True when this model's on-disk cache is usable (not a truncated download).
Config-only repos (``config_only: true`` in models.yaml e.g. pyannote's
diarisation pipeline, whose real weights live in referenced sub-repos) carry no
weight file of their own, so the weight check would false-positive them as
incomplete (#622 caveat). They're exempt: cache presence alone means complete.
A weight-bearing repo is complete only if at least one of its snapshots has
weights; if no snapshot dir is found on disk we can't prove truncation, so we
don't downgrade (the size-based caller already decided it's cached).
"""
if model.get("config_only"):
return True
dirs = _snapshot_dirs(model["repo_id"])
if not dirs:
return True
return any(snapshot_has_weights(d) for d in dirs)
def _is_cached_on_disk(repo_id: str) -> bool:
"""Direct-filesystem fallback for is_cached when scan_cache_dir is unavailable.
@@ -368,12 +220,8 @@ def is_cached(repo_id: str) -> bool:
except Exception as e:
# scan_cache_dir can raise on Windows (WinError 448 'untrusted mount
# point'); fall back to a direct disk check so a cached model isn't
# mistaken for missing and re-downloaded in a loop (#117/#118). Logged
# at WARNING with the exception type (MM2-09) so this fallback isn't
# invisible when triaging a Windows cache report — it previously logged
# at DEBUG and never showed at the default level.
logger.warning("is_cached: scan_cache_dir failed (%s: %s); using on-disk fallback for %s",
type(e).__name__, e, repo_id)
# mistaken for missing and re-downloaded in a loop (#117/#118).
logger.debug("scan_cache_dir failed (%s); using disk fallback", e)
return _is_cached_on_disk(repo_id)
@@ -434,15 +282,9 @@ def list_models():
out = []
for m in KNOWN_MODELS:
cached = cached_by_repo.get(m["repo_id"])
on_disk = cached is not None and cached["size_on_disk"] > 0
# A size-positive cache can still be a truncated download (config landed,
# weight shard didn't). Treat that as not-installed + incomplete so the
# wizard re-offers the download instead of stranding the user (#622).
incomplete = on_disk and not cache_is_complete(m)
out.append({
**m,
"installed": on_disk and not incomplete,
"incomplete": incomplete,
"installed": cached is not None and cached["size_on_disk"] > 0,
"size_on_disk_bytes": cached["size_on_disk"] if cached else 0,
"nb_files": cached["nb_files"] if cached else 0,
"supported": _model_supported(m),
@@ -451,10 +293,6 @@ def list_models():
"models": out,
"total_installed_bytes": sum(m["size_on_disk_bytes"] for m in out),
"hf_cache_dir": hf_cache_dir(),
# Free space on the cache volume, so the Model Store header can warn
# BEFORE an "Install all" overruns the disk (pairs with the per-install
# disk_space_error guard in setup/download.py).
"disk_free_gb": round(disk_free_bytes() / _GIB, 1),
"platform_tags": _current_platform_tags(),
}
_set_cache("models", response)
@@ -541,9 +379,6 @@ def recommendations():
entries = []
for rid in recommended_ids:
meta = known_by_id.get(rid, {})
# Mirror /models: a truncated cache (weights missing) is not installed, so
# the wizard counts it toward the remaining download instead of "all set".
installed = rid in cached_ids and cache_is_complete(meta or {"repo_id": rid})
entries.append({
"repo_id": rid,
"label": meta.get("label", rid),
@@ -551,7 +386,7 @@ def recommendations():
"size_gb": meta.get("size_gb", 0),
"required": bool(meta.get("required", False)),
"note": meta.get("note"),
"installed": installed,
"installed": rid in cached_ids,
})
to_download_gb = sum(e["size_gb"] for e in entries if not e["installed"])
+21 -52
View File
@@ -18,20 +18,33 @@ import sys
from fastapi import APIRouter
from api.schemas import SetupStatusResponse, PreflightResponse
# MIN_FREE_GB + disk_free_bytes are single-sourced in ``.models`` (the lowest
# module in the setup import graph) so the wizard gate, the /models header, and
# the per-install disk guard can't drift apart.
from .models import REQUIRED_MODELS, hf_cache_dir, is_cached, MIN_FREE_GB, disk_free_bytes
from .models import REQUIRED_MODELS, hf_cache_dir, is_cached
logger = logging.getLogger("omnivoice.setup.wizard")
router = APIRouter()
MIN_FREE_GB = 10
def _disk_free_gb(path: str) -> float:
"""Free GB on the volume containing *path* (thin GB wrapper over the shared
``models.disk_free_bytes``, which walks up to the nearest existing ancestor
for a not-yet-created path)."""
return disk_free_bytes(path) / (1024 ** 3)
"""Return free GB on the volume containing *path*.
If *path* doesn't exist yet (e.g. after a fresh wipe), walk up to the
nearest existing ancestor so ``shutil.disk_usage`` can still probe the
correct mount point.
"""
try:
from pathlib import Path
p = Path(path).resolve()
# Walk up until we find a directory that exists
while not p.exists():
parent = p.parent
if parent == p: # root
break
p = parent
return _shutil.disk_usage(str(p)).free / (1024 ** 3)
except Exception:
return 0.0
# ── Setup Status ───────────────────────────────────────────────────────────
@@ -360,46 +373,6 @@ def preflight():
"status": gpu_status, "detail": gpu_detail, "fix": gpu_fix,
})
# ── GPU routing for the ACTIVE TTS engine (#21 — no silent CPU fallback).
# Distinct from the hardware "gpu" check above: this asks "will the engine
# the user actually selected use that GPU on this host?" Built from the same
# canonical probe + resolver the Engine Compatibility Matrix uses.
try:
from services.tts_backend import gpu_routing_verdict
gpu_routing = gpu_routing_verdict()
except Exception as exc: # never break preflight on a routing hiccup
logger.warning("preflight gpu_routing failed: %s", exc)
gpu_routing = None
if gpu_routing:
_rs = gpu_routing.get("routing_status")
_eng = gpu_routing.get("engine") or "active engine"
_dev = gpu_routing.get("effective_device") or "?"
_why = gpu_routing.get("routing_reason")
if _rs == "accelerated" and not _why:
r_status, r_detail, r_fix = "pass", f"{_eng}{_dev} (accelerated)", None
elif _rs == "accelerated": # driver/arch caveat
r_status, r_detail, r_fix = "warn", f"{_eng}{_dev}: {_why}", (
"GPU selected but may fail at kernel launch — update drivers / "
"reinstall torch for this GPU architecture.")
elif _rs == "cpu_fallback":
r_status, r_detail, r_fix = "warn", (
f"{_eng} runs on CPU here: {_why or 'no GPU path for this host'}"), (
"Pick an engine that supports this host's GPU for a speedup, or "
"continue on CPU (slower).")
elif _rs == "cpu_only":
r_status, r_detail, r_fix = "pass", f"{_eng} → cpu (no accelerator on this host)", None
elif _rs == "unavailable":
r_status, r_detail, r_fix = "fail", (
f"{_eng} can't run on this host: {_why or 'needs a GPU this machine lacks'}"), (
"Select an engine with a CPU path in Settings → Engines.")
else: # "none" / unknown
r_status, r_detail, r_fix = "warn", "No active TTS engine resolved for routing.", (
"Pick an engine in Settings → Engines.")
checks.append({
"id": "gpu_routing", "label": "Active engine routing",
"status": r_status, "detail": r_detail, "fix": r_fix,
})
# ── Network
net_ok = _probe_network()
checks.append({
@@ -427,13 +400,9 @@ def preflight():
"gpu_available": gpu["available"],
"gpu_driver": gpu["driver"],
"gpu_device_name": gpu["device_name"],
# Canonical probe (distinguishes ROCm from CUDA):
"gpu_family": (gpu_routing or {}).get("host_family", "cpu"),
"vram_gb": (gpu_routing or {}).get("vram_gb", 0.0),
"ram_gb": round(ram, 1),
"disk_free_gb": round(free, 1),
},
"gpu_routing": gpu_routing,
}
+87 -122
View File
@@ -18,7 +18,7 @@ import shutil
from core.config import OUTPUTS_DIR, DATA_DIR, CRASH_LOG_PATH, LOG_PATH, IDLE_TIMEOUT_SECONDS
from core.version import APP_VERSION
from services.model_manager import get_model_status, get_best_device, resolve_omnivoice_checkpoint
from services.model_manager import get_model_status, get_best_device
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
# Router-level loopback gate. Every route mounted on `router` (GET + POST,
@@ -107,52 +107,6 @@ def _ui_port() -> int:
return 3901
def _fast_download_status() -> dict:
"""Report the download-acceleration state for the Settings UI (FDL-03).
Reports the *runtime* truth, not just whether hf_xet is importable. The app
currently sets ``HF_HUB_DISABLE_XET=1`` by default (main.py) Xet's chunked
transfer is fast but its progress bypasses our tqdm patch, so the legacy-LFS
path is forced to keep accurate byte progress. So:
* ``xet_installed`` hf_xet present
* ``xet_active`` installed AND not disabled via HF_HUB_DISABLE_XET
* ``xet_enabled`` alias of xet_active (what the UI badge keys off)
Must never throw: /system/info is called on every Settings load.
"""
installed = False
version = None
try:
import hf_xet # noqa: F401
installed = True
try:
from importlib.metadata import version as _ver
version = _ver("hf-xet")
except Exception:
version = None
except Exception:
installed = False
disabled = str(os.environ.get("HF_HUB_DISABLE_XET", "")).strip().lower() in {"1", "true", "yes", "on"}
active = installed and not disabled
try:
from core import prefs
high_perf = prefs.resolve(
"xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False
)
high_perf = high_perf if isinstance(high_perf, bool) else \
str(high_perf).strip().lower() in {"1", "true", "yes", "on"}
except Exception:
high_perf = False
return {
"xet_installed": installed,
"xet_active": active,
"xet_enabled": active, # UI badge: only true when Xet actually runs
"xet_version": version,
"high_performance": bool(high_perf),
}
def _has_hf_token() -> bool:
# Phase 1 AUTH-01..06 cascade. Delegates to the 3-source resolver
# (App → Env → HF-CLI) instead of reading env/HF-CLI directly. This
@@ -174,23 +128,88 @@ def model_status():
@router.get("/model/loaded")
def loaded_models():
"""List all currently loaded models for the flush dropdown (MM2-04).
Thin delegation to the model_lifecycle facade shape unchanged:
``{models, count}``."""
from services import model_lifecycle
return model_lifecycle.list_loaded()
"""Return details about all currently loaded models for the flush dropdown.
Returns a list of models with name, type, device, and estimated VRAM usage.
"""
import services.model_manager as mm
models = []
# 1. TTS model (OmniVoice)
if mm.model is not None:
device = "unknown"
vram_mb = 0
try:
device = str(next(mm.model.parameters()).device) if hasattr(mm.model, 'parameters') else get_best_device()
except Exception:
device = get_best_device()
try:
torch = mm._lazy_torch()
if torch.cuda.is_available():
vram_mb = torch.cuda.memory_allocated() / (1024 ** 2)
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
driver = getattr(torch.mps, "driver_allocated_memory", None)
if driver:
vram_mb = driver() / (1024 ** 2)
except Exception:
pass
models.append({
"id": "tts",
"name": "OmniVoice TTS",
"checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
"device": device,
"vram_mb": round(vram_mb, 1),
"unloadable": True,
})
# 2. ASR model (WhisperX)
if mm.model is not None and hasattr(mm.model, '_asr_pipe') and mm.model._asr_pipe is not None:
models.append({
"id": "asr",
"name": "WhisperX ASR",
"checkpoint": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
"device": "cpu",
"vram_mb": 0,
"unloadable": False, # tied to TTS model lifecycle
})
# 3. Diarization pipeline
if mm._diar_pipeline is not None:
models.append({
"id": "diarization",
"name": "Pyannote Diarization",
"checkpoint": "pyannote/speaker-diarization-3.1",
"device": get_best_device(),
"vram_mb": 0,
"unloadable": True,
})
return {"models": models, "count": len(models)}
@router.post("/model/unload/{model_id}")
async def unload_model(model_id: str):
"""Unload a specific model by id (MM2-04). Delegates to model_lifecycle;
an unknown id maps to HTTP 400. ``tts`` | ``diarization`` |
``sidecar:<id>`` | ``sidecars``."""
from services import model_lifecycle
try:
return await model_lifecycle.unload(model_id)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
"""Unload a specific model by ID."""
import services.model_manager as mm
if model_id == "tts":
async with mm._model_lock:
if mm.model is not None:
mm.model = None
mm.free_vram()
return {"unloaded": "tts", "success": True}
return {"unloaded": "tts", "success": False, "reason": "not loaded"}
elif model_id == "diarization":
if mm._diar_pipeline is not None:
mm._diar_pipeline = None
mm.free_vram()
return {"unloaded": "diarization", "success": True}
return {"unloaded": "diarization", "success": False, "reason": "not loaded"}
else:
raise HTTPException(status_code=400, detail=f"Unknown model id: {model_id}")
@router.get("/system/info", response_model=SystemInfoResponse)
@@ -208,11 +227,10 @@ def system_info():
"outputs_dir": OUTPUTS_DIR,
"crash_log_path": CRASH_LOG_PATH,
"idle_timeout_seconds": IDLE_TIMEOUT_SECONDS,
"model_checkpoint": resolve_omnivoice_checkpoint(), # #693: show the effective checkpoint, not a leaked raw value
"model_checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
"asr_model": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
"translate_provider": os.environ.get("TRANSLATE_PROVIDER", "google"),
"has_hf_token": _has_hf_token(),
"fast_download": _fast_download_status(),
"device": get_best_device(),
"python": sys.version.split()[0],
"platform": sys.platform,
@@ -304,10 +322,9 @@ def _tauri_log_candidates():
os.path.join(home, "Library/Logs/OmniVoice/backend_err.log"),
]
if sys.platform.startswith("linux"):
data_dir = os.environ.get("XDG_DATA_HOME") or os.path.join(home, ".local/share")
state_dir = os.environ.get("XDG_STATE_HOME") or os.path.join(home, ".local/state")
return [
os.path.join(data_dir, bid, "logs", "tauri.log"),
os.path.join(home, ".local/share", bid, "logs", "tauri.log"),
os.path.join(home, ".config", bid, "logs", "tauri.log"),
os.path.join(state_dir, "OmniVoice", "backend.log"),
os.path.join(state_dir, "OmniVoice", "backend_err.log"),
@@ -447,14 +464,6 @@ async def clear_system_logs():
status_code=500,
detail=f"Could not clear log at {p}: {e}. The file may be open in another process or read-only — close tailing tools and retry.",
)
if cleared_any:
# The crash log just shrank to zero — drop any stored ack so a stale
# byte count can't suppress the next 'crash-last-session' notice.
for key in ("crash_log_acked", "crash_log_acked_size"):
try:
prefs_delete(key)
except Exception:
pass
return {"cleared": cleared_any}
@@ -556,21 +565,6 @@ async def flush_memory(unload_model: bool = False):
# ── Actionable notifications ──────────────────────────────────────────────
_GPU_ARCH_WARNING: "list[str | None]" = [] # [-1] = computed result
def _gpu_arch_warning_cached() -> "str | None":
"""check_device_compatibility() once per process (it lazy-imports torch —
too heavy for the 30s notifications poll)."""
if not _GPU_ARCH_WARNING:
try:
from services.model_manager import check_device_compatibility
compatible, warning = check_device_compatibility()
_GPU_ARCH_WARNING.append(None if compatible else warning)
except Exception:
_GPU_ARCH_WARNING.append(None)
return _GPU_ARCH_WARNING[-1]
@router.get("/system/notifications")
def system_notifications():
@@ -602,20 +596,6 @@ def system_notifications():
},
})
# 1b. GPU compute capability unsupported by this torch build (#284) —
# the model "runs" but emits pure noise, the worst silent failure mode
# (RTX 50-series Blackwell sm_120 on pre-cu128 wheels). The loader logs
# this, but a log line never reached the affected users — surface it in
# the panel. Checked once per process: it lazy-imports torch.
gpu_warn = _gpu_arch_warning_cached()
if gpu_warn:
notes.append({
"id": "gpu-arch-unsupported",
"level": "error",
"title": "GPU not supported by this PyTorch build",
"message": gpu_warn + " Until then, output will be noise/garbage.",
})
# 2. Missing ffmpeg
ffmpeg_ok = False
try:
@@ -706,33 +686,18 @@ def _crashed_last_session() -> bool:
if not os.path.exists(CRASH_LOG_PATH):
return False
size = os.path.getsize(CRASH_LOG_PATH)
if size == 0:
acked = int(prefs_get("crash_log_acked_size", 0) or 0)
if size <= acked:
return False
mtime = os.path.getmtime(CRASH_LOG_PATH)
# Composite ack (size + mtime): a bare byte count goes stale after the log
# is truncated — the next crash log can stay smaller than the old acked
# size forever, silently suppressing 'crash-last-session'. The ack only
# holds while it still covers the file's current state.
ack = prefs_get("crash_log_acked")
if isinstance(ack, dict):
if float(ack.get("mtime", 0) or 0) >= mtime and int(ack.get("size", 0) or 0) >= size:
return False
else:
# Legacy size-only ack from older builds.
if size <= int(prefs_get("crash_log_acked_size", 0) or 0):
return False
return mtime < _PROCESS_START_TS
return os.path.getmtime(CRASH_LOG_PATH) < _PROCESS_START_TS
@router.post("/system/crash/ack")
async def ack_crash():
"""Mark the current crash log as seen — dismisses the
'crash-last-session' notification until the log changes again."""
size = mtime = 0
if os.path.exists(CRASH_LOG_PATH):
size = os.path.getsize(CRASH_LOG_PATH)
mtime = os.path.getmtime(CRASH_LOG_PATH)
prefs_set("crash_log_acked", {"size": size, "mtime": mtime})
'crash-last-session' notification until the log grows again."""
size = os.path.getsize(CRASH_LOG_PATH) if os.path.exists(CRASH_LOG_PATH) else 0
prefs_set("crash_log_acked_size", size)
return {"acked_size": size}
+34 -88
View File
@@ -98,30 +98,6 @@ async def ws_tts(websocket: WebSocket):
model = await get_model()
backend = get_active_tts_backend(model=model)
# ── Routing gate (#21 — no silent CPU fallback). WebSockets have
# no response headers, so this uses frames: an error frame +
# close on `unavailable`, a one-time `routing` frame on
# cpu_fallback / accelerated-with-caveat (before any audio).
from core.device_caps import detect_host_caps
from services.engine_routing import resolve_routing, routing_notice
from core.scrub import scrub_text
_routing = resolve_routing(
getattr(backend, "gpu_compat", ("cpu",)), detect_host_caps())
if _routing["routing_status"] == "unavailable":
await websocket.send_json({
"type": "error",
"detail": scrub_text(_routing["routing_reason"])
or "engine cannot run on this host",
})
continue # don't stream; wait for the next request
_notice = routing_notice(_routing)
if _notice:
await websocket.send_json({
"type": "routing",
"status": _notice[0],
"reason": scrub_text(_notice[1]) if _notice[1] else None,
})
# Build generation kwargs
kw: dict = {"speed": data.get("speed", 1.0)}
if data.get("language"):
@@ -168,80 +144,50 @@ async def ws_tts(websocket: WebSocket):
except Exception:
kw["voice"] = voice
# Wave 1.4: split the request into sentences so the first
# sentence's audio streams while later sentences are still
# synthesizing — this is the time-to-first-audio win. The
# chunker handles abbreviations/acronyms/decimals and CJK /
# non-Latin terminators; single-sentence requests behave
# exactly like the old single-shot path.
from services.sentence_chunker import SentenceChunker
_chunker = SentenceChunker(language=(data.get("language") or "en"))
sentences = _chunker.push(text)
sentences.extend(_chunker.flush())
if not sentences:
sentences = [text]
# Run generation in the GPU pool
import functools
from services.model_manager import run_on_gpu_pool_guarded
from services.model_manager import _gpu_pool
loop = asyncio.get_running_loop()
def _generate(sentence_text):
def _generate():
from services.audio_dsp import apply_mastering, normalize_audio
wav = backend.generate(sentence_text, **kw)
wav = backend.generate(text, **kw)
sr_actual = backend.sample_rate
# Like _run_tts in openai_compat: studio engines (VoxCPM2)
# opt out of the broadcast mastering chain. This is the
# other route that runs the active backend, so it needs the
# same guard. Loudness normalisation still runs.
if not getattr(backend, "applies_own_mastering", False):
wav = apply_mastering(wav, sample_rate=sr_actual)
wav = apply_mastering(wav, sample_rate=sr_actual)
wav = normalize_audio(wav, target_dBFS=-2.0)
return wav, sr_actual
wav_tensor, sr = await loop.run_in_executor(_gpu_pool, _generate)
# Send metadata after generation so sample_rate is real
await websocket.send_json({
"type": "start",
"sample_rate": sr,
"channels": 1,
"format": "pcm16",
"engine": backend.id,
})
# Stream PCM16 chunks over the WebSocket
import torch
total_samples = 0
sr = backend.sample_rate
started = False
# Convert to 16-bit PCM
pcm = (wav_tensor * 32767).clamp(-32768, 32767).to(torch.int16)
if pcm.ndim == 2:
pcm = pcm[0] # mono
pcm_bytes = pcm.numpy().tobytes()
for sentence in sentences:
# Bounded + pool-reset on hang so a wedged generate can't
# starve the GPU pool and brick the backend (#730 class). On
# timeout GpuJobTimeoutError propagates to the handler below,
# which sends an actionable error frame.
wav_tensor, sr = await run_on_gpu_pool_guarded(
functools.partial(_generate, sentence),
what="TTS generate",
)
total_samples = len(pcm)
sent_samples = 0
chunk_bytes = CHUNK_SAMPLES * 2 # 2 bytes per int16 sample
if not started:
# Send metadata after the first generation so
# sample_rate is real (lazy-loading engines report
# their true rate only once weights are up).
await websocket.send_json({
"type": "start",
"sample_rate": sr,
"channels": 1,
"format": "pcm16",
"engine": backend.id,
})
started = True
# Convert to 16-bit PCM and stream
pcm = (wav_tensor * 32767).clamp(-32768, 32767).to(torch.int16)
if pcm.ndim == 2:
pcm = pcm[0] # mono
pcm_bytes = pcm.numpy().tobytes()
n_samples = len(pcm)
sent_samples = 0
while sent_samples < n_samples:
end = min(sent_samples + CHUNK_SAMPLES, n_samples)
chunk = pcm_bytes[sent_samples * 2: end * 2]
await websocket.send_bytes(chunk)
sent_samples = end
# Yield to event loop between chunks for responsiveness
await asyncio.sleep(0)
total_samples += n_samples
while sent_samples < total_samples:
end = min(sent_samples + CHUNK_SAMPLES, total_samples)
start_byte = sent_samples * 2
end_byte = end * 2
chunk = pcm_bytes[start_byte:end_byte]
await websocket.send_bytes(chunk)
sent_samples = end
# Yield to event loop between chunks for responsiveness
await asyncio.sleep(0)
gen_time = round(time.perf_counter() - t0, 3)
duration = round(total_samples / sr, 3)
-27
View File
@@ -34,8 +34,6 @@ class SystemInfoResponse(BaseModel):
asr_model: str = "unknown"
translate_provider: str = "unknown"
has_hf_token: bool = False
# Xet fast-download backend state (FDL-03): {xet_enabled, xet_version, high_performance}
fast_download: dict | None = None
device: str = "cpu"
python: str = ""
platform: str = ""
@@ -129,41 +127,16 @@ class DeviceInfo(BaseModel):
gpu_available: bool = False
gpu_driver: str | None = None
gpu_device_name: str | None = None
# From the canonical device probe (core.device_caps) — distinguishes ROCm
# from CUDA, unlike the legacy nvidia-smi-based gpu_vendor/gpu_backend.
gpu_family: str = "cpu"
vram_gb: float = 0.0
ram_gb: float = 0.0
disk_free_gb: float = 0.0
class GpuRouting(BaseModel):
"""Routing verdict for the active TTS engine on THIS host (#21).
Distinct from the per-engine `routing_*` keys in `/engines`: this is the
single verdict for the *currently-selected* engine, surfaced in preflight +
diagnose so the user hears about a CPU fallback / unavailable GPU before a
slow or failed synth no silent CPU fallback.
"""
model_config = ConfigDict(extra="allow")
engine: str | None = None # active TTS engine id
effective_device: str | None = None # device it will actually use here
routing_status: str | None = None # accelerated|cpu_fallback|cpu_only|unavailable|none
routing_reason: str | None = None # scrubbed; null when none
host_family: str = "cpu" # detect_host_caps().family
vram_gb: float = 0.0
class PreflightResponse(BaseModel):
"""GET /setup/preflight"""
ok: bool
has_warnings: bool = False
checks: list[PreflightCheck] = Field(default_factory=list)
device: DeviceInfo
# Explicit field (PreflightResponse has no extra="allow") so the verdict
# survives serialization instead of being silently dropped.
gpu_routing: GpuRouting | None = None
class InstallModelRequest(BaseModel):
Binary file not shown.
-75
View File
@@ -14,10 +14,6 @@
# required (optional) — true if the app needs this model to function
# platforms (optional) — restrict to specific OS+arch tags (e.g. darwin-arm64, cuda)
# note (optional) — shown in the UI as a tooltip/footnote
# config_only (optional) — true for pipeline repos that ship no weight file of
# their own (weights live in referenced sub-repos). Such
# a cache is legitimately tiny, so the truncated-download
# (weights-missing) detector must NOT flag it incomplete.
# ─────────────────────────────────────────────────────────────────────────
models:
@@ -120,83 +116,12 @@ models:
size_gb: 0.05
note: "Smallest/fastest Moonshine, sub-200ms latency. Lower accuracy than base. Requires moonshine-onnx."
# ── sherpa-onnx live dictation (ONNX, CPU, streaming + offline) ────────
# Live faster-than-real-time dictation via the k2-fsa/sherpa-onnx runtime.
# `engine: sherpa-onnx`, `dictation_id` (backend model id), and `tag`
# (offline | streaming) are extra fields the model-store list passes through
# so the dictation UI can filter/group these (role=ASR, engine=sherpa-onnx).
# Requires `uv add sherpa-onnx` (CPU wheels, all platforms).
- repo_id: "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8"
label: "Parakeet TDT v3 (sherpa-onnx — dictation, 25 EU langs)"
role: ASR
size_gb: 0.18
engine: sherpa-onnx
dictation_id: sherpa-parakeet-tdt-v3
tag: offline
note: "Recommended live-dictation default. CPU, int8 ONNX. Requires sherpa-onnx."
- repo_id: "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8"
label: "Parakeet TDT v2 (sherpa-onnx — dictation, English)"
role: ASR
size_gb: 0.17
engine: sherpa-onnx
dictation_id: sherpa-parakeet-tdt-v2
tag: offline
note: "English live dictation. CPU, int8 ONNX. Requires sherpa-onnx."
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20"
label: "Zipformer Bilingual (sherpa-onnx — streaming, zh+en)"
role: ASR
size_gb: 0.13
engine: sherpa-onnx
dictation_id: sherpa-zipformer-bilingual-zh-en
tag: streaming
note: "True streaming partials as you speak (zh+en). CPU. Requires sherpa-onnx."
- repo_id: "csukuangfj/sherpa-onnx-streaming-paraformer-bilingual-zh-en"
label: "Paraformer Bilingual (sherpa-onnx — streaming, zh+en)"
role: ASR
size_gb: 0.115
engine: sherpa-onnx
dictation_id: sherpa-paraformer-bilingual-zh-en
tag: streaming
note: "True streaming partials (zh+en). CPU. Requires sherpa-onnx."
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-en-20M-2023-02-17"
label: "Zipformer Streaming EN 20M (sherpa-onnx — streaming, English)"
role: ASR
size_gb: 0.128
engine: sherpa-onnx
dictation_id: sherpa-zipformer-en-20m
tag: streaming
note: "Tiny English streaming model, very low latency. CPU. Requires sherpa-onnx."
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23"
label: "Zipformer Streaming ZH 14M (sherpa-onnx — streaming, Chinese)"
role: ASR
size_gb: 0.074
engine: sherpa-onnx
dictation_id: sherpa-zipformer-zh-14m
tag: streaming
note: "Tiny Chinese streaming model, very low latency. CPU. Requires sherpa-onnx."
- repo_id: "csukuangfj/sherpa-onnx-whisper-tiny"
label: "Whisper Tiny (sherpa-onnx — dictation, 90+ langs)"
role: ASR
size_gb: 0.116
engine: sherpa-onnx
dictation_id: sherpa-whisper-tiny
tag: offline
note: "Multilingual offline dictation (auto-detect). CPU, int8 ONNX. Requires sherpa-onnx."
# ── Diarisation ───────────────────────────────────────────────────────
- repo_id: "pyannote/speaker-diarization-3.1"
label: "pyannote speaker diarisation (multi-speaker videos)"
role: Diarisation
size_gb: 0.8
config_only: true # pipeline repo; real weights live in referenced sub-repos
note: "Needs an HF_TOKEN with license accepted."
# ── Optional TTS ──────────────────────────────────────────────────────
-138
View File
@@ -1,138 +0,0 @@
"""Parse the shipped CHANGELOG.md into structured release notes.
Feeds ``GET /api/settings/changelog`` the Settings Updates "What's new"
viewer. Local-first by design: the changelog ships with the app (repo root in
dev; copied into the packaged project dir by the Tauri bootstrap alongside
README.md), so the viewer works fully offline.
The house format (see CHANGELOG.md / the release-notes hard rule):
## [X.Y.Z] — DATE
one-paragraph headline (the "intro")
### Added / Fixed / Changed / ...
- **Bold one-line lead.** 1-3 lines of plain-English why. (#NNN)
Bullets may be a single long line (recent sections) *or* hard-wrapped across
indented continuation lines (older sections) the parser normalizes both to
one logical line per bullet. Bullets stay raw markdown-lite; the frontend's
safe renderer handles **bold** / `code` / (#NNN) refs.
"""
from __future__ import annotations
import os
import re
#: ``## [0.3.9] — 2026-07-02`` (em/en dash or hyphen; date optional).
_RELEASE_RE = re.compile(r"^##\s+\[(?P<version>[^\]]+)\]\s*(?:[—–-]\s*(?P<date>.+?))?\s*$")
_SECTION_RE = re.compile(r"^###\s+(?P<title>.+?)\s*$")
_BULLET_RE = re.compile(r"^\s*[-*]\s+(?P<text>.*\S)\s*$")
def changelog_path() -> str | None:
"""The shipped CHANGELOG.md, or None when this install doesn't have one.
``backend/core/changelog.py`` two levels up is the project root: the
repo root in dev, and ``<env>/project`` in packaged installs (where the
bootstrap copies CHANGELOG.md next to README.md). ``OMNIVOICE_CHANGELOG``
overrides for tests/containers.
"""
override = os.environ.get("OMNIVOICE_CHANGELOG")
if override:
return override if os.path.isfile(override) else None
here = os.path.dirname(os.path.abspath(__file__))
candidate = os.path.join(os.path.dirname(os.path.dirname(here)), "CHANGELOG.md")
return candidate if os.path.isfile(candidate) else None
def _looks_like_release_version(version: str) -> bool:
"""Only released ``X.Y.Z...`` sections (skip ``[Unreleased]`` etc.)."""
return bool(re.match(r"^v?\d", version.strip()))
def parse_changelog(text: str, limit_versions: int = 5) -> list[dict]:
"""CHANGELOG.md text → newest-first list of releases::
{"version": "0.3.9", "date": "2026-07-02", "intro": "",
"sections": [{"title": "Fixed", "bullets": ["", ]}, ]}
Tolerates both single-line bullets and older hard-wrapped bullets
(continuation lines are joined with a space). Content between the version
heading and the first ``###`` becomes ``intro`` (paragraphs joined by
blank lines).
"""
releases: list[dict] = []
release: dict | None = None
section: dict | None = None
intro_parts: list[str] = []
bullet_open = False # last bullet may still absorb continuation lines
intro_new_para = True
def close_release():
nonlocal release, section, intro_parts, bullet_open, intro_new_para
if release is not None:
release["intro"] = "\n\n".join(p for p in intro_parts if p)
release["sections"] = [s for s in release["sections"] if s["bullets"]]
releases.append(release)
release = None
section = None
intro_parts = []
bullet_open = False
intro_new_para = True
for raw in text.splitlines():
m = _RELEASE_RE.match(raw)
if m:
close_release()
if len(releases) >= limit_versions:
break
version = m.group("version").strip().lstrip("v")
if not _looks_like_release_version(version):
continue # e.g. [Unreleased] — skip until the next heading
release = {
"version": version,
"date": (m.group("date") or "").strip(),
"intro": "",
"sections": [],
}
continue
if release is None:
continue
line = raw.strip()
if not line:
bullet_open = False
intro_new_para = True
continue
sm = _SECTION_RE.match(raw)
if sm:
section = {"title": sm.group("title"), "bullets": []}
release["sections"].append(section)
bullet_open = False
continue
bm = _BULLET_RE.match(raw)
if bm:
if section is None:
# Rare: a bullet before any ### heading — group it untitled.
section = {"title": "", "bullets": []}
release["sections"].append(section)
section["bullets"].append(bm.group("text"))
bullet_open = True
continue
if section is not None:
if bullet_open and section["bullets"]:
# Hard-wrapped bullet continuation (older sections) → join.
section["bullets"][-1] += " " + line
continue
# Headline paragraph(s) before the first ### section.
if intro_new_para or not intro_parts:
intro_parts.append(line)
else:
intro_parts[-1] += " " + line
intro_new_para = False
close_release()
return releases[:limit_versions]
+8 -246
View File
@@ -3,8 +3,6 @@ import sqlite3
import logging
from contextlib import contextmanager
from core.config import DB_PATH
from core import db_backup
from core.version import APP_VERSION
logger = logging.getLogger("omnivoice.db")
@@ -51,12 +49,6 @@ _BASE_SCHEMA = """
personality TEXT DEFAULT '',
description TEXT DEFAULT '',
is_demo INTEGER DEFAULT 0,
verified_own_voice INTEGER DEFAULT 0,
consent_text TEXT DEFAULT '',
consent_audio_path TEXT DEFAULT '',
consent_recorded_at REAL DEFAULT NULL,
kind TEXT DEFAULT 'clone',
vd_states TEXT DEFAULT NULL,
created_at REAL
);
CREATE TABLE IF NOT EXISTS generation_history (
@@ -146,35 +138,6 @@ _BASE_SCHEMA = """
value TEXT NOT NULL,
updated_at REAL NOT NULL
);
-- Wave 2.2: per-agent MCP voice bindings. An MCP client (Claude Code,
-- Cursor, ) identified by the X-OmniVoice-Client-Id header it sends is
-- bound to a default voice profile / engine. Fresh installs create it
-- here; v0.3.x upgrades get it via alembic 0004.
CREATE TABLE IF NOT EXISTS mcp_client_bindings (
client_id TEXT PRIMARY KEY,
label TEXT NOT NULL DEFAULT '',
profile_id TEXT,
default_engine TEXT,
last_seen_at REAL,
created_at REAL
);
-- Expressive-TTS Spec 01 Phase 1: user pronunciation dictionary. A
-- per-language wordrespelling map applied as pure text substitution
-- before synthesis (Settings Pronunciation). Fresh installs create it
-- here; existing DBs get it via alembic 0008_pronunciation_dictionary.
-- Both paths converge on this identical schema (dual-path discipline).
CREATE TABLE IF NOT EXISTS pronunciation_entries (
id TEXT PRIMARY KEY,
term TEXT NOT NULL,
replacement TEXT NOT NULL DEFAULT '',
type TEXT NOT NULL DEFAULT 'respelling',
language TEXT NOT NULL DEFAULT '*',
enabled INTEGER NOT NULL DEFAULT 1,
created_at REAL
);
CREATE INDEX IF NOT EXISTS idx_pron_lang ON pronunciation_entries(language);
"""
# Only tables/columns this module is allowed to ALTER. Prevents SQL injection via
@@ -224,72 +187,6 @@ def _migrate(conn, current: int) -> int:
return current
def _reconcile_additive_columns(conn) -> None:
"""Make the live schema converge to ``_BASE_SCHEMA`` by ADDing any column the
canonical schema declares but an existing table is missing the belt for
when alembic can't run on an upgraded DB.
``CREATE TABLE IF NOT EXISTS`` (init_db) never adds columns to a table that
already exists, the legacy ``_migrate`` only knows pre-0.3 columns, and
``_run_alembic_upgrade`` swallows failures. So a DB whose ``alembic_version``
is stamped at a removed revision (e.g. after running a preview build), or
where alembic isn't importable in the bundled interpreter, would otherwise
lose every alembic-era additive column forever the ``no such column:
consent_audio_path`` 500 (#552/#547), and the same class for
``kind``/``vd_states``/``is_demo``/.... Additive only: never drops or retypes
a column, so it is safe and backward-compatible with existing user data. The
canonical names/types/defaults come solely from ``_BASE_SCHEMA`` (developer
controlled), so the ALTER is injection-safe.
"""
canon = sqlite3.connect(":memory:")
try:
canon.executescript(_BASE_SCHEMA)
_tables_sql = "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'"
live_tables = {r[0] for r in conn.execute(_tables_sql)}
for table in (r[0] for r in canon.execute(_tables_sql)):
if table not in live_tables:
continue # whole table missing → init_db's CREATE already made it
have = {r[1] for r in conn.execute(f"PRAGMA table_info({table})")}
# (cid, name, type, notnull, dflt_value, pk)
for _cid, name, ctype, notnull, dflt, _pk in canon.execute(f"PRAGMA table_info({table})"):
if name in have or not _IDENT_RE.match(name):
continue
ddl = f'ALTER TABLE "{table}" ADD COLUMN "{name}" {ctype or "TEXT"}'
if dflt is not None:
ddl += f" DEFAULT {dflt}"
elif notnull:
ddl += " DEFAULT ''" # SQLite requires a default to ADD a NOT NULL column
try:
conn.execute(ddl)
logger.info("schema reconcile: added missing column %s.%s", table, name)
except sqlite3.OperationalError as exc:
if "duplicate column" not in str(exc).lower():
logger.warning("schema reconcile ALTER %s.%s failed: %s", table, name, exc)
conn.commit()
finally:
canon.close()
def ensure_schema() -> None:
"""Idempotently ensure the base tables + additive columns exist.
A runtime self-heal for a DB that somehow missed init e.g. a write hitting
``no such table: generation_history`` (#710) because ``init_db()``'s
``executescript`` never took on that DB. Safe to call anytime: it's just
``CREATE ... IF NOT EXISTS`` plus the additive-only column reconcile, so it
never drops or retypes anything and is backward-compatible with user data.
Cheaper than ``init_db()`` (skips the legacy ``_migrate`` + alembic), so a
write path can call it on a schema error and retry without a 500.
"""
conn = get_db()
try:
conn.executescript(_BASE_SCHEMA)
_reconcile_additive_columns(conn)
conn.commit()
finally:
conn.close()
def init_db():
conn = get_db()
try:
@@ -298,11 +195,6 @@ def init_db():
new_version = _migrate(conn, version)
if new_version != version:
conn.execute(f"PRAGMA user_version = {new_version}")
# Converge any alembic-era additive columns that CREATE TABLE IF NOT
# EXISTS + the legacy _migrate don't add to a pre-existing table
# (consent_audio_path, kind, ...). Runs regardless of whether alembic
# below succeeds, so an unrunnable alembic can't leave a 500-ing schema.
_reconcile_additive_columns(conn)
conn.commit()
finally:
conn.close()
@@ -314,88 +206,12 @@ def init_db():
_run_alembic_upgrade()
class MigrationError(RuntimeError):
"""A schema migration failed *while executing*. Startup must NOT continue
on a possibly half-migrated database the caller lets this propagate so
the process stops with an actionable message naming the pre-migration
backup (see ``core.db_backup``). Restore is deliberately manual: silently
auto-restoring the snapshot could itself discard user data."""
def _reconcile_after_alembic_skip() -> None:
"""Converge the schema directly when alembic can't run at all (not
importable, or stamped at a removed revision #552/#547) so additive
columns still land instead of 500-ing on `no such column`. Only for the
"nothing was applied" classes; a mid-migration failure must NOT reach
here (see MigrationError)."""
try:
conn = get_db()
try:
_reconcile_additive_columns(conn)
finally:
conn.close()
except Exception as exc: # noqa: BLE001
logger.warning("schema reconcile after alembic skip also failed: %s", exc)
def _stamped_revisions(db_path: str) -> set | None:
"""Revisions recorded in ``alembic_version`` (empty set = never stamped),
or None when the DB can't be read."""
try:
conn = sqlite3.connect(db_path)
try:
try:
return {r[0] for r in conn.execute("SELECT version_num FROM alembic_version")}
except sqlite3.OperationalError:
return set() # table absent — nothing ever stamped
finally:
conn.close()
except Exception: # noqa: BLE001
return None
def _plan_alembic(cfg) -> str:
"""Decide what an ``upgrade head`` run would actually do:
- ``up_to_date`` stamped at head; upgrade is a no-op.
- ``pending`` migrations WILL execute (snapshot the DB first).
- ``unknown_revision`` stamped at a revision this build doesn't ship
(previewstable downgrade, #552/#547); upgrade would fail before
applying anything, so skip it and reconcile additively instead.
- ``indeterminate`` can't tell; treat like pending (snapshot, run).
"""
try:
from alembic.script import ScriptDirectory
script = ScriptDirectory.from_config(cfg)
known = {rev.revision for rev in script.walk_revisions()}
heads = set(script.get_heads())
stamped = _stamped_revisions(DB_PATH)
if stamped is None:
return "indeterminate"
if stamped and not stamped <= known:
return "unknown_revision"
if stamped == heads:
return "up_to_date"
return "pending"
except Exception: # noqa: BLE001
return "indeterminate"
def _run_alembic_upgrade() -> None:
"""`alembic upgrade head` on startup, wrapped in the data-safety net.
Failure classes are handled differently on purpose:
- alembic unavailable / stamped at an unknown revision **non-fatal**
(nothing was applied; warn + `_reconcile_additive_columns` keeps the
schema converged, exactly the pre-existing #552/#547 behavior).
- migrations actually pending the DB is snapshotted first
(``omnivoice.db.backup-<version>-<n>``, newest 3 kept), then upgraded.
- a migration fails **while executing** raise :class:`MigrationError`:
startup stops with a message naming the backup, instead of silently
running the app on a half-migrated DB.
"""
"""Best-effort `alembic upgrade head` on startup. Non-fatal: if alembic
isn't reachable (e.g. user running a stripped-down install or migrations
were already applied out-of-band), log a warning and move on. The
_BASE_SCHEMA CREATE TABLE IF NOT EXISTS above guarantees the runtime
schema is correct regardless."""
try:
import os
from alembic import command
@@ -411,62 +227,8 @@ def _run_alembic_upgrade() -> None:
return
cfg = Config(ini)
cfg.set_main_option("sqlalchemy.url", f"sqlite:///{DB_PATH}")
except Exception as exc: # noqa: BLE001 — alembic not importable / bad ini
logger.warning("alembic upgrade head skipped: %s", exc)
_reconcile_after_alembic_skip()
return
plan = _plan_alembic(cfg)
if plan == "up_to_date":
return
if plan == "unknown_revision":
logger.warning(
"alembic_version is stamped at a revision this build doesn't ship "
"(preview/newer build ran on this DB) — skipping alembic and "
"reconciling the schema additively (#552/#547)"
)
_reconcile_after_alembic_skip()
return
# Migrations may actually execute: snapshot the DB first so a failed or
# interrupted migration can never cost user data. A backup problem alone
# must not brick startup (the >500 MB skip is by design), so log and go on.
# ``db_backup``/``APP_VERSION`` are module-level imports (top of file), not
# re-imported here: a test that patches ``core.db_backup.MAX_BACKUP_DB_BYTES``
# on the object it imported at collection must see the same object this
# function uses. A lazy ``from core import db_backup`` would re-resolve
# through the (possibly re-imported) ``core`` package and silently miss the
# patch after another suite purged ``core.*`` from ``sys.modules``.
backup_path = None
try:
backup_path = db_backup.snapshot_before_migration(DB_PATH, APP_VERSION)
except Exception: # noqa: BLE001
logger.exception("Pre-migration DB backup failed — continuing without one")
try:
command.upgrade(cfg, "head")
except Exception as exc:
if "Can't locate revision" in str(exc):
# Belt for an unknown-revision case _plan_alembic missed: alembic
# bails before applying anything, so the old non-fatal path is safe.
logger.warning("alembic upgrade head skipped: %s", exc)
_reconcile_after_alembic_skip()
return
backup_note = (
f"A backup of your data from just before the migration is at: {backup_path}"
if backup_path
else "No pre-migration backup was written this run (see the log above)"
)
msg = (
f"Database migration failed while running: {exc}. "
f"OmniVoice stopped instead of running on a partially migrated database, "
f"and nothing was auto-restored (your database at {DB_PATH} was left "
f"exactly as the failed migration left it). "
f"{backup_note}. "
"What to do: relaunch to retry; if it keeps failing, report it at "
"https://github.com/debpalash/OmniVoice-Studio/issues (keep the backup file). "
"To roll back manually: quit the app, replace omnivoice.db with the backup "
"file, and reinstall the previous version."
)
logger.error(msg)
raise MigrationError(msg) from exc
# Don't block startup on a migration tooling problem. The runtime
# schema is already correct via _BASE_SCHEMA.
logger.warning("alembic upgrade head skipped: %s", exc)
-169
View File
@@ -1,169 +0,0 @@
"""Pre-migration SQLite safety net (data-safe updates).
Before ``alembic upgrade head`` applies *pending* migrations at startup
which is exactly the first launch of a new app version that changed the
schema the live database is snapshotted next to itself as
``omnivoice.db.backup-<version>-<n>`` so a failed or interrupted migration
can never cost user data (voices, projects, history, settings).
Design rules (owner intent: "never corrupt/erase user data on update"):
- Snapshots use the SQLite online-backup API (``sqlite3.Connection.backup``),
not a file copy the live DB runs in WAL mode, so a plain copy could miss
everything still sitting in ``omnivoice.db-wal``.
- Only the most recent ``KEEP_BACKUPS`` snapshots are kept; older ones are
pruned so backups can't grow without bound.
- DBs larger than ``MAX_BACKUP_DB_BYTES`` are skipped with a log line (a
multi-hundred-MB copy on every schema upgrade is worse than the risk it
hedges on those installs).
- Restore is NEVER automatic. On migration failure the caller
(``core.db._run_alembic_upgrade``) stops startup and names the backup path
so the user (or a support thread) decides a silent auto-restore could
itself discard data written after the snapshot.
"""
from __future__ import annotations
import logging
import os
import re
import sqlite3
import time
logger = logging.getLogger("omnivoice.db.backup")
#: Keep this many snapshots; older ones are pruned after each new snapshot.
KEEP_BACKUPS = 3
#: Skip the snapshot (with a log line) when the DB exceeds this size.
MAX_BACKUP_DB_BYTES = 500 * 1024 * 1024
#: ``<db name>.backup-<version>-<n>`` — ``<version>`` may itself contain
#: dashes (preview builds stamp ``0.3.9-41``), so the counter is the final
#: ``-<digits>`` group.
_BACKUP_SUFFIX_RE = re.compile(r"\.backup-(?P<version>.+)-(?P<n>\d+)$")
def _sanitize_version(version: str) -> str:
"""Version string → filesystem-safe fragment (defense in depth; real
versions are semver and already safe)."""
safe = re.sub(r"[^A-Za-z0-9._-]", "_", str(version).strip()) or "unknown"
return safe[:64]
def list_backups(db_path: str) -> list[str]:
"""All backup files for ``db_path``, newest first (mtime desc)."""
directory = os.path.dirname(os.path.abspath(db_path)) or "."
base = os.path.basename(db_path)
try:
names = os.listdir(directory)
except OSError:
return []
out = []
for name in names:
if not name.startswith(base + ".backup-"):
continue
if not _BACKUP_SUFFIX_RE.search(name[len(base):]):
continue
out.append(os.path.join(directory, name))
out.sort(key=lambda p: (_mtime(p), p), reverse=True)
return out
def _mtime(path: str) -> float:
try:
return os.path.getmtime(path)
except OSError:
return 0.0
def latest_backup(db_path: str) -> dict | None:
"""Newest backup as ``{"path", "created_at", "size_bytes"}`` or None."""
backups = list_backups(db_path)
if not backups:
return None
path = backups[0]
try:
st = os.stat(path)
except OSError:
return None
return {"path": path, "created_at": st.st_mtime, "size_bytes": st.st_size}
def _next_counter(db_path: str, safe_version: str) -> int:
"""Next free ``<n>`` for this version so a re-run never overwrites an
earlier snapshot of the same version."""
base = os.path.basename(db_path)
prefix = f"{base}.backup-{safe_version}-"
highest = 0
for path in list_backups(db_path):
name = os.path.basename(path)
if not name.startswith(prefix):
continue
tail = name[len(prefix):]
if tail.isdigit():
highest = max(highest, int(tail))
return highest + 1
def prune_backups(db_path: str, keep: int = KEEP_BACKUPS) -> list[str]:
"""Delete all but the ``keep`` newest backups. Returns deleted paths."""
deleted = []
for path in list_backups(db_path)[keep:]:
try:
os.remove(path)
deleted.append(path)
logger.info("Pruned old DB backup %s", path)
except OSError as exc:
logger.warning("Could not prune old DB backup %s: %s", path, exc)
return deleted
def snapshot_before_migration(db_path: str, version: str) -> str | None:
"""Snapshot ``db_path`` to ``<db>.backup-<version>-<n>``.
Returns the backup path, or None when skipped (no DB yet, or DB larger
than ``MAX_BACKUP_DB_BYTES``). Raises on an actual backup failure so the
caller can decide (the caller treats that as "continue without a backup",
logged loudly a backup problem must not brick startup by itself).
"""
if not os.path.isfile(db_path):
logger.debug("No DB at %s yet — nothing to back up", db_path)
return None
size = os.path.getsize(db_path)
if size > MAX_BACKUP_DB_BYTES:
logger.info(
"Skipping pre-migration DB backup: %s is %.0f MB (> %.0f MB limit)",
db_path, size / (1024 * 1024), MAX_BACKUP_DB_BYTES / (1024 * 1024),
)
return None
safe_version = _sanitize_version(version)
target = f"{db_path}.backup-{safe_version}-{_next_counter(db_path, safe_version)}"
tmp = f"{target}.part-{os.getpid()}"
src = sqlite3.connect(db_path)
try:
dst = sqlite3.connect(tmp)
try:
# Online backup: consistent snapshot including WAL contents.
src.backup(dst)
dst.commit()
finally:
dst.close()
except BaseException:
try:
os.remove(tmp)
except OSError:
pass
raise
finally:
src.close()
os.replace(tmp, target)
# A same-second rotation must still rank the new file newest.
try:
now = time.time()
os.utime(target, (now, now))
except OSError:
pass
logger.info("Pre-migration DB backup written: %s (%.1f MB)", target, size / (1024 * 1024))
prune_backups(db_path)
return target
-336
View File
@@ -1,336 +0,0 @@
"""Free-text voice-description → voice-design parameter mapper (issue #317).
Parity with the hosted omnivoice.app "Describe your voice" field, implemented
fully locally: a deterministic keyword/phrase mapper that projects a natural-
language description (e.g. ``"a warm elderly British storyteller, slightly
raspy"``) onto the **existing** voice-design parameter space — the same six
categories the Design tab's attribute picker drives (Gender / Age / Pitch /
Style / EnglishAccent / ChineseDialect).
Design notes
============
* **No model, no network.** This is an ordered synonym-table matcher, not an
LLM call it runs identically on macOS/Windows/Linux with zero deps beyond
the stdlib, preserving the local-first guarantee.
* **Single source of truth.** Every canonical token this module can emit is
validated at import time against the engine taxonomy in
``omnivoice/utils/voice_design.py`` (loaded via ``core.archetypes``), so the
mapper can never produce an instruct item the engine validator would reject
(the issue-#89 / #115 crash modes). The Chinese translations of each token
(e.g. ````/``中年``) are *derived* from that taxonomy, never hardcoded.
* **Ordered rules, first match wins.** Within a category, rules are checked in
a hand-ordered list so more specific phrases outrank generic ones
("young child" child, not young adult; "very deep" very low pitch, not
low pitch). Within one rule, the earliest occurrence in the text is reported
as the matched phrase. Deterministic by construction.
* **Graceful degradation.** Anything the taxonomy can't express (timbre words
like "raspy", role words like "storyteller") is returned in ``unmatched`` so
the UI can tell the user exactly which parts were ignored instead of failing
silently (issue #317's validation-feedback note). A description with no
matches at all yields all-``Auto`` attrs and an empty instruct.
Localization note (CLAUDE.md): the only hardcoded CJK here is
``DIALECT_PINYIN`` a functional pinyin Chinese-dialect-token mapping
(model vocabulary, like ``frontend/src/utils/constants.js``). Registered in
``tests/test_no_hardcoded_cjk.py``'s allowlist with this justification.
"""
from __future__ import annotations
import re
# Reuse the taxonomy already loaded (stdlib-only, by file path) by the
# archetype engine — same single source of truth, one loader to maintain.
from core.archetypes import _VD
_EN_TO_ZH = _VD._INSTRUCT_EN_TO_ZH # {"male": "男", ...}
_ZH_RE = _VD._ZH_RE
_VALID = _VD._INSTRUCT_ALL_VALID # every token the engine accepts
_DIALECTS = set(_VD._INSTRUCT_CATEGORIES[5]) # the 12 Chinese dialect tokens
# Category names match the frontend's CATEGORIES keys (utils/constants.js) and
# the archetype ``attrs`` shape, so the response drops straight into vdStates.
CATEGORY_ORDER = ("Gender", "Age", "Pitch", "Style", "EnglishAccent", "ChineseDialect")
# ── Pinyin / romanized names → Chinese-dialect tokens (functional vocabulary) ─
DIALECT_PINYIN = {
"henan": "河南话",
"shaanxi": "陕西话",
"sichuan": "四川话",
"szechuan": "四川话",
"guizhou": "贵州话",
"yunnan": "云南话",
"guilin": "桂林话",
"jinan": "济南话",
"shijiazhuang": "石家庄话",
"gansu": "甘肃话",
"ningxia": "宁夏话",
"qingdao": "青岛话",
"dongbei": "东北话",
"northeastern chinese": "东北话",
}
# ── Synonym tables ────────────────────────────────────────────────────────────
# Per category: ordered list of (canonical_token, [phrases]). First rule with
# any hit wins the category, so specific phrases must precede generic ones.
# Each canonical token's Chinese translation from the taxonomy is appended
# automatically at compile time (so "中年" maps to "middle-aged", etc.).
_GENDER_RULES = [
("female", [
"female", "woman", "women", "lady", "ladies", "girl", "girls",
"feminine", "gal", "grandma", "grandmother", "granny", "mother",
"mom", "mum", "aunt", "auntie", "queen", "princess", "actress",
"she", "her",
]),
("male", [
"male", "man", "men", "guy", "guys", "boy", "boys", "masculine",
"gentleman", "gentlemen", "dude", "grandpa", "grandfather", "father",
"dad", "uncle", "king", "prince", "actor", "he", "him", "his",
]),
]
# Order is load-bearing: "child" precedes "young adult" so "young child" →
# child; "middle-aged" precedes "elderly" so elderly's bare "aged" synonym
# can't fire inside the hyphenated "middle-aged" (hyphen is a \b boundary);
# "elderly" precedes "young adult" so grandparent words don't fall through.
_AGE_RULES = [
("child", [
"child", "children", "kid", "kiddo", "toddler", "little boy",
"little girl", "young boy", "young girl", "small child", "childlike",
]),
("teenager", ["teenager", "teen", "teenage", "adolescent"]),
("middle-aged", [
"middle-aged", "middle aged", "middle age", "midlife", "forties",
"fifties", "sixties", "mature",
]),
("elderly", [
"elderly", "old man", "old woman", "old lady", "older man",
"older woman", "elder", "senior", "aged", "grandpa", "grandfather",
"grandma", "grandmother", "granny", "retired", "seventies",
"eighties", "nineties", "old",
]),
("young adult", [
"young adult", "young woman", "young man", "young lady", "youthful",
"twenties", "thirties", "college", "young",
]),
]
# "very …" rules precede their plain counterparts so "very deep" doesn't stop
# at "deep". Bare "low"/"high" only count next to a voice word (pitch/voice/
# tone/register) to avoid false hits like "high quality" or "low effort".
_PITCH_RULES = [
("very low pitch", [
"very low pitch", "very low-pitched", "very low pitched",
"very low voice", "very low tone", "very deep", "extremely deep",
"extremely low", "ultra deep", "booming",
]),
("very high pitch", [
"very high pitch", "very high-pitched", "very high pitched",
"very high voice", "very high tone", "extremely high", "squeaky",
"shrill", "falsetto", "chipmunk",
]),
("low pitch", [
"low pitch", "low-pitched", "low pitched", "low voice", "low tone",
"low register", "deep", "deeper", "bass", "baritone", "husky",
]),
("high pitch", [
"high pitch", "high-pitched", "high pitched", "high voice",
"high tone", "high register", "soprano",
]),
("moderate pitch", [
"moderate pitch", "medium pitch", "medium-pitched", "medium pitched",
"mid-range", "midrange", "average pitch", "moderate",
]),
]
_STYLE_RULES = [
("whisper", [
"whisper", "whispering", "whispered", "whispery", "hushed",
"breathy", "soft-spoken", "soft spoken",
]),
]
# Bare "english" means the language, so only the explicit "english accent"
# phrase maps to british. "chinese" maps to the chinese *accent* (English
# speech with a Chinese accent); actual dialect words live in DIALECT_PINYIN.
_ACCENT_RULES = [
("american accent", [
"american", "america", "usa", "us accent", "midwestern",
"californian", "new york",
]),
("british accent", [
"british", "britain", "english accent", "england", "uk accent",
"london", "cockney", "posh", "received pronunciation",
]),
("australian accent", ["australian", "australia", "aussie"]),
("canadian accent", ["canadian", "canada"]),
("indian accent", ["indian", "india"]),
("chinese accent", ["chinese accent", "chinese-accented", "chinese"]),
("korean accent", ["korean", "korea"]),
("japanese accent", ["japanese", "japan"]),
("portuguese accent", ["portuguese", "portugal", "brazilian", "brazil"]),
("russian accent", ["russian", "russia"]),
]
_DIALECT_RULES = [
(token, [pinyin for pinyin, tok in DIALECT_PINYIN.items() if tok == token])
for token in sorted(_DIALECTS)
]
_RULES = {
"Gender": _GENDER_RULES,
"Age": _AGE_RULES,
"Pitch": _PITCH_RULES,
"Style": _STYLE_RULES,
"EnglishAccent": _ACCENT_RULES,
"ChineseDialect": _DIALECT_RULES,
}
# Import-time guard: every canonical token must be in the engine taxonomy, so
# a taxonomy rename upstream fails loudly here instead of at synthesis time.
for _cat_rules in _RULES.values():
for _token, _ in _cat_rules:
assert _token in _VALID, f"describe_voice token not in taxonomy: {_token!r}"
for _tok in DIALECT_PINYIN.values():
assert _tok in _DIALECTS, f"DIALECT_PINYIN value not a taxonomy dialect: {_tok!r}"
# ── Pattern compilation ───────────────────────────────────────────────────────
def _compile_phrase(phrase: str) -> re.Pattern:
"""Compile a synonym phrase to a regex.
Latin phrases get word boundaries (so "male" never fires inside "female",
"old" never inside "bold") and flexible separators (space or hyphen, so
"middle aged" also matches "middle-aged"). CJK phrases match as plain
substrings word boundaries are meaningless without spaces.
"""
if _ZH_RE.search(phrase):
return re.compile(re.escape(phrase))
parts = [re.escape(p) for p in re.split(r"[ -]+", phrase) if p]
return re.compile(r"\b" + r"[\s\-]+".join(parts) + r"\b")
def _compiled_rules():
out = {}
for cat, rules in _RULES.items():
compiled = []
for token, phrases in rules:
pats = list(phrases)
# Derive the Chinese form of each canonical token from the
# taxonomy (e.g. "middle-aged" → "中年") — never hardcoded here.
zh = _EN_TO_ZH.get(token)
if zh:
pats.append(zh)
if token not in pats:
pats.append(token) # the canonical token always matches itself
compiled.append((token, [_compile_phrase(p) for p in pats]))
out[cat] = compiled
return out
_COMPILED = _compiled_rules()
# "<N> year(s) old / <N>-year-old / <N> yo" → an age bracket. Runs before the
# keyword rules so the trailing "old" never misfires as elderly.
_AGE_NUM = re.compile(
r"\b(\d{1,3})(?:[\s\-]*(?:years?|yrs?|yr)[\s\-]*old|[\s\-]*(?:yo|y/o))\b"
)
def _age_token_for(years: int) -> str:
if years <= 12:
return "child"
if years <= 19:
return "teenager"
if years <= 39:
return "young adult"
if years <= 64:
return "middle-aged"
return "elderly"
def _normalize(description: str) -> str:
text = (description or "").lower()
text = text.replace("", "'").replace("", "'")
text = text.replace("", '"').replace("", '"')
return re.sub(r"[ \t]+", " ", text)
def _match_category(category: str, text: str):
"""Return (token, match) for the first rule with a hit, else None.
Rule order decides the winning token; within the winning rule the earliest
occurrence in the text is reported as the matched phrase.
"""
if category == "Age":
m = _AGE_NUM.search(text)
if m:
return _age_token_for(int(m.group(1))), m
for token, patterns in _COMPILED[category]:
best = None
for pat in patterns:
m = pat.search(text)
if m is not None and (best is None or m.start() < best.start()):
best = m
if best is not None:
return token, best
return None
# Fragment splitter for the "unmatched" report: clause separators (incl. the
# CJK comma/ideographic stop, which CJK descriptions use instead of ASCII).
_FRAGMENT = re.compile(r"[^,;.!?()\n,。;!?、]+")
_HAS_CONTENT = re.compile(r"[\w一-鿿]")
def parse_description(description: str) -> dict:
"""Map a free-text voice description onto the design parameter space.
Returns a dict with:
* ``attrs`` full category token map (``"Auto"`` where nothing
matched); same shape as the Design tab's ``vdStates``.
* ``instruct`` validator-safe instruct string built from the matched
tokens, in canonical category order (may be ``""``).
* ``matched`` list of ``{category, token, phrase}`` for transparency.
* ``unmatched`` clause fragments that contributed no attribute, so the
UI can show what was ignored instead of failing silently.
"""
text = _normalize(description)
attrs = {cat: "Auto" for cat in CATEGORY_ORDER}
matched = []
spans = []
for category in CATEGORY_ORDER:
hit = _match_category(category, text)
if hit is None:
continue
token, m = hit
attrs[category] = token
matched.append({"category": category, "token": token, "phrase": m.group(0)})
spans.append((m.start(), m.end()))
# Accents are English-only and dialects Chinese-only in the engine
# taxonomy; a dialect voice speaks Chinese, so an accent token alongside
# it is contradictory (the issue-#114 conflict class). Dialect wins.
if attrs["ChineseDialect"] != "Auto" and attrs["EnglishAccent"] != "Auto":
dropped = attrs["EnglishAccent"]
attrs["EnglishAccent"] = "Auto"
matched = [m for m in matched if not (m["category"] == "EnglishAccent" and m["token"] == dropped)]
instruct = ", ".join(attrs[c] for c in CATEGORY_ORDER if attrs[c] != "Auto")
unmatched = []
for frag in _FRAGMENT.finditer(text):
if not _HAS_CONTENT.search(frag.group(0)):
continue
lo, hi = frag.start(), frag.end()
if any(s < hi and e > lo for s, e in spans):
continue
unmatched.append(frag.group(0).strip())
return {
"attrs": attrs,
"instruct": instruct,
"matched": matched,
"unmatched": unmatched,
}
-279
View File
@@ -1,279 +0,0 @@
"""Canonical host compute-capability probe — the single source of truth for
"what can this machine actually accelerate on."
Every routing decision (the engine compatibility matrix, ``/setup/preflight``,
``/system/diagnose``, and the synth-time no-silent-fallback gating) reads from
``detect_host_caps()`` so the probe and the model loader can never disagree.
Design contract (load-bearing):
- **Never raises** to a caller. A broken torch / driver crash degrades to a
cached CPU-only ``probe_ok=False`` result; every endpoint stays responsive
(local-first: the app must work with no GPU and even with a broken torch).
- **No network call** driver/sysctl reads only, no tensor allocation, so it
stays kernel-free on cold start.
- **No new regex** on any driver/device string (CodeQL py/polynomial-redos):
the only string parse is the ``int(driver.split(".")[0])`` shape reused
from the wizard, and arch comparison is plain list membership.
- Distinguishes **ROCm from CUDA** (unlike the gguf ``hardware_probe``):
ROCm-on-HIP presents through ``torch.cuda`` but is reported ``family="rocm"``.
The ``get_best_device()`` loader (``services.model_manager``) delegates its
*family* decision here while keeping its own DirectML branch and the ROCm
``HSA_OVERRIDE_GFX_VERSION`` env side-effect the probe **reads**, the loader
**writes**. (The gguf ``hardware_probe.detect_capabilities()`` rebase onto this
module is a deliberate follow-up: it has its own torch-mocked test suite and a
VRAM-driven quant table that is unaffected by the family rename, so it is kept
out of this backend-only slice.)
"""
from __future__ import annotations
import functools
import platform as _platform
import sys
from dataclasses import dataclass
from typing import Literal
DeviceFamily = Literal["cuda", "rocm", "mps", "xpu", "cpu"]
# Stable substring stamped onto notes that represent a real kernel-launch risk
# (arch/driver mismatch) — as opposed to advisory notes (multi-GPU, VRAM query
# failed, DirectML present). ``engine_routing`` keys the "accelerated, but…"
# caveat off this marker so advisory notes never downgrade an accelerated badge.
KERNEL_RISK_MARKER = "may fail at kernel launch"
# Substring marking a DirectML-present (Windows GPU) host. The probe reports
# such hosts as ``family="cpu"`` (DirectML is not a torch device family); the
# router reads this marker to explain the neutral badge instead of "no GPU".
DIRECTML_MARKER = "DirectML device present"
# NOTE: the NVIDIA driver-version check (min R555 for the bundled CUDA runtime)
# is intentionally NOT done here — it requires shelling to ``nvidia-smi``, which
# would put a subprocess on the cold-start probe path. That check stays in
# ``wizard._detect_gpu`` (preflight), which already runs it. The probe only
# emits the torch-visible SM-arch caveat (cheap, metadata-only).
@dataclass(frozen=True)
class HostCaps:
"""Snapshot of the host's accelerator capability. Immutable + cached."""
family: DeviceFamily
"""Best available accelerator family, else ``"cpu"``."""
available_families: tuple[DeviceFamily, ...]
"""Everything usable; **always includes** ``"cpu"`` (invariant)."""
device_name: str = ""
"""Device 0's name, e.g. ``"NVIDIA RTX 4090"`` / ``"Apple Silicon (MPS)"``."""
vram_gb: float = 0.0
"""CUDA/ROCm total VRAM in GB; MPS = system RAM / 2; 0 for cpu/xpu."""
driver: str | None = None
"""Raw ROCm HIP version string (``torch.version.hip``) or ``None``. The
NVIDIA driver-version check is owned by ``wizard._detect_gpu`` (it already
shells to ``nvidia-smi``); the probe stays subprocess-free."""
notes: tuple[str, ...] = ()
"""Author-controlled English advisories (never user input). Empty on a
clean accelerated host."""
probe_ok: bool = True
"""``False`` only when torch could not be imported (degraded CPU-only)."""
def _probe() -> HostCaps:
"""Run the probe once. Enumerates every failure branch from the spec's
degradation contract; never raises."""
try:
import torch
except Exception:
return HostCaps(
family="cpu",
available_families=("cpu",),
notes=("torch not importable; treating host as CPU-only",),
probe_ok=False,
)
notes: list[str] = []
# Probe EVERY accelerator independently into this list (don't short-circuit
# after the first hit) so `available_families` is honest on hybrid hosts
# (e.g. an NVIDIA GPU + an Intel iGPU exposed via IPEX). The preferred
# `family` is chosen by priority at the end.
detected: list[DeviceFamily] = []
device_name = ""
vram_gb = 0.0
driver: str | None = None
# ── CUDA / ROCm (both present through torch.cuda) ────────────────────
cuda_ok = False
try:
cuda_ok = bool(torch.cuda.is_available())
except Exception as exc: # broken CUDA init (forked process / driver crash)
notes.append(f"CUDA init raised: {type(exc).__name__}")
if cuda_ok:
try:
count = int(torch.cuda.device_count())
except Exception:
count = 0
if count == 0:
notes.append("CUDA reports available but device_count==0")
else:
is_rocm = getattr(torch.version, "hip", None) is not None
detected.append("rocm" if is_rocm else "cuda")
if is_rocm:
driver = getattr(torch.version, "hip", None)
if count > 1:
notes.append(f"{count} GPUs detected; routing reflects device 0")
try:
device_name = torch.cuda.get_device_name(0)
except Exception:
device_name = ""
try:
_free, total = torch.cuda.mem_get_info()
vram_gb = float(total) / (1024 ** 3)
except Exception:
notes.append("VRAM query failed")
# SM-arch mismatch (mirrors model_manager.check_device_compatibility).
try:
major, minor = torch.cuda.get_device_capability(0)
arch_list = getattr(torch.cuda, "_get_arch_list", lambda: [])()
if arch_list:
sm_tag = f"sm_{major}{minor}"
compute_tag = f"compute_{major}{minor}"
if sm_tag not in arch_list and compute_tag not in arch_list:
notes.append(
f"{device_name or 'GPU'} ({sm_tag}) not in this torch "
f"build's archs ({', '.join(arch_list)}) — "
f"{KERNEL_RISK_MARKER}"
)
except Exception:
# Arch metadata unavailable on this torch build — skip the check
# (treated as compatible, exactly as check_device_compatibility).
pass
# ── Intel XPU via IPEX ───────────────────────────────────────────────
try:
import intel_extension_for_pytorch # noqa: F401
if hasattr(torch, "xpu") and torch.xpu.is_available():
detected.append("xpu")
if not device_name:
try:
device_name = torch.xpu.get_device_name(0)
except Exception:
# XPU present but unnamed — family classification still holds.
pass
notes.append("XPU VRAM not queried (unreliable across IPEX versions)")
except Exception:
# IPEX absent or XPU probe failed — no XPU on this host.
pass
# ── Apple Silicon MPS ────────────────────────────────────────────────
try:
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
detected.append("mps")
if not device_name:
device_name = "Apple Silicon (MPS)"
if not vram_gb:
try:
import psutil
vram_gb = float(psutil.virtual_memory().total) / (1024 ** 3) / 2
except Exception:
notes.append("psutil unavailable; MPS VRAM unknown")
except Exception:
# MPS probe raised on a non-Apple/old torch — treat as no MPS.
pass
# ── DirectML — Windows GPU, NOT a torch device family ────────────────
try:
import torch_directml
if torch_directml.device_count() > 0:
notes.append(
f"{DIRECTML_MARKER} (Windows GPU); torch-family probe treats "
f"as non-accelerated"
)
except Exception:
# torch_directml absent (the common case) — no DirectML on this host.
pass
# Preferred family by priority; cpu when nothing accelerated was detected.
family: DeviceFamily = "cpu"
for pref in ("cuda", "rocm", "xpu", "mps"):
if pref in detected:
family = pref # type: ignore[assignment]
break
# available_families: every detected accelerator + cpu, deduped, cpu last.
available: tuple[DeviceFamily, ...] = tuple(dict.fromkeys([*detected, "cpu"]))
return HostCaps(
family=family,
available_families=available,
device_name=device_name,
vram_gb=vram_gb,
driver=driver,
notes=tuple(notes),
probe_ok=True,
)
@functools.lru_cache(maxsize=1)
def detect_host_caps() -> HostCaps:
"""Cached per-process host capabilities. Never raises, makes no network
call, kernel-free on cold start. Host compute capability does not change at
runtime in any supported desktop flow (no GPU hot-plug; switching the active
engine does not re-probe routing is recomputed from these same caps), so
a single probe per process is correct. ``probe_ok=False`` is cached too."""
return _probe()
def refresh() -> HostCaps:
"""Clear the cache and re-probe. **TEST-ONLY** — nothing in the running app
calls this (host caps are immutable per process)."""
detect_host_caps.cache_clear()
return detect_host_caps()
def mlx_supported() -> tuple[bool, str]:
"""``(ok, reason)``. ``ok=True`` **only** on Apple Silicon
(``sys.platform == "darwin"`` and ``platform.machine() == "arm64"``) with
torch MPS available the shared gate for MLX-Audio / MLX-Whisper (#390).
Gates on exact-string equality (no regex no CodeQL surface). On any
non-Apple host it returns ``False`` **before** any package import, so a
stray ``mlx_*`` wheel on Linux/Windows never reports available.
"""
if sys.platform != "darwin" or _platform.machine() != "arm64":
if sys.platform == "darwin":
return (False, "MLX requires Apple Silicon; this Mac is Intel")
return (
False,
f"MLX requires Apple Silicon; this host is "
f"{sys.platform}/{_platform.machine()}",
)
try:
import torch
except Exception:
return (False, "torch not importable; cannot confirm MPS")
try:
if torch.backends.mps.is_available():
return (True, "")
except Exception:
# MPS query raised — fall through to the conservative unavailable path.
pass
return (
False,
"Apple Silicon detected but torch MPS unavailable; "
"reinstall torch with MPS support",
)
__all__ = [
"DeviceFamily",
"HostCaps",
"detect_host_caps",
"refresh",
"mlx_supported",
"KERNEL_RISK_MARKER",
"DIRECTML_MARKER",
]
+1 -47
View File
@@ -4,7 +4,7 @@ One pass over everything a working install needs: Python, compute device,
ffmpeg, HF token, disk, data-dir permissions, RAM, TTS engines, and (when
requested) network reachability of the HuggingFace hub. Surfaced two ways:
- ``GET /system/diagnose`` (Settings > About -> "Run self-check")
- ``GET /system/diagnose`` (Settings > About "Run self-check")
- ``python main.py --diagnose`` for headless installs / issue triage
Every ``detail``/``hint`` string is passed through ``core.scrub`` before it
@@ -194,49 +194,6 @@ def _check_engines() -> dict:
return _check("engines", "TTS engines", OK, detail)
def _check_gpu_routing() -> dict:
"""Routing verdict for the active TTS engine on THIS host (#21).
Surfaces a CPU fallback / unavailable-GPU *before* a slow or failed synth
the no-silent-fallback contract. `cpu_only` on a no-GPU machine is the
expected normal state and stays OK (never noise-warns)."""
try:
from services.tts_backend import gpu_routing_verdict
v = gpu_routing_verdict()
except Exception as e:
return _check("gpu_routing", "GPU routing", WARN, f"could not resolve: {e}")
status = v.get("routing_status")
engine = v.get("engine") or "active engine"
dev = v.get("effective_device") or "?"
reason = v.get("routing_reason")
host = v.get("host_family", "cpu")
if status == "accelerated":
if reason: # driver/arch caveat — accelerated but at risk
return _check("gpu_routing", "GPU routing", WARN,
f"{engine} -> {dev}: {reason}",
"The GPU is selected but may fail at kernel launch — "
"update drivers / reinstall torch for this GPU arch.")
return _check("gpu_routing", "GPU routing", OK, f"{engine} -> {dev} (accelerated)")
if status == "cpu_fallback":
return _check("gpu_routing", "GPU routing", WARN,
f"{engine} runs on CPU: {reason or 'no GPU path for this host'}",
"Pick an engine that supports this host's GPU for a big speedup, "
"or continue on CPU (slower).")
if status == "cpu_only":
return _check("gpu_routing", "GPU routing", OK,
f"{engine} -> cpu (no accelerator on this host)")
if status == "unavailable":
return _check("gpu_routing", "GPU routing", FAIL,
f"{engine} can't run on this host: {reason or f'needs a GPU; host is {host}'}",
"Select an engine with a CPU path in Settings -> Engines.")
# status == "none" / unknown — no active engine resolved.
return _check("gpu_routing", "GPU routing", WARN,
"No active TTS engine resolved for routing.",
"Pick an engine in Settings -> Engines.")
_DEEP_TIMEOUT_S = 180
@@ -307,8 +264,6 @@ def _check_network() -> dict:
# all model downloads need to get started. urllib honors HTTP(S)_PROXY.
import urllib.request
import urllib.error
if not _HUB_URL.startswith("https://"): # constant today; guard the sink anyway
raise ValueError(f"hub URL must be https, got {_HUB_URL!r}")
req = urllib.request.Request(_HUB_URL, method="HEAD")
try:
with urllib.request.urlopen(req, timeout=_HUB_TIMEOUT_S):
@@ -341,7 +296,6 @@ def run_diagnostics(include_network: bool = True, deep: bool = False) -> dict:
_check_data_dir(),
_check_ram(),
_check_engines(),
_check_gpu_routing(),
]
if include_network:
checks.append(_check_network())
+1 -8
View File
@@ -76,12 +76,6 @@ _CLASS_RULES: tuple[tuple[str, tuple[str, ...]], ...] = (
"connection refused",
"connection reset",
"connection aborted",
# transformers' download-failure wording ("We couldn't connect to
# '<endpoint>' to load the files") — the #874 mirror-down class was
# journaled as UNKNOWN without these.
"couldn't connect to",
"could not connect to",
"max retries exceeded",
"timed out",
"timeout",
"name or service not known",
@@ -117,8 +111,7 @@ def classify_exception(exc: BaseException, trace: str = "") -> str:
def _fingerprint(error_class: str, exc: BaseException) -> str:
import hashlib
raw = f"{error_class}|{type(exc).__name__}|{scrub_text(str(exc))[:200]}"
# Dedup key for the journal, not a security boundary.
return hashlib.sha1(raw.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:16]
return hashlib.sha1(raw.encode("utf-8", "replace")).hexdigest()[:16]
def _persist_locked() -> None:
+2 -201
View File
@@ -21,7 +21,6 @@ import re
import sys
from pathlib import Path
from typing import Any, Optional
from urllib.parse import urlsplit
from core import error_docs_map
from core.logging_filter import REDACTED, _HF_TOKEN_RE
@@ -33,141 +32,14 @@ _REDACTED_VALUE = "***REDACTED***"
# One-line "what to do" per docs-taxonomy key. Keys mirror error_docs_map's
# taxonomy; the docs URL itself stays owned by error_docs_map.
_HINTS: dict[str, str] = {
"PKG_RESOURCES_MISSING": "Run `uv pip install --reinstall 'setuptools>=75,<80'` in the backend venv (a plain install is skipped when setuptools' metadata is present but its pkg_resources files were removed by antivirus). Restart after.",
"PKG_RESOURCES_MISSING": "Install setuptools in the backend environment (provides pkg_resources).",
"GATEKEEPER_QUARANTINE": "Clear the macOS quarantine flag (xattr -cr the app), then reopen.",
"APPIMAGE_WEBKIT_WHITESCREEN": "Launch with WEBKIT_DISABLE_DMABUF_RENDERER=1 set.",
"HF_AUTH_FAILED": "Set a valid HF_TOKEN in Settings → Hugging Face and retry.",
"PYANNOTE_LICENSE_REQUIRED": "Accept the pyannote model licenses on Hugging Face, then retry.",
"COMPUTE_TYPE_UNSUPPORTED": "Your GPU doesn't support float16 — OmniVoice retried on int8. If transcription still fails, set OMNIVOICE/ASR_COMPUTE_TYPE=int8 or use CPU.",
"TRANSFORMERS_IMPORT": "Your transformers install is incomplete. Reinstall it (`uv pip install --reinstall transformers`) or switch ASR to faster-whisper (Settings → Models).",
"OS_INVALID_ARGUMENT": "The OS rejected a file operation (Errno 22 / invalid argument) — in the transcribe path this is the temporary WAV write before ASR. It's almost always the temp directory: missing, read-only, on a full or removed drive, or blocked by antivirus. Check that your system TEMP/TMP folder exists and is writable and the drive has free space (add an OmniVoice antivirus exclusion if you use one), then retry.",
"UNSUPPORTED_VIDEO_URL": "This link isn't a directly downloadable video. Paste a direct video page (e.g. a youtube.com/watch?v=… or douyin.com/video/<id> link), not a share/profile/feed link — or download the file and drop it in directly.",
"VIDEO_DOWNLOAD_NETWORK": "The connection to the video server dropped mid-download (often a transient CDN/network blip or a regional rate-limit). Just retry — OmniVoice already cleaned up the partial download. If it keeps failing, check your network/VPN.",
"BROKEN_VENV": "The Python backend environment was moved or damaged. OmniVoice rebuilds it automatically on the next launch; if it keeps failing, use Clean & Retry on the setup screen.",
# HF_MIRROR_UNREACHABLE has a DYNAMIC hint (it names the configured mirror)
# — see hf_mirror_hint(); build_failure special-cases it.
}
# ── HF mirror connectivity (#874) ────────────────────────────────────────────
# When a non-default HF_ENDPOINT (a mirror, e.g. hf-mirror.com — set via
# Settings → Models → Hugging Face mirror) is configured and a model
# download/load fails with a connectivity error, the raw transformers/hf_hub
# message ("We couldn't connect to 'https://hf-mirror.com' to load the files…")
# gives the user no next step. This is the single classifier for that class,
# shared by every surface: build_failure() (model status, dub/task events),
# the global 500 handler (main.py — covers /generate and every other route
# that can leak a model-load error), and the model-install SSE
# (setup/download.py).
_OFFICIAL_HF_ENDPOINTS = {"https://huggingface.co", "https://hf.co"}
# Connectivity signatures across the layers an HF download failure surfaces
# from: transformers' wording, huggingface_hub errors, requests/urllib3, and
# raw socket/DNS failures (Linux/macOS/Windows variants).
_HF_CONNECTIVITY_SIGNATURES = (
"couldn't connect to", # transformers: "We couldn't connect to '<endpoint>' …"
"could not connect to",
"connection error", # huggingface_hub / requests
"connection refused",
"connection reset",
"connection aborted",
"max retries exceeded", # urllib3 via requests
"failed to establish a new connection",
"name or service not known", # Linux DNS
"temporary failure in name resolution",
"nodename nor servname provided", # macOS DNS
"getaddrinfo failed", # Windows DNS
"timed out",
"an error happened while trying to locate the file on the hub", # LocalEntryNotFoundError
"we cannot find the requested files", # LocalEntryNotFoundError
)
# The failure must also be Hugging-Face-shaped — the configured endpoint/host
# named in the message, or HF-download wording — so a random socket error
# (e.g. a local LLM provider being down) doesn't get the mirror hint just
# because a mirror happens to be configured.
_HF_CONTEXT_MARKERS = (
"huggingface",
"hf_hub",
"hf-hub",
"load the files", # transformers
"cached files", # transformers
"the requested files", # LocalEntryNotFoundError
"locate the file on the hub",
"snapshot_download",
)
def configured_hf_mirror() -> str:
"""The non-default Hugging Face endpoint (mirror) in effect, or "".
Same resolution the download paths use: ``HF_ENDPOINT`` env (what
Settings Models Hugging Face mirror persists via user_env, and what
the HF libraries read) with the ``hf_endpoint`` pref as fallback
(mirrors setup/download.py's ``prefs.resolve``). Never raises.
"""
ep = (os.environ.get("HF_ENDPOINT") or "").strip()
if not ep:
try:
from core import prefs
ep = str(prefs.get("hf_endpoint", "") or "").strip()
except Exception:
ep = ""
ep = ep.rstrip("/")
if not ep or ep.lower() in _OFFICIAL_HF_ENDPOINTS:
return ""
return ep
def hf_mirror_hint(reason: Optional[str]) -> str:
"""Actionable hint when ``reason`` is an HF-download connectivity failure
and a non-default mirror endpoint is configured; "" otherwise.
The hint names the configured mirror, says it may be down, points at the
setting (Settings Models Hugging Face mirror), suggests the official
endpoint when the model isn't cached yet, and notes the restart
requirement (HF reads HF_ENDPOINT at import time see the hf-mirror
endpoints in api/routers/settings.py). Never raises.
"""
mirror = configured_hf_mirror()
if not mirror:
return ""
low = (reason or "").lower()
if not any(sig in low for sig in _HF_CONNECTIVITY_SIGNATURES):
return ""
try:
host = (urlsplit(mirror).netloc or "").lower()
except Exception:
host = ""
if not (
mirror.lower() in low
or (host and host in low)
or any(m in low for m in _HF_CONTEXT_MARKERS)
):
return ""
return (
f"Your Hugging Face mirror is set to {mirror}, which couldn't be "
"reached — the mirror may be down or blocked on your network. If the "
'model isn\'t in your local cache yet, switch to "Hugging Face '
'(official)" in Settings → Models → Hugging Face mirror (or wait for '
"the mirror to recover), then restart OmniVoice — the mirror setting "
"is applied when the app starts."
)
def append_hf_mirror_hint(text: str) -> str:
"""``"{text}{hint}"`` when the mirror-connectivity class applies;
``text`` unchanged otherwise. For surfaces that hand a raw error string to
the UI (the global 500 handler, the model-install SSE). Never raises."""
try:
hint = hf_mirror_hint(text)
except Exception:
return text
return f"{text}{hint}" if hint else text
def classify(reason: str) -> str:
"""Map a failure reason to a docs-taxonomy key, or "" when unknown.
@@ -183,78 +55,10 @@ def classify(reason: str) -> str:
return "APPIMAGE_WEBKIT_WHITESCREEN"
if "pyannote" in low or ("gated" in low and "model" in low) or "accept the" in low:
return "PYANNOTE_LICENSE_REQUIRED"
# ASR robustness (#551 / #549): name the class so the no-segments toast is
# actionable. Place before the generic returns so a compute-type/transformers
# failure gets its hint rather than falling through to "".
if "compute type" in low or "efficient float16" in low:
return "COMPUTE_TYPE_UNSUPPORTED"
# #763: a bare OS-level EINVAL ("[Errno 22] Invalid argument") while writing
# the per-chunk temp WAV for transcription (tempfile.NamedTemporaryFile /
# soundfile.write on the system temp dir) used to collapse into a dead-end
# "produced no segments. [Errno 22] Invalid argument" toast with no next
# step. errno 22 is EINVAL on every platform; in this path it's almost always
# a temp dir that's missing, read-only, on a full/removed drive, or blocked
# by antivirus. Name the class so build_failure attaches an actionable hint
# instead of a raw errno. Matching the errno (not the generic "invalid
# argument" wording) keeps this from mislabelling unrelated failures; the
# transformers "errno 2" rule below is unaffected — it also requires the
# transformers + site-packages markers, which this signature lacks.
if "errno 22" in low:
return "OS_INVALID_ARGUMENT"
if (
"could not import module" in low
or "autofeatureextractor" in low
# A corrupted/incomplete transformers install: a model load lazily
# resolves a module file that's MISSING from site-packages (an
# interrupted `uv sync`, antivirus removal, or a partial update), e.g.
# `[Errno 2] No such file or directory:
# '.../site-packages/transformers/models/qwen3/modeling_qwen3.py'`.
# That's a FileNotFoundError, not an ImportError, so the matches above
# miss it and the user got a useless "try restarting". Substring-match
# the package + the missing-file signal (separately, so it works on both
# POSIX `/` and Windows `\` paths).
or (
("no such file" in low or "errno 2" in low)
and "transformers" in low
and "site-packages" in low
)
):
return "TRANSFORMERS_IMPORT"
if ("huggingface" in low or "hf_token" in low or "401" in low or "unauthorized" in low) and (
"token" in low or "auth" in low or "401" in low or "unauthorized" in low
):
return "HF_AUTH_FAILED"
# #874: a model download that failed because the CONFIGURED HF mirror is
# unreachable. Env-aware by design — the class only exists when a
# non-default HF_ENDPOINT is configured. Checked BEFORE the video-download
# network class so a model download's "timed out"/"connection reset"
# names the mirror instead of the "video server".
if hf_mirror_hint(reason):
return "HF_MIRROR_UNREACHABLE"
# Video download (#554/#536): a non-downloadable URL shape vs a transient
# network drop — both previously surfaced as a bare yt-dlp string with no
# next step. UNSUPPORTED first (more specific) so "Unable to download video:
# Broken pipe" still classifies as a network blip.
if "unsupported url" in low or "no video formats" in low or "is not a valid url" in low:
return "UNSUPPORTED_VIDEO_URL"
if (
"broken pipe" in low
or "connection reset" in low
or "unable to download video" in low
or "remote end closed" in low
or "timed out" in low
):
return "VIDEO_DOWNLOAD_NETWORK"
# A relocated/corrupted venv whose interpreter can't bootstrap its stdlib —
# the Rust self-heal rebuilds it; this names the class for the toast.
if "no module named 'encodings'" in low:
return "BROKEN_VENV"
# #564: the interpreter starts fine but the backend can't import its OWN
# `omnivoice` package (a venv missing the editable install). Same self-heal
# class — Clean & Retry / the bootstrap repair rebuilds it. The trailing
# quote keeps a legitimately-named `omnivoice_*` helper from matching.
if "no module named 'omnivoice'" in low:
return "BROKEN_VENV"
return ""
@@ -350,15 +154,12 @@ def build_failure(
reason = sanitize(raw) or error_class
docs_topic = classify(raw)
# HF_MIRROR_UNREACHABLE's hint is dynamic (it names the configured mirror)
# so it can't live in the static _HINTS table.
hint = hf_mirror_hint(raw) if docs_topic == "HF_MIRROR_UNREACHABLE" else _HINTS.get(docs_topic, "")
fields: dict[str, Any] = {
"reason": reason,
"error": reason, # backward-compat mirror for older frontends
"error_class": error_class,
"stage": stage,
"hint": hint,
"hint": _HINTS.get(docs_topic, ""),
"docs_topic": docs_topic,
"docs_url": error_docs_map.ERROR_DOCS.get(docs_topic, ""),
"detail": sanitize(raw),
-77
View File
@@ -1,77 +0,0 @@
"""Resolve the project's own ``omnivoice`` package from source when the venv's
editable install is missing (#564).
``omnivoice`` is normally an editable install in the backend venv. An interrupted
or offline ``uv sync`` can install dependencies yet never lay the editable record
(``_editable_impl_omnivoice.pth``), or an antivirus quarantine can remove it
leaving a venv that starts uvicorn but cannot ``import omnivoice``, so it boots
fine and only fails at the first model call (``No module named 'omnivoice'``).
The desktop layout always copies ``omnivoice/`` next to ``backend/``, so we fall
back to importing it from there. The bootstrap now also gates on omnivoice being
importable (re-syncing to re-lay the editable install), but this keeps the
backend resilient even when that repair hasn't run yet.
"""
import os
import sys
def find_omnivoice_source_root(candidates):
"""Return the first candidate dir holding ``omnivoice/__init__.py``, else None."""
for root in candidates:
if root and os.path.isfile(os.path.join(root, "omnivoice", "__init__.py")):
return root
return None
def _candidate_roots(backend_dir):
"""Source roots to probe, most-specific first.
``OMNIVOICE_PROJECT_ROOT`` lets the launcher point at the staged project dir
explicitly; otherwise the desktop layout puts ``omnivoice/`` beside
``backend/`` (parent of ``backend_dir``).
"""
roots = []
env = os.environ.get("OMNIVOICE_PROJECT_ROOT")
if env:
roots.append(env)
roots.append(os.path.dirname(os.path.abspath(backend_dir)))
return roots
def _already_importable():
import importlib.util
try:
return importlib.util.find_spec("omnivoice") is not None
except (ImportError, ValueError):
# A half-laid spec (e.g. a stale .pth pointing at a deleted dir) raises
# rather than returning None — treat it as "not importable" so we fall
# back to the on-disk source.
return False
def ensure_omnivoice_importable(backend_dir, logger=None):
"""Make ``import omnivoice`` work, falling back to the sibling source tree.
No-op when the editable/site-packages install already resolves it. Otherwise
appends the first source root containing ``omnivoice/`` to ``sys.path``
(appended, never inserted, so a real install keeps precedence). Returns the
root that was added, or ``None`` if none was needed or found.
"""
if _already_importable():
return None
root = find_omnivoice_source_root(_candidate_roots(backend_dir))
if root and root not in sys.path:
sys.path.append(root)
if logger:
logger.warning(
"omnivoice not importable from the venv (missing/broken editable "
"install) — resolving it from source at %s (#564)", root,
)
elif logger and root is None:
logger.error(
"omnivoice is not importable and no source tree was found next to "
"%s — the install is incomplete; relaunch to let the bootstrap "
"repair the venv (#564)", backend_dir,
)
return root
+2 -8
View File
@@ -61,15 +61,9 @@ def seed_sample_project():
if count > 0:
return # Not first run — skip
# The demo clip is committed at backend/assets/samples/demo_voice.wav and
# bundled with the app (#621). If it's somehow absent (e.g. a partial
# checkout), skip the seed gracefully rather than seeding a profile that
# points at a missing file — run scripts/build_demos.sh to regenerate it.
# Check if demo audio exists
if not os.path.isfile(_DEMO_AUDIO):
logger.warning(
"Demo audio not found at %s — skipping onboarding seed "
"(regenerate with scripts/build_demos.sh)", _DEMO_AUDIO,
)
logger.warning("Demo audio not found at %s — skipping onboarding seed", _DEMO_AUDIO)
return
# Copy demo audio to voices directory
+6 -57
View File
@@ -36,39 +36,15 @@ _TOKEN_PATTERNS = (
re.compile(r"github_pat_[A-Za-z0-9_]{20,}"), # GitHub fine-grained PAT
re.compile(r"gh[pousr]_[A-Za-z0-9]{30,}"), # GitHub classic tokens
re.compile(r"sk-[A-Za-z0-9_\-]{20,}"), # OpenAI-style API keys
# A backend error can carry a secret from *any* provider (the LLM-providers
# feature ships a dozen), so match the common credential shapes too, not
# just the four vendors above — a leaked key in a public issue is real harm.
re.compile(r"eyJ[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{6,}"), # JWT (Bearer)
re.compile(r"AIza[0-9A-Za-z_\-]{35}"), # Google API key
re.compile(r"xox[baprs]-[A-Za-z0-9\-]{10,}"), # Slack token
re.compile(r"AKIA[0-9A-Z]{16}"), # AWS access key id
re.compile(r"(?i)bearer\s+[A-Za-z0-9._\-]{16,}"), # opaque bearer tokens
)
# Secrets carried in a URL query string (`?token=…`, `&api_key=…`). Redact the
# VALUE while keeping the param name + separator so the URL stays legible. Bare
# `key=` is intentionally excluded — too common in non-secret text; shaped keys
# are already caught above and named env vars by the sweep below.
_URL_SECRET_RE = re.compile(
r"((?:access[_-]?token|api[_-]?key|apikey|auth[_-]?token|token|secret|password|passwd|pwd)=)"
r"([^&\s\"'#]{6,})",
re.IGNORECASE,
)
# Home-directory shapes for all three supported platforms. Matched
# pattern-wise (not just this machine's $HOME) so paths quoted from a
# user's pasted log on another OS get cleaned too.
# IGNORECASE because Windows is case-insensitive and tools routinely emit the
# lowercase `c:\users\<name>` form, which the CLAUDE.md redaction spec still
# requires to become `~`. `Users`/`users`, `Home`/`home` all match.
_HOME_PATTERNS = (
# Windows-with-forward-slashes must run BEFORE the bare macOS shape, or
# `/Users/<name>` inside `C:/Users/<name>` gets eaten first, leaving `C:~`.
re.compile(r"[A-Za-z]:/Users/[^/\s\"']+", re.IGNORECASE), # Windows, forward slashes
re.compile(r"/Users/[^/\s\"']+", re.IGNORECASE), # macOS
re.compile(r"/home/[^/\s\"']+", re.IGNORECASE), # Linux
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+", re.IGNORECASE), # Windows, backslashes
re.compile(r"/Users/[^/\s\"']+"), # macOS
re.compile(r"/home/[^/\s\"']+"), # Linux
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+"), # Windows
)
# Values shorter than this are too entropy-poor to be real secrets and too
@@ -106,25 +82,19 @@ def scrub_text(text: str | None) -> str:
except Exception:
pass
# 2. Credential-shaped substrings + URL query secrets.
# 2. Credential-shaped substrings.
for pat in _TOKEN_PATTERNS:
try:
s = pat.sub(REDACTED, s)
except Exception:
pass
try:
s = _URL_SECRET_RE.sub(lambda m: m.group(1) + REDACTED, s)
except Exception:
pass
# 3. This process's real home dir (covers symlinked/nonstandard homes
# the generic patterns miss), then the per-OS shapes. Boundary-aware so
# a home of `/Users/john` doesn't rewrite `/Users/johnny` to `~ny`
# (leaking the fragment + mangling the path).
# the generic patterns miss), then the per-OS shapes.
try:
home = os.path.expanduser("~")
if home and home not in ("/", "~"):
s = re.sub(re.escape(home) + r"(?=[/\\\s\"']|$)", "~", s)
s = s.replace(home, "~")
except Exception:
pass
for pat in _HOME_PATTERNS:
@@ -134,24 +104,3 @@ def scrub_text(text: str | None) -> str:
pass
return s
def scrub_provider_error(detail: object, api_key: str | None = None) -> str:
"""UI-safe text for an LLM/translation provider failure.
Some OpenAI-compatible providers echo the caller's key or a stable
``user_id`` back inside their error bodies, and a raw ``str(exc)`` on the
translate / glossary paths would surface that verbatim. This redacts the
exact resolved ``api_key`` first (in the provider-registry case it isn't a
shaped/known-env secret, so ``scrub_text`` alone can miss it) then runs the
generic secret + home-path scrub. Never raises scrubbing must not mask a
failure with a new one. Mirrors ``settings._scrub_llm_detail`` so every
surface redacts identically.
"""
s = str(detail if detail is not None else "")
try:
if api_key and api_key != "local" and len(api_key) >= _MIN_SECRET_LEN:
s = s.replace(api_key, REDACTED)
except Exception:
pass
return scrub_text(s)
-25
View File
@@ -82,28 +82,3 @@ def unset_user_env(key: str, path: Optional[str] = None) -> None:
prefix = f"{key}="
lines = [ln for ln in _read_lines(path) if not ln.startswith(prefix)]
_write_lines(path, lines)
def load_into_environ(path: Optional[str] = None) -> bool:
"""Load the durable per-user env file into ``os.environ``, **overriding**
any value a launcher already injected. Returns True if a file was loaded.
This file is the in-app Settings source of truth. The desktop launcher
(Tauri) injects defaults like ``OMNIVOICE_CACHE_DIR`` (and ``HF_ENDPOINT``)
from its *own* config into the backend's environment *before* startup, so
loading this file with ``override=False`` meant a models directory the user
changed in Settings was silently ignored on every launch the effective
location stayed on the old one no matter how many restarts (#480). Both keys
this file can hold are the user's explicit Settings choice and should beat
the launcher's default, so we override. Restores this file's documented
"values written here take effect on the next backend launch" contract.
"""
path = path or os.environ.get("OMNIVOICE_ENV_FILE") or USER_ENV_PATH
if not os.path.isfile(path):
return False
try:
import dotenv
except ImportError:
return False
dotenv.load_dotenv(path, override=True)
return True
+4 -33
View File
@@ -2,43 +2,14 @@
Read from the installed package metadata (driven by ``pyproject.toml``) so the
FastAPI/API version and exported-bundle metadata never drift to a stale literal
the prior "0.4.0" / "0.2.7" bug, and the v0.3.6 desktop build that reported
"0.3.5" because the *frozen* backend couldn't read its own metadata.
Resolution order:
1. installed package metadata correct in any ``uv sync``'d env and, thanks
to ``copy_metadata('omnivoice')`` in ``backend.spec``, in the frozen build;
2. ``pyproject.toml`` walked up from this file correct for a raw source
checkout that was never installed;
3. ``_FALLBACK_VERSION`` a last resort, kept in lockstep with the four
version files by ``tests/test_app_version.py`` so it can never silently
drift again.
again (the prior "0.4.0" / "0.2.7" bug). Falls back to a literal only when
running from a raw source checkout that was never ``uv sync``'d.
"""
from __future__ import annotations
import re
from importlib.metadata import PackageNotFoundError, version
from pathlib import Path
# Last-resort literal. Guarded by
# tests/test_app_version.py::test_all_version_files_in_lockstep and bumped by
# release.yml's version-bump job, so it stays equal to
# pyproject/tauri.conf/Cargo/package.json.
_FALLBACK_VERSION = "0.3.9"
def _fallback_version() -> str:
"""Version for contexts where package metadata is unavailable."""
for parent in Path(__file__).resolve().parents:
pyproject = parent / "pyproject.toml"
if pyproject.is_file():
match = re.search(r'(?m)^version\s*=\s*"([^"]+)"', pyproject.read_text())
if match:
return match.group(1)
return _FALLBACK_VERSION
try:
APP_VERSION = version("omnivoice")
except PackageNotFoundError: # frozen build w/o metadata, or non-installed checkout
APP_VERSION = _fallback_version()
except PackageNotFoundError: # non-installed source checkout
APP_VERSION = "0.3.5"
-148
View File
@@ -1,148 +0,0 @@
"""Crash-isolated faster-whisper ASR sidecar (Wave 4.2 / Spec 7).
Runs faster-whisper in a child process so a CTranslate2 GPU-teardown segfault
becomes a failed job, not a dead backend. Speaks the SubprocessBackend wire
protocol (length-prefixed JSON over stdin/stdout):
on start {"op":"ready","engine":"faster-whisper-isolated"}
{"op":"ping"} {"op":"pong"}
{"op":"transcribe","audio_path":...,"word_timestamps":bool}
{"op":"segments","result":{"segments":[...],"language":...}}
{"op":"shutdown"} exit 0
error {"op":"error","message":...}
Runs under the PARENT venv (faster-whisper is already a dependency) only the
process boundary is new. torch/CTranslate2 import lazily inside transcribe so
the ready handshake fits the spawn timeout.
"""
from __future__ import annotations
import json
import os
import struct
import sys
import traceback
MAX_FRAME_BYTES = 64 * 1024 * 1024
_model = None
def _send(stream, obj):
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
stream.write(struct.pack("!I", len(body)))
stream.write(body)
stream.flush()
def _recv(stream):
header = stream.read(4)
if len(header) < 4:
return None
(n,) = struct.unpack("!I", header)
if n > MAX_FRAME_BYTES:
raise IOError(f"frame too large: {n}")
body = bytearray()
while len(body) < n:
chunk = stream.read(n - len(body))
if not chunk:
raise IOError("short read")
body.extend(chunk)
return json.loads(bytes(body).decode("utf-8"))
# NOTE: keep this compute_type fallback in lockstep with
# services/asr_backend.py:_compute_type_candidates / _is_compute_type_error.
# This sidecar runs in a child proc with a clean import path, so we duplicate a
# tiny copy rather than cross-importing the heavy services package (#551).
def _ct_candidates(device):
override = os.environ.get("ASR_COMPUTE_TYPE")
if override:
return [override]
return ["float16", "int8_float16", "int8"] if device == "cuda" else ["int8", "float32"]
def _is_ct_error(msg):
low = msg.lower()
return "compute type" in low or "efficient float16" in low
def _get_model():
global _model
if _model is None:
from faster_whisper import WhisperModel
name = os.environ.get("ASR_MODEL_FW", "large-v3")
try:
import torch
device = "cuda" if torch.cuda.is_available() else "cpu"
except Exception:
device = "cpu"
# Degrade fp16 → int8 rather than crash on GPUs without efficient fp16
# (older Maxwell/Pascal, GTX 16xx, CTranslate2/cuDNN mismatch) (#551).
last_err = None
for compute in _ct_candidates(device):
try:
_model = WhisperModel(name, device=device, compute_type=compute)
break
except (ValueError, RuntimeError) as e:
last_err = e
if _is_ct_error(str(e)):
continue
raise
else:
raise last_err
return _model
def _transcribe(audio_path, word_timestamps):
model = _get_model()
segments, info = model.transcribe(audio_path, word_timestamps=word_timestamps)
out = []
for s in segments:
seg = {"start": float(s.start), "end": float(s.end), "text": s.text}
if word_timestamps and getattr(s, "words", None):
seg["words"] = [
{"word": w.word, "start": float(w.start), "end": float(w.end),
"probability": float(getattr(w, "probability", 0.0))}
for w in s.words
]
out.append(seg)
return {
"segments": out,
"text": " ".join(s["text"].strip() for s in out).strip(),
"language": getattr(info, "language", "unknown"),
}
def main() -> int:
stdin, stdout = sys.stdin.buffer, sys.stdout.buffer
_send(stdout, {"op": "ready", "engine": "faster-whisper-isolated"})
while True:
try:
msg = _recv(stdin)
except Exception as exc:
_send(stdout, {"op": "error", "stage": "recv", "message": f"{type(exc).__name__}: {exc}"})
return 1
if msg is None:
return 0
op = msg.get("op")
try:
if op == "ping":
_send(stdout, {"op": "pong"})
elif op == "transcribe":
result = _transcribe(msg.get("audio_path"), bool(msg.get("word_timestamps", True)))
_send(stdout, {"op": "segments", "result": result})
elif op == "shutdown":
return 0
else:
_send(stdout, {"op": "error", "stage": "dispatch", "message": f"unknown op: {op!r}"})
except Exception as exc:
_send(stdout, {
"op": "error", "stage": "handler",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
return 0
if __name__ == "__main__":
sys.exit(main())
-19
View File
@@ -101,12 +101,6 @@ def main() -> int:
return 0
op = msg.get("op")
# Wave 4.2: deterministic "crash mid-transcription" hook — exit BEFORE
# sending any reply so the parent's blocking recv sees a dead pipe
# (reply=None). The crash-after-one hook below replies first, so it
# can't deterministically exercise the no-reply path.
if op == "transcribe" and os.environ.get("OMNIVOICE_ECHO_CRASH_NO_REPLY") == "1":
os._exit(1)
try:
if op == "ping":
_send(stdout, {"op": "pong"})
@@ -119,19 +113,6 @@ def main() -> int:
"sample_rate": sr,
"n_samples": n_samples,
})
elif op == "transcribe":
# Wave 4.2: echo ASR op — a canned segments result so the
# SubprocessASRBackend round-trip + respawn path is testable
# without a real ASR engine.
_send(stdout, {
"op": "segments",
"result": {
"segments": [{"start": 0.0, "end": 1.0,
"text": f"echo:{msg.get('audio_path', '')}"}],
"text": f"echo:{msg.get('audio_path', '')}",
"language": "en",
},
})
elif op == "shutdown":
return 0
elif op == "probe_env" and test_mode:
-134
View File
@@ -1,134 +0,0 @@
"""Confucius4-TTS sidecar package (issue #590).
Confucius4-TTS (netease-youdao) is an LLM-based multilingual / cross-lingual
zero-shot voice-cloning TTS: 14 languages, **no reference transcript required**,
cross-lingual voice transfer, Apache-2.0 (https://github.com/netease-youdao/Confucius4-TTS).
Like IndexTTS / MOSS-TTS-v1.5 / dots.tts it runs in its **own subprocess venv**
(upstream: Python 3.10 + CUDA 12.6 + its own deps), isolated from the OmniVoice
parent. It is **opt-in** selected in the engine picker and enabled only when
the user points ``OMNIVOICE_CONFUCIUS4_TTS_DIR`` at a clone so it can never
become a broken default on any platform (the strict default-parity rule).
Status (#590): **validated end-to-end** (2026-07-02, Apple Silicon, CPU) — the
synthesis API (``confuciustts.cli.inference.ConfuciusTTS``
``.generate(text, lang, prompt_wav)`` tensor, ``model.sample_rate``) produced
audible speech at 22 050 Hz; the sidecar's pure logic is unit-tested
(``tests/test_confucius4_sidecar.py``). CPU inference is slow (~17× realtime),
so CUDA is the recommended path. Gated off by default, so this affects no one
until they opt in.
Three entry points: ``Confucius4Backend`` (this module), ``main.py`` (the sidecar,
runs under the Confucius4 venv never imported by the parent), and
``bootstrap.py`` (venv probe + lazy bootstrap).
"""
from __future__ import annotations
import logging
from typing import TYPE_CHECKING
from services.subprocess_backend import SubprocessBackend
if TYPE_CHECKING:
import torch # noqa: F401
logger = logging.getLogger("omnivoice.confucius4")
class Confucius4Backend(SubprocessBackend):
"""Confucius4-TTS (netease-youdao) — LLM-based, 14 langs, zero-shot clone.
Runs in a long-lived sidecar over length-prefixed JSON-over-stdio in a
dedicated venv. First synthesize cold-loads the checkpoint; subsequent calls
reuse the process.
Installation::
git clone https://github.com/netease-youdao/Confucius4-TTS.git
cd Confucius4-TTS
uv venv --python 3.10 && uv pip install -r requirements.txt
(Upstream ships no pyproject.toml/setup.py, so there is nothing to
``pip install -e`` the sidecar sys.path-inserts the clone instead.)
Then set ``OMNIVOICE_CONFUCIUS4_TTS_DIR`` to the clone root and restart.
License: Apache-2.0. CUDA recommended; CPU validated but ~17× realtime.
"""
id = "confucius4-tts"
display_name = (
"Confucius4-TTS (LLM, 14 langs, cross-lingual zero-shot clone, CUDA/CPU, Apache-2.0)"
)
supports_voice_design = False # timbre comes from a reference clip
# Upstream vocoder rate (config target_sample_rate) — confirmed 22 050 Hz by
# a live run (2026-07-02); still re-read from the sidecar's ready/audio frames.
_DEFAULT_SAMPLE_RATE = 22050
# CUDA fast path + CPU fallback, both exercised (CPU end-to-end validated).
# No MPS claim — upstream has no Metal path.
gpu_compat = ("cuda", "cpu")
@classmethod
def is_available(cls) -> tuple[bool, str]:
# Verify the venv on disk only — do NOT import the engine here (separate
# interpreter). A real health-check runs on the user's "Test engine"
# action in Settings.
from engines.confucius4.bootstrap import (
CONFUCIUS4_SIDECAR_SCRIPT,
is_confucius4_installed,
)
if not is_confucius4_installed():
return False, (
"Confucius4-TTS venv not found. Set OMNIVOICE_CONFUCIUS4_TTS_DIR "
"to your Confucius4-TTS clone (the directory containing "
"requirements.txt) and restart OmniVoice. CUDA GPU recommended "
"(CPU works but is slow). See docs/engines/confucius4-tts.md."
)
if not CONFUCIUS4_SIDECAR_SCRIPT.exists():
return False, (
"Confucius4-TTS sidecar script missing at "
f"{CONFUCIUS4_SIDECAR_SCRIPT} — reinstall OmniVoice."
)
return True, "ok"
@classmethod
def venv_python(cls):
from engines.confucius4.bootstrap import resolve_confucius4_venv
return resolve_confucius4_venv()
@classmethod
def sidecar_script(cls):
from engines.confucius4.bootstrap import CONFUCIUS4_SIDECAR_SCRIPT
return CONFUCIUS4_SIDECAR_SCRIPT
@property
def sample_rate(self) -> int:
return self._DEFAULT_SAMPLE_RATE
@property
def supported_languages(self) -> list[str]:
# 14 languages with the caller's language passed through at synthesize
# time; "multi" on the protocol surface.
return ["multi"]
def generate(self, text: str, **kw) -> "torch.Tensor":
"""Synthesize one utterance through the Confucius4 sidecar.
kwargs honored:
* ``ref_audio`` reference clip path ``prompt_wav`` (zero-shot
cloning). Optional but recommended for a specific voice.
* ``language`` ISO code / name ``lang`` (cross-lingual transfer).
* ``ref_text`` is intentionally ignored Confucius4 is unconstrained
cloning (no reference transcript needed).
Returns a tensor of shape (1, n_samples) at :attr:`sample_rate`.
"""
forwarded: dict = {}
ref_audio = kw.get("ref_audio")
if ref_audio:
forwarded["ref_audio"] = ref_audio
language = kw.get("language")
if language:
forwarded["language"] = str(language)
return super().generate(text, **forwarded)
__all__ = ["Confucius4Backend"]
-217
View File
@@ -1,217 +0,0 @@
"""Confucius4-TTS venv probe + lazy bootstrap (issue #590).
Confucius4-TTS (netease-youdao) is an LLM-based multilingual zero-shot cloning
TTS 14 languages, no reference transcript required, Apache-2.0. Like the other
heavyweight opt-in engines (IndexTTS / MOSS-TTS-v1.5 / dots.tts) it runs in its
**own subprocess venv**: upstream targets Python 3.10 + CUDA 12.6 with its own
dependency set, which we keep off the parent interpreter.
Probe order (existing power-user installs win zero migration):
1. ``${OMNIVOICE_CONFUCIUS4_TTS_DIR}/.venv/`` the user's clone-level venv.
2. ``backend/engines/confucius4/.venv/`` this package's own venv.
3. Bootstrap: ``uv venv`` then ``uv pip install -r <clone>/requirements.txt``
(+ ``uv pip install -e <clone>`` only if upstream ever ships packaging).
Validated end-to-end 2026-07-02 (Apple Silicon, CPU): upstream ships **no
pyproject.toml/setup.py**, so ``confuciustts`` is importable only with the
clone root on ``sys.path`` the import probe and the sidecar both handle
that. The engine is opt-in (env-dir gated) and never touched unless
``OMNIVOICE_CONFUCIUS4_TTS_DIR`` is set, so this can't affect the default
install on any platform.
"""
from __future__ import annotations
import logging
import os
import shutil
import subprocess
import sys
from pathlib import Path
from typing import Optional
logger = logging.getLogger("omnivoice.confucius4.bootstrap")
#: Absolute path to the sidecar entrypoint.
CONFUCIUS4_SIDECAR_SCRIPT: Path = Path(__file__).parent / "main.py"
#: This package's owned venv (Probe 2).
_ENGINES_VENV_DIR: Path = Path(__file__).parent / ".venv"
#: Env var pointing at the user's Confucius4-TTS clone root.
_CLONE_DIR_ENV: str = "OMNIVOICE_CONFUCIUS4_TTS_DIR"
#: The package importable from the clone (verify against upstream).
_IMPORT_PROBE = "confuciustts"
_resolved_python: Optional[Path] = None
_IMPORT_PROBE_TIMEOUT_S = 15
_UV_VENV_TIMEOUT_S = 120
_UV_PIP_INSTALL_TIMEOUT_S = 1800
def invalidate() -> None:
"""Clear the resolved-python cache. Tests call this between scenarios."""
global _resolved_python
_resolved_python = None
def is_confucius4_installed() -> bool:
"""Cheap file-existence check for a usable venv (no subprocess spawn)."""
return any(cand.is_file() for cand in _probe_paths())
def resolve_confucius4_venv() -> Path:
"""Resolve the sidecar's Python interpreter (probe order in the docstring).
Memoised. Raises :exc:`RuntimeError` if none can be located and bootstrap
is unavailable."""
global _resolved_python
if _resolved_python is not None:
return _resolved_python
clone_dir = os.environ.get(_CLONE_DIR_ENV)
if clone_dir:
cand = _venv_python_path(Path(clone_dir) / ".venv")
if cand.is_file() and _venv_can_import(cand):
logger.info("Confucius4 venv resolved from %s: %s", _CLONE_DIR_ENV, cand)
_resolved_python = cand
return cand
cand = _venv_python_path(_ENGINES_VENV_DIR)
if cand.is_file() and _venv_can_import(cand):
logger.info("Confucius4 venv resolved from engines path: %s", cand)
_resolved_python = cand
return cand
if not clone_dir:
raise RuntimeError(
"Confucius4-TTS is not installed. Set the "
f"{_CLONE_DIR_ENV} environment variable to your Confucius4-TTS clone "
"(the directory that contains requirements.txt), then restart "
"OmniVoice. See docs/engines/confucius4-tts.md."
)
cand = _bootstrap_engines_venv(Path(clone_dir))
_resolved_python = cand
return cand
def _venv_python_path(venv_dir: Path) -> Path:
if sys.platform == "win32":
return venv_dir / "Scripts" / "python.exe"
return venv_dir / "bin" / "python"
def _probe_paths() -> list[Path]:
out: list[Path] = []
clone_dir = os.environ.get(_CLONE_DIR_ENV)
if clone_dir:
out.append(_venv_python_path(Path(clone_dir) / ".venv"))
out.append(_venv_python_path(_ENGINES_VENV_DIR))
return out
def _import_probe_code() -> str:
"""Probe snippet mirroring the sidecar's import semantics: upstream is not
pip-installable, so ``confuciustts`` resolves via the clone on sys.path."""
clone = os.environ.get(_CLONE_DIR_ENV, "")
if clone:
return f"import sys; sys.path.insert(0, {clone!r}); import {_IMPORT_PROBE}"
return f"import {_IMPORT_PROBE}"
def _venv_can_import(python_path: Path) -> bool:
"""Spawn the candidate python and verify ``import confuciustts`` works."""
try:
proc = subprocess.run(
[str(python_path), "-c", _import_probe_code()],
capture_output=True, timeout=_IMPORT_PROBE_TIMEOUT_S,
)
except (subprocess.TimeoutExpired, OSError) as exc:
logger.debug("Confucius4 import probe failed for %s: %s", python_path, exc)
return False
if proc.returncode != 0:
logger.debug(
"Confucius4 import probe non-zero for %s: %s",
python_path, proc.stderr.decode("utf-8", errors="replace")[:200],
)
return False
return True
def _locate_uv() -> Optional[str]:
bundled = os.environ.get("OMNIVOICE_BUNDLED_UV")
if bundled and Path(bundled).is_file():
return bundled
return shutil.which("uv")
def _bootstrap_engines_venv(clone_dir: Path) -> Path:
"""Create engines/confucius4/.venv and install the user's clone."""
uv = _locate_uv()
if not uv:
raise RuntimeError(
"uv is required to bootstrap the Confucius4-TTS venv but was not "
"found on PATH (and OMNIVOICE_BUNDLED_UV was not set). Install uv "
"from https://docs.astral.sh/uv/ and re-launch OmniVoice."
)
logger.info(
"Bootstrapping Confucius4 venv at %s from %s (several minutes on first "
"launch)", _ENGINES_VENV_DIR, clone_dir,
)
try:
subprocess.run(
[uv, "venv", "--python", "3.10", str(_ENGINES_VENV_DIR)],
check=True, timeout=_UV_VENV_TIMEOUT_S, capture_output=True,
)
except subprocess.CalledProcessError as exc:
raise RuntimeError(
f"uv venv failed for Confucius4 bootstrap at {_ENGINES_VENV_DIR}: "
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
) from exc
python_path = _venv_python_path(_ENGINES_VENV_DIR)
requirements = clone_dir / "requirements.txt"
try:
if requirements.is_file():
subprocess.run(
[uv, "pip", "install", "--python", str(python_path),
"-r", str(requirements)],
check=True, timeout=_UV_PIP_INSTALL_TIMEOUT_S, capture_output=True,
)
# Editable install only if upstream ever ships packaging metadata —
# as of 2026-07 there is none, and `uv pip install -e` on a bare clone
# fails outright. Import resolution is handled via sys.path instead.
if (clone_dir / "pyproject.toml").is_file() or (clone_dir / "setup.py").is_file():
subprocess.run(
[uv, "pip", "install", "--python", str(python_path), "-e", str(clone_dir)],
check=True, timeout=_UV_PIP_INSTALL_TIMEOUT_S, capture_output=True,
)
except subprocess.CalledProcessError as exc:
raise RuntimeError(
"uv pip install failed during Confucius4 bootstrap "
f"({clone_dir}): "
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}. "
"See docs/engines/confucius4-tts.md."
) from exc
if not _venv_can_import(python_path):
raise RuntimeError(
f"Confucius4 bootstrap completed but `import {_IMPORT_PROBE}` still "
f"fails from {python_path}. Verify {clone_dir} is a valid clone. "
"See docs/engines/confucius4-tts.md."
)
logger.info("Confucius4 venv bootstrap successful: %s", python_path)
return python_path
__all__ = [
"CONFUCIUS4_SIDECAR_SCRIPT",
"invalidate",
"is_confucius4_installed",
"resolve_confucius4_venv",
]
-216
View File
@@ -1,216 +0,0 @@
"""Confucius4-TTS sidecar entry point (issue #590).
Runs inside ``engines/confucius4/.venv`` (or the user's
``${OMNIVOICE_CONFUCIUS4_TTS_DIR}/.venv``), isolated from the OmniVoice parent.
Same isolation rationale as the IndexTTS / MOSS-TTS-v1.5 / dots.tts sidecars.
Stdlib-only at import time; ``confuciustts`` + torch are imported lazily on the
first synthesize op so the ``ready`` frame fits inside the parent's 30 s spawn
handshake.
Wire protocol length-prefixed JSON over stdin/stdout, byte-identical to
``backend/services/subprocess_backend.py``::
[ 4-byte big-endian uint32 length ][ N bytes UTF-8 JSON ]
Op flow: ready ping/pong synthesize ( progress, audio) shutdown.
Status (#590): the model API below
(``confuciustts.cli.inference.ConfuciusTTS(config_path=, device=)`` and
``model.generate(text=, lang=, prompt_wav=)`` audio tensor, ``model.sample_rate``)
is **validated end-to-end** (2026-07-02, Apple Silicon, CPU): live generate()
produced audible speech at 22 050 Hz. This sidecar's pure logic is unit-tested
in ``tests/test_confucius4_sidecar.py``. Opt-in, so it affects no one until
enabled.
Restrictions: NO imports from OmniVoice parent code. NO logging of os.environ.
"""
from __future__ import annotations
import base64
import json
import os
import struct
import sys
import traceback
MAX_FRAME_BYTES = 64 * 1024 * 1024
#: Upstream BigVGAN vocoder rate — ``target_sample_rate: 22050`` in
#: ``config/inference_config.yaml``, confirmed by a live end-to-end run
#: (2026-07-02). The real value is still re-read from ``model.sample_rate``
#: on each generate() so a future upstream change can't corrupt audio.
CONFUCIUS_SAMPLE_RATE = 22050
def _send(stream, obj: dict) -> None:
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
stream.write(struct.pack("!I", len(body)))
stream.write(body)
stream.flush()
def _recv(stream):
header = stream.read(4)
if len(header) < 4:
return None # EOF
(n,) = struct.unpack("!I", header)
if n > MAX_FRAME_BYTES:
raise IOError(f"frame too large: {n}")
body = bytearray()
while len(body) < n:
chunk = stream.read(n - len(body))
if not chunk:
raise IOError("short read")
body.extend(chunk)
return json.loads(bytes(body).decode("utf-8"))
def _measure_vram_mb() -> float:
try:
import torch
if torch.cuda.is_available():
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
except Exception:
pass
return 0.0
_model = None
def _config_path() -> str:
"""Locate Confucius4's inference config (``config/inference_config.yaml``)
under the clone, or an explicit override."""
explicit = os.environ.get("OMNIVOICE_CONFUCIUS4_CONFIG")
if explicit:
return explicit
clone = os.environ.get("OMNIVOICE_CONFUCIUS4_TTS_DIR", "")
return os.path.join(clone, "config", "inference_config.yaml")
def _ensure_clone_on_sys_path() -> None:
"""Make ``import confuciustts`` resolve from the user's clone.
Upstream Confucius4-TTS is **not pip-installable** (no pyproject.toml /
setup.py as of 2026-07); its own ``example.py`` sys.path-inserts the repo
root instead. Mirror that here so the sidecar works from a plain
``uv pip install -r requirements.txt`` venv. Inserted at position 0 so the
clone the user pointed at always wins over any stale installed copy.
"""
clone = os.environ.get("OMNIVOICE_CONFUCIUS4_TTS_DIR", "")
if clone and clone not in sys.path:
sys.path.insert(0, clone)
def _load_model(stdout):
"""Cold-construct the Confucius4 model (CUDA, else CPU — both validated)."""
global _model
if _model is not None:
return _model
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 0})
_ensure_clone_on_sys_path()
import torch
from confuciustts.cli.inference import ConfuciusTTS # type: ignore[import-not-found]
device = "cuda" if torch.cuda.is_available() else "cpu"
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 50})
_model = ConfuciusTTS(config_path=_config_path(), device=device)
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 100})
return _model
def _tensor_to_pcm_b64(audio, sample_rate: int) -> tuple[str, int, int]:
import numpy as np
arr = audio.detach().to("cpu").float().numpy() if hasattr(audio, "detach") else np.asarray(audio)
arr = np.asarray(arr, dtype=np.float32).squeeze()
if arr.ndim > 1:
arr = arr.mean(axis=0)
arr = np.clip(arr, -1.0, 1.0)
pcm = (arr * 32767.0).astype(np.int16).tobytes()
return base64.b64encode(pcm).decode("ascii"), int(sample_rate), int(arr.shape[0])
def _normalize_language(raw):
"""Confucius4 expects an ISO-ish language code (e.g. 'en', 'zh'). Empty /
'auto' 'en' as a safe default (the API requires a lang)."""
if not raw or not isinstance(raw, str):
return "en"
s = raw.strip().lower()
if not s or s == "auto":
return "en"
return s[:2] if (len(s) >= 2 and s[:2].isalpha()) else s
def _handle_synthesize(msg: dict, stdout) -> None:
text = msg.get("text")
if not text or not isinstance(text, str):
raise ValueError("synthesize: missing or non-string 'text'")
model = _load_model(stdout)
gen_kwargs: dict = {"text": text, "lang": _normalize_language(msg.get("language"))}
ref_audio = msg.get("ref_audio")
if ref_audio:
gen_kwargs["prompt_wav"] = ref_audio
audio = model.generate(**gen_kwargs)
sample_rate = int(getattr(model, "sample_rate", CONFUCIUS_SAMPLE_RATE))
pcm_b64, sr, n_samples = _tensor_to_pcm_b64(audio, sample_rate)
_send(stdout, {
"op": "audio",
"audio_pcm_b64": pcm_b64,
"sample_rate": sr,
"n_samples": n_samples,
})
def main() -> int:
stdin = sys.stdin.buffer
stdout = sys.stdout.buffer
_send(stdout, {
"op": "ready",
"engine": "confucius4-tts",
"sample_rate": CONFUCIUS_SAMPLE_RATE,
})
while True:
try:
msg = _recv(stdin)
except Exception as exc:
_send(stdout, {
"op": "error", "stage": "recv",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
return 1
if msg is None:
return 0
op = msg.get("op") if isinstance(msg, dict) else None
try:
if op == "ping":
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
elif op == "synthesize":
_handle_synthesize(msg, stdout)
elif op == "shutdown":
return 0
else:
_send(stdout, {"op": "error", "stage": "dispatch",
"message": f"unknown op: {op!r}"})
except Exception as exc:
_send(stdout, {
"op": "error", "stage": op or "unknown",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
if __name__ == "__main__":
sys.exit(main())
-187
View File
@@ -1,187 +0,0 @@
"""dots.tts sidecar package (issue #498).
dots.tts is rednote-hilab's 2B fully-continuous autoregressive TTS — widely
cited as among the strongest open zero-shot voice-cloning models. 24
languages, 48 kHz output, Apache-2.0 (code + checkpoints).
It runs in its own subprocess **and its own venv**, isolated from the
OmniVoice parent, for the same ``transformers`` reason as IndexTTS and
MOSS-TTS-v1.5: dots.tts pins ``transformers==4.57.0`` (verified against
``constraints/recommended.txt``), while OmniVoice pins
``transformers>=5.3.0``. The two cannot share one interpreter.
Cross-platform honesty (the strict default-parity rule): dots.tts's
upstream package declares **Linux + macOS** classifiers only **no
Windows** and its device code is **CUDA-or-CPU with no MPS branch**
(verified in ``runtime.py``). So:
* It is **opt-in** (engine-picker selection + a user-provided clone),
never a default so it never becomes a broken default on any platform.
* ``is_available()`` returns ``False`` with a clear reason on **Windows**
rather than offering an engine that can't run there. Windows users are
pointed at WSL2 / a Linux or macOS host.
* ``gpu_compat = ("cuda", "cpu")`` no MPS claim. On Apple Silicon the
upstream package runs on CPU (slow but correct); the faster MLX path is
a community port we deliberately don't auto-wire here.
Three public entry points: ``DotsTTSBackend`` (this module), ``main.py``
(sidecar, runs under dots.tts's ``transformers==4.57`` venv — never imported
by the parent), and ``bootstrap.py`` (venv probe + lazy bootstrap).
"""
from __future__ import annotations
import logging
import sys
from typing import TYPE_CHECKING
from services.subprocess_backend import SubprocessBackend
if TYPE_CHECKING:
import torch # noqa: F401
logger = logging.getLogger("omnivoice.dots_tts")
class DotsTTSBackend(SubprocessBackend):
"""dots.tts (rednote-hilab) — 2B, 24 langs, zero-shot clone, CUDA/CPU.
Runs in a long-lived sidecar over length-prefixed JSON-over-stdio in a
dedicated venv (``transformers==4.57.0``). First synthesize cold-loads
the ~9 GB checkpoint (bf16 on CUDA); subsequent calls reuse the process.
Installation (OmniVoice prefers a user's existing ``${DIR}/.venv``)::
git clone https://github.com/rednote-hilab/dots.tts.git
cd dots.tts
uv venv && uv pip install -e . -c constraints/recommended.txt
Set ``OMNIVOICE_DOTS_TTS_DIR`` to the clone root. OmniVoice creates
``backend/engines/dots_tts/.venv`` lazily on first launch if no venv
exists yet; the user's existing ``${DIR}/.venv`` is preferred if present.
Best cloning quality uses the ``dots.tts-soar`` checkpoint (the default)
and BOTH a reference clip and its exact transcript (continuation
cloning). License: Apache-2.0.
"""
id = "dots-tts"
display_name = (
"dots.tts (2B, 24 langs, zero-shot clone, CUDA/CPU, 48 kHz, Apache-2.0)"
)
supports_voice_design = False # requires ref audio for timbre cloning
# dots.tts emits 48 kHz (verified via checkpoint vocoder.sample_rate).
_DEFAULT_SAMPLE_RATE = 48000
# CUDA + CPU only; no MPS branch in upstream runtime.py.
gpu_compat = ("cuda", "cpu")
# ── availability ───────────────────────────────────────────────────────
@classmethod
def is_available(cls) -> tuple[bool, str]:
# Cross-platform parity: dots.tts upstream is Linux/macOS-only (no
# Windows classifier, no Windows install path). Refuse cleanly on
# Windows instead of advertising an engine that can't run.
if sys.platform == "win32":
return False, (
"dots.tts is not supported on Windows — upstream targets "
"Linux and macOS only. Run OmniVoice under WSL2, or use a "
"Linux/macOS host. See docs/engines/dots-tts.md."
)
# Do NOT import dots_tts here: it pins transformers==4.57, which can't
# coexist with the parent's transformers>=5.3 in one interpreter —
# the reason for the subprocess isolation. Verify the venv on disk
# only; a real health-check is gated on the user's "Test engine"
# action in Settings.
from engines.dots_tts.bootstrap import (
DOTS_TTS_SIDECAR_SCRIPT,
is_dots_tts_installed,
)
if not is_dots_tts_installed():
return False, (
"dots.tts venv not found. Set OMNIVOICE_DOTS_TTS_DIR to your "
"dots.tts clone (the directory containing pyproject.toml) and "
"restart OmniVoice. CUDA or CPU only (no MPS). See "
"docs/engines/dots-tts.md for the full install walk-through."
)
if not DOTS_TTS_SIDECAR_SCRIPT.exists():
return False, (
"dots.tts sidecar script missing at "
f"{DOTS_TTS_SIDECAR_SCRIPT} — reinstall OmniVoice."
)
return True, "ok (CUDA when present, else CPU)"
@classmethod
def venv_python(cls):
from engines.dots_tts.bootstrap import resolve_dots_tts_venv
return resolve_dots_tts_venv()
@classmethod
def sidecar_script(cls):
from engines.dots_tts.bootstrap import DOTS_TTS_SIDECAR_SCRIPT
return DOTS_TTS_SIDECAR_SCRIPT
# ── TTSBackend protocol ────────────────────────────────────────────────
@property
def sample_rate(self) -> int:
return self._DEFAULT_SAMPLE_RATE
@property
def supported_languages(self) -> list[str]:
# 24 languages with auto-detect; expose "multi" on the protocol
# surface and translate the caller's language at synthesize time.
return ["multi"]
# ── generate (parent-side arbitration) ─────────────────────────────────
def generate(self, text: str, **kw) -> "torch.Tensor":
"""Synthesize one utterance through the dots.tts sidecar.
kwargs honored:
* ``ref_audio`` reference clip path ``prompt_audio_path``
(zero-shot cloning). Optional.
* ``ref_text`` the reference transcript ``prompt_text``.
Best cloning fidelity ("continuation"). Upstream
REQUIRES ``prompt_audio_path`` when ``prompt_text``
is set, so we drop a stray ref_text with no
ref_audio rather than let the sidecar raise.
* ``language`` ISO code / name / None (auto-detect).
* ``num_step`` flow-matching steps ``num_steps`` (default 10;
use 4 for the ``dots.tts-mf`` checkpoint).
* ``guidance_scale`` CFG (default 1.2; >2 amplifies energy).
Returns a tensor of shape (1, n_samples) at :attr:`sample_rate`.
"""
forwarded: dict = {}
ref_audio = kw.get("ref_audio")
if ref_audio:
forwarded["ref_audio"] = ref_audio
ref_text = kw.get("ref_text")
if ref_text:
# continuation cloning — only valid alongside ref_audio.
forwarded["ref_text"] = ref_text
elif kw.get("ref_text"):
logger.info(
"dots-tts: ref_text supplied without ref_audio; ignoring "
"(upstream requires prompt_audio_path when prompt_text is set)."
)
language = kw.get("language")
if language:
forwarded["language"] = str(language)
# OmniVoice's generic num_step default is 16; dots.tts's own default
# is 10. Honor an explicit value, else use the dots-appropriate 10.
num_step = kw.get("num_step")
forwarded["num_steps"] = int(num_step) if num_step is not None else 10
# dots.tts's own CFG default is 1.2 (the generic 2.0 over-energises).
guidance = kw.get("guidance_scale")
forwarded["guidance_scale"] = float(guidance) if guidance is not None else 1.2
return super().generate(text, **forwarded)
__all__ = ["DotsTTSBackend"]
-226
View File
@@ -1,226 +0,0 @@
"""dots.tts venv probe + lazy bootstrap (issue #498).
Resolves which Python interpreter runs the dots.tts sidecar. Mirrors
``engines.indextts.bootstrap`` / ``engines.moss_tts_v15.bootstrap`` because
dots.tts has the same shape of problem: a hard ``transformers==4.57.0`` pin
that conflicts with the parent's ``transformers>=5.3`` — so it runs in its
own venv.
Probe order (priority existing power-user installs win, zero migration):
1. ``${OMNIVOICE_DOTS_TTS_DIR}/.venv/`` the user's clone-level venv.
2. ``backend/engines/dots_tts/.venv/`` this package's own venv.
3. Bootstrap: ``uv venv`` then ``uv pip install -e <clone> -c
<clone>/constraints/recommended.txt`` (the upstream-pinned stack:
torch==2.8.0, transformers==4.57.0, ).
Caching: memoised after first success. Tests reset via :func:`invalidate`.
Security: same posture as IndexTTS bootstrap never touches HF_TOKEN; the
sidecar's stderr is redacted by the parent's ``HFTokenRedactor``; the
editable install comes from a user-controlled clone they already trust.
"""
from __future__ import annotations
import logging
import os
import shutil
import subprocess
import sys
from pathlib import Path
from typing import Optional
logger = logging.getLogger("omnivoice.dots_tts.bootstrap")
#: Absolute path to the sidecar entrypoint.
DOTS_TTS_SIDECAR_SCRIPT: Path = Path(__file__).parent / "main.py"
#: This package's owned venv (Probe 2).
_ENGINES_VENV_DIR: Path = Path(__file__).parent / ".venv"
#: Env var pointing at the user's dots.tts clone root.
_CLONE_DIR_ENV: str = "OMNIVOICE_DOTS_TTS_DIR"
#: Per-process resolution cache. Cleared by :func:`invalidate` for tests.
_resolved_python: Optional[Path] = None
_IMPORT_PROBE_TIMEOUT_S = 15
_UV_VENV_TIMEOUT_S = 120
_UV_PIP_INSTALL_TIMEOUT_S = 1800
# ── public API ────────────────────────────────────────────────────────────
def invalidate() -> None:
"""Clear the resolved-python cache. Tests call this between scenarios."""
global _resolved_python
_resolved_python = None
def is_dots_tts_installed() -> bool:
"""Cheap file-existence check for a usable dots.tts venv. Does NOT spawn
the venv Python that's saved for :func:`resolve_dots_tts_venv`."""
for cand in _probe_paths():
if cand.is_file():
return True
return False
def resolve_dots_tts_venv() -> Path:
"""Resolve the sidecar's Python interpreter (probe order in the module
docstring). Memoised. Raises :exc:`RuntimeError` if none can be located
and the bootstrap path is unavailable."""
global _resolved_python
if _resolved_python is not None:
return _resolved_python
clone_dir = os.environ.get(_CLONE_DIR_ENV)
# Probe 1 — user's clone-level venv.
if clone_dir:
cand = _venv_python_path(Path(clone_dir) / ".venv")
if cand.is_file() and _venv_can_import_dots(cand):
logger.info(
"dots.tts venv resolved from %s: %s", _CLONE_DIR_ENV, cand,
)
_resolved_python = cand
return cand
# Probe 2 — this package's own venv.
cand = _venv_python_path(_ENGINES_VENV_DIR)
if cand.is_file() and _venv_can_import_dots(cand):
logger.info("dots.tts venv resolved from engines path: %s", cand)
_resolved_python = cand
return cand
# Probe 3 — bootstrap.
if not clone_dir:
raise RuntimeError(
"dots.tts is not installed. Set the "
f"{_CLONE_DIR_ENV} environment variable to your dots.tts clone "
"(the directory that contains pyproject.toml and constraints/), "
"then restart OmniVoice. See docs/engines/dots-tts.md for the "
"full install walk-through."
)
cand = _bootstrap_engines_venv(Path(clone_dir))
_resolved_python = cand
return cand
# ── internals ─────────────────────────────────────────────────────────────
def _venv_python_path(venv_dir: Path) -> Path:
if sys.platform == "win32":
return venv_dir / "Scripts" / "python.exe"
return venv_dir / "bin" / "python"
def _probe_paths() -> list[Path]:
out: list[Path] = []
clone_dir = os.environ.get(_CLONE_DIR_ENV)
if clone_dir:
out.append(_venv_python_path(Path(clone_dir) / ".venv"))
out.append(_venv_python_path(_ENGINES_VENV_DIR))
return out
def _venv_can_import_dots(python_path: Path) -> bool:
"""Spawn the candidate python and verify ``import dots_tts.runtime`` works.
Bounded by ``_IMPORT_PROBE_TIMEOUT_S``. False on any failure."""
try:
proc = subprocess.run(
[str(python_path), "-c", "import dots_tts.runtime"],
capture_output=True,
timeout=_IMPORT_PROBE_TIMEOUT_S,
)
except (subprocess.TimeoutExpired, OSError) as exc:
logger.debug("dots.tts import probe failed for %s: %s", python_path, exc)
return False
if proc.returncode != 0:
logger.debug(
"dots.tts import probe non-zero for %s: %s",
python_path,
proc.stderr.decode("utf-8", errors="replace")[:200],
)
return False
return True
def _locate_uv() -> Optional[str]:
bundled = os.environ.get("OMNIVOICE_BUNDLED_UV")
if bundled and Path(bundled).is_file():
return bundled
return shutil.which("uv")
def _bootstrap_engines_venv(clone_dir: Path) -> Path:
"""Create engines/dots_tts/.venv and editable-install the user's clone
with the upstream constraints file."""
uv = _locate_uv()
if not uv:
raise RuntimeError(
"uv is required to bootstrap the dots.tts venv but was not found "
"on PATH (and OMNIVOICE_BUNDLED_UV was not set). Install uv from "
"https://docs.astral.sh/uv/ and re-launch OmniVoice, or set "
"OMNIVOICE_BUNDLED_UV to the absolute path of a uv binary."
)
logger.info(
"Bootstrapping dots.tts venv at %s from %s (this can take several "
"minutes on first launch)", _ENGINES_VENV_DIR, clone_dir,
)
try:
subprocess.run(
[uv, "venv", str(_ENGINES_VENV_DIR)],
check=True, timeout=_UV_VENV_TIMEOUT_S, capture_output=True,
)
except subprocess.CalledProcessError as exc:
raise RuntimeError(
f"uv venv failed for dots.tts bootstrap at {_ENGINES_VENV_DIR}: "
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
) from exc
python_path = _venv_python_path(_ENGINES_VENV_DIR)
install_cmd = [
uv, "pip", "install",
"--python", str(python_path),
"-e", str(clone_dir),
]
# Apply the upstream pin set when it ships with the clone.
constraints = clone_dir / "constraints" / "recommended.txt"
if constraints.is_file():
install_cmd += ["-c", str(constraints)]
try:
subprocess.run(
install_cmd, check=True,
timeout=_UV_PIP_INSTALL_TIMEOUT_S, capture_output=True,
)
except subprocess.CalledProcessError as exc:
raise RuntimeError(
"uv pip install -e failed during dots.tts bootstrap "
f"({clone_dir}): "
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}. "
"See docs/engines/dots-tts.md."
) from exc
if not _venv_can_import_dots(python_path):
raise RuntimeError(
"dots.tts bootstrap completed but `import dots_tts.runtime` still "
f"fails from {python_path}. Verify that {clone_dir} is a valid "
"dots.tts clone. See docs/engines/dots-tts.md."
)
logger.info("dots.tts venv bootstrap successful: %s", python_path)
return python_path
__all__ = [
"DOTS_TTS_SIDECAR_SCRIPT",
"invalidate",
"is_dots_tts_installed",
"resolve_dots_tts_venv",
]
-255
View File
@@ -1,255 +0,0 @@
"""dots.tts sidecar entry point (issue #498).
Runs inside ``engines/dots_tts/.venv`` (or the user's existing
``${OMNIVOICE_DOTS_TTS_DIR}/.venv``) with ``transformers==4.57.0``, isolated
from the OmniVoice parent (``transformers>=5.3``). Same isolation rationale
as the IndexTTS / MOSS-TTS-v1.5 sidecars.
Stdlib-only at import time; ``dots_tts`` + torch are imported lazily on the
first synthesize op so the ``ready`` frame fits inside the parent's 30 s
spawn handshake even on a cold filesystem.
Wire protocol length-prefixed JSON over stdin/stdout, byte-identical to
``backend/services/subprocess_backend.py``::
[ 4-byte big-endian uint32 length ][ N bytes UTF-8 JSON ]
Op flow:
1. Sidecar -> parent: {"op": "ready", "engine": "dots-tts",
"sample_rate": 48000}
2. parent -> sidecar: {"op": "ping"} -> {"op": "pong", "vram_mb": N}
3. parent -> sidecar: {"op": "synthesize", "text": "...",
"ref_audio": "/path/ref.wav",
"ref_text": "transcript", "language": "EN",
"num_steps": 10, "guidance_scale": 1.2}
-> {"op": "progress", ...} (cold load) then
-> {"op": "audio", "audio_pcm_b64": "...", "sample_rate": 48000,
"n_samples": N}
4. parent -> sidecar: {"op": "shutdown"} -> exit 0
Restrictions: NO imports from OmniVoice parent code (different venv). NO
logging of ``os.environ`` contents. Single-frame DoS cap matches the
parent's ``MAX_FRAME_BYTES``.
"""
from __future__ import annotations
import base64
import json
import os
import struct
import sys
import traceback
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
MAX_FRAME_BYTES = 64 * 1024 * 1024
#: dots.tts emits 48 kHz (checkpoint vocoder.sample_rate). Advertised in the
#: ready frame; the real value is re-read from each generate() result.
DOTS_SAMPLE_RATE = 48000
#: Default checkpoint. ``-soar`` is the best-cloning variant; ``-mf`` is the
#: fastest (use num_steps=4). Overridable for air-gapped / mirror installs.
_DEFAULT_REPO = "rednote-hilab/dots.tts-soar"
# ── wire protocol ─────────────────────────────────────────────────────────
def _send(stream, obj: dict) -> None:
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
stream.write(struct.pack("!I", len(body)))
stream.write(body)
stream.flush()
def _recv(stream):
header = stream.read(4)
if len(header) < 4:
return None # EOF
(n,) = struct.unpack("!I", header)
if n > MAX_FRAME_BYTES:
raise IOError(f"frame too large: {n}")
body = bytearray()
while len(body) < n:
chunk = stream.read(n - len(body))
if not chunk:
raise IOError("short read")
body.extend(chunk)
return json.loads(bytes(body).decode("utf-8"))
def _measure_vram_mb() -> float:
"""This sidecar's own GPU memory in MB (MM2-08). 0 on CPU. Never raises."""
try:
import torch
if torch.cuda.is_available():
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
except Exception:
pass
return 0.0
# ── model loading (lazy, on first synthesize) ─────────────────────────────
# Module-level singleton — (runtime,). Device is auto-selected inside the
# dots.tts runtime (cuda-or-cpu, no MPS); we don't pass a device.
_runtime = None
def _load_runtime(stdout):
"""Cold-construct the dots.tts runtime.
``DotsTtsRuntime.from_pretrained`` auto-selects cuda-or-cpu internally
(no MPS path). precision is bf16 on CUDA; on CPU we fall back to fp32
(bf16 CPU kernels are spotty). Both overridable via env.
"""
global _runtime
if _runtime is not None:
return _runtime
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 0})
import torch
from dots_tts.runtime import DotsTtsRuntime # type: ignore[import-not-found]
repo = os.environ.get("OMNIVOICE_DOTS_TTS_MODEL", _DEFAULT_REPO)
default_precision = "bfloat16" if torch.cuda.is_available() else "float32"
precision = os.environ.get("OMNIVOICE_DOTS_TTS_PRECISION", default_precision)
optimize = os.environ.get("OMNIVOICE_DOTS_TTS_OPTIMIZE", "0") == "1"
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 50})
_runtime = DotsTtsRuntime.from_pretrained(
repo,
precision=precision,
optimize=optimize,
)
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 100})
return _runtime
def _tensor_to_pcm_b64(audio, sample_rate: int) -> tuple[str, int, int]:
"""Convert a torch waveform tensor (1, N) in [-1, 1] to base64 int16 PCM."""
import numpy as np
arr = audio.detach().to("cpu").float().numpy()
arr = np.asarray(arr, dtype=np.float32).squeeze()
if arr.ndim > 1:
arr = arr.mean(axis=0) # defensive downmix to mono
arr = np.clip(arr, -1.0, 1.0)
pcm = (arr * 32767.0).astype(np.int16).tobytes()
return base64.b64encode(pcm).decode("ascii"), int(sample_rate), int(arr.shape[0])
def _normalize_language(raw):
"""Map OmniVoice's language value to what dots.tts accepts, or None.
dots.tts accepts None/"auto_detect", ISO codes upper-cased ("EN"/"ZH"),
or names ("english"). A 2-letter ISO code is upper-cased; anything else
is passed through; empty / "auto" None (auto-detect)."""
if not raw or not isinstance(raw, str):
return None
s = raw.strip()
if not s or s.lower() == "auto":
return None
if len(s) == 2 and s.isalpha():
return s.upper()
return s
def _handle_synthesize(msg: dict, stdout) -> None:
"""Dispatch one synthesize request. Emits the audio frame or raises."""
text = msg.get("text")
if not text or not isinstance(text, str):
raise ValueError("synthesize: missing or non-string 'text'")
runtime = _load_runtime(stdout)
gen_kwargs: dict = {
"text": text,
"num_steps": int(msg.get("num_steps", 10)),
"guidance_scale": float(msg.get("guidance_scale", 1.2)),
}
ref_audio = msg.get("ref_audio")
if ref_audio:
gen_kwargs["prompt_audio_path"] = ref_audio
ref_text = msg.get("ref_text")
if ref_text:
# continuation cloning — upstream requires prompt_audio_path when
# prompt_text is set (the parent already enforces this).
gen_kwargs["prompt_text"] = ref_text
language = _normalize_language(msg.get("language"))
if language:
gen_kwargs["language"] = language
result = runtime.generate(**gen_kwargs)
audio = result["audio"]
sample_rate = int(result.get("sample_rate", DOTS_SAMPLE_RATE))
pcm_b64, sr, n_samples = _tensor_to_pcm_b64(audio, sample_rate)
_send(stdout, {
"op": "audio",
"audio_pcm_b64": pcm_b64,
"sample_rate": sr,
"n_samples": n_samples,
})
# ── main loop ─────────────────────────────────────────────────────────────
def main() -> int:
stdin = sys.stdin.buffer
stdout = sys.stdout.buffer
# Ready handshake fires BEFORE any heavy import.
_send(stdout, {
"op": "ready",
"engine": "dots-tts",
"sample_rate": DOTS_SAMPLE_RATE,
})
while True:
try:
msg = _recv(stdin)
except Exception as exc:
_send(stdout, {
"op": "error",
"stage": "recv",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
return 1
if msg is None:
return 0
op = msg.get("op") if isinstance(msg, dict) else None
try:
if op == "ping":
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
elif op == "synthesize":
_handle_synthesize(msg, stdout)
elif op == "shutdown":
return 0
else:
_send(stdout, {
"op": "error",
"stage": "dispatch",
"message": f"unknown op: {op!r}",
})
except Exception as exc:
_send(stdout, {
"op": "error",
"stage": op or "unknown",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
if __name__ == "__main__":
sys.exit(main())
-5
View File
@@ -79,11 +79,6 @@ class IndexTTS2Backend(SubprocessBackend):
display_name = "IndexTTS2 (emotion control, duration control, zero-shot)"
supports_voice_design = False # requires ref audio for timbre
_DEFAULT_SAMPLE_RATE = 24000
# Explicit so IndexTTS2 stops advertising the inherited CPU-only default:
# the sidecar runs the IndexTTS PyTorch model on CUDA when present, else
# CPU. ROCm left unclaimed (the sidecar's own venv would need a ROCm torch);
# a ROCm host honestly resolves to cpu_fallback.
gpu_compat = ("cuda", "cpu")
@classmethod
def is_available(cls) -> tuple[bool, str]:
+1 -19
View File
@@ -74,24 +74,6 @@ import traceback
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
MAX_FRAME_BYTES = 64 * 1024 * 1024
def _measure_vram_mb() -> float:
"""This sidecar's own GPU memory in MB, for the loaded-models panel
(MM2-08). The parent can't see a child's VRAM, so we self-report it in the
pong. Degrades to 0 on CPU / when torch isn't loaded yet — never raises."""
try:
import torch # already a dep inside the indextts venv
if torch.cuda.is_available():
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
mps = getattr(torch.backends, "mps", None)
if mps is not None and mps.is_available():
drv = getattr(torch.mps, "driver_allocated_memory", None)
if drv:
return round(drv() / (1024 ** 2), 1)
except Exception:
pass
return 0.0
# Sample rate IndexTTS-2 emits natively. Advertised in the ready frame so
# the parent doesn't have to import IndexTTS just to learn the rate.
INDEXTTS_SAMPLE_RATE = 24000
@@ -300,7 +282,7 @@ def main() -> int:
op = msg.get("op") if isinstance(msg, dict) else None
try:
if op == "ping":
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
_send(stdout, {"op": "pong"})
elif op == "synthesize":
_handle_synthesize(msg, stdout)
elif op == "shutdown":
-194
View File
@@ -1,194 +0,0 @@
"""MOSS-TTS-v1.5 sidecar package (issue #498).
MOSS-TTS-v1.5 is OpenMOSS's 8B flagship TTS — a Qwen3-8B language backbone
plus a 1.6B audio codec, 31 languages, zero-shot voice cloning, token-level
duration control and inline ``[pause Ns]`` markers. Apache-2.0.
It runs in its own subprocess **and its own venv**, isolated from the
OmniVoice parent process, for the *same* reason IndexTTS does: a hard
``transformers`` version conflict. MOSS-TTS-v1.5's ``torch-runtime`` extra
pins ``transformers==5.0.0`` (verified against the upstream
``pyproject.toml``), while OmniVoice pins ``transformers>=5.3.0``. The two
cannot share one interpreter so MOSS lives behind ``SubprocessBackend``
with a dedicated venv, exactly like ``engines.indextts``.
Three public entry points live in this package:
* ``MossTTSV15Backend`` (this module) the SubprocessBackend subclass
that ``services.tts_backend._LAZY_REGISTRY`` resolves on first access.
Defined HERE (not in ``services.tts_backend``) to break the import
cycle: ``services.subprocess_backend`` imports ``TTSBackend`` from
``services.tts_backend``, so the backend class must live downstream of
that module finishing its import. Same indirection as IndexTTS /
Supertonic-3.
* ``main.py`` the sidecar entrypoint (runs under MOSS's venv with
``transformers==5.0.0``; never imported by the parent).
* ``bootstrap.py`` the venv-probe + lazy-bootstrap helper.
Do NOT import ``main.py`` from the parent process it runs under a
different venv (``transformers==5.0.0``) and importing it in-process would
re-introduce the exact conflict this isolation exists to avoid.
Hardware honesty (cross-platform rule): MOSS-TTS-v1.5's upstream documents
only CUDA and CPU. There is **no documented or tested MPS path** the
custom ``trust_remote_code`` modelling code and the separate audio
tokenizer are unverified on Apple Silicon. We therefore advertise
``gpu_compat = ("cuda", "cpu")`` and the sidecar selects ``cuda`` when
present else ``cpu`` it never silently routes to MPS where it might
crash. On Apple Silicon the engine honestly resolves to CPU (slow but
correct), and the engine is opt-in regardless, so it never becomes a
broken default on any platform.
"""
from __future__ import annotations
import logging
from typing import TYPE_CHECKING
from services.subprocess_backend import SubprocessBackend
if TYPE_CHECKING:
import torch # noqa: F401
logger = logging.getLogger("omnivoice.moss_tts_v15")
#: 1 second of audio ≈ 12.5 codec tokens (MOSS-TTS-v1.5 model card). Used to
#: translate OmniVoice's ``duration`` (seconds) into the model's ``tokens``
#: duration-control argument.
TOKENS_PER_SECOND: float = 12.5
class MossTTSV15Backend(SubprocessBackend):
"""MOSS-TTS-v1.5 (OpenMOSS) — 8B, 31 langs, zero-shot clone, CUDA/CPU.
Runs in a long-lived sidecar over length-prefixed JSON-over-stdio in a
dedicated venv (``transformers==5.0.0``). The first synthesize cold-loads
~16 GB of bf16 weights (CUDA) / fp32 (CPU); subsequent calls reuse the
process and the in-memory model.
Installation (transparent to power users who already cloned MOSS-TTS
OmniVoice prefers their existing ``${DIR}/.venv``)::
git clone https://github.com/OpenMOSS/MOSS-TTS.git
cd MOSS-TTS
# CUDA host:
uv venv && uv pip install -e ".[torch-runtime]"
# non-CUDA host (CPU): install plain torch/transformers instead of +cu128
Set ``OMNIVOICE_MOSS_TTS_V15_DIR`` to the clone root. OmniVoice creates
``backend/engines/moss_tts_v15/.venv`` lazily on first launch if no venv
exists yet (CUDA hosts only the upstream ``torch-runtime`` extra is
``+cu128``); the user's existing ``${DIR}/.venv`` is preferred if
present, so no re-install is needed.
License: Apache-2.0 (code + weights) no acceptance gate needed.
"""
id = "moss-tts-v15"
display_name = (
"MOSS-TTS-v1.5 (8B, 31 langs, zero-shot clone, CUDA/CPU, Apache-2.0)"
)
supports_voice_design = False # requires ref audio for timbre cloning
_DEFAULT_SAMPLE_RATE = 24000
# Honest hardware surface: upstream documents CUDA + CPU only. MPS is
# undocumented / untested, so we do NOT claim it (cross-platform rule).
gpu_compat = ("cuda", "cpu")
# ── availability ───────────────────────────────────────────────────────
@classmethod
def is_available(cls) -> tuple[bool, str]:
# IMPORTANT: do NOT attempt to import MOSS / its transformers==5.0.0
# here. The parent pins transformers>=5.3 — co-importing the two in
# one interpreter is exactly the conflict this subprocess isolation
# exists to avoid. We only verify the venv exists on disk; a real
# health-check (spawn + ping) is gated on the user's "Test engine"
# action in Settings, same as IndexTTS.
from engines.moss_tts_v15.bootstrap import (
MOSS_TTS_V15_SIDECAR_SCRIPT,
is_moss_tts_v15_installed,
)
if not is_moss_tts_v15_installed():
return False, (
"MOSS-TTS-v1.5 venv not found. Set OMNIVOICE_MOSS_TTS_V15_DIR "
"to your MOSS-TTS clone (the directory containing pyproject.toml) "
"and restart OmniVoice. CUDA or CPU only (no MPS). See "
"docs/engines/moss-tts-v15.md for the full install walk-through."
)
if not MOSS_TTS_V15_SIDECAR_SCRIPT.exists():
return False, (
"MOSS-TTS-v1.5 sidecar script missing at "
f"{MOSS_TTS_V15_SIDECAR_SCRIPT} — reinstall OmniVoice."
)
return True, "ok (CUDA when present, else CPU)"
@classmethod
def venv_python(cls):
from engines.moss_tts_v15.bootstrap import resolve_moss_tts_v15_venv
return resolve_moss_tts_v15_venv()
@classmethod
def sidecar_script(cls):
from engines.moss_tts_v15.bootstrap import MOSS_TTS_V15_SIDECAR_SCRIPT
return MOSS_TTS_V15_SIDECAR_SCRIPT
# ── TTSBackend protocol ────────────────────────────────────────────────
@property
def sample_rate(self) -> int:
return self._DEFAULT_SAMPLE_RATE
@property
def supported_languages(self) -> list[str]:
# 31 languages with multilingual handling; expose "multi" on the
# protocol surface (same as OmniVoice / CosyVoice / Supertonic-3) and
# translate the caller's language at synthesize time.
return ["multi"]
# ── generate (parent-side arbitration) ─────────────────────────────────
def generate(self, text: str, **kw) -> "torch.Tensor":
"""Synthesize one utterance through the MOSS-TTS-v1.5 sidecar.
kwargs honored:
* ``ref_audio`` path to a reference clip. When present, MOSS
runs zero-shot voice cloning (``reference=``).
Optional: without it the model uses its own
default voice.
* ``ref_text`` accepted but unused in clone mode (MOSS's
zero-shot path needs only the audio); kept in
the signature so the common call-site doesn't
need engine-specific knowledge.
* ``language`` ISO code or name; mapped to a MOSS language name
in the sidecar, omitted (auto-detect) if unknown.
* ``duration`` target seconds ``tokens`` (1 s 12.5 tokens).
* ``max_new_tokens`` generation cap (default 4096).
Returns a tensor of shape (1, n_samples) at :attr:`sample_rate`.
"""
forwarded: dict = {}
ref_audio = kw.get("ref_audio")
if ref_audio:
forwarded["ref_audio"] = ref_audio
ref_text = kw.get("ref_text")
if ref_text:
forwarded["ref_text"] = ref_text
language = kw.get("language")
if language:
forwarded["language"] = str(language)
duration = kw.get("duration")
if duration is not None:
target_tokens = int(float(duration) * TOKENS_PER_SECOND)
if target_tokens > 0:
forwarded["tokens"] = target_tokens
max_new_tokens = kw.get("max_new_tokens")
if max_new_tokens is not None:
forwarded["max_new_tokens"] = int(max_new_tokens)
return super().generate(text, **forwarded)
__all__ = ["MossTTSV15Backend", "TOKENS_PER_SECOND"]
-272
View File
@@ -1,272 +0,0 @@
"""MOSS-TTS-v1.5 venv probe + lazy bootstrap (issue #498).
The parent process needs to know *which Python interpreter* to spawn the
MOSS-TTS-v1.5 sidecar under. This module owns that resolution. It mirrors
``engines.indextts.bootstrap`` because MOSS has the same shape of problem:
a hard ``transformers`` pin (``==5.0.0``) that conflicts with the parent's
``transformers>=5.3`` so MOSS runs in its own venv.
Probe order (priority existing power-user installs win, zero migration):
1. ``${OMNIVOICE_MOSS_TTS_V15_DIR}/.venv/`` the user's clone-level
venv. Highest priority: a user who already cloned MOSS-TTS and ran
``uv pip install -e ".[torch-runtime]"`` (per upstream docs) gets
reused verbatim, no re-download of the ~16 GB model.
2. ``backend/engines/moss_tts_v15/.venv/`` this package's own venv,
created by step 3 if needed.
3. Bootstrap: ``uv venv`` then ``uv pip install -e
"${DIR}[torch-runtime]"``. Requires ``OMNIVOICE_MOSS_TTS_V15_DIR``.
The upstream ``torch-runtime`` extra is CUDA (``+cu128``), so the
auto-bootstrap targets CUDA hosts; non-CUDA (CPU/Mac) users set up
their own venv per docs/engines/moss-tts-v15.md (Probe 1).
Caching: resolution is memoised after the first successful call. Tests
reset via :func:`invalidate`.
Security: bootstrap never touches HF_TOKEN; the sidecar's stderr is drained
by SubprocessBackend through the parent root logger where Phase 1's
``HFTokenRedactor`` strips token bytes. ``uv pip install -e`` installs from
a user-controlled clone the user already trusts (same posture as IndexTTS).
"""
from __future__ import annotations
import logging
import os
import shutil
import subprocess
import sys
from pathlib import Path
from typing import Optional
logger = logging.getLogger("omnivoice.moss_tts_v15.bootstrap")
#: Absolute path to the sidecar entrypoint. ``MossTTSV15Backend.sidecar_script``
#: returns this; SubprocessBackend spawns it with the resolved venv python.
MOSS_TTS_V15_SIDECAR_SCRIPT: Path = Path(__file__).parent / "main.py"
#: This package's owned venv (Probe 2). The MOSS-TTS clone, when bootstrapped,
#: is installed into this venv via ``uv pip install -e``.
_ENGINES_VENV_DIR: Path = Path(__file__).parent / ".venv"
#: Env var pointing at the user's MOSS-TTS clone root.
_CLONE_DIR_ENV: str = "OMNIVOICE_MOSS_TTS_V15_DIR"
#: Per-process resolution cache. Cleared by :func:`invalidate` for tests.
_resolved_python: Optional[Path] = None
# Timeouts — bounded so a wedged venv never hangs the parent. The bootstrap
# install can take many minutes on a cold cache (MOSS pulls a CUDA torch
# build + transformers + an audio codec stack).
_IMPORT_PROBE_TIMEOUT_S = 15
_UV_VENV_TIMEOUT_S = 120
_UV_PIP_INSTALL_TIMEOUT_S = 1800
# ── public API ────────────────────────────────────────────────────────────
def invalidate() -> None:
"""Clear the resolved-python cache. Tests call this between scenarios."""
global _resolved_python
_resolved_python = None
def is_moss_tts_v15_installed() -> bool:
"""Cheap file-existence check for a usable MOSS-TTS-v1.5 venv.
Returns True if either Probe 1 or Probe 2 has a Python executable on
disk. Does NOT spawn the venv Python that's saved for
:func:`resolve_moss_tts_v15_venv`, which is only invoked on the first
generate() / health_check(). This fires on every Settings render via
``MossTTSV15Backend.is_available()``, so it stays cheap.
"""
for cand in _probe_paths():
if cand.is_file():
return True
return False
def resolve_moss_tts_v15_venv() -> Path:
"""Resolve the path to the Python interpreter that runs the sidecar.
Probe order described in the module docstring. Memoised. Raises
:exc:`RuntimeError` if no working venv can be located AND the bootstrap
path is unavailable.
"""
global _resolved_python
if _resolved_python is not None:
return _resolved_python
clone_dir = os.environ.get(_CLONE_DIR_ENV)
# Probe 1 — user's clone-level venv (highest priority for back-compat).
if clone_dir:
cand = _venv_python_path(Path(clone_dir) / ".venv")
if cand.is_file() and _venv_can_import_moss(cand):
logger.info(
"MOSS-TTS-v1.5 venv resolved from %s: %s", _CLONE_DIR_ENV, cand,
)
_resolved_python = cand
return cand
# Probe 2 — this package's own venv.
cand = _venv_python_path(_ENGINES_VENV_DIR)
if cand.is_file() and _venv_can_import_moss(cand):
logger.info("MOSS-TTS-v1.5 venv resolved from engines path: %s", cand)
_resolved_python = cand
return cand
# Probe 3 — bootstrap.
if not clone_dir:
raise RuntimeError(
"MOSS-TTS-v1.5 is not installed. Set the "
f"{_CLONE_DIR_ENV} environment variable to your MOSS-TTS clone "
"(the directory that contains pyproject.toml), then restart "
"OmniVoice. See docs/engines/moss-tts-v15.md for the full "
"install walk-through."
)
cand = _bootstrap_engines_venv(Path(clone_dir))
_resolved_python = cand
return cand
# ── internals ─────────────────────────────────────────────────────────────
def _venv_python_path(venv_dir: Path) -> Path:
"""Return the python executable path inside a venv directory.
Handles the Unix (``bin/python``) vs Windows (``Scripts/python.exe``)
layout. No filesystem access caller checks .is_file().
"""
if sys.platform == "win32":
return venv_dir / "Scripts" / "python.exe"
return venv_dir / "bin" / "python"
def _probe_paths() -> list[Path]:
"""Ordered list of candidate venv-python paths (no .is_file() check)."""
out: list[Path] = []
clone_dir = os.environ.get(_CLONE_DIR_ENV)
if clone_dir:
out.append(_venv_python_path(Path(clone_dir) / ".venv"))
out.append(_venv_python_path(_ENGINES_VENV_DIR))
return out
def _venv_can_import_moss(python_path: Path) -> bool:
"""Spawn the candidate python and verify the MOSS stack imports.
MOSS-TTS-v1.5 loads via ``transformers`` + ``trust_remote_code`` (no
fixed top-level package to import), so the readiness signal is that the
venv has a working ``transformers`` + ``torch`` which only the
``[torch-runtime]`` install provides. Bounded by
``_IMPORT_PROBE_TIMEOUT_S`` so a wedged venv never hangs the parent.
Returns False on any failure (non-zero exit, timeout, OSError).
"""
try:
proc = subprocess.run(
[str(python_path), "-c", "import transformers, torch"],
capture_output=True,
timeout=_IMPORT_PROBE_TIMEOUT_S,
)
except (subprocess.TimeoutExpired, OSError) as exc:
logger.debug("moss-tts-v15 import probe failed for %s: %s", python_path, exc)
return False
if proc.returncode != 0:
logger.debug(
"moss-tts-v15 import probe non-zero for %s: %s",
python_path,
proc.stderr.decode("utf-8", errors="replace")[:200],
)
return False
return True
def _locate_uv() -> Optional[str]:
"""Find the uv binary — bundled first (Tauri-set env var), else PATH."""
bundled = os.environ.get("OMNIVOICE_BUNDLED_UV")
if bundled and Path(bundled).is_file():
return bundled
sys_uv = shutil.which("uv")
if sys_uv:
return sys_uv
return None
def _bootstrap_engines_venv(clone_dir: Path) -> Path:
"""Create engines/moss_tts_v15/.venv and install the user's clone into it.
Runs ``uv venv <engines_venv>`` then ``uv pip install --python
<engines_venv>/bin/python -e "<clone>[torch-runtime]"``. Verifies the
result by re-probing the import a successful uv invocation that still
can't import the stack indicates a deeper environment problem (e.g. the
``+cu128`` torch-runtime extra can't resolve on a non-CUDA host) and we
raise with whatever stderr we captured plus a docs pointer.
"""
uv = _locate_uv()
if not uv:
raise RuntimeError(
"uv is required to bootstrap the MOSS-TTS-v1.5 venv but was not "
"found on PATH (and OMNIVOICE_BUNDLED_UV was not set). Install uv "
"from https://docs.astral.sh/uv/ and re-launch OmniVoice, or set "
"OMNIVOICE_BUNDLED_UV to the absolute path of a uv binary."
)
logger.info(
"Bootstrapping MOSS-TTS-v1.5 venv at %s from %s (this can take "
"several minutes on first launch)", _ENGINES_VENV_DIR, clone_dir,
)
try:
subprocess.run(
[uv, "venv", str(_ENGINES_VENV_DIR)],
check=True,
timeout=_UV_VENV_TIMEOUT_S,
capture_output=True,
)
except subprocess.CalledProcessError as exc:
raise RuntimeError(
f"uv venv failed for MOSS-TTS-v1.5 bootstrap at {_ENGINES_VENV_DIR}: "
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
) from exc
python_path = _venv_python_path(_ENGINES_VENV_DIR)
try:
subprocess.run(
[
uv, "pip", "install",
"--python", str(python_path),
"-e", f"{clone_dir}[torch-runtime]",
],
check=True,
timeout=_UV_PIP_INSTALL_TIMEOUT_S,
capture_output=True,
)
except subprocess.CalledProcessError as exc:
raise RuntimeError(
"uv pip install -e failed during MOSS-TTS-v1.5 bootstrap "
f"({clone_dir}). On a non-CUDA host the upstream '[torch-runtime]' "
"extra (cu128) cannot resolve — set up the venv manually per "
"docs/engines/moss-tts-v15.md. Error: "
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
) from exc
if not _venv_can_import_moss(python_path):
raise RuntimeError(
"MOSS-TTS-v1.5 bootstrap completed but the transformers/torch "
f"import still fails from {python_path}. Verify that {clone_dir} "
"is a valid MOSS-TTS clone. See docs/engines/moss-tts-v15.md."
)
logger.info("MOSS-TTS-v1.5 venv bootstrap successful: %s", python_path)
return python_path
__all__ = [
"MOSS_TTS_V15_SIDECAR_SCRIPT",
"invalidate",
"is_moss_tts_v15_installed",
"resolve_moss_tts_v15_venv",
]
-303
View File
@@ -1,303 +0,0 @@
"""MOSS-TTS-v1.5 sidecar entry point (issue #498).
Runs inside ``engines/moss_tts_v15/.venv`` (or the user's existing
``${OMNIVOICE_MOSS_TTS_V15_DIR}/.venv``) with ``transformers==5.0.0``,
isolated from the OmniVoice parent process which pins ``transformers>=5.3``.
Same isolation rationale as the IndexTTS sidecar.
Stdlib-only at import time. The model + transformers + torch are imported
lazily on the first synthesize op so the sidecar emits its ``ready`` frame
inside the parent's 30 s spawn handshake even on a cold filesystem (an 8B
model takes well over 30 s to cold-load).
Wire protocol length-prefixed JSON over stdin/stdout, byte-identical to
``backend/services/subprocess_backend.py``::
[ 4-byte big-endian uint32 length ][ N bytes UTF-8 JSON ]
Op flow:
1. Sidecar -> parent: {"op": "ready", "engine": "moss-tts-v15",
"sample_rate": 24000}
2. parent -> sidecar: {"op": "ping"} -> {"op": "pong", "vram_mb": N}
3. parent -> sidecar: {"op": "synthesize", "text": "...",
"ref_audio": "/path/spk.wav", "language": "fr",
"tokens": 325, "max_new_tokens": 4096}
-> {"op": "progress", ...} (cold load only) then
-> {"op": "audio", "audio_pcm_b64": "...", "sample_rate": 24000,
"n_samples": N}
4. parent -> sidecar: {"op": "shutdown"} -> exit 0
Restrictions: NO imports from OmniVoice parent code (different venv). NO
logging of ``os.environ`` contents. Single-frame DoS cap matches the
parent's ``MAX_FRAME_BYTES``.
"""
from __future__ import annotations
import base64
import json
import os
import struct
import sys
import traceback
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
MAX_FRAME_BYTES = 64 * 1024 * 1024
#: Native sample rate MOSS-TTS-v1.5 emits. Advertised in the ready frame so
#: the parent doesn't have to import MOSS just to learn the rate. Confirmed
#: via ``processor.model_config.sampling_rate`` (the real value is read from
#: the loaded model at synthesize time; this is the handshake default).
MOSS_SAMPLE_RATE = 24000
#: HF repo id for the weights, overridable for air-gapped / mirror installs.
_DEFAULT_REPO = "OpenMOSS-Team/MOSS-TTS-v1.5"
#: ISO-639-1 → MOSS language name. MOSS's ``build_user_message`` takes a
#: language *name* ("French"), not a code. Unknown codes are omitted so the
#: model auto-detects. Covers the high-traffic subset of MOSS's 31 langs.
_ISO_TO_NAME = {
"en": "English", "zh": "Chinese", "ja": "Japanese", "ko": "Korean",
"fr": "French", "de": "German", "es": "Spanish", "it": "Italian",
"pt": "Portuguese", "ru": "Russian", "ar": "Arabic", "hi": "Hindi",
"nl": "Dutch", "pl": "Polish", "tr": "Turkish", "vi": "Vietnamese",
"th": "Thai", "id": "Indonesian", "cs": "Czech", "el": "Greek",
"he": "Hebrew", "fa": "Persian", "uk": "Ukrainian", "sv": "Swedish",
}
# ── wire protocol ─────────────────────────────────────────────────────────
def _send(stream, obj: dict) -> None:
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
stream.write(struct.pack("!I", len(body)))
stream.write(body)
stream.flush()
def _recv(stream):
header = stream.read(4)
if len(header) < 4:
return None # EOF
(n,) = struct.unpack("!I", header)
if n > MAX_FRAME_BYTES:
raise IOError(f"frame too large: {n}")
body = bytearray()
while len(body) < n:
chunk = stream.read(n - len(body))
if not chunk:
raise IOError("short read")
body.extend(chunk)
return json.loads(bytes(body).decode("utf-8"))
def _measure_vram_mb() -> float:
"""This sidecar's own GPU memory in MB (MM2-08). The parent can't see a
child's VRAM, so we self-report it in the pong. 0 on CPU. Never raises."""
try:
import torch
if torch.cuda.is_available():
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
except Exception:
pass
return 0.0
# ── model loading (lazy, on first synthesize) ─────────────────────────────
# Module-level singleton — populated on the first synthesize op and reused.
# Holds (processor, model, device, sample_rate).
_state = None
def _load_model(stdout):
"""Cold-construct the MOSS-TTS-v1.5 processor + model.
Device selection is CUDA-or-CPU only MOSS's upstream documents no MPS
path and the custom ``trust_remote_code`` modelling code is untested on
Apple Silicon, so we never route to MPS where it might crash. dtype is
bf16 on CUDA, fp32 on CPU (bf16 CPU ops are spotty). Emits progress
frames so the parent can surface the multi-GB cold-load latency.
"""
global _state
if _state is not None:
return _state
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 0})
import torch
from transformers import AutoModel, AutoProcessor
repo = os.environ.get("OMNIVOICE_MOSS_TTS_V15_MODEL", _DEFAULT_REPO)
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
# "sdpa" works on CUDA + CPU and needs no extra dep. flash_attention_2
# (Ampere+ CUDA, optional flash-attn) is opt-in via env.
attn = os.environ.get("OMNIVOICE_MOSS_TTS_V15_ATTN", "sdpa")
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
# The audio tokenizer is a separate sub-module that must be moved to the
# device independently (easy to miss — see upstream README).
processor.audio_tokenizer = processor.audio_tokenizer.to(device)
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 50})
model = AutoModel.from_pretrained(
repo,
trust_remote_code=True,
attn_implementation=attn,
torch_dtype=dtype,
).to(device)
model.eval()
sample_rate = int(getattr(processor.model_config, "sampling_rate", MOSS_SAMPLE_RATE))
_state = (processor, model, device, sample_rate)
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 100})
return _state
def _tensor_to_pcm_b64(audio, sample_rate: int) -> tuple[str, int, int]:
"""Convert a torch waveform tensor to base64 int16 PCM.
MOSS returns a float tensor in [-1, 1] (1-D or (1, N)); we squeeze to
mono, clip, scale to int16, and base64 so the wire frame stays JSON-safe.
"""
import numpy as np
arr = audio.detach().to("cpu").float().numpy()
arr = np.asarray(arr, dtype=np.float32).squeeze()
if arr.ndim > 1:
arr = arr.mean(axis=0) # defensive downmix to mono
arr = np.clip(arr, -1.0, 1.0)
pcm = (arr * 32767.0).astype(np.int16).tobytes()
return base64.b64encode(pcm).decode("ascii"), int(sample_rate), int(arr.shape[0])
def _resolve_language(raw):
"""Map OmniVoice's language value to a MOSS language name, or None.
Accepts an ISO-639-1 code or a full name. Unknown / empty / "auto"
values return None so MOSS auto-detects."""
if not raw or not isinstance(raw, str):
return None
s = raw.strip()
if not s or s.lower() == "auto":
return None
if s.lower() in _ISO_TO_NAME:
return _ISO_TO_NAME[s.lower()]
# Already a language name (or an unknown code) — pass it through; MOSS
# ignores a language it doesn't recognise.
return s
def _handle_synthesize(msg: dict, stdout) -> None:
"""Dispatch one synthesize request. Emits the audio frame or raises."""
import torch
text = msg.get("text")
if not text or not isinstance(text, str):
raise ValueError("synthesize: missing or non-string 'text'")
processor, model, device, sample_rate = _load_model(stdout)
user_kwargs: dict = {"text": text}
ref_audio = msg.get("ref_audio")
if ref_audio:
# Zero-shot voice cloning: the reference audio alone is enough in
# MOSS's clone mode (ref_text is not consumed here). The processor's
# audio tokenizer encodes the reference into the prompt.
user_kwargs["reference"] = [ref_audio]
language = _resolve_language(msg.get("language"))
if language:
user_kwargs["language"] = language
tokens = msg.get("tokens")
if tokens is not None:
user_kwargs["tokens"] = int(tokens)
max_new_tokens = int(msg.get("max_new_tokens", 4096))
conversations = [[processor.build_user_message(**user_kwargs)]]
with torch.no_grad():
batch = processor(conversations, mode="generation")
outputs = model.generate(
input_ids=batch["input_ids"].to(device),
attention_mask=batch["attention_mask"].to(device),
max_new_tokens=max_new_tokens,
)
decoded = processor.decode(outputs)
audio = decoded[0].audio_codes_list[0]
pcm_b64, sr, n_samples = _tensor_to_pcm_b64(audio, sample_rate)
_send(stdout, {
"op": "audio",
"audio_pcm_b64": pcm_b64,
"sample_rate": sr,
"n_samples": n_samples,
})
# ── main loop ─────────────────────────────────────────────────────────────
def main() -> int:
stdin = sys.stdin.buffer
stdout = sys.stdout.buffer
# Ready handshake fires BEFORE any heavy import — nothing above this line
# touches transformers/torch, so we make the 30 s spawn window even cold.
_send(stdout, {
"op": "ready",
"engine": "moss-tts-v15",
"sample_rate": MOSS_SAMPLE_RATE,
})
while True:
try:
msg = _recv(stdin)
except Exception as exc:
_send(stdout, {
"op": "error",
"stage": "recv",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
return 1
if msg is None:
return 0
op = msg.get("op") if isinstance(msg, dict) else None
try:
if op == "ping":
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
elif op == "synthesize":
_handle_synthesize(msg, stdout)
elif op == "shutdown":
return 0
else:
_send(stdout, {
"op": "error",
"stage": "dispatch",
"message": f"unknown op: {op!r}",
})
except Exception as exc:
# Per-op failure is recoverable — emit the error frame and stay
# alive so the parent can retry without paying the respawn +
# multi-GB model-load cost again.
_send(stdout, {
"op": "error",
"stage": op or "unknown",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
if __name__ == "__main__":
sys.exit(main())
+6 -67
View File
@@ -353,21 +353,6 @@ def _make_backend_class():
f"This clears the quarantine on the .app and its "
f"bundled binaries. See docs/install/macos.md."
)
# Execute bit (issue #437). A `git clone` / zip extract on POSIX
# can drop +x, which only surfaces at spawn time as
# "[Errno 13] Permission denied" — and the generic synth handler
# then mislabels it as out-of-memory. Self-heal here, AFTER the
# SHA check has confirmed this is the right file (so we never
# chmod a foreign binary). No-op on Windows.
if os.name == "posix" and not os.access(bin_path, os.X_OK):
try:
bin_path.chmod(bin_path.stat().st_mode | 0o111)
except OSError:
return False, (
f"GGUF binary {bin_path.name} isn't executable and "
f"couldn't be made so — run `chmod +x {bin_path}` "
f"and retry."
)
return True, "ready"
except Exception as exc:
return False, f"{type(exc).__name__}: {exc}"
@@ -508,12 +493,6 @@ def _make_backend_class():
* ``ref_audio`` (str/Path) speaker reference WAV for cloning.
* ``ref_text`` (str) transcript of ``ref_audio``.
* ``language`` (str) ISO code or omnivoice-tts lang label.
* ``instruct`` (str) style instruction.
* ``duration`` (float) target duration in seconds.
* ``seed`` (int) deterministic sampling seed.
* ``denoise`` (bool) omit denoise token when false.
* ``preprocess_prompt`` (bool) skip prompt preprocessing when false.
* ``chunk_duration`` / ``chunk_threshold`` (float) binary long-form controls.
"""
import soundfile as sf # local import keeps module import cheap
import torch
@@ -524,32 +503,15 @@ def _make_backend_class():
fd, out_str = tempfile.mkstemp(prefix="omnivoice-gguf-", suffix=".wav")
os.close(fd)
out_path = Path(out_str)
ref_text_path: Optional[Path] = None
try:
ref_text = kw.get("ref_text")
if kw.get("ref_audio") and ref_text:
text_fd, text_str = tempfile.mkstemp(
prefix="omnivoice-gguf-ref-", suffix=".txt"
)
os.close(text_fd)
ref_text_path = Path(text_str)
ref_text_path.write_text(str(ref_text), encoding="utf-8")
argv = self._build_argv(
base=base_path,
tokenizer=tok_path,
out_path=out_path,
ref_audio=kw.get("ref_audio"),
ref_text=str(ref_text_path) if ref_text_path else None,
ref_text=kw.get("ref_text"),
language=kw.get("language"),
instruct=kw.get("instruct"),
duration=kw.get("duration"),
seed=kw.get("seed"),
denoise=kw.get("denoise", True),
preprocess_prompt=kw.get("preprocess_prompt", True),
chunk_duration=kw.get("chunk_duration"),
chunk_threshold=kw.get("chunk_threshold"),
)
self._run_subprocess(argv, stdin_text=text)
wav, sr = sf.read(str(out_path))
@@ -558,11 +520,6 @@ def _make_backend_class():
out_path.unlink()
except OSError:
pass
if ref_text_path is not None:
try:
ref_text_path.unlink()
except OSError:
pass
# soundfile returns (n,) for mono or (n, c) for multichannel.
# OmniVoice/Higgs Audio v2 is mono → (n,). Wrap to (1, n).
@@ -587,13 +544,6 @@ def _make_backend_class():
ref_audio: Optional[str],
ref_text: Optional[str],
language: Optional[str],
instruct: Optional[str] = None,
duration: Optional[float] = None,
seed: Optional[int] = None,
denoise: bool = True,
preprocess_prompt: bool = True,
chunk_duration: Optional[float] = None,
chunk_threshold: Optional[float] = None,
) -> list[str]:
"""Compose argv from typed Path objects only (T-04-02)."""
argv: list[str] = [
@@ -605,20 +555,6 @@ def _make_backend_class():
lang = _iso_to_omnivoice_lang(language)
if lang:
argv += ["--lang", lang]
if instruct:
argv += ["--instruct", str(instruct)]
if duration is not None:
argv += ["--duration", str(float(duration))]
if seed is not None:
argv += ["--seed", str(int(seed))]
if denoise is False:
argv += ["--no-denoise"]
if preprocess_prompt is False:
argv += ["--no-preprocess-prompt"]
if chunk_duration is not None:
argv += ["--chunk-duration", str(float(chunk_duration))]
if chunk_threshold is not None:
argv += ["--chunk-threshold", str(float(chunk_threshold))]
if ref_audio:
# Two-stage validation (defense in depth):
# (a) Reject anything outside the project's voices /
@@ -652,8 +588,11 @@ def _make_backend_class():
)
argv += ["--ref-wav", str(ref_path)]
if ref_text:
# The C++ runtime expects a transcript file path.
# generate() creates this file in the system temp dir.
# ref_text is free-form text; pass via stdin would
# collide with the synthesis prompt, so the only safe
# channel is argv. The binary treats this as a quoted
# string at the OS layer (Popen escapes argv per
# platform); we don't pre-escape.
argv += ["--ref-text", str(ref_text)]
return argv
+11 -332
View File
@@ -9,16 +9,6 @@ _backend_dir = os.path.dirname(os.path.abspath(__file__))
if _backend_dir not in sys.path:
sys.path.insert(0, _backend_dir)
# #564: also make the project's OWN `omnivoice` package importable from source
# when the venv's editable install is missing/broken (interrupted/offline
# `uv sync`, antivirus-quarantined `_editable_impl_omnivoice.pth`, …). Without
# this the backend boots fine and only fails at the first model call with
# `No module named 'omnivoice'`. The bootstrap now gates on omnivoice being
# importable too (re-syncing to re-lay the editable install); this is the
# runtime safety net. See core/omnivoice_path.py for the full rationale.
from core.omnivoice_path import ensure_omnivoice_importable
ensure_omnivoice_importable(_backend_dir)
# Triton is unavailable on Windows — disable torch.compile / dynamo / inductor
# to prevent TritonMissing errors at inference time. Must be set before torch
# is imported (it is lazily imported in services/model_manager.py). Uses
@@ -53,13 +43,11 @@ try:
_project_env = os.path.join(os.path.dirname(_backend_dir), ".env")
if os.path.isfile(_project_env):
dotenv.load_dotenv(_project_env, override=False)
# Load the durable per-user config (the in-app Settings source of truth) so
# env vars set once survive Tauri/Finder launches that don't inherit a shell
# environment. This OVERRIDES launcher-injected defaults: the desktop app
# injects a stale OMNIVOICE_CACHE_DIR from its own config before startup, so
# without override a models dir changed in Settings was ignored forever (#480).
from core.user_env import load_into_environ as _load_user_env
_load_user_env()
# Also load the durable per-user config so env vars set once survive
# Tauri/Finder launches that don't inherit a shell environment.
_user_env = os.path.expanduser("~/.config/omnivoice/env")
if os.path.isfile(_user_env):
dotenv.load_dotenv(_user_env, override=False)
except ImportError:
pass
@@ -296,12 +284,7 @@ from fastapi.responses import JSONResponse, RedirectResponse, Response
from fastapi.staticfiles import StaticFiles
from fastapi.middleware.cors import CORSMiddleware
from starlette.datastructures import MutableHeaders
# Docs-only dependency: a venv created before scalar-fastapi entered the
# dependency set must still boot the backend (#307) — /docs degrades instead.
try:
from scalar_fastapi import get_scalar_api_reference
except ImportError:
get_scalar_api_reference = None
from scalar_fastapi import get_scalar_api_reference
import traceback
_crash_log_lock = threading.Lock()
@@ -330,22 +313,16 @@ from api.routers import (
setup,
gallery,
archetypes,
describe_voice,
community,
batch,
watermark,
events,
capture,
capture_ws,
dictation,
openai_compat,
tts_stream,
marketplace,
personas,
sonitranslate,
audiobook,
longform_jobs,
pronunciation, # Expressive-TTS Spec 01: user pronunciation dictionary
settings as settings_router, # Phase 1 AUTH-03: HF token save/clear/state
)
from utils import hf_progress
@@ -355,30 +332,6 @@ from utils import hf_progress
# the patched class, not the original.
hf_progress.install()
# Wire the overall download aggregator's byte sink onto the patched tqdm so
# parallel per-file updates feed one accurate overall bar (FDL-06).
try:
from utils import download_aggregator
download_aggregator.install()
except Exception:
pass
# Log the download-acceleration state once at startup (FDL-03) so a slow
# download report can be triaged from the logs without reproducing. Note: the
# app sets HF_HUB_DISABLE_XET=1 above by default (legacy LFS for byte progress),
# so xet_active is normally False even though hf_xet is installed.
try:
from api.routers.system import _fast_download_status as _fd_status
_fd = _fd_status()
_xet_ver = f" {_fd['xet_version']}" if _fd.get("xet_version") else ""
logging.getLogger("omnivoice.model").info(
"downloads: Xet %s (hf_xet%s installed=%s), high_perf=%s",
"ACTIVE" if _fd["xet_active"] else "disabled → legacy LFS",
_xet_ver, _fd["xet_installed"], _fd["high_performance"],
)
except Exception:
pass
def _env_flag(name: str, default: bool = False) -> bool:
value = os.environ.get(name)
@@ -387,116 +340,8 @@ def _env_flag(name: str, default: bool = False) -> bool:
return value.strip().lower() in {"1", "true", "yes", "on"}
def _capture_preload_delay_s() -> float:
"""Seconds after boot before the dictation (capture ASR) model warms.
Late enough that it never competes with startup I/O or the TTS preload;
overridable via OMNIVOICE_CAPTURE_PRELOAD_DELAY (mostly for tests)."""
raw = os.environ.get("OMNIVOICE_CAPTURE_PRELOAD_DELAY", "")
try:
v = float(raw)
if v >= 0:
return v
except (TypeError, ValueError):
pass
return 30.0
def _capture_preload_ram_ok(min_free_bytes: int = 4 * 1024**3) -> bool:
"""RAM guard for the dictation warm-up: skip below 4 GB free so the
background load never pushes a small machine into swap. If free memory
can't be measured, warm anyway (the load path has its own error handling)."""
try:
import psutil
return psutil.virtual_memory().available >= min_free_bytes
except Exception:
return True
def _mcp_start_timeout_s() -> float:
"""Seconds to wait for the MCP session manager to start before giving up
and serving without it (#632). Overridable via OMNIVOICE_MCP_START_TIMEOUT_S."""
raw = os.environ.get("OMNIVOICE_MCP_START_TIMEOUT_S", "")
try:
v = float(raw)
if v > 0:
return v
except (TypeError, ValueError):
pass
return 30.0
async def _serve_mcp(session_manager, ready: "asyncio.Event", stop: "asyncio.Event") -> None:
"""Own the MCP session manager's full enter→exit lifecycle in ONE task.
FastMCP's ``run()`` opens an anyio task group, and anyio requires the cancel
scope to be exited in the *same task* that entered it. So we must NOT enter
it via ``wait_for`` (which runs the enter in a throwaway sub-task) or on the
lifespan task and exit it elsewhere either raises "Attempted to exit cancel
scope in a different task". This coroutine enters and exits the context
itself: it signals ``ready`` once mounted, then idles until ``stop``.
"""
try:
async with session_manager.run():
ready.set()
await stop.wait()
except Exception as e:
logger.warning("MCP session manager stopped: %s", e)
finally:
ready.set() # never leave startup blocked on the readiness wait
async def _start_mcp_session_manager(session_manager, *, timeout: float):
"""Start MCP off the startup critical path; wait up to ``timeout`` for it to
signal ready. Returns ``(task, stop_event, mounted)``.
The MCP layer is best-effort and must never wedge backend startup. On some
platforms (observed: Apple-Silicon M1, #632) ``run()`` can *hang* on its
anyio task group; the old code awaited the enter before serving, so the hang
meant "Application startup complete" never fired and the whole backend was
unreachable with no error. Now the enter lives in its own task and we only
*optionally* wait on a ready signal a hang becomes a logged warning + a
backend that serves normally without MCP.
"""
stop = asyncio.Event()
if session_manager is None:
return None, stop, False
ready = asyncio.Event()
task = asyncio.create_task(_serve_mcp(session_manager, ready, stop))
try:
await asyncio.wait_for(ready.wait(), timeout=timeout)
mounted = not task.done() # ready is also set on failure → not mounted
except asyncio.TimeoutError:
logger.warning(
"MCP session manager did not signal ready within %.0fs (#632); "
"serving without waiting. Set OMNIVOICE_MCP_START_TIMEOUT_S to adjust.",
timeout,
)
mounted = False
return task, stop, mounted
@asynccontextmanager
async def lifespan(app: FastAPI):
# Startup watchdog (#632): a silent hang during startup (e.g. a model-load /
# MCP deadlock on some platforms) means "Application startup complete" never
# logs and the app sits forever with no error. If startup hasn't finished
# within the window, dump every thread's stack to stderr (→ backend_err.log)
# so the hang point is captured instead of invisible. Cancelled the instant
# startup completes, so a normal (even slow-download) boot never trips it.
# Tune with OMNIVOICE_STARTUP_WATCHDOG_S (seconds; 0 disables). Best-effort —
# never let the diagnostic itself break startup.
_watchdog_armed = False
try:
import faulthandler
_wd = float(os.environ.get("OMNIVOICE_STARTUP_WATCHDOG_S", "300"))
if _wd > 0 and hasattr(faulthandler, "dump_traceback_later"):
faulthandler.dump_traceback_later(_wd, repeat=False, exit=False)
_watchdog_armed = True
logger.info("Startup watchdog armed: thread dump if startup exceeds %.0fs (#632).", _wd)
except Exception:
pass
init_db()
# Network sharing is loopback-only by default; the PIN middleware stays
# inert until enable() sets a PIN. Seed the (disabled) state so the
@@ -545,19 +390,11 @@ async def lifespan(app: FastAPI):
worker_task = asyncio.create_task(task_manager.worker())
# Warm the TTS model in the background so first /generate is instant.
preload_task = asyncio.create_task(preload_model())
# Dictation v2: the capture ASR warms in the background BY DEFAULT — a
# deferred (~30s post-boot) load off the event loop, so startup stays
# lean and the first dictation is instant instead of a cold model load.
# OMNIVOICE_PRELOAD_CAPTURE_ASR=0 opts out; the warm-up is also skipped
# under 4 GB free RAM (checked at warm time, not boot time).
if _env_flag("OMNIVOICE_PRELOAD_CAPTURE_ASR", default=True):
# Capture ASR is useful to keep warm, but it is another large model in
# unified memory on Apple Silicon. Keep launch lean by default; users who
# prefer instant dictation can opt in with OMNIVOICE_PRELOAD_CAPTURE_ASR=1.
if _env_flag("OMNIVOICE_PRELOAD_CAPTURE_ASR"):
async def _preload_capture_asr():
await asyncio.sleep(_capture_preload_delay_s())
if not _capture_preload_ram_ok():
logger.info(
"Capture ASR preload skipped: <4GB free RAM; "
"dictation ASR will load on first use.")
return
loading_detail = None
prev_loading_detail = None
try:
@@ -585,39 +422,9 @@ async def lifespan(app: FastAPI):
capture_preload_task = asyncio.create_task(_preload_capture_asr())
else:
logger.info("Capture ASR preload disabled; dictation ASR will load on first use.")
# ── MCP session manager (Wave 2.2) ────────────────────────────────────
# FastMCP's Streamable-HTTP transport needs its session manager running for
# the lifetime of the app. Run it in its OWN task that owns the full
# enter→exit lifecycle (anyio task-affinity, see _serve_mcp) and only wait,
# with a timeout, for it to signal ready — so a hang on its anyio group
# (observed on M1, #632) can never wedge "Application startup complete".
_sm = getattr(app.state, "mcp_session_manager", None)
mcp_task, mcp_stop, mcp_mounted = await _start_mcp_session_manager(
_sm, timeout=_mcp_start_timeout_s()
)
if mcp_mounted:
logger.info("MCP server mounted at /mcp")
# Startup finished — disarm the hang watchdog before serving (#632).
if _watchdog_armed:
try:
import faulthandler
faulthandler.cancel_dump_traceback_later()
except Exception:
pass
yield
# ── Graceful shutdown (SIGTERM from Tauri, Ctrl+C, etc.) ────────────
logger.info("Shutdown: cleaning up…")
# Stop MCP first — signal its task to exit its own anyio context (correct
# task-affinity), then bound the wait so a wedged manager can't hang exit.
mcp_stop.set()
if mcp_task is not None:
try:
await asyncio.wait_for(mcp_task, timeout=5.0)
except (asyncio.TimeoutError, asyncio.CancelledError):
pass
except Exception:
pass
idle_task.cancel()
worker_task.cancel()
# Wait for tasks to finish their current iteration
@@ -664,14 +471,6 @@ app = FastAPI(
@app.get("/docs", include_in_schema=False)
async def scalar_docs():
"""Interactive API documentation powered by Scalar."""
if get_scalar_api_reference is None:
return JSONResponse(
status_code=503,
content={
"detail": "API docs unavailable: scalar-fastapi is not installed "
"in the backend environment (#307)."
},
)
return get_scalar_api_reference(
openapi_url=app.openapi_url,
title=app.title,
@@ -715,16 +514,8 @@ async def global_exception_handler(request: Request, exc: Exception):
headers["Access-Control-Allow-Origin"] = origin
headers["Access-Control-Allow-Credentials"] = "true"
headers["Vary"] = "Origin"
# #874: a model download that failed because the CONFIGURED Hugging Face
# mirror (HF_ENDPOINT) is unreachable used to leak the raw transformers
# message ("We couldn't connect to 'https://hf-mirror.com' …") as the 500
# detail with no next step. Appending the shared mirror hint HERE covers
# every route that can leak a model-load/download error (generate, dub,
# archetypes, …), not just TTS generate. append_hf_mirror_hint is a no-op
# for every other error and never raises.
from core.failure import append_hf_mirror_hint
return JSONResponse(
{"detail": append_hf_mirror_hint(str(exc)), "error_class": _entry.get("error_class")},
{"detail": str(exc), "error_class": _entry.get("error_class")},
status_code=500,
headers=headers,
)
@@ -788,68 +579,6 @@ class NetworkAccessMiddleware:
return await self.app(scope, receive, send)
class BearerKeyMiddleware:
"""When OMNIVOICE_API_KEY is set, non-loopback clients must present it on
every HTTP + WebSocket request: ``Authorization: Bearer <key>``,
``?api_key=<key>`` (browser WebSockets cannot set headers), or the
``ov_key`` cookie (set on the first successful HTTP auth). Loopback
always bypasses the desktop default is unchanged and the SPA shell
paths stay reachable so a remote UI can load and show what's wrong.
Inert when the env var is unset (the default). Pure ASGI for the same
no-buffering reason as NetworkAccessMiddleware above. Plain-HTTP caveat
is documented in docs/remote-gpu.md: the key is sniffable outside a
WireGuard (Tailscale) or TLS (tailscale serve) transport.
"""
def __init__(self, app):
self.app = app
async def __call__(self, scope, receive, send):
if scope["type"] not in ("http", "websocket"):
return await self.app(scope, receive, send)
key = os.environ.get("OMNIVOICE_API_KEY") or ""
if not key:
return await self.app(scope, receive, send)
client = scope["client"][0] if scope.get("client") else None
if client in _LOOPBACK_CLIENTS:
return await self.app(scope, receive, send)
path = scope.get("path", "")
if scope["type"] == "http" and (
path in _SHELL_PATHS or path.startswith("/assets/") or path.startswith("/favicon")
):
return await self.app(scope, receive, send)
from starlette.requests import HTTPConnection
conn = HTTPConnection(scope)
auth = conn.headers.get("authorization", "")
supplied = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
if not supplied:
supplied = conn.query_params.get("api_key") or conn.cookies.get("ov_key") or ""
if not secrets.compare_digest(supplied, key):
if scope["type"] == "websocket":
# Reject the handshake; 1008 = policy violation.
await receive() # consume websocket.connect
await send({"type": "websocket.close", "code": 1008})
return
resp = JSONResponse({"detail": "API key required"}, status_code=401)
return await resp(scope, receive, send)
if scope["type"] == "http" and conn.cookies.get("ov_key") != key:
async def send_with_cookie(message):
if message["type"] == "http.response.start":
headers = MutableHeaders(scope=message)
headers.append(
"set-cookie", f"ov_key={key}; Path=/; SameSite=Lax"
)
await send(message)
return await self.app(scope, receive, send_with_cookie)
return await self.app(scope, receive, send)
# UI dev-server port — single-sourced from OMNIVOICE_UI_PORT so a user who
# moves the Vite dev server off 3901 still gets a matching CORS allow-list.
def _ui_port() -> int:
@@ -881,29 +610,6 @@ app.add_middleware(
# applied even to the 401 PIN-required responses). Inert unless a PIN is set.
app.add_middleware(NetworkAccessMiddleware)
# Remote-backend bearer gate (parity program Wave 2.3 / §R2). Inert unless
# OMNIVOICE_API_KEY is set. Distinct from the PIN gate above: the PIN guards
# casual LAN-share guests for one session; the API key is the durable
# credential for running this backend remotely (Tailscale / Docker GPU box).
# Covers WebSockets too — the PIN gate never did, because every WS endpoint
# carried its own loopback guard; remote mode is exactly the case where a
# keyed non-loopback client must reach them.
app.add_middleware(BearerKeyMiddleware)
# Register canonical audio MIME types before any StaticFiles mount.
# Python's `mimetypes.guess_type()` returns `audio/x-wav` for `.wav` and
# `audio/x-flac` for `.flac` on most platforms — these are vendor-experimental
# (x- prefix, never IANA-registered). macOS Chrome/Safari MIME-sniff leniently
# via CoreAudio so playback works there, but Linux Chrome/Firefox (FFmpeg) and
# Android Chrome (ExoPlayer) strictly honor the declared type and treat the
# x- variants as download-only — manifesting as the play button silently
# doing nothing in the browser app while working in the Tauri desktop shell.
# `audio/wav` / `audio/flac` are the IANA-canonical types.
# Ref: https://www.iana.org/assignments/media-types/media-types.xhtml#audio
import mimetypes as _mimetypes
_mimetypes.add_type("audio/wav", ".wav")
_mimetypes.add_type("audio/flac", ".flac")
app.mount("/audio", StaticFiles(directory=OUTPUTS_DIR), name="audio")
app.mount("/voice_audio", StaticFiles(directory=VOICES_DIR), name="voice_audio")
@@ -946,44 +652,17 @@ app.include_router(stories.router)
app.include_router(setup.router)
app.include_router(gallery.router)
app.include_router(archetypes.router)
app.include_router(describe_voice.router) # issue #317: free-text voice design
app.include_router(community.router)
app.include_router(batch.router)
app.include_router(watermark.router)
app.include_router(events.router)
app.include_router(capture.router)
app.include_router(capture_ws.router)
app.include_router(dictation.router)
app.include_router(openai_compat.router)
app.include_router(tts_stream.router)
app.include_router(marketplace.router)
app.include_router(personas.router)
app.include_router(sonitranslate.router)
app.include_router(audiobook.router)
app.include_router(longform_jobs.router)
app.include_router(pronunciation.router) # Expressive-TTS Spec 01: pronunciation dictionary
app.include_router(settings_router.router) # Phase 1 AUTH-03 endpoints
from api.routers import mcp_bindings as _mcp_bindings_router # noqa: E402
app.include_router(_mcp_bindings_router.router) # Wave 2.2 per-agent voice bindings
# ── Mount the MCP server (Wave 2.2) ───────────────────────────────────────
# FastMCP's Streamable-HTTP app is sub-mounted at /mcp; its session manager is
# stashed on app.state for the lifespan above to run. Opt-out via
# OMNIVOICE_MCP_DISABLE=1; best-effort so a missing mcp package or a build
# without it never breaks startup.
if os.environ.get("OMNIVOICE_MCP_DISABLE", "").strip().lower() not in ("1", "true", "yes", "on"):
try:
from mcp_server import create_mcp_server
_mcp = create_mcp_server()
_mcp_app = _mcp.streamable_http_app()
app.state.mcp_session_manager = _mcp.session_manager
app.mount("/mcp", _mcp_app)
logging.getLogger("omnivoice.api").info("MCP app mounted at /mcp")
except Exception as _mcp_err: # noqa: BLE001
logging.getLogger("omnivoice.api").info(
"MCP server not mounted (%s); /mcp disabled.", _mcp_err
)
frontend_path = os.path.join(os.path.dirname(__file__), "..", "frontend", "dist")
if os.path.exists(frontend_path):
+1 -64
View File
@@ -53,14 +53,6 @@ def create_mcp_server():
"voice design, and video dubbing in 646 languages."
),
)
# Serve the Streamable-HTTP transport at the app root so mounting the whole
# app at "/mcp" on the main FastAPI yields the endpoint at "/mcp". FastMCP's
# default path is "/mcp", which would double-prefix to "/mcp/mcp" when
# sub-mounted. Harmless for the standalone CLI run() path.
try:
mcp.settings.streamable_http_path = "/"
except Exception:
pass
# ── Helpers ─────────────────────────────────────────────────────────
@@ -83,21 +75,6 @@ def create_mcp_server():
# ── Tools ───────────────────────────────────────────────────────────
def _current_client_id() -> str | None:
"""The X-OmniVoice-Client-Id of the calling MCP client, if any.
FastMCP exposes the HTTP request via its request context on the
Streamable-HTTP transport; stdio clients (and any version where the
accessor differs) simply resolve to None and fall back to the
global default voice."""
try:
req = mcp.get_context().request_context.request
if req is not None:
return req.headers.get("x-omnivoice-client-id")
except Exception:
pass
return None
@mcp.tool()
async def generate_speech(
text: str,
@@ -112,8 +89,7 @@ def create_mcp_server():
Args:
text: The text to synthesize into speech.
language: Target language (ISO code or 'Auto'). 646 languages supported.
profile_id: ID of a saved voice profile to clone. Omit to use this
agent's bound voice (Settings → MCP), else the global default.
profile_id: ID of a saved voice profile to clone. Omit for voice design mode.
instruct: Style instruction (e.g. 'whisper', 'excited', 'narrator').
speed: Speech speed multiplier (0.52.0, default 1.0).
steps: Diffusion steps (8=fast/draft, 16=balanced, 32=quality).
@@ -122,17 +98,6 @@ def create_mcp_server():
JSON with audio_id, generation_time, audio_duration, and
base64-encoded WAV data.
"""
# Per-agent voice binding (Wave 2.2): explicit arg wins; otherwise
# resolve this client's bound profile, then the global default.
client_id = _current_client_id()
try:
from services import mcp_bindings
resolved = mcp_bindings.resolve_voice(client_id, profile_id)
profile_id = resolved.get("profile_id")
mcp_bindings.touch_last_seen(client_id) if client_id else None
except Exception:
pass # binding layer unavailable — use whatever was passed
form = {
"text": text,
"language": language,
@@ -194,34 +159,6 @@ def create_mcp_server():
'],"note":"Pass any ISO 639 code or set language=Auto for detection."}'
)
@mcp.tool()
async def transcribe(audio_base64: str, language: str | None = None) -> str:
"""Transcribe spoken audio to text.
Args:
audio_base64: Base64-encoded audio bytes (wav/mp3/webm/m4a).
language: Optional language hint; omit for auto-detect.
Returns:
JSON with the recognized text, language, and duration.
"""
try:
raw = base64.b64decode(audio_base64, validate=True)
except Exception:
return '{"error":"audio_base64 is not valid base64"}'
# 200 MB cap — same spirit as voicebox's transcribe gate. Keeps a
# buggy/hostile agent from posting an unbounded blob.
if len(raw) > 200 * 1024 * 1024:
return '{"error":"audio exceeds 200 MB limit"}'
data = {}
if language:
data["language"] = language
r = await _api_post_form(
"/transcribe", data=data,
files={"audio": ("audio.wav", raw, "application/octet-stream")},
)
return str(r.json())
@mcp.tool()
async def check_health() -> str:
"""Check if the OmniVoice backend is running and what GPU device is active."""
-1
View File
@@ -1 +0,0 @@
"""omnivoice-mcp — stdio MCP shim for clients that only speak stdio."""
-176
View File
@@ -1,176 +0,0 @@
"""omnivoice-mcp — stdio ↔ Streamable-HTTP MCP proxy (Wave 2.2).
Adapted from voicebox (https://github.com/jamiepine/voicebox), MIT License,
Copyright (c) voicebox contributors.
Some MCP clients only speak stdio. They spawn this binary; we pipe each
JSON-RPC message to ``http://127.0.0.1:<port>/mcp/`` (the FastMCP app mounted
on the running OmniVoice backend) and stream the server's response back.
Environment variables:
OMNIVOICE_PORT backend port (default 3900).
OMNIVOICE_HOST host (default 127.0.0.1).
OMNIVOICE_CLIENT_ID forwarded as X-OmniVoice-Client-Id on every request
(drives per-agent voice binding).
Stdout is JSON-RPC only. Diagnostics go to stderr.
Exit 0 on clean EOF, 1 on transport error, 2 if the backend never answers.
Usage in an MCP client config (stdio):
command: python
args: ["-m", "backend.mcp_shim"]
env: { OMNIVOICE_CLIENT_ID: "claude-code" }
"""
from __future__ import annotations
import asyncio
import json
import os
import sys
from typing import Any
import httpx
CLIENT_ID_HEADER = "X-OmniVoice-Client-Id"
SESSION_HEADER = "mcp-session-id"
HEALTH_TIMEOUT_S = 30.0
DEFAULT_PORT = 3900
def _err(msg: str) -> None:
print(f"omnivoice-mcp: {msg}", file=sys.stderr, flush=True)
def _base_url() -> tuple[str, str]:
host = os.environ.get("OMNIVOICE_HOST", "127.0.0.1")
port = int(os.environ.get("OMNIVOICE_PORT", str(DEFAULT_PORT)))
return f"http://{host}:{port}/mcp/", f"http://{host}:{port}/health"
async def _wait_for_backend(client: httpx.AsyncClient, health_url: str) -> bool:
loop = asyncio.get_running_loop()
deadline = loop.time() + HEALTH_TIMEOUT_S
while loop.time() < deadline:
try:
r = await client.get(health_url, timeout=2.0)
if r.status_code == 200:
return True
except Exception:
pass
await asyncio.sleep(0.5)
return False
async def _read_stdin_line() -> str | None:
loop = asyncio.get_running_loop()
line = await loop.run_in_executor(None, sys.stdin.readline)
return line or None
def _write_stdout(obj: Any) -> None:
sys.stdout.write(json.dumps(obj, separators=(",", ":")))
sys.stdout.write("\n")
sys.stdout.flush()
async def _handle_request(
client: httpx.AsyncClient,
url: str,
raw: str,
headers: dict[str, str],
session_id: list[str | None],
) -> None:
try:
message = json.loads(raw)
except json.JSONDecodeError as exc:
_err(f"invalid JSON on stdin: {exc}")
return
req_headers = {
"Content-Type": "application/json",
"Accept": "application/json, text/event-stream",
**headers,
}
if session_id[0]:
req_headers[SESSION_HEADER] = session_id[0]
is_notification = isinstance(message, dict) and "id" not in message
async with client.stream("POST", url, headers=req_headers, content=raw.encode("utf-8")) as response:
if session_id[0] is None:
sid = response.headers.get(SESSION_HEADER)
if sid:
session_id[0] = sid
if response.status_code == 202:
return # notification acknowledged
if response.status_code >= 400:
body = await response.aread()
_err(f"server {response.status_code}: {body.decode('utf-8', errors='replace')[:400]}")
if is_notification:
return
_write_stdout({
"jsonrpc": "2.0",
"id": message.get("id"),
"error": {"code": -32000, "message": f"OmniVoice MCP proxy got HTTP {response.status_code}"},
})
return
ctype = response.headers.get("content-type", "")
if "text/event-stream" in ctype:
async for line in response.aiter_lines():
if line.startswith("data:"):
payload = line[5:].strip()
if not payload:
continue
try:
_write_stdout(json.loads(payload))
except json.JSONDecodeError:
_err(f"malformed SSE payload: {payload[:200]}")
else:
body = await response.aread()
try:
_write_stdout(json.loads(body))
except json.JSONDecodeError:
_err(f"non-JSON response ({ctype}): {body.decode('utf-8', errors='replace')[:200]}")
async def _run() -> int:
url, health_url = _base_url()
forward_headers: dict[str, str] = {}
client_id = os.environ.get("OMNIVOICE_CLIENT_ID")
if client_id:
forward_headers[CLIENT_ID_HEADER] = client_id
session_id: list[str | None] = [None]
async with httpx.AsyncClient(timeout=httpx.Timeout(300.0)) as client:
if not await _wait_for_backend(client, health_url):
_err(f"timed out waiting for OmniVoice at {health_url} — is the app running?")
return 2
try:
while True:
line = await _read_stdin_line()
if line is None:
return 0
line = line.strip()
if not line:
continue
await _handle_request(client, url, line, forward_headers, session_id)
except (KeyboardInterrupt, SystemExit):
return 0
except Exception as exc:
_err(f"proxy failed: {exc!r}")
return 1
def main() -> int:
try:
return asyncio.run(_run())
except KeyboardInterrupt:
return 0
if __name__ == "__main__":
sys.exit(main())
+1 -7
View File
@@ -17,13 +17,7 @@ from core.config import DB_PATH # noqa: E402 — backend/ is on sys.path via al
config = context.config
if config.config_file_name is not None:
# `disable_existing_loggers=False` is deliberate: this env runs *inside* the
# live app (startup `alembic upgrade head`), so the default (True) would
# disable every already-created application logger — e.g. silence
# `omnivoice.db.backup`'s "Skipping pre-migration DB backup" line and the
# rest of the app's logging for the remainder of the process. A migration
# must never mute the app (or leak that mute across a test session).
fileConfig(config.config_file_name, disable_existing_loggers=False)
fileConfig(config.config_file_name)
# SQLite file URL. Honour an externally-set URL (tests pass one via
# `cfg.set_main_option("sqlalchemy.url", ...)` to point at a fixture DB),
@@ -1,58 +0,0 @@
"""Parity program Wave 0.2: consent-locked voice profiles
Revision ID: 0003_voice_profile_consent
Revises: 0002_voice_profile_demo_fields
Create Date: 2026-06-12 00:00:00.000000
Adds four additive columns to ``voice_profiles`` backing the
``verified_own_voice`` consent lock (docs/competitive-analysis.md Action 22 /
parity program Wave 0.2). A profile becomes "verified" when its owner records
a spoken consent statement; agentic features and gallery sharing will require
the flag plain local synthesis never does.
* ``verified_own_voice INTEGER DEFAULT 0`` the consent lock itself.
* ``consent_text TEXT DEFAULT ''`` the statement that was read aloud.
* ``consent_audio_path TEXT DEFAULT ''`` filename of the recorded
statement in VOICES_DIR (kept as provenance, deletable via revoke).
* ``consent_recorded_at REAL DEFAULT NULL`` UNIX timestamp.
Behavior mirrors 0002: ``_has_column`` PRAGMA guards make upgrade a no-op on
fresh installs (where _BASE_SCHEMA already has the columns), satisfying the
"Backward-compatible project data" constraint; downgrade drops the columns
(SQLite >= 3.35).
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0003_voice_profile_consent"
down_revision: Union[str, None] = "0002_voice_profile_demo_fields"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
_COLUMNS = (
("verified_own_voice", sa.Column("verified_own_voice", sa.Integer(), nullable=False, server_default="0")),
("consent_text", sa.Column("consent_text", sa.Text(), nullable=False, server_default="")),
("consent_audio_path", sa.Column("consent_audio_path", sa.Text(), nullable=False, server_default="")),
("consent_recorded_at", sa.Column("consent_recorded_at", sa.Float(), nullable=True)),
)
def _has_column(table: str, column: str) -> bool:
bind = op.get_bind()
rows = bind.execute(sa.text(f"PRAGMA table_info({table})")).fetchall()
return any(r[1] == column for r in rows)
def upgrade() -> None:
for name, column in _COLUMNS:
if not _has_column("voice_profiles", name):
op.add_column("voice_profiles", column)
def downgrade() -> None:
for name, _ in reversed(_COLUMNS):
if _has_column("voice_profiles", name):
op.drop_column("voice_profiles", name)
@@ -1,60 +0,0 @@
"""Parity program Wave 2.2: per-agent MCP voice bindings
Revision ID: 0004_mcp_client_bindings
Revises: 0003_voice_profile_consent
Create Date: 2026-06-12 00:00:00.000000
Adds the ``mcp_client_bindings`` table backing per-agent voice binding
(docs/competitive-analysis.md Spec 2): each MCP client (identified by the
``X-OmniVoice-Client-Id`` header it sends) can be bound to a default voice
profile / engine, so "Claude Code speaks in Morgan, Cursor in Scarlett".
* ``client_id`` TEXT PRIMARY KEY the agent's stable id.
* ``label`` TEXT human label shown in Settings.
* ``profile_id`` TEXT voice profile to speak in (nullable FK-by-convention).
* ``default_engine`` TEXT engine override (nullable).
* ``last_seen_at`` REAL updated when the client calls a tool.
* ``created_at`` REAL.
Additive + idempotent (guarded by sqlite_master), matching 0002/0003, so
re-running on a fresh-install DB where _BASE_SCHEMA already created it is a
no-op (Backward-compatible project data constraint).
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
revision: str = "0004_mcp_client_bindings"
down_revision: Union[str, None] = "0003_voice_profile_consent"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def _has_table(name: str) -> bool:
bind = op.get_bind()
row = bind.execute(
sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name=:n"),
{"n": name},
).fetchone()
return row is not None
def upgrade() -> None:
if _has_table("mcp_client_bindings"):
return
op.create_table(
"mcp_client_bindings",
sa.Column("client_id", sa.Text(), primary_key=True),
sa.Column("label", sa.Text(), nullable=False, server_default=""),
sa.Column("profile_id", sa.Text(), nullable=True),
sa.Column("default_engine", sa.Text(), nullable=True),
sa.Column("last_seen_at", sa.Float(), nullable=True),
sa.Column("created_at", sa.Float(), nullable=True),
)
def downgrade() -> None:
if _has_table("mcp_client_bindings"):
op.drop_table("mcp_client_bindings")

Some files were not shown because too many files have changed in this diff Show More