Compare commits

..
Author SHA1 Message Date
debpalashandClaude Opus 4.8 362dca9ae1 docs(diagnostics): self-diagnosis section in troubleshooting + README pointer
Settings > About self-check / --diagnose / --deep / diagnostic bundle are
now the documented first step before the per-error entries — and the
support team's first ask on every issue.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:43:03 +05:30
debpalashandClaude Opus 4.8 bbac66ef76 feat(diagnostics): breadcrumbs, deep self-check, report sweep, issue search
Final slice of the bug-tracking work.

- toastErrorWithReport adopted at the high-traffic failure sites: TTS
  generation, dub upload/ingest/transcribe, engine install, engines-matrix
  load, voice profile save/delete/test, batch enqueue/cancel/delete.
  Validation toasts and cancellations stay plain on purpose.
- utils/breadcrumbs.js: local-only ring of the last 20 action names
  (closed-set names only — never content or paths), embedded as a
  'Recent actions' section in the prefilled report. Instrumented: view
  changes, generate, dub pipeline, export, engine switch.
- deep self-check: /system/diagnose?deep=true and --diagnose --deep load
  the active engine and synthesize a short utterance (num_step=4) —
  catches 'installed but broken'. 180s time-box, skips during model load,
  scrubbed failure detail. Verified live: cold-loaded omnivoice and
  produced 2.2s of audio in 43.9s on CUDA.
- 'Search similar issues' action on the ErrorBoundary: scrubbed,
  noise-stripped GitHub issue search URL — dedupe before filing.
- bug_report.md template now points at the diagnostic bundle and the
  --diagnose CLI so manual reports arrive with the same evidence.

Tests: pytest 107 passed (4 new deep-check tests, CJK gate green);
vitest 218 passed (breadcrumbs + issue-search suites); vite build green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:43:03 +05:30
debpalashandClaude Opus 4.8 3d74611be8 feat(diagnostics): error journal, diagnostic bundle, crash notice, global handlers
Second slice of the bug-tracking work — still zero outbound paths beyond
the user's own browser/file manager.

- core/error_journal.py: deduped ring of recent unhandled backend errors
  (fingerprint counts, error_class triage: GPU_OOM, HF_AUTH_FAILED,
  PYANNOTE_LICENSE_REQUIRED, DISK_FULL, FFMPEG_MISSING, NETWORK_ERROR),
  scrubbed, JSONL-persisted so the error that killed the last run survives
  restart. Wired into the global exception handler; 500 bodies now carry
  error_class; GET /system/errors/recent.
- core/diagnostic_bundle.py + POST /system/diagnostic-bundle + Settings >
  About 'Save diagnostic bundle': zip of self-check report, error journal,
  scrubbed log tails — drag onto a GitHub issue; bypasses the ~8k
  prefill-URL ceiling.
- crash-on-next-launch: /system/notifications flags a crash logged before
  this session started (size vs acked-size in prefs, mtime vs process
  start); POST /system/crash/ack; LogsFooter acks on action click.
- utils/globalErrorHandlers.js: uncaught errors + unhandled rejections get
  a throttled, noise-filtered 'Report this bug' toast.
- sidecar log parity fix: _tauri_log_candidates() now lists the Rust
  sidecar's backend.log/backend_err.log on Linux (XDG state dir) and
  Windows (LOCALAPPDATA) — sidecar crashes were only visible on macOS.

Tests: +19 pytest (journal, bundle); suite at 102 passed. Vitest 124
passed; vite build green. Live-verified: journal recorded and classified
a real HF 401 from the test run (HF_AUTH_FAILED, paths scrubbed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:43:03 +05:30
debpalashandClaude Opus 4.8 ee08f56e1b feat(diagnostics): local self-check + scrubbed bug-report pipeline
Closes the gap between 'something broke' and 'a useful GitHub issue
exists' — entirely within the local-first constraint: the only outbound
path remains the user's own browser opening a prefilled issues/new URL.

Backend:
- core/scrub.py: privacy scrubber for anything leaving the machine —
  env-var secret values (*TOKEN*|*KEY*|*SECRET*|*PASSWORD*), credential
  shapes (hf_/ghp_/github_pat_/sk-), home dirs on all three OSes
- core/diagnose.py: 9-check self-check (device+GPU, ffmpeg, HF token,
  disk, data-dir writability, RAM, engine registry, hub reachability),
  pre-scrubbed, ASCII-safe output
- GET /system/diagnose + 'python main.py --diagnose' (exit 0/1)
- /system/info: hardware inventory (os_version, cpu_model, cpu_count,
  ram_total_gb, gpu_name, vram_total_gb, disk_free_gb), cached statics

Frontend:
- utils/bugReport.js: single source for the prefilled-URL builder —
  scrubText twin, hardware context capture, scrubbed error+stack embed,
  URL-length cap; ReportBugButton refactored onto it
- ErrorBoundary 'Report this bug' action with the error attached
- utils/errorToast.jsx toastErrorWithReport(); wired into export toasts
- Settings > About 'Run self-check' with per-check status badges

Tests: 27 pytest (scrub, diagnose) + 15 vitest (bugReport); existing
suites green; verified live (--diagnose, TestClient, vite build).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-07 21:43:03 +05:30
1041 changed files with 28619 additions and 157927 deletions
-32
View File
@@ -1,32 +0,0 @@
# RTK - Rust Token Killer (Google Antigravity)
**Usage**: Token-optimized CLI proxy for shell commands.
## Rule
Always prefix shell commands with `rtk` to minimize token consumption.
Examples:
```bash
rtk git status
rtk cargo test
rtk ls src/
rtk grep "pattern" src/
rtk find "*.rs" .
rtk docker ps
rtk gh pr list
```
## Meta Commands
```bash
rtk gain # Show token savings
rtk gain --history # Command history with savings
rtk discover # Find missed RTK opportunities
rtk proxy <cmd> # Run raw (no filtering, for debugging)
```
## Why
RTK filters and compresses command output before it reaches the LLM context, saving 60-90% tokens on common operations. Always use `rtk <cmd>` instead of raw commands.
+3 -119
View File
@@ -7,26 +7,12 @@
language: "en-US"
early_access: false
# The review voice: a panel of senior domain experts, not a linter.
tone_instructions: >-
Review as a panel of principal engineers: ML inference, audio DSP, desktop
systems, product polish. Cite exact lines, name the failure mode, give the
concrete fix. No filler praise; raise nits only when they change a decision.
reviews:
# "chill" keeps the bot from blocking merges — it comments, it does not gate.
# Hard gating lives in CI (security.yml) and the constitution's human bar.
profile: chill
request_changes_workflow: false
high_level_summary: true
# Every walkthrough gets a visual: mermaid sequence diagrams for the
# mechanics, plus (via the summary instructions) an ASCII before/after
# sketch when the PR touches UI — so each PR is reviewable at a glance.
sequence_diagrams: true
high_level_summary_instructions: >-
If the PR changes UI (JSX/TSX/CSS/Tauri windows), include a compact ASCII
before/after sketch of the affected layout or component. If it changes
behavior, include a short mermaid flowchart of the new mechanism.
review_status: true
poem: false
@@ -53,8 +39,7 @@ reviews:
- "!**/*.onnx"
- "!tests/fixtures/**"
# One expert lens per subsystem — encode what a passionate senior in each
# domain would actually check, beyond what linters and CI already gate.
# Encode the project's hard constraints so the bot reviews against them.
path_instructions:
- path: "**/*.{py,rs,js,jsx,ts,tsx}"
instructions: >-
@@ -63,119 +48,18 @@ reviews:
model download, or an explicitly opt-in endpoint. Flag any code that
persists or logs values matching *TOKEN*/*KEY*/*SECRET* or absolute user
home paths (/Users/<name>/, C:\\Users\\<name>\\).
- path: "backend/services/**/*.py"
instructions: >-
Review as an ML-inference/audio engineer. Check: thread-safety of model
and cache state across the GPU worker pool; device/dtype assumptions
that break on one of CUDA/MPS/ROCm/CPU; VRAM lifecycle (load/unload,
leaks on the error path); sample-rate, channel-count and tensor-shape
assumptions at engine boundaries; blocking calls inside async paths;
model download/cache behavior when offline. Engine code must stay
backward-compatible with already-installed on-disk model state.
- path: "backend/**/*.py"
instructions: >-
Default features must behave identically on macOS, Windows and Linux.
Platform-specific implementation is allowed, but a divergent user-visible
default is a P0 bug — flag it and suggest an opt-in (Settings/env/flag).
Any DB schema change must go through an alembic migration with an upgrade
path; flag direct schema edits. The backend serves loopback HTTP: treat
every query/path/form param as hostile (path traversal, log injection,
CSRF from a browser tab), and never route user-chosen filesystem
destinations through HTTP — that authorization belongs in the Tauri
process.
- path: "frontend/src/**/*.{js,jsx,ts,tsx}"
instructions: >-
Review as a product-minded senior frontend engineer. Check: stale state
and races (async results landing after unmount or after newer requests);
every user-visible failure has an actionable, non-technical error
message; loading/disabled states during long operations. Every new
user-facing string must be an i18n t('...') key present in ALL 21
frontend/src/i18n/locales/*.json files — flag hardcoded UI strings and
keys missing from any locale.
- path: "frontend/src-tauri/**/*.rs"
instructions: >-
Review as a desktop-systems engineer. Check: every #[tauri::command] is
callable from the webview — validate inputs and scope filesystem/process
access accordingly; window and webview lifecycle on all three OSes;
child-process spawn/exit-code/stderr handling; no unwrap/expect on
user-controlled input; platform cfg blocks keep user-visible defaults
identical across macOS/Windows/Linux.
- path: "tests/**/*.py"
instructions: >-
Review as a test-infrastructure engineer. Check: the test would fail
before the fix and pass after (no tautologies); no sleeps as
synchronization; no module-level imports of app modules that go stale
under sys.modules pollution (resolve at run time); TestClient instances
are function-scoped and not lifespan-bound unless the test needs it;
new functional CJK is allowlisted in tests/test_no_hardcoded_cjk.py with
a justification.
path; flag direct schema edits. Engine code must stay backward-compatible
with already-installed on-disk model state (no forced reinstall).
- path: ".github/workflows/**"
instructions: >-
Pin actions to a major version tag at minimum. Flag any workflow that
grants write permissions it does not need.
# Non-gating pre-merge audits of the project's hard rules (warning mode —
# the human owner is the gate, these make the checklist visible per-PR).
pre_merge_checks:
title:
mode: warning
requirements: >-
Conventional-commit style with scope, e.g. "fix(dub): …", and the issue
reference in title or body. Never propose a version bump.
issue_assessment:
mode: warning
custom_checks:
- name: "Cross-platform default parity"
mode: warning
instructions: >-
If the PR changes any default-mode (out-of-the-box) behavior, verify
it behaves identically on macOS, Windows and Linux, or is moved behind
an explicit opt-in (Settings toggle, env var, or CLI flag). A
platform-divergent default is a P0 per CLAUDE.md — fail this check
and say which platform diverges.
- name: "i18n completeness (21 locales)"
mode: warning
instructions: >-
For every new or changed t('...') key in frontend code, verify the
key exists in all 21 files under frontend/src/i18n/locales/. List any
locale files missing the key. Also flag hardcoded user-facing strings
that bypass i18n entirely.
- name: "Local-first guarantee"
mode: warning
instructions: >-
Verify the PR adds no required cloud calls, accounts, API keys or
telemetry. Outbound traffic is only allowed to GitHub Issues (opt-in
bug reporting) and HuggingFace model downloads. The app must remain
fully functional offline and with reporting disabled.
- name: "Backward compatibility"
mode: warning
instructions: >-
Verify existing omnivoice_data/ (voices, projects, settings) and
already-installed engine model state keep working without manual
migration. Any DB schema change must ship an alembic migration with
an upgrade path. Flag anything that would force users to reinstall an
engine or re-download model weights.
finishing_touches:
docstrings:
enabled: true
unit_tests:
enabled: true
# Feed the bot the project constitution and docs, and let it accumulate
# learnings from review conversations ("@coderabbitai always/never …").
knowledge_base:
code_guidelines:
enabled: true
filePatterns:
- "CLAUDE.md"
- "docs/**/*.md"
learnings:
scope: auto
issues:
scope: auto
pull_requests:
scope: auto
chat:
auto_reply: true
+3 -6
View File
@@ -1,9 +1,6 @@
# These are supported funding model platforms
# GitHub Sponsors isn't set up for this account — fund via Ko-fi or PayPal.
ko_fi: debpalash
custom:
- "https://paypal.me/palashCoder"
- "https://github.com/debpalash/OmniVoice-Studio/blob/main/SPONSORS.md"
# github: [debpalash] # not available
github: [debpalash]
# ko_fi: omnivoice
# open_collective: omnivoice-studio
# custom: ["https://omnivoice.palash.dev/sponsor"]
+45
View File
@@ -0,0 +1,45 @@
---
name: 🐛 Bug Report
about: Report a bug to help us improve OmniVoice Studio
title: "[Bug] "
labels: ["bug", "triage"]
assignees: []
---
## Describe the bug
A clear and concise description of what the bug is.
## To reproduce
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '...'
3. See error
## Expected behavior
What you expected to happen.
## Screenshots / Logs
If applicable, add screenshots or paste relevant logs from **Settings → Logs**.
> **Tip:** **Settings → About → "Save diagnostic bundle"** produces a zip
> (self-check report, recent errors, scrubbed log tails) you can drag onto
> this issue — it answers most environment questions below automatically.
> Headless installs: `python backend/main.py --diagnose` prints the same
> self-check (`--deep` also test-loads the active engine).
## Environment
- **OS:** [e.g. macOS 15.2, Windows 11, Ubuntu 24.04]
- **Install method:** [Desktop app / Docker / From source]
- **Version:** [e.g. v0.2.7 — check Settings → About]
- **GPU:** [e.g. NVIDIA RTX 4090 / Apple M3 Pro / CPU only]
- **RAM:** [e.g. 16 GB]
- **Active TTS engine:** [e.g. omnivoice — check Settings → Engines]
## Additional context
Add any other context about the problem here.
-109
View File
@@ -1,109 +0,0 @@
name: 🐛 Bug report
description: Something works incorrectly or crashes (not a first-run/install problem — use the install template for those).
title: "[Bug] "
labels: ["bug", "triage"]
body:
- type: markdown
attributes:
value: |
Thanks for helping improve OmniVoice Studio! 🎙️
**Fastest path to a fix:** **Settings → About → "Save diagnostic bundle"** makes a
zip (self-check + recent errors + scrubbed log tails) — drag it onto this issue and
most of the environment questions below are answered automatically.
Headless: `python backend/main.py --diagnose` (add `--deep` to test-load the engine).
- type: checkboxes
id: preflight
attributes:
label: Before filing
options:
- label: I searched [existing issues](https://github.com/debpalash/OmniVoice-Studio/issues?q=is%3Aissue) and this isn't a duplicate.
required: true
- label: I'm on the latest release (or `main`) — older builds may already be fixed.
required: false
- type: textarea
id: what-happened
attributes:
label: What happened?
description: A clear description of the bug, including the exact error text / toast if any.
placeholder: "Voice cloning failed with '…' after I clicked Generate."
validations:
required: true
- type: textarea
id: repro
attributes:
label: Steps to reproduce
value: |
1.
2.
3.
validations:
required: true
- type: textarea
id: expected
attributes:
label: What did you expect instead?
validations:
required: true
- type: dropdown
id: os
attributes:
label: Operating system
options:
- macOS (Apple Silicon)
- macOS (Intel)
- Windows (x64)
- Linux (AppImage)
- Linux (.deb)
- Linux (other / from source)
- Docker
validations:
required: true
- type: dropdown
id: install
attributes:
label: How did you install it?
options:
- Desktop app (installer / AppImage)
- Docker image
- From source (uv sync)
validations:
required: true
- type: input
id: version
attributes:
label: Version
description: Settings → About (e.g. v0.3.5), or the Docker tag / git SHA.
placeholder: "v0.3.5"
validations:
required: true
- type: dropdown
id: device
attributes:
label: Compute device
options:
- NVIDIA GPU (CUDA)
- AMD GPU (ROCm)
- Apple Silicon (MPS)
- Intel GPU (XPU)
- CPU only
- Not sure
validations:
required: true
- type: input
id: engine
attributes:
label: Active TTS/ASR engine
description: Settings → Engines (e.g. omnivoice, cosyvoice, indextts2, whisperx).
placeholder: "omnivoice"
- type: textarea
id: logs
attributes:
label: Logs / diagnostic bundle
description: Drag the diagnostic bundle here, or paste relevant lines from **Settings → Logs**. Secrets are scrubbed automatically.
render: text
- type: textarea
id: extra
attributes:
label: Anything else?
description: Screenshots, the input that triggered it, RAM/VRAM, etc.
-11
View File
@@ -1,11 +0,0 @@
blank_issues_enabled: false
contact_links:
- name: 💬 Discord — questions & quick help
url: https://discord.gg/bzQavDfVV9
about: Usage questions, setup help, and chat. Faster than an issue for "how do I…".
- name: 🗣️ GitHub Discussions
url: https://github.com/debpalash/OmniVoice-Studio/discussions
about: Ideas, show-and-tell, and open-ended Q&A that isn't a bug or a specific feature ask.
- name: 🔒 Security vulnerability
url: https://github.com/debpalash/OmniVoice-Studio/security/policy
about: Please report security issues privately — do NOT open a public issue.
+23
View File
@@ -0,0 +1,23 @@
---
name: ✨ Feature Request
about: Suggest an idea for OmniVoice Studio
title: "[Feature] "
labels: ["enhancement"]
assignees: []
---
## Is your feature request related to a problem?
A clear description of what the problem is. Ex. "I'm always frustrated when..."
## Describe the solution you'd like
A clear description of what you want to happen.
## Describe alternatives you've considered
Any alternative solutions or features you've considered.
## Additional context
Add any other context, mockups, or screenshots about the feature request here.
@@ -1,50 +0,0 @@
name: ✨ Feature request
description: Suggest an improvement or a new capability.
title: "[Feature] "
labels: ["enhancement"]
body:
- type: checkboxes
id: preflight
attributes:
label: Before filing
options:
- label: I searched [existing issues](https://github.com/debpalash/OmniVoice-Studio/issues?q=is%3Aissue) and [discussions](https://github.com/debpalash/OmniVoice-Studio/discussions) for this idea.
required: true
- type: textarea
id: problem
attributes:
label: What problem does this solve?
description: The use case / friction this addresses ("When I … I can't …").
validations:
required: true
- type: textarea
id: solution
attributes:
label: Proposed solution
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives considered
- type: dropdown
id: area
attributes:
label: Area
options:
- Voice cloning
- Voice design
- Video dubbing
- Real-time dictation
- Audiobook / Stories (long-form)
- TTS/ASR engines
- Install / setup / packaging
- Other
validations:
required: true
- type: markdown
attributes:
value: |
> OmniVoice is **local-first** — features must work fully offline with no accounts,
API keys, or cloud calls, and behave identically on macOS/Windows/Linux. Proposals
that fit those constraints are easiest to land.
@@ -1,86 +0,0 @@
name: 🧩 Install / first-run problem
description: The app won't install, set up, download models, or reach a first working output.
title: "[Install] "
labels: ["install", "triage"]
body:
- type: markdown
attributes:
value: |
A first-run that *just works* is the whole point — sorry it didn't. Let's fix it.
If the app launched far enough to open Settings, **Settings → About → "Save diagnostic
bundle"** captures most of this; otherwise the fields below are enough.
- type: dropdown
id: stage
attributes:
label: Where did it fail?
options:
- App won't launch / blank or broken window
- Python / uv environment bootstrap
- Model download (HuggingFace)
- Engine install (CosyVoice / IndexTTS / MLX / etc.)
- First synthesis / dub never completes
- Other
validations:
required: true
- type: textarea
id: error
attributes:
label: The error
description: The exact message, traceback, or what you see on screen.
render: text
validations:
required: true
- type: dropdown
id: os
attributes:
label: Operating system
options:
- macOS (Apple Silicon)
- macOS (Intel)
- Windows (x64)
- Linux (AppImage)
- Linux (.deb)
- Linux (other / from source)
- Docker
validations:
required: true
- type: dropdown
id: install
attributes:
label: How are you installing it?
options:
- Desktop app (installer / AppImage)
- Docker image
- From source (uv sync)
validations:
required: true
- type: input
id: version
attributes:
label: Version
placeholder: "v0.3.5 (or installer build / git SHA)"
validations:
required: true
- type: dropdown
id: network
attributes:
label: Network conditions (model/dependency downloads)
description: Restricted networks are a known source of bootstrap failures (mirror fallback).
options:
- Normal / unrestricted
- Behind a corporate proxy / firewall
- Region with restricted access (e.g. China, Russia)
- Offline / air-gapped
- Not sure
validations:
required: true
- type: textarea
id: logs
attributes:
label: Logs / diagnostic bundle
description: Drag a diagnostic bundle, or paste the install/bootstrap log. Headless self-check — `python backend/main.py --diagnose`.
render: text
- type: textarea
id: tried
attributes:
label: What have you already tried?
-78
View File
@@ -1,78 +0,0 @@
name: 🤝 Sponsorship inquiry
description: Support OmniVoice and (optionally) claim a logo slot. Not for bugs or feature requests.
title: "Sponsorship inquiry: "
labels: ["sponsor"]
body:
- type: markdown
attributes:
value: |
Thanks for considering sponsoring **OmniVoice Studio** 💛
OmniVoice is free, local-first, and AGPL-3.0 — sponsorship keeps development going.
See **[SPONSORS.md](https://github.com/debpalash/OmniVoice-Studio/blob/main/SPONSORS.md)** for tiers, placements, and logo guidelines.
Prefer to just donate? [Ko-fi](https://ko-fi.com/debpalash) (recurring) or [PayPal](https://paypal.me/palashCoder) (one-time) — you don't need this form for that.
- type: input
id: name
attributes:
label: Name or organization
description: How you'd like to be credited (person or company).
validations:
required: true
- type: input
id: website
attributes:
label: Website / link
description: The URL your name or logo should link to (homepage, product page, profile…).
placeholder: https://example.com
- type: input
id: logo
attributes:
label: Logo URL (optional)
description: Link to your logo (SVG preferred, else 2× PNG, transparent background). You can also attach it in the description below.
placeholder: https://example.com/logo.svg
- type: dropdown
id: tier
attributes:
label: Tier you're interested in
description: See SPONSORS.md for what each tier includes. Not sure? Pick "Not sure yet".
options:
- Backer
- Bronze
- Silver
- Gold
- Not sure yet — let's talk
- Custom / annual arrangement
validations:
required: true
- type: dropdown
id: method
attributes:
label: How you'd like to support
options:
- Ko-fi (recurring)
- Ko-fi (one-time)
- PayPal (one-time)
- Not sure yet — let's discuss
validations:
required: true
- type: input
id: contact
attributes:
label: How should we reach you?
description: Email or another contact. (GitHub will also notify you on this issue.)
validations:
required: true
- type: textarea
id: notes
attributes:
label: Anything else?
description: Questions, constraints, timeline, or context. Attach your logo here if you didn't link it above.
- type: checkboxes
id: ack
attributes:
label: Acknowledgements
options:
- label: I understand sponsorship is a thank-you, not a paywall — OmniVoice stays fully free and AGPL-3.0, and sponsors don't get gated features.
required: true
- label: If I provide a logo, I have the right to use it and grant OmniVoice permission to display it in the README, the app, and the project website.
required: false
+13 -8
View File
@@ -18,7 +18,7 @@
- [ ] 📝 Documentation
- [ ] 🧪 Tests
- [ ] 🔧 CI / Build
- [ ] 🚀 Release prep
- [ ] 🚀 Release prep (RC or final)
## Testing
@@ -33,12 +33,17 @@
- [ ] No local machine paths, logs, or personal env details in this PR
- [ ] Version files are in sync (if version bump): `pyproject.toml`, `package.json`, `tauri.conf.json`, `Cargo.toml`
- [ ] If this PR changes runtime behavior, the regression fixture at `tests/fixtures/omnivoice_data/` still loads green on the `smoke-matrix` CI job (macOS + Windows + Linux)
- [ ] If this is part of a release, I've read the "Release cadence" section below and confirmed this PR targets the right RC
## Release cadence
## Release cadence (read once per RC)
OmniVoice ships **continuous-to-main** — no release candidates, no soak windows.
Every merged PR is immediately part of the rolling preview (`main`, Docker
`:latest`, the desktop Preview channel). Versioned releases are tagged from
`main` when it's ready; `main` then bumps to the next patch automatically.
Users who want stability pin a release tag / Docker `:stable` / the desktop
Stable channel.
OmniVoice ships every minor on a **two-RC cadence**:
- `vX.Y.0-rc1` — cut from `main` once all GATE-* requirements pass; clean-VM exercise on 4 OSes (per `REL-01`)
- 48-hour soak (no new commits to release branch except fix-forward)
- `vX.Y.0` — promotion if rc1 is clean
If your PR touches install / bootstrap / CI, it MUST land before rc1 cut, not between rc1 and the promotion. During a soak, any merge needs explicit OK from the release captain.
## Screenshots
<!-- If applicable, add screenshots or recordings. -->
+4 -26
View File
@@ -92,10 +92,7 @@ jobs:
- name: Install frontend deps
working-directory: frontend
# --frozen-lockfile so a frontend/package.json change that forgets to
# regenerate the root bun.lock fails HERE (fast) instead of only in the
# Docker build (deploy/Dockerfile), which is what reddened main on #485.
run: bun install --frozen-lockfile
run: bun install
# checkJs is true in tsconfig for IDE feedback, but 947 pre-existing
# JS errors remain. Override to false in CI so only .ts files block.
@@ -105,25 +102,9 @@ jobs:
working-directory: frontend
run: bun run typecheck:ci
# oxlint gate — fast Rust linter, blocks on errors so lint debt can't
# re-accumulate (warnings, incl. the react-compiler advisories in
# `lint:hooks`, are non-blocking). See frontend/.oxlintrc.json.
- name: Frontend lint (oxlint)
working-directory: frontend
run: bun run lint
# oxfmt format gate — JS/TS/JSX only (CSS/JSON/Tauri excluded; see
# frontend/.oxfmtrc.json). `bun run format` fixes locally.
- name: Frontend format check (oxfmt)
working-directory: frontend
run: bun run format:check
# `bun run test` (frontend/package.json), not `bunx vitest` — bunx
# resolves by npm package name and can miss workspace-hoisted bins,
# then falls back to fetching from npm (#962 class).
- name: Run Vitest (frontend)
working-directory: frontend
run: bun run test
run: bunx vitest run
# Legacy node:test runner for tests/frontend/*.test.mjs
- name: Run frontend node:test (legacy)
@@ -149,7 +130,7 @@ jobs:
- os: windows-2022
label: Windows
rust_target: x86_64-pc-windows-msvc
- os: ubuntu-24.04
- os: ubuntu-22.04
label: Linux
rust_target: x86_64-unknown-linux-gnu
runs-on: ${{ matrix.os }}
@@ -193,10 +174,7 @@ jobs:
- name: Install frontend deps
working-directory: frontend
# --frozen-lockfile so a frontend/package.json change that forgets to
# regenerate the root bun.lock fails HERE (fast) instead of only in the
# Docker build (deploy/Dockerfile), which is what reddened main on #485.
run: bun install --frozen-lockfile
run: bun install
# tauri-build's setup hook reads tauri.conf.json's `frontendDist`
# ("../dist"), which only exists after a frontend build. Without this,
+17 -62
View File
@@ -5,21 +5,14 @@
# - push to main branch → :main, :sha- (rolling "edge" build)
# - workflow_dispatch → :sha- only (ad-hoc test build)
#
# Tag ↔ image mapping (versioning hard rule, owner-set 2026-06-11:
# :latest IS the preview channel; stable users pin :stable or a version tag)
# :latest — rolling preview: latest commit on main (always last release + 1 dev)
# :main — alias of the same rolling main build (kept for back-compat)
# :stable — most recent versioned release (set on every v* tag push)
# :0.3.6 — exact version from the git tag
# Tag ↔ image mapping
# :latest — always the most recent versioned release (set on every v* tag push)
# :0.3.0 — exact version from the git tag
# :0.3 — major.minor floating tag (updated on every patch within the minor)
# :main — latest commit on main; may be ahead of the last tagged release
# :sha-xxxx — specific commit SHA; produced by workflow_dispatch
#
# Images land at: ghcr.io/debpalash/omnivoice-studio AND docker.io/palashdeb/omnivoice-studio
# (Docker Hub push gated on the DOCKERHUB_USERNAME/DOCKERHUB_TOKEN secrets;
# if unset the build still pushes to GHCR.)
#
# On main pushes the Docker Hub repository overview is also synced from
# deploy/dockerhub-overview.md (source of truth for the hub.docker.com page).
# Images land at: ghcr.io/debpalash/omnivoice-studio
#
# NOTE: the Docker image is the headless web-server build of OmniVoice (FastAPI
# backend + pre-built React frontend served over HTTP). The Tauri desktop
@@ -41,7 +34,6 @@ permissions:
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
DOCKERHUB_IMAGE: palashdeb/omnivoice-studio
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
jobs:
@@ -64,43 +56,28 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
# Docker Hub login — only when the secret is present, so forks / runs
# without the credential still publish to GHCR.
- name: Check Docker Hub credentials
id: dockerhub
run: echo "enabled=${{ secrets.DOCKERHUB_TOKEN != '' }}" >> "$GITHUB_OUTPUT"
- name: Log in to Docker Hub
if: steps.dockerhub.outputs.enabled == 'true'
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
# Tag strategy (`:sha-<short>` is emitted on every trigger):
# v0.3.6 tag push → :0.3.6, :0.3, :stable, :sha-
# main branch push → :latest, :main, :sha-
# v0.3.0 tag push → :0.3.0, :0.3, :latest, :sha-
# main branch push → :main, :sha-
# workflow_dispatch → :sha- only
#
# All mutable-tag rules stay gated on `github.event_name == 'push'` so a
# manual workflow_dispatch can only ever produce a throwaway `:sha-` tag
# (the stale-:latest fix from #249/#251). :stable excludes prerelease
# tags (those contain a `-`) so a prerelease can't clobber it.
# Fix for stale :latest (issues #249, #251):
# The previous rule used `enable={{is_default_branch}}`, which evaluates
# to false on tag pushes (detached HEAD) — so :latest was never updated
# when a release tag was pushed. The version / :latest / :main rules are
# gated on `github.event_name == 'push'` so a manual workflow_dispatch can
# only ever produce a throwaway `:sha-` tag (never republish a mutable
# tag), and :latest additionally excludes prerelease tags (those contain a
# `-`, e.g. v1.0.0-rc.1) so a prerelease can't clobber :latest.
- name: Extract metadata (tags, labels)
id: meta
uses: docker/metadata-action@v5
with:
# Same tag set applied to both registries. The Docker Hub line is
# blank when the secret is unset, so metadata-action emits GHCR-only
# tags in that case.
images: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
${{ steps.dockerhub.outputs.enabled == 'true' && env.DOCKERHUB_IMAGE || '' }}
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=semver,pattern={{version}},enable=${{ github.event_name == 'push' }}
type=semver,pattern={{major}}.{{minor}},enable=${{ github.event_name == 'push' }}
type=raw,value=stable,enable=${{ github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-') }}
type=raw,value=latest,enable=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
type=raw,value=latest,enable=${{ github.event_name == 'push' && github.ref_type == 'tag' && !contains(github.ref, '-') }}
type=raw,value=main,enable=${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
type=sha,prefix=sha-,format=short
@@ -114,25 +91,3 @@ jobs:
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
# Sync the Docker Hub repository overview from deploy/dockerhub-overview.md.
# Only on main pushes (the overview tracks the rolling preview) and only
# when Docker Hub creds are present, mirroring the push gating above.
#
# continue-on-error: the overview text is cosmetic, and the description
# PATCH 403s unless DOCKERHUB_TOKEN carries description-edit scope (many
# fine-grained Docker Hub tokens that can push still can't edit the
# description). The image build+push is what matters — a creds-scope
# mismatch on this cosmetic step must not fail the whole Docker run. To
# actually sync the overview, use a token with read/write (incl.
# description) scope, or the account password.
- name: Update Docker Hub description
if: steps.dockerhub.outputs.enabled == 'true' && github.event_name == 'push' && github.ref == 'refs/heads/main'
continue-on-error: true
uses: peter-evans/dockerhub-description@v4
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
repository: ${{ env.DOCKERHUB_IMAGE }}
short-description: "Local ElevenLabs alternative: voice cloning, design & video dubbing in 646 languages. No API keys."
readme-filepath: ./deploy/dockerhub-overview.md
-97
View File
@@ -1,97 +0,0 @@
# Docs drift — daily inventory-vs-docs check with a single rolling issue.
#
# docs/features.yaml is the canonical inventory; scripts/check-docs-drift.py
# diffs it against README.md, docs/, and the engine registries. On drift the
# job updates (or creates) ONE issue labeled `docs-drift` in place — no issue
# spam — and closes it automatically when the check is clean again.
#
# Companion to the PR-gating validate-install-docs.py step in ci.yml.
# Spec: docs/competitive-analysis.md Spec 9a / parity program Wave 0.1.
# Rolling-issue pattern adapted from Patter (MIT).
name: docs-drift
on:
schedule:
# Daily 03:30 UTC — after most merges, before EU morning triage.
- cron: "30 3 * * *"
workflow_dispatch:
permissions:
contents: read
issues: write
jobs:
drift:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install checker deps
run: pip install "pyyaml>=6"
- name: Check inventory vs README/docs/registries
id: drift
continue-on-error: true
run: python scripts/check-docs-drift.py --output drift-report.md
- name: Update rolling docs-drift issue
uses: actions/github-script@v7
env:
DRIFT_OUTCOME: ${{ steps.drift.outcome }}
with:
script: |
const fs = require('fs');
const drifted = process.env.DRIFT_OUTCOME === 'failure';
const { owner, repo } = context.repo;
const label = 'docs-drift';
const open = await github.rest.issues.listForRepo({
owner, repo, state: 'open', labels: label, per_page: 5,
});
if (drifted) {
let body = '';
try {
body = fs.readFileSync('drift-report.md', 'utf8');
} catch {
body = '# Docs drift report\n\nThe checker failed before writing a report — see the workflow run logs.';
}
body += `\n\n---\n_Last checked by [run ${context.runId}](https://github.com/${owner}/${repo}/actions/runs/${context.runId})._\n`;
if (open.data.length > 0) {
await github.rest.issues.update({
owner, repo, issue_number: open.data[0].number, body,
});
core.info(`Updated rolling issue #${open.data[0].number}`);
} else {
const created = await github.rest.issues.create({
owner, repo,
title: 'docs-drift: feature inventory vs docs mismatch',
body,
labels: [label, 'documentation'],
});
core.info(`Created rolling issue #${created.data.number}`);
}
} else {
for (const issue of open.data) {
await github.rest.issues.createComment({
owner, repo, issue_number: issue.number,
body: 'Drift resolved — nightly check is clean again. Closing automatically.',
});
await github.rest.issues.update({
owner, repo, issue_number: issue.number, state: 'closed',
});
core.info(`Closed rolling issue #${issue.number}`);
}
}
- name: Surface drift as a failed run
if: steps.drift.outcome == 'failure'
run: |
echo "Docs drift detected — see the rolling docs-drift issue."
exit 1
-55
View File
@@ -1,55 +0,0 @@
# LLM-judge evals — semantic quality suites, NEVER a gate.
#
# Hard rule (parity program Wave 0.3 / competitive-analysis Spec 9b): LLM
# judges never gate CI. This workflow is scheduled + manual only, the eval
# step is continue-on-error, and the JSON report is the deliverable
# (uploaded as an artifact). Deterministic probe judges in ci.yml remain
# the only gates.
#
# On the hosted runner there is no local LLM endpoint, so the run usually
# reports "skipped — no LLM backend configured"; the workflow exists so the
# suites run anywhere a TRANSLATE_BASE_URL secret/endpoint is provided
# (e.g. a self-hosted runner with Ollama).
name: evals
on:
schedule:
# Weekly, Sundays 04:00 UTC.
- cron: "0 4 * * 0"
workflow_dispatch:
permissions:
contents: read
jobs:
evals:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- uses: astral-sh/setup-uv@v3
with:
enable-cache: true
cache-dependency-glob: "uv.lock"
- name: Install deps
run: uv sync
- name: Run eval suites (non-gating)
continue-on-error: true
env:
TRANSLATE_BASE_URL: ${{ secrets.EVALS_LLM_BASE_URL }}
TRANSLATE_API_KEY: ${{ secrets.EVALS_LLM_API_KEY }}
run: uv run python tests/evals/run_evals.py --output eval-report.json
- name: Upload report artifact
uses: actions/upload-artifact@v4
with:
name: eval-report
path: eval-report.json
if-no-files-found: warn
+27 -231
View File
@@ -4,16 +4,12 @@
# - push of a tag matching `v*` (e.g. `v0.2.0`) → full STABLE release,
# publishes artifacts + signed updater manifest (`latest.json`) to the
# tag's GH Release. This is the default Stable updater channel.
# - schedule (nightly, 07:00 UTC) → rolling `preview` PRERELEASE built from
# `main` with its own signed `latest.json` at releases/download/preview/.
# Feeds the opt-in Preview updater channel (Settings → About → Update
# channel). The `preview-gate` job skips the matrix on nights when `main`
# didn't move, so an idle day costs only a ~30s gate job — keeping Preview
# ≤24h behind `main` at a predictable ~1-matrix/day cost. The stable
# `latest` release is untouched.
# - workflow_dispatch (publish_preview=true) → the same preview build on
# demand from the selected branch (e.g. to preview a feature branch, or to
# refresh immediately without waiting for the nightly).
# - workflow_dispatch (publish_preview=true) → builds the selected branch and
# publishes a rolling `preview` PRERELEASE with its own signed
# `latest.json` at releases/download/preview/. This feeds the opt-in
# Preview updater channel (Settings → About → Update channel). The stable
# `latest` release is untouched. Run this manually whenever you want to cut
# a preview from `main`.
# - workflow_dispatch (publish_preview=false) → on-demand build (prior
# behavior; draft release named after the branch).
#
@@ -32,10 +28,6 @@ name: Desktop Release
on:
push:
tags: ['v*']
schedule:
# 07:00 UTC daily — rolling `preview` prerelease from `main`. The
# preview-gate job no-ops the matrix when main hasn't moved in a day.
- cron: '0 7 * * *'
workflow_dispatch:
inputs:
draft:
@@ -129,41 +121,8 @@ jobs:
working-directory: frontend
run: node --experimental-strip-types --no-warnings --test ../tests/frontend/*.test.mjs
# Decide preview-vs-stable, and for nightly runs whether `main` actually
# moved in the last day. Outputs gate the expensive matrix (`build`) and the
# `preview-notes` job, so a no-commit night costs only this ~30s job.
preview-gate:
name: Preview gate
runs-on: ubuntu-22.04
outputs:
is_preview: ${{ steps.decide.outputs.is_preview }}
proceed: ${{ steps.decide.outputs.proceed }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 50
- id: decide
shell: bash
run: |
set -euo pipefail
event="${{ github.event_name }}"
if [ "$event" = "schedule" ] || { [ "$event" = "workflow_dispatch" ] && [ "${{ inputs.publish_preview }}" = "true" ]; }; then
echo "is_preview=true" >> "$GITHUB_OUTPUT"
else
echo "is_preview=false" >> "$GITHUB_OUTPUT"
fi
# Nightly: skip the matrix when main hasn't moved in the last day.
if [ "$event" = "schedule" ] && [ -z "$(git log --since='25 hours ago' --oneline)" ]; then
echo "No new commits on main in the last day — skipping nightly preview."
echo "proceed=false" >> "$GITHUB_OUTPUT"
else
echo "proceed=true" >> "$GITHUB_OUTPUT"
fi
build:
needs: [test, preview-gate]
# Nightly runs with no new commits on main skip the 4-platform matrix.
if: needs.preview-gate.outputs.proceed == 'true'
needs: test
strategy:
fail-fast: false
matrix:
@@ -174,36 +133,9 @@ jobs:
rust_target: aarch64-apple-darwin
bundles: "app,dmg,updater"
# macOS Intel (#279): reinstated. The earlier "Rosetta 2 runs the
# ARM build" rationale for dropping it was backwards — Rosetta only
# translates x86_64→arm64, so Intel Macs (supported through macOS
# Sequoia) simply cannot run the aarch64 bundle and had NO
# installable artifact. Runner: `macos-15-intel`, GitHub's
# designated migration target after macos-13 retired (Dec 2025);
# it's a standard (public-repo-free) image supported through
# August 2027 — the last x86_64 image Actions will offer. Building
# natively (not cross-compiling from the arm64 leg) keeps the
# per-TRIPLE uv/ffmpeg sidecar fetches, the DMG installer smoke,
# and the ad-hoc signing verification (scripts/
# verify-macos-signing.sh, PR #290) all exercising the real
# x86_64 artifact on real Intel hardware. The macos-13 queue
# backlog that motivated the original drop is contained by
# fail-fast:false — a slow Intel leg can delay the release run but
# can't fail the other targets.
#
# #889 (2026-07): Intel macOS is now UNSUPPORTED for the local
# backend — torch ≥2.3 ships no macOS x86_64 wheels, so the venv
# bootstrap can never succeed on Intel. The shipped x64 artifact is
# effectively UI-only (usable with a remote backend); the app now
# pre-fails first-run bootstrap with an honest message on Intel.
# Whether to keep shipping this x64 leg (UI-only) or drop it is an
# OWNER CALL — deliberately not changed in the #889 PR.
- os: macos-15-intel
arch: x86_64-apple-darwin
label: "macOS Intel"
rust_target: x86_64-apple-darwin
bundles: "app,dmg,updater"
# macOS Intel dropped: Apple shipped the last Intel Mac in 2023 and
# Rosetta 2 runs the ARM build natively. macos-13 runner backlog
# was also blocking every release tag for ~10 min.
# Windows: force MSI bundling via --bundles. NSIS fails at makensis
# because our PyInstaller payload approaches its ~2 GB stub limit.
- os: windows-2022
@@ -213,24 +145,13 @@ jobs:
bundles: "msi,updater"
# Linux: ship .AppImage only. AppImage is universal (no distro
# package-manager dep), runs on any glibc-2.39+ host, and is the
# package-manager dep), runs on any glibc-2.31+ host, and is the
# Linux auto-update target. The .deb target was dropped: tauri-bundler
# fails it with "Failed to create control scripts: No such file or
# directory" (no custom deb config of ours is at fault) — revisit on a
# tauri-cli bump. FUSE unavailability on GH runners is handled via
# APPIMAGE_EXTRACT_AND_RUN=1.
#
# Bumped from ubuntu-22.04 → ubuntu-24.04 (#961): the AppImage
# bundles whatever `libwebkit2gtk-4.1-dev` the build runner's apt
# repos resolve (see the "Linux system deps" step below) — 22.04's
# was meaningfully stale relative to what current Ubuntu/Fedora
# ship, and AppRun's LD_LIBRARY_PATH makes that bundled, stale copy
# take priority over a healthy system WebKitGTK at runtime. Raises
# the AppImage's glibc floor from 2.35 to 2.39 — pre-2022 distros
# (Ubuntu <22.04, Debian <12) lose support; no report of anyone on
# something that old has come in, and the project's own install
# docs already assume Debian 12 / Ubuntu 22.04+.
- os: ubuntu-24.04
- os: ubuntu-22.04
arch: x86_64-unknown-linux-gnu
label: "Linux x64"
rust_target: x86_64-unknown-linux-gnu
@@ -370,10 +291,7 @@ jobs:
case "$TRIPLE" in
aarch64-apple-darwin|x86_64-apple-darwin)
# evermeet.cx ships each binary as a separate .zip containing
# a single x86_64 Mach-O executable — natively correct on the
# Intel leg, and runs via Rosetta 2 on the arm64 leg. Both
# darwin TRIPLEs therefore bundle the same payload; only the
# sidecar filename suffix differs.
# a single x86_64 Mach-O executable (runs via Rosetta on arm64).
for TOOL in ffmpeg ffprobe; do
if [ "$TOOL" = "ffmpeg" ]; then
URL="https://evermeet.cx/ffmpeg/getrelease/zip"
@@ -480,21 +398,17 @@ jobs:
# always reported the static 0.3.0 never looked "newer", so no update was
# ever delivered). Ephemeral, CI-only — never committed. Tauri reads the
# bundle + updater version from tauri.conf.json, so rewriting it here
# stamps the artifacts + latest.json. Under the versioning hard rule
# (owner-set 2026-06-11) main is always last-release + 1, so BASE-N is a
# prerelease of the NEXT version and semver-sorts ABOVE the last stable
# (0.3.6-N > 0.3.5) — preview users naturally upgrade past stable, and
# the Windows MSI ProductVersion (which strips the prerelease → 0.3.6)
# is also correctly above the last stable.
# stamps the artifacts + latest.json. `0.3.0-preview.N` is a prerelease of
# the current target, so previews converge to stable when 0.3.0 ships
# (0.3.0 > 0.3.0-preview.N). NOTE: the Windows MSI ProductVersion strips
# the prerelease (→ 0.3.0), a wrinkle to verify for win preview→preview
# upgrades; mac/linux replace the bundle wholesale and are unaffected.
- name: Stamp preview version
if: needs.preview-gate.outputs.is_preview == 'true'
if: github.event_name == 'workflow_dispatch' && inputs.publish_preview
shell: bash
run: |
set -euo pipefail
# package.json is the single source of truth; tauri.conf.json reads its
# version from it ("version": "../package.json"), so stamping
# package.json restamps the whole bundle.
CONF=frontend/package.json
CONF=frontend/src-tauri/tauri.conf.json
BASE=$(jq -r .version "$CONF")
# MSI/WiX requires the semver pre-release identifier to be numeric-only
# (and <= 65535). "preview.N" hard-fails the Windows bundler, so the
@@ -517,12 +431,6 @@ jobs:
# only on the opt-in stable path, leaving them ABSENT (not "") on
# preview/unsigned paths so Tauri's bundler skips cert import. A static
# env: here would always set them to "" and break the mac build.
# Unsigned paths still get a VALID ad-hoc seal from tauri.conf.json
# (bundle.macOS.signingIdentity = "-"), so a downloaded build shows the
# GUI-bypassable "unidentified developer" prompt (right-click → Open /
# Settings → "Open Anyway") instead of the un-bypassable "damaged"
# error. On the signed path APPLE_SIGNING_IDENTITY (env) overrides the
# "-" default; once notarized, Gatekeeper accepts it with no prompt.
# GH runners disable FUSE, so linuxdeploy's AppImage can't mount
# itself at bundle time. This env tells linuxdeploy to extract-and-run
# instead, which works without FUSE.
@@ -534,13 +442,11 @@ jobs:
# rolling `preview` prerelease for the updater's Preview channel.
# Every other invocation — crucially the `v*` tag-push stable release
# — evaluates these expressions to exactly their prior values.
tagName: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'preview' || github.ref_name }}
# Version-first so the tag is readable in GitHub's truncated
# release-list sidebar (which clips the title mid-string).
releaseName: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'Preview — OmniVoice Studio' || format('{0} — OmniVoice Studio', github.ref_name) }}
tagName: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'preview' || github.ref_name }}
releaseName: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'OmniVoice Studio (Preview)' || format('OmniVoice Studio {0}', github.ref_name) }}
releaseBody: ${{ steps.changelog.outputs.body }}
releaseDraft: ${{ (needs.preview-gate.outputs.is_preview == 'true') && 'false' || (inputs.draft || 'true') }}
prerelease: ${{ needs.preview-gate.outputs.is_preview == 'true' }}
releaseDraft: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) && 'false' || (inputs.draft || 'true') }}
prerelease: ${{ (github.event_name == 'workflow_dispatch' && inputs.publish_preview) || false }}
updaterJsonPreferNsis: false
includeUpdaterJson: true
@@ -575,35 +481,6 @@ jobs:
echo "OK — bundle has shell + uv + backend resources"
hdiutil detach "$MOUNT" || true
# ── Signing / Gatekeeper / notarization verification ──────────────
# Runs codesign --verify, spctl (Gatekeeper), nested-binary, and
# stapler checks against the built .app (see docs/macos-signing-verification.md).
# STRICT (--require-signed) only on the opt-in signed stable path — same
# condition as "Configure Apple signing" above — so a failed or missing
# signature/notarization FAILS the job and STOPS the release instead of
# publishing an unsigned artifact. On every other (unsigned dev/preview)
# path it runs report-only and never breaks the build.
- name: Verify macOS signing
if: runner.os == 'macOS'
shell: bash
env:
STRICT: ${{ (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && vars.MACOS_SIGNING_ENABLED == 'true') && '1' || '0' }}
APPLE_ID: ${{ secrets.APPLE_ID }}
APPLE_PASSWORD: ${{ secrets.APPLE_PASSWORD }}
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
run: |
set -uo pipefail
APP=$(find "frontend/src-tauri/target/${{ matrix.rust_target }}/release/bundle/macos" -maxdepth 1 -name '*.app' | head -1)
[ -n "$APP" ] || { echo "FAIL — no .app found to verify"; exit 1; }
MODE=""
if [ "$STRICT" = "1" ]; then
MODE="--require-signed"
echo "Signed stable release → STRICT verification (release stops on failure)."
else
echo "Unsigned dev/preview path → report-only verification."
fi
bash scripts/verify-macos-signing.sh "$APP" $MODE
- name: Installer smoke (Windows)
if: runner.os == 'Windows'
timeout-minutes: 5
@@ -726,8 +603,8 @@ jobs:
# preview-only — stable `v*` releases keep their CHANGELOG section + the
# appended checksums.
preview-notes:
needs: [build, preview-gate]
if: needs.preview-gate.outputs.is_preview == 'true'
needs: build
if: github.event_name == 'workflow_dispatch' && inputs.publish_preview
runs-on: ubuntu-22.04
permissions:
contents: write
@@ -758,86 +635,5 @@ jobs:
echo ""
echo "$CONTRIB"
} > /tmp/preview-notes.md
# --prerelease re-asserts the flag every run: a non-prerelease
# `preview` release is eligible to become GitHub's "Latest", which is
# the exact URL the Stable updater channel reads — so it must never
# flip off.
gh release edit preview --repo "$REPO" --prerelease --notes-file /tmp/preview-notes.md
gh release edit preview --repo "$REPO" --notes-file /tmp/preview-notes.md
echo "Applied auto-generated release notes + contributors to the preview release."
- name: Verify preview updater manifest (prerelease + platform parity)
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REPO: ${{ github.repository }}
run: |
set -euo pipefail
# The preview release must stay a prerelease (or it can hijack the
# Stable channel's releases/latest endpoint), and its updater manifest
# must cover every platform stable does (else those users — e.g. Intel
# Mac — silently get no preview updates).
is_pre=$(gh release view preview --repo "$REPO" --json isPrerelease -q .isPrerelease)
test "$is_pre" = "true" || { echo "::error::preview release is not a prerelease"; exit 1; }
curl -fsSL "https://github.com/$REPO/releases/download/preview/latest.json" -o /tmp/preview-latest.json
curl -fsSL "https://github.com/$REPO/releases/latest/download/latest.json" -o /tmp/stable-latest.json
python3 - <<'PY'
import json, re
prev = json.load(open("/tmp/preview-latest.json"))
stab = json.load(open("/tmp/stable-latest.json"))
v = prev.get("version", "")
assert re.fullmatch(r"\d+\.\d+\.\d+-\d+", v), f"preview version not X.Y.Z-N: {v!r}"
pk, sk = set(prev.get("platforms", {})), set(stab.get("platforms", {}))
missing = sk - pk
assert not missing, f"preview manifest missing platforms vs stable: {sorted(missing)}"
print(f"preview manifest OK: {v} platforms={sorted(pk)}")
PY
# ── Post-release version bump (OWNER-GATED as of 2026-07-01) ──────────────
# Previously auto-ran after every stable v* tag to keep main = release + 1.
# The owner now controls bumps manually ("keep 0.3.8; I say when to bump"), so
# this job is OPT-IN: it runs ONLY when the repo variable AUTO_VERSION_BUMP is
# set to 'true' (Settings → Secrets and variables → Actions → Variables).
# Unset/anything-else → main stays at whatever it is after release. Re-enable
# by setting the variable; disable again by unsetting it.
version-bump:
if: >-
github.event_name == 'push' && github.ref_type == 'tag'
&& !contains(github.ref, '-')
&& vars.AUTO_VERSION_BUMP == 'true'
runs-on: ubuntu-22.04
permissions:
contents: write
steps:
- uses: actions/checkout@v4
with:
ref: main
fetch-depth: 0
- name: Bump main to released version + 1 patch
shell: bash
run: |
set -euo pipefail
RELEASED="${GITHUB_REF_NAME#v}"
IFS=. read -r MAJ MIN PAT <<< "$RELEASED"
NEXT="$MAJ.$MIN.$((PAT + 1))"
# frontend/package.json is the SINGLE SOURCE OF TRUTH: vite injects
# __APP_VERSION__ from it, and tauri.conf.json reads its bundle version
# from it ("version": "../package.json"). Read CURRENT from it.
CURRENT=$(jq -r .version frontend/package.json)
if [ "$(printf '%s\n' "$NEXT" "$CURRENT" | sort -V | tail -1)" = "$CURRENT" ] && [ "$NEXT" != "$CURRENT" ]; then
echo "main is already at $CURRENT (>= $NEXT) — nothing to bump"; exit 0
fi
# Bump the canonical (package.json), set absolutely so any prior drift
# self-heals. tauri.conf.json needs no edit — it derives from this.
tmp=$(mktemp)
jq --arg v "$NEXT" '.version = $v' frontend/package.json > "$tmp"
mv "$tmp" frontend/package.json
# The remaining files are CI-guarded mirrors (cargo/uv require a
# literal; the version.py literal is the frozen-backend last resort) —
# bump them in lockstep with the canonical.
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" frontend/src-tauri/Cargo.toml
sed -i "0,/^version = \"$CURRENT\"/s//version = \"$NEXT\"/" pyproject.toml
sed -i "0,/_FALLBACK_VERSION = \"$CURRENT\"/s//_FALLBACK_VERSION = \"$NEXT\"/" backend/core/version.py
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add frontend/package.json frontend/src-tauri/Cargo.toml pyproject.toml backend/core/version.py
git commit -m "chore(version): main -> $NEXT after $GITHUB_REF_NAME release"
git push origin main
+2 -6
View File
@@ -32,13 +32,9 @@ env:
permissions:
contents: read
# PR branches: a new push cancels the superseded scan (no wasted runners).
# main: every commit keeps its own group, so nothing is cancelled — a merge
# train used to leave a permanent red ✗ ("cancelled") on every intermediate
# commit in the history view even though nothing failed.
concurrency:
group: security-${{ github.ref }}-${{ github.ref == 'refs/heads/main' && github.sha || 'branch' }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
group: security-${{ github.ref }}
cancel-in-progress: true
jobs:
# ── Secret scanning (gating) ─────────────────────────────────────────────
+2 -9
View File
@@ -1,17 +1,10 @@
# SPIKE-02: Adopt `ModelsLab/omnivoice-singing` as singing variant of the existing engine
**Status:** ⚠️ **SUPERSEDED (2026-06-14)** by [`specs/006-dubbing-singing-mode/`](../../specs/006-dubbing-singing-mode/spec.md)
**Date:** 2026-05-18 (superseded 2026-06-14)
**Status:** Proposed (research-supported) — awaiting Phase 2 SubprocessBackend merge
**Date:** 2026-05-18
**Decision-makers:** [maintainer]
**Related:** ROADMAP Phase 4; REQUIREMENTS SING-01..05; `.planning/phases/04-adaptive-specialty-engines-spike-first/04-RESEARCH.md`
> **Superseded:** This chose `ModelsLab/omnivoice-singing` for singing, but that
> model has **no melody (F0/MIDI) conditioning** — it sings its own melody and
> cannot follow the *source song* a dub must preserve. SoulX-Singer (arXiv
> 2602.07803, published after this decision) provides F0/MIDI conditioning and is
> selected in plan-06. This ADR stays valid only if reframed as an
> expressive-TTS styling toggle, not melody-matched dubbing.
## Context
`ModelsLab/omnivoice-singing` (HuggingFace, 1,053 downloads/month, verified 2026-05-18) is a finetune of `k2-fsa/OmniVoice` — same Apache-2.0 license, same Qwen3-0.6B backbone, same Higgs Audio v2 codec at 24 kHz mono, same `omnivoice` PyPI library (0.1.5, 2026-04-28) already shipping in OmniVoice Studio v0.2.7. Trained on additional singing + emotion-tagged data and activated by a `[singing]` text control tag at generation time.
@@ -1,396 +0,0 @@
---
phase: 260613-fdl
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- pyproject.toml
- backend/api/routers/setup/download.py
- backend/api/routers/setup/models.py
- backend/utils/hf_progress.py
- backend/utils/download_aggregator.py # NEW
- backend/services/segmented_download.py # NEW
- backend/api/routers/system.py
- frontend/src/pages/Settings.jsx
- frontend/src/api/setup.ts
- docs/downloading-models.md # NEW (docs-sync rule)
- tests/backend/setup/test_download_preflight.py # NEW
- tests/backend/services/test_segmented_download.py # NEW
autonomous: true
requirements:
# ── Wave 0 — Spike / gate ───────────────────────────────────────────────────
- FDL-00 # Classify all catalog repos Xet-backed vs legacy-LFS; the result sizes Wave 3
# ── Wave 1 — Maximize + guarantee the Xet fast path (default, no new deps) ───
- FDL-01 # Explicitly pin huggingface_hub>=1.7 + hf-xet in pyproject (today transitive/unpinned)
- FDL-02 # Drive snapshot_download with explicit max_workers + tqdm_class + endpoint (not implicit monkeypatch)
- FDL-03 # /system/info reports fast_download {xet_enabled, xet_version, high_performance}; logged at startup
- FDL-04 # Opt-in HF_XET_HIGH_PERFORMANCE + HDD sequential-write toggles via prefs (env wins)
# ── Wave 2 — Accurate downloaded/remaining + speed (the user-visible win) ────
- FDL-05 # dry_run preflight -> emit install_plan {total_bytes, cached_bytes, to_download_bytes, n_files, n_cached}
- FDL-06 # Backend aggregate tracker -> single 'aggregate' event {bytes_done, total_bytes, rate, eta, files_done/total}
- FDL-07 # Frontend overall bar: speed + downloaded/remaining + ETA from aggregate; per-file detail collapsible; cached-skip shown
# ── Wave 3 — Opt-in IDM-style accelerator for legacy-LFS repos ───────────────
- FDL-08 # Custom httpx segmented downloader: parallel Range GETs, resume, auth-safe redirect, etag/sha verify, cancel (default OFF)
- FDL-09 # Dispatch: accelerator ON + repo is LFS (not Xet) -> segmented path; else xet. Same aggregate progress + weight validation
# ── Wave 4 — Opt-in mirror path + docs ───────────────────────────────────────
- FDL-10 # Opt-in HF_ENDPOINT mirror setting (prefs); documented as classic-LFS fallback (no Xet); pairs with FDL-08
- FDL-11 # Cancel-in-flight endpoint + cooldown interplay (composes with MM2-06 bounded cooldowns)
- FDL-12 # docs/downloading-models.md (speed, fast-download status, HDD/high-perf toggles, mirror/restricted-network) + README pointer
must_haves:
truths:
- "Xet is the default download backend and is provably engaged: /system/info reports fast_download.xet_enabled=true with the hf_xet version, and a Xet-backed repo downloads via parallel chunk range-gets (not single-stream LFS)."
- "Before any bytes flow, the UI shows an accurate denominator: total bytes to download, bytes already cached (skipped), and file count — sourced from snapshot_download(dry_run=True), not guessed from the first tqdm bar."
- "During a download the UI shows ONE overall progress bar with instantaneous speed (sampled over a window, not a single file's rate), bytes downloaded / bytes remaining, and ETA — accurate even while Xet fetches many chunks/files in parallel."
- "hf_transfer is NOT used or enabled anywhere (deprecated, breaks progress); the fast path is Xet only."
- "The custom segmented downloader is OPT-IN (default off), only engages for non-Xet/legacy-LFS repos, never forwards the HF Authorization header to the redirected CDN host, verifies the downloaded file against its expected size/etag before marking complete, resumes a partial .part file, and can be cancelled mid-flight."
- "Default download behavior is identical on macOS, Windows, Linux (Xet path, pure-Python). Every accelerator/mirror/high-perf knob is behind an explicit opt-in (Settings toggle or env var) per the cross-platform-parity strict rule — no bundled per-OS binary, no platform-divergent default."
- "No new on-disk model-state format; existing HF cache layout and already-installed models are untouched; the segmented downloader writes into the same HF cache blob/snapshot structure (or hands off to it) so a model it fetches is indistinguishable from one snapshot_download fetched."
- "uv run pytest tests/backend/setup/test_download_preflight.py tests/backend/services/test_segmented_download.py passes; existing download/install tests stay green."
- "pyproject pins huggingface_hub>=1.7 and hf-xet explicitly; uv.lock resolves with single versions (uv tree shows no duplicate huggingface_hub)."
artifacts:
- path: "backend/utils/download_aggregator.py"
provides: "Per-repo byte aggregator: sums bytes across parallel files/chunks, samples rate over a window, emits one 'aggregate' event"
contains: "class DownloadAggregator AND def snapshot"
- path: "backend/services/segmented_download.py"
provides: "Opt-in multi-connection Range downloader for legacy-LFS repos (auth-safe, resume, verify, cancel)"
contains: "async def segmented_download AND Range"
- path: "backend/api/routers/setup/download.py"
provides: "Driven snapshot_download (max_workers+tqdm_class+endpoint), dry_run preflight, dispatch to segmented path, cancel endpoint"
contains: "dry_run AND tqdm_class"
- path: "docs/downloading-models.md"
provides: "User docs for download speed, fast-download status, HDD/high-perf toggles, mirror/restricted-network"
contains: "Xet"
key_links:
- from: "install_model (download.py:122)"
to: "snapshot_download(dry_run=True) preflight"
via: "compute total/cached/remaining before the real download; emit 'install_plan'"
pattern: "dry_run\\s*=\\s*True"
- from: "snapshot_download / segmented_download byte updates"
to: "DownloadAggregator -> single 'aggregate' SSE event"
via: "tqdm_class forwards bytes into the aggregator; segmented path calls aggregator.add() directly"
pattern: "aggregate"
- from: "dispatch in install_model"
to: "segmented_download vs snapshot_download"
via: "prefs accelerator toggle AND repo-is-LFS classification (FDL-00 helper)"
pattern: "segmented_download"
- from: "system_info (system.py:245)"
to: "fast_download status block"
via: "probe hf_xet import + version + HF_XET_HIGH_PERFORMANCE"
pattern: "fast_download"
---
<objective>
Make model downloads as fast as possible AND show accurate speed / downloaded / remaining / ETA.
**Framing (validated by research — see 260613-fdl-RESEARCH below):** HuggingFace's **hf-xet** backend ALREADY implements the "IDM/uGet technique" — content-defined chunking, parallel byte-range fetches with adaptive concurrency, dedup, and automatic resume — and does it auth-safely. It ships by default in modern `huggingface_hub` and `hf_xet` is already installed here (huggingface_hub 1.7.2). HF closed the multi-connection-downloader feature request as "solved by Xet." So we do NOT build a custom segmented downloader as the default path; that would be redundant and would violate the cross-platform-parity rule.
What's actually missing:
1. **We don't drive Xet well.** `install_model` calls `snapshot_download(**dl_kwargs)` with no `max_workers`, no `tqdm_class`, no `dry_run`, and no explicit dependency pin — progress rides on a global tqdm monkeypatch.
2. **No pre-flight total**, so "downloaded/remaining" has no denominator until files appear, and aggregate speed is summed frontend-side from per-file events (inaccurate under parallel fetch).
3. **Legacy non-Xet (LFS) repos get zero intra-file parallelism** — this is the one place a real IDM-style multi-connection fetch still helps, so we add it as an OPT-IN accelerator.
Five waves, in order (each independently shippable, continuous-to-main per v0.3.0 cadence):
- **Wave 0 — Spike/gate (FDL-00):** classify every catalog repo Xet vs LFS. Sizes Wave 3's value; if ~all repos are Xet-backed, Wave 3 is low-priority polish.
- **Wave 1 — Maximize + guarantee Xet (FDL-01..04):** pin deps, drive snapshot_download explicitly, surface fast-download status, opt-in high-perf/HDD knobs. No new deps, all platforms.
- **Wave 2 — Accurate progress (FDL-05..07):** dry_run preflight + backend aggregate tracker + overall UI bar (speed/remaining/ETA). The biggest user-visible win.
- **Wave 3 — Opt-in segmented accelerator (FDL-08..09):** custom httpx Range downloader for LFS repos. Default OFF, opt-in toggle.
- **Wave 4 — Mirror path + docs (FDL-10..12):** opt-in HF_ENDPOINT, cancel endpoint, docs-sync.
Out of scope / explicitly rejected (call out, do NOT do):
- **hf_transfer / HF_HUB_ENABLE_HF_TRANSFER** — deprecated, breaks progress callbacks. Never enable.
- **Bundling aria2c** — per-OS GPLv2 binary + parity burden; the custom httpx path covers the same need without a binary.
- **Making the segmented downloader the default** — redundant vs Xet, violates parity rule. Always opt-in.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/STATE.md
@./CLAUDE.md
@.planning/quick/260613-fdl-fast-model-downloads/260613-fdl-RESEARCH.md
# Files under edit (read before editing)
@backend/api/routers/setup/download.py
@backend/api/routers/setup/models.py
@backend/utils/hf_progress.py
@backend/api/routers/system.py
@frontend/src/pages/Settings.jsx
# Reference only — patterns, do NOT modify
@backend/core/prefs.py
@frontend/src/api/setup.ts
@frontend/src/api/hooks.ts
<interfaces>
<!-- Verified during planning against the live env (huggingface_hub 1.7.2, hf_xet installed). -->
huggingface_hub 1.7.2 snapshot_download params (confirmed via inspect):
repo_id, repo_type, revision, cache_dir, local_dir, library_name, library_version,
user_agent, etag_timeout, force_download, token, local_files_only,
allow_patterns, ignore_patterns, max_workers, tqdm_class, headers, endpoint, dry_run
- dry_run=True -> returns per-file info incl. size + cached/not-cached (use for FDL-05 preflight).
- tqdm_class=<cls> -> drives the AGGREGATE bar; Xet feeds bytes into it (this is the xet-aware progress hook).
- max_workers -> parallel FILES (default 8); orthogonal to Xet intra-file chunk parallelism.
- endpoint -> per-call HF endpoint override (FDL-10 mirror, instead of process-wide HF_ENDPOINT).
backend/utils/hf_progress.py (existing):
- Monkeypatches huggingface_hub.utils.tqdm.tqdm -> TrackedTqdm (install() at startup).
- register_listener/unregister_listener; emit(event); current_repo_id contextvar stamps events.
- TrackedTqdm.update()/display() emit per-file {filename, downloaded, total, pct, rate, phase} throttled ~0.3s.
- GAP: per-file only, no aggregate, no preflight total. Wave 2 adds the aggregator on top (keep TrackedTqdm; feed it).
backend/api/routers/setup/download.py (existing):
- install_model (line 122): snapshot_download(**dl_kwargs) inside asyncio.to_thread; 5-retry backoff; heartbeat;
_validate_snapshot_has_weights (line 55); _install_cooldowns (line 27, see MM2-06 for bounding).
- SSE feed: GET /setup/download-stream (line 80) forwards hf_progress events.
backend/core/prefs.py:
- resolve(key, *, env=None, default=None) (line 75) — env wins, then store, then default. Use for all new toggles.
Xet env knobs (research): HF_XET_HIGH_PERFORMANCE=1 (opt-in max throughput; needs RAM/bandwidth),
HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1 (HDD), HF_XET_NUM_CONCURRENT_RANGE_GETS (default 16),
HF_XET_DATA_PROGRESS_UPDATE_INTERVAL (200ms). hf_xet is 64-bit only.
</interfaces>
</context>
<tasks>
<!-- ════════════ WAVE 0 — SPIKE / GATE ════════════ -->
<task type="auto">
<name>Task 0 (FDL-00): Classify catalog repos Xet vs LFS</name>
<files>.planning/quick/260613-fdl-fast-model-downloads/260613-fdl-SPIKE.md</files>
<action>
For every repo in backend/config/models.yaml (25 entries), determine whether it's Xet-backed or legacy Git-LFS. Use huggingface_hub: `HfApi().repo_info(repo_id, files_metadata=True)` and inspect each LFS blob for xet info, OR call the model-info endpoint and check the `xetEnabled`/blob `xet` field. For gated/unavailable repos, record "unknown (gated/offline)".
Write 260613-fdl-SPIKE.md: a table repo_id | role | backend (xet|lfs|unknown) | size, plus a one-line GO/LOW-PRIORITY verdict for Wave 3:
- If the majority of *user-facing default* models (OmniVoice TTS, the default ASR) are Xet-backed -> Wave 3 is LOW priority (xet already fast); still build it for the LFS long tail.
- If many defaults are still LFS -> Wave 3 is HIGH priority.
This is read-only network classification — do not download anything (use repo_info, not snapshot_download).
</action>
<verify>
<automated>test -f .planning/quick/260613-fast-model-downloads/260613-fdl-SPIKE.md || test -f .planning/quick/260613-fdl-fast-model-downloads/260613-fdl-SPIKE.md && echo "spike written"</automated>
</verify>
<done>SPIKE.md lists every catalog repo with its storage backend and a GO/LOW-PRIORITY verdict for Wave 3.</done>
</task>
<!-- ════════════ WAVE 1 — MAXIMIZE + GUARANTEE THE XET FAST PATH ════════════ -->
<task type="auto">
<name>Task 1 (FDL-01): Pin huggingface_hub + hf-xet explicitly</name>
<files>pyproject.toml</files>
<action>
Today huggingface_hub arrives transitively (1.7.2) and hf_xet is present but unpinned. Add explicit runtime pins so the fast path can never silently disappear on a resolve:
- huggingface_hub>=1.7 (keep compatible with transformers>=5.3.0 already in deps)
- hf-xet>=1.1 (the Xet backend; 64-bit only — fine for all OmniVoice targets)
Do NOT add hf_transfer. Run `uv sync` then `uv tree huggingface_hub` to confirm a single resolved version (no duplicate). If a transitive constraint conflicts, prefer the higher version and note it in the SUMMARY.
</action>
<verify>
<automated>grep -n "huggingface_hub\|hf-xet\|hf_xet\|hf-transfer\|hf_transfer" pyproject.toml</automated>
<automated>uv run python -c "import huggingface_hub,hf_xet; print('hub',huggingface_hub.__version__,'xet ok')"</automated>
</verify>
<done>pyproject pins huggingface_hub>=1.7 and hf-xet; no hf_transfer; uv resolves cleanly with one huggingface_hub.</done>
</task>
<task type="auto">
<name>Task 2 (FDL-02): Drive snapshot_download explicitly</name>
<files>backend/api/routers/setup/download.py</files>
<action>
In install_model's _do() (line ~148), build dl_kwargs with explicit, intentional args instead of the bare call:
- tqdm_class=<the TrackedTqdm class> so progress is deterministic and xet-aware rather than relying solely on the global monkeypatch. Expose TrackedTqdm from hf_progress (add a getter, e.g. hf_progress.tracked_tqdm_class()).
- max_workers: keep default 8 (don't crank — xet does intra-file parallelism; high max_workers multiplies buffer pressure). Make it prefs-overridable: prefs.resolve("download_max_workers", env="OMNIVOICE_DOWNLOAD_MAX_WORKERS", default=8).
- endpoint=prefs.resolve("hf_endpoint", env="HF_ENDPOINT", default=None) — wires FDL-10 mirror without process-wide env.
- Keep the existing 5-retry backoff, heartbeat, and _validate_snapshot_has_weights.
Do not remove the global monkeypatch (other libs — transformers/mlx_whisper — still rely on it); this task just makes the install path drive its own tqdm_class explicitly.
</action>
<verify>
<automated>grep -n "tqdm_class\|max_workers\|endpoint" backend/api/routers/setup/download.py</automated>
<automated>uv run pytest tests/ -k "download or install" -q 2>&amp;1 | tail -15</automated>
</verify>
<done>install_model drives snapshot_download with explicit tqdm_class + max_workers + endpoint; retry/validate intact; tests green.</done>
</task>
<task type="auto">
<name>Task 3 (FDL-03, FDL-04): fast_download status + opt-in xet knobs</name>
<files>backend/api/routers/system.py, backend/api/routers/setup/download.py</files>
<action>
- FDL-03: add a fast_download block to GET /system/info (system.py:245): {xet_enabled: bool, xet_version: str|None, high_performance: bool}. Probe by importing hf_xet (xet_enabled), reading its version, and reading the HF_XET_HIGH_PERFORMANCE env/pref. Must never throw (system_info is called on every Settings load). Log the same line once at startup ("fast download: Xet on (hf_xet X.Y), high_perf=...").
- FDL-04: opt-in knobs via prefs, applied at process/download setup (env wins):
high_performance = prefs.resolve("xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False)
hdd_sequential = prefs.resolve("xet_hdd_sequential_write", env="HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY", default=False)
When set, export the corresponding HF_XET_* env before the snapshot/segmented download runs. Both default OFF (high-perf can hurt low-RAM machines — surface that as a tooltip in Wave 2 UI).
</action>
<verify>
<automated>curl -s http://127.0.0.1:3900/system/info | python3 -c "import json,sys; print(json.load(sys.stdin).get('fast_download'))" 2>/dev/null || grep -n "fast_download" backend/api/routers/system.py</automated>
</verify>
<done>/system/info reports fast_download truthfully; high-perf + HDD knobs resolve via prefs with env precedence, default off; startup logs Xet status.</done>
</task>
<!-- ════════════ WAVE 2 — ACCURATE DOWNLOADED/REMAINING + SPEED ════════════ -->
<task type="auto">
<name>Task 4 (FDL-05): dry_run preflight -> install_plan event</name>
<files>backend/api/routers/setup/download.py</files>
<action>
Before the real download in install_model, run snapshot_download(repo_id, dry_run=True, endpoint=...) on the worker thread. From the returned per-file info compute: total_bytes, cached_bytes (files already present), to_download_bytes, n_files, n_cached. Emit a new phase event:
{repo_id, phase:"install_plan", total_bytes, cached_bytes, to_download_bytes, n_files, n_cached}
This gives the UI an accurate denominator and a "M GB already cached, N GB to download" line BEFORE bytes flow. Wrap dry_run in try/except — if it fails (older/gated repo), emit install_plan with totals=None and fall back to today's behavior (denominator fills in as files appear). dry_run must respect the 'resolving' heartbeat (it can take a couple seconds).
</action>
<verify>
<automated>grep -n "dry_run\|install_plan\|to_download_bytes" backend/api/routers/setup/download.py</automated>
<automated>uv run pytest tests/backend/setup/test_download_preflight.py -q 2>&amp;1 | tail -15</automated>
</verify>
<done>An install emits install_plan with accurate total/cached/remaining before download; dry_run failure degrades gracefully to old behavior.</done>
</task>
<task type="auto">
<name>Task 5 (FDL-06): Backend aggregate progress tracker</name>
<files>backend/utils/download_aggregator.py, backend/utils/hf_progress.py</files>
<action>
New backend/utils/download_aggregator.py: a per-repo DownloadAggregator that owns the TRUTH for overall progress, so the frontend stops summing potentially-misrouted per-file events.
- Seeded by the install_plan totals (total_bytes, n_files).
- add(filename, bytes_delta) / set_file(filename, downloaded, total): track bytes per file; bytes_done = sum.
- Rate: sampled over a sliding window (e.g. last ~5-10s of (t, bytes_done) samples), not a single tqdm bar's rate. eta = remaining / rate.
- snapshot() -> {repo_id, bytes_done, total_bytes, rate, eta_seconds, files_done, files_total, phase}.
- Emits one throttled (~0.3-0.5s) phase:"aggregate" event via hf_progress.emit().
Wire it: hf_progress's TrackedTqdm._emit_progress already has per-file (filename, downloaded, total) — also feed those into the active repo's aggregator (look up by current_repo_id). The segmented downloader (Wave 3) calls aggregator.add() directly. Keep the per-file events too (UI detail view) — aggregate is additive, not a replacement.
</action>
<verify>
<automated>uv run python -c "from utils.download_aggregator import DownloadAggregator as A; a=A('r',total_bytes=100,files_total=2); a.set_file('f1',50,50); a.set_file('f2',25,50); s=a.snapshot(); print(s['bytes_done'], s['total_bytes'])"</automated>
</verify>
<done>DownloadAggregator sums bytes across parallel files, samples rate over a window, emits a single 'aggregate' event; fed by both tqdm and the segmented path.</done>
</task>
<task type="auto">
<name>Task 6 (FDL-07): Frontend overall progress bar</name>
<files>frontend/src/pages/Settings.jsx, frontend/src/api/setup.ts</files>
<action>
- setup.ts: extend SetupProgressEvent phase union with "install_plan" | "aggregate" and their fields (total_bytes, cached_bytes, to_download_bytes, n_files, n_cached, bytes_done, rate, eta_seconds, files_done, files_total).
- Settings.jsx ModelStoreTab: when an aggregate event arrives for a repo, render ONE overall progress row: a bar (bytes_done/total_bytes), instantaneous speed (format rate as MB/s), "X.X GB of Y.Y GB" downloaded/remaining, and ETA (mm:ss from eta_seconds). Seed the denominator from install_plan (show "M GB cached, N GB to download" before bytes flow). Keep the existing per-file rows as a collapsible "details" section instead of the primary display. Show a small "⚡ fast download" badge when /system/info fast_download.xet_enabled is true.
- Prefer the backend aggregate's rate/eta over the frontend's own per-file ETA computation (Settings.jsx ~614-631) — replace that local ETA math with the aggregate fields; keep a fallback if no aggregate event has arrived yet.
</action>
<verify>
<automated>cd frontend && bun run typecheck 2>&amp;1 | tail -15</automated>
<automated>grep -n "aggregate\|install_plan\|eta_seconds\|fast download" frontend/src/pages/Settings.jsx frontend/src/api/setup.ts</automated>
</verify>
<done>UI shows one overall bar with live speed + downloaded/remaining + ETA from the aggregate event; per-file detail collapsible; fast-download badge; typecheck passes.</done>
</task>
<!-- ════════════ WAVE 3 — OPT-IN IDM-STYLE SEGMENTED ACCELERATOR (LFS REPOS) ════════════ -->
<task type="auto">
<name>Task 7 (FDL-08): Custom httpx segmented downloader</name>
<files>backend/services/segmented_download.py, tests/backend/services/test_segmented_download.py</files>
<action>
New backend/services/segmented_download.py — an OPT-IN multi-connection Range downloader for ONE file (the IDM/uGet technique) used only for legacy-LFS repos where Xet gives no intra-file parallelism. httpx is already a dep.
Contract (async def segmented_download(url, dest, *, token, expected_size, expected_etag=None, num_connections=8, chunk_aggregator=None, cancel_event=None)):
1. HEAD (or GET Range: bytes=0-0) the resolve URL to learn size + Accept-Ranges + the redirect target. If server doesn't honor Range (Accept-Ranges != bytes) -> fall back to a single streamed GET (still works, just not parallel).
2. AUTH SAFETY (critical): send Authorization: Bearer <token> ONLY to the huggingface.co host. When the resolve URL 302-redirects to the CDN (cloudfront/etc.), do NOT forward Authorization to the CDN host — the presigned URL already carries auth. Follow redirects manually so you control header propagation per-host.
3. Split expected_size into num_connections ranges; download each with Range: bytes=start-end concurrently (asyncio + httpx.AsyncClient). Write to dest+".part" at the right offsets (preallocate, or per-range temp files then concat).
4. RESUME: if dest+".part" exists with a sidecar manifest of completed ranges, skip completed ranges.
5. CANCEL: check cancel_event between chunks; on cancel, leave the .part for resume and raise CancelledError.
6. VERIFY: after assembly, check size == expected_size and (if given) sha256/etag matches; only then atomically rename .part -> dest. On mismatch, raise (caller's retry/validate handles it).
7. PROGRESS: call chunk_aggregator.add(filename, bytes_delta) as ranges complete bytes (feeds DownloadAggregator).
Tests (use a local mock HTTP server / httpx MockTransport): honors Range + parallel assembly == single-GET bytes; falls back when Accept-Ranges absent; does NOT send Authorization to a different host on redirect; resumes from a partial .part; cancels and leaves resumable state; size/etag mismatch raises.
</action>
<verify>
<automated>uv run pytest tests/backend/services/test_segmented_download.py -q 2>&amp;1 | tail -20</automated>
<automated>grep -n "Authorization\|Range\|cancel_event\|expected_size" backend/services/segmented_download.py</automated>
</verify>
<done>segmented_download fetches a file via parallel ranges, is auth-safe across the CDN redirect, resumes, cancels, and verifies size/etag before commit; all tests pass.</done>
</task>
<task type="auto">
<name>Task 8 (FDL-09): Dispatch — accelerator for LFS repos only</name>
<files>backend/api/routers/setup/download.py, backend/api/routers/setup/models.py</files>
<action>
- models.py: add a small helper is_xet_backed(repo_id) -> bool|None (reuse FDL-00's classification approach; cache result). Used to decide the path.
- download.py install_model dispatch:
accelerator_on = prefs.resolve("segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=False)
if accelerator_on and is_xet_backed(repo_id) is False:
-> resolve each LFS file's URL via hf_hub_url + HfApi file metadata, download via segmented_download into the HF cache layout (or download to a temp dir then place via the cache API so the result is a normal cache entry), feeding the same DownloadAggregator. Run _validate_snapshot_has_weights at the end.
else:
-> existing snapshot_download path (xet).
IMPORTANT: the segmented result MUST land in the same HF cache structure so /models install-state, delete, and is_cached() all keep working (truth: "indistinguishable from snapshot_download"). If matching the blob/snapshot symlink layout is too fiddly, the safe fallback is: segmented-download to a temp file, then hand the bytes to huggingface_hub so it finalizes the cache entry. Document the chosen approach in SUMMARY.
Default OFF -> zero behavior change unless the user opts in.
</action>
<verify>
<automated>grep -n "segmented_downloader\|is_xet_backed\|segmented_download" backend/api/routers/setup/download.py backend/api/routers/setup/models.py</automated>
<automated>uv run pytest tests/ -k "download or install or model" -q 2>&amp;1 | tail -20</automated>
</verify>
<done>With the toggle ON, LFS repos download via the segmented path into the normal HF cache; Xet repos and the default (toggle OFF) use snapshot_download; install-state/delete unaffected.</done>
</task>
<!-- ════════════ WAVE 4 — MIRROR PATH + CANCEL + DOCS ════════════ -->
<task type="auto">
<name>Task 9 (FDL-10, FDL-11): Mirror opt-in + cancel endpoint</name>
<files>backend/api/routers/setup/download.py</files>
<action>
- FDL-10: the endpoint= wiring from Task 2 already reads prefs hf_endpoint. Surface it as a setting and document (Task 10) that a mirror routes through the CLASSIC LFS path (no Xet) — so it pairs naturally with the FDL-08 segmented accelerator for speed on mirrors. No process-wide HF_ENDPOINT mutation; per-call endpoint only.
- FDL-11: add POST /models/install/cancel {repo_id} that sets the repo's cancel_event (segmented path) and, for the snapshot path, best-effort marks the install cancelled (snapshot_download isn't trivially cancellable mid-file — at minimum stop retries and emit install_cancelled). Compose with MM2-06: on success OR cancel, clear the _install_cooldowns entry so a cancelled download isn't rate-limited. Emit phase:"install_cancelled".
</action>
<verify>
<automated>grep -n "install/cancel\|cancel_event\|install_cancelled\|hf_endpoint" backend/api/routers/setup/download.py</automated>
</verify>
<done>Per-call mirror endpoint wired (opt-in); cancel endpoint stops the segmented path and clears cooldown; emits install_cancelled.</done>
</task>
<task type="auto">
<name>Task 10 (FDL-12): Docs — downloading-models.md + README pointer</name>
<files>docs/downloading-models.md, README.md</files>
<action>
Per the docs-sync hard rule, document the user-facing surface introduced here:
- How fast downloads work (Xet on by default; what the ⚡ badge means; how to check via Settings/system info).
- Advanced toggles: high-performance mode (warn: needs RAM/bandwidth, can hurt low-RAM machines), HDD sequential-write, max workers, segmented accelerator (opt-in, for legacy-LFS repos), and the mirror/restricted-network HF_ENDPOINT setting (note: mirror = classic LFS, no Xet; pair with the accelerator).
- A short troubleshooting section (slow downloads, stuck at resolving, restricted networks/China).
Add a one-line pointer from README.md to docs/downloading-models.md. Do NOT enable any opt-in by default in docs examples.
</action>
<verify>
<automated>test -f docs/downloading-models.md && grep -n "Xet\|HF_ENDPOINT\|high-performance\|segmented" docs/downloading-models.md | head</automated>
<automated>grep -n "downloading-models" README.md</automated>
</verify>
<done>docs/downloading-models.md covers speed, status, all opt-in knobs, mirror/restricted-network, troubleshooting; README links it; no opt-in shown as default.</done>
</task>
</tasks>
<verification>
Gate per wave; full set before the last PR:
1. `uv run pytest tests/backend/setup/test_download_preflight.py tests/backend/services/test_segmented_download.py tests/ -k "download or install or model" -q` — green.
2. Live smoke (backend running): an install emits install_plan (accurate total/cached/remaining) THEN aggregate events with rising bytes_done + a non-zero rate + decreasing ETA; on completion bytes_done == total_bytes.
3. /system/info reports fast_download.xet_enabled=true with a version.
4. Auth-safety unit test proves Authorization is NOT sent to a non-huggingface.co host on redirect.
5. Default-off proof: with no opt-in set, an install uses snapshot_download (xet) — `OMNIVOICE_SEGMENTED_DOWNLOAD` unset means the segmented path is never taken.
6. `uv tree huggingface_hub` shows one version; no hf_transfer anywhere (`grep -ri hf_transfer backend/` is empty).
7. `cd frontend && bun run typecheck` passes.
8. Cross-platform parity: the default path (Xet, pure-Python) is identical on all 3 OSes; every accelerator/mirror/high-perf knob is opt-in (Settings/env). No bundled binary added.
</verification>
<success_criteria>
- Fast: Xet is pinned, engaged, and driven with explicit args; high-perf/HDD knobs available opt-in; legacy-LFS repos can use the opt-in segmented accelerator for real multi-connection speed.
- Accurate: UI shows pre-flight total/cached/remaining, then one overall bar with live speed + downloaded/remaining + ETA sourced from a backend aggregate (not frontend guesswork).
- Safe & compatible: no hf_transfer; segmented downloader is opt-in, auth-safe, resumable, verified, cancellable, and lands in the normal HF cache; default behavior identical on all 3 OSes; no new on-disk model state; existing installs untouched.
- All listed tests + typecheck pass; docs updated in the same PR (docs-sync rule).
</success_criteria>
<risks>
- **Segmented downloader auth leak (FDL-08) — highest risk.** Forwarding the HF Authorization header to the CDN host on redirect would leak the token. Mitigation: manual redirect handling, per-host header allow-list (Authorization only to huggingface.co), and a dedicated unit test asserting no Authorization on the CDN hop. This is a must-have truth, not optional.
- **Cache-layout mismatch (FDL-09).** If the segmented path writes files outside the HF cache blob/snapshot structure, /models install-state + delete + is_cached() break. Mitigation: prefer the temp-file-then-hand-to-huggingface_hub finalization approach over hand-rolling the symlink/blob layout; assert is_cached(repo_id) is true after a segmented install in a test.
- **dry_run cost/availability (FDL-05).** dry_run adds a metadata round-trip and may not exist for gated/older repos. Mitigation: try/except -> totals=None fallback to current fill-in-as-you-go behavior; keep the resolving heartbeat so the UI isn't blank during preflight.
- **Aggregate vs per-file double-count (FDL-06).** Feeding both tqdm per-file events and the aggregator risks the UI showing two competing numbers. Mitigation: aggregate is the single source of truth for the overall bar; per-file events only drive the collapsible detail view; the frontend's old per-file ETA math is removed (Task 6).
- **High-performance mode hurting low-RAM machines (FDL-04).** HF_XET_HIGH_PERFORMANCE can need ~tens of GB RAM. Mitigation: default OFF, opt-in only, tooltip warning in the UI.
- **Mirror + Xet confusion (FDL-10).** Users may expect Xet speed through a mirror; mirrors fall back to classic LFS. Mitigation: document explicitly; that's exactly why the segmented accelerator pairs with the mirror path.
- **Scope: do not let the segmented path become default.** It's opt-in for LFS repos only. Xet stays the default; making it default would regress dedup + violate the parity rule.
</risks>
<output>
Write 260613-fdl-SPIKE.md (Task 0) and 260613-fdl-SUMMARY.md when done. SUMMARY must record: the Xet-vs-LFS catalog breakdown and how it changed Wave 3 priority; the cache-finalization approach chosen for the segmented path (and the is_cached-after-segmented test result); the exact new SSE event shapes (install_plan, aggregate); which opt-in prefs keys + env vars were added; and the auth-safety test output. Note any "use judgment" decision an executor made.
Docs-sync (CLAUDE.md hard rule): docs/downloading-models.md + README pointer ship in the SAME PR as the user-facing toggles (Task 10). If the Settings UI gains the new toggles, the docs describing them land together.
</output>
@@ -1,43 +0,0 @@
# RESEARCH — Fast HuggingFace model downloads (2026)
**Date:** 2026-06-13 · **For:** 260613-fdl-PLAN.md
## Bottom line
As of mid-2026 the fast path is **hf-xet, on by default** in modern `huggingface_hub`. Xet is itself a chunk-level, content-defined, massively-parallel downloader with adaptive concurrency — it **is** the "IDM/uGet-style segmented download," done for you and dedup-aware. `hf_transfer` is **deprecated**. Rolling your own segmented downloader or bridging to aria2 is **not worth it as a default**; the only thing we must build is (a) better driving + progress UI and (b) an **opt-in** segmented path for the legacy-LFS long tail (repos Xet doesn't back).
Installed in this repo: `huggingface_hub 1.7.2`, `hf_xet` present. `snapshot_download` here supports `max_workers`, `tqdm_class`, `endpoint`, `dry_run` (confirmed via inspect).
## 1. hf-xet — USE (default, no action needed beyond pinning)
Content-defined chunks grouped into blocks ("xorbs") in a content-addressable store; download = send file SHA256 → get reconstruction metadata + presigned URLs → fetch needed xorb ranges **in parallel** → reassemble; already-present chunks skipped (dedup). Auto-used by `snapshot_download`/`hf_hub_download` for Xet-backed repos since huggingface_hub 0.32. 23× over Git-LFS, up to ~1 GB/s.
Knobs (defaults already tuned): `HF_XET_NUM_CONCURRENT_RANGE_GETS` (16), adaptive concurrency ON (max 64), `HF_XET_DATA_MAX_CONCURRENT_FILE_DOWNLOADS` (8), chunk cache disabled by default (better for pure download), `HF_XET_HIGH_PERFORMANCE=1` (opt-in max throughput, needs RAM/bandwidth), `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1` (HDD). **64-bit only.**
- https://huggingface.co/docs/huggingface_hub/en/guides/download
- https://huggingface.co/docs/hub/en/xet/using-xet-storage
- https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables
## 2. hf_transfer — AVOID (deprecated)
`HF_HUB_ENABLE_HF_TRANSFER` flagged deprecated; Xet supersedes it. Historically **broke tqdm progress / had no callbacks** — directly conflicts with the accurate-progress goal. Successor for max throughput is `HF_XET_HIGH_PERFORMANCE=1`.
- https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables
- https://github.com/huggingface/hf_transfer/issues/63
## 3. huggingface_hub native concurrency — USE defaults
`snapshot_download(max_workers=...)` = parallel FILES (default 8), orthogonal to Xet's intra-file chunk parallelism. For OmniVoice's 1few-large-file models the win is mostly Xet's intra-file parallelism; don't crank max_workers (multiplies buffer pressure). Resume is automatic via cache + ETag (no `resume_download` flag to manage).
- https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
## 4. Custom IDM-style Range downloader — AVOID as default, BUILD as opt-in for LFS
`/resolve/<rev>/<file>` 302-redirects to CDN (Cloudfront) which honors Range + parallel byte-ranges. Catch: follow redirect, **do NOT forward Authorization to the CDN host** (presigned URL carries auth), verify ETag/sha256, auth on first hop only. Redundant vs Xet for Xet-backed repos (HF closed issue #3232 as "use Xet"), **but genuinely helps non-Xet/legacy-LFS repos** which get no intra-file parallelism. → our Wave 3 opt-in.
- https://github.com/huggingface/huggingface_hub/issues/3232
## 5. aria2 — OPTIONAL, rejected for OmniVoice
`aria2c -x16 -s16 -c --header="Authorization: Bearer <token>"` is 35× on plain LFS, but: no dedup (worse than Xet for Xet repos), per-OS GPLv2 binary to package (parity burden — would have to be opt-in anyway), stdout/RPC progress scraping. The custom httpx path covers the same need with no binary. → not bundled.
- https://gist.github.com/padeoe/697678ab8e528b85a2a7bddafea1fa4f
## 6. Mirrors / HF_ENDPOINT — OPTIONAL, region-gated, breaks Xet
`HF_ENDPOINT=https://hf-mirror.com` redirects Hub traffic (standard for China). **Xet CAS/presigned URLs point at HF infra → mirrors generally don't serve the Xet protocol → traffic falls back to classic LFS** (no dedup, no Xet parallelism). So mirror and Xet fast-path are mutually exclusive; the realistic China stack is mirror + LFS + (our opt-in) segmented accelerator. → our Wave 4 opt-in, per-call `endpoint=` not process-wide.
## 7. Progress / speed — USE `tqdm_class` (xet-aware) + `dry_run` preflight
Unlike hf_transfer, **Xet reports progress through the same tqdm interface**; huggingface_hub aggregates per-file/thread bytes into a shared bar and feeds the `tqdm_class` you pass. So `snapshot_download(tqdm_class=...)` yields reliable aggregate bytes/total/rate/ETA even under parallel fetch. `snapshot_download(dry_run=True)` returns per-file sizes + cached flags → use for "will download X of Y, N GB" preflight. Speed sampling tunable via `HF_XET_DATA_PROGRESS_UPDATE_INTERVAL` (200ms).
- https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
- https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/_snapshot_download.py
## Recommended architecture (→ plan)
Pin `huggingface_hub>=1.7` + `hf-xet`; let Xet be the default (it IS the IDM technique). Drive `snapshot_download(repo_id, tqdm_class=OmniVoiceProgress, max_workers=8, endpoint=<opt-in mirror>)`; `dry_run=True` first for total/remaining; aggregate bytes in a backend tracker → one overall bar (speed/remaining/ETA). Opt-in only: `HF_XET_HIGH_PERFORMANCE` (max speed), `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY` (HDD), a custom httpx **segmented downloader for legacy-LFS repos**, and an `HF_ENDPOINT` mirror (classic-LFS fallback). Never enable hf_transfer; never bundle aria2; never make the segmented path the default.
@@ -1,28 +0,0 @@
# SPIKE — FDL-00: Catalog Xet vs LFS classification
**Date:** 2026-06-13 · **Method:** HF API `GET /api/models/{repo}?expand[]=xetEnabled` (authoritative).
## Result: 25 / 25 catalog repos are Xet-backed
| backend | count |
|---------|-------|
| xet | 25 |
| lfs | 0 |
| unknown | 0 |
Every repo in `backend/config/models.yaml` — including both first-run defaults (`k2-fsa/OmniVoice` TTS, `Systran/faster-whisper-large-v3` ASR) — returns `xetEnabled: true`. Full list: all entries under TTS / ASR / Diarisation (k2-fsa, Systran×5, mlx-community×9, openai, nvidia×2, UsefulSensors×2, pyannote, OpenMOSS, KittenML, deepdml).
## Detection caveat (important for the executor)
The installed client is **huggingface_hub 1.7.2**, whose `repo_info(..., files_metadata=True)` siblings expose only `blob_id, lfs, rfilename, size`**no `xet_file`, and no `xet_enabled` on the info object.** A first pass that inferred backend from siblings wrongly reported "0/25 xet, all LFS." Do **not** classify Xet status from `repo_info` siblings on this client version. The reliable signal is the Hub API `xetEnabled` expand field (used here) or `hf_xet` actually engaging at download time. Re-check after any `huggingface_hub` bump — newer versions surface `xet_enabled` directly.
## Verdict for Wave 3 (segmented accelerator): LOW priority
Because the entire current catalog is Xet-backed and `hf_xet` is installed, Xet already provides chunked parallel range-gets (the IDM/uGet behavior) for **every** model we ship. The custom segmented downloader (Wave 3) is therefore **not needed to speed up any current default model** — it remains valuable only for:
- the **mirror / restricted-network path** (Wave 4: `HF_ENDPOINT` falls back to classic LFS, no Xet), and
- any **future non-Xet repo** a user adds.
**Recommendation:** proceed with W1 (maximize/guarantee Xet) and W2 (accurate progress) as the real wins for today's catalog; keep W3 as opt-in, build it alongside W4's mirror path where it actually pays off. This matches the PLAN's original framing — confirmed, not changed.
## Consequence for W1/W2 framing
W1 "guarantee the Xet fast path" is correctly the primary lever: these repos download via Xet **only if** the client engages it (hf_xet installed ✓ + huggingface_hub recent ✓). The W2 live smoke test should confirm Xet is actually used (fast parallel aggregate progress on a real install), since `xetEnabled=true` is a Hub-side capability, not proof the client took the Xet path.
@@ -1,49 +0,0 @@
# SUMMARY — FDL Waves 02 (fast model downloads)
**Date:** 2026-06-13 · **Scope shipped:** W0 (spike), W1 (maximize Xet), W2 (accurate progress). W3/W4 deferred.
## What landed
**W0 — spike (FDL-00).** Classified all 25 `models.yaml` repos via the HF API `xetEnabled` field → **25/25 Xet-backed** (incl. both first-run defaults). See `260613-fdl-SPIKE.md`. Verdict: Wave 3 (segmented accelerator) is **LOW priority** — Xet already gives parallel chunked transfer for every shipped model. Detection caveat recorded: `repo_info` siblings on hf_hub **1.7.2** expose no xet metadata; classify via the `xetEnabled` API field, not siblings.
**W1 — maximize + guarantee Xet (FDL-01..04).**
- `pyproject.toml`: pinned `huggingface_hub>=1.7` + `hf-xet>=1.1` explicitly (was transitive/unpinned); no `hf_transfer`. Resolves to hf_hub 1.7.2 / hf-xet 1.4.2, single version.
- `download.py`: `install_model` now drives `snapshot_download` with explicit `tqdm_class` (our progress-emitting subclass), `max_workers` (prefs `download_max_workers`, default 8), and `endpoint` (prefs `hf_endpoint` — W4 hook). `apply_xet_env()` applies opt-in `HF_XET_HIGH_PERFORMANCE` + `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY` (both default OFF, env wins).
- `system.py`: `/system/info` now returns `fast_download {xet_enabled, xet_version, high_performance}`; logged once at startup.
**W2 — accurate downloaded/remaining + speed (FDL-05..07).**
- Preflight `snapshot_download(dry_run=True)``compute_plan()``install_plan` SSE event with `total_bytes / cached_bytes / to_download_bytes / n_files / n_cached` **before bytes flow**. Degrades to totals=None on gated/older repos.
- New `utils/download_aggregator.py`: one source of truth for overall progress. Fed by a byte-sink on the patched tqdm; distinguishes byte bars (unit 'B', keyed by bar id) from the "Fetching N files" count bar; emits one throttled `aggregate` event (bytes_done/total/windowed rate/eta/files).
- Frontend `Settings.jsx` + `setup.ts`: overall bar driven by the aggregate; bar % = `max(byte%, file%)`; shows cached-skip + files-progress; `⚡ fast download` badge from `/system/info`. i18n keys added to `en.json`.
## Rebase reconciliation (main disabled Xet)
Rebasing onto latest main surfaced that main now sets **`HF_HUB_DISABLE_XET=1`** (main.py) — a deliberate choice to force the classic LFS path because Xet's progress bypasses the tqdm hook (the exact limitation found here). Reconciled rather than fought:
- `fast_download` status now reports the **runtime truth**: `xet_installed` + `xet_active` (active = installed AND not disabled) + `xet_enabled` alias. Default `xet_active=false`; the ⚡ badge only shows when Xet actually runs. Startup log: `downloads: Xet disabled → legacy LFS …`.
- Docs rewritten: default backend is **legacy LFS for accurate progress**; Xet is opt-in via `HF_HUB_DISABLE_XET=0` (coarser progress). The hf-xet pin stays (harmless; ready for a future Xet progress hook).
- Net: W2's progress is the value either way; W1's "maximize Xet" is dormant by main's design, not removed.
## Decisions / "use judgment" notes
- **Xet progress limitation (verified by live smoke).** Under Xet + hf_hub 1.7.2 the per-file **byte** bars never advance `n` and never `close()` through our tqdm (Xet fetches chunks out-of-band). Only the **file-count bar** is live. So: mid-download the overall bar is **file-granular** (moves 0→N files), and `complete()` flushes `bytes_done` to the exact preflight total on success (verified: final `74420620/74420620`, files 4/4). True live byte-speed is only available on classic-LFS/mirror repos (W4). This is a real constraint, not a bug — documented here and worth surfacing in W4 docs.
- Per-file detail kept inline (existing single-line summary, now aggregate-sourced) rather than a new collapsible panel — limited risk; can revisit.
## Drive-by fix
- `download.py` imported no `os`, but `_validate_snapshot_has_weights` uses `os.walk` → latent `NameError` on every install. Added `import os`.
## Verification
- `tests/backend/setup/test_download_preflight.py` — 10 pass (compute_plan splits, aggregator byte/count routing, close-credit, windowed rate/eta, registry feed + finish noop).
- `pytest -k "download or install or model or engine or setup"` — 149 passed, 7 skipped, 0 failed.
- `frontend typecheck:ci` — exit 0.
- Live smoke (real install of `mlx-community/whisper-tiny-mlx`, then deleted): `install_plan` exact; aggregate files 0→1→4; final bytes==total; `/system/info` + startup log correct.
## W4 — mirror + cancel + docs (FDL-10..12, shipped)
- **Mirror (FDL-10):** `snapshot_download(endpoint=…)` honours prefs `hf_endpoint` / env `HF_ENDPOINT` on both preflight and download — per-call, no process-wide mutation. Documented as the classic-LFS (non-Xet) path that restores continuous byte-speed.
- **Cancel (FDL-11):** `POST /models/install/cancel {repo_id}` sets a cancel flag checked at each retry boundary → emits `install_cancelled`, clears the cooldown (cancel ≠ failure). Limitation: an in-flight single-file fetch isn't interruptible in hf_hub 1.7.2; cancel lands at the next retry boundary. Frontend treats `install_cancelled` as a terminator (clears row + refetch).
- **Docs (FDL-12):** `docs/downloading-models.md` (Xet fast path, progress semantics incl. the byte-speed limitation, opt-in tuning knobs, mirror/restricted-network, cancel, troubleshooting) + README pointer. Docs-sync rule satisfied in-PR.
## W3 — opt-in segmented accelerator (FDL-08/09, shipped)
Reprioritised from LOW to HIGH after the rebase: since main forces Xet off, the default path is single-stream legacy LFS, so a segmented downloader is the way to get **both** parallel speed and live byte progress.
- `services/segmented_download.py`: async multi-connection Range downloader for one file — parallel byte-ranges, resume (`.part` + manifest), per-segment short-read truncation guard, optional sha256/etag verify, cancel, single-stream fallback when the server won't range. **Auth-safe**: the HF `Authorization` header goes only to `huggingface.co`/`hf.co`; never forwarded to a CDN host on redirect (unit-tested).
- Dispatch (`download.py`): opt-in via prefs `segmented_downloader` / env `OMNIVOICE_SEGMENTED_DOWNLOAD` (default OFF). When on and Xet inactive, fetches each repo file into the HF cache mirroring `hf_hub_download` (blobs + snapshot symlinks + `refs/main`), feeding **real bytes** to the aggregator. Any failure falls back to `snapshot_download` — the accelerator can never break a correct install.
- Verified live (accelerator ON): real mid-download byte progress (1.5 KB → 71 MB, rate ramping to **16.6 MB/s**), final `bytes_done == total`, `/models` shows `installed: True`, delete frees the right bytes.
- Fixed a `complete()` double-count (was adding a full total on top of accumulated segmented bytes → 2×); now replaces byte bars so the sum is exactly total.
- Tests: `tests/backend/services/test_segmented_download.py` (7 cases) covering parallel range reassembly, single-stream fallback, the auth header reaching only the HF host (never a CDN), size/truncation rejection, cancellation, and byte-callback totals — plus an aggregator double-count regression.
@@ -1,363 +0,0 @@
---
phase: 260613-mm2
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- backend/services/tts_backend.py
- backend/services/model_manager.py
- backend/services/subprocess_backend.py
- backend/services/model_lifecycle.py # NEW
- backend/api/routers/system.py
- backend/api/routers/setup/download.py
- backend/api/routers/setup/models.py
- tests/test_engines.py
- tests/backend/services/test_model_lifecycle.py # NEW
- tests/backend/services/test_subprocess_reaper.py
autonomous: true
requirements:
# ── Tier 1 — Correctness (Wave 1) ──────────────────────────────────────────
- MM2-01 # Registry reuses one active instance and calls unload() on engine switch
- MM2-02 # Per-engine unload() overrides (in-process drop+free_vram; subprocess -> unload_sidecar)
- MM2-03 # /model/loaded + /model/unload report ASR honestly; no unloadable:False-but-loaded lies
# ── Tier 2 — Single lifecycle surface (Wave 2) ─────────────────────────────
- MM2-04 # model_lifecycle facade owns list_loaded/unload/unload_all/free_vram across all 3 worlds
- MM2-05 # Idle/timeout config unified through core.prefs.resolve (env still wins); no duplicated constants
# ── Tier 3 — Robustness & observability (Wave 3) ───────────────────────────
- MM2-06 # _install_cooldowns bounded (evict on success + TTL); no unbounded growth
- MM2-07 # Snapshot weight validation is per-role, not one 5 MB magic number
- MM2-08 # Subprocess sidecars self-report VRAM in pong; panel shows real MB, not 0
- MM2-09 # scan_cache_dir() -> disk-walk fallback logs WHY it fell back (WinError #117/#118)
must_haves:
truths:
- "Switching the active TTS backend in Settings releases the outgoing engine's VRAM before the new one loads — verified by asserting the outgoing instance's unload() was called exactly once on switch."
- "TTSBackend.unload() is overridden by OmniVoiceBackend (drops model ref + free_vram) and by every SubprocessBackend subclass (routes to unload_sidecar); all overrides are idempotent and safe before first generate()."
- "/model/loaded never reports a model as loaded with a misleading unloadable flag: the ASR row's unloadable reflects whether it can actually be released independently of the TTS lifecycle."
- "services.model_lifecycle is the single import surface for list_loaded()/unload(id)/unload_all()/free_vram(); system.py routers call it instead of re-enumerating models inline."
- "Idle timeouts for the in-process model and subprocess sidecars resolve through core.prefs.resolve(... env=...) so an env var still wins and the Settings store can override; no module duplicates IDLE_TIMEOUT_SECONDS by hand."
- "_install_cooldowns cannot grow without bound: entries are removed on successful install and stale entries are evicted by TTL."
- "A live subprocess sidecar reports a non-zero vram_mb in /model/loaded when it actually holds GPU memory (pong carries the figure); CPU-only sidecars report 0 truthfully."
- "When scan_cache_dir() raises and the code falls back to the on-disk walk, the reason is logged at WARNING with the exception type (the #117/#118 WinError-448 path is no longer silent)."
- "uv run pytest tests/test_engines.py tests/backend/services/test_model_lifecycle.py tests/backend/services/test_subprocess_reaper.py tests/test_model_load_timeout.py passes."
- "No on-disk model state changes; no new runtime dependency added; behavior degrades gracefully (not errors) on MPS/CPU where VRAM APIs are sparse."
artifacts:
- path: "backend/services/tts_backend.py"
provides: "Active-instance reuse + unload-on-switch in get_active_tts_backend(); per-engine unload() overrides"
contains: "_active_instance AND (def unload)"
- path: "backend/services/model_lifecycle.py"
provides: "Facade owning list_loaded/unload/unload_all/free_vram across in-process + subprocess models"
contains: "def list_loaded AND def unload_all"
- path: "backend/api/routers/system.py"
provides: "Thin /model/loaded + /model/unload routers delegating to model_lifecycle"
contains: "model_lifecycle"
key_links:
- from: "get_active_tts_backend() (tts_backend.py:1235)"
to: "outgoing backend.unload()"
via: "module-level _active_instance compared against newly-resolved active_backend_id()"
pattern: "_active_instance"
- from: "system.py /model/loaded + /model/unload (system.py:129, 210)"
to: "model_lifecycle.list_loaded() / model_lifecycle.unload()"
via: "import services.model_lifecycle"
pattern: "model_lifecycle\\.(list_loaded|unload)"
- from: "subprocess sidecar pong reply (subprocess_backend.py:435-438)"
to: "list_live_sidecars() vram_mb field"
via: "ping reply carries allocated VRAM measured inside the sidecar process"
pattern: "vram_mb"
---
<objective>
Clean up OmniVoice's model-management subsystem ("v2"). Today load / unload / list / free-VRAM each behave differently across three worlds — the in-process model (`model_manager.py`), the TTS backend registry (`tts_backend.py`), and subprocess sidecars (`subprocess_backend.py`) — with no single lifecycle owner. This produces one real user-facing bug (VRAM leak on engine switch), inaccurate VRAM/unloadable reporting, an unbounded cooldown dict, and a silent cache fallback.
This is **cleanup + correctness, not a rewrite.** The Wave 13 idle-reaper and the SubprocessBackend primitive are sound and stay. The `TTSBackend.unload()` contract already exists as a documented default no-op (`tts_backend.py:149`) explicitly deferred to "Phase 2"; this plan *is* that Phase-2 follow-through — wire the registry to call it, override it per engine, and unify the surrounding surface.
Three tiers, executed in order (each independently shippable, continuous-to-main per the v0.3.0 cadence):
- **Wave 1 / Tier 1 — Correctness:** MM2-01..03. The VRAM leak on switch + honest unload reporting. Highest value; ship first.
- **Wave 2 / Tier 2 — Single lifecycle surface:** MM2-04..05. Extract `model_lifecycle` facade + unify idle/timeout config.
- **Wave 3 / Tier 3 — Robustness & observability:** MM2-06..09. Bounded cooldowns, per-role weight validation, sidecar VRAM self-report, cache-fallback logging.
Output: PRs on branches off `main` (one per wave is fine), each green on the listed pytest selection. No push until the orchestrator merges; tests added with each wave.
Out of scope (call out, do not touch): GPU-pool per-engine sizing (`model_manager.py:42`, `_GPU_VRAM_PER_JOB_GB`) and torch.compile tuning — those are performance, not cleanup, and carry regression risk against #278/#315.
</objective>
<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/STATE.md
@./CLAUDE.md
# Files under edit (read before editing)
@backend/services/tts_backend.py
@backend/services/model_manager.py
@backend/services/subprocess_backend.py
@backend/api/routers/system.py
@backend/api/routers/setup/download.py
@backend/api/routers/setup/models.py
# Reference only — establish patterns, do NOT modify
@backend/core/prefs.py
@tests/test_engines.py
@tests/backend/services/test_subprocess_reaper.py
<interfaces>
<!-- Verified during planning. Executor should use these directly. -->
backend/services/tts_backend.py
- class TTSBackend(ABC) (line 59); unload() default no-op (line 149) — contract already documented:
idempotent, synchronous, safe before first generate().
- OmniVoiceBackend.__init__(self, model=None) (line 174); self._model reuses model_manager singleton.
- _REGISTRY: dict[str, type[TTSBackend]] (line 1109, a _LazyRegistry).
- active_backend_id() (line 1228) -> prefs.resolve("tts_backend", env="OMNIVOICE_TTS_BACKEND", default="omnivoice").
- get_active_tts_backend(*, model=None) (line 1235) — builds a FRESH instance every call, no teardown. THE leak.
backend/core/prefs.py
- resolve(key: str, *, env: Optional[str] = None, default: Any = None) -> Any (line 75) — env wins, then store, then default.
backend/services/model_manager.py
- module global `model` (line 111); `_last_used`; free_vram() (line 678); idle_worker() (line 667).
- IDLE_TIMEOUT_SECONDS imported from core.config (line 33); duplicated as _IDLE_TIMEOUT_SECONDS (line 114). Collapse.
- offload_tts_for_asr() (line 701) / restore_tts_after_asr() — ad-hoc ASR<->TTS VRAM juggling; _diar_pipeline global.
backend/services/subprocess_backend.py
- protocol op set (line 74); SIDECAR_IDLE_TIMEOUT_S = env-only float (line 107) — move to prefs.resolve.
- list_live_sidecars() -> list[dict] (line 181); unload_sidecar(engine_id) (line 199); unload_all_sidecars() (line 205).
- health/ping: _send({"op":"ping"}) then expect {"op":"pong"} (lines 435-438). Add vram_mb to the pong here AND
in the sidecar entry-point that answers ping (search the sidecar worker for the "ping"->"pong" handler).
backend/api/routers/system.py
- GET /model/loaded (line 129) — ~80 lines of inline enumeration of TTS/ASR/diar/sidecars. Replace body with
model_lifecycle.list_loaded().
- POST /model/unload/{model_id} (line 210) — handles "tts" | "diarization" | "sidecar:<id>" | "sidecars".
Replace body with model_lifecycle.unload(model_id).
backend/api/routers/setup/download.py
- _install_cooldowns dict (line 27) — unbounded. _validate_snapshot_has_weights (line 55) + _MIN_WEIGHT_BYTES 5 MB
(line 45) — single magic number across roles.
backend/api/routers/setup/models.py
- scan_cache_dir() with silent disk-walk fallback (~line 268-280) + _scan_cache_on_disk (line 177).
</interfaces>
</context>
<tasks>
<!-- ════════════ WAVE 1 / TIER 1 — CORRECTNESS ════════════ -->
<task type="auto">
<name>Task 1 (MM2-02): Per-engine unload() overrides</name>
<files>backend/services/tts_backend.py</files>
<action>
The base-class `unload()` no-op already exists (tts_backend.py:149) with a documented contract. Override it where it matters. Do this BEFORE Task 2 — the registry switch (Task 2) calls these.
- OmniVoiceBackend (line 162): override `unload(self)`. Drop the local model ref (`self._model = None`) and, because OmniVoice shares the singleton owned by model_manager, also release that: `import services.model_manager as mm; mm.model = None; mm.free_vram()`. Idempotent — guard on `mm.model is not None` before free_vram(). Safe before first generate() (no-op when nothing loaded).
- Every SubprocessBackend subclass: implement `unload(self)` on the SubprocessBackend base (subprocess_backend.py — the duck-typed `_is_subprocess_isolated` class) so all subclasses inherit it. It must call `unload_sidecar(self.id)` (force-shut this engine's sidecar; busy sidecars are skipped, never interrupted — existing semantics). Idempotent: unload_sidecar on a non-running engine returns 0, no raise.
- In-process non-OmniVoice engines that hold their own model (e.g. KittenTTS/VoxCPM2 keep refs in __init__): override unload() to drop the ref + best-effort empty_cache via the existing free_vram() helper if they used GPU. Where an engine genuinely holds nothing resident, leave the base no-op (and note it in the SUMMARY so the future CI gate knows it's intentional, not missed).
Honor the contract comment verbatim: idempotent, synchronous, safe pre-load.
</action>
<verify>
<automated>grep -n "def unload" backend/services/tts_backend.py backend/services/subprocess_backend.py</automated>
<automated>uv run python -c "from services.tts_backend import OmniVoiceBackend; b=OmniVoiceBackend(); b.unload(); b.unload(); print('idempotent ok')"</automated>
</verify>
<done>
- OmniVoiceBackend.unload() drops both self._model and mm.model and calls free_vram(), guarded for idempotency.
- SubprocessBackend.unload() routes to unload_sidecar(self.id); inherited by all subprocess engines.
- Calling unload() twice, and before any generate(), never raises.
</done>
</task>
<task type="auto">
<name>Task 2 (MM2-01): Registry reuses one active instance + unloads on switch</name>
<files>backend/services/tts_backend.py</files>
<action>
Fix the leak at get_active_tts_backend() (line 1235). Today it builds a fresh instance every call with no teardown of the prior engine — switching engines (or repeated synth) leaks VRAM until GC. This is the root cause behind the #278 comment thread.
- Add a module-level cache: `_active_instance: TTSBackend | None = None` and `_active_instance_id: str | None = None`.
- In get_active_tts_backend(): resolve `bid = active_backend_id()`. If `_active_instance is not None` and `_active_instance_id != bid`, call `_active_instance.unload()` (best-effort, wrap in try/except so a bad unload can't block the switch — log on failure) before discarding it.
- Build the new instance, store it as `_active_instance` + `_active_instance_id = bid`, return it.
- IMPORTANT subtlety: OmniVoiceBackend takes `model=`. When `model=` is passed (the caller already has a loaded model), do NOT cache that instance as the shared `_active_instance` blindly — it's a per-call view over the shared singleton. Keep current behavior for the `model=` path (return a fresh OmniVoiceBackend(model=model)) but still trigger unload() of a *different* outgoing engine first. Pick the simplest correct rule: the cache tracks the configured backend id; passing model= for the SAME id reuses, switching id always unloads the previous. Document the rule in a comment.
- Add a module-level `reset_active_backend()` helper that unloads + clears the cache, for app shutdown and tests.
</action>
<verify>
<automated>grep -n "_active_instance\|def reset_active_backend\|def get_active_tts_backend" backend/services/tts_backend.py</automated>
</verify>
<done>
- Switching backend id calls the outgoing instance's unload() exactly once before the new instance is built.
- A bad/raising unload() is caught + logged, never blocks the switch.
- reset_active_backend() exists and is idempotent.
- The model= fast-path for OmniVoice still works (no double-load).
</done>
</task>
<task type="auto">
<name>Task 3 (MM2-03): Honest /model/loaded + /model/unload for ASR</name>
<files>backend/api/routers/system.py</files>
<action>
The ASR row (system.py:166-175) is reported as unloadable:False, vram_mb:0 even when loaded on GPU, and /model/unload doesn't expose the offload-to-CPU path. Make reporting truthful WITHOUT changing the ASR<->TTS lifecycle coupling (that coupling is intentional — offload_tts_for_asr/restore_tts_after_asr).
- ASR row: keep unloadable reflecting reality. If ASR truly cannot be released independently of TTS, keep unloadable:False but add a `note` field ("released with TTS") so the UI explains it rather than showing a dead button. Do not invent a separate ASR unload that breaks the WhisperX large-v3 offload path.
- vram_mb: if ASR currently runs on CPU (device "cpu" in the row), 0 is correct — leave it but make the device value derive from where the pipe actually is, not a hardcoded "cpu".
- This task is intentionally small; the bigger restructure is Task 4 (facade). Land MM2-03 as the honest-reporting fix, then Task 4 moves the enumeration into the facade.
</action>
<verify>
<automated>uv run pytest tests/test_engines.py -q 2>&amp;1 | tail -15</automated>
</verify>
<done>
- No row reports loaded-but-with-a-misleading-unloadable flag; ASR carries an explanatory note when unloadable:False.
- Device field reflects the actual device of the ASR pipe.
</done>
</task>
<task type="auto">
<name>Task 4 (MM2-01..03 tests): Wave 1 regression tests</name>
<files>tests/test_engines.py</files>
<action>
Add tests proving the leak fix and the unload contract:
- test_switching_backend_unloads_previous: monkeypatch two fake backends into _REGISTRY, set active to A (get_active_tts_backend), switch prefs to B, assert A.unload() was called exactly once before B is returned.
- test_unload_is_idempotent_and_preload_safe: OmniVoiceBackend().unload() twice + before generate() never raises.
- test_reset_active_backend_clears_cache: after reset_active_backend(), the next get_active_tts_backend() builds fresh.
Reuse the existing fixture style in tests/test_engines.py (it already monkeypatches the registry / availability). Keep tests CPU-only (no real model load).
</action>
<verify>
<automated>uv run pytest tests/test_engines.py -q 2>&amp;1 | tail -20</automated>
</verify>
<done>All three new tests pass; existing test_engines.py tests still green.</done>
</task>
<!-- ════════════ WAVE 2 / TIER 2 — SINGLE LIFECYCLE SURFACE ════════════ -->
<task type="auto">
<name>Task 5 (MM2-04): Extract services/model_lifecycle.py facade</name>
<files>backend/services/model_lifecycle.py</files>
<action>
Create backend/services/model_lifecycle.py as the single owner of cross-world model lifecycle. It composes the existing pieces — it does NOT reimplement loading.
Public surface:
- list_loaded() -> list[dict]: returns the unified rows currently assembled inline in system.py:129-207 (TTS, ASR, diarization, subprocess sidecars). Move that logic here verbatim first, then improve (MM2-03 note field, MM2-08 sidecar vram once Task 8 lands).
- unload(model_id: str) -> dict: the dispatch currently inline in system.py:210-242 ("tts" | "diarization" | "sidecar:<id>" | "sidecars"). Move here; keep async-lock semantics for the in-process model (mm._model_lock).
- unload_all() -> dict: unload every releasable model (in-process TTS + diar + all sidecars). New convenience used by app shutdown.
- free_vram(): thin re-export of model_manager.free_vram() so callers have one import.
Keep the "never let sidecar enumeration break the panel" try/except guard.
</action>
<verify>
<automated>uv run python -c "import services.model_lifecycle as ml; print([f for f in ('list_loaded','unload','unload_all','free_vram') if hasattr(ml,f)])"</automated>
</verify>
<done>model_lifecycle exposes list_loaded/unload/unload_all/free_vram; logic moved out of system.py (not duplicated).</done>
</task>
<task type="auto">
<name>Task 6 (MM2-04): Thin system.py routers + facade tests</name>
<files>backend/api/routers/system.py, tests/backend/services/test_model_lifecycle.py</files>
<action>
- Replace the bodies of GET /model/loaded (line 129) and POST /model/unload/{model_id} (line 210) with calls to model_lifecycle.list_loaded() / model_lifecycle.unload(model_id). Preserve the exact response shapes (frontend hooks.ts useModelStatus/useFlushMemory + the flush dropdown depend on {models, count} and {unloaded, success, ...}). The 400 on unknown model_id stays.
- New tests/backend/services/test_model_lifecycle.py: list_loaded with nothing loaded returns {models:[], count:0}; unload("tts") when not loaded returns success:False reason:"not loaded"; unload("sidecars") with no sidecars returns count:0; unknown id raises/400 path. Mock model_manager + subprocess_backend so no real models load.
</action>
<verify>
<automated>uv run pytest tests/backend/services/test_model_lifecycle.py -q 2>&amp;1 | tail -20</automated>
<automated>grep -n "model_lifecycle" backend/api/routers/system.py</automated>
</verify>
<done>system.py routers are thin delegations; response shapes unchanged; new facade tests pass.</done>
</task>
<task type="auto">
<name>Task 7 (MM2-05): Unify idle/timeout config through prefs.resolve</name>
<files>backend/services/model_manager.py, backend/services/subprocess_backend.py</files>
<action>
- model_manager.py: remove the duplicated `_IDLE_TIMEOUT_SECONDS = IDLE_TIMEOUT_SECONDS` (line 114). Resolve at use-site in idle_worker() via prefs: `prefs.resolve("idle_timeout_seconds", env="OMNIVOICE_IDLE_TIMEOUT_S", default=IDLE_TIMEOUT_SECONDS)`. Keep core.config.IDLE_TIMEOUT_SECONDS as the default source.
- subprocess_backend.py: replace the env-only `SIDECAR_IDLE_TIMEOUT_S` (line 107) read with prefs.resolve("sidecar_idle_timeout_seconds", env="OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S", default=300.0). Preserve "<=0 disables reaping" semantics and the existing reaper-start guard (line 222). Resolve lazily (function call), not at import, so a test/setting change takes effect — but keep a sensible cached default for the hot reaper loop.
- Both must keep env precedence (env wins over store) — that's exactly what prefs.resolve already does.
</action>
<verify>
<automated>grep -n "_IDLE_TIMEOUT_SECONDS\|prefs.resolve\|SIDECAR_IDLE_TIMEOUT" backend/services/model_manager.py backend/services/subprocess_backend.py</automated>
<automated>uv run pytest tests/backend/services/test_subprocess_reaper.py -q 2>&amp;1 | tail -20</automated>
</verify>
<done>
- No hand-duplicated IDLE_TIMEOUT constant; both timeouts resolve via prefs with env precedence.
- Reaper "<=0 disables" + busy-skip behavior unchanged; all 10+ reaper tests still pass.
</done>
</task>
<!-- ════════════ WAVE 3 / TIER 3 — ROBUSTNESS & OBSERVABILITY ════════════ -->
<task type="auto">
<name>Task 8 (MM2-08): Subprocess sidecars self-report VRAM in pong</name>
<files>backend/services/subprocess_backend.py</files>
<action>
Sidecar VRAM is reported as 0 (system.py:192-203 / list_live_sidecars) because the parent can't measure a child's GPU memory. Have the child measure itself.
- In the sidecar worker's ping handler (the code that answers {"op":"ping"} with {"op":"pong"} — find it in the sidecar entry-point module), include `vram_mb`: measure inside the child via torch.cuda.memory_allocated() (CUDA) or torch.mps.driver_allocated_memory() (MPS, guarded), else 0. Same degrade-gracefully pattern as system.py:147-156.
- Parent: in the health-check ping/pong path (subprocess_backend.py:435-438), capture reply["vram_mb"] and stash it on the sidecar record so list_live_sidecars() (line 181) can surface it. Refresh opportunistically on each successful ping; default to last-known or 0 if never measured.
- Keep the contract that enumeration never breaks the panel.
This is CUDA/MPS-aware and degrades to 0 on CPU — honoring cross-platform parity (default behavior identical; the number is just more accurate where the API exists).
</action>
<verify>
<automated>grep -n "vram_mb" backend/services/subprocess_backend.py</automated>
<automated>uv run pytest tests/backend/services/test_subprocess_reaper.py -q 2>&amp;1 | tail -15</automated>
</verify>
<done>list_live_sidecars() exposes a vram_mb sourced from the child's own measurement; 0 only when truly CPU/unmeasured; reaper tests still green.</done>
</task>
<task type="auto">
<name>Task 9 (MM2-06, MM2-07): Bounded cooldowns + per-role weight validation</name>
<files>backend/api/routers/setup/download.py</files>
<action>
- MM2-06: _install_cooldowns (line 27) grows unbounded. On a successful install, delete the repo's cooldown entry. Add a TTL sweep: when reading/writing the dict, evict entries older than a fixed window (reuse the existing cooldown window constant; pick the larger of cooldown-window and e.g. 1h). Keep it simple — a dict + timestamps, swept on access. No new dep.
- MM2-07: _validate_snapshot_has_weights (line 55) + _MIN_WEIGHT_BYTES 5 MB (line 45) is one magic number for all roles. Make the threshold per-role/per-extension: safetensors/bin/ckpt expect the existing floor; .onnx models (kittentts, supertonic, sherpa) can be legitimately smaller — set a lower, role-aware floor so a valid small ONNX model isn't flagged as truncated. Keep the #352 truncation-catch intent (catch a 0-byte / KB-sized partial), just stop false-positiving small-but-complete models.
</action>
<verify>
<automated>grep -n "_install_cooldowns\|_MIN_WEIGHT_BYTES\|def _validate_snapshot_has_weights" backend/api/routers/setup/download.py</automated>
<automated>uv run pytest tests/ -k "download or install or model" -q 2>&amp;1 | tail -20</automated>
</verify>
<done>Cooldown dict is bounded (evict-on-success + TTL sweep); weight validation floor varies by role/extension; #352 truncation still caught.</done>
</task>
<task type="auto">
<name>Task 10 (MM2-09): Log why scan_cache_dir() fell back to disk walk</name>
<files>backend/api/routers/setup/models.py</files>
<action>
The scan_cache_dir() -> _scan_cache_on_disk() fallback (~line 268-280, helper at line 177) silently swallows the exception — this is the #117/#118 Windows WinError-448 path. Wrap the fallback so it logs at WARNING with the exception type and a one-line reason ("scan_cache_dir failed (%s); falling back to on-disk walk of %s") before walking. Do not change the fallback behavior itself — just stop it being invisible in logs. Keep it from ever raising out (the panel must still render).
</action>
<verify>
<automated>grep -n "falling back\|logger.warning\|_scan_cache_on_disk\|scan_cache_dir" backend/api/routers/setup/models.py | head</automated>
</verify>
<done>The disk-walk fallback logs a WARNING naming the exception type; behavior otherwise unchanged; never raises out.</done>
</task>
</tasks>
<verification>
Full-suite gate after each wave (run the relevant subset per wave, full set before the last PR):
1. `uv run pytest tests/test_engines.py tests/backend/services/test_model_lifecycle.py tests/backend/services/test_subprocess_reaper.py tests/test_model_load_timeout.py tests/test_model_manager_preload.py -q` — all green.
2. `uv run pytest tests/ -k "download or install or model or engine" -q` — green (Tier 3 touch points).
3. Response-shape guard: GET /model/loaded still returns {models, count}; POST /model/unload returns {unloaded, success, ...}; 400 on unknown id. (Covered by test_model_lifecycle.py.)
4. No new runtime dependency: `git diff pyproject.toml uv.lock` is empty.
5. Localization/CJK + redaction gates unaffected: `uv run pytest tests/test_no_hardcoded_cjk.py -q`.
</verification>
<success_criteria>
- Tier 1: switching the active backend releases the previous engine's VRAM (unload() called once on switch); contract overridden for OmniVoice + all subprocess engines; ASR reporting is honest. (MM2-01..03)
- Tier 2: services.model_lifecycle is the single lifecycle surface; system.py routers are thin delegations with unchanged response shapes; idle/timeout config flows through prefs.resolve with env precedence and no duplicated constants. (MM2-04..05)
- Tier 3: cooldown dict bounded; weight validation is per-role; sidecars self-report real VRAM; cache-fallback logs its reason. (MM2-06..09)
- All listed pytest selections pass; no on-disk model-state change; no new dep; cross-platform default behavior identical (VRAM numbers degrade gracefully on MPS/CPU).
</success_criteria>
<risks>
- **unload() correctness for the shared OmniVoice singleton (MM2-01/02):** OmniVoiceBackend shares model_manager's `model` global. unload() must release the shared singleton, but the idle_worker() + offload_tts_for_asr() paths also touch it. Risk: a switch during an in-flight ASR offload double-frees or races. Mitigation: take mm._model_lock around the shared release in unload(); guard on `mm.model is not None`; keep unload best-effort (try/except) so it can never wedge a switch. Add the idempotency test (Task 4).
- **Response-shape drift (MM2-04):** Moving /model/loaded + /model/unload bodies into the facade risks changing the JSON the frontend depends on (hooks.ts, flush dropdown). Mitigation: move verbatim first, assert shapes in test_model_lifecycle.py, only then layer MM2-03/08 improvements.
- **Sidecar protocol change (MM2-08):** Adding vram_mb to pong touches the parent/child wire format. Older sidecars (a long-running session mid-upgrade) won't send it. Mitigation: treat vram_mb as optional in the parent (`reply.get("vram_mb", <last-known or 0>)`); never require it; never break the existing pong==success check.
- **prefs.resolve at import time (MM2-05):** Resolving timeouts at import freezes them; the reaper loop reads SIDECAR_IDLE_TIMEOUT_S. Mitigation: resolve lazily inside the reaper tick / idle_worker tick (cheap) so a settings change takes effect, while keeping the import-time default for the start-guard.
- **Per-role weight floor (MM2-07):** Lowering the ONNX floor could let a genuinely-truncated ONNX through (#352 regression). Mitigation: keep a non-zero floor for every role (e.g. ONNX floor still >> a partial KB), key on extension, and keep the "largest file" heuristic — only the threshold becomes role-aware.
- **Scope creep into perf:** GPU-pool sizing and torch.compile are explicitly out of scope. If an executor is tempted, stop — those regress #278/#315.
</risks>
<output>
Write `.planning/quick/260613-mm2-clean-model-management-v2/260613-mm2-SUMMARY.md` when done (per wave or once at the end), documenting: which engines got real unload() overrides vs intentional no-ops (for the future CI gate), the exact response shapes preserved on the two endpoints, the per-role weight-validation thresholds chosen, and the pytest output for the verification selection. Note any decision an executor made where the plan said "use judgment."
Docs-sync check (CLAUDE.md hard rule): this is internal lifecycle cleanup with no user-facing install/Docker/versioning change, so no README/docs edit is expected. If MM2-05 surfaces the new idle-timeout settings keys in the Settings UI, add them to the relevant settings doc in the same PR.
</output>
@@ -1,32 +0,0 @@
# SUMMARY — model-management v2 cleanup (mm2)
**Date:** 2026-06-13 · **Scope:** all 3 tiers (MM2-01..09). Backend-only; no frontend, no on-disk model-state change, no new deps.
## Tier 1 — correctness
- **MM2-01 (VRAM leak on engine switch):** `get_active_tts_backend()` now caches one instance per configured backend id and calls the outgoing engine's `unload()` before switching. Added `reset_active_backend()` for shutdown/tests. The `model=` OmniVoice fast-path still returns a fresh view over the shared singleton (no double-load) but a switch *away from* another engine still releases it. `tts_backend.py`.
- **MM2-02 (per-engine unload()):** `OmniVoiceBackend.unload()` drops the local ref + the shared `model_manager.model` singleton + `free_vram()` (idempotent, preload-safe, best-effort — no async lock from the sync path). `SubprocessBackend.unload()` routes to `unload_sidecar(self.id)` (busy sidecars skipped) and is inherited by every subprocess engine.
- **MM2-03 (honest ASR row):** `/model/loaded` ASR row now reports the pipe's actual device and carries a `note: "released with the TTS model"` so the disabled unload button is explained rather than silent.
## Tier 2 — single lifecycle surface
- **MM2-04 (`services/model_lifecycle.py`):** new facade owns `list_loaded()` / `unload(id)` / `unload_all()` / `free_vram()` across in-process TTS+ASR, diarization, and sidecars. `system.py` `/model/loaded` + `/model/unload` are now thin delegations; **response shapes preserved exactly** (`{models,count}`, `{unloaded,success,...}`, 400 on unknown id) — frontend untouched.
- **MM2-05 (unified idle config):** removed the duplicated `_IDLE_TIMEOUT_SECONDS`; the in-process idle timeout and the sidecar idle timeout both resolve per-tick via `prefs.resolve(... env=...)` (env wins, settings can tune without restart). New keys: `idle_timeout_seconds` (`OMNIVOICE_IDLE_TIMEOUT_S`), `sidecar_idle_timeout_seconds` (`OMNIVOICE_SIDECAR_IDLE_TIMEOUT_S`). `<=0` still disables sidecar reaping.
## Tier 3 — robustness & observability
- **MM2-06 (bounded cooldowns):** `_install_cooldowns` is swept (TTL 1h) on each install check and cleared on success — can no longer grow unbounded.
- **MM2-07 (per-role weight floor):** `_validate_snapshot_has_weights` uses per-extension floors (tensor formats keep 5 MB; `.onnx` floor 64 KB) **OR** the original ≥5 MB catch — strictly more lenient, so a small-but-complete ONNX model is no longer false-flagged as truncated while a 0/KB partial is still rejected (#352 intact).
- **MM2-08 (sidecar VRAM self-report):** the parent can't see a child's VRAM, so the GPU sidecar (`engines/indextts`) now reports `vram_mb` in its `pong` (CUDA/MPS-aware, 0 on CPU); the parent stashes the last-known figure and `list_live_sidecars()` surfaces it. CPU/absent sidecars honestly report 0.
- **MM2-09 (cache-fallback logging):** the `is_cached` `scan_cache_dir → on-disk` fallback now logs at WARNING with the exception type (was DEBUG/invisible) — the #117/#118 WinError-448 path is triagable from logs.
## Out of scope (as planned, not done)
GPU-pool per-engine sizing (`_GPU_VRAM_PER_JOB_GB`) and torch.compile tuning — perf, not cleanup; risk regressing #278/#315.
## Verification
- New `tests/test_mm2_lifecycle.py` — 15 tests (reuse/switch-unload/reset/idempotent-unload, facade list/unload/unknown/sidecars shapes + honest ASR, env-wins idle config, cooldown sweep, per-role weight floor ×3).
- Affected existing: `test_engines.py`, `test_subprocess_reaper.py`, `test_model_load_timeout.py`, `test_model_manager_preload.py` — green (no regressions).
- **Full suite: 1379 passed, 0 failed.** Live: facade endpoints return preserved shapes; engine switch calls the previous engine's `unload()` exactly once.
## Test placement note
MM2 tests live at top-level `tests/` (not `tests/backend/`) on purpose: adding files under `tests/backend/` reorders collection and can expose a pre-existing `sys.modules`-isolation leak in other backend fixtures (the issue debugged in the FDL PR). Top-level placement keeps `tests/backend/` order identical.
## Docs-sync
The new idle-timeout settings keys are internal env/prefs knobs with no UI surface, so no README/docs change is required by the docs-sync rule. If a future Settings panel exposes them, document there.
@@ -1,205 +0,0 @@
# Adjacent open-source projects — research notes (2026-07-10)
Owner-requested research on five neighboring projects, read against OmniVoice
Studio's current feature-maturity map. Each section ends with what we should
take from it. Priorities are consolidated at the bottom.
| Project | Stars | License | Status | Why it matters to us |
|---|---|---|---|---|
| [Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning) | ~60k | MIT | Retired (models frozen 2019, maintainer quit 2020) | Positioning/SEO opportunity, cautionary tales |
| [VoxCPM](https://github.com/OpenBMB/VoxCPM) | ~33k | Apache-2.0 | Very active (VoxCPM2, Apr 2026) | **Upstream of our `voxcpm2` engine** — sync items below |
| [ebook2audiobook](https://github.com/DrewThomasson/ebook2audiobook) | ~19.5k | Apache-2.0 (default XTTS weights are CPML non-commercial) | Very active, weekly releases | The playbook for our weakest shipped surface (audiobook) |
| [VideoLingo](https://github.com/Huanshere/VideoLingo) | ~17.7k | Apache-2.0 | Active, bursty | Dub-pipeline techniques (translation loop, timeline fit) |
| [voicebox](https://github.com/jamiepine/voicebox) | ~40.2k | MIT | Very active, post-viral triage debt | **Direct competitor** — same stack, same pitch, 10x the audience |
## 1. Real-Time-Voice-Cloning — the retired ancestor
The 2019 SV2TTS implementation ("clone a voice in 5 seconds") that created the
DIY voice-cloning category. Explicitly retired: the maintainer said in 2020 he
won't develop it again; the README now calls itself old and redirects users to
Chatterbox. Models are frozen 2019 checkpoints — 16 kHz, English-only, weak
similarity, Tacotron+WaveRNN. Community PRs keep the install alive (uv
one-command install landed Sept 2025), but ~163 open issues are mostly "how do
I make it sound good" — the answer is: you can't.
**Integrating it as an engine: no.** Strictly worse than everything we ship,
plus PyQt/legacy baggage.
**Take:**
- 60k stars of traffic reads a README that says "go elsewhere," and the
redirect target is a model repo, not a product. An honest
"Real-Time-Voice-Cloning alternative" comparison page is cheap, truthful,
and lands exactly our pitch (local, free, modern quality, 646 languages,
actual installer).
- Its headline copy discipline ("Clone a voice in 5 seconds, generate
arbitrary speech in real-time") is better than ours; our 3-second-reference
claim deserves the same outcome-first, time-boxed phrasing.
- Its failure modes validate our Core Value: out-of-band model links rotted
for years; a research toolbox without packaging drowned in install issues.
## 2. VoxCPM — upstream of our `voxcpm2` engine
Tokenizer-free TTS on a MiniCPM-4 backbone. Current model is **VoxCPM2**
(Apr 2026): 2B params, 30 languages + 9 Chinese dialects, 48 kHz, ~8 GB VRAM,
RTF ~0.30 (0.13 with Nano-vLLM). Latest tag v2.0.3 (May 2026); main has
unreleased seed support and timestamp alignment. Apache-2.0, healthy cadence,
~868k monthly HF downloads.
**Sync items for our integration** (we install `voxcpm` unpinned):
1. **Floor the install at `voxcpm>=2.0.3`** — it carries the MPS
audio-quality fix (low-precision dtypes promoted to float32 on Apple
Silicon). Directly relevant to our default-platform-parity rule.
2. **v2.0.1 removed reference-audio auto-trim** — if we hand raw user clips
to cloning, we now own trim/normalize. Verify our clone path; cloning
quality may have silently regressed when upstream released 2.0.1.
3. **Trailing-audio guard**: end-of-audio gibberish/hallucination is a known
open upstream bug (#352). A trailing-silence/garbage trim on our side is
cheap insurance.
4. **Later, when tagged**: seed support (reproducible generation — currently
buggy upstream, #351) and timestamp alignment (useful for dub sync);
`generate_streaming()` is a candidate for `tts_stream.py`.
5. **Risk**: unpinned dependency + active upstream = next release lands
silently in fresh installs. Consider pinning a tested range.
## 3. ebook2audiobook — the audiobook playbook
Any-format ebook (epub/pdf/docx/even scanned images via OCR) → Calibre
normalize to EPUB → TOC/spine chapters ("blocks") → per-language sentence
split → per-sentence TTS → chapterized m4b with metadata/cover. Gradio UI +
headless CLI + Docker for every accelerator. Engine roster is 2023-era Coqui
(XTTSv2 default, Bark, Piper, MMS…), with voice-conversion post-processing to
fake cloning on non-cloning engines. 19.5k stars, near-weekly releases, only
4 open issues.
This is the mature version of exactly the surface where we're weakest: our
audiobook/stories feature is a thin UI over per-chapter render caching, with
no server-side ebook parsing and no per-segment regeneration.
**Take (prioritized):**
1. **Per-sentence render cache + content-hashed blocks + missing-file
resume.** Every sentence is its own file; restart re-renders only what's
missing; editing a block invalidates only that block. This closes our
biggest audiobook gap (per-chapter cache, no crash resume) and is the same
span-level model spec 03 already calls for — dub's `incremental.py`
pattern, extended to longform.
2. **Normalize-to-EPUB ingestion** (Calibre `ebook-convert`) instead of
building N format parsers; blocks carry keep/drop flags for front matter.
3. **Engine-agnostic text-normalization pre-pass**: per-language abbreviation
maps, num2words, roman numerals, and a non-text character filter that
kills TTS hallucination triggers. Benefits every engine we ship, not just
audiobooks.
4. **Chapterized m4b output** (ffmpeg FFMETADATA chapters, cover art, VTT
sidecar) — small work, high perceived value.
5. **Inline voice/pause tags** for multi-voice narration — our cloning
quality makes this worth more to us than it is to them.
**Where we already win:** native desktop UX, modern engine quality
(CosyVoice3/IndexTTS2/VoxCPM2 vs 2023 Coqui), real zero-shot cloning without
VC hacks, no Calibre-wall install, and a commercially-clean default engine
(their default XTTS weights are CPML non-commercial).
## 4. VideoLingo — dub-pipeline techniques
"Netflix-quality subtitles + dubbing" as a 14-stage Streamlit pipeline:
yt-dlp → WhisperX word-level ASR → spaCy + LLM two-candidate semantic split →
summarize-first terminology glossary → 3-step TranslateReflectAdapt →
length-constrained subtitles → duration-aware dub-chunk planning →
per-chunk reference audio → TTS → merge. Its recommended path is
cloud-heavy (API LLM/TTS, optionally API ASR); fully-local is possible but
fragile. Single-speaker only — it explicitly gave up on diarized multi-voice
dubbing. Apache-2.0, ~17.7k stars, bursty maintenance, install pain on
Windows/CUDA.
**Take (prioritized):**
1. **TranslateReflectAdapt** — add a reflection/critique pass to our
per-segment translation prompt. Prompt-level change, meaningful quality
win on idiomatic output.
2. **Summarize-first glossary** — extract theme + terminology once per video,
inject into every segment's translation. Fixes term drift on long videos.
3. **Duration-aware chunk planning** — estimate TTS duration *before*
generating; classify each line ok / needs-speedup / impossible; borrow
inter-subtitle gap time and merge adjacent segments before resorting to
atempo; for impossible lines, LLM-trim filler from the dub text instead of
chipmunking. Our smart-fit handles the tail of this; their pre-planning
avoids generating doomed audio at all.
4. **Two-candidate split prompt** — generate two `[br]` segmentations, have
the LLM pick, instead of accepting the first.
**Where we already win:** fully local by design, per-segment regeneration +
directorial AI (they have coarse folder-state resume, no per-segment redo),
cross-platform installers, cloning stable across languages. Their
single-speaker ceiling is our opening if diarized multi-voice dubbing ever
ships.
## 5. voicebox — the direct competitor
Jamie Pine's (Spacedrive founder) "open-source AI voice studio. Clone,
dictate, create." — architecturally a near-twin: **Tauri + React/TS +
FastAPI/Python + SQLite**, MIT, local-first, explicitly pitched as
ElevenLabs-out + WisprFlow-in replacement. Launched Jan 29, 2026; the launch
post did ~17M views on X, and it sits at **~40.2k stars** with ~10 community
contributors and heavy AI co-authorship. Latest tagged release v0.5.0
(Apr 2026); main is active but untagged for ~10 weeks, with **434 open
issues / 105 open PRs** — a polished happy path with thin edges.
Engines: Qwen3-TTS 0.6B/1.7B (flagship cloner), Qwen CustomVoice, LuxTTS,
Chatterbox Multilingual (23 langs) + Turbo, HumeAI TADA, Kokoro. Features
where they lead: global-hotkey dictation overlay with LLM transcript cleanup
(macOS-verified), Pedalboard post-FX chain, generation versioning/starring,
multi-track Stories editor, **MCP per-client voice bindings** ("Claude Code
speaks in your cloned voice") used as a viral wedge, DirectML/Intel-Arc
coverage, and an agent-facing CONTRIBUTING pattern that farms drive-by
contributions.
Two strategic facts:
- **They are adding accounts.** "Log in with browser" auth for a
`voicebox.sh` cloud tier merged July 5 (their PR #812). Open-core with a
paid cloud is visibly forming — which cuts against the pitch that won them
their audience.
- **Press already flagged their missing consent/misuse policy** — we ship
watermarking by default and consent attestation in `.ovsvoice`.
**Where we're ahead:** 646 languages vs 23, video dubbing (they have none),
voice design from text descriptions (roadmap item for them, shipped for us),
engine breadth (CosyVoice3/VoxCPM2/IndexTTS2/GPT-SoVITS/sherpa-onnx), and
backward-compat/release discipline.
**Take:**
1. **Positioning: own "no accounts, ever."** Their cloud login is our
opening — state the local-first guarantee in the README as a permanent
commitment, next to the 646-language and dubbing advantages they can't
match today.
2. **Tell the MCP agent-voice story loudly.** We already ship an MCP server
and Agent Skills; per-client voice bindings + a speak-in-your-voice demo
was their single best growth hook and costs us mostly marketing effort.
3. **Generation versioning/starring and post-FX presets** — cheap,
high-perceived-value Studio features worth absorbing.
4. **Watch their triage debt** (434 open issues): our absorb-or-decline
queue discipline is a real contributor-trust differentiator — keep it.
## Consolidated priorities
Ordered by (user impact on already-shipped surfaces) × (effort):
1. **voxcpm2 upstream sync** (§2 items 13): version floor, ref-clip trim
audit, trailing-audio guard. Small, protects an engine users already run.
2. **Dub translation quality loop** (§4 items 12): reflect pass + glossary.
Prompt-level, no new deps, lifts the flagship dubbing feature.
3. **Audiobook maturity via per-sentence cache + resume** (§3 item 1): the
established pattern for the feature the maturity survey ranked weakest —
and it's the same architecture spec 03 already prescribes.
4. **Text-normalization pre-pass** (§3 item 3): engine-agnostic hallucination
reduction; pairs with the pronunciation dictionary we already shipped.
5. **Duration-aware dub planning** (§4 item 3) and **chapterized m4b export**
(§3 item 4): next tier, both self-contained.
6. **Competitive positioning vs voicebox** (§5 items 12): own "no accounts,
ever" while they onboard a cloud tier, and tell the MCP agent-voice story
we already technically ship.
7. **RTVC comparison/migration page** (§1): marketing, not engineering;
cheap and honest.
*Method note: compiled from five parallel research passes over the repos'
READMEs, releases, issues, and (for ebook2audiobook) source; figures as of
2026-07-10.*
-1109
View File
File diff suppressed because it is too large Load Diff
+175 -26
View File
@@ -3,7 +3,7 @@
**OmniVoice Studio**
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. It's an active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord. The current version lives in `frontend/package.json` (the single source of truth — see Versioning); the latest stable tag is on the [Releases page](https://github.com/debpalash/OmniVoice-Studio/releases/latest). With `AUTO_VERSION_BUMP` off (the current owner setting), `main` holds at the released version between releases.
OmniVoice Studio is an open-source, fully-local ElevenLabs alternative — a desktop app for voice cloning, voice design, video dubbing, and real-time dictation across 646 languages. It runs entirely on the user's machine (CUDA/MPS/ROCm/CPU auto-detect), with no API keys, no accounts, and no cloud dependencies. Today it's a v0.2.7 active beta with a growing user base who hit it with real workloads (50-video batches, multi-engine setups, edge-OS platforms) and report friction in GitHub Issues and Discord.
**Core Value:** **A first-run that actually works.** A user who downloads the installer (or clones the repo) should reach a working voice-cloning or dubbing output without hitting a wall — and when something does go wrong, the error or docs should tell them exactly what to do.
@@ -16,42 +16,184 @@ Everything else (new engines, fancy features) is downstream of "the thing instal
- **Default features must work on every platform (strict rule, 2026-05-20):** A feature that ships in default mode — out-of-the-box, no user customization, no opt-in toggle — must behave identically on macOS, Windows, and Linux. Platform-specific *implementation code* is allowed for OS APIs / shells / packaging, but the user-visible *default behavior* cannot diverge. Platform-only features (e.g., a macOS-only global shortcut, a Windows-only path picker) must go behind explicit user opt-in: Settings toggle, env var, or CLI flag. When a default doesn't work on a platform, that's a P0 bug — either fix it on the missing platform or move it behind opt-in. No third option.
- **Backward-compatible project data**: Existing `omnivoice_data/` (user voices, projects, settings) must keep working without manual migration. Any DB schema change goes through alembic with a tested upgrade path.
- **Local-first guarantee preserved**: Auto bug reporting (new addition) must be **opt-in**, must submit only to GitHub Issues (no third-party telemetry endpoint), and the app must remain fully functional with reporting disabled. No required cloud calls, accounts, or API keys.
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** the v0.3.x line has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main; the owner tags a patch (`v0.3.Z`) from main whenever the current state is worth cutting. No `-rc` tags. No phased release. No `v0.4` deferrals while the v0.3.x line is open — every open issue and every open community PR gets absorbed into the v0.3.x line or explicitly declined. Users follow `main` for previews; users wanting stable stay on the latest tagged release. ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
- **Beta release cadence (no RC, no ceremony — strict rule, 2026-05-20):** v0.3.0 has **no release candidates, no 48h soak, no formal release ceremony**. Every fix goes continuous-to-main. Tag `v0.3.0` once when the user calls "actually useful" — a qualitative bar, not a checklist. No `v0.3.0-rc1`. No phased release. No `v0.4` deferrals while v0.3.0 is open — every open issue and every open community PR gets absorbed into the v0.3.0 line or explicitly declined. Users follow `main` for previews; users wanting stable stay on `v0.2.7`. ROADMAP.md's Phase 6 "Release/Verify/Retro" entries are obsolete unless the user revives them.
<!-- GSD:project-end -->
<!-- GSD:stack-start source:research/STACK.md -->
## Technology Stack
The May-2026 stack research that used to live here served five capabilities that have all since shipped (HF-token Settings panel, prefilled-URL bug reporting, uv mirror fallback for restricted networks, the Supertonic-3 engine, in-repo Markdown docs). Follow the patterns in the code itself; the durable *don'ts* that research established:
- **No third-party telemetry endpoints, ever** (`sentry-tauri` was evaluated and rejected) — bug reporting stays opt-in via prefilled GitHub-issue URLs.
- **No PAT/token-based GitHub posting from the app** — the user submits from their own browser.
- **Don't recommend `setx` for env vars on Windows** (silent truncation, no current-shell propagation) — use the in-app Settings panel or PowerShell `[Environment]::SetEnvironmentVariable`.
- **Don't adopt Material for MkDocs** for any future docs site (maintenance mode since Nov 2025) — Astro Starlight is the precedent if docs ever outgrow the repo.
- **`hf_transfer` is deprecated** — default `huggingface_hub` (hf-xet) handles downloads.
For anything new: prefer what's already pinned in `pyproject.toml` / `frontend/package.json`, and check `uv tree` for conflicts before adding a dependency.
## Recommended Stack — Per Capability
### Capability 1 — HuggingFace Token Persistence (issue #35)
| Technology | Version | Purpose | Why Recommended |
|------------|---------|---------|-----------------|
| `huggingface_hub` (already pinned transitively by `transformers>=5.3.0`) | `≥1.12.x` (latest 2026) | Auth + cache + token storage | Canonical, used by every HF library already in the stack. `HfFolder` is **superseded** in v1.x by the higher-level `login()` / `auth_list()` / `auth_switch()` API. |
| `keyring` (Python) | `≥25.x` | Optional OS-keychain backing | Only adopt if a future hardening pass wants Keychain/Credential-Manager/SecretService. **Not recommended for this milestone** — adds a native dep (`dbus`, `pywin32`) per platform with no real security win over `0600` file storage in `HF_HOME`. |
| Shell | One-liner to persist `HF_TOKEN` |
|-------|---------------------------------|
| macOS zsh (default since 10.15) | `echo 'export HF_TOKEN=hf_xxx' >> ~/.zshrc && source ~/.zshrc` |
| Linux bash | `echo 'export HF_TOKEN=hf_xxx' >> ~/.bashrc && source ~/.bashrc` |
| Windows PowerShell (user scope) | `[Environment]::SetEnvironmentVariable("HF_TOKEN","hf_xxx","User")` (new shells only) |
| Windows cmd | `setx HF_TOKEN "hf_xxx"` (user scope, new shells only) |
- [HF environment variables docs](https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables) — HIGH confidence (official, current)
- [HF authentication API docs](https://huggingface.co/docs/huggingface_hub/en/package_reference/authentication) — HIGH confidence
- [Microsoft `setx` docs](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/setx) — HIGH confidence
### Capability 2 — In-App Structured Bug Reporting (opt-in, GitHub Issues)
| Technology | Version | Purpose | Why Recommended |
|------------|---------|---------|-----------------|
| GitHub REST API `POST /repos/{owner}/{repo}/issues` | `2026-03-10` API version | Server-side issue creation | Official, stable. Requires auth. |
| **Prefilled-URL pattern** (`github.com/{owner}/{repo}/issues/new?title=…&body=…&labels=…`) | n/a | Zero-auth fallback | **This is the recommended primary path for v0.3.x.** No token needed, no GitHub App registration needed, user's browser opens with a prefilled form, they review and click Submit. They own the issue, the OSS project gets the report, and OmniVoice never holds a credential. |
| `gh-app-jwt` + GitHub App (Rust crate `octocrab` or Python `pygithub`) | only if we later want fully-automated submission | Programmatic posting under an app identity | **Defer to a later milestone.** Requires registering a public GitHub App, hosting a token-exchange endpoint, and managing rate-limit quotas — disproportionate for stabilization scope. |
| `platform`, `psutil`, `torch.cuda` (already in deps) | already pinned | Capture OS, CPU/GPU/VRAM info | No new deps. |
| `httpx` (already in `dev-dependencies`, promote to runtime if needed) | `≥0.28.1` | HTTP for the API call path (if/when we add auth) | Modern async-first, already used in test suite. |
- ✓ No token storage in OmniVoice → no security surface
- ✓ Opt-in by definition (user has to click Submit on github.com)
- ✓ User owns the issue → can be replied to, edited, closed by them
- ✓ Zero infra cost — no proxy, no app, no rate-limit management
- ✓ Works identically on macOS / Windows / Linux via Tauri's `shell.open`
- ✓ Survives our project being forked (just change the URL)
- OS name + version (`platform.platform()`)
- Python version (`sys.version`)
- OmniVoice version (`pyproject.toml`)
- Backend git SHA (if installed from source) or installer build ID
- CPU model, RAM (`psutil.cpu_count()`, `psutil.virtual_memory()`)
- GPU vendor/model/VRAM (`torch.cuda.get_device_name()`, `torch.cuda.mem_get_info()`, MPS detect)
- Active TTS engine + list of installed engines
- Frontend: bun version, OS shell
- Last error message + stack trace if launched from an error toast
- Audio file contents (privacy — reference samples may contain user's voice)
- File paths containing `/Users/<name>/` (strip home dir → `~/`)
- HF token, OpenAI keys, any env var matching `*TOKEN*|*KEY*|*SECRET*`
- [GitHub URL query parameters for issues](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/creating-an-issue#creating-an-issue-from-a-url-query) — HIGH confidence
- [sindresorhus/new-github-issue-url](https://github.com/sindresorhus/new-github-issue-url) — HIGH (widely used reference impl)
- [GitHub REST API: Create an issue](https://docs.github.com/en/rest/issues/issues#create-an-issue) — HIGH confidence (for the future auto-submit path)
- [sentry-tauri](https://github.com/timfish/sentry-tauri) — reviewed, **rejected for milestone** due to local-first constraint
### Capability 3 — `uv venv` Mirror Fallback for Restricted Networks (issues #57, #60)
| Technology | Version | Purpose | Why Recommended |
|------------|---------|---------|-----------------|
| `uv` (already used) | `≥0.5.x` | Python+venv bootstrap | Existing dep. |
| `UV_PYTHON_INSTALL_MIRROR` env var | uv `0.4.x`+ | Override python-build-standalone download URL | **Official, current.** Replaces `https://github.com/astral-sh/python-build-standalone/releases/download/...` in download URL construction. No built-in fallback if mirror fails. |
| `UV_PYTHON_PREFERENCE=only-system` (or CLI flag `--python-preference only-system`) | uv `0.4.x`+ | Skip the python-build-standalone download entirely; use the user's system Python | **The reliable escape hatch** when no mirror works. Requires a compatible Python `>=3.11` to already be on PATH. |
| `UV_HTTP_TIMEOUT`, `UV_HTTP_CONNECT_TIMEOUT`, `UV_HTTP_RETRIES` | uv `0.4.x`+ | Tune retry behavior for flaky links | Defaults are 30s / 10s / 3 — bump to 120s / 30s / 5 for restricted networks. |
# Pseudocode for the bootstrap
# Final fallback: don't download Python at all
- `UV_DEFAULT_INDEX=https://pypi.tuna.tsinghua.edu.cn/simple` (Tsinghua — fastest in China)
- `UV_DEFAULT_INDEX=https://mirrors.aliyun.com/pypi/simple` (Aliyun fallback)
- Russia: no major government-blessed PyPI mirror; users typically tunnel via VPN. Document this honestly rather than ship a broken default.
- [uv environment variables reference](https://docs.astral.sh/uv/reference/environment/) — HIGH (official)
- [uv issue #5224 — python-build-standalone mirror support](https://github.com/astral-sh/uv/issues/5224) — HIGH (the feature was added)
- [uv issue #14187 — venv on Chinese network](https://github.com/astral-sh/uv/issues/14187) — HIGH (confirms real user pain, no built-in fallback)
- [uv python-versions concepts](https://github.com/astral-sh/uv/blob/main/docs/concepts/python-versions.md) — HIGH (documents `python-preference` semantics)
- [dautovri/mirrors-china](https://github.com/dautovri/mirrors-china) — MEDIUM (community-maintained mirror list; verify each URL still works before shipping)
### Capability 4 — Supertonic-3 TTS Engine
| Technology | Version | Purpose | Why Recommended |
|------------|---------|---------|-----------------|
| `supertonic` (PyPI) | `1.3.1` (latest, May 18 2026 — Phase 3 Wave 1 to verify constructor signature before bump) | Official Supertonic-3 inference SDK | Authoritative wrapper from Supertone Inc. Wraps the ONNX session orchestration so we don't have to. |
| `onnxruntime` | `≥1.17.x` (any recent) | ONNX inference runtime | Already a transitive dep of WhisperX (via CTranslate2 path is separate, but `onnxruntime` itself ships for kittentts and audioseal). Verify with `uv tree` after adding — should resolve cleanly. |
| `huggingface_hub` (already pinned) | `≥1.12.x` | Model weight download (~400 MB on first use) | Reuses existing HF token + cache infrastructure. The user's existing `HF_TOKEN` (Capability 1) works for the Supertonic model download too. |
| `numpy`, `soundfile` (already pinned) | already pinned | Audio I/O + array math | No new deps. |
- `text_encoder.onnx`
- `latent_denoiser.onnx`
- `voice_decoder.onnx`
- 44.1 kHz sample rate, 24-dim latent, 128-dim style
- ~99M parameters total
- Tokenizer: `AutoTokenizer.from_pretrained(model_path)` — loads from `tokenizer.json` shipped with model
- [Supertone/supertonic-3 model card](https://huggingface.co/Supertone/supertonic-3) — HIGH (official)
- [supertone-inc/supertonic GitHub](https://github.com/supertone-inc/supertonic) — HIGH (official)
- [supertonic PyPI page](https://pypi.org/project/supertonic/) — HIGH (`1.3.1` confirmed 2026-05-18; same publisher, MIT, same 4 deps)
- [onnx-community/Supertonic-TTS-ONNX](https://huggingface.co/onnx-community/Supertonic-TTS-ONNX) — HIGH (ONNX file structure details)
### Capability 5 — Cross-Platform Documentation Tooling
| Technology | Version | Purpose | Why Recommended |
|------------|---------|---------|-----------------|
| Plain Markdown in `docs/` + GitHub-rendered (current state) | n/a | Install tutorial, troubleshooting | Zero new infra. Renders inline on GitHub for issue-replies. No build step to break. |
| Existing `scripts/smoke-test.sh` + Playwright `tests/` (already in `package.json`) | already pinned | Verify install paths actually work | **This is the real solution to "docs drift."** If smoke-test exercises the install path described in docs, docs that drift will break CI. |
| **Future** (defer): Astro Starlight | `≥0.30` | Standalone docs site at `docs.omnivoice.studio` | Adopt only when docs exceed ~20 markdown files and need search/versioning. Tauri, the framework OmniVoice already depends on, uses Starlight — well-traveled choice. Material for MkDocs entered maintenance mode in November 2025 per Docsio's 2026 review — **avoid** for new docs. |
| Project | What they do |
|---------|--------------|
| **OBS Studio** | Docs at `obsproject.com/docs` (Sphinx, separate repo). Install paths in README, wiki for community-contributed. CI doesn't gate on docs drift. |
| **Audacity** | Manual at `manual.audacityteam.org` (MediaWiki). README is minimal. Install path = "use the installer." No automated sync. |
| **Tauri** | Docs at `v2.tauri.app` (Astro Starlight, separate repo `tauri-apps/tauri-docs`). README is minimal. Heavy reliance on community contributions and PR review. |
| **VS Code** | Docs at `code.visualstudio.com/docs` (separate repo, Markdown). README is minimal. Manual sync; docs team is staffed. |
- [Tauri docs (Astro Starlight)](https://github.com/tauri-apps/tauri-docs) — HIGH (reference for "if we ever move off README")
- [OBS Studio docs](https://docs.obsproject.com/) — HIGH (Sphinx, separate site reference)
- [Audacity Manual](https://manual.audacityteam.org/) — HIGH (MediaWiki reference)
- [Docsio: Material for MkDocs 2026 review (maintenance mode)](https://docsio.co/blog/mkdocs-material) — MEDIUM (third-party review, but signal aligns with project's own GitHub activity)
- [Docsio: Starlight 2026 review](https://docsio.co/blog/starlight-docs) — MEDIUM
## Installation
# No new Python dependencies needed for Capabilities 1, 2, 3, 5.
# Only Capability 4 adds a runtime dep:
# Verify no regressions:
# Should show single versions of each; no duplicates.
## Alternatives Considered
| Recommended | Alternative | When to Use Alternative |
|-------------|-------------|-------------------------|
| HF token via in-app Settings → `huggingface_hub.login()` | OS keyring via `keyring` package | Only if a security hardening milestone later demands OS-native credential storage. Not worth the cross-platform native-dep cost for v0.3.x. |
| Prefilled-URL GitHub Issues | GitHub App + device flow + authenticated POST | When milestone budget can afford registering a public GitHub App and hosting a token-exchange function. Defer. |
| Prefilled-URL GitHub Issues | Sentry / `sentry-tauri` | Never — violates the "no third-party telemetry endpoint" constraint in PROJECT.md. |
| `UV_PYTHON_INSTALL_MIRROR` chain + `only-system` fallback | Bundle Python in the Tauri installer | Adds ~30 MB to every installer for ~5% of users. Revisit if the bootstrap is still a top complaint in v0.4. |
| In-repo Markdown docs | Astro Starlight standalone site | When docs grow past ~20 pages and need full-text search. Tauri provides a precedent if/when we get there. |
| In-repo Markdown docs | MkDocs / Material for MkDocs | **Avoid** for new sites — Material for MkDocs is in maintenance mode as of Nov 2025. |
## What NOT to Use
| Avoid | Why | Use Instead |
|-------|-----|-------------|
| `HfFolder.save_token()` directly | Older API; v1.x `login()` does the same plus git-credential integration and is the documented path | `huggingface_hub.login(token=val, add_to_git_credential=False)` |
| Setting `HF_TOKEN` via shell rc files as the *only* persistence mechanism | Different per OS, fragile, opaque to the user, breaks in installer-launched processes that don't source shell rc | Write to `$HF_HOME/token` via `login()`. Document env var as override only. |
| `setx` for HF token persistence | Doesn't propagate to current shell; common source of "I set it but it's empty" bug reports | `[Environment]::SetEnvironmentVariable(...,"User")` in PowerShell, or the in-app Settings field |
| PAT-based GitHub Issues posting from OmniVoice | Would require shipping or asking for a token; breaks local-first promise | Prefilled-URL pattern (user submits from their browser) |
| `sentry-tauri` for OmniVoice | Third-party telemetry endpoint — violates PROJECT.md constraint | Local-only `backend.log` rotation + opt-in prefilled-URL reporter |
| `hf_transfer` for downloads | Deprecated in favor of `hf-xet` per HF docs | Default `huggingface_hub` (uses `hf-xet` automatically when available) |
| `--python-preference managed` (default) without mirror config in restricted-network installers | Hits GitHub CDN, times out, user sees raw `uv` error | Configure `UV_PYTHON_INSTALL_MIRROR` + retry chain + `only-system` final fallback |
| Material for MkDocs as a *new* docs choice | Entered maintenance mode November 2025 | If docs site is eventually needed, use Astro Starlight (Tauri precedent) |
## Stack Patterns by Variant
- Set `UV_PYTHON_INSTALL_MIRROR` to one of the gh-proxy URLs at install time
- Set `UV_DEFAULT_INDEX=https://pypi.tuna.tsinghua.edu.cn/simple` (China) or document VPN requirement (Russia)
- Fall back to `UV_PYTHON_PREFERENCE=only-system` if all mirrors fail
- Increase `UV_HTTP_TIMEOUT=120`, `UV_HTTP_RETRIES=5`
- Default path: in-app Settings field → `login()` → file at `$HF_HOME/token`
- Power-user path: `export HF_TOKEN=...` in shell rc (documented but not promoted)
- Both paths are read at HF library import time; env var wins on conflict
- Default path: in-app "Report a bug" → prefilled GitHub Issues URL → user reviews + submits in browser
- All optional capture toggles default ON except "include reproduction file" (privacy)
- No path posts to any URL except `github.com/{owner}/{repo}/issues/new` (rendered locally as a URL, opened via `shell.open`)
- `uv add supertonic` → new TTSBackend subclass in `backend/services/tts_backend.py`
- Auto-detected and added to the engine picker in Settings
- ~400 MB model download on first synthesize call, cached in `$HF_HUB_CACHE`
- Existing IndexTTS/CosyVoice/etc. installs are untouched (no shared model weights)
## Version Compatibility
| Package A | Compatible With | Notes |
|-----------|-----------------|-------|
| `supertonic@1.3.1` | `onnxruntime>=1.17`, `numpy>=1.24`, `huggingface_hub>=0.20` | All deps already satisfied transitively by current `pyproject.toml`. |
| `huggingface_hub>=1.12` | `transformers>=5.3.0` (current pin) | `HfFolder` retained as deprecated alias; `login()`/`get_token()` are the canonical APIs. |
| `uv>=0.5` | `UV_PYTHON_INSTALL_MIRROR`, `UV_PYTHON_PREFERENCE` | Both env vars stable since uv 0.4.x. |
| Tauri v2 + `@tauri-apps/api/shell` | `shell.open()` for the prefilled-URL pattern | Already in the desktop app; no new permission needed beyond what the existing "open external link" plugin grants. |
## Sources
- [Hugging Face Hub environment variables](https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables) — HIGH (verified against v1.12.1 docs, current 2026)
- [Hugging Face Hub authentication API](https://huggingface.co/docs/huggingface_hub/en/package_reference/authentication) — HIGH (verified `login()` is the canonical 1.x API)
- [Microsoft `setx` reference](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/setx) — HIGH (confirms "current shell" gotcha)
- [PowerShell `about_Environment_Variables`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_environment_variables) — HIGH
- [uv environment variables reference](https://docs.astral.sh/uv/reference/environment/) — HIGH (verified all mirror + retry env vars)
- [uv issue #5224 — python-build-standalone mirror](https://github.com/astral-sh/uv/issues/5224) — HIGH (feature shipped)
- [uv issue #14187 — venv on Chinese network](https://github.com/astral-sh/uv/issues/14187) — HIGH (confirms user pain, justifies fallback chain)
- [uv `python-preference` semantics](https://github.com/astral-sh/uv/blob/main/docs/concepts/python-versions.md) — HIGH
- [Supertone/supertonic-3 model card](https://huggingface.co/Supertone/supertonic-3) — HIGH (official, 99M params, 31 languages, OpenRAIL-M)
- [supertone-inc/supertonic GitHub](https://github.com/supertone-inc/supertonic) — HIGH (official inference API)
- [supertonic 1.3.1 on PyPI](https://pypi.org/project/supertonic/) — HIGH (released 2026-05-18, MIT code license; bumped from 1.2.3 after Phase 3 research)
- [onnx-community/Supertonic-TTS-ONNX](https://huggingface.co/onnx-community/Supertonic-TTS-ONNX) — HIGH (ONNX file structure)
- [GitHub Docs: Authenticating to the REST API](https://docs.github.com/en/rest/authentication/authenticating-to-the-rest-api) — HIGH
- [GitHub Docs: Generating a user access token for a GitHub App](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/generating-a-user-access-token-for-a-github-app) — HIGH (device flow reference)
- [sindresorhus/new-github-issue-url](https://github.com/sindresorhus/new-github-issue-url) — HIGH (canonical prefilled-URL reference impl)
- [sentry-tauri](https://github.com/timfish/sentry-tauri) — MEDIUM (reviewed, rejected on PROJECT.md constraint, not on quality)
- [dautovri/mirrors-china](https://github.com/dautovri/mirrors-china) — MEDIUM (community-maintained, verify URLs are still live before pinning in production)
- [Tauri 2 docs (Astro Starlight reference)](https://v2.tauri.app/) — HIGH (precedent for docs framework if we ever migrate)
- [Docsio: Material for MkDocs entered maintenance mode Nov 2025](https://docsio.co/blog/mkdocs-material) — MEDIUM (third-party review, but signal aligns with the project's own GitHub commit activity)
<!-- GSD:stack-end -->
<!-- GSD:conventions-start source:CONVENTIONS.md -->
## Conventions
**Versioning (hard rule, owner-set 2026-06-11; single-source 2026-06-16):** main is always **latest release + 1 patch**. **`frontend/package.json` is the SINGLE SOURCE OF TRUTH for the app version** — vite injects `__APP_VERSION__` from it (first-run footer + every auto bug report), and `frontend/src-tauri/tauri.conf.json` reads its bundle version from it (`"version": "../package.json"`, so the MSI/dmg/updater version can't drift from the UI). Three toolchain-required **mirrors** are kept equal to it and bumped in lockstep — `frontend/src-tauri/Cargo.toml` + `pyproject.toml` (cargo/uv need a literal) and `backend/core/version.py`'s `_FALLBACK_VERSION` (the frozen-backend last resort; at runtime the backend reads its version from package metadata via `importlib.metadata`, which `backend.spec`'s `copy_metadata('omnivoice')` makes work in the frozen build too). Never hand-edit any mirror or re-hardcode a literal in `tauri.conf.json`. Guarded by `tests/test_app_version.py` (`test_all_version_files_in_lockstep` + `test_tauri_version_derives_from_package_json`). The moment `vX.Y.Z` is released, bump `package.json` (+ the mirrors) to `X.Y.(Z+1)`. Consequences:
- Every PR and preview build identifies as the **next** version. Preview builds stamp `X.Y.(Z+1)-N` (run number), which semver-sorts **above** the last stable `X.Y.Z` — the updater ordering is natural, no comparator tricks needed.
- Releasing = tag `vX.Y.(Z+1)` from main (version files already match), then immediately bump main to `X.Y.(Z+2)`. **Owner override (2026-07-01): the post-release bump is now MANUAL — the `version-bump` job in release.yml is opt-in behind the `AUTO_VERSION_BUMP` repo variable (default off), so `main` stays at the released version until the owner explicitly asks to bump.** (Historically the bump auto-ran; re-enable that by setting `AUTO_VERSION_BUMP=true`.) When pinned, `main` == the released tag; preview-build ordering and "release + 1" only resume once a bump is requested.
- Docker: `ghcr.io/debpalash/omnivoice-studio:latest` = **main** (rolling preview); `:X.Y.Z` + `:X.Y` + `:stable` = tagged releases. `:latest` is the preview channel by design — stable users pin `:stable` or a version tag.
- Do not bump minor/major or invent RCs/codenames without the owner asking. No "defer to next version" labels — scope is absorbed or declined, never re-versioned.
**Docs-sync (hard rule, owner-set 2026-06-11):** any change that alters something these docs describe — README.md, CONTRIBUTING.md, SECURITY.md, SUPPORT.md, LICENSE, or `docs/**` (install flows, Docker tag semantics, platform support, versioning/release behavior, review process, supported versions) — must update those docs **in the same PR** as the change. If a doc impact is discovered after merge, the docs fix is the immediate next commit, not backlog. Stale docs are treated as bugs.
**Release notes / changelog (hard rule, owner-set 2026-06-16):** every tagged release gets a **high-quality, user-facing `## [X.Y.Z] — DATE` section in `CHANGELOG.md`** before (or in the same hour as) the tag — never the "Auto-generated release for vX.Y.Z…" fallback. `release.yml` extracts that section verbatim as the GitHub Release body (the `Extract CHANGELOG section for tag` step), so a missing/empty section ships a bare release. Quality bar = the existing house style: a one-paragraph headline, then `### Added` / `### Fixed` / `### Changed` / `### License` / `### CI` subsections; each entry is a **bold one-line lead** (what the user gets), 13 lines of plain-English why, and the `(#NNN)` issue/PR ref — grouped by theme, written for users, **not** raw commit dumps. This applies to **preview builds too**: preview release notes summarize what's new on `main` since the last stable, in the same style. Workflow: as features merge, keep `## [Unreleased]` current; at release time rename it to the version + date. If a release was already cut with the fallback body, the next action is to backfill `CHANGELOG.md` **and** `gh release edit <tag>` the live body — not backlog.
**Versioning (hard rule):** Everything ships on `v0.3.0`. Never mention, suggest, or label anything with a version bump — no v0.4, no RCs, no "defer to next version", no future-version labels — unless the user explicitly asks to bump. Zero unprompted version chatter.
**Localization (hard rule):** No hardcoded non-English (CJK) **user-facing text** anywhere in the codebase except the translation layer (`frontend/src/i18n/`). All UI strings go through i18n (`t('...')` keys in `locales/*.json`); native language names live in `i18n/index.ts` (`LANGUAGES`). Functional CJK is allowed and tracked via the allowlist in `tests/test_no_hardcoded_cjk.py` — text-processing regexes, model/engine vocabulary & identifiers (e.g. CosyVoice speaker IDs), localized error matching, demo/eval data, and test fixtures. CI fails on any hardcoded CJK outside the allowlist; to add legitimate functional CJK, extend `_ALLOWED_FILES` there with a justification.
**Fix quality (hard rule, owner-set 2026-06-16):** Fix issues *properly* and future-maintenance-proof — don't stop at the symptom. Root-cause fully, fix the whole **class** of the bug (not just the one reported instance), add a fail-before/pass-after regression test, and harden against recurrence (e.g. if a lockfile drift only fails in Docker, also make CI catch it). Go the extra mile where it durably pays off. Be token-efficient about it — extra **effort**, not extra **verbosity**: no padding, no redundant re-checks, the smallest correct change that is also recurrence-proof. Don't be shy to spend the effort a proper fix needs; do be shy about wasting tokens.
**Keep main green (hard rule, owner-set 2026-06-16):** A merge must **never break `main`'s CI**. Before a change lands, verify the *full* CI matrix would pass — every workflow in `.github/workflows/` **and** `deploy/Dockerfile`, not only the checks you happened to run. Dependency / lockfile / config changes must be validated against **all** consumers. Specifically: `frontend/` is a bun **workspace monorepo** — the lockfile is the repo-root `bun.lock`, and `deploy/Dockerfile` runs `bun install --frozen-lockfile`, so any `frontend/package.json` change requires regenerating root `bun.lock` and confirming `bun install --frozen-lockfile` passes (plain `bun install` in `ci.yml` silently tolerates drift, so CI-green ≠ Docker-green). Likewise re-check CodeQL/Security on code changes and the Tauri `cargo` build on Rust/dep changes.
Other conventions not yet established. Will populate as patterns emerge during development.
<!-- GSD:conventions-end -->
@@ -68,9 +210,16 @@ No project skills found. Add skills to any of: `.claude/skills/`, `.agents/skill
<!-- GSD:skills-end -->
<!-- GSD:workflow-start source:GSD defaults -->
## Workflow
## GSD Workflow Enforcement
Direct repo edits are authorized (owner decision, 2026-07-08). The GSD command gate that used to live here referenced `/gsd-quick` / `/gsd-debug` / `/gsd-execute-phase` skills that are not installed in this environment; the owner chose to keep working directly rather than restore them. The working conventions that matter are in **Conventions** above — versioning, docs-sync, changelog, localization, fix quality, keep-main-green — plus: gate every merge on the "Tests (backend + frontend)" check passing and the PR being MERGEABLE, and check the open-PR queue before implementing any community-reported fix (contributors may have already submitted one).
Before using Edit, Write, or other file-changing tools, start work through a GSD command so planning artifacts and execution context stay in sync.
Use these entry points:
- `/gsd-quick` for small fixes, doc updates, and ad-hoc tasks
- `/gsd-debug` for investigation and bug fixing
- `/gsd-execute-phase` for planned phase work
Do not make direct repo edits outside a GSD workflow unless the user explicitly asks to bypass it.
<!-- GSD:workflow-end -->
+2 -101
View File
@@ -18,7 +18,6 @@ Thanks for your interest in improving OmniVoice Studio! This guide covers everyt
### Prerequisites
- [Git](https://git-scm.com/)
- `curl` (used by the Bun / uv / rustup install one-liners on macOS and Linux)
- [Bun](https://bun.sh/) (frontend package manager)
- [uv](https://docs.astral.sh/uv/) (Python environment manager)
- [ffmpeg](https://ffmpeg.org/) (audio/video processing)
@@ -43,22 +42,11 @@ This starts both services:
### Desktop App (Tauri)
```bash
bun run desktop # dev: hot-reload Tauri shell + backend
bun run desktop-prod # production: builds, bundles the backend, then launches
bun run desktop
```
Both run `uv sync` first (so the Python backend env is set up) and start the
backend automatically — you do **not** start it separately. Use the exact script
names: there is no `desktop=prod` (note the **hyphen** in `desktop-prod`).
`desktop-prod` is Windows-aware (auto-detects bash/git; see `scripts/desktop-prod.mjs`).
Requires [Rust](https://rustup.rs/) and platform-specific Tauri dependencies — see the [Tauri prerequisites](https://v2.tauri.app/start/prerequisites/).
If the app opens but stays on the **setup splash with no buttons**, the Python
backend didn't finish starting — the splash surfaces the stall reason, a log
panel, and a **Retry** button (and Settings → Logs → Backend has the full trace).
The most common from-source cause is `uv` or Python not being on your PATH.
---
## Project Structure
@@ -160,7 +148,7 @@ class MyEngineBackend(TTSBackend):
- **Components**: Functional components with hooks
- **State**: Zustand stores in `src/stores/`, organized by slice
- **CSS**: **Utilities-first + shadcn/ui, one stylesheet.** UI is built on the shadcn/ui primitives in `src/components/ui/` (wrapped by the `src/ui/` barrel, themed to the OmniVoice palette), composed with Tailwind v4 utility classes. **All styling now lives in a single file — `src/index.css`**: the `@theme` / `[data-theme]` token foundation plus the irreducible set utilities can't express (`@keyframes`, glassmorphism/`backdrop-filter`, pseudo-elements, `:has()`, unlayered cascade overrides, and styling hooks on library-generated DOM like virtualized rows / WaveSurfer). The per-component `.css` files were eliminated in the CSS→Tailwind/shadcn migration — **do not create new ones.** Reach for shadcn primitives + utilities; if a rule is genuinely irreducible, add it to `src/index.css` with a provenance comment. (The only other `.css` is the test-only visual harness. See `docs/shadcn-migration.md`.)
- **CSS**: Vanilla CSS in component-level files — no Tailwind
- **Naming**: `PascalCase` for components, `camelCase` for hooks and utils
### Rust (Tauri)
@@ -170,30 +158,6 @@ class MyEngineBackend(TTSBackend):
---
## Frontend file structure & size limits
Frontend code stays modular so an edit loads one small file, not a 1900-line
one. The rules:
- **Size caps:** **soft 300 lines**, **hard 500 lines** per `.jsx` file.
Anything over 500 lines must be split. (The cap does **not** apply to
`src/index.css` — it is the single, intentional styling foundation and the
only app stylesheet; see the CSS rule above.)
- **Pages are thin orchestrators.** A file in `frontend/src/pages/` is just
layout + routing + state wiring that composes feature components — no inline
sub-component over ~50 lines.
- **One component per file.** Co-locate `Foo.jsx` + `Foo.test.jsx` together in a
per-page feature folder under `frontend/src/components/` (e.g.
`components/settings/`, `components/dub/`). Styling is **not** co-located —
it's utilities + shadcn, with any irreducible rules in `src/index.css`.
- **Shared bits go in a `primitives/` folder** inside the feature folder
(`components/settings/primitives/` is the existing example).
- **Enforced by ESLint `max-lines`** (`max: 500`) — **warn-only for now** so it
never breaks CI, with the goal of upgrading to `error` once the backlog of
oversized files clears.
---
## Commit Messages
Write clear, concise messages. The PR title becomes the squash-merge commit.
@@ -228,69 +192,6 @@ cd frontend/src-tauri && cargo check
---
---
## What code review looks like
Every PR is reviewed by two AI reviewers before a human looks at it:
- **CodeRabbit** posts a walkthrough (with a sequence diagram, and an ASCII
before/after sketch for UI changes), inline findings, and warning-mode
pre-merge checks against the project's hard rules.
- **Greptile** reviews with the same project rubrics and learns from 👍/👎
reactions on its comments — react to train it.
Both are advisory, not gating: CI and the maintainer's approval decide. Don't
be surprised by detailed bot comments minutes after you open a PR — address
what's right, push back (in a reply) on what's wrong.
**Commit & PR conventions:** conventional-commit style with a scope
(`fix(dub): …`, `feat(setup): …`) and link the issue (`Closes #N` / `Refs #N`)
in the title or body.
## Quality gates your PR must pass
- **Cross-platform parity (hard rule):** anything that ships in default mode
must behave identically on macOS, Windows, and Linux. Platform-specific
*implementation* is fine; platform-divergent *default behavior* is a P0.
Platform-only features go behind an explicit opt-in (Settings toggle, env
var, or CLI flag).
- **i18n — all 21 locales (hard rule):** every user-facing string goes through
`t('...')` and the key must exist in **all 21** files under
`frontend/src/i18n/locales/`. Translate; don't copy English into non-English
locales. CI fails on hardcoded CJK outside the allowlist in
`tests/test_no_hardcoded_cjk.py` (extend `_ALLOWED_FILES` with a
justification for legitimate functional CJK).
- **DB schema changes** go through an alembic migration with a tested upgrade
path — existing `omnivoice_data/` must keep working with no manual steps.
- **Engine back-compat:** already-installed engines (model weights on disk)
must not require reinstall or re-download.
- **Local-first:** no new outbound calls except GitHub Issues (opt-in
reporting) and HuggingFace model downloads. Never log or persist secrets or
absolute home paths.
- **Security posture:** the backend serves loopback HTTP — treat every
query/path/form parameter as hostile. User-chosen filesystem destinations
are authorized in the Tauri process (save dialog), never via HTTP params.
## Contribution licensing
OmniVoice Studio is **AGPL-3.0-only**, and the maintainer also offers a
**commercial license** (see [LICENSE](LICENSE)). By submitting a contribution
you agree that:
1. you have the right to submit it (your own work, or compatibly licensed);
2. it is licensed to the project under **AGPL-3.0**; and
3. you grant the project maintainer a perpetual, worldwide, non-exclusive
right to also distribute your contribution under the project's commercial
license terms.
This inbound grant is what keeps the dual-license model viable. If you can't
agree to (3) for a particular contribution, say so in the PR and we'll discuss
before merging. Adding a `Signed-off-by:` line (DCO) to your commits is
appreciated but not required.
---
## Need Help?
- **Stuck on setup?** Ask in [Discord #help](https://discord.gg/bzQavDfVV9)
+175 -392
View File
@@ -4,49 +4,44 @@
<h3>The open-source ElevenLabs alternative.</h3>
<p>Real-time dictation, zero-shot voice cloning, and cinematic video dubbing — all on your desktop.<br/>Open-source, no API keys, fully local. <b>646 languages.</b></p>
<p>
<a href="https://github.com/debpalash/OmniVoice-Studio/stargazers"><img src="https://img.shields.io/github/stars/debpalash/OmniVoice-Studio?style=flat-square&color=f59e0b" alt="Stars" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/OmniVoice-Studio?style=flat-square&color=10b981" alt="Release" /></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="License" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/issues"><img src="https://img.shields.io/github/issues/debpalash/OmniVoice-Studio?style=flat-square&color=ef4444" alt="Issues" /></a>
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Discord-Join_Community-5865F2?style=flat-square&logo=discord&logoColor=white" alt="Discord" /></a>
</p>
<p>
<a href="#quickstart">Quickstart</a> ·
<a href="#features">Features</a> ·
<a href="#why-ovs">Why OVS</a> ·
<a href="#why-omnivoice-studio">Why OmniVoice Studio?</a> ·
<a href="#tts-engines">TTS Engines</a> ·
<a href="#asr-engines">ASR Engines</a> ·
<a href="#openai-api">API</a> ·
<a href="#sponsors">Sponsors</a> ·
<a href="#sponsor--donate">Donate</a> ·
<a href="#contributing">Contributing</a> ·
<a href="https://discord.gg/bzQavDfVV9">Discord</a> ·
<a href="README_CN.md"><strong>简体中文</strong></a>
</p>
<p>
<a href="https://github.com/debpalash/OmniVoice-Studio/stargazers"><img src="https://img.shields.io/github/stars/debpalash/OmniVoice-Studio?style=flat-square&color=f59e0b" alt="Stars" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/github/v/release/debpalash/OmniVoice-Studio?style=flat-square&color=10b981" alt="Release" /></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-AGPL--3.0-blue?style=flat-square" alt="License" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/issues"><img src="https://img.shields.io/github/issues/debpalash/OmniVoice-Studio?style=flat-square&color=ef4444" alt="Issues" /></a>
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/Discord-Join_Community-5865F2?style=flat-square&logo=discord&logoColor=white" alt="Discord" /></a>
<a href="https://ko-fi.com/debpalash"><img src="https://img.shields.io/badge/Ko--fi-Support_Us-FF5E5B?style=flat-square&logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
<a href="https://paypal.me/palashCoder"><img src="https://img.shields.io/badge/PayPal-Donate-00457C?style=flat-square&logo=paypal&logoColor=white" alt="PayPal" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_aarch64.dmg"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_x64_en-US.msi"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_amd64.AppImage"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/download/v0.2.7/OmniVoice.Studio_0.2.7_amd64.deb"><img src="https://img.shields.io/badge/Debian-.deb-A81D33?style=for-the-badge&logo=debian&logoColor=white" alt="Download Debian .deb" /></a>
</p>
<p>
<sub><b>macOS:</b> first launch needs a one-time approval — right-click → <b>Open</b> (or System Settings → Privacy &amp; Security → <b>"Open Anyway"</b> on macOS 15). No Terminal needed. <a href="docs/install/macos.md#gatekeeper-quarantine">Why?</a></sub>
</p>
</div>
<br/>
<div align="center">
<img src="docs/screenshot-launchpad.png" alt="OmniVoice Studio — Launchpad" width="100%"/>
</div>
> **Your voice is the most personal data you have. So why rent it back from a cloud?** Every mainstream voice tool ships your audio to someone else's server and bills you monthly for the privilege. OmniVoice Studio flips that: clone, design, dub, and dictate on your own hardware — 646 languages, no meter running, nothing leaving your machine.
<div align="center">
| 🔑 No API keys | 🙅 No accounts | ☁️ No cloud | 💳 No subscription |
|:---:|:---:|:---:|:---:|
| nothing to paste in | nothing to sign up for | your audio stays home | it's your computer |
<img src=".github/assets/social-preview.png" alt="OmniVoice Studio — The open-source ElevenLabs alternative" width="100%"/>
</div>
> [!WARNING]
> **OmniVoice Studio is in active beta.** Things may break between releases — for the latest features and fixes, clone the repo and run from source rather than the pre-built installers. Bug reports and PRs are very welcome: [open an issue](https://github.com/debpalash/OmniVoice-Studio/issues) or [join Discord](https://discord.gg/bzQavDfVV9).
> **OmniVoice Studio is in active beta.** Things may break between releases. For the latest features and fixes, clone the repo and run from source rather than using pre-built installers. Bug reports and PRs are very welcome [open an issue](https://github.com/debpalash/OmniVoice-Studio/issues) or [join Discord](https://discord.gg/bzQavDfVV9).
<div align="center">
<br/>
@@ -58,157 +53,79 @@
<br/>
<a id="screenshots"></a>
## 📸 See it in action
<table>
<tr>
<td align="center" width="50%">
<img src="docs/screenshot-studio.png" alt="Studio" width="100%"/>
<br/><b>Studio</b><br/>
<sub>Generate &amp; clone in one workspace — a 3-second clip mirrors any voice, 646 languages, zero-shot.</sub>
</td>
<td align="center" width="50%">
<img src="docs/screenshot-design.png" alt="Voice Design" width="100%"/>
<br/><b>Voice Design</b><br/>
<sub>Build new voices from scratch — gender, age, accent, pitch, emotion, dialect.</sub>
</td>
</tr>
<tr>
<td align="center">
<img src="docs/screenshot-gallery.png" alt="Voice Gallery" width="100%"/>
<br/><b>Voice Gallery</b><br/>
<sub>Browse ready-made archetype voices with language filters — or build your own library.</sub>
</td>
<td align="center">
<img src="docs/screenshot-dub.png" alt="Video Dubbing" width="100%"/>
<br/><b>Video Dubbing</b><br/>
<sub>A real dub, end to end: 37 segments transcribed, translated to Bengali, re-voiced, and timed — ready to export as MP4.</sub>
</td>
</tr>
<tr>
<td align="center">
<img src="docs/screenshot-engines.png" alt="Settings — Engines" width="100%"/>
<br/><b>Settings → Engines</b><br/>
<sub>The engine compatibility matrix — 14 TTS engines with per-engine GPU preflight, no silent CPU fallback.</sub>
</td>
<td align="center">
<img src="docs/screenshot-settings.png" alt="Settings — Models" width="100%"/>
<br/><b>Settings → Models</b><br/>
<sub>One-click model store — auto-detects your platform (CUDA / MPS / CPU) and recommends the right models.</sub>
</td>
</tr>
<tr>
<td align="center">
<img src="docs/screenshot-openapi.png" alt="Settings — API Reference" width="100%"/>
<br/><b>API Reference</b><br/>
<sub>The full local REST API, embedded — every endpoint documented with copy-paste client snippets.</sub>
</td>
<td align="center">
<img src="docs/screenshot-updates.png" alt="Settings — What's New" width="100%"/>
<br/><b>What's New</b><br/>
<sub>In-app changelog reader — see exactly what shipped in each release without leaving the app.</sub>
</td>
</tr>
</table>
---
<a id="features"></a>
## ✨ Features
The eight headliners — and twelve more waiting under the fold.
## Features
<table>
<tr>
<td align="center" width="25%">
<td align="center" width="33%">
<h3>🎙️ Voice Cloning</h3>
<p>3-second clip → mirror any voice.<br/><b>646 languages</b>, zero-shot.</p>
</td>
<td align="center" width="25%">
<td align="center" width="33%">
<h3>🎨 Voice Design</h3>
<p>Gender, age, accent, pitch, speed,<br/>emotion, dialect — <b>dial it in</b>.</p>
</td>
<td align="center" width="25%">
<td align="center" width="33%">
<h3>🎬 Video Dubbing</h3>
<p>YouTube URL or file → transcribe →<br/>translate → re-voice → <b>MP4</b>.</p>
</td>
<td align="center" width="25%">
<h3>📖 Audiobook Editor</h3>
<p>Import text, EPUB, or PDF. Auto-chapter,<br/>loudnorm, metadata. Export <b>.m4b</b>.</p>
</td>
</tr>
<tr>
<td align="center" valign="top">
<h3>🎭 Stories</h3>
<p>Multi-voice editor. Assign voices<br/>per-line, preview, <b>export full cast</b>.</p>
</td>
<td align="center" valign="top">
<h3>⌨️ Dictation Widget</h3>
<p><kbd>⌘</kbd>+<kbd>⇧</kbd>+<kbd>Space</kbd> from <b>any app</b>.<br/>Transcribes, auto-pastes, disappears.</p>
<p><code>⌘+⇧+Space</code> from <b>any app</b>.<br/>Transcribes, auto-pastes, disappears.</p>
</td>
<td align="center" valign="top">
<h3>🔐 100% Local</h3>
<p>No keys, no cloud, no accounts.<br/><b>Your machine only</b>.</p>
<h3>🔊 Vocal Isolation</h3>
<p>Demucs-powered. Splits speech<br/>from music, <b>keeps the background</b>.</p>
</td>
<td align="center" valign="top">
<h3>👥 Speaker Diarization</h3>
<p>Pyannote + WhisperX.<br/><b>Auto-identifies</b> who said what.</p>
</td>
</tr>
<tr>
<td align="center" valign="top">
<h3>📦 Batch Queue</h3>
<p>Drop <b>50 videos</b>, walk away.<br/>Progress bars per job.</p>
</td>
<td align="center" valign="top">
<h3>🤖 MCP Server</h3>
<p>Use OmniVoice from <b>Claude</b>,<br/>Cursor, or any MCP client.</p>
</td>
<td align="center" valign="top">
<h3>🛡️ AI Watermark</h3>
<p>AudioSeal (Meta). <b>Invisible</b>,<br/>survives compression.</p>
</td>
</tr>
<tr>
<td align="center" valign="top">
<h3>🔐 100% Local</h3>
<p>No keys, no cloud, no accounts.<br/><b>Your machine only</b>.</p>
</td>
<td align="center" valign="top">
<h3>⚡ GPU Auto-Detect</h3>
<p>CUDA · MPS · ROCm · CPU.<br/>≤8 GB? <b>Auto-offloads</b>.</p>
</td>
<td align="center" valign="top">
<h3>🧩 Extensible</h3>
<p>Subclass <code>TTSBackend</code>,<br/>add any engine in <b>~50 lines</b>.</p>
</td>
</tr>
</table>
<details>
<summary><b>…and 12 more</b> — isolation, diarization, batch, watermarking, diagnostics, and friends</summary>
<br/>
- 🔊 **Vocal Isolation** — Demucs-powered: splits speech from music and keeps the background bed.
- 👥 **Speaker Diarization** — Pyannote + WhisperX auto-identify who said what.
- 📦 **Batch Queue** — drop 50 videos, walk away; per-job progress bars.
- 🛡️ **AI Watermark** — AudioSeal (Meta): invisible, survives compression.
- 🔬 **Diagnostics** — self-check suite, error journal, scrubbed diagnostic bundles.
- ⚡ **GPU Auto-Detect** — CUDA · MPS · ROCm (Linux, opt-in) · CPU; ≤8 GB VRAM auto-offloads.
- 🧭 **Engine routing** — preflight GPU check per engine; no silent CPU fallback.
- 🧩 **Extensible** — subclass `TTSBackend`, add any engine in ~50 lines.
- 🎒 **Portable personas** — export voices as `.ovsvoice` bundles: identity + watermark.
- ♾️ **Unlimited TTS** — sentence-chunked generation, no length cap, streaming via WebSocket.
- 🌐 **Remote backend** — point the UI at a remote server; Tailscale-friendly, bearer auth.
- 🧠 **Dictation + LLM** — local-LLM cleanup of transcripts, optional echo cancellation.
</details>
---
<a id="quickstart"></a>
## Quickstart
## ⚡ Quickstart
Per-OS install guides — pick yours and follow it end-to-end:
<div align="center">
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/macOS-DMG_(Apple_Silicon)-000?style=for-the-badge&logo=apple&logoColor=white" alt="Download macOS DMG" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Windows-MSI_(x64)-0078D4?style=for-the-badge&logo=windows&logoColor=white" alt="Download Windows MSI" /></a>
<a href="https://github.com/debpalash/OmniVoice-Studio/releases/latest"><img src="https://img.shields.io/badge/Linux-AppImage_(x64)-FCC624?style=for-the-badge&logo=linux&logoColor=black" alt="Download Linux AppImage" /></a>
<br/>
<sub><b>macOS:</b> first launch needs a one-time approval — right-click → <b>Open</b> (or System Settings → Privacy &amp; Security → <b>"Open Anyway"</b> on macOS 15). No Terminal needed. <a href="docs/install/macos.md#gatekeeper-quarantine">Why?</a></sub>
<br/>
<sub><b>Intel Macs are not supported for the local backend:</b> the app UI installs, but the Python backend cannot run because PyTorch no longer ships Intel-Mac (x86_64) wheels (<a href="https://github.com/debpalash/OmniVoice-Studio/issues/889">#889</a>) — see <a href="docs/install/macos.md">docs/install/macos.md</a>.</sub>
</div>
- **macOS** — [docs/install/macos.md](docs/install/macos.md)
- **Windows** — [docs/install/windows.md](docs/install/windows.md)
- **Linux** — [docs/install/linux.md](docs/install/linux.md)
- **Docker** — [docs/install/docker.md](docs/install/docker.md)
Pick your OS and follow the guide end-to-end:
- 🍎 **macOS** — [docs/install/macos.md](docs/install/macos.md)
- 🪟 **Windows** — [docs/install/windows.md](docs/install/windows.md)
- 🐧 **Linux** — [docs/install/linux.md](docs/install/linux.md)
- 🐳 **Docker** — [docs/install/docker.md](docs/install/docker.md) · [Docker Hub: `palashdeb/omnivoice-studio`](https://hub.docker.com/r/palashdeb/omnivoice-studio)
<details>
<summary><b>🧰 Stuck? Self-checks, tokens &amp; restricted networks</b></summary>
<br/>
Run the built-in self-check first — **Settings → About → "Run
Stuck? Run the built-in self-check first — **Settings → About → "Run
self-check"** in the app, or `uv run python backend/main.py --diagnose` from
a checkout (`--deep` also test-loads the active engine). Then see
[docs/install/troubleshooting.md](docs/install/troubleshooting.md) for the
@@ -219,17 +136,59 @@ bundle"** packages scrubbed logs + the self-check report for bug reports.
For Hugging Face token setup, see
[docs/setup/huggingface-token.md](docs/setup/huggingface-token.md). For
diarization-specific gating, see
[docs/features/diarization.md](docs/features/diarization.md). For download
speed, the ⚡ fast-download (Xet) status, and restricted-network / mirror
options, see [docs/downloading-models.md](docs/downloading-models.md).
[docs/features/diarization.md](docs/features/diarization.md).
</details>
## Screenshots
<table>
<tr>
<td align="center" width="50%">
<img src="docs/screenshot-clone.png" alt="Voice Clone" width="100%"/>
<br/><b>Voice Clone</b><br/>
<sub>Drop a 3-second clip → mirror any voice. 646 languages, zero-shot.</sub>
</td>
<td align="center" width="50%">
<img src="docs/screenshot-design.png" alt="Voice Design" width="100%"/>
<br/><b>Voice Design</b><br/>
<sub>Build new voices from scratch — gender, age, accent, pitch, style.</sub>
</td>
</tr>
<tr>
<td align="center">
<img src="docs/screenshot-dub.png" alt="Video Dubbing" width="100%"/>
<br/><b>Video Dubbing</b><br/>
<sub>Upload or paste a YouTube URL. Transcribe, translate, re-voice, export.</sub>
</td>
<td align="center">
<img src="docs/screenshot-gallery.png" alt="Voice Gallery" width="100%"/>
<br/><b>Voice Gallery</b><br/>
<sub>Search YouTube, browse categories, download clips, build your library.</sub>
</td>
</tr>
<tr>
<td align="center">
<img src="docs/screenshot-settings.png" alt="Settings — Models" width="100%"/>
<br/><b>Settings → Models</b><br/>
<sub>15 models. One-click install. Auto-detects your platform (CUDA / MPS / CPU).</sub>
</td>
<td align="center">
<img src="docs/screenshot-libraryprojects.png" alt="Projects" width="100%"/>
<br/><b>Projects</b><br/>
<sub>Dub projects, voice profiles, generation history, exports — all searchable.</sub>
</td>
</tr>
<tr>
<td align="center" colspan="2">
<img src="docs/screenshot-logs.png" alt="Settings — Logs" width="100%"/>
<br/><b>Settings → Logs</b><br/>
<sub>Live backend, frontend, and Tauri runtime logs. Filter, refresh, clear.</sub>
</td>
</tr>
</table>
---
<a id="why-ovs"></a>
## 💡 Why OmniVoice?
## Why OmniVoice Studio?
ElevenLabs charges **$5$330/mo** and processes your audio on their servers. OmniVoice Studio runs **on your hardware, with no usage limits.**
@@ -238,20 +197,15 @@ ElevenLabs charges **$5$330/mo** and processes your audio on their servers. O
| **Pricing** | $5$330/mo, per-character billing | Free & open-source (AGPL-3.0) · [Commercial license](#license) for proprietary use |
| **Voice Cloning** | ✅ 3s clip | ✅ 3s clip, zero-shot |
| **Voice Design** | ✅ Gender, age | ✅ Gender, age, accent, pitch, style, dialect |
| **Audiobook / Stories** | ❌ | ✅ Full audiobook editor + multi-voice stories (EPUB/PDF import, .m4b export) |
| **Languages** | 32 | **646** |
| **Video Dubbing** | ✅ Cloud-only | ✅ Fully local |
| **Data Privacy** | Audio sent to cloud | **Nothing leaves your machine** |
| **API Keys** | Required | Not needed |
| **GPU Support** | N/A (cloud) | CUDA · Apple Silicon · ROCm (Linux) · CPU |
| **GPU Support** | N/A (cloud) | CUDA · Apple Silicon · ROCm · CPU |
| **Desktop App** | ❌ | ✅ macOS · Windows · Linux |
| **TTS Engines** | 1 | **14** (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS) |
| **ASR Engines** | 1 | **9** (WhisperX, Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet, Moonshine, FunASR, isolated Faster-Whisper, sherpa-onnx live dictation) |
| **MCP Server** | ❌ | ✅ Use from Claude, Cursor, any MCP client |
| **Self-check** | ❌ | ✅ Diagnostics suite, error journal, scrubbed debug bundles |
| **Customizable** | ❌ Closed | ✅ Fork it, extend it, ship it |
Professional-grade voice AI, minus the subscription and the cloud.
OmniVoice Studio gives you professional-grade AI tools without the subscription or the cloud.
<div align="center">
<br/>
@@ -262,271 +216,125 @@ Professional-grade voice AI, minus the subscription and the cloud.
---
## 🖥️ System Requirements
## System Requirements
| | **Minimum** | **Recommended** |
|---|---|---|
| **OS** | Windows 10, macOS 12+ (Apple Silicon), Ubuntu 24.04+ (glibc 2.39+) | Any modern 64-bit OS |
| **OS** | Windows 10, macOS 12+, Ubuntu 20.04+ | Any modern 64-bit OS |
| **RAM** | 8 GB | 16 GB+ |
| **VRAM (GPU)** | 4 GB (auto-offloads TTS to CPU) | 8 GB+ (NVIDIA RTX 3060+) |
| **Disk** | 10 GB free (models + cache) | 20 GB+ SSD |
| **Python** | 3.10+ (managed by `uv`) | 3.113.12 |
| **GPU** | Optional — CPU works | NVIDIA CUDA · Apple Silicon MPS · AMD ROCm (Linux only) |
| **GPU** | Optional — CPU works | NVIDIA CUDA · Apple Silicon MPS · AMD ROCm |
> [!TIP]
> On GPUs with **≤8 GB VRAM**, OmniVoice automatically offloads TTS to CPU during transcription — no config needed. A dedicated GPU is not required; the entire pipeline runs on CPU (just slower).
> [!NOTE]
> **AMD GPUs:** ROCm acceleration is **Linux-only and opt-in** — pick **"AMD GPU (ROCm)"** on the first-run setup screen or set `OMNIVOICE_TORCH_VARIANT=rocm` ([docs/install/linux.md](docs/install/linux.md#amd-gpu-rocm)). **On Windows, AMD GPUs (incl. Ryzen AI iGPUs) run CPU-only**: PyTorch has no Windows ROCm wheels, so Windows GPU acceleration is NVIDIA/CUDA-only ([docs/install/windows.md](docs/install/windows.md#gpu-support)).
### TTS Engines
> [!IMPORTANT]
> **macOS Intel (x86_64) is unsupported for the local backend:** the app UI installs, but the Python backend cannot run because PyTorch no longer ships Intel-Mac wheels ([#889](https://github.com/debpalash/OmniVoice-Studio/issues/889)). Intel-Mac users can still point the UI at a remote backend on another machine — see [docs/install/macos.md](docs/install/macos.md).
<a id="tts-engines"></a>
### 🗣️ TTS Engines
**14 engines, one picker.** OmniVoice (default, 600+ languages) is always available; CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, and Sherpa-ONNX are opt-in and auto-detected — plus six lazy-installed heavyweights (IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS). Switch in **Settings → TTS Engine** or via the `OMNIVOICE_TTS_BACKEND` env var — the selection applies everywhere synthesis happens: single-clip generation, Voice Cloning, Video Dubbing, and Batch TTS.
<details>
<summary><b>📊 The full matrix</b> — 14 engines × platform × clone/instruct × license</summary>
<br/>
OmniVoice ships a multi-engine TTS backend. The default engine (OmniVoice) is always available; additional engines are opt-in and auto-detected. Switch engines in **Settings → TTS Engine** or via the `OMNIVOICE_TTS_BACKEND` env var.
| Engine | Languages | Clone | Instruct | Linux | macOS ARM | Windows | License |
|--------|:---------:|:-----:|:--------:|:-----:|:---------:|:-------:|:-------:|
| **OmniVoice** (default) | 600+ | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Built-in |
| **CosyVoice 3** | 9 + 18 dialects | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
| **GPT-SoVITS** | 5 | ✅ | — | ✅ CUDA/CPU | — | ✅ CUDA/CPU | MIT |
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
| **MOSS-TTS-Nano** | 20 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **KittenTTS** | English | — | — | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
| **MLX-Audio** (Kokoro, Qwen3-TTS, CSM, Dia, …) | Multi | Varies | Varies | ❌ | ✅ Native | ❌ | Varies |
| **Sherpa-ONNX** | 20+ | | | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **IndexTTS 2** ⚡ | Multi | ✅ | | ✅ CUDA | — | ✅ CUDA | Apache-2.0 |
| **OmniVoice GGUF** ⚡ | 600+ | | | ✅ CPU | ✅ CPU | ✅ CPU | Built-in |
| **Supertonic 3** ⚡ | 31 | — | — | ✅ CPU | ✅ CPU | ✅ CPU | OpenRAIL-M |
| **MOSS-TTS-v1.5** ⚡ (8B) | 31 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **dots.tts** ⚡ (2B) | 24 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ❌ | Apache-2.0 |
| **Confucius4-TTS** ⚡ | 14 | ✅ | — | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **VoxCPM2** | 30 | | | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
| **MOSS-TTS-Nano** | 20 | ✅ | | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **KittenTTS** | English | | | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
> **CUDA** = GPU-accelerated · **MPS** = Apple Silicon Metal · **CPU** = runs everywhere, slower for large models · KittenTTS and MOSS-TTS-Nano run realtime on CPU · MLX-Audio is Apple Silicon only · ⚡ = lazy-registered (installed on first use)
>
> **Clone** matters beyond single-clip generation: Video Dubbing (and any Batch job with a pinned voice) needs reference-audio cloning to preserve speaker identity, so picking a Clone-less engine (KittenTTS, Sherpa-ONNX, Supertonic 3) as the active engine fails those jobs up front with an actionable message instead of silently falling back to OmniVoice.
>
> **MOSS-TTS-v1.5** (8B, ~16 GB weights) and **dots.tts** (2B, ~9 GB weights) are heavyweight opt-in engines that run in their own isolated venv from a local clone — see [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) and [dots.tts](docs/engines/dots-tts.md). Neither claims Apple-Silicon **MPS** (upstream is CUDA/CPU only; on a Mac they run on CPU). dots.tts upstream is Linux/macOS only — no Windows path. **Confucius4-TTS** (14-language cross-lingual zero-shot cloning) is similar — its own Python 3.10 venv from a clone; CUDA recommended, CPU validated end-to-end (slow, ~17× realtime; no MPS — tested slower than CPU); see [Confucius4-TTS](docs/engines/confucius4-tts.md).
> **CUDA** = GPU-accelerated · **MPS** = Apple Silicon Metal · **CPU** = runs everywhere, slower for large models · KittenTTS and MOSS-TTS-Nano run realtime on CPU · MLX-Audio is Apple Silicon only.
</details>
### ASR Engines
<a id="asr-engines"></a>
### 🎧 ASR Engines
**10 engines** — they power dictation, video dubbing, and subtitles. **WhisperX** is the cross-platform default (~100 languages, word-level timing); the rest are opt-in and auto-detected. Switch in **Settings → Engines** (the ASR Engines table — same picker TTS has), or pin one with the `OMNIVOICE_ASR_BACKEND` env var (the env var wins over the Settings pick). Nine run fully on-device; one (OpenAI-compatible) is an optional remote client for pointing at Qwen3-ASR or another compatible server — see below.
<details>
<summary><b>📊 The full lineup</b> — 10 engines, what each is best at, and compute-type notes</summary>
<br/>
OmniVoice ships a multi-engine ASR (speech-to-text) backend that powers dictation, video dubbing, and subtitle generation — all fully local. **WhisperX** is the cross-platform default; the rest are opt-in and auto-detected. Switch in **Settings → ASR Engine** or via the `OMNIVOICE_ASR_BACKEND` env var.
| Engine | `OMNIVOICE_ASR_BACKEND` | Languages | Best for |
|--------|-------------------------|:---------:|----------|
| **WhisperX** (default) | `whisperx` | ~100 | Dubbing & subtitles — word-level timing via wav2vec2 forced alignment |
| **Faster-Whisper** | `faster-whisper` | ~100 | Fast transcription on Linux / macOS / Windows (CTranslate2) |
| **Faster-Whisper (isolated)** | `faster-whisper-isolated` | ~100 | Same as Faster-Whisper but crash-isolated in a subprocess — an ASR crash won't take down the app |
| **MLX Whisper** | `mlx-whisper` | ~100 | Native Apple Silicon speed (Apple MLX / Metal) |
| **PyTorch Whisper** | `pytorch-whisper` | ~100 | CUDA / CPU fallback via 🤗 Transformers (no cuDNN 8 needed) |
| **Parakeet TDT** | `nemo-parakeet` | English + 25 EU | SOTA accuracy at ~10× realtime even on CPU, auto language detection (NVIDIA NeMo, CUDA/CPU) |
| **PyTorch Whisper** | `pytorch-whisper` | ~100 | CUDA / CPU fallback via 🤗 Transformers |
| **Parakeet TDT** | `nemo-parakeet` | English + 25 EU | SOTA English accuracy, auto language detection (NVIDIA NeMo, GPU only) |
| **Moonshine** | `moonshine` | English | Edge / low-latency, ONNX |
| **FunASR** | `funasr` | 50+ | All-in-one multilingual — built-in VAD + inline speaker diarization (SenseVoice) |
| **sherpa-onnx** (live dictation) | `sherpa-onnx-asr` | 25 EU + 90+ | Live, faster-than-real-time dictation — small streaming/offline ONNX models (Parakeet TDT v3/v2, streaming Zipformer & Paraformer, Whisper Tiny), CPU, identical on macOS / Windows / Linux. Picked per-model in **Settings → Voice**. |
| **OpenAI-compatible** ⚠️ remote | `openai-compat-asr` | Server-dependent | A path to **Qwen3-ASR** today (self-hosted server, no transformers wait), any OpenAI-compatible transcription endpoint, or OpenAI's own API — no install, configure in **Settings → Models**. Audio leaves your machine to whatever server you point it at; see [docs/engines/openai-compatible-asr.md](docs/engines/openai-compatible-asr.md). |
> Whisper-family engines cover ~100 languages; **FunASR / SenseVoice** adds an all-in-one multilingual path with built-in voice-activity detection and inline speaker diarization. **sherpa-onnx** powers the live dictation model picker — you talk and text appears as you speak. Every engine runs on-device — no API keys, no cloud.
> **GPU without efficient float16?** On older NVIDIA GPUs (Maxwell/Pascal, GTX 16xx) or after a CTranslate2/cuDNN mismatch, the CTranslate2 ASR engines (WhisperX, Faster-Whisper) can't run `float16` and OmniVoice automatically retries on `int8` — no config needed. If transcription still fails, pin the compute type with the `ASR_COMPUTE_TYPE` env var (escape hatch): `ASR_COMPUTE_TYPE=int8` (or `float32` for CPU). Set it to `int8` and restart the backend.
</details>
> Whisper-family engines cover ~100 languages; **FunASR / SenseVoice** adds an all-in-one multilingual path with built-in voice-activity detection and inline speaker diarization. Every engine runs on-device — no API keys, no cloud.
---
## 🏗️ Architecture
## Architecture
```
┌─────────────────────────────────────────────────────────────
Frontend (React)
│ DubTab · VoiceConsole · Stories · Audiobook · Gallery
│ Dictation · BatchQueue · Diagnostics · MCP Client │
├─────────────────────────────────────────────────────────────┤
Backend (FastAPI)
│ 100+ API endpoints · SSE+WSS streaming · SQLite │
├──────────┬──────────┬──────────┬──────────┬────────────────┤
WhisperX │ Demucs │OmniVoice │ Pyannote │ Engine Routing
(+7 ASR │ Source │ (+10 │ Diariz- │ ↳ GPU preflight
│ engines) │ Sep. │ TTS) │ ation │ ↳ No silent CPU │
└──────────┴──────────┴──────────┴──────────┴────────────────┘
CUDA / MPS / ROCm / CPU (auto-detected + routed)
┌─────────────────────────────────────────────────┐
│ Frontend (React) │
│ DubTab · VoicePreview · BatchQueue · Gallery │
├─────────────────────────────────────────────────┤
│ Backend (FastAPI) │
97 API endpoints · SSE streaming · SQLite
├──────────┬──────────┬──────────┬────────────────┤
│ WhisperX │ Demucs │OmniVoice │ Pyannote │
ASR │ Source TTS │ Diarization
│ Sep. │ │
└──────────┴──────────┴──────────┴────────────────┘
CUDA / MPS / ROCm / CPU (auto-detected)
```
<a id="openai-api"></a>
## 🔌 OpenAI-compatible API
Already have a script, agent, or tool that speaks OpenAI's audio API? Point it at `http://localhost:3900/v1` — no key needed, no code changes. The backend ships a drop-in surface for the audio endpoints, wired to whichever TTS/ASR engine you have active (and yes, `voice` accepts your cloned voice-profile IDs).
| Endpoint | What it does |
|---|---|
| `POST /v1/audio/speech` | TTS — text in; `mp3` / `wav` / `flac` / `opus` / `pcm` out. `tts-1` / `tts-1-hd` map to your active engine; OpenAI voice names (`alloy`, …) are accepted. |
| `POST /v1/audio/transcriptions` | STT — audio file in; `json`, `text`, `verbose_json`, `srt`, or `vtt` out. `whisper-1` maps to your active ASR engine. |
| `GET /v1/audio/voices` | OmniVoice extension — lists every voice profile and engine, so clients can discover your clones. |
```sh
curl http://localhost:3900/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"model": "tts-1", "voice": "alloy", "input": "Generated on my own hardware.", "response_format": "wav"}' \
--output speech.wav
```
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:3900/v1", api_key="none") # any string works — nothing checks it
result = client.audio.transcriptions.create(model="whisper-1", file=open("clip.wav", "rb"))
print(result.text)
```
Want the whole surface (100+ endpoints)? The full REST API reference is embedded in the app — **Settings → OpenAPI Reference** (Scalar-powered), or the `{}` button in the footer.
### 📓 Run on Google Colab (community)
No local GPU? A community member ([@shakib30](https://github.com/shakib30)) maintains a working Colab notebook: [shakib30/OmniVoice-Studio-google-colab](https://github.com/shakib30/OmniVoice-Studio-google-colab). Community-maintained — issues with the notebook go there; issues with OmniVoice itself come here.
### 🤝 Agent Skills
Teach your AI agent (Claude Code, Cursor, Codex, …) to use OmniVoice with one command:
```sh
npx skills add debpalash/omnivoice-studio
```
Ships two [skills](https://skills.sh): **`omnivoice`** — speak and transcribe through your local install (including your cloned voices) from any agent, free and offline; and **`oss-maintainer`** — the maintainer methodology this project is run with, for anyone running their own OSS project with an agent.
---
## 🗺️ Roadmap
## Roadmap
### ✅ Shipped
| Category | Features |
|----------|----------|
| **Dubbing** | Full pipeline (transcribe→translate→synthesize→mux), scene-aware splitting, lip-sync scoring, streaming TTS |
| **Voice** | Zero-shot cloning, voice design, A/B comparison, voice preview widget, gallery with favorites/tags |
| **Audio** | Demucs vocal isolation, per-segment gain, selective track export, stem/SRT/VTT/MP3 export |
| **Multi-Lang** | Multi-language batch picker, batch dubbing queue with sequential GPU execution |
| **Diarization** | Pyannote ML diarization, auto speaker clone extraction, per-speaker voice assignment |
| **Infra** | Docker deployment, CUDA/MPS/ROCm auto-detect, cuDNN 8 compat, VRAM-aware model offloading |
| **AI Provenance** | AudioSeal invisible watermarking (SynthID-like), video logo overlay, watermark detection API |
| **UX** | Undo/redo, keyboard shortcuts, drag-and-drop, session persistence, glassmorphism design system |
| **Real-time Events** | WebSocket event bus — instant sidebar refresh on data mutations, exponential backoff reconnect |
| **State Management** | Zustand store migration — `uiSlice`, `pillSlice`, `dubSlice`, `generateSlice`, `prefsSlice`, `glossarySlice` |
| **Desktop** | Cross-platform Tauri installers (macOS DMG, Windows MSI, Linux deb/AppImage), auto-update infrastructure |
| **Windows Hardening** | Cross-platform log paths, Triton workaround, HF symlink bypass, 300s health check timeout |
| **Dictation** | Global system-wide hotkey (`⌘+⇧+Space`), frameless floating widget, streaming ASR via WebSocket, auto-paste |
| **Batch Pipeline** | Full batch TTS: extract → transcribe → translate → generate → mix → export, with live progress tracking |
### 🔜 Up Next
- 🎬 **Lip-sync v2** — visual speech timing with wav2lip
- 📖 **Audiobook Editor** — chapter-aware long-form narration
- 🌐 **Hosted Demo** — try OmniVoice without installing anything
- 🔌 **Plugin Marketplace** — community-contributed TTS engines and effects
- 🎵 **Real-time Voice Changer** — live microphone transformation during calls
<details>
<summary><b>✅ Everything shipped so far</b> — the receipts, by category</summary>
<br/>
| Category | Features |
|----------|----------|
| **Longform** | Audiobook editor (text/EPUB/PDF → chaptered .m4b), Stories multi-voice editor, two-pass loudnorm mastering, crash-resume for interrupted renders, pronunciation control + SSML-lite prosody |
| **Dubbing** | Full pipeline (transcribe→translate→synthesize→mux), scene-aware splitting, lip-sync scoring, streaming TTS, per-speaker voice assignment, Smart Fit timing + second-pass QC, dedicated Dub home |
| **Voice** | Zero-shot cloning, voice design, A/B comparison, voice preview widget, gallery with favorites/tags, portable persona bundles (`.ovsvoice`), voice console workspace |
| **Audio** | Demucs vocal isolation, per-segment gain, selective track export, stem/SRT/VTT/MP3 export, unlimited-length TTS via sentence-chunked generation |
| **Multi-Lang** | Multi-language batch picker, batch dubbing queue with sequential GPU execution |
| **Diarization** | Pyannote ML diarization, auto speaker clone extraction, per-speaker voice assignment |
| **ASR** | 9 engines (WhisperX, Faster-Whisper, isolated Faster-Whisper, MLX Whisper, PyTorch Whisper, Parakeet TDT, Moonshine, FunASR/SenseVoice, sherpa-onnx live dictation), crash-isolated subprocess backend |
| **TTS** | 14 engines (OmniVoice, CosyVoice 3, GPT-SoVITS, VoxCPM2, MOSS-TTS-Nano, KittenTTS, MLX-Audio, Sherpa-ONNX, + lazy: IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, Confucius4-TTS), engine routing with GPU preflight |
| **Infra** | Docker deployment, CUDA/MPS/ROCm auto-detect, cuDNN 8 compat, VRAM-aware model offloading, engine routing (no silent CPU fallback), diagnostics suite & error journal, restricted-network mirror support |
| **AI Provenance** | AudioSeal invisible watermarking (SynthID-like), video logo overlay, watermark detection API |
| **UX** | Undo/redo, keyboard shortcuts, drag-and-drop, session persistence, glassmorphism design system, UI scale fix for Linux/WebKitGTK |
| **Real-time Events** | WebSocket event bus — instant sidebar refresh on data mutations, exponential backoff reconnect |
| **State Management** | Zustand store migration — `uiSlice`, `pillSlice`, `dubSlice`, `generateSlice`, `prefsSlice`, `glossarySlice` |
| **Desktop** | Cross-platform Tauri installers (macOS DMG — Apple Silicon; Intel unsupported for the local backend, #889 — Windows MSI, Linux deb/AppImage), auto-update infrastructure, single-instance enforcement, close-to-tray, macOS Gatekeeper fix |
| **Dictation** | Global system-wide hotkey (`⌘+⇧+Space`), frameless floating widget, streaming ASR via WebSocket, auto-paste, customizable hotkey, local-LLM transcript refinement |
| **Batch Pipeline** | Full batch TTS: extract → transcribe → translate → generate → mix → export, with live progress tracking |
| **MCP Server** | OmniVoice as a local TTS/STT provider for Claude, Cursor, and any MCP client |
| **Remote Backend** | Point the desktop UI at a remote backend URL with bearer auth (Tailscale-documented) |
| **Reliability** | Stall watchdog on bootstrap splash, per-engine GPU compatibility matrix, actionable errors for non-executable engine binaries, setuptools auto-repair |
</details>
---
<a id="sponsor--donate"></a>
## 💜 Sponsor / Donate
OmniVoice Studio is built by one developer using Claude Code and AI agents — and the agent bills are real. Over the last three months I've spent thousands of dollars on Claude subscriptions to keep the features shipping, the bugs fixed, and your issues answered. If OmniVoice has created value for you, helping cover those bills means I can keep developing full-time.
<div align="center">
**This month's agent bill fund**
<img src="https://img.shields.io/badge/raised_%2410_of_%24200-5%25-EAB308?style=for-the-badge" alt="$10 / $200 raised" />
<br/><br/>
<a href="https://ko-fi.com/debpalash"><img src="https://img.shields.io/badge/Ko--fi-Support_❤️-FF5E5B?style=for-the-badge&logo=ko-fi&logoColor=white" alt="Ko-fi" /></a>
&nbsp;&nbsp;
<a href="https://paypal.me/palashCoder"><img src="https://img.shields.io/badge/PayPal-Donate-00457C?style=for-the-badge&logo=paypal&logoColor=white" alt="PayPal" /></a>
<br/>
<sub>Every dollar goes directly to agent bills — keeping OmniVoice development continuous.</sub>
</div>
<a id="sponsors"></a>
### 🌟 Sponsors
OmniVoice is **free** and **AGPL-3.0** — no paid tier, no SaaS revenue. Sponsors keep development going, and in return get a logo slot here, in the app, and (for top tiers) on the project website. It's a thank-you, never a paywall. **[See tiers & become a sponsor →](SPONSORS.md)**
<div align="center">
<!-- SPONSORS:START — logo slots are filled here as sponsors come aboard; see SPONSORS.md -->
**Your logo here** — [become a sponsor](SPONSORS.md)
<!-- SPONSORS:END -->
</div>
<sub>💡 GitHub also shows a **Sponsor** button at the top of this repo, wired to the same links via <a href=".github/FUNDING.yml"><code>.github/FUNDING.yml</code></a>.</sub>
---
## 💬 Community
## Community
<div align="center">
<a href="https://discord.gg/bzQavDfVV9"><img src="https://img.shields.io/badge/💬_Discord-Join_Community-5865F2?style=for-the-badge&logo=discord&logoColor=white" alt="Join Discord" /></a>
<br/>
<sub>We respond to setup questions within hours, not days.</sub>
</div>
<details>
<summary><b>What happens in there</b></summary>
<br/>
| Channel | What happens there |
|---------|--------------------|
| `#announcements` | Release news and the big moments — new versions land here first |
| `#releases` + `#changelog` | Every build and exactly what's inside it |
| `#issues` | Bug reports as forum posts — triaged straight into GitHub issues |
| `#ideas` | Feature requests, discussed and voted on |
| `#discuss-ideas` | Design talk before things get built |
| `#general` | Setup help, GPU troubleshooting, and showing off your dubs |
| `#showcase` | Members share their dubs, clones, and voice designs |
| `#help` | Setup issues, GPU troubleshooting, model questions |
| `#feature-requests` | Vote on what gets built next |
| `#dev` | Architecture discussions, PR reviews, engine integrations |
| `#announcements` | Release notes, breaking changes, early access |
</details>
**[→ Join the Discord](https://discord.gg/bzQavDfVV9)** — we respond to setup questions within hours, not days.
---
<a id="contributing"></a>
## Contributing
## 🤝 Contributing
Yes please — bug fixes, new TTS engine adapters, UI improvements, docs, translations. All of it.
We welcome contributions of all kinds — bug fixes, new TTS engine adapters, UI improvements, docs, and translations.
- 📖 Read the **[Contributing Guide](CONTRIBUTING.md)** for setup, code style, and PR workflow
- 🐛 Browse [good first issues](https://github.com/debpalash/OmniVoice-Studio/labels/good%20first%20issue)
@@ -534,26 +342,18 @@ Yes please — bug fixes, new TTS engine adapters, UI improvements, docs, transl
---
## FAQ
## FAQ
<details>
<summary><b>Is this really as good as ElevenLabs?</b></summary>
<br/>
Honest answer: <b>it depends on what you're doing.</b>
<b>Where OmniVoice is genuinely competitive:</b> voice cloning from a clean reference clip (state-of-the-art open diffusion TTS), language coverage (646 languages vs. their 32), and everything structural — no per-character billing, no usage caps, no audio leaving your machine, full pipeline customizability (10 TTS engines, 10 ASR engines, your choice of translation).
<b>Where ElevenLabs still wins:</b> out-of-the-box consistency and polish, especially for English TTS. Their one model is heavily tuned; our quality depends on which engine you pick, your hardware, and — for cloning — the reference audio (a dry, close-mic clip clones dramatically better than a noisy or echoey one).
<b>For dubbing specifically:</b> a dub is a chain — transcription → translation → cloning → synthesis — and the output is only as good as its weakest link on <i>your</i> source material. Noisy or accented source audio degrades transcription, which degrades everything downstream; some language pairs translate better than others. If parts of a dub come out incoherent, check the segment table's <i>original</i> text first: if the transcription is already wrong there, switch the ASR engine (Settings → Engines) or use cleaner source audio — that's usually the fix, not the voice.
Try it on your real material — it's free and takes one download. Many users find it replaces ElevenLabs outright; some keep both for different jobs. Both outcomes are fine with us.
For voice cloning and dubbing, yes — OmniVoice uses a state-of-the-art diffusion TTS model with 646 languages (ElevenLabs supports 32). Quality is comparable for most use cases. Where ElevenLabs wins is in their polished cloud API and pre-made voice library. OmniVoice wins on privacy, cost, language coverage, and customizability.
</details>
<details>
<summary><b>Does it work on Apple Silicon (M1/M2/M3/M4)?</b></summary>
<br/>
Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are available for faster transcription on Apple hardware. <b>Intel Macs are not supported</b>: the app UI installs, but the local Python backend cannot run because PyTorch no longer ships Intel-Mac wheels (<a href="https://github.com/debpalash/OmniVoice-Studio/issues/889">#889</a>) — an Intel Mac can only be used with a remote backend.
Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are available for faster transcription on Apple hardware.
</details>
<details>
@@ -577,14 +377,12 @@ Yes. MPS acceleration is auto-detected. MLX-optimized Whisper models are availab
<details>
<summary><b>Can I add my own TTS engine?</b></summary>
<br/>
Yes. OmniVoice uses a <b>built-in backend registry</b>. To add an engine in ~50 lines, subclass <code>TTSBackend</code> in <code>backend/services/tts_backend.py</code> and add it to the <code>_REGISTRY</code> dictionary. Fourteen engines are built in: OmniVoice, CosyVoice 3, GPT-SoVITS, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, KittenTTS, Sherpa-ONNX, plus lazy-registered IndexTTS 2, OmniVoice GGUF, Supertonic 3, MOSS-TTS-v1.5, dots.tts, and Confucius4-TTS. See the <a href="#tts-engines">TTS Engines</a> section for details.
Yes. OmniVoice uses a <b>built-in backend registry</b>. To add an engine in ~50 lines, subclass <code>TTSBackend</code> in <code>backend/services/tts_backend.py</code> and add it to the <code>_REGISTRY</code> dictionary at the bottom. Six engines are built in: OmniVoice, CosyVoice, MLX-Audio (14+ sub-engines), VoxCPM2, MOSS-TTS-Nano, and KittenTTS. See the <a href="#tts-engines">TTS Engines</a> section for details.
</details>
---
<a id="license"></a>
## 📜 License
## License
OmniVoice Studio is free and open-source software under the [**GNU Affero General Public License v3.0 (AGPL-3.0)**](https://www.gnu.org/licenses/agpl-3.0.html).
@@ -596,7 +394,7 @@ The bundled `omnivoice/` TTS model by Han Zhu remains Apache-2.0 upstream. See [
---
## 🙏 Acknowledgments
## Acknowledgments
OmniVoice Studio is built on the shoulders of exceptional open-source work:
@@ -609,20 +407,6 @@ OmniVoice Studio is built on the shoulders of exceptional open-source work:
| [**CTranslate2**](https://github.com/OpenNMT/CTranslate2) | Optimized Transformer inference on CPU and GPU |
| [**AudioSeal (Meta)**](https://github.com/facebookresearch/audioseal) | Invisible neural audio watermarking for AI provenance |
| [**Tauri**](https://tauri.app) | Native desktop app framework |
| [**Supertone / Supertonic 3**](https://huggingface.co/Supertone/supertonic-3) | ONNX TTS engine — 31 languages, CPU-efficient |
| [**Sherpa-ONNX**](https://github.com/k2-fsa/sherpa-onnx) | WASM-ready universal TTS/ASR runtime |
| [**GPT-SoVITS**](https://github.com/RVC-Boss/GPT-SoVITS) | Zero-shot TTS engine — 5 languages, RTF 0.014 |
---
## 🧰 More local open-source from the maker
Like the local-first philosophy? It runs in the family:
| Project | What it is |
|---------|------------|
| [**Opal**](https://github.com/debpalash/Opal) 💠 | **Play everything.** The evolved media player for the next decades of entertainment — video, anime, comics, torrents, Jellyfin/Plex, with local AI built in. |
| [**memxt**](https://github.com/debpalash/memxt) 🧠 | **The fastest benchmarked open-source AI memory system.** 100% local memory for AI agents, with MCP support. |
---
@@ -632,8 +416,7 @@ Like the local-first philosophy? It runs in the family:
If you read this far, you're our kind of person.<br/>
**[⭐ Star this repo](https://github.com/debpalash/OmniVoice-Studio)** so others can find it too.<br/>
**[💬 Join the Discord](https://discord.gg/bzQavDfVV9)** to share what you build.<br/>
**[❤️ Support development](https://ko-fi.com/debpalash)** — fund the AI agent bills that keep OmniVoice shipping.
**[💬 Join the Discord](https://discord.gg/bzQavDfVV9)** to share what you build.
<br/>
-4
View File
@@ -372,13 +372,9 @@ OmniVoice 配备多引擎 TTS 后端。默认引擎(OmniVoice)始终可用
| **MLX-Audio**Kokoro, Qwen3-TTS, CSM, Dia 等) | 多语言 | 因引擎而异 | 因引擎而异 | ❌ | ✅ 原生 | ❌ | 因引擎而异 |
| **VoxCPM2** | 30 | ✅ | ✅ | ✅ CUDA/CPU | ✅ MPS | ✅ CUDA/CPU | Apache-2.0 |
| **MOSS-TTS-Nano** | 20 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **MOSS-TTS-v1.5**(8B,可选装) | 31 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ✅ CUDA/CPU | Apache-2.0 |
| **dots.tts**(2B,可选装) | 24 | ✅ | ❌ | ✅ CUDA/CPU | ✅ CPU | ❌ | Apache-2.0 |
| **KittenTTS** | 英语 | ❌ | ❌ | ✅ CPU | ✅ CPU | ✅ CPU | MIT |
> **CUDA** = GPU 加速 · **MPS** = Apple Silicon Metal · **CPU** = 随处可运行,大模型较慢 · KittenTTS 和 MOSS-TTS-Nano 可在 CPU 上实时运行 · MLX-Audio 仅限 Apple Silicon。
>
> **MOSS-TTS-v1.5**8B,约 16 GB 权重)和 **dots.tts**(2B,约 9 GB 权重)是重量级可选引擎,从本地克隆在独立 venv 中运行——参见 [MOSS-TTS-v1.5](docs/engines/moss-tts-v15.md) 和 [dots.tts](docs/engines/dots-tts.md)。两者均不支持 Apple Silicon **MPS**(上游仅支持 CUDA/CPU;在 Mac 上以 CPU 运行)。dots.tts 上游仅支持 Linux/macOS——无 Windows 路径。
---
+4 -14
View File
@@ -2,20 +2,10 @@
## Supported Versions
| Version | Supported |
|---------|-----------|
| 0.3.x (latest release + `main` previews) | ✅ Current — all fixes land here |
| 0.2.7 | ⚠️ Legacy stable — security fixes only, upgrade recommended |
| < 0.2.7 | ❌ No longer supported |
## Model supply chain
OmniVoice supports models from **public, verifiable sources only** (Hugging
Face repos, official project releases). Privately sold or gated model files
are not supported: an archive from a private source can carry anything
(bundled executables, modified configs), and nobody else can verify or
reproduce it. Treat any privately distributed model file as an untrusted
download, and never run executables bundled with model archives.
| Version | Supported |
|---------|--------------------|
| 0.2.x | ✅ Current release |
| < 0.2 | ❌ No longer supported |
## Reporting a Vulnerability
-119
View File
@@ -1,119 +0,0 @@
<div align="center">
<img src="docs/logo.png" alt="OmniVoice Logo" width="96" />
<h1>Sponsor OmniVoice Studio</h1>
<p><b>Keep the open-source ElevenLabs alternative free, local, and shipping.</b></p>
</div>
---
## Why sponsor?
OmniVoice Studio is built by one developer, in the open, using Claude Code and AI agents — and the agent bills are real. Over the last few months I've spent thousands of dollars on Claude subscriptions to keep features shipping, bugs fixed, and your issues answered.
OmniVoice is **free**, **fully local**, and **AGPL-3.0**. There's no paid tier, no accounts, no cloud, and no SaaS revenue — nothing runs on a server we bill you for, because nothing runs on a server at all. That's the whole point, and it's also why there's no recurring revenue to fund development. Sponsorship is what makes continued full-time work possible.
If OmniVoice has created value for you or your company, sponsoring means the next release keeps coming — and you get a thank-you (and, at most tiers, a logo slot) in return.
### Where your money goes
Every dollar goes to the cost of building OmniVoice — chiefly the **AI agent bills that keep it shipping** (Claude subscriptions and API usage), plus the occasional signing certificate, test hardware, and model-hosting costs. It is not a salary top-up; it's what keeps the lights on for continuous development.
---
## Sponsorship tiers
Tiers are about **visibility and gratitude** — what you get is placement, not gated features (see [Not a paywall](#not-a-paywall)). Higher tiers include everything in the tiers below them.
| Tier | Suggested monthly | What you get |
|------|-------------------|--------------|
| **🥉 Backer** | _set by owner_ <!-- OWNER: set amounts --> | Your name or handle listed in the **Backers** section of this file, with a link of your choice. |
| **🟫 Bronze** | _set by owner_ <!-- OWNER: set amounts --> | Everything above, **plus** a small logo in `SPONSORS.md` **and** in the README [Sponsors section](README.md#sponsors). |
| **🥈 Silver** | _set by owner_ <!-- OWNER: set amounts --> | Everything above, **plus** your logo in the **README** and in the app's **in-app Sponsors page footer** (as that page ships). |
| **🥇 Gold** | _set by owner_ <!-- OWNER: set amounts --> | Everything above, **plus** a **prominent logo slot** and link on the project **website / landing page**. |
> **Amounts are set by the maintainer** — look for the `<!-- OWNER: set amounts -->` markers in this file's source. If you don't see a price that fits, say so in your inquiry; custom and annual arrangements are welcome.
Placements marked "as that page ships" (the in-app Sponsors page and the project website) are on the near-term roadmap. Until they exist, Silver/Gold logos live in `SPONSORS.md` and the README, and are added to the app and site the moment those land — no re-application needed.
---
## How to become a sponsor
**1. Open a sponsorship inquiry (recommended).** This opens a short GitHub form (name/org, logo, tier, contact) so we can get you set up:
> **[→ Open a sponsorship inquiry](https://github.com/debpalash/OmniVoice-Studio/issues/new?template=sponsor.yml)**
**2. Or start recurring support directly:**
- **Ko-fi (recurring or one-time):** [ko-fi.com/debpalash](https://ko-fi.com/debpalash)
- **PayPal (one-time):** [paypal.me/palashCoder](https://paypal.me/palashCoder)
If you sponsor via Ko-fi/PayPal and want a logo slot, still open an inquiry (or drop a note there) so we know who to credit and where to link.
**3. Prefer to talk first?** Reach out directly:
- Email: <!-- OWNER: add your sponsor contact email here if you want one public -->
- Or ask in the `#dev` / `#announcements` channels on [Discord](https://discord.gg/bzQavDfVV9).
---
## Logo & asset guidelines
To make your logo look sharp everywhere (README on GitHub, the in-app page, the website), please send:
- **Format:** **SVG preferred** (scales cleanly); otherwise **PNG at 2× resolution**.
- **Background:** **transparent** — no baked-in white/black box.
- **Contrast:** send a variant that stays legible on **both light and dark** backgrounds, or one light-mode and one dark-mode file (GitHub and the app both render in either theme).
- **Dimensions:** legible at **~40px tall**; keep the wordmark within roughly **480px wide**. Landscape/wordmark shapes work best in the README row.
- **File size:** keep SVGs under ~50 KB and PNGs under ~100 KB.
- **Link target:** the destination URL you want the logo to point to (usually your homepage).
**How your logo gets added:**
- **Easiest:** attach the asset and link in your [sponsorship inquiry](https://github.com/debpalash/OmniVoice-Studio/issues/new?template=sponsor.yml) — the maintainer places it.
- **Or open a PR:** add your asset under `docs/sponsors/` and an entry to the tables in this file. Silver/Gold logos are also wired into the app's in-app Sponsors page (via the `sponsors.js` manifest) and the project website as those surfaces ship.
By sponsoring you confirm you have the right to use the submitted logo and grant OmniVoice permission to display it in the contexts above. We won't alter your logo beyond scaling, and we'll remove it promptly on request.
---
## Current sponsors
OmniVoice doesn't have any sponsors yet — **you could be the first.** These slots fill in as sponsors come aboard.
### 🥇 Gold
_Be the first Gold sponsor — [claim this slot](#how-to-become-a-sponsor)._
### 🥈 Silver
_Open — [become a Silver sponsor](#how-to-become-a-sponsor)._
### 🟫 Bronze
_Open — [become a Bronze sponsor](#how-to-become-a-sponsor)._
### 🥉 Backers
_Open — [become a Backer](#how-to-become-a-sponsor)._
<!-- When a sponsor joins, add them to the matching section above:
- Logo tiers (Bronze+): <a href="https://sponsor.example"><img src="docs/sponsors/name.svg" alt="Name" height="48" /></a>
- Backers: - [Name / handle](https://link) -->
---
## Not a paywall
Sponsorship is a **thank-you, never a paywall.**
Every feature of OmniVoice Studio is and will remain **free** and **open-source under [AGPL-3.0](LICENSE)**. Sponsors do **not** get private builds, gated features, license exceptions, or anything that degrades the experience for people who don't (or can't) pay. What sponsors get is **visibility and our gratitude** — and the knowledge that they're directly funding the next release.
OmniVoice stays local-first and fully functional with zero dollars spent. Sponsoring just helps it keep getting better, faster.
---
<div align="center">
<sub>Thank you for keeping local-first voice AI alive and free. ❤️</sub><br/>
<sub>Questions? <a href="https://github.com/debpalash/OmniVoice-Studio/issues/new?template=sponsor.yml">Open an inquiry</a> · <a href="https://discord.gg/bzQavDfVV9">Discord</a></sub>
</div>
-36
View File
@@ -1,36 +0,0 @@
# Support
## Where to get help
| Channel | Best for |
|---|---|
| [Discord](https://discord.gg/bzQavDfVV9) — `#help` | Setup problems, quick questions, sharing results |
| [GitHub Issues](https://github.com/debpalash/OmniVoice-Studio/issues) | Bugs and feature requests — use the templates; attach the diagnostic bundle (Settings → About → "Save diagnostic bundle") |
| [GitHub Discussions](https://github.com/debpalash/OmniVoice-Studio/discussions) | Design questions, ideas, show & tell |
| Security issues | **Never a public issue** — see [SECURITY.md](SECURITY.md) for private reporting |
## Model sources we support
OmniVoice is built on the idea that everything it runs is **open and available
to everyone**: free, public models with verifiable sources and licenses
(Hugging Face repos, official project releases), so the whole community can
use, test, and debug the same thing.
**We do not support privately sold, paywalled, or gated model files.** A model
delivered privately can't be verified, reproduced, or shared — it doesn't fit
the project's goals, and issues involving such models will be politely closed.
As a general safety rule, never run executables bundled inside any model
archive.
## Before filing a bug
1. Update to the latest release (or `main` if you follow previews) — fixes ship continuously.
2. Run the in-app self-check: **Settings → About → Run self-check**.
3. Search existing issues; add a 👍 + your details to an existing one rather than opening a duplicate.
## Response expectations
This is an open-source project maintained with the help of an automated triage
bot: issues are typically triaged within hours and every report gets a human-
approved response. Reproducible reports with a diagnostic bundle get fixed
fastest.
+1 -20
View File
@@ -13,18 +13,12 @@
# Run: uv run pyinstaller backend.spec --noconfirm --clean
import platform
import sys
from PyInstaller.utils.hooks import collect_data_files, collect_all, collect_submodules, copy_metadata
from PyInstaller.utils.hooks import collect_data_files, collect_all, collect_submodules
IS_MAC_ARM = sys.platform == "darwin" and platform.machine() == "arm64"
datas = []
binaries = []
# Bundle the omnivoice package's .dist-info so importlib.metadata.version()
# resolves inside the frozen build. Without it the backend can't read its own
# version and falls back to the literal in backend/core/version.py — which is
# how a 0.3.6 desktop build shipped reporting "0.3.5" in About + bug reports.
datas += copy_metadata('omnivoice')
hiddenimports = [
# Web stack
'uvicorn', 'uvicorn.logging', 'uvicorn.loops', 'uvicorn.loops.auto',
@@ -33,14 +27,6 @@ hiddenimports = [
'uvicorn.lifespan', 'uvicorn.lifespan.on',
'fastapi', 'fastapi.responses', 'starlette',
'multipart',
# SOCKS proxy support (#959). httpx imports socksio lazily inside a
# try/except (only when a socks5:// proxy env var is set), so
# PyInstaller's static tracer never sees it — without this entry the
# frozen installers keep raising "Using SOCKS proxy, but the 'socksio'
# package is not installed" on every model load under a SOCKS proxy,
# even though pyproject.toml ships the package. Guarded by
# tests/test_socks_proxy.py.
'socksio',
# Core
'uuid', 'asyncio',
@@ -69,11 +55,6 @@ hiddenimports = [
# Pipeline
'yt_dlp', 'demucs', 'demucs.separate',
# Numbers→words for the pre-TTS text normalization pass
# (services/text_normalization.py). Imported inside a function (lazy),
# so pin it explicitly rather than trusting the tracer.
'num2words',
# OmniVoice's own package
'omnivoice', 'omnivoice.models', 'omnivoice.models.omnivoice',
]
-32
View File
@@ -7,13 +7,9 @@ composed at the route or router level without surprises.
Currently exposed:
- `require_loopback`: 403 unless the request came from a loopback origin
(bypassed in explicit server mode see `_server_mode`).
- `ws_remote_authorized`: whether a WebSocket handshake from a non-loopback
client carries the remote API key (Wave 2.3) used by WS endpoints that
keep their own inline loopback guards.
"""
import os
import secrets
from fastapi import HTTPException, Request
@@ -75,31 +71,3 @@ def require_loopback(request: Request) -> None:
if _server_mode():
return
raise HTTPException(status_code=403, detail="loopback origin required")
def remote_api_key() -> str | None:
"""The remote-backend bearer key (Wave 2.3), or None when remote mode is
off. Read at call time so tests can monkeypatch the env."""
return os.environ.get("OMNIVOICE_API_KEY") or None
def ws_remote_authorized(websocket) -> bool:
"""Whether a WebSocket handshake presents the remote API key.
Browser WebSockets cannot set an Authorization header, so the key may
arrive as ``?api_key=`` or via the ``ov_key`` cookie that the bearer
middleware sets on the first authenticated HTTP request. Returns False
when remote mode is off callers keep their loopback-only behavior.
"""
key = remote_api_key()
if not key:
return False
auth = websocket.headers.get("authorization", "")
supplied = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
if not supplied:
supplied = (
websocket.query_params.get("api_key")
or websocket.cookies.get("ov_key")
or ""
)
return secrets.compare_digest(supplied, key)
+6 -7
View File
@@ -18,6 +18,7 @@ Design notes
"""
from __future__ import annotations
import asyncio
import hashlib
import logging
import os
@@ -136,7 +137,7 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
from api.routers.generation import ( # noqa: WPS433 — intentional lazy import
get_model,
_run_inference,
run_on_gpu_pool_guarded,
_gpu_pool,
_safe_torchaudio_save,
)
@@ -146,6 +147,8 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
language = None
text = (a.get("sample_script") or "").strip() or _FALLBACK_SCRIPT
loop = asyncio.get_running_loop()
def _infer(seed: int):
return _run_inference(
model, # _model
@@ -168,18 +171,14 @@ async def _render_archetype_wav(a: dict, out_path: Path) -> None:
"broadcast", # effect_preset
)
# Bounded + pool-reset on hang so a wedged preview render can't starve the
# GPU pool and brick the backend (#730 class).
audio_tensor = await run_on_gpu_pool_guarded(
lambda: _infer(_PREVIEW_SEED), what="Archetype preview generate")
audio_tensor = await loop.run_in_executor(_gpu_pool, _infer, _PREVIEW_SEED)
if _is_unusable_audio(audio_tensor):
# Blank OR a degenerate tonal buzz — retry once on a different seed to
# step off the bad diffusion trajectory. Static message only: the
# archetype id is request-derived (CodeQL log-injection); the seed is a
# module constant, safe to log.
logger.warning("Archetype rendered unusable at seed %d — retrying once", _PREVIEW_SEED)
audio_tensor = await run_on_gpu_pool_guarded(
lambda: _infer(_PREVIEW_SEED + 1), what="Archetype preview generate")
audio_tensor = await loop.run_in_executor(_gpu_pool, _infer, _PREVIEW_SEED + 1)
if _is_unusable_audio(audio_tensor):
raise RuntimeError("the voice engine returned no audible audio for this archetype")
-798
View File
@@ -1,798 +0,0 @@
"""Audiobook creator endpoints (parity Wave 5).
``POST /audiobook/plan`` pure preview: parse a chapter-delimited script
(Markdown ``# H1`` chapters, inline ``[voice:NAME]`` / ``[pause …]``) into the
chapter/span plan, no synthesis.
``POST /audiobook`` the synth job: render each chapter through the active TTS
backend (reusing ``services.audiobook.synthesize_chapter`` + ``chunked_tts``),
then mux the chapter WAVs into a chapterized **m4b** (FFMETADATA1 chapters via
``build_m4b_cmd``). Progress streams as Server-Sent Events, mirroring the dub
pipeline. ffmpeg-gated without ffmpeg the job reports an error event and
stops (the m4b is the only output format).
``GET /audiobook/jobs`` + ``POST /audiobook/resume/{job_id}`` durable
crash-resume: an interrupted render persists its plan + params to a
``resume.json`` manifest in the job work dir, so it can be resumed later (the
content-addressed chapter cache makes finished chapters instant) even without
the original script. The resume UI affordance remains a follow-up.
epub/pdf ingest, ACX mastering shipped; the resume UI surface remains a follow-up.
"""
import asyncio
import json
import logging
import os
import re
import uuid
from fastapi import APIRouter, File, HTTPException, UploadFile
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from services.audiobook import (
parse_audiobook_script,
synthesize_chapter,
)
from services.longform_render import (
LOUDNESS_PRESETS,
build_concat_list,
build_ffmetadata,
build_render_cmd,
prune_cache_dir,
)
from services import longform_resume # pure (no torch) — durable resume manifest
logger = logging.getLogger("omnivoice.audiobook")
router = APIRouter()
# A cover filename as produced by /audiobook/cover: 12 hex chars + image ext.
# An exact-match allowlist is the strongest barrier (and the one CodeQL's
# path-injection query recognizes) — anything else is rejected outright.
_COVER_NAME_RE = re.compile(r"^[0-9a-f]{12}\.(?:jpg|jpeg|png)$")
def _safe_cover_path(cover_path: str | None) -> str | None:
"""Confine a user-supplied cover to the upload directory before it can flow
into ffmpeg.
Covers only ever come from ``/audiobook/cover``, which writes them to
``OUTPUTS_DIR/audiobook_covers`` with a generated name. We rebuild the path
from the basename alone (``os.path.basename`` strips any directory component
or ``..`` traversal) joined onto that fixed directory, so no caller-supplied
path absolute or relative can escape it. Returns the path only if the
file actually exists there, else None."""
if not cover_path:
return None
from core.config import OUTPUTS_DIR
name = os.path.basename(cover_path)
if not _COVER_NAME_RE.match(name):
return None # not a name the upload endpoint could have produced
cover_dir = os.path.realpath(os.path.join(OUTPUTS_DIR, "audiobook_covers"))
real = os.path.realpath(os.path.join(cover_dir, name))
# Containment check on the resolved path itself — it must live inside the
# covers dir. Belt-and-suspenders over the regex+basename above; the
# commonpath form is the path-injection barrier static analysis recognizes.
if os.path.commonpath([real, cover_dir]) != cover_dir:
return None
return real if os.path.isfile(real) else None
class AudiobookPlanRequest(BaseModel):
text: str
default_voice: str | None = None
@router.post("/audiobook/plan")
def audiobook_plan(req: AudiobookPlanRequest) -> dict:
"""Parse a script into a chapter/span plan (pure preview, no synthesis)."""
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
return plan.to_dict()
#: Cover size cap mirrors longform_render's guard (8 MB — a book cover, not a
#: payload). Kept in sync intentionally; the render builder re-validates too.
_COVER_MAX_BYTES = 8 * 1024 * 1024
#: Import upload cap — a generous ceiling for a .txt/.md/.epub manuscript that
#: still stops a memory-exhaustion upload (the whole file is read into RAM).
_IMPORT_MAX_BYTES = 64 * 1024 * 1024
#: Upper bound on chapters in a single /longform/render plan — far above any real
#: book, but stops a pathological request from allocating/holding the job forever.
_MAX_CHAPTERS = 10_000
@router.post("/audiobook/import")
async def audiobook_import(file: UploadFile = File(...)) -> dict:
"""Import a ``.txt``/``.md``/``.epub``/``.pdf`` into a chapter-delimited script.
EPUB is parsed in spine order (stdlib only, local); PDF text is extracted
with pypdf (pure-Python) then chapterized; plain text gets ``# `` headings
inserted ahead of obvious chapter-title lines. Returns the script text (for
the editor) + the resulting chapter count."""
from services.longform_import import (
chapterize_plaintext,
epub_to_chapter_script,
pdf_to_chapter_script,
)
name = (file.filename or "").lower()
data = await file.read()
if not data:
raise HTTPException(status_code=400, detail="empty file")
if len(data) > _IMPORT_MAX_BYTES:
raise HTTPException(status_code=400, detail="file too large (max 64 MB)")
if name.endswith(".epub"):
try:
script = epub_to_chapter_script(data)
except ValueError as e:
raise HTTPException(status_code=400, detail=f"couldn't parse EPUB: {e}")
elif name.endswith(".pdf"):
try:
script = pdf_to_chapter_script(data)
except ValueError as e:
raise HTTPException(status_code=400, detail=f"couldn't parse PDF: {e}")
else:
script = chapterize_plaintext(data.decode("utf-8", "ignore"))
if not script.strip():
raise HTTPException(status_code=400, detail="no text found in the file")
plan = parse_audiobook_script(script)
return {"text": script, "chapters": plan.chapter_count}
@router.post("/audiobook/cover")
async def audiobook_cover(cover: UploadFile = File(...)) -> dict:
"""Upload a cover image; returns a server-side ``path`` to pass back as
``cover_path`` in the synth request. Validated here (jpg/png + size cap) and
again at render time."""
from core.config import OUTPUTS_DIR
ext = os.path.splitext(cover.filename or "")[1].lower()
if ext not in (".jpg", ".jpeg", ".png"):
raise HTTPException(status_code=400, detail="cover must be a .jpg or .png")
data = await cover.read()
if not data or len(data) > _COVER_MAX_BYTES:
raise HTTPException(status_code=400, detail="cover must be between 1 byte and 8 MB")
cover_dir = os.path.join(OUTPUTS_DIR, "audiobook_covers")
os.makedirs(cover_dir, exist_ok=True)
path = os.path.join(cover_dir, f"{uuid.uuid4().hex[:12]}{ext}")
with open(path, "wb") as f:
f.write(data)
return {"path": path}
class AudiobookRequest(BaseModel):
text: str
default_voice: str | None = None # voice profile id; None = engine default
language: str | None = None # None/"Auto" → profile language, else autodetect (#505)
bitrate: str = "128k"
format: str = "m4b" # "m4b" | "mp3"
loudness: str | None = None # None/"off" | "acx" | "podcast" (opt-in)
cover_path: str | None = None # server-side path to a jpg/png cover
# Global tags embedded in the output: {title, author, narrator, year,
# genre, description}. Player-visible (Apple Books / Audible read these).
metadata: dict | None = None
# Optional pronunciation lexicon {word: respelling} applied before synthesis.
lexicon: dict | None = None
def _resolve_voice(profile_id: str | None) -> dict:
"""Map a voice-profile id to (ref_audio, ref_text, instruct, seed).
Compact form of the resolver in generation.py covers locked, design and
clone profiles. Returns all-None for the engine default (no profile).
"""
out = {"ref_audio": None, "ref_text": None, "instruct": None, "seed": None}
if not profile_id:
return out
from core.config import VOICES_DIR
from core.db import db_conn
with db_conn() as conn:
row = conn.execute("SELECT * FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
if not row:
return out
try:
kind = row["kind"] or "clone"
except (KeyError, IndexError):
kind = "clone"
if row["is_locked"] and row["locked_audio_path"]:
out["ref_audio"] = os.path.join(VOICES_DIR, row["locked_audio_path"])
out["ref_text"] = row["ref_text"]
out["instruct"] = row["instruct"]
elif kind == "design":
out["ref_audio"] = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
out["ref_text"] = row["ref_text"] if out["ref_audio"] else None
out["instruct"] = row["instruct"]
else:
out["ref_audio"] = os.path.join(VOICES_DIR, row["ref_audio_path"]) if row["ref_audio_path"] else None
out["ref_text"] = row["ref_text"]
out["instruct"] = row["instruct"]
try:
if row["seed"] is not None:
out["seed"] = row["seed"]
except (KeyError, IndexError):
pass
return out
def _resolve_default_language(language: str | None, default_voice: str | None) -> str | None:
"""Pick the language to thread into the longform synth callable.
Priority (mirrors the single-shot /generate path, #533): an explicit
non-Auto request ``language`` wins; otherwise the selected profile's stored
language drives it; otherwise ``None`` (genuine Auto the engine
autodetects, exactly as before). Hardcoding ``None`` here (#505 B2) let the
engine re-autodetect per chunk, so a non-English clone flipped to the wrong
language on short/ambiguous chapters.
"""
if language and language != "Auto":
return language
if default_voice:
from core.db import db_conn
with db_conn() as conn:
row = conn.execute(
"SELECT language FROM voice_profiles WHERE id=?", (default_voice,)
).fetchone()
if row:
try:
prof_lang = row["language"]
except (KeyError, IndexError):
prof_lang = None
if prof_lang and prof_lang != "Auto":
return prof_lang
return None
def _build_synth(default_voice: str | None, language: str | None = None) -> dict:
"""Describe how to synthesize for the active TTS engine.
Returns a dict with ``mode``, ``resolve`` (voice-id resolved refs, cached
per id) and ``engine_id``. For OmniVoice it also carries the async
``get_model``; other engines carry a ready ``synth`` + ``sample_rate``.
:func:`_prepare_synth` turns this into a uniform ``(synth, sr, resolve,
engine_id)`` once the (async) model is in hand.
``language`` (already resolved by :func:`_resolve_default_language`) is
threaded into every chunk's ``generate`` so a non-English clone stays in its
language instead of re-autodetecting per chunk (#505 B2). ``None`` keeps the
engine's autodetect behavior unchanged.
"""
from services.tts_backend import OmniVoiceBackend, active_backend_id, get_backend_class
cache: dict = {}
def resolve(voice_id):
key = voice_id or default_voice
if key not in cache:
cache[key] = _resolve_voice(key)
return cache[key]
engine_id = active_backend_id()
cls = get_backend_class(engine_id)
if cls is OmniVoiceBackend:
from services.model_manager import get_model
return {"mode": "omnivoice", "resolve": resolve,
"engine_id": engine_id, "get_model": get_model, "language": language}
backend = cls()
def synth(text, voice_id, speed=None):
v = resolve(voice_id)
return backend.generate(
text, language=language, ref_audio=v["ref_audio"],
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
speed=float(speed) if speed else 1.0,
)
return {"mode": "generic", "resolve": resolve, "engine_id": engine_id,
"synth": synth, "sample_rate": backend.sample_rate}
async def _prepare_synth(default_voice: str | None, language: str | None = None):
"""Resolve :func:`_build_synth` into ``(synth, sample_rate, resolve,
engine_id)`` awaiting the OmniVoice model load when needed. Shared by the
full job and the per-chapter preview. ``language`` is threaded into every
chunk so a non-English clone holds its language (#505 B2)."""
info = _build_synth(default_voice, language=language)
resolve, engine_id = info["resolve"], info["engine_id"]
if info["mode"] == "omnivoice":
lang = info["language"]
model = await info["get_model"]()
sr = getattr(model, "sampling_rate", 24000)
def synth(text, voice_id, speed=None):
v = resolve(voice_id)
return model.generate(
text=text, language=lang, ref_audio=v["ref_audio"],
ref_text=v["ref_text"], instruct=v["instruct"], duration=None,
speed=float(speed) if speed else 1.0,
)[0]
return synth, sr, resolve, engine_id
return info["synth"], info["sample_rate"], resolve, engine_id
def _render_chapter_cached(chapter, synth, sr, engine_id, resolve, cache_dir, lexicon=None,
language=None):
"""Render one chapter, content-addressed so a re-run reuses it (resume).
Returns ``(wav_path, duration_s, was_cached, seg_stats)``. Two cache
layers:
* Outer the WAV at ``cache_dir/<key>.wav`` where ``key`` is
:func:`chapter_cache_key` over the chapter's spans + sample rate +
engine + each voice's resolved signature (+ the lexicon, so a lexicon
edit re-renders). A fully-unchanged chapter hits here and never touches
segment files; the key derivation is unchanged, so chapter caches
written by released versions keep hitting. ``seg_stats`` is ``None``.
* Inner on a chapter miss, each spoken span goes through the
:class:`services.longform_render.SegmentCache` under
``cache_dir/segments``: cached segments load from disk, only the
edited/missing ones synthesize, and each fresh segment persists the
moment it renders (an interrupted chapter resumes from them).
``seg_stats`` is ``{"total": spoken_spans, "cached": reused}``.
Span text is normalized (``services.text_normalization``) up front BEFORE
either cache key and BEFORE ``synthesize_chapter``'s lexicon pass, so the
per-project dictionary operates on normalized text and toggling / changing
normalization output naturally invalidates cached chapters and segments.
Runs in the GPU-pool executor.
"""
import json
import wave
from services.audio_io import atomic_save_wav
from services.audiobook import Span
from services.longform_render import SegmentCache, chapter_cache_key
from services.pronunciation import normalize_lexicon
from services.text_normalization import normalize_for_tts
spans = [Span(voice_id=s.voice_id, text=normalize_for_tts(s.text, language),
pause_ms_after=s.pause_ms_after, speed=getattr(s, "speed", None))
for s in chapter.spans]
spans_tuples = [(s.voice_id, s.text, s.pause_ms_after, getattr(s, "speed", None))
for s in spans]
voice_sigs: dict = {}
for s in spans:
k = s.voice_id or ""
if k not in voice_sigs:
v = resolve(s.voice_id)
voice_sigs[k] = f"{v.get('ref_audio')}|{v.get('ref_text')}|{v.get('instruct')}|{v.get('seed')}"
sig: dict = dict(voice_sigs)
lex_sig = ""
if lexicon:
# Fold the lexicon into the cache key so editing pronunciations
# invalidates cached chapters (reserved key can't collide with a voice id).
lex_sig = json.dumps(normalize_lexicon(lexicon), sort_keys=True)
sig["\x00lexicon"] = lex_sig
key = chapter_cache_key(spans_tuples, sample_rate=sr, engine_id=engine_id, voice_sig=sig)
wav_path = os.path.join(cache_dir, f"{key}.wav")
if os.path.exists(wav_path):
try:
with wave.open(wav_path, "rb") as w:
dur = w.getnframes() / float(w.getframerate() or sr)
return wav_path, dur, True, None
except Exception:
pass # corrupt cache entry — fall through and re-render
seg_cache = SegmentCache(cache_dir, sample_rate=sr, engine_id=engine_id,
voice_sig=voice_sigs, extra_sig=lex_sig)
audio, dur = synthesize_chapter(spans, synth, sr, lexicon=lexicon,
segment_cache=seg_cache)
atomic_save_wav(wav_path, audio, sr)
return wav_path, dur, False, {"total": seg_cache.hits + seg_cache.misses,
"cached": seg_cache.hits}
class AudiobookPreviewRequest(BaseModel):
text: str
chapter_index: int = 0
default_voice: str | None = None
language: str | None = None # None/"Auto" → profile language, else autodetect
lexicon: dict | None = None
@router.post("/audiobook/preview")
async def audiobook_preview(req: AudiobookPreviewRequest) -> dict:
"""Render a single chapter so the user can audition it before the full run.
Reuses the same content-addressed cache as the job, so a preview warms the
cache (the later full render reuses it) and a re-preview is instant.
"""
from core.config import OUTPUTS_DIR
from services.model_manager import _gpu_pool
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
if not plan.chapters:
raise HTTPException(status_code=400, detail="no chapters parsed from the script")
n = len(plan.chapters)
if not (0 <= req.chapter_index < n):
raise HTTPException(status_code=400, detail=f"chapter_index out of range (0..{n - 1})")
chapter = plan.chapters[req.chapter_index]
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache") # shared with _render_longform_sse
os.makedirs(cache_dir, exist_ok=True)
resolved_lang = _resolve_default_language(req.language, req.default_voice)
synth, sr, resolve, engine_id = await _prepare_synth(
req.default_voice,
language=resolved_lang,
)
loop = asyncio.get_running_loop()
wav_path, dur, was_cached, _seg_stats = await loop.run_in_executor(
_gpu_pool, _render_chapter_cached, chapter, synth, sr, engine_id, resolve, cache_dir,
req.lexicon, resolved_lang,
)
return {
"output": os.path.relpath(wav_path, OUTPUTS_DIR), # served via /audio
"duration_s": round(dur, 2),
"cached": was_cached,
"title": chapter.title,
}
async def _render_longform_sse(
plan,
*,
default_voice: str | None,
language: str | None = None,
fmt: str = "m4b",
bitrate: str = "128k",
loudness: str | None = None,
cover_path: str | None = None,
metadata: dict | None = None,
lexicon: dict | None = None,
job_type: str = "audiobook",
job_id: str | None = None,
resume: bool = False,
):
"""Shared chapterized-render SSE generator for Audiobook *and* Stories.
Takes a ready ``plan`` (``.chapters`` ``.title`` + ``.spans``) Audiobook
parses it from a script, Stories compiles it from cast/lines and renders
each chapter (content-addressed cache resume), isolating per-chapter
failures, then muxes the successful chapters into a tagged file. This is the
convergence point: one renderer, two front doors.
"""
from core.config import OUTPUTS_DIR
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
from services.model_manager import _gpu_pool
# Resume reuses the original job_id (continuing the same job row + cached
# chapters); a fresh render generates a new one. The id may arrive from the
# /resume/{job_id} path param, so strip it to a safe token (no path
# separators, no CR/LF) before it ever reaches a filesystem path or a log
# line — CodeQL py/path-injection + py/log-injection. Empty after the strip
# → a fresh id.
job_id = re.sub(r"[^A-Za-z0-9_-]", "", job_id or "")[:64] or uuid.uuid4().hex[:16]
try:
from core import job_store
if not resume:
job_store.create(job_id, type=job_type)
job_store.mark_running(job_id)
except Exception:
job_store = None # job history is best-effort; never block synthesis
# Persist a durable resume manifest (plan + params) so an interrupted render
# can be resumed later even without the original script. Best-effort.
try:
title = (metadata or {}).get("title") or (plan.chapters[0].title if plan.chapters else "")
longform_resume.write_manifest(longform_resume.build_manifest(
job_id=job_id, job_type=job_type, title=title,
plan_chapters=[
{"title": c.title, "spans": [s.to_dict() for s in c.spans]}
for c in plan.chapters
],
params={
"default_voice": default_voice, "language": language,
"fmt": fmt, "bitrate": bitrate,
"loudness": loudness, "cover_path": cover_path,
"metadata": metadata, "lexicon": lexicon,
},
))
except Exception: # resume durability is an enhancement; never block the render
logger.debug("[%s] resume manifest write skipped", job_id, exc_info=True)
def _emit(payload: dict) -> str:
if job_store is not None:
try:
job_store.append_event(job_id, json.dumps(payload))
except Exception:
pass # best-effort job history; never block the stream
return f"data: {json.dumps(payload)}\n\n"
if not plan.chapters:
yield _emit({"type": "error", "error": "nothing to render (no chapters)"})
return
ffmpeg = find_ffmpeg()
if not ffmpeg:
yield _emit({"type": "error", "error": "ffmpeg not available; the output needs it"})
return
# Confined work dir (job_id is already token-sanitized above; work_dir adds
# the basename + realpath barrier so CodeQL sees a clean path).
work = longform_resume.work_dir(job_type, job_id)
if work is None:
yield _emit({"type": "error", "error": "invalid job id"})
return
os.makedirs(work, exist_ok=True)
# Chapter WAVs are content-addressed in a shared cache so a re-run (after a
# failure or interruption) reuses what already rendered — only the
# missing/changed chapters synthesize again (resume). Shared across both
# front doors: an identical chapter renders once.
cache_dir = os.path.join(OUTPUTS_DIR, "longform_cache")
os.makedirs(cache_dir, exist_ok=True)
prune_cache_dir(cache_dir) # bound disk before this job adds its chapters
loop = asyncio.get_running_loop()
try:
resolved_lang = _resolve_default_language(language, default_voice)
synth, sr, resolve, engine_id = await _prepare_synth(
default_voice, language=resolved_lang
)
total = len(plan.chapters)
chapter_files: list[str] = []
chapters_meta: list[tuple[str, int]] = []
cached_n = 0
failed: list[int] = []
yield _emit({"type": "started", "job_id": job_id, "chapters": total})
for i, chapter in enumerate(plan.chapters):
try:
wav_path, dur, was_cached, seg_stats = await loop.run_in_executor(
_gpu_pool, _render_chapter_cached,
chapter, synth, sr, engine_id, resolve, cache_dir, lexicon,
resolved_lang,
)
except Exception: # isolate a bad chapter — keep going
logger.warning("[%s] chapter %d (%s) failed to render",
job_id, i, chapter.title, exc_info=True)
failed.append(i)
yield _emit({"type": "chapter_error", "index": i, "total": total,
"title": chapter.title, "error": "chapter failed to render"})
continue
chapter_files.append(wav_path)
chapters_meta.append((chapter.title, int(round(dur * 1000))))
cached_n += 1 if was_cached else 0
ev = {"type": "chapter", "index": i, "total": total,
"title": chapter.title, "duration_s": round(dur, 2),
"cached": was_cached}
if seg_stats is not None:
# Additive fields (old clients ignore them): segment-level
# reuse inside a re-rendered chapter.
ev["segments"] = seg_stats["total"]
ev["cached_segments"] = seg_stats["cached"]
yield _emit(ev)
if not chapter_files:
yield _emit({"type": "error", "error": "all chapters failed to render"})
return
yield _emit({"type": "assembling"})
meta_path = os.path.join(work, "chapters.ffmeta")
with open(meta_path, "w", encoding="utf-8") as f:
f.write(build_ffmetadata(chapters_meta, global_meta=metadata))
concat_path = os.path.join(work, "concat.txt")
with open(concat_path, "w", encoding="utf-8") as f:
f.write(build_concat_list(chapter_files))
ext = "mp3" if (fmt or "").lower() == "mp3" else "m4b"
out_name = f"{job_type}_{job_id}.{ext}"
out_path = os.path.join(OUTPUTS_DIR, out_name)
# Two-pass loudness master (#28): for a known preset, measure the
# concatenated program first, then feed the measured values back into the
# single mux encode. `measured is None` (skip OR any failure) → the mux
# falls back to single-pass. Gated identically to the pure builders
# (.lower(), no strip), so off/None/unknown/whitespace skip cleanly.
measured = None
norm = (loudness or "").lower()
if norm in LOUDNESS_PRESETS:
yield _emit({"type": "mastering", "preset": norm})
from services.loudness import measure_loudness
measured = await measure_loudness(ffmpeg, concat_path, norm, job_id=job_id)
await run_ffmpeg(
build_render_cmd(
ffmpeg, concat_path, meta_path, out_path,
fmt=ext, bitrate=bitrate, cover_path=_safe_cover_path(cover_path),
loudness=loudness, measured=measured,
),
job_id=job_id,
)
if job_store is not None:
try:
job_store.mark_done(job_id)
except Exception:
pass # best-effort job history
# The render finished — drop the resume manifest so this job is no longer
# offered for resume.
longform_resume.clear_manifest(job_type, job_id)
total_s = sum(d for _, d in chapters_meta) / 1000.0
done = {"type": "done", "output": out_name,
"chapters": len(chapter_files), "duration_s": round(total_s, 2),
"cached_chapters": cached_n, "failed_chapters": failed}
# Loudness verdict only when a preset was requested — off/None paths keep
# the exact legacy `done` shape (additive, old clients unaffected).
if norm in LOUDNESS_PRESETS:
p = LOUDNESS_PRESETS[norm]
done["loudness"] = {
"preset": norm, "target_i": p.i, "target_tp": p.tp,
"two_pass": measured is not None,
"measured_i": measured.input_i if measured else None,
}
yield _emit(done)
except Exception as e: # surface, don't 500 the stream
logger.exception("[%s] longform render failed", job_id)
if job_store is not None:
try:
job_store.mark_failed(job_id, str(e))
except Exception:
pass # best-effort job history
# Generic message only — don't leak the stack/exception text to the client.
yield _emit({"type": "error", "error": "render failed (see backend log)"})
@router.post("/audiobook")
async def audiobook_synthesize(req: AudiobookRequest):
"""Synthesize a chapterized audiobook from a script, streaming SSE progress."""
plan = parse_audiobook_script(req.text, default_voice=req.default_voice)
return StreamingResponse(
_render_longform_sse(
plan, default_voice=req.default_voice, language=req.language,
fmt=req.format, bitrate=req.bitrate,
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
lexicon=req.lexicon, job_type="audiobook",
),
media_type="text/event-stream",
)
# ── Shared longform render: Stories (and any future front door) post a plan ──
class LongformSpan(BaseModel):
voice_id: str | None = None
text: str
pause_ms_after: int = 0
speed: float | None = None
class LongformChapter(BaseModel):
title: str = ""
spans: list[LongformSpan] = []
class LongformRenderRequest(BaseModel):
chapters: list[LongformChapter] = []
default_voice: str | None = None
language: str | None = None # None/"Auto" → profile language, else autodetect (#505)
bitrate: str = "128k"
format: str = "m4b"
loudness: str | None = None
cover_path: str | None = None
metadata: dict | None = None
lexicon: dict | None = None
@router.post("/longform/render")
async def longform_render(req: LongformRenderRequest):
"""Render a pre-built chapter/span plan (the Stories Editor's compiled
cast+lines) through the shared chapterized renderer same resume, loudness,
cover, metadata, and output formats as the Audiobook job."""
from services.audiobook import AudiobookPlan, Chapter, Span
if len(req.chapters) > _MAX_CHAPTERS:
raise HTTPException(status_code=422, detail=f"too many chapters (max {_MAX_CHAPTERS})")
chapters = []
for i, c in enumerate(req.chapters):
# Keep a span if it has text to speak OR a pause to render (pause-only
# spans carry inter-line silence with empty text).
spans = [Span(voice_id=s.voice_id, text=(s.text or "").strip(),
pause_ms_after=max(0, int(s.pause_ms_after)), speed=s.speed)
for s in c.spans if ((s.text and s.text.strip()) or s.pause_ms_after > 0)]
if spans:
chapters.append(Chapter(title=c.title or f"Chapter {i + 1}", spans=spans))
plan = AudiobookPlan(chapters=chapters)
return StreamingResponse(
_render_longform_sse(
plan, default_voice=req.default_voice, language=req.language,
fmt=req.format, bitrate=req.bitrate,
loudness=req.loudness, cover_path=req.cover_path, metadata=req.metadata,
lexicon=req.lexicon, job_type="story",
),
media_type="text/event-stream",
)
# ── Durable resume: interrupted longform renders ────────────────────────────
def _chapters_done(job_id: str) -> int:
"""Count chapters that finished rendering, from the job's persisted events.
Best-effort (0 if unavailable) used only to show resume progress."""
try:
from core import job_store
n = 0
for ev in job_store.events_since(job_id, 0, limit=100_000):
try:
if json.loads(ev["payload"]).get("type") == "chapter":
n += 1
except (ValueError, KeyError, TypeError):
continue
return n
except Exception:
return 0
@router.get("/audiobook/jobs")
def list_resumable_jobs() -> dict:
"""List interrupted longform renders that can be resumed — a work dir that
still holds a resume manifest (a job left mid-render by a crash/quit). The
ids come from scanning the filesystem, so the UI can offer one-click resume."""
from core import job_store
out = []
for e in longform_resume.scan_resumable():
jid = e["job_id"]
manifest = longform_resume.load_manifest_file(e["manifest_path"]) or {}
job = job_store.get(jid) or {}
out.append({
"job_id": jid,
"type": e["job_type"],
"status": job.get("status", "interrupted"),
"title": manifest.get("title", ""),
"total_chapters": manifest.get("total_chapters", 0),
"chapters_done": _chapters_done(jid),
"created_at": job.get("created_at"),
})
return {"jobs": out}
@router.post("/audiobook/resume/{job_id}")
async def resume_longform(job_id: str):
"""Resume an interrupted longform render from its persisted manifest. The
already-rendered chapters are content-addressed in the shared cache, so they
return instantly only the unrendered chapters synthesize again. Streams the
same SSE event shape as the original render, under the original job_id."""
from services.audiobook import AudiobookPlan, Chapter, Span
# Find the requested job among the trusted filesystem scan (every path there
# is os.listdir-sourced, never request input) and read its manifest via the
# scan's own trusted path — the request job_id is used ONLY to *select* an
# entry, never to build a path. No request-controlled value reaches a file
# operation (CodeQL py/path-injection-safe).
entry = next((e for e in longform_resume.scan_resumable()
if e["job_id"] == job_id), None)
if entry is None:
raise HTTPException(status_code=404, detail="No resumable job for that id")
manifest = longform_resume.load_manifest_file(entry["manifest_path"])
if not manifest:
raise HTTPException(status_code=404, detail="No resume manifest for that job")
chapters = [
Chapter(title=c.get("title", ""),
spans=[Span(**s) for s in c.get("spans", [])])
for c in manifest["plan"]
]
plan = AudiobookPlan(chapters=chapters)
p = manifest.get("params", {})
# Retire the interrupted job's manifest (trusted scan path) so it stops
# showing as resumable once we've kicked off the fresh-id resume.
longform_resume.discard_manifest_file(entry["manifest_path"])
# Resume under a FRESH job id (job_id=None → a server uuid in the renderer).
# The chapter cache is content-addressed (keyed by chapter content, not the
# job id), so the already-rendered chapters still hit instantly — only the
# unrendered ones synthesize. Using a fresh id means the request's job_id
# never names a work dir / output file (defence-in-depth path-injection).
return StreamingResponse(
_render_longform_sse(
plan, default_voice=p.get("default_voice"), language=p.get("language"),
fmt=p.get("fmt", "m4b"), bitrate=p.get("bitrate", "128k"),
loudness=p.get("loudness"), cover_path=p.get("cover_path"),
metadata=p.get("metadata"), lexicon=p.get("lexicon"),
job_type=entry["job_type"],
),
media_type="text/event-stream",
)
+13 -36
View File
@@ -142,7 +142,7 @@ async def _run_batch_pipeline(job_id: str, job: dict):
_set_progress(job, "transcribe", 0)
from services.asr_backend import get_active_asr_backend
from services.model_manager import _gpu_pool, _cpu_pool, run_on_gpu_pool_guarded
from services.model_manager import _gpu_pool, _cpu_pool
from services.segmentation import (
segment_transcript, assign_speakers_heuristic,
)
@@ -162,12 +162,7 @@ async def _run_batch_pipeline(job_id: str, job: dict):
pass
return segments, detected_lang
# Bound the batch transcribe (#730) so a wedged whisperx/CTranslate2 call
# can't hold its GPU-pool worker forever and starve the rest of the backend
# ("can't reach backend"); run_transcribe_guarded also resets the pool on
# timeout to restore capacity.
from services.asr_backend import run_transcribe_guarded
segments, source_lang = await run_transcribe_guarded(_gpu_pool, _transcribe, what="Batch")
segments, source_lang = await loop.run_in_executor(_gpu_pool, _transcribe)
source_lang = (source_lang or "en").split("_")[0][:2].lower()
job["segments"] = segments
job["source_lang"] = source_lang
@@ -179,21 +174,6 @@ async def _run_batch_pipeline(job_id: str, job: dict):
job["status"] = "failed"
return
# ── Engine resolution (issue #312 class) ────────────────────────────
# Batch used to hardcode OmniVoice via get_model() regardless of the
# engine selected in Settings → Engines. require_cloning only when a
# specific voice is pinned (job["voice_id"]) — an unpinned job is fine on
# any active engine. Resolved ONCE for the whole job (every language
# below shares the same active engine); an uncaught ValueError here
# propagates to _worker()'s existing except-Exception handling, which
# already records a structured job failure via core.failure.build_failure.
from services.tts_backend import resolve_generation_backend
backend = await resolve_generation_backend(
require_cloning=bool(job.get("voice_id")),
cloning_purpose="this batch job's pinned voice",
)
sr = backend.sample_rate
# ── 3. Translate + Generate per language ───────────────────────────
total_langs = len(langs)
outputs = {}
@@ -258,10 +238,13 @@ async def _run_batch_pipeline(job_id: str, job: dict):
total_segments=len(translated_segments),
)
from services.model_manager import get_model
from services.audio_dsp import apply_mastering, normalize_audio
from services.audio_io import atomic_save_wav
import torch
_model = await get_model()
sr = _model.sampling_rate
total_samples = int(duration * sr)
full_audio = torch.zeros(1, total_samples)
total_segs = len(translated_segments)
@@ -287,13 +270,6 @@ async def _run_batch_pipeline(job_id: str, job: dict):
continue
def _gen(text=seg_text, lang=target_lang, dur=seg_duration):
# Normalize once at the segment's text→engine choke point —
# the same pre-pass as /generate and dub_generate's _gen.
# `lang` is the job's target language code. Pref-gated,
# idempotent, never raises.
from services.text_normalization import normalize_for_tts
text = normalize_for_tts(text, lang)
ref_audio = None
ref_text = None
@@ -314,24 +290,25 @@ async def _run_batch_pipeline(job_id: str, job: dict):
ref_text = row.get("ref_text")
try:
audio_out = backend.generate(
audios = _model.generate(
text=text, language=lang,
ref_audio=ref_audio, ref_text=ref_text,
duration=dur, num_step=16,
guidance_scale=2.0, speed=1.0,
denoise=True, postprocess_output=True,
)
if not getattr(backend, "applies_own_mastering", False):
audio_out = apply_mastering(audio_out, sample_rate=sr)
return normalize_audio(audio_out, target_dBFS=-2.0)
audio_out = audios[0]
mastered = apply_mastering(
audio_out,
sample_rate=sr,
)
return normalize_audio(mastered, target_dBFS=-2.0)
except Exception as e:
logger.warning("TTS failed for seg %d (lang=%s): %s", i, lang, e)
return torch.zeros(1, int(dur * sr))
try:
# Bounded + pool-reset on hang so a wedged batch segment can't
# starve the GPU pool and brick the backend (#730 class).
audio_tensor = await run_on_gpu_pool_guarded(_gen, what="Batch generate")
audio_tensor = await loop.run_in_executor(_gpu_pool, _gen)
# Fit to slot
target_samples_seg = int(seg_duration * sr)
+5 -62
View File
@@ -18,26 +18,19 @@ import os
import tempfile
import time
from fastapi import APIRouter, File, Form, HTTPException, UploadFile
from fastapi import APIRouter, File, Form, UploadFile
from typing import Optional
router = APIRouter()
logger = logging.getLogger("omnivoice.capture")
def _truthy(value: Optional[str]) -> bool:
"""Parse a multipart form flag. Treats '1'/'true'/'yes'/'on'/'auto'
(any case) as on; everything else including None as off."""
return (value or "").strip().lower() in {"1", "true", "yes", "on", "auto"}
@router.post("/transcribe")
async def transcribe_audio(
audio: UploadFile = File(...),
language: Optional[str] = Form(None),
model: Optional[str] = Form(None),
mode: Optional[str] = Form(None),
refine: Optional[str] = Form(None),
):
"""Transcribe an audio file to text.
@@ -47,19 +40,10 @@ async def transcribe_audio(
model: Whisper model size (legacy; ignored in dual-mode architecture).
mode: 'fast' (default) uses MLX Turbo for speed; 'accurate' uses
WhisperX with forced alignment for word-level timing.
refine: Opt-in local-LLM cleanup of the final text (disfluencies,
self-corrections, punctuation) same pipeline the live
dictation socket uses. Off by default so MCP/CLI callers don't
pay LLM latency unless they ask; honours the user's
Settings Dictation-refinement config and silently passes
through when no LLM backend is configured. The raw ``text``
is always returned; ``refined_text`` is added only when the
LLM actually changed something.
Returns:
{
"text": "full transcription",
"refined_text": "cleaned text", # only when refine=true changed it
"segments": [ {"start": 0.0, "end": 1.5, "text": "..."}, ... ],
"language": "en",
"duration_s": 4.2,
@@ -96,17 +80,9 @@ async def transcribe_audio(
return result, backend.id
from services.model_manager import _gpu_pool
from services.asr_backend import ASRTimeoutError, run_transcribe_guarded
loop = asyncio.get_running_loop()
t0 = time.perf_counter()
try:
result, engine_id = await run_transcribe_guarded(
_gpu_pool, _run, what="Dictation",
)
except ASRTimeoutError as e:
# Backend is alive — ASR couldn't finish. 504 with guidance, not a
# silent hang the UI reads as "can't reach the local backend".
logger.warning("Capture transcription timed out: %s", e)
raise HTTPException(status_code=504, detail=str(e))
result, engine_id = await loop.run_in_executor(_gpu_pool, _run)
elapsed = round(time.perf_counter() - t0, 2)
# Normalize result shape
@@ -115,20 +91,6 @@ async def transcribe_audio(
if not full_text and segments:
full_text = " ".join(s.get("text", "") for s in segments).strip()
# Wave 1.1: strip Whisper hallucination loops from the final text.
# Segments keep the raw recognition so their timings stay truthful.
from services.refinement import collapse_repetitive_artifacts
full_text = collapse_repetitive_artifacts(full_text)
# Cross-transport parity: deterministically polish the final text
# (leading capital + terminal punctuation) exactly like the live
# dictation socket (capture_ws) does, so the widget's POST fallback and
# MCP/CLI callers get the same typed-looking result the WS returns —
# not the raw "...test" the REST path used to leak. Segments stay raw
# (their timings/verbatim recognition are the contract).
from services.text_polish import polish_text
full_text = polish_text(full_text)
# Calculate audio duration from segments if available
duration = 0.0
if segments:
@@ -136,28 +98,12 @@ async def transcribe_audio(
detected_lang = result.get("language", language or "unknown")
# Opt-in Wave 2.1 refinement, mirroring the live-dictation socket
# (capture_ws). Off-thread (it's a network call, not GPU); never
# raises — maybe_refine swallows failures and a missing LLM into a
# None pass-through, so the raw text always stands.
refined_text = None
if _truthy(refine) and full_text:
from services.refinement import maybe_refine
refined = await asyncio.to_thread(maybe_refine, full_text)
if refined:
# Polish the refined text too, so both surfaced strings read as
# typed text (mirrors the raw-vs-refined contract of the WS).
refined = polish_text(refined)
if refined != full_text:
refined_text = refined
logger.info(
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s, refined=%s",
"Capture transcription done: engine=%s, elapsed=%.2fs, duration=%.1fs, mode=%s",
engine_id, elapsed, duration, "accurate" if use_accurate else "fast",
refined_text is not None,
)
response = {
return {
"text": full_text,
"segments": [
{
@@ -172,9 +118,6 @@ async def transcribe_audio(
"transcription_time_s": elapsed,
"engine": engine_id,
}
if refined_text is not None:
response["refined_text"] = refined_text
return response
finally:
try:
os.unlink(tmp.name)
+16 -587
View File
@@ -8,26 +8,12 @@ live dictation feedback.
Protocol:
Client sends binary audio frames (16-bit PCM or WebM/Opus blobs)
Server sends JSON messages:
Opt-in AEC mode (``?aec=1[&sr=16000]``, parity Action 8b): for dictating
while the app plays audio. Frames must be raw int16 mono PCM, each tagged
with a 1-byte prefix 0x00 = microphone, 0x01 = playback reference. The
server runs an NLMS echo canceller, cleaning the mic against the reference
before transcription. Without the param the protocol is unchanged.
{"type": "partial", "text": "Hello wor..."} interim result
{"type": "final", "text": "Hello world.", committed result
"segments": [...], "language": "en",
"duration_s": 4.2, "transcription_time_s": 0.8,
"engine": "mlx-whisper"}
{"type": "status", "stage": "downloading"|"loading"|"ready"}
model cold-start
{"type": "error", "message": "...", "kind": "...",
"detail": "..."} error ("detail"
kept for legacy)
Every ``final`` text is normalised by services.text_polish (leading
capital for Latin scripts, terminal punctuation, single-spaced) so the
pasted result reads like typed text. Partials are raw.
{"type": "error", "detail": "..."} error
"""
from __future__ import annotations
@@ -39,8 +25,7 @@ import time
from fastapi import APIRouter, WebSocket, WebSocketDisconnect
from api.dependencies import _LOOPBACK_HOSTS, ws_remote_authorized
from services.text_polish import polish_text
from api.dependencies import _LOOPBACK_HOSTS
router = APIRouter()
logger = logging.getLogger("omnivoice.capture_ws")
@@ -59,80 +44,6 @@ MIN_BUFFER_BYTES = 64000 # ~2s of 16-bit mono 16kHz — needs enough WebM frame
# to transcribe whatever the user recorded, even short utterances.
MIN_FINAL_BUFFER_BYTES = 4000 # ~125ms of 16-bit mono 16kHz
# ── Dictate-over-playback AEC (parity Action 8b, opt-in) ──────────────────
# Activated by the ``?aec=1`` query param. When OFF (the default), the
# protocol and behaviour are byte-for-byte unchanged. When ON, the client
# streams raw int16 mono PCM frames tagged with a 1-byte type prefix so the
# server can tell mic audio from the playback reference it must cancel:
_AEC_NEAR = 0x00 # microphone frame (clean it, then buffer for ASR)
_AEC_FAR = 0x01 # playback reference frame (feed the echo model only)
def _demux_aec_frame(data: bytes) -> tuple[str, bytes]:
"""Split a prefixed AEC binary frame into ``(kind, pcm)``.
``kind`` is ``"near"`` (mic) or ``"far"`` (playback reference). An empty
or prefix-only frame yields an empty payload. Unknown prefixes are treated
as ``"near"`` so a malformed tag degrades to plain dictation rather than
dropping audio.
"""
if not data:
return "near", b""
kind = "far" if data[0] == _AEC_FAR else "near"
return kind, data[1:]
def _pcm16_to_wav(pcm: bytes, sample_rate: int) -> str | None:
"""Write raw int16 mono PCM to a temp WAV via stdlib ``wave`` (no ffmpeg).
Used on the AEC path, where frames are already decoded PCM the cleaned
samples have no container, so the ffmpeg-sniffing ``_chunks_to_wav`` would
misdetect them. Returns the temp path, or ``None`` for a too-short buffer.
"""
if not pcm or len(pcm) < 100:
return None
import wave
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".wav")
tmp.close()
try:
with wave.open(tmp.name, "wb") as wf:
wf.setnchannels(1)
wf.setsampwidth(2) # int16
wf.setframerate(sample_rate)
wf.writeframes(pcm)
return tmp.name
except Exception as e:
logger.debug("PCM->WAV failed: %s", e)
try:
os.unlink(tmp.name)
except OSError:
pass
return None
def _select_sherpa_spec(websocket: WebSocket):
"""Resolve the sherpa dictation model for this WS session, or None.
A ``?model=<id>`` query param wins (the frontend can pin a model per
session); otherwise the persisted ``dictation.model_id`` pref is used (only
when dictation is enabled). Returns the :class:`SherpaModelSpec` or None
(None the legacy Whisper/WebM path runs unchanged).
"""
try:
from services import sherpa_dictation as sd
except Exception:
return None
requested = websocket.query_params.get("model")
if requested:
return sd.get_spec(requested) # explicit selection (may be None if bad)
# Fall back to the persisted dictation pref.
try:
from services.asr_backend import dictation_model_id
mid = dictation_model_id()
except Exception:
mid = None
return sd.get_spec(mid) if mid else None
@router.websocket("/ws/transcribe")
async def ws_transcribe(websocket: WebSocket):
@@ -142,51 +53,13 @@ async def ws_transcribe(websocket: WebSocket):
# WebSocket dependency injection differs across FastAPI versions, so we
# inline the check before accept(). Without it, any local process could
# stream the user's microphone over this endpoint.
# Wave 2.3 (remote backend): a non-loopback client that presents the
# OMNIVOICE_API_KEY bearer is the thin-client dictation case — the mic
# lives on the user's machine, the GPU here — and is allowed through.
host = websocket.client.host if websocket.client else None
if host not in _LOOPBACK_HOSTS and not ws_remote_authorized(websocket):
if host not in _LOOPBACK_HOSTS:
await websocket.close(code=1008, reason="loopback origin required")
return
await websocket.accept()
# Live-dictation engine selection. When a sherpa-onnx model is selected
# (via ?model= or the dictation.model_id pref) AND sherpa is installed,
# run the dedicated low-latency handler. Otherwise fall through to the
# legacy Whisper/WebM path, byte-for-byte unchanged.
spec = _select_sherpa_spec(websocket)
if spec is not None:
from services.asr_backend import SherpaDictationBackend
ok, _reason = SherpaDictationBackend.is_available()
if ok:
if spec.streaming:
await _run_sherpa_streaming(websocket, spec)
else:
await _run_sherpa_offline(websocket, spec)
return
# sherpa not installed → fall through to the legacy path so the user
# still gets dictation (just not live partials).
logger.info("sherpa dictation selected but unavailable — legacy path")
# Opt-in dictate-over-playback AEC (parity Action 8b). Default OFF →
# identical legacy behaviour. When on, frames are 1-byte-tagged raw PCM
# and the cleaned mic stream is muxed via stdlib wave (not ffmpeg).
aec = None
pcm_sr: int | None = None
if websocket.query_params.get("aec") in ("1", "true", "on"):
try:
pcm_sr = int(websocket.query_params.get("sr", "16000"))
from services.aec import NlmsEchoCanceller
aec = NlmsEchoCanceller(sample_rate=pcm_sr)
logger.info("AEC enabled for dictation session (sr=%d)", pcm_sr)
except Exception as e:
# Bad sr or import failure → fall back to plain dictation.
logger.warning("AEC requested but disabled: %s", e)
aec = None
pcm_sr = None
audio_chunks: list[bytes] = []
total_bytes = 0
last_audio_time = time.monotonic()
@@ -224,16 +97,6 @@ async def ws_transcribe(websocket: WebSocket):
# Empty binary frame also acts as EOF — connection stays open.
running = False
break
if aec is not None:
# Tagged PCM: route the playback reference into the echo
# model and clean the mic before it reaches the buffer.
kind, payload = _demux_aec_frame(data)
if kind == "far":
aec.push_far_end(payload)
continue
if not payload:
continue
data = aec.process_near_end(payload)
audio_chunks.append(data)
total_bytes += len(data)
last_audio_time = time.monotonic()
@@ -280,7 +143,7 @@ async def ws_transcribe(websocket: WebSocket):
# Transcribe current buffer
try:
text = await _transcribe_buffer(audio_chunks[:], pcm_sr=pcm_sr)
text = await _transcribe_buffer(audio_chunks[:])
if text and text != partial_text:
partial_text = text
await _safe_send({
@@ -310,30 +173,12 @@ async def ws_transcribe(websocket: WebSocket):
# Final transcription on complete buffer — skip if client already gone.
if total_bytes > MIN_FINAL_BUFFER_BYTES:
try:
result = await _transcribe_buffer_full(audio_chunks, pcm_sr=pcm_sr)
# Dictation v2: deterministic polish so the pasted final reads
# like typed text (leading capital, terminal punctuation).
result["text"] = polish_text(result.get("text", ""))
# Wave 2.1: optional local-LLM refinement of the final text.
# HARD-BOUNDED (maybe_refine_async, ~4s OMNIVOICE_REFINE_TIMEOUT_S):
# a slow/dead LLM can never delay this `final` beyond the budget —
# it falls back to the unrefined (but polished) text. Best-effort:
# never let refinement turn a good final into an error. The raw
# text always ships too — clients paste refined_text ?? text.
if result.get("text"):
try:
from services.refinement import maybe_refine_async
refined = await maybe_refine_async(result["text"])
if refined and refined != result["text"]:
result["refined_text"] = refined
except Exception as e: # noqa: BLE001
logger.debug("Dictation refinement skipped: %s", e)
result = await _transcribe_buffer_full(audio_chunks)
if not await _safe_send({"type": "final", **result}):
logger.debug("Skipped final send — client already disconnected")
except Exception as e:
logger.error("Final transcription failed: %s", e)
await _safe_send({"type": "error", "message": str(e),
"kind": "transcribe", "detail": str(e)})
await _safe_send({"type": "error", "detail": str(e)})
else:
await _safe_send({
"type": "final",
@@ -352,433 +197,24 @@ async def ws_transcribe(websocket: WebSocket):
pass
# ── sherpa-onnx live dictation handlers ─────────────────────────────────────
#
# Both handlers read raw int16 mono PCM frames (reusing the AEC framing: an
# opt-in 1-byte type prefix when ?aec=1, else bare PCM) at ?sr= (default 16000).
# This is the low-latency transport — no WebM/ffmpeg in the hot path.
# How often the offline-kind handler re-decodes the live window for a partial
# (streaming-kind decodes every frame, no cadence needed).
SHERPA_OFFLINE_PARTIAL_S = float(os.environ.get("OMNIVOICE_SHERPA_OFFLINE_PARTIAL", "0.8"))
# Utterance gate for the offline-kind handler: once the trailing this-many
# seconds of the live buffer fall below the RMS floor, the utterance is
# COMMITTED — decoded, flushed as a `final`, and dropped from the buffer. Each
# decode is thereby bounded by one utterance instead of the whole session
# (the old full-buffer re-decode was O(n²)), and a sentence commits ~0.6s
# after the user stops speaking instead of only at EOF.
SHERPA_OFFLINE_SILENCE_S = float(os.environ.get("OMNIVOICE_SHERPA_OFFLINE_SILENCE", "0.6"))
SHERPA_OFFLINE_RMS_FLOOR = float(os.environ.get("OMNIVOICE_SHERPA_OFFLINE_RMS", "0.01"))
def _pcm16_to_f32(pcm: bytes):
"""int16 little-endian mono PCM bytes → float32 numpy in [-1, 1]."""
import numpy as np
if not pcm:
return np.zeros(0, dtype=np.float32)
# Guard against an odd trailing byte from a split frame.
if len(pcm) % 2:
pcm = pcm[:-1]
return np.frombuffer(pcm, dtype=np.int16).astype(np.float32) / 32768.0
async def _sherpa_session(websocket: WebSocket):
"""Shared WS receive setup for the sherpa handlers.
Returns ``(get_frame, state)`` where ``get_frame`` is an async callable
that yields the next near-end (mic) PCM bytes, ``b""`` for a keepalive/ref
frame, or ``None`` on EOF/disconnect. ``state`` carries sample rate, AEC,
and the disconnect flag for the caller's finaliser.
"""
pcm_sr = 16000
try:
pcm_sr = int(websocket.query_params.get("sr", "16000"))
except (TypeError, ValueError):
pcm_sr = 16000
aec = None
if websocket.query_params.get("aec") in ("1", "true", "on"):
try:
from services.aec import NlmsEchoCanceller
aec = NlmsEchoCanceller(sample_rate=pcm_sr)
except Exception as e:
logger.warning("AEC requested but disabled (sherpa): %s", e)
aec = None
return pcm_sr, aec
async def _recv_pcm_frame(websocket: WebSocket, aec):
"""Receive one frame; return (kind, pcm_bytes).
kind {"near","eof","skip"}. Demuxes AEC-tagged frames when ``aec`` is on
and feeds the playback reference into the canceller. A text "EOF" or an
empty/closed socket yields kind "eof".
"""
msg = await websocket.receive()
mtype = msg.get("type")
if mtype == "websocket.disconnect":
return "eof", b""
if mtype != "websocket.receive":
return "skip", b""
data = msg.get("bytes")
if data is not None:
if len(data) == 0:
return "eof", b""
if aec is not None:
kind, payload = _demux_aec_frame(data)
if kind == "far":
aec.push_far_end(payload)
return "skip", b""
if not payload:
return "skip", b""
return "near", aec.process_near_end(payload)
return "near", data
if msg.get("text") == "EOF":
return "eof", b""
return "skip", b""
async def _sherpa_load_with_status(websocket: WebSocket, backend, spec) -> bool:
"""Build the recognizer off the event loop, narrating cold-start progress.
Sends ``{"type":"status","stage":"downloading"|"loading"}`` before the
load ("downloading" when the pinned assets aren't in the HF cache yet;
stage-only HF's per-file progress isn't worth a callback plumb-through)
and ``{"type":"status","stage":"ready"}`` after, so the widget can show
*why* the first dictation takes a moment. Returns False when the load
failed (the error frame is sent and the socket closed here).
"""
try:
from services import sherpa_dictation as _sd
stage = "loading" if _sd.is_installed(spec) else "downloading"
except Exception:
stage = "loading"
try:
await websocket.send_json({"type": "status", "stage": stage})
except Exception:
pass
try:
await asyncio.to_thread(backend.ensure_loaded)
except Exception as e:
logger.error("sherpa dictation load failed (%s): %s", spec.id, e)
try:
await websocket.send_json({"type": "error", "message": str(e),
"kind": "load", "detail": str(e)})
await websocket.close()
except Exception:
pass
return False
try:
await websocket.send_json({"type": "status", "stage": "ready"})
except Exception:
pass
return True
async def _run_sherpa_streaming(websocket: WebSocket, spec):
"""True streaming: feed the OnlineRecognizer frame-by-frame, emit `partial`
every time the decoded text grows, and `final` on sherpa's endpoint (silence)
detection and on EOF. <300ms perceived latency on CPU for the tiny models.
"""
import numpy as np
from services.asr_backend import get_sherpa_dictation_backend
pcm_sr, aec = await _sherpa_session(websocket)
logger.info("sherpa streaming dictation: model=%s sr=%d aec=%s",
spec.id, pcm_sr, bool(aec))
# Reuse the shared, per-model warm backend (#888): the recognizer is built
# once and shared across sessions instead of rebuilt (1.32.5s) per connect,
# so the first dictation is instant when the preload warmed it. Each session
# still gets its own decode stream below.
backend = get_sherpa_dictation_backend(spec.id)
# Build the recognizer off the event loop if it isn't warm yet
# (download-on-first-use + ONNX session init can take a moment); status
# frames keep the widget honest.
if not await _sherpa_load_with_status(websocket, backend, spec):
return
rec = backend._rec
stream = rec.create_stream()
last_partial = ""
committed: list[str] = [] # finalized utterances this session
client_disconnected = False
async def _send(payload) -> bool:
nonlocal client_disconnected
if client_disconnected:
return False
try:
await websocket.send_json(payload)
return True
except Exception:
client_disconnected = True
return False
def _decode_after_feed(pcm: bytes):
"""Blocking: feed one PCM frame, decode, return (text, is_endpoint).
Runs in a thread so the ONNX work never blocks the event loop."""
samples = _pcm16_to_f32(pcm)
if len(samples):
stream.accept_waveform(pcm_sr, samples)
while rec.is_ready(stream):
rec.decode_stream(stream)
endpoint = rec.is_endpoint(stream)
text = (rec.get_result(stream) or "").strip()
return text, endpoint
def _flush_final():
"""Blocking: pad + drain the stream for the trailing utterance."""
tail = np.zeros(int(0.5 * pcm_sr), dtype=np.float32)
stream.accept_waveform(pcm_sr, tail)
stream.input_finished()
while rec.is_ready(stream):
rec.decode_stream(stream)
return (rec.get_result(stream) or "").strip()
try:
while True:
kind, pcm = await _recv_pcm_frame(websocket, aec)
if kind == "eof":
break
if kind == "skip":
continue
text, endpoint = await asyncio.to_thread(_decode_after_feed, pcm)
if endpoint:
# Commit this utterance (polished — it gets pasted); reset
# for the next one.
text = polish_text(text)
if text:
committed.append(text)
await _send({"type": "final", "text": text,
"segments": [{"start": 0.0, "end": None, "text": text}],
"language": "auto", "engine": backend.id})
rec.reset(stream)
last_partial = ""
elif text and text != last_partial:
last_partial = text
await _send({"type": "partial", "text": text})
except WebSocketDisconnect:
client_disconnected = True
except Exception as e:
logger.warning("sherpa streaming loop ended: %s", e)
client_disconnected = True
# Drain the trailing (un-endpointed) utterance on EOF.
try:
tail_text = await asyncio.to_thread(_flush_final)
except Exception as e:
logger.debug("sherpa streaming flush failed: %s", e)
tail_text = ""
tail_text = polish_text(tail_text)
if tail_text and tail_text != (committed[-1] if committed else None):
committed.append(tail_text)
# Pieces are already polished; the join is too (polish is idempotent).
full = " ".join(t for t in committed if t).strip()
segments = [{"start": 0.0, "end": None, "text": t} for t in committed if t]
if not client_disconnected:
if full:
# Hard-bounded refinement (~4s): never delays this summary `final`
# beyond OMNIVOICE_REFINE_TIMEOUT_S even with a dead LLM endpoint.
try:
from services.refinement import maybe_refine_async
refined = await maybe_refine_async(full)
except Exception:
refined = None
payload = {"type": "final", "text": full, "segments": segments,
"language": "auto", "engine": backend.id}
if refined and refined != full:
payload["refined_text"] = refined
await _send(payload)
else:
await _send({"type": "final", "text": "", "segments": [],
"language": "auto", "engine": backend.id})
try:
await websocket.close()
except Exception:
pass
async def _run_sherpa_offline(websocket: WebSocket, spec):
"""Offline-kind sherpa model with live partials, utterance-windowed.
Raw PCM accumulates in a *live* buffer holding only the current
(uncommitted) utterance. Every ~800ms the live window is re-decoded for a
``partial``; when the trailing ~0.6s of it fall below the RMS floor the
utterance is committed decoded once more, flushed as a ``final``, and
its samples dropped so per-partial cost is bounded by one utterance
(not the whole session) and sentences commit as the user pauses instead
of only at EOF."""
from services.asr_backend import get_sherpa_dictation_backend
pcm_sr, aec = await _sherpa_session(websocket)
logger.info("sherpa offline dictation: model=%s sr=%d aec=%s",
spec.id, pcm_sr, bool(aec))
# Shared, per-model warm backend (#888) — built once, reused per session.
backend = get_sherpa_dictation_backend(spec.id)
if not await _sherpa_load_with_status(websocket, backend, spec):
return
buf = bytearray() # live (uncommitted) PCM only
committed: list[str] = [] # polished utterances already flushed
last_partial = ""
running = True
client_disconnected = False
last_audio = time.monotonic()
# Trailing-silence gate window, in bytes of int16 mono PCM.
sil_bytes = max(2, int(SHERPA_OFFLINE_SILENCE_S * pcm_sr) * 2)
async def _send(payload) -> bool:
nonlocal client_disconnected
if client_disconnected:
return False
try:
await websocket.send_json(payload)
return True
except Exception:
client_disconnected = True
return False
def _rms(pcm: bytes) -> float:
samples = _pcm16_to_f32(pcm)
if not len(samples):
return 0.0
return float((samples * samples).mean() ** 0.5)
def _decode_window(pcm: bytes) -> str:
samples = _pcm16_to_f32(pcm)
if not len(samples):
return ""
return backend._decode_offline(samples, pcm_sr)
async def receive():
nonlocal running, client_disconnected, last_audio
try:
while running:
kind, pcm = await _recv_pcm_frame(websocket, aec)
if kind == "eof":
running = False
break
if kind == "skip":
continue
buf.extend(pcm)
last_audio = time.monotonic()
except WebSocketDisconnect:
client_disconnected = True
running = False
except Exception as e:
logger.debug("sherpa offline receive ended: %s", e)
running = False
async def _commit(snapshot: bytes):
"""Finalize one utterance: decode it off-thread, flush a polished
`final`, drop its samples from the live buffer. `receive()` may
append while we decode only the snapshot's prefix is dropped."""
nonlocal last_partial
try:
text = await asyncio.to_thread(_decode_window, snapshot)
except Exception as e:
logger.debug("sherpa offline commit decode failed: %s", e)
return
del buf[:len(snapshot)]
last_partial = ""
text = polish_text(text)
if text:
committed.append(text)
await _send({"type": "final", "text": text,
"segments": [{"start": 0.0, "end": None, "text": text}],
"language": "auto", "engine": backend.id})
async def partials():
nonlocal last_partial, running
while running:
await asyncio.sleep(SHERPA_OFFLINE_PARTIAL_S)
if not running or len(buf) < 2000:
continue
snapshot = bytes(buf)
if len(snapshot) > sil_bytes and \
_rms(snapshot[-sil_bytes:]) < SHERPA_OFFLINE_RMS_FLOOR:
if _rms(snapshot[:-sil_bytes]) >= SHERPA_OFFLINE_RMS_FLOOR:
await _commit(snapshot)
else:
# Pure silence — drop it (keep the gate window for
# continuity) so a long pause can't grow the buffer.
del buf[:len(snapshot) - sil_bytes]
continue
try:
text = await asyncio.to_thread(_decode_window, snapshot)
except Exception as e:
logger.debug("sherpa offline partial failed: %s", e)
continue
if text and text != last_partial:
last_partial = text
await _send({"type": "partial", "text": text})
recv_task = asyncio.create_task(receive())
part_task = asyncio.create_task(partials())
await asyncio.wait([recv_task, part_task], return_when=asyncio.FIRST_COMPLETED)
running = False
for t in (recv_task, part_task):
if not t.done():
t.cancel()
try:
await t
except (asyncio.CancelledError, Exception):
pass
# Drain the trailing (un-committed) utterance on EOF.
try:
tail = await asyncio.to_thread(_decode_window, bytes(buf))
except Exception as e:
logger.error("sherpa offline final failed: %s", e)
tail = ""
tail = polish_text(tail)
if tail:
committed.append(tail)
# Pieces are already polished; the join is too (polish is idempotent).
full = " ".join(committed).strip()
segments = [{"start": 0.0, "end": None, "text": t} for t in committed]
if not client_disconnected:
payload = {"type": "final", "text": full, "segments": segments,
"language": "auto", "engine": backend.id}
if full:
# Hard-bounded refinement (~4s) — never delays the `final`.
try:
from services.refinement import maybe_refine_async
refined = await maybe_refine_async(full)
if refined and refined != full:
payload["refined_text"] = refined
except Exception:
pass
await _send(payload)
try:
await websocket.close()
except Exception:
pass
async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None) -> str:
async def _transcribe_buffer(chunks: list[bytes]) -> str:
"""Quick partial transcription of the current audio buffer."""
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
tmp = _chunks_to_wav(chunks)
if tmp is None:
return ""
try:
from services.model_manager import _gpu_pool
from services.asr_backend import get_capture_asr_backend, run_transcribe_guarded
from services.asr_backend import get_capture_asr_backend
def _run():
backend = get_capture_asr_backend()
result = backend.transcribe(tmp, word_timestamps=False)
return result.get("text", "")
# Bound dictation transcribes (#730): a wedged whisperx/CTranslate2 call
# must not hold its GPU-pool worker forever and starve TTS / other ASR
# into a "can't reach backend"; on timeout the pool is reset to recover.
text = await run_transcribe_guarded(_gpu_pool, _run, what="Dictation")
loop = asyncio.get_running_loop()
text = await loop.run_in_executor(_gpu_pool, _run)
return text.strip()
finally:
try:
@@ -787,16 +223,16 @@ async def _transcribe_buffer(chunks: list[bytes], *, pcm_sr: int | None = None)
pass
async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = None) -> dict:
async def _transcribe_buffer_full(chunks: list[bytes]) -> dict:
"""Full transcription with timing info for the final result."""
tmp = _pcm16_to_wav(b"".join(chunks), pcm_sr) if pcm_sr else _chunks_to_wav(chunks)
tmp = _chunks_to_wav(chunks)
if tmp is None:
return {"text": "", "segments": [], "language": "unknown",
"duration_s": 0, "transcription_time_s": 0, "engine": "none"}
try:
from services.model_manager import _gpu_pool
from services.asr_backend import get_capture_asr_backend, run_transcribe_guarded
from services.asr_backend import get_capture_asr_backend
def _run():
backend = get_capture_asr_backend()
@@ -809,12 +245,6 @@ async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = N
if not full_text and segments:
full_text = " ".join(s.get("text", "") for s in segments).strip()
# Wave 1.1: strip Whisper hallucination loops from the final
# text (the string that gets auto-pasted). Segments keep the
# raw recognition so their timings stay truthful.
from services.refinement import collapse_repetitive_artifacts
full_text = collapse_repetitive_artifacts(full_text)
duration = max((s.get("end", 0) for s in segments), default=0.0)
return {
@@ -831,9 +261,8 @@ async def _transcribe_buffer_full(chunks: list[bytes], *, pcm_sr: int | None = N
"engine": backend.id,
}
# Bounded + pool-resetting on timeout (#730), same rationale as the
# partial path above.
return await run_transcribe_guarded(_gpu_pool, _run, what="Dictation")
loop = asyncio.get_running_loop()
return await loop.run_in_executor(_gpu_pool, _run)
finally:
try:
os.unlink(tmp)
+3 -9
View File
@@ -258,19 +258,13 @@ async def community_use(item_id: str, name: Optional[str] = Query(None)):
raise HTTPException(status_code=503, detail=f"Couldn't add this voice right now. Error: {e}")
try:
# A community "preset" is a synthetic designed voice (rendered from an
# instruct string) → kind='design'; a "voice" carries a real reference
# clip → kind='clone'. Setting kind makes the persona-gallery
# synthetic-only gating work (§R3) instead of defaulting all imports to
# 'clone'.
kind = "design" if item["type"] == "preset" else "clone"
with db_conn() as conn:
conn.execute(
"INSERT INTO voice_profiles "
"(id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at, kind) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
"(id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(profile_id, profile_name, audio_filename, ref_text, instruct,
item.get("language", "Auto"), None, item["id"], time.time(), kind),
item.get("language", "Auto"), None, item["id"], time.time()),
)
except Exception:
with __import__("contextlib").suppress(OSError):
-35
View File
@@ -1,35 +0,0 @@
"""Voice-design "describe your voice" API (issue #317).
Maps a free-text voice description onto the existing design parameter space
via the deterministic keyword mapper in ``core.describe_voice``. Pure CPU +
stdlib no model, no network so it imports and responds instantly in any
environment, including test/CI without model weights.
"""
from __future__ import annotations
from fastapi import APIRouter
from pydantic import BaseModel, Field
from core.describe_voice import parse_description
router = APIRouter()
class DescribeRequest(BaseModel):
description: str = Field(default="", max_length=2000)
@router.post("/design/describe")
def describe_voice(req: DescribeRequest) -> dict:
"""Parse a free-text description into design attrs + a validator-safe instruct.
Response shape::
{
"attrs": {"Gender": "female", "Age": "elderly", ... or "Auto"},
"instruct": "female, elderly, low pitch, british accent",
"matched": [{"category": "Age", "token": "elderly", "phrase": "elderly"}, ...],
"unmatched": ["slightly raspy"]
}
"""
return parse_description(req.description)
-127
View File
@@ -1,127 +0,0 @@
"""
Dictation router sherpa-onnx live-dictation engine.
Exposes the seven sherpa-onnx dictation models and the dictation prefs the
frontend dictation UI binds to.
GET /dictation/models the 7 models + install state (frontend model list)
GET /dictation/prefs { enabled, mode, model_id }
POST /dictation/prefs persist any subset of those prefs
Install state reuses the same HF-cache check the model store uses, so a model
shown "installed" here is the same snapshot the backend will load.
Prefs are stored in the shared ``prefs.json`` store under the ``dictation.*``
namespace (``dictation.enabled``, ``dictation.mode``, ``dictation.model_id``),
mirroring how the ASR/TTS engine picks persist.
"""
from __future__ import annotations
import logging
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from typing import Optional
from api.dependencies import require_loopback
from core import prefs
from services import sherpa_dictation as sd
router = APIRouter()
logger = logging.getLogger("omnivoice.dictation")
# Pref keys (the binding contract — the frontend writes exactly these).
PREF_ENABLED = "dictation.enabled"
PREF_MODE = "dictation.mode"
PREF_MODEL_ID = "dictation.model_id"
_DEFAULT_ENABLED = True
_DEFAULT_MODE = "toggle"
_VALID_MODES = ("toggle", "hold")
def _read_prefs() -> dict:
mid = prefs.get(PREF_MODEL_ID, sd.DEFAULT_MODEL_ID)
if not sd.is_sherpa_model(mid):
mid = sd.DEFAULT_MODEL_ID
mode = prefs.get(PREF_MODE, _DEFAULT_MODE)
if mode not in _VALID_MODES:
mode = _DEFAULT_MODE
return {
"enabled": bool(prefs.get(PREF_ENABLED, _DEFAULT_ENABLED)),
"mode": mode,
"model_id": mid,
}
@router.get("/dictation/models", dependencies=[Depends(require_loopback)])
def list_dictation_models():
"""The seven sherpa-onnx dictation models + install state.
Each entry: id, repo_id, label, tag ("offline"|"streaming"), recommended,
size_gb, languages, kind, and install state (installed/installing). The
``installed`` flag is computed from the same HF cache the model store reads,
so it matches the model-store row state.
"""
available, reason = sd.sherpa_available()
out = []
for spec in sd.list_specs():
out.append({
"id": spec.id,
"repo_id": spec.repo_id,
"label": spec.label,
"tag": spec.tag,
"recommended": spec.recommended,
"size_gb": spec.size_gb,
"languages": spec.languages,
"kind": spec.kind,
"installed": sd.is_installed(spec),
})
return {
"models": out,
"engine_available": available,
"engine_reason": None if available else reason,
"default_model_id": sd.DEFAULT_MODEL_ID,
}
@router.get("/dictation/prefs", dependencies=[Depends(require_loopback)])
def get_dictation_prefs():
return _read_prefs()
class DictationPrefsUpdate(BaseModel):
enabled: Optional[bool] = None
mode: Optional[str] = None
model_id: Optional[str] = None
@router.post("/dictation/prefs", dependencies=[Depends(require_loopback)])
def set_dictation_prefs(req: DictationPrefsUpdate):
"""Persist any subset of the dictation prefs. Validates ``mode`` and
``model_id`` so a bad value can't wedge the capture engine."""
if req.mode is not None:
if req.mode not in _VALID_MODES:
raise HTTPException(
status_code=400,
detail=f"mode must be one of {_VALID_MODES}",
)
prefs.set_(PREF_MODE, req.mode)
if req.model_id is not None:
if not sd.is_sherpa_model(req.model_id):
raise HTTPException(
status_code=400,
detail=f"unknown dictation model_id {req.model_id!r}",
)
# Normalise to the canonical dictation id (accept repo_id too).
prefs.set_(PREF_MODEL_ID, sd.get_spec(req.model_id).id)
if req.enabled is not None:
prefs.set_(PREF_ENABLED, bool(req.enabled))
# Rebuild the cached capture singleton so the change takes effect at once.
try:
from services import asr_backend
asr_backend._capture_backend = None
asr_backend._capture_backend_key = None
except Exception:
pass
return _read_prefs()
+89 -463
View File
@@ -16,7 +16,6 @@ from core.tasks import task_manager
from core import event_bus
from schemas.requests import DubIngestUrlRequest
from services.model_manager import get_model, _gpu_pool, _cpu_pool, get_diarization_pipeline, offload_tts_for_asr, restore_tts_after_asr
from services.asr_backend import ASRTimeoutError, reset_pool_after_wedge, run_transcribe_guarded
from services.audio_io import _safe_soundfile_write
from services.ffmpeg_utils import find_ffmpeg
from services.segmentation import (
@@ -24,18 +23,13 @@ from services.segmentation import (
assign_speakers_from_diarization,
assign_speakers_from_turns,
assign_speakers_heuristic,
resplit_segments_by_diarization,
resplit_segments_by_turns,
_words_from_whisper,
clean_up_segments,
)
from services.onset_align import snap_segment_starts
from services import dub_pipeline
router = APIRouter()
logger = logging.getLogger("omnivoice.api")
# ── Legacy-name aliases to services/dub_pipeline.py ────────────────────────
# Phase 2.4 moved the business logic into a service. Other routers
# (dub_generate, dub_translate, dub_export) + internal call sites below still
@@ -365,48 +359,14 @@ async def dub_ingest_url(req: DubIngestUrlRequest):
TRANSCRIBE_CHUNK_S = float(os.environ.get("OMNIVOICE_TRANSCRIBE_CHUNK_S", "30.0"))
TRANSCRIBE_CHUNK_TIMEOUT_S = float(os.environ.get("OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S", "120.0"))
#: How many times to attempt each transcribe chunk before giving up on it. A
#: transient wedge (esp. the first chunk, where whisperx cold-loads its model)
#: shouldn't silently drop that whole window — retry once on a fresh pool so the
#: transcript doesn't come back "missing the beginning".
_CHUNK_TRANSCRIBE_ATTEMPTS = max(1, int(os.environ.get("OMNIVOICE_TRANSCRIBE_CHUNK_ATTEMPTS", "2")))
_sse_event = dub_pipeline.sse_event
_prep_event_helper = dub_pipeline.prep_event # alias; we keep the module-local _prep_event below for the inline one-liner shape
#: User-facing warning emitted when auto voice cloning is skipped because the
#: speaker labels came from the silence-gap heuristic (see _diarize /
#: extract_speaker_clones — gap-based labels routinely mix two people's audio
#: into one reference, which is how "made up" clone voices happen).
CLONE_SKIP_HEURISTIC_MSG = (
"auto voice cloning skipped: speaker labels are gap-based estimates — "
"set up diarization (Settings → Models → pyannote) for per-speaker clones"
)
def _clamp_num_speakers(value) -> Optional[int]:
"""Clamp the user's speaker-count hint to a sane 120 range.
Shared by the SSE and legacy transcribe endpoints so the two can't drift.
None / non-int / out-of-range None (auto-detect), so a bad query string
can never break a diarization call.
"""
if value is None:
return None
try:
value = int(value)
except (TypeError, ValueError):
return None
return value if 1 <= value <= 20 else None
@router.get("/dub/transcribe-stream/{job_id}")
async def dub_transcribe_stream(
job_id: str,
num_speakers: Optional[int] = None,
per_segment_refs: bool = True,
):
async def dub_transcribe_stream(job_id: str, num_speakers: Optional[int] = None):
"""Stream per-chunk segments via SSE, then emit diarized final pass.
Pre-flight checks (missing job, missing audio, ASR not loaded) are emitted
@@ -418,14 +378,15 @@ async def dub_transcribe_stream(
pyannote auto-detects the count but its auto-detect can collapse a
multi-speaker clip to a single speaker (issue #274). When the user knows
the exact count, supplying it forces pyannote to return that many speakers.
On paths that can't honor the hint exactly (inline ASR turns, the
silence-gap heuristic) it is never silently dropped: the heuristic cycles
the requested count and a `warning` SSE event tells the user how far the
labels can be trusted.
"""
# Clamp to a sane range; ignore anything non-positive / absurd so a bad
# query string can never break the diarization call. None → auto-detect.
num_speakers = _clamp_num_speakers(num_speakers)
if num_speakers is not None:
try:
num_speakers = int(num_speakers)
num_speakers = num_speakers if 1 <= num_speakers <= 20 else None
except (TypeError, ValueError):
num_speakers = None
job = _get_job(job_id)
@@ -452,11 +413,6 @@ async def dub_transcribe_stream(
asr_audio_target = job.get("vocals_path")
if not asr_audio_target or not os.path.exists(asr_audio_target):
asr_audio_target = job.get("audio_path")
# #963: onset snapping is only trustworthy on the Demucs vocals
# track. When separation failed/was skipped, dub_pipeline sets
# vocals_path to the mixed audio_path — so compare paths instead
# of trusting the key's presence.
asr_on_vocals = bool(asr_audio_target) and asr_audio_target != job.get("audio_path")
if not asr_audio_target or not os.path.exists(asr_audio_target):
preflight_error = "No audio available for transcription."
else:
@@ -464,22 +420,10 @@ async def dub_transcribe_stream(
try:
# The PyTorch-Whisper backend lazily builds its own pipeline
# when no preloaded `_asr_pipe` is present (issue #255), so it
# no longer needs OMNIVOICE_PRELOAD_TTS_ASR=1.
# no longer needs OMNIVOICE_PRELOAD_TTS_ASR=1 — don't reject it
# here; any load failure surfaces per-chunk with a real cause.
_asr_backend = get_active_asr_backend(asr_pipe=getattr(_model, "_asr_pipe", None))
# Eagerly load the model HERE so a real load failure (e.g.
# WhisperX: missing weights, CTranslate2/cuDNN mismatch, the
# torch-2.6 weights-only VAD regression) surfaces once, with
# its actual cause, as a clean preflight `error` event —
# instead of being buried in N cryptic per-chunk failures
# and retried on every chunk (#578). Run in a thread so the
# (blocking) load doesn't stall the event loop.
_ensure_loaded = getattr(_asr_backend, "ensure_loaded", None)
if callable(_ensure_loaded):
await asyncio.get_running_loop().run_in_executor(
_gpu_pool, _ensure_loaded
)
except Exception as e:
logger.exception("transcribe preflight: ASR load failed (job=%s)", job_id)
from core.failure import build_failure
f = build_failure(e, stage="transcribe-preflight", include_diagnostic=False)
preflight_error = "ASR backend initialization failed: " + f["reason"] + (
@@ -487,16 +431,9 @@ async def dub_transcribe_stream(
)
scene_cuts = job.get("scene_cuts") or []
async def _gen_body():
async def gen():
if preflight_error:
# Always follow a terminal `error` with `done` so the stream closes
# via a named event, not a raw connection drop. A bare error+close
# races the browser's native EventSource error (which carries no
# `data`); if that native error wins, the client falls back to the
# misleading generic "stream dropped … ASR backend failed" message
# and the real cause (in `detail`) is lost (#578).
yield _sse_event("error", {"detail": preflight_error, "retryable": True})
yield _sse_event("done", {})
yield _sse_event("error", {"detail": preflight_error})
return
import math
import tempfile
@@ -511,9 +448,7 @@ async def dub_transcribe_stream(
try:
audio_np, sr = await loop.run_in_executor(_cpu_pool, _load)
except Exception as e:
# Terminal error → always emit `done` (see preflight note, #578).
yield _sse_event("error", {"detail": f"audio load failed: {e}", "retryable": True})
yield _sse_event("done", {})
yield _sse_event("error", {"detail": f"audio load failed: {e}"})
return
total = float(len(audio_np)) / float(sr) if sr else 0.0
@@ -530,9 +465,6 @@ async def dub_transcribe_stream(
logger.warning("offload_tts_for_asr failed (continuing): %s", e)
all_segments: list[dict] = []
# Words (global-timeline) retained so diarization can re-split a segment
# that spans two speakers' turns at the word boundary (#486).
all_words: list = []
detected_lang = None
next_seg_id = 0
chunk_errors: list[str] = []
@@ -584,59 +516,31 @@ async def dub_transcribe_stream(
logger.exception("chunk transcribe failed (backend=%s)", _asr_backend.id)
return {"chunks": [], "language": None, "error": str(e)}
# Retry a failed/timed-out chunk once on a fresh pool before giving
# up. Otherwise a transient wedge on the FIRST chunk (whisperx often
# cold-loads its model there, the #730 hang) drops that whole window
# and the transcript is "missing the beginning, only middle+end".
# The retry reuses the same audio window, so a recovered chunk fills
# the hole instead of leaving silent gaps.
part = None
for _attempt in range(1, _CHUNK_TRANSCRIBE_ATTEMPTS + 1):
# A wedged chunk gets the SAME guarded-timeout + pool-reset
# semantics as the whole-file paths (#730/#851):
# run_transcribe_guarded bounds the call, abandons the poisoned
# pool so the retry (and any concurrent TTS work) gets a fresh
# worker, and raises the actionable ASRTimeoutError. Run it as
# a task and poll so we can keep yielding pings — the
# EventSource connection drops without them.
pool_reset_by_guard = False
task = asyncio.ensure_future(run_transcribe_guarded(
_gpu_pool, _transcribe_chunk,
what=f"Dub chunk {i + 1}/{chunks_n}",
timeout=TRANSCRIBE_CHUNK_TIMEOUT_S,
timeout_env="OMNIVOICE_TRANSCRIBE_CHUNK_TIMEOUT_S",
))
try:
# wait_for in a loop to yield pings so the EventSource connection doesn't drop
fut = loop.run_in_executor(_gpu_pool, _transcribe_chunk)
waited = 0.0
part = None
while True:
done, _pending = await asyncio.wait({task}, timeout=5.0)
done, pending = await asyncio.wait([fut], timeout=5.0)
if done:
part = done.pop().result()
break
yield _sse_event("ping", {})
try:
part = task.result()
except ASRTimeoutError as e:
# The guard already reset the pool; keep the actionable
# message (it names the durable fixes, and — after repeated
# timeouts — the crash-isolated engine escape hatch).
pool_reset_by_guard = True
logger.error(
"Transcribe chunk %d/%d timed out after %.0fs (attempt %d/%d, job=%s)",
i + 1, chunks_n, TRANSCRIBE_CHUNK_TIMEOUT_S, _attempt,
_CHUNK_TRANSCRIBE_ATTEMPTS, job_id,
)
part = {"chunks": [], "language": None, "error": str(e)}
# Success → keep it. Failure/timeout → retry once on a fresh
# worker (the internal _transcribe_chunk except returns an
# error-part; the timeout path already reset the pool).
if part is not None and not part.get("error"):
break
if _attempt < _CHUNK_TRANSCRIBE_ATTEMPTS:
logger.warning(
"Retrying transcribe chunk %d/%d after failure/timeout (next attempt %d/%d, job=%s)",
i + 1, chunks_n, _attempt + 1, _CHUNK_TRANSCRIBE_ATTEMPTS, job_id,
)
if not pool_reset_by_guard:
reset_pool_after_wedge(
_gpu_pool, what=f"Dub chunk {i + 1}/{chunks_n}")
waited += 5.0
if waited >= TRANSCRIBE_CHUNK_TIMEOUT_S:
# Re-raise TimeoutError if we exceed the overall limit
raise asyncio.TimeoutError()
except asyncio.TimeoutError:
logger.error(
"Transcribe chunk %d/%d timed out after %.0fs (job=%s)",
i + 1, chunks_n, TRANSCRIBE_CHUNK_TIMEOUT_S, job_id,
)
part = {
"chunks": [], "language": None,
"error": f"Chunk {i+1} timed out after {TRANSCRIBE_CHUNK_TIMEOUT_S:.0f}s — "
f"ASR backend may be stuck. Try restarting the server.",
}
if part.get("error"):
chunk_errors.append(part["error"])
logger.warning("Chunk %d/%d error: %s", i + 1, chunks_n, part["error"])
@@ -644,29 +548,7 @@ async def dub_transcribe_stream(
detected_lang = part["language"]
asr_speaker_turns.extend(part.get("speaker_turns") or [])
chunk_segs = segment_transcript(part, duration=t1, scene_cuts=scene_cuts)
# Same word source segment_transcript used (already global-timeline),
# kept for the post-diarization speaker re-split (#486).
try:
all_words.extend(_words_from_whisper(part))
except Exception:
pass
# #280: Whisper often stretches a segment's start back over
# leading music/silence (classic case: speech begins at 0:03,
# transcript says 0.0 → the dub plays 3 s early). Snap starts
# forward to the actual speech onset. `audio_np` is the same
# track ASR ran on — vocals.wav when Demucs succeeded. #963:
# when it didn't (mixed audio), snapping is disabled — every
# footstep/sigh/score cue is a false onset candidate there.
try:
snap_segment_starts(chunk_segs, audio_np, sr,
separated_vocals=asr_on_vocals)
except Exception as e:
logger.warning("onset alignment skipped for chunk %d: %s", i, e)
# Provisional per-chunk labels for the streaming UI only — the
# final diarization pass below overwrites them. Honor the user's
# speaker-count hint here too so the interim view doesn't flip
# between 2 and N speakers.
chunk_segs = assign_speakers_heuristic(chunk_segs, num_speakers)
chunk_segs = assign_speakers_heuristic(chunk_segs)
for s in chunk_segs:
s["id"] = f"s{next_seg_id:05x}"
s["text_original"] = s.get("text", "")
@@ -720,110 +602,30 @@ async def dub_transcribe_stream(
return
def _diarize():
"""Returns (segments, warning_payload_or_None, labels_source).
`labels_source` records where the speaker labels came from
`"pyannote"` | `"turns"` | `"heuristic"` so downstream
auto-clone extraction can refuse to cut reference audio from
gap-based estimates (a mixed-speaker reference is how "made up"
clone voices happen).
"""Returns (segments, warning_payload_or_None).
`warning_payload` is a structured dict
`{detail, error_class, docs_url}` whenever we silently fell back
to the silence-gap heuristic (no HF_TOKEN, model unavailable,
license not accepted, or pyannote raised) or whenever the
user's `num_speakers` hint could not be honored exactly. The
heuristic only detects speaker turns from >1.2s silences, so a
rapid-fire manwoman exchange will read as one speaker. Issue
#78 — we attach an `error_class` so the front-end's errorDocsMap
can render a "See docs" deeplink instead of a dead-end toast.
license not accepted, or pyannote raised). The heuristic only
detects speaker turns from >1.2s silences, so a rapid-fire
manwoman exchange will read as one speaker. Issue #78 — we
attach an `error_class` so the front-end's errorDocsMap can
render a "See docs" deeplink instead of a dead-end toast.
"""
# The active ASR backend already diarized inline (FunASR cam++):
# use its speaker turns directly and skip pyannote entirely (#182).
if asr_speaker_turns:
logger.info("Using inline ASR diarization (%d turns); skipping pyannote.", len(asr_speaker_turns))
return assign_speakers_from_turns(all_segments, asr_speaker_turns), None
from services.model_manager import (
DIARIZATION_ERR_LICENSE,
DIARIZATION_ERR_NO_TOKEN,
)
from core import error_docs_map
def _hint_suffix() -> str:
"""Honest caveat appended to heuristic-fallback warnings when a
multi-speaker hint is set: the count is now honored, but the
heuristic can't attribute voices. (A hint of 1 IS fully
honored one label so it needs no caveat.)"""
if not num_speakers or num_speakers < 2:
return ""
return (
f" Your speaker-count setting ({num_speakers}) is only "
f"approximately honored: the heuristic cycles "
f"{num_speakers} speaker labels on silence gaps instead "
f"of recognizing voices, so lines may be attributed to "
f"the wrong speaker."
)
def _use_turns(crash: Exception | None = None, err_sentinel=None):
"""Label from the ASR backend's inline speaker turns; warn when
that means the user's explicit count can't be enforced."""
logger.info(
"Using inline ASR diarization (%d turns)%s.",
len(asr_speaker_turns),
"" if crash else "; skipping pyannote",
)
assigned = assign_speakers_from_turns(all_segments, asr_speaker_turns)
# #486: split any segment that spans two speakers' turns at the
# word boundary (single-speaker segments pass through unchanged).
resplit = resplit_segments_by_turns(assigned, all_words, asr_speaker_turns)
if not num_speakers:
return resplit, None, "turns"
error_class = (
"HF_AUTH_FAILED"
if err_sentinel == DIARIZATION_ERR_NO_TOKEN
else "PYANNOTE_LICENSE_REQUIRED"
)
if crash:
detail = (
f"Speaker diarization crashed mid-run "
f"({type(crash).__name__}); falling back to the ASR "
f"engine's built-in speaker turns. Speaker-count hint "
f"ignored: the detected count may differ from the "
f"{num_speakers} you set."
)
else:
detail = (
f"Speaker-count hint ignored: pyannote diarization is "
f"unavailable, so the ASR engine's built-in speaker "
f"turns were used and the detected count may differ "
f"from the {num_speakers} you set. Set up diarization "
f"(Settings → Models → pyannote) to enforce an exact "
f"speaker count."
)
return resplit, {
"detail": detail,
"error_class": error_class,
"docs_url": error_docs_map.lookup(error_class),
"speaker_hint": {"requested": num_speakers, "status": "ignored"},
}, "turns"
# The active ASR backend already diarized inline (FunASR cam++):
# its turns are the fast path and skip pyannote entirely (#182) —
# but ONLY when the user didn't set an explicit speaker count.
# Inline turns are labeled per-30s-chunk and can't be forced to N
# speakers, so a set num_speakers prefers pyannote — the one
# engine that honors an exact count. When pyannote can't load,
# the turns are still the best labels available; use them and say
# so instead of silently eating the hint.
diar_pipe = None
err_sentinel = None
if asr_speaker_turns:
if num_speakers:
diar_pipe, err_sentinel = get_diarization_pipeline(return_error=True)
if not diar_pipe:
return _use_turns(err_sentinel=err_sentinel)
logger.info(
"num_speakers=%d set: preferring pyannote over %d inline "
"ASR turns (only pyannote honors an exact count).",
num_speakers, len(asr_speaker_turns),
)
else:
diar_pipe, err_sentinel = get_diarization_pipeline(return_error=True)
diar_pipe, err_sentinel = get_diarization_pipeline(return_error=True)
if not diar_pipe:
# Phase 1 AUTH-01: ask the resolver (App → Env → HF-CLI),
# not just the env var. This is the #35 fix — users who
@@ -874,20 +676,13 @@ async def dub_transcribe_stream(
f"heuristic; rapid speaker turns may be merged."
)
error_class = "PYANNOTE_LICENSE_REQUIRED"
warning = {
"detail": detail + _hint_suffix(),
"error_class": error_class,
"docs_url": error_docs_map.lookup(error_class),
}
if num_speakers:
warning["speaker_hint"] = {
"requested": num_speakers,
"status": "approximate" if num_speakers > 1 else "honored",
}
return (
assign_speakers_heuristic(all_segments, num_speakers),
warning,
"heuristic",
assign_speakers_heuristic(all_segments),
{
"detail": detail,
"error_class": error_class,
"docs_url": error_docs_map.lookup(error_class),
},
)
try:
# Pass the user's speaker-count hint through to pyannote when
@@ -899,17 +694,9 @@ async def dub_transcribe_stream(
diar = diar_pipe(asr_audio_target, num_speakers=num_speakers)
else:
diar = diar_pipe(asr_audio_target)
assigned = assign_speakers_from_diarization(all_segments, diar)
# #486: split any segment that spans two speakers' turns at the
# word boundary (single-speaker segments pass through unchanged).
return resplit_segments_by_diarization(assigned, all_words, diar), None, "pyannote"
return assign_speakers_from_diarization(all_segments, diar), None
except Exception as e:
logger.error(f"Diarization failed: {e}")
# Inline ASR turns beat the silence-gap heuristic as a crash
# fallback (this path is reachable with turns present since a
# set num_speakers routes turns-jobs through pyannote).
if asr_speaker_turns:
return _use_turns(crash=e)
# Mid-run failure — classify against the same sentinels so a
# post-load 401 (rare but possible after a token rotation)
# still gets the right docs deeplink.
@@ -920,50 +707,36 @@ async def dub_transcribe_stream(
if err_class_post == DIARIZATION_ERR_LICENSE
else "PYANNOTE_LICENSE_REQUIRED" # LOAD failures land here too
)
warning = {
"detail": (
f"Speaker diarization crashed mid-run "
f"({type(e).__name__}); falling back to a silence-gap "
f"heuristic. Rapid speaker turns may be merged."
+ _hint_suffix()
),
"error_class": error_class,
"docs_url": error_docs_map.lookup(error_class),
}
if num_speakers:
warning["speaker_hint"] = {
"requested": num_speakers,
"status": "approximate" if num_speakers > 1 else "honored",
}
return (
assign_speakers_heuristic(all_segments, num_speakers),
warning,
"heuristic",
assign_speakers_heuristic(all_segments),
{
"detail": (
f"Speaker diarization crashed mid-run "
f"({type(e).__name__}); falling back to a silence-gap "
f"heuristic. Rapid speaker turns may be merged."
),
"error_class": error_class,
"docs_url": error_docs_map.lookup(error_class),
},
)
fut_diar = loop.run_in_executor(_gpu_pool, _diarize)
final_segs = None
diar_warning = None
labels_source = "heuristic"
while True:
done, pending = await asyncio.wait([fut_diar], timeout=5.0)
if done:
final_segs, diar_warning, labels_source = done.pop().result()
final_segs, diar_warning = done.pop().result()
break
yield _sse_event("ping", {})
if diar_warning:
logger.warning("diarization fallback: %s", diar_warning.get("detail"))
payload = {
yield _sse_event("warning", {
"detail": diar_warning.get("detail"),
"source": "diarization",
"error_class": diar_warning.get("error_class"),
"docs_url": diar_warning.get("docs_url"),
}
# Machine-readable trail of what happened to the user's
# speaker-count hint (the `detail` text carries the human story).
if diar_warning.get("speaker_hint"):
payload["speaker_hint"] = diar_warning["speaker_hint"]
yield _sse_event("warning", payload)
})
job["segments"] = final_segs
@@ -975,123 +748,27 @@ async def dub_transcribe_stream(
try:
from services.speaker_clone import extract_speaker_clones, auto_profile_id
vocals_for_clone = job.get("vocals_path") or asr_audio_target
clones = {}
if labels_source == "heuristic":
# Clone-purity guard: heuristic labels are silence-gap
# estimates, not voice identity — a per-speaker reference cut
# from them routinely concatenates two people's audio and the
# clone sounds "made up". Skip auto-clones and say so instead
# of shipping bad ones. (extract_speaker_clones enforces the
# same guard internally; this branch exists to surface the
# warning to the user.)
logger.info(
"auto speaker clones skipped (labels_source=heuristic, job=%s)",
job_id,
)
yield _sse_event("warning", {
"detail": CLONE_SKIP_HEURISTIC_MSG,
"source": "speaker_clone",
})
else:
fut_clones = loop.run_in_executor(
_cpu_pool, lambda: extract_speaker_clones(
vocals_for_clone, final_segs,
os.path.dirname(vocals_for_clone),
labels_source=labels_source,
),
)
while True:
done, pending = await asyncio.wait([fut_clones], timeout=5.0)
if done:
clones = done.pop().result()
break
yield _sse_event("ping", {})
if clones:
from services.speaker_clone import refine_ref_texts
# Bound the re-transcribe like every other ASR dispatch in
# this file (#730): a wedged transcribe would otherwise hold
# the GPU-pool worker forever and starve later work into a
# "can't reach backend". On timeout the guard resets the pool
# and raises — keep the original (unrefined) clones, matching
# refine_ref_text's own "failure is a strict no-op" fallback.
try:
clones = await run_transcribe_guarded(
_gpu_pool,
lambda: refine_ref_texts(clones, _asr_backend),
what="Dub clone ref-text refine",
)
except ASRTimeoutError as e:
logger.warning(
"clone ref-text refine timed out; keeping original ref_text: %s", e
)
# Wave 3.2: per-segment clone refs. Cut each long-enough segment's
# own reference from the vocals so the dub of each line matches the
# prosody of its source line. Short lines fall back to the
# per-speaker clone below. Default on; the user can force
# per-speaker by disabling it (job["per_segment_refs"]).
seg_clones = {}
job["per_segment_refs"] = per_segment_refs
if per_segment_refs:
try:
from services.speaker_clone import extract_segment_refs
seg_ids_for_clone = [s.get("id", i) for i, s in enumerate(final_segs)]
seg_clones = await loop.run_in_executor(
_cpu_pool, lambda: extract_segment_refs(
vocals_for_clone, final_segs,
os.path.dirname(vocals_for_clone),
seg_ids=seg_ids_for_clone,
),
)
if seg_clones:
from services.speaker_clone import refine_ref_texts
# Same guard as the per-speaker refine above (#730):
# keep the original seg_clones on a wedge/timeout.
try:
seg_clones = await run_transcribe_guarded(
_gpu_pool,
lambda: refine_ref_texts(seg_clones, _asr_backend),
what="Dub segment ref-text refine",
)
except ASRTimeoutError as e:
logger.warning(
"segment ref-text refine timed out; keeping original ref_text: %s", e
)
job["segment_clones"] = seg_clones
except Exception as e:
logger.warning("per-segment clone refs skipped: %s", e)
if clones or seg_clones:
if clones:
job["speaker_clones"] = clones
# Default each segment's profile_id to its detected speaker's
# auto-clone — but only if the user hasn't already assigned
# something. (#486)
#
# We prefer the UI-visible `auto:{speaker}` id over the
# per-segment `auto-seg:{id}` id even when a per-segment ref
# exists, because the dub editor's Voice dropdown only renders
# `auto:` options ("From Video → Speaker N"). An `auto-seg:`
# value matches no <option>, so the row silently read
# "Default" while the speaker was actually bound — exactly the
# reported bug. The per-segment ref is NOT lost: dub_generate's
# `auto:` branch transparently prefers this segment's own
# per-segment ref (job["segment_clones"][seg_id]) when present,
# so a row shown as "Speaker 1" still clones from its own line
# when that line is long enough.
fut_clones = loop.run_in_executor(
_cpu_pool, extract_speaker_clones,
vocals_for_clone, final_segs, os.path.dirname(vocals_for_clone),
)
clones = None
while True:
done, pending = await asyncio.wait([fut_clones], timeout=5.0)
if done:
clones = done.pop().result()
break
yield _sse_event("ping", {})
if clones:
job["speaker_clones"] = clones
# Default each segment's profile_id to its speaker's auto-clone,
# but only if the user hasn't already assigned something.
for s in final_segs:
if s.get("profile_id"):
continue
spk = s.get("speaker_id") or "Speaker 1"
if spk in clones:
s["profile_id"] = auto_profile_id(spk)
continue
# No per-speaker clone for this speaker (too little usable
# audio overall) but this single line was long enough for
# its own ref — fall back to the per-segment id. The editor
# can't render it, but generation still clones correctly.
sid = str(s.get("id", ""))
if sid and sid in seg_clones:
s["profile_id"] = f"auto-seg:{sid}"
except Exception as e:
logger.warning("speaker_clone extraction skipped: %s", e)
@@ -1120,25 +797,6 @@ async def dub_transcribe_stream(
})
yield _sse_event("done", {})
async def gen():
# Terminal-event guard (#516): the SSE stream must NEVER close without a
# terminal event. Any unanticipated exception in the body (e.g. an ASR
# load that escapes the per-chunk handler) previously dropped the
# connection, which the frontend can only report as "stream dropped,
# likely ASR failed" — hiding the real cause. Emit a structured `error`
# (with the actionable hint from build_failure) then `done`, so the user
# sees the real failure + a Retry instead of a silent disconnect.
try:
async for ev in _gen_body():
yield ev
except Exception as e: # noqa: BLE001 — last-resort stream finalizer
logger.exception("transcribe stream crashed (job=%s)", job_id)
from core.failure import build_failure
f = build_failure(e, stage="transcribe", include_diagnostic=False)
detail = f["reason"] + (f"{f['hint']}" if f.get("hint") else "")
yield _sse_event("error", {"detail": detail, "retryable": True})
yield _sse_event("done", {})
return StreamingResponse(
gen(),
media_type="text/event-stream",
@@ -1150,28 +808,18 @@ async def dub_transcribe_stream(
@router.post("/dub/transcribe/{job_id}")
async def dub_transcribe(job_id: str, num_speakers: Optional[int] = None):
"""Legacy synchronous transcribe (kept for the headless CLI).
`num_speakers` mirrors the SSE endpoint's query param (same 120 clamp):
an exact speaker count forwarded to pyannote, or cycled by the silence-gap
heuristic when pyannote is unavailable. None auto-detect.
"""
num_speakers = _clamp_num_speakers(num_speakers)
async def dub_transcribe(job_id: str):
job = _get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
_model = await get_model()
def _transcribe():
asr_audio_target = job.get("vocals_path")
if not asr_audio_target or not os.path.exists(asr_audio_target):
asr_audio_target = job.get("audio_path")
# #963: same source-awareness as the SSE endpoint — vocals_path
# falls back to the mixed audio_path when Demucs failed/skipped.
asr_on_vocals = bool(asr_audio_target) and asr_audio_target != job.get("audio_path")
import torch
detected_lang = None
@@ -1214,35 +862,17 @@ async def dub_transcribe(job_id: str, num_speakers: Optional[int] = None):
scene_cuts = job.get("scene_cuts") or []
segments = segment_transcript(result, duration=job.get("duration", 0.0), scene_cuts=scene_cuts)
# #280: snap segment starts forward to the actual speech onset so the
# dub doesn't begin seconds before the original speaker does. #963:
# only on the separated vocals track — on mixed audio every ambient
# sound is a false onset candidate, so snapping is disabled.
try:
audio_for_onset, onset_sr = sf.read(asr_audio_target, dtype="float32")
snap_segment_starts(segments, audio_for_onset, onset_sr,
separated_vocals=asr_on_vocals)
except Exception as e:
logger.warning("onset alignment skipped: %s", e)
diar_pipe = get_diarization_pipeline()
if diar_pipe:
try:
diar_target = job.get("vocals_path") or job.get("audio_path")
# Same hint pass-through as the SSE endpoint (#274): omit the
# kwarg entirely when unset so we don't depend on it existing
# in every pyannote build.
if num_speakers:
logger.info("Diarizing with num_speakers=%d (user hint)", num_speakers)
diarization = diar_pipe(diar_target, num_speakers=num_speakers)
else:
diarization = diar_pipe(diar_target)
diarization = diar_pipe(diar_target)
segments = assign_speakers_from_diarization(segments, diarization)
except Exception as e:
logger.error(f"Pyannote diarization failed during inference: {e}. Falling back to heuristic.")
segments = assign_speakers_heuristic(segments, num_speakers)
segments = assign_speakers_heuristic(segments)
else:
segments = assign_speakers_heuristic(segments, num_speakers)
segments = assign_speakers_heuristic(segments)
# Previously ran `segment_for_subtitles(segments)` here. Removed 2026-04-21 —
# that splitter enforces Netflix's 17 CPS reading-speed ceiling which
@@ -1262,11 +892,7 @@ async def dub_transcribe(job_id: str, num_speakers: Optional[int] = None):
try:
loop = asyncio.get_running_loop()
try:
# Bound the whole-file transcribe (#730): a wedged whisperx/CTranslate2
# call would otherwise hold its GPU-pool worker forever and starve
# every other request into a "can't reach backend". run_transcribe_guarded
# also resets the pool on timeout so capacity is restored.
segments_result = await run_transcribe_guarded(_gpu_pool, _transcribe, what="Dub")
segments_result = await loop.run_in_executor(_gpu_pool, _transcribe)
except asyncio.CancelledError:
job["aborted"] = True
raise
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+75 -562
View File
@@ -2,14 +2,13 @@ import os
import time
import asyncio
import logging
from typing import Optional
from fastapi import APIRouter
from fastapi.responses import JSONResponse
from schemas.requests import TranslateRequest
from services.model_manager import _cpu_pool, _gpu_pool
from services.translator import cinematic_available, cinematic_refine_many, _cinematic_budget
from api.routers.dub_core import _get_job, _save_job
from services.translator import cinematic_available, cinematic_refine_many
from api.routers.dub_core import _get_job
router = APIRouter()
logger = logging.getLogger("omnivoice.api")
@@ -46,70 +45,6 @@ LANG_NAMES = {
"id": "Indonesian", "uk": "Ukrainian",
}
# Regional dialect hints (#280 item 2). Maps a BCP-47 dialect code to the
# instruction injected into LLM translation prompts so the output uses that
# region's vocabulary and grammar (the reporter's example: choosing Argentina
# should yield "Vos sos muy listo", not the Peninsular "Tú eres muy listo").
# Only LLM-backed paths can honor these — provider="openai" and the
# quality="cinematic" refine pass. Keep entries short: they ride on every
# per-segment prompt, so verbosity = wall time.
DIALECT_HINTS = {
# Spanish
"es-ES": "European Spanish (Spain): use tú/vosotros forms and Peninsular vocabulary.",
"es-MX": "Mexican Spanish: use tú/ustedes forms and Mexican vocabulary.",
"es-AR": "Rioplatense Spanish (Argentina): use voseo — 'vos' with its verb forms (e.g. 'vos sos', 'tenés') and 'ustedes'; prefer Argentinian vocabulary.",
"es-CO": "Colombian Spanish: use tú/usted as natural in Colombia and Colombian vocabulary.",
"es-CL": "Chilean Spanish: use Chilean vocabulary and expressions.",
# Portuguese
"pt-BR": "Brazilian Portuguese: use 'você' forms, Brazilian vocabulary and spelling.",
"pt-PT": "European Portuguese: use European vocabulary, spelling, and 'tu' where natural.",
# English
"en-US": "American English: use US spelling and vocabulary.",
"en-GB": "British English: use UK spelling and vocabulary.",
"en-AU": "Australian English: use Australian spelling and vocabulary.",
"en-IN": "Indian English: use Indian English vocabulary and conventions.",
# French
"fr-FR": "Metropolitan French (France): use standard French vocabulary.",
"fr-CA": "Canadian French (Québec): use Québécois vocabulary and expressions.",
"fr-BE": "Belgian French: use Belgian vocabulary (e.g. septante, nonante).",
# German
"de-DE": "Standard German (Germany): use Federal German vocabulary.",
"de-AT": "Austrian German: use Austrian vocabulary (e.g. Jänner, Erdapfel).",
"de-CH": "Swiss Standard German: use Swiss vocabulary and 'ss' instead of 'ß'.",
# Arabic
"ar-EG": "Egyptian Arabic: use Egyptian colloquial vocabulary where natural for dubbing.",
"ar-SA": "Gulf/Saudi Arabic flavor: prefer vocabulary natural to the Gulf region.",
"ar-MA": "Moroccan Arabic (Darija) flavor: prefer vocabulary natural to Morocco.",
# Dutch
"nl-NL": "Netherlands Dutch: use vocabulary standard in the Netherlands.",
"nl-BE": "Belgian Dutch (Flemish): use Flemish vocabulary and expressions.",
}
def dialect_clause(dialect: Optional[str]) -> str:
"""Prompt fragment for a requested dialect, or '' when unset/unknown.
Unknown-but-plausible codes (e.g. "es-PE") still get a generic regional
clause so users aren't limited to the curated list.
"""
if not dialect or not str(dialect).strip():
return ""
code = str(dialect).strip()
hint = DIALECT_HINTS.get(code)
if hint:
return f" Target dialect — {hint}"
# Generic fallback for any lang-REGION shaped code we don't curate.
if "-" in code:
lang, _, region = code.partition("-")
lang_name = LANG_NAMES.get(lang, lang)
if region:
return (
f" Use the vocabulary, grammar, and expressions of {lang_name} "
f"as spoken in the region '{region}'."
)
return ""
# Per-language script enforcement. Maps language code → required Unicode
# block(s) the translation must contain. Used as a sanity gate after the
# LLM responds: if the output contains <50% characters from the expected
@@ -157,89 +92,15 @@ _nllb_tokenizer = None
_nllb_device = None
def _dialect_flags(req, applied: bool) -> dict:
"""Response fields describing whether the requested dialect was honored.
Empty dict when no dialect was requested, so existing response shapes
stay byte-identical for callers that never send one.
"""
if not getattr(req, "dialect", None):
return {}
return {"dialect": req.dialect, "dialect_applied": bool(applied)}
def _guess_lang_from_text(segments) -> str | None:
"""Best-effort source language from segment text, by script.
Used only as a last resort when neither the request nor the job carries a
detected language. Without this, the bare "en" fallback below forces
en -> en on non-English audio (e.g. Korean), which has no Argos package and
fails every segment even though ASR detected the language correctly.
"""
text = " ".join((getattr(s, "text", "") or "") for s in (segments or [])[:8])
has = lambda lo, hi: any(lo <= ord(c) <= hi for c in text)
if has(0x3040, 0x30FF):
return "ja" # Hiragana/Katakana — check before CJK (Japanese uses Kanji too)
if has(0xAC00, 0xD7A3) or has(0x1100, 0x11FF):
return "ko" # Hangul
if has(0x4E00, 0x9FFF):
return "zh" # CJK ideographs
if has(0x0400, 0x04FF):
return "ru" # Cyrillic
if has(0x0600, 0x06FF):
return "ar" # Arabic
return None
def _resolve_source_lang(req: TranslateRequest) -> str:
"""Pick source language: explicit request > job.source_lang > text guess > 'en'."""
"""Pick source language: explicit request > job.source_lang > 'en' fallback."""
if getattr(req, "source_lang", None):
return req.source_lang
if getattr(req, "job_id", None):
job = _get_job(req.job_id)
if job and job.get("source_lang"):
return job["source_lang"]
return _guess_lang_from_text(getattr(req, "segments", None)) or "en"
def _resolve_translation_context(req, client, model_name: str, timeout: float,
src_lang: str) -> Optional[dict]:
"""Cached auto-glossary context (theme + terms) for this job/target.
Cache lives on the dub job dict (``job["translation_context"][target]``)
and persists through the existing ``job_data`` JSON blob via ``_save_job``
no schema change. A transcript fingerprint keys the cache so an edited
transcript re-extracts; an unchanged transcript costs zero LLM calls on
re-translate. Any failure returns None translation proceeds without
context, never fails because of it. Blocking; run in an executor.
"""
from services import translation_quality as tq
texts = [(s.text or "") for s in req.segments]
fp = tq.transcript_fingerprint(texts)
job = _get_job(req.job_id) if getattr(req, "job_id", None) else None
if job is not None:
cached = (job.get("translation_context") or {}).get(req.target_lang)
if isinstance(cached, dict) and cached.get("fingerprint") == fp:
return cached
ctx = tq.extract_context_sync(
client, model_name, timeout,
segment_texts=texts,
source_lang=src_lang,
target_lang=req.target_lang,
source_name=LANG_NAMES.get(src_lang, src_lang),
target_name=LANG_NAMES.get(req.target_lang, req.target_lang),
)
if ctx is None:
return None
ctx = {**ctx, "fingerprint": fp}
if job is not None:
try:
job.setdefault("translation_context", {})[req.target_lang] = ctx
_save_job(req.job_id, job)
except Exception: # noqa: BLE001 — persistence is best-effort
logger.debug("translation context persist skipped", exc_info=True)
return ctx
return "en"
def _unload_nllb():
@@ -342,90 +203,14 @@ async def dub_translate(req: TranslateRequest):
translated = await loop.run_in_executor(_gpu_pool, _translate_nllb)
if os.environ.get("OMNIVOICE_UNLOAD_NLLB", "1") == "1":
_unload_nllb()
# Cinematic/Autofit refine + rate-ratio badges must run for NLLB too
# (previously this returned before _maybe_cinematic, so a Cinematic
# pick on NLLB silently produced plain Fast output). Unloading NLLB
# first is fine — the refine LLM is a separate network provider.
return await _maybe_cinematic(translated, req, src_lang, loop)
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
# LLM translation — resolves through the LLM Skills registry: per-skill
# "Dub translation" override → global active provider (Settings → LLM
# Providers). The keys users configure + test in the app now actually
# power this engine; the raw TRANSLATE_* env vars stay working as a
# power-user override so pre-skills setups see zero behavior change.
# OpenAI / Ollama Local LLM Translation
if provider == "openai":
from services import llm_skills
llm_timeout = llm_skills._default_timeout()
handle = None
try:
handle = llm_skills.resolve_skill_client("dub_translation")
except Exception: # noqa: BLE001 — resolution must never 500 a translate
logger.exception("dub_translation skill resolution failed; trying env fallback")
if handle is not None:
client = handle.client
model_name = handle.model
llm_timeout = handle.timeout
# The provider-store key never touches env; resolve it so the
# error scrubber below can redact it if a provider echoes it.
try:
from services import llm_providers
api_key = llm_providers.resolve_api_key(
llm_skills.effective_provider("dub_translation")) or api_key
except Exception: # noqa: BLE001 — scrub-key resolution is best-effort
pass
elif os.environ.get("TRANSLATE_BASE_URL") or api_key:
# Legacy env-only setup (no provider configured in-app).
from openai import OpenAI
# max_retries=0: a 429 + long Retry-After must not let one segment's
# SDK call sleep+retry and blow the overall translate wall time.
client = OpenAI(base_url=os.environ.get("TRANSLATE_BASE_URL"),
api_key=api_key or "local", max_retries=0)
model_name = os.environ.get("TRANSLATE_MODEL", "gpt-4o-mini")
else:
# Nothing configured anywhere — name the exact next step instead
# of letting an empty key surface as a raw 401 per segment.
try:
reason = llm_skills.resolve_skill("dub_translation").reason
except Exception: # noqa: BLE001
reason = None
if reason == "disabled":
friendly = (
"The LLM translation engine is turned off — enable the "
"'Dub translation' skill in Settings → LLM Skills, or "
"pick another engine in the Engine dropdown."
)
else:
friendly = (
"The LLM translation engine has no provider configured. "
"Add and test one in Settings → LLM Providers (it powers "
"this engine; route it per-skill in Settings → LLM "
"Skills), or set TRANSLATE_BASE_URL + TRANSLATE_API_KEY "
"+ TRANSLATE_MODEL. Or pick another engine in the "
"Engine dropdown."
)
return JSONResponse(status_code=400, content={"error": friendly})
from services import translation_quality as tq
# Two-stage quality toggles. None (old clients) = ON — an LLM
# translator is active on this branch by definition.
auto_glossary_on = req.auto_glossary if req.auto_glossary is not None else True
reflect_on = req.reflect if req.reflect is not None else True
# Stage 1 — auto-glossary: ONE pass over the full transcript for a
# theme summary + terminology map (cached per job/target/transcript),
# merged with the user's manual glossary (user entries win) and
# injected into every per-segment prompt below. With the toggle off
# the manual glossary still rides along — that costs no extra call.
auto_ctx = None
if auto_glossary_on:
auto_ctx = await loop.run_in_executor(
_cpu_pool, _resolve_translation_context,
req, client, model_name, llm_timeout, src_lang,
)
merged_terms = tq.merge_glossary(req.glossary, (auto_ctx or {}).get("terms"))
context_extra = tq.context_clause((auto_ctx or {}).get("theme", ""), merged_terms)
base_url = os.environ.get("TRANSLATE_BASE_URL")
model_name = os.environ.get("TRANSLATE_MODEL", "gpt-3.5-turbo")
from openai import OpenAI
client = OpenAI(base_url=base_url, api_key=api_key or "local")
def _build_prompt(src_code: str, tgt_code: str) -> str:
"""Build a system prompt that resists hallucinations on small
@@ -451,25 +236,13 @@ async def dub_translate(req: TranslateRequest):
f"only — do not use Latin/Roman letters, do not "
f"transliterate, do not output any other language."
)
# #280 item 2 — regional dialect/vocabulary. Only applied when
# the dialect belongs to the target language (a leftover
# "es-AR" must not contaminate a French translation).
dia_clause = ""
if req.dialect and str(req.dialect).lower().startswith(str(tgt_code).lower()[:2]):
dia_clause = dialect_clause(req.dialect)
base = (
return (
f"You are a professional dubbing translator. "
f"Translate the user's text from {src_name} into "
f"{tgt_name}.{script_clause}{dia_clause} "
f"{tgt_name}.{script_clause} "
f"Reply ONLY with the translated {tgt_name} text, do not "
f"add quotes, notes, headers, explanations, or commentary."
)
# Auto-glossary theme + merged terminology (user terms win) —
# every segment prompt carries the same brief, so recurring
# names/terms come out consistent across the whole dub.
if context_extra:
base = base + "\n\n" + context_extra
return base
def _translate_llm(seg):
if not seg.text or not seg.text.strip():
@@ -493,7 +266,6 @@ async def dub_translate(req: TranslateRequest):
res = client.chat.completions.create(
model=model_name,
temperature=0.2, # less drift than default 1.0
timeout=llm_timeout, # bound per call (OMNIVOICE_LLM_TIMEOUT, 45s default)
messages=[
{"role": "system", "content": sys_for_attempt},
{"role": "user", "content": seg.text},
@@ -513,33 +285,6 @@ async def dub_translate(req: TranslateRequest):
seg.id, attempt + 1, last_err,
)
continue
# Stage 2 — reflect pass: critique→rewrite the direct
# translation into natural spoken dialogue. Returns None
# on ANY failure/timeout/divergence, in which case the
# direct translation stands — refinement can never fail
# a segment that already translated fine. The belt-and-
# braces except keeps that guarantee even if the helper
# itself ever raised: without it, the enclosing attempt
# handler would burn a retry on a segment that already
# translated successfully.
if reflect_on:
polished = None
try:
polished = tq.reflect_translation_sync(
client, model_name, llm_timeout,
source_text=seg.text,
direct_text=out_text,
source_lang=src_lang,
target_lang=tgt_code,
target_name=LANG_NAMES.get(tgt_code, tgt_code),
extra_clause=context_extra,
)
except Exception as e: # noqa: BLE001
logger.warning("reflect pass skipped for %s: %s",
seg.id, e)
if polished:
return {"id": seg.id, "text": polished,
"literal": out_text}
return {"id": seg.id, "text": out_text}
except Exception as e:
last_err = f"{type(e).__name__}: {e}"
@@ -548,38 +293,24 @@ async def dub_translate(req: TranslateRequest):
seg.id, attempt + 1, e,
)
# Both attempts failed — keep source text + flag error so the
# frontend can surface "fallback to literal" warning. Scrub the
# provider error: some OpenAI-compatible providers echo the key
# or a user_id in the body, which must not reach the UI verbatim.
from core.scrub import scrub_provider_error
return {"id": seg.id, "text": seg.text,
"error": scrub_provider_error(last_err, api_key) or "llm-failed"}
# frontend can surface "fallback to literal" warning.
return {"id": seg.id, "text": seg.text, "error": last_err or "llm-failed"}
tasks = [loop.run_in_executor(_cpu_pool, _translate_llm, seg) for seg in req.segments]
translated = await asyncio.gather(*tasks)
translated.sort(key=lambda x: str(x["id"]))
# provider="openai" is already an LLM translation — _maybe_cinematic
# skips the reflect/adapt re-refine (already_llm) but still stamps
# rate-ratio badges and runs the bounded Autofit fit pass. Before
# this it returned here, so Cinematic/Autofit on the LLM engine did
# nothing.
return await _maybe_cinematic(translated, req, src_lang, loop, already_llm=True)
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
# Offline Argos Translate
if provider == "argos" or provider == "libretranslate":
try:
import argostranslate # noqa: F401
except ImportError:
# Single-source the install command from the engine registry so
# this 400 and the proactive Install button in the Engine
# selector can never drift (see translation_engines.install_command).
from services.translation_engines import install_command
cmd = install_command("argos") or "uv pip install argostranslate"
friendly = (
f"The '{provider}' translation engine needs the optional "
f"`argostranslate` Python package, which isn't installed in "
f"this backend. Install it with `{cmd}` "
f"and restart the server, or "
f"this backend. Install it with `uv pip install argostranslate` "
f"(or `pip install argostranslate`) and restart the server, or "
f"switch the Engine dropdown to another provider."
)
return JSONResponse(status_code=400, content={"error": friendly})
@@ -622,11 +353,7 @@ async def dub_translate(req: TranslateRequest):
return results
translated = await loop.run_in_executor(_cpu_pool, _translate_argos)
# Argos is the DEFAULT engine — routing it through _maybe_cinematic is
# the headline fix: a user who picks Cinematic/Autofit on Argos now
# gets the LLM refine + fit pass (and rate-ratio badges in Fast mode)
# instead of silent plain-Fast output.
return await _maybe_cinematic(translated, req, src_lang, loop)
return {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang}
# Legacy / API Deep_Translator logic.
# Preflight the optional `deep_translator` dep once so we fail with a
@@ -635,16 +362,11 @@ async def dub_translate(req: TranslateRequest):
try:
import deep_translator # noqa: F401
except ImportError:
# Same single-source install command as the Engine selector's Install
# button (translation_engines.install_command) — google/deepl/
# microsoft/mymemory all share the deep_translator package.
from services.translation_engines import install_command
cmd = install_command(provider) or "uv pip install deep_translator"
friendly = (
f"The '{provider}' translation engine needs the optional "
f"`deep_translator` Python package, which isn't installed in "
f"this backend. Install it with `{cmd}` "
f"and restart the server, or "
f"this backend. Install it with `uv pip install deep_translator` "
f"(or `pip install deep_translator`) and restart the server, or "
f"switch the Engine dropdown to Argos (local, bundled), NLLB "
f"(local, heavier), or OpenAI (LLM)."
)
@@ -700,11 +422,7 @@ async def dub_translate(req: TranslateRequest):
)
time.sleep(0.25 * (attempt + 1))
logger.error("translate %s -> %s gave up (provider=%s): %s", src_arg, seg_lc, provider, last_err)
# Scrub before it reaches the UI — DeepL/Microsoft errors can echo
# the API key (same class as the OpenAI user_id leak).
from core.scrub import scrub_provider_error
return {"id": seg.id, "text": seg.text,
"error": scrub_provider_error(last_err, _deepl_key or _msft_key or api_key) or "unknown"}
return {"id": seg.id, "text": seg.text, "error": last_err or "unknown"}
tasks = [loop.run_in_executor(_cpu_pool, _translate_single, seg) for seg in req.segments]
translated = await asyncio.gather(*tasks)
@@ -718,145 +436,24 @@ async def dub_translate(req: TranslateRequest):
return JSONResponse(status_code=500, content={"error": str(e)})
def _stamp_duration_plan(rows, req) -> None:
"""Attach a pre-synthesis duration-plan verdict to every row (in place).
Pure planning (services/duration_planner.py): estimate the natural
speech duration of each row's FINAL text — self-calibrated from this
job's already-synthesized segments when possible — and classify it
against slot + borrowable gap using fit_planner's own caps. The verdict
rides on the row as ``plan`` so the segment table can badge tight/
impossible segments BEFORE any GPU time is spent. Informational only
generation is never blocked. Never raises.
"""
try:
from services.duration_planner import calibration_from_job, classify_segments
timed = [
s for s in req.segments
if getattr(s, "start", None) is not None and getattr(s, "end", None) is not None
]
if not timed:
return # old client — no timeline info, no plan
text_by_id = {str(r["id"]): (r.get("text") or "") for r in rows}
segs = sorted(
(
{
"id": str(s.id),
"start": float(s.start),
"end": float(s.end),
"text": text_by_id.get(str(s.id), ""),
}
for s in timed
),
key=lambda d: d["start"],
)
calib = None
total_dur = 0.0
if getattr(req, "job_id", None):
job = _get_job(req.job_id)
if job:
calib = calibration_from_job(job, req.target_lang)
total_dur = float(job.get("duration") or 0.0)
verdicts = {
v["id"]: v
for v in classify_segments(
segs, req.target_lang, calibration=calib, total_dur_s=total_dur,
)
}
for row in rows:
v = verdicts.get(str(row["id"]))
if v is None or row.get("error") or not (row.get("text") or "").strip():
continue
row["plan"] = {
"status": v["status"],
"est_dur_s": v["est_dur_s"],
"available_s": v["available_s"],
"est_overrun_s": v["est_overrun_s"],
"calibrated": v["calibrated"],
}
except Exception as e: # noqa: BLE001 — planning must never sink a translate
logger.debug("duration-plan stamping skipped: %s", e)
async def _apply_condense_pass(rows, req, loop) -> None:
"""Opt-in LLM condensation for ``impossible`` rows (in place).
Fans ``condense_for_slot`` out on the CPU pool under the same wall-clock
budget the cinematic phase uses, so a slow LLM can't hang the translate.
Suggestions land as ``plan.suggested_text`` the user applies them per
segment; the row's ``text`` is never touched here. Every failure mode
(no LLM, LLM error, divergent reply, budget) degrades to no suggestion.
"""
targets = [
row for row in rows
if (row.get("plan") or {}).get("status") == "impossible"
and (row.get("text") or "").strip() and not row.get("error")
]
if not targets:
return
try:
from services.duration_planner import calibration_from_job, condense_for_slot
calib = None
if getattr(req, "job_id", None):
job = _get_job(req.job_id)
if job:
calib = calibration_from_job(job, req.target_lang)
source_by_id = {str(s.id): s.text for s in req.segments}
sem = asyncio.Semaphore(int(os.environ.get("OMNIVOICE_LLM_CONCURRENCY", "6")))
async def _one(row):
async with sem:
res = await loop.run_in_executor(
_cpu_pool,
lambda: condense_for_slot(
row["text"],
available_s=float(row["plan"]["available_s"]),
target_lang=req.target_lang,
source_text=source_by_id.get(str(row["id"])),
calibration=calib,
),
)
if res.get("applied") and res.get("text"):
row["plan"]["suggested_text"] = res["text"]
row["plan"]["suggested_est_dur_s"] = res.get("est_dur_s")
tasks = [asyncio.ensure_future(_one(row)) for row in targets]
budget = _cinematic_budget()
done, pending = await asyncio.wait(
tasks, timeout=budget if budget and budget > 0 else None,
)
for task in pending:
task.cancel() # abandon the executor thread (#730 pattern)
for task in done:
exc = task.exception()
if exc is not None:
logger.warning("condense pass segment failed: %s", exc)
except Exception as e: # noqa: BLE001 — a suggestion pass must never sink a translate
logger.warning("condense pass skipped: %s", e)
async def _finalize_duration_plan(rows, req, loop) -> None:
"""Stamp plan verdicts on the FINAL row texts, then (opt-in) condense."""
_stamp_duration_plan(rows, req)
if getattr(req, "condense", False):
await _apply_condense_pass(rows, req, loop)
def _stamp_predicted_rate_ratio(translated, req) -> None:
"""Stamp a predicted ``rate_ratio`` on every row that has a known slot.
No LLM needed just the per-language CPS table from ``services/speech_rate``.
The UI's ``seg-rate-badge`` reads it (Fast mode included) to show which
segments will compress hard at generation time, so users can edit text or
pick a heavier quality. Mutates ``translated`` in place; never raises.
async def _maybe_cinematic(translated, req, src_lang, loop):
"""If quality=cinematic and a usable LLM is configured, run REFLECT+ADAPT.
Otherwise return Fast-mode shape unchanged.
"""
quality = (getattr(req, "quality", None) or "fast").lower()
# Stamp the predicted rate_ratio on every translated row that has a
# known slot. Works for Fast mode too — no LLM needed; just the CPS
# table from services/speech_rate. The UI's `seg-rate-badge` reads
# this value and shows users which segments will compress hard at
# generation time, so they can edit text or pick Cinematic quality.
try:
from services.speech_rate import rate_ratio as _predict_rate_ratio
slots = {str(s.id): getattr(s, "slot_seconds", None) for s in req.segments}
for row in translated:
slot = slots.get(str(row["id"]))
seg_ref = next(
(s for s in req.segments if str(s.id) == str(row["id"])),
None,
)
slot = getattr(seg_ref, "slot_seconds", None) if seg_ref else None
text = (row.get("text") or "").strip()
if slot and text and not row.get("error"):
row["rate_ratio"] = round(
@@ -865,126 +462,18 @@ def _stamp_predicted_rate_ratio(translated, req) -> None:
except Exception as e:
logger.debug("non-LLM rate_ratio prediction skipped: %s", e)
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang, "quality_used": "fast"}
async def _apply_fit_pass(rows, req, slots_by_id, source_by_id, quality, loop, deadline) -> None:
"""Run the Autofit slot-fit pass over ``rows`` concurrently, in place.
Bounded by ``deadline`` (shared with the cinematic refine) so a slow /
rate-limited LLM can't spin the fit pass per-segment unbounded — the old
behavior, which ran one blocking ``adjust_for_slot`` per segment in the
merge loop, outside any budget. Segments still running at the deadline keep
their current text and get ``rate_error='fit-budget'``. Only rows with a
slot + text + no prior error participate.
"""
strict = (quality == "autofit")
items = []
for row in rows:
seg_id = str(row["id"])
slot = slots_by_id.get(seg_id)
text = row.get("text") or ""
if slot and text and not row.get("error"):
items.append((seg_id, text, float(slot), req.target_lang,
source_by_id.get(seg_id), strict))
if not items:
return
try:
from services.speech_rate import adjust_for_slot_many
fits = await adjust_for_slot_many(
items, executor=_cpu_pool, deadline=deadline, loop=loop,
)
except Exception as e:
logger.warning("rate-fit pass skipped: %s", e)
return
for row in rows:
f = fits.get(str(row["id"]))
if not f:
continue
if f.get("text"):
row["text"] = f["text"]
if f.get("rate_ratio") is not None:
row["rate_ratio"] = f["rate_ratio"]
if f.get("error"):
row["rate_error"] = f["error"]
async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False):
"""Post-process a literal translation into Cinematic/Autofit output.
Runs for EVERY provider now (Argos/NLLB/Google//OpenAI). The three
LLM-independent branches (nllb/argos) and the openai branch used to return
*before* reaching this, so a Cinematic/Autofit pick on them including the
DEFAULT Argos engine silently produced plain Fast output with a success
toast. Fast mode still returns the plain translation (plus rate-ratio badges).
``already_llm`` (provider="openai"): the translation was itself produced by
an LLM, so the REFLECT+ADAPT *re*-refine is skipped, but the bounded Autofit
fit pass + rate-ratio stamping still run, and the dialect the translate
prompt already baked in is reported as applied.
"""
quality = (getattr(req, "quality", None) or "fast").lower()
_stamp_predicted_rate_ratio(translated, req)
# #280 item 2 — regional dialect hint, guarded against a stale dialect from
# another language. For already_llm the initial translate prompt already
# applied it, so it's reported applied in the Fast-shape base too.
dialect_hint = ""
_dialect = getattr(req, "dialect", None)
if _dialect and str(_dialect).lower().startswith(str(req.target_lang).lower()[:2]):
dialect_hint = dialect_clause(_dialect)
base = {"translated": translated, "target_lang": req.target_lang, "source_lang": src_lang,
"quality_used": "fast",
**_dialect_flags(req, applied=(already_llm and bool(dialect_hint)))}
# Fast (and anything unrecognised) returns the plain translation unchanged
# (plus the pre-synthesis duration-plan badges — no LLM needed for those).
if quality not in ("cinematic", "autofit"):
await _finalize_duration_plan(translated, req, loop)
if quality != "cinematic":
return base
source_by_id: dict[str, str] = {str(s.id): s.text for s in req.segments}
slots_by_id = {
str(s.id): getattr(s, "slot_seconds", None)
for s in req.segments
if getattr(s, "slot_seconds", None)
}
# One wall-clock deadline shared by the whole LLM phase (refine + fit), so a
# slow/rate-limited provider can't run either pass unbounded. <=0 disables.
budget = _cinematic_budget()
deadline = (loop.time() + budget) if budget and budget > 0 else None
# provider="openai": already an LLM translation → skip REFLECT+ADAPT, keep
# the rate-ratio badges, still run the bounded fit pass.
if already_llm:
merged = []
for row in translated:
# A reflect-pass row already carries its pre-polish direct
# translation as `literal` — keep it instead of clobbering.
out = {"id": row["id"],
"text": row.get("text", "") or "",
"literal": row.get("literal") or row.get("text", "") or ""}
if row.get("error"):
out["error"] = row["error"]
if "rate_ratio" in row:
out["rate_ratio"] = row["rate_ratio"]
merged.append(out)
await _apply_fit_pass(merged, req, slots_by_id, source_by_id, quality, loop, deadline)
# Plan AFTER the fit pass — verdicts must describe the final text.
await _finalize_duration_plan(merged, req, loop)
return {"translated": merged, "target_lang": req.target_lang,
"source_lang": src_lang, "quality_used": quality,
**_dialect_flags(req, applied=bool(dialect_hint))}
# Non-LLM provider → the reflect/adapt refine needs a separately-configured
# LLM (Settings → LLM Providers). Without one, degrade to Fast with a flag.
if not cinematic_available():
logger.warning("%s requested but no LLM configured — returning Fast result.", quality)
logger.warning("cinematic requested but no LLM configured — returning Fast result.")
base["cinematic_skipped"] = "no-llm-configured"
await _finalize_duration_plan(translated, req, loop)
return base
# Build a map from id → original segment (to fetch source text + direction).
source_by_id: dict[str, str] = {str(s.id): s.text for s in req.segments}
directions: dict[str, str] = {
str(s.id): s.direction
for s in req.segments
@@ -992,7 +481,7 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
}
pairs = []
passthrough_index = {}
for row in translated:
for i, row in enumerate(translated):
seg_id = str(row["id"])
literal = row.get("text", "") or ""
if row.get("error") or not literal.strip():
@@ -1001,7 +490,6 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
pairs.append((seg_id, source_by_id.get(seg_id, ""), literal))
if not pairs:
await _finalize_duration_plan(translated, req, loop)
return base
refined = await cinematic_refine_many(
@@ -1010,11 +498,20 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
target_lang=req.target_lang,
glossary=req.glossary,
directions=directions,
dialect_hint=dialect_hint,
executor=_cpu_pool,
)
refined_by_id = {r["id"]: r for r in refined}
# Phase 4.4 — speech-rate fit pass. Segment boundaries aren't in the
# translate request (by design — translator is boundary-agnostic), so we
# only run it when the caller supplied `slot_seconds` on each segment.
# The frontend populates this for Cinematic calls from the edit view.
slots_by_id = {
str(s.id): getattr(s, "slot_seconds", None)
for s in req.segments
if getattr(s, "slot_seconds", None)
}
merged = []
for row in translated:
seg_id = str(row["id"])
@@ -1033,18 +530,34 @@ async def _maybe_cinematic(translated, req, src_lang, loop, *, already_llm=False
}
if r.get("error"):
out["error"] = r["error"]
# Optional slot-fit pass — only when the caller asked for cinematic
# *and* provided a slot. Runs best-effort; no-LLM or mid-loop failure
# just leaves the cinematic text untouched.
slot = slots_by_id.get(seg_id)
if slot and out["text"]:
try:
from services.speech_rate import adjust_for_slot
fit = await asyncio.to_thread(
adjust_for_slot,
out["text"],
slot_seconds=float(slot),
target_lang=req.target_lang,
source_text=source_by_id.get(seg_id),
)
if fit.get("text"):
out["text"] = fit["text"]
out["rate_ratio"] = fit.get("rate_ratio")
if fit.get("error"):
out["rate_error"] = fit["error"]
except Exception as e:
logger.warning("rate-fit skipped for %s: %s", seg_id, e)
merged.append(out)
# Phase 4.4 speech-rate fit pass — now concurrent + bounded (see helper).
await _apply_fit_pass(merged, req, slots_by_id, source_by_id, quality, loop, deadline)
# Plan AFTER the fit pass — verdicts must describe the final text.
await _finalize_duration_plan(merged, req, loop)
return {
"translated": merged,
"target_lang": req.target_lang,
"source_lang": src_lang,
"quality_used": quality,
**_dialect_flags(req, applied=bool(dialect_hint)),
"quality_used": "cinematic",
}
+6 -224
View File
@@ -15,9 +15,6 @@ Environment variables (`OMNIVOICE_TTS_BACKEND`, `OMNIVOICE_ASR_BACKEND`,
`OMNIVOICE_LLM_BACKEND`) still win over the UI choice so power-users can pin
a backend without Settings silently undoing it.
"""
import os
import re
import threading
from time import perf_counter
from fastapi import APIRouter, Depends, HTTPException
@@ -264,199 +261,16 @@ def engine_health(engine_id: str):
}
# ── Real-synthesis self-test (in-process TTS engines) ──────────────────────
#
# ``/health`` above is a liveness/import probe — for an in-process backend it
# only calls ``is_available()`` and the UI labels the result "deps OK". This
# route goes one step further: for an AVAILABLE, IN-PROCESS TTS engine it runs
# a *tiny real synthesis* from a fixed short phrase and reports duration +
# sample-rate + sample count, proving the engine actually emits audio rather
# than merely importing. The Compat Matrix's "Self-test" button calls it.
#
# Guardrails (kept identical across macOS/Windows/Linux per the default-feature
# rule — the phrase, timeout and gating don't branch on OS):
# * TTS family + available + in-process only. Subprocess engines keep their
# spawn-and-ping ``health_check`` (a real synth there is a sidecar
# cold-start — out of scope for a click-to-test affordance).
# * Bounded wall-clock timeout (``OMNIVOICE_SELFTEST_TIMEOUT_S``, default 90s):
# a runaway synth returns ``ok=False`` / ``timed_out=True`` instead of
# hanging the Settings panel. The orphaned worker is best-effort daemon.
# * A process-wide lock serialises self-tests so a click-storm can't stack
# concurrent model loads.
# * Only ever on user click (POST) — never on Settings load. Loopback-gated.
# Deliberately short + ASCII so the synth stays CPU-cheap and the phrase never
# trips the no-hardcoded-CJK guard.
_SELFTEST_PHRASE = "OmniVoice engine self test."
_SELFTEST_LOCK = threading.Lock()
def _selftest_timeout_s() -> float:
try:
return max(1.0, float(os.environ.get("OMNIVOICE_SELFTEST_TIMEOUT_S", "90")))
except (TypeError, ValueError):
return 90.0
def _sample_count(audio) -> int:
"""Total sample count of an engine's ``generate()`` return, tolerant of
torch.Tensor / numpy.ndarray / list shapes. 0 when it can't be measured."""
try:
shape = getattr(audio, "shape", None)
if shape is not None and len(shape) > 0:
return int(shape[-1])
return int(len(audio))
except Exception:
return 0
def _run_synth_bounded(backend, timeout_s: float) -> dict | None:
"""Run one tiny synthesis in a daemon thread, bounded by ``timeout_s``.
Returns ``{"audio": .., "duration_ms": ..}`` on success, ``{"error": exc}``
on a synth exception, or ``None`` when the timeout elapsed (worker left
running best-effort Python threads can't be force-killed)."""
box: dict = {}
def _worker():
t0 = perf_counter()
try:
audio = backend.generate(_SELFTEST_PHRASE, language="en", num_step=8)
box["audio"] = audio
except Exception as exc: # noqa: BLE001 — surfaced to the caller as ok=False
box["error"] = exc
finally:
box["duration_ms"] = (perf_counter() - t0) * 1000.0
th = threading.Thread(target=_worker, name="engine-selftest", daemon=True)
th.start()
th.join(timeout_s)
if th.is_alive():
return None
return box
class SelfTestResponse(BaseModel):
id: str
ok: bool
message: str
duration_ms: float
sample_rate: int | None = None
num_samples: int | None = None
audio_seconds: float | None = None
timed_out: bool = False
@router.post(
"/engines/{engine_id}/selftest",
response_model=SelfTestResponse,
dependencies=[Depends(require_loopback)],
)
def engine_selftest(engine_id: str):
"""Run a bounded, real synthesis on an available in-process TTS engine.
404 for an unknown TTS id; 400 when the engine is subprocess-isolated or
not currently available (a real synth on either is meaningless). Never
raises through to a 500 on a synth failure the exception is captured into
``ok=False`` / ``message`` so the panel renders a per-row failure."""
if engine_id not in tts_backend._REGISTRY:
raise HTTPException(
status_code=404,
detail=f"unknown TTS engine id: {engine_id!r}",
)
cls = tts_backend._REGISTRY[engine_id]
if getattr(cls, "_is_subprocess_isolated", False):
raise HTTPException(
status_code=400,
detail=(
f"{engine_id} is subprocess-isolated — self-test runs real "
"synthesis for in-process engines only. Use Test engine "
"(spawn-and-ping) for subprocess engines."
),
)
try:
ok, msg = cls.is_available()
except Exception as exc: # noqa: BLE001
ok, msg = False, f"{type(exc).__name__}: {exc}"
if not ok:
raise HTTPException(
status_code=400,
detail=(
f"{engine_id} is not available: {tts_backend._mask_hf_tokens(msg)}. "
"Install/enable the engine, then self-test."
),
)
timeout_s = _selftest_timeout_s()
# Serialise so a click-storm can't stack concurrent model loads.
with _SELFTEST_LOCK:
backend = _get_engine_instance(cls)
res = _run_synth_bounded(backend, timeout_s)
if res is None:
return SelfTestResponse(
id=engine_id,
ok=False,
message=f"timed out after {timeout_s:.0f}s (model still loading?)",
duration_ms=timeout_s * 1000.0,
timed_out=True,
)
if "error" in res:
exc = res["error"]
return SelfTestResponse(
id=engine_id,
ok=False,
message=tts_backend._mask_hf_tokens(f"{type(exc).__name__}: {exc}"),
duration_ms=res.get("duration_ms", 0.0),
)
n = _sample_count(res.get("audio"))
try:
sr = int(getattr(backend, "sample_rate", 0) or 0) or None
except Exception:
sr = None
secs = round(n / sr, 3) if (sr and n) else None
return SelfTestResponse(
id=engine_id,
ok=n > 0,
message="synthesized" if n > 0 else "engine returned no audio",
duration_ms=res["duration_ms"],
sample_rate=sr,
num_samples=n or None,
audio_seconds=secs,
)
class SelectEngineRequest(BaseModel):
family: str # "tts" | "asr" | "llm"
backend_id: str
# Only meaningful for family="tts", backend_id="mlx-audio" (#981) — picks
# which of mlx-audio's curated models is actually loaded. A curated key
# ("kokoro") or a raw HF repo id ("mlx-community/Kokoro-82M-bf16") — the
# same tolerance MLXAudioBackend.__init__ already has. Ignored otherwise.
model_id: str | None = None
class SelectEngineResponse(BaseModel):
family: str
active: str
env_override: bool
# Routing verdict for the selected engine on THIS host (#21). Always present
# so the UI can show a confirm/warning toast on a cpu_fallback pick without
# branching on key presence; defaults match a legacy/degraded row.
routing_status: str = "cpu_only"
effective_device: str = "cpu"
routing_reason: str | None = None
@router.post("/engines/select", response_model=SelectEngineResponse)
@router.post("/engines/select")
def select_engine(req: SelectEngineRequest):
"""Persist a family's engine pick to prefs.json. Refuses unknown backends,
backends whose deps aren't installed, AND backends that cannot run on THIS
host's hardware (routing_status == "unavailable") — so the UI can't silently
brick a pipeline by picking an engine that needs a GPU this machine lacks.
A `cpu_fallback` pick is allowed (it runs, just slower) only a hard
`unavailable` is blocked. LLM is never routing-gated (its status is "n/a")."""
"""Persist a family's engine pick to prefs.json. Refuses unknown backends
+ refuses backends whose deps aren't installed (so the UI can't silently
brick a pipeline by picking an unavailable engine)."""
family = _FAMILIES.get(req.family)
if not family:
raise HTTPException(400, f"Unknown family: {req.family}. Expected one of tts/asr/llm.")
@@ -464,44 +278,12 @@ def select_engine(req: SelectEngineRequest):
available = {b["id"]: b for b in module.list_backends()}
if req.backend_id not in available:
raise HTTPException(400, f"Unknown {req.family} backend: {req.backend_id!r}")
entry = available[req.backend_id]
if not entry["available"]:
reason = entry.get("reason") or "unavailable"
if not available[req.backend_id]["available"]:
reason = available[req.backend_id].get("reason") or "unavailable"
raise HTTPException(400, f"Backend {req.backend_id} not ready: {reason}")
# Host-routing gate (no silent CPU fallback). `.get` is defensive so an
# older/legacy payload without routing keys still selects cleanly.
if entry.get("routing_status") == "unavailable":
why = entry.get("routing_reason") or "requires a GPU this host doesn't have"
raise HTTPException(
400,
f"Backend {req.backend_id} can't run on this machine: {why}. "
f"Pick an engine with a CPU path, or one that supports this host's GPU.",
)
# #981: mlx-audio multiplexes 7+ curated models behind one backend id —
# persist the model pick alongside the backend id so the UI can actually
# select which curated model gets loaded (previously it always defaulted
# to Kokoro no matter what the user downloaded in Settings → Models).
if req.family == "tts" and req.backend_id == "mlx-audio" and req.model_id is not None:
known_keys = tts_backend.MLXAudioBackend.CURATED_MODELS
# Accept a curated key OR a raw HF repo id ("owner/name") — the same
# tolerance MLXAudioBackend.__init__ already has for power users.
# Anything else (typo'd key, malformed id) is rejected outright
# rather than silently persisted as a "custom repo" that then fails
# to resolve at load time.
if req.model_id not in known_keys and not re.fullmatch(r"[\w.-]+/[\w.-]+", req.model_id):
raise HTTPException(
400,
f"Unknown mlx-audio model: {req.model_id!r}. Expected one of "
f"{sorted(known_keys)} or a HF repo id like 'owner/name'.",
)
prefs.set_("mlx_audio_model_id", req.model_id)
prefs.set_(pref_key, req.backend_id)
return {
"family": req.family,
"active": module.active_backend_id(),
"env_override": bool(__import__("os").environ.get(f"OMNIVOICE_{req.family.upper()}_BACKEND")),
# Echo the routing verdict so the UI can warn on a cpu_fallback pick.
"routing_status": entry.get("routing_status", "cpu_only"),
"effective_device": entry.get("effective_device", "cpu"),
"routing_reason": entry.get("routing_reason"),
}
+2 -10
View File
@@ -161,18 +161,13 @@ async def search_youtube(
list. Users are responsible for the licensing of whatever they import.
"""
try:
# yt-dlp is an importable module, never a PATH requirement — run it
# via the interpreter (honors the Settings → Audio tools overlay).
from services.media_tools import ytdlp_invocation
ytdlp_argv, ytdlp_env = ytdlp_invocation()
result = await spawn_subprocess(
*ytdlp_argv,
"yt-dlp",
"--dump-json",
"--remote-components", "ejs:github",
f"ytsearch{max_results}:{query}",
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
env=ytdlp_env,
)
stdout, stderr = await result.communicate()
@@ -223,10 +218,8 @@ async def download_youtube_clip(
temp_path = str(VOICE_GALLERY_DIR / f"{voice_id}.%(ext)s")
try:
from services.media_tools import ytdlp_invocation
ytdlp_argv, ytdlp_env = ytdlp_invocation()
cmd = [
*ytdlp_argv,
"yt-dlp",
"--remote-components", "ejs:github",
"-f",
"bestaudio",
@@ -246,7 +239,6 @@ async def download_youtube_clip(
*cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
env=ytdlp_env,
)
stdout, stderr = await result.communicate()
File diff suppressed because it is too large Load Diff
+9 -24
View File
@@ -189,21 +189,15 @@ def auto_extract(project_id: str, req: AutoExtractRequest):
Writes them as `auto=1` rows. Existing terms with the same (source,target)
are NOT duplicated. Returns the full current glossary after the pass.
"""
# Resolved through the LLM Skills registry so auto-extract can be toggled
# or routed to its own provider (Settings → LLM Skills) independently of
# the translation pipeline. None == disabled or no provider configured.
from services import llm_skills
from services.translator import _llm_client, _llm_model, _llm_timeout # reuse same client
handle = llm_skills.resolve_skill_client("glossary_extract")
if handle is None:
client = _llm_client()
if client is None:
raise HTTPException(
status_code=503,
detail=(
"Auto-extract needs an LLM. Set one up in Settings → LLM Providers "
"(pick a provider, add its key, choose a model, Test) — or use local "
"Ollama / LM Studio for a fully offline setup — and make sure the "
"Glossary auto-extract skill is enabled in Settings → LLM Skills, "
"then try again."
"Auto-extract needs an LLM. Set TRANSLATE_BASE_URL + TRANSLATE_API_KEY "
"(Ollama works locally: base_url=http://localhost:11434/v1) and try again."
),
)
@@ -226,9 +220,9 @@ def auto_extract(project_id: str, req: AutoExtractRequest):
)
try:
res = handle.client.chat.completions.create(
model=handle.model,
timeout=handle.timeout,
res = client.chat.completions.create(
model=_llm_model(),
timeout=_llm_timeout(),
messages=[
{"role": "system", "content": system},
{"role": "user", "content": user},
@@ -237,18 +231,9 @@ def auto_extract(project_id: str, req: AutoExtractRequest):
body = (res.choices[0].message.content or "").strip()
except Exception as e:
logger.warning("auto-extract LLM call failed: %s", e)
# Scrub the provider error — some OpenAI-compatible providers echo the
# API key or a user_id in the body, which must not reach the UI verbatim.
from core.scrub import scrub_provider_error
from services import llm_providers
_p = llm_providers.active_provider()
_key = llm_providers.resolve_api_key(_p) if _p else None
raise HTTPException(
status_code=502,
detail=(
"LLM didn't respond. Check Settings → Logs → Backend for the trace. "
f"Error: {scrub_provider_error(e, _key)}"
),
detail=f"LLM didn't respond. Check Settings → Logs → Backend for the trace. Error: {e}",
)
# Parse: SOURCE || TARGET || note (lines are allowed to be sloppy — we're forgiving).
-158
View File
@@ -1,158 +0,0 @@
"""Longform Job Library (PR 7).
``GET /longform/jobs`` list finished Audiobook + Story renders so the user can
re-download them from the Projects view. The render itself (the m4b/mp3) already
landed in ``OUTPUTS_DIR`` and is served at ``/audio/<output>``; here we just
recover, from each finished job's persisted SSE tail, the output filename plus
the chapter count and duration the ``done`` event carried.
Pure recovery, no synthesis. Defensive by construction: a job whose ``done``
event is missing or unparseable is skipped, never surfaced and never a 500.
The work lives in :func:`build_longform_library`, a pure function over the
job-store callables, so it's unit-testable without importing ``main`` (and the
torch graph behind it).
"""
from __future__ import annotations
import json
import logging
from typing import Callable, Optional
from fastapi import APIRouter, Query
logger = logging.getLogger("omnivoice.longform_jobs")
router = APIRouter()
#: Job types this library surfaces. Both flow through the shared longform
#: renderer (``_render_longform_sse``) and emit the same ``done`` event shape.
_LONGFORM_TYPES = ("audiobook", "story")
def _done_payload_from_events(events: list[dict]) -> Optional[dict]:
"""Recover the final ``{"type": "done", ...}`` payload from a job's SSE tail.
Each row's ``payload`` is the JSON the renderer stored via
``job_store.append_event(job_id, json.dumps(payload))``. We scan newest-first
and return the first parseable ``done`` event. Anything malformed is skipped
this never raises.
"""
for ev in reversed(events):
raw = ev.get("payload") if isinstance(ev, dict) else None
if not raw or not isinstance(raw, str):
continue
try:
obj = json.loads(raw)
except (ValueError, TypeError):
continue
if isinstance(obj, dict) and obj.get("type") == "done":
return obj
return None
def _coerce_int(value, default: int = 0) -> int:
try:
return int(value)
except (TypeError, ValueError):
return default
def _coerce_float(value, default: float = 0.0) -> float:
try:
return float(value)
except (TypeError, ValueError):
return default
def build_longform_library(
list_jobs: Callable[..., list[dict]],
events_since: Callable[..., list[dict]],
*,
limit: int = 50,
) -> list[dict]:
"""Build the newest-first list of finished longform renders.
Pure over the two job-store callables so tests can pass them directly:
* ``list_jobs(status="done", limit=...)`` all done jobs, newest-first.
* ``events_since(job_id)`` that job's persisted SSE events.
Returns ``[{job_id, type, title?, output, duration_s, chapters,
created_at}]``. Jobs that aren't a longform type, or whose ``done`` event /
output filename can't be recovered, are silently skipped — the library only
ever lists things the user can actually re-download.
"""
limit = max(1, min(_coerce_int(limit, 50), 500))
try:
# Over-fetch: non-longform done jobs (dub, etc.) get filtered out below,
# so ask for more rows than the caller's limit to still fill the page.
rows = list_jobs(status="done", limit=limit * 4)
except Exception:
logger.warning("longform library: list_jobs failed", exc_info=True)
return []
out: list[dict] = []
for row in rows or []:
if len(out) >= limit:
break
try:
job_type = row.get("type")
job_id = row.get("id")
if job_type not in _LONGFORM_TYPES or not job_id:
continue
try:
events = events_since(job_id)
except Exception:
logger.warning("longform library: events_since failed for %s",
job_id, exc_info=True)
continue
done = _done_payload_from_events(events or [])
if not done:
continue
output = done.get("output")
if not output or not isinstance(output, str):
continue # nothing to re-download → not worth listing
item = {
"job_id": job_id,
"type": job_type,
"output": output,
"duration_s": round(_coerce_float(done.get("duration_s")), 2),
"chapters": _coerce_int(done.get("chapters")),
"created_at": row.get("created_at"),
}
# Title is optional — prefer the done event, fall back to job meta.
title = done.get("title")
if not title:
meta_raw = row.get("meta_json")
if isinstance(meta_raw, str) and meta_raw:
try:
meta = json.loads(meta_raw)
if isinstance(meta, dict):
title = meta.get("title")
except (ValueError, TypeError):
title = None
if title:
item["title"] = title
out.append(item)
except Exception:
# Per-row isolation: one bad row never sinks the whole list.
logger.warning("longform library: skipping unparseable job row",
exc_info=True)
continue
return out
@router.get("/longform/jobs")
def longform_jobs(limit: int = Query(50, ge=1, le=500)) -> dict:
"""Finished Audiobook + Story renders, newest-first, ready to re-download.
Each item's ``output`` is served at ``/audio/<output>``. Never 500s — on any
backend hiccup it returns an empty list rather than an error.
"""
from core import job_store
jobs = build_longform_library(
job_store.list_jobs, job_store.events_since, limit=limit,
)
return {"jobs": jobs}
+28 -40
View File
@@ -58,31 +58,6 @@ MAX_BUNDLE_BYTES = 100 * 1024 * 1024
# ── Export ──────────────────────────────────────────────────────────────────
def _bundle_metadata(profile: dict, **extra) -> dict:
"""Common .omnivoice metadata for export + publish.
Captures ``kind`` and ``vd_states`` so a *designed* persona survives the
bundle round-trip as a design (not silently demoted to a clone) required
for the synthetic-only gating of the persona gallery (§R3). Old bundles
without these keys import as ``kind='clone'`` (backward-compatible).
"""
meta = {
"bundle_version": BUNDLE_VERSION,
"profile_name": profile.get("name", "Unnamed"),
"ref_text": profile.get("ref_text", ""),
"instruct": profile.get("instruct", ""),
"language": profile.get("language", "Auto"),
"personality": profile.get("personality", ""),
"seed": profile.get("seed"),
"kind": profile.get("kind") or "clone",
"vd_states": profile.get("vd_states"),
"is_locked": bool(profile.get("is_locked")),
"omnivoice_version": APP_VERSION,
}
meta.update(extra)
return meta
@router.post("/export/{profile_id}")
def export_profile(profile_id: str):
"""Export a voice profile as a downloadable .omnivoice bundle (ZIP)."""
@@ -100,9 +75,19 @@ def export_profile(profile_id: str):
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf:
# Metadata
metadata = _bundle_metadata(
profile, created_at=profile.get("created_at"), exported_at=time.time(),
)
metadata = {
"bundle_version": BUNDLE_VERSION,
"profile_name": profile.get("name", "Unnamed"),
"ref_text": profile.get("ref_text", ""),
"instruct": profile.get("instruct", ""),
"language": profile.get("language", "Auto"),
"personality": profile.get("personality", ""),
"seed": profile.get("seed"),
"is_locked": bool(profile.get("is_locked")),
"created_at": profile.get("created_at"),
"exported_at": time.time(),
"omnivoice_version": APP_VERSION,
}
zf.writestr("metadata.json", json.dumps(metadata, indent=2))
# Reference audio
@@ -206,9 +191,8 @@ async def import_profile(
conn.execute(
"""INSERT INTO voice_profiles
(id, name, ref_audio_path, ref_text, instruct, language,
seed, personality, is_locked, locked_audio_path, created_at,
kind, vd_states)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
seed, personality, is_locked, locked_audio_path, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(
profile_id,
metadata.get("profile_name", "Imported Voice"),
@@ -221,10 +205,6 @@ async def import_profile(
1 if is_locked else 0,
locked_audio_filename or "",
time.time(),
# Preserve the design/clone distinction across the round-trip;
# old bundles without these keys import as a clone.
metadata.get("kind") or "clone",
metadata.get("vd_states"),
),
)
@@ -273,11 +253,19 @@ def publish_to_marketplace(
# Build the bundle
with zipfile.ZipFile(str(bundle_path), "w", zipfile.ZIP_DEFLATED) as zf:
metadata = _bundle_metadata(
profile,
tags=[t.strip() for t in tags.split(",") if t.strip()],
published_at=time.time(),
)
metadata = {
"bundle_version": BUNDLE_VERSION,
"profile_name": profile.get("name", "Unnamed"),
"ref_text": profile.get("ref_text", ""),
"instruct": profile.get("instruct", ""),
"language": profile.get("language", "Auto"),
"personality": profile.get("personality", ""),
"seed": profile.get("seed"),
"is_locked": bool(profile.get("is_locked")),
"tags": [t.strip() for t in tags.split(",") if t.strip()],
"published_at": time.time(),
"omnivoice_version": APP_VERSION,
}
zf.writestr("metadata.json", json.dumps(metadata, indent=2))
ref_path = profile.get("ref_audio_path")
-53
View File
@@ -1,53 +0,0 @@
"""REST CRUD for per-agent MCP voice bindings (Wave 2.2 / Spec 2).
Loopback-gated the Settings UI manages bindings here. The MCP tools
themselves resolve voices via ``services.mcp_bindings.resolve_voice``.
"""
from __future__ import annotations
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from api.dependencies import require_loopback
from services import mcp_bindings
router = APIRouter(
prefix="/api/mcp",
tags=["mcp"],
dependencies=[Depends(require_loopback)],
)
class _BindingBody(BaseModel):
client_id: str = Field(..., min_length=1, max_length=128)
label: str | None = None
profile_id: str | None = None
default_engine: str | None = None
@router.get("/bindings")
def list_bindings():
"""All per-agent voice bindings, most-recently-seen first."""
return mcp_bindings.list_bindings()
@router.put("/bindings")
def upsert_binding(body: _BindingBody):
"""Create or update the binding for an MCP client id."""
try:
return mcp_bindings.upsert_binding(
body.client_id,
label=body.label,
profile_id=body.profile_id,
default_engine=body.default_engine,
)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@router.delete("/bindings/{client_id}")
def delete_binding(client_id: str):
if not mcp_bindings.delete_binding(client_id):
raise HTTPException(status_code=404, detail="No binding for that client id")
return {"deleted": client_id}
-85
View File
@@ -1,85 +0,0 @@
"""Media-tools endpoints — the backend for Settings → Audio tools and the
wizard's invisible media-engine self-heal.
Every route is loopback-gated: ``custom-path`` / ``use-system`` point the app
at an arbitrary executable (an RCE primitive if remote-reachable), and the
rest mutate local state. Same contract as ``/system/set-env``.
"""
from __future__ import annotations
import logging
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from api.dependencies import require_loopback
logger = logging.getLogger("omnivoice.api")
router = APIRouter(dependencies=[Depends(require_loopback)])
class CustomPathRequest(BaseModel):
path: str
def _svc():
# Late import so a service-level failure surfaces as a 500 with detail,
# not an app-boot failure.
from services import media_tools
return media_tools
@router.get("/media-tools/status")
def media_tools_status():
"""Per-tool {ok, path, version, origin} + background-op states."""
return _svc().status()
@router.post("/media-tools/acquire")
def media_tools_acquire():
"""(Re-)fetch the pinned, checksummed static ffmpeg/ffprobe build in the
background. Idempotent; poll /media-tools/status for progress."""
return _svc().acquire_bundled()
# Literal ytdlp routes MUST register before the parametrized {tool} routes —
# FastAPI matches in declaration order, and `/media-tools/{tool}/restore`
# would otherwise swallow `/media-tools/ytdlp/restore` into a 400.
@router.post("/media-tools/ytdlp/update")
def media_tools_ytdlp_update():
"""Fetch the newest yt-dlp wheel (sha256-verified against PyPI metadata)
into the update-surviving overlay. Applies on next backend start."""
return _svc().update_ytdlp()
@router.post("/media-tools/ytdlp/restore")
def media_tools_ytdlp_restore():
"""Drop the overlay — the app-tested, locked yt-dlp takes over on next
start. Always safe (the locked install is never modified)."""
return _svc().restore_ytdlp()
@router.post("/media-tools/{tool}/custom-path")
def media_tools_custom_path(tool: str, body: CustomPathRequest):
try:
return _svc().set_custom_path(tool, body.path)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
@router.post("/media-tools/{tool}/use-system")
def media_tools_use_system(tool: str):
try:
return _svc().use_system(tool)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
except LookupError as e:
raise HTTPException(status_code=404, detail=str(e))
@router.post("/media-tools/{tool}/restore")
def media_tools_restore(tool: str):
try:
return _svc().restore_bundled(tool)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
+12 -145
View File
@@ -22,6 +22,7 @@ from __future__ import annotations
import io
import logging
import os
import asyncio
import tempfile
from typing import Literal, Optional
@@ -29,7 +30,7 @@ from fastapi import APIRouter, File, Form, HTTPException, UploadFile
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, Field
from services.model_manager import _gpu_pool, run_on_gpu_pool_guarded
from services.model_manager import _gpu_pool
logger = logging.getLogger("omnivoice.openai_compat")
@@ -81,50 +82,6 @@ class SpeechRequest(BaseModel):
"E.g. 'young female, warm tone, slight British accent'.",
)
instruct: Optional[str] = Field(default=None, description="Style instruction for the TTS engine.")
duration: Optional[float] = Field(
default=None,
gt=0,
description="OmniVoice extension: target output duration in seconds.",
)
seed: Optional[int] = Field(
default=None,
description="OmniVoice extension: deterministic sampling seed.",
)
denoise: bool = Field(
default=True,
description="OmniVoice extension: prepend denoise control when supported.",
)
preprocess_prompt: bool = Field(
default=True,
description="OmniVoice extension: trim/preprocess reference prompt when supported.",
)
chunk_duration: Optional[float] = Field(
default=None,
ge=0,
description="OmniVoice GGUF extension: long-form internal chunk duration.",
)
chunk_threshold: Optional[float] = Field(
default=None,
ge=0,
description="OmniVoice GGUF extension: long-form internal chunk threshold.",
)
# #1014: these two were silently DISCARDED before (pydantic ignores
# undeclared fields) — a 200 OK that quietly dropped the caller's quality
# knobs. Declared now and passed through, matching the native /generate
# form fields (defaults there: num_step=16, guidance_scale=2.0; the
# model's documented "quality" preset is num_step=32).
num_step: Optional[int] = Field(
default=None,
ge=1,
le=128,
description="OmniVoice extension: iterative unmasking steps (app default 16; 32 = the model's documented quality preset).",
)
guidance_scale: Optional[float] = Field(
default=None,
gt=0,
le=20,
description="OmniVoice extension: classifier-free guidance scale (app default 2.0).",
)
class TranscriptionResponse(BaseModel):
@@ -252,14 +209,7 @@ def _run_tts(backend, text: str, kw: dict):
from services.audio_dsp import apply_mastering, normalize_audio
wav = backend.generate(text, **kw)
sr = backend.sample_rate
# Engines that already emit mastered, studio-grade audio (e.g. VoxCPM2's
# native 48 kHz) opt out of apply_mastering via `applies_own_mastering`.
# That chain's highpass + Compressor is tuned for OmniVoice's 24 kHz clone
# output; applied to a studio engine it adds an audible level pump that
# degrades the very output we want clean. Loudness normalisation still
# runs — it's a benign peak scale, not dynamics.
if not getattr(backend, "applies_own_mastering", False):
wav = apply_mastering(wav, sample_rate=sr)
wav = apply_mastering(wav, sample_rate=sr)
wav = normalize_audio(wav, target_dBFS=-2.0)
return wav, sr
@@ -269,32 +219,10 @@ async def create_speech(req: SpeechRequest):
"""Generate audio from text. Compatible with OpenAI's POST /v1/audio/speech."""
backend = _resolve_engine(req.model)
# Routing gate (#21 — no silent CPU fallback), identical to REST /generate.
from core.device_caps import detect_host_caps
from services.engine_routing import resolve_routing, routing_notice
_routing = resolve_routing(getattr(backend, "gpu_compat", ("cpu",)), detect_host_caps())
if _routing["routing_status"] == "unavailable":
raise HTTPException(status_code=400, detail=_routing["routing_reason"])
_routing_notice = routing_notice(_routing) # (status, reason) or None
# Build kwargs for the backend's generate() method
kw: dict = {
"speed": req.speed,
"denoise": req.denoise,
"preprocess_prompt": req.preprocess_prompt,
}
if req.duration is not None:
kw["duration"] = req.duration
if req.seed is not None:
kw["seed"] = req.seed
if req.chunk_duration is not None:
kw["chunk_duration"] = req.chunk_duration
if req.chunk_threshold is not None:
kw["chunk_threshold"] = req.chunk_threshold
if req.num_step is not None:
kw["num_step"] = req.num_step
if req.guidance_scale is not None:
kw["guidance_scale"] = req.guidance_scale
if req.language:
kw["language"] = req.language
if req.instruct:
@@ -323,8 +251,6 @@ async def create_speech(req: SpeechRequest):
kw["ref_text"] = row["ref_text"]
if row["instruct"] and not req.instruct:
kw["instruct"] = row["instruct"]
if req.seed is None and row["seed"] is not None:
kw["seed"] = row["seed"]
else:
# Not a profile ID — forward as engine preset name
kw["voice"] = voice
@@ -332,72 +258,22 @@ async def create_speech(req: SpeechRequest):
# Not a profile ID — might be a KittenTTS preset or similar
kw["voice"] = voice
# Engine-agnostic text normalization (junk strip, numbers→words,
# abbreviations) at this route's text→engine choke point — the same
# pre-pass as /generate, applied exactly once per request. `req.language`
# is everything this route knows about the language (None → universal
# safety filters only). Pref-gated (default ON), idempotent, never raises.
from services.text_normalization import normalize_for_tts
text = normalize_for_tts(req.input, req.language)
# ── #1033/#1037/#1014: warm the engine under the LOAD budget before the
# generate clock starts. The T4 verification (#1014) measured a fresh
# install's first /v1/audio/speech burning its whole 300s generate budget
# on the multi-GB checkpoint download (0% GPU util throughout) and dying
# with a misleading "too heavy for the available compute" error. Model
# loading gets OMNIVOICE_MODEL_LOAD_TIMEOUT (default 1200s); once warm
# this is a per-request no-op.
from services.model_manager import _model_load_timeout
try:
await run_on_gpu_pool_guarded(
backend.ensure_ready,
what=f"TTS engine '{backend.id}' model load",
timeout=_model_load_timeout(),
)
# Catch the BUILTIN TimeoutError base, not GpuJobTimeoutError by name:
# several tests reload services.model_manager mid-suite, so a class
# imported at call time can differ in identity from the one the guard
# (bound at this module's import) actually raises — the except would
# silently miss. The builtin base has one identity forever. (Caught by
# this exact test failing CI-only, in full-suite order.)
except TimeoutError as e:
logger.warning("engine load exceeded the model-load budget: %s", e)
raise HTTPException(
status_code=503,
detail=(
f"TTS engine '{backend.id}' did not finish loading within its "
f"model-load budget — on a first run this usually means the weight "
f"download is slow or stalled (check Settings → Models for "
f"progress), not that generation failed. Retry once the model "
f"shows as installed."
),
) from e
try:
# Bounded + pool-reset on hang so a wedged TTS request can't starve the
# GPU pool and brick the backend (#730 class).
wav, sr = await run_on_gpu_pool_guarded(
lambda: _run_tts(backend, text, kw), what="OpenAI TTS generate")
loop = asyncio.get_running_loop()
wav, sr = await loop.run_in_executor(_gpu_pool, _run_tts, backend, req.input, kw)
except Exception as e:
logger.exception("OpenAI TTS failed: %s", e)
raise HTTPException(status_code=500, detail=str(e))
audio_bytes, mime_type, ext = _encode_audio(wav, sr, req.response_format)
_headers = {
"Content-Length": str(len(audio_bytes)),
"Content-Disposition": f'inline; filename="speech.{ext}"',
}
if _routing_notice:
from services.engine_routing import header_safe_reason
_headers["X-OmniVoice-Routing"] = _routing_notice[0]
_hr = header_safe_reason(_routing_notice[1])
if _hr:
_headers["X-OmniVoice-Routing-Reason"] = _hr
return StreamingResponse(
io.BytesIO(audio_bytes),
media_type=mime_type,
headers=_headers,
headers={
"Content-Length": str(len(audio_bytes)),
"Content-Disposition": f'inline; filename="speech.{ext}"',
},
)
@@ -447,15 +323,12 @@ async def create_transcription(
try:
backend = get_active_asr_backend()
# Run transcription in the thread pool to avoid blocking the event loop,
# bounded so a stuck/starved ASR returns a 504 with guidance instead of
# hanging the request forever (see run_transcribe_guarded).
from services.asr_backend import run_transcribe_guarded
# Run transcription in the thread pool to avoid blocking the event loop
loop = asyncio.get_running_loop()
word_ts = response_format == "verbose_json"
result = await run_transcribe_guarded(
result = await loop.run_in_executor(
_gpu_pool,
lambda: backend.transcribe(tmp_path, word_timestamps=word_ts),
what="OpenAI",
)
# Extract the full text from segments
@@ -522,12 +395,6 @@ async def create_transcription(
# Default: json
return TranscriptionResponse(text=full_text)
except HTTPException:
raise
except TimeoutError as e:
# ASRTimeoutError (subclass): backend alive, ASR too heavy for compute.
logger.warning("OpenAI transcription timed out: %s", e)
raise HTTPException(status_code=504, detail=str(e))
except Exception as e:
logger.exception("OpenAI transcription failed: %s", e)
raise HTTPException(status_code=500, detail=str(e))
-332
View File
@@ -1,332 +0,0 @@
"""HTTP layer for the `.ovsvoice` portable persona format (#29 / parity §R3 G1).
Thin router over `services.persona_bundle`:
POST /personas/export/{profile_id} stream a downloadable .ovsvoice
POST /personas/import create a profile from a bundle
POST /personas/inspect read a bundle's manifest, no writes
Mirrors the legacy `.omnivoice` endpoints (`marketplace.py`) and reuses the
same path-confinement (`_voices_path`) + consent floor. `.ovsvoice` is additive;
`.omnivoice` import stays a compatible legacy reader.
"""
from __future__ import annotations
import asyncio
import functools
import logging
import os
import time
import uuid
from fastapi import APIRouter, File, HTTPException, Query, UploadFile
from fastapi.responses import StreamingResponse
from core import event_bus
from core.config import VOICES_DIR # noqa: F401 — re-exported for tests/monkeypatch
from core.db import db_conn
from core.version import APP_VERSION
from services import persona_bundle as pb
router = APIRouter()
logger = logging.getLogger("omnivoice.personas")
def _safe_name(name: str, profile_id: str) -> str:
"""Sanitised download filename stem (marketplace idiom); empty → persona_<id>."""
cleaned = "".join(
c if c.isalnum() or c in "-_ " else "" for c in (name or "")
).strip().replace(" ", "_")[:40]
return cleaned or f"persona_{profile_id}"
# ── Export ────────────────────────────────────────────────────────────────
@router.post("/personas/export/{profile_id}")
async def export_persona(
profile_id: str,
license_spdx: str = Query(pb.DEFAULT_LICENSE),
tags: str = Query(""),
include_reference: bool = Query(True),
):
"""Build + stream a `.ovsvoice` bundle for a profile."""
with db_conn() as conn:
row = conn.execute(
"SELECT * FROM voice_profiles WHERE id = ?", (profile_id,)
).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Voice profile not found")
profile = dict(row)
tag_list = [t.strip() for t in tags.split(",") if t.strip()]
# #693: if OMNIVOICE_MODEL is set, record the *resolved* checkpoint in the
# exported bundle so a leaked engine id (e.g. "omnivoice") can't be baked in;
# keep "" when unset (the bundle's "engine unspecified" marker).
from services.model_manager import resolve_omnivoice_checkpoint
engine_id = resolve_omnivoice_checkpoint() if os.environ.get("OMNIVOICE_MODEL", "").strip() else ""
try:
loop = asyncio.get_running_loop()
content = await loop.run_in_executor(
None,
functools.partial(
pb.build_persona_bundle,
profile,
license_spdx=license_spdx,
tags=tag_list,
include_reference=include_reference,
engine_id=engine_id,
omnivoice_version=APP_VERSION,
),
)
except pb.NoPreviewSource:
raise HTTPException(
status_code=503,
detail="This profile has no readable reference or locked audio to "
"build a preview from — re-create or re-import it.",
)
except Exception:
logger.exception("persona export failed for %s", profile_id)
raise HTTPException(
status_code=503,
detail="Could not build the persona bundle — see Settings → Logs.",
)
filename = f"{_safe_name(profile.get('name'), profile_id)}.ovsvoice"
from io import BytesIO
return StreamingResponse(
BytesIO(content),
media_type="application/zip",
headers={
"Content-Disposition": f'attachment; filename="{filename}"',
"Content-Length": str(len(content)),
},
)
# ── Import ────────────────────────────────────────────────────────────────
def _voices_dest(filename: str) -> str:
"""Resolve an output filename inside VOICES_DIR; 400 on escape (belt+braces —
the name is always server-generated `{profile_id}`)."""
from api.routers.profiles import _voices_path
path = _voices_path(filename)
if path is None:
raise HTTPException(status_code=400, detail="Invalid profile id")
return path
def _consent_verified(parsed: pb.ParsedPersona, consent_path: str | None) -> bool:
"""B12-B16: trust verified-own-voice ONLY with a real recording (≥ floor) AND
non-empty consent_text AND a consent.json present. The manifest flag alone
can't forge it."""
if not parsed.consent or not consent_path:
return False
if os.path.getsize(consent_path) < pb._MIN_CONSENT_AUDIO_BYTES:
return False
return bool((parsed.consent.get("consent_text") or "").strip())
@router.post("/personas/import")
async def import_persona(file: UploadFile = File(...)):
"""Create a new voice profile from a `.ovsvoice` (or legacy `.omnivoice`) bundle."""
name = (file.filename or "").lower()
if not name.endswith(".ovsvoice") and not name.endswith(".omnivoice"):
raise HTTPException(status_code=400, detail="File must be a .ovsvoice or .omnivoice bundle")
content = await file.read()
try:
parsed = pb.parse_persona_bundle(content)
except pb.BundleError as e:
raise HTTPException(status_code=e.status, detail=e.detail)
persona = parsed.manifest.get("persona") or {}
written: list[str] = []
def _gen_id() -> str:
return str(uuid.uuid4())[:8]
profile_id = _gen_id()
try:
# ── Audio members → server-named files (never the member name). ──
ref_filename = None
locked_filename = None
if "ref_audio" in parsed.members:
ref_filename = f"{profile_id}{parsed.member_ext('ref_audio')}"
dest = _voices_dest(ref_filename)
parsed.extract_member("ref_audio", dest); written.append(dest)
if "locked_audio" in parsed.members:
locked_filename = f"{profile_id}_locked{parsed.member_ext('locked_audio')}"
dest = _voices_dest(locked_filename)
parsed.extract_member("locked_audio", dest); written.append(dest)
# Preview-only bundle (A12/B8): use the preview as the usable ref clip.
if ref_filename is None and locked_filename is None and "preview" in parsed.members:
ref_filename = f"{profile_id}{parsed.member_ext('preview')}"
dest = _voices_dest(ref_filename)
parsed.extract_member("preview", dest); written.append(dest)
if ref_filename is None and locked_filename is None:
raise HTTPException(status_code=400, detail="bundle has no usable audio")
# ── Consent recording (optional) ──
consent_filename = None
consent_path = None
if "consent_audio" in parsed.members:
consent_filename = f"{profile_id}_consent{parsed.member_ext('consent_audio')}"
consent_path = _voices_dest(consent_filename)
parsed.extract_member("consent_audio", consent_path); written.append(consent_path)
verified = _consent_verified(parsed, consent_path)
consent_text = ((parsed.consent or {}).get("consent_text") or "").strip()
recorded_at = None
if verified:
try:
recorded_at = float(parsed.consent.get("recorded_at"))
except (TypeError, ValueError):
recorded_at = time.time()
is_locked = bool(persona.get("is_locked") and locked_filename)
ref_for_db = ref_filename or locked_filename # at least one is set
def _insert(pid: str):
with db_conn() as conn:
conn.execute(
"""INSERT INTO voice_profiles
(id, name, ref_audio_path, ref_text, instruct, language,
seed, personality, is_locked, locked_audio_path, created_at,
kind, vd_states,
verified_own_voice, consent_text, consent_audio_path, consent_recorded_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(
pid,
persona.get("name") or "Imported Voice",
ref_for_db,
persona.get("ref_text", ""),
persona.get("instruct", ""),
persona.get("language", "Auto"),
persona.get("seed"),
persona.get("personality", ""),
1 if is_locked else 0,
locked_filename or "",
time.time(),
persona.get("kind") or "clone",
persona.get("vd_states"),
1 if verified else 0,
# Keep the attestation text so the user can re-attest locally,
# even when imported unverified.
consent_text,
consent_filename if verified else "",
recorded_at if verified else None,
),
)
import sqlite3
try:
_insert(profile_id)
except sqlite3.IntegrityError:
profile_id = _gen_id() # one retry on id collision (B20)
# rename the on-disk files to the new id so they still match the row
written = _rename_for_new_id(written, profile_id)
ref_for_db = _retarget(ref_for_db, profile_id)
locked_filename = _retarget(locked_filename, profile_id)
consent_filename = _retarget(consent_filename, profile_id)
_insert(profile_id)
except HTTPException:
_cleanup(written)
raise
except Exception:
_cleanup(written)
logger.exception("persona import failed")
raise HTTPException(status_code=500, detail="Import failed; no files were kept.")
event_bus.emit("profiles", {"action": "created", "id": profile_id})
logger.info("Imported persona %r as %s (verified=%s)", persona.get("name"), profile_id, verified)
return {
"success": True,
"profile_id": profile_id,
"name": persona.get("name") or "Imported Voice",
"kind": persona.get("kind") or "clone",
"verified_own_voice": verified,
"preview_only": parsed.preview_only,
"license_spdx": parsed.license_spdx,
"watermarked_preview": parsed.watermarked_preview,
"source_bundle": file.filename,
"schema_version_ahead": parsed.schema_version_ahead,
}
def _cleanup(paths: list[str]) -> None:
for p in paths:
try:
if p and os.path.exists(p):
os.remove(p)
except OSError:
pass
def _rename_for_new_id(written: list[str], new_id: str) -> list[str]:
"""After an id-collision retry, rename each written file to carry the new id
(filenames are `{old_id}`; swap the leading 8-char stem)."""
out = []
for p in written:
d, base = os.path.split(p)
# base looks like {id}{ext} | {id}_locked{ext} | {id}_consent{ext}
new_base = new_id + base[8:]
new_path = os.path.join(d, new_base)
try:
os.replace(p, new_path)
out.append(new_path)
except OSError:
out.append(p)
return out
def _retarget(filename: str | None, new_id: str) -> str | None:
return new_id + filename[8:] if filename else filename
# ── Inspect (no-write preview) ──────────────────────────────────────────────
@router.post("/personas/inspect")
async def inspect_persona(file: UploadFile = File(...)):
"""Read a bundle's manifest + consent summary WITHOUT writing any file or row."""
name = (file.filename or "").lower()
if not name.endswith(".ovsvoice") and not name.endswith(".omnivoice"):
raise HTTPException(status_code=400, detail="File must be a .ovsvoice or .omnivoice bundle")
content = await file.read()
try:
parsed = pb.parse_persona_bundle(content)
except pb.BundleError as e:
raise HTTPException(status_code=e.status, detail=e.detail)
persona = parsed.manifest.get("persona") or {}
consent_summary = None
if parsed.consent:
has_recording = "consent_audio" in parsed.members
consent_summary = {
"verified_claimed": bool(parsed.consent.get("verified_own_voice")),
"method": parsed.consent.get("method", ""),
"has_recording": has_recording,
# would_verify mirrors import's gate, minus the byte-floor check
# (inspect never extracts to measure size — advisory only).
"would_verify": has_recording and bool((parsed.consent.get("consent_text") or "").strip()),
}
return {
"format": "omnivoice-legacy" if parsed.is_legacy else pb.OVSVOICE_FORMAT,
"schema_version": parsed.manifest.get("schema_version", pb.OVSVOICE_SCHEMA_VERSION),
"name": persona.get("name") or "Imported Voice",
"kind": persona.get("kind") or "clone",
"language": persona.get("language", "Auto"),
"personality": persona.get("personality", ""),
"is_locked": bool(persona.get("is_locked")),
"license_spdx": parsed.license_spdx,
"tags": parsed.manifest.get("tags") or [],
"preview_only": parsed.preview_only,
"watermarked_preview": parsed.watermarked_preview,
"consent": consent_summary,
"schema_version_ahead": parsed.schema_version_ahead,
}
+18 -299
View File
@@ -1,5 +1,4 @@
import os
import re
import uuid
import time
import shutil
@@ -12,7 +11,6 @@ from core.db import db_conn
from core.config import VOICES_DIR, OUTPUTS_DIR
from core import event_bus
from core.personalities import get_personalities
from omnivoice.utils.voice_design import heal_design_instruct, sanitize_instruct
router = APIRouter()
@@ -36,121 +34,29 @@ def list_profiles():
rows = conn.execute("SELECT * FROM voice_profiles ORDER BY created_at DESC").fetchall()
return [dict(r) for r in rows]
_DESIGN_SEED = 42 # deterministic sample render, same as archetype previews
@router.post("/profiles")
async def create_profile(
name: str = Form(...),
ref_audio: Optional[UploadFile] = File(None),
ref_audio: UploadFile = File(...),
ref_text: str = Form(""),
instruct: str = Form(""),
language: str = Form("Auto"),
seed: Optional[int] = Form(None),
personality: str = Form(""),
kind: str = Form("clone"),
vd_states: Optional[str] = Form(None),
):
"""Create a voice profile (spec: docs/specs/voice-studio-unification.md §5).
kind='clone' requires `ref_audio` (the user's reference recording).
kind='design' requires `vd_states` (JSON of category picks); the server
renders a deterministic sample WAV (seed 42, same path as
archetype materialization) and stores it as the profile's
reference so the voice identity is stable across runs.
"""
if kind not in ("clone", "design"):
raise HTTPException(status_code=422, detail="kind must be 'clone' or 'design'")
if kind == "clone" and ref_audio is None:
raise HTTPException(status_code=422, detail="clone profiles require ref_audio")
if kind == "design":
if not (vd_states or "").strip():
raise HTTPException(status_code=422, detail="design profiles require vd_states")
import json as _json
try:
parsed = _json.loads(vd_states)
if not isinstance(parsed, dict):
raise ValueError("not an object")
except ValueError:
raise HTTPException(status_code=422, detail="vd_states must be a JSON object")
# Root-cause close for #983: a design profile must never be PERSISTED
# with a partial vd_states shape, regardless of which client (older
# frontend build, hand-edited payload, third-party API caller) created
# it — a missing category key crashes DesignMethodPanel's render on
# every future client that selects this profile. CATEGORY_ORDER is the
# same single source of truth the frontend's CATEGORIES keys mirror
# (core/describe_voice.py), so this can't drift from the picker.
from core.describe_voice import CATEGORY_ORDER
for _cat in CATEGORY_ORDER:
parsed.setdefault(_cat, "Auto")
vd_states = _json.dumps(parsed)
# An all-Auto design (every category left on "Auto") yields an empty
# instruct — that's still a valid, saveable voice: synthesis falls back
# to neutral instruct-only conditioning (see generation.py design path).
# Don't gate save on a non-empty instruct.
#
# Defence-in-depth against the "[object Object]" / freeform-prose poison
# (#550 #571 #594 #596): never persist an instruct the engine validator
# would reject. Sanitize the submitted instruct and, if it's unusable,
# rebuild the tags from vd_states — so the row is always generation-safe
# regardless of which frontend build saved it.
instruct = heal_design_instruct(instruct, parsed)
profile_id = str(uuid.uuid4())[:8]
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
audio_filename = f"{profile_id}{ext}"
audio_path = os.path.join(VOICES_DIR, audio_filename)
if kind == "clone":
ext = os.path.splitext(ref_audio.filename or ".wav")[1]
audio_filename = f"{profile_id}{ext}"
audio_path = os.path.join(VOICES_DIR, audio_filename)
with open(audio_path, "wb") as f:
f.write(await ref_audio.read())
used_seed = seed
else:
# Saving a design profile is a pure persistence operation — it must not
# depend on a loaded TTS model (issue #476: on a fresh model-less Docker
# image the render forced a full model load + inference that 503'd, so
# the save failed). We try the deterministic identity sample opportunist-
# ically through the one shared TTS path (archetypes' renderer, never a
# second inference code path); if the engine isn't ready it's rendered
# lazily on first preview/use. The row carries vd_states + instruct, so
# the voice is fully usable without the sample (synthesis falls back to
# instruct-only conditioning — see generation.py's design path).
from pathlib import Path
from api.routers.archetypes import _render_archetype_wav
audio_filename = f"{profile_id}.wav"
audio_path = os.path.join(VOICES_DIR, audio_filename)
try:
await _render_archetype_wav(
{
"language": language,
"sample_script": ref_text, # optional custom sample line
"instruct": instruct,
},
Path(audio_path),
)
except Exception:
# Engine unavailable / OOM / inference failure — defer the sample.
# Store the row with no ref_audio_path; the identity sample is
# rendered on first preview or use. Never let this block the save.
import logging
logging.getLogger("omnivoice.profiles").info(
"Design profile %s saved with sample pending — "
"voice engine not ready; will render on first use", profile_id,
)
if os.path.exists(audio_path): # partial/blank render: don't keep it
with __import__("contextlib").suppress(OSError):
os.remove(audio_path)
audio_filename = None
used_seed = seed if seed is not None else _DESIGN_SEED
with open(audio_path, "wb") as f:
f.write(await ref_audio.read())
try:
with db_conn() as conn:
conn.execute(
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, "
"language, seed, personality, kind, vd_states, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
(profile_id, name, audio_filename, ref_text, instruct, language,
used_seed, personality, kind, vd_states, time.time())
"INSERT INTO voice_profiles (id, name, ref_audio_path, ref_text, instruct, language, seed, personality, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(profile_id, name, audio_filename, ref_text, instruct, language, seed, personality, time.time())
)
except Exception:
# Clean up orphaned audio file if DB insert fails
@@ -158,7 +64,7 @@ async def create_profile(
os.remove(audio_path)
raise
event_bus.emit("profiles", {"action": "created", "id": profile_id})
return {"id": profile_id, "name": name, "kind": kind}
return {"id": profile_id, "name": name}
@router.get("/profiles/{profile_id}")
def get_profile(profile_id: str):
@@ -186,10 +92,6 @@ def update_profile(profile_id: str, patch: ProfileUpdate):
continue
if col == "name" and not val.strip():
raise HTTPException(status_code=400, detail="A voice profile needs a name.")
if col == "instruct":
# Never let an edit persist a validator-rejecting instruct (prose /
# "[object Object]"); keep only whitelist tags (#550 #571 #594 #596).
val = sanitize_instruct(val)
fields.append(f"{col} = ?")
params.append(val.strip() if col in ("name", "language") else val)
if not fields:
@@ -261,96 +163,20 @@ def get_profile_usage(profile_id: str):
}
# profile_id is a request path param and the audio filename derives from it, so
# constrain it to the generated-id charset (no separators / `..` possible) before
# any path use, and read only a *direct child* of VOICES_DIR — os.path.basename()
# strips any directory component (a path-injection / CWE-22 barrier).
_PROFILE_ID_RE = re.compile(r"[A-Za-z0-9_-]{1,64}")
@router.get("/profiles/{profile_id}/audio")
async def get_profile_audio(profile_id: str):
if not _PROFILE_ID_RE.fullmatch(profile_id or ""):
return Response("Profile not found", status_code=404)
def get_profile_audio(profile_id: str):
with db_conn() as conn:
row = conn.execute(
"SELECT ref_audio_path, locked_audio_path, kind, instruct, language, ref_text "
"FROM voice_profiles WHERE id=?",
(profile_id,),
).fetchone()
row = conn.execute("SELECT ref_audio_path, locked_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
if not row:
return Response("Profile not found", status_code=404)
audio_file = row["locked_audio_path"] or row["ref_audio_path"]
if not audio_file:
# A design profile saved before the engine was ready (issue #476) has no
# identity sample yet. Render it lazily now — the deterministic seed-42
# sample is reproducible, so a deferred render matches a save-time one.
rendered = await _materialize_design_sample(profile_id, row)
if rendered is None:
return Response("No audio available", status_code=404)
audio_file = rendered
# CWE-22: resolve the DB-stored filename strictly inside VOICES_DIR via the
# shared guard — _voices_path() applies the os.path.basename() barrier plus
# symlink-resolved containment (same path the consent endpoint trusts).
audio_path = _voices_path(str(audio_file))
if audio_path is None or not os.path.exists(audio_path):
return Response("No audio available", status_code=404)
audio_path = os.path.join(VOICES_DIR, audio_file)
if not os.path.exists(audio_path):
return Response("Audio file missing", status_code=404)
return FileResponse(audio_path, media_type="audio/wav")
async def _materialize_design_sample(profile_id: str, row) -> Optional[str]:
"""Render a design profile's pending identity sample on first request.
Returns the stored filename on success, or None if this isn't a renderable
design row. Raises HTTPException(503) with a precise "model not ready"
message if the engine is genuinely unavailable saving never depends on
this, but a user who explicitly asks for the sample gets a clear signal.
"""
try:
kind = row["kind"]
except (KeyError, IndexError):
kind = "clone"
if kind != "design":
return None
from pathlib import Path
from api.routers.archetypes import _render_archetype_wav
audio_filename = f"{profile_id}.wav"
# CWE-22: resolve under VOICES_DIR via the shared basename + containment
# guard before rendering (rejects any escape).
audio_path = _voices_path(audio_filename)
if audio_path is None:
raise HTTPException(status_code=400, detail="invalid profile identifier")
try:
await _render_archetype_wav(
{
"language": row["language"] or "Auto",
"sample_script": row["ref_text"] or "",
"instruct": row["instruct"] or "",
},
Path(audio_path),
)
except Exception as e:
with __import__("contextlib").suppress(OSError):
if os.path.exists(audio_path):
os.remove(audio_path)
raise HTTPException(
status_code=503,
detail=(
"The voice engine isn't ready yet, so this designed voice's "
"preview sample can't be rendered. Finish setup / download a "
f"model, then try again. ({e})"
),
)
with db_conn() as conn:
conn.execute(
"UPDATE voice_profiles SET ref_audio_path=? WHERE id=?",
(audio_filename, profile_id),
)
return audio_filename
@router.post("/profiles/{profile_id}/lock")
async def lock_profile(
profile_id: str,
@@ -408,122 +234,15 @@ async def unlock_profile(profile_id: str):
event_bus.emit("profiles", {"action": "unlocked", "id": profile_id})
return {"unlocked": True, "profile_id": profile_id}
# ── Consent lock (parity program Wave 0.2) ─────────────────────────────────
#
# A profile becomes "verified own voice" when its owner records themselves
# reading a consent statement. The recording is provenance, not a voiceprint
# check — agentic features and gallery sharing gate on the flag; plain local
# synthesis never does. Spec: docs/competitive-analysis.md Action 22.
_MIN_CONSENT_AUDIO_BYTES = 1000 # same floor as the frontend recorder
# Upload filename extension whitelist — anything else falls back to .wav so a
# crafted filename can never influence the on-disk path (py/path-injection).
_CONSENT_EXT_RE = re.compile(r"^\.[A-Za-z0-9]{1,8}$")
def _voices_path(filename: str) -> Optional[str]:
"""Resolve a DB-stored audio filename strictly inside VOICES_DIR.
Rejects anything that isn't a bare filename or that escapes the voices
directory after symlink resolution. Returns None instead of raising so
cleanup paths can simply skip bad values.
"""
if not filename or os.path.basename(filename) != filename:
return None
root = os.path.realpath(VOICES_DIR)
path = os.path.realpath(os.path.join(root, filename))
if not path.startswith(root + os.sep):
return None
return path
@router.post("/profiles/{profile_id}/consent")
async def record_consent(
profile_id: str,
consent_audio: UploadFile = File(...),
consent_text: str = Form(...),
):
if not consent_text.strip():
raise HTTPException(status_code=422, detail="consent_text must not be empty")
data = await consent_audio.read()
if len(data) < _MIN_CONSENT_AUDIO_BYTES:
raise HTTPException(status_code=422, detail="consent recording is too short")
with db_conn() as conn:
row = conn.execute(
"SELECT id, consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)
).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Profile not found")
ext = os.path.splitext(consent_audio.filename or "")[1]
if not _CONSENT_EXT_RE.match(ext):
ext = ".wav"
audio_filename = f"{profile_id}_consent{ext}"
audio_path = _voices_path(audio_filename)
if audio_path is None: # profile_id is server-generated; this is belt+braces
raise HTTPException(status_code=400, detail="Invalid profile id")
with open(audio_path, "wb") as f:
f.write(data)
# A re-record may change the extension; drop the superseded file.
old = row["consent_audio_path"]
if old and old != audio_filename:
old_path = _voices_path(old)
if old_path and os.path.exists(old_path):
os.remove(old_path)
recorded_at = time.time()
try:
with db_conn() as conn:
conn.execute(
"UPDATE voice_profiles SET verified_own_voice=1, consent_text=?, "
"consent_audio_path=?, consent_recorded_at=? WHERE id=?",
(consent_text.strip(), audio_filename, recorded_at, profile_id),
)
except Exception:
if os.path.exists(audio_path):
os.remove(audio_path)
raise
event_bus.emit("profiles", {"action": "consent_recorded", "id": profile_id})
return {
"id": profile_id,
"verified_own_voice": True,
"consent_recorded_at": recorded_at,
}
@router.delete("/profiles/{profile_id}/consent")
def revoke_consent(profile_id: str):
with db_conn() as conn:
row = conn.execute(
"SELECT consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)
).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Profile not found")
conn.execute(
"UPDATE voice_profiles SET verified_own_voice=0, consent_text='', "
"consent_audio_path='', consent_recorded_at=NULL WHERE id=?",
(profile_id,),
)
if row["consent_audio_path"]:
path = _voices_path(row["consent_audio_path"])
if path and os.path.exists(path):
os.remove(path)
event_bus.emit("profiles", {"action": "consent_revoked", "id": profile_id})
return {"id": profile_id, "verified_own_voice": False}
@router.delete("/profiles/{profile_id}")
def delete_profile(profile_id: str):
with db_conn() as conn:
row = conn.execute("SELECT ref_audio_path, locked_audio_path, consent_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
row = conn.execute("SELECT ref_audio_path, locked_audio_path FROM voice_profiles WHERE id=?", (profile_id,)).fetchone()
if row:
for col in ["ref_audio_path", "locked_audio_path", "consent_audio_path"]:
for col in ["ref_audio_path", "locked_audio_path"]:
if row[col]:
path = _voices_path(row[col])
if path and os.path.exists(path):
path = os.path.join(VOICES_DIR, row[col])
if os.path.exists(path):
os.remove(path)
# Prevent FOREIGN KEY constraint failure
conn.execute("UPDATE generation_history SET profile_id = NULL WHERE profile_id=?", (profile_id,))
-25
View File
@@ -2,7 +2,6 @@ import uuid
import time
import json
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
from core.db import db_conn
from core import event_bus
@@ -10,10 +9,6 @@ from schemas.requests import ProjectSaveRequest
router = APIRouter()
class ProjectRenameRequest(BaseModel):
name: str
@router.get("/projects")
async def list_projects():
with db_conn() as conn:
@@ -64,26 +59,6 @@ async def update_project(project_id: str, req: ProjectSaveRequest):
event_bus.emit("projects", {"action": "updated", "id": project_id})
return {"id": project_id, "name": req.name, "updated_at": now}
@router.patch("/projects/{project_id}")
async def rename_project(project_id: str, req: ProjectRenameRequest):
"""Lightweight rename — updates only the project name (and updated_at),
without re-serialising the whole state blob like PUT does."""
name = req.name.strip()
if not name:
raise HTTPException(status_code=400, detail="Project name cannot be empty")
now = time.time()
with db_conn() as conn:
row = conn.execute("SELECT id FROM studio_projects WHERE id=?", (project_id,)).fetchone()
if not row:
raise HTTPException(status_code=404, detail="Project not found")
conn.execute(
"UPDATE studio_projects SET name=?, updated_at=? WHERE id=?",
(name, now, project_id),
)
event_bus.emit("projects", {"action": "renamed", "id": project_id})
return {"id": project_id, "name": name, "updated_at": now}
@router.delete("/projects/{project_id}")
async def delete_project(project_id: str):
with db_conn() as conn:
-306
View File
@@ -1,306 +0,0 @@
"""
Pronunciation dictionary router Expressive-TTS Spec 01 Phase 1.
CRUD for the DB-backed, per-language pronunciation dictionary the
``PronunciationPanel`` (Settings Pronunciation) edits, plus a model-free
``/pronunciation/test`` dry-run. Entries are applied as pure text substitution
before synthesis (see ``services/pronunciation.apply_pronunciation`` and the
generate path), so a saved entry actually changes the audio on every engine.
Endpoints (loopback-only, like the dictation router):
GET /pronunciation list every entry
POST /pronunciation create one entry
PUT /pronunciation/{entry_id} update an entry (partial)
DELETE /pronunciation/{entry_id} remove an entry
POST /pronunciation/test dry-run substitution (no model)
GET /pronunciation/export all entries as JSON (round-trips import)
POST /pronunciation/import bulk add entries from JSON
Scope: ``language='*'`` is global (applies to every request); a 2-letter code
(``'en'``, ``'de'``) applies only when the request language matches.
"""
from __future__ import annotations
import logging
import re
import time
import uuid
from typing import List, Optional
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from api.dependencies import require_loopback
from core.db import db_conn
from services.pronunciation import apply_pronunciation, entries_for_language
logger = logging.getLogger("omnivoice.pronunciation")
router = APIRouter()
_VALID_TYPES = ("respelling", "ipa", "cmu")
_ALL_LANG = "*"
# IPA: the input is validated as a non-empty string of Unicode letters / IPA
# extension codepoints + the usual suprasegmental marks; we reject ASCII control
# and the bracket/pipe chars that would collide with the inline grammar. This is
# a charset gate (catches obvious garbage early), not a full IPA grammar.
_IPA_BAD = re.compile(r"[\[\]\|\x00-\x1f]")
# CMU / ARPABET: space-separated phoneme tokens (letters + an optional 0-2 stress
# digit), e.g. "N AH0 V AE1 D AH0". Reject anything else.
_CMU_TOKEN = re.compile(r"^[A-Za-z]{1,3}[0-2]?$")
def _validate_type_replacement(etype: str, replacement: str) -> None:
"""Raise 400 on a phoneme replacement that's obviously malformed.
Respelling rows accept any text. IPA rows must be a non-empty string free of
bracket/pipe/control chars. CMU rows must be space-separated ARPABET tokens.
Validating on save (not at synth) means a model never sees garbage phonemes
(Spec 01 §R3 never pass unvalidated phoneme strings to a model).
"""
if etype == "respelling":
return
rep = (replacement or "").strip()
if not rep:
raise HTTPException(
status_code=400,
detail=f"A {etype.upper()} entry needs a phoneme string in 'replacement'.",
)
if etype == "ipa":
if _IPA_BAD.search(rep):
raise HTTPException(
status_code=400,
detail="That IPA string contains brackets, a pipe, or control characters. "
"Use plain IPA symbols, e.g. ˈnɛvʌdə.",
)
elif etype == "cmu":
tokens = rep.split()
if not tokens or any(not _CMU_TOKEN.match(tok) for tok in tokens):
raise HTTPException(
status_code=400,
detail="That doesn't look like CMU/ARPABET. Use space-separated tokens with "
"optional stress digits, e.g. N AH0 V AE1 D AH0.",
)
def _norm_language(language: Optional[str]) -> str:
"""Normalize a scope to '*' (global) or a lowercase 2-letter code."""
if not language:
return _ALL_LANG
s = str(language).strip()
if not s or s == _ALL_LANG or s.lower() == "auto":
return _ALL_LANG
return s.lower()[:2]
def _row_to_dict(r) -> dict:
d = dict(r)
d["enabled"] = bool(d.get("enabled"))
# ``scope`` is the UI-facing alias for ``language`` ('*' shows as Global).
d["scope"] = d.get("language") or _ALL_LANG
return d
# ── Schemas ──────────────────────────────────────────────────────────────────
class PronEntry(BaseModel):
term: str
replacement: str = ""
type: str = "respelling"
language: str = _ALL_LANG
enabled: bool = True
class PronEntryUpdate(BaseModel):
term: Optional[str] = None
replacement: Optional[str] = None
type: Optional[str] = None
language: Optional[str] = None
enabled: Optional[bool] = None
class PronTestRequest(BaseModel):
text: str
language: Optional[str] = None
class PronImportRequest(BaseModel):
entries: List[PronEntry]
replace: bool = False # True → clear existing rows first
# ── CRUD ─────────────────────────────────────────────────────────────────────
@router.get("/pronunciation", dependencies=[Depends(require_loopback)])
def list_entries():
with db_conn() as conn:
rows = conn.execute(
"SELECT id, term, replacement, type, language, enabled, created_at "
"FROM pronunciation_entries ORDER BY created_at ASC, id ASC"
).fetchall()
return [_row_to_dict(r) for r in rows]
@router.post("/pronunciation", dependencies=[Depends(require_loopback)])
def create_entry(entry: PronEntry):
term = entry.term.strip()
if not term:
raise HTTPException(status_code=400, detail="A pronunciation entry needs a term.")
etype = (entry.type or "respelling").strip().lower()
if etype not in _VALID_TYPES:
raise HTTPException(
status_code=400,
detail=f"Unknown entry type {entry.type!r}. Use one of: {', '.join(_VALID_TYPES)}.",
)
_validate_type_replacement(etype, entry.replacement)
eid = str(uuid.uuid4())[:12]
now = time.time()
lang = _norm_language(entry.language)
with db_conn() as conn:
conn.execute(
"INSERT INTO pronunciation_entries (id, term, replacement, type, language, enabled, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?)",
(eid, term, entry.replacement, etype, lang, 1 if entry.enabled else 0, now),
)
row = conn.execute(
"SELECT id, term, replacement, type, language, enabled, created_at "
"FROM pronunciation_entries WHERE id = ?", (eid,)
).fetchone()
return _row_to_dict(row)
@router.put("/pronunciation/{entry_id}", dependencies=[Depends(require_loopback)])
def update_entry(entry_id: str, patch: PronEntryUpdate):
with db_conn() as conn:
existing = conn.execute(
"SELECT id, term, replacement, type, language, enabled, created_at "
"FROM pronunciation_entries WHERE id = ?", (entry_id,)
).fetchone()
if existing is None:
raise HTTPException(status_code=404, detail="No such pronunciation entry.")
# Resolve the post-update type + replacement so phoneme validation runs
# against the final state (e.g. switching type without changing text).
new_type = (patch.type.strip().lower() if patch.type is not None else existing["type"]) or "respelling"
if new_type not in _VALID_TYPES:
raise HTTPException(
status_code=400,
detail=f"Unknown entry type {patch.type!r}. Use one of: {', '.join(_VALID_TYPES)}.",
)
new_replacement = patch.replacement if patch.replacement is not None else existing["replacement"]
_validate_type_replacement(new_type, new_replacement)
fields, params = [], []
if patch.term is not None:
term = patch.term.strip()
if not term:
raise HTTPException(status_code=400, detail="A pronunciation entry needs a term.")
fields.append("term = ?"); params.append(term)
if patch.replacement is not None:
fields.append("replacement = ?"); params.append(patch.replacement)
if patch.type is not None:
fields.append("type = ?"); params.append(new_type)
if patch.language is not None:
fields.append("language = ?"); params.append(_norm_language(patch.language))
if patch.enabled is not None:
fields.append("enabled = ?"); params.append(1 if patch.enabled else 0)
if not fields:
raise HTTPException(
status_code=400,
detail="PUT body was empty. Include at least one field to change, or DELETE the entry.",
)
params.append(entry_id)
# nosec B608 - `fields` are fixed literal assignments ("term = ?", …) from
# the allowlist above; every user value is a bound `?` parameter, never
# interpolated. The f-string only joins constant column fragments.
conn.execute(
f"UPDATE pronunciation_entries SET {', '.join(fields)} WHERE id = ?", # nosec B608
params,
)
row = conn.execute(
"SELECT id, term, replacement, type, language, enabled, created_at "
"FROM pronunciation_entries WHERE id = ?", (entry_id,)
).fetchone()
return _row_to_dict(row)
@router.delete("/pronunciation/{entry_id}", dependencies=[Depends(require_loopback)])
def delete_entry(entry_id: str):
with db_conn() as conn:
cur = conn.execute("DELETE FROM pronunciation_entries WHERE id = ?", (entry_id,))
return {"deleted": cur.rowcount > 0}
# ── Dry-run + import/export ───────────────────────────────────────────────────
@router.post("/pronunciation/test", dependencies=[Depends(require_loopback)])
def test_substitution(req: PronTestRequest):
"""Show the post-substitution text for ``req.text`` — no model call.
Applies the same dictionary + inline ``[[]]`` resolution the synth path
runs, so the user sees exactly what the engine will be handed.
"""
with db_conn() as conn:
rows = conn.execute(
"SELECT id, term, replacement, type, language, enabled, created_at "
"FROM pronunciation_entries"
).fetchall()
substituted = apply_pronunciation(req.text, rows, req.language)
applied = entries_for_language(rows, req.language)
return {
"input": req.text,
"substituted": substituted,
"changed": substituted != req.text,
"applied_terms": sorted(applied.keys(), key=len, reverse=True),
}
@router.get("/pronunciation/export", dependencies=[Depends(require_loopback)])
def export_entries():
"""Every entry as a JSON-serializable list (round-trips ``/import``)."""
with db_conn() as conn:
rows = conn.execute(
"SELECT term, replacement, type, language, enabled "
"FROM pronunciation_entries ORDER BY created_at ASC, id ASC"
).fetchall()
return {"entries": [
{"term": r["term"], "replacement": r["replacement"], "type": r["type"],
"language": r["language"], "enabled": bool(r["enabled"])}
for r in rows
]}
@router.post("/pronunciation/import", dependencies=[Depends(require_loopback)])
def import_entries(req: PronImportRequest):
"""Bulk-add entries. ``replace=true`` clears the table first.
Each entry is validated like ``POST /pronunciation``; one bad row fails the
whole import (400) so the table is never left half-applied.
"""
now = time.time()
cleaned = []
for e in req.entries:
term = e.term.strip()
if not term:
continue # silently skip blank terms — they're a no-op anyway
etype = (e.type or "respelling").strip().lower()
if etype not in _VALID_TYPES:
raise HTTPException(
status_code=400,
detail=f"Entry {term!r}: unknown type {e.type!r}.",
)
_validate_type_replacement(etype, e.replacement)
cleaned.append((str(uuid.uuid4())[:12], term, e.replacement, etype,
_norm_language(e.language), 1 if e.enabled else 0, now))
with db_conn() as conn:
if req.replace:
conn.execute("DELETE FROM pronunciation_entries")
conn.executemany(
"INSERT INTO pronunciation_entries (id, term, replacement, type, language, enabled, created_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?)",
cleaned,
)
return {"imported": len(cleaned), "replaced": req.replace}
+2 -624
View File
@@ -12,7 +12,6 @@ The state endpoint duplicates `/system/hf-token/state` (which lives on
"""
from __future__ import annotations
import asyncio
import logging
import os
from dataclasses import asdict
@@ -77,17 +76,8 @@ def clear_hf_token(also_clear_hf_cli: bool = Query(False)):
@router.get("/hf-token/state")
def get_hf_token_state(fresh: bool = Query(False)):
"""3-source HF token cascade state for the Settings UI.
``fresh=1`` drops the resolver's whoami validation cache first so the
response re-runs whoami for every source this is what the panel's
"Test now" button sends. Plain GETs (panel mounts) keep the 300s cache
so repeat Settings visits don't hammer the HF API.
"""
from services import token_resolver
if fresh:
token_resolver.invalidate_cache()
def get_hf_token_state():
"""3-source HF token cascade state for the Settings UI."""
return _state_response()
@@ -133,409 +123,6 @@ def set_torch_compile_disabled(body: _TorchCompileBody):
return _torch_compile_state()
# ── Generation-history retention (Studio takes rail) ──────────────────────
class _HistoryRetentionBody(BaseModel):
cap: int = Field(
...,
ge=0,
le=100000,
description="Max takes kept before the oldest UNstarred ones (rows + WAVs) are pruned; 0 = unlimited",
)
def _history_retention_state() -> dict:
from api.routers.generation import DEFAULT_HISTORY_CAP, _history_cap
return {"cap": _history_cap(), "default": DEFAULT_HISTORY_CAP}
@router.get("/history-retention")
def get_history_retention():
"""Current generation-history retention cap (Settings → Storage)."""
return _history_retention_state()
@router.put("/history-retention")
def set_history_retention(body: _HistoryRetentionBody):
"""Persist the retention cap. Enforced after every generation: the oldest
unstarred takes over the cap are pruned (rows + their audio files);
starred takes are never pruned. 0 disables pruning entirely."""
from core import prefs
from api.routers.generation import HISTORY_CAP_PREF_KEY
try:
prefs.set_(HISTORY_CAP_PREF_KEY, int(body.cap))
except Exception:
logger.exception("set_history_retention failed")
raise HTTPException(status_code=500, detail="Failed to persist setting")
return _history_retention_state()
# ── Dictation refinement (parity program Wave 2.1 / Spec 3 phase 2) ───────
class _RefinementBody(BaseModel):
auto: bool | None = None
smart_cleanup: bool | None = None
self_correction: bool | None = None
preserve_technical: bool | None = None
def _refinement_state():
from services.refinement import (
_skill_llm,
get_last_refine_status,
get_refinement_config,
)
cfg = get_refinement_config()
# `llm_ready` only means "an endpoint is CONFIGURED" — a placeholder/dead
# endpoint still reads ready. It's resolved through the LLM Skills registry
# so a disabled dictation_refinement skill / per-skill provider override
# reads the same here as on the actual refine path. The honesty layer is
# `last_refine_status`: {ok, reason, at} from the most recent final, so the
# panel can flag a configured-but-failing LLM (the real safety is the hard
# refine timeout, which keeps a dead endpoint from ever stalling the final).
cfg["llm_ready"] = _skill_llm().id != "off"
cfg["last_refine_status"] = get_last_refine_status()
return cfg
@router.get("/dictation-refinement")
def get_dictation_refinement():
"""Current refinement config + whether an LLM backend is configured."""
return _refinement_state()
@router.put("/dictation-refinement")
def set_dictation_refinement(body: _RefinementBody):
from services.refinement import set_refinement_config
try:
set_refinement_config({k: v for k, v in body.model_dump().items() if v is not None})
except Exception:
logger.exception("set_dictation_refinement failed")
raise HTTPException(status_code=500, detail="Failed to persist setting")
return _refinement_state()
# ── LLM endpoint (parity program Wave 2.4 / §R2 rung 4) ───────────────────
# Focused configuration for the OpenAI-compatible LLM endpoint that powers
# cinematic translate, glossary auto-extract, and dictation refinement.
# Persistence rides the existing TRANSLATE_BASE_URL / TRANSLATE_API_KEY /
# TRANSLATE_MODEL env vars (already in system.py PERSISTENT_KEYS, restored
# at startup) so the resolution path in llm_backend/translator is unchanged.
class _LLMEndpointBody(BaseModel):
base_url: str | None = None
model: str | None = None
api_key: str | None = None # None = leave unchanged; "" = clear
def _mask(secret: str | None) -> str | None:
if not secret:
return None
return f"{secret[-4:]}" if len(secret) > 4 else "set"
def _llm_endpoint_state():
from services.llm_backend import OpenAICompatBackend
ok, reason = OpenAICompatBackend.is_available()
return {
"base_url": os.environ.get("TRANSLATE_BASE_URL", ""),
"model": os.environ.get("TRANSLATE_MODEL", ""),
"api_key_masked": _mask(
os.environ.get("TRANSLATE_API_KEY") or os.environ.get("OPENAI_API_KEY")
),
"available": ok,
"reason": None if ok else reason,
}
@router.get("/llm-endpoint")
def get_llm_endpoint():
"""Current OpenAI-compatible LLM endpoint config + live availability."""
return _llm_endpoint_state()
@router.put("/llm-endpoint")
def set_llm_endpoint(body: _LLMEndpointBody):
"""Persist base URL / model / API key for the OpenAI-compatible endpoint.
Reuses the env-var persistence path (prefs.json, restored at startup):
base_url -> TRANSLATE_BASE_URL, model -> TRANSLATE_MODEL,
api_key -> TRANSLATE_API_KEY. A None field is left unchanged; an empty
string clears it. Ollama ignores the key; vLLM / LM Studio require it.
"""
from core.prefs import set_ as prefs_set, delete as prefs_delete
mapping = {
"TRANSLATE_BASE_URL": body.base_url,
"TRANSLATE_MODEL": body.model,
"TRANSLATE_API_KEY": body.api_key,
}
for env_key, val in mapping.items():
if val is None:
continue # untouched
val = val.strip()
if val:
os.environ[env_key] = val
prefs_set(f"env.{env_key}", val)
else:
os.environ.pop(env_key, None)
prefs_delete(f"env.{env_key}")
# get_active_llm_backend() builds a fresh backend (and its OpenAI client
# reads env at construction) on every call, so there's no singleton to
# invalidate — the next translate/refine picks up the new values.
return _llm_endpoint_state()
# ── Multi-provider LLM registry (Settings → LLM Providers) ────────────────
# Keys persist ENCRYPTED via settings_store.set_secret (never .env, never
# returned). base_url/model/account overrides are non-secret. Loopback-gated
# by the router dep, so LAN peers can't read masks or write keys.
class _LLMProviderBody(BaseModel):
api_key: str | None = Field(None, description="API key; '' clears it, None leaves unchanged")
base_url: str | None = None
model: str | None = None
account_id: str | None = Field(None, description="Cloudflare account id")
make_active: bool = False
class _LLMActiveBody(BaseModel):
provider: str = Field(..., description="provider id to activate")
@router.get("/llm-providers")
def list_llm_providers():
"""All providers with resolved base_url/model + whether a key is configured.
Never returns key material only `has_key`/`key_from_env` booleans.
"""
from services import llm_providers
return {
"active": llm_providers.active_provider_id(),
"providers": [llm_providers.describe(p) for p in llm_providers.all_providers()],
}
@router.put("/llm-providers/{provider_id}")
def save_llm_provider(provider_id: str, body: _LLMProviderBody):
"""Save a provider's key (encrypted) + optional base_url/model/account.
A None field is left unchanged; an empty api_key clears the stored key.
"""
from services import llm_providers
p = llm_providers.get_provider(provider_id)
if p is None:
raise HTTPException(status_code=404, detail=f"unknown provider {provider_id!r}")
if body.api_key is not None:
llm_providers.save_key(provider_id, body.api_key.strip())
llm_providers.save_overrides(
provider_id, base_url=body.base_url, model=body.model,
account_id=body.account_id,
)
# An explicit save also claims the active slot when the user has never
# chosen a provider (#963). Without this, a saved-and-tested local
# provider (Ollama/LM Studio) evaporates on restart: active_provider_id()
# deliberately excludes local providers from auto-select, so the plain
# "Save" left nothing persisted to resolve. Gated on the STORED selection
# only — an explicit prior choice is never stolen by a plain save, and an
# unconfigured provider can't claim the slot.
if body.make_active or (
llm_providers.stored_active_provider_id() is None
and llm_providers.is_configured(p)
):
llm_providers.set_active_provider(provider_id)
return list_llm_providers()
@router.post("/llm-providers/active")
def set_active_llm_provider(body: _LLMActiveBody):
from services import llm_providers
if llm_providers.get_provider(body.provider) is None:
raise HTTPException(status_code=404, detail=f"unknown provider {body.provider!r}")
llm_providers.set_active_provider(body.provider)
return list_llm_providers()
def _scrub_llm_detail(e: Exception, api_key: str | None) -> str:
"""Scrubbed, UI-safe failure text. scrub_text() covers env secrets and
home paths but a STORE-persisted key isn't in the env, and some
providers echo the key in error bodies, so redact the exact resolved key
explicitly before the generic pass."""
from core.scrub import scrub_text
detail = f"{type(e).__name__}: {e}"
if api_key and api_key != "local" and len(api_key) >= 8:
detail = detail.replace(api_key, "•••")
return scrub_text(detail)
def _classify_llm_error(e: Exception) -> str:
"""Map a provider-call failure to an actionable kind the UI can localize.
Kinds: auth (bad/missing key), not_found (model or endpoint path),
rate_limit, network (DNS/conn/timeout), error (everything else).
Status codes win when the OpenAI SDK provides one; exception-family
names catch the non-HTTP failures (DNS, refused, TLS, timeout).
"""
status = getattr(e, "status_code", None)
if status in (401, 403):
return "auth"
if status == 404:
return "not_found"
if status == 429:
return "rate_limit"
name = type(e).__name__
if name in ("APIConnectionError", "APITimeoutError", "ConnectError",
"ConnectTimeout", "TimeoutError"):
return "network"
if name == "AuthenticationError":
return "auth"
if name == "NotFoundError":
return "not_found"
if name == "RateLimitError":
return "rate_limit"
return "error"
@router.post("/llm-providers/{provider_id}/test")
def test_llm_provider(provider_id: str):
"""One cheap round-trip against a provider to prove the key/URL work.
Temporarily activates the provider for the probe by resolving its config
directly (does not change the persisted active selection). Returns
latency_ms plus, on failure, a classified ``kind`` (config / auth /
not_found / rate_limit / network / error) so the UI shows an actionable,
localizable message instead of a raw exception string.
"""
import time as _time
from services import llm_providers
p = llm_providers.get_provider(provider_id)
if p is None:
raise HTTPException(status_code=404, detail=f"unknown provider {provider_id!r}")
base_url = llm_providers.resolve_base_url(p)
api_key = llm_providers.resolve_api_key(p)
if not base_url:
return {"ok": False, "kind": "config", "detail": "No Base URL set for this provider."}
if not api_key:
return {"ok": False, "kind": "config", "detail": "No API key configured for this provider."}
t0 = _time.monotonic()
try:
from openai import OpenAI
# max_retries=0: this is an interactive probe with a live spinner — the
# SDK's default 2 automatic retries turn a 429/timeout into a ~34s hang.
# Surface the first failure immediately instead.
client = OpenAI(api_key=api_key, base_url=base_url, max_retries=0)
res = client.chat.completions.create(
model=llm_providers.resolve_model(p),
messages=[{"role": "user", "content": "Reply with the single word: ok"}],
timeout=20,
)
reply = (res.choices[0].message.content or "").strip()
return {
"ok": True,
"model": llm_providers.resolve_model(p),
"reply": reply[:80],
"latency_ms": int((_time.monotonic() - t0) * 1000),
}
except Exception as e: # noqa: BLE001 — surface a clean, scrubbed error to the UI
return {
"ok": False,
"kind": _classify_llm_error(e),
"detail": _scrub_llm_detail(e, api_key),
"latency_ms": int((_time.monotonic() - t0) * 1000),
}
@router.get("/llm-providers/{provider_id}/models")
def list_llm_provider_models(provider_id: str):
"""List model ids the provider's key can access (OpenAI-compat /models).
Powers the model-picker datalist in Settings LLM Providers so users
don't have to guess model names. Read-only; failures return the same
classified shape as /test; capped so a huge catalog can't bloat the UI.
"""
from services import llm_providers
p = llm_providers.get_provider(provider_id)
if p is None:
raise HTTPException(status_code=404, detail=f"unknown provider {provider_id!r}")
base_url = llm_providers.resolve_base_url(p)
api_key = llm_providers.resolve_api_key(p)
if not base_url or not api_key:
return {"ok": False, "kind": "config", "models": []}
try:
from openai import OpenAI
# max_retries=0: interactive probe — fail fast, don't burn ~34s on the
# SDK's default retry ladder when the key/URL is wrong (matches /test).
client = OpenAI(api_key=api_key, base_url=base_url, max_retries=0)
ids = sorted(m.id for m in client.models.list(timeout=10))
# Cap so a huge catalog can't bloat the datalist; flag the cap so the UI
# can say "first 200 shown" rather than implying it's the full list.
return {"ok": True, "models": ids[:200], "truncated": len(ids) > 200}
except Exception as e: # noqa: BLE001
return {
"ok": False,
"kind": _classify_llm_error(e),
"detail": _scrub_llm_detail(e, api_key),
"models": [],
}
# ── LLM Skills (Settings → LLM Skills) ─────────────────────────────────────
# Per-feature enable/route control for every LLM consumption point. Each
# skill can be toggled off (degrades exactly like "no LLM configured") or
# routed to a specific provider (local Ollama/LM Studio vs a remote key)
# instead of the one global active provider. Loopback-gated (router dep).
class _LLMSkillBody(BaseModel):
enabled: bool | None = Field(None, description="None leaves the toggle unchanged")
provider_override: str | None = Field(
None,
description="provider id to route this skill to; '' or null clears "
"it (skill follows the active provider). Omit to leave "
"unchanged.",
)
@router.get("/llm-skills")
def list_llm_skills():
"""Every LLM skill with its toggle, routing, and resolved ready status."""
from services import llm_skills
return {"skills": [llm_skills.describe(s.id) for s in llm_skills.all_skills()]}
@router.put("/llm-skills/{skill_id}")
def set_llm_skill(skill_id: str, body: _LLMSkillBody):
"""Toggle a skill and/or set its provider routing.
Field semantics match the providers PUT: an omitted field is left
unchanged; ``provider_override: ""``/``null`` clears the override.
404 for an unknown skill or an unknown provider id.
"""
from services import llm_skills
if llm_skills.get_skill(skill_id) is None:
raise HTTPException(status_code=404, detail=f"unknown LLM skill {skill_id!r}")
kwargs = {}
if body.enabled is not None:
kwargs["enabled"] = body.enabled
if "provider_override" in body.model_fields_set:
kwargs["provider_override"] = body.provider_override
try:
if kwargs:
llm_skills.configure_skill(skill_id, **kwargs)
except ValueError as e:
raise HTTPException(status_code=404, detail=str(e))
return list_llm_skills()
# ── License acceptance (Phase 3 Plan 03-01 / TTS-05) ──────────────────────
# Frontend ``SupertonicLicenseDialog`` flips the engine-license bit via this
# endpoint. The handler is loopback-gated (router-level dep) and the
@@ -697,212 +284,3 @@ def set_models_dir(body: _ModelsDirBody):
user_env.set_user_env(_MODELS_DIR_ENV, path)
return {"configured": path, "effective": _effective_models_dir(), "restart_required": True}
# ── Storage report (Settings → Storage) ────────────────────────────────────
# Per-volume disk totals + du-style sizes for everything the app owns (HF
# model cache, app data subtotals, engine venvs, temp files) with server-side
# warnings. Heavy directory walks run in a worker thread with per-category
# deadlines and a 5-minute in-process cache (services.storage_report), so the
# endpoint stays cheap on repeat Settings visits. Loopback-gated via the
# router-level dep like every sibling.
@router.get("/storage")
async def get_storage_report(refresh: bool = Query(False)):
"""Disk + per-category storage usage for the Settings → Storage panel.
`refresh=1` bypasses the 5-minute cache and rescans. `min_free_gb`
reuses the setup wizard's constant so both surfaces warn at the same
threshold.
"""
from api.routers.setup.wizard import MIN_FREE_GB
from core.config import DATA_DIR
from services import storage_report
try:
return await asyncio.to_thread(
storage_report.get_report,
data_dir=DATA_DIR,
hf_cache_dir=_effective_models_dir(),
app_venv=storage_report.default_app_venv(),
min_free_gb=MIN_FREE_GB,
refresh=refresh,
)
except Exception:
logger.exception("storage report failed")
raise HTTPException(status_code=500, detail="Failed to compute storage report")
@router.post("/storage/temp/clear")
async def clear_temp_files():
"""Delete OmniVoice-owned temp files (Settings → Storage → Temporary files).
Removes only the ``omnivoice*`` entries in the OS temp dir the exact
population the storage report's "temp" category counts — and invalidates
the cached report so the next scan reflects the reclaimed space. Partial
failures (files held open by a running job) are returned per entry.
"""
from services import storage_report
try:
result = await asyncio.to_thread(storage_report.clear_temp)
storage_report.clear_cache()
return result
except Exception:
logger.exception("clear temp files failed")
raise HTTPException(status_code=500, detail="Failed to clear temporary files")
# ── HF mirror endpoint (parity program Wave 4.3 / §R4 c) ──────────────────
# Restricted-network users (e.g. behind the Great Firewall) need to point
# huggingface_hub at a mirror. HF reads HF_ENDPOINT at import time, so a
# change takes effect on the next backend start — persisted to the durable
# per-user env so it survives Tauri/Finder launches that don't inherit a
# shell. Loopback-gated via the router dep.
_HF_ENDPOINT_ENV = "HF_ENDPOINT"
# A few well-known mirrors, surfaced as quick-picks in the UI. hf-mirror.com
# is the community mirror most-used in China; the official endpoint clears it.
_HF_MIRROR_PRESETS = [
{"label": "Hugging Face (official)", "url": ""},
{"label": "hf-mirror.com (community, China)", "url": "https://hf-mirror.com"},
]
class _HFMirrorBody(BaseModel):
url: str = Field("", description="HF_ENDPOINT URL; empty string clears it (official endpoint)")
@router.get("/hf-mirror")
def get_hf_mirror():
from core import user_env
configured = user_env.get_user_env(_HF_ENDPOINT_ENV) or ""
return {
# The value that will apply after restart (persisted), and what's
# live in this process (env may differ until then).
"configured": configured,
"effective": os.environ.get(_HF_ENDPOINT_ENV, ""),
"presets": _HF_MIRROR_PRESETS,
}
@router.put("/hf-mirror")
def set_hf_mirror(body: _HFMirrorBody):
from core import user_env
url = (body.url or "").strip().rstrip("/")
if url and not url.startswith(("http://", "https://")):
raise HTTPException(status_code=400, detail="Mirror URL must start with http(s)://")
# Compare against the currently-persisted value (normalised the same way) so
# a no-op save doesn't nag the user to restart. Only a real change to the
# persisted endpoint can require a restart.
previous = (user_env.get_user_env(_HF_ENDPOINT_ENV) or "").strip().rstrip("/")
changed = url != previous
try:
if url:
user_env.set_user_env(_HF_ENDPOINT_ENV, url)
os.environ[_HF_ENDPOINT_ENV] = url # best-effort for new downloads this session
else:
user_env.unset_user_env(_HF_ENDPOINT_ENV)
os.environ.pop(_HF_ENDPOINT_ENV, None)
except Exception:
logger.exception("set_hf_mirror failed")
raise HTTPException(status_code=500, detail="Failed to persist mirror setting")
# Model Store downloads pick up the new mirror immediately — the download
# path resolves the endpoint per-call and we updated os.environ above. Only
# transformers-side model *loads* (which read HF_ENDPOINT at import time)
# need a restart, so restart_required is True ONLY when the value actually
# changed — a no-op re-save never asks for a restart.
return {"configured": url, "restart_required": changed, "presets": _HF_MIRROR_PRESETS}
# ── OpenAI-compatible remote ASR (#877) ─────────────────────────────────────
# A path to Qwen3-ASR/FunASR/SenseVoice — or OpenAI's own Whisper API — today,
# without waiting on transformers to ship a direct Qwen3-ASR integration.
# base_url/model are plain settings_store text rows; the key is encrypted via
# settings_store.set_secret — same convention as /llm-providers, never
# returned to the client, '' clears it, omitted/None leaves it unchanged.
class _ASROpenAICompatBody(BaseModel):
base_url: str | None = None
model: str | None = None
api_key: str | None = Field(None, description="'' clears it, None leaves unchanged")
@router.get("/asr-openai-compat")
def get_asr_openai_compat():
from services import asr_backend
return {
"base_url": asr_backend.resolve_openai_compat_asr_base_url(),
"model": asr_backend.resolve_openai_compat_asr_model(),
"has_key": asr_backend.openai_compat_asr_has_key(),
}
@router.put("/asr-openai-compat")
def set_asr_openai_compat(body: _ASROpenAICompatBody):
from services import asr_backend, settings_store
if body.base_url is not None:
url = body.base_url.strip().rstrip("/")
if url and not url.startswith(("http://", "https://")):
raise HTTPException(status_code=400, detail="Base URL must start with http(s)://")
settings_store.set_text(asr_backend._ASR_OPENAI_COMPAT_BASE_URL_KEY, url)
if body.model is not None:
settings_store.set_text(
asr_backend._ASR_OPENAI_COMPAT_MODEL_KEY, body.model.strip() or "whisper-1"
)
if body.api_key is not None:
settings_store.set_secret(
asr_backend._ASR_OPENAI_COMPAT_SECRET_NAME, body.api_key.strip()
)
return get_asr_openai_compat()
# ── Updates panel: shipped changelog + pre-migration DB backup state ────────
# (feat/safe-updates). Both are read-only, local-first surfaces for
# Settings → Updates: the "What's new" viewer reads the CHANGELOG.md that
# ships with the app, and the backup line shows the newest pre-migration
# snapshot written by core.db_backup before `alembic upgrade head` runs.
@router.get("/changelog")
def get_changelog(limit_versions: int = Query(5, ge=1, le=50)):
"""Structured release notes from the shipped CHANGELOG.md (newest first).
Bullets are raw markdown-lite (bold leads, `code`, (#NNN) refs) — the
frontend renders them safely without HTML. `available: false` when this
install has no changelog (never an error: the viewer just hides)."""
from core import changelog
path = changelog.changelog_path()
if not path:
return {"available": False, "releases": []}
try:
with open(path, encoding="utf-8") as fh:
releases = changelog.parse_changelog(fh.read(), limit_versions)
except Exception:
logger.exception("changelog parse failed")
return {"available": False, "releases": []}
return {"available": bool(releases), "releases": releases}
@router.get("/db-backup")
def get_db_backup_state():
"""Newest pre-migration database backup (or none yet). Feeds the
"your data is backed up before every update" line in Settings Updates."""
from core import db_backup
from core.config import DB_PATH
latest = db_backup.latest_backup(DB_PATH)
return {
"available": latest is not None,
"latest": latest,
"count": len(db_backup.list_backups(DB_PATH)),
"keep": db_backup.KEEP_BACKUPS,
}
+4 -362
View File
@@ -11,28 +11,14 @@ from __future__ import annotations
import asyncio
import json
import logging
import os
import sys
from fastapi import APIRouter, HTTPException
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from core import prefs
from utils import hf_progress
from utils import download_aggregator
# Weight-floor scan (MM2-07 / #352) lives in ``models.py`` — the lowest module in
# the setup import graph — so install-time validation here, the first-run
# install-state detector (#622), and load-time repair share one set of floors and
# can't drift apart. ``_MIN_WEIGHT_BYTES``/``_WEIGHT_FLOORS`` re-exported for tests.
from .models import ( # noqa: F401
KNOWN_MODELS,
invalidate_cache,
snapshot_has_weights,
disk_space_error,
_MIN_WEIGHT_BYTES,
_WEIGHT_FLOORS,
)
from .models import KNOWN_MODELS, invalidate_cache
logger = logging.getLogger("omnivoice.setup.download")
router = APIRouter()
@@ -40,181 +26,6 @@ router = APIRouter()
# Cooldown: prevent rapid re-install after a failure. Maps repo_id → last_fail_time.
_install_cooldowns: dict[str, float] = {}
_COOLDOWN_SECS = 60.0
# Evict cooldown entries older than this so the dict can't grow unbounded across
# a long-lived process (MM2-06). Anything past the cooldown window is dead state.
_COOLDOWN_TTL_SECS = 3600.0
def _sweep_cooldowns(now: float) -> None:
"""Drop cooldown entries older than the TTL (MM2-06). Keeps the dict bounded
without this it accumulated one entry per ever-failed repo forever."""
stale = [k for k, t in _install_cooldowns.items() if (now - t) > _COOLDOWN_TTL_SECS]
for k in stale:
_install_cooldowns.pop(k, None)
# Repo_ids the user asked to cancel (FDL-11). Checked between retry attempts.
# Note: a single in-flight snapshot_download/Xet fetch is not interruptible
# mid-file in hf_hub 1.7.2 — cancel stops further retries, marks the row
# cancelled, and clears the cooldown so a cancel isn't rate-limited.
_cancelled: set[str] = set()
def _download_max_workers() -> int:
"""Parallel-FILES worker count for snapshot_download (FDL-02). Default 8 —
don't crank it: Xet already parallelises *within* each file via concurrent
byte-range gets, so a high count just multiplies buffer pressure. Override
via prefs / OMNIVOICE_DOWNLOAD_MAX_WORKERS for power users."""
raw = prefs.resolve("download_max_workers", env="OMNIVOICE_DOWNLOAD_MAX_WORKERS", default=8)
try:
return max(1, int(raw))
except (TypeError, ValueError):
return 8
def _download_endpoint() -> "str | None":
"""Optional HF endpoint override (FDL-10 mirror path, opt-in). Returned as a
per-call ``endpoint=`` rather than a process-wide HF_ENDPOINT mutation. A
mirror routes through the classic LFS path (no Xet) documented in
docs/downloading-models.md."""
ep = prefs.resolve("hf_endpoint", env="HF_ENDPOINT", default=None)
return ep or None
def apply_xet_env() -> None:
"""Apply opt-in Xet tuning knobs to the environment before a download
(FDL-04). Both default OFF; env wins over the prefs store. high-performance
can *hurt* low-RAM machines (needs lots of RAM/bandwidth); HDD-sequential
avoids parallel-write thrash on spinning disks. Idempotent."""
import os as _os
high_perf = prefs.resolve("xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False)
if _truthy(high_perf):
_os.environ["HF_XET_HIGH_PERFORMANCE"] = "1"
hdd_seq = prefs.resolve("xet_hdd_sequential_write", env="HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY", default=False)
if _truthy(hdd_seq):
_os.environ["HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY"] = "1"
def _truthy(v) -> bool:
if isinstance(v, bool):
return v
return str(v).strip().lower() in {"1", "true", "yes", "on"}
class _InstallCancelled(Exception):
"""Raised inside the install worker when the user cancels (FDL-11)."""
def compute_plan(plan_files) -> dict:
"""Summarise a snapshot_download(dry_run=True) result into the install_plan
payload (FDL-05): total bytes, bytes already cached (skipped), bytes that
will actually download, and file counts. ``will_download`` defaults to
``not is_cached`` for forward-compat with older DryRunFileInfo shapes."""
total = sum(int(getattr(f, "file_size", 0) or 0) for f in plan_files)
cached = sum(
int(getattr(f, "file_size", 0) or 0)
for f in plan_files if getattr(f, "is_cached", False)
)
will = [
f for f in plan_files
if getattr(f, "will_download", not getattr(f, "is_cached", False))
]
to_dl = sum(int(getattr(f, "file_size", 0) or 0) for f in will)
n_files = len(plan_files)
n_cached = sum(1 for f in plan_files if getattr(f, "is_cached", False))
return {
"total_bytes": total,
"cached_bytes": cached,
"to_download_bytes": to_dl,
"n_files": n_files,
"n_cached": n_cached,
}
def _segmented_enabled() -> bool:
"""IDM-style multi-connection accelerator (FDL-09), default **ON**. The app
forces the legacy-LFS path (HF_HUB_DISABLE_XET=1) for clear progress, but that
path is single-stream and slow this restores parallel byte-range speed AND
real live progress, and falls back to snapshot_download on any error so it
can never compromise a correct install. Default-on so first-run downloads are
fast out of the box (pairs with an HF token for higher rate limits); set
OMNIVOICE_SEGMENTED_DOWNLOAD=0 to force the single-stream path."""
return _truthy(prefs.resolve(
"segmented_downloader", env="OMNIVOICE_SEGMENTED_DOWNLOAD", default=True,
))
def _xet_active() -> bool:
"""True only when hf_xet is installed AND not disabled. The app sets
HF_HUB_DISABLE_XET=1 by default, so this is normally False which is when
the segmented accelerator pays off."""
import importlib.util
if importlib.util.find_spec("hf_xet") is None:
return False
return os.environ.get("HF_HUB_DISABLE_XET", "").strip().lower() not in {"1", "true", "yes", "on"}
def _repo_cancelled(repo_id: str) -> bool:
return repo_id in _cancelled
def _segmented_snapshot(repo_id: str, *, endpoint: "str | None") -> str:
"""Fetch every file of a repo via the segmented downloader into the HF
cache, mirroring hf_hub_download's blob+snapshot+refs layout so the result
is indistinguishable from snapshot_download (FDL-09) keeping /models
install-state, is_cached, and delete working. Feeds real bytes to the
aggregator. Raises on any error; the caller falls back to snapshot_download.
"""
import asyncio as _asyncio
from huggingface_hub import HfApi, constants as _C
from huggingface_hub.file_download import (
hf_hub_url, get_hf_file_metadata, repo_folder_name, _create_symlink,
)
from services.segmented_download import segmented_download
from services.token_resolver import resolve as _resolve_token
token = _resolve_token()
api = HfApi(endpoint=endpoint, token=token)
info = api.repo_info(repo_id, repo_type="model")
commit = info.sha
files = [s.rfilename for s in (info.siblings or [])]
if not commit or not files:
raise RuntimeError("repo_info returned no commit/siblings")
repo_dir = os.path.join(_C.HF_HUB_CACHE, repo_folder_name(repo_id=repo_id, repo_type="model"))
blobs_dir = os.path.join(repo_dir, "blobs")
snap_dir = os.path.join(repo_dir, "snapshots", commit)
refs_dir = os.path.join(repo_dir, "refs")
for d in (blobs_dir, snap_dir, refs_dir):
os.makedirs(d, exist_ok=True)
for rel in files:
if _repo_cancelled(repo_id):
raise _InstallCancelled()
url = hf_hub_url(repo_id, rel, endpoint=endpoint, revision=commit)
meta = get_hf_file_metadata(url, token=token)
etag = (meta.etag or "").strip('"')
if not etag:
raise RuntimeError(f"no etag for {rel}")
blob_path = os.path.join(blobs_dir, etag)
pointer = os.path.join(snap_dir, rel)
os.makedirs(os.path.dirname(pointer), exist_ok=True)
if not os.path.exists(blob_path):
_asyncio.run(segmented_download(
meta.location or url, blob_path,
token=token, expected_size=meta.size, expected_etag=etag,
on_bytes=lambda d, k=rel: download_aggregator.add_bytes(repo_id, k, d),
cancel_check=lambda: _repo_cancelled(repo_id),
))
if not os.path.lexists(pointer):
_create_symlink(blob_path, pointer, new_blob=True)
# refs/main → commit so scan_cache_dir maps the revision correctly.
try:
with open(os.path.join(refs_dir, "main"), "w") as f:
f.write(commit)
except OSError:
pass
return snap_dir
# ── SSE Download Stream ───────────────────────────────────────────────────
@@ -231,41 +42,6 @@ def _safe_put(queue: asyncio.Queue, event) -> None:
pass
# Minimum size for "this snapshot actually contains model weights". An
# interrupted snapshot_download can leave config/tokenizer files but no
# weights; the install then looks complete and synthesis later fails with
# "does not appear to have a file named pytorch_model.bin or
# model.safetensors" (#352). 5 MB clears every weight format we ship
# (safetensors/bin shards, onnx, pt, gguf) without false-positiving on
# config-only aux repos.
def _validate_snapshot_has_weights(repo_id: str, snapshot_path: str) -> None:
"""Raise OSError when a finished snapshot has no plausible weight file —
surfaces the truncated-download class (#352) at install time, where the
retry loop and the UI's re-download path can deal with it, instead of at
first synthesis with an opaque transformers error.
Delegates the weight check to ``models.snapshot_has_weights`` (single source of
the floors); only the install-time error message lives here."""
if snapshot_has_weights(snapshot_path):
return
biggest = 0
try:
for root, _dirs, files in os.walk(snapshot_path, followlinks=True):
for f in files:
try:
biggest = max(biggest, os.path.getsize(os.path.join(root, f)))
except OSError:
continue
except OSError:
pass
raise OSError(
f"{repo_id}: download finished but no model weights were found in the "
"snapshot (largest file "
f"{biggest} bytes). The download was likely interrupted — delete the "
"model in Settings → Models and install it again."
)
@router.get("/setup/download-stream")
async def setup_download_stream():
"""SSE: forward every HuggingFace download tqdm update as a JSON event."""
@@ -322,7 +98,6 @@ async def install_model(req: InstallModelRequest):
)
# Cooldown guard — don't retry if the same model just failed.
import time as _time_check
_sweep_cooldowns(_time_check.time()) # bound the dict (MM2-06)
last_fail = _install_cooldowns.get(req.repo_id)
if last_fail and (_time_check.time() - last_fail) < _COOLDOWN_SECS:
remaining = int(_COOLDOWN_SECS - (_time_check.time() - last_fail))
@@ -337,7 +112,6 @@ async def install_model(req: InstallModelRequest):
def _do():
token = hf_progress.current_repo_id.set(req.repo_id)
_cancelled.discard(req.repo_id) # clear any stale cancel from a prior run
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
@@ -351,22 +125,7 @@ async def install_model(req: InstallModelRequest):
LocalEntryNotFoundError,
)
logger.info("model install starting: %s", req.repo_id)
# Apply opt-in Xet tuning knobs (high-perf / HDD) before downloading.
apply_xet_env()
# Drive snapshot_download explicitly (FDL-02): pass our progress-
# emitting tqdm subclass so progress is deterministic + Xet-aware
# (Xet feeds bytes into whatever tqdm_class is supplied), bound the
# parallel-files worker count, and honour an optional mirror endpoint.
dl_kwargs: dict = {
"repo_id": req.repo_id,
"max_workers": _download_max_workers(),
}
_tqdm_cls = hf_progress.tracked_tqdm_class()
if _tqdm_cls is not None:
dl_kwargs["tqdm_class"] = _tqdm_cls
_endpoint = _download_endpoint()
if _endpoint:
dl_kwargs["endpoint"] = _endpoint
dl_kwargs: dict = {"repo_id": req.repo_id}
if sys.platform == "win32":
dl_kwargs["local_dir_use_symlinks"] = False
@@ -394,91 +153,12 @@ async def install_model(req: InstallModelRequest):
hb = threading.Thread(target=_heartbeat, daemon=True)
hb.start()
# Pre-flight (FDL-05): a dry-run resolve gives the UI an accurate
# denominator — total bytes, bytes already cached (skipped), and the
# bytes that will actually download — BEFORE any byte flows. Seeds
# the overall aggregator so its bar/ETA are correct from the first
# event. Degrades gracefully (totals=None) on older/gated repos.
_preflight_kwargs = {"repo_id": req.repo_id, "dry_run": True}
if _endpoint:
_preflight_kwargs["endpoint"] = _endpoint
try:
_plan = snapshot_download(**_preflight_kwargs)
_summary = compute_plan(_plan)
# Disk-space guard (before a single byte flows): the preflight
# gives an exact "to download" size, so reject an install that
# would overrun the cache volume — with the numbers named —
# instead of failing mid-download with a cryptic OSError. No-op
# when it fits or the size is unknown. Same on every platform.
_disk_err = disk_space_error(_summary["to_download_bytes"])
if _disk_err:
logger.info("model install %s: rejected — %s", req.repo_id, _disk_err)
_resolving.set() # stop the heartbeat thread before we bail
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"downloaded": 0, "total": 0, "pct": 0.0,
"phase": "install_error",
"error": _disk_err,
})
# A disk-full is not a transient network failure — don't set
# a cooldown (freeing space, not waiting, is the fix). The
# outer finally still cleans up the aggregator + context.
return
download_aggregator.start(
req.repo_id,
total_bytes=_summary["to_download_bytes"],
files_total=max(0, _summary["n_files"] - _summary["n_cached"]),
)
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"phase": "install_plan",
**_summary,
})
except Exception as _pf_err:
# No preflight (older/gated repo, mirror without dry-run, etc.):
# fall back to today's fill-in-as-files-appear behaviour.
logger.info("model install %s: preflight unavailable (%s)", req.repo_id, _pf_err)
download_aggregator.start(req.repo_id)
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"phase": "install_plan",
"total_bytes": None,
"cached_bytes": None,
"to_download_bytes": None,
"n_files": None,
"n_cached": None,
})
_max_attempts = 5
_attempt = 0
while True:
if req.repo_id in _cancelled:
raise _InstallCancelled()
_attempt += 1
try:
# Segmented accelerator (FDL-09, default ON): parallel
# byte-range fetch with real live progress, for the
# legacy-LFS path. Any failure falls through to
# snapshot_download — the accelerator can never compromise a
# correct install.
_snapshot_path = None
if _attempt == 1 and _segmented_enabled() and not _xet_active():
try:
_snapshot_path = _segmented_snapshot(req.repo_id, endpoint=_endpoint)
except _InstallCancelled:
raise
except Exception as _seg_err:
logger.info(
"segmented download for %s failed (%s); falling back to snapshot_download",
req.repo_id, _seg_err,
)
_snapshot_path = None
if _snapshot_path is None:
_snapshot_path = snapshot_download(**dl_kwargs)
_validate_snapshot_has_weights(req.repo_id, _snapshot_path)
snapshot_download(**dl_kwargs)
break
except (HfHubHTTPError, LocalEntryNotFoundError, OSError) as net_err:
if _attempt >= _max_attempts:
@@ -499,10 +179,6 @@ async def install_model(req: InstallModelRequest):
_t.sleep(_backoff)
# Stop heartbeat once download completes
_resolving.set()
# Flush the overall bar to 100% with the true byte total (FDL-06):
# under Xet the per-file byte bars don't surface completion, so the
# aggregator can sit below 100% even though every file landed.
download_aggregator.complete(req.repo_id)
logger.info("model install done: %s", req.repo_id)
hf_progress.emit({
"repo_id": req.repo_id,
@@ -510,60 +186,26 @@ async def install_model(req: InstallModelRequest):
"downloaded": 0, "total": 0, "pct": 1.0,
"phase": "install_done",
})
_install_cooldowns.pop(req.repo_id, None) # success clears any cooldown (MM2-06)
invalidate_cache()
except _InstallCancelled:
_resolving.set()
logger.info("model install cancelled: %s", req.repo_id)
# A cancel is user intent, not a failure — don't set a cooldown.
_install_cooldowns.pop(req.repo_id, None)
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"downloaded": 0, "total": 0, "pct": 0.0,
"phase": "install_cancelled",
})
except Exception as e:
_resolving.set()
logger.info("model install failed for %s: %s", req.repo_id, e)
import time as _time_fail
_install_cooldowns[req.repo_id] = _time_fail.time()
# #874: when the install failed because the configured HF mirror is
# unreachable, name the mirror + the setting instead of leaking the
# raw connectivity error. #959: likewise for the SOCKS-proxy class
# (missing socksio fails the download's session construction).
# No-op for every other failure.
from core.failure import append_hint
hf_progress.emit({
"repo_id": req.repo_id,
"filename": req.repo_id,
"downloaded": 0, "total": 0, "pct": 0.0,
"phase": "install_error",
"error": append_hint(str(e)),
"error": str(e),
})
finally:
_cancelled.discard(req.repo_id)
download_aggregator.finish(req.repo_id)
hf_progress.current_repo_id.reset(token)
loop.create_task(asyncio.to_thread(_do))
return {"status": "install_started", "repo_id": req.repo_id}
@router.post("/models/install/cancel")
async def cancel_install(req: InstallModelRequest):
"""Request cancellation of an in-flight install (FDL-11).
Best-effort: stops further retry attempts and marks the row cancelled. A
single in-flight snapshot_download/Xet fetch isn't interruptible mid-file
in hf_hub 1.7.2, so an already-streaming file finishes; the cancel takes
effect at the next retry boundary. Clears the cooldown so the user can
immediately restart."""
_cancelled.add(req.repo_id)
_install_cooldowns.pop(req.repo_id, None)
return {"cancelling": req.repo_id}
# ── Delete ─────────────────────────────────────────────────────────────────
@router.delete("/models/{repo_id:path}")
+4 -169
View File
@@ -123,66 +123,6 @@ def hf_cache_dir() -> str:
)
# ── Disk-space guard (shared, single-sourced) ──────────────────────────────
# MIN_FREE_GB is the headroom we insist on keeping free on the model-cache
# volume — the wizard's absolute pre-install floor AND the extra buffer the
# per-install check demands on top of the download itself, so an "Install all"
# can't fill the disk to the brim (setup/download.py). Lives here — the lowest
# module in the setup import graph — so the wizard, the /models header, and the
# install endpoint can't drift apart (mirrors the weight-floor single-sourcing).
_GIB = 1024 ** 3
MIN_FREE_GB = 10
def disk_free_bytes(path: "str | None" = None) -> int:
"""Free bytes on the volume backing *path* (defaults to the HF cache).
Walks up to the nearest existing ancestor so a not-yet-created cache dir
still probes the correct mount point. ``shutil.disk_usage`` is cross-platform
(macOS/Windows/Linux) so this behaves identically everywhere. Never raises.
"""
import shutil
try:
p = Path(path or hf_cache_dir()).resolve()
while not p.exists():
parent = p.parent
if parent == p: # reached the volume root
break
p = parent
return int(shutil.disk_usage(str(p)).free)
except Exception:
return 0
def disk_space_error(to_download_bytes: "int | None", *, cache_dir: "str | None" = None) -> "str | None":
"""Actionable message when *to_download_bytes* (+ MIN_FREE_GB headroom) won't
fit on the cache volume; ``None`` when it fits, the size is unknown, or the
volume can't be probed (never block on missing information).
Names the three numbers a user needs to act needs X, headroom Y, have Z
so "Install all" can't silently overrun the disk (issue: no pre-install disk
check). Platform-agnostic; applied identically on macOS/Windows/Linux.
"""
if not to_download_bytes or to_download_bytes <= 0:
return None # unknown plan (older/gated repo, mirror without dry-run) → don't block
cache = cache_dir or hf_cache_dir()
free = disk_free_bytes(cache)
if free <= 0:
return None # couldn't probe the volume → don't block on missing info
required = int(to_download_bytes) + MIN_FREE_GB * _GIB
if free >= required:
return None
def _gb(n: int) -> str:
return f"{n / _GIB:.1f} GB"
return (
f"Not enough disk space to install: this download needs {_gb(int(to_download_bytes))} "
f"plus {MIN_FREE_GB} GB free headroom ({_gb(required)} total), but only {_gb(free)} "
f"is free at {cache}. Free up space (or move the model cache to a bigger volume) and retry."
)
def _repo_dir_name(repo_id: str) -> str:
"""HF cache dir name for a repo: 'k2-fsa/OmniVoice''models--k2-fsa--OmniVoice'."""
return "models--" + repo_id.replace("/", "--")
@@ -206,94 +146,6 @@ def _hub_cache_roots() -> list[str]:
return roots
# ── Weight-presence (truncated-cache) detection ─────────────────────────────
# A cache that downloaded config/tokenizer files but not the weight shard still
# occupies bytes on disk, so a size-only "installed" check (#352/#581/#606) reads
# it as installed and the first-run wizard hides the re-download button, stranding
# the user (#622). These helpers tell a *complete* snapshot from a truncated one by
# checking for a plausible weight file — the same class `download.py` guards at
# install time and `model_manager.py` repairs at load time. Shared here (the lowest
# module in the setup import graph; `download.py` imports from this module) so the
# floors live in exactly one place and can't drift between the three call sites.
_MIN_WEIGHT_BYTES = 5 * 1024 * 1024 # tensor formats: a real shard is ≥ a few MB
# Per-extension floors. ONNX graphs are legitimately small (a complete model can be
# well under 5 MB), so they get a lower floor that still rejects a bytes-only partial.
_WEIGHT_FLOORS = {
".safetensors": _MIN_WEIGHT_BYTES,
".bin": _MIN_WEIGHT_BYTES,
".ckpt": _MIN_WEIGHT_BYTES,
".pt": _MIN_WEIGHT_BYTES,
".pth": _MIN_WEIGHT_BYTES,
".gguf": _MIN_WEIGHT_BYTES,
".onnx": 64 * 1024,
}
def snapshot_has_weights(snapshot_path: str) -> bool:
"""True when a finished snapshot dir holds a plausible weight file.
A snapshot is complete if it contains a recognized weight file meeting its
per-extension floor OR any file the global 5 MB floor (the lenient catch for
non-standard weight names). Returns True when the path can't be inspected — an
un-walkable dir must never be reported as truncated, only a confirmed weight-less
one. `getsize` follows symlinks, so HF's snapshot→blob links resolve correctly;
a broken link (missing blob) raises OSError and is skipped, i.e. counts as absent.
"""
try:
for root, _dirs, files in os.walk(snapshot_path, followlinks=True):
for f in files:
try:
size = os.path.getsize(os.path.join(root, f))
except OSError:
continue
ext = os.path.splitext(f)[1].lower()
floor = _WEIGHT_FLOORS.get(ext)
if floor is not None and size >= floor:
return True
if size >= _MIN_WEIGHT_BYTES:
return True
except OSError:
return True # can't inspect — don't mislabel as truncated
return False
def _snapshot_dirs(repo_id: str) -> list[str]:
"""Existing snapshot revision dirs for a repo across the candidate cache roots."""
name = _repo_dir_name(repo_id)
dirs: list[str] = []
for root in _hub_cache_roots():
snaps = os.path.join(root, name, "snapshots")
try:
for rev in os.listdir(snaps):
rev_dir = os.path.join(snaps, rev)
if os.path.isdir(rev_dir):
dirs.append(rev_dir)
except OSError:
continue
return dirs
def cache_is_complete(model: dict) -> bool:
"""True when this model's on-disk cache is usable (not a truncated download).
Config-only repos (``config_only: true`` in models.yaml e.g. pyannote's
diarisation pipeline, whose real weights live in referenced sub-repos) carry no
weight file of their own, so the weight check would false-positive them as
incomplete (#622 caveat). They're exempt: cache presence alone means complete.
A weight-bearing repo is complete only if at least one of its snapshots has
weights; if no snapshot dir is found on disk we can't prove truncation, so we
don't downgrade (the size-based caller already decided it's cached).
"""
if model.get("config_only"):
return True
dirs = _snapshot_dirs(model["repo_id"])
if not dirs:
return True
return any(snapshot_has_weights(d) for d in dirs)
def _is_cached_on_disk(repo_id: str) -> bool:
"""Direct-filesystem fallback for is_cached when scan_cache_dir is unavailable.
@@ -368,12 +220,8 @@ def is_cached(repo_id: str) -> bool:
except Exception as e:
# scan_cache_dir can raise on Windows (WinError 448 'untrusted mount
# point'); fall back to a direct disk check so a cached model isn't
# mistaken for missing and re-downloaded in a loop (#117/#118). Logged
# at WARNING with the exception type (MM2-09) so this fallback isn't
# invisible when triaging a Windows cache report — it previously logged
# at DEBUG and never showed at the default level.
logger.warning("is_cached: scan_cache_dir failed (%s: %s); using on-disk fallback for %s",
type(e).__name__, e, repo_id)
# mistaken for missing and re-downloaded in a loop (#117/#118).
logger.debug("scan_cache_dir failed (%s); using disk fallback", e)
return _is_cached_on_disk(repo_id)
@@ -434,15 +282,9 @@ def list_models():
out = []
for m in KNOWN_MODELS:
cached = cached_by_repo.get(m["repo_id"])
on_disk = cached is not None and cached["size_on_disk"] > 0
# A size-positive cache can still be a truncated download (config landed,
# weight shard didn't). Treat that as not-installed + incomplete so the
# wizard re-offers the download instead of stranding the user (#622).
incomplete = on_disk and not cache_is_complete(m)
out.append({
**m,
"installed": on_disk and not incomplete,
"incomplete": incomplete,
"installed": cached is not None and cached["size_on_disk"] > 0,
"size_on_disk_bytes": cached["size_on_disk"] if cached else 0,
"nb_files": cached["nb_files"] if cached else 0,
"supported": _model_supported(m),
@@ -451,10 +293,6 @@ def list_models():
"models": out,
"total_installed_bytes": sum(m["size_on_disk_bytes"] for m in out),
"hf_cache_dir": hf_cache_dir(),
# Free space on the cache volume, so the Model Store header can warn
# BEFORE an "Install all" overruns the disk (pairs with the per-install
# disk_space_error guard in setup/download.py).
"disk_free_gb": round(disk_free_bytes() / _GIB, 1),
"platform_tags": _current_platform_tags(),
}
_set_cache("models", response)
@@ -541,9 +379,6 @@ def recommendations():
entries = []
for rid in recommended_ids:
meta = known_by_id.get(rid, {})
# Mirror /models: a truncated cache (weights missing) is not installed, so
# the wizard counts it toward the remaining download instead of "all set".
installed = rid in cached_ids and cache_is_complete(meta or {"repo_id": rid})
entries.append({
"repo_id": rid,
"label": meta.get("label", rid),
@@ -551,7 +386,7 @@ def recommendations():
"size_gb": meta.get("size_gb", 0),
"required": bool(meta.get("required", False)),
"note": meta.get("note"),
"installed": installed,
"installed": rid in cached_ids,
})
to_download_gb = sum(e["size_gb"] for e in entries if not e["installed"])
+85 -136
View File
@@ -3,9 +3,7 @@
Extracted from the monolithic ``setup.py``.
- ``GET /setup/status`` missing-model gate for boot screen
- ``GET /setup/preflight`` system health check (OS, RAM, disk, GPU, network
genuine user facts only; the media engine (ffmpeg/ffprobe/yt-dlp) is an
internal concern that self-heals via ``services.media_tools``)
- ``GET /setup/preflight`` system health check (OS, RAM, GPU, ffmpeg)
- ``POST /setup/warmup`` background model pre-load
"""
from __future__ import annotations
@@ -14,25 +12,39 @@ import asyncio
import logging
import os
import platform as _platform
import shutil as _shutil
import sys
from fastapi import APIRouter
from api.schemas import SetupStatusResponse, PreflightResponse
# MIN_FREE_GB + disk_free_bytes are single-sourced in ``.models`` (the lowest
# module in the setup import graph) so the wizard gate, the /models header, and
# the per-install disk guard can't drift apart.
from .models import REQUIRED_MODELS, hf_cache_dir, is_cached, MIN_FREE_GB, disk_free_bytes
from .models import REQUIRED_MODELS, hf_cache_dir, is_cached
logger = logging.getLogger("omnivoice.setup.wizard")
router = APIRouter()
MIN_FREE_GB = 10
def _disk_free_gb(path: str) -> float:
"""Free GB on the volume containing *path* (thin GB wrapper over the shared
``models.disk_free_bytes``, which walks up to the nearest existing ancestor
for a not-yet-created path)."""
return disk_free_bytes(path) / (1024 ** 3)
"""Return free GB on the volume containing *path*.
If *path* doesn't exist yet (e.g. after a fresh wipe), walk up to the
nearest existing ancestor so ``shutil.disk_usage`` can still probe the
correct mount point.
"""
try:
from pathlib import Path
p = Path(path).resolve()
# Walk up until we find a directory that exists
while not p.exists():
parent = p.parent
if parent == p: # root
break
p = parent
return _shutil.disk_usage(str(p)).free / (1024 ** 3)
except Exception:
return 0.0
# ── Setup Status ───────────────────────────────────────────────────────────
@@ -169,40 +181,16 @@ def _detect_gpu() -> dict:
return info
def _probe_network(host: str = "huggingface.co", port: int = 443, timeout: float = 2.0) -> bool:
def _probe_network(host: str = "huggingface.co", timeout: float = 2.0) -> bool:
"""Tiny TCP connect test."""
import socket
try:
with socket.create_connection((host, port), timeout=timeout):
with socket.create_connection((host, 443), timeout=timeout):
return True
except Exception:
return False
def _hf_endpoint_host() -> tuple[str, int]:
"""Host/port of the Hugging Face endpoint actually in effect.
Mirror-aware: restricted-network users (e.g. behind the Great Firewall)
point HF_ENDPOINT at a mirror via Settings Models Hugging Face
mirror. Probing hardcoded huggingface.co would fail them even when their
configured mirror works fine.
"""
try:
from core.failure import configured_hf_mirror
mirror = configured_hf_mirror()
except Exception:
mirror = ""
if mirror:
try:
from urllib.parse import urlsplit
u = urlsplit(mirror)
if u.hostname:
return u.hostname, u.port or (80 if u.scheme == "http" else 443)
except Exception:
pass
return "huggingface.co", 443
def _ram_gb() -> float:
try:
import psutil
@@ -281,19 +269,59 @@ def preflight():
f"Fix write permissions on {cache} or point HF_HOME elsewhere.",
})
# ── Media engine (ffmpeg/ffprobe/yt-dlp) — deliberately NOT a check row.
# These are internal dependencies the app provisions for itself, not user
# facts: when the resolution chain has no tier at all, preflight kicks the
# bundled acquisition in the background and the wizard shows a quiet
# progress line (a failure card only if that fails — with Retry / use a
# system copy). yt-dlp is an importable locked module and never appears.
# Power users manage all three in Settings → Audio tools.
media_tools = None
# ── FFmpeg
ffmpeg_path = None
try:
from services.media_tools import summary as _media_summary
media_tools = _media_summary(auto_acquire=True)
except Exception as exc: # never break preflight on the media engine
logger.warning("preflight media_tools summary failed: %s", exc)
from services.ffmpeg_utils import find_ffmpeg
ffmpeg_path = find_ffmpeg()
except Exception as e:
checks.append({
"id": "ffmpeg", "label": "FFmpeg", "status": "fail",
"detail": str(e)[:200],
"fix": "Install ffmpeg via your package manager "
"(brew install ffmpeg / apt install ffmpeg / choco install ffmpeg).",
})
else:
checks.append({
"id": "ffmpeg", "label": "FFmpeg", "status": "pass",
"detail": ffmpeg_path, "fix": None,
})
# ── FFprobe
ffprobe_path = None
try:
from services.ffmpeg_utils import find_ffprobe
ffprobe_path = find_ffprobe()
except Exception:
pass
if ffprobe_path:
checks.append({
"id": "ffprobe", "label": "FFprobe", "status": "pass",
"detail": ffprobe_path, "fix": None,
})
else:
checks.append({
"id": "ffprobe", "label": "FFprobe", "status": "warn",
"detail": "Not bundled alongside ffmpeg.",
"fix": "File-probe endpoint (/tools/probe) will 501. "
"Install system ffmpeg (includes ffprobe) to enable it.",
})
# ── yt-dlp
yt_dlp_path = _shutil.which("yt-dlp")
if yt_dlp_path:
rc_ytv, yt_ver = _run_cmd([yt_dlp_path, "--version"], timeout=3.0)
yt_version = yt_ver.strip() if rc_ytv == 0 else "unknown"
checks.append({
"id": "yt-dlp", "label": "yt-dlp", "status": "pass",
"detail": f"{yt_dlp_path} (v{yt_version})", "fix": None,
})
else:
checks.append({
"id": "yt-dlp", "label": "yt-dlp", "status": "warn",
"detail": "Not found in system PATH.",
"fix": "YouTube clip downloads in Voice Gallery will fail. Download the standalone binary from https://github.com/yt-dlp/yt-dlp/releases and place it in your PATH.",
})
# ── GPU
gpu = _detect_gpu()
@@ -345,89 +373,15 @@ def preflight():
"status": gpu_status, "detail": gpu_detail, "fix": gpu_fix,
})
# ── GPU routing for the ACTIVE TTS engine (#21 — no silent CPU fallback).
# Distinct from the hardware "gpu" check above: this asks "will the engine
# the user actually selected use that GPU on this host?" Built from the same
# canonical probe + resolver the Engine Compatibility Matrix uses.
try:
from services.tts_backend import gpu_routing_verdict
gpu_routing = gpu_routing_verdict()
except Exception as exc: # never break preflight on a routing hiccup
logger.warning("preflight gpu_routing failed: %s", exc)
gpu_routing = None
if gpu_routing:
_rs = gpu_routing.get("routing_status")
_eng = gpu_routing.get("engine") or "active engine"
_dev = gpu_routing.get("effective_device") or "?"
_why = gpu_routing.get("routing_reason")
if _rs == "accelerated" and not _why:
r_status, r_detail, r_fix = "pass", f"{_eng}{_dev} (accelerated)", None
elif _rs == "accelerated": # driver/arch caveat
r_status, r_detail, r_fix = "warn", f"{_eng}{_dev}: {_why}", (
"GPU selected but may fail at kernel launch — update drivers / "
"reinstall torch for this GPU architecture.")
elif _rs == "cpu_fallback":
r_status, r_detail, r_fix = "warn", (
f"{_eng} runs on CPU here: {_why or 'no GPU path for this host'}"), (
"Pick an engine that supports this host's GPU for a speedup, or "
"continue on CPU (slower).")
elif _rs == "cpu_only":
r_status, r_detail, r_fix = "pass", f"{_eng} → cpu (no accelerator on this host)", None
elif _rs == "unavailable":
r_status, r_detail, r_fix = "fail", (
f"{_eng} can't run on this host: {_why or 'needs a GPU this machine lacks'}"), (
"Select an engine with a CPU path in Settings → Engines.")
else: # "none" / unknown
r_status, r_detail, r_fix = "warn", "No active TTS engine resolved for routing.", (
"Pick an engine in Settings → Engines.")
checks.append({
"id": "gpu_routing", "label": "Active engine routing",
"status": r_status, "detail": r_detail, "fix": r_fix,
})
# ── Network — probes the HF endpoint actually in effect (mirror-aware),
# and a dead network is a WARNING, not a blocker. The app is local-first:
# already-downloaded models work offline, and a hard fail here dead-ends
# restricted-network users (e.g. China, where huggingface.co is blocked)
# on the very first screen — before they can reach the mirror setting
# that fixes it. Model downloads surface their own actionable errors.
net_host, net_port = _hf_endpoint_host()
net_ok = _probe_network(net_host, net_port)
mirror_reachable = False
if not net_ok and net_host == "huggingface.co":
# Official endpoint blocked — if the community mirror is reachable,
# tell the user exactly which switch unblocks them.
mirror_reachable = _probe_network("hf-mirror.com")
if net_ok:
net_fix = None
elif mirror_reachable:
net_fix = (
"huggingface.co is blocked on this network, but the hf-mirror.com "
"community mirror is reachable — apply it below and re-check. "
"Model downloads will use the mirror immediately."
)
elif net_host != "huggingface.co":
net_fix = (
f"Your configured Hugging Face mirror ({net_host}) is unreachable "
"— it may be down or blocked. Pick another mirror or the official "
"endpoint below, or continue offline: models already downloaded "
"keep working."
)
else:
net_fix = (
"Check internet connection, VPN, or corporate firewall whitelist "
"for huggingface.co. You can continue — models already downloaded "
"keep working offline; new downloads need a connection or a "
"mirror (configurable below)."
)
# ── Network
net_ok = _probe_network()
checks.append({
"id": "network", "label": f"Network ({net_host})",
"status": "pass" if net_ok else "warn",
"detail": "Reachable" if net_ok else f"Unreachable on port {net_port}",
"fix": net_fix,
# Frontend affordance hint: the wizard offers the mirror quick-pick
# when the endpoint is unreachable (PreflightCheck allows extras).
"mirror_reachable": mirror_reachable,
"id": "network", "label": "Network (huggingface.co)",
"status": "pass" if net_ok else "fail",
"detail": "Reachable" if net_ok else "Unreachable on port 443",
"fix": None if net_ok else
"Check internet connection, VPN, or corporate firewall "
"whitelist for huggingface.co.",
})
# Aggregate
@@ -446,14 +400,9 @@ def preflight():
"gpu_available": gpu["available"],
"gpu_driver": gpu["driver"],
"gpu_device_name": gpu["device_name"],
# Canonical probe (distinguishes ROCm from CUDA):
"gpu_family": (gpu_routing or {}).get("host_family", "cpu"),
"vram_gb": (gpu_routing or {}).get("vram_gb", 0.0),
"ram_gb": round(ram, 1),
"disk_free_gb": round(free, 1),
},
"gpu_routing": gpu_routing,
"media_tools": media_tools,
}
+93 -130
View File
@@ -18,7 +18,7 @@ import shutil
from core.config import OUTPUTS_DIR, DATA_DIR, CRASH_LOG_PATH, LOG_PATH, IDLE_TIMEOUT_SECONDS
from core.version import APP_VERSION
from services.model_manager import get_model_status, get_best_device, resolve_omnivoice_checkpoint
from services.model_manager import get_model_status, get_best_device
from services.ffmpeg_utils import find_ffmpeg, run_ffmpeg
# Router-level loopback gate. Every route mounted on `router` (GET + POST,
@@ -107,52 +107,6 @@ def _ui_port() -> int:
return 3901
def _fast_download_status() -> dict:
"""Report the download-acceleration state for the Settings UI (FDL-03).
Reports the *runtime* truth, not just whether hf_xet is importable. The app
currently sets ``HF_HUB_DISABLE_XET=1`` by default (main.py) Xet's chunked
transfer is fast but its progress bypasses our tqdm patch, so the legacy-LFS
path is forced to keep accurate byte progress. So:
* ``xet_installed`` hf_xet present
* ``xet_active`` installed AND not disabled via HF_HUB_DISABLE_XET
* ``xet_enabled`` alias of xet_active (what the UI badge keys off)
Must never throw: /system/info is called on every Settings load.
"""
installed = False
version = None
try:
import hf_xet # noqa: F401
installed = True
try:
from importlib.metadata import version as _ver
version = _ver("hf-xet")
except Exception:
version = None
except Exception:
installed = False
disabled = str(os.environ.get("HF_HUB_DISABLE_XET", "")).strip().lower() in {"1", "true", "yes", "on"}
active = installed and not disabled
try:
from core import prefs
high_perf = prefs.resolve(
"xet_high_performance", env="HF_XET_HIGH_PERFORMANCE", default=False
)
high_perf = high_perf if isinstance(high_perf, bool) else \
str(high_perf).strip().lower() in {"1", "true", "yes", "on"}
except Exception:
high_perf = False
return {
"xet_installed": installed,
"xet_active": active,
"xet_enabled": active, # UI badge: only true when Xet actually runs
"xet_version": version,
"high_performance": bool(high_perf),
}
def _has_hf_token() -> bool:
# Phase 1 AUTH-01..06 cascade. Delegates to the 3-source resolver
# (App → Env → HF-CLI) instead of reading env/HF-CLI directly. This
@@ -174,23 +128,88 @@ def model_status():
@router.get("/model/loaded")
def loaded_models():
"""List all currently loaded models for the flush dropdown (MM2-04).
Thin delegation to the model_lifecycle facade shape unchanged:
``{models, count}``."""
from services import model_lifecycle
return model_lifecycle.list_loaded()
"""Return details about all currently loaded models for the flush dropdown.
Returns a list of models with name, type, device, and estimated VRAM usage.
"""
import services.model_manager as mm
models = []
# 1. TTS model (OmniVoice)
if mm.model is not None:
device = "unknown"
vram_mb = 0
try:
device = str(next(mm.model.parameters()).device) if hasattr(mm.model, 'parameters') else get_best_device()
except Exception:
device = get_best_device()
try:
torch = mm._lazy_torch()
if torch.cuda.is_available():
vram_mb = torch.cuda.memory_allocated() / (1024 ** 2)
elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
driver = getattr(torch.mps, "driver_allocated_memory", None)
if driver:
vram_mb = driver() / (1024 ** 2)
except Exception:
pass
models.append({
"id": "tts",
"name": "OmniVoice TTS",
"checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
"device": device,
"vram_mb": round(vram_mb, 1),
"unloadable": True,
})
# 2. ASR model (WhisperX)
if mm.model is not None and hasattr(mm.model, '_asr_pipe') and mm.model._asr_pipe is not None:
models.append({
"id": "asr",
"name": "WhisperX ASR",
"checkpoint": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
"device": "cpu",
"vram_mb": 0,
"unloadable": False, # tied to TTS model lifecycle
})
# 3. Diarization pipeline
if mm._diar_pipeline is not None:
models.append({
"id": "diarization",
"name": "Pyannote Diarization",
"checkpoint": "pyannote/speaker-diarization-3.1",
"device": get_best_device(),
"vram_mb": 0,
"unloadable": True,
})
return {"models": models, "count": len(models)}
@router.post("/model/unload/{model_id}")
async def unload_model(model_id: str):
"""Unload a specific model by id (MM2-04). Delegates to model_lifecycle;
an unknown id maps to HTTP 400. ``tts`` | ``diarization`` |
``sidecar:<id>`` | ``sidecars``."""
from services import model_lifecycle
try:
return await model_lifecycle.unload(model_id)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
"""Unload a specific model by ID."""
import services.model_manager as mm
if model_id == "tts":
async with mm._model_lock:
if mm.model is not None:
mm.model = None
mm.free_vram()
return {"unloaded": "tts", "success": True}
return {"unloaded": "tts", "success": False, "reason": "not loaded"}
elif model_id == "diarization":
if mm._diar_pipeline is not None:
mm._diar_pipeline = None
mm.free_vram()
return {"unloaded": "diarization", "success": True}
return {"unloaded": "diarization", "success": False, "reason": "not loaded"}
else:
raise HTTPException(status_code=400, detail=f"Unknown model id: {model_id}")
@router.get("/system/info", response_model=SystemInfoResponse)
@@ -208,11 +227,10 @@ def system_info():
"outputs_dir": OUTPUTS_DIR,
"crash_log_path": CRASH_LOG_PATH,
"idle_timeout_seconds": IDLE_TIMEOUT_SECONDS,
"model_checkpoint": resolve_omnivoice_checkpoint(), # #693: show the effective checkpoint, not a leaked raw value
"model_checkpoint": os.environ.get("OMNIVOICE_MODEL", "k2-fsa/OmniVoice"),
"asr_model": os.environ.get("ASR_MODEL", "Systran/faster-whisper-large-v3"),
"translate_provider": os.environ.get("TRANSLATE_PROVIDER", "google"),
"has_hf_token": _has_hf_token(),
"fast_download": _fast_download_status(),
"device": get_best_device(),
"python": sys.version.split()[0],
"platform": sys.platform,
@@ -304,10 +322,9 @@ def _tauri_log_candidates():
os.path.join(home, "Library/Logs/OmniVoice/backend_err.log"),
]
if sys.platform.startswith("linux"):
data_dir = os.environ.get("XDG_DATA_HOME") or os.path.join(home, ".local/share")
state_dir = os.environ.get("XDG_STATE_HOME") or os.path.join(home, ".local/state")
return [
os.path.join(data_dir, bid, "logs", "tauri.log"),
os.path.join(home, ".local/share", bid, "logs", "tauri.log"),
os.path.join(home, ".config", bid, "logs", "tauri.log"),
os.path.join(state_dir, "OmniVoice", "backend.log"),
os.path.join(state_dir, "OmniVoice", "backend_err.log"),
@@ -447,14 +464,6 @@ async def clear_system_logs():
status_code=500,
detail=f"Could not clear log at {p}: {e}. The file may be open in another process or read-only — close tailing tools and retry.",
)
if cleared_any:
# The crash log just shrank to zero — drop any stored ack so a stale
# byte count can't suppress the next 'crash-last-session' notice.
for key in ("crash_log_acked", "crash_log_acked_size"):
try:
prefs_delete(key)
except Exception:
pass
return {"cleared": cleared_any}
@@ -556,21 +565,6 @@ async def flush_memory(unload_model: bool = False):
# ── Actionable notifications ──────────────────────────────────────────────
_GPU_ARCH_WARNING: "list[str | None]" = [] # [-1] = computed result
def _gpu_arch_warning_cached() -> "str | None":
"""check_device_compatibility() once per process (it lazy-imports torch —
too heavy for the 30s notifications poll)."""
if not _GPU_ARCH_WARNING:
try:
from services.model_manager import check_device_compatibility
compatible, warning = check_device_compatibility()
_GPU_ARCH_WARNING.append(None if compatible else warning)
except Exception:
_GPU_ARCH_WARNING.append(None)
return _GPU_ARCH_WARNING[-1]
@router.get("/system/notifications")
def system_notifications():
@@ -602,20 +596,6 @@ def system_notifications():
},
})
# 1b. GPU compute capability unsupported by this torch build (#284) —
# the model "runs" but emits pure noise, the worst silent failure mode
# (RTX 50-series Blackwell sm_120 on pre-cu128 wheels). The loader logs
# this, but a log line never reached the affected users — surface it in
# the panel. Checked once per process: it lazy-imports torch.
gpu_warn = _gpu_arch_warning_cached()
if gpu_warn:
notes.append({
"id": "gpu-arch-unsupported",
"level": "error",
"title": "GPU not supported by this PyTorch build",
"message": gpu_warn + " Until then, output will be noise/garbage.",
})
# 2. Missing ffmpeg
ffmpeg_ok = False
try:
@@ -629,17 +609,15 @@ def system_notifications():
notes.append({
"id": "ffmpeg-missing",
"level": "error",
"title": "Media engine unavailable",
"title": "ffmpeg not found",
"message": (
"Video processing, audio conversion, and dubbing need the "
"media engine (ffmpeg), which the app normally provisions "
"itself. Open Settings > Audio tools and press Restore "
"bundled to re-download it, or point it at a system copy."
"Video processing, audio conversion, and dubbing require ffmpeg. "
"Install it with: brew install ffmpeg (macOS) or apt install ffmpeg (Linux)."
),
"action": {
"label": "Open Audio tools",
"type": "settings-tab",
"target": "audio-tools",
"label": "Install guide",
"type": "link",
"target": "https://ffmpeg.org/download.html",
},
})
@@ -708,33 +686,18 @@ def _crashed_last_session() -> bool:
if not os.path.exists(CRASH_LOG_PATH):
return False
size = os.path.getsize(CRASH_LOG_PATH)
if size == 0:
acked = int(prefs_get("crash_log_acked_size", 0) or 0)
if size <= acked:
return False
mtime = os.path.getmtime(CRASH_LOG_PATH)
# Composite ack (size + mtime): a bare byte count goes stale after the log
# is truncated — the next crash log can stay smaller than the old acked
# size forever, silently suppressing 'crash-last-session'. The ack only
# holds while it still covers the file's current state.
ack = prefs_get("crash_log_acked")
if isinstance(ack, dict):
if float(ack.get("mtime", 0) or 0) >= mtime and int(ack.get("size", 0) or 0) >= size:
return False
else:
# Legacy size-only ack from older builds.
if size <= int(prefs_get("crash_log_acked_size", 0) or 0):
return False
return mtime < _PROCESS_START_TS
return os.path.getmtime(CRASH_LOG_PATH) < _PROCESS_START_TS
@router.post("/system/crash/ack")
async def ack_crash():
"""Mark the current crash log as seen — dismisses the
'crash-last-session' notification until the log changes again."""
size = mtime = 0
if os.path.exists(CRASH_LOG_PATH):
size = os.path.getsize(CRASH_LOG_PATH)
mtime = os.path.getmtime(CRASH_LOG_PATH)
prefs_set("crash_log_acked", {"size": size, "mtime": mtime})
'crash-last-session' notification until the log grows again."""
size = os.path.getsize(CRASH_LOG_PATH) if os.path.exists(CRASH_LOG_PATH) else 0
prefs_set("crash_log_acked_size", size)
return {"acked_size": size}
-5
View File
@@ -77,10 +77,6 @@ async def probe(req: ProbeReq):
class IncrementalReq(BaseModel):
segments: list[dict]
stored_hashes: Optional[dict[str, str]] = None
# P1.3 — the ACTIVE track's language code. When set, fingerprints are
# scoped to that language (pass that language's stored hashes alongside);
# omitted → legacy language-agnostic hashing, kept for old callers.
lang: Optional[str] = None
@router.post("/tools/incremental")
@@ -88,7 +84,6 @@ def plan_incremental(req: IncrementalReq):
return incremental.plan_incremental(
req.segments,
stored_hashes=req.stored_hashes or {},
track_lang=req.lang,
)
+35 -104
View File
@@ -98,30 +98,6 @@ async def ws_tts(websocket: WebSocket):
model = await get_model()
backend = get_active_tts_backend(model=model)
# ── Routing gate (#21 — no silent CPU fallback). WebSockets have
# no response headers, so this uses frames: an error frame +
# close on `unavailable`, a one-time `routing` frame on
# cpu_fallback / accelerated-with-caveat (before any audio).
from core.device_caps import detect_host_caps
from services.engine_routing import resolve_routing, routing_notice
from core.scrub import scrub_text
_routing = resolve_routing(
getattr(backend, "gpu_compat", ("cpu",)), detect_host_caps())
if _routing["routing_status"] == "unavailable":
await websocket.send_json({
"type": "error",
"detail": scrub_text(_routing["routing_reason"])
or "engine cannot run on this host",
})
continue # don't stream; wait for the next request
_notice = routing_notice(_routing)
if _notice:
await websocket.send_json({
"type": "routing",
"status": _notice[0],
"reason": scrub_text(_notice[1]) if _notice[1] else None,
})
# Build generation kwargs
kw: dict = {"speed": data.get("speed", 1.0)}
if data.get("language"):
@@ -136,10 +112,7 @@ async def ws_tts(websocket: WebSocket):
kw["emo_text"] = data["emo_text"]
if data.get("emo_audio"):
kw["emo_audio"] = data["emo_audio"]
# Default 1.0 when absent: a missing key must not trip the
# `!= 1.0` branch into a KeyError (any minimal request that
# omitted emo_alpha got an error frame instead of audio).
if data.get("emo_alpha", 1.0) != 1.0:
if data.get("emo_alpha") != 1.0:
kw["emo_alpha"] = data["emo_alpha"]
# Resolve voice profile
@@ -171,92 +144,50 @@ async def ws_tts(websocket: WebSocket):
except Exception:
kw["voice"] = voice
# Engine-agnostic text normalization (junk strip,
# numbers→words, abbreviations) — the same pre-pass as
# /generate, applied exactly ONCE per request, on the whole
# text BEFORE the sentence chunker fans it out (so per-sentence
# generates never re-normalize, and expanded abbreviations
# can't confuse the sentence splitter). The request's
# `language` is all this route knows (None → universal safety
# filters only). Pref-gated (default ON), idempotent, never
# raises.
from services.text_normalization import normalize_for_tts
text = normalize_for_tts(text, data.get("language"))
# Wave 1.4: split the request into sentences so the first
# sentence's audio streams while later sentences are still
# synthesizing — this is the time-to-first-audio win. The
# chunker handles abbreviations/acronyms/decimals and CJK /
# non-Latin terminators; single-sentence requests behave
# exactly like the old single-shot path.
from services.sentence_chunker import SentenceChunker
_chunker = SentenceChunker(language=(data.get("language") or "en"))
sentences = _chunker.push(text)
sentences.extend(_chunker.flush())
if not sentences:
sentences = [text]
# Run generation in the GPU pool
import functools
from services.model_manager import run_on_gpu_pool_guarded
from services.model_manager import _gpu_pool
loop = asyncio.get_running_loop()
def _generate(sentence_text):
def _generate():
from services.audio_dsp import apply_mastering, normalize_audio
wav = backend.generate(sentence_text, **kw)
wav = backend.generate(text, **kw)
sr_actual = backend.sample_rate
# Like _run_tts in openai_compat: studio engines (VoxCPM2)
# opt out of the broadcast mastering chain. This is the
# other route that runs the active backend, so it needs the
# same guard. Loudness normalisation still runs.
if not getattr(backend, "applies_own_mastering", False):
wav = apply_mastering(wav, sample_rate=sr_actual)
wav = apply_mastering(wav, sample_rate=sr_actual)
wav = normalize_audio(wav, target_dBFS=-2.0)
return wav, sr_actual
wav_tensor, sr = await loop.run_in_executor(_gpu_pool, _generate)
# Send metadata after generation so sample_rate is real
await websocket.send_json({
"type": "start",
"sample_rate": sr,
"channels": 1,
"format": "pcm16",
"engine": backend.id,
})
# Stream PCM16 chunks over the WebSocket
import torch
total_samples = 0
sr = backend.sample_rate
started = False
# Convert to 16-bit PCM
pcm = (wav_tensor * 32767).clamp(-32768, 32767).to(torch.int16)
if pcm.ndim == 2:
pcm = pcm[0] # mono
pcm_bytes = pcm.numpy().tobytes()
for sentence in sentences:
# Bounded + pool-reset on hang so a wedged generate can't
# starve the GPU pool and brick the backend (#730 class). On
# timeout GpuJobTimeoutError propagates to the handler below,
# which sends an actionable error frame.
wav_tensor, sr = await run_on_gpu_pool_guarded(
functools.partial(_generate, sentence),
what="TTS generate",
)
total_samples = len(pcm)
sent_samples = 0
chunk_bytes = CHUNK_SAMPLES * 2 # 2 bytes per int16 sample
if not started:
# Send metadata after the first generation so
# sample_rate is real (lazy-loading engines report
# their true rate only once weights are up).
await websocket.send_json({
"type": "start",
"sample_rate": sr,
"channels": 1,
"format": "pcm16",
"engine": backend.id,
})
started = True
# Convert to 16-bit PCM and stream
pcm = (wav_tensor * 32767).clamp(-32768, 32767).to(torch.int16)
if pcm.ndim == 2:
pcm = pcm[0] # mono
pcm_bytes = pcm.numpy().tobytes()
n_samples = len(pcm)
sent_samples = 0
while sent_samples < n_samples:
end = min(sent_samples + CHUNK_SAMPLES, n_samples)
chunk = pcm_bytes[sent_samples * 2: end * 2]
await websocket.send_bytes(chunk)
sent_samples = end
# Yield to event loop between chunks for responsiveness
await asyncio.sleep(0)
total_samples += n_samples
while sent_samples < total_samples:
end = min(sent_samples + CHUNK_SAMPLES, total_samples)
start_byte = sent_samples * 2
end_byte = end * 2
chunk = pcm_bytes[start_byte:end_byte]
await websocket.send_bytes(chunk)
sent_samples = end
# Yield to event loop between chunks for responsiveness
await asyncio.sleep(0)
gen_time = round(time.perf_counter() - t0, 3)
duration = round(total_samples / sr, 3)
-32
View File
@@ -34,8 +34,6 @@ class SystemInfoResponse(BaseModel):
asr_model: str = "unknown"
translate_provider: str = "unknown"
has_hf_token: bool = False
# Xet fast-download backend state (FDL-03): {xet_enabled, xet_version, high_performance}
fast_download: dict | None = None
device: str = "cpu"
python: str = ""
platform: str = ""
@@ -129,46 +127,16 @@ class DeviceInfo(BaseModel):
gpu_available: bool = False
gpu_driver: str | None = None
gpu_device_name: str | None = None
# From the canonical device probe (core.device_caps) — distinguishes ROCm
# from CUDA, unlike the legacy nvidia-smi-based gpu_vendor/gpu_backend.
gpu_family: str = "cpu"
vram_gb: float = 0.0
ram_gb: float = 0.0
disk_free_gb: float = 0.0
class GpuRouting(BaseModel):
"""Routing verdict for the active TTS engine on THIS host (#21).
Distinct from the per-engine `routing_*` keys in `/engines`: this is the
single verdict for the *currently-selected* engine, surfaced in preflight +
diagnose so the user hears about a CPU fallback / unavailable GPU before a
slow or failed synth no silent CPU fallback.
"""
model_config = ConfigDict(extra="allow")
engine: str | None = None # active TTS engine id
effective_device: str | None = None # device it will actually use here
routing_status: str | None = None # accelerated|cpu_fallback|cpu_only|unavailable|none
routing_reason: str | None = None # scrubbed; null when none
host_family: str = "cpu" # detect_host_caps().family
vram_gb: float = 0.0
class PreflightResponse(BaseModel):
"""GET /setup/preflight"""
ok: bool
has_warnings: bool = False
checks: list[PreflightCheck] = Field(default_factory=list)
device: DeviceInfo
# Explicit field (PreflightResponse has no extra="allow") so the verdict
# survives serialization instead of being silently dropped.
gpu_routing: GpuRouting | None = None
# Media-engine verdict (ffmpeg/ffprobe) — NOT a check row: an internal
# dependency the app provisions for itself. Shape: {ready, acquire:
# {state, progress, error}}. The wizard renders a quiet progress line /
# failure card from it instead of "install ffmpeg" system requirements.
media_tools: dict | None = None
class InstallModelRequest(BaseModel):
Binary file not shown.
-75
View File
@@ -14,10 +14,6 @@
# required (optional) — true if the app needs this model to function
# platforms (optional) — restrict to specific OS+arch tags (e.g. darwin-arm64, cuda)
# note (optional) — shown in the UI as a tooltip/footnote
# config_only (optional) — true for pipeline repos that ship no weight file of
# their own (weights live in referenced sub-repos). Such
# a cache is legitimately tiny, so the truncated-download
# (weights-missing) detector must NOT flag it incomplete.
# ─────────────────────────────────────────────────────────────────────────
models:
@@ -120,83 +116,12 @@ models:
size_gb: 0.05
note: "Smallest/fastest Moonshine, sub-200ms latency. Lower accuracy than base. Requires moonshine-onnx."
# ── sherpa-onnx live dictation (ONNX, CPU, streaming + offline) ────────
# Live faster-than-real-time dictation via the k2-fsa/sherpa-onnx runtime.
# `engine: sherpa-onnx`, `dictation_id` (backend model id), and `tag`
# (offline | streaming) are extra fields the model-store list passes through
# so the dictation UI can filter/group these (role=ASR, engine=sherpa-onnx).
# Requires `uv add sherpa-onnx` (CPU wheels, all platforms).
- repo_id: "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8"
label: "Parakeet TDT v3 (sherpa-onnx — dictation, 25 EU langs)"
role: ASR
size_gb: 0.18
engine: sherpa-onnx
dictation_id: sherpa-parakeet-tdt-v3
tag: offline
note: "Recommended live-dictation default. CPU, int8 ONNX. Requires sherpa-onnx."
- repo_id: "csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8"
label: "Parakeet TDT v2 (sherpa-onnx — dictation, English)"
role: ASR
size_gb: 0.17
engine: sherpa-onnx
dictation_id: sherpa-parakeet-tdt-v2
tag: offline
note: "English live dictation. CPU, int8 ONNX. Requires sherpa-onnx."
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20"
label: "Zipformer Bilingual (sherpa-onnx — streaming, zh+en)"
role: ASR
size_gb: 0.13
engine: sherpa-onnx
dictation_id: sherpa-zipformer-bilingual-zh-en
tag: streaming
note: "True streaming partials as you speak (zh+en). CPU. Requires sherpa-onnx."
- repo_id: "csukuangfj/sherpa-onnx-streaming-paraformer-bilingual-zh-en"
label: "Paraformer Bilingual (sherpa-onnx — streaming, zh+en)"
role: ASR
size_gb: 0.115
engine: sherpa-onnx
dictation_id: sherpa-paraformer-bilingual-zh-en
tag: streaming
note: "True streaming partials (zh+en). CPU. Requires sherpa-onnx."
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-en-20M-2023-02-17"
label: "Zipformer Streaming EN 20M (sherpa-onnx — streaming, English)"
role: ASR
size_gb: 0.128
engine: sherpa-onnx
dictation_id: sherpa-zipformer-en-20m
tag: streaming
note: "Tiny English streaming model, very low latency. CPU. Requires sherpa-onnx."
- repo_id: "csukuangfj/sherpa-onnx-streaming-zipformer-zh-14M-2023-02-23"
label: "Zipformer Streaming ZH 14M (sherpa-onnx — streaming, Chinese)"
role: ASR
size_gb: 0.074
engine: sherpa-onnx
dictation_id: sherpa-zipformer-zh-14m
tag: streaming
note: "Tiny Chinese streaming model, very low latency. CPU. Requires sherpa-onnx."
- repo_id: "csukuangfj/sherpa-onnx-whisper-tiny"
label: "Whisper Tiny (sherpa-onnx — dictation, 90+ langs)"
role: ASR
size_gb: 0.116
engine: sherpa-onnx
dictation_id: sherpa-whisper-tiny
tag: offline
note: "Multilingual offline dictation (auto-detect). CPU, int8 ONNX. Requires sherpa-onnx."
# ── Diarisation ───────────────────────────────────────────────────────
- repo_id: "pyannote/speaker-diarization-3.1"
label: "pyannote speaker diarisation (multi-speaker videos)"
role: Diarisation
size_gb: 0.8
config_only: true # pipeline repo; real weights live in referenced sub-repos
note: "Needs an HF_TOKEN with license accepted."
# ── Optional TTS ──────────────────────────────────────────────────────
-138
View File
@@ -1,138 +0,0 @@
"""Parse the shipped CHANGELOG.md into structured release notes.
Feeds ``GET /api/settings/changelog`` the Settings Updates "What's new"
viewer. Local-first by design: the changelog ships with the app (repo root in
dev; copied into the packaged project dir by the Tauri bootstrap alongside
README.md), so the viewer works fully offline.
The house format (see CHANGELOG.md / the release-notes hard rule):
## [X.Y.Z] — DATE
one-paragraph headline (the "intro")
### Added / Fixed / Changed / ...
- **Bold one-line lead.** 1-3 lines of plain-English why. (#NNN)
Bullets may be a single long line (recent sections) *or* hard-wrapped across
indented continuation lines (older sections) the parser normalizes both to
one logical line per bullet. Bullets stay raw markdown-lite; the frontend's
safe renderer handles **bold** / `code` / (#NNN) refs.
"""
from __future__ import annotations
import os
import re
#: ``## [0.3.9] — 2026-07-02`` (em/en dash or hyphen; date optional).
_RELEASE_RE = re.compile(r"^##\s+\[(?P<version>[^\]]+)\]\s*(?:[—–-]\s*(?P<date>.+?))?\s*$")
_SECTION_RE = re.compile(r"^###\s+(?P<title>.+?)\s*$")
_BULLET_RE = re.compile(r"^\s*[-*]\s+(?P<text>.*\S)\s*$")
def changelog_path() -> str | None:
"""The shipped CHANGELOG.md, or None when this install doesn't have one.
``backend/core/changelog.py`` two levels up is the project root: the
repo root in dev, and ``<env>/project`` in packaged installs (where the
bootstrap copies CHANGELOG.md next to README.md). ``OMNIVOICE_CHANGELOG``
overrides for tests/containers.
"""
override = os.environ.get("OMNIVOICE_CHANGELOG")
if override:
return override if os.path.isfile(override) else None
here = os.path.dirname(os.path.abspath(__file__))
candidate = os.path.join(os.path.dirname(os.path.dirname(here)), "CHANGELOG.md")
return candidate if os.path.isfile(candidate) else None
def _looks_like_release_version(version: str) -> bool:
"""Only released ``X.Y.Z...`` sections (skip ``[Unreleased]`` etc.)."""
return bool(re.match(r"^v?\d", version.strip()))
def parse_changelog(text: str, limit_versions: int = 5) -> list[dict]:
"""CHANGELOG.md text → newest-first list of releases::
{"version": "0.3.9", "date": "2026-07-02", "intro": "",
"sections": [{"title": "Fixed", "bullets": ["", ]}, ]}
Tolerates both single-line bullets and older hard-wrapped bullets
(continuation lines are joined with a space). Content between the version
heading and the first ``###`` becomes ``intro`` (paragraphs joined by
blank lines).
"""
releases: list[dict] = []
release: dict | None = None
section: dict | None = None
intro_parts: list[str] = []
bullet_open = False # last bullet may still absorb continuation lines
intro_new_para = True
def close_release():
nonlocal release, section, intro_parts, bullet_open, intro_new_para
if release is not None:
release["intro"] = "\n\n".join(p for p in intro_parts if p)
release["sections"] = [s for s in release["sections"] if s["bullets"]]
releases.append(release)
release = None
section = None
intro_parts = []
bullet_open = False
intro_new_para = True
for raw in text.splitlines():
m = _RELEASE_RE.match(raw)
if m:
close_release()
if len(releases) >= limit_versions:
break
version = m.group("version").strip().lstrip("v")
if not _looks_like_release_version(version):
continue # e.g. [Unreleased] — skip until the next heading
release = {
"version": version,
"date": (m.group("date") or "").strip(),
"intro": "",
"sections": [],
}
continue
if release is None:
continue
line = raw.strip()
if not line:
bullet_open = False
intro_new_para = True
continue
sm = _SECTION_RE.match(raw)
if sm:
section = {"title": sm.group("title"), "bullets": []}
release["sections"].append(section)
bullet_open = False
continue
bm = _BULLET_RE.match(raw)
if bm:
if section is None:
# Rare: a bullet before any ### heading — group it untitled.
section = {"title": "", "bullets": []}
release["sections"].append(section)
section["bullets"].append(bm.group("text"))
bullet_open = True
continue
if section is not None:
if bullet_open and section["bullets"]:
# Hard-wrapped bullet continuation (older sections) → join.
section["bullets"][-1] += " " + line
continue
# Headline paragraph(s) before the first ### section.
if intro_new_para or not intro_parts:
intro_parts.append(line)
else:
intro_parts[-1] += " " + line
intro_new_para = False
close_release()
return releases[:limit_versions]
+8 -247
View File
@@ -3,8 +3,6 @@ import sqlite3
import logging
from contextlib import contextmanager
from core.config import DB_PATH
from core import db_backup
from core.version import APP_VERSION
logger = logging.getLogger("omnivoice.db")
@@ -51,12 +49,6 @@ _BASE_SCHEMA = """
personality TEXT DEFAULT '',
description TEXT DEFAULT '',
is_demo INTEGER DEFAULT 0,
verified_own_voice INTEGER DEFAULT 0,
consent_text TEXT DEFAULT '',
consent_audio_path TEXT DEFAULT '',
consent_recorded_at REAL DEFAULT NULL,
kind TEXT DEFAULT 'clone',
vd_states TEXT DEFAULT NULL,
created_at REAL
);
CREATE TABLE IF NOT EXISTS generation_history (
@@ -70,7 +62,6 @@ _BASE_SCHEMA = """
duration_seconds REAL,
generation_time REAL,
seed INTEGER DEFAULT NULL,
starred INTEGER DEFAULT 0,
created_at REAL,
FOREIGN KEY (profile_id) REFERENCES voice_profiles(id)
);
@@ -147,35 +138,6 @@ _BASE_SCHEMA = """
value TEXT NOT NULL,
updated_at REAL NOT NULL
);
-- Wave 2.2: per-agent MCP voice bindings. An MCP client (Claude Code,
-- Cursor, ) identified by the X-OmniVoice-Client-Id header it sends is
-- bound to a default voice profile / engine. Fresh installs create it
-- here; v0.3.x upgrades get it via alembic 0004.
CREATE TABLE IF NOT EXISTS mcp_client_bindings (
client_id TEXT PRIMARY KEY,
label TEXT NOT NULL DEFAULT '',
profile_id TEXT,
default_engine TEXT,
last_seen_at REAL,
created_at REAL
);
-- Expressive-TTS Spec 01 Phase 1: user pronunciation dictionary. A
-- per-language wordrespelling map applied as pure text substitution
-- before synthesis (Settings Pronunciation). Fresh installs create it
-- here; existing DBs get it via alembic 0008_pronunciation_dictionary.
-- Both paths converge on this identical schema (dual-path discipline).
CREATE TABLE IF NOT EXISTS pronunciation_entries (
id TEXT PRIMARY KEY,
term TEXT NOT NULL,
replacement TEXT NOT NULL DEFAULT '',
type TEXT NOT NULL DEFAULT 'respelling',
language TEXT NOT NULL DEFAULT '*',
enabled INTEGER NOT NULL DEFAULT 1,
created_at REAL
);
CREATE INDEX IF NOT EXISTS idx_pron_lang ON pronunciation_entries(language);
"""
# Only tables/columns this module is allowed to ALTER. Prevents SQL injection via
@@ -225,72 +187,6 @@ def _migrate(conn, current: int) -> int:
return current
def _reconcile_additive_columns(conn) -> None:
"""Make the live schema converge to ``_BASE_SCHEMA`` by ADDing any column the
canonical schema declares but an existing table is missing the belt for
when alembic can't run on an upgraded DB.
``CREATE TABLE IF NOT EXISTS`` (init_db) never adds columns to a table that
already exists, the legacy ``_migrate`` only knows pre-0.3 columns, and
``_run_alembic_upgrade`` swallows failures. So a DB whose ``alembic_version``
is stamped at a removed revision (e.g. after running a preview build), or
where alembic isn't importable in the bundled interpreter, would otherwise
lose every alembic-era additive column forever the ``no such column:
consent_audio_path`` 500 (#552/#547), and the same class for
``kind``/``vd_states``/``is_demo``/.... Additive only: never drops or retypes
a column, so it is safe and backward-compatible with existing user data. The
canonical names/types/defaults come solely from ``_BASE_SCHEMA`` (developer
controlled), so the ALTER is injection-safe.
"""
canon = sqlite3.connect(":memory:")
try:
canon.executescript(_BASE_SCHEMA)
_tables_sql = "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'"
live_tables = {r[0] for r in conn.execute(_tables_sql)}
for table in (r[0] for r in canon.execute(_tables_sql)):
if table not in live_tables:
continue # whole table missing → init_db's CREATE already made it
have = {r[1] for r in conn.execute(f"PRAGMA table_info({table})")}
# (cid, name, type, notnull, dflt_value, pk)
for _cid, name, ctype, notnull, dflt, _pk in canon.execute(f"PRAGMA table_info({table})"):
if name in have or not _IDENT_RE.match(name):
continue
ddl = f'ALTER TABLE "{table}" ADD COLUMN "{name}" {ctype or "TEXT"}'
if dflt is not None:
ddl += f" DEFAULT {dflt}"
elif notnull:
ddl += " DEFAULT ''" # SQLite requires a default to ADD a NOT NULL column
try:
conn.execute(ddl)
logger.info("schema reconcile: added missing column %s.%s", table, name)
except sqlite3.OperationalError as exc:
if "duplicate column" not in str(exc).lower():
logger.warning("schema reconcile ALTER %s.%s failed: %s", table, name, exc)
conn.commit()
finally:
canon.close()
def ensure_schema() -> None:
"""Idempotently ensure the base tables + additive columns exist.
A runtime self-heal for a DB that somehow missed init e.g. a write hitting
``no such table: generation_history`` (#710) because ``init_db()``'s
``executescript`` never took on that DB. Safe to call anytime: it's just
``CREATE ... IF NOT EXISTS`` plus the additive-only column reconcile, so it
never drops or retypes anything and is backward-compatible with user data.
Cheaper than ``init_db()`` (skips the legacy ``_migrate`` + alembic), so a
write path can call it on a schema error and retry without a 500.
"""
conn = get_db()
try:
conn.executescript(_BASE_SCHEMA)
_reconcile_additive_columns(conn)
conn.commit()
finally:
conn.close()
def init_db():
conn = get_db()
try:
@@ -299,11 +195,6 @@ def init_db():
new_version = _migrate(conn, version)
if new_version != version:
conn.execute(f"PRAGMA user_version = {new_version}")
# Converge any alembic-era additive columns that CREATE TABLE IF NOT
# EXISTS + the legacy _migrate don't add to a pre-existing table
# (consent_audio_path, kind, ...). Runs regardless of whether alembic
# below succeeds, so an unrunnable alembic can't leave a 500-ing schema.
_reconcile_additive_columns(conn)
conn.commit()
finally:
conn.close()
@@ -315,88 +206,12 @@ def init_db():
_run_alembic_upgrade()
class MigrationError(RuntimeError):
"""A schema migration failed *while executing*. Startup must NOT continue
on a possibly half-migrated database the caller lets this propagate so
the process stops with an actionable message naming the pre-migration
backup (see ``core.db_backup``). Restore is deliberately manual: silently
auto-restoring the snapshot could itself discard user data."""
def _reconcile_after_alembic_skip() -> None:
"""Converge the schema directly when alembic can't run at all (not
importable, or stamped at a removed revision #552/#547) so additive
columns still land instead of 500-ing on `no such column`. Only for the
"nothing was applied" classes; a mid-migration failure must NOT reach
here (see MigrationError)."""
try:
conn = get_db()
try:
_reconcile_additive_columns(conn)
finally:
conn.close()
except Exception as exc: # noqa: BLE001
logger.warning("schema reconcile after alembic skip also failed: %s", exc)
def _stamped_revisions(db_path: str) -> set | None:
"""Revisions recorded in ``alembic_version`` (empty set = never stamped),
or None when the DB can't be read."""
try:
conn = sqlite3.connect(db_path)
try:
try:
return {r[0] for r in conn.execute("SELECT version_num FROM alembic_version")}
except sqlite3.OperationalError:
return set() # table absent — nothing ever stamped
finally:
conn.close()
except Exception: # noqa: BLE001
return None
def _plan_alembic(cfg) -> str:
"""Decide what an ``upgrade head`` run would actually do:
- ``up_to_date`` stamped at head; upgrade is a no-op.
- ``pending`` migrations WILL execute (snapshot the DB first).
- ``unknown_revision`` stamped at a revision this build doesn't ship
(previewstable downgrade, #552/#547); upgrade would fail before
applying anything, so skip it and reconcile additively instead.
- ``indeterminate`` can't tell; treat like pending (snapshot, run).
"""
try:
from alembic.script import ScriptDirectory
script = ScriptDirectory.from_config(cfg)
known = {rev.revision for rev in script.walk_revisions()}
heads = set(script.get_heads())
stamped = _stamped_revisions(DB_PATH)
if stamped is None:
return "indeterminate"
if stamped and not stamped <= known:
return "unknown_revision"
if stamped == heads:
return "up_to_date"
return "pending"
except Exception: # noqa: BLE001
return "indeterminate"
def _run_alembic_upgrade() -> None:
"""`alembic upgrade head` on startup, wrapped in the data-safety net.
Failure classes are handled differently on purpose:
- alembic unavailable / stamped at an unknown revision **non-fatal**
(nothing was applied; warn + `_reconcile_additive_columns` keeps the
schema converged, exactly the pre-existing #552/#547 behavior).
- migrations actually pending the DB is snapshotted first
(``omnivoice.db.backup-<version>-<n>``, newest 3 kept), then upgraded.
- a migration fails **while executing** raise :class:`MigrationError`:
startup stops with a message naming the backup, instead of silently
running the app on a half-migrated DB.
"""
"""Best-effort `alembic upgrade head` on startup. Non-fatal: if alembic
isn't reachable (e.g. user running a stripped-down install or migrations
were already applied out-of-band), log a warning and move on. The
_BASE_SCHEMA CREATE TABLE IF NOT EXISTS above guarantees the runtime
schema is correct regardless."""
try:
import os
from alembic import command
@@ -412,62 +227,8 @@ def _run_alembic_upgrade() -> None:
return
cfg = Config(ini)
cfg.set_main_option("sqlalchemy.url", f"sqlite:///{DB_PATH}")
except Exception as exc: # noqa: BLE001 — alembic not importable / bad ini
logger.warning("alembic upgrade head skipped: %s", exc)
_reconcile_after_alembic_skip()
return
plan = _plan_alembic(cfg)
if plan == "up_to_date":
return
if plan == "unknown_revision":
logger.warning(
"alembic_version is stamped at a revision this build doesn't ship "
"(preview/newer build ran on this DB) — skipping alembic and "
"reconciling the schema additively (#552/#547)"
)
_reconcile_after_alembic_skip()
return
# Migrations may actually execute: snapshot the DB first so a failed or
# interrupted migration can never cost user data. A backup problem alone
# must not brick startup (the >500 MB skip is by design), so log and go on.
# ``db_backup``/``APP_VERSION`` are module-level imports (top of file), not
# re-imported here: a test that patches ``core.db_backup.MAX_BACKUP_DB_BYTES``
# on the object it imported at collection must see the same object this
# function uses. A lazy ``from core import db_backup`` would re-resolve
# through the (possibly re-imported) ``core`` package and silently miss the
# patch after another suite purged ``core.*`` from ``sys.modules``.
backup_path = None
try:
backup_path = db_backup.snapshot_before_migration(DB_PATH, APP_VERSION)
except Exception: # noqa: BLE001
logger.exception("Pre-migration DB backup failed — continuing without one")
try:
command.upgrade(cfg, "head")
except Exception as exc:
if "Can't locate revision" in str(exc):
# Belt for an unknown-revision case _plan_alembic missed: alembic
# bails before applying anything, so the old non-fatal path is safe.
logger.warning("alembic upgrade head skipped: %s", exc)
_reconcile_after_alembic_skip()
return
backup_note = (
f"A backup of your data from just before the migration is at: {backup_path}"
if backup_path
else "No pre-migration backup was written this run (see the log above)"
)
msg = (
f"Database migration failed while running: {exc}. "
f"OmniVoice stopped instead of running on a partially migrated database, "
f"and nothing was auto-restored (your database at {DB_PATH} was left "
f"exactly as the failed migration left it). "
f"{backup_note}. "
"What to do: relaunch to retry; if it keeps failing, report it at "
"https://github.com/debpalash/OmniVoice-Studio/issues (keep the backup file). "
"To roll back manually: quit the app, replace omnivoice.db with the backup "
"file, and reinstall the previous version."
)
logger.error(msg)
raise MigrationError(msg) from exc
# Don't block startup on a migration tooling problem. The runtime
# schema is already correct via _BASE_SCHEMA.
logger.warning("alembic upgrade head skipped: %s", exc)
-169
View File
@@ -1,169 +0,0 @@
"""Pre-migration SQLite safety net (data-safe updates).
Before ``alembic upgrade head`` applies *pending* migrations at startup
which is exactly the first launch of a new app version that changed the
schema the live database is snapshotted next to itself as
``omnivoice.db.backup-<version>-<n>`` so a failed or interrupted migration
can never cost user data (voices, projects, history, settings).
Design rules (owner intent: "never corrupt/erase user data on update"):
- Snapshots use the SQLite online-backup API (``sqlite3.Connection.backup``),
not a file copy the live DB runs in WAL mode, so a plain copy could miss
everything still sitting in ``omnivoice.db-wal``.
- Only the most recent ``KEEP_BACKUPS`` snapshots are kept; older ones are
pruned so backups can't grow without bound.
- DBs larger than ``MAX_BACKUP_DB_BYTES`` are skipped with a log line (a
multi-hundred-MB copy on every schema upgrade is worse than the risk it
hedges on those installs).
- Restore is NEVER automatic. On migration failure the caller
(``core.db._run_alembic_upgrade``) stops startup and names the backup path
so the user (or a support thread) decides a silent auto-restore could
itself discard data written after the snapshot.
"""
from __future__ import annotations
import logging
import os
import re
import sqlite3
import time
logger = logging.getLogger("omnivoice.db.backup")
#: Keep this many snapshots; older ones are pruned after each new snapshot.
KEEP_BACKUPS = 3
#: Skip the snapshot (with a log line) when the DB exceeds this size.
MAX_BACKUP_DB_BYTES = 500 * 1024 * 1024
#: ``<db name>.backup-<version>-<n>`` — ``<version>`` may itself contain
#: dashes (preview builds stamp ``0.3.9-41``), so the counter is the final
#: ``-<digits>`` group.
_BACKUP_SUFFIX_RE = re.compile(r"\.backup-(?P<version>.+)-(?P<n>\d+)$")
def _sanitize_version(version: str) -> str:
"""Version string → filesystem-safe fragment (defense in depth; real
versions are semver and already safe)."""
safe = re.sub(r"[^A-Za-z0-9._-]", "_", str(version).strip()) or "unknown"
return safe[:64]
def list_backups(db_path: str) -> list[str]:
"""All backup files for ``db_path``, newest first (mtime desc)."""
directory = os.path.dirname(os.path.abspath(db_path)) or "."
base = os.path.basename(db_path)
try:
names = os.listdir(directory)
except OSError:
return []
out = []
for name in names:
if not name.startswith(base + ".backup-"):
continue
if not _BACKUP_SUFFIX_RE.search(name[len(base):]):
continue
out.append(os.path.join(directory, name))
out.sort(key=lambda p: (_mtime(p), p), reverse=True)
return out
def _mtime(path: str) -> float:
try:
return os.path.getmtime(path)
except OSError:
return 0.0
def latest_backup(db_path: str) -> dict | None:
"""Newest backup as ``{"path", "created_at", "size_bytes"}`` or None."""
backups = list_backups(db_path)
if not backups:
return None
path = backups[0]
try:
st = os.stat(path)
except OSError:
return None
return {"path": path, "created_at": st.st_mtime, "size_bytes": st.st_size}
def _next_counter(db_path: str, safe_version: str) -> int:
"""Next free ``<n>`` for this version so a re-run never overwrites an
earlier snapshot of the same version."""
base = os.path.basename(db_path)
prefix = f"{base}.backup-{safe_version}-"
highest = 0
for path in list_backups(db_path):
name = os.path.basename(path)
if not name.startswith(prefix):
continue
tail = name[len(prefix):]
if tail.isdigit():
highest = max(highest, int(tail))
return highest + 1
def prune_backups(db_path: str, keep: int = KEEP_BACKUPS) -> list[str]:
"""Delete all but the ``keep`` newest backups. Returns deleted paths."""
deleted = []
for path in list_backups(db_path)[keep:]:
try:
os.remove(path)
deleted.append(path)
logger.info("Pruned old DB backup %s", path)
except OSError as exc:
logger.warning("Could not prune old DB backup %s: %s", path, exc)
return deleted
def snapshot_before_migration(db_path: str, version: str) -> str | None:
"""Snapshot ``db_path`` to ``<db>.backup-<version>-<n>``.
Returns the backup path, or None when skipped (no DB yet, or DB larger
than ``MAX_BACKUP_DB_BYTES``). Raises on an actual backup failure so the
caller can decide (the caller treats that as "continue without a backup",
logged loudly a backup problem must not brick startup by itself).
"""
if not os.path.isfile(db_path):
logger.debug("No DB at %s yet — nothing to back up", db_path)
return None
size = os.path.getsize(db_path)
if size > MAX_BACKUP_DB_BYTES:
logger.info(
"Skipping pre-migration DB backup: %s is %.0f MB (> %.0f MB limit)",
db_path, size / (1024 * 1024), MAX_BACKUP_DB_BYTES / (1024 * 1024),
)
return None
safe_version = _sanitize_version(version)
target = f"{db_path}.backup-{safe_version}-{_next_counter(db_path, safe_version)}"
tmp = f"{target}.part-{os.getpid()}"
src = sqlite3.connect(db_path)
try:
dst = sqlite3.connect(tmp)
try:
# Online backup: consistent snapshot including WAL contents.
src.backup(dst)
dst.commit()
finally:
dst.close()
except BaseException:
try:
os.remove(tmp)
except OSError:
pass
raise
finally:
src.close()
os.replace(tmp, target)
# A same-second rotation must still rank the new file newest.
try:
now = time.time()
os.utime(target, (now, now))
except OSError:
pass
logger.info("Pre-migration DB backup written: %s (%.1f MB)", target, size / (1024 * 1024))
prune_backups(db_path)
return target
-336
View File
@@ -1,336 +0,0 @@
"""Free-text voice-description → voice-design parameter mapper (issue #317).
Parity with the hosted omnivoice.app "Describe your voice" field, implemented
fully locally: a deterministic keyword/phrase mapper that projects a natural-
language description (e.g. ``"a warm elderly British storyteller, slightly
raspy"``) onto the **existing** voice-design parameter space — the same six
categories the Design tab's attribute picker drives (Gender / Age / Pitch /
Style / EnglishAccent / ChineseDialect).
Design notes
============
* **No model, no network.** This is an ordered synonym-table matcher, not an
LLM call it runs identically on macOS/Windows/Linux with zero deps beyond
the stdlib, preserving the local-first guarantee.
* **Single source of truth.** Every canonical token this module can emit is
validated at import time against the engine taxonomy in
``omnivoice/utils/voice_design.py`` (loaded via ``core.archetypes``), so the
mapper can never produce an instruct item the engine validator would reject
(the issue-#89 / #115 crash modes). The Chinese translations of each token
(e.g. ````/``中年``) are *derived* from that taxonomy, never hardcoded.
* **Ordered rules, first match wins.** Within a category, rules are checked in
a hand-ordered list so more specific phrases outrank generic ones
("young child" child, not young adult; "very deep" very low pitch, not
low pitch). Within one rule, the earliest occurrence in the text is reported
as the matched phrase. Deterministic by construction.
* **Graceful degradation.** Anything the taxonomy can't express (timbre words
like "raspy", role words like "storyteller") is returned in ``unmatched`` so
the UI can tell the user exactly which parts were ignored instead of failing
silently (issue #317's validation-feedback note). A description with no
matches at all yields all-``Auto`` attrs and an empty instruct.
Localization note (CLAUDE.md): the only hardcoded CJK here is
``DIALECT_PINYIN`` a functional pinyin Chinese-dialect-token mapping
(model vocabulary, like ``frontend/src/utils/constants.js``). Registered in
``tests/test_no_hardcoded_cjk.py``'s allowlist with this justification.
"""
from __future__ import annotations
import re
# Reuse the taxonomy already loaded (stdlib-only, by file path) by the
# archetype engine — same single source of truth, one loader to maintain.
from core.archetypes import _VD
_EN_TO_ZH = _VD._INSTRUCT_EN_TO_ZH # {"male": "男", ...}
_ZH_RE = _VD._ZH_RE
_VALID = _VD._INSTRUCT_ALL_VALID # every token the engine accepts
_DIALECTS = set(_VD._INSTRUCT_CATEGORIES[5]) # the 12 Chinese dialect tokens
# Category names match the frontend's CATEGORIES keys (utils/constants.js) and
# the archetype ``attrs`` shape, so the response drops straight into vdStates.
CATEGORY_ORDER = ("Gender", "Age", "Pitch", "Style", "EnglishAccent", "ChineseDialect")
# ── Pinyin / romanized names → Chinese-dialect tokens (functional vocabulary) ─
DIALECT_PINYIN = {
"henan": "河南话",
"shaanxi": "陕西话",
"sichuan": "四川话",
"szechuan": "四川话",
"guizhou": "贵州话",
"yunnan": "云南话",
"guilin": "桂林话",
"jinan": "济南话",
"shijiazhuang": "石家庄话",
"gansu": "甘肃话",
"ningxia": "宁夏话",
"qingdao": "青岛话",
"dongbei": "东北话",
"northeastern chinese": "东北话",
}
# ── Synonym tables ────────────────────────────────────────────────────────────
# Per category: ordered list of (canonical_token, [phrases]). First rule with
# any hit wins the category, so specific phrases must precede generic ones.
# Each canonical token's Chinese translation from the taxonomy is appended
# automatically at compile time (so "中年" maps to "middle-aged", etc.).
_GENDER_RULES = [
("female", [
"female", "woman", "women", "lady", "ladies", "girl", "girls",
"feminine", "gal", "grandma", "grandmother", "granny", "mother",
"mom", "mum", "aunt", "auntie", "queen", "princess", "actress",
"she", "her",
]),
("male", [
"male", "man", "men", "guy", "guys", "boy", "boys", "masculine",
"gentleman", "gentlemen", "dude", "grandpa", "grandfather", "father",
"dad", "uncle", "king", "prince", "actor", "he", "him", "his",
]),
]
# Order is load-bearing: "child" precedes "young adult" so "young child" →
# child; "middle-aged" precedes "elderly" so elderly's bare "aged" synonym
# can't fire inside the hyphenated "middle-aged" (hyphen is a \b boundary);
# "elderly" precedes "young adult" so grandparent words don't fall through.
_AGE_RULES = [
("child", [
"child", "children", "kid", "kiddo", "toddler", "little boy",
"little girl", "young boy", "young girl", "small child", "childlike",
]),
("teenager", ["teenager", "teen", "teenage", "adolescent"]),
("middle-aged", [
"middle-aged", "middle aged", "middle age", "midlife", "forties",
"fifties", "sixties", "mature",
]),
("elderly", [
"elderly", "old man", "old woman", "old lady", "older man",
"older woman", "elder", "senior", "aged", "grandpa", "grandfather",
"grandma", "grandmother", "granny", "retired", "seventies",
"eighties", "nineties", "old",
]),
("young adult", [
"young adult", "young woman", "young man", "young lady", "youthful",
"twenties", "thirties", "college", "young",
]),
]
# "very …" rules precede their plain counterparts so "very deep" doesn't stop
# at "deep". Bare "low"/"high" only count next to a voice word (pitch/voice/
# tone/register) to avoid false hits like "high quality" or "low effort".
_PITCH_RULES = [
("very low pitch", [
"very low pitch", "very low-pitched", "very low pitched",
"very low voice", "very low tone", "very deep", "extremely deep",
"extremely low", "ultra deep", "booming",
]),
("very high pitch", [
"very high pitch", "very high-pitched", "very high pitched",
"very high voice", "very high tone", "extremely high", "squeaky",
"shrill", "falsetto", "chipmunk",
]),
("low pitch", [
"low pitch", "low-pitched", "low pitched", "low voice", "low tone",
"low register", "deep", "deeper", "bass", "baritone", "husky",
]),
("high pitch", [
"high pitch", "high-pitched", "high pitched", "high voice",
"high tone", "high register", "soprano",
]),
("moderate pitch", [
"moderate pitch", "medium pitch", "medium-pitched", "medium pitched",
"mid-range", "midrange", "average pitch", "moderate",
]),
]
_STYLE_RULES = [
("whisper", [
"whisper", "whispering", "whispered", "whispery", "hushed",
"breathy", "soft-spoken", "soft spoken",
]),
]
# Bare "english" means the language, so only the explicit "english accent"
# phrase maps to british. "chinese" maps to the chinese *accent* (English
# speech with a Chinese accent); actual dialect words live in DIALECT_PINYIN.
_ACCENT_RULES = [
("american accent", [
"american", "america", "usa", "us accent", "midwestern",
"californian", "new york",
]),
("british accent", [
"british", "britain", "english accent", "england", "uk accent",
"london", "cockney", "posh", "received pronunciation",
]),
("australian accent", ["australian", "australia", "aussie"]),
("canadian accent", ["canadian", "canada"]),
("indian accent", ["indian", "india"]),
("chinese accent", ["chinese accent", "chinese-accented", "chinese"]),
("korean accent", ["korean", "korea"]),
("japanese accent", ["japanese", "japan"]),
("portuguese accent", ["portuguese", "portugal", "brazilian", "brazil"]),
("russian accent", ["russian", "russia"]),
]
_DIALECT_RULES = [
(token, [pinyin for pinyin, tok in DIALECT_PINYIN.items() if tok == token])
for token in sorted(_DIALECTS)
]
_RULES = {
"Gender": _GENDER_RULES,
"Age": _AGE_RULES,
"Pitch": _PITCH_RULES,
"Style": _STYLE_RULES,
"EnglishAccent": _ACCENT_RULES,
"ChineseDialect": _DIALECT_RULES,
}
# Import-time guard: every canonical token must be in the engine taxonomy, so
# a taxonomy rename upstream fails loudly here instead of at synthesis time.
for _cat_rules in _RULES.values():
for _token, _ in _cat_rules:
assert _token in _VALID, f"describe_voice token not in taxonomy: {_token!r}"
for _tok in DIALECT_PINYIN.values():
assert _tok in _DIALECTS, f"DIALECT_PINYIN value not a taxonomy dialect: {_tok!r}"
# ── Pattern compilation ───────────────────────────────────────────────────────
def _compile_phrase(phrase: str) -> re.Pattern:
"""Compile a synonym phrase to a regex.
Latin phrases get word boundaries (so "male" never fires inside "female",
"old" never inside "bold") and flexible separators (space or hyphen, so
"middle aged" also matches "middle-aged"). CJK phrases match as plain
substrings word boundaries are meaningless without spaces.
"""
if _ZH_RE.search(phrase):
return re.compile(re.escape(phrase))
parts = [re.escape(p) for p in re.split(r"[ -]+", phrase) if p]
return re.compile(r"\b" + r"[\s\-]+".join(parts) + r"\b")
def _compiled_rules():
out = {}
for cat, rules in _RULES.items():
compiled = []
for token, phrases in rules:
pats = list(phrases)
# Derive the Chinese form of each canonical token from the
# taxonomy (e.g. "middle-aged" → "中年") — never hardcoded here.
zh = _EN_TO_ZH.get(token)
if zh:
pats.append(zh)
if token not in pats:
pats.append(token) # the canonical token always matches itself
compiled.append((token, [_compile_phrase(p) for p in pats]))
out[cat] = compiled
return out
_COMPILED = _compiled_rules()
# "<N> year(s) old / <N>-year-old / <N> yo" → an age bracket. Runs before the
# keyword rules so the trailing "old" never misfires as elderly.
_AGE_NUM = re.compile(
r"\b(\d{1,3})(?:[\s\-]*(?:years?|yrs?|yr)[\s\-]*old|[\s\-]*(?:yo|y/o))\b"
)
def _age_token_for(years: int) -> str:
if years <= 12:
return "child"
if years <= 19:
return "teenager"
if years <= 39:
return "young adult"
if years <= 64:
return "middle-aged"
return "elderly"
def _normalize(description: str) -> str:
text = (description or "").lower()
text = text.replace("", "'").replace("", "'")
text = text.replace("", '"').replace("", '"')
return re.sub(r"[ \t]+", " ", text)
def _match_category(category: str, text: str):
"""Return (token, match) for the first rule with a hit, else None.
Rule order decides the winning token; within the winning rule the earliest
occurrence in the text is reported as the matched phrase.
"""
if category == "Age":
m = _AGE_NUM.search(text)
if m:
return _age_token_for(int(m.group(1))), m
for token, patterns in _COMPILED[category]:
best = None
for pat in patterns:
m = pat.search(text)
if m is not None and (best is None or m.start() < best.start()):
best = m
if best is not None:
return token, best
return None
# Fragment splitter for the "unmatched" report: clause separators (incl. the
# CJK comma/ideographic stop, which CJK descriptions use instead of ASCII).
_FRAGMENT = re.compile(r"[^,;.!?()\n,。;!?、]+")
_HAS_CONTENT = re.compile(r"[\w一-鿿]")
def parse_description(description: str) -> dict:
"""Map a free-text voice description onto the design parameter space.
Returns a dict with:
* ``attrs`` full category token map (``"Auto"`` where nothing
matched); same shape as the Design tab's ``vdStates``.
* ``instruct`` validator-safe instruct string built from the matched
tokens, in canonical category order (may be ``""``).
* ``matched`` list of ``{category, token, phrase}`` for transparency.
* ``unmatched`` clause fragments that contributed no attribute, so the
UI can show what was ignored instead of failing silently.
"""
text = _normalize(description)
attrs = {cat: "Auto" for cat in CATEGORY_ORDER}
matched = []
spans = []
for category in CATEGORY_ORDER:
hit = _match_category(category, text)
if hit is None:
continue
token, m = hit
attrs[category] = token
matched.append({"category": category, "token": token, "phrase": m.group(0)})
spans.append((m.start(), m.end()))
# Accents are English-only and dialects Chinese-only in the engine
# taxonomy; a dialect voice speaks Chinese, so an accent token alongside
# it is contradictory (the issue-#114 conflict class). Dialect wins.
if attrs["ChineseDialect"] != "Auto" and attrs["EnglishAccent"] != "Auto":
dropped = attrs["EnglishAccent"]
attrs["EnglishAccent"] = "Auto"
matched = [m for m in matched if not (m["category"] == "EnglishAccent" and m["token"] == dropped)]
instruct = ", ".join(attrs[c] for c in CATEGORY_ORDER if attrs[c] != "Auto")
unmatched = []
for frag in _FRAGMENT.finditer(text):
if not _HAS_CONTENT.search(frag.group(0)):
continue
lo, hi = frag.start(), frag.end()
if any(s < hi and e > lo for s, e in spans):
continue
unmatched.append(frag.group(0).strip())
return {
"attrs": attrs,
"instruct": instruct,
"matched": matched,
"unmatched": unmatched,
}
-279
View File
@@ -1,279 +0,0 @@
"""Canonical host compute-capability probe — the single source of truth for
"what can this machine actually accelerate on."
Every routing decision (the engine compatibility matrix, ``/setup/preflight``,
``/system/diagnose``, and the synth-time no-silent-fallback gating) reads from
``detect_host_caps()`` so the probe and the model loader can never disagree.
Design contract (load-bearing):
- **Never raises** to a caller. A broken torch / driver crash degrades to a
cached CPU-only ``probe_ok=False`` result; every endpoint stays responsive
(local-first: the app must work with no GPU and even with a broken torch).
- **No network call** driver/sysctl reads only, no tensor allocation, so it
stays kernel-free on cold start.
- **No new regex** on any driver/device string (CodeQL py/polynomial-redos):
the only string parse is the ``int(driver.split(".")[0])`` shape reused
from the wizard, and arch comparison is plain list membership.
- Distinguishes **ROCm from CUDA** (unlike the gguf ``hardware_probe``):
ROCm-on-HIP presents through ``torch.cuda`` but is reported ``family="rocm"``.
The ``get_best_device()`` loader (``services.model_manager``) delegates its
*family* decision here while keeping its own DirectML branch and the ROCm
``HSA_OVERRIDE_GFX_VERSION`` env side-effect the probe **reads**, the loader
**writes**. (The gguf ``hardware_probe.detect_capabilities()`` rebase onto this
module is a deliberate follow-up: it has its own torch-mocked test suite and a
VRAM-driven quant table that is unaffected by the family rename, so it is kept
out of this backend-only slice.)
"""
from __future__ import annotations
import functools
import platform as _platform
import sys
from dataclasses import dataclass
from typing import Literal
DeviceFamily = Literal["cuda", "rocm", "mps", "xpu", "cpu"]
# Stable substring stamped onto notes that represent a real kernel-launch risk
# (arch/driver mismatch) — as opposed to advisory notes (multi-GPU, VRAM query
# failed, DirectML present). ``engine_routing`` keys the "accelerated, but…"
# caveat off this marker so advisory notes never downgrade an accelerated badge.
KERNEL_RISK_MARKER = "may fail at kernel launch"
# Substring marking a DirectML-present (Windows GPU) host. The probe reports
# such hosts as ``family="cpu"`` (DirectML is not a torch device family); the
# router reads this marker to explain the neutral badge instead of "no GPU".
DIRECTML_MARKER = "DirectML device present"
# NOTE: the NVIDIA driver-version check (min R555 for the bundled CUDA runtime)
# is intentionally NOT done here — it requires shelling to ``nvidia-smi``, which
# would put a subprocess on the cold-start probe path. That check stays in
# ``wizard._detect_gpu`` (preflight), which already runs it. The probe only
# emits the torch-visible SM-arch caveat (cheap, metadata-only).
@dataclass(frozen=True)
class HostCaps:
"""Snapshot of the host's accelerator capability. Immutable + cached."""
family: DeviceFamily
"""Best available accelerator family, else ``"cpu"``."""
available_families: tuple[DeviceFamily, ...]
"""Everything usable; **always includes** ``"cpu"`` (invariant)."""
device_name: str = ""
"""Device 0's name, e.g. ``"NVIDIA RTX 4090"`` / ``"Apple Silicon (MPS)"``."""
vram_gb: float = 0.0
"""CUDA/ROCm total VRAM in GB; MPS = system RAM / 2; 0 for cpu/xpu."""
driver: str | None = None
"""Raw ROCm HIP version string (``torch.version.hip``) or ``None``. The
NVIDIA driver-version check is owned by ``wizard._detect_gpu`` (it already
shells to ``nvidia-smi``); the probe stays subprocess-free."""
notes: tuple[str, ...] = ()
"""Author-controlled English advisories (never user input). Empty on a
clean accelerated host."""
probe_ok: bool = True
"""``False`` only when torch could not be imported (degraded CPU-only)."""
def _probe() -> HostCaps:
"""Run the probe once. Enumerates every failure branch from the spec's
degradation contract; never raises."""
try:
import torch
except Exception:
return HostCaps(
family="cpu",
available_families=("cpu",),
notes=("torch not importable; treating host as CPU-only",),
probe_ok=False,
)
notes: list[str] = []
# Probe EVERY accelerator independently into this list (don't short-circuit
# after the first hit) so `available_families` is honest on hybrid hosts
# (e.g. an NVIDIA GPU + an Intel iGPU exposed via IPEX). The preferred
# `family` is chosen by priority at the end.
detected: list[DeviceFamily] = []
device_name = ""
vram_gb = 0.0
driver: str | None = None
# ── CUDA / ROCm (both present through torch.cuda) ────────────────────
cuda_ok = False
try:
cuda_ok = bool(torch.cuda.is_available())
except Exception as exc: # broken CUDA init (forked process / driver crash)
notes.append(f"CUDA init raised: {type(exc).__name__}")
if cuda_ok:
try:
count = int(torch.cuda.device_count())
except Exception:
count = 0
if count == 0:
notes.append("CUDA reports available but device_count==0")
else:
is_rocm = getattr(torch.version, "hip", None) is not None
detected.append("rocm" if is_rocm else "cuda")
if is_rocm:
driver = getattr(torch.version, "hip", None)
if count > 1:
notes.append(f"{count} GPUs detected; routing reflects device 0")
try:
device_name = torch.cuda.get_device_name(0)
except Exception:
device_name = ""
try:
_free, total = torch.cuda.mem_get_info()
vram_gb = float(total) / (1024 ** 3)
except Exception:
notes.append("VRAM query failed")
# SM-arch mismatch (mirrors model_manager.check_device_compatibility).
try:
major, minor = torch.cuda.get_device_capability(0)
arch_list = getattr(torch.cuda, "_get_arch_list", lambda: [])()
if arch_list:
sm_tag = f"sm_{major}{minor}"
compute_tag = f"compute_{major}{minor}"
if sm_tag not in arch_list and compute_tag not in arch_list:
notes.append(
f"{device_name or 'GPU'} ({sm_tag}) not in this torch "
f"build's archs ({', '.join(arch_list)}) — "
f"{KERNEL_RISK_MARKER}"
)
except Exception:
# Arch metadata unavailable on this torch build — skip the check
# (treated as compatible, exactly as check_device_compatibility).
pass
# ── Intel XPU via IPEX ───────────────────────────────────────────────
try:
import intel_extension_for_pytorch # noqa: F401
if hasattr(torch, "xpu") and torch.xpu.is_available():
detected.append("xpu")
if not device_name:
try:
device_name = torch.xpu.get_device_name(0)
except Exception:
# XPU present but unnamed — family classification still holds.
pass
notes.append("XPU VRAM not queried (unreliable across IPEX versions)")
except Exception:
# IPEX absent or XPU probe failed — no XPU on this host.
pass
# ── Apple Silicon MPS ────────────────────────────────────────────────
try:
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
detected.append("mps")
if not device_name:
device_name = "Apple Silicon (MPS)"
if not vram_gb:
try:
import psutil
vram_gb = float(psutil.virtual_memory().total) / (1024 ** 3) / 2
except Exception:
notes.append("psutil unavailable; MPS VRAM unknown")
except Exception:
# MPS probe raised on a non-Apple/old torch — treat as no MPS.
pass
# ── DirectML — Windows GPU, NOT a torch device family ────────────────
try:
import torch_directml
if torch_directml.device_count() > 0:
notes.append(
f"{DIRECTML_MARKER} (Windows GPU); torch-family probe treats "
f"as non-accelerated"
)
except Exception:
# torch_directml absent (the common case) — no DirectML on this host.
pass
# Preferred family by priority; cpu when nothing accelerated was detected.
family: DeviceFamily = "cpu"
for pref in ("cuda", "rocm", "xpu", "mps"):
if pref in detected:
family = pref # type: ignore[assignment]
break
# available_families: every detected accelerator + cpu, deduped, cpu last.
available: tuple[DeviceFamily, ...] = tuple(dict.fromkeys([*detected, "cpu"]))
return HostCaps(
family=family,
available_families=available,
device_name=device_name,
vram_gb=vram_gb,
driver=driver,
notes=tuple(notes),
probe_ok=True,
)
@functools.lru_cache(maxsize=1)
def detect_host_caps() -> HostCaps:
"""Cached per-process host capabilities. Never raises, makes no network
call, kernel-free on cold start. Host compute capability does not change at
runtime in any supported desktop flow (no GPU hot-plug; switching the active
engine does not re-probe routing is recomputed from these same caps), so
a single probe per process is correct. ``probe_ok=False`` is cached too."""
return _probe()
def refresh() -> HostCaps:
"""Clear the cache and re-probe. **TEST-ONLY** — nothing in the running app
calls this (host caps are immutable per process)."""
detect_host_caps.cache_clear()
return detect_host_caps()
def mlx_supported() -> tuple[bool, str]:
"""``(ok, reason)``. ``ok=True`` **only** on Apple Silicon
(``sys.platform == "darwin"`` and ``platform.machine() == "arm64"``) with
torch MPS available the shared gate for MLX-Audio / MLX-Whisper (#390).
Gates on exact-string equality (no regex no CodeQL surface). On any
non-Apple host it returns ``False`` **before** any package import, so a
stray ``mlx_*`` wheel on Linux/Windows never reports available.
"""
if sys.platform != "darwin" or _platform.machine() != "arm64":
if sys.platform == "darwin":
return (False, "MLX requires Apple Silicon; this Mac is Intel")
return (
False,
f"MLX requires Apple Silicon; this host is "
f"{sys.platform}/{_platform.machine()}",
)
try:
import torch
except Exception:
return (False, "torch not importable; cannot confirm MPS")
try:
if torch.backends.mps.is_available():
return (True, "")
except Exception:
# MPS query raised — fall through to the conservative unavailable path.
pass
return (
False,
"Apple Silicon detected but torch MPS unavailable; "
"reinstall torch with MPS support",
)
__all__ = [
"DeviceFamily",
"HostCaps",
"detect_host_caps",
"refresh",
"mlx_supported",
"KERNEL_RISK_MARKER",
"DIRECTML_MARKER",
]
+9 -71
View File
@@ -4,7 +4,7 @@ One pass over everything a working install needs: Python, compute device,
ffmpeg, HF token, disk, data-dir permissions, RAM, TTS engines, and (when
requested) network reachability of the HuggingFace hub. Surfaced two ways:
- ``GET /system/diagnose`` (Settings > About -> "Run self-check")
- ``GET /system/diagnose`` (Settings > About "Run self-check")
- ``python main.py --diagnose`` for headless installs / issue triage
Every ``detail``/``hint`` string is passed through ``core.scrub`` before it
@@ -88,33 +88,17 @@ def _check_device() -> dict:
def _check_ffmpeg() -> dict:
"""Media engine (ffmpeg + ffprobe) — an internal dependency the app
bundles/acquires itself, so a failure here means the self-heal also has
nothing to work with (and the hint says where the controls live)."""
ffmpeg = ffprobe = None
try:
from services.ffmpeg_utils import find_ffmpeg, find_ffprobe
ffmpeg = find_ffmpeg()
ffprobe = find_ffprobe()
from services.ffmpeg_utils import find_ffmpeg
path = find_ffmpeg()
except Exception:
pass
if ffmpeg and ffprobe:
return _check("ffmpeg", "Media engine (ffmpeg)", OK,
f"ffmpeg: {ffmpeg}; ffprobe: {ffprobe}")
if ffmpeg:
return _check(
"ffmpeg", "Media engine (ffmpeg)", WARN,
f"ffmpeg: {ffmpeg}; ffprobe missing",
"Media probing (Smart Fit, file inspection) is degraded. Open "
"Settings > Audio tools and press Restore bundled to fetch the "
"app's own ffprobe, or point it at a system copy there.",
)
path = None
if path:
return _check("ffmpeg", "ffmpeg", OK, str(path))
return _check(
"ffmpeg", "Media engine (ffmpeg)", FAIL,
"no runnable ffmpeg in any tier (sidecar, bundled, system, custom)",
"Dubbing and audio conversion are unavailable. The app normally "
"provisions ffmpeg itself — open Settings > Audio tools and press "
"Restore bundled (needs network once), or choose a system copy there.",
"ffmpeg", "ffmpeg", FAIL,
"not found on PATH or FFMPEG_PATH",
"Dubbing and audio conversion need ffmpeg: brew install ffmpeg (macOS), apt install ffmpeg (Linux), or set the path in Settings > General.",
)
@@ -210,49 +194,6 @@ def _check_engines() -> dict:
return _check("engines", "TTS engines", OK, detail)
def _check_gpu_routing() -> dict:
"""Routing verdict for the active TTS engine on THIS host (#21).
Surfaces a CPU fallback / unavailable-GPU *before* a slow or failed synth
the no-silent-fallback contract. `cpu_only` on a no-GPU machine is the
expected normal state and stays OK (never noise-warns)."""
try:
from services.tts_backend import gpu_routing_verdict
v = gpu_routing_verdict()
except Exception as e:
return _check("gpu_routing", "GPU routing", WARN, f"could not resolve: {e}")
status = v.get("routing_status")
engine = v.get("engine") or "active engine"
dev = v.get("effective_device") or "?"
reason = v.get("routing_reason")
host = v.get("host_family", "cpu")
if status == "accelerated":
if reason: # driver/arch caveat — accelerated but at risk
return _check("gpu_routing", "GPU routing", WARN,
f"{engine} -> {dev}: {reason}",
"The GPU is selected but may fail at kernel launch — "
"update drivers / reinstall torch for this GPU arch.")
return _check("gpu_routing", "GPU routing", OK, f"{engine} -> {dev} (accelerated)")
if status == "cpu_fallback":
return _check("gpu_routing", "GPU routing", WARN,
f"{engine} runs on CPU: {reason or 'no GPU path for this host'}",
"Pick an engine that supports this host's GPU for a big speedup, "
"or continue on CPU (slower).")
if status == "cpu_only":
return _check("gpu_routing", "GPU routing", OK,
f"{engine} -> cpu (no accelerator on this host)")
if status == "unavailable":
return _check("gpu_routing", "GPU routing", FAIL,
f"{engine} can't run on this host: {reason or f'needs a GPU; host is {host}'}",
"Select an engine with a CPU path in Settings -> Engines.")
# status == "none" / unknown — no active engine resolved.
return _check("gpu_routing", "GPU routing", WARN,
"No active TTS engine resolved for routing.",
"Pick an engine in Settings -> Engines.")
_DEEP_TIMEOUT_S = 180
@@ -323,8 +264,6 @@ def _check_network() -> dict:
# all model downloads need to get started. urllib honors HTTP(S)_PROXY.
import urllib.request
import urllib.error
if not _HUB_URL.startswith("https://"): # constant today; guard the sink anyway
raise ValueError(f"hub URL must be https, got {_HUB_URL!r}")
req = urllib.request.Request(_HUB_URL, method="HEAD")
try:
with urllib.request.urlopen(req, timeout=_HUB_TIMEOUT_S):
@@ -357,7 +296,6 @@ def run_diagnostics(include_network: bool = True, deep: bool = False) -> dict:
_check_data_dir(),
_check_ram(),
_check_engines(),
_check_gpu_routing(),
]
if include_network:
checks.append(_check_network())
+1 -8
View File
@@ -76,12 +76,6 @@ _CLASS_RULES: tuple[tuple[str, tuple[str, ...]], ...] = (
"connection refused",
"connection reset",
"connection aborted",
# transformers' download-failure wording ("We couldn't connect to
# '<endpoint>' to load the files") — the #874 mirror-down class was
# journaled as UNKNOWN without these.
"couldn't connect to",
"could not connect to",
"max retries exceeded",
"timed out",
"timeout",
"name or service not known",
@@ -117,8 +111,7 @@ def classify_exception(exc: BaseException, trace: str = "") -> str:
def _fingerprint(error_class: str, exc: BaseException) -> str:
import hashlib
raw = f"{error_class}|{type(exc).__name__}|{scrub_text(str(exc))[:200]}"
# Dedup key for the journal, not a security boundary.
return hashlib.sha1(raw.encode("utf-8", "replace"), usedforsecurity=False).hexdigest()[:16]
return hashlib.sha1(raw.encode("utf-8", "replace")).hexdigest()[:16]
def _persist_locked() -> None:
+2 -265
View File
@@ -21,7 +21,6 @@ import re
import sys
from pathlib import Path
from typing import Any, Optional
from urllib.parse import urlsplit
from core import error_docs_map
from core.logging_filter import REDACTED, _HF_TOKEN_RE
@@ -33,172 +32,14 @@ _REDACTED_VALUE = "***REDACTED***"
# One-line "what to do" per docs-taxonomy key. Keys mirror error_docs_map's
# taxonomy; the docs URL itself stays owned by error_docs_map.
_HINTS: dict[str, str] = {
"PKG_RESOURCES_MISSING": "Run `uv pip install --reinstall 'setuptools>=75,<80'` in the backend venv (a plain install is skipped when setuptools' metadata is present but its pkg_resources files were removed by antivirus). Restart after.",
"PKG_RESOURCES_MISSING": "Install setuptools in the backend environment (provides pkg_resources).",
"GATEKEEPER_QUARANTINE": "Clear the macOS quarantine flag (xattr -cr the app), then reopen.",
"APPIMAGE_WEBKIT_WHITESCREEN": "Launch with WEBKIT_DISABLE_DMABUF_RENDERER=1 set.",
"HF_AUTH_FAILED": "Set a valid HF_TOKEN in Settings → Hugging Face and retry.",
"PYANNOTE_LICENSE_REQUIRED": "Accept the pyannote model licenses on Hugging Face, then retry.",
"COMPUTE_TYPE_UNSUPPORTED": "Your GPU doesn't support float16 — OmniVoice retried on int8. If transcription still fails, set OMNIVOICE/ASR_COMPUTE_TYPE=int8 or use CPU.",
"TRANSFORMERS_IMPORT": "Your transformers install is incomplete. Reinstall it (`uv pip install --reinstall transformers`) or switch ASR to faster-whisper (Settings → Models).",
"OS_INVALID_ARGUMENT": "The OS rejected a file operation (Errno 22 / invalid argument) — in the transcribe path this is the temporary WAV write before ASR. It's almost always the temp directory: missing, read-only, on a full or removed drive, or blocked by antivirus. Check that your system TEMP/TMP folder exists and is writable and the drive has free space (add an OmniVoice antivirus exclusion if you use one), then retry.",
"SOCKS_PROXY_SUPPORT_MISSING": "A SOCKS proxy is configured in your environment (ALL_PROXY/HTTPS_PROXY=socks5://…) and the backend's HTTP client is missing SOCKS support. Newer OmniVoice builds ship SOCKS support (the socksio package) — update the app. If you still see this, unset ALL_PROXY/HTTPS_PROXY for OmniVoice, or run `uv pip install 'httpx[socks]'` in the backend venv, then restart.",
"SSL_HANDSHAKE_FAILURE": "A corporate or antivirus proxy is intercepting HTTPS traffic and re-signing certificates with its own CA — your OS trusts that CA, but Python's bundled certifi CA list doesn't, so the TLS handshake fails even though the connection reached the server. Newer OmniVoice builds trust the OS certificate store at startup (the truststore package), which should already fix this — update the app and retry. If you still see this, add an HTTPS-scanning exclusion for OmniVoice/Python in your antivirus, or ask IT for the proxy's CA bundle and set SSL_CERT_FILE to it, then restart.",
"UNSUPPORTED_VIDEO_URL": "This link isn't a directly downloadable video. Paste a direct video page (e.g. a youtube.com/watch?v=… or douyin.com/video/<id> link), not a share/profile/feed link — or download the file and drop it in directly.",
"VIDEO_DOWNLOAD_NETWORK": "The connection to the video server dropped mid-download (often a transient CDN/network blip or a regional rate-limit). Just retry — OmniVoice already cleaned up the partial download. If it keeps failing, check your network/VPN.",
"BROKEN_VENV": "The Python backend environment was moved or damaged. OmniVoice rebuilds it automatically on the next launch; if it keeps failing, use Clean & Retry on the setup screen.",
"MODEL_CACHE_CORRUPT": "The model cache had broken file links — snapshot entries that no longer point at their downloaded data (interrupted renames or antivirus interference can cause this). OmniVoice repairs this automatically and retries the load once. If the error persists, quit OmniVoice, delete the model's models--<org>--<name> folder inside the Hugging Face cache, and restart — the model re-downloads automatically.",
# HF_MIRROR_UNREACHABLE has a DYNAMIC hint (it names the configured mirror)
# — see hf_mirror_hint(); build_failure special-cases it.
}
# ── HF mirror connectivity (#874) ────────────────────────────────────────────
# When a non-default HF_ENDPOINT (a mirror, e.g. hf-mirror.com — set via
# Settings → Models → Hugging Face mirror) is configured and a model
# download/load fails with a connectivity error, the raw transformers/hf_hub
# message ("We couldn't connect to 'https://hf-mirror.com' to load the files…")
# gives the user no next step. This is the single classifier for that class,
# shared by every surface: build_failure() (model status, dub/task events),
# the global 500 handler (main.py — covers /generate and every other route
# that can leak a model-load error), and the model-install SSE
# (setup/download.py).
_OFFICIAL_HF_ENDPOINTS = {"https://huggingface.co", "https://hf.co"}
# Connectivity signatures across the layers an HF download failure surfaces
# from: transformers' wording, huggingface_hub errors, requests/urllib3, and
# raw socket/DNS failures (Linux/macOS/Windows variants).
_HF_CONNECTIVITY_SIGNATURES = (
"couldn't connect to", # transformers: "We couldn't connect to '<endpoint>' …"
"could not connect to",
"connection error", # huggingface_hub / requests
"connection refused",
"connection reset",
"connection aborted",
"max retries exceeded", # urllib3 via requests
"failed to establish a new connection",
"name or service not known", # Linux DNS
"temporary failure in name resolution",
"nodename nor servname provided", # macOS DNS
"getaddrinfo failed", # Windows DNS
"timed out",
"an error happened while trying to locate the file on the hub", # LocalEntryNotFoundError
"we cannot find the requested files", # LocalEntryNotFoundError
)
# The failure must also be Hugging-Face-shaped — the configured endpoint/host
# named in the message, or HF-download wording — so a random socket error
# (e.g. a local LLM provider being down) doesn't get the mirror hint just
# because a mirror happens to be configured.
_HF_CONTEXT_MARKERS = (
"huggingface",
"hf_hub",
"hf-hub",
"load the files", # transformers
"cached files", # transformers
"the requested files", # LocalEntryNotFoundError
"locate the file on the hub",
"snapshot_download",
)
def configured_hf_mirror() -> str:
"""The non-default Hugging Face endpoint (mirror) in effect, or "".
Same resolution the download paths use: ``HF_ENDPOINT`` env (what
Settings Models Hugging Face mirror persists via user_env, and what
the HF libraries read) with the ``hf_endpoint`` pref as fallback
(mirrors setup/download.py's ``prefs.resolve``). Never raises.
"""
ep = (os.environ.get("HF_ENDPOINT") or "").strip()
if not ep:
try:
from core import prefs
ep = str(prefs.get("hf_endpoint", "") or "").strip()
except Exception:
ep = ""
ep = ep.rstrip("/")
if not ep or ep.lower() in _OFFICIAL_HF_ENDPOINTS:
return ""
return ep
def hf_mirror_hint(reason: Optional[str]) -> str:
"""Actionable hint when ``reason`` is an HF-download connectivity failure
and a non-default mirror endpoint is configured; "" otherwise.
The hint names the configured mirror, says it may be down, points at the
setting (Settings Models Hugging Face mirror), suggests the official
endpoint when the model isn't cached yet, and notes the restart
requirement (HF reads HF_ENDPOINT at import time see the hf-mirror
endpoints in api/routers/settings.py). Never raises.
"""
mirror = configured_hf_mirror()
if not mirror:
return ""
low = (reason or "").lower()
if not any(sig in low for sig in _HF_CONNECTIVITY_SIGNATURES):
return ""
try:
host = (urlsplit(mirror).netloc or "").lower()
except Exception:
host = ""
if not (
mirror.lower() in low
or (host and host in low)
or any(m in low for m in _HF_CONTEXT_MARKERS)
):
return ""
return (
f"Your Hugging Face mirror is set to {mirror}, which couldn't be "
"reached — the mirror may be down or blocked on your network. If the "
'model isn\'t in your local cache yet, switch to "Hugging Face '
'(official)" in Settings → Models → Hugging Face mirror (or wait for '
"the mirror to recover), then restart OmniVoice — the mirror setting "
"is applied when the app starts."
)
def append_hf_mirror_hint(text: str) -> str:
"""``"{text}{hint}"`` when the mirror-connectivity class applies;
``text`` unchanged otherwise. For surfaces that hand a raw error string to
the UI (the global 500 handler, the model-install SSE). Never raises."""
try:
hint = hf_mirror_hint(text)
except Exception:
return text
return f"{text}{hint}" if hint else text
# Classes whose hint is safe to attach on the CONTEXT-FREE surfaces (the
# global 500 handler in main.py, the model-install SSE in setup/download.py),
# where all we have is a raw error string with no stage. Only classes whose
# classify() trigger is unmistakable belong here — e.g. VIDEO_DOWNLOAD_NETWORK
# must NOT be added: its bare "timed out" trigger would stamp a "video server"
# hint on a model-load timeout that leaks through the 500 handler.
_CONTEXT_FREE_HINT_CLASSES = frozenset({
"SOCKS_PROXY_SUPPORT_MISSING",
"SSL_HANDSHAKE_FAILURE",
})
def append_hint(text: str) -> str:
"""``"{text}{hint}"`` for raw-string surfaces (the global 500 handler,
the model-install SSE): the dynamic mirror hint (#874) when that class
applies, else a context-free static class hint (#959). ``text`` unchanged
otherwise a no-op for every other error. Never raises."""
try:
hint = hf_mirror_hint(text)
if not hint:
topic = classify(text)
if topic in _CONTEXT_FREE_HINT_CLASSES:
hint = _HINTS.get(topic, "")
except Exception:
return text
return f"{text}{hint}" if hint else text
def classify(reason: str) -> str:
"""Map a failure reason to a docs-taxonomy key, or "" when unknown.
@@ -214,111 +55,10 @@ def classify(reason: str) -> str:
return "APPIMAGE_WEBKIT_WHITESCREEN"
if "pyannote" in low or ("gated" in low and "model" in low) or "accept the" in low:
return "PYANNOTE_LICENSE_REQUIRED"
# ASR robustness (#551 / #549): name the class so the no-segments toast is
# actionable. Place before the generic returns so a compute-type/transformers
# failure gets its hint rather than falling through to "".
if "compute type" in low or "efficient float16" in low:
return "COMPUTE_TYPE_UNSUPPORTED"
# #763: a bare OS-level EINVAL ("[Errno 22] Invalid argument") while writing
# the per-chunk temp WAV for transcription (tempfile.NamedTemporaryFile /
# soundfile.write on the system temp dir) used to collapse into a dead-end
# "produced no segments. [Errno 22] Invalid argument" toast with no next
# step. errno 22 is EINVAL on every platform; in this path it's almost always
# a temp dir that's missing, read-only, on a full/removed drive, or blocked
# by antivirus. Name the class so build_failure attaches an actionable hint
# instead of a raw errno. Matching the errno (not the generic "invalid
# argument" wording) keeps this from mislabelling unrelated failures; the
# transformers "errno 2" rule below is unaffected — it also requires the
# transformers + site-packages markers, which this signature lacks.
if "errno 22" in low:
return "OS_INVALID_ARGUMENT"
# An HF cache whose snapshot entries don't resolve (dangling symlinks /
# zero-byte stand-ins): transformers reports the weights missing ("does
# not appear to have a file named pytorch_model.bin or model.safetensors")
# even though the blobs are fully on disk. model_manager self-heals this
# (delete broken entries → snapshot_download → retry once); the class here
# covers both the raw transformers wording (any load surface can leak it)
# and OmniVoice's own repair messages, so the user-facing error and the
# auto bug report name the class and its automatic repair.
if ("does not appear to have a file named" in low
or "broken file link" in low):
return "MODEL_CACHE_CORRUPT"
if (
"could not import module" in low
or "autofeatureextractor" in low
# A corrupted/incomplete transformers install: a model load lazily
# resolves a module file that's MISSING from site-packages (an
# interrupted `uv sync`, antivirus removal, or a partial update), e.g.
# `[Errno 2] No such file or directory:
# '.../site-packages/transformers/models/qwen3/modeling_qwen3.py'`.
# That's a FileNotFoundError, not an ImportError, so the matches above
# miss it and the user got a useless "try restarting". Substring-match
# the package + the missing-file signal (separately, so it works on both
# POSIX `/` and Windows `\` paths).
or (
("no such file" in low or "errno 2" in low)
and "transformers" in low
and "site-packages" in low
)
):
return "TRANSFORMERS_IMPORT"
# #959: httpx raises ImportError AT CLIENT CONSTRUCTION ("Using SOCKS
# proxy, but the 'socksio' package is not installed. Make sure to install
# httpx using `pip install httpx[socks]`.") when ALL_PROXY/HTTPS_PROXY is
# socks5:// and socksio isn't importable. It surfaced from
# huggingface_hub's get_session() inside model load — a bare 500 on
# /generate with no next step. Checked BEFORE the HF-auth/mirror rules so
# a message that also carries HF wording still names this class.
if "socks proxy" in low or "socksio" in low:
return "SOCKS_PROXY_SUPPORT_MISSING"
# #976: a TLS handshake failing AFTER the TCP connection succeeds — the
# signature of a corporate/antivirus proxy that TLS-inspects traffic and
# re-signs certificates with a CA the OS trusts but Python's bundled
# certifi list doesn't (a different failure mode from #984's TCP-level
# "can't reach the host at all"). Requires "ssl" plus a handshake/cert-
# verify marker so a generic connection error isn't mislabelled.
if "ssl" in low and (
"handshake" in low
or "certificate verify failed" in low
or "sslv3_alert" in low
or "sslcertverificationerror" in low
):
return "SSL_HANDSHAKE_FAILURE"
if ("huggingface" in low or "hf_token" in low or "401" in low or "unauthorized" in low) and (
"token" in low or "auth" in low or "401" in low or "unauthorized" in low
):
return "HF_AUTH_FAILED"
# #874: a model download that failed because the CONFIGURED HF mirror is
# unreachable. Env-aware by design — the class only exists when a
# non-default HF_ENDPOINT is configured. Checked BEFORE the video-download
# network class so a model download's "timed out"/"connection reset"
# names the mirror instead of the "video server".
if hf_mirror_hint(reason):
return "HF_MIRROR_UNREACHABLE"
# Video download (#554/#536): a non-downloadable URL shape vs a transient
# network drop — both previously surfaced as a bare yt-dlp string with no
# next step. UNSUPPORTED first (more specific) so "Unable to download video:
# Broken pipe" still classifies as a network blip.
if "unsupported url" in low or "no video formats" in low or "is not a valid url" in low:
return "UNSUPPORTED_VIDEO_URL"
if (
"broken pipe" in low
or "connection reset" in low
or "unable to download video" in low
or "remote end closed" in low
or "timed out" in low
):
return "VIDEO_DOWNLOAD_NETWORK"
# A relocated/corrupted venv whose interpreter can't bootstrap its stdlib —
# the Rust self-heal rebuilds it; this names the class for the toast.
if "no module named 'encodings'" in low:
return "BROKEN_VENV"
# #564: the interpreter starts fine but the backend can't import its OWN
# `omnivoice` package (a venv missing the editable install). Same self-heal
# class — Clean & Retry / the bootstrap repair rebuilds it. The trailing
# quote keeps a legitimately-named `omnivoice_*` helper from matching.
if "no module named 'omnivoice'" in low:
return "BROKEN_VENV"
return ""
@@ -414,15 +154,12 @@ def build_failure(
reason = sanitize(raw) or error_class
docs_topic = classify(raw)
# HF_MIRROR_UNREACHABLE's hint is dynamic (it names the configured mirror)
# so it can't live in the static _HINTS table.
hint = hf_mirror_hint(raw) if docs_topic == "HF_MIRROR_UNREACHABLE" else _HINTS.get(docs_topic, "")
fields: dict[str, Any] = {
"reason": reason,
"error": reason, # backward-compat mirror for older frontends
"error_class": error_class,
"stage": stage,
"hint": hint,
"hint": _HINTS.get(docs_topic, ""),
"docs_topic": docs_topic,
"docs_url": error_docs_map.ERROR_DOCS.get(docs_topic, ""),
"detail": sanitize(raw),
-77
View File
@@ -1,77 +0,0 @@
"""Resolve the project's own ``omnivoice`` package from source when the venv's
editable install is missing (#564).
``omnivoice`` is normally an editable install in the backend venv. An interrupted
or offline ``uv sync`` can install dependencies yet never lay the editable record
(``_editable_impl_omnivoice.pth``), or an antivirus quarantine can remove it
leaving a venv that starts uvicorn but cannot ``import omnivoice``, so it boots
fine and only fails at the first model call (``No module named 'omnivoice'``).
The desktop layout always copies ``omnivoice/`` next to ``backend/``, so we fall
back to importing it from there. The bootstrap now also gates on omnivoice being
importable (re-syncing to re-lay the editable install), but this keeps the
backend resilient even when that repair hasn't run yet.
"""
import os
import sys
def find_omnivoice_source_root(candidates):
"""Return the first candidate dir holding ``omnivoice/__init__.py``, else None."""
for root in candidates:
if root and os.path.isfile(os.path.join(root, "omnivoice", "__init__.py")):
return root
return None
def _candidate_roots(backend_dir):
"""Source roots to probe, most-specific first.
``OMNIVOICE_PROJECT_ROOT`` lets the launcher point at the staged project dir
explicitly; otherwise the desktop layout puts ``omnivoice/`` beside
``backend/`` (parent of ``backend_dir``).
"""
roots = []
env = os.environ.get("OMNIVOICE_PROJECT_ROOT")
if env:
roots.append(env)
roots.append(os.path.dirname(os.path.abspath(backend_dir)))
return roots
def _already_importable():
import importlib.util
try:
return importlib.util.find_spec("omnivoice") is not None
except (ImportError, ValueError):
# A half-laid spec (e.g. a stale .pth pointing at a deleted dir) raises
# rather than returning None — treat it as "not importable" so we fall
# back to the on-disk source.
return False
def ensure_omnivoice_importable(backend_dir, logger=None):
"""Make ``import omnivoice`` work, falling back to the sibling source tree.
No-op when the editable/site-packages install already resolves it. Otherwise
appends the first source root containing ``omnivoice/`` to ``sys.path``
(appended, never inserted, so a real install keeps precedence). Returns the
root that was added, or ``None`` if none was needed or found.
"""
if _already_importable():
return None
root = find_omnivoice_source_root(_candidate_roots(backend_dir))
if root and root not in sys.path:
sys.path.append(root)
if logger:
logger.warning(
"omnivoice not importable from the venv (missing/broken editable "
"install) — resolving it from source at %s (#564)", root,
)
elif logger and root is None:
logger.error(
"omnivoice is not importable and no source tree was found next to "
"%s — the install is incomplete; relaunch to let the bootstrap "
"repair the venv (#564)", backend_dir,
)
return root
+2 -8
View File
@@ -61,15 +61,9 @@ def seed_sample_project():
if count > 0:
return # Not first run — skip
# The demo clip is committed at backend/assets/samples/demo_voice.wav and
# bundled with the app (#621). If it's somehow absent (e.g. a partial
# checkout), skip the seed gracefully rather than seeding a profile that
# points at a missing file — run scripts/build_demos.sh to regenerate it.
# Check if demo audio exists
if not os.path.isfile(_DEMO_AUDIO):
logger.warning(
"Demo audio not found at %s — skipping onboarding seed "
"(regenerate with scripts/build_demos.sh)", _DEMO_AUDIO,
)
logger.warning("Demo audio not found at %s — skipping onboarding seed", _DEMO_AUDIO)
return
# Copy demo audio to voices directory
+6 -57
View File
@@ -36,39 +36,15 @@ _TOKEN_PATTERNS = (
re.compile(r"github_pat_[A-Za-z0-9_]{20,}"), # GitHub fine-grained PAT
re.compile(r"gh[pousr]_[A-Za-z0-9]{30,}"), # GitHub classic tokens
re.compile(r"sk-[A-Za-z0-9_\-]{20,}"), # OpenAI-style API keys
# A backend error can carry a secret from *any* provider (the LLM-providers
# feature ships a dozen), so match the common credential shapes too, not
# just the four vendors above — a leaked key in a public issue is real harm.
re.compile(r"eyJ[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{8,}\.[A-Za-z0-9_\-]{6,}"), # JWT (Bearer)
re.compile(r"AIza[0-9A-Za-z_\-]{35}"), # Google API key
re.compile(r"xox[baprs]-[A-Za-z0-9\-]{10,}"), # Slack token
re.compile(r"AKIA[0-9A-Z]{16}"), # AWS access key id
re.compile(r"(?i)bearer\s+[A-Za-z0-9._\-]{16,}"), # opaque bearer tokens
)
# Secrets carried in a URL query string (`?token=…`, `&api_key=…`). Redact the
# VALUE while keeping the param name + separator so the URL stays legible. Bare
# `key=` is intentionally excluded — too common in non-secret text; shaped keys
# are already caught above and named env vars by the sweep below.
_URL_SECRET_RE = re.compile(
r"((?:access[_-]?token|api[_-]?key|apikey|auth[_-]?token|token|secret|password|passwd|pwd)=)"
r"([^&\s\"'#]{6,})",
re.IGNORECASE,
)
# Home-directory shapes for all three supported platforms. Matched
# pattern-wise (not just this machine's $HOME) so paths quoted from a
# user's pasted log on another OS get cleaned too.
# IGNORECASE because Windows is case-insensitive and tools routinely emit the
# lowercase `c:\users\<name>` form, which the CLAUDE.md redaction spec still
# requires to become `~`. `Users`/`users`, `Home`/`home` all match.
_HOME_PATTERNS = (
# Windows-with-forward-slashes must run BEFORE the bare macOS shape, or
# `/Users/<name>` inside `C:/Users/<name>` gets eaten first, leaving `C:~`.
re.compile(r"[A-Za-z]:/Users/[^/\s\"']+", re.IGNORECASE), # Windows, forward slashes
re.compile(r"/Users/[^/\s\"']+", re.IGNORECASE), # macOS
re.compile(r"/home/[^/\s\"']+", re.IGNORECASE), # Linux
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+", re.IGNORECASE), # Windows, backslashes
re.compile(r"/Users/[^/\s\"']+"), # macOS
re.compile(r"/home/[^/\s\"']+"), # Linux
re.compile(r"[A-Za-z]:\\Users\\[^\\\s\"']+"), # Windows
)
# Values shorter than this are too entropy-poor to be real secrets and too
@@ -106,25 +82,19 @@ def scrub_text(text: str | None) -> str:
except Exception:
pass
# 2. Credential-shaped substrings + URL query secrets.
# 2. Credential-shaped substrings.
for pat in _TOKEN_PATTERNS:
try:
s = pat.sub(REDACTED, s)
except Exception:
pass
try:
s = _URL_SECRET_RE.sub(lambda m: m.group(1) + REDACTED, s)
except Exception:
pass
# 3. This process's real home dir (covers symlinked/nonstandard homes
# the generic patterns miss), then the per-OS shapes. Boundary-aware so
# a home of `/Users/john` doesn't rewrite `/Users/johnny` to `~ny`
# (leaking the fragment + mangling the path).
# the generic patterns miss), then the per-OS shapes.
try:
home = os.path.expanduser("~")
if home and home not in ("/", "~"):
s = re.sub(re.escape(home) + r"(?=[/\\\s\"']|$)", "~", s)
s = s.replace(home, "~")
except Exception:
pass
for pat in _HOME_PATTERNS:
@@ -134,24 +104,3 @@ def scrub_text(text: str | None) -> str:
pass
return s
def scrub_provider_error(detail: object, api_key: str | None = None) -> str:
"""UI-safe text for an LLM/translation provider failure.
Some OpenAI-compatible providers echo the caller's key or a stable
``user_id`` back inside their error bodies, and a raw ``str(exc)`` on the
translate / glossary paths would surface that verbatim. This redacts the
exact resolved ``api_key`` first (in the provider-registry case it isn't a
shaped/known-env secret, so ``scrub_text`` alone can miss it) then runs the
generic secret + home-path scrub. Never raises scrubbing must not mask a
failure with a new one. Mirrors ``settings._scrub_llm_detail`` so every
surface redacts identically.
"""
s = str(detail if detail is not None else "")
try:
if api_key and api_key != "local" and len(api_key) >= _MIN_SECRET_LEN:
s = s.replace(api_key, REDACTED)
except Exception:
pass
return scrub_text(s)
-25
View File
@@ -82,28 +82,3 @@ def unset_user_env(key: str, path: Optional[str] = None) -> None:
prefix = f"{key}="
lines = [ln for ln in _read_lines(path) if not ln.startswith(prefix)]
_write_lines(path, lines)
def load_into_environ(path: Optional[str] = None) -> bool:
"""Load the durable per-user env file into ``os.environ``, **overriding**
any value a launcher already injected. Returns True if a file was loaded.
This file is the in-app Settings source of truth. The desktop launcher
(Tauri) injects defaults like ``OMNIVOICE_CACHE_DIR`` (and ``HF_ENDPOINT``)
from its *own* config into the backend's environment *before* startup, so
loading this file with ``override=False`` meant a models directory the user
changed in Settings was silently ignored on every launch the effective
location stayed on the old one no matter how many restarts (#480). Both keys
this file can hold are the user's explicit Settings choice and should beat
the launcher's default, so we override. Restores this file's documented
"values written here take effect on the next backend launch" contract.
"""
path = path or os.environ.get("OMNIVOICE_ENV_FILE") or USER_ENV_PATH
if not os.path.isfile(path):
return False
try:
import dotenv
except ImportError:
return False
dotenv.load_dotenv(path, override=True)
return True
+4 -33
View File
@@ -2,43 +2,14 @@
Read from the installed package metadata (driven by ``pyproject.toml``) so the
FastAPI/API version and exported-bundle metadata never drift to a stale literal
the prior "0.4.0" / "0.2.7" bug, and the v0.3.6 desktop build that reported
"0.3.5" because the *frozen* backend couldn't read its own metadata.
Resolution order:
1. installed package metadata correct in any ``uv sync``'d env and, thanks
to ``copy_metadata('omnivoice')`` in ``backend.spec``, in the frozen build;
2. ``pyproject.toml`` walked up from this file correct for a raw source
checkout that was never installed;
3. ``_FALLBACK_VERSION`` a last resort, kept in lockstep with the four
version files by ``tests/test_app_version.py`` so it can never silently
drift again.
again (the prior "0.4.0" / "0.2.7" bug). Falls back to a literal only when
running from a raw source checkout that was never ``uv sync``'d.
"""
from __future__ import annotations
import re
from importlib.metadata import PackageNotFoundError, version
from pathlib import Path
# Last-resort literal. Guarded by
# tests/test_app_version.py::test_all_version_files_in_lockstep and bumped by
# release.yml's version-bump job, so it stays equal to
# pyproject/tauri.conf/Cargo/package.json.
_FALLBACK_VERSION = "0.3.17"
def _fallback_version() -> str:
"""Version for contexts where package metadata is unavailable."""
for parent in Path(__file__).resolve().parents:
pyproject = parent / "pyproject.toml"
if pyproject.is_file():
match = re.search(r'(?m)^version\s*=\s*"([^"]+)"', pyproject.read_text())
if match:
return match.group(1)
return _FALLBACK_VERSION
try:
APP_VERSION = version("omnivoice")
except PackageNotFoundError: # frozen build w/o metadata, or non-installed checkout
APP_VERSION = _fallback_version()
except PackageNotFoundError: # non-installed source checkout
APP_VERSION = "0.3.5"
-148
View File
@@ -1,148 +0,0 @@
"""Crash-isolated faster-whisper ASR sidecar (Wave 4.2 / Spec 7).
Runs faster-whisper in a child process so a CTranslate2 GPU-teardown segfault
becomes a failed job, not a dead backend. Speaks the SubprocessBackend wire
protocol (length-prefixed JSON over stdin/stdout):
on start {"op":"ready","engine":"faster-whisper-isolated"}
{"op":"ping"} {"op":"pong"}
{"op":"transcribe","audio_path":...,"word_timestamps":bool}
{"op":"segments","result":{"segments":[...],"language":...}}
{"op":"shutdown"} exit 0
error {"op":"error","message":...}
Runs under the PARENT venv (faster-whisper is already a dependency) only the
process boundary is new. torch/CTranslate2 import lazily inside transcribe so
the ready handshake fits the spawn timeout.
"""
from __future__ import annotations
import json
import os
import struct
import sys
import traceback
MAX_FRAME_BYTES = 64 * 1024 * 1024
_model = None
def _send(stream, obj):
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
stream.write(struct.pack("!I", len(body)))
stream.write(body)
stream.flush()
def _recv(stream):
header = stream.read(4)
if len(header) < 4:
return None
(n,) = struct.unpack("!I", header)
if n > MAX_FRAME_BYTES:
raise IOError(f"frame too large: {n}")
body = bytearray()
while len(body) < n:
chunk = stream.read(n - len(body))
if not chunk:
raise IOError("short read")
body.extend(chunk)
return json.loads(bytes(body).decode("utf-8"))
# NOTE: keep this compute_type fallback in lockstep with
# services/asr_backend.py:_compute_type_candidates / _is_compute_type_error.
# This sidecar runs in a child proc with a clean import path, so we duplicate a
# tiny copy rather than cross-importing the heavy services package (#551).
def _ct_candidates(device):
override = os.environ.get("ASR_COMPUTE_TYPE")
if override:
return [override]
return ["float16", "int8_float16", "int8"] if device == "cuda" else ["int8", "float32"]
def _is_ct_error(msg):
low = msg.lower()
return "compute type" in low or "efficient float16" in low
def _get_model():
global _model
if _model is None:
from faster_whisper import WhisperModel
name = os.environ.get("ASR_MODEL_FW", "large-v3")
try:
import torch
device = "cuda" if torch.cuda.is_available() else "cpu"
except Exception:
device = "cpu"
# Degrade fp16 → int8 rather than crash on GPUs without efficient fp16
# (older Maxwell/Pascal, GTX 16xx, CTranslate2/cuDNN mismatch) (#551).
last_err = None
for compute in _ct_candidates(device):
try:
_model = WhisperModel(name, device=device, compute_type=compute)
break
except (ValueError, RuntimeError) as e:
last_err = e
if _is_ct_error(str(e)):
continue
raise
else:
raise last_err
return _model
def _transcribe(audio_path, word_timestamps):
model = _get_model()
segments, info = model.transcribe(audio_path, word_timestamps=word_timestamps)
out = []
for s in segments:
seg = {"start": float(s.start), "end": float(s.end), "text": s.text}
if word_timestamps and getattr(s, "words", None):
seg["words"] = [
{"word": w.word, "start": float(w.start), "end": float(w.end),
"probability": float(getattr(w, "probability", 0.0))}
for w in s.words
]
out.append(seg)
return {
"segments": out,
"text": " ".join(s["text"].strip() for s in out).strip(),
"language": getattr(info, "language", "unknown"),
}
def main() -> int:
stdin, stdout = sys.stdin.buffer, sys.stdout.buffer
_send(stdout, {"op": "ready", "engine": "faster-whisper-isolated"})
while True:
try:
msg = _recv(stdin)
except Exception as exc:
_send(stdout, {"op": "error", "stage": "recv", "message": f"{type(exc).__name__}: {exc}"})
return 1
if msg is None:
return 0
op = msg.get("op")
try:
if op == "ping":
_send(stdout, {"op": "pong"})
elif op == "transcribe":
result = _transcribe(msg.get("audio_path"), bool(msg.get("word_timestamps", True)))
_send(stdout, {"op": "segments", "result": result})
elif op == "shutdown":
return 0
else:
_send(stdout, {"op": "error", "stage": "dispatch", "message": f"unknown op: {op!r}"})
except Exception as exc:
_send(stdout, {
"op": "error", "stage": "handler",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
return 0
if __name__ == "__main__":
sys.exit(main())
-19
View File
@@ -101,12 +101,6 @@ def main() -> int:
return 0
op = msg.get("op")
# Wave 4.2: deterministic "crash mid-transcription" hook — exit BEFORE
# sending any reply so the parent's blocking recv sees a dead pipe
# (reply=None). The crash-after-one hook below replies first, so it
# can't deterministically exercise the no-reply path.
if op == "transcribe" and os.environ.get("OMNIVOICE_ECHO_CRASH_NO_REPLY") == "1":
os._exit(1)
try:
if op == "ping":
_send(stdout, {"op": "pong"})
@@ -119,19 +113,6 @@ def main() -> int:
"sample_rate": sr,
"n_samples": n_samples,
})
elif op == "transcribe":
# Wave 4.2: echo ASR op — a canned segments result so the
# SubprocessASRBackend round-trip + respawn path is testable
# without a real ASR engine.
_send(stdout, {
"op": "segments",
"result": {
"segments": [{"start": 0.0, "end": 1.0,
"text": f"echo:{msg.get('audio_path', '')}"}],
"text": f"echo:{msg.get('audio_path', '')}",
"language": "en",
},
})
elif op == "shutdown":
return 0
elif op == "probe_env" and test_mode:
-134
View File
@@ -1,134 +0,0 @@
"""Confucius4-TTS sidecar package (issue #590).
Confucius4-TTS (netease-youdao) is an LLM-based multilingual / cross-lingual
zero-shot voice-cloning TTS: 14 languages, **no reference transcript required**,
cross-lingual voice transfer, Apache-2.0 (https://github.com/netease-youdao/Confucius4-TTS).
Like IndexTTS / MOSS-TTS-v1.5 / dots.tts it runs in its **own subprocess venv**
(upstream: Python 3.10 + CUDA 12.6 + its own deps), isolated from the OmniVoice
parent. It is **opt-in** selected in the engine picker and enabled only when
the user points ``OMNIVOICE_CONFUCIUS4_TTS_DIR`` at a clone so it can never
become a broken default on any platform (the strict default-parity rule).
Status (#590): **validated end-to-end** (2026-07-02, Apple Silicon, CPU) — the
synthesis API (``confuciustts.cli.inference.ConfuciusTTS``
``.generate(text, lang, prompt_wav)`` tensor, ``model.sample_rate``) produced
audible speech at 22 050 Hz; the sidecar's pure logic is unit-tested
(``tests/test_confucius4_sidecar.py``). CPU inference is slow (~17× realtime),
so CUDA is the recommended path. Gated off by default, so this affects no one
until they opt in.
Three entry points: ``Confucius4Backend`` (this module), ``main.py`` (the sidecar,
runs under the Confucius4 venv never imported by the parent), and
``bootstrap.py`` (venv probe + lazy bootstrap).
"""
from __future__ import annotations
import logging
from typing import TYPE_CHECKING
from services.subprocess_backend import SubprocessBackend
if TYPE_CHECKING:
import torch # noqa: F401
logger = logging.getLogger("omnivoice.confucius4")
class Confucius4Backend(SubprocessBackend):
"""Confucius4-TTS (netease-youdao) — LLM-based, 14 langs, zero-shot clone.
Runs in a long-lived sidecar over length-prefixed JSON-over-stdio in a
dedicated venv. First synthesize cold-loads the checkpoint; subsequent calls
reuse the process.
Installation::
git clone https://github.com/netease-youdao/Confucius4-TTS.git
cd Confucius4-TTS
uv venv --python 3.10 && uv pip install -r requirements.txt
(Upstream ships no pyproject.toml/setup.py, so there is nothing to
``pip install -e`` the sidecar sys.path-inserts the clone instead.)
Then set ``OMNIVOICE_CONFUCIUS4_TTS_DIR`` to the clone root and restart.
License: Apache-2.0. CUDA recommended; CPU validated but ~17× realtime.
"""
id = "confucius4-tts"
display_name = (
"Confucius4-TTS (LLM, 14 langs, cross-lingual zero-shot clone, CUDA/CPU, Apache-2.0)"
)
supports_voice_design = False # timbre comes from a reference clip
# Upstream vocoder rate (config target_sample_rate) — confirmed 22 050 Hz by
# a live run (2026-07-02); still re-read from the sidecar's ready/audio frames.
_DEFAULT_SAMPLE_RATE = 22050
# CUDA fast path + CPU fallback, both exercised (CPU end-to-end validated).
# No MPS claim — upstream has no Metal path.
gpu_compat = ("cuda", "cpu")
@classmethod
def is_available(cls) -> tuple[bool, str]:
# Verify the venv on disk only — do NOT import the engine here (separate
# interpreter). A real health-check runs on the user's "Test engine"
# action in Settings.
from engines.confucius4.bootstrap import (
CONFUCIUS4_SIDECAR_SCRIPT,
is_confucius4_installed,
)
if not is_confucius4_installed():
return False, (
"Confucius4-TTS venv not found. Set OMNIVOICE_CONFUCIUS4_TTS_DIR "
"to your Confucius4-TTS clone (the directory containing "
"requirements.txt) and restart OmniVoice. CUDA GPU recommended "
"(CPU works but is slow). See docs/engines/confucius4-tts.md."
)
if not CONFUCIUS4_SIDECAR_SCRIPT.exists():
return False, (
"Confucius4-TTS sidecar script missing at "
f"{CONFUCIUS4_SIDECAR_SCRIPT} — reinstall OmniVoice."
)
return True, "ok"
@classmethod
def venv_python(cls):
from engines.confucius4.bootstrap import resolve_confucius4_venv
return resolve_confucius4_venv()
@classmethod
def sidecar_script(cls):
from engines.confucius4.bootstrap import CONFUCIUS4_SIDECAR_SCRIPT
return CONFUCIUS4_SIDECAR_SCRIPT
@property
def sample_rate(self) -> int:
return self._DEFAULT_SAMPLE_RATE
@property
def supported_languages(self) -> list[str]:
# 14 languages with the caller's language passed through at synthesize
# time; "multi" on the protocol surface.
return ["multi"]
def generate(self, text: str, **kw) -> "torch.Tensor":
"""Synthesize one utterance through the Confucius4 sidecar.
kwargs honored:
* ``ref_audio`` reference clip path ``prompt_wav`` (zero-shot
cloning). Optional but recommended for a specific voice.
* ``language`` ISO code / name ``lang`` (cross-lingual transfer).
* ``ref_text`` is intentionally ignored Confucius4 is unconstrained
cloning (no reference transcript needed).
Returns a tensor of shape (1, n_samples) at :attr:`sample_rate`.
"""
forwarded: dict = {}
ref_audio = kw.get("ref_audio")
if ref_audio:
forwarded["ref_audio"] = ref_audio
language = kw.get("language")
if language:
forwarded["language"] = str(language)
return super().generate(text, **forwarded)
__all__ = ["Confucius4Backend"]
-217
View File
@@ -1,217 +0,0 @@
"""Confucius4-TTS venv probe + lazy bootstrap (issue #590).
Confucius4-TTS (netease-youdao) is an LLM-based multilingual zero-shot cloning
TTS 14 languages, no reference transcript required, Apache-2.0. Like the other
heavyweight opt-in engines (IndexTTS / MOSS-TTS-v1.5 / dots.tts) it runs in its
**own subprocess venv**: upstream targets Python 3.10 + CUDA 12.6 with its own
dependency set, which we keep off the parent interpreter.
Probe order (existing power-user installs win zero migration):
1. ``${OMNIVOICE_CONFUCIUS4_TTS_DIR}/.venv/`` the user's clone-level venv.
2. ``backend/engines/confucius4/.venv/`` this package's own venv.
3. Bootstrap: ``uv venv`` then ``uv pip install -r <clone>/requirements.txt``
(+ ``uv pip install -e <clone>`` only if upstream ever ships packaging).
Validated end-to-end 2026-07-02 (Apple Silicon, CPU): upstream ships **no
pyproject.toml/setup.py**, so ``confuciustts`` is importable only with the
clone root on ``sys.path`` the import probe and the sidecar both handle
that. The engine is opt-in (env-dir gated) and never touched unless
``OMNIVOICE_CONFUCIUS4_TTS_DIR`` is set, so this can't affect the default
install on any platform.
"""
from __future__ import annotations
import logging
import os
import shutil
import subprocess
import sys
from pathlib import Path
from typing import Optional
logger = logging.getLogger("omnivoice.confucius4.bootstrap")
#: Absolute path to the sidecar entrypoint.
CONFUCIUS4_SIDECAR_SCRIPT: Path = Path(__file__).parent / "main.py"
#: This package's owned venv (Probe 2).
_ENGINES_VENV_DIR: Path = Path(__file__).parent / ".venv"
#: Env var pointing at the user's Confucius4-TTS clone root.
_CLONE_DIR_ENV: str = "OMNIVOICE_CONFUCIUS4_TTS_DIR"
#: The package importable from the clone (verify against upstream).
_IMPORT_PROBE = "confuciustts"
_resolved_python: Optional[Path] = None
_IMPORT_PROBE_TIMEOUT_S = 15
_UV_VENV_TIMEOUT_S = 120
_UV_PIP_INSTALL_TIMEOUT_S = 1800
def invalidate() -> None:
"""Clear the resolved-python cache. Tests call this between scenarios."""
global _resolved_python
_resolved_python = None
def is_confucius4_installed() -> bool:
"""Cheap file-existence check for a usable venv (no subprocess spawn)."""
return any(cand.is_file() for cand in _probe_paths())
def resolve_confucius4_venv() -> Path:
"""Resolve the sidecar's Python interpreter (probe order in the docstring).
Memoised. Raises :exc:`RuntimeError` if none can be located and bootstrap
is unavailable."""
global _resolved_python
if _resolved_python is not None:
return _resolved_python
clone_dir = os.environ.get(_CLONE_DIR_ENV)
if clone_dir:
cand = _venv_python_path(Path(clone_dir) / ".venv")
if cand.is_file() and _venv_can_import(cand):
logger.info("Confucius4 venv resolved from %s: %s", _CLONE_DIR_ENV, cand)
_resolved_python = cand
return cand
cand = _venv_python_path(_ENGINES_VENV_DIR)
if cand.is_file() and _venv_can_import(cand):
logger.info("Confucius4 venv resolved from engines path: %s", cand)
_resolved_python = cand
return cand
if not clone_dir:
raise RuntimeError(
"Confucius4-TTS is not installed. Set the "
f"{_CLONE_DIR_ENV} environment variable to your Confucius4-TTS clone "
"(the directory that contains requirements.txt), then restart "
"OmniVoice. See docs/engines/confucius4-tts.md."
)
cand = _bootstrap_engines_venv(Path(clone_dir))
_resolved_python = cand
return cand
def _venv_python_path(venv_dir: Path) -> Path:
if sys.platform == "win32":
return venv_dir / "Scripts" / "python.exe"
return venv_dir / "bin" / "python"
def _probe_paths() -> list[Path]:
out: list[Path] = []
clone_dir = os.environ.get(_CLONE_DIR_ENV)
if clone_dir:
out.append(_venv_python_path(Path(clone_dir) / ".venv"))
out.append(_venv_python_path(_ENGINES_VENV_DIR))
return out
def _import_probe_code() -> str:
"""Probe snippet mirroring the sidecar's import semantics: upstream is not
pip-installable, so ``confuciustts`` resolves via the clone on sys.path."""
clone = os.environ.get(_CLONE_DIR_ENV, "")
if clone:
return f"import sys; sys.path.insert(0, {clone!r}); import {_IMPORT_PROBE}"
return f"import {_IMPORT_PROBE}"
def _venv_can_import(python_path: Path) -> bool:
"""Spawn the candidate python and verify ``import confuciustts`` works."""
try:
proc = subprocess.run(
[str(python_path), "-c", _import_probe_code()],
capture_output=True, timeout=_IMPORT_PROBE_TIMEOUT_S,
)
except (subprocess.TimeoutExpired, OSError) as exc:
logger.debug("Confucius4 import probe failed for %s: %s", python_path, exc)
return False
if proc.returncode != 0:
logger.debug(
"Confucius4 import probe non-zero for %s: %s",
python_path, proc.stderr.decode("utf-8", errors="replace")[:200],
)
return False
return True
def _locate_uv() -> Optional[str]:
bundled = os.environ.get("OMNIVOICE_BUNDLED_UV")
if bundled and Path(bundled).is_file():
return bundled
return shutil.which("uv")
def _bootstrap_engines_venv(clone_dir: Path) -> Path:
"""Create engines/confucius4/.venv and install the user's clone."""
uv = _locate_uv()
if not uv:
raise RuntimeError(
"uv is required to bootstrap the Confucius4-TTS venv but was not "
"found on PATH (and OMNIVOICE_BUNDLED_UV was not set). Install uv "
"from https://docs.astral.sh/uv/ and re-launch OmniVoice."
)
logger.info(
"Bootstrapping Confucius4 venv at %s from %s (several minutes on first "
"launch)", _ENGINES_VENV_DIR, clone_dir,
)
try:
subprocess.run(
[uv, "venv", "--python", "3.10", str(_ENGINES_VENV_DIR)],
check=True, timeout=_UV_VENV_TIMEOUT_S, capture_output=True,
)
except subprocess.CalledProcessError as exc:
raise RuntimeError(
f"uv venv failed for Confucius4 bootstrap at {_ENGINES_VENV_DIR}: "
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
) from exc
python_path = _venv_python_path(_ENGINES_VENV_DIR)
requirements = clone_dir / "requirements.txt"
try:
if requirements.is_file():
subprocess.run(
[uv, "pip", "install", "--python", str(python_path),
"-r", str(requirements)],
check=True, timeout=_UV_PIP_INSTALL_TIMEOUT_S, capture_output=True,
)
# Editable install only if upstream ever ships packaging metadata —
# as of 2026-07 there is none, and `uv pip install -e` on a bare clone
# fails outright. Import resolution is handled via sys.path instead.
if (clone_dir / "pyproject.toml").is_file() or (clone_dir / "setup.py").is_file():
subprocess.run(
[uv, "pip", "install", "--python", str(python_path), "-e", str(clone_dir)],
check=True, timeout=_UV_PIP_INSTALL_TIMEOUT_S, capture_output=True,
)
except subprocess.CalledProcessError as exc:
raise RuntimeError(
"uv pip install failed during Confucius4 bootstrap "
f"({clone_dir}): "
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}. "
"See docs/engines/confucius4-tts.md."
) from exc
if not _venv_can_import(python_path):
raise RuntimeError(
f"Confucius4 bootstrap completed but `import {_IMPORT_PROBE}` still "
f"fails from {python_path}. Verify {clone_dir} is a valid clone. "
"See docs/engines/confucius4-tts.md."
)
logger.info("Confucius4 venv bootstrap successful: %s", python_path)
return python_path
__all__ = [
"CONFUCIUS4_SIDECAR_SCRIPT",
"invalidate",
"is_confucius4_installed",
"resolve_confucius4_venv",
]
-216
View File
@@ -1,216 +0,0 @@
"""Confucius4-TTS sidecar entry point (issue #590).
Runs inside ``engines/confucius4/.venv`` (or the user's
``${OMNIVOICE_CONFUCIUS4_TTS_DIR}/.venv``), isolated from the OmniVoice parent.
Same isolation rationale as the IndexTTS / MOSS-TTS-v1.5 / dots.tts sidecars.
Stdlib-only at import time; ``confuciustts`` + torch are imported lazily on the
first synthesize op so the ``ready`` frame fits inside the parent's 30 s spawn
handshake.
Wire protocol length-prefixed JSON over stdin/stdout, byte-identical to
``backend/services/subprocess_backend.py``::
[ 4-byte big-endian uint32 length ][ N bytes UTF-8 JSON ]
Op flow: ready ping/pong synthesize ( progress, audio) shutdown.
Status (#590): the model API below
(``confuciustts.cli.inference.ConfuciusTTS(config_path=, device=)`` and
``model.generate(text=, lang=, prompt_wav=)`` audio tensor, ``model.sample_rate``)
is **validated end-to-end** (2026-07-02, Apple Silicon, CPU): live generate()
produced audible speech at 22 050 Hz. This sidecar's pure logic is unit-tested
in ``tests/test_confucius4_sidecar.py``. Opt-in, so it affects no one until
enabled.
Restrictions: NO imports from OmniVoice parent code. NO logging of os.environ.
"""
from __future__ import annotations
import base64
import json
import os
import struct
import sys
import traceback
MAX_FRAME_BYTES = 64 * 1024 * 1024
#: Upstream BigVGAN vocoder rate — ``target_sample_rate: 22050`` in
#: ``config/inference_config.yaml``, confirmed by a live end-to-end run
#: (2026-07-02). The real value is still re-read from ``model.sample_rate``
#: on each generate() so a future upstream change can't corrupt audio.
CONFUCIUS_SAMPLE_RATE = 22050
def _send(stream, obj: dict) -> None:
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
stream.write(struct.pack("!I", len(body)))
stream.write(body)
stream.flush()
def _recv(stream):
header = stream.read(4)
if len(header) < 4:
return None # EOF
(n,) = struct.unpack("!I", header)
if n > MAX_FRAME_BYTES:
raise IOError(f"frame too large: {n}")
body = bytearray()
while len(body) < n:
chunk = stream.read(n - len(body))
if not chunk:
raise IOError("short read")
body.extend(chunk)
return json.loads(bytes(body).decode("utf-8"))
def _measure_vram_mb() -> float:
try:
import torch
if torch.cuda.is_available():
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
except Exception:
pass
return 0.0
_model = None
def _config_path() -> str:
"""Locate Confucius4's inference config (``config/inference_config.yaml``)
under the clone, or an explicit override."""
explicit = os.environ.get("OMNIVOICE_CONFUCIUS4_CONFIG")
if explicit:
return explicit
clone = os.environ.get("OMNIVOICE_CONFUCIUS4_TTS_DIR", "")
return os.path.join(clone, "config", "inference_config.yaml")
def _ensure_clone_on_sys_path() -> None:
"""Make ``import confuciustts`` resolve from the user's clone.
Upstream Confucius4-TTS is **not pip-installable** (no pyproject.toml /
setup.py as of 2026-07); its own ``example.py`` sys.path-inserts the repo
root instead. Mirror that here so the sidecar works from a plain
``uv pip install -r requirements.txt`` venv. Inserted at position 0 so the
clone the user pointed at always wins over any stale installed copy.
"""
clone = os.environ.get("OMNIVOICE_CONFUCIUS4_TTS_DIR", "")
if clone and clone not in sys.path:
sys.path.insert(0, clone)
def _load_model(stdout):
"""Cold-construct the Confucius4 model (CUDA, else CPU — both validated)."""
global _model
if _model is not None:
return _model
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 0})
_ensure_clone_on_sys_path()
import torch
from confuciustts.cli.inference import ConfuciusTTS # type: ignore[import-not-found]
device = "cuda" if torch.cuda.is_available() else "cpu"
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 50})
_model = ConfuciusTTS(config_path=_config_path(), device=device)
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 100})
return _model
def _tensor_to_pcm_b64(audio, sample_rate: int) -> tuple[str, int, int]:
import numpy as np
arr = audio.detach().to("cpu").float().numpy() if hasattr(audio, "detach") else np.asarray(audio)
arr = np.asarray(arr, dtype=np.float32).squeeze()
if arr.ndim > 1:
arr = arr.mean(axis=0)
arr = np.clip(arr, -1.0, 1.0)
pcm = (arr * 32767.0).astype(np.int16).tobytes()
return base64.b64encode(pcm).decode("ascii"), int(sample_rate), int(arr.shape[0])
def _normalize_language(raw):
"""Confucius4 expects an ISO-ish language code (e.g. 'en', 'zh'). Empty /
'auto' 'en' as a safe default (the API requires a lang)."""
if not raw or not isinstance(raw, str):
return "en"
s = raw.strip().lower()
if not s or s == "auto":
return "en"
return s[:2] if (len(s) >= 2 and s[:2].isalpha()) else s
def _handle_synthesize(msg: dict, stdout) -> None:
text = msg.get("text")
if not text or not isinstance(text, str):
raise ValueError("synthesize: missing or non-string 'text'")
model = _load_model(stdout)
gen_kwargs: dict = {"text": text, "lang": _normalize_language(msg.get("language"))}
ref_audio = msg.get("ref_audio")
if ref_audio:
gen_kwargs["prompt_wav"] = ref_audio
audio = model.generate(**gen_kwargs)
sample_rate = int(getattr(model, "sample_rate", CONFUCIUS_SAMPLE_RATE))
pcm_b64, sr, n_samples = _tensor_to_pcm_b64(audio, sample_rate)
_send(stdout, {
"op": "audio",
"audio_pcm_b64": pcm_b64,
"sample_rate": sr,
"n_samples": n_samples,
})
def main() -> int:
stdin = sys.stdin.buffer
stdout = sys.stdout.buffer
_send(stdout, {
"op": "ready",
"engine": "confucius4-tts",
"sample_rate": CONFUCIUS_SAMPLE_RATE,
})
while True:
try:
msg = _recv(stdin)
except Exception as exc:
_send(stdout, {
"op": "error", "stage": "recv",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
return 1
if msg is None:
return 0
op = msg.get("op") if isinstance(msg, dict) else None
try:
if op == "ping":
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
elif op == "synthesize":
_handle_synthesize(msg, stdout)
elif op == "shutdown":
return 0
else:
_send(stdout, {"op": "error", "stage": "dispatch",
"message": f"unknown op: {op!r}"})
except Exception as exc:
_send(stdout, {
"op": "error", "stage": op or "unknown",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
if __name__ == "__main__":
sys.exit(main())
-187
View File
@@ -1,187 +0,0 @@
"""dots.tts sidecar package (issue #498).
dots.tts is rednote-hilab's 2B fully-continuous autoregressive TTS — widely
cited as among the strongest open zero-shot voice-cloning models. 24
languages, 48 kHz output, Apache-2.0 (code + checkpoints).
It runs in its own subprocess **and its own venv**, isolated from the
OmniVoice parent, for the same ``transformers`` reason as IndexTTS and
MOSS-TTS-v1.5: dots.tts pins ``transformers==4.57.0`` (verified against
``constraints/recommended.txt``), while OmniVoice pins
``transformers>=5.3.0``. The two cannot share one interpreter.
Cross-platform honesty (the strict default-parity rule): dots.tts's
upstream package declares **Linux + macOS** classifiers only **no
Windows** and its device code is **CUDA-or-CPU with no MPS branch**
(verified in ``runtime.py``). So:
* It is **opt-in** (engine-picker selection + a user-provided clone),
never a default so it never becomes a broken default on any platform.
* ``is_available()`` returns ``False`` with a clear reason on **Windows**
rather than offering an engine that can't run there. Windows users are
pointed at WSL2 / a Linux or macOS host.
* ``gpu_compat = ("cuda", "cpu")`` no MPS claim. On Apple Silicon the
upstream package runs on CPU (slow but correct); the faster MLX path is
a community port we deliberately don't auto-wire here.
Three public entry points: ``DotsTTSBackend`` (this module), ``main.py``
(sidecar, runs under dots.tts's ``transformers==4.57`` venv — never imported
by the parent), and ``bootstrap.py`` (venv probe + lazy bootstrap).
"""
from __future__ import annotations
import logging
import sys
from typing import TYPE_CHECKING
from services.subprocess_backend import SubprocessBackend
if TYPE_CHECKING:
import torch # noqa: F401
logger = logging.getLogger("omnivoice.dots_tts")
class DotsTTSBackend(SubprocessBackend):
"""dots.tts (rednote-hilab) — 2B, 24 langs, zero-shot clone, CUDA/CPU.
Runs in a long-lived sidecar over length-prefixed JSON-over-stdio in a
dedicated venv (``transformers==4.57.0``). First synthesize cold-loads
the ~9 GB checkpoint (bf16 on CUDA); subsequent calls reuse the process.
Installation (OmniVoice prefers a user's existing ``${DIR}/.venv``)::
git clone https://github.com/rednote-hilab/dots.tts.git
cd dots.tts
uv venv && uv pip install -e . -c constraints/recommended.txt
Set ``OMNIVOICE_DOTS_TTS_DIR`` to the clone root. OmniVoice creates
``backend/engines/dots_tts/.venv`` lazily on first launch if no venv
exists yet; the user's existing ``${DIR}/.venv`` is preferred if present.
Best cloning quality uses the ``dots.tts-soar`` checkpoint (the default)
and BOTH a reference clip and its exact transcript (continuation
cloning). License: Apache-2.0.
"""
id = "dots-tts"
display_name = (
"dots.tts (2B, 24 langs, zero-shot clone, CUDA/CPU, 48 kHz, Apache-2.0)"
)
supports_voice_design = False # requires ref audio for timbre cloning
# dots.tts emits 48 kHz (verified via checkpoint vocoder.sample_rate).
_DEFAULT_SAMPLE_RATE = 48000
# CUDA + CPU only; no MPS branch in upstream runtime.py.
gpu_compat = ("cuda", "cpu")
# ── availability ───────────────────────────────────────────────────────
@classmethod
def is_available(cls) -> tuple[bool, str]:
# Cross-platform parity: dots.tts upstream is Linux/macOS-only (no
# Windows classifier, no Windows install path). Refuse cleanly on
# Windows instead of advertising an engine that can't run.
if sys.platform == "win32":
return False, (
"dots.tts is not supported on Windows — upstream targets "
"Linux and macOS only. Run OmniVoice under WSL2, or use a "
"Linux/macOS host. See docs/engines/dots-tts.md."
)
# Do NOT import dots_tts here: it pins transformers==4.57, which can't
# coexist with the parent's transformers>=5.3 in one interpreter —
# the reason for the subprocess isolation. Verify the venv on disk
# only; a real health-check is gated on the user's "Test engine"
# action in Settings.
from engines.dots_tts.bootstrap import (
DOTS_TTS_SIDECAR_SCRIPT,
is_dots_tts_installed,
)
if not is_dots_tts_installed():
return False, (
"dots.tts venv not found. Set OMNIVOICE_DOTS_TTS_DIR to your "
"dots.tts clone (the directory containing pyproject.toml) and "
"restart OmniVoice. CUDA or CPU only (no MPS). See "
"docs/engines/dots-tts.md for the full install walk-through."
)
if not DOTS_TTS_SIDECAR_SCRIPT.exists():
return False, (
"dots.tts sidecar script missing at "
f"{DOTS_TTS_SIDECAR_SCRIPT} — reinstall OmniVoice."
)
return True, "ok (CUDA when present, else CPU)"
@classmethod
def venv_python(cls):
from engines.dots_tts.bootstrap import resolve_dots_tts_venv
return resolve_dots_tts_venv()
@classmethod
def sidecar_script(cls):
from engines.dots_tts.bootstrap import DOTS_TTS_SIDECAR_SCRIPT
return DOTS_TTS_SIDECAR_SCRIPT
# ── TTSBackend protocol ────────────────────────────────────────────────
@property
def sample_rate(self) -> int:
return self._DEFAULT_SAMPLE_RATE
@property
def supported_languages(self) -> list[str]:
# 24 languages with auto-detect; expose "multi" on the protocol
# surface and translate the caller's language at synthesize time.
return ["multi"]
# ── generate (parent-side arbitration) ─────────────────────────────────
def generate(self, text: str, **kw) -> "torch.Tensor":
"""Synthesize one utterance through the dots.tts sidecar.
kwargs honored:
* ``ref_audio`` reference clip path ``prompt_audio_path``
(zero-shot cloning). Optional.
* ``ref_text`` the reference transcript ``prompt_text``.
Best cloning fidelity ("continuation"). Upstream
REQUIRES ``prompt_audio_path`` when ``prompt_text``
is set, so we drop a stray ref_text with no
ref_audio rather than let the sidecar raise.
* ``language`` ISO code / name / None (auto-detect).
* ``num_step`` flow-matching steps ``num_steps`` (default 10;
use 4 for the ``dots.tts-mf`` checkpoint).
* ``guidance_scale`` CFG (default 1.2; >2 amplifies energy).
Returns a tensor of shape (1, n_samples) at :attr:`sample_rate`.
"""
forwarded: dict = {}
ref_audio = kw.get("ref_audio")
if ref_audio:
forwarded["ref_audio"] = ref_audio
ref_text = kw.get("ref_text")
if ref_text:
# continuation cloning — only valid alongside ref_audio.
forwarded["ref_text"] = ref_text
elif kw.get("ref_text"):
logger.info(
"dots-tts: ref_text supplied without ref_audio; ignoring "
"(upstream requires prompt_audio_path when prompt_text is set)."
)
language = kw.get("language")
if language:
forwarded["language"] = str(language)
# OmniVoice's generic num_step default is 16; dots.tts's own default
# is 10. Honor an explicit value, else use the dots-appropriate 10.
num_step = kw.get("num_step")
forwarded["num_steps"] = int(num_step) if num_step is not None else 10
# dots.tts's own CFG default is 1.2 (the generic 2.0 over-energises).
guidance = kw.get("guidance_scale")
forwarded["guidance_scale"] = float(guidance) if guidance is not None else 1.2
return super().generate(text, **forwarded)
__all__ = ["DotsTTSBackend"]
-226
View File
@@ -1,226 +0,0 @@
"""dots.tts venv probe + lazy bootstrap (issue #498).
Resolves which Python interpreter runs the dots.tts sidecar. Mirrors
``engines.indextts.bootstrap`` / ``engines.moss_tts_v15.bootstrap`` because
dots.tts has the same shape of problem: a hard ``transformers==4.57.0`` pin
that conflicts with the parent's ``transformers>=5.3`` — so it runs in its
own venv.
Probe order (priority existing power-user installs win, zero migration):
1. ``${OMNIVOICE_DOTS_TTS_DIR}/.venv/`` the user's clone-level venv.
2. ``backend/engines/dots_tts/.venv/`` this package's own venv.
3. Bootstrap: ``uv venv`` then ``uv pip install -e <clone> -c
<clone>/constraints/recommended.txt`` (the upstream-pinned stack:
torch==2.8.0, transformers==4.57.0, ).
Caching: memoised after first success. Tests reset via :func:`invalidate`.
Security: same posture as IndexTTS bootstrap never touches HF_TOKEN; the
sidecar's stderr is redacted by the parent's ``HFTokenRedactor``; the
editable install comes from a user-controlled clone they already trust.
"""
from __future__ import annotations
import logging
import os
import shutil
import subprocess
import sys
from pathlib import Path
from typing import Optional
logger = logging.getLogger("omnivoice.dots_tts.bootstrap")
#: Absolute path to the sidecar entrypoint.
DOTS_TTS_SIDECAR_SCRIPT: Path = Path(__file__).parent / "main.py"
#: This package's owned venv (Probe 2).
_ENGINES_VENV_DIR: Path = Path(__file__).parent / ".venv"
#: Env var pointing at the user's dots.tts clone root.
_CLONE_DIR_ENV: str = "OMNIVOICE_DOTS_TTS_DIR"
#: Per-process resolution cache. Cleared by :func:`invalidate` for tests.
_resolved_python: Optional[Path] = None
_IMPORT_PROBE_TIMEOUT_S = 15
_UV_VENV_TIMEOUT_S = 120
_UV_PIP_INSTALL_TIMEOUT_S = 1800
# ── public API ────────────────────────────────────────────────────────────
def invalidate() -> None:
"""Clear the resolved-python cache. Tests call this between scenarios."""
global _resolved_python
_resolved_python = None
def is_dots_tts_installed() -> bool:
"""Cheap file-existence check for a usable dots.tts venv. Does NOT spawn
the venv Python that's saved for :func:`resolve_dots_tts_venv`."""
for cand in _probe_paths():
if cand.is_file():
return True
return False
def resolve_dots_tts_venv() -> Path:
"""Resolve the sidecar's Python interpreter (probe order in the module
docstring). Memoised. Raises :exc:`RuntimeError` if none can be located
and the bootstrap path is unavailable."""
global _resolved_python
if _resolved_python is not None:
return _resolved_python
clone_dir = os.environ.get(_CLONE_DIR_ENV)
# Probe 1 — user's clone-level venv.
if clone_dir:
cand = _venv_python_path(Path(clone_dir) / ".venv")
if cand.is_file() and _venv_can_import_dots(cand):
logger.info(
"dots.tts venv resolved from %s: %s", _CLONE_DIR_ENV, cand,
)
_resolved_python = cand
return cand
# Probe 2 — this package's own venv.
cand = _venv_python_path(_ENGINES_VENV_DIR)
if cand.is_file() and _venv_can_import_dots(cand):
logger.info("dots.tts venv resolved from engines path: %s", cand)
_resolved_python = cand
return cand
# Probe 3 — bootstrap.
if not clone_dir:
raise RuntimeError(
"dots.tts is not installed. Set the "
f"{_CLONE_DIR_ENV} environment variable to your dots.tts clone "
"(the directory that contains pyproject.toml and constraints/), "
"then restart OmniVoice. See docs/engines/dots-tts.md for the "
"full install walk-through."
)
cand = _bootstrap_engines_venv(Path(clone_dir))
_resolved_python = cand
return cand
# ── internals ─────────────────────────────────────────────────────────────
def _venv_python_path(venv_dir: Path) -> Path:
if sys.platform == "win32":
return venv_dir / "Scripts" / "python.exe"
return venv_dir / "bin" / "python"
def _probe_paths() -> list[Path]:
out: list[Path] = []
clone_dir = os.environ.get(_CLONE_DIR_ENV)
if clone_dir:
out.append(_venv_python_path(Path(clone_dir) / ".venv"))
out.append(_venv_python_path(_ENGINES_VENV_DIR))
return out
def _venv_can_import_dots(python_path: Path) -> bool:
"""Spawn the candidate python and verify ``import dots_tts.runtime`` works.
Bounded by ``_IMPORT_PROBE_TIMEOUT_S``. False on any failure."""
try:
proc = subprocess.run(
[str(python_path), "-c", "import dots_tts.runtime"],
capture_output=True,
timeout=_IMPORT_PROBE_TIMEOUT_S,
)
except (subprocess.TimeoutExpired, OSError) as exc:
logger.debug("dots.tts import probe failed for %s: %s", python_path, exc)
return False
if proc.returncode != 0:
logger.debug(
"dots.tts import probe non-zero for %s: %s",
python_path,
proc.stderr.decode("utf-8", errors="replace")[:200],
)
return False
return True
def _locate_uv() -> Optional[str]:
bundled = os.environ.get("OMNIVOICE_BUNDLED_UV")
if bundled and Path(bundled).is_file():
return bundled
return shutil.which("uv")
def _bootstrap_engines_venv(clone_dir: Path) -> Path:
"""Create engines/dots_tts/.venv and editable-install the user's clone
with the upstream constraints file."""
uv = _locate_uv()
if not uv:
raise RuntimeError(
"uv is required to bootstrap the dots.tts venv but was not found "
"on PATH (and OMNIVOICE_BUNDLED_UV was not set). Install uv from "
"https://docs.astral.sh/uv/ and re-launch OmniVoice, or set "
"OMNIVOICE_BUNDLED_UV to the absolute path of a uv binary."
)
logger.info(
"Bootstrapping dots.tts venv at %s from %s (this can take several "
"minutes on first launch)", _ENGINES_VENV_DIR, clone_dir,
)
try:
subprocess.run(
[uv, "venv", str(_ENGINES_VENV_DIR)],
check=True, timeout=_UV_VENV_TIMEOUT_S, capture_output=True,
)
except subprocess.CalledProcessError as exc:
raise RuntimeError(
f"uv venv failed for dots.tts bootstrap at {_ENGINES_VENV_DIR}: "
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
) from exc
python_path = _venv_python_path(_ENGINES_VENV_DIR)
install_cmd = [
uv, "pip", "install",
"--python", str(python_path),
"-e", str(clone_dir),
]
# Apply the upstream pin set when it ships with the clone.
constraints = clone_dir / "constraints" / "recommended.txt"
if constraints.is_file():
install_cmd += ["-c", str(constraints)]
try:
subprocess.run(
install_cmd, check=True,
timeout=_UV_PIP_INSTALL_TIMEOUT_S, capture_output=True,
)
except subprocess.CalledProcessError as exc:
raise RuntimeError(
"uv pip install -e failed during dots.tts bootstrap "
f"({clone_dir}): "
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}. "
"See docs/engines/dots-tts.md."
) from exc
if not _venv_can_import_dots(python_path):
raise RuntimeError(
"dots.tts bootstrap completed but `import dots_tts.runtime` still "
f"fails from {python_path}. Verify that {clone_dir} is a valid "
"dots.tts clone. See docs/engines/dots-tts.md."
)
logger.info("dots.tts venv bootstrap successful: %s", python_path)
return python_path
__all__ = [
"DOTS_TTS_SIDECAR_SCRIPT",
"invalidate",
"is_dots_tts_installed",
"resolve_dots_tts_venv",
]
-255
View File
@@ -1,255 +0,0 @@
"""dots.tts sidecar entry point (issue #498).
Runs inside ``engines/dots_tts/.venv`` (or the user's existing
``${OMNIVOICE_DOTS_TTS_DIR}/.venv``) with ``transformers==4.57.0``, isolated
from the OmniVoice parent (``transformers>=5.3``). Same isolation rationale
as the IndexTTS / MOSS-TTS-v1.5 sidecars.
Stdlib-only at import time; ``dots_tts`` + torch are imported lazily on the
first synthesize op so the ``ready`` frame fits inside the parent's 30 s
spawn handshake even on a cold filesystem.
Wire protocol length-prefixed JSON over stdin/stdout, byte-identical to
``backend/services/subprocess_backend.py``::
[ 4-byte big-endian uint32 length ][ N bytes UTF-8 JSON ]
Op flow:
1. Sidecar -> parent: {"op": "ready", "engine": "dots-tts",
"sample_rate": 48000}
2. parent -> sidecar: {"op": "ping"} -> {"op": "pong", "vram_mb": N}
3. parent -> sidecar: {"op": "synthesize", "text": "...",
"ref_audio": "/path/ref.wav",
"ref_text": "transcript", "language": "EN",
"num_steps": 10, "guidance_scale": 1.2}
-> {"op": "progress", ...} (cold load) then
-> {"op": "audio", "audio_pcm_b64": "...", "sample_rate": 48000,
"n_samples": N}
4. parent -> sidecar: {"op": "shutdown"} -> exit 0
Restrictions: NO imports from OmniVoice parent code (different venv). NO
logging of ``os.environ`` contents. Single-frame DoS cap matches the
parent's ``MAX_FRAME_BYTES``.
"""
from __future__ import annotations
import base64
import json
import os
import struct
import sys
import traceback
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
MAX_FRAME_BYTES = 64 * 1024 * 1024
#: dots.tts emits 48 kHz (checkpoint vocoder.sample_rate). Advertised in the
#: ready frame; the real value is re-read from each generate() result.
DOTS_SAMPLE_RATE = 48000
#: Default checkpoint. ``-soar`` is the best-cloning variant; ``-mf`` is the
#: fastest (use num_steps=4). Overridable for air-gapped / mirror installs.
_DEFAULT_REPO = "rednote-hilab/dots.tts-soar"
# ── wire protocol ─────────────────────────────────────────────────────────
def _send(stream, obj: dict) -> None:
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
stream.write(struct.pack("!I", len(body)))
stream.write(body)
stream.flush()
def _recv(stream):
header = stream.read(4)
if len(header) < 4:
return None # EOF
(n,) = struct.unpack("!I", header)
if n > MAX_FRAME_BYTES:
raise IOError(f"frame too large: {n}")
body = bytearray()
while len(body) < n:
chunk = stream.read(n - len(body))
if not chunk:
raise IOError("short read")
body.extend(chunk)
return json.loads(bytes(body).decode("utf-8"))
def _measure_vram_mb() -> float:
"""This sidecar's own GPU memory in MB (MM2-08). 0 on CPU. Never raises."""
try:
import torch
if torch.cuda.is_available():
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
except Exception:
pass
return 0.0
# ── model loading (lazy, on first synthesize) ─────────────────────────────
# Module-level singleton — (runtime,). Device is auto-selected inside the
# dots.tts runtime (cuda-or-cpu, no MPS); we don't pass a device.
_runtime = None
def _load_runtime(stdout):
"""Cold-construct the dots.tts runtime.
``DotsTtsRuntime.from_pretrained`` auto-selects cuda-or-cpu internally
(no MPS path). precision is bf16 on CUDA; on CPU we fall back to fp32
(bf16 CPU kernels are spotty). Both overridable via env.
"""
global _runtime
if _runtime is not None:
return _runtime
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 0})
import torch
from dots_tts.runtime import DotsTtsRuntime # type: ignore[import-not-found]
repo = os.environ.get("OMNIVOICE_DOTS_TTS_MODEL", _DEFAULT_REPO)
default_precision = "bfloat16" if torch.cuda.is_available() else "float32"
precision = os.environ.get("OMNIVOICE_DOTS_TTS_PRECISION", default_precision)
optimize = os.environ.get("OMNIVOICE_DOTS_TTS_OPTIMIZE", "0") == "1"
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 50})
_runtime = DotsTtsRuntime.from_pretrained(
repo,
precision=precision,
optimize=optimize,
)
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 100})
return _runtime
def _tensor_to_pcm_b64(audio, sample_rate: int) -> tuple[str, int, int]:
"""Convert a torch waveform tensor (1, N) in [-1, 1] to base64 int16 PCM."""
import numpy as np
arr = audio.detach().to("cpu").float().numpy()
arr = np.asarray(arr, dtype=np.float32).squeeze()
if arr.ndim > 1:
arr = arr.mean(axis=0) # defensive downmix to mono
arr = np.clip(arr, -1.0, 1.0)
pcm = (arr * 32767.0).astype(np.int16).tobytes()
return base64.b64encode(pcm).decode("ascii"), int(sample_rate), int(arr.shape[0])
def _normalize_language(raw):
"""Map OmniVoice's language value to what dots.tts accepts, or None.
dots.tts accepts None/"auto_detect", ISO codes upper-cased ("EN"/"ZH"),
or names ("english"). A 2-letter ISO code is upper-cased; anything else
is passed through; empty / "auto" None (auto-detect)."""
if not raw or not isinstance(raw, str):
return None
s = raw.strip()
if not s or s.lower() == "auto":
return None
if len(s) == 2 and s.isalpha():
return s.upper()
return s
def _handle_synthesize(msg: dict, stdout) -> None:
"""Dispatch one synthesize request. Emits the audio frame or raises."""
text = msg.get("text")
if not text or not isinstance(text, str):
raise ValueError("synthesize: missing or non-string 'text'")
runtime = _load_runtime(stdout)
gen_kwargs: dict = {
"text": text,
"num_steps": int(msg.get("num_steps", 10)),
"guidance_scale": float(msg.get("guidance_scale", 1.2)),
}
ref_audio = msg.get("ref_audio")
if ref_audio:
gen_kwargs["prompt_audio_path"] = ref_audio
ref_text = msg.get("ref_text")
if ref_text:
# continuation cloning — upstream requires prompt_audio_path when
# prompt_text is set (the parent already enforces this).
gen_kwargs["prompt_text"] = ref_text
language = _normalize_language(msg.get("language"))
if language:
gen_kwargs["language"] = language
result = runtime.generate(**gen_kwargs)
audio = result["audio"]
sample_rate = int(result.get("sample_rate", DOTS_SAMPLE_RATE))
pcm_b64, sr, n_samples = _tensor_to_pcm_b64(audio, sample_rate)
_send(stdout, {
"op": "audio",
"audio_pcm_b64": pcm_b64,
"sample_rate": sr,
"n_samples": n_samples,
})
# ── main loop ─────────────────────────────────────────────────────────────
def main() -> int:
stdin = sys.stdin.buffer
stdout = sys.stdout.buffer
# Ready handshake fires BEFORE any heavy import.
_send(stdout, {
"op": "ready",
"engine": "dots-tts",
"sample_rate": DOTS_SAMPLE_RATE,
})
while True:
try:
msg = _recv(stdin)
except Exception as exc:
_send(stdout, {
"op": "error",
"stage": "recv",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
return 1
if msg is None:
return 0
op = msg.get("op") if isinstance(msg, dict) else None
try:
if op == "ping":
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
elif op == "synthesize":
_handle_synthesize(msg, stdout)
elif op == "shutdown":
return 0
else:
_send(stdout, {
"op": "error",
"stage": "dispatch",
"message": f"unknown op: {op!r}",
})
except Exception as exc:
_send(stdout, {
"op": "error",
"stage": op or "unknown",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
if __name__ == "__main__":
sys.exit(main())
-5
View File
@@ -79,11 +79,6 @@ class IndexTTS2Backend(SubprocessBackend):
display_name = "IndexTTS2 (emotion control, duration control, zero-shot)"
supports_voice_design = False # requires ref audio for timbre
_DEFAULT_SAMPLE_RATE = 24000
# Explicit so IndexTTS2 stops advertising the inherited CPU-only default:
# the sidecar runs the IndexTTS PyTorch model on CUDA when present, else
# CPU. ROCm left unclaimed (the sidecar's own venv would need a ROCm torch);
# a ROCm host honestly resolves to cpu_fallback.
gpu_compat = ("cuda", "cpu")
@classmethod
def is_available(cls) -> tuple[bool, str]:
+1 -19
View File
@@ -74,24 +74,6 @@ import traceback
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
MAX_FRAME_BYTES = 64 * 1024 * 1024
def _measure_vram_mb() -> float:
"""This sidecar's own GPU memory in MB, for the loaded-models panel
(MM2-08). The parent can't see a child's VRAM, so we self-report it in the
pong. Degrades to 0 on CPU / when torch isn't loaded yet — never raises."""
try:
import torch # already a dep inside the indextts venv
if torch.cuda.is_available():
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
mps = getattr(torch.backends, "mps", None)
if mps is not None and mps.is_available():
drv = getattr(torch.mps, "driver_allocated_memory", None)
if drv:
return round(drv() / (1024 ** 2), 1)
except Exception:
pass
return 0.0
# Sample rate IndexTTS-2 emits natively. Advertised in the ready frame so
# the parent doesn't have to import IndexTTS just to learn the rate.
INDEXTTS_SAMPLE_RATE = 24000
@@ -300,7 +282,7 @@ def main() -> int:
op = msg.get("op") if isinstance(msg, dict) else None
try:
if op == "ping":
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
_send(stdout, {"op": "pong"})
elif op == "synthesize":
_handle_synthesize(msg, stdout)
elif op == "shutdown":
-194
View File
@@ -1,194 +0,0 @@
"""MOSS-TTS-v1.5 sidecar package (issue #498).
MOSS-TTS-v1.5 is OpenMOSS's 8B flagship TTS — a Qwen3-8B language backbone
plus a 1.6B audio codec, 31 languages, zero-shot voice cloning, token-level
duration control and inline ``[pause Ns]`` markers. Apache-2.0.
It runs in its own subprocess **and its own venv**, isolated from the
OmniVoice parent process, for the *same* reason IndexTTS does: a hard
``transformers`` version conflict. MOSS-TTS-v1.5's ``torch-runtime`` extra
pins ``transformers==5.0.0`` (verified against the upstream
``pyproject.toml``), while OmniVoice pins ``transformers>=5.3.0``. The two
cannot share one interpreter so MOSS lives behind ``SubprocessBackend``
with a dedicated venv, exactly like ``engines.indextts``.
Three public entry points live in this package:
* ``MossTTSV15Backend`` (this module) the SubprocessBackend subclass
that ``services.tts_backend._LAZY_REGISTRY`` resolves on first access.
Defined HERE (not in ``services.tts_backend``) to break the import
cycle: ``services.subprocess_backend`` imports ``TTSBackend`` from
``services.tts_backend``, so the backend class must live downstream of
that module finishing its import. Same indirection as IndexTTS /
Supertonic-3.
* ``main.py`` the sidecar entrypoint (runs under MOSS's venv with
``transformers==5.0.0``; never imported by the parent).
* ``bootstrap.py`` the venv-probe + lazy-bootstrap helper.
Do NOT import ``main.py`` from the parent process it runs under a
different venv (``transformers==5.0.0``) and importing it in-process would
re-introduce the exact conflict this isolation exists to avoid.
Hardware honesty (cross-platform rule): MOSS-TTS-v1.5's upstream documents
only CUDA and CPU. There is **no documented or tested MPS path** the
custom ``trust_remote_code`` modelling code and the separate audio
tokenizer are unverified on Apple Silicon. We therefore advertise
``gpu_compat = ("cuda", "cpu")`` and the sidecar selects ``cuda`` when
present else ``cpu`` it never silently routes to MPS where it might
crash. On Apple Silicon the engine honestly resolves to CPU (slow but
correct), and the engine is opt-in regardless, so it never becomes a
broken default on any platform.
"""
from __future__ import annotations
import logging
from typing import TYPE_CHECKING
from services.subprocess_backend import SubprocessBackend
if TYPE_CHECKING:
import torch # noqa: F401
logger = logging.getLogger("omnivoice.moss_tts_v15")
#: 1 second of audio ≈ 12.5 codec tokens (MOSS-TTS-v1.5 model card). Used to
#: translate OmniVoice's ``duration`` (seconds) into the model's ``tokens``
#: duration-control argument.
TOKENS_PER_SECOND: float = 12.5
class MossTTSV15Backend(SubprocessBackend):
"""MOSS-TTS-v1.5 (OpenMOSS) — 8B, 31 langs, zero-shot clone, CUDA/CPU.
Runs in a long-lived sidecar over length-prefixed JSON-over-stdio in a
dedicated venv (``transformers==5.0.0``). The first synthesize cold-loads
~16 GB of bf16 weights (CUDA) / fp32 (CPU); subsequent calls reuse the
process and the in-memory model.
Installation (transparent to power users who already cloned MOSS-TTS
OmniVoice prefers their existing ``${DIR}/.venv``)::
git clone https://github.com/OpenMOSS/MOSS-TTS.git
cd MOSS-TTS
# CUDA host:
uv venv && uv pip install -e ".[torch-runtime]"
# non-CUDA host (CPU): install plain torch/transformers instead of +cu128
Set ``OMNIVOICE_MOSS_TTS_V15_DIR`` to the clone root. OmniVoice creates
``backend/engines/moss_tts_v15/.venv`` lazily on first launch if no venv
exists yet (CUDA hosts only the upstream ``torch-runtime`` extra is
``+cu128``); the user's existing ``${DIR}/.venv`` is preferred if
present, so no re-install is needed.
License: Apache-2.0 (code + weights) no acceptance gate needed.
"""
id = "moss-tts-v15"
display_name = (
"MOSS-TTS-v1.5 (8B, 31 langs, zero-shot clone, CUDA/CPU, Apache-2.0)"
)
supports_voice_design = False # requires ref audio for timbre cloning
_DEFAULT_SAMPLE_RATE = 24000
# Honest hardware surface: upstream documents CUDA + CPU only. MPS is
# undocumented / untested, so we do NOT claim it (cross-platform rule).
gpu_compat = ("cuda", "cpu")
# ── availability ───────────────────────────────────────────────────────
@classmethod
def is_available(cls) -> tuple[bool, str]:
# IMPORTANT: do NOT attempt to import MOSS / its transformers==5.0.0
# here. The parent pins transformers>=5.3 — co-importing the two in
# one interpreter is exactly the conflict this subprocess isolation
# exists to avoid. We only verify the venv exists on disk; a real
# health-check (spawn + ping) is gated on the user's "Test engine"
# action in Settings, same as IndexTTS.
from engines.moss_tts_v15.bootstrap import (
MOSS_TTS_V15_SIDECAR_SCRIPT,
is_moss_tts_v15_installed,
)
if not is_moss_tts_v15_installed():
return False, (
"MOSS-TTS-v1.5 venv not found. Set OMNIVOICE_MOSS_TTS_V15_DIR "
"to your MOSS-TTS clone (the directory containing pyproject.toml) "
"and restart OmniVoice. CUDA or CPU only (no MPS). See "
"docs/engines/moss-tts-v15.md for the full install walk-through."
)
if not MOSS_TTS_V15_SIDECAR_SCRIPT.exists():
return False, (
"MOSS-TTS-v1.5 sidecar script missing at "
f"{MOSS_TTS_V15_SIDECAR_SCRIPT} — reinstall OmniVoice."
)
return True, "ok (CUDA when present, else CPU)"
@classmethod
def venv_python(cls):
from engines.moss_tts_v15.bootstrap import resolve_moss_tts_v15_venv
return resolve_moss_tts_v15_venv()
@classmethod
def sidecar_script(cls):
from engines.moss_tts_v15.bootstrap import MOSS_TTS_V15_SIDECAR_SCRIPT
return MOSS_TTS_V15_SIDECAR_SCRIPT
# ── TTSBackend protocol ────────────────────────────────────────────────
@property
def sample_rate(self) -> int:
return self._DEFAULT_SAMPLE_RATE
@property
def supported_languages(self) -> list[str]:
# 31 languages with multilingual handling; expose "multi" on the
# protocol surface (same as OmniVoice / CosyVoice / Supertonic-3) and
# translate the caller's language at synthesize time.
return ["multi"]
# ── generate (parent-side arbitration) ─────────────────────────────────
def generate(self, text: str, **kw) -> "torch.Tensor":
"""Synthesize one utterance through the MOSS-TTS-v1.5 sidecar.
kwargs honored:
* ``ref_audio`` path to a reference clip. When present, MOSS
runs zero-shot voice cloning (``reference=``).
Optional: without it the model uses its own
default voice.
* ``ref_text`` accepted but unused in clone mode (MOSS's
zero-shot path needs only the audio); kept in
the signature so the common call-site doesn't
need engine-specific knowledge.
* ``language`` ISO code or name; mapped to a MOSS language name
in the sidecar, omitted (auto-detect) if unknown.
* ``duration`` target seconds ``tokens`` (1 s 12.5 tokens).
* ``max_new_tokens`` generation cap (default 4096).
Returns a tensor of shape (1, n_samples) at :attr:`sample_rate`.
"""
forwarded: dict = {}
ref_audio = kw.get("ref_audio")
if ref_audio:
forwarded["ref_audio"] = ref_audio
ref_text = kw.get("ref_text")
if ref_text:
forwarded["ref_text"] = ref_text
language = kw.get("language")
if language:
forwarded["language"] = str(language)
duration = kw.get("duration")
if duration is not None:
target_tokens = int(float(duration) * TOKENS_PER_SECOND)
if target_tokens > 0:
forwarded["tokens"] = target_tokens
max_new_tokens = kw.get("max_new_tokens")
if max_new_tokens is not None:
forwarded["max_new_tokens"] = int(max_new_tokens)
return super().generate(text, **forwarded)
__all__ = ["MossTTSV15Backend", "TOKENS_PER_SECOND"]
-272
View File
@@ -1,272 +0,0 @@
"""MOSS-TTS-v1.5 venv probe + lazy bootstrap (issue #498).
The parent process needs to know *which Python interpreter* to spawn the
MOSS-TTS-v1.5 sidecar under. This module owns that resolution. It mirrors
``engines.indextts.bootstrap`` because MOSS has the same shape of problem:
a hard ``transformers`` pin (``==5.0.0``) that conflicts with the parent's
``transformers>=5.3`` so MOSS runs in its own venv.
Probe order (priority existing power-user installs win, zero migration):
1. ``${OMNIVOICE_MOSS_TTS_V15_DIR}/.venv/`` the user's clone-level
venv. Highest priority: a user who already cloned MOSS-TTS and ran
``uv pip install -e ".[torch-runtime]"`` (per upstream docs) gets
reused verbatim, no re-download of the ~16 GB model.
2. ``backend/engines/moss_tts_v15/.venv/`` this package's own venv,
created by step 3 if needed.
3. Bootstrap: ``uv venv`` then ``uv pip install -e
"${DIR}[torch-runtime]"``. Requires ``OMNIVOICE_MOSS_TTS_V15_DIR``.
The upstream ``torch-runtime`` extra is CUDA (``+cu128``), so the
auto-bootstrap targets CUDA hosts; non-CUDA (CPU/Mac) users set up
their own venv per docs/engines/moss-tts-v15.md (Probe 1).
Caching: resolution is memoised after the first successful call. Tests
reset via :func:`invalidate`.
Security: bootstrap never touches HF_TOKEN; the sidecar's stderr is drained
by SubprocessBackend through the parent root logger where Phase 1's
``HFTokenRedactor`` strips token bytes. ``uv pip install -e`` installs from
a user-controlled clone the user already trusts (same posture as IndexTTS).
"""
from __future__ import annotations
import logging
import os
import shutil
import subprocess
import sys
from pathlib import Path
from typing import Optional
logger = logging.getLogger("omnivoice.moss_tts_v15.bootstrap")
#: Absolute path to the sidecar entrypoint. ``MossTTSV15Backend.sidecar_script``
#: returns this; SubprocessBackend spawns it with the resolved venv python.
MOSS_TTS_V15_SIDECAR_SCRIPT: Path = Path(__file__).parent / "main.py"
#: This package's owned venv (Probe 2). The MOSS-TTS clone, when bootstrapped,
#: is installed into this venv via ``uv pip install -e``.
_ENGINES_VENV_DIR: Path = Path(__file__).parent / ".venv"
#: Env var pointing at the user's MOSS-TTS clone root.
_CLONE_DIR_ENV: str = "OMNIVOICE_MOSS_TTS_V15_DIR"
#: Per-process resolution cache. Cleared by :func:`invalidate` for tests.
_resolved_python: Optional[Path] = None
# Timeouts — bounded so a wedged venv never hangs the parent. The bootstrap
# install can take many minutes on a cold cache (MOSS pulls a CUDA torch
# build + transformers + an audio codec stack).
_IMPORT_PROBE_TIMEOUT_S = 15
_UV_VENV_TIMEOUT_S = 120
_UV_PIP_INSTALL_TIMEOUT_S = 1800
# ── public API ────────────────────────────────────────────────────────────
def invalidate() -> None:
"""Clear the resolved-python cache. Tests call this between scenarios."""
global _resolved_python
_resolved_python = None
def is_moss_tts_v15_installed() -> bool:
"""Cheap file-existence check for a usable MOSS-TTS-v1.5 venv.
Returns True if either Probe 1 or Probe 2 has a Python executable on
disk. Does NOT spawn the venv Python that's saved for
:func:`resolve_moss_tts_v15_venv`, which is only invoked on the first
generate() / health_check(). This fires on every Settings render via
``MossTTSV15Backend.is_available()``, so it stays cheap.
"""
for cand in _probe_paths():
if cand.is_file():
return True
return False
def resolve_moss_tts_v15_venv() -> Path:
"""Resolve the path to the Python interpreter that runs the sidecar.
Probe order described in the module docstring. Memoised. Raises
:exc:`RuntimeError` if no working venv can be located AND the bootstrap
path is unavailable.
"""
global _resolved_python
if _resolved_python is not None:
return _resolved_python
clone_dir = os.environ.get(_CLONE_DIR_ENV)
# Probe 1 — user's clone-level venv (highest priority for back-compat).
if clone_dir:
cand = _venv_python_path(Path(clone_dir) / ".venv")
if cand.is_file() and _venv_can_import_moss(cand):
logger.info(
"MOSS-TTS-v1.5 venv resolved from %s: %s", _CLONE_DIR_ENV, cand,
)
_resolved_python = cand
return cand
# Probe 2 — this package's own venv.
cand = _venv_python_path(_ENGINES_VENV_DIR)
if cand.is_file() and _venv_can_import_moss(cand):
logger.info("MOSS-TTS-v1.5 venv resolved from engines path: %s", cand)
_resolved_python = cand
return cand
# Probe 3 — bootstrap.
if not clone_dir:
raise RuntimeError(
"MOSS-TTS-v1.5 is not installed. Set the "
f"{_CLONE_DIR_ENV} environment variable to your MOSS-TTS clone "
"(the directory that contains pyproject.toml), then restart "
"OmniVoice. See docs/engines/moss-tts-v15.md for the full "
"install walk-through."
)
cand = _bootstrap_engines_venv(Path(clone_dir))
_resolved_python = cand
return cand
# ── internals ─────────────────────────────────────────────────────────────
def _venv_python_path(venv_dir: Path) -> Path:
"""Return the python executable path inside a venv directory.
Handles the Unix (``bin/python``) vs Windows (``Scripts/python.exe``)
layout. No filesystem access caller checks .is_file().
"""
if sys.platform == "win32":
return venv_dir / "Scripts" / "python.exe"
return venv_dir / "bin" / "python"
def _probe_paths() -> list[Path]:
"""Ordered list of candidate venv-python paths (no .is_file() check)."""
out: list[Path] = []
clone_dir = os.environ.get(_CLONE_DIR_ENV)
if clone_dir:
out.append(_venv_python_path(Path(clone_dir) / ".venv"))
out.append(_venv_python_path(_ENGINES_VENV_DIR))
return out
def _venv_can_import_moss(python_path: Path) -> bool:
"""Spawn the candidate python and verify the MOSS stack imports.
MOSS-TTS-v1.5 loads via ``transformers`` + ``trust_remote_code`` (no
fixed top-level package to import), so the readiness signal is that the
venv has a working ``transformers`` + ``torch`` which only the
``[torch-runtime]`` install provides. Bounded by
``_IMPORT_PROBE_TIMEOUT_S`` so a wedged venv never hangs the parent.
Returns False on any failure (non-zero exit, timeout, OSError).
"""
try:
proc = subprocess.run(
[str(python_path), "-c", "import transformers, torch"],
capture_output=True,
timeout=_IMPORT_PROBE_TIMEOUT_S,
)
except (subprocess.TimeoutExpired, OSError) as exc:
logger.debug("moss-tts-v15 import probe failed for %s: %s", python_path, exc)
return False
if proc.returncode != 0:
logger.debug(
"moss-tts-v15 import probe non-zero for %s: %s",
python_path,
proc.stderr.decode("utf-8", errors="replace")[:200],
)
return False
return True
def _locate_uv() -> Optional[str]:
"""Find the uv binary — bundled first (Tauri-set env var), else PATH."""
bundled = os.environ.get("OMNIVOICE_BUNDLED_UV")
if bundled and Path(bundled).is_file():
return bundled
sys_uv = shutil.which("uv")
if sys_uv:
return sys_uv
return None
def _bootstrap_engines_venv(clone_dir: Path) -> Path:
"""Create engines/moss_tts_v15/.venv and install the user's clone into it.
Runs ``uv venv <engines_venv>`` then ``uv pip install --python
<engines_venv>/bin/python -e "<clone>[torch-runtime]"``. Verifies the
result by re-probing the import a successful uv invocation that still
can't import the stack indicates a deeper environment problem (e.g. the
``+cu128`` torch-runtime extra can't resolve on a non-CUDA host) and we
raise with whatever stderr we captured plus a docs pointer.
"""
uv = _locate_uv()
if not uv:
raise RuntimeError(
"uv is required to bootstrap the MOSS-TTS-v1.5 venv but was not "
"found on PATH (and OMNIVOICE_BUNDLED_UV was not set). Install uv "
"from https://docs.astral.sh/uv/ and re-launch OmniVoice, or set "
"OMNIVOICE_BUNDLED_UV to the absolute path of a uv binary."
)
logger.info(
"Bootstrapping MOSS-TTS-v1.5 venv at %s from %s (this can take "
"several minutes on first launch)", _ENGINES_VENV_DIR, clone_dir,
)
try:
subprocess.run(
[uv, "venv", str(_ENGINES_VENV_DIR)],
check=True,
timeout=_UV_VENV_TIMEOUT_S,
capture_output=True,
)
except subprocess.CalledProcessError as exc:
raise RuntimeError(
f"uv venv failed for MOSS-TTS-v1.5 bootstrap at {_ENGINES_VENV_DIR}: "
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
) from exc
python_path = _venv_python_path(_ENGINES_VENV_DIR)
try:
subprocess.run(
[
uv, "pip", "install",
"--python", str(python_path),
"-e", f"{clone_dir}[torch-runtime]",
],
check=True,
timeout=_UV_PIP_INSTALL_TIMEOUT_S,
capture_output=True,
)
except subprocess.CalledProcessError as exc:
raise RuntimeError(
"uv pip install -e failed during MOSS-TTS-v1.5 bootstrap "
f"({clone_dir}). On a non-CUDA host the upstream '[torch-runtime]' "
"extra (cu128) cannot resolve — set up the venv manually per "
"docs/engines/moss-tts-v15.md. Error: "
f"{exc.stderr.decode('utf-8', errors='replace') if exc.stderr else exc}"
) from exc
if not _venv_can_import_moss(python_path):
raise RuntimeError(
"MOSS-TTS-v1.5 bootstrap completed but the transformers/torch "
f"import still fails from {python_path}. Verify that {clone_dir} "
"is a valid MOSS-TTS clone. See docs/engines/moss-tts-v15.md."
)
logger.info("MOSS-TTS-v1.5 venv bootstrap successful: %s", python_path)
return python_path
__all__ = [
"MOSS_TTS_V15_SIDECAR_SCRIPT",
"invalidate",
"is_moss_tts_v15_installed",
"resolve_moss_tts_v15_venv",
]
-303
View File
@@ -1,303 +0,0 @@
"""MOSS-TTS-v1.5 sidecar entry point (issue #498).
Runs inside ``engines/moss_tts_v15/.venv`` (or the user's existing
``${OMNIVOICE_MOSS_TTS_V15_DIR}/.venv``) with ``transformers==5.0.0``,
isolated from the OmniVoice parent process which pins ``transformers>=5.3``.
Same isolation rationale as the IndexTTS sidecar.
Stdlib-only at import time. The model + transformers + torch are imported
lazily on the first synthesize op so the sidecar emits its ``ready`` frame
inside the parent's 30 s spawn handshake even on a cold filesystem (an 8B
model takes well over 30 s to cold-load).
Wire protocol length-prefixed JSON over stdin/stdout, byte-identical to
``backend/services/subprocess_backend.py``::
[ 4-byte big-endian uint32 length ][ N bytes UTF-8 JSON ]
Op flow:
1. Sidecar -> parent: {"op": "ready", "engine": "moss-tts-v15",
"sample_rate": 24000}
2. parent -> sidecar: {"op": "ping"} -> {"op": "pong", "vram_mb": N}
3. parent -> sidecar: {"op": "synthesize", "text": "...",
"ref_audio": "/path/spk.wav", "language": "fr",
"tokens": 325, "max_new_tokens": 4096}
-> {"op": "progress", ...} (cold load only) then
-> {"op": "audio", "audio_pcm_b64": "...", "sample_rate": 24000,
"n_samples": N}
4. parent -> sidecar: {"op": "shutdown"} -> exit 0
Restrictions: NO imports from OmniVoice parent code (different venv). NO
logging of ``os.environ`` contents. Single-frame DoS cap matches the
parent's ``MAX_FRAME_BYTES``.
"""
from __future__ import annotations
import base64
import json
import os
import struct
import sys
import traceback
# Mirrors backend/services/subprocess_backend.py::MAX_FRAME_BYTES.
MAX_FRAME_BYTES = 64 * 1024 * 1024
#: Native sample rate MOSS-TTS-v1.5 emits. Advertised in the ready frame so
#: the parent doesn't have to import MOSS just to learn the rate. Confirmed
#: via ``processor.model_config.sampling_rate`` (the real value is read from
#: the loaded model at synthesize time; this is the handshake default).
MOSS_SAMPLE_RATE = 24000
#: HF repo id for the weights, overridable for air-gapped / mirror installs.
_DEFAULT_REPO = "OpenMOSS-Team/MOSS-TTS-v1.5"
#: ISO-639-1 → MOSS language name. MOSS's ``build_user_message`` takes a
#: language *name* ("French"), not a code. Unknown codes are omitted so the
#: model auto-detects. Covers the high-traffic subset of MOSS's 31 langs.
_ISO_TO_NAME = {
"en": "English", "zh": "Chinese", "ja": "Japanese", "ko": "Korean",
"fr": "French", "de": "German", "es": "Spanish", "it": "Italian",
"pt": "Portuguese", "ru": "Russian", "ar": "Arabic", "hi": "Hindi",
"nl": "Dutch", "pl": "Polish", "tr": "Turkish", "vi": "Vietnamese",
"th": "Thai", "id": "Indonesian", "cs": "Czech", "el": "Greek",
"he": "Hebrew", "fa": "Persian", "uk": "Ukrainian", "sv": "Swedish",
}
# ── wire protocol ─────────────────────────────────────────────────────────
def _send(stream, obj: dict) -> None:
body = json.dumps(obj, separators=(",", ":")).encode("utf-8")
stream.write(struct.pack("!I", len(body)))
stream.write(body)
stream.flush()
def _recv(stream):
header = stream.read(4)
if len(header) < 4:
return None # EOF
(n,) = struct.unpack("!I", header)
if n > MAX_FRAME_BYTES:
raise IOError(f"frame too large: {n}")
body = bytearray()
while len(body) < n:
chunk = stream.read(n - len(body))
if not chunk:
raise IOError("short read")
body.extend(chunk)
return json.loads(bytes(body).decode("utf-8"))
def _measure_vram_mb() -> float:
"""This sidecar's own GPU memory in MB (MM2-08). The parent can't see a
child's VRAM, so we self-report it in the pong. 0 on CPU. Never raises."""
try:
import torch
if torch.cuda.is_available():
return round(torch.cuda.memory_allocated() / (1024 ** 2), 1)
except Exception:
pass
return 0.0
# ── model loading (lazy, on first synthesize) ─────────────────────────────
# Module-level singleton — populated on the first synthesize op and reused.
# Holds (processor, model, device, sample_rate).
_state = None
def _load_model(stdout):
"""Cold-construct the MOSS-TTS-v1.5 processor + model.
Device selection is CUDA-or-CPU only MOSS's upstream documents no MPS
path and the custom ``trust_remote_code`` modelling code is untested on
Apple Silicon, so we never route to MPS where it might crash. dtype is
bf16 on CUDA, fp32 on CPU (bf16 CPU ops are spotty). Emits progress
frames so the parent can surface the multi-GB cold-load latency.
"""
global _state
if _state is not None:
return _state
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 0})
import torch
from transformers import AutoModel, AutoProcessor
repo = os.environ.get("OMNIVOICE_MOSS_TTS_V15_MODEL", _DEFAULT_REPO)
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
# "sdpa" works on CUDA + CPU and needs no extra dep. flash_attention_2
# (Ampere+ CUDA, optional flash-attn) is opt-in via env.
attn = os.environ.get("OMNIVOICE_MOSS_TTS_V15_ATTN", "sdpa")
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
# The audio tokenizer is a separate sub-module that must be moved to the
# device independently (easy to miss — see upstream README).
processor.audio_tokenizer = processor.audio_tokenizer.to(device)
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 50})
model = AutoModel.from_pretrained(
repo,
trust_remote_code=True,
attn_implementation=attn,
torch_dtype=dtype,
).to(device)
model.eval()
sample_rate = int(getattr(processor.model_config, "sampling_rate", MOSS_SAMPLE_RATE))
_state = (processor, model, device, sample_rate)
_send(stdout, {"op": "progress", "stage": "loading_model", "percent": 100})
return _state
def _tensor_to_pcm_b64(audio, sample_rate: int) -> tuple[str, int, int]:
"""Convert a torch waveform tensor to base64 int16 PCM.
MOSS returns a float tensor in [-1, 1] (1-D or (1, N)); we squeeze to
mono, clip, scale to int16, and base64 so the wire frame stays JSON-safe.
"""
import numpy as np
arr = audio.detach().to("cpu").float().numpy()
arr = np.asarray(arr, dtype=np.float32).squeeze()
if arr.ndim > 1:
arr = arr.mean(axis=0) # defensive downmix to mono
arr = np.clip(arr, -1.0, 1.0)
pcm = (arr * 32767.0).astype(np.int16).tobytes()
return base64.b64encode(pcm).decode("ascii"), int(sample_rate), int(arr.shape[0])
def _resolve_language(raw):
"""Map OmniVoice's language value to a MOSS language name, or None.
Accepts an ISO-639-1 code or a full name. Unknown / empty / "auto"
values return None so MOSS auto-detects."""
if not raw or not isinstance(raw, str):
return None
s = raw.strip()
if not s or s.lower() == "auto":
return None
if s.lower() in _ISO_TO_NAME:
return _ISO_TO_NAME[s.lower()]
# Already a language name (or an unknown code) — pass it through; MOSS
# ignores a language it doesn't recognise.
return s
def _handle_synthesize(msg: dict, stdout) -> None:
"""Dispatch one synthesize request. Emits the audio frame or raises."""
import torch
text = msg.get("text")
if not text or not isinstance(text, str):
raise ValueError("synthesize: missing or non-string 'text'")
processor, model, device, sample_rate = _load_model(stdout)
user_kwargs: dict = {"text": text}
ref_audio = msg.get("ref_audio")
if ref_audio:
# Zero-shot voice cloning: the reference audio alone is enough in
# MOSS's clone mode (ref_text is not consumed here). The processor's
# audio tokenizer encodes the reference into the prompt.
user_kwargs["reference"] = [ref_audio]
language = _resolve_language(msg.get("language"))
if language:
user_kwargs["language"] = language
tokens = msg.get("tokens")
if tokens is not None:
user_kwargs["tokens"] = int(tokens)
max_new_tokens = int(msg.get("max_new_tokens", 4096))
conversations = [[processor.build_user_message(**user_kwargs)]]
with torch.no_grad():
batch = processor(conversations, mode="generation")
outputs = model.generate(
input_ids=batch["input_ids"].to(device),
attention_mask=batch["attention_mask"].to(device),
max_new_tokens=max_new_tokens,
)
decoded = processor.decode(outputs)
audio = decoded[0].audio_codes_list[0]
pcm_b64, sr, n_samples = _tensor_to_pcm_b64(audio, sample_rate)
_send(stdout, {
"op": "audio",
"audio_pcm_b64": pcm_b64,
"sample_rate": sr,
"n_samples": n_samples,
})
# ── main loop ─────────────────────────────────────────────────────────────
def main() -> int:
stdin = sys.stdin.buffer
stdout = sys.stdout.buffer
# Ready handshake fires BEFORE any heavy import — nothing above this line
# touches transformers/torch, so we make the 30 s spawn window even cold.
_send(stdout, {
"op": "ready",
"engine": "moss-tts-v15",
"sample_rate": MOSS_SAMPLE_RATE,
})
while True:
try:
msg = _recv(stdin)
except Exception as exc:
_send(stdout, {
"op": "error",
"stage": "recv",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
return 1
if msg is None:
return 0
op = msg.get("op") if isinstance(msg, dict) else None
try:
if op == "ping":
_send(stdout, {"op": "pong", "vram_mb": _measure_vram_mb()})
elif op == "synthesize":
_handle_synthesize(msg, stdout)
elif op == "shutdown":
return 0
else:
_send(stdout, {
"op": "error",
"stage": "dispatch",
"message": f"unknown op: {op!r}",
})
except Exception as exc:
# Per-op failure is recoverable — emit the error frame and stay
# alive so the parent can retry without paying the respawn +
# multi-GB model-load cost again.
_send(stdout, {
"op": "error",
"stage": op or "unknown",
"message": f"{type(exc).__name__}: {exc}",
"traceback": traceback.format_exc(),
})
if __name__ == "__main__":
sys.exit(main())
+6 -67
View File
@@ -353,21 +353,6 @@ def _make_backend_class():
f"This clears the quarantine on the .app and its "
f"bundled binaries. See docs/install/macos.md."
)
# Execute bit (issue #437). A `git clone` / zip extract on POSIX
# can drop +x, which only surfaces at spawn time as
# "[Errno 13] Permission denied" — and the generic synth handler
# then mislabels it as out-of-memory. Self-heal here, AFTER the
# SHA check has confirmed this is the right file (so we never
# chmod a foreign binary). No-op on Windows.
if os.name == "posix" and not os.access(bin_path, os.X_OK):
try:
bin_path.chmod(bin_path.stat().st_mode | 0o111)
except OSError:
return False, (
f"GGUF binary {bin_path.name} isn't executable and "
f"couldn't be made so — run `chmod +x {bin_path}` "
f"and retry."
)
return True, "ready"
except Exception as exc:
return False, f"{type(exc).__name__}: {exc}"
@@ -508,12 +493,6 @@ def _make_backend_class():
* ``ref_audio`` (str/Path) speaker reference WAV for cloning.
* ``ref_text`` (str) transcript of ``ref_audio``.
* ``language`` (str) ISO code or omnivoice-tts lang label.
* ``instruct`` (str) style instruction.
* ``duration`` (float) target duration in seconds.
* ``seed`` (int) deterministic sampling seed.
* ``denoise`` (bool) omit denoise token when false.
* ``preprocess_prompt`` (bool) skip prompt preprocessing when false.
* ``chunk_duration`` / ``chunk_threshold`` (float) binary long-form controls.
"""
import soundfile as sf # local import keeps module import cheap
import torch
@@ -524,32 +503,15 @@ def _make_backend_class():
fd, out_str = tempfile.mkstemp(prefix="omnivoice-gguf-", suffix=".wav")
os.close(fd)
out_path = Path(out_str)
ref_text_path: Optional[Path] = None
try:
ref_text = kw.get("ref_text")
if kw.get("ref_audio") and ref_text:
text_fd, text_str = tempfile.mkstemp(
prefix="omnivoice-gguf-ref-", suffix=".txt"
)
os.close(text_fd)
ref_text_path = Path(text_str)
ref_text_path.write_text(str(ref_text), encoding="utf-8")
argv = self._build_argv(
base=base_path,
tokenizer=tok_path,
out_path=out_path,
ref_audio=kw.get("ref_audio"),
ref_text=str(ref_text_path) if ref_text_path else None,
ref_text=kw.get("ref_text"),
language=kw.get("language"),
instruct=kw.get("instruct"),
duration=kw.get("duration"),
seed=kw.get("seed"),
denoise=kw.get("denoise", True),
preprocess_prompt=kw.get("preprocess_prompt", True),
chunk_duration=kw.get("chunk_duration"),
chunk_threshold=kw.get("chunk_threshold"),
)
self._run_subprocess(argv, stdin_text=text)
wav, sr = sf.read(str(out_path))
@@ -558,11 +520,6 @@ def _make_backend_class():
out_path.unlink()
except OSError:
pass
if ref_text_path is not None:
try:
ref_text_path.unlink()
except OSError:
pass
# soundfile returns (n,) for mono or (n, c) for multichannel.
# OmniVoice/Higgs Audio v2 is mono → (n,). Wrap to (1, n).
@@ -587,13 +544,6 @@ def _make_backend_class():
ref_audio: Optional[str],
ref_text: Optional[str],
language: Optional[str],
instruct: Optional[str] = None,
duration: Optional[float] = None,
seed: Optional[int] = None,
denoise: bool = True,
preprocess_prompt: bool = True,
chunk_duration: Optional[float] = None,
chunk_threshold: Optional[float] = None,
) -> list[str]:
"""Compose argv from typed Path objects only (T-04-02)."""
argv: list[str] = [
@@ -605,20 +555,6 @@ def _make_backend_class():
lang = _iso_to_omnivoice_lang(language)
if lang:
argv += ["--lang", lang]
if instruct:
argv += ["--instruct", str(instruct)]
if duration is not None:
argv += ["--duration", str(float(duration))]
if seed is not None:
argv += ["--seed", str(int(seed))]
if denoise is False:
argv += ["--no-denoise"]
if preprocess_prompt is False:
argv += ["--no-preprocess-prompt"]
if chunk_duration is not None:
argv += ["--chunk-duration", str(float(chunk_duration))]
if chunk_threshold is not None:
argv += ["--chunk-threshold", str(float(chunk_threshold))]
if ref_audio:
# Two-stage validation (defense in depth):
# (a) Reject anything outside the project's voices /
@@ -652,8 +588,11 @@ def _make_backend_class():
)
argv += ["--ref-wav", str(ref_path)]
if ref_text:
# The C++ runtime expects a transcript file path.
# generate() creates this file in the system temp dir.
# ref_text is free-form text; pass via stdin would
# collide with the synthesis prompt, so the only safe
# channel is argv. The binary treats this as a quoted
# string at the OS layer (Popen escapes argv per
# platform); we don't pre-escape.
argv += ["--ref-text", str(ref_text)]
return argv
-1
View File
@@ -70,7 +70,6 @@ class Supertonic3Backend(SubprocessBackend):
id = "supertonic3"
display_name = "Supertonic-3 (31 langs, CPU ONNX, 7 preset voices, OpenRAIL-M)"
supports_voice_design = False # preset voices only
supports_cloning = False # preset voices only; generate() never reads ref_audio
# TTS-04: honest hardware reporting. Supertonic-3 has no CUDA / MPS
# path in the SDK ‑‑ ONNX Runtime CPU EP only.
gpu_compat: tuple[str, ...] = ("cpu",)
+17 -434
View File
@@ -9,16 +9,6 @@ _backend_dir = os.path.dirname(os.path.abspath(__file__))
if _backend_dir not in sys.path:
sys.path.insert(0, _backend_dir)
# #564: also make the project's OWN `omnivoice` package importable from source
# when the venv's editable install is missing/broken (interrupted/offline
# `uv sync`, antivirus-quarantined `_editable_impl_omnivoice.pth`, …). Without
# this the backend boots fine and only fails at the first model call with
# `No module named 'omnivoice'`. The bootstrap now gates on omnivoice being
# importable too (re-syncing to re-lay the editable install); this is the
# runtime safety net. See core/omnivoice_path.py for the full rationale.
from core.omnivoice_path import ensure_omnivoice_importable
ensure_omnivoice_importable(_backend_dir)
# Triton is unavailable on Windows — disable torch.compile / dynamo / inductor
# to prevent TritonMissing errors at inference time. Must be set before torch
# is imported (it is lazily imported in services/model_manager.py). Uses
@@ -53,13 +43,11 @@ try:
_project_env = os.path.join(os.path.dirname(_backend_dir), ".env")
if os.path.isfile(_project_env):
dotenv.load_dotenv(_project_env, override=False)
# Load the durable per-user config (the in-app Settings source of truth) so
# env vars set once survive Tauri/Finder launches that don't inherit a shell
# environment. This OVERRIDES launcher-injected defaults: the desktop app
# injects a stale OMNIVOICE_CACHE_DIR from its own config before startup, so
# without override a models dir changed in Settings was ignored forever (#480).
from core.user_env import load_into_environ as _load_user_env
_load_user_env()
# Also load the durable per-user config so env vars set once survive
# Tauri/Finder launches that don't inherit a shell environment.
_user_env = os.path.expanduser("~/.config/omnivoice/env")
if os.path.isfile(_user_env):
dotenv.load_dotenv(_user_env, override=False)
except ImportError:
pass
@@ -139,24 +127,6 @@ os.environ.setdefault("HF_HUB_DISABLE_XET", "1")
os.environ.setdefault("HF_HUB_ETAG_TIMEOUT", "15")
os.environ.setdefault("HF_HUB_DOWNLOAD_TIMEOUT", "30")
# ── OS trust store for TLS (#976) ───────────────────────────────────────────
# Users behind a corporate/antivirus proxy that TLS-inspects HTTPS traffic get
# a raw "[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] ssl/tls alert handshake failure"
# on every model install — the TCP connection succeeds (a different failure
# mode from #984's TCP-level blocked-host case), but the proxy re-signs the
# certificate with its own root CA, which the OS trusts (Windows CryptoAPI/
# SChannel) and Python's bundled `certifi` CA list does not. `inject_into_ssl`
# patches `ssl.SSLContext` process-wide to verify against the OS trust store
# instead, which is the actual fix (not just a nicer error message). Must run
# here — at MODULE level, before huggingface_hub/requests/httpx do any network
# I/O — not inside lifespan(), which runs too late. Not platform-gated: it's a
# correctness improvement everywhere. Best-effort: never block startup.
try:
import truststore
truststore.inject_into_ssl()
except Exception:
pass
# Prevent torchaudio from lazy-importing torchcodec (broken on some installs).
# Proper fix = exclude torchcodec in pyproject.toml; this is a belt-and-braces guard.
@@ -173,16 +143,6 @@ from logging.handlers import RotatingFileHandler
# written to prefs.json so they survive backend restarts. Read them back
# here — before any user code reads os.environ — so the values are available
# from startup.
#
# Legacy (≤v0.3.7) Translation-LLM rows (env.TRANSLATE_*) must migrate into
# the custom LLM provider's settings store BEFORE the re-import below — once
# TRANSLATE_BASE_URL lands in os.environ it hijacks the LLM provider
# selection for the whole session (#963). Real env vars are untouched.
try:
from services.llm_providers import migrate_legacy_translate_prefs
migrate_legacy_translate_prefs()
except Exception:
pass # never block startup on the migration; it retries next launch
_PERSISTED_ENV_PREFIX = "env."
try:
from core.prefs import _load as _load_all_prefs
@@ -195,18 +155,6 @@ try:
except Exception:
pass # prefs.json missing or broken — fine on first run
# ── Activate the yt-dlp user-update overlay (Settings → Audio tools) ──────
# Must run before anything imports yt_dlp so a user-updated version (stored
# under DATA_DIR, surviving app updates and uv drift syncs) wins over the
# locked wheel. Best-effort: a broken overlay must never block startup.
try:
from services.media_tools import activate_ytdlp_overlay
activate_ytdlp_overlay()
except Exception:
# Best-effort by design: a broken/corrupt overlay must never block
# startup — the locked wheel on sys.path is the fallback.
pass
warnings.filterwarnings("ignore", category=UserWarning)
torchaudio.set_audio_backend("soundfile")
@@ -336,12 +284,7 @@ from fastapi.responses import JSONResponse, RedirectResponse, Response
from fastapi.staticfiles import StaticFiles
from fastapi.middleware.cors import CORSMiddleware
from starlette.datastructures import MutableHeaders
# Docs-only dependency: a venv created before scalar-fastapi entered the
# dependency set must still boot the backend (#307) — /docs degrades instead.
try:
from scalar_fastapi import get_scalar_api_reference
except ImportError:
get_scalar_api_reference = None
from scalar_fastapi import get_scalar_api_reference
import traceback
_crash_log_lock = threading.Lock()
@@ -370,24 +313,17 @@ from api.routers import (
setup,
gallery,
archetypes,
describe_voice,
community,
batch,
watermark,
events,
capture,
capture_ws,
dictation,
openai_compat,
tts_stream,
marketplace,
personas,
sonitranslate,
audiobook,
longform_jobs,
pronunciation, # Expressive-TTS Spec 01: user pronunciation dictionary
settings as settings_router, # Phase 1 AUTH-03: HF token save/clear/state
media_tools as media_tools_router, # Audio tools: ffmpeg/ffprobe/yt-dlp management
)
from utils import hf_progress
@@ -396,30 +332,6 @@ from utils import hf_progress
# the patched class, not the original.
hf_progress.install()
# Wire the overall download aggregator's byte sink onto the patched tqdm so
# parallel per-file updates feed one accurate overall bar (FDL-06).
try:
from utils import download_aggregator
download_aggregator.install()
except Exception:
pass
# Log the download-acceleration state once at startup (FDL-03) so a slow
# download report can be triaged from the logs without reproducing. Note: the
# app sets HF_HUB_DISABLE_XET=1 above by default (legacy LFS for byte progress),
# so xet_active is normally False even though hf_xet is installed.
try:
from api.routers.system import _fast_download_status as _fd_status
_fd = _fd_status()
_xet_ver = f" {_fd['xet_version']}" if _fd.get("xet_version") else ""
logging.getLogger("omnivoice.model").info(
"downloads: Xet %s (hf_xet%s installed=%s), high_perf=%s",
"ACTIVE" if _fd["xet_active"] else "disabled → legacy LFS",
_xet_ver, _fd["xet_installed"], _fd["high_performance"],
)
except Exception:
pass
def _env_flag(name: str, default: bool = False) -> bool:
value = os.environ.get(name)
@@ -428,145 +340,8 @@ def _env_flag(name: str, default: bool = False) -> bool:
return value.strip().lower() in {"1", "true", "yes", "on"}
def _capture_preload_delay_s() -> float:
"""Seconds after boot before the dictation (capture ASR) model warms.
Late enough that it never competes with startup I/O or the TTS preload;
overridable via OMNIVOICE_CAPTURE_PRELOAD_DELAY (mostly for tests)."""
raw = os.environ.get("OMNIVOICE_CAPTURE_PRELOAD_DELAY", "")
try:
v = float(raw)
if v >= 0:
return v
except (TypeError, ValueError):
pass
return 30.0
def _capture_preload_ram_ok(min_free_bytes: int = 4 * 1024**3) -> bool:
"""RAM guard for the dictation warm-up: skip below 4 GB free so the
background load never pushes a small machine into swap. If free memory
can't be measured, warm anyway (the load path has its own error handling)."""
try:
import psutil
return psutil.virtual_memory().available >= min_free_bytes
except Exception:
return True
def _mcp_start_timeout_s() -> float:
"""Seconds to wait for the MCP session manager to start before giving up
and serving without it (#632). Overridable via OMNIVOICE_MCP_START_TIMEOUT_S."""
raw = os.environ.get("OMNIVOICE_MCP_START_TIMEOUT_S", "")
try:
v = float(raw)
if v > 0:
return v
except (TypeError, ValueError):
pass
return 30.0
async def _serve_mcp(session_manager, ready: "asyncio.Event", stop: "asyncio.Event") -> None:
"""Own the MCP session manager's full enter→exit lifecycle in ONE task.
FastMCP's ``run()`` opens an anyio task group, and anyio requires the cancel
scope to be exited in the *same task* that entered it. So we must NOT enter
it via ``wait_for`` (which runs the enter in a throwaway sub-task) or on the
lifespan task and exit it elsewhere either raises "Attempted to exit cancel
scope in a different task". This coroutine enters and exits the context
itself: it signals ``ready`` once mounted, then idles until ``stop``.
"""
try:
async with session_manager.run():
ready.set()
await stop.wait()
except Exception as e:
logger.warning("MCP session manager stopped: %s", e)
finally:
ready.set() # never leave startup blocked on the readiness wait
async def _start_mcp_session_manager(session_manager, *, timeout: float):
"""Start MCP off the startup critical path; wait up to ``timeout`` for it to
signal ready. Returns ``(task, stop_event, mounted)``.
The MCP layer is best-effort and must never wedge backend startup. On some
platforms (observed: Apple-Silicon M1, #632) ``run()`` can *hang* on its
anyio task group; the old code awaited the enter before serving, so the hang
meant "Application startup complete" never fired and the whole backend was
unreachable with no error. Now the enter lives in its own task and we only
*optionally* wait on a ready signal a hang becomes a logged warning + a
backend that serves normally without MCP.
"""
stop = asyncio.Event()
if session_manager is None:
return None, stop, False
ready = asyncio.Event()
task = asyncio.create_task(_serve_mcp(session_manager, ready, stop))
try:
await asyncio.wait_for(ready.wait(), timeout=timeout)
mounted = not task.done() # ready is also set on failure → not mounted
except asyncio.TimeoutError:
logger.warning(
"MCP session manager did not signal ready within %.0fs (#632); "
"serving without waiting. Set OMNIVOICE_MCP_START_TIMEOUT_S to adjust.",
timeout,
)
mounted = False
return task, stop, mounted
async def _cancel_and_await_tasks(*tasks, timeout: float = 3.0) -> None:
"""Cancel each background task and give it a bounded chance to actually
finish before shutdown proceeds ``None`` entries are skipped (a task
that's conditionally created, e.g. ``capture_preload_task``, may not
exist).
``task.cancel()`` alone is not enough for a task awaiting
``run_in_executor()``: once the underlying OS thread is inside blocking
native/import work, cancellation can't stop it, so cancel-and-move-on lets
shutdown finish while that thread is still running invisible to
asyncio, but very much alive when the interpreter starts tearing down
module state under it (#1000 class). Awaiting with a bound (instead of
just cancelling) gives an early-stage task a real chance to exit cleanly
first; a task that's genuinely still deep in blocking work times out here
same as before, and the caller's own GPU-pool reset handles that case.
"""
for t in tasks:
if t is None:
continue
t.cancel()
for t in tasks:
if t is None:
continue
try:
await asyncio.wait_for(t, timeout=timeout)
except (asyncio.CancelledError, asyncio.TimeoutError):
pass
@asynccontextmanager
async def lifespan(app: FastAPI):
# Startup watchdog (#632): a silent hang during startup (e.g. a model-load /
# MCP deadlock on some platforms) means "Application startup complete" never
# logs and the app sits forever with no error. If startup hasn't finished
# within the window, dump every thread's stack to stderr (→ backend_err.log)
# so the hang point is captured instead of invisible. Cancelled the instant
# startup completes, so a normal (even slow-download) boot never trips it.
# Tune with OMNIVOICE_STARTUP_WATCHDOG_S (seconds; 0 disables). Best-effort —
# never let the diagnostic itself break startup.
_watchdog_armed = False
try:
import faulthandler
_wd = float(os.environ.get("OMNIVOICE_STARTUP_WATCHDOG_S", "300"))
if _wd > 0 and hasattr(faulthandler, "dump_traceback_later"):
faulthandler.dump_traceback_later(_wd, repeat=False, exit=False)
_watchdog_armed = True
logger.info("Startup watchdog armed: thread dump if startup exceeds %.0fs (#632).", _wd)
except Exception:
pass
init_db()
# Network sharing is loopback-only by default; the PIN middleware stays
# inert until enable() sets a PIN. Seed the (disabled) state so the
@@ -615,20 +390,11 @@ async def lifespan(app: FastAPI):
worker_task = asyncio.create_task(task_manager.worker())
# Warm the TTS model in the background so first /generate is instant.
preload_task = asyncio.create_task(preload_model())
# Dictation v2: the capture ASR warms in the background BY DEFAULT — a
# deferred (~30s post-boot) load off the event loop, so startup stays
# lean and the first dictation is instant instead of a cold model load.
# OMNIVOICE_PRELOAD_CAPTURE_ASR=0 opts out; the warm-up is also skipped
# under 4 GB free RAM (checked at warm time, not boot time).
capture_preload_task = None # only assigned when the preload actually runs (#1000 class)
if _env_flag("OMNIVOICE_PRELOAD_CAPTURE_ASR", default=True):
# Capture ASR is useful to keep warm, but it is another large model in
# unified memory on Apple Silicon. Keep launch lean by default; users who
# prefer instant dictation can opt in with OMNIVOICE_PRELOAD_CAPTURE_ASR=1.
if _env_flag("OMNIVOICE_PRELOAD_CAPTURE_ASR"):
async def _preload_capture_asr():
await asyncio.sleep(_capture_preload_delay_s())
if not _capture_preload_ram_ok():
logger.info(
"Capture ASR preload skipped: <4GB free RAM; "
"dictation ASR will load on first use.")
return
loading_detail = None
prev_loading_detail = None
try:
@@ -656,66 +422,17 @@ async def lifespan(app: FastAPI):
capture_preload_task = asyncio.create_task(_preload_capture_asr())
else:
logger.info("Capture ASR preload disabled; dictation ASR will load on first use.")
# ── MCP session manager (Wave 2.2) ────────────────────────────────────
# FastMCP's Streamable-HTTP transport needs its session manager running for
# the lifetime of the app. Run it in its OWN task that owns the full
# enter→exit lifecycle (anyio task-affinity, see _serve_mcp) and only wait,
# with a timeout, for it to signal ready — so a hang on its anyio group
# (observed on M1, #632) can never wedge "Application startup complete".
_sm = getattr(app.state, "mcp_session_manager", None)
mcp_task, mcp_stop, mcp_mounted = await _start_mcp_session_manager(
_sm, timeout=_mcp_start_timeout_s()
)
if mcp_mounted:
logger.info("MCP server mounted at /mcp")
# Startup finished — disarm the hang watchdog before serving (#632).
if _watchdog_armed:
try:
import faulthandler
faulthandler.cancel_dump_traceback_later()
except Exception:
pass
yield
# ── Graceful shutdown (SIGTERM from Tauri, Ctrl+C, etc.) ────────────
logger.info("Shutdown: cleaning up…")
# Stop MCP first — signal its task to exit its own anyio context (correct
# task-affinity), then bound the wait so a wedged manager can't hang exit.
mcp_stop.set()
if mcp_task is not None:
idle_task.cancel()
worker_task.cancel()
# Wait for tasks to finish their current iteration
for t in (idle_task, worker_task):
try:
await asyncio.wait_for(mcp_task, timeout=5.0)
except (asyncio.TimeoutError, asyncio.CancelledError):
await asyncio.wait_for(t, timeout=3.0)
except (asyncio.CancelledError, asyncio.TimeoutError):
pass
except Exception:
pass
# preload_task/capture_preload_task matter most here (#1000 class): a quit
# mid-preload used to fall straight through to "Shutdown: done." while the
# model load was still running on a GPU-pool thread — cancel() can't stop
# a thread already inside blocking import/load work, so the process
# reported a clean exit while that background thread was still mid-
# `import transformers`, and got torn down by interpreter finalization
# instead. That surfaced as a misleading "Could not import module
# 'AutoFeatureExtractor'" — transformers' own generic lazy-import wrapper,
# not a real dependency problem. Awaiting here lets an early-stage load
# (still importing, not yet mid weight-download) finish cleanly before we
# report done; a load that's genuinely deep into a multi-GB download still
# times out — _reset_gpu_pool() below abandons it either way.
#
# 20s, not the original 3s (code-review finding post-merge): a cold
# transformers import alone can take longer than 3s on a slow disk or a
# first-ever launch, so the original bound left a real residual window —
# cancellation detaches the asyncio task, but the underlying OS thread
# keeps running past it, and shutdown could still report "done" while
# that thread was alive. Python cannot forcibly kill a running thread, so
# no finite bound eliminates this outright — 20s just shrinks the window
# from "any preload" to "an unusually slow cold-import," which is the
# practical ceiling before a longer shutdown itself becomes the
# complaint. A thread that's still running past 20s was never going to
# finish in a shutdown-appropriate timeframe regardless.
await _cancel_and_await_tasks(
idle_task, worker_task, preload_task, capture_preload_task, timeout=20.0,
)
# Unload the model and free GPU memory
try:
import services.model_manager as mm
@@ -723,10 +440,6 @@ async def lifespan(app: FastAPI):
mm.model = None
logger.info("Shutdown: model unloaded.")
mm.free_vram()
# Abandon a still-running preload's GPU-pool thread (Python can't kill
# a thread mid blocking call) so it can't outlive this shutdown block
# holding a reference into module state that's about to be torn down.
mm._reset_gpu_pool()
except Exception:
pass
# Run GC to release any remaining references
@@ -758,14 +471,6 @@ app = FastAPI(
@app.get("/docs", include_in_schema=False)
async def scalar_docs():
"""Interactive API documentation powered by Scalar."""
if get_scalar_api_reference is None:
return JSONResponse(
status_code=503,
content={
"detail": "API docs unavailable: scalar-fastapi is not installed "
"in the backend environment (#307)."
},
)
return get_scalar_api_reference(
openapi_url=app.openapi_url,
title=app.title,
@@ -809,17 +514,8 @@ async def global_exception_handler(request: Request, exc: Exception):
headers["Access-Control-Allow-Origin"] = origin
headers["Access-Control-Allow-Credentials"] = "true"
headers["Vary"] = "Origin"
# #874: a model download that failed because the CONFIGURED Hugging Face
# mirror (HF_ENDPOINT) is unreachable used to leak the raw transformers
# message ("We couldn't connect to 'https://hf-mirror.com' …") as the 500
# detail with no next step. #959: same story for the SOCKS-proxy class
# ("Using SOCKS proxy, but the 'socksio' package is not installed").
# Appending the shared hints HERE covers every route that can leak a
# model-load/download error (generate, dub, archetypes, …), not just TTS
# generate. append_hint is a no-op for every other error and never raises.
from core.failure import append_hint
return JSONResponse(
{"detail": append_hint(str(exc)), "error_class": _entry.get("error_class")},
{"detail": str(exc), "error_class": _entry.get("error_class")},
status_code=500,
headers=headers,
)
@@ -883,68 +579,6 @@ class NetworkAccessMiddleware:
return await self.app(scope, receive, send)
class BearerKeyMiddleware:
"""When OMNIVOICE_API_KEY is set, non-loopback clients must present it on
every HTTP + WebSocket request: ``Authorization: Bearer <key>``,
``?api_key=<key>`` (browser WebSockets cannot set headers), or the
``ov_key`` cookie (set on the first successful HTTP auth). Loopback
always bypasses the desktop default is unchanged and the SPA shell
paths stay reachable so a remote UI can load and show what's wrong.
Inert when the env var is unset (the default). Pure ASGI for the same
no-buffering reason as NetworkAccessMiddleware above. Plain-HTTP caveat
is documented in docs/remote-gpu.md: the key is sniffable outside a
WireGuard (Tailscale) or TLS (tailscale serve) transport.
"""
def __init__(self, app):
self.app = app
async def __call__(self, scope, receive, send):
if scope["type"] not in ("http", "websocket"):
return await self.app(scope, receive, send)
key = os.environ.get("OMNIVOICE_API_KEY") or ""
if not key:
return await self.app(scope, receive, send)
client = scope["client"][0] if scope.get("client") else None
if client in _LOOPBACK_CLIENTS:
return await self.app(scope, receive, send)
path = scope.get("path", "")
if scope["type"] == "http" and (
path in _SHELL_PATHS or path.startswith("/assets/") or path.startswith("/favicon")
):
return await self.app(scope, receive, send)
from starlette.requests import HTTPConnection
conn = HTTPConnection(scope)
auth = conn.headers.get("authorization", "")
supplied = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
if not supplied:
supplied = conn.query_params.get("api_key") or conn.cookies.get("ov_key") or ""
if not secrets.compare_digest(supplied, key):
if scope["type"] == "websocket":
# Reject the handshake; 1008 = policy violation.
await receive() # consume websocket.connect
await send({"type": "websocket.close", "code": 1008})
return
resp = JSONResponse({"detail": "API key required"}, status_code=401)
return await resp(scope, receive, send)
if scope["type"] == "http" and conn.cookies.get("ov_key") != key:
async def send_with_cookie(message):
if message["type"] == "http.response.start":
headers = MutableHeaders(scope=message)
headers.append(
"set-cookie", f"ov_key={key}; Path=/; SameSite=Lax"
)
await send(message)
return await self.app(scope, receive, send_with_cookie)
return await self.app(scope, receive, send)
# UI dev-server port — single-sourced from OMNIVOICE_UI_PORT so a user who
# moves the Vite dev server off 3901 still gets a matching CORS allow-list.
def _ui_port() -> int:
@@ -976,29 +610,6 @@ app.add_middleware(
# applied even to the 401 PIN-required responses). Inert unless a PIN is set.
app.add_middleware(NetworkAccessMiddleware)
# Remote-backend bearer gate (parity program Wave 2.3 / §R2). Inert unless
# OMNIVOICE_API_KEY is set. Distinct from the PIN gate above: the PIN guards
# casual LAN-share guests for one session; the API key is the durable
# credential for running this backend remotely (Tailscale / Docker GPU box).
# Covers WebSockets too — the PIN gate never did, because every WS endpoint
# carried its own loopback guard; remote mode is exactly the case where a
# keyed non-loopback client must reach them.
app.add_middleware(BearerKeyMiddleware)
# Register canonical audio MIME types before any StaticFiles mount.
# Python's `mimetypes.guess_type()` returns `audio/x-wav` for `.wav` and
# `audio/x-flac` for `.flac` on most platforms — these are vendor-experimental
# (x- prefix, never IANA-registered). macOS Chrome/Safari MIME-sniff leniently
# via CoreAudio so playback works there, but Linux Chrome/Firefox (FFmpeg) and
# Android Chrome (ExoPlayer) strictly honor the declared type and treat the
# x- variants as download-only — manifesting as the play button silently
# doing nothing in the browser app while working in the Tauri desktop shell.
# `audio/wav` / `audio/flac` are the IANA-canonical types.
# Ref: https://www.iana.org/assignments/media-types/media-types.xhtml#audio
import mimetypes as _mimetypes
_mimetypes.add_type("audio/wav", ".wav")
_mimetypes.add_type("audio/flac", ".flac")
app.mount("/audio", StaticFiles(directory=OUTPUTS_DIR), name="audio")
app.mount("/voice_audio", StaticFiles(directory=VOICES_DIR), name="voice_audio")
@@ -1041,45 +652,17 @@ app.include_router(stories.router)
app.include_router(setup.router)
app.include_router(gallery.router)
app.include_router(archetypes.router)
app.include_router(describe_voice.router) # issue #317: free-text voice design
app.include_router(community.router)
app.include_router(batch.router)
app.include_router(watermark.router)
app.include_router(events.router)
app.include_router(capture.router)
app.include_router(capture_ws.router)
app.include_router(dictation.router)
app.include_router(openai_compat.router)
app.include_router(tts_stream.router)
app.include_router(marketplace.router)
app.include_router(personas.router)
app.include_router(sonitranslate.router)
app.include_router(audiobook.router)
app.include_router(longform_jobs.router)
app.include_router(pronunciation.router) # Expressive-TTS Spec 01: pronunciation dictionary
app.include_router(settings_router.router) # Phase 1 AUTH-03 endpoints
app.include_router(media_tools_router.router) # Settings → Audio tools + wizard media-engine self-heal
from api.routers import mcp_bindings as _mcp_bindings_router # noqa: E402
app.include_router(_mcp_bindings_router.router) # Wave 2.2 per-agent voice bindings
# ── Mount the MCP server (Wave 2.2) ───────────────────────────────────────
# FastMCP's Streamable-HTTP app is sub-mounted at /mcp; its session manager is
# stashed on app.state for the lifespan above to run. Opt-out via
# OMNIVOICE_MCP_DISABLE=1; best-effort so a missing mcp package or a build
# without it never breaks startup.
if os.environ.get("OMNIVOICE_MCP_DISABLE", "").strip().lower() not in ("1", "true", "yes", "on"):
try:
from mcp_server import create_mcp_server
_mcp = create_mcp_server()
_mcp_app = _mcp.streamable_http_app()
app.state.mcp_session_manager = _mcp.session_manager
app.mount("/mcp", _mcp_app)
logging.getLogger("omnivoice.api").info("MCP app mounted at /mcp")
except Exception as _mcp_err: # noqa: BLE001
logging.getLogger("omnivoice.api").info(
"MCP server not mounted (%s); /mcp disabled.", _mcp_err
)
frontend_path = os.path.join(os.path.dirname(__file__), "..", "frontend", "dist")
if os.path.exists(frontend_path):
+1 -64
View File
@@ -53,14 +53,6 @@ def create_mcp_server():
"voice design, and video dubbing in 646 languages."
),
)
# Serve the Streamable-HTTP transport at the app root so mounting the whole
# app at "/mcp" on the main FastAPI yields the endpoint at "/mcp". FastMCP's
# default path is "/mcp", which would double-prefix to "/mcp/mcp" when
# sub-mounted. Harmless for the standalone CLI run() path.
try:
mcp.settings.streamable_http_path = "/"
except Exception:
pass
# ── Helpers ─────────────────────────────────────────────────────────
@@ -83,21 +75,6 @@ def create_mcp_server():
# ── Tools ───────────────────────────────────────────────────────────
def _current_client_id() -> str | None:
"""The X-OmniVoice-Client-Id of the calling MCP client, if any.
FastMCP exposes the HTTP request via its request context on the
Streamable-HTTP transport; stdio clients (and any version where the
accessor differs) simply resolve to None and fall back to the
global default voice."""
try:
req = mcp.get_context().request_context.request
if req is not None:
return req.headers.get("x-omnivoice-client-id")
except Exception:
pass
return None
@mcp.tool()
async def generate_speech(
text: str,
@@ -112,8 +89,7 @@ def create_mcp_server():
Args:
text: The text to synthesize into speech.
language: Target language (ISO code or 'Auto'). 646 languages supported.
profile_id: ID of a saved voice profile to clone. Omit to use this
agent's bound voice (Settings → MCP), else the global default.
profile_id: ID of a saved voice profile to clone. Omit for voice design mode.
instruct: Style instruction (e.g. 'whisper', 'excited', 'narrator').
speed: Speech speed multiplier (0.52.0, default 1.0).
steps: Diffusion steps (8=fast/draft, 16=balanced, 32=quality).
@@ -122,17 +98,6 @@ def create_mcp_server():
JSON with audio_id, generation_time, audio_duration, and
base64-encoded WAV data.
"""
# Per-agent voice binding (Wave 2.2): explicit arg wins; otherwise
# resolve this client's bound profile, then the global default.
client_id = _current_client_id()
try:
from services import mcp_bindings
resolved = mcp_bindings.resolve_voice(client_id, profile_id)
profile_id = resolved.get("profile_id")
mcp_bindings.touch_last_seen(client_id) if client_id else None
except Exception:
pass # binding layer unavailable — use whatever was passed
form = {
"text": text,
"language": language,
@@ -194,34 +159,6 @@ def create_mcp_server():
'],"note":"Pass any ISO 639 code or set language=Auto for detection."}'
)
@mcp.tool()
async def transcribe(audio_base64: str, language: str | None = None) -> str:
"""Transcribe spoken audio to text.
Args:
audio_base64: Base64-encoded audio bytes (wav/mp3/webm/m4a).
language: Optional language hint; omit for auto-detect.
Returns:
JSON with the recognized text, language, and duration.
"""
try:
raw = base64.b64decode(audio_base64, validate=True)
except Exception:
return '{"error":"audio_base64 is not valid base64"}'
# 200 MB cap — same spirit as voicebox's transcribe gate. Keeps a
# buggy/hostile agent from posting an unbounded blob.
if len(raw) > 200 * 1024 * 1024:
return '{"error":"audio exceeds 200 MB limit"}'
data = {}
if language:
data["language"] = language
r = await _api_post_form(
"/transcribe", data=data,
files={"audio": ("audio.wav", raw, "application/octet-stream")},
)
return str(r.json())
@mcp.tool()
async def check_health() -> str:
"""Check if the OmniVoice backend is running and what GPU device is active."""

Some files were not shown because too many files have changed in this diff Show More